2024-03-06T07:10:56.153 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:10:56.153 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --verbose --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583068 --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583068/orig.config.yaml 2024-03-06T07:10:56.197 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:10:56.282 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583068 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583068' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd kclient: syntax: v2 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGnLa4PUoQbbr777PfGO5g31qCUBVETbPZq9txwJ4Lxtdzps2caejRBUHlg+pkJr04pulBsJMdAfxPL0IKh9o14= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuwpLlO9gIMelvUOWA133EQLE3kA0YlFdD17jgdGOnoCBjYoz/fyj8X1WIjxYiwHQ1ZGPdII5hTaDUN2gjmjFA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:10:56.283 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:10:56.283 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:10:56.284 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:10:56.285 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:10:56.309 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:10:56.310 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:10:56.310 INFO:teuthology.packaging:ref: None 2024-03-06T07:10:56.310 INFO:teuthology.packaging:tag: None 2024-03-06T07:10:56.310 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:10:56.310 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:10:56.310 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:10:56.458 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:10:56.460 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:10:56.467 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:10:56.467 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:10:56.474 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:10:56.491 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:10:56.497 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:10:56.523 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583068', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:43.609283', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGnLa4PUoQbbr777PfGO5g31qCUBVETbPZq9txwJ4Lxtdzps2caejRBUHlg+pkJr04pulBsJMdAfxPL0IKh9o14='} 2024-03-06T07:10:56.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583068', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:43.607797', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuwpLlO9gIMelvUOWA133EQLE3kA0YlFdD17jgdGOnoCBjYoz/fyj8X1WIjxYiwHQ1ZGPdII5hTaDUN2gjmjFA='} 2024-03-06T07:10:56.546 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:10:56.552 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:10:56.552 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:10:56.552 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:10:56.623 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9ad5e88f70>, signals=[15]) 2024-03-06T07:10:56.623 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:10:56.629 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:10:56.630 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:10:56.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:56.741 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:10:56.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:56.841 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:10:56.848 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-06T07:10:56.865 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-06T07:10:56.865 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:56.922 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:56.923 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:56.923 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:56.923 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:56.923 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:56.923 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-06T07:10:56.948 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-03-06T07:10:56.965 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-03-06T07:10:56.965 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="9" 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="9" 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:57.022 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:57.023 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:57.023 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-03-06T07:10:57.047 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:10:57.056 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:10:57.063 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:10:57.063 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:57.065 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:57.081 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:10:57.088 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:10:57.088 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:57.122 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:57.224 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:10:57.278 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:10:57.294 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-06T07:10:57.294 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-06T07:10:57.294 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-06T07:10:57.295 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:57.295 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:57.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:10:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:57.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-06T07:10:57.542 DEBUG:teuthology.task.kernel:sha1 for client.1 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:57.542 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:10:57.542 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-03-06T07:10:57.559 INFO:teuthology.orchestra.run.smithi178.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:57.559 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:57.560 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:10:57.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum upgrade -y linux-firmware 2024-03-06T07:10:57.566 DEBUG:teuthology.task.kernel:sha1 for client.0 is b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:57.566 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "b47552b3c0e3993e47c2c21f2f0d35fc872144cc"... 2024-03-06T07:10:57.566 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-06T07:10:57.584 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:57.584 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:10:57.584 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-06T07:10:57.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum upgrade -y linux-firmware 2024-03-06T07:10:59.724 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-06T07:10:59.788 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-06T07:11:01.099 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-06T07:11:01.222 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-06T07:11:12.145 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 777 kB/s | 8.0 MB 00:10 2024-03-06T07:11:12.688 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 731 kB/s | 8.0 MB 00:11 2024-03-06T07:11:33.803 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 961 kB/s | 19 MB 00:20 2024-03-06T07:11:38.768 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 754 kB/s | 19 MB 00:25 2024-03-06T07:11:38.861 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-06T07:11:40.537 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-03-06T07:11:42.805 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-06T07:11:47.780 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-06T07:11:50.079 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:11:50.088 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-06T07:11:50.089 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-03-06T07:11:50.089 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 1 Package 2024-03-06T07:11:50.089 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:11:50.089 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 375 M 2024-03-06T07:11:50.089 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-06T07:11:55.193 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 491 kB/s | 6.1 MB 00:12 2024-03-06T07:11:56.902 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-06T07:12:00.056 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-03-06T07:12:05.120 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-03-06T07:12:07.459 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-06T07:12:07.473 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-06T07:12:07.473 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-06T07:12:07.473 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-06T07:12:07.473 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-06T07:12:07.473 INFO:teuthology.orchestra.run.smithi006.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 1 Package 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 375 M 2024-03-06T07:12:07.474 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-06T07:12:17.029 INFO:teuthology.orchestra.run.smithi006.stdout:linux-firmware-20240219-143.el9.noarch.rpm 39 MB/s | 375 MB 00:09 2024-03-06T07:12:17.030 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:12:17.031 INFO:teuthology.orchestra.run.smithi006.stdout:Total 39 MB/s | 375 MB 00:09 2024-03-06T07:12:17.031 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-06T07:12:17.122 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-06T07:12:17.122 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-06T07:12:19.617 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-06T07:12:19.617 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-06T07:12:26.009 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-06T07:12:26.251 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:26.340 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:27.458 INFO:teuthology.orchestra.run.smithi178.stdout:linux-firmware-20240219-143.el9.noarch.rpm 10 MB/s | 375 MB 00:37 2024-03-06T07:12:27.459 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:12:27.460 INFO:teuthology.orchestra.run.smithi178.stdout:Total 10 MB/s | 375 MB 00:37 2024-03-06T07:12:27.460 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-06T07:12:27.545 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:27.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:27.560 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-06T07:12:27.560 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-06T07:12:27.948 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:27.948 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:12:27.948 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-06T07:12:27.948 INFO:teuthology.orchestra.run.smithi006.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:12:27.948 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:12:27.948 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-06T07:12:28.234 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:12:28.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:12:28.388 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:12:28.388 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:12:28.391 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi006.front.sepia.ceph.com to complete... 2024-03-06T07:12:29.945 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-06T07:12:29.945 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-06T07:12:36.258 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-06T07:12:36.459 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:36.549 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:37.513 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:37.514 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-06T07:12:37.900 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-06T07:12:37.900 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:12:37.900 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-03-06T07:12:37.900 INFO:teuthology.orchestra.run.smithi178.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-06T07:12:37.900 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:12:37.900 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-06T07:12:38.230 INFO:teuthology.task.kernel:Downloading kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:12:38.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b47552b3c0e3993e47c2c21f2f0d35fc872144cc 2024-03-06T07:12:38.404 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ 2024-03-06T07:12:38.404 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-06T07:12:38.407 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi178.front.sepia.ceph.com to complete... 2024-03-06T07:14:25.423 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06 07:14:25 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:14:25.423 INFO:teuthology.orchestra.run.smithi178.stderr:FINISHED --2024-03-06 07:14:25-- 2024-03-06T07:14:25.423 INFO:teuthology.orchestra.run.smithi178.stderr:Total wall clock time: 1m 47s 2024-03-06T07:14:25.423 INFO:teuthology.orchestra.run.smithi178.stderr:Downloaded: 1 files, 320M in 1m 46s (3.01 MB/s) 2024-03-06T07:14:25.427 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.1... 2024-03-06T07:14:25.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:14:27.353 INFO:teuthology.orchestra.run.smithi178.stdout:Verifying... ######################################## 2024-03-06T07:14:27.686 INFO:teuthology.orchestra.run.smithi178.stdout:Preparing... ######################################## 2024-03-06T07:14:27.764 INFO:teuthology.orchestra.run.smithi178.stdout:Updating / installing... 2024-03-06T07:14:35.228 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:14:49.759 INFO:teuthology.orchestra.run.smithi178.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:08.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06 07:15:08 URL:https://3.chacra.ceph.com/r/kernel/testing/b47552b3c0e3993e47c2c21f2f0d35fc872144cc/centos/9/flavors/default/x86_64/kernel-6.8.0_rc6_gb47552b3c0e3-1.x86_64.rpm [336022261/336022261] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-06T07:15:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:FINISHED --2024-03-06 07:15:08-- 2024-03-06T07:15:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:Total wall clock time: 2m 40s 2024-03-06T07:15:08.698 INFO:teuthology.orchestra.run.smithi006.stderr:Downloaded: 1 files, 320M in 2m 40s (2.01 MB/s) 2024-03-06T07:15:08.704 INFO:teuthology.task.kernel:Installing kernel b47552b3c0e3993e47c2c21f2f0d35fc872144cc on client.0... 2024-03-06T07:15:08.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-06T07:15:10.740 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying... ######################################## 2024-03-06T07:15:10.811 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing... ######################################## 2024-03-06T07:15:10.895 INFO:teuthology.orchestra.run.smithi006.stdout:Updating / installing... 2024-03-06T07:15:12.307 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:15:12.307 DEBUG:teuthology.orchestra.run.smithi178:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:15:12.384 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:15:12.385 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:15:12.386 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:15:12.387 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:15:12.388 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:15:12.389 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:15:12.390 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:15:12.390 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:15:12.390 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:15:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:15:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:15:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:15:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:15:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:15:12.393 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:15:12.394 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:15:12.395 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:15:12.396 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:15:12.397 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:15:12.398 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:15:12.399 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:15:12.400 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:15:12.401 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:15:12.402 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:15:12.403 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:15:12.404 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:15:12.405 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:15:12.406 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:15:12.407 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:15:12.408 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:15:12.409 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:15:12.410 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:15:12.411 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:15:12.412 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:15:12.413 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:15:12.414 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:15:12.415 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:15:12.416 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:15:12.417 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:15:12.418 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:15:12.419 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:15:12.420 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:15:12.421 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:15:12.422 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:15:12.423 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:15:12.424 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:15:12.425 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:15:12.426 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:15:12.427 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:15:12.428 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:15:12.429 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:15:12.430 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:15:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:15:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:15:12.433 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:15:12.434 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:15:12.435 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:15:12.436 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:15:12.437 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:15:12.438 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:15:12.439 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:15:12.440 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:15:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:15:12.442 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:15:12.443 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:15:12.444 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:15:12.445 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:15:12.446 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:15:12.447 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:15:12.448 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:15:12.449 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:15:12.450 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:15:12.451 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:15:12.452 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:15:12.453 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:15:12.454 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:15:12.455 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:15:12.456 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:15:12.457 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:15:12.458 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:15:12.459 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:15:12.460 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:15:12.461 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:15:12.462 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:15:12.463 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:15:12.464 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:15:12.465 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:15:12.466 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:15:12.467 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:15:12.468 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:15:12.469 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:15:12.470 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:15:12.471 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:15:12.472 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:15:12.473 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:15:12.474 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:15:12.475 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:15:12.476 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:15:12.477 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:15:12.478 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:15:12.479 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:15:12.480 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:15:12.481 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:15:12.482 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:15:12.483 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:15:12.484 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:15:12.485 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:15:12.486 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:15:12.487 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:15:12.488 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:15:12.489 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:15:12.490 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:15:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:15:12.492 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:15:12.493 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:15:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:15:12.495 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:15:12.496 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:15:12.497 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:15:12.498 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:15:12.499 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:15:12.500 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:15:12.501 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:15:12.502 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:15:12.503 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:15:12.504 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:15:12.505 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:15:12.506 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:15:12.507 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:15:12.508 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:15:12.509 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:15:12.510 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:15:12.511 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:15:12.512 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:15:12.513 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:15:12.514 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:15:12.515 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:15:12.516 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:15:12.517 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:15:12.518 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:15:12.519 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:15:12.520 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:15:12.521 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:15:12.522 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:15:12.523 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:15:12.524 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:15:12.525 INFO:teuthology.orchestra.run.smithi178.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:12.529 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:12.529 DEBUG:teuthology.orchestra.run.smithi178:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:12.551 INFO:teuthology.orchestra.run.smithi178.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:fi 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout: fi 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:done 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:12.552 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:12.553 INFO:teuthology.orchestra.run.smithi178.stdout:fi 2024-03-06T07:15:12.553 INFO:teuthology.orchestra.run.smithi178.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:12.553 INFO:teuthology.orchestra.run.smithi178.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:15:12.553 INFO:teuthology.orchestra.run.smithi178.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:12.553 INFO:teuthology.orchestra.run.smithi178.stdout:fi 2024-03-06T07:15:12.554 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2024-03-06T07:15:12.628 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2024-03-06T07:15:12.628 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.06 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 68.el9 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:Group : Unspecified 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 8264254 2024-03-06T07:15:12.629 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2024-03-06T07:15:12.630 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:15:12.630 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:15:12.630 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:15:12.630 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:15:12.630 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : builder@centos.org 2024-03-06T07:15:12.630 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:15:12.631 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:15:12.632 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2024-03-06T07:15:12.632 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:15:12.632 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:15:12.633 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:15:12.633 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:12.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:15:13.151 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-03-06T07:15:14.555 INFO:teuthology.orchestra.run.smithi178.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:15:14.584 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-03-06T07:15:14.596 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-03-06T07:15:14.611 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.9864XH3qER 2024-03-06T07:15:14.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9864XH3qER 2024-03-06T07:15:14.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.9864XH3qER 2024-03-06T07:15:14.831 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.9864XH3qER is 6KB 2024-03-06T07:15:14.879 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.9864XH3qER 2024-03-06T07:15:14.893 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:15:14.967 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:15:14.967 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:15:14.968 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:15:14.968 INFO:teuthology.orchestra.run.smithi178.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:15:14.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:15.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-03-06T07:15:18.728 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-6.8.0_rc6_gb47552b3c0e3-1 ######################################## 2024-03-06T07:15:33.185 INFO:teuthology.orchestra.run.smithi006.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:15:45.119 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-06T07:15:45.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-06T07:15:45.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:15:56.488 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-06T07:15:56.489 DEBUG:teuthology.orchestra.run.smithi006:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:56.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:56.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/System.map 2024-03-06T07:15:56.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/config 2024-03-06T07:15:56.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel 2024-03-06T07:15:56.574 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-06T07:15:56.575 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce 2024-03-06T07:15:56.576 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/arch/x86/kvm/kvm.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ansi_cprng.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/anubis.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/arc4.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_memcpy.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_pq.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-06T07:15:56.577 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_tx.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/async_xor.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/async_tx/raid6test.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authenc.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/authencesn.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blake2b_generic.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_common.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/blowfish_generic.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/camellia_generic.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast5_generic.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast6_generic.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cast_common.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/ccm.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cmac.ko 2024-03-06T07:15:56.578 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crc32_generic.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_engine.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/crypto_user.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/cts.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/des_generic.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/dh_generic.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/echainiv.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/essiv.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/fcrypt.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/khazad.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/lz4hc.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/md4.ko 2024-03-06T07:15:56.579 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/michael_mic.ko 2024-03-06T07:15:56.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcbc.ko 2024-03-06T07:15:56.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/pcrypt.ko 2024-03-06T07:15:56.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/rmd160.ko 2024-03-06T07:15:56.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/seed.ko 2024-03-06T07:15:56.580 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/serpent_generic.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tcrypt.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/tea.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_common.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/twofish_generic.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/vmac.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/wp512.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xcbc.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xor.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/crypto/xxhash_generic.ko 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers 2024-03-06T07:15:56.581 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/acpi_pad.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/custom_method.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/ec_sys.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/platform_profile.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbs.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/sbshc.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/acpi/video.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/acard-ahci.ko 2024-03-06T07:15:56.582 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ahci_platform.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/ata_generic.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/libahci_platform.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_acpi.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ali.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_amd.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_artop.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atiixp.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_atp867x.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd640.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cmd64x.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_cypress.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_efar.ko 2024-03-06T07:15:56.583 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt366.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt37x.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it8213.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_it821x.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_jmicron.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_marvell.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_mpiix.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_netcell.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ninja32.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87410.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_ns87415.ko 2024-03-06T07:15:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_oldpiix.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_opti.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_optidma.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pcmcia.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_piccolo.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_rdc.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sch.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_serverworks.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sil680.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sis.ko 2024-03-06T07:15:56.585 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_sl82c105.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_triflex.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pata_via.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/pdc_adma.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_inic162x.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_mv.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_nv.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_promise.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_qstor.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil.ko 2024-03-06T07:15:56.586 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sil24.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sis.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_svw.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_sx4.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_uli.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_via.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ata/sata_vsc.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/auxdisplay/ks0108.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-06T07:15:56.587 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/bcma/bcma.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/aoe/aoe.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/brd.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/drbd/drbd.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/floppy.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/loop.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/nbd.ko 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk 2024-03-06T07:15:56.588 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/null_blk/null_blk.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/pktcdvd.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/rbd.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/virtio_blk.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/xen-blkfront.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/block/zram/zram.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hangcheck-timer.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-06T07:15:56.589 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/via-rng.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/lp.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/mwave/mwave.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/ppdev.ko 2024-03-06T07:15:56.590 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tlclk.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/uv_mmtimer.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/char/virtio_console.ko 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq 2024-03-06T07:15:56.591 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-06T07:15:56.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-06T07:15:56.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-06T07:15:56.592 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-06T07:15:56.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-06T07:15:56.595 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/ccp/ccp.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-aes.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/padlock-sha.ko 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio 2024-03-06T07:15:56.596 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dca/dca.ko 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-06T07:15:56.597 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/amd64_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/e752x_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/edac_mce_amd.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3000_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i3200_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5100_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i5400_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7300_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i7core_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/i82975x_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/ie31200_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/sb_edac.ko 2024-03-06T07:15:56.598 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/edac/x38_edac.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/edd.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-ich.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-appleir.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-aureal.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-axff.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-dr.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elecom.ko 2024-03-06T07:15:56.599 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-elo.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-emsff.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gaff.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gt683r.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-gyration.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-holtekff.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-hyperv.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-icade.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-keytouch.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-kye.ko 2024-03-06T07:15:56.600 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lcpower.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-led.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-lenovo.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-multitouch.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-ortek.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-penmount.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-petalynx.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-picolcd.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-pl.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-plantronics.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-primax.ko 2024-03-06T07:15:56.601 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-rmi.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-common.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-roccat.ko 2024-03-06T07:15:56.602 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-saitek.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-samsung.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sjoy.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sony.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-speedlink.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-steelseries.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-sunplus.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tivo.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-tmff.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-topseed.ko 2024-03-06T07:15:56.603 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-twinhan.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-uclogic.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-waltop.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-wiimote.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-xinmo.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zpff.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/hid-zydacron.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/uhid.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hid/wacom.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_balloon.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_utils.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hv/hv_vmbus.ko 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon 2024-03-06T07:15:56.604 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/abituguru3.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7414.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ad7418.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adc128d818.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1025.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1026.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1029.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm1031.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adm9240.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ads7828.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7410.ko 2024-03-06T07:15:56.605 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7411.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7462.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7470.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7475.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/adt7x10.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/amc6821.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/applesmc.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asb100.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asc7621.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/atxp1.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/coretemp.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-06T07:15:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/dme1737.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds1621.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ds620.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc1403.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/emc6w201.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71805f.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f71882fg.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/f75375s.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fam15h_power.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/fschmd.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g760a.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/g762.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl518sm.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/gl520sm.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5500_temp.ko 2024-03-06T07:15:56.607 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/i5k_amb.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmaem.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ibmpex.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina209.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ina2xx.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/it87.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k10temp.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/k8temp.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lineage-pem.ko 2024-03-06T07:15:56.608 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm63.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm73.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm75.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm77.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm78.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm80.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm83.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm85.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm87.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm90.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm92.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm93.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95234.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95241.ko 2024-03-06T07:15:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/lm95245.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc2945.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4151.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4215.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4222.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4245.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4260.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/ltc4261.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max16065.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1619.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max1668.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max197.ko 2024-03-06T07:15:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6639.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6650.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/max6697.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/mcp3021.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6683.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775-core.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct6775.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/nct7802.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87360.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pc87427.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pcf8591.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-06T07:15:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/powr1220.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5627.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch5636.ko 2024-03-06T07:15:56.612 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht15.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sht21.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/shtc1.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/sis5595.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47b397.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m1.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/smsc47m192.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/thmc50.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp102.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp103.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp401.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/tmp421.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via-cputemp.ko 2024-03-06T07:15:56.613 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/via686a.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt1211.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/vt8231.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627ehf.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83627hf.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83781d.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83791d.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83792d.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83793.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83795.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l785ts.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/hwmon/w83l786ng.ko 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos 2024-03-06T07:15:56.614 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-06T07:15:56.615 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-dev.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-mux.ko 2024-03-06T07:15:56.616 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-smbus.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/i2c/i2c-stub.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_core.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-06T07:15:56.617 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp 2024-03-06T07:15:56.618 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/gameport/gameport.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/joydev.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/appletouch.ko 2024-03-06T07:15:56.619 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/bcm5974.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/cyapatp.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/sermouse.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/altera_ps2.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/arc_ps2.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/serio/serio_raw.ko 2024-03-06T07:15:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/input/sparse-keymap.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-blinkm.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lm3530.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-lp3944.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/leds-ss4200.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-06T07:15:56.621 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/bcache/bcache.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-bio-prison.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache-smq.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-cache.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-crypt.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-delay.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-flakey.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-log-userspace.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-multipath.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-queue-length.ko 2024-03-06T07:15:56.622 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-raid.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-round-robin.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-service-time.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-switch.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-thin-pool.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/dm-verity.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid0.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid1.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid10.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/md/raid456.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptbase.ko 2024-03-06T07:15:56.623 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptctl.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptfc.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptlan.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptsas.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptscsih.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/message/fusion/mptspi.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_ich.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/lpc_sch.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/mfd-core.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/sm501.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/viperboard.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/vx855.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/mfd/wl1273-core.ko 2024-03-06T07:15:56.624 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds9802als.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/apds990x.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/bh1770glc.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/cb710/cb710.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/at24.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/eeprom/max6875.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/enclosure.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/hpilo.ko 2024-03-06T07:15:56.625 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29003.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/isl29020.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-me.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei-txe.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/mei/mei.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-06T07:15:56.626 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_7xx1.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tifm_core.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/tsl2550.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_balloon.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/bonding/bonding.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-06T07:15:56.627 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/dummy.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/eql.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-06T07:15:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros 2024-03-06T07:15:56.629 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-06T07:15:56.630 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-06T07:15:56.631 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink 2024-03-06T07:15:56.632 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/dnet.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ethoc.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/fealnx.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-06T07:15:56.633 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/jme.ko 2024-03-06T07:15:56.634 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-06T07:15:56.635 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-06T07:15:56.636 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan 2024-03-06T07:15:56.637 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-06T07:15:56.638 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-06T07:15:56.639 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/geneve.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ifb.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvlan.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/macvtap.ko 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio 2024-03-06T07:15:56.640 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio.ko 2024-03-06T07:15:56.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-06T07:15:56.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-06T07:15:56.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-06T07:15:56.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/mii.ko 2024-03-06T07:15:56.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/net_failover.ko 2024-03-06T07:15:56.641 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/netconsole.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/nlmon.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/amd.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/bcm87xx.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/broadcom.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/cicada.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/davicom.ko 2024-03-06T07:15:56.642 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/dp83640.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/et1011c.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/icplus.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/lxt.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/marvell.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/micrel.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/national.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/phylink.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/qsemi.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/realtek.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/smsc.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/ste10Xp.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/phy/vitesse.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-06T07:15:56.643 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_async.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppoe.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pppox.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/ppp/pptp.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slhc.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/slip/slip.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/sungem_phy.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tap.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team.ko 2024-03-06T07:15:56.644 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_random.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/tun.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/veth.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/virtio_net.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/vxlan/vxlan.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-06T07:15:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/net/xen-netfront.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ntb/ntb.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme-core.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/nvme/host/nvme.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_cs.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_pc.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/parport/parport_serial.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-06T07:15:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pci/xen-pcifront.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/i82092.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/pd6729.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/acerhdf.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-06T07:15:56.647 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/hdaps.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/intel_ips.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-06T07:15:56.648 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/platform/x86/wmi.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/clients/pps_parport.ko 2024-03-06T07:15:56.649 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/pps/pps_core.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ptp/ptp_kvm.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-06T07:15:56.650 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-em3027.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-max6900.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-06T07:15:56.651 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/rtc/rtc-x1205.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-9xxx.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-sas.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/3w-xxxx.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/BusLogic.ko 2024-03-06T07:15:56.652 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/a100u2w.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/advansys.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/am53c974.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/atp870u.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bfa/bfa.ko 2024-03-06T07:15:56.653 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ch.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dc395x.ko 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler 2024-03-06T07:15:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/dmx3191d.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/esp_scsi.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/fnic/fnic.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hpsa.ko 2024-03-06T07:15:56.655 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hptiop.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/hv_storvsc.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/initio.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ipr.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ips.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/isci/isci.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libfc/libfc.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas 2024-03-06T07:15:56.656 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/libsas/libsas.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/mvumi.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/pmcraid.ko 2024-03-06T07:15:56.657 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla1280.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/raid_class.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_debug.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/ses.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/st.ko 2024-03-06T07:15:56.658 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/stex.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/virtio_scsi.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/scsi/wd719x.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ssb/ssb.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_file.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_iblock.ko 2024-03-06T07:15:56.659 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_mod.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_pscsi.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/target_core_user.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-06T07:15:56.660 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_gsm.ko 2024-03-06T07:15:56.661 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/n_hdlc.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/nozomi.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/arc_uart.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio.ko 2024-03-06T07:15:56.662 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_aec.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_cif.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_pci_generic.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/uio/uio_sercos3.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-acm.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usblp.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/class/usbtmc.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/mdc800.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/image/microtek.ko 2024-03-06T07:15:56.663 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/adutux.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/appledisplay.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi26.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/emi62.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ezusb.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/idmouse.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/iowarrior.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/ldusb.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/legousbtower.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-06T07:15:56.664 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usb3503.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usblcd.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/uss720.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/misc/yurex.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/phy/phy-generic.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/aircable.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ark3116.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-06T07:15:56.665 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ch341.ko 2024-03-06T07:15:56.666 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cp210x.ko 2024-03-06T07:15:56.666 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cyberjack.ko 2024-03-06T07:15:56.666 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/empeg.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/io_ti.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipaq.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ipw.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ir-usb.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-06T07:15:56.667 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mct_u232.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7720.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/mos7840.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/navman.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/omninet.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/opticon.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/option.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/oti6858.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/pl2303.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcaux.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/qcserial.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/quatech2.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/safe_serial.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/sierra.ko 2024-03-06T07:15:56.668 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ssu100.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/symbolserial.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_debug.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/visor.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/whiteheat.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/uas.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-06T07:15:56.669 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-karma.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/storage/usb-storage.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-06T07:15:56.670 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_net.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/vhost/vhost_scsi.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/apple_bl.ko 2024-03-06T07:15:56.671 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/lcd.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_balloon.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_mmio.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/virtio/virtio_ring.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2482.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/masters/ds2490.ko 2024-03-06T07:15:56.672 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/w1/wire.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-06T07:15:56.673 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/hpwdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/i6300esb.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ib700wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ibmasr.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/it87_wdt.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/machzwd.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/nv_tco.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-06T07:15:56.674 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/softdog.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/via_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/wdt_pci.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/watchdog/xen_wdt.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-evtchn.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntalloc.ko 2024-03-06T07:15:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-gntdev.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xen-privcmd.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/9p/9p.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/affs/affs.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/befs/befs.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/binfmt_misc.ko 2024-03-06T07:15:56.676 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/btrfs/btrfs.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cachefiles/cachefiles.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ceph/ceph.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/coda/coda.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/cramfs/cramfs.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/dlm/dlm.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-06T07:15:56.677 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/f2fs/f2fs.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/fat.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/msdos.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fat/vfat.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/cuse.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/fuse/fuse.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/gfs2/gfs2.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfs/hfs.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/hfsplus/hfsplus.ko 2024-03-06T07:15:56.678 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/isofs/isofs.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/jfs/jfs.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/lockd/lockd.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/minix/minix.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/netfs/netfs.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-06T07:15:56.679 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfs.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv3.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs/nfsv4.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/grace.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfs_common/nfs_acl.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nfsd/nfsd.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nilfs2/nilfs2.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-celtic.ko 2024-03-06T07:15:56.680 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-centeuro.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-croatian.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-cyrillic.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-gaelic.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-greek.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-iceland.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-inuit.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-roman.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-romanian.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/mac-turkish.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1250.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1251.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp1255.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp737.ko 2024-03-06T07:15:56.681 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp775.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp850.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp852.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp855.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp857.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp860.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp861.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp862.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp863.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp864.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp865.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp866.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp869.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp874.ko 2024-03-06T07:15:56.682 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp932.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp936.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp949.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_cp950.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_euc-jp.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-1.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-13.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-14.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-15.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-2.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-3.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-4.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-5.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-6.ko 2024-03-06T07:15:56.683 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-7.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_iso8859-9.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-r.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-ru.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_koi8-u.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/nls/nls_utf8.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-06T07:15:56.684 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/overlayfs/overlay.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/pstore/ramoops.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/reiserfs/reiserfs.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/romfs/romfs.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/client/cifs.ko 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common 2024-03-06T07:15:56.685 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_arc4.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/smb/common/cifs_md4.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/squashfs/squashfs.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/sysv/sysv.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/udf/udf.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/ufs/ufs.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/fs/xfs/xfs.ko 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu 2024-03-06T07:15:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/rcu/rcutorture.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/torture.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/asn1_encoder.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc-itu-t.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crc8.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libarc4.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/crypto/libdes.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/libcrc32c.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lru_cache.ko 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4 2024-03-06T07:15:56.687 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4_compress.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/lz4/lz4hc_compress.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/math/cordic.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/raid6/raid6_pq.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_bm.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_fsm.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/ts_kmp.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/lib/zstd/zstd_compress.ko 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm 2024-03-06T07:15:56.688 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/mm/hwpoison-inject.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/6lowpan.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_dest.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_fragment.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_hop.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_mobility.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_routing.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/6lowpan/nhc_udp.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/garp.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/mrp.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/p8022.ko 2024-03-06T07:15:56.689 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/psnap.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/802/stp.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/8021q/8021q.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_fd.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_rdma.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/9p/9pnet_virtio.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/batman-adv/batman-adv.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/br_netfilter.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/bridge.ko 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter 2024-03-06T07:15:56.690 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-06T07:15:56.691 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/ebtables.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ceph/libceph.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/failover.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/core/pktgen.ko 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp 2024-03-06T07:15:56.692 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_diag.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv4.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dccp/dccp_ipv6.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dns_resolver/dns_resolver.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/dsa_core.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_brcm.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_dsa.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_none.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/dsa/tag_trailer.ko 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan 2024-03-06T07:15:56.693 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ah4.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/esp4.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/fou.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/gre.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/inet_diag.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_gre.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_tunnel.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ip_vti.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipcomp.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/ipip.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-06T07:15:56.694 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-06T07:15:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_bic.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_dctcp.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_diag.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_highspeed.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_htcp.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_hybla.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_illinois.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_lp.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_scalable.ko 2024-03-06T07:15:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_vegas.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_veno.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_westwood.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tcp_yeah.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/tunnel4.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_diag.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/udp_tunnel.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ah6.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/esp6.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/fou6.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_tunnel.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ip6_vti.ko 2024-03-06T07:15:56.697 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/ipcomp6.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-06T07:15:56.698 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/sit.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/tunnel6.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-06T07:15:56.699 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/key/af_key.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_core.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_eth.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ip6.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_netlink.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/l2tp/l2tp_ppp.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/llc/llc.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mac802154/mac802154.ko 2024-03-06T07:15:56.700 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/mpls/mpls_gso.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-06T07:15:56.701 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-06T07:15:56.702 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conncount.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-06T07:15:56.703 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_log_syslog.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat.ko 2024-03-06T07:15:56.704 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_irc.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_sip.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nf_tables.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_log.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_chain_nat.ko 2024-03-06T07:15:56.708 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_compat.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_ct.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fib.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_hash.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_limit.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_log.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_masq.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_nat.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_numgen.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_queue.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_quota.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_redir.ko 2024-03-06T07:15:56.709 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_reject_inet.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_socket.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/nft_tunnel.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_AUDIT.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_CT.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_DSCP.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HL.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_HMARK.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LED.ko 2024-03-06T07:15:56.710 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_LOG.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NETMAP.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFLOG.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_RATEEST.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_SECMARK.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TEE.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TPROXY.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_TRACE.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_addrtype.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_bpf.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cgroup.ko 2024-03-06T07:15:56.711 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cluster.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_comment.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connbytes.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlabel.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connlimit.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_connmark.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_conntrack.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_cpu.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dccp.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_devgroup.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_dscp.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ecn.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_esp.ko 2024-03-06T07:15:56.712 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hashlimit.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_helper.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_hl.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipcomp.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_iprange.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_ipvs.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_l2tp.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_length.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_limit.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mac.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_mark.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_multiport.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nat.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_nfacct.ko 2024-03-06T07:15:56.713 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_osf.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_owner.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_physdev.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_pkttype.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_policy.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_quota.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_rateest.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_realm.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_recent.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_sctp.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_set.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_socket.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_state.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_statistic.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_string.ko 2024-03-06T07:15:56.714 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_tcpmss.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_time.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netfilter/xt_u32.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/netlink/netlink_diag.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/nsh/nsh.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/openvswitch.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-geneve.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-gre.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/openvswitch/vport-vxlan.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/packet/af_packet_diag.ko 2024-03-06T07:15:56.715 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds 2024-03-06T07:15:56.716 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds.ko 2024-03-06T07:15:56.716 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_rdma.ko 2024-03-06T07:15:56.716 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/rds/rds_tcp.ko 2024-03-06T07:15:56.716 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched 2024-03-06T07:15:56.717 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_csum.ko 2024-03-06T07:15:56.717 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_gact.ko 2024-03-06T07:15:56.717 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_mirred.ko 2024-03-06T07:15:56.717 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_nat.ko 2024-03-06T07:15:56.717 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_pedit.ko 2024-03-06T07:15:56.718 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_police.ko 2024-03-06T07:15:56.719 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_simple.ko 2024-03-06T07:15:56.719 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_skbedit.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/act_vlan.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_basic.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_bpf.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_flow.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_fw.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_route.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/cls_u32.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_cmp.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_ipset.ko 2024-03-06T07:15:56.720 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_meta.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_nbyte.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_text.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/em_u32.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_choke.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_codel.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_drr.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_fq.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_gred.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hfsc.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_hhf.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_htb.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_ingress.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio.ko 2024-03-06T07:15:56.721 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_mqprio_lib.ko 2024-03-06T07:15:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_multiq.ko 2024-03-06T07:15:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_netem.ko 2024-03-06T07:15:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_pie.ko 2024-03-06T07:15:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_plug.ko 2024-03-06T07:15:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_prio.ko 2024-03-06T07:15:56.722 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_qfq.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_red.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfb.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_sfq.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_tbf.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sched/sch_teql.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sctp/sctp_diag.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/sunrpc.ko 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma 2024-03-06T07:15:56.723 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-06T07:15:56.724 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc 2024-03-06T07:15:56.724 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/diag.ko 2024-03-06T07:15:56.724 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/tipc/tipc.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/unix/unix_diag.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys 2024-03-06T07:15:56.725 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/security/keys/trusted-keys/trusted.ko 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/kernel/virt/lib/irqbypass.ko 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.alias.bin 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.alias.bin 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.bin 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.builtin.modinfo 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.dep.bin 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.devname 2024-03-06T07:15:56.726 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.order 2024-03-06T07:15:56.727 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.softdep 2024-03-06T07:15:56.727 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols 2024-03-06T07:15:56.727 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/modules.symbols.bin 2024-03-06T07:15:56.727 INFO:teuthology.orchestra.run.smithi006.stdout:/lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:56.732 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:56.732 DEBUG:teuthology.orchestra.run.smithi006:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-06T07:15:56.758 INFO:teuthology.orchestra.run.smithi006.stdout:postinstall scriptlet (using /bin/sh): 2024-03-06T07:15:56.758 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:56.758 INFO:teuthology.orchestra.run.smithi006.stdout: /usr/bin/kernel-install add 6.8.0-rc6-gb47552b3c0e3 /lib/modules/6.8.0-rc6-gb47552b3c0e3/vmlinuz 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:for file in vmlinuz System.map config; do 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3"; then 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout: cp "/lib/modules/6.8.0-rc6-gb47552b3c0e3/${file}" "/boot/${file}-6.8.0-rc6-gb47552b3c0e3" 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout: fi 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-06T07:15:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:new-kernel-pkg --remove 6.8.0-rc6-gb47552b3c0e3 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc6-gb47552b3c0e3.img 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-install remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/update-bootloader --remove 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:56.760 INFO:teuthology.orchestra.run.smithi006.stdout:fi 2024-03-06T07:15:56.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-06T07:15:56.856 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-06T07:15:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-06T07:15:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-06T07:15:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-06T07:15:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-06T07:15:56.857 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:15:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-06T07:15:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-06T07:15:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-06T07:15:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:15:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:15:56.858 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:15:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:15:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-06T07:15:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-06T07:15:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:15:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-06T07:15:56.859 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:15:56.860 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:15:56.862 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:15:56.862 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:56.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:15:57.454 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-06T07:15:58.845 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:15:58.889 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-06T07:15:58.891 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-06T07:15:58.907 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.D6DOQvDMyu 2024-03-06T07:15:58.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.D6DOQvDMyu 2024-03-06T07:15:58.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.D6DOQvDMyu 2024-03-06T07:15:59.130 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.D6DOQvDMyu is 6KB 2024-03-06T07:15:59.181 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.D6DOQvDMyu 2024-03-06T07:15:59.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:15:59.265 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:15:59.266 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:15:59.266 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:15:59.266 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3.conf 2024-03-06T07:15:59.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:15:59.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-06T07:16:19.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-03-06T07:16:28.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-06T07:16:28.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:16:29.413 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-06T07:16:29.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-06T07:16:29.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:28.713 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:17:29.416 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:17:38.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-06T07:17:38.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:38.859 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-06T07:17:39.609 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-06T07:17:39.609 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:17:39.609 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-06T07:17:39.648 INFO:teuthology.orchestra.run.smithi006.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:39.648 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:39.648 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:17:39.648 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:17:40.652 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:40.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-06T07:17:40.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:41.210 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-06T07:17:41.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-06T07:17:41.818 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc6-gb47552b3c0e3"... 2024-03-06T07:17:41.818 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-03-06T07:17:41.868 INFO:teuthology.orchestra.run.smithi178.stdout:6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:41.869 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc6-gb47552b3c0e3 vs 6.8.0-rc6-gb47552b3c0e3 2024-03-06T07:17:41.869 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:17:41.869 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:17:42.870 DEBUG:teuthology.parallel:result is None 2024-03-06T07:17:42.871 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:17:42.881 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:17:42.882 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:17:42.891 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:17:42.946 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:17:42.953 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:17:42.961 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:17:42.962 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:17:42.966 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:17:43.036 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:17:43.042 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:17:43.043 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:17:43.047 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:17:43.164 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:43.186 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:43.265 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:43.269 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:17:43.275 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:17:43.282 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:17:43.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:17:43.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:17:43.384 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:17:43.393 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:17:43.394 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:17:43.398 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:17:43.452 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:43.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:43.616 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:43.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:43.774 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:17:43.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:17:43.873 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:43.957 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:17:44.073 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:44.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:17:44.254 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:17:44.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:17:44.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-06T07:17:44.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-03-06T07:17:44.505 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:17:44.507 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:17:44.967 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:17:44.976 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:17:44.976 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:17:44.992 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:17:45.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-06T07:17:45.109 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-06T07:17:45.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-03-06T07:17:45.250 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-03-06T07:17:45.255 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:17:45.255 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-06T07:17:45.305 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-06T07:17:45.306 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-03-06T07:17:45.387 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-03-06T07:17:45.388 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:17:45.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:17:45.473 INFO:teuthology.orchestra.run.smithi006.stdout:type=AVC msg=audit(1709709465.192:175): avc: denied { checkpoint_restore } for pid=1207 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1 2024-03-06T07:17:45.474 DEBUG:teuthology.task.selinux:ubuntu@smithi006.front.sepia.ceph.com has 1 denials 2024-03-06T07:17:45.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:17:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:17:45.553 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:17:45.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:17:45.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:17:45.733 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:17:45.743 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:17:45.761 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:17:45.762 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:23:47.167 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-03-06T07:23:47.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-06T07:23:47.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:23:47.317 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-06T07:23:47.476 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-06T07:23:47.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-06T07:23:47.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:23:47.621 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-03-06T07:23:47.770 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-03-06T07:23:47.770 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:23:47.782 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:23:47.782 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:23:47.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:23:47.790 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:23:47.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:23:47.890 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:23:47.892 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:23:47.943 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:23:47.950 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:23:48.034 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-03-06T07:23:48.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-06T07:23:48.076 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-03-06T07:23:48.099 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-06T07:23:48.126 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:23:48.157 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:23:48.179 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:23:48.299 INFO:teuthology.orchestra.run.smithi178.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.308 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.349 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:23:48.358 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:23:48.358 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.359 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:23:48.360 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:23:48.369 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:23:48.369 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:23:48.369 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:23:48.369 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:23:48.375 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:23:48.375 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:23:48.376 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:23:48.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:23:48.378 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:23:48.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:23:48.540 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:23:48.541 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:23:48.616 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:23:48.616 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:23:48.728 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:23:48.728 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:23:48.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:23:48.800 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:23:48.800 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:23:48.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:23:48.829 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:23:48.829 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:23:48.907 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:23:48.907 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:23:48.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:23:49.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:23:49.097 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:23:49.180 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-03-06T07:23:49.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-03-06T07:23:49.187 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:23:49.283 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-03-06T07:23:49.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-03-06T07:23:49.535 INFO:teuthology.orchestra.run.smithi178.stdout:74 files removed 2024-03-06T07:23:49.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:23:49.662 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-03-06T07:23:49.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:23:50.631 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 149 kB/s | 83 kB 00:00 2024-03-06T07:23:50.775 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 150 kB/s | 83 kB 00:00 2024-03-06T07:23:51.140 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-06T07:23:51.309 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-06T07:23:51.641 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-03-06T07:23:51.885 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-06T07:23:52.278 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:23:52.588 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-06T07:23:53.979 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-06T07:23:54.247 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-06T07:23:55.058 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-06T07:23:55.651 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-03-06T07:24:00.044 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-03-06T07:24:04.728 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-06T07:24:06.696 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-03-06T07:24:10.631 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 20 MB 00:03 2024-03-06T07:24:11.799 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-06T07:24:16.548 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-06T07:24:16.741 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.1 MB 00:01 2024-03-06T07:24:18.558 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-06T07:24:19.486 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:19.488 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:19.512 INFO:teuthology.orchestra.run.smithi006.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:24:19.513 INFO:teuthology.orchestra.run.smithi006.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:24:19.515 INFO:teuthology.orchestra.run.smithi006.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:24:19.516 INFO:teuthology.orchestra.run.smithi006.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:19.520 INFO:teuthology.orchestra.run.smithi006.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:19.523 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:24:19.525 INFO:teuthology.orchestra.run.smithi006.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:24:19.621 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:24:19.638 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:24:19.639 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:24:19.640 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:24:19.641 INFO:teuthology.orchestra.run.smithi006.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:24:19.642 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:24:19.643 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:19.644 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:24:19.645 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:24:19.646 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:24:19.647 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:24:19.648 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:24:19.649 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:24:19.650 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout:Install 150 Packages 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 26 Packages 2024-03-06T07:24:19.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:24:19.652 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 224 M 2024-03-06T07:24:19.652 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-06T07:24:20.374 INFO:teuthology.orchestra.run.smithi006.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-06T07:24:21.683 INFO:teuthology.orchestra.run.smithi006.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 680 kB/s | 888 kB 00:01 2024-03-06T07:24:21.949 INFO:teuthology.orchestra.run.smithi006.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 546 kB/s | 145 kB 00:00 2024-03-06T07:24:22.925 INFO:teuthology.orchestra.run.smithi006.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.8 MB/s | 5.3 MB 00:02 2024-03-06T07:24:23.392 INFO:teuthology.orchestra.run.smithi006.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 3.2 MB/s | 1.5 MB 00:00 2024-03-06T07:24:24.592 INFO:teuthology.orchestra.run.smithi006.stdout:(6/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-03-06T07:24:24.651 INFO:teuthology.orchestra.run.smithi006.stdout:(7/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 876 kB/s | 2.3 MB 00:02 2024-03-06T07:24:26.793 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 21 MB 00:08 2024-03-06T07:24:28.159 INFO:teuthology.orchestra.run.smithi006.stdout:(8/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x86 4.7 MB/s | 17 MB 00:03 2024-03-06T07:24:28.242 INFO:teuthology.orchestra.run.smithi006.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 306 kB/s | 25 kB 00:00 2024-03-06T07:24:32.467 INFO:teuthology.orchestra.run.smithi006.stdout:(10/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el 1.3 MB/s | 9.8 MB 00:07 2024-03-06T07:24:32.517 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-06T07:24:32.650 INFO:teuthology.orchestra.run.smithi006.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.2 MB/s | 220 kB 00:00 2024-03-06T07:24:32.751 INFO:teuthology.orchestra.run.smithi006.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 270 kB/s | 27 kB 00:00 2024-03-06T07:24:32.852 INFO:teuthology.orchestra.run.smithi006.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 316 kB/s | 31 kB 00:00 2024-03-06T07:24:33.484 INFO:teuthology.orchestra.run.smithi006.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.2 MB/s | 747 kB 00:00 2024-03-06T07:24:33.686 INFO:teuthology.orchestra.run.smithi006.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 817 kB/s | 163 kB 00:00 2024-03-06T07:24:33.886 INFO:teuthology.orchestra.run.smithi006.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 636 kB/s | 127 kB 00:00 2024-03-06T07:24:34.444 INFO:teuthology.orchestra.run.smithi006.stdout:(17/176): ceph-common-19.0.0-1739.g1bc250c7.el9 1.4 MB/s | 20 MB 00:14 2024-03-06T07:24:34.478 INFO:teuthology.orchestra.run.smithi006.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 848 kB/s | 501 kB 00:00 2024-03-06T07:24:34.587 INFO:teuthology.orchestra.run.smithi006.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 420 kB/s | 45 kB 00:00 2024-03-06T07:24:34.780 INFO:teuthology.orchestra.run.smithi006.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 687 kB/s | 131 kB 00:00 2024-03-06T07:24:34.971 INFO:teuthology.orchestra.run.smithi006.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 846 kB/s | 162 kB 00:00 2024-03-06T07:24:35.187 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:35.189 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:24:35.207 INFO:teuthology.orchestra.run.smithi178.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:24:35.208 INFO:teuthology.orchestra.run.smithi178.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:24:35.210 INFO:teuthology.orchestra.run.smithi178.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:24:35.211 INFO:teuthology.orchestra.run.smithi178.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:35.213 INFO:teuthology.orchestra.run.smithi178.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:24:35.216 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:24:35.218 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:24:35.263 INFO:teuthology.orchestra.run.smithi006.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 1.1 MB/s | 321 kB 00:00 2024-03-06T07:24:35.307 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:24:35.333 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:24:35.334 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:24:35.335 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:24:35.336 INFO:teuthology.orchestra.run.smithi178.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:24:35.337 INFO:teuthology.orchestra.run.smithi178.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:24:35.338 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:24:35.339 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:24:35.340 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:24:35.343 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi178.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout:Install 150 Packages 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 26 Packages 2024-03-06T07:24:35.346 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:24:35.347 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 224 M 2024-03-06T07:24:35.347 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-03-06T07:24:35.563 INFO:teuthology.orchestra.run.smithi006.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 995 kB/s | 298 kB 00:00 2024-03-06T07:24:35.671 INFO:teuthology.orchestra.run.smithi006.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 924 kB/s | 99 kB 00:00 2024-03-06T07:24:35.772 INFO:teuthology.orchestra.run.smithi006.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 859 kB/s | 85 kB 00:00 2024-03-06T07:24:36.027 INFO:teuthology.orchestra.run.smithi178.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-03-06T07:24:36.985 INFO:teuthology.orchestra.run.smithi178.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 928 kB/s | 888 kB 00:00 2024-03-06T07:24:37.077 INFO:teuthology.orchestra.run.smithi178.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 1.6 MB/s | 145 kB 00:00 2024-03-06T07:24:37.686 INFO:teuthology.orchestra.run.smithi178.stdout:(4/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 3.8 MB/s | 2.3 MB 00:00 2024-03-06T07:24:38.103 INFO:teuthology.orchestra.run.smithi178.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 3.6 MB/s | 1.5 MB 00:00 2024-03-06T07:24:38.803 INFO:teuthology.orchestra.run.smithi178.stdout:(6/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.7 MB/s | 5.3 MB 00:03 2024-03-06T07:24:38.838 INFO:teuthology.orchestra.run.smithi006.stdout:(26/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 1.0 MB/s | 3.1 MB 00:03 2024-03-06T07:24:39.064 INFO:teuthology.orchestra.run.smithi006.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 762 kB/s | 171 kB 00:00 2024-03-06T07:24:39.078 INFO:teuthology.orchestra.run.smithi178.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 4.8 MB/s | 4.7 MB 00:00 2024-03-06T07:24:39.172 INFO:teuthology.orchestra.run.smithi006.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 292 kB/s | 31 kB 00:00 2024-03-06T07:24:39.431 INFO:teuthology.orchestra.run.smithi006.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 536 kB/s | 138 kB 00:00 2024-03-06T07:24:40.315 INFO:teuthology.orchestra.run.smithi006.stdout:(30/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 893 kB/s | 5.1 MB 00:05 2024-03-06T07:24:40.853 INFO:teuthology.orchestra.run.smithi178.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 5.5 MB/s | 9.8 MB 00:01 2024-03-06T07:24:40.912 INFO:teuthology.orchestra.run.smithi178.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 441 kB/s | 25 kB 00:00 2024-03-06T07:24:41.140 INFO:teuthology.orchestra.run.smithi006.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 1.0 MB/s | 1.8 MB 00:01 2024-03-06T07:24:41.465 INFO:teuthology.orchestra.run.smithi006.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 766 kB/s | 248 kB 00:00 2024-03-06T07:24:41.607 INFO:teuthology.orchestra.run.smithi006.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 350 kB/s | 49 kB 00:00 2024-03-06T07:24:41.740 INFO:teuthology.orchestra.run.smithi006.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 115 kB/s | 15 kB 00:00 2024-03-06T07:24:42.049 INFO:teuthology.orchestra.run.smithi006.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 855 kB/s | 264 kB 00:00 2024-03-06T07:24:42.783 INFO:teuthology.orchestra.run.smithi006.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.0 MB/s | 748 kB 00:00 2024-03-06T07:24:42.891 INFO:teuthology.orchestra.run.smithi006.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 170 kB/s | 18 kB 00:00 2024-03-06T07:24:43.159 INFO:teuthology.orchestra.run.smithi006.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-06T07:24:43.284 INFO:teuthology.orchestra.run.smithi006.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 7.3 MB/s | 920 kB 00:00 2024-03-06T07:24:43.326 INFO:teuthology.orchestra.run.smithi006.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-06T07:24:43.359 INFO:teuthology.orchestra.run.smithi006.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 622 kB/s | 20 kB 00:00 2024-03-06T07:24:43.410 INFO:teuthology.orchestra.run.smithi006.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 307 kB 00:00 2024-03-06T07:24:43.452 INFO:teuthology.orchestra.run.smithi006.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 772 kB/s | 32 kB 00:00 2024-03-06T07:24:43.619 INFO:teuthology.orchestra.run.smithi006.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-06T07:24:43.661 INFO:teuthology.orchestra.run.smithi006.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 420 kB/s | 17 kB 00:00 2024-03-06T07:24:43.703 INFO:teuthology.orchestra.run.smithi006.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 244 kB/s | 10 kB 00:00 2024-03-06T07:24:43.753 INFO:teuthology.orchestra.run.smithi006.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-06T07:24:43.795 INFO:teuthology.orchestra.run.smithi006.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-06T07:24:43.854 INFO:teuthology.orchestra.run.smithi006.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-06T07:24:44.029 INFO:teuthology.orchestra.run.smithi006.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 3.2 MB/s | 578 kB 00:00 2024-03-06T07:24:44.079 INFO:teuthology.orchestra.run.smithi006.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 495 kB/s | 24 kB 00:00 2024-03-06T07:24:44.130 INFO:teuthology.orchestra.run.smithi006.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 936 kB/s | 46 kB 00:00 2024-03-06T07:24:44.180 INFO:teuthology.orchestra.run.smithi006.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.6 MB/s | 282 kB 00:00 2024-03-06T07:24:44.247 INFO:teuthology.orchestra.run.smithi006.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-03-06T07:24:44.289 INFO:teuthology.orchestra.run.smithi006.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 835 kB/s | 34 kB 00:00 2024-03-06T07:24:44.531 INFO:teuthology.orchestra.run.smithi006.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-06T07:24:44.570 INFO:teuthology.orchestra.run.smithi178.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 2.9 MB/s | 17 MB 00:05 2024-03-06T07:24:44.681 INFO:teuthology.orchestra.run.smithi006.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 20 MB/s | 3.0 MB 00:00 2024-03-06T07:24:44.728 INFO:teuthology.orchestra.run.smithi178.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 1.4 MB/s | 220 kB 00:00 2024-03-06T07:24:44.791 INFO:teuthology.orchestra.run.smithi006.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-03-06T07:24:44.821 INFO:teuthology.orchestra.run.smithi178.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 297 kB/s | 27 kB 00:00 2024-03-06T07:24:44.865 INFO:teuthology.orchestra.run.smithi006.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-06T07:24:44.908 INFO:teuthology.orchestra.run.smithi006.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-06T07:24:44.921 INFO:teuthology.orchestra.run.smithi178.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 315 kB/s | 31 kB 00:00 2024-03-06T07:24:45.166 INFO:teuthology.orchestra.run.smithi006.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-06T07:24:45.283 INFO:teuthology.orchestra.run.smithi006.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-03-06T07:24:45.341 INFO:teuthology.orchestra.run.smithi006.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-06T07:24:45.346 INFO:teuthology.orchestra.run.smithi178.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.7 MB/s | 747 kB 00:00 2024-03-06T07:24:45.470 INFO:teuthology.orchestra.run.smithi178.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.3 MB/s | 163 kB 00:00 2024-03-06T07:24:45.580 INFO:teuthology.orchestra.run.smithi178.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 1.1 MB/s | 127 kB 00:00 2024-03-06T07:24:45.839 INFO:teuthology.orchestra.run.smithi178.stdout:(17/176): libradosstriper1-19.0.0-1739.g1bc250c 1.9 MB/s | 501 kB 00:00 2024-03-06T07:24:45.901 INFO:teuthology.orchestra.run.smithi006.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-06T07:24:45.959 INFO:teuthology.orchestra.run.smithi006.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 3.9 MB/s | 227 kB 00:00 2024-03-06T07:24:45.993 INFO:teuthology.orchestra.run.smithi006.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 750 kB/s | 24 kB 00:00 2024-03-06T07:24:46.018 INFO:teuthology.orchestra.run.smithi006.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 739 kB/s | 18 kB 00:00 2024-03-06T07:24:46.060 INFO:teuthology.orchestra.run.smithi006.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-06T07:24:46.102 INFO:teuthology.orchestra.run.smithi006.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 771 kB/s | 32 kB 00:00 2024-03-06T07:24:46.327 INFO:teuthology.orchestra.run.smithi006.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-06T07:24:46.378 INFO:teuthology.orchestra.run.smithi006.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-06T07:24:46.428 INFO:teuthology.orchestra.run.smithi006.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-06T07:24:46.470 INFO:teuthology.orchestra.run.smithi006.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-06T07:24:46.512 INFO:teuthology.orchestra.run.smithi006.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 999 kB/s | 41 kB 00:00 2024-03-06T07:24:46.571 INFO:teuthology.orchestra.run.smithi006.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi006.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-06T07:24:46.671 INFO:teuthology.orchestra.run.smithi006.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.2 MB/s | 211 kB 00:00 2024-03-06T07:24:46.713 INFO:teuthology.orchestra.run.smithi006.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-06T07:24:46.755 INFO:teuthology.orchestra.run.smithi006.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-06T07:24:47.515 INFO:teuthology.orchestra.run.smithi006.stdout:(80/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-03-06T07:24:47.556 INFO:teuthology.orchestra.run.smithi006.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 950 kB/s | 39 kB 00:00 2024-03-06T07:24:47.581 INFO:teuthology.orchestra.run.smithi178.stdout:(18/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 2.9 MB/s | 5.1 MB 00:01 2024-03-06T07:24:47.590 INFO:teuthology.orchestra.run.smithi006.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-06T07:24:47.657 INFO:teuthology.orchestra.run.smithi006.stdout:(83/176): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 215 kB 00:00 2024-03-06T07:24:47.673 INFO:teuthology.orchestra.run.smithi178.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 496 kB/s | 45 kB 00:00 2024-03-06T07:24:47.707 INFO:teuthology.orchestra.run.smithi006.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 692 kB/s | 34 kB 00:00 2024-03-06T07:24:47.741 INFO:teuthology.orchestra.run.smithi006.stdout:(85/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-06T07:24:47.768 INFO:teuthology.orchestra.run.smithi178.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 1.4 MB/s | 131 kB 00:00 2024-03-06T07:24:47.791 INFO:teuthology.orchestra.run.smithi006.stdout:(86/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-06T07:24:47.833 INFO:teuthology.orchestra.run.smithi006.stdout:(87/176): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-06T07:24:47.857 INFO:teuthology.orchestra.run.smithi178.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 1.8 MB/s | 162 kB 00:00 2024-03-06T07:24:47.892 INFO:teuthology.orchestra.run.smithi006.stdout:(88/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2024-03-06T07:24:47.942 INFO:teuthology.orchestra.run.smithi006.stdout:(89/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-06T07:24:48.001 INFO:teuthology.orchestra.run.smithi006.stdout:(90/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 760 kB/s | 44 kB 00:00 2024-03-06T07:24:48.007 INFO:teuthology.orchestra.run.smithi178.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 2.1 MB/s | 321 kB 00:00 2024-03-06T07:24:48.051 INFO:teuthology.orchestra.run.smithi006.stdout:(91/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-06T07:24:48.125 INFO:teuthology.orchestra.run.smithi006.stdout:(92/176): jq-1.6-16.el9.x86_64.rpm 2.5 MB/s | 186 kB 00:00 2024-03-06T07:24:48.141 INFO:teuthology.orchestra.run.smithi178.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 2.2 MB/s | 298 kB 00:00 2024-03-06T07:24:48.210 INFO:teuthology.orchestra.run.smithi006.stdout:(93/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.3 MB/s | 802 kB 00:00 2024-03-06T07:24:48.225 INFO:teuthology.orchestra.run.smithi178.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 1.2 MB/s | 99 kB 00:00 2024-03-06T07:24:48.317 INFO:teuthology.orchestra.run.smithi178.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 937 kB/s | 85 kB 00:00 2024-03-06T07:24:48.360 INFO:teuthology.orchestra.run.smithi006.stdout:(94/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 194 kB 00:00 2024-03-06T07:24:48.427 INFO:teuthology.orchestra.run.smithi006.stdout:(95/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-06T07:24:48.477 INFO:teuthology.orchestra.run.smithi006.stdout:(96/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-03-06T07:24:48.637 INFO:teuthology.orchestra.run.smithi006.stdout:(97/176): python3-cryptography-36.0.1-4.el9.x86 7.2 MB/s | 1.1 MB 00:00 2024-03-06T07:24:48.687 INFO:teuthology.orchestra.run.smithi006.stdout:(98/176): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-06T07:24:48.728 INFO:teuthology.orchestra.run.smithi006.stdout:(99/176): python3-pycparser-2.20-6.el9.noarch.r 3.0 MB/s | 124 kB 00:00 2024-03-06T07:24:48.779 INFO:teuthology.orchestra.run.smithi006.stdout:(100/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-06T07:24:48.829 INFO:teuthology.orchestra.run.smithi006.stdout:(101/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-06T07:24:48.988 INFO:teuthology.orchestra.run.smithi006.stdout:(102/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.5 MB/s | 245 kB 00:00 2024-03-06T07:24:49.038 INFO:teuthology.orchestra.run.smithi006.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 454 kB/s | 22 kB 00:00 2024-03-06T07:24:49.155 INFO:teuthology.orchestra.run.smithi006.stdout:(104/176): protobuf-compiler-3.14.0-13.el9.x86_ 7.3 MB/s | 863 kB 00:00 2024-03-06T07:24:49.333 INFO:teuthology.orchestra.run.smithi178.stdout:(26/176): ceph-test-19.0.0-1739.g1bc250c7.el9.x 5.6 MB/s | 47 MB 00:08 2024-03-06T07:24:49.434 INFO:teuthology.orchestra.run.smithi178.stdout:(27/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 2.8 MB/s | 3.1 MB 00:01 2024-03-06T07:24:49.467 INFO:teuthology.orchestra.run.smithi178.stdout:(28/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 1.3 MB/s | 171 kB 00:00 2024-03-06T07:24:49.509 INFO:teuthology.orchestra.run.smithi178.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1739.g 422 kB/s | 31 kB 00:00 2024-03-06T07:24:49.560 INFO:teuthology.orchestra.run.smithi178.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 1.5 MB/s | 138 kB 00:00 2024-03-06T07:24:50.268 INFO:teuthology.orchestra.run.smithi178.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 2.4 MB/s | 1.8 MB 00:00 2024-03-06T07:24:50.439 INFO:teuthology.orchestra.run.smithi006.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 427 kB/s | 548 kB 00:01 2024-03-06T07:24:50.452 INFO:teuthology.orchestra.run.smithi178.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 1.3 MB/s | 248 kB 00:00 2024-03-06T07:24:50.544 INFO:teuthology.orchestra.run.smithi178.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 539 kB/s | 49 kB 00:00 2024-03-06T07:24:50.636 INFO:teuthology.orchestra.run.smithi178.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 169 kB/s | 15 kB 00:00 2024-03-06T07:24:50.803 INFO:teuthology.orchestra.run.smithi178.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 1.6 MB/s | 264 kB 00:00 2024-03-06T07:24:50.831 INFO:teuthology.orchestra.run.smithi006.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 790 kB/s | 309 kB 00:00 2024-03-06T07:24:50.898 INFO:teuthology.orchestra.run.smithi006.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 296 kB/s | 19 kB 00:00 2024-03-06T07:24:51.270 INFO:teuthology.orchestra.run.smithi178.stdout:(36/176): ceph-mgr-diskprediction-local-19.0.0- 4.3 MB/s | 7.4 MB 00:01 2024-03-06T07:24:51.312 INFO:teuthology.orchestra.run.smithi178.stdout:(37/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.4 MB/s | 748 kB 00:00 2024-03-06T07:24:51.354 INFO:teuthology.orchestra.run.smithi178.stdout:(38/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 220 kB/s | 18 kB 00:00 2024-03-06T07:24:51.532 INFO:teuthology.orchestra.run.smithi006.stdout:(108/176): ceph-mgr-diskprediction-local-19.0.0 676 kB/s | 7.4 MB 00:11 2024-03-06T07:24:51.621 INFO:teuthology.orchestra.run.smithi178.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 920 kB 00:00 2024-03-06T07:24:51.705 INFO:teuthology.orchestra.run.smithi178.stdout:(40/176): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-03-06T07:24:51.738 INFO:teuthology.orchestra.run.smithi178.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 920 kB/s | 106 kB 00:00 2024-03-06T07:24:51.764 INFO:teuthology.orchestra.run.smithi178.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 358 kB/s | 20 kB 00:00 2024-03-06T07:24:51.798 INFO:teuthology.orchestra.run.smithi006.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 98 kB/s | 26 kB 00:00 2024-03-06T07:24:51.847 INFO:teuthology.orchestra.run.smithi178.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 307 kB 00:00 2024-03-06T07:24:51.873 INFO:teuthology.orchestra.run.smithi178.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 295 kB/s | 32 kB 00:00 2024-03-06T07:24:51.907 INFO:teuthology.orchestra.run.smithi006.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 453 kB/s | 49 kB 00:00 2024-03-06T07:24:51.965 INFO:teuthology.orchestra.run.smithi006.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-06T07:24:52.057 INFO:teuthology.orchestra.run.smithi178.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 94 kB/s | 17 kB 00:00 2024-03-06T07:24:52.099 INFO:teuthology.orchestra.run.smithi006.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-03-06T07:24:52.223 INFO:teuthology.orchestra.run.smithi178.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.0 MB/s | 3.0 MB 00:00 2024-03-06T07:24:52.248 INFO:teuthology.orchestra.run.smithi178.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 53 kB/s | 10 kB 00:00 2024-03-06T07:24:52.282 INFO:teuthology.orchestra.run.smithi178.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-06T07:24:52.324 INFO:teuthology.orchestra.run.smithi178.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 595 kB/s | 44 kB 00:00 2024-03-06T07:24:52.458 INFO:teuthology.orchestra.run.smithi178.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-06T07:24:52.500 INFO:teuthology.orchestra.run.smithi178.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 3.2 MB/s | 578 kB 00:00 2024-03-06T07:24:52.525 INFO:teuthology.orchestra.run.smithi178.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 390 kB/s | 24 kB 00:00 2024-03-06T07:24:52.542 INFO:teuthology.orchestra.run.smithi178.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-06T07:24:52.592 INFO:teuthology.orchestra.run.smithi178.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-06T07:24:52.643 INFO:teuthology.orchestra.run.smithi178.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-03-06T07:24:52.660 INFO:teuthology.orchestra.run.smithi178.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 34 kB 00:00 2024-03-06T07:24:52.733 INFO:teuthology.orchestra.run.smithi006.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-03-06T07:24:53.142 INFO:teuthology.orchestra.run.smithi006.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 1.3 MB/s | 548 kB 00:00 2024-03-06T07:24:53.185 INFO:teuthology.orchestra.run.smithi178.stdout:(57/176): ceph-common-19.0.0-1739.g1bc250c7.el9 1.2 MB/s | 20 MB 00:17 2024-03-06T07:24:53.234 INFO:teuthology.orchestra.run.smithi006.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 479 kB/s | 43 kB 00:00 2024-03-06T07:24:53.317 INFO:teuthology.orchestra.run.smithi006.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 389 kB/s | 32 kB 00:00 2024-03-06T07:24:53.369 INFO:teuthology.orchestra.run.smithi178.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 6.5 MB/s | 4.7 MB 00:00 2024-03-06T07:24:53.376 INFO:teuthology.orchestra.run.smithi006.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 261 kB/s | 15 kB 00:00 2024-03-06T07:24:53.461 INFO:teuthology.orchestra.run.smithi178.stdout:(59/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-03-06T07:24:53.520 INFO:teuthology.orchestra.run.smithi178.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 4.6 MB/s | 1.5 MB 00:00 2024-03-06T07:24:53.561 INFO:teuthology.orchestra.run.smithi178.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 4.5 MB/s | 811 kB 00:00 2024-03-06T07:24:53.568 INFO:teuthology.orchestra.run.smithi006.stdout:(118/176): python3-cheroot-8.6.0-4.el9.noarch.r 900 kB/s | 172 kB 00:00 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi178.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-06T07:24:53.754 INFO:teuthology.orchestra.run.smithi178.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-06T07:24:53.776 INFO:teuthology.orchestra.run.smithi006.stdout:(119/176): python3-cherrypy-18.6.1-2.el9.noarch 1.7 MB/s | 358 kB 00:00 2024-03-06T07:24:53.904 INFO:teuthology.orchestra.run.smithi178.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 248 kB/s | 78 kB 00:00 2024-03-06T07:24:54.021 INFO:teuthology.orchestra.run.smithi178.stdout:(65/176): postgresql-server-13.11-1.el9.x86_64. 12 MB/s | 5.7 MB 00:00 2024-03-06T07:24:54.197 INFO:teuthology.orchestra.run.smithi178.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-06T07:24:54.198 INFO:teuthology.orchestra.run.smithi006.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 721 kB/s | 303 kB 00:00 2024-03-06T07:24:54.230 INFO:teuthology.orchestra.run.smithi178.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 708 kB/s | 227 kB 00:00 2024-03-06T07:24:54.255 INFO:teuthology.orchestra.run.smithi178.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 111 kB/s | 24 kB 00:00 2024-03-06T07:24:54.277 INFO:teuthology.orchestra.run.smithi006.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 453 kB/s | 35 kB 00:00 2024-03-06T07:24:54.280 INFO:teuthology.orchestra.run.smithi178.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 224 kB/s | 18 kB 00:00 2024-03-06T07:24:54.306 INFO:teuthology.orchestra.run.smithi178.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-06T07:24:54.331 INFO:teuthology.orchestra.run.smithi178.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 426 kB/s | 32 kB 00:00 2024-03-06T07:24:54.469 INFO:teuthology.orchestra.run.smithi006.stdout:(122/176): libarrow-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 4.4 MB 00:03 2024-03-06T07:24:54.494 INFO:teuthology.orchestra.run.smithi006.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 938 kB/s | 203 kB 00:00 2024-03-06T07:24:54.548 INFO:teuthology.orchestra.run.smithi178.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-06T07:24:54.581 INFO:teuthology.orchestra.run.smithi178.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 909 kB/s | 217 kB 00:00 2024-03-06T07:24:54.595 INFO:teuthology.orchestra.run.smithi006.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-06T07:24:54.607 INFO:teuthology.orchestra.run.smithi178.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 661 kB/s | 174 kB 00:00 2024-03-06T07:24:54.623 INFO:teuthology.orchestra.run.smithi178.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 951 kB/s | 69 kB 00:00 2024-03-06T07:24:54.668 INFO:teuthology.orchestra.run.smithi178.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 697 kB/s | 41 kB 00:00 2024-03-06T07:24:54.670 INFO:teuthology.orchestra.run.smithi006.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 144 kB/s | 11 kB 00:00 2024-03-06T07:24:54.699 INFO:teuthology.orchestra.run.smithi178.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 2.6 MB/s | 238 kB 00:00 2024-03-06T07:24:54.716 INFO:teuthology.orchestra.run.smithi178.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-06T07:24:54.754 INFO:teuthology.orchestra.run.smithi006.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 214 kB/s | 18 kB 00:00 2024-03-06T07:24:54.761 INFO:teuthology.orchestra.run.smithi178.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 211 kB 00:00 2024-03-06T07:24:54.783 INFO:teuthology.orchestra.run.smithi178.stdout:(80/176): python3-pytz-2021.1-5.el9.noarch.rpm 737 kB/s | 47 kB 00:00 2024-03-06T07:24:54.808 INFO:teuthology.orchestra.run.smithi178.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 479 kB/s | 43 kB 00:00 2024-03-06T07:24:54.820 INFO:teuthology.orchestra.run.smithi006.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 351 kB/s | 23 kB 00:00 2024-03-06T07:24:54.899 INFO:teuthology.orchestra.run.smithi006.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 261 kB/s | 19 kB 00:00 2024-03-06T07:24:54.979 INFO:teuthology.orchestra.run.smithi006.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 239 kB/s | 20 kB 00:00 2024-03-06T07:24:55.076 INFO:teuthology.orchestra.run.smithi178.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 151 kB/s | 39 kB 00:00 2024-03-06T07:24:55.103 INFO:teuthology.orchestra.run.smithi178.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 145 kB/s | 41 kB 00:00 2024-03-06T07:24:55.543 INFO:teuthology.orchestra.run.smithi178.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-06T07:24:55.568 INFO:teuthology.orchestra.run.smithi178.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 438 kB/s | 215 kB 00:00 2024-03-06T07:24:55.593 INFO:teuthology.orchestra.run.smithi178.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 69 kB/s | 34 kB 00:00 2024-03-06T07:24:55.652 INFO:teuthology.orchestra.run.smithi178.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 762 kB/s | 43 kB 00:00 2024-03-06T07:24:55.702 INFO:teuthology.orchestra.run.smithi178.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 527 kB/s | 56 kB 00:00 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi178.stdout:(89/176): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-06T07:24:55.770 INFO:teuthology.orchestra.run.smithi178.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 747 kB/s | 87 kB 00:00 2024-03-06T07:24:55.795 INFO:teuthology.orchestra.run.smithi178.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 706 kB/s | 63 kB 00:00 2024-03-06T07:24:55.828 INFO:teuthology.orchestra.run.smithi178.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 772 kB/s | 44 kB 00:00 2024-03-06T07:24:55.854 INFO:teuthology.orchestra.run.smithi178.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 956 kB/s | 79 kB 00:00 2024-03-06T07:24:55.879 INFO:teuthology.orchestra.run.smithi178.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 186 kB 00:00 2024-03-06T07:24:56.046 INFO:teuthology.orchestra.run.smithi178.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 802 kB 00:00 2024-03-06T07:24:56.071 INFO:teuthology.orchestra.run.smithi178.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 901 kB/s | 194 kB 00:00 2024-03-06T07:24:56.096 INFO:teuthology.orchestra.run.smithi178.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-06T07:24:56.147 INFO:teuthology.orchestra.run.smithi178.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-03-06T07:24:56.180 INFO:teuthology.orchestra.run.smithi178.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-06T07:24:56.196 INFO:teuthology.orchestra.run.smithi006.stdout:(130/176): python3-kubernetes-26.1.0-2.el9.noar 860 kB/s | 1.0 MB 00:01 2024-03-06T07:24:56.230 INFO:teuthology.orchestra.run.smithi178.stdout:(100/176): python3-cryptography-36.0.1-4.el9.x8 8.6 MB/s | 1.1 MB 00:00 2024-03-06T07:24:56.272 INFO:teuthology.orchestra.run.smithi178.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.0 MB/s | 124 kB 00:00 2024-03-06T07:24:56.280 INFO:teuthology.orchestra.run.smithi006.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 563 kB/s | 46 kB 00:00 2024-03-06T07:24:56.298 INFO:teuthology.orchestra.run.smithi178.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.0 MB/s | 113 kB 00:00 2024-03-06T07:24:56.440 INFO:teuthology.orchestra.run.smithi178.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 1.1 MB/s | 187 kB 00:00 2024-03-06T07:24:56.455 INFO:teuthology.orchestra.run.smithi006.stdout:(132/176): python3-grpcio-1.46.7-10.el9.x86_64. 1.0 MB/s | 2.0 MB 00:01 2024-03-06T07:24:56.490 INFO:teuthology.orchestra.run.smithi178.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 117 kB/s | 22 kB 00:00 2024-03-06T07:24:56.495 INFO:teuthology.orchestra.run.smithi006.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 367 kB/s | 79 kB 00:00 2024-03-06T07:24:56.563 INFO:teuthology.orchestra.run.smithi006.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 542 kB/s | 58 kB 00:00 2024-03-06T07:24:56.646 INFO:teuthology.orchestra.run.smithi006.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 199 kB/s | 16 kB 00:00 2024-03-06T07:24:56.797 INFO:teuthology.orchestra.run.smithi006.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 600 kB/s | 90 kB 00:00 2024-03-06T07:24:56.880 INFO:teuthology.orchestra.run.smithi006.stdout:(137/176): python3-pyperclip-1.8.2-2.el9.noarch 342 kB/s | 28 kB 00:00 2024-03-06T07:24:56.939 INFO:teuthology.orchestra.run.smithi006.stdout:(138/176): python3-pecan-1.4.2-3.el9.noarch.rpm 616 kB/s | 272 kB 00:00 2024-03-06T07:24:56.964 INFO:teuthology.orchestra.run.smithi006.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 372 kB/s | 31 kB 00:00 2024-03-06T07:24:57.073 INFO:teuthology.orchestra.run.smithi006.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 552 kB/s | 59 kB 00:00 2024-03-06T07:24:57.074 INFO:teuthology.orchestra.run.smithi178.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 307 kB/s | 245 kB 00:00 2024-03-06T07:24:57.140 INFO:teuthology.orchestra.run.smithi006.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 544 kB/s | 36 kB 00:00 2024-03-06T07:24:57.165 INFO:teuthology.orchestra.run.smithi006.stdout:(142/176): python3-routes-2.5.1-5.el9.noarch.rp 945 kB/s | 188 kB 00:00 2024-03-06T07:24:57.216 INFO:teuthology.orchestra.run.smithi178.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-03-06T07:24:57.265 INFO:teuthology.orchestra.run.smithi006.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 410 kB/s | 51 kB 00:00 2024-03-06T07:24:57.266 INFO:teuthology.orchestra.run.smithi178.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 707 kB/s | 548 kB 00:00 2024-03-06T07:24:57.382 INFO:teuthology.orchestra.run.smithi006.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-03-06T07:24:57.407 INFO:teuthology.orchestra.run.smithi006.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 635 kB/s | 90 kB 00:00 2024-03-06T07:24:57.466 INFO:teuthology.orchestra.run.smithi006.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 348 kB/s | 20 kB 00:00 2024-03-06T07:24:57.475 INFO:teuthology.orchestra.run.smithi178.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 774 kB/s | 309 kB 00:00 2024-03-06T07:24:57.491 INFO:teuthology.orchestra.run.smithi178.stdout:(109/176): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-06T07:24:57.525 INFO:teuthology.orchestra.run.smithi178.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 526 kB/s | 26 kB 00:00 2024-03-06T07:24:57.575 INFO:teuthology.orchestra.run.smithi178.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 585 kB/s | 49 kB 00:00 2024-03-06T07:24:57.601 INFO:teuthology.orchestra.run.smithi178.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 905 kB/s | 67 kB 00:00 2024-03-06T07:24:57.675 INFO:teuthology.orchestra.run.smithi006.stdout:(147/176): re2-20211101-3.el9.x86_64.rpm 924 kB/s | 192 kB 00:00 2024-03-06T07:24:57.676 INFO:teuthology.orchestra.run.smithi178.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-06T07:24:57.717 INFO:teuthology.orchestra.run.smithi006.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 1.3 MB/s | 427 kB 00:00 2024-03-06T07:24:57.768 INFO:teuthology.orchestra.run.smithi178.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-06T07:24:57.810 INFO:teuthology.orchestra.run.smithi178.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-03-06T07:24:57.827 INFO:teuthology.orchestra.run.smithi178.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 751 kB/s | 43 kB 00:00 2024-03-06T07:24:57.886 INFO:teuthology.orchestra.run.smithi178.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 431 kB/s | 32 kB 00:00 2024-03-06T07:24:57.920 INFO:teuthology.orchestra.run.smithi178.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 164 kB/s | 15 kB 00:00 2024-03-06T07:24:57.978 INFO:teuthology.orchestra.run.smithi178.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-06T07:24:58.029 INFO:teuthology.orchestra.run.smithi178.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-03-06T07:24:58.062 INFO:teuthology.orchestra.run.smithi178.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-06T07:24:58.096 INFO:teuthology.orchestra.run.smithi178.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 531 kB/s | 35 kB 00:00 2024-03-06T07:24:58.129 INFO:teuthology.orchestra.run.smithi178.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-06T07:24:58.195 INFO:teuthology.orchestra.run.smithi178.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 2.2 MB/s | 144 kB 00:00 2024-03-06T07:24:58.254 INFO:teuthology.orchestra.run.smithi178.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 186 kB/s | 11 kB 00:00 2024-03-06T07:24:58.321 INFO:teuthology.orchestra.run.smithi178.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 268 kB/s | 18 kB 00:00 2024-03-06T07:24:58.380 INFO:teuthology.orchestra.run.smithi178.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 401 kB/s | 23 kB 00:00 2024-03-06T07:24:58.439 INFO:teuthology.orchestra.run.smithi178.stdout:(128/176): python3-grpcio-1.46.7-10.el9.x86_64. 6.0 MB/s | 2.0 MB 00:00 2024-03-06T07:24:58.464 INFO:teuthology.orchestra.run.smithi178.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 234 kB/s | 19 kB 00:00 2024-03-06T07:24:58.506 INFO:teuthology.orchestra.run.smithi178.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 298 kB/s | 20 kB 00:00 2024-03-06T07:24:58.573 INFO:teuthology.orchestra.run.smithi178.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 702 kB/s | 46 kB 00:00 2024-03-06T07:24:58.600 INFO:teuthology.orchestra.run.smithi006.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 1.7 MB/s | 1.6 MB 00:00 2024-03-06T07:24:58.623 INFO:teuthology.orchestra.run.smithi178.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-06T07:24:58.649 INFO:teuthology.orchestra.run.smithi178.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-06T07:24:58.764 INFO:teuthology.orchestra.run.smithi178.stdout:(134/176): libarrow-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-03-06T07:24:58.799 INFO:teuthology.orchestra.run.smithi178.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 329 kB/s | 58 kB 00:00 2024-03-06T07:24:58.825 INFO:teuthology.orchestra.run.smithi178.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 277 kB/s | 16 kB 00:00 2024-03-06T07:24:58.858 INFO:teuthology.orchestra.run.smithi178.stdout:(137/176): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-06T07:24:58.884 INFO:teuthology.orchestra.run.smithi178.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 486 kB/s | 28 kB 00:00 2024-03-06T07:24:58.901 INFO:teuthology.orchestra.run.smithi178.stdout:(139/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 898 kB/s | 90 kB 00:00 2024-03-06T07:24:58.926 INFO:teuthology.orchestra.run.smithi178.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 470 kB/s | 31 kB 00:00 2024-03-06T07:24:58.960 INFO:teuthology.orchestra.run.smithi178.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-03-06T07:24:58.985 INFO:teuthology.orchestra.run.smithi178.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2024-03-06T07:24:59.010 INFO:teuthology.orchestra.run.smithi178.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 429 kB/s | 36 kB 00:00 2024-03-06T07:24:59.036 INFO:teuthology.orchestra.run.smithi178.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 681 kB/s | 51 kB 00:00 2024-03-06T07:24:59.069 INFO:teuthology.orchestra.run.smithi178.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-06T07:24:59.094 INFO:teuthology.orchestra.run.smithi178.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-03-06T07:24:59.128 INFO:teuthology.orchestra.run.smithi178.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 344 kB/s | 20 kB 00:00 2024-03-06T07:24:59.162 INFO:teuthology.orchestra.run.smithi178.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-06T07:24:59.187 INFO:teuthology.orchestra.run.smithi178.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-06T07:24:59.346 INFO:teuthology.orchestra.run.smithi178.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-06T07:24:59.388 INFO:teuthology.orchestra.run.smithi178.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-06T07:24:59.596 INFO:teuthology.orchestra.run.smithi178.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-06T07:24:59.705 INFO:teuthology.orchestra.run.smithi178.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-06T07:24:59.730 INFO:teuthology.orchestra.run.smithi178.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.7 MB/s | 41 kB 00:00 2024-03-06T07:24:59.818 INFO:teuthology.orchestra.run.smithi178.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 2.8 MB/s | 247 kB 00:00 2024-03-06T07:24:59.856 INFO:teuthology.orchestra.run.smithi178.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.8 MB/s | 69 kB 00:00 2024-03-06T07:24:59.890 INFO:teuthology.orchestra.run.smithi178.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-03-06T07:24:59.932 INFO:teuthology.orchestra.run.smithi178.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-03-06T07:24:59.974 INFO:teuthology.orchestra.run.smithi178.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-03-06T07:25:00.033 INFO:teuthology.orchestra.run.smithi178.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 14 MB/s | 802 kB 00:00 2024-03-06T07:25:00.083 INFO:teuthology.orchestra.run.smithi178.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.6 MB/s | 433 kB 00:00 2024-03-06T07:25:00.125 INFO:teuthology.orchestra.run.smithi178.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 536 kB/s | 22 kB 00:00 2024-03-06T07:25:00.167 INFO:teuthology.orchestra.run.smithi178.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 107 kB 00:00 2024-03-06T07:25:00.201 INFO:teuthology.orchestra.run.smithi178.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 4.6 MB/s | 154 kB 00:00 2024-03-06T07:25:00.242 INFO:teuthology.orchestra.run.smithi178.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:25:00.268 INFO:teuthology.orchestra.run.smithi178.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 2.5 MB/s | 63 kB 00:00 2024-03-06T07:25:00.293 INFO:teuthology.orchestra.run.smithi178.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-03-06T07:25:00.368 INFO:teuthology.orchestra.run.smithi178.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 15 MB/s | 1.2 MB 00:00 2024-03-06T07:25:00.477 INFO:teuthology.orchestra.run.smithi178.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-06T07:25:00.767 INFO:teuthology.orchestra.run.smithi006.stdout:(150/176): librados2-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 3.4 MB 00:03 2024-03-06T07:25:00.786 INFO:teuthology.orchestra.run.smithi178.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 7.1 MB/s | 2.2 MB 00:00 2024-03-06T07:25:00.809 INFO:teuthology.orchestra.run.smithi006.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-06T07:25:00.878 INFO:teuthology.orchestra.run.smithi178.stdout:(171/176): librados2-19.0.0-1739.g1bc250c7.el9. 2.0 MB/s | 3.4 MB 00:01 2024-03-06T07:25:00.911 INFO:teuthology.orchestra.run.smithi178.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 438 kB 00:00 2024-03-06T07:25:00.934 INFO:teuthology.orchestra.run.smithi178.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 584 kB/s | 30 kB 00:00 2024-03-06T07:25:00.951 INFO:teuthology.orchestra.run.smithi006.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-06T07:25:00.967 INFO:teuthology.orchestra.run.smithi178.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.7 MB/s | 321 kB 00:00 2024-03-06T07:25:01.018 INFO:teuthology.orchestra.run.smithi178.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-06T07:25:01.277 INFO:teuthology.orchestra.run.smithi006.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 5.6 MB/s | 1.8 MB 00:00 2024-03-06T07:25:01.310 INFO:teuthology.orchestra.run.smithi006.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.3 MB/s | 41 kB 00:00 2024-03-06T07:25:01.352 INFO:teuthology.orchestra.run.smithi006.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.9 MB/s | 247 kB 00:00 2024-03-06T07:25:01.359 INFO:teuthology.orchestra.run.smithi178.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.4 MB/s | 3.1 MB 00:02 2024-03-06T07:25:01.374 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:25:01.375 INFO:teuthology.orchestra.run.smithi178.stdout:Total 8.6 MB/s | 224 MB 00:26 2024-03-06T07:25:01.394 INFO:teuthology.orchestra.run.smithi006.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2024-03-06T07:25:01.428 INFO:teuthology.orchestra.run.smithi006.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-03-06T07:25:01.495 INFO:teuthology.orchestra.run.smithi006.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 191 kB 00:00 2024-03-06T07:25:01.508 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-03-06T07:25:01.537 INFO:teuthology.orchestra.run.smithi006.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-03-06T07:25:01.612 INFO:teuthology.orchestra.run.smithi006.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-06T07:25:01.662 INFO:teuthology.orchestra.run.smithi006.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.7 MB/s | 433 kB 00:00 2024-03-06T07:25:01.696 INFO:teuthology.orchestra.run.smithi006.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 665 kB/s | 22 kB 00:00 2024-03-06T07:25:01.738 INFO:teuthology.orchestra.run.smithi006.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 107 kB 00:00 2024-03-06T07:25:01.780 INFO:teuthology.orchestra.run.smithi006.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.7 MB/s | 154 kB 00:00 2024-03-06T07:25:01.822 INFO:teuthology.orchestra.run.smithi006.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-03-06T07:25:01.856 INFO:teuthology.orchestra.run.smithi006.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.9 MB/s | 63 kB 00:00 2024-03-06T07:25:01.881 INFO:teuthology.orchestra.run.smithi006.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 28 kB 00:00 2024-03-06T07:25:01.883 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-03-06T07:25:01.883 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-03-06T07:25:01.973 INFO:teuthology.orchestra.run.smithi006.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-06T07:25:02.082 INFO:teuthology.orchestra.run.smithi006.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-06T07:25:02.190 INFO:teuthology.orchestra.run.smithi006.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-06T07:25:02.243 INFO:teuthology.orchestra.run.smithi006.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 8.7 MB/s | 438 kB 00:00 2024-03-06T07:25:02.274 INFO:teuthology.orchestra.run.smithi006.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 920 kB/s | 30 kB 00:00 2024-03-06T07:25:02.316 INFO:teuthology.orchestra.run.smithi006.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-03-06T07:25:02.392 INFO:teuthology.orchestra.run.smithi006.stdout:(174/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-06T07:25:02.867 INFO:teuthology.orchestra.run.smithi006.stdout:(175/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.4 MB/s | 47 MB 00:34 2024-03-06T07:25:04.203 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-03-06T07:25:04.203 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-03-06T07:25:04.400 INFO:teuthology.orchestra.run.smithi006.stdout:(176/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 553 kB/s | 3.1 MB 00:05 2024-03-06T07:25:04.414 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:25:04.414 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.0 MB/s | 224 MB 00:44 2024-03-06T07:25:04.558 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-06T07:25:04.960 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-06T07:25:04.960 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-06T07:25:07.362 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-06T07:25:07.363 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-06T07:25:11.631 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-03-06T07:25:11.732 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:11.790 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:11.828 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:11.916 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:11.980 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:12.055 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:12.169 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:12.234 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:12.272 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:12.372 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:12.805 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:12.853 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:13.014 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:13.046 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:13.192 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:13.235 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:13.491 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:13.541 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:13.567 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:13.711 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:13.743 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:13.832 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:13.887 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:13.911 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:13.981 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:14.307 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:14.353 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:14.403 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:14.735 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:14.827 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-06T07:25:14.885 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:14.973 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:15.071 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:25:15.117 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:25:15.188 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:15.193 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:15.246 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:15.273 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:25:15.341 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:25:15.437 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:25:15.543 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:25:15.579 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:15.610 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:25:15.647 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:25:15.754 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:25:15.759 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:25:15.885 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:25:15.927 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:25:16.026 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:25:16.089 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:25:16.122 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:25:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:25:16.200 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:25:16.227 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:16.252 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:25:16.356 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:25:16.399 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:25:16.415 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:25:16.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:16.462 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:25:16.505 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:25:16.544 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:25:16.571 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:25:16.604 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:25:16.659 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:25:16.698 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:25:16.751 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:25:16.788 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:25:16.867 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:25:16.904 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:25:16.942 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:25:16.964 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:25:16.966 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:17.064 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:25:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:25:17.143 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:17.235 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:25:17.241 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:25:17.294 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:25:17.301 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:25:17.327 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:17.394 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:25:17.727 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:17.761 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:25:17.803 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:25:17.995 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:25:18.090 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:25:18.145 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:25:18.191 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:25:18.267 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:18.291 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:18.310 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:18.352 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:25:18.398 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:25:18.406 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:25:18.460 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:25:18.602 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:18.620 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:25:18.655 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:18.669 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:25:19.001 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:25:19.003 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:25:19.181 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:25:19.200 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:25:19.311 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:25:19.320 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:25:19.352 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:25:19.379 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:25:19.450 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:25:19.530 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:25:19.565 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:25:19.581 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:25:19.610 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:25:19.654 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:25:19.657 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:25:19.731 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:25:19.763 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:25:19.808 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:25:19.823 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:25:19.861 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:25:19.875 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:25:19.900 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:25:19.966 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:25:20.005 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:25:20.017 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:25:20.080 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:25:20.147 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:25:20.293 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:25:20.337 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:25:20.389 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:25:20.427 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:25:20.439 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:25:20.475 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:25:20.487 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:25:20.558 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:25:20.593 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:25:20.631 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:25:20.651 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:25:20.695 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:25:20.724 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:25:20.725 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:25:20.757 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:25:20.778 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:25:20.871 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:25:20.896 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:25:21.053 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:25:21.063 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:25:21.122 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:25:21.210 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:25:21.268 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:25:21.330 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:25:21.430 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:25:21.678 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:25:21.731 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:25:21.862 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:25:21.904 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:25:21.953 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:25:21.957 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:25:21.994 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:25:22.034 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:25:22.073 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:25:22.110 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:25:22.121 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:25:22.152 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:25:22.178 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:22.198 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:25:22.207 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:25:22.265 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:25:22.271 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:25:22.328 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:25:22.332 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:25:22.400 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:25:22.413 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:25:22.536 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:25:22.573 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:25:22.609 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:25:22.616 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:25:22.671 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:25:22.691 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:25:22.720 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:25:22.866 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:25:22.981 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:25:23.050 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:25:23.061 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:25:23.086 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:25:23.143 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:25:23.277 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:25:23.281 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:25:23.344 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:25:23.359 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:25:23.398 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:25:23.596 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:25:23.673 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:25:23.755 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:25:23.880 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:25:23.922 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:25:23.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:25:24.032 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:25:24.079 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:25:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:25:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:25:24.303 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:25:24.359 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:25:24.373 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:25:24.392 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:25:24.425 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:25:24.463 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:25:24.482 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:25:24.521 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:25:24.596 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:25:24.635 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:25:24.667 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:25:24.704 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:25:24.750 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:25:24.819 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:25:24.845 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:25:24.938 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:25:25.023 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:25:25.090 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:25:25.199 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:25:25.222 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:25:25.357 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:25:25.413 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:25:25.484 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:25:25.560 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:25:25.809 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:25:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:25:26.021 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:25:26.043 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:25:26.093 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:25:26.130 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:25:26.154 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:25:26.171 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:25:26.219 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:25:26.255 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:25:26.299 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:25:26.353 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:25:26.423 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:25:26.476 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:25:26.524 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:25:26.629 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:25:26.685 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:25:26.720 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:25:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:25:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:25:26.962 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:25:27.084 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:25:27.137 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:25:27.172 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:25:27.232 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:25:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:25:27.463 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:25:27.696 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:25:27.787 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:25:27.822 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:25:28.182 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:25:28.207 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:28.326 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:28.355 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:25:28.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:25:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:25:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:25:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:25:28.601 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:25:28.714 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:25:28.754 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:25:28.921 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:25:28.953 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:28.994 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:25:29.026 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:25:29.043 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:29.071 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:25:29.122 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:25:29.961 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:25:30.101 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:25:31.692 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:31.708 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:25:31.778 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:31.799 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:25:31.833 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:25:32.205 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:25:32.236 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:32.357 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:25:32.392 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:25:32.402 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:32.447 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:25:32.606 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:32.606 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:25:32.606 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:25:32.607 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:32.656 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:32.983 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:25:33.014 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:33.097 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:25:33.116 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:25:35.778 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:35.829 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:25:36.541 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:36.721 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:25:36.721 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:25:36.721 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:25:36.721 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:36.767 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /root 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-03-06T07:25:45.592 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:45.826 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:25:45.869 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:25:45.869 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:45.869 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:25:45.869 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:47.521 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:25:47.597 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:25:47.597 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:47.597 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:25:47.597 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:25:47.597 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:25:47.597 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:47.829 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:25:47.898 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:25:47.899 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:47.899 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:25:47.899 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:25:47.899 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:25:47.899 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:48.380 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:25:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:25:48.449 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:48.449 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:25:48.449 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:25:48.449 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:25:48.449 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:48.565 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:25:49.320 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:25:49.349 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:25:49.397 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:25:49.484 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:25:49.528 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:25:49.848 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:25:49.895 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-03-06T07:25:50.181 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-03-06T07:25:50.182 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-03-06T07:25:50.182 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:50.390 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:25:50.442 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:25:50.442 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:50.442 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:25:50.442 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:51.004 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:25:51.037 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:25:51.177 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:25:51.350 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:25:51.381 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:25:51.448 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:25:51.448 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:51.448 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:25:51.449 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:25:51.449 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:25:51.449 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:51.507 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:25:51.544 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:25:51.654 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:25:52.108 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:25:52.189 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:25:52.189 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:52.189 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:25:52.189 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:25:52.190 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:25:52.190 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:52.459 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:25:52.542 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:25:52.542 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:52.542 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:25:52.542 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:25:52.542 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:25:52.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:52.591 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:25:52.626 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:25:52.693 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:25:52.693 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:52.693 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:25:52.694 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:25:52.694 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:25:52.694 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:52.760 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:25:52.830 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:25:52.830 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:52.830 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:25:52.830 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:52.910 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:25:52.996 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:25:52.996 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:52.996 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:25:52.996 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:25:52.997 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:25:52.997 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:53.059 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:25:53.132 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:25:53.133 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:53.133 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:25:53.133 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:25:53.133 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:25:53.133 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:53.253 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:25:53.394 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:25:53.466 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:25:53.466 INFO:teuthology.orchestra.run.smithi178.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:53.466 INFO:teuthology.orchestra.run.smithi178.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:25:53.466 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:25:53.466 INFO:teuthology.orchestra.run.smithi178.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:25:53.466 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:25:53.871 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:25:53.913 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:25:53.965 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:25:54.068 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:25:54.108 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:25:54.445 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:25:55.648 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:25:55.684 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:25:55.832 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:25:56.039 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:25:56.069 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:25:56.153 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:25:56.153 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:56.153 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:25:56.153 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:25:56.153 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:25:56.154 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:56.228 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:25:56.268 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:25:56.371 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:25:57.405 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:25:57.439 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:25:57.511 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:25:57.511 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:57.511 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:25:57.511 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:25:57.511 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:25:57.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:57.565 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:25:57.637 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:25:57.637 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:57.637 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:25:57.637 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:57.724 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:25:57.804 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:25:57.804 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:57.804 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:25:57.804 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:25:57.804 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:25:57.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:58.224 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:25:58.233 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:25:58.278 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:25:58.298 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:25:58.298 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:25:58.299 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:25:58.299 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:25:58.299 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:25:58.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:25:58.316 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:25:58.353 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:25:58.631 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:25:58.761 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:25:58.810 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:25:58.861 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:25:58.925 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:25:59.032 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:25:59.102 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:25:59.149 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:25:59.177 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:25:59.228 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:25:59.339 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:25:59.403 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:25:59.436 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:26:00.300 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:00.342 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:00.394 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:01.705 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:01.780 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:01.852 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:01.978 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:02.016 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:02.067 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:02.100 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:02.141 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:02.142 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:02.283 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:02.333 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:02.379 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:02.379 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:02.568 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:02.611 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:02.668 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:02.707 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:02.754 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:02.800 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:02.882 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:02.963 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:03.013 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:03.046 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:03.080 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:03.113 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:03.147 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:03.183 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:03.183 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:03.381 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:26:03.425 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:26:03.461 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:26:03.488 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:03.488 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:03.499 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:26:03.636 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:03.689 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:03.725 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:03.758 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:03.794 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:26:03.929 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:26:03.939 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:26:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:26:04.121 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi178.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:04.229 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:04.232 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:26:04.274 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:26:04.320 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:26:04.347 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:26:04.400 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:26:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:26:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:26:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:26:05.679 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:05.728 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:05.785 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:26:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:26:07.291 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:26:07.337 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:26:07.455 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:26:07.496 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:26:07.529 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:26:07.572 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:26:07.615 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:26:07.616 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:07.746 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:26:07.799 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:26:07.843 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:26:07.843 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:07.982 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:26:08.028 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:26:08.091 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:26:08.130 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:26:08.176 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:26:08.222 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:26:08.292 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:26:08.369 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:26:08.444 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:26:08.476 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:26:08.510 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:26:08.555 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:08.585 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:08.622 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:26:08.970 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:09.108 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:26:09.161 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:09.198 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:26:09.230 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:26:09.270 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:09.271 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:09.272 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:09.286 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:26:09.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:09.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:09.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:09.288 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:09.289 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:09.290 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:09.291 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:09.292 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:09.293 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:09.294 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:09.295 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:09.296 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:09.297 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:09.298 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:09.299 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:09.300 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:09.416 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:09.661 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:26:09.661 INFO:teuthology.orchestra.run.smithi006.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-06T07:26:09.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:11.875 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.876 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:11.877 INFO:teuthology.orchestra.run.smithi178.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.878 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:11.879 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:11.880 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:11.881 INFO:teuthology.orchestra.run.smithi178.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:11.882 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:11.883 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:11.884 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:11.885 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:11.886 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:11.887 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:11.888 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:11.889 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:11.889 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:11.889 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-03-06T07:26:12.554 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:14.526 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:26:14.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:26:14.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:26:14.529 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:26:14.530 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:26:14.531 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:26:14.532 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:26:14.533 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:26:14.533 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:26:14.533 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:26:14.533 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:26:14.533 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:26:14.533 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:26:14.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:26:14.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:26:14.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:26:14.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:26:14.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:26:14.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:26:14.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:26:14.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:26:14.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:26:14.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:26:14.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:26:14.535 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:26:14.536 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:26:14.536 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:26:14.536 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:26:14.536 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:26:14.536 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:26:14.537 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:26:14.538 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:26:14.539 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:26:14.539 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:26:14.539 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:26:14.539 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:26:14.539 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:26:14.539 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:26:14.540 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:26:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:26:14.542 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:26:14.542 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:26:14.542 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:26:14.542 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:26:14.542 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:26:14.542 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:26:14.543 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:26:14.543 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:26:14.543 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:26:14.543 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:26:14.543 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:26:14.543 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:26:14.544 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:26:14.544 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:26:14.544 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:26:14.544 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:26:14.544 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:26:14.544 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:26:14.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:26:14.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:26:14.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:26:14.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:26:14.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:26:14.545 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:26:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:26:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:26:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:26:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:26:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:26:14.546 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:26:14.547 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:26:14.548 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:26:14.548 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:26:14.548 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:26:14.548 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:26:14.548 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:26:14.549 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:26:14.550 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:26:14.550 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:26:14.550 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:26:14.550 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:26:14.550 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:26:14.550 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:26:14.551 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:26:14.552 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:26:14.552 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:26:14.552 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:26:14.552 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:26:14.552 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:26:14.553 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:26:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:26:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:26:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:26:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:26:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:26:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:26:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:26:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:26:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:26:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:17.044 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-06T07:26:17.716 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:17.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:26:17.938 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:26:17.996 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:17.996 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:17.996 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:26:17.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:26:18.220 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:26:18.290 INFO:teuthology.orchestra.run.smithi178.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:18.291 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:26:18.291 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:26:18.293 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:26:18.293 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:18.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:26:18.375 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:18.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:26:18.470 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:26:18.470 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:18.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:26:18.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:26:18.661 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:18.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:26:18.759 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:26:18.853 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:26:18.853 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:18.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:26:18.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:26:19.047 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:19.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:26:19.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:26:19.276 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:26:19.277 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:19.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:26:19.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:26:19.463 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:19.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:26:19.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:26:19.648 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:26:19.732 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:26:19.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-03-06T07:26:19.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /var/log/ceph 2024-03-06T07:26:19.830 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:26:19.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:26:19.835 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:26:19.929 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:26:19.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:26:19.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:26:20.028 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:26:20.029 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:26:20.029 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:26:20.029 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:20.029 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:26:20.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:20.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:20.341 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:20.341 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:26:20.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:20.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:20.763 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:26:20.763 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:26:20.764 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583068', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583068', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'kclient': {'syntax': 'v2'}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGnLa4PUoQbbr777PfGO5g31qCUBVETbPZq9txwJ4Lxtdzps2caejRBUHlg+pkJr04pulBsJMdAfxPL0IKh9o14=', 'smithi178.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuwpLlO9gIMelvUOWA133EQLE3kA0YlFdD17jgdGOnoCBjYoz/fyj8X1WIjxYiwHQ1ZGPdII5hTaDUN2gjmjFA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:26:20.764 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:26:20.769 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:26:20.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:26:20.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:26:20.940 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:20.940 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:26:20.978 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:26:20.979 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-06 07:26:10.091831239 +0000 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-06 07:23:15.286966452 +0000 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-06 07:23:15.286966452 +0000 2024-03-06T07:26:21.061 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-06 07:23:15.286966452 +0000 2024-03-06T07:26:21.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:26:21.176 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-06T07:26:21.176 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-06T07:26:21.176 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000323757 s, 1.6 MB/s 2024-03-06T07:26:21.179 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:26:21.218 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-06T07:26:21.296 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-06 07:26:10.092831227 +0000 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-06 07:23:15.884959750 +0000 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-06 07:23:15.884959750 +0000 2024-03-06T07:26:21.297 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-06 07:23:15.884959750 +0000 2024-03-06T07:26:21.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:26:21.400 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-06T07:26:21.401 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000256502 s, 2.0 MB/s 2024-03-06T07:26:21.404 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:26:21.443 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-06T07:26:21.525 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-06 07:26:10.092831227 +0000 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-06 07:23:16.483953036 +0000 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-06 07:23:16.483953036 +0000 2024-03-06T07:26:21.526 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-06 07:23:16.483953036 +0000 2024-03-06T07:26:21.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:26:21.638 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-06T07:26:21.638 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-06T07:26:21.638 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000339738 s, 1.5 MB/s 2024-03-06T07:26:21.641 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:26:21.680 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-06T07:26:21.762 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:26:21.762 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-06 07:26:10.093831216 +0000 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-06 07:23:17.078946368 +0000 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-06 07:23:17.078946368 +0000 2024-03-06T07:26:21.763 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-06 07:23:17.078946368 +0000 2024-03-06T07:26:21.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:26:21.872 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-06T07:26:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-06T07:26:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000307111 s, 1.7 MB/s 2024-03-06T07:26:21.875 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:26:21.919 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:21.919 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:21.920 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:26:21.978 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:26:21.978 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2024-03-06T07:26:22.029 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:26:22.053 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-06 07:26:04.673618861 +0000 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-06 07:23:06.375692750 +0000 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-06 07:23:06.375692750 +0000 2024-03-06T07:26:22.054 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-06 07:23:06.375692750 +0000 2024-03-06T07:26:22.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-06T07:26:22.147 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000227756 s, 2.2 MB/s 2024-03-06T07:26:22.151 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:26:22.208 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2024-03-06T07:26:22.257 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:26:22.257 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:22.257 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-06T07:26:22.257 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:22.257 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:22.258 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-06 07:26:04.673618861 +0000 2024-03-06T07:26:22.258 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-06 07:23:06.933682824 +0000 2024-03-06T07:26:22.258 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-06 07:23:06.933682824 +0000 2024-03-06T07:26:22.258 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-06 07:23:06.933682824 +0000 2024-03-06T07:26:22.258 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:26:22.342 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-06T07:26:22.343 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-06T07:26:22.343 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000215392 s, 2.4 MB/s 2024-03-06T07:26:22.347 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:26:22.401 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-06 07:26:04.675618825 +0000 2024-03-06T07:26:22.450 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-06 07:23:07.499672756 +0000 2024-03-06T07:26:22.451 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-06 07:23:07.499672756 +0000 2024-03-06T07:26:22.451 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-06 07:23:07.499672756 +0000 2024-03-06T07:26:22.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:26:22.529 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-06T07:26:22.529 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-06T07:26:22.530 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000291711 s, 1.8 MB/s 2024-03-06T07:26:22.534 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:26:22.587 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2024-03-06 07:26:04.675618825 +0000 2024-03-06T07:26:22.636 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2024-03-06 07:23:08.050662954 +0000 2024-03-06T07:26:22.637 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2024-03-06 07:23:08.050662954 +0000 2024-03-06T07:26:22.637 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: 2024-03-06 07:23:08.050662954 +0000 2024-03-06T07:26:22.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:26:22.722 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2024-03-06T07:26:22.723 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2024-03-06T07:26:22.723 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000222083 s, 2.3 MB/s 2024-03-06T07:26:22.726 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:26:22.782 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:22.782 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:26:22.782 INFO:tasks.ceph:Generating config... 2024-03-06T07:26:22.783 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:26:22.783 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:26:22.783 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:26:22.784 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:26:22.784 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:26:22.784 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:26:22.784 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:26:22.784 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:26:22.785 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:26:22.785 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:26:22.786 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:26:22.786 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:26:22.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:26:22.968 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:26:22.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:26:23.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:23.208 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.178'), ('mon.c', '[v2:172.21.15.178:3301,v1:172.21.15.178:6790]')] 2024-03-06T07:26:23.209 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:26:23.210 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:23.210 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:26:23.267 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.178 --addv c '[v2:172.21.15.178:3301,v1:172.21.15.178:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:23.417 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:fsid e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-03-06T07:26:23.416873+0000 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-03-06T07:26:23.416873+0000 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon.b 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] mon.c 2024-03-06T07:26:23.418 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:26:23.421 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:26:23.474 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e57361cb-720d-49c4-a325-3e55e1c43024... 2024-03-06T07:26:23.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:23.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.648 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.649 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:26:23.650 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:23.651 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:23.652 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:26:23.653 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:26:23.654 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:26:23.655 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.656 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:26:23.657 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:26:23.658 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:23.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.660 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.661 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.662 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:26:23.663 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:26:23.664 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.665 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:23.666 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:23.667 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:26:23.668 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:26:23.669 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:26:23.703 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:26:23.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:26:23.857 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:26:23.857 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:23.858 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:26:23.906 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:23.906 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:26:24.003 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:26:24.003 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:24.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:26:24.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:24.157 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:24.157 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:24.257 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:26:24.257 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:24.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:26:24.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:26:24.407 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:24.408 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:24.503 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:26:24.503 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:26:24.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:26:24.704 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:26:24.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:26:24.906 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:26:24.919 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:26:24.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:26:25.116 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:26:25.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:26:25.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:26:25.429 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:26:25.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:26:25.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:26:25.713 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:26:25.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:26:25.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:26:25.991 INFO:teuthology.orchestra.run.smithi178.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:26:26.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:26:26.091 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:26:26.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:26:26.226 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:26:26.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:26:26.420 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:26:26.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:26:26.608 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:26:26.662 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:26:26.787 INFO:teuthology.orchestra.run.smithi178.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:26:26.840 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:26:26.841 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi178.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:26:26.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:26.947 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:26.947 INFO:tasks.ceph:role: osd.0 2024-03-06T07:26:26.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:26:26.947 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:26:27.053 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:27.054 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:27.563 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-06T07:26:27.570 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-06T07:26:27.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:27.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:27.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:27.963 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:27.963 INFO:tasks.ceph:role: osd.1 2024-03-06T07:26:27.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:26:27.963 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:26:28.063 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:28.063 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:28.063 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:28.063 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:28.063 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:28.064 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:28.064 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:28.064 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:28.064 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:28.064 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:28.555 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-06T07:26:28.562 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-06T07:26:28.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:28.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:28.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:28.914 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:28.914 INFO:tasks.ceph:role: osd.2 2024-03-06T07:26:28.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:26:28.914 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:26:29.015 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:29.016 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:29.531 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-06T07:26:29.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-06T07:26:29.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:29.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:29.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:29.944 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:29.944 INFO:tasks.ceph:role: osd.3 2024-03-06T07:26:29.944 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:26:29.944 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:26:30.047 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:30.047 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:30.047 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:30.047 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:30.047 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:30.048 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:30.048 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:30.048 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:30.048 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:30.048 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:30.523 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-03-06T07:26:30.530 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-03-06T07:26:30.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:30.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:30.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:30.971+0000 7fc7ab102640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:30.977+0000 7fc7ab102640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:30.977+0000 7fc7ab102640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:30.977+0000 7fc7ab102640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:30.977+0000 7fc7ab102640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:26:30.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:30.989+0000 7fc7ab102640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:26:34.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:26:34.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:34.407+0000 7fee2d2b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:26:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:34.408+0000 7fee2d2b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:26:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:34.408+0000 7fee2d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:34.408+0000 7fee2d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:34.408+0000 7fee2d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:26:34.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:34.411+0000 7fee2d2b5640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:26:37.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:26:37.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:37.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:37.836+0000 7f1ab6ad9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:26:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:37.837+0000 7f1ab6ad9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:26:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:37.837+0000 7f1ab6ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:26:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:37.837+0000 7f1ab6ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:26:37.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:37.837+0000 7f1ab6ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:26:37.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:37.839+0000 7f1ab6ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:26:40.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:26:40.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:41.025+0000 7f6c51068640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:41.025+0000 7f6c51068640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:41.025+0000 7f6c51068640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:41.025+0000 7f6c51068640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:41.025+0000 7f6c51068640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:26:41.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:26:41.028+0000 7f6c51068640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:26:43.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:26:44.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:44.164 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:44.165 INFO:tasks.ceph:role: osd.4 2024-03-06T07:26:44.165 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:26:44.165 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:44.257 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:44.290 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-06T07:26:44.297 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-06T07:26:44.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:44.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:44.494 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:26:44.569 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:44.569 INFO:tasks.ceph:role: osd.5 2024-03-06T07:26:44.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:26:44.570 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:44.651 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:44.652 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:44.684 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-06T07:26:44.690 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-06T07:26:44.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:26:44.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:26:44.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:26:44.946 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:44.946 INFO:tasks.ceph:role: osd.6 2024-03-06T07:26:44.946 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:26:44.946 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:45.016 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:45.049 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-06T07:26:45.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-06T07:26:45.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:26:45.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:26:45.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:26:45.288 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:26:45.288 INFO:tasks.ceph:role: osd.7 2024-03-06T07:26:45.289 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:26:45.289 DEBUG:teuthology.orchestra.run.smithi178:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:26:45.361 INFO:teuthology.orchestra.run.smithi178.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:26:45.361 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi178.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:26:45.395 INFO:teuthology.orchestra.run.smithi178.stdout:Discarding blocks...Done. 2024-03-06T07:26:45.402 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi178.front.sepia.ceph.com -o noatime 2024-03-06T07:26:45.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:26:45.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:26:45.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:45.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:45.708+0000 7fb7a9a21640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:26:45.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:45.709+0000 7fb7a9a21640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:26:45.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:45.709+0000 7fb7a9a21640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:26:45.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:45.709+0000 7fb7a9a21640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:26:45.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:45.709+0000 7fb7a9a21640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:26:45.714 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:45.712+0000 7fb7a9a21640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:26:48.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:26:48.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:48.865 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:48.863+0000 7f5215a1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:26:48.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:48.863+0000 7f5215a1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:26:48.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:48.863+0000 7f5215a1d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:26:48.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:48.864+0000 7f5215a1d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:26:48.866 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:48.864+0000 7f5215a1d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:26:48.868 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:48.866+0000 7f5215a1d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:26:50.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:50.773 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:51.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:26:51.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:52.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:52.009+0000 7f2ccc015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:26:52.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:52.010+0000 7f2ccc015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:26:52.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:52.010+0000 7f2ccc015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:26:52.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:52.010+0000 7f2ccc015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:26:52.012 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:52.010+0000 7f2ccc015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:26:52.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:52.012+0000 7f2ccc015640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:26:55.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:26:55.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:26:55.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:55.411+0000 7f5ed420d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:26:55.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:55.412+0000 7f5ed420d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:26:55.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:55.412+0000 7f5ed420d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:26:55.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:55.412+0000 7f5ed420d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:26:55.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:55.412+0000 7f5ed420d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:26:55.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:26:55.414+0000 7f5ed420d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:26:58.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:26:58.439 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:26:58.440 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:58.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:26:58.544 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:58.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:26:58.645 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:58.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:26:58.746 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:58.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:26:58.848 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:58.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:26:58.952 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:58.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:26:59.058 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:59.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:26:59.147 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:26:59.246 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.246 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:26:59.339 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.339 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:26:59.422 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:26:59.496 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:26:59.631 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:26:59.715 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:26:59.806 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:59.806 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:26:59.868 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:26:59.868 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:26:59.922 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.922 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:26:59.978 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:26:59.978 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:27:00.030 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:27:00.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:27:00.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:27:00.127 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.y] 2024-03-06T07:27:00.127 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA== 2024-03-06T07:27:00.127 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:27:00.127 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow== 2024-03-06T07:27:00.127 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:27:00.127 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg== 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.0] 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng== 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.1] 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ== 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.2] 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w== 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.3] 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw== 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr.x] 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g== 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:27:00.128 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ== 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg== 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.4] 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww== 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.5] 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA== 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.6] 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ== 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout:[osd.7] 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw== 2024-03-06T07:27:00.129 INFO:teuthology.orchestra.run.smithi178.stdout:[client.0] 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout:[client.2] 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ== 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout:[client.1] 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ== 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout:[client.3] 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi178.stdout: key = AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw== 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-03-06T07:27:00.130 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA== 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow== 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg== 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng== 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ== 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w== 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-03-06T07:27:00.131 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw== 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g== 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ== 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg== 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww== 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA== 2024-03-06T07:27:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ== 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw== 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout:[client.2] 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ== 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout:[client.1] 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ== 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout:[client.3] 2024-03-06T07:27:00.133 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw== 2024-03-06T07:27:00.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:00.142 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:00.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:00.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:00.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:00.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:00.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:00.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:00.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:00.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:00.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:00.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:01.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:27:01.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:01.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:01.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:01.492 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:27:01.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.799 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:01.951 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:02.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:02.103 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:27:02.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:27:02.849 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:27:02.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:27:02.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:03.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:27:03.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:27:03.307 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:03.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:27:03.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:27:03.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:27:03.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:27:03.870 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:03.877 DEBUG:teuthology.orchestra.run.smithi178:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:27:03.932 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:27:03.932 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:27:03.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:27:03.938 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:27:03.938 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:27:03.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:27:03.942 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:27:03.942 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:27:03.942 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:27:03.947 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:27:03.947 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:27:03.947 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:27:03.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:27:03.953 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:27:03.953 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:27:03.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:27:03.958 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:27:03.958 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:27:03.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:27:04.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.141+0000 7f3711600640 1 Processor -- start 2024-03-06T07:27:04.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.141+0000 7f3711600640 1 -- start start 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.142+0000 7f3711600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 0x7f370c0d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.142+0000 7f3711600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 0x7f370c0d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f3711600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f3711600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c0d20d0 con 0x7f370c0d0f80 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f3711600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c0d4df0 con 0x7f370c161930 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f3711600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0d1df0 con 0x7f370c0c7d10 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:49340/0 (socket says 172.21.15.6:49340) 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 -- 172.21.15.6:0/3694175087 learned_addr learned my addr 172.21.15.6:0/3694175087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:04.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 -- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 msgr2=0x7f370c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 -- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 msgr2=0x7f370c0c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 --2- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370b400640 1 --2- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370aa00640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 0x7f370c0d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370aa00640 1 -- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 msgr2=0x7f370c0d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370aa00640 1 -- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 msgr2=0x7f370c0d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.143+0000 7f370aa00640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 0x7f370c0d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.144+0000 7f370aa00640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 0x7f370c0d24c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.144+0000 7f3703e00640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 0x7f370c0d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.144+0000 7f3703e00640 1 -- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 msgr2=0x7f370c0d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.144+0000 7f3703e00640 1 -- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 msgr2=0x7f370c0d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.144+0000 7f3703e00640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 0x7f370c0d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:27:04.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.144+0000 7f3703e00640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 0x7f370c0d2a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:27:04.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.344+0000 7f370b400640 1 --2- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.345+0000 7f370b400640 1 -- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 msgr2=0x7f370c0d24c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:27:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.345+0000 7f370b400640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 0x7f370c0d24c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.345+0000 7f370b400640 1 -- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 msgr2=0x7f370c0d2a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:27:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.345+0000 7f370b400640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 0x7f370c0d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.345+0000 7f370b400640 1 -- 172.21.15.6:0/3694175087 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c0d4b10 con 0x7f370c0c7d10 2024-03-06T07:27:04.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.347+0000 7f370b400640 1 --2- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f370c08d000 tx=0x7f370402c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=729b7ab7b0bf1d58 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.348+0000 7f370a000640 1 -- 172.21.15.6:0/3694175087 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704032070 con 0x7f370c0c7d10 2024-03-06T07:27:04.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.348+0000 7f370a000640 1 -- 172.21.15.6:0/3694175087 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370402cdb0 con 0x7f370c0c7d10 2024-03-06T07:27:04.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.348+0000 7f370a000640 1 -- 172.21.15.6:0/3694175087 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704036600 con 0x7f370c0c7d10 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 -- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 msgr2=0x7f370c0c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 --2- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f370c08d000 tx=0x7f370402c000 comp rx=0 tx=0).stop 2024-03-06T07:27:04.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 -- 172.21.15.6:0/3694175087 shutdown_connections 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 --2- 172.21.15.6:0/3694175087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c0c8180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.6:3300/0 conn(0x7f370c0d0f80 0x7f370c0d2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 --2- 172.21.15.6:0/3694175087 >> v2:172.21.15.178:3300/0 conn(0x7f370c161930 0x7f370c0d24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 -- 172.21.15.6:0/3694175087 >> 172.21.15.6:0/3694175087 conn(0x7f370c0d3ee0 msgr2=0x7f370c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.349+0000 7f3711600640 1 -- 172.21.15.6:0/3694175087 shutdown_connections 2024-03-06T07:27:04.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.350+0000 7f3711600640 1 -- 172.21.15.6:0/3694175087 wait complete. 2024-03-06T07:27:04.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.352+0000 7f3711600640 1 Processor -- start 2024-03-06T07:27:04.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.353+0000 7f3711600640 1 -- start start 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3711600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3711600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 0x7f370c15d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3711600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c161930 0x7f370c06c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3711600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0c86c0 con 0x7f370c0d0f80 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3711600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0b4f30 con 0x7f370c161930 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3711600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0c8840 con 0x7f370c0c7d10 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.354+0000 7f3703e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 0x7f370c15d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 0x7f370c15d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39918/0 (socket says 172.21.15.6:39918) 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 -- 172.21.15.6:0/3192837749 learned_addr learned my addr 172.21.15.6:0/3192837749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f370aa00640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f370b400640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c161930 0x7f370c06c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 -- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 msgr2=0x7f370c1542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c1542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 -- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c161930 msgr2=0x7f370c06c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c161930 0x7f370c06c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.355+0000 7f3703e00640 1 -- 172.21.15.6:0/3192837749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c0a1b60 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3703e00640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 0x7f370c15d3d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f36f400d800 tx=0x7f36f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f40096a0 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f4019070 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f4009a60 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c0a0ac0 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c0a1080 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.356+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f36f4009c00 con 0x7f370c0d0f80 2024-03-06T07:27:04.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.357+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f36f4014630 con 0x7f370c0d0f80 2024-03-06T07:27:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.358+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c04f330 con 0x7f370c0d0f80 2024-03-06T07:27:04.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.360+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f36f4014830 con 0x7f370c0d0f80 2024-03-06T07:27:04.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.436+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f370c0d0b60 con 0x7f370c0d0f80 2024-03-06T07:27:04.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.440+0000 7f3702a00640 1 -- 172.21.15.6:0/3192837749 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f36f4014a10 con 0x7f370c0d0f80 2024-03-06T07:27:04.443 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 msgr2=0x7f370c15d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 0x7f370c15d3d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f36f400d800 tx=0x7f36f400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 shutdown_connections 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f370c161930 0x7f370c06c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f370c0d0f80 0x7f370c15d3d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 --2- 172.21.15.6:0/3192837749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f370c0c7d10 0x7f370c1542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.442+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 >> 172.21.15.6:0/3192837749 conn(0x7f370c0d3ee0 msgr2=0x7f370c15ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.443+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 shutdown_connections 2024-03-06T07:27:04.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:04.443+0000 7f3711600640 1 -- 172.21.15.6:0/3192837749 wait complete. 2024-03-06T07:27:04.461 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:27:04.462 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:27:04.462 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:27:04.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:27:04.545 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:27:04.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:27:04.674 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:27:04.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:27:04.757 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:27:04.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:27:04.889 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:27:04.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:27:04.970 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:04.917+0000 7fb3af817040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:27:04.970 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:04.947+0000 7fce2aa4d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:27:04.988 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:04.985+0000 7fb3af817040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:27:05.019 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:05.017+0000 7fce2aa4d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:27:05.049 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:27:05.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:27:05.174 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:27:05.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:27:05.298 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:27:05.298 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:27:05.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 4d3c7a7e-7b00-4ed6-829d-322f3c73cc69 0 2024-03-06T07:27:05.608 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:05.606+0000 7fb3af817040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:27:05.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.622+0000 7fc964a00640 1 Processor -- start 2024-03-06T07:27:05.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.622+0000 7fc964a00640 1 -- start start 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc964a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc960161840 0x7fc9600d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc964a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9600cfad0 0x7fc9600d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc964a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9600c7a00 con 0x7fc9600d4a40 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc964a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc960161c20 con 0x7fc960161840 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc964a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9600d2530 con 0x7fc9600cfad0 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc95e800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9600cfad0 0x7fc9600d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc95e800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9600cfad0 0x7fc9600d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33550/0 (socket says 172.21.15.178:33550) 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc95e800640 1 -- 172.21.15.178:0/2456784273 learned_addr learned my addr 172.21.15.178:0/2456784273 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc95de00640 1 --2- 172.21.15.178:0/2456784273 >> v2:172.21.15.6:3300/0 conn(0x7fc960161840 0x7fc9600d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc95e800640 1 -- 172.21.15.178:0/2456784273 >> v2:172.21.15.6:3300/0 conn(0x7fc960161840 msgr2=0x7fc9600d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.624+0000 7fc95e800640 1 --2- 172.21.15.178:0/2456784273 >> v2:172.21.15.6:3300/0 conn(0x7fc960161840 0x7fc9600d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.625+0000 7fc95e800640 1 -- 172.21.15.178:0/2456784273 >> v1:172.21.15.178:6789/0 conn(0x7fc9600d4a40 legacy=0x7fc9600c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.625+0000 7fc95e800640 1 -- 172.21.15.178:0/2456784273 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600d2810 con 0x7fc9600cfad0 2024-03-06T07:27:05.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.626+0000 7fc95e800640 1 --2- 172.21.15.178:0/2456784273 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600cfad0 0x7fc9600d0d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc96008cd60 tx=0x7fc95802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45b4a16314cd9db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.626+0000 7fc95ca00640 1 -- 172.21.15.178:0/2456784273 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95802ca80 con 0x7fc9600cfad0 2024-03-06T07:27:05.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.626+0000 7fc95ca00640 1 -- 172.21.15.178:0/2456784273 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95802cc20 con 0x7fc9600cfad0 2024-03-06T07:27:05.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.626+0000 7fc95ca00640 1 -- 172.21.15.178:0/2456784273 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9580343f0 con 0x7fc9600cfad0 2024-03-06T07:27:05.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 -- 172.21.15.178:0/2456784273 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600cfad0 msgr2=0x7fc9600d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 --2- 172.21.15.178:0/2456784273 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600cfad0 0x7fc9600d0d30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc96008cd60 tx=0x7fc95802bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 -- 172.21.15.178:0/2456784273 shutdown_connections 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 --2- 172.21.15.178:0/2456784273 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600cfad0 0x7fc9600d0d30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 --2- 172.21.15.178:0/2456784273 >> v2:172.21.15.6:3300/0 conn(0x7fc960161840 0x7fc9600d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 -- 172.21.15.178:0/2456784273 >> 172.21.15.178:0/2456784273 conn(0x7fc9600d3eb0 msgr2=0x7fc9600d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 -- 172.21.15.178:0/2456784273 shutdown_connections 2024-03-06T07:27:05.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.627+0000 7fc964a00640 1 -- 172.21.15.178:0/2456784273 wait complete. 2024-03-06T07:27:05.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.629+0000 7fc964a00640 1 Processor -- start 2024-03-06T07:27:05.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.629+0000 7fc964a00640 1 -- start start 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc964a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 0x7fc96014f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc964a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 0x7fc9600b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc964a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc960161840 0x7fc9600b8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc964a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600d1750 con 0x7fc960161840 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc964a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600d2dd0 con 0x7fc9600d4a40 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc964a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9600c8610 con 0x7fc9600cfad0 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95d400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 0x7fc9600b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 0x7fc96014f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 0x7fc96014f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50482/0 (socket says 172.21.15.178:50482) 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 -- 172.21.15.178:0/1305245402 learned_addr learned my addr 172.21.15.178:0/1305245402 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:05.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95d400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 0x7fc9600b89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33554/0 (socket says 172.21.15.178:33554) 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95e800640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc960161840 0x7fc9600b8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 -- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 msgr2=0x7fc9600b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 0x7fc9600b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 -- 172.21.15.178:0/1305245402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc960161840 msgr2=0x7fc9600b8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc960161840 0x7fc9600b8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.630+0000 7fc95de00640 1 -- 172.21.15.178:0/1305245402 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9600b7e10 con 0x7fc9600cfad0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc95d400640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 0x7fc9600b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc95de00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 0x7fc96014f1b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e870 tx=0x7fc95000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950017070 con 0x7fc9600cfad0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95000c940 con 0x7fc9600cfad0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9600cbb60 con 0x7fc9600cfad0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000cd30 con 0x7fc9600cfad0 2024-03-06T07:27:05.635 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9600cc120 con 0x7fc9600cfad0 2024-03-06T07:27:05.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.631+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc95001a070 con 0x7fc9600cfad0 2024-03-06T07:27:05.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.632+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc950013470 con 0x7fc9600cfad0 2024-03-06T07:27:05.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.632+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc960068010 con 0x7fc9600cfad0 2024-03-06T07:27:05.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.635+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc9600b7e10 con 0x7fc9600cfad0 2024-03-06T07:27:05.677 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:05.674+0000 7fb3af817040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:27:05.689 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:05.687+0000 7fce2aa4d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:27:05.716 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.714+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4d3c7a7e-7b00-4ed6-829d-322f3c73cc69", "id": 0} v 0) v1 -- 0x7fc9600d03c0 con 0x7fc9600cfad0 2024-03-06T07:27:05.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.721+0000 7fc956000640 1 -- 172.21.15.178:0/1305245402 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d3c7a7e-7b00-4ed6-829d-322f3c73cc69", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc950005990 con 0x7fc9600cfad0 2024-03-06T07:27:05.724 INFO:teuthology.orchestra.run.smithi178.stdout:0 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 msgr2=0x7fc96014f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 0x7fc96014f1b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e870 tx=0x7fc95000ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:05.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 shutdown_connections 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9600d4a40 0x7fc9600b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc960161840 0x7fc9600b8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 --2- 172.21.15.178:0/1305245402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9600cfad0 0x7fc96014f1b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 >> 172.21.15.178:0/1305245402 conn(0x7fc9600d3eb0 msgr2=0x7fc9601605b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 shutdown_connections 2024-03-06T07:27:05.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.723+0000 7fc964a00640 1 -- 172.21.15.178:0/1305245402 wait complete. 2024-03-06T07:27:05.734 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:05.732+0000 7fb3af817040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:27:05.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new be5e1794-8b2f-44e0-a366-01f108e9a562 1 2024-03-06T07:27:05.757 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:05.755+0000 7fce2aa4d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:27:05.804 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:05.802+0000 7fb3af817040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:27:05.810 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:05.808+0000 7fce2aa4d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:27:05.857 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:05.855+0000 7fb3af817040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:27:05.875 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:05.874+0000 7fce2aa4d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:27:05.928 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:05.926+0000 7fce2aa4d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.940+0000 7f7341e00640 1 Processor -- start 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.941+0000 7f7341e00640 1 -- start start 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7341e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f733c161580 0x7f733c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7341e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f733c161960 0x7f733c0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7341e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c0b3150 con 0x7f733c0d4800 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7341e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733c0d26f0 con 0x7f733c161580 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7341e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733c0b2b90 con 0x7f733c161960 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f733be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f733c161960 0x7f733c0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f733be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f733c161960 0x7f733c0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33560/0 (socket says 172.21.15.178:33560) 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f733be00640 1 -- 172.21.15.178:0/972432297 learned_addr learned my addr 172.21.15.178:0/972432297 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7341400640 1 --1- 172.21.15.178:0/972432297 >> v1:172.21.15.178:6789/0 conn(0x7f733c0d4800 0x7f733c0b0d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:47482/0 (socket says 172.21.15.178:47482) 2024-03-06T07:27:05.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.942+0000 7f7340a00640 1 --2- 172.21.15.178:0/972432297 >> v2:172.21.15.6:3300/0 conn(0x7f733c161580 0x7f733c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.943+0000 7f733be00640 1 -- 172.21.15.178:0/972432297 >> v2:172.21.15.6:3300/0 conn(0x7f733c161580 msgr2=0x7f733c0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.943+0000 7f733be00640 1 --2- 172.21.15.178:0/972432297 >> v2:172.21.15.6:3300/0 conn(0x7f733c161580 0x7f733c0b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.943+0000 7f733be00640 1 -- 172.21.15.178:0/972432297 >> v1:172.21.15.178:6789/0 conn(0x7f733c0d4800 legacy=0x7f733c0b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.943+0000 7f733be00640 1 -- 172.21.15.178:0/972432297 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0b2e70 con 0x7f733c161960 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.944+0000 7f733be00640 1 --2- 172.21.15.178:0/972432297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c161960 0x7f733c0b0820 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f733c08d630 tx=0x7f732c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92bc21f9afc6e0ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.944+0000 7f733b400640 1 -- 172.21.15.178:0/972432297 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c035020 con 0x7f733c161960 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.944+0000 7f733b400640 1 -- 172.21.15.178:0/972432297 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c031870 con 0x7f733c161960 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.944+0000 7f733b400640 1 -- 172.21.15.178:0/972432297 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c031c40 con 0x7f733c161960 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 -- 172.21.15.178:0/972432297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c161960 msgr2=0x7f733c0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 --2- 172.21.15.178:0/972432297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c161960 0x7f733c0b0820 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f733c08d630 tx=0x7f732c02b910 comp rx=0 tx=0).stop 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 -- 172.21.15.178:0/972432297 shutdown_connections 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 --2- 172.21.15.178:0/972432297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c161960 0x7f733c0b0820 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 --2- 172.21.15.178:0/972432297 >> v2:172.21.15.6:3300/0 conn(0x7f733c161580 0x7f733c0b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 -- 172.21.15.178:0/972432297 >> 172.21.15.178:0/972432297 conn(0x7f733c0d3c70 msgr2=0x7f733c0d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 -- 172.21.15.178:0/972432297 shutdown_connections 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.945+0000 7f7341e00640 1 -- 172.21.15.178:0/972432297 wait complete. 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.947+0000 7f7341e00640 1 Processor -- start 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.947+0000 7f7341e00640 1 -- start start 2024-03-06T07:27:05.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c0d4800 0x7f733c15aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f733c161580 0x7f733c14ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f733c161960 0x7f733c0c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0b1470 con 0x7f733c161960 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0b2780 con 0x7f733c0d4800 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0b15f0 con 0x7f733c161580 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7340a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c0d4800 0x7f733c15aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f733be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f733c161580 0x7f733c14ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7340a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c0d4800 0x7f733c15aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33562/0 (socket says 172.21.15.178:33562) 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7340a00640 1 -- 172.21.15.178:0/3336994848 learned_addr learned my addr 172.21.15.178:0/3336994848 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f7341400640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f733c161960 0x7f733c0c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f733be00640 1 -- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c0d4800 msgr2=0x7f733c15aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f733be00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c0d4800 0x7f733c15aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f733be00640 1 -- 172.21.15.178:0/3336994848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f733c161960 msgr2=0x7f733c0c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f733be00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f733c161960 0x7f733c0c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.948+0000 7f733be00640 1 -- 172.21.15.178:0/3336994848 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0c9330 con 0x7f733c161580 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f733be00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f733c161580 0x7f733c14ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f732c033000 tx=0x7f732c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c040070 con 0x7f733c161580 2024-03-06T07:27:05.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c030d00 con 0x7f733c161580 2024-03-06T07:27:05.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c035040 con 0x7f733c161580 2024-03-06T07:27:05.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c14b150 con 0x7f733c161580 2024-03-06T07:27:05.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c0cb1c0 con 0x7f733c161580 2024-03-06T07:27:05.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.949+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f732c04b060 con 0x7f733c161580 2024-03-06T07:27:05.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.950+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f732c04d970 con 0x7f733c161580 2024-03-06T07:27:05.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.950+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c04f330 con 0x7f733c161580 2024-03-06T07:27:05.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:05.953+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f732c052020 con 0x7f733c161580 2024-03-06T07:27:05.996 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:05.994+0000 7fb3af817040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:27:06.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.031+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "be5e1794-8b2f-44e0-a366-01f108e9a562", "id": 1} v 0) v1 -- 0x7f733c06d060 con 0x7f733c161580 2024-03-06T07:27:06.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.037+0000 7f7338c00640 1 -- 172.21.15.178:0/3336994848 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be5e1794-8b2f-44e0-a366-01f108e9a562", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f732c04db70 con 0x7f733c161580 2024-03-06T07:27:06.040 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f733c161580 msgr2=0x7f733c14ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f733c161580 0x7f733c14ac10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f732c033000 tx=0x7f732c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 shutdown_connections 2024-03-06T07:27:06.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f733c161960 0x7f733c0c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f733c161580 0x7f733c14ac10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 --2- 172.21.15.178:0/3336994848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f733c0d4800 0x7f733c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 >> 172.21.15.178:0/3336994848 conn(0x7f733c0d3c70 msgr2=0x7f733c0b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 shutdown_connections 2024-03-06T07:27:06.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.039+0000 7f7341e00640 1 -- 172.21.15.178:0/3336994848 wait complete. 2024-03-06T07:27:06.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 16638d16-3365-4de8-bb6c-b3f4dca67a9f 2 2024-03-06T07:27:06.065 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.063+0000 7fce2aa4d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:27:06.081 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:06.079+0000 7fb3af817040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:06.138 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:06.136+0000 7fb3af817040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:27:06.150 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.148+0000 7fce2aa4d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:06.203 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:06.200+0000 7fb3af817040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:27:06.206 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.204+0000 7fce2aa4d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:27:06.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.243+0000 7fe112200640 1 Processor -- start 2024-03-06T07:27:06.246 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.243+0000 7fe112200640 1 -- start start 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe112200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15bb90 0x7fe10c06b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe112200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c15f4a0 con 0x7fe10c15efd0 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe112200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c072050 con 0x7fe10c15bf70 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe112200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c1521d0 con 0x7fe10c15bb90 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe103e00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe10c15bf70 0x7fe10c15e8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:47498/0 (socket says 172.21.15.178:47498) 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe103e00640 1 -- 172.21.15.178:0/976996618 learned_addr learned my addr 172.21.15.178:0/976996618 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-06T07:27:06.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.245+0000 7fe10b400640 1 --2- 172.21.15.178:0/976996618 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15bb90 0x7fe10c06b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.246+0000 7fe10aa00640 1 -- 172.21.15.178:0/976996618 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561710079 0 0) 0x7fe10c072050 con 0x7fe10c15bf70 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.246+0000 7fe10aa00640 1 -- 172.21.15.178:0/976996618 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0003520 con 0x7fe10c15bf70 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.246+0000 7fe10b400640 1 -- 172.21.15.178:0/976996618 >> v1:172.21.15.178:6789/0 conn(0x7fe10c15bf70 legacy=0x7fe10c15e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.246+0000 7fe10b400640 1 -- 172.21.15.178:0/976996618 >> v1:172.21.15.6:6789/0 conn(0x7fe10c15efd0 legacy=0x7fe10c06d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.249 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.246+0000 7fe10b400640 1 -- 172.21.15.178:0/976996618 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c072330 con 0x7fe10c15bb90 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.247+0000 7fe10b400640 1 --2- 172.21.15.178:0/976996618 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15bb90 0x7fe10c06b1f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0097d0 tx=0x7fe0fc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60d6f2a1fc4e800 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.247+0000 7fe10aa00640 1 -- 172.21.15.178:0/976996618 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc031780 con 0x7fe10c15bb90 2024-03-06T07:27:06.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.247+0000 7fe10aa00640 1 -- 172.21.15.178:0/976996618 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc037070 con 0x7fe10c15bb90 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.247+0000 7fe10aa00640 1 -- 172.21.15.178:0/976996618 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc031b40 con 0x7fe10c15bb90 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 -- 172.21.15.178:0/976996618 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15bb90 msgr2=0x7fe10c06b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 --2- 172.21.15.178:0/976996618 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15bb90 0x7fe10c06b1f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0097d0 tx=0x7fe0fc02b820 comp rx=0 tx=0).stop 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 -- 172.21.15.178:0/976996618 shutdown_connections 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 --2- 172.21.15.178:0/976996618 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15bb90 0x7fe10c06b1f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 -- 172.21.15.178:0/976996618 >> 172.21.15.178:0/976996618 conn(0x7fe10c0b1ad0 msgr2=0x7fe10c0b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 -- 172.21.15.178:0/976996618 shutdown_connections 2024-03-06T07:27:06.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.248+0000 7fe112200640 1 -- 172.21.15.178:0/976996618 wait complete. 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.250+0000 7fe112200640 1 Processor -- start 2024-03-06T07:27:06.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.250+0000 7fe112200640 1 -- start start 2024-03-06T07:27:06.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe112200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe10c15bb90 0x7fe10c0b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe112200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe10c15bf70 0x7fe10c0ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.254 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe112200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 0x7fe10c143610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe112200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c15b660 con 0x7fe10c15bb90 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe112200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c15f780 con 0x7fe10c15bf70 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe112200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c072b30 con 0x7fe10c15efd0 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe103e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe10c15bf70 0x7fe10c0ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 0x7fe10c143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 0x7fe10c143610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50500/0 (socket says 172.21.15.178:50500) 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 -- 172.21.15.178:0/1227807634 learned_addr learned my addr 172.21.15.178:0/1227807634 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10b400640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe10c15bb90 0x7fe10c0b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 -- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe10c15bf70 msgr2=0x7fe10c0ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe10c15bf70 0x7fe10c0ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 -- 172.21.15.178:0/1227807634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe10c15bb90 msgr2=0x7fe10c0b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe10c15bb90 0x7fe10c0b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10be00640 1 -- 172.21.15.178:0/1227807634 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c0cf2b0 con 0x7fe10c15efd0 2024-03-06T07:27:06.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.251+0000 7fe10b400640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe10c15bb90 0x7fe10c0b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe10be00640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 0x7fe10c143610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe10400b970 tx=0x7fe10400be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104019070 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10400cb40 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c0b0f20 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104014440 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c09abb0 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe10401c070 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.252+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe104004750 con 0x7fe10c15efd0 2024-03-06T07:27:06.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.253+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c0b1440 con 0x7fe10c15efd0 2024-03-06T07:27:06.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.256+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe10c0b1440 con 0x7fe10c15efd0 2024-03-06T07:27:06.261 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:06.259+0000 7fb3af817040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:27:06.270 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.268+0000 7fce2aa4d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:27:06.327 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.325+0000 7fce2aa4d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:27:06.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.334+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "16638d16-3365-4de8-bb6c-b3f4dca67a9f", "id": 2} v 0) v1 -- 0x7fe10c15eb20 con 0x7fe10c15efd0 2024-03-06T07:27:06.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.341+0000 7fe103400640 1 -- 172.21.15.178:0/1227807634 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "16638d16-3365-4de8-bb6c-b3f4dca67a9f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe10c15eb20 con 0x7fe10c15efd0 2024-03-06T07:27:06.344 INFO:teuthology.orchestra.run.smithi178.stdout:2 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 msgr2=0x7fe10c143610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 0x7fe10c143610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe10400b970 tx=0x7fe10400be30 comp rx=0 tx=0).stop 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 shutdown_connections 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe10c15bf70 0x7fe10c0ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe10c15efd0 0x7fe10c143610 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 --2- 172.21.15.178:0/1227807634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe10c15bb90 0x7fe10c0b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 >> 172.21.15.178:0/1227807634 conn(0x7fe10c0b1ad0 msgr2=0x7fe10c15e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 shutdown_connections 2024-03-06T07:27:06.345 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.342+0000 7fe112200640 1 -- 172.21.15.178:0/1227807634 wait complete. 2024-03-06T07:27:06.362 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new a8a80788-be86-4b6f-8af6-c4b2fba8ac90 3 2024-03-06T07:27:06.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.582+0000 7f3050a00640 1 Processor -- start 2024-03-06T07:27:06.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.582+0000 7f3050a00640 1 -- start start 2024-03-06T07:27:06.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.583+0000 7f3050a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f304c161840 0x7f304c0d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.583+0000 7f3050a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f304c0d4a40 0x7f304c0d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3050a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c0c7a00 con 0x7f304c0c7f00 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3050a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c161c20 con 0x7f304c161840 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3050a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c0d2530 con 0x7f304c0d4a40 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f304c0d4a40 0x7f304c0d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f304c0d4a40 0x7f304c0d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33592/0 (socket says 172.21.15.178:33592) 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 -- 172.21.15.178:0/650449459 learned_addr learned my addr 172.21.15.178:0/650449459 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3049e00640 1 --2- 172.21.15.178:0/650449459 >> v2:172.21.15.6:3300/0 conn(0x7f304c161840 0x7f304c0d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 -- 172.21.15.178:0/650449459 >> v2:172.21.15.6:3300/0 conn(0x7f304c161840 msgr2=0x7f304c0d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 --2- 172.21.15.178:0/650449459 >> v2:172.21.15.6:3300/0 conn(0x7f304c161840 0x7f304c0d07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 -- 172.21.15.178:0/650449459 >> v1:172.21.15.6:6789/0 conn(0x7f304c0c7f00 legacy=0x7f304c0c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.584+0000 7f3043e00640 1 -- 172.21.15.178:0/650449459 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0d2810 con 0x7f304c0d4a40 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.585+0000 7f3043e00640 1 --2- 172.21.15.178:0/650449459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0d4a40 0x7f304c0d0d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f304c08cf80 tx=0x7f303402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f94e5704273046aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3049400640 1 -- 172.21.15.178:0/650449459 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303402e070 con 0x7f304c0d4a40 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3049400640 1 -- 172.21.15.178:0/650449459 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3034037070 con 0x7f304c0d4a40 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3049400640 1 -- 172.21.15.178:0/650449459 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034031a00 con 0x7f304c0d4a40 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 -- 172.21.15.178:0/650449459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0d4a40 msgr2=0x7f304c0d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 --2- 172.21.15.178:0/650449459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0d4a40 0x7f304c0d0d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f304c08cf80 tx=0x7f303402b880 comp rx=0 tx=0).stop 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 -- 172.21.15.178:0/650449459 shutdown_connections 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 --2- 172.21.15.178:0/650449459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0d4a40 0x7f304c0d0d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 --2- 172.21.15.178:0/650449459 >> v2:172.21.15.6:3300/0 conn(0x7f304c161840 0x7f304c0d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 -- 172.21.15.178:0/650449459 >> 172.21.15.178:0/650449459 conn(0x7f304c0d3eb0 msgr2=0x7f304c0d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 -- 172.21.15.178:0/650449459 shutdown_connections 2024-03-06T07:27:06.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.586+0000 7f3050a00640 1 -- 172.21.15.178:0/650449459 wait complete. 2024-03-06T07:27:06.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.588+0000 7f3050a00640 1 Processor -- start 2024-03-06T07:27:06.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.588+0000 7f3050a00640 1 -- start start 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3050a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0c7f00 0x7f304c153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3050a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 0x7f304c15ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3050a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f304c161840 0x7f304c06ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3050a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c0d1750 con 0x7f304c161840 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3050a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c0d2dd0 con 0x7f304c0c7f00 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3050a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c0cfad0 con 0x7f304c0d4a40 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3043e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 0x7f304c15ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3049e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0c7f00 0x7f304c153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3049e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0c7f00 0x7f304c153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33600/0 (socket says 172.21.15.178:33600) 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3049e00640 1 -- 172.21.15.178:0/2753365140 learned_addr learned my addr 172.21.15.178:0/2753365140 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f3043e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 0x7f304c15ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50508/0 (socket says 172.21.15.178:50508) 2024-03-06T07:27:06.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.589+0000 7f304a800640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f304c161840 0x7f304c06ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3043e00640 1 -- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0c7f00 msgr2=0x7f304c153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3043e00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0c7f00 0x7f304c153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3043e00640 1 -- 172.21.15.178:0/2753365140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f304c161840 msgr2=0x7f304c06ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3043e00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f304c161840 0x7f304c06ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3043e00640 1 -- 172.21.15.178:0/2753365140 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0b8a50 con 0x7f304c0d4a40 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3043e00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 0x7f304c15ab90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3034033000 tx=0x7f30340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034032880 con 0x7f304c0d4a40 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3034032a60 con 0x7f304c0d4a40 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3034032e50 con 0x7f304c0d4a40 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c159dd0 con 0x7f304c0d4a40 2024-03-06T07:27:06.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.590+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c15a390 con 0x7f304c0d4a40 2024-03-06T07:27:06.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.591+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f303404c060 con 0x7f304c0d4a40 2024-03-06T07:27:06.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.591+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f303404ec70 con 0x7f304c0d4a40 2024-03-06T07:27:06.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.592+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304c068010 con 0x7f304c0d4a40 2024-03-06T07:27:06.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.594+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3034040070 con 0x7f304c0d4a40 2024-03-06T07:27:06.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.672+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a8a80788-be86-4b6f-8af6-c4b2fba8ac90", "id": 3} v 0) v1 -- 0x7f304c0d03c0 con 0x7f304c0d4a40 2024-03-06T07:27:06.684 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.682+0000 7f3042000640 1 -- 172.21.15.178:0/2753365140 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a8a80788-be86-4b6f-8af6-c4b2fba8ac90", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f303404e7e0 con 0x7f304c0d4a40 2024-03-06T07:27:06.684 INFO:teuthology.orchestra.run.smithi178.stdout:3 2024-03-06T07:27:06.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 msgr2=0x7f304c15ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 0x7f304c15ab90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3034033000 tx=0x7f30340046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 shutdown_connections 2024-03-06T07:27:06.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f304c161840 0x7f304c06ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f304c0d4a40 0x7f304c15ab90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 --2- 172.21.15.178:0/2753365140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f304c0c7f00 0x7f304c153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 >> 172.21.15.178:0/2753365140 conn(0x7f304c0d3eb0 msgr2=0x7f304c0c96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 shutdown_connections 2024-03-06T07:27:06.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.684+0000 7f3050a00640 1 -- 172.21.15.178:0/2753365140 wait complete. 2024-03-06T07:27:06.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 04f8a1e4-842b-4372-b1f3-1ef04081a7bf 4 2024-03-06T07:27:06.742 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:06.740+0000 7fb3af817040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:27:06.801 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.799+0000 7fce2aa4d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:27:06.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.901+0000 7f4ceba00640 1 Processor -- start 2024-03-06T07:27:06.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.901+0000 7f4ceba00640 1 -- start start 2024-03-06T07:27:06.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.902+0000 7f4ceba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ce40d4680 0x7f4ce4161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.902+0000 7f4ceba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0f20 0x7f4ce41618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.905 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.902+0000 7f4ceba00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce40cf7e0 con 0x7f4ce40d0b40 2024-03-06T07:27:06.905 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.902+0000 7f4ceba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce40cf590 con 0x7f4ce40d4680 2024-03-06T07:27:06.905 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.902+0000 7f4ceba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b6dc0 con 0x7f4ce40d0f20 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9000640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f4ce40d0b40 0x7f4ce40b4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:47508/0 (socket says 172.21.15.178:47508) 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9000640 1 -- 172.21.15.178:0/4084029859 learned_addr learned my addr 172.21.15.178:0/4084029859 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9a00640 1 --2- 172.21.15.178:0/4084029859 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0f20 0x7f4ce41618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce3e00640 1 --2- 172.21.15.178:0/4084029859 >> v2:172.21.15.6:3300/0 conn(0x7f4ce40d4680 0x7f4ce4161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce3400640 1 -- 172.21.15.178:0/4084029859 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831787699 0 0) 0x7f4ce40cf7e0 con 0x7f4ce40d0b40 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce3400640 1 -- 172.21.15.178:0/4084029859 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ccc003520 con 0x7f4ce40d0b40 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9a00640 1 -- 172.21.15.178:0/4084029859 >> v2:172.21.15.6:3300/0 conn(0x7f4ce40d4680 msgr2=0x7f4ce4161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9a00640 1 --2- 172.21.15.178:0/4084029859 >> v2:172.21.15.6:3300/0 conn(0x7f4ce40d4680 0x7f4ce4161370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9a00640 1 -- 172.21.15.178:0/4084029859 >> v1:172.21.15.178:6789/0 conn(0x7f4ce40d0b40 legacy=0x7f4ce40b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.906 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.903+0000 7f4ce9a00640 1 -- 172.21.15.178:0/4084029859 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40cfac0 con 0x7f4ce40d0f20 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.904+0000 7f4ce9a00640 1 --2- 172.21.15.178:0/4084029859 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0f20 0x7f4ce41618b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009dc0 tx=0x7f4cdc02bdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4579983c4e0e0334 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.904+0000 7f4ce3400640 1 -- 172.21.15.178:0/4084029859 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc039070 con 0x7f4ce40d0f20 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.904+0000 7f4ce3400640 1 -- 172.21.15.178:0/4084029859 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cdc02cba0 con 0x7f4ce40d0f20 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.904+0000 7f4ce3400640 1 -- 172.21.15.178:0/4084029859 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc034440 con 0x7f4ce40d0f20 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 -- 172.21.15.178:0/4084029859 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0f20 msgr2=0x7f4ce41618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 --2- 172.21.15.178:0/4084029859 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0f20 0x7f4ce41618b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc009dc0 tx=0x7f4cdc02bdf0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 -- 172.21.15.178:0/4084029859 shutdown_connections 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 --2- 172.21.15.178:0/4084029859 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0f20 0x7f4ce41618b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 --2- 172.21.15.178:0/4084029859 >> v2:172.21.15.6:3300/0 conn(0x7f4ce40d4680 0x7f4ce4161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 -- 172.21.15.178:0/4084029859 >> 172.21.15.178:0/4084029859 conn(0x7f4ce409d280 msgr2=0x7f4ce40d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 -- 172.21.15.178:0/4084029859 shutdown_connections 2024-03-06T07:27:06.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.905+0000 7f4ceba00640 1 -- 172.21.15.178:0/4084029859 wait complete. 2024-03-06T07:27:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.907+0000 7f4ceba00640 1 Processor -- start 2024-03-06T07:27:06.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.907+0000 7f4ceba00640 1 -- start start 2024-03-06T07:27:06.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ceba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 0x7f4ce4157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ceba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ce40d0f20 0x7f4ce415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ceba00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ce40d4680 0x7f4ce41f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ceba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b73d0 con 0x7f4ce40d0f20 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ceba00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b7250 con 0x7f4ce40d4680 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ceba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce40b7550 con 0x7f4ce40d0b40 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ce9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 0x7f4ce4157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ce9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 0x7f4ce4157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50522/0 (socket says 172.21.15.178:50522) 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ce9000640 1 -- 172.21.15.178:0/2570570472 learned_addr learned my addr 172.21.15.178:0/2570570472 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ce9a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ce40d4680 0x7f4ce41f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.908+0000 7f4ce3e00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ce40d0f20 0x7f4ce415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9000640 1 -- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ce40d4680 msgr2=0x7f4ce41f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9000640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ce40d4680 0x7f4ce41f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9000640 1 -- 172.21.15.178:0/2570570472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ce40d0f20 msgr2=0x7f4ce415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9000640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ce40d0f20 0x7f4ce415a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9000640 1 -- 172.21.15.178:0/2570570472 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce41596c0 con 0x7f4ce40d0b40 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce3e00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ce40d0f20 0x7f4ce415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9a00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ce40d4680 0x7f4ce41f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce9000640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 0x7f4ce4157df0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c460 tx=0x7f4cd800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8017070 con 0x7f4ce40d0b40 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd8005ce0 con 0x7f4ce40d0b40 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8007290 con 0x7f4ce40d0b40 2024-03-06T07:27:06.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce4158620 con 0x7f4ce40d0b40 2024-03-06T07:27:06.913 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:06.909+0000 7fb3af817040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:27:06.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.909+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce4158be0 con 0x7f4ce40d0b40 2024-03-06T07:27:06.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.910+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4cd801a070 con 0x7f4ce40d0b40 2024-03-06T07:27:06.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.910+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f4cd8012910 con 0x7f4ce40d0b40 2024-03-06T07:27:06.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.910+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce4068720 con 0x7f4ce40d0b40 2024-03-06T07:27:06.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.914+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4cd8021070 con 0x7f4ce40d0b40 2024-03-06T07:27:06.970 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:06.968+0000 7fce2aa4d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:27:06.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:06.994+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "04f8a1e4-842b-4372-b1f3-1ef04081a7bf", "id": 4} v 0) v1 -- 0x7f4ce40d3d50 con 0x7f4ce40d0b40 2024-03-06T07:27:07.002 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.000+0000 7f4ce0c00640 1 -- 172.21.15.178:0/2570570472 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04f8a1e4-842b-4372-b1f3-1ef04081a7bf", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4cd8007660 con 0x7f4ce40d0b40 2024-03-06T07:27:07.002 INFO:teuthology.orchestra.run.smithi178.stdout:4 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 msgr2=0x7f4ce4157df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 0x7f4ce4157df0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800c460 tx=0x7f4cd800c920 comp rx=0 tx=0).stop 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 shutdown_connections 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ce40d4680 0x7f4ce41f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ce40d0f20 0x7f4ce415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 --2- 172.21.15.178:0/2570570472 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ce40d0b40 0x7f4ce4157df0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 >> 172.21.15.178:0/2570570472 conn(0x7f4ce409d280 msgr2=0x7f4ce40d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 shutdown_connections 2024-03-06T07:27:07.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.001+0000 7f4ceba00640 1 -- 172.21.15.178:0/2570570472 wait complete. 2024-03-06T07:27:07.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 6b8b8205-1df1-4537-b6c6-3477a9e720ef 5 2024-03-06T07:27:07.057 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.055+0000 7fb3af817040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:27:07.114 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.107+0000 7fb3af817040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:27:07.118 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.117+0000 7fce2aa4d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:27:07.176 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.173+0000 7fce2aa4d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:27:07.244 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.242+0000 7fb3af817040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:07.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.303+0000 7f0e16600640 1 Processor -- start 2024-03-06T07:27:07.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.303+0000 7f0e16600640 1 -- start start 2024-03-06T07:27:07.307 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.304+0000 7f0e16600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0e100d4800 0x7f0e100b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.304+0000 7f0e16600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100b3150 con 0x7f0e10161580 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.304+0000 7f0e16600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100b2b90 con 0x7f0e10161960 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.304+0000 7f0e16600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e100d26f0 con 0x7f0e100d4800 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e0fe00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f0e10161960 0x7f0e100b3490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:47522/0 (socket says 172.21.15.178:47522) 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e0fe00640 1 -- 172.21.15.178:0/3454135170 learned_addr learned my addr 172.21.15.178:0/3454135170 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e15c00640 1 --2- 172.21.15.178:0/3454135170 >> v2:172.21.15.6:3300/0 conn(0x7f0e100d4800 0x7f0e100b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e0f400640 1 -- 172.21.15.178:0/3454135170 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118229942 0 0) 0x7f0e100b2b90 con 0x7f0e10161960 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e0f400640 1 -- 172.21.15.178:0/3454135170 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df8003520 con 0x7f0e10161960 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e15c00640 1 -- 172.21.15.178:0/3454135170 >> v1:172.21.15.178:6789/0 conn(0x7f0e10161960 legacy=0x7f0e100b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e0f400640 1 -- 172.21.15.178:0/3454135170 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 342526106 0 0) 0x7f0df8003520 con 0x7f0e10161960 2024-03-06T07:27:07.308 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e15c00640 1 -- 172.21.15.178:0/3454135170 >> v1:172.21.15.6:6789/0 conn(0x7f0e10161580 legacy=0x7f0e100b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.309 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.305+0000 7f0e15c00640 1 -- 172.21.15.178:0/3454135170 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100b2e70 con 0x7f0e100d4800 2024-03-06T07:27:07.309 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.306+0000 7fce2aa4d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:07.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.307+0000 7f0e15c00640 1 --2- 172.21.15.178:0/3454135170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100b4bb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e100d2870 tx=0x7f0e0802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538f734c8f78d5ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.307+0000 7f0e0f400640 1 -- 172.21.15.178:0/3454135170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08039070 con 0x7f0e100d4800 2024-03-06T07:27:07.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.307+0000 7f0e0f400640 1 -- 172.21.15.178:0/3454135170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0802c8c0 con 0x7f0e100d4800 2024-03-06T07:27:07.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.307+0000 7f0e0f400640 1 -- 172.21.15.178:0/3454135170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc001670 con 0x7f0e100d4800 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 -- 172.21.15.178:0/3454135170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 msgr2=0x7f0e100b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 --2- 172.21.15.178:0/3454135170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100b4bb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e100d2870 tx=0x7f0e0802bd90 comp rx=0 tx=0).stop 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 -- 172.21.15.178:0/3454135170 shutdown_connections 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 --2- 172.21.15.178:0/3454135170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100b4bb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 -- 172.21.15.178:0/3454135170 >> 172.21.15.178:0/3454135170 conn(0x7f0e100d3c70 msgr2=0x7f0e100d4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 -- 172.21.15.178:0/3454135170 shutdown_connections 2024-03-06T07:27:07.311 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.308+0000 7f0e16600640 1 -- 172.21.15.178:0/3454135170 wait complete. 2024-03-06T07:27:07.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.309+0000 7f0e16600640 1 Processor -- start 2024-03-06T07:27:07.312 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.309+0000 7f0e16600640 1 -- start start 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.310+0000 7f0e16600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.310+0000 7f0e16600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 0x7f0e100cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.310+0000 7f0e16600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 0x7f0e100cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.310+0000 7f0e16600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100b5270 con 0x7f0e100d4800 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.310+0000 7f0e16600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e100b50f0 con 0x7f0e10161960 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.310+0000 7f0e16600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e10168e40 con 0x7f0e10161580 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 0x7f0e100cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.314 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e15c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 0x7f0e100cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e15c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 0x7f0e100cfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33618/0 (socket says 172.21.15.178:33618) 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 0x7f0e100cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50530/0 (socket says 172.21.15.178:50530) 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e15c00640 1 -- 172.21.15.178:0/1597316844 learned_addr learned my addr 172.21.15.178:0/1597316844 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e15200640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 -- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 msgr2=0x7f0e100cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 0x7f0e100cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 -- 172.21.15.178:0/1597316844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 msgr2=0x7f0e100c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100c8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 -- 172.21.15.178:0/1597316844 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100d00e0 con 0x7f0e10161580 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e15c00640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 0x7f0e100cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.311+0000 7f0e0fe00640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 0x7f0e100cf660 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000c00 tx=0x7f0dfc00cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc018070 con 0x7f0e10161580 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc0072d0 con 0x7f0e10161580 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0047e0 con 0x7f0e10161580 2024-03-06T07:27:07.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e10156f00 con 0x7f0e10161580 2024-03-06T07:27:07.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e101574c0 con 0x7f0e10161580 2024-03-06T07:27:07.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0dfc01b070 con 0x7f0e10161580 2024-03-06T07:27:07.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.312+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f0dfc00a610 con 0x7f0e10161580 2024-03-06T07:27:07.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.313+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1004f330 con 0x7f0e10161580 2024-03-06T07:27:07.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.315+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0dfc022080 con 0x7f0e10161580 2024-03-06T07:27:07.363 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.361+0000 7fb3af817040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:27:07.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.394+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6b8b8205-1df1-4537-b6c6-3477a9e720ef", "id": 5} v 0) v1 -- 0x7f0e10159430 con 0x7f0e10161580 2024-03-06T07:27:07.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.401+0000 7f0e0cc00640 1 -- 172.21.15.178:0/1597316844 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b8b8205-1df1-4537-b6c6-3477a9e720ef", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0dfc00ac60 con 0x7f0e10161580 2024-03-06T07:27:07.404 INFO:teuthology.orchestra.run.smithi178.stdout:5 2024-03-06T07:27:07.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.402+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 msgr2=0x7f0e100cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.402+0000 7f0e16600640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 0x7f0e100cf660 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000c00 tx=0x7f0dfc00cbe0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 shutdown_connections 2024-03-06T07:27:07.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e10161960 0x7f0e100cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e10161580 0x7f0e100cf660 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 --2- 172.21.15.178:0/1597316844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e100d4800 0x7f0e100c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 >> 172.21.15.178:0/1597316844 conn(0x7f0e100d3c70 msgr2=0x7f0e100d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 shutdown_connections 2024-03-06T07:27:07.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.403+0000 7f0e16600640 1 -- 172.21.15.178:0/1597316844 wait complete. 2024-03-06T07:27:07.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new 4348e701-3acb-4a7c-b0a9-8951f623e015 6 2024-03-06T07:27:07.426 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.424+0000 7fce2aa4d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:27:07.550 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.548+0000 7fb3af817040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:07.602 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.600+0000 7fb3af817040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:07.611 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.609+0000 7fce2aa4d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.610+0000 7ff966400640 1 Processor -- start 2024-03-06T07:27:07.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.610+0000 7ff966400640 1 -- start start 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.611+0000 7ff966400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 0x7ff960161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.611+0000 7ff966400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff9600d0f20 0x7ff9601618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.611+0000 7ff966400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9600cf7e0 con 0x7ff9600d0b40 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.611+0000 7ff966400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9600cf590 con 0x7ff9600d0f20 2024-03-06T07:27:07.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.611+0000 7ff966400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600b6dc0 con 0x7ff9600d4680 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 0x7ff960161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 0x7ff960161370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50532/0 (socket says 172.21.15.178:50532) 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 -- 172.21.15.178:0/2483544510 learned_addr learned my addr 172.21.15.178:0/2483544510 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff95fe00640 1 --2- 172.21.15.178:0/2483544510 >> v2:172.21.15.6:3300/0 conn(0x7ff9600d0f20 0x7ff9601618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 -- 172.21.15.178:0/2483544510 >> v2:172.21.15.6:3300/0 conn(0x7ff9600d0f20 msgr2=0x7ff9601618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 --2- 172.21.15.178:0/2483544510 >> v2:172.21.15.6:3300/0 conn(0x7ff9600d0f20 0x7ff9601618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 -- 172.21.15.178:0/2483544510 >> v1:172.21.15.6:6789/0 conn(0x7ff9600d0b40 legacy=0x7ff9600b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.615 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.612+0000 7ff957e00640 1 -- 172.21.15.178:0/2483544510 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600cfac0 con 0x7ff9600d4680 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.613+0000 7ff957e00640 1 --2- 172.21.15.178:0/2483544510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 0x7ff960161370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff948009870 tx=0x7ff94802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe2aabc94e421ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.613+0000 7ff95ea00640 1 -- 172.21.15.178:0/2483544510 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948035020 con 0x7ff9600d4680 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.613+0000 7ff95ea00640 1 -- 172.21.15.178:0/2483544510 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff948031640 con 0x7ff9600d4680 2024-03-06T07:27:07.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.613+0000 7ff95ea00640 1 -- 172.21.15.178:0/2483544510 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94802d040 con 0x7ff9600d4680 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 -- 172.21.15.178:0/2483544510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 msgr2=0x7ff960161370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 --2- 172.21.15.178:0/2483544510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 0x7ff960161370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff948009870 tx=0x7ff94802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 -- 172.21.15.178:0/2483544510 shutdown_connections 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 --2- 172.21.15.178:0/2483544510 >> v2:172.21.15.6:3300/0 conn(0x7ff9600d0f20 0x7ff9601618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 --2- 172.21.15.178:0/2483544510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d4680 0x7ff960161370 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 -- 172.21.15.178:0/2483544510 >> 172.21.15.178:0/2483544510 conn(0x7ff96009d280 msgr2=0x7ff9600d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 -- 172.21.15.178:0/2483544510 shutdown_connections 2024-03-06T07:27:07.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.614+0000 7ff966400640 1 -- 172.21.15.178:0/2483544510 wait complete. 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.616+0000 7ff966400640 1 Processor -- start 2024-03-06T07:27:07.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.616+0000 7ff966400640 1 -- start start 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff966400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 0x7ff960157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff966400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600d0f20 0x7ff9600cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff966400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600d4680 0x7ff9600c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff966400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600b73d0 con 0x7ff9600d4680 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff966400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600b7250 con 0x7ff9600d0f20 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff966400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600b7550 con 0x7ff9600d0b40 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 0x7ff960157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 0x7ff960157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.178:50540/0 (socket says 172.21.15.178:50540) 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 -- 172.21.15.178:0/1832853278 learned_addr learned my addr 172.21.15.178:0/1832853278 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff957e00640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600d0f20 0x7ff9600cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95fe00640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600d4680 0x7ff9600c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 -- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600d0f20 msgr2=0x7ff9600cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600d0f20 0x7ff9600cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 -- 172.21.15.178:0/1832853278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600d4680 msgr2=0x7ff9600c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.617+0000 7ff95f400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600d4680 0x7ff9600c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff95f400640 1 -- 172.21.15.178:0/1832853278 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600cc310 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff95f400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 0x7ff960157390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff95000c480 tx=0x7ff95000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9500052d0 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950005470 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950012870 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9600cba70 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9600cc030 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff950012a10 con 0x7ff9600d0b40 2024-03-06T07:27:07.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.618+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7ff950003be0 con 0x7ff9600d0b40 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.619+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff960068720 con 0x7ff9600d0b40 2024-03-06T07:27:07.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.621+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff950012d00 con 0x7ff9600d0b40 2024-03-06T07:27:07.663 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.661+0000 7fce2aa4d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:27:07.702 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.699+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4348e701-3acb-4a7c-b0a9-8951f623e015", "id": 6} v 0) v1 -- 0x7ff960158ca0 con 0x7ff9600d0b40 2024-03-06T07:27:07.709 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.707+0000 7ff957400640 1 -- 172.21.15.178:0/1832853278 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4348e701-3acb-4a7c-b0a9-8951f623e015", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff9500035f0 con 0x7ff9600d0b40 2024-03-06T07:27:07.709 INFO:teuthology.orchestra.run.smithi178.stdout:6 2024-03-06T07:27:07.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 msgr2=0x7ff960157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 0x7ff960157390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff95000c480 tx=0x7ff95000c940 comp rx=0 tx=0).stop 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 shutdown_connections 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600d0f20 0x7ff9600cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600d4680 0x7ff9600c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 --2- 172.21.15.178:0/1832853278 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9600d0b40 0x7ff960157390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 >> 172.21.15.178:0/1832853278 conn(0x7ff96009d280 msgr2=0x7ff9600b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 shutdown_connections 2024-03-06T07:27:07.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.708+0000 7ff966400640 1 -- 172.21.15.178:0/1832853278 wait complete. 2024-03-06T07:27:07.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo ceph --cluster ceph osd new bee5fa45-a2e4-49e5-b380-e32387dbb269 7 2024-03-06T07:27:07.791 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:07.789+0000 7fb3af817040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:27:07.846 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:07.844+0000 7fce2aa4d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:27:07.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.916+0000 7f140e200640 1 Processor -- start 2024-03-06T07:27:07.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.916+0000 7f140e200640 1 -- start start 2024-03-06T07:27:07.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.917+0000 7f140e200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14080c8f10 0x7f14080cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.917+0000 7f140e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14080c7a00 con 0x7f14080d0f00 2024-03-06T07:27:07.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.917+0000 7f140e200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1408161c20 con 0x7f1408161840 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.917+0000 7f140e200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14080d2530 con 0x7f14080c8f10 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1407400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f1408161840 0x7f14080d07f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.178:47532/0 (socket says 172.21.15.178:47532) 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1407400640 1 -- 172.21.15.178:0/2215230341 learned_addr learned my addr 172.21.15.178:0/2215230341 (peer_addr_for_me v1:172.21.15.178:0/0) 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1407e00640 1 --2- 172.21.15.178:0/2215230341 >> v2:172.21.15.6:3300/0 conn(0x7f14080c8f10 0x7f14080cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1406000640 1 -- 172.21.15.178:0/2215230341 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682991345 0 0) 0x7f1408161c20 con 0x7f1408161840 2024-03-06T07:27:07.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1406000640 1 -- 172.21.15.178:0/2215230341 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec003520 con 0x7f1408161840 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1407e00640 1 -- 172.21.15.178:0/2215230341 >> v1:172.21.15.178:6789/0 conn(0x7f1408161840 legacy=0x7f14080d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.918+0000 7f1407e00640 1 -- 172.21.15.178:0/2215230341 >> v1:172.21.15.6:6789/0 conn(0x7f14080d0f00 legacy=0x7f14080ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.919+0000 7f1407e00640 1 -- 172.21.15.178:0/2215230341 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14080d2810 con 0x7f14080c8f10 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.920+0000 7f1407e00640 1 --2- 172.21.15.178:0/2215230341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14080c8f10 0x7f14080cf1d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1400009d90 tx=0x7f140002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e41e44b76e6b28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.920+0000 7f1406000640 1 -- 172.21.15.178:0/2215230341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400039070 con 0x7f14080c8f10 2024-03-06T07:27:07.922 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.920+0000 7f1406000640 1 -- 172.21.15.178:0/2215230341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140002cc60 con 0x7f14080c8f10 2024-03-06T07:27:07.923 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.920+0000 7f1406000640 1 -- 172.21.15.178:0/2215230341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400034640 con 0x7f14080c8f10 2024-03-06T07:27:07.923 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 -- 172.21.15.178:0/2215230341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14080c8f10 msgr2=0x7f14080cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.923 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 --2- 172.21.15.178:0/2215230341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14080c8f10 0x7f14080cf1d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1400009d90 tx=0x7f140002beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 -- 172.21.15.178:0/2215230341 shutdown_connections 2024-03-06T07:27:07.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 --2- 172.21.15.178:0/2215230341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14080c8f10 0x7f14080cf1d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 -- 172.21.15.178:0/2215230341 >> 172.21.15.178:0/2215230341 conn(0x7f14080d3eb0 msgr2=0x7f14080d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 -- 172.21.15.178:0/2215230341 shutdown_connections 2024-03-06T07:27:07.924 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.921+0000 7f140e200640 1 -- 172.21.15.178:0/2215230341 wait complete. 2024-03-06T07:27:07.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.923+0000 7f140e200640 1 Processor -- start 2024-03-06T07:27:07.925 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.923+0000 7f140e200640 1 -- start start 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f140e200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14080c8f10 0x7f140815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f140e200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 0x7f140809df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f140e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1408161840 0x7f140809e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f140e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080d1750 con 0x7f1408161840 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f140e200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080d2dd0 con 0x7f14080d0f00 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f140e200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14080d4a40 con 0x7f14080c8f10 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f1406a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 0x7f140809df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.928 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f1406a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 0x7f140809df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:33634/0 (socket says 172.21.15.178:33634) 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f1406a00640 1 -- 172.21.15.178:0/2605839320 learned_addr learned my addr 172.21.15.178:0/2605839320 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f1407400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14080c8f10 0x7f140815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.924+0000 7f1407e00640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1408161840 0x7f140809e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1406a00640 1 -- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14080c8f10 msgr2=0x7f140815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1406a00640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14080c8f10 0x7f140815c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1406a00640 1 -- 172.21.15.178:0/2605839320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1408161840 msgr2=0x7f140809e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1406a00640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1408161840 0x7f140809e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1406a00640 1 -- 172.21.15.178:0/2605839320 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14080a0af0 con 0x7f14080d0f00 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1407400640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14080c8f10 0x7f140815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f1406a00640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 0x7f140809df10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f13f400c400 tx=0x7f13f400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f40052d0 con 0x7f14080d0f00 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f4005490 con 0x7f14080d0f00 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f4007260 con 0x7f14080d0f00 2024-03-06T07:27:07.929 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14080a0dd0 con 0x7f14080d0f00 2024-03-06T07:27:07.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.925+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14080a1390 con 0x7f14080d0f00 2024-03-06T07:27:07.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.926+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f13f4007400 con 0x7f14080d0f00 2024-03-06T07:27:07.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.926+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f13f4003a00 con 0x7f14080d0f00 2024-03-06T07:27:07.930 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.927+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1408068010 con 0x7f14080d0f00 2024-03-06T07:27:07.932 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:07.930+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f13f40035f0 con 0x7f14080d0f00 2024-03-06T07:27:08.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.021+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bee5fa45-a2e4-49e5-b380-e32387dbb269", "id": 7} v 0) v1 -- 0x7f14080b9b00 con 0x7f14080d0f00 2024-03-06T07:27:08.031 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.029+0000 7f13ff400640 1 -- 172.21.15.178:0/2605839320 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bee5fa45-a2e4-49e5-b380-e32387dbb269", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f13f4003d20 con 0x7f14080d0f00 2024-03-06T07:27:08.032 INFO:teuthology.orchestra.run.smithi178.stdout:7 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.030+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 msgr2=0x7f140809df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.030+0000 7f140e200640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 0x7f140809df10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f13f400c400 tx=0x7f13f400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 shutdown_connections 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14080d0f00 0x7f140809df10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1408161840 0x7f140809e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 --2- 172.21.15.178:0/2605839320 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14080c8f10 0x7f140815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 >> 172.21.15.178:0/2605839320 conn(0x7f14080d3eb0 msgr2=0x7f14080d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 shutdown_connections 2024-03-06T07:27:08.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-03-06T07:27:08.031+0000 7f140e200640 1 -- 172.21.15.178:0/2605839320 wait complete. 2024-03-06T07:27:08.050 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:27:08.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:27:08.058 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:27:08.058 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:27:08.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:27:08.065 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:27:08.065 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:27:08.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:27:08.070 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:27:08.070 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:27:08.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:27:08.076 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:27:08.076 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:27:08.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:27:08.084 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:27:08.084 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:27:08.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:27:08.090 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:27:08.090 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:27:08.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:27:08.097 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:27:08.098 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:27:08.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:27:08.102 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:27:08.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:08.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.333+0000 7f0882e00640 1 Processor -- start 2024-03-06T07:27:08.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.333+0000 7f0882e00640 1 -- start start 2024-03-06T07:27:08.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f0882e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 0x7f087c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f0882e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f087c0ae0a0 0x7f087c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f0882e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087c0ce9a0 con 0x7f087c06e240 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f0882e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f087c0730d0 con 0x7f087c0ae0a0 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f0882e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cec80 con 0x7f087c161670 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f087be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 0x7f087c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.334+0000 7f0880e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f087c0ae0a0 0x7f087c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f087b400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f087c06e240 0x7f087c06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:45492/0 (socket says 172.21.15.6:45492) 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f087be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 0x7f087c070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:49420/0 (socket says 172.21.15.6:49420) 2024-03-06T07:27:08.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f087b400640 1 -- 172.21.15.6:0/1139578606 learned_addr learned my addr 172.21.15.6:0/1139578606 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f0880e00640 1 -- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 msgr2=0x7f087c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f0880e00640 1 --2- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 0x7f087c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f0880e00640 1 -- 172.21.15.6:0/1139578606 >> v1:172.21.15.178:6789/0 conn(0x7f087c06e240 legacy=0x7f087c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f0880e00640 1 -- 172.21.15.6:0/1139578606 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c0cef60 con 0x7f087c0ae0a0 2024-03-06T07:27:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.335+0000 7f087be00640 1 --2- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 0x7f087c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:08.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.337+0000 7f0880e00640 1 --2- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c0ae0a0 0x7f087c0711b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0874009d90 tx=0x7f087402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1228f540b5573a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.337+0000 7f087aa00640 1 -- 172.21.15.6:0/1139578606 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874039070 con 0x7f087c0ae0a0 2024-03-06T07:27:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.337+0000 7f087aa00640 1 -- 172.21.15.6:0/1139578606 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087402cc50 con 0x7f087c0ae0a0 2024-03-06T07:27:08.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.337+0000 7f087aa00640 1 -- 172.21.15.6:0/1139578606 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874034440 con 0x7f087c0ae0a0 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 -- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c0ae0a0 msgr2=0x7f087c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 --2- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c0ae0a0 0x7f087c0711b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0874009d90 tx=0x7f087402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 -- 172.21.15.6:0/1139578606 shutdown_connections 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 --2- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c0ae0a0 0x7f087c0711b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 --2- 172.21.15.6:0/1139578606 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c161670 0x7f087c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 -- 172.21.15.6:0/1139578606 >> 172.21.15.6:0/1139578606 conn(0x7f087c06afb0 msgr2=0x7f087c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 -- 172.21.15.6:0/1139578606 shutdown_connections 2024-03-06T07:27:08.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.338+0000 7f0882e00640 1 -- 172.21.15.6:0/1139578606 wait complete. 2024-03-06T07:27:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.340+0000 7f0882e00640 1 Processor -- start 2024-03-06T07:27:08.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.340+0000 7f0882e00640 1 -- start start 2024-03-06T07:27:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0882e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 0x7f087c149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0882e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c0ae0a0 0x7f087c0b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0882e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c161670 0x7f087c0b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0882e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0ce600 con 0x7f087c06e240 2024-03-06T07:27:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0882e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0cf520 con 0x7f087c161670 2024-03-06T07:27:08.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0882e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087c0ce780 con 0x7f087c0ae0a0 2024-03-06T07:27:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 0x7f087c149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 0x7f087c149560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39970/0 (socket says 172.21.15.6:39970) 2024-03-06T07:27:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 -- 172.21.15.6:0/996345596 learned_addr learned my addr 172.21.15.6:0/996345596 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f0880e00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c161670 0x7f087c0b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087b400640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c0ae0a0 0x7f087c0b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 -- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c0ae0a0 msgr2=0x7f087c0b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c0ae0a0 0x7f087c0b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 -- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c161670 msgr2=0x7f087c0b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c161670 0x7f087c0b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.341+0000 7f087be00640 1 -- 172.21.15.6:0/996345596 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f087c09ab40 con 0x7f087c06e240 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f087be00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 0x7f087c149560 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e960 tx=0x7f086c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c017070 con 0x7f087c06e240 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c00ca00 con 0x7f087c06e240 2024-03-06T07:27:08.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c09ae20 con 0x7f087c06e240 2024-03-06T07:27:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00cdd0 con 0x7f087c06e240 2024-03-06T07:27:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c09b900 con 0x7f087c06e240 2024-03-06T07:27:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f086c01a070 con 0x7f087c06e240 2024-03-06T07:27:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.342+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f086c01cb20 con 0x7f087c06e240 2024-03-06T07:27:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.343+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c069510 con 0x7f087c06e240 2024-03-06T07:27:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.346+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f086c01c450 con 0x7f087c06e240 2024-03-06T07:27:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.411+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f087c04f3a0 con 0x7f087c06e240 2024-03-06T07:27:08.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.412+0000 7f0873e00640 1 -- 172.21.15.6:0/996345596 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f086c01c630 con 0x7f087c06e240 2024-03-06T07:27:08.414 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:08.414 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":10,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:08.024203+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:08.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 msgr2=0x7f087c149560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 0x7f087c149560 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e960 tx=0x7f086c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 shutdown_connections 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f087c161670 0x7f087c0b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f087c0ae0a0 0x7f087c0b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 --2- 172.21.15.6:0/996345596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f087c06e240 0x7f087c149560 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 >> 172.21.15.6:0/996345596 conn(0x7f087c06afb0 msgr2=0x7f087c0ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 shutdown_connections 2024-03-06T07:27:08.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:08.413+0000 7f0882e00640 1 -- 172.21.15.6:0/996345596 wait complete. 2024-03-06T07:27:08.435 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:27:08.436 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:27:08.771 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-06T07:27:08.768+0000 7f535c49b640 -1 Falling back to public interface 2024-03-06T07:27:08.771 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-06T07:27:08.768+0000 7f1c6fc11640 -1 Falling back to public interface 2024-03-06T07:27:08.774 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-06T07:27:08.772+0000 7f25a520f640 -1 Falling back to public interface 2024-03-06T07:27:08.774 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-06T07:27:08.772+0000 7fc60588b640 -1 Falling back to public interface 2024-03-06T07:27:08.784 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-06T07:27:08.782+0000 7f29aa448640 -1 Falling back to public interface 2024-03-06T07:27:08.791 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-06T07:27:08.790+0000 7f068d622640 -1 Falling back to public interface 2024-03-06T07:27:08.800 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-06T07:27:08.794+0000 7fafe9c9b640 -1 Falling back to public interface 2024-03-06T07:27:08.806 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-06T07:27:08.804+0000 7fec69215640 -1 Falling back to public interface 2024-03-06T07:27:08.914 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:08.912+0000 7fb3af817040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:27:08.969 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:08.967+0000 7fce2aa4d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:27:09.226 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:09.223+0000 7fb3af817040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:27:09.241 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:09.284 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:09.282+0000 7fb3af817040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:27:09.299 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:09.298+0000 7fce2aa4d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:27:09.364 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:09.362+0000 7fce2aa4d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:27:09.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.404+0000 7f218bc00640 1 Processor -- start 2024-03-06T07:27:09.406 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.404+0000 7f218bc00640 1 -- start start 2024-03-06T07:27:09.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.405+0000 7f218bc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f218415e2b0 0x7f21840d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.405+0000 7f218bc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f21840cf570 0x7f21840d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.405+0000 7f218bc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21840d0db0 0x7f21840d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.405+0000 7f218bc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218415e690 con 0x7f218415e2b0 2024-03-06T07:27:09.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.405+0000 7f218bc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218406bb30 con 0x7f21840cf570 2024-03-06T07:27:09.408 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.405+0000 7f218bc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21840cfdf0 con 0x7f21840d0db0 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f218415e2b0 0x7f21840d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f218415e2b0 0x7f21840d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39972/0 (socket says 172.21.15.6:39972) 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 -- 172.21.15.6:0/2569012049 learned_addr learned my addr 172.21.15.6:0/2569012049 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2183e00640 1 --2- 172.21.15.6:0/2569012049 >> v2:172.21.15.178:3300/0 conn(0x7f21840cf570 0x7f21840d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189c00640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21840d0db0 0x7f21840d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 -- 172.21.15.6:0/2569012049 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21840d0db0 msgr2=0x7f21840d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21840d0db0 0x7f21840d1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 -- 172.21.15.6:0/2569012049 >> v2:172.21.15.178:3300/0 conn(0x7f21840cf570 msgr2=0x7f21840d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 --2- 172.21.15.6:0/2569012049 >> v2:172.21.15.178:3300/0 conn(0x7f21840cf570 0x7f21840d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189200640 1 -- 172.21.15.6:0/2569012049 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218406bdd0 con 0x7f218415e2b0 2024-03-06T07:27:09.409 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.406+0000 7f2189c00640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21840d0db0 0x7f21840d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:09.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.407+0000 7f2189200640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f218415e2b0 0x7f21840d15e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f218408c5b0 tx=0x7f217802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe02b2814e23b49f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.410 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f2183400640 1 -- 172.21.15.6:0/2569012049 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217802e070 con 0x7f218415e2b0 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f2183400640 1 -- 172.21.15.6:0/2569012049 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2178037070 con 0x7f218415e2b0 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f2183400640 1 -- 172.21.15.6:0/2569012049 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178031a00 con 0x7f218415e2b0 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 -- 172.21.15.6:0/2569012049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f218415e2b0 msgr2=0x7f21840d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f218415e2b0 0x7f21840d15e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f218408c5b0 tx=0x7f217802b880 comp rx=0 tx=0).stop 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 -- 172.21.15.6:0/2569012049 shutdown_connections 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21840d0db0 0x7f21840d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 --2- 172.21.15.6:0/2569012049 >> v2:172.21.15.178:3300/0 conn(0x7f21840cf570 0x7f21840d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 --2- 172.21.15.6:0/2569012049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f218415e2b0 0x7f21840d15e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 -- 172.21.15.6:0/2569012049 >> 172.21.15.6:0/2569012049 conn(0x7f21840d3fa0 msgr2=0x7f21840d43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 -- 172.21.15.6:0/2569012049 shutdown_connections 2024-03-06T07:27:09.411 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.408+0000 7f218bc00640 1 -- 172.21.15.6:0/2569012049 wait complete. 2024-03-06T07:27:09.412 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.410+0000 7f218bc00640 1 Processor -- start 2024-03-06T07:27:09.412 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.410+0000 7f218bc00640 1 -- start start 2024-03-06T07:27:09.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f218bc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21840cf570 0x7f2184153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.414 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f218bc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 0x7f2184160c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f218bc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f218415e2b0 0x7f21840c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f218bc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21840b0bc0 con 0x7f21840d0db0 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f218bc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218406c390 con 0x7f21840cf570 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f218bc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21840cf9f0 con 0x7f218415e2b0 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f2183e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 0x7f2184160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f2183e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 0x7f2184160c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39976/0 (socket says 172.21.15.6:39976) 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f2183e00640 1 -- 172.21.15.6:0/1446303741 learned_addr learned my addr 172.21.15.6:0/1446303741 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f2189200640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21840cf570 0x7f2184153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.411+0000 7f2189c00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f218415e2b0 0x7f21840c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2183e00640 1 -- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f218415e2b0 msgr2=0x7f21840c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2183e00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f218415e2b0 0x7f21840c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2183e00640 1 -- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21840cf570 msgr2=0x7f2184153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2183e00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21840cf570 0x7f2184153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.415 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2183e00640 1 -- 172.21.15.6:0/1446303741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840c91c0 con 0x7f21840d0db0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2183e00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 0x7f2184160c50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f217000e890 tx=0x7f217000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2170017070 con 0x7f21840d0db0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217000c940 con 0x7f21840d0db0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217000cd10 con 0x7f21840d0db0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.412+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2184161190 con 0x7f21840d0db0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.413+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218415fe90 con 0x7f21840d0db0 2024-03-06T07:27:09.416 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.413+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f217001a070 con 0x7f21840d0db0 2024-03-06T07:27:09.417 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:09.413+0000 7fb3af817040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:27:09.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.413+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2170013b40 con 0x7f21840d0db0 2024-03-06T07:27:09.417 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.414+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218404f330 con 0x7f21840d0db0 2024-03-06T07:27:09.418 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.416+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f21840c91c0 con 0x7f21840d0db0 2024-03-06T07:27:09.424 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.422+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mgrmap(e 2) v1 ==== 44915+0+0 (secure 0 0 0) 0x7f2170013470 con 0x7f21840d0db0 2024-03-06T07:27:09.491 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.489+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f218406d930 con 0x7f21840d0db0 2024-03-06T07:27:09.492 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.490+0000 7f2180c00640 1 -- 172.21.15.6:0/1446303741 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f2170012e20 con 0x7f21840d0db0 2024-03-06T07:27:09.492 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-06T07:27:09.492 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":10,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:08.024203+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:09.493 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 msgr2=0x7f2184160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 0x7f2184160c50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f217000e890 tx=0x7f217000ed50 comp rx=0 tx=0).stop 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 shutdown_connections 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f218415e2b0 0x7f21840c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21840d0db0 0x7f2184160c50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 --2- 172.21.15.6:0/1446303741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21840cf570 0x7f2184153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.491+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 >> 172.21.15.6:0/1446303741 conn(0x7f21840d3fa0 msgr2=0x7f21840d4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.492+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 shutdown_connections 2024-03-06T07:27:09.494 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:09.492+0000 7f218bc00640 1 -- 172.21.15.6:0/1446303741 wait complete. 2024-03-06T07:27:09.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:27:09.509 INFO:tasks.ceph.mgr.y.smithi006.stderr:2024-03-06T07:27:09.508+0000 7fce2aa4d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:27:10.444 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:10.442+0000 7fb34fe00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:27:10.445 INFO:tasks.ceph.mgr.x.smithi178.stderr:2024-03-06T07:27:10.443+0000 7fb34fe00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:27:12.324 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-06T07:27:12.322+0000 7f535c49b640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:27:12.351 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-06T07:27:12.349+0000 7f1c6fc11640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:27:12.353 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-06T07:27:12.352+0000 7f068d622640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:27:12.384 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-06T07:27:12.382+0000 7fec69215640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:27:12.569 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-06T07:27:12.567+0000 7f25a520f640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:27:12.598 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-06T07:27:12.595+0000 7fc60588b640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:27:12.600 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-06T07:27:12.599+0000 7f29aa448640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:27:12.615 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-06T07:27:12.613+0000 7fafe9c9b640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:27:14.742 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-06T07:27:14.741+0000 7fec63000640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:27:14.751 INFO:tasks.ceph.osd.6.smithi178.stderr:2024-03-06T07:27:14.750+0000 7fec59a00640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:14.757 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-06T07:27:14.756+0000 7f1c69a00640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:27:14.762 INFO:tasks.ceph.osd.5.smithi178.stderr:2024-03-06T07:27:14.761+0000 7f1c60400640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:15.117 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-06T07:27:15.115+0000 7f0687600640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:27:15.137 INFO:tasks.ceph.osd.3.smithi006.stderr:2024-03-06T07:27:15.135+0000 7f067e000640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:15.218 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-06T07:27:15.216+0000 7f5358800640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:27:15.240 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-03-06T07:27:15.238+0000 7f534d400640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:15.662 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-06T07:27:15.660+0000 7f259f600640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:27:15.674 INFO:tasks.ceph.osd.7.smithi178.stderr:2024-03-06T07:27:15.673+0000 7f2596000640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:15.718 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-06T07:27:15.717+0000 7fc601c00640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:27:15.724 INFO:tasks.ceph.osd.4.smithi178.stderr:2024-03-06T07:27:15.723+0000 7fc5f6800640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:15.946 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-06T07:27:15.945+0000 7f29a4800640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:27:15.959 INFO:tasks.ceph.osd.1.smithi006.stderr:2024-03-06T07:27:15.957+0000 7f299b200640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:15.989 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-06T07:27:15.987+0000 7fafe4200640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:27:15.997 INFO:tasks.ceph.osd.2.smithi006.stderr:2024-03-06T07:27:15.995+0000 7fafdac00640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:27:16.312 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:16.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.502+0000 7fde24e00640 1 Processor -- start 2024-03-06T07:27:16.504 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.502+0000 7fde24e00640 1 -- start start 2024-03-06T07:27:16.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.504+0000 7fde24e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fde200b22e0 0x7fde200b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.504+0000 7fde24e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2015ca90 con 0x7fde200cf620 2024-03-06T07:27:16.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.504+0000 7fde24e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde20145bb0 con 0x7fde201457d0 2024-03-06T07:27:16.506 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.504+0000 7fde24e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde200b33b0 con 0x7fde200b22e0 2024-03-06T07:27:16.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.504+0000 7fde1d400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde200cf620 0x7fde200b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36382/0 (socket says 172.21.15.6:36382) 2024-03-06T07:27:16.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1d400640 1 -- 172.21.15.6:0/183972177 learned_addr learned my addr 172.21.15.6:0/183972177 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:16.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1e800640 1 --2- 172.21.15.6:0/183972177 >> v2:172.21.15.178:3300/0 conn(0x7fde200b22e0 0x7fde200b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1ca00640 1 -- 172.21.15.6:0/183972177 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371790293 0 0) 0x7fde2015ca90 con 0x7fde200cf620 2024-03-06T07:27:16.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1ca00640 1 -- 172.21.15.6:0/183972177 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde04003520 con 0x7fde200cf620 2024-03-06T07:27:16.507 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1e800640 1 -- 172.21.15.6:0/183972177 >> v1:172.21.15.178:6789/0 conn(0x7fde201457d0 legacy=0x7fde2015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1ca00640 1 -- 172.21.15.6:0/183972177 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1543151232 0 0) 0x7fde04003520 con 0x7fde200cf620 2024-03-06T07:27:16.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1e800640 1 -- 172.21.15.6:0/183972177 >> v1:172.21.15.6:6789/0 conn(0x7fde200cf620 legacy=0x7fde200b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.505+0000 7fde1e800640 1 -- 172.21.15.6:0/183972177 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde200b3690 con 0x7fde200b22e0 2024-03-06T07:27:16.508 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.507+0000 7fde1e800640 1 --2- 172.21.15.6:0/183972177 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200b22e0 0x7fde200b4450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fde180097a0 tx=0x7fde1802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f66f9a52583ea244 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.507+0000 7fde1ca00640 1 -- 172.21.15.6:0/183972177 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde18038070 con 0x7fde200b22e0 2024-03-06T07:27:16.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.507+0000 7fde1ca00640 1 -- 172.21.15.6:0/183972177 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde18031500 con 0x7fde200b22e0 2024-03-06T07:27:16.509 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.507+0000 7fde1ca00640 1 -- 172.21.15.6:0/183972177 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080019f0 con 0x7fde200b22e0 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 -- 172.21.15.6:0/183972177 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200b22e0 msgr2=0x7fde200b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 --2- 172.21.15.6:0/183972177 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200b22e0 0x7fde200b4450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fde180097a0 tx=0x7fde1802b820 comp rx=0 tx=0).stop 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 -- 172.21.15.6:0/183972177 shutdown_connections 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 --2- 172.21.15.6:0/183972177 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200b22e0 0x7fde200b4450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 -- 172.21.15.6:0/183972177 >> 172.21.15.6:0/183972177 conn(0x7fde2006af70 msgr2=0x7fde2006b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 -- 172.21.15.6:0/183972177 shutdown_connections 2024-03-06T07:27:16.510 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.508+0000 7fde24e00640 1 -- 172.21.15.6:0/183972177 wait complete. 2024-03-06T07:27:16.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.510+0000 7fde24e00640 1 Processor -- start 2024-03-06T07:27:16.512 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.510+0000 7fde24e00640 1 -- start start 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde24e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 0x7fde20135160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde24e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200cf620 0x7fde201f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde24e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde201457d0 0x7fde201f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde24e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde200b3f30 con 0x7fde200b22e0 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde24e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde200b3c50 con 0x7fde200cf620 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde24e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde200731f0 con 0x7fde201457d0 2024-03-06T07:27:16.514 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde1de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 0x7fde20135160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde1de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 0x7fde20135160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45976/0 (socket says 172.21.15.6:45976) 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde1de00640 1 -- 172.21.15.6:0/2741389110 learned_addr learned my addr 172.21.15.6:0/2741389110 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde1e800640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde201457d0 0x7fde201f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.511+0000 7fde1d400640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200cf620 0x7fde201f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1de00640 1 -- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde201457d0 msgr2=0x7fde201f8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1de00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde201457d0 0x7fde201f8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1de00640 1 -- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200cf620 msgr2=0x7fde201f7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1de00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200cf620 0x7fde201f7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1de00640 1 -- 172.21.15.6:0/2741389110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20137030 con 0x7fde200b22e0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1d400640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200cf620 0x7fde201f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde1de00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 0x7fde20135160 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c400 tx=0x7fde1000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10017070 con 0x7fde200b22e0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde100052d0 con 0x7fde200b22e0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10003830 con 0x7fde200b22e0 2024-03-06T07:27:16.515 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde20137310 con 0x7fde200b22e0 2024-03-06T07:27:16.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.512+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde20144900 con 0x7fde200b22e0 2024-03-06T07:27:16.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.513+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde200b2f40 con 0x7fde200b22e0 2024-03-06T07:27:16.516 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.514+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fde1001a070 con 0x7fde200b22e0 2024-03-06T07:27:16.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.515+0000 7fde16000640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddfc06b440 0x7fddfc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.517 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.515+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3911+0+0 (secure 0 0 0) 0x7fde10088e20 con 0x7fde200b22e0 2024-03-06T07:27:16.518 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.516+0000 7fde1d400640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddfc06b440 0x7fddfc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.519 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.517+0000 7fde1d400640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddfc06b440 0x7fddfc06d900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fde08001560 tx=0x7fde08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.523 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.521+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde200b2f40 con 0x7fde200b22e0 2024-03-06T07:27:16.722 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.720+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde2004f330 con 0x7fde200b22e0 2024-03-06T07:27:16.724 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.722+0000 7fde16000640 1 -- 172.21.15.6:0/2741389110 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9913 (secure 0 0 0) 0x7fde100583e0 con 0x7fde200b22e0 2024-03-06T07:27:16.724 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-06T07:27:16.725 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":14,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:15.743461+0000","last_up_change":"2024-03-06T07:27:15.743461+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddfc06b440 msgr2=0x7fddfc06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddfc06b440 0x7fddfc06d900 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fde08001560 tx=0x7fde08009040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 msgr2=0x7fde20135160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 0x7fde20135160 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c400 tx=0x7fde1000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 shutdown_connections 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddfc06b440 0x7fddfc06d900 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde200cf620 0x7fde201f7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde201457d0 0x7fde201f8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 --2- 172.21.15.6:0/2741389110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde200b22e0 0x7fde20135160 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 >> 172.21.15.6:0/2741389110 conn(0x7fde2006af70 msgr2=0x7fde200afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.728 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 shutdown_connections 2024-03-06T07:27:16.729 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:16.725+0000 7fde24e00640 1 -- 172.21.15.6:0/2741389110 wait complete. 2024-03-06T07:27:16.741 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-03-06T07:27:20.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:20.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:23.548 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:23.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.739+0000 7fd8eda00640 1 Processor -- start 2024-03-06T07:27:23.742 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.740+0000 7fd8eda00640 1 -- start start 2024-03-06T07:27:23.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.741+0000 7fd8eda00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8e815e2c0 0x7fd8e815f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.741+0000 7fd8eda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8e8071e50 0x7fd8e80b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.741+0000 7fd8eda00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e815e6a0 con 0x7fd8e80b2110 2024-03-06T07:27:23.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8eda00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e80b2580 con 0x7fd8e8071e50 2024-03-06T07:27:23.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8eda00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e815f850 con 0x7fd8e815e2c0 2024-03-06T07:27:23.745 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8e8071e50 0x7fd8e80b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd8e8071e50 0x7fd8e80b1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37326/0 (socket says 172.21.15.6:37326) 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 -- 172.21.15.6:0/628351372 learned_addr learned my addr 172.21.15.6:0/628351372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6a00640 1 --2- 172.21.15.6:0/628351372 >> v2:172.21.15.178:3300/0 conn(0x7fd8e815e2c0 0x7fd8e815f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 -- 172.21.15.6:0/628351372 >> v2:172.21.15.178:3300/0 conn(0x7fd8e815e2c0 msgr2=0x7fd8e815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 --2- 172.21.15.6:0/628351372 >> v2:172.21.15.178:3300/0 conn(0x7fd8e815e2c0 0x7fd8e815f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 -- 172.21.15.6:0/628351372 >> v1:172.21.15.178:6789/0 conn(0x7fd8e80b2110 legacy=0x7fd8e80ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6000640 1 -- 172.21.15.6:0/628351372 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8071390 con 0x7fd8e8071e50 2024-03-06T07:27:23.746 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.742+0000 7fd8e6a00640 1 --2- 172.21.15.6:0/628351372 >> v2:172.21.15.178:3300/0 conn(0x7fd8e815e2c0 0x7fd8e815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.744+0000 7fd8e6000640 1 --2- 172.21.15.6:0/628351372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e8071e50 0x7fd8e80b1bd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8e806f1f0 tx=0x7fd8d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899c80145cc9dc20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.744+0000 7fd8e5600640 1 -- 172.21.15.6:0/628351372 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d402e070 con 0x7fd8e8071e50 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.744+0000 7fd8e5600640 1 -- 172.21.15.6:0/628351372 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d4037070 con 0x7fd8e8071e50 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.744+0000 7fd8e5600640 1 -- 172.21.15.6:0/628351372 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4031a00 con 0x7fd8e8071e50 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.744+0000 7fd8eda00640 1 -- 172.21.15.6:0/628351372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e8071e50 msgr2=0x7fd8e80b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.744+0000 7fd8eda00640 1 --2- 172.21.15.6:0/628351372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e8071e50 0x7fd8e80b1bd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd8e806f1f0 tx=0x7fd8d402b880 comp rx=0 tx=0).stop 2024-03-06T07:27:23.749 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.745+0000 7fd8eda00640 1 -- 172.21.15.6:0/628351372 shutdown_connections 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.745+0000 7fd8eda00640 1 --2- 172.21.15.6:0/628351372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e8071e50 0x7fd8e80b1bd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.745+0000 7fd8eda00640 1 --2- 172.21.15.6:0/628351372 >> v2:172.21.15.178:3300/0 conn(0x7fd8e815e2c0 0x7fd8e815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.745+0000 7fd8eda00640 1 -- 172.21.15.6:0/628351372 >> 172.21.15.6:0/628351372 conn(0x7fd8e806af70 msgr2=0x7fd8e806b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.745+0000 7fd8eda00640 1 -- 172.21.15.6:0/628351372 shutdown_connections 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.745+0000 7fd8eda00640 1 -- 172.21.15.6:0/628351372 wait complete. 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.746+0000 7fd8eda00640 1 Processor -- start 2024-03-06T07:27:23.750 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.747+0000 7fd8eda00640 1 -- start start 2024-03-06T07:27:23.753 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8eda00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8e8071e50 0x7fd8e81548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8eda00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8e80b2110 0x7fd8e81e4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8eda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 0x7fd8e814a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8eda00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8071650 con 0x7fd8e815e2c0 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8eda00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e80ccf70 con 0x7fd8e8071e50 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8eda00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8071930 con 0x7fd8e80b2110 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 0x7fd8e814a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 0x7fd8e814a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37336/0 (socket says 172.21.15.6:37336) 2024-03-06T07:27:23.754 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 -- 172.21.15.6:0/1273624659 learned_addr learned my addr 172.21.15.6:0/1273624659 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e6a00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8e8071e50 0x7fd8e81548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e6000640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8e80b2110 0x7fd8e81e4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 -- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8e80b2110 msgr2=0x7fd8e81e4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8e80b2110 0x7fd8e81e4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 -- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8e8071e50 msgr2=0x7fd8e81548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8e8071e50 0x7fd8e81548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.748+0000 7fd8e7400640 1 -- 172.21.15.6:0/1273624659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e81e4ca0 con 0x7fd8e815e2c0 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.749+0000 7fd8e7400640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 0x7fd8e814a7b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c780 tx=0x7fd8e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.755 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.749+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0017070 con 0x7fd8e815e2c0 2024-03-06T07:27:23.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.749+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8e0003da0 con 0x7fd8e815e2c0 2024-03-06T07:27:23.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.749+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00052b0 con 0x7fd8e815e2c0 2024-03-06T07:27:23.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.749+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e8159d80 con 0x7fd8e815e2c0 2024-03-06T07:27:23.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.749+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e80ae390 con 0x7fd8e815e2c0 2024-03-06T07:27:23.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.750+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e804f330 con 0x7fd8e815e2c0 2024-03-06T07:27:23.756 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.751+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8e0005450 con 0x7fd8e815e2c0 2024-03-06T07:27:23.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.753+0000 7fd8dea00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8c806b4d0 0x7fd8c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:23.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.753+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd8e0089410 con 0x7fd8e815e2c0 2024-03-06T07:27:23.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.753+0000 7fd8e6a00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8c806b4d0 0x7fd8c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:23.757 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.754+0000 7fd8e6a00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8c806b4d0 0x7fd8c806d990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8e815b6b0 tx=0x7fd8d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:23.760 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.758+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8e0058670 con 0x7fd8e815e2c0 2024-03-06T07:27:23.959 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.957+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8e80b2770 con 0x7fd8e815e2c0 2024-03-06T07:27:23.960 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.959+0000 7fd8dea00640 1 -- 172.21.15.6:0/1273624659 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12812 (secure 0 0 0) 0x7fd8e0057f70 con 0x7fd8e815e2c0 2024-03-06T07:27:23.961 INFO:teuthology.misc.health.smithi006.stdout: 2024-03-06T07:27:23.961 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":18,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:19.967437+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8c806b4d0 msgr2=0x7fd8c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8c806b4d0 0x7fd8c806d990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd8e815b6b0 tx=0x7fd8d8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 msgr2=0x7fd8e814a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 0x7fd8e814a7b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000c780 tx=0x7fd8e000cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 shutdown_connections 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8c806b4d0 0x7fd8c806d990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8e815e2c0 0x7fd8e814a7b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8e80b2110 0x7fd8e81e4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 --2- 172.21.15.6:0/1273624659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8e8071e50 0x7fd8e81548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.961+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 >> 172.21.15.6:0/1273624659 conn(0x7fd8e806af70 msgr2=0x7fd8e80ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.962+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 shutdown_connections 2024-03-06T07:27:23.964 INFO:teuthology.misc.health.smithi006.stderr:2024-03-06T07:27:23.962+0000 7fd8eda00640 1 -- 172.21.15.6:0/1273624659 wait complete. 2024-03-06T07:27:23.978 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:27:23.978 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:27:23.978 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:23.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:23.983 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:23.983 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:23.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:23.987 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:23.987 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:23.988 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:23.995 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:23.995 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:23.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:24.002 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:24.002 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:27:24.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:24.135 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:27:24.136 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:27:24.156 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:27:24.158 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:27:24.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.184+0000 7f2fb9c00640 1 Processor -- start 2024-03-06T07:27:24.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.184+0000 7f2fb9c00640 1 -- start start 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb9c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2fb415e0b0 0x7f2fb40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb9c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d0e50 0x7f2fb40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb9c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40b3710 con 0x7f2fb40d4a10 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb9c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb40b09e0 con 0x7f2fb415e0b0 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb9c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40b39f0 con 0x7f2fb40d0e50 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2fb415e0b0 0x7f2fb40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2fb40d4a10 0x7f2fb40d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50862/0 (socket says 172.21.15.6:50862) 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb3400640 1 -- 172.21.15.6:0/658707059 learned_addr learned my addr 172.21.15.6:0/658707059 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.185+0000 7f2fb9200640 1 --2- 172.21.15.6:0/658707059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d0e50 0x7f2fb40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.186+0000 7f2fb2a00640 1 -- 172.21.15.6:0/658707059 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703844023 0 0) 0x7f2fb40b3710 con 0x7f2fb40d4a10 2024-03-06T07:27:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.186+0000 7f2fb2a00640 1 -- 172.21.15.6:0/658707059 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f98003520 con 0x7f2fb40d4a10 2024-03-06T07:27:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.186+0000 7f2fb9200640 1 -- 172.21.15.6:0/658707059 >> v2:172.21.15.6:3300/0 conn(0x7f2fb415e0b0 msgr2=0x7f2fb40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.186+0000 7f2fb9200640 1 --2- 172.21.15.6:0/658707059 >> v2:172.21.15.6:3300/0 conn(0x7f2fb415e0b0 0x7f2fb40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.186+0000 7f2fb9200640 1 -- 172.21.15.6:0/658707059 >> v1:172.21.15.6:6789/0 conn(0x7f2fb40d4a10 legacy=0x7f2fb40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.186+0000 7f2fb9200640 1 -- 172.21.15.6:0/658707059 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb40b3cd0 con 0x7f2fb40d0e50 2024-03-06T07:27:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.187+0000 7f2fb9200640 1 --2- 172.21.15.6:0/658707059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d0e50 0x7f2fb40b44d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009d90 tx=0x7f2fac02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f82f21c3de7ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.187+0000 7f2fb2a00640 1 -- 172.21.15.6:0/658707059 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac039070 con 0x7f2fb40d0e50 2024-03-06T07:27:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.187+0000 7f2fb2a00640 1 -- 172.21.15.6:0/658707059 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac02cc60 con 0x7f2fb40d0e50 2024-03-06T07:27:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.187+0000 7f2fb2a00640 1 -- 172.21.15.6:0/658707059 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac034660 con 0x7f2fb40d0e50 2024-03-06T07:27:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 -- 172.21.15.6:0/658707059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d0e50 msgr2=0x7f2fb40b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/658707059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d0e50 0x7f2fb40b44d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009d90 tx=0x7f2fac02beb0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 -- 172.21.15.6:0/658707059 shutdown_connections 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/658707059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d0e50 0x7f2fb40b44d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/658707059 >> v2:172.21.15.6:3300/0 conn(0x7f2fb415e0b0 0x7f2fb40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 -- 172.21.15.6:0/658707059 >> 172.21.15.6:0/658707059 conn(0x7f2fb40d3d30 msgr2=0x7f2fb40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 -- 172.21.15.6:0/658707059 shutdown_connections 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.188+0000 7f2fb9c00640 1 -- 172.21.15.6:0/658707059 wait complete. 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.189+0000 7f2fb9c00640 1 Processor -- start 2024-03-06T07:27:24.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 -- start start 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 0x7f2fb4152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d4a10 0x7f2fb40b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fb415e0b0 0x7f2fb40b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40cfc60 con 0x7f2fb40d0e50 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb415e490 con 0x7f2fb415e0b0 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.190+0000 7f2fb9c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40cf850 con 0x7f2fb40d4a10 2024-03-06T07:27:24.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 0x7f2fb4152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 0x7f2fb4152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37380/0 (socket says 172.21.15.6:37380) 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 -- 172.21.15.6:0/555340373 learned_addr learned my addr 172.21.15.6:0/555340373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3400640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d4a10 0x7f2fb40b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb9200640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fb415e0b0 0x7f2fb40b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 -- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d4a10 msgr2=0x7f2fb40b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d4a10 0x7f2fb40b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 -- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fb415e0b0 msgr2=0x7f2fb40b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fb415e0b0 0x7f2fb40b0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3e00640 1 -- 172.21.15.6:0/555340373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb41f3dd0 con 0x7f2fb40d0e50 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.191+0000 7f2fb3400640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d4a10 0x7f2fb40b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.192+0000 7f2fb3e00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 0x7f2fb4152320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400e870 tx=0x7f2fa400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.192+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4017070 con 0x7f2fb40d0e50 2024-03-06T07:27:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.192+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400c940 con 0x7f2fb40d0e50 2024-03-06T07:27:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.192+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400cd30 con 0x7f2fb40d0e50 2024-03-06T07:27:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.192+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb41f40b0 con 0x7f2fb40d0e50 2024-03-06T07:27:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.192+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb41f4670 con 0x7f2fb40d0e50 2024-03-06T07:27:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.193+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb404f330 con 0x7f2fb40d0e50 2024-03-06T07:27:24.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.194+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fa401a070 con 0x7f2fb40d0e50 2024-03-06T07:27:24.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.195+0000 7f2fabe00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f9406b4d0 0x7f2f9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.195+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2fa408a120 con 0x7f2fb40d0e50 2024-03-06T07:27:24.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.196+0000 7f2fb3400640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f9406b4d0 0x7f2f9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.196+0000 7f2fb3400640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f9406b4d0 0x7f2f9406d990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2fb415f780 tx=0x7f2fa0000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.200+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fa4058c00 con 0x7f2fb40d0e50 2024-03-06T07:27:24.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.396+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fb40b64e0 con 0x7f2fb40d0e50 2024-03-06T07:27:24.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.398+0000 7f2fabe00640 1 -- 172.21.15.6:0/555340373 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12812 (secure 0 0 0) 0x7f2fa405cca0 con 0x7f2fb40d0e50 2024-03-06T07:27:24.400 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:24.400 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:19.967437+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f9406b4d0 msgr2=0x7f2f9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f9406b4d0 0x7f2f9406d990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2fb415f780 tx=0x7f2fa0000bc0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 msgr2=0x7f2fb4152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 0x7f2fb4152320 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400e870 tx=0x7f2fa400ed30 comp rx=0 tx=0).stop 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 shutdown_connections 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f9406b4d0 0x7f2f9406d990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fb415e0b0 0x7f2fb40b0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fb40d4a10 0x7f2fb40b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 --2- 172.21.15.6:0/555340373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fb40d0e50 0x7f2fb4152320 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.400+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 >> 172.21.15.6:0/555340373 conn(0x7f2fb40d3d30 msgr2=0x7f2fb40cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.401+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 shutdown_connections 2024-03-06T07:27:24.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.401+0000 7f2fb9c00640 1 -- 172.21.15.6:0/555340373 wait complete. 2024-03-06T07:27:24.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:24.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:24.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.630+0000 7fcc57e00640 1 Processor -- start 2024-03-06T07:27:24.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.630+0000 7fcc57e00640 1 -- start start 2024-03-06T07:27:24.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 0x7fcc580d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc57e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc580ceca0 con 0x7fcc5815dee0 2024-03-06T07:27:24.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc57e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc580cef80 con 0x7fcc580cfa60 2024-03-06T07:27:24.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc57e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580cf5d0 con 0x7fcc580d4860 2024-03-06T07:27:24.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc56a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcc5815dee0 0x7fcc580d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50866/0 (socket says 172.21.15.6:50866) 2024-03-06T07:27:24.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc56a00640 1 -- 172.21.15.6:0/3986099348 learned_addr learned my addr 172.21.15.6:0/3986099348 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.632+0000 7fcc56000640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 0x7fcc580d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126050917 0 0) 0x7fcc580ceca0 con 0x7fcc5815dee0 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c003520 con 0x7fcc5815dee0 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272736563 0 0) 0x7fcc3c003520 con 0x7fcc5815dee0 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 msgr2=0x7fcc580d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 0x7fcc580d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 >> v1:172.21.15.178:6789/0 conn(0x7fcc580cfa60 legacy=0x7fcc580b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc580cf260 con 0x7fcc5815dee0 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fcc48001670 con 0x7fcc5815dee0 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 >> v1:172.21.15.6:6789/0 conn(0x7fcc5815dee0 legacy=0x7fcc580d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc3c0039e0 0x7fcc3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 0x7fcc3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc3c004d40 0x7fcc3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580cf5d0 con 0x7fcc3c004d40 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c00baa0 con 0x7fcc3c0039e0 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c00bca0 con 0x7fcc580d4860 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.633+0000 7fcc57400640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc3c004d40 0x7fcc3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc56a00640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc3c0039e0 0x7fcc3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc56000640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 0x7fcc3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc57400640 1 -- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 msgr2=0x7fcc3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc57400640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc580d4860 0x7fcc3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc57400640 1 -- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc3c0039e0 msgr2=0x7fcc3c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc57400640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc3c0039e0 0x7fcc3c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.634+0000 7fcc57400640 1 -- 172.21.15.6:0/3986099348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c003520 con 0x7fcc3c004d40 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.635+0000 7fcc57400640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc3c004d40 0x7fcc3c0051c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcc50001e20 tx=0x7fcc3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d65f7244cf93fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.635+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50001a70 con 0x7fcc3c004d40 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.635+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50003b70 con 0x7fcc3c004d40 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.635+0000 7fcc55600640 1 -- 172.21.15.6:0/3986099348 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500050d0 con 0x7fcc3c004d40 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57e00640 1 -- 172.21.15.6:0/3986099348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc3c004d40 msgr2=0x7fcc3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3986099348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc3c004d40 0x7fcc3c0051c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcc50001e20 tx=0x7fcc3c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57400640 1 -- 172.21.15.6:0/3986099348 reap_dead start 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57e00640 1 -- 172.21.15.6:0/3986099348 shutdown_connections 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57e00640 1 -- 172.21.15.6:0/3986099348 >> 172.21.15.6:0/3986099348 conn(0x7fcc580d3cd0 msgr2=0x7fcc580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57e00640 1 -- 172.21.15.6:0/3986099348 shutdown_connections 2024-03-06T07:27:24.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.636+0000 7fcc57e00640 1 -- 172.21.15.6:0/3986099348 wait complete. 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.638+0000 7fcc57e00640 1 Processor -- start 2024-03-06T07:27:24.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.638+0000 7fcc57e00640 1 -- start start 2024-03-06T07:27:24.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc3c004d40 0x7fcc58153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc580b9840 0x7fcc580b7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 0x7fcc58158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580b3c20 con 0x7fcc58157fa0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5815e2c0 con 0x7fcc580b9840 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc580b3da0 con 0x7fcc3c004d40 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 0x7fcc58158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 0x7fcc58158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37386/0 (socket says 172.21.15.6:37386) 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc57400640 1 -- 172.21.15.6:0/3893113073 learned_addr learned my addr 172.21.15.6:0/3893113073 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc56000640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc580b9840 0x7fcc580b7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.639+0000 7fcc56a00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc3c004d40 0x7fcc58153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57400640 1 -- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc3c004d40 msgr2=0x7fcc58153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57400640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc3c004d40 0x7fcc58153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57400640 1 -- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc580b9840 msgr2=0x7fcc580b7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57400640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc580b9840 0x7fcc580b7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57400640 1 -- 172.21.15.6:0/3893113073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc58153dc0 con 0x7fcc58157fa0 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57400640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 0x7fcc58158450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc5002c040 tx=0x7fcc500014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50007db0 con 0x7fcc58157fa0 2024-03-06T07:27:24.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50003970 con 0x7fcc58157fa0 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50006d90 con 0x7fcc58157fa0 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc580b7ea0 con 0x7fcc58157fa0 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.640+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc580c90c0 con 0x7fcc58157fa0 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.642+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5804f330 con 0x7fcc58157fa0 2024-03-06T07:27:24.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.642+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc50007030 con 0x7fcc58157fa0 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.643+0000 7fcc4ea00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc3806b4d0 0x7fcc3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.643+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcc500aaff0 con 0x7fcc58157fa0 2024-03-06T07:27:24.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.644+0000 7fcc56a00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc3806b4d0 0x7fcc3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.645+0000 7fcc56a00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc3806b4d0 0x7fcc3806d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc48001050 tx=0x7fcc48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.648+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc50033020 con 0x7fcc58157fa0 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.849+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc580b4430 con 0x7fcc58157fa0 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.850+0000 7fcc4ea00640 1 -- 172.21.15.6:0/3893113073 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc50003b40 con 0x7fcc58157fa0 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc3806b4d0 msgr2=0x7fcc3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc3806b4d0 0x7fcc3806d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcc48001050 tx=0x7fcc48009040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 msgr2=0x7fcc58158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 0x7fcc58158450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc5002c040 tx=0x7fcc500014a0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 shutdown_connections 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc3806b4d0 0x7fcc3806d990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc580b9840 0x7fcc580b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc58157fa0 0x7fcc58158450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 --2- 172.21.15.6:0/3893113073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc3c004d40 0x7fcc58153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 >> 172.21.15.6:0/3893113073 conn(0x7fcc580d3cd0 msgr2=0x7fcc5806b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 shutdown_connections 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:24.853+0000 7fcc57e00640 1 -- 172.21.15.6:0/3893113073 wait complete. 2024-03-06T07:27:24.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:25.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.079+0000 7fc04b200640 1 Processor -- start 2024-03-06T07:27:25.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.079+0000 7fc04b200640 1 -- start start 2024-03-06T07:27:25.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.081+0000 7fc04b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc04c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.081+0000 7fc04b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c15ed10 con 0x7fc04c1552b0 2024-03-06T07:27:25.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.081+0000 7fc04b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c15eff0 con 0x7fc04c0cd060 2024-03-06T07:27:25.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.081+0000 7fc04b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c15f2d0 con 0x7fc04c15e810 2024-03-06T07:27:25.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.081+0000 7fc049e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc04c1552b0 0x7fc04c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50874/0 (socket says 172.21.15.6:50874) 2024-03-06T07:27:25.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.081+0000 7fc049e00640 1 -- 172.21.15.6:0/1843570642 learned_addr learned my addr 172.21.15.6:0/1843570642 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:25.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc04a800640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc04c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130825282 0 0) 0x7fc04c15ed10 con 0x7fc04c1552b0 2024-03-06T07:27:25.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030003520 con 0x7fc04c1552b0 2024-03-06T07:27:25.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3543037940 0 0) 0x7fc030003520 con 0x7fc04c1552b0 2024-03-06T07:27:25.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 msgr2=0x7fc04c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc04c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 >> v1:172.21.15.178:6789/0 conn(0x7fc04c0cd060 legacy=0x7fc04c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c15f5b0 con 0x7fc04c1552b0 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc03c001690 con 0x7fc04c1552b0 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 >> v1:172.21.15.6:6789/0 conn(0x7fc04c1552b0 legacy=0x7fc04c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0300039e0 0x7fc030003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc030004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.082+0000 7fc048a00640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc0300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c15f2d0 con 0x7fc030004d40 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03000baf0 con 0x7fc0300039e0 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03000bcf0 con 0x7fc04c15e810 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc04a800640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc0300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc049e00640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0300039e0 0x7fc030003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc049400640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc030004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc04a800640 1 -- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 msgr2=0x7fc030004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc04a800640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc030004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc04a800640 1 -- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0300039e0 msgr2=0x7fc030003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc04a800640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0300039e0 0x7fc030003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc04a800640 1 -- 172.21.15.6:0/1843570642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030003520 con 0x7fc030004d40 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc049e00640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0300039e0 0x7fc030003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.083+0000 7fc049400640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c15e810 0x7fc030004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.084+0000 7fc04a800640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc0300051c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc04c15ec40 tx=0x7fc030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94b3cfe2ff8fa63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.084+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044039070 con 0x7fc030004d40 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.084+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04402cb90 con 0x7fc030004d40 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.084+0000 7fc048a00640 1 -- 172.21.15.6:0/1843570642 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044009c10 con 0x7fc030004d40 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04b200640 1 -- 172.21.15.6:0/1843570642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 msgr2=0x7fc0300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04b200640 1 --2- 172.21.15.6:0/1843570642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc0300051c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc04c15ec40 tx=0x7fc030005830 comp rx=0 tx=0).stop 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04a800640 1 -- 172.21.15.6:0/1843570642 reap_dead start 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04b200640 1 -- 172.21.15.6:0/1843570642 shutdown_connections 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04b200640 1 -- 172.21.15.6:0/1843570642 >> 172.21.15.6:0/1843570642 conn(0x7fc04c06b3d0 msgr2=0x7fc04c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04b200640 1 -- 172.21.15.6:0/1843570642 shutdown_connections 2024-03-06T07:27:25.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.085+0000 7fc04b200640 1 -- 172.21.15.6:0/1843570642 wait complete. 2024-03-06T07:27:25.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.087+0000 7fc04b200640 1 Processor -- start 2024-03-06T07:27:25.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.087+0000 7fc04b200640 1 -- start start 2024-03-06T07:27:25.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc04b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc04c153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc04b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c0af720 0x7fc04c1532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc04b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc04c14b210 0x7fc04c14b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc04b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c06dd30 con 0x7fc030004d40 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc04b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0b3760 con 0x7fc04c14b210 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc04b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c06deb0 con 0x7fc04c0af720 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc049e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc04c153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc049e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc04c153e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37396/0 (socket says 172.21.15.6:37396) 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.088+0000 7fc049e00640 1 -- 172.21.15.6:0/915909536 learned_addr learned my addr 172.21.15.6:0/915909536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049400640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c0af720 0x7fc04c1532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc04a800640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc04c14b210 0x7fc04c14b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049e00640 1 -- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c0af720 msgr2=0x7fc04c1532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049e00640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c0af720 0x7fc04c1532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049e00640 1 -- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc04c14b210 msgr2=0x7fc04c14b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049e00640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc04c14b210 0x7fc04c14b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049e00640 1 -- 172.21.15.6:0/915909536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c14bc00 con 0x7fc030004d40 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc049e00640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc04c153e60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00fcf0 tx=0x7fc03c00d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c015070 con 0x7fc030004d40 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c0070d0 con 0x7fc030004d40 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c007480 con 0x7fc030004d40 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c0afbe0 con 0x7fc030004d40 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.089+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c0b01a0 con 0x7fc030004d40 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.091+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c04f330 con 0x7fc030004d40 2024-03-06T07:27:25.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.091+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc03c00c070 con 0x7fc030004d40 2024-03-06T07:27:25.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.092+0000 7fc042000640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc02c06b500 0x7fc02c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.092+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc03c08c900 con 0x7fc030004d40 2024-03-06T07:27:25.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.093+0000 7fc049400640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc02c06b500 0x7fc02c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.094+0000 7fc049400640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc02c06b500 0x7fc02c06d9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc04c06b850 tx=0x7fc038009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.097+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc03c05ba30 con 0x7fc030004d40 2024-03-06T07:27:25.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.329+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc04c0cf4a0 con 0x7fc030004d40 2024-03-06T07:27:25.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.330+0000 7fc042000640 1 -- 172.21.15.6:0/915909536 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7fc03c05b330 con 0x7fc030004d40 2024-03-06T07:27:25.332 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:25.332 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-06T07:27:25.332 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 3 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc02c06b500 msgr2=0x7fc02c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc02c06b500 0x7fc02c06d9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc04c06b850 tx=0x7fc038009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 msgr2=0x7fc04c153e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc04c153e60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00fcf0 tx=0x7fc03c00d6b0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 shutdown_connections 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc02c06b500 0x7fc02c06d9c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc04c14b210 0x7fc04c14b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc04c0af720 0x7fc04c1532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 --2- 172.21.15.6:0/915909536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc030004d40 0x7fc04c153e60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 >> 172.21.15.6:0/915909536 conn(0x7fc04c06b3d0 msgr2=0x7fc04c0ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 shutdown_connections 2024-03-06T07:27:25.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.333+0000 7fc04b200640 1 -- 172.21.15.6:0/915909536 wait complete. 2024-03-06T07:27:25.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:25.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.550+0000 7fc2d5800640 1 Processor -- start 2024-03-06T07:27:25.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.550+0000 7fc2d5800640 1 -- start start 2024-03-06T07:27:25.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.551+0000 7fc2d5800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc2d00cd060 0x7fc2d0161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.551+0000 7fc2d5800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2d0070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.551+0000 7fc2d5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d01627f0 con 0x7fc2d015efc0 2024-03-06T07:27:25.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.551+0000 7fc2d5800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d015f3a0 con 0x7fc2d00cd060 2024-03-06T07:27:25.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.551+0000 7fc2d5800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d006d190 con 0x7fc2d0162000 2024-03-06T07:27:25.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.552+0000 7fc2cfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc2d015efc0 0x7fc2d00cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50888/0 (socket says 172.21.15.6:50888) 2024-03-06T07:27:25.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.552+0000 7fc2cfe00640 1 -- 172.21.15.6:0/3633763425 learned_addr learned my addr 172.21.15.6:0/3633763425 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:25.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.552+0000 7fc2cf400640 1 --2- 172.21.15.6:0/3633763425 >> v2:172.21.15.178:3300/0 conn(0x7fc2d00cd060 0x7fc2d0161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.552+0000 7fc2d4e00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2d0070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.552+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116442895 0 0) 0x7fc2d01627f0 con 0x7fc2d015efc0 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.552+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b4003520 con 0x7fc2d015efc0 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173018933 0 0) 0x7fc2b4003520 con 0x7fc2d015efc0 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 msgr2=0x7fc2d0070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2d0070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 >> v2:172.21.15.178:3300/0 conn(0x7fc2d00cd060 msgr2=0x7fc2d0161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 --2- 172.21.15.6:0/3633763425 >> v2:172.21.15.178:3300/0 conn(0x7fc2d00cd060 0x7fc2d0161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d006c860 con 0x7fc2d015efc0 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc2c00019f0 con 0x7fc2d015efc0 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 >> v1:172.21.15.6:6789/0 conn(0x7fc2d015efc0 legacy=0x7fc2d00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2b40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4003f80 0x7fc2b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b4004d40 0x7fc2b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d015f3a0 con 0x7fc2b4004d40 2024-03-06T07:27:25.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d006d190 con 0x7fc2b4003f80 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b400b9c0 con 0x7fc2d0162000 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2d4e00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b4004d40 0x7fc2b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2b40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.553+0000 7fc2cf400640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4003f80 0x7fc2b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2d4e00640 1 -- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 msgr2=0x7fc2b40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2d4e00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2b40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2d4e00640 1 -- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4003f80 msgr2=0x7fc2b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2d4e00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4003f80 0x7fc2b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2d4e00640 1 -- 172.21.15.6:0/3633763425 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b4003520 con 0x7fc2b4004d40 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0162000 0x7fc2b40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:25.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.554+0000 7fc2cf400640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4003f80 0x7fc2b4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.555+0000 7fc2d4e00640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b4004d40 0x7fc2b40051c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009af0 tx=0x7fc2b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1286bf0e4eb7fa59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.555+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8039070 con 0x7fc2b4004d40 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.555+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c802c6b0 con 0x7fc2b4004d40 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.555+0000 7fc2cea00640 1 -- 172.21.15.6:0/3633763425 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8009690 con 0x7fc2b4004d40 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d5800640 1 -- 172.21.15.6:0/3633763425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b4004d40 msgr2=0x7fc2b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d5800640 1 --2- 172.21.15.6:0/3633763425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2b4004d40 0x7fc2b40051c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8009af0 tx=0x7fc2b4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d4e00640 1 -- 172.21.15.6:0/3633763425 reap_dead start 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d5800640 1 -- 172.21.15.6:0/3633763425 shutdown_connections 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d5800640 1 -- 172.21.15.6:0/3633763425 >> 172.21.15.6:0/3633763425 conn(0x7fc2d006b3d0 msgr2=0x7fc2d006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d5800640 1 -- 172.21.15.6:0/3633763425 shutdown_connections 2024-03-06T07:27:25.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.556+0000 7fc2d5800640 1 -- 172.21.15.6:0/3633763425 wait complete. 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.558+0000 7fc2d5800640 1 Processor -- start 2024-03-06T07:27:25.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.558+0000 7fc2d5800640 1 -- start start 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2d5800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4004d40 0x7fc2d0157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2d5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 0x7fc2d0158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2d5800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d015a480 0x7fc2d015a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2d5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d01624b0 con 0x7fc2d0158a40 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2d5800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d006ce20 con 0x7fc2b4004d40 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2d5800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d006d620 con 0x7fc2d015a480 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 0x7fc2d0158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 0x7fc2d0158ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37418/0 (socket says 172.21.15.6:37418) 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.559+0000 7fc2cf400640 1 -- 172.21.15.6:0/2086493480 learned_addr learned my addr 172.21.15.6:0/2086493480 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4004d40 0x7fc2d0157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2d4e00640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d015a480 0x7fc2d015a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cf400640 1 -- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d015a480 msgr2=0x7fc2d015a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cf400640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d015a480 0x7fc2d015a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cf400640 1 -- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4004d40 msgr2=0x7fc2d0157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cf400640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4004d40 0x7fc2d0157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cf400640 1 -- 172.21.15.6:0/2086493480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d01594c0 con 0x7fc2d0158a40 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2cf400640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 0x7fc2d0158ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc011570 tx=0x7fc2bc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc015870 con 0x7fc2d0158a40 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc015a10 con 0x7fc2d0158a40 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc015dc0 con 0x7fc2d0158a40 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.560+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d01597a0 con 0x7fc2d0158a40 2024-03-06T07:27:25.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.561+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d0159d60 con 0x7fc2d0158a40 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.562+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d004f330 con 0x7fc2d0158a40 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.563+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2bc00f460 con 0x7fc2d0158a40 2024-03-06T07:27:25.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.563+0000 7fc2c7e00640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2b006b4d0 0x7fc2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.564+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc2bc08a820 con 0x7fc2d0158a40 2024-03-06T07:27:25.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.564+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2b006b4d0 0x7fc2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.565+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2b006b4d0 0x7fc2b006d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0001330 tx=0x7fc2c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.568+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2bc019030 con 0x7fc2d0158a40 2024-03-06T07:27:25.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.771+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2d00af2f0 con 0x7fc2d0158a40 2024-03-06T07:27:25.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.773+0000 7fc2c7e00640 1 -- 172.21.15.6:0/2086493480 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12812 (secure 0 0 0) 0x7fc2bc059950 con 0x7fc2d0158a40 2024-03-06T07:27:25.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:25.776 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:19.967437+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2b006b4d0 msgr2=0x7fc2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2b006b4d0 0x7fc2b006d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0001330 tx=0x7fc2c0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 msgr2=0x7fc2d0158ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 0x7fc2d0158ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc011570 tx=0x7fc2bc011a30 comp rx=0 tx=0).stop 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 shutdown_connections 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2b006b4d0 0x7fc2b006d990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d015a480 0x7fc2d015a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d0158a40 0x7fc2d0158ec0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 --2- 172.21.15.6:0/2086493480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2b4004d40 0x7fc2d0157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.776+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 >> 172.21.15.6:0/2086493480 conn(0x7fc2d006b3d0 msgr2=0x7fc2d0072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.777+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 shutdown_connections 2024-03-06T07:27:25.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:25.777+0000 7fc2d5800640 1 -- 172.21.15.6:0/2086493480 wait complete. 2024-03-06T07:27:25.798 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:25.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:26.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.016+0000 7f3d07e00640 1 Processor -- start 2024-03-06T07:27:26.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.016+0000 7f3d07e00640 1 -- start start 2024-03-06T07:27:26.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d07e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d000d2050 0x7f3d000b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d0006f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d07e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0015cbc0 con 0x7f3d00156420 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d07e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0015cea0 con 0x7f3d000d2050 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d07e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0015d180 con 0x7f3d0015c420 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d000d2050 0x7f3d000b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d000d2050 0x7f3d000b6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37426/0 (socket says 172.21.15.6:37426) 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 -- 172.21.15.6:0/2840019394 learned_addr learned my addr 172.21.15.6:0/2840019394 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d05e00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d0006f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 -- 172.21.15.6:0/2840019394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 msgr2=0x7f3d0006f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d0006f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 -- 172.21.15.6:0/2840019394 >> v1:172.21.15.178:6789/0 conn(0x7f3d00156420 legacy=0x7f3d000d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.018+0000 7f3d04a00640 1 -- 172.21.15.6:0/2840019394 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0006efb0 con 0x7f3d000d2050 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.019+0000 7f3d05e00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d0006f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:26.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.020+0000 7f3d04a00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d000b6090 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009870 tx=0x7f3cec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380225b9c9edd284 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.020+0000 7f3cffe00640 1 -- 172.21.15.6:0/2840019394 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec035020 con 0x7f3d000d2050 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.020+0000 7f3cffe00640 1 -- 172.21.15.6:0/2840019394 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cec031640 con 0x7f3d000d2050 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.020+0000 7f3cffe00640 1 -- 172.21.15.6:0/2840019394 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec02d040 con 0x7f3d000d2050 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 -- 172.21.15.6:0/2840019394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 msgr2=0x7f3d000b6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d000b6090 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3cec009870 tx=0x7f3cec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 -- 172.21.15.6:0/2840019394 shutdown_connections 2024-03-06T07:27:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d0006f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 --2- 172.21.15.6:0/2840019394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d000b6090 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 -- 172.21.15.6:0/2840019394 >> 172.21.15.6:0/2840019394 conn(0x7f3d000732d0 msgr2=0x7f3d0009aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 -- 172.21.15.6:0/2840019394 shutdown_connections 2024-03-06T07:27:26.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.021+0000 7f3d07e00640 1 -- 172.21.15.6:0/2840019394 wait complete. 2024-03-06T07:27:26.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.023+0000 7f3d07e00640 1 Processor -- start 2024-03-06T07:27:26.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.023+0000 7f3d07e00640 1 -- start start 2024-03-06T07:27:26.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d0015bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d00156420 0x7f3d00154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d07e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d000cf160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d07e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006fa10 con 0x7f3d000d2050 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d07e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00156800 con 0x7f3d00156420 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d07e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006fb90 con 0x7f3d0015c420 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d0015bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d0015bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37430/0 (socket says 172.21.15.6:37430) 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d05400640 1 -- 172.21.15.6:0/330342829 learned_addr learned my addr 172.21.15.6:0/330342829 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d04a00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d00156420 0x7f3d00154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.024+0000 7f3d05e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d000cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d05400640 1 -- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 msgr2=0x7f3d000cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d05400640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d000cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d05400640 1 -- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d00156420 msgr2=0x7f3d00154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d05400640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d00156420 0x7f3d00154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d05400640 1 -- 172.21.15.6:0/330342829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d000cfa40 con 0x7f3d000d2050 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d05400640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d0015bda0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400c7c0 tx=0x7f3cf400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4016020 con 0x7f3d000d2050 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf400aa50 con 0x7f3d000d2050 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d000cfd20 con 0x7f3d000d2050 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf400ae40 con 0x7f3d000d2050 2024-03-06T07:27:26.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.025+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d000d02e0 con 0x7f3d000d2050 2024-03-06T07:27:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.027+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0004f330 con 0x7f3d000d2050 2024-03-06T07:27:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.027+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cf4019020 con 0x7f3d000d2050 2024-03-06T07:27:26.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.028+0000 7f3cfd600640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3ce006b4d0 0x7f3ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.028+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3cf408a060 con 0x7f3d000d2050 2024-03-06T07:27:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.029+0000 7f3d04a00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3ce006b4d0 0x7f3ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.030+0000 7f3d04a00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3ce006b4d0 0x7f3ce006d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3cec036000 tx=0x7f3cec03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.033+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3cf4020030 con 0x7f3d000d2050 2024-03-06T07:27:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.232+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d000caf10 con 0x7f3d000d2050 2024-03-06T07:27:26.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.233+0000 7f3cfd600640 1 -- 172.21.15.6:0/330342829 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3cf4059190 con 0x7f3d000d2050 2024-03-06T07:27:26.235 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.235+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3ce006b4d0 msgr2=0x7f3ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.235+0000 7f3d07e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3ce006b4d0 0x7f3ce006d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3cec036000 tx=0x7f3cec03c040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.235+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 msgr2=0x7f3d0015bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.235+0000 7f3d07e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d0015bda0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400c7c0 tx=0x7f3cf400cc80 comp rx=0 tx=0).stop 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 shutdown_connections 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3ce006b4d0 0x7f3ce006d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d00156420 0x7f3d00154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d0015c420 0x7f3d000cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 --2- 172.21.15.6:0/330342829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d000d2050 0x7f3d0015bda0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 >> 172.21.15.6:0/330342829 conn(0x7f3d000732d0 msgr2=0x7f3d0006ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 shutdown_connections 2024-03-06T07:27:26.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.236+0000 7f3d07e00640 1 -- 172.21.15.6:0/330342829 wait complete. 2024-03-06T07:27:26.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:26.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.476+0000 7f0a70e00640 1 Processor -- start 2024-03-06T07:27:26.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.477+0000 7f0a70e00640 1 -- start start 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.478+0000 7f0a70e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a6c0d4a10 0x7f0a6c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.478+0000 7f0a70e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d0c80 0x7f0a6c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.478+0000 7f0a70e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c0b3710 con 0x7f0a6c15e0b0 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.478+0000 7f0a70e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b09e0 con 0x7f0a6c0d4a10 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.478+0000 7f0a70e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0cfa20 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a6c0d4a10 0x7f0a6c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a6c0d4a10 0x7f0a6c0d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37434/0 (socket says 172.21.15.6:37434) 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 -- 172.21.15.6:0/1936055059 learned_addr learned my addr 172.21.15.6:0/1936055059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:26.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6be00640 1 --2- 172.21.15.6:0/1936055059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d0c80 0x7f0a6c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 -- 172.21.15.6:0/1936055059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d0c80 msgr2=0x7f0a6c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 --2- 172.21.15.6:0/1936055059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d0c80 0x7f0a6c0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 -- 172.21.15.6:0/1936055059 >> v1:172.21.15.178:6789/0 conn(0x7f0a6c15e0b0 legacy=0x7f0a6c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.479+0000 7f0a6aa00640 1 -- 172.21.15.6:0/1936055059 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0b39f0 con 0x7f0a6c0d4a10 2024-03-06T07:27:26.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.481+0000 7f0a6aa00640 1 --2- 172.21.15.6:0/1936055059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d4a10 0x7f0a6c0d0740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009870 tx=0x7f0a5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be729bf9b3118d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.481+0000 7f0a6a000640 1 -- 172.21.15.6:0/1936055059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58034070 con 0x7f0a6c0d4a10 2024-03-06T07:27:26.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.481+0000 7f0a6a000640 1 -- 172.21.15.6:0/1936055059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a58030840 con 0x7f0a6c0d4a10 2024-03-06T07:27:26.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.481+0000 7f0a6a000640 1 -- 172.21.15.6:0/1936055059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a58030c10 con 0x7f0a6c0d4a10 2024-03-06T07:27:26.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 -- 172.21.15.6:0/1936055059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d4a10 msgr2=0x7f0a6c0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 --2- 172.21.15.6:0/1936055059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d4a10 0x7f0a6c0d0740 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009870 tx=0x7f0a5802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:26.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 -- 172.21.15.6:0/1936055059 shutdown_connections 2024-03-06T07:27:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 --2- 172.21.15.6:0/1936055059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d0c80 0x7f0a6c0b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 --2- 172.21.15.6:0/1936055059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d4a10 0x7f0a6c0d0740 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 -- 172.21.15.6:0/1936055059 >> 172.21.15.6:0/1936055059 conn(0x7f0a6c0d3d30 msgr2=0x7f0a6c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 -- 172.21.15.6:0/1936055059 shutdown_connections 2024-03-06T07:27:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.482+0000 7f0a70e00640 1 -- 172.21.15.6:0/1936055059 wait complete. 2024-03-06T07:27:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.484+0000 7f0a70e00640 1 Processor -- start 2024-03-06T07:27:26.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.484+0000 7f0a70e00640 1 -- start start 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a70e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 0x7f0a6c14f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a70e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d4a10 0x7f0a6c15fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a70e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c15e0b0 0x7f0a6c06c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a70e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c15e490 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a70e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0b3fb0 con 0x7f0a6c15e0b0 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a70e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0cfc60 con 0x7f0a6c0d4a10 2024-03-06T07:27:26.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.485+0000 7f0a6b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 0x7f0a6c14f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 0x7f0a6c14f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37442/0 (socket says 172.21.15.6:37442) 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 -- 172.21.15.6:0/2286384749 learned_addr learned my addr 172.21.15.6:0/2286384749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6be00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c15e0b0 0x7f0a6c06c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6aa00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d4a10 0x7f0a6c15fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 -- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d4a10 msgr2=0x7f0a6c15fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d4a10 0x7f0a6c15fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 -- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c15e0b0 msgr2=0x7f0a6c06c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c15e0b0 0x7f0a6c06c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 -- 172.21.15.6:0/2286384749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c160690 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a6b400640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 0x7f0a6c14f210 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c870 tx=0x7f0a5c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.486+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c003b70 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.487+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c003d30 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.487+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00aa90 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.487+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c160970 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.487+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c160f30 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.488+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c0d3480 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.489+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a5c00ac30 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.489+0000 7f0a63400640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4c06b4d0 0x7f0a4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.490+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a5c089f00 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.490+0000 7f0a6aa00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4c06b4d0 0x7f0a4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.491+0000 7f0a6aa00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4c06b4d0 0x7f0a4c06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a58036000 tx=0x7f0a5803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.494+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a6c0d3480 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.693+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0a6c04f330 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.694+0000 7f0a63400640 1 -- 172.21.15.6:0/2286384749 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0a5c059030 con 0x7f0a6c0d0c80 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:27:26.697 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:27:26.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4c06b4d0 msgr2=0x7f0a4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4c06b4d0 0x7f0a4c06d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a58036000 tx=0x7f0a5803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 msgr2=0x7f0a6c14f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 0x7f0a6c14f210 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c870 tx=0x7f0a5c00cd30 comp rx=0 tx=0).stop 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 shutdown_connections 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4c06b4d0 0x7f0a4c06d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c15e0b0 0x7f0a6c06c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0d4a10 0x7f0a6c15fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 --2- 172.21.15.6:0/2286384749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c0d0c80 0x7f0a6c14f210 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 >> 172.21.15.6:0/2286384749 conn(0x7f0a6c0d3d30 msgr2=0x7f0a6c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 shutdown_connections 2024-03-06T07:27:26.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.697+0000 7f0a70e00640 1 -- 172.21.15.6:0/2286384749 wait complete. 2024-03-06T07:27:26.719 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:26.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:26.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.947+0000 7f9238a00640 1 Processor -- start 2024-03-06T07:27:26.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.947+0000 7f9238a00640 1 -- start start 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9238a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92340d0d90 0x7f92340ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9238a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f92340b4900 0x7f92340cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9238a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92340d23c0 con 0x7f923415e1f0 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9238a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340d4c30 con 0x7f92340d0d90 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9238a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340d26a0 con 0x7f92340b4900 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9233400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f923415e1f0 0x7f92340d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50904/0 (socket says 172.21.15.6:50904) 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9233400640 1 -- 172.21.15.6:0/150237275 learned_addr learned my addr 172.21.15.6:0/150237275 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.949+0000 7f9232a00640 1 --2- 172.21.15.6:0/150237275 >> v2:172.21.15.6:3300/0 conn(0x7f92340d0d90 0x7f92340ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9233e00640 1 --2- 172.21.15.6:0/150237275 >> v2:172.21.15.178:3300/0 conn(0x7f92340b4900 0x7f92340cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9232a00640 1 -- 172.21.15.6:0/150237275 >> v2:172.21.15.178:3300/0 conn(0x7f92340b4900 msgr2=0x7f92340cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:26.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9232a00640 1 --2- 172.21.15.6:0/150237275 >> v2:172.21.15.178:3300/0 conn(0x7f92340b4900 0x7f92340cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9232000640 1 -- 172.21.15.6:0/150237275 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001542600 0 0) 0x7f92340d23c0 con 0x7f923415e1f0 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9232a00640 1 -- 172.21.15.6:0/150237275 >> v1:172.21.15.6:6789/0 conn(0x7f923415e1f0 legacy=0x7f92340d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9232a00640 1 -- 172.21.15.6:0/150237275 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92340d2980 con 0x7f92340d0d90 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.950+0000 7f9233e00640 1 --2- 172.21.15.6:0/150237275 >> v2:172.21.15.178:3300/0 conn(0x7f92340b4900 0x7f92340cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.951+0000 7f9232a00640 1 --2- 172.21.15.6:0/150237275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340d0d90 0x7f92340ce8c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f923408dbc0 tx=0x7f922002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25b683b74c5a9e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.951+0000 7f9232000640 1 -- 172.21.15.6:0/150237275 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92200315a0 con 0x7f92340d0d90 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.951+0000 7f9232000640 1 -- 172.21.15.6:0/150237275 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9220037070 con 0x7f92340d0d90 2024-03-06T07:27:26.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.951+0000 7f9232000640 1 -- 172.21.15.6:0/150237275 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220031960 con 0x7f92340d0d90 2024-03-06T07:27:26.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 -- 172.21.15.6:0/150237275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340d0d90 msgr2=0x7f92340ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 --2- 172.21.15.6:0/150237275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340d0d90 0x7f92340ce8c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f923408dbc0 tx=0x7f922002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:26.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 -- 172.21.15.6:0/150237275 shutdown_connections 2024-03-06T07:27:26.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 --2- 172.21.15.6:0/150237275 >> v2:172.21.15.178:3300/0 conn(0x7f92340b4900 0x7f92340cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 --2- 172.21.15.6:0/150237275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340d0d90 0x7f92340ce8c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 -- 172.21.15.6:0/150237275 >> 172.21.15.6:0/150237275 conn(0x7f923409d000 msgr2=0x7f92340d3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:26.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 -- 172.21.15.6:0/150237275 shutdown_connections 2024-03-06T07:27:26.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.952+0000 7f9238a00640 1 -- 172.21.15.6:0/150237275 wait complete. 2024-03-06T07:27:26.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.954+0000 7f9238a00640 1 Processor -- start 2024-03-06T07:27:26.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.955+0000 7f9238a00640 1 -- start start 2024-03-06T07:27:26.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9238a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 0x7f92340b9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9238a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92340d0d90 0x7f92340cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9238a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923415e1f0 0x7f92340b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9238a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340d4660 con 0x7f92340b4900 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9238a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340b2da0 con 0x7f923415e1f0 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9238a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92340d47e0 con 0x7f92340d0d90 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9233400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 0x7f92340b9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9233400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 0x7f92340b9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37460/0 (socket says 172.21.15.6:37460) 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9233400640 1 -- 172.21.15.6:0/2216435632 learned_addr learned my addr 172.21.15.6:0/2216435632 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.956+0000 7f9232a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92340d0d90 0x7f92340cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233e00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923415e1f0 0x7f92340b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233400640 1 -- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92340d0d90 msgr2=0x7f92340cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233400640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92340d0d90 0x7f92340cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233400640 1 -- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923415e1f0 msgr2=0x7f92340b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:26.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233400640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923415e1f0 0x7f92340b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233400640 1 -- 172.21.15.6:0/2216435632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92340b9a40 con 0x7f92340b4900 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9232a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92340d0d90 0x7f92340cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233e00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923415e1f0 0x7f92340b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.957+0000 7f9233400640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 0x7f92340b9500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f922400c8c0 tx=0x7f922400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.958+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224018070 con 0x7f92340b4900 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.958+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400a640 con 0x7f92340b4900 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.958+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922400aa30 con 0x7f92340b4900 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.958+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923415fb00 con 0x7f92340b4900 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.958+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92341600c0 con 0x7f92340b4900 2024-03-06T07:27:26.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.959+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923404f330 con 0x7f92340b4900 2024-03-06T07:27:26.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.960+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f922400abd0 con 0x7f92340b4900 2024-03-06T07:27:26.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.961+0000 7f922b400640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f921406b4d0 0x7f921406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:26.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.961+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9224089700 con 0x7f92340b4900 2024-03-06T07:27:26.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.962+0000 7f9232a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f921406b4d0 0x7f921406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:26.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.963+0000 7f9232a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f921406b4d0 0x7f921406d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9220036000 tx=0x7f922003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:26.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:26.966+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92240587a0 con 0x7f92340b4900 2024-03-06T07:27:27.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:27.165+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f92340b6100 con 0x7f92340b4900 2024-03-06T07:27:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.179+0000 7f922b400640 1 -- 172.21.15.6:0/2216435632 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f92240580a0 con 0x7f92340b4900 2024-03-06T07:27:28.182 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f921406b4d0 msgr2=0x7f921406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f921406b4d0 0x7f921406d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9220036000 tx=0x7f922003d040 comp rx=0 tx=0).stop 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 msgr2=0x7f92340b9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 0x7f92340b9500 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f922400c8c0 tx=0x7f922400cd80 comp rx=0 tx=0).stop 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 shutdown_connections 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f921406b4d0 0x7f921406d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923415e1f0 0x7f92340b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92340d0d90 0x7f92340cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 --2- 172.21.15.6:0/2216435632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92340b4900 0x7f92340b9500 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 >> 172.21.15.6:0/2216435632 conn(0x7f923409d000 msgr2=0x7f923406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.184+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 shutdown_connections 2024-03-06T07:27:28.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.185+0000 7f9238a00640 1 -- 172.21.15.6:0/2216435632 wait complete. 2024-03-06T07:27:28.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.421+0000 7fb844c00640 1 Processor -- start 2024-03-06T07:27:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.421+0000 7fb844c00640 1 -- start start 2024-03-06T07:27:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.423+0000 7fb844c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8400ad6f0 0x7fb84015c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.423+0000 7fb844c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb84006d5d0 0x7fb84006da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.423+0000 7fb844c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb840156830 con 0x7fb84015f960 2024-03-06T07:27:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.423+0000 7fb844c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84006ce60 con 0x7fb84006d5d0 2024-03-06T07:27:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.423+0000 7fb844c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb840160120 con 0x7fb8400ad6f0 2024-03-06T07:27:28.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb837e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb84015f960 0x7fb8400adad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50920/0 (socket says 172.21.15.6:50920) 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb837e00640 1 -- 172.21.15.6:0/339635832 learned_addr learned my addr 172.21.15.6:0/339635832 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83e800640 1 --2- 172.21.15.6:0/339635832 >> v2:172.21.15.6:3300/0 conn(0x7fb84006d5d0 0x7fb84006da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83de00640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8400ad6f0 0x7fb84015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83d400640 1 -- 172.21.15.6:0/339635832 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481863136 0 0) 0x7fb840156830 con 0x7fb84015f960 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83d400640 1 -- 172.21.15.6:0/339635832 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb824003520 con 0x7fb84015f960 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83e800640 1 -- 172.21.15.6:0/339635832 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8400ad6f0 msgr2=0x7fb84015c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83e800640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8400ad6f0 0x7fb84015c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83e800640 1 -- 172.21.15.6:0/339635832 >> v1:172.21.15.6:6789/0 conn(0x7fb84015f960 legacy=0x7fb8400adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83e800640 1 -- 172.21.15.6:0/339635832 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb840160400 con 0x7fb84006d5d0 2024-03-06T07:27:28.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.424+0000 7fb83de00640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8400ad6f0 0x7fb84015c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:28.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.425+0000 7fb83e800640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb84006d5d0 0x7fb84006da40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb838009d90 tx=0x7fb83802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54fd59e6366f48d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.426+0000 7fb83d400640 1 -- 172.21.15.6:0/339635832 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838039070 con 0x7fb84006d5d0 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.426+0000 7fb83d400640 1 -- 172.21.15.6:0/339635832 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83802cc50 con 0x7fb84006d5d0 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.426+0000 7fb83d400640 1 -- 172.21.15.6:0/339635832 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838034440 con 0x7fb84006d5d0 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.426+0000 7fb844c00640 1 -- 172.21.15.6:0/339635832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb84006d5d0 msgr2=0x7fb84006da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.426+0000 7fb844c00640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb84006d5d0 0x7fb84006da40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb838009d90 tx=0x7fb83802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.427+0000 7fb844c00640 1 -- 172.21.15.6:0/339635832 shutdown_connections 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.427+0000 7fb844c00640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb84006d5d0 0x7fb84006da40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.427+0000 7fb844c00640 1 --2- 172.21.15.6:0/339635832 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8400ad6f0 0x7fb84015c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.427+0000 7fb844c00640 1 -- 172.21.15.6:0/339635832 >> 172.21.15.6:0/339635832 conn(0x7fb84006b3d0 msgr2=0x7fb84006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.427+0000 7fb844c00640 1 -- 172.21.15.6:0/339635832 shutdown_connections 2024-03-06T07:27:28.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.427+0000 7fb844c00640 1 -- 172.21.15.6:0/339635832 wait complete. 2024-03-06T07:27:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.428+0000 7fb844c00640 1 Processor -- start 2024-03-06T07:27:28.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 -- start start 2024-03-06T07:27:28.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb84006d5d0 0x7fb840154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 0x7fb8401443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb84015f960 0x7fb84015de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84015fdc0 con 0x7fb8400ad6f0 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84006d0b0 con 0x7fb84006d5d0 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.429+0000 7fb844c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84006df80 con 0x7fb84015f960 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 0x7fb8401443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 0x7fb8401443a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37492/0 (socket says 172.21.15.6:37492) 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 -- 172.21.15.6:0/1316732289 learned_addr learned my addr 172.21.15.6:0/1316732289 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb83e800640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb84015f960 0x7fb84015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb83de00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb84006d5d0 0x7fb840154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 -- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb84015f960 msgr2=0x7fb84015de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb84015f960 0x7fb84015de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 -- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb84006d5d0 msgr2=0x7fb840154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb84006d5d0 0x7fb840154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.430+0000 7fb837e00640 1 -- 172.21.15.6:0/1316732289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb840153ec0 con 0x7fb8400ad6f0 2024-03-06T07:27:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.431+0000 7fb837e00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 0x7fb8401443a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb82800c390 tx=0x7fb82800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.431+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb828017070 con 0x7fb8400ad6f0 2024-03-06T07:27:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.431+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb828005ce0 con 0x7fb8400ad6f0 2024-03-06T07:27:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.431+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb840154160 con 0x7fb8400ad6f0 2024-03-06T07:27:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.431+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8280072f0 con 0x7fb8400ad6f0 2024-03-06T07:27:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.431+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb840154a00 con 0x7fb8400ad6f0 2024-03-06T07:27:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.432+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb840154550 con 0x7fb8400ad6f0 2024-03-06T07:27:28.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.433+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb82801a070 con 0x7fb8400ad6f0 2024-03-06T07:27:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.434+0000 7fb836000640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81c06b4d0 0x7fb81c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.434+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb828089540 con 0x7fb8400ad6f0 2024-03-06T07:27:28.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.435+0000 7fb83de00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81c06b4d0 0x7fb81c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.436+0000 7fb83de00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81c06b4d0 0x7fb81c06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb830006fd0 tx=0x7fb830008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.439+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb828058470 con 0x7fb8400ad6f0 2024-03-06T07:27:28.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:28.639+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb840154330 con 0x7fb8400ad6f0 2024-03-06T07:27:29.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.186+0000 7fb836000640 1 -- 172.21.15.6:0/1316732289 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb828057d70 con 0x7fb8400ad6f0 2024-03-06T07:27:29.188 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.189+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81c06b4d0 msgr2=0x7fb81c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.189+0000 7fb844c00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81c06b4d0 0x7fb81c06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb830006fd0 tx=0x7fb830008040 comp rx=0 tx=0).stop 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.189+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 msgr2=0x7fb8401443a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.189+0000 7fb844c00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 0x7fb8401443a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb82800c390 tx=0x7fb82800c850 comp rx=0 tx=0).stop 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 shutdown_connections 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81c06b4d0 0x7fb81c06d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb84015f960 0x7fb84015de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8400ad6f0 0x7fb8401443a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 --2- 172.21.15.6:0/1316732289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb84006d5d0 0x7fb840154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 >> 172.21.15.6:0/1316732289 conn(0x7fb84006b3d0 msgr2=0x7fb8400b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 shutdown_connections 2024-03-06T07:27:29.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.190+0000 7fb844c00640 1 -- 172.21.15.6:0/1316732289 wait complete. 2024-03-06T07:27:29.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:29.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.414+0000 7f45e6200640 1 Processor -- start 2024-03-06T07:27:29.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.414+0000 7f45e6200640 1 -- start start 2024-03-06T07:27:29.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.415+0000 7f45e6200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45e006da40 0x7f45e006e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.415+0000 7f45e6200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f45e015e680 0x7f45e006c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.415+0000 7f45e6200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e00ccad0 0x7f45e00ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.415+0000 7f45e6200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e0161c70 con 0x7f45e006da40 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.415+0000 7f45e6200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e015eb80 con 0x7f45e015e680 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.415+0000 7f45e6200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e015ee60 con 0x7f45e00ccad0 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45e006da40 0x7f45e006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45e006da40 0x7f45e006e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37502/0 (socket says 172.21.15.6:37502) 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 -- 172.21.15.6:0/62304533 learned_addr learned my addr 172.21.15.6:0/62304533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45dea00640 1 --2- 172.21.15.6:0/62304533 >> v2:172.21.15.178:3300/0 conn(0x7f45e015e680 0x7f45e006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45dfe00640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e00ccad0 0x7f45e00ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 -- 172.21.15.6:0/62304533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e00ccad0 msgr2=0x7f45e00ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e00ccad0 0x7f45e00ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 -- 172.21.15.6:0/62304533 >> v2:172.21.15.178:3300/0 conn(0x7f45e015e680 msgr2=0x7f45e006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 --2- 172.21.15.6:0/62304533 >> v2:172.21.15.178:3300/0 conn(0x7f45e015e680 0x7f45e006c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45df400640 1 -- 172.21.15.6:0/62304533 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e015f140 con 0x7f45e006da40 2024-03-06T07:27:29.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.416+0000 7f45dfe00640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e00ccad0 0x7f45e00ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.417+0000 7f45df400640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e006e340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45e006bfd0 tx=0x7f45d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f05b03dc0d5c1ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.418+0000 7f45de000640 1 -- 172.21.15.6:0/62304533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0035020 con 0x7f45e006da40 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.418+0000 7f45de000640 1 -- 172.21.15.6:0/62304533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0031870 con 0x7f45e006da40 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.418+0000 7f45de000640 1 -- 172.21.15.6:0/62304533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0031c40 con 0x7f45e006da40 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.418+0000 7f45e6200640 1 -- 172.21.15.6:0/62304533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 msgr2=0x7f45e006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.418+0000 7f45e6200640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e006e340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45e006bfd0 tx=0x7f45d002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 -- 172.21.15.6:0/62304533 shutdown_connections 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e00ccad0 0x7f45e00ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 --2- 172.21.15.6:0/62304533 >> v2:172.21.15.178:3300/0 conn(0x7f45e015e680 0x7f45e006c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 --2- 172.21.15.6:0/62304533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e006e340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 -- 172.21.15.6:0/62304533 >> 172.21.15.6:0/62304533 conn(0x7f45e006b3d0 msgr2=0x7f45e006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 -- 172.21.15.6:0/62304533 shutdown_connections 2024-03-06T07:27:29.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.419+0000 7f45e6200640 1 -- 172.21.15.6:0/62304533 wait complete. 2024-03-06T07:27:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.420+0000 7f45e6200640 1 Processor -- start 2024-03-06T07:27:29.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 -- start start 2024-03-06T07:27:29.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e01556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45e00ccad0 0x7f45e00add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e015e680 0x7f45e00ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e006ea00 con 0x7f45e006da40 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e006e880 con 0x7f45e00ccad0 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.421+0000 7f45e6200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e006cec0 con 0x7f45e015e680 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e01556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e01556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37510/0 (socket says 172.21.15.6:37510) 2024-03-06T07:27:29.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 -- 172.21.15.6:0/4195430295 learned_addr learned my addr 172.21.15.6:0/4195430295 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:29.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 -- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e015e680 msgr2=0x7f45e00ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e015e680 0x7f45e00ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 -- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45e00ccad0 msgr2=0x7f45e00add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45e00ccad0 0x7f45e00add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 -- 172.21.15.6:0/4195430295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e00ca380 con 0x7f45e006da40 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.422+0000 7f45df400640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e01556c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45d00097c0 tx=0x7f45d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.423+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0040070 con 0x7f45e006da40 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.423+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0030d00 con 0x7f45e006da40 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.423+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e00ca660 con 0x7f45e006da40 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.423+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0035040 con 0x7f45e006da40 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.423+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e00cac20 con 0x7f45e006da40 2024-03-06T07:27:29.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.424+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e004f330 con 0x7f45e006da40 2024-03-06T07:27:29.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.425+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45d004b060 con 0x7f45e006da40 2024-03-06T07:27:29.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.426+0000 7f45d7400640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45c006b520 0x7f45c006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.426+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f45d00abea0 con 0x7f45e006da40 2024-03-06T07:27:29.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.427+0000 7f45dea00640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45c006b520 0x7f45c006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.428+0000 7f45dea00640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45c006b520 0x7f45c006d9e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f45cc006fd0 tx=0x7f45cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.432+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45d007adf0 con 0x7f45e006da40 2024-03-06T07:27:29.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:29.671+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f45e0071080 con 0x7f45e006da40 2024-03-06T07:27:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.228+0000 7f45d7400640 1 -- 172.21.15.6:0/4195430295 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f45d007a6f0 con 0x7f45e006da40 2024-03-06T07:27:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:27:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.238+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45c006b520 msgr2=0x7f45c006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.238+0000 7f45e6200640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45c006b520 0x7f45c006d9e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f45cc006fd0 tx=0x7f45cc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.238+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 msgr2=0x7f45e01556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.238+0000 7f45e6200640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e01556c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45d00097c0 tx=0x7f45d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.239+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 shutdown_connections 2024-03-06T07:27:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.239+0000 7f45e6200640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45c006b520 0x7f45c006d9e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.239+0000 7f45e6200640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45e00ccad0 0x7f45e00add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.239+0000 7f45e6200640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45e015e680 0x7f45e00ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.239+0000 7f45e6200640 1 --2- 172.21.15.6:0/4195430295 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45e006da40 0x7f45e01556c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.239+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 >> 172.21.15.6:0/4195430295 conn(0x7f45e006b3d0 msgr2=0x7f45e00afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.240+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 shutdown_connections 2024-03-06T07:27:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.240+0000 7f45e6200640 1 -- 172.21.15.6:0/4195430295 wait complete. 2024-03-06T07:27:30.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.445+0000 7f2ff9200640 1 Processor -- start 2024-03-06T07:27:30.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.446+0000 7f2ff9200640 1 -- start start 2024-03-06T07:27:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 0x7f2ff40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff9200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2ff40cce90 0x7f2ff4161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff9200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff41627f0 con 0x7f2ff4162000 2024-03-06T07:27:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff9200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff415f3a0 con 0x7f2ff40cce90 2024-03-06T07:27:30.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406d190 con 0x7f2ff415efc0 2024-03-06T07:27:30.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2ff4162000 0x7f2ff40ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50932/0 (socket says 172.21.15.6:50932) 2024-03-06T07:27:30.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.447+0000 7f2ff3e00640 1 -- 172.21.15.6:0/3756469086 learned_addr learned my addr 172.21.15.6:0/3756469086 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:30.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff3400640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 0x7f2ff40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2febe00640 1 --2- 172.21.15.6:0/3756469086 >> v2:172.21.15.178:3300/0 conn(0x7f2ff40cce90 0x7f2ff4161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721276746 0 0) 0x7f2ff41627f0 con 0x7f2ff4162000 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd8003520 con 0x7f2ff4162000 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107236105 0 0) 0x7f2fd8003520 con 0x7f2ff4162000 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 msgr2=0x7f2ff40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 0x7f2ff40cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 >> v2:172.21.15.178:3300/0 conn(0x7f2ff40cce90 msgr2=0x7f2ff4161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 --2- 172.21.15.6:0/3756469086 >> v2:172.21.15.178:3300/0 conn(0x7f2ff40cce90 0x7f2ff4161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff406c860 con 0x7f2ff4162000 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2fec0019f0 con 0x7f2ff4162000 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 >> v1:172.21.15.6:6789/0 conn(0x7f2ff4162000 legacy=0x7f2ff40ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fd80039b0 0x7f2fd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 0x7f2fd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2fd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff415f3a0 con 0x7f2fd8004d40 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406d190 con 0x7f2fd80039b0 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.448+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd800b9c0 con 0x7f2ff415efc0 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.449+0000 7f2ff3e00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2fd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.449+0000 7f2ff3e00640 1 -- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 msgr2=0x7f2fd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.449+0000 7f2ff3e00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff415efc0 0x7f2fd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.449+0000 7f2ff3e00640 1 -- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fd80039b0 msgr2=0x7f2fd8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.449+0000 7f2ff3e00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fd80039b0 0x7f2fd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.449+0000 7f2ff3e00640 1 -- 172.21.15.6:0/3756469086 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8003520 con 0x7f2fd8004d40 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.450+0000 7f2ff3e00640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2fd80051c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0013d0 tx=0x7f2fd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3203325c1a9568 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.450+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00a4e0 con 0x7f2fd8004d40 2024-03-06T07:27:30.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.450+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec034650 con 0x7f2fd8004d40 2024-03-06T07:27:30.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.450+0000 7f2ff2a00640 1 -- 172.21.15.6:0/3756469086 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec034a40 con 0x7f2fd8004d40 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff9200640 1 -- 172.21.15.6:0/3756469086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 msgr2=0x7f2fd80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff9200640 1 --2- 172.21.15.6:0/3756469086 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2fd80051c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2fec0013d0 tx=0x7f2fd8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff3e00640 1 -- 172.21.15.6:0/3756469086 reap_dead start 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff9200640 1 -- 172.21.15.6:0/3756469086 shutdown_connections 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff9200640 1 -- 172.21.15.6:0/3756469086 >> 172.21.15.6:0/3756469086 conn(0x7f2ff406b3d0 msgr2=0x7f2ff406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff9200640 1 -- 172.21.15.6:0/3756469086 shutdown_connections 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.451+0000 7f2ff9200640 1 -- 172.21.15.6:0/3756469086 wait complete. 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.453+0000 7f2ff9200640 1 Processor -- start 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.453+0000 7f2ff9200640 1 -- start start 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2ff41f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff9200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ff41f79a0 0x7f2ff41580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff41468b0 0x7f2ff4146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4071c30 con 0x7f2fd8004d40 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff9200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406ce20 con 0x7f2ff41f79a0 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4162490 con 0x7f2ff41468b0 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2ff41f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2ff41f39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37532/0 (socket says 172.21.15.6:37532) 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 -- 172.21.15.6:0/2046609537 learned_addr learned my addr 172.21.15.6:0/2046609537 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2febe00640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ff41f79a0 0x7f2ff41580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 -- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff41468b0 msgr2=0x7f2ff4146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3e00640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff41468b0 0x7f2ff4146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff41468b0 0x7f2ff4146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 -- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ff41f79a0 msgr2=0x7f2ff41580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ff41f79a0 0x7f2ff41580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2ff3400640 1 -- 172.21.15.6:0/2046609537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff41f7e00 con 0x7f2fd8004d40 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.454+0000 7f2febe00640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ff41f79a0 0x7f2ff41580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.455+0000 7f2ff3400640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2ff41f39f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400e780 tx=0x7f2fe400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.455+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4017070 con 0x7f2fd8004d40 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.455+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400ee90 con 0x7f2fd8004d40 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.455+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe400cb20 con 0x7f2fd8004d40 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.455+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff41f4100 con 0x7f2fd8004d40 2024-03-06T07:27:30.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.455+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff41f46c0 con 0x7f2fd8004d40 2024-03-06T07:27:30.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.456+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff404f330 con 0x7f2fd8004d40 2024-03-06T07:27:30.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.457+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fe401a070 con 0x7f2fd8004d40 2024-03-06T07:27:30.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.458+0000 7f2feb400640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2fd006b500 0x7f2fd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.458+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fe408a060 con 0x7f2fd8004d40 2024-03-06T07:27:30.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.459+0000 7f2febe00640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2fd006b500 0x7f2fd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.460+0000 7f2febe00640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2fd006b500 0x7f2fd006d9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4159800 tx=0x7f2fdc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.462+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fe4058e40 con 0x7f2fd8004d40 2024-03-06T07:27:30.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.653+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ff40af2f0 con 0x7f2fd8004d40 2024-03-06T07:27:30.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.656+0000 7f2feb400640 1 -- 172.21.15.6:0/2046609537 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16546 (secure 0 0 0) 0x7f2fe4058740 con 0x7f2fd8004d40 2024-03-06T07:27:30.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:30.659 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":21,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:29.673432+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2fd006b500 msgr2=0x7f2fd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2fd006b500 0x7f2fd006d9c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4159800 tx=0x7f2fdc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 msgr2=0x7f2ff41f39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2ff41f39f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2fe400e780 tx=0x7f2fe400ec40 comp rx=0 tx=0).stop 2024-03-06T07:27:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 shutdown_connections 2024-03-06T07:27:30.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2fd006b500 0x7f2fd006d9c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ff41f79a0 0x7f2ff41580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ff41468b0 0x7f2ff4146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 --2- 172.21.15.6:0/2046609537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fd8004d40 0x7f2ff41f39f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 >> 172.21.15.6:0/2046609537 conn(0x7f2ff406b3d0 msgr2=0x7f2ff40ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 shutdown_connections 2024-03-06T07:27:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.659+0000 7f2ff9200640 1 -- 172.21.15.6:0/2046609537 wait complete. 2024-03-06T07:27:30.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.893+0000 7f65bb200640 1 Processor -- start 2024-03-06T07:27:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.893+0000 7f65bb200640 1 -- start start 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65bb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b40cd060 0x7f65b41667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65bb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40714d0 con 0x7f65b4070f10 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65bb200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40717b0 con 0x7f65b4145300 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65bb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4071a90 con 0x7f65b40cd060 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65b9400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f65b4070f10 0x7f65b40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50940/0 (socket says 172.21.15.6:50940) 2024-03-06T07:27:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65b9400640 1 -- 172.21.15.6:0/2863793721 learned_addr learned my addr 172.21.15.6:0/2863793721 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65ba800640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b40cd060 0x7f65b41667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.895+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1668162608 0 0) 0x7f65b40714d0 con 0x7f65b4070f10 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659c003520 con 0x7f65b4070f10 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 923780779 0 0) 0x7f659c003520 con 0x7f65b4070f10 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b40cd060 msgr2=0x7f65b41667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b40cd060 0x7f65b41667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 >> v1:172.21.15.178:6789/0 conn(0x7f65b4145300 legacy=0x7f65b40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4071d70 con 0x7f65b4070f10 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f65a0001670 con 0x7f65b4070f10 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 >> v1:172.21.15.6:6789/0 conn(0x7f65b4070f10 legacy=0x7f65b40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f659c0039e0 0x7f659c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f659c0043d0 0x7f659c0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f659c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4071a90 con 0x7f659c005140 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c00c060 con 0x7f659c0043d0 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c00c260 con 0x7f659c0039e0 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.896+0000 7f65ba800640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b40cd060 0x7f65b41667e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65ba800640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f659c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65b9400640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f659c0043d0 0x7f659c0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65b9e00640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f659c0039e0 0x7f659c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65ba800640 1 -- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f659c0039e0 msgr2=0x7f659c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65ba800640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f659c0039e0 0x7f659c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65ba800640 1 -- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f659c0043d0 msgr2=0x7f659c0048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65ba800640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f659c0043d0 0x7f659c0048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.897+0000 7f65ba800640 1 -- 172.21.15.6:0/2863793721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c003520 con 0x7f659c005140 2024-03-06T07:27:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.898+0000 7f65ba800640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f659c00b9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65b000e1f0 tx=0x7f659c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcb2ce734b8f5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.898+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0009d40 con 0x7f659c005140 2024-03-06T07:27:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.898+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b002cce0 con 0x7f659c005140 2024-03-06T07:27:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.898+0000 7f65b8a00640 1 -- 172.21.15.6:0/2863793721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0003bb0 con 0x7f659c005140 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65bb200640 1 -- 172.21.15.6:0/2863793721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 msgr2=0x7f659c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65bb200640 1 --2- 172.21.15.6:0/2863793721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f659c00b9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f65b000e1f0 tx=0x7f659c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65ba800640 1 -- 172.21.15.6:0/2863793721 reap_dead start 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65bb200640 1 -- 172.21.15.6:0/2863793721 shutdown_connections 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65bb200640 1 -- 172.21.15.6:0/2863793721 >> 172.21.15.6:0/2863793721 conn(0x7f65b406b3d0 msgr2=0x7f65b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65bb200640 1 -- 172.21.15.6:0/2863793721 shutdown_connections 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.899+0000 7f65bb200640 1 -- 172.21.15.6:0/2863793721 wait complete. 2024-03-06T07:27:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.901+0000 7f65bb200640 1 Processor -- start 2024-03-06T07:27:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.901+0000 7f65bb200640 1 -- start start 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65bb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f65b415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65bb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65b4142e30 0x7f65b40b07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65bb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b41432b0 0x7f65b40b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65bb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b415d700 con 0x7f659c005140 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65bb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b415d580 con 0x7f65b4142e30 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65bb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b415d880 con 0x7f65b41432b0 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65b9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f65b415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.902+0000 7f65b9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f65b415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37540/0 (socket says 172.21.15.6:37540) 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 -- 172.21.15.6:0/4272423286 learned_addr learned my addr 172.21.15.6:0/4272423286 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65ba800640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b41432b0 0x7f65b40b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9400640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65b4142e30 0x7f65b40b07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 -- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b41432b0 msgr2=0x7f65b40b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b41432b0 0x7f65b40b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 -- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65b4142e30 msgr2=0x7f65b40b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65b4142e30 0x7f65b40b07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 -- 172.21.15.6:0/4272423286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40aa480 con 0x7f659c005140 2024-03-06T07:27:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9400640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65b4142e30 0x7f65b40b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.903+0000 7f65b9e00640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f65b415b340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65a800e580 tx=0x7f65a800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.904+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80052e0 con 0x7f659c005140 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.904+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8018070 con 0x7f659c005140 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.904+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b40aa1a0 con 0x7f659c005140 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.904+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8003810 con 0x7f659c005140 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.904+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b4142630 con 0x7f659c005140 2024-03-06T07:27:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.905+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b404f330 con 0x7f659c005140 2024-03-06T07:27:30.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.906+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65a80039b0 con 0x7f659c005140 2024-03-06T07:27:30.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.907+0000 7f65ae000640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f659406b4d0 0x7f659406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:30.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.907+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65a8089f10 con 0x7f659c005140 2024-03-06T07:27:30.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.908+0000 7f65b9400640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f659406b4d0 0x7f659406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.908+0000 7f65b9400640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f659406b4d0 0x7f659406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f65b40cc480 tx=0x7f65a0001db0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:30.911+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65a8058d00 con 0x7f659c005140 2024-03-06T07:27:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.108+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65b415b880 con 0x7f659c005140 2024-03-06T07:27:31.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.111+0000 7f65ae000640 1 -- 172.21.15.6:0/4272423286 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16546 (secure 0 0 0) 0x7f65a8058600 con 0x7f659c005140 2024-03-06T07:27:31.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:31.114 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":21,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:29.673432+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:31.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f659406b4d0 msgr2=0x7f659406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f659406b4d0 0x7f659406d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f65b40cc480 tx=0x7f65a0001db0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 msgr2=0x7f65b415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f65b415b340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65a800e580 tx=0x7f65a800ea40 comp rx=0 tx=0).stop 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 shutdown_connections 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f659406b4d0 0x7f659406d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65b4142e30 0x7f65b40b07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65b41432b0 0x7f65b40b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 --2- 172.21.15.6:0/4272423286 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f659c005140 0x7f65b415b340 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 >> 172.21.15.6:0/4272423286 conn(0x7f65b406b3d0 msgr2=0x7f65b415ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 shutdown_connections 2024-03-06T07:27:31.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.114+0000 7f65bb200640 1 -- 172.21.15.6:0/4272423286 wait complete. 2024-03-06T07:27:31.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.359+0000 7ff244000640 1 Processor -- start 2024-03-06T07:27:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.359+0000 7ff244000640 1 -- start start 2024-03-06T07:27:31.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff244000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff23c15bf20 0x7ff23c0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff244000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c072df0 con 0x7ff23c071e20 2024-03-06T07:27:31.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff244000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c0cdab0 con 0x7ff23c072520 2024-03-06T07:27:31.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff244000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23c0b4d70 con 0x7ff23c15bf20 2024-03-06T07:27:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff242000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff23c071e20 0x7ff23c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50948/0 (socket says 172.21.15.6:50948) 2024-03-06T07:27:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff242000640 1 -- 172.21.15.6:0/24164249 learned_addr learned my addr 172.21.15.6:0/24164249 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.361+0000 7ff241600640 1 --2- 172.21.15.6:0/24164249 >> v2:172.21.15.178:3300/0 conn(0x7ff23c15bf20 0x7ff23c0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995480810 0 0) 0x7ff23c072df0 con 0x7ff23c071e20 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff224003520 con 0x7ff23c071e20 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830912832 0 0) 0x7ff23c0cdab0 con 0x7ff23c072520 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff23c072df0 con 0x7ff23c072520 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 921315692 0 0) 0x7ff224003520 con 0x7ff23c071e20 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff241600640 1 -- 172.21.15.6:0/24164249 >> v1:172.21.15.178:6789/0 conn(0x7ff23c072520 legacy=0x7ff23c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff241600640 1 -- 172.21.15.6:0/24164249 >> v1:172.21.15.6:6789/0 conn(0x7ff23c071e20 legacy=0x7ff23c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.362+0000 7ff241600640 1 -- 172.21.15.6:0/24164249 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c0b5050 con 0x7ff23c15bf20 2024-03-06T07:27:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.363+0000 7ff241600640 1 --2- 172.21.15.6:0/24164249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c15bf20 0x7ff23c0acd40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2300097a0 tx=0x7ff23002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c0a999987ad652 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.364+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230038070 con 0x7ff23c15bf20 2024-03-06T07:27:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.364+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff230031500 con 0x7ff23c15bf20 2024-03-06T07:27:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.364+0000 7ff23be00640 1 -- 172.21.15.6:0/24164249 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2340019f0 con 0x7ff23c15bf20 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.364+0000 7ff244000640 1 -- 172.21.15.6:0/24164249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c15bf20 msgr2=0x7ff23c0acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.364+0000 7ff244000640 1 --2- 172.21.15.6:0/24164249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c15bf20 0x7ff23c0acd40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2300097a0 tx=0x7ff23002b820 comp rx=0 tx=0).stop 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.365+0000 7ff244000640 1 -- 172.21.15.6:0/24164249 shutdown_connections 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.365+0000 7ff244000640 1 --2- 172.21.15.6:0/24164249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c15bf20 0x7ff23c0acd40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.365+0000 7ff244000640 1 -- 172.21.15.6:0/24164249 >> 172.21.15.6:0/24164249 conn(0x7ff23c06b3d0 msgr2=0x7ff23c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.365+0000 7ff244000640 1 -- 172.21.15.6:0/24164249 shutdown_connections 2024-03-06T07:27:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.365+0000 7ff244000640 1 -- 172.21.15.6:0/24164249 wait complete. 2024-03-06T07:27:31.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.366+0000 7ff244000640 1 Processor -- start 2024-03-06T07:27:31.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.367+0000 7ff244000640 1 -- start start 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.367+0000 7ff244000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff23c071e20 0x7ff23c14d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.367+0000 7ff244000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c072520 0x7ff23c155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff244000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 0x7ff23c1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff244000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0b4b10 con 0x7ff23c15bf20 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff244000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0b4990 con 0x7ff23c072520 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff244000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0ce6c0 con 0x7ff23c071e20 2024-03-06T07:27:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 0x7ff23c1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 0x7ff23c1557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37542/0 (socket says 172.21.15.6:37542) 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 -- 172.21.15.6:0/1398232057 learned_addr learned my addr 172.21.15.6:0/1398232057 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff241600640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff23c071e20 0x7ff23c14d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff240c00640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c072520 0x7ff23c155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 -- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff23c071e20 msgr2=0x7ff23c14d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff23c071e20 0x7ff23c14d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 -- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c072520 msgr2=0x7ff23c155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c072520 0x7ff23c155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff242000640 1 -- 172.21.15.6:0/1398232057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c14f280 con 0x7ff23c15bf20 2024-03-06T07:27:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.368+0000 7ff240c00640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c072520 0x7ff23c155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.369+0000 7ff242000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 0x7ff23c1557d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff234001560 tx=0x7ff23400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.369+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234012400 con 0x7ff23c15bf20 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.369+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2340125a0 con 0x7ff23c15bf20 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.369+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2340129d0 con 0x7ff23c15bf20 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.369+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23c14f520 con 0x7ff23c15bf20 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.369+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23c14fae0 con 0x7ff23c15bf20 2024-03-06T07:27:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.370+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c04f330 con 0x7ff23c15bf20 2024-03-06T07:27:31.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.371+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff234012b70 con 0x7ff23c15bf20 2024-03-06T07:27:31.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.372+0000 7ff239600640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff21c06b4d0 0x7ff21c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.372+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff23408aab0 con 0x7ff23c15bf20 2024-03-06T07:27:31.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.373+0000 7ff241600640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff21c06b4d0 0x7ff21c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.374+0000 7ff241600640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff21c06b4d0 0x7ff21c06d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff230035000 tx=0x7ff230008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.377+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff234059890 con 0x7ff23c15bf20 2024-03-06T07:27:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:31.623+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff23c15f720 con 0x7ff23c15bf20 2024-03-06T07:27:32.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.245+0000 7ff239600640 1 -- 172.21.15.6:0/1398232057 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff234059190 con 0x7ff23c15bf20 2024-03-06T07:27:32.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff21c06b4d0 msgr2=0x7ff21c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff21c06b4d0 0x7ff21c06d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff230035000 tx=0x7ff230008030 comp rx=0 tx=0).stop 2024-03-06T07:27:32.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 msgr2=0x7ff23c1557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 0x7ff23c1557d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff234001560 tx=0x7ff23400a430 comp rx=0 tx=0).stop 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 shutdown_connections 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff21c06b4d0 0x7ff21c06d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff23c072520 0x7ff23c155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff23c15bf20 0x7ff23c1557d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 --2- 172.21.15.6:0/1398232057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff23c071e20 0x7ff23c14d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 >> 172.21.15.6:0/1398232057 conn(0x7ff23c06b3d0 msgr2=0x7ff23c0ad630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 shutdown_connections 2024-03-06T07:27:32.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.250+0000 7ff244000640 1 -- 172.21.15.6:0/1398232057 wait complete. 2024-03-06T07:27:32.274 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:32.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.494+0000 7f2645800640 1 Processor -- start 2024-03-06T07:27:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.494+0000 7f2645800640 1 -- start start 2024-03-06T07:27:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.496+0000 7f2645800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2640145430 0x7f264015f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.496+0000 7f2645800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f264015cc90 0x7f26400b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.496+0000 7f2645800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26400b3fe0 con 0x7f26400b2080 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.496+0000 7f2645800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26400b24d0 con 0x7f264015cc90 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.496+0000 7f2645800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264015fe40 con 0x7f2640145430 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f264015cc90 0x7f26400b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f264015cc90 0x7f26400b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42130/0 (socket says 172.21.15.6:42130) 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 -- 172.21.15.6:0/2942972949 learned_addr learned my addr 172.21.15.6:0/2942972949 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263fe00640 1 --2- 172.21.15.6:0/2942972949 >> v2:172.21.15.178:3300/0 conn(0x7f2640145430 0x7f264015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 -- 172.21.15.6:0/2942972949 >> v2:172.21.15.178:3300/0 conn(0x7f2640145430 msgr2=0x7f264015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 --2- 172.21.15.6:0/2942972949 >> v2:172.21.15.178:3300/0 conn(0x7f2640145430 0x7f264015f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 -- 172.21.15.6:0/2942972949 >> v1:172.21.15.178:6789/0 conn(0x7f26400b2080 legacy=0x7f264006c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.497+0000 7f263f400640 1 -- 172.21.15.6:0/2942972949 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640160120 con 0x7f264015cc90 2024-03-06T07:27:32.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.499+0000 7f263f400640 1 --2- 172.21.15.6:0/2942972949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f264015cc90 0x7f26400b1b40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f26400b2670 tx=0x7f262c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abb3b14a95cca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.499+0000 7f263ea00640 1 -- 172.21.15.6:0/2942972949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c034070 con 0x7f264015cc90 2024-03-06T07:27:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.499+0000 7f263ea00640 1 -- 172.21.15.6:0/2942972949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c030860 con 0x7f264015cc90 2024-03-06T07:27:32.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.499+0000 7f263ea00640 1 -- 172.21.15.6:0/2942972949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c030c30 con 0x7f264015cc90 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 -- 172.21.15.6:0/2942972949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f264015cc90 msgr2=0x7f26400b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 --2- 172.21.15.6:0/2942972949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f264015cc90 0x7f26400b1b40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f26400b2670 tx=0x7f262c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 -- 172.21.15.6:0/2942972949 shutdown_connections 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 --2- 172.21.15.6:0/2942972949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f264015cc90 0x7f26400b1b40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 --2- 172.21.15.6:0/2942972949 >> v2:172.21.15.178:3300/0 conn(0x7f2640145430 0x7f264015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 -- 172.21.15.6:0/2942972949 >> 172.21.15.6:0/2942972949 conn(0x7f264006b3a0 msgr2=0x7f264006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 -- 172.21.15.6:0/2942972949 shutdown_connections 2024-03-06T07:27:32.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.500+0000 7f2645800640 1 -- 172.21.15.6:0/2942972949 wait complete. 2024-03-06T07:27:32.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.502+0000 7f2645800640 1 Processor -- start 2024-03-06T07:27:32.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.502+0000 7f2645800640 1 -- start start 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2645800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 0x7f2640071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2645800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2640145430 0x7f26401413c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2645800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f264015cc90 0x7f264015c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2645800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264015d0d0 con 0x7f26400b2080 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2645800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26401606e0 con 0x7f2640145430 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2645800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2640145810 con 0x7f264015cc90 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 0x7f2640071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 0x7f2640071ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42144/0 (socket says 172.21.15.6:42144) 2024-03-06T07:27:32.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 -- 172.21.15.6:0/1684397851 learned_addr learned my addr 172.21.15.6:0/1684397851 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263f400640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2640145430 0x7f26401413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2644e00640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f264015cc90 0x7f264015c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 -- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f264015cc90 msgr2=0x7f264015c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f264015cc90 0x7f264015c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 -- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2640145430 msgr2=0x7f26401413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2640145430 0x7f26401413c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f263fe00640 1 -- 172.21.15.6:0/1684397851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2640141a50 con 0x7f26400b2080 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.503+0000 7f2644e00640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f264015cc90 0x7f264015c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.504+0000 7f263fe00640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 0x7f2640071ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f263000e9b0 tx=0x7f263000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.504+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630017070 con 0x7f26400b2080 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.504+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263000c9a0 con 0x7f26400b2080 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.504+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263000cd90 con 0x7f26400b2080 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.504+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2640141d30 con 0x7f26400b2080 2024-03-06T07:27:32.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.504+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26401422f0 con 0x7f26400b2080 2024-03-06T07:27:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.505+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264004f330 con 0x7f26400b2080 2024-03-06T07:27:32.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.506+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f263001a070 con 0x7f26400b2080 2024-03-06T07:27:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.507+0000 7f2637e00640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f262006b4d0 0x7f262006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.507+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f263008a680 con 0x7f26400b2080 2024-03-06T07:27:32.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.508+0000 7f263f400640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f262006b4d0 0x7f262006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.509+0000 7f263f400640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f262006b4d0 0x7f262006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f262c036000 tx=0x7f262c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.512+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2630059460 con 0x7f26400b2080 2024-03-06T07:27:32.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:32.751+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f264006b990 con 0x7f26400b2080 2024-03-06T07:27:33.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.248+0000 7f2637e00640 1 -- 172.21.15.6:0/1684397851 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2630058d60 con 0x7f26400b2080 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.252+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f262006b4d0 msgr2=0x7f262006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.252+0000 7f2645800640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f262006b4d0 0x7f262006d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f262c036000 tx=0x7f262c03b040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.252+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 msgr2=0x7f2640071ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.252+0000 7f2645800640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 0x7f2640071ef0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f263000e9b0 tx=0x7f263000ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 shutdown_connections 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f262006b4d0 0x7f262006d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2640145430 0x7f26401413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f264015cc90 0x7f264015c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 --2- 172.21.15.6:0/1684397851 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f26400b2080 0x7f2640071ef0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 >> 172.21.15.6:0/1684397851 conn(0x7f264006b3a0 msgr2=0x7f26400b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 shutdown_connections 2024-03-06T07:27:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.253+0000 7f2645800640 1 -- 172.21.15.6:0/1684397851 wait complete. 2024-03-06T07:27:33.274 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:33.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.491+0000 7fafef600640 1 Processor -- start 2024-03-06T07:27:33.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.491+0000 7fafef600640 1 -- start start 2024-03-06T07:27:33.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.492+0000 7fafef600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fafe81607e0 0x7fafe80d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.492+0000 7fafef600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.492+0000 7fafef600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe80ceac0 con 0x7fafe80d27d0 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.492+0000 7fafef600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe80cf690 con 0x7fafe81607e0 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.492+0000 7fafef600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80d0060 con 0x7fafe80b3f60 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.493+0000 7fafed600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.493+0000 7fafed600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47380/0 (socket says 172.21.15.6:47380) 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.493+0000 7fafed600640 1 -- 172.21.15.6:0/1056827870 learned_addr learned my addr 172.21.15.6:0/1056827870 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:33.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.493+0000 7fafecc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fafe81607e0 0x7fafe80d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.494+0000 7fafed600640 1 -- 172.21.15.6:0/1056827870 >> v2:172.21.15.178:3300/0 conn(0x7fafe81607e0 msgr2=0x7fafe80d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.494+0000 7fafed600640 1 --2- 172.21.15.6:0/1056827870 >> v2:172.21.15.178:3300/0 conn(0x7fafe81607e0 0x7fafe80d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.494+0000 7fafed600640 1 -- 172.21.15.6:0/1056827870 >> v1:172.21.15.178:6789/0 conn(0x7fafe80d27d0 legacy=0x7fafe80b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.494+0000 7fafed600640 1 -- 172.21.15.6:0/1056827870 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80b43f0 con 0x7fafe80b3f60 2024-03-06T07:27:33.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.494+0000 7fafecc00640 1 --2- 172.21.15.6:0/1056827870 >> v2:172.21.15.178:3300/0 conn(0x7fafe81607e0 0x7fafe80d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:33.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.495+0000 7fafed600640 1 --2- 172.21.15.6:0/1056827870 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cfb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009870 tx=0x7fafe002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39eddf9ef831bfae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.496+0000 7fafe7400640 1 -- 172.21.15.6:0/1056827870 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0034070 con 0x7fafe80b3f60 2024-03-06T07:27:33.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.496+0000 7fafe7400640 1 -- 172.21.15.6:0/1056827870 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe0030840 con 0x7fafe80b3f60 2024-03-06T07:27:33.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.496+0000 7fafe7400640 1 -- 172.21.15.6:0/1056827870 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0030c10 con 0x7fafe80b3f60 2024-03-06T07:27:33.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.496+0000 7fafef600640 1 -- 172.21.15.6:0/1056827870 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 msgr2=0x7fafe80cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.496+0000 7fafef600640 1 --2- 172.21.15.6:0/1056827870 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cfb20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009870 tx=0x7fafe002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.497+0000 7fafef600640 1 -- 172.21.15.6:0/1056827870 shutdown_connections 2024-03-06T07:27:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.497+0000 7fafef600640 1 --2- 172.21.15.6:0/1056827870 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cfb20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.497+0000 7fafef600640 1 --2- 172.21.15.6:0/1056827870 >> v2:172.21.15.178:3300/0 conn(0x7fafe81607e0 0x7fafe80d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.497+0000 7fafef600640 1 -- 172.21.15.6:0/1056827870 >> 172.21.15.6:0/1056827870 conn(0x7fafe80d3ef0 msgr2=0x7fafe80d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.497+0000 7fafef600640 1 -- 172.21.15.6:0/1056827870 shutdown_connections 2024-03-06T07:27:33.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.497+0000 7fafef600640 1 -- 172.21.15.6:0/1056827870 wait complete. 2024-03-06T07:27:33.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.499+0000 7fafef600640 1 Processor -- start 2024-03-06T07:27:33.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.499+0000 7fafef600640 1 -- start start 2024-03-06T07:27:33.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.500+0000 7fafef600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.500+0000 7fafef600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 0x7fafe81f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.500+0000 7fafef600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafe81607e0 0x7fafe80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.500+0000 7fafef600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80cf320 con 0x7fafe80d27d0 2024-03-06T07:27:33.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.500+0000 7fafef600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80cf040 con 0x7fafe81607e0 2024-03-06T07:27:33.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.500+0000 7fafef600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80d2c10 con 0x7fafe80b3f60 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 0x7fafe81f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 0x7fafe81f3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42156/0 (socket says 172.21.15.6:42156) 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 -- 172.21.15.6:0/4089629528 learned_addr learned my addr 172.21.15.6:0/4089629528 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafecc00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafed600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafe81607e0 0x7fafe80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 -- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 msgr2=0x7fafe80cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 -- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafe81607e0 msgr2=0x7fafe80cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafe81607e0 0x7fafe80cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.501+0000 7fafe7e00640 1 -- 172.21.15.6:0/4089629528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe81f3c50 con 0x7fafe80d27d0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.502+0000 7fafe7e00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 0x7fafe81f3710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fafd000c3c0 tx=0x7fafd000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.502+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd00052d0 con 0x7fafe80d27d0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.502+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd0005490 con 0x7fafe80d27d0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.502+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe80cbee0 con 0x7fafe80d27d0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.502+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0007260 con 0x7fafe80d27d0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.502+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe80cc470 con 0x7fafe80d27d0 2024-03-06T07:27:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.503+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe804f3a0 con 0x7fafe80d27d0 2024-03-06T07:27:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.504+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafd0007400 con 0x7fafe80d27d0 2024-03-06T07:27:33.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.505+0000 7fafe4c00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafc406b520 0x7fafc406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.505+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafd0089600 con 0x7fafe80d27d0 2024-03-06T07:27:33.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.505+0000 7fafecc00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafc406b520 0x7fafc406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.506+0000 7fafecc00640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafc406b520 0x7fafc406d9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fafe80c7520 tx=0x7fafd800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.509+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafd00584f0 con 0x7fafe80d27d0 2024-03-06T07:27:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.737+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafe8074ab0 con 0x7fafe80d27d0 2024-03-06T07:27:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.738+0000 7fafe4c00640 1 -- 172.21.15.6:0/4089629528 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fafd0057df0 con 0x7fafe80d27d0 2024-03-06T07:27:33.740 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:33.740 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:27:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:27:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafc406b520 msgr2=0x7fafc406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafc406b520 0x7fafc406d9e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fafe80c7520 tx=0x7fafd800b040 comp rx=0 tx=0).stop 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 msgr2=0x7fafe81f3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 0x7fafe81f3710 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fafd000c3c0 tx=0x7fafd000c880 comp rx=0 tx=0).stop 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 shutdown_connections 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafc406b520 0x7fafc406d9e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafe81607e0 0x7fafe80cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafe80d27d0 0x7fafe81f3710 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 --2- 172.21.15.6:0/4089629528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafe80b3f60 0x7fafe80cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.740+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 >> 172.21.15.6:0/4089629528 conn(0x7fafe80d3ef0 msgr2=0x7fafe80b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.741+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 shutdown_connections 2024-03-06T07:27:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.741+0000 7fafef600640 1 -- 172.21.15.6:0/4089629528 wait complete. 2024-03-06T07:27:33.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:33.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.978+0000 7f8310a00640 1 Processor -- start 2024-03-06T07:27:33.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.978+0000 7f8310a00640 1 -- start start 2024-03-06T07:27:33.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.980+0000 7f8310a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f830c145300 0x7f830c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.980+0000 7f8310a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0714d0 con 0x7f830c070f10 2024-03-06T07:27:33.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.980+0000 7f8310a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c0717b0 con 0x7f830c0cd060 2024-03-06T07:27:33.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.980+0000 7f8310a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c071a90 con 0x7f830c145300 2024-03-06T07:27:33.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.980+0000 7f830aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f830c070f10 0x7f830c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37420/0 (socket says 172.21.15.6:37420) 2024-03-06T07:27:33.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.980+0000 7f830aa00640 1 -- 172.21.15.6:0/1547261353 learned_addr learned my addr 172.21.15.6:0/1547261353 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:33.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830b400640 1 --2- 172.21.15.6:0/1547261353 >> v2:172.21.15.178:3300/0 conn(0x7f830c145300 0x7f830c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209406134 0 0) 0x7f830c0714d0 con 0x7f830c070f10 2024-03-06T07:27:33.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0003520 con 0x7f830c070f10 2024-03-06T07:27:33.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2240284806 0 0) 0x7f82f0003520 con 0x7f830c070f10 2024-03-06T07:27:33.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 >> v2:172.21.15.178:3300/0 conn(0x7f830c145300 msgr2=0x7f830c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 --2- 172.21.15.6:0/1547261353 >> v2:172.21.15.178:3300/0 conn(0x7f830c145300 0x7f830c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 >> v1:172.21.15.178:6789/0 conn(0x7f830c0cd060 legacy=0x7f830c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c071d70 con 0x7f830c070f10 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f82f8001670 con 0x7f830c070f10 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 >> v1:172.21.15.6:6789/0 conn(0x7f830c070f10 legacy=0x7f830c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.981+0000 7f830a000640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f00039b0 0x7f82f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830a000640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82f0004800 0x7f82f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830a000640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82f00051c0 0x7f82f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c071a90 con 0x7f82f0004800 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f000c160 con 0x7f82f00039b0 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f000c360 con 0x7f82f00051c0 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830aa00640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82f0004800 0x7f82f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830b400640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f00039b0 0x7f82f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830aa00640 1 -- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82f00051c0 msgr2=0x7f82f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830aa00640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82f00051c0 0x7f82f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830aa00640 1 -- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f00039b0 msgr2=0x7f82f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:33.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830aa00640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f00039b0 0x7f82f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830aa00640 1 -- 172.21.15.6:0/1547261353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0003520 con 0x7f82f0004800 2024-03-06T07:27:33.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.982+0000 7f830be00640 1 -- 172.21.15.6:0/1547261353 reap_dead start 2024-03-06T07:27:33.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.983+0000 7f830aa00640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82f0004800 0x7f82f0004c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c240 tx=0x7f82f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9445925475ffed8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.983+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8007d10 con 0x7f82f0004800 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.983+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8030070 con 0x7f82f0004800 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.983+0000 7f830a000640 1 -- 172.21.15.6:0/1547261353 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f802cda0 con 0x7f82f0004800 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 -- 172.21.15.6:0/1547261353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82f0004800 msgr2=0x7f82f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82f0004800 0x7f82f0004c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c240 tx=0x7f82f0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 -- 172.21.15.6:0/1547261353 shutdown_connections 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 --2- 172.21.15.6:0/1547261353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82f0004800 0x7f82f0004c80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 -- 172.21.15.6:0/1547261353 >> 172.21.15.6:0/1547261353 conn(0x7f830c06b3d0 msgr2=0x7f830c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 -- 172.21.15.6:0/1547261353 shutdown_connections 2024-03-06T07:27:33.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.984+0000 7f8310a00640 1 -- 172.21.15.6:0/1547261353 wait complete. 2024-03-06T07:27:33.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.986+0000 7f8310a00640 1 Processor -- start 2024-03-06T07:27:33.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.986+0000 7f8310a00640 1 -- start start 2024-03-06T07:27:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f8310a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f0004800 0x7f830c136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f8310a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 0x7f830c0b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f8310a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f830c1e7920 0x7f830c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f8310a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0ccad0 con 0x7f830c1433d0 2024-03-06T07:27:33.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f8310a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0cc950 con 0x7f82f0004800 2024-03-06T07:27:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f8310a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0ccc50 con 0x7f830c1e7920 2024-03-06T07:27:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f830aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 0x7f830c0b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f830aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 0x7f830c0b0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42166/0 (socket says 172.21.15.6:42166) 2024-03-06T07:27:33.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.987+0000 7f830aa00640 1 -- 172.21.15.6:0/3430465163 learned_addr learned my addr 172.21.15.6:0/3430465163 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830be00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f830c1e7920 0x7f830c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830b400640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f0004800 0x7f830c136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830aa00640 1 -- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f830c1e7920 msgr2=0x7f830c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830aa00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f830c1e7920 0x7f830c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830aa00640 1 -- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f0004800 msgr2=0x7f830c136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830aa00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f0004800 0x7f830c136380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830aa00640 1 -- 172.21.15.6:0/3430465163 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c1e6880 con 0x7f830c1433d0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f830aa00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 0x7f830c0b0630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c240 tx=0x7f82f802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8007d10 con 0x7f830c1433d0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f8004bf0 con 0x7f830c1433d0 2024-03-06T07:27:33.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.988+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8007070 con 0x7f830c1433d0 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.989+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c1e6b60 con 0x7f830c1433d0 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.989+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c1e7120 con 0x7f830c1433d0 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.990+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c04f330 con 0x7f830c1433d0 2024-03-06T07:27:33.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.991+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82f8048020 con 0x7f830c1433d0 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.992+0000 7f8303400640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82ec06b4d0 0x7f82ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:33.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.992+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82f80ab780 con 0x7f830c1433d0 2024-03-06T07:27:33.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.992+0000 7f830b400640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82ec06b4d0 0x7f82ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.993+0000 7f830b400640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82ec06b4d0 0x7f82ec06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f830c0b2170 tx=0x7f82fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:33.996+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82f807a5f0 con 0x7f830c1433d0 2024-03-06T07:27:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.198+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f830c144960 con 0x7f830c1433d0 2024-03-06T07:27:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.201+0000 7f8303400640 1 -- 172.21.15.6:0/3430465163 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f82f8079ef0 con 0x7f830c1433d0 2024-03-06T07:27:34.204 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:34.204 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82ec06b4d0 msgr2=0x7f82ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82ec06b4d0 0x7f82ec06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f830c0b2170 tx=0x7f82fc008040 comp rx=0 tx=0).stop 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 msgr2=0x7f830c0b0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 0x7f830c0b0630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f82f800c240 tx=0x7f82f802e070 comp rx=0 tx=0).stop 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 shutdown_connections 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82ec06b4d0 0x7f82ec06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f830c1e7920 0x7f830c0cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f830c1433d0 0x7f830c0b0630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 --2- 172.21.15.6:0/3430465163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82f0004800 0x7f830c136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 >> 172.21.15.6:0/3430465163 conn(0x7f830c06b3d0 msgr2=0x7f830c0b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 shutdown_connections 2024-03-06T07:27:34.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.204+0000 7f8310a00640 1 -- 172.21.15.6:0/3430465163 wait complete. 2024-03-06T07:27:34.227 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:34.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:34.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.444+0000 7fbd43e00640 1 Processor -- start 2024-03-06T07:27:34.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.444+0000 7fbd43e00640 1 -- start start 2024-03-06T07:27:34.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.446+0000 7fbd43e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 0x7fbd4415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.446+0000 7fbd43e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4415ed10 con 0x7fbd441552b0 2024-03-06T07:27:34.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.446+0000 7fbd43e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4415eff0 con 0x7fbd4415e810 2024-03-06T07:27:34.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.446+0000 7fbd43e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4415f2d0 con 0x7fbd440cd060 2024-03-06T07:27:34.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.446+0000 7fbd42a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd441552b0 0x7fbd440cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37426/0 (socket says 172.21.15.6:37426) 2024-03-06T07:27:34.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.446+0000 7fbd42a00640 1 -- 172.21.15.6:0/49516671 learned_addr learned my addr 172.21.15.6:0/49516671 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd3a000640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 0x7fbd4415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612553544 0 0) 0x7fbd4415ed10 con 0x7fbd441552b0 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd24003520 con 0x7fbd441552b0 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251184154 0 0) 0x7fbd24003520 con 0x7fbd441552b0 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 msgr2=0x7fbd4415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 0x7fbd4415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 >> v1:172.21.15.178:6789/0 conn(0x7fbd4415e810 legacy=0x7fbd440b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4415f5b0 con 0x7fbd441552b0 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fbd34001670 con 0x7fbd441552b0 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 >> v1:172.21.15.6:6789/0 conn(0x7fbd441552b0 legacy=0x7fbd440cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.447+0000 7fbd42000640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 0x7fbd240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd42000640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd24003f60 0x7fbd24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd42000640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd24004d40 0x7fbd240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4415f2d0 con 0x7fbd24004d40 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2400bb50 con 0x7fbd24003f60 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2400df60 con 0x7fbd440cd060 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd43400640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd24004d40 0x7fbd240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd42a00640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 0x7fbd240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd3a000640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd24003f60 0x7fbd24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd43400640 1 -- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 msgr2=0x7fbd240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd43400640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd440cd060 0x7fbd240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd43400640 1 -- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd24003f60 msgr2=0x7fbd24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd43400640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd24003f60 0x7fbd24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.448+0000 7fbd43400640 1 -- 172.21.15.6:0/49516671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24003520 con 0x7fbd24004d40 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.449+0000 7fbd43400640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd24004d40 0x7fbd240051c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbd4415ec40 tx=0x7fbd24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcbd55dd71d3286 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c001a70 con 0x7fbd24004d40 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c003b70 con 0x7fbd24004d40 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd42000640 1 -- 172.21.15.6:0/49516671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c0050d0 con 0x7fbd24004d40 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43e00640 1 -- 172.21.15.6:0/49516671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd24004d40 msgr2=0x7fbd240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43e00640 1 --2- 172.21.15.6:0/49516671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd24004d40 0x7fbd240051c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbd4415ec40 tx=0x7fbd24005830 comp rx=0 tx=0).stop 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43400640 1 -- 172.21.15.6:0/49516671 reap_dead start 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43e00640 1 -- 172.21.15.6:0/49516671 shutdown_connections 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43e00640 1 -- 172.21.15.6:0/49516671 >> 172.21.15.6:0/49516671 conn(0x7fbd4406b3d0 msgr2=0x7fbd4406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43e00640 1 -- 172.21.15.6:0/49516671 shutdown_connections 2024-03-06T07:27:34.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.450+0000 7fbd43e00640 1 -- 172.21.15.6:0/49516671 wait complete. 2024-03-06T07:27:34.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.452+0000 7fbd43e00640 1 Processor -- start 2024-03-06T07:27:34.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.453+0000 7fbd43e00640 1 -- start start 2024-03-06T07:27:34.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd44143450 0x7fbd4414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 0x7fbd4415d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd44152e00 0x7fbd44153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4406dd30 con 0x7fbd4415cb90 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440b3760 con 0x7fbd44152e00 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4406deb0 con 0x7fbd44143450 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 0x7fbd4415d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 0x7fbd4415d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42186/0 (socket says 172.21.15.6:42186) 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 -- 172.21.15.6:0/1361010313 learned_addr learned my addr 172.21.15.6:0/1361010313 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd42a00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd44143450 0x7fbd4414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 -- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd44143450 msgr2=0x7fbd4414f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd44143450 0x7fbd4414f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd43400640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd44152e00 0x7fbd44153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 -- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd44152e00 msgr2=0x7fbd44153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd44152e00 0x7fbd44153280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.454+0000 7fbd3a000640 1 -- 172.21.15.6:0/1361010313 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd44143830 con 0x7fbd4415cb90 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.455+0000 7fbd3a000640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 0x7fbd4415d010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c011630 tx=0x7fbd2c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.455+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00f8c0 con 0x7fbd4415cb90 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.455+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c00fa60 con 0x7fbd4415cb90 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.455+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00fe10 con 0x7fbd4415cb90 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.455+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4414e100 con 0x7fbd4415cb90 2024-03-06T07:27:34.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.455+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4414e6c0 con 0x7fbd4415cb90 2024-03-06T07:27:34.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.457+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4404f330 con 0x7fbd4415cb90 2024-03-06T07:27:34.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.458+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd2c015460 con 0x7fbd4415cb90 2024-03-06T07:27:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.459+0000 7fbd3be00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.459+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd2c08ab00 con 0x7fbd4415cb90 2024-03-06T07:27:34.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.459+0000 7fbd42a00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.461+0000 7fbd42a00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd34001eb0 tx=0x7fbd34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.464+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd2c019030 con 0x7fbd4415cb90 2024-03-06T07:27:34.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.659+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd440cf4a0 con 0x7fbd2006b500 2024-03-06T07:27:34.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.699+0000 7fbd3be00640 1 -- 172.21.15.6:0/1361010313 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251903 (secure 0 0 0) 0x7fbd440cf4a0 con 0x7fbd2006b500 2024-03-06T07:27:34.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:34.706 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.703+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 msgr2=0x7fbd2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.703+0000 7fbd43e00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbd34001eb0 tx=0x7fbd34009040 comp rx=0 tx=0).stop 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.703+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 msgr2=0x7fbd4415d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.703+0000 7fbd43e00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 0x7fbd4415d010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c011630 tx=0x7fbd2c011af0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 shutdown_connections 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd44152e00 0x7fbd44153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4415cb90 0x7fbd4415d010 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 --2- 172.21.15.6:0/1361010313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd44143450 0x7fbd4414f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 >> 172.21.15.6:0/1361010313 conn(0x7fbd4406b3d0 msgr2=0x7fbd440cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 shutdown_connections 2024-03-06T07:27:34.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.704+0000 7fbd43e00640 1 -- 172.21.15.6:0/1361010313 wait complete. 2024-03-06T07:27:34.724 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:27:33.437845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235700,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2949120,"data_stored":1772764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.254847"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.250766+0000","last_change":"2024-03-06T07:27:30.239373+0000","last_active":"2024-03-06T07:27:30.250766+0000","last_peered":"2024-03-06T07:27:30.250766+0000","last_clean":"2024-03-06T07:27:30.250766+0000","last_became_active":"2024-03-06T07:27:30.239046+0000","last_became_peered":"2024-03-06T07:27:30.239046+0000","last_unstale":"2024-03-06T07:27:30.250766+0000","last_undegraded":"2024-03-06T07:27:30.250766+0000","last_fullsized":"2024-03-06T07:27:30.250766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:58.456356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239467+0000","last_change":"2024-03-06T07:27:30.234823+0000","last_active":"2024-03-06T07:27:31.239467+0000","last_peered":"2024-03-06T07:27:31.239467+0000","last_clean":"2024-03-06T07:27:31.239467+0000","last_became_active":"2024-03-06T07:27:30.234521+0000","last_became_peered":"2024-03-06T07:27:30.234521+0000","last_unstale":"2024-03-06T07:27:31.239467+0000","last_undegraded":"2024-03-06T07:27:31.239467+0000","last_fullsized":"2024-03-06T07:27:31.239467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:17.318504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.247178+0000","last_change":"2024-03-06T07:27:30.244772+0000","last_active":"2024-03-06T07:27:30.247178+0000","last_peered":"2024-03-06T07:27:30.247178+0000","last_clean":"2024-03-06T07:27:30.247178+0000","last_became_active":"2024-03-06T07:27:30.244401+0000","last_became_peered":"2024-03-06T07:27:30.244401+0000","last_unstale":"2024-03-06T07:27:30.247178+0000","last_undegraded":"2024-03-06T07:27:30.247178+0000","last_fullsized":"2024-03-06T07:27:30.247178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:22.855579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269899+0000","last_change":"2024-03-06T07:27:30.260180+0000","last_active":"2024-03-06T07:27:30.269899+0000","last_peered":"2024-03-06T07:27:30.269899+0000","last_clean":"2024-03-06T07:27:30.269899+0000","last_became_active":"2024-03-06T07:27:30.259183+0000","last_became_peered":"2024-03-06T07:27:30.259183+0000","last_unstale":"2024-03-06T07:27:30.269899+0000","last_undegraded":"2024-03-06T07:27:30.269899+0000","last_fullsized":"2024-03-06T07:27:30.269899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:48.495220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.260632+0000","last_change":"2024-03-06T07:27:30.254425+0000","last_active":"2024-03-06T07:27:30.260632+0000","last_peered":"2024-03-06T07:27:30.260632+0000","last_clean":"2024-03-06T07:27:30.260632+0000","last_became_active":"2024-03-06T07:27:30.253238+0000","last_became_peered":"2024-03-06T07:27:30.253238+0000","last_unstale":"2024-03-06T07:27:30.260632+0000","last_undegraded":"2024-03-06T07:27:30.260632+0000","last_fullsized":"2024-03-06T07:27:30.260632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:02.502387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269776+0000","last_change":"2024-03-06T07:27:30.259178+0000","last_active":"2024-03-06T07:27:30.269776+0000","last_peered":"2024-03-06T07:27:30.269776+0000","last_clean":"2024-03-06T07:27:30.269776+0000","last_became_active":"2024-03-06T07:27:30.258928+0000","last_became_peered":"2024-03-06T07:27:30.258928+0000","last_unstale":"2024-03-06T07:27:30.269776+0000","last_undegraded":"2024-03-06T07:27:30.269776+0000","last_fullsized":"2024-03-06T07:27:30.269776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:54.202774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.252318+0000","last_change":"2024-03-06T07:27:30.244331+0000","last_active":"2024-03-06T07:27:30.252318+0000","last_peered":"2024-03-06T07:27:30.252318+0000","last_clean":"2024-03-06T07:27:30.252318+0000","last_became_active":"2024-03-06T07:27:30.244078+0000","last_became_peered":"2024-03-06T07:27:30.244078+0000","last_unstale":"2024-03-06T07:27:30.252318+0000","last_undegraded":"2024-03-06T07:27:30.252318+0000","last_fullsized":"2024-03-06T07:27:30.252318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:50.573971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241726+0000","last_change":"2024-03-06T07:27:30.256607+0000","last_active":"2024-03-06T07:27:31.241726+0000","last_peered":"2024-03-06T07:27:31.241726+0000","last_clean":"2024-03-06T07:27:31.241726+0000","last_became_active":"2024-03-06T07:27:30.256321+0000","last_became_peered":"2024-03-06T07:27:30.256321+0000","last_unstale":"2024-03-06T07:27:31.241726+0000","last_undegraded":"2024-03-06T07:27:31.241726+0000","last_fullsized":"2024-03-06T07:27:31.241726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262343+0000","last_change":"2024-03-06T07:27:30.260973+0000","last_active":"2024-03-06T07:27:30.262343+0000","last_peered":"2024-03-06T07:27:30.262343+0000","last_clean":"2024-03-06T07:27:30.262343+0000","last_became_active":"2024-03-06T07:27:30.260401+0000","last_became_peered":"2024-03-06T07:27:30.260401+0000","last_unstale":"2024-03-06T07:27:30.262343+0000","last_undegraded":"2024-03-06T07:27:30.262343+0000","last_fullsized":"2024-03-06T07:27:30.262343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:18.053835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243083+0000","last_change":"2024-03-06T07:27:30.247832+0000","last_active":"2024-03-06T07:27:31.243083+0000","last_peered":"2024-03-06T07:27:31.243083+0000","last_clean":"2024-03-06T07:27:31.243083+0000","last_became_active":"2024-03-06T07:27:30.247503+0000","last_became_peered":"2024-03-06T07:27:30.247503+0000","last_unstale":"2024-03-06T07:27:31.243083+0000","last_undegraded":"2024-03-06T07:27:31.243083+0000","last_fullsized":"2024-03-06T07:27:31.243083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242309+0000","last_change":"2024-03-06T07:27:30.256946+0000","last_active":"2024-03-06T07:27:31.242309+0000","last_peered":"2024-03-06T07:27:31.242309+0000","last_clean":"2024-03-06T07:27:31.242309+0000","last_became_active":"2024-03-06T07:27:30.256643+0000","last_became_peered":"2024-03-06T07:27:30.256643+0000","last_unstale":"2024-03-06T07:27:31.242309+0000","last_undegraded":"2024-03-06T07:27:31.242309+0000","last_fullsized":"2024-03-06T07:27:31.242309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262105+0000","last_change":"2024-03-06T07:27:30.248076+0000","last_active":"2024-03-06T07:27:30.262105+0000","last_peered":"2024-03-06T07:27:30.262105+0000","last_clean":"2024-03-06T07:27:30.262105+0000","last_became_active":"2024-03-06T07:27:30.247733+0000","last_became_peered":"2024-03-06T07:27:30.247733+0000","last_unstale":"2024-03-06T07:27:30.262105+0000","last_undegraded":"2024-03-06T07:27:30.262105+0000","last_fullsized":"2024-03-06T07:27:30.262105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:19.764301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243474+0000","last_change":"2024-03-06T07:27:30.238620+0000","last_active":"2024-03-06T07:27:31.243474+0000","last_peered":"2024-03-06T07:27:31.243474+0000","last_clean":"2024-03-06T07:27:31.243474+0000","last_became_active":"2024-03-06T07:27:30.238357+0000","last_became_peered":"2024-03-06T07:27:30.238357+0000","last_unstale":"2024-03-06T07:27:31.243474+0000","last_undegraded":"2024-03-06T07:27:31.243474+0000","last_fullsized":"2024-03-06T07:27:31.243474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.245543+0000","last_change":"2024-03-06T07:27:30.242346+0000","last_active":"2024-03-06T07:27:30.245543+0000","last_peered":"2024-03-06T07:27:30.245543+0000","last_clean":"2024-03-06T07:27:30.245543+0000","last_became_active":"2024-03-06T07:27:30.242081+0000","last_became_peered":"2024-03-06T07:27:30.242081+0000","last_unstale":"2024-03-06T07:27:30.245543+0000","last_undegraded":"2024-03-06T07:27:30.245543+0000","last_fullsized":"2024-03-06T07:27:30.245543+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:46.777176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239135+0000","last_change":"2024-03-06T07:27:30.252254+0000","last_active":"2024-03-06T07:27:31.239135+0000","last_peered":"2024-03-06T07:27:31.239135+0000","last_clean":"2024-03-06T07:27:31.239135+0000","last_became_active":"2024-03-06T07:27:30.251915+0000","last_became_peered":"2024-03-06T07:27:30.251915+0000","last_unstale":"2024-03-06T07:27:31.239135+0000","last_undegraded":"2024-03-06T07:27:31.239135+0000","last_fullsized":"2024-03-06T07:27:31.239135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:42.209474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270906+0000","last_change":"2024-03-06T07:27:30.266564+0000","last_active":"2024-03-06T07:27:30.270906+0000","last_peered":"2024-03-06T07:27:30.270906+0000","last_clean":"2024-03-06T07:27:30.270906+0000","last_became_active":"2024-03-06T07:27:30.265165+0000","last_became_peered":"2024-03-06T07:27:30.265165+0000","last_unstale":"2024-03-06T07:27:30.270906+0000","last_undegraded":"2024-03-06T07:27:30.270906+0000","last_fullsized":"2024-03-06T07:27:30.270906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270924+0000","last_change":"2024-03-06T07:27:30.266828+0000","last_active":"2024-03-06T07:27:30.270924+0000","last_peered":"2024-03-06T07:27:30.270924+0000","last_clean":"2024-03-06T07:27:30.270924+0000","last_became_active":"2024-03-06T07:27:30.265102+0000","last_became_peered":"2024-03-06T07:27:30.265102+0000","last_unstale":"2024-03-06T07:27:30.270924+0000","last_undegraded":"2024-03-06T07:27:30.270924+0000","last_fullsized":"2024-03-06T07:27:30.270924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695842+0000","last_change":"2024-03-06T07:27:30.239978+0000","last_active":"2024-03-06T07:27:31.695842+0000","last_peered":"2024-03-06T07:27:31.695842+0000","last_clean":"2024-03-06T07:27:31.695842+0000","last_became_active":"2024-03-06T07:27:30.239682+0000","last_became_peered":"2024-03-06T07:27:30.239682+0000","last_unstale":"2024-03-06T07:27:31.695842+0000","last_undegraded":"2024-03-06T07:27:31.695842+0000","last_fullsized":"2024-03-06T07:27:31.695842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:46.446447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266214+0000","last_change":"2024-03-06T07:27:30.245372+0000","last_active":"2024-03-06T07:27:30.266214+0000","last_peered":"2024-03-06T07:27:30.266214+0000","last_clean":"2024-03-06T07:27:30.266214+0000","last_became_active":"2024-03-06T07:27:30.245096+0000","last_became_peered":"2024-03-06T07:27:30.245096+0000","last_unstale":"2024-03-06T07:27:30.266214+0000","last_undegraded":"2024-03-06T07:27:30.266214+0000","last_fullsized":"2024-03-06T07:27:30.266214+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:35.043873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242869+0000","last_change":"2024-03-06T07:27:30.244785+0000","last_active":"2024-03-06T07:27:31.242869+0000","last_peered":"2024-03-06T07:27:31.242869+0000","last_clean":"2024-03-06T07:27:31.242869+0000","last_became_active":"2024-03-06T07:27:30.244505+0000","last_became_peered":"2024-03-06T07:27:30.244505+0000","last_unstale":"2024-03-06T07:27:31.242869+0000","last_undegraded":"2024-03-06T07:27:31.242869+0000","last_fullsized":"2024-03-06T07:27:31.242869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.257896+0000","last_change":"2024-03-06T07:27:30.253741+0000","last_active":"2024-03-06T07:27:30.257896+0000","last_peered":"2024-03-06T07:27:30.257896+0000","last_clean":"2024-03-06T07:27:30.257896+0000","last_became_active":"2024-03-06T07:27:30.251912+0000","last_became_peered":"2024-03-06T07:27:30.251912+0000","last_unstale":"2024-03-06T07:27:30.257896+0000","last_undegraded":"2024-03-06T07:27:30.257896+0000","last_fullsized":"2024-03-06T07:27:30.257896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:08.043724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.254285+0000","last_change":"2024-03-06T07:27:30.240983+0000","last_active":"2024-03-06T07:27:30.254285+0000","last_peered":"2024-03-06T07:27:30.254285+0000","last_clean":"2024-03-06T07:27:30.254285+0000","last_became_active":"2024-03-06T07:27:30.240088+0000","last_became_peered":"2024-03-06T07:27:30.240088+0000","last_unstale":"2024-03-06T07:27:30.254285+0000","last_undegraded":"2024-03-06T07:27:30.254285+0000","last_fullsized":"2024-03-06T07:27:30.254285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:08.043724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241714+0000","last_change":"2024-03-06T07:27:30.254933+0000","last_active":"2024-03-06T07:27:31.241714+0000","last_peered":"2024-03-06T07:27:31.241714+0000","last_clean":"2024-03-06T07:27:31.241714+0000","last_became_active":"2024-03-06T07:27:30.253777+0000","last_became_peered":"2024-03-06T07:27:30.253777+0000","last_unstale":"2024-03-06T07:27:31.241714+0000","last_undegraded":"2024-03-06T07:27:31.241714+0000","last_fullsized":"2024-03-06T07:27:31.241714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266696+0000","last_change":"2024-03-06T07:27:30.254617+0000","last_active":"2024-03-06T07:27:30.266696+0000","last_peered":"2024-03-06T07:27:30.266696+0000","last_clean":"2024-03-06T07:27:30.266696+0000","last_became_active":"2024-03-06T07:27:30.253826+0000","last_became_peered":"2024-03-06T07:27:30.253826+0000","last_unstale":"2024-03-06T07:27:30.266696+0000","last_undegraded":"2024-03-06T07:27:30.266696+0000","last_fullsized":"2024-03-06T07:27:30.266696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:49.587341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243679+0000","last_change":"2024-03-06T07:27:30.245625+0000","last_active":"2024-03-06T07:27:31.243679+0000","last_peered":"2024-03-06T07:27:31.243679+0000","last_clean":"2024-03-06T07:27:31.243679+0000","last_became_active":"2024-03-06T07:27:30.245310+0000","last_became_peered":"2024-03-06T07:27:30.245310+0000","last_unstale":"2024-03-06T07:27:31.243679+0000","last_undegraded":"2024-03-06T07:27:31.243679+0000","last_fullsized":"2024-03-06T07:27:31.243679+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.246068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243256+0000","last_change":"2024-03-06T07:27:30.241785+0000","last_active":"2024-03-06T07:27:31.243256+0000","last_peered":"2024-03-06T07:27:31.243256+0000","last_clean":"2024-03-06T07:27:31.243256+0000","last_became_active":"2024-03-06T07:27:30.241456+0000","last_became_peered":"2024-03-06T07:27:30.241456+0000","last_unstale":"2024-03-06T07:27:31.243256+0000","last_undegraded":"2024-03-06T07:27:31.243256+0000","last_fullsized":"2024-03-06T07:27:31.243256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239562+0000","last_change":"2024-03-06T07:27:30.239090+0000","last_active":"2024-03-06T07:27:31.239562+0000","last_peered":"2024-03-06T07:27:31.239562+0000","last_clean":"2024-03-06T07:27:31.239562+0000","last_became_active":"2024-03-06T07:27:30.238809+0000","last_became_peered":"2024-03-06T07:27:30.238809+0000","last_unstale":"2024-03-06T07:27:31.239562+0000","last_undegraded":"2024-03-06T07:27:31.239562+0000","last_fullsized":"2024-03-06T07:27:31.239562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:18.053835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243221+0000","last_change":"2024-03-06T07:27:30.250473+0000","last_active":"2024-03-06T07:27:31.243221+0000","last_peered":"2024-03-06T07:27:31.243221+0000","last_clean":"2024-03-06T07:27:31.243221+0000","last_became_active":"2024-03-06T07:27:30.250084+0000","last_became_peered":"2024-03-06T07:27:30.250084+0000","last_unstale":"2024-03-06T07:27:31.243221+0000","last_undegraded":"2024-03-06T07:27:31.243221+0000","last_fullsized":"2024-03-06T07:27:31.243221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.257877+0000","last_change":"2024-03-06T07:27:30.247866+0000","last_active":"2024-03-06T07:27:30.257877+0000","last_peered":"2024-03-06T07:27:30.257877+0000","last_clean":"2024-03-06T07:27:30.257877+0000","last_became_active":"2024-03-06T07:27:30.247526+0000","last_became_peered":"2024-03-06T07:27:30.247526+0000","last_unstale":"2024-03-06T07:27:30.257877+0000","last_undegraded":"2024-03-06T07:27:30.257877+0000","last_fullsized":"2024-03-06T07:27:30.257877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:47.912605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.255793+0000","last_change":"2024-03-06T07:27:30.252736+0000","last_active":"2024-03-06T07:27:30.255793+0000","last_peered":"2024-03-06T07:27:30.255793+0000","last_clean":"2024-03-06T07:27:30.255793+0000","last_became_active":"2024-03-06T07:27:30.252267+0000","last_became_peered":"2024-03-06T07:27:30.252267+0000","last_unstale":"2024-03-06T07:27:30.255793+0000","last_undegraded":"2024-03-06T07:27:30.255793+0000","last_fullsized":"2024-03-06T07:27:30.255793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:52.878333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695816+0000","last_change":"2024-03-06T07:27:30.265689+0000","last_active":"2024-03-06T07:27:31.695816+0000","last_peered":"2024-03-06T07:27:31.695816+0000","last_clean":"2024-03-06T07:27:31.695816+0000","last_became_active":"2024-03-06T07:27:30.265410+0000","last_became_peered":"2024-03-06T07:27:30.265410+0000","last_unstale":"2024-03-06T07:27:31.695816+0000","last_undegraded":"2024-03-06T07:27:31.695816+0000","last_fullsized":"2024-03-06T07:27:31.695816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:00.010236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242466+0000","last_change":"2024-03-06T07:27:30.238867+0000","last_active":"2024-03-06T07:27:31.242466+0000","last_peered":"2024-03-06T07:27:31.242466+0000","last_clean":"2024-03-06T07:27:31.242466+0000","last_became_active":"2024-03-06T07:27:30.238543+0000","last_became_peered":"2024-03-06T07:27:30.238543+0000","last_unstale":"2024-03-06T07:27:31.242466+0000","last_undegraded":"2024-03-06T07:27:31.242466+0000","last_fullsized":"2024-03-06T07:27:31.242466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.260931+0000","last_change":"2024-03-06T07:27:30.244526+0000","last_active":"2024-03-06T07:27:30.260931+0000","last_peered":"2024-03-06T07:27:30.260931+0000","last_clean":"2024-03-06T07:27:30.260931+0000","last_became_active":"2024-03-06T07:27:30.244266+0000","last_became_peered":"2024-03-06T07:27:30.244266+0000","last_unstale":"2024-03-06T07:27:30.260931+0000","last_undegraded":"2024-03-06T07:27:30.260931+0000","last_fullsized":"2024-03-06T07:27:30.260931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:15.978688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242632+0000","last_change":"2024-03-06T07:27:29.224405+0000","last_active":"2024-03-06T07:27:31.242632+0000","last_peered":"2024-03-06T07:27:31.242632+0000","last_clean":"2024-03-06T07:27:31.242632+0000","last_became_active":"2024-03-06T07:27:29.223485+0000","last_became_peered":"2024-03-06T07:27:29.223485+0000","last_unstale":"2024-03-06T07:27:31.242632+0000","last_undegraded":"2024-03-06T07:27:31.242632+0000","last_fullsized":"2024-03-06T07:27:31.242632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.257854+0000","last_change":"2024-03-06T07:27:30.249021+0000","last_active":"2024-03-06T07:27:30.257854+0000","last_peered":"2024-03-06T07:27:30.257854+0000","last_clean":"2024-03-06T07:27:30.257854+0000","last_became_active":"2024-03-06T07:27:30.248148+0000","last_became_peered":"2024-03-06T07:27:30.248148+0000","last_unstale":"2024-03-06T07:27:30.257854+0000","last_undegraded":"2024-03-06T07:27:30.257854+0000","last_fullsized":"2024-03-06T07:27:30.257854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:48.493765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243028+0000","last_change":"2024-03-06T07:27:29.230274+0000","last_active":"2024-03-06T07:27:31.243028+0000","last_peered":"2024-03-06T07:27:31.243028+0000","last_clean":"2024-03-06T07:27:31.243028+0000","last_became_active":"2024-03-06T07:27:29.230009+0000","last_became_peered":"2024-03-06T07:27:29.230009+0000","last_unstale":"2024-03-06T07:27:31.243028+0000","last_undegraded":"2024-03-06T07:27:31.243028+0000","last_fullsized":"2024-03-06T07:27:31.243028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241766+0000","last_change":"2024-03-06T07:27:29.221872+0000","last_active":"2024-03-06T07:27:31.241766+0000","last_peered":"2024-03-06T07:27:31.241766+0000","last_clean":"2024-03-06T07:27:31.241766+0000","last_became_active":"2024-03-06T07:27:29.221430+0000","last_became_peered":"2024-03-06T07:27:29.221430+0000","last_unstale":"2024-03-06T07:27:31.241766+0000","last_undegraded":"2024-03-06T07:27:31.241766+0000","last_fullsized":"2024-03-06T07:27:31.241766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241901+0000","last_change":"2024-03-06T07:27:30.235213+0000","last_active":"2024-03-06T07:27:31.241901+0000","last_peered":"2024-03-06T07:27:31.241901+0000","last_clean":"2024-03-06T07:27:31.241901+0000","last_became_active":"2024-03-06T07:27:30.234891+0000","last_became_peered":"2024-03-06T07:27:30.234891+0000","last_unstale":"2024-03-06T07:27:31.241901+0000","last_undegraded":"2024-03-06T07:27:31.241901+0000","last_fullsized":"2024-03-06T07:27:31.241901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.226087+0000","last_change":"2024-03-06T07:27:29.228355+0000","last_active":"2024-03-06T07:27:30.226087+0000","last_peered":"2024-03-06T07:27:30.226087+0000","last_clean":"2024-03-06T07:27:30.226087+0000","last_became_active":"2024-03-06T07:27:29.228054+0000","last_became_peered":"2024-03-06T07:27:29.228054+0000","last_unstale":"2024-03-06T07:27:30.226087+0000","last_undegraded":"2024-03-06T07:27:30.226087+0000","last_fullsized":"2024-03-06T07:27:30.226087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240414+0000","last_change":"2024-03-06T07:27:30.236691+0000","last_active":"2024-03-06T07:27:31.240414+0000","last_peered":"2024-03-06T07:27:31.240414+0000","last_clean":"2024-03-06T07:27:31.240414+0000","last_became_active":"2024-03-06T07:27:30.236318+0000","last_became_peered":"2024-03-06T07:27:30.236318+0000","last_unstale":"2024-03-06T07:27:31.240414+0000","last_undegraded":"2024-03-06T07:27:31.240414+0000","last_fullsized":"2024-03-06T07:27:31.240414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:06.911342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242168+0000","last_change":"2024-03-06T07:27:29.218901+0000","last_active":"2024-03-06T07:27:31.242168+0000","last_peered":"2024-03-06T07:27:31.242168+0000","last_clean":"2024-03-06T07:27:31.242168+0000","last_became_active":"2024-03-06T07:27:29.218549+0000","last_became_peered":"2024-03-06T07:27:29.218549+0000","last_unstale":"2024-03-06T07:27:31.242168+0000","last_undegraded":"2024-03-06T07:27:31.242168+0000","last_fullsized":"2024-03-06T07:27:31.242168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.271314+0000","last_change":"2024-03-06T07:27:30.269139+0000","last_active":"2024-03-06T07:27:30.271314+0000","last_peered":"2024-03-06T07:27:30.271314+0000","last_clean":"2024-03-06T07:27:30.271314+0000","last_became_active":"2024-03-06T07:27:30.265940+0000","last_became_peered":"2024-03-06T07:27:30.265940+0000","last_unstale":"2024-03-06T07:27:30.271314+0000","last_undegraded":"2024-03-06T07:27:30.271314+0000","last_fullsized":"2024-03-06T07:27:30.271314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.238792+0000","last_change":"2024-03-06T07:27:29.224828+0000","last_active":"2024-03-06T07:27:30.238792+0000","last_peered":"2024-03-06T07:27:30.238792+0000","last_clean":"2024-03-06T07:27:30.238792+0000","last_became_active":"2024-03-06T07:27:29.223899+0000","last_became_peered":"2024-03-06T07:27:29.223899+0000","last_unstale":"2024-03-06T07:27:30.238792+0000","last_undegraded":"2024-03-06T07:27:30.238792+0000","last_fullsized":"2024-03-06T07:27:30.238792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261170+0000","last_change":"2024-03-06T07:27:30.258895+0000","last_active":"2024-03-06T07:27:30.261170+0000","last_peered":"2024-03-06T07:27:30.261170+0000","last_clean":"2024-03-06T07:27:30.261170+0000","last_became_active":"2024-03-06T07:27:30.258204+0000","last_became_peered":"2024-03-06T07:27:30.258204+0000","last_unstale":"2024-03-06T07:27:30.261170+0000","last_undegraded":"2024-03-06T07:27:30.261170+0000","last_fullsized":"2024-03-06T07:27:30.261170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:54.202774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239843+0000","last_change":"2024-03-06T07:27:29.232043+0000","last_active":"2024-03-06T07:27:31.239843+0000","last_peered":"2024-03-06T07:27:31.239843+0000","last_clean":"2024-03-06T07:27:31.239843+0000","last_became_active":"2024-03-06T07:27:29.231659+0000","last_became_peered":"2024-03-06T07:27:29.231659+0000","last_unstale":"2024-03-06T07:27:31.239843+0000","last_undegraded":"2024-03-06T07:27:31.239843+0000","last_fullsized":"2024-03-06T07:27:31.239843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262284+0000","last_change":"2024-03-06T07:27:30.253450+0000","last_active":"2024-03-06T07:27:30.262284+0000","last_peered":"2024-03-06T07:27:30.262284+0000","last_clean":"2024-03-06T07:27:30.262284+0000","last_became_active":"2024-03-06T07:27:30.250531+0000","last_became_peered":"2024-03-06T07:27:30.250531+0000","last_unstale":"2024-03-06T07:27:30.262284+0000","last_undegraded":"2024-03-06T07:27:30.262284+0000","last_fullsized":"2024-03-06T07:27:30.262284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:14.695885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242917+0000","last_change":"2024-03-06T07:27:29.216258+0000","last_active":"2024-03-06T07:27:31.242917+0000","last_peered":"2024-03-06T07:27:31.242917+0000","last_clean":"2024-03-06T07:27:31.242917+0000","last_became_active":"2024-03-06T07:27:29.215924+0000","last_became_peered":"2024-03-06T07:27:29.215924+0000","last_unstale":"2024-03-06T07:27:31.242917+0000","last_undegraded":"2024-03-06T07:27:31.242917+0000","last_fullsized":"2024-03-06T07:27:31.242917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266248+0000","last_change":"2024-03-06T07:27:30.248670+0000","last_active":"2024-03-06T07:27:30.266248+0000","last_peered":"2024-03-06T07:27:30.266248+0000","last_clean":"2024-03-06T07:27:30.266248+0000","last_became_active":"2024-03-06T07:27:30.248398+0000","last_became_peered":"2024-03-06T07:27:30.248398+0000","last_unstale":"2024-03-06T07:27:30.266248+0000","last_undegraded":"2024-03-06T07:27:30.266248+0000","last_fullsized":"2024-03-06T07:27:30.266248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:55.908106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266659+0000","last_change":"2024-03-06T07:27:29.228193+0000","last_active":"2024-03-06T07:27:30.266659+0000","last_peered":"2024-03-06T07:27:30.266659+0000","last_clean":"2024-03-06T07:27:30.266659+0000","last_became_active":"2024-03-06T07:27:29.227840+0000","last_became_peered":"2024-03-06T07:27:29.227840+0000","last_unstale":"2024-03-06T07:27:30.266659+0000","last_undegraded":"2024-03-06T07:27:30.266659+0000","last_fullsized":"2024-03-06T07:27:30.266659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:21.775657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.243820+0000","last_change":"2024-03-06T07:27:30.240692+0000","last_active":"2024-03-06T07:27:30.243820+0000","last_peered":"2024-03-06T07:27:30.243820+0000","last_clean":"2024-03-06T07:27:30.243820+0000","last_became_active":"2024-03-06T07:27:30.240410+0000","last_became_peered":"2024-03-06T07:27:30.240410+0000","last_unstale":"2024-03-06T07:27:30.243820+0000","last_undegraded":"2024-03-06T07:27:30.243820+0000","last_fullsized":"2024-03-06T07:27:30.243820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:16.421421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.227276+0000","last_change":"2024-03-06T07:27:29.231958+0000","last_active":"2024-03-06T07:27:30.227276+0000","last_peered":"2024-03-06T07:27:30.227276+0000","last_clean":"2024-03-06T07:27:30.227276+0000","last_became_active":"2024-03-06T07:27:29.231690+0000","last_became_peered":"2024-03-06T07:27:29.231690+0000","last_unstale":"2024-03-06T07:27:30.227276+0000","last_undegraded":"2024-03-06T07:27:30.227276+0000","last_fullsized":"2024-03-06T07:27:30.227276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261652+0000","last_change":"2024-03-06T07:27:30.258434+0000","last_active":"2024-03-06T07:27:30.261652+0000","last_peered":"2024-03-06T07:27:30.261652+0000","last_clean":"2024-03-06T07:27:30.261652+0000","last_became_active":"2024-03-06T07:27:30.258094+0000","last_became_peered":"2024-03-06T07:27:30.258094+0000","last_unstale":"2024-03-06T07:27:30.261652+0000","last_undegraded":"2024-03-06T07:27:30.261652+0000","last_fullsized":"2024-03-06T07:27:30.261652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:42.209474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241544+0000","last_change":"2024-03-06T07:27:29.231347+0000","last_active":"2024-03-06T07:27:31.241544+0000","last_peered":"2024-03-06T07:27:31.241544+0000","last_clean":"2024-03-06T07:27:31.241544+0000","last_became_active":"2024-03-06T07:27:29.230852+0000","last_became_peered":"2024-03-06T07:27:29.230852+0000","last_unstale":"2024-03-06T07:27:31.241544+0000","last_undegraded":"2024-03-06T07:27:31.241544+0000","last_fullsized":"2024-03-06T07:27:31.241544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242563+0000","last_change":"2024-03-06T07:27:30.240275+0000","last_active":"2024-03-06T07:27:31.242563+0000","last_peered":"2024-03-06T07:27:31.242563+0000","last_clean":"2024-03-06T07:27:31.242563+0000","last_became_active":"2024-03-06T07:27:30.239945+0000","last_became_peered":"2024-03-06T07:27:30.239945+0000","last_unstale":"2024-03-06T07:27:31.242563+0000","last_undegraded":"2024-03-06T07:27:31.242563+0000","last_fullsized":"2024-03-06T07:27:31.242563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695318+0000","last_change":"2024-03-06T07:27:29.217898+0000","last_active":"2024-03-06T07:27:31.695318+0000","last_peered":"2024-03-06T07:27:31.695318+0000","last_clean":"2024-03-06T07:27:31.695318+0000","last_became_active":"2024-03-06T07:27:29.217605+0000","last_became_peered":"2024-03-06T07:27:29.217605+0000","last_unstale":"2024-03-06T07:27:31.695318+0000","last_undegraded":"2024-03-06T07:27:31.695318+0000","last_fullsized":"2024-03-06T07:27:31.695318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.683045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261715+0000","last_change":"2024-03-06T07:27:30.258444+0000","last_active":"2024-03-06T07:27:30.261715+0000","last_peered":"2024-03-06T07:27:30.261715+0000","last_clean":"2024-03-06T07:27:30.261715+0000","last_became_active":"2024-03-06T07:27:30.258119+0000","last_became_peered":"2024-03-06T07:27:30.258119+0000","last_unstale":"2024-03-06T07:27:30.261715+0000","last_undegraded":"2024-03-06T07:27:30.261715+0000","last_fullsized":"2024-03-06T07:27:30.261715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:17.318504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262526+0000","last_change":"2024-03-06T07:27:29.228474+0000","last_active":"2024-03-06T07:27:30.262526+0000","last_peered":"2024-03-06T07:27:30.262526+0000","last_clean":"2024-03-06T07:27:30.262526+0000","last_became_active":"2024-03-06T07:27:29.228168+0000","last_became_peered":"2024-03-06T07:27:29.228168+0000","last_unstale":"2024-03-06T07:27:30.262526+0000","last_undegraded":"2024-03-06T07:27:30.262526+0000","last_fullsized":"2024-03-06T07:27:30.262526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:36.193215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.254491+0000","last_change":"2024-03-06T07:27:30.243658+0000","last_active":"2024-03-06T07:27:30.254491+0000","last_peered":"2024-03-06T07:27:30.254491+0000","last_clean":"2024-03-06T07:27:30.254491+0000","last_became_active":"2024-03-06T07:27:30.243145+0000","last_became_peered":"2024-03-06T07:27:30.243145+0000","last_unstale":"2024-03-06T07:27:30.254491+0000","last_undegraded":"2024-03-06T07:27:30.254491+0000","last_fullsized":"2024-03-06T07:27:30.254491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.947928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695709+0000","last_change":"2024-03-06T07:27:29.221215+0000","last_active":"2024-03-06T07:27:31.695709+0000","last_peered":"2024-03-06T07:27:31.695709+0000","last_clean":"2024-03-06T07:27:31.695709+0000","last_became_active":"2024-03-06T07:27:29.220256+0000","last_became_peered":"2024-03-06T07:27:29.220256+0000","last_unstale":"2024-03-06T07:27:31.695709+0000","last_undegraded":"2024-03-06T07:27:31.695709+0000","last_fullsized":"2024-03-06T07:27:31.695709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261816+0000","last_change":"2024-03-06T07:27:30.258256+0000","last_active":"2024-03-06T07:27:30.261816+0000","last_peered":"2024-03-06T07:27:30.261816+0000","last_clean":"2024-03-06T07:27:30.261816+0000","last_became_active":"2024-03-06T07:27:30.255209+0000","last_became_peered":"2024-03-06T07:27:30.255209+0000","last_unstale":"2024-03-06T07:27:30.261816+0000","last_undegraded":"2024-03-06T07:27:30.261816+0000","last_fullsized":"2024-03-06T07:27:30.261816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:52.703156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240185+0000","last_change":"2024-03-06T07:27:29.225919+0000","last_active":"2024-03-06T07:27:31.240185+0000","last_peered":"2024-03-06T07:27:31.240185+0000","last_clean":"2024-03-06T07:27:31.240185+0000","last_became_active":"2024-03-06T07:27:29.225690+0000","last_became_peered":"2024-03-06T07:27:29.225690+0000","last_unstale":"2024-03-06T07:27:31.240185+0000","last_undegraded":"2024-03-06T07:27:31.240185+0000","last_fullsized":"2024-03-06T07:27:31.240185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269657+0000","last_change":"2024-03-06T07:27:30.259916+0000","last_active":"2024-03-06T07:27:30.269657+0000","last_peered":"2024-03-06T07:27:30.269657+0000","last_clean":"2024-03-06T07:27:30.269657+0000","last_became_active":"2024-03-06T07:27:30.259100+0000","last_became_peered":"2024-03-06T07:27:30.259100+0000","last_unstale":"2024-03-06T07:27:30.269657+0000","last_undegraded":"2024-03-06T07:27:30.269657+0000","last_fullsized":"2024-03-06T07:27:30.269657+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:44.205908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695658+0000","last_change":"2024-03-06T07:27:29.231688+0000","last_active":"2024-03-06T07:27:31.695658+0000","last_peered":"2024-03-06T07:27:31.695658+0000","last_clean":"2024-03-06T07:27:31.695658+0000","last_became_active":"2024-03-06T07:27:29.230816+0000","last_became_peered":"2024-03-06T07:27:29.230816+0000","last_unstale":"2024-03-06T07:27:31.695658+0000","last_undegraded":"2024-03-06T07:27:31.695658+0000","last_fullsized":"2024-03-06T07:27:31.695658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:51.657884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.267821+0000","last_change":"2024-03-06T07:27:30.264213+0000","last_active":"2024-03-06T07:27:30.267821+0000","last_peered":"2024-03-06T07:27:30.267821+0000","last_clean":"2024-03-06T07:27:30.267821+0000","last_became_active":"2024-03-06T07:27:30.263986+0000","last_became_peered":"2024-03-06T07:27:30.263986+0000","last_unstale":"2024-03-06T07:27:30.267821+0000","last_undegraded":"2024-03-06T07:27:30.267821+0000","last_fullsized":"2024-03-06T07:27:30.267821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.744486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":168,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.228124+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:27:30.228124+0000","last_peered":"2024-03-06T07:27:30.228124+0000","last_clean":"2024-03-06T07:27:30.228124+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:27:30.228124+0000","last_undegraded":"2024-03-06T07:27:30.228124+0000","last_fullsized":"2024-03-06T07:27:30.228124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.231617+0000","last_change":"2024-03-06T07:27:29.232294+0000","last_active":"2024-03-06T07:27:30.231617+0000","last_peered":"2024-03-06T07:27:30.231617+0000","last_clean":"2024-03-06T07:27:30.231617+0000","last_became_active":"2024-03-06T07:27:29.231839+0000","last_became_peered":"2024-03-06T07:27:29.231839+0000","last_unstale":"2024-03-06T07:27:30.231617+0000","last_undegraded":"2024-03-06T07:27:30.231617+0000","last_fullsized":"2024-03-06T07:27:30.231617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241928+0000","last_change":"2024-03-06T07:27:30.245626+0000","last_active":"2024-03-06T07:27:31.241928+0000","last_peered":"2024-03-06T07:27:31.241928+0000","last_clean":"2024-03-06T07:27:31.241928+0000","last_became_active":"2024-03-06T07:27:30.245324+0000","last_became_peered":"2024-03-06T07:27:30.245324+0000","last_unstale":"2024-03-06T07:27:31.241928+0000","last_undegraded":"2024-03-06T07:27:31.241928+0000","last_fullsized":"2024-03-06T07:27:31.241928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.228052+0000","last_change":"2024-03-06T07:27:29.234333+0000","last_active":"2024-03-06T07:27:30.228052+0000","last_peered":"2024-03-06T07:27:30.228052+0000","last_clean":"2024-03-06T07:27:30.228052+0000","last_became_active":"2024-03-06T07:27:29.234024+0000","last_became_peered":"2024-03-06T07:27:29.234024+0000","last_unstale":"2024-03-06T07:27:30.228052+0000","last_undegraded":"2024-03-06T07:27:30.228052+0000","last_fullsized":"2024-03-06T07:27:30.228052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.260516+0000","last_change":"2024-03-06T07:27:30.254664+0000","last_active":"2024-03-06T07:27:30.260516+0000","last_peered":"2024-03-06T07:27:30.260516+0000","last_clean":"2024-03-06T07:27:30.260516+0000","last_became_active":"2024-03-06T07:27:30.254371+0000","last_became_peered":"2024-03-06T07:27:30.254371+0000","last_unstale":"2024-03-06T07:27:30.260516+0000","last_undegraded":"2024-03-06T07:27:30.260516+0000","last_fullsized":"2024-03-06T07:27:30.260516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:52.878333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695651+0000","last_change":"2024-03-06T07:27:29.220564+0000","last_active":"2024-03-06T07:27:31.695651+0000","last_peered":"2024-03-06T07:27:31.695651+0000","last_clean":"2024-03-06T07:27:31.695651+0000","last_became_active":"2024-03-06T07:27:29.220231+0000","last_became_peered":"2024-03-06T07:27:29.220231+0000","last_unstale":"2024-03-06T07:27:31.695651+0000","last_undegraded":"2024-03-06T07:27:31.695651+0000","last_fullsized":"2024-03-06T07:27:31.695651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:18.751461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.238983+0000","last_change":"2024-03-06T07:27:30.244146+0000","last_active":"2024-03-06T07:27:31.238983+0000","last_peered":"2024-03-06T07:27:31.238983+0000","last_clean":"2024-03-06T07:27:31.238983+0000","last_became_active":"2024-03-06T07:27:30.241811+0000","last_became_peered":"2024-03-06T07:27:30.241811+0000","last_unstale":"2024-03-06T07:27:31.238983+0000","last_undegraded":"2024-03-06T07:27:31.238983+0000","last_fullsized":"2024-03-06T07:27:31.238983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:52.703156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.227697+0000","last_change":"2024-03-06T07:27:29.221902+0000","last_active":"2024-03-06T07:27:30.227697+0000","last_peered":"2024-03-06T07:27:30.227697+0000","last_clean":"2024-03-06T07:27:30.227697+0000","last_became_active":"2024-03-06T07:27:29.221598+0000","last_became_peered":"2024-03-06T07:27:29.221598+0000","last_unstale":"2024-03-06T07:27:30.227697+0000","last_undegraded":"2024-03-06T07:27:30.227697+0000","last_fullsized":"2024-03-06T07:27:30.227697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.254349+0000","last_change":"2024-03-06T07:27:30.243366+0000","last_active":"2024-03-06T07:27:30.254349+0000","last_peered":"2024-03-06T07:27:30.254349+0000","last_clean":"2024-03-06T07:27:30.254349+0000","last_became_active":"2024-03-06T07:27:30.243054+0000","last_became_peered":"2024-03-06T07:27:30.243054+0000","last_unstale":"2024-03-06T07:27:30.254349+0000","last_undegraded":"2024-03-06T07:27:30.254349+0000","last_fullsized":"2024-03-06T07:27:30.254349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:00.926447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239297+0000","last_change":"2024-03-06T07:27:29.220848+0000","last_active":"2024-03-06T07:27:31.239297+0000","last_peered":"2024-03-06T07:27:31.239297+0000","last_clean":"2024-03-06T07:27:31.239297+0000","last_became_active":"2024-03-06T07:27:29.220557+0000","last_became_peered":"2024-03-06T07:27:29.220557+0000","last_unstale":"2024-03-06T07:27:31.239297+0000","last_undegraded":"2024-03-06T07:27:31.239297+0000","last_fullsized":"2024-03-06T07:27:31.239297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269534+0000","last_change":"2024-03-06T07:27:30.259510+0000","last_active":"2024-03-06T07:27:30.269534+0000","last_peered":"2024-03-06T07:27:30.269534+0000","last_clean":"2024-03-06T07:27:30.269534+0000","last_became_active":"2024-03-06T07:27:30.258936+0000","last_became_peered":"2024-03-06T07:27:30.258936+0000","last_unstale":"2024-03-06T07:27:30.269534+0000","last_undegraded":"2024-03-06T07:27:30.269534+0000","last_fullsized":"2024-03-06T07:27:30.269534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:18.107539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.224951+0000","last_change":"2024-03-06T07:27:29.237638+0000","last_active":"2024-03-06T07:27:30.224951+0000","last_peered":"2024-03-06T07:27:30.224951+0000","last_clean":"2024-03-06T07:27:30.224951+0000","last_became_active":"2024-03-06T07:27:29.237304+0000","last_became_peered":"2024-03-06T07:27:29.237304+0000","last_unstale":"2024-03-06T07:27:30.224951+0000","last_undegraded":"2024-03-06T07:27:30.224951+0000","last_fullsized":"2024-03-06T07:27:30.224951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240309+0000","last_change":"2024-03-06T07:27:30.241439+0000","last_active":"2024-03-06T07:27:31.240309+0000","last_peered":"2024-03-06T07:27:31.240309+0000","last_clean":"2024-03-06T07:27:31.240309+0000","last_became_active":"2024-03-06T07:27:30.241163+0000","last_became_peered":"2024-03-06T07:27:30.241163+0000","last_unstale":"2024-03-06T07:27:31.240309+0000","last_undegraded":"2024-03-06T07:27:31.240309+0000","last_fullsized":"2024-03-06T07:27:31.240309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.744486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.253333+0000","last_change":"2024-03-06T07:27:29.223056+0000","last_active":"2024-03-06T07:27:31.253333+0000","last_peered":"2024-03-06T07:27:31.253333+0000","last_clean":"2024-03-06T07:27:31.253333+0000","last_became_active":"2024-03-06T07:27:29.222744+0000","last_became_peered":"2024-03-06T07:27:29.222744+0000","last_unstale":"2024-03-06T07:27:31.253333+0000","last_undegraded":"2024-03-06T07:27:31.253333+0000","last_fullsized":"2024-03-06T07:27:31.253333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243861+0000","last_change":"2024-03-06T07:27:30.241735+0000","last_active":"2024-03-06T07:27:31.243861+0000","last_peered":"2024-03-06T07:27:31.243861+0000","last_clean":"2024-03-06T07:27:31.243861+0000","last_became_active":"2024-03-06T07:27:30.241436+0000","last_became_peered":"2024-03-06T07:27:30.241436+0000","last_unstale":"2024-03-06T07:27:31.243861+0000","last_undegraded":"2024-03-06T07:27:31.243861+0000","last_fullsized":"2024-03-06T07:27:31.243861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:58.613904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242269+0000","last_change":"2024-03-06T07:27:29.226601+0000","last_active":"2024-03-06T07:27:31.242269+0000","last_peered":"2024-03-06T07:27:31.242269+0000","last_clean":"2024-03-06T07:27:31.242269+0000","last_became_active":"2024-03-06T07:27:29.225390+0000","last_became_peered":"2024-03-06T07:27:29.225390+0000","last_unstale":"2024-03-06T07:27:31.242269+0000","last_undegraded":"2024-03-06T07:27:31.242269+0000","last_fullsized":"2024-03-06T07:27:31.242269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.243578+0000","last_change":"2024-03-06T07:27:30.242063+0000","last_active":"2024-03-06T07:27:30.243578+0000","last_peered":"2024-03-06T07:27:30.243578+0000","last_clean":"2024-03-06T07:27:30.243578+0000","last_became_active":"2024-03-06T07:27:30.241737+0000","last_became_peered":"2024-03-06T07:27:30.241737+0000","last_unstale":"2024-03-06T07:27:30.243578+0000","last_undegraded":"2024-03-06T07:27:30.243578+0000","last_fullsized":"2024-03-06T07:27:30.243578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:30.484610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.227853+0000","last_change":"2024-03-06T07:27:29.235266+0000","last_active":"2024-03-06T07:27:30.227853+0000","last_peered":"2024-03-06T07:27:30.227853+0000","last_clean":"2024-03-06T07:27:30.227853+0000","last_became_active":"2024-03-06T07:27:29.234878+0000","last_became_peered":"2024-03-06T07:27:29.234878+0000","last_unstale":"2024-03-06T07:27:30.227853+0000","last_undegraded":"2024-03-06T07:27:30.227853+0000","last_fullsized":"2024-03-06T07:27:30.227853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262179+0000","last_change":"2024-03-06T07:27:30.250799+0000","last_active":"2024-03-06T07:27:30.262179+0000","last_peered":"2024-03-06T07:27:30.262179+0000","last_clean":"2024-03-06T07:27:30.262179+0000","last_became_active":"2024-03-06T07:27:30.250475+0000","last_became_peered":"2024-03-06T07:27:30.250475+0000","last_unstale":"2024-03-06T07:27:30.262179+0000","last_undegraded":"2024-03-06T07:27:30.262179+0000","last_fullsized":"2024-03-06T07:27:30.262179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:52.670724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239430+0000","last_change":"2024-03-06T07:27:29.224306+0000","last_active":"2024-03-06T07:27:31.239430+0000","last_peered":"2024-03-06T07:27:31.239430+0000","last_clean":"2024-03-06T07:27:31.239430+0000","last_became_active":"2024-03-06T07:27:29.223923+0000","last_became_peered":"2024-03-06T07:27:29.223923+0000","last_unstale":"2024-03-06T07:27:31.239430+0000","last_undegraded":"2024-03-06T07:27:31.239430+0000","last_fullsized":"2024-03-06T07:27:31.239430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242506+0000","last_change":"2024-03-06T07:27:30.246857+0000","last_active":"2024-03-06T07:27:31.242506+0000","last_peered":"2024-03-06T07:27:31.242506+0000","last_clean":"2024-03-06T07:27:31.242506+0000","last_became_active":"2024-03-06T07:27:30.246519+0000","last_became_peered":"2024-03-06T07:27:30.246519+0000","last_unstale":"2024-03-06T07:27:31.242506+0000","last_undegraded":"2024-03-06T07:27:31.242506+0000","last_fullsized":"2024-03-06T07:27:31.242506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239222+0000","last_change":"2024-03-06T07:27:29.217568+0000","last_active":"2024-03-06T07:27:31.239222+0000","last_peered":"2024-03-06T07:27:31.239222+0000","last_clean":"2024-03-06T07:27:31.239222+0000","last_became_active":"2024-03-06T07:27:29.217204+0000","last_became_peered":"2024-03-06T07:27:29.217204+0000","last_unstale":"2024-03-06T07:27:31.239222+0000","last_undegraded":"2024-03-06T07:27:31.239222+0000","last_fullsized":"2024-03-06T07:27:31.239222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242343+0000","last_change":"2024-03-06T07:27:30.244509+0000","last_active":"2024-03-06T07:27:31.242343+0000","last_peered":"2024-03-06T07:27:31.242343+0000","last_clean":"2024-03-06T07:27:31.242343+0000","last_became_active":"2024-03-06T07:27:30.243804+0000","last_became_peered":"2024-03-06T07:27:30.243804+0000","last_unstale":"2024-03-06T07:27:31.242343+0000","last_undegraded":"2024-03-06T07:27:31.242343+0000","last_fullsized":"2024-03-06T07:27:31.242343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.263087+0000","last_change":"2024-03-06T07:27:29.235464+0000","last_active":"2024-03-06T07:27:30.263087+0000","last_peered":"2024-03-06T07:27:30.263087+0000","last_clean":"2024-03-06T07:27:30.263087+0000","last_became_active":"2024-03-06T07:27:29.234532+0000","last_became_peered":"2024-03-06T07:27:29.234532+0000","last_unstale":"2024-03-06T07:27:30.263087+0000","last_undegraded":"2024-03-06T07:27:30.263087+0000","last_fullsized":"2024-03-06T07:27:30.263087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242177+0000","last_change":"2024-03-06T07:27:30.257067+0000","last_active":"2024-03-06T07:27:31.242177+0000","last_peered":"2024-03-06T07:27:31.242177+0000","last_clean":"2024-03-06T07:27:31.242177+0000","last_became_active":"2024-03-06T07:27:30.256842+0000","last_became_peered":"2024-03-06T07:27:30.256842+0000","last_unstale":"2024-03-06T07:27:31.242177+0000","last_undegraded":"2024-03-06T07:27:31.242177+0000","last_fullsized":"2024-03-06T07:27:31.242177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270955+0000","last_change":"2024-03-06T07:27:30.266494+0000","last_active":"2024-03-06T07:27:30.270955+0000","last_peered":"2024-03-06T07:27:30.270955+0000","last_clean":"2024-03-06T07:27:30.270955+0000","last_became_active":"2024-03-06T07:27:30.265258+0000","last_became_peered":"2024-03-06T07:27:30.265258+0000","last_unstale":"2024-03-06T07:27:30.270955+0000","last_undegraded":"2024-03-06T07:27:30.270955+0000","last_fullsized":"2024-03-06T07:27:30.270955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.271387+0000","last_change":"2024-03-06T07:27:29.235012+0000","last_active":"2024-03-06T07:27:30.271387+0000","last_peered":"2024-03-06T07:27:30.271387+0000","last_clean":"2024-03-06T07:27:30.271387+0000","last_became_active":"2024-03-06T07:27:29.234638+0000","last_became_peered":"2024-03-06T07:27:29.234638+0000","last_unstale":"2024-03-06T07:27:30.271387+0000","last_undegraded":"2024-03-06T07:27:30.271387+0000","last_fullsized":"2024-03-06T07:27:30.271387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270622+0000","last_change":"2024-03-06T07:27:30.265713+0000","last_active":"2024-03-06T07:27:30.270622+0000","last_peered":"2024-03-06T07:27:30.270622+0000","last_clean":"2024-03-06T07:27:30.270622+0000","last_became_active":"2024-03-06T07:27:30.259972+0000","last_became_peered":"2024-03-06T07:27:30.259972+0000","last_unstale":"2024-03-06T07:27:30.270622+0000","last_undegraded":"2024-03-06T07:27:30.270622+0000","last_fullsized":"2024-03-06T07:27:30.270622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270292+0000","last_change":"2024-03-06T07:27:29.236520+0000","last_active":"2024-03-06T07:27:30.270292+0000","last_peered":"2024-03-06T07:27:30.270292+0000","last_clean":"2024-03-06T07:27:30.270292+0000","last_became_active":"2024-03-06T07:27:29.235298+0000","last_became_peered":"2024-03-06T07:27:29.235298+0000","last_unstale":"2024-03-06T07:27:30.270292+0000","last_undegraded":"2024-03-06T07:27:30.270292+0000","last_fullsized":"2024-03-06T07:27:30.270292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261148+0000","last_change":"2024-03-06T07:27:30.258866+0000","last_active":"2024-03-06T07:27:30.261148+0000","last_peered":"2024-03-06T07:27:30.261148+0000","last_clean":"2024-03-06T07:27:30.261148+0000","last_became_active":"2024-03-06T07:27:30.258557+0000","last_became_peered":"2024-03-06T07:27:30.258557+0000","last_unstale":"2024-03-06T07:27:30.261148+0000","last_undegraded":"2024-03-06T07:27:30.261148+0000","last_fullsized":"2024-03-06T07:27:30.261148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:49.587341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243647+0000","last_change":"2024-03-06T07:27:29.221773+0000","last_active":"2024-03-06T07:27:31.243647+0000","last_peered":"2024-03-06T07:27:31.243647+0000","last_clean":"2024-03-06T07:27:31.243647+0000","last_became_active":"2024-03-06T07:27:29.221365+0000","last_became_peered":"2024-03-06T07:27:29.221365+0000","last_unstale":"2024-03-06T07:27:31.243647+0000","last_undegraded":"2024-03-06T07:27:31.243647+0000","last_fullsized":"2024-03-06T07:27:31.243647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.408581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243868+0000","last_change":"2024-03-06T07:27:30.245108+0000","last_active":"2024-03-06T07:27:31.243868+0000","last_peered":"2024-03-06T07:27:31.243868+0000","last_clean":"2024-03-06T07:27:31.243868+0000","last_became_active":"2024-03-06T07:27:30.244795+0000","last_became_peered":"2024-03-06T07:27:30.244795+0000","last_unstale":"2024-03-06T07:27:31.243868+0000","last_undegraded":"2024-03-06T07:27:31.243868+0000","last_fullsized":"2024-03-06T07:27:31.243868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.111110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.271690+0000","last_change":"2024-03-06T07:27:29.229397+0000","last_active":"2024-03-06T07:27:30.271690+0000","last_peered":"2024-03-06T07:27:30.271690+0000","last_clean":"2024-03-06T07:27:30.271690+0000","last_became_active":"2024-03-06T07:27:29.229097+0000","last_became_peered":"2024-03-06T07:27:29.229097+0000","last_unstale":"2024-03-06T07:27:30.271690+0000","last_undegraded":"2024-03-06T07:27:30.271690+0000","last_fullsized":"2024-03-06T07:27:30.271690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":662776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":73438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":71952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":79738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":662832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:34.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.992+0000 7f613f200640 1 Processor -- start 2024-03-06T07:27:34.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.992+0000 7f613f200640 1 -- start start 2024-03-06T07:27:34.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.994+0000 7f613f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6140145300 0x7f61400b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:34.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.994+0000 7f613f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61400714d0 con 0x7f61400cd060 2024-03-06T07:27:34.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.994+0000 7f613f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61400717b0 con 0x7f6140070f10 2024-03-06T07:27:34.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.994+0000 7f613f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6140071a90 con 0x7f6140145300 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61400cd060 0x7f614015c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37428/0 (socket says 172.21.15.6:37428) 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613e800640 1 -- 172.21.15.6:0/2037637611 learned_addr learned my addr 172.21.15.6:0/2037637611 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613de00640 1 --2- 172.21.15.6:0/2037637611 >> v2:172.21.15.178:3300/0 conn(0x7f6140145300 0x7f61400b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613ca00640 1 -- 172.21.15.6:0/2037637611 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108859578 0 0) 0x7f61400714d0 con 0x7f61400cd060 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613ca00640 1 -- 172.21.15.6:0/2037637611 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6124003520 con 0x7f61400cd060 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613de00640 1 -- 172.21.15.6:0/2037637611 >> v1:172.21.15.178:6789/0 conn(0x7f6140070f10 legacy=0x7f61400cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613de00640 1 -- 172.21.15.6:0/2037637611 >> v1:172.21.15.6:6789/0 conn(0x7f61400cd060 legacy=0x7f614015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:34.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.995+0000 7f613de00640 1 -- 172.21.15.6:0/2037637611 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6140071d70 con 0x7f6140145300 2024-03-06T07:27:34.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.997+0000 7f613de00640 1 --2- 172.21.15.6:0/2037637611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f61400b49a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f61300097a0 tx=0x7f613002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa6b3b5c4263513e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:34.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.997+0000 7f613ca00640 1 -- 172.21.15.6:0/2037637611 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130035020 con 0x7f6140145300 2024-03-06T07:27:34.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.997+0000 7f613ca00640 1 -- 172.21.15.6:0/2037637611 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6130031830 con 0x7f6140145300 2024-03-06T07:27:34.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.997+0000 7f613ca00640 1 -- 172.21.15.6:0/2037637611 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130031c00 con 0x7f6140145300 2024-03-06T07:27:35.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 -- 172.21.15.6:0/2037637611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 msgr2=0x7f61400b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 --2- 172.21.15.6:0/2037637611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f61400b49a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f61300097a0 tx=0x7f613002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 -- 172.21.15.6:0/2037637611 shutdown_connections 2024-03-06T07:27:35.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 --2- 172.21.15.6:0/2037637611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f61400b49a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 -- 172.21.15.6:0/2037637611 >> 172.21.15.6:0/2037637611 conn(0x7f614006b3d0 msgr2=0x7f614006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 -- 172.21.15.6:0/2037637611 shutdown_connections 2024-03-06T07:27:35.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:34.998+0000 7f613f200640 1 -- 172.21.15.6:0/2037637611 wait complete. 2024-03-06T07:27:35.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.000+0000 7f613f200640 1 Processor -- start 2024-03-06T07:27:35.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.000+0000 7f613f200640 1 -- start start 2024-03-06T07:27:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 0x7f61401e91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61400cd060 0x7f614015b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f614015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614015cc80 con 0x7f6140070f10 2024-03-06T07:27:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61401456e0 con 0x7f6140145300 2024-03-06T07:27:35.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614015ce00 con 0x7f61400cd060 2024-03-06T07:27:35.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 0x7f61401e91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 0x7f61401e91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42202/0 (socket says 172.21.15.6:42202) 2024-03-06T07:27:35.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.001+0000 7f613de00640 1 -- 172.21.15.6:0/520320146 learned_addr learned my addr 172.21.15.6:0/520320146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613e800640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f614015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613d400640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61400cd060 0x7f614015b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613de00640 1 -- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61400cd060 msgr2=0x7f614015b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613de00640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61400cd060 0x7f614015b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613de00640 1 -- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 msgr2=0x7f614015b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613de00640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f614015b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613de00640 1 -- 172.21.15.6:0/520320146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61401e5b60 con 0x7f6140070f10 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613de00640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 0x7f61401e91c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6130034040 tx=0x7f61300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130040070 con 0x7f6140070f10 2024-03-06T07:27:35.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613003a910 con 0x7f6140070f10 2024-03-06T07:27:35.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130035040 con 0x7f6140070f10 2024-03-06T07:27:35.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.002+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61401e5e40 con 0x7f6140070f10 2024-03-06T07:27:35.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.003+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614012ffe0 con 0x7f6140070f10 2024-03-06T07:27:35.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.004+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614004f330 con 0x7f6140070f10 2024-03-06T07:27:35.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.005+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6130038020 con 0x7f6140070f10 2024-03-06T07:27:35.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.005+0000 7f6136000640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f612006b500 0x7f612006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.006+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61300abc80 con 0x7f6140070f10 2024-03-06T07:27:35.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.006+0000 7f613d400640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f612006b500 0x7f612006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.007+0000 7f613d400640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f612006b500 0x7f612006d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6140143bb0 tx=0x7f612c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.010+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6130031800 con 0x7f6140070f10 2024-03-06T07:27:35.171 INFO:tasks.ceph.mds.b.smithi178.stderr:2024-03-06T07:27:35.170+0000 7f63f2400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:35.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.208+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6140074ab0 con 0x7f612006b500 2024-03-06T07:27:35.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.229+0000 7f6136000640 1 -- 172.21.15.6:0/520320146 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251903 (secure 0 0 0) 0x7f6140074ab0 con 0x7f612006b500 2024-03-06T07:27:35.234 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:35.236 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:27:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.234+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f612006b500 msgr2=0x7f612006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.234+0000 7f613f200640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f612006b500 0x7f612006d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6140143bb0 tx=0x7f612c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.234+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 msgr2=0x7f61401e91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.234+0000 7f613f200640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 0x7f61401e91c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6130034040 tx=0x7f61300046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 shutdown_connections 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f612006b500 0x7f612006d9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6140145300 0x7f614015b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61400cd060 0x7f614015b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 --2- 172.21.15.6:0/520320146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6140070f10 0x7f61401e91c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 >> 172.21.15.6:0/520320146 conn(0x7f614006b3d0 msgr2=0x7f614015c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 shutdown_connections 2024-03-06T07:27:35.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.235+0000 7f613f200640 1 -- 172.21.15.6:0/520320146 wait complete. 2024-03-06T07:27:35.257 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:27:33.437845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235700,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2949120,"data_stored":1772764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.254847"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.250766+0000","last_change":"2024-03-06T07:27:30.239373+0000","last_active":"2024-03-06T07:27:30.250766+0000","last_peered":"2024-03-06T07:27:30.250766+0000","last_clean":"2024-03-06T07:27:30.250766+0000","last_became_active":"2024-03-06T07:27:30.239046+0000","last_became_peered":"2024-03-06T07:27:30.239046+0000","last_unstale":"2024-03-06T07:27:30.250766+0000","last_undegraded":"2024-03-06T07:27:30.250766+0000","last_fullsized":"2024-03-06T07:27:30.250766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:56:58.456356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239467+0000","last_change":"2024-03-06T07:27:30.234823+0000","last_active":"2024-03-06T07:27:31.239467+0000","last_peered":"2024-03-06T07:27:31.239467+0000","last_clean":"2024-03-06T07:27:31.239467+0000","last_became_active":"2024-03-06T07:27:30.234521+0000","last_became_peered":"2024-03-06T07:27:30.234521+0000","last_unstale":"2024-03-06T07:27:31.239467+0000","last_undegraded":"2024-03-06T07:27:31.239467+0000","last_fullsized":"2024-03-06T07:27:31.239467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:17.318504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.247178+0000","last_change":"2024-03-06T07:27:30.244772+0000","last_active":"2024-03-06T07:27:30.247178+0000","last_peered":"2024-03-06T07:27:30.247178+0000","last_clean":"2024-03-06T07:27:30.247178+0000","last_became_active":"2024-03-06T07:27:30.244401+0000","last_became_peered":"2024-03-06T07:27:30.244401+0000","last_unstale":"2024-03-06T07:27:30.247178+0000","last_undegraded":"2024-03-06T07:27:30.247178+0000","last_fullsized":"2024-03-06T07:27:30.247178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:22.855579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269899+0000","last_change":"2024-03-06T07:27:30.260180+0000","last_active":"2024-03-06T07:27:30.269899+0000","last_peered":"2024-03-06T07:27:30.269899+0000","last_clean":"2024-03-06T07:27:30.269899+0000","last_became_active":"2024-03-06T07:27:30.259183+0000","last_became_peered":"2024-03-06T07:27:30.259183+0000","last_unstale":"2024-03-06T07:27:30.269899+0000","last_undegraded":"2024-03-06T07:27:30.269899+0000","last_fullsized":"2024-03-06T07:27:30.269899+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:48.495220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.260632+0000","last_change":"2024-03-06T07:27:30.254425+0000","last_active":"2024-03-06T07:27:30.260632+0000","last_peered":"2024-03-06T07:27:30.260632+0000","last_clean":"2024-03-06T07:27:30.260632+0000","last_became_active":"2024-03-06T07:27:30.253238+0000","last_became_peered":"2024-03-06T07:27:30.253238+0000","last_unstale":"2024-03-06T07:27:30.260632+0000","last_undegraded":"2024-03-06T07:27:30.260632+0000","last_fullsized":"2024-03-06T07:27:30.260632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:02.502387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269776+0000","last_change":"2024-03-06T07:27:30.259178+0000","last_active":"2024-03-06T07:27:30.269776+0000","last_peered":"2024-03-06T07:27:30.269776+0000","last_clean":"2024-03-06T07:27:30.269776+0000","last_became_active":"2024-03-06T07:27:30.258928+0000","last_became_peered":"2024-03-06T07:27:30.258928+0000","last_unstale":"2024-03-06T07:27:30.269776+0000","last_undegraded":"2024-03-06T07:27:30.269776+0000","last_fullsized":"2024-03-06T07:27:30.269776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:54.202774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.252318+0000","last_change":"2024-03-06T07:27:30.244331+0000","last_active":"2024-03-06T07:27:30.252318+0000","last_peered":"2024-03-06T07:27:30.252318+0000","last_clean":"2024-03-06T07:27:30.252318+0000","last_became_active":"2024-03-06T07:27:30.244078+0000","last_became_peered":"2024-03-06T07:27:30.244078+0000","last_unstale":"2024-03-06T07:27:30.252318+0000","last_undegraded":"2024-03-06T07:27:30.252318+0000","last_fullsized":"2024-03-06T07:27:30.252318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:50.573971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241726+0000","last_change":"2024-03-06T07:27:30.256607+0000","last_active":"2024-03-06T07:27:31.241726+0000","last_peered":"2024-03-06T07:27:31.241726+0000","last_clean":"2024-03-06T07:27:31.241726+0000","last_became_active":"2024-03-06T07:27:30.256321+0000","last_became_peered":"2024-03-06T07:27:30.256321+0000","last_unstale":"2024-03-06T07:27:31.241726+0000","last_undegraded":"2024-03-06T07:27:31.241726+0000","last_fullsized":"2024-03-06T07:27:31.241726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262343+0000","last_change":"2024-03-06T07:27:30.260973+0000","last_active":"2024-03-06T07:27:30.262343+0000","last_peered":"2024-03-06T07:27:30.262343+0000","last_clean":"2024-03-06T07:27:30.262343+0000","last_became_active":"2024-03-06T07:27:30.260401+0000","last_became_peered":"2024-03-06T07:27:30.260401+0000","last_unstale":"2024-03-06T07:27:30.262343+0000","last_undegraded":"2024-03-06T07:27:30.262343+0000","last_fullsized":"2024-03-06T07:27:30.262343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:18.053835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243083+0000","last_change":"2024-03-06T07:27:30.247832+0000","last_active":"2024-03-06T07:27:31.243083+0000","last_peered":"2024-03-06T07:27:31.243083+0000","last_clean":"2024-03-06T07:27:31.243083+0000","last_became_active":"2024-03-06T07:27:30.247503+0000","last_became_peered":"2024-03-06T07:27:30.247503+0000","last_unstale":"2024-03-06T07:27:31.243083+0000","last_undegraded":"2024-03-06T07:27:31.243083+0000","last_fullsized":"2024-03-06T07:27:31.243083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242309+0000","last_change":"2024-03-06T07:27:30.256946+0000","last_active":"2024-03-06T07:27:31.242309+0000","last_peered":"2024-03-06T07:27:31.242309+0000","last_clean":"2024-03-06T07:27:31.242309+0000","last_became_active":"2024-03-06T07:27:30.256643+0000","last_became_peered":"2024-03-06T07:27:30.256643+0000","last_unstale":"2024-03-06T07:27:31.242309+0000","last_undegraded":"2024-03-06T07:27:31.242309+0000","last_fullsized":"2024-03-06T07:27:31.242309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262105+0000","last_change":"2024-03-06T07:27:30.248076+0000","last_active":"2024-03-06T07:27:30.262105+0000","last_peered":"2024-03-06T07:27:30.262105+0000","last_clean":"2024-03-06T07:27:30.262105+0000","last_became_active":"2024-03-06T07:27:30.247733+0000","last_became_peered":"2024-03-06T07:27:30.247733+0000","last_unstale":"2024-03-06T07:27:30.262105+0000","last_undegraded":"2024-03-06T07:27:30.262105+0000","last_fullsized":"2024-03-06T07:27:30.262105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:19.764301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243474+0000","last_change":"2024-03-06T07:27:30.238620+0000","last_active":"2024-03-06T07:27:31.243474+0000","last_peered":"2024-03-06T07:27:31.243474+0000","last_clean":"2024-03-06T07:27:31.243474+0000","last_became_active":"2024-03-06T07:27:30.238357+0000","last_became_peered":"2024-03-06T07:27:30.238357+0000","last_unstale":"2024-03-06T07:27:31.243474+0000","last_undegraded":"2024-03-06T07:27:31.243474+0000","last_fullsized":"2024-03-06T07:27:31.243474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.245543+0000","last_change":"2024-03-06T07:27:30.242346+0000","last_active":"2024-03-06T07:27:30.245543+0000","last_peered":"2024-03-06T07:27:30.245543+0000","last_clean":"2024-03-06T07:27:30.245543+0000","last_became_active":"2024-03-06T07:27:30.242081+0000","last_became_peered":"2024-03-06T07:27:30.242081+0000","last_unstale":"2024-03-06T07:27:30.245543+0000","last_undegraded":"2024-03-06T07:27:30.245543+0000","last_fullsized":"2024-03-06T07:27:30.245543+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:46.777176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239135+0000","last_change":"2024-03-06T07:27:30.252254+0000","last_active":"2024-03-06T07:27:31.239135+0000","last_peered":"2024-03-06T07:27:31.239135+0000","last_clean":"2024-03-06T07:27:31.239135+0000","last_became_active":"2024-03-06T07:27:30.251915+0000","last_became_peered":"2024-03-06T07:27:30.251915+0000","last_unstale":"2024-03-06T07:27:31.239135+0000","last_undegraded":"2024-03-06T07:27:31.239135+0000","last_fullsized":"2024-03-06T07:27:31.239135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:42.209474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270906+0000","last_change":"2024-03-06T07:27:30.266564+0000","last_active":"2024-03-06T07:27:30.270906+0000","last_peered":"2024-03-06T07:27:30.270906+0000","last_clean":"2024-03-06T07:27:30.270906+0000","last_became_active":"2024-03-06T07:27:30.265165+0000","last_became_peered":"2024-03-06T07:27:30.265165+0000","last_unstale":"2024-03-06T07:27:30.270906+0000","last_undegraded":"2024-03-06T07:27:30.270906+0000","last_fullsized":"2024-03-06T07:27:30.270906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270924+0000","last_change":"2024-03-06T07:27:30.266828+0000","last_active":"2024-03-06T07:27:30.270924+0000","last_peered":"2024-03-06T07:27:30.270924+0000","last_clean":"2024-03-06T07:27:30.270924+0000","last_became_active":"2024-03-06T07:27:30.265102+0000","last_became_peered":"2024-03-06T07:27:30.265102+0000","last_unstale":"2024-03-06T07:27:30.270924+0000","last_undegraded":"2024-03-06T07:27:30.270924+0000","last_fullsized":"2024-03-06T07:27:30.270924+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695842+0000","last_change":"2024-03-06T07:27:30.239978+0000","last_active":"2024-03-06T07:27:31.695842+0000","last_peered":"2024-03-06T07:27:31.695842+0000","last_clean":"2024-03-06T07:27:31.695842+0000","last_became_active":"2024-03-06T07:27:30.239682+0000","last_became_peered":"2024-03-06T07:27:30.239682+0000","last_unstale":"2024-03-06T07:27:31.695842+0000","last_undegraded":"2024-03-06T07:27:31.695842+0000","last_fullsized":"2024-03-06T07:27:31.695842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:46.446447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266214+0000","last_change":"2024-03-06T07:27:30.245372+0000","last_active":"2024-03-06T07:27:30.266214+0000","last_peered":"2024-03-06T07:27:30.266214+0000","last_clean":"2024-03-06T07:27:30.266214+0000","last_became_active":"2024-03-06T07:27:30.245096+0000","last_became_peered":"2024-03-06T07:27:30.245096+0000","last_unstale":"2024-03-06T07:27:30.266214+0000","last_undegraded":"2024-03-06T07:27:30.266214+0000","last_fullsized":"2024-03-06T07:27:30.266214+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:35.043873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242869+0000","last_change":"2024-03-06T07:27:30.244785+0000","last_active":"2024-03-06T07:27:31.242869+0000","last_peered":"2024-03-06T07:27:31.242869+0000","last_clean":"2024-03-06T07:27:31.242869+0000","last_became_active":"2024-03-06T07:27:30.244505+0000","last_became_peered":"2024-03-06T07:27:30.244505+0000","last_unstale":"2024-03-06T07:27:31.242869+0000","last_undegraded":"2024-03-06T07:27:31.242869+0000","last_fullsized":"2024-03-06T07:27:31.242869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.257896+0000","last_change":"2024-03-06T07:27:30.253741+0000","last_active":"2024-03-06T07:27:30.257896+0000","last_peered":"2024-03-06T07:27:30.257896+0000","last_clean":"2024-03-06T07:27:30.257896+0000","last_became_active":"2024-03-06T07:27:30.251912+0000","last_became_peered":"2024-03-06T07:27:30.251912+0000","last_unstale":"2024-03-06T07:27:30.257896+0000","last_undegraded":"2024-03-06T07:27:30.257896+0000","last_fullsized":"2024-03-06T07:27:30.257896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:08.043724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.254285+0000","last_change":"2024-03-06T07:27:30.240983+0000","last_active":"2024-03-06T07:27:30.254285+0000","last_peered":"2024-03-06T07:27:30.254285+0000","last_clean":"2024-03-06T07:27:30.254285+0000","last_became_active":"2024-03-06T07:27:30.240088+0000","last_became_peered":"2024-03-06T07:27:30.240088+0000","last_unstale":"2024-03-06T07:27:30.254285+0000","last_undegraded":"2024-03-06T07:27:30.254285+0000","last_fullsized":"2024-03-06T07:27:30.254285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:08.043724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241714+0000","last_change":"2024-03-06T07:27:30.254933+0000","last_active":"2024-03-06T07:27:31.241714+0000","last_peered":"2024-03-06T07:27:31.241714+0000","last_clean":"2024-03-06T07:27:31.241714+0000","last_became_active":"2024-03-06T07:27:30.253777+0000","last_became_peered":"2024-03-06T07:27:30.253777+0000","last_unstale":"2024-03-06T07:27:31.241714+0000","last_undegraded":"2024-03-06T07:27:31.241714+0000","last_fullsized":"2024-03-06T07:27:31.241714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266696+0000","last_change":"2024-03-06T07:27:30.254617+0000","last_active":"2024-03-06T07:27:30.266696+0000","last_peered":"2024-03-06T07:27:30.266696+0000","last_clean":"2024-03-06T07:27:30.266696+0000","last_became_active":"2024-03-06T07:27:30.253826+0000","last_became_peered":"2024-03-06T07:27:30.253826+0000","last_unstale":"2024-03-06T07:27:30.266696+0000","last_undegraded":"2024-03-06T07:27:30.266696+0000","last_fullsized":"2024-03-06T07:27:30.266696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:49.587341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243679+0000","last_change":"2024-03-06T07:27:30.245625+0000","last_active":"2024-03-06T07:27:31.243679+0000","last_peered":"2024-03-06T07:27:31.243679+0000","last_clean":"2024-03-06T07:27:31.243679+0000","last_became_active":"2024-03-06T07:27:30.245310+0000","last_became_peered":"2024-03-06T07:27:30.245310+0000","last_unstale":"2024-03-06T07:27:31.243679+0000","last_undegraded":"2024-03-06T07:27:31.243679+0000","last_fullsized":"2024-03-06T07:27:31.243679+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.246068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243256+0000","last_change":"2024-03-06T07:27:30.241785+0000","last_active":"2024-03-06T07:27:31.243256+0000","last_peered":"2024-03-06T07:27:31.243256+0000","last_clean":"2024-03-06T07:27:31.243256+0000","last_became_active":"2024-03-06T07:27:30.241456+0000","last_became_peered":"2024-03-06T07:27:30.241456+0000","last_unstale":"2024-03-06T07:27:31.243256+0000","last_undegraded":"2024-03-06T07:27:31.243256+0000","last_fullsized":"2024-03-06T07:27:31.243256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239562+0000","last_change":"2024-03-06T07:27:30.239090+0000","last_active":"2024-03-06T07:27:31.239562+0000","last_peered":"2024-03-06T07:27:31.239562+0000","last_clean":"2024-03-06T07:27:31.239562+0000","last_became_active":"2024-03-06T07:27:30.238809+0000","last_became_peered":"2024-03-06T07:27:30.238809+0000","last_unstale":"2024-03-06T07:27:31.239562+0000","last_undegraded":"2024-03-06T07:27:31.239562+0000","last_fullsized":"2024-03-06T07:27:31.239562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:18.053835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243221+0000","last_change":"2024-03-06T07:27:30.250473+0000","last_active":"2024-03-06T07:27:31.243221+0000","last_peered":"2024-03-06T07:27:31.243221+0000","last_clean":"2024-03-06T07:27:31.243221+0000","last_became_active":"2024-03-06T07:27:30.250084+0000","last_became_peered":"2024-03-06T07:27:30.250084+0000","last_unstale":"2024-03-06T07:27:31.243221+0000","last_undegraded":"2024-03-06T07:27:31.243221+0000","last_fullsized":"2024-03-06T07:27:31.243221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.257877+0000","last_change":"2024-03-06T07:27:30.247866+0000","last_active":"2024-03-06T07:27:30.257877+0000","last_peered":"2024-03-06T07:27:30.257877+0000","last_clean":"2024-03-06T07:27:30.257877+0000","last_became_active":"2024-03-06T07:27:30.247526+0000","last_became_peered":"2024-03-06T07:27:30.247526+0000","last_unstale":"2024-03-06T07:27:30.257877+0000","last_undegraded":"2024-03-06T07:27:30.257877+0000","last_fullsized":"2024-03-06T07:27:30.257877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:47.912605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.255793+0000","last_change":"2024-03-06T07:27:30.252736+0000","last_active":"2024-03-06T07:27:30.255793+0000","last_peered":"2024-03-06T07:27:30.255793+0000","last_clean":"2024-03-06T07:27:30.255793+0000","last_became_active":"2024-03-06T07:27:30.252267+0000","last_became_peered":"2024-03-06T07:27:30.252267+0000","last_unstale":"2024-03-06T07:27:30.255793+0000","last_undegraded":"2024-03-06T07:27:30.255793+0000","last_fullsized":"2024-03-06T07:27:30.255793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:52.878333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695816+0000","last_change":"2024-03-06T07:27:30.265689+0000","last_active":"2024-03-06T07:27:31.695816+0000","last_peered":"2024-03-06T07:27:31.695816+0000","last_clean":"2024-03-06T07:27:31.695816+0000","last_became_active":"2024-03-06T07:27:30.265410+0000","last_became_peered":"2024-03-06T07:27:30.265410+0000","last_unstale":"2024-03-06T07:27:31.695816+0000","last_undegraded":"2024-03-06T07:27:31.695816+0000","last_fullsized":"2024-03-06T07:27:31.695816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:00.010236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242466+0000","last_change":"2024-03-06T07:27:30.238867+0000","last_active":"2024-03-06T07:27:31.242466+0000","last_peered":"2024-03-06T07:27:31.242466+0000","last_clean":"2024-03-06T07:27:31.242466+0000","last_became_active":"2024-03-06T07:27:30.238543+0000","last_became_peered":"2024-03-06T07:27:30.238543+0000","last_unstale":"2024-03-06T07:27:31.242466+0000","last_undegraded":"2024-03-06T07:27:31.242466+0000","last_fullsized":"2024-03-06T07:27:31.242466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.260931+0000","last_change":"2024-03-06T07:27:30.244526+0000","last_active":"2024-03-06T07:27:30.260931+0000","last_peered":"2024-03-06T07:27:30.260931+0000","last_clean":"2024-03-06T07:27:30.260931+0000","last_became_active":"2024-03-06T07:27:30.244266+0000","last_became_peered":"2024-03-06T07:27:30.244266+0000","last_unstale":"2024-03-06T07:27:30.260931+0000","last_undegraded":"2024-03-06T07:27:30.260931+0000","last_fullsized":"2024-03-06T07:27:30.260931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:15.978688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242632+0000","last_change":"2024-03-06T07:27:29.224405+0000","last_active":"2024-03-06T07:27:31.242632+0000","last_peered":"2024-03-06T07:27:31.242632+0000","last_clean":"2024-03-06T07:27:31.242632+0000","last_became_active":"2024-03-06T07:27:29.223485+0000","last_became_peered":"2024-03-06T07:27:29.223485+0000","last_unstale":"2024-03-06T07:27:31.242632+0000","last_undegraded":"2024-03-06T07:27:31.242632+0000","last_fullsized":"2024-03-06T07:27:31.242632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.257854+0000","last_change":"2024-03-06T07:27:30.249021+0000","last_active":"2024-03-06T07:27:30.257854+0000","last_peered":"2024-03-06T07:27:30.257854+0000","last_clean":"2024-03-06T07:27:30.257854+0000","last_became_active":"2024-03-06T07:27:30.248148+0000","last_became_peered":"2024-03-06T07:27:30.248148+0000","last_unstale":"2024-03-06T07:27:30.257854+0000","last_undegraded":"2024-03-06T07:27:30.257854+0000","last_fullsized":"2024-03-06T07:27:30.257854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:48.493765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243028+0000","last_change":"2024-03-06T07:27:29.230274+0000","last_active":"2024-03-06T07:27:31.243028+0000","last_peered":"2024-03-06T07:27:31.243028+0000","last_clean":"2024-03-06T07:27:31.243028+0000","last_became_active":"2024-03-06T07:27:29.230009+0000","last_became_peered":"2024-03-06T07:27:29.230009+0000","last_unstale":"2024-03-06T07:27:31.243028+0000","last_undegraded":"2024-03-06T07:27:31.243028+0000","last_fullsized":"2024-03-06T07:27:31.243028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241766+0000","last_change":"2024-03-06T07:27:29.221872+0000","last_active":"2024-03-06T07:27:31.241766+0000","last_peered":"2024-03-06T07:27:31.241766+0000","last_clean":"2024-03-06T07:27:31.241766+0000","last_became_active":"2024-03-06T07:27:29.221430+0000","last_became_peered":"2024-03-06T07:27:29.221430+0000","last_unstale":"2024-03-06T07:27:31.241766+0000","last_undegraded":"2024-03-06T07:27:31.241766+0000","last_fullsized":"2024-03-06T07:27:31.241766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241901+0000","last_change":"2024-03-06T07:27:30.235213+0000","last_active":"2024-03-06T07:27:31.241901+0000","last_peered":"2024-03-06T07:27:31.241901+0000","last_clean":"2024-03-06T07:27:31.241901+0000","last_became_active":"2024-03-06T07:27:30.234891+0000","last_became_peered":"2024-03-06T07:27:30.234891+0000","last_unstale":"2024-03-06T07:27:31.241901+0000","last_undegraded":"2024-03-06T07:27:31.241901+0000","last_fullsized":"2024-03-06T07:27:31.241901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.226087+0000","last_change":"2024-03-06T07:27:29.228355+0000","last_active":"2024-03-06T07:27:30.226087+0000","last_peered":"2024-03-06T07:27:30.226087+0000","last_clean":"2024-03-06T07:27:30.226087+0000","last_became_active":"2024-03-06T07:27:29.228054+0000","last_became_peered":"2024-03-06T07:27:29.228054+0000","last_unstale":"2024-03-06T07:27:30.226087+0000","last_undegraded":"2024-03-06T07:27:30.226087+0000","last_fullsized":"2024-03-06T07:27:30.226087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240414+0000","last_change":"2024-03-06T07:27:30.236691+0000","last_active":"2024-03-06T07:27:31.240414+0000","last_peered":"2024-03-06T07:27:31.240414+0000","last_clean":"2024-03-06T07:27:31.240414+0000","last_became_active":"2024-03-06T07:27:30.236318+0000","last_became_peered":"2024-03-06T07:27:30.236318+0000","last_unstale":"2024-03-06T07:27:31.240414+0000","last_undegraded":"2024-03-06T07:27:31.240414+0000","last_fullsized":"2024-03-06T07:27:31.240414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:06.911342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242168+0000","last_change":"2024-03-06T07:27:29.218901+0000","last_active":"2024-03-06T07:27:31.242168+0000","last_peered":"2024-03-06T07:27:31.242168+0000","last_clean":"2024-03-06T07:27:31.242168+0000","last_became_active":"2024-03-06T07:27:29.218549+0000","last_became_peered":"2024-03-06T07:27:29.218549+0000","last_unstale":"2024-03-06T07:27:31.242168+0000","last_undegraded":"2024-03-06T07:27:31.242168+0000","last_fullsized":"2024-03-06T07:27:31.242168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.271314+0000","last_change":"2024-03-06T07:27:30.269139+0000","last_active":"2024-03-06T07:27:30.271314+0000","last_peered":"2024-03-06T07:27:30.271314+0000","last_clean":"2024-03-06T07:27:30.271314+0000","last_became_active":"2024-03-06T07:27:30.265940+0000","last_became_peered":"2024-03-06T07:27:30.265940+0000","last_unstale":"2024-03-06T07:27:30.271314+0000","last_undegraded":"2024-03-06T07:27:30.271314+0000","last_fullsized":"2024-03-06T07:27:30.271314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.238792+0000","last_change":"2024-03-06T07:27:29.224828+0000","last_active":"2024-03-06T07:27:30.238792+0000","last_peered":"2024-03-06T07:27:30.238792+0000","last_clean":"2024-03-06T07:27:30.238792+0000","last_became_active":"2024-03-06T07:27:29.223899+0000","last_became_peered":"2024-03-06T07:27:29.223899+0000","last_unstale":"2024-03-06T07:27:30.238792+0000","last_undegraded":"2024-03-06T07:27:30.238792+0000","last_fullsized":"2024-03-06T07:27:30.238792+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261170+0000","last_change":"2024-03-06T07:27:30.258895+0000","last_active":"2024-03-06T07:27:30.261170+0000","last_peered":"2024-03-06T07:27:30.261170+0000","last_clean":"2024-03-06T07:27:30.261170+0000","last_became_active":"2024-03-06T07:27:30.258204+0000","last_became_peered":"2024-03-06T07:27:30.258204+0000","last_unstale":"2024-03-06T07:27:30.261170+0000","last_undegraded":"2024-03-06T07:27:30.261170+0000","last_fullsized":"2024-03-06T07:27:30.261170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:54.202774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239843+0000","last_change":"2024-03-06T07:27:29.232043+0000","last_active":"2024-03-06T07:27:31.239843+0000","last_peered":"2024-03-06T07:27:31.239843+0000","last_clean":"2024-03-06T07:27:31.239843+0000","last_became_active":"2024-03-06T07:27:29.231659+0000","last_became_peered":"2024-03-06T07:27:29.231659+0000","last_unstale":"2024-03-06T07:27:31.239843+0000","last_undegraded":"2024-03-06T07:27:31.239843+0000","last_fullsized":"2024-03-06T07:27:31.239843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262284+0000","last_change":"2024-03-06T07:27:30.253450+0000","last_active":"2024-03-06T07:27:30.262284+0000","last_peered":"2024-03-06T07:27:30.262284+0000","last_clean":"2024-03-06T07:27:30.262284+0000","last_became_active":"2024-03-06T07:27:30.250531+0000","last_became_peered":"2024-03-06T07:27:30.250531+0000","last_unstale":"2024-03-06T07:27:30.262284+0000","last_undegraded":"2024-03-06T07:27:30.262284+0000","last_fullsized":"2024-03-06T07:27:30.262284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:14.695885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242917+0000","last_change":"2024-03-06T07:27:29.216258+0000","last_active":"2024-03-06T07:27:31.242917+0000","last_peered":"2024-03-06T07:27:31.242917+0000","last_clean":"2024-03-06T07:27:31.242917+0000","last_became_active":"2024-03-06T07:27:29.215924+0000","last_became_peered":"2024-03-06T07:27:29.215924+0000","last_unstale":"2024-03-06T07:27:31.242917+0000","last_undegraded":"2024-03-06T07:27:31.242917+0000","last_fullsized":"2024-03-06T07:27:31.242917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266248+0000","last_change":"2024-03-06T07:27:30.248670+0000","last_active":"2024-03-06T07:27:30.266248+0000","last_peered":"2024-03-06T07:27:30.266248+0000","last_clean":"2024-03-06T07:27:30.266248+0000","last_became_active":"2024-03-06T07:27:30.248398+0000","last_became_peered":"2024-03-06T07:27:30.248398+0000","last_unstale":"2024-03-06T07:27:30.266248+0000","last_undegraded":"2024-03-06T07:27:30.266248+0000","last_fullsized":"2024-03-06T07:27:30.266248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:55.908106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.266659+0000","last_change":"2024-03-06T07:27:29.228193+0000","last_active":"2024-03-06T07:27:30.266659+0000","last_peered":"2024-03-06T07:27:30.266659+0000","last_clean":"2024-03-06T07:27:30.266659+0000","last_became_active":"2024-03-06T07:27:29.227840+0000","last_became_peered":"2024-03-06T07:27:29.227840+0000","last_unstale":"2024-03-06T07:27:30.266659+0000","last_undegraded":"2024-03-06T07:27:30.266659+0000","last_fullsized":"2024-03-06T07:27:30.266659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:21.775657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.243820+0000","last_change":"2024-03-06T07:27:30.240692+0000","last_active":"2024-03-06T07:27:30.243820+0000","last_peered":"2024-03-06T07:27:30.243820+0000","last_clean":"2024-03-06T07:27:30.243820+0000","last_became_active":"2024-03-06T07:27:30.240410+0000","last_became_peered":"2024-03-06T07:27:30.240410+0000","last_unstale":"2024-03-06T07:27:30.243820+0000","last_undegraded":"2024-03-06T07:27:30.243820+0000","last_fullsized":"2024-03-06T07:27:30.243820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:16.421421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.227276+0000","last_change":"2024-03-06T07:27:29.231958+0000","last_active":"2024-03-06T07:27:30.227276+0000","last_peered":"2024-03-06T07:27:30.227276+0000","last_clean":"2024-03-06T07:27:30.227276+0000","last_became_active":"2024-03-06T07:27:29.231690+0000","last_became_peered":"2024-03-06T07:27:29.231690+0000","last_unstale":"2024-03-06T07:27:30.227276+0000","last_undegraded":"2024-03-06T07:27:30.227276+0000","last_fullsized":"2024-03-06T07:27:30.227276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261652+0000","last_change":"2024-03-06T07:27:30.258434+0000","last_active":"2024-03-06T07:27:30.261652+0000","last_peered":"2024-03-06T07:27:30.261652+0000","last_clean":"2024-03-06T07:27:30.261652+0000","last_became_active":"2024-03-06T07:27:30.258094+0000","last_became_peered":"2024-03-06T07:27:30.258094+0000","last_unstale":"2024-03-06T07:27:30.261652+0000","last_undegraded":"2024-03-06T07:27:30.261652+0000","last_fullsized":"2024-03-06T07:27:30.261652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:42.209474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241544+0000","last_change":"2024-03-06T07:27:29.231347+0000","last_active":"2024-03-06T07:27:31.241544+0000","last_peered":"2024-03-06T07:27:31.241544+0000","last_clean":"2024-03-06T07:27:31.241544+0000","last_became_active":"2024-03-06T07:27:29.230852+0000","last_became_peered":"2024-03-06T07:27:29.230852+0000","last_unstale":"2024-03-06T07:27:31.241544+0000","last_undegraded":"2024-03-06T07:27:31.241544+0000","last_fullsized":"2024-03-06T07:27:31.241544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242563+0000","last_change":"2024-03-06T07:27:30.240275+0000","last_active":"2024-03-06T07:27:31.242563+0000","last_peered":"2024-03-06T07:27:31.242563+0000","last_clean":"2024-03-06T07:27:31.242563+0000","last_became_active":"2024-03-06T07:27:30.239945+0000","last_became_peered":"2024-03-06T07:27:30.239945+0000","last_unstale":"2024-03-06T07:27:31.242563+0000","last_undegraded":"2024-03-06T07:27:31.242563+0000","last_fullsized":"2024-03-06T07:27:31.242563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695318+0000","last_change":"2024-03-06T07:27:29.217898+0000","last_active":"2024-03-06T07:27:31.695318+0000","last_peered":"2024-03-06T07:27:31.695318+0000","last_clean":"2024-03-06T07:27:31.695318+0000","last_became_active":"2024-03-06T07:27:29.217605+0000","last_became_peered":"2024-03-06T07:27:29.217605+0000","last_unstale":"2024-03-06T07:27:31.695318+0000","last_undegraded":"2024-03-06T07:27:31.695318+0000","last_fullsized":"2024-03-06T07:27:31.695318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.683045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261715+0000","last_change":"2024-03-06T07:27:30.258444+0000","last_active":"2024-03-06T07:27:30.261715+0000","last_peered":"2024-03-06T07:27:30.261715+0000","last_clean":"2024-03-06T07:27:30.261715+0000","last_became_active":"2024-03-06T07:27:30.258119+0000","last_became_peered":"2024-03-06T07:27:30.258119+0000","last_unstale":"2024-03-06T07:27:30.261715+0000","last_undegraded":"2024-03-06T07:27:30.261715+0000","last_fullsized":"2024-03-06T07:27:30.261715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:17.318504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262526+0000","last_change":"2024-03-06T07:27:29.228474+0000","last_active":"2024-03-06T07:27:30.262526+0000","last_peered":"2024-03-06T07:27:30.262526+0000","last_clean":"2024-03-06T07:27:30.262526+0000","last_became_active":"2024-03-06T07:27:29.228168+0000","last_became_peered":"2024-03-06T07:27:29.228168+0000","last_unstale":"2024-03-06T07:27:30.262526+0000","last_undegraded":"2024-03-06T07:27:30.262526+0000","last_fullsized":"2024-03-06T07:27:30.262526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:36.193215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.254491+0000","last_change":"2024-03-06T07:27:30.243658+0000","last_active":"2024-03-06T07:27:30.254491+0000","last_peered":"2024-03-06T07:27:30.254491+0000","last_clean":"2024-03-06T07:27:30.254491+0000","last_became_active":"2024-03-06T07:27:30.243145+0000","last_became_peered":"2024-03-06T07:27:30.243145+0000","last_unstale":"2024-03-06T07:27:30.254491+0000","last_undegraded":"2024-03-06T07:27:30.254491+0000","last_fullsized":"2024-03-06T07:27:30.254491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:53.947928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695709+0000","last_change":"2024-03-06T07:27:29.221215+0000","last_active":"2024-03-06T07:27:31.695709+0000","last_peered":"2024-03-06T07:27:31.695709+0000","last_clean":"2024-03-06T07:27:31.695709+0000","last_became_active":"2024-03-06T07:27:29.220256+0000","last_became_peered":"2024-03-06T07:27:29.220256+0000","last_unstale":"2024-03-06T07:27:31.695709+0000","last_undegraded":"2024-03-06T07:27:31.695709+0000","last_fullsized":"2024-03-06T07:27:31.695709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261816+0000","last_change":"2024-03-06T07:27:30.258256+0000","last_active":"2024-03-06T07:27:30.261816+0000","last_peered":"2024-03-06T07:27:30.261816+0000","last_clean":"2024-03-06T07:27:30.261816+0000","last_became_active":"2024-03-06T07:27:30.255209+0000","last_became_peered":"2024-03-06T07:27:30.255209+0000","last_unstale":"2024-03-06T07:27:30.261816+0000","last_undegraded":"2024-03-06T07:27:30.261816+0000","last_fullsized":"2024-03-06T07:27:30.261816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:52.703156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240185+0000","last_change":"2024-03-06T07:27:29.225919+0000","last_active":"2024-03-06T07:27:31.240185+0000","last_peered":"2024-03-06T07:27:31.240185+0000","last_clean":"2024-03-06T07:27:31.240185+0000","last_became_active":"2024-03-06T07:27:29.225690+0000","last_became_peered":"2024-03-06T07:27:29.225690+0000","last_unstale":"2024-03-06T07:27:31.240185+0000","last_undegraded":"2024-03-06T07:27:31.240185+0000","last_fullsized":"2024-03-06T07:27:31.240185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269657+0000","last_change":"2024-03-06T07:27:30.259916+0000","last_active":"2024-03-06T07:27:30.269657+0000","last_peered":"2024-03-06T07:27:30.269657+0000","last_clean":"2024-03-06T07:27:30.269657+0000","last_became_active":"2024-03-06T07:27:30.259100+0000","last_became_peered":"2024-03-06T07:27:30.259100+0000","last_unstale":"2024-03-06T07:27:30.269657+0000","last_undegraded":"2024-03-06T07:27:30.269657+0000","last_fullsized":"2024-03-06T07:27:30.269657+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:44.205908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695658+0000","last_change":"2024-03-06T07:27:29.231688+0000","last_active":"2024-03-06T07:27:31.695658+0000","last_peered":"2024-03-06T07:27:31.695658+0000","last_clean":"2024-03-06T07:27:31.695658+0000","last_became_active":"2024-03-06T07:27:29.230816+0000","last_became_peered":"2024-03-06T07:27:29.230816+0000","last_unstale":"2024-03-06T07:27:31.695658+0000","last_undegraded":"2024-03-06T07:27:31.695658+0000","last_fullsized":"2024-03-06T07:27:31.695658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:51.657884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.267821+0000","last_change":"2024-03-06T07:27:30.264213+0000","last_active":"2024-03-06T07:27:30.267821+0000","last_peered":"2024-03-06T07:27:30.267821+0000","last_clean":"2024-03-06T07:27:30.267821+0000","last_became_active":"2024-03-06T07:27:30.263986+0000","last_became_peered":"2024-03-06T07:27:30.263986+0000","last_unstale":"2024-03-06T07:27:30.267821+0000","last_undegraded":"2024-03-06T07:27:30.267821+0000","last_fullsized":"2024-03-06T07:27:30.267821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.744486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":168,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.228124+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:27:30.228124+0000","last_peered":"2024-03-06T07:27:30.228124+0000","last_clean":"2024-03-06T07:27:30.228124+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:27:30.228124+0000","last_undegraded":"2024-03-06T07:27:30.228124+0000","last_fullsized":"2024-03-06T07:27:30.228124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.231617+0000","last_change":"2024-03-06T07:27:29.232294+0000","last_active":"2024-03-06T07:27:30.231617+0000","last_peered":"2024-03-06T07:27:30.231617+0000","last_clean":"2024-03-06T07:27:30.231617+0000","last_became_active":"2024-03-06T07:27:29.231839+0000","last_became_peered":"2024-03-06T07:27:29.231839+0000","last_unstale":"2024-03-06T07:27:30.231617+0000","last_undegraded":"2024-03-06T07:27:30.231617+0000","last_fullsized":"2024-03-06T07:27:30.231617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241928+0000","last_change":"2024-03-06T07:27:30.245626+0000","last_active":"2024-03-06T07:27:31.241928+0000","last_peered":"2024-03-06T07:27:31.241928+0000","last_clean":"2024-03-06T07:27:31.241928+0000","last_became_active":"2024-03-06T07:27:30.245324+0000","last_became_peered":"2024-03-06T07:27:30.245324+0000","last_unstale":"2024-03-06T07:27:31.241928+0000","last_undegraded":"2024-03-06T07:27:31.241928+0000","last_fullsized":"2024-03-06T07:27:31.241928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.228052+0000","last_change":"2024-03-06T07:27:29.234333+0000","last_active":"2024-03-06T07:27:30.228052+0000","last_peered":"2024-03-06T07:27:30.228052+0000","last_clean":"2024-03-06T07:27:30.228052+0000","last_became_active":"2024-03-06T07:27:29.234024+0000","last_became_peered":"2024-03-06T07:27:29.234024+0000","last_unstale":"2024-03-06T07:27:30.228052+0000","last_undegraded":"2024-03-06T07:27:30.228052+0000","last_fullsized":"2024-03-06T07:27:30.228052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.260516+0000","last_change":"2024-03-06T07:27:30.254664+0000","last_active":"2024-03-06T07:27:30.260516+0000","last_peered":"2024-03-06T07:27:30.260516+0000","last_clean":"2024-03-06T07:27:30.260516+0000","last_became_active":"2024-03-06T07:27:30.254371+0000","last_became_peered":"2024-03-06T07:27:30.254371+0000","last_unstale":"2024-03-06T07:27:30.260516+0000","last_undegraded":"2024-03-06T07:27:30.260516+0000","last_fullsized":"2024-03-06T07:27:30.260516+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:52.878333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695651+0000","last_change":"2024-03-06T07:27:29.220564+0000","last_active":"2024-03-06T07:27:31.695651+0000","last_peered":"2024-03-06T07:27:31.695651+0000","last_clean":"2024-03-06T07:27:31.695651+0000","last_became_active":"2024-03-06T07:27:29.220231+0000","last_became_peered":"2024-03-06T07:27:29.220231+0000","last_unstale":"2024-03-06T07:27:31.695651+0000","last_undegraded":"2024-03-06T07:27:31.695651+0000","last_fullsized":"2024-03-06T07:27:31.695651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:18.751461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.238983+0000","last_change":"2024-03-06T07:27:30.244146+0000","last_active":"2024-03-06T07:27:31.238983+0000","last_peered":"2024-03-06T07:27:31.238983+0000","last_clean":"2024-03-06T07:27:31.238983+0000","last_became_active":"2024-03-06T07:27:30.241811+0000","last_became_peered":"2024-03-06T07:27:30.241811+0000","last_unstale":"2024-03-06T07:27:31.238983+0000","last_undegraded":"2024-03-06T07:27:31.238983+0000","last_fullsized":"2024-03-06T07:27:31.238983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:52.703156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.227697+0000","last_change":"2024-03-06T07:27:29.221902+0000","last_active":"2024-03-06T07:27:30.227697+0000","last_peered":"2024-03-06T07:27:30.227697+0000","last_clean":"2024-03-06T07:27:30.227697+0000","last_became_active":"2024-03-06T07:27:29.221598+0000","last_became_peered":"2024-03-06T07:27:29.221598+0000","last_unstale":"2024-03-06T07:27:30.227697+0000","last_undegraded":"2024-03-06T07:27:30.227697+0000","last_fullsized":"2024-03-06T07:27:30.227697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.254349+0000","last_change":"2024-03-06T07:27:30.243366+0000","last_active":"2024-03-06T07:27:30.254349+0000","last_peered":"2024-03-06T07:27:30.254349+0000","last_clean":"2024-03-06T07:27:30.254349+0000","last_became_active":"2024-03-06T07:27:30.243054+0000","last_became_peered":"2024-03-06T07:27:30.243054+0000","last_unstale":"2024-03-06T07:27:30.254349+0000","last_undegraded":"2024-03-06T07:27:30.254349+0000","last_fullsized":"2024-03-06T07:27:30.254349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:00.926447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239297+0000","last_change":"2024-03-06T07:27:29.220848+0000","last_active":"2024-03-06T07:27:31.239297+0000","last_peered":"2024-03-06T07:27:31.239297+0000","last_clean":"2024-03-06T07:27:31.239297+0000","last_became_active":"2024-03-06T07:27:29.220557+0000","last_became_peered":"2024-03-06T07:27:29.220557+0000","last_unstale":"2024-03-06T07:27:31.239297+0000","last_undegraded":"2024-03-06T07:27:31.239297+0000","last_fullsized":"2024-03-06T07:27:31.239297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.269534+0000","last_change":"2024-03-06T07:27:30.259510+0000","last_active":"2024-03-06T07:27:30.269534+0000","last_peered":"2024-03-06T07:27:30.269534+0000","last_clean":"2024-03-06T07:27:30.269534+0000","last_became_active":"2024-03-06T07:27:30.258936+0000","last_became_peered":"2024-03-06T07:27:30.258936+0000","last_unstale":"2024-03-06T07:27:30.269534+0000","last_undegraded":"2024-03-06T07:27:30.269534+0000","last_fullsized":"2024-03-06T07:27:30.269534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:36:18.107539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.224951+0000","last_change":"2024-03-06T07:27:29.237638+0000","last_active":"2024-03-06T07:27:30.224951+0000","last_peered":"2024-03-06T07:27:30.224951+0000","last_clean":"2024-03-06T07:27:30.224951+0000","last_became_active":"2024-03-06T07:27:29.237304+0000","last_became_peered":"2024-03-06T07:27:29.237304+0000","last_unstale":"2024-03-06T07:27:30.224951+0000","last_undegraded":"2024-03-06T07:27:30.224951+0000","last_fullsized":"2024-03-06T07:27:30.224951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240309+0000","last_change":"2024-03-06T07:27:30.241439+0000","last_active":"2024-03-06T07:27:31.240309+0000","last_peered":"2024-03-06T07:27:31.240309+0000","last_clean":"2024-03-06T07:27:31.240309+0000","last_became_active":"2024-03-06T07:27:30.241163+0000","last_became_peered":"2024-03-06T07:27:30.241163+0000","last_unstale":"2024-03-06T07:27:31.240309+0000","last_undegraded":"2024-03-06T07:27:31.240309+0000","last_fullsized":"2024-03-06T07:27:31.240309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.744486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.253333+0000","last_change":"2024-03-06T07:27:29.223056+0000","last_active":"2024-03-06T07:27:31.253333+0000","last_peered":"2024-03-06T07:27:31.253333+0000","last_clean":"2024-03-06T07:27:31.253333+0000","last_became_active":"2024-03-06T07:27:29.222744+0000","last_became_peered":"2024-03-06T07:27:29.222744+0000","last_unstale":"2024-03-06T07:27:31.253333+0000","last_undegraded":"2024-03-06T07:27:31.253333+0000","last_fullsized":"2024-03-06T07:27:31.253333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243861+0000","last_change":"2024-03-06T07:27:30.241735+0000","last_active":"2024-03-06T07:27:31.243861+0000","last_peered":"2024-03-06T07:27:31.243861+0000","last_clean":"2024-03-06T07:27:31.243861+0000","last_became_active":"2024-03-06T07:27:30.241436+0000","last_became_peered":"2024-03-06T07:27:30.241436+0000","last_unstale":"2024-03-06T07:27:31.243861+0000","last_undegraded":"2024-03-06T07:27:31.243861+0000","last_fullsized":"2024-03-06T07:27:31.243861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:58.613904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242269+0000","last_change":"2024-03-06T07:27:29.226601+0000","last_active":"2024-03-06T07:27:31.242269+0000","last_peered":"2024-03-06T07:27:31.242269+0000","last_clean":"2024-03-06T07:27:31.242269+0000","last_became_active":"2024-03-06T07:27:29.225390+0000","last_became_peered":"2024-03-06T07:27:29.225390+0000","last_unstale":"2024-03-06T07:27:31.242269+0000","last_undegraded":"2024-03-06T07:27:31.242269+0000","last_fullsized":"2024-03-06T07:27:31.242269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.243578+0000","last_change":"2024-03-06T07:27:30.242063+0000","last_active":"2024-03-06T07:27:30.243578+0000","last_peered":"2024-03-06T07:27:30.243578+0000","last_clean":"2024-03-06T07:27:30.243578+0000","last_became_active":"2024-03-06T07:27:30.241737+0000","last_became_peered":"2024-03-06T07:27:30.241737+0000","last_unstale":"2024-03-06T07:27:30.243578+0000","last_undegraded":"2024-03-06T07:27:30.243578+0000","last_fullsized":"2024-03-06T07:27:30.243578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:30.484610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.227853+0000","last_change":"2024-03-06T07:27:29.235266+0000","last_active":"2024-03-06T07:27:30.227853+0000","last_peered":"2024-03-06T07:27:30.227853+0000","last_clean":"2024-03-06T07:27:30.227853+0000","last_became_active":"2024-03-06T07:27:29.234878+0000","last_became_peered":"2024-03-06T07:27:29.234878+0000","last_unstale":"2024-03-06T07:27:30.227853+0000","last_undegraded":"2024-03-06T07:27:30.227853+0000","last_fullsized":"2024-03-06T07:27:30.227853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.262179+0000","last_change":"2024-03-06T07:27:30.250799+0000","last_active":"2024-03-06T07:27:30.262179+0000","last_peered":"2024-03-06T07:27:30.262179+0000","last_clean":"2024-03-06T07:27:30.262179+0000","last_became_active":"2024-03-06T07:27:30.250475+0000","last_became_peered":"2024-03-06T07:27:30.250475+0000","last_unstale":"2024-03-06T07:27:30.262179+0000","last_undegraded":"2024-03-06T07:27:30.262179+0000","last_fullsized":"2024-03-06T07:27:30.262179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:52.670724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239430+0000","last_change":"2024-03-06T07:27:29.224306+0000","last_active":"2024-03-06T07:27:31.239430+0000","last_peered":"2024-03-06T07:27:31.239430+0000","last_clean":"2024-03-06T07:27:31.239430+0000","last_became_active":"2024-03-06T07:27:29.223923+0000","last_became_peered":"2024-03-06T07:27:29.223923+0000","last_unstale":"2024-03-06T07:27:31.239430+0000","last_undegraded":"2024-03-06T07:27:31.239430+0000","last_fullsized":"2024-03-06T07:27:31.239430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242506+0000","last_change":"2024-03-06T07:27:30.246857+0000","last_active":"2024-03-06T07:27:31.242506+0000","last_peered":"2024-03-06T07:27:31.242506+0000","last_clean":"2024-03-06T07:27:31.242506+0000","last_became_active":"2024-03-06T07:27:30.246519+0000","last_became_peered":"2024-03-06T07:27:30.246519+0000","last_unstale":"2024-03-06T07:27:31.242506+0000","last_undegraded":"2024-03-06T07:27:31.242506+0000","last_fullsized":"2024-03-06T07:27:31.242506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239222+0000","last_change":"2024-03-06T07:27:29.217568+0000","last_active":"2024-03-06T07:27:31.239222+0000","last_peered":"2024-03-06T07:27:31.239222+0000","last_clean":"2024-03-06T07:27:31.239222+0000","last_became_active":"2024-03-06T07:27:29.217204+0000","last_became_peered":"2024-03-06T07:27:29.217204+0000","last_unstale":"2024-03-06T07:27:31.239222+0000","last_undegraded":"2024-03-06T07:27:31.239222+0000","last_fullsized":"2024-03-06T07:27:31.239222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242343+0000","last_change":"2024-03-06T07:27:30.244509+0000","last_active":"2024-03-06T07:27:31.242343+0000","last_peered":"2024-03-06T07:27:31.242343+0000","last_clean":"2024-03-06T07:27:31.242343+0000","last_became_active":"2024-03-06T07:27:30.243804+0000","last_became_peered":"2024-03-06T07:27:30.243804+0000","last_unstale":"2024-03-06T07:27:31.242343+0000","last_undegraded":"2024-03-06T07:27:31.242343+0000","last_fullsized":"2024-03-06T07:27:31.242343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.263087+0000","last_change":"2024-03-06T07:27:29.235464+0000","last_active":"2024-03-06T07:27:30.263087+0000","last_peered":"2024-03-06T07:27:30.263087+0000","last_clean":"2024-03-06T07:27:30.263087+0000","last_became_active":"2024-03-06T07:27:29.234532+0000","last_became_peered":"2024-03-06T07:27:29.234532+0000","last_unstale":"2024-03-06T07:27:30.263087+0000","last_undegraded":"2024-03-06T07:27:30.263087+0000","last_fullsized":"2024-03-06T07:27:30.263087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242177+0000","last_change":"2024-03-06T07:27:30.257067+0000","last_active":"2024-03-06T07:27:31.242177+0000","last_peered":"2024-03-06T07:27:31.242177+0000","last_clean":"2024-03-06T07:27:31.242177+0000","last_became_active":"2024-03-06T07:27:30.256842+0000","last_became_peered":"2024-03-06T07:27:30.256842+0000","last_unstale":"2024-03-06T07:27:31.242177+0000","last_undegraded":"2024-03-06T07:27:31.242177+0000","last_fullsized":"2024-03-06T07:27:31.242177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270955+0000","last_change":"2024-03-06T07:27:30.266494+0000","last_active":"2024-03-06T07:27:30.270955+0000","last_peered":"2024-03-06T07:27:30.270955+0000","last_clean":"2024-03-06T07:27:30.270955+0000","last_became_active":"2024-03-06T07:27:30.265258+0000","last_became_peered":"2024-03-06T07:27:30.265258+0000","last_unstale":"2024-03-06T07:27:30.270955+0000","last_undegraded":"2024-03-06T07:27:30.270955+0000","last_fullsized":"2024-03-06T07:27:30.270955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.271387+0000","last_change":"2024-03-06T07:27:29.235012+0000","last_active":"2024-03-06T07:27:30.271387+0000","last_peered":"2024-03-06T07:27:30.271387+0000","last_clean":"2024-03-06T07:27:30.271387+0000","last_became_active":"2024-03-06T07:27:29.234638+0000","last_became_peered":"2024-03-06T07:27:29.234638+0000","last_unstale":"2024-03-06T07:27:30.271387+0000","last_undegraded":"2024-03-06T07:27:30.271387+0000","last_fullsized":"2024-03-06T07:27:30.271387+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270622+0000","last_change":"2024-03-06T07:27:30.265713+0000","last_active":"2024-03-06T07:27:30.270622+0000","last_peered":"2024-03-06T07:27:30.270622+0000","last_clean":"2024-03-06T07:27:30.270622+0000","last_became_active":"2024-03-06T07:27:30.259972+0000","last_became_peered":"2024-03-06T07:27:30.259972+0000","last_unstale":"2024-03-06T07:27:30.270622+0000","last_undegraded":"2024-03-06T07:27:30.270622+0000","last_fullsized":"2024-03-06T07:27:30.270622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.270292+0000","last_change":"2024-03-06T07:27:29.236520+0000","last_active":"2024-03-06T07:27:30.270292+0000","last_peered":"2024-03-06T07:27:30.270292+0000","last_clean":"2024-03-06T07:27:30.270292+0000","last_became_active":"2024-03-06T07:27:29.235298+0000","last_became_peered":"2024-03-06T07:27:29.235298+0000","last_unstale":"2024-03-06T07:27:30.270292+0000","last_undegraded":"2024-03-06T07:27:30.270292+0000","last_fullsized":"2024-03-06T07:27:30.270292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.261148+0000","last_change":"2024-03-06T07:27:30.258866+0000","last_active":"2024-03-06T07:27:30.261148+0000","last_peered":"2024-03-06T07:27:30.261148+0000","last_clean":"2024-03-06T07:27:30.261148+0000","last_became_active":"2024-03-06T07:27:30.258557+0000","last_became_peered":"2024-03-06T07:27:30.258557+0000","last_unstale":"2024-03-06T07:27:30.261148+0000","last_undegraded":"2024-03-06T07:27:30.261148+0000","last_fullsized":"2024-03-06T07:27:30.261148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:49.587341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243647+0000","last_change":"2024-03-06T07:27:29.221773+0000","last_active":"2024-03-06T07:27:31.243647+0000","last_peered":"2024-03-06T07:27:31.243647+0000","last_clean":"2024-03-06T07:27:31.243647+0000","last_became_active":"2024-03-06T07:27:29.221365+0000","last_became_peered":"2024-03-06T07:27:29.221365+0000","last_unstale":"2024-03-06T07:27:31.243647+0000","last_undegraded":"2024-03-06T07:27:31.243647+0000","last_fullsized":"2024-03-06T07:27:31.243647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.408581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243868+0000","last_change":"2024-03-06T07:27:30.245108+0000","last_active":"2024-03-06T07:27:31.243868+0000","last_peered":"2024-03-06T07:27:31.243868+0000","last_clean":"2024-03-06T07:27:31.243868+0000","last_became_active":"2024-03-06T07:27:30.244795+0000","last_became_peered":"2024-03-06T07:27:30.244795+0000","last_unstale":"2024-03-06T07:27:31.243868+0000","last_undegraded":"2024-03-06T07:27:31.243868+0000","last_fullsized":"2024-03-06T07:27:31.243868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.111110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:27:30.271690+0000","last_change":"2024-03-06T07:27:29.229397+0000","last_active":"2024-03-06T07:27:30.271690+0000","last_peered":"2024-03-06T07:27:30.271690+0000","last_clean":"2024-03-06T07:27:30.271690+0000","last_became_active":"2024-03-06T07:27:29.229097+0000","last_became_peered":"2024-03-06T07:27:29.229097+0000","last_unstale":"2024-03-06T07:27:30.271690+0000","last_undegraded":"2024-03-06T07:27:30.271690+0000","last_fullsized":"2024-03-06T07:27:30.271690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509445,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":662776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":73438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":221184,"data_stored":71952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":79738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":662832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:35.260 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:35.260 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:27:35.260 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:27:35.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:27:35.261 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:27:35.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.515+0000 7fe91a600640 1 Processor -- start 2024-03-06T07:27:35.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.515+0000 7fe91a600640 1 -- start start 2024-03-06T07:27:35.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe91a600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe9140d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe91a600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9140b3710 con 0x7fe91415e0b0 2024-03-06T07:27:35.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe91a600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9140b39f0 con 0x7fe9140d0c80 2024-03-06T07:27:35.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe91a600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140b09e0 con 0x7fe9140d4a10 2024-03-06T07:27:35.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe919200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe91415e0b0 0x7fe9140cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37432/0 (socket says 172.21.15.6:37432) 2024-03-06T07:27:35.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe919200640 1 -- 172.21.15.6:0/1212616538 learned_addr learned my addr 172.21.15.6:0/1212616538 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82368806 0 0) 0x7fe9140b3710 con 0x7fe91415e0b0 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f8003520 con 0x7fe91415e0b0 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.517+0000 7fe913e00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe9140d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41742401 0 0) 0x7fe8f8003520 con 0x7fe91415e0b0 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 msgr2=0x7fe9140d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe9140d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 >> v1:172.21.15.178:6789/0 conn(0x7fe9140d0c80 legacy=0x7fe9140b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140b3cd0 con 0x7fe91415e0b0 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fe8fc001670 con 0x7fe91415e0b0 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 >> v1:172.21.15.6:6789/0 conn(0x7fe91415e0b0 legacy=0x7fe9140cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8f80039e0 0x7fe8f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe8f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8f8004d40 0x7fe8f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f800bb50 con 0x7fe8f8004d40 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f800bcd0 con 0x7fe8f80039e0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f800bea0 con 0x7fe9140d4a10 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe919c00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8f8004d40 0x7fe8f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe919200640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8f80039e0 0x7fe8f8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe913e00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe8f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe919c00640 1 -- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 msgr2=0x7fe8f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe919c00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe8f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe919c00640 1 -- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8f80039e0 msgr2=0x7fe8f8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.518+0000 7fe919c00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8f80039e0 0x7fe8f8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.519+0000 7fe919c00640 1 -- 172.21.15.6:0/1212616538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f8003520 con 0x7fe8f8004d40 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.519+0000 7fe913e00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9140d4a10 0x7fe8f8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe919c00640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8f8004d40 0x7fe8f80051c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe90c001780 tx=0x7fe8f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9c823539e06ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c001370 con 0x7fe8f8004d40 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c0052d0 con 0x7fe8f8004d40 2024-03-06T07:27:35.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe913400640 1 -- 172.21.15.6:0/1212616538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c003830 con 0x7fe8f8004d40 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe91a600640 1 -- 172.21.15.6:0/1212616538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8f8004d40 msgr2=0x7fe8f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe91a600640 1 --2- 172.21.15.6:0/1212616538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8f8004d40 0x7fe8f80051c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe90c001780 tx=0x7fe8f8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:35.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe919c00640 1 -- 172.21.15.6:0/1212616538 reap_dead start 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe91a600640 1 -- 172.21.15.6:0/1212616538 shutdown_connections 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.520+0000 7fe91a600640 1 -- 172.21.15.6:0/1212616538 >> 172.21.15.6:0/1212616538 conn(0x7fe9140d3d30 msgr2=0x7fe9140d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.521+0000 7fe91a600640 1 -- 172.21.15.6:0/1212616538 shutdown_connections 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.521+0000 7fe91a600640 1 -- 172.21.15.6:0/1212616538 wait complete. 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.522+0000 7fe91a600640 1 Processor -- start 2024-03-06T07:27:35.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.523+0000 7fe91a600640 1 -- start start 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.523+0000 7fe91a600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8f8004d40 0x7fe91414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.523+0000 7fe91a600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe91415b550 0x7fe9140b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.523+0000 7fe91a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 0x7fe9140cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.523+0000 7fe91a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91415e490 con 0x7fe9140cd3f0 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe91a600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140cfa20 con 0x7fe91415b550 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe91a600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9140cfc60 con 0x7fe8f8004d40 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 0x7fe9140cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 0x7fe9140cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42210/0 (socket says 172.21.15.6:42210) 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 -- 172.21.15.6:0/3351010631 learned_addr learned my addr 172.21.15.6:0/3351010631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe913e00640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe91415b550 0x7fe9140b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919200640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8f8004d40 0x7fe91414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 -- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8f8004d40 msgr2=0x7fe91414f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8f8004d40 0x7fe91414f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 -- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe91415b550 msgr2=0x7fe9140b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe91415b550 0x7fe9140b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 -- 172.21.15.6:0/3351010631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9140cc350 con 0x7fe9140cd3f0 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe913e00640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe91415b550 0x7fe9140b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.524+0000 7fe919c00640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 0x7fe9140cd870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe90c005040 tx=0x7fe90c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.525+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c001370 con 0x7fe9140cd3f0 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.525+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90c005480 con 0x7fe9140cd3f0 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.525+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9140cc630 con 0x7fe9140cd3f0 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.525+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90c007a30 con 0x7fe9140cd3f0 2024-03-06T07:27:35.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.525+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9140ccbf0 con 0x7fe9140cd3f0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.526+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91404f330 con 0x7fe9140cd3f0 2024-03-06T07:27:35.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.527+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe90c046020 con 0x7fe9140cd3f0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.528+0000 7fe910c00640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe8f006b4c0 0x7fe8f006d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.528+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe90c0ab070 con 0x7fe9140cd3f0 2024-03-06T07:27:35.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.528+0000 7fe919200640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe8f006b4c0 0x7fe8f006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:35.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.529+0000 7fe919200640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe8f006b4c0 0x7fe8f006d980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc001eb0 tx=0x7fe8fc001dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:35.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.532+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe90c04d020 con 0x7fe9140cd3f0 2024-03-06T07:27:35.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.781+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe914161640 con 0x7fe9140cd3f0 2024-03-06T07:27:35.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.785+0000 7fe910c00640 1 -- 172.21.15.6:0/3351010631 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171791 (secure 0 0 0) 0x7fe90c079e50 con 0x7fe9140cd3f0 2024-03-06T07:27:35.790 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:35.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.789+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe8f006b4c0 msgr2=0x7fe8f006d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.789+0000 7fe91a600640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe8f006b4c0 0x7fe8f006d980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc001eb0 tx=0x7fe8fc001dc0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.789+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 msgr2=0x7fe9140cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.789+0000 7fe91a600640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 0x7fe9140cd870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe90c005040 tx=0x7fe90c001120 comp rx=0 tx=0).stop 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 shutdown_connections 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe8f006b4c0 0x7fe8f006d980 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe91415b550 0x7fe9140b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9140cd3f0 0x7fe9140cd870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 --2- 172.21.15.6:0/3351010631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8f8004d40 0x7fe91414f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 >> 172.21.15.6:0/3351010631 conn(0x7fe9140d3d30 msgr2=0x7fe9140b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 shutdown_connections 2024-03-06T07:27:35.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:35.790+0000 7fe91a600640 1 -- 172.21.15.6:0/3351010631 wait complete. 2024-03-06T07:27:35.808 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":3849979937},{"type":"v1","addr":"172.21.15.178:6833","nonce":3849979937}]},"active_addr":"172.21.15.178:6833/3849979937","active_change":"2024-03-06T07:27:09.417456+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":4190444168}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":1563615027}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":3879980522}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.178:0","nonce":408284990}]}]} 2024-03-06T07:27:35.811 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:27:35.811 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:27:35.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:36.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.025+0000 7f3746c00640 1 Processor -- start 2024-03-06T07:27:36.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.025+0000 7f3746c00640 1 -- start start 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3746c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f37400af580 0x7f374015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3746c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37400cdc20 0x7f37400b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3746c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374015cc40 con 0x7f37400af1a0 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3746c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374015cf20 con 0x7f37400cdc20 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3746c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374015d200 con 0x7f37400af580 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3744c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37400cdc20 0x7f37400b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3744c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37400cdc20 0x7f37400b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42218/0 (socket says 172.21.15.6:42218) 2024-03-06T07:27:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f3744c00640 1 -- 172.21.15.6:0/834052533 learned_addr learned my addr 172.21.15.6:0/834052533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.027+0000 7f373f400640 1 --2- 172.21.15.6:0/834052533 >> v2:172.21.15.178:3300/0 conn(0x7f37400af580 0x7f374015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.028+0000 7f3744c00640 1 -- 172.21.15.6:0/834052533 >> v2:172.21.15.178:3300/0 conn(0x7f37400af580 msgr2=0x7f374015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.028+0000 7f3744c00640 1 --2- 172.21.15.6:0/834052533 >> v2:172.21.15.178:3300/0 conn(0x7f37400af580 0x7f374015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.028+0000 7f3744c00640 1 -- 172.21.15.6:0/834052533 >> v1:172.21.15.178:6789/0 conn(0x7f37400af1a0 legacy=0x7f37400b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.028+0000 7f3744c00640 1 -- 172.21.15.6:0/834052533 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374015d4e0 con 0x7f37400cdc20 2024-03-06T07:27:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.029+0000 7f3744c00640 1 --2- 172.21.15.6:0/834052533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400cdc20 0x7f37400b2dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37400b54c0 tx=0x7f373802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5367a8c4275470ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.029+0000 7f373ea00640 1 -- 172.21.15.6:0/834052533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373802ca80 con 0x7f37400cdc20 2024-03-06T07:27:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.029+0000 7f373ea00640 1 -- 172.21.15.6:0/834052533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373802cc20 con 0x7f37400cdc20 2024-03-06T07:27:36.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.029+0000 7f373ea00640 1 -- 172.21.15.6:0/834052533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380343f0 con 0x7f37400cdc20 2024-03-06T07:27:36.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 -- 172.21.15.6:0/834052533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400cdc20 msgr2=0x7f37400b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 --2- 172.21.15.6:0/834052533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400cdc20 0x7f37400b2dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37400b54c0 tx=0x7f373802bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 -- 172.21.15.6:0/834052533 shutdown_connections 2024-03-06T07:27:36.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 --2- 172.21.15.6:0/834052533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400cdc20 0x7f37400b2dd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 --2- 172.21.15.6:0/834052533 >> v2:172.21.15.178:3300/0 conn(0x7f37400af580 0x7f374015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 -- 172.21.15.6:0/834052533 >> 172.21.15.6:0/834052533 conn(0x7f374006afb0 msgr2=0x7f374006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 -- 172.21.15.6:0/834052533 shutdown_connections 2024-03-06T07:27:36.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.030+0000 7f3746c00640 1 -- 172.21.15.6:0/834052533 wait complete. 2024-03-06T07:27:36.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.032+0000 7f3746c00640 1 Processor -- start 2024-03-06T07:27:36.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.032+0000 7f3746c00640 1 -- start start 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f3746c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 0x7f3740149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f3746c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400af580 0x7f37400c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f3746c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37400cdc20 0x7f37400b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f3746c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37400b36a0 con 0x7f37400af1a0 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f3746c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37400b3520 con 0x7f37400af580 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f3746c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37400b3820 con 0x7f37400cdc20 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f373fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 0x7f3740149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f373fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 0x7f3740149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42234/0 (socket says 172.21.15.6:42234) 2024-03-06T07:27:36.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f373fe00640 1 -- 172.21.15.6:0/840560865 learned_addr learned my addr 172.21.15.6:0/840560865 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.033+0000 7f373f400640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400af580 0x7f37400c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f373fe00640 1 -- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37400cdc20 msgr2=0x7f37400b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f373fe00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37400cdc20 0x7f37400b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f373fe00640 1 -- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400af580 msgr2=0x7f37400c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f373fe00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400af580 0x7f37400c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f373fe00640 1 -- 172.21.15.6:0/840560865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374014be80 con 0x7f37400af1a0 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f373fe00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 0x7f3740149240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f373000c450 tx=0x7f373000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300052d0 con 0x7f37400af1a0 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3730005470 con 0x7f37400af1a0 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730012870 con 0x7f37400af1a0 2024-03-06T07:27:36.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374014c160 con 0x7f37400af1a0 2024-03-06T07:27:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.034+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374014c720 con 0x7f37400af1a0 2024-03-06T07:27:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.035+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3740073050 con 0x7f37400af1a0 2024-03-06T07:27:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.036+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3730012a10 con 0x7f37400af1a0 2024-03-06T07:27:36.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.037+0000 7f3737e00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f372006b4d0 0x7f372006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.037+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3730089fc0 con 0x7f37400af1a0 2024-03-06T07:27:36.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.038+0000 7f373f400640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f372006b4d0 0x7f372006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.039+0000 7f373f400640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f372006b4d0 0x7f372006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3740157ed0 tx=0x7f372c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.042+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3740073050 con 0x7f37400af1a0 2024-03-06T07:27:36.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.240+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f374004f3a0 con 0x7f37400af1a0 2024-03-06T07:27:36.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.243+0000 7f3737e00640 1 -- 172.21.15.6:0/840560865 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f3730058d10 con 0x7f37400af1a0 2024-03-06T07:27:36.245 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:36.245 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f372006b4d0 msgr2=0x7f372006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f372006b4d0 0x7f372006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3740157ed0 tx=0x7f372c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 msgr2=0x7f3740149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 0x7f3740149240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f373000c450 tx=0x7f373000c910 comp rx=0 tx=0).stop 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 shutdown_connections 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f372006b4d0 0x7f372006d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400af580 0x7f37400c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37400cdc20 0x7f37400b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 --2- 172.21.15.6:0/840560865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37400af1a0 0x7f3740149240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.245+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 >> 172.21.15.6:0/840560865 conn(0x7f374006afb0 msgr2=0x7f3740071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.246+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 shutdown_connections 2024-03-06T07:27:36.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.246+0000 7f3746c00640 1 -- 172.21.15.6:0/840560865 wait complete. 2024-03-06T07:27:36.268 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:27:36.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:36.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.482+0000 7f413de00640 1 Processor -- start 2024-03-06T07:27:36.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.482+0000 7f413de00640 1 -- start start 2024-03-06T07:27:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.483+0000 7f413de00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f41380af1a0 0x7f413815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.483+0000 7f413de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f41380b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.483+0000 7f413de00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413815cc40 con 0x7f41380af580 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.483+0000 7f413de00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413815cf20 con 0x7f41380af1a0 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.483+0000 7f413de00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413815d200 con 0x7f41380cdc20 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.484+0000 7f4136a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f41380af1a0 0x7f413815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.484+0000 7f4137400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f41380b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.484+0000 7f4136a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f41380af1a0 0x7f413815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:33738/0 (socket says 172.21.15.6:33738) 2024-03-06T07:27:36.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.484+0000 7f4136a00640 1 -- 172.21.15.6:0/2894885713 learned_addr learned my addr 172.21.15.6:0/2894885713 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:36.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.485+0000 7f4136a00640 1 -- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 msgr2=0x7f41380b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:36.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.485+0000 7f4136a00640 1 --2- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f41380b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.485+0000 7f4136a00640 1 -- 172.21.15.6:0/2894885713 >> v1:172.21.15.178:6789/0 conn(0x7f41380af580 legacy=0x7f41380b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.485+0000 7f4136a00640 1 -- 172.21.15.6:0/2894885713 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413815d4e0 con 0x7f41380af1a0 2024-03-06T07:27:36.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.487+0000 7f4136a00640 1 --2- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af1a0 0x7f413815c600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4128009870 tx=0x7f412802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61c523ada43fc52f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.487+0000 7f4135600640 1 -- 172.21.15.6:0/2894885713 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128035020 con 0x7f41380af1a0 2024-03-06T07:27:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.487+0000 7f4135600640 1 -- 172.21.15.6:0/2894885713 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4128031640 con 0x7f41380af1a0 2024-03-06T07:27:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.487+0000 7f4135600640 1 -- 172.21.15.6:0/2894885713 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412802d040 con 0x7f41380af1a0 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 -- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af1a0 msgr2=0x7f413815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 --2- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af1a0 0x7f413815c600 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4128009870 tx=0x7f412802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 -- 172.21.15.6:0/2894885713 shutdown_connections 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 --2- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f41380b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 --2- 172.21.15.6:0/2894885713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af1a0 0x7f413815c600 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 -- 172.21.15.6:0/2894885713 >> 172.21.15.6:0/2894885713 conn(0x7f413806afb0 msgr2=0x7f413806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 -- 172.21.15.6:0/2894885713 shutdown_connections 2024-03-06T07:27:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.488+0000 7f413de00640 1 -- 172.21.15.6:0/2894885713 wait complete. 2024-03-06T07:27:36.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.490+0000 7f413de00640 1 Processor -- start 2024-03-06T07:27:36.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.491+0000 7f413de00640 1 -- start start 2024-03-06T07:27:36.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f413de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 0x7f41381488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f413de00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af580 0x7f4138151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f413de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f413815e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f413de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41380b3720 con 0x7f41380af1a0 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f413de00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41380b35a0 con 0x7f41380af580 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f413de00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4138155e70 con 0x7f41380cdc20 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f4136a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 0x7f41381488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f4136a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 0x7f41381488a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42246/0 (socket says 172.21.15.6:42246) 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f4136a00640 1 -- 172.21.15.6:0/1240182427 learned_addr learned my addr 172.21.15.6:0/1240182427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.492+0000 7f4136000640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af580 0x7f4138151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4137400640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f413815e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4136a00640 1 -- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 msgr2=0x7f413815e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4136a00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f413815e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4136a00640 1 -- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af580 msgr2=0x7f4138151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4136a00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af580 0x7f4138151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4136a00640 1 -- 172.21.15.6:0/1240182427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413815d8a0 con 0x7f41380af1a0 2024-03-06T07:27:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f4136a00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 0x7f41381488a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4128009990 tx=0x7f41280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128040070 con 0x7f41380af1a0 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412803e070 con 0x7f41380af1a0 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128035040 con 0x7f41380af1a0 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.493+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413815db80 con 0x7f41380af1a0 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.494+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413815e140 con 0x7f41380af1a0 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.495+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4138069510 con 0x7f41380af1a0 2024-03-06T07:27:36.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.496+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4128038020 con 0x7f41380af1a0 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.496+0000 7f412f400640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f411406b500 0x7f411406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:36.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.496+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41280abbd0 con 0x7f41380af1a0 2024-03-06T07:27:36.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.497+0000 7f4136000640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f411406b500 0x7f411406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:36.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.498+0000 7f4136000640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f411406b500 0x7f411406d9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f413815f810 tx=0x7f4120009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:36.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.501+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f412807a9b0 con 0x7f41380af1a0 2024-03-06T07:27:36.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.701+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f413806b670 con 0x7f41380af1a0 2024-03-06T07:27:36.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.702+0000 7f412f400640 1 -- 172.21.15.6:0/1240182427 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f412807a2b0 con 0x7f41380af1a0 2024-03-06T07:27:36.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:36.705 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.704+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f411406b500 msgr2=0x7f411406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.704+0000 7f413de00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f411406b500 0x7f411406d9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f413815f810 tx=0x7f4120009040 comp rx=0 tx=0).stop 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 msgr2=0x7f41381488a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 0x7f41381488a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4128009990 tx=0x7f41280046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 shutdown_connections 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f411406b500 0x7f411406d9c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f41380af580 0x7f4138151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41380cdc20 0x7f413815e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 --2- 172.21.15.6:0/1240182427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41380af1a0 0x7f41381488a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:36.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 >> 172.21.15.6:0/1240182427 conn(0x7f413806afb0 msgr2=0x7f41380ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:36.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 shutdown_connections 2024-03-06T07:27:36.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:36.705+0000 7f413de00640 1 -- 172.21.15.6:0/1240182427 wait complete. 2024-03-06T07:27:36.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:27:36.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:27:36.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:27:36.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:27:36.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:27:36.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:27:36.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:27:36.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:27:37.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.027+0000 7fe002400640 1 Processor -- start 2024-03-06T07:27:37.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.027+0000 7f1630a00640 1 Processor -- start 2024-03-06T07:27:37.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.028+0000 7fe002400640 1 -- start start 2024-03-06T07:27:37.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.029+0000 7f40e9800640 1 Processor -- start 2024-03-06T07:27:37.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7fe002400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdff40e3420 0x7fdff403c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.028+0000 7f1630a00640 1 -- start start 2024-03-06T07:27:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7f1630a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f162c15a210 0x7f162c0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7f1630a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f162c06d440 0x7f162c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7f1630a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c06d900 con 0x7f162c06e5e0 2024-03-06T07:27:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7f1630a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162c06dbe0 con 0x7f162c15a210 2024-03-06T07:27:37.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7f1630a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162c0cddc0 con 0x7f162c06d440 2024-03-06T07:27:37.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.031+0000 7f1629e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f162c15a210 0x7f162c0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.031+0000 7f1629e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f162c15a210 0x7f162c0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42264/0 (socket says 172.21.15.6:42264) 2024-03-06T07:27:37.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.031+0000 7f1629e00640 1 -- 172.21.15.6:0/882941534 learned_addr learned my addr 172.21.15.6:0/882941534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7fe002400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff4058980 con 0x7fdff40e78d0 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7fe002400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff4058c60 con 0x7fdff40e3040 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7fe002400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdff4006860 con 0x7fdff40e3420 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7fe001a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdff40e78d0 0x7fdff4053d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37438/0 (socket says 172.21.15.6:37438) 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7fe001a00640 1 -- 172.21.15.6:0/4163713615 learned_addr learned my addr 172.21.15.6:0/4163713615 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710658085 0 0) 0x7fdff4058980 con 0x7fdff40e78d0 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe4003520 con 0x7fdff40e78d0 2024-03-06T07:27:37.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7fdffbe00640 1 --2- 172.21.15.6:0/4163713615 >> v2:172.21.15.6:3300/0 conn(0x7fdff40e3420 0x7fdff403c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7f1629e00640 1 -- 172.21.15.6:0/882941534 >> v2:172.21.15.178:3300/0 conn(0x7f162c06d440 msgr2=0x7f162c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7f1629e00640 1 --2- 172.21.15.6:0/882941534 >> v2:172.21.15.178:3300/0 conn(0x7f162c06d440 0x7f162c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7f1629e00640 1 -- 172.21.15.6:0/882941534 >> v1:172.21.15.6:6789/0 conn(0x7f162c06e5e0 legacy=0x7f162c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7f1629e00640 1 -- 172.21.15.6:0/882941534 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c0acdb0 con 0x7f162c15a210 2024-03-06T07:27:37.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620735193 0 0) 0x7fdff4058c60 con 0x7fdff40e3040 2024-03-06T07:27:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe4003880 con 0x7fdff40e3040 2024-03-06T07:27:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7fdffbe00640 1 -- 172.21.15.6:0/4163713615 >> v1:172.21.15.178:6789/0 conn(0x7fdff40e3040 legacy=0x7fdff4058130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7fdffbe00640 1 -- 172.21.15.6:0/4163713615 >> v1:172.21.15.6:6789/0 conn(0x7fdff40e78d0 legacy=0x7fdff4053d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7fdffbe00640 1 -- 172.21.15.6:0/4163713615 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff4006210 con 0x7fdff40e3420 2024-03-06T07:27:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.030+0000 7f40e9800640 1 -- start start 2024-03-06T07:27:37.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7f1629e00640 1 --2- 172.21.15.6:0/882941534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c0cbfc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f162c06ec50 tx=0x7f161c02b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2524ba9e881db6b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7f40e9800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f40e414bb40 0x7f40e406db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.032+0000 7f40e9800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40b4750 0x7f40e406e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7f40e9800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f40e40719e0 0x7f40e4071e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7f40e9800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e40b3460 con 0x7f40e414bb40 2024-03-06T07:27:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7f40e9800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40e4072370 con 0x7f40e40719e0 2024-03-06T07:27:37.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.033+0000 7f40e9800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40b36b0 con 0x7f40e40b4750 2024-03-06T07:27:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7f40e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f40e414bb40 0x7f40e406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7f40e2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f40e414bb40 0x7f40e406db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42276/0 (socket says 172.21.15.6:42276) 2024-03-06T07:27:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7f40e2a00640 1 -- 172.21.15.6:0/4094878941 learned_addr learned my addr 172.21.15.6:0/4094878941 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.034+0000 7f40e2000640 1 --2- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40b4750 0x7f40e406e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f1628a00640 1 -- 172.21.15.6:0/882941534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c037070 con 0x7f162c15a210 2024-03-06T07:27:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f40e3400640 1 --2- 172.21.15.6:0/4094878941 >> v2:172.21.15.178:3300/0 conn(0x7f40e40719e0 0x7f40e4071e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f1628a00640 1 -- 172.21.15.6:0/882941534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f161c030710 con 0x7f162c15a210 2024-03-06T07:27:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f1628a00640 1 -- 172.21.15.6:0/882941534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c030b20 con 0x7f162c15a210 2024-03-06T07:27:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f40e3400640 1 -- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40b4750 msgr2=0x7f40e406e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f40e3400640 1 --2- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40b4750 0x7f40e406e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f40e3400640 1 -- 172.21.15.6:0/4094878941 >> v2:172.21.15.6:3300/0 conn(0x7f40e414bb40 msgr2=0x7f40e406db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.035+0000 7f40e3400640 1 --2- 172.21.15.6:0/4094878941 >> v2:172.21.15.6:3300/0 conn(0x7f40e414bb40 0x7f40e406db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.036+0000 7f40e3400640 1 -- 172.21.15.6:0/4094878941 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e40b3990 con 0x7f40e40719e0 2024-03-06T07:27:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.036+0000 7f1630a00640 1 -- 172.21.15.6:0/882941534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 msgr2=0x7f162c0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.036+0000 7fdffbe00640 1 --2- 172.21.15.6:0/4163713615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff403c180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfe80097a0 tx=0x7fdfe802b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be83179a26fbb2c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7f1630a00640 1 --2- 172.21.15.6:0/882941534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c0cbfc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f162c06ec50 tx=0x7f161c02b770 comp rx=0 tx=0).stop 2024-03-06T07:27:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8037070 con 0x7fdff40e3420 2024-03-06T07:27:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe8030510 con 0x7fdff40e3420 2024-03-06T07:27:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7fdffb400640 1 -- 172.21.15.6:0/4163713615 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff0001650 con 0x7fdff40e3420 2024-03-06T07:27:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7f1630a00640 1 -- 172.21.15.6:0/882941534 shutdown_connections 2024-03-06T07:27:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7f1630a00640 1 --2- 172.21.15.6:0/882941534 >> v2:172.21.15.178:3300/0 conn(0x7f162c06d440 0x7f162c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7f1630a00640 1 --2- 172.21.15.6:0/882941534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c0cbfc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7f1630a00640 1 -- 172.21.15.6:0/882941534 >> 172.21.15.6:0/882941534 conn(0x7f162c06afb0 msgr2=0x7f162c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.037+0000 7f40e3400640 1 --2- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40719e0 0x7f40e4071e30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40e4070a70 tx=0x7f40dc02be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d8895f72e3ac90e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7f1630a00640 1 -- 172.21.15.6:0/882941534 shutdown_connections 2024-03-06T07:27:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 -- 172.21.15.6:0/4163713615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 msgr2=0x7fdff403c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 --2- 172.21.15.6:0/4163713615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff403c180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdfe80097a0 tx=0x7fdfe802b6b0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7f1630a00640 1 -- 172.21.15.6:0/882941534 wait complete. 2024-03-06T07:27:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fc020200640 1 Processor -- start 2024-03-06T07:27:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7f40e1600640 1 -- 172.21.15.6:0/4094878941 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc039070 con 0x7f40e40719e0 2024-03-06T07:27:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7f40e1600640 1 -- 172.21.15.6:0/4094878941 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40dc02cbe0 con 0x7f40e40719e0 2024-03-06T07:27:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 -- 172.21.15.6:0/4163713615 shutdown_connections 2024-03-06T07:27:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 --2- 172.21.15.6:0/4163713615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff403c180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 -- 172.21.15.6:0/4163713615 >> 172.21.15.6:0/4163713615 conn(0x7fdff403aa40 msgr2=0x7fdff403ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 -- 172.21.15.6:0/4163713615 shutdown_connections 2024-03-06T07:27:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7f40e1600640 1 -- 172.21.15.6:0/4094878941 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40dc034440 con 0x7f40e40719e0 2024-03-06T07:27:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fe002400640 1 -- 172.21.15.6:0/4163713615 wait complete. 2024-03-06T07:27:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7f8f0fe00640 1 Processor -- start 2024-03-06T07:27:37.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7f8f0fe00640 1 -- start start 2024-03-06T07:27:37.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.039+0000 7f40e9800640 1 -- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40719e0 msgr2=0x7f40e4071e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.039+0000 7f40e9800640 1 --2- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40719e0 0x7f40e4071e30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f40e4070a70 tx=0x7f40dc02be30 comp rx=0 tx=0).stop 2024-03-06T07:27:37.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.038+0000 7fc020200640 1 -- start start 2024-03-06T07:27:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f1630a00640 1 Processor -- start 2024-03-06T07:27:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7fc020200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc018070ee0 0x7fc018162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7fc020200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc018072da0 0x7fc0180cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7fc020200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0180ce090 con 0x7fc0180712c0 2024-03-06T07:27:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7fc020200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0180ceca0 con 0x7fc018072da0 2024-03-06T07:27:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.040+0000 7fc020200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180ce370 con 0x7fc018070ee0 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7fe002400640 1 Processor -- start 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7fe002400640 1 -- start start 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f40e9800640 1 -- 172.21.15.6:0/4094878941 shutdown_connections 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f40e9800640 1 --2- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40719e0 0x7f40e4071e30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f40e9800640 1 --2- 172.21.15.6:0/4094878941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40b4750 0x7f40e406e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f40e9800640 1 --2- 172.21.15.6:0/4094878941 >> v2:172.21.15.6:3300/0 conn(0x7f40e414bb40 0x7f40e406db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f40e9800640 1 -- 172.21.15.6:0/4094878941 >> 172.21.15.6:0/4094878941 conn(0x7f40e40b1750 msgr2=0x7f40e40b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7fc01e200640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fc0180712c0 0x7fc01806e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37098/0 (socket says 172.21.15.6:37098) 2024-03-06T07:27:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7fc01e200640 1 -- 172.21.15.6:0/1362126810 learned_addr learned my addr 172.21.15.6:0/1362126810 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7fc01ce00640 1 --2- 172.21.15.6:0/1362126810 >> v2:172.21.15.6:3300/0 conn(0x7fc018072da0 0x7fc0180cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f080361b0 0x7f8f08039b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f8f0fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0803b340 con 0x7f8f080529f0 2024-03-06T07:27:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f8f0fe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0803b5e0 con 0x7f8f0804a320 2024-03-06T07:27:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f8f0fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0803b8c0 con 0x7f8f080361b0 2024-03-06T07:27:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7fc01d800640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc018070ee0 0x7fc018162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7fc01ce00640 1 -- 172.21.15.6:0/1362126810 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc018070ee0 msgr2=0x7fc018162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.041+0000 7f1630a00640 1 -- start start 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f1630a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f162c06d440 0x7f162c138580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f1630a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f162c06e5e0 0x7f162c14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f1630a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c09c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f1630a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c06bc20 con 0x7f162c15a210 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f1630a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c06baa0 con 0x7f162c06d440 2024-03-06T07:27:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f1630a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162c15a5f0 con 0x7f162c06e5e0 2024-03-06T07:27:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f8f0ea00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f8f0804a320 0x7f8f0803c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37104/0 (socket says 172.21.15.6:37104) 2024-03-06T07:27:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f8f0ea00640 1 -- 172.21.15.6:0/75583948 learned_addr learned my addr 172.21.15.6:0/75583948 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7fc017e00640 1 -- 172.21.15.6:0/1362126810 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309905837 0 0) 0x7fc0180ce090 con 0x7fc0180712c0 2024-03-06T07:27:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f162a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c09c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f40e9800640 1 -- 172.21.15.6:0/4094878941 shutdown_connections 2024-03-06T07:27:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.042+0000 7f40e9800640 1 -- 172.21.15.6:0/4094878941 wait complete. 2024-03-06T07:27:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7fc01ce00640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc018070ee0 0x7fc018162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7fc01ce00640 1 -- 172.21.15.6:0/1362126810 >> v1:172.21.15.178:6789/0 conn(0x7fc0180712c0 legacy=0x7fc01806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f162a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c09c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42286/0 (socket says 172.21.15.6:42286) 2024-03-06T07:27:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f162a800640 1 -- 172.21.15.6:0/1120646229 learned_addr learned my addr 172.21.15.6:0/1120646229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f1629400640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f162c06e5e0 0x7f162c14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7fc01ce00640 1 -- 172.21.15.6:0/1362126810 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180ce650 con 0x7fc018072da0 2024-03-06T07:27:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7fc01d800640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc018070ee0 0x7fc018162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:37.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f1629e00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f162c06d440 0x7f162c138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.044+0000 7fe002400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdff40e3040 0x7fdff4050020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.043+0000 7f8f0e000640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f080361b0 0x7f8f08039b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.044+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711023951 0 0) 0x7f8f0803b340 con 0x7f8f080529f0 2024-03-06T07:27:37.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fe002400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff4051d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fe002400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdff40e78d0 0x7fdff40da0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fe002400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff4054620 con 0x7fdff40e3420 2024-03-06T07:27:37.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f40e9800640 1 Processor -- start 2024-03-06T07:27:37.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.044+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef4003520 con 0x7f8f080529f0 2024-03-06T07:27:37.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.044+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 415140101 0 0) 0x7f8ef4003520 con 0x7f8f080529f0 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.044+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f080361b0 msgr2=0x7f8f08039b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f080361b0 0x7f8f08039b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 >> v1:172.21.15.178:6789/0 conn(0x7f8f0804a320 legacy=0x7f8f0803c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0803bba0 con 0x7f8f080529f0 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8f1004fb80 con 0x7f8f080529f0 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 >> v1:172.21.15.6:6789/0 conn(0x7f8f080529f0 legacy=0x7f8f08054870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40039e0 0x7f8ef4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ef4004800 0x7f8ef4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef40051c0 0x7f8ef400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fe002400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff40544a0 con 0x7fdff40e78d0 2024-03-06T07:27:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fe002400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff40547a0 con 0x7fdff40e3040 2024-03-06T07:27:37.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.044+0000 7f1629400640 1 -- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f162c06d440 msgr2=0x7f162c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f40e9800640 1 -- start start 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff4051d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fdffbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff4051d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42300/0 (socket says 172.21.15.6:42300) 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fdffbe00640 1 -- 172.21.15.6:0/3318031794 learned_addr learned my addr 172.21.15.6:0/3318031794 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7fe001a00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdff40e78d0 0x7fdff40da0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f1629400640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f162c06d440 0x7f162c138580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f1629400640 1 -- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 msgr2=0x7f162c09c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f1629400640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c09c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f1629400640 1 -- 172.21.15.6:0/1120646229 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c138ac0 con 0x7f162c06e5e0 2024-03-06T07:27:37.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7fc01ce00640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc0180cf130 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0097a0 tx=0x7fc00c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654913c82c40bff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7fe001000640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdff40e3040 0x7fdff4050020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f162a800640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f162c15a210 0x7f162c09c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:37.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f1629e00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f162c06d440 0x7f162c138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0803b8c0 con 0x7f8ef40051c0 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef400bfb0 con 0x7f8ef4004800 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.045+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef400c180 con 0x7f8ef40039e0 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0e000640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ef4004800 0x7f8ef4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0f400640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef40051c0 0x7f8ef400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0ea00640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40039e0 0x7f8ef4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0f400640 1 -- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40039e0 msgr2=0x7f8ef4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f40e9800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40719e0 0x7f40e4161fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0f400640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40039e0 0x7f8ef4003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0f400640 1 -- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ef4004800 msgr2=0x7f8ef4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0f400640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ef4004800 0x7f8ef4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.046+0000 7f8f0f400640 1 -- 172.21.15.6:0/75583948 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0803bba0 con 0x7f8ef40051c0 2024-03-06T07:27:37.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f40e9800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40b4750 0x7f40e41398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f40e9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e414bb40 0x7f40e4139e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f40e9800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40b4230 con 0x7f40e414bb40 2024-03-06T07:27:37.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fe001000640 1 -- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdff40e78d0 msgr2=0x7fdff40da0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fe001000640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdff40e78d0 0x7fdff40da0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fe001000640 1 -- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 msgr2=0x7fdff4051d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fe001000640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdff40e3420 0x7fdff4051d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fe001000640 1 -- 172.21.15.6:0/3318031794 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff4050870 con 0x7fdff40e3040 2024-03-06T07:27:37.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f1629400640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f162c06e5e0 0x7f162c14fbd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f161800c780 tx=0x7f161800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fc017e00640 1 -- 172.21.15.6:0/1362126810 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c035020 con 0x7fc018072da0 2024-03-06T07:27:37.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f40e9800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e40b3f50 con 0x7f40e40b4750 2024-03-06T07:27:37.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fc017e00640 1 -- 172.21.15.6:0/1362126810 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c031810 con 0x7fc018072da0 2024-03-06T07:27:37.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7fc017e00640 1 -- 172.21.15.6:0/1362126810 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c031be0 con 0x7fc018072da0 2024-03-06T07:27:37.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f40e9800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40e414bf20 con 0x7f40e40719e0 2024-03-06T07:27:37.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f40e2a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40719e0 0x7f40e4161fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f40e2a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40719e0 0x7f40e4161fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47578/0 (socket says 172.21.15.6:47578) 2024-03-06T07:27:37.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f40e2a00640 1 -- 172.21.15.6:0/3214133612 learned_addr learned my addr 172.21.15.6:0/3214133612 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.047+0000 7f045cc00640 1 Processor -- start 2024-03-06T07:27:37.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f045cc00640 1 -- start start 2024-03-06T07:27:37.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f40e2000640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40b4750 0x7f40e41398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7fe001000640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdff40e3040 0x7fdff4050020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdffc069ad0 tx=0x7fdffc069430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618017070 con 0x7f162c06e5e0 2024-03-06T07:27:37.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1618005270 con 0x7f162c06e5e0 2024-03-06T07:27:37.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161800a640 con 0x7f162c06e5e0 2024-03-06T07:27:37.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f8f0f400640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef40051c0 0x7f8ef400b9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f10050c30 tx=0x7f8ef4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dccea93b9464d79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c09db70 con 0x7f162c06e5e0 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c09d5b0 con 0x7f162c06e5e0 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc090070 con 0x7fdff40e3040 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdffc073260 con 0x7fdff40e3040 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc070690 con 0x7fdff40e3040 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff4050b50 con 0x7fdff40e3040 2024-03-06T07:27:37.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f100b4070 con 0x7f8ef40051c0 2024-03-06T07:27:37.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f100a7c50 con 0x7f8ef40051c0 2024-03-06T07:27:37.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f8f0d600640 1 -- 172.21.15.6:0/75583948 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10068010 con 0x7f8ef40051c0 2024-03-06T07:27:37.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc020200640 1 -- 172.21.15.6:0/1362126810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 msgr2=0x7fc0180cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc020200640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc0180cf130 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0097a0 tx=0x7fc00c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.048+0000 7f40e3400640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e414bb40 0x7f40e4139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff40d5ae0 con 0x7fdff40e3040 2024-03-06T07:27:37.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc020200640 1 -- 172.21.15.6:0/1362126810 shutdown_connections 2024-03-06T07:27:37.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc020200640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc0180cf130 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc020200640 1 --2- 172.21.15.6:0/1362126810 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc018070ee0 0x7fc018162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc020200640 1 -- 172.21.15.6:0/1362126810 >> 172.21.15.6:0/1362126810 conn(0x7fc01806afb0 msgr2=0x7fc01806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc0ec600640 1 Processor -- start 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7f40e2000640 1 -- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40719e0 msgr2=0x7f40e4161fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7f40e2000640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f40e40719e0 0x7f40e4161fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7f40e2000640 1 -- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e414bb40 msgr2=0x7f40e4139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7f40e2000640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e414bb40 0x7f40e4139e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7f40e2000640 1 -- 172.21.15.6:0/3214133612 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e4162b30 con 0x7f40e40b4750 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7f40e3400640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f40e414bb40 0x7f40e4139e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:37.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f40e2000640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40b4750 0x7f40e41398d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40d400e850 tx=0x7f40d400ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.049+0000 7fc0ec600640 1 -- start start 2024-03-06T07:27:37.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f8f0fe00640 1 -- 172.21.15.6:0/75583948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef40051c0 msgr2=0x7f8ef400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f8f0fe00640 1 --2- 172.21.15.6:0/75583948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ef40051c0 0x7f8ef400b9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f10050c30 tx=0x7f8ef4005830 comp rx=0 tx=0).stop 2024-03-06T07:27:37.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d4017070 con 0x7f40e40b4750 2024-03-06T07:27:37.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d400c940 con 0x7f40e40b4750 2024-03-06T07:27:37.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e4162e10 con 0x7f40e40b4750 2024-03-06T07:27:37.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e41633d0 con 0x7f40e40b4750 2024-03-06T07:27:37.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d400cd10 con 0x7f40e40b4750 2024-03-06T07:27:37.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f162c0b4f50 con 0x7f162c06e5e0 2024-03-06T07:27:37.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7fc020200640 1 -- 172.21.15.6:0/1362126810 shutdown_connections 2024-03-06T07:27:37.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7fc020200640 1 -- 172.21.15.6:0/1362126810 wait complete. 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f8f0f400640 1 -- 172.21.15.6:0/75583948 reap_dead start 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f8f0fe00640 1 -- 172.21.15.6:0/75583948 shutdown_connections 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f8f0fe00640 1 -- 172.21.15.6:0/75583948 >> 172.21.15.6:0/75583948 conn(0x7f8f080537b0 msgr2=0x7f8f08053b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f8f0fe00640 1 -- 172.21.15.6:0/75583948 shutdown_connections 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7f8f0fe00640 1 -- 172.21.15.6:0/75583948 wait complete. 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7fc0ec600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc0e4070ee0 0x7fc0e4162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7fc0ec600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7fc0ec600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e40ce090 con 0x7fc0e4072da0 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7fc0ec600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e40ceca0 con 0x7fc0e4070ee0 2024-03-06T07:27:37.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7fc0ec600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40ce370 con 0x7fc0e40712c0 2024-03-06T07:27:37.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f045cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f045815e2e0 0x7f04580d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f045cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580b3940 con 0x7f04580b50e0 2024-03-06T07:27:37.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f045cc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04580cf580 con 0x7f04580d0d70 2024-03-06T07:27:37.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.050+0000 7f045cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580d4e70 con 0x7f045815e2e0 2024-03-06T07:27:37.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.051+0000 7fc0e9c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc0e4070ee0 0x7fc0e4162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7fc0e9200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc0e4072da0 0x7fc0e406e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37464/0 (socket says 172.21.15.6:37464) 2024-03-06T07:27:37.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7f0455400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f04580d0d70 0x7f04580d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37118/0 (socket says 172.21.15.6:37118) 2024-03-06T07:27:37.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7f0455400640 1 -- 172.21.15.6:0/60885804 learned_addr learned my addr 172.21.15.6:0/60885804 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7f0455e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f045815e2e0 0x7f04580d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7fc0e9200640 1 -- 172.21.15.6:0/2108910304 learned_addr learned my addr 172.21.15.6:0/2108910304 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7fc0ea600640 1 --2- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7fc0e3e00640 1 -- 172.21.15.6:0/2108910304 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384222415 0 0) 0x7fc0e40ce090 con 0x7fc0e4072da0 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7fc0e3e00640 1 -- 172.21.15.6:0/2108910304 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0cc003520 con 0x7fc0e4072da0 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7fc0e9c00640 1 -- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 msgr2=0x7fc0e406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7fc0e9c00640 1 --2- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e406e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7fc0e9c00640 1 -- 172.21.15.6:0/2108910304 >> v1:172.21.15.6:6789/0 conn(0x7fc0e4072da0 legacy=0x7fc0e406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.052+0000 7f0456800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f04580b50e0 0x7f04580d2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37454/0 (socket says 172.21.15.6:37454) 2024-03-06T07:27:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7fc0e9c00640 1 -- 172.21.15.6:0/2108910304 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e40ce650 con 0x7fc0e4070ee0 2024-03-06T07:27:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7f8f0fe00640 1 Processor -- start 2024-03-06T07:27:37.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.054+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdfd8001030 con 0x7fdff40e3040 2024-03-06T07:27:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7f0454a00640 1 -- 172.21.15.6:0/60885804 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525848746 0 0) 0x7f04580cf580 con 0x7f04580d0d70 2024-03-06T07:27:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.054+0000 7f0454a00640 1 -- 172.21.15.6:0/60885804 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f043c003520 con 0x7f04580d0d70 2024-03-06T07:27:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.054+0000 7f0455e00640 1 -- 172.21.15.6:0/60885804 >> v1:172.21.15.178:6789/0 conn(0x7f04580d0d70 legacy=0x7f04580d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.053+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f161801a070 con 0x7f162c06e5e0 2024-03-06T07:27:37.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7fdc24200640 1 Processor -- start 2024-03-06T07:27:37.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f1622000640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f160c06b520 0x7f160c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7fdc24200640 1 -- start start 2024-03-06T07:27:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40bc001030 con 0x7f40e40b4750 2024-03-06T07:27:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f0454a00640 1 -- 172.21.15.6:0/60885804 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013588814 0 0) 0x7f04580b3940 con 0x7f04580b50e0 2024-03-06T07:27:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f0455e00640 1 -- 172.21.15.6:0/60885804 >> v1:172.21.15.6:6789/0 conn(0x7f04580b50e0 legacy=0x7f04580d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f0455e00640 1 -- 172.21.15.6:0/60885804 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04580cf860 con 0x7f045815e2e0 2024-03-06T07:27:37.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1618056730 con 0x7f162c06e5e0 2024-03-06T07:27:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.054+0000 7f8f0fe00640 1 -- start start 2024-03-06T07:27:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdffc093070 con 0x7fdff40e3040 2024-03-06T07:27:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40051c0 0x7f8f080df5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0804e2b0 0x7f8f08181240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f8f0fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0804f080 0x7f8f0804f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f8f0fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080eb4d0 con 0x7f8f0804e2b0 2024-03-06T07:27:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f8f0fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f080eb350 con 0x7f8f0804f080 2024-03-06T07:27:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.055+0000 7f8f0fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f08054f80 con 0x7f8ef40051c0 2024-03-06T07:27:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f1622000640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f160c074410 0x7f160c0768b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f160c076f80 con 0x7f160c074410 2024-03-06T07:27:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7fc020200640 1 Processor -- start 2024-03-06T07:27:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16180550c0 con 0x7f162c06e5e0 2024-03-06T07:27:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f1629e00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f160c06b520 0x7f160c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f162a800640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f160c074410 0x7f160c0768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7fc020200640 1 -- start start 2024-03-06T07:27:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f8f0e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0804e2b0 0x7f8f08181240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f8f0ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40051c0 0x7f8f080df5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f8f0ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40051c0 0x7f8f080df5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47586/0 (socket says 172.21.15.6:47586) 2024-03-06T07:27:37.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7f8f0ea00640 1 -- 172.21.15.6:0/3718467688 learned_addr learned my addr 172.21.15.6:0/3718467688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7fc0e9c00640 1 --2- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4070ee0 0x7fc0e4162c40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40cee20 tx=0x7fc0d802be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8676d4c0ad3a569d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7fc0e3e00640 1 -- 172.21.15.6:0/2108910304 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8039070 con 0x7fc0e4070ee0 2024-03-06T07:27:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.056+0000 7fc0e3e00640 1 -- 172.21.15.6:0/2108910304 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d802cc40 con 0x7fc0e4070ee0 2024-03-06T07:27:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7fc0e3e00640 1 -- 172.21.15.6:0/2108910304 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d8034440 con 0x7fc0e4070ee0 2024-03-06T07:27:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7f8f0f400640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0804f080 0x7f8f0804f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7fc0ec600640 1 -- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4070ee0 msgr2=0x7fc0e4162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7fc0ec600640 1 --2- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4070ee0 0x7fc0e4162c40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40cee20 tx=0x7fc0d802be90 comp rx=0 tx=0).stop 2024-03-06T07:27:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7f0455e00640 1 --2- 172.21.15.6:0/60885804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f04580d0830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0450009d90 tx=0x7f045002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2eb3c7ff81ad33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc0ec600640 1 -- 172.21.15.6:0/2108910304 shutdown_connections 2024-03-06T07:27:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc0ec600640 1 --2- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e406e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc0ec600640 1 --2- 172.21.15.6:0/2108910304 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4070ee0 0x7fc0e4162c40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc0ec600640 1 -- 172.21.15.6:0/2108910304 >> 172.21.15.6:0/2108910304 conn(0x7fc0e406afb0 msgr2=0x7fc0e406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc020200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc018070ee0 0x7fc01814e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc020200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0180712c0 0x7fc018143d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc020200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc018144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc020200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180cdda0 con 0x7fc018072da0 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc020200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180cdc20 con 0x7fc018070ee0 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc020200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01806e7d0 con 0x7fc0180712c0 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7fdc24200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 0x7fdc1c0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fdc24200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc1c06d440 0x7fdc1c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7f8f0ea00640 1 -- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0804f080 msgr2=0x7f8f0804f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7f8f0ea00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0804f080 0x7f8f0804f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7f1629e00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f160c06b520 0x7f160c06d9e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f161c034000 tx=0x7f161c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7f162a800640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f160c074410 0x7f160c0768b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fdc24200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1c06d900 con 0x7fdc1c06e5e0 2024-03-06T07:27:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fdc24200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1c06dbe0 con 0x7fdc1c06d440 2024-03-06T07:27:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fdc24200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c0cddc0 con 0x7fdc1c06baa0 2024-03-06T07:27:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7f0454a00640 1 -- 172.21.15.6:0/60885804 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450039070 con 0x7f045815e2e0 2024-03-06T07:27:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7f8f0ea00640 1 -- 172.21.15.6:0/3718467688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0804e2b0 msgr2=0x7f8f08181240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fdc21800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 0x7fdc1c0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc21800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 0x7fdc1c0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47598/0 (socket says 172.21.15.6:47598) 2024-03-06T07:27:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc21800640 1 -- 172.21.15.6:0/2424087443 learned_addr learned my addr 172.21.15.6:0/2424087443 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc22200640 1 --2- 172.21.15.6:0/2424087443 >> v2:172.21.15.6:3300/0 conn(0x7fdc1c06d440 0x7fdc1c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdff8c00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdfdc06b5a0 0x7fdfdc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdffc1048d0 con 0x7fdff40e3040 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdff8c00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7fdfdc0737c0 0x7fdfdc075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdffc08ee80 con 0x7fdfdc0737c0 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdffc1092a0 con 0x7fdff40e3040 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fdffbe00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdfdc06b5a0 0x7fdfdc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fe001a00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7fdfdc0737c0 0x7fdfdc075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== osd.2 v2:172.21.15.6:6822/1963740543 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f160c076f80 con 0x7f160c074410 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.057+0000 7f8f0ea00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0804e2b0 0x7f8f08181240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7f8f0ea00640 1 -- 172.21.15.6:0/3718467688 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f08182150 con 0x7f8ef40051c0 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc0ec600640 1 -- 172.21.15.6:0/2108910304 shutdown_connections 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.058+0000 7fc0ec600640 1 -- 172.21.15.6:0/2108910304 wait complete. 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc0ec600640 1 Processor -- start 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc0ec600640 1 -- start start 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7f0454a00640 1 -- 172.21.15.6:0/60885804 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045002ccf0 con 0x7f045815e2e0 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7f0454a00640 1 -- 172.21.15.6:0/60885804 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450039070 con 0x7f045815e2e0 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7f045cc00640 1 -- 172.21.15.6:0/60885804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 msgr2=0x7f04580d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7f045cc00640 1 --2- 172.21.15.6:0/60885804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f04580d0830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0450009d90 tx=0x7f045002bf40 comp rx=0 tx=0).stop 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f045cc00640 1 -- 172.21.15.6:0/60885804 shutdown_connections 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f045cc00640 1 --2- 172.21.15.6:0/60885804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f04580d0830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f045cc00640 1 -- 172.21.15.6:0/60885804 >> 172.21.15.6:0/60885804 conn(0x7f04580d3f60 msgr2=0x7f04580d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f045cc00640 1 -- 172.21.15.6:0/60885804 shutdown_connections 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fc01d800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc018070ee0 0x7fc01814e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fc01e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc018144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fc01e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc018144290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42362/0 (socket says 172.21.15.6:42362) 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fc01e200640 1 -- 172.21.15.6:0/730348263 learned_addr learned my addr 172.21.15.6:0/730348263 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc01e200640 1 -- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0180712c0 msgr2=0x7fc018143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc01e200640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0180712c0 0x7fc018143d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc01e200640 1 -- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc018070ee0 msgr2=0x7fc01814e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc01e200640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc018070ee0 0x7fc01814e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc01e200640 1 -- 172.21.15.6:0/730348263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01814d780 con 0x7fc018072da0 2024-03-06T07:27:37.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7fc01e200640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc018144290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc01000c7e0 tx=0x7fc01000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40d401a070 con 0x7f40e40b4750 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc21800640 1 -- 172.21.15.6:0/2424087443 >> v2:172.21.15.6:3300/0 conn(0x7fdc1c06d440 msgr2=0x7fdc1c0ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc21800640 1 --2- 172.21.15.6:0/2424087443 >> v2:172.21.15.6:3300/0 conn(0x7fdc1c06d440 0x7fdc1c0ce5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc21800640 1 -- 172.21.15.6:0/2424087443 >> v1:172.21.15.6:6789/0 conn(0x7fdc1c06e5e0 legacy=0x7fdc1c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.059+0000 7fdc21800640 1 -- 172.21.15.6:0/2424087443 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0acdb0 con 0x7fdc1c06baa0 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010018070 con 0x7fc018072da0 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01000a640 con 0x7fc018072da0 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01000aa30 con 0x7fc018072da0 2024-03-06T07:27:37.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7fe001a00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7fdfdc0737c0 0x7fdfdc075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.060+0000 7f045cc00640 1 -- 172.21.15.6:0/60885804 wait complete. 2024-03-06T07:27:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01809cbe0 con 0x7fc018072da0 2024-03-06T07:27:37.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01809d1a0 con 0x7fc018072da0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.061+0000 7f8f0ea00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40051c0 0x7f8f080df5e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d5f0 tx=0x7f8efc00dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f40daa00640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40c406b5a0 0x7f40c406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7fdffbe00640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdfdc06b5a0 0x7fdfdc06da60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdfe802b6f0 tx=0x7fdfe8008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc019070 con 0x7f8ef40051c0 2024-03-06T07:27:37.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0074e0 con 0x7f8ef40051c0 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc004a00 con 0x7f8ef40051c0 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f08181b90 con 0x7f8ef40051c0 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7fdc21800640 1 --2- 172.21.15.6:0/2424087443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 0x7fdc1c0cbfc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009e50 tx=0x7fdc1402be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e45fb284fc21cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f40e2a00640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40c406b5a0 0x7f40c406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.062+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0804e880 con 0x7f8ef40051c0 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.063+0000 7f045cc00640 1 Processor -- start 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.063+0000 7fdc1be00640 1 -- 172.21.15.6:0/2424087443 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1402c9f0 con 0x7fdc1c06baa0 2024-03-06T07:27:37.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.063+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40d408a3f0 con 0x7f40e40b4750 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.063+0000 7fdc1be00640 1 -- 172.21.15.6:0/2424087443 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1402cb90 con 0x7fdc1c06baa0 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.063+0000 7f40daa00640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f40c4073740 0x7f40c4075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.063+0000 7f045cc00640 1 -- start start 2024-03-06T07:27:37.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== osd.0 v2:172.21.15.6:6800/2712143005 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdffc08ee80 con 0x7fdfdc0737c0 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40d4016ec0 con 0x7f40c4073740 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40d4091050 con 0x7f40e40b4750 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f0804e6d0 con 0x7f8ef40051c0 2024-03-06T07:27:37.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ec600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0e4070ee0 0x7fc0e415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fdc1be00640 1 -- 172.21.15.6:0/2424087443 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1402c9f0 con 0x7fdc1c06baa0 2024-03-06T07:27:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ec600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e409c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ec600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4072da0 0x7fc0e409ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ec600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40cdda0 con 0x7fc0e4070ee0 2024-03-06T07:27:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ec600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40cdc20 con 0x7fc0e4072da0 2024-03-06T07:27:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ec600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40cf130 con 0x7fc0e40712c0 2024-03-06T07:27:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fdc24200640 1 -- 172.21.15.6:0/2424087443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 msgr2=0x7fdc1c0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fdc24200640 1 --2- 172.21.15.6:0/2424087443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 0x7fdc1c0cbfc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009e50 tx=0x7fdc1402be50 comp rx=0 tx=0).stop 2024-03-06T07:27:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc01804f3a0 con 0x7fc018072da0 2024-03-06T07:27:37.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7f40e3400640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f40c4073740 0x7f40c4075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.064+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4072da0 0x7fc0e409ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fdc24200640 1 -- 172.21.15.6:0/2424087443 shutdown_connections 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fdc24200640 1 --2- 172.21.15.6:0/2424087443 >> v2:172.21.15.6:3300/0 conn(0x7fdc1c06d440 0x7fdc1c0ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fdc24200640 1 --2- 172.21.15.6:0/2424087443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06baa0 0x7fdc1c0cbfc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fdc24200640 1 -- 172.21.15.6:0/2424087443 >> 172.21.15.6:0/2424087443 conn(0x7fdc1c06afb0 msgr2=0x7fdc1c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fdc24200640 1 -- 172.21.15.6:0/2424087443 shutdown_connections 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4072da0 0x7fc0e409ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:33852/0 (socket says 172.21.15.6:33852) 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fc0ea600640 1 -- 172.21.15.6:0/2626662052 learned_addr learned my addr 172.21.15.6:0/2626662052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fdc24200640 1 -- 172.21.15.6:0/2424087443 wait complete. 2024-03-06T07:27:37.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7fc0e9200640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e409c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7f045cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04580b50e0 0x7f04580c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7f045cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04580d0d70 0x7f04580c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7f045cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f0458149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7f045cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04580cfe20 con 0x7f045815e2e0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7f045cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04580b3c40 con 0x7f04580b50e0 2024-03-06T07:27:37.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.065+0000 7f045cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04580d2950 con 0x7f04580d0d70 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7fc0ea600640 1 -- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 msgr2=0x7fc0e409c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7fc0ea600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0e40712c0 0x7fc0e409c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7fc0ea600640 1 -- 172.21.15.6:0/2626662052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0e4070ee0 msgr2=0x7fc0e415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7fc0ea600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0e4070ee0 0x7fc0e415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7fc0ea600640 1 -- 172.21.15.6:0/2626662052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e409b740 con 0x7fc0e4072da0 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7f0456800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f0458149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7f0455400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04580d0d70 0x7f04580c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7f40e2a00640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40c406b5a0 0x7f40c406da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40e40ce890 tx=0x7f40d0009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7f0455400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04580d0d70 0x7f04580c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47612/0 (socket says 172.21.15.6:47612) 2024-03-06T07:27:37.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7f0455400640 1 -- 172.21.15.6:0/2179463360 learned_addr learned my addr 172.21.15.6:0/2179463360 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc01001b070 con 0x7fc018072da0 2024-03-06T07:27:37.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.067+0000 7fc0ea600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4072da0 0x7fc0e409ca90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00e960 tx=0x7fc0dc00ee20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.066+0000 7f0455e00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04580b50e0 0x7f04580c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.067+0000 7f40e3400640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f40c4073740 0x7f40c4075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.067+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8efc01c070 con 0x7f8ef40051c0 2024-03-06T07:27:37.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7f8f06a00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b520 0x7f8ef006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8efc08a440 con 0x7f8ef40051c0 2024-03-06T07:27:37.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.067+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc017070 con 0x7fc0e4072da0 2024-03-06T07:27:37.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.067+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0dc00c980 con 0x7fc0e4072da0 2024-03-06T07:27:37.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.067+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc00cd70 con 0x7fc0e4072da0 2024-03-06T07:27:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7fc015600640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbffc06b5d0 0x7fbffc06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7fdc24200640 1 Processor -- start 2024-03-06T07:27:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7f8f06a00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8ef0073740 0x7f8ef0075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8efc018ec0 con 0x7f8ef0073740 2024-03-06T07:27:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8efc08a920 con 0x7f8ef40051c0 2024-03-06T07:27:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e409ba20 con 0x7fc0e4072da0 2024-03-06T07:27:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e409bfe0 con 0x7fc0e4072da0 2024-03-06T07:27:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.068+0000 7fdc24200640 1 -- start start 2024-03-06T07:27:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f8f0e000640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b520 0x7f8ef006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc010089830 con 0x7fc018072da0 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f0455e00640 1 -- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04580d0d70 msgr2=0x7f04580c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f0455e00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04580d0d70 0x7f04580c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f0455e00640 1 -- 172.21.15.6:0/2179463360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 msgr2=0x7f0458149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f0455e00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f0458149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f0455e00640 1 -- 172.21.15.6:0/2179463360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045815c850 con 0x7f04580b50e0 2024-03-06T07:27:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f0456800640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045815e2e0 0x7f0458149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7fc01d800640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbffc06b5d0 0x7fbffc06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7fc015600640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7fbffc0737f0 0x7fbffc075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== osd.1 v2:172.21.15.6:6808/1355309078 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f40d4016ec0 con 0x7f40c4073740 2024-03-06T07:27:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbffc0762c0 con 0x7fbffc0737f0 2024-03-06T07:27:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.069+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc01008e2a0 con 0x7fc018072da0 2024-03-06T07:27:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7f0455400640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04580d0d70 0x7f04580c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7f8f0f400640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8ef0073740 0x7f8ef0075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7fdc24200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc1c06baa0 0x7fdc1c0ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7fdc24200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c06d440 0x7fdc1c0ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0e404f3a0 con 0x7fc0e4072da0 2024-03-06T07:27:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7fdc24200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06e5e0 0x7fdc1c14b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7fc01d800640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbffc06b5d0 0x7fbffc06da90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc01809c6a0 tx=0x7fc008008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.070+0000 7fc01ce00640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7fbffc0737f0 0x7fbffc075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7f0455e00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04580b50e0 0x7f04580c8160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0450033000 tx=0x7f0450004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc24200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c15a390 con 0x7fdc1c06d440 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc24200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c15a210 con 0x7fdc1c06baa0 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc24200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1c15a510 con 0x7fdc1c06e5e0 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc21800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc1c06baa0 0x7fdc1c0ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc21800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc1c06baa0 0x7fdc1c0ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:33880/0 (socket says 172.21.15.6:33880) 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc21800640 1 -- 172.21.15.6:0/1902109166 learned_addr learned my addr 172.21.15.6:0/1902109166 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7fdc20e00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c06d440 0x7fdc1c0ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7f8f0f400640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8ef0073740 0x7f8ef0075b60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450042020 con 0x7f04580b50e0 2024-03-06T07:27:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045003a040 con 0x7f04580b50e0 2024-03-06T07:27:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.071+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450039070 con 0x7f04580b50e0 2024-03-06T07:27:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fc01ce00640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7fbffc0737f0 0x7fbffc075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc22200640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06e5e0 0x7fdc1c14b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7f8f0e000640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b520 0x7f8ef006d9e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f080564d0 tx=0x7f8f00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc20e00640 1 -- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06e5e0 msgr2=0x7fdc1c14b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc20e00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06e5e0 0x7fdc1c14b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc20e00640 1 -- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc1c06baa0 msgr2=0x7fdc1c0ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc20e00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc1c06baa0 0x7fdc1c0ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc20e00640 1 -- 172.21.15.6:0/1902109166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c149da0 con 0x7fdc1c06d440 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.072+0000 7fdc22200640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc1c06e5e0 0x7fdc1c14b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.073+0000 7fdc20e00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c06d440 0x7fdc1c0ae6a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000b780 tx=0x7fdc1000bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.073+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10004220 con 0x7fdc1c06d440 2024-03-06T07:27:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc100043c0 con 0x7fdc1c06d440 2024-03-06T07:27:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc10003830 con 0x7fdc1c06d440 2024-03-06T07:27:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.073+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04580c8d40 con 0x7f04580b50e0 2024-03-06T07:27:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.073+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04580c9300 con 0x7f04580b50e0 2024-03-06T07:27:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7fdc24200640 1 -- 172.21.15.6:0/1902109166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1c14a080 con 0x7fdc1c06d440 2024-03-06T07:27:37.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7fdc24200640 1 -- 172.21.15.6:0/1902109166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1c14a640 con 0x7fdc1c06d440 2024-03-06T07:27:37.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== osd.4 v2:172.21.15.178:6809/2203214816 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8efc018ec0 con 0x7f8ef0073740 2024-03-06T07:27:37.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== osd.3 v2:172.21.15.6:6816/1271003476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbffc0762c0 con 0x7fbffc0737f0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.073+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0dc01a070 con 0x7fc0e4072da0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.074+0000 7fc0e1600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0c406b4d0 0x7fc0c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0dc08afc0 con 0x7fc0e4072da0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fc0e1600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7fc0c40736f0 0x7fc0c4075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0dc00ae90 con 0x7fc0c40736f0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0dc08b3b0 con 0x7fc0e4072da0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fc0e9200640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7fc0c40736f0 0x7fc0c4075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fc0e9c00640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0c406b4d0 0x7fc0c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.076+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04580b9e20 con 0x7f04580b50e0 2024-03-06T07:27:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.075+0000 7fdc24200640 1 -- 172.21.15.6:0/1902109166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdc1c04f3a0 con 0x7fdc1c06d440 2024-03-06T07:27:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.077+0000 7fc0e9c00640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0c406b4d0 0x7fc0c406d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8036000 tx=0x7fc0d802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.077+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0450045020 con 0x7f04580b50e0 2024-03-06T07:27:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.078+0000 7fc0e9200640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7fc0c40736f0 0x7fc0c4075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.078+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc100039d0 con 0x7fdc1c06d440 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.078+0000 7f044e000640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f043806b4d0 0x7f043806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.079+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04500ac4c0 con 0x7f04580b50e0 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.079+0000 7f044e000640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f04380736d0 0x7f0438075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.079+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f045004de50 con 0x7f04380736d0 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.079+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04500b3050 con 0x7f04580b50e0 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.079+0000 7f0455400640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f043806b4d0 0x7f043806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7f0456800640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f04380736d0 0x7f0438075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7fdc19600640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbfc06b5a0 0x7fdbfc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc10089f60 con 0x7fdc1c06d440 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7fdc19600640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x7fdbfc0737c0 0x7fdbfc075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc10008eb0 con 0x7fdbfc0737c0 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc1008a350 con 0x7fdc1c06d440 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.080+0000 7fdc21800640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbfc06b5a0 0x7fdbfc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.081+0000 7f0456800640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f04380736d0 0x7f0438075af0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.081+0000 7f0455400640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f043806b4d0 0x7f043806d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04581fb610 tx=0x7f0444009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.081+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== osd.5 v2:172.21.15.178:6800/1128394266 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc0dc00ae90 con 0x7fc0c40736f0 2024-03-06T07:27:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.081+0000 7fdc22200640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x7fdbfc0737c0 0x7fdbfc075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.082+0000 7fdc21800640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbfc06b5a0 0x7fdbfc06da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c159c20 tx=0x7fdc1403f040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.082+0000 7fdc22200640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x7fdbfc0737c0 0x7fdbfc075be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.085+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== osd.6 v2:172.21.15.178:6824/196133891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f045004de50 con 0x7f04380736d0 2024-03-06T07:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.085+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== osd.7 v2:172.21.15.178:6808/56642032 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdc10008eb0 con 0x7fdbfc0737c0 2024-03-06T07:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.088+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f162c04f3a0 con 0x7f160c074410 2024-03-06T07:27:37.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.089+0000 7f1622000640 1 -- 172.21.15.6:0/1120646229 <== osd.2 v2:172.21.15.6:6822/1963740543 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f162c04f3a0 con 0x7f160c074410 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.091+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f160c074410 msgr2=0x7f160c0768b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.091+0000 7f1630a00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f160c074410 0x7f160c0768b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f160c06b520 msgr2=0x7f160c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f1630a00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f160c06b520 0x7f160c06d9e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f161c034000 tx=0x7f161c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f162c06e5e0 msgr2=0x7f162c14fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f1630a00640 1 --2- 172.21.15.6:0/1120646229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f162c06e5e0 0x7f162c14fbd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f161800c780 tx=0x7f161800cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f162a800640 1 -- 172.21.15.6:0/1120646229 reap_dead start 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 shutdown_connections 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.092+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 >> 172.21.15.6:0/1120646229 conn(0x7f162c06afb0 msgr2=0x7f162c14e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.095+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 shutdown_connections 2024-03-06T07:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.095+0000 7f1630a00640 1 -- 172.21.15.6:0/1120646229 wait complete. 2024-03-06T07:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.099+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdfd8002dd0 con 0x7fdfdc0737c0 2024-03-06T07:27:37.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.100+0000 7fdff8c00640 1 -- 172.21.15.6:0/3318031794 <== osd.0 v2:172.21.15.6:6800/2712143005 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdfd8002dd0 con 0x7fdfdc0737c0 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.101+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7fdfdc0737c0 msgr2=0x7fdfdc075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.101+0000 7fe002400640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7fdfdc0737c0 0x7fdfdc075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.101+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdfdc06b5a0 msgr2=0x7fdfdc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.101+0000 7fe002400640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdfdc06b5a0 0x7fdfdc06da60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdfe802b6f0 tx=0x7fdfe8008030 comp rx=0 tx=0).stop 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.101+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdff40e3040 msgr2=0x7fdff4050020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.101+0000 7fe002400640 1 --2- 172.21.15.6:0/3318031794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdff40e3040 0x7fdff4050020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdffc069ad0 tx=0x7fdffc069430 comp rx=0 tx=0).stop 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.102+0000 7fe001a00640 1 -- 172.21.15.6:0/3318031794 reap_dead start 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.102+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 shutdown_connections 2024-03-06T07:27:37.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.102+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 >> 172.21.15.6:0/3318031794 conn(0x7fdff403aa40 msgr2=0x7fdff403e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.104+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40bc002dd0 con 0x7f40c4073740 2024-03-06T07:27:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.104+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc01814d950 con 0x7fbffc0737f0 2024-03-06T07:27:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.105+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 shutdown_connections 2024-03-06T07:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.105+0000 7f40daa00640 1 -- 172.21.15.6:0/3214133612 <== osd.1 v2:172.21.15.6:6808/1355309078 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f40bc002dd0 con 0x7f40c4073740 2024-03-06T07:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.105+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8f08039b80 con 0x7f8ef0073740 2024-03-06T07:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7fe002400640 1 -- 172.21.15.6:0/3318031794 wait complete. 2024-03-06T07:27:37.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f40c4073740 msgr2=0x7f40c4075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f40e9800640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f40c4073740 0x7f40c4075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7fc015600640 1 -- 172.21.15.6:0/730348263 <== osd.3 v2:172.21.15.6:6816/1271003476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc01814d950 con 0x7fbffc0737f0 2024-03-06T07:27:37.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f8f06a00640 1 -- 172.21.15.6:0/3718467688 <== osd.4 v2:172.21.15.178:6809/2203214816 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8f08039b80 con 0x7f8ef0073740 2024-03-06T07:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40c406b5a0 msgr2=0x7f40c406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f40e9800640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40c406b5a0 0x7f40c406da60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40e40ce890 tx=0x7f40d0009240 comp rx=0 tx=0).stop 2024-03-06T07:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40b4750 msgr2=0x7f40e41398d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.106+0000 7f40e9800640 1 --2- 172.21.15.6:0/3214133612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f40e40b4750 0x7f40e41398d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f40d400e850 tx=0x7f40d400ed10 comp rx=0 tx=0).stop 2024-03-06T07:27:37.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f40e3400640 1 -- 172.21.15.6:0/3214133612 reap_dead start 2024-03-06T07:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 shutdown_connections 2024-03-06T07:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 >> 172.21.15.6:0/3214133612 conn(0x7f40e40b1750 msgr2=0x7f40e406d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 shutdown_connections 2024-03-06T07:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8ef0073740 msgr2=0x7f8ef0075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f8f0fe00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8ef0073740 0x7f8ef0075b60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.107+0000 7f40e9800640 1 -- 172.21.15.6:0/3214133612 wait complete. 2024-03-06T07:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7fbffc0737f0 msgr2=0x7fbffc075c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7fbffc0737f0 0x7fbffc075c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b520 msgr2=0x7f8ef006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0fe00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b520 0x7f8ef006d9e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f080564d0 tx=0x7f8f00008040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40051c0 msgr2=0x7f8f080df5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0fe00640 1 --2- 172.21.15.6:0/3718467688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ef40051c0 0x7f8f080df5e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00d5f0 tx=0x7f8efc00dab0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbffc06b5d0 msgr2=0x7fbffc06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbffc06b5d0 0x7fbffc06da90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc01809c6a0 tx=0x7fc008008040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 msgr2=0x7fc018144290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 --2- 172.21.15.6:0/730348263 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc018072da0 0x7fc018144290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc01000c7e0 tx=0x7fc01000cca0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0f400640 1 -- 172.21.15.6:0/3718467688 reap_dead start 2024-03-06T07:27:37.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc01e200640 1 -- 172.21.15.6:0/730348263 reap_dead start 2024-03-06T07:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 shutdown_connections 2024-03-06T07:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 >> 172.21.15.6:0/730348263 conn(0x7fc01806afb0 msgr2=0x7fc018165490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 shutdown_connections 2024-03-06T07:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 >> 172.21.15.6:0/3718467688 conn(0x7f8f080537b0 msgr2=0x7f8f08052050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.109+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 shutdown_connections 2024-03-06T07:27:37.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.110+0000 7fc020200640 1 -- 172.21.15.6:0/730348263 wait complete. 2024-03-06T07:27:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.110+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 shutdown_connections 2024-03-06T07:27:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.110+0000 7f8f0fe00640 1 -- 172.21.15.6:0/3718467688 wait complete. 2024-03-06T07:27:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.113+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc0e40b52d0 con 0x7fc0c40736f0 2024-03-06T07:27:37.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.114+0000 7fc0e1600640 1 -- 172.21.15.6:0/2626662052 <== osd.5 v2:172.21.15.178:6800/1128394266 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc0e40b52d0 con 0x7fc0c40736f0 2024-03-06T07:27:37.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.115+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7fc0c40736f0 msgr2=0x7fc0c4075b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.115+0000 7fc0ec600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7fc0c40736f0 0x7fc0c4075b10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.116+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0c406b4d0 msgr2=0x7fc0c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.116+0000 7fc0ec600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0c406b4d0 0x7fc0c406d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8036000 tx=0x7fc0d802e000 comp rx=0 tx=0).stop 2024-03-06T07:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.116+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f045804f3a0 con 0x7f04380736d0 2024-03-06T07:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.116+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4072da0 msgr2=0x7fc0e409ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.116+0000 7fc0ec600640 1 --2- 172.21.15.6:0/2626662052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0e4072da0 0x7fc0e409ca90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00e960 tx=0x7fc0dc00ee20 comp rx=0 tx=0).stop 2024-03-06T07:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.116+0000 7fc0ea600640 1 -- 172.21.15.6:0/2626662052 reap_dead start 2024-03-06T07:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.117+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 shutdown_connections 2024-03-06T07:27:37.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.117+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 >> 172.21.15.6:0/2626662052 conn(0x7fc0e406afb0 msgr2=0x7fc0e4164a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.117+0000 7f044e000640 1 -- 172.21.15.6:0/2179463360 <== osd.6 v2:172.21.15.178:6824/196133891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f045804f3a0 con 0x7f04380736d0 2024-03-06T07:27:37.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.117+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 shutdown_connections 2024-03-06T07:27:37.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.117+0000 7fc0ec600640 1 -- 172.21.15.6:0/2626662052 wait complete. 2024-03-06T07:27:37.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.118+0000 7fdc24200640 1 -- 172.21.15.6:0/1902109166 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdc1c14aa90 con 0x7fdbfc0737c0 2024-03-06T07:27:37.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.118+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f04380736d0 msgr2=0x7f0438075af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.118+0000 7f045cc00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f04380736d0 0x7f0438075af0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.118+0000 7fdc19600640 1 -- 172.21.15.6:0/1902109166 <== osd.7 v2:172.21.15.178:6808/56642032 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdc1c14aa90 con 0x7fdbfc0737c0 2024-03-06T07:27:37.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f043806b4d0 msgr2=0x7f043806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f045cc00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f043806b4d0 0x7f043806d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04581fb610 tx=0x7f0444009040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04580b50e0 msgr2=0x7f04580c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f045cc00640 1 --2- 172.21.15.6:0/2179463360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f04580b50e0 0x7f04580c8160 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0450033000 tx=0x7f0450004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:37.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f0456800640 1 -- 172.21.15.6:0/2179463360 reap_dead start 2024-03-06T07:27:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 shutdown_connections 2024-03-06T07:27:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 >> 172.21.15.6:0/2179463360 conn(0x7f04580d3f60 msgr2=0x7f04580b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.151 INFO:teuthology.orchestra.run.smithi006.stdout:64424509445 2024-03-06T07:27:37.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x7fdbfc0737c0 msgr2=0x7fdbfc075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.119+0000 7fdc0ea00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x7fdbfc0737c0 0x7fdbfc075be0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbfc06b5a0 msgr2=0x7fdbfc06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbfc06b5a0 0x7fdbfc06da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c159c20 tx=0x7fdc1403f040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c06d440 msgr2=0x7fdc1c0ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 --2- 172.21.15.6:0/1902109166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc1c06d440 0x7fdc1c0ae6a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000b780 tx=0x7fdc1000bc40 comp rx=0 tx=0).stop 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc22200640 1 -- 172.21.15.6:0/1902109166 reap_dead start 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 shutdown_connections 2024-03-06T07:27:37.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 >> 172.21.15.6:0/1902109166 conn(0x7fdc1c06afb0 msgr2=0x7fdc1c0cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 shutdown_connections 2024-03-06T07:27:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7f045cc00640 1 -- 172.21.15.6:0/2179463360 wait complete. 2024-03-06T07:27:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.120+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 shutdown_connections 2024-03-06T07:27:37.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.121+0000 7fdc0ea00640 1 -- 172.21.15.6:0/1902109166 wait complete. 2024-03-06T07:27:37.155 INFO:teuthology.orchestra.run.smithi006.stdout:60129542151 2024-03-06T07:27:37.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:27:37.156 INFO:teuthology.orchestra.run.smithi006.stdout:60129542151 2024-03-06T07:27:37.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:27:37.157 INFO:teuthology.orchestra.run.smithi006.stdout:60129542150 2024-03-06T07:27:37.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:27:37.158 INFO:teuthology.orchestra.run.smithi006.stdout:64424509446 2024-03-06T07:27:37.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:27:37.159 INFO:teuthology.orchestra.run.smithi006.stdout:60129542151 2024-03-06T07:27:37.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:27:37.160 INFO:teuthology.orchestra.run.smithi006.stdout:60129542150 2024-03-06T07:27:37.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:27:37.161 INFO:teuthology.orchestra.run.smithi006.stdout:60129542150 2024-03-06T07:27:37.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:27:37.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.486+0000 7f31ca600640 1 Processor -- start 2024-03-06T07:27:37.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.486+0000 7f31ca600640 1 -- start start 2024-03-06T07:27:37.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.488+0000 7f31ca600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f31bc0e7b10 0x7f31bc006920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.489+0000 7f31ca600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc056ee0 con 0x7f31bc0535a0 2024-03-06T07:27:37.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.489+0000 7f31ca600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0571c0 con 0x7f31bc058b10 2024-03-06T07:27:37.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.489+0000 7f31ca600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31bc057af0 con 0x7f31bc0e7b10 2024-03-06T07:27:37.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.489+0000 7f31c3e00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f31bc058b10 0x7f31bc006210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37128/0 (socket says 172.21.15.6:37128) 2024-03-06T07:27:37.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.489+0000 7f31c3e00640 1 -- 172.21.15.6:0/2363453417 learned_addr learned my addr 172.21.15.6:0/2363453417 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.490+0000 7f31c9c00640 1 --2- 172.21.15.6:0/2363453417 >> v2:172.21.15.178:3300/0 conn(0x7f31bc0e7b10 0x7f31bc006920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.490+0000 7f31c3400640 1 -- 172.21.15.6:0/2363453417 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022689133 0 0) 0x7f31bc0571c0 con 0x7f31bc058b10 2024-03-06T07:27:37.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.490+0000 7f31c3400640 1 -- 172.21.15.6:0/2363453417 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ac003520 con 0x7f31bc058b10 2024-03-06T07:27:37.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.490+0000 7f31c9c00640 1 -- 172.21.15.6:0/2363453417 >> v1:172.21.15.178:6789/0 conn(0x7f31bc058b10 legacy=0x7f31bc006210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.491+0000 7f31c9c00640 1 -- 172.21.15.6:0/2363453417 >> v1:172.21.15.6:6789/0 conn(0x7f31bc0535a0 legacy=0x7f31bc053980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.491+0000 7f31c9c00640 1 -- 172.21.15.6:0/2363453417 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0574a0 con 0x7f31bc0e7b10 2024-03-06T07:27:37.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.492+0000 7f31c9c00640 1 --2- 172.21.15.6:0/2363453417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 0x7f31bc006920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f31b80097a0 tx=0x7f31b802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19267428f28eea4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.493+0000 7f31c3400640 1 -- 172.21.15.6:0/2363453417 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8034070 con 0x7f31bc0e7b10 2024-03-06T07:27:37.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.493+0000 7f31c3400640 1 -- 172.21.15.6:0/2363453417 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b8030610 con 0x7f31bc0e7b10 2024-03-06T07:27:37.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.494+0000 7f31ca600640 1 -- 172.21.15.6:0/2363453417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 msgr2=0x7f31bc006920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.494+0000 7f31ca600640 1 --2- 172.21.15.6:0/2363453417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 0x7f31bc006920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f31b80097a0 tx=0x7f31b802b960 comp rx=0 tx=0).stop 2024-03-06T07:27:37.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.495+0000 7f31ca600640 1 -- 172.21.15.6:0/2363453417 shutdown_connections 2024-03-06T07:27:37.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.495+0000 7f31ca600640 1 --2- 172.21.15.6:0/2363453417 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 0x7f31bc006920 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.495+0000 7f31ca600640 1 -- 172.21.15.6:0/2363453417 >> 172.21.15.6:0/2363453417 conn(0x7f31bc03aa40 msgr2=0x7f31bc03ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.495+0000 7f31ca600640 1 -- 172.21.15.6:0/2363453417 shutdown_connections 2024-03-06T07:27:37.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.495+0000 7f31ca600640 1 -- 172.21.15.6:0/2363453417 wait complete. 2024-03-06T07:27:37.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.498+0000 7f7098c00640 1 Processor -- start 2024-03-06T07:27:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.498+0000 7f31ca600640 1 Processor -- start 2024-03-06T07:27:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.498+0000 7f7098c00640 1 -- start start 2024-03-06T07:27:37.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.498+0000 7f31ca600640 1 -- start start 2024-03-06T07:27:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.500+0000 7f7098c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f708c0e3150 0x7f708c03a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.500+0000 7f7098c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 0x7f708c0e7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.500+0000 7f7098c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708c0e7a80 con 0x7f708c0e6c90 2024-03-06T07:27:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.500+0000 7f7098c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708c0e7640 con 0x7f708c0e3150 2024-03-06T07:27:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.500+0000 7f7098c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f708c053050 con 0x7f708c0544f0 2024-03-06T07:27:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.501+0000 7f31ca600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 0x7f31bc051a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.501+0000 7f7091e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f708c0e3150 0x7f708c03a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.501+0000 7f7091400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f708c0e6c90 0x7f708c005ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37490/0 (socket says 172.21.15.6:37490) 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.501+0000 7f7091e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f708c0e3150 0x7f708c03a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42410/0 (socket says 172.21.15.6:42410) 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.501+0000 7f7092800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 0x7f708c0e7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f7092800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 0x7f708c0e7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:33906/0 (socket says 172.21.15.6:33906) 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f7091e00640 1 -- 172.21.15.6:0/2805157288 learned_addr learned my addr 172.21.15.6:0/2805157288 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f31ca600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31bc058b10 0x7f31bc050a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f7090a00640 1 -- 172.21.15.6:0/2805157288 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252368331 0 0) 0x7f708c0e7a80 con 0x7f708c0e6c90 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f31c9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 0x7f31bc051a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f31c9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 0x7f31bc051a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42424/0 (socket says 172.21.15.6:42424) 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f31ca600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 0x7f31bc050f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc057f80 con 0x7f31bc0535a0 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0e1fa0 con 0x7f31bc0e7b10 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc056b20 con 0x7f31bc058b10 2024-03-06T07:27:37.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.502+0000 7f31c9200640 1 -- 172.21.15.6:0/2143340981 learned_addr learned my addr 172.21.15.6:0/2143340981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f31c3e00640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31bc058b10 0x7f31bc050a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f31c9200640 1 -- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31bc058b10 msgr2=0x7f31bc050a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f31c9200640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31bc058b10 0x7f31bc050a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.504+0000 7f31c9200640 1 -- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 msgr2=0x7f31bc050f70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.504+0000 7f31c9200640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 0x7f31bc050f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.504+0000 7f31c9200640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0dddf0 con 0x7f31bc0535a0 2024-03-06T07:27:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.504+0000 7f31c9200640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 0x7f31bc051a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31c40697f0 tx=0x7f31c4069430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f7091e00640 1 -- 172.21.15.6:0/2805157288 >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 msgr2=0x7f708c0e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.505+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4090070 con 0x7f31bc0535a0 2024-03-06T07:27:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f7091e00640 1 --2- 172.21.15.6:0/2805157288 >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 0x7f708c0e7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f7091e00640 1 -- 172.21.15.6:0/2805157288 >> v1:172.21.15.6:6789/0 conn(0x7f708c0e6c90 legacy=0x7f708c005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.503+0000 7f7091e00640 1 -- 172.21.15.6:0/2805157288 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c0071c0 con 0x7f708c0e3150 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.504+0000 7f7092800640 1 --2- 172.21.15.6:0/2805157288 >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 0x7f708c0e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.505+0000 7f7091e00640 1 --2- 172.21.15.6:0/2805157288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e3150 0x7f708c03a390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f708c034370 tx=0x7f708002ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b360d07ee4b006e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.505+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4073260 con 0x7f31bc0535a0 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.505+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4070690 con 0x7f31bc0535a0 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.505+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc0de0d0 con 0x7f31bc0535a0 2024-03-06T07:27:37.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.506+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc0de690 con 0x7f31bc0535a0 2024-03-06T07:27:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.506+0000 7f7090a00640 1 -- 172.21.15.6:0/2805157288 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708002d820 con 0x7f708c0e3150 2024-03-06T07:27:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.506+0000 7f0861800640 1 Processor -- start 2024-03-06T07:27:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.506+0000 7f7090a00640 1 -- 172.21.15.6:0/2805157288 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708002d9c0 con 0x7f708c0e3150 2024-03-06T07:27:37.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.506+0000 7f7090a00640 1 -- 172.21.15.6:0/2805157288 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708002d820 con 0x7f708c0e3150 2024-03-06T07:27:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.508+0000 7f7098c00640 1 -- 172.21.15.6:0/2805157288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e3150 msgr2=0x7f708c03a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.508+0000 7f7098c00640 1 --2- 172.21.15.6:0/2805157288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e3150 0x7f708c03a390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f708c034370 tx=0x7f708002ba60 comp rx=0 tx=0).stop 2024-03-06T07:27:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.507+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc0052c0 con 0x7f31bc0535a0 2024-03-06T07:27:37.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.506+0000 7f0861800640 1 -- start start 2024-03-06T07:27:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f7098c00640 1 -- 172.21.15.6:0/2805157288 shutdown_connections 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f7098c00640 1 --2- 172.21.15.6:0/2805157288 >> v2:172.21.15.178:3300/0 conn(0x7f708c0544f0 0x7f708c0e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f7098c00640 1 --2- 172.21.15.6:0/2805157288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e3150 0x7f708c03a390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f7098c00640 1 -- 172.21.15.6:0/2805157288 >> 172.21.15.6:0/2805157288 conn(0x7f708c009fa0 msgr2=0x7f708c00a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f7098c00640 1 -- 172.21.15.6:0/2805157288 shutdown_connections 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f7098c00640 1 -- 172.21.15.6:0/2805157288 wait complete. 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.510+0000 7f0861800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f085c15e2e0 0x7f085c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.510+0000 7f0861800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f0861800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f085c0b3940 con 0x7f085c0d0d70 2024-03-06T07:27:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f085a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f085c0d0d70 0x7f085c0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37492/0 (socket says 172.21.15.6:37492) 2024-03-06T07:27:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f085a000640 1 -- 172.21.15.6:0/817506406 learned_addr learned my addr 172.21.15.6:0/817506406 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f085c0d4e70 con 0x7f085c15e2e0 2024-03-06T07:27:37.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f085b400640 1 --2- 172.21.15.6:0/817506406 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.509+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31c4093070 con 0x7f31bc0535a0 2024-03-06T07:27:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f31c0c00640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31a406b5a0 0x7f31a406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.511+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31c4104a90 con 0x7f31bc0535a0 2024-03-06T07:27:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.512+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085c0b3c40 con 0x7f085c0b50e0 2024-03-06T07:27:37.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.512+0000 7f31c3e00640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31a406b5a0 0x7f31a406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.512+0000 7f085aa00640 1 --2- 172.21.15.6:0/817506406 >> v2:172.21.15.178:3300/0 conn(0x7f085c15e2e0 0x7f085c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.512+0000 7f7098c00640 1 Processor -- start 2024-03-06T07:27:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.512+0000 7f085b400640 1 -- 172.21.15.6:0/817506406 >> v2:172.21.15.178:3300/0 conn(0x7f085c15e2e0 msgr2=0x7f085c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.513+0000 7f3868a00640 1 Processor -- start 2024-03-06T07:27:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.513+0000 7f7098c00640 1 -- start start 2024-03-06T07:27:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.513+0000 7f3868a00640 1 -- start start 2024-03-06T07:27:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.513+0000 7f085b400640 1 --2- 172.21.15.6:0/817506406 >> v2:172.21.15.178:3300/0 conn(0x7f085c15e2e0 0x7f085c0d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.513+0000 7f085b400640 1 -- 172.21.15.6:0/817506406 >> v1:172.21.15.6:6789/0 conn(0x7f085c0d0d70 legacy=0x7f085c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.514+0000 7f085b400640 1 -- 172.21.15.6:0/817506406 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085c0cf580 con 0x7f085c0b50e0 2024-03-06T07:27:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.514+0000 7f7098c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f708c0544f0 0x7f708c037430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.514+0000 7f7098c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 0x7f708c0e0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.514+0000 7f7098c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e6c90 0x7f708c044330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f085aa00640 1 --2- 172.21.15.6:0/817506406 >> v2:172.21.15.178:3300/0 conn(0x7f085c15e2e0 0x7f085c0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f31c3e00640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31a406b5a0 0x7f31a406da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f31bc053570 tx=0x7f31b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f3868a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38640d26f0 0x7f38640d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f3868a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f38640d1920 0x7f38640b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f3868a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38640cf4a0 con 0x7f386415e220 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f3868a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38640b36b0 con 0x7f38640d26f0 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f3868a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38640cf780 con 0x7f38640d1920 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f3863400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f386415e220 0x7f38640d1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37138/0 (socket says 172.21.15.6:37138) 2024-03-06T07:27:37.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f3863400640 1 -- 172.21.15.6:0/2764648253 learned_addr learned my addr 172.21.15.6:0/2764648253 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.514+0000 7f7098c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c052ce0 con 0x7f708c0e6c90 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f7098c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c007780 con 0x7f708c0544f0 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f7098c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c0e3530 con 0x7f708c0e3150 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f7091400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 0x7f708c0e0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f7091400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 0x7f708c0e0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47666/0 (socket says 172.21.15.6:47666) 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f7091400640 1 -- 172.21.15.6:0/2400346330 learned_addr learned my addr 172.21.15.6:0/2400346330 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f7092800640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e6c90 0x7f708c044330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.515+0000 7f085b400640 1 --2- 172.21.15.6:0/817506406 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c0d11d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0854009d90 tx=0x7f085402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fb0f359d5af20d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f7091e00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f708c0544f0 0x7f708c037430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f0859600640 1 -- 172.21.15.6:0/817506406 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854039070 con 0x7f085c0b50e0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f0859600640 1 -- 172.21.15.6:0/817506406 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085402cc90 con 0x7f085c0b50e0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f0859600640 1 -- 172.21.15.6:0/817506406 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854034660 con 0x7f085c0b50e0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862a00640 1 --2- 172.21.15.6:0/2764648253 >> v2:172.21.15.6:3300/0 conn(0x7f38640d26f0 0x7f38640d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3863e00640 1 --2- 172.21.15.6:0/2764648253 >> v2:172.21.15.178:3300/0 conn(0x7f38640d1920 0x7f38640b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.516+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489572666 0 0) 0x7f38640cf4a0 con 0x7f386415e220 2024-03-06T07:27:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3848003520 con 0x7f386415e220 2024-03-06T07:27:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f7091400640 1 -- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f708c0544f0 msgr2=0x7f708c037430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f7091400640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f708c0544f0 0x7f708c037430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f7091400640 1 -- 172.21.15.6:0/2400346330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e6c90 msgr2=0x7f708c044330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7feb3f600640 1 Processor -- start 2024-03-06T07:27:37.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f7091400640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e6c90 0x7f708c044330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f7091400640 1 -- 172.21.15.6:0/2400346330 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c042d70 con 0x7f708c0e3150 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f7092800640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e6c90 0x7f708c044330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356766911 0 0) 0x7f3848003520 con 0x7f386415e220 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 >> v2:172.21.15.178:3300/0 conn(0x7f38640d1920 msgr2=0x7f38640b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 --2- 172.21.15.6:0/2764648253 >> v2:172.21.15.178:3300/0 conn(0x7f38640d1920 0x7f38640b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 >> v2:172.21.15.6:3300/0 conn(0x7f38640d26f0 msgr2=0x7f38640d4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 --2- 172.21.15.6:0/2764648253 >> v2:172.21.15.6:3300/0 conn(0x7f38640d26f0 0x7f38640d4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.517+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38640cfa60 con 0x7f386415e220 2024-03-06T07:27:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f385c0019f0 con 0x7f386415e220 2024-03-06T07:27:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 >> v1:172.21.15.178:6789/0 conn(0x7f386415e220 legacy=0x7f38640d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 msgr2=0x7f085c0d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f0861800640 1 --2- 172.21.15.6:0/817506406 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c0d11d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0854009d90 tx=0x7f085402bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38480039b0 0x7f3848003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7feb3f600640 1 -- start start 2024-03-06T07:27:37.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3848004800 0x7f3848004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f384800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 shutdown_connections 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f0861800640 1 --2- 172.21.15.6:0/817506406 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c0d11d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f0861800640 1 --2- 172.21.15.6:0/817506406 >> v2:172.21.15.178:3300/0 conn(0x7f085c15e2e0 0x7f085c0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 >> 172.21.15.6:0/817506406 conn(0x7f085c0d3f60 msgr2=0x7f085c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 shutdown_connections 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f0861800640 1 -- 172.21.15.6:0/817506406 wait complete. 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640b36b0 con 0x7f38480051c0 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640cf780 con 0x7f3848004800 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384800c170 con 0x7f38480039b0 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f3863400640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38480039b0 0x7f3848003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f3863e00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f384800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f3862a00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3848004800 0x7f3848004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.518+0000 7f7091400640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 0x7f708c0e0170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f708400c420 tx=0x7f708400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70840074e0 con 0x7f708c0e3150 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70840047e0 con 0x7f708c0e3150 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7084004c10 con 0x7f708c0e3150 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c043050 con 0x7f708c0e3150 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.519+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c043610 con 0x7f708c0e3150 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f3863e00640 1 -- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38480039b0 msgr2=0x7f3848003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f24ba400640 1 Processor -- start 2024-03-06T07:27:37.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7feb3f600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb38070ee0 0x7feb38162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f3863e00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38480039b0 0x7f3848003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7feb3f600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb38072da0 0x7feb380cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31c40d38f0 con 0x7f31bc0535a0 2024-03-06T07:27:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f3863e00640 1 -- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3848004800 msgr2=0x7f3848004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f3863e00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3848004800 0x7f3848004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f3863e00640 1 -- 172.21.15.6:0/2764648253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38640cfa60 con 0x7f38480051c0 2024-03-06T07:27:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f3862a00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3848004800 0x7f3848004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:37.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.520+0000 7f24ba400640 1 -- start start 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb3f600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb380ce090 con 0x7feb380712c0 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb3f600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb380ceca0 con 0x7feb38072da0 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb3f600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb380ce370 con 0x7feb38070ee0 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb3cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb38070ee0 0x7feb38162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb37e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb38072da0 0x7feb380cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb37e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb38072da0 0x7feb380cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42468/0 (socket says 172.21.15.6:42468) 2024-03-06T07:27:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.521+0000 7feb37e00640 1 -- 172.21.15.6:0/1810400536 learned_addr learned my addr 172.21.15.6:0/1810400536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f0861800640 1 Processor -- start 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f0861800640 1 -- start start 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f24ba400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f24b415e2e0 0x7f24b40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f24ba400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24b40b50e0 0x7f24b40d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f24ba400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b40b3940 con 0x7f24b40d0d70 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f24ba400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b40d4e70 con 0x7f24b40b50e0 2024-03-06T07:27:37.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c0052b0 con 0x7f708c0e3150 2024-03-06T07:27:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f3863e00640 1 -- 172.21.15.6:0/2764648253 reap_dead start 2024-03-06T07:27:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.522+0000 7f24ba400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b40b3c40 con 0x7f24b415e2e0 2024-03-06T07:27:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.523+0000 7fe52c200640 1 Processor -- start 2024-03-06T07:27:37.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.523+0000 7f24b3400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f24b415e2e0 0x7f24b40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f3863e00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f384800b9c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f385000ba30 tx=0x7f3848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eea2301ff99ef01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.523+0000 7f24b3400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f24b415e2e0 0x7f24b40d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:33974/0 (socket says 172.21.15.6:33974) 2024-03-06T07:27:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f24b3400640 1 -- 172.21.15.6:0/363199045 learned_addr learned my addr 172.21.15.6:0/363199045 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f24b3e00640 1 --2- 172.21.15.6:0/363199045 >> v2:172.21.15.6:3300/0 conn(0x7f24b40b50e0 0x7f24b40d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f0861800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c14aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7fe52c200640 1 -- start start 2024-03-06T07:27:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f0861800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f085c0d0d70 0x7f085c15e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f0861800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 0x7f085c15ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7feb37400640 1 -- 172.21.15.6:0/1810400536 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489255855 0 0) 0x7feb380ce090 con 0x7feb380712c0 2024-03-06T07:27:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7feb37400640 1 -- 172.21.15.6:0/1810400536 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb20003520 con 0x7feb380712c0 2024-03-06T07:27:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f0861800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c0cfe20 con 0x7f085c15e2e0 2024-03-06T07:27:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f0861800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c0cfb40 con 0x7f085c0d0d70 2024-03-06T07:27:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f0861800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f085c0d2240 con 0x7f085c0b50e0 2024-03-06T07:27:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7f24b3400640 1 -- 172.21.15.6:0/363199045 >> v2:172.21.15.6:3300/0 conn(0x7f24b40b50e0 msgr2=0x7f24b40d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 0x7f085c15ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 0x7f085c15ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42478/0 (socket says 172.21.15.6:42478) 2024-03-06T07:27:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 -- 172.21.15.6:0/1214181070 learned_addr learned my addr 172.21.15.6:0/1214181070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850035070 con 0x7f38480051c0 2024-03-06T07:27:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3850004260 con 0x7f38480051c0 2024-03-06T07:27:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f3862000640 1 -- 172.21.15.6:0/2764648253 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385000b600 con 0x7f38480051c0 2024-03-06T07:27:37.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085aa00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c14aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085a000640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f085c0d0d70 0x7f085c15e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 -- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 msgr2=0x7f085c14aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c14aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 -- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f085c0d0d70 msgr2=0x7f085c15e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f085c0d0d70 0x7f085c15e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f085b400640 1 -- 172.21.15.6:0/1214181070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085c0b7e30 con 0x7f085c15e2e0 2024-03-06T07:27:37.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f24b2000640 1 -- 172.21.15.6:0/363199045 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454745335 0 0) 0x7f24b40b3940 con 0x7f24b40d0d70 2024-03-06T07:27:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f085aa00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c14aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f085b400640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 0x7f085c15ec00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0854009e80 tx=0x7f0854004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f3868a00640 1 -- 172.21.15.6:0/2764648253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 msgr2=0x7f384800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f3868a00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f384800b9c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f385000ba30 tx=0x7f3848005830 comp rx=0 tx=0).stop 2024-03-06T07:27:37.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.525+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7084011070 con 0x7f708c0e3150 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7feb37e00640 1 -- 172.21.15.6:0/1810400536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb38070ee0 msgr2=0x7feb38162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7feb37e00640 1 --2- 172.21.15.6:0/1810400536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb38070ee0 0x7feb38162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7feb37e00640 1 -- 172.21.15.6:0/1810400536 >> v1:172.21.15.6:6789/0 conn(0x7feb380712c0 legacy=0x7feb3806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.524+0000 7feb37e00640 1 -- 172.21.15.6:0/1810400536 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb380ce650 con 0x7feb38072da0 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f3868a00640 1 -- 172.21.15.6:0/2764648253 shutdown_connections 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f3868a00640 1 --2- 172.21.15.6:0/2764648253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f384800b9c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f3868a00640 1 -- 172.21.15.6:0/2764648253 >> 172.21.15.6:0/2764648253 conn(0x7f38640d3d50 msgr2=0x7f38640d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f24b3400640 1 --2- 172.21.15.6:0/363199045 >> v2:172.21.15.6:3300/0 conn(0x7f24b40b50e0 0x7f24b40d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f24b3400640 1 -- 172.21.15.6:0/363199045 >> v1:172.21.15.178:6789/0 conn(0x7f24b40d0d70 legacy=0x7f24b40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f24b3400640 1 -- 172.21.15.6:0/363199045 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40cf580 con 0x7f24b415e2e0 2024-03-06T07:27:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854044020 con 0x7f085c15e2e0 2024-03-06T07:27:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7feb37e00640 1 --2- 172.21.15.6:0/1810400536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38072da0 0x7feb380cf130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb24009870 tx=0x7feb2402b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c637331d8b3aa4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f708a000640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f707406b5a0 0x7f707406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7084089d00 con 0x7f708c0e3150 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f0861800640 1 -- 172.21.15.6:0/1214181070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085c0b8110 con 0x7f085c15e2e0 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f0861800640 1 -- 172.21.15.6:0/1214181070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085c0b86d0 con 0x7f085c15e2e0 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085403f470 con 0x7f085c15e2e0 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0854044020 con 0x7f085c15e2e0 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7feb37400640 1 -- 172.21.15.6:0/1810400536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24034070 con 0x7feb38072da0 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7feb37400640 1 -- 172.21.15.6:0/1810400536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24030880 con 0x7feb38072da0 2024-03-06T07:27:37.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7feb37400640 1 -- 172.21.15.6:0/1810400536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24030c90 con 0x7feb38072da0 2024-03-06T07:27:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f3868a00640 1 -- 172.21.15.6:0/2764648253 shutdown_connections 2024-03-06T07:27:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.527+0000 7f3868a00640 1 -- 172.21.15.6:0/2764648253 wait complete. 2024-03-06T07:27:37.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.526+0000 7f24b3e00640 1 --2- 172.21.15.6:0/363199045 >> v2:172.21.15.6:3300/0 conn(0x7f24b40b50e0 0x7f24b40d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.528+0000 7fe52c200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe5240d2b40 0x7fe5240d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.528+0000 7fe52c200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5240d1480 0x7fe5240b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.528+0000 7fe52c200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5240b3940 con 0x7fe52415e2e0 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe52c200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5240d4e70 con 0x7fe5240d1480 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe529800640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe52415e2e0 0x7fe5240d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37168/0 (socket says 172.21.15.6:37168) 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe529800640 1 -- 172.21.15.6:0/3533619234 learned_addr learned my addr 172.21.15.6:0/3533619234 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe52c200640 1 -- 172.21.15.6:0/3533619234 --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5240b3c40 con 0x7fe5240d2b40 2024-03-06T07:27:37.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7f0861800640 1 -- 172.21.15.6:0/1214181070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085c0c7690 con 0x7f085c15e2e0 2024-03-06T07:27:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe528e00640 1 --2- 172.21.15.6:0/3533619234 >> v2:172.21.15.178:3300/0 conn(0x7fe5240d2b40 0x7fe5240d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe523e00640 1 -- 172.21.15.6:0/3533619234 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405045657 0 0) 0x7fe5240b3940 con 0x7fe52415e2e0 2024-03-06T07:27:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7fe523e00640 1 -- 172.21.15.6:0/3533619234 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c003520 con 0x7fe52415e2e0 2024-03-06T07:27:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7f24b3400640 1 --2- 172.21.15.6:0/363199045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b415e2e0 0x7f24b40d0830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24b408d8b0 tx=0x7f24a402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be4801aed9afd9f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7f0182800640 1 Processor -- start 2024-03-06T07:27:37.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7fe528e00640 1 -- 172.21.15.6:0/3533619234 >> v2:172.21.15.6:3300/0 conn(0x7fe5240d1480 msgr2=0x7fe5240b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:37.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7fe528e00640 1 --2- 172.21.15.6:0/3533619234 >> v2:172.21.15.6:3300/0 conn(0x7fe5240d1480 0x7fe5240b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7fe528e00640 1 -- 172.21.15.6:0/3533619234 >> v1:172.21.15.178:6789/0 conn(0x7fe52415e2e0 legacy=0x7fe5240d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7fe528e00640 1 -- 172.21.15.6:0/3533619234 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240cf580 con 0x7fe5240d2b40 2024-03-06T07:27:37.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7f0182800640 1 -- start start 2024-03-06T07:27:37.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.529+0000 7f7091e00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f707406b5a0 0x7f707406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7feb3f600640 1 -- 172.21.15.6:0/1810400536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38072da0 msgr2=0x7feb380cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7feb3f600640 1 --2- 172.21.15.6:0/1810400536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38072da0 0x7feb380cf130 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feb24009870 tx=0x7feb2402b9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7f7091e00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f707406b5a0 0x7f707406da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7080036000 tx=0x7f7080008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.530+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0854047020 con 0x7f085c15e2e0 2024-03-06T07:27:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7feb3f600640 1 -- 172.21.15.6:0/1810400536 shutdown_connections 2024-03-06T07:27:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7feb3f600640 1 --2- 172.21.15.6:0/1810400536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38072da0 0x7feb380cf130 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7feb3f600640 1 --2- 172.21.15.6:0/1810400536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb38070ee0 0x7feb38162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7feb3f600640 1 -- 172.21.15.6:0/1810400536 >> 172.21.15.6:0/1810400536 conn(0x7feb3806afb0 msgr2=0x7feb3806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7f3868a00640 1 Processor -- start 2024-03-06T07:27:37.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7f24b2000640 1 -- 172.21.15.6:0/363199045 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4034070 con 0x7f24b415e2e0 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7f0182800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7f0182800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017c0b3940 con 0x7f017c0d1480 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7f0182800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017c0cf580 con 0x7f017c15e2e0 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7f0182800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0d4e70 con 0x7f017c0d2b40 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7f24b2000640 1 -- 172.21.15.6:0/363199045 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a4030860 con 0x7f24b415e2e0 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.531+0000 7f24b2000640 1 -- 172.21.15.6:0/363199045 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4030c30 con 0x7f24b415e2e0 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7feb3f600640 1 -- 172.21.15.6:0/1810400536 shutdown_connections 2024-03-06T07:27:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7feb3f600640 1 -- 172.21.15.6:0/1810400536 wait complete. 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7f3868a00640 1 -- start start 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f3868a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f38641497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f3868a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f386415d820 0x7f38640c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f3868a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3864147f30 0x7f38640c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f3868a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640d4870 con 0x7f38480051c0 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f3868a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640d1d50 con 0x7f3864147f30 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.532+0000 7f24ba400640 1 -- 172.21.15.6:0/363199045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b415e2e0 msgr2=0x7f24b40d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f24ba400640 1 --2- 172.21.15.6:0/363199045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b415e2e0 0x7f24b40d0830 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24b408d8b0 tx=0x7f24a402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f24ba400640 1 -- 172.21.15.6:0/363199045 shutdown_connections 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f24ba400640 1 --2- 172.21.15.6:0/363199045 >> v2:172.21.15.6:3300/0 conn(0x7f24b40b50e0 0x7f24b40d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f24ba400640 1 --2- 172.21.15.6:0/363199045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b415e2e0 0x7f24b40d0830 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24b408d8b0 tx=0x7f24a402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f24ba400640 1 -- 172.21.15.6:0/363199045 >> 172.21.15.6:0/363199045 conn(0x7f24b40d3f60 msgr2=0x7f24b40d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f017aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f017aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0d0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47716/0 (socket says 172.21.15.6:47716) 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f017aa00640 1 -- 172.21.15.6:0/1131715349 learned_addr learned my addr 172.21.15.6:0/1131715349 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f017aa00640 1 -- 172.21.15.6:0/1131715349 >> v1:172.21.15.178:6789/0 conn(0x7f017c15e2e0 legacy=0x7f017c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f017aa00640 1 -- 172.21.15.6:0/1131715349 >> v1:172.21.15.6:6789/0 conn(0x7f017c0d1480 legacy=0x7f017c0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f017aa00640 1 -- 172.21.15.6:0/1131715349 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c0cf860 con 0x7f017c0d2b40 2024-03-06T07:27:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.533+0000 7f3868a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640d49f0 con 0x7f386415d820 2024-03-06T07:27:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3862a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f386415d820 0x7f38640c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f38641497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7fe528e00640 1 --2- 172.21.15.6:0/3533619234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5240d2b40 0x7fe5240d0f40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe5240d3510 tx=0x7fe51c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f692358417a90e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3864147f30 0x7f38640c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f38641497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42494/0 (socket says 172.21.15.6:42494) 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 -- 172.21.15.6:0/1497192899 learned_addr learned my addr 172.21.15.6:0/1497192899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 -- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f386415d820 msgr2=0x7f38640c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f386415d820 0x7f38640c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 -- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3864147f30 msgr2=0x7f38640c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3864147f30 0x7f38640c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7f3863400640 1 -- 172.21.15.6:0/1497192899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38640ca430 con 0x7f38480051c0 2024-03-06T07:27:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.534+0000 7feb3f600640 1 Processor -- start 2024-03-06T07:27:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.535+0000 7f3862a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f386415d820 0x7f38640c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.535+0000 7fe523e00640 1 -- 172.21.15.6:0/3533619234 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c035020 con 0x7fe5240d2b40 2024-03-06T07:27:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.535+0000 7fe523e00640 1 -- 172.21.15.6:0/3533619234 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c031640 con 0x7fe5240d2b40 2024-03-06T07:27:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.535+0000 7fe523e00640 1 -- 172.21.15.6:0/3533619234 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c02d040 con 0x7fe5240d2b40 2024-03-06T07:27:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.535+0000 7feb3f600640 1 -- start start 2024-03-06T07:27:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.535+0000 7f3863400640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f38641497d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f385c007a70 tx=0x7f385c00a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7f017aa00640 1 --2- 172.21.15.6:0/1131715349 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0d0f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01680098a0 tx=0x7f016802ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbbd20348c28fa70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7fe52c200640 1 -- 172.21.15.6:0/3533619234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5240d2b40 msgr2=0x7fe5240d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7fe52c200640 1 --2- 172.21.15.6:0/3533619234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5240d2b40 0x7fe5240d0f40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe5240d3510 tx=0x7fe51c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7fe52c200640 1 -- 172.21.15.6:0/3533619234 shutdown_connections 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7fe52c200640 1 --2- 172.21.15.6:0/3533619234 >> v2:172.21.15.6:3300/0 conn(0x7fe5240d1480 0x7fe5240b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7fe52c200640 1 --2- 172.21.15.6:0/3533619234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5240d2b40 0x7fe5240d0f40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7fe52c200640 1 -- 172.21.15.6:0/3533619234 >> 172.21.15.6:0/3533619234 conn(0x7fe5240d3f60 msgr2=0x7fe5240d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7f0852a00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f083c06b700 0x7f083c06dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08540ac7e0 con 0x7f085c15e2e0 2024-03-06T07:27:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7feb3f600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 0x7feb3815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7feb3f600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb380712c0 0x7feb3809bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7feb3f600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb38072da0 0x7feb381f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7feb3f600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb380cdda0 con 0x7feb38070ee0 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7feb3f600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb380cdc20 con 0x7feb38072da0 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.536+0000 7feb3f600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3806e7d0 con 0x7feb380712c0 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7fe52c200640 1 -- 172.21.15.6:0/3533619234 shutdown_connections 2024-03-06T07:27:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7f085aa00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f083c06b700 0x7f083c06dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7fe52c200640 1 -- 172.21.15.6:0/3533619234 wait complete. 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 0x7feb3815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 0x7feb3815e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42502/0 (socket says 172.21.15.6:42502) 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 -- 172.21.15.6:0/2087914644 learned_addr learned my addr 172.21.15.6:0/2087914644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb37e00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb380712c0 0x7feb3809bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7f24ba400640 1 -- 172.21.15.6:0/363199045 shutdown_connections 2024-03-06T07:27:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7f24ba400640 1 -- 172.21.15.6:0/363199045 wait complete. 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3d600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb38072da0 0x7feb381f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 -- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb380712c0 msgr2=0x7feb3809bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb380712c0 0x7feb3809bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 -- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb38072da0 msgr2=0x7feb381f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb38072da0 0x7feb381f77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.537+0000 7feb3cc00640 1 -- 172.21.15.6:0/2087914644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3809c390 con 0x7feb38070ee0 2024-03-06T07:27:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7feb3cc00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 0x7feb3815e5d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00e9b0 tx=0x7feb2c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c017020 con 0x7f38480051c0 2024-03-06T07:27:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c00ae20 con 0x7f38480051c0 2024-03-06T07:27:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c017070 con 0x7feb38070ee0 2024-03-06T07:27:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c00c9a0 con 0x7feb38070ee0 2024-03-06T07:27:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3809c670 con 0x7feb38070ee0 2024-03-06T07:27:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.538+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c013600 con 0x7f38480051c0 2024-03-06T07:27:37.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.539+0000 7f24ba400640 1 Processor -- start 2024-03-06T07:27:37.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.539+0000 7f24ba400640 1 -- start start 2024-03-06T07:27:37.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.539+0000 7fe52c200640 1 Processor -- start 2024-03-06T07:27:37.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.539+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38640ca6e0 con 0x7f38480051c0 2024-03-06T07:27:37.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.540+0000 7fe52c200640 1 -- start start 2024-03-06T07:27:37.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.539+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c00cd90 con 0x7feb38070ee0 2024-03-06T07:27:37.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.539+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3809cc30 con 0x7feb38070ee0 2024-03-06T07:27:37.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.540+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38640c8e00 con 0x7f38480051c0 2024-03-06T07:27:37.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.541+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38640c9000 con 0x7f38480051c0 2024-03-06T07:27:37.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.541+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb14005290 con 0x7feb38070ee0 2024-03-06T07:27:37.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.541+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f085c0c7690 con 0x7f085c15e2e0 2024-03-06T07:27:37.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.541+0000 7f085aa00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f083c06b700 0x7f083c06dbc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f084c006fd0 tx=0x7f084c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb2c01a070 con 0x7feb38070ee0 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7f24ba400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 0x7f24b415fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7f24ba400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b40d0d70 0x7f24b40ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7f24ba400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24b415ee40 0x7f24b4171940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7f24ba400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40cfe20 con 0x7f24b40b50e0 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7f24ba400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40cfb40 con 0x7f24b40d0d70 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7f24ba400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40d2240 con 0x7f24b415ee40 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f24b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 0x7f24b415fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.541+0000 7fe52c200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5240d2b40 0x7fe5240c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.541+0000 7fe52c200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52415e2e0 0x7fe5240cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52c200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 0x7fe5240cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52c200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240cfe20 con 0x7fe5240722f0 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52c200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240cfb40 con 0x7fe52415e2e0 2024-03-06T07:27:37.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52c200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5240b4610 con 0x7fe5240d2b40 2024-03-06T07:27:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 0x7fe5240cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 0x7fe5240cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42518/0 (socket says 172.21.15.6:42518) 2024-03-06T07:27:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.542+0000 7fe52a200640 1 -- 172.21.15.6:0/1094095605 learned_addr learned my addr 172.21.15.6:0/1094095605 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f385c01a020 con 0x7f38480051c0 2024-03-06T07:27:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7fe52a200640 1 -- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5240d2b40 msgr2=0x7fe5240c9f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:27:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7fe52a200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5240d2b40 0x7fe5240c9f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7fe52a200640 1 -- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52415e2e0 msgr2=0x7fe5240cb030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:27:37.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7fe52a200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52415e2e0 0x7fe5240cb030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7fe52a200640 1 -- 172.21.15.6:0/1094095605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5241f3e90 con 0x7fe5240722f0 2024-03-06T07:27:37.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f24b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 0x7f24b415fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42508/0 (socket says 172.21.15.6:42508) 2024-03-06T07:27:37.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f385b400640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f384406b550 0x7f384406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f24b3400640 1 -- 172.21.15.6:0/1278363056 learned_addr learned my addr 172.21.15.6:0/1278363056 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f24b3e00640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24b415ee40 0x7f24b4171940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7fe52a200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 0x7fe5240cb570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe51000b3f0 tx=0x7fe51000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f24b3400640 1 -- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24b415ee40 msgr2=0x7f24b4171940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7f24b3400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24b415ee40 0x7f24b4171940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f24b2a00640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b40d0d70 0x7f24b40ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f24b3400640 1 -- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b40d0d70 msgr2=0x7f24b40ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f24b3400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b40d0d70 0x7f24b40ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f24b3400640 1 -- 172.21.15.6:0/1278363056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40c93e0 con 0x7f24b40b50e0 2024-03-06T07:27:37.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f24b2a00640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b40d0d70 0x7f24b40ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:37.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f385c08abf0 con 0x7f38480051c0 2024-03-06T07:27:37.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7feb34c00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1806b520 0x7feb1806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.543+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb2c08a460 con 0x7feb38070ee0 2024-03-06T07:27:37.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7feb37e00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1806b520 0x7feb1806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.544+0000 7f24b3400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 0x7f24b415fcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f24a402bf30 tx=0x7f24a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.545+0000 7f3862a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f384406b550 0x7f384406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.546+0000 7feb37e00640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1806b520 0x7feb1806d9e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb2403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.546+0000 7f3862a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f384406b550 0x7f384406da10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3854006fd0 tx=0x7f3854008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.547+0000 7f017a000640 1 -- 172.21.15.6:0/1131715349 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016802f820 con 0x7f017c0d2b40 2024-03-06T07:27:37.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.547+0000 7f017a000640 1 -- 172.21.15.6:0/1131715349 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016802f9c0 con 0x7f017c0d2b40 2024-03-06T07:27:37.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.547+0000 7f017a000640 1 -- 172.21.15.6:0/1131715349 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016802fd90 con 0x7f017c0d2b40 2024-03-06T07:27:37.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.548+0000 7f0182800640 1 -- 172.21.15.6:0/1131715349 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 msgr2=0x7f017c0d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.548+0000 7f0182800640 1 --2- 172.21.15.6:0/1131715349 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0d0f40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01680098a0 tx=0x7f016802ba00 comp rx=0 tx=0).stop 2024-03-06T07:27:37.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.548+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f708401a2b0 con 0x7f708c0e3150 2024-03-06T07:27:37.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.548+0000 7f0182800640 1 -- 172.21.15.6:0/1131715349 shutdown_connections 2024-03-06T07:27:37.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.548+0000 7f0182800640 1 --2- 172.21.15.6:0/1131715349 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0d0f40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.548+0000 7f0182800640 1 -- 172.21.15.6:0/1131715349 >> 172.21.15.6:0/1131715349 conn(0x7f017c0d3f60 msgr2=0x7f017c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.549+0000 7f0182800640 1 -- 172.21.15.6:0/1131715349 shutdown_connections 2024-03-06T07:27:37.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.549+0000 7f0182800640 1 -- 172.21.15.6:0/1131715349 wait complete. 2024-03-06T07:27:37.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.550+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb2c058d70 con 0x7feb38070ee0 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.551+0000 7f0182800640 1 Processor -- start 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.552+0000 7f0182800640 1 -- start start 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.553+0000 7f0182800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0b9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.553+0000 7f0182800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f017c15e2e0 0x7f017c160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.553+0000 7f0182800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f017c0b7bd0 0x7f017c0b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.553+0000 7f0182800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0cfe20 con 0x7f017c0b7bd0 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.553+0000 7f0182800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0b3c40 con 0x7f017c15e2e0 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.553+0000 7f0182800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0b47e0 con 0x7f017c0d2b40 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0b9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0b9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47748/0 (socket says 172.21.15.6:47748) 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 -- 172.21.15.6:0/340599837 learned_addr learned my addr 172.21.15.6:0/340599837 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:37.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017aa00640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f017c15e2e0 0x7f017c160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017be00640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f017c0b7bd0 0x7f017c0b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 -- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f017c15e2e0 msgr2=0x7f017c160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f017c15e2e0 0x7f017c160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 -- 172.21.15.6:0/340599837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f017c0b7bd0 msgr2=0x7f017c0b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f017c0b7bd0 0x7f017c0b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017b400640 1 -- 172.21.15.6:0/340599837 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c160df0 con 0x7f017c0d2b40 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.554+0000 7f017be00640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f017c0b7bd0 0x7f017c0b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.555+0000 7f017b400640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0b9130 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f016c00c430 tx=0x7f016c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.555+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c017070 con 0x7f017c0d2b40 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.556+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c0052d0 con 0x7f017c0d2b40 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.556+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c003830 con 0x7f017c0d2b40 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.557+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c1610d0 con 0x7f017c0d2b40 2024-03-06T07:27:37.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.557+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c161690 con 0x7f017c0d2b40 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4048070 con 0x7f24b40b50e0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a4039d90 con 0x7f24b40b50e0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4034040 con 0x7f24b40b50e0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.561+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38640c9000 con 0x7f38480051c0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510015020 con 0x7fe5240722f0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510011850 con 0x7fe5240722f0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510011c40 con 0x7fe5240722f0 2024-03-06T07:27:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.559+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5240ca480 con 0x7fe5240722f0 2024-03-06T07:27:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.560+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5240c9180 con 0x7fe5240722f0 2024-03-06T07:27:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.558+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c1618d0 con 0x7f017c0d2b40 2024-03-06T07:27:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.561+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b40c96c0 con 0x7f24b40b50e0 2024-03-06T07:27:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.561+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f016c01a070 con 0x7f017c0d2b40 2024-03-06T07:27:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.562+0000 7f0173400640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f015c06b4d0 0x7f015c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.562+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f016c0894b0 con 0x7f017c0d2b40 2024-03-06T07:27:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.563+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe510018020 con 0x7fe5240722f0 2024-03-06T07:27:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.564+0000 7f017aa00640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f015c06b4d0 0x7f015c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.564+0000 7fe521600640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.564+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe51008a120 con 0x7fe5240722f0 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.563+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b40c9c80 con 0x7f24b40b50e0 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.564+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b404f3a0 con 0x7f24b40b50e0 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.565+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24a4037070 con 0x7f24b40b50e0 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.566+0000 7f24ab400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f249406b4d0 0x7f249406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.566+0000 7f017aa00640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f015c06b4d0 0x7f015c06d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0168036000 tx=0x7f0168008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.566+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24a40abfb0 con 0x7f24b40b50e0 2024-03-06T07:27:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.566+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5240b9df0 con 0x7fe5240722f0 2024-03-06T07:27:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.566+0000 7f24b2a00640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f249406b4d0 0x7f249406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.568+0000 7f24b2a00640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f249406b4d0 0x7f249406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f24b415d730 tx=0x7f24a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.568+0000 7fe529800640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.569+0000 7fe529800640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe50406b4d0 0x7fe50406d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe518001f80 tx=0x7fe5180015b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:37.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.571+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24a407a740 con 0x7f24b40b50e0 2024-03-06T07:27:37.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.576+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5240b9df0 con 0x7fe5240722f0 2024-03-06T07:27:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.577+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f016c057cd0 con 0x7f017c0d2b40 2024-03-06T07:27:37.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.947+0000 7f31ca600640 1 -- 172.21.15.6:0/2143340981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f31bc038240 con 0x7f31bc0535a0 2024-03-06T07:27:37.951 INFO:teuthology.orchestra.run.smithi006.stdout:64424509444 2024-03-06T07:27:37.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.948+0000 7f31c0c00640 1 -- 172.21.15.6:0/2143340981 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f31c40d31f0 con 0x7f31bc0535a0 2024-03-06T07:27:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.952+0000 7f31b6000640 1 -- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31a406b5a0 msgr2=0x7f31a406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.952+0000 7f31b6000640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31a406b5a0 0x7f31a406da60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f31bc053570 tx=0x7f31b0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.952+0000 7f31b6000640 1 -- 172.21.15.6:0/2143340981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 msgr2=0x7f31bc051a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.952+0000 7f31b6000640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 0x7f31bc051a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31c40697f0 tx=0x7f31c4069430 comp rx=0 tx=0).stop 2024-03-06T07:27:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.953+0000 7f31b6000640 1 -- 172.21.15.6:0/2143340981 shutdown_connections 2024-03-06T07:27:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.953+0000 7f31b6000640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31a406b5a0 0x7f31a406da60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.953+0000 7f31b6000640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31bc0e7b10 0x7f31bc050f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.953+0000 7f31b6000640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31bc058b10 0x7f31bc050a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.953+0000 7f31b6000640 1 --2- 172.21.15.6:0/2143340981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31bc0535a0 0x7f31bc051a90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.953+0000 7f31b6000640 1 -- 172.21.15.6:0/2143340981 >> 172.21.15.6:0/2143340981 conn(0x7f31bc03aa40 msgr2=0x7f31bc03d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.954+0000 7f31b6000640 1 -- 172.21.15.6:0/2143340981 shutdown_connections 2024-03-06T07:27:37.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.954+0000 7f31b6000640 1 -- 172.21.15.6:0/2143340981 wait complete. 2024-03-06T07:27:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.970+0000 7f0861800640 1 -- 172.21.15.6:0/1214181070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f085c04f3a0 con 0x7f085c15e2e0 2024-03-06T07:27:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.971+0000 7f0852a00640 1 -- 172.21.15.6:0/1214181070 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f085404e030 con 0x7f085c15e2e0 2024-03-06T07:27:37.976 INFO:teuthology.orchestra.run.smithi006.stdout:60129542149 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.976+0000 7f0847e00640 1 -- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f083c06b700 msgr2=0x7f083c06dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.976+0000 7f0847e00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f083c06b700 0x7f083c06dbc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f084c006fd0 tx=0x7f084c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.976+0000 7f0847e00640 1 -- 172.21.15.6:0/1214181070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 msgr2=0x7f085c15ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.976+0000 7f0847e00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 0x7f085c15ec00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0854009e80 tx=0x7f0854004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.977+0000 7f0847e00640 1 -- 172.21.15.6:0/1214181070 shutdown_connections 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.977+0000 7f0847e00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f083c06b700 0x7f083c06dbc0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f084c006fd0 tx=0x7f084c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.977+0000 7f0847e00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f085c0d0d70 0x7f085c15e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.977+0000 7f0847e00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f085c15e2e0 0x7f085c15ec00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.977+0000 7f0847e00640 1 --2- 172.21.15.6:0/1214181070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f085c0b50e0 0x7f085c14aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.977+0000 7f0847e00640 1 -- 172.21.15.6:0/1214181070 >> 172.21.15.6:0/1214181070 conn(0x7f085c0d3f60 msgr2=0x7f085c0d11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.978+0000 7f0847e00640 1 -- 172.21.15.6:0/1214181070 shutdown_connections 2024-03-06T07:27:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.978+0000 7f0847e00640 1 -- 172.21.15.6:0/1214181070 wait complete. 2024-03-06T07:27:37.983 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.2 2024-03-06T07:27:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.985+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f708c00a660 con 0x7f708c0e3150 2024-03-06T07:27:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.986+0000 7f708a000640 1 -- 172.21.15.6:0/2400346330 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7084058bf0 con 0x7f708c0e3150 2024-03-06T07:27:37.988 INFO:teuthology.orchestra.run.smithi006.stdout:60129542149 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.988+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f707406b5a0 msgr2=0x7f707406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.988+0000 7f7098c00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f707406b5a0 0x7f707406da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7080036000 tx=0x7f7080008030 comp rx=0 tx=0).stop 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.988+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 msgr2=0x7f708c0e0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.988+0000 7f7098c00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 0x7f708c0e0170 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f708400c420 tx=0x7f708400c820 comp rx=0 tx=0).stop 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.988+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 shutdown_connections 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f707406b5a0 0x7f707406da60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f708c0e6c90 0x7f708c044330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f708c0e3150 0x7f708c0e0170 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 --2- 172.21.15.6:0/2400346330 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f708c0544f0 0x7f708c037430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 >> 172.21.15.6:0/2400346330 conn(0x7f708c009fa0 msgr2=0x7f708c005d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 shutdown_connections 2024-03-06T07:27:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.989+0000 7f7098c00640 1 -- 172.21.15.6:0/2400346330 wait complete. 2024-03-06T07:27:37.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.994+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feb14005850 con 0x7feb38070ee0 2024-03-06T07:27:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.995+0000 7feb34c00640 1 -- 172.21.15.6:0/2087914644 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb2c05ce10 con 0x7feb38070ee0 2024-03-06T07:27:37.997 INFO:teuthology.orchestra.run.smithi006.stdout:60129542148 2024-03-06T07:27:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.998+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1806b520 msgr2=0x7feb1806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1806b520 0x7feb1806d9e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb24036000 tx=0x7feb2403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 msgr2=0x7feb3815e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 0x7feb3815e5d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feb2c00e9b0 tx=0x7feb2c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 shutdown_connections 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1806b520 0x7feb1806d9e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb38072da0 0x7feb381f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb380712c0 0x7feb3809bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 --2- 172.21.15.6:0/2087914644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb38070ee0 0x7feb3815e5d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:37.999+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 >> 172.21.15.6:0/2087914644 conn(0x7feb3806afb0 msgr2=0x7feb381668c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.000+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 shutdown_connections 2024-03-06T07:27:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.000+0000 7feb3f600640 1 -- 172.21.15.6:0/2087914644 wait complete. 2024-03-06T07:27:38.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.3 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.007+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f017c04f3a0 con 0x7f017c0d2b40 2024-03-06T07:27:38.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.007+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f386404f330 con 0x7f38480051c0 2024-03-06T07:27:38.010 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.0 2024-03-06T07:27:38.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.007+0000 7f385b400640 1 -- 172.21.15.6:0/1497192899 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f385c021020 con 0x7f38480051c0 2024-03-06T07:27:38.010 INFO:teuthology.orchestra.run.smithi006.stdout:64424509445 2024-03-06T07:27:38.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.008+0000 7f0173400640 1 -- 172.21.15.6:0/340599837 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f016c05bd70 con 0x7f017c0d2b40 2024-03-06T07:27:38.011 INFO:teuthology.orchestra.run.smithi006.stdout:60129542148 2024-03-06T07:27:38.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.009+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f24b406c670 con 0x7f24b40b50e0 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.010+0000 7f24ab400640 1 -- 172.21.15.6:0/1278363056 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f24a407e7e0 con 0x7f24b40b50e0 2024-03-06T07:27:38.012 INFO:teuthology.orchestra.run.smithi006.stdout:60129542149 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f384406b550 msgr2=0x7f384406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f384406b550 0x7f384406da10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3854006fd0 tx=0x7f3854008040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 msgr2=0x7f38641497d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f38641497d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f385c007a70 tx=0x7f385c00a770 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 shutdown_connections 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f384406b550 0x7f384406da10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3864147f30 0x7f38640c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f386415d820 0x7f38640c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 --2- 172.21.15.6:0/1497192899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38480051c0 0x7f38641497d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 >> 172.21.15.6:0/1497192899 conn(0x7f38640d3d50 msgr2=0x7f38640b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 shutdown_connections 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f3868a00640 1 -- 172.21.15.6:0/1497192899 wait complete. 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f015c06b4d0 msgr2=0x7f015c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f015c06b4d0 0x7f015c06d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0168036000 tx=0x7f0168008030 comp rx=0 tx=0).stop 2024-03-06T07:27:38.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 msgr2=0x7f017c0b9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0b9130 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f016c00c430 tx=0x7f016c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 shutdown_connections 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f015c06b4d0 0x7f015c06d990 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f017c15e2e0 0x7f017c160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f017c0b7bd0 0x7f017c0b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 --2- 172.21.15.6:0/340599837 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f017c0d2b40 0x7f017c0b9130 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 >> 172.21.15.6:0/340599837 conn(0x7f017c0d3f60 msgr2=0x7f017c06c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.011+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 shutdown_connections 2024-03-06T07:27:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.012+0000 7f0182800640 1 -- 172.21.15.6:0/340599837 wait complete. 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f249406b4d0 msgr2=0x7f249406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f249406b4d0 0x7f249406d990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f24b415d730 tx=0x7f24a0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 msgr2=0x7f24b415fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 0x7f24b415fcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f24a402bf30 tx=0x7f24a4004730 comp rx=0 tx=0).stop 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 shutdown_connections 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f249406b4d0 0x7f249406d990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f24b40d0d70 0x7f24b40ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24b415ee40 0x7f24b4171940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 --2- 172.21.15.6:0/1278363056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f24b40b50e0 0x7f24b415fcd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 >> 172.21.15.6:0/1278363056 conn(0x7f24b40d3f60 msgr2=0x7f24b40d1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 shutdown_connections 2024-03-06T07:27:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.013+0000 7f24ba400640 1 -- 172.21.15.6:0/1278363056 wait complete. 2024-03-06T07:27:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.014+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe52406c650 con 0x7fe5240722f0 2024-03-06T07:27:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.015+0000 7fe521600640 1 -- 172.21.15.6:0/1094095605 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe51001f020 con 0x7fe5240722f0 2024-03-06T07:27:38.017 INFO:teuthology.orchestra.run.smithi006.stdout:60129542148 2024-03-06T07:27:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.018+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe50406b4d0 msgr2=0x7fe50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.018+0000 7fe52c200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe50406b4d0 0x7fe50406d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe518001f80 tx=0x7fe5180015b0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.018+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 msgr2=0x7fe5240cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.018+0000 7fe52c200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 0x7fe5240cb570 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe51000b3f0 tx=0x7fe51000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 shutdown_connections 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe50406b4d0 0x7fe50406d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52415e2e0 0x7fe5240cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5240722f0 0x7fe5240cb570 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 --2- 172.21.15.6:0/1094095605 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5240d2b40 0x7fe5240c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 >> 172.21.15.6:0/1094095605 conn(0x7fe5240d3f60 msgr2=0x7fe5240b6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 shutdown_connections 2024-03-06T07:27:38.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:38.019+0000 7fe52c200640 1 -- 172.21.15.6:0/1094095605 wait complete. 2024-03-06T07:27:38.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.4 2024-03-06T07:27:38.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.7 2024-03-06T07:27:38.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542148 for osd.6 2024-03-06T07:27:38.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.1 2024-03-06T07:27:38.041 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.5 2024-03-06T07:27:38.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:27:39.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:27:39.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:27:39.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:27:39.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:27:39.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:27:39.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:27:39.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:27:39.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.253+0000 7fac1f200640 1 Processor -- start 2024-03-06T07:27:39.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.253+0000 7fac1f200640 1 -- start start 2024-03-06T07:27:39.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.255+0000 7fac1f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fac180e3160 0x7fac1803a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.255+0000 7fac1f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fac180e6ca0 0x7fac180e7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.255+0000 7fac1f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac180e7a90 con 0x7fac18005ee0 2024-03-06T07:27:39.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.255+0000 7fac1f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac180e7650 con 0x7fac180e3160 2024-03-06T07:27:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.255+0000 7fac1f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac18053060 con 0x7fac180e6ca0 2024-03-06T07:27:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.256+0000 7fac1e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fac18005ee0 0x7fac18006300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37534/0 (socket says 172.21.15.6:37534) 2024-03-06T07:27:39.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.256+0000 7fac1e800640 1 -- 172.21.15.6:0/1346095937 learned_addr learned my addr 172.21.15.6:0/1346095937 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.256+0000 7fac1d400640 1 --2- 172.21.15.6:0/1346095937 >> v2:172.21.15.178:3300/0 conn(0x7fac180e6ca0 0x7fac180e7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.257+0000 7fac1de00640 1 --2- 172.21.15.6:0/1346095937 >> v2:172.21.15.6:3300/0 conn(0x7fac180e3160 0x7fac1803a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.259+0000 7fac1d400640 1 -- 172.21.15.6:0/1346095937 >> v2:172.21.15.6:3300/0 conn(0x7fac180e3160 msgr2=0x7fac1803a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.259+0000 7fac1ca00640 1 -- 172.21.15.6:0/1346095937 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105513958 0 0) 0x7fac180e7a90 con 0x7fac18005ee0 2024-03-06T07:27:39.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.259+0000 7fac1d400640 1 --2- 172.21.15.6:0/1346095937 >> v2:172.21.15.6:3300/0 conn(0x7fac180e3160 0x7fac1803a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.259+0000 7fac1d400640 1 -- 172.21.15.6:0/1346095937 >> v1:172.21.15.6:6789/0 conn(0x7fac18005ee0 legacy=0x7fac18006300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.259+0000 7fac1d400640 1 -- 172.21.15.6:0/1346095937 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac180071d0 con 0x7fac180e6ca0 2024-03-06T07:27:39.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.260+0000 7fac1de00640 1 --2- 172.21.15.6:0/1346095937 >> v2:172.21.15.6:3300/0 conn(0x7fac180e3160 0x7fac1803a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:39.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.261+0000 7fac1d400640 1 --2- 172.21.15.6:0/1346095937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac180e7110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac2002d580 tx=0x7fac200a6b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e4764890930dae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.262+0000 7fac1ca00640 1 -- 172.21.15.6:0/1346095937 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200a89c0 con 0x7fac180e6ca0 2024-03-06T07:27:39.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.262+0000 7fac1ca00640 1 -- 172.21.15.6:0/1346095937 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac200a8b60 con 0x7fac180e6ca0 2024-03-06T07:27:39.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.262+0000 7fac1ca00640 1 -- 172.21.15.6:0/1346095937 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200b0420 con 0x7fac180e6ca0 2024-03-06T07:27:39.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.263+0000 7fac1f200640 1 -- 172.21.15.6:0/1346095937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 msgr2=0x7fac180e7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.263+0000 7fac1f200640 1 --2- 172.21.15.6:0/1346095937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac180e7110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fac2002d580 tx=0x7fac200a6b40 comp rx=0 tx=0).stop 2024-03-06T07:27:39.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.264+0000 7fac1f200640 1 -- 172.21.15.6:0/1346095937 shutdown_connections 2024-03-06T07:27:39.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.264+0000 7fac1f200640 1 --2- 172.21.15.6:0/1346095937 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac180e7110 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.264+0000 7fac1f200640 1 --2- 172.21.15.6:0/1346095937 >> v2:172.21.15.6:3300/0 conn(0x7fac180e3160 0x7fac1803a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.264+0000 7fac1f200640 1 -- 172.21.15.6:0/1346095937 >> 172.21.15.6:0/1346095937 conn(0x7fac18009fb0 msgr2=0x7fac1800a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.265+0000 7fac1f200640 1 -- 172.21.15.6:0/1346095937 shutdown_connections 2024-03-06T07:27:39.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.265+0000 7fac1f200640 1 -- 172.21.15.6:0/1346095937 wait complete. 2024-03-06T07:27:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.268+0000 7fac1f200640 1 Processor -- start 2024-03-06T07:27:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.268+0000 7fac1f200640 1 -- start start 2024-03-06T07:27:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.270+0000 7fac1f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 0x7fac180d7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.270+0000 7fac1f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 0x7fac1804bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.270+0000 7fac1f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac1804c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.271+0000 7fac1d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 0x7fac1804bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.271+0000 7fac1d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 0x7fac1804bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42542/0 (socket says 172.21.15.6:42542) 2024-03-06T07:27:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.272+0000 7fac1de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 0x7fac180d7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.272+0000 7fac1de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 0x7fac180d7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47750/0 (socket says 172.21.15.6:47750) 2024-03-06T07:27:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.272+0000 7fac1d400640 1 -- 172.21.15.6:0/3588658865 learned_addr learned my addr 172.21.15.6:0/3588658865 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.272+0000 7fac1e800640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac1804c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.272+0000 7fac1f200640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18052cf0 con 0x7fac180e3160 2024-03-06T07:27:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.273+0000 7fac1f200640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac18007790 con 0x7fac180e6ca0 2024-03-06T07:27:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.273+0000 7fac1f200640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac180e3540 con 0x7fac18005ee0 2024-03-06T07:27:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.274+0000 7fac1de00640 1 -- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 msgr2=0x7fac1804c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.274+0000 7fac1de00640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac1804c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.274+0000 7fac1de00640 1 -- 172.21.15.6:0/3588658865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 msgr2=0x7fac1804bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.275+0000 7fac1de00640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 0x7fac1804bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.275+0000 7fac1de00640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac18036d50 con 0x7fac18005ee0 2024-03-06T07:27:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.275+0000 7fac1d400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 0x7fac1804bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.276+0000 7fac1de00640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 0x7fac180d7230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac1000e900 tx=0x7fac1000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.277+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10017070 con 0x7fac18005ee0 2024-03-06T07:27:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.277+0000 7fac1f200640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac18037030 con 0x7fac18005ee0 2024-03-06T07:27:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.277+0000 7fac1f200640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac180375f0 con 0x7fac18005ee0 2024-03-06T07:27:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.279+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac10009300 con 0x7fac18005ee0 2024-03-06T07:27:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.279+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac10017070 con 0x7fac18005ee0 2024-03-06T07:27:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.281+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf8005290 con 0x7fac18005ee0 2024-03-06T07:27:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.284+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac1001a070 con 0x7fac18005ee0 2024-03-06T07:27:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.286+0000 7fac16000640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fac0006b6d0 0x7fac0006db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.287+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac1008a100 con 0x7fac18005ee0 2024-03-06T07:27:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.287+0000 7fac1d400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fac0006b6d0 0x7fac0006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.289+0000 7fac1d400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fac0006b6d0 0x7fac0006db90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac2007abf0 tx=0x7fac200ac000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.300+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fac10058890 con 0x7fac18005ee0 2024-03-06T07:27:39.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.307+0000 7fc802e00640 1 Processor -- start 2024-03-06T07:27:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.307+0000 7fc802e00640 1 -- start start 2024-03-06T07:27:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.309+0000 7fc802e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7fc070ee0 0x7fc7fc162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.309+0000 7fc802e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc7fc0712c0 0x7fc7fc06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.309+0000 7fc802e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc0ce090 con 0x7fc7fc072da0 2024-03-06T07:27:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.309+0000 7fc802e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7fc0ceca0 con 0x7fc7fc070ee0 2024-03-06T07:27:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.309+0000 7fc802e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7fc0ce370 con 0x7fc7fc0712c0 2024-03-06T07:27:39.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.311+0000 7fc7fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7fc070ee0 0x7fc7fc162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.311+0000 7fc800e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc7fc0712c0 0x7fc7fc06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.312+0000 7fc7fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7fc070ee0 0x7fc7fc162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42552/0 (socket says 172.21.15.6:42552) 2024-03-06T07:27:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.312+0000 7fc7fbe00640 1 -- 172.21.15.6:0/475434460 learned_addr learned my addr 172.21.15.6:0/475434460 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.311+0000 7fc7fb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc7fc072da0 0x7fc7fc06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37540/0 (socket says 172.21.15.6:37540) 2024-03-06T07:27:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.313+0000 7fc7faa00640 1 -- 172.21.15.6:0/475434460 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347769308 0 0) 0x7fc7fc0ce090 con 0x7fc7fc072da0 2024-03-06T07:27:39.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7fc7faa00640 1 -- 172.21.15.6:0/475434460 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0003520 con 0x7fc7fc072da0 2024-03-06T07:27:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7fc7fbe00640 1 -- 172.21.15.6:0/475434460 >> v2:172.21.15.178:3300/0 conn(0x7fc7fc0712c0 msgr2=0x7fc7fc06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7fc7fbe00640 1 --2- 172.21.15.6:0/475434460 >> v2:172.21.15.178:3300/0 conn(0x7fc7fc0712c0 0x7fc7fc06e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.312+0000 7ff78e400640 1 Processor -- start 2024-03-06T07:27:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.312+0000 7ff78e400640 1 -- start start 2024-03-06T07:27:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7ff78e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7880b8310 0x7ff78816d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7ff78e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7880b6e50 con 0x7ff7880b52d0 2024-03-06T07:27:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7ff78e400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7880d1530 con 0x7ff7880c7320 2024-03-06T07:27:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7ff78e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7880cf720 con 0x7ff7880b8310 2024-03-06T07:27:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.315+0000 7ff786a00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7ff7880c7320 0x7ff7880d0e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37190/0 (socket says 172.21.15.6:37190) 2024-03-06T07:27:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.315+0000 7ff786a00640 1 -- 172.21.15.6:0/4162069596 learned_addr learned my addr 172.21.15.6:0/4162069596 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.316+0000 7ff787e00640 1 --2- 172.21.15.6:0/4162069596 >> v2:172.21.15.6:3300/0 conn(0x7ff7880b8310 0x7ff78816d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7fc7fbe00640 1 -- 172.21.15.6:0/475434460 >> v1:172.21.15.6:6789/0 conn(0x7fc7fc072da0 legacy=0x7fc7fc06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7fc7fbe00640 1 -- 172.21.15.6:0/475434460 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc0ce650 con 0x7fc7fc070ee0 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.314+0000 7fc800e00640 1 --2- 172.21.15.6:0/475434460 >> v2:172.21.15.178:3300/0 conn(0x7fc7fc0712c0 0x7fc7fc06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.316+0000 7fc7fbe00640 1 --2- 172.21.15.6:0/475434460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc070ee0 0x7fc7fc162c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc06f070 tx=0x7fc7e402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa360dc37caed7cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7fc7faa00640 1 -- 172.21.15.6:0/475434460 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034070 con 0x7fc7fc070ee0 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7fc7faa00640 1 -- 172.21.15.6:0/475434460 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e4030870 con 0x7fc7fc070ee0 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7fc7faa00640 1 -- 172.21.15.6:0/475434460 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4030c40 con 0x7fc7fc070ee0 2024-03-06T07:27:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7f19b5200640 1 Processor -- start 2024-03-06T07:27:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454789018 0 0) 0x7ff7880d1530 con 0x7ff7880c7320 2024-03-06T07:27:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff76c003520 con 0x7ff7880c7320 2024-03-06T07:27:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.317+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251857960 0 0) 0x7ff7880b6e50 con 0x7ff7880b52d0 2024-03-06T07:27:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 -- 172.21.15.6:0/475434460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc070ee0 msgr2=0x7fc7fc162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 --2- 172.21.15.6:0/475434460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc070ee0 0x7fc7fc162c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc06f070 tx=0x7fc7e402b9b0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7880d1530 con 0x7ff7880b52d0 2024-03-06T07:27:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7f19b5200640 1 -- start start 2024-03-06T07:27:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 <== mon.1 v1:172.21.15.178:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570598820 0 0) 0x7ff76c003520 con 0x7ff7880c7320 2024-03-06T07:27:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 >> v2:172.21.15.6:3300/0 conn(0x7ff7880b8310 msgr2=0x7ff78816d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 -- 172.21.15.6:0/475434460 shutdown_connections 2024-03-06T07:27:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 --2- 172.21.15.6:0/475434460 >> v2:172.21.15.178:3300/0 conn(0x7fc7fc0712c0 0x7fc7fc06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 --2- 172.21.15.6:0/475434460 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc070ee0 0x7fc7fc162c40 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc06f070 tx=0x7fc7e402b9b0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 -- 172.21.15.6:0/475434460 >> 172.21.15.6:0/475434460 conn(0x7fc7fc06afb0 msgr2=0x7fc7fc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7fc802e00640 1 -- 172.21.15.6:0/475434460 shutdown_connections 2024-03-06T07:27:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.319+0000 7fc802e00640 1 -- 172.21.15.6:0/475434460 wait complete. 2024-03-06T07:27:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 --2- 172.21.15.6:0/4162069596 >> v2:172.21.15.6:3300/0 conn(0x7ff7880b8310 0x7ff78816d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 >> v1:172.21.15.6:6789/0 conn(0x7ff7880b52d0 legacy=0x7ff7880d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 --> v1:172.21.15.178:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880cfa00 con 0x7ff7880c7320 2024-03-06T07:27:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 <== mon.1 v1:172.21.15.178:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff774001670 con 0x7ff7880c7320 2024-03-06T07:27:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 >> v1:172.21.15.178:6789/0 conn(0x7ff7880c7320 legacy=0x7ff7880d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c003bb0 0x7ff76c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff76c004800 0x7ff76c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff76c0051c0 0x7ff76c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880cf720 con 0x7ff76c004800 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c00c040 con 0x7ff76c003bb0 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.318+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c00c240 con 0x7ff76c0051c0 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.319+0000 7ff787e00640 1 --2- 172.21.15.6:0/4162069596 >> v2:172.21.15.6:3300/0 conn(0x7ff7880b8310 0x7ff78816d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7ff787e00640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff76c0051c0 0x7ff76c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7f19b5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19b00b2300 0x7f19b0140360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7f19b5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b006ce20 0x7f19b006d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7f19b5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19b0160e70 con 0x7f19b00b2ae0 2024-03-06T07:27:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7f19b5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19b01408a0 con 0x7f19b00b2300 2024-03-06T07:27:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7f19b5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b006e4b0 con 0x7f19b006ce20 2024-03-06T07:27:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.320+0000 7ff787400640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c003bb0 0x7ff76c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7f19aea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b006ce20 0x7f19b006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7f19afe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f19b00b2ae0 0x7f19b015a500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37560/0 (socket says 172.21.15.6:37560) 2024-03-06T07:27:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7f19afe00640 1 -- 172.21.15.6:0/2062671515 learned_addr learned my addr 172.21.15.6:0/2062671515 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7ff787e00640 1 -- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c003bb0 msgr2=0x7ff76c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7fc802e00640 1 Processor -- start 2024-03-06T07:27:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7f19af400640 1 --2- 172.21.15.6:0/2062671515 >> v2:172.21.15.6:3300/0 conn(0x7f19b00b2300 0x7f19b0140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.322+0000 7f19af400640 1 -- 172.21.15.6:0/2062671515 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b006ce20 msgr2=0x7f19b006d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7f19ae000640 1 -- 172.21.15.6:0/2062671515 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82686155 0 0) 0x7f19b0160e70 con 0x7f19b00b2ae0 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7ff787e00640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c003bb0 0x7ff76c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7ff787e00640 1 -- 172.21.15.6:0/4162069596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff76c004800 msgr2=0x7ff76c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7ff787e00640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff76c004800 0x7ff76c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.321+0000 7ff787e00640 1 -- 172.21.15.6:0/4162069596 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880cfa00 con 0x7ff76c0051c0 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7ff787e00640 1 -- 172.21.15.6:0/4162069596 reap_dead start 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.324+0000 7ff787e00640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff76c0051c0 0x7ff76c00b9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff78000e350 tx=0x7ff76c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20549c44a49f629 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7fc802e00640 1 -- start start 2024-03-06T07:27:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.324+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780009de0 con 0x7ff76c0051c0 2024-03-06T07:27:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7f19af400640 1 --2- 172.21.15.6:0/2062671515 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b006ce20 0x7f19b006d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7f19af400640 1 -- 172.21.15.6:0/2062671515 >> v1:172.21.15.6:6789/0 conn(0x7f19b00b2ae0 legacy=0x7f19b015a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.324+0000 7ff786000640 1 -- 172.21.15.6:0/4162069596 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78002ce40 con 0x7ff76c0051c0 2024-03-06T07:27:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.323+0000 7f19af400640 1 -- 172.21.15.6:0/2062671515 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b006db80 con 0x7f19b00b2300 2024-03-06T07:27:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.324+0000 7f19aea00640 1 --2- 172.21.15.6:0/2062671515 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b006ce20 0x7f19b006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7fc802e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 0x7fc7fc150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7f19af400640 1 --2- 172.21.15.6:0/2062671515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b0140360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009d90 tx=0x7f19a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdaf0e25e915124 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7fc802e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc072da0 0x7fc7fc150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7fc802e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7fc14ea70 0x7fc7fc1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7ff78e400640 1 -- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff76c0051c0 msgr2=0x7ff76c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7ff78e400640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff76c0051c0 0x7ff76c00b9c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff78000e350 tx=0x7ff76c005830 comp rx=0 tx=0).stop 2024-03-06T07:27:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7fc802e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc0cdda0 con 0x7fc7fc072da0 2024-03-06T07:27:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7ff78e400640 1 -- 172.21.15.6:0/4162069596 shutdown_connections 2024-03-06T07:27:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7ff78e400640 1 --2- 172.21.15.6:0/4162069596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff76c0051c0 0x7ff76c00b9c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7ff78e400640 1 -- 172.21.15.6:0/4162069596 >> 172.21.15.6:0/4162069596 conn(0x7ff7880d3f60 msgr2=0x7ff7880d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7fc802e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc0cdc20 con 0x7fc7fc14ea70 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.325+0000 7fc802e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc0cf130 con 0x7fc7fc0712c0 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 0x7fc7fc150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 0x7fc7fc150240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47774/0 (socket says 172.21.15.6:47774) 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 -- 172.21.15.6:0/3334816686 learned_addr learned my addr 172.21.15.6:0/3334816686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fb400640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc072da0 0x7fc7fc150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7ff78e400640 1 -- 172.21.15.6:0/4162069596 shutdown_connections 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 -- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7fc14ea70 msgr2=0x7fc7fc1511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7fc14ea70 0x7fc7fc1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 -- 172.21.15.6:0/3334816686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc072da0 msgr2=0x7fc7fc150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc072da0 0x7fc7fc150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.326+0000 7fc7fbe00640 1 -- 172.21.15.6:0/3334816686 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc0b3db0 con 0x7fc7fc0712c0 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.327+0000 7fc7fbe00640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 0x7fc7fc150240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009960 tx=0x7fc7e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.327+0000 7f19ae000640 1 -- 172.21.15.6:0/2062671515 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8039070 con 0x7f19b00b2300 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.327+0000 7f19ae000640 1 -- 172.21.15.6:0/2062671515 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19a802cc50 con 0x7f19b00b2300 2024-03-06T07:27:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.327+0000 7f19ae000640 1 -- 172.21.15.6:0/2062671515 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a8034440 con 0x7f19b00b2300 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.327+0000 7ff78e400640 1 -- 172.21.15.6:0/4162069596 wait complete. 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034040 con 0x7fc7fc0712c0 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7fc802e00640 1 -- 172.21.15.6:0/3334816686 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7fc0b4090 con 0x7fc7fc0712c0 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e4030670 con 0x7fc7fc0712c0 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7fc802e00640 1 -- 172.21.15.6:0/3334816686 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7fc150780 con 0x7fc7fc0712c0 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7f19b5200640 1 -- 172.21.15.6:0/2062671515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 msgr2=0x7f19b0140360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7f19b5200640 1 --2- 172.21.15.6:0/2062671515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b0140360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009d90 tx=0x7f19a802bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.328+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034040 con 0x7fc7fc0712c0 2024-03-06T07:27:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.329+0000 7f19b5200640 1 -- 172.21.15.6:0/2062671515 shutdown_connections 2024-03-06T07:27:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.329+0000 7f19b5200640 1 --2- 172.21.15.6:0/2062671515 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b006ce20 0x7f19b006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.329+0000 7f19b5200640 1 --2- 172.21.15.6:0/2062671515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b0140360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.329+0000 7f19b5200640 1 -- 172.21.15.6:0/2062671515 >> 172.21.15.6:0/2062671515 conn(0x7f19b00b17e0 msgr2=0x7f19b00b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.330+0000 7ff78e400640 1 Processor -- start 2024-03-06T07:27:39.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.331+0000 7ff78e400640 1 -- start start 2024-03-06T07:27:39.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.330+0000 7f19b5200640 1 -- 172.21.15.6:0/2062671515 shutdown_connections 2024-03-06T07:27:39.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.330+0000 7f19b5200640 1 -- 172.21.15.6:0/2062671515 wait complete. 2024-03-06T07:27:39.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7ff78e400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 0x7ff788145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7f19b5200640 1 Processor -- start 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7ff78e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7880cd260 0x7ff7880c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7ff78e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7881489b0 0x7ff788148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7ff78e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880b56b0 con 0x7ff7880cd260 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7ff78e400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880cffc0 con 0x7ff76c0051c0 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7ff78e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7880b5830 con 0x7ff7881489b0 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7ff787400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 0x7ff788145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7ff787400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 0x7ff788145df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:34100/0 (socket says 172.21.15.6:34100) 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7ff787400640 1 -- 172.21.15.6:0/2134698876 learned_addr learned my addr 172.21.15.6:0/2134698876 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7ff787e00640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7881489b0 0x7ff788148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7f19b5200640 1 -- start start 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7e4037070 con 0x7fc7fc0712c0 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.333+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7fc069510 con 0x7fc7fc0712c0 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7fc7f3e00640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7d806b6d0 0x7fc7d806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.334+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7e40ac310 con 0x7fc7fc0712c0 2024-03-06T07:27:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7ff786a00640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7880cd260 0x7ff7880c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7ff787400640 1 -- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7881489b0 msgr2=0x7ff788148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7ff787400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7881489b0 0x7ff788148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7ff787400640 1 -- 172.21.15.6:0/2134698876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7880cd260 msgr2=0x7ff7880c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7ff787400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7880cd260 0x7ff7880c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7fc7fb400640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7d806b6d0 0x7fc7d806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.335+0000 7ff787400640 1 -- 172.21.15.6:0/2134698876 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7880cc4e0 con 0x7ff76c0051c0 2024-03-06T07:27:39.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19b5200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19b006ce20 0x7f19b013ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19b5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b013ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19b5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b00b2ae0 0x7f19b00abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19b5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b015dcb0 con 0x7f19b00b2300 2024-03-06T07:27:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19b5200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b006e140 con 0x7f19b006ce20 2024-03-06T07:27:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19b5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19b015de30 con 0x7f19b00b2ae0 2024-03-06T07:27:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19aea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b013ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19aea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b013ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42590/0 (socket says 172.21.15.6:42590) 2024-03-06T07:27:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19aea00640 1 -- 172.21.15.6:0/3022102950 learned_addr learned my addr 172.21.15.6:0/3022102950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7ff787400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 0x7ff788145df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff77800f8d0 tx=0x7ff77800fd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.336+0000 7f19af400640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19b006ce20 0x7f19b013ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19afe00640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b00b2ae0 0x7f19b00abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19aea00640 1 -- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b00b2ae0 msgr2=0x7f19b00abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77801a070 con 0x7ff76c0051c0 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19aea00640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b00b2ae0 0x7f19b00abfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19aea00640 1 -- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19b006ce20 msgr2=0x7f19b013ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19aea00640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19b006ce20 0x7f19b013ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff778005320 con 0x7ff76c0051c0 2024-03-06T07:27:39.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19aea00640 1 -- 172.21.15.6:0/3022102950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b00adff0 con 0x7f19b00b2300 2024-03-06T07:27:39.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff778007290 con 0x7ff76c0051c0 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.337+0000 7f19aea00640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b013ef70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f199c00c7f0 tx=0x7f199c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.338+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c015070 con 0x7f19b00b2300 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.338+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c007a30 con 0x7f19b00b2300 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.338+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c007e00 con 0x7f19b00b2300 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.338+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b00ae290 con 0x7f19b00b2300 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.338+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff788146330 con 0x7ff76c0051c0 2024-03-06T07:27:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.338+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7880cc780 con 0x7ff76c0051c0 2024-03-06T07:27:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.341+0000 7fc7fb400640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7d806b6d0 0x7fc7d806db90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0ae960 tx=0x7fc7ec007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.340+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b00ae850 con 0x7f19b00b2300 2024-03-06T07:27:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.341+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b004f330 con 0x7f19b00b2300 2024-03-06T07:27:39.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.342+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f199c018070 con 0x7f19b00b2300 2024-03-06T07:27:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.344+0000 7f19a7400640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f199006b520 0x7f199006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.344+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f199c08a120 con 0x7f19b00b2300 2024-03-06T07:27:39.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.344+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78804f3a0 con 0x7ff76c0051c0 2024-03-06T07:27:39.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.345+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff778007430 con 0x7ff76c0051c0 2024-03-06T07:27:39.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.347+0000 7f19af400640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f199006b520 0x7f199006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.347+0000 7ff77f400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff76806b500 0x7ff76806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.347+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff778089850 con 0x7ff76c0051c0 2024-03-06T07:27:39.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.348+0000 7ff786a00640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff76806b500 0x7ff76806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.348+0000 7f19af400640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f199006b520 0x7f199006d9e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19a8036000 tx=0x7f19a8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.349+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f199c058930 con 0x7f19b00b2300 2024-03-06T07:27:39.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.349+0000 7ff786a00640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff76806b500 0x7ff76806d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff788148910 tx=0x7ff774009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.355+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7e407b180 con 0x7fc7fc0712c0 2024-03-06T07:27:39.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.364+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7780580a0 con 0x7ff76c0051c0 2024-03-06T07:27:39.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.376+0000 7fc93ca00640 1 Processor -- start 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.377+0000 7fc93ca00640 1 -- start start 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.379+0000 7fc93ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 0x7fc9380d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.379+0000 7fc93ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9380d0d70 0x7fc9380d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.379+0000 7fc93ca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380b3940 con 0x7fc9380d1b30 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.379+0000 7fc93ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9380d4e70 con 0x7fc9380d0d70 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.379+0000 7fc93ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380b3c40 con 0x7fc93815e2e0 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.380+0000 7fc935400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc9380d0d70 0x7fc9380d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.380+0000 7fc935e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 0x7fc9380d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.380+0000 7fc935e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 0x7fc9380d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47804/0 (socket says 172.21.15.6:47804) 2024-03-06T07:27:39.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.380+0000 7fc935e00640 1 -- 172.21.15.6:0/1407271614 learned_addr learned my addr 172.21.15.6:0/1407271614 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.381+0000 7fc934a00640 1 -- 172.21.15.6:0/1407271614 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321486340 0 0) 0x7fc9380b3940 con 0x7fc9380d1b30 2024-03-06T07:27:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.381+0000 7fc935e00640 1 -- 172.21.15.6:0/1407271614 >> v2:172.21.15.178:3300/0 conn(0x7fc9380d0d70 msgr2=0x7fc9380d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.381+0000 7fc935e00640 1 --2- 172.21.15.6:0/1407271614 >> v2:172.21.15.178:3300/0 conn(0x7fc9380d0d70 0x7fc9380d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.381+0000 7fc935e00640 1 -- 172.21.15.6:0/1407271614 >> v1:172.21.15.178:6789/0 conn(0x7fc9380d1b30 legacy=0x7fc9380d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.381+0000 7fc935e00640 1 -- 172.21.15.6:0/1407271614 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380cf580 con 0x7fc93815e2e0 2024-03-06T07:27:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.386+0000 7fc935e00640 1 --2- 172.21.15.6:0/1407271614 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 0x7fc9380d0830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc920009870 tx=0x7fc92002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9d46e96ec020874 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.387+0000 7fc934a00640 1 -- 172.21.15.6:0/1407271614 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920034070 con 0x7fc93815e2e0 2024-03-06T07:27:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.387+0000 7fc934a00640 1 -- 172.21.15.6:0/1407271614 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920030880 con 0x7fc93815e2e0 2024-03-06T07:27:39.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.387+0000 7fc934a00640 1 -- 172.21.15.6:0/1407271614 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920030c50 con 0x7fc93815e2e0 2024-03-06T07:27:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.388+0000 7fc93ca00640 1 -- 172.21.15.6:0/1407271614 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 msgr2=0x7fc9380d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.388+0000 7fc93ca00640 1 --2- 172.21.15.6:0/1407271614 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 0x7fc9380d0830 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc920009870 tx=0x7fc92002b9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.388+0000 7fc93ca00640 1 -- 172.21.15.6:0/1407271614 shutdown_connections 2024-03-06T07:27:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.388+0000 7fc93ca00640 1 --2- 172.21.15.6:0/1407271614 >> v2:172.21.15.178:3300/0 conn(0x7fc9380d0d70 0x7fc9380d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.388+0000 7fc93ca00640 1 --2- 172.21.15.6:0/1407271614 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc93815e2e0 0x7fc9380d0830 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.388+0000 7fc93ca00640 1 -- 172.21.15.6:0/1407271614 >> 172.21.15.6:0/1407271614 conn(0x7fc9380d3f60 msgr2=0x7fc9380d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.389+0000 7fc93ca00640 1 -- 172.21.15.6:0/1407271614 shutdown_connections 2024-03-06T07:27:39.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.389+0000 7fc93ca00640 1 -- 172.21.15.6:0/1407271614 wait complete. 2024-03-06T07:27:39.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.391+0000 7fc93ca00640 1 Processor -- start 2024-03-06T07:27:39.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.392+0000 7fc93ca00640 1 -- start start 2024-03-06T07:27:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.393+0000 7fc93ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9380d0d70 0x7fc9380cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.393+0000 7fc93ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9380d1b30 0x7fc9380cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.393+0000 7fc93ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 0x7fc938153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.394+0000 7fc936800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 0x7fc938153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.394+0000 7fc936800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 0x7fc938153fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42606/0 (socket says 172.21.15.6:42606) 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.394+0000 7fc935400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9380d1b30 0x7fc9380cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7fc935e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9380d0d70 0x7fc9380cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7fc936800640 1 -- 172.21.15.6:0/2149063966 learned_addr learned my addr 172.21.15.6:0/2149063966 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380cfe20 con 0x7fc93815e2e0 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380cfb40 con 0x7fc9380d1b30 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380b50e0 con 0x7fc9380d0d70 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7fc936800640 1 -- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9380d0d70 msgr2=0x7fc9380cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7fc936800640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9380d0d70 0x7fc9380cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7fc936800640 1 -- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9380d1b30 msgr2=0x7fc9380cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7fc936800640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9380d1b30 0x7fc9380cb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7f7463800640 1 Processor -- start 2024-03-06T07:27:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7fc936800640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380ccb20 con 0x7fc93815e2e0 2024-03-06T07:27:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7fc935400640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9380d1b30 0x7fc9380cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7fc935e00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9380d0d70 0x7fc9380cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:39.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.397+0000 7fc936800640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 0x7fc938153fd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc928001560 tx=0x7fc92800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928017070 con 0x7fc93815e2e0 2024-03-06T07:27:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.395+0000 7f0a27e00640 1 Processor -- start 2024-03-06T07:27:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7f0a27e00640 1 -- start start 2024-03-06T07:27:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a20053c20 0x7f0a20007640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f0a27e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200e79b0 con 0x7f0a200589a0 2024-03-06T07:27:39.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f0a27e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200065a0 con 0x7f0a200e75d0 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f0a27e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20007b80 con 0x7f0a20053c20 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.399+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc928003dc0 con 0x7fc93815e2e0 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.396+0000 7f7463800640 1 -- start start 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f7463800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f745c15a210 0x7f745c0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f7463800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f745c06d440 0x7f745c0ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f7463800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745c06d900 con 0x7f745c06e5e0 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f7463800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c06dbe0 con 0x7f745c15a210 2024-03-06T07:27:39.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.398+0000 7f7463800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745c0cddc0 con 0x7f745c06d440 2024-03-06T07:27:39.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.399+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9280052b0 con 0x7fc93815e2e0 2024-03-06T07:27:39.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.399+0000 7f0a26a00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f0a200e75d0 0x7f0a20053510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:37204/0 (socket says 172.21.15.6:37204) 2024-03-06T07:27:39.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.399+0000 7f0a26a00640 1 -- 172.21.15.6:0/1648753153 learned_addr learned my addr 172.21.15.6:0/1648753153 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:39.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.399+0000 7f0a26000640 1 --2- 172.21.15.6:0/1648753153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a20053c20 0x7f0a20007640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.399+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9380cba80 con 0x7fc93815e2e0 2024-03-06T07:27:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f7461800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f745c06d440 0x7f745c0ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f7461800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f745c06d440 0x7f745c0ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:34134/0 (socket says 172.21.15.6:34134) 2024-03-06T07:27:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f7461800640 1 -- 172.21.15.6:0/2093038751 learned_addr learned my addr 172.21.15.6:0/2093038751 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f7460e00640 1 --2- 172.21.15.6:0/2093038751 >> v2:172.21.15.6:3300/0 conn(0x7f745c15a210 0x7f745c0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9380cc040 con 0x7fc93815e2e0 2024-03-06T07:27:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f5c70a00640 1 Processor -- start 2024-03-06T07:27:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.401+0000 7f7461800640 1 -- 172.21.15.6:0/2093038751 >> v2:172.21.15.6:3300/0 conn(0x7f745c15a210 msgr2=0x7f745c0cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.401+0000 7f7461800640 1 --2- 172.21.15.6:0/2093038751 >> v2:172.21.15.6:3300/0 conn(0x7f745c15a210 0x7f745c0cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.401+0000 7f7461800640 1 -- 172.21.15.6:0/2093038751 >> v1:172.21.15.178:6789/0 conn(0x7f745c06e5e0 legacy=0x7f745c06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.401+0000 7f7461800640 1 -- 172.21.15.6:0/2093038751 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0acdb0 con 0x7f745c06d440 2024-03-06T07:27:39.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157589052 0 0) 0x7f0a200065a0 con 0x7f0a200e75d0 2024-03-06T07:27:39.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c003520 con 0x7f0a200e75d0 2024-03-06T07:27:39.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775230982 0 0) 0x7f0a200e79b0 con 0x7f0a200589a0 2024-03-06T07:27:39.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.400+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a200065a0 con 0x7f0a200589a0 2024-03-06T07:27:39.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.401+0000 7f5c70a00640 1 -- start start 2024-03-06T07:27:39.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.402+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93804f3a0 con 0x7fc93815e2e0 2024-03-06T07:27:39.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.402+0000 7f0a26000640 1 -- 172.21.15.6:0/1648753153 >> v1:172.21.15.178:6789/0 conn(0x7f0a200e75d0 legacy=0x7f0a20053510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.403+0000 7f5c70a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c6c15e2e0 0x7f5c6c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.402+0000 7f0a26000640 1 -- 172.21.15.6:0/1648753153 >> v1:172.21.15.6:6789/0 conn(0x7f0a200589a0 legacy=0x7f0a2003bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.402+0000 7f0a26000640 1 -- 172.21.15.6:0/1648753153 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20006880 con 0x7f0a20053c20 2024-03-06T07:27:39.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.403+0000 7f7461800640 1 --2- 172.21.15.6:0/2093038751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06d440 0x7f745c0ce5c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f745c06ec50 tx=0x7f7448030090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c58efb1947231b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.403+0000 7f5c70a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5c6c0d0d70 0x7f5c6c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f0a26000640 1 --2- 172.21.15.6:0/1648753153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a20053c20 0x7f0a20007640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0a180097d0 tx=0x7f0a1802b680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c7345fd963014a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f745b400640 1 -- 172.21.15.6:0/2093038751 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448036070 con 0x7f745c06d440 2024-03-06T07:27:39.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f745b400640 1 -- 172.21.15.6:0/2093038751 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448030e40 con 0x7f745c06d440 2024-03-06T07:27:39.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f745b400640 1 -- 172.21.15.6:0/2093038751 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744803a620 con 0x7f745c06d440 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c70a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c6c0b3940 con 0x7f5c6c0d1b30 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c70a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c6c0d4e70 con 0x7f5c6c0d0d70 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c70a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0b3c40 con 0x7f5c6c15e2e0 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c69e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c6c15e2e0 0x7f5c6c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c6a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5c6c0d1b30 0x7f5c6c0d1fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37576/0 (socket says 172.21.15.6:37576) 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c69e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c6c15e2e0 0x7f5c6c0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47830/0 (socket says 172.21.15.6:47830) 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f5c69e00640 1 -- 172.21.15.6:0/2460050625 learned_addr learned my addr 172.21.15.6:0/2460050625 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.404+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18037070 con 0x7f0a20053c20 2024-03-06T07:27:39.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a18030510 con 0x7f0a20053c20 2024-03-06T07:27:39.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f7463800640 1 -- 172.21.15.6:0/2093038751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06d440 msgr2=0x7f745c0ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f7463800640 1 --2- 172.21.15.6:0/2093038751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06d440 0x7f745c0ce5c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f745c06ec50 tx=0x7f7448030090 comp rx=0 tx=0).stop 2024-03-06T07:27:39.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f0a25600640 1 -- 172.21.15.6:0/1648753153 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2804fb80 con 0x7f0a20053c20 2024-03-06T07:27:39.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736641912 0 0) 0x7f5c6c0b3940 con 0x7f5c6c0d1b30 2024-03-06T07:27:39.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c50003520 con 0x7f5c6c0d1b30 2024-03-06T07:27:39.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f7463800640 1 -- 172.21.15.6:0/2093038751 shutdown_connections 2024-03-06T07:27:39.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f7463800640 1 --2- 172.21.15.6:0/2093038751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06d440 0x7f745c0ce5c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f7463800640 1 --2- 172.21.15.6:0/2093038751 >> v2:172.21.15.6:3300/0 conn(0x7f745c15a210 0x7f745c0cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.405+0000 7f7463800640 1 -- 172.21.15.6:0/2093038751 >> 172.21.15.6:0/2093038751 conn(0x7f745c06afb0 msgr2=0x7f745c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c69400640 1 --2- 172.21.15.6:0/2460050625 >> v2:172.21.15.178:3300/0 conn(0x7f5c6c0d0d70 0x7f5c6c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267708653 0 0) 0x7f5c50003520 con 0x7f5c6c0d1b30 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c6c15e2e0 msgr2=0x7f5c6c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c6c15e2e0 0x7f5c6c0d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 >> v2:172.21.15.178:3300/0 conn(0x7f5c6c0d0d70 msgr2=0x7f5c6c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 --2- 172.21.15.6:0/2460050625 >> v2:172.21.15.178:3300/0 conn(0x7f5c6c0d0d70 0x7f5c6c0d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0cf580 con 0x7f5c6c0d1b30 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5c640019f0 con 0x7f5c6c0d1b30 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 >> v1:172.21.15.6:6789/0 conn(0x7f5c6c0d1b30 legacy=0x7f5c6c0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c50003a20 0x7f5c50003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c50004800 0x7f5c50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0b3c40 con 0x7f5c50003a20 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5000bfe0 con 0x7f5c50004800 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5000c1e0 con 0x7f5c500051c0 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.406+0000 7f0a27e00640 1 -- 172.21.15.6:0/1648753153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a20053c20 msgr2=0x7f0a20007640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f0a27e00640 1 --2- 172.21.15.6:0/1648753153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a20053c20 0x7f0a20007640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0a180097d0 tx=0x7f0a1802b680 comp rx=0 tx=0).stop 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f0a27e00640 1 -- 172.21.15.6:0/1648753153 shutdown_connections 2024-03-06T07:27:39.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f0a27e00640 1 --2- 172.21.15.6:0/1648753153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a20053c20 0x7f0a20007640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f5c6a800640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f5c69e00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c50003a20 0x7f5c50003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f0a27e00640 1 -- 172.21.15.6:0/1648753153 >> 172.21.15.6:0/1648753153 conn(0x7f0a20058ec0 msgr2=0x7f0a200592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f0a27e00640 1 -- 172.21.15.6:0/1648753153 shutdown_connections 2024-03-06T07:27:39.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f0a27e00640 1 -- 172.21.15.6:0/1648753153 wait complete. 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.407+0000 7f5c69400640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c50004800 0x7f5c50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7f7463800640 1 -- 172.21.15.6:0/2093038751 shutdown_connections 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.409+0000 7f7463800640 1 -- 172.21.15.6:0/2093038751 wait complete. 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc92801a070 con 0x7fc93815e2e0 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.409+0000 7fc92e000640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc91406b500 0x7fc91406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.409+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc928089800 con 0x7fc93815e2e0 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.411+0000 7f7463800640 1 Processor -- start 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.410+0000 7f0a27e00640 1 Processor -- start 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.411+0000 7f7463800640 1 -- start start 2024-03-06T07:27:39.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.411+0000 7f0a27e00640 1 -- start start 2024-03-06T07:27:39.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7f5c6a800640 1 -- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c50004800 msgr2=0x7f5c50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7f5c6a800640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c50004800 0x7f5c50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7f5c6a800640 1 -- 172.21.15.6:0/2460050625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c50003a20 msgr2=0x7f5c50003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:39.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7f5c6a800640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c50003a20 0x7f5c50003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.408+0000 7f5c6a800640 1 -- 172.21.15.6:0/2460050625 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0cf580 con 0x7f5c500051c0 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f7463800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c06d440 0x7f745c1fad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f7463800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06e5e0 0x7f745c0b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f5c69e00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c50003a20 0x7f5c50003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f7463800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 0x7f745c149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f7463800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c06bc20 con 0x7f745c15a210 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f7463800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c06baa0 con 0x7f745c06e5e0 2024-03-06T07:27:39.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f7463800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c15a5f0 con 0x7f745c06d440 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7f5c6a800640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c5000b9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c520 tx=0x7f5c50005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=426a58141dd0fc81 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.412+0000 7fc935e00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc91406b500 0x7fc91406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 0x7f745c149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 0x7f745c149740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42642/0 (socket says 172.21.15.6:42642) 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f745be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06e5e0 0x7f745c0b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 -- 172.21.15.6:0/1221027916 learned_addr learned my addr 172.21.15.6:0/1221027916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7460e00640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c06d440 0x7f745c1fad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 -- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c06d440 msgr2=0x7f745c1fad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c06d440 0x7f745c1fad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 -- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06e5e0 msgr2=0x7f745c0b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06e5e0 0x7f745c0b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.413+0000 7f7461800640 1 -- 172.21.15.6:0/1221027916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c0b0e00 con 0x7f745c15a210 2024-03-06T07:27:39.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.414+0000 7fc935e00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc91406b500 0x7fc91406d9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc920036000 tx=0x7fc92003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.414+0000 7f7461800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 0x7f745c149740 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7448008960 tx=0x7f7448004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.414+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 0x7f0a2004ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.414+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 0x7f0a20050120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.414+0000 7f0a26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 0x7f0a2004ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.414+0000 7f0a26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 0x7f0a2004ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42656/0 (socket says 172.21.15.6:42656) 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a200e75d0 0x7f0a2017d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400c120 con 0x7f5c500051c0 2024-03-06T07:27:39.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a26000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 0x7f0a20050120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a26000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 0x7f0a20050120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:34172/0 (socket says 172.21.15.6:34172) 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a26000640 1 -- 172.21.15.6:0/222059827 learned_addr learned my addr 172.21.15.6:0/222059827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20007120 con 0x7f0a20053c20 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20006e40 con 0x7f0a200589a0 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a200061c0 con 0x7f0a200e75d0 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f0a27400640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a200e75d0 0x7f0a2017d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64030070 con 0x7f5c500051c0 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.415+0000 7f5c68a00640 1 -- 172.21.15.6:0/2460050625 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64005dd0 con 0x7f5c500051c0 2024-03-06T07:27:39.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f5c70a00640 1 -- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 msgr2=0x7f5c5000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f5c70a00640 1 --2- 172.21.15.6:0/2460050625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c5000b9c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c520 tx=0x7f5c50005830 comp rx=0 tx=0).stop 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f5c6a800640 1 -- 172.21.15.6:0/2460050625 reap_dead start 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f5c70a00640 1 -- 172.21.15.6:0/2460050625 shutdown_connections 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f0a26000640 1 -- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a200e75d0 msgr2=0x7f0a2017d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f5c70a00640 1 -- 172.21.15.6:0/2460050625 >> 172.21.15.6:0/2460050625 conn(0x7f5c6c0d3f60 msgr2=0x7f5c6c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f0a26000640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a200e75d0 0x7f0a2017d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f0a26000640 1 -- 172.21.15.6:0/222059827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 msgr2=0x7f0a2004ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f0a26000640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 0x7f0a2004ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f0a26000640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20053170 con 0x7f0a200589a0 2024-03-06T07:27:39.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.416+0000 7f0a26a00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 0x7f0a2004ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:39.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.417+0000 7f5c70a00640 1 -- 172.21.15.6:0/2460050625 shutdown_connections 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.417+0000 7f0a26000640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 0x7f0a20050120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009720 tx=0x7f0a180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.418+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18042070 con 0x7f0a200589a0 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.418+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a20053450 con 0x7f0a200589a0 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.418+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a20180c80 con 0x7f0a200589a0 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.418+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1803e870 con 0x7f0a200589a0 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.418+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18037040 con 0x7f0a200589a0 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.417+0000 7f5c70a00640 1 -- 172.21.15.6:0/2460050625 wait complete. 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.420+0000 7f5c70a00640 1 Processor -- start 2024-03-06T07:27:39.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.420+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a00005290 con 0x7f0a200589a0 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.420+0000 7f5c70a00640 1 -- start start 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448043480 con 0x7f745c15a210 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745c09ac60 con 0x7f745c15a210 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745c14a4e0 con 0x7f745c15a210 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448043640 con 0x7f745c15a210 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448043a30 con 0x7f745c15a210 2024-03-06T07:27:39.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a18045070 con 0x7f0a200589a0 2024-03-06T07:27:39.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.422+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc928058020 con 0x7fc93815e2e0 2024-03-06T07:27:39.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.423+0000 7f5c70a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c6c160d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.424+0000 7f5c70a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c6c15c080 0x7f5c6c15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.424+0000 7f0a1ea00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a0806b500 0x7f0a0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.424+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a180ac5f0 con 0x7f0a200589a0 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.424+0000 7f5c69e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c6c160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.424+0000 7f5c69e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c6c160d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47876/0 (socket says 172.21.15.6:47876) 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.424+0000 7f5c69e00640 1 -- 172.21.15.6:0/3356870519 learned_addr learned my addr 172.21.15.6:0/3356870519 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c6c0c9e90 0x7f5c6c0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0cfe20 con 0x7f5c6c15c080 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0cfb40 con 0x7f5c6c0c9e90 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f5c69400640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c6c15c080 0x7f5c6c15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f5c6a800640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c6c0c9e90 0x7f5c6c0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6c0b50e0 con 0x7f5c500051c0 2024-03-06T07:27:39.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.425+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434005290 con 0x7f745c15a210 2024-03-06T07:27:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.426+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f744804a020 con 0x7f745c15a210 2024-03-06T07:27:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.426+0000 7f5c69400640 1 -- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 msgr2=0x7f5c6c160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.427+0000 7f5c69400640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c6c160d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.427+0000 7f5c69400640 1 -- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c6c0c9e90 msgr2=0x7f5c6c0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.427+0000 7f5c69400640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c6c0c9e90 0x7f5c6c0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.427+0000 7f5c69400640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c15ca40 con 0x7f5c6c15c080 2024-03-06T07:27:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.427+0000 7f7458c00640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743c06b5a0 0x7f743c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.428+0000 7f5c69400640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c6c15c080 0x7f5c6c15c500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800b7c0 tx=0x7f5c5800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.427+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74480abb00 con 0x7f745c15a210 2024-03-06T07:27:39.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.428+0000 7f0a26a00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a0806b500 0x7f0a0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.428+0000 7f7460e00640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743c06b5a0 0x7f743c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.429+0000 7f0a26a00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a0806b500 0x7f0a0806d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a14007fd0 tx=0x7f0a14007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.430+0000 7f7460e00640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743c06b5a0 0x7f743c06da60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f745c0b4510 tx=0x7f745400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.432+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58018070 con 0x7f5c6c15c080 2024-03-06T07:27:39.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.432+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7448012070 con 0x7f745c15a210 2024-03-06T07:27:39.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.432+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c6c0ca850 con 0x7f5c6c15c080 2024-03-06T07:27:39.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.432+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c6c0cb4b0 con 0x7f5c6c15c080 2024-03-06T07:27:39.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.432+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c58010830 con 0x7f5c6c15c080 2024-03-06T07:27:39.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.432+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c58010c00 con 0x7f5c6c15c080 2024-03-06T07:27:39.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.436+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6c0b4b20 con 0x7f5c6c15c080 2024-03-06T07:27:39.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.436+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c5801b070 con 0x7f5c6c15c080 2024-03-06T07:27:39.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.437+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a1807ae90 con 0x7f0a200589a0 2024-03-06T07:27:39.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.438+0000 7f5c62000640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c4c06b5a0 0x7f5c4c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:39.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.438+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c5808a5a0 con 0x7f5c6c15c080 2024-03-06T07:27:39.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.444+0000 7f5c69e00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c4c06b5a0 0x7f5c4c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.449+0000 7f5c69e00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c4c06b5a0 0x7f5c4c06da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009400 tx=0x7f5c5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:39.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.450+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c6c0b4b20 con 0x7f5c6c15c080 2024-03-06T07:27:39.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.743+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fabf8005850 con 0x7fac18005ee0 2024-03-06T07:27:39.746 INFO:teuthology.orchestra.run.smithi006.stdout:64424509445 2024-03-06T07:27:39.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.744+0000 7fac16000640 1 -- 172.21.15.6:0/3588658865 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fac1005c930 con 0x7fac18005ee0 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.747+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fac0006b6d0 msgr2=0x7fac0006db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.747+0000 7fac0b400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fac0006b6d0 0x7fac0006db90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac2007abf0 tx=0x7fac200ac000 comp rx=0 tx=0).stop 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.748+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 msgr2=0x7fac180d7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.748+0000 7fac0b400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 0x7fac180d7230 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac1000e900 tx=0x7fac1000edc0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.748+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 shutdown_connections 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.748+0000 7fac0b400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fac0006b6d0 0x7fac0006db90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.749+0000 7fac0b400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fac180e6ca0 0x7fac1804c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.749+0000 7fac0b400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fac180e3160 0x7fac1804bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.749+0000 7fac0b400640 1 --2- 172.21.15.6:0/3588658865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fac18005ee0 0x7fac180d7230 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.749+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 >> 172.21.15.6:0/3588658865 conn(0x7fac18009fb0 msgr2=0x7fac180e5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.749+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 shutdown_connections 2024-03-06T07:27:39.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.749+0000 7fac0b400640 1 -- 172.21.15.6:0/3588658865 wait complete. 2024-03-06T07:27:39.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.2 2024-03-06T07:27:39.777 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.778+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f19b00b3e90 con 0x7f19b00b2300 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.778+0000 7f19a7400640 1 -- 172.21.15.6:0/3022102950 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f199c05c9d0 con 0x7f19b00b2300 2024-03-06T07:27:39.780 INFO:teuthology.orchestra.run.smithi006.stdout:60129542151 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f199006b520 msgr2=0x7f199006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f199006b520 0x7f199006d9e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f19a8036000 tx=0x7f19a8030000 comp rx=0 tx=0).stop 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 msgr2=0x7f19b013ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b013ef70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f199c00c7f0 tx=0x7f199c00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 shutdown_connections 2024-03-06T07:27:39.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f199006b520 0x7f199006d9e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19b00b2ae0 0x7f19b00abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19b00b2300 0x7f19b013ef70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 --2- 172.21.15.6:0/3022102950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19b006ce20 0x7f19b013ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 >> 172.21.15.6:0/3022102950 conn(0x7f19b00b17e0 msgr2=0x7f19b015a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 shutdown_connections 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.782+0000 7f19b5200640 1 -- 172.21.15.6:0/3022102950 wait complete. 2024-03-06T07:27:39.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.783+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc7fc04f3a0 con 0x7fc7fc0712c0 2024-03-06T07:27:39.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.784+0000 7fc7f3e00640 1 -- 172.21.15.6:0/3334816686 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc7e407aa80 con 0x7fc7fc0712c0 2024-03-06T07:27:39.789 INFO:teuthology.orchestra.run.smithi006.stdout:60129542151 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7d806b6d0 msgr2=0x7fc7d806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7d806b6d0 0x7fc7d806db90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc0ae960 tx=0x7fc7ec007d30 comp rx=0 tx=0).stop 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 msgr2=0x7fc7fc150240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 0x7fc7fc150240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4009960 tx=0x7fc7e4004730 comp rx=0 tx=0).stop 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 shutdown_connections 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7d806b6d0 0x7fc7d806db90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7fc14ea70 0x7fc7fc1511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7fc072da0 0x7fc7fc150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 --2- 172.21.15.6:0/3334816686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7fc0712c0 0x7fc7fc150240 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 >> 172.21.15.6:0/3334816686 conn(0x7fc7fc06afb0 msgr2=0x7fc7fc1656b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.788+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 shutdown_connections 2024-03-06T07:27:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.789+0000 7fc7f1600640 1 -- 172.21.15.6:0/3334816686 wait complete. 2024-03-06T07:27:39.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.789+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff78814fdd0 con 0x7ff76c0051c0 2024-03-06T07:27:39.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.790+0000 7ff77f400640 1 -- 172.21.15.6:0/2134698876 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff77805c140 con 0x7ff76c0051c0 2024-03-06T07:27:39.793 INFO:teuthology.orchestra.run.smithi006.stdout:60129542150 2024-03-06T07:27:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff76806b500 msgr2=0x7ff76806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff76806b500 0x7ff76806d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff788148910 tx=0x7ff774009040 comp rx=0 tx=0).stop 2024-03-06T07:27:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 msgr2=0x7ff788145df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 0x7ff788145df0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff77800f8d0 tx=0x7ff77800fd90 comp rx=0 tx=0).stop 2024-03-06T07:27:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 shutdown_connections 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff76806b500 0x7ff76806d9c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7881489b0 0x7ff788148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7880cd260 0x7ff7880c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 --2- 172.21.15.6:0/2134698876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0051c0 0x7ff788145df0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 >> 172.21.15.6:0/2134698876 conn(0x7ff7880d3f60 msgr2=0x7ff78806c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.794+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 shutdown_connections 2024-03-06T07:27:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.795+0000 7ff78e400640 1 -- 172.21.15.6:0/2134698876 wait complete. 2024-03-06T07:27:39.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-03-06T07:27:39.814 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.3 2024-03-06T07:27:39.814 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.4 2024-03-06T07:27:39.820 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.825+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7434005580 con 0x7f745c15a210 2024-03-06T07:27:39.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.826+0000 7f7458c00640 1 -- 172.21.15.6:0/1221027916 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f744807a960 con 0x7f745c15a210 2024-03-06T07:27:39.828 INFO:teuthology.orchestra.run.smithi006.stdout:60129542151 2024-03-06T07:27:39.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743c06b5a0 msgr2=0x7f743c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743c06b5a0 0x7f743c06da60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f745c0b4510 tx=0x7f745400b040 comp rx=0 tx=0).stop 2024-03-06T07:27:39.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 msgr2=0x7f745c149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 0x7f745c149740 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7448008960 tx=0x7f7448004730 comp rx=0 tx=0).stop 2024-03-06T07:27:39.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 shutdown_connections 2024-03-06T07:27:39.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743c06b5a0 0x7f743c06da60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c06e5e0 0x7f745c0b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c15a210 0x7f745c149740 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 --2- 172.21.15.6:0/1221027916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c06d440 0x7f745c1fad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 >> 172.21.15.6:0/1221027916 conn(0x7f745c06afb0 msgr2=0x7f745c158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 shutdown_connections 2024-03-06T07:27:39.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.828+0000 7f7463800640 1 -- 172.21.15.6:0/1221027916 wait complete. 2024-03-06T07:27:39.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-03-06T07:27:39.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.844+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5c6c0d02e0 con 0x7f5c6c15c080 2024-03-06T07:27:39.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.845+0000 7f5c62000640 1 -- 172.21.15.6:0/3356870519 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5c58059400 con 0x7f5c6c15c080 2024-03-06T07:27:39.847 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.847 INFO:teuthology.orchestra.run.smithi006.stdout:64424509446 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c4c06b5a0 msgr2=0x7f5c4c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c4c06b5a0 0x7f5c4c06da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009400 tx=0x7f5c5c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c6c15c080 msgr2=0x7f5c6c15c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c6c15c080 0x7f5c6c15c500 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800b7c0 tx=0x7f5c5800bc80 comp rx=0 tx=0).stop 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 shutdown_connections 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c4c06b5a0 0x7f5c4c06da60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c6c0c9e90 0x7f5c6c0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c6c15c080 0x7f5c6c15c500 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 --2- 172.21.15.6:0/3356870519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c500051c0 0x7f5c6c160d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 >> 172.21.15.6:0/3356870519 conn(0x7f5c6c0d3f60 msgr2=0x7f5c6c06d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 shutdown_connections 2024-03-06T07:27:39.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.847+0000 7f5c70a00640 1 -- 172.21.15.6:0/3356870519 wait complete. 2024-03-06T07:27:39.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.1 2024-03-06T07:27:39.865 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.867+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc9380d00d0 con 0x7fc93815e2e0 2024-03-06T07:27:39.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.867+0000 7fc92e000640 1 -- 172.21.15.6:0/2149063966 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc92805c0c0 con 0x7fc93815e2e0 2024-03-06T07:27:39.869 INFO:teuthology.orchestra.run.smithi006.stdout:60129542150 2024-03-06T07:27:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc91406b500 msgr2=0x7fc91406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc91406b500 0x7fc91406d9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc920036000 tx=0x7fc92003b040 comp rx=0 tx=0).stop 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 msgr2=0x7fc938153fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 0x7fc938153fd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc928001560 tx=0x7fc92800cc70 comp rx=0 tx=0).stop 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 shutdown_connections 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc91406b500 0x7fc91406d9c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc9380d1b30 0x7fc9380cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc93815e2e0 0x7fc938153fd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 --2- 172.21.15.6:0/2149063966 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc9380d0d70 0x7fc9380cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 >> 172.21.15.6:0/2149063966 conn(0x7fc9380d3f60 msgr2=0x7fc9380cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 shutdown_connections 2024-03-06T07:27:39.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.870+0000 7fc93ca00640 1 -- 172.21.15.6:0/2149063966 wait complete. 2024-03-06T07:27:39.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.882+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0a00005850 con 0x7f0a200589a0 2024-03-06T07:27:39.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.882+0000 7f0a1ea00640 1 -- 172.21.15.6:0/222059827 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a1807ef30 con 0x7f0a200589a0 2024-03-06T07:27:39.884 INFO:teuthology.orchestra.run.smithi006.stdout:60129542150 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a0806b500 msgr2=0x7f0a0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a0806b500 0x7f0a0806d9c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a14007fd0 tx=0x7f0a14007de0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 msgr2=0x7f0a20050120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 0x7f0a20050120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009720 tx=0x7f0a180046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 shutdown_connections 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a0806b500 0x7f0a0806d9c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a200589a0 0x7f0a20050120 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.885+0000 7f0a27e00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a200e75d0 0x7f0a2017d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.886+0000 7f0a27e00640 1 --2- 172.21.15.6:0/222059827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a20053c20 0x7f0a2004ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.886+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 >> 172.21.15.6:0/222059827 conn(0x7f0a20058ec0 msgr2=0x7f0a200e6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.886+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 shutdown_connections 2024-03-06T07:27:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:39.886+0000 7f0a27e00640 1 -- 172.21.15.6:0/222059827 wait complete. 2024-03-06T07:27:39.893 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2024-03-06T07:27:39.893 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2024-03-06T07:27:39.904 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:39.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:27:39.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:40.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.165+0000 7f3f61800640 1 Processor -- start 2024-03-06T07:27:40.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.165+0000 7f3f61800640 1 -- start start 2024-03-06T07:27:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f61800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f61800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f5c070f10 0x7f3f5c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f61800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5c0714d0 con 0x7f3f5c0cce90 2024-03-06T07:27:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f61800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5c0cd300 con 0x7f3f5c070f10 2024-03-06T07:27:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f61800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0717b0 con 0x7f3f5c145300 2024-03-06T07:27:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f5b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f5c070f10 0x7f3f5c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f5b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f5c070f10 0x7f3f5c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42668/0 (socket says 172.21.15.6:42668) 2024-03-06T07:27:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f60e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f5c0cce90 0x7f3f5c0cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37584/0 (socket says 172.21.15.6:37584) 2024-03-06T07:27:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f5b400640 1 -- 172.21.15.6:0/581373459 learned_addr learned my addr 172.21.15.6:0/581373459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.167+0000 7f3f5be00640 1 --2- 172.21.15.6:0/581373459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.168+0000 7f3f5aa00640 1 -- 172.21.15.6:0/581373459 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760502301 0 0) 0x7f3f5c0714d0 con 0x7f3f5c0cce90 2024-03-06T07:27:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.168+0000 7f3f5b400640 1 -- 172.21.15.6:0/581373459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 msgr2=0x7f3f5c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.168+0000 7f3f5b400640 1 --2- 172.21.15.6:0/581373459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.168+0000 7f3f5b400640 1 -- 172.21.15.6:0/581373459 >> v1:172.21.15.6:6789/0 conn(0x7f3f5c0cce90 legacy=0x7f3f5c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.168+0000 7f3f5b400640 1 -- 172.21.15.6:0/581373459 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c071a90 con 0x7f3f5c070f10 2024-03-06T07:27:40.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.169+0000 7f3f5b400640 1 --2- 172.21.15.6:0/581373459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c070f10 0x7f3f5c0cc950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009870 tx=0x7f3f4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8caa1806b378a135 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.169+0000 7f3f5aa00640 1 -- 172.21.15.6:0/581373459 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034070 con 0x7f3f5c070f10 2024-03-06T07:27:40.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.169+0000 7f3f5aa00640 1 -- 172.21.15.6:0/581373459 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48030840 con 0x7f3f5c070f10 2024-03-06T07:27:40.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.169+0000 7f3f5aa00640 1 -- 172.21.15.6:0/581373459 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48030c10 con 0x7f3f5c070f10 2024-03-06T07:27:40.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 -- 172.21.15.6:0/581373459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c070f10 msgr2=0x7f3f5c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 --2- 172.21.15.6:0/581373459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c070f10 0x7f3f5c0cc950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009870 tx=0x7f3f4802b980 comp rx=0 tx=0).stop 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 -- 172.21.15.6:0/581373459 shutdown_connections 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 --2- 172.21.15.6:0/581373459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c070f10 0x7f3f5c0cc950 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 --2- 172.21.15.6:0/581373459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 -- 172.21.15.6:0/581373459 >> 172.21.15.6:0/581373459 conn(0x7f3f5c06b3d0 msgr2=0x7f3f5c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 -- 172.21.15.6:0/581373459 shutdown_connections 2024-03-06T07:27:40.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.170+0000 7f3f61800640 1 -- 172.21.15.6:0/581373459 wait complete. 2024-03-06T07:27:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.171+0000 7f3f61800640 1 Processor -- start 2024-03-06T07:27:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 -- start start 2024-03-06T07:27:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c070f10 0x7f3f5c1f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 0x7f3f5c0c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c1456e0 con 0x7f3f5c0cce90 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0b4ee0 con 0x7f3f5c070f10 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.172+0000 7f3f61800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c15d580 con 0x7f3f5c145300 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 0x7f3f5c0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 0x7f3f5c0c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42674/0 (socket says 172.21.15.6:42674) 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 -- 172.21.15.6:0/1849588791 learned_addr learned my addr 172.21.15.6:0/1849588791 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5be00640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c070f10 0x7f3f5c1f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f60e00640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 -- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 msgr2=0x7f3f5c0b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 -- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c070f10 msgr2=0x7f3f5c1f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c070f10 0x7f3f5c1f46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f5b400640 1 -- 172.21.15.6:0/1849588791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c141290 con 0x7f3f5c0cce90 2024-03-06T07:27:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.173+0000 7f3f60e00640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.174+0000 7f3f5b400640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 0x7f3f5c0c6a10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3f4802b9c0 tx=0x7f3f48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.174+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48048070 con 0x7f3f5c0cce90 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.174+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48030640 con 0x7f3f5c0cce90 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.174+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034040 con 0x7f3f5c0cce90 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.174+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c141570 con 0x7f3f5c0cce90 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.174+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c141b30 con 0x7f3f5c0cce90 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.175+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c04f330 con 0x7f3f5c0cce90 2024-03-06T07:27:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.176+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f4804b070 con 0x7f3f5c0cce90 2024-03-06T07:27:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.177+0000 7f3f53e00640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3c06b4d0 0x7f3f3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.177+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f480ac0a0 con 0x7f3f5c0cce90 2024-03-06T07:27:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.178+0000 7f3f5be00640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3c06b4d0 0x7f3f3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.179+0000 7f3f5be00640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3c06b4d0 0x7f3f3c06d990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c142450 tx=0x7f3f4c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.181+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f4807af10 con 0x7f3f5c0cce90 2024-03-06T07:27:40.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.371+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f5c074ab0 con 0x7f3f3c06b4d0 2024-03-06T07:27:40.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.392+0000 7f3f53e00640 1 -- 172.21.15.6:0/1849588791 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7f3f5c074ab0 con 0x7f3f3c06b4d0 2024-03-06T07:27:40.397 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:40.399 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:27:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.396+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3c06b4d0 msgr2=0x7f3f3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.396+0000 7f3f61800640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3c06b4d0 0x7f3f3c06d990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c142450 tx=0x7f3f4c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.396+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 msgr2=0x7f3f5c0c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.396+0000 7f3f61800640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 0x7f3f5c0c6a10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3f4802b9c0 tx=0x7f3f48004730 comp rx=0 tx=0).stop 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 shutdown_connections 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3c06b4d0 0x7f3f3c06d990 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c145300 0x7f3f5c0b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0cce90 0x7f3f5c0c6a10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 --2- 172.21.15.6:0/1849588791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c070f10 0x7f3f5c1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 >> 172.21.15.6:0/1849588791 conn(0x7f3f5c06b3d0 msgr2=0x7f3f5c0aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 shutdown_connections 2024-03-06T07:27:40.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.397+0000 7f3f61800640 1 -- 172.21.15.6:0/1849588791 wait complete. 2024-03-06T07:27:40.417 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-06T07:27:39.440318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235772,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738948,"statfs":{"total":773094113280,"available":772852682752,"internally_reserved":0,"allocated":3022848,"data_stored":1810564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.230310"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699359+0000","last_change":"2024-03-06T07:27:30.239373+0000","last_active":"2024-03-06T07:27:31.699359+0000","last_peered":"2024-03-06T07:27:31.699359+0000","last_clean":"2024-03-06T07:27:31.699359+0000","last_became_active":"2024-03-06T07:27:30.239046+0000","last_became_peered":"2024-03-06T07:27:30.239046+0000","last_unstale":"2024-03-06T07:27:31.699359+0000","last_undegraded":"2024-03-06T07:27:31.699359+0000","last_fullsized":"2024-03-06T07:27:31.699359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:38.519257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239467+0000","last_change":"2024-03-06T07:27:30.234823+0000","last_active":"2024-03-06T07:27:31.239467+0000","last_peered":"2024-03-06T07:27:31.239467+0000","last_clean":"2024-03-06T07:27:31.239467+0000","last_became_active":"2024-03-06T07:27:30.234521+0000","last_became_peered":"2024-03-06T07:27:30.234521+0000","last_unstale":"2024-03-06T07:27:31.239467+0000","last_undegraded":"2024-03-06T07:27:31.239467+0000","last_fullsized":"2024-03-06T07:27:31.239467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:17.318504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634508+0000","last_change":"2024-03-06T07:27:30.244772+0000","last_active":"2024-03-06T07:27:31.634508+0000","last_peered":"2024-03-06T07:27:31.634508+0000","last_clean":"2024-03-06T07:27:31.634508+0000","last_became_active":"2024-03-06T07:27:30.244401+0000","last_became_peered":"2024-03-06T07:27:30.244401+0000","last_unstale":"2024-03-06T07:27:31.634508+0000","last_undegraded":"2024-03-06T07:27:31.634508+0000","last_fullsized":"2024-03-06T07:27:31.634508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699779+0000","last_change":"2024-03-06T07:27:30.260180+0000","last_active":"2024-03-06T07:27:31.699779+0000","last_peered":"2024-03-06T07:27:31.699779+0000","last_clean":"2024-03-06T07:27:31.699779+0000","last_became_active":"2024-03-06T07:27:30.259183+0000","last_became_peered":"2024-03-06T07:27:30.259183+0000","last_unstale":"2024-03-06T07:27:31.699779+0000","last_undegraded":"2024-03-06T07:27:31.699779+0000","last_fullsized":"2024-03-06T07:27:31.699779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:46.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700449+0000","last_change":"2024-03-06T07:27:30.254425+0000","last_active":"2024-03-06T07:27:31.700449+0000","last_peered":"2024-03-06T07:27:31.700449+0000","last_clean":"2024-03-06T07:27:31.700449+0000","last_became_active":"2024-03-06T07:27:30.253238+0000","last_became_peered":"2024-03-06T07:27:30.253238+0000","last_unstale":"2024-03-06T07:27:31.700449+0000","last_undegraded":"2024-03-06T07:27:31.700449+0000","last_fullsized":"2024-03-06T07:27:31.700449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:45.798875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696752+0000","last_change":"2024-03-06T07:27:30.259178+0000","last_active":"2024-03-06T07:27:31.696752+0000","last_peered":"2024-03-06T07:27:31.696752+0000","last_clean":"2024-03-06T07:27:31.696752+0000","last_became_active":"2024-03-06T07:27:30.258928+0000","last_became_peered":"2024-03-06T07:27:30.258928+0000","last_unstale":"2024-03-06T07:27:31.696752+0000","last_undegraded":"2024-03-06T07:27:31.696752+0000","last_fullsized":"2024-03-06T07:27:31.696752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.246068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698449+0000","last_change":"2024-03-06T07:27:30.244331+0000","last_active":"2024-03-06T07:27:31.698449+0000","last_peered":"2024-03-06T07:27:31.698449+0000","last_clean":"2024-03-06T07:27:31.698449+0000","last_became_active":"2024-03-06T07:27:30.244078+0000","last_became_peered":"2024-03-06T07:27:30.244078+0000","last_unstale":"2024-03-06T07:27:31.698449+0000","last_undegraded":"2024-03-06T07:27:31.698449+0000","last_fullsized":"2024-03-06T07:27:31.698449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:18.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241726+0000","last_change":"2024-03-06T07:27:30.256607+0000","last_active":"2024-03-06T07:27:31.241726+0000","last_peered":"2024-03-06T07:27:31.241726+0000","last_clean":"2024-03-06T07:27:31.241726+0000","last_became_active":"2024-03-06T07:27:30.256321+0000","last_became_peered":"2024-03-06T07:27:30.256321+0000","last_unstale":"2024-03-06T07:27:31.241726+0000","last_undegraded":"2024-03-06T07:27:31.241726+0000","last_fullsized":"2024-03-06T07:27:31.241726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699557+0000","last_change":"2024-03-06T07:27:30.260973+0000","last_active":"2024-03-06T07:27:31.699557+0000","last_peered":"2024-03-06T07:27:31.699557+0000","last_clean":"2024-03-06T07:27:31.699557+0000","last_became_active":"2024-03-06T07:27:30.260401+0000","last_became_peered":"2024-03-06T07:27:30.260401+0000","last_unstale":"2024-03-06T07:27:31.699557+0000","last_undegraded":"2024-03-06T07:27:31.699557+0000","last_fullsized":"2024-03-06T07:27:31.699557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:14.052739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243083+0000","last_change":"2024-03-06T07:27:30.247832+0000","last_active":"2024-03-06T07:27:31.243083+0000","last_peered":"2024-03-06T07:27:31.243083+0000","last_clean":"2024-03-06T07:27:31.243083+0000","last_became_active":"2024-03-06T07:27:30.247503+0000","last_became_peered":"2024-03-06T07:27:30.247503+0000","last_unstale":"2024-03-06T07:27:31.243083+0000","last_undegraded":"2024-03-06T07:27:31.243083+0000","last_fullsized":"2024-03-06T07:27:31.243083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242309+0000","last_change":"2024-03-06T07:27:30.256946+0000","last_active":"2024-03-06T07:27:31.242309+0000","last_peered":"2024-03-06T07:27:31.242309+0000","last_clean":"2024-03-06T07:27:31.242309+0000","last_became_active":"2024-03-06T07:27:30.256643+0000","last_became_peered":"2024-03-06T07:27:30.256643+0000","last_unstale":"2024-03-06T07:27:31.242309+0000","last_undegraded":"2024-03-06T07:27:31.242309+0000","last_fullsized":"2024-03-06T07:27:31.242309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700787+0000","last_change":"2024-03-06T07:27:30.248076+0000","last_active":"2024-03-06T07:27:31.700787+0000","last_peered":"2024-03-06T07:27:31.700787+0000","last_clean":"2024-03-06T07:27:31.700787+0000","last_became_active":"2024-03-06T07:27:30.247733+0000","last_became_peered":"2024-03-06T07:27:30.247733+0000","last_unstale":"2024-03-06T07:27:31.700787+0000","last_undegraded":"2024-03-06T07:27:31.700787+0000","last_fullsized":"2024-03-06T07:27:31.700787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:00.010236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243474+0000","last_change":"2024-03-06T07:27:30.238620+0000","last_active":"2024-03-06T07:27:31.243474+0000","last_peered":"2024-03-06T07:27:31.243474+0000","last_clean":"2024-03-06T07:27:31.243474+0000","last_became_active":"2024-03-06T07:27:30.238357+0000","last_became_peered":"2024-03-06T07:27:30.238357+0000","last_unstale":"2024-03-06T07:27:31.243474+0000","last_undegraded":"2024-03-06T07:27:31.243474+0000","last_fullsized":"2024-03-06T07:27:31.243474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634214+0000","last_change":"2024-03-06T07:27:30.242346+0000","last_active":"2024-03-06T07:27:31.634214+0000","last_peered":"2024-03-06T07:27:31.634214+0000","last_clean":"2024-03-06T07:27:31.634214+0000","last_became_active":"2024-03-06T07:27:30.242081+0000","last_became_peered":"2024-03-06T07:27:30.242081+0000","last_unstale":"2024-03-06T07:27:31.634214+0000","last_undegraded":"2024-03-06T07:27:31.634214+0000","last_fullsized":"2024-03-06T07:27:31.634214+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:48.495220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239135+0000","last_change":"2024-03-06T07:27:30.252254+0000","last_active":"2024-03-06T07:27:31.239135+0000","last_peered":"2024-03-06T07:27:31.239135+0000","last_clean":"2024-03-06T07:27:31.239135+0000","last_became_active":"2024-03-06T07:27:30.251915+0000","last_became_peered":"2024-03-06T07:27:30.251915+0000","last_unstale":"2024-03-06T07:27:31.239135+0000","last_undegraded":"2024-03-06T07:27:31.239135+0000","last_fullsized":"2024-03-06T07:27:31.239135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:42.209474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696450+0000","last_change":"2024-03-06T07:27:30.266564+0000","last_active":"2024-03-06T07:27:31.696450+0000","last_peered":"2024-03-06T07:27:31.696450+0000","last_clean":"2024-03-06T07:27:31.696450+0000","last_became_active":"2024-03-06T07:27:30.265165+0000","last_became_peered":"2024-03-06T07:27:30.265165+0000","last_unstale":"2024-03-06T07:27:31.696450+0000","last_undegraded":"2024-03-06T07:27:31.696450+0000","last_fullsized":"2024-03-06T07:27:31.696450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697066+0000","last_change":"2024-03-06T07:27:30.266828+0000","last_active":"2024-03-06T07:27:31.697066+0000","last_peered":"2024-03-06T07:27:31.697066+0000","last_clean":"2024-03-06T07:27:31.697066+0000","last_became_active":"2024-03-06T07:27:30.265102+0000","last_became_peered":"2024-03-06T07:27:30.265102+0000","last_unstale":"2024-03-06T07:27:31.697066+0000","last_undegraded":"2024-03-06T07:27:31.697066+0000","last_fullsized":"2024-03-06T07:27:31.697066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:58.613904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695842+0000","last_change":"2024-03-06T07:27:30.239978+0000","last_active":"2024-03-06T07:27:31.695842+0000","last_peered":"2024-03-06T07:27:31.695842+0000","last_clean":"2024-03-06T07:27:31.695842+0000","last_became_active":"2024-03-06T07:27:30.239682+0000","last_became_peered":"2024-03-06T07:27:30.239682+0000","last_unstale":"2024-03-06T07:27:31.695842+0000","last_undegraded":"2024-03-06T07:27:31.695842+0000","last_fullsized":"2024-03-06T07:27:31.695842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:46.446447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698649+0000","last_change":"2024-03-06T07:27:30.245372+0000","last_active":"2024-03-06T07:27:31.698649+0000","last_peered":"2024-03-06T07:27:31.698649+0000","last_clean":"2024-03-06T07:27:31.698649+0000","last_became_active":"2024-03-06T07:27:30.245096+0000","last_became_peered":"2024-03-06T07:27:30.245096+0000","last_unstale":"2024-03-06T07:27:31.698649+0000","last_undegraded":"2024-03-06T07:27:31.698649+0000","last_fullsized":"2024-03-06T07:27:31.698649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:08.875835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242869+0000","last_change":"2024-03-06T07:27:30.244785+0000","last_active":"2024-03-06T07:27:31.242869+0000","last_peered":"2024-03-06T07:27:31.242869+0000","last_clean":"2024-03-06T07:27:31.242869+0000","last_became_active":"2024-03-06T07:27:30.244505+0000","last_became_peered":"2024-03-06T07:27:30.244505+0000","last_unstale":"2024-03-06T07:27:31.242869+0000","last_undegraded":"2024-03-06T07:27:31.242869+0000","last_fullsized":"2024-03-06T07:27:31.242869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634309+0000","last_change":"2024-03-06T07:27:30.253741+0000","last_active":"2024-03-06T07:27:31.634309+0000","last_peered":"2024-03-06T07:27:31.634309+0000","last_clean":"2024-03-06T07:27:31.634309+0000","last_became_active":"2024-03-06T07:27:30.251912+0000","last_became_peered":"2024-03-06T07:27:30.251912+0000","last_unstale":"2024-03-06T07:27:31.634309+0000","last_undegraded":"2024-03-06T07:27:31.634309+0000","last_fullsized":"2024-03-06T07:27:31.634309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696681+0000","last_change":"2024-03-06T07:27:30.240983+0000","last_active":"2024-03-06T07:27:31.696681+0000","last_peered":"2024-03-06T07:27:31.696681+0000","last_clean":"2024-03-06T07:27:31.696681+0000","last_became_active":"2024-03-06T07:27:30.240088+0000","last_became_peered":"2024-03-06T07:27:30.240088+0000","last_unstale":"2024-03-06T07:27:31.696681+0000","last_undegraded":"2024-03-06T07:27:31.696681+0000","last_fullsized":"2024-03-06T07:27:31.696681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241714+0000","last_change":"2024-03-06T07:27:30.254933+0000","last_active":"2024-03-06T07:27:31.241714+0000","last_peered":"2024-03-06T07:27:31.241714+0000","last_clean":"2024-03-06T07:27:31.241714+0000","last_became_active":"2024-03-06T07:27:30.253777+0000","last_became_peered":"2024-03-06T07:27:30.253777+0000","last_unstale":"2024-03-06T07:27:31.241714+0000","last_undegraded":"2024-03-06T07:27:31.241714+0000","last_fullsized":"2024-03-06T07:27:31.241714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696716+0000","last_change":"2024-03-06T07:27:30.254617+0000","last_active":"2024-03-06T07:27:31.696716+0000","last_peered":"2024-03-06T07:27:31.696716+0000","last_clean":"2024-03-06T07:27:31.696716+0000","last_became_active":"2024-03-06T07:27:30.253826+0000","last_became_peered":"2024-03-06T07:27:30.253826+0000","last_unstale":"2024-03-06T07:27:31.696716+0000","last_undegraded":"2024-03-06T07:27:31.696716+0000","last_fullsized":"2024-03-06T07:27:31.696716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243679+0000","last_change":"2024-03-06T07:27:30.245625+0000","last_active":"2024-03-06T07:27:31.243679+0000","last_peered":"2024-03-06T07:27:31.243679+0000","last_clean":"2024-03-06T07:27:31.243679+0000","last_became_active":"2024-03-06T07:27:30.245310+0000","last_became_peered":"2024-03-06T07:27:30.245310+0000","last_unstale":"2024-03-06T07:27:31.243679+0000","last_undegraded":"2024-03-06T07:27:31.243679+0000","last_fullsized":"2024-03-06T07:27:31.243679+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.246068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243256+0000","last_change":"2024-03-06T07:27:30.241785+0000","last_active":"2024-03-06T07:27:31.243256+0000","last_peered":"2024-03-06T07:27:31.243256+0000","last_clean":"2024-03-06T07:27:31.243256+0000","last_became_active":"2024-03-06T07:27:30.241456+0000","last_became_peered":"2024-03-06T07:27:30.241456+0000","last_unstale":"2024-03-06T07:27:31.243256+0000","last_undegraded":"2024-03-06T07:27:31.243256+0000","last_fullsized":"2024-03-06T07:27:31.243256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239562+0000","last_change":"2024-03-06T07:27:30.239090+0000","last_active":"2024-03-06T07:27:31.239562+0000","last_peered":"2024-03-06T07:27:31.239562+0000","last_clean":"2024-03-06T07:27:31.239562+0000","last_became_active":"2024-03-06T07:27:30.238809+0000","last_became_peered":"2024-03-06T07:27:30.238809+0000","last_unstale":"2024-03-06T07:27:31.239562+0000","last_undegraded":"2024-03-06T07:27:31.239562+0000","last_fullsized":"2024-03-06T07:27:31.239562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:18.053835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243221+0000","last_change":"2024-03-06T07:27:30.250473+0000","last_active":"2024-03-06T07:27:31.243221+0000","last_peered":"2024-03-06T07:27:31.243221+0000","last_clean":"2024-03-06T07:27:31.243221+0000","last_became_active":"2024-03-06T07:27:30.250084+0000","last_became_peered":"2024-03-06T07:27:30.250084+0000","last_unstale":"2024-03-06T07:27:31.243221+0000","last_undegraded":"2024-03-06T07:27:31.243221+0000","last_fullsized":"2024-03-06T07:27:31.243221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634430+0000","last_change":"2024-03-06T07:27:30.247866+0000","last_active":"2024-03-06T07:27:31.634430+0000","last_peered":"2024-03-06T07:27:31.634430+0000","last_clean":"2024-03-06T07:27:31.634430+0000","last_became_active":"2024-03-06T07:27:30.247526+0000","last_became_peered":"2024-03-06T07:27:30.247526+0000","last_unstale":"2024-03-06T07:27:31.634430+0000","last_undegraded":"2024-03-06T07:27:31.634430+0000","last_fullsized":"2024-03-06T07:27:31.634430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696909+0000","last_change":"2024-03-06T07:27:30.252736+0000","last_active":"2024-03-06T07:27:31.696909+0000","last_peered":"2024-03-06T07:27:31.696909+0000","last_clean":"2024-03-06T07:27:31.696909+0000","last_became_active":"2024-03-06T07:27:30.252267+0000","last_became_peered":"2024-03-06T07:27:30.252267+0000","last_unstale":"2024-03-06T07:27:31.696909+0000","last_undegraded":"2024-03-06T07:27:31.696909+0000","last_fullsized":"2024-03-06T07:27:31.696909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.111110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695816+0000","last_change":"2024-03-06T07:27:30.265689+0000","last_active":"2024-03-06T07:27:31.695816+0000","last_peered":"2024-03-06T07:27:31.695816+0000","last_clean":"2024-03-06T07:27:31.695816+0000","last_became_active":"2024-03-06T07:27:30.265410+0000","last_became_peered":"2024-03-06T07:27:30.265410+0000","last_unstale":"2024-03-06T07:27:31.695816+0000","last_undegraded":"2024-03-06T07:27:31.695816+0000","last_fullsized":"2024-03-06T07:27:31.695816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:00.010236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242466+0000","last_change":"2024-03-06T07:27:30.238867+0000","last_active":"2024-03-06T07:27:31.242466+0000","last_peered":"2024-03-06T07:27:31.242466+0000","last_clean":"2024-03-06T07:27:31.242466+0000","last_became_active":"2024-03-06T07:27:30.238543+0000","last_became_peered":"2024-03-06T07:27:30.238543+0000","last_unstale":"2024-03-06T07:27:31.242466+0000","last_undegraded":"2024-03-06T07:27:31.242466+0000","last_fullsized":"2024-03-06T07:27:31.242466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699273+0000","last_change":"2024-03-06T07:27:30.244526+0000","last_active":"2024-03-06T07:27:31.699273+0000","last_peered":"2024-03-06T07:27:31.699273+0000","last_clean":"2024-03-06T07:27:31.699273+0000","last_became_active":"2024-03-06T07:27:30.244266+0000","last_became_peered":"2024-03-06T07:27:30.244266+0000","last_unstale":"2024-03-06T07:27:31.699273+0000","last_undegraded":"2024-03-06T07:27:31.699273+0000","last_fullsized":"2024-03-06T07:27:31.699273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:10.823443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242632+0000","last_change":"2024-03-06T07:27:29.224405+0000","last_active":"2024-03-06T07:27:31.242632+0000","last_peered":"2024-03-06T07:27:31.242632+0000","last_clean":"2024-03-06T07:27:31.242632+0000","last_became_active":"2024-03-06T07:27:29.223485+0000","last_became_peered":"2024-03-06T07:27:29.223485+0000","last_unstale":"2024-03-06T07:27:31.242632+0000","last_undegraded":"2024-03-06T07:27:31.242632+0000","last_fullsized":"2024-03-06T07:27:31.242632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634481+0000","last_change":"2024-03-06T07:27:30.249021+0000","last_active":"2024-03-06T07:27:31.634481+0000","last_peered":"2024-03-06T07:27:31.634481+0000","last_clean":"2024-03-06T07:27:31.634481+0000","last_became_active":"2024-03-06T07:27:30.248148+0000","last_became_peered":"2024-03-06T07:27:30.248148+0000","last_unstale":"2024-03-06T07:27:31.634481+0000","last_undegraded":"2024-03-06T07:27:31.634481+0000","last_fullsized":"2024-03-06T07:27:31.634481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243028+0000","last_change":"2024-03-06T07:27:29.230274+0000","last_active":"2024-03-06T07:27:31.243028+0000","last_peered":"2024-03-06T07:27:31.243028+0000","last_clean":"2024-03-06T07:27:31.243028+0000","last_became_active":"2024-03-06T07:27:29.230009+0000","last_became_peered":"2024-03-06T07:27:29.230009+0000","last_unstale":"2024-03-06T07:27:31.243028+0000","last_undegraded":"2024-03-06T07:27:31.243028+0000","last_fullsized":"2024-03-06T07:27:31.243028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241766+0000","last_change":"2024-03-06T07:27:29.221872+0000","last_active":"2024-03-06T07:27:31.241766+0000","last_peered":"2024-03-06T07:27:31.241766+0000","last_clean":"2024-03-06T07:27:31.241766+0000","last_became_active":"2024-03-06T07:27:29.221430+0000","last_became_peered":"2024-03-06T07:27:29.221430+0000","last_unstale":"2024-03-06T07:27:31.241766+0000","last_undegraded":"2024-03-06T07:27:31.241766+0000","last_fullsized":"2024-03-06T07:27:31.241766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241901+0000","last_change":"2024-03-06T07:27:30.235213+0000","last_active":"2024-03-06T07:27:31.241901+0000","last_peered":"2024-03-06T07:27:31.241901+0000","last_clean":"2024-03-06T07:27:31.241901+0000","last_became_active":"2024-03-06T07:27:30.234891+0000","last_became_peered":"2024-03-06T07:27:30.234891+0000","last_unstale":"2024-03-06T07:27:31.241901+0000","last_undegraded":"2024-03-06T07:27:31.241901+0000","last_fullsized":"2024-03-06T07:27:31.241901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696609+0000","last_change":"2024-03-06T07:27:29.228355+0000","last_active":"2024-03-06T07:27:31.696609+0000","last_peered":"2024-03-06T07:27:31.696609+0000","last_clean":"2024-03-06T07:27:31.696609+0000","last_became_active":"2024-03-06T07:27:29.228054+0000","last_became_peered":"2024-03-06T07:27:29.228054+0000","last_unstale":"2024-03-06T07:27:31.696609+0000","last_undegraded":"2024-03-06T07:27:31.696609+0000","last_fullsized":"2024-03-06T07:27:31.696609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240414+0000","last_change":"2024-03-06T07:27:30.236691+0000","last_active":"2024-03-06T07:27:31.240414+0000","last_peered":"2024-03-06T07:27:31.240414+0000","last_clean":"2024-03-06T07:27:31.240414+0000","last_became_active":"2024-03-06T07:27:30.236318+0000","last_became_peered":"2024-03-06T07:27:30.236318+0000","last_unstale":"2024-03-06T07:27:31.240414+0000","last_undegraded":"2024-03-06T07:27:31.240414+0000","last_fullsized":"2024-03-06T07:27:31.240414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:06.911342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242168+0000","last_change":"2024-03-06T07:27:29.218901+0000","last_active":"2024-03-06T07:27:31.242168+0000","last_peered":"2024-03-06T07:27:31.242168+0000","last_clean":"2024-03-06T07:27:31.242168+0000","last_became_active":"2024-03-06T07:27:29.218549+0000","last_became_peered":"2024-03-06T07:27:29.218549+0000","last_unstale":"2024-03-06T07:27:31.242168+0000","last_undegraded":"2024-03-06T07:27:31.242168+0000","last_fullsized":"2024-03-06T07:27:31.242168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697521+0000","last_change":"2024-03-06T07:27:30.269139+0000","last_active":"2024-03-06T07:27:31.697521+0000","last_peered":"2024-03-06T07:27:31.697521+0000","last_clean":"2024-03-06T07:27:31.697521+0000","last_became_active":"2024-03-06T07:27:30.265940+0000","last_became_peered":"2024-03-06T07:27:30.265940+0000","last_unstale":"2024-03-06T07:27:31.697521+0000","last_undegraded":"2024-03-06T07:27:31.697521+0000","last_fullsized":"2024-03-06T07:27:31.697521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:13.515252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697588+0000","last_change":"2024-03-06T07:27:29.224828+0000","last_active":"2024-03-06T07:27:31.697588+0000","last_peered":"2024-03-06T07:27:31.697588+0000","last_clean":"2024-03-06T07:27:31.697588+0000","last_became_active":"2024-03-06T07:27:29.223899+0000","last_became_peered":"2024-03-06T07:27:29.223899+0000","last_unstale":"2024-03-06T07:27:31.697588+0000","last_undegraded":"2024-03-06T07:27:31.697588+0000","last_fullsized":"2024-03-06T07:27:31.697588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635385+0000","last_change":"2024-03-06T07:27:30.258895+0000","last_active":"2024-03-06T07:27:31.635385+0000","last_peered":"2024-03-06T07:27:31.635385+0000","last_clean":"2024-03-06T07:27:31.635385+0000","last_became_active":"2024-03-06T07:27:30.258204+0000","last_became_peered":"2024-03-06T07:27:30.258204+0000","last_unstale":"2024-03-06T07:27:31.635385+0000","last_undegraded":"2024-03-06T07:27:31.635385+0000","last_fullsized":"2024-03-06T07:27:31.635385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239843+0000","last_change":"2024-03-06T07:27:29.232043+0000","last_active":"2024-03-06T07:27:31.239843+0000","last_peered":"2024-03-06T07:27:31.239843+0000","last_clean":"2024-03-06T07:27:31.239843+0000","last_became_active":"2024-03-06T07:27:29.231659+0000","last_became_peered":"2024-03-06T07:27:29.231659+0000","last_unstale":"2024-03-06T07:27:31.239843+0000","last_undegraded":"2024-03-06T07:27:31.239843+0000","last_fullsized":"2024-03-06T07:27:31.239843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700788+0000","last_change":"2024-03-06T07:27:30.253450+0000","last_active":"2024-03-06T07:27:31.700788+0000","last_peered":"2024-03-06T07:27:31.700788+0000","last_clean":"2024-03-06T07:27:31.700788+0000","last_became_active":"2024-03-06T07:27:30.250531+0000","last_became_peered":"2024-03-06T07:27:30.250531+0000","last_unstale":"2024-03-06T07:27:31.700788+0000","last_undegraded":"2024-03-06T07:27:31.700788+0000","last_fullsized":"2024-03-06T07:27:31.700788+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:46.446447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242917+0000","last_change":"2024-03-06T07:27:29.216258+0000","last_active":"2024-03-06T07:27:31.242917+0000","last_peered":"2024-03-06T07:27:31.242917+0000","last_clean":"2024-03-06T07:27:31.242917+0000","last_became_active":"2024-03-06T07:27:29.215924+0000","last_became_peered":"2024-03-06T07:27:29.215924+0000","last_unstale":"2024-03-06T07:27:31.242917+0000","last_undegraded":"2024-03-06T07:27:31.242917+0000","last_fullsized":"2024-03-06T07:27:31.242917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697945+0000","last_change":"2024-03-06T07:27:30.248670+0000","last_active":"2024-03-06T07:27:31.697945+0000","last_peered":"2024-03-06T07:27:31.697945+0000","last_clean":"2024-03-06T07:27:31.697945+0000","last_became_active":"2024-03-06T07:27:30.248398+0000","last_became_peered":"2024-03-06T07:27:30.248398+0000","last_unstale":"2024-03-06T07:27:31.697945+0000","last_undegraded":"2024-03-06T07:27:31.697945+0000","last_fullsized":"2024-03-06T07:27:31.697945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:06.911342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634619+0000","last_change":"2024-03-06T07:27:29.228193+0000","last_active":"2024-03-06T07:27:31.634619+0000","last_peered":"2024-03-06T07:27:31.634619+0000","last_clean":"2024-03-06T07:27:31.634619+0000","last_became_active":"2024-03-06T07:27:29.227840+0000","last_became_peered":"2024-03-06T07:27:29.227840+0000","last_unstale":"2024-03-06T07:27:31.634619+0000","last_undegraded":"2024-03-06T07:27:31.634619+0000","last_fullsized":"2024-03-06T07:27:31.634619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:21.775657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634157+0000","last_change":"2024-03-06T07:27:30.240692+0000","last_active":"2024-03-06T07:27:31.634157+0000","last_peered":"2024-03-06T07:27:31.634157+0000","last_clean":"2024-03-06T07:27:31.634157+0000","last_became_active":"2024-03-06T07:27:30.240410+0000","last_became_peered":"2024-03-06T07:27:30.240410+0000","last_unstale":"2024-03-06T07:27:31.634157+0000","last_undegraded":"2024-03-06T07:27:31.634157+0000","last_fullsized":"2024-03-06T07:27:31.634157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:44.205908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696997+0000","last_change":"2024-03-06T07:27:29.231958+0000","last_active":"2024-03-06T07:27:31.696997+0000","last_peered":"2024-03-06T07:27:31.696997+0000","last_clean":"2024-03-06T07:27:31.696997+0000","last_became_active":"2024-03-06T07:27:29.231690+0000","last_became_peered":"2024-03-06T07:27:29.231690+0000","last_unstale":"2024-03-06T07:27:31.696997+0000","last_undegraded":"2024-03-06T07:27:31.696997+0000","last_fullsized":"2024-03-06T07:27:31.696997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700469+0000","last_change":"2024-03-06T07:27:30.258434+0000","last_active":"2024-03-06T07:27:31.700469+0000","last_peered":"2024-03-06T07:27:31.700469+0000","last_clean":"2024-03-06T07:27:31.700469+0000","last_became_active":"2024-03-06T07:27:30.258094+0000","last_became_peered":"2024-03-06T07:27:30.258094+0000","last_unstale":"2024-03-06T07:27:31.700469+0000","last_undegraded":"2024-03-06T07:27:31.700469+0000","last_fullsized":"2024-03-06T07:27:31.700469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:58.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241544+0000","last_change":"2024-03-06T07:27:29.231347+0000","last_active":"2024-03-06T07:27:31.241544+0000","last_peered":"2024-03-06T07:27:31.241544+0000","last_clean":"2024-03-06T07:27:31.241544+0000","last_became_active":"2024-03-06T07:27:29.230852+0000","last_became_peered":"2024-03-06T07:27:29.230852+0000","last_unstale":"2024-03-06T07:27:31.241544+0000","last_undegraded":"2024-03-06T07:27:31.241544+0000","last_fullsized":"2024-03-06T07:27:31.241544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242563+0000","last_change":"2024-03-06T07:27:30.240275+0000","last_active":"2024-03-06T07:27:31.242563+0000","last_peered":"2024-03-06T07:27:31.242563+0000","last_clean":"2024-03-06T07:27:31.242563+0000","last_became_active":"2024-03-06T07:27:30.239945+0000","last_became_peered":"2024-03-06T07:27:30.239945+0000","last_unstale":"2024-03-06T07:27:31.242563+0000","last_undegraded":"2024-03-06T07:27:31.242563+0000","last_fullsized":"2024-03-06T07:27:31.242563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695318+0000","last_change":"2024-03-06T07:27:29.217898+0000","last_active":"2024-03-06T07:27:31.695318+0000","last_peered":"2024-03-06T07:27:31.695318+0000","last_clean":"2024-03-06T07:27:31.695318+0000","last_became_active":"2024-03-06T07:27:29.217605+0000","last_became_peered":"2024-03-06T07:27:29.217605+0000","last_unstale":"2024-03-06T07:27:31.695318+0000","last_undegraded":"2024-03-06T07:27:31.695318+0000","last_fullsized":"2024-03-06T07:27:31.695318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.683045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698824+0000","last_change":"2024-03-06T07:27:30.258444+0000","last_active":"2024-03-06T07:27:31.698824+0000","last_peered":"2024-03-06T07:27:31.698824+0000","last_clean":"2024-03-06T07:27:31.698824+0000","last_became_active":"2024-03-06T07:27:30.258119+0000","last_became_peered":"2024-03-06T07:27:30.258119+0000","last_unstale":"2024-03-06T07:27:31.698824+0000","last_undegraded":"2024-03-06T07:27:31.698824+0000","last_fullsized":"2024-03-06T07:27:31.698824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:52.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699271+0000","last_change":"2024-03-06T07:27:29.228474+0000","last_active":"2024-03-06T07:27:31.699271+0000","last_peered":"2024-03-06T07:27:31.699271+0000","last_clean":"2024-03-06T07:27:31.699271+0000","last_became_active":"2024-03-06T07:27:29.228168+0000","last_became_peered":"2024-03-06T07:27:29.228168+0000","last_unstale":"2024-03-06T07:27:31.699271+0000","last_undegraded":"2024-03-06T07:27:31.699271+0000","last_fullsized":"2024-03-06T07:27:31.699271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:36.193215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699323+0000","last_change":"2024-03-06T07:27:30.243658+0000","last_active":"2024-03-06T07:27:31.699323+0000","last_peered":"2024-03-06T07:27:31.699323+0000","last_clean":"2024-03-06T07:27:31.699323+0000","last_became_active":"2024-03-06T07:27:30.243145+0000","last_became_peered":"2024-03-06T07:27:30.243145+0000","last_unstale":"2024-03-06T07:27:31.699323+0000","last_undegraded":"2024-03-06T07:27:31.699323+0000","last_fullsized":"2024-03-06T07:27:31.699323+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:10.001805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695709+0000","last_change":"2024-03-06T07:27:29.221215+0000","last_active":"2024-03-06T07:27:31.695709+0000","last_peered":"2024-03-06T07:27:31.695709+0000","last_clean":"2024-03-06T07:27:31.695709+0000","last_became_active":"2024-03-06T07:27:29.220256+0000","last_became_peered":"2024-03-06T07:27:29.220256+0000","last_unstale":"2024-03-06T07:27:31.695709+0000","last_undegraded":"2024-03-06T07:27:31.695709+0000","last_fullsized":"2024-03-06T07:27:31.695709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699734+0000","last_change":"2024-03-06T07:27:30.258256+0000","last_active":"2024-03-06T07:27:31.699734+0000","last_peered":"2024-03-06T07:27:31.699734+0000","last_clean":"2024-03-06T07:27:31.699734+0000","last_became_active":"2024-03-06T07:27:30.255209+0000","last_became_peered":"2024-03-06T07:27:30.255209+0000","last_unstale":"2024-03-06T07:27:31.699734+0000","last_undegraded":"2024-03-06T07:27:31.699734+0000","last_fullsized":"2024-03-06T07:27:31.699734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:11.942117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240185+0000","last_change":"2024-03-06T07:27:29.225919+0000","last_active":"2024-03-06T07:27:31.240185+0000","last_peered":"2024-03-06T07:27:31.240185+0000","last_clean":"2024-03-06T07:27:31.240185+0000","last_became_active":"2024-03-06T07:27:29.225690+0000","last_became_peered":"2024-03-06T07:27:29.225690+0000","last_unstale":"2024-03-06T07:27:31.240185+0000","last_undegraded":"2024-03-06T07:27:31.240185+0000","last_fullsized":"2024-03-06T07:27:31.240185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698122+0000","last_change":"2024-03-06T07:27:30.259916+0000","last_active":"2024-03-06T07:27:31.698122+0000","last_peered":"2024-03-06T07:27:31.698122+0000","last_clean":"2024-03-06T07:27:31.698122+0000","last_became_active":"2024-03-06T07:27:30.259100+0000","last_became_peered":"2024-03-06T07:27:30.259100+0000","last_unstale":"2024-03-06T07:27:31.698122+0000","last_undegraded":"2024-03-06T07:27:31.698122+0000","last_fullsized":"2024-03-06T07:27:31.698122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:19.907027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695658+0000","last_change":"2024-03-06T07:27:29.231688+0000","last_active":"2024-03-06T07:27:31.695658+0000","last_peered":"2024-03-06T07:27:31.695658+0000","last_clean":"2024-03-06T07:27:31.695658+0000","last_became_active":"2024-03-06T07:27:29.230816+0000","last_became_peered":"2024-03-06T07:27:29.230816+0000","last_unstale":"2024-03-06T07:27:31.695658+0000","last_undegraded":"2024-03-06T07:27:31.695658+0000","last_fullsized":"2024-03-06T07:27:31.695658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:51.657884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698198+0000","last_change":"2024-03-06T07:27:30.264213+0000","last_active":"2024-03-06T07:27:31.698198+0000","last_peered":"2024-03-06T07:27:31.698198+0000","last_clean":"2024-03-06T07:27:31.698198+0000","last_became_active":"2024-03-06T07:27:30.263986+0000","last_became_peered":"2024-03-06T07:27:30.263986+0000","last_unstale":"2024-03-06T07:27:31.698198+0000","last_undegraded":"2024-03-06T07:27:31.698198+0000","last_fullsized":"2024-03-06T07:27:31.698198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:14.580315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698115+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:27:31.698115+0000","last_peered":"2024-03-06T07:27:31.698115+0000","last_clean":"2024-03-06T07:27:31.698115+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:27:31.698115+0000","last_undegraded":"2024-03-06T07:27:31.698115+0000","last_fullsized":"2024-03-06T07:27:31.698115+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696277+0000","last_change":"2024-03-06T07:27:29.232294+0000","last_active":"2024-03-06T07:27:31.696277+0000","last_peered":"2024-03-06T07:27:31.696277+0000","last_clean":"2024-03-06T07:27:31.696277+0000","last_became_active":"2024-03-06T07:27:29.231839+0000","last_became_peered":"2024-03-06T07:27:29.231839+0000","last_unstale":"2024-03-06T07:27:31.696277+0000","last_undegraded":"2024-03-06T07:27:31.696277+0000","last_fullsized":"2024-03-06T07:27:31.696277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241928+0000","last_change":"2024-03-06T07:27:30.245626+0000","last_active":"2024-03-06T07:27:31.241928+0000","last_peered":"2024-03-06T07:27:31.241928+0000","last_clean":"2024-03-06T07:27:31.241928+0000","last_became_active":"2024-03-06T07:27:30.245324+0000","last_became_peered":"2024-03-06T07:27:30.245324+0000","last_unstale":"2024-03-06T07:27:31.241928+0000","last_undegraded":"2024-03-06T07:27:31.241928+0000","last_fullsized":"2024-03-06T07:27:31.241928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697983+0000","last_change":"2024-03-06T07:27:29.234333+0000","last_active":"2024-03-06T07:27:31.697983+0000","last_peered":"2024-03-06T07:27:31.697983+0000","last_clean":"2024-03-06T07:27:31.697983+0000","last_became_active":"2024-03-06T07:27:29.234024+0000","last_became_peered":"2024-03-06T07:27:29.234024+0000","last_unstale":"2024-03-06T07:27:31.697983+0000","last_undegraded":"2024-03-06T07:27:31.697983+0000","last_fullsized":"2024-03-06T07:27:31.697983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634297+0000","last_change":"2024-03-06T07:27:30.254664+0000","last_active":"2024-03-06T07:27:31.634297+0000","last_peered":"2024-03-06T07:27:31.634297+0000","last_clean":"2024-03-06T07:27:31.634297+0000","last_became_active":"2024-03-06T07:27:30.254371+0000","last_became_peered":"2024-03-06T07:27:30.254371+0000","last_unstale":"2024-03-06T07:27:31.634297+0000","last_undegraded":"2024-03-06T07:27:31.634297+0000","last_fullsized":"2024-03-06T07:27:31.634297+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695651+0000","last_change":"2024-03-06T07:27:29.220564+0000","last_active":"2024-03-06T07:27:31.695651+0000","last_peered":"2024-03-06T07:27:31.695651+0000","last_clean":"2024-03-06T07:27:31.695651+0000","last_became_active":"2024-03-06T07:27:29.220231+0000","last_became_peered":"2024-03-06T07:27:29.220231+0000","last_unstale":"2024-03-06T07:27:31.695651+0000","last_undegraded":"2024-03-06T07:27:31.695651+0000","last_fullsized":"2024-03-06T07:27:31.695651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:18.751461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.238983+0000","last_change":"2024-03-06T07:27:30.244146+0000","last_active":"2024-03-06T07:27:31.238983+0000","last_peered":"2024-03-06T07:27:31.238983+0000","last_clean":"2024-03-06T07:27:31.238983+0000","last_became_active":"2024-03-06T07:27:30.241811+0000","last_became_peered":"2024-03-06T07:27:30.241811+0000","last_unstale":"2024-03-06T07:27:31.238983+0000","last_undegraded":"2024-03-06T07:27:31.238983+0000","last_fullsized":"2024-03-06T07:27:31.238983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:52.703156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697062+0000","last_change":"2024-03-06T07:27:29.221902+0000","last_active":"2024-03-06T07:27:31.697062+0000","last_peered":"2024-03-06T07:27:31.697062+0000","last_clean":"2024-03-06T07:27:31.697062+0000","last_became_active":"2024-03-06T07:27:29.221598+0000","last_became_peered":"2024-03-06T07:27:29.221598+0000","last_unstale":"2024-03-06T07:27:31.697062+0000","last_undegraded":"2024-03-06T07:27:31.697062+0000","last_fullsized":"2024-03-06T07:27:31.697062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700176+0000","last_change":"2024-03-06T07:27:30.243366+0000","last_active":"2024-03-06T07:27:31.700176+0000","last_peered":"2024-03-06T07:27:31.700176+0000","last_clean":"2024-03-06T07:27:31.700176+0000","last_became_active":"2024-03-06T07:27:30.243054+0000","last_became_peered":"2024-03-06T07:27:30.243054+0000","last_unstale":"2024-03-06T07:27:31.700176+0000","last_undegraded":"2024-03-06T07:27:31.700176+0000","last_fullsized":"2024-03-06T07:27:31.700176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:00.207295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239297+0000","last_change":"2024-03-06T07:27:29.220848+0000","last_active":"2024-03-06T07:27:31.239297+0000","last_peered":"2024-03-06T07:27:31.239297+0000","last_clean":"2024-03-06T07:27:31.239297+0000","last_became_active":"2024-03-06T07:27:29.220557+0000","last_became_peered":"2024-03-06T07:27:29.220557+0000","last_unstale":"2024-03-06T07:27:31.239297+0000","last_undegraded":"2024-03-06T07:27:31.239297+0000","last_fullsized":"2024-03-06T07:27:31.239297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698368+0000","last_change":"2024-03-06T07:27:30.259510+0000","last_active":"2024-03-06T07:27:31.698368+0000","last_peered":"2024-03-06T07:27:31.698368+0000","last_clean":"2024-03-06T07:27:31.698368+0000","last_became_active":"2024-03-06T07:27:30.258936+0000","last_became_peered":"2024-03-06T07:27:30.258936+0000","last_unstale":"2024-03-06T07:27:31.698368+0000","last_undegraded":"2024-03-06T07:27:31.698368+0000","last_fullsized":"2024-03-06T07:27:31.698368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:24.977160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698863+0000","last_change":"2024-03-06T07:27:29.237638+0000","last_active":"2024-03-06T07:27:31.698863+0000","last_peered":"2024-03-06T07:27:31.698863+0000","last_clean":"2024-03-06T07:27:31.698863+0000","last_became_active":"2024-03-06T07:27:29.237304+0000","last_became_peered":"2024-03-06T07:27:29.237304+0000","last_unstale":"2024-03-06T07:27:31.698863+0000","last_undegraded":"2024-03-06T07:27:31.698863+0000","last_fullsized":"2024-03-06T07:27:31.698863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240309+0000","last_change":"2024-03-06T07:27:30.241439+0000","last_active":"2024-03-06T07:27:31.240309+0000","last_peered":"2024-03-06T07:27:31.240309+0000","last_clean":"2024-03-06T07:27:31.240309+0000","last_became_active":"2024-03-06T07:27:30.241163+0000","last_became_peered":"2024-03-06T07:27:30.241163+0000","last_unstale":"2024-03-06T07:27:31.240309+0000","last_undegraded":"2024-03-06T07:27:31.240309+0000","last_fullsized":"2024-03-06T07:27:31.240309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.744486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.253333+0000","last_change":"2024-03-06T07:27:29.223056+0000","last_active":"2024-03-06T07:27:31.253333+0000","last_peered":"2024-03-06T07:27:31.253333+0000","last_clean":"2024-03-06T07:27:31.253333+0000","last_became_active":"2024-03-06T07:27:29.222744+0000","last_became_peered":"2024-03-06T07:27:29.222744+0000","last_unstale":"2024-03-06T07:27:31.253333+0000","last_undegraded":"2024-03-06T07:27:31.253333+0000","last_fullsized":"2024-03-06T07:27:31.253333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243861+0000","last_change":"2024-03-06T07:27:30.241735+0000","last_active":"2024-03-06T07:27:31.243861+0000","last_peered":"2024-03-06T07:27:31.243861+0000","last_clean":"2024-03-06T07:27:31.243861+0000","last_became_active":"2024-03-06T07:27:30.241436+0000","last_became_peered":"2024-03-06T07:27:30.241436+0000","last_unstale":"2024-03-06T07:27:31.243861+0000","last_undegraded":"2024-03-06T07:27:31.243861+0000","last_fullsized":"2024-03-06T07:27:31.243861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:58.613904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242269+0000","last_change":"2024-03-06T07:27:29.226601+0000","last_active":"2024-03-06T07:27:31.242269+0000","last_peered":"2024-03-06T07:27:31.242269+0000","last_clean":"2024-03-06T07:27:31.242269+0000","last_became_active":"2024-03-06T07:27:29.225390+0000","last_became_peered":"2024-03-06T07:27:29.225390+0000","last_unstale":"2024-03-06T07:27:31.242269+0000","last_undegraded":"2024-03-06T07:27:31.242269+0000","last_fullsized":"2024-03-06T07:27:31.242269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635023+0000","last_change":"2024-03-06T07:27:30.242063+0000","last_active":"2024-03-06T07:27:31.635023+0000","last_peered":"2024-03-06T07:27:31.635023+0000","last_clean":"2024-03-06T07:27:31.635023+0000","last_became_active":"2024-03-06T07:27:30.241737+0000","last_became_peered":"2024-03-06T07:27:30.241737+0000","last_unstale":"2024-03-06T07:27:31.635023+0000","last_undegraded":"2024-03-06T07:27:31.635023+0000","last_fullsized":"2024-03-06T07:27:31.635023+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697825+0000","last_change":"2024-03-06T07:27:29.235266+0000","last_active":"2024-03-06T07:27:31.697825+0000","last_peered":"2024-03-06T07:27:31.697825+0000","last_clean":"2024-03-06T07:27:31.697825+0000","last_became_active":"2024-03-06T07:27:29.234878+0000","last_became_peered":"2024-03-06T07:27:29.234878+0000","last_unstale":"2024-03-06T07:27:31.697825+0000","last_undegraded":"2024-03-06T07:27:31.697825+0000","last_fullsized":"2024-03-06T07:27:31.697825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698617+0000","last_change":"2024-03-06T07:27:30.250799+0000","last_active":"2024-03-06T07:27:31.698617+0000","last_peered":"2024-03-06T07:27:31.698617+0000","last_clean":"2024-03-06T07:27:31.698617+0000","last_became_active":"2024-03-06T07:27:30.250475+0000","last_became_peered":"2024-03-06T07:27:30.250475+0000","last_unstale":"2024-03-06T07:27:31.698617+0000","last_undegraded":"2024-03-06T07:27:31.698617+0000","last_fullsized":"2024-03-06T07:27:31.698617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:47.520042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239430+0000","last_change":"2024-03-06T07:27:29.224306+0000","last_active":"2024-03-06T07:27:31.239430+0000","last_peered":"2024-03-06T07:27:31.239430+0000","last_clean":"2024-03-06T07:27:31.239430+0000","last_became_active":"2024-03-06T07:27:29.223923+0000","last_became_peered":"2024-03-06T07:27:29.223923+0000","last_unstale":"2024-03-06T07:27:31.239430+0000","last_undegraded":"2024-03-06T07:27:31.239430+0000","last_fullsized":"2024-03-06T07:27:31.239430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242506+0000","last_change":"2024-03-06T07:27:30.246857+0000","last_active":"2024-03-06T07:27:31.242506+0000","last_peered":"2024-03-06T07:27:31.242506+0000","last_clean":"2024-03-06T07:27:31.242506+0000","last_became_active":"2024-03-06T07:27:30.246519+0000","last_became_peered":"2024-03-06T07:27:30.246519+0000","last_unstale":"2024-03-06T07:27:31.242506+0000","last_undegraded":"2024-03-06T07:27:31.242506+0000","last_fullsized":"2024-03-06T07:27:31.242506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239222+0000","last_change":"2024-03-06T07:27:29.217568+0000","last_active":"2024-03-06T07:27:31.239222+0000","last_peered":"2024-03-06T07:27:31.239222+0000","last_clean":"2024-03-06T07:27:31.239222+0000","last_became_active":"2024-03-06T07:27:29.217204+0000","last_became_peered":"2024-03-06T07:27:29.217204+0000","last_unstale":"2024-03-06T07:27:31.239222+0000","last_undegraded":"2024-03-06T07:27:31.239222+0000","last_fullsized":"2024-03-06T07:27:31.239222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242343+0000","last_change":"2024-03-06T07:27:30.244509+0000","last_active":"2024-03-06T07:27:31.242343+0000","last_peered":"2024-03-06T07:27:31.242343+0000","last_clean":"2024-03-06T07:27:31.242343+0000","last_became_active":"2024-03-06T07:27:30.243804+0000","last_became_peered":"2024-03-06T07:27:30.243804+0000","last_unstale":"2024-03-06T07:27:31.242343+0000","last_undegraded":"2024-03-06T07:27:31.242343+0000","last_fullsized":"2024-03-06T07:27:31.242343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700592+0000","last_change":"2024-03-06T07:27:29.235464+0000","last_active":"2024-03-06T07:27:31.700592+0000","last_peered":"2024-03-06T07:27:31.700592+0000","last_clean":"2024-03-06T07:27:31.700592+0000","last_became_active":"2024-03-06T07:27:29.234532+0000","last_became_peered":"2024-03-06T07:27:29.234532+0000","last_unstale":"2024-03-06T07:27:31.700592+0000","last_undegraded":"2024-03-06T07:27:31.700592+0000","last_fullsized":"2024-03-06T07:27:31.700592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242177+0000","last_change":"2024-03-06T07:27:30.257067+0000","last_active":"2024-03-06T07:27:31.242177+0000","last_peered":"2024-03-06T07:27:31.242177+0000","last_clean":"2024-03-06T07:27:31.242177+0000","last_became_active":"2024-03-06T07:27:30.256842+0000","last_became_peered":"2024-03-06T07:27:30.256842+0000","last_unstale":"2024-03-06T07:27:31.242177+0000","last_undegraded":"2024-03-06T07:27:31.242177+0000","last_fullsized":"2024-03-06T07:27:31.242177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697945+0000","last_change":"2024-03-06T07:27:30.266494+0000","last_active":"2024-03-06T07:27:31.697945+0000","last_peered":"2024-03-06T07:27:31.697945+0000","last_clean":"2024-03-06T07:27:31.697945+0000","last_became_active":"2024-03-06T07:27:30.265258+0000","last_became_peered":"2024-03-06T07:27:30.265258+0000","last_unstale":"2024-03-06T07:27:31.697945+0000","last_undegraded":"2024-03-06T07:27:31.697945+0000","last_fullsized":"2024-03-06T07:27:31.697945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:55.013108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699044+0000","last_change":"2024-03-06T07:27:29.235012+0000","last_active":"2024-03-06T07:27:31.699044+0000","last_peered":"2024-03-06T07:27:31.699044+0000","last_clean":"2024-03-06T07:27:31.699044+0000","last_became_active":"2024-03-06T07:27:29.234638+0000","last_became_peered":"2024-03-06T07:27:29.234638+0000","last_unstale":"2024-03-06T07:27:31.699044+0000","last_undegraded":"2024-03-06T07:27:31.699044+0000","last_fullsized":"2024-03-06T07:27:31.699044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697715+0000","last_change":"2024-03-06T07:27:30.265713+0000","last_active":"2024-03-06T07:27:31.697715+0000","last_peered":"2024-03-06T07:27:31.697715+0000","last_clean":"2024-03-06T07:27:31.697715+0000","last_became_active":"2024-03-06T07:27:30.259972+0000","last_became_peered":"2024-03-06T07:27:30.259972+0000","last_unstale":"2024-03-06T07:27:31.697715+0000","last_undegraded":"2024-03-06T07:27:31.697715+0000","last_fullsized":"2024-03-06T07:27:31.697715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:12.658478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698343+0000","last_change":"2024-03-06T07:27:29.236520+0000","last_active":"2024-03-06T07:27:31.698343+0000","last_peered":"2024-03-06T07:27:31.698343+0000","last_clean":"2024-03-06T07:27:31.698343+0000","last_became_active":"2024-03-06T07:27:29.235298+0000","last_became_peered":"2024-03-06T07:27:29.235298+0000","last_unstale":"2024-03-06T07:27:31.698343+0000","last_undegraded":"2024-03-06T07:27:31.698343+0000","last_fullsized":"2024-03-06T07:27:31.698343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635179+0000","last_change":"2024-03-06T07:27:30.258866+0000","last_active":"2024-03-06T07:27:31.635179+0000","last_peered":"2024-03-06T07:27:31.635179+0000","last_clean":"2024-03-06T07:27:31.635179+0000","last_became_active":"2024-03-06T07:27:30.258557+0000","last_became_peered":"2024-03-06T07:27:30.258557+0000","last_unstale":"2024-03-06T07:27:31.635179+0000","last_undegraded":"2024-03-06T07:27:31.635179+0000","last_fullsized":"2024-03-06T07:27:31.635179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243647+0000","last_change":"2024-03-06T07:27:29.221773+0000","last_active":"2024-03-06T07:27:31.243647+0000","last_peered":"2024-03-06T07:27:31.243647+0000","last_clean":"2024-03-06T07:27:31.243647+0000","last_became_active":"2024-03-06T07:27:29.221365+0000","last_became_peered":"2024-03-06T07:27:29.221365+0000","last_unstale":"2024-03-06T07:27:31.243647+0000","last_undegraded":"2024-03-06T07:27:31.243647+0000","last_fullsized":"2024-03-06T07:27:31.243647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.408581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243868+0000","last_change":"2024-03-06T07:27:30.245108+0000","last_active":"2024-03-06T07:27:31.243868+0000","last_peered":"2024-03-06T07:27:31.243868+0000","last_clean":"2024-03-06T07:27:31.243868+0000","last_became_active":"2024-03-06T07:27:30.244795+0000","last_became_peered":"2024-03-06T07:27:30.244795+0000","last_unstale":"2024-03-06T07:27:31.243868+0000","last_undegraded":"2024-03-06T07:27:31.243868+0000","last_fullsized":"2024-03-06T07:27:31.243868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.111110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635325+0000","last_change":"2024-03-06T07:27:29.229397+0000","last_active":"2024-03-06T07:27:31.635325+0000","last_peered":"2024-03-06T07:27:31.635325+0000","last_clean":"2024-03-06T07:27:31.635325+0000","last_became_active":"2024-03-06T07:27:29.229097+0000","last_became_peered":"2024-03-06T07:27:29.229097+0000","last_unstale":"2024-03-06T07:27:31.635325+0000","last_undegraded":"2024-03-06T07:27:31.635325+0000","last_fullsized":"2024-03-06T07:27:31.635325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":78126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":669076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":79738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":225280,"data_stored":78216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":233472,"data_stored":78252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":79738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":827392,"data_stored":669132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:40.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.676+0000 7fc2afa00640 1 Processor -- start 2024-03-06T07:27:40.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.676+0000 7fc2afa00640 1 -- start start 2024-03-06T07:27:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2afa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2a80af1a0 0x7fc2a815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2afa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc2a80af580 0x7fc2a80b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2afa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2afa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a815cc40 con 0x7fc2a80af1a0 2024-03-06T07:27:40.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2afa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a815cf20 con 0x7fc2a80af580 2024-03-06T07:27:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2afa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a815d200 con 0x7fc2a80cdc20 2024-03-06T07:27:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2ad000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2a80af1a0 0x7fc2a815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2ad000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2a80af1a0 0x7fc2a815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42678/0 (socket says 172.21.15.6:42678) 2024-03-06T07:27:40.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.678+0000 7fc2ad000640 1 -- 172.21.15.6:0/4091280464 learned_addr learned my addr 172.21.15.6:0/4091280464 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2a7e00640 1 --2- 172.21.15.6:0/4091280464 >> v2:172.21.15.178:3300/0 conn(0x7fc2a80af580 0x7fc2a80b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2ada00640 1 --2- 172.21.15.6:0/4091280464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2ad000640 1 -- 172.21.15.6:0/4091280464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 msgr2=0x7fc2a80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2ad000640 1 --2- 172.21.15.6:0/4091280464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2ad000640 1 -- 172.21.15.6:0/4091280464 >> v2:172.21.15.178:3300/0 conn(0x7fc2a80af580 msgr2=0x7fc2a80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2ad000640 1 --2- 172.21.15.6:0/4091280464 >> v2:172.21.15.178:3300/0 conn(0x7fc2a80af580 0x7fc2a80b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.679+0000 7fc2ad000640 1 -- 172.21.15.6:0/4091280464 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a815d4e0 con 0x7fc2a80af1a0 2024-03-06T07:27:40.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.680+0000 7fc2ad000640 1 --2- 172.21.15.6:0/4091280464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a815c600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80b54c0 tx=0x7fc29c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91eecbfd61a812c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.681+0000 7fc2a7400640 1 -- 172.21.15.6:0/4091280464 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c02e070 con 0x7fc2a80af1a0 2024-03-06T07:27:40.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.681+0000 7fc2a7400640 1 -- 172.21.15.6:0/4091280464 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c037070 con 0x7fc2a80af1a0 2024-03-06T07:27:40.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.681+0000 7fc2a7400640 1 -- 172.21.15.6:0/4091280464 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c031a00 con 0x7fc2a80af1a0 2024-03-06T07:27:40.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.681+0000 7fc2afa00640 1 -- 172.21.15.6:0/4091280464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 msgr2=0x7fc2a815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.681+0000 7fc2afa00640 1 --2- 172.21.15.6:0/4091280464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a815c600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80b54c0 tx=0x7fc29c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:40.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 -- 172.21.15.6:0/4091280464 shutdown_connections 2024-03-06T07:27:40.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 --2- 172.21.15.6:0/4091280464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 --2- 172.21.15.6:0/4091280464 >> v2:172.21.15.178:3300/0 conn(0x7fc2a80af580 0x7fc2a80b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 --2- 172.21.15.6:0/4091280464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a815c600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 -- 172.21.15.6:0/4091280464 >> 172.21.15.6:0/4091280464 conn(0x7fc2a806afb0 msgr2=0x7fc2a806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 -- 172.21.15.6:0/4091280464 shutdown_connections 2024-03-06T07:27:40.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.682+0000 7fc2afa00640 1 -- 172.21.15.6:0/4091280464 wait complete. 2024-03-06T07:27:40.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.684+0000 7fc2afa00640 1 Processor -- start 2024-03-06T07:27:40.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.684+0000 7fc2afa00640 1 -- start start 2024-03-06T07:27:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.685+0000 7fc2afa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.685+0000 7fc2afa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2a80af580 0x7fc2a8151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.685+0000 7fc2afa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.685+0000 7fc2afa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80b3490 con 0x7fc2a80af1a0 2024-03-06T07:27:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.685+0000 7fc2afa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80b3310 con 0x7fc2a80af580 2024-03-06T07:27:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.685+0000 7fc2afa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a80b3610 con 0x7fc2a80cdc20 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a806e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42682/0 (socket says 172.21.15.6:42682) 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 -- 172.21.15.6:0/2413129898 learned_addr learned my addr 172.21.15.6:0/2413129898 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 -- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 msgr2=0x7fc2a80b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 -- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2a80af580 msgr2=0x7fc2a8151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2a80af580 0x7fc2a8151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 -- 172.21.15.6:0/2413129898 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a8151bf0 con 0x7fc2a80af1a0 2024-03-06T07:27:40.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.686+0000 7fc2ad000640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a806e7f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc29c000c00 tx=0x7fc29c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.687+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c032670 con 0x7fc2a80af1a0 2024-03-06T07:27:40.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.687+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c032830 con 0x7fc2a80af1a0 2024-03-06T07:27:40.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.687+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c032c20 con 0x7fc2a80af1a0 2024-03-06T07:27:40.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.687+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a815d8c0 con 0x7fc2a80af1a0 2024-03-06T07:27:40.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.687+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a815de80 con 0x7fc2a80af1a0 2024-03-06T07:27:40.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.688+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a804f3a0 con 0x7fc2a80af1a0 2024-03-06T07:27:40.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.689+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc29c04c060 con 0x7fc2a80af1a0 2024-03-06T07:27:40.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.690+0000 7fc2a4c00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc28806b4d0 0x7fc28806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:40.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.690+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc29c0ab8c0 con 0x7fc2a80af1a0 2024-03-06T07:27:40.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.690+0000 7fc2a7e00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc28806b4d0 0x7fc28806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.691+0000 7fc2a7e00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc28806b4d0 0x7fc28806d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc2a81581f0 tx=0x7fc294008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:40.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.699+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc29c040070 con 0x7fc2a80af1a0 2024-03-06T07:27:40.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.893+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2a80cc930 con 0x7fc28806b4d0 2024-03-06T07:27:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.912+0000 7fc2a4c00640 1 -- 172.21.15.6:0/2413129898 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7fc2a80cc930 con 0x7fc28806b4d0 2024-03-06T07:27:40.917 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc28806b4d0 msgr2=0x7fc28806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc28806b4d0 0x7fc28806d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc2a81581f0 tx=0x7fc294008040 comp rx=0 tx=0).stop 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 msgr2=0x7fc2a806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a806e7f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc29c000c00 tx=0x7fc29c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 shutdown_connections 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc28806b4d0 0x7fc28806d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2a80af580 0x7fc2a8151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2a80cdc20 0x7fc2a80b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 --2- 172.21.15.6:0/2413129898 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2a80af1a0 0x7fc2a806e7f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 >> 172.21.15.6:0/2413129898 conn(0x7fc2a806afb0 msgr2=0x7fc2a80ad520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 shutdown_connections 2024-03-06T07:27:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:40.917+0000 7fc2afa00640 1 -- 172.21.15.6:0/2413129898 wait complete. 2024-03-06T07:27:40.938 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-06T07:27:39.440318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235772,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738948,"statfs":{"total":773094113280,"available":772852682752,"internally_reserved":0,"allocated":3022848,"data_stored":1810564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.230310"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699359+0000","last_change":"2024-03-06T07:27:30.239373+0000","last_active":"2024-03-06T07:27:31.699359+0000","last_peered":"2024-03-06T07:27:31.699359+0000","last_clean":"2024-03-06T07:27:31.699359+0000","last_became_active":"2024-03-06T07:27:30.239046+0000","last_became_peered":"2024-03-06T07:27:30.239046+0000","last_unstale":"2024-03-06T07:27:31.699359+0000","last_undegraded":"2024-03-06T07:27:31.699359+0000","last_fullsized":"2024-03-06T07:27:31.699359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:38.519257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239467+0000","last_change":"2024-03-06T07:27:30.234823+0000","last_active":"2024-03-06T07:27:31.239467+0000","last_peered":"2024-03-06T07:27:31.239467+0000","last_clean":"2024-03-06T07:27:31.239467+0000","last_became_active":"2024-03-06T07:27:30.234521+0000","last_became_peered":"2024-03-06T07:27:30.234521+0000","last_unstale":"2024-03-06T07:27:31.239467+0000","last_undegraded":"2024-03-06T07:27:31.239467+0000","last_fullsized":"2024-03-06T07:27:31.239467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:17.318504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634508+0000","last_change":"2024-03-06T07:27:30.244772+0000","last_active":"2024-03-06T07:27:31.634508+0000","last_peered":"2024-03-06T07:27:31.634508+0000","last_clean":"2024-03-06T07:27:31.634508+0000","last_became_active":"2024-03-06T07:27:30.244401+0000","last_became_peered":"2024-03-06T07:27:30.244401+0000","last_unstale":"2024-03-06T07:27:31.634508+0000","last_undegraded":"2024-03-06T07:27:31.634508+0000","last_fullsized":"2024-03-06T07:27:31.634508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699779+0000","last_change":"2024-03-06T07:27:30.260180+0000","last_active":"2024-03-06T07:27:31.699779+0000","last_peered":"2024-03-06T07:27:31.699779+0000","last_clean":"2024-03-06T07:27:31.699779+0000","last_became_active":"2024-03-06T07:27:30.259183+0000","last_became_peered":"2024-03-06T07:27:30.259183+0000","last_unstale":"2024-03-06T07:27:31.699779+0000","last_undegraded":"2024-03-06T07:27:31.699779+0000","last_fullsized":"2024-03-06T07:27:31.699779+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:46.385603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700449+0000","last_change":"2024-03-06T07:27:30.254425+0000","last_active":"2024-03-06T07:27:31.700449+0000","last_peered":"2024-03-06T07:27:31.700449+0000","last_clean":"2024-03-06T07:27:31.700449+0000","last_became_active":"2024-03-06T07:27:30.253238+0000","last_became_peered":"2024-03-06T07:27:30.253238+0000","last_unstale":"2024-03-06T07:27:31.700449+0000","last_undegraded":"2024-03-06T07:27:31.700449+0000","last_fullsized":"2024-03-06T07:27:31.700449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:45.798875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696752+0000","last_change":"2024-03-06T07:27:30.259178+0000","last_active":"2024-03-06T07:27:31.696752+0000","last_peered":"2024-03-06T07:27:31.696752+0000","last_clean":"2024-03-06T07:27:31.696752+0000","last_became_active":"2024-03-06T07:27:30.258928+0000","last_became_peered":"2024-03-06T07:27:30.258928+0000","last_unstale":"2024-03-06T07:27:31.696752+0000","last_undegraded":"2024-03-06T07:27:31.696752+0000","last_fullsized":"2024-03-06T07:27:31.696752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.246068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698449+0000","last_change":"2024-03-06T07:27:30.244331+0000","last_active":"2024-03-06T07:27:31.698449+0000","last_peered":"2024-03-06T07:27:31.698449+0000","last_clean":"2024-03-06T07:27:31.698449+0000","last_became_active":"2024-03-06T07:27:30.244078+0000","last_became_peered":"2024-03-06T07:27:30.244078+0000","last_unstale":"2024-03-06T07:27:31.698449+0000","last_undegraded":"2024-03-06T07:27:31.698449+0000","last_fullsized":"2024-03-06T07:27:31.698449+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:36:18.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241726+0000","last_change":"2024-03-06T07:27:30.256607+0000","last_active":"2024-03-06T07:27:31.241726+0000","last_peered":"2024-03-06T07:27:31.241726+0000","last_clean":"2024-03-06T07:27:31.241726+0000","last_became_active":"2024-03-06T07:27:30.256321+0000","last_became_peered":"2024-03-06T07:27:30.256321+0000","last_unstale":"2024-03-06T07:27:31.241726+0000","last_undegraded":"2024-03-06T07:27:31.241726+0000","last_fullsized":"2024-03-06T07:27:31.241726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699557+0000","last_change":"2024-03-06T07:27:30.260973+0000","last_active":"2024-03-06T07:27:31.699557+0000","last_peered":"2024-03-06T07:27:31.699557+0000","last_clean":"2024-03-06T07:27:31.699557+0000","last_became_active":"2024-03-06T07:27:30.260401+0000","last_became_peered":"2024-03-06T07:27:30.260401+0000","last_unstale":"2024-03-06T07:27:31.699557+0000","last_undegraded":"2024-03-06T07:27:31.699557+0000","last_fullsized":"2024-03-06T07:27:31.699557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:14.052739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243083+0000","last_change":"2024-03-06T07:27:30.247832+0000","last_active":"2024-03-06T07:27:31.243083+0000","last_peered":"2024-03-06T07:27:31.243083+0000","last_clean":"2024-03-06T07:27:31.243083+0000","last_became_active":"2024-03-06T07:27:30.247503+0000","last_became_peered":"2024-03-06T07:27:30.247503+0000","last_unstale":"2024-03-06T07:27:31.243083+0000","last_undegraded":"2024-03-06T07:27:31.243083+0000","last_fullsized":"2024-03-06T07:27:31.243083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242309+0000","last_change":"2024-03-06T07:27:30.256946+0000","last_active":"2024-03-06T07:27:31.242309+0000","last_peered":"2024-03-06T07:27:31.242309+0000","last_clean":"2024-03-06T07:27:31.242309+0000","last_became_active":"2024-03-06T07:27:30.256643+0000","last_became_peered":"2024-03-06T07:27:30.256643+0000","last_unstale":"2024-03-06T07:27:31.242309+0000","last_undegraded":"2024-03-06T07:27:31.242309+0000","last_fullsized":"2024-03-06T07:27:31.242309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700787+0000","last_change":"2024-03-06T07:27:30.248076+0000","last_active":"2024-03-06T07:27:31.700787+0000","last_peered":"2024-03-06T07:27:31.700787+0000","last_clean":"2024-03-06T07:27:31.700787+0000","last_became_active":"2024-03-06T07:27:30.247733+0000","last_became_peered":"2024-03-06T07:27:30.247733+0000","last_unstale":"2024-03-06T07:27:31.700787+0000","last_undegraded":"2024-03-06T07:27:31.700787+0000","last_fullsized":"2024-03-06T07:27:31.700787+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:00.010236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243474+0000","last_change":"2024-03-06T07:27:30.238620+0000","last_active":"2024-03-06T07:27:31.243474+0000","last_peered":"2024-03-06T07:27:31.243474+0000","last_clean":"2024-03-06T07:27:31.243474+0000","last_became_active":"2024-03-06T07:27:30.238357+0000","last_became_peered":"2024-03-06T07:27:30.238357+0000","last_unstale":"2024-03-06T07:27:31.243474+0000","last_undegraded":"2024-03-06T07:27:31.243474+0000","last_fullsized":"2024-03-06T07:27:31.243474+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634214+0000","last_change":"2024-03-06T07:27:30.242346+0000","last_active":"2024-03-06T07:27:31.634214+0000","last_peered":"2024-03-06T07:27:31.634214+0000","last_clean":"2024-03-06T07:27:31.634214+0000","last_became_active":"2024-03-06T07:27:30.242081+0000","last_became_peered":"2024-03-06T07:27:30.242081+0000","last_unstale":"2024-03-06T07:27:31.634214+0000","last_undegraded":"2024-03-06T07:27:31.634214+0000","last_fullsized":"2024-03-06T07:27:31.634214+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:48.495220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239135+0000","last_change":"2024-03-06T07:27:30.252254+0000","last_active":"2024-03-06T07:27:31.239135+0000","last_peered":"2024-03-06T07:27:31.239135+0000","last_clean":"2024-03-06T07:27:31.239135+0000","last_became_active":"2024-03-06T07:27:30.251915+0000","last_became_peered":"2024-03-06T07:27:30.251915+0000","last_unstale":"2024-03-06T07:27:31.239135+0000","last_undegraded":"2024-03-06T07:27:31.239135+0000","last_fullsized":"2024-03-06T07:27:31.239135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:42.209474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696450+0000","last_change":"2024-03-06T07:27:30.266564+0000","last_active":"2024-03-06T07:27:31.696450+0000","last_peered":"2024-03-06T07:27:31.696450+0000","last_clean":"2024-03-06T07:27:31.696450+0000","last_became_active":"2024-03-06T07:27:30.265165+0000","last_became_peered":"2024-03-06T07:27:30.265165+0000","last_unstale":"2024-03-06T07:27:31.696450+0000","last_undegraded":"2024-03-06T07:27:31.696450+0000","last_fullsized":"2024-03-06T07:27:31.696450+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697066+0000","last_change":"2024-03-06T07:27:30.266828+0000","last_active":"2024-03-06T07:27:31.697066+0000","last_peered":"2024-03-06T07:27:31.697066+0000","last_clean":"2024-03-06T07:27:31.697066+0000","last_became_active":"2024-03-06T07:27:30.265102+0000","last_became_peered":"2024-03-06T07:27:30.265102+0000","last_unstale":"2024-03-06T07:27:31.697066+0000","last_undegraded":"2024-03-06T07:27:31.697066+0000","last_fullsized":"2024-03-06T07:27:31.697066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:58.613904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695842+0000","last_change":"2024-03-06T07:27:30.239978+0000","last_active":"2024-03-06T07:27:31.695842+0000","last_peered":"2024-03-06T07:27:31.695842+0000","last_clean":"2024-03-06T07:27:31.695842+0000","last_became_active":"2024-03-06T07:27:30.239682+0000","last_became_peered":"2024-03-06T07:27:30.239682+0000","last_unstale":"2024-03-06T07:27:31.695842+0000","last_undegraded":"2024-03-06T07:27:31.695842+0000","last_fullsized":"2024-03-06T07:27:31.695842+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:46.446447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698649+0000","last_change":"2024-03-06T07:27:30.245372+0000","last_active":"2024-03-06T07:27:31.698649+0000","last_peered":"2024-03-06T07:27:31.698649+0000","last_clean":"2024-03-06T07:27:31.698649+0000","last_became_active":"2024-03-06T07:27:30.245096+0000","last_became_peered":"2024-03-06T07:27:30.245096+0000","last_unstale":"2024-03-06T07:27:31.698649+0000","last_undegraded":"2024-03-06T07:27:31.698649+0000","last_fullsized":"2024-03-06T07:27:31.698649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:01:08.875835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242869+0000","last_change":"2024-03-06T07:27:30.244785+0000","last_active":"2024-03-06T07:27:31.242869+0000","last_peered":"2024-03-06T07:27:31.242869+0000","last_clean":"2024-03-06T07:27:31.242869+0000","last_became_active":"2024-03-06T07:27:30.244505+0000","last_became_peered":"2024-03-06T07:27:30.244505+0000","last_unstale":"2024-03-06T07:27:31.242869+0000","last_undegraded":"2024-03-06T07:27:31.242869+0000","last_fullsized":"2024-03-06T07:27:31.242869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634309+0000","last_change":"2024-03-06T07:27:30.253741+0000","last_active":"2024-03-06T07:27:31.634309+0000","last_peered":"2024-03-06T07:27:31.634309+0000","last_clean":"2024-03-06T07:27:31.634309+0000","last_became_active":"2024-03-06T07:27:30.251912+0000","last_became_peered":"2024-03-06T07:27:30.251912+0000","last_unstale":"2024-03-06T07:27:31.634309+0000","last_undegraded":"2024-03-06T07:27:31.634309+0000","last_fullsized":"2024-03-06T07:27:31.634309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696681+0000","last_change":"2024-03-06T07:27:30.240983+0000","last_active":"2024-03-06T07:27:31.696681+0000","last_peered":"2024-03-06T07:27:31.696681+0000","last_clean":"2024-03-06T07:27:31.696681+0000","last_became_active":"2024-03-06T07:27:30.240088+0000","last_became_peered":"2024-03-06T07:27:30.240088+0000","last_unstale":"2024-03-06T07:27:31.696681+0000","last_undegraded":"2024-03-06T07:27:31.696681+0000","last_fullsized":"2024-03-06T07:27:31.696681+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241714+0000","last_change":"2024-03-06T07:27:30.254933+0000","last_active":"2024-03-06T07:27:31.241714+0000","last_peered":"2024-03-06T07:27:31.241714+0000","last_clean":"2024-03-06T07:27:31.241714+0000","last_became_active":"2024-03-06T07:27:30.253777+0000","last_became_peered":"2024-03-06T07:27:30.253777+0000","last_unstale":"2024-03-06T07:27:31.241714+0000","last_undegraded":"2024-03-06T07:27:31.241714+0000","last_fullsized":"2024-03-06T07:27:31.241714+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696716+0000","last_change":"2024-03-06T07:27:30.254617+0000","last_active":"2024-03-06T07:27:31.696716+0000","last_peered":"2024-03-06T07:27:31.696716+0000","last_clean":"2024-03-06T07:27:31.696716+0000","last_became_active":"2024-03-06T07:27:30.253826+0000","last_became_peered":"2024-03-06T07:27:30.253826+0000","last_unstale":"2024-03-06T07:27:31.696716+0000","last_undegraded":"2024-03-06T07:27:31.696716+0000","last_fullsized":"2024-03-06T07:27:31.696716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243679+0000","last_change":"2024-03-06T07:27:30.245625+0000","last_active":"2024-03-06T07:27:31.243679+0000","last_peered":"2024-03-06T07:27:31.243679+0000","last_clean":"2024-03-06T07:27:31.243679+0000","last_became_active":"2024-03-06T07:27:30.245310+0000","last_became_peered":"2024-03-06T07:27:30.245310+0000","last_unstale":"2024-03-06T07:27:31.243679+0000","last_undegraded":"2024-03-06T07:27:31.243679+0000","last_fullsized":"2024-03-06T07:27:31.243679+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:28.246068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243256+0000","last_change":"2024-03-06T07:27:30.241785+0000","last_active":"2024-03-06T07:27:31.243256+0000","last_peered":"2024-03-06T07:27:31.243256+0000","last_clean":"2024-03-06T07:27:31.243256+0000","last_became_active":"2024-03-06T07:27:30.241456+0000","last_became_peered":"2024-03-06T07:27:30.241456+0000","last_unstale":"2024-03-06T07:27:31.243256+0000","last_undegraded":"2024-03-06T07:27:31.243256+0000","last_fullsized":"2024-03-06T07:27:31.243256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239562+0000","last_change":"2024-03-06T07:27:30.239090+0000","last_active":"2024-03-06T07:27:31.239562+0000","last_peered":"2024-03-06T07:27:31.239562+0000","last_clean":"2024-03-06T07:27:31.239562+0000","last_became_active":"2024-03-06T07:27:30.238809+0000","last_became_peered":"2024-03-06T07:27:30.238809+0000","last_unstale":"2024-03-06T07:27:31.239562+0000","last_undegraded":"2024-03-06T07:27:31.239562+0000","last_fullsized":"2024-03-06T07:27:31.239562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:48:18.053835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243221+0000","last_change":"2024-03-06T07:27:30.250473+0000","last_active":"2024-03-06T07:27:31.243221+0000","last_peered":"2024-03-06T07:27:31.243221+0000","last_clean":"2024-03-06T07:27:31.243221+0000","last_became_active":"2024-03-06T07:27:30.250084+0000","last_became_peered":"2024-03-06T07:27:30.250084+0000","last_unstale":"2024-03-06T07:27:31.243221+0000","last_undegraded":"2024-03-06T07:27:31.243221+0000","last_fullsized":"2024-03-06T07:27:31.243221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634430+0000","last_change":"2024-03-06T07:27:30.247866+0000","last_active":"2024-03-06T07:27:31.634430+0000","last_peered":"2024-03-06T07:27:31.634430+0000","last_clean":"2024-03-06T07:27:31.634430+0000","last_became_active":"2024-03-06T07:27:30.247526+0000","last_became_peered":"2024-03-06T07:27:30.247526+0000","last_unstale":"2024-03-06T07:27:31.634430+0000","last_undegraded":"2024-03-06T07:27:31.634430+0000","last_fullsized":"2024-03-06T07:27:31.634430+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696909+0000","last_change":"2024-03-06T07:27:30.252736+0000","last_active":"2024-03-06T07:27:31.696909+0000","last_peered":"2024-03-06T07:27:31.696909+0000","last_clean":"2024-03-06T07:27:31.696909+0000","last_became_active":"2024-03-06T07:27:30.252267+0000","last_became_peered":"2024-03-06T07:27:30.252267+0000","last_unstale":"2024-03-06T07:27:31.696909+0000","last_undegraded":"2024-03-06T07:27:31.696909+0000","last_fullsized":"2024-03-06T07:27:31.696909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.111110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695816+0000","last_change":"2024-03-06T07:27:30.265689+0000","last_active":"2024-03-06T07:27:31.695816+0000","last_peered":"2024-03-06T07:27:31.695816+0000","last_clean":"2024-03-06T07:27:31.695816+0000","last_became_active":"2024-03-06T07:27:30.265410+0000","last_became_peered":"2024-03-06T07:27:30.265410+0000","last_unstale":"2024-03-06T07:27:31.695816+0000","last_undegraded":"2024-03-06T07:27:31.695816+0000","last_fullsized":"2024-03-06T07:27:31.695816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:00.010236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242466+0000","last_change":"2024-03-06T07:27:30.238867+0000","last_active":"2024-03-06T07:27:31.242466+0000","last_peered":"2024-03-06T07:27:31.242466+0000","last_clean":"2024-03-06T07:27:31.242466+0000","last_became_active":"2024-03-06T07:27:30.238543+0000","last_became_peered":"2024-03-06T07:27:30.238543+0000","last_unstale":"2024-03-06T07:27:31.242466+0000","last_undegraded":"2024-03-06T07:27:31.242466+0000","last_fullsized":"2024-03-06T07:27:31.242466+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699273+0000","last_change":"2024-03-06T07:27:30.244526+0000","last_active":"2024-03-06T07:27:31.699273+0000","last_peered":"2024-03-06T07:27:31.699273+0000","last_clean":"2024-03-06T07:27:31.699273+0000","last_became_active":"2024-03-06T07:27:30.244266+0000","last_became_peered":"2024-03-06T07:27:30.244266+0000","last_unstale":"2024-03-06T07:27:31.699273+0000","last_undegraded":"2024-03-06T07:27:31.699273+0000","last_fullsized":"2024-03-06T07:27:31.699273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:10.823443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242632+0000","last_change":"2024-03-06T07:27:29.224405+0000","last_active":"2024-03-06T07:27:31.242632+0000","last_peered":"2024-03-06T07:27:31.242632+0000","last_clean":"2024-03-06T07:27:31.242632+0000","last_became_active":"2024-03-06T07:27:29.223485+0000","last_became_peered":"2024-03-06T07:27:29.223485+0000","last_unstale":"2024-03-06T07:27:31.242632+0000","last_undegraded":"2024-03-06T07:27:31.242632+0000","last_fullsized":"2024-03-06T07:27:31.242632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634481+0000","last_change":"2024-03-06T07:27:30.249021+0000","last_active":"2024-03-06T07:27:31.634481+0000","last_peered":"2024-03-06T07:27:31.634481+0000","last_clean":"2024-03-06T07:27:31.634481+0000","last_became_active":"2024-03-06T07:27:30.248148+0000","last_became_peered":"2024-03-06T07:27:30.248148+0000","last_unstale":"2024-03-06T07:27:31.634481+0000","last_undegraded":"2024-03-06T07:27:31.634481+0000","last_fullsized":"2024-03-06T07:27:31.634481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243028+0000","last_change":"2024-03-06T07:27:29.230274+0000","last_active":"2024-03-06T07:27:31.243028+0000","last_peered":"2024-03-06T07:27:31.243028+0000","last_clean":"2024-03-06T07:27:31.243028+0000","last_became_active":"2024-03-06T07:27:29.230009+0000","last_became_peered":"2024-03-06T07:27:29.230009+0000","last_unstale":"2024-03-06T07:27:31.243028+0000","last_undegraded":"2024-03-06T07:27:31.243028+0000","last_fullsized":"2024-03-06T07:27:31.243028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241766+0000","last_change":"2024-03-06T07:27:29.221872+0000","last_active":"2024-03-06T07:27:31.241766+0000","last_peered":"2024-03-06T07:27:31.241766+0000","last_clean":"2024-03-06T07:27:31.241766+0000","last_became_active":"2024-03-06T07:27:29.221430+0000","last_became_peered":"2024-03-06T07:27:29.221430+0000","last_unstale":"2024-03-06T07:27:31.241766+0000","last_undegraded":"2024-03-06T07:27:31.241766+0000","last_fullsized":"2024-03-06T07:27:31.241766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241901+0000","last_change":"2024-03-06T07:27:30.235213+0000","last_active":"2024-03-06T07:27:31.241901+0000","last_peered":"2024-03-06T07:27:31.241901+0000","last_clean":"2024-03-06T07:27:31.241901+0000","last_became_active":"2024-03-06T07:27:30.234891+0000","last_became_peered":"2024-03-06T07:27:30.234891+0000","last_unstale":"2024-03-06T07:27:31.241901+0000","last_undegraded":"2024-03-06T07:27:31.241901+0000","last_fullsized":"2024-03-06T07:27:31.241901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:44.036284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696609+0000","last_change":"2024-03-06T07:27:29.228355+0000","last_active":"2024-03-06T07:27:31.696609+0000","last_peered":"2024-03-06T07:27:31.696609+0000","last_clean":"2024-03-06T07:27:31.696609+0000","last_became_active":"2024-03-06T07:27:29.228054+0000","last_became_peered":"2024-03-06T07:27:29.228054+0000","last_unstale":"2024-03-06T07:27:31.696609+0000","last_undegraded":"2024-03-06T07:27:31.696609+0000","last_fullsized":"2024-03-06T07:27:31.696609+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240414+0000","last_change":"2024-03-06T07:27:30.236691+0000","last_active":"2024-03-06T07:27:31.240414+0000","last_peered":"2024-03-06T07:27:31.240414+0000","last_clean":"2024-03-06T07:27:31.240414+0000","last_became_active":"2024-03-06T07:27:30.236318+0000","last_became_peered":"2024-03-06T07:27:30.236318+0000","last_unstale":"2024-03-06T07:27:31.240414+0000","last_undegraded":"2024-03-06T07:27:31.240414+0000","last_fullsized":"2024-03-06T07:27:31.240414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:06.911342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242168+0000","last_change":"2024-03-06T07:27:29.218901+0000","last_active":"2024-03-06T07:27:31.242168+0000","last_peered":"2024-03-06T07:27:31.242168+0000","last_clean":"2024-03-06T07:27:31.242168+0000","last_became_active":"2024-03-06T07:27:29.218549+0000","last_became_peered":"2024-03-06T07:27:29.218549+0000","last_unstale":"2024-03-06T07:27:31.242168+0000","last_undegraded":"2024-03-06T07:27:31.242168+0000","last_fullsized":"2024-03-06T07:27:31.242168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697521+0000","last_change":"2024-03-06T07:27:30.269139+0000","last_active":"2024-03-06T07:27:31.697521+0000","last_peered":"2024-03-06T07:27:31.697521+0000","last_clean":"2024-03-06T07:27:31.697521+0000","last_became_active":"2024-03-06T07:27:30.265940+0000","last_became_peered":"2024-03-06T07:27:30.265940+0000","last_unstale":"2024-03-06T07:27:31.697521+0000","last_undegraded":"2024-03-06T07:27:31.697521+0000","last_fullsized":"2024-03-06T07:27:31.697521+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:13.515252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697588+0000","last_change":"2024-03-06T07:27:29.224828+0000","last_active":"2024-03-06T07:27:31.697588+0000","last_peered":"2024-03-06T07:27:31.697588+0000","last_clean":"2024-03-06T07:27:31.697588+0000","last_became_active":"2024-03-06T07:27:29.223899+0000","last_became_peered":"2024-03-06T07:27:29.223899+0000","last_unstale":"2024-03-06T07:27:31.697588+0000","last_undegraded":"2024-03-06T07:27:31.697588+0000","last_fullsized":"2024-03-06T07:27:31.697588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635385+0000","last_change":"2024-03-06T07:27:30.258895+0000","last_active":"2024-03-06T07:27:31.635385+0000","last_peered":"2024-03-06T07:27:31.635385+0000","last_clean":"2024-03-06T07:27:31.635385+0000","last_became_active":"2024-03-06T07:27:30.258204+0000","last_became_peered":"2024-03-06T07:27:30.258204+0000","last_unstale":"2024-03-06T07:27:31.635385+0000","last_undegraded":"2024-03-06T07:27:31.635385+0000","last_fullsized":"2024-03-06T07:27:31.635385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:42.088931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239843+0000","last_change":"2024-03-06T07:27:29.232043+0000","last_active":"2024-03-06T07:27:31.239843+0000","last_peered":"2024-03-06T07:27:31.239843+0000","last_clean":"2024-03-06T07:27:31.239843+0000","last_became_active":"2024-03-06T07:27:29.231659+0000","last_became_peered":"2024-03-06T07:27:29.231659+0000","last_unstale":"2024-03-06T07:27:31.239843+0000","last_undegraded":"2024-03-06T07:27:31.239843+0000","last_fullsized":"2024-03-06T07:27:31.239843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700788+0000","last_change":"2024-03-06T07:27:30.253450+0000","last_active":"2024-03-06T07:27:31.700788+0000","last_peered":"2024-03-06T07:27:31.700788+0000","last_clean":"2024-03-06T07:27:31.700788+0000","last_became_active":"2024-03-06T07:27:30.250531+0000","last_became_peered":"2024-03-06T07:27:30.250531+0000","last_unstale":"2024-03-06T07:27:31.700788+0000","last_undegraded":"2024-03-06T07:27:31.700788+0000","last_fullsized":"2024-03-06T07:27:31.700788+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:46.446447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242917+0000","last_change":"2024-03-06T07:27:29.216258+0000","last_active":"2024-03-06T07:27:31.242917+0000","last_peered":"2024-03-06T07:27:31.242917+0000","last_clean":"2024-03-06T07:27:31.242917+0000","last_became_active":"2024-03-06T07:27:29.215924+0000","last_became_peered":"2024-03-06T07:27:29.215924+0000","last_unstale":"2024-03-06T07:27:31.242917+0000","last_undegraded":"2024-03-06T07:27:31.242917+0000","last_fullsized":"2024-03-06T07:27:31.242917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:47.480925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697945+0000","last_change":"2024-03-06T07:27:30.248670+0000","last_active":"2024-03-06T07:27:31.697945+0000","last_peered":"2024-03-06T07:27:31.697945+0000","last_clean":"2024-03-06T07:27:31.697945+0000","last_became_active":"2024-03-06T07:27:30.248398+0000","last_became_peered":"2024-03-06T07:27:30.248398+0000","last_unstale":"2024-03-06T07:27:31.697945+0000","last_undegraded":"2024-03-06T07:27:31.697945+0000","last_fullsized":"2024-03-06T07:27:31.697945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:06.911342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634619+0000","last_change":"2024-03-06T07:27:29.228193+0000","last_active":"2024-03-06T07:27:31.634619+0000","last_peered":"2024-03-06T07:27:31.634619+0000","last_clean":"2024-03-06T07:27:31.634619+0000","last_became_active":"2024-03-06T07:27:29.227840+0000","last_became_peered":"2024-03-06T07:27:29.227840+0000","last_unstale":"2024-03-06T07:27:31.634619+0000","last_undegraded":"2024-03-06T07:27:31.634619+0000","last_fullsized":"2024-03-06T07:27:31.634619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:21.775657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634157+0000","last_change":"2024-03-06T07:27:30.240692+0000","last_active":"2024-03-06T07:27:31.634157+0000","last_peered":"2024-03-06T07:27:31.634157+0000","last_clean":"2024-03-06T07:27:31.634157+0000","last_became_active":"2024-03-06T07:27:30.240410+0000","last_became_peered":"2024-03-06T07:27:30.240410+0000","last_unstale":"2024-03-06T07:27:31.634157+0000","last_undegraded":"2024-03-06T07:27:31.634157+0000","last_fullsized":"2024-03-06T07:27:31.634157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:44.205908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696997+0000","last_change":"2024-03-06T07:27:29.231958+0000","last_active":"2024-03-06T07:27:31.696997+0000","last_peered":"2024-03-06T07:27:31.696997+0000","last_clean":"2024-03-06T07:27:31.696997+0000","last_became_active":"2024-03-06T07:27:29.231690+0000","last_became_peered":"2024-03-06T07:27:29.231690+0000","last_unstale":"2024-03-06T07:27:31.696997+0000","last_undegraded":"2024-03-06T07:27:31.696997+0000","last_fullsized":"2024-03-06T07:27:31.696997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:45.764336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700469+0000","last_change":"2024-03-06T07:27:30.258434+0000","last_active":"2024-03-06T07:27:31.700469+0000","last_peered":"2024-03-06T07:27:31.700469+0000","last_clean":"2024-03-06T07:27:31.700469+0000","last_became_active":"2024-03-06T07:27:30.258094+0000","last_became_peered":"2024-03-06T07:27:30.258094+0000","last_unstale":"2024-03-06T07:27:31.700469+0000","last_undegraded":"2024-03-06T07:27:31.700469+0000","last_fullsized":"2024-03-06T07:27:31.700469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:58.740494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241544+0000","last_change":"2024-03-06T07:27:29.231347+0000","last_active":"2024-03-06T07:27:31.241544+0000","last_peered":"2024-03-06T07:27:31.241544+0000","last_clean":"2024-03-06T07:27:31.241544+0000","last_became_active":"2024-03-06T07:27:29.230852+0000","last_became_peered":"2024-03-06T07:27:29.230852+0000","last_unstale":"2024-03-06T07:27:31.241544+0000","last_undegraded":"2024-03-06T07:27:31.241544+0000","last_fullsized":"2024-03-06T07:27:31.241544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:29.471770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242563+0000","last_change":"2024-03-06T07:27:30.240275+0000","last_active":"2024-03-06T07:27:31.242563+0000","last_peered":"2024-03-06T07:27:31.242563+0000","last_clean":"2024-03-06T07:27:31.242563+0000","last_became_active":"2024-03-06T07:27:30.239945+0000","last_became_peered":"2024-03-06T07:27:30.239945+0000","last_unstale":"2024-03-06T07:27:31.242563+0000","last_undegraded":"2024-03-06T07:27:31.242563+0000","last_fullsized":"2024-03-06T07:27:31.242563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695318+0000","last_change":"2024-03-06T07:27:29.217898+0000","last_active":"2024-03-06T07:27:31.695318+0000","last_peered":"2024-03-06T07:27:31.695318+0000","last_clean":"2024-03-06T07:27:31.695318+0000","last_became_active":"2024-03-06T07:27:29.217605+0000","last_became_peered":"2024-03-06T07:27:29.217605+0000","last_unstale":"2024-03-06T07:27:31.695318+0000","last_undegraded":"2024-03-06T07:27:31.695318+0000","last_fullsized":"2024-03-06T07:27:31.695318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.683045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698824+0000","last_change":"2024-03-06T07:27:30.258444+0000","last_active":"2024-03-06T07:27:31.698824+0000","last_peered":"2024-03-06T07:27:31.698824+0000","last_clean":"2024-03-06T07:27:31.698824+0000","last_became_active":"2024-03-06T07:27:30.258119+0000","last_became_peered":"2024-03-06T07:27:30.258119+0000","last_unstale":"2024-03-06T07:27:31.698824+0000","last_undegraded":"2024-03-06T07:27:31.698824+0000","last_fullsized":"2024-03-06T07:27:31.698824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:52.196155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699271+0000","last_change":"2024-03-06T07:27:29.228474+0000","last_active":"2024-03-06T07:27:31.699271+0000","last_peered":"2024-03-06T07:27:31.699271+0000","last_clean":"2024-03-06T07:27:31.699271+0000","last_became_active":"2024-03-06T07:27:29.228168+0000","last_became_peered":"2024-03-06T07:27:29.228168+0000","last_unstale":"2024-03-06T07:27:31.699271+0000","last_undegraded":"2024-03-06T07:27:31.699271+0000","last_fullsized":"2024-03-06T07:27:31.699271+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:36.193215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699323+0000","last_change":"2024-03-06T07:27:30.243658+0000","last_active":"2024-03-06T07:27:31.699323+0000","last_peered":"2024-03-06T07:27:31.699323+0000","last_clean":"2024-03-06T07:27:31.699323+0000","last_became_active":"2024-03-06T07:27:30.243145+0000","last_became_peered":"2024-03-06T07:27:30.243145+0000","last_unstale":"2024-03-06T07:27:31.699323+0000","last_undegraded":"2024-03-06T07:27:31.699323+0000","last_fullsized":"2024-03-06T07:27:31.699323+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:57:10.001805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695709+0000","last_change":"2024-03-06T07:27:29.221215+0000","last_active":"2024-03-06T07:27:31.695709+0000","last_peered":"2024-03-06T07:27:31.695709+0000","last_clean":"2024-03-06T07:27:31.695709+0000","last_became_active":"2024-03-06T07:27:29.220256+0000","last_became_peered":"2024-03-06T07:27:29.220256+0000","last_unstale":"2024-03-06T07:27:31.695709+0000","last_undegraded":"2024-03-06T07:27:31.695709+0000","last_fullsized":"2024-03-06T07:27:31.695709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699734+0000","last_change":"2024-03-06T07:27:30.258256+0000","last_active":"2024-03-06T07:27:31.699734+0000","last_peered":"2024-03-06T07:27:31.699734+0000","last_clean":"2024-03-06T07:27:31.699734+0000","last_became_active":"2024-03-06T07:27:30.255209+0000","last_became_peered":"2024-03-06T07:27:30.255209+0000","last_unstale":"2024-03-06T07:27:31.699734+0000","last_undegraded":"2024-03-06T07:27:31.699734+0000","last_fullsized":"2024-03-06T07:27:31.699734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:11.942117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240185+0000","last_change":"2024-03-06T07:27:29.225919+0000","last_active":"2024-03-06T07:27:31.240185+0000","last_peered":"2024-03-06T07:27:31.240185+0000","last_clean":"2024-03-06T07:27:31.240185+0000","last_became_active":"2024-03-06T07:27:29.225690+0000","last_became_peered":"2024-03-06T07:27:29.225690+0000","last_unstale":"2024-03-06T07:27:31.240185+0000","last_undegraded":"2024-03-06T07:27:31.240185+0000","last_fullsized":"2024-03-06T07:27:31.240185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698122+0000","last_change":"2024-03-06T07:27:30.259916+0000","last_active":"2024-03-06T07:27:31.698122+0000","last_peered":"2024-03-06T07:27:31.698122+0000","last_clean":"2024-03-06T07:27:31.698122+0000","last_became_active":"2024-03-06T07:27:30.259100+0000","last_became_peered":"2024-03-06T07:27:30.259100+0000","last_unstale":"2024-03-06T07:27:31.698122+0000","last_undegraded":"2024-03-06T07:27:31.698122+0000","last_fullsized":"2024-03-06T07:27:31.698122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:19.907027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695658+0000","last_change":"2024-03-06T07:27:29.231688+0000","last_active":"2024-03-06T07:27:31.695658+0000","last_peered":"2024-03-06T07:27:31.695658+0000","last_clean":"2024-03-06T07:27:31.695658+0000","last_became_active":"2024-03-06T07:27:29.230816+0000","last_became_peered":"2024-03-06T07:27:29.230816+0000","last_unstale":"2024-03-06T07:27:31.695658+0000","last_undegraded":"2024-03-06T07:27:31.695658+0000","last_fullsized":"2024-03-06T07:27:31.695658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:26:51.657884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698198+0000","last_change":"2024-03-06T07:27:30.264213+0000","last_active":"2024-03-06T07:27:31.698198+0000","last_peered":"2024-03-06T07:27:31.698198+0000","last_clean":"2024-03-06T07:27:31.698198+0000","last_became_active":"2024-03-06T07:27:30.263986+0000","last_became_peered":"2024-03-06T07:27:30.263986+0000","last_unstale":"2024-03-06T07:27:31.698198+0000","last_undegraded":"2024-03-06T07:27:31.698198+0000","last_fullsized":"2024-03-06T07:27:31.698198+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:14.580315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":170,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698115+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:27:31.698115+0000","last_peered":"2024-03-06T07:27:31.698115+0000","last_clean":"2024-03-06T07:27:31.698115+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:27:31.698115+0000","last_undegraded":"2024-03-06T07:27:31.698115+0000","last_fullsized":"2024-03-06T07:27:31.698115+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.696277+0000","last_change":"2024-03-06T07:27:29.232294+0000","last_active":"2024-03-06T07:27:31.696277+0000","last_peered":"2024-03-06T07:27:31.696277+0000","last_clean":"2024-03-06T07:27:31.696277+0000","last_became_active":"2024-03-06T07:27:29.231839+0000","last_became_peered":"2024-03-06T07:27:29.231839+0000","last_unstale":"2024-03-06T07:27:31.696277+0000","last_undegraded":"2024-03-06T07:27:31.696277+0000","last_fullsized":"2024-03-06T07:27:31.696277+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:46.899765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.241928+0000","last_change":"2024-03-06T07:27:30.245626+0000","last_active":"2024-03-06T07:27:31.241928+0000","last_peered":"2024-03-06T07:27:31.241928+0000","last_clean":"2024-03-06T07:27:31.241928+0000","last_became_active":"2024-03-06T07:27:30.245324+0000","last_became_peered":"2024-03-06T07:27:30.245324+0000","last_unstale":"2024-03-06T07:27:31.241928+0000","last_undegraded":"2024-03-06T07:27:31.241928+0000","last_fullsized":"2024-03-06T07:27:31.241928+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:18.452094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697983+0000","last_change":"2024-03-06T07:27:29.234333+0000","last_active":"2024-03-06T07:27:31.697983+0000","last_peered":"2024-03-06T07:27:31.697983+0000","last_clean":"2024-03-06T07:27:31.697983+0000","last_became_active":"2024-03-06T07:27:29.234024+0000","last_became_peered":"2024-03-06T07:27:29.234024+0000","last_unstale":"2024-03-06T07:27:31.697983+0000","last_undegraded":"2024-03-06T07:27:31.697983+0000","last_fullsized":"2024-03-06T07:27:31.697983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:58.997396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.634297+0000","last_change":"2024-03-06T07:27:30.254664+0000","last_active":"2024-03-06T07:27:31.634297+0000","last_peered":"2024-03-06T07:27:31.634297+0000","last_clean":"2024-03-06T07:27:31.634297+0000","last_became_active":"2024-03-06T07:27:30.254371+0000","last_became_peered":"2024-03-06T07:27:30.254371+0000","last_unstale":"2024-03-06T07:27:31.634297+0000","last_undegraded":"2024-03-06T07:27:31.634297+0000","last_fullsized":"2024-03-06T07:27:31.634297+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:03.061047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.695651+0000","last_change":"2024-03-06T07:27:29.220564+0000","last_active":"2024-03-06T07:27:31.695651+0000","last_peered":"2024-03-06T07:27:31.695651+0000","last_clean":"2024-03-06T07:27:31.695651+0000","last_became_active":"2024-03-06T07:27:29.220231+0000","last_became_peered":"2024-03-06T07:27:29.220231+0000","last_unstale":"2024-03-06T07:27:31.695651+0000","last_undegraded":"2024-03-06T07:27:31.695651+0000","last_fullsized":"2024-03-06T07:27:31.695651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:18.751461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.238983+0000","last_change":"2024-03-06T07:27:30.244146+0000","last_active":"2024-03-06T07:27:31.238983+0000","last_peered":"2024-03-06T07:27:31.238983+0000","last_clean":"2024-03-06T07:27:31.238983+0000","last_became_active":"2024-03-06T07:27:30.241811+0000","last_became_peered":"2024-03-06T07:27:30.241811+0000","last_unstale":"2024-03-06T07:27:31.238983+0000","last_undegraded":"2024-03-06T07:27:31.238983+0000","last_fullsized":"2024-03-06T07:27:31.238983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:52.703156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697062+0000","last_change":"2024-03-06T07:27:29.221902+0000","last_active":"2024-03-06T07:27:31.697062+0000","last_peered":"2024-03-06T07:27:31.697062+0000","last_clean":"2024-03-06T07:27:31.697062+0000","last_became_active":"2024-03-06T07:27:29.221598+0000","last_became_peered":"2024-03-06T07:27:29.221598+0000","last_unstale":"2024-03-06T07:27:31.697062+0000","last_undegraded":"2024-03-06T07:27:31.697062+0000","last_fullsized":"2024-03-06T07:27:31.697062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700176+0000","last_change":"2024-03-06T07:27:30.243366+0000","last_active":"2024-03-06T07:27:31.700176+0000","last_peered":"2024-03-06T07:27:31.700176+0000","last_clean":"2024-03-06T07:27:31.700176+0000","last_became_active":"2024-03-06T07:27:30.243054+0000","last_became_peered":"2024-03-06T07:27:30.243054+0000","last_unstale":"2024-03-06T07:27:31.700176+0000","last_undegraded":"2024-03-06T07:27:31.700176+0000","last_fullsized":"2024-03-06T07:27:31.700176+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:29:00.207295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239297+0000","last_change":"2024-03-06T07:27:29.220848+0000","last_active":"2024-03-06T07:27:31.239297+0000","last_peered":"2024-03-06T07:27:31.239297+0000","last_clean":"2024-03-06T07:27:31.239297+0000","last_became_active":"2024-03-06T07:27:29.220557+0000","last_became_peered":"2024-03-06T07:27:29.220557+0000","last_unstale":"2024-03-06T07:27:31.239297+0000","last_undegraded":"2024-03-06T07:27:31.239297+0000","last_fullsized":"2024-03-06T07:27:31.239297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698368+0000","last_change":"2024-03-06T07:27:30.259510+0000","last_active":"2024-03-06T07:27:31.698368+0000","last_peered":"2024-03-06T07:27:31.698368+0000","last_clean":"2024-03-06T07:27:31.698368+0000","last_became_active":"2024-03-06T07:27:30.258936+0000","last_became_peered":"2024-03-06T07:27:30.258936+0000","last_unstale":"2024-03-06T07:27:31.698368+0000","last_undegraded":"2024-03-06T07:27:31.698368+0000","last_fullsized":"2024-03-06T07:27:31.698368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:24.977160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698863+0000","last_change":"2024-03-06T07:27:29.237638+0000","last_active":"2024-03-06T07:27:31.698863+0000","last_peered":"2024-03-06T07:27:31.698863+0000","last_clean":"2024-03-06T07:27:31.698863+0000","last_became_active":"2024-03-06T07:27:29.237304+0000","last_became_peered":"2024-03-06T07:27:29.237304+0000","last_unstale":"2024-03-06T07:27:31.698863+0000","last_undegraded":"2024-03-06T07:27:31.698863+0000","last_fullsized":"2024-03-06T07:27:31.698863+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:09.810603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.240309+0000","last_change":"2024-03-06T07:27:30.241439+0000","last_active":"2024-03-06T07:27:31.240309+0000","last_peered":"2024-03-06T07:27:31.240309+0000","last_clean":"2024-03-06T07:27:31.240309+0000","last_became_active":"2024-03-06T07:27:30.241163+0000","last_became_peered":"2024-03-06T07:27:30.241163+0000","last_unstale":"2024-03-06T07:27:31.240309+0000","last_undegraded":"2024-03-06T07:27:31.240309+0000","last_fullsized":"2024-03-06T07:27:31.240309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:01.744486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.253333+0000","last_change":"2024-03-06T07:27:29.223056+0000","last_active":"2024-03-06T07:27:31.253333+0000","last_peered":"2024-03-06T07:27:31.253333+0000","last_clean":"2024-03-06T07:27:31.253333+0000","last_became_active":"2024-03-06T07:27:29.222744+0000","last_became_peered":"2024-03-06T07:27:29.222744+0000","last_unstale":"2024-03-06T07:27:31.253333+0000","last_undegraded":"2024-03-06T07:27:31.253333+0000","last_fullsized":"2024-03-06T07:27:31.253333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243861+0000","last_change":"2024-03-06T07:27:30.241735+0000","last_active":"2024-03-06T07:27:31.243861+0000","last_peered":"2024-03-06T07:27:31.243861+0000","last_clean":"2024-03-06T07:27:31.243861+0000","last_became_active":"2024-03-06T07:27:30.241436+0000","last_became_peered":"2024-03-06T07:27:30.241436+0000","last_unstale":"2024-03-06T07:27:31.243861+0000","last_undegraded":"2024-03-06T07:27:31.243861+0000","last_fullsized":"2024-03-06T07:27:31.243861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:53:58.613904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242269+0000","last_change":"2024-03-06T07:27:29.226601+0000","last_active":"2024-03-06T07:27:31.242269+0000","last_peered":"2024-03-06T07:27:31.242269+0000","last_clean":"2024-03-06T07:27:31.242269+0000","last_became_active":"2024-03-06T07:27:29.225390+0000","last_became_peered":"2024-03-06T07:27:29.225390+0000","last_unstale":"2024-03-06T07:27:31.242269+0000","last_undegraded":"2024-03-06T07:27:31.242269+0000","last_fullsized":"2024-03-06T07:27:31.242269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:21.842739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635023+0000","last_change":"2024-03-06T07:27:30.242063+0000","last_active":"2024-03-06T07:27:31.635023+0000","last_peered":"2024-03-06T07:27:31.635023+0000","last_clean":"2024-03-06T07:27:31.635023+0000","last_became_active":"2024-03-06T07:27:30.241737+0000","last_became_peered":"2024-03-06T07:27:30.241737+0000","last_unstale":"2024-03-06T07:27:31.635023+0000","last_undegraded":"2024-03-06T07:27:31.635023+0000","last_fullsized":"2024-03-06T07:27:31.635023+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697825+0000","last_change":"2024-03-06T07:27:29.235266+0000","last_active":"2024-03-06T07:27:31.697825+0000","last_peered":"2024-03-06T07:27:31.697825+0000","last_clean":"2024-03-06T07:27:31.697825+0000","last_became_active":"2024-03-06T07:27:29.234878+0000","last_became_peered":"2024-03-06T07:27:29.234878+0000","last_unstale":"2024-03-06T07:27:31.697825+0000","last_undegraded":"2024-03-06T07:27:31.697825+0000","last_fullsized":"2024-03-06T07:27:31.697825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698617+0000","last_change":"2024-03-06T07:27:30.250799+0000","last_active":"2024-03-06T07:27:31.698617+0000","last_peered":"2024-03-06T07:27:31.698617+0000","last_clean":"2024-03-06T07:27:31.698617+0000","last_became_active":"2024-03-06T07:27:30.250475+0000","last_became_peered":"2024-03-06T07:27:30.250475+0000","last_unstale":"2024-03-06T07:27:31.698617+0000","last_undegraded":"2024-03-06T07:27:31.698617+0000","last_fullsized":"2024-03-06T07:27:31.698617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:47.520042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239430+0000","last_change":"2024-03-06T07:27:29.224306+0000","last_active":"2024-03-06T07:27:31.239430+0000","last_peered":"2024-03-06T07:27:31.239430+0000","last_clean":"2024-03-06T07:27:31.239430+0000","last_became_active":"2024-03-06T07:27:29.223923+0000","last_became_peered":"2024-03-06T07:27:29.223923+0000","last_unstale":"2024-03-06T07:27:31.239430+0000","last_undegraded":"2024-03-06T07:27:31.239430+0000","last_fullsized":"2024-03-06T07:27:31.239430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:59.194456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242506+0000","last_change":"2024-03-06T07:27:30.246857+0000","last_active":"2024-03-06T07:27:31.242506+0000","last_peered":"2024-03-06T07:27:31.242506+0000","last_clean":"2024-03-06T07:27:31.242506+0000","last_became_active":"2024-03-06T07:27:30.246519+0000","last_became_peered":"2024-03-06T07:27:30.246519+0000","last_unstale":"2024-03-06T07:27:31.242506+0000","last_undegraded":"2024-03-06T07:27:31.242506+0000","last_fullsized":"2024-03-06T07:27:31.242506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:16:59.134577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.239222+0000","last_change":"2024-03-06T07:27:29.217568+0000","last_active":"2024-03-06T07:27:31.239222+0000","last_peered":"2024-03-06T07:27:31.239222+0000","last_clean":"2024-03-06T07:27:31.239222+0000","last_became_active":"2024-03-06T07:27:29.217204+0000","last_became_peered":"2024-03-06T07:27:29.217204+0000","last_unstale":"2024-03-06T07:27:31.239222+0000","last_undegraded":"2024-03-06T07:27:31.239222+0000","last_fullsized":"2024-03-06T07:27:31.239222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242343+0000","last_change":"2024-03-06T07:27:30.244509+0000","last_active":"2024-03-06T07:27:31.242343+0000","last_peered":"2024-03-06T07:27:31.242343+0000","last_clean":"2024-03-06T07:27:31.242343+0000","last_became_active":"2024-03-06T07:27:30.243804+0000","last_became_peered":"2024-03-06T07:27:30.243804+0000","last_unstale":"2024-03-06T07:27:31.242343+0000","last_undegraded":"2024-03-06T07:27:31.242343+0000","last_fullsized":"2024-03-06T07:27:31.242343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:32.877653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.700592+0000","last_change":"2024-03-06T07:27:29.235464+0000","last_active":"2024-03-06T07:27:31.700592+0000","last_peered":"2024-03-06T07:27:31.700592+0000","last_clean":"2024-03-06T07:27:31.700592+0000","last_became_active":"2024-03-06T07:27:29.234532+0000","last_became_peered":"2024-03-06T07:27:29.234532+0000","last_unstale":"2024-03-06T07:27:31.700592+0000","last_undegraded":"2024-03-06T07:27:31.700592+0000","last_fullsized":"2024-03-06T07:27:31.700592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:42:45.433607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.242177+0000","last_change":"2024-03-06T07:27:30.257067+0000","last_active":"2024-03-06T07:27:31.242177+0000","last_peered":"2024-03-06T07:27:31.242177+0000","last_clean":"2024-03-06T07:27:31.242177+0000","last_became_active":"2024-03-06T07:27:30.256842+0000","last_became_peered":"2024-03-06T07:27:30.256842+0000","last_unstale":"2024-03-06T07:27:31.242177+0000","last_undegraded":"2024-03-06T07:27:31.242177+0000","last_fullsized":"2024-03-06T07:27:31.242177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:23:19.075851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697945+0000","last_change":"2024-03-06T07:27:30.266494+0000","last_active":"2024-03-06T07:27:31.697945+0000","last_peered":"2024-03-06T07:27:31.697945+0000","last_clean":"2024-03-06T07:27:31.697945+0000","last_became_active":"2024-03-06T07:27:30.265258+0000","last_became_peered":"2024-03-06T07:27:30.265258+0000","last_unstale":"2024-03-06T07:27:31.697945+0000","last_undegraded":"2024-03-06T07:27:31.697945+0000","last_fullsized":"2024-03-06T07:27:31.697945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:55.013108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.699044+0000","last_change":"2024-03-06T07:27:29.235012+0000","last_active":"2024-03-06T07:27:31.699044+0000","last_peered":"2024-03-06T07:27:31.699044+0000","last_clean":"2024-03-06T07:27:31.699044+0000","last_became_active":"2024-03-06T07:27:29.234638+0000","last_became_peered":"2024-03-06T07:27:29.234638+0000","last_unstale":"2024-03-06T07:27:31.699044+0000","last_undegraded":"2024-03-06T07:27:31.699044+0000","last_fullsized":"2024-03-06T07:27:31.699044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:58.295551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.697715+0000","last_change":"2024-03-06T07:27:30.265713+0000","last_active":"2024-03-06T07:27:31.697715+0000","last_peered":"2024-03-06T07:27:31.697715+0000","last_clean":"2024-03-06T07:27:31.697715+0000","last_became_active":"2024-03-06T07:27:30.259972+0000","last_became_peered":"2024-03-06T07:27:30.259972+0000","last_unstale":"2024-03-06T07:27:31.697715+0000","last_undegraded":"2024-03-06T07:27:31.697715+0000","last_fullsized":"2024-03-06T07:27:31.697715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:12.658478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.698343+0000","last_change":"2024-03-06T07:27:29.236520+0000","last_active":"2024-03-06T07:27:31.698343+0000","last_peered":"2024-03-06T07:27:31.698343+0000","last_clean":"2024-03-06T07:27:31.698343+0000","last_became_active":"2024-03-06T07:27:29.235298+0000","last_became_peered":"2024-03-06T07:27:29.235298+0000","last_unstale":"2024-03-06T07:27:31.698343+0000","last_undegraded":"2024-03-06T07:27:31.698343+0000","last_fullsized":"2024-03-06T07:27:31.698343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:46.507202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635179+0000","last_change":"2024-03-06T07:27:30.258866+0000","last_active":"2024-03-06T07:27:31.635179+0000","last_peered":"2024-03-06T07:27:31.635179+0000","last_clean":"2024-03-06T07:27:31.635179+0000","last_became_active":"2024-03-06T07:27:30.258557+0000","last_became_peered":"2024-03-06T07:27:30.258557+0000","last_unstale":"2024-03-06T07:27:31.635179+0000","last_undegraded":"2024-03-06T07:27:31.635179+0000","last_fullsized":"2024-03-06T07:27:31.635179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:31.100344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243647+0000","last_change":"2024-03-06T07:27:29.221773+0000","last_active":"2024-03-06T07:27:31.243647+0000","last_peered":"2024-03-06T07:27:31.243647+0000","last_clean":"2024-03-06T07:27:31.243647+0000","last_became_active":"2024-03-06T07:27:29.221365+0000","last_became_peered":"2024-03-06T07:27:29.221365+0000","last_unstale":"2024-03-06T07:27:31.243647+0000","last_undegraded":"2024-03-06T07:27:31.243647+0000","last_fullsized":"2024-03-06T07:27:31.243647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:15.408581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.243868+0000","last_change":"2024-03-06T07:27:30.245108+0000","last_active":"2024-03-06T07:27:31.243868+0000","last_peered":"2024-03-06T07:27:31.243868+0000","last_clean":"2024-03-06T07:27:31.243868+0000","last_became_active":"2024-03-06T07:27:30.244795+0000","last_became_peered":"2024-03-06T07:27:30.244795+0000","last_unstale":"2024-03-06T07:27:31.243868+0000","last_undegraded":"2024-03-06T07:27:31.243868+0000","last_fullsized":"2024-03-06T07:27:31.243868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:29.180291+0000","last_clean_scrub_stamp":"2024-03-06T07:27:29.180291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:13:03.111110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-06T07:27:31.635325+0000","last_change":"2024-03-06T07:27:29.229397+0000","last_active":"2024-03-06T07:27:31.635325+0000","last_peered":"2024-03-06T07:27:31.635325+0000","last_clean":"2024-03-06T07:27:31.635325+0000","last_became_active":"2024-03-06T07:27:29.229097+0000","last_became_peered":"2024-03-06T07:27:29.229097+0000","last_unstale":"2024-03-06T07:27:31.635325+0000","last_undegraded":"2024-03-06T07:27:31.635325+0000","last_fullsized":"2024-03-06T07:27:31.635325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:28.167451+0000","last_clean_scrub_stamp":"2024-03-06T07:27:28.167451+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:17.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":78126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":78286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":819200,"data_stored":669076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":79738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":225280,"data_stored":78216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":233472,"data_stored":78252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":233472,"data_stored":79738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":827392,"data_stored":669132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:40.942 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:27:40.942 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:27:40.942 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:27:40.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:27:41.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.199+0000 7ffba7200640 1 Processor -- start 2024-03-06T07:27:41.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.200+0000 7ffba7200640 1 -- start start 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.201+0000 7ffba7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ffba815dff0 0x7ffba80ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.201+0000 7ffba7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffba80cf6d0 0x7ffba80ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.201+0000 7ffba7200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba815e3d0 con 0x7ffba80cd400 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.201+0000 7ffba7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba80cdca0 con 0x7ffba80cf6d0 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.201+0000 7ffba7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba80cd9c0 con 0x7ffba815dff0 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffba80cf6d0 0x7ffba80ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffba80cf6d0 0x7ffba80ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42694/0 (socket says 172.21.15.6:42694) 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 -- 172.21.15.6:0/206033667 learned_addr learned my addr 172.21.15.6:0/206033667 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:41.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5e00640 1 --2- 172.21.15.6:0/206033667 >> v2:172.21.15.178:3300/0 conn(0x7ffba815dff0 0x7ffba80ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 -- 172.21.15.6:0/206033667 >> v2:172.21.15.178:3300/0 conn(0x7ffba815dff0 msgr2=0x7ffba80ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 --2- 172.21.15.6:0/206033667 >> v2:172.21.15.178:3300/0 conn(0x7ffba815dff0 0x7ffba80ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 -- 172.21.15.6:0/206033667 >> v1:172.21.15.178:6789/0 conn(0x7ffba80cd400 legacy=0x7ffba80cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.202+0000 7ffba5400640 1 -- 172.21.15.6:0/206033667 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba806bea0 con 0x7ffba80cf6d0 2024-03-06T07:27:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.203+0000 7ffba5400640 1 --2- 172.21.15.6:0/206033667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba80cf6d0 0x7ffba80ce930 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffba80718c0 tx=0x7ffb9402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7da77313034443 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba4a00640 1 -- 172.21.15.6:0/206033667 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9402c9f0 con 0x7ffba80cf6d0 2024-03-06T07:27:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba4a00640 1 -- 172.21.15.6:0/206033667 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9402cb90 con 0x7ffba80cf6d0 2024-03-06T07:27:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba4a00640 1 -- 172.21.15.6:0/206033667 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb940343f0 con 0x7ffba80cf6d0 2024-03-06T07:27:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba7200640 1 -- 172.21.15.6:0/206033667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba80cf6d0 msgr2=0x7ffba80ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba7200640 1 --2- 172.21.15.6:0/206033667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba80cf6d0 0x7ffba80ce930 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffba80718c0 tx=0x7ffb9402be50 comp rx=0 tx=0).stop 2024-03-06T07:27:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba7200640 1 -- 172.21.15.6:0/206033667 shutdown_connections 2024-03-06T07:27:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba7200640 1 --2- 172.21.15.6:0/206033667 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba80cf6d0 0x7ffba80ce930 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba7200640 1 --2- 172.21.15.6:0/206033667 >> v2:172.21.15.178:3300/0 conn(0x7ffba815dff0 0x7ffba80ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.204+0000 7ffba7200640 1 -- 172.21.15.6:0/206033667 >> 172.21.15.6:0/206033667 conn(0x7ffba80b36d0 msgr2=0x7ffba80b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.205+0000 7ffba7200640 1 -- 172.21.15.6:0/206033667 shutdown_connections 2024-03-06T07:27:41.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.205+0000 7ffba7200640 1 -- 172.21.15.6:0/206033667 wait complete. 2024-03-06T07:27:41.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.206+0000 7ffba7200640 1 Processor -- start 2024-03-06T07:27:41.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.207+0000 7ffba7200640 1 -- start start 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.207+0000 7ffba7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffba80cd400 0x7ffba814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.207+0000 7ffba7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffba80cf6d0 0x7ffba806fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.207+0000 7ffba7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 0x7ffba8070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80cf150 con 0x7ffba815dff0 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba806c460 con 0x7ffba80cd400 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba806d990 con 0x7ffba80cf6d0 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 0x7ffba8070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 0x7ffba8070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42696/0 (socket says 172.21.15.6:42696) 2024-03-06T07:27:41.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 -- 172.21.15.6:0/1772422497 learned_addr learned my addr 172.21.15.6:0/1772422497 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba5400640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffba80cf6d0 0x7ffba806fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba5e00640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffba80cd400 0x7ffba814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 -- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffba80cf6d0 msgr2=0x7ffba806fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffba80cf6d0 0x7ffba806fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 -- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffba80cd400 msgr2=0x7ffba814ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffba80cd400 0x7ffba814ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba6800640 1 -- 172.21.15.6:0/1772422497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba81f7270 con 0x7ffba815dff0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.208+0000 7ffba5400640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffba80cf6d0 0x7ffba806fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.209+0000 7ffba6800640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 0x7ffba8070250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffba000c410 tx=0x7ffba000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.209+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011790 con 0x7ffba815dff0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.209+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba0017070 con 0x7ffba815dff0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.209+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0011bb0 con 0x7ffba815dff0 2024-03-06T07:27:41.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.209+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba81f7550 con 0x7ffba815dff0 2024-03-06T07:27:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.209+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba81f7b10 con 0x7ffba815dff0 2024-03-06T07:27:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.210+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba804f330 con 0x7ffba815dff0 2024-03-06T07:27:41.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.211+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffba001c460 con 0x7ffba815dff0 2024-03-06T07:27:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.212+0000 7ffb9e000640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb8806b500 0x7ffb8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.212+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffba0089df0 con 0x7ffba815dff0 2024-03-06T07:27:41.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.212+0000 7ffba5e00640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb8806b500 0x7ffb8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.213+0000 7ffba5e00640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb8806b500 0x7ffb8806d9c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.216+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffba0021020 con 0x7ffba815dff0 2024-03-06T07:27:41.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.459+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffba8150450 con 0x7ffba815dff0 2024-03-06T07:27:41.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.459+0000 7ffb9e000640 1 -- 172.21.15.6:0/1772422497 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ffba0058c70 con 0x7ffba815dff0 2024-03-06T07:27:41.462 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:41.462 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb8806b500 msgr2=0x7ffb8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb8806b500 0x7ffb8806d9c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb98006fd0 tx=0x7ffb98008040 comp rx=0 tx=0).stop 2024-03-06T07:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 msgr2=0x7ffba8070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 0x7ffba8070250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffba000c410 tx=0x7ffba000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 shutdown_connections 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb8806b500 0x7ffb8806d9c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffba815dff0 0x7ffba8070250 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffba80cf6d0 0x7ffba806fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 --2- 172.21.15.6:0/1772422497 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffba80cd400 0x7ffba814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.462+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 >> 172.21.15.6:0/1772422497 conn(0x7ffba80b36d0 msgr2=0x7ffba80d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.463+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 shutdown_connections 2024-03-06T07:27:41.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.463+0000 7ffba7200640 1 -- 172.21.15.6:0/1772422497 wait complete. 2024-03-06T07:27:41.484 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:27:41.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:41.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.708+0000 7fa8cbe00640 1 Processor -- start 2024-03-06T07:27:41.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.708+0000 7fa8cbe00640 1 -- start start 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8cbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 0x7fa8cc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8cbe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc15ed10 con 0x7fa8cc1552b0 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8cbe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc15eff0 con 0x7fa8cc15e810 2024-03-06T07:27:41.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8cbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc15f2d0 con 0x7fa8cc0cd060 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8caa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8cc1552b0 0x7fa8cc0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37592/0 (socket says 172.21.15.6:37592) 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8caa00640 1 -- 172.21.15.6:0/1549533000 learned_addr learned my addr 172.21.15.6:0/1549533000 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.710+0000 7fa8ca000640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 0x7fa8cc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179609545 0 0) 0x7fa8cc15ed10 con 0x7fa8cc1552b0 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0003520 con 0x7fa8cc1552b0 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1989086134 0 0) 0x7fa8b0003520 con 0x7fa8cc1552b0 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 msgr2=0x7fa8cc15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 0x7fa8cc15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 >> v1:172.21.15.178:6789/0 conn(0x7fa8cc15e810 legacy=0x7fa8cc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc15f5b0 con 0x7fa8cc1552b0 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fa8bc001670 con 0x7fa8cc1552b0 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 >> v1:172.21.15.6:6789/0 conn(0x7fa8cc1552b0 legacy=0x7fa8cc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 0x7fa8b00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0003f60 0x7fa8b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc15f2d0 con 0x7fa8b0004d40 2024-03-06T07:27:41.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000dd60 con 0x7fa8b0003f60 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.711+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000df60 con 0x7fa8cc0cd060 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8cb400640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8caa00640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 0x7fa8b00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8ca000640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0003f60 0x7fa8b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8cb400640 1 -- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 msgr2=0x7fa8b00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8cb400640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0cd060 0x7fa8b00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8cb400640 1 -- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0003f60 msgr2=0x7fa8b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8cb400640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0003f60 0x7fa8b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.712+0000 7fa8cb400640 1 -- 172.21.15.6:0/1549533000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0003520 con 0x7fa8b0004d40 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.713+0000 7fa8cb400640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc15ec40 tx=0x7fa8b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfe934910f7adc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4001a70 con 0x7fa8b0004d40 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4003b70 con 0x7fa8b0004d40 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8c9600640 1 -- 172.21.15.6:0/1549533000 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40050d0 con 0x7fa8b0004d40 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8cbe00640 1 -- 172.21.15.6:0/1549533000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0004d40 msgr2=0x7fa8b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/1549533000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0004d40 0x7fa8b00051c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc15ec40 tx=0x7fa8b0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8cb400640 1 -- 172.21.15.6:0/1549533000 reap_dead start 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8cbe00640 1 -- 172.21.15.6:0/1549533000 shutdown_connections 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8cbe00640 1 -- 172.21.15.6:0/1549533000 >> 172.21.15.6:0/1549533000 conn(0x7fa8cc06b3d0 msgr2=0x7fa8cc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.714+0000 7fa8cbe00640 1 -- 172.21.15.6:0/1549533000 shutdown_connections 2024-03-06T07:27:41.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.715+0000 7fa8cbe00640 1 -- 172.21.15.6:0/1549533000 wait complete. 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.716+0000 7fa8cbe00640 1 Processor -- start 2024-03-06T07:27:41.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.716+0000 7fa8cbe00640 1 -- start start 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.717+0000 7fa8cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 0x7fa8cc1516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.717+0000 7fa8cbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0b4740 0x7fa8cc14fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.717+0000 7fa8cbe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc150100 0x7fa8cc159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.717+0000 7fa8cbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc06dd30 con 0x7fa8cc0b4360 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.717+0000 7fa8cbe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc0b3760 con 0x7fa8cc150100 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.717+0000 7fa8cbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc06deb0 con 0x7fa8cc0b4740 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 0x7fa8cc1516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 0x7fa8cc1516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42710/0 (socket says 172.21.15.6:42710) 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 -- 172.21.15.6:0/393428645 learned_addr learned my addr 172.21.15.6:0/393428645 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8ca000640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0b4740 0x7fa8cc14fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8cb400640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc150100 0x7fa8cc159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 -- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0b4740 msgr2=0x7fa8cc14fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0b4740 0x7fa8cc14fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 -- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc150100 msgr2=0x7fa8cc159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc150100 0x7fa8cc159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 -- 172.21.15.6:0/393428645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc150640 con 0x7fa8cc0b4360 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8ca000640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0b4740 0x7fa8cc14fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.718+0000 7fa8caa00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 0x7fa8cc1516e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc001b80 tx=0x7fa8bc007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.719+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc00f960 con 0x7fa8cc0b4360 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.719+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc018070 con 0x7fa8cc0b4360 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.719+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc00fd80 con 0x7fa8cc0b4360 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.719+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc150920 con 0x7fa8cc0b4360 2024-03-06T07:27:41.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.719+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc150ee0 con 0x7fa8cc0b4360 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.720+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc04f330 con 0x7fa8cc0b4360 2024-03-06T07:27:41.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.721+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8bc01d460 con 0x7fa8cc0b4360 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.722+0000 7fa8c3400640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8ac06b500 0x7fa8ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.722+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8bc08a470 con 0x7fa8cc0b4360 2024-03-06T07:27:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.722+0000 7fa8ca000640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8ac06b500 0x7fa8ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:41.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.724+0000 7fa8ca000640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8ac06b500 0x7fa8ac06d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0b4ed0 tx=0x7fa8b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:41.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.727+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8bc059250 con 0x7fa8cc0b4360 2024-03-06T07:27:41.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.927+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8cc0cf4a0 con 0x7fa8cc0b4360 2024-03-06T07:27:41.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.931+0000 7fa8c3400640 1 -- 172.21.15.6:0/393428645 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7fa8bc058b50 con 0x7fa8cc0b4360 2024-03-06T07:27:41.934 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:41.934 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:41.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.934+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8ac06b500 msgr2=0x7fa8ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.934+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8ac06b500 0x7fa8ac06d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0b4ed0 tx=0x7fa8b8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.934+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 msgr2=0x7fa8cc1516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.934+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 0x7fa8cc1516e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc001b80 tx=0x7fa8bc007720 comp rx=0 tx=0).stop 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 shutdown_connections 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8ac06b500 0x7fa8ac06d9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc150100 0x7fa8cc159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8cc0b4740 0x7fa8cc14fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 --2- 172.21.15.6:0/393428645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8cc0b4360 0x7fa8cc1516e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 >> 172.21.15.6:0/393428645 conn(0x7fa8cc06b3d0 msgr2=0x7fa8cc152010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 shutdown_connections 2024-03-06T07:27:41.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:41.935+0000 7fa8cbe00640 1 -- 172.21.15.6:0/393428645 wait complete. 2024-03-06T07:27:41.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:41.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.179+0000 7fecd1800640 1 Processor -- start 2024-03-06T07:27:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.180+0000 7fecd1800640 1 -- start start 2024-03-06T07:27:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.181+0000 7fecd1800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7feccc162bb0 0x7feccc163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.181+0000 7fecd1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feccc0adb00 0x7feccc0adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.181+0000 7fecd1800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc163950 con 0x7feccc0ccad0 2024-03-06T07:27:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.181+0000 7fecd1800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc0acd40 con 0x7feccc0adb00 2024-03-06T07:27:42.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.181+0000 7fecd1800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc0ad020 con 0x7feccc162bb0 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.182+0000 7feccaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feccc0ccad0 0x7feccc0cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52330/0 (socket says 172.21.15.6:52330) 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.182+0000 7feccb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feccc0adb00 0x7feccc0adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.182+0000 7feccaa00640 1 -- 172.21.15.6:0/3966463075 learned_addr learned my addr 172.21.15.6:0/3966463075 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.182+0000 7fecc3e00640 1 --2- 172.21.15.6:0/3966463075 >> v2:172.21.15.178:3300/0 conn(0x7feccc162bb0 0x7feccc163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.182+0000 7fecca000640 1 -- 172.21.15.6:0/3966463075 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816627635 0 0) 0x7feccc163950 con 0x7feccc0ccad0 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.182+0000 7fecca000640 1 -- 172.21.15.6:0/3966463075 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0003520 con 0x7feccc0ccad0 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.183+0000 7feccb400640 1 -- 172.21.15.6:0/3966463075 >> v2:172.21.15.178:3300/0 conn(0x7feccc162bb0 msgr2=0x7feccc163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.183+0000 7feccb400640 1 --2- 172.21.15.6:0/3966463075 >> v2:172.21.15.178:3300/0 conn(0x7feccc162bb0 0x7feccc163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.183+0000 7fecca000640 1 -- 172.21.15.6:0/3966463075 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469293980 0 0) 0x7fecb0003520 con 0x7feccc0ccad0 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.183+0000 7feccb400640 1 -- 172.21.15.6:0/3966463075 >> v1:172.21.15.6:6789/0 conn(0x7feccc0ccad0 legacy=0x7feccc0cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.183+0000 7feccb400640 1 -- 172.21.15.6:0/3966463075 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0ad300 con 0x7feccc0adb00 2024-03-06T07:27:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.184+0000 7feccb400640 1 --2- 172.21.15.6:0/3966463075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0adb00 0x7feccc0adf10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feccc070b90 tx=0x7fecc402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36061964bcf5041 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.185+0000 7fecca000640 1 -- 172.21.15.6:0/3966463075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4039070 con 0x7feccc0adb00 2024-03-06T07:27:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.185+0000 7fecca000640 1 -- 172.21.15.6:0/3966463075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc402c8c0 con 0x7feccc0adb00 2024-03-06T07:27:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.185+0000 7fecca000640 1 -- 172.21.15.6:0/3966463075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc001670 con 0x7feccc0adb00 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.185+0000 7fecd1800640 1 -- 172.21.15.6:0/3966463075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0adb00 msgr2=0x7feccc0adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.185+0000 7fecd1800640 1 --2- 172.21.15.6:0/3966463075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0adb00 0x7feccc0adf10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7feccc070b90 tx=0x7fecc402bd90 comp rx=0 tx=0).stop 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.186+0000 7fecd1800640 1 -- 172.21.15.6:0/3966463075 shutdown_connections 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.186+0000 7fecd1800640 1 --2- 172.21.15.6:0/3966463075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0adb00 0x7feccc0adf10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.186+0000 7fecd1800640 1 --2- 172.21.15.6:0/3966463075 >> v2:172.21.15.178:3300/0 conn(0x7feccc162bb0 0x7feccc163040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.186+0000 7fecd1800640 1 -- 172.21.15.6:0/3966463075 >> 172.21.15.6:0/3966463075 conn(0x7feccc06b3d0 msgr2=0x7feccc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.186+0000 7fecd1800640 1 -- 172.21.15.6:0/3966463075 shutdown_connections 2024-03-06T07:27:42.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.186+0000 7fecd1800640 1 -- 172.21.15.6:0/3966463075 wait complete. 2024-03-06T07:27:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.188+0000 7fecd1800640 1 Processor -- start 2024-03-06T07:27:42.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.188+0000 7fecd1800640 1 -- start start 2024-03-06T07:27:42.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecd1800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feccc0adb00 0x7feccc09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecd1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 0x7feccc0aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecd1800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feccc162bb0 0x7feccc0af460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecd1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06c980 con 0x7feccc0ccad0 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecd1800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06e7d0 con 0x7feccc0adb00 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecd1800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc163580 con 0x7feccc162bb0 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 0x7feccc0aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 0x7feccc0aef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54938/0 (socket says 172.21.15.6:54938) 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 -- 172.21.15.6:0/914822487 learned_addr learned my addr 172.21.15.6:0/914822487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7feccaa00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feccc0adb00 0x7feccc09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 -- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feccc162bb0 msgr2=0x7feccc0af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feccc162bb0 0x7feccc0af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 -- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feccc0adb00 msgr2=0x7feccc09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feccc0adb00 0x7feccc09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7fecc3e00640 1 -- 172.21.15.6:0/914822487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0b2ff0 con 0x7feccc0ccad0 2024-03-06T07:27:42.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.189+0000 7feccaa00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feccc0adb00 0x7feccc09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:42.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.190+0000 7fecc3e00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 0x7feccc0aef20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fecb400e9b0 tx=0x7fecb400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.190+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4017070 con 0x7feccc0ccad0 2024-03-06T07:27:42.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.190+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb400c9a0 con 0x7feccc0ccad0 2024-03-06T07:27:42.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.190+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb400cd90 con 0x7feccc0ccad0 2024-03-06T07:27:42.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.190+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feccc0b00d0 con 0x7feccc0ccad0 2024-03-06T07:27:42.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.190+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc0b0690 con 0x7feccc0ccad0 2024-03-06T07:27:42.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.191+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc067830 con 0x7feccc0ccad0 2024-03-06T07:27:42.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.192+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecb401a070 con 0x7feccc0ccad0 2024-03-06T07:27:42.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.193+0000 7fecc2a00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feca806b4d0 0x7feca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.193+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecb408a1d0 con 0x7feccc0ccad0 2024-03-06T07:27:42.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.194+0000 7feccaa00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feca806b4d0 0x7feca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.195+0000 7feccaa00640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feca806b4d0 0x7feca806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fecbc001380 tx=0x7fecbc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.198+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feccc067830 con 0x7feccc0ccad0 2024-03-06T07:27:42.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.398+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feccc09b610 con 0x7feccc0ccad0 2024-03-06T07:27:42.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.399+0000 7fecc2a00640 1 -- 172.21.15.6:0/914822487 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fecb4059040 con 0x7feccc0ccad0 2024-03-06T07:27:42.401 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:27:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feca806b4d0 msgr2=0x7feca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feca806b4d0 0x7feca806d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fecbc001380 tx=0x7fecbc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 msgr2=0x7feccc0aef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 0x7feccc0aef20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fecb400e9b0 tx=0x7fecb400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 shutdown_connections 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feca806b4d0 0x7feca806d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feccc162bb0 0x7feccc0af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feccc0ccad0 0x7feccc0aef20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 --2- 172.21.15.6:0/914822487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feccc0adb00 0x7feccc09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 >> 172.21.15.6:0/914822487 conn(0x7feccc06b3d0 msgr2=0x7feccc1619b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.401+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 shutdown_connections 2024-03-06T07:27:42.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.402+0000 7fecd1800640 1 -- 172.21.15.6:0/914822487 wait complete. 2024-03-06T07:27:42.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:42.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.645+0000 7f9013e00640 1 Processor -- start 2024-03-06T07:27:42.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.646+0000 7f9013e00640 1 -- start start 2024-03-06T07:27:42.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.647+0000 7f9013e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f901415dee0 0x7f90140d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.647+0000 7f9013e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140d4860 0x7f90140d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.647+0000 7f9013e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90140ceca0 con 0x7f90140cfa60 2024-03-06T07:27:42.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.647+0000 7f9013e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90140cf5d0 con 0x7f901415dee0 2024-03-06T07:27:42.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.647+0000 7f9013e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140cef80 con 0x7f90140d4860 2024-03-06T07:27:42.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9013400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f90140cfa60 0x7f90140b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52338/0 (socket says 172.21.15.6:52338) 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9013400640 1 -- 172.21.15.6:0/832306543 learned_addr learned my addr 172.21.15.6:0/832306543 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9012a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f901415dee0 0x7f90140d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9012000640 1 --2- 172.21.15.6:0/832306543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140d4860 0x7f90140d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9012a00640 1 -- 172.21.15.6:0/832306543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140d4860 msgr2=0x7f90140d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9012a00640 1 --2- 172.21.15.6:0/832306543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140d4860 0x7f90140d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9012a00640 1 -- 172.21.15.6:0/832306543 >> v1:172.21.15.6:6789/0 conn(0x7f90140cfa60 legacy=0x7f90140b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9012a00640 1 -- 172.21.15.6:0/832306543 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140cf260 con 0x7f901415dee0 2024-03-06T07:27:42.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.648+0000 7f9011600640 1 -- 172.21.15.6:0/832306543 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591555836 0 0) 0x7f90140ceca0 con 0x7f90140cfa60 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.649+0000 7f9012a00640 1 --2- 172.21.15.6:0/832306543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90140d18a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9004009d90 tx=0x7f900402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802cd4a0fdc5e6e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9011600640 1 -- 172.21.15.6:0/832306543 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004039070 con 0x7f901415dee0 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9011600640 1 -- 172.21.15.6:0/832306543 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900402cc50 con 0x7f901415dee0 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9011600640 1 -- 172.21.15.6:0/832306543 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004034440 con 0x7f901415dee0 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 -- 172.21.15.6:0/832306543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 msgr2=0x7f90140d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 --2- 172.21.15.6:0/832306543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90140d18a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9004009d90 tx=0x7f900402bea0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 -- 172.21.15.6:0/832306543 shutdown_connections 2024-03-06T07:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 --2- 172.21.15.6:0/832306543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140d4860 0x7f90140d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 --2- 172.21.15.6:0/832306543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90140d18a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 -- 172.21.15.6:0/832306543 >> 172.21.15.6:0/832306543 conn(0x7f90140d3cd0 msgr2=0x7f90140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.650+0000 7f9013e00640 1 -- 172.21.15.6:0/832306543 shutdown_connections 2024-03-06T07:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.651+0000 7f9013e00640 1 -- 172.21.15.6:0/832306543 wait complete. 2024-03-06T07:27:42.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.652+0000 7f9013e00640 1 Processor -- start 2024-03-06T07:27:42.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.652+0000 7f9013e00640 1 -- start start 2024-03-06T07:27:42.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.653+0000 7f9013e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140cfa60 0x7f90140c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.653+0000 7f9013e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90140d4860 0x7f90140cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.653+0000 7f9013e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90141f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.653+0000 7f9013e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90140d1de0 con 0x7f901415dee0 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.653+0000 7f9013e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f901415e2c0 con 0x7f90140d4860 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.653+0000 7f9013e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90140d1f60 con 0x7f90140cfa60 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90141f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90141f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54950/0 (socket says 172.21.15.6:54950) 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 -- 172.21.15.6:0/2169119057 learned_addr learned my addr 172.21.15.6:0/2169119057 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:42.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9012a00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140cfa60 0x7f90140c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9012000640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90140d4860 0x7f90140cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 -- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140cfa60 msgr2=0x7f90140c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140cfa60 0x7f90140c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 -- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90140d4860 msgr2=0x7f90140cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90140d4860 0x7f90140cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 -- 172.21.15.6:0/2169119057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90140c9230 con 0x7f901415dee0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.654+0000 7f9013400640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90141f6ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f900c00c450 tx=0x7f900c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.655+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c018070 con 0x7f901415dee0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.655+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c007500 con 0x7f901415dee0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.655+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c004a20 con 0x7f901415dee0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.655+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90140c9510 con 0x7f901415dee0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.655+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90140c9ad0 con 0x7f901415dee0 2024-03-06T07:27:42.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.656+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901404f330 con 0x7f901415dee0 2024-03-06T07:27:42.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.657+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f900c004bc0 con 0x7f901415dee0 2024-03-06T07:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.658+0000 7f900aa00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ff006b500 0x7f8ff006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:42.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.658+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f900c089480 con 0x7f901415dee0 2024-03-06T07:27:42.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.659+0000 7f9012a00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ff006b500 0x7f8ff006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:42.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.660+0000 7f9012a00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ff006b500 0x7f8ff006d9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9004036000 tx=0x7f9004030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.663+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f900c0582f0 con 0x7f901415dee0 2024-03-06T07:27:42.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.866+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f901408d7d0 con 0x7f901415dee0 2024-03-06T07:27:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.867+0000 7f900aa00640 1 -- 172.21.15.6:0/2169119057 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f900c057bf0 con 0x7f901415dee0 2024-03-06T07:27:42.869 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:27:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ff006b500 msgr2=0x7f8ff006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ff006b500 0x7f8ff006d9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9004036000 tx=0x7f9004030000 comp rx=0 tx=0).stop 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 msgr2=0x7f90141f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90141f6ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f900c00c450 tx=0x7f900c00c910 comp rx=0 tx=0).stop 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 shutdown_connections 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ff006b500 0x7f8ff006d9c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90140d4860 0x7f90140cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f901415dee0 0x7f90141f6ea0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 --2- 172.21.15.6:0/2169119057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90140cfa60 0x7f90140c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 >> 172.21.15.6:0/2169119057 conn(0x7f90140d3cd0 msgr2=0x7f901415cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 shutdown_connections 2024-03-06T07:27:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:42.870+0000 7f9013e00640 1 -- 172.21.15.6:0/2169119057 wait complete. 2024-03-06T07:27:42.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.110+0000 7fb7cac00640 1 Processor -- start 2024-03-06T07:27:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.111+0000 7fb7cac00640 1 -- start start 2024-03-06T07:27:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.112+0000 7fb7cac00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb7c4147590 0x7fb7c4071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.112+0000 7fb7cac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7c40b2250 0x7fb7c40b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.112+0000 7fb7cac00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c40719c0 con 0x7fb7c406d8d0 2024-03-06T07:27:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.112+0000 7fb7cac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c4147970 con 0x7fb7c40b2250 2024-03-06T07:27:43.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.112+0000 7fb7cac00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c40b31b0 con 0x7fb7c4147590 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7c8e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb7c406d8d0 0x7fb7c40b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52344/0 (socket says 172.21.15.6:52344) 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7c8e00640 1 -- 172.21.15.6:0/3635847341 learned_addr learned my addr 172.21.15.6:0/3635847341 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7ca200640 1 --2- 172.21.15.6:0/3635847341 >> v2:172.21.15.6:3300/0 conn(0x7fb7c40b2250 0x7fb7c40b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7c9800640 1 --2- 172.21.15.6:0/3635847341 >> v2:172.21.15.178:3300/0 conn(0x7fb7c4147590 0x7fb7c4071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7c3e00640 1 -- 172.21.15.6:0/3635847341 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604370737 0 0) 0x7fb7c40719c0 con 0x7fb7c406d8d0 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7c3e00640 1 -- 172.21.15.6:0/3635847341 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ac003520 con 0x7fb7c406d8d0 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7ca200640 1 -- 172.21.15.6:0/3635847341 >> v2:172.21.15.178:3300/0 conn(0x7fb7c4147590 msgr2=0x7fb7c4071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7ca200640 1 --2- 172.21.15.6:0/3635847341 >> v2:172.21.15.178:3300/0 conn(0x7fb7c4147590 0x7fb7c4071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.113+0000 7fb7ca200640 1 -- 172.21.15.6:0/3635847341 >> v1:172.21.15.6:6789/0 conn(0x7fb7c406d8d0 legacy=0x7fb7c40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.114+0000 7fb7ca200640 1 -- 172.21.15.6:0/3635847341 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c406de40 con 0x7fb7c40b2250 2024-03-06T07:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.115+0000 7fb7ca200640 1 --2- 172.21.15.6:0/3635847341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c40b2250 0x7fb7c40b3640 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40b2790 tx=0x7fb7bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc155ead4da2f4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.115+0000 7fb7c3e00640 1 -- 172.21.15.6:0/3635847341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc039070 con 0x7fb7c40b2250 2024-03-06T07:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.115+0000 7fb7c3e00640 1 -- 172.21.15.6:0/3635847341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc02cc80 con 0x7fb7c40b2250 2024-03-06T07:27:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.115+0000 7fb7c3e00640 1 -- 172.21.15.6:0/3635847341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc034660 con 0x7fb7c40b2250 2024-03-06T07:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 -- 172.21.15.6:0/3635847341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c40b2250 msgr2=0x7fb7c40b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3635847341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c40b2250 0x7fb7c40b3640 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40b2790 tx=0x7fb7bc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 -- 172.21.15.6:0/3635847341 shutdown_connections 2024-03-06T07:27:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3635847341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c40b2250 0x7fb7c40b3640 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3635847341 >> v2:172.21.15.178:3300/0 conn(0x7fb7c4147590 0x7fb7c4071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 -- 172.21.15.6:0/3635847341 >> 172.21.15.6:0/3635847341 conn(0x7fb7c406b3a0 msgr2=0x7fb7c406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 -- 172.21.15.6:0/3635847341 shutdown_connections 2024-03-06T07:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.116+0000 7fb7cac00640 1 -- 172.21.15.6:0/3635847341 wait complete. 2024-03-06T07:27:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.118+0000 7fb7cac00640 1 Processor -- start 2024-03-06T07:27:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.118+0000 7fb7cac00640 1 -- start start 2024-03-06T07:27:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7cac00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7c406d8d0 0x7fb7c4145420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7cac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7c40b2250 0x7fb7c4142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7cac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 0x7fb7c4143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7cac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40b2b50 con 0x7fb7c4147590 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7cac00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40b2e40 con 0x7fb7c406d8d0 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7cac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c40b3b80 con 0x7fb7c40b2250 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7ca200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 0x7fb7c4143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7ca200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 0x7fb7c4143410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54976/0 (socket says 172.21.15.6:54976) 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7ca200640 1 -- 172.21.15.6:0/3228564007 learned_addr learned my addr 172.21.15.6:0/3228564007 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7c9800640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7c406d8d0 0x7fb7c4145420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.119+0000 7fb7c8e00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7c40b2250 0x7fb7c4142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7ca200640 1 -- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7c40b2250 msgr2=0x7fb7c4142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7ca200640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7c40b2250 0x7fb7c4142ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7ca200640 1 -- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7c406d8d0 msgr2=0x7fb7c4145420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7ca200640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7c406d8d0 0x7fb7c4145420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7ca200640 1 -- 172.21.15.6:0/3228564007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c4145960 con 0x7fb7c4147590 2024-03-06T07:27:43.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7c9800640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7c406d8d0 0x7fb7c4145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7ca200640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 0x7fb7c4143410 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009e80 tx=0x7fb7bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc039050 con 0x7fb7c4147590 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc045070 con 0x7fb7c4147590 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc009450 con 0x7fb7c4147590 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.120+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c4141e00 con 0x7fb7c4147590 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.121+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c41423c0 con 0x7fb7c4147590 2024-03-06T07:27:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.122+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c404f330 con 0x7fb7c4147590 2024-03-06T07:27:43.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.123+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7bc03e850 con 0x7fb7c4147590 2024-03-06T07:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.124+0000 7fb7c1600640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.124+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7bc0acfa0 con 0x7fb7c4147590 2024-03-06T07:27:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.124+0000 7fb7c9800640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.125+0000 7fb7c9800640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb7a406b4d0 0x7fb7a406d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7c406bad0 tx=0x7fb7b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.129+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7bc032e20 con 0x7fb7c4147590 2024-03-06T07:27:43.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.340+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb7c40b48b0 con 0x7fb7c4147590 2024-03-06T07:27:43.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.341+0000 7fb7c1600640 1 -- 172.21.15.6:0/3228564007 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb7bc07bea0 con 0x7fb7c4147590 2024-03-06T07:27:43.343 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:27:43.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb7a406b4d0 msgr2=0x7fb7a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb7a406b4d0 0x7fb7a406d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7c406bad0 tx=0x7fb7b8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:43.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 msgr2=0x7fb7c4143410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 0x7fb7c4143410 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009e80 tx=0x7fb7bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:43.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 shutdown_connections 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb7a406b4d0 0x7fb7a406d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7c4147590 0x7fb7c4143410 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7c40b2250 0x7fb7c4142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 --2- 172.21.15.6:0/3228564007 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7c406d8d0 0x7fb7c4145420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 >> 172.21.15.6:0/3228564007 conn(0x7fb7c406b3a0 msgr2=0x7fb7c40b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 shutdown_connections 2024-03-06T07:27:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.344+0000 7fb7cac00640 1 -- 172.21.15.6:0/3228564007 wait complete. 2024-03-06T07:27:43.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.585+0000 7fa570a00640 1 Processor -- start 2024-03-06T07:27:43.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.585+0000 7fa570a00640 1 -- start start 2024-03-06T07:27:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.586+0000 7fa570a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa56c145300 0x7fa56c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa570a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa56c0cd060 0x7fa56c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa570a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c0714d0 con 0x7fa56c070f10 2024-03-06T07:27:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa570a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c0717b0 con 0x7fa56c0cd060 2024-03-06T07:27:43.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa570a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c071a90 con 0x7fa56c145300 2024-03-06T07:27:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa563e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa56c070f10 0x7fa56c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52358/0 (socket says 172.21.15.6:52358) 2024-03-06T07:27:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa563e00640 1 -- 172.21.15.6:0/3629664321 learned_addr learned my addr 172.21.15.6:0/3629664321 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:43.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa56be00640 1 --2- 172.21.15.6:0/3629664321 >> v2:172.21.15.6:3300/0 conn(0x7fa56c0cd060 0x7fa56c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.587+0000 7fa56b400640 1 --2- 172.21.15.6:0/3629664321 >> v2:172.21.15.178:3300/0 conn(0x7fa56c145300 0x7fa56c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.588+0000 7fa56aa00640 1 -- 172.21.15.6:0/3629664321 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236762558 0 0) 0x7fa56c0714d0 con 0x7fa56c070f10 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.588+0000 7fa56aa00640 1 -- 172.21.15.6:0/3629664321 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa550003520 con 0x7fa56c070f10 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.588+0000 7fa56be00640 1 -- 172.21.15.6:0/3629664321 >> v2:172.21.15.178:3300/0 conn(0x7fa56c145300 msgr2=0x7fa56c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.588+0000 7fa56be00640 1 --2- 172.21.15.6:0/3629664321 >> v2:172.21.15.178:3300/0 conn(0x7fa56c145300 0x7fa56c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.588+0000 7fa56be00640 1 -- 172.21.15.6:0/3629664321 >> v1:172.21.15.6:6789/0 conn(0x7fa56c070f10 legacy=0x7fa56c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.588+0000 7fa56be00640 1 -- 172.21.15.6:0/3629664321 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c071d70 con 0x7fa56c0cd060 2024-03-06T07:27:43.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.589+0000 7fa56be00640 1 --2- 172.21.15.6:0/3629664321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c0cd060 0x7fa56c15c570 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0cf990 tx=0x7fa56402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b138d27fce271bfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.589+0000 7fa56aa00640 1 -- 172.21.15.6:0/3629664321 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564039070 con 0x7fa56c0cd060 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.589+0000 7fa56aa00640 1 -- 172.21.15.6:0/3629664321 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56402cbf0 con 0x7fa56c0cd060 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.589+0000 7fa56aa00640 1 -- 172.21.15.6:0/3629664321 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564034440 con 0x7fa56c0cd060 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 -- 172.21.15.6:0/3629664321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c0cd060 msgr2=0x7fa56c15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 --2- 172.21.15.6:0/3629664321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c0cd060 0x7fa56c15c570 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0cf990 tx=0x7fa56402be40 comp rx=0 tx=0).stop 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 -- 172.21.15.6:0/3629664321 shutdown_connections 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 --2- 172.21.15.6:0/3629664321 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c0cd060 0x7fa56c15c570 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 --2- 172.21.15.6:0/3629664321 >> v2:172.21.15.178:3300/0 conn(0x7fa56c145300 0x7fa56c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 -- 172.21.15.6:0/3629664321 >> 172.21.15.6:0/3629664321 conn(0x7fa56c06b3d0 msgr2=0x7fa56c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 -- 172.21.15.6:0/3629664321 shutdown_connections 2024-03-06T07:27:43.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.590+0000 7fa570a00640 1 -- 172.21.15.6:0/3629664321 wait complete. 2024-03-06T07:27:43.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.592+0000 7fa570a00640 1 Processor -- start 2024-03-06T07:27:43.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.592+0000 7fa570a00640 1 -- start start 2024-03-06T07:27:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa570a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 0x7fa56c135f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa570a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa56c0cd060 0x7fa56c1e9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa570a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa56c145300 0x7fa56c1e9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa570a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c15cab0 con 0x7fa56c070f10 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa570a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1456e0 con 0x7fa56c145300 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa570a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c15cc30 con 0x7fa56c0cd060 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa56b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 0x7fa56c135f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa56b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 0x7fa56c135f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55004/0 (socket says 172.21.15.6:55004) 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.593+0000 7fa56b400640 1 -- 172.21.15.6:0/3239288377 learned_addr learned my addr 172.21.15.6:0/3239288377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa563e00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa56c0cd060 0x7fa56c1e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56be00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa56c145300 0x7fa56c1e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56b400640 1 -- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa56c0cd060 msgr2=0x7fa56c1e9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56b400640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa56c0cd060 0x7fa56c1e9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56b400640 1 -- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa56c145300 msgr2=0x7fa56c1e9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56b400640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa56c145300 0x7fa56c1e9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56b400640 1 -- 172.21.15.6:0/3239288377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c1e68c0 con 0x7fa56c070f10 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56be00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa56c145300 0x7fa56c1e9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa56b400640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 0x7fa56c135f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e8a0 tx=0x7fa55c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c017070 con 0x7fa56c070f10 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55c00c940 con 0x7fa56c070f10 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00cd30 con 0x7fa56c070f10 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.594+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c1e5c40 con 0x7fa56c070f10 2024-03-06T07:27:43.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.595+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c1e6e40 con 0x7fa56c070f10 2024-03-06T07:27:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.596+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c04f330 con 0x7fa56c070f10 2024-03-06T07:27:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.597+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa55c01a070 con 0x7fa56c070f10 2024-03-06T07:27:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.597+0000 7fa563400640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa54806b4d0 0x7fa54806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:43.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.598+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa55c08a090 con 0x7fa56c070f10 2024-03-06T07:27:43.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.598+0000 7fa563e00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa54806b4d0 0x7fa54806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:43.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.599+0000 7fa563e00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa54806b4d0 0x7fa54806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0cc390 tx=0x7fa554009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:43.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.602+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa55c058f10 con 0x7fa56c070f10 2024-03-06T07:27:43.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.843+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa56c072470 con 0x7fa56c070f10 2024-03-06T07:27:43.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.845+0000 7fa563400640 1 -- 172.21.15.6:0/3239288377 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fa55c058810 con 0x7fa56c070f10 2024-03-06T07:27:43.847 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:43.847 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:27:43.847 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.847+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa54806b4d0 msgr2=0x7fa54806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.847+0000 7fa570a00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa54806b4d0 0x7fa54806d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0cc390 tx=0x7fa554009040 comp rx=0 tx=0).stop 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.847+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 msgr2=0x7fa56c135f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.847+0000 7fa570a00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 0x7fa56c135f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e8a0 tx=0x7fa55c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 shutdown_connections 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa54806b4d0 0x7fa54806d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa56c145300 0x7fa56c1e9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa56c0cd060 0x7fa56c1e9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 --2- 172.21.15.6:0/3239288377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa56c070f10 0x7fa56c135f30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 >> 172.21.15.6:0/3239288377 conn(0x7fa56c06b3d0 msgr2=0x7fa56c0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 shutdown_connections 2024-03-06T07:27:43.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:43.848+0000 7fa570a00640 1 -- 172.21.15.6:0/3239288377 wait complete. 2024-03-06T07:27:43.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:44.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.082+0000 7f3155200640 1 Processor -- start 2024-03-06T07:27:44.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.083+0000 7f3155200640 1 -- start start 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f3155200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31500d0cd0 0x7f31500b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f3155200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31500d2dc0 con 0x7f31500d2960 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f3155200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31500d18c0 con 0x7f315015e070 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f3155200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31500d1ba0 con 0x7f31500d0cd0 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f314fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31500d0cd0 0x7f31500b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f314ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31500d2960 0x7f31500d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52372/0 (socket says 172.21.15.6:52372) 2024-03-06T07:27:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.084+0000 7f314ea00640 1 -- 172.21.15.6:0/94914730 learned_addr learned my addr 172.21.15.6:0/94914730 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.085+0000 7f314fe00640 1 -- 172.21.15.6:0/94914730 >> v1:172.21.15.178:6789/0 conn(0x7f315015e070 legacy=0x7f31500cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.085+0000 7f314fe00640 1 -- 172.21.15.6:0/94914730 >> v1:172.21.15.6:6789/0 conn(0x7f31500d2960 legacy=0x7f31500d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.085+0000 7f314fe00640 1 -- 172.21.15.6:0/94914730 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31500d1e80 con 0x7f31500d0cd0 2024-03-06T07:27:44.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.085+0000 7f314e000640 1 -- 172.21.15.6:0/94914730 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075902778 0 0) 0x7f31500d2dc0 con 0x7f31500d2960 2024-03-06T07:27:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.086+0000 7f314fe00640 1 --2- 172.21.15.6:0/94914730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31500d0cd0 0x7f31500b3630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f31480097a0 tx=0x7f314802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835be1887cfa5d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f314e000640 1 -- 172.21.15.6:0/94914730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148034070 con 0x7f31500d0cd0 2024-03-06T07:27:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f314e000640 1 -- 172.21.15.6:0/94914730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148030610 con 0x7f31500d0cd0 2024-03-06T07:27:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f314e000640 1 -- 172.21.15.6:0/94914730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31480309e0 con 0x7f31500d0cd0 2024-03-06T07:27:44.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 -- 172.21.15.6:0/94914730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31500d0cd0 msgr2=0x7f31500b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 --2- 172.21.15.6:0/94914730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31500d0cd0 0x7f31500b3630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f31480097a0 tx=0x7f314802b960 comp rx=0 tx=0).stop 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 -- 172.21.15.6:0/94914730 shutdown_connections 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 --2- 172.21.15.6:0/94914730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31500d0cd0 0x7f31500b3630 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 -- 172.21.15.6:0/94914730 >> 172.21.15.6:0/94914730 conn(0x7f315008d2a0 msgr2=0x7f31500d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 -- 172.21.15.6:0/94914730 shutdown_connections 2024-03-06T07:27:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.087+0000 7f3155200640 1 -- 172.21.15.6:0/94914730 wait complete. 2024-03-06T07:27:44.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.089+0000 7f3155200640 1 Processor -- start 2024-03-06T07:27:44.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.089+0000 7f3155200640 1 -- start start 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f3155200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31500d0cd0 0x7f31500c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f3155200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31500d2960 0x7f315015c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f3155200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 0x7f31500ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f3155200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31500d45b0 con 0x7f315015e070 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f3155200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31500d2440 con 0x7f31500d0cd0 2024-03-06T07:27:44.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f3155200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31500d4730 con 0x7f31500d2960 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.090+0000 7f314fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 0x7f31500ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 0x7f31500ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55022/0 (socket says 172.21.15.6:55022) 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 -- 172.21.15.6:0/734990651 learned_addr learned my addr 172.21.15.6:0/734990651 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314ea00640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31500d2960 0x7f315015c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314f400640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31500d0cd0 0x7f31500c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 -- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31500d2960 msgr2=0x7f315015c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31500d2960 0x7f315015c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 -- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31500d0cd0 msgr2=0x7f31500c86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31500d0cd0 0x7f31500c86f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 -- 172.21.15.6:0/734990651 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31500c8cd0 con 0x7f315015e070 2024-03-06T07:27:44.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.091+0000 7f314fe00640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 0x7f31500ca160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f31480098c0 tx=0x7f31480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.092+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148034040 con 0x7f315015e070 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.092+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148039990 con 0x7f315015e070 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.092+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148039d80 con 0x7f315015e070 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.092+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31500c8fb0 con 0x7f315015e070 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.092+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31500c9570 con 0x7f315015e070 2024-03-06T07:27:44.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.093+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315004f330 con 0x7f315015e070 2024-03-06T07:27:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.094+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3148037070 con 0x7f315015e070 2024-03-06T07:27:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.095+0000 7f3147400640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f312c06b4d0 0x7f312c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.095+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31480ac350 con 0x7f315015e070 2024-03-06T07:27:44.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.095+0000 7f314f400640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f312c06b4d0 0x7f312c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.096+0000 7f314f400640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f312c06b4d0 0x7f312c06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f31400012e0 tx=0x7f3140009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.099+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f314807b130 con 0x7f315015e070 2024-03-06T07:27:44.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.306+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31500d1370 con 0x7f315015e070 2024-03-06T07:27:44.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.308+0000 7f3147400640 1 -- 172.21.15.6:0/734990651 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f314807aa30 con 0x7f315015e070 2024-03-06T07:27:44.311 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:44.311 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f312c06b4d0 msgr2=0x7f312c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f312c06b4d0 0x7f312c06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f31400012e0 tx=0x7f3140009040 comp rx=0 tx=0).stop 2024-03-06T07:27:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 msgr2=0x7f31500ca160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 0x7f31500ca160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f31480098c0 tx=0x7f31480046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 shutdown_connections 2024-03-06T07:27:44.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f312c06b4d0 0x7f312c06d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f315015e070 0x7f31500ca160 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31500d2960 0x7f315015c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 --2- 172.21.15.6:0/734990651 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31500d0cd0 0x7f31500c86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 >> 172.21.15.6:0/734990651 conn(0x7f315008d2a0 msgr2=0x7f315015d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 shutdown_connections 2024-03-06T07:27:44.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.312+0000 7f3155200640 1 -- 172.21.15.6:0/734990651 wait complete. 2024-03-06T07:27:44.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:44.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:44.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.549+0000 7f9edba00640 1 Processor -- start 2024-03-06T07:27:44.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.549+0000 7f9edba00640 1 -- start start 2024-03-06T07:27:44.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.551+0000 7f9edba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9ed40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.551+0000 7f9edba00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9ed4071e20 0x7f9ed40ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.551+0000 7f9edba00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed4072df0 con 0x7f9ed415bf20 2024-03-06T07:27:44.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.551+0000 7f9edba00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed40cdab0 con 0x7f9ed4071e20 2024-03-06T07:27:44.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.551+0000 7f9edba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40b4d70 con 0x7f9ed4072520 2024-03-06T07:27:44.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed9000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9ed415bf20 0x7f9ed40acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52386/0 (socket says 172.21.15.6:52386) 2024-03-06T07:27:44.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed9000640 1 -- 172.21.15.6:0/3476867952 learned_addr learned my addr 172.21.15.6:0/3476867952 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3e00640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9ed40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed9a00640 1 --2- 172.21.15.6:0/3476867952 >> v2:172.21.15.178:3300/0 conn(0x7f9ed4071e20 0x7f9ed40ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687556077 0 0) 0x7f9ed4072df0 con 0x7f9ed415bf20 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb8003520 con 0x7f9ed415bf20 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295747081 0 0) 0x7f9eb8003520 con 0x7f9ed415bf20 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 msgr2=0x7f9ed40cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3400640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9ed40cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.552+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 >> v2:172.21.15.178:3300/0 conn(0x7f9ed4071e20 msgr2=0x7f9ed40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 --2- 172.21.15.6:0/3476867952 >> v2:172.21.15.178:3300/0 conn(0x7f9ed4071e20 0x7f9ed40ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed40b5050 con 0x7f9ed415bf20 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9ec40019f0 con 0x7f9ed415bf20 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 >> v1:172.21.15.6:6789/0 conn(0x7f9ed415bf20 legacy=0x7f9ed40acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eb80039b0 0x7f9eb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9eb8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb8004d40 0x7f9eb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40cdab0 con 0x7f9eb80039b0 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40b4d70 con 0x7f9eb8004d40 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb800b9c0 con 0x7f9ed4072520 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eb80039b0 0x7f9eb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3e00640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9eb8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed9000640 1 -- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 msgr2=0x7f9eb8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9eb8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed9000640 1 -- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb8004d40 msgr2=0x7f9eb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb8004d40 0x7f9eb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed9000640 1 -- 172.21.15.6:0/3476867952 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb8003520 con 0x7f9eb80039b0 2024-03-06T07:27:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.553+0000 7f9ed3e00640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed4072520 0x7f9eb8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:44.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.554+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eb80039b0 0x7f9eb8003dd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400a7d0 tx=0x7f9eb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77f85dfcc611c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400a3f0 con 0x7f9eb80039b0 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec4034650 con 0x7f9eb80039b0 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9ed3400640 1 -- 172.21.15.6:0/3476867952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4034a40 con 0x7f9eb80039b0 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9edba00640 1 -- 172.21.15.6:0/3476867952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eb80039b0 msgr2=0x7f9eb8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9edba00640 1 --2- 172.21.15.6:0/3476867952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eb80039b0 0x7f9eb8003dd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400a7d0 tx=0x7f9eb8005830 comp rx=0 tx=0).stop 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9ed9a00640 1 -- 172.21.15.6:0/3476867952 reap_dead start 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9edba00640 1 -- 172.21.15.6:0/3476867952 shutdown_connections 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9edba00640 1 -- 172.21.15.6:0/3476867952 >> 172.21.15.6:0/3476867952 conn(0x7f9ed406b3d0 msgr2=0x7f9ed406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.555+0000 7f9edba00640 1 -- 172.21.15.6:0/3476867952 shutdown_connections 2024-03-06T07:27:44.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.556+0000 7f9edba00640 1 -- 172.21.15.6:0/3476867952 wait complete. 2024-03-06T07:27:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.557+0000 7f9edba00640 1 Processor -- start 2024-03-06T07:27:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.557+0000 7f9edba00640 1 -- start start 2024-03-06T07:27:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.558+0000 7f9edba00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb80039b0 0x7f9ed4150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.558+0000 7f9edba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed40af7d0 0x7f9ed409b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.558+0000 7f9edba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 0x7f9ed409ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.558+0000 7f9edba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40ad450 con 0x7f9ed40afc50 2024-03-06T07:27:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.558+0000 7f9edba00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed40b4990 con 0x7f9eb80039b0 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.558+0000 7f9edba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed4154c30 con 0x7f9ed40af7d0 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 0x7f9ed409ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 0x7f9ed409ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55052/0 (socket says 172.21.15.6:55052) 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 -- 172.21.15.6:0/3841605112 learned_addr learned my addr 172.21.15.6:0/3841605112 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed3e00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed40af7d0 0x7f9ed409b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb80039b0 0x7f9ed4150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 -- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed40af7d0 msgr2=0x7f9ed409b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed40af7d0 0x7f9ed409b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 -- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb80039b0 msgr2=0x7f9ed4150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb80039b0 0x7f9ed4150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.559+0000 7f9ed9a00640 1 -- 172.21.15.6:0/3841605112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed4154f10 con 0x7f9ed40afc50 2024-03-06T07:27:44.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.560+0000 7f9ed9a00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 0x7f9ed409ba90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9ed406c030 tx=0x7f9ecc00e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.560+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc013680 con 0x7f9ed40afc50 2024-03-06T07:27:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.560+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc01a070 con 0x7f9ed40afc50 2024-03-06T07:27:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.560+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc013a60 con 0x7f9ed40afc50 2024-03-06T07:27:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.560+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed41551f0 con 0x7f9ed40afc50 2024-03-06T07:27:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.560+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed41557b0 con 0x7f9ed40afc50 2024-03-06T07:27:44.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.561+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed404f330 con 0x7f9ed40afc50 2024-03-06T07:27:44.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.562+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ecc013c00 con 0x7f9ed40afc50 2024-03-06T07:27:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.563+0000 7f9ed0c00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9eb406b500 0x7f9eb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.564+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ecc089fc0 con 0x7f9ed40afc50 2024-03-06T07:27:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.564+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9eb406b500 0x7f9eb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.565+0000 7f9ed9000640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9eb406b500 0x7f9eb406d9c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4034000 tx=0x7f9ec4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.568+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ecc058e30 con 0x7f9ed40afc50 2024-03-06T07:27:44.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.766+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ed4155980 con 0x7f9ed40afc50 2024-03-06T07:27:44.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.767+0000 7f9ed0c00640 1 -- 172.21.15.6:0/3841605112 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ecc058730 con 0x7f9ed40afc50 2024-03-06T07:27:44.770 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:27:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9eb406b500 msgr2=0x7f9eb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9eb406b500 0x7f9eb406d9c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4034000 tx=0x7f9ec4009030 comp rx=0 tx=0).stop 2024-03-06T07:27:44.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 msgr2=0x7f9ed409ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 0x7f9ed409ba90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9ed406c030 tx=0x7f9ecc00e9f0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 shutdown_connections 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9eb406b500 0x7f9eb406d9c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ed40afc50 0x7f9ed409ba90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ed40af7d0 0x7f9ed409b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 --2- 172.21.15.6:0/3841605112 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eb80039b0 0x7f9ed4150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 >> 172.21.15.6:0/3841605112 conn(0x7f9ed406b3d0 msgr2=0x7f9ed415ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 shutdown_connections 2024-03-06T07:27:44.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:44.770+0000 7f9edba00640 1 -- 172.21.15.6:0/3841605112 wait complete. 2024-03-06T07:27:44.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.008+0000 7fef0da00640 1 Processor -- start 2024-03-06T07:27:45.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.009+0000 7fef0da00640 1 -- start start 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef0da00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef0815bf20 0x7fef080acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef0da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fef08072520 0x7fef080cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef0da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef08072df0 con 0x7fef08071e20 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef0da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef080cdab0 con 0x7fef08072520 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef0da00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef080b4d70 con 0x7fef0815bf20 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef07400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef08071e20 0x7fef080ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52398/0 (socket says 172.21.15.6:52398) 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7feefe000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fef08072520 0x7fef080cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef07400640 1 -- 172.21.15.6:0/2328292312 learned_addr learned my addr 172.21.15.6:0/2328292312 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.011+0000 7fef06a00640 1 --2- 172.21.15.6:0/2328292312 >> v2:172.21.15.178:3300/0 conn(0x7fef0815bf20 0x7fef080acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.012+0000 7feefe000640 1 -- 172.21.15.6:0/2328292312 >> v2:172.21.15.178:3300/0 conn(0x7fef0815bf20 msgr2=0x7fef080acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.012+0000 7feefe000640 1 --2- 172.21.15.6:0/2328292312 >> v2:172.21.15.178:3300/0 conn(0x7fef0815bf20 0x7fef080acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.012+0000 7feefe000640 1 -- 172.21.15.6:0/2328292312 >> v1:172.21.15.6:6789/0 conn(0x7fef08071e20 legacy=0x7fef080ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.012+0000 7feefe000640 1 -- 172.21.15.6:0/2328292312 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080b5050 con 0x7fef08072520 2024-03-06T07:27:45.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.012+0000 7fef06000640 1 -- 172.21.15.6:0/2328292312 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587264198 0 0) 0x7fef08072df0 con 0x7fef08071e20 2024-03-06T07:27:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.013+0000 7feefe000640 1 --2- 172.21.15.6:0/2328292312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08072520 0x7fef080cdfb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fef0806ec40 tx=0x7feef002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81858a06463d4822 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.013+0000 7fef06000640 1 -- 172.21.15.6:0/2328292312 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0035020 con 0x7fef08072520 2024-03-06T07:27:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.013+0000 7fef06000640 1 -- 172.21.15.6:0/2328292312 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef0031830 con 0x7fef08072520 2024-03-06T07:27:45.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.013+0000 7fef06000640 1 -- 172.21.15.6:0/2328292312 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0031c00 con 0x7fef08072520 2024-03-06T07:27:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 -- 172.21.15.6:0/2328292312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08072520 msgr2=0x7fef080cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 --2- 172.21.15.6:0/2328292312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08072520 0x7fef080cdfb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fef0806ec40 tx=0x7feef002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 -- 172.21.15.6:0/2328292312 shutdown_connections 2024-03-06T07:27:45.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 --2- 172.21.15.6:0/2328292312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08072520 0x7fef080cdfb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 --2- 172.21.15.6:0/2328292312 >> v2:172.21.15.178:3300/0 conn(0x7fef0815bf20 0x7fef080acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 -- 172.21.15.6:0/2328292312 >> 172.21.15.6:0/2328292312 conn(0x7fef0806b3d0 msgr2=0x7fef0806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 -- 172.21.15.6:0/2328292312 shutdown_connections 2024-03-06T07:27:45.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.014+0000 7fef0da00640 1 -- 172.21.15.6:0/2328292312 wait complete. 2024-03-06T07:27:45.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.016+0000 7fef0da00640 1 Processor -- start 2024-03-06T07:27:45.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.016+0000 7fef0da00640 1 -- start start 2024-03-06T07:27:45.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef0da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 0x7fef08147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef0da00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef08072520 0x7fef08156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef0da00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef0815bf20 0x7fef081574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef0da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef080b4b10 con 0x7fef08071e20 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef0da00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef080b4990 con 0x7fef0815bf20 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef0da00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef080cd1e0 con 0x7fef08072520 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef06a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 0x7fef08147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef06a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 0x7fef08147340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55066/0 (socket says 172.21.15.6:55066) 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.017+0000 7fef06a00640 1 -- 172.21.15.6:0/2708664693 learned_addr learned my addr 172.21.15.6:0/2708664693 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7feefe000640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef08072520 0x7fef08156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef07400640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef0815bf20 0x7fef081574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef06a00640 1 -- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef08072520 msgr2=0x7fef08156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef06a00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef08072520 0x7fef08156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef06a00640 1 -- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef0815bf20 msgr2=0x7fef081574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef06a00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef0815bf20 0x7fef081574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef06a00640 1 -- 172.21.15.6:0/2708664693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0806cbd0 con 0x7fef08071e20 2024-03-06T07:27:45.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef07400640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef0815bf20 0x7fef081574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef06a00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 0x7fef08147340 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feef800d620 tx=0x7feef800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef800b620 con 0x7fef08071e20 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef8019070 con 0x7fef08071e20 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0806ceb0 con 0x7fef08071e20 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef800ba00 con 0x7fef08071e20 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.018+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0806d470 con 0x7fef08071e20 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.020+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0804f330 con 0x7fef08071e20 2024-03-06T07:27:45.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.021+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feef800bba0 con 0x7fef08071e20 2024-03-06T07:27:45.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.021+0000 7feeff400640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feee406b500 0x7feee406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.022+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feef808a750 con 0x7fef08071e20 2024-03-06T07:27:45.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.022+0000 7feefe000640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feee406b500 0x7feee406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.023+0000 7feefe000640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feee406b500 0x7feee406d9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feef0036000 tx=0x7feef0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.026+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feef80594a0 con 0x7fef08071e20 2024-03-06T07:27:45.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.227+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef0815f720 con 0x7fef08071e20 2024-03-06T07:27:45.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.228+0000 7feeff400640 1 -- 172.21.15.6:0/2708664693 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7feef8058da0 con 0x7fef08071e20 2024-03-06T07:27:45.230 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:27:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feee406b500 msgr2=0x7feee406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feee406b500 0x7feee406d9c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feef0036000 tx=0x7feef0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:45.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 msgr2=0x7fef08147340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 0x7fef08147340 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7feef800d620 tx=0x7feef800dae0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 shutdown_connections 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feee406b500 0x7feee406d9c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef0815bf20 0x7fef081574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef08072520 0x7fef08156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 --2- 172.21.15.6:0/2708664693 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef08071e20 0x7fef08147340 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 >> 172.21.15.6:0/2708664693 conn(0x7fef0806b3d0 msgr2=0x7fef080adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 shutdown_connections 2024-03-06T07:27:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.231+0000 7fef0da00640 1 -- 172.21.15.6:0/2708664693 wait complete. 2024-03-06T07:27:45.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.471+0000 7f1b19000640 1 Processor -- start 2024-03-06T07:27:45.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.471+0000 7f1b19000640 1 -- start start 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b19000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1b140cfdf0 0x7f1b140d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b19000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b140ccc50 con 0x7f1b140d07d0 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b19000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b140ccf30 con 0x7f1b1415c0e0 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b19000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1415c4c0 con 0x7f1b140cfdf0 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b12800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1b140d07d0 0x7f1b140b4b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52414/0 (socket says 172.21.15.6:52414) 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b12800640 1 -- 172.21.15.6:0/491945088 learned_addr learned my addr 172.21.15.6:0/491945088 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.473+0000 7f1b11400640 1 --2- 172.21.15.6:0/491945088 >> v2:172.21.15.178:3300/0 conn(0x7f1b140cfdf0 0x7f1b140d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.474+0000 7f1b10a00640 1 -- 172.21.15.6:0/491945088 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550872697 0 0) 0x7f1b140ccc50 con 0x7f1b140d07d0 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.474+0000 7f1b10a00640 1 -- 172.21.15.6:0/491945088 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8003520 con 0x7f1b140d07d0 2024-03-06T07:27:45.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.474+0000 7f1b11400640 1 -- 172.21.15.6:0/491945088 >> v1:172.21.15.178:6789/0 conn(0x7f1b1415c0e0 legacy=0x7f1b140cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.474+0000 7f1b11400640 1 -- 172.21.15.6:0/491945088 >> v1:172.21.15.6:6789/0 conn(0x7f1b140d07d0 legacy=0x7f1b140b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.474+0000 7f1b11400640 1 -- 172.21.15.6:0/491945088 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b140cd210 con 0x7f1b140cfdf0 2024-03-06T07:27:45.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.475+0000 7f1b11400640 1 --2- 172.21.15.6:0/491945088 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140cfdf0 0x7f1b140d0290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009d90 tx=0x7f1b0402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c868824968ea89c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.476+0000 7f1b10a00640 1 -- 172.21.15.6:0/491945088 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04039070 con 0x7f1b140cfdf0 2024-03-06T07:27:45.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.476+0000 7f1b10a00640 1 -- 172.21.15.6:0/491945088 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0402cd80 con 0x7f1b140cfdf0 2024-03-06T07:27:45.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.476+0000 7f1b10a00640 1 -- 172.21.15.6:0/491945088 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04034660 con 0x7f1b140cfdf0 2024-03-06T07:27:45.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 -- 172.21.15.6:0/491945088 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140cfdf0 msgr2=0x7f1b140d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 --2- 172.21.15.6:0/491945088 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140cfdf0 0x7f1b140d0290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009d90 tx=0x7f1b0402bfd0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 -- 172.21.15.6:0/491945088 shutdown_connections 2024-03-06T07:27:45.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 --2- 172.21.15.6:0/491945088 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140cfdf0 0x7f1b140d0290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 -- 172.21.15.6:0/491945088 >> 172.21.15.6:0/491945088 conn(0x7f1b140b3820 msgr2=0x7f1b140b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 -- 172.21.15.6:0/491945088 shutdown_connections 2024-03-06T07:27:45.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.477+0000 7f1b19000640 1 -- 172.21.15.6:0/491945088 wait complete. 2024-03-06T07:27:45.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.480+0000 7f1b19000640 1 Processor -- start 2024-03-06T07:27:45.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.480+0000 7f1b19000640 1 -- start start 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.481+0000 7f1b19000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b140cfdf0 0x7f1b1415c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.481+0000 7f1b19000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140d07d0 0x7f1b14148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.481+0000 7f1b19000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 0x7f1b14148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.481+0000 7f1b19000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140cd580 con 0x7f1b1415c0e0 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.481+0000 7f1b19000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140cf630 con 0x7f1b140d07d0 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.481+0000 7f1b19000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b140cda10 con 0x7f1b140cfdf0 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 0x7f1b14148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b11e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b140cfdf0 0x7f1b1415c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 0x7f1b14148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55070/0 (socket says 172.21.15.6:55070) 2024-03-06T07:27:45.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 -- 172.21.15.6:0/2072104082 learned_addr learned my addr 172.21.15.6:0/2072104082 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:45.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b11400640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140d07d0 0x7f1b14148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 -- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b140cfdf0 msgr2=0x7f1b1415c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b140cfdf0 0x7f1b1415c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 -- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140d07d0 msgr2=0x7f1b14148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140d07d0 0x7f1b14148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b12800640 1 -- 172.21.15.6:0/2072104082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b140b6510 con 0x7f1b1415c0e0 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.482+0000 7f1b11400640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140d07d0 0x7f1b14148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.483+0000 7f1b12800640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 0x7f1b14148b70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00c460 tx=0x7f1b0c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.483+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c017070 con 0x7f1b1415c0e0 2024-03-06T07:27:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.483+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0c005ce0 con 0x7f1b1415c0e0 2024-03-06T07:27:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.483+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c0072b0 con 0x7f1b1415c0e0 2024-03-06T07:27:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.483+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b140b67f0 con 0x7f1b1415c0e0 2024-03-06T07:27:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.483+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b140b6db0 con 0x7f1b1415c0e0 2024-03-06T07:27:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.484+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1404f330 con 0x7f1b1415c0e0 2024-03-06T07:27:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.485+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b0c01a070 con 0x7f1b1415c0e0 2024-03-06T07:27:45.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.486+0000 7f1b0a000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1af006b4d0 0x7f1af006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.486+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b0c0894c0 con 0x7f1b1415c0e0 2024-03-06T07:27:45.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.487+0000 7f1b11e00640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1af006b4d0 0x7f1af006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.488+0000 7f1b11e00640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1af006b4d0 0x7f1af006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1b1409fde0 tx=0x7f1afc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.490+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b0c057ce0 con 0x7f1b1415c0e0 2024-03-06T07:27:45.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.694+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b140b7940 con 0x7f1b1415c0e0 2024-03-06T07:27:45.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.694+0000 7f1b0a000640 1 -- 172.21.15.6:0/2072104082 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1b0c05bd80 con 0x7f1b1415c0e0 2024-03-06T07:27:45.696 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:27:45.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1af006b4d0 msgr2=0x7f1af006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1af006b4d0 0x7f1af006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1b1409fde0 tx=0x7f1afc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 msgr2=0x7f1b14148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 0x7f1b14148b70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00c460 tx=0x7f1b0c00c920 comp rx=0 tx=0).stop 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 shutdown_connections 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1af006b4d0 0x7f1af006d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b140d07d0 0x7f1b14148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b1415c0e0 0x7f1b14148b70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 --2- 172.21.15.6:0/2072104082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b140cfdf0 0x7f1b1415c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.697+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 >> 172.21.15.6:0/2072104082 conn(0x7f1b140b3820 msgr2=0x7f1b1406da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.698+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 shutdown_connections 2024-03-06T07:27:45.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.698+0000 7f1b19000640 1 -- 172.21.15.6:0/2072104082 wait complete. 2024-03-06T07:27:45.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:45.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.937+0000 7f9e67e00640 1 Processor -- start 2024-03-06T07:27:45.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.938+0000 7f9e67e00640 1 -- start start 2024-03-06T07:27:45.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e6815e0b0 0x7f9e680cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e680d4a10 0x7f9e680d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e680b3710 con 0x7f9e680d0c80 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e680b09e0 con 0x7f9e6815e0b0 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680b39f0 con 0x7f9e680d4a10 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9e680d0c80 0x7f9e680d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52420/0 (socket says 172.21.15.6:52420) 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9e6815e0b0 0x7f9e680cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.939+0000 7f9e67400640 1 -- 172.21.15.6:0/348103338 learned_addr learned my addr 172.21.15.6:0/348103338 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:45.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e66000640 1 --2- 172.21.15.6:0/348103338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e680d4a10 0x7f9e680d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e65600640 1 -- 172.21.15.6:0/348103338 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552959563 0 0) 0x7f9e680b3710 con 0x7f9e680d0c80 2024-03-06T07:27:45.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e65600640 1 -- 172.21.15.6:0/348103338 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e4c003520 con 0x7f9e680d0c80 2024-03-06T07:27:45.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e66a00640 1 -- 172.21.15.6:0/348103338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e680d4a10 msgr2=0x7f9e680d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:45.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e66a00640 1 --2- 172.21.15.6:0/348103338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e680d4a10 0x7f9e680d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e66a00640 1 -- 172.21.15.6:0/348103338 >> v1:172.21.15.6:6789/0 conn(0x7f9e680d0c80 legacy=0x7f9e680d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.940+0000 7f9e66a00640 1 -- 172.21.15.6:0/348103338 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e680b3cd0 con 0x7f9e6815e0b0 2024-03-06T07:27:45.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.941+0000 7f9e66a00640 1 --2- 172.21.15.6:0/348103338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e6815e0b0 0x7f9e680cf310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9e580097a0 tx=0x7f9e5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a197ecf374449e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.941+0000 7f9e65600640 1 -- 172.21.15.6:0/348103338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58035020 con 0x7f9e6815e0b0 2024-03-06T07:27:45.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e65600640 1 -- 172.21.15.6:0/348103338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e58031830 con 0x7f9e6815e0b0 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e65600640 1 -- 172.21.15.6:0/348103338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58031c00 con 0x7f9e6815e0b0 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 -- 172.21.15.6:0/348103338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e6815e0b0 msgr2=0x7f9e680cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 --2- 172.21.15.6:0/348103338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e6815e0b0 0x7f9e680cf310 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9e580097a0 tx=0x7f9e5802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 -- 172.21.15.6:0/348103338 shutdown_connections 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 --2- 172.21.15.6:0/348103338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e680d4a10 0x7f9e680d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 --2- 172.21.15.6:0/348103338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e6815e0b0 0x7f9e680cf310 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 -- 172.21.15.6:0/348103338 >> 172.21.15.6:0/348103338 conn(0x7f9e680d3d30 msgr2=0x7f9e680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 -- 172.21.15.6:0/348103338 shutdown_connections 2024-03-06T07:27:45.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.942+0000 7f9e67e00640 1 -- 172.21.15.6:0/348103338 wait complete. 2024-03-06T07:27:45.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.944+0000 7f9e67e00640 1 Processor -- start 2024-03-06T07:27:45.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.944+0000 7f9e67e00640 1 -- start start 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e680d0c80 0x7f9e681f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 0x7f9e680ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e67e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e6815e0b0 0x7f9e680b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e67e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680cfc60 con 0x7f9e680d4a10 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e67e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6815e490 con 0x7f9e680d0c80 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e67e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e680b54e0 con 0x7f9e6815e0b0 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e66000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 0x7f9e680ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e66000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 0x7f9e680ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55094/0 (socket says 172.21.15.6:55094) 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.945+0000 7f9e66000640 1 -- 172.21.15.6:0/94032659 learned_addr learned my addr 172.21.15.6:0/94032659 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e67400640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e6815e0b0 0x7f9e680b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66a00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e680d0c80 0x7f9e681f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66000640 1 -- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e6815e0b0 msgr2=0x7f9e680b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66000640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e6815e0b0 0x7f9e680b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66000640 1 -- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e680d0c80 msgr2=0x7f9e681f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:45.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66000640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e680d0c80 0x7f9e681f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66000640 1 -- 172.21.15.6:0/94032659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e681f7810 con 0x7f9e680d4a10 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e66000640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 0x7f9e680ca500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000e980 tx=0x7f9e5000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e50017070 con 0x7f9e680d4a10 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5000ca00 con 0x7f9e680d4a10 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5000cdd0 con 0x7f9e680d4a10 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.946+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e681f7af0 con 0x7f9e680d4a10 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.947+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e681f7470 con 0x7f9e680d4a10 2024-03-06T07:27:45.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.948+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e681f38f0 con 0x7f9e680d4a10 2024-03-06T07:27:45.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.949+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e5001a070 con 0x7f9e680d4a10 2024-03-06T07:27:45.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.950+0000 7f9e5ea00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e4406b4d0 0x7f9e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:45.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.950+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e5008a5c0 con 0x7f9e680d4a10 2024-03-06T07:27:45.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.950+0000 7f9e66a00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e4406b4d0 0x7f9e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:45.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.951+0000 7f9e66a00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e4406b4d0 0x7f9e4406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9e58036000 tx=0x7f9e5803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:45.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:45.954+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e500593a0 con 0x7f9e680d4a10 2024-03-06T07:27:46.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.193+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e68161640 con 0x7f9e680d4a10 2024-03-06T07:27:46.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.193+0000 7f9e5ea00640 1 -- 172.21.15.6:0/94032659 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f9e50058ca0 con 0x7f9e680d4a10 2024-03-06T07:27:46.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:46.196 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:27:46.196 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e4406b4d0 msgr2=0x7f9e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e4406b4d0 0x7f9e4406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9e58036000 tx=0x7f9e5803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 msgr2=0x7f9e680ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 0x7f9e680ca500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000e980 tx=0x7f9e5000ee40 comp rx=0 tx=0).stop 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 shutdown_connections 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e4406b4d0 0x7f9e4406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e6815e0b0 0x7f9e680b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e680d4a10 0x7f9e680ca500 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 --2- 172.21.15.6:0/94032659 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e680d0c80 0x7f9e681f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.196+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 >> 172.21.15.6:0/94032659 conn(0x7f9e680d3d30 msgr2=0x7f9e6806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.197+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 shutdown_connections 2024-03-06T07:27:46.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.197+0000 7f9e67e00640 1 -- 172.21.15.6:0/94032659 wait complete. 2024-03-06T07:27:46.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:46.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.442+0000 7f6d22000640 1 Processor -- start 2024-03-06T07:27:46.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.442+0000 7f6d22000640 1 -- start start 2024-03-06T07:27:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.443+0000 7f6d22000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c070f10 0x7f6d1c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.443+0000 7f6d22000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6d1c0cce90 0x7f6d1c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.443+0000 7f6d22000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c0714d0 con 0x7f6d1c145300 2024-03-06T07:27:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.443+0000 7f6d22000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0cd300 con 0x7f6d1c0cce90 2024-03-06T07:27:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.443+0000 7f6d22000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0717b0 con 0x7f6d1c070f10 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.444+0000 7f6d21600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6d1c0cce90 0x7f6d1c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.444+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c070f10 0x7f6d1c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.444+0000 7f6d21600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6d1c0cce90 0x7f6d1c15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:46780/0 (socket says 172.21.15.6:46780) 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.444+0000 7f6d21600640 1 -- 172.21.15.6:0/2308940735 learned_addr learned my addr 172.21.15.6:0/2308940735 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.445+0000 7f6d1be00640 1 -- 172.21.15.6:0/2308940735 >> v2:172.21.15.178:3300/0 conn(0x7f6d1c0cce90 msgr2=0x7f6d1c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.445+0000 7f6d1be00640 1 --2- 172.21.15.6:0/2308940735 >> v2:172.21.15.178:3300/0 conn(0x7f6d1c0cce90 0x7f6d1c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.445+0000 7f6d1be00640 1 -- 172.21.15.6:0/2308940735 >> v1:172.21.15.178:6789/0 conn(0x7f6d1c145300 legacy=0x7f6d1c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.445+0000 7f6d1be00640 1 -- 172.21.15.6:0/2308940735 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c071a90 con 0x7f6d1c070f10 2024-03-06T07:27:46.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.446+0000 7f6d1be00640 1 --2- 172.21.15.6:0/2308940735 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c070f10 0x7f6d1c0cc950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d080097a0 tx=0x7f6d0802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9bec1e2ce5f06ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.446+0000 7f6d1b400640 1 -- 172.21.15.6:0/2308940735 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08035020 con 0x7f6d1c070f10 2024-03-06T07:27:46.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d1b400640 1 -- 172.21.15.6:0/2308940735 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d08031810 con 0x7f6d1c070f10 2024-03-06T07:27:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d1b400640 1 -- 172.21.15.6:0/2308940735 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08031be0 con 0x7f6d1c070f10 2024-03-06T07:27:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 -- 172.21.15.6:0/2308940735 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c070f10 msgr2=0x7f6d1c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 --2- 172.21.15.6:0/2308940735 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c070f10 0x7f6d1c0cc950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d080097a0 tx=0x7f6d0802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 -- 172.21.15.6:0/2308940735 shutdown_connections 2024-03-06T07:27:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 --2- 172.21.15.6:0/2308940735 >> v2:172.21.15.178:3300/0 conn(0x7f6d1c0cce90 0x7f6d1c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 --2- 172.21.15.6:0/2308940735 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c070f10 0x7f6d1c0cc950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 -- 172.21.15.6:0/2308940735 >> 172.21.15.6:0/2308940735 conn(0x7f6d1c06b3d0 msgr2=0x7f6d1c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 -- 172.21.15.6:0/2308940735 shutdown_connections 2024-03-06T07:27:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.447+0000 7f6d22000640 1 -- 172.21.15.6:0/2308940735 wait complete. 2024-03-06T07:27:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.449+0000 7f6d22000640 1 Processor -- start 2024-03-06T07:27:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.449+0000 7f6d22000640 1 -- start start 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d22000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 0x7f6d1c1e5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d22000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c0cce90 0x7f6d1c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d22000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d1c145300 0x7f6d1c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d22000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c15cab0 con 0x7f6d1c070f10 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d22000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c1456e0 con 0x7f6d1c145300 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d22000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c15cc30 con 0x7f6d1c0cce90 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d20c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 0x7f6d1c1e5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.450+0000 7f6d20c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 0x7f6d1c1e5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55096/0 (socket says 172.21.15.6:55096) 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 -- 172.21.15.6:0/2158227159 learned_addr learned my addr 172.21.15.6:0/2158227159 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d21600640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d1c145300 0x7f6d1c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d1be00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c0cce90 0x7f6d1c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 -- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c0cce90 msgr2=0x7f6d1c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c0cce90 0x7f6d1c0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 -- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d1c145300 msgr2=0x7f6d1c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d1c145300 0x7f6d1c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 -- 172.21.15.6:0/2158227159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c1e6f10 con 0x7f6d1c070f10 2024-03-06T07:27:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.451+0000 7f6d20c00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 0x7f6d1c1e5fb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000c3c0 tx=0x7f6d1000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.452+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100052d0 con 0x7f6d1c070f10 2024-03-06T07:27:46.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.452+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d10005490 con 0x7f6d1c070f10 2024-03-06T07:27:46.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.452+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10007260 con 0x7f6d1c070f10 2024-03-06T07:27:46.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.452+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c1e6950 con 0x7f6d1c070f10 2024-03-06T07:27:46.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.452+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c133260 con 0x7f6d1c070f10 2024-03-06T07:27:46.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.453+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1c04f330 con 0x7f6d1c070f10 2024-03-06T07:27:46.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.454+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d10007400 con 0x7f6d1c070f10 2024-03-06T07:27:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.455+0000 7f6d18c00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6cfc06b500 0x7f6cfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.455+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d10089510 con 0x7f6d1c070f10 2024-03-06T07:27:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.456+0000 7f6d1be00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6cfc06b500 0x7f6cfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.457+0000 7f6d1be00640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6cfc06b500 0x7f6cfc06d9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6d08036000 tx=0x7f6d08008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.460+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d10057dd0 con 0x7f6d1c070f10 2024-03-06T07:27:46.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.663+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d1c0b2c30 con 0x7f6d1c070f10 2024-03-06T07:27:46.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.665+0000 7f6d18c00640 1 -- 172.21.15.6:0/2158227159 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f6d1005be70 con 0x7f6d1c070f10 2024-03-06T07:27:46.667 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:46.667 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:46.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6cfc06b500 msgr2=0x7f6cfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6cfc06b500 0x7f6cfc06d9c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6d08036000 tx=0x7f6d08008030 comp rx=0 tx=0).stop 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 msgr2=0x7f6d1c1e5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 0x7f6d1c1e5fb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000c3c0 tx=0x7f6d1000c880 comp rx=0 tx=0).stop 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 shutdown_connections 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6cfc06b500 0x7f6cfc06d9c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d1c145300 0x7f6d1c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d1c0cce90 0x7f6d1c0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 --2- 172.21.15.6:0/2158227159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d1c070f10 0x7f6d1c1e5fb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 >> 172.21.15.6:0/2158227159 conn(0x7f6d1c06b3d0 msgr2=0x7f6d1c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 shutdown_connections 2024-03-06T07:27:46.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.668+0000 7f6d22000640 1 -- 172.21.15.6:0/2158227159 wait complete. 2024-03-06T07:27:46.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:46.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.905+0000 7f3360400640 1 Processor -- start 2024-03-06T07:27:46.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.906+0000 7f3360400640 1 -- start start 2024-03-06T07:27:46.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.907+0000 7f3360400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f335806e770 0x7f33580b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.907+0000 7f3360400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3358072d80 0x7f335815e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.907+0000 7f3360400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3358071400 con 0x7f335815eb10 2024-03-06T07:27:46.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.907+0000 7f3360400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33580b3ed0 con 0x7f335806e770 2024-03-06T07:27:46.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.907+0000 7f3360400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33580716e0 con 0x7f3358072d80 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.908+0000 7f335da00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f335806e770 0x7f33580b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.908+0000 7f335da00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f335806e770 0x7f33580b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:46804/0 (socket says 172.21.15.6:46804) 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.908+0000 7f335da00640 1 -- 172.21.15.6:0/1453176249 learned_addr learned my addr 172.21.15.6:0/1453176249 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.908+0000 7f335d000640 1 --2- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3358072d80 0x7f335815e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.909+0000 7f335da00640 1 -- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3358072d80 msgr2=0x7f335815e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.909+0000 7f335da00640 1 --2- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3358072d80 0x7f335815e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.909+0000 7f335da00640 1 -- 172.21.15.6:0/1453176249 >> v1:172.21.15.178:6789/0 conn(0x7f335815eb10 legacy=0x7f335806cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.909+0000 7f335da00640 1 -- 172.21.15.6:0/1453176249 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33580b4120 con 0x7f335806e770 2024-03-06T07:27:46.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.910+0000 7f335da00640 1 --2- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335806e770 0x7f33580b4920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f334c009e50 tx=0x7f334c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=275db332d3c94fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.910+0000 7f3357e00640 1 -- 172.21.15.6:0/1453176249 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c02c9f0 con 0x7f335806e770 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3357e00640 1 -- 172.21.15.6:0/1453176249 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c02cb90 con 0x7f335806e770 2024-03-06T07:27:46.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3357e00640 1 -- 172.21.15.6:0/1453176249 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0343f0 con 0x7f335806e770 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 -- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335806e770 msgr2=0x7f33580b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 --2- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335806e770 0x7f33580b4920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f334c009e50 tx=0x7f334c02be50 comp rx=0 tx=0).stop 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 -- 172.21.15.6:0/1453176249 shutdown_connections 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 --2- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3358072d80 0x7f335815e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 --2- 172.21.15.6:0/1453176249 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f335806e770 0x7f33580b4920 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 -- 172.21.15.6:0/1453176249 >> 172.21.15.6:0/1453176249 conn(0x7f33580b1bb0 msgr2=0x7f33580b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.911+0000 7f3360400640 1 -- 172.21.15.6:0/1453176249 shutdown_connections 2024-03-06T07:27:46.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.912+0000 7f3360400640 1 -- 172.21.15.6:0/1453176249 wait complete. 2024-03-06T07:27:46.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.914+0000 7f3360400640 1 Processor -- start 2024-03-06T07:27:46.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.914+0000 7f3360400640 1 -- start start 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.915+0000 7f3360400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f335806e770 0x7f3358155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.915+0000 7f3360400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3358072d80 0x7f33581f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.915+0000 7f3360400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 0x7f335815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.915+0000 7f3360400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33580b4e60 con 0x7f335815eb10 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.915+0000 7f3360400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335815ef70 con 0x7f3358072d80 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.915+0000 7f3360400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3358071090 con 0x7f335806e770 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 0x7f335815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 0x7f335815a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55100/0 (socket says 172.21.15.6:55100) 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 -- 172.21.15.6:0/640342908 learned_addr learned my addr 172.21.15.6:0/640342908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335da00640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f335806e770 0x7f3358155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335d000640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3358072d80 0x7f33581f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 -- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f335806e770 msgr2=0x7f3358155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f335806e770 0x7f3358155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 -- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3358072d80 msgr2=0x7f33581f7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3358072d80 0x7f33581f7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335e400640 1 -- 172.21.15.6:0/640342908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33581f3eb0 con 0x7f335815eb10 2024-03-06T07:27:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.916+0000 7f335da00640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f335806e770 0x7f3358155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.917+0000 7f335e400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 0x7f335815a5d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f335000c440 tx=0x7f335000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.917+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350011790 con 0x7f335815eb10 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.917+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3350017070 con 0x7f335815eb10 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.917+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350011b90 con 0x7f335815eb10 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.917+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3358155d50 con 0x7f335815eb10 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.917+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33581f4450 con 0x7f335815eb10 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.918+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335804f3a0 con 0x7f335815eb10 2024-03-06T07:27:46.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.919+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f335001c460 con 0x7f335815eb10 2024-03-06T07:27:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.920+0000 7f3355600640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f333806b4d0 0x7f333806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.920+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3350089fe0 con 0x7f335815eb10 2024-03-06T07:27:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.921+0000 7f335da00640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f333806b4d0 0x7f333806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:46.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.922+0000 7f335da00640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f333806b4d0 0x7f333806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f334c035000 tx=0x7f334c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:46.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:46.924+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3350058e50 con 0x7f335815eb10 2024-03-06T07:27:47.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.166+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f335806ebd0 con 0x7f335815eb10 2024-03-06T07:27:47.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.167+0000 7f3355600640 1 -- 172.21.15.6:0/640342908 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f3350058750 con 0x7f335815eb10 2024-03-06T07:27:47.169 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:27:47.169 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:27:47.169 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:27:47.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.169+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f333806b4d0 msgr2=0x7f333806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.169+0000 7f3360400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f333806b4d0 0x7f333806d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f334c035000 tx=0x7f334c030000 comp rx=0 tx=0).stop 2024-03-06T07:27:47.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.169+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 msgr2=0x7f335815a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.169+0000 7f3360400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 0x7f335815a5d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f335000c440 tx=0x7f335000c900 comp rx=0 tx=0).stop 2024-03-06T07:27:47.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 shutdown_connections 2024-03-06T07:27:47.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f333806b4d0 0x7f333806d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3358072d80 0x7f33581f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f335815eb10 0x7f335815a5d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 --2- 172.21.15.6:0/640342908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f335806e770 0x7f3358155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 >> 172.21.15.6:0/640342908 conn(0x7f33580b1bb0 msgr2=0x7f3358159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 shutdown_connections 2024-03-06T07:27:47.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.170+0000 7f3360400640 1 -- 172.21.15.6:0/640342908 wait complete. 2024-03-06T07:27:47.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:29.674138+0000', 'modified': '2024-03-06T07:27:32.753541+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.178:6837/4203957884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 4203957884}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 4203957884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-03-06T07:27:47.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:47.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:27:47.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.820+0000 7f4612000640 1 Processor -- start 2024-03-06T07:27:47.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.820+0000 7f4612000640 1 -- start start 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4612000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f460c0cd060 0x7f460c16d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4612000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0cdad0 con 0x7f460c15c920 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4612000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c071500 con 0x7f460c0ce850 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4612000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460c0cdd70 con 0x7f460c0cd060 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4610c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f460c15c920 0x7f460c070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52436/0 (socket says 172.21.15.6:52436) 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4610c00640 1 -- 172.21.15.6:0/1102224976 learned_addr learned my addr 172.21.15.6:0/1102224976 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:47.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f4611600640 1 --2- 172.21.15.6:0/1102224976 >> v2:172.21.15.6:3300/0 conn(0x7f460c0cd060 0x7f460c16d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f460be00640 1 -- 172.21.15.6:0/1102224976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788366322 0 0) 0x7f460c0cdad0 con 0x7f460c15c920 2024-03-06T07:27:47.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.822+0000 7f460be00640 1 -- 172.21.15.6:0/1102224976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45ec003520 con 0x7f460c15c920 2024-03-06T07:27:47.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.823+0000 7f4611600640 1 -- 172.21.15.6:0/1102224976 >> v1:172.21.15.178:6789/0 conn(0x7f460c0ce850 legacy=0x7f460c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.823+0000 7f4611600640 1 -- 172.21.15.6:0/1102224976 >> v1:172.21.15.6:6789/0 conn(0x7f460c15c920 legacy=0x7f460c070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.823+0000 7f4611600640 1 -- 172.21.15.6:0/1102224976 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0ce050 con 0x7f460c0cd060 2024-03-06T07:27:47.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.824+0000 7f4611600640 1 --2- 172.21.15.6:0/1102224976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0cd060 0x7f460c16d220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4604009d90 tx=0x7f460402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839ae0c84bcda459 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.824+0000 7f460be00640 1 -- 172.21.15.6:0/1102224976 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604039070 con 0x7f460c0cd060 2024-03-06T07:27:47.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.824+0000 7f460be00640 1 -- 172.21.15.6:0/1102224976 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460402cd00 con 0x7f460c0cd060 2024-03-06T07:27:47.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.824+0000 7f460be00640 1 -- 172.21.15.6:0/1102224976 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604034640 con 0x7f460c0cd060 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 -- 172.21.15.6:0/1102224976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0cd060 msgr2=0x7f460c16d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 --2- 172.21.15.6:0/1102224976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0cd060 0x7f460c16d220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4604009d90 tx=0x7f460402bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 -- 172.21.15.6:0/1102224976 shutdown_connections 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 --2- 172.21.15.6:0/1102224976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0cd060 0x7f460c16d220 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 -- 172.21.15.6:0/1102224976 >> 172.21.15.6:0/1102224976 conn(0x7f460c06b3d0 msgr2=0x7f460c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 -- 172.21.15.6:0/1102224976 shutdown_connections 2024-03-06T07:27:47.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.825+0000 7f4612000640 1 -- 172.21.15.6:0/1102224976 wait complete. 2024-03-06T07:27:47.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.826+0000 7f4612000640 1 Processor -- start 2024-03-06T07:27:47.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.827+0000 7f4612000640 1 -- start start 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.827+0000 7f4612000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f460c0cd060 0x7f460c0b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.827+0000 7f4612000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 0x7f460c15a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.827+0000 7f4612000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f460c15c920 0x7f460c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.827+0000 7f4612000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c071150 con 0x7f460c0ce850 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.827+0000 7f4612000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c163790 con 0x7f460c0cd060 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4612000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c0b0a60 con 0x7f460c15c920 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 0x7f460c15a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 0x7f460c15a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55112/0 (socket says 172.21.15.6:55112) 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 -- 172.21.15.6:0/690297608 learned_addr learned my addr 172.21.15.6:0/690297608 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:47.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4610c00640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f460c0cd060 0x7f460c0b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4611600640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f460c15c920 0x7f460c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 -- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f460c15c920 msgr2=0x7f460c15adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f460c15c920 0x7f460c15adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 -- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f460c0cd060 msgr2=0x7f460c0b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f460c0cd060 0x7f460c0b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.828+0000 7f4603e00640 1 -- 172.21.15.6:0/690297608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0b0d40 con 0x7f460c0ce850 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.829+0000 7f4603e00640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 0x7f460c15a870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00c400 tx=0x7f45fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.829+0000 7f4609600640 1 -- 172.21.15.6:0/690297608 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc0052d0 con 0x7f460c0ce850 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.829+0000 7f4609600640 1 -- 172.21.15.6:0/690297608 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc005490 con 0x7f460c0ce850 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.829+0000 7f4609600640 1 -- 172.21.15.6:0/690297608 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc007260 con 0x7f460c0ce850 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.829+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460c0b1020 con 0x7f460c0ce850 2024-03-06T07:27:47.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.829+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c0b15e0 con 0x7f460c0ce850 2024-03-06T07:27:47.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.831+0000 7f4609600640 1 -- 172.21.15.6:0/690297608 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45fc007400 con 0x7f460c0ce850 2024-03-06T07:27:47.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.832+0000 7f4609600640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45e806b500 0x7f45e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.832+0000 7f4609600640 1 -- 172.21.15.6:0/690297608 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45fc089740 con 0x7f460c0ce850 2024-03-06T07:27:47.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.833+0000 7f4616229740 1 Processor -- start 2024-03-06T07:27:47.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.833+0000 7f4616229740 1 -- start start 2024-03-06T07:27:47.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.833+0000 7f4610c00640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45e806b500 0x7f45e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97165930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4616229740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97195720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97165e70 0x560e97195c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4616229740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e970cf600 con 0x560e97192d70 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e97078930 con 0x560e97165e70 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e970cb6a0 con 0x560e97192990 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97195720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55120/0 (socket says 172.21.15.6:55120) 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 -- 172.21.15.6:0/3830764812 learned_addr learned my addr 172.21.15.6:0/3830764812 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4610c00640 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97165930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4611600640 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97165e70 0x560e97195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4610c00640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45e806b500 0x7f45e806d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45f4005880 tx=0x7f45f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 -- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 msgr2=0x560e97165930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97165930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 -- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97165e70 msgr2=0x560e97195c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97165e70 0x560e97195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.834+0000 7f4603e00640 1 -- 172.21.15.6:0/3830764812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e96de42d0 con 0x560e97192d70 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.835+0000 7f4603e00640 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97195720 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45fc060000 tx=0x7f45fc00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cbdfcaa076bbd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.835+0000 7f4601200640 1 -- 172.21.15.6:0/3830764812 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc056120 con 0x560e97192d70 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.835+0000 7f4601200640 1 -- 172.21.15.6:0/3830764812 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc05e0f0 con 0x560e97192d70 2024-03-06T07:27:47.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.835+0000 7f4601200640 1 -- 172.21.15.6:0/3830764812 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc057270 con 0x560e97192d70 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.835+0000 7f4616229740 1 -- 172.21.15.6:0/3830764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 msgr2=0x560e97195720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.835+0000 7f4616229740 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97195720 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45fc060000 tx=0x7f45fc00a000 comp rx=0 tx=0).stop 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 -- 172.21.15.6:0/3830764812 shutdown_connections 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97165e70 0x560e97195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97195720 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 --2- 172.21.15.6:0/3830764812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97165930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 -- 172.21.15.6:0/3830764812 >> 172.21.15.6:0/3830764812 conn(0x560e97167ba0 msgr2=0x560e971b0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 -- 172.21.15.6:0/3830764812 shutdown_connections 2024-03-06T07:27:47.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 -- 172.21.15.6:0/3830764812 wait complete. 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.836+0000 7f4616229740 1 -- start start 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.837+0000 7f4616229740 1 Processor -- start 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.837+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97191850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.837+0000 7f4616229740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97198a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e971aa730 0x560e971aab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4616229740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e970cf600 con 0x560e97192d70 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e9708bd90 con 0x560e971aa730 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e9716a140 con 0x560e97192990 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4616229740 10 client.0 _ll_register_callbacks cb 0x560e970c7350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55122/0 (socket says 172.21.15.6:55122) 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 -- 172.21.15.6:0/2375611702 learned_addr learned my addr 172.21.15.6:0/2375611702 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4610c00640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97191850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4611600640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e971aa730 0x560e971aab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 -- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 msgr2=0x560e97191850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97191850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 -- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e971aa730 msgr2=0x560e971aab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e971aa730 0x560e971aab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 -- 172.21.15.6:0/2375611702 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560e96eae6b0 con 0x560e97192d70 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f4603e00640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97198a80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f45fc007850 tx=0x7f45fc087aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e765301dab1795 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.838+0000 7f45f9600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f45f9600640 1 -- 172.21.15.6:0/2375611702 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc074060 con 0x560e97192d70 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f45f9600640 1 -- 172.21.15.6:0/2375611702 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45fc06fbb0 con 0x560e97192d70 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45fc05e0f0 con 0x560e97192d70 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f45f9600640 1 -- 172.21.15.6:0/2375611702 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45fc056120 con 0x560e97192d70 2024-03-06T07:27:47.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 10 client.4689 fetch_fsmap learned FSMap version 8 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560e971a90c0 con 0x560e97192d70 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f45f9600640 1 -- 172.21.15.6:0/2375611702 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc05e0f0 con 0x560e97192d70 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f45f9600640 1 -- 172.21.15.6:0/2375611702 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f45fc057270 con 0x560e97192d70 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 10 client.4689 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 10 client.4689 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 20 client.4689 populate_metadata read hostname 'smithi006' 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e971acf90 0x560e971ad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 4 client.4689 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.839+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 --> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560e970dcb40 con 0x560e971acf90 2024-03-06T07:27:47.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.840+0000 7f4610c00640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e971acf90 0x560e971ad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.841+0000 7f4610c00640 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e971acf90 0x560e971ad370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80e761d529f49656 server_cookie=3307570601a7f160 in_seq=0 out_seq=0 2024-03-06T07:27:47.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.841+0000 7f45f9600640 10 client.4689 ms_handle_connect on v2:172.21.15.178:6836/4203957884 2024-03-06T07:27:47.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f45f9600640 1 -- 172.21.15.6:0/2375611702 <== mds.0 v2:172.21.15.178:6836/4203957884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560e970dcb40 con 0x560e971acf90 2024-03-06T07:27:47.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f45f9600640 10 client.4689 handle_command_reply: tid=0 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 client.4689 shutdown 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 msgr2=0x560e97198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97198a80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f45fc007850 tx=0x7f45fc087aa0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 shutdown_connections 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e971aa730 0x560e971aab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e971acf90 0x560e971ad370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192d70 0x560e97198a80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 --2- 172.21.15.6:0/2375611702 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192990 0x560e97191850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 >> 172.21.15.6:0/2375611702 conn(0x560e97165e70 msgr2=0x560e97169840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f45f9600640 0 client.4689 ms_handle_reset on v2:172.21.15.178:6836/4203957884 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.843+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 shutdown_connections 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.844+0000 7f4616229740 1 -- 172.21.15.6:0/2375611702 wait complete. 2024-03-06T07:27:47.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.844+0000 7f4616229740 20 client.4689 trim_cache size 0 max 16384 2024-03-06T07:27:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.862+0000 7f4616229740 1 Processor -- start 2024-03-06T07:27:47.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.862+0000 7f4616229740 1 -- start start 2024-03-06T07:27:47.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4616229740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 0x560e97169530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192d70 0x560e97169a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e971aa730 0x560e97191660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4616229740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c2036d0 con 0x560e97192990 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c202ad0 con 0x560e97192d70 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c2030d0 con 0x560e971aa730 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 0x560e97169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 0x560e97169530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55126/0 (socket says 172.21.15.6:55126) 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 -- 172.21.15.6:0/1671003814 learned_addr learned my addr 172.21.15.6:0/1671003814 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4603e00640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192d70 0x560e97169a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4611600640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e971aa730 0x560e97191660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 -- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e971aa730 msgr2=0x560e97191660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e971aa730 0x560e97191660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 -- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192d70 msgr2=0x560e97169a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192d70 0x560e97169a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4610c00640 1 -- 172.21.15.6:0/1671003814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e97197e60 con 0x560e97192990 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.864+0000 7f4603e00640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192d70 0x560e97169a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.863+0000 7f4611600640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e971aa730 0x560e97191660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:47.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.864+0000 7f4610c00640 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 0x560e97169530 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f460c0cdca0 tx=0x7f45f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936352b2ee09e378 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.864+0000 7f45fbe00640 1 -- 172.21.15.6:0/1671003814 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f401a040 con 0x560e97192990 2024-03-06T07:27:47.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.864+0000 7f45fbe00640 1 -- 172.21.15.6:0/1671003814 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f402a020 con 0x560e97192990 2024-03-06T07:27:47.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.864+0000 7f45fbe00640 1 -- 172.21.15.6:0/1671003814 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f4026d90 con 0x560e97192990 2024-03-06T07:27:47.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 -- 172.21.15.6:0/1671003814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 msgr2=0x560e97169530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 0x560e97169530 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f460c0cdca0 tx=0x7f45f4014000 comp rx=0 tx=0).stop 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 -- 172.21.15.6:0/1671003814 shutdown_connections 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192d70 0x560e97169a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e971aa730 0x560e97191660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 --2- 172.21.15.6:0/1671003814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e97192990 0x560e97169530 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 -- 172.21.15.6:0/1671003814 >> 172.21.15.6:0/1671003814 conn(0x560e97165e70 msgr2=0x560e97160b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 -- 172.21.15.6:0/1671003814 shutdown_connections 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.865+0000 7f4616229740 1 -- 172.21.15.6:0/1671003814 wait complete. 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.866+0000 7f4616229740 1 -- start start 2024-03-06T07:27:47.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.866+0000 7f4616229740 1 Processor -- start 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192990 0x560e971921c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192d70 0x560e9721fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 0x560e9721e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e970cf600 con 0x560e971aa730 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e9708bd90 con 0x560e97192990 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e971a93a0 con 0x560e97192d70 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4616229740 10 client.0 _ll_register_callbacks cb 0x7f45fc058520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 0x560e9721e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 0x560e9721e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55136/0 (socket says 172.21.15.6:55136) 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 -- 172.21.15.6:0/4255107234 learned_addr learned my addr 172.21.15.6:0/4255107234 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4610c00640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192990 0x560e971921c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4603e00640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192d70 0x560e9721fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 -- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192d70 msgr2=0x560e9721fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192d70 0x560e9721fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 -- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192990 msgr2=0x560e971921c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192990 0x560e971921c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4611600640 1 -- 172.21.15.6:0/4255107234 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560e9721eed0 con 0x560e971aa730 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.867+0000 7f4603e00640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192d70 0x560e9721fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f4611600640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 0x560e9721e990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4604009d90 tx=0x7f4604004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d802d755e506aeec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f45f9600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f45f9600640 1 -- 172.21.15.6:0/4255107234 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604044070 con 0x560e971aa730 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f45f9600640 1 -- 172.21.15.6:0/4255107234 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4604004040 con 0x560e971aa730 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45f402a020 con 0x560e971aa730 2024-03-06T07:27:47.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f45f9600640 1 -- 172.21.15.6:0/4255107234 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4604051050 con 0x560e971aa730 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f4616229740 10 client.4695 fetch_fsmap learned FSMap version 8 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.868+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560e971a8f60 con 0x560e971aa730 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f45f9600640 1 -- 172.21.15.6:0/4255107234 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f402a020 con 0x560e971aa730 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f45f9600640 1 -- 172.21.15.6:0/4255107234 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f4604054020 con 0x560e971aa730 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4616229740 10 client.4695 fetch_fsmap finished waiting for FSMap version 8 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4616229740 10 client.4695 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4616229740 20 client.4695 populate_metadata read hostname 'smithi006' 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4616229740 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e97169ba0 0x560e9718aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4616229740 4 client.4695 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 --> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560e970d6e90 con 0x560e97169ba0 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.869+0000 7f4610c00640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e97169ba0 0x560e9718aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.870+0000 7f4610c00640 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e97169ba0 0x560e9718aa40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa9737beb13f54bd server_cookie=ec41339428416941 in_seq=0 out_seq=0 2024-03-06T07:27:47.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.870+0000 7f45f9600640 10 client.4695 ms_handle_connect on v2:172.21.15.178:6836/4203957884 2024-03-06T07:27:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f45f9600640 1 -- 172.21.15.6:0/4255107234 <== mds.0 v2:172.21.15.178:6836/4203957884 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x560e970d6e90 con 0x560e97169ba0 2024-03-06T07:27:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f45f9600640 10 client.4695 handle_command_reply: tid=0 2024-03-06T07:27:47.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 client.4695 shutdown 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 msgr2=0x560e9721e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 0x560e9721e990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4604009d90 tx=0x7f4604004ab0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 shutdown_connections 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:6836/4203957884,v1:172.21.15.178:6837/4203957884] conn(0x560e97169ba0 0x560e9718aa40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x560e971aa730 0x560e9721e990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x560e97192d70 0x560e9721fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 --2- 172.21.15.6:0/4255107234 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560e97192990 0x560e971921c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f45f9600640 0 client.4695 ms_handle_reset on v2:172.21.15.178:6836/4203957884 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 >> 172.21.15.6:0/4255107234 conn(0x560e97165e70 msgr2=0x560e971951e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 shutdown_connections 2024-03-06T07:27:47.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 1 -- 172.21.15.6:0/4255107234 wait complete. 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.871+0000 7f4616229740 20 client.4695 trim_cache size 0 max 16384 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4294, 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 17.631951206, 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 8, 2024-03-06T07:27:47.877 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 22, 2024-03-06T07:27:47.878 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 22, 2024-03-06T07:27:47.878 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 23.710231193999999 2024-03-06T07:27:47.878 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:27:47.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45e806b500 msgr2=0x7f45e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45e806b500 0x7f45e806d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45f4005880 tx=0x7f45f4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 msgr2=0x7f460c15a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 0x7f460c15a870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00c400 tx=0x7f45fc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 shutdown_connections 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45e806b500 0x7f45e806d9c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f460c15c920 0x7f460c15adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f460c0ce850 0x7f460c15a870 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 --2- 172.21.15.6:0/690297608 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f460c0cd060 0x7f460c0b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 >> 172.21.15.6:0/690297608 conn(0x7f460c06b3d0 msgr2=0x7f460c0b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 shutdown_connections 2024-03-06T07:27:47.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:47.872+0000 7f4612000640 1 -- 172.21.15.6:0/690297608 wait complete. 2024-03-06T07:27:47.891 INFO:teuthology.run_tasks:Running task kclient... 2024-03-06T07:27:47.915 INFO:tasks.kclient:Mounting kernel clients... 2024-03-06T07:27:47.915 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:27:47.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:47.915 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:27:47.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:47.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-03-06T07:27:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:48.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:48.075 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:27:48.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:48.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link delete ceph-brx 2024-03-06T07:27:48.220 INFO:teuthology.orchestra.run.smithi006.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:48.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:48.224 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list 2024-03-06T07:27:48.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:48.279 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link delete ceph-brx 2024-03-06T07:27:48.374 INFO:teuthology.orchestra.run.smithi178.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:48.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:48.380 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list 2024-03-06T07:27:48.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:48.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link delete ceph-brx 2024-03-06T07:27:48.558 INFO:teuthology.orchestra.run.smithi178.stderr:Cannot find device "ceph-brx" 2024-03-06T07:27:48.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:27:48.561 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-06T07:27:48.562 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:27:48.562 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:48.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:48.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.784+0000 7f06d2e00640 1 Processor -- start 2024-03-06T07:27:48.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.784+0000 7f06d2e00640 1 -- start start 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.785+0000 7f06d2e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc070ee0 0x7f06cc162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.786+0000 7f06d2e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc0ce090 con 0x7f06cc072da0 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.786+0000 7f06d2e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc0ce370 con 0x7f06cc0712c0 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.786+0000 7f06d2e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc0ceca0 con 0x7f06cc070ee0 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.786+0000 7f06cb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06cc072da0 0x7f06cc06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:52444/0 (socket says 172.21.15.6:52444) 2024-03-06T07:27:48.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.786+0000 7f06cb400640 1 -- 172.21.15.6:0/3444691344 learned_addr learned my addr 172.21.15.6:0/3444691344 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.786+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc070ee0 0x7f06cc162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235333897 0 0) 0x7f06cc0ce090 con 0x7f06cc072da0 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b0003520 con 0x7f06cc072da0 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3886051740 0 0) 0x7f06b0003520 con 0x7f06cc072da0 2024-03-06T07:27:48.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc070ee0 msgr2=0x7f06cc162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc070ee0 0x7f06cc162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 >> v1:172.21.15.178:6789/0 conn(0x7f06cc0712c0 legacy=0x7f06cc06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0ce650 con 0x7f06cc072da0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f06b4001670 con 0x7f06cc072da0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 >> v1:172.21.15.6:6789/0 conn(0x7f06cc072da0 legacy=0x7f06cc06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06b0003a50 0x7f06b0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06b00043d0 0x7f06b00048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc0ceca0 con 0x7f06b00050a0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b000bfb0 con 0x7f06b00043d0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b000c1b0 con 0x7f06b0003a50 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc070ee0 0x7f06cc162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.787+0000 7f06d0e00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06cb400640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06b00043d0 0x7f06b00048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06b0003a50 0x7f06b0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06d0e00640 1 -- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06b0003a50 msgr2=0x7f06b0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06d0e00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06b0003a50 0x7f06b0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06d0e00640 1 -- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06b00043d0 msgr2=0x7f06b00048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06d0e00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06b00043d0 0x7f06b00048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06d0e00640 1 -- 172.21.15.6:0/3444691344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b0003520 con 0x7f06b00050a0 2024-03-06T07:27:48.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.788+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06b0003a50 0x7f06b0003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.789+0000 7f06d0e00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06b000b9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f06c4001f60 tx=0x7f06b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72fb5863db1ca02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.789+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4001b30 con 0x7f06b00050a0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.789+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c402d710 con 0x7f06b00050a0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.789+0000 7f06caa00640 1 -- 172.21.15.6:0/3444691344 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c402db00 con 0x7f06b00050a0 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d2e00640 1 -- 172.21.15.6:0/3444691344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 msgr2=0x7f06b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3444691344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06b000b9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f06c4001f60 tx=0x7f06b0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d0e00640 1 -- 172.21.15.6:0/3444691344 reap_dead start 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d2e00640 1 -- 172.21.15.6:0/3444691344 shutdown_connections 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d2e00640 1 -- 172.21.15.6:0/3444691344 >> 172.21.15.6:0/3444691344 conn(0x7f06cc06afb0 msgr2=0x7f06cc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d2e00640 1 -- 172.21.15.6:0/3444691344 shutdown_connections 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.790+0000 7f06d2e00640 1 -- 172.21.15.6:0/3444691344 wait complete. 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.792+0000 7f06d2e00640 1 Processor -- start 2024-03-06T07:27:48.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.792+0000 7f06d2e00640 1 -- start start 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d2e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06cc15bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d2e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06cc15f040 0x7f06cc15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d2e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc15fa00 0x7f06cc09c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d2e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc0cdda0 con 0x7f06b00050a0 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d2e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc0cdc20 con 0x7f06cc15f040 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d2e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06cc162c40 con 0x7f06cc15fa00 2024-03-06T07:27:48.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06cc15bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06cc15bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55142/0 (socket says 172.21.15.6:55142) 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06cbe00640 1 -- 172.21.15.6:0/3182980009 learned_addr learned my addr 172.21.15.6:0/3182980009 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06cb400640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06cc15f040 0x7f06cc15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.793+0000 7f06d0e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc15fa00 0x7f06cc09c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06cbe00640 1 -- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc15fa00 msgr2=0x7f06cc09c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc15fa00 0x7f06cc09c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06cbe00640 1 -- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06cc15f040 msgr2=0x7f06cc15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06cc15f040 0x7f06cc15f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06cbe00640 1 -- 172.21.15.6:0/3182980009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc160db0 con 0x7f06b00050a0 2024-03-06T07:27:48.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06cbe00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06cc15bfb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f06bc010930 tx=0x7f06bc010df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc00f040 con 0x7f06b00050a0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06bc00ebc0 con 0x7f06b00050a0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc161090 con 0x7f06b00050a0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.794+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06bc01c400 con 0x7f06b00050a0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.795+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc161650 con 0x7f06b00050a0 2024-03-06T07:27:48.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.796+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06cc069510 con 0x7f06b00050a0 2024-03-06T07:27:48.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.797+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06bc01c5a0 con 0x7f06b00050a0 2024-03-06T07:27:48.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.798+0000 7f06c3e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06a806b550 0x7f06a806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:48.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.798+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06bc08a920 con 0x7f06b00050a0 2024-03-06T07:27:48.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.798+0000 7f06cb400640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06a806b550 0x7f06a806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:48.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.799+0000 7f06cb400640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06a806b550 0x7f06a806da10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f06b4001eb0 tx=0x7f06b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:48.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:48.803+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06bc018030 con 0x7f06b00050a0 2024-03-06T07:27:49.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.040+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f06cc09c0f0 con 0x7f06b00050a0 2024-03-06T07:27:49.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.040+0000 7f06c3e00640 1 -- 172.21.15.6:0/3182980009 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f06cc09c0f0 con 0x7f06b00050a0 2024-03-06T07:27:49.043 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06a806b550 msgr2=0x7f06a806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06a806b550 0x7f06a806da10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f06b4001eb0 tx=0x7f06b4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 msgr2=0x7f06cc15bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06cc15bfb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f06bc010930 tx=0x7f06bc010df0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 shutdown_connections 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06a806b550 0x7f06a806da10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06cc15f040 0x7f06cc15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06cc15fa00 0x7f06cc09c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 --2- 172.21.15.6:0/3182980009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f06b00050a0 0x7f06cc15bfb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 >> 172.21.15.6:0/3182980009 conn(0x7f06cc06afb0 msgr2=0x7f06cc163dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 shutdown_connections 2024-03-06T07:27:49.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:49.043+0000 7f06d2e00640 1 -- 172.21.15.6:0/3182980009 wait complete. 2024-03-06T07:27:49.062 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:49.062 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:49.063 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:49.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:49.063 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:49.117 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42588sec preferred_lft 42588sec 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:49.118 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:27:49.119 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:27:49.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr flush dev ceph-brx 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set ceph-brx up 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:27:49.120 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:49.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:49.577 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:49.657 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:27:49.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:49.661 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:27:49.725 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:27:49.725 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:49.725 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:27:49.725 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:27:49.726 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:49.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:49.727 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:49.727 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:49.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:27:49.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:27:49.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:27:49.727 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:50.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:50.484 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:27:50.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:50.543 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-06T07:27:50.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:50.596 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:50.596 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:50.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:50.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:27:50.596 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:50.746 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:27:50.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:27:50.747 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:51.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:51.049 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:51.049 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:51.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:27:51.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:27:51.049 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:51.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:51.104 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:51.212 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:51.212 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:51.258 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:51.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:51.259 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:51.351 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:27:51.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:51.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:51.475 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:51.475 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:27:52.128 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:52.128 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:27:52.128 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:27:52.128 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:27:52.128 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:27:52.130 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:27:52.130 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:27:52.130 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:27:52.131 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:27:52.131 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:27:52.135 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:27:52.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:52.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:52.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:52.216 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:52.289 INFO:teuthology.orchestra.run.smithi006.stdout:client4704 2024-03-06T07:27:52.289 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:27:52.289 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:52.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:52.332 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:52.452 INFO:teuthology.orchestra.run.smithi006.stdout:client4704 2024-03-06T07:27:52.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:52.453 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:52.523 INFO:teuthology.orchestra.run.smithi006.stdout:client4704 2024-03-06T07:27:52.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:52.524 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e57361cb-720d-49c4-a325-3e55e1c43024.client4704/status') 2024-03-06T07:27:52.626 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.4704 (3)192.168.0.1:0/2050158191 2024-03-06T07:27:52.626 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-03-06T07:27:52.626 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-03-06T07:27:52.626 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-03-06T07:27:52.626 INFO:teuthology.orchestra.run.smithi006.stderr:69 bytes copied, 8.0476e-05 s, 857 kB/s 2024-03-06T07:27:52.631 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-06T07:27:52.631 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:27:52.631 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:27:52.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:52.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.845+0000 7f2e34e00640 1 Processor -- start 2024-03-06T07:27:52.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.846+0000 7f2e34e00640 1 -- start start 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.847+0000 7f2e34e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e3015e030 0x7f2e30161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.847+0000 7f2e34e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e300cfd50 0x7f2e30162260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.847+0000 7f2e34e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2e3015e410 0x7f2e300d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.847+0000 7f2e34e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300cea50 con 0x7f2e3015e030 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.847+0000 7f2e34e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300cecf0 con 0x7f2e3015e410 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.847+0000 7f2e34e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300cef90 con 0x7f2e300cfd50 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e3015e030 0x7f2e30161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e3015e030 0x7f2e30161d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47304/0 (socket says 172.21.15.6:47304) 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 -- 172.21.15.6:0/1224631787 learned_addr learned my addr 172.21.15.6:0/1224631787 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2d400640 1 --2- 172.21.15.6:0/1224631787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e300cfd50 0x7f2e30162260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2e800640 1 --2- 172.21.15.6:0/1224631787 >> v2:172.21.15.178:3300/0 conn(0x7f2e3015e410 0x7f2e300d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 -- 172.21.15.6:0/1224631787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e300cfd50 msgr2=0x7f2e30162260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 --2- 172.21.15.6:0/1224631787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e300cfd50 0x7f2e30162260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 -- 172.21.15.6:0/1224631787 >> v2:172.21.15.178:3300/0 conn(0x7f2e3015e410 msgr2=0x7f2e300d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 --2- 172.21.15.6:0/1224631787 >> v2:172.21.15.178:3300/0 conn(0x7f2e3015e410 0x7f2e300d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2de00640 1 -- 172.21.15.6:0/1224631787 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300cf270 con 0x7f2e3015e030 2024-03-06T07:27:52.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.848+0000 7f2e2e800640 1 --2- 172.21.15.6:0/1224631787 >> v2:172.21.15.178:3300/0 conn(0x7f2e3015e410 0x7f2e300d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:52.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.849+0000 7f2e2de00640 1 --2- 172.21.15.6:0/1224631787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e3015e030 0x7f2e30161d20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2e30162cb0 tx=0x7f2e1802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86942b70b94ae6e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e2ca00640 1 -- 172.21.15.6:0/1224631787 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18034070 con 0x7f2e3015e030 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e2ca00640 1 -- 172.21.15.6:0/1224631787 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e18030850 con 0x7f2e3015e030 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e2ca00640 1 -- 172.21.15.6:0/1224631787 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18030c20 con 0x7f2e3015e030 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 -- 172.21.15.6:0/1224631787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e3015e030 msgr2=0x7f2e30161d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 --2- 172.21.15.6:0/1224631787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e3015e030 0x7f2e30161d20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2e30162cb0 tx=0x7f2e1802b990 comp rx=0 tx=0).stop 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 -- 172.21.15.6:0/1224631787 shutdown_connections 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 --2- 172.21.15.6:0/1224631787 >> v2:172.21.15.178:3300/0 conn(0x7f2e3015e410 0x7f2e300d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 --2- 172.21.15.6:0/1224631787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e300cfd50 0x7f2e30162260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 --2- 172.21.15.6:0/1224631787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e3015e030 0x7f2e30161d20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 -- 172.21.15.6:0/1224631787 >> 172.21.15.6:0/1224631787 conn(0x7f2e300d3eb0 msgr2=0x7f2e300d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.850+0000 7f2e34e00640 1 -- 172.21.15.6:0/1224631787 shutdown_connections 2024-03-06T07:27:52.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.851+0000 7f2e34e00640 1 -- 172.21.15.6:0/1224631787 wait complete. 2024-03-06T07:27:52.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.852+0000 7f2e34e00640 1 Processor -- start 2024-03-06T07:27:52.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.852+0000 7f2e34e00640 1 -- start start 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.853+0000 7f2e34e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e3015e030 0x7f2e301f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.853+0000 7f2e34e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e3015e410 0x7f2e300b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.853+0000 7f2e34e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 0x7f2e300c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.853+0000 7f2e34e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300d29d0 con 0x7f2e300c8c30 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.853+0000 7f2e34e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300cf830 con 0x7f2e3015e030 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.853+0000 7f2e34e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300d2b50 con 0x7f2e3015e410 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 0x7f2e300c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 0x7f2e300c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47312/0 (socket says 172.21.15.6:47312) 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 -- 172.21.15.6:0/2498694121 learned_addr learned my addr 172.21.15.6:0/2498694121 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2de00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e3015e030 0x7f2e301f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2d400640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e3015e410 0x7f2e300b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 -- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e3015e410 msgr2=0x7f2e300b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e3015e410 0x7f2e300b7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 -- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e3015e030 msgr2=0x7f2e301f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e3015e030 0x7f2e301f4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:52.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 -- 172.21.15.6:0/2498694121 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e301f5500 con 0x7f2e300c8c30 2024-03-06T07:27:52.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.854+0000 7f2e2e800640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 0x7f2e300c9810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800c570 tx=0x7f2e2800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.855+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28015070 con 0x7f2e300c8c30 2024-03-06T07:27:52.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.855+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e28004220 con 0x7f2e300c8c30 2024-03-06T07:27:52.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.855+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28003610 con 0x7f2e300c8c30 2024-03-06T07:27:52.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.855+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300c7b90 con 0x7f2e300c8c30 2024-03-06T07:27:52.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.855+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300c8150 con 0x7f2e300c8c30 2024-03-06T07:27:52.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.856+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300b2a90 con 0x7f2e300c8c30 2024-03-06T07:27:52.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.857+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e280037b0 con 0x7f2e300c8c30 2024-03-06T07:27:52.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.858+0000 7f2e26000640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e0c06b4d0 0x7f2e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:52.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.858+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e28089fe0 con 0x7f2e300c8c30 2024-03-06T07:27:52.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.858+0000 7f2e2de00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e0c06b4d0 0x7f2e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:52.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.859+0000 7f2e2de00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e0c06b4d0 0x7f2e0c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e18036000 tx=0x7f2e1803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:52.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:52.862+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e300b2a90 con 0x7f2e300c8c30 2024-03-06T07:27:53.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.107+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2e3004f3a0 con 0x7f2e300c8c30 2024-03-06T07:27:53.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.108+0000 7f2e26000640 1 -- 172.21.15.6:0/2498694121 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2e301f5500 con 0x7f2e300c8c30 2024-03-06T07:27:53.110 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e0c06b4d0 msgr2=0x7f2e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e0c06b4d0 0x7f2e0c06d990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e18036000 tx=0x7f2e1803b040 comp rx=0 tx=0).stop 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 msgr2=0x7f2e300c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 0x7f2e300c9810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800c570 tx=0x7f2e2800ca30 comp rx=0 tx=0).stop 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 shutdown_connections 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e0c06b4d0 0x7f2e0c06d990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e300c8c30 0x7f2e300c9810 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e3015e410 0x7f2e300b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 --2- 172.21.15.6:0/2498694121 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e3015e030 0x7f2e301f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:53.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.110+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 >> 172.21.15.6:0/2498694121 conn(0x7f2e300d3eb0 msgr2=0x7f2e3006d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:53.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.111+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 shutdown_connections 2024-03-06T07:27:53.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:53.111+0000 7f2e34e00640 1 -- 172.21.15.6:0/2498694121 wait complete. 2024-03-06T07:27:53.130 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:53.130 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:53.130 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:27:53.130 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:27:53.131 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:27:53.131 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:53.131 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:53.131 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:53.131 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:27:53.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.131 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:27:53.179 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:53.179 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:27:53.180 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42584sec preferred_lft 42584sec 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:27:53.181 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.182 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:53.269 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:27:53.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.274 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:27:53.324 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:27:53.325 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:53.325 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:27:53.325 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:27:53.325 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:53.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.326 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:53.326 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:53.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:27:53.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:27:53.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:27:53.326 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:53.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.523 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:27:53.573 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:53.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.574 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list-id 2024-03-06T07:27:53.623 INFO:teuthology.orchestra.run.smithi006.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:27:53.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.624 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:53.624 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:53.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:53.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:27:53.625 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:53.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:27:53.864 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:7e:e8:79:16:1a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c7e:e8ff:fe79:161a/64 scope link proto kernel_ll 2024-03-06T07:27:53.865 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:53.966 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 7e:7e:e8:79:16:1a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:27:53.967 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:27:53.967 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.967 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7c7e:e8ff:fe79:161a/64 scope link proto kernel_ll 2024-03-06T07:27:53.967 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:53.971 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:27:53.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:53.971 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:53.971 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:53.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:27:53.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:27:53.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:27:53.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:27:53.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:27:53.972 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:54.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.249 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:27:54.249 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:27:54.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.1 up 2024-03-06T07:27:54.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:27:54.249 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:27:54.426 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:54.426 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:27:54.477 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:27:54.478 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:54.478 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:27:54.532 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-06T07:27:54.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:54.676 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:54.676 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:27:54.711 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:54.711 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:27:54.711 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:27:54.711 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 2@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:27:54.711 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:27:54.713 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-06T07:27:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:27:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:27:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:27:54.713 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:27:54.719 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:27:54.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:27:54.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.811 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:54.867 INFO:teuthology.orchestra.run.smithi006.stdout:client4713 2024-03-06T07:27:54.867 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:27:54.867 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:54.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:54.913 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:55.020 INFO:teuthology.orchestra.run.smithi006.stdout:client4713 2024-03-06T07:27:55.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.020 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:55.081 INFO:teuthology.orchestra.run.smithi006.stdout:client4713 2024-03-06T07:27:55.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.082 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e57361cb-720d-49c4-a325-3e55e1c43024.client4713/status') 2024-03-06T07:27:55.181 INFO:teuthology.orchestra.run.smithi006.stdout:instance: client.4713 (3)192.168.0.2:0/273819986 2024-03-06T07:27:55.181 INFO:teuthology.orchestra.run.smithi006.stdout:blocklisted: false 2024-03-06T07:27:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records in 2024-03-06T07:27:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:0+1 records out 2024-03-06T07:27:55.181 INFO:teuthology.orchestra.run.smithi006.stderr:68 bytes copied, 7.5005e-05 s, 907 kB/s 2024-03-06T07:27:55.186 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-06T07:27:55.186 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:27:55.186 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:27:55.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.409+0000 7ff20a000640 1 Processor -- start 2024-03-06T07:27:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.409+0000 7ff20a000640 1 -- start start 2024-03-06T07:27:55.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff20a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff204071380 0x7ff2040ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff20a000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff2040ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff20a000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040ce9a0 con 0x7ff204161670 2024-03-06T07:27:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff20a000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2040730d0 con 0x7ff204071380 2024-03-06T07:27:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff20a000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cec80 con 0x7ff20406e240 2024-03-06T07:27:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff202a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff204071380 0x7ff2040ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff203400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff204161670 0x7ff204070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60776/0 (socket says 172.21.15.6:60776) 2024-03-06T07:27:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.411+0000 7ff203400640 1 -- 172.21.15.6:0/1683677773 learned_addr learned my addr 172.21.15.6:0/1683677773 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.412+0000 7ff203e00640 1 --2- 172.21.15.6:0/1683677773 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff2040ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.412+0000 7ff202a00640 1 -- 172.21.15.6:0/1683677773 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 msgr2=0x7ff2040ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.412+0000 7ff202a00640 1 --2- 172.21.15.6:0/1683677773 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff2040ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.412+0000 7ff202a00640 1 -- 172.21.15.6:0/1683677773 >> v1:172.21.15.6:6789/0 conn(0x7ff204161670 legacy=0x7ff204070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.412+0000 7ff202a00640 1 -- 172.21.15.6:0/1683677773 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2040cef60 con 0x7ff204071380 2024-03-06T07:27:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.412+0000 7ff202000640 1 -- 172.21.15.6:0/1683677773 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341104618 0 0) 0x7ff2040ce9a0 con 0x7ff204161670 2024-03-06T07:27:55.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.413+0000 7ff202a00640 1 --2- 172.21.15.6:0/1683677773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff2040ad090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00097a0 tx=0x7ff1f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8f8d0d0aae7878 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.413+0000 7ff202000640 1 -- 172.21.15.6:0/1683677773 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f00315a0 con 0x7ff204071380 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.413+0000 7ff202000640 1 -- 172.21.15.6:0/1683677773 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f0037070 con 0x7ff204071380 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.413+0000 7ff202000640 1 -- 172.21.15.6:0/1683677773 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0031960 con 0x7ff204071380 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 -- 172.21.15.6:0/1683677773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 msgr2=0x7ff2040ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 --2- 172.21.15.6:0/1683677773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff2040ad090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00097a0 tx=0x7ff1f002b840 comp rx=0 tx=0).stop 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 -- 172.21.15.6:0/1683677773 shutdown_connections 2024-03-06T07:27:55.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 --2- 172.21.15.6:0/1683677773 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff2040ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 --2- 172.21.15.6:0/1683677773 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff2040ad090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 -- 172.21.15.6:0/1683677773 >> 172.21.15.6:0/1683677773 conn(0x7ff20406afb0 msgr2=0x7ff20406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 -- 172.21.15.6:0/1683677773 shutdown_connections 2024-03-06T07:27:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.414+0000 7ff20a000640 1 -- 172.21.15.6:0/1683677773 wait complete. 2024-03-06T07:27:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.416+0000 7ff20a000640 1 Processor -- start 2024-03-06T07:27:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.416+0000 7ff20a000640 1 -- start start 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.417+0000 7ff20a000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff20409be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.417+0000 7ff20a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff20414a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.417+0000 7ff20a000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff204161670 0x7ff2040cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.417+0000 7ff20a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040ce600 con 0x7ff204071380 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.417+0000 7ff20a000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cf520 con 0x7ff204161670 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.417+0000 7ff20a000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040ce780 con 0x7ff20406e240 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff20414a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff20414a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47336/0 (socket says 172.21.15.6:47336) 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 -- 172.21.15.6:0/1206470289 learned_addr learned my addr 172.21.15.6:0/1206470289 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff203400640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff20409be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff203e00640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff204161670 0x7ff2040cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 -- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 msgr2=0x7ff20409be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff20409be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 -- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff204161670 msgr2=0x7ff2040cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff204161670 0x7ff2040cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff202a00640 1 -- 172.21.15.6:0/1206470289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20409add0 con 0x7ff204071380 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff203400640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff20409be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.418+0000 7ff203e00640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff204161670 0x7ff2040cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.419+0000 7ff202a00640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff20414a610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00097a0 tx=0x7ff1f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.419+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0004230 con 0x7ff204071380 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.419+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f00043d0 con 0x7ff204071380 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.419+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0003850 con 0x7ff204071380 2024-03-06T07:27:55.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.419+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20409b0b0 con 0x7ff204071380 2024-03-06T07:27:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.419+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20409b670 con 0x7ff204071380 2024-03-06T07:27:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.420+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff204069510 con 0x7ff204071380 2024-03-06T07:27:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.421+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1f00039f0 con 0x7ff204071380 2024-03-06T07:27:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.422+0000 7ff1fb400640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff1e406b4d0 0x7ff1e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.422+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1f002f030 con 0x7ff204071380 2024-03-06T07:27:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.423+0000 7ff203400640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff1e406b4d0 0x7ff1e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.424+0000 7ff203400640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff1e406b4d0 0x7ff1e406d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000f60 tx=0x7ff1f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:55.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.427+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1f0041070 con 0x7ff204071380 2024-03-06T07:27:55.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.668+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff2040b18f0 con 0x7ff204071380 2024-03-06T07:27:55.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.669+0000 7ff1fb400640 1 -- 172.21.15.6:0/1206470289 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff2040b18f0 con 0x7ff204071380 2024-03-06T07:27:55.671 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff1e406b4d0 msgr2=0x7ff1e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff1e406b4d0 0x7ff1e406d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000f60 tx=0x7ff1f4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 msgr2=0x7ff20414a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff20414a610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00097a0 tx=0x7ff1f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 shutdown_connections 2024-03-06T07:27:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff1e406b4d0 0x7ff1e406d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff204161670 0x7ff2040cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff204071380 0x7ff20414a610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 --2- 172.21.15.6:0/1206470289 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff20406e240 0x7ff20409be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:55.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 >> 172.21.15.6:0/1206470289 conn(0x7ff20406afb0 msgr2=0x7ff2040ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:55.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 shutdown_connections 2024-03-06T07:27:55.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:55.672+0000 7ff20a000640 1 -- 172.21.15.6:0/1206470289 wait complete. 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi178.front.sepia.ceph.com 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:55.694 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:55.695 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:27:55.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.695 DEBUG:teuthology.orchestra.run.smithi178:> ip addr 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:d9:3b:20 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: altname enp7s0f0 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:55.747 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:d9:3b:21 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: altname enp7s0f1 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:88:7e:f8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:88:7e:f9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: inet 172.21.15.178/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft 42555sec preferred_lft 42555sec 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::ec4:7aff:fe88:7ef9/64 scope link proto kernel_ll 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:55.748 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:27:55.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:55.748 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:27:55.749 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:27:55.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:27:55.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip addr flush dev ceph-brx 2024-03-06T07:27:55.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set ceph-brx up 2024-03-06T07:27:55.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:27:55.749 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:27:56.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:56.213 DEBUG:teuthology.orchestra.run.smithi178:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:56.287 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-03-06T07:27:56.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:56.290 DEBUG:teuthology.orchestra.run.smithi178:> route 2024-03-06T07:27:56.343 INFO:teuthology.orchestra.run.smithi178.stdout:Kernel IP routing table 2024-03-06T07:27:56.343 INFO:teuthology.orchestra.run.smithi178.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:56.343 INFO:teuthology.orchestra.run.smithi178.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:56.343 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:56.344 INFO:teuthology.orchestra.run.smithi178.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:56.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:56.344 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:27:56.344 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:27:56.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:56.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:56.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:56.344 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:27:57.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:57.040 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list 2024-03-06T07:27:57.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:57.092 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list-id 2024-03-06T07:27:57.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:57.138 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:27:57.138 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:27:57.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:27:57.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:27:57.138 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:27:57.327 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:27:57.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:27:57.327 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:27:57.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:57.588 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:27:57.588 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:27:57.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set brx.0 up 2024-03-06T07:27:57.589 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:27:57.589 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:27:57.744 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:57.744 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run.smithi178.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:27:57.783 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:57.783 DEBUG:teuthology.orchestra.run.smithi178:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:27:57.872 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-06T07:27:57.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:57.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:58.037 INFO:teuthology.orchestra.run.smithi178.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:58.037 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:27:58.684 INFO:teuthology.orchestra.run.smithi178.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:27:58.685 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:27:58.685 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:27:58.685 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: trying mount with new device syntax: 1@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:27:58.685 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:27:58.687 INFO:teuthology.orchestra.run.smithi178.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-06T07:27:58.687 INFO:teuthology.orchestra.run.smithi178.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:27:58.687 INFO:teuthology.orchestra.run.smithi178.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:27:58.687 INFO:teuthology.orchestra.run.smithi178.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:27:58.688 INFO:teuthology.orchestra.run.smithi178.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:27:58.691 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:27:58.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:58.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:27:58.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:58.775 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:58.827 INFO:teuthology.orchestra.run.smithi178.stdout:client4469 2024-03-06T07:27:58.827 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:27:58.827 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:58.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:58.872 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:58.966 INFO:teuthology.orchestra.run.smithi178.stdout:client4469 2024-03-06T07:27:58.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:58.966 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:27:59.010 INFO:teuthology.orchestra.run.smithi178.stdout:client4469 2024-03-06T07:27:59.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.011 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e57361cb-720d-49c4-a325-3e55e1c43024.client4469/status') 2024-03-06T07:27:59.133 INFO:teuthology.orchestra.run.smithi178.stdout:instance: client.4469 (3)192.168.0.1:0/3437114021 2024-03-06T07:27:59.133 INFO:teuthology.orchestra.run.smithi178.stdout:blocklisted: false 2024-03-06T07:27:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:0+1 records in 2024-03-06T07:27:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:0+1 records out 2024-03-06T07:27:59.134 INFO:teuthology.orchestra.run.smithi178.stderr:69 bytes copied, 5.7872e-05 s, 1.2 MB/s 2024-03-06T07:27:59.137 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-06T07:27:59.137 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:27:59.137 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:27:59.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:59.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.364+0000 7faf02200640 1 Processor -- start 2024-03-06T07:27:59.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.364+0000 7faf02200640 1 -- start start 2024-03-06T07:27:59.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.366+0000 7faf02200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 0x7faefc162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.366+0000 7faf02200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc161a80 con 0x7faefc15dd00 2024-03-06T07:27:59.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.366+0000 7faf02200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faefc15efa0 con 0x7faefc15e0e0 2024-03-06T07:27:59.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.366+0000 7faf02200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faefc15f280 con 0x7faefc06c6d0 2024-03-06T07:27:59.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.366+0000 7faefb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faefc15dd00 0x7faefc06bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60788/0 (socket says 172.21.15.6:60788) 2024-03-06T07:27:59.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.366+0000 7faefb400640 1 -- 172.21.15.6:0/2784269694 learned_addr learned my addr 172.21.15.6:0/2784269694 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:27:59.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faef3e00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 0x7faefc162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587233823 0 0) 0x7faefc161a80 con 0x7faefc15dd00 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee0003520 con 0x7faefc15dd00 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2844567452 0 0) 0x7faee0003520 con 0x7faefc15dd00 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 msgr2=0x7faefc162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 0x7faefc162520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 >> v1:172.21.15.178:6789/0 conn(0x7faefc15e0e0 legacy=0x7faefc15e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc161d20 con 0x7faefc15dd00 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7faeec001690 con 0x7faefc15dd00 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 >> v1:172.21.15.6:6789/0 conn(0x7faefc15dd00 legacy=0x7faefc06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0003a50 0x7faee0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.367+0000 7faefaa00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 0x7faee0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefaa00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee0004d90 0x7faee0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc15f280 con 0x7faee0004d90 2024-03-06T07:27:59.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee000bb80 con 0x7faee0003a50 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee000bd80 con 0x7faefc06c6d0 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefbe00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee0004d90 0x7faee0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefb400640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0003a50 0x7faee0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faef3e00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 0x7faee0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefbe00640 1 -- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 msgr2=0x7faee0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefbe00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc06c6d0 0x7faee0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefbe00640 1 -- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0003a50 msgr2=0x7faee0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefbe00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0003a50 0x7faee0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.368+0000 7faefbe00640 1 -- 172.21.15.6:0/2784269694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0003520 con 0x7faee0004d90 2024-03-06T07:27:59.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.369+0000 7faefbe00640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee0004d90 0x7faee0005210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faef4001e50 tx=0x7faee0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8d30adf6bae346 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.370+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4001a70 con 0x7faee0004d90 2024-03-06T07:27:59.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.370+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef4003b70 con 0x7faee0004d90 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.370+0000 7faefaa00640 1 -- 172.21.15.6:0/2784269694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef40050d0 con 0x7faee0004d90 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.370+0000 7faf02200640 1 -- 172.21.15.6:0/2784269694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee0004d90 msgr2=0x7faee0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.370+0000 7faf02200640 1 --2- 172.21.15.6:0/2784269694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee0004d90 0x7faee0005210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faef4001e50 tx=0x7faee0005830 comp rx=0 tx=0).stop 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.370+0000 7faefbe00640 1 -- 172.21.15.6:0/2784269694 reap_dead start 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.371+0000 7faf02200640 1 -- 172.21.15.6:0/2784269694 shutdown_connections 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.371+0000 7faf02200640 1 -- 172.21.15.6:0/2784269694 >> 172.21.15.6:0/2784269694 conn(0x7faefc0714d0 msgr2=0x7faefc0718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.371+0000 7faf02200640 1 -- 172.21.15.6:0/2784269694 shutdown_connections 2024-03-06T07:27:59.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.371+0000 7faf02200640 1 -- 172.21.15.6:0/2784269694 wait complete. 2024-03-06T07:27:59.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.372+0000 7faf02200640 1 Processor -- start 2024-03-06T07:27:59.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 -- start start 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0004d90 0x7faefc14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 0x7faefc06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc157a80 0x7faefc157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc06f280 con 0x7faefc06b440 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc15ec40 con 0x7faee0004d90 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.373+0000 7faf02200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faefc162520 con 0x7faefc157a80 2024-03-06T07:27:59.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 0x7faefc06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 0x7faefc06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47350/0 (socket says 172.21.15.6:47350) 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 -- 172.21.15.6:0/1040464507 learned_addr learned my addr 172.21.15.6:0/1040464507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faefb400640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0004d90 0x7faefc14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faefbe00640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc157a80 0x7faefc157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 -- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc157a80 msgr2=0x7faefc157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc157a80 0x7faefc157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 -- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0004d90 msgr2=0x7faefc14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0004d90 0x7faefc14fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faef3e00640 1 -- 172.21.15.6:0/1040464507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc1503e0 con 0x7faefc06b440 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.374+0000 7faefbe00640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc157a80 0x7faefc157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.375+0000 7faef3e00640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 0x7faefc06b8c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faee400f5b0 tx=0x7faee400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.375+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4019020 con 0x7faefc06b440 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.375+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee4014870 con 0x7faefc06b440 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.375+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4014c60 con 0x7faefc06b440 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.375+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc1587e0 con 0x7faefc06b440 2024-03-06T07:27:59.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.375+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc158da0 con 0x7faefc06b440 2024-03-06T07:27:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.376+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faefc04f3a0 con 0x7faefc06b440 2024-03-06T07:27:59.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.377+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faee401c020 con 0x7faefc06b440 2024-03-06T07:27:59.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.378+0000 7faef3400640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faed806b4d0 0x7faed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:59.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.378+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faee408a9f0 con 0x7faefc06b440 2024-03-06T07:27:59.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.379+0000 7faefb400640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faed806b4d0 0x7faed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:59.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.380+0000 7faefb400640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faed806b4d0 0x7faed806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faeec001eb0 tx=0x7faeec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:59.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.383+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faee4023020 con 0x7faefc06b440 2024-03-06T07:27:59.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.623+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faefc0724c0 con 0x7faefc06b440 2024-03-06T07:27:59.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.624+0000 7faef3400640 1 -- 172.21.15.6:0/1040464507 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7faee4059860 con 0x7faefc06b440 2024-03-06T07:27:59.626 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faed806b4d0 msgr2=0x7faed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faed806b4d0 0x7faed806d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faeec001eb0 tx=0x7faeec009040 comp rx=0 tx=0).stop 2024-03-06T07:27:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 msgr2=0x7faefc06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 0x7faefc06b8c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faee400f5b0 tx=0x7faee400fa70 comp rx=0 tx=0).stop 2024-03-06T07:27:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 shutdown_connections 2024-03-06T07:27:59.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faed806b4d0 0x7faed806d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faefc157a80 0x7faefc157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faefc06b440 0x7faefc06b8c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 --2- 172.21.15.6:0/1040464507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee0004d90 0x7faefc14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 >> 172.21.15.6:0/1040464507 conn(0x7faefc0714d0 msgr2=0x7faefc0b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 shutdown_connections 2024-03-06T07:27:59.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:27:59.627+0000 7faf02200640 1 -- 172.21.15.6:0/1040464507 wait complete. 2024-03-06T07:27:59.646 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:59.646 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi178.front.sepia.ceph.com 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:59.647 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:27:59.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.647 DEBUG:teuthology.orchestra.run.smithi178:> ip addr 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:59.704 INFO:teuthology.orchestra.run.smithi178.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:d9:3b:20 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: altname enp7s0f0 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:d9:3b:21 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: altname enp7s0f1 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:88:7e:f8 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 0c:c4:7a:88:7e:f9 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: inet 172.21.15.178/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft 42551sec preferred_lft 42551sec 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::ec4:7aff:fe88:7ef9/64 scope link proto kernel_ll 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:59.705 INFO:teuthology.orchestra.run.smithi178.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether ce:ab:ad:ca:71:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::885:eaff:fe57:611e/64 scope link proto kernel_ll 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether ce:ab:ad:ca:71:f4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::ccab:adff:feca:71f4/64 scope link proto kernel_ll 2024-03-06T07:27:59.706 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:59.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.707 DEBUG:teuthology.orchestra.run.smithi178:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:59.789 INFO:teuthology.orchestra.run.smithi178.stdout:1 2024-03-06T07:27:59.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.793 DEBUG:teuthology.orchestra.run.smithi178:> route 2024-03-06T07:27:59.840 INFO:teuthology.orchestra.run.smithi178.stdout:Kernel IP routing table 2024-03-06T07:27:59.840 INFO:teuthology.orchestra.run.smithi178.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:59.840 INFO:teuthology.orchestra.run.smithi178.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:59.840 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:59.840 INFO:teuthology.orchestra.run.smithi178.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:59.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:59.840 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:27:59.841 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:27:59.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:59.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:59.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:59.841 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:28:00.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.059 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list 2024-03-06T07:28:00.108 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:28:00.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.109 DEBUG:teuthology.orchestra.run.smithi178:> ip netns list-id 2024-03-06T07:28:00.156 INFO:teuthology.orchestra.run.smithi178.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:28:00.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.157 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:28:00.157 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:28:00.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:28:00.157 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:28:00.157 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:28:00.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:28:00.379 INFO:teuthology.orchestra.run.smithi178.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 9a:fa:a4:41:8f:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::98fa:a4ff:fe41:8f14/64 scope link proto kernel_ll 2024-03-06T07:28:00.380 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:28:00.476 INFO:teuthology.orchestra.run.smithi178.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:00.476 INFO:teuthology.orchestra.run.smithi178.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:00.476 INFO:teuthology.orchestra.run.smithi178.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 ::1/128 scope host proto kernel_lo 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: link/ether 9a:fa:a4:41:8f:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: inet6 fe80::98fa:a4ff:fe41:8f14/64 scope link proto kernel_ll 2024-03-06T07:28:00.477 INFO:teuthology.orchestra.run.smithi178.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:00.480 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:28:00.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:28:00.481 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:28:00.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:00.755 DEBUG:teuthology.orchestra.run.smithi178:> (cd / && exec bash -c ' 2024-03-06T07:28:00.755 DEBUG:teuthology.orchestra.run.smithi178:> set -e 2024-03-06T07:28:00.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set brx.1 up 2024-03-06T07:28:00.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:28:00.755 DEBUG:teuthology.orchestra.run.smithi178:> ') 2024-03-06T07:28:00.918 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:00.919 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:00.963 INFO:teuthology.orchestra.run.smithi178.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:28:00.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:00.964 DEBUG:teuthology.orchestra.run.smithi178:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:01.051 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-06T07:28:01.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:01.224 INFO:teuthology.orchestra.run.smithi178.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:01.224 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:01.259 INFO:teuthology.orchestra.run.smithi178.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:01.259 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:01.259 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:28:01.259 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: trying mount with new device syntax: 3@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:28:01.259 INFO:teuthology.orchestra.run.smithi178.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:28:01.261 INFO:teuthology.orchestra.run.smithi178.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-06T07:28:01.261 INFO:teuthology.orchestra.run.smithi178.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:01.261 INFO:teuthology.orchestra.run.smithi178.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:01.261 INFO:teuthology.orchestra.run.smithi178.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:01.261 INFO:teuthology.orchestra.run.smithi178.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:01.264 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:01.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:28:01.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.345 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:01.399 INFO:teuthology.orchestra.run.smithi178.stdout:client4475 2024-03-06T07:28:01.399 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:01.399 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:01.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.449 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:01.545 INFO:teuthology.orchestra.run.smithi178.stdout:client4475 2024-03-06T07:28:01.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.545 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:01.595 INFO:teuthology.orchestra.run.smithi178.stdout:client4475 2024-03-06T07:28:01.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:01.596 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e57361cb-720d-49c4-a325-3e55e1c43024.client4475/status') 2024-03-06T07:28:01.671 INFO:teuthology.orchestra.run.smithi178.stdout:instance: client.4475 (3)192.168.0.2:0/3047285095 2024-03-06T07:28:01.671 INFO:teuthology.orchestra.run.smithi178.stdout:blocklisted: false 2024-03-06T07:28:01.671 INFO:teuthology.orchestra.run.smithi178.stderr:0+1 records in 2024-03-06T07:28:01.671 INFO:teuthology.orchestra.run.smithi178.stderr:0+1 records out 2024-03-06T07:28:01.671 INFO:teuthology.orchestra.run.smithi178.stderr:69 bytes copied, 6.6794e-05 s, 1.0 MB/s 2024-03-06T07:28:01.675 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:28:01.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:01.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.920+0000 7fa7b5000640 1 Processor -- start 2024-03-06T07:28:01.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.921+0000 7fa7b5000640 1 -- start start 2024-03-06T07:28:01.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7b5000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa7b015df70 0x7fa7b0070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7b5000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7b00b6230 0x7fa7b00b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7b5000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b006f0a0 con 0x7fa7b0161cc0 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7b5000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b006fcb0 con 0x7fa7b00b6230 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7b5000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b006f380 con 0x7fa7b015df70 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7a5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7b00b6230 0x7fa7b00b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7a5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7b00b6230 0x7fa7b00b6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32828/0 (socket says 172.21.15.6:32828) 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.922+0000 7fa7a5400640 1 -- 172.21.15.6:0/1023581605 learned_addr learned my addr 172.21.15.6:0/1023581605 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.923+0000 7fa7ade00640 1 --2- 172.21.15.6:0/1023581605 >> v2:172.21.15.178:3300/0 conn(0x7fa7b015df70 0x7fa7b0070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.923+0000 7fa7a5400640 1 -- 172.21.15.6:0/1023581605 >> v2:172.21.15.178:3300/0 conn(0x7fa7b015df70 msgr2=0x7fa7b0070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.923+0000 7fa7a5400640 1 --2- 172.21.15.6:0/1023581605 >> v2:172.21.15.178:3300/0 conn(0x7fa7b015df70 0x7fa7b0070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.923+0000 7fa7a5400640 1 -- 172.21.15.6:0/1023581605 >> v1:172.21.15.6:6789/0 conn(0x7fa7b0161cc0 legacy=0x7fa7b006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.923+0000 7fa7a5400640 1 -- 172.21.15.6:0/1023581605 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b006f660 con 0x7fa7b00b6230 2024-03-06T07:28:01.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.924+0000 7fa7a5400640 1 --2- 172.21.15.6:0/1023581605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b00b6650 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00696b0 tx=0x7fa79802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d3948ce02f28e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.924+0000 7fa7ad400640 1 -- 172.21.15.6:0/1023581605 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798034070 con 0x7fa7b00b6230 2024-03-06T07:28:01.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.924+0000 7fa7ad400640 1 -- 172.21.15.6:0/1023581605 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa798030860 con 0x7fa7b00b6230 2024-03-06T07:28:01.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.924+0000 7fa7ad400640 1 -- 172.21.15.6:0/1023581605 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa798030c30 con 0x7fa7b00b6230 2024-03-06T07:28:01.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 -- 172.21.15.6:0/1023581605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 msgr2=0x7fa7b00b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 --2- 172.21.15.6:0/1023581605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b00b6650 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00696b0 tx=0x7fa79802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 -- 172.21.15.6:0/1023581605 shutdown_connections 2024-03-06T07:28:01.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 --2- 172.21.15.6:0/1023581605 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b00b6650 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 --2- 172.21.15.6:0/1023581605 >> v2:172.21.15.178:3300/0 conn(0x7fa7b015df70 0x7fa7b0070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 -- 172.21.15.6:0/1023581605 >> 172.21.15.6:0/1023581605 conn(0x7fa7b009aa10 msgr2=0x7fa7b009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:01.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 -- 172.21.15.6:0/1023581605 shutdown_connections 2024-03-06T07:28:01.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.925+0000 7fa7b5000640 1 -- 172.21.15.6:0/1023581605 wait complete. 2024-03-06T07:28:01.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.927+0000 7fa7b5000640 1 Processor -- start 2024-03-06T07:28:01.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.927+0000 7fa7b5000640 1 -- start start 2024-03-06T07:28:01.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7b5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b0154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7b5000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7b015df70 0x7fa7b015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7b5000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7b0161cc0 0x7fa7b0151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7b5000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0070800 con 0x7fa7b00b6230 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7b5000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b0070680 con 0x7fa7b0161cc0 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7b5000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b015f360 con 0x7fa7b015df70 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7ade00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b0154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7ade00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b0154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32844/0 (socket says 172.21.15.6:32844) 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7ade00640 1 -- 172.21.15.6:0/225173117 learned_addr learned my addr 172.21.15.6:0/225173117 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7a5400640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7b015df70 0x7fa7b015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7ae800640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7b0161cc0 0x7fa7b0151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7ade00640 1 -- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7b015df70 msgr2=0x7fa7b015d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.928+0000 7fa7ade00640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7b015df70 0x7fa7b015d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7ade00640 1 -- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7b0161cc0 msgr2=0x7fa7b0151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7ade00640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7b0161cc0 0x7fa7b0151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7ade00640 1 -- 172.21.15.6:0/225173117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0156b70 con 0x7fa7b00b6230 2024-03-06T07:28:01.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7ae800640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7b0161cc0 0x7fa7b0151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7ade00640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b0154f30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0007ef0 tx=0x7fa7a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0014620 con 0x7fa7b00b6230 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0019070 con 0x7fa7b00b6230 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b01f77a0 con 0x7fa7b00b6230 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0014a00 con 0x7fa7b00b6230 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.929+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b01f7d60 con 0x7fa7b00b6230 2024-03-06T07:28:01.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.930+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b01f75c0 con 0x7fa7b00b6230 2024-03-06T07:28:01.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.931+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7a0014ba0 con 0x7fa7b00b6230 2024-03-06T07:28:01.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.932+0000 7fa7a6a00640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa78c06b4d0 0x7fa78c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:01.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.933+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7a008abe0 con 0x7fa7b00b6230 2024-03-06T07:28:01.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.933+0000 7fa7a5400640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa78c06b4d0 0x7fa78c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:01.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.934+0000 7fa7a5400640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa78c06b4d0 0x7fa78c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa798036000 tx=0x7fa79803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:01.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:01.937+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7b01f75c0 con 0x7fa7b00b6230 2024-03-06T07:28:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.136+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7b004f3a0 con 0x7fa7b00b6230 2024-03-06T07:28:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.139+0000 7fa7a6a00640 1 -- 172.21.15.6:0/225173117 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7fa7a00599c0 con 0x7fa7b00b6230 2024-03-06T07:28:02.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:02.142 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.141+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa78c06b4d0 msgr2=0x7fa78c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.141+0000 7fa7b5000640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa78c06b4d0 0x7fa78c06d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa798036000 tx=0x7fa79803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 msgr2=0x7fa7b0154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b0154f30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0007ef0 tx=0x7fa7a000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 shutdown_connections 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa78c06b4d0 0x7fa78c06d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7b0161cc0 0x7fa7b0151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7b015df70 0x7fa7b015d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 --2- 172.21.15.6:0/225173117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7b00b6230 0x7fa7b0154f30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 >> 172.21.15.6:0/225173117 conn(0x7fa7b009aa10 msgr2=0x7fa7b0160c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 shutdown_connections 2024-03-06T07:28:02.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.142+0000 7fa7b5000640 1 -- 172.21.15.6:0/225173117 wait complete. 2024-03-06T07:28:02.163 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:02.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:02.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.390+0000 7f80c8a00640 1 Processor -- start 2024-03-06T07:28:02.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.390+0000 7f80c8a00640 1 -- start start 2024-03-06T07:28:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.391+0000 7f80c8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 0x7f80c4168dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.391+0000 7f80c8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c40b2c00 con 0x7f80c4161700 2024-03-06T07:28:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.391+0000 7f80c8a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c4161ae0 con 0x7f80c40d4890 2024-03-06T07:28:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.391+0000 7f80c8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40b56e0 con 0x7f80c40d24f0 2024-03-06T07:28:02.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.391+0000 7f80c3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f80c4161700 0x7f80c40b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44332/0 (socket says 172.21.15.6:44332) 2024-03-06T07:28:02.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.391+0000 7f80c3400640 1 -- 172.21.15.6:0/2611583797 learned_addr learned my addr 172.21.15.6:0/2611583797 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531685985 0 0) 0x7f80c40b2c00 con 0x7f80c4161700 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a8003520 con 0x7f80c4161700 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c3e00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 0x7f80c4168dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2215722343 0 0) 0x7f80a8003520 con 0x7f80c4161700 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 msgr2=0x7f80c4168dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 0x7f80c4168dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 >> v1:172.21.15.178:6789/0 conn(0x7f80c40d4890 legacy=0x7f80c40d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40b2ea0 con 0x7f80c4161700 2024-03-06T07:28:02.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f80b40019f0 con 0x7f80c4161700 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 >> v1:172.21.15.6:6789/0 conn(0x7f80c4161700 legacy=0x7f80c40b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a80039e0 0x7f80a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a8004800 0x7f80a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 0x7f80a8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40b56e0 con 0x7f80a8004800 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.392+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a800bb20 con 0x7f80a80039e0 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a800bd20 con 0x7f80c40d24f0 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2a00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a8004800 0x7f80a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c3400640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a80039e0 0x7f80a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c3e00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 0x7f80a8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2a00640 1 -- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 msgr2=0x7f80a8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2a00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c40d24f0 0x7f80a8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2a00640 1 -- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a80039e0 msgr2=0x7f80a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2a00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a80039e0 0x7f80a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c2a00640 1 -- 172.21.15.6:0/2611583797 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8003520 con 0x7f80a8004800 2024-03-06T07:28:02.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.393+0000 7f80c3400640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a80039e0 0x7f80a8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.394+0000 7f80c2a00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a8004800 0x7f80a8004c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f80ac001750 tx=0x7f80a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997a1419d595867d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac001370 con 0x7f80a8004800 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0052d0 con 0x7f80a8004800 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c2000640 1 -- 172.21.15.6:0/2611583797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac003830 con 0x7f80a8004800 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c8a00640 1 -- 172.21.15.6:0/2611583797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a8004800 msgr2=0x7f80a8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c8a00640 1 --2- 172.21.15.6:0/2611583797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a8004800 0x7f80a8004c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f80ac001750 tx=0x7f80a8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c3e00640 1 -- 172.21.15.6:0/2611583797 reap_dead start 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c8a00640 1 -- 172.21.15.6:0/2611583797 shutdown_connections 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c8a00640 1 -- 172.21.15.6:0/2611583797 >> 172.21.15.6:0/2611583797 conn(0x7f80c40d3d70 msgr2=0x7f80c40d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c8a00640 1 -- 172.21.15.6:0/2611583797 shutdown_connections 2024-03-06T07:28:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.395+0000 7f80c8a00640 1 -- 172.21.15.6:0/2611583797 wait complete. 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.397+0000 7f80c8a00640 1 Processor -- start 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.397+0000 7f80c8a00640 1 -- start start 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.398+0000 7f80c8a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a8004800 0x7f80c409f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.398+0000 7f80c8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c415c540 0x7f80c40cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.398+0000 7f80c8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 0x7f80c41fb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.398+0000 7f80c8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40b0d00 con 0x7f80c40cc310 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.398+0000 7f80c8a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40b0a20 con 0x7f80a8004800 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.398+0000 7f80c8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c40b2230 con 0x7f80c415c540 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 0x7f80c41fb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 0x7f80c41fb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32848/0 (socket says 172.21.15.6:32848) 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 -- 172.21.15.6:0/1751187077 learned_addr learned my addr 172.21.15.6:0/1751187077 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3400640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a8004800 0x7f80c409f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 -- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c415c540 msgr2=0x7f80c40cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c415c540 0x7f80c40cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 -- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a8004800 msgr2=0x7f80c409f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a8004800 0x7f80c409f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 -- 172.21.15.6:0/1751187077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40cc7e0 con 0x7f80c40cc310 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3400640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a8004800 0x7f80c409f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.399+0000 7f80c3e00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 0x7f80c41fb6e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80c408c480 tx=0x7f80bc00f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.400+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc004440 con 0x7f80c40cc310 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.400+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc019070 con 0x7f80c40cc310 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.400+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc003850 con 0x7f80c40cc310 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.400+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c40ccac0 con 0x7f80c40cc310 2024-03-06T07:28:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.400+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c40c8bd0 con 0x7f80c40cc310 2024-03-06T07:28:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.401+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c404f330 con 0x7f80c40cc310 2024-03-06T07:28:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.402+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80bc012070 con 0x7f80c40cc310 2024-03-06T07:28:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.403+0000 7f80bb400640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80a006b4d0 0x7f80a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.403+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80bc089b70 con 0x7f80c40cc310 2024-03-06T07:28:02.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.403+0000 7f80c3400640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80a006b4d0 0x7f80a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.404+0000 7f80c3400640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80a006b4d0 0x7f80a006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80b4002230 tx=0x7f80b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.407+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80bc058950 con 0x7f80c40cc310 2024-03-06T07:28:02.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.615+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80c40d0d10 con 0x7f80c40cc310 2024-03-06T07:28:02.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.615+0000 7f80bb400640 1 -- 172.21.15.6:0/1751187077 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80bc058250 con 0x7f80c40cc310 2024-03-06T07:28:02.617 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80a006b4d0 msgr2=0x7f80a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80a006b4d0 0x7f80a006d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80b4002230 tx=0x7f80b4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 msgr2=0x7f80c41fb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 0x7f80c41fb6e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80c408c480 tx=0x7f80bc00f6f0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 shutdown_connections 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80a006b4d0 0x7f80a006d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80c40cc310 0x7f80c41fb6e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80c415c540 0x7f80c40cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 --2- 172.21.15.6:0/1751187077 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a8004800 0x7f80c409f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.618+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 >> 172.21.15.6:0/1751187077 conn(0x7f80c40d3d70 msgr2=0x7f80c415d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.619+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 shutdown_connections 2024-03-06T07:28:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.619+0000 7f80c8a00640 1 -- 172.21.15.6:0/1751187077 wait complete. 2024-03-06T07:28:02.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.863+0000 7fe081a00640 1 Processor -- start 2024-03-06T07:28:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.863+0000 7fe081a00640 1 -- start start 2024-03-06T07:28:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe081a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c15dee0 0x7fe07c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe081a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe07c0d0d40 0x7fe07c0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe081a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c0ceca0 con 0x7fe07c0d4860 2024-03-06T07:28:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe081a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07c0cf5d0 con 0x7fe07c0d0d40 2024-03-06T07:28:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe081a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c0cef80 con 0x7fe07c15dee0 2024-03-06T07:28:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe073e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe07c0d4860 0x7fe07c0d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44344/0 (socket says 172.21.15.6:44344) 2024-03-06T07:28:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe073e00640 1 -- 172.21.15.6:0/3689777407 learned_addr learned my addr 172.21.15.6:0/3689777407 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe081000640 1 --2- 172.21.15.6:0/3689777407 >> v2:172.21.15.6:3300/0 conn(0x7fe07c0d0d40 0x7fe07c0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.865+0000 7fe07be00640 1 --2- 172.21.15.6:0/3689777407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c15dee0 0x7fe07c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.866+0000 7fe07b400640 1 -- 172.21.15.6:0/3689777407 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043583036 0 0) 0x7fe07c0ceca0 con 0x7fe07c0d4860 2024-03-06T07:28:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.866+0000 7fe07b400640 1 -- 172.21.15.6:0/3689777407 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe060003520 con 0x7fe07c0d4860 2024-03-06T07:28:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.866+0000 7fe081000640 1 -- 172.21.15.6:0/3689777407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c15dee0 msgr2=0x7fe07c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.866+0000 7fe081000640 1 --2- 172.21.15.6:0/3689777407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c15dee0 0x7fe07c0d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.866+0000 7fe081000640 1 -- 172.21.15.6:0/3689777407 >> v1:172.21.15.6:6789/0 conn(0x7fe07c0d4860 legacy=0x7fe07c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.866+0000 7fe081000640 1 -- 172.21.15.6:0/3689777407 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c0cf260 con 0x7fe07c0d0d40 2024-03-06T07:28:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.867+0000 7fe081000640 1 --2- 172.21.15.6:0/3689777407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d0d40 0x7fe07c0b3510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe074009d90 tx=0x7fe07402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9f71f7a76b46ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe07b400640 1 -- 172.21.15.6:0/3689777407 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074039070 con 0x7fe07c0d0d40 2024-03-06T07:28:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe07b400640 1 -- 172.21.15.6:0/3689777407 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07402cc50 con 0x7fe07c0d0d40 2024-03-06T07:28:02.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe07b400640 1 -- 172.21.15.6:0/3689777407 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074034440 con 0x7fe07c0d0d40 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe081a00640 1 -- 172.21.15.6:0/3689777407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d0d40 msgr2=0x7fe07c0b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe081a00640 1 --2- 172.21.15.6:0/3689777407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d0d40 0x7fe07c0b3510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe074009d90 tx=0x7fe07402bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe081a00640 1 -- 172.21.15.6:0/3689777407 shutdown_connections 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe081a00640 1 --2- 172.21.15.6:0/3689777407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d0d40 0x7fe07c0b3510 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.868+0000 7fe081a00640 1 --2- 172.21.15.6:0/3689777407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c15dee0 0x7fe07c0d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.869+0000 7fe081a00640 1 -- 172.21.15.6:0/3689777407 >> 172.21.15.6:0/3689777407 conn(0x7fe07c0d3cd0 msgr2=0x7fe07c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.869+0000 7fe081a00640 1 -- 172.21.15.6:0/3689777407 shutdown_connections 2024-03-06T07:28:02.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.869+0000 7fe081a00640 1 -- 172.21.15.6:0/3689777407 wait complete. 2024-03-06T07:28:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.870+0000 7fe081a00640 1 Processor -- start 2024-03-06T07:28:02.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.870+0000 7fe081a00640 1 -- start start 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.871+0000 7fe081a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c0d0d40 0x7fe07c0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.871+0000 7fe081a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 0x7fe07c0cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.871+0000 7fe081a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe07c15dee0 0x7fe07c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.871+0000 7fe081a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c0d1de0 con 0x7fe07c0d4860 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.871+0000 7fe081a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c15e2c0 con 0x7fe07c15dee0 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.871+0000 7fe081a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c0d1f60 con 0x7fe07c0d0d40 2024-03-06T07:28:02.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 0x7fe07c0cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 0x7fe07c0cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32876/0 (socket says 172.21.15.6:32876) 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 -- 172.21.15.6:0/3569457533 learned_addr learned my addr 172.21.15.6:0/3569457533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe07be00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c0d0d40 0x7fe07c0c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe081000640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe07c15dee0 0x7fe07c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 -- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c0d0d40 msgr2=0x7fe07c0c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c0d0d40 0x7fe07c0c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 -- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe07c15dee0 msgr2=0x7fe07c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe07c15dee0 0x7fe07c0cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.872+0000 7fe073e00640 1 -- 172.21.15.6:0/3569457533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c15e8b0 con 0x7fe07c0d4860 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.873+0000 7fe073e00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 0x7fe07c0cc930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c430 tx=0x7fe06400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.873+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064017070 con 0x7fe07c0d4860 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.873+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe064005d00 con 0x7fe07c0d4860 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.873+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c15eb90 con 0x7fe07c0d4860 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.873+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064007310 con 0x7fe07c0d4860 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.873+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c15f150 con 0x7fe07c0d4860 2024-03-06T07:28:02.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.874+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07c04f330 con 0x7fe07c0d4860 2024-03-06T07:28:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.875+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe06401a070 con 0x7fe07c0d4860 2024-03-06T07:28:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.876+0000 7fe078c00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe05806b4d0 0x7fe05806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.876+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe064089610 con 0x7fe07c0d4860 2024-03-06T07:28:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.877+0000 7fe07be00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe05806b4d0 0x7fe05806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.878+0000 7fe07be00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe05806b4d0 0x7fe05806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009a30 tx=0x7fe06c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:02.880+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe064058500 con 0x7fe07c0d4860 2024-03-06T07:28:03.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.083+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe07c08d7d0 con 0x7fe07c0d4860 2024-03-06T07:28:03.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.084+0000 7fe078c00640 1 -- 172.21.15.6:0/3569457533 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe064057e00 con 0x7fe07c0d4860 2024-03-06T07:28:03.086 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:03.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe05806b4d0 msgr2=0x7fe05806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe05806b4d0 0x7fe05806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009a30 tx=0x7fe06c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:03.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 msgr2=0x7fe07c0cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 0x7fe07c0cc930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe06400c430 tx=0x7fe06400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 shutdown_connections 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe05806b4d0 0x7fe05806d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe07c15dee0 0x7fe07c0cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe07c0d4860 0x7fe07c0cc930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 --2- 172.21.15.6:0/3569457533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe07c0d0d40 0x7fe07c0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 >> 172.21.15.6:0/3569457533 conn(0x7fe07c0d3cd0 msgr2=0x7fe07c06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 shutdown_connections 2024-03-06T07:28:03.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.087+0000 7fe081a00640 1 -- 172.21.15.6:0/3569457533 wait complete. 2024-03-06T07:28:03.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:03.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.339+0000 7f126b200640 1 Processor -- start 2024-03-06T07:28:03.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.339+0000 7f126b200640 1 -- start start 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f126b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f126c0af4e0 0x7f126c0b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f126b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f126c0b4b20 0x7f126c0af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f126b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c0b1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f126b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126c0b0950 con 0x7f126c0af4e0 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f126b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126c0b0ba0 con 0x7f126c0b4b20 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f126b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0b0e80 con 0x7f126c0afe00 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f1269e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f126c0af4e0 0x7f126c0b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f1269e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f126c0af4e0 0x7f126c0b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32890/0 (socket says 172.21.15.6:32890) 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f1269e00640 1 -- 172.21.15.6:0/3521940024 learned_addr learned my addr 172.21.15.6:0/3521940024 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.341+0000 7f1269400640 1 --2- 172.21.15.6:0/3521940024 >> v2:172.21.15.178:3300/0 conn(0x7f126c0b4b20 0x7f126c0af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f126a800640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c0b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f1269e00640 1 -- 172.21.15.6:0/3521940024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 msgr2=0x7f126c0b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f1269e00640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c0b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f1269e00640 1 -- 172.21.15.6:0/3521940024 >> v2:172.21.15.178:3300/0 conn(0x7f126c0b4b20 msgr2=0x7f126c0af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f1269e00640 1 --2- 172.21.15.6:0/3521940024 >> v2:172.21.15.178:3300/0 conn(0x7f126c0b4b20 0x7f126c0af8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f1269e00640 1 -- 172.21.15.6:0/3521940024 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c0b1160 con 0x7f126c0af4e0 2024-03-06T07:28:03.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.342+0000 7f126a800640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c0b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:03.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.343+0000 7f1269e00640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c0b5990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f125c009870 tx=0x7f125c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8242f2c569974d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.343+0000 7f1268a00640 1 -- 172.21.15.6:0/3521940024 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c0317c0 con 0x7f126c0af4e0 2024-03-06T07:28:03.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.343+0000 7f1268a00640 1 -- 172.21.15.6:0/3521940024 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c037070 con 0x7f126c0af4e0 2024-03-06T07:28:03.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.343+0000 7f1268a00640 1 -- 172.21.15.6:0/3521940024 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c031b80 con 0x7f126c0af4e0 2024-03-06T07:28:03.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 -- 172.21.15.6:0/3521940024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 msgr2=0x7f126c0b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c0b5990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f125c009870 tx=0x7f125c02b860 comp rx=0 tx=0).stop 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 -- 172.21.15.6:0/3521940024 shutdown_connections 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c0b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 --2- 172.21.15.6:0/3521940024 >> v2:172.21.15.178:3300/0 conn(0x7f126c0b4b20 0x7f126c0af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 --2- 172.21.15.6:0/3521940024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c0b5990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 -- 172.21.15.6:0/3521940024 >> 172.21.15.6:0/3521940024 conn(0x7f126c09aa10 msgr2=0x7f126c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 -- 172.21.15.6:0/3521940024 shutdown_connections 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.344+0000 7f126b200640 1 -- 172.21.15.6:0/3521940024 wait complete. 2024-03-06T07:28:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.346+0000 7f126b200640 1 Processor -- start 2024-03-06T07:28:03.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.346+0000 7f126b200640 1 -- start start 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f126b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f126b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f126b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f126c0b4b20 0x7f126c158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f126b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0b2020 con 0x7f126c0af4e0 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f126b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0b1ea0 con 0x7f126c0b4b20 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f126b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0b21a0 con 0x7f126c0afe00 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f1269e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f1269e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32902/0 (socket says 172.21.15.6:32902) 2024-03-06T07:28:03.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f1269e00640 1 -- 172.21.15.6:0/1162348031 learned_addr learned my addr 172.21.15.6:0/1162348031 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:03.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.347+0000 7f1269400640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1269e00640 1 -- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 msgr2=0x7f126c145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f126a800640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f126c0b4b20 0x7f126c158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1269e00640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1269e00640 1 -- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f126c0b4b20 msgr2=0x7f126c158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1269e00640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f126c0b4b20 0x7f126c158600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1269e00640 1 -- 172.21.15.6:0/1162348031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c158b40 con 0x7f126c0af4e0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1269e00640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c156600 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f125c0097c0 tx=0x7f125c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c032650 con 0x7f126c0af4e0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125c032810 con 0x7f126c0af4e0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125c032c00 con 0x7f126c0af4e0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c156be0 con 0x7f126c0af4e0 2024-03-06T07:28:03.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.348+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c1571a0 con 0x7f126c0af4e0 2024-03-06T07:28:03.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.349+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c04f330 con 0x7f126c0af4e0 2024-03-06T07:28:03.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.350+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f125c04c060 con 0x7f126c0af4e0 2024-03-06T07:28:03.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.351+0000 7f1262000640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124806b4d0 0x7f124806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.351+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f125c0ab890 con 0x7f126c0af4e0 2024-03-06T07:28:03.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.352+0000 7f1269400640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124806b4d0 0x7f124806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.353+0000 7f1269400640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124806b4d0 0x7f124806d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1254006fd0 tx=0x7f1254008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.356+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f125c040070 con 0x7f126c0af4e0 2024-03-06T07:28:03.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.555+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f126c0b3230 con 0x7f126c0af4e0 2024-03-06T07:28:03.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.556+0000 7f1262000640 1 -- 172.21.15.6:0/1162348031 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f125c07a830 con 0x7f126c0af4e0 2024-03-06T07:28:03.558 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124806b4d0 msgr2=0x7f124806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124806b4d0 0x7f124806d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1254006fd0 tx=0x7f1254008040 comp rx=0 tx=0).stop 2024-03-06T07:28:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 msgr2=0x7f126c156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c156600 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f125c0097c0 tx=0x7f125c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 shutdown_connections 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124806b4d0 0x7f124806d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f126c0b4b20 0x7f126c158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f126c0afe00 0x7f126c145600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 --2- 172.21.15.6:0/1162348031 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126c0af4e0 0x7f126c156600 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 >> 172.21.15.6:0/1162348031 conn(0x7f126c09aa10 msgr2=0x7f126c15c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 shutdown_connections 2024-03-06T07:28:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.559+0000 7f126b200640 1 -- 172.21.15.6:0/1162348031 wait complete. 2024-03-06T07:28:03.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:03.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.802+0000 7ff130a00640 1 Processor -- start 2024-03-06T07:28:03.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.803+0000 7ff130a00640 1 -- start start 2024-03-06T07:28:03.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff130a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff12c0af1a0 0x7ff12c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff130a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 0x7ff12c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff130a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c15cc40 con 0x7ff12c0af580 2024-03-06T07:28:03.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff130a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff12c15cf20 con 0x7ff12c0af1a0 2024-03-06T07:28:03.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff130a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c15d200 con 0x7ff12c0cdc20 2024-03-06T07:28:03.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff129400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff12c0af580 0x7ff12c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44360/0 (socket says 172.21.15.6:44360) 2024-03-06T07:28:03.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.804+0000 7ff129400640 1 -- 172.21.15.6:0/340354604 learned_addr learned my addr 172.21.15.6:0/340354604 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff129e00640 1 --2- 172.21.15.6:0/340354604 >> v2:172.21.15.178:3300/0 conn(0x7ff12c0af1a0 0x7ff12c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff12a800640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 0x7ff12c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269892864 0 0) 0x7ff12c15cc40 con 0x7ff12c0af580 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff10c003520 con 0x7ff12c0af580 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1558886099 0 0) 0x7ff10c003520 con 0x7ff12c0af580 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 msgr2=0x7ff12c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 0x7ff12c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 >> v2:172.21.15.178:3300/0 conn(0x7ff12c0af1a0 msgr2=0x7ff12c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 --2- 172.21.15.6:0/340354604 >> v2:172.21.15.178:3300/0 conn(0x7ff12c0af1a0 0x7ff12c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c15d4e0 con 0x7ff12c0af580 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff1140019f0 con 0x7ff12c0af580 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 >> v1:172.21.15.6:6789/0 conn(0x7ff12c0af580 legacy=0x7ff12c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 0x7ff10c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10c003fa0 0x7ff10c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c004d40 0x7ff10c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c15cf20 con 0x7ff10c003fa0 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c15d200 con 0x7ff10c004d40 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.805+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c00ba70 con 0x7ff12c0cdc20 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.806+0000 7ff129400640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10c003fa0 0x7ff10c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.806+0000 7ff129400640 1 -- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 msgr2=0x7ff10c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.806+0000 7ff129400640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0cdc20 0x7ff10c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.806+0000 7ff129400640 1 -- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c004d40 msgr2=0x7ff10c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.806+0000 7ff129400640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c004d40 0x7ff10c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.806+0000 7ff129400640 1 -- 172.21.15.6:0/340354604 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c003520 con 0x7ff10c003fa0 2024-03-06T07:28:03.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.807+0000 7ff129400640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10c003fa0 0x7ff10c004800 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff11400a890 tx=0x7ff10c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c83e916fe68e705 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.807+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11400a4e0 con 0x7ff10c003fa0 2024-03-06T07:28:03.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.807+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114034650 con 0x7ff10c003fa0 2024-03-06T07:28:03.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.807+0000 7ff128a00640 1 -- 172.21.15.6:0/340354604 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114034a80 con 0x7ff10c003fa0 2024-03-06T07:28:03.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff130a00640 1 -- 172.21.15.6:0/340354604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10c003fa0 msgr2=0x7ff10c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff130a00640 1 --2- 172.21.15.6:0/340354604 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff10c003fa0 0x7ff10c004800 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff11400a890 tx=0x7ff10c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff12a800640 1 -- 172.21.15.6:0/340354604 reap_dead start 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff130a00640 1 -- 172.21.15.6:0/340354604 shutdown_connections 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff130a00640 1 -- 172.21.15.6:0/340354604 >> 172.21.15.6:0/340354604 conn(0x7ff12c06afb0 msgr2=0x7ff12c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff130a00640 1 -- 172.21.15.6:0/340354604 shutdown_connections 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.808+0000 7ff130a00640 1 -- 172.21.15.6:0/340354604 wait complete. 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.810+0000 7ff130a00640 1 Processor -- start 2024-03-06T07:28:03.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.810+0000 7ff130a00640 1 -- start start 2024-03-06T07:28:03.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff130a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c003fa0 0x7ff12c15ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff130a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 0x7ff12c0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff130a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0c5a60 0x7ff12c15bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff130a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c0b3160 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff130a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c0b2fe0 con 0x7ff10c003fa0 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff130a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c0b32e0 con 0x7ff12c0c5a60 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff129400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 0x7ff12c0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff129400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 0x7ff12c0c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32920/0 (socket says 172.21.15.6:32920) 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.811+0000 7ff129400640 1 -- 172.21.15.6:0/3660970643 learned_addr learned my addr 172.21.15.6:0/3660970643 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129e00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c003fa0 0x7ff12c15ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff12a800640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0c5a60 0x7ff12c15bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129400640 1 -- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0c5a60 msgr2=0x7ff12c15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129400640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0c5a60 0x7ff12c15bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129400640 1 -- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c003fa0 msgr2=0x7ff12c15ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129400640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c003fa0 0x7ff12c15ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129400640 1 -- 172.21.15.6:0/3660970643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0c5ee0 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff12a800640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0c5a60 0x7ff12c15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff129400640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 0x7ff12c0c5520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff11400a9b0 tx=0x7ff114001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114039040 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114043070 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12c0c61c0 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.812+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114034800 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.813+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12c15b070 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.814+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12c0c65b0 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.815+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1140349c0 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.816+0000 7ff122000640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff10806b520 0x7ff10806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.816+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1140ac3b0 con 0x7ff12c0c4ff0 2024-03-06T07:28:03.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.816+0000 7ff129e00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff10806b520 0x7ff10806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.818+0000 7ff129e00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff10806b520 0x7ff10806d9e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009ad0 tx=0x7ff11c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:03.820+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff11407b210 con 0x7ff12c0c4ff0 2024-03-06T07:28:04.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.021+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff12c069510 con 0x7ff12c0c4ff0 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.024+0000 7ff122000640 1 -- 172.21.15.6:0/3660970643 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7ff11407ab10 con 0x7ff12c0c4ff0 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:04.026 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:04.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff10806b520 msgr2=0x7ff10806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff10806b520 0x7ff10806d9e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009ad0 tx=0x7ff11c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 msgr2=0x7ff12c0c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 0x7ff12c0c5520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff11400a9b0 tx=0x7ff114001dc0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 shutdown_connections 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff10806b520 0x7ff10806d9e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff12c0c5a60 0x7ff12c15bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff12c0c4ff0 0x7ff12c0c5520 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 --2- 172.21.15.6:0/3660970643 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff10c003fa0 0x7ff12c15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 >> 172.21.15.6:0/3660970643 conn(0x7ff12c06afb0 msgr2=0x7ff12c15b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 shutdown_connections 2024-03-06T07:28:04.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.027+0000 7ff130a00640 1 -- 172.21.15.6:0/3660970643 wait complete. 2024-03-06T07:28:04.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:04.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.272+0000 7f8f12a00640 1 Processor -- start 2024-03-06T07:28:04.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.272+0000 7f8f12a00640 1 -- start start 2024-03-06T07:28:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.273+0000 7f8f12a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8f0c15dee0 0x7f8f0c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.273+0000 7f8f12a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f0c0d4860 0x7f8f0c0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.273+0000 7f8f12a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0c0ceca0 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.273+0000 7f8f12a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c0cf5d0 con 0x7f8f0c0d4860 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.273+0000 7f8f12a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0c0cef80 con 0x7f8f0c15dee0 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f0c0d4860 0x7f8f0c0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f0c0d4860 0x7f8f0c0d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32928/0 (socket says 172.21.15.6:32928) 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 -- 172.21.15.6:0/2825561871 learned_addr learned my addr 172.21.15.6:0/2825561871 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f11600640 1 --2- 172.21.15.6:0/2825561871 >> v2:172.21.15.178:3300/0 conn(0x7f8f0c15dee0 0x7f8f0c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 -- 172.21.15.6:0/2825561871 >> v2:172.21.15.178:3300/0 conn(0x7f8f0c15dee0 msgr2=0x7f8f0c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 --2- 172.21.15.6:0/2825561871 >> v2:172.21.15.178:3300/0 conn(0x7f8f0c15dee0 0x7f8f0c0d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 -- 172.21.15.6:0/2825561871 >> v1:172.21.15.178:6789/0 conn(0x7f8f0c0cfa60 legacy=0x7f8f0c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.274+0000 7f8f10c00640 1 -- 172.21.15.6:0/2825561871 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c0cf260 con 0x7f8f0c0d4860 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.275+0000 7f8f10c00640 1 --2- 172.21.15.6:0/2825561871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0d4860 0x7f8f0c0d0630 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c08c6f0 tx=0x7f8f0002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f73c3f9d4bff62e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f0be00640 1 -- 172.21.15.6:0/2825561871 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0002cb20 con 0x7f8f0c0d4860 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f0be00640 1 -- 172.21.15.6:0/2825561871 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0002ccc0 con 0x7f8f0c0d4860 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f0be00640 1 -- 172.21.15.6:0/2825561871 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00034610 con 0x7f8f0c0d4860 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f12a00640 1 -- 172.21.15.6:0/2825561871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0d4860 msgr2=0x7f8f0c0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f12a00640 1 --2- 172.21.15.6:0/2825561871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0d4860 0x7f8f0c0d0630 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c08c6f0 tx=0x7f8f0002bf80 comp rx=0 tx=0).stop 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f12a00640 1 -- 172.21.15.6:0/2825561871 shutdown_connections 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f12a00640 1 --2- 172.21.15.6:0/2825561871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0d4860 0x7f8f0c0d0630 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f12a00640 1 --2- 172.21.15.6:0/2825561871 >> v2:172.21.15.178:3300/0 conn(0x7f8f0c15dee0 0x7f8f0c0d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.276+0000 7f8f12a00640 1 -- 172.21.15.6:0/2825561871 >> 172.21.15.6:0/2825561871 conn(0x7f8f0c0d3cd0 msgr2=0x7f8f0c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.277+0000 7f8f12a00640 1 -- 172.21.15.6:0/2825561871 shutdown_connections 2024-03-06T07:28:04.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.277+0000 7f8f12a00640 1 -- 172.21.15.6:0/2825561871 wait complete. 2024-03-06T07:28:04.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.278+0000 7f8f12a00640 1 Processor -- start 2024-03-06T07:28:04.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.278+0000 7f8f12a00640 1 -- start start 2024-03-06T07:28:04.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f12a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 0x7f8f0c15bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f12a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f0c0d4860 0x7f8f0c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f12a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0c15dee0 0x7f8f0c0c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f12a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0d1de0 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f12a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c15e2c0 con 0x7f8f0c15dee0 2024-03-06T07:28:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f12a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c0d1f60 con 0x7f8f0c0d4860 2024-03-06T07:28:04.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.279+0000 7f8f11600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 0x7f8f0c15bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 0x7f8f0c15bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32936/0 (socket says 172.21.15.6:32936) 2024-03-06T07:28:04.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 -- 172.21.15.6:0/731178248 learned_addr learned my addr 172.21.15.6:0/731178248 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:04.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f10c00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f0c0d4860 0x7f8f0c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f12000640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0c15dee0 0x7f8f0c0c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 -- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f0c0d4860 msgr2=0x7f8f0c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f0c0d4860 0x7f8f0c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 -- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0c15dee0 msgr2=0x7f8f0c0c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0c15dee0 0x7f8f0c0c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 -- 172.21.15.6:0/731178248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c0c8200 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f10c00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f0c0d4860 0x7f8f0c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.280+0000 7f8f11600640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 0x7f8f0c15bd40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00e870 tx=0x7f8efc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.281+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc017070 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.281+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc00c940 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.281+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0c0c8eb0 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.281+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc00cd30 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.281+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0c15c280 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.282+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0c04f330 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.283+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8efc01a070 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.285+0000 7f8f09600640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b4d0 0x7f8ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.285+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8efc08a490 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.286+0000 7f8f10c00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b4d0 0x7f8ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.287+0000 7f8f10c00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b4d0 0x7f8ef006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8f00035000 tx=0x7f8f00030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.290+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8efc059300 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.494+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f0c06d490 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.495+0000 7f8f09600640 1 -- 172.21.15.6:0/731178248 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8efc058c00 con 0x7f8f0c0cfa60 2024-03-06T07:28:04.497 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.497+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b4d0 msgr2=0x7f8ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.497+0000 7f8f12a00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b4d0 0x7f8ef006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8f00035000 tx=0x7f8f00030000 comp rx=0 tx=0).stop 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.497+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 msgr2=0x7f8f0c15bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.497+0000 7f8f12a00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 0x7f8f0c15bd40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8efc00e870 tx=0x7f8efc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 shutdown_connections 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ef006b4d0 0x7f8ef006d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f0c15dee0 0x7f8f0c0c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f0c0d4860 0x7f8f0c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 --2- 172.21.15.6:0/731178248 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f0c0cfa60 0x7f8f0c15bd40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 >> 172.21.15.6:0/731178248 conn(0x7f8f0c0d3cd0 msgr2=0x7f8f0c0b8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 shutdown_connections 2024-03-06T07:28:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.498+0000 7f8f12a00640 1 -- 172.21.15.6:0/731178248 wait complete. 2024-03-06T07:28:04.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.744+0000 7f9de3200640 1 Processor -- start 2024-03-06T07:28:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.744+0000 7f9de3200640 1 -- start start 2024-03-06T07:28:04.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.745+0000 7f9de3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9de415e0c0 0x7f9de4072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.745+0000 7f9de3200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9de406b440 0x7f9de40cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.745+0000 7f9de3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9de40b5800 con 0x7f9de406afb0 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.745+0000 7f9de3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de406dd30 con 0x7f9de415e0c0 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.745+0000 7f9de3200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9de40b5aa0 con 0x7f9de406b440 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9de415e0c0 0x7f9de4072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9de415e0c0 0x7f9de4072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32952/0 (socket says 172.21.15.6:32952) 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 -- 172.21.15.6:0/3342827610 learned_addr learned my addr 172.21.15.6:0/3342827610 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:04.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1400640 1 --1- 172.21.15.6:0/3342827610 >> v1:172.21.15.6:6789/0 conn(0x7f9de406afb0 0x7f9de40cd600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44370/0 (socket says 172.21.15.6:44370) 2024-03-06T07:28:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 -- 172.21.15.6:0/3342827610 >> v2:172.21.15.178:3300/0 conn(0x7f9de406b440 msgr2=0x7f9de40cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 --2- 172.21.15.6:0/3342827610 >> v2:172.21.15.178:3300/0 conn(0x7f9de406b440 0x7f9de40cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 -- 172.21.15.6:0/3342827610 >> v1:172.21.15.6:6789/0 conn(0x7f9de406afb0 legacy=0x7f9de40cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de1e00640 1 -- 172.21.15.6:0/3342827610 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de40b5d80 con 0x7f9de415e0c0 2024-03-06T07:28:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.746+0000 7f9de0a00640 1 -- 172.21.15.6:0/3342827610 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739632545 0 0) 0x7f9de40b5800 con 0x7f9de406afb0 2024-03-06T07:28:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.747+0000 7f9de1e00640 1 --2- 172.21.15.6:0/3342827610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de415e0c0 0x7f9de4072270 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9de40ce6f0 tx=0x7f9dd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2a86e15b51b876 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de0a00640 1 -- 172.21.15.6:0/3342827610 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4034070 con 0x7f9de415e0c0 2024-03-06T07:28:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de0a00640 1 -- 172.21.15.6:0/3342827610 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd4030860 con 0x7f9de415e0c0 2024-03-06T07:28:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de0a00640 1 -- 172.21.15.6:0/3342827610 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4030c30 con 0x7f9de415e0c0 2024-03-06T07:28:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 -- 172.21.15.6:0/3342827610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de415e0c0 msgr2=0x7f9de4072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 --2- 172.21.15.6:0/3342827610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de415e0c0 0x7f9de4072270 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9de40ce6f0 tx=0x7f9dd402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 -- 172.21.15.6:0/3342827610 shutdown_connections 2024-03-06T07:28:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 --2- 172.21.15.6:0/3342827610 >> v2:172.21.15.178:3300/0 conn(0x7f9de406b440 0x7f9de40cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 --2- 172.21.15.6:0/3342827610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de415e0c0 0x7f9de4072270 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 -- 172.21.15.6:0/3342827610 >> 172.21.15.6:0/3342827610 conn(0x7f9de40d0c00 msgr2=0x7f9de40d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 -- 172.21.15.6:0/3342827610 shutdown_connections 2024-03-06T07:28:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.748+0000 7f9de3200640 1 -- 172.21.15.6:0/3342827610 wait complete. 2024-03-06T07:28:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.750+0000 7f9de3200640 1 Processor -- start 2024-03-06T07:28:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.750+0000 7f9de3200640 1 -- start start 2024-03-06T07:28:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9de406afb0 0x7f9de41547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 0x7f9de406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9de415e0c0 0x7f9de406f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de406b850 con 0x7f9de406b440 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40cef60 con 0x7f9de415e0c0 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9de40727b0 con 0x7f9de406afb0 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.751+0000 7f9de1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 0x7f9de406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 0x7f9de406ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32960/0 (socket says 172.21.15.6:32960) 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 -- 172.21.15.6:0/540436378 learned_addr learned my addr 172.21.15.6:0/540436378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1e00640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9de406afb0 0x7f9de41547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de2800640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9de415e0c0 0x7f9de406f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 -- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9de406afb0 msgr2=0x7f9de41547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9de406afb0 0x7f9de41547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 -- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9de415e0c0 msgr2=0x7f9de406f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9de415e0c0 0x7f9de406f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 -- 172.21.15.6:0/540436378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de40af3b0 con 0x7f9de406b440 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.752+0000 7f9de1400640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 0x7f9de406ef10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00c4b0 tx=0x7f9dcc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.753+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc018070 con 0x7f9de406b440 2024-03-06T07:28:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.753+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc007480 con 0x7f9de406b440 2024-03-06T07:28:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.753+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc004820 con 0x7f9de406b440 2024-03-06T07:28:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.753+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de41f7b20 con 0x7f9de406b440 2024-03-06T07:28:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.753+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de415e820 con 0x7f9de406b440 2024-03-06T07:28:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.754+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de404f330 con 0x7f9de406b440 2024-03-06T07:28:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.755+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dcc0049c0 con 0x7f9de406b440 2024-03-06T07:28:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.756+0000 7f9dda000640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9dc006b500 0x7f9dc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.756+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dcc089340 con 0x7f9de406b440 2024-03-06T07:28:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.757+0000 7f9de1e00640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9dc006b500 0x7f9dc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.758+0000 7f9de1e00640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9dc006b500 0x7f9dc006d9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4036000 tx=0x7f9dd4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.760+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dcc058310 con 0x7f9de406b440 2024-03-06T07:28:04.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.960+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9de406bcb0 con 0x7f9de406b440 2024-03-06T07:28:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.961+0000 7f9dda000640 1 -- 172.21.15.6:0/540436378 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9dcc057c10 con 0x7f9de406b440 2024-03-06T07:28:04.963 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:04.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9dc006b500 msgr2=0x7f9dc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9dc006b500 0x7f9dc006d9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4036000 tx=0x7f9dd4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 msgr2=0x7f9de406ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 0x7f9de406ef10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00c4b0 tx=0x7f9dcc00c970 comp rx=0 tx=0).stop 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 shutdown_connections 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9dc006b500 0x7f9dc006d9c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9de415e0c0 0x7f9de406f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9de406b440 0x7f9de406ef10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 --2- 172.21.15.6:0/540436378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9de406afb0 0x7f9de41547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 >> 172.21.15.6:0/540436378 conn(0x7f9de40d0c00 msgr2=0x7f9de40cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 shutdown_connections 2024-03-06T07:28:04.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:04.964+0000 7f9de3200640 1 -- 172.21.15.6:0/540436378 wait complete. 2024-03-06T07:28:04.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.202+0000 7fb01ce00640 1 Processor -- start 2024-03-06T07:28:05.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.203+0000 7fb01ce00640 1 -- start start 2024-03-06T07:28:05.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb01ce00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb01815dee0 0x7fb0180d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb01ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0180d4860 0x7fb0180d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb01ce00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0180d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb01ce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180cf5d0 con 0x7fb0180d4860 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb01ce00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180ceca0 con 0x7fb01815dee0 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb01ce00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180cef80 con 0x7fb0180cfa60 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb016a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0180d4860 0x7fb0180d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb016a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0180d4860 0x7fb0180d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32968/0 (socket says 172.21.15.6:32968) 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb016a00640 1 -- 172.21.15.6:0/1750633730 learned_addr learned my addr 172.21.15.6:0/1750633730 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.204+0000 7fb017400640 1 --2- 172.21.15.6:0/1750633730 >> v2:172.21.15.178:3300/0 conn(0x7fb01815dee0 0x7fb0180d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.205+0000 7fb016a00640 1 -- 172.21.15.6:0/1750633730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 msgr2=0x7fb0180d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.205+0000 7fb017e00640 1 --2- 172.21.15.6:0/1750633730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0180d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.205+0000 7fb016a00640 1 --2- 172.21.15.6:0/1750633730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0180d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.205+0000 7fb016a00640 1 -- 172.21.15.6:0/1750633730 >> v2:172.21.15.178:3300/0 conn(0x7fb01815dee0 msgr2=0x7fb0180d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.205+0000 7fb016a00640 1 --2- 172.21.15.6:0/1750633730 >> v2:172.21.15.178:3300/0 conn(0x7fb01815dee0 0x7fb0180d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.205+0000 7fb016a00640 1 -- 172.21.15.6:0/1750633730 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180cf260 con 0x7fb0180d4860 2024-03-06T07:28:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.206+0000 7fb016a00640 1 --2- 172.21.15.6:0/1750633730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0180d4860 0x7fb0180d0630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb01808c6f0 tx=0x7fb00402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1750171c38fa9d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.206+0000 7fb016000640 1 -- 172.21.15.6:0/1750633730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004035020 con 0x7fb0180d4860 2024-03-06T07:28:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.206+0000 7fb016000640 1 -- 172.21.15.6:0/1750633730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb004031870 con 0x7fb0180d4860 2024-03-06T07:28:05.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.206+0000 7fb016000640 1 -- 172.21.15.6:0/1750633730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004031c40 con 0x7fb0180d4860 2024-03-06T07:28:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 -- 172.21.15.6:0/1750633730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0180d4860 msgr2=0x7fb0180d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 --2- 172.21.15.6:0/1750633730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0180d4860 0x7fb0180d0630 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb01808c6f0 tx=0x7fb00402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 -- 172.21.15.6:0/1750633730 shutdown_connections 2024-03-06T07:28:05.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 --2- 172.21.15.6:0/1750633730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0180d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 --2- 172.21.15.6:0/1750633730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0180d4860 0x7fb0180d0630 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 --2- 172.21.15.6:0/1750633730 >> v2:172.21.15.178:3300/0 conn(0x7fb01815dee0 0x7fb0180d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 -- 172.21.15.6:0/1750633730 >> 172.21.15.6:0/1750633730 conn(0x7fb0180d3cd0 msgr2=0x7fb0180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 -- 172.21.15.6:0/1750633730 shutdown_connections 2024-03-06T07:28:05.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.207+0000 7fb01ce00640 1 -- 172.21.15.6:0/1750633730 wait complete. 2024-03-06T07:28:05.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.209+0000 7fb01ce00640 1 Processor -- start 2024-03-06T07:28:05.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.209+0000 7fb01ce00640 1 -- start start 2024-03-06T07:28:05.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb01ce00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0181604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb01ce00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0180d4860 0x7fb0181f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb01ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 0x7fb0181f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb01ce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180d1de0 con 0x7fb01815dee0 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb01ce00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01815e2c0 con 0x7fb0180d4860 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb01ce00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180d1f60 con 0x7fb0180cfa60 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb017e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 0x7fb0181f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb017e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 0x7fb0181f79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32980/0 (socket says 172.21.15.6:32980) 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb017e00640 1 -- 172.21.15.6:0/720108914 learned_addr learned my addr 172.21.15.6:0/720108914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb016a00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0180d4860 0x7fb0181f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.210+0000 7fb017400640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0181604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb017e00640 1 -- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 msgr2=0x7fb0181604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb017e00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0181604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb017e00640 1 -- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0180d4860 msgr2=0x7fb0181f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb017e00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0180d4860 0x7fb0181f7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb017e00640 1 -- 172.21.15.6:0/720108914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180c7f80 con 0x7fb01815dee0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb017e00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 0x7fb0181f79d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb01000ccf0 tx=0x7fb01000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb010003b90 con 0x7fb01815dee0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb010003d30 con 0x7fb01815dee0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb010004240 con 0x7fb01815dee0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.211+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180c8260 con 0x7fb01815dee0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.212+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0180c8820 con 0x7fb01815dee0 2024-03-06T07:28:05.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.213+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01804f330 con 0x7fb01815dee0 2024-03-06T07:28:05.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.213+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0100043e0 con 0x7fb01815dee0 2024-03-06T07:28:05.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.214+0000 7fb00f400640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faff806b4d0 0x7faff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.215+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb01008ad10 con 0x7fb01815dee0 2024-03-06T07:28:05.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.215+0000 7fb017400640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faff806b4d0 0x7faff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.216+0000 7fb017400640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faff806b4d0 0x7faff806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb008006fd0 tx=0x7fb008008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.219+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb010090020 con 0x7fb01815dee0 2024-03-06T07:28:05.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.423+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb01815cbd0 con 0x7fb01815dee0 2024-03-06T07:28:05.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.424+0000 7fb00f400640 1 -- 172.21.15.6:0/720108914 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb010059b80 con 0x7fb01815dee0 2024-03-06T07:28:05.426 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.426+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faff806b4d0 msgr2=0x7faff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.426+0000 7fb01ce00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faff806b4d0 0x7faff806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb008006fd0 tx=0x7fb008008040 comp rx=0 tx=0).stop 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.426+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 msgr2=0x7fb0181f79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.426+0000 7fb01ce00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 0x7fb0181f79d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb01000ccf0 tx=0x7fb01000a660 comp rx=0 tx=0).stop 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.426+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 shutdown_connections 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faff806b4d0 0x7faff806d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0180d4860 0x7fb0181f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb01815dee0 0x7fb0181f79d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 --2- 172.21.15.6:0/720108914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0180cfa60 0x7fb0181604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 >> 172.21.15.6:0/720108914 conn(0x7fb0180d3cd0 msgr2=0x7fb01806cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 shutdown_connections 2024-03-06T07:28:05.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.427+0000 7fb01ce00640 1 -- 172.21.15.6:0/720108914 wait complete. 2024-03-06T07:28:05.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:05.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.671+0000 7f50df200640 1 Processor -- start 2024-03-06T07:28:05.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.672+0000 7f50df200640 1 -- start start 2024-03-06T07:28:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.673+0000 7f50df200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 0x7f50e00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.673+0000 7f50df200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e00b3710 con 0x7f50e00d0e50 2024-03-06T07:28:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.673+0000 7f50df200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e00b39f0 con 0x7f50e00d4a10 2024-03-06T07:28:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.673+0000 7f50df200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00b09e0 con 0x7f50e015e0b0 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50de800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f50e00d0e50 0x7f50e00b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44376/0 (socket says 172.21.15.6:44376) 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50de800640 1 -- 172.21.15.6:0/1033080221 learned_addr learned my addr 172.21.15.6:0/1033080221 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50dde00640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 0x7f50e00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468672359 0 0) 0x7f50e00b3710 con 0x7f50e00d0e50 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c0003520 con 0x7f50e00d0e50 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995772125 0 0) 0x7f50e00b39f0 con 0x7f50e00d4a10 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e00b3710 con 0x7f50e00d4a10 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.674+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387131382 0 0) 0x7f50c0003520 con 0x7f50e00d0e50 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 msgr2=0x7f50e00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 0x7f50e00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 >> v1:172.21.15.178:6789/0 conn(0x7f50e00d4a10 legacy=0x7f50e00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e00b3cd0 con 0x7f50e00d0e50 2024-03-06T07:28:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f50d80019f0 con 0x7f50e00d0e50 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 >> v1:172.21.15.6:6789/0 conn(0x7f50e00d0e50 legacy=0x7f50e00b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 0x7f50c0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50c0004d40 0x7f50c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c000bad0 con 0x7f50c0004120 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c000bc80 con 0x7f50c0004d40 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c000be80 con 0x7f50e015e0b0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dd400640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dde00640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 0x7f50c0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50de800640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50c0004d40 0x7f50c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dd400640 1 -- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 msgr2=0x7f50c0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dd400640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e015e0b0 0x7f50c0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dd400640 1 -- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50c0004d40 msgr2=0x7f50c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dd400640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50c0004d40 0x7f50c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.675+0000 7f50dd400640 1 -- 172.21.15.6:0/1033080221 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0003520 con 0x7f50c0004120 2024-03-06T07:28:05.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.676+0000 7f50de800640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50c0004d40 0x7f50c00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.677+0000 7f50dd400640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50c0004800 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f50c80018a0 tx=0x7f50c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163b6c4355942e78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.677+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80014f0 con 0x7f50c0004120 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.677+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c80052d0 con 0x7f50c0004120 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.677+0000 7f50dca00640 1 -- 172.21.15.6:0/1033080221 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c8003810 con 0x7f50c0004120 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50df200640 1 -- 172.21.15.6:0/1033080221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 msgr2=0x7f50c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50df200640 1 --2- 172.21.15.6:0/1033080221 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50c0004800 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f50c80018a0 tx=0x7f50c0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50de800640 1 -- 172.21.15.6:0/1033080221 reap_dead start 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50df200640 1 -- 172.21.15.6:0/1033080221 shutdown_connections 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50df200640 1 -- 172.21.15.6:0/1033080221 >> 172.21.15.6:0/1033080221 conn(0x7f50e00d3d30 msgr2=0x7f50e00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50df200640 1 -- 172.21.15.6:0/1033080221 shutdown_connections 2024-03-06T07:28:05.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.678+0000 7f50df200640 1 -- 172.21.15.6:0/1033080221 wait complete. 2024-03-06T07:28:05.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.680+0000 7f50df200640 1 Processor -- start 2024-03-06T07:28:05.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.680+0000 7f50df200640 1 -- start start 2024-03-06T07:28:05.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50df200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50e01f7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50df200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e01f2fa0 0x7f50e00b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50df200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e01f3420 0x7f50e00b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50df200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00cfc60 con 0x7f50c0004120 2024-03-06T07:28:05.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50df200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e015e490 con 0x7f50e01f3420 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50df200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00cf310 con 0x7f50e01f2fa0 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50e01f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50e01f7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32998/0 (socket says 172.21.15.6:32998) 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 -- 172.21.15.6:0/3143904303 learned_addr learned my addr 172.21.15.6:0/3143904303 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 -- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e01f2fa0 msgr2=0x7f50e00b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e01f2fa0 0x7f50e00b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 -- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e01f3420 msgr2=0x7f50e00b86f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e01f3420 0x7f50e00b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.681+0000 7f50dde00640 1 -- 172.21.15.6:0/3143904303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e01f7d60 con 0x7f50c0004120 2024-03-06T07:28:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.682+0000 7f50dde00640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50e01f7820 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f50d0011630 tx=0x7f50d0011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.682+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d000f920 con 0x7f50c0004120 2024-03-06T07:28:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.682+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d000fac0 con 0x7f50c0004120 2024-03-06T07:28:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.682+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d000fe70 con 0x7f50c0004120 2024-03-06T07:28:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.682+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e00b6cc0 con 0x7f50c0004120 2024-03-06T07:28:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.682+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e00b7280 con 0x7f50c0004120 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.683+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e004f330 con 0x7f50c0004120 2024-03-06T07:28:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.684+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50d0015460 con 0x7f50c0004120 2024-03-06T07:28:05.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.686+0000 7f50d6000640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50bc06b4d0 0x7f50bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.686+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50d008a810 con 0x7f50c0004120 2024-03-06T07:28:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.686+0000 7f50dd400640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50bc06b4d0 0x7f50bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.687+0000 7f50dd400640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50bc06b4d0 0x7f50bc06d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f50c8033000 tx=0x7f50c8009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.692+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50e01f7d60 con 0x7f50c0004120 2024-03-06T07:28:05.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.893+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50e00740f0 con 0x7f50c0004120 2024-03-06T07:28:05.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.896+0000 7f50d6000640 1 -- 172.21.15.6:0/3143904303 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f50d0019030 con 0x7f50c0004120 2024-03-06T07:28:05.899 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:05.899 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:05.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50bc06b4d0 msgr2=0x7f50bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50bc06b4d0 0x7f50bc06d990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f50c8033000 tx=0x7f50c8009030 comp rx=0 tx=0).stop 2024-03-06T07:28:05.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 msgr2=0x7f50e01f7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50e01f7820 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f50d0011630 tx=0x7f50d0011af0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 shutdown_connections 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50bc06b4d0 0x7f50bc06d990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e01f3420 0x7f50e00b86f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e01f2fa0 0x7f50e00b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 --2- 172.21.15.6:0/3143904303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50c0004120 0x7f50e01f7820 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 >> 172.21.15.6:0/3143904303 conn(0x7f50e00d3d30 msgr2=0x7f50e00b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 shutdown_connections 2024-03-06T07:28:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:05.900+0000 7f50df200640 1 -- 172.21.15.6:0/3143904303 wait complete. 2024-03-06T07:28:05.921 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:05.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.144+0000 7f55f3c00640 1 Processor -- start 2024-03-06T07:28:06.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.144+0000 7f55f3c00640 1 -- start start 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.146+0000 7f55f3c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f55ec071e20 0x7f55ec0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.146+0000 7f55f3c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ec072df0 con 0x7f55ec15bf20 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.146+0000 7f55f3c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ec0cdab0 con 0x7f55ec072520 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.146+0000 7f55f3c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ec0b4d70 con 0x7f55ec071e20 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.146+0000 7f55f1200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f55ec15bf20 0x7f55ec0acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44384/0 (socket says 172.21.15.6:44384) 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.146+0000 7f55f1200640 1 -- 172.21.15.6:0/636003364 learned_addr learned my addr 172.21.15.6:0/636003364 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55f1c00640 1 --2- 172.21.15.6:0/636003364 >> v2:172.21.15.178:3300/0 conn(0x7f55ec071e20 0x7f55ec0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55eb400640 1 -- 172.21.15.6:0/636003364 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560521415 0 0) 0x7f55ec072df0 con 0x7f55ec15bf20 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55eb400640 1 -- 172.21.15.6:0/636003364 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d4003520 con 0x7f55ec15bf20 2024-03-06T07:28:06.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55f1c00640 1 -- 172.21.15.6:0/636003364 >> v1:172.21.15.178:6789/0 conn(0x7f55ec072520 legacy=0x7f55ec0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55eb400640 1 -- 172.21.15.6:0/636003364 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3693817303 0 0) 0x7f55d4003520 con 0x7f55ec15bf20 2024-03-06T07:28:06.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55f1c00640 1 -- 172.21.15.6:0/636003364 >> v1:172.21.15.6:6789/0 conn(0x7f55ec15bf20 legacy=0x7f55ec0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.147+0000 7f55f1c00640 1 -- 172.21.15.6:0/636003364 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec0b5050 con 0x7f55ec071e20 2024-03-06T07:28:06.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.149+0000 7f55f1c00640 1 --2- 172.21.15.6:0/636003364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec071e20 0x7f55ec0ccad0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009d90 tx=0x7f55e402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=886f1586c82263de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.149+0000 7f55eb400640 1 -- 172.21.15.6:0/636003364 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e4039070 con 0x7f55ec071e20 2024-03-06T07:28:06.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.149+0000 7f55eb400640 1 -- 172.21.15.6:0/636003364 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e402c940 con 0x7f55ec071e20 2024-03-06T07:28:06.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.149+0000 7f55eb400640 1 -- 172.21.15.6:0/636003364 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0001670 con 0x7f55ec071e20 2024-03-06T07:28:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 -- 172.21.15.6:0/636003364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec071e20 msgr2=0x7f55ec0ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 --2- 172.21.15.6:0/636003364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec071e20 0x7f55ec0ccad0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f55e4009d90 tx=0x7f55e402be10 comp rx=0 tx=0).stop 2024-03-06T07:28:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 -- 172.21.15.6:0/636003364 shutdown_connections 2024-03-06T07:28:06.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 --2- 172.21.15.6:0/636003364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec071e20 0x7f55ec0ccad0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 -- 172.21.15.6:0/636003364 >> 172.21.15.6:0/636003364 conn(0x7f55ec06b3d0 msgr2=0x7f55ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 -- 172.21.15.6:0/636003364 shutdown_connections 2024-03-06T07:28:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.150+0000 7f55f3c00640 1 -- 172.21.15.6:0/636003364 wait complete. 2024-03-06T07:28:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.152+0000 7f55f3c00640 1 Processor -- start 2024-03-06T07:28:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.152+0000 7f55f3c00640 1 -- start start 2024-03-06T07:28:06.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f3c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 0x7f55ec1507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f3c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec072520 0x7f55ec0cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f3c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55ec15bf20 0x7f55ec09b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f3c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0ad450 con 0x7f55ec071e20 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f3c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0b4990 con 0x7f55ec072520 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f3c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec0ca350 con 0x7f55ec15bf20 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 0x7f55ec1507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 0x7f55ec1507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33000/0 (socket says 172.21.15.6:33000) 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.153+0000 7f55f1200640 1 -- 172.21.15.6:0/2563817347 learned_addr learned my addr 172.21.15.6:0/2563817347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:06.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55ebe00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec072520 0x7f55ec0cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55f1200640 1 -- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55ec15bf20 msgr2=0x7f55ec09b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55f1200640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55ec15bf20 0x7f55ec09b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55f1200640 1 -- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec072520 msgr2=0x7f55ec0cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55f1200640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec072520 0x7f55ec0cadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55f1200640 1 -- 172.21.15.6:0/2563817347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ec09bec0 con 0x7f55ec071e20 2024-03-06T07:28:06.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55f1200640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 0x7f55ec1507b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f55e000c6d0 tx=0x7f55e000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e0005ad0 con 0x7f55ec071e20 2024-03-06T07:28:06.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e0005c90 con 0x7f55ec071e20 2024-03-06T07:28:06.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.154+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e00072b0 con 0x7f55ec071e20 2024-03-06T07:28:06.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.155+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec0ca5f0 con 0x7f55ec071e20 2024-03-06T07:28:06.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.155+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec150cf0 con 0x7f55ec071e20 2024-03-06T07:28:06.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.156+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec04f330 con 0x7f55ec071e20 2024-03-06T07:28:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.157+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55e0007450 con 0x7f55ec071e20 2024-03-06T07:28:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.158+0000 7f55e8c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55cc06b500 0x7f55cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.158+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55e0089fe0 con 0x7f55ec071e20 2024-03-06T07:28:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.159+0000 7f55ebe00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55cc06b500 0x7f55cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.160+0000 7f55ebe00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55cc06b500 0x7f55cc06d9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f55d8007fd0 tx=0x7f55d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.163+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55e0058e70 con 0x7f55ec071e20 2024-03-06T07:28:06.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.362+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55ec15f720 con 0x7f55ec071e20 2024-03-06T07:28:06.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.362+0000 7f55e8c00640 1 -- 172.21.15.6:0/2563817347 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55e0058770 con 0x7f55ec071e20 2024-03-06T07:28:06.364 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55cc06b500 msgr2=0x7f55cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55cc06b500 0x7f55cc06d9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f55d8007fd0 tx=0x7f55d8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 msgr2=0x7f55ec1507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 0x7f55ec1507b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f55e000c6d0 tx=0x7f55e000cb90 comp rx=0 tx=0).stop 2024-03-06T07:28:06.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 shutdown_connections 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55cc06b500 0x7f55cc06d9c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55ec072520 0x7f55ec0cadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55ec15bf20 0x7f55ec09b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 --2- 172.21.15.6:0/2563817347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55ec071e20 0x7f55ec1507b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 >> 172.21.15.6:0/2563817347 conn(0x7f55ec06b3d0 msgr2=0x7f55ec15b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 shutdown_connections 2024-03-06T07:28:06.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.365+0000 7f55f3c00640 1 -- 172.21.15.6:0/2563817347 wait complete. 2024-03-06T07:28:06.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.609+0000 7f4972200640 1 Processor -- start 2024-03-06T07:28:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.610+0000 7f4972200640 1 -- start start 2024-03-06T07:28:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.611+0000 7f4972200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f496c15dee0 0x7f496c0d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.611+0000 7f4972200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0ceca0 con 0x7f496c0d4860 2024-03-06T07:28:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.611+0000 7f4972200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f496c0cef80 con 0x7f496c0d0d40 2024-03-06T07:28:06.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.611+0000 7f4972200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496c0cf5d0 con 0x7f496c15dee0 2024-03-06T07:28:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.612+0000 7f496be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f496c0d4860 0x7f496c0d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44396/0 (socket says 172.21.15.6:44396) 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.612+0000 7f496be00640 1 -- 172.21.15.6:0/68177891 learned_addr learned my addr 172.21.15.6:0/68177891 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.612+0000 7f4970e00640 1 --2- 172.21.15.6:0/68177891 >> v2:172.21.15.178:3300/0 conn(0x7f496c15dee0 0x7f496c0d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.612+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555673701 0 0) 0x7f496c0ceca0 con 0x7f496c0d4860 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.612+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4950003520 con 0x7f496c0d4860 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789733294 0 0) 0x7f4950003520 con 0x7f496c0d4860 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 >> v2:172.21.15.178:3300/0 conn(0x7f496c15dee0 msgr2=0x7f496c0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 --2- 172.21.15.6:0/68177891 >> v2:172.21.15.178:3300/0 conn(0x7f496c15dee0 0x7f496c0d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 >> v1:172.21.15.178:6789/0 conn(0x7f496c0d0d40 legacy=0x7f496c0b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c0cf260 con 0x7f496c0d4860 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f495c001670 con 0x7f496c0d4860 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 >> v1:172.21.15.6:6789/0 conn(0x7f496c0d4860 legacy=0x7f496c0d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f4950003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4950004800 0x7f4950004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49500051c0 0x7f495000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0cf5d0 con 0x7f49500039b0 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495000c060 con 0x7f49500051c0 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495000c260 con 0x7f4950004800 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f4970e00640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f4950003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.613+0000 7f4971800640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49500051c0 0x7f495000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4970e00640 1 -- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4950004800 msgr2=0x7f4950004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4970e00640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4950004800 0x7f4950004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4970e00640 1 -- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49500051c0 msgr2=0x7f495000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4970e00640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49500051c0 0x7f495000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4970e00640 1 -- 172.21.15.6:0/68177891 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4950003520 con 0x7f49500039b0 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4971800640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f49500051c0 0x7f495000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.614+0000 7f4971800640 1 -- 172.21.15.6:0/68177891 reap_dead start 2024-03-06T07:28:06.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.615+0000 7f4970e00640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f4950003dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4954009ac0 tx=0x7f4950005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a2f1280c86349 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.615+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954039070 con 0x7f49500039b0 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.615+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495402c6b0 con 0x7f49500039b0 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.615+0000 7f496b400640 1 -- 172.21.15.6:0/68177891 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954009690 con 0x7f49500039b0 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 -- 172.21.15.6:0/68177891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 msgr2=0x7f4950003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f4950003dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4954009ac0 tx=0x7f4950005830 comp rx=0 tx=0).stop 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 -- 172.21.15.6:0/68177891 shutdown_connections 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 --2- 172.21.15.6:0/68177891 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f4950003dd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 -- 172.21.15.6:0/68177891 >> 172.21.15.6:0/68177891 conn(0x7f496c0d3cd0 msgr2=0x7f496c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 -- 172.21.15.6:0/68177891 shutdown_connections 2024-03-06T07:28:06.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.616+0000 7f4972200640 1 -- 172.21.15.6:0/68177891 wait complete. 2024-03-06T07:28:06.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.618+0000 7f4972200640 1 Processor -- start 2024-03-06T07:28:06.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.618+0000 7f4972200640 1 -- start start 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4972200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f496c1512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4972200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f496c160ed0 0x7f496c161350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4972200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f496c0cdc40 0x7f496c0ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4972200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0cfa60 con 0x7f49500039b0 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4972200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496c15e2c0 con 0x7f496c160ed0 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4972200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c0cfbe0 con 0x7f496c0cdc40 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f496c1512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f496c1512d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33026/0 (socket says 172.21.15.6:33026) 2024-03-06T07:28:06.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 -- 172.21.15.6:0/1816240447 learned_addr learned my addr 172.21.15.6:0/1816240447 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 -- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f496c0cdc40 msgr2=0x7f496c0ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f496be00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f496c160ed0 0x7f496c161350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f496c0cdc40 0x7f496c0ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 -- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f496c160ed0 msgr2=0x7f496c161350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.619+0000 7f4970e00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f496c160ed0 0x7f496c161350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4970e00640 1 -- 172.21.15.6:0/1816240447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c15fe30 con 0x7f49500039b0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4970e00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f496c1512d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4954037040 tx=0x7f4954004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954042020 con 0x7f49500039b0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495403a040 con 0x7f49500039b0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496c160110 con 0x7f49500039b0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4954039070 con 0x7f49500039b0 2024-03-06T07:28:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.620+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496c1606d0 con 0x7f49500039b0 2024-03-06T07:28:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.622+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496c06d670 con 0x7f49500039b0 2024-03-06T07:28:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.623+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4954045020 con 0x7f49500039b0 2024-03-06T07:28:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.624+0000 7f4968c00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f494806b500 0x7f494806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.624+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49540aae80 con 0x7f49500039b0 2024-03-06T07:28:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.624+0000 7f496be00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f494806b500 0x7f494806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.626+0000 7f496be00640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f494806b500 0x7f494806d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f495c000f60 tx=0x7f495c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.629+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f496c06d670 con 0x7f49500039b0 2024-03-06T07:28:06.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.832+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f496c08d7d0 con 0x7f49500039b0 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.833+0000 7f4968c00640 1 -- 172.21.15.6:0/1816240447 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4954079d00 con 0x7f49500039b0 2024-03-06T07:28:06.835 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:06.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f494806b500 msgr2=0x7f494806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f494806b500 0x7f494806d9c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f495c000f60 tx=0x7f495c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 msgr2=0x7f496c1512d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f496c1512d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4954037040 tx=0x7f4954004730 comp rx=0 tx=0).stop 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 shutdown_connections 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f494806b500 0x7f494806d9c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f496c160ed0 0x7f496c161350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f496c0cdc40 0x7f496c0ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 --2- 172.21.15.6:0/1816240447 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49500039b0 0x7f496c1512d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 >> 172.21.15.6:0/1816240447 conn(0x7f496c0d3cd0 msgr2=0x7f496c06cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.836+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 shutdown_connections 2024-03-06T07:28:06.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:06.837+0000 7f4972200640 1 -- 172.21.15.6:0/1816240447 wait complete. 2024-03-06T07:28:06.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:07.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.080+0000 7f349ee00640 1 Processor -- start 2024-03-06T07:28:07.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.080+0000 7f349ee00640 1 -- start start 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f349ee00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3498162bb0 0x7f3498163040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f349ee00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 0x7f34980adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f349ee00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3498163950 con 0x7f34980ccad0 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f349ee00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34980acd40 con 0x7f3498162bb0 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f349ee00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980ad020 con 0x7f34980adb00 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f3497e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f34980ccad0 0x7f34980cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44404/0 (socket says 172.21.15.6:44404) 2024-03-06T07:28:07.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.082+0000 7f3497e00640 1 -- 172.21.15.6:0/1670623687 learned_addr learned my addr 172.21.15.6:0/1670623687 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080462576 0 0) 0x7f3498163950 con 0x7f34980ccad0 2024-03-06T07:28:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c003520 con 0x7f34980ccad0 2024-03-06T07:28:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f349ce00640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 0x7f34980adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f348f400640 1 --2- 172.21.15.6:0/1670623687 >> v2:172.21.15.178:3300/0 conn(0x7f3498162bb0 0x7f3498163040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3921689403 0 0) 0x7f347c003520 con 0x7f34980ccad0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 msgr2=0x7f34980adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 0x7f34980adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 >> v2:172.21.15.178:3300/0 conn(0x7f3498162bb0 msgr2=0x7f3498163040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 --2- 172.21.15.6:0/1670623687 >> v2:172.21.15.178:3300/0 conn(0x7f3498162bb0 0x7f3498163040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980ad300 con 0x7f34980ccad0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f3488001670 con 0x7f34980ccad0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 >> v1:172.21.15.6:6789/0 conn(0x7f34980ccad0 legacy=0x7f34980cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f347c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 0x7f347c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980acd40 con 0x7f347c0039b0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34980ad020 con 0x7f347c004800 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347c00bac0 con 0x7f34980adb00 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.083+0000 7f3497e00640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f347c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f348f400640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f3497e00640 1 -- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 msgr2=0x7f347c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f3497e00640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f34980adb00 0x7f347c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f3497e00640 1 -- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f347c004800 msgr2=0x7f347c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f3497e00640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f3497e00640 1 -- 172.21.15.6:0/1670623687 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c003520 con 0x7f347c0039b0 2024-03-06T07:28:07.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.084+0000 7f348f400640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f347c004800 0x7f347c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.085+0000 7f3497e00640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f347c003dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3488001eb0 tx=0x7f347c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f663054de45a416a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.085+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488007ca0 con 0x7f347c0039b0 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.085+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488030070 con 0x7f347c0039b0 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.085+0000 7f3497400640 1 -- 172.21.15.6:0/1670623687 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488005d10 con 0x7f347c0039b0 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ee00640 1 -- 172.21.15.6:0/1670623687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 msgr2=0x7f347c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ee00640 1 --2- 172.21.15.6:0/1670623687 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f347c003dd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3488001eb0 tx=0x7f347c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:07.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ce00640 1 -- 172.21.15.6:0/1670623687 reap_dead start 2024-03-06T07:28:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ee00640 1 -- 172.21.15.6:0/1670623687 shutdown_connections 2024-03-06T07:28:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ee00640 1 -- 172.21.15.6:0/1670623687 >> 172.21.15.6:0/1670623687 conn(0x7f349806b3d0 msgr2=0x7f349806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ee00640 1 -- 172.21.15.6:0/1670623687 shutdown_connections 2024-03-06T07:28:07.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.086+0000 7f349ee00640 1 -- 172.21.15.6:0/1670623687 wait complete. 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.088+0000 7f349ee00640 1 Processor -- start 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.088+0000 7f349ee00640 1 -- start start 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f349814c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ee00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f349809b2c0 0x7f34981ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ee00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34981eb260 0x7f349814eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349806c980 con 0x7f347c0039b0 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ee00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349806e7d0 con 0x7f34981eb260 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ee00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3498163580 con 0x7f349809b2c0 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f349814c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f349814c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33052/0 (socket says 172.21.15.6:33052) 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 -- 172.21.15.6:0/2084413782 learned_addr learned my addr 172.21.15.6:0/2084413782 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f349ce00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34981eb260 0x7f349814eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f348f400640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f349809b2c0 0x7f34981ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 -- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f349809b2c0 msgr2=0x7f34981ead20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f349809b2c0 0x7f34981ead20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 -- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34981eb260 msgr2=0x7f349814eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34981eb260 0x7f349814eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f3497e00640 1 -- 172.21.15.6:0/2084413782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349814d200 con 0x7f347c0039b0 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.089+0000 7f348f400640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f349809b2c0 0x7f34981ead20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.090+0000 7f3497e00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f349814c660 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f348802c460 tx=0x7f3488000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.090+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488004bf0 con 0x7f347c0039b0 2024-03-06T07:28:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.090+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3488004d90 con 0x7f347c0039b0 2024-03-06T07:28:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.090+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880072b0 con 0x7f347c0039b0 2024-03-06T07:28:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.090+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349814d4e0 con 0x7f347c0039b0 2024-03-06T07:28:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.090+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349814daa0 con 0x7f347c0039b0 2024-03-06T07:28:07.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.092+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349804f330 con 0x7f347c0039b0 2024-03-06T07:28:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.093+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3488047020 con 0x7f347c0039b0 2024-03-06T07:28:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.094+0000 7f3494c00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f347406b520 0x7f347406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.094+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34880ab3f0 con 0x7f347c0039b0 2024-03-06T07:28:07.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.094+0000 7f348f400640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f347406b520 0x7f347406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.095+0000 7f348f400640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f347406b520 0x7f347406d9e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34981ec720 tx=0x7f3480008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.098+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f348807a2c0 con 0x7f347c0039b0 2024-03-06T07:28:07.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.302+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34980cefd0 con 0x7f347c0039b0 2024-03-06T07:28:07.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.303+0000 7f3494c00640 1 -- 172.21.15.6:0/2084413782 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3488079bc0 con 0x7f347c0039b0 2024-03-06T07:28:07.305 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:07.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f347406b520 msgr2=0x7f347406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f347406b520 0x7f347406d9e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34981ec720 tx=0x7f3480008040 comp rx=0 tx=0).stop 2024-03-06T07:28:07.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 msgr2=0x7f349814c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f349814c660 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f348802c460 tx=0x7f3488000fe0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 shutdown_connections 2024-03-06T07:28:07.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f347406b520 0x7f347406d9e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34981eb260 0x7f349814eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f349809b2c0 0x7f34981ead20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 --2- 172.21.15.6:0/2084413782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f347c0039b0 0x7f349814c660 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.305+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 >> 172.21.15.6:0/2084413782 conn(0x7f349806b3d0 msgr2=0x7f34980cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.306+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 shutdown_connections 2024-03-06T07:28:07.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.306+0000 7f349ee00640 1 -- 172.21.15.6:0/2084413782 wait complete. 2024-03-06T07:28:07.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:07.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.547+0000 7efdd4200640 1 Processor -- start 2024-03-06T07:28:07.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.547+0000 7efdd4200640 1 -- start start 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd4200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdcc070ee0 0x7efdcc162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd4200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc072da0 0x7efdcc0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd4200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7efdcc0712c0 0x7efdcc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd4200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0ceca0 con 0x7efdcc070ee0 2024-03-06T07:28:07.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd4200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0ce090 con 0x7efdcc0712c0 2024-03-06T07:28:07.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd4200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0ce370 con 0x7efdcc072da0 2024-03-06T07:28:07.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdcc070ee0 0x7efdcc162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd1800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdcc070ee0 0x7efdcc162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33062/0 (socket says 172.21.15.6:33062) 2024-03-06T07:28:07.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.549+0000 7efdd1800640 1 -- 172.21.15.6:0/3725768535 learned_addr learned my addr 172.21.15.6:0/3725768535 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd2200640 1 --2- 172.21.15.6:0/3725768535 >> v2:172.21.15.178:3300/0 conn(0x7efdcc0712c0 0x7efdcc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd0e00640 1 --2- 172.21.15.6:0/3725768535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc072da0 0x7efdcc0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd1800640 1 -- 172.21.15.6:0/3725768535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc072da0 msgr2=0x7efdcc0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd1800640 1 --2- 172.21.15.6:0/3725768535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc072da0 0x7efdcc0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd1800640 1 -- 172.21.15.6:0/3725768535 >> v2:172.21.15.178:3300/0 conn(0x7efdcc0712c0 msgr2=0x7efdcc06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd1800640 1 --2- 172.21.15.6:0/3725768535 >> v2:172.21.15.178:3300/0 conn(0x7efdcc0712c0 0x7efdcc06e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.550+0000 7efdd1800640 1 -- 172.21.15.6:0/3725768535 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0ce650 con 0x7efdcc070ee0 2024-03-06T07:28:07.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.551+0000 7efdd1800640 1 --2- 172.21.15.6:0/3725768535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc070ee0 0x7efdcc162c40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efdcc06f040 tx=0x7efdc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79817fa663fee04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.551+0000 7efdcbe00640 1 -- 172.21.15.6:0/3725768535 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0035020 con 0x7efdcc070ee0 2024-03-06T07:28:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.551+0000 7efdcbe00640 1 -- 172.21.15.6:0/3725768535 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc0031870 con 0x7efdcc070ee0 2024-03-06T07:28:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.551+0000 7efdcbe00640 1 -- 172.21.15.6:0/3725768535 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0031c40 con 0x7efdcc070ee0 2024-03-06T07:28:07.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 -- 172.21.15.6:0/3725768535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc070ee0 msgr2=0x7efdcc162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 --2- 172.21.15.6:0/3725768535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc070ee0 0x7efdcc162c40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efdcc06f040 tx=0x7efdc002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 -- 172.21.15.6:0/3725768535 shutdown_connections 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 --2- 172.21.15.6:0/3725768535 >> v2:172.21.15.178:3300/0 conn(0x7efdcc0712c0 0x7efdcc06e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 --2- 172.21.15.6:0/3725768535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc072da0 0x7efdcc0cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 --2- 172.21.15.6:0/3725768535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc070ee0 0x7efdcc162c40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 -- 172.21.15.6:0/3725768535 >> 172.21.15.6:0/3725768535 conn(0x7efdcc06afb0 msgr2=0x7efdcc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 -- 172.21.15.6:0/3725768535 shutdown_connections 2024-03-06T07:28:07.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.552+0000 7efdd4200640 1 -- 172.21.15.6:0/3725768535 wait complete. 2024-03-06T07:28:07.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.554+0000 7efdd4200640 1 Processor -- start 2024-03-06T07:28:07.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.554+0000 7efdd4200640 1 -- start start 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd4200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc070ee0 0x7efdcc09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd4200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 0x7efdcc14c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd4200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdcc072da0 0x7efdcc09c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd4200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0cdda0 con 0x7efdcc0712c0 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd4200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0cdc20 con 0x7efdcc072da0 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd4200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc06e600 con 0x7efdcc070ee0 2024-03-06T07:28:07.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 0x7efdcc14c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.555+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 0x7efdcc14c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33078/0 (socket says 172.21.15.6:33078) 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 -- 172.21.15.6:0/4249251877 learned_addr learned my addr 172.21.15.6:0/4249251877 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd1800640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc070ee0 0x7efdcc09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd2200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdcc072da0 0x7efdcc09c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 -- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc070ee0 msgr2=0x7efdcc09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc070ee0 0x7efdcc09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 -- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdcc072da0 msgr2=0x7efdcc09c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdcc072da0 0x7efdcc09c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 -- 172.21.15.6:0/4249251877 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc14c370 con 0x7efdcc0712c0 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd2200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdcc072da0 0x7efdcc09c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdd0e00640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 0x7efdcc14c890 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efdb800d5f0 tx=0x7efdb800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.556+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80125f0 con 0x7efdcc0712c0 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.557+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb8019070 con 0x7efdcc0712c0 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.557+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80129d0 con 0x7efdcc0712c0 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.557+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc09b310 con 0x7efdcc0712c0 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.557+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc09b8d0 con 0x7efdcc0712c0 2024-03-06T07:28:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.558+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc04f3a0 con 0x7efdcc0712c0 2024-03-06T07:28:07.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.559+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdb8012b70 con 0x7efdcc0712c0 2024-03-06T07:28:07.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.560+0000 7efdc9600640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efdac06b4d0 0x7efdac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:07.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.560+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdb808a6a0 con 0x7efdcc0712c0 2024-03-06T07:28:07.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.560+0000 7efdd1800640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efdac06b4d0 0x7efdac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:07.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.562+0000 7efdd1800640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efdac06b4d0 0x7efdac06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efdc0036000 tx=0x7efdc003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:07.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.565+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdb8059480 con 0x7efdcc0712c0 2024-03-06T07:28:07.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.766+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdcc0b52d0 con 0x7efdcc0712c0 2024-03-06T07:28:07.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.769+0000 7efdc9600640 1 -- 172.21.15.6:0/4249251877 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16533 (secure 0 0 0) 0x7efdb8058d80 con 0x7efdcc0712c0 2024-03-06T07:28:07.772 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:07.772 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":22,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:27:31.226911+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efdac06b4d0 msgr2=0x7efdac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efdac06b4d0 0x7efdac06d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efdc0036000 tx=0x7efdc003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 msgr2=0x7efdcc14c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 0x7efdcc14c890 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efdb800d5f0 tx=0x7efdb800dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 shutdown_connections 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efdac06b4d0 0x7efdac06d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdcc072da0 0x7efdcc09c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdcc0712c0 0x7efdcc14c890 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 --2- 172.21.15.6:0/4249251877 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdcc070ee0 0x7efdcc09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 >> 172.21.15.6:0/4249251877 conn(0x7efdcc06afb0 msgr2=0x7efdcc166030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 shutdown_connections 2024-03-06T07:28:07.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:07.772+0000 7efdd4200640 1 -- 172.21.15.6:0/4249251877 wait complete. 2024-03-06T07:28:07.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:07.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:08.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.008+0000 7fdf84200640 1 Processor -- start 2024-03-06T07:28:08.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.008+0000 7fdf84200640 1 -- start start 2024-03-06T07:28:08.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf84200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c072520 0x7fdf7c0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf84200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf7c071e20 0x7fdf7c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf84200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c072df0 con 0x7fdf7c15bf20 2024-03-06T07:28:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf84200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c0cdab0 con 0x7fdf7c071e20 2024-03-06T07:28:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf84200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0b4d70 con 0x7fdf7c072520 2024-03-06T07:28:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf82200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf7c071e20 0x7fdf7c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf82200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdf7c071e20 0x7fdf7c0ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33088/0 (socket says 172.21.15.6:33088) 2024-03-06T07:28:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.010+0000 7fdf82200640 1 -- 172.21.15.6:0/2225870677 learned_addr learned my addr 172.21.15.6:0/2225870677 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.011+0000 7fdf80e00640 1 --2- 172.21.15.6:0/2225870677 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c072520 0x7fdf7c0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.011+0000 7fdf82200640 1 -- 172.21.15.6:0/2225870677 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c072520 msgr2=0x7fdf7c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.011+0000 7fdf82200640 1 --2- 172.21.15.6:0/2225870677 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c072520 0x7fdf7c0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.011+0000 7fdf82200640 1 -- 172.21.15.6:0/2225870677 >> v1:172.21.15.178:6789/0 conn(0x7fdf7c15bf20 legacy=0x7fdf7c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.011+0000 7fdf82200640 1 -- 172.21.15.6:0/2225870677 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0b5050 con 0x7fdf7c071e20 2024-03-06T07:28:08.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.012+0000 7fdf82200640 1 --2- 172.21.15.6:0/2225870677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0ce4f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009e50 tx=0x7fdf7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b9c372e4b6ab0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.012+0000 7fdf7be00640 1 -- 172.21.15.6:0/2225870677 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7402cb00 con 0x7fdf7c071e20 2024-03-06T07:28:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.012+0000 7fdf7be00640 1 -- 172.21.15.6:0/2225870677 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7402cca0 con 0x7fdf7c071e20 2024-03-06T07:28:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf7be00640 1 -- 172.21.15.6:0/2225870677 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74034610 con 0x7fdf7c071e20 2024-03-06T07:28:08.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 -- 172.21.15.6:0/2225870677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 msgr2=0x7fdf7c0ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 --2- 172.21.15.6:0/2225870677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0ce4f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009e50 tx=0x7fdf7402bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 -- 172.21.15.6:0/2225870677 shutdown_connections 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 --2- 172.21.15.6:0/2225870677 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0ce4f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 --2- 172.21.15.6:0/2225870677 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c072520 0x7fdf7c0cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 -- 172.21.15.6:0/2225870677 >> 172.21.15.6:0/2225870677 conn(0x7fdf7c06b3d0 msgr2=0x7fdf7c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 -- 172.21.15.6:0/2225870677 shutdown_connections 2024-03-06T07:28:08.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.013+0000 7fdf84200640 1 -- 172.21.15.6:0/2225870677 wait complete. 2024-03-06T07:28:08.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.015+0000 7fdf84200640 1 Processor -- start 2024-03-06T07:28:08.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.015+0000 7fdf84200640 1 -- start start 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf84200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf84200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf7c072520 0x7fdf7c149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf84200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c15bf20 0x7fdf7c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf84200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0ad450 con 0x7fdf7c071e20 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf84200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0b4990 con 0x7fdf7c072520 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf84200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0ca510 con 0x7fdf7c15bf20 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf81800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf81800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33096/0 (socket says 172.21.15.6:33096) 2024-03-06T07:28:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.016+0000 7fdf81800640 1 -- 172.21.15.6:0/1128351914 learned_addr learned my addr 172.21.15.6:0/1128351914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf80e00640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf7c072520 0x7fdf7c149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf82200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c15bf20 0x7fdf7c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf81800640 1 -- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c15bf20 msgr2=0x7fdf7c0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf81800640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c15bf20 0x7fdf7c0affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf81800640 1 -- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf7c072520 msgr2=0x7fdf7c149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf81800640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf7c072520 0x7fdf7c149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf81800640 1 -- 172.21.15.6:0/1128351914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0ca7f0 con 0x7fdf7c071e20 2024-03-06T07:28:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf80e00640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf7c072520 0x7fdf7c149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.017+0000 7fdf81800640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0b2050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c3c0 tx=0x7fdf7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.018+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700052d0 con 0x7fdf7c071e20 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.018+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70005490 con 0x7fdf7c071e20 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.018+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c0caad0 con 0x7fdf7c071e20 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.018+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70007260 con 0x7fdf7c071e20 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.018+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c0b2590 con 0x7fdf7c071e20 2024-03-06T07:28:08.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.019+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c0caec0 con 0x7fdf7c071e20 2024-03-06T07:28:08.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.020+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf70007400 con 0x7fdf7c071e20 2024-03-06T07:28:08.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.021+0000 7fdf79600640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdf5c06b4d0 0x7fdf5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.021+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf70089810 con 0x7fdf7c071e20 2024-03-06T07:28:08.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.022+0000 7fdf80e00640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdf5c06b4d0 0x7fdf5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.023+0000 7fdf80e00640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdf5c06b4d0 0x7fdf5c06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf68006fd0 tx=0x7fdf68008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.025+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf70058560 con 0x7fdf7c071e20 2024-03-06T07:28:08.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.223+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf7c0caca0 con 0x7fdf7c071e20 2024-03-06T07:28:08.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.224+0000 7fdf79600640 1 -- 172.21.15.6:0/1128351914 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf70057e60 con 0x7fdf7c071e20 2024-03-06T07:28:08.226 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.226+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdf5c06b4d0 msgr2=0x7fdf5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdf5c06b4d0 0x7fdf5c06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf68006fd0 tx=0x7fdf68008040 comp rx=0 tx=0).stop 2024-03-06T07:28:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 msgr2=0x7fdf7c0b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0b2050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c3c0 tx=0x7fdf7000c880 comp rx=0 tx=0).stop 2024-03-06T07:28:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 shutdown_connections 2024-03-06T07:28:08.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdf5c06b4d0 0x7fdf5c06d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf7c072520 0x7fdf7c149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf7c15bf20 0x7fdf7c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 --2- 172.21.15.6:0/1128351914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf7c071e20 0x7fdf7c0b2050 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 >> 172.21.15.6:0/1128351914 conn(0x7fdf7c06b3d0 msgr2=0x7fdf7c0ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 shutdown_connections 2024-03-06T07:28:08.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.227+0000 7fdf84200640 1 -- 172.21.15.6:0/1128351914 wait complete. 2024-03-06T07:28:08.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.468+0000 7fe85fe00640 1 Processor -- start 2024-03-06T07:28:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.468+0000 7fe85fe00640 1 -- start start 2024-03-06T07:28:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.469+0000 7fe85fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe86006b440 0x7fe8600cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.469+0000 7fe85fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8600b5800 con 0x7fe86015e0c0 2024-03-06T07:28:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.469+0000 7fe85fe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8600b5aa0 con 0x7fe86006afb0 2024-03-06T07:28:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.469+0000 7fe85fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86006dd30 con 0x7fe86006b440 2024-03-06T07:28:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.469+0000 7fe85ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe86015e0c0 0x7fe860072270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44408/0 (socket says 172.21.15.6:44408) 2024-03-06T07:28:08.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85ea00640 1 -- 172.21.15.6:0/726439538 learned_addr learned my addr 172.21.15.6:0/726439538 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85f400640 1 --2- 172.21.15.6:0/726439538 >> v2:172.21.15.178:3300/0 conn(0x7fe86006b440 0x7fe8600cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039896887 0 0) 0x7fe8600b5800 con 0x7fe86015e0c0 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe844003520 con 0x7fe86015e0c0 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057123421 0 0) 0x7fe844003520 con 0x7fe86015e0c0 2024-03-06T07:28:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 >> v2:172.21.15.178:3300/0 conn(0x7fe86006b440 msgr2=0x7fe8600cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 --2- 172.21.15.6:0/726439538 >> v2:172.21.15.178:3300/0 conn(0x7fe86006b440 0x7fe8600cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 >> v1:172.21.15.178:6789/0 conn(0x7fe86006afb0 legacy=0x7fe8600cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8600b5d80 con 0x7fe86015e0c0 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.470+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fe848001670 con 0x7fe86015e0c0 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 >> v1:172.21.15.6:6789/0 conn(0x7fe86015e0c0 legacy=0x7fe860072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe844003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe844004800 0x7fe844004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8440051c0 0x7fe84400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84400c030 con 0x7fe8440039b0 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84400c1e0 con 0x7fe844004800 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84400c3e0 con 0x7fe8440051c0 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85ea00640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe844003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85e000640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe844004800 0x7fe844004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85ea00640 1 -- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8440051c0 msgr2=0x7fe84400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85ea00640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8440051c0 0x7fe84400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85ea00640 1 -- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe844004800 msgr2=0x7fe844004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85ea00640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe844004800 0x7fe844004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85ea00640 1 -- 172.21.15.6:0/726439538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe844003520 con 0x7fe8440039b0 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.471+0000 7fe85f400640 1 -- 172.21.15.6:0/726439538 reap_dead start 2024-03-06T07:28:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.472+0000 7fe85ea00640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe844003dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe84800c240 tx=0x7fe844005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7dc7b2218c15b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848007d10 con 0x7fe8440039b0 2024-03-06T07:28:08.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe848030070 con 0x7fe8440039b0 2024-03-06T07:28:08.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85d600640 1 -- 172.21.15.6:0/726439538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848005d70 con 0x7fe8440039b0 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 -- 172.21.15.6:0/726439538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 msgr2=0x7fe844003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe844003dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe84800c240 tx=0x7fe844005830 comp rx=0 tx=0).stop 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 -- 172.21.15.6:0/726439538 shutdown_connections 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 --2- 172.21.15.6:0/726439538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe844003dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 -- 172.21.15.6:0/726439538 >> 172.21.15.6:0/726439538 conn(0x7fe8600d0c00 msgr2=0x7fe8600d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 -- 172.21.15.6:0/726439538 shutdown_connections 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.473+0000 7fe85fe00640 1 -- 172.21.15.6:0/726439538 wait complete. 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.475+0000 7fe85fe00640 1 Processor -- start 2024-03-06T07:28:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.475+0000 7fe85fe00640 1 -- start start 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe8601f78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8601408a0 0x7fe8601f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8601f3dd0 0x7fe8600aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86006b850 con 0x7fe8440039b0 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8600cef60 con 0x7fe8601408a0 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe860072cc0 con 0x7fe8601f3dd0 2024-03-06T07:28:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe8601f78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe8601f78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33112/0 (socket says 172.21.15.6:33112) 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.476+0000 7fe85ea00640 1 -- 172.21.15.6:0/3504841814 learned_addr learned my addr 172.21.15.6:0/3504841814 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85e000640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8601408a0 0x7fe8601f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85ea00640 1 -- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8601f3dd0 msgr2=0x7fe8600aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85ea00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8601f3dd0 0x7fe8600aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85ea00640 1 -- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8601408a0 msgr2=0x7fe8601f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85ea00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8601408a0 0x7fe8601f3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85ea00640 1 -- 172.21.15.6:0/3504841814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8601f42e0 con 0x7fe8440039b0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85ea00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe8601f78b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe84800c240 tx=0x7fe84802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848045070 con 0x7fe8440039b0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe848004bf0 con 0x7fe8440039b0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe848007d10 con 0x7fe8440039b0 2024-03-06T07:28:08.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.477+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8601f45c0 con 0x7fe8440039b0 2024-03-06T07:28:08.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.478+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe860140d20 con 0x7fe8440039b0 2024-03-06T07:28:08.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.479+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe86004f330 con 0x7fe8440039b0 2024-03-06T07:28:08.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.480+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe848007070 con 0x7fe8440039b0 2024-03-06T07:28:08.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.481+0000 7fe857400640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.481+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8480ab130 con 0x7fe8440039b0 2024-03-06T07:28:08.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.482+0000 7fe85e000640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.483+0000 7fe85e000640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe83c06b4d0 0x7fe83c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe850000c00 tx=0x7fe850009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.486+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe848079fb0 con 0x7fe8440039b0 2024-03-06T07:28:08.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.688+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8601f4790 con 0x7fe8440039b0 2024-03-06T07:28:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.689+0000 7fe857400640 1 -- 172.21.15.6:0/3504841814 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe8480798b0 con 0x7fe8440039b0 2024-03-06T07:28:08.692 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.692+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe83c06b4d0 msgr2=0x7fe83c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.692+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe83c06b4d0 0x7fe83c06d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe850000c00 tx=0x7fe850009040 comp rx=0 tx=0).stop 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.692+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 msgr2=0x7fe8601f78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.692+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe8601f78b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe84800c240 tx=0x7fe84802e070 comp rx=0 tx=0).stop 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 shutdown_connections 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe83c06b4d0 0x7fe83c06d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8601408a0 0x7fe8601f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8601f3dd0 0x7fe8600aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 --2- 172.21.15.6:0/3504841814 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8440039b0 0x7fe8601f78b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 >> 172.21.15.6:0/3504841814 conn(0x7fe8600d0c00 msgr2=0x7fe8600b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 shutdown_connections 2024-03-06T07:28:08.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.693+0000 7fe85fe00640 1 -- 172.21.15.6:0/3504841814 wait complete. 2024-03-06T07:28:08.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:08.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.939+0000 7f4327e00640 1 Processor -- start 2024-03-06T07:28:08.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.940+0000 7f4327e00640 1 -- start start 2024-03-06T07:28:08.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.941+0000 7f4327e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280d4860 0x7f43280d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.941+0000 7f4327e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43280cfa60 0x7f43280d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.941+0000 7f4327e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280ceca0 con 0x7f432815dee0 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.941+0000 7f4327e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280cf5d0 con 0x7f43280cfa60 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.941+0000 7f4327e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280cef80 con 0x7f43280d4860 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4326a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f432815dee0 0x7f43280d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44422/0 (socket says 172.21.15.6:44422) 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4327400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43280cfa60 0x7f43280d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4326a00640 1 -- 172.21.15.6:0/3879653029 learned_addr learned my addr 172.21.15.6:0/3879653029 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:08.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4326000640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280d4860 0x7f43280d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4327400640 1 -- 172.21.15.6:0/3879653029 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280d4860 msgr2=0x7f43280d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4327400640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280d4860 0x7f43280d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4327400640 1 -- 172.21.15.6:0/3879653029 >> v1:172.21.15.6:6789/0 conn(0x7f432815dee0 legacy=0x7f43280d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4327400640 1 -- 172.21.15.6:0/3879653029 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43280cf260 con 0x7f43280cfa60 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4325600640 1 -- 172.21.15.6:0/3879653029 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602239193 0 0) 0x7f43280ceca0 con 0x7f432815dee0 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.942+0000 7f4326000640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280d4860 0x7f43280d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.943+0000 7f4327400640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280cfa60 0x7f43280d0b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4320009d90 tx=0x7f432002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ff0373667b8076 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4325600640 1 -- 172.21.15.6:0/3879653029 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320039070 con 0x7f43280cfa60 2024-03-06T07:28:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4325600640 1 -- 172.21.15.6:0/3879653029 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432002cc50 con 0x7f43280cfa60 2024-03-06T07:28:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4325600640 1 -- 172.21.15.6:0/3879653029 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320034440 con 0x7f43280cfa60 2024-03-06T07:28:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4327e00640 1 -- 172.21.15.6:0/3879653029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280cfa60 msgr2=0x7f43280d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4327e00640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280cfa60 0x7f43280d0b70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4320009d90 tx=0x7f432002bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4327e00640 1 -- 172.21.15.6:0/3879653029 shutdown_connections 2024-03-06T07:28:08.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4327e00640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280cfa60 0x7f43280d0b70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4327e00640 1 --2- 172.21.15.6:0/3879653029 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280d4860 0x7f43280d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.944+0000 7f4327e00640 1 -- 172.21.15.6:0/3879653029 >> 172.21.15.6:0/3879653029 conn(0x7f43280d3cd0 msgr2=0x7f43280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:08.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.945+0000 7f4327e00640 1 -- 172.21.15.6:0/3879653029 shutdown_connections 2024-03-06T07:28:08.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.945+0000 7f4327e00640 1 -- 172.21.15.6:0/3879653029 wait complete. 2024-03-06T07:28:08.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.946+0000 7f4327e00640 1 Processor -- start 2024-03-06T07:28:08.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.946+0000 7f4327e00640 1 -- start start 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.947+0000 7f4327e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280cfa60 0x7f43281500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.947+0000 7f4327e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 0x7f43280b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.947+0000 7f4327e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432815dee0 0x7f43280b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.947+0000 7f4327e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280b3510 con 0x7f43280d4860 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.947+0000 7f4327e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432815e2c0 con 0x7f432815dee0 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.947+0000 7f4327e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280b3690 con 0x7f43280cfa60 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 0x7f43280b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 0x7f43280b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33140/0 (socket says 172.21.15.6:33140) 2024-03-06T07:28:08.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 -- 172.21.15.6:0/248993305 learned_addr learned my addr 172.21.15.6:0/248993305 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:08.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326a00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280cfa60 0x7f43281500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4327400640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432815dee0 0x7f43280b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 -- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280cfa60 msgr2=0x7f43281500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280cfa60 0x7f43281500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 -- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432815dee0 msgr2=0x7f43280b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432815dee0 0x7f43280b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4326000640 1 -- 172.21.15.6:0/248993305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43281f3b70 con 0x7f43280d4860 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.948+0000 7f4327400640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432815dee0 0x7f43280b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.949+0000 7f4326000640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 0x7f43280b2480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f431400c7f0 tx=0x7f431400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.949+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314015070 con 0x7f43280d4860 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.949+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314007a30 con 0x7f43280d4860 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.949+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314007e00 con 0x7f43280d4860 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.949+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43281f3e50 con 0x7f43280d4860 2024-03-06T07:28:08.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.949+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43281f4410 con 0x7f43280d4860 2024-03-06T07:28:08.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.951+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432804f330 con 0x7f43280d4860 2024-03-06T07:28:08.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.951+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4314018070 con 0x7f43280d4860 2024-03-06T07:28:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.952+0000 7f431ea00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b4d0 0x7f430806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:08.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.952+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f431408a130 con 0x7f43280d4860 2024-03-06T07:28:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.953+0000 7f4326a00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b4d0 0x7f430806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:08.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.954+0000 7f4326a00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b4d0 0x7f430806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4318000f60 tx=0x7f4318009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:08.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:08.957+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4314059040 con 0x7f43280d4860 2024-03-06T07:28:09.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.158+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f432808d7d0 con 0x7f43280d4860 2024-03-06T07:28:09.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.159+0000 7f431ea00640 1 -- 172.21.15.6:0/248993305 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4314058940 con 0x7f43280d4860 2024-03-06T07:28:09.161 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.161+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b4d0 msgr2=0x7f430806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.161+0000 7f4327e00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b4d0 0x7f430806d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4318000f60 tx=0x7f4318009040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.161+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 msgr2=0x7f43280b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.161+0000 7f4327e00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 0x7f43280b2480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f431400c7f0 tx=0x7f431400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 shutdown_connections 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b4d0 0x7f430806d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432815dee0 0x7f43280b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43280d4860 0x7f43280b2480 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 --2- 172.21.15.6:0/248993305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43280cfa60 0x7f43281500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 >> 172.21.15.6:0/248993305 conn(0x7f43280d3cd0 msgr2=0x7f432806c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 shutdown_connections 2024-03-06T07:28:09.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.162+0000 7f4327e00640 1 -- 172.21.15.6:0/248993305 wait complete. 2024-03-06T07:28:09.194 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:28:09.195 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:09.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:09.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.425+0000 7f1323600640 1 Processor -- start 2024-03-06T07:28:09.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.425+0000 7f1323600640 1 -- start start 2024-03-06T07:28:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.427+0000 7f1323600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f131c0af580 0x7f131c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.427+0000 7f1323600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c15cc40 con 0x7f131c0cdc20 2024-03-06T07:28:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.427+0000 7f1323600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c15cf20 con 0x7f131c0af1a0 2024-03-06T07:28:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.427+0000 7f1323600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131c15d200 con 0x7f131c0af580 2024-03-06T07:28:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.427+0000 7f1321600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f131c0cdc20 0x7f131c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44426/0 (socket says 172.21.15.6:44426) 2024-03-06T07:28:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f1321600640 1 -- 172.21.15.6:0/1803443130 learned_addr learned my addr 172.21.15.6:0/1803443130 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131be00640 1 --2- 172.21.15.6:0/1803443130 >> v2:172.21.15.178:3300/0 conn(0x7f131c0af580 0x7f131c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484739218 0 0) 0x7f131c15cc40 con 0x7f131c0cdc20 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1304003520 con 0x7f131c0cdc20 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052127695 0 0) 0x7f1304003520 con 0x7f131c0cdc20 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 >> v2:172.21.15.178:3300/0 conn(0x7f131c0af580 msgr2=0x7f131c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 --2- 172.21.15.6:0/1803443130 >> v2:172.21.15.178:3300/0 conn(0x7f131c0af580 0x7f131c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 >> v1:172.21.15.178:6789/0 conn(0x7f131c0af1a0 legacy=0x7f131c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c15d4e0 con 0x7f131c0cdc20 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f13140019f0 con 0x7f131c0cdc20 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 >> v1:172.21.15.6:6789/0 conn(0x7f131c0cdc20 legacy=0x7f131c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.428+0000 7f131b400640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1304003a70 0x7f1304003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f131b400640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304004800 0x7f1304004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f131b400640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13040052e0 0x7f130400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c15d200 con 0x7f13040052e0 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130400c1b0 con 0x7f1304004800 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130400c3b0 con 0x7f1304003a70 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1321600640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13040052e0 0x7f130400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1320c00640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1304003a70 0x7f1304003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f131be00640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304004800 0x7f1304004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1321600640 1 -- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1304003a70 msgr2=0x7f1304003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1321600640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1304003a70 0x7f1304003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1321600640 1 -- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304004800 msgr2=0x7f1304004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1321600640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304004800 0x7f1304004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.429+0000 7f1321600640 1 -- 172.21.15.6:0/1803443130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304003520 con 0x7f13040052e0 2024-03-06T07:28:09.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.431+0000 7f1321600640 1 -- 172.21.15.6:0/1803443130 reap_dead start 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.431+0000 7f1321600640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13040052e0 0x7f130400b9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1314002230 tx=0x7f1304005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8786940d0ad2824 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.431+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131400a3f0 con 0x7f13040052e0 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.431+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1314034650 con 0x7f13040052e0 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.431+0000 7f131b400640 1 -- 172.21.15.6:0/1803443130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314034a40 con 0x7f13040052e0 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 -- 172.21.15.6:0/1803443130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13040052e0 msgr2=0x7f130400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13040052e0 0x7f130400b9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1314002230 tx=0x7f1304005830 comp rx=0 tx=0).stop 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 -- 172.21.15.6:0/1803443130 shutdown_connections 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 --2- 172.21.15.6:0/1803443130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13040052e0 0x7f130400b9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 -- 172.21.15.6:0/1803443130 >> 172.21.15.6:0/1803443130 conn(0x7f131c06afb0 msgr2=0x7f131c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 -- 172.21.15.6:0/1803443130 shutdown_connections 2024-03-06T07:28:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.432+0000 7f1323600640 1 -- 172.21.15.6:0/1803443130 wait complete. 2024-03-06T07:28:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.434+0000 7f1323600640 1 Processor -- start 2024-03-06T07:28:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.434+0000 7f1323600640 1 -- start start 2024-03-06T07:28:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1323600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13040052e0 0x7f131c14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1323600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f131c1f4d60 0x7f131c1f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1323600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 0x7f131c15eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1323600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c15c990 con 0x7f131c15ea20 2024-03-06T07:28:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1323600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c15c810 con 0x7f131c1f4d60 2024-03-06T07:28:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1323600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0b4c20 con 0x7f13040052e0 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1321600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 0x7f131c15eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1321600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 0x7f131c15eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33160/0 (socket says 172.21.15.6:33160) 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.435+0000 7f1321600640 1 -- 172.21.15.6:0/1873428638 learned_addr learned my addr 172.21.15.6:0/1873428638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1320c00640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13040052e0 0x7f131c14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f131be00640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f131c1f4d60 0x7f131c1f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1321600640 1 -- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13040052e0 msgr2=0x7f131c14ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1321600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13040052e0 0x7f131c14ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1321600640 1 -- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f131c1f4d60 msgr2=0x7f131c1f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1321600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f131c1f4d60 0x7f131c1f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1321600640 1 -- 172.21.15.6:0/1873428638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c14fa70 con 0x7f131c15ea20 2024-03-06T07:28:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1321600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 0x7f131c15eed0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1314002230 tx=0x7f1314001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.436+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314043070 con 0x7f131c15ea20 2024-03-06T07:28:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.437+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1314004b60 con 0x7f131c15ea20 2024-03-06T07:28:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.437+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1314039040 con 0x7f131c15ea20 2024-03-06T07:28:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.437+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c1f58d0 con 0x7f131c15ea20 2024-03-06T07:28:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.437+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c0c4d40 con 0x7f131c15ea20 2024-03-06T07:28:09.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.438+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c069510 con 0x7f131c15ea20 2024-03-06T07:28:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.439+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1314034650 con 0x7f131c15ea20 2024-03-06T07:28:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.440+0000 7f1318c00640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12fc06b500 0x7f12fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.440+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13140ab690 con 0x7f131c15ea20 2024-03-06T07:28:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.440+0000 7f1320c00640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12fc06b500 0x7f12fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.441+0000 7f1320c00640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12fc06b500 0x7f12fc06d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1310004dc0 tx=0x7f1310009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.445+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f131407a470 con 0x7f131c15ea20 2024-03-06T07:28:09.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.654+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f131c0b3320 con 0x7f131c15ea20 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.656+0000 7f1318c00640 1 -- 172.21.15.6:0/1873428638 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f1314079d70 con 0x7f131c15ea20 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:09.659 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":23,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:09.580214+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12fc06b500 msgr2=0x7f12fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12fc06b500 0x7f12fc06d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1310004dc0 tx=0x7f1310009040 comp rx=0 tx=0).stop 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 msgr2=0x7f131c15eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 0x7f131c15eed0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1314002230 tx=0x7f1314001cf0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 shutdown_connections 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12fc06b500 0x7f12fc06d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f131c1f4d60 0x7f131c1f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f131c15ea20 0x7f131c15eed0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 --2- 172.21.15.6:0/1873428638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13040052e0 0x7f131c14ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.659+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 >> 172.21.15.6:0/1873428638 conn(0x7f131c06afb0 msgr2=0x7f131c15bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.660+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 shutdown_connections 2024-03-06T07:28:09.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.660+0000 7f1323600640 1 -- 172.21.15.6:0/1873428638 wait complete. 2024-03-06T07:28:09.683 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:09.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:09.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.904+0000 7f6f39c00640 1 Processor -- start 2024-03-06T07:28:09.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.904+0000 7f6f39c00640 1 -- start start 2024-03-06T07:28:09.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f3415f810 0x7f6f340b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f340b2080 0x7f6f340b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3415fe40 con 0x7f6f34145430 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f34160120 con 0x7f6f340b2080 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f34160400 con 0x7f6f3415f810 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f340b2080 0x7f6f340b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.906+0000 7f6f39200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6f340b2080 0x7f6f340b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33164/0 (socket says 172.21.15.6:33164) 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f39200640 1 -- 172.21.15.6:0/4103377397 learned_addr learned my addr 172.21.15.6:0/4103377397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:09.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f33400640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f3415f810 0x7f6f340b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f39200640 1 -- 172.21.15.6:0/4103377397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f3415f810 msgr2=0x7f6f340b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:09.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f39200640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f3415f810 0x7f6f340b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f39200640 1 -- 172.21.15.6:0/4103377397 >> v1:172.21.15.178:6789/0 conn(0x7f6f34145430 legacy=0x7f6f3415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f39200640 1 -- 172.21.15.6:0/4103377397 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f341606e0 con 0x7f6f340b2080 2024-03-06T07:28:09.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.907+0000 7f6f33400640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f3415f810 0x7f6f340b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:09.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.909+0000 7f6f39200640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f340b2080 0x7f6f340b2fd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c009e50 tx=0x7f6f2c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08fbce5f739c351 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.909+0000 7f6f32a00640 1 -- 172.21.15.6:0/4103377397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c02cb00 con 0x7f6f340b2080 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.909+0000 7f6f32a00640 1 -- 172.21.15.6:0/4103377397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c02cca0 con 0x7f6f340b2080 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.909+0000 7f6f32a00640 1 -- 172.21.15.6:0/4103377397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c034610 con 0x7f6f340b2080 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.909+0000 7f6f39c00640 1 -- 172.21.15.6:0/4103377397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f340b2080 msgr2=0x7f6f340b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.909+0000 7f6f39c00640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f340b2080 0x7f6f340b2fd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c009e50 tx=0x7f6f2c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.910+0000 7f6f39c00640 1 -- 172.21.15.6:0/4103377397 shutdown_connections 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.910+0000 7f6f39c00640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f340b2080 0x7f6f340b2fd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.910+0000 7f6f39c00640 1 --2- 172.21.15.6:0/4103377397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f3415f810 0x7f6f340b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.910+0000 7f6f39c00640 1 -- 172.21.15.6:0/4103377397 >> 172.21.15.6:0/4103377397 conn(0x7f6f3406b3a0 msgr2=0x7f6f3406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.910+0000 7f6f39c00640 1 -- 172.21.15.6:0/4103377397 shutdown_connections 2024-03-06T07:28:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.910+0000 7f6f39c00640 1 -- 172.21.15.6:0/4103377397 wait complete. 2024-03-06T07:28:09.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.911+0000 7f6f39c00640 1 Processor -- start 2024-03-06T07:28:09.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.911+0000 7f6f39c00640 1 -- start start 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.912+0000 7f6f39c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f340b2080 0x7f6f34150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.912+0000 7f6f39c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 0x7f6f3414bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.912+0000 7f6f39c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f3415f810 0x7f6f34152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.912+0000 7f6f39c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340cdf80 con 0x7f6f34145430 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.912+0000 7f6f39c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340cde00 con 0x7f6f3415f810 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.912+0000 7f6f39c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f340b3510 con 0x7f6f340b2080 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 0x7f6f3414bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 0x7f6f3414bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33178/0 (socket says 172.21.15.6:33178) 2024-03-06T07:28:09.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 -- 172.21.15.6:0/2369016337 learned_addr learned my addr 172.21.15.6:0/2369016337 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:09.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33e00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f340b2080 0x7f6f34150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f39200640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f3415f810 0x7f6f34152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 -- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f340b2080 msgr2=0x7f6f34150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f340b2080 0x7f6f34150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 -- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f3415f810 msgr2=0x7f6f34152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f3415f810 0x7f6f34152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 -- 172.21.15.6:0/2369016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3414b420 con 0x7f6f34145430 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f39200640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f3415f810 0x7f6f34152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.913+0000 7f6f33400640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 0x7f6f3414bfd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000d7d0 tx=0x7f6f2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.914+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f20004030 con 0x7f6f34145430 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.914+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f200041d0 con 0x7f6f34145430 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.914+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2000b6c0 con 0x7f6f34145430 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.914+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3414ff60 con 0x7f6f34145430 2024-03-06T07:28:09.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.914+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f34150520 con 0x7f6f34145430 2024-03-06T07:28:09.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.915+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f340b1d20 con 0x7f6f34145430 2024-03-06T07:28:09.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.916+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f2000b860 con 0x7f6f34145430 2024-03-06T07:28:09.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.917+0000 7f6f2be00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f1406b4d0 0x7f6f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:09.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.918+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6f2008a590 con 0x7f6f34145430 2024-03-06T07:28:09.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.918+0000 7f6f33e00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f1406b4d0 0x7f6f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:09.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.919+0000 7f6f33e00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f1406b4d0 0x7f6f1406d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6f24000f60 tx=0x7f6f24009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:09.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:09.922+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f340b1d20 con 0x7f6f34145430 2024-03-06T07:28:10.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.123+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f3414fc80 con 0x7f6f34145430 2024-03-06T07:28:10.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.124+0000 7f6f2be00640 1 -- 172.21.15.6:0/2369016337 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f3414fc80 con 0x7f6f34145430 2024-03-06T07:28:10.126 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f1406b4d0 msgr2=0x7f6f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f1406b4d0 0x7f6f1406d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6f24000f60 tx=0x7f6f24009040 comp rx=0 tx=0).stop 2024-03-06T07:28:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 msgr2=0x7f6f3414bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 0x7f6f3414bfd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6f2000d7d0 tx=0x7f6f2000dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 shutdown_connections 2024-03-06T07:28:10.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f1406b4d0 0x7f6f1406d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f3415f810 0x7f6f34152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f34145430 0x7f6f3414bfd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 --2- 172.21.15.6:0/2369016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f340b2080 0x7f6f34150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 >> 172.21.15.6:0/2369016337 conn(0x7f6f3406b3a0 msgr2=0x7f6f340b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 shutdown_connections 2024-03-06T07:28:10.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.127+0000 7f6f39c00640 1 -- 172.21.15.6:0/2369016337 wait complete. 2024-03-06T07:28:10.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.377+0000 7f6a80a00640 1 Processor -- start 2024-03-06T07:28:10.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.377+0000 7f6a80a00640 1 -- start start 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a80a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6a7c147590 0x7f6a7c071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a80a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a80a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7c0719c0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a80a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7c0b24f0 con 0x7f6a7c147590 2024-03-06T07:28:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a80a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c147970 con 0x7f6a7c06d8d0 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a7b200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6a7c0b2080 0x7f6a7c0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44430/0 (socket says 172.21.15.6:44430) 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a7b200640 1 -- 172.21.15.6:0/3715199816 learned_addr learned my addr 172.21.15.6:0/3715199816 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.379+0000 7f6a79e00640 1 --2- 172.21.15.6:0/3715199816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a7a800640 1 --2- 172.21.15.6:0/3715199816 >> v2:172.21.15.178:3300/0 conn(0x7f6a7c147590 0x7f6a7c071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79400640 1 -- 172.21.15.6:0/3715199816 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086080817 0 0) 0x7f6a7c0719c0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79400640 1 -- 172.21.15.6:0/3715199816 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a60003520 con 0x7f6a7c0b2080 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79e00640 1 -- 172.21.15.6:0/3715199816 >> v2:172.21.15.178:3300/0 conn(0x7f6a7c147590 msgr2=0x7f6a7c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79e00640 1 --2- 172.21.15.6:0/3715199816 >> v2:172.21.15.178:3300/0 conn(0x7f6a7c147590 0x7f6a7c071480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79e00640 1 -- 172.21.15.6:0/3715199816 >> v1:172.21.15.6:6789/0 conn(0x7f6a7c0b2080 legacy=0x7f6a7c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79400640 1 -- 172.21.15.6:0/3715199816 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777430082 0 0) 0x7f6a60003520 con 0x7f6a7c0b2080 2024-03-06T07:28:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.380+0000 7f6a79e00640 1 -- 172.21.15.6:0/3715199816 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c06de40 con 0x7f6a7c06d8d0 2024-03-06T07:28:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.382+0000 7f6a79e00640 1 --2- 172.21.15.6:0/3715199816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0b1b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a680097a0 tx=0x7f6a6802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dcacd7223c2c3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.382+0000 7f6a79400640 1 -- 172.21.15.6:0/3715199816 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68035020 con 0x7f6a7c06d8d0 2024-03-06T07:28:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.382+0000 7f6a79400640 1 -- 172.21.15.6:0/3715199816 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a68031810 con 0x7f6a7c06d8d0 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a79400640 1 -- 172.21.15.6:0/3715199816 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68031be0 con 0x7f6a7c06d8d0 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 -- 172.21.15.6:0/3715199816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 msgr2=0x7f6a7c0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 --2- 172.21.15.6:0/3715199816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0b1b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6a680097a0 tx=0x7f6a6802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 -- 172.21.15.6:0/3715199816 shutdown_connections 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 --2- 172.21.15.6:0/3715199816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0b1b40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 --2- 172.21.15.6:0/3715199816 >> v2:172.21.15.178:3300/0 conn(0x7f6a7c147590 0x7f6a7c071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 -- 172.21.15.6:0/3715199816 >> 172.21.15.6:0/3715199816 conn(0x7f6a7c06b3a0 msgr2=0x7f6a7c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 -- 172.21.15.6:0/3715199816 shutdown_connections 2024-03-06T07:28:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.383+0000 7f6a80a00640 1 -- 172.21.15.6:0/3715199816 wait complete. 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.385+0000 7f6a80a00640 1 Processor -- start 2024-03-06T07:28:10.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.385+0000 7f6a80a00640 1 -- start start 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a80a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a80a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 0x7f6a7c1407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a80a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a7c147590 0x7f6a7c1f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a80a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0b2e40 con 0x7f6a7c0b2080 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a80a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0b31b0 con 0x7f6a7c147590 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a80a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7c0b2b50 con 0x7f6a7c06d8d0 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a79e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 0x7f6a7c1407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a79e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 0x7f6a7c1407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33188/0 (socket says 172.21.15.6:33188) 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.386+0000 7f6a79e00640 1 -- 172.21.15.6:0/4172057338 learned_addr learned my addr 172.21.15.6:0/4172057338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a7a800640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a7b200640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a7c147590 0x7f6a7c1f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a79e00640 1 -- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 msgr2=0x7f6a7c0c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a79e00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a79e00640 1 -- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a7c147590 msgr2=0x7f6a7c1f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a79e00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a7c147590 0x7f6a7c1f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a79e00640 1 -- 172.21.15.6:0/4172057338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7c1410c0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a7b200640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a7c147590 0x7f6a7c1f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.387+0000 7f6a79e00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 0x7f6a7c1407e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6a6802bec0 tx=0x7f6a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.388+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68041070 con 0x7f6a7c0b2080 2024-03-06T07:28:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.388+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a68031e40 con 0x7f6a7c0b2080 2024-03-06T07:28:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.388+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7c1413a0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.388+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a68035040 con 0x7f6a7c0b2080 2024-03-06T07:28:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.388+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7c141960 con 0x7f6a7c0b2080 2024-03-06T07:28:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.389+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7c04f330 con 0x7f6a7c0b2080 2024-03-06T07:28:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.390+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a6804c060 con 0x7f6a7c0b2080 2024-03-06T07:28:10.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.391+0000 7f6a72a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.391+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6a680abd70 con 0x7f6a7c0b2080 2024-03-06T07:28:10.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.391+0000 7f6a7a800640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.392+0000 7f6a7a800640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009aa0 tx=0x7f6a6c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.395+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a6807aad0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.593+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a7c074ab0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.604 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:10.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.601+0000 7f6a72a00640 1 -- 172.21.15.6:0/4172057338 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a6807a3d0 con 0x7f6a7c0b2080 2024-03-06T07:28:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5c06b4d0 msgr2=0x7f6a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009aa0 tx=0x7f6a6c009190 comp rx=0 tx=0).stop 2024-03-06T07:28:10.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 msgr2=0x7f6a7c1407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 0x7f6a7c1407e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6a6802bec0 tx=0x7f6a680046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 shutdown_connections 2024-03-06T07:28:10.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5c06b4d0 0x7f6a5c06d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a7c147590 0x7f6a7c1f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a7c0b2080 0x7f6a7c1407e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 --2- 172.21.15.6:0/4172057338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a7c06d8d0 0x7f6a7c0c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.605+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 >> 172.21.15.6:0/4172057338 conn(0x7f6a7c06b3a0 msgr2=0x7f6a7c0af100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.606+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 shutdown_connections 2024-03-06T07:28:10.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.606+0000 7f6a80a00640 1 -- 172.21.15.6:0/4172057338 wait complete. 2024-03-06T07:28:10.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.837+0000 7f3656e00640 1 Processor -- start 2024-03-06T07:28:10.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.837+0000 7f3656e00640 1 -- start start 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3656e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f365015dee0 0x7f36500d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3656e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500ceca0 con 0x7f36500d0d40 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3656e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36500cef80 con 0x7f36500d4860 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3656e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36500cf5d0 con 0x7f365015dee0 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3655a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f365015dee0 0x7f36500d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3656400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f36500d0d40 0x7f36500b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44432/0 (socket says 172.21.15.6:44432) 2024-03-06T07:28:10.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.839+0000 7f3656400640 1 -- 172.21.15.6:0/1485197025 learned_addr learned my addr 172.21.15.6:0/1485197025 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.840+0000 7f3655a00640 1 -- 172.21.15.6:0/1485197025 >> v1:172.21.15.178:6789/0 conn(0x7f36500d4860 legacy=0x7f36500d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.840+0000 7f3655a00640 1 -- 172.21.15.6:0/1485197025 >> v1:172.21.15.6:6789/0 conn(0x7f36500d0d40 legacy=0x7f36500b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.840+0000 7f3655a00640 1 -- 172.21.15.6:0/1485197025 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500cf260 con 0x7f365015dee0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.841+0000 7f3655a00640 1 --2- 172.21.15.6:0/1485197025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f365015dee0 0x7f36500d18a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f36440097a0 tx=0x7f364402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a174be876a9d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.841+0000 7f364fe00640 1 -- 172.21.15.6:0/1485197025 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644035020 con 0x7f365015dee0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.841+0000 7f364fe00640 1 -- 172.21.15.6:0/1485197025 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3644031830 con 0x7f365015dee0 2024-03-06T07:28:10.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.841+0000 7f364fe00640 1 -- 172.21.15.6:0/1485197025 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644031c00 con 0x7f365015dee0 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 -- 172.21.15.6:0/1485197025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f365015dee0 msgr2=0x7f36500d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 --2- 172.21.15.6:0/1485197025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f365015dee0 0x7f36500d18a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f36440097a0 tx=0x7f364402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 -- 172.21.15.6:0/1485197025 shutdown_connections 2024-03-06T07:28:10.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 --2- 172.21.15.6:0/1485197025 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f365015dee0 0x7f36500d18a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 -- 172.21.15.6:0/1485197025 >> 172.21.15.6:0/1485197025 conn(0x7f36500d3cd0 msgr2=0x7f36500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 -- 172.21.15.6:0/1485197025 shutdown_connections 2024-03-06T07:28:10.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.842+0000 7f3656e00640 1 -- 172.21.15.6:0/1485197025 wait complete. 2024-03-06T07:28:10.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.844+0000 7f3656e00640 1 Processor -- start 2024-03-06T07:28:10.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.844+0000 7f3656e00640 1 -- start start 2024-03-06T07:28:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3656e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 0x7f36500cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3656e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36500d4860 0x7f36500b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3656e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f365015dee0 0x7f36500b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3656e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500d1de0 con 0x7f36500d0d40 2024-03-06T07:28:10.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3656e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365015e2c0 con 0x7f36500d4860 2024-03-06T07:28:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3656e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500d1f60 con 0x7f365015dee0 2024-03-06T07:28:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3655a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 0x7f36500cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3655a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 0x7f36500cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33206/0 (socket says 172.21.15.6:33206) 2024-03-06T07:28:10.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.845+0000 7f3655a00640 1 -- 172.21.15.6:0/2238057206 learned_addr learned my addr 172.21.15.6:0/2238057206 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655000640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36500d4860 0x7f36500b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655a00640 1 -- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f365015dee0 msgr2=0x7f36500b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655a00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f365015dee0 0x7f36500b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655a00640 1 -- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36500d4860 msgr2=0x7f36500b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655a00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36500d4860 0x7f36500b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655a00640 1 -- 172.21.15.6:0/2238057206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500cd1f0 con 0x7f36500d0d40 2024-03-06T07:28:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655000640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36500d4860 0x7f36500b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f3655a00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 0x7f36500cd6d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f36440098c0 tx=0x7f36440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.846+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644041070 con 0x7f36500d0d40 2024-03-06T07:28:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.847+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364403aa30 con 0x7f36500d0d40 2024-03-06T07:28:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.847+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36500a1880 con 0x7f36500d0d40 2024-03-06T07:28:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.847+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644035040 con 0x7f36500d0d40 2024-03-06T07:28:10.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.847+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36501f7a90 con 0x7f36500d0d40 2024-03-06T07:28:10.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.848+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365004f330 con 0x7f36500d0d40 2024-03-06T07:28:10.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.849+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f364404c060 con 0x7f36500d0d40 2024-03-06T07:28:10.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.850+0000 7f364d600640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f363006b4d0 0x7f363006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:10.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.850+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f36440abd20 con 0x7f36500d0d40 2024-03-06T07:28:10.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.850+0000 7f3655000640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f363006b4d0 0x7f363006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:10.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.851+0000 7f3655000640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f363006b4d0 0x7f363006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3650074550 tx=0x7f363c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:10.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:10.854+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f364407ab10 con 0x7f36500d0d40 2024-03-06T07:28:11.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.056+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f36500cd4b0 con 0x7f36500d0d40 2024-03-06T07:28:11.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.057+0000 7f364d600640 1 -- 172.21.15.6:0/2238057206 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f364407a410 con 0x7f36500d0d40 2024-03-06T07:28:11.059 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f363006b4d0 msgr2=0x7f363006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f363006b4d0 0x7f363006d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3650074550 tx=0x7f363c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 msgr2=0x7f36500cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 0x7f36500cd6d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f36440098c0 tx=0x7f36440046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 shutdown_connections 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f363006b4d0 0x7f363006d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36500d4860 0x7f36500b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f365015dee0 0x7f36500b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.059+0000 7f3656e00640 1 --2- 172.21.15.6:0/2238057206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36500d0d40 0x7f36500cd6d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.060+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 >> 172.21.15.6:0/2238057206 conn(0x7f36500d3cd0 msgr2=0x7f36500c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.060+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 shutdown_connections 2024-03-06T07:28:11.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.060+0000 7f3656e00640 1 -- 172.21.15.6:0/2238057206 wait complete. 2024-03-06T07:28:11.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:28:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.301+0000 7f9dc1800640 1 Processor -- start 2024-03-06T07:28:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.302+0000 7f9dc1800640 1 -- start start 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.303+0000 7f9dc1800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9dbc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.303+0000 7f9dc1800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9dbc0cdc20 0x7f9dbc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.303+0000 7f9dc1800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc15cc40 con 0x7f9dbc0af580 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9dc1800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dbc15cf20 con 0x7f9dbc0cdc20 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9dc1800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc15d200 con 0x7f9dbc0af1a0 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9dba000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9dbc0af580 0x7f9dbc0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44448/0 (socket says 172.21.15.6:44448) 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9dba000640 1 -- 172.21.15.6:0/1375447375 learned_addr learned my addr 172.21.15.6:0/1375447375 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9dbb400640 1 --2- 172.21.15.6:0/1375447375 >> v2:172.21.15.178:3300/0 conn(0x7f9dbc0cdc20 0x7f9dbc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9dbc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063713911 0 0) 0x7f9dbc15cc40 con 0x7f9dbc0af580 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.304+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d9c003520 con 0x7f9dbc0af580 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4272829601 0 0) 0x7f9d9c003520 con 0x7f9dbc0af580 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 msgr2=0x7f9dbc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9dbc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 >> v2:172.21.15.178:3300/0 conn(0x7f9dbc0cdc20 msgr2=0x7f9dbc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 --2- 172.21.15.6:0/1375447375 >> v2:172.21.15.178:3300/0 conn(0x7f9dbc0cdc20 0x7f9dbc0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc15d4e0 con 0x7f9dbc0af580 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9da40019f0 con 0x7f9dbc0af580 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9dbc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 >> v1:172.21.15.6:6789/0 conn(0x7f9dbc0af580 legacy=0x7f9dbc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9d9c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c003f60 0x7f9d9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d9c004d40 0x7f9d9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc15cf20 con 0x7f9d9c004d40 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc15d200 con 0x7f9d9c003f60 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c00baf0 con 0x7f9dbc0af1a0 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.305+0000 7f9dbb400640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d9c004d40 0x7f9d9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dba000640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c003f60 0x7f9d9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9d9c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbb400640 1 -- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 msgr2=0x7f9d9c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbb400640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9d9c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbb400640 1 -- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c003f60 msgr2=0x7f9d9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbb400640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c003f60 0x7f9d9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbb400640 1 -- 172.21.15.6:0/1375447375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c003520 con 0x7f9d9c004d40 2024-03-06T07:28:11.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.306+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc0af1a0 0x7f9d9c003a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.307+0000 7f9dbb400640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d9c004d40 0x7f9d9c0051c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009ac0 tx=0x7f9d9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f60604be32a4a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.307+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db4039070 con 0x7f9d9c004d40 2024-03-06T07:28:11.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.307+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db402c710 con 0x7f9d9c004d40 2024-03-06T07:28:11.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.307+0000 7f9db9600640 1 -- 172.21.15.6:0/1375447375 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db4009690 con 0x7f9d9c004d40 2024-03-06T07:28:11.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dc1800640 1 -- 172.21.15.6:0/1375447375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d9c004d40 msgr2=0x7f9d9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dc1800640 1 --2- 172.21.15.6:0/1375447375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d9c004d40 0x7f9d9c0051c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9db4009ac0 tx=0x7f9d9c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dbb400640 1 -- 172.21.15.6:0/1375447375 reap_dead start 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dc1800640 1 -- 172.21.15.6:0/1375447375 shutdown_connections 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dc1800640 1 -- 172.21.15.6:0/1375447375 >> 172.21.15.6:0/1375447375 conn(0x7f9dbc06afb0 msgr2=0x7f9dbc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dc1800640 1 -- 172.21.15.6:0/1375447375 shutdown_connections 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.308+0000 7f9dc1800640 1 -- 172.21.15.6:0/1375447375 wait complete. 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.310+0000 7f9dc1800640 1 Processor -- start 2024-03-06T07:28:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.310+0000 7f9dc1800640 1 -- start start 2024-03-06T07:28:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dc1800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c004d40 0x7f9dbc0c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dc1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 0x7f9dbc0cb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dc1800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc15ada0 0x7f9dbc15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dc1800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc15c990 con 0x7f9dbc1f2280 2024-03-06T07:28:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dc1800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc15c810 con 0x7f9d9c004d40 2024-03-06T07:28:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dc1800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc0b3310 con 0x7f9dbc15ada0 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dba000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 0x7f9dbc0cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dba000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 0x7f9dbc0cb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33230/0 (socket says 172.21.15.6:33230) 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dba000640 1 -- 172.21.15.6:0/940548546 learned_addr learned my addr 172.21.15.6:0/940548546 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.311+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c004d40 0x7f9dbc0c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dbb400640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc15ada0 0x7f9dbc15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dba000640 1 -- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc15ada0 msgr2=0x7f9dbc15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dba000640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc15ada0 0x7f9dbc15b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dba000640 1 -- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c004d40 msgr2=0x7f9dbc0c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dba000640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c004d40 0x7f9dbc0c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dba000640 1 -- 172.21.15.6:0/940548546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc0cbba0 con 0x7f9dbc1f2280 2024-03-06T07:28:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dbb400640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc15ada0 0x7f9dbc15b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dba000640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 0x7f9dbc0cb660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9da4007a90 tx=0x7f9da400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4013420 con 0x7f9dbc1f2280 2024-03-06T07:28:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da4018070 con 0x7f9dbc1f2280 2024-03-06T07:28:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc0c55c0 con 0x7f9dbc1f2280 2024-03-06T07:28:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da40137e0 con 0x7f9dbc1f2280 2024-03-06T07:28:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.312+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc1f2c10 con 0x7f9dbc1f2280 2024-03-06T07:28:11.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.314+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc069510 con 0x7f9dbc1f2280 2024-03-06T07:28:11.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.314+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9da4013980 con 0x7f9dbc1f2280 2024-03-06T07:28:11.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.316+0000 7f9db2a00640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d9806b4d0 0x7f9d9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.316+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9da408a990 con 0x7f9dbc1f2280 2024-03-06T07:28:11.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.316+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d9806b4d0 0x7f9d9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.317+0000 7f9dbaa00640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d9806b4d0 0x7f9d9806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9dac006fd0 tx=0x7f9dac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.320+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9da4059780 con 0x7f9dbc1f2280 2024-03-06T07:28:11.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.594+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f9dbc15a5a0 con 0x7f9dbc1f2280 2024-03-06T07:28:11.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.608+0000 7f9db2a00640 1 -- 172.21.15.6:0/940548546 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v63) v1 ==== 135+0+0 (secure 0 0 0) 0x7f9da4059080 con 0x7f9dbc1f2280 2024-03-06T07:28:11.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d9806b4d0 msgr2=0x7f9d9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d9806b4d0 0x7f9d9806d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9dac006fd0 tx=0x7f9dac008040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 msgr2=0x7f9dbc0cb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 0x7f9dbc0cb660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9da4007a90 tx=0x7f9da400e530 comp rx=0 tx=0).stop 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 shutdown_connections 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d9806b4d0 0x7f9d9806d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dbc15ada0 0x7f9dbc15b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dbc1f2280 0x7f9dbc0cb660 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 --2- 172.21.15.6:0/940548546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d9c004d40 0x7f9dbc0c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 >> 172.21.15.6:0/940548546 conn(0x7f9dbc06afb0 msgr2=0x7f9dbc071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 shutdown_connections 2024-03-06T07:28:11.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.611+0000 7f9dc1800640 1 -- 172.21.15.6:0/940548546 wait complete. 2024-03-06T07:28:11.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.853+0000 7f1e14a00640 1 Processor -- start 2024-03-06T07:28:11.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.853+0000 7f1e14a00640 1 -- start start 2024-03-06T07:28:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.854+0000 7f1e14a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1006afb0 0x7f1e1006b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.854+0000 7f1e14a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e100cf480 0x7f1e100cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e14a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e100b5800 con 0x7f1e1015e0c0 2024-03-06T07:28:11.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e14a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1006dd30 con 0x7f1e100cf480 2024-03-06T07:28:11.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e14a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1006b950 con 0x7f1e1006afb0 2024-03-06T07:28:11.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e100cf480 0x7f1e100cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e100cf480 0x7f1e100cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33246/0 (socket says 172.21.15.6:33246) 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 -- 172.21.15.6:0/3046438074 learned_addr learned my addr 172.21.15.6:0/3046438074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0de00640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1006afb0 0x7f1e1006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 -- 172.21.15.6:0/3046438074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1006afb0 msgr2=0x7f1e1006b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1006afb0 0x7f1e1006b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 -- 172.21.15.6:0/3046438074 >> v1:172.21.15.178:6789/0 conn(0x7f1e1015e0c0 legacy=0x7f1e10072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0f200640 1 -- 172.21.15.6:0/3046438074 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e100b5aa0 con 0x7f1e100cf480 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.855+0000 7f1e0de00640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1006afb0 0x7f1e1006b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.857+0000 7f1e0f200640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e100cf8f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009e50 tx=0x7f1e0802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69779aa230b54d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.857+0000 7f1e0d400640 1 -- 172.21.15.6:0/3046438074 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0802ca70 con 0x7f1e100cf480 2024-03-06T07:28:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.857+0000 7f1e0d400640 1 -- 172.21.15.6:0/3046438074 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0802cc10 con 0x7f1e100cf480 2024-03-06T07:28:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.857+0000 7f1e0d400640 1 -- 172.21.15.6:0/3046438074 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080343f0 con 0x7f1e100cf480 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 -- 172.21.15.6:0/3046438074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 msgr2=0x7f1e100cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e100cf8f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009e50 tx=0x7f1e0802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 -- 172.21.15.6:0/3046438074 shutdown_connections 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e100cf8f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 --2- 172.21.15.6:0/3046438074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1006afb0 0x7f1e1006b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 -- 172.21.15.6:0/3046438074 >> 172.21.15.6:0/3046438074 conn(0x7f1e100d0c00 msgr2=0x7f1e100d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 -- 172.21.15.6:0/3046438074 shutdown_connections 2024-03-06T07:28:11.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.858+0000 7f1e14a00640 1 -- 172.21.15.6:0/3046438074 wait complete. 2024-03-06T07:28:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.860+0000 7f1e14a00640 1 Processor -- start 2024-03-06T07:28:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.860+0000 7f1e14a00640 1 -- start start 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e14a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e1006afb0 0x7f1e101530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e14a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e1015a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e14a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1015e0c0 0x7f1e100b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e14a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e100cef60 con 0x7f1e100cf480 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e14a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e100cec80 con 0x7f1e1006afb0 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e14a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e10072cc0 con 0x7f1e1015e0c0 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e0de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e1015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e0de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e1015a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56030/0 (socket says 172.21.15.6:56030) 2024-03-06T07:28:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.861+0000 7f1e0de00640 1 -- 172.21.15.6:0/1875852672 learned_addr learned my addr 172.21.15.6:0/1875852672 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0e800640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e1006afb0 0x7f1e101530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0f200640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1015e0c0 0x7f1e100b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0de00640 1 -- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1015e0c0 msgr2=0x7f1e100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0de00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1015e0c0 0x7f1e100b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0de00640 1 -- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e1006afb0 msgr2=0x7f1e101530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0de00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e1006afb0 0x7f1e101530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0de00640 1 -- 172.21.15.6:0/1875852672 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1015ab40 con 0x7f1e100cf480 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0e800640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e1006afb0 0x7f1e101530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e0de00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e1015a210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000c820 tx=0x7f1e0000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00015070 con 0x7f1e100cf480 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e00007a30 con 0x7f1e100cf480 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00007e00 con 0x7f1e100cf480 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1015ae20 con 0x7f1e100cf480 2024-03-06T07:28:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.862+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1015b3e0 con 0x7f1e100cf480 2024-03-06T07:28:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.864+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1004f330 con 0x7f1e100cf480 2024-03-06T07:28:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.865+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e00018070 con 0x7f1e100cf480 2024-03-06T07:28:11.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.865+0000 7f1e06a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1dec06b4d0 0x7f1dec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.866+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1e0008a260 con 0x7f1e100cf480 2024-03-06T07:28:11.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.866+0000 7f1e0e800640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1dec06b4d0 0x7f1dec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.867+0000 7f1e0e800640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1dec06b4d0 0x7f1dec06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1df800a830 tx=0x7f1df8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:11.870+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e00058fc0 con 0x7f1e100cf480 2024-03-06T07:28:12.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.071+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1e10153600 con 0x7f1e100cf480 2024-03-06T07:28:12.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.079+0000 7f1e06a00640 1 -- 172.21.15.6:0/1875852672 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1e000588c0 con 0x7f1e100cf480 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.081+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1dec06b4d0 msgr2=0x7f1dec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.081+0000 7f1e14a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1dec06b4d0 0x7f1dec06d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1df800a830 tx=0x7f1df8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.081+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 msgr2=0x7f1e1015a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.081+0000 7f1e14a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e1015a210 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000c820 tx=0x7f1e0000cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 shutdown_connections 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1dec06b4d0 0x7f1dec06d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e1015e0c0 0x7f1e100b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e100cf480 0x7f1e1015a210 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 --2- 172.21.15.6:0/1875852672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e1006afb0 0x7f1e101530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 >> 172.21.15.6:0/1875852672 conn(0x7f1e100d0c00 msgr2=0x7f1e100d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 shutdown_connections 2024-03-06T07:28:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.082+0000 7f1e14a00640 1 -- 172.21.15.6:0/1875852672 wait complete. 2024-03-06T07:28:12.104 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:12.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:12.220 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:12.335 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:12.336 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:12.336 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:12.394 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:12.394 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:12.448 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:12.448 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:12.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.726+0000 7fb4b8c00640 1 Processor -- start 2024-03-06T07:28:12.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.726+0000 7fb4b8c00640 1 -- start start 2024-03-06T07:28:12.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.728+0000 7fb4b8c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb4b415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.728+0000 7fb4b8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b415cc40 con 0x7fb4b40af1a0 2024-03-06T07:28:12.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.728+0000 7fb4b8c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b415cf20 con 0x7fb4b40cdc20 2024-03-06T07:28:12.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.728+0000 7fb4b8c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b415d200 con 0x7fb4b40af580 2024-03-06T07:28:12.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.728+0000 7fb4b1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb4b40af1a0 0x7fb4b40b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32854/0 (socket says 172.21.15.6:32854) 2024-03-06T07:28:12.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.728+0000 7fb4b1e00640 1 -- 172.21.15.6:0/153601450 learned_addr learned my addr 172.21.15.6:0/153601450 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4abe00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb4b415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050171245 0 0) 0x7fb4b415cc40 con 0x7fb4b40af1a0 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb498003520 con 0x7fb4b40af1a0 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 125018966 0 0) 0x7fb498003520 con 0x7fb4b40af1a0 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 msgr2=0x7fb4b415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb4b415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 >> v1:172.21.15.178:6789/0 conn(0x7fb4b40cdc20 legacy=0x7fb4b40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b415d4e0 con 0x7fb4b40af1a0 2024-03-06T07:28:12.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb4a4001670 con 0x7fb4b40af1a0 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 >> v1:172.21.15.6:6789/0 conn(0x7fb4b40af1a0 legacy=0x7fb4b40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.729+0000 7fb4b1400640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb498003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1400640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb498003fd0 0x7fb498004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1400640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb498004d40 0x7fb4980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b415d200 con 0x7fb498003fd0 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49800bbb0 con 0x7fb498004d40 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49800bdb0 con 0x7fb4b40af580 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4abe00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb498003fd0 0x7fb498004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1e00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb498003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4abe00640 1 -- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 msgr2=0x7fb498003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b2800640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb498004d40 0x7fb4980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4abe00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb498003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4abe00640 1 -- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb498004d40 msgr2=0x7fb4980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4abe00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb498004d40 0x7fb4980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4abe00640 1 -- 172.21.15.6:0/153601450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb498003520 con 0x7fb498003fd0 2024-03-06T07:28:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b1e00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4b40af580 0x7fb498003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.730+0000 7fb4b2800640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb498004d40 0x7fb4980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4abe00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb498003fd0 0x7fb498004800 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009ac0 tx=0x7fb498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437f76373c579cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c039070 con 0x7fb498003fd0 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49c02c6b0 con 0x7fb498003fd0 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4b1400640 1 -- 172.21.15.6:0/153601450 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb49c009690 con 0x7fb498003fd0 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4b8c00640 1 -- 172.21.15.6:0/153601450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb498003fd0 msgr2=0x7fb498004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/153601450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb498003fd0 0x7fb498004800 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb49c009ac0 tx=0x7fb498005830 comp rx=0 tx=0).stop 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.732+0000 7fb4b2800640 1 -- 172.21.15.6:0/153601450 reap_dead start 2024-03-06T07:28:12.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.733+0000 7fb4b8c00640 1 -- 172.21.15.6:0/153601450 shutdown_connections 2024-03-06T07:28:12.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.733+0000 7fb4b8c00640 1 -- 172.21.15.6:0/153601450 >> 172.21.15.6:0/153601450 conn(0x7fb4b406afb0 msgr2=0x7fb4b406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.733+0000 7fb4b8c00640 1 -- 172.21.15.6:0/153601450 shutdown_connections 2024-03-06T07:28:12.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.733+0000 7fb4b8c00640 1 -- 172.21.15.6:0/153601450 wait complete. 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.735+0000 7fb4b8c00640 1 Processor -- start 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.735+0000 7fb4b8c00640 1 -- start start 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b8c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb498003fd0 0x7fb4b4155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b8c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb4b4151810 0x7fb4b40c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 0x7fb4b415e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b415c600 con 0x7fb4b40b1370 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b8c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40b36f0 con 0x7fb4b4151810 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b8c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b415c780 con 0x7fb498003fd0 2024-03-06T07:28:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 0x7fb4b415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 0x7fb4b415e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56050/0 (socket says 172.21.15.6:56050) 2024-03-06T07:28:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b2800640 1 -- 172.21.15.6:0/3448136421 learned_addr learned my addr 172.21.15.6:0/3448136421 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4abe00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb4b4151810 0x7fb4b40c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b2800640 1 -- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb498003fd0 msgr2=0x7fb4b4155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.736+0000 7fb4b1e00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb498003fd0 0x7fb4b4155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b2800640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb498003fd0 0x7fb4b4155660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b2800640 1 -- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb4b4151810 msgr2=0x7fb4b40c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b2800640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb4b4151810 0x7fb4b40c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b2800640 1 -- 172.21.15.6:0/3448136421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4155c40 con 0x7fb4b40b1370 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4abe00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb4b4151810 0x7fb4b40c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b2800640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 0x7fb4b415e980 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb4b406c0c0 tx=0x7fb4ac00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac006d50 con 0x7fb4b40b1370 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac003970 con 0x7fb4b40b1370 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac003d60 con 0x7fb4b40b1370 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b4155f20 con 0x7fb4b40b1370 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.737+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b406cdf0 con 0x7fb4b40b1370 2024-03-06T07:28:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.738+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b4069510 con 0x7fb4b40b1370 2024-03-06T07:28:12.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.739+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4ac01b020 con 0x7fb4b40b1370 2024-03-06T07:28:12.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.740+0000 7fb4aa000640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb49006b4d0 0x7fb49006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.740+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb4ac089d20 con 0x7fb4b40b1370 2024-03-06T07:28:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.741+0000 7fb4b1e00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb49006b4d0 0x7fb49006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.742+0000 7fb4b1e00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb49006b4d0 0x7fb49006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4001200 tx=0x7fb4a4000f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.745+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4ac058b10 con 0x7fb4b40b1370 2024-03-06T07:28:12.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.943+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4b41560f0 con 0x7fb4b40b1370 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.946+0000 7fb4aa000640 1 -- 172.21.15.6:0/3448136421 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7fb4ac058410 con 0x7fb4b40b1370 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb49006b4d0 msgr2=0x7fb49006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb49006b4d0 0x7fb49006d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4001200 tx=0x7fb4a4000f10 comp rx=0 tx=0).stop 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 msgr2=0x7fb4b415e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 0x7fb4b415e980 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb4b406c0c0 tx=0x7fb4ac00dc00 comp rx=0 tx=0).stop 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 shutdown_connections 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb49006b4d0 0x7fb49006d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb4b4151810 0x7fb4b40c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4b40b1370 0x7fb4b415e980 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 --2- 172.21.15.6:0/3448136421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb498003fd0 0x7fb4b4155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 >> 172.21.15.6:0/3448136421 conn(0x7fb4b406afb0 msgr2=0x7fb4b406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 shutdown_connections 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:12.949+0000 7fb4b8c00640 1 -- 172.21.15.6:0/3448136421 wait complete. 2024-03-06T07:28:12.971 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:12.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:13.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.201+0000 7f0754c00640 1 Processor -- start 2024-03-06T07:28:13.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.202+0000 7f0754c00640 1 -- start start 2024-03-06T07:28:13.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.203+0000 7f0754c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f075006afb0 0x7f075015f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.203+0000 7f0754c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075006dcd0 con 0x7f0750161af0 2024-03-06T07:28:13.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.203+0000 7f0754c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07500b5980 con 0x7f075006b390 2024-03-06T07:28:13.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.204+0000 7f0754c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075006fc60 con 0x7f075006afb0 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.204+0000 7f074e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0750161af0 0x7f0750161ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32864/0 (socket says 172.21.15.6:32864) 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.204+0000 7f074e800640 1 -- 172.21.15.6:0/1153807155 learned_addr learned my addr 172.21.15.6:0/1153807155 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074de00640 1 --2- 172.21.15.6:0/1153807155 >> v2:172.21.15.178:3300/0 conn(0x7f075006afb0 0x7f075015f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797032681 0 0) 0x7f075006dcd0 con 0x7f0750161af0 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0734003520 con 0x7f0750161af0 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934217128 0 0) 0x7f0734003520 con 0x7f0750161af0 2024-03-06T07:28:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 >> v2:172.21.15.178:3300/0 conn(0x7f075006afb0 msgr2=0x7f075015f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 --2- 172.21.15.6:0/1153807155 >> v2:172.21.15.178:3300/0 conn(0x7f075006afb0 0x7f075015f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 >> v1:172.21.15.178:6789/0 conn(0x7f075006b390 legacy=0x7f07500b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500b5c60 con 0x7f0750161af0 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f07480019f0 con 0x7f0750161af0 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 >> v1:172.21.15.6:6789/0 conn(0x7f0750161af0 legacy=0x7f0750161ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0734003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0734004800 0x7f0734004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07340052e0 0x7f073400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075006fc60 con 0x7f07340039b0 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073400c190 con 0x7f0734004800 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.205+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073400c390 con 0x7f07340052e0 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074de00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0734003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074de00640 1 -- 172.21.15.6:0/1153807155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07340052e0 msgr2=0x7f073400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074de00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07340052e0 0x7f073400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074de00640 1 -- 172.21.15.6:0/1153807155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0734004800 msgr2=0x7f0734004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074de00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0734004800 0x7f0734004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074de00640 1 -- 172.21.15.6:0/1153807155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0734003520 con 0x7f07340039b0 2024-03-06T07:28:13.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.206+0000 7f074e800640 1 -- 172.21.15.6:0/1153807155 reap_dead start 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.207+0000 7f074de00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0734003db0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f074000ba00 tx=0x7f0734005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a70a3314815f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.207+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740036020 con 0x7f07340039b0 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.207+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07400325f0 con 0x7f07340039b0 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.207+0000 7f074ca00640 1 -- 172.21.15.6:0/1153807155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074000b600 con 0x7f07340039b0 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 -- 172.21.15.6:0/1153807155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 msgr2=0x7f0734003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0734003db0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f074000ba00 tx=0x7f0734005830 comp rx=0 tx=0).stop 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 -- 172.21.15.6:0/1153807155 shutdown_connections 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 --2- 172.21.15.6:0/1153807155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0734003db0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 -- 172.21.15.6:0/1153807155 >> 172.21.15.6:0/1153807155 conn(0x7f07500b2ae0 msgr2=0x7f07500b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 -- 172.21.15.6:0/1153807155 shutdown_connections 2024-03-06T07:28:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.208+0000 7f0754c00640 1 -- 172.21.15.6:0/1153807155 wait complete. 2024-03-06T07:28:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.210+0000 7f0754c00640 1 Processor -- start 2024-03-06T07:28:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.210+0000 7f0754c00640 1 -- start start 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f0754c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0750146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f0754c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0750158b30 0x7f075015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f0754c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f075015ffe0 0x7f0750160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f0754c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075006f460 con 0x7f07340039b0 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f0754c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075006f7d0 con 0x7f075015ffe0 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f0754c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07500b07d0 con 0x7f0750158b30 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f074de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0750146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f074de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0750146690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56074/0 (socket says 172.21.15.6:56074) 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.211+0000 7f074de00640 1 -- 172.21.15.6:0/880919398 learned_addr learned my addr 172.21.15.6:0/880919398 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074e800640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f075015ffe0 0x7f0750160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074d400640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0750158b30 0x7f075015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074de00640 1 -- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0750158b30 msgr2=0x7f075015dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074de00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0750158b30 0x7f075015dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074de00640 1 -- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f075015ffe0 msgr2=0x7f0750160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074de00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f075015ffe0 0x7f0750160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074de00640 1 -- 172.21.15.6:0/880919398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0750146bd0 con 0x7f07340039b0 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074d400640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0750158b30 0x7f075015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.212+0000 7f074de00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0750146690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f074000bb20 tx=0x7f0740004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.213+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740042070 con 0x7f07340039b0 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.213+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074003ba00 con 0x7f07340039b0 2024-03-06T07:28:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.213+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07501590a0 con 0x7f07340039b0 2024-03-06T07:28:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.213+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740036040 con 0x7f07340039b0 2024-03-06T07:28:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.213+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075015b770 con 0x7f07340039b0 2024-03-06T07:28:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.214+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075004f330 con 0x7f07340039b0 2024-03-06T07:28:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.215+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0740039020 con 0x7f07340039b0 2024-03-06T07:28:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.216+0000 7f0746000640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f073006b500 0x7f073006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.216+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f07400ac220 con 0x7f07340039b0 2024-03-06T07:28:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.216+0000 7f074d400640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f073006b500 0x7f073006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.217+0000 7f074d400640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f073006b500 0x7f073006d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f07500b1db0 tx=0x7f073c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.220+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f074002d040 con 0x7f07340039b0 2024-03-06T07:28:13.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.420+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0750074170 con 0x7f07340039b0 2024-03-06T07:28:13.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.420+0000 7f0746000640 1 -- 172.21.15.6:0/880919398 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f074007b010 con 0x7f07340039b0 2024-03-06T07:28:13.423 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f073006b500 msgr2=0x7f073006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f073006b500 0x7f073006d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f07500b1db0 tx=0x7f073c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 msgr2=0x7f0750146690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0750146690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f074000bb20 tx=0x7f0740004730 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 shutdown_connections 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f073006b500 0x7f073006d9c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f075015ffe0 0x7f0750160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0750158b30 0x7f075015dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 --2- 172.21.15.6:0/880919398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07340039b0 0x7f0750146690 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.423+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 >> 172.21.15.6:0/880919398 conn(0x7f07500b2ae0 msgr2=0x7f075006ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.424+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 shutdown_connections 2024-03-06T07:28:13.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.424+0000 7f0754c00640 1 -- 172.21.15.6:0/880919398 wait complete. 2024-03-06T07:28:13.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:13.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.660+0000 7f02d3400640 1 Processor -- start 2024-03-06T07:28:13.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.660+0000 7f02d3400640 1 -- start start 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d3400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f02cc072520 0x7f02cc0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d3400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 0x7f02cc0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d3400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc072df0 con 0x7f02cc15bf20 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d3400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02cc0cdab0 con 0x7f02cc072520 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d3400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0b4d70 con 0x7f02cc071e20 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d0a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f02cc15bf20 0x7f02cc0acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32872/0 (socket says 172.21.15.6:32872) 2024-03-06T07:28:13.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d0a00640 1 -- 172.21.15.6:0/2784033466 learned_addr learned my addr 172.21.15.6:0/2784033466 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02cbe00640 1 --2- 172.21.15.6:0/2784033466 >> v2:172.21.15.178:3300/0 conn(0x7f02cc072520 0x7f02cc0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.662+0000 7f02d1400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 0x7f02cc0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370290466 0 0) 0x7f02cc072df0 con 0x7f02cc15bf20 2024-03-06T07:28:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4003520 con 0x7f02cc15bf20 2024-03-06T07:28:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975508478 0 0) 0x7f02b4003520 con 0x7f02cc15bf20 2024-03-06T07:28:13.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 msgr2=0x7f02cc0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 0x7f02cc0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 >> v2:172.21.15.178:3300/0 conn(0x7f02cc072520 msgr2=0x7f02cc0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 --2- 172.21.15.6:0/2784033466 >> v2:172.21.15.178:3300/0 conn(0x7f02cc072520 0x7f02cc0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0b5050 con 0x7f02cc15bf20 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f02c0001670 con 0x7f02cc15bf20 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 >> v1:172.21.15.6:6789/0 conn(0x7f02cc15bf20 legacy=0x7f02cc0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40039b0 0x7f02b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02b4004800 0x7f02b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 0x7f02b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0cdab0 con 0x7f02b40039b0 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0b4d70 con 0x7f02b4004800 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.663+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b400b9c0 con 0x7f02cc071e20 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02d0a00640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40039b0 0x7f02b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02cbe00640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02b4004800 0x7f02b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02d1400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 0x7f02b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02d0a00640 1 -- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 msgr2=0x7f02b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02d0a00640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02cc071e20 0x7f02b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02d0a00640 1 -- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02b4004800 msgr2=0x7f02b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.664+0000 7f02d0a00640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02b4004800 0x7f02b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.665+0000 7f02d0a00640 1 -- 172.21.15.6:0/2784033466 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b4003520 con 0x7f02b40039b0 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.666+0000 7f02d0a00640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40039b0 0x7f02b4003dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f02c0001050 tx=0x7f02b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99764dec6614f4d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.666+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0007ca0 con 0x7f02b40039b0 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0030070 con 0x7f02b40039b0 2024-03-06T07:28:13.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02cb400640 1 -- 172.21.15.6:0/2784033466 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0005d10 con 0x7f02b40039b0 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d3400640 1 -- 172.21.15.6:0/2784033466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40039b0 msgr2=0x7f02b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d3400640 1 --2- 172.21.15.6:0/2784033466 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02b40039b0 0x7f02b4003dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f02c0001050 tx=0x7f02b4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d1400640 1 -- 172.21.15.6:0/2784033466 reap_dead start 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d3400640 1 -- 172.21.15.6:0/2784033466 shutdown_connections 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d3400640 1 -- 172.21.15.6:0/2784033466 >> 172.21.15.6:0/2784033466 conn(0x7f02cc06b3d0 msgr2=0x7f02cc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d3400640 1 -- 172.21.15.6:0/2784033466 shutdown_connections 2024-03-06T07:28:13.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.667+0000 7f02d3400640 1 -- 172.21.15.6:0/2784033466 wait complete. 2024-03-06T07:28:13.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.669+0000 7f02d3400640 1 Processor -- start 2024-03-06T07:28:13.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.669+0000 7f02d3400640 1 -- start start 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.670+0000 7f02d3400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02b40039b0 0x7f02cc14e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.670+0000 7f02d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 0x7f02cc06e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.670+0000 7f02d3400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02cc159fd0 0x7f02cc15a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.670+0000 7f02d3400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0ad450 con 0x7f02cc06e160 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.670+0000 7f02d3400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0b4990 con 0x7f02cc159fd0 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.670+0000 7f02d3400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc09bb90 con 0x7f02b40039b0 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 0x7f02cc06e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 0x7f02cc06e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56094/0 (socket says 172.21.15.6:56094) 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 -- 172.21.15.6:0/1122343390 learned_addr learned my addr 172.21.15.6:0/1122343390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02d0a00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02b40039b0 0x7f02cc14e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 -- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02b40039b0 msgr2=0x7f02cc14e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02b40039b0 0x7f02cc14e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02d1400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02cc159fd0 0x7f02cc15a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 -- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02cc159fd0 msgr2=0x7f02cc15a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02cc159fd0 0x7f02cc15a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 -- 172.21.15.6:0/1122343390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc14ebd0 con 0x7f02cc06e160 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.671+0000 7f02cbe00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 0x7f02cc06e5e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f02b80119c0 tx=0x7f02b8011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.672+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b800fb50 con 0x7f02cc06e160 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.672+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b800fcf0 con 0x7f02cc06e160 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.672+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b800fea0 con 0x7f02cc06e160 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.672+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc06ccf0 con 0x7f02cc06e160 2024-03-06T07:28:13.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.672+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc06d2b0 con 0x7f02cc06e160 2024-03-06T07:28:13.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.673+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc04f330 con 0x7f02cc06e160 2024-03-06T07:28:13.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.674+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02b8012040 con 0x7f02cc06e160 2024-03-06T07:28:13.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.675+0000 7f02c8c00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02ac06b4d0 0x7f02ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.675+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f02b808a880 con 0x7f02cc06e160 2024-03-06T07:28:13.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.676+0000 7f02d0a00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02ac06b4d0 0x7f02ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.677+0000 7f02d0a00640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02ac06b4d0 0x7f02ac06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02c0034000 tx=0x7f02c0007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.679+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02b8019030 con 0x7f02cc06e160 2024-03-06T07:28:13.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.882+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f02cc15f720 con 0x7f02cc06e160 2024-03-06T07:28:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.883+0000 7f02c8c00640 1 -- 172.21.15.6:0/1122343390 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f02b8058f90 con 0x7f02cc06e160 2024-03-06T07:28:13.886 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02ac06b4d0 msgr2=0x7f02ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02ac06b4d0 0x7f02ac06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f02c0034000 tx=0x7f02c0007000 comp rx=0 tx=0).stop 2024-03-06T07:28:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 msgr2=0x7f02cc06e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 0x7f02cc06e5e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f02b80119c0 tx=0x7f02b8011e80 comp rx=0 tx=0).stop 2024-03-06T07:28:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 shutdown_connections 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02ac06b4d0 0x7f02ac06d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02cc159fd0 0x7f02cc15a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02cc06e160 0x7f02cc06e5e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 --2- 172.21.15.6:0/1122343390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02b40039b0 0x7f02cc14e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 >> 172.21.15.6:0/1122343390 conn(0x7f02cc06b3d0 msgr2=0x7f02cc0ccaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 shutdown_connections 2024-03-06T07:28:13.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:13.886+0000 7f02d3400640 1 -- 172.21.15.6:0/1122343390 wait complete. 2024-03-06T07:28:13.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:14.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.121+0000 7f93eb200640 1 Processor -- start 2024-03-06T07:28:14.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.121+0000 7f93eb200640 1 -- start start 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93eb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93ec145430 0x7f93ec15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93eb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec15cc90 0x7f93ec0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93eb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ec0b3fe0 con 0x7f93ec0b2080 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93eb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ec15fe40 con 0x7f93ec145430 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93eb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec160120 con 0x7f93ec15cc90 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93e9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93ec145430 0x7f93ec15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93ea800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f93ec0b2080 0x7f93ec06c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32882/0 (socket says 172.21.15.6:32882) 2024-03-06T07:28:14.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93ea800640 1 -- 172.21.15.6:0/325730986 learned_addr learned my addr 172.21.15.6:0/325730986 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:14.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.123+0000 7f93e9400640 1 --2- 172.21.15.6:0/325730986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec15cc90 0x7f93ec0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.124+0000 7f93e9e00640 1 -- 172.21.15.6:0/325730986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec15cc90 msgr2=0x7f93ec0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.124+0000 7f93e9e00640 1 --2- 172.21.15.6:0/325730986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec15cc90 0x7f93ec0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.124+0000 7f93e9e00640 1 -- 172.21.15.6:0/325730986 >> v1:172.21.15.6:6789/0 conn(0x7f93ec0b2080 legacy=0x7f93ec06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.124+0000 7f93e9e00640 1 -- 172.21.15.6:0/325730986 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec160400 con 0x7f93ec145430 2024-03-06T07:28:14.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.124+0000 7f93e8a00640 1 -- 172.21.15.6:0/325730986 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513195179 0 0) 0x7f93ec0b3fe0 con 0x7f93ec0b2080 2024-03-06T07:28:14.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.125+0000 7f93e9e00640 1 --2- 172.21.15.6:0/325730986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec145430 0x7f93ec15f810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0097a0 tx=0x7f93dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dabb24a6fe0164f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.125+0000 7f93e8a00640 1 -- 172.21.15.6:0/325730986 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc035020 con 0x7f93ec145430 2024-03-06T07:28:14.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.125+0000 7f93e8a00640 1 -- 172.21.15.6:0/325730986 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc031830 con 0x7f93ec145430 2024-03-06T07:28:14.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.125+0000 7f93e8a00640 1 -- 172.21.15.6:0/325730986 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc031c00 con 0x7f93ec145430 2024-03-06T07:28:14.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 -- 172.21.15.6:0/325730986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec145430 msgr2=0x7f93ec15f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 --2- 172.21.15.6:0/325730986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec145430 0x7f93ec15f810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0097a0 tx=0x7f93dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 -- 172.21.15.6:0/325730986 shutdown_connections 2024-03-06T07:28:14.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 --2- 172.21.15.6:0/325730986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec15cc90 0x7f93ec0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 --2- 172.21.15.6:0/325730986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec145430 0x7f93ec15f810 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 -- 172.21.15.6:0/325730986 >> 172.21.15.6:0/325730986 conn(0x7f93ec06b3a0 msgr2=0x7f93ec06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 -- 172.21.15.6:0/325730986 shutdown_connections 2024-03-06T07:28:14.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.126+0000 7f93eb200640 1 -- 172.21.15.6:0/325730986 wait complete. 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.128+0000 7f93eb200640 1 Processor -- start 2024-03-06T07:28:14.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.128+0000 7f93eb200640 1 -- start start 2024-03-06T07:28:14.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93eb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 0x7f93ec15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93eb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec145430 0x7f93ec141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93eb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93ec15cc90 0x7f93ec13e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93eb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec145810 con 0x7f93ec0b2080 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93eb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec15d0d0 con 0x7f93ec15cc90 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93eb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec145990 con 0x7f93ec145430 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 0x7f93ec15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 0x7f93ec15b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56114/0 (socket says 172.21.15.6:56114) 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 -- 172.21.15.6:0/3919131798 learned_addr learned my addr 172.21.15.6:0/3919131798 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 -- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec145430 msgr2=0x7f93ec141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec145430 0x7f93ec141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 -- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93ec15cc90 msgr2=0x7f93ec13e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93ec15cc90 0x7f93ec13e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.129+0000 7f93e9e00640 1 -- 172.21.15.6:0/3919131798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec1f27f0 con 0x7f93ec0b2080 2024-03-06T07:28:14.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.130+0000 7f93e9e00640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 0x7f93ec15b680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f93dc000c00 tx=0x7f93dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.130+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc004230 con 0x7f93ec0b2080 2024-03-06T07:28:14.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.130+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93dc0043d0 con 0x7f93ec0b2080 2024-03-06T07:28:14.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.130+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93dc03a960 con 0x7f93ec0b2080 2024-03-06T07:28:14.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.130+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ec1f1a30 con 0x7f93ec0b2080 2024-03-06T07:28:14.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.130+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ec1f1ff0 con 0x7f93ec0b2080 2024-03-06T07:28:14.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.131+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ec04f330 con 0x7f93ec0b2080 2024-03-06T07:28:14.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.133+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93dc038020 con 0x7f93ec0b2080 2024-03-06T07:28:14.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.134+0000 7f93e2000640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f93cc06b4d0 0x7f93cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.134+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f93dc0ac450 con 0x7f93ec0b2080 2024-03-06T07:28:14.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.135+0000 7f93e9400640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f93cc06b4d0 0x7f93cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.136+0000 7f93e9400640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f93cc06b4d0 0x7f93cc06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f93d80070b0 tx=0x7f93d8007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.139+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93dc041070 con 0x7f93ec0b2080 2024-03-06T07:28:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.339+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93ec06e240 con 0x7f93ec0b2080 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.340+0000 7f93e2000640 1 -- 172.21.15.6:0/3919131798 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93dc043390 con 0x7f93ec0b2080 2024-03-06T07:28:14.342 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.342+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f93cc06b4d0 msgr2=0x7f93cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.342+0000 7f93eb200640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f93cc06b4d0 0x7f93cc06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f93d80070b0 tx=0x7f93d8007040 comp rx=0 tx=0).stop 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.342+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 msgr2=0x7f93ec15b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.342+0000 7f93eb200640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 0x7f93ec15b680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f93dc000c00 tx=0x7f93dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 shutdown_connections 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f93cc06b4d0 0x7f93cc06d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93ec15cc90 0x7f93ec13e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93ec145430 0x7f93ec141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 --2- 172.21.15.6:0/3919131798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93ec0b2080 0x7f93ec15b680 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 >> 172.21.15.6:0/3919131798 conn(0x7f93ec06b3a0 msgr2=0x7f93ec140390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 shutdown_connections 2024-03-06T07:28:14.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.343+0000 7f93eb200640 1 -- 172.21.15.6:0/3919131798 wait complete. 2024-03-06T07:28:14.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:14.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.584+0000 7f13cda00640 1 Processor -- start 2024-03-06T07:28:14.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.584+0000 7f13cda00640 1 -- start start 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.586+0000 7f13cda00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f13c815e810 0x7f13c815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.586+0000 7f13cda00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c815ed10 con 0x7f13c80cd060 2024-03-06T07:28:14.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.586+0000 7f13cda00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c815eff0 con 0x7f13c81552b0 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.586+0000 7f13cda00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c815f2d0 con 0x7f13c815e810 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.586+0000 7f13c7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f13c80cd060 0x7f13c80b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32888/0 (socket says 172.21.15.6:32888) 2024-03-06T07:28:14.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.586+0000 7f13c7400640 1 -- 172.21.15.6:0/2260032993 learned_addr learned my addr 172.21.15.6:0/2260032993 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13cd000640 1 --2- 172.21.15.6:0/2260032993 >> v2:172.21.15.178:3300/0 conn(0x7f13c815e810 0x7f13c815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642983142 0 0) 0x7f13c815ed10 con 0x7f13c80cd060 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ac003520 con 0x7f13c80cd060 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980000681 0 0) 0x7f13ac003520 con 0x7f13c80cd060 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 >> v2:172.21.15.178:3300/0 conn(0x7f13c815e810 msgr2=0x7f13c815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 --2- 172.21.15.6:0/2260032993 >> v2:172.21.15.178:3300/0 conn(0x7f13c815e810 0x7f13c815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 >> v1:172.21.15.178:6789/0 conn(0x7f13c81552b0 legacy=0x7f13c80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c815f5b0 con 0x7f13c80cd060 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f13b0001690 con 0x7f13c80cd060 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.587+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 >> v1:172.21.15.6:6789/0 conn(0x7f13c80cd060 legacy=0x7f13c80b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13c6a00640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0039b0 0x7f13ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13c6a00640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13ac004800 0x7f13ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13c6a00640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13ac0051c0 0x7f13ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c815f2d0 con 0x7f13ac0051c0 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac00c060 con 0x7f13ac004800 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac00c260 con 0x7f13ac0039b0 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13cd000640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13ac0051c0 0x7f13ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13cd000640 1 -- 172.21.15.6:0/2260032993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0039b0 msgr2=0x7f13ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13cd000640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0039b0 0x7f13ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13cd000640 1 -- 172.21.15.6:0/2260032993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13ac004800 msgr2=0x7f13ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13cd000640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13ac004800 0x7f13ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.588+0000 7f13cd000640 1 -- 172.21.15.6:0/2260032993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac003520 con 0x7f13ac0051c0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.589+0000 7f13cd000640 1 -- 172.21.15.6:0/2260032993 reap_dead start 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.589+0000 7f13cd000640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13ac0051c0 0x7f13ac00b9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f13c000bd30 tx=0x7f13ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44c629294882b91 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c000b930 con 0x7f13ac0051c0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c0004220 con 0x7f13ac0051c0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13c6a00640 1 -- 172.21.15.6:0/2260032993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0003610 con 0x7f13ac0051c0 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 -- 172.21.15.6:0/2260032993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13ac0051c0 msgr2=0x7f13ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13ac0051c0 0x7f13ac00b9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f13c000bd30 tx=0x7f13ac005830 comp rx=0 tx=0).stop 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 -- 172.21.15.6:0/2260032993 shutdown_connections 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 --2- 172.21.15.6:0/2260032993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13ac0051c0 0x7f13ac00b9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 -- 172.21.15.6:0/2260032993 >> 172.21.15.6:0/2260032993 conn(0x7f13c806b3d0 msgr2=0x7f13c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 -- 172.21.15.6:0/2260032993 shutdown_connections 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.590+0000 7f13cda00640 1 -- 172.21.15.6:0/2260032993 wait complete. 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.592+0000 7f13cda00640 1 Processor -- start 2024-03-06T07:28:14.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.592+0000 7f13cda00640 1 -- start start 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.593+0000 7f13cda00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0051c0 0x7f13c81e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.593+0000 7f13cda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 0x7f13c81f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.593+0000 7f13cda00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c81536e0 0x7f13c81f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.593+0000 7f13cda00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c806dd30 con 0x7f13c81e7060 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.593+0000 7f13cda00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80b3760 con 0x7f13c81536e0 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.593+0000 7f13cda00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c806deb0 con 0x7f13ac0051c0 2024-03-06T07:28:14.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 0x7f13c81f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 0x7f13c81f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56144/0 (socket says 172.21.15.6:56144) 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 -- 172.21.15.6:0/2868951464 learned_addr learned my addr 172.21.15.6:0/2868951464 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13cd000640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c81536e0 0x7f13c81f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7e00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0051c0 0x7f13c81e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 -- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0051c0 msgr2=0x7f13c81e6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0051c0 0x7f13c81e6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 -- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c81536e0 msgr2=0x7f13c81f4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c81536e0 0x7f13c81f4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 -- 172.21.15.6:0/2868951464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c81e4880 con 0x7f13c81e7060 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13cd000640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c81536e0 0x7f13c81f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:14.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.594+0000 7f13c7400640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 0x7f13c81f3ae0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c700 tx=0x7f13b000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.595+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0018070 con 0x7f13c81e7060 2024-03-06T07:28:14.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.595+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b0007500 con 0x7f13c81e7060 2024-03-06T07:28:14.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.595+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c81e4b60 con 0x7f13c81e7060 2024-03-06T07:28:14.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.595+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0004a20 con 0x7f13c81e7060 2024-03-06T07:28:14.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.595+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c81e5120 con 0x7f13c81e7060 2024-03-06T07:28:14.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.596+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c804f330 con 0x7f13c81e7060 2024-03-06T07:28:14.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.597+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13b001b070 con 0x7f13c81e7060 2024-03-06T07:28:14.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.598+0000 7f13bfe00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a406b520 0x7f13a406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:14.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.598+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f13b0089830 con 0x7f13c81e7060 2024-03-06T07:28:14.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.599+0000 7f13c7e00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a406b520 0x7f13a406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:14.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.600+0000 7f13c7e00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a406b520 0x7f13a406d9e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13c815b680 tx=0x7f13b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:14.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.603+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13b00586a0 con 0x7f13c81e7060 2024-03-06T07:28:14.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.820+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f13c81e52f0 con 0x7f13c81e7060 2024-03-06T07:28:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.821+0000 7f13bfe00640 1 -- 172.21.15.6:0/2868951464 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f13b0057fa0 con 0x7f13c81e7060 2024-03-06T07:28:14.823 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:28:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.823+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a406b520 msgr2=0x7f13a406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a406b520 0x7f13a406d9e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13c815b680 tx=0x7f13b8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 msgr2=0x7f13c81f3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 0x7f13c81f3ae0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c700 tx=0x7f13b000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 shutdown_connections 2024-03-06T07:28:14.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a406b520 0x7f13a406d9e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c81536e0 0x7f13c81f4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c81e7060 0x7f13c81f3ae0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 --2- 172.21.15.6:0/2868951464 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13ac0051c0 0x7f13c81e6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 >> 172.21.15.6:0/2868951464 conn(0x7f13c806b3d0 msgr2=0x7f13c80b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 shutdown_connections 2024-03-06T07:28:14.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:14.824+0000 7f13cda00640 1 -- 172.21.15.6:0/2868951464 wait complete. 2024-03-06T07:28:14.844 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:28:14.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:14.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-06T07:28:14.943 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:28:14.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:14.943 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:15.014 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:15.014 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:15.071 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:15.071 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:15.127 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:15.127 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:15.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:15.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.400+0000 7fdcc8800640 1 Processor -- start 2024-03-06T07:28:15.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.400+0000 7fdcc8800640 1 -- start start 2024-03-06T07:28:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc8800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc8800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcc00cdc20 0x7fdcc00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc8800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc015cc40 con 0x7fdcc00af580 2024-03-06T07:28:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc8800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc015cf20 con 0x7fdcc00cdc20 2024-03-06T07:28:15.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc8800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc015d200 con 0x7fdcc00af1a0 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcc00cdc20 0x7fdcc00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcc00cdc20 0x7fdcc00b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56152/0 (socket says 172.21.15.6:56152) 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.402+0000 7fdcc6800640 1 -- 172.21.15.6:0/1574827079 learned_addr learned my addr 172.21.15.6:0/1574827079 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.403+0000 7fdcc5e00640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.403+0000 7fdcc6800640 1 -- 172.21.15.6:0/1574827079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 msgr2=0x7fdcc015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.403+0000 7fdcc6800640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.403+0000 7fdcc6800640 1 -- 172.21.15.6:0/1574827079 >> v1:172.21.15.178:6789/0 conn(0x7fdcc00af580 legacy=0x7fdcc00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.403+0000 7fdcc6800640 1 -- 172.21.15.6:0/1574827079 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc015d4e0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.403+0000 7fdcc5e00640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:15.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.404+0000 7fdcc6800640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00b2dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009e50 tx=0x7fdcbc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4b2aaf02924556 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.404+0000 7fdcc4a00640 1 -- 172.21.15.6:0/1574827079 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc02ca80 con 0x7fdcc00cdc20 2024-03-06T07:28:15.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.404+0000 7fdcc4a00640 1 -- 172.21.15.6:0/1574827079 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc02cc20 con 0x7fdcc00cdc20 2024-03-06T07:28:15.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc4a00640 1 -- 172.21.15.6:0/1574827079 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc0343f0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 -- 172.21.15.6:0/1574827079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 msgr2=0x7fdcc00b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00b2dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009e50 tx=0x7fdcbc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 -- 172.21.15.6:0/1574827079 shutdown_connections 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00b2dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 --2- 172.21.15.6:0/1574827079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 -- 172.21.15.6:0/1574827079 >> 172.21.15.6:0/1574827079 conn(0x7fdcc006afb0 msgr2=0x7fdcc006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 -- 172.21.15.6:0/1574827079 shutdown_connections 2024-03-06T07:28:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.405+0000 7fdcc8800640 1 -- 172.21.15.6:0/1574827079 wait complete. 2024-03-06T07:28:15.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.407+0000 7fdcc8800640 1 Processor -- start 2024-03-06T07:28:15.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.407+0000 7fdcc8800640 1 -- start start 2024-03-06T07:28:15.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.408+0000 7fdcc8800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.408+0000 7fdcc8800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcc00af580 0x7fdcc00ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.408+0000 7fdcc8800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.408+0000 7fdcc8800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b36a0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.408+0000 7fdcc8800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b3520 con 0x7fdcc00af580 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.408+0000 7fdcc8800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b3820 con 0x7fdcc00af1a0 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56160/0 (socket says 172.21.15.6:56160) 2024-03-06T07:28:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 -- 172.21.15.6:0/2496325366 learned_addr learned my addr 172.21.15.6:0/2496325366 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 -- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 msgr2=0x7fdcc014dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 -- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcc00af580 msgr2=0x7fdcc00ad3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc5400640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcc00af580 0x7fdcc00ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcc00af580 0x7fdcc00ad3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 -- 172.21.15.6:0/2496325366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc01f4b30 con 0x7fdcc00cdc20 2024-03-06T07:28:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc5400640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcc00af580 0x7fdcc00ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.409+0000 7fdcc6800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00ad920 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc036040 tx=0x7fdcbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.410+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc044020 con 0x7fdcc00cdc20 2024-03-06T07:28:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.410+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc040670 con 0x7fdcc00cdc20 2024-03-06T07:28:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.410+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc009510 con 0x7fdcc00cdc20 2024-03-06T07:28:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.410+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc014f2b0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.410+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc014e7d0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.411+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc004f3a0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.412+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcbc047020 con 0x7fdcc00cdc20 2024-03-06T07:28:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.413+0000 7fdcba000640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdca006b4d0 0x7fdca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.413+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdcbc0acec0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.414+0000 7fdcc5e00640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdca006b4d0 0x7fdca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.415+0000 7fdcc5e00640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdca006b4d0 0x7fdca006d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4006fd0 tx=0x7fdcb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.418+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcbc04e030 con 0x7fdcc00cdc20 2024-03-06T07:28:15.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.620+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc00ae0b0 con 0x7fdcc00cdc20 2024-03-06T07:28:15.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.622+0000 7fdcba000640 1 -- 172.21.15.6:0/2496325366 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7fdcbc07bc20 con 0x7fdcc00cdc20 2024-03-06T07:28:15.625 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:15.626 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.625+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdca006b4d0 msgr2=0x7fdca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.625+0000 7fdcc8800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdca006b4d0 0x7fdca006d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdcb4006fd0 tx=0x7fdcb4008040 comp rx=0 tx=0).stop 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.625+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 msgr2=0x7fdcc00ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.625+0000 7fdcc8800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00ad920 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc036040 tx=0x7fdcbc004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 shutdown_connections 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdca006b4d0 0x7fdca006d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcc00af580 0x7fdcc00ad3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcc00cdc20 0x7fdcc00ad920 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 --2- 172.21.15.6:0/2496325366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcc00af1a0 0x7fdcc014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 >> 172.21.15.6:0/2496325366 conn(0x7fdcc006afb0 msgr2=0x7fdcc00cb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 shutdown_connections 2024-03-06T07:28:15.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.626+0000 7fdcc8800640 1 -- 172.21.15.6:0/2496325366 wait complete. 2024-03-06T07:28:15.649 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:15.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:15.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.861+0000 7f1f32c00640 1 Processor -- start 2024-03-06T07:28:15.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.861+0000 7f1f32c00640 1 -- start start 2024-03-06T07:28:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.863+0000 7f1f32c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 0x7f1f2c0b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.863+0000 7f1f32c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f2c0d2050 0x7f1f2c06f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.863+0000 7f1f32c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c15cbc0 con 0x7f1f2c15c420 2024-03-06T07:28:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.863+0000 7f1f32c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c15cea0 con 0x7f1f2c0d2050 2024-03-06T07:28:15.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.863+0000 7f1f32c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c15d180 con 0x7f1f2c156420 2024-03-06T07:28:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.864+0000 7f1f2be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 0x7f1f2c0b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.864+0000 7f1f30c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f2c0d2050 0x7f1f2c06f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.864+0000 7f1f30c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f2c0d2050 0x7f1f2c06f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:43892/0 (socket says 172.21.15.6:43892) 2024-03-06T07:28:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.864+0000 7f1f30c00640 1 -- 172.21.15.6:0/323337210 learned_addr learned my addr 172.21.15.6:0/323337210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:15.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.864+0000 7f1f2be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 0x7f1f2c0b6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:38596/0 (socket says 172.21.15.6:38596) 2024-03-06T07:28:15.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.865+0000 7f1f30c00640 1 -- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 msgr2=0x7f1f2c0b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:15.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.865+0000 7f1f30c00640 1 --2- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 0x7f1f2c0b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.865+0000 7f1f30c00640 1 -- 172.21.15.6:0/323337210 >> v1:172.21.15.178:6789/0 conn(0x7f1f2c15c420 legacy=0x7f1f2c0d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.865+0000 7f1f30c00640 1 -- 172.21.15.6:0/323337210 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c06efb0 con 0x7f1f2c0d2050 2024-03-06T07:28:15.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.865+0000 7f1f2be00640 1 --2- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 0x7f1f2c0b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:15.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.866+0000 7f1f30c00640 1 --2- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c06f4d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009870 tx=0x7f1f2402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b433a9ae86df813 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f2aa00640 1 -- 172.21.15.6:0/323337210 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24034070 con 0x7f1f2c0d2050 2024-03-06T07:28:15.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f2aa00640 1 -- 172.21.15.6:0/323337210 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f24030840 con 0x7f1f2c0d2050 2024-03-06T07:28:15.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f2aa00640 1 -- 172.21.15.6:0/323337210 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24030c10 con 0x7f1f2c0d2050 2024-03-06T07:28:15.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 -- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c0d2050 msgr2=0x7f1f2c06f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 --2- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c06f4d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009870 tx=0x7f1f2402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 -- 172.21.15.6:0/323337210 shutdown_connections 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 --2- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c06f4d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 --2- 172.21.15.6:0/323337210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c156420 0x7f1f2c0b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 -- 172.21.15.6:0/323337210 >> 172.21.15.6:0/323337210 conn(0x7f1f2c0732d0 msgr2=0x7f1f2c09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 -- 172.21.15.6:0/323337210 shutdown_connections 2024-03-06T07:28:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.867+0000 7f1f32c00640 1 -- 172.21.15.6:0/323337210 wait complete. 2024-03-06T07:28:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.869+0000 7f1f32c00640 1 Processor -- start 2024-03-06T07:28:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.869+0000 7f1f32c00640 1 -- start start 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.870+0000 7f1f32c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c1f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.870+0000 7f1f32c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c156420 0x7f1f2c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.870+0000 7f1f32c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c15c420 0x7f1f2c15ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.870+0000 7f1f32c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c06fa10 con 0x7f1f2c0d2050 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.870+0000 7f1f32c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c156800 con 0x7f1f2c156420 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.870+0000 7f1f32c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c06fb90 con 0x7f1f2c15c420 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c1f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c1f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56174/0 (socket says 172.21.15.6:56174) 2024-03-06T07:28:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 -- 172.21.15.6:0/394363686 learned_addr learned my addr 172.21.15.6:0/394363686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2b400640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c156420 0x7f1f2c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f30c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c15c420 0x7f1f2c15ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 -- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c15c420 msgr2=0x7f1f2c15ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c15c420 0x7f1f2c15ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 -- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c156420 msgr2=0x7f1f2c0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c156420 0x7f1f2c0b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 -- 172.21.15.6:0/394363686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0b50d0 con 0x7f1f2c0d2050 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2b400640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c156420 0x7f1f2c0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.871+0000 7f1f2be00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c1f3180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00ebe0 tx=0x7f1f1c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.872+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c017070 con 0x7f1f2c0d2050 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.872+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c00cc30 con 0x7f1f2c0d2050 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.872+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c013470 con 0x7f1f2c0d2050 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.872+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c0b53b0 con 0x7f1f2c0d2050 2024-03-06T07:28:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.872+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c0b5970 con 0x7f1f2c0d2050 2024-03-06T07:28:15.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.873+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c06cc00 con 0x7f1f2c0d2050 2024-03-06T07:28:15.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.874+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f1c01a070 con 0x7f1f2c0d2050 2024-03-06T07:28:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.875+0000 7f1f23e00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f0c06b520 0x7f1f0c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.875+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1f1c08a5b0 con 0x7f1f2c0d2050 2024-03-06T07:28:15.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.876+0000 7f1f2b400640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f0c06b520 0x7f1f0c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.877+0000 7f1f2b400640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f0c06b520 0x7f1f0c06d9e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0b3920 tx=0x7f1f18009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:15.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:15.880+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f1c059390 con 0x7f1f2c0d2050 2024-03-06T07:28:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.080+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f2c0d1ac0 con 0x7f1f2c0d2050 2024-03-06T07:28:16.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.081+0000 7f1f23e00640 1 -- 172.21.15.6:0/394363686 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f1c058c90 con 0x7f1f2c0d2050 2024-03-06T07:28:16.084 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:16.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f0c06b520 msgr2=0x7f1f0c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f0c06b520 0x7f1f0c06d9e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0b3920 tx=0x7f1f18009040 comp rx=0 tx=0).stop 2024-03-06T07:28:16.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 msgr2=0x7f1f2c1f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c1f3180 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00ebe0 tx=0x7f1f1c00c520 comp rx=0 tx=0).stop 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 shutdown_connections 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f0c06b520 0x7f1f0c06d9e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f2c156420 0x7f1f2c0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f2c15c420 0x7f1f2c15ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 --2- 172.21.15.6:0/394363686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f2c0d2050 0x7f1f2c1f3180 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 >> 172.21.15.6:0/394363686 conn(0x7f1f2c0732d0 msgr2=0x7f1f2c0b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 shutdown_connections 2024-03-06T07:28:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.084+0000 7f1f32c00640 1 -- 172.21.15.6:0/394363686 wait complete. 2024-03-06T07:28:16.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:16.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.327+0000 7fe2f2e00640 1 Processor -- start 2024-03-06T07:28:16.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.327+0000 7fe2f2e00640 1 -- start start 2024-03-06T07:28:16.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2f2e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 0x7fe2ec0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2f2e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec072df0 con 0x7fe2ec071e20 2024-03-06T07:28:16.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2f2e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec0cdab0 con 0x7fe2ec072520 2024-03-06T07:28:16.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2f2e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0b4d70 con 0x7fe2ec15bf20 2024-03-06T07:28:16.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2f0e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2ec071e20 0x7fe2ec0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32904/0 (socket says 172.21.15.6:32904) 2024-03-06T07:28:16.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2f0e00640 1 -- 172.21.15.6:0/3927141133 learned_addr learned my addr 172.21.15.6:0/3927141133 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:16.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.329+0000 7fe2ebe00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 0x7fe2ec0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766009491 0 0) 0x7fe2ec072df0 con 0x7fe2ec071e20 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d0003520 con 0x7fe2ec071e20 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608042045 0 0) 0x7fe2d0003520 con 0x7fe2ec071e20 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 msgr2=0x7fe2ec0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 0x7fe2ec0acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 >> v1:172.21.15.178:6789/0 conn(0x7fe2ec072520 legacy=0x7fe2ec0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0b5050 con 0x7fe2ec071e20 2024-03-06T07:28:16.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fe2e40019f0 con 0x7fe2ec071e20 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 >> v1:172.21.15.6:6789/0 conn(0x7fe2ec071e20 legacy=0x7fe2ec0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 0x7fe2d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2d0003f60 0x7fe2d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2d0004d40 0x7fe2d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0b4d70 con 0x7fe2d0004d40 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d000feb0 con 0x7fe2d0003f60 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d00100b0 con 0x7fe2ec15bf20 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.330+0000 7fe2f0e00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2d0004d40 0x7fe2d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2eb400640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2d0003f60 0x7fe2d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2f0e00640 1 -- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 msgr2=0x7fe2d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2f0e00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec15bf20 0x7fe2d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2f0e00640 1 -- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2d0003f60 msgr2=0x7fe2d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2f0e00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2d0003f60 0x7fe2d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2f0e00640 1 -- 172.21.15.6:0/3927141133 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d0003520 con 0x7fe2d0004d40 2024-03-06T07:28:16.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.331+0000 7fe2eb400640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2d0003f60 0x7fe2d0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.332+0000 7fe2f0e00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2d0004d40 0x7fe2d00051c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400a860 tx=0x7fe2d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d208d17d3c7a64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.332+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400a480 con 0x7fe2d0004d40 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.332+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4034630 con 0x7fe2d0004d40 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.332+0000 7fe2eaa00640 1 -- 172.21.15.6:0/3927141133 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4034a80 con 0x7fe2d0004d40 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f2e00640 1 -- 172.21.15.6:0/3927141133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2d0004d40 msgr2=0x7fe2d00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/3927141133 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2d0004d40 0x7fe2d00051c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400a860 tx=0x7fe2d0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f0e00640 1 -- 172.21.15.6:0/3927141133 reap_dead start 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f2e00640 1 -- 172.21.15.6:0/3927141133 shutdown_connections 2024-03-06T07:28:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f2e00640 1 -- 172.21.15.6:0/3927141133 >> 172.21.15.6:0/3927141133 conn(0x7fe2ec06b3d0 msgr2=0x7fe2ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f2e00640 1 -- 172.21.15.6:0/3927141133 shutdown_connections 2024-03-06T07:28:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.333+0000 7fe2f2e00640 1 -- 172.21.15.6:0/3927141133 wait complete. 2024-03-06T07:28:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.335+0000 7fe2f2e00640 1 Processor -- start 2024-03-06T07:28:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.335+0000 7fe2f2e00640 1 -- start start 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2f2e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 0x7fe2ec09ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2f2e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec09d1f0 0x7fe2ec159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2f2e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2ec15a430 0x7fe2ec0af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2f2e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0b4b10 con 0x7fe2ec14d230 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2f2e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0b4990 con 0x7fe2ec15a430 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2f2e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0ce6c0 con 0x7fe2ec09d1f0 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 0x7fe2ec09ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 0x7fe2ec09ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56192/0 (socket says 172.21.15.6:56192) 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.336+0000 7fe2ebe00640 1 -- 172.21.15.6:0/4074560097 learned_addr learned my addr 172.21.15.6:0/4074560097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2f0e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2ec15a430 0x7fe2ec0af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2ebe00640 1 -- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec09d1f0 msgr2=0x7fe2ec159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2eb400640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec09d1f0 0x7fe2ec159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2ebe00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec09d1f0 0x7fe2ec159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2ebe00640 1 -- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2ec15a430 msgr2=0x7fe2ec0af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2ebe00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2ec15a430 0x7fe2ec0af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2ebe00640 1 -- 172.21.15.6:0/4074560097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec09d820 con 0x7fe2ec14d230 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2ebe00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 0x7fe2ec09ccb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00e820 tx=0x7fe2dc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc017070 con 0x7fe2ec14d230 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc00c940 con 0x7fe2ec14d230 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.337+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec1f01b0 con 0x7fe2ec14d230 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.338+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00cd10 con 0x7fe2ec14d230 2024-03-06T07:28:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.338+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec1f0770 con 0x7fe2ec14d230 2024-03-06T07:28:16.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.339+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec04f330 con 0x7fe2ec14d230 2024-03-06T07:28:16.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.340+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2dc01a070 con 0x7fe2ec14d230 2024-03-06T07:28:16.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.340+0000 7fe2e3e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.341+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe2dc08a110 con 0x7fe2ec14d230 2024-03-06T07:28:16.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.341+0000 7fe2eb400640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.342+0000 7fe2eb400640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec15b070 tx=0x7fe2d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.345+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2dc058e70 con 0x7fe2ec14d230 2024-03-06T07:28:16.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.544+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe2ec15a810 con 0x7fe2ec14d230 2024-03-06T07:28:16.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.545+0000 7fe2e3e00640 1 -- 172.21.15.6:0/4074560097 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe2dc058770 con 0x7fe2ec14d230 2024-03-06T07:28:16.547 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.547+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe2cc06b500 msgr2=0x7fe2cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.547+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec15b070 tx=0x7fe2d8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.547+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 msgr2=0x7fe2ec09ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.547+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 0x7fe2ec09ccb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00e820 tx=0x7fe2dc00ece0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 shutdown_connections 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe2ec15a430 0x7fe2ec0af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2ec09d1f0 0x7fe2ec159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 --2- 172.21.15.6:0/4074560097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2ec14d230 0x7fe2ec09ccb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 >> 172.21.15.6:0/4074560097 conn(0x7fe2ec06b3d0 msgr2=0x7fe2ec0acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 shutdown_connections 2024-03-06T07:28:16.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.548+0000 7fe2f2e00640 1 -- 172.21.15.6:0/4074560097 wait complete. 2024-03-06T07:28:16.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:16.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.794+0000 7f3f5ca00640 1 Processor -- start 2024-03-06T07:28:16.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.794+0000 7f3f5ca00640 1 -- start start 2024-03-06T07:28:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f5ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3f5815f810 0x7f3f580b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f5ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f580b2080 0x7f3f580b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f5ca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5815fe40 con 0x7f3f58145430 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f5ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f580b24d0 con 0x7f3f580b2080 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f5ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f58160120 con 0x7f3f5815f810 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f580b2080 0x7f3f580b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f580b2080 0x7f3f580b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56196/0 (socket says 172.21.15.6:56196) 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f57e00640 1 -- 172.21.15.6:0/1004465764 learned_addr learned my addr 172.21.15.6:0/1004465764 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.796+0000 7f3f4fe00640 1 --2- 172.21.15.6:0/1004465764 >> v2:172.21.15.178:3300/0 conn(0x7f3f5815f810 0x7f3f580b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/1004465764 >> v2:172.21.15.178:3300/0 conn(0x7f3f5815f810 msgr2=0x7f3f580b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.797+0000 7f3f57e00640 1 --2- 172.21.15.6:0/1004465764 >> v2:172.21.15.178:3300/0 conn(0x7f3f5815f810 0x7f3f580b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/1004465764 >> v1:172.21.15.178:6789/0 conn(0x7f3f58145430 legacy=0x7f3f5815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:16.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.797+0000 7f3f57e00640 1 -- 172.21.15.6:0/1004465764 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f58160400 con 0x7f3f580b2080 2024-03-06T07:28:16.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.798+0000 7f3f57e00640 1 --2- 172.21.15.6:0/1004465764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f580b2080 0x7f3f580b2fd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806d8c0 tx=0x7f3f5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1cf703e8c839af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.798+0000 7f3f56a00640 1 -- 172.21.15.6:0/1004465764 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5002ca80 con 0x7f3f580b2080 2024-03-06T07:28:16.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f56a00640 1 -- 172.21.15.6:0/1004465764 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5002cc20 con 0x7f3f580b2080 2024-03-06T07:28:16.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f56a00640 1 -- 172.21.15.6:0/1004465764 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f500343f0 con 0x7f3f580b2080 2024-03-06T07:28:16.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 -- 172.21.15.6:0/1004465764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f580b2080 msgr2=0x7f3f580b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/1004465764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f580b2080 0x7f3f580b2fd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3f5806d8c0 tx=0x7f3f5002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 -- 172.21.15.6:0/1004465764 shutdown_connections 2024-03-06T07:28:16.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/1004465764 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f580b2080 0x7f3f580b2fd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/1004465764 >> v2:172.21.15.178:3300/0 conn(0x7f3f5815f810 0x7f3f580b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 -- 172.21.15.6:0/1004465764 >> 172.21.15.6:0/1004465764 conn(0x7f3f5806b3a0 msgr2=0x7f3f5806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:16.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 -- 172.21.15.6:0/1004465764 shutdown_connections 2024-03-06T07:28:16.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.799+0000 7f3f5ca00640 1 -- 172.21.15.6:0/1004465764 wait complete. 2024-03-06T07:28:16.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.801+0000 7f3f5ca00640 1 Processor -- start 2024-03-06T07:28:16.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.801+0000 7f3f5ca00640 1 -- start start 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.802+0000 7f3f5ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f580b2080 0x7f3f581f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.802+0000 7f3f5ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 0x7f3f5813c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.802+0000 7f3f5ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5815f810 0x7f3f58143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.802+0000 7f3f5ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f58145990 con 0x7f3f58145430 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.802+0000 7f3f5ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f58145810 con 0x7f3f580b2080 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.802+0000 7f3f5ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f580cde00 con 0x7f3f5815f810 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 0x7f3f5813c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 0x7f3f5813c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56198/0 (socket says 172.21.15.6:56198) 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2092259478 learned_addr learned my addr 172.21.15.6:0/2092259478 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f57400640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f580b2080 0x7f3f581f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5815f810 0x7f3f58143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5815f810 msgr2=0x7f3f58143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5815f810 0x7f3f58143ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f580b2080 msgr2=0x7f3f581f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f580b2080 0x7f3f581f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 -- 172.21.15.6:0/2092259478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5813e0c0 con 0x7f3f58145430 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f57e00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5815f810 0x7f3f58143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f57400640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f580b2080 0x7f3f581f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.803+0000 7f3f4fe00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 0x7f3f5813c350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000d5f0 tx=0x7f3f4000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.804+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f400125f0 con 0x7f3f58145430 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.804+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f40019070 con 0x7f3f58145430 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.804+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f400129d0 con 0x7f3f58145430 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.804+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5813e390 con 0x7f3f58145430 2024-03-06T07:28:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.804+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5813ebd0 con 0x7f3f58145430 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.805+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5804f330 con 0x7f3f58145430 2024-03-06T07:28:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.806+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f40012b70 con 0x7f3f58145430 2024-03-06T07:28:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.807+0000 7f3f4f400640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3406b500 0x7f3f3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.807+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3f4008a660 con 0x7f3f58145430 2024-03-06T07:28:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.808+0000 7f3f57400640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3406b500 0x7f3f3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:16.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.809+0000 7f3f57400640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3406b500 0x7f3f3406d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f580aed10 tx=0x7f3f48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:16.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:16.811+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f5813e390 con 0x7f3f58145430 2024-03-06T07:28:17.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.014+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3f5806ed30 con 0x7f3f58145430 2024-03-06T07:28:17.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.015+0000 7f3f4f400640 1 -- 172.21.15.6:0/2092259478 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3f40059450 con 0x7f3f58145430 2024-03-06T07:28:17.017 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.017+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3406b500 msgr2=0x7f3f3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.017+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3406b500 0x7f3f3406d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3f580aed10 tx=0x7f3f48009040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.017+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 msgr2=0x7f3f5813c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.017+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 0x7f3f5813c350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000d5f0 tx=0x7f3f4000dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 shutdown_connections 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3406b500 0x7f3f3406d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5815f810 0x7f3f58143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f58145430 0x7f3f5813c350 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 --2- 172.21.15.6:0/2092259478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f580b2080 0x7f3f581f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 >> 172.21.15.6:0/2092259478 conn(0x7f3f5806b3a0 msgr2=0x7f3f580ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 shutdown_connections 2024-03-06T07:28:17.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.018+0000 7f3f5ca00640 1 -- 172.21.15.6:0/2092259478 wait complete. 2024-03-06T07:28:17.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:17.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.255+0000 7f341fc00640 1 Processor -- start 2024-03-06T07:28:17.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.255+0000 7f341fc00640 1 -- start start 2024-03-06T07:28:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.256+0000 7f341fc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f34180b3c90 0x7f34180cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.256+0000 7f341fc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f34180cf5c0 0x7f34180b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.256+0000 7f341fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f341806cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.256+0000 7f341fc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180ce160 con 0x7f34180cf5c0 2024-03-06T07:28:17.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.256+0000 7f341fc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34180b45b0 con 0x7f34180b3c90 2024-03-06T07:28:17.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.256+0000 7f341fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b4860 con 0x7f341806cb10 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f34180cf5c0 0x7f34180b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f34180cf5c0 0x7f34180b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56202/0 (socket says 172.21.15.6:56202) 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 -- 172.21.15.6:0/3107433064 learned_addr learned my addr 172.21.15.6:0/3107433064 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f341d200640 1 --2- 172.21.15.6:0/3107433064 >> v2:172.21.15.178:3300/0 conn(0x7f34180b3c90 0x7f34180cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f341dc00640 1 --2- 172.21.15.6:0/3107433064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f341806cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 -- 172.21.15.6:0/3107433064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 msgr2=0x7f341806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 --2- 172.21.15.6:0/3107433064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f341806cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 -- 172.21.15.6:0/3107433064 >> v2:172.21.15.178:3300/0 conn(0x7f34180b3c90 msgr2=0x7f34180cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 --2- 172.21.15.6:0/3107433064 >> v2:172.21.15.178:3300/0 conn(0x7f34180b3c90 0x7f34180cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.257+0000 7f3417e00640 1 -- 172.21.15.6:0/3107433064 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180b4b40 con 0x7f34180cf5c0 2024-03-06T07:28:17.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.258+0000 7f3417e00640 1 --2- 172.21.15.6:0/3107433064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180cf5c0 0x7f34180b4070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3418070800 tx=0x7f340402dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c945554ca8c9f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f3417400640 1 -- 172.21.15.6:0/3107433064 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340403b070 con 0x7f34180cf5c0 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f3417400640 1 -- 172.21.15.6:0/3107433064 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340402e950 con 0x7f34180cf5c0 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f3417400640 1 -- 172.21.15.6:0/3107433064 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340402ed20 con 0x7f34180cf5c0 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 -- 172.21.15.6:0/3107433064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180cf5c0 msgr2=0x7f34180b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 --2- 172.21.15.6:0/3107433064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180cf5c0 0x7f34180b4070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3418070800 tx=0x7f340402dba0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 -- 172.21.15.6:0/3107433064 shutdown_connections 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 --2- 172.21.15.6:0/3107433064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f341806cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 --2- 172.21.15.6:0/3107433064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180cf5c0 0x7f34180b4070 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 --2- 172.21.15.6:0/3107433064 >> v2:172.21.15.178:3300/0 conn(0x7f34180b3c90 0x7f34180cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.259+0000 7f341fc00640 1 -- 172.21.15.6:0/3107433064 >> 172.21.15.6:0/3107433064 conn(0x7f341806afb0 msgr2=0x7f341806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.260+0000 7f341fc00640 1 -- 172.21.15.6:0/3107433064 shutdown_connections 2024-03-06T07:28:17.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.260+0000 7f341fc00640 1 -- 172.21.15.6:0/3107433064 wait complete. 2024-03-06T07:28:17.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.261+0000 7f341fc00640 1 Processor -- start 2024-03-06T07:28:17.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.261+0000 7f341fc00640 1 -- start start 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.262+0000 7f341fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f34180721d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.262+0000 7f341fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 0x7f3418072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.262+0000 7f341fc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34180cf5c0 0x7f34181f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.262+0000 7f341fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341815b340 con 0x7f34180b3c90 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.262+0000 7f341fc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34180b5100 con 0x7f34180cf5c0 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.262+0000 7f341fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341806d4c0 con 0x7f341806cb10 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 0x7f3418072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 0x7f3418072710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56210/0 (socket says 172.21.15.6:56210) 2024-03-06T07:28:17.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 -- 172.21.15.6:0/502472638 learned_addr learned my addr 172.21.15.6:0/502472638 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f341dc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34180cf5c0 0x7f34181f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f341d200640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f34180721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 -- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 msgr2=0x7f34180721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f34180721d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 -- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34180cf5c0 msgr2=0x7f34181f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34180cf5c0 0x7f34181f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 -- 172.21.15.6:0/502472638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341809bc40 con 0x7f34180b3c90 2024-03-06T07:28:17.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.263+0000 7f3417e00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 0x7f3418072710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3404000c00 tx=0x7f34040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.264+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340403b070 con 0x7f34180b3c90 2024-03-06T07:28:17.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.264+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3404036d00 con 0x7f34180b3c90 2024-03-06T07:28:17.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.264+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404041650 con 0x7f34180b3c90 2024-03-06T07:28:17.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.264+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341809bf20 con 0x7f34180b3c90 2024-03-06T07:28:17.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.264+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341814a2b0 con 0x7f34180b3c90 2024-03-06T07:28:17.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.265+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3418069510 con 0x7f34180b3c90 2024-03-06T07:28:17.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.266+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f340403f030 con 0x7f34180b3c90 2024-03-06T07:28:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.267+0000 7f3414c00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f33f806b4d0 0x7f33f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.267+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f34040aba90 con 0x7f34180b3c90 2024-03-06T07:28:17.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.267+0000 7f341d200640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f33f806b4d0 0x7f33f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:17.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.269+0000 7f341d200640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f33f806b4d0 0x7f33f806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:17.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.271+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f340407a880 con 0x7f34180b3c90 2024-03-06T07:28:17.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.475+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f34180cdf80 con 0x7f34180b3c90 2024-03-06T07:28:17.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.475+0000 7f3414c00640 1 -- 172.21.15.6:0/502472638 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f34180cdf80 con 0x7f34180b3c90 2024-03-06T07:28:17.477 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f33f806b4d0 msgr2=0x7f33f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f33f806b4d0 0x7f33f806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408008040 comp rx=0 tx=0).stop 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 msgr2=0x7f3418072710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 0x7f3418072710 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3404000c00 tx=0x7f34040046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 shutdown_connections 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f33f806b4d0 0x7f33f806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f34180cf5c0 0x7f34181f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f34180b3c90 0x7f3418072710 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 --2- 172.21.15.6:0/502472638 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f341806cb10 0x7f34180721d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:17.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 >> 172.21.15.6:0/502472638 conn(0x7f341806afb0 msgr2=0x7f34180b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:17.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 shutdown_connections 2024-03-06T07:28:17.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:17.478+0000 7f341fc00640 1 -- 172.21.15.6:0/502472638 wait complete. 2024-03-06T07:28:17.497 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-06T07:28:17.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-06T07:28:17.605 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:28:17.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:17.605 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:17.660 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:17.661 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:17.713 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:17.713 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:17.764 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:17.764 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:17.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:18.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.090+0000 7f573b200640 1 Processor -- start 2024-03-06T07:28:18.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.091+0000 7f573b200640 1 -- start start 2024-03-06T07:28:18.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.092+0000 7f573b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 0x7f57340d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.092+0000 7f573b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f57340d0c80 0x7f57340b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.092+0000 7f573b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340b3710 con 0x7f573415e0b0 2024-03-06T07:28:18.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.092+0000 7f573b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57340b09e0 con 0x7f57340d0c80 2024-03-06T07:28:18.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.092+0000 7f573b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57340cfa20 con 0x7f57340d4a10 2024-03-06T07:28:18.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.093+0000 7f5739e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f573415e0b0 0x7f57340cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32912/0 (socket says 172.21.15.6:32912) 2024-03-06T07:28:18.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.093+0000 7f5739e00640 1 -- 172.21.15.6:0/1453631523 learned_addr learned my addr 172.21.15.6:0/1453631523 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:18.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.093+0000 7f573a800640 1 --2- 172.21.15.6:0/1453631523 >> v2:172.21.15.178:3300/0 conn(0x7f57340d0c80 0x7f57340b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.093+0000 7f5739400640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 0x7f57340d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.093+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903449406 0 0) 0x7f57340b3710 con 0x7f573415e0b0 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.093+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c003520 con 0x7f573415e0b0 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 817128503 0 0) 0x7f571c003520 con 0x7f573415e0b0 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 msgr2=0x7f57340d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 0x7f57340d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 >> v2:172.21.15.178:3300/0 conn(0x7f57340d0c80 msgr2=0x7f57340b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 --2- 172.21.15.6:0/1453631523 >> v2:172.21.15.178:3300/0 conn(0x7f57340d0c80 0x7f57340b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340b39f0 con 0x7f573415e0b0 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5728001670 con 0x7f573415e0b0 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 >> v1:172.21.15.6:6789/0 conn(0x7f573415e0b0 legacy=0x7f57340cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f571c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 0x7f571c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57340cfa20 con 0x7f571c004800 2024-03-06T07:28:18.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c00b9c0 con 0x7f571c0039b0 2024-03-06T07:28:18.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c00bb40 con 0x7f57340d4a10 2024-03-06T07:28:18.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5739400640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f571c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.094+0000 7f5739e00640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.095+0000 7f573a800640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 0x7f571c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.095+0000 7f5739400640 1 -- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 msgr2=0x7f571c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.095+0000 7f5739400640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57340d4a10 0x7f571c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.095+0000 7f5739400640 1 -- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f571c0039b0 msgr2=0x7f571c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.095+0000 7f5739400640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f571c0039b0 0x7f571c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.095+0000 7f5739400640 1 -- 172.21.15.6:0/1453631523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c003520 con 0x7f571c004800 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.096+0000 7f5739400640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f571c004c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5724009a90 tx=0x7f571c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba8044298c0f965 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.096+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724039070 con 0x7f571c004800 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.096+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572402c6b0 con 0x7f571c004800 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.096+0000 7f5738a00640 1 -- 172.21.15.6:0/1453631523 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724009690 con 0x7f571c004800 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573b200640 1 -- 172.21.15.6:0/1453631523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 msgr2=0x7f571c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573b200640 1 --2- 172.21.15.6:0/1453631523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f571c004c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5724009a90 tx=0x7f571c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573a800640 1 -- 172.21.15.6:0/1453631523 reap_dead start 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573b200640 1 -- 172.21.15.6:0/1453631523 shutdown_connections 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573b200640 1 -- 172.21.15.6:0/1453631523 >> 172.21.15.6:0/1453631523 conn(0x7f57340d3d30 msgr2=0x7f57340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573b200640 1 -- 172.21.15.6:0/1453631523 shutdown_connections 2024-03-06T07:28:18.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.097+0000 7f573b200640 1 -- 172.21.15.6:0/1453631523 wait complete. 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.099+0000 7f573b200640 1 Processor -- start 2024-03-06T07:28:18.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.099+0000 7f573b200640 1 -- start start 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f573b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f57340cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f573b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f573414f050 0x7f573414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f573b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f573414f490 0x7f57340b7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f573b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573415e490 con 0x7f571c004800 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f573b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57340b3fb0 con 0x7f573414f050 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f573b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57340cfc60 con 0x7f573414f490 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f5739e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f57340cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f5739e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f57340cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56226/0 (socket says 172.21.15.6:56226) 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.100+0000 7f5739e00640 1 -- 172.21.15.6:0/1622724377 learned_addr learned my addr 172.21.15.6:0/1622724377 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:18.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f573a800640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f573414f490 0x7f57340b7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739e00640 1 -- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f573414f490 msgr2=0x7f57340b7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739400640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f573414f050 0x7f573414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739e00640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f573414f490 0x7f57340b7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739e00640 1 -- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f573414f050 msgr2=0x7f573414fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739e00640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f573414f050 0x7f573414fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739e00640 1 -- 172.21.15.6:0/1622724377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57340cb340 con 0x7f571c004800 2024-03-06T07:28:18.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f5739e00640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f57340cae00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f57280018a0 tx=0x7f572800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572800c6c0 con 0x7f571c004800 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728018070 con 0x7f571c004800 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572800caa0 con 0x7f571c004800 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.101+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573415fc60 con 0x7f571c004800 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.102+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5734160220 con 0x7f571c004800 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.103+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573404f330 con 0x7f571c004800 2024-03-06T07:28:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.104+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f572801d460 con 0x7f571c004800 2024-03-06T07:28:18.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.105+0000 7f572e000640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f571806b4d0 0x7f571806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.105+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f572808a3e0 con 0x7f571c004800 2024-03-06T07:28:18.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.105+0000 7f5739400640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f571806b4d0 0x7f571806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.106+0000 7f5739400640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f571806b4d0 0x7f571806d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5724036000 tx=0x7f5724030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.109+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57280591d0 con 0x7f571c004800 2024-03-06T07:28:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.315+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57340d31a0 con 0x7f571c004800 2024-03-06T07:28:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.317+0000 7f572e000640 1 -- 172.21.15.6:0/1622724377 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f5728058ad0 con 0x7f571c004800 2024-03-06T07:28:18.320 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:18.320 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f571806b4d0 msgr2=0x7f571806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f571806b4d0 0x7f571806d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5724036000 tx=0x7f5724030000 comp rx=0 tx=0).stop 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 msgr2=0x7f57340cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f57340cae00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f57280018a0 tx=0x7f572800eb90 comp rx=0 tx=0).stop 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 shutdown_connections 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f571806b4d0 0x7f571806d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f573414f050 0x7f573414fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f573414f490 0x7f57340b7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 --2- 172.21.15.6:0/1622724377 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f571c004800 0x7f57340cae00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 >> 172.21.15.6:0/1622724377 conn(0x7f57340d3d30 msgr2=0x7f57340b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 shutdown_connections 2024-03-06T07:28:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.320+0000 7f573b200640 1 -- 172.21.15.6:0/1622724377 wait complete. 2024-03-06T07:28:18.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:18.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.571+0000 7f8d97800640 1 Processor -- start 2024-03-06T07:28:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.572+0000 7f8d97800640 1 -- start start 2024-03-06T07:28:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.573+0000 7f8d97800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8d90072520 0x7f8d900cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.573+0000 7f8d97800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d90071e20 0x7f8d900ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.573+0000 7f8d97800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d90072df0 con 0x7f8d9015bf20 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.573+0000 7f8d97800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d900cdab0 con 0x7f8d90072520 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.573+0000 7f8d97800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900b4d70 con 0x7f8d90071e20 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d94e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8d9015bf20 0x7f8d900acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32926/0 (socket says 172.21.15.6:32926) 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d94e00640 1 -- 172.21.15.6:0/3456268690 learned_addr learned my addr 172.21.15.6:0/3456268690 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8fe00640 1 --2- 172.21.15.6:0/3456268690 >> v2:172.21.15.6:3300/0 conn(0x7f8d90072520 0x7f8d900cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d95800640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d90071e20 0x7f8d900ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8f400640 1 -- 172.21.15.6:0/3456268690 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968072932 0 0) 0x7f8d90072df0 con 0x7f8d9015bf20 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8f400640 1 -- 172.21.15.6:0/3456268690 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d74003520 con 0x7f8d9015bf20 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3456268690 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d90071e20 msgr2=0x7f8d900ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8fe00640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d90071e20 0x7f8d900ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3456268690 >> v1:172.21.15.6:6789/0 conn(0x7f8d9015bf20 legacy=0x7f8d900acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3456268690 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900b5050 con 0x7f8d90072520 2024-03-06T07:28:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.574+0000 7f8d95800640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d90071e20 0x7f8d900ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.576+0000 7f8d8fe00640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d900cdfb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8d780097a0 tx=0x7f8d7802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c091b8823bae87f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.576+0000 7f8d8f400640 1 -- 172.21.15.6:0/3456268690 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78035020 con 0x7f8d90072520 2024-03-06T07:28:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.576+0000 7f8d8f400640 1 -- 172.21.15.6:0/3456268690 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d78031810 con 0x7f8d90072520 2024-03-06T07:28:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.576+0000 7f8d8f400640 1 -- 172.21.15.6:0/3456268690 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78031be0 con 0x7f8d90072520 2024-03-06T07:28:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.576+0000 7f8d97800640 1 -- 172.21.15.6:0/3456268690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 msgr2=0x7f8d900cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d900cdfb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8d780097a0 tx=0x7f8d7802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 -- 172.21.15.6:0/3456268690 shutdown_connections 2024-03-06T07:28:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d90071e20 0x7f8d900ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 --2- 172.21.15.6:0/3456268690 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d900cdfb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 -- 172.21.15.6:0/3456268690 >> 172.21.15.6:0/3456268690 conn(0x7f8d9006b3d0 msgr2=0x7f8d9006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 -- 172.21.15.6:0/3456268690 shutdown_connections 2024-03-06T07:28:18.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.577+0000 7f8d97800640 1 -- 172.21.15.6:0/3456268690 wait complete. 2024-03-06T07:28:18.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.579+0000 7f8d97800640 1 Processor -- start 2024-03-06T07:28:18.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.579+0000 7f8d97800640 1 -- start start 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d97800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d90071e20 0x7f8d9015c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d97800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d901514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d97800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d9015bf20 0x7f8d900ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d97800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900ad450 con 0x7f8d90072520 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d97800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900b4990 con 0x7f8d90071e20 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d97800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9015b780 con 0x7f8d9015bf20 2024-03-06T07:28:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d901514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d901514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56248/0 (socket says 172.21.15.6:56248) 2024-03-06T07:28:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3936851756 learned_addr learned my addr 172.21.15.6:0/3936851756 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d94e00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d90071e20 0x7f8d9015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.580+0000 7f8d95800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d9015bf20 0x7f8d900ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d9015bf20 msgr2=0x7f8d900ad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8fe00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d9015bf20 0x7f8d900ad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d90071e20 msgr2=0x7f8d9015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8fe00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d90071e20 0x7f8d9015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8fe00640 1 -- 172.21.15.6:0/3936851756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900b2370 con 0x7f8d90072520 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d95800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d9015bf20 0x7f8d900ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8fe00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d901514f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8d78034040 tx=0x7f8d780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78040070 con 0x7f8d90072520 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d78039cf0 con 0x7f8d90072520 2024-03-06T07:28:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78035040 con 0x7f8d90072520 2024-03-06T07:28:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d900b2650 con 0x7f8d90072520 2024-03-06T07:28:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.581+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9015ef30 con 0x7f8d90072520 2024-03-06T07:28:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.583+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9004f330 con 0x7f8d90072520 2024-03-06T07:28:18.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.583+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d78038020 con 0x7f8d90072520 2024-03-06T07:28:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.584+0000 7f8d8cc00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.584+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8d780abe10 con 0x7f8d90072520 2024-03-06T07:28:18.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.585+0000 7f8d94e00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.586+0000 7f8d94e00640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8d80005c00 tx=0x7f8d80009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.589+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d780af030 con 0x7f8d90072520 2024-03-06T07:28:18.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.791+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d900b2820 con 0x7f8d90072520 2024-03-06T07:28:18.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.792+0000 7f8d8cc00640 1 -- 172.21.15.6:0/3936851756 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d7807ac00 con 0x7f8d90072520 2024-03-06T07:28:18.794 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d6c06b4d0 msgr2=0x7f8d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8d80005c00 tx=0x7f8d80009040 comp rx=0 tx=0).stop 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 msgr2=0x7f8d901514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d901514f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8d78034040 tx=0x7f8d780046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 shutdown_connections 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d6c06b4d0 0x7f8d6c06d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d9015bf20 0x7f8d900ad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d90072520 0x7f8d901514f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 --2- 172.21.15.6:0/3936851756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d90071e20 0x7f8d9015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 >> 172.21.15.6:0/3936851756 conn(0x7f8d9006b3d0 msgr2=0x7f8d901554d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.794+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 shutdown_connections 2024-03-06T07:28:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:18.795+0000 7f8d97800640 1 -- 172.21.15.6:0/3936851756 wait complete. 2024-03-06T07:28:18.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.047+0000 7f2c73400640 1 Processor -- start 2024-03-06T07:28:19.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.047+0000 7f2c73400640 1 -- start start 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.048+0000 7f2c73400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c6c0d07d0 0x7f2c6c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.048+0000 7f2c73400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d4930 0x7f2c6c15dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.048+0000 7f2c73400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2c6c0d0bb0 0x7f2c6c15e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.048+0000 7f2c73400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c0b3770 con 0x7f2c6c0d07d0 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.048+0000 7f2c73400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c0cf780 con 0x7f2c6c0d0bb0 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.048+0000 7f2c73400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0cfa60 con 0x7f2c6c0d4930 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c6c0d07d0 0x7f2c6c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c6c0d07d0 0x7f2c6c0b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56258/0 (socket says 172.21.15.6:56258) 2024-03-06T07:28:19.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 -- 172.21.15.6:0/3157685960 learned_addr learned my addr 172.21.15.6:0/3157685960 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c71400640 1 --2- 172.21.15.6:0/3157685960 >> v2:172.21.15.178:3300/0 conn(0x7f2c6c0d0bb0 0x7f2c6c15e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c63e00640 1 --2- 172.21.15.6:0/3157685960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d4930 0x7f2c6c15dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 -- 172.21.15.6:0/3157685960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d4930 msgr2=0x7f2c6c15dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 --2- 172.21.15.6:0/3157685960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d4930 0x7f2c6c15dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 -- 172.21.15.6:0/3157685960 >> v2:172.21.15.178:3300/0 conn(0x7f2c6c0d0bb0 msgr2=0x7f2c6c15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 --2- 172.21.15.6:0/3157685960 >> v2:172.21.15.178:3300/0 conn(0x7f2c6c0d0bb0 0x7f2c6c15e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.049+0000 7f2c70a00640 1 -- 172.21.15.6:0/3157685960 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c0cfd40 con 0x7f2c6c0d07d0 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.051+0000 7f2c70a00640 1 --2- 172.21.15.6:0/3157685960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b4f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c08d2b0 tx=0x7f2c5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd846ca78bfdd50 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.051+0000 7f2c6be00640 1 -- 172.21.15.6:0/3157685960 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54034070 con 0x7f2c6c0d07d0 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.051+0000 7f2c6be00640 1 -- 172.21.15.6:0/3157685960 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c54030860 con 0x7f2c6c0d07d0 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.051+0000 7f2c6be00640 1 -- 172.21.15.6:0/3157685960 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54030c30 con 0x7f2c6c0d07d0 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.051+0000 7f2c73400640 1 -- 172.21.15.6:0/3157685960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d07d0 msgr2=0x7f2c6c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.051+0000 7f2c73400640 1 --2- 172.21.15.6:0/3157685960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b4f80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c08d2b0 tx=0x7f2c5402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 -- 172.21.15.6:0/3157685960 shutdown_connections 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 --2- 172.21.15.6:0/3157685960 >> v2:172.21.15.178:3300/0 conn(0x7f2c6c0d0bb0 0x7f2c6c15e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 --2- 172.21.15.6:0/3157685960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d4930 0x7f2c6c15dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 --2- 172.21.15.6:0/3157685960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b4f80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 -- 172.21.15.6:0/3157685960 >> 172.21.15.6:0/3157685960 conn(0x7f2c6c0d3e10 msgr2=0x7f2c6c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 -- 172.21.15.6:0/3157685960 shutdown_connections 2024-03-06T07:28:19.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.052+0000 7f2c73400640 1 -- 172.21.15.6:0/3157685960 wait complete. 2024-03-06T07:28:19.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.053+0000 7f2c73400640 1 Processor -- start 2024-03-06T07:28:19.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.054+0000 7f2c73400640 1 -- start start 2024-03-06T07:28:19.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.054+0000 7f2c73400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.054+0000 7f2c73400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c6c0d0bb0 0x7f2c6c1f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c73400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 0x7f2c6c1f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c73400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0d1d70 con 0x7f2c6c0d4930 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c73400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0d1bf0 con 0x7f2c6c0d0bb0 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c73400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0d1ef0 con 0x7f2c6c0d07d0 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 0x7f2c6c1f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 0x7f2c6c1f3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56270/0 (socket says 172.21.15.6:56270) 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 -- 172.21.15.6:0/2521385292 learned_addr learned my addr 172.21.15.6:0/2521385292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c70a00640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c63e00640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c6c0d0bb0 0x7f2c6c1f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 -- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d07d0 msgr2=0x7f2c6c0b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 -- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c6c0d0bb0 msgr2=0x7f2c6c1f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c6c0d0bb0 0x7f2c6c1f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c71400640 1 -- 172.21.15.6:0/2521385292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c0b7370 con 0x7f2c6c0d4930 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c63e00640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c6c0d0bb0 0x7f2c6c1f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.055+0000 7f2c70a00640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.056+0000 7f2c71400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 0x7f2c6c1f3fd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2c64007e60 tx=0x7f2c6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.056+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64014620 con 0x7f2c6c0d4930 2024-03-06T07:28:19.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.056+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c64019070 con 0x7f2c6c0d4930 2024-03-06T07:28:19.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.056+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64014a20 con 0x7f2c6c0d4930 2024-03-06T07:28:19.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.056+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6c0b7650 con 0x7f2c6c0d4930 2024-03-06T07:28:19.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.056+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c0b7ef0 con 0x7f2c6c0d4930 2024-03-06T07:28:19.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.057+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c04f330 con 0x7f2c6c0d4930 2024-03-06T07:28:19.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.058+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c64014bc0 con 0x7f2c6c0d4930 2024-03-06T07:28:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.059+0000 7f2c69600640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c4806b520 0x7f2c4806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.059+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2c6408aaa0 con 0x7f2c6c0d4930 2024-03-06T07:28:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.060+0000 7f2c70a00640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c4806b520 0x7f2c4806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.061+0000 7f2c70a00640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c4806b520 0x7f2c4806d9e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c54036000 tx=0x7f2c5403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.065+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c64059910 con 0x7f2c6c0d4930 2024-03-06T07:28:19.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.268+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c6c0b2c90 con 0x7f2c6c0d4930 2024-03-06T07:28:19.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.269+0000 7f2c69600640 1 -- 172.21.15.6:0/2521385292 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c64059210 con 0x7f2c6c0d4930 2024-03-06T07:28:19.271 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:19.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c4806b520 msgr2=0x7f2c4806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c4806b520 0x7f2c4806d9e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2c54036000 tx=0x7f2c5403b040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 msgr2=0x7f2c6c1f3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 0x7f2c6c1f3fd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2c64007e60 tx=0x7f2c6400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 shutdown_connections 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c4806b520 0x7f2c4806d9e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c6c0d0bb0 0x7f2c6c1f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c6c0d4930 0x7f2c6c1f3fd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 --2- 172.21.15.6:0/2521385292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c6c0d07d0 0x7f2c6c0b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 >> 172.21.15.6:0/2521385292 conn(0x7f2c6c0d3e10 msgr2=0x7f2c6c161270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 shutdown_connections 2024-03-06T07:28:19.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.272+0000 7f2c73400640 1 -- 172.21.15.6:0/2521385292 wait complete. 2024-03-06T07:28:19.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:19.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.506+0000 7f650a600640 1 Processor -- start 2024-03-06T07:28:19.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.507+0000 7f650a600640 1 -- start start 2024-03-06T07:28:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.508+0000 7f650a600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6504147590 0x7f6504071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.508+0000 7f650a600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65040b2250 0x7f65040b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.508+0000 7f650a600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65040719c0 con 0x7f650406d8d0 2024-03-06T07:28:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.508+0000 7f650a600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6504147970 con 0x7f6504147590 2024-03-06T07:28:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.508+0000 7f650a600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65040b31b0 con 0x7f65040b2250 2024-03-06T07:28:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.509+0000 7f6509200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6504147590 0x7f6504071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.509+0000 7f6509200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6504147590 0x7f6504071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:44046/0 (socket says 172.21.15.6:44046) 2024-03-06T07:28:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.509+0000 7f6509200640 1 -- 172.21.15.6:0/3929261518 learned_addr learned my addr 172.21.15.6:0/3929261518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:19.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.509+0000 7f6509c00640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65040b2250 0x7f65040b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6503400640 1 -- 172.21.15.6:0/3929261518 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932706870 0 0) 0x7f65040719c0 con 0x7f650406d8d0 2024-03-06T07:28:19.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6503400640 1 -- 172.21.15.6:0/3929261518 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ec003520 con 0x7f650406d8d0 2024-03-06T07:28:19.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6509200640 1 -- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65040b2250 msgr2=0x7f65040b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6509200640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65040b2250 0x7f65040b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6509200640 1 -- 172.21.15.6:0/3929261518 >> v1:172.21.15.178:6789/0 conn(0x7f650406d8d0 legacy=0x7f65040b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6509200640 1 -- 172.21.15.6:0/3929261518 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650406de40 con 0x7f6504147590 2024-03-06T07:28:19.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.510+0000 7f6509c00640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65040b2250 0x7f65040b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:19.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.512+0000 7f6509200640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6504147590 0x7f6504071480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009870 tx=0x7f64f802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7090471b5e195c28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f6503400640 1 -- 172.21.15.6:0/3929261518 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8034070 con 0x7f6504147590 2024-03-06T07:28:19.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f6503400640 1 -- 172.21.15.6:0/3929261518 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f8030840 con 0x7f6504147590 2024-03-06T07:28:19.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f6503400640 1 -- 172.21.15.6:0/3929261518 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8030bf0 con 0x7f6504147590 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 -- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6504147590 msgr2=0x7f6504071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6504147590 0x7f6504071480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009870 tx=0x7f64f802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 -- 172.21.15.6:0/3929261518 shutdown_connections 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f65040b2250 0x7f65040b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 --2- 172.21.15.6:0/3929261518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6504147590 0x7f6504071480 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 -- 172.21.15.6:0/3929261518 >> 172.21.15.6:0/3929261518 conn(0x7f650406b3a0 msgr2=0x7f650406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 -- 172.21.15.6:0/3929261518 shutdown_connections 2024-03-06T07:28:19.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.513+0000 7f650a600640 1 -- 172.21.15.6:0/3929261518 wait complete. 2024-03-06T07:28:19.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.515+0000 7f650a600640 1 Processor -- start 2024-03-06T07:28:19.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.515+0000 7f650a600640 1 -- start start 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.516+0000 7f650a600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f650406d8d0 0x7f6504136c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.516+0000 7f650a600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65040b2250 0x7f650415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.516+0000 7f650a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 0x7f650415fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.516+0000 7f650a600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65040b2b50 con 0x7f6504147590 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.516+0000 7f650a600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65040b2e40 con 0x7f65040b2250 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.516+0000 7f650a600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65040b3b80 con 0x7f650406d8d0 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 0x7f650415fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 0x7f650415fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56276/0 (socket says 172.21.15.6:56276) 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 -- 172.21.15.6:0/2478983865 learned_addr learned my addr 172.21.15.6:0/2478983865 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6503e00640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65040b2250 0x7f650415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509200640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f650406d8d0 0x7f6504136c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 -- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f650406d8d0 msgr2=0x7f6504136c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f650406d8d0 0x7f6504136c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 -- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65040b2250 msgr2=0x7f650415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65040b2250 0x7f650415d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.517+0000 7f6509c00640 1 -- 172.21.15.6:0/2478983865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65040c6290 con 0x7f6504147590 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.518+0000 7f6509c00640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 0x7f650415fa40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00efb0 tx=0x7f64fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.518+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc018020 con 0x7f6504147590 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.518+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64fc014480 con 0x7f6504147590 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.518+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc014850 con 0x7f6504147590 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.518+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65040c6530 con 0x7f6504147590 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.518+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65040c6af0 con 0x7f6504147590 2024-03-06T07:28:19.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.519+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650404f330 con 0x7f6504147590 2024-03-06T07:28:19.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.520+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64fc005d10 con 0x7f6504147590 2024-03-06T07:28:19.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.521+0000 7f6500c00640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64e406b4d0 0x7f64e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.521+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f64fc08b060 con 0x7f6504147590 2024-03-06T07:28:19.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.522+0000 7f6509200640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64e406b4d0 0x7f64e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.523+0000 7f6509200640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64e406b4d0 0x7f64e406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f64f8036000 tx=0x7f64f803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.526+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64fc022030 con 0x7f6504147590 2024-03-06T07:28:19.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.726+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65040cf1c0 con 0x7f6504147590 2024-03-06T07:28:19.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.727+0000 7f6500c00640 1 -- 172.21.15.6:0/2478983865 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f64fc010040 con 0x7f6504147590 2024-03-06T07:28:19.729 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:19.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64e406b4d0 msgr2=0x7f64e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64e406b4d0 0x7f64e406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f64f8036000 tx=0x7f64f803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 msgr2=0x7f650415fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 0x7f650415fa40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00efb0 tx=0x7f64fc00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 shutdown_connections 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64e406b4d0 0x7f64e406d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f65040b2250 0x7f650415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6504147590 0x7f650415fa40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 --2- 172.21.15.6:0/2478983865 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f650406d8d0 0x7f6504136c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.730+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 >> 172.21.15.6:0/2478983865 conn(0x7f650406b3a0 msgr2=0x7f6504070b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.731+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 shutdown_connections 2024-03-06T07:28:19.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.731+0000 7f650a600640 1 -- 172.21.15.6:0/2478983865 wait complete. 2024-03-06T07:28:19.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:19.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.974+0000 7f7df9200640 1 Processor -- start 2024-03-06T07:28:19.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.975+0000 7f7df9200640 1 -- start start 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.976+0000 7f7df9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 0x7f7df40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.976+0000 7f7df9200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df40b3710 con 0x7f7df40d0c80 2024-03-06T07:28:19.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.976+0000 7f7df9200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df40b39f0 con 0x7f7df415e0b0 2024-03-06T07:28:19.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.976+0000 7f7df9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40b09e0 con 0x7f7df40d4a10 2024-03-06T07:28:19.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.977+0000 7f7df3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7df40d0c80 0x7f7df40b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32940/0 (socket says 172.21.15.6:32940) 2024-03-06T07:28:19.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.977+0000 7f7df3e00640 1 -- 172.21.15.6:0/1560737394 learned_addr learned my addr 172.21.15.6:0/1560737394 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.977+0000 7f7df2a00640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 0x7f7df40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.977+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955330562 0 0) 0x7f7df40b3710 con 0x7f7df40d0c80 2024-03-06T07:28:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.977+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8003520 con 0x7f7df40d0c80 2024-03-06T07:28:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156644086 0 0) 0x7f7dd8003520 con 0x7f7df40d0c80 2024-03-06T07:28:19.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 msgr2=0x7f7df40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 0x7f7df40d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 >> v1:172.21.15.178:6789/0 conn(0x7f7df415e0b0 legacy=0x7f7df40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40b3cd0 con 0x7f7df40d0c80 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7dec001670 con 0x7f7df40d0c80 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 >> v1:172.21.15.6:6789/0 conn(0x7f7df40d0c80 legacy=0x7f7df40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dd80039e0 0x7f7dd8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd8004800 0x7f7dd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 0x7f7dd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd800ba70 con 0x7f7dd80039e0 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd800bbf0 con 0x7f7dd8004800 2024-03-06T07:28:19.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd800bdf0 con 0x7f7df40d4a10 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df3400640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dd80039e0 0x7f7dd8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df3e00640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 0x7f7dd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.978+0000 7f7df2a00640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd8004800 0x7f7dd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.979+0000 7f7df3400640 1 -- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 msgr2=0x7f7dd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.979+0000 7f7df3400640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7df40d4a10 0x7f7dd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.979+0000 7f7df3400640 1 -- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd8004800 msgr2=0x7f7dd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.979+0000 7f7df3400640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd8004800 0x7f7dd8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.979+0000 7f7df3400640 1 -- 172.21.15.6:0/1560737394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8003520 con 0x7f7dd80039e0 2024-03-06T07:28:19.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.979+0000 7f7df2a00640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7dd8004800 0x7f7dd8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.980+0000 7f7df3400640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dd80039e0 0x7f7dd8003e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001e20 tx=0x7f7dd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5226281d45b7482 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.980+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de4001a70 con 0x7f7dd80039e0 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.980+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de4003b70 con 0x7f7dd80039e0 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.980+0000 7f7df2000640 1 -- 172.21.15.6:0/1560737394 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40050d0 con 0x7f7dd80039e0 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df9200640 1 -- 172.21.15.6:0/1560737394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dd80039e0 msgr2=0x7f7dd8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df9200640 1 --2- 172.21.15.6:0/1560737394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7dd80039e0 0x7f7dd8003e00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7de4001e20 tx=0x7f7dd8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df3e00640 1 -- 172.21.15.6:0/1560737394 reap_dead start 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df9200640 1 -- 172.21.15.6:0/1560737394 shutdown_connections 2024-03-06T07:28:19.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df9200640 1 -- 172.21.15.6:0/1560737394 >> 172.21.15.6:0/1560737394 conn(0x7f7df40d3d30 msgr2=0x7f7df40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df9200640 1 -- 172.21.15.6:0/1560737394 shutdown_connections 2024-03-06T07:28:19.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.981+0000 7f7df9200640 1 -- 172.21.15.6:0/1560737394 wait complete. 2024-03-06T07:28:19.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.983+0000 7f7df9200640 1 Processor -- start 2024-03-06T07:28:19.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.983+0000 7f7df9200640 1 -- start start 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7dd80039e0 0x7f7df415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 0x7f7df40b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df9200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7df41f3d90 0x7f7df4160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df415e490 con 0x7f7df40b78b0 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df9200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40cfa20 con 0x7f7df41f3d90 2024-03-06T07:28:19.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40cfc60 con 0x7f7dd80039e0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 0x7f7df40b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 0x7f7df40b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56292/0 (socket says 172.21.15.6:56292) 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.984+0000 7f7df2a00640 1 -- 172.21.15.6:0/3649865792 learned_addr learned my addr 172.21.15.6:0/3649865792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df3400640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7dd80039e0 0x7f7df415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df3e00640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7df41f3d90 0x7f7df4160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df2a00640 1 -- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7dd80039e0 msgr2=0x7f7df415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df2a00640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7dd80039e0 0x7f7df415bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df2a00640 1 -- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7df41f3d90 msgr2=0x7f7df4160c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df2a00640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7df41f3d90 0x7f7df4160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df2a00640 1 -- 172.21.15.6:0/3649865792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40b92d0 con 0x7f7df40b78b0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df3400640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7dd80039e0 0x7f7df415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df2a00640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 0x7f7df40b8730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc010890 tx=0x7f7ddc010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc00f040 con 0x7f7df40b78b0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc00eb60 con 0x7f7df40b78b0 2024-03-06T07:28:19.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc01c400 con 0x7f7df40b78b0 2024-03-06T07:28:19.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.985+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df40b8ff0 con 0x7f7df40b78b0 2024-03-06T07:28:19.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.986+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df415ea70 con 0x7f7df40b78b0 2024-03-06T07:28:19.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.987+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df404f330 con 0x7f7df40b78b0 2024-03-06T07:28:19.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.988+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ddc01c5a0 con 0x7f7df40b78b0 2024-03-06T07:28:19.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.988+0000 7f7deb400640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7dd006b4d0 0x7f7dd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.988+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7ddc08a8d0 con 0x7f7df40b78b0 2024-03-06T07:28:19.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.989+0000 7f7df3400640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7dd006b4d0 0x7f7dd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.990+0000 7f7df3400640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7dd006b4d0 0x7f7dd006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7de4034000 tx=0x7f7de4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:19.993+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ddc018030 con 0x7f7df40b78b0 2024-03-06T07:28:20.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.210+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7df40b09e0 con 0x7f7df40b78b0 2024-03-06T07:28:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.211+0000 7f7deb400640 1 -- 172.21.15.6:0/3649865792 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7ddc059630 con 0x7f7df40b78b0 2024-03-06T07:28:20.213 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:28:20.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7dd006b4d0 msgr2=0x7f7dd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7dd006b4d0 0x7f7dd006d990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7de4034000 tx=0x7f7de4009030 comp rx=0 tx=0).stop 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 msgr2=0x7f7df40b8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 0x7f7df40b8730 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc010890 tx=0x7f7ddc010c90 comp rx=0 tx=0).stop 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 shutdown_connections 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7dd006b4d0 0x7f7dd006d990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7df41f3d90 0x7f7df4160c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7df40b78b0 0x7f7df40b8730 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 --2- 172.21.15.6:0/3649865792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7dd80039e0 0x7f7df415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 >> 172.21.15.6:0/3649865792 conn(0x7f7df40d3d30 msgr2=0x7f7df40b0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 shutdown_connections 2024-03-06T07:28:20.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.214+0000 7f7df9200640 1 -- 172.21.15.6:0/3649865792 wait complete. 2024-03-06T07:28:20.236 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-06T07:28:20.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-06T07:28:20.335 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:28:20.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:20.336 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:20.390 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:20.390 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:20.444 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:20.444 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:20.497 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:28:20.497 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:20.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:20.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.775+0000 7fa2d7c00640 1 Processor -- start 2024-03-06T07:28:20.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.775+0000 7fa2d7c00640 1 -- start start 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2d7c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2d00af1a0 0x7fa2d015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2d7c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d015cc40 con 0x7fa2d00af580 2024-03-06T07:28:20.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2d7c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d015cf20 con 0x7fa2d00cdc20 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2d7c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d015d200 con 0x7fa2d00af1a0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2d5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2d00af1a0 0x7fa2d015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2cfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa2d00af580 0x7fa2d00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32948/0 (socket says 172.21.15.6:32948) 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.777+0000 7fa2cfe00640 1 -- 172.21.15.6:0/3729036622 learned_addr learned my addr 172.21.15.6:0/3729036622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:20.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.778+0000 7fa2d5200640 1 -- 172.21.15.6:0/3729036622 >> v1:172.21.15.178:6789/0 conn(0x7fa2d00cdc20 legacy=0x7fa2d00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.778+0000 7fa2d5200640 1 -- 172.21.15.6:0/3729036622 >> v1:172.21.15.6:6789/0 conn(0x7fa2d00af580 legacy=0x7fa2d00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.778+0000 7fa2cf400640 1 -- 172.21.15.6:0/3729036622 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536757614 0 0) 0x7fa2d015cc40 con 0x7fa2d00af580 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.778+0000 7fa2d5200640 1 -- 172.21.15.6:0/3729036622 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d015d4e0 con 0x7fa2d00af1a0 2024-03-06T07:28:20.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.779+0000 7fa2d5200640 1 --2- 172.21.15.6:0/3729036622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af1a0 0x7fa2d015c600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40097a0 tx=0x7fa2c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a5780416422ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2cf400640 1 -- 172.21.15.6:0/3729036622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4035020 con 0x7fa2d00af1a0 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2cf400640 1 -- 172.21.15.6:0/3729036622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c4031830 con 0x7fa2d00af1a0 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2cf400640 1 -- 172.21.15.6:0/3729036622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4031c00 con 0x7fa2d00af1a0 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2d7c00640 1 -- 172.21.15.6:0/3729036622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af1a0 msgr2=0x7fa2d015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/3729036622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af1a0 0x7fa2d015c600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40097a0 tx=0x7fa2c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2d7c00640 1 -- 172.21.15.6:0/3729036622 shutdown_connections 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/3729036622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af1a0 0x7fa2d015c600 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.780+0000 7fa2d7c00640 1 -- 172.21.15.6:0/3729036622 >> 172.21.15.6:0/3729036622 conn(0x7fa2d006afb0 msgr2=0x7fa2d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.781+0000 7fa2d7c00640 1 -- 172.21.15.6:0/3729036622 shutdown_connections 2024-03-06T07:28:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.781+0000 7fa2d7c00640 1 -- 172.21.15.6:0/3729036622 wait complete. 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.782+0000 7fa2d7c00640 1 Processor -- start 2024-03-06T07:28:20.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.782+0000 7fa2d7c00640 1 -- start start 2024-03-06T07:28:20.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.783+0000 7fa2d7c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00af1a0 0x7fa2d015bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.783+0000 7fa2d7c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 0x7fa2d006e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.783+0000 7fa2d7c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d00cdc20 0x7fa2d014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2d7c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0155e70 con 0x7fa2d00af580 2024-03-06T07:28:20.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2d7c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b3730 con 0x7fa2d00cdc20 2024-03-06T07:28:20.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2d7c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0155ff0 con 0x7fa2d00af1a0 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 0x7fa2d006e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 0x7fa2d006e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56306/0 (socket says 172.21.15.6:56306) 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 -- 172.21.15.6:0/2097173188 learned_addr learned my addr 172.21.15.6:0/2097173188 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2d5c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d00cdc20 0x7fa2d014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2d5200640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00af1a0 0x7fa2d015bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 -- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00af1a0 msgr2=0x7fa2d015bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00af1a0 0x7fa2d015bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 -- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d00cdc20 msgr2=0x7fa2d014ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d00cdc20 0x7fa2d014ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2cfe00640 1 -- 172.21.15.6:0/2097173188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00b0720 con 0x7fa2d00af580 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.784+0000 7fa2d5200640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00af1a0 0x7fa2d015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.785+0000 7fa2cfe00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 0x7fa2d006e5a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c460 tx=0x7fa2bc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.785+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc018070 con 0x7fa2d00af580 2024-03-06T07:28:20.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.785+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc007520 con 0x7fa2d00af580 2024-03-06T07:28:20.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.785+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc004a20 con 0x7fa2d00af580 2024-03-06T07:28:20.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.785+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d014c3a0 con 0x7fa2d00af580 2024-03-06T07:28:20.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.785+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d014c960 con 0x7fa2d00af580 2024-03-06T07:28:20.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.786+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d015c190 con 0x7fa2d00af580 2024-03-06T07:28:20.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.787+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2bc01b070 con 0x7fa2d00af580 2024-03-06T07:28:20.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.788+0000 7fa2ccc00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.788+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa2bc089770 con 0x7fa2d00af580 2024-03-06T07:28:20.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.789+0000 7fa2d5200640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.790+0000 7fa2d5200640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4036000 tx=0x7fa2c4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.793+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2bc0584d0 con 0x7fa2d00af580 2024-03-06T07:28:20.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.993+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2d0069510 con 0x7fa2d00af580 2024-03-06T07:28:20.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.997+0000 7fa2ccc00640 1 -- 172.21.15.6:0/2097173188 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7fa2bc057dd0 con 0x7fa2d00af580 2024-03-06T07:28:20.999 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:20.999 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 msgr2=0x7fa2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4036000 tx=0x7fa2c4008030 comp rx=0 tx=0).stop 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 msgr2=0x7fa2d006e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 0x7fa2d006e5a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00c460 tx=0x7fa2bc00c920 comp rx=0 tx=0).stop 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 shutdown_connections 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d00cdc20 0x7fa2d014ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00af580 0x7fa2d006e5a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:20.999+0000 7fa2d7c00640 1 --2- 172.21.15.6:0/2097173188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00af1a0 0x7fa2d015bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.000+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 >> 172.21.15.6:0/2097173188 conn(0x7fa2d006afb0 msgr2=0x7fa2d00ade30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.000+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 shutdown_connections 2024-03-06T07:28:21.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.000+0000 7fa2d7c00640 1 -- 172.21.15.6:0/2097173188 wait complete. 2024-03-06T07:28:21.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:21.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:21.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:21.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:21.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.245+0000 7f80a7800640 1 Processor -- start 2024-03-06T07:28:21.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.245+0000 7f80a7800640 1 -- start start 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a7800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a00ccad0 0x7f80a00cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a7800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f80a00adb00 0x7f80a00adf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a7800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00cd3f0 con 0x7f80a0162bb0 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a7800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a0163720 con 0x7f80a00adb00 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a7800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0163950 con 0x7f80a00ccad0 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f809fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f80a0162bb0 0x7f80a0163010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32952/0 (socket says 172.21.15.6:32952) 2024-03-06T07:28:21.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f809fe00640 1 -- 172.21.15.6:0/2814625186 learned_addr learned my addr 172.21.15.6:0/2814625186 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:21.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a5800640 1 --2- 172.21.15.6:0/2814625186 >> v2:172.21.15.178:3300/0 conn(0x7f80a00adb00 0x7f80a00adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f80a4e00640 1 --2- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a00ccad0 0x7f80a00cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.247+0000 7f809f400640 1 -- 172.21.15.6:0/2814625186 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819478069 0 0) 0x7f80a00cd3f0 con 0x7f80a0162bb0 2024-03-06T07:28:21.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.248+0000 7f809f400640 1 -- 172.21.15.6:0/2814625186 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8088003520 con 0x7f80a0162bb0 2024-03-06T07:28:21.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.248+0000 7f80a5800640 1 -- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a00ccad0 msgr2=0x7f80a00cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.248+0000 7f80a5800640 1 --2- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a00ccad0 0x7f80a00cceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.248+0000 7f809f400640 1 -- 172.21.15.6:0/2814625186 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368068554 0 0) 0x7f8088003520 con 0x7f80a0162bb0 2024-03-06T07:28:21.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.248+0000 7f80a5800640 1 -- 172.21.15.6:0/2814625186 >> v1:172.21.15.6:6789/0 conn(0x7f80a0162bb0 legacy=0x7f80a0163010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.248+0000 7f80a5800640 1 -- 172.21.15.6:0/2814625186 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a00acd40 con 0x7f80a00adb00 2024-03-06T07:28:21.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.249+0000 7f80a5800640 1 --2- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00adb00 0x7f80a00adf10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8098009d90 tx=0x7f809802bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=110eb50774e37de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.249+0000 7f809f400640 1 -- 172.21.15.6:0/2814625186 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098039070 con 0x7f80a00adb00 2024-03-06T07:28:21.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f809f400640 1 -- 172.21.15.6:0/2814625186 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809802c8a0 con 0x7f80a00adb00 2024-03-06T07:28:21.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f809f400640 1 -- 172.21.15.6:0/2814625186 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c001670 con 0x7f80a00adb00 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f80a7800640 1 -- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00adb00 msgr2=0x7f80a00adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f80a7800640 1 --2- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00adb00 0x7f80a00adf10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8098009d90 tx=0x7f809802bd70 comp rx=0 tx=0).stop 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f80a7800640 1 -- 172.21.15.6:0/2814625186 shutdown_connections 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f80a7800640 1 --2- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00adb00 0x7f80a00adf10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f80a7800640 1 --2- 172.21.15.6:0/2814625186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a00ccad0 0x7f80a00cceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.250+0000 7f80a7800640 1 -- 172.21.15.6:0/2814625186 >> 172.21.15.6:0/2814625186 conn(0x7f80a006b3d0 msgr2=0x7f80a006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.251+0000 7f80a7800640 1 -- 172.21.15.6:0/2814625186 shutdown_connections 2024-03-06T07:28:21.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.251+0000 7f80a7800640 1 -- 172.21.15.6:0/2814625186 wait complete. 2024-03-06T07:28:21.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.252+0000 7f80a7800640 1 Processor -- start 2024-03-06T07:28:21.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.252+0000 7f80a7800640 1 -- start start 2024-03-06T07:28:21.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.253+0000 7f80a7800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 0x7f80a01f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.253+0000 7f80a7800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00ccad0 0x7f80a014e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.253+0000 7f80a7800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a0162bb0 0x7f80a00b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.253+0000 7f80a7800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00ad5e0 con 0x7f80a00adb00 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.253+0000 7f80a7800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00ad300 con 0x7f80a00ccad0 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.253+0000 7f80a7800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a006e7d0 con 0x7f80a0162bb0 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 0x7f80a01f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 0x7f80a01f8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56310/0 (socket says 172.21.15.6:56310) 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 -- 172.21.15.6:0/4278116843 learned_addr learned my addr 172.21.15.6:0/4278116843 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a5800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a0162bb0 0x7f80a00b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f809fe00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00ccad0 0x7f80a014e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 -- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a0162bb0 msgr2=0x7f80a00b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a0162bb0 0x7f80a00b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 -- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00ccad0 msgr2=0x7f80a014e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00ccad0 0x7f80a014e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.254+0000 7f80a4e00640 1 -- 172.21.15.6:0/4278116843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a01f8f90 con 0x7f80a00adb00 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.255+0000 7f80a4e00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 0x7f80a01f8a50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f809400e870 tx=0x7f809400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.255+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094017070 con 0x7f80a00adb00 2024-03-06T07:28:21.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.255+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809400c940 con 0x7f80a00adb00 2024-03-06T07:28:21.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.255+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809400cd30 con 0x7f80a00adb00 2024-03-06T07:28:21.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.255+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a00aef30 con 0x7f80a00adb00 2024-03-06T07:28:21.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.255+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a00af4b0 con 0x7f80a00adb00 2024-03-06T07:28:21.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.257+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a004f330 con 0x7f80a00adb00 2024-03-06T07:28:21.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.257+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f809401a070 con 0x7f80a00adb00 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.258+0000 7f809cc00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.258+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f809408a0d0 con 0x7f80a00adb00 2024-03-06T07:28:21.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.259+0000 7f809fe00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.260+0000 7f809fe00640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808006b4d0 0x7f808006d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f808c0013c0 tx=0x7f808c001350 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.263+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80a01f8f90 con 0x7f80a00adb00 2024-03-06T07:28:21.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.464+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80a006ba60 con 0x7f80a00adb00 2024-03-06T07:28:21.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.465+0000 7f809cc00640 1 -- 172.21.15.6:0/4278116843 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8094058860 con 0x7f80a00adb00 2024-03-06T07:28:21.467 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.467+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808006b4d0 msgr2=0x7f808006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.467+0000 7f80a7800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808006b4d0 0x7f808006d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f808c0013c0 tx=0x7f808c001350 comp rx=0 tx=0).stop 2024-03-06T07:28:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.467+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 msgr2=0x7f80a01f8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.467+0000 7f80a7800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 0x7f80a01f8a50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f809400e870 tx=0x7f809400ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 shutdown_connections 2024-03-06T07:28:21.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808006b4d0 0x7f808006d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80a00ccad0 0x7f80a014e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80a0162bb0 0x7f80a00b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 --2- 172.21.15.6:0/4278116843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80a00adb00 0x7f80a01f8a50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 >> 172.21.15.6:0/4278116843 conn(0x7f80a006b3d0 msgr2=0x7f80a01610d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 shutdown_connections 2024-03-06T07:28:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.468+0000 7f80a7800640 1 -- 172.21.15.6:0/4278116843 wait complete. 2024-03-06T07:28:21.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.715+0000 7faa07200640 1 Processor -- start 2024-03-06T07:28:21.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.715+0000 7faa07200640 1 -- start start 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.716+0000 7faa07200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 0x7faa0815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7faa07200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0815fe40 con 0x7faa0815cc90 2024-03-06T07:28:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7faa07200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa08160120 con 0x7faa080b2250 2024-03-06T07:28:21.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7faa07200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa08160400 con 0x7faa08145430 2024-03-06T07:28:21.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7fa9ffe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faa0815cc90 0x7faa080b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:32966/0 (socket says 172.21.15.6:32966) 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7fa9ffe00640 1 -- 172.21.15.6:0/2316117166 learned_addr learned my addr 172.21.15.6:0/2316117166 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7faa05e00640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 0x7faa0815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515835688 0 0) 0x7faa0815fe40 con 0x7faa0815cc90 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.717+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ec003520 con 0x7faa0815cc90 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2561274406 0 0) 0x7fa9ec003520 con 0x7faa0815cc90 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 msgr2=0x7faa0815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 0x7faa0815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 >> v1:172.21.15.178:6789/0 conn(0x7faa080b2250 legacy=0x7faa080b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa081606e0 con 0x7faa0815cc90 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fa9f0001670 con 0x7faa0815cc90 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 >> v1:172.21.15.6:6789/0 conn(0x7faa0815cc90 legacy=0x7faa080b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9ec0039e0 0x7fa9ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9ec004800 0x7fa9ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 0x7fa9ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08160400 con 0x7fa9ec004800 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec00baa0 con 0x7fa9ec0039e0 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec00bca0 con 0x7faa08145430 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.718+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9ec004800 0x7fa9ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.719+0000 7faa05e00640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9ec0039e0 0x7fa9ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.719+0000 7fa9ffe00640 1 -- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 msgr2=0x7fa9ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.719+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa08145430 0x7fa9ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.719+0000 7fa9ffe00640 1 -- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9ec0039e0 msgr2=0x7fa9ec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.719+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9ec0039e0 0x7fa9ec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.719+0000 7fa9ffe00640 1 -- 172.21.15.6:0/2316117166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec003520 con 0x7fa9ec004800 2024-03-06T07:28:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.720+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9ec004800 0x7fa9ec004c80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c0e0 tx=0x7fa9ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c4eb64ed611541 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.720+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0007c40 con 0x7fa9ec004800 2024-03-06T07:28:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.720+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f0030070 con 0x7fa9ec004800 2024-03-06T07:28:21.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.720+0000 7faa05400640 1 -- 172.21.15.6:0/2316117166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0005d30 con 0x7fa9ec004800 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa07200640 1 -- 172.21.15.6:0/2316117166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9ec004800 msgr2=0x7fa9ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa07200640 1 --2- 172.21.15.6:0/2316117166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9ec004800 0x7fa9ec004c80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c0e0 tx=0x7fa9ec005830 comp rx=0 tx=0).stop 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa06800640 1 -- 172.21.15.6:0/2316117166 reap_dead start 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa07200640 1 -- 172.21.15.6:0/2316117166 shutdown_connections 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa07200640 1 -- 172.21.15.6:0/2316117166 >> 172.21.15.6:0/2316117166 conn(0x7faa0806b3a0 msgr2=0x7faa0806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa07200640 1 -- 172.21.15.6:0/2316117166 shutdown_connections 2024-03-06T07:28:21.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.721+0000 7faa07200640 1 -- 172.21.15.6:0/2316117166 wait complete. 2024-03-06T07:28:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.723+0000 7faa07200640 1 Processor -- start 2024-03-06T07:28:21.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.723+0000 7faa07200640 1 -- start start 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7faa07200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9ec004800 0x7faa0815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7faa07200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 0x7faa08141df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7faa07200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa080b05f0 0x7faa080b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7faa07200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08145990 con 0x7faa081e6160 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7faa07200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08145810 con 0x7faa080b05f0 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7faa07200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa080cde00 con 0x7fa9ec004800 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7fa9ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 0x7faa08141df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7fa9ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 0x7faa08141df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56326/0 (socket says 172.21.15.6:56326) 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.724+0000 7fa9ffe00640 1 -- 172.21.15.6:0/88303265 learned_addr learned my addr 172.21.15.6:0/88303265 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7faa05e00640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9ec004800 0x7faa0815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7fa9ffe00640 1 -- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9ec004800 msgr2=0x7faa0815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9ec004800 0x7faa0815b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7fa9ffe00640 1 -- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa080b05f0 msgr2=0x7faa080b0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7faa06800640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa080b05f0 0x7faa080b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa080b05f0 0x7faa080b0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7fa9ffe00640 1 -- 172.21.15.6:0/88303265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa08142810 con 0x7faa081e6160 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7faa06800640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa080b05f0 0x7faa080b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.725+0000 7fa9ffe00640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 0x7faa08141df0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c200 tx=0x7faa08160010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.726+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0045070 con 0x7faa081e6160 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.726+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f0000ed0 con 0x7faa081e6160 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.726+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0007c40 con 0x7faa081e6160 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.726+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08142af0 con 0x7faa081e6160 2024-03-06T07:28:21.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.726+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa081430b0 con 0x7faa081e6160 2024-03-06T07:28:21.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.727+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa080b0fb0 con 0x7faa081e6160 2024-03-06T07:28:21.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.728+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9f0007070 con 0x7faa081e6160 2024-03-06T07:28:21.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.729+0000 7fa9fe000640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9e406b500 0x7fa9e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.729+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa9f00ab6f0 con 0x7faa081e6160 2024-03-06T07:28:21.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.729+0000 7faa05e00640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9e406b500 0x7fa9e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.731+0000 7faa05e00640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9e406b500 0x7fa9e406d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009ab0 tx=0x7fa9f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.733+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa9f007a4e0 con 0x7faa081e6160 2024-03-06T07:28:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.936+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa0804f330 con 0x7faa081e6160 2024-03-06T07:28:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.936+0000 7fa9fe000640 1 -- 172.21.15.6:0/88303265 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9f0079de0 con 0x7faa081e6160 2024-03-06T07:28:21.938 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9e406b500 msgr2=0x7fa9e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9e406b500 0x7fa9e406d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009ab0 tx=0x7fa9f8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 msgr2=0x7faa08141df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 0x7faa08141df0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c200 tx=0x7faa08160010 comp rx=0 tx=0).stop 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 shutdown_connections 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9e406b500 0x7fa9e406d9c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa080b05f0 0x7faa080b0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa081e6160 0x7faa08141df0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 --2- 172.21.15.6:0/88303265 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9ec004800 0x7faa0815b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.939+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 >> 172.21.15.6:0/88303265 conn(0x7faa0806b3a0 msgr2=0x7faa080cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.940+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 shutdown_connections 2024-03-06T07:28:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:21.940+0000 7faa07200640 1 -- 172.21.15.6:0/88303265 wait complete. 2024-03-06T07:28:21.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.179+0000 7f4420a00640 1 Processor -- start 2024-03-06T07:28:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.179+0000 7f4420a00640 1 -- start start 2024-03-06T07:28:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.180+0000 7f4420a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 0x7f441c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.180+0000 7f4420a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c071c60 con 0x7f441c0cd1e0 2024-03-06T07:28:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.180+0000 7f4420a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c071f40 con 0x7f441c070ce0 2024-03-06T07:28:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.180+0000 7f4420a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0cfa10 con 0x7f441c15afa0 2024-03-06T07:28:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f441a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f441c0cd1e0 0x7f441c0b3ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59536/0 (socket says 172.21.15.6:59536) 2024-03-06T07:28:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f441a800640 1 -- 172.21.15.6:0/11198551 learned_addr learned my addr 172.21.15.6:0/11198551 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4419e00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 0x7f441c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153175550 0 0) 0x7f441c071c60 con 0x7f441c0cd1e0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4400003520 con 0x7f441c0cd1e0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1406922055 0 0) 0x7f4400003520 con 0x7f441c0cd1e0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 msgr2=0x7f441c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 0x7f441c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 >> v1:172.21.15.178:6789/0 conn(0x7f441c070ce0 legacy=0x7f441c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.181+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c072220 con 0x7f441c0cd1e0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f44140019f0 con 0x7f441c0cd1e0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 >> v1:172.21.15.6:6789/0 conn(0x7f441c0cd1e0 legacy=0x7f441c0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 0x7f44000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4400003fb0 0x7f4400004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400004d40 0x7f44000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0cfa10 con 0x7f4400003fb0 2024-03-06T07:28:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440000bbb0 con 0x7f4400004d40 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440000bdb0 con 0x7f441c15afa0 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419400640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4400003fb0 0x7f4400004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f441a800640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400004d40 0x7f44000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419e00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 0x7f44000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419400640 1 -- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 msgr2=0x7f44000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419400640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c15afa0 0x7f44000039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419400640 1 -- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400004d40 msgr2=0x7f44000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419400640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400004d40 0x7f44000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.182+0000 7f4419400640 1 -- 172.21.15.6:0/11198551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400003520 con 0x7f4400003fb0 2024-03-06T07:28:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f4419400640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4400003fb0 0x7f4400004800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4408001750 tx=0x7f4400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2f4ba5576555d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408001370 con 0x7f4400003fb0 2024-03-06T07:28:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44080052d0 con 0x7f4400003fb0 2024-03-06T07:28:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f4418a00640 1 -- 172.21.15.6:0/11198551 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408003830 con 0x7f4400003fb0 2024-03-06T07:28:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f4420a00640 1 -- 172.21.15.6:0/11198551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4400003fb0 msgr2=0x7f4400004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f4420a00640 1 --2- 172.21.15.6:0/11198551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4400003fb0 0x7f4400004800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4408001750 tx=0x7f4400005830 comp rx=0 tx=0).stop 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.184+0000 7f441a800640 1 -- 172.21.15.6:0/11198551 reap_dead start 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.185+0000 7f4420a00640 1 -- 172.21.15.6:0/11198551 shutdown_connections 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.185+0000 7f4420a00640 1 -- 172.21.15.6:0/11198551 >> 172.21.15.6:0/11198551 conn(0x7f441c06b3d0 msgr2=0x7f441c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.185+0000 7f4420a00640 1 -- 172.21.15.6:0/11198551 shutdown_connections 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.185+0000 7f4420a00640 1 -- 172.21.15.6:0/11198551 wait complete. 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.186+0000 7f4420a00640 1 Processor -- start 2024-03-06T07:28:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.187+0000 7f4420a00640 1 -- start start 2024-03-06T07:28:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4420a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400003fb0 0x7f441c09d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4420a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 0x7f441c09ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4420a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c148690 0x7f441c148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4420a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0cf3a0 con 0x7f441c09aa10 2024-03-06T07:28:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4420a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c071140 con 0x7f4400003fb0 2024-03-06T07:28:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4420a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0cf520 con 0x7f441c148690 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 0x7f441c09ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 0x7f441c09ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43650/0 (socket says 172.21.15.6:43650) 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 -- 172.21.15.6:0/4253879904 learned_addr learned my addr 172.21.15.6:0/4253879904 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419e00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400003fb0 0x7f441c09d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f441a800640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c148690 0x7f441c148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 -- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c148690 msgr2=0x7f441c148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c148690 0x7f441c148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 -- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400003fb0 msgr2=0x7f441c09d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400003fb0 0x7f441c09d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f4419400640 1 -- 172.21.15.6:0/4253879904 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c1475f0 con 0x7f441c09aa10 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.188+0000 7f441a800640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c148690 0x7f441c148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.189+0000 7f4419400640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 0x7f441c09ae90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4408005040 tx=0x7f4408001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.189+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408041070 con 0x7f441c09aa10 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.189+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4408005480 con 0x7f441c09aa10 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.189+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408007a30 con 0x7f441c09aa10 2024-03-06T07:28:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.189+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c1478d0 con 0x7f441c09aa10 2024-03-06T07:28:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.189+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c147e90 con 0x7f441c09aa10 2024-03-06T07:28:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.191+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c15e260 con 0x7f441c09aa10 2024-03-06T07:28:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.192+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4408044070 con 0x7f441c09aa10 2024-03-06T07:28:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.192+0000 7f4412000640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43fc06b500 0x7f43fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.192+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f44080abd90 con 0x7f441c09aa10 2024-03-06T07:28:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.193+0000 7f4419e00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43fc06b500 0x7f43fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.194+0000 7f4419e00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43fc06b500 0x7f43fc06d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f441c14ad90 tx=0x7f440c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.197+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f441c15e260 con 0x7f441c09aa10 2024-03-06T07:28:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.400+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f441c04f330 con 0x7f441c09aa10 2024-03-06T07:28:22.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.401+0000 7f4412000640 1 -- 172.21.15.6:0/4253879904 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f440807ac10 con 0x7f441c09aa10 2024-03-06T07:28:22.403 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43fc06b500 msgr2=0x7f43fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43fc06b500 0x7f43fc06d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f441c14ad90 tx=0x7f440c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 msgr2=0x7f441c09ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 0x7f441c09ae90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4408005040 tx=0x7f4408001120 comp rx=0 tx=0).stop 2024-03-06T07:28:22.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 shutdown_connections 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43fc06b500 0x7f43fc06d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f441c148690 0x7f441c148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f441c09aa10 0x7f441c09ae90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 --2- 172.21.15.6:0/4253879904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4400003fb0 0x7f441c09d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 >> 172.21.15.6:0/4253879904 conn(0x7f441c06b3d0 msgr2=0x7f441c15d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 shutdown_connections 2024-03-06T07:28:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.404+0000 7f4420a00640 1 -- 172.21.15.6:0/4253879904 wait complete. 2024-03-06T07:28:22.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:22.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.643+0000 7f9c77e00640 1 Processor -- start 2024-03-06T07:28:22.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.643+0000 7f9c77e00640 1 -- start start 2024-03-06T07:28:22.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c77e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c77e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c7815e810 0x7f9c780b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c77e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7815ed10 con 0x7f9c781552b0 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c77e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7815eff0 con 0x7f9c7815e810 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c77e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7815f2d0 con 0x7f9c780cd060 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c76a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9c781552b0 0x7f9c780cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59544/0 (socket says 172.21.15.6:59544) 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.644+0000 7f9c76a00640 1 -- 172.21.15.6:0/3033573312 learned_addr learned my addr 172.21.15.6:0/3033573312 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:22.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c77400640 1 --2- 172.21.15.6:0/3033573312 >> v2:172.21.15.6:3300/0 conn(0x7f9c7815e810 0x7f9c780b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c76000640 1 -- 172.21.15.6:0/3033573312 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344081904 0 0) 0x7f9c7815ed10 con 0x7f9c781552b0 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c6fe00640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c76000640 1 -- 172.21.15.6:0/3033573312 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c003520 con 0x7f9c781552b0 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c77400640 1 -- 172.21.15.6:0/3033573312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 msgr2=0x7f9c7815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c77400640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c77400640 1 -- 172.21.15.6:0/3033573312 >> v1:172.21.15.6:6789/0 conn(0x7f9c781552b0 legacy=0x7f9c780cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c77400640 1 -- 172.21.15.6:0/3033573312 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7815f5b0 con 0x7f9c7815e810 2024-03-06T07:28:22.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.645+0000 7f9c6fe00640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:22.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.646+0000 7f9c77400640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c780b3cb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9c70009d90 tx=0x7f9c7002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa14d46af2bbfa96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.646+0000 7f9c76000640 1 -- 172.21.15.6:0/3033573312 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70039070 con 0x7f9c7815e810 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c76000640 1 -- 172.21.15.6:0/3033573312 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7002cbc0 con 0x7f9c7815e810 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c76000640 1 -- 172.21.15.6:0/3033573312 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70034440 con 0x7f9c7815e810 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 -- 172.21.15.6:0/3033573312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 msgr2=0x7f9c780b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c780b3cb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9c70009d90 tx=0x7f9c7002be10 comp rx=0 tx=0).stop 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 -- 172.21.15.6:0/3033573312 shutdown_connections 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c780b3cb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 --2- 172.21.15.6:0/3033573312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 -- 172.21.15.6:0/3033573312 >> 172.21.15.6:0/3033573312 conn(0x7f9c7806b3d0 msgr2=0x7f9c7806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 -- 172.21.15.6:0/3033573312 shutdown_connections 2024-03-06T07:28:22.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.647+0000 7f9c77e00640 1 -- 172.21.15.6:0/3033573312 wait complete. 2024-03-06T07:28:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.649+0000 7f9c77e00640 1 Processor -- start 2024-03-06T07:28:22.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.649+0000 7f9c77e00640 1 -- start start 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7814e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c781552b0 0x7f9c7814fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c78137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7806dd30 con 0x7f9c7815e810 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780b3760 con 0x7f9c781552b0 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7806deb0 con 0x7f9c780cd060 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c78137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c78137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43668/0 (socket says 172.21.15.6:43668) 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.650+0000 7f9c77400640 1 -- 172.21.15.6:0/558261647 learned_addr learned my addr 172.21.15.6:0/558261647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c76a00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7814e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c77400640 1 -- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 msgr2=0x7f9c7814e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c77400640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7814e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c77400640 1 -- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c781552b0 msgr2=0x7f9c7814fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c77400640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c781552b0 0x7f9c7814fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c77400640 1 -- 172.21.15.6:0/558261647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c78154660 con 0x7f9c7815e810 2024-03-06T07:28:22.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c77400640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c78137e70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c70009eb0 tx=0x7f9c70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70039050 con 0x7f9c7815e810 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.651+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c70045070 con 0x7f9c7815e810 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.652+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70009450 con 0x7f9c7815e810 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.652+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c78151ea0 con 0x7f9c7815e810 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.652+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7814e720 con 0x7f9c7815e810 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.653+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c78151cc0 con 0x7f9c7815e810 2024-03-06T07:28:22.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.654+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c7003e850 con 0x7f9c7815e810 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.654+0000 7f9c6f400640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.655+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9c700ac970 con 0x7f9c7815e810 2024-03-06T07:28:22.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.655+0000 7f9c76a00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.656+0000 7f9c76a00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c5406b500 0x7f9c5406d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c68000f60 tx=0x7f9c68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.659+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c78151cc0 con 0x7f9c7815e810 2024-03-06T07:28:22.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.871+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9c7804f330 con 0x7f9c7815e810 2024-03-06T07:28:22.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.872+0000 7f9c6f400640 1 -- 172.21.15.6:0/558261647 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9c7004e020 con 0x7f9c7815e810 2024-03-06T07:28:22.874 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:28:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c5406b500 msgr2=0x7f9c5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c5406b500 0x7f9c5406d9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c68000f60 tx=0x7f9c68009040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 msgr2=0x7f9c78137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c78137e70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c70009eb0 tx=0x7f9c70004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 shutdown_connections 2024-03-06T07:28:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c5406b500 0x7f9c5406d9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c781552b0 0x7f9c7814fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c7815e810 0x7f9c78137e70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 --2- 172.21.15.6:0/558261647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c780cd060 0x7f9c7814e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 >> 172.21.15.6:0/558261647 conn(0x7f9c7806b3d0 msgr2=0x7f9c78071970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 shutdown_connections 2024-03-06T07:28:22.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:22.875+0000 7f9c77e00640 1 -- 172.21.15.6:0/558261647 wait complete. 2024-03-06T07:28:22.898 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-06T07:28:22.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:22.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-06T07:28:22.986 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi178.front.sepia.ceph.com 2024-03-06T07:28:22.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:22.987 DEBUG:teuthology.orchestra.run.smithi178:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:23.029 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:23.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.256+0000 7fcd9b200640 1 Processor -- start 2024-03-06T07:28:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.256+0000 7fcd9b200640 1 -- start start 2024-03-06T07:28:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.257+0000 7fcd9b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcd9c0cf080 0x7fcd9c0d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.257+0000 7fcd9b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c0b1370 con 0x7fcd9c0b0f90 2024-03-06T07:28:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.257+0000 7fcd9b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c0ccf80 con 0x7fcd9c0d05e0 2024-03-06T07:28:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.257+0000 7fcd9b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c0d0a10 con 0x7fcd9c0cf080 2024-03-06T07:28:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.258+0000 7fcd99e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcd9c0b0f90 0x7fcd9c0cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59550/0 (socket says 172.21.15.6:59550) 2024-03-06T07:28:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.258+0000 7fcd99e00640 1 -- 172.21.15.6:0/3796584707 learned_addr learned my addr 172.21.15.6:0/3796584707 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.258+0000 7fcd91400640 1 --2- 172.21.15.6:0/3796584707 >> v2:172.21.15.6:3300/0 conn(0x7fcd9c0cf080 0x7fcd9c0d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.258+0000 7fcd99400640 1 -- 172.21.15.6:0/3796584707 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403851579 0 0) 0x7fcd9c0b1370 con 0x7fcd9c0b0f90 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.258+0000 7fcd99400640 1 -- 172.21.15.6:0/3796584707 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd80003520 con 0x7fcd9c0b0f90 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.258+0000 7fcd91400640 1 -- 172.21.15.6:0/3796584707 >> v1:172.21.15.178:6789/0 conn(0x7fcd9c0d05e0 legacy=0x7fcd9c15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.259+0000 7fcd91400640 1 -- 172.21.15.6:0/3796584707 >> v1:172.21.15.6:6789/0 conn(0x7fcd9c0b0f90 legacy=0x7fcd9c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.259+0000 7fcd91400640 1 -- 172.21.15.6:0/3796584707 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0cd260 con 0x7fcd9c0cf080 2024-03-06T07:28:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.260+0000 7fcd91400640 1 --2- 172.21.15.6:0/3796584707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c0d00a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009d90 tx=0x7fcd8402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04b8b8d9e438805 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.260+0000 7fcd99400640 1 -- 172.21.15.6:0/3796584707 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84039070 con 0x7fcd9c0cf080 2024-03-06T07:28:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.260+0000 7fcd99400640 1 -- 172.21.15.6:0/3796584707 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8402cc70 con 0x7fcd9c0cf080 2024-03-06T07:28:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.260+0000 7fcd99400640 1 -- 172.21.15.6:0/3796584707 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84034640 con 0x7fcd9c0cf080 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 -- 172.21.15.6:0/3796584707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 msgr2=0x7fcd9c0d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3796584707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c0d00a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009d90 tx=0x7fcd8402bec0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 -- 172.21.15.6:0/3796584707 shutdown_connections 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3796584707 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c0d00a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 -- 172.21.15.6:0/3796584707 >> 172.21.15.6:0/3796584707 conn(0x7fcd9c0d1ae0 msgr2=0x7fcd9c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 -- 172.21.15.6:0/3796584707 shutdown_connections 2024-03-06T07:28:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.261+0000 7fcd9b200640 1 -- 172.21.15.6:0/3796584707 wait complete. 2024-03-06T07:28:23.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.262+0000 7fcd9b200640 1 Processor -- start 2024-03-06T07:28:23.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.263+0000 7fcd9b200640 1 -- start start 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.263+0000 7fcd9b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd9c0b0f90 0x7fcd9c15a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd9b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd9b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd9c0d05e0 0x7fcd9c15e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd9b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c06ffe0 con 0x7fcd9c0cf080 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd9b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cd820 con 0x7fcd9c0b0f90 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd9b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c070160 con 0x7fcd9c0d05e0 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd91400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd91400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c15ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43684/0 (socket says 172.21.15.6:43684) 2024-03-06T07:28:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd91400640 1 -- 172.21.15.6:0/3235516674 learned_addr learned my addr 172.21.15.6:0/3235516674 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd9a800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd9c0d05e0 0x7fcd9c15e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.264+0000 7fcd99e00640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd9c0b0f90 0x7fcd9c15a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd91400640 1 -- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd9c0d05e0 msgr2=0x7fcd9c15e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd91400640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd9c0d05e0 0x7fcd9c15e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd91400640 1 -- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd9c0b0f90 msgr2=0x7fcd9c15a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd91400640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd9c0b0f90 0x7fcd9c15a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd91400640 1 -- 172.21.15.6:0/3235516674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c15c630 con 0x7fcd9c0cf080 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd91400640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c15ba50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009d90 tx=0x7fcd84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84039050 con 0x7fcd9c0cf080 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8403a040 con 0x7fcd9c0cf080 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84009450 con 0x7fcd9c0cf080 2024-03-06T07:28:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.265+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c15c350 con 0x7fcd9c0cf080 2024-03-06T07:28:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.266+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c1f3d30 con 0x7fcd9c0cf080 2024-03-06T07:28:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.267+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c04f330 con 0x7fcd9c0cf080 2024-03-06T07:28:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.268+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd84047020 con 0x7fcd9c0cf080 2024-03-06T07:28:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.268+0000 7fcd92a00640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcd7806b4d0 0x7fcd7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.269+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcd840ac3b0 con 0x7fcd9c0cf080 2024-03-06T07:28:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.269+0000 7fcd99e00640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcd7806b4d0 0x7fcd7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.270+0000 7fcd99e00640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcd7806b4d0 0x7fcd7806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0011e0 tx=0x7fcd8c007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.273+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd8404e020 con 0x7fcd9c0cf080 2024-03-06T07:28:23.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.513+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd9c1613a0 con 0x7fcd9c0cf080 2024-03-06T07:28:23.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.515+0000 7fcd92a00640 1 -- 172.21.15.6:0/3235516674 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7fcd8402ce20 con 0x7fcd9c0cf080 2024-03-06T07:28:23.517 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:23.517 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:28:23.518 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:28:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcd7806b4d0 msgr2=0x7fcd7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcd7806b4d0 0x7fcd7806d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0011e0 tx=0x7fcd8c007d30 comp rx=0 tx=0).stop 2024-03-06T07:28:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 msgr2=0x7fcd9c15ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c15ba50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009d90 tx=0x7fcd84004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 shutdown_connections 2024-03-06T07:28:23.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcd7806b4d0 0x7fcd7806d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd9c0d05e0 0x7fcd9c15e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd9c0cf080 0x7fcd9c15ba50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 --2- 172.21.15.6:0/3235516674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd9c0b0f90 0x7fcd9c15a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 >> 172.21.15.6:0/3235516674 conn(0x7fcd9c0d1ae0 msgr2=0x7fcd9c06be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 shutdown_connections 2024-03-06T07:28:23.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.518+0000 7fcd9b200640 1 -- 172.21.15.6:0/3235516674 wait complete. 2024-03-06T07:28:23.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:23.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.756+0000 7f5dd3200640 1 Processor -- start 2024-03-06T07:28:23.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.757+0000 7f5dd3200640 1 -- start start 2024-03-06T07:28:23.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.758+0000 7f5dd3200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5dd415e810 0x7f5dd415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.758+0000 7f5dd3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd415ed10 con 0x7f5dd41552b0 2024-03-06T07:28:23.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.758+0000 7f5dd3200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd415eff0 con 0x7f5dd40cd060 2024-03-06T07:28:23.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.758+0000 7f5dd3200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd415f2d0 con 0x7f5dd415e810 2024-03-06T07:28:23.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5dd41552b0 0x7f5dd40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59554/0 (socket says 172.21.15.6:59554) 2024-03-06T07:28:23.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1e00640 1 -- 172.21.15.6:0/1642944887 learned_addr learned my addr 172.21.15.6:0/1642944887 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd2800640 1 --2- 172.21.15.6:0/1642944887 >> v2:172.21.15.178:3300/0 conn(0x7f5dd415e810 0x7f5dd415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131731705 0 0) 0x7f5dd415ed10 con 0x7f5dd41552b0 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8003520 con 0x7f5dd41552b0 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 521088939 0 0) 0x7f5db8003520 con 0x7f5dd41552b0 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 >> v2:172.21.15.178:3300/0 conn(0x7f5dd415e810 msgr2=0x7f5dd415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 --2- 172.21.15.6:0/1642944887 >> v2:172.21.15.178:3300/0 conn(0x7f5dd415e810 0x7f5dd415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 >> v1:172.21.15.178:6789/0 conn(0x7f5dd40cd060 legacy=0x7f5dd40b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd415f5b0 con 0x7f5dd41552b0 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.759+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5dc4001690 con 0x7f5dd41552b0 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 >> v1:172.21.15.6:6789/0 conn(0x7f5dd41552b0 legacy=0x7f5dd40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db80039b0 0x7f5db8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db8004800 0x7f5db8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5db80051c0 0x7f5db800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd415f2d0 con 0x7f5db8004800 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db800c060 con 0x7f5db80039b0 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db800c260 con 0x7f5db80051c0 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dc9400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db8004800 0x7f5db8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd1e00640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db80039b0 0x7f5db8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd2800640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5db80051c0 0x7f5db800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dc9400640 1 -- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5db80051c0 msgr2=0x7f5db800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dc9400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5db80051c0 0x7f5db800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dc9400640 1 -- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db80039b0 msgr2=0x7f5db8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dc9400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db80039b0 0x7f5db8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dc9400640 1 -- 172.21.15.6:0/1642944887 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8003520 con 0x7f5db8004800 2024-03-06T07:28:23.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd2800640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5db80051c0 0x7f5db800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.760+0000 7f5dd2800640 1 -- 172.21.15.6:0/1642944887 reap_dead start 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dc9400640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db8004800 0x7f5db8004c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc001330 tx=0x7f5db8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf195b1dd6f81598 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc001370 con 0x7f5db8004800 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc0052d0 con 0x7f5db8004800 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd1400640 1 -- 172.21.15.6:0/1642944887 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc003830 con 0x7f5db8004800 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd3200640 1 -- 172.21.15.6:0/1642944887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db8004800 msgr2=0x7f5db8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd3200640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db8004800 0x7f5db8004c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc001330 tx=0x7f5db8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd3200640 1 -- 172.21.15.6:0/1642944887 shutdown_connections 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd3200640 1 --2- 172.21.15.6:0/1642944887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5db8004800 0x7f5db8004c80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.762+0000 7f5dd3200640 1 -- 172.21.15.6:0/1642944887 >> 172.21.15.6:0/1642944887 conn(0x7f5dd406b3d0 msgr2=0x7f5dd406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.763+0000 7f5dd3200640 1 -- 172.21.15.6:0/1642944887 shutdown_connections 2024-03-06T07:28:23.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.763+0000 7f5dd3200640 1 -- 172.21.15.6:0/1642944887 wait complete. 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.765+0000 7f5dd3200640 1 Processor -- start 2024-03-06T07:28:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.765+0000 7f5dd3200640 1 -- start start 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db8004800 0x7f5dd4143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd414f570 0x7f5dd415d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 0x7f5dd415d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd406dd30 con 0x7f5dd414f9d0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40b3760 con 0x7f5db8004800 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd406deb0 con 0x7f5dd414f570 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 0x7f5dd415d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 0x7f5dd415d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43706/0 (socket says 172.21.15.6:43706) 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.766+0000 7f5dd2800640 1 -- 172.21.15.6:0/3056946188 learned_addr learned my addr 172.21.15.6:0/3056946188 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd1e00640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db8004800 0x7f5dd4143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dc9400640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd414f570 0x7f5dd415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd2800640 1 -- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd414f570 msgr2=0x7f5dd415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd2800640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd414f570 0x7f5dd415d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd2800640 1 -- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db8004800 msgr2=0x7f5dd4143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd2800640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db8004800 0x7f5dd4143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd2800640 1 -- 172.21.15.6:0/3056946188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd414e4d0 con 0x7f5dd414f9d0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd2800640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 0x7f5dd415d950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40b1770 tx=0x7f5dcc00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc016420 con 0x7f5dd414f9d0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc01b070 con 0x7f5dd414f9d0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.767+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd414e7b0 con 0x7f5dd414f9d0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.768+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc0167c0 con 0x7f5dd414f9d0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.768+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd414ed70 con 0x7f5dd414f9d0 2024-03-06T07:28:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.769+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd404f330 con 0x7f5dd414f9d0 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.770+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5dcc005d10 con 0x7f5dd414f9d0 2024-03-06T07:28:23.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.771+0000 7f5dcaa00640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db006b4d0 0x7f5db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.771+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5dcc08aea0 con 0x7f5dd414f9d0 2024-03-06T07:28:23.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.771+0000 7f5dd1e00640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db006b4d0 0x7f5db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.772+0000 7f5dd1e00640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db006b4d0 0x7f5db006d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4001050 tx=0x7f5dc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.775+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dcc025020 con 0x7f5dd414f9d0 2024-03-06T07:28:23.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.974+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dd40cf4a0 con 0x7f5dd414f9d0 2024-03-06T07:28:23.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.976+0000 7f5dcaa00640 1 -- 172.21.15.6:0/3056946188 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f5dcc059c00 con 0x7f5dd414f9d0 2024-03-06T07:28:23.978 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:23.978 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:23.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.978+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db006b4d0 msgr2=0x7f5db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.978+0000 7f5dd3200640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db006b4d0 0x7f5db006d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4001050 tx=0x7f5dc4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.978+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 msgr2=0x7f5dd415d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.978+0000 7f5dd3200640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 0x7f5dd415d950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40b1770 tx=0x7f5dcc00f520 comp rx=0 tx=0).stop 2024-03-06T07:28:23.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 shutdown_connections 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db006b4d0 0x7f5db006d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd414f9d0 0x7f5dd415d950 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd414f570 0x7f5dd415d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 --2- 172.21.15.6:0/3056946188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5db8004800 0x7f5dd4143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 >> 172.21.15.6:0/3056946188 conn(0x7f5dd406b3d0 msgr2=0x7f5dd40b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 shutdown_connections 2024-03-06T07:28:23.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:23.979+0000 7f5dd3200640 1 -- 172.21.15.6:0/3056946188 wait complete. 2024-03-06T07:28:24.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:27.226348+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:28.693445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:24.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:24.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.226+0000 7f3f4ac00640 1 Processor -- start 2024-03-06T07:28:24.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.226+0000 7f3f4ac00640 1 -- start start 2024-03-06T07:28:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f4ac00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3f440cd5a0 0x7f3f440cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f4ac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f4ac00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4406fb40 con 0x7f3f44144640 2024-03-06T07:28:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f4ac00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4406e460 con 0x7f3f440cd5a0 2024-03-06T07:28:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f4ac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4415f640 con 0x7f3f4406f190 2024-03-06T07:28:24.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f43e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f44144640 0x7f3f440cb850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59560/0 (socket says 172.21.15.6:59560) 2024-03-06T07:28:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f43e00640 1 -- 172.21.15.6:0/2907175524 learned_addr learned my addr 172.21.15.6:0/2907175524 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.228+0000 7f3f43400640 1 --2- 172.21.15.6:0/2907175524 >> v2:172.21.15.178:3300/0 conn(0x7f3f440cd5a0 0x7f3f440cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f48c00640 1 --2- 172.21.15.6:0/2907175524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f42a00640 1 -- 172.21.15.6:0/2907175524 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043916204 0 0) 0x7f3f4406fb40 con 0x7f3f44144640 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f42a00640 1 -- 172.21.15.6:0/2907175524 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f28005730 con 0x7f3f44144640 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f48c00640 1 -- 172.21.15.6:0/2907175524 >> v2:172.21.15.178:3300/0 conn(0x7f3f440cd5a0 msgr2=0x7f3f440cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f48c00640 1 --2- 172.21.15.6:0/2907175524 >> v2:172.21.15.178:3300/0 conn(0x7f3f440cd5a0 0x7f3f440cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f48c00640 1 -- 172.21.15.6:0/2907175524 >> v1:172.21.15.6:6789/0 conn(0x7f3f44144640 legacy=0x7f3f440cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.229+0000 7f3f48c00640 1 -- 172.21.15.6:0/2907175524 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4406e6b0 con 0x7f3f4406f190 2024-03-06T07:28:24.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.230+0000 7f3f48c00640 1 --2- 172.21.15.6:0/2907175524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4406f600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009d90 tx=0x7f3f3c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bbbf0f090a111e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.230+0000 7f3f42a00640 1 -- 172.21.15.6:0/2907175524 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c039070 con 0x7f3f4406f190 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f42a00640 1 -- 172.21.15.6:0/2907175524 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c02cbc0 con 0x7f3f4406f190 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f42a00640 1 -- 172.21.15.6:0/2907175524 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c034440 con 0x7f3f4406f190 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2907175524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 msgr2=0x7f3f4406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2907175524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4406f600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c009d90 tx=0x7f3f3c02be10 comp rx=0 tx=0).stop 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2907175524 shutdown_connections 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2907175524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4406f600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2907175524 >> v2:172.21.15.178:3300/0 conn(0x7f3f440cd5a0 0x7f3f440cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2907175524 >> 172.21.15.6:0/2907175524 conn(0x7f3f440b2860 msgr2=0x7f3f440b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.231+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2907175524 shutdown_connections 2024-03-06T07:28:24.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.232+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2907175524 wait complete. 2024-03-06T07:28:24.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.233+0000 7f3f4ac00640 1 Processor -- start 2024-03-06T07:28:24.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.233+0000 7f3f4ac00640 1 -- start start 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.234+0000 7f3f4ac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4413ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.234+0000 7f3f4ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 0x7f3f4415d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.234+0000 7f3f4ac00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f44144640 0x7f3f4415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.234+0000 7f3f4ac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f44144a20 con 0x7f3f440cd5a0 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.234+0000 7f3f4ac00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4406ec70 con 0x7f3f44144640 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.234+0000 7f3f4ac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440cbf60 con 0x7f3f4406f190 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 0x7f3f4415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 0x7f3f4415d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43714/0 (socket says 172.21.15.6:43714) 2024-03-06T07:28:24.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 -- 172.21.15.6:0/2625768175 learned_addr learned my addr 172.21.15.6:0/2625768175 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43e00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4413ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f48c00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f44144640 0x7f3f4415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 -- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 msgr2=0x7f3f4413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4413ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 -- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f44144640 msgr2=0x7f3f4415d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f44144640 0x7f3f4415d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.235+0000 7f3f43400640 1 -- 172.21.15.6:0/2625768175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4413b1d0 con 0x7f3f440cd5a0 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.236+0000 7f3f43400640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 0x7f3f4415d330 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000e9b0 tx=0x7f3f3000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.236+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30017070 con 0x7f3f440cd5a0 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.236+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3000c9a0 con 0x7f3f440cd5a0 2024-03-06T07:28:24.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.236+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3000cd90 con 0x7f3f440cd5a0 2024-03-06T07:28:24.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.236+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4413b4b0 con 0x7f3f440cd5a0 2024-03-06T07:28:24.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.236+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4413ba70 con 0x7f3f440cd5a0 2024-03-06T07:28:24.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.237+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4404f330 con 0x7f3f440cd5a0 2024-03-06T07:28:24.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.238+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f3001a070 con 0x7f3f440cd5a0 2024-03-06T07:28:24.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.239+0000 7f3f3be00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f2406b4d0 0x7f3f2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.239+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3f3008a520 con 0x7f3f440cd5a0 2024-03-06T07:28:24.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.240+0000 7f3f43e00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f2406b4d0 0x7f3f2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.240+0000 7f3f43e00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f2406b4d0 0x7f3f2406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f34001200 tx=0x7f3f34000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.244+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f30058d60 con 0x7f3f440cd5a0 2024-03-06T07:28:24.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.445+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f4415e1f0 con 0x7f3f440cd5a0 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.446+0000 7f3f3be00640 1 -- 172.21.15.6:0/2625768175 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f3005ce00 con 0x7f3f440cd5a0 2024-03-06T07:28:24.448 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f2406b4d0 msgr2=0x7f3f2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f2406b4d0 0x7f3f2406d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3f34001200 tx=0x7f3f34000bc0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 msgr2=0x7f3f4415d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 0x7f3f4415d330 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000e9b0 tx=0x7f3f3000ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 shutdown_connections 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f2406b4d0 0x7f3f2406d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f44144640 0x7f3f4415d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f440cd5a0 0x7f3f4415d330 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 --2- 172.21.15.6:0/2625768175 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f4406f190 0x7f3f4413ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 >> 172.21.15.6:0/2625768175 conn(0x7f3f440b2860 msgr2=0x7f3f4415e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 shutdown_connections 2024-03-06T07:28:24.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.449+0000 7f3f4ac00640 1 -- 172.21.15.6:0/2625768175 wait complete. 2024-03-06T07:28:24.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:24.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.702+0000 7f95f4200640 1 Processor -- start 2024-03-06T07:28:24.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.702+0000 7f95f4200640 1 -- start start 2024-03-06T07:28:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f4200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f4200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f95ec070ce0 0x7f95ec0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f4200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f95ec0cd010 0x7f95ec06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f4200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec0cfa10 con 0x7f95ec0cd010 2024-03-06T07:28:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f4200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ec071c60 con 0x7f95ec070ce0 2024-03-06T07:28:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f4200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec071f40 con 0x7f95ec15afa0 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f2200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f95ec0cd010 0x7f95ec06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f2200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f95ec0cd010 0x7f95ec06c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43720/0 (socket says 172.21.15.6:43720) 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f2200640 1 -- 172.21.15.6:0/4098125952 learned_addr learned my addr 172.21.15.6:0/4098125952 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f1800640 1 --2- 172.21.15.6:0/4098125952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f0e00640 1 --2- 172.21.15.6:0/4098125952 >> v2:172.21.15.178:3300/0 conn(0x7f95ec070ce0 0x7f95ec0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f2200640 1 -- 172.21.15.6:0/4098125952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 msgr2=0x7f95ec0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f2200640 1 --2- 172.21.15.6:0/4098125952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.704+0000 7f95f2200640 1 -- 172.21.15.6:0/4098125952 >> v2:172.21.15.178:3300/0 conn(0x7f95ec070ce0 msgr2=0x7f95ec0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.705+0000 7f95f2200640 1 --2- 172.21.15.6:0/4098125952 >> v2:172.21.15.178:3300/0 conn(0x7f95ec070ce0 0x7f95ec0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.705+0000 7f95f2200640 1 -- 172.21.15.6:0/4098125952 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec072220 con 0x7f95ec0cd010 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.705+0000 7f95f0e00640 1 --2- 172.21.15.6:0/4098125952 >> v2:172.21.15.178:3300/0 conn(0x7f95ec070ce0 0x7f95ec0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.706+0000 7f95f2200640 1 --2- 172.21.15.6:0/4098125952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec0cd010 0x7f95ec06c980 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f95ec06c050 tx=0x7f95e402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b39ae0c22be7bc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.706+0000 7f95ebe00640 1 -- 172.21.15.6:0/4098125952 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e402c9f0 con 0x7f95ec0cd010 2024-03-06T07:28:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.706+0000 7f95ebe00640 1 -- 172.21.15.6:0/4098125952 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e402cb90 con 0x7f95ec0cd010 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.706+0000 7f95ebe00640 1 -- 172.21.15.6:0/4098125952 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e40343f0 con 0x7f95ec0cd010 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 -- 172.21.15.6:0/4098125952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec0cd010 msgr2=0x7f95ec06c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 --2- 172.21.15.6:0/4098125952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec0cd010 0x7f95ec06c980 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f95ec06c050 tx=0x7f95e402be50 comp rx=0 tx=0).stop 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 -- 172.21.15.6:0/4098125952 shutdown_connections 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 --2- 172.21.15.6:0/4098125952 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec0cd010 0x7f95ec06c980 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 --2- 172.21.15.6:0/4098125952 >> v2:172.21.15.178:3300/0 conn(0x7f95ec070ce0 0x7f95ec0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 --2- 172.21.15.6:0/4098125952 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 -- 172.21.15.6:0/4098125952 >> 172.21.15.6:0/4098125952 conn(0x7f95ec06b3d0 msgr2=0x7f95ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 -- 172.21.15.6:0/4098125952 shutdown_connections 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.707+0000 7f95f4200640 1 -- 172.21.15.6:0/4098125952 wait complete. 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.709+0000 7f95f4200640 1 Processor -- start 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.709+0000 7f95f4200640 1 -- start start 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f4200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 0x7f95ec144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f4200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95ec0cd010 0x7f95ec148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f4200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f4200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec06cec0 con 0x7f95ec070ce0 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f4200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec071140 con 0x7f95ec0cd010 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f4200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ec06d040 con 0x7f95ec15afa0 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 0x7f95ec144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 0x7f95ec144e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43728/0 (socket says 172.21.15.6:43728) 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 -- 172.21.15.6:0/573597433 learned_addr learned my addr 172.21.15.6:0/573597433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f0e00640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95ec0cd010 0x7f95ec148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 -- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 msgr2=0x7f95ec0ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 -- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95ec0cd010 msgr2=0x7f95ec148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95ec0cd010 0x7f95ec148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f1800640 1 -- 172.21.15.6:0/573597433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ec1453a0 con 0x7f95ec070ce0 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.710+0000 7f95f0e00640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95ec0cd010 0x7f95ec148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:24.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.711+0000 7f95f1800640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 0x7f95ec144e60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f95e000c820 tx=0x7f95e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.711+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e0015070 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.711+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95e0004020 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.711+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e00043f0 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.711+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ec148640 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.711+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ec0ad5c0 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.712+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ec04f330 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.713+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95e0007830 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.715+0000 7f95e9600640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95cc06b4d0 0x7f95cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.715+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f95e008a070 con 0x7f95ec070ce0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.716+0000 7f95f0e00640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95cc06b4d0 0x7f95cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.717+0000 7f95f0e00640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95cc06b4d0 0x7f95cc06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f95d8006fd0 tx=0x7f95d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.720+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f95e0058dd0 con 0x7f95ec070ce0 2024-03-06T07:28:24.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.918+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95ec0ccb40 con 0x7f95ec070ce0 2024-03-06T07:28:24.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.919+0000 7f95e9600640 1 -- 172.21.15.6:0/573597433 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95e00586d0 con 0x7f95ec070ce0 2024-03-06T07:28:24.921 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95cc06b4d0 msgr2=0x7f95cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95cc06b4d0 0x7f95cc06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f95d8006fd0 tx=0x7f95d8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 msgr2=0x7f95ec144e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 0x7f95ec144e60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f95e000c820 tx=0x7f95e000cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 shutdown_connections 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95cc06b4d0 0x7f95cc06d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95ec0cd010 0x7f95ec148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f95ec15afa0 0x7f95ec0ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 --2- 172.21.15.6:0/573597433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f95ec070ce0 0x7f95ec144e60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 >> 172.21.15.6:0/573597433 conn(0x7f95ec06b3d0 msgr2=0x7f95ec06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 shutdown_connections 2024-03-06T07:28:24.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:24.922+0000 7f95f4200640 1 -- 172.21.15.6:0/573597433 wait complete. 2024-03-06T07:28:24.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.158+0000 7f84b7e00640 1 Processor -- start 2024-03-06T07:28:25.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.158+0000 7f84b7e00640 1 -- start start 2024-03-06T07:28:25.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.160+0000 7f84b7e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84b8145430 0x7f84b815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.160+0000 7f84b7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b80b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.160+0000 7f84b7e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b815fe40 con 0x7f84b815cc90 2024-03-06T07:28:25.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.160+0000 7f84b7e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b8160120 con 0x7f84b8145430 2024-03-06T07:28:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.160+0000 7f84b7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b8160400 con 0x7f84b80b2250 2024-03-06T07:28:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.161+0000 7f84b7400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b80b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.161+0000 7f84b6a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84b8145430 0x7f84b815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.161+0000 7f84b7400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b80b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:52004/0 (socket says 172.21.15.6:52004) 2024-03-06T07:28:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.161+0000 7f84b7400640 1 -- 172.21.15.6:0/4242418595 learned_addr learned my addr 172.21.15.6:0/4242418595 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.161+0000 7f84b6a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84b8145430 0x7f84b815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:54340/0 (socket says 172.21.15.6:54340) 2024-03-06T07:28:25.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.162+0000 7f84b6a00640 1 -- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 msgr2=0x7f84b80b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.162+0000 7f84b6a00640 1 --2- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b80b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.162+0000 7f84b6a00640 1 -- 172.21.15.6:0/4242418595 >> v1:172.21.15.178:6789/0 conn(0x7f84b815cc90 legacy=0x7f84b80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.162+0000 7f84b6a00640 1 -- 172.21.15.6:0/4242418595 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b81606e0 con 0x7f84b8145430 2024-03-06T07:28:25.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.162+0000 7f84b5600640 1 -- 172.21.15.6:0/4242418595 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016100792 0 0) 0x7f84b815fe40 con 0x7f84b815cc90 2024-03-06T07:28:25.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.164+0000 7f84b6a00640 1 --2- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b8145430 0x7f84b815f810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f84a80097a0 tx=0x7f84a802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43d32d28ab281360 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b5600640 1 -- 172.21.15.6:0/4242418595 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8035020 con 0x7f84b8145430 2024-03-06T07:28:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b5600640 1 -- 172.21.15.6:0/4242418595 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8031830 con 0x7f84b8145430 2024-03-06T07:28:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b5600640 1 -- 172.21.15.6:0/4242418595 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a8031c00 con 0x7f84b8145430 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b7e00640 1 -- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b8145430 msgr2=0x7f84b815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b7e00640 1 --2- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b8145430 0x7f84b815f810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f84a80097a0 tx=0x7f84a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b7e00640 1 -- 172.21.15.6:0/4242418595 shutdown_connections 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b7e00640 1 --2- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b80b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.165+0000 7f84b7e00640 1 --2- 172.21.15.6:0/4242418595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b8145430 0x7f84b815f810 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.166+0000 7f84b7e00640 1 -- 172.21.15.6:0/4242418595 >> 172.21.15.6:0/4242418595 conn(0x7f84b806b3a0 msgr2=0x7f84b806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.166+0000 7f84b7e00640 1 -- 172.21.15.6:0/4242418595 shutdown_connections 2024-03-06T07:28:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.166+0000 7f84b7e00640 1 -- 172.21.15.6:0/4242418595 wait complete. 2024-03-06T07:28:25.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.167+0000 7f84b7e00640 1 Processor -- start 2024-03-06T07:28:25.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.168+0000 7f84b7e00640 1 -- start start 2024-03-06T07:28:25.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.168+0000 7f84b7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b813ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.168+0000 7f84b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 0x7f84b8151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b7e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b815cc90 0x7f84b80af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b8145990 con 0x7f84b8145430 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b7e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b8145810 con 0x7f84b815cc90 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b80cde00 con 0x7f84b80b2250 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 0x7f84b8151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 0x7f84b8151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43732/0 (socket says 172.21.15.6:43732) 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b6000640 1 -- 172.21.15.6:0/631169749 learned_addr learned my addr 172.21.15.6:0/631169749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b6a00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b813ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.169+0000 7f84b7400640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b815cc90 0x7f84b80af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6000640 1 -- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 msgr2=0x7f84b813ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6000640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b813ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6000640 1 -- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b815cc90 msgr2=0x7f84b80af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6000640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b815cc90 0x7f84b80af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6000640 1 -- 172.21.15.6:0/631169749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b813b3a0 con 0x7f84b8145430 2024-03-06T07:28:25.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6a00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b813ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b6000640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 0x7f84b8151070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f84a400c410 tx=0x7f84a400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4017070 con 0x7f84b8145430 2024-03-06T07:28:25.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a40035f0 con 0x7f84b8145430 2024-03-06T07:28:25.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a40039e0 con 0x7f84b8145430 2024-03-06T07:28:25.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.170+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b8151c50 con 0x7f84b8145430 2024-03-06T07:28:25.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.171+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b8152210 con 0x7f84b8145430 2024-03-06T07:28:25.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.172+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b804f330 con 0x7f84b8145430 2024-03-06T07:28:25.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.173+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84a401a070 con 0x7f84b8145430 2024-03-06T07:28:25.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.175+0000 7f84aea00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f849806b4d0 0x7f849806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.175+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f84a40896b0 con 0x7f84b8145430 2024-03-06T07:28:25.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.175+0000 7f84b6a00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f849806b4d0 0x7f849806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.177+0000 7f84b6a00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f849806b4d0 0x7f849806d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84a8036000 tx=0x7f84a803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.179+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84b813b3a0 con 0x7f84b8145430 2024-03-06T07:28:25.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.383+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84b806e240 con 0x7f84b8145430 2024-03-06T07:28:25.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.384+0000 7f84aea00640 1 -- 172.21.15.6:0/631169749 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84a40584a0 con 0x7f84b8145430 2024-03-06T07:28:25.386 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.386+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f849806b4d0 msgr2=0x7f849806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.386+0000 7f84b7e00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f849806b4d0 0x7f849806d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84a8036000 tx=0x7f84a803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.386+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 msgr2=0x7f84b8151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.386+0000 7f84b7e00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 0x7f84b8151070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f84a400c410 tx=0x7f84a400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 shutdown_connections 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f849806b4d0 0x7f849806d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84b815cc90 0x7f84b80af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84b8145430 0x7f84b8151070 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 --2- 172.21.15.6:0/631169749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84b80b2250 0x7f84b813ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 >> 172.21.15.6:0/631169749 conn(0x7f84b806b3a0 msgr2=0x7f84b80cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 shutdown_connections 2024-03-06T07:28:25.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.387+0000 7f84b7e00640 1 -- 172.21.15.6:0/631169749 wait complete. 2024-03-06T07:28:25.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.599+0000 7f382be00640 1 Processor -- start 2024-03-06T07:28:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.600+0000 7f382be00640 1 -- start start 2024-03-06T07:28:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.601+0000 7f382be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 0x7f382c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.601+0000 7f382be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382c0cd0c0 con 0x7f382c15f450 2024-03-06T07:28:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.601+0000 7f382be00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382c0cd3a0 con 0x7f382c0d04b0 2024-03-06T07:28:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.601+0000 7f382be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c0d29e0 con 0x7f382c0700c0 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.601+0000 7f382aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f382c15f450 0x7f382c0cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59570/0 (socket says 172.21.15.6:59570) 2024-03-06T07:28:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.601+0000 7f382aa00640 1 -- 172.21.15.6:0/3689576892 learned_addr learned my addr 172.21.15.6:0/3689576892 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057912724 0 0) 0x7f382c0cd0c0 con 0x7f382c15f450 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c003520 con 0x7f382c15f450 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f382b400640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 0x7f382c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3736419393 0 0) 0x7f380c003520 con 0x7f382c15f450 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 msgr2=0x7f382c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 0x7f382c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 >> v1:172.21.15.178:6789/0 conn(0x7f382c0d04b0 legacy=0x7f382c0cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c0cd680 con 0x7f382c15f450 2024-03-06T07:28:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f381c001670 con 0x7f382c15f450 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 >> v1:172.21.15.6:6789/0 conn(0x7f382c15f450 legacy=0x7f382c0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f380c0039e0 0x7f380c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f380c004800 0x7f380c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 0x7f380c005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c0d29e0 con 0x7f380c004800 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c00bb50 con 0x7f380c0039e0 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.602+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c00bd50 con 0x7f382c0700c0 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382a000640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f380c004800 0x7f380c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382aa00640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f380c0039e0 0x7f380c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382a000640 1 -- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 msgr2=0x7f380c005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382a000640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382c0700c0 0x7f380c005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382a000640 1 -- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f380c0039e0 msgr2=0x7f380c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382a000640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f380c0039e0 0x7f380c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.603+0000 7f382a000640 1 -- 172.21.15.6:0/3689576892 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c003520 con 0x7f380c004800 2024-03-06T07:28:25.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.604+0000 7f382a000640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f380c004800 0x7f380c004c80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3814001e50 tx=0x7f380c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdf8a7d347c4dfc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.604+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814001a70 con 0x7f380c004800 2024-03-06T07:28:25.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.604+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3814003b70 con 0x7f380c004800 2024-03-06T07:28:25.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.604+0000 7f3829600640 1 -- 172.21.15.6:0/3689576892 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38140050d0 con 0x7f380c004800 2024-03-06T07:28:25.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382be00640 1 -- 172.21.15.6:0/3689576892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f380c004800 msgr2=0x7f380c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382be00640 1 --2- 172.21.15.6:0/3689576892 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f380c004800 0x7f380c004c80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3814001e50 tx=0x7f380c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382b400640 1 -- 172.21.15.6:0/3689576892 reap_dead start 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382be00640 1 -- 172.21.15.6:0/3689576892 shutdown_connections 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382be00640 1 -- 172.21.15.6:0/3689576892 >> 172.21.15.6:0/3689576892 conn(0x7f382c0d1ae0 msgr2=0x7f382c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382be00640 1 -- 172.21.15.6:0/3689576892 shutdown_connections 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.605+0000 7f382be00640 1 -- 172.21.15.6:0/3689576892 wait complete. 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.607+0000 7f382be00640 1 Processor -- start 2024-03-06T07:28:25.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.607+0000 7f382be00640 1 -- start start 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f380c004800 0x7f382c0b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f382c0b6ee0 0x7f382c0c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 0x7f382c1f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c15f830 con 0x7f382c0c66a0 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c0ae8b0 con 0x7f382c0b6ee0 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c15f9b0 con 0x7f380c004800 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 0x7f382c1f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 0x7f382c1f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43752/0 (socket says 172.21.15.6:43752) 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 -- 172.21.15.6:0/2573043480 learned_addr learned my addr 172.21.15.6:0/2573043480 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382aa00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f380c004800 0x7f382c0b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 -- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f380c004800 msgr2=0x7f382c0b69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f380c004800 0x7f382c0b69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 -- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f382c0b6ee0 msgr2=0x7f382c0c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382a000640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f382c0b6ee0 0x7f382c0c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f382c0b6ee0 0x7f382c0c6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.608+0000 7f382b400640 1 -- 172.21.15.6:0/2573043480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c14f880 con 0x7f382c0c66a0 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.609+0000 7f382b400640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 0x7f382c1f6e50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f382c071e00 tx=0x7f382400d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.609+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824004030 con 0x7f382c0c66a0 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.609+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38240041f0 con 0x7f382c0c66a0 2024-03-06T07:28:25.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.609+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3824003610 con 0x7f382c0c66a0 2024-03-06T07:28:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.609+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382c14fb60 con 0x7f382c0c66a0 2024-03-06T07:28:25.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.609+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382c150120 con 0x7f382c0c66a0 2024-03-06T07:28:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.610+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382c04f330 con 0x7f382c0c66a0 2024-03-06T07:28:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.611+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f38240037b0 con 0x7f382c0c66a0 2024-03-06T07:28:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.612+0000 7f3823400640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b500 0x7f380806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.612+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f382408a0e0 con 0x7f382c0c66a0 2024-03-06T07:28:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.612+0000 7f382aa00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b500 0x7f380806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.614+0000 7f382aa00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b500 0x7f380806d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f381c001eb0 tx=0x7f381c000f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.616+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f382401b300 con 0x7f382c0c66a0 2024-03-06T07:28:25.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.854+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f382c0b72c0 con 0x7f382c0c66a0 2024-03-06T07:28:25.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.855+0000 7f3823400640 1 -- 172.21.15.6:0/2573043480 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f3824058ed0 con 0x7f382c0c66a0 2024-03-06T07:28:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:25.857 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:28:25.857 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.857+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b500 msgr2=0x7f380806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.857+0000 7f382be00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b500 0x7f380806d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f381c001eb0 tx=0x7f381c000f10 comp rx=0 tx=0).stop 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.857+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 msgr2=0x7f382c1f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.857+0000 7f382be00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 0x7f382c1f6e50 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f382c071e00 tx=0x7f382400d8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 shutdown_connections 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b500 0x7f380806d9c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f382c0b6ee0 0x7f382c0c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f382c0c66a0 0x7f382c1f6e50 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 --2- 172.21.15.6:0/2573043480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f380c004800 0x7f382c0b69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 >> 172.21.15.6:0/2573043480 conn(0x7f382c0d1ae0 msgr2=0x7f382c0d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 shutdown_connections 2024-03-06T07:28:25.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:25.858+0000 7f382be00640 1 -- 172.21.15.6:0/2573043480 wait complete. 2024-03-06T07:28:25.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:26.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.095+0000 7faf76400640 1 Processor -- start 2024-03-06T07:28:26.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.096+0000 7faf76400640 1 -- start start 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.097+0000 7faf76400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf70072da0 0x7faf70162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.097+0000 7faf76400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf700712c0 0x7faf7006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.097+0000 7faf76400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf700ce090 con 0x7faf70070ee0 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf76400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf700ceca0 con 0x7faf70072da0 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf76400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700ce370 con 0x7faf700712c0 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faf70070ee0 0x7faf7006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59572/0 (socket says 172.21.15.6:59572) 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6f400640 1 -- 172.21.15.6:0/1283428836 learned_addr learned my addr 172.21.15.6:0/1283428836 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:26.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6ea00640 1 --2- 172.21.15.6:0/1283428836 >> v2:172.21.15.6:3300/0 conn(0x7faf70072da0 0x7faf70162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6fe00640 1 --2- 172.21.15.6:0/1283428836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf700712c0 0x7faf7006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6ea00640 1 -- 172.21.15.6:0/1283428836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf700712c0 msgr2=0x7faf7006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6ea00640 1 --2- 172.21.15.6:0/1283428836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf700712c0 0x7faf7006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6ea00640 1 -- 172.21.15.6:0/1283428836 >> v1:172.21.15.6:6789/0 conn(0x7faf70070ee0 legacy=0x7faf7006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6e000640 1 -- 172.21.15.6:0/1283428836 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698176503 0 0) 0x7faf700ce090 con 0x7faf70070ee0 2024-03-06T07:28:26.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.098+0000 7faf6ea00640 1 -- 172.21.15.6:0/1283428836 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700ce650 con 0x7faf70072da0 2024-03-06T07:28:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.100+0000 7faf6ea00640 1 --2- 172.21.15.6:0/1283428836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70072da0 0x7faf70162c40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faf700cee20 tx=0x7faf5c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bbf0b5f28370e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.100+0000 7faf6e000640 1 -- 172.21.15.6:0/1283428836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c039070 con 0x7faf70072da0 2024-03-06T07:28:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.100+0000 7faf6e000640 1 -- 172.21.15.6:0/1283428836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c02cc40 con 0x7faf70072da0 2024-03-06T07:28:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.100+0000 7faf6e000640 1 -- 172.21.15.6:0/1283428836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c034440 con 0x7faf70072da0 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.100+0000 7faf76400640 1 -- 172.21.15.6:0/1283428836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70072da0 msgr2=0x7faf70162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.100+0000 7faf76400640 1 --2- 172.21.15.6:0/1283428836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70072da0 0x7faf70162c40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7faf700cee20 tx=0x7faf5c02be90 comp rx=0 tx=0).stop 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.101+0000 7faf76400640 1 -- 172.21.15.6:0/1283428836 shutdown_connections 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.101+0000 7faf76400640 1 --2- 172.21.15.6:0/1283428836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf700712c0 0x7faf7006e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.101+0000 7faf76400640 1 --2- 172.21.15.6:0/1283428836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf70072da0 0x7faf70162c40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.101+0000 7faf76400640 1 -- 172.21.15.6:0/1283428836 >> 172.21.15.6:0/1283428836 conn(0x7faf7006afb0 msgr2=0x7faf7006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.101+0000 7faf76400640 1 -- 172.21.15.6:0/1283428836 shutdown_connections 2024-03-06T07:28:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.101+0000 7faf76400640 1 -- 172.21.15.6:0/1283428836 wait complete. 2024-03-06T07:28:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.102+0000 7faf76400640 1 Processor -- start 2024-03-06T07:28:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.103+0000 7faf76400640 1 -- start start 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.103+0000 7faf76400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf70070ee0 0x7faf70152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf76400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 0x7faf700ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf76400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf70072da0 0x7faf700ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf76400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700cdda0 con 0x7faf700712c0 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf76400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700cdc20 con 0x7faf70072da0 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf76400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700cf130 con 0x7faf70070ee0 2024-03-06T07:28:26.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 0x7faf700ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 0x7faf700ae6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43760/0 (socket says 172.21.15.6:43760) 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 -- 172.21.15.6:0/1302544462 learned_addr learned my addr 172.21.15.6:0/1302544462 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6f400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf70070ee0 0x7faf70152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6fe00640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf70072da0 0x7faf700ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 -- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf70070ee0 msgr2=0x7faf70152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf70070ee0 0x7faf70152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 -- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf70072da0 msgr2=0x7faf700ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf70072da0 0x7faf700ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6ea00640 1 -- 172.21.15.6:0/1302544462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf701532c0 con 0x7faf700712c0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.104+0000 7faf6fe00640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf70072da0 0x7faf700ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.105+0000 7faf6ea00640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 0x7faf700ae6f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faf5c037040 tx=0x7faf5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.105+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c039050 con 0x7faf700712c0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.105+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c045070 con 0x7faf700712c0 2024-03-06T07:28:26.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.105+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c009450 con 0x7faf700712c0 2024-03-06T07:28:26.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.105+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf700aef40 con 0x7faf700712c0 2024-03-06T07:28:26.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.105+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf700ace50 con 0x7faf700712c0 2024-03-06T07:28:26.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.106+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7004f3a0 con 0x7faf700712c0 2024-03-06T07:28:26.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.107+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf5c03e850 con 0x7faf700712c0 2024-03-06T07:28:26.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.108+0000 7faf67400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5006b4d0 0x7faf5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.108+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faf5c0ace30 con 0x7faf700712c0 2024-03-06T07:28:26.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.109+0000 7faf6f400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5006b4d0 0x7faf5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.110+0000 7faf6f400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5006b4d0 0x7faf5006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf60000f60 tx=0x7faf60007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.113+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf5c032e20 con 0x7faf700712c0 2024-03-06T07:28:26.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.318+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf700b12c0 con 0x7faf700712c0 2024-03-06T07:28:26.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.321+0000 7faf67400640 1 -- 172.21.15.6:0/1302544462 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7faf5c07bc20 con 0x7faf700712c0 2024-03-06T07:28:26.324 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:26.324 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5006b4d0 msgr2=0x7faf5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5006b4d0 0x7faf5006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf60000f60 tx=0x7faf60007b50 comp rx=0 tx=0).stop 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 msgr2=0x7faf700ae6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 0x7faf700ae6f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faf5c037040 tx=0x7faf5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 shutdown_connections 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5006b4d0 0x7faf5006d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf70072da0 0x7faf700ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf700712c0 0x7faf700ae6f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 --2- 172.21.15.6:0/1302544462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf70070ee0 0x7faf70152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 >> 172.21.15.6:0/1302544462 conn(0x7faf7006afb0 msgr2=0x7faf7006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 shutdown_connections 2024-03-06T07:28:26.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.324+0000 7faf76400640 1 -- 172.21.15.6:0/1302544462 wait complete. 2024-03-06T07:28:26.346 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:26.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.563+0000 7f7367200640 1 Processor -- start 2024-03-06T07:28:26.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.563+0000 7f7367200640 1 -- start start 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f7367200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f73680d0e50 0x7f7368168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f7367200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73680b3710 con 0x7f73680d4a10 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f7367200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73680b39f0 con 0x7f736815e0b0 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f7367200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73680b09e0 con 0x7f73680d0e50 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f735d400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73680d4a10 0x7f73680d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59576/0 (socket says 172.21.15.6:59576) 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f735d400640 1 -- 172.21.15.6:0/3402615529 learned_addr learned my addr 172.21.15.6:0/3402615529 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.565+0000 7f7366800640 1 --2- 172.21.15.6:0/3402615529 >> v2:172.21.15.6:3300/0 conn(0x7f73680d0e50 0x7f7368168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.566+0000 7f7365400640 1 -- 172.21.15.6:0/3402615529 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751199184 0 0) 0x7f73680b3710 con 0x7f73680d4a10 2024-03-06T07:28:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.566+0000 7f7365400640 1 -- 172.21.15.6:0/3402615529 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c003520 con 0x7f73680d4a10 2024-03-06T07:28:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.566+0000 7f7366800640 1 -- 172.21.15.6:0/3402615529 >> v1:172.21.15.178:6789/0 conn(0x7f736815e0b0 legacy=0x7f73680cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.566+0000 7f7366800640 1 -- 172.21.15.6:0/3402615529 >> v1:172.21.15.6:6789/0 conn(0x7f73680d4a10 legacy=0x7f73680d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.566+0000 7f7366800640 1 -- 172.21.15.6:0/3402615529 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73680b3cd0 con 0x7f73680d0e50 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.567+0000 7f7366800640 1 --2- 172.21.15.6:0/3402615529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d0e50 0x7f7368168c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7360009d90 tx=0x7f736002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9c203ec7a7332 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.567+0000 7f7365400640 1 -- 172.21.15.6:0/3402615529 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360039070 con 0x7f73680d0e50 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7365400640 1 -- 172.21.15.6:0/3402615529 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736002ccf0 con 0x7f73680d0e50 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7365400640 1 -- 172.21.15.6:0/3402615529 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360034660 con 0x7f73680d0e50 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7367200640 1 -- 172.21.15.6:0/3402615529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d0e50 msgr2=0x7f7368168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7367200640 1 --2- 172.21.15.6:0/3402615529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d0e50 0x7f7368168c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7360009d90 tx=0x7f736002bf40 comp rx=0 tx=0).stop 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7367200640 1 -- 172.21.15.6:0/3402615529 shutdown_connections 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7367200640 1 --2- 172.21.15.6:0/3402615529 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d0e50 0x7f7368168c50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.568+0000 7f7367200640 1 -- 172.21.15.6:0/3402615529 >> 172.21.15.6:0/3402615529 conn(0x7f73680d3d30 msgr2=0x7f73680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.569+0000 7f7367200640 1 -- 172.21.15.6:0/3402615529 shutdown_connections 2024-03-06T07:28:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.569+0000 7f7367200640 1 -- 172.21.15.6:0/3402615529 wait complete. 2024-03-06T07:28:26.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.570+0000 7f7367200640 1 Processor -- start 2024-03-06T07:28:26.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.570+0000 7f7367200640 1 -- start start 2024-03-06T07:28:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.571+0000 7f7367200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73680d0e50 0x7f73681531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.571+0000 7f7367200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 0x7f73681f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.571+0000 7f7367200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f736815e0b0 0x7f73680cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f7367200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736815e490 con 0x7f73680d4a10 2024-03-06T07:28:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f7367200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680cfa20 con 0x7f73680d0e50 2024-03-06T07:28:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f7367200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680cfc60 con 0x7f736815e0b0 2024-03-06T07:28:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 0x7f73681f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 0x7f73681f31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43770/0 (socket says 172.21.15.6:43770) 2024-03-06T07:28:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 -- 172.21.15.6:0/1075110106 learned_addr learned my addr 172.21.15.6:0/1075110106 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f7365e00640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73680d0e50 0x7f73681531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 -- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f736815e0b0 msgr2=0x7f73680cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f736815e0b0 0x7f73680cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 -- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73680d0e50 msgr2=0x7f73681531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73680d0e50 0x7f73681531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.572+0000 7f735d400640 1 -- 172.21.15.6:0/1075110106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73681f3700 con 0x7f73680d4a10 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.573+0000 7f735d400640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 0x7f73681f31c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f735000c4d0 tx=0x7f735000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.573+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350017070 con 0x7f73680d4a10 2024-03-06T07:28:26.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.573+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7350005ce0 con 0x7f73680d4a10 2024-03-06T07:28:26.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.573+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7350007270 con 0x7f73680d4a10 2024-03-06T07:28:26.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.573+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73680cb340 con 0x7f73680d4a10 2024-03-06T07:28:26.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.573+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73680cb8c0 con 0x7f73680d4a10 2024-03-06T07:28:26.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.574+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736804f330 con 0x7f73680d4a10 2024-03-06T07:28:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.575+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f735001a070 con 0x7f73680d4a10 2024-03-06T07:28:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.576+0000 7f735ea00640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f734406b4d0 0x7f734406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.576+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7350089530 con 0x7f73680d4a10 2024-03-06T07:28:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.577+0000 7f7365e00640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f734406b4d0 0x7f734406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.578+0000 7f7365e00640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f734406b4d0 0x7f734406d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7358000f60 tx=0x7f7358009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.580+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73681f3700 con 0x7f73680d4a10 2024-03-06T07:28:26.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.821+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73680b61e0 con 0x7f73680d4a10 2024-03-06T07:28:26.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.822+0000 7f735ea00640 1 -- 172.21.15.6:0/1075110106 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f7350057cd0 con 0x7f73680d4a10 2024-03-06T07:28:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:26.824 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:28:26.825 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f734406b4d0 msgr2=0x7f734406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f734406b4d0 0x7f734406d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7358000f60 tx=0x7f7358009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 msgr2=0x7f73681f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 0x7f73681f31c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f735000c4d0 tx=0x7f735000c990 comp rx=0 tx=0).stop 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 shutdown_connections 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f734406b4d0 0x7f734406d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f736815e0b0 0x7f73680cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73680d4a10 0x7f73681f31c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 --2- 172.21.15.6:0/1075110106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73680d0e50 0x7f73681531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 >> 172.21.15.6:0/1075110106 conn(0x7f73680d3d30 msgr2=0x7f73680cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 shutdown_connections 2024-03-06T07:28:26.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:26.825+0000 7f7367200640 1 -- 172.21.15.6:0/1075110106 wait complete. 2024-03-06T07:28:26.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:27.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.065+0000 7f5425a00640 1 Processor -- start 2024-03-06T07:28:27.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.065+0000 7f5425a00640 1 -- start start 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f5425a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 0x7f54200d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f5425a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54200d0c80 0x7f54200b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f5425a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54200b3710 con 0x7f542015e0b0 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f5425a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54200b09e0 con 0x7f54200d0c80 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f5425a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54200cfa20 con 0x7f54200d4a10 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f541fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f542015e0b0 0x7f54200cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59592/0 (socket says 172.21.15.6:59592) 2024-03-06T07:28:27.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.067+0000 7f541fe00640 1 -- 172.21.15.6:0/2404992652 learned_addr learned my addr 172.21.15.6:0/2404992652 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:27.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541f400640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 0x7f54200d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f5425000640 1 --2- 172.21.15.6:0/2404992652 >> v2:172.21.15.178:3300/0 conn(0x7f54200d0c80 0x7f54200b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860091091 0 0) 0x7f54200b3710 con 0x7f542015e0b0 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5404003520 con 0x7f542015e0b0 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223185398 0 0) 0x7f5404003520 con 0x7f542015e0b0 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 msgr2=0x7f54200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 0x7f54200d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 >> v2:172.21.15.178:3300/0 conn(0x7f54200d0c80 msgr2=0x7f54200b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 --2- 172.21.15.6:0/2404992652 >> v2:172.21.15.178:3300/0 conn(0x7f54200d0c80 0x7f54200b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200b39f0 con 0x7f542015e0b0 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5410001670 con 0x7f542015e0b0 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 >> v1:172.21.15.6:6789/0 conn(0x7f542015e0b0 legacy=0x7f54200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54040039b0 0x7f5404003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 0x7f5404004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.068+0000 7f541ea00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5404004d40 0x7f54040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200cfa20 con 0x7f5404004d40 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540400b9c0 con 0x7f54040039b0 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540400bb40 con 0x7f54200d4a10 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f5425000640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5404004d40 0x7f54040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f541fe00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54040039b0 0x7f5404003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f541f400640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 0x7f5404004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f5425000640 1 -- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 msgr2=0x7f5404004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f5425000640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54200d4a10 0x7f5404004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f5425000640 1 -- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54040039b0 msgr2=0x7f5404003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f5425000640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54040039b0 0x7f5404003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f5425000640 1 -- 172.21.15.6:0/2404992652 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404003520 con 0x7f5404004d40 2024-03-06T07:28:27.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.069+0000 7f541fe00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54040039b0 0x7f5404003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f5425000640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5404004d40 0x7f54040051c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5418009fc0 tx=0x7f5404005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1d0efcfd15938a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418039070 con 0x7f5404004d40 2024-03-06T07:28:27.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541802cc60 con 0x7f5404004d40 2024-03-06T07:28:27.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f541ea00640 1 -- 172.21.15.6:0/2404992652 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418009c10 con 0x7f5404004d40 2024-03-06T07:28:27.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f5425a00640 1 -- 172.21.15.6:0/2404992652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5404004d40 msgr2=0x7f54040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f5425a00640 1 --2- 172.21.15.6:0/2404992652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5404004d40 0x7f54040051c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5418009fc0 tx=0x7f5404005830 comp rx=0 tx=0).stop 2024-03-06T07:28:27.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f5425000640 1 -- 172.21.15.6:0/2404992652 reap_dead start 2024-03-06T07:28:27.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f5425a00640 1 -- 172.21.15.6:0/2404992652 shutdown_connections 2024-03-06T07:28:27.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.071+0000 7f5425a00640 1 -- 172.21.15.6:0/2404992652 >> 172.21.15.6:0/2404992652 conn(0x7f54200d3d30 msgr2=0x7f54200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.072+0000 7f5425a00640 1 -- 172.21.15.6:0/2404992652 shutdown_connections 2024-03-06T07:28:27.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.072+0000 7f5425a00640 1 -- 172.21.15.6:0/2404992652 wait complete. 2024-03-06T07:28:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.074+0000 7f5425a00640 1 Processor -- start 2024-03-06T07:28:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.074+0000 7f5425a00640 1 -- start start 2024-03-06T07:28:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5404004d40 0x7f54201f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5420147920 0x7f54200b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 0x7f54200cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542015e490 con 0x7f54200b1d60 2024-03-06T07:28:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54200b3fb0 con 0x7f5420147920 2024-03-06T07:28:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54200cfc60 con 0x7f5404004d40 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.075+0000 7f5425000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 0x7f54200cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 0x7f54200cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43788/0 (socket says 172.21.15.6:43788) 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 -- 172.21.15.6:0/2441843532 learned_addr learned my addr 172.21.15.6:0/2441843532 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f541fe00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5404004d40 0x7f54201f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f541f400640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5420147920 0x7f54200b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 -- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5404004d40 msgr2=0x7f54201f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5404004d40 0x7f54201f7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 -- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5420147920 msgr2=0x7f54200b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5420147920 0x7f54200b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 -- 172.21.15.6:0/2441843532 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54200b25e0 con 0x7f54200b1d60 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f541f400640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5420147920 0x7f54200b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.076+0000 7f5425000640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 0x7f54200cd0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f541800e230 tx=0x7f5418009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.077+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418039070 con 0x7f54200b1d60 2024-03-06T07:28:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.077+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541803a040 con 0x7f54200b1d60 2024-03-06T07:28:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.077+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418003bd0 con 0x7f54200b1d60 2024-03-06T07:28:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.077+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54200cc600 con 0x7f54200b1d60 2024-03-06T07:28:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.077+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54200cc2d0 con 0x7f54200b1d60 2024-03-06T07:28:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.078+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542004f330 con 0x7f54200b1d60 2024-03-06T07:28:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.079+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5418004030 con 0x7f54200b1d60 2024-03-06T07:28:27.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.080+0000 7f5417e00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.081+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f54180ac7c0 con 0x7f54200b1d60 2024-03-06T07:28:27.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.081+0000 7f541fe00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.082+0000 7f541fe00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540006b4d0 0x7f540006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5410000c00 tx=0x7f5410001ba0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.085+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f541807b5b0 con 0x7f54200b1d60 2024-03-06T07:28:27.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.324+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54200cf3f0 con 0x7f54200b1d60 2024-03-06T07:28:27.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.325+0000 7f5417e00640 1 -- 172.21.15.6:0/2441843532 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4971 (secure 0 0 0) 0x7f541807aeb0 con 0x7f54200b1d60 2024-03-06T07:28:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:27.327 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4279,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1756235741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":1756235741},{"type":"v1","addr":"172.21.15.6:6835","nonce":1756235741}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4281,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/2536798883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2536798883},{"type":"v1","addr":"172.21.15.6:6834","nonce":2536798883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4291,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3503522622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3503522622},{"type":"v1","addr":"172.21.15.178:6835","nonce":3503522622}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:29.674138+0000","modified":"2024-03-06T07:27:32.753541+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/4203957884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":4203957884},{"type":"v1","addr":"172.21.15.178:6837","nonce":4203957884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-03-06T07:28:27.328 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 8 2024-03-06T07:28:27.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540006b4d0 msgr2=0x7f540006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540006b4d0 0x7f540006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5410000c00 tx=0x7f5410001ba0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 msgr2=0x7f54200cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 0x7f54200cd0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f541800e230 tx=0x7f5418009750 comp rx=0 tx=0).stop 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 shutdown_connections 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540006b4d0 0x7f540006d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5420147920 0x7f54200b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54200b1d60 0x7f54200cd0e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 --2- 172.21.15.6:0/2441843532 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5404004d40 0x7f54201f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.327+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 >> 172.21.15.6:0/2441843532 conn(0x7f54200d3d30 msgr2=0x7f542015d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.328+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 shutdown_connections 2024-03-06T07:28:27.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.328+0000 7f5425a00640 1 -- 172.21.15.6:0/2441843532 wait complete. 2024-03-06T07:28:27.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:27.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.567+0000 7f8a1a800640 1 Processor -- start 2024-03-06T07:28:27.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.568+0000 7f8a1a800640 1 -- start start 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a1a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a140af1a0 0x7f8a1415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a1a800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a140b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a1a800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8a140cdc20 0x7f8a140b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a1a800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1415cc40 con 0x7f8a140af1a0 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a1a800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1415cf20 con 0x7f8a140cdc20 2024-03-06T07:28:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a1a800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1415d200 con 0x7f8a140af580 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a13400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a140af1a0 0x7f8a1415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a13400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8a140af1a0 0x7f8a1415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43800/0 (socket says 172.21.15.6:43800) 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.569+0000 7f8a13400640 1 -- 172.21.15.6:0/902845511 learned_addr learned my addr 172.21.15.6:0/902845511 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.570+0000 7f8a13400640 1 -- 172.21.15.6:0/902845511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 msgr2=0x7f8a140b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.570+0000 7f8a13400640 1 --2- 172.21.15.6:0/902845511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a140b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.570+0000 7f8a13400640 1 -- 172.21.15.6:0/902845511 >> v2:172.21.15.178:3300/0 conn(0x7f8a140cdc20 msgr2=0x7f8a140b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.570+0000 7f8a13400640 1 --2- 172.21.15.6:0/902845511 >> v2:172.21.15.178:3300/0 conn(0x7f8a140cdc20 0x7f8a140b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.570+0000 7f8a13400640 1 -- 172.21.15.6:0/902845511 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1415d4e0 con 0x7f8a140af1a0 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.571+0000 7f8a13400640 1 --2- 172.21.15.6:0/902845511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a1415c600 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8a140cda30 tx=0x7f8a0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c5c7254a1a005 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.571+0000 7f8a12000640 1 -- 172.21.15.6:0/902845511 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0402e070 con 0x7f8a140af1a0 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.571+0000 7f8a12000640 1 -- 172.21.15.6:0/902845511 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04037070 con 0x7f8a140af1a0 2024-03-06T07:28:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.571+0000 7f8a12000640 1 -- 172.21.15.6:0/902845511 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04031a00 con 0x7f8a140af1a0 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 -- 172.21.15.6:0/902845511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 msgr2=0x7f8a1415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 --2- 172.21.15.6:0/902845511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a1415c600 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8a140cda30 tx=0x7f8a0402b880 comp rx=0 tx=0).stop 2024-03-06T07:28:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 -- 172.21.15.6:0/902845511 shutdown_connections 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 --2- 172.21.15.6:0/902845511 >> v2:172.21.15.178:3300/0 conn(0x7f8a140cdc20 0x7f8a140b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 --2- 172.21.15.6:0/902845511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a140b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 --2- 172.21.15.6:0/902845511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a1415c600 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 -- 172.21.15.6:0/902845511 >> 172.21.15.6:0/902845511 conn(0x7f8a1406afb0 msgr2=0x7f8a1406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 -- 172.21.15.6:0/902845511 shutdown_connections 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.572+0000 7f8a1a800640 1 -- 172.21.15.6:0/902845511 wait complete. 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.574+0000 7f8a1a800640 1 Processor -- start 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.574+0000 7f8a1a800640 1 -- start start 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a14152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a1a800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a1409b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a1a800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a140cdc20 0x7f8a140cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a1a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140b3490 con 0x7f8a140af1a0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a1a800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140b3310 con 0x7f8a140cdc20 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a1a800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a140b3610 con 0x7f8a140af580 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a13400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a14152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a13400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a14152630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43814/0 (socket says 172.21.15.6:43814) 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.575+0000 7f8a13400640 1 -- 172.21.15.6:0/1975332507 learned_addr learned my addr 172.21.15.6:0/1975332507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a12a00640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a1409b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13e00640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a140cdc20 0x7f8a140cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13400640 1 -- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 msgr2=0x7f8a1409b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13400640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a1409b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13400640 1 -- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a140cdc20 msgr2=0x7f8a140cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13400640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a140cdc20 0x7f8a140cc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13400640 1 -- 172.21.15.6:0/1975332507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a140cada0 con 0x7f8a140af1a0 2024-03-06T07:28:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13e00640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a140cdc20 0x7f8a140cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.576+0000 7f8a13400640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a14152630 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8a0402b8c0 tx=0x7f8a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.577+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04032880 con 0x7f8a140af1a0 2024-03-06T07:28:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.577+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04032a20 con 0x7f8a140af1a0 2024-03-06T07:28:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.577+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04032e10 con 0x7f8a140af1a0 2024-03-06T07:28:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.577+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a140cb080 con 0x7f8a140af1a0 2024-03-06T07:28:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.577+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a14152110 con 0x7f8a140af1a0 2024-03-06T07:28:27.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.578+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1404f3a0 con 0x7f8a140af1a0 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.579+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a040453f0 con 0x7f8a140af1a0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.580+0000 7f8a0b400640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89f006b4d0 0x7f89f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.580+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8a040ac320 con 0x7f8a140af1a0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.580+0000 7f8a12a00640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89f006b4d0 0x7f89f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.582+0000 7f8a12a00640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89f006b4d0 0x7f89f006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a141f74a0 tx=0x7f89fc0094f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.584+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a04040070 con 0x7f8a140af1a0 2024-03-06T07:28:27.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.786+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a140ad1c0 con 0x7f8a140af1a0 2024-03-06T07:28:27.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.790+0000 7f8a0b400640 1 -- 172.21.15.6:0/1975332507 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16783 (secure 0 0 0) 0x7f8a0407b1a0 con 0x7f8a140af1a0 2024-03-06T07:28:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:27.792 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":24,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:10.589654+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:27.226348+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:28.693445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:27.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89f006b4d0 msgr2=0x7f89f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89f006b4d0 0x7f89f006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8a141f74a0 tx=0x7f89fc0094f0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 msgr2=0x7f8a14152630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a14152630 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8a0402b8c0 tx=0x7f8a040046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 shutdown_connections 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89f006b4d0 0x7f89f006d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a140cdc20 0x7f8a140cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a140af580 0x7f8a1409b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 --2- 172.21.15.6:0/1975332507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a140af1a0 0x7f8a14152630 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 >> 172.21.15.6:0/1975332507 conn(0x7f8a1406afb0 msgr2=0x7f8a140ada30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 shutdown_connections 2024-03-06T07:28:27.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:27.793+0000 7f8a1a800640 1 -- 172.21.15.6:0/1975332507 wait complete. 2024-03-06T07:28:27.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.029+0000 7f5d14400640 1 Processor -- start 2024-03-06T07:28:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.029+0000 7f5d14400640 1 -- start start 2024-03-06T07:28:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.031+0000 7f5d14400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5d0c0af580 0x7f5d0c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.031+0000 7f5d14400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c15cc40 con 0x7f5d0c0af1a0 2024-03-06T07:28:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.031+0000 7f5d14400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c15cf20 con 0x7f5d0c0cdc20 2024-03-06T07:28:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.031+0000 7f5d14400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c15d200 con 0x7f5d0c0af580 2024-03-06T07:28:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.031+0000 7f5d11a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5d0c0af1a0 0x7f5d0c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59606/0 (socket says 172.21.15.6:59606) 2024-03-06T07:28:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.031+0000 7f5d11a00640 1 -- 172.21.15.6:0/2849739407 learned_addr learned my addr 172.21.15.6:0/2849739407 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:28.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595850210 0 0) 0x7f5d0c15cc40 con 0x7f5d0c0af1a0 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4003520 con 0x7f5d0c0af1a0 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d11000640 1 --2- 172.21.15.6:0/2849739407 >> v2:172.21.15.178:3300/0 conn(0x7f5d0c0af580 0x7f5d0c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2238752027 0 0) 0x7f5cf4003520 con 0x7f5d0c0af1a0 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 >> v2:172.21.15.178:3300/0 conn(0x7f5d0c0af580 msgr2=0x7f5d0c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 --2- 172.21.15.6:0/2849739407 >> v2:172.21.15.178:3300/0 conn(0x7f5d0c0af580 0x7f5d0c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 >> v1:172.21.15.178:6789/0 conn(0x7f5d0c0cdc20 legacy=0x7f5d0c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c15d4e0 con 0x7f5d0c0af1a0 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5d00001670 con 0x7f5d0c0af1a0 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 >> v1:172.21.15.6:6789/0 conn(0x7f5d0c0af1a0 legacy=0x7f5d0c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cf4003a70 0x7f5cf4003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5cf40053c0 0x7f5cf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c15d200 con 0x7f5cf4003a70 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf400c260 con 0x7f5cf4004800 2024-03-06T07:28:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.032+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf400c460 con 0x7f5cf40053c0 2024-03-06T07:28:28.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cf4003a70 0x7f5cf4003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d12400640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5cf40053c0 0x7f5cf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11000640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11a00640 1 -- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5cf40053c0 msgr2=0x7f5cf400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5cf40053c0 0x7f5cf400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11a00640 1 -- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4004800 msgr2=0x7f5cf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11a00640 1 -- 172.21.15.6:0/2849739407 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4003520 con 0x7f5cf4003a70 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d12400640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5cf40053c0 0x7f5cf400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d11000640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.033+0000 7f5d12400640 1 -- 172.21.15.6:0/2849739407 reap_dead start 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.034+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cf4003a70 0x7f5cf4003e90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c190 tx=0x7f5cf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cc6d5f3f0fe381 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00007c60 con 0x7f5cf4003a70 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00030070 con 0x7f5cf4003a70 2024-03-06T07:28:28.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d0be00640 1 -- 172.21.15.6:0/2849739407 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00005d90 con 0x7f5cf4003a70 2024-03-06T07:28:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 -- 172.21.15.6:0/2849739407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cf4003a70 msgr2=0x7f5cf4003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cf4003a70 0x7f5cf4003e90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c190 tx=0x7f5cf4005830 comp rx=0 tx=0).stop 2024-03-06T07:28:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 -- 172.21.15.6:0/2849739407 shutdown_connections 2024-03-06T07:28:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 --2- 172.21.15.6:0/2849739407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5cf4003a70 0x7f5cf4003e90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 -- 172.21.15.6:0/2849739407 >> 172.21.15.6:0/2849739407 conn(0x7f5d0c06afb0 msgr2=0x7f5d0c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 -- 172.21.15.6:0/2849739407 shutdown_connections 2024-03-06T07:28:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.035+0000 7f5d14400640 1 -- 172.21.15.6:0/2849739407 wait complete. 2024-03-06T07:28:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.037+0000 7f5d14400640 1 Processor -- start 2024-03-06T07:28:28.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.037+0000 7f5d14400640 1 -- start start 2024-03-06T07:28:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d14400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4003a70 0x7f5d0c06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d14400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d0c09c1b0 0x7f5d0c06cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d14400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 0x7f5d0c151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d14400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c15c990 con 0x7f5d0c15a620 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d14400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c15c810 con 0x7f5cf4003a70 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d14400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c0b4c20 con 0x7f5d0c09c1b0 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.038+0000 7f5d12400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 0x7f5d0c151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 0x7f5d0c151250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43828/0 (socket says 172.21.15.6:43828) 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 -- 172.21.15.6:0/2763612355 learned_addr learned my addr 172.21.15.6:0/2763612355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 -- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d0c09c1b0 msgr2=0x7f5d0c06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d0c09c1b0 0x7f5d0c06cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 -- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4003a70 msgr2=0x7f5d0c06e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4003a70 0x7f5d0c06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4003a70 0x7f5d0c06e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 -- 172.21.15.6:0/2763612355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c09c610 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.039+0000 7f5d12400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 0x7f5d0c151250 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c0cda30 tx=0x7f5d0400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.040+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04018070 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.040+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d04003b90 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.040+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04005070 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.040+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c06d360 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.040+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c06d920 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.041+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c04f3a0 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.042+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d0401b070 con 0x7f5d0c15a620 2024-03-06T07:28:28.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.043+0000 7f5d09600640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cec06b4d0 0x7f5cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.043+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5d04089a70 con 0x7f5d0c15a620 2024-03-06T07:28:28.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.043+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cec06b4d0 0x7f5cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.045+0000 7f5d11a00640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cec06b4d0 0x7f5cec06d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c160 tx=0x7f5d00009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.047+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d040587d0 con 0x7f5d0c15a620 2024-03-06T07:28:28.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.292+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5d0c06d050 con 0x7f5d0c15a620 2024-03-06T07:28:28.295 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:28:28.293+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:28.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.297+0000 7f5d09600640 1 -- 172.21.15.6:0/2763612355 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5d040580d0 con 0x7f5d0c15a620 2024-03-06T07:28:28.299 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:28.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.299+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cec06b4d0 msgr2=0x7f5cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.299+0000 7f5d14400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cec06b4d0 0x7f5cec06d990 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000c160 tx=0x7f5d00009030 comp rx=0 tx=0).stop 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.299+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 msgr2=0x7f5d0c151250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.299+0000 7f5d14400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 0x7f5d0c151250 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c0cda30 tx=0x7f5d0400dc00 comp rx=0 tx=0).stop 2024-03-06T07:28:28.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 shutdown_connections 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cec06b4d0 0x7f5cec06d990 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d0c15a620 0x7f5d0c151250 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d0c09c1b0 0x7f5d0c06cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 --2- 172.21.15.6:0/2763612355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cf4003a70 0x7f5d0c06e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 >> 172.21.15.6:0/2763612355 conn(0x7f5d0c06afb0 msgr2=0x7f5d0c0aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 shutdown_connections 2024-03-06T07:28:28.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.300+0000 7f5d14400640 1 -- 172.21.15.6:0/2763612355 wait complete. 2024-03-06T07:28:28.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:28.338 INFO:tasks.ceph.mds.b.smithi178.stderr: -7760> 2024-03-06T07:27:35.170+0000 7f63f2400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:28.414 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:28:28.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.543+0000 7f6501600640 1 Processor -- start 2024-03-06T07:28:28.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.543+0000 7f6501600640 1 -- start start 2024-03-06T07:28:28.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f6501600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64fc0b3c20 0x7f64fc0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f6501600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f64fc0b1b40 0x7f64fc0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f6501600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64fc071930 con 0x7f64fc0b4000 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f6501600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64fc0b4490 con 0x7f64fc0b3c20 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f6501600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64fc0b24e0 con 0x7f64fc0b1b40 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f64fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64fc0b3c20 0x7f64fc0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f64fbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64fc0b3c20 0x7f64fc0729d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43834/0 (socket says 172.21.15.6:43834) 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.545+0000 7f64fbe00640 1 -- 172.21.15.6:0/1486492245 learned_addr learned my addr 172.21.15.6:0/1486492245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:28.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.546+0000 7f6500c00640 1 --2- 172.21.15.6:0/1486492245 >> v2:172.21.15.178:3300/0 conn(0x7f64fc0b1b40 0x7f64fc0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.546+0000 7f64fbe00640 1 -- 172.21.15.6:0/1486492245 >> v2:172.21.15.178:3300/0 conn(0x7f64fc0b1b40 msgr2=0x7f64fc0b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.546+0000 7f64fbe00640 1 --2- 172.21.15.6:0/1486492245 >> v2:172.21.15.178:3300/0 conn(0x7f64fc0b1b40 0x7f64fc0b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.546+0000 7f64fbe00640 1 -- 172.21.15.6:0/1486492245 >> v1:172.21.15.178:6789/0 conn(0x7f64fc0b4000 legacy=0x7f64fc15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.546+0000 7f64fbe00640 1 -- 172.21.15.6:0/1486492245 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc071c10 con 0x7f64fc0b3c20 2024-03-06T07:28:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.547+0000 7f64fbe00640 1 --2- 172.21.15.6:0/1486492245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0729d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0af660 tx=0x7f64ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708e3942506b506c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.547+0000 7f64faa00640 1 -- 172.21.15.6:0/1486492245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02ca80 con 0x7f64fc0b3c20 2024-03-06T07:28:28.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.547+0000 7f64faa00640 1 -- 172.21.15.6:0/1486492245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec02cc20 con 0x7f64fc0b3c20 2024-03-06T07:28:28.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.547+0000 7f64faa00640 1 -- 172.21.15.6:0/1486492245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec0343f0 con 0x7f64fc0b3c20 2024-03-06T07:28:28.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 -- 172.21.15.6:0/1486492245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 msgr2=0x7f64fc0729d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 --2- 172.21.15.6:0/1486492245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0729d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0af660 tx=0x7f64ec02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 -- 172.21.15.6:0/1486492245 shutdown_connections 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 --2- 172.21.15.6:0/1486492245 >> v2:172.21.15.178:3300/0 conn(0x7f64fc0b1b40 0x7f64fc0b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 --2- 172.21.15.6:0/1486492245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0729d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 -- 172.21.15.6:0/1486492245 >> 172.21.15.6:0/1486492245 conn(0x7f64fc06b3a0 msgr2=0x7f64fc06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 -- 172.21.15.6:0/1486492245 shutdown_connections 2024-03-06T07:28:28.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.548+0000 7f6501600640 1 -- 172.21.15.6:0/1486492245 wait complete. 2024-03-06T07:28:28.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.550+0000 7f6501600640 1 Processor -- start 2024-03-06T07:28:28.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.550+0000 7f6501600640 1 -- start start 2024-03-06T07:28:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.551+0000 7f6501600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64fc0b1b40 0x7f64fc1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.551+0000 7f6501600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.551+0000 7f6501600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64fc0b4000 0x7f64fc0cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.551+0000 7f6501600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0721d0 con 0x7f64fc0b3c20 2024-03-06T07:28:28.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.551+0000 7f6501600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc15bfb0 con 0x7f64fc0b1b40 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.551+0000 7f6501600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64fc0724b0 con 0x7f64fc0b4000 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43842/0 (socket says 172.21.15.6:43842) 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 -- 172.21.15.6:0/4158842191 learned_addr learned my addr 172.21.15.6:0/4158842191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fbe00640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64fc0b1b40 0x7f64fc1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f6500c00640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64fc0b4000 0x7f64fc0cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 -- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64fc0b4000 msgr2=0x7f64fc0cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64fc0b4000 0x7f64fc0cea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 -- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64fc0b1b40 msgr2=0x7f64fc1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64fc0b1b40 0x7f64fc1423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 -- 172.21.15.6:0/4158842191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64fc1557c0 con 0x7f64fc0b3c20 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fbe00640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64fc0b1b40 0x7f64fc1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.552+0000 7f64fb400640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0ce4c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c460 tx=0x7f64e400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.553+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4016020 con 0x7f64fc0b3c20 2024-03-06T07:28:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.553+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4012870 con 0x7f64fc0b3c20 2024-03-06T07:28:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.553+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4012c60 con 0x7f64fc0b3c20 2024-03-06T07:28:28.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.553+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64fc1428f0 con 0x7f64fc0b3c20 2024-03-06T07:28:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.553+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64fc155d10 con 0x7f64fc0b3c20 2024-03-06T07:28:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.554+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc04f330 con 0x7f64fc0b3c20 2024-03-06T07:28:28.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.555+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64e4019020 con 0x7f64fc0b3c20 2024-03-06T07:28:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.556+0000 7f64f3e00640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64d806b4d0 0x7f64d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.556+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f64e408a000 con 0x7f64fc0b3c20 2024-03-06T07:28:28.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.557+0000 7f64fbe00640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64d806b4d0 0x7f64d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.558+0000 7f64fbe00640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64d806b4d0 0x7f64d806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64ec035000 tx=0x7f64ec030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.561+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64e4020030 con 0x7f64fc0b3c20 2024-03-06T07:28:28.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:28.805+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f64fc06b8b0 con 0x7f64fc0b3c20 2024-03-06T07:28:29.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.300+0000 7f64f3e00640 1 -- 172.21.15.6:0/4158842191 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f64e4058da0 con 0x7f64fc0b3c20 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.303+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64d806b4d0 msgr2=0x7f64d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.303+0000 7f6501600640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64d806b4d0 0x7f64d806d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f64ec035000 tx=0x7f64ec030000 comp rx=0 tx=0).stop 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 msgr2=0x7f64fc0ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0ce4c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c460 tx=0x7f64e400c920 comp rx=0 tx=0).stop 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 shutdown_connections 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64d806b4d0 0x7f64d806d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64fc0b4000 0x7f64fc0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64fc0b3c20 0x7f64fc0ce4c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 --2- 172.21.15.6:0/4158842191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64fc0b1b40 0x7f64fc1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 >> 172.21.15.6:0/4158842191 conn(0x7f64fc06b3a0 msgr2=0x7f64fc15da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 shutdown_connections 2024-03-06T07:28:29.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.304+0000 7f6501600640 1 -- 172.21.15.6:0/4158842191 wait complete. 2024-03-06T07:28:29.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:29.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.552+0000 7f4d32200640 1 Processor -- start 2024-03-06T07:28:29.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.552+0000 7f4d32200640 1 -- start start 2024-03-06T07:28:29.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.554+0000 7f4d32200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.554+0000 7f4d32200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4d2c0d4610 0x7f4d2c0d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.554+0000 7f4d32200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c0d1e40 con 0x7f4d2c0d23e0 2024-03-06T07:28:29.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.554+0000 7f4d32200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2c0d1be0 con 0x7f4d2c0d4610 2024-03-06T07:28:29.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.554+0000 7f4d32200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0d2100 con 0x7f4d2c15e180 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2aa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4d2c0d4610 0x7f4d2c0d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c0b35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:52148/0 (socket says 172.21.15.6:52148) 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2b400640 1 -- 172.21.15.6:0/3200164281 learned_addr learned my addr 172.21.15.6:0/3200164281 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 msgr2=0x7f4d2c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2aa00640 1 --2- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c0b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.555+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3200164281 >> v1:172.21.15.178:6789/0 conn(0x7f4d2c0d23e0 legacy=0x7f4d2c0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.556+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3200164281 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0ceed0 con 0x7f4d2c0d4610 2024-03-06T07:28:29.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.557+0000 7f4d2aa00640 1 --2- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c0d4a30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097a0 tx=0x7f4d1802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4708d6c70e7739fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.557+0000 7f4d2a000640 1 -- 172.21.15.6:0/3200164281 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18035020 con 0x7f4d2c0d4610 2024-03-06T07:28:29.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.557+0000 7f4d2a000640 1 -- 172.21.15.6:0/3200164281 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d18031810 con 0x7f4d2c0d4610 2024-03-06T07:28:29.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.557+0000 7f4d2a000640 1 -- 172.21.15.6:0/3200164281 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18031be0 con 0x7f4d2c0d4610 2024-03-06T07:28:29.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 -- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d4610 msgr2=0x7f4d2c0d4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 --2- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c0d4a30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097a0 tx=0x7f4d1802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 -- 172.21.15.6:0/3200164281 shutdown_connections 2024-03-06T07:28:29.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 --2- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c0d4a30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 --2- 172.21.15.6:0/3200164281 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c0b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 -- 172.21.15.6:0/3200164281 >> 172.21.15.6:0/3200164281 conn(0x7f4d2c09d2b0 msgr2=0x7f4d2c0d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 -- 172.21.15.6:0/3200164281 shutdown_connections 2024-03-06T07:28:29.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.558+0000 7f4d32200640 1 -- 172.21.15.6:0/3200164281 wait complete. 2024-03-06T07:28:29.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.561+0000 7f4d32200640 1 Processor -- start 2024-03-06T07:28:29.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.561+0000 7f4d32200640 1 -- start start 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d32200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d23e0 0x7f4d2c1602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d32200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c15c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d32200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d32200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0b32a0 con 0x7f4d2c0d4610 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d32200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0cebc0 con 0x7f4d2c0d23e0 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d32200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c0b8680 con 0x7f4d2c15e180 2024-03-06T07:28:29.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.562+0000 7f4d2aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c15c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43856/0 (socket says 172.21.15.6:43856) 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3290710954 learned_addr learned my addr 172.21.15.6:0/3290710954 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2b400640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d23e0 0x7f4d2c1602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2be00640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 msgr2=0x7f4d2c06ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c06ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d23e0 msgr2=0x7f4d2c1602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d23e0 0x7f4d2c1602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 -- 172.21.15.6:0/3290710954 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c160820 con 0x7f4d2c0d4610 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2b400640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d23e0 0x7f4d2c1602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:29.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.563+0000 7f4d2aa00640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c15c280 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4d180098c0 tx=0x7f4d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.564+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18006e10 con 0x7f4d2c0d4610 2024-03-06T07:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.564+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1803bd70 con 0x7f4d2c0d4610 2024-03-06T07:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.564+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18045630 con 0x7f4d2c0d4610 2024-03-06T07:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.564+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c0b7a70 con 0x7f4d2c0d4610 2024-03-06T07:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.564+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c0b8030 con 0x7f4d2c0d4610 2024-03-06T07:28:29.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.565+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c068010 con 0x7f4d2c0d4610 2024-03-06T07:28:29.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.566+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d1804c060 con 0x7f4d2c0d4610 2024-03-06T07:28:29.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.567+0000 7f4d23400640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d0c06b500 0x7f4d0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.567+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f4d180ac230 con 0x7f4d2c0d4610 2024-03-06T07:28:29.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.567+0000 7f4d2b400640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d0c06b500 0x7f4d0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.569+0000 7f4d2b400640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d0c06b500 0x7f4d0c06d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c160c60 tx=0x7f4d1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.572+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d18040070 con 0x7f4d2c0d4610 2024-03-06T07:28:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:29.771+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4d2c0d1580 con 0x7f4d2c0d4610 2024-03-06T07:28:30.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.727+0000 7f4d23400640 1 -- 172.21.15.6:0/3290710954 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4d1807af40 con 0x7f4d2c0d4610 2024-03-06T07:28:30.730 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d0c06b500 msgr2=0x7f4d0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d0c06b500 0x7f4d0c06d9c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c160c60 tx=0x7f4d1c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 msgr2=0x7f4d2c15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c15c280 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4d180098c0 tx=0x7f4d180046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 shutdown_connections 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d0c06b500 0x7f4d0c06d9c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4d2c15e180 0x7f4d2c06ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d2c0d4610 0x7f4d2c15c280 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 --2- 172.21.15.6:0/3290710954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d2c0d23e0 0x7f4d2c1602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 >> 172.21.15.6:0/3290710954 conn(0x7f4d2c09d2b0 msgr2=0x7f4d2c0b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 shutdown_connections 2024-03-06T07:28:30.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.732+0000 7f4d32200640 1 -- 172.21.15.6:0/3290710954 wait complete. 2024-03-06T07:28:30.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.967+0000 7f3e3e800640 1 Processor -- start 2024-03-06T07:28:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.967+0000 7f3e3e800640 1 -- start start 2024-03-06T07:28:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e3e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e380ae7b0 0x7f3e380aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e3e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e381531f0 con 0x7f3e380b3640 2024-03-06T07:28:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e3e800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b2b80 con 0x7f3e3815cdc0 2024-03-06T07:28:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e3e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380d04d0 con 0x7f3e380ae7b0 2024-03-06T07:28:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e36a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3e380b3640 0x7f3e380aeb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59608/0 (socket says 172.21.15.6:59608) 2024-03-06T07:28:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e36a00640 1 -- 172.21.15.6:0/4234305935 learned_addr learned my addr 172.21.15.6:0/4234305935 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.969+0000 7f3e37400640 1 --2- 172.21.15.6:0/4234305935 >> v2:172.21.15.6:3300/0 conn(0x7f3e380ae7b0 0x7f3e380aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.970+0000 7f3e36000640 1 -- 172.21.15.6:0/4234305935 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687538848 0 0) 0x7f3e381531f0 con 0x7f3e380b3640 2024-03-06T07:28:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.970+0000 7f3e36000640 1 -- 172.21.15.6:0/4234305935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c003520 con 0x7f3e380b3640 2024-03-06T07:28:30.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.970+0000 7f3e37400640 1 -- 172.21.15.6:0/4234305935 >> v1:172.21.15.178:6789/0 conn(0x7f3e3815cdc0 legacy=0x7f3e3806cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.970+0000 7f3e37400640 1 -- 172.21.15.6:0/4234305935 >> v1:172.21.15.6:6789/0 conn(0x7f3e380b3640 legacy=0x7f3e380aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.970+0000 7f3e37400640 1 -- 172.21.15.6:0/4234305935 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3806ebd0 con 0x7f3e380ae7b0 2024-03-06T07:28:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.971+0000 7f3e37400640 1 --2- 172.21.15.6:0/4234305935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e380aced0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3e280097a0 tx=0x7f3e2802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbecad96ef5aa3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e36000640 1 -- 172.21.15.6:0/4234305935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28034070 con 0x7f3e380ae7b0 2024-03-06T07:28:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e36000640 1 -- 172.21.15.6:0/4234305935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28030610 con 0x7f3e380ae7b0 2024-03-06T07:28:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e36000640 1 -- 172.21.15.6:0/4234305935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e280309e0 con 0x7f3e380ae7b0 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 -- 172.21.15.6:0/4234305935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 msgr2=0x7f3e380aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 --2- 172.21.15.6:0/4234305935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e380aced0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3e280097a0 tx=0x7f3e2802b960 comp rx=0 tx=0).stop 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 -- 172.21.15.6:0/4234305935 shutdown_connections 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 --2- 172.21.15.6:0/4234305935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e380aced0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 -- 172.21.15.6:0/4234305935 >> 172.21.15.6:0/4234305935 conn(0x7f3e3806afb0 msgr2=0x7f3e3806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 -- 172.21.15.6:0/4234305935 shutdown_connections 2024-03-06T07:28:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.972+0000 7f3e3e800640 1 -- 172.21.15.6:0/4234305935 wait complete. 2024-03-06T07:28:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.974+0000 7f3e3e800640 1 Processor -- start 2024-03-06T07:28:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.974+0000 7f3e3e800640 1 -- start start 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.975+0000 7f3e3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e38147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.975+0000 7f3e3e800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e380b3640 0x7f3e38153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.975+0000 7f3e3e800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e3815cdc0 0x7f3e38159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.975+0000 7f3e3e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b3120 con 0x7f3e380ae7b0 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.975+0000 7f3e3e800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b2e40 con 0x7f3e3815cdc0 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.975+0000 7f3e3e800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3815d250 con 0x7f3e380b3640 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e38147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e38147580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43876/0 (socket says 172.21.15.6:43876) 2024-03-06T07:28:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 -- 172.21.15.6:0/2048563235 learned_addr learned my addr 172.21.15.6:0/2048563235 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 -- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e380b3640 msgr2=0x7f3e38153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e380b3640 0x7f3e38153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 -- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e3815cdc0 msgr2=0x7f3e38159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e3815cdc0 0x7f3e38159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 -- 172.21.15.6:0/2048563235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38147ac0 con 0x7f3e380ae7b0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.976+0000 7f3e37400640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e38147580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3e280098c0 tx=0x7f3e280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.977+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28034040 con 0x7f3e380ae7b0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.977+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28030410 con 0x7f3e380ae7b0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.977+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28039c50 con 0x7f3e380ae7b0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.977+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e38157fd0 con 0x7f3e380ae7b0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.977+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e38158590 con 0x7f3e380ae7b0 2024-03-06T07:28:30.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.978+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380d0130 con 0x7f3e380ae7b0 2024-03-06T07:28:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.979+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e28037070 con 0x7f3e380ae7b0 2024-03-06T07:28:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.980+0000 7f3e2f400640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e1806b4d0 0x7f3e1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.981+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f3e280ac1b0 con 0x7f3e380ae7b0 2024-03-06T07:28:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.981+0000 7f3e36a00640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e1806b4d0 0x7f3e1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.983+0000 7f3e36a00640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e1806b4d0 0x7f3e1806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e24000f60 tx=0x7f3e24007970 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:30.989+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e380d0130 con 0x7f3e380ae7b0 2024-03-06T07:28:31.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.189+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3e380b1630 con 0x7f3e380ae7b0 2024-03-06T07:28:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.731+0000 7f3e2f400640 1 -- 172.21.15.6:0/2048563235 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3e380b1630 con 0x7f3e380ae7b0 2024-03-06T07:28:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e1806b4d0 msgr2=0x7f3e1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e1806b4d0 0x7f3e1806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e24000f60 tx=0x7f3e24007970 comp rx=0 tx=0).stop 2024-03-06T07:28:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 msgr2=0x7f3e38147580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e38147580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3e280098c0 tx=0x7f3e280046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 shutdown_connections 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e1806b4d0 0x7f3e1806d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e3815cdc0 0x7f3e38159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e380b3640 0x7f3e38153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 --2- 172.21.15.6:0/2048563235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e380ae7b0 0x7f3e38147580 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.735+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 >> 172.21.15.6:0/2048563235 conn(0x7f3e3806afb0 msgr2=0x7f3e380b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.736+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 shutdown_connections 2024-03-06T07:28:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:31.736+0000 7f3e3e800640 1 -- 172.21.15.6:0/2048563235 wait complete. 2024-03-06T07:28:31.764 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:31.764 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:31.765 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:31.765 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:31.765 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:31.765 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:31.765 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:31.765 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:31.765 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:31.766 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:31.766 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:31.766 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:31.766 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:31.766 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:31.766 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:31.767 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:31.784 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:31.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:31.786 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:31.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:31.787 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:31.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:31.788 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:31.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:31.832 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:31.832 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:31.832 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:31.833 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:31.837 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:31.837 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:31.839 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:31.839 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:31.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:31.990 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:28:31.991 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:28:31.992 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:28:31.993 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:28:32.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.028+0000 7f1b48200640 1 Processor -- start 2024-03-06T07:28:32.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.028+0000 7f1b48200640 1 -- start start 2024-03-06T07:28:32.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b48200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1b40070ee0 0x7f1b40162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b48200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b40072da0 0x7f1b400cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b48200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400ce090 con 0x7f1b400712c0 2024-03-06T07:28:32.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b48200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b400ceca0 con 0x7f1b40072da0 2024-03-06T07:28:32.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b48200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b400ce370 con 0x7f1b40070ee0 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b44e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b40072da0 0x7f1b400cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b44e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1b40072da0 0x7f1b400cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54984/0 (socket says 172.21.15.6:54984) 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b44e00640 1 -- 172.21.15.6:0/2455192360 learned_addr learned my addr 172.21.15.6:0/2455192360 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.030+0000 7f1b45800640 1 --2- 172.21.15.6:0/2455192360 >> v2:172.21.15.178:3300/0 conn(0x7f1b40070ee0 0x7f1b40162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.031+0000 7f1b44e00640 1 -- 172.21.15.6:0/2455192360 >> v2:172.21.15.178:3300/0 conn(0x7f1b40070ee0 msgr2=0x7f1b40162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.031+0000 7f1b44e00640 1 --2- 172.21.15.6:0/2455192360 >> v2:172.21.15.178:3300/0 conn(0x7f1b40070ee0 0x7f1b40162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.031+0000 7f1b44e00640 1 -- 172.21.15.6:0/2455192360 >> v1:172.21.15.178:6789/0 conn(0x7f1b400712c0 legacy=0x7f1b4006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.031+0000 7f1b44e00640 1 -- 172.21.15.6:0/2455192360 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400ce650 con 0x7f1b40072da0 2024-03-06T07:28:32.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.032+0000 7f1b44e00640 1 --2- 172.21.15.6:0/2455192360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40072da0 0x7f1b400cf130 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1b40073610 tx=0x7f1b2c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcdc42114d896d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.032+0000 7f1b3fe00640 1 -- 172.21.15.6:0/2455192360 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c034070 con 0x7f1b40072da0 2024-03-06T07:28:32.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.032+0000 7f1b3fe00640 1 -- 172.21.15.6:0/2455192360 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c030860 con 0x7f1b40072da0 2024-03-06T07:28:32.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.032+0000 7f1b3fe00640 1 -- 172.21.15.6:0/2455192360 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c030c30 con 0x7f1b40072da0 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 -- 172.21.15.6:0/2455192360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40072da0 msgr2=0x7f1b400cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 --2- 172.21.15.6:0/2455192360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40072da0 0x7f1b400cf130 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1b40073610 tx=0x7f1b2c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 -- 172.21.15.6:0/2455192360 shutdown_connections 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 --2- 172.21.15.6:0/2455192360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40072da0 0x7f1b400cf130 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 --2- 172.21.15.6:0/2455192360 >> v2:172.21.15.178:3300/0 conn(0x7f1b40070ee0 0x7f1b40162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 -- 172.21.15.6:0/2455192360 >> 172.21.15.6:0/2455192360 conn(0x7f1b4006afb0 msgr2=0x7f1b4006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 -- 172.21.15.6:0/2455192360 shutdown_connections 2024-03-06T07:28:32.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.033+0000 7f1b48200640 1 -- 172.21.15.6:0/2455192360 wait complete. 2024-03-06T07:28:32.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.035+0000 7f1b48200640 1 Processor -- start 2024-03-06T07:28:32.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.035+0000 7f1b48200640 1 -- start start 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b48200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 0x7f1b4014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b48200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b400712c0 0x7f1b400b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b48200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b40072da0 0x7f1b4015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b48200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400cdda0 con 0x7f1b40070ee0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b48200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400cdc20 con 0x7f1b40072da0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b48200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4006e7d0 con 0x7f1b400712c0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b45800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 0x7f1b4014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b45800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 0x7f1b4014fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55000/0 (socket says 172.21.15.6:55000) 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b45800640 1 -- 172.21.15.6:0/1396099528 learned_addr learned my addr 172.21.15.6:0/1396099528 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b44e00640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b400712c0 0x7f1b400b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b46200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b40072da0 0x7f1b4015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b45800640 1 -- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b400712c0 msgr2=0x7f1b400b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.036+0000 7f1b45800640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b400712c0 0x7f1b400b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b45800640 1 -- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b40072da0 msgr2=0x7f1b4015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b45800640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b40072da0 0x7f1b4015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b45800640 1 -- 172.21.15.6:0/1396099528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4015d800 con 0x7f1b40070ee0 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b46200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b40072da0 0x7f1b4015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b45800640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 0x7f1b4014fe80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e9b0 tx=0x7f1b3400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b34017070 con 0x7f1b40070ee0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3400c9a0 con 0x7f1b40070ee0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400cd90 con 0x7f1b40070ee0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b401503c0 con 0x7f1b40070ee0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.037+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b400aef20 con 0x7f1b40070ee0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.039+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4004f3a0 con 0x7f1b40070ee0 2024-03-06T07:28:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.039+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b3401a070 con 0x7f1b40070ee0 2024-03-06T07:28:32.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.040+0000 7f1b3d600640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1b2006b520 0x7f1b2006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.040+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1b34089f70 con 0x7f1b40070ee0 2024-03-06T07:28:32.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.041+0000 7f1b44e00640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1b2006b520 0x7f1b2006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.042+0000 7f1b44e00640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1b2006b520 0x7f1b2006d9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c036000 tx=0x7f1b2c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.045+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b401503c0 con 0x7f1b40070ee0 2024-03-06T07:28:32.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.261+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1b400b0e00 con 0x7f1b40070ee0 2024-03-06T07:28:32.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.770+0000 7f1b3d600640 1 -- 172.21.15.6:0/1396099528 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1b4015d800 con 0x7f1b40070ee0 2024-03-06T07:28:32.772 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:28:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.773+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1b2006b520 msgr2=0x7f1b2006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.773+0000 7f1b48200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1b2006b520 0x7f1b2006d9e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c036000 tx=0x7f1b2c03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.773+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 msgr2=0x7f1b4014fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.773+0000 7f1b48200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 0x7f1b4014fe80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e9b0 tx=0x7f1b3400ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 shutdown_connections 2024-03-06T07:28:32.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1b2006b520 0x7f1b2006d9e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1b40072da0 0x7f1b4015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1b400712c0 0x7f1b400b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 --2- 172.21.15.6:0/1396099528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1b40070ee0 0x7f1b4014fe80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 >> 172.21.15.6:0/1396099528 conn(0x7f1b4006afb0 msgr2=0x7f1b40164f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 shutdown_connections 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.774+0000 7f1b48200640 1 -- 172.21.15.6:0/1396099528 wait complete. 2024-03-06T07:28:32.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.983+0000 7f6be0a00640 1 Processor -- start 2024-03-06T07:28:32.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.983+0000 7f6be0a00640 1 -- start start 2024-03-06T07:28:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.985+0000 7f6be0a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 0x7f6bdc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.985+0000 7f6be0a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc15ed10 con 0x7f6bdc15e810 2024-03-06T07:28:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.985+0000 7f6be0a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc15eff0 con 0x7f6bdc0cd060 2024-03-06T07:28:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.985+0000 7f6be0a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc15f2d0 con 0x7f6bdc1552b0 2024-03-06T07:28:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.985+0000 7f6bd9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6bdc15e810 0x7f6bdc0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44558/0 (socket says 172.21.15.6:44558) 2024-03-06T07:28:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.985+0000 7f6bd9e00640 1 -- 172.21.15.6:0/3336744995 learned_addr learned my addr 172.21.15.6:0/3336744995 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bda800640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 0x7f6bdc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018145919 0 0) 0x7f6bdc15ed10 con 0x7f6bdc15e810 2024-03-06T07:28:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc0003520 con 0x7f6bdc15e810 2024-03-06T07:28:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1214457885 0 0) 0x7f6bc0003520 con 0x7f6bdc15e810 2024-03-06T07:28:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 msgr2=0x7f6bdc15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 0x7f6bdc15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 >> v1:172.21.15.178:6789/0 conn(0x7f6bdc0cd060 legacy=0x7f6bdc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc15f5b0 con 0x7f6bdc15e810 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f6bc4001670 con 0x7f6bdc15e810 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 >> v1:172.21.15.6:6789/0 conn(0x7f6bdc15e810 legacy=0x7f6bdc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bc00039e0 0x7f6bc0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 0x7f6bc0005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc15f2d0 con 0x7f6bc0004800 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc000bb50 con 0x7f6bc00039e0 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc000df60 con 0x7f6bdc1552b0 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.986+0000 7f6bd9e00640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bda800640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bc00039e0 0x7f6bc0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bd9e00640 1 -- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 msgr2=0x7f6bc0005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bdb200640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 0x7f6bc0005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bd9e00640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bdc1552b0 0x7f6bc0005270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bd9e00640 1 -- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bc00039e0 msgr2=0x7f6bc0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bd9e00640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bc00039e0 0x7f6bc0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.987+0000 7f6bd9e00640 1 -- 172.21.15.6:0/3336744995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0003520 con 0x7f6bc0004800 2024-03-06T07:28:32.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.988+0000 7f6bd9e00640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c110 tx=0x7f6bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d61de3a930bc79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.988+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4007c40 con 0x7f6bc0004800 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.988+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc4030070 con 0x7f6bc0004800 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.988+0000 7f6bd9400640 1 -- 172.21.15.6:0/3336744995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4005d30 con 0x7f6bc0004800 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6be0a00640 1 -- 172.21.15.6:0/3336744995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc0004800 msgr2=0x7f6bc0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6be0a00640 1 --2- 172.21.15.6:0/3336744995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bc0004800 0x7f6bc0004c80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400c110 tx=0x7f6bc0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6bdb200640 1 -- 172.21.15.6:0/3336744995 reap_dead start 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6be0a00640 1 -- 172.21.15.6:0/3336744995 shutdown_connections 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6be0a00640 1 -- 172.21.15.6:0/3336744995 >> 172.21.15.6:0/3336744995 conn(0x7f6bdc06b3d0 msgr2=0x7f6bdc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6be0a00640 1 -- 172.21.15.6:0/3336744995 shutdown_connections 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.989+0000 7f6be0a00640 1 -- 172.21.15.6:0/3336744995 wait complete. 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.990+0000 7f6be0a00640 1 Processor -- start 2024-03-06T07:28:32.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.991+0000 7f6be0a00640 1 -- start start 2024-03-06T07:28:32.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6be0a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bc0004800 0x7f6bdc1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6be0a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bdc157570 0x7f6bdc0afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6be0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 0x7f6bdc145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6be0a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc06dd30 con 0x7f6bdc0b0410 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6be0a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0b3760 con 0x7f6bdc157570 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6be0a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc06deb0 con 0x7f6bc0004800 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 0x7f6bdc145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 0x7f6bdc145cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55024/0 (socket says 172.21.15.6:55024) 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 -- 172.21.15.6:0/1638151147 learned_addr learned my addr 172.21.15.6:0/1638151147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bd9e00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bdc157570 0x7f6bdc0afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bda800640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bc0004800 0x7f6bdc1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 -- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bc0004800 msgr2=0x7f6bdc1598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bc0004800 0x7f6bdc1598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 -- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bdc157570 msgr2=0x7f6bdc0afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.992+0000 7f6bdb200640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bdc157570 0x7f6bdc0afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6bdb200640 1 -- 172.21.15.6:0/1638151147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc157a00 con 0x7f6bdc0b0410 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6bdb200640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 0x7f6bdc145cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0004780 tx=0x7f6bd400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4018070 con 0x7f6bdc0b0410 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40052f0 con 0x7f6bdc0b0410 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc157ce0 con 0x7f6bdc0b0410 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4012840 con 0x7f6bdc0b0410 2024-03-06T07:28:32.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.993+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc1582a0 con 0x7f6bdc0b0410 2024-03-06T07:28:32.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.994+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc04f330 con 0x7f6bdc0b0410 2024-03-06T07:28:32.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.995+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bd401b070 con 0x7f6bdc0b0410 2024-03-06T07:28:32.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.996+0000 7f6bd2a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6bb806b500 0x7f6bb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.996+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6bd40890d0 con 0x7f6bdc0b0410 2024-03-06T07:28:32.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.997+0000 7f6bda800640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6bb806b500 0x7f6bb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:32.998+0000 7f6bda800640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6bb806b500 0x7f6bb806d9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc007520 tx=0x7f6bcc00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.000+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6bd4058210 con 0x7f6bdc0b0410 2024-03-06T07:28:33.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.270+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6bdc0ccb70 con 0x7f6bdc0b0410 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.271+0000 7f6bd2a00640 1 -- 172.21.15.6:0/1638151147 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f6bd4057b10 con 0x7f6bdc0b0410 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:28:33.273 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:28:33.274 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:28:33.276 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.277 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:28:33.278 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.280 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:28:33.281 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:28:33.283 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:28:33.284 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQDIGuhlts1rCBAAaBfVhFV/s4G6/hGoc3rvPQ==", 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQDIGuhlQtdrCBAAP+l6dTMtEmNIuli+eXkt1w==", 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQDIGuhlU+NrCBAAKW6tix18VWFFtURw2gKR0g==", 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.285 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQDIGuhl5+trCBAARE9dmr55a5oVsZk5hS/P5g==", 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQDIGuhl3/RrCBAA8vj6If8iFoBtqahtkpeOHQ==", 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQDIGuhlr/1rCBAAZcbDQZ3/f/0QLn2IlV2wlg==", 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:28:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:28:33.288 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6bb806b500 msgr2=0x7f6bb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6bb806b500 0x7f6bb806d9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc007520 tx=0x7f6bcc00b040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 msgr2=0x7f6bdc145cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 0x7f6bdc145cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0004780 tx=0x7f6bd400d890 comp rx=0 tx=0).stop 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 shutdown_connections 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6bb806b500 0x7f6bb806d9c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.273+0000 7f6be0a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6bdc157570 0x7f6bdc0afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.274+0000 7f6be0a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6bdc0b0410 0x7f6bdc145cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.274+0000 7f6be0a00640 1 --2- 172.21.15.6:0/1638151147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6bc0004800 0x7f6bdc1598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.274+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 >> 172.21.15.6:0/1638151147 conn(0x7f6bdc06b3d0 msgr2=0x7f6bdc1524d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.274+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 shutdown_connections 2024-03-06T07:28:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.274+0000 7f6be0a00640 1 -- 172.21.15.6:0/1638151147 wait complete. 2024-03-06T07:28:33.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:28:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.517+0000 7f33a7e00640 1 Processor -- start 2024-03-06T07:28:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.517+0000 7f33a7e00640 1 -- start start 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 0x7f33a80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80714d0 con 0x7f33a8145300 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a7e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80717b0 con 0x7f33a80cce90 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a80cd300 con 0x7f33a8070f10 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f33a8145300 0x7f33a80b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44572/0 (socket says 172.21.15.6:44572) 2024-03-06T07:28:33.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a6a00640 1 -- 172.21.15.6:0/2796301571 learned_addr learned my addr 172.21.15.6:0/2796301571 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.519+0000 7f33a6000640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 0x7f33a80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592178918 0 0) 0x7f33a80714d0 con 0x7f33a8145300 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c003520 con 0x7f33a8145300 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026779417 0 0) 0x7f33a80717b0 con 0x7f33a80cce90 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a80714d0 con 0x7f33a80cce90 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713407972 0 0) 0x7f338c003520 con 0x7f33a8145300 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 msgr2=0x7f33a80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 0x7f33a80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 >> v1:172.21.15.178:6789/0 conn(0x7f33a80cce90 legacy=0x7f33a815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a8071a90 con 0x7f33a8145300 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f3398001670 con 0x7f33a8145300 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.520+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 >> v1:172.21.15.6:6789/0 conn(0x7f33a8145300 legacy=0x7f33a80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a5600640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f338c003b70 0x7f338c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a5600640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 0x7f338c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a5600640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c004d40 0x7f338c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a80cd300 con 0x7f338c004d40 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c00fe80 con 0x7f338c003b70 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c010030 con 0x7f33a8070f10 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a7400640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c004d40 0x7f338c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a6000640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 0x7f338c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.521+0000 7f33a6a00640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f338c003b70 0x7f338c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.522+0000 7f33a7400640 1 -- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 msgr2=0x7f338c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.522+0000 7f33a7400640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f33a8070f10 0x7f338c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.522+0000 7f33a7400640 1 -- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f338c003b70 msgr2=0x7f338c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.522+0000 7f33a7400640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f338c003b70 0x7f338c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.522+0000 7f33a7400640 1 -- 172.21.15.6:0/2796301571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c003520 con 0x7f338c004d40 2024-03-06T07:28:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.523+0000 7f33a7400640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c004d40 0x7f338c0051c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f33a0001fd0 tx=0x7f338c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525ab8d73ad10fde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.523+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a0001bf0 con 0x7f338c004d40 2024-03-06T07:28:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.523+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a0003b70 con 0x7f338c004d40 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.523+0000 7f33a5600640 1 -- 172.21.15.6:0/2796301571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a00050d0 con 0x7f338c004d40 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7e00640 1 -- 172.21.15.6:0/2796301571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c004d40 msgr2=0x7f338c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2796301571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f338c004d40 0x7f338c0051c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f33a0001fd0 tx=0x7f338c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7400640 1 -- 172.21.15.6:0/2796301571 reap_dead start 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7e00640 1 -- 172.21.15.6:0/2796301571 shutdown_connections 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7e00640 1 -- 172.21.15.6:0/2796301571 >> 172.21.15.6:0/2796301571 conn(0x7f33a806b3d0 msgr2=0x7f33a806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7e00640 1 -- 172.21.15.6:0/2796301571 shutdown_connections 2024-03-06T07:28:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.524+0000 7f33a7e00640 1 -- 172.21.15.6:0/2796301571 wait complete. 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.526+0000 7f33a7e00640 1 Processor -- start 2024-03-06T07:28:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.526+0000 7f33a7e00640 1 -- start start 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f338c004d40 0x7f33a813b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 0x7f33a80b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a7e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33a813c430 0x7f33a813c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a80cc950 con 0x7f33a80b1300 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a7e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a81456e0 con 0x7f33a813c430 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a80ccad0 con 0x7f338c004d40 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.527+0000 7f33a6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 0x7f33a80b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 0x7f33a80b1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55036/0 (socket says 172.21.15.6:55036) 2024-03-06T07:28:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 -- 172.21.15.6:0/2144739039 learned_addr learned my addr 172.21.15.6:0/2144739039 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6a00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f338c004d40 0x7f33a813b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a7400640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33a813c430 0x7f33a813c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 -- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f338c004d40 msgr2=0x7f33a813b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f338c004d40 0x7f33a813b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 -- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33a813c430 msgr2=0x7f33a813c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33a813c430 0x7f33a813c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 -- 172.21.15.6:0/2144739039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a813bcf0 con 0x7f33a80b1300 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.528+0000 7f33a6000640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 0x7f33a80b1760 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f33900119d0 tx=0x7f3390011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.529+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000fa90 con 0x7f33a80b1300 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.529+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339000fc30 con 0x7f33a80b1300 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.529+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390014440 con 0x7f33a80b1300 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.529+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a813d160 con 0x7f33a80b1300 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.529+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a813d720 con 0x7f33a80b1300 2024-03-06T07:28:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.530+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a804f330 con 0x7f33a80b1300 2024-03-06T07:28:33.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.531+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3390012040 con 0x7f33a80b1300 2024-03-06T07:28:33.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.532+0000 7f339ea00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f338406b4d0 0x7f338406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.532+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f339008a560 con 0x7f33a80b1300 2024-03-06T07:28:33.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.533+0000 7f33a6a00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f338406b4d0 0x7f338406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.534+0000 7f33a6a00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f338406b4d0 0x7f338406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3398001050 tx=0x7f3398009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.536+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3390019030 con 0x7f33a80b1300 2024-03-06T07:28:33.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.812+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f33a815f130 con 0x7f33a80b1300 2024-03-06T07:28:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.816+0000 7f339ea00640 1 -- 172.21.15.6:0/2144739039 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3390059600 con 0x7f33a80b1300 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f338406b4d0 msgr2=0x7f338406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f338406b4d0 0x7f338406d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3398001050 tx=0x7f3398009040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 msgr2=0x7f33a80b1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 0x7f33a80b1760 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f33900119d0 tx=0x7f3390011e90 comp rx=0 tx=0).stop 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 shutdown_connections 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f338406b4d0 0x7f338406d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33a813c430 0x7f33a813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f33a80b1300 0x7f33a80b1760 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 --2- 172.21.15.6:0/2144739039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f338c004d40 0x7f33a813b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 >> 172.21.15.6:0/2144739039 conn(0x7f33a806b3d0 msgr2=0x7f33a806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 shutdown_connections 2024-03-06T07:28:33.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:33.819+0000 7f33a7e00640 1 -- 172.21.15.6:0/2144739039 wait complete. 2024-03-06T07:28:33.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:28:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.057+0000 7f80b2800640 1 Processor -- start 2024-03-06T07:28:34.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.057+0000 7f80b2800640 1 -- start start 2024-03-06T07:28:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b2800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80ac161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b2800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f80ac162000 0x7f80ac070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b2800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ac1627f0 con 0x7f80ac15efc0 2024-03-06T07:28:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b2800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ac15f3a0 con 0x7f80ac162000 2024-03-06T07:28:34.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b2800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac06d190 con 0x7f80ac0cd060 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b1400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f80ac15efc0 0x7f80ac0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44576/0 (socket says 172.21.15.6:44576) 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.059+0000 7f80b1400640 1 -- 172.21.15.6:0/3625736258 learned_addr learned my addr 172.21.15.6:0/3625736258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80b0a00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80ac161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80b1e00640 1 --2- 172.21.15.6:0/3625736258 >> v2:172.21.15.178:3300/0 conn(0x7f80ac162000 0x7f80ac070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714598689 0 0) 0x7f80ac1627f0 con 0x7f80ac15efc0 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8094003520 con 0x7f80ac15efc0 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071650899 0 0) 0x7f8094003520 con 0x7f80ac15efc0 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 msgr2=0x7f80ac161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80ac161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 >> v2:172.21.15.178:3300/0 conn(0x7f80ac162000 msgr2=0x7f80ac070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 --2- 172.21.15.6:0/3625736258 >> v2:172.21.15.178:3300/0 conn(0x7f80ac162000 0x7f80ac070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac06c860 con 0x7f80ac15efc0 2024-03-06T07:28:34.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f80a0001670 con 0x7f80ac15efc0 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 >> v1:172.21.15.6:6789/0 conn(0x7f80ac15efc0 legacy=0x7f80ac0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80940039b0 0x7f8094003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.060+0000 7f80abe00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8094004800 0x7f8094004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80abe00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac15f3a0 con 0x7f8094004800 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac06d190 con 0x7f80940039b0 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809400b9c0 con 0x7f80ac0cd060 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b0a00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8094004800 0x7f8094004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b1400640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80940039b0 0x7f8094003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b1e00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b0a00640 1 -- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 msgr2=0x7f80940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b0a00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b0a00640 1 -- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80940039b0 msgr2=0x7f8094003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b0a00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80940039b0 0x7f8094003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b0a00640 1 -- 172.21.15.6:0/3625736258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8094003520 con 0x7f8094004800 2024-03-06T07:28:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.061+0000 7f80b1e00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80ac0cd060 0x7f80940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.062+0000 7f80b0a00640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8094004800 0x7f8094004c80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8098009ac0 tx=0x7f8094005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afbc2fd2190804c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098039070 con 0x7f8094004800 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809802c6b0 con 0x7f8094004800 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80abe00640 1 -- 172.21.15.6:0/3625736258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098009690 con 0x7f8094004800 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80b2800640 1 -- 172.21.15.6:0/3625736258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8094004800 msgr2=0x7f8094004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80b2800640 1 --2- 172.21.15.6:0/3625736258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8094004800 0x7f8094004c80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8098009ac0 tx=0x7f8094005830 comp rx=0 tx=0).stop 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80b1e00640 1 -- 172.21.15.6:0/3625736258 reap_dead start 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80b2800640 1 -- 172.21.15.6:0/3625736258 shutdown_connections 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.063+0000 7f80b2800640 1 -- 172.21.15.6:0/3625736258 >> 172.21.15.6:0/3625736258 conn(0x7f80ac06b3d0 msgr2=0x7f80ac06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.064+0000 7f80b2800640 1 -- 172.21.15.6:0/3625736258 shutdown_connections 2024-03-06T07:28:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.064+0000 7f80b2800640 1 -- 172.21.15.6:0/3625736258 wait complete. 2024-03-06T07:28:34.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.065+0000 7f80b2800640 1 Processor -- start 2024-03-06T07:28:34.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 -- start start 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8094004800 0x7f80ac144cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 0x7f80ac1475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80ac158000 0x7f80ac147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac1624b0 con 0x7f80ac157ba0 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac06ce20 con 0x7f80ac158000 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.066+0000 7f80b2800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac06d620 con 0x7f8094004800 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 0x7f80ac1475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 0x7f80ac1475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55048/0 (socket says 172.21.15.6:55048) 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 -- 172.21.15.6:0/2332648906 learned_addr learned my addr 172.21.15.6:0/2332648906 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b1400640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8094004800 0x7f80ac144cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b1e00640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80ac158000 0x7f80ac147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 -- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8094004800 msgr2=0x7f80ac144cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8094004800 0x7f80ac144cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 -- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80ac158000 msgr2=0x7f80ac147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80ac158000 0x7f80ac147af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.067+0000 7f80b0a00640 1 -- 172.21.15.6:0/2332648906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac145230 con 0x7f80ac157ba0 2024-03-06T07:28:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.068+0000 7f80b0a00640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 0x7f80ac1475b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8098009be0 tx=0x7f8098004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.068+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098042020 con 0x7f80ac157ba0 2024-03-06T07:28:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.068+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809803a040 con 0x7f80ac157ba0 2024-03-06T07:28:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.068+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098039070 con 0x7f80ac157ba0 2024-03-06T07:28:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.068+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ac1e8280 con 0x7f80ac157ba0 2024-03-06T07:28:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.068+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ac1e8840 con 0x7f80ac157ba0 2024-03-06T07:28:34.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.069+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ac04f330 con 0x7f80ac157ba0 2024-03-06T07:28:34.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.070+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8098045020 con 0x7f80ac157ba0 2024-03-06T07:28:34.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.071+0000 7f80a9600640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808c06b500 0x7f808c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.072+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f80980aafe0 con 0x7f80ac157ba0 2024-03-06T07:28:34.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.072+0000 7f80b1400640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808c06b500 0x7f808c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.073+0000 7f80b1400640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808c06b500 0x7f808c06d9c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f80a0001830 tx=0x7f80a0001e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.076+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f809804c020 con 0x7f80ac157ba0 2024-03-06T07:28:34.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.348+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f80ac0affe0 con 0x7f80ac157ba0 2024-03-06T07:28:34.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.353+0000 7f80a9600640 1 -- 172.21.15.6:0/2332648906 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7f809807a130 con 0x7f80ac157ba0 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.356+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808c06b500 msgr2=0x7f808c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808c06b500 0x7f808c06d9c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f80a0001830 tx=0x7f80a0001e30 comp rx=0 tx=0).stop 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 msgr2=0x7f80ac1475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 0x7f80ac1475b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8098009be0 tx=0x7f8098004730 comp rx=0 tx=0).stop 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 shutdown_connections 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f808c06b500 0x7f808c06d9c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80ac158000 0x7f80ac147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80ac157ba0 0x7f80ac1475b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 --2- 172.21.15.6:0/2332648906 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8094004800 0x7f80ac144cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 >> 172.21.15.6:0/2332648906 conn(0x7f80ac06b3d0 msgr2=0x7f80ac0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 shutdown_connections 2024-03-06T07:28:34.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.357+0000 7f80b2800640 1 -- 172.21.15.6:0/2332648906 wait complete. 2024-03-06T07:28:34.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:28:34.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.595+0000 7fbd49400640 1 Processor -- start 2024-03-06T07:28:34.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.595+0000 7fbd49400640 1 -- start start 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd49400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbd44070f10 0x7fbd440cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd49400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd440714d0 con 0x7fbd440cce90 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd49400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd440717b0 con 0x7fbd44145300 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd49400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd440cd300 con 0x7fbd44070f10 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd48a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd440cce90 0x7fbd4415c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44580/0 (socket says 172.21.15.6:44580) 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd48a00640 1 -- 172.21.15.6:0/3449395127 learned_addr learned my addr 172.21.15.6:0/3449395127 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:34.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd3be00640 1 --2- 172.21.15.6:0/3449395127 >> v2:172.21.15.178:3300/0 conn(0x7fbd44070f10 0x7fbd440cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.597+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889758780 0 0) 0x7fbd440714d0 con 0x7fbd440cce90 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd28003520 con 0x7fbd440cce90 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994510314 0 0) 0x7fbd28003520 con 0x7fbd440cce90 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 >> v2:172.21.15.178:3300/0 conn(0x7fbd44070f10 msgr2=0x7fbd440cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 --2- 172.21.15.6:0/3449395127 >> v2:172.21.15.178:3300/0 conn(0x7fbd44070f10 0x7fbd440cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 >> v1:172.21.15.178:6789/0 conn(0x7fbd44145300 legacy=0x7fbd440b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd44071a90 con 0x7fbd440cce90 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fbd3c0019f0 con 0x7fbd440cce90 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 >> v1:172.21.15.6:6789/0 conn(0x7fbd440cce90 legacy=0x7fbd4415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28003a00 0x7fbd28003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd28004800 0x7fbd28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd28005210 0x7fbd2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440cd300 con 0x7fbd28005210 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2800c160 con 0x7fbd28003a00 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2800c360 con 0x7fbd28004800 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.598+0000 7fbd48a00640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd28005210 0x7fbd2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.599+0000 7fbd48a00640 1 -- 172.21.15.6:0/3449395127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd28004800 msgr2=0x7fbd28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.599+0000 7fbd48a00640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd28004800 0x7fbd28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.599+0000 7fbd48a00640 1 -- 172.21.15.6:0/3449395127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28003a00 msgr2=0x7fbd28003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.599+0000 7fbd48a00640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28003a00 0x7fbd28003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.599+0000 7fbd48a00640 1 -- 172.21.15.6:0/3449395127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd28003520 con 0x7fbd28005210 2024-03-06T07:28:34.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.600+0000 7fbd48a00640 1 -- 172.21.15.6:0/3449395127 reap_dead start 2024-03-06T07:28:34.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.600+0000 7fbd48a00640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd28005210 0x7fbd2800b9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00a760 tx=0x7fbd28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f428e7698500d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.600+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c00a3b0 con 0x7fbd28005210 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.600+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c034630 con 0x7fbd28005210 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.600+0000 7fbd43400640 1 -- 172.21.15.6:0/3449395127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c034a20 con 0x7fbd28005210 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 -- 172.21.15.6:0/3449395127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd28005210 msgr2=0x7fbd2800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd28005210 0x7fbd2800b9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00a760 tx=0x7fbd28005830 comp rx=0 tx=0).stop 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 -- 172.21.15.6:0/3449395127 shutdown_connections 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 --2- 172.21.15.6:0/3449395127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd28005210 0x7fbd2800b9c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 -- 172.21.15.6:0/3449395127 >> 172.21.15.6:0/3449395127 conn(0x7fbd4406b3d0 msgr2=0x7fbd4406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 -- 172.21.15.6:0/3449395127 shutdown_connections 2024-03-06T07:28:34.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.601+0000 7fbd49400640 1 -- 172.21.15.6:0/3449395127 wait complete. 2024-03-06T07:28:34.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.603+0000 7fbd49400640 1 Processor -- start 2024-03-06T07:28:34.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.603+0000 7fbd49400640 1 -- start start 2024-03-06T07:28:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd49400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28005210 0x7fbd441540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd49400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd441f7080 0x7fbd441f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd49400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 0x7fbd4406dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd49400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440cc950 con 0x7fbd4406d7d0 2024-03-06T07:28:34.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd49400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd441456e0 con 0x7fbd28005210 2024-03-06T07:28:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd49400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd440ccad0 con 0x7fbd441f7080 2024-03-06T07:28:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd48a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 0x7fbd4406dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd48a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 0x7fbd4406dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55074/0 (socket says 172.21.15.6:55074) 2024-03-06T07:28:34.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.604+0000 7fbd48a00640 1 -- 172.21.15.6:0/2106107498 learned_addr learned my addr 172.21.15.6:0/2106107498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd43e00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28005210 0x7fbd441540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd48a00640 1 -- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd441f7080 msgr2=0x7fbd441f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd48a00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd441f7080 0x7fbd441f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd48a00640 1 -- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28005210 msgr2=0x7fbd441540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd48a00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28005210 0x7fbd441540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd48a00640 1 -- 172.21.15.6:0/2106107498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd440af1d0 con 0x7fbd4406d7d0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd48a00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 0x7fbd4406dc80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00a760 tx=0x7fbd44071980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c043070 con 0x7fbd4406d7d0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3c004b60 con 0x7fbd4406d7d0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c039040 con 0x7fbd4406d7d0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.605+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd440ae650 con 0x7fbd4406d7d0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.606+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd440aec10 con 0x7fbd4406d7d0 2024-03-06T07:28:34.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.607+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4404f330 con 0x7fbd4406d7d0 2024-03-06T07:28:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.608+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd3c034630 con 0x7fbd4406d7d0 2024-03-06T07:28:34.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.609+0000 7fbd40c00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.609+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbd3c0ab1b0 con 0x7fbd4406d7d0 2024-03-06T07:28:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.609+0000 7fbd43e00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.611+0000 7fbd43e00640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd34001640 tx=0x7fbd34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.614+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd3c07a370 con 0x7fbd4406d7d0 2024-03-06T07:28:34.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.887+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fbd44072760 con 0x7fbd4406d7d0 2024-03-06T07:28:34.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.892+0000 7fbd40c00640 1 -- 172.21.15.6:0/2106107498 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fbd3c079c70 con 0x7fbd4406d7d0 2024-03-06T07:28:34.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 msgr2=0x7fbd2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd34001640 tx=0x7fbd34009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 msgr2=0x7fbd4406dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 0x7fbd4406dc80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c00a760 tx=0x7fbd44071980 comp rx=0 tx=0).stop 2024-03-06T07:28:34.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 shutdown_connections 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbd2006b500 0x7fbd2006d9c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd4406d7d0 0x7fbd4406dc80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbd441f7080 0x7fbd441f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 --2- 172.21.15.6:0/2106107498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbd28005210 0x7fbd441540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 >> 172.21.15.6:0/2106107498 conn(0x7fbd4406b3d0 msgr2=0x7fbd440b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 shutdown_connections 2024-03-06T07:28:34.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:34.895+0000 7fbd49400640 1 -- 172.21.15.6:0/2106107498 wait complete. 2024-03-06T07:28:34.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:28:35.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.137+0000 7fb232a00640 1 Processor -- start 2024-03-06T07:28:35.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.137+0000 7fb232a00640 1 -- start start 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb232a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb22c0af1a0 0x7fb22c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb232a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb22c0cdc20 0x7fb22c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb232a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c15cc40 con 0x7fb22c0af580 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb232a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22c15cf20 con 0x7fb22c0cdc20 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb232a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22c15d200 con 0x7fb22c0af1a0 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb230a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb22c0cdc20 0x7fb22c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb230a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb22c0cdc20 0x7fb22c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55082/0 (socket says 172.21.15.6:55082) 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb230a00640 1 -- 172.21.15.6:0/2062503281 learned_addr learned my addr 172.21.15.6:0/2062503281 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.139+0000 7fb22be00640 1 --2- 172.21.15.6:0/2062503281 >> v2:172.21.15.178:3300/0 conn(0x7fb22c0af1a0 0x7fb22c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.140+0000 7fb230a00640 1 -- 172.21.15.6:0/2062503281 >> v2:172.21.15.178:3300/0 conn(0x7fb22c0af1a0 msgr2=0x7fb22c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.140+0000 7fb230a00640 1 --2- 172.21.15.6:0/2062503281 >> v2:172.21.15.178:3300/0 conn(0x7fb22c0af1a0 0x7fb22c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.140+0000 7fb230a00640 1 -- 172.21.15.6:0/2062503281 >> v1:172.21.15.178:6789/0 conn(0x7fb22c0af580 legacy=0x7fb22c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.140+0000 7fb230a00640 1 -- 172.21.15.6:0/2062503281 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c15d4e0 con 0x7fb22c0cdc20 2024-03-06T07:28:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.141+0000 7fb230a00640 1 --2- 172.21.15.6:0/2062503281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0cdc20 0x7fb22c0b2dd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb22c074960 tx=0x7fb22402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f6682f688bd750 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.141+0000 7fb22aa00640 1 -- 172.21.15.6:0/2062503281 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22402c9f0 con 0x7fb22c0cdc20 2024-03-06T07:28:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.141+0000 7fb22aa00640 1 -- 172.21.15.6:0/2062503281 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22402cb90 con 0x7fb22c0cdc20 2024-03-06T07:28:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.141+0000 7fb22aa00640 1 -- 172.21.15.6:0/2062503281 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2240343f0 con 0x7fb22c0cdc20 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 -- 172.21.15.6:0/2062503281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0cdc20 msgr2=0x7fb22c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 --2- 172.21.15.6:0/2062503281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0cdc20 0x7fb22c0b2dd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb22c074960 tx=0x7fb22402be50 comp rx=0 tx=0).stop 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 -- 172.21.15.6:0/2062503281 shutdown_connections 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 --2- 172.21.15.6:0/2062503281 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0cdc20 0x7fb22c0b2dd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 --2- 172.21.15.6:0/2062503281 >> v2:172.21.15.178:3300/0 conn(0x7fb22c0af1a0 0x7fb22c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 -- 172.21.15.6:0/2062503281 >> 172.21.15.6:0/2062503281 conn(0x7fb22c06afb0 msgr2=0x7fb22c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 -- 172.21.15.6:0/2062503281 shutdown_connections 2024-03-06T07:28:35.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.142+0000 7fb232a00640 1 -- 172.21.15.6:0/2062503281 wait complete. 2024-03-06T07:28:35.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.144+0000 7fb232a00640 1 Processor -- start 2024-03-06T07:28:35.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.144+0000 7fb232a00640 1 -- start start 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb232a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 0x7fb22c06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb232a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb22c0af580 0x7fb22c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb232a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb22c0cdc20 0x7fb22c15edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb232a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0b36a0 con 0x7fb22c0af1a0 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb232a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0b3520 con 0x7fb22c0af580 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb232a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0b3820 con 0x7fb22c0cdc20 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 0x7fb22c06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 0x7fb22c06dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55092/0 (socket says 172.21.15.6:55092) 2024-03-06T07:28:35.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 -- 172.21.15.6:0/4020866548 learned_addr learned my addr 172.21.15.6:0/4020866548 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22b400640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb22c0af580 0x7fb22c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 -- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb22c0cdc20 msgr2=0x7fb22c15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb22c0cdc20 0x7fb22c15edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 -- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb22c0af580 msgr2=0x7fb22c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb22c0af580 0x7fb22c15e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.145+0000 7fb22be00640 1 -- 172.21.15.6:0/4020866548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c0c93a0 con 0x7fb22c0af1a0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.146+0000 7fb22be00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 0x7fb22c06dba0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00d830 tx=0x7fb21c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.146+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00b6c0 con 0x7fb22c0af1a0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.146+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21c019070 con 0x7fb22c0af1a0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.146+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c0c9680 con 0x7fb22c0af1a0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.146+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00ba80 con 0x7fb22c0af1a0 2024-03-06T07:28:35.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.146+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c0c9c40 con 0x7fb22c0af1a0 2024-03-06T07:28:35.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.147+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c069510 con 0x7fb22c0af1a0 2024-03-06T07:28:35.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.148+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb21c01e460 con 0x7fb22c0af1a0 2024-03-06T07:28:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.149+0000 7fb223e00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb20c06b520 0x7fb20c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.149+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb21c08a410 con 0x7fb22c0af1a0 2024-03-06T07:28:35.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.150+0000 7fb22b400640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb20c06b520 0x7fb20c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.151+0000 7fb22b400640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb20c06b520 0x7fb20c06d9e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb22c06d8c0 tx=0x7fb218009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.154+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb21c059750 con 0x7fb22c0af1a0 2024-03-06T07:28:35.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.426+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fb22c04f3a0 con 0x7fb22c0af1a0 2024-03-06T07:28:35.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.430+0000 7fb223e00640 1 -- 172.21.15.6:0/4020866548 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb21c059050 con 0x7fb22c0af1a0 2024-03-06T07:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.432+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb20c06b520 msgr2=0x7fb20c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.432+0000 7fb232a00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb20c06b520 0x7fb20c06d9e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb22c06d8c0 tx=0x7fb218009040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.432+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 msgr2=0x7fb22c06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.432+0000 7fb232a00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 0x7fb22c06dba0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00d830 tx=0x7fb21c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 shutdown_connections 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb20c06b520 0x7fb20c06d9e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb22c0af580 0x7fb22c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb22c0cdc20 0x7fb22c15edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 --2- 172.21.15.6:0/4020866548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb22c0af1a0 0x7fb22c06dba0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 >> 172.21.15.6:0/4020866548 conn(0x7fb22c06afb0 msgr2=0x7fb22c15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 shutdown_connections 2024-03-06T07:28:35.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.433+0000 7fb232a00640 1 -- 172.21.15.6:0/4020866548 wait complete. 2024-03-06T07:28:35.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:28:35.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.665+0000 7f09c2800640 1 Processor -- start 2024-03-06T07:28:35.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.665+0000 7f09c2800640 1 -- start start 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c2800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09bc145300 0x7f09bc0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c2800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09bc0cce90 0x7f09bc0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c2800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc0cd300 con 0x7f09bc0cce90 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c2800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc0714d0 con 0x7f09bc145300 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c2800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0717b0 con 0x7f09bc070f10 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09bc0cce90 0x7f09bc0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09bc0cce90 0x7f09bc0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55108/0 (socket says 172.21.15.6:55108) 2024-03-06T07:28:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c1e00640 1 -- 172.21.15.6:0/1040217092 learned_addr learned my addr 172.21.15.6:0/1040217092 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.667+0000 7f09c1400640 1 --2- 172.21.15.6:0/1040217092 >> v2:172.21.15.178:3300/0 conn(0x7f09bc145300 0x7f09bc0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c1e00640 1 -- 172.21.15.6:0/1040217092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 msgr2=0x7f09bc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c0a00640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c1e00640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c1e00640 1 -- 172.21.15.6:0/1040217092 >> v2:172.21.15.178:3300/0 conn(0x7f09bc145300 msgr2=0x7f09bc0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c1e00640 1 --2- 172.21.15.6:0/1040217092 >> v2:172.21.15.178:3300/0 conn(0x7f09bc145300 0x7f09bc0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c1e00640 1 -- 172.21.15.6:0/1040217092 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc071a90 con 0x7f09bc0cce90 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c0a00640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.668+0000 7f09c1400640 1 --2- 172.21.15.6:0/1040217092 >> v2:172.21.15.178:3300/0 conn(0x7f09bc145300 0x7f09bc0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.669+0000 7f09c1e00640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0cd930 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0cf990 tx=0x7f09b402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd1ccfdce8f13f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.669+0000 7f09bbe00640 1 -- 172.21.15.6:0/1040217092 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b402ca80 con 0x7f09bc0cce90 2024-03-06T07:28:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.669+0000 7f09bbe00640 1 -- 172.21.15.6:0/1040217092 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b402cc20 con 0x7f09bc0cce90 2024-03-06T07:28:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.669+0000 7f09bbe00640 1 -- 172.21.15.6:0/1040217092 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b40343f0 con 0x7f09bc0cce90 2024-03-06T07:28:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 -- 172.21.15.6:0/1040217092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 msgr2=0x7f09bc0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0cd930 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0cf990 tx=0x7f09b402bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 -- 172.21.15.6:0/1040217092 shutdown_connections 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0cd930 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 --2- 172.21.15.6:0/1040217092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 --2- 172.21.15.6:0/1040217092 >> v2:172.21.15.178:3300/0 conn(0x7f09bc145300 0x7f09bc0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 -- 172.21.15.6:0/1040217092 >> 172.21.15.6:0/1040217092 conn(0x7f09bc06b3d0 msgr2=0x7f09bc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 -- 172.21.15.6:0/1040217092 shutdown_connections 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.670+0000 7f09c2800640 1 -- 172.21.15.6:0/1040217092 wait complete. 2024-03-06T07:28:35.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.672+0000 7f09c2800640 1 Processor -- start 2024-03-06T07:28:35.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.672+0000 7f09c2800640 1 -- start start 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c2800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c2800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09bc145300 0x7f09bc15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c2800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc1456e0 con 0x7f09bc0cce90 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c2800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0b4ee0 con 0x7f09bc145300 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c2800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0cde70 con 0x7f09bc070f10 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0ca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55124/0 (socket says 172.21.15.6:55124) 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 -- 172.21.15.6:0/747809212 learned_addr learned my addr 172.21.15.6:0/747809212 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 -- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 msgr2=0x7f09bc144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 -- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09bc145300 msgr2=0x7f09bc15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09bc145300 0x7f09bc15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.673+0000 7f09c0a00640 1 -- 172.21.15.6:0/747809212 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0c81c0 con 0x7f09bc0cce90 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.674+0000 7f09c0a00640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0ca510 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c820 tx=0x7f09ac00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.674+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac015070 con 0x7f09bc0cce90 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.674+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac007a30 con 0x7f09bc0cce90 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.674+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac007e00 con 0x7f09bc0cce90 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.674+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc0c84a0 con 0x7f09bc0cce90 2024-03-06T07:28:35.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.674+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc0c8a60 con 0x7f09bc0cce90 2024-03-06T07:28:35.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.675+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc04f330 con 0x7f09bc0cce90 2024-03-06T07:28:35.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.676+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09ac018070 con 0x7f09bc0cce90 2024-03-06T07:28:35.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.677+0000 7f09b9600640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09a006b4d0 0x7f09a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.677+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09ac089ea0 con 0x7f09bc0cce90 2024-03-06T07:28:35.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.678+0000 7f09c1400640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09a006b4d0 0x7f09a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.679+0000 7f09c1400640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09a006b4d0 0x7f09a006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09b0006b10 tx=0x7f09b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.682+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09ac059060 con 0x7f09bc0cce90 2024-03-06T07:28:35.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.951+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f09bc074ab0 con 0x7f09bc0cce90 2024-03-06T07:28:35.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.955+0000 7f09b9600640 1 -- 172.21.15.6:0/747809212 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f09ac058960 con 0x7f09bc0cce90 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09a006b4d0 msgr2=0x7f09a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09a006b4d0 0x7f09a006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f09b0006b10 tx=0x7f09b0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 msgr2=0x7f09bc0ca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0ca510 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c820 tx=0x7f09ac00cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 shutdown_connections 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09a006b4d0 0x7f09a006d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09bc145300 0x7f09bc15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09bc0cce90 0x7f09bc0ca510 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 --2- 172.21.15.6:0/747809212 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09bc070f10 0x7f09bc144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 >> 172.21.15.6:0/747809212 conn(0x7f09bc06b3d0 msgr2=0x7f09bc0725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 shutdown_connections 2024-03-06T07:28:35.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:35.957+0000 7f09c2800640 1 -- 172.21.15.6:0/747809212 wait complete. 2024-03-06T07:28:35.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:28:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.192+0000 7f745a000640 1 Processor -- start 2024-03-06T07:28:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.192+0000 7f745a000640 1 -- start start 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f745a000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af1a0 0x7f745415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f745a000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f74540af580 0x7f74540b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f745a000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74540cdc20 0x7f74540b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f745a000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745415cc40 con 0x7f74540cdc20 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f745a000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745415cf20 con 0x7f74540af580 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f745a000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745415d200 con 0x7f74540af1a0 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f7453e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74540cdc20 0x7f74540b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f7453e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74540cdc20 0x7f74540b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55128/0 (socket says 172.21.15.6:55128) 2024-03-06T07:28:36.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f7453e00640 1 -- 172.21.15.6:0/937641312 learned_addr learned my addr 172.21.15.6:0/937641312 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f7453400640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af1a0 0x7f745415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.194+0000 7f744aa00640 1 --2- 172.21.15.6:0/937641312 >> v2:172.21.15.178:3300/0 conn(0x7f74540af580 0x7f74540b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f7453e00640 1 -- 172.21.15.6:0/937641312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af1a0 msgr2=0x7f745415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f7453e00640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af1a0 0x7f745415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f7453e00640 1 -- 172.21.15.6:0/937641312 >> v2:172.21.15.178:3300/0 conn(0x7f74540af580 msgr2=0x7f74540b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f7453e00640 1 --2- 172.21.15.6:0/937641312 >> v2:172.21.15.178:3300/0 conn(0x7f74540af580 0x7f74540b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f7453e00640 1 -- 172.21.15.6:0/937641312 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745415d4e0 con 0x7f74540cdc20 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f7453400640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af1a0 0x7f745415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.195+0000 7f744aa00640 1 --2- 172.21.15.6:0/937641312 >> v2:172.21.15.178:3300/0 conn(0x7f74540af580 0x7f74540b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.196+0000 7f7453e00640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540cdc20 0x7f74540b2dd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f74540748c0 tx=0x7f744c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4c204c18893bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.196+0000 7f7452a00640 1 -- 172.21.15.6:0/937641312 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c02c9f0 con 0x7f74540cdc20 2024-03-06T07:28:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.196+0000 7f7452a00640 1 -- 172.21.15.6:0/937641312 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c02cb90 con 0x7f74540cdc20 2024-03-06T07:28:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.196+0000 7f7452a00640 1 -- 172.21.15.6:0/937641312 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0343f0 con 0x7f74540cdc20 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 -- 172.21.15.6:0/937641312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540cdc20 msgr2=0x7f74540b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540cdc20 0x7f74540b2dd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f74540748c0 tx=0x7f744c02be50 comp rx=0 tx=0).stop 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 -- 172.21.15.6:0/937641312 shutdown_connections 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540cdc20 0x7f74540b2dd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 --2- 172.21.15.6:0/937641312 >> v2:172.21.15.178:3300/0 conn(0x7f74540af580 0x7f74540b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 --2- 172.21.15.6:0/937641312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af1a0 0x7f745415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 -- 172.21.15.6:0/937641312 >> 172.21.15.6:0/937641312 conn(0x7f745406afb0 msgr2=0x7f745406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 -- 172.21.15.6:0/937641312 shutdown_connections 2024-03-06T07:28:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.197+0000 7f745a000640 1 -- 172.21.15.6:0/937641312 wait complete. 2024-03-06T07:28:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.199+0000 7f745a000640 1 Processor -- start 2024-03-06T07:28:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.199+0000 7f745a000640 1 -- start start 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f745a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 0x7f745414c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f745a000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af580 0x7f745409d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f745a000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74540cdc20 0x7f745414bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f745a000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74540b3490 con 0x7f74540af1a0 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f745a000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74540b3310 con 0x7f74540cdc20 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f745a000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74540b3610 con 0x7f74540af580 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f7453400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 0x7f745414c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f7453400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 0x7f745414c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55134/0 (socket says 172.21.15.6:55134) 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f7453400640 1 -- 172.21.15.6:0/1936017402 learned_addr learned my addr 172.21.15.6:0/1936017402 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f744aa00640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af580 0x7f745409d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.200+0000 7f7453e00640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74540cdc20 0x7f745414bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453400640 1 -- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af580 msgr2=0x7f745409d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453400640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af580 0x7f745409d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453400640 1 -- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74540cdc20 msgr2=0x7f745414bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453400640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74540cdc20 0x7f745414bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453400640 1 -- 172.21.15.6:0/1936017402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74541f73f0 con 0x7f74540af1a0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453e00640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74540cdc20 0x7f745414bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f7453400640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 0x7f745414c970 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f744400c7c0 tx=0x7f744400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7444015070 con 0x7f74540af1a0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7444007a30 con 0x7f74540af1a0 2024-03-06T07:28:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7444007e00 con 0x7f74540af1a0 2024-03-06T07:28:36.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74541f76d0 con 0x7f74540af1a0 2024-03-06T07:28:36.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.201+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74541f7c90 con 0x7f74540af1a0 2024-03-06T07:28:36.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.203+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7454069510 con 0x7f74540af1a0 2024-03-06T07:28:36.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.204+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7444018070 con 0x7f74540af1a0 2024-03-06T07:28:36.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.204+0000 7f744be00640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743006b4d0 0x7f743006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.204+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7444089ec0 con 0x7f74540af1a0 2024-03-06T07:28:36.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.205+0000 7f744aa00640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743006b4d0 0x7f743006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.206+0000 7f744aa00640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743006b4d0 0x7f743006d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f745415b930 tx=0x7f743c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.209+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7444058ff0 con 0x7f74540af1a0 2024-03-06T07:28:36.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.478+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f745415c700 con 0x7f74540af1a0 2024-03-06T07:28:36.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.483+0000 7f744be00640 1 -- 172.21.15.6:0/1936017402 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f74440588f0 con 0x7f74540af1a0 2024-03-06T07:28:36.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743006b4d0 msgr2=0x7f743006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743006b4d0 0x7f743006d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f745415b930 tx=0x7f743c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 msgr2=0x7f745414c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 0x7f745414c970 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f744400c7c0 tx=0x7f744400cc80 comp rx=0 tx=0).stop 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 shutdown_connections 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f743006b4d0 0x7f743006d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f74540cdc20 0x7f745414bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f74540af580 0x7f745409d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 --2- 172.21.15.6:0/1936017402 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74540af1a0 0x7f745414c970 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 >> 172.21.15.6:0/1936017402 conn(0x7f745406afb0 msgr2=0x7f745406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 shutdown_connections 2024-03-06T07:28:36.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.486+0000 7f745a000640 1 -- 172.21.15.6:0/1936017402 wait complete. 2024-03-06T07:28:36.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.717+0000 7fbbff400640 1 Processor -- start 2024-03-06T07:28:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.717+0000 7fbbff400640 1 -- start start 2024-03-06T07:28:36.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.719+0000 7fbbff400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbbf80af580 0x7fbbf815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.719+0000 7fbbff400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf815cc40 con 0x7fbbf80af1a0 2024-03-06T07:28:36.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.719+0000 7fbbff400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf815cf20 con 0x7fbbf80cdc20 2024-03-06T07:28:36.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.719+0000 7fbbff400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf815d200 con 0x7fbbf80af580 2024-03-06T07:28:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.719+0000 7fbbfca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbbf80af1a0 0x7fbbf80b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44590/0 (socket says 172.21.15.6:44590) 2024-03-06T07:28:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.719+0000 7fbbfca00640 1 -- 172.21.15.6:0/2589603124 learned_addr learned my addr 172.21.15.6:0/2589603124 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/2589603124 >> v2:172.21.15.178:3300/0 conn(0x7fbbf80af580 0x7fbbf815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7400640 1 -- 172.21.15.6:0/2589603124 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622734590 0 0) 0x7fbbf815cc40 con 0x7fbbf80af1a0 2024-03-06T07:28:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7400640 1 -- 172.21.15.6:0/2589603124 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe0003520 con 0x7fbbf80af1a0 2024-03-06T07:28:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7400640 1 -- 172.21.15.6:0/2589603124 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2675353928 0 0) 0x7fbbe0003520 con 0x7fbbf80af1a0 2024-03-06T07:28:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7e00640 1 -- 172.21.15.6:0/2589603124 >> v1:172.21.15.178:6789/0 conn(0x7fbbf80cdc20 legacy=0x7fbbf80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7e00640 1 -- 172.21.15.6:0/2589603124 >> v1:172.21.15.6:6789/0 conn(0x7fbbf80af1a0 legacy=0x7fbbf80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.720+0000 7fbbf7e00640 1 -- 172.21.15.6:0/2589603124 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf815d4e0 con 0x7fbbf80af580 2024-03-06T07:28:36.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.722+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/2589603124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af580 0x7fbbf815c600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40097a0 tx=0x7fbbe402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3b0d7230f387256 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.722+0000 7fbbf7400640 1 -- 172.21.15.6:0/2589603124 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4037070 con 0x7fbbf80af580 2024-03-06T07:28:36.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.722+0000 7fbbf7400640 1 -- 172.21.15.6:0/2589603124 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe4030510 con 0x7fbbf80af580 2024-03-06T07:28:36.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.722+0000 7fbbf7400640 1 -- 172.21.15.6:0/2589603124 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec001670 con 0x7fbbf80af580 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 -- 172.21.15.6:0/2589603124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af580 msgr2=0x7fbbf815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 --2- 172.21.15.6:0/2589603124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af580 0x7fbbf815c600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40097a0 tx=0x7fbbe402b790 comp rx=0 tx=0).stop 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 -- 172.21.15.6:0/2589603124 shutdown_connections 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 --2- 172.21.15.6:0/2589603124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af580 0x7fbbf815c600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 -- 172.21.15.6:0/2589603124 >> 172.21.15.6:0/2589603124 conn(0x7fbbf806afb0 msgr2=0x7fbbf806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 -- 172.21.15.6:0/2589603124 shutdown_connections 2024-03-06T07:28:36.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.723+0000 7fbbff400640 1 -- 172.21.15.6:0/2589603124 wait complete. 2024-03-06T07:28:36.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.724+0000 7fbbff400640 1 Processor -- start 2024-03-06T07:28:36.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.725+0000 7fbbff400640 1 -- start start 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.725+0000 7fbbff400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af1a0 0x7fbbf8148550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.725+0000 7fbbff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 0x7fbbf809d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbff400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf80cdc20 0x7fbbf809b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbff400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8155e70 con 0x7fbbf80af580 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbff400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf80b3730 con 0x7fbbf80af1a0 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbff400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8155ff0 con 0x7fbbf80cdc20 2024-03-06T07:28:36.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 0x7fbbf809d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 0x7fbbf809d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55144/0 (socket says 172.21.15.6:55144) 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 -- 172.21.15.6:0/1690288710 learned_addr learned my addr 172.21.15.6:0/1690288710 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbfca00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af1a0 0x7fbbf8148550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 -- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf80cdc20 msgr2=0x7fbbf809b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbfd400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf80cdc20 0x7fbbf809b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf80cdc20 0x7fbbf809b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 -- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af1a0 msgr2=0x7fbbf8148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af1a0 0x7fbbf8148550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.726+0000 7fbbf7e00640 1 -- 172.21.15.6:0/1690288710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf806cc40 con 0x7fbbf80af580 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.727+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 0x7fbbf809d220 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40098c0 tx=0x7fbbe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.727+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4042070 con 0x7fbbf80af580 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.727+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe403e870 con 0x7fbbf80af580 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.727+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4037040 con 0x7fbbf80af580 2024-03-06T07:28:36.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.727+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf806cf20 con 0x7fbbf80af580 2024-03-06T07:28:36.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.727+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf806d4e0 con 0x7fbbf80af580 2024-03-06T07:28:36.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.728+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf80b38b0 con 0x7fbbf80af580 2024-03-06T07:28:36.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.729+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbe4045070 con 0x7fbbf80af580 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.730+0000 7fbbf4c00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd806b4d0 0x7fbbd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.730+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbbe40abae0 con 0x7fbbf80af580 2024-03-06T07:28:36.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.731+0000 7fbbfca00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd806b4d0 0x7fbbd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.732+0000 7fbbfca00640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd806b4d0 0x7fbbd806d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbbec005880 tx=0x7fbbec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.735+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbf80b38b0 con 0x7fbbf80af580 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.934+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbf8155870 con 0x7fbbf80af580 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.937+0000 7fbbf4c00640 1 -- 172.21.15.6:0/1690288710 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12819 (secure 0 0 0) 0x7fbbe407adc0 con 0x7fbbf80af580 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":31,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:32.785522+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd806b4d0 msgr2=0x7fbbd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd806b4d0 0x7fbbd806d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbbec005880 tx=0x7fbbec009040 comp rx=0 tx=0).stop 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 msgr2=0x7fbbf809d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 0x7fbbf809d220 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40098c0 tx=0x7fbbe40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 shutdown_connections 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd806b4d0 0x7fbbd806d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf80cdc20 0x7fbbf809b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf80af580 0x7fbbf809d220 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 --2- 172.21.15.6:0/1690288710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf80af1a0 0x7fbbf8148550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.939+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 >> 172.21.15.6:0/1690288710 conn(0x7fbbf806afb0 msgr2=0x7fbbf80ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.940+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 shutdown_connections 2024-03-06T07:28:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:36.940+0000 7fbbff400640 1 -- 172.21.15.6:0/1690288710 wait complete. 2024-03-06T07:28:36.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:36.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:37.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.179+0000 7f9ddd200640 1 Processor -- start 2024-03-06T07:28:37.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.179+0000 7f9ddd200640 1 -- start start 2024-03-06T07:28:37.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9ddd200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9dd80cd1e0 0x7f9dd80cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9ddd200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9dd8070ce0 0x7f9dd80b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9ddd200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd8071c60 con 0x7f9dd815afa0 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9ddd200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd80cfa10 con 0x7f9dd80cd1e0 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9ddd200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd8071f40 con 0x7f9dd8070ce0 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9dd5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9dd80cd1e0 0x7f9dd80cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9dd5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9dd80cd1e0 0x7f9dd80cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55154/0 (socket says 172.21.15.6:55154) 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9dd5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9dd815afa0 0x7f9dd80ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44602/0 (socket says 172.21.15.6:44602) 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.181+0000 7f9dd5400640 1 -- 172.21.15.6:0/2273802716 learned_addr learned my addr 172.21.15.6:0/2273802716 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.182+0000 7f9dd6800640 1 --2- 172.21.15.6:0/2273802716 >> v2:172.21.15.178:3300/0 conn(0x7f9dd8070ce0 0x7f9dd80b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.182+0000 7f9dd5400640 1 -- 172.21.15.6:0/2273802716 >> v2:172.21.15.178:3300/0 conn(0x7f9dd8070ce0 msgr2=0x7f9dd80b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.182+0000 7f9dd5400640 1 --2- 172.21.15.6:0/2273802716 >> v2:172.21.15.178:3300/0 conn(0x7f9dd8070ce0 0x7f9dd80b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.182+0000 7f9dd5400640 1 -- 172.21.15.6:0/2273802716 >> v1:172.21.15.6:6789/0 conn(0x7f9dd815afa0 legacy=0x7f9dd80ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.182+0000 7f9dd5400640 1 -- 172.21.15.6:0/2273802716 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd8072220 con 0x7f9dd80cd1e0 2024-03-06T07:28:37.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.183+0000 7f9dd5400640 1 --2- 172.21.15.6:0/2273802716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd80cd1e0 0x7f9dd80cf3a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80b52b0 tx=0x7f9dc002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11edf29471ea7078 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.183+0000 7f9dd4a00640 1 -- 172.21.15.6:0/2273802716 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc003b070 con 0x7f9dd80cd1e0 2024-03-06T07:28:37.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.183+0000 7f9dd4a00640 1 -- 172.21.15.6:0/2273802716 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc0036440 con 0x7f9dd80cd1e0 2024-03-06T07:28:37.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.183+0000 7f9dd4a00640 1 -- 172.21.15.6:0/2273802716 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0036810 con 0x7f9dd80cd1e0 2024-03-06T07:28:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 -- 172.21.15.6:0/2273802716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd80cd1e0 msgr2=0x7f9dd80cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 --2- 172.21.15.6:0/2273802716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd80cd1e0 0x7f9dd80cf3a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80b52b0 tx=0x7f9dc002e0f0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 -- 172.21.15.6:0/2273802716 shutdown_connections 2024-03-06T07:28:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 --2- 172.21.15.6:0/2273802716 >> v2:172.21.15.178:3300/0 conn(0x7f9dd8070ce0 0x7f9dd80b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 --2- 172.21.15.6:0/2273802716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd80cd1e0 0x7f9dd80cf3a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 -- 172.21.15.6:0/2273802716 >> 172.21.15.6:0/2273802716 conn(0x7f9dd806b3d0 msgr2=0x7f9dd806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 -- 172.21.15.6:0/2273802716 shutdown_connections 2024-03-06T07:28:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.184+0000 7f9ddd200640 1 -- 172.21.15.6:0/2273802716 wait complete. 2024-03-06T07:28:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.186+0000 7f9ddd200640 1 Processor -- start 2024-03-06T07:28:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.186+0000 7f9ddd200640 1 -- start start 2024-03-06T07:28:37.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9ddd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 0x7f9dd809bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9ddd200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd80cd1e0 0x7f9dd81f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9ddd200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd815afa0 0x7f9dd81f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9ddd200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80b4000 con 0x7f9dd8070ce0 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9ddd200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd8071110 con 0x7f9dd815afa0 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9ddd200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80b4180 con 0x7f9dd80cd1e0 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.187+0000 7f9dd5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 0x7f9dd809bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 0x7f9dd809bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55164/0 (socket says 172.21.15.6:55164) 2024-03-06T07:28:37.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 -- 172.21.15.6:0/3214872793 learned_addr learned my addr 172.21.15.6:0/3214872793 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:37.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5400640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd80cd1e0 0x7f9dd81f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd6800640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd815afa0 0x7f9dd81f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 -- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd80cd1e0 msgr2=0x7f9dd81f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd80cd1e0 0x7f9dd81f3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 -- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd815afa0 msgr2=0x7f9dd81f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd815afa0 0x7f9dd81f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 -- 172.21.15.6:0/3214872793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd809abb0 con 0x7f9dd8070ce0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.188+0000 7f9dd5e00640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 0x7f9dd809bc10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c730 tx=0x7f9dc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.189+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8017070 con 0x7f9dd8070ce0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.189+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc800ce40 con 0x7f9dd8070ce0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.189+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc800a860 con 0x7f9dd8070ce0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.189+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd809ae50 con 0x7f9dd8070ce0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.189+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd809b410 con 0x7f9dd8070ce0 2024-03-06T07:28:37.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.190+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd804f330 con 0x7f9dd8070ce0 2024-03-06T07:28:37.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.191+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dc801a070 con 0x7f9dd8070ce0 2024-03-06T07:28:37.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.192+0000 7f9dce000640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db406b4d0 0x7f9db406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.192+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9dc80894e0 con 0x7f9dd8070ce0 2024-03-06T07:28:37.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.192+0000 7f9dd5400640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db406b4d0 0x7f9db406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.193+0000 7f9dd5400640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db406b4d0 0x7f9db406d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0038000 tx=0x7f9dc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.196+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dc80586b0 con 0x7f9dd8070ce0 2024-03-06T07:28:37.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.394+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9dd80cf110 con 0x7f9dd8070ce0 2024-03-06T07:28:37.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.395+0000 7f9dce000640 1 -- 172.21.15.6:0/3214872793 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9dc8057fb0 con 0x7f9dd8070ce0 2024-03-06T07:28:37.398 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:28:37.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db406b4d0 msgr2=0x7f9db406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db406b4d0 0x7f9db406d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0038000 tx=0x7f9dc0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 msgr2=0x7f9dd809bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 0x7f9dd809bc10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800c730 tx=0x7f9dc800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 shutdown_connections 2024-03-06T07:28:37.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db406b4d0 0x7f9db406d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd815afa0 0x7f9dd81f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd80cd1e0 0x7f9dd81f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 --2- 172.21.15.6:0/3214872793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8070ce0 0x7f9dd809bc10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 >> 172.21.15.6:0/3214872793 conn(0x7f9dd806b3d0 msgr2=0x7f9dd80ce6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 shutdown_connections 2024-03-06T07:28:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.398+0000 7f9ddd200640 1 -- 172.21.15.6:0/3214872793 wait complete. 2024-03-06T07:28:37.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:37.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.634+0000 7fb91b000640 1 Processor -- start 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.634+0000 7fb91b000640 1 -- start start 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.635+0000 7fb91b000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb914145300 0x7fb9140b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.635+0000 7fb91b000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb914070f10 0x7fb9140cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.635+0000 7fb91b000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb9140cce90 0x7fb9140cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb91b000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9140cd300 con 0x7fb914070f10 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb91b000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9140714d0 con 0x7fb9140cce90 2024-03-06T07:28:37.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb91b000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140717b0 con 0x7fb914145300 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb914070f10 0x7fb9140cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb914070f10 0x7fb9140cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55176/0 (socket says 172.21.15.6:55176) 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 -- 172.21.15.6:0/3772893757 learned_addr learned my addr 172.21.15.6:0/3772893757 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919c00640 1 --2- 172.21.15.6:0/3772893757 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb914145300 0x7fb9140b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb91a600640 1 --2- 172.21.15.6:0/3772893757 >> v2:172.21.15.178:3300/0 conn(0x7fb9140cce90 0x7fb9140cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 -- 172.21.15.6:0/3772893757 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb914145300 msgr2=0x7fb9140b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 --2- 172.21.15.6:0/3772893757 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb914145300 0x7fb9140b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 -- 172.21.15.6:0/3772893757 >> v2:172.21.15.178:3300/0 conn(0x7fb9140cce90 msgr2=0x7fb9140cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 --2- 172.21.15.6:0/3772893757 >> v2:172.21.15.178:3300/0 conn(0x7fb9140cce90 0x7fb9140cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.636+0000 7fb919200640 1 -- 172.21.15.6:0/3772893757 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914071a90 con 0x7fb914070f10 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.638+0000 7fb919200640 1 --2- 172.21.15.6:0/3772893757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9140cc950 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb914072420 tx=0x7fb90002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec34414a8487c375 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.638+0000 7fb913e00640 1 -- 172.21.15.6:0/3772893757 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb900035020 con 0x7fb914070f10 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.638+0000 7fb913e00640 1 -- 172.21.15.6:0/3772893757 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb900031870 con 0x7fb914070f10 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.638+0000 7fb913e00640 1 -- 172.21.15.6:0/3772893757 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb900031c40 con 0x7fb914070f10 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.638+0000 7fb91b000640 1 -- 172.21.15.6:0/3772893757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 msgr2=0x7fb9140cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.638+0000 7fb91b000640 1 --2- 172.21.15.6:0/3772893757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9140cc950 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb914072420 tx=0x7fb90002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:37.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 -- 172.21.15.6:0/3772893757 shutdown_connections 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 --2- 172.21.15.6:0/3772893757 >> v2:172.21.15.178:3300/0 conn(0x7fb9140cce90 0x7fb9140cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 --2- 172.21.15.6:0/3772893757 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9140cc950 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 --2- 172.21.15.6:0/3772893757 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb914145300 0x7fb9140b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 -- 172.21.15.6:0/3772893757 >> 172.21.15.6:0/3772893757 conn(0x7fb91406b3d0 msgr2=0x7fb91406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 -- 172.21.15.6:0/3772893757 shutdown_connections 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.639+0000 7fb91b000640 1 -- 172.21.15.6:0/3772893757 wait complete. 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.640+0000 7fb91b000640 1 Processor -- start 2024-03-06T07:28:37.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 -- start start 2024-03-06T07:28:37.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9141f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9140cce90 0x7fb9140ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914145300 0x7fb9141f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141456e0 con 0x7fb914070f10 2024-03-06T07:28:37.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140b4ee0 con 0x7fb914145300 2024-03-06T07:28:37.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.641+0000 7fb91b000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140cde70 con 0x7fb9140cce90 2024-03-06T07:28:37.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.642+0000 7fb919c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9141f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.642+0000 7fb919c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9141f11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55180/0 (socket says 172.21.15.6:55180) 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.642+0000 7fb919c00640 1 -- 172.21.15.6:0/3958227078 learned_addr learned my addr 172.21.15.6:0/3958227078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.642+0000 7fb91a600640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914145300 0x7fb9141f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.642+0000 7fb919200640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9140cce90 0x7fb9140ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb919c00640 1 -- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9140cce90 msgr2=0x7fb9140ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb919c00640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9140cce90 0x7fb9140ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb919c00640 1 -- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914145300 msgr2=0x7fb9141f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb919c00640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914145300 0x7fb9141f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb919c00640 1 -- 172.21.15.6:0/3958227078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140af210 con 0x7fb914070f10 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb919c00640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9141f11d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb90800d5f0 tx=0x7fb90800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9080125f0 con 0x7fb914070f10 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb908019070 con 0x7fb914070f10 2024-03-06T07:28:37.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9140aec50 con 0x7fb914070f10 2024-03-06T07:28:37.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9080129d0 con 0x7fb914070f10 2024-03-06T07:28:37.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.643+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9141f50a0 con 0x7fb914070f10 2024-03-06T07:28:37.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.644+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91404f330 con 0x7fb914070f10 2024-03-06T07:28:37.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.645+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb908012b70 con 0x7fb914070f10 2024-03-06T07:28:37.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.646+0000 7fb911600640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8f406b4d0 0x7fb8f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.646+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb90808a230 con 0x7fb914070f10 2024-03-06T07:28:37.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.647+0000 7fb919200640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8f406b4d0 0x7fb8f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.648+0000 7fb919200640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8f406b4d0 0x7fb8f406d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb900036000 tx=0x7fb90003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.651+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9080593f0 con 0x7fb914070f10 2024-03-06T07:28:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.855+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb9140b09b0 con 0x7fb914070f10 2024-03-06T07:28:37.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.856+0000 7fb911600640 1 -- 172.21.15.6:0/3958227078 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb908058cf0 con 0x7fb914070f10 2024-03-06T07:28:37.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:37.859 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:28:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:28:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:28:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:28:37.859 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:28:37.859 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:28:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8f406b4d0 msgr2=0x7fb8f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8f406b4d0 0x7fb8f406d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb900036000 tx=0x7fb90003c040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 msgr2=0x7fb9141f11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9141f11d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb90800d5f0 tx=0x7fb90800dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 shutdown_connections 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8f406b4d0 0x7fb8f406d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914145300 0x7fb9141f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9140cce90 0x7fb9140ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 --2- 172.21.15.6:0/3958227078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914070f10 0x7fb9141f11d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 >> 172.21.15.6:0/3958227078 conn(0x7fb91406b3d0 msgr2=0x7fb9140b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 shutdown_connections 2024-03-06T07:28:37.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:37.859+0000 7fb91b000640 1 -- 172.21.15.6:0/3958227078 wait complete. 2024-03-06T07:28:37.880 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:37.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:38.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.100+0000 7fba33200640 1 Processor -- start 2024-03-06T07:28:38.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.100+0000 7fba33200640 1 -- start start 2024-03-06T07:28:38.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba33200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c145570 0x7fba2c0cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba33200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fba2c071450 0x7fba2c06daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba33200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fba2c06e030 0x7fba2c0ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba33200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2c06e4a0 con 0x7fba2c071450 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba33200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2c071c50 con 0x7fba2c06e030 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba33200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c071f30 con 0x7fba2c145570 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba31400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fba2c071450 0x7fba2c06daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.102+0000 7fba31400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fba2c071450 0x7fba2c06daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55192/0 (socket says 172.21.15.6:55192) 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31400640 1 -- 172.21.15.6:0/2411510821 learned_addr learned my addr 172.21.15.6:0/2411510821 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba32800640 1 --2- 172.21.15.6:0/2411510821 >> v2:172.21.15.178:3300/0 conn(0x7fba2c06e030 0x7fba2c0ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31e00640 1 --2- 172.21.15.6:0/2411510821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c145570 0x7fba2c0cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31400640 1 -- 172.21.15.6:0/2411510821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c145570 msgr2=0x7fba2c0cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31400640 1 --2- 172.21.15.6:0/2411510821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c145570 0x7fba2c0cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31400640 1 -- 172.21.15.6:0/2411510821 >> v2:172.21.15.178:3300/0 conn(0x7fba2c06e030 msgr2=0x7fba2c0ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31400640 1 --2- 172.21.15.6:0/2411510821 >> v2:172.21.15.178:3300/0 conn(0x7fba2c06e030 0x7fba2c0ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba31400640 1 -- 172.21.15.6:0/2411510821 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c072210 con 0x7fba2c071450 2024-03-06T07:28:38.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.103+0000 7fba32800640 1 --2- 172.21.15.6:0/2411510821 >> v2:172.21.15.178:3300/0 conn(0x7fba2c06e030 0x7fba2c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.105+0000 7fba31400640 1 --2- 172.21.15.6:0/2411510821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c071450 0x7fba2c06daf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fba2c071e60 tx=0x7fba1c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1547e3df70639496 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.105+0000 7fba30a00640 1 -- 172.21.15.6:0/2411510821 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c0317c0 con 0x7fba2c071450 2024-03-06T07:28:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.105+0000 7fba30a00640 1 -- 172.21.15.6:0/2411510821 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1c037070 con 0x7fba2c071450 2024-03-06T07:28:38.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.105+0000 7fba30a00640 1 -- 172.21.15.6:0/2411510821 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c031b80 con 0x7fba2c071450 2024-03-06T07:28:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 -- 172.21.15.6:0/2411510821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c071450 msgr2=0x7fba2c06daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 --2- 172.21.15.6:0/2411510821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c071450 0x7fba2c06daf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fba2c071e60 tx=0x7fba1c02b860 comp rx=0 tx=0).stop 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 -- 172.21.15.6:0/2411510821 shutdown_connections 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 --2- 172.21.15.6:0/2411510821 >> v2:172.21.15.178:3300/0 conn(0x7fba2c06e030 0x7fba2c0ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 --2- 172.21.15.6:0/2411510821 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c071450 0x7fba2c06daf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 --2- 172.21.15.6:0/2411510821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c145570 0x7fba2c0cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 -- 172.21.15.6:0/2411510821 >> 172.21.15.6:0/2411510821 conn(0x7fba2c0b17c0 msgr2=0x7fba2c0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 -- 172.21.15.6:0/2411510821 shutdown_connections 2024-03-06T07:28:38.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.106+0000 7fba33200640 1 -- 172.21.15.6:0/2411510821 wait complete. 2024-03-06T07:28:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.108+0000 7fba33200640 1 Processor -- start 2024-03-06T07:28:38.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.109+0000 7fba33200640 1 -- start start 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba33200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 0x7fba2c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba33200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c071450 0x7fba2c1f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba33200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba2c145570 0x7fba2c1f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba33200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c0718b0 con 0x7fba2c06e030 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba33200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c0727d0 con 0x7fba2c145570 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba33200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c071a30 con 0x7fba2c071450 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba31e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 0x7fba2c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba31e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 0x7fba2c0cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55208/0 (socket says 172.21.15.6:55208) 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.110+0000 7fba31e00640 1 -- 172.21.15.6:0/3461095388 learned_addr learned my addr 172.21.15.6:0/3461095388 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31400640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c071450 0x7fba2c1f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31e00640 1 -- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c071450 msgr2=0x7fba2c1f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31e00640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c071450 0x7fba2c1f6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31e00640 1 -- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba2c145570 msgr2=0x7fba2c1f7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31e00640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba2c145570 0x7fba2c1f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31e00640 1 -- 172.21.15.6:0/3461095388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c0ca010 con 0x7fba2c06e030 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31400640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c071450 0x7fba2c1f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba31e00640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 0x7fba2c0cb0b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fba2000d5f0 tx=0x7fba2000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200125f0 con 0x7fba2c06e030 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba20019070 con 0x7fba2c06e030 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200129d0 con 0x7fba2c06e030 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.111+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c0ca2f0 con 0x7fba2c06e030 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.112+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c0ca8b0 con 0x7fba2c06e030 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.113+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c04f330 con 0x7fba2c06e030 2024-03-06T07:28:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.114+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba20012b70 con 0x7fba2c06e030 2024-03-06T07:28:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.115+0000 7fba26000640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba1006b4d0 0x7fba1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.115+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba2008a360 con 0x7fba2c06e030 2024-03-06T07:28:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.115+0000 7fba31400640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba1006b4d0 0x7fba1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.117+0000 7fba31400640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba1006b4d0 0x7fba1006d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fba1c036000 tx=0x7fba1c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.120+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba20059490 con 0x7fba2c06e030 2024-03-06T07:28:38.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.324+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fba2c15eb90 con 0x7fba2c06e030 2024-03-06T07:28:38.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.787+0000 7fba26000640 1 -- 172.21.15.6:0/3461095388 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7fba20058d90 con 0x7fba2c06e030 2024-03-06T07:28:38.790 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:38.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba1006b4d0 msgr2=0x7fba1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba1006b4d0 0x7fba1006d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fba1c036000 tx=0x7fba1c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 msgr2=0x7fba2c0cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 0x7fba2c0cb0b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fba2000d5f0 tx=0x7fba2000dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 shutdown_connections 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba1006b4d0 0x7fba1006d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba2c145570 0x7fba2c1f7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba2c071450 0x7fba2c1f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 --2- 172.21.15.6:0/3461095388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba2c06e030 0x7fba2c0cb0b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 >> 172.21.15.6:0/3461095388 conn(0x7fba2c0b17c0 msgr2=0x7fba2c15dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.791+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 shutdown_connections 2024-03-06T07:28:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:38.792+0000 7fba33200640 1 -- 172.21.15.6:0/3461095388 wait complete. 2024-03-06T07:28:38.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.025+0000 7fe7d7400640 1 Processor -- start 2024-03-06T07:28:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.026+0000 7fe7d7400640 1 -- start start 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.027+0000 7fe7d7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7d006ca10 0x7fe7d00cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.027+0000 7fe7d7400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe7d00cfb10 0x7fe7d0157f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.027+0000 7fe7d7400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d015f6a0 con 0x7fe7d006b8a0 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.027+0000 7fe7d7400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d00d0300 con 0x7fe7d006ca10 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.027+0000 7fe7d7400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d015e8e0 con 0x7fe7d00cfb10 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.028+0000 7fe7cfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7d006ca10 0x7fe7d00cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.028+0000 7fe7cfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7d006ca10 0x7fe7d00cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55224/0 (socket says 172.21.15.6:55224) 2024-03-06T07:28:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.028+0000 7fe7cfe00640 1 -- 172.21.15.6:0/3592018172 learned_addr learned my addr 172.21.15.6:0/3592018172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.028+0000 7fe7d5400640 1 --2- 172.21.15.6:0/3592018172 >> v2:172.21.15.178:3300/0 conn(0x7fe7d00cfb10 0x7fe7d0157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.028+0000 7fe7cfe00640 1 -- 172.21.15.6:0/3592018172 >> v2:172.21.15.178:3300/0 conn(0x7fe7d00cfb10 msgr2=0x7fe7d0157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.029+0000 7fe7cfe00640 1 --2- 172.21.15.6:0/3592018172 >> v2:172.21.15.178:3300/0 conn(0x7fe7d00cfb10 0x7fe7d0157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.029+0000 7fe7cfe00640 1 -- 172.21.15.6:0/3592018172 >> v1:172.21.15.178:6789/0 conn(0x7fe7d006b8a0 legacy=0x7fe7d006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.029+0000 7fe7cfe00640 1 -- 172.21.15.6:0/3592018172 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d015ebc0 con 0x7fe7d006ca10 2024-03-06T07:28:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.030+0000 7fe7cfe00640 1 --2- 172.21.15.6:0/3592018172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d006ca10 0x7fe7d00cf5d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe7d006fe00 tx=0x7fe7bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752da2a3b96cafd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.030+0000 7fe7cf400640 1 -- 172.21.15.6:0/3592018172 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc034070 con 0x7fe7d006ca10 2024-03-06T07:28:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.030+0000 7fe7cf400640 1 -- 172.21.15.6:0/3592018172 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7bc030860 con 0x7fe7d006ca10 2024-03-06T07:28:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.030+0000 7fe7cf400640 1 -- 172.21.15.6:0/3592018172 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc030c30 con 0x7fe7d006ca10 2024-03-06T07:28:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 -- 172.21.15.6:0/3592018172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d006ca10 msgr2=0x7fe7d00cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 --2- 172.21.15.6:0/3592018172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d006ca10 0x7fe7d00cf5d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe7d006fe00 tx=0x7fe7bc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 -- 172.21.15.6:0/3592018172 shutdown_connections 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 --2- 172.21.15.6:0/3592018172 >> v2:172.21.15.178:3300/0 conn(0x7fe7d00cfb10 0x7fe7d0157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 --2- 172.21.15.6:0/3592018172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d006ca10 0x7fe7d00cf5d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 -- 172.21.15.6:0/3592018172 >> 172.21.15.6:0/3592018172 conn(0x7fe7d00d0ea0 msgr2=0x7fe7d00d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 -- 172.21.15.6:0/3592018172 shutdown_connections 2024-03-06T07:28:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.031+0000 7fe7d7400640 1 -- 172.21.15.6:0/3592018172 wait complete. 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.033+0000 7fe7d7400640 1 Processor -- start 2024-03-06T07:28:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.033+0000 7fe7d7400640 1 -- start start 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.034+0000 7fe7d7400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d006b8a0 0x7fe7d01e7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.034+0000 7fe7d7400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7d006ca10 0x7fe7d0156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.034+0000 7fe7d7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 0x7fe7d00adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.034+0000 7fe7d7400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d006e5a0 con 0x7fe7d00cfb10 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.034+0000 7fe7d7400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d015f180 con 0x7fe7d006b8a0 2024-03-06T07:28:39.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.034+0000 7fe7d7400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d00cff70 con 0x7fe7d006ca10 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 0x7fe7d00adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 0x7fe7d00adfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55232/0 (socket says 172.21.15.6:55232) 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 -- 172.21.15.6:0/1874357527 learned_addr learned my addr 172.21.15.6:0/1874357527 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7cfe00640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7d006ca10 0x7fe7d0156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 -- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7d006ca10 msgr2=0x7fe7d0156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d4a00640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d006b8a0 0x7fe7d01e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7d006ca10 0x7fe7d0156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 -- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d006b8a0 msgr2=0x7fe7d01e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d006b8a0 0x7fe7d01e7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.035+0000 7fe7d5400640 1 -- 172.21.15.6:0/1874357527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d01555b0 con 0x7fe7d00cfb10 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.036+0000 7fe7d5400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 0x7fe7d00adfc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8007e90 tx=0x7fe7c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.036+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8014420 con 0x7fe7d00cfb10 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.036+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c8019070 con 0x7fe7d00cfb10 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.036+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c8014800 con 0x7fe7d00cfb10 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.036+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d0155890 con 0x7fe7d00cfb10 2024-03-06T07:28:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.036+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d0156130 con 0x7fe7d00cfb10 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.037+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d0155c80 con 0x7fe7d00cfb10 2024-03-06T07:28:39.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.039+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7c80149a0 con 0x7fe7d00cfb10 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.039+0000 7fe7ccc00640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7b006b4d0 0x7fe7b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.040+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe7c808b690 con 0x7fe7d00cfb10 2024-03-06T07:28:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.040+0000 7fe7d4a00640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7b006b4d0 0x7fe7b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.041+0000 7fe7d4a00640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7b006b4d0 0x7fe7b006d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00015a0 tx=0x7fe7c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.044+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7c805a7d0 con 0x7fe7d00cfb10 2024-03-06T07:28:39.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.244+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe7d0069510 con 0x7fe7d00cfb10 2024-03-06T07:28:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.794+0000 7fe7ccc00640 1 -- 172.21.15.6:0/1874357527 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe7c805a0d0 con 0x7fe7d00cfb10 2024-03-06T07:28:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:28:39.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7b006b4d0 msgr2=0x7fe7b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7b006b4d0 0x7fe7b006d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00015a0 tx=0x7fe7c0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 msgr2=0x7fe7d00adfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 0x7fe7d00adfc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8007e90 tx=0x7fe7c800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 shutdown_connections 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7b006b4d0 0x7fe7b006d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7d00cfb10 0x7fe7d00adfc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7d006ca10 0x7fe7d0156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 --2- 172.21.15.6:0/1874357527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7d006b8a0 0x7fe7d01e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.798+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 >> 172.21.15.6:0/1874357527 conn(0x7fe7d00d0ea0 msgr2=0x7fe7d006c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.799+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 shutdown_connections 2024-03-06T07:28:39.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:39.799+0000 7fe7d7400640 1 -- 172.21.15.6:0/1874357527 wait complete. 2024-03-06T07:28:39.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:40.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.021+0000 7fee90400640 1 Processor -- start 2024-03-06T07:28:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.021+0000 7fee90400640 1 -- start start 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee90400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fee8806f9c0 0x7fee8806fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee90400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fee8806edc0 0x7fee8815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee90400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee880b1530 con 0x7fee880d0c20 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee90400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee880cff10 con 0x7fee8806edc0 2024-03-06T07:28:40.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee90400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8815f700 con 0x7fee8806f9c0 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee8d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fee8806edc0 0x7fee8815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee8d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fee8806edc0 0x7fee8815f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55244/0 (socket says 172.21.15.6:55244) 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.023+0000 7fee8d000640 1 -- 172.21.15.6:0/4044119320 learned_addr learned my addr 172.21.15.6:0/4044119320 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.024+0000 7fee8da00640 1 --2- 172.21.15.6:0/4044119320 >> v2:172.21.15.178:3300/0 conn(0x7fee8806f9c0 0x7fee8806fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.024+0000 7fee8d000640 1 -- 172.21.15.6:0/4044119320 >> v2:172.21.15.178:3300/0 conn(0x7fee8806f9c0 msgr2=0x7fee8806fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.024+0000 7fee8d000640 1 --2- 172.21.15.6:0/4044119320 >> v2:172.21.15.178:3300/0 conn(0x7fee8806f9c0 0x7fee8806fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.024+0000 7fee8d000640 1 -- 172.21.15.6:0/4044119320 >> v1:172.21.15.178:6789/0 conn(0x7fee880d0c20 legacy=0x7fee8815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.024+0000 7fee8d000640 1 -- 172.21.15.6:0/4044119320 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8806f2c0 con 0x7fee8806edc0 2024-03-06T07:28:40.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.025+0000 7fee8d000640 1 --2- 172.21.15.6:0/4044119320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee8815f1c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fee880b48e0 tx=0x7fee7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed77337ea40cdba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee87e00640 1 -- 172.21.15.6:0/4044119320 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74035020 con 0x7fee8806edc0 2024-03-06T07:28:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee87e00640 1 -- 172.21.15.6:0/4044119320 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee74031870 con 0x7fee8806edc0 2024-03-06T07:28:40.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee87e00640 1 -- 172.21.15.6:0/4044119320 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74031c40 con 0x7fee8806edc0 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee90400640 1 -- 172.21.15.6:0/4044119320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 msgr2=0x7fee8815f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee90400640 1 --2- 172.21.15.6:0/4044119320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee8815f1c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fee880b48e0 tx=0x7fee7402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee90400640 1 -- 172.21.15.6:0/4044119320 shutdown_connections 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee90400640 1 --2- 172.21.15.6:0/4044119320 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee8815f1c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee90400640 1 --2- 172.21.15.6:0/4044119320 >> v2:172.21.15.178:3300/0 conn(0x7fee8806f9c0 0x7fee8806fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.026+0000 7fee90400640 1 -- 172.21.15.6:0/4044119320 >> 172.21.15.6:0/4044119320 conn(0x7fee880b3820 msgr2=0x7fee880b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.027+0000 7fee90400640 1 -- 172.21.15.6:0/4044119320 shutdown_connections 2024-03-06T07:28:40.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.027+0000 7fee90400640 1 -- 172.21.15.6:0/4044119320 wait complete. 2024-03-06T07:28:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.028+0000 7fee90400640 1 Processor -- start 2024-03-06T07:28:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.028+0000 7fee90400640 1 -- start start 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.029+0000 7fee90400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee881f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.029+0000 7fee90400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee8806f9c0 0x7fee8815ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.029+0000 7fee90400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee880d0c20 0x7fee88151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.029+0000 7fee90400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee880d0790 con 0x7fee8806edc0 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.029+0000 7fee90400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8806f5a0 con 0x7fee880d0c20 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.029+0000 7fee90400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee880d0420 con 0x7fee8806f9c0 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee881f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee881f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55248/0 (socket says 172.21.15.6:55248) 2024-03-06T07:28:40.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 -- 172.21.15.6:0/3161310985 learned_addr learned my addr 172.21.15.6:0/3161310985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8d000640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee8806f9c0 0x7fee8815ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 -- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee8806f9c0 msgr2=0x7fee8815ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8e400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee880d0c20 0x7fee88151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee8806f9c0 0x7fee8815ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 -- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee880d0c20 msgr2=0x7fee88151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee880d0c20 0x7fee88151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8da00640 1 -- 172.21.15.6:0/3161310985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee880c5a10 con 0x7fee8806edc0 2024-03-06T07:28:40.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.030+0000 7fee8e400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee880d0c20 0x7fee88151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.031+0000 7fee8da00640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee881f8960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00d620 tx=0x7fee7c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.031+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00b620 con 0x7fee8806edc0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.031+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c019070 con 0x7fee8806edc0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.031+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00ba00 con 0x7fee8806edc0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.031+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee880c73e0 con 0x7fee8806edc0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.031+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8815c720 con 0x7fee8806edc0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.032+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8804f330 con 0x7fee8806edc0 2024-03-06T07:28:40.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.033+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee7c00bba0 con 0x7fee8806edc0 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.034+0000 7fee85600640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee6806b500 0x7fee6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.034+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fee7c08a610 con 0x7fee8806edc0 2024-03-06T07:28:40.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.035+0000 7fee8d000640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee6806b500 0x7fee6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.036+0000 7fee8d000640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee6806b500 0x7fee6806d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fee74036000 tx=0x7fee74008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.040+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee7c059450 con 0x7fee8806edc0 2024-03-06T07:28:40.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.280+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fee880b4340 con 0x7fee8806edc0 2024-03-06T07:28:40.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.850+0000 7fee85600640 1 -- 172.21.15.6:0/3161310985 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fee7c058d50 con 0x7fee8806edc0 2024-03-06T07:28:40.857 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:28:40.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.857+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee6806b500 msgr2=0x7fee6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.857+0000 7fee90400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee6806b500 0x7fee6806d9c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fee74036000 tx=0x7fee74008030 comp rx=0 tx=0).stop 2024-03-06T07:28:40.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.857+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 msgr2=0x7fee881f8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.857+0000 7fee90400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee881f8960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00d620 tx=0x7fee7c00dae0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.861+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 shutdown_connections 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.861+0000 7fee90400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee6806b500 0x7fee6806d9c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.861+0000 7fee90400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee880d0c20 0x7fee88151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.861+0000 7fee90400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee8806f9c0 0x7fee8815ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.862+0000 7fee90400640 1 --2- 172.21.15.6:0/3161310985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee8806edc0 0x7fee881f8960 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.862+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 >> 172.21.15.6:0/3161310985 conn(0x7fee880b3820 msgr2=0x7fee880d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.862+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 shutdown_connections 2024-03-06T07:28:40.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:40.863+0000 7fee90400640 1 -- 172.21.15.6:0/3161310985 wait complete. 2024-03-06T07:28:40.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:41.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.078+0000 7f9344600640 1 Processor -- start 2024-03-06T07:28:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.078+0000 7f9344600640 1 -- start start 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9344600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f933c0cf5c0 0x7f933c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9344600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 0x7f933c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9344600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0b4860 con 0x7f933c0b3c90 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9344600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0ce160 con 0x7f933c0cf5c0 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9344600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0b4b40 con 0x7f933c071050 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9341c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f933c0b3c90 0x7f933c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44616/0 (socket says 172.21.15.6:44616) 2024-03-06T07:28:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.080+0000 7f9341c00640 1 -- 172.21.15.6:0/1604382694 learned_addr learned my addr 172.21.15.6:0/1604382694 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f9341200640 1 --2- 172.21.15.6:0/1604382694 >> v2:172.21.15.178:3300/0 conn(0x7f933c0cf5c0 0x7f933c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f9342600640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 0x7f933c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572187157 0 0) 0x7f933c0b4860 con 0x7f933c0b3c90 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9324003520 con 0x7f933c0b3c90 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3785645771 0 0) 0x7f9324003520 con 0x7f933c0b3c90 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 msgr2=0x7f933c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 0x7f933c071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 >> v2:172.21.15.178:3300/0 conn(0x7f933c0cf5c0 msgr2=0x7f933c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 --2- 172.21.15.6:0/1604382694 >> v2:172.21.15.178:3300/0 conn(0x7f933c0cf5c0 0x7f933c0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0b4e20 con 0x7f933c0b3c90 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9330001670 con 0x7f933c0b3c90 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 >> v1:172.21.15.6:6789/0 conn(0x7f933c0b3c90 legacy=0x7f933c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9324003a50 0x7f9324003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 0x7f93240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0ce160 con 0x7f9324003a50 2024-03-06T07:28:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0b4b40 con 0x7f9324004800 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.081+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932400ba70 con 0x7f933c071050 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341c00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9324003a50 0x7f9324003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341200640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341c00640 1 -- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 msgr2=0x7f93240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9342600640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 0x7f93240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341c00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933c071050 0x7f93240051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341c00640 1 -- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9324004800 msgr2=0x7f9324004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341c00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341c00640 1 -- 172.21.15.6:0/1604382694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324003520 con 0x7f9324003a50 2024-03-06T07:28:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.082+0000 7f9341200640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.083+0000 7f9341c00640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9324003a50 0x7f9324003e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f933000c240 tx=0x7f9324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d5278a8edc1330 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9330007d10 con 0x7f9324003a50 2024-03-06T07:28:41.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330030070 con 0x7f9324003a50 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f933be00640 1 -- 172.21.15.6:0/1604382694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933002cd20 con 0x7f9324003a50 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9344600640 1 -- 172.21.15.6:0/1604382694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9324003a50 msgr2=0x7f9324003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9344600640 1 --2- 172.21.15.6:0/1604382694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9324003a50 0x7f9324003e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f933000c240 tx=0x7f9324005830 comp rx=0 tx=0).stop 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9342600640 1 -- 172.21.15.6:0/1604382694 reap_dead start 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9344600640 1 -- 172.21.15.6:0/1604382694 shutdown_connections 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9344600640 1 -- 172.21.15.6:0/1604382694 >> 172.21.15.6:0/1604382694 conn(0x7f933c06afb0 msgr2=0x7f933c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9344600640 1 -- 172.21.15.6:0/1604382694 shutdown_connections 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.084+0000 7f9344600640 1 -- 172.21.15.6:0/1604382694 wait complete. 2024-03-06T07:28:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.086+0000 7f9344600640 1 Processor -- start 2024-03-06T07:28:41.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.086+0000 7f9344600640 1 -- start start 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.087+0000 7f9344600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9324003a50 0x7f933c09d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.087+0000 7f9344600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f933c158f30 0x7f933c158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.087+0000 7f9344600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 0x7f933c147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.087+0000 7f9344600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c071490 con 0x7f933c156720 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.087+0000 7f9344600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c15b340 con 0x7f933c158f30 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.087+0000 7f9344600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c071610 con 0x7f9324003a50 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 0x7f933c147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 0x7f933c147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55264/0 (socket says 172.21.15.6:55264) 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 -- 172.21.15.6:0/77249143 learned_addr learned my addr 172.21.15.6:0/77249143 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9341200640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f933c158f30 0x7f933c158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9341c00640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9324003a50 0x7f933c09d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 -- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9324003a50 msgr2=0x7f933c09d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9324003a50 0x7f933c09d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 -- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f933c158f30 msgr2=0x7f933c158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f933c158f30 0x7f933c158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 -- 172.21.15.6:0/77249143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c157470 con 0x7f933c156720 2024-03-06T07:28:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9341200640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f933c158f30 0x7f933c158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.088+0000 7f9342600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 0x7f933c147470 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f933c0cda70 tx=0x7f933400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.089+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933401c070 con 0x7f933c156720 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.089+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9334017400 con 0x7f933c156720 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.089+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c157750 con 0x7f933c156720 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.089+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340177f0 con 0x7f933c156720 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.089+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c157d10 con 0x7f933c156720 2024-03-06T07:28:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.090+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c04f3a0 con 0x7f933c156720 2024-03-06T07:28:41.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.091+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9334005d10 con 0x7f933c156720 2024-03-06T07:28:41.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.092+0000 7f9339600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931c06b500 0x7f931c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.092+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f933408a430 con 0x7f933c156720 2024-03-06T07:28:41.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.092+0000 7f9341c00640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931c06b500 0x7f931c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.094+0000 7f9341c00640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931c06b500 0x7f931c06d9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9330034000 tx=0x7f9330009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.096+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9334012070 con 0x7f933c156720 2024-03-06T07:28:41.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.299+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f933c071a40 con 0x7f933c156720 2024-03-06T07:28:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.301+0000 7f9339600640 1 -- 172.21.15.6:0/77249143 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16610 (secure 0 0 0) 0x7f9334059210 con 0x7f933c156720 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:41.304 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:40.282516+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931c06b500 msgr2=0x7f931c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931c06b500 0x7f931c06d9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9330034000 tx=0x7f9330009030 comp rx=0 tx=0).stop 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 msgr2=0x7f933c147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 0x7f933c147470 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f933c0cda70 tx=0x7f933400f4a0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 shutdown_connections 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931c06b500 0x7f931c06d9c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f933c158f30 0x7f933c158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933c156720 0x7f933c147470 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 --2- 172.21.15.6:0/77249143 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9324003a50 0x7f933c09d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.304+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 >> 172.21.15.6:0/77249143 conn(0x7f933c06afb0 msgr2=0x7f933c0adba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.305+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 shutdown_connections 2024-03-06T07:28:41.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.305+0000 7f9344600640 1 -- 172.21.15.6:0/77249143 wait complete. 2024-03-06T07:28:41.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:41.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.548+0000 7fcf02c00640 1 Processor -- start 2024-03-06T07:28:41.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.548+0000 7fcf02c00640 1 -- start start 2024-03-06T07:28:41.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.550+0000 7fcf02c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcefc0af1a0 0x7fcefc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.550+0000 7fcf02c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.550+0000 7fcf02c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc15cc40 con 0x7fcefc0cdc20 2024-03-06T07:28:41.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.550+0000 7fcf02c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc15cf20 con 0x7fcefc0af1a0 2024-03-06T07:28:41.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.550+0000 7fcf02c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc15d200 con 0x7fcefc0af580 2024-03-06T07:28:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.551+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.551+0000 7fcefbe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcefc0af1a0 0x7fcefc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.551+0000 7fcf00c00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fcefc0cdc20 0x7fcefc0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:49650/0 (socket says 172.21.15.6:49650) 2024-03-06T07:28:41.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.551+0000 7fcf00c00640 1 -- 172.21.15.6:0/1445870198 learned_addr learned my addr 172.21.15.6:0/1445870198 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcefb400640 1 -- 172.21.15.6:0/1445870198 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185163870 0 0) 0x7fcefc15cc40 con 0x7fcefc0cdc20 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcefb400640 1 -- 172.21.15.6:0/1445870198 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003520 con 0x7fcefc0cdc20 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcef3e00640 1 -- 172.21.15.6:0/1445870198 >> v2:172.21.15.178:3300/0 conn(0x7fcefc0af1a0 msgr2=0x7fcefc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcef3e00640 1 --2- 172.21.15.6:0/1445870198 >> v2:172.21.15.178:3300/0 conn(0x7fcefc0af1a0 0x7fcefc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcef3e00640 1 -- 172.21.15.6:0/1445870198 >> v1:172.21.15.178:6789/0 conn(0x7fcefc0cdc20 legacy=0x7fcefc0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcef3e00640 1 -- 172.21.15.6:0/1445870198 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc15d4e0 con 0x7fcefc0af580 2024-03-06T07:28:41.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.552+0000 7fcefbe00640 1 --2- 172.21.15.6:0/1445870198 >> v2:172.21.15.178:3300/0 conn(0x7fcefc0af1a0 0x7fcefc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:41.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.553+0000 7fcef3e00640 1 --2- 172.21.15.6:0/1445870198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc0b4c20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcee40097a0 tx=0x7fcee402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb2f35eb4456b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcefb400640 1 -- 172.21.15.6:0/1445870198 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4031780 con 0x7fcefc0af580 2024-03-06T07:28:41.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcefb400640 1 -- 172.21.15.6:0/1445870198 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee4037070 con 0x7fcefc0af580 2024-03-06T07:28:41.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcefb400640 1 -- 172.21.15.6:0/1445870198 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4031b40 con 0x7fcefc0af580 2024-03-06T07:28:41.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcf02c00640 1 -- 172.21.15.6:0/1445870198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 msgr2=0x7fcefc0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1445870198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc0b4c20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcee40097a0 tx=0x7fcee402b820 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcf02c00640 1 -- 172.21.15.6:0/1445870198 shutdown_connections 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1445870198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc0b4c20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1445870198 >> v2:172.21.15.178:3300/0 conn(0x7fcefc0af1a0 0x7fcefc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.554+0000 7fcf02c00640 1 -- 172.21.15.6:0/1445870198 >> 172.21.15.6:0/1445870198 conn(0x7fcefc06afb0 msgr2=0x7fcefc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.555+0000 7fcf02c00640 1 -- 172.21.15.6:0/1445870198 shutdown_connections 2024-03-06T07:28:41.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.555+0000 7fcf02c00640 1 -- 172.21.15.6:0/1445870198 wait complete. 2024-03-06T07:28:41.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.556+0000 7fcf02c00640 1 Processor -- start 2024-03-06T07:28:41.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.557+0000 7fcf02c00640 1 -- start start 2024-03-06T07:28:41.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.557+0000 7fcf02c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 0x7fcefc1f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcf02c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc1f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcf02c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcefc0cdc20 0x7fcefc0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcf02c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b3660 con 0x7fcefc0af1a0 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcf02c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b34e0 con 0x7fcefc0cdc20 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcf02c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc0b37e0 con 0x7fcefc0af580 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 0x7fcefc1f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 0x7fcefc1f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55272/0 (socket says 172.21.15.6:55272) 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 -- 172.21.15.6:0/1074446089 learned_addr learned my addr 172.21.15.6:0/1074446089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcf00c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcefc0cdc20 0x7fcefc0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcef3e00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc1f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 -- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 msgr2=0x7fcefc1f33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc1f33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 -- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcefc0cdc20 msgr2=0x7fcefc0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcefc0cdc20 0x7fcefc0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.558+0000 7fcefbe00640 1 -- 172.21.15.6:0/1074446089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc0adef0 con 0x7fcefc0af1a0 2024-03-06T07:28:41.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.559+0000 7fcefbe00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 0x7fcefc1f4750 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fceec00ebe0 tx=0x7fceec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.559+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec017070 con 0x7fcefc0af1a0 2024-03-06T07:28:41.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.559+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec00cc30 con 0x7fcefc0af1a0 2024-03-06T07:28:41.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.559+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc0ae190 con 0x7fcefc0af1a0 2024-03-06T07:28:41.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.559+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec013470 con 0x7fcefc0af1a0 2024-03-06T07:28:41.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.559+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc06e8a0 con 0x7fcefc0af1a0 2024-03-06T07:28:41.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.560+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc069510 con 0x7fcefc0af1a0 2024-03-06T07:28:41.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.561+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceec01a070 con 0x7fcefc0af1a0 2024-03-06T07:28:41.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.563+0000 7fcef8c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fced806b4d0 0x7fced806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.563+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceec08a510 con 0x7fcefc0af1a0 2024-03-06T07:28:41.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.563+0000 7fcef3e00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fced806b4d0 0x7fced806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.565+0000 7fcef3e00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fced806b4d0 0x7fced806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcee4036000 tx=0x7fcee403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.568+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fceec059380 con 0x7fcefc0af1a0 2024-03-06T07:28:41.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.767+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcefc1f43a0 con 0x7fcefc0af1a0 2024-03-06T07:28:41.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.770+0000 7fcef8c00640 1 -- 172.21.15.6:0/1074446089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16610 (secure 0 0 0) 0x7fcefc1f43a0 con 0x7fcefc0af1a0 2024-03-06T07:28:41.773 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:41.773 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":34,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:40.282516+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fced806b4d0 msgr2=0x7fced806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fced806b4d0 0x7fced806d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcee4036000 tx=0x7fcee403d040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 msgr2=0x7fcefc1f4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 0x7fcefc1f4750 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fceec00ebe0 tx=0x7fceec00c520 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 shutdown_connections 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fced806b4d0 0x7fced806d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcefc0cdc20 0x7fcefc0c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcefc0af580 0x7fcefc1f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 --2- 172.21.15.6:0/1074446089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcefc0af1a0 0x7fcefc1f4750 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 >> 172.21.15.6:0/1074446089 conn(0x7fcefc06afb0 msgr2=0x7fcefc15a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 shutdown_connections 2024-03-06T07:28:41.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:41.773+0000 7fcf02c00640 1 -- 172.21.15.6:0/1074446089 wait complete. 2024-03-06T07:28:41.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.012+0000 7f2b50800640 1 Processor -- start 2024-03-06T07:28:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.012+0000 7f2b50800640 1 -- start start 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b50800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b4815e260 0x7f2b480d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b50800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2b480cf500 0x7f2b480d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b50800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4815e640 con 0x7f2b480d2530 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b50800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b480d4ca0 con 0x7f2b4815e260 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b50800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b480cfa70 con 0x7f2b480cf500 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b4de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b4815e260 0x7f2b480d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b4de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b4815e260 0x7f2b480d1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42730/0 (socket says 172.21.15.6:42730) 2024-03-06T07:28:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b4e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2b480d2530 0x7f2b480d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48246/0 (socket says 172.21.15.6:48246) 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.015+0000 7f2b4de00640 1 -- 172.21.15.6:0/3297475648 learned_addr learned my addr 172.21.15.6:0/3297475648 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.016+0000 7f2b4d400640 1 --2- 172.21.15.6:0/3297475648 >> v2:172.21.15.178:3300/0 conn(0x7f2b480cf500 0x7f2b480d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.016+0000 7f2b4de00640 1 -- 172.21.15.6:0/3297475648 >> v2:172.21.15.178:3300/0 conn(0x7f2b480cf500 msgr2=0x7f2b480d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.016+0000 7f2b4de00640 1 --2- 172.21.15.6:0/3297475648 >> v2:172.21.15.178:3300/0 conn(0x7f2b480cf500 0x7f2b480d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.016+0000 7f2b4de00640 1 -- 172.21.15.6:0/3297475648 >> v1:172.21.15.6:6789/0 conn(0x7f2b480d2530 legacy=0x7f2b480d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.016+0000 7f2b4de00640 1 -- 172.21.15.6:0/3297475648 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b480cfd50 con 0x7f2b4815e260 2024-03-06T07:28:42.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.017+0000 7f2b4de00640 1 --2- 172.21.15.6:0/3297475648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b4815e260 0x7f2b480d1ab0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009870 tx=0x7f2b3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1989126e8667fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.017+0000 7f2b4ca00640 1 -- 172.21.15.6:0/3297475648 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c035020 con 0x7f2b4815e260 2024-03-06T07:28:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.017+0000 7f2b4ca00640 1 -- 172.21.15.6:0/3297475648 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c031870 con 0x7f2b4815e260 2024-03-06T07:28:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.017+0000 7f2b4ca00640 1 -- 172.21.15.6:0/3297475648 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c031c40 con 0x7f2b4815e260 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 -- 172.21.15.6:0/3297475648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b4815e260 msgr2=0x7f2b480d1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 --2- 172.21.15.6:0/3297475648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b4815e260 0x7f2b480d1ab0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009870 tx=0x7f2b3c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 -- 172.21.15.6:0/3297475648 shutdown_connections 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 --2- 172.21.15.6:0/3297475648 >> v2:172.21.15.178:3300/0 conn(0x7f2b480cf500 0x7f2b480d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 --2- 172.21.15.6:0/3297475648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b4815e260 0x7f2b480d1ab0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 -- 172.21.15.6:0/3297475648 >> 172.21.15.6:0/3297475648 conn(0x7f2b480d3ee0 msgr2=0x7f2b480d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 -- 172.21.15.6:0/3297475648 shutdown_connections 2024-03-06T07:28:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.018+0000 7f2b50800640 1 -- 172.21.15.6:0/3297475648 wait complete. 2024-03-06T07:28:42.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.020+0000 7f2b50800640 1 Processor -- start 2024-03-06T07:28:42.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.020+0000 7f2b50800640 1 -- start start 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b50800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b480cf500 0x7f2b4815a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b50800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 0x7f2b4815e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b50800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b4815e260 0x7f2b4815ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b50800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480b3e80 con 0x7f2b480d2530 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b50800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480b3ba0 con 0x7f2b480cf500 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b50800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480b53b0 con 0x7f2b4815e260 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b4d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 0x7f2b4815e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b4d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 0x7f2b4815e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42746/0 (socket says 172.21.15.6:42746) 2024-03-06T07:28:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.021+0000 7f2b4d400640 1 -- 172.21.15.6:0/2942164501 learned_addr learned my addr 172.21.15.6:0/2942164501 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4de00640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b480cf500 0x7f2b4815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4e800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b4815e260 0x7f2b4815ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4d400640 1 -- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b4815e260 msgr2=0x7f2b4815ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4d400640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b4815e260 0x7f2b4815ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4d400640 1 -- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b480cf500 msgr2=0x7f2b4815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4d400640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b480cf500 0x7f2b4815a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4d400640 1 -- 172.21.15.6:0/2942164501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b480cd7d0 con 0x7f2b480d2530 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.022+0000 7f2b4d400640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 0x7f2b4815e810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800d800 tx=0x7f2b3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.023+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3800b6c0 con 0x7f2b480d2530 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.023+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b38019070 con 0x7f2b480d2530 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.023+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3800ba80 con 0x7f2b480d2530 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.023+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b480b7a20 con 0x7f2b480d2530 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.023+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b480b7fe0 con 0x7f2b480d2530 2024-03-06T07:28:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.025+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4804f330 con 0x7f2b480d2530 2024-03-06T07:28:42.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.025+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b3800bc20 con 0x7f2b480d2530 2024-03-06T07:28:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.026+0000 7f2b42000640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b2c06b4d0 0x7f2b2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.026+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b3808a6d0 con 0x7f2b480d2530 2024-03-06T07:28:42.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.027+0000 7f2b4de00640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b2c06b4d0 0x7f2b2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.028+0000 7f2b4de00640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b2c06b4d0 0x7f2b2c06d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c036000 tx=0x7f2b3c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.030+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b38059540 con 0x7f2b480d2530 2024-03-06T07:28:42.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.285+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2b480b34f0 con 0x7f2b480d2530 2024-03-06T07:28:42.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.862+0000 7f2b42000640 1 -- 172.21.15.6:0/2942164501 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2b38058e40 con 0x7f2b480d2530 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.866+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b2c06b4d0 msgr2=0x7f2b2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.866+0000 7f2b50800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b2c06b4d0 0x7f2b2c06d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c036000 tx=0x7f2b3c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.866+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 msgr2=0x7f2b4815e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.866+0000 7f2b50800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 0x7f2b4815e810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800d800 tx=0x7f2b3800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 shutdown_connections 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b2c06b4d0 0x7f2b2c06d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b4815e260 0x7f2b4815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b480d2530 0x7f2b4815e810 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 --2- 172.21.15.6:0/2942164501 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b480cf500 0x7f2b4815a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 >> 172.21.15.6:0/2942164501 conn(0x7f2b480d3ee0 msgr2=0x7f2b480d29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 shutdown_connections 2024-03-06T07:28:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:42.867+0000 7f2b50800640 1 -- 172.21.15.6:0/2942164501 wait complete. 2024-03-06T07:28:42.888 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:42.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.105+0000 7f7a67e00640 1 Processor -- start 2024-03-06T07:28:43.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.105+0000 7f7a67e00640 1 -- start start 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.106+0000 7f7a67e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a68161700 0x7f7a680b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.106+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a680d4890 0x7f7a680d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.106+0000 7f7a67e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7a680d2320 0x7f7a680b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.106+0000 7f7a67e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a680b2c00 con 0x7f7a68161700 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.106+0000 7f7a67e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a68161ae0 con 0x7f7a680d2320 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.106+0000 7f7a67e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b2ea0 con 0x7f7a680d4890 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a68161700 0x7f7a680b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7a68161700 0x7f7a680b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42762/0 (socket says 172.21.15.6:42762) 2024-03-06T07:28:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 -- 172.21.15.6:0/4069137442 learned_addr learned my addr 172.21.15.6:0/4069137442 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a67400640 1 --2- 172.21.15.6:0/4069137442 >> v2:172.21.15.178:3300/0 conn(0x7f7a680d2320 0x7f7a680b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66000640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a680d4890 0x7f7a680d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 -- 172.21.15.6:0/4069137442 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a680d4890 msgr2=0x7f7a680d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a680d4890 0x7f7a680d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 -- 172.21.15.6:0/4069137442 >> v2:172.21.15.178:3300/0 conn(0x7f7a680d2320 msgr2=0x7f7a680b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 --2- 172.21.15.6:0/4069137442 >> v2:172.21.15.178:3300/0 conn(0x7f7a680d2320 0x7f7a680b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66a00640 1 -- 172.21.15.6:0/4069137442 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680b3180 con 0x7f7a68161700 2024-03-06T07:28:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.107+0000 7f7a66000640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a680d4890 0x7f7a680d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.109+0000 7f7a66a00640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68161700 0x7f7a680b4fd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7a680c7850 tx=0x7f7a5002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eb3349d13f2db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.109+0000 7f7a65600640 1 -- 172.21.15.6:0/4069137442 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50035020 con 0x7f7a68161700 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.109+0000 7f7a65600640 1 -- 172.21.15.6:0/4069137442 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5002d040 con 0x7f7a68161700 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.109+0000 7f7a65600640 1 -- 172.21.15.6:0/4069137442 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50031a80 con 0x7f7a68161700 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.109+0000 7f7a67e00640 1 -- 172.21.15.6:0/4069137442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68161700 msgr2=0x7f7a680b4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.109+0000 7f7a67e00640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68161700 0x7f7a680b4fd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7a680c7850 tx=0x7f7a5002b900 comp rx=0 tx=0).stop 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 -- 172.21.15.6:0/4069137442 shutdown_connections 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 --2- 172.21.15.6:0/4069137442 >> v2:172.21.15.178:3300/0 conn(0x7f7a680d2320 0x7f7a680b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a680d4890 0x7f7a680d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 --2- 172.21.15.6:0/4069137442 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a68161700 0x7f7a680b4fd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 -- 172.21.15.6:0/4069137442 >> 172.21.15.6:0/4069137442 conn(0x7f7a680d3d70 msgr2=0x7f7a680d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 -- 172.21.15.6:0/4069137442 shutdown_connections 2024-03-06T07:28:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.110+0000 7f7a67e00640 1 -- 172.21.15.6:0/4069137442 wait complete. 2024-03-06T07:28:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.111+0000 7f7a67e00640 1 Processor -- start 2024-03-06T07:28:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.111+0000 7f7a67e00640 1 -- start start 2024-03-06T07:28:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 0x7f7a6814e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a680d4890 0x7f7a6814edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a67e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a68161700 0x7f7a680cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a67e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b5510 con 0x7f7a680d2320 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a67e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b0d00 con 0x7f7a680d4890 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a67e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680b5690 con 0x7f7a68161700 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a66a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 0x7f7a6814e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a66a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 0x7f7a6814e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42778/0 (socket says 172.21.15.6:42778) 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.113+0000 7f7a66a00640 1 -- 172.21.15.6:0/1875699738 learned_addr learned my addr 172.21.15.6:0/1875699738 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a66a00640 1 -- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a68161700 msgr2=0x7f7a680cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a68161700 0x7f7a680cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a66a00640 1 -- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a680d4890 msgr2=0x7f7a6814edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a680d4890 0x7f7a6814edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a66a00640 1 -- 172.21.15.6:0/1875699738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680c85a0 con 0x7f7a680d2320 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a66a00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 0x7f7a6814e5d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7a50000c00 tx=0x7f7a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50040070 con 0x7f7a680d2320 2024-03-06T07:28:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a50039bf0 con 0x7f7a680d2320 2024-03-06T07:28:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50035040 con 0x7f7a680d2320 2024-03-06T07:28:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.114+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a680c7a20 con 0x7f7a680d2320 2024-03-06T07:28:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.115+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a680c7fe0 con 0x7f7a680d2320 2024-03-06T07:28:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.116+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6804f330 con 0x7f7a680d2320 2024-03-06T07:28:43.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.117+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a5004b060 con 0x7f7a680d2320 2024-03-06T07:28:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.117+0000 7f7a5ea00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.117+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a500abe00 con 0x7f7a680d2320 2024-03-06T07:28:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.118+0000 7f7a66000640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.119+0000 7f7a66000640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a4406b4d0 0x7f7a4406d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a681f70b0 tx=0x7f7a580083d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.122+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a5007ac60 con 0x7f7a680d2320 2024-03-06T07:28:43.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.388+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7a680ce2e0 con 0x7f7a680d2320 2024-03-06T07:28:43.778 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-06T07:28:43.776+0000 7f1367800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:43.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.874+0000 7f7a5ea00640 1 -- 172.21.15.6:0/1875699738 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7a5007a560 con 0x7f7a680d2320 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a4406b4d0 msgr2=0x7f7a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a4406b4d0 0x7f7a4406d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a681f70b0 tx=0x7f7a580083d0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 msgr2=0x7f7a6814e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 0x7f7a6814e5d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7a50000c00 tx=0x7f7a500046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 shutdown_connections 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a680d4890 0x7f7a6814edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a68161700 0x7f7a680cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 --2- 172.21.15.6:0/1875699738 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a680d2320 0x7f7a6814e5d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.877+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 >> 172.21.15.6:0/1875699738 conn(0x7f7a680d3d70 msgr2=0x7f7a680b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.878+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 shutdown_connections 2024-03-06T07:28:43.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:43.878+0000 7f7a67e00640 1 -- 172.21.15.6:0/1875699738 wait complete. 2024-03-06T07:28:43.901 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:43.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:44.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.113+0000 7f01d2200640 1 Processor -- start 2024-03-06T07:28:44.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.113+0000 7f01d2200640 1 -- start start 2024-03-06T07:28:44.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.115+0000 7f01d2200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.115+0000 7f01d2200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f01cc0cf130 0x7f01cc0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.115+0000 7f01d2200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc0d0910 con 0x7f01cc144c00 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.115+0000 7f01d2200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc06b950 con 0x7f01cc0cf130 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.115+0000 7f01d2200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0d0bf0 con 0x7f01cc06afb0 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01cbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01d1800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f01cc0cf130 0x7f01cc0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01d1800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f01cc0cf130 0x7f01cc0cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:53604/0 (socket says 172.21.15.6:53604) 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01d1800640 1 -- 172.21.15.6:0/857909433 learned_addr learned my addr 172.21.15.6:0/857909433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01cbe00640 1 -- 172.21.15.6:0/857909433 >> v2:172.21.15.178:3300/0 conn(0x7f01cc0cf130 msgr2=0x7f01cc0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01cbe00640 1 --2- 172.21.15.6:0/857909433 >> v2:172.21.15.178:3300/0 conn(0x7f01cc0cf130 0x7f01cc0cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01cbe00640 1 -- 172.21.15.6:0/857909433 >> v1:172.21.15.178:6789/0 conn(0x7f01cc144c00 legacy=0x7f01cc145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.116+0000 7f01cbe00640 1 -- 172.21.15.6:0/857909433 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc0d0ed0 con 0x7f01cc06afb0 2024-03-06T07:28:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.118+0000 7f01cbe00640 1 --2- 172.21.15.6:0/857909433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc06b410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f01b80097a0 tx=0x7f01b802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afafc1ce38fd30f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.119+0000 7f01cb400640 1 -- 172.21.15.6:0/857909433 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8035020 con 0x7f01cc06afb0 2024-03-06T07:28:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.119+0000 7f01cb400640 1 -- 172.21.15.6:0/857909433 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8031830 con 0x7f01cc06afb0 2024-03-06T07:28:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.119+0000 7f01cb400640 1 -- 172.21.15.6:0/857909433 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8031c00 con 0x7f01cc06afb0 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.119+0000 7f01d2200640 1 -- 172.21.15.6:0/857909433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 msgr2=0x7f01cc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.119+0000 7f01d2200640 1 --2- 172.21.15.6:0/857909433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc06b410 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f01b80097a0 tx=0x7f01b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.120+0000 7f01d2200640 1 -- 172.21.15.6:0/857909433 shutdown_connections 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.120+0000 7f01d2200640 1 --2- 172.21.15.6:0/857909433 >> v2:172.21.15.178:3300/0 conn(0x7f01cc0cf130 0x7f01cc0cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.120+0000 7f01d2200640 1 --2- 172.21.15.6:0/857909433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc06b410 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.120+0000 7f01d2200640 1 -- 172.21.15.6:0/857909433 >> 172.21.15.6:0/857909433 conn(0x7f01cc06c0c0 msgr2=0x7f01cc06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.120+0000 7f01d2200640 1 -- 172.21.15.6:0/857909433 shutdown_connections 2024-03-06T07:28:44.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.120+0000 7f01d2200640 1 -- 172.21.15.6:0/857909433 wait complete. 2024-03-06T07:28:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.122+0000 7f01d2200640 1 Processor -- start 2024-03-06T07:28:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.123+0000 7f01d2200640 1 -- start start 2024-03-06T07:28:44.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.124+0000 7f01d2200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc0b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.124+0000 7f01d2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 0x7f01cc139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.124+0000 7f01d2200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01cc144c00 0x7f01cc156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.124+0000 7f01d2200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc144fe0 con 0x7f01cc0cf130 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.124+0000 7f01d2200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc0d1490 con 0x7f01cc144c00 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.124+0000 7f01d2200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc145990 con 0x7f01cc06afb0 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 0x7f01cc139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 0x7f01cc139140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42792/0 (socket says 172.21.15.6:42792) 2024-03-06T07:28:44.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 -- 172.21.15.6:0/2629517981 learned_addr learned my addr 172.21.15.6:0/2629517981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:44.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01d1800640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01cc144c00 0x7f01cc156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01d0e00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc0b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 -- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 msgr2=0x7f01cc0b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc0b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 -- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01cc144c00 msgr2=0x7f01cc156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01cc144c00 0x7f01cc156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01cbe00640 1 -- 172.21.15.6:0/2629517981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc0ade20 con 0x7f01cc0cf130 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.125+0000 7f01d0e00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc0b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.126+0000 7f01cbe00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 0x7f01cc139140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f01b8034040 tx=0x7f01b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.126+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8040070 con 0x7f01cc0cf130 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.126+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b803a910 con 0x7f01cc0cf130 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.126+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8035040 con 0x7f01cc0cf130 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.126+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc0b4ec0 con 0x7f01cc0cf130 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.126+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc0b5480 con 0x7f01cc0cf130 2024-03-06T07:28:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.127+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc04f330 con 0x7f01cc0cf130 2024-03-06T07:28:44.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.129+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01b8038020 con 0x7f01cc0cf130 2024-03-06T07:28:44.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.130+0000 7f01c8c00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01ac06b4d0 0x7f01ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.130+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01b80abd20 con 0x7f01cc0cf130 2024-03-06T07:28:44.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.130+0000 7f01d0e00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01ac06b4d0 0x7f01ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.132+0000 7f01d0e00640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01ac06b4d0 0x7f01ac06d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01c0001680 tx=0x7f01c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.135+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01b8031800 con 0x7f01cc0cf130 2024-03-06T07:28:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.373+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01cc0722f0 con 0x7f01cc0cf130 2024-03-06T07:28:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.375+0000 7f01c8c00640 1 -- 172.21.15.6:0/2629517981 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f01b807ab00 con 0x7f01cc0cf130 2024-03-06T07:28:44.377 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:44.377 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:28:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:28:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01ac06b4d0 msgr2=0x7f01ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01ac06b4d0 0x7f01ac06d990 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f01c0001680 tx=0x7f01c0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 msgr2=0x7f01cc139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 0x7f01cc139140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f01b8034040 tx=0x7f01b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 shutdown_connections 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01ac06b4d0 0x7f01ac06d990 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01cc144c00 0x7f01cc156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01cc0cf130 0x7f01cc139140 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 --2- 172.21.15.6:0/2629517981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01cc06afb0 0x7f01cc0b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 >> 172.21.15.6:0/2629517981 conn(0x7f01cc06c0c0 msgr2=0x7f01cc0cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 shutdown_connections 2024-03-06T07:28:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.377+0000 7f01d2200640 1 -- 172.21.15.6:0/2629517981 wait complete. 2024-03-06T07:28:44.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:44.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.602+0000 7f84cbe00640 1 Processor -- start 2024-03-06T07:28:44.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.602+0000 7f84cbe00640 1 -- start start 2024-03-06T07:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.604+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 0x7f84c4162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.604+0000 7f84cbe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84c406bfc0 0x7f84c406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.604+0000 7f84cbe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c4161a80 con 0x7f84c415e0e0 2024-03-06T07:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.604+0000 7f84cbe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c406c960 con 0x7f84c406bfc0 2024-03-06T07:28:44.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.604+0000 7f84cbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c415efa0 con 0x7f84c415dd00 2024-03-06T07:28:44.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.605+0000 7f84c9400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 0x7f84c4162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.605+0000 7f84c8a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84c406bfc0 0x7f84c406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.605+0000 7f84c9400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 0x7f84c4162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:41918/0 (socket says 172.21.15.6:41918) 2024-03-06T07:28:44.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.605+0000 7f84c9400640 1 -- 172.21.15.6:0/1787651696 learned_addr learned my addr 172.21.15.6:0/1787651696 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.606+0000 7f84c9400640 1 -- 172.21.15.6:0/1787651696 >> v2:172.21.15.178:3300/0 conn(0x7f84c406bfc0 msgr2=0x7f84c406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.606+0000 7f84c9400640 1 --2- 172.21.15.6:0/1787651696 >> v2:172.21.15.178:3300/0 conn(0x7f84c406bfc0 0x7f84c406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.606+0000 7f84c9400640 1 -- 172.21.15.6:0/1787651696 >> v1:172.21.15.178:6789/0 conn(0x7f84c415e0e0 legacy=0x7f84c415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.606+0000 7f84c9400640 1 -- 172.21.15.6:0/1787651696 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c415f280 con 0x7f84c415dd00 2024-03-06T07:28:44.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.607+0000 7f84c9400640 1 --2- 172.21.15.6:0/1787651696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 0x7f84c4162520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009870 tx=0x7f84b802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2d57300cb903a74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.608+0000 7f84c3e00640 1 -- 172.21.15.6:0/1787651696 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8034070 con 0x7f84c415dd00 2024-03-06T07:28:44.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.608+0000 7f84c3e00640 1 -- 172.21.15.6:0/1787651696 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b8030840 con 0x7f84c415dd00 2024-03-06T07:28:44.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.608+0000 7f84c3e00640 1 -- 172.21.15.6:0/1787651696 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b8030c10 con 0x7f84c415dd00 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 -- 172.21.15.6:0/1787651696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 msgr2=0x7f84c4162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 --2- 172.21.15.6:0/1787651696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 0x7f84c4162520 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009870 tx=0x7f84b802b980 comp rx=0 tx=0).stop 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 -- 172.21.15.6:0/1787651696 shutdown_connections 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 --2- 172.21.15.6:0/1787651696 >> v2:172.21.15.178:3300/0 conn(0x7f84c406bfc0 0x7f84c406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 --2- 172.21.15.6:0/1787651696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c415dd00 0x7f84c4162520 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 -- 172.21.15.6:0/1787651696 >> 172.21.15.6:0/1787651696 conn(0x7f84c40714d0 msgr2=0x7f84c40718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 -- 172.21.15.6:0/1787651696 shutdown_connections 2024-03-06T07:28:44.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.609+0000 7f84cbe00640 1 -- 172.21.15.6:0/1787651696 wait complete. 2024-03-06T07:28:44.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.611+0000 7f84cbe00640 1 Processor -- start 2024-03-06T07:28:44.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.611+0000 7f84cbe00640 1 -- start start 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.612+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c406bfc0 0x7f84c415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.612+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c415dd00 0x7f84c40c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.612+0000 7f84cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 0x7f84c40c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.612+0000 7f84cbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c415ec60 con 0x7f84c415e0e0 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.612+0000 7f84cbe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c4162000 con 0x7f84c415dd00 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.612+0000 7f84cbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c406f280 con 0x7f84c406bfc0 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 0x7f84c40c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 0x7f84c40c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42798/0 (socket says 172.21.15.6:42798) 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 -- 172.21.15.6:0/2517976920 learned_addr learned my addr 172.21.15.6:0/2517976920 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:44.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c8a00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c415dd00 0x7f84c40c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9400640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c406bfc0 0x7f84c415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 -- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c406bfc0 msgr2=0x7f84c415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c406bfc0 0x7f84c415c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 -- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c415dd00 msgr2=0x7f84c40c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c415dd00 0x7f84c40c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.613+0000 7f84c9e00640 1 -- 172.21.15.6:0/2517976920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40c5f10 con 0x7f84c415e0e0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.614+0000 7f84c9e00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 0x7f84c40c59d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00c730 tx=0x7f84bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.614+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc003b90 con 0x7f84c415e0e0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.614+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc003d30 con 0x7f84c415e0e0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.614+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc00aa90 con 0x7f84c415e0e0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.614+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c4148110 con 0x7f84c415e0e0 2024-03-06T07:28:44.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.614+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c4145840 con 0x7f84c415e0e0 2024-03-06T07:28:44.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.616+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c41471a0 con 0x7f84c415e0e0 2024-03-06T07:28:44.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.616+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f84bc00ac30 con 0x7f84c415e0e0 2024-03-06T07:28:44.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.617+0000 7f84c1600640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84a406b4d0 0x7f84a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:44.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.617+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84bc08a0f0 con 0x7f84c415e0e0 2024-03-06T07:28:44.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.618+0000 7f84c9400640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84a406b4d0 0x7f84a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:44.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.619+0000 7f84c9400640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84a406b4d0 0x7f84a406d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f84b8036000 tx=0x7f84b803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:44.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.623+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f84c41471a0 con 0x7f84c415e0e0 2024-03-06T07:28:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.908+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c404f3a0 con 0x7f84c415e0e0 2024-03-06T07:28:44.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.911+0000 7f84c1600640 1 -- 172.21.15.6:0/2517976920 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f84bc058f70 con 0x7f84c415e0e0 2024-03-06T07:28:44.914 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:44.914 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:41.851077+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:44.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84a406b4d0 msgr2=0x7f84a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84a406b4d0 0x7f84a406d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f84b8036000 tx=0x7f84b803b040 comp rx=0 tx=0).stop 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 msgr2=0x7f84c40c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 0x7f84c40c59d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00c730 tx=0x7f84bc00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 shutdown_connections 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84a406b4d0 0x7f84a406d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84c415dd00 0x7f84c40c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84c415e0e0 0x7f84c40c59d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 --2- 172.21.15.6:0/2517976920 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84c406bfc0 0x7f84c415c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.914+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 >> 172.21.15.6:0/2517976920 conn(0x7f84c40714d0 msgr2=0x7f84c40af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.915+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 shutdown_connections 2024-03-06T07:28:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:44.915+0000 7f84cbe00640 1 -- 172.21.15.6:0/2517976920 wait complete. 2024-03-06T07:28:44.935 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:44.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.148+0000 7f5c21400640 1 Processor -- start 2024-03-06T07:28:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.148+0000 7f5c21400640 1 -- start start 2024-03-06T07:28:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.149+0000 7f5c21400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5c1c1477e0 0x7f5c1c162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c21400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c147bc0 con 0x7f5c1c06ef20 2024-03-06T07:28:45.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c21400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0b3d70 con 0x7f5c1c06b6c0 2024-03-06T07:28:45.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c21400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c161f90 con 0x7f5c1c1477e0 2024-03-06T07:28:45.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c1b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5c1c06ef20 0x7f5c1c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48252/0 (socket says 172.21.15.6:48252) 2024-03-06T07:28:45.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c1b400640 1 -- 172.21.15.6:0/1713558955 learned_addr learned my addr 172.21.15.6:0/1713558955 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c1be00640 1 --2- 172.21.15.6:0/1713558955 >> v2:172.21.15.178:3300/0 conn(0x7f5c1c1477e0 0x7f5c1c162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237319443 0 0) 0x7f5c1c147bc0 con 0x7f5c1c06ef20 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.150+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c00003520 con 0x7f5c1c06ef20 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465718582 0 0) 0x7f5c00003520 con 0x7f5c1c06ef20 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 >> v2:172.21.15.178:3300/0 conn(0x7f5c1c1477e0 msgr2=0x7f5c1c162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 --2- 172.21.15.6:0/1713558955 >> v2:172.21.15.178:3300/0 conn(0x7f5c1c1477e0 0x7f5c1c162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 >> v1:172.21.15.178:6789/0 conn(0x7f5c1c06b6c0 legacy=0x7f5c1c0d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0b4050 con 0x7f5c1c06ef20 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5c08001670 con 0x7f5c1c06ef20 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 >> v1:172.21.15.6:6789/0 conn(0x7f5c1c06ef20 legacy=0x7f5c1c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000039b0 0x7f5c00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c00004800 0x7f5c00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c000051c0 0x7f5c0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c161f90 con 0x7f5c000051c0 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0000c160 con 0x7f5c000039b0 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0000c360 con 0x7f5c00004800 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.151+0000 7f5c20a00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c000051c0 0x7f5c0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c1be00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000039b0 0x7f5c00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c1b400640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c00004800 0x7f5c00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c20a00640 1 -- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c00004800 msgr2=0x7f5c00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c20a00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c00004800 0x7f5c00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c20a00640 1 -- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000039b0 msgr2=0x7f5c00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c20a00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000039b0 0x7f5c00003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c20a00640 1 -- 172.21.15.6:0/1713558955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c00003520 con 0x7f5c000051c0 2024-03-06T07:28:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.152+0000 7f5c1b400640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c00004800 0x7f5c00004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.153+0000 7f5c20a00640 1 -- 172.21.15.6:0/1713558955 reap_dead start 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.153+0000 7f5c20a00640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c000051c0 0x7f5c0000b9c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5c14001e20 tx=0x7f5c00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6f84d57d365925 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14001a70 con 0x7f5c000051c0 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14003b70 con 0x7f5c000051c0 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c1aa00640 1 -- 172.21.15.6:0/1713558955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c140050d0 con 0x7f5c000051c0 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c21400640 1 -- 172.21.15.6:0/1713558955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c000051c0 msgr2=0x7f5c0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c21400640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c000051c0 0x7f5c0000b9c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5c14001e20 tx=0x7f5c00005830 comp rx=0 tx=0).stop 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c21400640 1 -- 172.21.15.6:0/1713558955 shutdown_connections 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c21400640 1 --2- 172.21.15.6:0/1713558955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c000051c0 0x7f5c0000b9c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.154+0000 7f5c21400640 1 -- 172.21.15.6:0/1713558955 >> 172.21.15.6:0/1713558955 conn(0x7f5c1c06c0c0 msgr2=0x7f5c1c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.155+0000 7f5c21400640 1 -- 172.21.15.6:0/1713558955 shutdown_connections 2024-03-06T07:28:45.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.155+0000 7f5c21400640 1 -- 172.21.15.6:0/1713558955 wait complete. 2024-03-06T07:28:45.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.156+0000 7f5c21400640 1 Processor -- start 2024-03-06T07:28:45.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.157+0000 7f5c21400640 1 -- start start 2024-03-06T07:28:45.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c21400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000051c0 0x7f5c1c0ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c21400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 0x7f5c1c160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c21400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c1c1f2da0 0x7f5c1c1f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c21400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c06f8f0 con 0x7f5c1c142170 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c21400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c161c20 con 0x7f5c000051c0 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c21400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c06fa70 con 0x7f5c1c1f2da0 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.158+0000 7f5c1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 0x7f5c1c160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 0x7f5c1c160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42816/0 (socket says 172.21.15.6:42816) 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 -- 172.21.15.6:0/2155476990 learned_addr learned my addr 172.21.15.6:0/2155476990 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1be00640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000051c0 0x7f5c1c0ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c20a00640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c1c1f2da0 0x7f5c1c1f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 -- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c1c1f2da0 msgr2=0x7f5c1c1f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c1c1f2da0 0x7f5c1c1f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 -- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000051c0 msgr2=0x7f5c1c0ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000051c0 0x7f5c1c0ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 -- 172.21.15.6:0/2155476990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0cd290 con 0x7f5c1c142170 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1be00640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000051c0 0x7f5c1c0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.159+0000 7f5c1b400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 0x7f5c1c160690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800e7b0 tx=0x7f5c0800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.160+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0800c870 con 0x7f5c1c142170 2024-03-06T07:28:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.160+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c08018070 con 0x7f5c1c142170 2024-03-06T07:28:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.160+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0800cc10 con 0x7f5c1c142170 2024-03-06T07:28:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.160+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c1f3790 con 0x7f5c1c142170 2024-03-06T07:28:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.160+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c13e0f0 con 0x7f5c1c142170 2024-03-06T07:28:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.161+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c04f330 con 0x7f5c1c142170 2024-03-06T07:28:45.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.162+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c0801d460 con 0x7f5c1c142170 2024-03-06T07:28:45.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.163+0000 7f5c13e00640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.163+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c0808a070 con 0x7f5c1c142170 2024-03-06T07:28:45.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.164+0000 7f5c1be00640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.165+0000 7f5c1be00640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009a40 tx=0x7f5c0c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.167+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c08058ee0 con 0x7f5c1c142170 2024-03-06T07:28:45.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.421+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c1c072ae0 con 0x7f5bfc06b4d0 2024-03-06T07:28:45.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.441+0000 7f5c13e00640 1 -- 172.21.15.6:0/2155476990 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251897 (secure 0 0 0) 0x7f5c1c072ae0 con 0x7f5bfc06b4d0 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:28:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5bfc06b4d0 msgr2=0x7f5bfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009a40 tx=0x7f5c0c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 msgr2=0x7f5c1c160690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 0x7f5c1c160690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800e7b0 tx=0x7f5c0800ec70 comp rx=0 tx=0).stop 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 shutdown_connections 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5bfc06b4d0 0x7f5bfc06d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c1c1f2da0 0x7f5c1c1f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c1c142170 0x7f5c1c160690 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 --2- 172.21.15.6:0/2155476990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c000051c0 0x7f5c1c0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.447+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 >> 172.21.15.6:0/2155476990 conn(0x7f5c1c06c0c0 msgr2=0x7f5c1c145370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.448+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 shutdown_connections 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.448+0000 7f5c21400640 1 -- 172.21.15.6:0/2155476990 wait complete. 2024-03-06T07:28:45.470 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-03-06T07:28:43.462893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236976,"kb_used_data":4144,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737744,"statfs":{"total":773094113280,"available":772851449856,"internally_reserved":0,"allocated":4243456,"data_stored":2357664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.672774"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871675+0000","last_change":"2024-03-06T07:28:40.869031+0000","last_active":"2024-03-06T07:28:40.871675+0000","last_peered":"2024-03-06T07:28:40.871675+0000","last_clean":"2024-03-06T07:28:40.871675+0000","last_became_active":"2024-03-06T07:28:40.868760+0000","last_became_peered":"2024-03-06T07:28:40.868760+0000","last_unstale":"2024-03-06T07:28:40.871675+0000","last_undegraded":"2024-03-06T07:28:40.871675+0000","last_fullsized":"2024-03-06T07:28:40.871675+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:47.710001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.857551+0000","last_change":"2024-03-06T07:28:40.848398+0000","last_active":"2024-03-06T07:28:40.857551+0000","last_peered":"2024-03-06T07:28:40.857551+0000","last_clean":"2024-03-06T07:28:40.857551+0000","last_became_active":"2024-03-06T07:28:40.847621+0000","last_became_peered":"2024-03-06T07:28:40.847621+0000","last_unstale":"2024-03-06T07:28:40.857551+0000","last_undegraded":"2024-03-06T07:28:40.857551+0000","last_fullsized":"2024-03-06T07:28:40.857551+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.565225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881867+0000","last_change":"2024-03-06T07:28:40.879289+0000","last_active":"2024-03-06T07:28:40.881867+0000","last_peered":"2024-03-06T07:28:40.881867+0000","last_clean":"2024-03-06T07:28:40.881867+0000","last_became_active":"2024-03-06T07:28:40.879050+0000","last_became_peered":"2024-03-06T07:28:40.879050+0000","last_unstale":"2024-03-06T07:28:40.881867+0000","last_undegraded":"2024-03-06T07:28:40.881867+0000","last_fullsized":"2024-03-06T07:28:40.881867+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:36.401757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.865307+0000","last_change":"2024-03-06T07:28:40.871486+0000","last_active":"2024-03-06T07:28:41.865307+0000","last_peered":"2024-03-06T07:28:41.865307+0000","last_clean":"2024-03-06T07:28:41.865307+0000","last_became_active":"2024-03-06T07:28:40.865725+0000","last_became_peered":"2024-03-06T07:28:40.865725+0000","last_unstale":"2024-03-06T07:28:41.865307+0000","last_undegraded":"2024-03-06T07:28:41.865307+0000","last_fullsized":"2024-03-06T07:28:41.865307+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:05.689350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056067+0000","last_change":"2024-03-06T07:28:40.861325+0000","last_active":"2024-03-06T07:28:42.056067+0000","last_peered":"2024-03-06T07:28:42.056067+0000","last_clean":"2024-03-06T07:28:42.056067+0000","last_became_active":"2024-03-06T07:28:40.860847+0000","last_became_peered":"2024-03-06T07:28:40.860847+0000","last_unstale":"2024-03-06T07:28:42.056067+0000","last_undegraded":"2024-03-06T07:28:42.056067+0000","last_fullsized":"2024-03-06T07:28:42.056067+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:59.058344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.870081+0000","last_change":"2024-03-06T07:28:40.867112+0000","last_active":"2024-03-06T07:28:40.870081+0000","last_peered":"2024-03-06T07:28:40.870081+0000","last_clean":"2024-03-06T07:28:40.870081+0000","last_became_active":"2024-03-06T07:28:40.861082+0000","last_became_peered":"2024-03-06T07:28:40.861082+0000","last_unstale":"2024-03-06T07:28:40.870081+0000","last_undegraded":"2024-03-06T07:28:40.870081+0000","last_fullsized":"2024-03-06T07:28:40.870081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:57.724310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873051+0000","last_change":"2024-03-06T07:28:40.869166+0000","last_active":"2024-03-06T07:28:40.873051+0000","last_peered":"2024-03-06T07:28:40.873051+0000","last_clean":"2024-03-06T07:28:40.873051+0000","last_became_active":"2024-03-06T07:28:40.868642+0000","last_became_peered":"2024-03-06T07:28:40.868642+0000","last_unstale":"2024-03-06T07:28:40.873051+0000","last_undegraded":"2024-03-06T07:28:40.873051+0000","last_fullsized":"2024-03-06T07:28:40.873051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:48.955548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867843+0000","last_change":"2024-03-06T07:28:40.861508+0000","last_active":"2024-03-06T07:28:40.867843+0000","last_peered":"2024-03-06T07:28:40.867843+0000","last_clean":"2024-03-06T07:28:40.867843+0000","last_became_active":"2024-03-06T07:28:40.861226+0000","last_became_peered":"2024-03-06T07:28:40.861226+0000","last_unstale":"2024-03-06T07:28:40.867843+0000","last_undegraded":"2024-03-06T07:28:40.867843+0000","last_fullsized":"2024-03-06T07:28:40.867843+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:56.854757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.877919+0000","last_change":"2024-03-06T07:28:40.876335+0000","last_active":"2024-03-06T07:28:40.877919+0000","last_peered":"2024-03-06T07:28:40.877919+0000","last_clean":"2024-03-06T07:28:40.877919+0000","last_became_active":"2024-03-06T07:28:40.875996+0000","last_became_peered":"2024-03-06T07:28:40.875996+0000","last_unstale":"2024-03-06T07:28:40.877919+0000","last_undegraded":"2024-03-06T07:28:40.877919+0000","last_fullsized":"2024-03-06T07:28:40.877919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.411787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867283+0000","last_change":"2024-03-06T07:28:40.876626+0000","last_active":"2024-03-06T07:28:41.867283+0000","last_peered":"2024-03-06T07:28:41.867283+0000","last_clean":"2024-03-06T07:28:41.867283+0000","last_became_active":"2024-03-06T07:28:40.876362+0000","last_became_peered":"2024-03-06T07:28:40.876362+0000","last_unstale":"2024-03-06T07:28:41.867283+0000","last_undegraded":"2024-03-06T07:28:41.867283+0000","last_fullsized":"2024-03-06T07:28:41.867283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.694349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881493+0000","last_change":"2024-03-06T07:28:40.877223+0000","last_active":"2024-03-06T07:28:40.881493+0000","last_peered":"2024-03-06T07:28:40.881493+0000","last_clean":"2024-03-06T07:28:40.881493+0000","last_became_active":"2024-03-06T07:28:40.876932+0000","last_became_peered":"2024-03-06T07:28:40.876932+0000","last_unstale":"2024-03-06T07:28:40.881493+0000","last_undegraded":"2024-03-06T07:28:40.881493+0000","last_fullsized":"2024-03-06T07:28:40.881493+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:18.515752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867544+0000","last_change":"2024-03-06T07:28:40.864107+0000","last_active":"2024-03-06T07:28:40.867544+0000","last_peered":"2024-03-06T07:28:40.867544+0000","last_clean":"2024-03-06T07:28:40.867544+0000","last_became_active":"2024-03-06T07:28:40.861454+0000","last_became_peered":"2024-03-06T07:28:40.861454+0000","last_unstale":"2024-03-06T07:28:40.867544+0000","last_undegraded":"2024-03-06T07:28:40.867544+0000","last_fullsized":"2024-03-06T07:28:40.867544+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.565225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056262+0000","last_change":"2024-03-06T07:28:40.861149+0000","last_active":"2024-03-06T07:28:42.056262+0000","last_peered":"2024-03-06T07:28:42.056262+0000","last_clean":"2024-03-06T07:28:42.056262+0000","last_became_active":"2024-03-06T07:28:40.860453+0000","last_became_peered":"2024-03-06T07:28:40.860453+0000","last_unstale":"2024-03-06T07:28:42.056262+0000","last_undegraded":"2024-03-06T07:28:42.056262+0000","last_fullsized":"2024-03-06T07:28:42.056262+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:22.581189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869064+0000","last_change":"2024-03-06T07:28:40.858071+0000","last_active":"2024-03-06T07:28:40.869064+0000","last_peered":"2024-03-06T07:28:40.869064+0000","last_clean":"2024-03-06T07:28:40.869064+0000","last_became_active":"2024-03-06T07:28:40.853230+0000","last_became_peered":"2024-03-06T07:28:40.853230+0000","last_unstale":"2024-03-06T07:28:40.869064+0000","last_undegraded":"2024-03-06T07:28:40.869064+0000","last_fullsized":"2024-03-06T07:28:40.869064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:15.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.865831+0000","last_change":"2024-03-06T07:28:40.860185+0000","last_active":"2024-03-06T07:28:40.865831+0000","last_peered":"2024-03-06T07:28:40.865831+0000","last_clean":"2024-03-06T07:28:40.865831+0000","last_became_active":"2024-03-06T07:28:40.859859+0000","last_became_peered":"2024-03-06T07:28:40.859859+0000","last_unstale":"2024-03-06T07:28:40.865831+0000","last_undegraded":"2024-03-06T07:28:40.865831+0000","last_fullsized":"2024-03-06T07:28:40.865831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:57.408991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873563+0000","last_change":"2024-03-06T07:28:40.868249+0000","last_active":"2024-03-06T07:28:40.873563+0000","last_peered":"2024-03-06T07:28:40.873563+0000","last_clean":"2024-03-06T07:28:40.873563+0000","last_became_active":"2024-03-06T07:28:40.868010+0000","last_became_peered":"2024-03-06T07:28:40.868010+0000","last_unstale":"2024-03-06T07:28:40.873563+0000","last_undegraded":"2024-03-06T07:28:40.873563+0000","last_fullsized":"2024-03-06T07:28:40.873563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:08.961770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871283+0000","last_change":"2024-03-06T07:28:40.863703+0000","last_active":"2024-03-06T07:28:40.871283+0000","last_peered":"2024-03-06T07:28:40.871283+0000","last_clean":"2024-03-06T07:28:40.871283+0000","last_became_active":"2024-03-06T07:28:40.863403+0000","last_became_peered":"2024-03-06T07:28:40.863403+0000","last_unstale":"2024-03-06T07:28:40.871283+0000","last_undegraded":"2024-03-06T07:28:40.871283+0000","last_fullsized":"2024-03-06T07:28:40.871283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:24.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882440+0000","last_change":"2024-03-06T07:28:40.877591+0000","last_active":"2024-03-06T07:28:40.882440+0000","last_peered":"2024-03-06T07:28:40.882440+0000","last_clean":"2024-03-06T07:28:40.882440+0000","last_became_active":"2024-03-06T07:28:40.876954+0000","last_became_peered":"2024-03-06T07:28:40.876954+0000","last_unstale":"2024-03-06T07:28:40.882440+0000","last_undegraded":"2024-03-06T07:28:40.882440+0000","last_fullsized":"2024-03-06T07:28:40.882440+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:14.014615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869158+0000","last_change":"2024-03-06T07:28:40.866649+0000","last_active":"2024-03-06T07:28:40.869158+0000","last_peered":"2024-03-06T07:28:40.869158+0000","last_clean":"2024-03-06T07:28:40.869158+0000","last_became_active":"2024-03-06T07:28:40.866348+0000","last_became_peered":"2024-03-06T07:28:40.866348+0000","last_unstale":"2024-03-06T07:28:40.869158+0000","last_undegraded":"2024-03-06T07:28:40.869158+0000","last_fullsized":"2024-03-06T07:28:40.869158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:13.090146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056192+0000","last_change":"2024-03-06T07:28:40.868159+0000","last_active":"2024-03-06T07:28:42.056192+0000","last_peered":"2024-03-06T07:28:42.056192+0000","last_clean":"2024-03-06T07:28:42.056192+0000","last_became_active":"2024-03-06T07:28:40.864562+0000","last_became_peered":"2024-03-06T07:28:40.864562+0000","last_unstale":"2024-03-06T07:28:42.056192+0000","last_undegraded":"2024-03-06T07:28:42.056192+0000","last_fullsized":"2024-03-06T07:28:42.056192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:35.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869739+0000","last_change":"2024-03-06T07:28:40.856912+0000","last_active":"2024-03-06T07:28:40.869739+0000","last_peered":"2024-03-06T07:28:40.869739+0000","last_clean":"2024-03-06T07:28:40.869739+0000","last_became_active":"2024-03-06T07:28:40.856733+0000","last_became_peered":"2024-03-06T07:28:40.856733+0000","last_unstale":"2024-03-06T07:28:40.869739+0000","last_undegraded":"2024-03-06T07:28:40.869739+0000","last_fullsized":"2024-03-06T07:28:40.869739+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:11.454733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.877381+0000","last_change":"2024-03-06T07:28:40.872043+0000","last_active":"2024-03-06T07:28:40.877381+0000","last_peered":"2024-03-06T07:28:40.877381+0000","last_clean":"2024-03-06T07:28:40.877381+0000","last_became_active":"2024-03-06T07:28:40.871701+0000","last_became_peered":"2024-03-06T07:28:40.871701+0000","last_unstale":"2024-03-06T07:28:40.877381+0000","last_undegraded":"2024-03-06T07:28:40.877381+0000","last_fullsized":"2024-03-06T07:28:40.877381+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:06.193722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.863350+0000","last_change":"2024-03-06T07:28:40.856856+0000","last_active":"2024-03-06T07:28:40.863350+0000","last_peered":"2024-03-06T07:28:40.863350+0000","last_clean":"2024-03-06T07:28:40.863350+0000","last_became_active":"2024-03-06T07:28:40.856526+0000","last_became_peered":"2024-03-06T07:28:40.856526+0000","last_unstale":"2024-03-06T07:28:40.863350+0000","last_undegraded":"2024-03-06T07:28:40.863350+0000","last_fullsized":"2024-03-06T07:28:40.863350+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:54.840821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.864076+0000","last_change":"2024-03-06T07:28:40.856769+0000","last_active":"2024-03-06T07:28:40.864076+0000","last_peered":"2024-03-06T07:28:40.864076+0000","last_clean":"2024-03-06T07:28:40.864076+0000","last_became_active":"2024-03-06T07:28:40.856430+0000","last_became_peered":"2024-03-06T07:28:40.856430+0000","last_unstale":"2024-03-06T07:28:40.864076+0000","last_undegraded":"2024-03-06T07:28:40.864076+0000","last_fullsized":"2024-03-06T07:28:40.864076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:17.684202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867052+0000","last_change":"2024-03-06T07:28:40.871472+0000","last_active":"2024-03-06T07:28:41.867052+0000","last_peered":"2024-03-06T07:28:41.867052+0000","last_clean":"2024-03-06T07:28:41.867052+0000","last_became_active":"2024-03-06T07:28:40.864661+0000","last_became_peered":"2024-03-06T07:28:40.864661+0000","last_unstale":"2024-03-06T07:28:41.867052+0000","last_undegraded":"2024-03-06T07:28:41.867052+0000","last_fullsized":"2024-03-06T07:28:41.867052+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.461309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869128+0000","last_change":"2024-03-06T07:28:40.864124+0000","last_active":"2024-03-06T07:28:40.869128+0000","last_peered":"2024-03-06T07:28:40.869128+0000","last_clean":"2024-03-06T07:28:40.869128+0000","last_became_active":"2024-03-06T07:28:40.862825+0000","last_became_peered":"2024-03-06T07:28:40.862825+0000","last_unstale":"2024-03-06T07:28:40.869128+0000","last_undegraded":"2024-03-06T07:28:40.869128+0000","last_fullsized":"2024-03-06T07:28:40.869128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:07.122213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867094+0000","last_change":"2024-03-06T07:28:40.871729+0000","last_active":"2024-03-06T07:28:41.867094+0000","last_peered":"2024-03-06T07:28:41.867094+0000","last_clean":"2024-03-06T07:28:41.867094+0000","last_became_active":"2024-03-06T07:28:40.865342+0000","last_became_peered":"2024-03-06T07:28:40.865342+0000","last_unstale":"2024-03-06T07:28:41.867094+0000","last_undegraded":"2024-03-06T07:28:41.867094+0000","last_fullsized":"2024-03-06T07:28:41.867094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:27.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.864386+0000","last_change":"2024-03-06T07:28:40.853164+0000","last_active":"2024-03-06T07:28:40.864386+0000","last_peered":"2024-03-06T07:28:40.864386+0000","last_clean":"2024-03-06T07:28:40.864386+0000","last_became_active":"2024-03-06T07:28:40.852837+0000","last_became_peered":"2024-03-06T07:28:40.852837+0000","last_unstale":"2024-03-06T07:28:40.864386+0000","last_undegraded":"2024-03-06T07:28:40.864386+0000","last_fullsized":"2024-03-06T07:28:40.864386+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:59.287242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869765+0000","last_change":"2024-03-06T07:28:40.860765+0000","last_active":"2024-03-06T07:28:40.869765+0000","last_peered":"2024-03-06T07:28:40.869765+0000","last_clean":"2024-03-06T07:28:40.869765+0000","last_became_active":"2024-03-06T07:28:40.860376+0000","last_became_peered":"2024-03-06T07:28:40.860376+0000","last_unstale":"2024-03-06T07:28:40.869765+0000","last_undegraded":"2024-03-06T07:28:40.869765+0000","last_fullsized":"2024-03-06T07:28:40.869765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:55.894729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869898+0000","last_change":"2024-03-06T07:28:40.862685+0000","last_active":"2024-03-06T07:28:40.869898+0000","last_peered":"2024-03-06T07:28:40.869898+0000","last_clean":"2024-03-06T07:28:40.869898+0000","last_became_active":"2024-03-06T07:28:40.862399+0000","last_became_peered":"2024-03-06T07:28:40.862399+0000","last_unstale":"2024-03-06T07:28:40.869898+0000","last_undegraded":"2024-03-06T07:28:40.869898+0000","last_fullsized":"2024-03-06T07:28:40.869898+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:35.590273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871566+0000","last_change":"2024-03-06T07:28:40.865321+0000","last_active":"2024-03-06T07:28:40.871566+0000","last_peered":"2024-03-06T07:28:40.871566+0000","last_clean":"2024-03-06T07:28:40.871566+0000","last_became_active":"2024-03-06T07:28:40.865038+0000","last_became_peered":"2024-03-06T07:28:40.865038+0000","last_unstale":"2024-03-06T07:28:40.871566+0000","last_undegraded":"2024-03-06T07:28:40.871566+0000","last_fullsized":"2024-03-06T07:28:40.871566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:37.635530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873521+0000","last_change":"2024-03-06T07:28:40.867951+0000","last_active":"2024-03-06T07:28:40.873521+0000","last_peered":"2024-03-06T07:28:40.873521+0000","last_clean":"2024-03-06T07:28:40.873521+0000","last_became_active":"2024-03-06T07:28:40.867609+0000","last_became_peered":"2024-03-06T07:28:40.867609+0000","last_unstale":"2024-03-06T07:28:40.873521+0000","last_undegraded":"2024-03-06T07:28:40.873521+0000","last_fullsized":"2024-03-06T07:28:40.873521+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:06.193722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.868903+0000","last_change":"2024-03-06T07:28:40.862477+0000","last_active":"2024-03-06T07:28:40.868903+0000","last_peered":"2024-03-06T07:28:40.868903+0000","last_clean":"2024-03-06T07:28:40.868903+0000","last_became_active":"2024-03-06T07:28:40.862208+0000","last_became_peered":"2024-03-06T07:28:40.862208+0000","last_unstale":"2024-03-06T07:28:40.868903+0000","last_undegraded":"2024-03-06T07:28:40.868903+0000","last_fullsized":"2024-03-06T07:28:40.868903+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:32.044942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882365+0000","last_change":"2024-03-06T07:28:39.831689+0000","last_active":"2024-03-06T07:28:40.882365+0000","last_peered":"2024-03-06T07:28:40.882365+0000","last_clean":"2024-03-06T07:28:40.882365+0000","last_became_active":"2024-03-06T07:28:39.831154+0000","last_became_peered":"2024-03-06T07:28:39.831154+0000","last_unstale":"2024-03-06T07:28:40.882365+0000","last_undegraded":"2024-03-06T07:28:40.882365+0000","last_fullsized":"2024-03-06T07:28:40.882365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:55.978361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871697+0000","last_change":"2024-03-06T07:28:40.869278+0000","last_active":"2024-03-06T07:28:40.871697+0000","last_peered":"2024-03-06T07:28:40.871697+0000","last_clean":"2024-03-06T07:28:40.871697+0000","last_became_active":"2024-03-06T07:28:40.867903+0000","last_became_peered":"2024-03-06T07:28:40.867903+0000","last_unstale":"2024-03-06T07:28:40.871697+0000","last_undegraded":"2024-03-06T07:28:40.871697+0000","last_fullsized":"2024-03-06T07:28:40.871697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:24.261260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.879244+0000","last_change":"2024-03-06T07:28:39.831295+0000","last_active":"2024-03-06T07:28:40.879244+0000","last_peered":"2024-03-06T07:28:40.879244+0000","last_clean":"2024-03-06T07:28:40.879244+0000","last_became_active":"2024-03-06T07:28:39.830638+0000","last_became_peered":"2024-03-06T07:28:39.830638+0000","last_unstale":"2024-03-06T07:28:40.879244+0000","last_undegraded":"2024-03-06T07:28:40.879244+0000","last_fullsized":"2024-03-06T07:28:40.879244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:36.630626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.877219+0000","last_change":"2024-03-06T07:28:40.871580+0000","last_active":"2024-03-06T07:28:40.877219+0000","last_peered":"2024-03-06T07:28:40.877219+0000","last_clean":"2024-03-06T07:28:40.877219+0000","last_became_active":"2024-03-06T07:28:40.870854+0000","last_became_peered":"2024-03-06T07:28:40.870854+0000","last_unstale":"2024-03-06T07:28:40.877219+0000","last_undegraded":"2024-03-06T07:28:40.877219+0000","last_fullsized":"2024-03-06T07:28:40.877219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:24.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.885641+0000","last_change":"2024-03-06T07:28:39.833946+0000","last_active":"2024-03-06T07:28:40.885641+0000","last_peered":"2024-03-06T07:28:40.885641+0000","last_clean":"2024-03-06T07:28:40.885641+0000","last_became_active":"2024-03-06T07:28:39.833665+0000","last_became_peered":"2024-03-06T07:28:39.833665+0000","last_unstale":"2024-03-06T07:28:40.885641+0000","last_undegraded":"2024-03-06T07:28:40.885641+0000","last_fullsized":"2024-03-06T07:28:40.885641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:16.679298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867303+0000","last_change":"2024-03-06T07:28:40.871442+0000","last_active":"2024-03-06T07:28:41.867303+0000","last_peered":"2024-03-06T07:28:41.867303+0000","last_clean":"2024-03-06T07:28:41.867303+0000","last_became_active":"2024-03-06T07:28:40.863935+0000","last_became_peered":"2024-03-06T07:28:40.863935+0000","last_unstale":"2024-03-06T07:28:41.867303+0000","last_undegraded":"2024-03-06T07:28:41.867303+0000","last_fullsized":"2024-03-06T07:28:41.867303+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:11.062252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056335+0000","last_change":"2024-03-06T07:28:39.833643+0000","last_active":"2024-03-06T07:28:42.056335+0000","last_peered":"2024-03-06T07:28:42.056335+0000","last_clean":"2024-03-06T07:28:42.056335+0000","last_became_active":"2024-03-06T07:28:39.833170+0000","last_became_peered":"2024-03-06T07:28:39.833170+0000","last_unstale":"2024-03-06T07:28:42.056335+0000","last_undegraded":"2024-03-06T07:28:42.056335+0000","last_fullsized":"2024-03-06T07:28:42.056335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:36.553751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.874948+0000","last_change":"2024-03-06T07:28:40.872246+0000","last_active":"2024-03-06T07:28:40.874948+0000","last_peered":"2024-03-06T07:28:40.874948+0000","last_clean":"2024-03-06T07:28:40.874948+0000","last_became_active":"2024-03-06T07:28:40.871404+0000","last_became_peered":"2024-03-06T07:28:40.871404+0000","last_unstale":"2024-03-06T07:28:40.874948+0000","last_undegraded":"2024-03-06T07:28:40.874948+0000","last_fullsized":"2024-03-06T07:28:40.874948+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:05.259044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882996+0000","last_change":"2024-03-06T07:28:39.834605+0000","last_active":"2024-03-06T07:28:40.882996+0000","last_peered":"2024-03-06T07:28:40.882996+0000","last_clean":"2024-03-06T07:28:40.882996+0000","last_became_active":"2024-03-06T07:28:39.834291+0000","last_became_peered":"2024-03-06T07:28:39.834291+0000","last_unstale":"2024-03-06T07:28:40.882996+0000","last_undegraded":"2024-03-06T07:28:40.882996+0000","last_fullsized":"2024-03-06T07:28:40.882996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:09.296468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.865538+0000","last_change":"2024-03-06T07:28:40.859013+0000","last_active":"2024-03-06T07:28:40.865538+0000","last_peered":"2024-03-06T07:28:40.865538+0000","last_clean":"2024-03-06T07:28:40.865538+0000","last_became_active":"2024-03-06T07:28:40.858711+0000","last_became_peered":"2024-03-06T07:28:40.858711+0000","last_unstale":"2024-03-06T07:28:40.865538+0000","last_undegraded":"2024-03-06T07:28:40.865538+0000","last_fullsized":"2024-03-06T07:28:40.865538+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:11.722633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.883622+0000","last_change":"2024-03-06T07:28:39.840615+0000","last_active":"2024-03-06T07:28:40.883622+0000","last_peered":"2024-03-06T07:28:40.883622+0000","last_clean":"2024-03-06T07:28:40.883622+0000","last_became_active":"2024-03-06T07:28:39.840213+0000","last_became_peered":"2024-03-06T07:28:39.840213+0000","last_unstale":"2024-03-06T07:28:40.883622+0000","last_undegraded":"2024-03-06T07:28:40.883622+0000","last_fullsized":"2024-03-06T07:28:40.883622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:58.053440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869443+0000","last_change":"2024-03-06T07:28:40.864315+0000","last_active":"2024-03-06T07:28:40.869443+0000","last_peered":"2024-03-06T07:28:40.869443+0000","last_clean":"2024-03-06T07:28:40.869443+0000","last_became_active":"2024-03-06T07:28:40.863397+0000","last_became_peered":"2024-03-06T07:28:40.863397+0000","last_unstale":"2024-03-06T07:28:40.869443+0000","last_undegraded":"2024-03-06T07:28:40.869443+0000","last_fullsized":"2024-03-06T07:28:40.869443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:01.073459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.835349+0000","last_change":"2024-03-06T07:28:39.825334+0000","last_active":"2024-03-06T07:28:40.835349+0000","last_peered":"2024-03-06T07:28:40.835349+0000","last_clean":"2024-03-06T07:28:40.835349+0000","last_became_active":"2024-03-06T07:28:39.824980+0000","last_became_peered":"2024-03-06T07:28:39.824980+0000","last_unstale":"2024-03-06T07:28:40.835349+0000","last_undegraded":"2024-03-06T07:28:40.835349+0000","last_fullsized":"2024-03-06T07:28:40.835349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:08.188611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864679+0000","last_change":"2024-03-06T07:28:40.872116+0000","last_active":"2024-03-06T07:28:41.864679+0000","last_peered":"2024-03-06T07:28:41.864679+0000","last_clean":"2024-03-06T07:28:41.864679+0000","last_became_active":"2024-03-06T07:28:40.871809+0000","last_became_peered":"2024-03-06T07:28:40.871809+0000","last_unstale":"2024-03-06T07:28:41.864679+0000","last_undegraded":"2024-03-06T07:28:41.864679+0000","last_fullsized":"2024-03-06T07:28:41.864679+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:44.690312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.846159+0000","last_change":"2024-03-06T07:28:39.829966+0000","last_active":"2024-03-06T07:28:40.846159+0000","last_peered":"2024-03-06T07:28:40.846159+0000","last_clean":"2024-03-06T07:28:40.846159+0000","last_became_active":"2024-03-06T07:28:39.828232+0000","last_became_peered":"2024-03-06T07:28:39.828232+0000","last_unstale":"2024-03-06T07:28:40.846159+0000","last_undegraded":"2024-03-06T07:28:40.846159+0000","last_fullsized":"2024-03-06T07:28:40.846159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:36.659740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867489+0000","last_change":"2024-03-06T07:28:40.861779+0000","last_active":"2024-03-06T07:28:40.867489+0000","last_peered":"2024-03-06T07:28:40.867489+0000","last_clean":"2024-03-06T07:28:40.867489+0000","last_became_active":"2024-03-06T07:28:40.861389+0000","last_became_peered":"2024-03-06T07:28:40.861389+0000","last_unstale":"2024-03-06T07:28:40.867489+0000","last_undegraded":"2024-03-06T07:28:40.867489+0000","last_fullsized":"2024-03-06T07:28:40.867489+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:22.055775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.838635+0000","last_change":"2024-03-06T07:28:39.830244+0000","last_active":"2024-03-06T07:28:40.838635+0000","last_peered":"2024-03-06T07:28:40.838635+0000","last_clean":"2024-03-06T07:28:40.838635+0000","last_became_active":"2024-03-06T07:28:39.829818+0000","last_became_peered":"2024-03-06T07:28:39.829818+0000","last_unstale":"2024-03-06T07:28:40.838635+0000","last_undegraded":"2024-03-06T07:28:40.838635+0000","last_fullsized":"2024-03-06T07:28:40.838635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:10.717729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882357+0000","last_change":"2024-03-06T07:28:40.877281+0000","last_active":"2024-03-06T07:28:40.882357+0000","last_peered":"2024-03-06T07:28:40.882357+0000","last_clean":"2024-03-06T07:28:40.882357+0000","last_became_active":"2024-03-06T07:28:40.876953+0000","last_became_peered":"2024-03-06T07:28:40.876953+0000","last_unstale":"2024-03-06T07:28:40.882357+0000","last_undegraded":"2024-03-06T07:28:40.882357+0000","last_fullsized":"2024-03-06T07:28:40.882357+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:44.246160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.831215+0000","last_change":"2024-03-06T07:28:39.832385+0000","last_active":"2024-03-06T07:28:40.831215+0000","last_peered":"2024-03-06T07:28:40.831215+0000","last_clean":"2024-03-06T07:28:40.831215+0000","last_became_active":"2024-03-06T07:28:39.832146+0000","last_became_peered":"2024-03-06T07:28:39.832146+0000","last_unstale":"2024-03-06T07:28:40.831215+0000","last_undegraded":"2024-03-06T07:28:40.831215+0000","last_fullsized":"2024-03-06T07:28:40.831215+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:46.705097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.868106+0000","last_change":"2024-03-06T07:28:40.855877+0000","last_active":"2024-03-06T07:28:40.868106+0000","last_peered":"2024-03-06T07:28:40.868106+0000","last_clean":"2024-03-06T07:28:40.868106+0000","last_became_active":"2024-03-06T07:28:40.855602+0000","last_became_peered":"2024-03-06T07:28:40.855602+0000","last_unstale":"2024-03-06T07:28:40.868106+0000","last_undegraded":"2024-03-06T07:28:40.868106+0000","last_fullsized":"2024-03-06T07:28:40.868106+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:50.125370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.883184+0000","last_change":"2024-03-06T07:28:39.836497+0000","last_active":"2024-03-06T07:28:40.883184+0000","last_peered":"2024-03-06T07:28:40.883184+0000","last_clean":"2024-03-06T07:28:40.883184+0000","last_became_active":"2024-03-06T07:28:39.832020+0000","last_became_peered":"2024-03-06T07:28:39.832020+0000","last_unstale":"2024-03-06T07:28:40.883184+0000","last_undegraded":"2024-03-06T07:28:40.883184+0000","last_fullsized":"2024-03-06T07:28:40.883184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:08.188611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881846+0000","last_change":"2024-03-06T07:28:40.879191+0000","last_active":"2024-03-06T07:28:40.881846+0000","last_peered":"2024-03-06T07:28:40.881846+0000","last_clean":"2024-03-06T07:28:40.881846+0000","last_became_active":"2024-03-06T07:28:40.878686+0000","last_became_peered":"2024-03-06T07:28:40.878686+0000","last_unstale":"2024-03-06T07:28:40.881846+0000","last_undegraded":"2024-03-06T07:28:40.881846+0000","last_fullsized":"2024-03-06T07:28:40.881846+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:15.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.897062+0000","last_change":"2024-03-06T07:28:39.837789+0000","last_active":"2024-03-06T07:28:40.897062+0000","last_peered":"2024-03-06T07:28:40.897062+0000","last_clean":"2024-03-06T07:28:40.897062+0000","last_became_active":"2024-03-06T07:28:39.836249+0000","last_became_peered":"2024-03-06T07:28:39.836249+0000","last_unstale":"2024-03-06T07:28:40.897062+0000","last_undegraded":"2024-03-06T07:28:40.897062+0000","last_fullsized":"2024-03-06T07:28:40.897062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:58.282339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864386+0000","last_change":"2024-03-06T07:28:40.871379+0000","last_active":"2024-03-06T07:28:41.864386+0000","last_peered":"2024-03-06T07:28:41.864386+0000","last_clean":"2024-03-06T07:28:41.864386+0000","last_became_active":"2024-03-06T07:28:40.866071+0000","last_became_peered":"2024-03-06T07:28:40.866071+0000","last_unstale":"2024-03-06T07:28:41.864386+0000","last_undegraded":"2024-03-06T07:28:41.864386+0000","last_fullsized":"2024-03-06T07:28:41.864386+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:18.515752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882930+0000","last_change":"2024-03-06T07:28:39.834756+0000","last_active":"2024-03-06T07:28:40.882930+0000","last_peered":"2024-03-06T07:28:40.882930+0000","last_clean":"2024-03-06T07:28:40.882930+0000","last_became_active":"2024-03-06T07:28:39.833731+0000","last_became_peered":"2024-03-06T07:28:39.833731+0000","last_unstale":"2024-03-06T07:28:40.882930+0000","last_undegraded":"2024-03-06T07:28:40.882930+0000","last_fullsized":"2024-03-06T07:28:40.882930+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:34.199312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867878+0000","last_change":"2024-03-06T07:28:40.853500+0000","last_active":"2024-03-06T07:28:40.867878+0000","last_peered":"2024-03-06T07:28:40.867878+0000","last_clean":"2024-03-06T07:28:40.867878+0000","last_became_active":"2024-03-06T07:28:40.853015+0000","last_became_peered":"2024-03-06T07:28:40.853015+0000","last_unstale":"2024-03-06T07:28:40.867878+0000","last_undegraded":"2024-03-06T07:28:40.867878+0000","last_fullsized":"2024-03-06T07:28:40.867878+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:34.394133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864771+0000","last_change":"2024-03-06T07:28:39.814957+0000","last_active":"2024-03-06T07:28:41.864771+0000","last_peered":"2024-03-06T07:28:41.864771+0000","last_clean":"2024-03-06T07:28:41.864771+0000","last_became_active":"2024-03-06T07:28:39.814664+0000","last_became_peered":"2024-03-06T07:28:39.814664+0000","last_unstale":"2024-03-06T07:28:41.864771+0000","last_undegraded":"2024-03-06T07:28:41.864771+0000","last_fullsized":"2024-03-06T07:28:41.864771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:22.337525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873544+0000","last_change":"2024-03-06T07:28:40.868167+0000","last_active":"2024-03-06T07:28:40.873544+0000","last_peered":"2024-03-06T07:28:40.873544+0000","last_clean":"2024-03-06T07:28:40.873544+0000","last_became_active":"2024-03-06T07:28:40.867692+0000","last_became_peered":"2024-03-06T07:28:40.867692+0000","last_unstale":"2024-03-06T07:28:40.873544+0000","last_undegraded":"2024-03-06T07:28:40.873544+0000","last_fullsized":"2024-03-06T07:28:40.873544+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.411787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.834893+0000","last_change":"2024-03-06T07:28:39.829648+0000","last_active":"2024-03-06T07:28:40.834893+0000","last_peered":"2024-03-06T07:28:40.834893+0000","last_clean":"2024-03-06T07:28:40.834893+0000","last_became_active":"2024-03-06T07:28:39.828961+0000","last_became_peered":"2024-03-06T07:28:39.828961+0000","last_unstale":"2024-03-06T07:28:40.834893+0000","last_undegraded":"2024-03-06T07:28:40.834893+0000","last_fullsized":"2024-03-06T07:28:40.834893+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:31.040038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867477+0000","last_change":"2024-03-06T07:28:40.858454+0000","last_active":"2024-03-06T07:28:40.867477+0000","last_peered":"2024-03-06T07:28:40.867477+0000","last_clean":"2024-03-06T07:28:40.867477+0000","last_became_active":"2024-03-06T07:28:40.858149+0000","last_became_peered":"2024-03-06T07:28:40.858149+0000","last_unstale":"2024-03-06T07:28:40.867477+0000","last_undegraded":"2024-03-06T07:28:40.867477+0000","last_fullsized":"2024-03-06T07:28:40.867477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:28.464391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.893528+0000","last_change":"2024-03-06T07:28:39.834458+0000","last_active":"2024-03-06T07:28:40.893528+0000","last_peered":"2024-03-06T07:28:40.893528+0000","last_clean":"2024-03-06T07:28:40.893528+0000","last_became_active":"2024-03-06T07:28:39.833866+0000","last_became_peered":"2024-03-06T07:28:39.833866+0000","last_unstale":"2024-03-06T07:28:40.893528+0000","last_undegraded":"2024-03-06T07:28:40.893528+0000","last_fullsized":"2024-03-06T07:28:40.893528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:56.404088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869653+0000","last_change":"2024-03-06T07:28:40.867062+0000","last_active":"2024-03-06T07:28:40.869653+0000","last_peered":"2024-03-06T07:28:40.869653+0000","last_clean":"2024-03-06T07:28:40.869653+0000","last_became_active":"2024-03-06T07:28:40.866709+0000","last_became_peered":"2024-03-06T07:28:40.866709+0000","last_unstale":"2024-03-06T07:28:40.869653+0000","last_undegraded":"2024-03-06T07:28:40.869653+0000","last_fullsized":"2024-03-06T07:28:40.869653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:56.854757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867281+0000","last_change":"2024-03-06T07:28:39.835450+0000","last_active":"2024-03-06T07:28:41.867281+0000","last_peered":"2024-03-06T07:28:41.867281+0000","last_clean":"2024-03-06T07:28:41.867281+0000","last_became_active":"2024-03-06T07:28:39.835174+0000","last_became_peered":"2024-03-06T07:28:39.835174+0000","last_unstale":"2024-03-06T07:28:41.867281+0000","last_undegraded":"2024-03-06T07:28:41.867281+0000","last_fullsized":"2024-03-06T07:28:41.867281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:00.068555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.870324+0000","last_change":"2024-03-06T07:28:40.866818+0000","last_active":"2024-03-06T07:28:40.870324+0000","last_peered":"2024-03-06T07:28:40.870324+0000","last_clean":"2024-03-06T07:28:40.870324+0000","last_became_active":"2024-03-06T07:28:40.866526+0000","last_became_peered":"2024-03-06T07:28:40.866526+0000","last_unstale":"2024-03-06T07:28:40.870324+0000","last_undegraded":"2024-03-06T07:28:40.870324+0000","last_fullsized":"2024-03-06T07:28:40.870324+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:34.394133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882604+0000","last_change":"2024-03-06T07:28:39.834560+0000","last_active":"2024-03-06T07:28:40.882604+0000","last_peered":"2024-03-06T07:28:40.882604+0000","last_clean":"2024-03-06T07:28:40.882604+0000","last_became_active":"2024-03-06T07:28:39.834193+0000","last_became_peered":"2024-03-06T07:28:39.834193+0000","last_unstale":"2024-03-06T07:28:40.882604+0000","last_undegraded":"2024-03-06T07:28:40.882604+0000","last_fullsized":"2024-03-06T07:28:40.882604+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:22.337525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.865513+0000","last_change":"2024-03-06T07:28:40.873964+0000","last_active":"2024-03-06T07:28:41.865513+0000","last_peered":"2024-03-06T07:28:41.865513+0000","last_clean":"2024-03-06T07:28:41.865513+0000","last_became_active":"2024-03-06T07:28:40.873682+0000","last_became_peered":"2024-03-06T07:28:40.873682+0000","last_unstale":"2024-03-06T07:28:41.865513+0000","last_undegraded":"2024-03-06T07:28:41.865513+0000","last_fullsized":"2024-03-06T07:28:41.865513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:33.591548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882857+0000","last_change":"2024-03-06T07:28:39.836333+0000","last_active":"2024-03-06T07:28:40.882857+0000","last_peered":"2024-03-06T07:28:40.882857+0000","last_clean":"2024-03-06T07:28:40.882857+0000","last_became_active":"2024-03-06T07:28:39.836040+0000","last_became_peered":"2024-03-06T07:28:39.836040+0000","last_unstale":"2024-03-06T07:28:40.882857+0000","last_undegraded":"2024-03-06T07:28:40.882857+0000","last_fullsized":"2024-03-06T07:28:40.882857+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:22.325631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056463+0000","last_change":"2024-03-06T07:28:40.868335+0000","last_active":"2024-03-06T07:28:42.056463+0000","last_peered":"2024-03-06T07:28:42.056463+0000","last_clean":"2024-03-06T07:28:42.056463+0000","last_became_active":"2024-03-06T07:28:40.865156+0000","last_became_peered":"2024-03-06T07:28:40.865156+0000","last_unstale":"2024-03-06T07:28:42.056463+0000","last_undegraded":"2024-03-06T07:28:42.056463+0000","last_fullsized":"2024-03-06T07:28:42.056463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:55.894729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.849230+0000","last_change":"2024-03-06T07:28:39.834169+0000","last_active":"2024-03-06T07:28:40.849230+0000","last_peered":"2024-03-06T07:28:40.849230+0000","last_clean":"2024-03-06T07:28:40.849230+0000","last_became_active":"2024-03-06T07:28:39.833753+0000","last_became_peered":"2024-03-06T07:28:39.833753+0000","last_unstale":"2024-03-06T07:28:40.849230+0000","last_undegraded":"2024-03-06T07:28:40.849230+0000","last_fullsized":"2024-03-06T07:28:40.849230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:21.576285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056339+0000","last_change":"2024-03-06T07:28:40.867535+0000","last_active":"2024-03-06T07:28:42.056339+0000","last_peered":"2024-03-06T07:28:42.056339+0000","last_clean":"2024-03-06T07:28:42.056339+0000","last_became_active":"2024-03-06T07:28:40.859951+0000","last_became_peered":"2024-03-06T07:28:40.859951+0000","last_unstale":"2024-03-06T07:28:42.056339+0000","last_undegraded":"2024-03-06T07:28:42.056339+0000","last_fullsized":"2024-03-06T07:28:42.056339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:23.330534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881513+0000","last_change":"2024-03-06T07:28:39.823828+0000","last_active":"2024-03-06T07:28:40.881513+0000","last_peered":"2024-03-06T07:28:40.881513+0000","last_clean":"2024-03-06T07:28:40.881513+0000","last_became_active":"2024-03-06T07:28:39.823039+0000","last_became_peered":"2024-03-06T07:28:39.823039+0000","last_unstale":"2024-03-06T07:28:40.881513+0000","last_undegraded":"2024-03-06T07:28:40.881513+0000","last_fullsized":"2024-03-06T07:28:40.881513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:55.978361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056418+0000","last_change":"2024-03-06T07:28:40.861016+0000","last_active":"2024-03-06T07:28:42.056418+0000","last_peered":"2024-03-06T07:28:42.056418+0000","last_clean":"2024-03-06T07:28:42.056418+0000","last_became_active":"2024-03-06T07:28:40.857378+0000","last_became_peered":"2024-03-06T07:28:40.857378+0000","last_unstale":"2024-03-06T07:28:42.056418+0000","last_undegraded":"2024-03-06T07:28:42.056418+0000","last_fullsized":"2024-03-06T07:28:42.056418+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:11.454733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.850677+0000","last_change":"2024-03-06T07:28:39.821996+0000","last_active":"2024-03-06T07:28:40.850677+0000","last_peered":"2024-03-06T07:28:40.850677+0000","last_clean":"2024-03-06T07:28:40.850677+0000","last_became_active":"2024-03-06T07:28:39.820771+0000","last_became_peered":"2024-03-06T07:28:39.820771+0000","last_unstale":"2024-03-06T07:28:40.850677+0000","last_undegraded":"2024-03-06T07:28:40.850677+0000","last_fullsized":"2024-03-06T07:28:40.850677+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:43.981321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056093+0000","last_change":"2024-03-06T07:28:40.867025+0000","last_active":"2024-03-06T07:28:42.056093+0000","last_peered":"2024-03-06T07:28:42.056093+0000","last_clean":"2024-03-06T07:28:42.056093+0000","last_became_active":"2024-03-06T07:28:40.855104+0000","last_became_peered":"2024-03-06T07:28:40.855104+0000","last_unstale":"2024-03-06T07:28:42.056093+0000","last_undegraded":"2024-03-06T07:28:42.056093+0000","last_fullsized":"2024-03-06T07:28:42.056093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:10.301372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.847544+0000","last_change":"2024-03-06T07:28:39.837722+0000","last_active":"2024-03-06T07:28:40.847544+0000","last_peered":"2024-03-06T07:28:40.847544+0000","last_clean":"2024-03-06T07:28:40.847544+0000","last_became_active":"2024-03-06T07:28:39.837079+0000","last_became_peered":"2024-03-06T07:28:39.837079+0000","last_unstale":"2024-03-06T07:28:40.847544+0000","last_undegraded":"2024-03-06T07:28:40.847544+0000","last_fullsized":"2024-03-06T07:28:40.847544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:42.568653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.866152+0000","last_change":"2024-03-06T07:28:40.860010+0000","last_active":"2024-03-06T07:28:40.866152+0000","last_peered":"2024-03-06T07:28:40.866152+0000","last_clean":"2024-03-06T07:28:40.866152+0000","last_became_active":"2024-03-06T07:28:40.859737+0000","last_became_peered":"2024-03-06T07:28:40.859737+0000","last_unstale":"2024-03-06T07:28:40.866152+0000","last_undegraded":"2024-03-06T07:28:40.866152+0000","last_fullsized":"2024-03-06T07:28:40.866152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:01.073459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.840047+0000","last_change":"2024-03-06T07:28:39.834432+0000","last_active":"2024-03-06T07:28:40.840047+0000","last_peered":"2024-03-06T07:28:40.840047+0000","last_clean":"2024-03-06T07:28:40.840047+0000","last_became_active":"2024-03-06T07:28:39.834132+0000","last_became_peered":"2024-03-06T07:28:39.834132+0000","last_unstale":"2024-03-06T07:28:40.840047+0000","last_undegraded":"2024-03-06T07:28:40.840047+0000","last_fullsized":"2024-03-06T07:28:40.840047+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:31.040038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056393+0000","last_change":"2024-03-06T07:28:40.868822+0000","last_active":"2024-03-06T07:28:42.056393+0000","last_peered":"2024-03-06T07:28:42.056393+0000","last_clean":"2024-03-06T07:28:42.056393+0000","last_became_active":"2024-03-06T07:28:40.868585+0000","last_became_peered":"2024-03-06T07:28:40.868585+0000","last_unstale":"2024-03-06T07:28:42.056393+0000","last_undegraded":"2024-03-06T07:28:42.056393+0000","last_fullsized":"2024-03-06T07:28:42.056393+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:23.342428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867434+0000","last_change":"2024-03-06T07:28:39.826331+0000","last_active":"2024-03-06T07:28:41.867434+0000","last_peered":"2024-03-06T07:28:41.867434+0000","last_clean":"2024-03-06T07:28:41.867434+0000","last_became_active":"2024-03-06T07:28:39.825988+0000","last_became_peered":"2024-03-06T07:28:39.825988+0000","last_unstale":"2024-03-06T07:28:41.867434+0000","last_undegraded":"2024-03-06T07:28:41.867434+0000","last_fullsized":"2024-03-06T07:28:41.867434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:55.849853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.880074+0000","last_change":"2024-03-06T07:28:40.877256+0000","last_active":"2024-03-06T07:28:40.880074+0000","last_peered":"2024-03-06T07:28:40.880074+0000","last_clean":"2024-03-06T07:28:40.880074+0000","last_became_active":"2024-03-06T07:28:40.875647+0000","last_became_peered":"2024-03-06T07:28:40.875647+0000","last_unstale":"2024-03-06T07:28:40.880074+0000","last_undegraded":"2024-03-06T07:28:40.880074+0000","last_fullsized":"2024-03-06T07:28:40.880074+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:50.125370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.831333+0000","last_change":"2024-03-06T07:28:39.822567+0000","last_active":"2024-03-06T07:28:40.831333+0000","last_peered":"2024-03-06T07:28:40.831333+0000","last_clean":"2024-03-06T07:28:40.831333+0000","last_became_active":"2024-03-06T07:28:39.822273+0000","last_became_peered":"2024-03-06T07:28:39.822273+0000","last_unstale":"2024-03-06T07:28:40.831333+0000","last_undegraded":"2024-03-06T07:28:40.831333+0000","last_fullsized":"2024-03-06T07:28:40.831333+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:23.256357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881924+0000","last_change":"2024-03-06T07:28:40.877693+0000","last_active":"2024-03-06T07:28:40.881924+0000","last_peered":"2024-03-06T07:28:40.881924+0000","last_clean":"2024-03-06T07:28:40.881924+0000","last_became_active":"2024-03-06T07:28:40.875906+0000","last_became_peered":"2024-03-06T07:28:40.875906+0000","last_unstale":"2024-03-06T07:28:40.881924+0000","last_undegraded":"2024-03-06T07:28:40.881924+0000","last_fullsized":"2024-03-06T07:28:40.881924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:59.287242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.865654+0000","last_change":"2024-03-06T07:28:39.825495+0000","last_active":"2024-03-06T07:28:41.865654+0000","last_peered":"2024-03-06T07:28:41.865654+0000","last_clean":"2024-03-06T07:28:41.865654+0000","last_became_active":"2024-03-06T07:28:39.825214+0000","last_became_peered":"2024-03-06T07:28:39.825214+0000","last_unstale":"2024-03-06T07:28:41.865654+0000","last_undegraded":"2024-03-06T07:28:41.865654+0000","last_fullsized":"2024-03-06T07:28:41.865654+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:54.889825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.875487+0000","last_change":"2024-03-06T07:28:40.872905+0000","last_active":"2024-03-06T07:28:40.875487+0000","last_peered":"2024-03-06T07:28:40.875487+0000","last_clean":"2024-03-06T07:28:40.875487+0000","last_became_active":"2024-03-06T07:28:40.872581+0000","last_became_peered":"2024-03-06T07:28:40.872581+0000","last_unstale":"2024-03-06T07:28:40.875487+0000","last_undegraded":"2024-03-06T07:28:40.875487+0000","last_fullsized":"2024-03-06T07:28:40.875487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:54.840821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.830185+0000","last_change":"2024-03-06T07:28:39.823880+0000","last_active":"2024-03-06T07:28:40.830185+0000","last_peered":"2024-03-06T07:28:40.830185+0000","last_clean":"2024-03-06T07:28:40.830185+0000","last_became_active":"2024-03-06T07:28:39.823084+0000","last_became_peered":"2024-03-06T07:28:39.823084+0000","last_unstale":"2024-03-06T07:28:40.830185+0000","last_undegraded":"2024-03-06T07:28:40.830185+0000","last_fullsized":"2024-03-06T07:28:40.830185+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:27.459488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.870408+0000","last_change":"2024-03-06T07:28:40.862951+0000","last_active":"2024-03-06T07:28:40.870408+0000","last_peered":"2024-03-06T07:28:40.870408+0000","last_clean":"2024-03-06T07:28:40.870408+0000","last_became_active":"2024-03-06T07:28:40.861646+0000","last_became_peered":"2024-03-06T07:28:40.861646+0000","last_unstale":"2024-03-06T07:28:40.870408+0000","last_undegraded":"2024-03-06T07:28:40.870408+0000","last_fullsized":"2024-03-06T07:28:40.870408+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:09.193514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864923+0000","last_change":"2024-03-06T07:28:39.826645+0000","last_active":"2024-03-06T07:28:41.864923+0000","last_peered":"2024-03-06T07:28:41.864923+0000","last_clean":"2024-03-06T07:28:41.864923+0000","last_became_active":"2024-03-06T07:28:39.826365+0000","last_became_peered":"2024-03-06T07:28:39.826365+0000","last_unstale":"2024-03-06T07:28:41.864923+0000","last_undegraded":"2024-03-06T07:28:41.864923+0000","last_fullsized":"2024-03-06T07:28:41.864923+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:10.449829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869093+0000","last_change":"2024-03-06T07:28:40.858520+0000","last_active":"2024-03-06T07:28:40.869093+0000","last_peered":"2024-03-06T07:28:40.869093+0000","last_clean":"2024-03-06T07:28:40.869093+0000","last_became_active":"2024-03-06T07:28:40.858181+0000","last_became_peered":"2024-03-06T07:28:40.858181+0000","last_unstale":"2024-03-06T07:28:40.869093+0000","last_undegraded":"2024-03-06T07:28:40.869093+0000","last_fullsized":"2024-03-06T07:28:40.869093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:36.401757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.847767+0000","last_change":"2024-03-06T07:28:39.835477+0000","last_active":"2024-03-06T07:28:40.847767+0000","last_peered":"2024-03-06T07:28:40.847767+0000","last_clean":"2024-03-06T07:28:40.847767+0000","last_became_active":"2024-03-06T07:28:39.835166+0000","last_became_peered":"2024-03-06T07:28:39.835166+0000","last_unstale":"2024-03-06T07:28:40.847767+0000","last_undegraded":"2024-03-06T07:28:40.847767+0000","last_fullsized":"2024-03-06T07:28:40.847767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:00.877196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869789+0000","last_change":"2024-03-06T07:28:40.867485+0000","last_active":"2024-03-06T07:28:40.869789+0000","last_peered":"2024-03-06T07:28:40.869789+0000","last_clean":"2024-03-06T07:28:40.869789+0000","last_became_active":"2024-03-06T07:28:40.866013+0000","last_became_peered":"2024-03-06T07:28:40.866013+0000","last_unstale":"2024-03-06T07:28:40.869789+0000","last_undegraded":"2024-03-06T07:28:40.869789+0000","last_fullsized":"2024-03-06T07:28:40.869789+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:35.590273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.841509+0000","last_change":"2024-03-06T07:28:39.831389+0000","last_active":"2024-03-06T07:28:40.841509+0000","last_peered":"2024-03-06T07:28:40.841509+0000","last_clean":"2024-03-06T07:28:40.841509+0000","last_became_active":"2024-03-06T07:28:39.831100+0000","last_became_peered":"2024-03-06T07:28:39.831100+0000","last_unstale":"2024-03-06T07:28:40.841509+0000","last_undegraded":"2024-03-06T07:28:40.841509+0000","last_fullsized":"2024-03-06T07:28:40.841509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:21.050872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.875406+0000","last_change":"2024-03-06T07:28:40.873299+0000","last_active":"2024-03-06T07:28:40.875406+0000","last_peered":"2024-03-06T07:28:40.875406+0000","last_clean":"2024-03-06T07:28:40.875406+0000","last_became_active":"2024-03-06T07:28:40.872979+0000","last_became_peered":"2024-03-06T07:28:40.872979+0000","last_unstale":"2024-03-06T07:28:40.875406+0000","last_undegraded":"2024-03-06T07:28:40.875406+0000","last_fullsized":"2024-03-06T07:28:40.875406+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:25.580334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.853355+0000","last_change":"2024-03-06T07:28:39.832618+0000","last_active":"2024-03-06T07:28:40.853355+0000","last_peered":"2024-03-06T07:28:40.853355+0000","last_clean":"2024-03-06T07:28:40.853355+0000","last_became_active":"2024-03-06T07:28:39.832334+0000","last_became_peered":"2024-03-06T07:28:39.832334+0000","last_unstale":"2024-03-06T07:28:40.853355+0000","last_undegraded":"2024-03-06T07:28:40.853355+0000","last_fullsized":"2024-03-06T07:28:40.853355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:10.797789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":191,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.850363+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:28:40.850363+0000","last_peered":"2024-03-06T07:28:40.850363+0000","last_clean":"2024-03-06T07:28:40.850363+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:28:40.850363+0000","last_undegraded":"2024-03-06T07:28:40.850363+0000","last_fullsized":"2024-03-06T07:28:40.850363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509459,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":155442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":731101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":364544,"data_stored":140733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":385024,"data_stored":148180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":385024,"data_stored":147612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":149142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":970752,"data_stored":737876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.718+0000 7f2df1000640 1 Processor -- start 2024-03-06T07:28:45.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.718+0000 7f2df1000640 1 -- start start 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2df1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2dec145300 0x7f2dec0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2df1000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec070f10 0x7f2dec0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2df1000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dec0714d0 con 0x7f2dec0cce90 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2df1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0cd300 con 0x7f2dec145300 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2df1000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0717b0 con 0x7f2dec070f10 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2deb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2dec145300 0x7f2dec0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2deb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2dec145300 0x7f2dec0b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42822/0 (socket says 172.21.15.6:42822) 2024-03-06T07:28:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2deb400640 1 -- 172.21.15.6:0/4031701179 learned_addr learned my addr 172.21.15.6:0/4031701179 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.720+0000 7f2deaa00640 1 --2- 172.21.15.6:0/4031701179 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec070f10 0x7f2dec0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.721+0000 7f2deb400640 1 -- 172.21.15.6:0/4031701179 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec070f10 msgr2=0x7f2dec0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.721+0000 7f2deb400640 1 --2- 172.21.15.6:0/4031701179 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec070f10 0x7f2dec0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.721+0000 7f2deb400640 1 -- 172.21.15.6:0/4031701179 >> v1:172.21.15.178:6789/0 conn(0x7f2dec0cce90 legacy=0x7f2dec0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.721+0000 7f2deb400640 1 -- 172.21.15.6:0/4031701179 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec071a90 con 0x7f2dec145300 2024-03-06T07:28:45.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.722+0000 7f2deb400640 1 --2- 172.21.15.6:0/4031701179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec0b49a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009870 tx=0x7f2ddc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7085dbbc06247402 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.722+0000 7f2dea000640 1 -- 172.21.15.6:0/4031701179 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc034070 con 0x7f2dec145300 2024-03-06T07:28:45.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.722+0000 7f2dea000640 1 -- 172.21.15.6:0/4031701179 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc030840 con 0x7f2dec145300 2024-03-06T07:28:45.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.722+0000 7f2dea000640 1 -- 172.21.15.6:0/4031701179 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc030c10 con 0x7f2dec145300 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 -- 172.21.15.6:0/4031701179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 msgr2=0x7f2dec0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 --2- 172.21.15.6:0/4031701179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec0b49a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009870 tx=0x7f2ddc02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 -- 172.21.15.6:0/4031701179 shutdown_connections 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 --2- 172.21.15.6:0/4031701179 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec070f10 0x7f2dec0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 --2- 172.21.15.6:0/4031701179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec0b49a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 -- 172.21.15.6:0/4031701179 >> 172.21.15.6:0/4031701179 conn(0x7f2dec06b3d0 msgr2=0x7f2dec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 -- 172.21.15.6:0/4031701179 shutdown_connections 2024-03-06T07:28:45.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.723+0000 7f2df1000640 1 -- 172.21.15.6:0/4031701179 wait complete. 2024-03-06T07:28:45.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.724+0000 7f2df1000640 1 Processor -- start 2024-03-06T07:28:45.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.725+0000 7f2df1000640 1 -- start start 2024-03-06T07:28:45.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.725+0000 7f2df1000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2dec070f10 0x7f2dec15b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2df1000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec0cce90 0x7f2dec1e7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2df1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec1e7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2df1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec1456e0 con 0x7f2dec145300 2024-03-06T07:28:45.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2df1000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0b4ee0 con 0x7f2dec070f10 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2df1000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec15d580 con 0x7f2dec0cce90 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec1e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec1e7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42834/0 (socket says 172.21.15.6:42834) 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 -- 172.21.15.6:0/1758270274 learned_addr learned my addr 172.21.15.6:0/1758270274 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2deaa00640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec0cce90 0x7f2dec1e7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2deb400640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2dec070f10 0x7f2dec15b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 -- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec0cce90 msgr2=0x7f2dec1e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec0cce90 0x7f2dec1e7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 -- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2dec070f10 msgr2=0x7f2dec15b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2dec070f10 0x7f2dec15b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.726+0000 7f2debe00640 1 -- 172.21.15.6:0/1758270274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec1e4960 con 0x7f2dec145300 2024-03-06T07:28:45.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.727+0000 7f2debe00640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec1e7780 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2de400c7a0 tx=0x7f2de400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.727+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de4003b70 con 0x7f2dec145300 2024-03-06T07:28:45.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.727+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de4003d30 con 0x7f2dec145300 2024-03-06T07:28:45.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.727+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de400aa90 con 0x7f2dec145300 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.727+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec1e4bd0 con 0x7f2dec145300 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.727+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec1e5470 con 0x7f2dec145300 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.728+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dec1e4fc0 con 0x7f2dec145300 2024-03-06T07:28:45.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.729+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2de400ac30 con 0x7f2dec145300 2024-03-06T07:28:45.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.731+0000 7f2de3400640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2dcc06b4d0 0x7f2dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.731+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2de408a150 con 0x7f2dec145300 2024-03-06T07:28:45.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.731+0000 7f2deb400640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2dcc06b4d0 0x7f2dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.732+0000 7f2deb400640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2dcc06b4d0 0x7f2dcc06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc036000 tx=0x7f2ddc03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.736+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2de4058fd0 con 0x7f2dec145300 2024-03-06T07:28:45.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.932+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2dec1e4da0 con 0x7f2dcc06b4d0 2024-03-06T07:28:45.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.952+0000 7f2de3400640 1 -- 172.21.15.6:0/1758270274 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251897 (secure 0 0 0) 0x7f2dec1e4da0 con 0x7f2dcc06b4d0 2024-03-06T07:28:45.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:45.958 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2dcc06b4d0 msgr2=0x7f2dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2dcc06b4d0 0x7f2dcc06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc036000 tx=0x7f2ddc03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 msgr2=0x7f2dec1e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec1e7780 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2de400c7a0 tx=0x7f2de400cc60 comp rx=0 tx=0).stop 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 shutdown_connections 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2dcc06b4d0 0x7f2dcc06d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2dec145300 0x7f2dec1e7780 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2dec0cce90 0x7f2dec1e7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 --2- 172.21.15.6:0/1758270274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2dec070f10 0x7f2dec15b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 >> 172.21.15.6:0/1758270274 conn(0x7f2dec06b3d0 msgr2=0x7f2dec0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 shutdown_connections 2024-03-06T07:28:45.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:45.957+0000 7f2df1000640 1 -- 172.21.15.6:0/1758270274 wait complete. 2024-03-06T07:28:45.979 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-03-06T07:28:45.463602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236976,"kb_used_data":4144,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737744,"statfs":{"total":773094113280,"available":772851449856,"internally_reserved":0,"allocated":4243456,"data_stored":2357664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001952"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871675+0000","last_change":"2024-03-06T07:28:40.869031+0000","last_active":"2024-03-06T07:28:40.871675+0000","last_peered":"2024-03-06T07:28:40.871675+0000","last_clean":"2024-03-06T07:28:40.871675+0000","last_became_active":"2024-03-06T07:28:40.868760+0000","last_became_peered":"2024-03-06T07:28:40.868760+0000","last_unstale":"2024-03-06T07:28:40.871675+0000","last_undegraded":"2024-03-06T07:28:40.871675+0000","last_fullsized":"2024-03-06T07:28:40.871675+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:47.710001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.857551+0000","last_change":"2024-03-06T07:28:40.848398+0000","last_active":"2024-03-06T07:28:40.857551+0000","last_peered":"2024-03-06T07:28:40.857551+0000","last_clean":"2024-03-06T07:28:40.857551+0000","last_became_active":"2024-03-06T07:28:40.847621+0000","last_became_peered":"2024-03-06T07:28:40.847621+0000","last_unstale":"2024-03-06T07:28:40.857551+0000","last_undegraded":"2024-03-06T07:28:40.857551+0000","last_fullsized":"2024-03-06T07:28:40.857551+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.565225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881867+0000","last_change":"2024-03-06T07:28:40.879289+0000","last_active":"2024-03-06T07:28:40.881867+0000","last_peered":"2024-03-06T07:28:40.881867+0000","last_clean":"2024-03-06T07:28:40.881867+0000","last_became_active":"2024-03-06T07:28:40.879050+0000","last_became_peered":"2024-03-06T07:28:40.879050+0000","last_unstale":"2024-03-06T07:28:40.881867+0000","last_undegraded":"2024-03-06T07:28:40.881867+0000","last_fullsized":"2024-03-06T07:28:40.881867+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:36.401757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.865307+0000","last_change":"2024-03-06T07:28:40.871486+0000","last_active":"2024-03-06T07:28:41.865307+0000","last_peered":"2024-03-06T07:28:41.865307+0000","last_clean":"2024-03-06T07:28:41.865307+0000","last_became_active":"2024-03-06T07:28:40.865725+0000","last_became_peered":"2024-03-06T07:28:40.865725+0000","last_unstale":"2024-03-06T07:28:41.865307+0000","last_undegraded":"2024-03-06T07:28:41.865307+0000","last_fullsized":"2024-03-06T07:28:41.865307+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:57:05.689350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056067+0000","last_change":"2024-03-06T07:28:40.861325+0000","last_active":"2024-03-06T07:28:42.056067+0000","last_peered":"2024-03-06T07:28:42.056067+0000","last_clean":"2024-03-06T07:28:42.056067+0000","last_became_active":"2024-03-06T07:28:40.860847+0000","last_became_peered":"2024-03-06T07:28:40.860847+0000","last_unstale":"2024-03-06T07:28:42.056067+0000","last_undegraded":"2024-03-06T07:28:42.056067+0000","last_fullsized":"2024-03-06T07:28:42.056067+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:59.058344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.870081+0000","last_change":"2024-03-06T07:28:40.867112+0000","last_active":"2024-03-06T07:28:40.870081+0000","last_peered":"2024-03-06T07:28:40.870081+0000","last_clean":"2024-03-06T07:28:40.870081+0000","last_became_active":"2024-03-06T07:28:40.861082+0000","last_became_peered":"2024-03-06T07:28:40.861082+0000","last_unstale":"2024-03-06T07:28:40.870081+0000","last_undegraded":"2024-03-06T07:28:40.870081+0000","last_fullsized":"2024-03-06T07:28:40.870081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:31:57.724310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873051+0000","last_change":"2024-03-06T07:28:40.869166+0000","last_active":"2024-03-06T07:28:40.873051+0000","last_peered":"2024-03-06T07:28:40.873051+0000","last_clean":"2024-03-06T07:28:40.873051+0000","last_became_active":"2024-03-06T07:28:40.868642+0000","last_became_peered":"2024-03-06T07:28:40.868642+0000","last_unstale":"2024-03-06T07:28:40.873051+0000","last_undegraded":"2024-03-06T07:28:40.873051+0000","last_fullsized":"2024-03-06T07:28:40.873051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:48.955548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867843+0000","last_change":"2024-03-06T07:28:40.861508+0000","last_active":"2024-03-06T07:28:40.867843+0000","last_peered":"2024-03-06T07:28:40.867843+0000","last_clean":"2024-03-06T07:28:40.867843+0000","last_became_active":"2024-03-06T07:28:40.861226+0000","last_became_peered":"2024-03-06T07:28:40.861226+0000","last_unstale":"2024-03-06T07:28:40.867843+0000","last_undegraded":"2024-03-06T07:28:40.867843+0000","last_fullsized":"2024-03-06T07:28:40.867843+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:56.854757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.877919+0000","last_change":"2024-03-06T07:28:40.876335+0000","last_active":"2024-03-06T07:28:40.877919+0000","last_peered":"2024-03-06T07:28:40.877919+0000","last_clean":"2024-03-06T07:28:40.877919+0000","last_became_active":"2024-03-06T07:28:40.875996+0000","last_became_peered":"2024-03-06T07:28:40.875996+0000","last_unstale":"2024-03-06T07:28:40.877919+0000","last_undegraded":"2024-03-06T07:28:40.877919+0000","last_fullsized":"2024-03-06T07:28:40.877919+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.411787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867283+0000","last_change":"2024-03-06T07:28:40.876626+0000","last_active":"2024-03-06T07:28:41.867283+0000","last_peered":"2024-03-06T07:28:41.867283+0000","last_clean":"2024-03-06T07:28:41.867283+0000","last_became_active":"2024-03-06T07:28:40.876362+0000","last_became_peered":"2024-03-06T07:28:40.876362+0000","last_unstale":"2024-03-06T07:28:41.867283+0000","last_undegraded":"2024-03-06T07:28:41.867283+0000","last_fullsized":"2024-03-06T07:28:41.867283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:19.694349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881493+0000","last_change":"2024-03-06T07:28:40.877223+0000","last_active":"2024-03-06T07:28:40.881493+0000","last_peered":"2024-03-06T07:28:40.881493+0000","last_clean":"2024-03-06T07:28:40.881493+0000","last_became_active":"2024-03-06T07:28:40.876932+0000","last_became_peered":"2024-03-06T07:28:40.876932+0000","last_unstale":"2024-03-06T07:28:40.881493+0000","last_undegraded":"2024-03-06T07:28:40.881493+0000","last_fullsized":"2024-03-06T07:28:40.881493+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:18.515752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867544+0000","last_change":"2024-03-06T07:28:40.864107+0000","last_active":"2024-03-06T07:28:40.867544+0000","last_peered":"2024-03-06T07:28:40.867544+0000","last_clean":"2024-03-06T07:28:40.867544+0000","last_became_active":"2024-03-06T07:28:40.861454+0000","last_became_peered":"2024-03-06T07:28:40.861454+0000","last_unstale":"2024-03-06T07:28:40.867544+0000","last_undegraded":"2024-03-06T07:28:40.867544+0000","last_fullsized":"2024-03-06T07:28:40.867544+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.565225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056262+0000","last_change":"2024-03-06T07:28:40.861149+0000","last_active":"2024-03-06T07:28:42.056262+0000","last_peered":"2024-03-06T07:28:42.056262+0000","last_clean":"2024-03-06T07:28:42.056262+0000","last_became_active":"2024-03-06T07:28:40.860453+0000","last_became_peered":"2024-03-06T07:28:40.860453+0000","last_unstale":"2024-03-06T07:28:42.056262+0000","last_undegraded":"2024-03-06T07:28:42.056262+0000","last_fullsized":"2024-03-06T07:28:42.056262+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:22.581189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869064+0000","last_change":"2024-03-06T07:28:40.858071+0000","last_active":"2024-03-06T07:28:40.869064+0000","last_peered":"2024-03-06T07:28:40.869064+0000","last_clean":"2024-03-06T07:28:40.869064+0000","last_became_active":"2024-03-06T07:28:40.853230+0000","last_became_peered":"2024-03-06T07:28:40.853230+0000","last_unstale":"2024-03-06T07:28:40.869064+0000","last_undegraded":"2024-03-06T07:28:40.869064+0000","last_fullsized":"2024-03-06T07:28:40.869064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:15.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.865831+0000","last_change":"2024-03-06T07:28:40.860185+0000","last_active":"2024-03-06T07:28:40.865831+0000","last_peered":"2024-03-06T07:28:40.865831+0000","last_clean":"2024-03-06T07:28:40.865831+0000","last_became_active":"2024-03-06T07:28:40.859859+0000","last_became_peered":"2024-03-06T07:28:40.859859+0000","last_unstale":"2024-03-06T07:28:40.865831+0000","last_undegraded":"2024-03-06T07:28:40.865831+0000","last_fullsized":"2024-03-06T07:28:40.865831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:57.408991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873563+0000","last_change":"2024-03-06T07:28:40.868249+0000","last_active":"2024-03-06T07:28:40.873563+0000","last_peered":"2024-03-06T07:28:40.873563+0000","last_clean":"2024-03-06T07:28:40.873563+0000","last_became_active":"2024-03-06T07:28:40.868010+0000","last_became_peered":"2024-03-06T07:28:40.868010+0000","last_unstale":"2024-03-06T07:28:40.873563+0000","last_undegraded":"2024-03-06T07:28:40.873563+0000","last_fullsized":"2024-03-06T07:28:40.873563+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:08.961770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871283+0000","last_change":"2024-03-06T07:28:40.863703+0000","last_active":"2024-03-06T07:28:40.871283+0000","last_peered":"2024-03-06T07:28:40.871283+0000","last_clean":"2024-03-06T07:28:40.871283+0000","last_became_active":"2024-03-06T07:28:40.863403+0000","last_became_peered":"2024-03-06T07:28:40.863403+0000","last_unstale":"2024-03-06T07:28:40.871283+0000","last_undegraded":"2024-03-06T07:28:40.871283+0000","last_fullsized":"2024-03-06T07:28:40.871283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:24.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882440+0000","last_change":"2024-03-06T07:28:40.877591+0000","last_active":"2024-03-06T07:28:40.882440+0000","last_peered":"2024-03-06T07:28:40.882440+0000","last_clean":"2024-03-06T07:28:40.882440+0000","last_became_active":"2024-03-06T07:28:40.876954+0000","last_became_peered":"2024-03-06T07:28:40.876954+0000","last_unstale":"2024-03-06T07:28:40.882440+0000","last_undegraded":"2024-03-06T07:28:40.882440+0000","last_fullsized":"2024-03-06T07:28:40.882440+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:14.014615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869158+0000","last_change":"2024-03-06T07:28:40.866649+0000","last_active":"2024-03-06T07:28:40.869158+0000","last_peered":"2024-03-06T07:28:40.869158+0000","last_clean":"2024-03-06T07:28:40.869158+0000","last_became_active":"2024-03-06T07:28:40.866348+0000","last_became_peered":"2024-03-06T07:28:40.866348+0000","last_unstale":"2024-03-06T07:28:40.869158+0000","last_undegraded":"2024-03-06T07:28:40.869158+0000","last_fullsized":"2024-03-06T07:28:40.869158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:13.090146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056192+0000","last_change":"2024-03-06T07:28:40.868159+0000","last_active":"2024-03-06T07:28:42.056192+0000","last_peered":"2024-03-06T07:28:42.056192+0000","last_clean":"2024-03-06T07:28:42.056192+0000","last_became_active":"2024-03-06T07:28:40.864562+0000","last_became_peered":"2024-03-06T07:28:40.864562+0000","last_unstale":"2024-03-06T07:28:42.056192+0000","last_undegraded":"2024-03-06T07:28:42.056192+0000","last_fullsized":"2024-03-06T07:28:42.056192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:35.204216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869739+0000","last_change":"2024-03-06T07:28:40.856912+0000","last_active":"2024-03-06T07:28:40.869739+0000","last_peered":"2024-03-06T07:28:40.869739+0000","last_clean":"2024-03-06T07:28:40.869739+0000","last_became_active":"2024-03-06T07:28:40.856733+0000","last_became_peered":"2024-03-06T07:28:40.856733+0000","last_unstale":"2024-03-06T07:28:40.869739+0000","last_undegraded":"2024-03-06T07:28:40.869739+0000","last_fullsized":"2024-03-06T07:28:40.869739+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:11.454733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.877381+0000","last_change":"2024-03-06T07:28:40.872043+0000","last_active":"2024-03-06T07:28:40.877381+0000","last_peered":"2024-03-06T07:28:40.877381+0000","last_clean":"2024-03-06T07:28:40.877381+0000","last_became_active":"2024-03-06T07:28:40.871701+0000","last_became_peered":"2024-03-06T07:28:40.871701+0000","last_unstale":"2024-03-06T07:28:40.877381+0000","last_undegraded":"2024-03-06T07:28:40.877381+0000","last_fullsized":"2024-03-06T07:28:40.877381+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:06.193722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.863350+0000","last_change":"2024-03-06T07:28:40.856856+0000","last_active":"2024-03-06T07:28:40.863350+0000","last_peered":"2024-03-06T07:28:40.863350+0000","last_clean":"2024-03-06T07:28:40.863350+0000","last_became_active":"2024-03-06T07:28:40.856526+0000","last_became_peered":"2024-03-06T07:28:40.856526+0000","last_unstale":"2024-03-06T07:28:40.863350+0000","last_undegraded":"2024-03-06T07:28:40.863350+0000","last_fullsized":"2024-03-06T07:28:40.863350+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:54.840821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.864076+0000","last_change":"2024-03-06T07:28:40.856769+0000","last_active":"2024-03-06T07:28:40.864076+0000","last_peered":"2024-03-06T07:28:40.864076+0000","last_clean":"2024-03-06T07:28:40.864076+0000","last_became_active":"2024-03-06T07:28:40.856430+0000","last_became_peered":"2024-03-06T07:28:40.856430+0000","last_unstale":"2024-03-06T07:28:40.864076+0000","last_undegraded":"2024-03-06T07:28:40.864076+0000","last_fullsized":"2024-03-06T07:28:40.864076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:17.684202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867052+0000","last_change":"2024-03-06T07:28:40.871472+0000","last_active":"2024-03-06T07:28:41.867052+0000","last_peered":"2024-03-06T07:28:41.867052+0000","last_clean":"2024-03-06T07:28:41.867052+0000","last_became_active":"2024-03-06T07:28:40.864661+0000","last_became_peered":"2024-03-06T07:28:40.864661+0000","last_unstale":"2024-03-06T07:28:41.867052+0000","last_undegraded":"2024-03-06T07:28:41.867052+0000","last_fullsized":"2024-03-06T07:28:41.867052+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:42.461309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869128+0000","last_change":"2024-03-06T07:28:40.864124+0000","last_active":"2024-03-06T07:28:40.869128+0000","last_peered":"2024-03-06T07:28:40.869128+0000","last_clean":"2024-03-06T07:28:40.869128+0000","last_became_active":"2024-03-06T07:28:40.862825+0000","last_became_peered":"2024-03-06T07:28:40.862825+0000","last_unstale":"2024-03-06T07:28:40.869128+0000","last_undegraded":"2024-03-06T07:28:40.869128+0000","last_fullsized":"2024-03-06T07:28:40.869128+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:07.122213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867094+0000","last_change":"2024-03-06T07:28:40.871729+0000","last_active":"2024-03-06T07:28:41.867094+0000","last_peered":"2024-03-06T07:28:41.867094+0000","last_clean":"2024-03-06T07:28:41.867094+0000","last_became_active":"2024-03-06T07:28:40.865342+0000","last_became_peered":"2024-03-06T07:28:40.865342+0000","last_unstale":"2024-03-06T07:28:41.867094+0000","last_undegraded":"2024-03-06T07:28:41.867094+0000","last_fullsized":"2024-03-06T07:28:41.867094+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:27.441497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.864386+0000","last_change":"2024-03-06T07:28:40.853164+0000","last_active":"2024-03-06T07:28:40.864386+0000","last_peered":"2024-03-06T07:28:40.864386+0000","last_clean":"2024-03-06T07:28:40.864386+0000","last_became_active":"2024-03-06T07:28:40.852837+0000","last_became_peered":"2024-03-06T07:28:40.852837+0000","last_unstale":"2024-03-06T07:28:40.864386+0000","last_undegraded":"2024-03-06T07:28:40.864386+0000","last_fullsized":"2024-03-06T07:28:40.864386+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:59.287242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869765+0000","last_change":"2024-03-06T07:28:40.860765+0000","last_active":"2024-03-06T07:28:40.869765+0000","last_peered":"2024-03-06T07:28:40.869765+0000","last_clean":"2024-03-06T07:28:40.869765+0000","last_became_active":"2024-03-06T07:28:40.860376+0000","last_became_peered":"2024-03-06T07:28:40.860376+0000","last_unstale":"2024-03-06T07:28:40.869765+0000","last_undegraded":"2024-03-06T07:28:40.869765+0000","last_fullsized":"2024-03-06T07:28:40.869765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:55.894729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869898+0000","last_change":"2024-03-06T07:28:40.862685+0000","last_active":"2024-03-06T07:28:40.869898+0000","last_peered":"2024-03-06T07:28:40.869898+0000","last_clean":"2024-03-06T07:28:40.869898+0000","last_became_active":"2024-03-06T07:28:40.862399+0000","last_became_peered":"2024-03-06T07:28:40.862399+0000","last_unstale":"2024-03-06T07:28:40.869898+0000","last_undegraded":"2024-03-06T07:28:40.869898+0000","last_fullsized":"2024-03-06T07:28:40.869898+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:35.590273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871566+0000","last_change":"2024-03-06T07:28:40.865321+0000","last_active":"2024-03-06T07:28:40.871566+0000","last_peered":"2024-03-06T07:28:40.871566+0000","last_clean":"2024-03-06T07:28:40.871566+0000","last_became_active":"2024-03-06T07:28:40.865038+0000","last_became_peered":"2024-03-06T07:28:40.865038+0000","last_unstale":"2024-03-06T07:28:40.871566+0000","last_undegraded":"2024-03-06T07:28:40.871566+0000","last_fullsized":"2024-03-06T07:28:40.871566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:37.635530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873521+0000","last_change":"2024-03-06T07:28:40.867951+0000","last_active":"2024-03-06T07:28:40.873521+0000","last_peered":"2024-03-06T07:28:40.873521+0000","last_clean":"2024-03-06T07:28:40.873521+0000","last_became_active":"2024-03-06T07:28:40.867609+0000","last_became_peered":"2024-03-06T07:28:40.867609+0000","last_unstale":"2024-03-06T07:28:40.873521+0000","last_undegraded":"2024-03-06T07:28:40.873521+0000","last_fullsized":"2024-03-06T07:28:40.873521+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:06.193722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.868903+0000","last_change":"2024-03-06T07:28:40.862477+0000","last_active":"2024-03-06T07:28:40.868903+0000","last_peered":"2024-03-06T07:28:40.868903+0000","last_clean":"2024-03-06T07:28:40.868903+0000","last_became_active":"2024-03-06T07:28:40.862208+0000","last_became_peered":"2024-03-06T07:28:40.862208+0000","last_unstale":"2024-03-06T07:28:40.868903+0000","last_undegraded":"2024-03-06T07:28:40.868903+0000","last_fullsized":"2024-03-06T07:28:40.868903+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:32.044942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882365+0000","last_change":"2024-03-06T07:28:39.831689+0000","last_active":"2024-03-06T07:28:40.882365+0000","last_peered":"2024-03-06T07:28:40.882365+0000","last_clean":"2024-03-06T07:28:40.882365+0000","last_became_active":"2024-03-06T07:28:39.831154+0000","last_became_peered":"2024-03-06T07:28:39.831154+0000","last_unstale":"2024-03-06T07:28:40.882365+0000","last_undegraded":"2024-03-06T07:28:40.882365+0000","last_fullsized":"2024-03-06T07:28:40.882365+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:55.978361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.871697+0000","last_change":"2024-03-06T07:28:40.869278+0000","last_active":"2024-03-06T07:28:40.871697+0000","last_peered":"2024-03-06T07:28:40.871697+0000","last_clean":"2024-03-06T07:28:40.871697+0000","last_became_active":"2024-03-06T07:28:40.867903+0000","last_became_peered":"2024-03-06T07:28:40.867903+0000","last_unstale":"2024-03-06T07:28:40.871697+0000","last_undegraded":"2024-03-06T07:28:40.871697+0000","last_fullsized":"2024-03-06T07:28:40.871697+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:24.261260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.879244+0000","last_change":"2024-03-06T07:28:39.831295+0000","last_active":"2024-03-06T07:28:40.879244+0000","last_peered":"2024-03-06T07:28:40.879244+0000","last_clean":"2024-03-06T07:28:40.879244+0000","last_became_active":"2024-03-06T07:28:39.830638+0000","last_became_peered":"2024-03-06T07:28:39.830638+0000","last_unstale":"2024-03-06T07:28:40.879244+0000","last_undegraded":"2024-03-06T07:28:40.879244+0000","last_fullsized":"2024-03-06T07:28:40.879244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:36.630626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.877219+0000","last_change":"2024-03-06T07:28:40.871580+0000","last_active":"2024-03-06T07:28:40.877219+0000","last_peered":"2024-03-06T07:28:40.877219+0000","last_clean":"2024-03-06T07:28:40.877219+0000","last_became_active":"2024-03-06T07:28:40.870854+0000","last_became_peered":"2024-03-06T07:28:40.870854+0000","last_unstale":"2024-03-06T07:28:40.877219+0000","last_undegraded":"2024-03-06T07:28:40.877219+0000","last_fullsized":"2024-03-06T07:28:40.877219+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:37:24.089323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.885641+0000","last_change":"2024-03-06T07:28:39.833946+0000","last_active":"2024-03-06T07:28:40.885641+0000","last_peered":"2024-03-06T07:28:40.885641+0000","last_clean":"2024-03-06T07:28:40.885641+0000","last_became_active":"2024-03-06T07:28:39.833665+0000","last_became_peered":"2024-03-06T07:28:39.833665+0000","last_unstale":"2024-03-06T07:28:40.885641+0000","last_undegraded":"2024-03-06T07:28:40.885641+0000","last_fullsized":"2024-03-06T07:28:40.885641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:41:16.679298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867303+0000","last_change":"2024-03-06T07:28:40.871442+0000","last_active":"2024-03-06T07:28:41.867303+0000","last_peered":"2024-03-06T07:28:41.867303+0000","last_clean":"2024-03-06T07:28:41.867303+0000","last_became_active":"2024-03-06T07:28:40.863935+0000","last_became_peered":"2024-03-06T07:28:40.863935+0000","last_unstale":"2024-03-06T07:28:41.867303+0000","last_undegraded":"2024-03-06T07:28:41.867303+0000","last_fullsized":"2024-03-06T07:28:41.867303+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:11.062252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056335+0000","last_change":"2024-03-06T07:28:39.833643+0000","last_active":"2024-03-06T07:28:42.056335+0000","last_peered":"2024-03-06T07:28:42.056335+0000","last_clean":"2024-03-06T07:28:42.056335+0000","last_became_active":"2024-03-06T07:28:39.833170+0000","last_became_peered":"2024-03-06T07:28:39.833170+0000","last_unstale":"2024-03-06T07:28:42.056335+0000","last_undegraded":"2024-03-06T07:28:42.056335+0000","last_fullsized":"2024-03-06T07:28:42.056335+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:36.553751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.874948+0000","last_change":"2024-03-06T07:28:40.872246+0000","last_active":"2024-03-06T07:28:40.874948+0000","last_peered":"2024-03-06T07:28:40.874948+0000","last_clean":"2024-03-06T07:28:40.874948+0000","last_became_active":"2024-03-06T07:28:40.871404+0000","last_became_peered":"2024-03-06T07:28:40.871404+0000","last_unstale":"2024-03-06T07:28:40.874948+0000","last_undegraded":"2024-03-06T07:28:40.874948+0000","last_fullsized":"2024-03-06T07:28:40.874948+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:09:05.259044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882996+0000","last_change":"2024-03-06T07:28:39.834605+0000","last_active":"2024-03-06T07:28:40.882996+0000","last_peered":"2024-03-06T07:28:40.882996+0000","last_clean":"2024-03-06T07:28:40.882996+0000","last_became_active":"2024-03-06T07:28:39.834291+0000","last_became_peered":"2024-03-06T07:28:39.834291+0000","last_unstale":"2024-03-06T07:28:40.882996+0000","last_undegraded":"2024-03-06T07:28:40.882996+0000","last_fullsized":"2024-03-06T07:28:40.882996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:09.296468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.865538+0000","last_change":"2024-03-06T07:28:40.859013+0000","last_active":"2024-03-06T07:28:40.865538+0000","last_peered":"2024-03-06T07:28:40.865538+0000","last_clean":"2024-03-06T07:28:40.865538+0000","last_became_active":"2024-03-06T07:28:40.858711+0000","last_became_peered":"2024-03-06T07:28:40.858711+0000","last_unstale":"2024-03-06T07:28:40.865538+0000","last_undegraded":"2024-03-06T07:28:40.865538+0000","last_fullsized":"2024-03-06T07:28:40.865538+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:11.722633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.883622+0000","last_change":"2024-03-06T07:28:39.840615+0000","last_active":"2024-03-06T07:28:40.883622+0000","last_peered":"2024-03-06T07:28:40.883622+0000","last_clean":"2024-03-06T07:28:40.883622+0000","last_became_active":"2024-03-06T07:28:39.840213+0000","last_became_peered":"2024-03-06T07:28:39.840213+0000","last_unstale":"2024-03-06T07:28:40.883622+0000","last_undegraded":"2024-03-06T07:28:40.883622+0000","last_fullsized":"2024-03-06T07:28:40.883622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:58.053440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869443+0000","last_change":"2024-03-06T07:28:40.864315+0000","last_active":"2024-03-06T07:28:40.869443+0000","last_peered":"2024-03-06T07:28:40.869443+0000","last_clean":"2024-03-06T07:28:40.869443+0000","last_became_active":"2024-03-06T07:28:40.863397+0000","last_became_peered":"2024-03-06T07:28:40.863397+0000","last_unstale":"2024-03-06T07:28:40.869443+0000","last_undegraded":"2024-03-06T07:28:40.869443+0000","last_fullsized":"2024-03-06T07:28:40.869443+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:01.073459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.835349+0000","last_change":"2024-03-06T07:28:39.825334+0000","last_active":"2024-03-06T07:28:40.835349+0000","last_peered":"2024-03-06T07:28:40.835349+0000","last_clean":"2024-03-06T07:28:40.835349+0000","last_became_active":"2024-03-06T07:28:39.824980+0000","last_became_peered":"2024-03-06T07:28:39.824980+0000","last_unstale":"2024-03-06T07:28:40.835349+0000","last_undegraded":"2024-03-06T07:28:40.835349+0000","last_fullsized":"2024-03-06T07:28:40.835349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:08.188611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864679+0000","last_change":"2024-03-06T07:28:40.872116+0000","last_active":"2024-03-06T07:28:41.864679+0000","last_peered":"2024-03-06T07:28:41.864679+0000","last_clean":"2024-03-06T07:28:41.864679+0000","last_became_active":"2024-03-06T07:28:40.871809+0000","last_became_peered":"2024-03-06T07:28:40.871809+0000","last_unstale":"2024-03-06T07:28:41.864679+0000","last_undegraded":"2024-03-06T07:28:41.864679+0000","last_fullsized":"2024-03-06T07:28:41.864679+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:24:44.690312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.846159+0000","last_change":"2024-03-06T07:28:39.829966+0000","last_active":"2024-03-06T07:28:40.846159+0000","last_peered":"2024-03-06T07:28:40.846159+0000","last_clean":"2024-03-06T07:28:40.846159+0000","last_became_active":"2024-03-06T07:28:39.828232+0000","last_became_peered":"2024-03-06T07:28:39.828232+0000","last_unstale":"2024-03-06T07:28:40.846159+0000","last_undegraded":"2024-03-06T07:28:40.846159+0000","last_fullsized":"2024-03-06T07:28:40.846159+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:36.659740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867489+0000","last_change":"2024-03-06T07:28:40.861779+0000","last_active":"2024-03-06T07:28:40.867489+0000","last_peered":"2024-03-06T07:28:40.867489+0000","last_clean":"2024-03-06T07:28:40.867489+0000","last_became_active":"2024-03-06T07:28:40.861389+0000","last_became_peered":"2024-03-06T07:28:40.861389+0000","last_unstale":"2024-03-06T07:28:40.867489+0000","last_undegraded":"2024-03-06T07:28:40.867489+0000","last_fullsized":"2024-03-06T07:28:40.867489+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:22.055775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.838635+0000","last_change":"2024-03-06T07:28:39.830244+0000","last_active":"2024-03-06T07:28:40.838635+0000","last_peered":"2024-03-06T07:28:40.838635+0000","last_clean":"2024-03-06T07:28:40.838635+0000","last_became_active":"2024-03-06T07:28:39.829818+0000","last_became_peered":"2024-03-06T07:28:39.829818+0000","last_unstale":"2024-03-06T07:28:40.838635+0000","last_undegraded":"2024-03-06T07:28:40.838635+0000","last_fullsized":"2024-03-06T07:28:40.838635+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:10.717729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882357+0000","last_change":"2024-03-06T07:28:40.877281+0000","last_active":"2024-03-06T07:28:40.882357+0000","last_peered":"2024-03-06T07:28:40.882357+0000","last_clean":"2024-03-06T07:28:40.882357+0000","last_became_active":"2024-03-06T07:28:40.876953+0000","last_became_peered":"2024-03-06T07:28:40.876953+0000","last_unstale":"2024-03-06T07:28:40.882357+0000","last_undegraded":"2024-03-06T07:28:40.882357+0000","last_fullsized":"2024-03-06T07:28:40.882357+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:44.246160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.831215+0000","last_change":"2024-03-06T07:28:39.832385+0000","last_active":"2024-03-06T07:28:40.831215+0000","last_peered":"2024-03-06T07:28:40.831215+0000","last_clean":"2024-03-06T07:28:40.831215+0000","last_became_active":"2024-03-06T07:28:39.832146+0000","last_became_peered":"2024-03-06T07:28:39.832146+0000","last_unstale":"2024-03-06T07:28:40.831215+0000","last_undegraded":"2024-03-06T07:28:40.831215+0000","last_fullsized":"2024-03-06T07:28:40.831215+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:46.705097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.868106+0000","last_change":"2024-03-06T07:28:40.855877+0000","last_active":"2024-03-06T07:28:40.868106+0000","last_peered":"2024-03-06T07:28:40.868106+0000","last_clean":"2024-03-06T07:28:40.868106+0000","last_became_active":"2024-03-06T07:28:40.855602+0000","last_became_peered":"2024-03-06T07:28:40.855602+0000","last_unstale":"2024-03-06T07:28:40.868106+0000","last_undegraded":"2024-03-06T07:28:40.868106+0000","last_fullsized":"2024-03-06T07:28:40.868106+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:50.125370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.883184+0000","last_change":"2024-03-06T07:28:39.836497+0000","last_active":"2024-03-06T07:28:40.883184+0000","last_peered":"2024-03-06T07:28:40.883184+0000","last_clean":"2024-03-06T07:28:40.883184+0000","last_became_active":"2024-03-06T07:28:39.832020+0000","last_became_peered":"2024-03-06T07:28:39.832020+0000","last_unstale":"2024-03-06T07:28:40.883184+0000","last_undegraded":"2024-03-06T07:28:40.883184+0000","last_fullsized":"2024-03-06T07:28:40.883184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:08.188611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881846+0000","last_change":"2024-03-06T07:28:40.879191+0000","last_active":"2024-03-06T07:28:40.881846+0000","last_peered":"2024-03-06T07:28:40.881846+0000","last_clean":"2024-03-06T07:28:40.881846+0000","last_became_active":"2024-03-06T07:28:40.878686+0000","last_became_peered":"2024-03-06T07:28:40.878686+0000","last_unstale":"2024-03-06T07:28:40.881846+0000","last_undegraded":"2024-03-06T07:28:40.881846+0000","last_fullsized":"2024-03-06T07:28:40.881846+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:31:15.814867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.897062+0000","last_change":"2024-03-06T07:28:39.837789+0000","last_active":"2024-03-06T07:28:40.897062+0000","last_peered":"2024-03-06T07:28:40.897062+0000","last_clean":"2024-03-06T07:28:40.897062+0000","last_became_active":"2024-03-06T07:28:39.836249+0000","last_became_peered":"2024-03-06T07:28:39.836249+0000","last_unstale":"2024-03-06T07:28:40.897062+0000","last_undegraded":"2024-03-06T07:28:40.897062+0000","last_fullsized":"2024-03-06T07:28:40.897062+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:58.282339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864386+0000","last_change":"2024-03-06T07:28:40.871379+0000","last_active":"2024-03-06T07:28:41.864386+0000","last_peered":"2024-03-06T07:28:41.864386+0000","last_clean":"2024-03-06T07:28:41.864386+0000","last_became_active":"2024-03-06T07:28:40.866071+0000","last_became_peered":"2024-03-06T07:28:40.866071+0000","last_unstale":"2024-03-06T07:28:41.864386+0000","last_undegraded":"2024-03-06T07:28:41.864386+0000","last_fullsized":"2024-03-06T07:28:41.864386+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:00:18.515752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882930+0000","last_change":"2024-03-06T07:28:39.834756+0000","last_active":"2024-03-06T07:28:40.882930+0000","last_peered":"2024-03-06T07:28:40.882930+0000","last_clean":"2024-03-06T07:28:40.882930+0000","last_became_active":"2024-03-06T07:28:39.833731+0000","last_became_peered":"2024-03-06T07:28:39.833731+0000","last_unstale":"2024-03-06T07:28:40.882930+0000","last_undegraded":"2024-03-06T07:28:40.882930+0000","last_fullsized":"2024-03-06T07:28:40.882930+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:34.199312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867878+0000","last_change":"2024-03-06T07:28:40.853500+0000","last_active":"2024-03-06T07:28:40.867878+0000","last_peered":"2024-03-06T07:28:40.867878+0000","last_clean":"2024-03-06T07:28:40.867878+0000","last_became_active":"2024-03-06T07:28:40.853015+0000","last_became_peered":"2024-03-06T07:28:40.853015+0000","last_unstale":"2024-03-06T07:28:40.867878+0000","last_undegraded":"2024-03-06T07:28:40.867878+0000","last_fullsized":"2024-03-06T07:28:40.867878+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:34.394133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864771+0000","last_change":"2024-03-06T07:28:39.814957+0000","last_active":"2024-03-06T07:28:41.864771+0000","last_peered":"2024-03-06T07:28:41.864771+0000","last_clean":"2024-03-06T07:28:41.864771+0000","last_became_active":"2024-03-06T07:28:39.814664+0000","last_became_peered":"2024-03-06T07:28:39.814664+0000","last_unstale":"2024-03-06T07:28:41.864771+0000","last_undegraded":"2024-03-06T07:28:41.864771+0000","last_fullsized":"2024-03-06T07:28:41.864771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:22.337525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.873544+0000","last_change":"2024-03-06T07:28:40.868167+0000","last_active":"2024-03-06T07:28:40.873544+0000","last_peered":"2024-03-06T07:28:40.873544+0000","last_clean":"2024-03-06T07:28:40.873544+0000","last_became_active":"2024-03-06T07:28:40.867692+0000","last_became_peered":"2024-03-06T07:28:40.867692+0000","last_unstale":"2024-03-06T07:28:40.873544+0000","last_undegraded":"2024-03-06T07:28:40.873544+0000","last_fullsized":"2024-03-06T07:28:40.873544+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:16.411787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.834893+0000","last_change":"2024-03-06T07:28:39.829648+0000","last_active":"2024-03-06T07:28:40.834893+0000","last_peered":"2024-03-06T07:28:40.834893+0000","last_clean":"2024-03-06T07:28:40.834893+0000","last_became_active":"2024-03-06T07:28:39.828961+0000","last_became_peered":"2024-03-06T07:28:39.828961+0000","last_unstale":"2024-03-06T07:28:40.834893+0000","last_undegraded":"2024-03-06T07:28:40.834893+0000","last_fullsized":"2024-03-06T07:28:40.834893+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:31.040038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.867477+0000","last_change":"2024-03-06T07:28:40.858454+0000","last_active":"2024-03-06T07:28:40.867477+0000","last_peered":"2024-03-06T07:28:40.867477+0000","last_clean":"2024-03-06T07:28:40.867477+0000","last_became_active":"2024-03-06T07:28:40.858149+0000","last_became_peered":"2024-03-06T07:28:40.858149+0000","last_unstale":"2024-03-06T07:28:40.867477+0000","last_undegraded":"2024-03-06T07:28:40.867477+0000","last_fullsized":"2024-03-06T07:28:40.867477+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:28.464391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.893528+0000","last_change":"2024-03-06T07:28:39.834458+0000","last_active":"2024-03-06T07:28:40.893528+0000","last_peered":"2024-03-06T07:28:40.893528+0000","last_clean":"2024-03-06T07:28:40.893528+0000","last_became_active":"2024-03-06T07:28:39.833866+0000","last_became_peered":"2024-03-06T07:28:39.833866+0000","last_unstale":"2024-03-06T07:28:40.893528+0000","last_undegraded":"2024-03-06T07:28:40.893528+0000","last_fullsized":"2024-03-06T07:28:40.893528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:56.404088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869653+0000","last_change":"2024-03-06T07:28:40.867062+0000","last_active":"2024-03-06T07:28:40.869653+0000","last_peered":"2024-03-06T07:28:40.869653+0000","last_clean":"2024-03-06T07:28:40.869653+0000","last_became_active":"2024-03-06T07:28:40.866709+0000","last_became_peered":"2024-03-06T07:28:40.866709+0000","last_unstale":"2024-03-06T07:28:40.869653+0000","last_undegraded":"2024-03-06T07:28:40.869653+0000","last_fullsized":"2024-03-06T07:28:40.869653+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:56.854757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867281+0000","last_change":"2024-03-06T07:28:39.835450+0000","last_active":"2024-03-06T07:28:41.867281+0000","last_peered":"2024-03-06T07:28:41.867281+0000","last_clean":"2024-03-06T07:28:41.867281+0000","last_became_active":"2024-03-06T07:28:39.835174+0000","last_became_peered":"2024-03-06T07:28:39.835174+0000","last_unstale":"2024-03-06T07:28:41.867281+0000","last_undegraded":"2024-03-06T07:28:41.867281+0000","last_fullsized":"2024-03-06T07:28:41.867281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:00.068555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.870324+0000","last_change":"2024-03-06T07:28:40.866818+0000","last_active":"2024-03-06T07:28:40.870324+0000","last_peered":"2024-03-06T07:28:40.870324+0000","last_clean":"2024-03-06T07:28:40.870324+0000","last_became_active":"2024-03-06T07:28:40.866526+0000","last_became_peered":"2024-03-06T07:28:40.866526+0000","last_unstale":"2024-03-06T07:28:40.870324+0000","last_undegraded":"2024-03-06T07:28:40.870324+0000","last_fullsized":"2024-03-06T07:28:40.870324+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:34.394133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882604+0000","last_change":"2024-03-06T07:28:39.834560+0000","last_active":"2024-03-06T07:28:40.882604+0000","last_peered":"2024-03-06T07:28:40.882604+0000","last_clean":"2024-03-06T07:28:40.882604+0000","last_became_active":"2024-03-06T07:28:39.834193+0000","last_became_peered":"2024-03-06T07:28:39.834193+0000","last_unstale":"2024-03-06T07:28:40.882604+0000","last_undegraded":"2024-03-06T07:28:40.882604+0000","last_fullsized":"2024-03-06T07:28:40.882604+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:22.337525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.865513+0000","last_change":"2024-03-06T07:28:40.873964+0000","last_active":"2024-03-06T07:28:41.865513+0000","last_peered":"2024-03-06T07:28:41.865513+0000","last_clean":"2024-03-06T07:28:41.865513+0000","last_became_active":"2024-03-06T07:28:40.873682+0000","last_became_peered":"2024-03-06T07:28:40.873682+0000","last_unstale":"2024-03-06T07:28:41.865513+0000","last_undegraded":"2024-03-06T07:28:41.865513+0000","last_fullsized":"2024-03-06T07:28:41.865513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:33.591548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.882857+0000","last_change":"2024-03-06T07:28:39.836333+0000","last_active":"2024-03-06T07:28:40.882857+0000","last_peered":"2024-03-06T07:28:40.882857+0000","last_clean":"2024-03-06T07:28:40.882857+0000","last_became_active":"2024-03-06T07:28:39.836040+0000","last_became_peered":"2024-03-06T07:28:39.836040+0000","last_unstale":"2024-03-06T07:28:40.882857+0000","last_undegraded":"2024-03-06T07:28:40.882857+0000","last_fullsized":"2024-03-06T07:28:40.882857+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:22.325631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056463+0000","last_change":"2024-03-06T07:28:40.868335+0000","last_active":"2024-03-06T07:28:42.056463+0000","last_peered":"2024-03-06T07:28:42.056463+0000","last_clean":"2024-03-06T07:28:42.056463+0000","last_became_active":"2024-03-06T07:28:40.865156+0000","last_became_peered":"2024-03-06T07:28:40.865156+0000","last_unstale":"2024-03-06T07:28:42.056463+0000","last_undegraded":"2024-03-06T07:28:42.056463+0000","last_fullsized":"2024-03-06T07:28:42.056463+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:55.894729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.849230+0000","last_change":"2024-03-06T07:28:39.834169+0000","last_active":"2024-03-06T07:28:40.849230+0000","last_peered":"2024-03-06T07:28:40.849230+0000","last_clean":"2024-03-06T07:28:40.849230+0000","last_became_active":"2024-03-06T07:28:39.833753+0000","last_became_peered":"2024-03-06T07:28:39.833753+0000","last_unstale":"2024-03-06T07:28:40.849230+0000","last_undegraded":"2024-03-06T07:28:40.849230+0000","last_fullsized":"2024-03-06T07:28:40.849230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:21.576285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056339+0000","last_change":"2024-03-06T07:28:40.867535+0000","last_active":"2024-03-06T07:28:42.056339+0000","last_peered":"2024-03-06T07:28:42.056339+0000","last_clean":"2024-03-06T07:28:42.056339+0000","last_became_active":"2024-03-06T07:28:40.859951+0000","last_became_peered":"2024-03-06T07:28:40.859951+0000","last_unstale":"2024-03-06T07:28:42.056339+0000","last_undegraded":"2024-03-06T07:28:42.056339+0000","last_fullsized":"2024-03-06T07:28:42.056339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:23.330534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881513+0000","last_change":"2024-03-06T07:28:39.823828+0000","last_active":"2024-03-06T07:28:40.881513+0000","last_peered":"2024-03-06T07:28:40.881513+0000","last_clean":"2024-03-06T07:28:40.881513+0000","last_became_active":"2024-03-06T07:28:39.823039+0000","last_became_peered":"2024-03-06T07:28:39.823039+0000","last_unstale":"2024-03-06T07:28:40.881513+0000","last_undegraded":"2024-03-06T07:28:40.881513+0000","last_fullsized":"2024-03-06T07:28:40.881513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:55.978361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056418+0000","last_change":"2024-03-06T07:28:40.861016+0000","last_active":"2024-03-06T07:28:42.056418+0000","last_peered":"2024-03-06T07:28:42.056418+0000","last_clean":"2024-03-06T07:28:42.056418+0000","last_became_active":"2024-03-06T07:28:40.857378+0000","last_became_peered":"2024-03-06T07:28:40.857378+0000","last_unstale":"2024-03-06T07:28:42.056418+0000","last_undegraded":"2024-03-06T07:28:42.056418+0000","last_fullsized":"2024-03-06T07:28:42.056418+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:11.454733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.850677+0000","last_change":"2024-03-06T07:28:39.821996+0000","last_active":"2024-03-06T07:28:40.850677+0000","last_peered":"2024-03-06T07:28:40.850677+0000","last_clean":"2024-03-06T07:28:40.850677+0000","last_became_active":"2024-03-06T07:28:39.820771+0000","last_became_peered":"2024-03-06T07:28:39.820771+0000","last_unstale":"2024-03-06T07:28:40.850677+0000","last_undegraded":"2024-03-06T07:28:40.850677+0000","last_fullsized":"2024-03-06T07:28:40.850677+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:43.981321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056093+0000","last_change":"2024-03-06T07:28:40.867025+0000","last_active":"2024-03-06T07:28:42.056093+0000","last_peered":"2024-03-06T07:28:42.056093+0000","last_clean":"2024-03-06T07:28:42.056093+0000","last_became_active":"2024-03-06T07:28:40.855104+0000","last_became_peered":"2024-03-06T07:28:40.855104+0000","last_unstale":"2024-03-06T07:28:42.056093+0000","last_undegraded":"2024-03-06T07:28:42.056093+0000","last_fullsized":"2024-03-06T07:28:42.056093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:10.301372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.847544+0000","last_change":"2024-03-06T07:28:39.837722+0000","last_active":"2024-03-06T07:28:40.847544+0000","last_peered":"2024-03-06T07:28:40.847544+0000","last_clean":"2024-03-06T07:28:40.847544+0000","last_became_active":"2024-03-06T07:28:39.837079+0000","last_became_peered":"2024-03-06T07:28:39.837079+0000","last_unstale":"2024-03-06T07:28:40.847544+0000","last_undegraded":"2024-03-06T07:28:40.847544+0000","last_fullsized":"2024-03-06T07:28:40.847544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:42.568653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.866152+0000","last_change":"2024-03-06T07:28:40.860010+0000","last_active":"2024-03-06T07:28:40.866152+0000","last_peered":"2024-03-06T07:28:40.866152+0000","last_clean":"2024-03-06T07:28:40.866152+0000","last_became_active":"2024-03-06T07:28:40.859737+0000","last_became_peered":"2024-03-06T07:28:40.859737+0000","last_unstale":"2024-03-06T07:28:40.866152+0000","last_undegraded":"2024-03-06T07:28:40.866152+0000","last_fullsized":"2024-03-06T07:28:40.866152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:26:01.073459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.840047+0000","last_change":"2024-03-06T07:28:39.834432+0000","last_active":"2024-03-06T07:28:40.840047+0000","last_peered":"2024-03-06T07:28:40.840047+0000","last_clean":"2024-03-06T07:28:40.840047+0000","last_became_active":"2024-03-06T07:28:39.834132+0000","last_became_peered":"2024-03-06T07:28:39.834132+0000","last_unstale":"2024-03-06T07:28:40.840047+0000","last_undegraded":"2024-03-06T07:28:40.840047+0000","last_fullsized":"2024-03-06T07:28:40.840047+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:31.040038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:42.056393+0000","last_change":"2024-03-06T07:28:40.868822+0000","last_active":"2024-03-06T07:28:42.056393+0000","last_peered":"2024-03-06T07:28:42.056393+0000","last_clean":"2024-03-06T07:28:42.056393+0000","last_became_active":"2024-03-06T07:28:40.868585+0000","last_became_peered":"2024-03-06T07:28:40.868585+0000","last_unstale":"2024-03-06T07:28:42.056393+0000","last_undegraded":"2024-03-06T07:28:42.056393+0000","last_fullsized":"2024-03-06T07:28:42.056393+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:23.342428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.867434+0000","last_change":"2024-03-06T07:28:39.826331+0000","last_active":"2024-03-06T07:28:41.867434+0000","last_peered":"2024-03-06T07:28:41.867434+0000","last_clean":"2024-03-06T07:28:41.867434+0000","last_became_active":"2024-03-06T07:28:39.825988+0000","last_became_peered":"2024-03-06T07:28:39.825988+0000","last_unstale":"2024-03-06T07:28:41.867434+0000","last_undegraded":"2024-03-06T07:28:41.867434+0000","last_fullsized":"2024-03-06T07:28:41.867434+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:55.849853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.880074+0000","last_change":"2024-03-06T07:28:40.877256+0000","last_active":"2024-03-06T07:28:40.880074+0000","last_peered":"2024-03-06T07:28:40.880074+0000","last_clean":"2024-03-06T07:28:40.880074+0000","last_became_active":"2024-03-06T07:28:40.875647+0000","last_became_peered":"2024-03-06T07:28:40.875647+0000","last_unstale":"2024-03-06T07:28:40.880074+0000","last_undegraded":"2024-03-06T07:28:40.880074+0000","last_fullsized":"2024-03-06T07:28:40.880074+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:50.125370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.831333+0000","last_change":"2024-03-06T07:28:39.822567+0000","last_active":"2024-03-06T07:28:40.831333+0000","last_peered":"2024-03-06T07:28:40.831333+0000","last_clean":"2024-03-06T07:28:40.831333+0000","last_became_active":"2024-03-06T07:28:39.822273+0000","last_became_peered":"2024-03-06T07:28:39.822273+0000","last_unstale":"2024-03-06T07:28:40.831333+0000","last_undegraded":"2024-03-06T07:28:40.831333+0000","last_fullsized":"2024-03-06T07:28:40.831333+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:23.256357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.881924+0000","last_change":"2024-03-06T07:28:40.877693+0000","last_active":"2024-03-06T07:28:40.881924+0000","last_peered":"2024-03-06T07:28:40.881924+0000","last_clean":"2024-03-06T07:28:40.881924+0000","last_became_active":"2024-03-06T07:28:40.875906+0000","last_became_peered":"2024-03-06T07:28:40.875906+0000","last_unstale":"2024-03-06T07:28:40.881924+0000","last_undegraded":"2024-03-06T07:28:40.881924+0000","last_fullsized":"2024-03-06T07:28:40.881924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:59.287242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.865654+0000","last_change":"2024-03-06T07:28:39.825495+0000","last_active":"2024-03-06T07:28:41.865654+0000","last_peered":"2024-03-06T07:28:41.865654+0000","last_clean":"2024-03-06T07:28:41.865654+0000","last_became_active":"2024-03-06T07:28:39.825214+0000","last_became_peered":"2024-03-06T07:28:39.825214+0000","last_unstale":"2024-03-06T07:28:41.865654+0000","last_undegraded":"2024-03-06T07:28:41.865654+0000","last_fullsized":"2024-03-06T07:28:41.865654+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:54.889825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.875487+0000","last_change":"2024-03-06T07:28:40.872905+0000","last_active":"2024-03-06T07:28:40.875487+0000","last_peered":"2024-03-06T07:28:40.875487+0000","last_clean":"2024-03-06T07:28:40.875487+0000","last_became_active":"2024-03-06T07:28:40.872581+0000","last_became_peered":"2024-03-06T07:28:40.872581+0000","last_unstale":"2024-03-06T07:28:40.875487+0000","last_undegraded":"2024-03-06T07:28:40.875487+0000","last_fullsized":"2024-03-06T07:28:40.875487+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:54.840821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.830185+0000","last_change":"2024-03-06T07:28:39.823880+0000","last_active":"2024-03-06T07:28:40.830185+0000","last_peered":"2024-03-06T07:28:40.830185+0000","last_clean":"2024-03-06T07:28:40.830185+0000","last_became_active":"2024-03-06T07:28:39.823084+0000","last_became_peered":"2024-03-06T07:28:39.823084+0000","last_unstale":"2024-03-06T07:28:40.830185+0000","last_undegraded":"2024-03-06T07:28:40.830185+0000","last_fullsized":"2024-03-06T07:28:40.830185+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:27.459488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.870408+0000","last_change":"2024-03-06T07:28:40.862951+0000","last_active":"2024-03-06T07:28:40.870408+0000","last_peered":"2024-03-06T07:28:40.870408+0000","last_clean":"2024-03-06T07:28:40.870408+0000","last_became_active":"2024-03-06T07:28:40.861646+0000","last_became_peered":"2024-03-06T07:28:40.861646+0000","last_unstale":"2024-03-06T07:28:40.870408+0000","last_undegraded":"2024-03-06T07:28:40.870408+0000","last_fullsized":"2024-03-06T07:28:40.870408+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:36:09.193514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-06T07:28:41.864923+0000","last_change":"2024-03-06T07:28:39.826645+0000","last_active":"2024-03-06T07:28:41.864923+0000","last_peered":"2024-03-06T07:28:41.864923+0000","last_clean":"2024-03-06T07:28:41.864923+0000","last_became_active":"2024-03-06T07:28:39.826365+0000","last_became_peered":"2024-03-06T07:28:39.826365+0000","last_unstale":"2024-03-06T07:28:41.864923+0000","last_undegraded":"2024-03-06T07:28:41.864923+0000","last_fullsized":"2024-03-06T07:28:41.864923+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:10.449829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869093+0000","last_change":"2024-03-06T07:28:40.858520+0000","last_active":"2024-03-06T07:28:40.869093+0000","last_peered":"2024-03-06T07:28:40.869093+0000","last_clean":"2024-03-06T07:28:40.869093+0000","last_became_active":"2024-03-06T07:28:40.858181+0000","last_became_peered":"2024-03-06T07:28:40.858181+0000","last_unstale":"2024-03-06T07:28:40.869093+0000","last_undegraded":"2024-03-06T07:28:40.869093+0000","last_fullsized":"2024-03-06T07:28:40.869093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:36.401757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.847767+0000","last_change":"2024-03-06T07:28:39.835477+0000","last_active":"2024-03-06T07:28:40.847767+0000","last_peered":"2024-03-06T07:28:40.847767+0000","last_clean":"2024-03-06T07:28:40.847767+0000","last_became_active":"2024-03-06T07:28:39.835166+0000","last_became_peered":"2024-03-06T07:28:39.835166+0000","last_unstale":"2024-03-06T07:28:40.847767+0000","last_undegraded":"2024-03-06T07:28:40.847767+0000","last_fullsized":"2024-03-06T07:28:40.847767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:00.877196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.869789+0000","last_change":"2024-03-06T07:28:40.867485+0000","last_active":"2024-03-06T07:28:40.869789+0000","last_peered":"2024-03-06T07:28:40.869789+0000","last_clean":"2024-03-06T07:28:40.869789+0000","last_became_active":"2024-03-06T07:28:40.866013+0000","last_became_peered":"2024-03-06T07:28:40.866013+0000","last_unstale":"2024-03-06T07:28:40.869789+0000","last_undegraded":"2024-03-06T07:28:40.869789+0000","last_fullsized":"2024-03-06T07:28:40.869789+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:30:35.590273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.841509+0000","last_change":"2024-03-06T07:28:39.831389+0000","last_active":"2024-03-06T07:28:40.841509+0000","last_peered":"2024-03-06T07:28:40.841509+0000","last_clean":"2024-03-06T07:28:40.841509+0000","last_became_active":"2024-03-06T07:28:39.831100+0000","last_became_peered":"2024-03-06T07:28:39.831100+0000","last_unstale":"2024-03-06T07:28:40.841509+0000","last_undegraded":"2024-03-06T07:28:40.841509+0000","last_fullsized":"2024-03-06T07:28:40.841509+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:21.050872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.875406+0000","last_change":"2024-03-06T07:28:40.873299+0000","last_active":"2024-03-06T07:28:40.875406+0000","last_peered":"2024-03-06T07:28:40.875406+0000","last_clean":"2024-03-06T07:28:40.875406+0000","last_became_active":"2024-03-06T07:28:40.872979+0000","last_became_peered":"2024-03-06T07:28:40.872979+0000","last_unstale":"2024-03-06T07:28:40.875406+0000","last_undegraded":"2024-03-06T07:28:40.875406+0000","last_fullsized":"2024-03-06T07:28:40.875406+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:39.787637+0000","last_clean_scrub_stamp":"2024-03-06T07:28:39.787637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:25.580334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.853355+0000","last_change":"2024-03-06T07:28:39.832618+0000","last_active":"2024-03-06T07:28:40.853355+0000","last_peered":"2024-03-06T07:28:40.853355+0000","last_clean":"2024-03-06T07:28:40.853355+0000","last_became_active":"2024-03-06T07:28:39.832334+0000","last_became_peered":"2024-03-06T07:28:39.832334+0000","last_unstale":"2024-03-06T07:28:40.853355+0000","last_undegraded":"2024-03-06T07:28:40.853355+0000","last_fullsized":"2024-03-06T07:28:40.853355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:38.782733+0000","last_clean_scrub_stamp":"2024-03-06T07:28:38.782733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:10.797789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":191,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.850363+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:28:40.850363+0000","last_peered":"2024-03-06T07:28:40.850363+0000","last_clean":"2024-03-06T07:28:40.850363+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:28:40.850363+0000","last_undegraded":"2024-03-06T07:28:40.850363+0000","last_fullsized":"2024-03-06T07:28:40.850363+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509459,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":405504,"data_stored":155442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":731101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":364544,"data_stored":140733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":385024,"data_stored":148180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":385024,"data_stored":147612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":149142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":970752,"data_stored":737876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.984 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:45.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:46.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.244+0000 7fef9ae00640 1 Processor -- start 2024-03-06T07:28:46.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.244+0000 7fef9ae00640 1 -- start start 2024-03-06T07:28:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.246+0000 7fef9ae00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef94144340 0x7fef940ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.246+0000 7fef9ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef940cee10 0x7fef940b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.246+0000 7fef9ae00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef940b2b80 con 0x7fef940b4b40 2024-03-06T07:28:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.246+0000 7fef9ae00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef940cc780 con 0x7fef94144340 2024-03-06T07:28:46.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.246+0000 7fef9ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef940b3790 con 0x7fef940cee10 2024-03-06T07:28:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.247+0000 7fef99a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef94144340 0x7fef940ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.247+0000 7fef99a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef94144340 0x7fef940ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:53694/0 (socket says 172.21.15.6:53694) 2024-03-06T07:28:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.247+0000 7fef99a00640 1 -- 172.21.15.6:0/1831687043 learned_addr learned my addr 172.21.15.6:0/1831687043 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.247+0000 7fef99000640 1 --2- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef940cee10 0x7fef940b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.248+0000 7fef99a00640 1 -- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef940cee10 msgr2=0x7fef940b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.248+0000 7fef99a00640 1 --2- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef940cee10 0x7fef940b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.248+0000 7fef99a00640 1 -- 172.21.15.6:0/1831687043 >> v1:172.21.15.178:6789/0 conn(0x7fef940b4b40 legacy=0x7fef9415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.248+0000 7fef99a00640 1 -- 172.21.15.6:0/1831687043 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef940b2e60 con 0x7fef94144340 2024-03-06T07:28:46.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.249+0000 7fef99a00640 1 --2- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef94144340 0x7fef940ce8d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fef88009870 tx=0x7fef8802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc3fc21c8ac6869e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef93e00640 1 -- 172.21.15.6:0/1831687043 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88035020 con 0x7fef94144340 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef93e00640 1 -- 172.21.15.6:0/1831687043 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef88031640 con 0x7fef94144340 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef93e00640 1 -- 172.21.15.6:0/1831687043 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8802d040 con 0x7fef94144340 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 -- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef94144340 msgr2=0x7fef940ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef94144340 0x7fef940ce8d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fef88009870 tx=0x7fef8802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 -- 172.21.15.6:0/1831687043 shutdown_connections 2024-03-06T07:28:46.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef940cee10 0x7fef940b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1831687043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef94144340 0x7fef940ce8d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 -- 172.21.15.6:0/1831687043 >> 172.21.15.6:0/1831687043 conn(0x7fef9406b190 msgr2=0x7fef9406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 -- 172.21.15.6:0/1831687043 shutdown_connections 2024-03-06T07:28:46.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.250+0000 7fef9ae00640 1 -- 172.21.15.6:0/1831687043 wait complete. 2024-03-06T07:28:46.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.252+0000 7fef9ae00640 1 Processor -- start 2024-03-06T07:28:46.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.252+0000 7fef9ae00640 1 -- start start 2024-03-06T07:28:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.253+0000 7fef9ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 0x7fef9413eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.253+0000 7fef9ae00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef940cee10 0x7fef9413b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.253+0000 7fef9ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef94144340 0x7fef94142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.253+0000 7fef9ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef94144720 con 0x7fef940b4b40 2024-03-06T07:28:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.253+0000 7fef9ae00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef940b3420 con 0x7fef940cee10 2024-03-06T07:28:46.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.253+0000 7fef9ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef940b3c20 con 0x7fef94144340 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 0x7fef9413eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 0x7fef9413eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42844/0 (socket says 172.21.15.6:42844) 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 -- 172.21.15.6:0/1520435978 learned_addr learned my addr 172.21.15.6:0/1520435978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99000640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef940cee10 0x7fef9413b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef9a400640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef94144340 0x7fef94142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 -- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef94144340 msgr2=0x7fef94142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef94144340 0x7fef94142380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 -- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef940cee10 msgr2=0x7fef9413b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef940cee10 0x7fef9413b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.254+0000 7fef99a00640 1 -- 172.21.15.6:0/1520435978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9413d7b0 con 0x7fef940b4b40 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.255+0000 7fef99a00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 0x7fef9413eaf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fef88000c00 tx=0x7fef880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.255+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88040070 con 0x7fef940b4b40 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.255+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef880399f0 con 0x7fef940b4b40 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.255+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88035040 con 0x7fef940b4b40 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.255+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9413da50 con 0x7fef940b4b40 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.255+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9413e010 con 0x7fef940b4b40 2024-03-06T07:28:46.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.256+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9404f330 con 0x7fef940b4b40 2024-03-06T07:28:46.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.258+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef8804b060 con 0x7fef940b4b40 2024-03-06T07:28:46.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.259+0000 7fef91600640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef7406b4d0 0x7fef7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.260+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef880aba90 con 0x7fef940b4b40 2024-03-06T07:28:46.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.260+0000 7fef99000640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef7406b4d0 0x7fef7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.261+0000 7fef99000640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef7406b4d0 0x7fef7406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fef9415ba80 tx=0x7fef80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.264+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef8807a870 con 0x7fef940b4b40 2024-03-06T07:28:46.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.538+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fef9406e6e0 con 0x7fef940b4b40 2024-03-06T07:28:46.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.544+0000 7fef91600640 1 -- 172.21.15.6:0/1520435978 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7fef8807a170 con 0x7fef940b4b40 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:46.546 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef7406b4d0 msgr2=0x7fef7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef7406b4d0 0x7fef7406d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fef9415ba80 tx=0x7fef80008040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 msgr2=0x7fef9413eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 0x7fef9413eaf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fef88000c00 tx=0x7fef880046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 shutdown_connections 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef7406b4d0 0x7fef7406d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef940cee10 0x7fef9413b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef94144340 0x7fef94142380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 --2- 172.21.15.6:0/1520435978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef940b4b40 0x7fef9413eaf0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 >> 172.21.15.6:0/1520435978 conn(0x7fef9406b190 msgr2=0x7fef940717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 shutdown_connections 2024-03-06T07:28:46.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.547+0000 7fef9ae00640 1 -- 172.21.15.6:0/1520435978 wait complete. 2024-03-06T07:28:46.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:46.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.786+0000 7fef32000640 1 Processor -- start 2024-03-06T07:28:46.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.786+0000 7fef32000640 1 -- start start 2024-03-06T07:28:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.787+0000 7fef32000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef2c161670 0x7fef2c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.787+0000 7fef32000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c06de00 con 0x7fef2c0ae0a0 2024-03-06T07:28:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef32000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c0ce9a0 con 0x7fef2c06e240 2024-03-06T07:28:46.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef32000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2c0730d0 con 0x7fef2c161670 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef2be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef2c0ae0a0 0x7fef2c06d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48258/0 (socket says 172.21.15.6:48258) 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef2be00640 1 -- 172.21.15.6:0/3356943040 learned_addr learned my addr 172.21.15.6:0/3356943040 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef2b400640 1 --2- 172.21.15.6:0/3356943040 >> v2:172.21.15.178:3300/0 conn(0x7fef2c161670 0x7fef2c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865417846 0 0) 0x7fef2c06de00 con 0x7fef2c0ae0a0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.788+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef10003520 con 0x7fef2c0ae0a0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243174289 0 0) 0x7fef10003520 con 0x7fef2c0ae0a0 2024-03-06T07:28:46.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 >> v2:172.21.15.178:3300/0 conn(0x7fef2c161670 msgr2=0x7fef2c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 --2- 172.21.15.6:0/3356943040 >> v2:172.21.15.178:3300/0 conn(0x7fef2c161670 0x7fef2c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 >> v1:172.21.15.178:6789/0 conn(0x7fef2c06e240 legacy=0x7fef2c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0cec80 con 0x7fef2c0ae0a0 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fef24001670 con 0x7fef2c0ae0a0 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 >> v1:172.21.15.6:6789/0 conn(0x7fef2c0ae0a0 legacy=0x7fef2c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef10003a20 0x7fef10003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef10005230 0x7fef1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0730d0 con 0x7fef10005230 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c190 con 0x7fef10004800 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1000c390 con 0x7fef10003a20 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.789+0000 7fef2be00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef10005230 0x7fef1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef22a00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef2b400640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef10003a20 0x7fef10003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef2be00640 1 -- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef10003a20 msgr2=0x7fef10003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef2be00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef10003a20 0x7fef10003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef2be00640 1 -- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10004800 msgr2=0x7fef10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef2be00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10004800 0x7fef10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.790+0000 7fef2be00640 1 -- 172.21.15.6:0/3356943040 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10003520 con 0x7fef10005230 2024-03-06T07:28:46.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.791+0000 7fef2be00640 1 -- 172.21.15.6:0/3356943040 reap_dead start 2024-03-06T07:28:46.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.792+0000 7fef2be00640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef10005230 0x7fef1000b9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fef2400c240 tx=0x7fef10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2100a611df2b9ab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.792+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24007cf0 con 0x7fef10005230 2024-03-06T07:28:46.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.792+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef24030070 con 0x7fef10005230 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.792+0000 7fef2aa00640 1 -- 172.21.15.6:0/3356943040 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24005d30 con 0x7fef10005230 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 -- 172.21.15.6:0/3356943040 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef10005230 msgr2=0x7fef1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef10005230 0x7fef1000b9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fef2400c240 tx=0x7fef10005830 comp rx=0 tx=0).stop 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 -- 172.21.15.6:0/3356943040 shutdown_connections 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 --2- 172.21.15.6:0/3356943040 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef10005230 0x7fef1000b9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 -- 172.21.15.6:0/3356943040 >> 172.21.15.6:0/3356943040 conn(0x7fef2c06afb0 msgr2=0x7fef2c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 -- 172.21.15.6:0/3356943040 shutdown_connections 2024-03-06T07:28:46.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.793+0000 7fef32000640 1 -- 172.21.15.6:0/3356943040 wait complete. 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.795+0000 7fef32000640 1 Processor -- start 2024-03-06T07:28:46.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.795+0000 7fef32000640 1 -- start start 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef32000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10005230 0x7fef2c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef32000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef2c158d90 0x7fef2c1591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef32000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 0x7fef2c0ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef32000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0cf520 con 0x7fef2c0ca390 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef32000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0cf240 con 0x7fef10005230 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef32000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0ce600 con 0x7fef2c158d90 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 0x7fef2c0ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 0x7fef2c0ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42866/0 (socket says 172.21.15.6:42866) 2024-03-06T07:28:46.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 -- 172.21.15.6:0/381397884 learned_addr learned my addr 172.21.15.6:0/381397884 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2b400640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10005230 0x7fef2c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef22a00640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef2c158d90 0x7fef2c1591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 -- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef2c158d90 msgr2=0x7fef2c1591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef2c158d90 0x7fef2c1591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 -- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10005230 msgr2=0x7fef2c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10005230 0x7fef2c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef2be00640 1 -- 172.21.15.6:0/381397884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0cb030 con 0x7fef2c0ca390 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.796+0000 7fef22a00640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef2c158d90 0x7fef2c1591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.797+0000 7fef2be00640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 0x7fef2c0ca810 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fef2402c480 tx=0x7fef2c0cee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.797+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24001c30 con 0x7fef2c0ca390 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.797+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef24000ed0 con 0x7fef2c0ca390 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.797+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c0cb310 con 0x7fef2c0ca390 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.797+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef24004e30 con 0x7fef2c0ca390 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.797+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c0cb8d0 con 0x7fef2c0ca390 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.798+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c069510 con 0x7fef2c0ca390 2024-03-06T07:28:46.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.800+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef24007070 con 0x7fef2c0ca390 2024-03-06T07:28:46.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.800+0000 7fef23e00640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef0806b500 0x7fef0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.801+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef240ab4a0 con 0x7fef2c0ca390 2024-03-06T07:28:46.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.801+0000 7fef2b400640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef0806b500 0x7fef0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.802+0000 7fef2b400640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef0806b500 0x7fef0806d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009fd0 tx=0x7fef1c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:46.810+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef2407a3b0 con 0x7fef2c0ca390 2024-03-06T07:28:47.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.050+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2c04f3a0 con 0x7fef2c0ca390 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.051+0000 7fef23e00640 1 -- 172.21.15.6:0/381397884 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fef24079cb0 con 0x7fef2c0ca390 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:28:47.054 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef0806b500 msgr2=0x7fef0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef0806b500 0x7fef0806d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009fd0 tx=0x7fef1c00b040 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 msgr2=0x7fef2c0ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 0x7fef2c0ca810 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fef2402c480 tx=0x7fef2c0cee50 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 shutdown_connections 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef0806b500 0x7fef0806d9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef2c0ca390 0x7fef2c0ca810 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef2c158d90 0x7fef2c1591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 --2- 172.21.15.6:0/381397884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef10005230 0x7fef2c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 >> 172.21.15.6:0/381397884 conn(0x7fef2c06afb0 msgr2=0x7fef2c0b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.054+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 shutdown_connections 2024-03-06T07:28:47.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.055+0000 7fef32000640 1 -- 172.21.15.6:0/381397884 wait complete. 2024-03-06T07:28:47.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:40.283024+0000', 'modified': '2024-03-06T07:28:43.390057+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5124}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5124': {'gid': 5124, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/2554316293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 2554316293}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 2554316293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5124, 'qdb_cluster': [5124]} 2024-03-06T07:28:47.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:47.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:47.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.701+0000 7f84a4a00640 1 Processor -- start 2024-03-06T07:28:47.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.701+0000 7f84a4a00640 1 -- start start 2024-03-06T07:28:47.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.702+0000 7f84a4a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84a00d2700 0x7f84a00c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.702+0000 7f84a4a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00c4ff0 0x7f84a00af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.702+0000 7f84a4a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a00c5400 con 0x7f84a0162410 2024-03-06T07:28:47.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.702+0000 7f84a4a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a00cf450 con 0x7f84a00d2700 2024-03-06T07:28:47.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.702+0000 7f84a4a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00cebc0 con 0x7f84a00c4ff0 2024-03-06T07:28:47.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.703+0000 7f849e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f84a0162410 0x7f84a00ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48272/0 (socket says 172.21.15.6:48272) 2024-03-06T07:28:47.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.703+0000 7f849e800640 1 -- 172.21.15.6:0/451492896 learned_addr learned my addr 172.21.15.6:0/451492896 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:47.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.703+0000 7f849f200640 1 --2- 172.21.15.6:0/451492896 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00c4ff0 0x7f84a00af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.703+0000 7f8495e00640 1 --2- 172.21.15.6:0/451492896 >> v2:172.21.15.178:3300/0 conn(0x7f84a00d2700 0x7f84a00c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.703+0000 7f849de00640 1 -- 172.21.15.6:0/451492896 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091112728 0 0) 0x7f8490001490 con 0x7f84a0162410 2024-03-06T07:28:47.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.703+0000 7f849de00640 1 -- 172.21.15.6:0/451492896 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484003520 con 0x7f84a0162410 2024-03-06T07:28:47.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.704+0000 7f849f200640 1 -- 172.21.15.6:0/451492896 >> v2:172.21.15.178:3300/0 conn(0x7f84a00d2700 msgr2=0x7f84a00c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.704+0000 7f849f200640 1 --2- 172.21.15.6:0/451492896 >> v2:172.21.15.178:3300/0 conn(0x7f84a00d2700 0x7f84a00c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.704+0000 7f849f200640 1 -- 172.21.15.6:0/451492896 >> v1:172.21.15.6:6789/0 conn(0x7f84a0162410 legacy=0x7f84a00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.704+0000 7f849f200640 1 -- 172.21.15.6:0/451492896 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00cee90 con 0x7f84a00c4ff0 2024-03-06T07:28:47.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.705+0000 7f849f200640 1 --2- 172.21.15.6:0/451492896 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00c4ff0 0x7f84a00af1e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84980097a0 tx=0x7f849802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddd13c59b060f409 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f849de00640 1 -- 172.21.15.6:0/451492896 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498031780 con 0x7f84a00c4ff0 2024-03-06T07:28:47.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f849de00640 1 -- 172.21.15.6:0/451492896 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498037070 con 0x7f84a00c4ff0 2024-03-06T07:28:47.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f849de00640 1 -- 172.21.15.6:0/451492896 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498031b40 con 0x7f84a00c4ff0 2024-03-06T07:28:47.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 -- 172.21.15.6:0/451492896 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00c4ff0 msgr2=0x7f84a00af1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 --2- 172.21.15.6:0/451492896 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00c4ff0 0x7f84a00af1e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84980097a0 tx=0x7f849802b820 comp rx=0 tx=0).stop 2024-03-06T07:28:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 -- 172.21.15.6:0/451492896 shutdown_connections 2024-03-06T07:28:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 --2- 172.21.15.6:0/451492896 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00c4ff0 0x7f84a00af1e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 --2- 172.21.15.6:0/451492896 >> v2:172.21.15.178:3300/0 conn(0x7f84a00d2700 0x7f84a00c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 -- 172.21.15.6:0/451492896 >> 172.21.15.6:0/451492896 conn(0x7f84a00d1ae0 msgr2=0x7f84a00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 -- 172.21.15.6:0/451492896 shutdown_connections 2024-03-06T07:28:47.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.706+0000 7f84a4a00640 1 -- 172.21.15.6:0/451492896 wait complete. 2024-03-06T07:28:47.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.709+0000 7f84a4a00640 1 Processor -- start 2024-03-06T07:28:47.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.709+0000 7f84a4a00640 1 -- start start 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.710+0000 7f84a4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 0x7f84a00c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.710+0000 7f84a4a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00d2700 0x7f84a0156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.710+0000 7f84a4a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84a0162410 0x7f84a015ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.710+0000 7f84a4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00aecc0 con 0x7f84a00c4ff0 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.710+0000 7f84a4a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00ae9e0 con 0x7f84a0162410 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.710+0000 7f84a4a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00af720 con 0x7f84a00d2700 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 0x7f84a00c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 0x7f84a00c5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42870/0 (socket says 172.21.15.6:42870) 2024-03-06T07:28:47.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 -- 172.21.15.6:0/1229164059 learned_addr learned my addr 172.21.15.6:0/1229164059 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f8495e00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00d2700 0x7f84a0156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849f200640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84a0162410 0x7f84a015ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 -- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00d2700 msgr2=0x7f84a0156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00d2700 0x7f84a0156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 -- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84a0162410 msgr2=0x7f84a015ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84a0162410 0x7f84a015ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.711+0000 7f849e800640 1 -- 172.21.15.6:0/1229164059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00c8d50 con 0x7f84a00c4ff0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.712+0000 7f849e800640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 0x7f84a00c5a00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f849000cb10 tx=0x7f849000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.712+0000 7f8497400640 1 -- 172.21.15.6:0/1229164059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490017070 con 0x7f84a00c4ff0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.712+0000 7f8497400640 1 -- 172.21.15.6:0/1229164059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849000ac10 con 0x7f84a00c4ff0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.712+0000 7f8497400640 1 -- 172.21.15.6:0/1229164059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490012400 con 0x7f84a00c4ff0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.712+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a00c9030 con 0x7f84a00c4ff0 2024-03-06T07:28:47.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.712+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a015d360 con 0x7f84a00c4ff0 2024-03-06T07:28:47.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.716+0000 7f8497400640 1 -- 172.21.15.6:0/1229164059 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f849001a070 con 0x7f84a00c4ff0 2024-03-06T07:28:47.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.716+0000 7f84a81f1740 1 Processor -- start 2024-03-06T07:28:47.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.716+0000 7f84a81f1740 1 -- start start 2024-03-06T07:28:47.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f8497400640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f8497400640 1 -- 172.21.15.6:0/1229164059 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8490089f00 con 0x7f84a00c4ff0 2024-03-06T07:28:47.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f849f200640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42af930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42df720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 0x564ca42dfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f84a81f1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ca4219600 con 0x564ca42afe70 2024-03-06T07:28:47.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ca41c2930 con 0x564ca42dc990 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.717+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564ca42156a0 con 0x564ca42dcd70 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 0x564ca42dfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 0x564ca42dfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42874/0 (socket says 172.21.15.6:42874) 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 -- 172.21.15.6:0/3744485397 learned_addr learned my addr 172.21.15.6:0/3744485397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849e800640 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f8495e00640 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42df720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 -- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 msgr2=0x564ca42df720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42df720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 -- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 msgr2=0x564ca42af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.718+0000 7f849f200640 1 -- 172.21.15.6:0/3744485397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ca3f2e2d0 con 0x564ca42afe70 2024-03-06T07:28:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.719+0000 7f849f200640 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 0x564ca42dfc60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f849802b860 tx=0x7f8498004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f249bb72443f9143 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.719+0000 7f848f000640 1 -- 172.21.15.6:0/3744485397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498041030 con 0x564ca42afe70 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.719+0000 7f848f000640 1 -- 172.21.15.6:0/3744485397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498037040 con 0x564ca42afe70 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.719+0000 7f848f000640 1 -- 172.21.15.6:0/3744485397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498044040 con 0x564ca42afe70 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 -- 172.21.15.6:0/3744485397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 msgr2=0x564ca42dfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 0x564ca42dfc60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f849802b860 tx=0x7f8498004730 comp rx=0 tx=0).stop 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f849f200640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f847c06b4d0 0x7f847c06d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f84980098c0 tx=0x7f849803d000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 -- 172.21.15.6:0/3744485397 shutdown_connections 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42afe70 0x564ca42dfc60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42df720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3744485397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 -- 172.21.15.6:0/3744485397 >> 172.21.15.6:0/3744485397 conn(0x564ca42b1ba0 msgr2=0x564ca42fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 -- 172.21.15.6:0/3744485397 shutdown_connections 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.720+0000 7f84a81f1740 1 -- 172.21.15.6:0/3744485397 wait complete. 2024-03-06T07:28:47.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.721+0000 7f84a81f1740 1 -- start start 2024-03-06T07:28:47.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.721+0000 7f84a81f1740 1 Processor -- start 2024-03-06T07:28:47.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42e1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 0x564ca42e2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ca42b4140 con 0x564ca42fab20 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ca41d5d90 con 0x564ca42dc990 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564ca42b3530 con 0x564ca42dcd70 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f84a81f1740 10 client.0 _ll_register_callbacks cb 0x564ca4211350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f849f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 0x564ca42e2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f849f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 0x564ca42e2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42890/0 (socket says 172.21.15.6:42890) 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f849f200640 1 -- 172.21.15.6:0/3313832199 learned_addr learned my addr 172.21.15.6:0/3313832199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f849e800640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.722+0000 7f8495e00640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42e1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849f200640 1 -- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 msgr2=0x564ca42e1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849f200640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42e1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849f200640 1 -- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 msgr2=0x564ca42cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849f200640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849f200640 1 -- 172.21.15.6:0/3313832199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ca3ff86b0 con 0x564ca42fab20 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849e800640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:47.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f849f200640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 0x564ca42e2100 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f849804a000 tx=0x7f8498008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1bc056fd726d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f8483400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f8483400640 1 -- 172.21.15.6:0/3313832199 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8498041070 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f8483400640 1 -- 172.21.15.6:0/3313832199 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8498031e40 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.723+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8498037040 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f8483400640 1 -- 172.21.15.6:0/3313832199 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8498044040 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 10 client.5274 fetch_fsmap learned FSMap version 17 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564ca42cb8f0 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f8483400640 1 -- 172.21.15.6:0/3313832199 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8498037040 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f8483400640 1 -- 172.21.15.6:0/3313832199 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8498005990 con 0x564ca42fab20 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 10 client.5274 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 10 client.5274 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 20 client.5274 populate_metadata read hostname 'smithi006' 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca42f1890 0x564ca42cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 4 client.5274 mds_command: new command op to 5124 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:47.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.724+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 --> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ca4226b40 con 0x564ca42f1890 2024-03-06T07:28:47.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.725+0000 7f849e800640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca42f1890 0x564ca42cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.725+0000 7f849e800640 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca42f1890 0x564ca42cc3e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d5949cf0c9598c6 server_cookie=950377f3d9652ea in_seq=0 out_seq=0 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.725+0000 7f8483400640 10 client.5274 ms_handle_connect on v2:172.21.15.6:6833/2554316293 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f8483400640 1 -- 172.21.15.6:0/3313832199 <== mds.0 v2:172.21.15.6:6833/2554316293 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564ca4226b40 con 0x564ca42f1890 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f8483400640 10 client.5274 handle_command_reply: tid=0 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 client.5274 shutdown 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 msgr2=0x564ca42e2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 0x564ca42e2100 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f849804a000 tx=0x7f8498008030 comp rx=0 tx=0).stop 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 shutdown_connections 2024-03-06T07:28:47.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca42f1890 0x564ca42cc3e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42fab20 0x564ca42e2100 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dcd70 0x564ca42e1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3313832199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42dc990 0x564ca42cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 >> 172.21.15.6:0/3313832199 conn(0x564ca42afe70 msgr2=0x564ca42df710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f8483400640 0 client.5274 ms_handle_reset on v2:172.21.15.6:6833/2554316293 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.726+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 shutdown_connections 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.727+0000 7f84a81f1740 1 -- 172.21.15.6:0/3313832199 wait complete. 2024-03-06T07:28:47.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.727+0000 7f84a81f1740 20 client.5274 trim_cache size 0 max 16384 2024-03-06T07:28:47.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.745+0000 7f84a81f1740 1 Processor -- start 2024-03-06T07:28:47.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.745+0000 7f84a81f1740 1 -- start start 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42aec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca42cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f84a81f1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00ca380 con 0x564ca42dcd70 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a01fc7e0 con 0x564ca42fab20 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00c9d80 con 0x564ca42dc990 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f8495e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f8495e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42894/0 (socket says 172.21.15.6:42894) 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.746+0000 7f8495e00640 1 -- 172.21.15.6:0/2869006619 learned_addr learned my addr 172.21.15.6:0/2869006619 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f849e800640 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f8495e00640 1 -- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 msgr2=0x564ca42aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f8495e00640 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f8495e00640 1 -- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 msgr2=0x564ca42cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f8495e00640 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca42cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f8495e00640 1 -- 172.21.15.6:0/2869006619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ca42b1300 con 0x564ca42dcd70 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.747+0000 7f8495e00640 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42cc3b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8488009a90 tx=0x7f8488009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e495413073846620 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f848e600640 1 -- 172.21.15.6:0/2869006619 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848801b070 con 0x564ca42dcd70 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f848e600640 1 -- 172.21.15.6:0/2869006619 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488015c60 con 0x564ca42dcd70 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f848e600640 1 -- 172.21.15.6:0/2869006619 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488020460 con 0x564ca42dcd70 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 -- 172.21.15.6:0/2869006619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 msgr2=0x564ca42cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42cc3b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8488009a90 tx=0x7f8488009190 comp rx=0 tx=0).stop 2024-03-06T07:28:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 -- 172.21.15.6:0/2869006619 shutdown_connections 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca42cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42cc3b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 --2- 172.21.15.6:0/2869006619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42aec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 -- 172.21.15.6:0/2869006619 >> 172.21.15.6:0/2869006619 conn(0x564ca42afe70 msgr2=0x564ca42d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 -- 172.21.15.6:0/2869006619 shutdown_connections 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.748+0000 7f84a81f1740 1 -- 172.21.15.6:0/2869006619 wait complete. 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.749+0000 7f84a81f1740 1 -- start start 2024-03-06T07:28:47.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.749+0000 7f84a81f1740 1 Processor -- start 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42fcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca431d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ca42f1710 con 0x564ca42dcd70 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ca42b4140 con 0x564ca42fab20 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564ca42f1d10 con 0x564ca42dc990 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.750+0000 7f84a81f1740 10 client.0 _ll_register_callbacks cb 0x7f8488009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42fcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42fcb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42908/0 (socket says 172.21.15.6:42908) 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 -- 172.21.15.6:0/3044099723 learned_addr learned my addr 172.21.15.6:0/3044099723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f849f200640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca431d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f849e800640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 -- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 msgr2=0x564ca42fc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42fc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 -- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 msgr2=0x564ca431d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca431d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 -- 172.21.15.6:0/3044099723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ca42b6d60 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8495e00640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42fcb30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f848800a010 tx=0x7f8488004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f73d2243c048280 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.751+0000 7f8483400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f8483400640 1 -- 172.21.15.6:0/3044099723 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8488031060 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f8483400640 1 -- 172.21.15.6:0/3044099723 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8488020870 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8488015c60 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f8483400640 1 -- 172.21.15.6:0/3044099723 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8488037050 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f84a81f1740 10 client.5280 fetch_fsmap learned FSMap version 17 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x564ca42aecc0 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.752+0000 7f8483400640 1 -- 172.21.15.6:0/3044099723 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8488015c60 con 0x564ca42dcd70 2024-03-06T07:28:47.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f8483400640 1 -- 172.21.15.6:0/3044099723 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 17) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f848803a020 con 0x564ca42dcd70 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f84a81f1740 10 client.5280 fetch_fsmap finished waiting for FSMap version 17 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f84a81f1740 10 client.5280 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f84a81f1740 20 client.5280 populate_metadata read hostname 'smithi006' 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca431e3e0 0x564ca436a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f84a81f1740 4 client.5280 mds_command: new command op to 5124 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 --> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564ca4220e90 con 0x564ca431e3e0 2024-03-06T07:28:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f849e800640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca431e3e0 0x564ca436a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f849e800640 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca431e3e0 0x564ca436a5b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad20f39b6831bc3a server_cookie=4d76bbf3a68144f1 in_seq=0 out_seq=0 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.753+0000 7f8483400640 10 client.5280 ms_handle_connect on v2:172.21.15.6:6833/2554316293 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f8483400640 1 -- 172.21.15.6:0/3044099723 <== mds.0 v2:172.21.15.6:6833/2554316293 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x564ca4220e90 con 0x564ca431e3e0 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f8483400640 10 client.5280 handle_command_reply: tid=0 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 client.5280 shutdown 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 msgr2=0x564ca42fcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42fcb30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f848800a010 tx=0x7f8488004730 comp rx=0 tx=0).stop 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 shutdown_connections 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:6833/2554316293,v1:172.21.15.6:6835/2554316293] conn(0x564ca431e3e0 0x564ca436a5b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564ca42fab20 0x564ca431d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564ca42dcd70 0x564ca42fcb30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 --2- 172.21.15.6:0/3044099723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564ca42dc990 0x564ca42fc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f8483400640 0 client.5280 ms_handle_reset on v2:172.21.15.6:6833/2554316293 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 >> 172.21.15.6:0/3044099723 conn(0x564ca42afe70 msgr2=0x564ca42cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 shutdown_connections 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 1 -- 172.21.15.6:0/3044099723 wait complete. 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.754+0000 7f84a81f1740 20 client.5280 trim_cache size 0 max 16384 2024-03-06T07:28:47.759 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5124, 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.9030078509999999, 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 17, 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 35, 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 35, 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 15.756696372 2024-03-06T07:28:47.760 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f847c06b4d0 msgr2=0x7f847c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f847c06b4d0 0x7f847c06d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f84980098c0 tx=0x7f849803d000 comp rx=0 tx=0).stop 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 msgr2=0x7f84a00c5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 0x7f84a00c5a00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f849000cb10 tx=0x7f849000a060 comp rx=0 tx=0).stop 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 shutdown_connections 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f847c06b4d0 0x7f847c06d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84a0162410 0x7f84a015ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84a00d2700 0x7f84a0156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 --2- 172.21.15.6:0/1229164059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84a00c4ff0 0x7f84a00c5a00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 >> 172.21.15.6:0/1229164059 conn(0x7f84a00d1ae0 msgr2=0x7f84a006fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 shutdown_connections 2024-03-06T07:28:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.755+0000 7f84a4a00640 1 -- 172.21.15.6:0/1229164059 wait complete. 2024-03-06T07:28:47.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.983+0000 7fa23b400640 1 Processor -- start 2024-03-06T07:28:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.983+0000 7fa23b400640 1 -- start start 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.985+0000 7fa23b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234070ee0 0x7fa234162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.985+0000 7fa23b400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa2340712c0 0x7fa23406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.985+0000 7fa23b400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2340ce090 con 0x7fa234072da0 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.985+0000 7fa23b400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2340ceca0 con 0x7fa2340712c0 2024-03-06T07:28:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.985+0000 7fa23b400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2340ce370 con 0x7fa234070ee0 2024-03-06T07:28:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.986+0000 7fa239400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa2340712c0 0x7fa23406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.986+0000 7fa238a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234070ee0 0x7fa234162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.986+0000 7fa239400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa2340712c0 0x7fa23406e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:53798/0 (socket says 172.21.15.6:53798) 2024-03-06T07:28:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.986+0000 7fa239400640 1 -- 172.21.15.6:0/2031652908 learned_addr learned my addr 172.21.15.6:0/2031652908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.987+0000 7fa239400640 1 -- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234070ee0 msgr2=0x7fa234162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.987+0000 7fa239400640 1 --2- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234070ee0 0x7fa234162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.987+0000 7fa239400640 1 -- 172.21.15.6:0/2031652908 >> v1:172.21.15.178:6789/0 conn(0x7fa234072da0 legacy=0x7fa23406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.987+0000 7fa239400640 1 -- 172.21.15.6:0/2031652908 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2340ce650 con 0x7fa2340712c0 2024-03-06T07:28:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.987+0000 7fa238a00640 1 --2- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234070ee0 0x7fa234162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.988+0000 7fa239400640 1 --2- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa23406e7d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009e50 tx=0x7fa22c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8889bfcdf4fa3325 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa233400640 1 -- 172.21.15.6:0/2031652908 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c02cb00 con 0x7fa2340712c0 2024-03-06T07:28:47.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa233400640 1 -- 172.21.15.6:0/2031652908 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22c02cca0 con 0x7fa2340712c0 2024-03-06T07:28:47.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa233400640 1 -- 172.21.15.6:0/2031652908 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c034610 con 0x7fa2340712c0 2024-03-06T07:28:47.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 -- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 msgr2=0x7fa23406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 --2- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa23406e7d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009e50 tx=0x7fa22c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 -- 172.21.15.6:0/2031652908 shutdown_connections 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 --2- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa23406e7d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 --2- 172.21.15.6:0/2031652908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234070ee0 0x7fa234162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 -- 172.21.15.6:0/2031652908 >> 172.21.15.6:0/2031652908 conn(0x7fa23406afb0 msgr2=0x7fa23406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.989+0000 7fa23b400640 1 -- 172.21.15.6:0/2031652908 shutdown_connections 2024-03-06T07:28:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.990+0000 7fa23b400640 1 -- 172.21.15.6:0/2031652908 wait complete. 2024-03-06T07:28:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.992+0000 7fa23b400640 1 Processor -- start 2024-03-06T07:28:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.992+0000 7fa23b400640 1 -- start start 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.993+0000 7fa23b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 0x7fa234152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.993+0000 7fa23b400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa2340b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.993+0000 7fa23b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234072da0 0x7fa234162020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.993+0000 7fa23b400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2340cdda0 con 0x7fa234070ee0 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.993+0000 7fa23b400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2340cdc20 con 0x7fa2340712c0 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.993+0000 7fa23b400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2340cf130 con 0x7fa234072da0 2024-03-06T07:28:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 0x7fa234152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 0x7fa234152980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42924/0 (socket says 172.21.15.6:42924) 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 -- 172.21.15.6:0/3969793064 learned_addr learned my addr 172.21.15.6:0/3969793064 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa239400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234072da0 0x7fa234162020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa233e00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa2340b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 -- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234072da0 msgr2=0x7fa234162020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234072da0 0x7fa234162020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 -- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 msgr2=0x7fa2340b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa2340b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.994+0000 7fa238a00640 1 -- 172.21.15.6:0/3969793064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23409aab0 con 0x7fa234070ee0 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.995+0000 7fa238a00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 0x7fa234152980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa22800ebe0 tx=0x7fa22800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.995+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228017070 con 0x7fa234070ee0 2024-03-06T07:28:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.995+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22800cc30 con 0x7fa234070ee0 2024-03-06T07:28:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.995+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa23409adf0 con 0x7fa234070ee0 2024-03-06T07:28:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.995+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa228013470 con 0x7fa234070ee0 2024-03-06T07:28:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.995+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23415f7f0 con 0x7fa234070ee0 2024-03-06T07:28:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.996+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234069510 con 0x7fa234070ee0 2024-03-06T07:28:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.997+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa22801a070 con 0x7fa234070ee0 2024-03-06T07:28:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.999+0000 7fa230c00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa21406b4d0 0x7fa21406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.999+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa22808a250 con 0x7fa234070ee0 2024-03-06T07:28:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:47.999+0000 7fa233e00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa21406b4d0 0x7fa21406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.001+0000 7fa233e00640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa21406b4d0 0x7fa21406d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa220007fd0 tx=0x7fa220009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.003+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa23409adf0 con 0x7fa234070ee0 2024-03-06T07:28:48.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.268+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa23404f3a0 con 0x7fa234070ee0 2024-03-06T07:28:48.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.269+0000 7fa230c00640 1 -- 172.21.15.6:0/3969793064 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa228059040 con 0x7fa234070ee0 2024-03-06T07:28:48.271 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa21406b4d0 msgr2=0x7fa21406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa21406b4d0 0x7fa21406d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa220007fd0 tx=0x7fa220009040 comp rx=0 tx=0).stop 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 msgr2=0x7fa234152980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 0x7fa234152980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa22800ebe0 tx=0x7fa22800c520 comp rx=0 tx=0).stop 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 shutdown_connections 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa21406b4d0 0x7fa21406d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2340712c0 0x7fa2340b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa234072da0 0x7fa234162020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 --2- 172.21.15.6:0/3969793064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa234070ee0 0x7fa234152980 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.271+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 >> 172.21.15.6:0/3969793064 conn(0x7fa23406afb0 msgr2=0x7fa23406e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.272+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 shutdown_connections 2024-03-06T07:28:48.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:48.272+0000 7fa23b400640 1 -- 172.21.15.6:0/3969793064 wait complete. 2024-03-06T07:28:48.293 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:28:48.293 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:48.293 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:28:48.293 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:48.293 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.294 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:28:48.294 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:28:48.294 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:48.294 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:48.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.294 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:48.349 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42529sec preferred_lft 42529sec 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:48.350 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:28:48.351 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:48.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.352 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:48.437 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:28:48.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.442 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:28:48.494 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:28:48.494 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:48.494 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:28:48.494 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:28:48.495 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:48.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.495 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:28:48.495 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:28:48.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:28:48.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:28:48.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:28:48.495 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:28:48.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.691 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:28:48.741 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:48.741 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:48.742 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:48.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:28:48.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:48.849 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.902 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:48.902 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:48.902 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.947 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:28:48.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:49.122 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:49.122 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:49.156 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:28:49.157 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:28:49.157 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:28:49.157 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:28:49.157 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:28:49.159 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:28:49.159 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:28:49.159 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:28:49.159 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:28:49.159 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:28:49.163 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:28:49.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:49.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:49.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:49.242 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:28:49.304 INFO:teuthology.orchestra.run.smithi006.stdout:client5286 2024-03-06T07:28:49.305 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:28:49.305 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> subdir = "delete_me" 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> size = 1024 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> file_count = 1000 2024-03-06T07:28:49.357 DEBUG:teuthology.orchestra.run.smithi006:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:28:49.358 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(0, file_count): 2024-03-06T07:28:49.358 DEBUG:teuthology.orchestra.run.smithi006:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:28:49.358 DEBUG:teuthology.orchestra.run.smithi006:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:28:49.358 DEBUG:teuthology.orchestra.run.smithi006:> f.write(size * '"'"'x'"'"') 2024-03-06T07:28:49.358 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:28:50.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:50.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:51.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.000+0000 7fecdac00640 1 Processor -- start 2024-03-06T07:28:51.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.000+0000 7fecdac00640 1 -- start start 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecdac00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fecd415f450 0x7fecd4070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecdac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecdac00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fecd40d02e0 0x7fecd406ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecdac00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd40d29e0 con 0x7fecd415f450 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecdac00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd40cd0c0 con 0x7fecd40d02e0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecdac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40cd3a0 con 0x7fecd40700c0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecd9800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fecd415f450 0x7fecd4070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecd9800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fecd415f450 0x7fecd4070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42926/0 (socket says 172.21.15.6:42926) 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.002+0000 7fecd9800640 1 -- 172.21.15.6:0/3933208684 learned_addr learned my addr 172.21.15.6:0/3933208684 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecd8e00640 1 --2- 172.21.15.6:0/3933208684 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecda200640 1 --2- 172.21.15.6:0/3933208684 >> v2:172.21.15.178:3300/0 conn(0x7fecd40d02e0 0x7fecd406ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecd9800640 1 -- 172.21.15.6:0/3933208684 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 msgr2=0x7fecd40cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecd9800640 1 --2- 172.21.15.6:0/3933208684 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecd9800640 1 -- 172.21.15.6:0/3933208684 >> v2:172.21.15.178:3300/0 conn(0x7fecd40d02e0 msgr2=0x7fecd406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecd9800640 1 --2- 172.21.15.6:0/3933208684 >> v2:172.21.15.178:3300/0 conn(0x7fecd40d02e0 0x7fecd406ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecd9800640 1 -- 172.21.15.6:0/3933208684 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd40cd680 con 0x7fecd415f450 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.003+0000 7fecda200640 1 --2- 172.21.15.6:0/3933208684 >> v2:172.21.15.178:3300/0 conn(0x7fecd40d02e0 0x7fecd406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:51.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.004+0000 7fecd9800640 1 --2- 172.21.15.6:0/3933208684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd415f450 0x7fecd4070990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fecd406b9c0 tx=0x7fecc802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd234304d780515b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.004+0000 7fecd3e00640 1 -- 172.21.15.6:0/3933208684 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8035020 con 0x7fecd415f450 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.004+0000 7fecd3e00640 1 -- 172.21.15.6:0/3933208684 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc802d040 con 0x7fecd415f450 2024-03-06T07:28:51.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.004+0000 7fecd3e00640 1 -- 172.21.15.6:0/3933208684 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8031a80 con 0x7fecd415f450 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 -- 172.21.15.6:0/3933208684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd415f450 msgr2=0x7fecd4070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 --2- 172.21.15.6:0/3933208684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd415f450 0x7fecd4070990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fecd406b9c0 tx=0x7fecc802b900 comp rx=0 tx=0).stop 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 -- 172.21.15.6:0/3933208684 shutdown_connections 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 --2- 172.21.15.6:0/3933208684 >> v2:172.21.15.178:3300/0 conn(0x7fecd40d02e0 0x7fecd406ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 --2- 172.21.15.6:0/3933208684 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 --2- 172.21.15.6:0/3933208684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd415f450 0x7fecd4070990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 -- 172.21.15.6:0/3933208684 >> 172.21.15.6:0/3933208684 conn(0x7fecd40d1ae0 msgr2=0x7fecd40d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 -- 172.21.15.6:0/3933208684 shutdown_connections 2024-03-06T07:28:51.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.005+0000 7fecdac00640 1 -- 172.21.15.6:0/3933208684 wait complete. 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.007+0000 7fecdac00640 1 Processor -- start 2024-03-06T07:28:51.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.007+0000 7fecdac00640 1 -- start start 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.008+0000 7fecdac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.008+0000 7fecdac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 0x7fecd40c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.008+0000 7fecdac00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecd415f450 0x7fecd414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.008+0000 7fecdac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd415f830 con 0x7fecd40d02e0 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.008+0000 7fecdac00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40ae8b0 con 0x7fecd415f450 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.008+0000 7fecdac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd415f9b0 con 0x7fecd40700c0 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 0x7fecd40c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 0x7fecd40c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42936/0 (socket says 172.21.15.6:42936) 2024-03-06T07:28:51.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 -- 172.21.15.6:0/3755358572 learned_addr learned my addr 172.21.15.6:0/3755358572 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd9800640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecda200640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecd415f450 0x7fecd414f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 -- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 msgr2=0x7fecd40ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 -- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecd415f450 msgr2=0x7fecd414f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecd415f450 0x7fecd414f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 -- 172.21.15.6:0/3755358572 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4150330 con 0x7fecd40d02e0 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd9800640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecda200640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecd415f450 0x7fecd414f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.009+0000 7fecd8e00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 0x7fecd40c7a30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fecc000d800 tx=0x7fecc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.010+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0018030 con 0x7fecd40d02e0 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.010+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc00096a0 con 0x7fecd40d02e0 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.010+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0009870 con 0x7fecd40d02e0 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.010+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd40c92f0 con 0x7fecd40d02e0 2024-03-06T07:28:51.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.010+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd40c9b90 con 0x7fecd40d02e0 2024-03-06T07:28:51.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.011+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd40c96e0 con 0x7fecd40d02e0 2024-03-06T07:28:51.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.013+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecc001b070 con 0x7fecd40d02e0 2024-03-06T07:28:51.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.014+0000 7fecd1600640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecb406b4d0 0x7fecb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:51.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.014+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecc008a460 con 0x7fecd40d02e0 2024-03-06T07:28:51.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.015+0000 7fecd9800640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecb406b4d0 0x7fecb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:51.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.016+0000 7fecd9800640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecb406b4d0 0x7fecb406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fecc8036000 tx=0x7fecc803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:51.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.019+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecc00592c0 con 0x7fecd40d02e0 2024-03-06T07:28:51.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:51.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:51.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.384+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecd40c87b0 con 0x7fecd40d02e0 2024-03-06T07:28:51.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.385+0000 7fecd1600640 1 -- 172.21.15.6:0/3755358572 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fecc0058bc0 con 0x7fecd40d02e0 2024-03-06T07:28:51.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:51.387 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:28:51.388 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.387+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecb406b4d0 msgr2=0x7fecb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.387+0000 7fecdac00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecb406b4d0 0x7fecb406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fecc8036000 tx=0x7fecc803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.387+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 msgr2=0x7fecd40c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.387+0000 7fecdac00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 0x7fecd40c7a30 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fecc000d800 tx=0x7fecc000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 shutdown_connections 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecb406b4d0 0x7fecb406d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fecd415f450 0x7fecd414f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fecd40d02e0 0x7fecd40c7a30 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 --2- 172.21.15.6:0/3755358572 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fecd40700c0 0x7fecd40ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 >> 172.21.15.6:0/3755358572 conn(0x7fecd40d1ae0 msgr2=0x7fecd415bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:51.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 shutdown_connections 2024-03-06T07:28:51.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:51.388+0000 7fecdac00640 1 -- 172.21.15.6:0/3755358572 wait complete. 2024-03-06T07:28:51.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:52.070 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:28:52.071 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:52.071 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:28:52.071 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:28:52.072 DEBUG:teuthology.orchestra.run.smithi006:> import stat 2024-03-06T07:28:52.072 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:28:52.072 DEBUG:teuthology.orchestra.run.smithi006:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:28:52.072 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:28:52.181 INFO:teuthology.orchestra.run.smithi006.stdout:1099511627776 2024-03-06T07:28:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:52.194 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:28:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.329+0000 7fe3c3439f40 1 Processor -- start 2024-03-06T07:28:52.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.329+0000 7fe3c3439f40 1 -- start start 2024-03-06T07:28:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.330+0000 7fe3c3439f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d11e0 0x562109863a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.330+0000 7fe3c3439f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 0x562109871900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.330+0000 7fe3c3439f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 0x562109871e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.330+0000 7fe3c3439f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621097efbb0 con 0x5621097d2a90 2024-03-06T07:28:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.330+0000 7fe3c3439f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56210974cfd0 con 0x5621097d11e0 2024-03-06T07:28:52.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.330+0000 7fe3c3439f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5621097a9620 con 0x56210987b330 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 0x562109871900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 0x562109871900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57172/0 (socket says 172.21.15.6:57172) 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 -- 172.21.15.6:0/234080809 learned_addr learned my addr 172.21.15.6:0/234080809 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3c0800640 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 0x562109871e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 -- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 msgr2=0x562109871e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 0x562109871e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 -- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d11e0 msgr2=0x562109863a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d11e0 0x562109863a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 -- 172.21.15.6:0/234080809 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621097d5220 con 0x5621097d2a90 2024-03-06T07:28:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.331+0000 7fe3bf400640 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 0x562109871900 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0097c0 tx=0x7fe3ac009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e1c1d2f972b0be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3be000640 1 -- 172.21.15.6:0/234080809 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac017070 con 0x5621097d2a90 2024-03-06T07:28:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3be000640 1 -- 172.21.15.6:0/234080809 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ac0090d0 con 0x5621097d2a90 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3be000640 1 -- 172.21.15.6:0/234080809 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac006b60 con 0x5621097d2a90 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 -- 172.21.15.6:0/234080809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 msgr2=0x562109871900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 0x562109871900 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0097c0 tx=0x7fe3ac009c80 comp rx=0 tx=0).stop 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 -- 172.21.15.6:0/234080809 shutdown_connections 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 0x562109871e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d2a90 0x562109871900 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/234080809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d11e0 0x562109863a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 -- 172.21.15.6:0/234080809 >> 172.21.15.6:0/234080809 conn(0x5621097d1670 msgr2=0x5621097a9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 -- 172.21.15.6:0/234080809 shutdown_connections 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.332+0000 7fe3c3439f40 1 -- 172.21.15.6:0/234080809 wait complete. 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.333+0000 7fe3c3439f40 1 Processor -- start 2024-03-06T07:28:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 -- start start 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d11e0 0x56210986e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d2a90 0x56210985df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 0x56210985f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621097d54c0 con 0x5621097d11e0 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562109863590 con 0x5621097d2a90 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.334+0000 7fe3c3439f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x562109863220 con 0x56210987b330 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d11e0 0x56210986e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d11e0 0x56210986e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57184/0 (socket says 172.21.15.6:57184) 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 -- 172.21.15.6:0/2751463105 learned_addr learned my addr 172.21.15.6:0/2751463105 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 -- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 msgr2=0x56210985f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56210987b330 0x56210985f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 -- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d2a90 msgr2=0x56210985df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5621097d2a90 0x56210985df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 -- 172.21.15.6:0/2751463105 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56210986ef10 con 0x5621097d11e0 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.335+0000 7fe3bfe00640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d11e0 0x56210986e830 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000bd10 tx=0x7fe3b0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.336+0000 7fe3bcc00640 1 -- 172.21.15.6:0/2751463105 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0008070 con 0x5621097d11e0 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.336+0000 7fe3bcc00640 1 -- 172.21.15.6:0/2751463105 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b000f430 con 0x5621097d11e0 2024-03-06T07:28:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.336+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56210986f1f0 con 0x5621097d11e0 2024-03-06T07:28:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.336+0000 7fe3bcc00640 1 -- 172.21.15.6:0/2751463105 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000f800 con 0x5621097d11e0 2024-03-06T07:28:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.336+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56210986f7b0 con 0x5621097d11e0 2024-03-06T07:28:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.338+0000 7fe3bcc00640 1 -- 172.21.15.6:0/2751463105 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b000f9a0 con 0x5621097d11e0 2024-03-06T07:28:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.339+0000 7fe3bcc00640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3a0069eb0 0x7fe3a006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.339+0000 7fe3bcc00640 1 -- 172.21.15.6:0/2751463105 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b0088eb0 con 0x5621097d11e0 2024-03-06T07:28:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.339+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x562109870300 0x562109870710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.339+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v9 -- 0x562109870c50 con 0x562109870300 2024-03-06T07:28:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.339+0000 7fe3bf400640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3a0069eb0 0x7fe3a006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.339+0000 7fe3c0800640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x562109870300 0x562109870710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.340+0000 7fe3c0800640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x562109870300 0x562109870710 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3bf400640 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3a0069eb0 0x7fe3a006c370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x5621097d6e30 tx=0x7fe3ac00e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c0800640 1 -- 172.21.15.6:0/2751463105 <== osd.3 v2:172.21.15.6:6816/1271003476 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7fe3b80525e0 con 0x562109870300 2024-03-06T07:28:52.343 INFO:teuthology.orchestra.run.smithi006.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:28:51.000000+0000, size 0 2024-03-06T07:28:52.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x562109870300 msgr2=0x562109870710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x562109870300 0x562109870710 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3a0069eb0 msgr2=0x7fe3a006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3a0069eb0 0x7fe3a006c370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x5621097d6e30 tx=0x7fe3ac00e000 comp rx=0 tx=0).stop 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d11e0 msgr2=0x56210986e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 --2- 172.21.15.6:0/2751463105 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5621097d11e0 0x56210986e830 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000bd10 tx=0x7fe3b0009560 comp rx=0 tx=0).stop 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c0800640 1 -- 172.21.15.6:0/2751463105 reap_dead start 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 shutdown_connections 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 >> 172.21.15.6:0/2751463105 conn(0x5621097d1670 msgr2=0x562109871900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:52.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 shutdown_connections 2024-03-06T07:28:52.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:52.341+0000 7fe3c3439f40 1 -- 172.21.15.6:0/2751463105 wait complete. 2024-03-06T07:28:52.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:52.351 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:28:55.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:56.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:55.999+0000 7fd90de00640 1 Processor -- start 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:55.999+0000 7fd90de00640 1 -- start start 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.001+0000 7fd90de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd908072da0 0x7fd908162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.001+0000 7fd90de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 0x7fd90806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.001+0000 7fd90de00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9080ce090 con 0x7fd908070ee0 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.001+0000 7fd90de00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9080ceca0 con 0x7fd908072da0 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.001+0000 7fd90de00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080ce370 con 0x7fd9080712c0 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.002+0000 7fd907400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 0x7fd90806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.002+0000 7fd907400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 0x7fd90806e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:48760/0 (socket says 172.21.15.6:48760) 2024-03-06T07:28:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.002+0000 7fd907400640 1 -- 172.21.15.6:0/82353318 learned_addr learned my addr 172.21.15.6:0/82353318 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.002+0000 7fd906a00640 1 --1- 172.21.15.6:0/82353318 >> v1:172.21.15.178:6789/0 conn(0x7fd908070ee0 0x7fd90806e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:53628/0 (socket says 172.21.15.6:53628) 2024-03-06T07:28:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.002+0000 7fd906000640 1 --2- 172.21.15.6:0/82353318 >> v2:172.21.15.6:3300/0 conn(0x7fd908072da0 0x7fd908162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.003+0000 7fd905600640 1 -- 172.21.15.6:0/82353318 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130817211 0 0) 0x7fd9080ce090 con 0x7fd908070ee0 2024-03-06T07:28:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.003+0000 7fd905600640 1 -- 172.21.15.6:0/82353318 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ec003520 con 0x7fd908070ee0 2024-03-06T07:28:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.003+0000 7fd907400640 1 -- 172.21.15.6:0/82353318 >> v2:172.21.15.6:3300/0 conn(0x7fd908072da0 msgr2=0x7fd908162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.003+0000 7fd907400640 1 --2- 172.21.15.6:0/82353318 >> v2:172.21.15.6:3300/0 conn(0x7fd908072da0 0x7fd908162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.003+0000 7fd907400640 1 -- 172.21.15.6:0/82353318 >> v1:172.21.15.178:6789/0 conn(0x7fd908070ee0 legacy=0x7fd90806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.003+0000 7fd907400640 1 -- 172.21.15.6:0/82353318 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080ce650 con 0x7fd9080712c0 2024-03-06T07:28:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.005+0000 7fd907400640 1 --2- 172.21.15.6:0/82353318 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 0x7fd90806e7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9080cee20 tx=0x7fd90002bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9725652fd3c27fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.006+0000 7fd905600640 1 -- 172.21.15.6:0/82353318 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90002cae0 con 0x7fd9080712c0 2024-03-06T07:28:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.006+0000 7fd905600640 1 -- 172.21.15.6:0/82353318 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90002cc80 con 0x7fd9080712c0 2024-03-06T07:28:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.006+0000 7fd905600640 1 -- 172.21.15.6:0/82353318 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900034610 con 0x7fd9080712c0 2024-03-06T07:28:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.007+0000 7fd90de00640 1 -- 172.21.15.6:0/82353318 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 msgr2=0x7fd90806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.007+0000 7fd90de00640 1 --2- 172.21.15.6:0/82353318 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 0x7fd90806e7d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9080cee20 tx=0x7fd90002bf40 comp rx=0 tx=0).stop 2024-03-06T07:28:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.007+0000 7fd90de00640 1 -- 172.21.15.6:0/82353318 shutdown_connections 2024-03-06T07:28:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.007+0000 7fd90de00640 1 --2- 172.21.15.6:0/82353318 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9080712c0 0x7fd90806e7d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.007+0000 7fd90de00640 1 --2- 172.21.15.6:0/82353318 >> v2:172.21.15.6:3300/0 conn(0x7fd908072da0 0x7fd908162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.007+0000 7fd90de00640 1 -- 172.21.15.6:0/82353318 >> 172.21.15.6:0/82353318 conn(0x7fd90806afb0 msgr2=0x7fd90806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.008+0000 7fd90de00640 1 -- 172.21.15.6:0/82353318 shutdown_connections 2024-03-06T07:28:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.008+0000 7fd90de00640 1 -- 172.21.15.6:0/82353318 wait complete. 2024-03-06T07:28:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.011+0000 7fd90de00640 1 Processor -- start 2024-03-06T07:28:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.011+0000 7fd90de00640 1 -- start start 2024-03-06T07:28:56.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.013+0000 7fd90de00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd908070ee0 0x7fd90809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.014+0000 7fd90de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9080712c0 0x7fd90809af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.014+0000 7fd90de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 0x7fd90814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.015+0000 7fd90de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080cdda0 con 0x7fd9080712c0 2024-03-06T07:28:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.016+0000 7fd907400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 0x7fd90814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.016+0000 7fd907400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 0x7fd90814e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:48774/0 (socket says 172.21.15.6:48774) 2024-03-06T07:28:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.016+0000 7fd907400640 1 -- 172.21.15.6:0/3346532961 learned_addr learned my addr 172.21.15.6:0/3346532961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.016+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080cdc20 con 0x7fd908070ee0 2024-03-06T07:28:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.016+0000 7fd906a00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd908070ee0 0x7fd90809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.016+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080cf130 con 0x7fd908072da0 2024-03-06T07:28:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.017+0000 7fd906000640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9080712c0 0x7fd90809af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.019+0000 7fd906000640 1 -- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 msgr2=0x7fd90814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.019+0000 7fd906000640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 0x7fd90814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.019+0000 7fd906000640 1 -- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd908070ee0 msgr2=0x7fd90809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.019+0000 7fd906000640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd908070ee0 0x7fd90809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.020+0000 7fd906000640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908161fa0 con 0x7fd9080712c0 2024-03-06T07:28:56.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.021+0000 7fd907400640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 0x7fd90814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.021+0000 7fd906a00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd908070ee0 0x7fd90809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.022+0000 7fd906000640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9080712c0 0x7fd90809af50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400d520 tx=0x7fd8f400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.022+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40125f0 con 0x7fd9080712c0 2024-03-06T07:28:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.023+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4019070 con 0x7fd9080712c0 2024-03-06T07:28:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.023+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90814a5f0 con 0x7fd9080712c0 2024-03-06T07:28:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.023+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90814abb0 con 0x7fd9080712c0 2024-03-06T07:28:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.023+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4012990 con 0x7fd9080712c0 2024-03-06T07:28:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.025+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e0005290 con 0x7fd9080712c0 2024-03-06T07:28:56.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.029+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8f4012b30 con 0x7fd9080712c0 2024-03-06T07:28:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.033+0000 7fd8ff400640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8e806b4d0 0x7fd8e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.033+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8f408a9e0 con 0x7fd9080712c0 2024-03-06T07:28:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.034+0000 7fd906a00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8e806b4d0 0x7fd8e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.036+0000 7fd906a00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8e806b4d0 0x7fd8e806d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8000f60 tx=0x7fd8f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.041+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8f4023020 con 0x7fd9080712c0 2024-03-06T07:28:56.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.350+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8e0005580 con 0x7fd9080712c0 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.351+0000 7fd8ff400640 1 -- 172.21.15.6:0/3346532961 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd8f405f030 con 0x7fd9080712c0 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:28:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:28:56.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8e806b4d0 msgr2=0x7fd8e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8e806b4d0 0x7fd8e806d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8000f60 tx=0x7fd8f8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9080712c0 msgr2=0x7fd90809af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9080712c0 0x7fd90809af50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400d520 tx=0x7fd8f400d9e0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 shutdown_connections 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8e806b4d0 0x7fd8e806d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd908072da0 0x7fd90814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9080712c0 0x7fd90809af50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 --2- 172.21.15.6:0/3346532961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd908070ee0 0x7fd90809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 >> 172.21.15.6:0/3346532961 conn(0x7fd90806afb0 msgr2=0x7fd908164a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.353+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 shutdown_connections 2024-03-06T07:28:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.354+0000 7fd90de00640 1 -- 172.21.15.6:0/3346532961 wait complete. 2024-03-06T07:28:56.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:28:56.773 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:28:56.773 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:56.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:56.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.992+0000 7f848be00640 1 Processor -- start 2024-03-06T07:28:56.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.992+0000 7f848be00640 1 -- start start 2024-03-06T07:28:56.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f848be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f84840af4e0 0x7f848415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f848be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848406db30 con 0x7f848406ed60 2024-03-06T07:28:56.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f848be00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848406ddf0 con 0x7f848406d6a0 2024-03-06T07:28:56.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f848be00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848406e360 con 0x7f84840af4e0 2024-03-06T07:28:56.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f8489400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f848406ed60 0x7f8484071760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56634/0 (socket says 172.21.15.6:56634) 2024-03-06T07:28:56.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f8489400640 1 -- 172.21.15.6:0/2217120168 learned_addr learned my addr 172.21.15.6:0/2217120168 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:56.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.995+0000 7f8489e00640 1 --2- 172.21.15.6:0/2217120168 >> v2:172.21.15.178:3300/0 conn(0x7f84840af4e0 0x7f848415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.996+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568325143 0 0) 0x7f848406db30 con 0x7f848406ed60 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.997+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f846c005730 con 0x7f848406ed60 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.997+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197469790 0 0) 0x7f848406ddf0 con 0x7f848406d6a0 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.997+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848406db30 con 0x7f848406d6a0 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.997+0000 7f8489e00640 1 -- 172.21.15.6:0/2217120168 >> v1:172.21.15.178:6789/0 conn(0x7f848406d6a0 legacy=0x7f84840aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.997+0000 7f8489e00640 1 -- 172.21.15.6:0/2217120168 >> v1:172.21.15.6:6789/0 conn(0x7f848406ed60 legacy=0x7f8484071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:56.997+0000 7f8489e00640 1 -- 172.21.15.6:0/2217120168 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848406e5b0 con 0x7f84840af4e0 2024-03-06T07:28:57.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.000+0000 7f8489e00640 1 --2- 172.21.15.6:0/2217120168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84840af4e0 0x7f848415dd20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f847c00aea0 tx=0x7f847c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad6eeb37e49a73a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.002+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c02c9c0 con 0x7f84840af4e0 2024-03-06T07:28:57.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.002+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c02cb60 con 0x7f84840af4e0 2024-03-06T07:28:57.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.002+0000 7f8483e00640 1 -- 172.21.15.6:0/2217120168 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c008440 con 0x7f84840af4e0 2024-03-06T07:28:57.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.003+0000 7f848be00640 1 -- 172.21.15.6:0/2217120168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84840af4e0 msgr2=0x7f848415dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.003+0000 7f848be00640 1 --2- 172.21.15.6:0/2217120168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84840af4e0 0x7f848415dd20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f847c00aea0 tx=0x7f847c02be20 comp rx=0 tx=0).stop 2024-03-06T07:28:57.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.004+0000 7f848be00640 1 -- 172.21.15.6:0/2217120168 shutdown_connections 2024-03-06T07:28:57.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.004+0000 7f848be00640 1 --2- 172.21.15.6:0/2217120168 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f84840af4e0 0x7f848415dd20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.004+0000 7f848be00640 1 -- 172.21.15.6:0/2217120168 >> 172.21.15.6:0/2217120168 conn(0x7f848406afb0 msgr2=0x7f848406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.004+0000 7f848be00640 1 -- 172.21.15.6:0/2217120168 shutdown_connections 2024-03-06T07:28:57.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.005+0000 7f848be00640 1 -- 172.21.15.6:0/2217120168 wait complete. 2024-03-06T07:28:57.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.007+0000 7f848be00640 1 Processor -- start 2024-03-06T07:28:57.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.007+0000 7f848be00640 1 -- start start 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.008+0000 7f848be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 0x7f8484159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.008+0000 7f848be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f848406ed60 0x7f84841583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.009+0000 7f848be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84840af4e0 0x7f8484158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.009+0000 7f848be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84840cf120 con 0x7f84840af4e0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.009+0000 7f848be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84840cefa0 con 0x7f848406d6a0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.009+0000 7f848be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84840cf2a0 con 0x7f848406ed60 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.009+0000 7f8489400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 0x7f8484159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.009+0000 7f8489e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84840af4e0 0x7f8484158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.010+0000 7f8488a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f848406ed60 0x7f84841583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.010+0000 7f8489400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 0x7f8484159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:50858/0 (socket says 172.21.15.6:50858) 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.010+0000 7f8489400640 1 -- 172.21.15.6:0/2953940120 learned_addr learned my addr 172.21.15.6:0/2953940120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.011+0000 7f8489400640 1 -- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f848406ed60 msgr2=0x7f84841583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.011+0000 7f8489400640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f848406ed60 0x7f84841583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.011+0000 7f8489400640 1 -- 172.21.15.6:0/2953940120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84840af4e0 msgr2=0x7f8484158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.012+0000 7f8489400640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84840af4e0 0x7f8484158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.012+0000 7f8489400640 1 -- 172.21.15.6:0/2953940120 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84840c64a0 con 0x7f848406d6a0 2024-03-06T07:28:57.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.012+0000 7f8489e00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84840af4e0 0x7f8484158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:57.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.013+0000 7f8489400640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 0x7f8484159910 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f847800c500 tx=0x7f847800c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.014+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478017070 con 0x7f848406d6a0 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.014+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478005ce0 con 0x7f848406d6a0 2024-03-06T07:28:57.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.014+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478007330 con 0x7f848406d6a0 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.014+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84840c5920 con 0x7f848406d6a0 2024-03-06T07:28:57.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.014+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84840c5ee0 con 0x7f848406d6a0 2024-03-06T07:28:57.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.016+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8484067830 con 0x7f848406d6a0 2024-03-06T07:28:57.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.019+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f847801a070 con 0x7f848406d6a0 2024-03-06T07:28:57.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.020+0000 7f8481600640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.021+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8478089610 con 0x7f848406d6a0 2024-03-06T07:28:57.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.021+0000 7f8488a00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.022+0000 7f8488a00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f846406b4d0 0x7f846406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f84841f0ce0 tx=0x7f8470009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.034+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8484067830 con 0x7f848406d6a0 2024-03-06T07:28:57.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.273+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8484071460 con 0x7f848406d6a0 2024-03-06T07:28:57.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.274+0000 7f8481600640 1 -- 172.21.15.6:0/2953940120 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f8478057ef0 con 0x7f848406d6a0 2024-03-06T07:28:57.277 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:57.277 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:28:57.277 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f846406b4d0 msgr2=0x7f846406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f846406b4d0 0x7f846406d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f84841f0ce0 tx=0x7f8470009040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 msgr2=0x7f8484159910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 0x7f8484159910 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f847800c500 tx=0x7f847800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 shutdown_connections 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f846406b4d0 0x7f846406d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f84840af4e0 0x7f8484158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f848406ed60 0x7f84841583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 --2- 172.21.15.6:0/2953940120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f848406d6a0 0x7f8484159910 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.277+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 >> 172.21.15.6:0/2953940120 conn(0x7f848406afb0 msgr2=0x7f84840ae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.278+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 shutdown_connections 2024-03-06T07:28:57.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.278+0000 7f848be00640 1 -- 172.21.15.6:0/2953940120 wait complete. 2024-03-06T07:28:57.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:28:57.522 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:28:57.523 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:28:57.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.744+0000 7ff95fe00640 1 Processor -- start 2024-03-06T07:28:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.744+0000 7ff95fe00640 1 -- start start 2024-03-06T07:28:57.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.746+0000 7ff95fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff9600d0e50 0x7ff960168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.746+0000 7ff95fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9600b3710 con 0x7ff96015e0b0 2024-03-06T07:28:57.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.746+0000 7ff95fe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9600b39f0 con 0x7ff9600d4a10 2024-03-06T07:28:57.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.746+0000 7ff95fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9600b09e0 con 0x7ff9600d0e50 2024-03-06T07:28:57.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.746+0000 7ff95ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff96015e0b0 0x7ff9600cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56650/0 (socket says 172.21.15.6:56650) 2024-03-06T07:28:57.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.746+0000 7ff95ea00640 1 -- 172.21.15.6:0/2362360039 learned_addr learned my addr 172.21.15.6:0/2362360039 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95f400640 1 --2- 172.21.15.6:0/2362360039 >> v2:172.21.15.178:3300/0 conn(0x7ff9600d0e50 0x7ff960168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732603674 0 0) 0x7ff9600b3710 con 0x7ff96015e0b0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff944003520 con 0x7ff96015e0b0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2327152927 0 0) 0x7ff944003520 con 0x7ff96015e0b0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 >> v2:172.21.15.178:3300/0 conn(0x7ff9600d0e50 msgr2=0x7ff960168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 --2- 172.21.15.6:0/2362360039 >> v2:172.21.15.178:3300/0 conn(0x7ff9600d0e50 0x7ff960168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 >> v1:172.21.15.178:6789/0 conn(0x7ff9600d4a10 legacy=0x7ff9600d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600b3cd0 con 0x7ff96015e0b0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff9500019f0 con 0x7ff96015e0b0 2024-03-06T07:28:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 >> v1:172.21.15.6:6789/0 conn(0x7ff96015e0b0 legacy=0x7ff9600cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440039b0 0x7ff944003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.747+0000 7ff95e000640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff944004800 0x7ff944004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95e000640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440051c0 0x7ff94400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400c030 con 0x7ff9440039b0 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400c1e0 con 0x7ff944004800 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400c3e0 con 0x7ff9440051c0 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440039b0 0x7ff944003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff956000640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff944004800 0x7ff944004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95ea00640 1 -- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440051c0 msgr2=0x7ff94400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440051c0 0x7ff94400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95ea00640 1 -- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff944004800 msgr2=0x7ff944004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff944004800 0x7ff944004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95ea00640 1 -- 172.21.15.6:0/2362360039 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944003520 con 0x7ff9440039b0 2024-03-06T07:28:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.748+0000 7ff95f400640 1 -- 172.21.15.6:0/2362360039 reap_dead start 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.749+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440039b0 0x7ff944003dd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff95000a860 tx=0x7ff944005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193d50742be53ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95000a480 con 0x7ff9440039b0 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950034630 con 0x7ff9440039b0 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95e000640 1 -- 172.21.15.6:0/2362360039 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950034aa0 con 0x7ff9440039b0 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95fe00640 1 -- 172.21.15.6:0/2362360039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440039b0 msgr2=0x7ff944003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440039b0 0x7ff944003dd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff95000a860 tx=0x7ff944005830 comp rx=0 tx=0).stop 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95fe00640 1 -- 172.21.15.6:0/2362360039 shutdown_connections 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2362360039 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9440039b0 0x7ff944003dd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.750+0000 7ff95fe00640 1 -- 172.21.15.6:0/2362360039 >> 172.21.15.6:0/2362360039 conn(0x7ff9600d3d30 msgr2=0x7ff9600d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.751+0000 7ff95fe00640 1 -- 172.21.15.6:0/2362360039 shutdown_connections 2024-03-06T07:28:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.751+0000 7ff95fe00640 1 -- 172.21.15.6:0/2362360039 wait complete. 2024-03-06T07:28:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.752+0000 7ff95fe00640 1 Processor -- start 2024-03-06T07:28:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.753+0000 7ff95fe00640 1 -- start start 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.753+0000 7ff95fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440039b0 0x7ff960155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.753+0000 7ff95fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600b71d0 0x7ff9600b7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 0x7ff9600b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600cfc60 con 0x7ff9600b54a0 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96015e490 con 0x7ff9600b71d0 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600d0740 con 0x7ff9440039b0 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 0x7ff9600b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 0x7ff9600b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57244/0 (socket says 172.21.15.6:57244) 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 -- 172.21.15.6:0/2774963379 learned_addr learned my addr 172.21.15.6:0/2774963379 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440039b0 0x7ff960155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff956000640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600b71d0 0x7ff9600b7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 -- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440039b0 msgr2=0x7ff960155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440039b0 0x7ff960155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 -- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600b71d0 msgr2=0x7ff9600b7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600b71d0 0x7ff9600b7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.754+0000 7ff95f400640 1 -- 172.21.15.6:0/2774963379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600b6080 con 0x7ff9600b54a0 2024-03-06T07:28:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.755+0000 7ff95f400640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 0x7ff9600b5900 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff96008dff0 tx=0x7ff95800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.755+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958017070 con 0x7ff9600b54a0 2024-03-06T07:28:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.755+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95800c680 con 0x7ff9600b54a0 2024-03-06T07:28:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.755+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95800ca70 con 0x7ff9600b54a0 2024-03-06T07:28:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.755+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9600b6360 con 0x7ff9600b54a0 2024-03-06T07:28:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.755+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9600c8880 con 0x7ff9600b54a0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.757+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96004f330 con 0x7ff9600b54a0 2024-03-06T07:28:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.757+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff95801a070 con 0x7ff9600b54a0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.759+0000 7ff957400640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff93c06b4d0 0x7ff93c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.759+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff958089cf0 con 0x7ff9600b54a0 2024-03-06T07:28:57.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.760+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff93c06b4d0 0x7ff93c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.761+0000 7ff95ea00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff93c06b4d0 0x7ff93c06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff950034000 tx=0x7ff950036040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:57.763+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff95808e070 con 0x7ff9600b54a0 2024-03-06T07:28:58.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.008+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9600b4570 con 0x7ff9600b54a0 2024-03-06T07:28:58.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.008+0000 7ff957400640 1 -- 172.21.15.6:0/2774963379 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7ff958058c00 con 0x7ff9600b54a0 2024-03-06T07:28:58.011 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:28:58.011 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:28:58.012 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:28:58.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff93c06b4d0 msgr2=0x7ff93c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff93c06b4d0 0x7ff93c06d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff950034000 tx=0x7ff950036040 comp rx=0 tx=0).stop 2024-03-06T07:28:58.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 msgr2=0x7ff9600b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:58.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 0x7ff9600b5900 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff96008dff0 tx=0x7ff95800ec00 comp rx=0 tx=0).stop 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 shutdown_connections 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff93c06b4d0 0x7ff93c06d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9600b71d0 0x7ff9600b7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9600b54a0 0x7ff9600b5900 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.011+0000 7ff95fe00640 1 --2- 172.21.15.6:0/2774963379 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9440039b0 0x7ff960155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.012+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 >> 172.21.15.6:0/2774963379 conn(0x7ff9600d3d30 msgr2=0x7ff96006cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:58.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.012+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 shutdown_connections 2024-03-06T07:28:58.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:28:58.012+0000 7ff95fe00640 1 -- 172.21.15.6:0/2774963379 wait complete. 2024-03-06T07:28:58.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:58.261 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":1000}} 2024-03-06T07:28:58.261 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1000 } } 2024-03-06T07:28:58.261 DEBUG:tasks.ceph_test_case:wait_until_equal: 1000 != 1001, waiting (timeout=600)... 2024-03-06T07:29:03.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:03.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.485+0000 7fd4caa00640 1 Processor -- start 2024-03-06T07:29:03.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.485+0000 7fd4caa00640 1 -- start start 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.487+0000 7fd4caa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd4c415e2a0 0x7fd4c40cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.487+0000 7fd4caa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40d1120 con 0x7fd4c40cf460 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.487+0000 7fd4caa00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c40d1400 con 0x7fd4c415dec0 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.487+0000 7fd4caa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c40b0ea0 con 0x7fd4c415e2a0 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.488+0000 7fd4ca000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd4c40cf460 0x7fd4c40b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34618/0 (socket says 172.21.15.6:34618) 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.488+0000 7fd4ca000640 1 -- 172.21.15.6:0/2149025353 learned_addr learned my addr 172.21.15.6:0/2149025353 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:03.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.488+0000 7fd4c8c00640 1 --2- 172.21.15.6:0/2149025353 >> v2:172.21.15.178:3300/0 conn(0x7fd4c415e2a0 0x7fd4c40cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.488+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808756490 0 0) 0x7fd4c40d1120 con 0x7fd4c40cf460 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.488+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ac003520 con 0x7fd4c40cf460 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.488+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163351844 0 0) 0x7fd4ac003520 con 0x7fd4c40cf460 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 >> v2:172.21.15.178:3300/0 conn(0x7fd4c415e2a0 msgr2=0x7fd4c40cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 --2- 172.21.15.6:0/2149025353 >> v2:172.21.15.178:3300/0 conn(0x7fd4c415e2a0 0x7fd4c40cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 >> v1:172.21.15.178:6789/0 conn(0x7fd4c415dec0 legacy=0x7fd4c40ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40b0b30 con 0x7fd4c40cf460 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd4bc0019f0 con 0x7fd4c40cf460 2024-03-06T07:29:03.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 >> v1:172.21.15.6:6789/0 conn(0x7fd4c40cf460 legacy=0x7fd4c40b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0039b0 0x7fd4ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4ac004800 0x7fd4ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0051c0 0x7fd4ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40b0ea0 con 0x7fd4ac0051c0 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac00c060 con 0x7fd4ac0039b0 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ac00c260 con 0x7fd4ac004800 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c8c00640 1 --2- 172.21.15.6:0/2149025353 >> v2:172.21.15.178:3300/0 conn(0x7fd4c415e2a0 0x7fd4c40cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4ca000640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0051c0 0x7fd4ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c9600640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0039b0 0x7fd4ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4c8c00640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4ac004800 0x7fd4ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.489+0000 7fd4ca000640 1 -- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4ac004800 msgr2=0x7fd4ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.490+0000 7fd4ca000640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4ac004800 0x7fd4ac004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.490+0000 7fd4ca000640 1 -- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0039b0 msgr2=0x7fd4ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.490+0000 7fd4ca000640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0039b0 0x7fd4ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.490+0000 7fd4ca000640 1 -- 172.21.15.6:0/2149025353 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac003520 con 0x7fd4ac0051c0 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.490+0000 7fd4ca000640 1 -- 172.21.15.6:0/2149025353 reap_dead start 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.491+0000 7fd4ca000640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0051c0 0x7fd4ac00b9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40cf8d0 tx=0x7fd4ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da925bd7135101d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.491+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc00a4a0 con 0x7fd4ac0051c0 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.491+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc034650 con 0x7fd4ac0051c0 2024-03-06T07:29:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.491+0000 7fd4c3e00640 1 -- 172.21.15.6:0/2149025353 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc034a40 con 0x7fd4ac0051c0 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.491+0000 7fd4caa00640 1 -- 172.21.15.6:0/2149025353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0051c0 msgr2=0x7fd4ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.491+0000 7fd4caa00640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0051c0 0x7fd4ac00b9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40cf8d0 tx=0x7fd4ac005830 comp rx=0 tx=0).stop 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.492+0000 7fd4caa00640 1 -- 172.21.15.6:0/2149025353 shutdown_connections 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.492+0000 7fd4caa00640 1 --2- 172.21.15.6:0/2149025353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4ac0051c0 0x7fd4ac00b9c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.492+0000 7fd4caa00640 1 -- 172.21.15.6:0/2149025353 >> 172.21.15.6:0/2149025353 conn(0x7fd4c40d3d40 msgr2=0x7fd4c40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.492+0000 7fd4caa00640 1 -- 172.21.15.6:0/2149025353 shutdown_connections 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.492+0000 7fd4caa00640 1 -- 172.21.15.6:0/2149025353 wait complete. 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.494+0000 7fd4caa00640 1 Processor -- start 2024-03-06T07:29:03.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.494+0000 7fd4caa00640 1 -- start start 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4caa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0051c0 0x7fd4c41f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4caa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 0x7fd4c41f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4caa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4c40d0270 0x7fd4c40cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4caa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40ce990 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4caa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40ce810 con 0x7fd4ac0051c0 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4caa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c40ceb10 con 0x7fd4c40d0270 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4c8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 0x7fd4c41f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4c8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 0x7fd4c41f3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53188/0 (socket says 172.21.15.6:53188) 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4c8c00640 1 -- 172.21.15.6:0/536695565 learned_addr learned my addr 172.21.15.6:0/536695565 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4c9600640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0051c0 0x7fd4c41f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.495+0000 7fd4ca000640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4c40d0270 0x7fd4c40cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c8c00640 1 -- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4c40d0270 msgr2=0x7fd4c40cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c8c00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4c40d0270 0x7fd4c40cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c8c00640 1 -- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0051c0 msgr2=0x7fd4c41f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c8c00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0051c0 0x7fd4c41f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c8c00640 1 -- 172.21.15.6:0/536695565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c40cae10 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c9600640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0051c0 0x7fd4c41f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c8c00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 0x7fd4c41f3b70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0010920 tx=0x7fd4b0010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b000f040 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b000ec20 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c41f4430 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.496+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b001c400 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.497+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c41f49f0 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.498+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c404f330 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.498+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4b001c5a0 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.499+0000 7fd4c1600640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.499+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4b008aaa0 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.500+0000 7fd4c9600640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:03.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.501+0000 7fd4c9600640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd4a406b4d0 0x7fd4a406d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40d0b90 tx=0x7fd4b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:03.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.505+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4b0018030 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.749+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4c415fce0 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.750+0000 7fd4c1600640 1 -- 172.21.15.6:0/536695565 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd4b0059880 con 0x7fd4c40b8fd0 2024-03-06T07:29:03.752 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:03.753 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:29:03.753 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:29:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd4a406b4d0 msgr2=0x7fd4a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd4a406b4d0 0x7fd4a406d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40d0b90 tx=0x7fd4b8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 msgr2=0x7fd4c41f3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 0x7fd4c41f3b70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0010920 tx=0x7fd4b0010de0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 shutdown_connections 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd4a406b4d0 0x7fd4a406d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd4c40d0270 0x7fd4c40cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4c40b8fd0 0x7fd4c41f3b70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 --2- 172.21.15.6:0/536695565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4ac0051c0 0x7fd4c41f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 >> 172.21.15.6:0/536695565 conn(0x7fd4c40d3d40 msgr2=0x7fd4c40d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 shutdown_connections 2024-03-06T07:29:03.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:03.753+0000 7fd4caa00640 1 -- 172.21.15.6:0/536695565 wait complete. 2024-03-06T07:29:03.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:04.024 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:29:04.024 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:29:04.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:04.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.256+0000 7fa267200640 1 Processor -- start 2024-03-06T07:29:04.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.256+0000 7fa267200640 1 -- start start 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa267200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa26015e0b0 0x7fa2600cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa267200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600b3710 con 0x7fa2600d0e50 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa267200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2600b39f0 con 0x7fa2600d4a10 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa267200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2600b09e0 con 0x7fa26015e0b0 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa266800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa2600d0e50 0x7fa2600b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34622/0 (socket says 172.21.15.6:34622) 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa266800640 1 -- 172.21.15.6:0/1501790328 learned_addr learned my addr 172.21.15.6:0/1501790328 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:04.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.259+0000 7fa265e00640 1 --2- 172.21.15.6:0/1501790328 >> v2:172.21.15.178:3300/0 conn(0x7fa26015e0b0 0x7fa2600cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.260+0000 7fa264a00640 1 -- 172.21.15.6:0/1501790328 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102677688 0 0) 0x7fa2600b3710 con 0x7fa2600d0e50 2024-03-06T07:29:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.260+0000 7fa264a00640 1 -- 172.21.15.6:0/1501790328 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa248003520 con 0x7fa2600d0e50 2024-03-06T07:29:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.260+0000 7fa265e00640 1 -- 172.21.15.6:0/1501790328 >> v1:172.21.15.178:6789/0 conn(0x7fa2600d4a10 legacy=0x7fa2600d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.260+0000 7fa265e00640 1 -- 172.21.15.6:0/1501790328 >> v1:172.21.15.6:6789/0 conn(0x7fa2600d0e50 legacy=0x7fa2600b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.260+0000 7fa264a00640 1 -- 172.21.15.6:0/1501790328 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3738302391 0 0) 0x7fa248003520 con 0x7fa2600d0e50 2024-03-06T07:29:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.260+0000 7fa265e00640 1 -- 172.21.15.6:0/1501790328 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2600b3cd0 con 0x7fa26015e0b0 2024-03-06T07:29:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.262+0000 7fa265e00640 1 --2- 172.21.15.6:0/1501790328 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cf310 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097a0 tx=0x7fa25402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bfeb4134dcdc5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.262+0000 7fa264a00640 1 -- 172.21.15.6:0/1501790328 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254038070 con 0x7fa26015e0b0 2024-03-06T07:29:04.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.262+0000 7fa264a00640 1 -- 172.21.15.6:0/1501790328 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa254031500 con 0x7fa26015e0b0 2024-03-06T07:29:04.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.262+0000 7fa264a00640 1 -- 172.21.15.6:0/1501790328 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c0019f0 con 0x7fa26015e0b0 2024-03-06T07:29:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 -- 172.21.15.6:0/1501790328 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 msgr2=0x7fa2600cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 --2- 172.21.15.6:0/1501790328 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cf310 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa2540097a0 tx=0x7fa25402b820 comp rx=0 tx=0).stop 2024-03-06T07:29:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 -- 172.21.15.6:0/1501790328 shutdown_connections 2024-03-06T07:29:04.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 --2- 172.21.15.6:0/1501790328 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cf310 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 -- 172.21.15.6:0/1501790328 >> 172.21.15.6:0/1501790328 conn(0x7fa2600d3d30 msgr2=0x7fa2600d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 -- 172.21.15.6:0/1501790328 shutdown_connections 2024-03-06T07:29:04.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.263+0000 7fa267200640 1 -- 172.21.15.6:0/1501790328 wait complete. 2024-03-06T07:29:04.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.265+0000 7fa267200640 1 Processor -- start 2024-03-06T07:29:04.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.265+0000 7fa267200640 1 -- start start 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa267200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2600d0e50 0x7fa2601f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa267200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 0x7fa2600cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa267200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa267200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600cfc60 con 0x7fa2600d4a10 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa267200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26015e490 con 0x7fa26015e0b0 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa267200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2600cf850 con 0x7fa2600d0e50 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa265400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 0x7fa2600cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa265400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 0x7fa2600cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53200/0 (socket says 172.21.15.6:53200) 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.266+0000 7fa265400640 1 -- 172.21.15.6:0/314851147 learned_addr learned my addr 172.21.15.6:0/314851147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265e00640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2600d0e50 0x7fa2601f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa266800640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265400640 1 -- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2600d0e50 msgr2=0x7fa2601f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265400640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2600d0e50 0x7fa2601f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265400640 1 -- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 msgr2=0x7fa2600cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265400640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265400640 1 -- 172.21.15.6:0/314851147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26015ce10 con 0x7fa2600d4a10 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa265400640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 0x7fa2600cd2f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c430 tx=0x7fa25000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250017070 con 0x7fa2600d4a10 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2500052d0 con 0x7fa2600d4a10 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.267+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250003830 con 0x7fa2600d4a10 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.268+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2600cb470 con 0x7fa2600d4a10 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.268+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2600cba30 con 0x7fa2600d4a10 2024-03-06T07:29:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.269+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa260072ba0 con 0x7fa2600d4a10 2024-03-06T07:29:04.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.270+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa25001a070 con 0x7fa2600d4a10 2024-03-06T07:29:04.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.271+0000 7fa25a000640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa24406b4d0 0x7fa24406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.271+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa250089610 con 0x7fa2600d4a10 2024-03-06T07:29:04.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.271+0000 7fa265e00640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa24406b4d0 0x7fa24406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.273+0000 7fa265e00640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa24406b4d0 0x7fa24406d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa254035000 tx=0x7fa254008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.276+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa260072ba0 con 0x7fa2600d4a10 2024-03-06T07:29:04.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.514+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa26004f330 con 0x7fa2600d4a10 2024-03-06T07:29:04.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.515+0000 7fa25a000640 1 -- 172.21.15.6:0/314851147 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fa250058480 con 0x7fa2600d4a10 2024-03-06T07:29:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:04.518 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:29:04.518 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa24406b4d0 msgr2=0x7fa24406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa24406b4d0 0x7fa24406d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa254035000 tx=0x7fa254008030 comp rx=0 tx=0).stop 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 msgr2=0x7fa2600cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 0x7fa2600cd2f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c430 tx=0x7fa25000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 shutdown_connections 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa24406b4d0 0x7fa24406d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa26015e0b0 0x7fa2600cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2600d4a10 0x7fa2600cd2f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 --2- 172.21.15.6:0/314851147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2600d0e50 0x7fa2601f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 >> 172.21.15.6:0/314851147 conn(0x7fa2600d3d30 msgr2=0x7fa2600b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 shutdown_connections 2024-03-06T07:29:04.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.518+0000 7fa267200640 1 -- 172.21.15.6:0/314851147 wait complete. 2024-03-06T07:29:04.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:29:04.777 INFO:teuthology.orchestra.run.smithi006.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:29:04.777 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:29:04.778 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:29:04.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.894+0000 7fc63841ef40 1 Processor -- start 2024-03-06T07:29:04.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.894+0000 7fc63841ef40 1 -- start start 2024-03-06T07:29:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc63841ef40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 0x5591a6c70030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc63841ef40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6bc91a0 0x5591a6c67ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc63841ef40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6c71330 0x5591a6c62fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc63841ef40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591a6b42fd0 con 0x5591a6bc71e0 2024-03-06T07:29:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc63841ef40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591a6b46f30 con 0x5591a6c71330 2024-03-06T07:29:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc63841ef40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591a6be5bb0 con 0x5591a6bc91a0 2024-03-06T07:29:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.895+0000 7fc635600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 0x5591a6c70030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 0x5591a6c70030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53208/0 (socket says 172.21.15.6:53208) 2024-03-06T07:29:04.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 -- 172.21.15.6:0/928109588 learned_addr learned my addr 172.21.15.6:0/928109588 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:04.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc634c00640 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6bc91a0 0x5591a6c67ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc636000640 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6c71330 0x5591a6c62fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 -- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6bc91a0 msgr2=0x5591a6c67ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6bc91a0 0x5591a6c67ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 -- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6c71330 msgr2=0x5591a6c62fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6c71330 0x5591a6c62fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 -- 172.21.15.6:0/928109588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591a6c67960 con 0x5591a6bc71e0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.896+0000 7fc635600640 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 0x5591a6c70030 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc628001d90 tx=0x7fc62800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c819af04fbff8430 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc62f400640 1 -- 172.21.15.6:0/928109588 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800bd80 con 0x5591a6bc71e0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc62f400640 1 -- 172.21.15.6:0/928109588 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62800f400 con 0x5591a6bc71e0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc62f400640 1 -- 172.21.15.6:0/928109588 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800f7d0 con 0x5591a6bc71e0 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 -- 172.21.15.6:0/928109588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 msgr2=0x5591a6c70030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 0x5591a6c70030 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc628001d90 tx=0x7fc62800b5b0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 -- 172.21.15.6:0/928109588 shutdown_connections 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6c71330 0x5591a6c62fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6bc91a0 0x5591a6c67ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 --2- 172.21.15.6:0/928109588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc71e0 0x5591a6c70030 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 -- 172.21.15.6:0/928109588 >> 172.21.15.6:0/928109588 conn(0x5591a6bc7670 msgr2=0x5591a6c6f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.897+0000 7fc63841ef40 1 -- 172.21.15.6:0/928109588 shutdown_connections 2024-03-06T07:29:04.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.898+0000 7fc63841ef40 1 -- 172.21.15.6:0/928109588 wait complete. 2024-03-06T07:29:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.898+0000 7fc63841ef40 1 Processor -- start 2024-03-06T07:29:04.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.898+0000 7fc63841ef40 1 -- start start 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.899+0000 7fc63841ef40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6bc71e0 0x5591a6c6b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.899+0000 7fc63841ef40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc91a0 0x5591a6c71c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.899+0000 7fc63841ef40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6c71330 0x5591a6c586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.899+0000 7fc63841ef40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591a6c5a040 con 0x5591a6bc91a0 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.899+0000 7fc63841ef40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591a6c6cdb0 con 0x5591a6bc71e0 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.899+0000 7fc63841ef40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5591a6c68d30 con 0x5591a6c71330 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc91a0 0x5591a6c71c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc91a0 0x5591a6c71c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53224/0 (socket says 172.21.15.6:53224) 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 -- 172.21.15.6:0/4260182978 learned_addr learned my addr 172.21.15.6:0/4260182978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc636000640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6c71330 0x5591a6c586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc635600640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6bc71e0 0x5591a6c6b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 -- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6c71330 msgr2=0x5591a6c586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5591a6c71330 0x5591a6c586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 -- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6bc71e0 msgr2=0x5591a6c6b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5591a6bc71e0 0x5591a6c6b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.900+0000 7fc634c00640 1 -- 172.21.15.6:0/4260182978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591a6c721c0 con 0x5591a6bc91a0 2024-03-06T07:29:04.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.901+0000 7fc634c00640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc91a0 0x5591a6c71c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc624001ce0 tx=0x7fc62400a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.901+0000 7fc62e000640 1 -- 172.21.15.6:0/4260182978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62400e640 con 0x5591a6bc91a0 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.901+0000 7fc62e000640 1 -- 172.21.15.6:0/4260182978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62400e800 con 0x5591a6bc91a0 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.901+0000 7fc62e000640 1 -- 172.21.15.6:0/4260182978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62400ebf0 con 0x5591a6bc91a0 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.901+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5591a6c68fa0 con 0x5591a6bc91a0 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.901+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5591a6c69560 con 0x5591a6bc91a0 2024-03-06T07:29:04.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.903+0000 7fc62e000640 1 -- 172.21.15.6:0/4260182978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc624009030 con 0x5591a6bc91a0 2024-03-06T07:29:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.904+0000 7fc62e000640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc61806e2f0 0x7fc6180707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.904+0000 7fc62e000640 1 -- 172.21.15.6:0/4260182978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc624089380 con 0x5591a6bc91a0 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.905+0000 7fc63841ef40 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x5591a6c58c00 0x5591a6d0bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.905+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35) v9 -- 0x5591a6c737e0 con 0x5591a6c58c00 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.905+0000 7fc635600640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc61806e2f0 0x7fc6180707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.905+0000 7fc636000640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x5591a6c58c00 0x5591a6d0bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:04.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc636000640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x5591a6c58c00 0x5591a6d0bc40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc636000640 1 -- 172.21.15.6:0/4260182978 <== osd.3 v2:172.21.15.6:6816/1271003476 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7fc630052640 con 0x5591a6c58c00 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x5591a6c58c00 msgr2=0x5591a6d0bc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc63841ef40 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x5591a6c58c00 0x5591a6d0bc40 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc635600640 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc61806e2f0 0x7fc6180707b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc628001d90 tx=0x7fc628009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc61806e2f0 msgr2=0x7fc6180707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc63841ef40 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc61806e2f0 0x7fc6180707b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc628001d90 tx=0x7fc628009000 comp rx=0 tx=0).stop 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc91a0 msgr2=0x5591a6c71c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:04.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc63841ef40 1 --2- 172.21.15.6:0/4260182978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5591a6bc91a0 0x5591a6c71c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc624001ce0 tx=0x7fc62400a6b0 comp rx=0 tx=0).stop 2024-03-06T07:29:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.906+0000 7fc636000640 1 -- 172.21.15.6:0/4260182978 reap_dead start 2024-03-06T07:29:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.907+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 shutdown_connections 2024-03-06T07:29:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.907+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 >> 172.21.15.6:0/4260182978 conn(0x5591a6bc7670 msgr2=0x5591a6c6c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.907+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 shutdown_connections 2024-03-06T07:29:04.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:04.907+0000 7fc63841ef40 1 -- 172.21.15.6:0/4260182978 wait complete. 2024-03-06T07:29:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:04.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:29:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.112+0000 7fe897c00640 1 Processor -- start 2024-03-06T07:29:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.113+0000 7fe897c00640 1 -- start start 2024-03-06T07:29:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.115+0000 7fe897c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe89015e2e0 0x7fe8900d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.115+0000 7fe897c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8900d0d70 0x7fe8900d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.115+0000 7fe897c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8900b3940 con 0x7fe8900d1b30 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.115+0000 7fe897c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900d4e70 con 0x7fe89015e2e0 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.115+0000 7fe897c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900b3c40 con 0x7fe8900d0d70 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe89015e2e0 0x7fe8900d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe89015e2e0 0x7fe8900d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53236/0 (socket says 172.21.15.6:53236) 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 -- 172.21.15.6:0/114269290 learned_addr learned my addr 172.21.15.6:0/114269290 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe88fe00640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8900d0d70 0x7fe8900d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 -- 172.21.15.6:0/114269290 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8900d0d70 msgr2=0x7fe8900d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8900d0d70 0x7fe8900d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 -- 172.21.15.6:0/114269290 >> v1:172.21.15.6:6789/0 conn(0x7fe8900d1b30 legacy=0x7fe8900d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe895200640 1 -- 172.21.15.6:0/114269290 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900cf580 con 0x7fe89015e2e0 2024-03-06T07:29:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.116+0000 7fe88fe00640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8900d0d70 0x7fe8900d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.117+0000 7fe895200640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe89015e2e0 0x7fe8900d0830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe884009870 tx=0x7fe88402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6cce0a2ce2f72e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe88f400640 1 -- 172.21.15.6:0/114269290 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884034070 con 0x7fe89015e2e0 2024-03-06T07:29:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe88f400640 1 -- 172.21.15.6:0/114269290 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884030840 con 0x7fe89015e2e0 2024-03-06T07:29:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe88f400640 1 -- 172.21.15.6:0/114269290 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884030c10 con 0x7fe89015e2e0 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe897c00640 1 -- 172.21.15.6:0/114269290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe89015e2e0 msgr2=0x7fe8900d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe897c00640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe89015e2e0 0x7fe8900d0830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe884009870 tx=0x7fe88402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe897c00640 1 -- 172.21.15.6:0/114269290 shutdown_connections 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe897c00640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe8900d0d70 0x7fe8900d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe897c00640 1 --2- 172.21.15.6:0/114269290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe89015e2e0 0x7fe8900d0830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.118+0000 7fe897c00640 1 -- 172.21.15.6:0/114269290 >> 172.21.15.6:0/114269290 conn(0x7fe8900d3f60 msgr2=0x7fe8900d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.119+0000 7fe897c00640 1 -- 172.21.15.6:0/114269290 shutdown_connections 2024-03-06T07:29:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.119+0000 7fe897c00640 1 -- 172.21.15.6:0/114269290 wait complete. 2024-03-06T07:29:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.120+0000 7fe897c00640 1 Processor -- start 2024-03-06T07:29:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.121+0000 7fe897c00640 1 -- start start 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.121+0000 7fe897c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 0x7fe89015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe897c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8900d1b30 0x7fe890160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe897c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe89015e2e0 0x7fe89014cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe897c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900cfe20 con 0x7fe8900d0d70 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe897c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900cfb40 con 0x7fe8900d1b30 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe897c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900b50e0 con 0x7fe89015e2e0 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 0x7fe89015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 0x7fe89015e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53238/0 (socket says 172.21.15.6:53238) 2024-03-06T07:29:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 -- 172.21.15.6:0/4100450554 learned_addr learned my addr 172.21.15.6:0/4100450554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 -- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe89015e2e0 msgr2=0x7fe89014cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe89015e2e0 0x7fe89014cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe88fe00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8900d1b30 0x7fe890160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe89015e2e0 0x7fe89014cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 -- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8900d1b30 msgr2=0x7fe890160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8900d1b30 0x7fe890160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895200640 1 -- 172.21.15.6:0/4100450554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900ca530 con 0x7fe8900d0d70 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.122+0000 7fe895c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe89015e2e0 0x7fe89014cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.123+0000 7fe895200640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 0x7fe89015e6c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe884000c00 tx=0x7fe8840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.123+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884034040 con 0x7fe8900d0d70 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.123+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884039d90 con 0x7fe8900d0d70 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.123+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884043610 con 0x7fe8900d0d70 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.123+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe890160b60 con 0x7fe8900d0d70 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.123+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe890160860 con 0x7fe8900d0d70 2024-03-06T07:29:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.125+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89004f3a0 con 0x7fe8900d0d70 2024-03-06T07:29:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.126+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe88404a060 con 0x7fe8900d0d70 2024-03-06T07:29:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.127+0000 7fe88cc00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.127+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8840ac3c0 con 0x7fe8900d0d70 2024-03-06T07:29:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.128+0000 7fe88fe00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.129+0000 7fe88fe00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe87006b4d0 0x7fe87006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe87c008850 tx=0x7fe87c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.132+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe88403e070 con 0x7fe8900d0d70 2024-03-06T07:29:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.553+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fe8900d1700 con 0x7fe8900d0d70 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.554+0000 7fe88cc00640 1 -- 172.21.15.6:0/4100450554 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1652 (secure 0 0 0) 0x7fe88402d330 con 0x7fe8900d0d70 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772827963392, 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 266149888, 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 266149888, 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00034426583442837, 2024-03-06T07:29:05.556 INFO:teuthology.orchestra.run.smithi006.stdout: "num_osds": 8, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_osds": 8, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "stats_by_class": { 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "ssd": { 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772827963392, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 266149888, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 266149888, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00034426583442837 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "name": ".mgr", 2024-03-06T07:29:05.557 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 1, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 590368, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 2, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 1160, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 1187840, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 1.6179851627384778e-06, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367073230848 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_metadata", 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 4, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 1130975, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 23, 2024-03-06T07:29:05.558 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 2272, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 2326528, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 3.1690142350271344e-06, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367073230848 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_data", 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 0, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 0, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 0, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 0, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 0, 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367073230848 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:05.559 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:05.560 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:29:05.560 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:29:05.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.556+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe87006b4d0 msgr2=0x7fe87006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.556+0000 7fe897c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe87006b4d0 0x7fe87006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe87c008850 tx=0x7fe87c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.556+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 msgr2=0x7fe89015e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.556+0000 7fe897c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 0x7fe89015e6c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe884000c00 tx=0x7fe8840046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 shutdown_connections 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe87006b4d0 0x7fe87006d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe8900d1b30 0x7fe890160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe89015e2e0 0x7fe89014cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 --2- 172.21.15.6:0/4100450554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe8900d0d70 0x7fe89015e6c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 >> 172.21.15.6:0/4100450554 conn(0x7fe8900d3f60 msgr2=0x7fe89006d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 shutdown_connections 2024-03-06T07:29:05.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:05.557+0000 7fe897c00640 1 -- 172.21.15.6:0/4100450554 wait complete. 2024-03-06T07:29:05.579 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:29:05.579 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:05.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:05.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:05.806 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:05.806 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:05.806 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:05.860 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:05.860 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:05.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:06.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.118+0000 7f6a78800640 1 Processor -- start 2024-03-06T07:29:06.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.118+0000 7f6a78800640 1 -- start start 2024-03-06T07:29:06.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.120+0000 7f6a78800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a7015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.120+0000 7f6a78800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6a700cdc20 0x7f6a700b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.120+0000 7f6a78800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a7015cc40 con 0x7f6a700af1a0 2024-03-06T07:29:06.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.120+0000 7f6a78800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a7015cf20 con 0x7f6a700cdc20 2024-03-06T07:29:06.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.120+0000 7f6a78800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7015d200 con 0x7f6a700af580 2024-03-06T07:29:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.121+0000 7f6a76800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6a700cdc20 0x7f6a700b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.121+0000 7f6a76800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6a700cdc20 0x7f6a700b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:52288/0 (socket says 172.21.15.6:52288) 2024-03-06T07:29:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.121+0000 7f6a76800640 1 -- 172.21.15.6:0/3509946963 learned_addr learned my addr 172.21.15.6:0/3509946963 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.121+0000 7f6a75400640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a7015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.122+0000 7f6a76800640 1 -- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 msgr2=0x7f6a7015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.122+0000 7f6a76800640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a7015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.122+0000 7f6a76800640 1 -- 172.21.15.6:0/3509946963 >> v1:172.21.15.178:6789/0 conn(0x7f6a700af1a0 legacy=0x7f6a700b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.122+0000 7f6a76800640 1 -- 172.21.15.6:0/3509946963 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7015d4e0 con 0x7f6a700cdc20 2024-03-06T07:29:06.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.122+0000 7f6a75400640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a7015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:06.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.123+0000 7f6a76800640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a700b2dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009e50 tx=0x7f6a6c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4edc6e32181cc7f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a74a00640 1 -- 172.21.15.6:0/3509946963 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c02ca80 con 0x7f6a700cdc20 2024-03-06T07:29:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a74a00640 1 -- 172.21.15.6:0/3509946963 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c02cc20 con 0x7f6a700cdc20 2024-03-06T07:29:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a74a00640 1 -- 172.21.15.6:0/3509946963 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0343f0 con 0x7f6a700cdc20 2024-03-06T07:29:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 -- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 msgr2=0x7f6a700b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a700b2dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009e50 tx=0x7f6a6c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 -- 172.21.15.6:0/3509946963 shutdown_connections 2024-03-06T07:29:06.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a700b2dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 --2- 172.21.15.6:0/3509946963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a7015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 -- 172.21.15.6:0/3509946963 >> 172.21.15.6:0/3509946963 conn(0x7f6a7006afb0 msgr2=0x7f6a7006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.124+0000 7f6a78800640 1 -- 172.21.15.6:0/3509946963 shutdown_connections 2024-03-06T07:29:06.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.125+0000 7f6a78800640 1 -- 172.21.15.6:0/3509946963 wait complete. 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.126+0000 7f6a78800640 1 Processor -- start 2024-03-06T07:29:06.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.126+0000 7f6a78800640 1 -- start start 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.127+0000 7f6a78800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 0x7f6a7014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.127+0000 7f6a78800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a70151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.127+0000 7f6a78800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a70152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.127+0000 7f6a78800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700b36a0 con 0x7f6a700af1a0 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.127+0000 7f6a78800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700b3520 con 0x7f6a700cdc20 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.127+0000 7f6a78800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700b3820 con 0x7f6a700af580 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 0x7f6a7014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 0x7f6a7014f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53252/0 (socket says 172.21.15.6:53252) 2024-03-06T07:29:06.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 -- 172.21.15.6:0/3182575091 learned_addr learned my addr 172.21.15.6:0/3182575091 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a76800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a70152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75400640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a70151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 -- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 msgr2=0x7f6a70151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a70151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 -- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 msgr2=0x7f6a70152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a70152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.128+0000 7f6a75e00640 1 -- 172.21.15.6:0/3182575091 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7009acf0 con 0x7f6a700af1a0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.129+0000 7f6a75e00640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 0x7f6a7014f050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c450 tx=0x7f6a6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.129+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640052d0 con 0x7f6a700af1a0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.129+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64005470 con 0x7f6a700af1a0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.129+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64012870 con 0x7f6a700af1a0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.129+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a701527b0 con 0x7f6a700af1a0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.129+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7006cb10 con 0x7f6a700af1a0 2024-03-06T07:29:06.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.130+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70069510 con 0x7f6a700af1a0 2024-03-06T07:29:06.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.132+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a64012a10 con 0x7f6a700af1a0 2024-03-06T07:29:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.132+0000 7f6a6a000640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5406b4d0 0x7f6a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.133+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a64089e50 con 0x7f6a700af1a0 2024-03-06T07:29:06.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.133+0000 7f6a75400640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5406b4d0 0x7f6a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.134+0000 7f6a75400640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5406b4d0 0x7f6a5406d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009560 tx=0x7f6a60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.137+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a64058cc0 con 0x7f6a700af1a0 2024-03-06T07:29:06.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.340+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a70072eb0 con 0x7f6a700af1a0 2024-03-06T07:29:06.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.342+0000 7f6a6a000640 1 -- 172.21.15.6:0/3182575091 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f6a640585c0 con 0x7f6a700af1a0 2024-03-06T07:29:06.345 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:06.345 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:41.851077+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:06.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5406b4d0 msgr2=0x7f6a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5406b4d0 0x7f6a5406d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009560 tx=0x7f6a60008040 comp rx=0 tx=0).stop 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 msgr2=0x7f6a7014f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 0x7f6a7014f050 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c450 tx=0x7f6a6400c910 comp rx=0 tx=0).stop 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 shutdown_connections 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a5406b4d0 0x7f6a5406d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a700cdc20 0x7f6a70152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a700af580 0x7f6a70151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 --2- 172.21.15.6:0/3182575091 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a700af1a0 0x7f6a7014f050 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 >> 172.21.15.6:0/3182575091 conn(0x7f6a7006afb0 msgr2=0x7f6a7006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 shutdown_connections 2024-03-06T07:29:06.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.345+0000 7f6a78800640 1 -- 172.21.15.6:0/3182575091 wait complete. 2024-03-06T07:29:06.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:38.380358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:39.297336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:06.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:06.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.572+0000 7fe7e6e00640 1 Processor -- start 2024-03-06T07:29:06.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.573+0000 7fe7e6e00640 1 -- start start 2024-03-06T07:29:06.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.574+0000 7fe7e6e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7e0161b10 0x7fe7e006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.574+0000 7fe7e6e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e00b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.574+0000 7fe7e6e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e0067030 con 0x7fe7e0161ef0 2024-03-06T07:29:06.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.574+0000 7fe7e6e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e006f3e0 con 0x7fe7e0161b10 2024-03-06T07:29:06.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.574+0000 7fe7e6e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e006f6c0 con 0x7fe7e00b3960 2024-03-06T07:29:06.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7e0161b10 0x7fe7e006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe7e0161b10 0x7fe7e006ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53256/0 (socket says 172.21.15.6:53256) 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 -- 172.21.15.6:0/3616411751 learned_addr learned my addr 172.21.15.6:0/3616411751 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e00b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 -- 172.21.15.6:0/3616411751 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 msgr2=0x7fe7e00b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e00b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 -- 172.21.15.6:0/3616411751 >> v1:172.21.15.178:6789/0 conn(0x7fe7e0161ef0 legacy=0x7fe7e0162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7dfe00640 1 -- 172.21.15.6:0/3616411751 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e006f9a0 con 0x7fe7e0161b10 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.575+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e00b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:06.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7dfe00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161b10 0x7fe7e006ed90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2fba1e5c09c7e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7dea00640 1 -- 172.21.15.6:0/3616411751 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0034070 con 0x7fe7e0161b10 2024-03-06T07:29:06.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7dea00640 1 -- 172.21.15.6:0/3616411751 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d0030840 con 0x7fe7e0161b10 2024-03-06T07:29:06.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7dea00640 1 -- 172.21.15.6:0/3616411751 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0030c10 con 0x7fe7e0161b10 2024-03-06T07:29:06.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7e6e00640 1 -- 172.21.15.6:0/3616411751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161b10 msgr2=0x7fe7e006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161b10 0x7fe7e006ed90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7e6e00640 1 -- 172.21.15.6:0/3616411751 shutdown_connections 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e00b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/3616411751 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161b10 0x7fe7e006ed90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.577+0000 7fe7e6e00640 1 -- 172.21.15.6:0/3616411751 >> 172.21.15.6:0/3616411751 conn(0x7fe7e00722d0 msgr2=0x7fe7e006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.578+0000 7fe7e6e00640 1 -- 172.21.15.6:0/3616411751 shutdown_connections 2024-03-06T07:29:06.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.578+0000 7fe7e6e00640 1 -- 172.21.15.6:0/3616411751 wait complete. 2024-03-06T07:29:06.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.579+0000 7fe7e6e00640 1 Processor -- start 2024-03-06T07:29:06.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 -- start start 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e0151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 0x7fe7e01537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7e013ed40 0x7fe7e00adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e01551e0 con 0x7fe7e0161ef0 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e00b3440 con 0x7fe7e013ed40 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.580+0000 7fe7e6e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e0155360 con 0x7fe7e00b3960 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 0x7fe7e01537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 0x7fe7e01537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53270/0 (socket says 172.21.15.6:53270) 2024-03-06T07:29:06.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 -- 172.21.15.6:0/32259558 learned_addr learned my addr 172.21.15.6:0/32259558 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:06.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7e013ed40 0x7fe7e00adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 -- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 msgr2=0x7fe7e0151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e0151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 -- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7e013ed40 msgr2=0x7fe7e00adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7e013ed40 0x7fe7e00adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7df400640 1 -- 172.21.15.6:0/32259558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0154660 con 0x7fe7e0161ef0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.581+0000 7fe7e4e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7e013ed40 0x7fe7e00adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.582+0000 7fe7df400640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 0x7fe7e01537c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c3f0 tx=0x7fe7cc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.582+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0052d0 con 0x7fe7e0161ef0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.582+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc005490 con 0x7fe7e0161ef0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.582+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc007260 con 0x7fe7e0161ef0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.582+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e01540a0 con 0x7fe7e0161ef0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.582+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e0143c70 con 0x7fe7e0161ef0 2024-03-06T07:29:06.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.583+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e004f330 con 0x7fe7e0161ef0 2024-03-06T07:29:06.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.584+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7cc007400 con 0x7fe7e0161ef0 2024-03-06T07:29:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.585+0000 7fe7d7e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.585+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7cc0896f0 con 0x7fe7e0161ef0 2024-03-06T07:29:06.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.586+0000 7fe7dfe00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:06.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.587+0000 7fe7dfe00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7c006b4d0 0x7fe7c006d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0036000 tx=0x7fe7d003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:06.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.590+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7cc058560 con 0x7fe7e0161ef0 2024-03-06T07:29:06.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.787+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7e00b51c0 con 0x7fe7e0161ef0 2024-03-06T07:29:06.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.788+0000 7fe7d7e00640 1 -- 172.21.15.6:0/32259558 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7cc057e60 con 0x7fe7e0161ef0 2024-03-06T07:29:06.790 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7c006b4d0 msgr2=0x7fe7c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7c006b4d0 0x7fe7c006d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0036000 tx=0x7fe7d003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 msgr2=0x7fe7e01537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 0x7fe7e01537c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c3f0 tx=0x7fe7cc00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 shutdown_connections 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe7c006b4d0 0x7fe7c006d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe7e013ed40 0x7fe7e00adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe7e0161ef0 0x7fe7e01537c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 --2- 172.21.15.6:0/32259558 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe7e00b3960 0x7fe7e0151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 >> 172.21.15.6:0/32259558 conn(0x7fe7e00722d0 msgr2=0x7fe7e00b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 shutdown_connections 2024-03-06T07:29:06.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:06.791+0000 7fe7e6e00640 1 -- 172.21.15.6:0/32259558 wait complete. 2024-03-06T07:29:06.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.036+0000 7fc903200640 1 Processor -- start 2024-03-06T07:29:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.036+0000 7fc903200640 1 -- start start 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc903200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8fc06ecf0 0x7fc8fc06f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc903200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc06dc50 con 0x7fc8fc0ce6e0 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc903200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc06df30 con 0x7fc8fc06afb0 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc903200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8fc06e210 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc901200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8fc06ecf0 0x7fc8fc06f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc901200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8fc06ecf0 0x7fc8fc06f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53274/0 (socket says 172.21.15.6:53274) 2024-03-06T07:29:07.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc901200640 1 -- 172.21.15.6:0/3796885985 learned_addr learned my addr 172.21.15.6:0/3796885985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.040+0000 7fc901200640 1 -- 172.21.15.6:0/3796885985 >> v1:172.21.15.178:6789/0 conn(0x7fc8fc06afb0 legacy=0x7fc8fc06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.041+0000 7fc901200640 1 -- 172.21.15.6:0/3796885985 >> v1:172.21.15.6:6789/0 conn(0x7fc8fc0ce6e0 legacy=0x7fc8fc0c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.041+0000 7fc901200640 1 -- 172.21.15.6:0/3796885985 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc06e4f0 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.042+0000 7fc901200640 1 --2- 172.21.15.6:0/3796885985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc06f100 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009870 tx=0x7fc8f402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a559a500ad791 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.042+0000 7fc8fb400640 1 -- 172.21.15.6:0/3796885985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f402f850 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.042+0000 7fc8fb400640 1 -- 172.21.15.6:0/3796885985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f402f9f0 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.042+0000 7fc8fb400640 1 -- 172.21.15.6:0/3796885985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f402fdc0 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.042+0000 7fc903200640 1 -- 172.21.15.6:0/3796885985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 msgr2=0x7fc8fc06f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.042+0000 7fc903200640 1 --2- 172.21.15.6:0/3796885985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc06f100 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009870 tx=0x7fc8f402ba30 comp rx=0 tx=0).stop 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.043+0000 7fc903200640 1 -- 172.21.15.6:0/3796885985 shutdown_connections 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.043+0000 7fc903200640 1 --2- 172.21.15.6:0/3796885985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc06f100 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.043+0000 7fc903200640 1 -- 172.21.15.6:0/3796885985 >> 172.21.15.6:0/3796885985 conn(0x7fc8fc0b2760 msgr2=0x7fc8fc0b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.043+0000 7fc903200640 1 -- 172.21.15.6:0/3796885985 shutdown_connections 2024-03-06T07:29:07.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.043+0000 7fc903200640 1 -- 172.21.15.6:0/3796885985 wait complete. 2024-03-06T07:29:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.045+0000 7fc903200640 1 Processor -- start 2024-03-06T07:29:07.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.045+0000 7fc903200640 1 -- start start 2024-03-06T07:29:07.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc903200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc06afb0 0x7fc8fc0aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc903200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc903200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8fc0ce6e0 0x7fc8fc1546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc903200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0c7780 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc903200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc15e2a0 con 0x7fc8fc06afb0 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc903200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc15f230 con 0x7fc8fc0ce6e0 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53282/0 (socket says 172.21.15.6:53282) 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 -- 172.21.15.6:0/3269587784 learned_addr learned my addr 172.21.15.6:0/3269587784 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc901200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8fc0ce6e0 0x7fc8fc1546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8fbe00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc06afb0 0x7fc8fc0aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 -- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8fc0ce6e0 msgr2=0x7fc8fc1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8fc0ce6e0 0x7fc8fc1546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 -- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc06afb0 msgr2=0x7fc8fc0aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc06afb0 0x7fc8fc0aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8f3e00640 1 -- 172.21.15.6:0/3269587784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc142480 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.046+0000 7fc8fbe00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc06afb0 0x7fc8fc0aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:07.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.047+0000 7fc8f3e00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc154180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400c400 tx=0x7fc8e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.047+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4017070 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.047+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e40052d0 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.047+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4003830 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.047+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc142760 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.047+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc142d20 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.048+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc04f330 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.050+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8e401a070 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.051+0000 7fc8f8c00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8d806b4d0 0x7fc8d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.051+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8e4089570 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.052+0000 7fc8fbe00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8d806b4d0 0x7fc8d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.054+0000 7fc8fbe00640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8d806b4d0 0x7fc8d806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0af8b0 tx=0x7fc8ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.060+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8e40583e0 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.260+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc8fc06f470 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.261+0000 7fc8f8c00640 1 -- 172.21.15.6:0/3269587784 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc8e4087e20 con 0x7fc8fc06ecf0 2024-03-06T07:29:07.263 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8d806b4d0 msgr2=0x7fc8d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8d806b4d0 0x7fc8d806d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc0af8b0 tx=0x7fc8ec009040 comp rx=0 tx=0).stop 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 msgr2=0x7fc8fc154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc154180 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400c400 tx=0x7fc8e400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 shutdown_connections 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8d806b4d0 0x7fc8d806d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8fc0ce6e0 0x7fc8fc1546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8fc06ecf0 0x7fc8fc154180 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 --2- 172.21.15.6:0/3269587784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8fc06afb0 0x7fc8fc0aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 >> 172.21.15.6:0/3269587784 conn(0x7fc8fc0b2760 msgr2=0x7fc8fc0b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 shutdown_connections 2024-03-06T07:29:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.263+0000 7fc903200640 1 -- 172.21.15.6:0/3269587784 wait complete. 2024-03-06T07:29:07.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.489+0000 7fc3a8c00640 1 Processor -- start 2024-03-06T07:29:07.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.489+0000 7fc3a8c00640 1 -- start start 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a8c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 0x7fc3a40b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a8c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc3a4165b60 0x7fc3a4167f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a40b5da0 con 0x7fc3a406cab0 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a8c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a40b6080 con 0x7fc3a4165b60 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a8c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b6360 con 0x7fc3a415e2d0 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc3a406cab0 0x7fc3a406ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34652/0 (socket says 172.21.15.6:34652) 2024-03-06T07:29:07.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.491+0000 7fc3a3400640 1 -- 172.21.15.6:0/1467503258 learned_addr learned my addr 172.21.15.6:0/1467503258 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:07.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a3e00640 1 --2- 172.21.15.6:0/1467503258 >> v2:172.21.15.178:3300/0 conn(0x7fc3a4165b60 0x7fc3a4167f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796340053 0 0) 0x7fc3a40b5da0 con 0x7fc3a406cab0 2024-03-06T07:29:07.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388003520 con 0x7fc3a406cab0 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 0x7fc3a40b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398123998 0 0) 0x7fc388003520 con 0x7fc3a406cab0 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 msgr2=0x7fc3a40b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 0x7fc3a40b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 >> v2:172.21.15.178:3300/0 conn(0x7fc3a4165b60 msgr2=0x7fc3a4167f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1467503258 >> v2:172.21.15.178:3300/0 conn(0x7fc3a4165b60 0x7fc3a4167f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40b6640 con 0x7fc3a406cab0 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc394001670 con 0x7fc3a406cab0 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 >> v1:172.21.15.6:6789/0 conn(0x7fc3a406cab0 legacy=0x7fc3a406ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3880039b0 0x7fc388003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 0x7fc388004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b6080 con 0x7fc3880039b0 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b6360 con 0x7fc388004d40 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800baf0 con 0x7fc3a415e2d0 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a3e00640 1 --2- 172.21.15.6:0/1467503258 >> v2:172.21.15.178:3300/0 conn(0x7fc3a4165b60 0x7fc3a4167f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.492+0000 7fc3a3400640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3880039b0 0x7fc388003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 0x7fc388004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a3e00640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a3400640 1 -- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 msgr2=0x7fc388004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a3400640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a415e2d0 0x7fc388004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a3400640 1 -- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc388004d40 msgr2=0x7fc3880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a3400640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.493+0000 7fc3a3400640 1 -- 172.21.15.6:0/1467503258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388003520 con 0x7fc3880039b0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.494+0000 7fc3a3400640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3880039b0 0x7fc388003dd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c190 tx=0x7fc388005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725fb44888ecf405 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.494+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394007c80 con 0x7fc3880039b0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394030070 con 0x7fc3880039b0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a2000640 1 -- 172.21.15.6:0/1467503258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394005d30 con 0x7fc3880039b0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1467503258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3880039b0 msgr2=0x7fc388003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1467503258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3880039b0 0x7fc388003dd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c190 tx=0x7fc388005830 comp rx=0 tx=0).stop 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a3e00640 1 -- 172.21.15.6:0/1467503258 reap_dead start 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1467503258 shutdown_connections 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1467503258 >> 172.21.15.6:0/1467503258 conn(0x7fc3a409aa10 msgr2=0x7fc3a409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1467503258 shutdown_connections 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.495+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1467503258 wait complete. 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.497+0000 7fc3a8c00640 1 Processor -- start 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.497+0000 7fc3a8c00640 1 -- start start 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.498+0000 7fc3a8c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3880039b0 0x7fc3a4158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.498+0000 7fc3a8c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a4154460 0x7fc3a4153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.498+0000 7fc3a8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154990 0x7fc3a41555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.498+0000 7fc3a8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415e8b0 con 0x7fc3a4154990 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.498+0000 7fc3a8c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415e730 con 0x7fc3880039b0 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.498+0000 7fc3a8c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b2630 con 0x7fc3a4154460 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3880039b0 0x7fc3a4158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154990 0x7fc3a41555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3880039b0 0x7fc3a4158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:52330/0 (socket says 172.21.15.6:52330) 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3400640 1 -- 172.21.15.6:0/1180512660 learned_addr learned my addr 172.21.15.6:0/1180512660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a2a00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a4154460 0x7fc3a4153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3e00640 1 -- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a4154460 msgr2=0x7fc3a4153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3e00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a4154460 0x7fc3a4153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3e00640 1 -- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3880039b0 msgr2=0x7fc3a4158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3e00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3880039b0 0x7fc3a4158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.499+0000 7fc3a3e00640 1 -- 172.21.15.6:0/1180512660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4156f70 con 0x7fc3a4154990 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.500+0000 7fc3a3e00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154990 0x7fc3a41555c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4072960 tx=0x7fc39c004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.500+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c019070 con 0x7fc3a4154990 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.500+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c007c90 con 0x7fc3a4154990 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.500+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c014640 con 0x7fc3a4154990 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.500+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a4157250 con 0x7fc3a4154990 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.500+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a4157810 con 0x7fc3a4154990 2024-03-06T07:29:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.501+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a404f330 con 0x7fc3a4154990 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.502+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc39c01c070 con 0x7fc3a4154990 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.503+0000 7fc39b400640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38406b500 0x7fc38406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.503+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc39c089c10 con 0x7fc3a4154990 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.504+0000 7fc3a3400640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38406b500 0x7fc38406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.505+0000 7fc3a3400640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38406b500 0x7fc38406d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc394007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.511+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc39c0583a0 con 0x7fc3a4154990 2024-03-06T07:29:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.706+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3a415c020 con 0x7fc3a4154990 2024-03-06T07:29:07.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.706+0000 7fc39b400640 1 -- 172.21.15.6:0/1180512660 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc39c05c440 con 0x7fc3a4154990 2024-03-06T07:29:07.709 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:29:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38406b500 msgr2=0x7fc38406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38406b500 0x7fc38406d9c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc394007000 comp rx=0 tx=0).stop 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154990 msgr2=0x7fc3a41555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154990 0x7fc3a41555c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4072960 tx=0x7fc39c004840 comp rx=0 tx=0).stop 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 shutdown_connections 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38406b500 0x7fc38406d9c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a4154990 0x7fc3a41555c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a4154460 0x7fc3a4153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 --2- 172.21.15.6:0/1180512660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3880039b0 0x7fc3a4158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.709+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 >> 172.21.15.6:0/1180512660 conn(0x7fc3a409aa10 msgr2=0x7fc3a40b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.710+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 shutdown_connections 2024-03-06T07:29:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.710+0000 7fc3a8c00640 1 -- 172.21.15.6:0/1180512660 wait complete. 2024-03-06T07:29:07.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.926+0000 7f4260e00640 1 Processor -- start 2024-03-06T07:29:07.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.926+0000 7f4260e00640 1 -- start start 2024-03-06T07:29:07.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4260e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f425c0d4a10 0x7f425c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4260e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c0d0c80 0x7f425c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4260e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f425c0b3710 con 0x7f425c15e0b0 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4260e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425c0b09e0 con 0x7f425c0d4a10 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4260e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425c0cfa20 con 0x7f425c0d0c80 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4252a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f425c0d4a10 0x7f425c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4252a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f425c0d4a10 0x7f425c0d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53304/0 (socket says 172.21.15.6:53304) 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4252a00640 1 -- 172.21.15.6:0/3162024568 learned_addr learned my addr 172.21.15.6:0/3162024568 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f425b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f425c15e0b0 0x7f425c0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34656/0 (socket says 172.21.15.6:34656) 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4252a00640 1 -- 172.21.15.6:0/3162024568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c0d0c80 msgr2=0x7f425c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f425be00640 1 --2- 172.21.15.6:0/3162024568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c0d0c80 0x7f425c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.929+0000 7f4252a00640 1 --2- 172.21.15.6:0/3162024568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c0d0c80 0x7f425c0b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.930+0000 7f4252a00640 1 -- 172.21.15.6:0/3162024568 >> v1:172.21.15.6:6789/0 conn(0x7f425c15e0b0 legacy=0x7f425c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.930+0000 7f4252a00640 1 -- 172.21.15.6:0/3162024568 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c0b39f0 con 0x7f425c0d4a10 2024-03-06T07:29:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.931+0000 7f4252a00640 1 --2- 172.21.15.6:0/3162024568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d4a10 0x7f425c0d0740 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4244009870 tx=0x7f424402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bf78972c10276c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.931+0000 7f425aa00640 1 -- 172.21.15.6:0/3162024568 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244034070 con 0x7f425c0d4a10 2024-03-06T07:29:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.931+0000 7f425aa00640 1 -- 172.21.15.6:0/3162024568 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4244030840 con 0x7f425c0d4a10 2024-03-06T07:29:07.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.931+0000 7f425aa00640 1 -- 172.21.15.6:0/3162024568 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244030c10 con 0x7f425c0d4a10 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.931+0000 7f4260e00640 1 -- 172.21.15.6:0/3162024568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d4a10 msgr2=0x7f425c0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.931+0000 7f4260e00640 1 --2- 172.21.15.6:0/3162024568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d4a10 0x7f425c0d0740 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4244009870 tx=0x7f424402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.932+0000 7f4260e00640 1 -- 172.21.15.6:0/3162024568 shutdown_connections 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.932+0000 7f4260e00640 1 --2- 172.21.15.6:0/3162024568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c0d0c80 0x7f425c0b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.932+0000 7f4260e00640 1 --2- 172.21.15.6:0/3162024568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d4a10 0x7f425c0d0740 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.932+0000 7f4260e00640 1 -- 172.21.15.6:0/3162024568 >> 172.21.15.6:0/3162024568 conn(0x7f425c0d3d30 msgr2=0x7f425c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.932+0000 7f4260e00640 1 -- 172.21.15.6:0/3162024568 shutdown_connections 2024-03-06T07:29:07.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.932+0000 7f4260e00640 1 -- 172.21.15.6:0/3162024568 wait complete. 2024-03-06T07:29:07.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.934+0000 7f4260e00640 1 Processor -- start 2024-03-06T07:29:07.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.935+0000 7f4260e00640 1 -- start start 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f4260e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 0x7f425c15a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f4260e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f425c0d4a10 0x7f425c1f8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f4260e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c15e0b0 0x7f425c159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f4260e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c15e490 con 0x7f425c0d0c80 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f4260e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c0b3fb0 con 0x7f425c0d4a10 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f4260e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425c0cfc60 con 0x7f425c15e0b0 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f425b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 0x7f425c15a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f425b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 0x7f425c15a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53316/0 (socket says 172.21.15.6:53316) 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.936+0000 7f425b400640 1 -- 172.21.15.6:0/2487153120 learned_addr learned my addr 172.21.15.6:0/2487153120 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f425b400640 1 -- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c15e0b0 msgr2=0x7f425c159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f425b400640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c15e0b0 0x7f425c159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f425b400640 1 -- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f425c0d4a10 msgr2=0x7f425c1f8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f425b400640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f425c0d4a10 0x7f425c1f8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f425b400640 1 -- 172.21.15.6:0/2487153120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c15a720 con 0x7f425c0d0c80 2024-03-06T07:29:07.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f425b400640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 0x7f425c15a140 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f424c00c7c0 tx=0x7f424c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c017070 con 0x7f425c0d0c80 2024-03-06T07:29:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424c00a640 con 0x7f425c0d0c80 2024-03-06T07:29:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424c00a9f0 con 0x7f425c0d0c80 2024-03-06T07:29:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c15aa00 con 0x7f425c0d0c80 2024-03-06T07:29:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.937+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c15afc0 con 0x7f425c0d0c80 2024-03-06T07:29:07.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.939+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c04f330 con 0x7f425c0d0c80 2024-03-06T07:29:07.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.940+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f424c01a070 con 0x7f425c0d0c80 2024-03-06T07:29:07.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.941+0000 7f4253e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f423806b4d0 0x7f423806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.942+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f424c089750 con 0x7f425c0d0c80 2024-03-06T07:29:07.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.942+0000 7f4252a00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f423806b4d0 0x7f423806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.943+0000 7f4252a00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f423806b4d0 0x7f423806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4244036000 tx=0x7f424403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:07.946+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f424c057f80 con 0x7f425c0d0c80 2024-03-06T07:29:08.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.160+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f425c161640 con 0x7f425c0d0c80 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.161+0000 7f4253e00640 1 -- 172.21.15.6:0/2487153120 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v35) v1 ==== 80+0+0 (secure 0 0 0) 0x7f424c05c020 con 0x7f425c0d0c80 2024-03-06T07:29:08.163 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f423806b4d0 msgr2=0x7f423806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f423806b4d0 0x7f423806d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4244036000 tx=0x7f424403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 msgr2=0x7f425c15a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 0x7f425c15a140 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f424c00c7c0 tx=0x7f424c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 shutdown_connections 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f423806b4d0 0x7f423806d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f425c0d4a10 0x7f425c1f8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f425c15e0b0 0x7f425c159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 --2- 172.21.15.6:0/2487153120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f425c0d0c80 0x7f425c15a140 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.163+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 >> 172.21.15.6:0/2487153120 conn(0x7f425c0d3d30 msgr2=0x7f425c0b0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.164+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 shutdown_connections 2024-03-06T07:29:08.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.164+0000 7f4260e00640 1 -- 172.21.15.6:0/2487153120 wait complete. 2024-03-06T07:29:08.182 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:29:08.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:08.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:08.295 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:29:08.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:08.295 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:08.343 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:29:08.343 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:08.384 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:08.384 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:08.384 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:08.421 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:08.421 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:29:08.421 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:08.475 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:08.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.674+0000 7f5891e00640 1 Processor -- start 2024-03-06T07:29:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.674+0000 7f5891e00640 1 -- start start 2024-03-06T07:29:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5891e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f588c0cd060 0x7f588c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5891e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5891e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c15ed10 con 0x7f588c1552b0 2024-03-06T07:29:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5891e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588c15eff0 con 0x7f588c0cd060 2024-03-06T07:29:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5891e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c15f2d0 con 0x7f588c15e810 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5890a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f588c1552b0 0x7f588c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34670/0 (socket says 172.21.15.6:34670) 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5890a00640 1 -- 172.21.15.6:0/3986153401 learned_addr learned my addr 172.21.15.6:0/3986153401 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.676+0000 7f5883e00640 1 --2- 172.21.15.6:0/3986153401 >> v2:172.21.15.6:3300/0 conn(0x7f588c0cd060 0x7f588c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.677+0000 7f5891400640 1 --2- 172.21.15.6:0/3986153401 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.677+0000 7f588be00640 1 -- 172.21.15.6:0/3986153401 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998396817 0 0) 0x7f588c15ed10 con 0x7f588c1552b0 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.677+0000 7f5883e00640 1 -- 172.21.15.6:0/3986153401 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 msgr2=0x7f588c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.677+0000 7f5883e00640 1 --2- 172.21.15.6:0/3986153401 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.677+0000 7f5883e00640 1 -- 172.21.15.6:0/3986153401 >> v1:172.21.15.6:6789/0 conn(0x7f588c1552b0 legacy=0x7f588c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.677+0000 7f5883e00640 1 -- 172.21.15.6:0/3986153401 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c15f5b0 con 0x7f588c0cd060 2024-03-06T07:29:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.678+0000 7f5883e00640 1 --2- 172.21.15.6:0/3986153401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c15b1d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f58740097a0 tx=0x7f587402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d6e270ce00658b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.678+0000 7f588be00640 1 -- 172.21.15.6:0/3986153401 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740315a0 con 0x7f588c0cd060 2024-03-06T07:29:08.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.678+0000 7f588be00640 1 -- 172.21.15.6:0/3986153401 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874037070 con 0x7f588c0cd060 2024-03-06T07:29:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.678+0000 7f588be00640 1 -- 172.21.15.6:0/3986153401 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874031960 con 0x7f588c0cd060 2024-03-06T07:29:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 -- 172.21.15.6:0/3986153401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 msgr2=0x7f588c15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 --2- 172.21.15.6:0/3986153401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c15b1d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f58740097a0 tx=0x7f587402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 -- 172.21.15.6:0/3986153401 shutdown_connections 2024-03-06T07:29:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 --2- 172.21.15.6:0/3986153401 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 --2- 172.21.15.6:0/3986153401 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c15b1d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 -- 172.21.15.6:0/3986153401 >> 172.21.15.6:0/3986153401 conn(0x7f588c06b3d0 msgr2=0x7f588c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 -- 172.21.15.6:0/3986153401 shutdown_connections 2024-03-06T07:29:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.679+0000 7f5891e00640 1 -- 172.21.15.6:0/3986153401 wait complete. 2024-03-06T07:29:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.680+0000 7f5891e00640 1 Processor -- start 2024-03-06T07:29:08.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.681+0000 7f5891e00640 1 -- start start 2024-03-06T07:29:08.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5891e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c14b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5891e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f588c1552b0 0x7f588c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5891e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5891e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c06dd30 con 0x7f588c0cd060 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5891e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0b3760 con 0x7f588c1552b0 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5891e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c06deb0 con 0x7f588c15e810 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.682+0000 7f5890a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c14b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c14b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53338/0 (socket says 172.21.15.6:53338) 2024-03-06T07:29:08.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 -- 172.21.15.6:0/2921402869 learned_addr learned my addr 172.21.15.6:0/2921402869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 -- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 msgr2=0x7f588c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 -- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f588c1552b0 msgr2=0x7f588c159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f588c1552b0 0x7f588c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 -- 172.21.15.6:0/2921402869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c14bc50 con 0x7f588c0cd060 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5890a00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c14b710 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f587c00c400 tx=0x7f587c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.683+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c017070 con 0x7f588c0cd060 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.684+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587c005ce0 con 0x7f588c0cd060 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.684+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c007310 con 0x7f588c0cd060 2024-03-06T07:29:08.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.684+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c1578a0 con 0x7f588c0cd060 2024-03-06T07:29:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.684+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c157e60 con 0x7f588c0cd060 2024-03-06T07:29:08.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.685+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c04f330 con 0x7f588c0cd060 2024-03-06T07:29:08.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.686+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f587c01a070 con 0x7f588c0cd060 2024-03-06T07:29:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.687+0000 7f5889600640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f586806b4d0 0x7f586806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.688+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f587c0896b0 con 0x7f588c0cd060 2024-03-06T07:29:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.688+0000 7f5883e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f586806b4d0 0x7f586806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.690+0000 7f5883e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f586806b4d0 0x7f586806d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5874036000 tx=0x7f5874030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:08.692+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f587c058490 con 0x7f588c0cd060 2024-03-06T07:29:09.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.002+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f588c0af8f0 con 0x7f588c0cd060 2024-03-06T07:29:09.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.003+0000 7f5889600640 1 -- 172.21.15.6:0/2921402869 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f587c057d90 con 0x7f588c0cd060 2024-03-06T07:29:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:09.005 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:29:09.006 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:29:09.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f586806b4d0 msgr2=0x7f586806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f586806b4d0 0x7f586806d990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5874036000 tx=0x7f5874030000 comp rx=0 tx=0).stop 2024-03-06T07:29:09.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 msgr2=0x7f588c14b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c14b710 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f587c00c400 tx=0x7f587c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 shutdown_connections 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f586806b4d0 0x7f586806d990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f588c1552b0 0x7f588c159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f588c15e810 0x7f588c0cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 --2- 172.21.15.6:0/2921402869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f588c0cd060 0x7f588c14b710 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 >> 172.21.15.6:0/2921402869 conn(0x7f588c06b3d0 msgr2=0x7f588c06e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 shutdown_connections 2024-03-06T07:29:09.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.006+0000 7f5891e00640 1 -- 172.21.15.6:0/2921402869 wait complete. 2024-03-06T07:29:09.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.234+0000 7f8b53200640 1 Processor -- start 2024-03-06T07:29:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.234+0000 7f8b53200640 1 -- start start 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b53200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b54066d00 0x7f8b5406ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b53200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b540ce370 0x7f8b540ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b53200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540cf1b0 con 0x7f8b5406c8b0 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b53200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5415e470 con 0x7f8b540ce370 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b53200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5415ea00 con 0x7f8b54066d00 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b52800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b540ce370 0x7f8b540ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b52800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8b540ce370 0x7f8b540ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53342/0 (socket says 172.21.15.6:53342) 2024-03-06T07:29:09.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.236+0000 7f8b52800640 1 -- 172.21.15.6:0/1744540006 learned_addr learned my addr 172.21.15.6:0/1744540006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.237+0000 7f8b4be00640 1 --2- 172.21.15.6:0/1744540006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b54066d00 0x7f8b5406ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.237+0000 7f8b52800640 1 -- 172.21.15.6:0/1744540006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b54066d00 msgr2=0x7f8b5406ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.237+0000 7f8b52800640 1 --2- 172.21.15.6:0/1744540006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b54066d00 0x7f8b5406ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.237+0000 7f8b52800640 1 -- 172.21.15.6:0/1744540006 >> v1:172.21.15.178:6789/0 conn(0x7f8b5406c8b0 legacy=0x7f8b5415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.237+0000 7f8b52800640 1 -- 172.21.15.6:0/1744540006 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5406d420 con 0x7f8b540ce370 2024-03-06T07:29:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.238+0000 7f8b52800640 1 --2- 172.21.15.6:0/1744540006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b540ce7e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009e50 tx=0x7f8b4c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf764d74422991 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.238+0000 7f8b51400640 1 -- 172.21.15.6:0/1744540006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c02cb00 con 0x7f8b540ce370 2024-03-06T07:29:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.238+0000 7f8b51400640 1 -- 172.21.15.6:0/1744540006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c02cca0 con 0x7f8b540ce370 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.238+0000 7f8b51400640 1 -- 172.21.15.6:0/1744540006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c034610 con 0x7f8b540ce370 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 -- 172.21.15.6:0/1744540006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 msgr2=0x7f8b540ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 --2- 172.21.15.6:0/1744540006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b540ce7e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009e50 tx=0x7f8b4c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:29:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 -- 172.21.15.6:0/1744540006 shutdown_connections 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 --2- 172.21.15.6:0/1744540006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b540ce7e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 --2- 172.21.15.6:0/1744540006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b54066d00 0x7f8b5406ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 -- 172.21.15.6:0/1744540006 >> 172.21.15.6:0/1744540006 conn(0x7f8b540ccf70 msgr2=0x7f8b540cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 -- 172.21.15.6:0/1744540006 shutdown_connections 2024-03-06T07:29:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.239+0000 7f8b53200640 1 -- 172.21.15.6:0/1744540006 wait complete. 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.241+0000 7f8b53200640 1 Processor -- start 2024-03-06T07:29:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.241+0000 7f8b53200640 1 -- start start 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.242+0000 7f8b53200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b54066d00 0x7f8b54146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b53200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b5406c8b0 0x7f8b5415f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b53200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b541e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b53200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540ceea0 con 0x7f8b540ce370 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b53200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540ced20 con 0x7f8b54066d00 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b53200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5406afb0 con 0x7f8b5406c8b0 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b52800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b541e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b52800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b541e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53356/0 (socket says 172.21.15.6:53356) 2024-03-06T07:29:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b52800640 1 -- 172.21.15.6:0/1154763356 learned_addr learned my addr 172.21.15.6:0/1154763356 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b51e00640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b54066d00 0x7f8b54146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b4be00640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b5406c8b0 0x7f8b5415f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.243+0000 7f8b52800640 1 -- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b5406c8b0 msgr2=0x7f8b5415f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b52800640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b5406c8b0 0x7f8b5415f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b52800640 1 -- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b54066d00 msgr2=0x7f8b54146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b52800640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b54066d00 0x7f8b54146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b52800640 1 -- 172.21.15.6:0/1154763356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5415fd40 con 0x7f8b540ce370 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b4be00640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b5406c8b0 0x7f8b5415f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b51e00640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b54066d00 0x7f8b54146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b52800640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b541e84f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009e50 tx=0x7f8b4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c043020 con 0x7f8b540ce370 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c02ce50 con 0x7f8b540ce370 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c009510 con 0x7f8b540ce370 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.244+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b54160020 con 0x7f8b540ce370 2024-03-06T07:29:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.245+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b541605e0 con 0x7f8b540ce370 2024-03-06T07:29:09.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.246+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5404f330 con 0x7f8b540ce370 2024-03-06T07:29:09.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.247+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b4c046020 con 0x7f8b540ce370 2024-03-06T07:29:09.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.248+0000 7f8b4a000640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8b3006b4d0 0x7f8b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.248+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b4c0ac920 con 0x7f8b540ce370 2024-03-06T07:29:09.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.248+0000 7f8b51e00640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8b3006b4d0 0x7f8b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.249+0000 7f8b51e00640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8b3006b4d0 0x7f8b3006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8b44001700 tx=0x7f8b44009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.258+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b4c04d030 con 0x7f8b540ce370 2024-03-06T07:29:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.464+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b540cdde0 con 0x7f8b540ce370 2024-03-06T07:29:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.466+0000 7f8b4a000640 1 -- 172.21.15.6:0/1154763356 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f8b4c07b700 con 0x7f8b540ce370 2024-03-06T07:29:09.469 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:09.469 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:41.851077+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8b3006b4d0 msgr2=0x7f8b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8b3006b4d0 0x7f8b3006d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8b44001700 tx=0x7f8b44009040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 msgr2=0x7f8b541e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b541e84f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009e50 tx=0x7f8b4c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 shutdown_connections 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8b3006b4d0 0x7f8b3006d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8b540ce370 0x7f8b541e84f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8b5406c8b0 0x7f8b5415f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 --2- 172.21.15.6:0/1154763356 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b54066d00 0x7f8b54146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 >> 172.21.15.6:0/1154763356 conn(0x7f8b540ccf70 msgr2=0x7f8b540b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 shutdown_connections 2024-03-06T07:29:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.469+0000 7f8b53200640 1 -- 172.21.15.6:0/1154763356 wait complete. 2024-03-06T07:29:09.490 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:38.380358+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:39.297336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:09.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:09.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.706+0000 7fc0dca00640 1 Processor -- start 2024-03-06T07:29:09.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.707+0000 7fc0dca00640 1 -- start start 2024-03-06T07:29:09.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0dca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0d80d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0dca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80d0080 con 0x7fc0d80d4a00 2024-03-06T07:29:09.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0dca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80cf500 con 0x7fc0d815e280 2024-03-06T07:29:09.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0dca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80cf7e0 con 0x7fc0d80d2990 2024-03-06T07:29:09.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0d6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc0d80d4a00 0x7fc0d80cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34678/0 (socket says 172.21.15.6:34678) 2024-03-06T07:29:09.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0d6800640 1 -- 172.21.15.6:0/1435874819 learned_addr learned my addr 172.21.15.6:0/1435874819 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:09.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.708+0000 7fc0d5400640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0d80d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506422186 0 0) 0x7fc0d80d0080 con 0x7fc0d80d4a00 2024-03-06T07:29:09.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc003520 con 0x7fc0d80d4a00 2024-03-06T07:29:09.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979144207 0 0) 0x7fc0d80cf500 con 0x7fc0d815e280 2024-03-06T07:29:09.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d80d0080 con 0x7fc0d815e280 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226322143 0 0) 0x7fc0bc003520 con 0x7fc0d80d4a00 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 msgr2=0x7fc0d80d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0d80d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 >> v1:172.21.15.178:6789/0 conn(0x7fc0d815e280 legacy=0x7fc0d80d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80cfac0 con 0x7fc0d80d4a00 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc0d00019f0 con 0x7fc0d80d4a00 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 >> v1:172.21.15.6:6789/0 conn(0x7fc0d80d4a00 legacy=0x7fc0d80cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0bc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0bc0040d0 0x7fc0bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80cf7e0 con 0x7fc0bc004d40 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc00bb50 con 0x7fc0bc0040d0 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.709+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0bc00bd20 con 0x7fc0d80d2990 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d6800640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d5400640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0bc0040d0 0x7fc0bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d5e00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0bc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d6800640 1 -- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 msgr2=0x7fc0bc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d6800640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0bc003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d6800640 1 -- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0bc0040d0 msgr2=0x7fc0bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d6800640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0bc0040d0 0x7fc0bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d6800640 1 -- 172.21.15.6:0/1435874819 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0bc003520 con 0x7fc0bc004d40 2024-03-06T07:29:09.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.710+0000 7fc0d5e00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0d80d2990 0x7fc0bc003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.711+0000 7fc0d6800640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000a810 tx=0x7fc0bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38faec995f9bae9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d000a430 con 0x7fc0bc004d40 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d0034650 con 0x7fc0bc004d40 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0d4a00640 1 -- 172.21.15.6:0/1435874819 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0034a80 con 0x7fc0bc004d40 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0dca00640 1 -- 172.21.15.6:0/1435874819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0bc004d40 msgr2=0x7fc0bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1435874819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0bc004d40 0x7fc0bc0051c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000a810 tx=0x7fc0bc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0d6800640 1 -- 172.21.15.6:0/1435874819 reap_dead start 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0dca00640 1 -- 172.21.15.6:0/1435874819 shutdown_connections 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0dca00640 1 -- 172.21.15.6:0/1435874819 >> 172.21.15.6:0/1435874819 conn(0x7fc0d80d3ee0 msgr2=0x7fc0d80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.712+0000 7fc0dca00640 1 -- 172.21.15.6:0/1435874819 shutdown_connections 2024-03-06T07:29:09.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.713+0000 7fc0dca00640 1 -- 172.21.15.6:0/1435874819 wait complete. 2024-03-06T07:29:09.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.715+0000 7fc0dca00640 1 Processor -- start 2024-03-06T07:29:09.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.716+0000 7fc0dca00640 1 -- start start 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.717+0000 7fc0dca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0bc004d40 0x7fc0d80c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.717+0000 7fc0dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 0x7fc0d80c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.717+0000 7fc0dca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0d8157930 0x7fc0d81607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.717+0000 7fc0dca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d815e660 con 0x7fc0d81574b0 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.717+0000 7fc0dca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80cb0e0 con 0x7fc0d8157930 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.717+0000 7fc0dca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80d1cc0 con 0x7fc0bc004d40 2024-03-06T07:29:09.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 0x7fc0d80c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 0x7fc0d80c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53378/0 (socket says 172.21.15.6:53378) 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 -- 172.21.15.6:0/1473553462 learned_addr learned my addr 172.21.15.6:0/1473553462 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5e00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0bc004d40 0x7fc0d80c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 -- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0bc004d40 msgr2=0x7fc0d80c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d6800640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0d8157930 0x7fc0d81607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0bc004d40 0x7fc0d80c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 -- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0d8157930 msgr2=0x7fc0d81607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0d8157930 0x7fc0d81607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 -- 172.21.15.6:0/1473553462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80c84b0 con 0x7fc0d81574b0 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5e00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0bc004d40 0x7fc0d80c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:09.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.718+0000 7fc0d5400640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 0x7fc0d80c7a10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4011600 tx=0x7fc0c4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.719+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c400f920 con 0x7fc0d81574b0 2024-03-06T07:29:09.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.719+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c400fac0 con 0x7fc0d81574b0 2024-03-06T07:29:09.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.719+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c400fe70 con 0x7fc0d81574b0 2024-03-06T07:29:09.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.719+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d80c8790 con 0x7fc0d81574b0 2024-03-06T07:29:09.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.719+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d80c8d50 con 0x7fc0d81574b0 2024-03-06T07:29:09.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.721+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d804f330 con 0x7fc0d81574b0 2024-03-06T07:29:09.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.721+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0c4015460 con 0x7fc0d81574b0 2024-03-06T07:29:09.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.722+0000 7fc0ce000640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0b806b4d0 0x7fc0b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.722+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0c408a850 con 0x7fc0d81574b0 2024-03-06T07:29:09.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.723+0000 7fc0d5e00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0b806b4d0 0x7fc0b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.724+0000 7fc0d5e00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0b806b4d0 0x7fc0b806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8000f60 tx=0x7fc0c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.727+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0c4019030 con 0x7fc0d81574b0 2024-03-06T07:29:09.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.960+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0d80c9a90 con 0x7fc0d81574b0 2024-03-06T07:29:09.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.961+0000 7fc0ce000640 1 -- 172.21.15.6:0/1473553462 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0c4059630 con 0x7fc0d81574b0 2024-03-06T07:29:09.963 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0b806b4d0 msgr2=0x7fc0b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0b806b4d0 0x7fc0b806d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8000f60 tx=0x7fc0c8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 msgr2=0x7fc0d80c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 0x7fc0d80c7a10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4011600 tx=0x7fc0c4011ac0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 shutdown_connections 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0b806b4d0 0x7fc0b806d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0d8157930 0x7fc0d81607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d81574b0 0x7fc0d80c7a10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 --2- 172.21.15.6:0/1473553462 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0bc004d40 0x7fc0d80c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.963+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 >> 172.21.15.6:0/1473553462 conn(0x7fc0d80d3ee0 msgr2=0x7fc0d806ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.964+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 shutdown_connections 2024-03-06T07:29:09.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:09.964+0000 7fc0dca00640 1 -- 172.21.15.6:0/1473553462 wait complete. 2024-03-06T07:29:09.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:10.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.193+0000 7f8218e00640 1 Processor -- start 2024-03-06T07:29:10.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.193+0000 7f8218e00640 1 -- start start 2024-03-06T07:29:10.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.195+0000 7f8218e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8214144560 0x7f821406e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.195+0000 7f8218e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8214147510 0x7f821406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.195+0000 7f8218e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f82140cd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.195+0000 7f8218e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8214162ae0 con 0x7f8214147510 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.195+0000 7f8218e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821406b930 con 0x7f8214144560 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.195+0000 7f8218e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8214161d20 con 0x7f821406b4f0 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8214147510 0x7f821406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8214147510 0x7f821406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53394/0 (socket says 172.21.15.6:53394) 2024-03-06T07:29:10.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 -- 172.21.15.6:0/2720793571 learned_addr learned my addr 172.21.15.6:0/2720793571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f8211e00640 1 --2- 172.21.15.6:0/2720793571 >> v2:172.21.15.178:3300/0 conn(0x7f8214144560 0x7f821406e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f8212800640 1 --2- 172.21.15.6:0/2720793571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f82140cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 -- 172.21.15.6:0/2720793571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 msgr2=0x7f82140cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 --2- 172.21.15.6:0/2720793571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f82140cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 -- 172.21.15.6:0/2720793571 >> v2:172.21.15.178:3300/0 conn(0x7f8214144560 msgr2=0x7f821406e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 --2- 172.21.15.6:0/2720793571 >> v2:172.21.15.178:3300/0 conn(0x7f8214144560 0x7f821406e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f820be00640 1 -- 172.21.15.6:0/2720793571 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8214162000 con 0x7f8214147510 2024-03-06T07:29:10.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.196+0000 7f8211e00640 1 --2- 172.21.15.6:0/2720793571 >> v2:172.21.15.178:3300/0 conn(0x7f8214144560 0x7f821406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.198+0000 7f820be00640 1 --2- 172.21.15.6:0/2720793571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214147510 0x7f821406afb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f82140b3580 tx=0x7f81fc02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b00bfdcf067950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.198+0000 7f8211400640 1 -- 172.21.15.6:0/2720793571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc0317d0 con 0x7f8214147510 2024-03-06T07:29:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.198+0000 7f8211400640 1 -- 172.21.15.6:0/2720793571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc037070 con 0x7f8214147510 2024-03-06T07:29:10.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.198+0000 7f8211400640 1 -- 172.21.15.6:0/2720793571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc031b90 con 0x7f8214147510 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 -- 172.21.15.6:0/2720793571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214147510 msgr2=0x7f821406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 --2- 172.21.15.6:0/2720793571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214147510 0x7f821406afb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f82140b3580 tx=0x7f81fc02b870 comp rx=0 tx=0).stop 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 -- 172.21.15.6:0/2720793571 shutdown_connections 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 --2- 172.21.15.6:0/2720793571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f82140cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 --2- 172.21.15.6:0/2720793571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214147510 0x7f821406afb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 --2- 172.21.15.6:0/2720793571 >> v2:172.21.15.178:3300/0 conn(0x7f8214144560 0x7f821406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 -- 172.21.15.6:0/2720793571 >> 172.21.15.6:0/2720793571 conn(0x7f82140b2860 msgr2=0x7f82140b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 -- 172.21.15.6:0/2720793571 shutdown_connections 2024-03-06T07:29:10.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.199+0000 7f8218e00640 1 -- 172.21.15.6:0/2720793571 wait complete. 2024-03-06T07:29:10.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.202+0000 7f8218e00640 1 Processor -- start 2024-03-06T07:29:10.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.202+0000 7f8218e00640 1 -- start start 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.203+0000 7f8218e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f821415f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8218e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 0x7f821415d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8218e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8214147510 0x7f82140cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8218e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821406efd0 con 0x7f8214144560 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8218e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82141625c0 con 0x7f8214147510 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8218e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821406ebc0 con 0x7f821406b4f0 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 0x7f821415d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 0x7f821415d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53402/0 (socket says 172.21.15.6:53402) 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 -- 172.21.15.6:0/1701148470 learned_addr learned my addr 172.21.15.6:0/1701148470 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:10.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8211e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f821415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8212800640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8214147510 0x7f82140cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 -- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 msgr2=0x7f821415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f821415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 -- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8214147510 msgr2=0x7f82140cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8214147510 0x7f82140cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f820be00640 1 -- 172.21.15.6:0/1701148470 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821415e6f0 con 0x7f8214144560 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.204+0000 7f8212800640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8214147510 0x7f82140cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:10.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.205+0000 7f820be00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 0x7f821415d880 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f81fc000c00 tx=0x7f81fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.205+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc032660 con 0x7f8214144560 2024-03-06T07:29:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.205+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81fc032840 con 0x7f8214144560 2024-03-06T07:29:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.205+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821415e960 con 0x7f8214144560 2024-03-06T07:29:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.205+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81fc032c30 con 0x7f8214144560 2024-03-06T07:29:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.205+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821415ef20 con 0x7f8214144560 2024-03-06T07:29:10.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.207+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821404f330 con 0x7f8214144560 2024-03-06T07:29:10.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.208+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81fc04d060 con 0x7f8214144560 2024-03-06T07:29:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.209+0000 7f820a000640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81f006b4d0 0x7f81f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.209+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81fc0ac390 con 0x7f8214144560 2024-03-06T07:29:10.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.210+0000 7f8211e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81f006b4d0 0x7f81f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.211+0000 7f8211e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81f006b4d0 0x7f81f006d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f820400a8d0 tx=0x7f8204008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.214+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81fc040070 con 0x7f8214144560 2024-03-06T07:29:10.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.415+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8214147bc0 con 0x7f8214144560 2024-03-06T07:29:10.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.416+0000 7f820a000640 1 -- 172.21.15.6:0/1701148470 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v35) v1 ==== 104+0+11 (secure 0 0 0) 0x7f81fc07b0e0 con 0x7f8214144560 2024-03-06T07:29:10.418 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:29:10.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81f006b4d0 msgr2=0x7f81f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81f006b4d0 0x7f81f006d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f820400a8d0 tx=0x7f8204008040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 msgr2=0x7f821415d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 0x7f821415d880 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f81fc000c00 tx=0x7f81fc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 shutdown_connections 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81f006b4d0 0x7f81f006d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8214147510 0x7f82140cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8214144560 0x7f821415d880 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 --2- 172.21.15.6:0/1701148470 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821406b4f0 0x7f821415f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 >> 172.21.15.6:0/1701148470 conn(0x7f82140b2860 msgr2=0x7f8214071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 shutdown_connections 2024-03-06T07:29:10.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.419+0000 7f8218e00640 1 -- 172.21.15.6:0/1701148470 wait complete. 2024-03-06T07:29:10.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:10.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.660+0000 7f0bcda00640 1 Processor -- start 2024-03-06T07:29:10.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.660+0000 7f0bcda00640 1 -- start start 2024-03-06T07:29:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.662+0000 7f0bcda00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0bc815acb0 0x7f0bc8066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.662+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc80b4730 0x7f0bc80b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.662+0000 7f0bcda00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc80afaa0 con 0x7f0bc806d330 2024-03-06T07:29:10.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.662+0000 7f0bcda00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc80cfdb0 con 0x7f0bc815acb0 2024-03-06T07:29:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.662+0000 7f0bcda00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80b50d0 con 0x7f0bc80b4730 2024-03-06T07:29:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bcd000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0bc806d330 0x7f0bc806d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34680/0 (socket says 172.21.15.6:34680) 2024-03-06T07:29:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bcd000640 1 -- 172.21.15.6:0/701634246 learned_addr learned my addr 172.21.15.6:0/701634246 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:10.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7e00640 1 --2- 172.21.15.6:0/701634246 >> v2:172.21.15.178:3300/0 conn(0x7f0bc815acb0 0x7f0bc8066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bbf400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc80b4730 0x7f0bc80b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136410105 0 0) 0x7f0bc80afaa0 con 0x7f0bc806d330 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bac003520 con 0x7f0bc806d330 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3176195633 0 0) 0x7f0bac003520 con 0x7f0bc806d330 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc80b4730 msgr2=0x7f0bc80b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc80b4730 0x7f0bc80b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 >> v2:172.21.15.178:3300/0 conn(0x7f0bc815acb0 msgr2=0x7f0bc8066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 --2- 172.21.15.6:0/701634246 >> v2:172.21.15.178:3300/0 conn(0x7f0bc815acb0 0x7f0bc8066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.663+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc80afd80 con 0x7f0bc806d330 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0bc0001670 con 0x7f0bc806d330 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 >> v1:172.21.15.6:6789/0 conn(0x7f0bc806d330 legacy=0x7f0bc806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bac0039b0 0x7f0bac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bac004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bac004ef0 0x7f0bac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80cfdb0 con 0x7f0bac0043a0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac00bf00 con 0x7f0bac004ef0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac00c080 con 0x7f0bac0039b0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bbf400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bac004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bc7e00640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bac0039b0 0x7f0bac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.664+0000 7f0bcd000640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bac004ef0 0x7f0bac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bbf400640 1 -- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bac0039b0 msgr2=0x7f0bac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bbf400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bac0039b0 0x7f0bac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bbf400640 1 -- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bac004ef0 msgr2=0x7f0bac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bbf400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bac004ef0 0x7f0bac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bbf400640 1 -- 172.21.15.6:0/701634246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac003520 con 0x7f0bac0043a0 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bc7e00640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bac0039b0 0x7f0bac003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.665+0000 7f0bcd000640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bac004ef0 0x7f0bac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.666+0000 7f0bbf400640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bac004880 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8072900 tx=0x7f0bac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2ae91b8db2d1d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.666+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0039070 con 0x7f0bac0043a0 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.666+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb002c580 con 0x7f0bac0043a0 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.666+0000 7f0bc7400640 1 -- 172.21.15.6:0/701634246 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0009650 con 0x7f0bac0043a0 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcda00640 1 -- 172.21.15.6:0/701634246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 msgr2=0x7f0bac004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcda00640 1 --2- 172.21.15.6:0/701634246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bac004880 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8072900 tx=0x7f0bac005830 comp rx=0 tx=0).stop 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcd000640 1 -- 172.21.15.6:0/701634246 reap_dead start 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcda00640 1 -- 172.21.15.6:0/701634246 shutdown_connections 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcda00640 1 -- 172.21.15.6:0/701634246 >> 172.21.15.6:0/701634246 conn(0x7f0bc80723d0 msgr2=0x7f0bc806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcda00640 1 -- 172.21.15.6:0/701634246 shutdown_connections 2024-03-06T07:29:10.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.667+0000 7f0bcda00640 1 -- 172.21.15.6:0/701634246 wait complete. 2024-03-06T07:29:10.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.669+0000 7f0bcda00640 1 Processor -- start 2024-03-06T07:29:10.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.669+0000 7f0bcda00640 1 -- start start 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bc813fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc8149120 0x7f0bc81499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bcda00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc81495a0 0x7f0bc80b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bcda00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc8067240 con 0x7f0bac0043a0 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bcda00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80cf920 con 0x7f0bc8149120 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bcda00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80ccff0 con 0x7f0bc81495a0 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bc7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bc813fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bc7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bc813fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53426/0 (socket says 172.21.15.6:53426) 2024-03-06T07:29:10.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.670+0000 7f0bc7e00640 1 -- 172.21.15.6:0/3409168879 learned_addr learned my addr 172.21.15.6:0/3409168879 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bbf400640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc8149120 0x7f0bc81499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bcd000640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc81495a0 0x7f0bc80b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc7e00640 1 -- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc81495a0 msgr2=0x7f0bc80b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc7e00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc81495a0 0x7f0bc80b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc7e00640 1 -- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc8149120 msgr2=0x7f0bc81499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc7e00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc8149120 0x7f0bc81499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc7e00640 1 -- 172.21.15.6:0/3409168879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8140160 con 0x7f0bac0043a0 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bbf400640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc8149120 0x7f0bc81499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:10.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc7e00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bc813fc20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80101c0 tx=0x7f0bb8010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb801a030 con 0x7f0bac0043a0 2024-03-06T07:29:10.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.671+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb8003990 con 0x7f0bac0043a0 2024-03-06T07:29:10.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.672+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8003d60 con 0x7f0bac0043a0 2024-03-06T07:29:10.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.672+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc8146160 con 0x7f0bac0043a0 2024-03-06T07:29:10.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.672+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc806e5a0 con 0x7f0bac0043a0 2024-03-06T07:29:10.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.673+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc804f330 con 0x7f0bac0043a0 2024-03-06T07:29:10.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.674+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bb8005d10 con 0x7f0bac0043a0 2024-03-06T07:29:10.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.675+0000 7f0bc4c00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b4d0 0x7f0ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:10.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.675+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bb808b2e0 con 0x7f0bac0043a0 2024-03-06T07:29:10.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.675+0000 7f0bbf400640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b4d0 0x7f0ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:10.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.677+0000 7f0bbf400640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b4d0 0x7f0ba406d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0036000 tx=0x7f0bb0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:10.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.684+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bb8024070 con 0x7f0bac0043a0 2024-03-06T07:29:10.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.907+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0bc806c2e0 con 0x7f0bac0043a0 2024-03-06T07:29:10.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.908+0000 7f0bc4c00640 1 -- 172.21.15.6:0/3409168879 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v35) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0bb801de40 con 0x7f0bac0043a0 2024-03-06T07:29:10.910 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b4d0 msgr2=0x7f0ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b4d0 0x7f0ba406d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0036000 tx=0x7f0bb0030000 comp rx=0 tx=0).stop 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 msgr2=0x7f0bc813fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bc813fc20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80101c0 tx=0x7f0bb8010680 comp rx=0 tx=0).stop 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 shutdown_connections 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b4d0 0x7f0ba406d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc8149120 0x7f0bc81499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc81495a0 0x7f0bc80b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.910+0000 7f0bcda00640 1 --2- 172.21.15.6:0/3409168879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bac0043a0 0x7f0bc813fc20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.911+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 >> 172.21.15.6:0/3409168879 conn(0x7f0bc80723d0 msgr2=0x7f0bc80b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.911+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 shutdown_connections 2024-03-06T07:29:10.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:10.911+0000 7f0bcda00640 1 -- 172.21.15.6:0/3409168879 wait complete. 2024-03-06T07:29:10.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:11.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.127+0000 7f11b5800640 1 Processor -- start 2024-03-06T07:29:11.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.128+0000 7f11b5800640 1 -- start start 2024-03-06T07:29:11.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.129+0000 7f11b5800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f11b015e0b0 0x7f11b00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.129+0000 7f11b5800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11b00d4a10 0x7f11b00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.129+0000 7f11b5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b00b3710 con 0x7f11b00d0c80 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.129+0000 7f11b5800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00d10f0 con 0x7f11b00d4a10 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.129+0000 7f11b5800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00b09e0 con 0x7f11b015e0b0 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11af400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11b00d4a10 0x7f11b00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11b4e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f11b00d0c80 0x7f11b00d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34682/0 (socket says 172.21.15.6:34682) 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11b4e00640 1 -- 172.21.15.6:0/2592108861 learned_addr learned my addr 172.21.15.6:0/2592108861 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11afe00640 1 --2- 172.21.15.6:0/2592108861 >> v2:172.21.15.178:3300/0 conn(0x7f11b015e0b0 0x7f11b00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11af400640 1 -- 172.21.15.6:0/2592108861 >> v2:172.21.15.178:3300/0 conn(0x7f11b015e0b0 msgr2=0x7f11b00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11af400640 1 --2- 172.21.15.6:0/2592108861 >> v2:172.21.15.178:3300/0 conn(0x7f11b015e0b0 0x7f11b00cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11af400640 1 -- 172.21.15.6:0/2592108861 >> v1:172.21.15.6:6789/0 conn(0x7f11b00d0c80 legacy=0x7f11b00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11af400640 1 -- 172.21.15.6:0/2592108861 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b00b39f0 con 0x7f11b00d4a10 2024-03-06T07:29:11.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.130+0000 7f11aea00640 1 -- 172.21.15.6:0/2592108861 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645191387 0 0) 0x7f11b00b3710 con 0x7f11b00d0c80 2024-03-06T07:29:11.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.132+0000 7f11af400640 1 --2- 172.21.15.6:0/2592108861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d4a10 0x7f11b00d0740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f11b00b68c0 tx=0x7f119802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e29358ece5208ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.132+0000 7f11aea00640 1 -- 172.21.15.6:0/2592108861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198035020 con 0x7f11b00d4a10 2024-03-06T07:29:11.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.132+0000 7f11aea00640 1 -- 172.21.15.6:0/2592108861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198031830 con 0x7f11b00d4a10 2024-03-06T07:29:11.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.132+0000 7f11aea00640 1 -- 172.21.15.6:0/2592108861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198031c00 con 0x7f11b00d4a10 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 -- 172.21.15.6:0/2592108861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d4a10 msgr2=0x7f11b00d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 --2- 172.21.15.6:0/2592108861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d4a10 0x7f11b00d0740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f11b00b68c0 tx=0x7f119802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 -- 172.21.15.6:0/2592108861 shutdown_connections 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 --2- 172.21.15.6:0/2592108861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d4a10 0x7f11b00d0740 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 --2- 172.21.15.6:0/2592108861 >> v2:172.21.15.178:3300/0 conn(0x7f11b015e0b0 0x7f11b00cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 -- 172.21.15.6:0/2592108861 >> 172.21.15.6:0/2592108861 conn(0x7f11b00d3d30 msgr2=0x7f11b00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 -- 172.21.15.6:0/2592108861 shutdown_connections 2024-03-06T07:29:11.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.133+0000 7f11b5800640 1 -- 172.21.15.6:0/2592108861 wait complete. 2024-03-06T07:29:11.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.135+0000 7f11b5800640 1 Processor -- start 2024-03-06T07:29:11.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.135+0000 7f11b5800640 1 -- start start 2024-03-06T07:29:11.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11b5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 0x7f11b015f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11b5800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b00d4a10 0x7f11b00cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11b5800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f11b015e0b0 0x7f11b00b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11b5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b015e490 con 0x7f11b00d0c80 2024-03-06T07:29:11.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11b5800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b00b3fb0 con 0x7f11b00d4a10 2024-03-06T07:29:11.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11b5800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b00cfc60 con 0x7f11b015e0b0 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 0x7f11b015f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 0x7f11b015f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53452/0 (socket says 172.21.15.6:53452) 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.137+0000 7f11afe00640 1 -- 172.21.15.6:0/3195097528 learned_addr learned my addr 172.21.15.6:0/3195097528 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11afe00640 1 -- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f11b015e0b0 msgr2=0x7f11b00b4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11afe00640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f11b015e0b0 0x7f11b00b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11afe00640 1 -- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b00d4a10 msgr2=0x7f11b00cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11af400640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b00d4a10 0x7f11b00cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11afe00640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b00d4a10 0x7f11b00cc2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11afe00640 1 -- 172.21.15.6:0/3195097528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b01f4970 con 0x7f11b00d0c80 2024-03-06T07:29:11.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11afe00640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 0x7f11b015f290 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f11a000efc0 tx=0x7f11a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0018020 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a0014480 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0014850 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b015e7b0 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.138+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b00b4cb0 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.140+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b0068a70 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.142+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f11a0005d10 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.143+0000 7f11a7e00640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f118c06b500 0x7f118c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.143+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11a008ac90 con 0x7f11b00d0c80 2024-03-06T07:29:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.143+0000 7f11af400640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f118c06b500 0x7f118c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.145+0000 7f11af400640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f118c06b500 0x7f118c06d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1198036000 tx=0x7f1198008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.148+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11b0068a70 con 0x7f11b00d0c80 2024-03-06T07:29:11.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.471+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b004f330 con 0x7f11b00d0c80 2024-03-06T07:29:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.472+0000 7f11a7e00640 1 -- 172.21.15.6:0/3195097528 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f11a0022070 con 0x7f11b00d0c80 2024-03-06T07:29:11.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:11.475 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:29:11.475 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:29:11.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f118c06b500 msgr2=0x7f118c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f118c06b500 0x7f118c06d9c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1198036000 tx=0x7f1198008030 comp rx=0 tx=0).stop 2024-03-06T07:29:11.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 msgr2=0x7f11b015f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 0x7f11b015f290 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f11a000efc0 tx=0x7f11a000c490 comp rx=0 tx=0).stop 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 shutdown_connections 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f118c06b500 0x7f118c06d9c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f11b00d4a10 0x7f11b00cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f11b015e0b0 0x7f11b00b4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 --2- 172.21.15.6:0/3195097528 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11b00d0c80 0x7f11b015f290 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 >> 172.21.15.6:0/3195097528 conn(0x7f11b00d3d30 msgr2=0x7f11b00b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 shutdown_connections 2024-03-06T07:29:11.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.475+0000 7f11b5800640 1 -- 172.21.15.6:0/3195097528 wait complete. 2024-03-06T07:29:11.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.689+0000 7f9bb9000640 1 Processor -- start 2024-03-06T07:29:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.689+0000 7f9bb9000640 1 -- start start 2024-03-06T07:29:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.691+0000 7f9bb9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9bb40cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.691+0000 7f9bb9000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb40cf1b0 con 0x7f9bb406c8b0 2024-03-06T07:29:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.691+0000 7f9bb9000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb406d420 con 0x7f9bb4066d00 2024-03-06T07:29:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.691+0000 7f9bb9000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb415e470 con 0x7f9bb406ef10 2024-03-06T07:29:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.691+0000 7f9bb3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9bb406c8b0 0x7f9bb415ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34694/0 (socket says 172.21.15.6:34694) 2024-03-06T07:29:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.691+0000 7f9bb3400640 1 -- 172.21.15.6:0/2046365696 learned_addr learned my addr 172.21.15.6:0/2046365696 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb3e00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9bb40cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082412847 0 0) 0x7f9bb40cf1b0 con 0x7f9bb406c8b0 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b98003520 con 0x7f9bb406c8b0 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428245464 0 0) 0x7f9b98003520 con 0x7f9bb406c8b0 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 msgr2=0x7f9bb40cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9bb40cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 >> v1:172.21.15.178:6789/0 conn(0x7f9bb4066d00 legacy=0x7f9bb40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb415e690 con 0x7f9bb406c8b0 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9ba4001670 con 0x7f9bb406c8b0 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 >> v1:172.21.15.6:6789/0 conn(0x7f9bb406c8b0 legacy=0x7f9bb415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b980039e0 0x7f9b98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98004800 0x7f9b98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.692+0000 7f9bb2000640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9b980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb415e470 con 0x7f9b98004800 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9800baa0 con 0x7f9b980039e0 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9800bca0 con 0x7f9bb406ef10 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2a00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98004800 0x7f9b98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb3e00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9b980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb3400640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b980039e0 0x7f9b98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2a00640 1 -- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 msgr2=0x7f9b980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2a00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9b980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2a00640 1 -- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b980039e0 msgr2=0x7f9b98003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2a00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b980039e0 0x7f9b98003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb2a00640 1 -- 172.21.15.6:0/2046365696 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98003520 con 0x7f9b98004800 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb3e00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9bb406ef10 0x7f9b980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.693+0000 7f9bb3400640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b980039e0 0x7f9b98003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.694+0000 7f9bb2a00640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98004800 0x7f9b98004c80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0001720 tx=0x7f9b98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800ad965906416b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.694+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0001370 con 0x7f9b98004800 2024-03-06T07:29:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.694+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba00052d0 con 0x7f9b98004800 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb2000640 1 -- 172.21.15.6:0/2046365696 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0003830 con 0x7f9b98004800 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb9000640 1 -- 172.21.15.6:0/2046365696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98004800 msgr2=0x7f9b98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb9000640 1 --2- 172.21.15.6:0/2046365696 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b98004800 0x7f9b98004c80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0001720 tx=0x7f9b98005830 comp rx=0 tx=0).stop 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb3e00640 1 -- 172.21.15.6:0/2046365696 reap_dead start 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb9000640 1 -- 172.21.15.6:0/2046365696 shutdown_connections 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb9000640 1 -- 172.21.15.6:0/2046365696 >> 172.21.15.6:0/2046365696 conn(0x7f9bb40ccf70 msgr2=0x7f9bb40cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb9000640 1 -- 172.21.15.6:0/2046365696 shutdown_connections 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.695+0000 7f9bb9000640 1 -- 172.21.15.6:0/2046365696 wait complete. 2024-03-06T07:29:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.697+0000 7f9bb9000640 1 Processor -- start 2024-03-06T07:29:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.697+0000 7f9bb9000640 1 -- start start 2024-03-06T07:29:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b98004800 0x7f9bb4154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb9000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9bb415d030 0x7f9bb40ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 0x7f9bb41e9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40ceed0 con 0x7f9bb41e9180 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb9000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40ced50 con 0x7f9bb415d030 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb9000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb415ea00 con 0x7f9b98004800 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.698+0000 7f9bb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 0x7f9bb41e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 0x7f9bb41e9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53468/0 (socket says 172.21.15.6:53468) 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 -- 172.21.15.6:0/1071358546 learned_addr learned my addr 172.21.15.6:0/1071358546 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3400640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b98004800 0x7f9bb4154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb2a00640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9bb415d030 0x7f9bb40ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 -- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b98004800 msgr2=0x7f9bb4154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b98004800 0x7f9bb4154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 -- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9bb415d030 msgr2=0x7f9bb40ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9bb415d030 0x7f9bb40ae290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 -- 172.21.15.6:0/1071358546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb41f5330 con 0x7f9bb41e9180 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3400640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b98004800 0x7f9bb4154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bb3e00640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 0x7f9bb41e9600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4067a30 tx=0x7f9bac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.699+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac01c070 con 0x7f9bb41e9180 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.700+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac017400 con 0x7f9bb41e9180 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.700+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0177d0 con 0x7f9bb41e9180 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.700+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb40aeae0 con 0x7f9bb41e9180 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.700+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb40af0a0 con 0x7f9bb41e9180 2024-03-06T07:29:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.701+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb406e250 con 0x7f9bb41e9180 2024-03-06T07:29:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.702+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bac005d10 con 0x7f9bb41e9180 2024-03-06T07:29:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.703+0000 7f9bab400640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b9406b500 0x7f9b9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.703+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bac08a3a0 con 0x7f9bb41e9180 2024-03-06T07:29:11.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.703+0000 7f9bb3400640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b9406b500 0x7f9b9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.704+0000 7f9bb3400640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b9406b500 0x7f9b9406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4157c50 tx=0x7f9ba4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.707+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bb406e250 con 0x7f9bb41e9180 2024-03-06T07:29:11.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.985+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bb404f330 con 0x7f9bb41e9180 2024-03-06T07:29:11.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.987+0000 7f9bab400640 1 -- 172.21.15.6:0/1071358546 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f9bac012070 con 0x7f9bb41e9180 2024-03-06T07:29:11.989 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:11.990 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:41.851077+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.989+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b9406b500 msgr2=0x7f9b9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b9406b500 0x7f9b9406d9c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4157c50 tx=0x7f9ba4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 msgr2=0x7f9bb41e9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 0x7f9bb41e9600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4067a30 tx=0x7f9bac00f4a0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 shutdown_connections 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b9406b500 0x7f9b9406d9c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9bb415d030 0x7f9bb40ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9bb41e9180 0x7f9bb41e9600 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 --2- 172.21.15.6:0/1071358546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b98004800 0x7f9bb4154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 >> 172.21.15.6:0/1071358546 conn(0x7f9bb40ccf70 msgr2=0x7f9bb40b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 shutdown_connections 2024-03-06T07:29:11.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:11.990+0000 7f9bb9000640 1 -- 172.21.15.6:0/1071358546 wait complete. 2024-03-06T07:29:12.010 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:12.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.212+0000 7f45c1a00640 1 Processor -- start 2024-03-06T07:29:12.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.212+0000 7f45c1a00640 1 -- start start 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.213+0000 7f45c1a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45bc15e2e0 0x7f45bc0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45c1a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d0d70 0x7f45bc0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45c1a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f45bc0d1b30 0x7f45bc0d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45c1a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc0d4e70 con 0x7f45bc15e2e0 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45c1a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45bc0b3c40 con 0x7f45bc0d1b30 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45c1a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0b3940 con 0x7f45bc0d0d70 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45bc15e2e0 0x7f45bc0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45bc15e2e0 0x7f45bc0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51440/0 (socket says 172.21.15.6:51440) 2024-03-06T07:29:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 -- 172.21.15.6:0/2844469051 learned_addr learned my addr 172.21.15.6:0/2844469051 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 -- 172.21.15.6:0/2844469051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d0d70 msgr2=0x7f45bc0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45ba000640 1 --2- 172.21.15.6:0/2844469051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d0d70 0x7f45bc0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45bb400640 1 --2- 172.21.15.6:0/2844469051 >> v2:172.21.15.178:3300/0 conn(0x7f45bc0d1b30 0x7f45bc0d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 --2- 172.21.15.6:0/2844469051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d0d70 0x7f45bc0d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 -- 172.21.15.6:0/2844469051 >> v2:172.21.15.178:3300/0 conn(0x7f45bc0d1b30 msgr2=0x7f45bc0d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 --2- 172.21.15.6:0/2844469051 >> v2:172.21.15.178:3300/0 conn(0x7f45bc0d1b30 0x7f45bc0d1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45baa00640 1 -- 172.21.15.6:0/2844469051 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0cf580 con 0x7f45bc15e2e0 2024-03-06T07:29:12.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.214+0000 7f45bb400640 1 --2- 172.21.15.6:0/2844469051 >> v2:172.21.15.178:3300/0 conn(0x7f45bc0d1b30 0x7f45bc0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.216+0000 7f45baa00640 1 --2- 172.21.15.6:0/2844469051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0d0830 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0d37f0 tx=0x7f45ac02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7288e0cd269b9ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.216+0000 7f45b9600640 1 -- 172.21.15.6:0/2844469051 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac035020 con 0x7f45bc15e2e0 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.216+0000 7f45b9600640 1 -- 172.21.15.6:0/2844469051 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ac02d040 con 0x7f45bc15e2e0 2024-03-06T07:29:12.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.216+0000 7f45b9600640 1 -- 172.21.15.6:0/2844469051 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac031a80 con 0x7f45bc15e2e0 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.216+0000 7f45c1a00640 1 -- 172.21.15.6:0/2844469051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 msgr2=0x7f45bc0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.216+0000 7f45c1a00640 1 --2- 172.21.15.6:0/2844469051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0d0830 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0d37f0 tx=0x7f45ac02b900 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 -- 172.21.15.6:0/2844469051 shutdown_connections 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 --2- 172.21.15.6:0/2844469051 >> v2:172.21.15.178:3300/0 conn(0x7f45bc0d1b30 0x7f45bc0d1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 --2- 172.21.15.6:0/2844469051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d0d70 0x7f45bc0d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 --2- 172.21.15.6:0/2844469051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0d0830 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 -- 172.21.15.6:0/2844469051 >> 172.21.15.6:0/2844469051 conn(0x7f45bc0d3f60 msgr2=0x7f45bc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 -- 172.21.15.6:0/2844469051 shutdown_connections 2024-03-06T07:29:12.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.217+0000 7f45c1a00640 1 -- 172.21.15.6:0/2844469051 wait complete. 2024-03-06T07:29:12.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.218+0000 7f45c1a00640 1 Processor -- start 2024-03-06T07:29:12.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 -- start start 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45bc0d0d70 0x7f45bc1504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d1b30 0x7f45bc0b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0cfe20 con 0x7f45bc15e2e0 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0cfb40 con 0x7f45bc0d0d70 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.219+0000 7f45c1a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0d2500 con 0x7f45bc0d1b30 2024-03-06T07:29:12.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0b8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51450/0 (socket says 172.21.15.6:51450) 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 -- 172.21.15.6:0/3146961507 learned_addr learned my addr 172.21.15.6:0/3146961507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45ba000640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d1b30 0x7f45bc0b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45baa00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45bc0d0d70 0x7f45bc1504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 -- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d1b30 msgr2=0x7f45bc0b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d1b30 0x7f45bc0b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 -- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45bc0d0d70 msgr2=0x7f45bc1504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45bc0d0d70 0x7f45bc1504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.220+0000 7f45bb400640 1 -- 172.21.15.6:0/3146961507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc0b7560 con 0x7f45bc15e2e0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.221+0000 7f45bb400640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0b8eb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45b400cf20 tx=0x7f45b400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.221+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b401a070 con 0x7f45bc15e2e0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.221+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b400d840 con 0x7f45bc15e2e0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.221+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b400dc30 con 0x7f45bc15e2e0 2024-03-06T07:29:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.221+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc0b7840 con 0x7f45bc15e2e0 2024-03-06T07:29:12.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.221+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc0b7e00 con 0x7f45bc15e2e0 2024-03-06T07:29:12.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.222+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc06d840 con 0x7f45bc15e2e0 2024-03-06T07:29:12.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.223+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45b401d070 con 0x7f45bc15e2e0 2024-03-06T07:29:12.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.225+0000 7f45b2a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f459c06b500 0x7f459c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.225+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45b408aaa0 con 0x7f45bc15e2e0 2024-03-06T07:29:12.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.226+0000 7f45baa00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f459c06b500 0x7f459c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.227+0000 7f45baa00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f459c06b500 0x7f459c06d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f45ac036000 tx=0x7f45ac03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.229+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45bc06d840 con 0x7f45bc15e2e0 2024-03-06T07:29:12.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.469+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45bc073210 con 0x7f45bc15e2e0 2024-03-06T07:29:12.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.470+0000 7f45b2a00640 1 -- 172.21.15.6:0/3146961507 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f45bc073210 con 0x7f45bc15e2e0 2024-03-06T07:29:12.472 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:12.472 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:29:12.473 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:29:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f459c06b500 msgr2=0x7f459c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f459c06b500 0x7f459c06d9c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f45ac036000 tx=0x7f45ac03c040 comp rx=0 tx=0).stop 2024-03-06T07:29:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 msgr2=0x7f45bc0b8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0b8eb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45b400cf20 tx=0x7f45b400a7d0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 shutdown_connections 2024-03-06T07:29:12.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f459c06b500 0x7f459c06d9c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45bc15e2e0 0x7f45bc0b8eb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45bc0d1b30 0x7f45bc0b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 --2- 172.21.15.6:0/3146961507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45bc0d0d70 0x7f45bc1504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 >> 172.21.15.6:0/3146961507 conn(0x7f45bc0d3f60 msgr2=0x7f45bc0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 shutdown_connections 2024-03-06T07:29:12.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.473+0000 7f45c1a00640 1 -- 172.21.15.6:0/3146961507 wait complete. 2024-03-06T07:29:12.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.733+0000 7fe5f6200640 1 Processor -- start 2024-03-06T07:29:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.733+0000 7fe5f6200640 1 -- start start 2024-03-06T07:29:12.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f6200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5f01452e0 0x7fe5f00ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f6200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f015f760 0x7fe5f006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f6200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f00b6050 con 0x7fe5f006b4f0 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f6200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f006b940 con 0x7fe5f01452e0 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f6200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f015fbc0 con 0x7fe5f015f760 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5f01452e0 0x7fe5f00ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5f01452e0 0x7fe5f00ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51460/0 (socket says 172.21.15.6:51460) 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f5800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe5f006b4f0 0x7fe5f0071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33048/0 (socket says 172.21.15.6:33048) 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.735+0000 7fe5f4e00640 1 -- 172.21.15.6:0/1259672339 learned_addr learned my addr 172.21.15.6:0/1259672339 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.736+0000 7fe5efe00640 1 --2- 172.21.15.6:0/1259672339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f015f760 0x7fe5f006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.736+0000 7fe5f4e00640 1 -- 172.21.15.6:0/1259672339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f015f760 msgr2=0x7fe5f006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.736+0000 7fe5f4e00640 1 --2- 172.21.15.6:0/1259672339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f015f760 0x7fe5f006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.736+0000 7fe5f4e00640 1 -- 172.21.15.6:0/1259672339 >> v1:172.21.15.6:6789/0 conn(0x7fe5f006b4f0 legacy=0x7fe5f0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.736+0000 7fe5f4e00640 1 -- 172.21.15.6:0/1259672339 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f015fe90 con 0x7fe5f01452e0 2024-03-06T07:29:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.737+0000 7fe5f4e00640 1 --2- 172.21.15.6:0/1259672339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f01452e0 0x7fe5f00ce7b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009870 tx=0x7fe5e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b791c972aa3cddbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.737+0000 7fe5ef400640 1 -- 172.21.15.6:0/1259672339 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4034070 con 0x7fe5f01452e0 2024-03-06T07:29:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.737+0000 7fe5ef400640 1 -- 172.21.15.6:0/1259672339 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e4030840 con 0x7fe5f01452e0 2024-03-06T07:29:12.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.737+0000 7fe5ef400640 1 -- 172.21.15.6:0/1259672339 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4030c10 con 0x7fe5f01452e0 2024-03-06T07:29:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 -- 172.21.15.6:0/1259672339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f01452e0 msgr2=0x7fe5f00ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 --2- 172.21.15.6:0/1259672339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f01452e0 0x7fe5f00ce7b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009870 tx=0x7fe5e402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 -- 172.21.15.6:0/1259672339 shutdown_connections 2024-03-06T07:29:12.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 --2- 172.21.15.6:0/1259672339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f015f760 0x7fe5f006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 --2- 172.21.15.6:0/1259672339 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f01452e0 0x7fe5f00ce7b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 -- 172.21.15.6:0/1259672339 >> 172.21.15.6:0/1259672339 conn(0x7fe5f006c0c0 msgr2=0x7fe5f006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 -- 172.21.15.6:0/1259672339 shutdown_connections 2024-03-06T07:29:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.738+0000 7fe5f6200640 1 -- 172.21.15.6:0/1259672339 wait complete. 2024-03-06T07:29:12.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.740+0000 7fe5f6200640 1 Processor -- start 2024-03-06T07:29:12.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.740+0000 7fe5f6200640 1 -- start start 2024-03-06T07:29:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f6200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f006b4f0 0x7fe5f01423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f6200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5f01452e0 0x7fe5f0135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f6200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 0x7fe5f015e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f6200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f00b4b20 con 0x7fe5f015f760 2024-03-06T07:29:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f6200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0160450 con 0x7fe5f01452e0 2024-03-06T07:29:12.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f6200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f015c1f0 con 0x7fe5f006b4f0 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5f5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 0x7fe5f015e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.741+0000 7fe5efe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5f01452e0 0x7fe5f0135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f006b4f0 0x7fe5f01423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 0x7fe5f015e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51462/0 (socket says 172.21.15.6:51462) 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 -- 172.21.15.6:0/3752506346 learned_addr learned my addr 172.21.15.6:0/3752506346 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 -- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f006b4f0 msgr2=0x7fe5f01423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f006b4f0 0x7fe5f01423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 -- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5f01452e0 msgr2=0x7fe5f0135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5f01452e0 0x7fe5f0135250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 -- 172.21.15.6:0/3752506346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f013f790 con 0x7fe5f015f760 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.742+0000 7fe5f5800640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 0x7fe5f015e470 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c730 tx=0x7fe5e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.743+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8018070 con 0x7fe5f015f760 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.743+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e8004d80 con 0x7fe5f015f760 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.743+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0140700 con 0x7fe5f015f760 2024-03-06T07:29:12.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.743+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e800a880 con 0x7fe5f015f760 2024-03-06T07:29:12.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.743+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f01409a0 con 0x7fe5f015f760 2024-03-06T07:29:12.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.744+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f006f900 con 0x7fe5f015f760 2024-03-06T07:29:12.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.745+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5e801b070 con 0x7fe5f015f760 2024-03-06T07:29:12.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.746+0000 7fe5ecc00640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5d006b4d0 0x7fe5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.747+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5e8089770 con 0x7fe5f015f760 2024-03-06T07:29:12.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.747+0000 7fe5f4e00640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5d006b4d0 0x7fe5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.748+0000 7fe5f4e00640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5d006b4d0 0x7fe5d006d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4036000 tx=0x7fe5e4008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:12.750+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5f006f900 con 0x7fe5f015f760 2024-03-06T07:29:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.004+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5f00aff80 con 0x7fe5f015f760 2024-03-06T07:29:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.005+0000 7fe5ecc00640 1 -- 172.21.15.6:0/3752506346 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fe5f00aff80 con 0x7fe5f015f760 2024-03-06T07:29:13.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:13.008 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:40.283024+0000","modified":"2024-03-06T07:28:43.390057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5124},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5124":{"gid":5124,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/2554316293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2554316293},{"type":"v1","addr":"172.21.15.6:6835","nonce":2554316293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5124,"qdb_cluster":[5124]},"id":2}]} 2024-03-06T07:29:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 17 2024-03-06T07:29:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5d006b4d0 msgr2=0x7fe5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5d006b4d0 0x7fe5d006d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4036000 tx=0x7fe5e4008030 comp rx=0 tx=0).stop 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 msgr2=0x7fe5f015e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 0x7fe5f015e470 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe5e800c730 tx=0x7fe5e800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 shutdown_connections 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5d006b4d0 0x7fe5d006d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5f01452e0 0x7fe5f0135250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5f015f760 0x7fe5f015e470 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 --2- 172.21.15.6:0/3752506346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5f006b4f0 0x7fe5f01423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 >> 172.21.15.6:0/3752506346 conn(0x7fe5f006c0c0 msgr2=0x7fe5f00ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 shutdown_connections 2024-03-06T07:29:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.008+0000 7fe5f6200640 1 -- 172.21.15.6:0/3752506346 wait complete. 2024-03-06T07:29:13.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:13.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.242+0000 7f2aa7c00640 1 Processor -- start 2024-03-06T07:29:13.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.243+0000 7f2aa7c00640 1 -- start start 2024-03-06T07:29:13.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.244+0000 7f2aa7c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 0x7f2aa015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.244+0000 7f2aa7c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa015cc40 con 0x7f2aa00cdc20 2024-03-06T07:29:13.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.244+0000 7f2aa7c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa015cf20 con 0x7f2aa00af1a0 2024-03-06T07:29:13.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.244+0000 7f2aa7c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa015d200 con 0x7f2aa00af580 2024-03-06T07:29:13.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.245+0000 7f2aa5c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2aa00cdc20 0x7f2aa00b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33054/0 (socket says 172.21.15.6:33054) 2024-03-06T07:29:13.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.245+0000 7f2aa5c00640 1 -- 172.21.15.6:0/2958242325 learned_addr learned my addr 172.21.15.6:0/2958242325 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.245+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 0x7f2aa015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.245+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053007202 0 0) 0x7f2aa015cc40 con 0x7f2aa00cdc20 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.245+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a88003520 con 0x7f2aa00cdc20 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1863764656 0 0) 0x7f2a88003520 con 0x7f2aa00cdc20 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 msgr2=0x7f2aa015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 0x7f2aa015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 >> v1:172.21.15.178:6789/0 conn(0x7f2aa00af1a0 legacy=0x7f2aa00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa015d4e0 con 0x7f2aa00cdc20 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2a980019f0 con 0x7f2aa00cdc20 2024-03-06T07:29:13.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 >> v1:172.21.15.6:6789/0 conn(0x7f2aa00cdc20 legacy=0x7f2aa00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a88003a50 0x7f2a88003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a88004800 0x7f2a88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 0x7f2a880052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa015d200 con 0x7f2a88003a50 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8800bba0 con 0x7f2a88004800 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8800bda0 con 0x7f2aa00af580 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.246+0000 7f2aa5200640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a88003a50 0x7f2a88003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a88004800 0x7f2a88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2aa5c00640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 0x7f2a880052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2aa5200640 1 -- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 msgr2=0x7f2a880052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2aa5200640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00af580 0x7f2a880052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2aa5200640 1 -- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a88004800 msgr2=0x7f2a88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2aa5200640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2a88004800 0x7f2a88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.247+0000 7f2aa5200640 1 -- 172.21.15.6:0/2958242325 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a88003520 con 0x7f2a88003a50 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.249+0000 7f2aa5200640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a88003a50 0x7f2a88003e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2a94001720 tx=0x7f2a88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fc970e14507a35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.249+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94001370 con 0x7f2a88003a50 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.249+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a940052d0 con 0x7f2a88003a50 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.249+0000 7f2a9f400640 1 -- 172.21.15.6:0/2958242325 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94003830 con 0x7f2a88003a50 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa7c00640 1 -- 172.21.15.6:0/2958242325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a88003a50 msgr2=0x7f2a88003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/2958242325 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2a88003a50 0x7f2a88003e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2a94001720 tx=0x7f2a88005830 comp rx=0 tx=0).stop 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa5c00640 1 -- 172.21.15.6:0/2958242325 reap_dead start 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa7c00640 1 -- 172.21.15.6:0/2958242325 shutdown_connections 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa7c00640 1 -- 172.21.15.6:0/2958242325 >> 172.21.15.6:0/2958242325 conn(0x7f2aa006afb0 msgr2=0x7f2aa006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa7c00640 1 -- 172.21.15.6:0/2958242325 shutdown_connections 2024-03-06T07:29:13.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.250+0000 7f2aa7c00640 1 -- 172.21.15.6:0/2958242325 wait complete. 2024-03-06T07:29:13.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.252+0000 7f2aa7c00640 1 Processor -- start 2024-03-06T07:29:13.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.252+0000 7f2aa7c00640 1 -- start start 2024-03-06T07:29:13.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa7c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2a88003a50 0x7f2aa00cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa7c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 0x7f2aa00b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa7c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00b14a0 0x7f2aa014f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa7c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa015c600 con 0x7f2aa00af960 2024-03-06T07:29:13.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa7c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00b36f0 con 0x7f2aa00b14a0 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa7c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa015c780 con 0x7f2a88003a50 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 0x7f2aa00b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 0x7f2aa00b0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51476/0 (socket says 172.21.15.6:51476) 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 -- 172.21.15.6:0/1671879094 learned_addr learned my addr 172.21.15.6:0/1671879094 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa5200640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2a88003a50 0x7f2aa00cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa5c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00b14a0 0x7f2aa014f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 -- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2a88003a50 msgr2=0x7f2aa00cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2a88003a50 0x7f2aa00cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 -- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00b14a0 msgr2=0x7f2aa014f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00b14a0 0x7f2aa014f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2a9fe00640 1 -- 172.21.15.6:0/1671879094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa006cb10 con 0x7f2aa00af960 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa5200640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2a88003a50 0x7f2aa00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:13.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.253+0000 7f2aa5c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00b14a0 0x7f2aa014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.254+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 0x7f2aa00b0f60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2a90011600 tx=0x7f2a90011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.254+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000f920 con 0x7f2aa00af960 2024-03-06T07:29:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.254+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9000fac0 con 0x7f2aa00af960 2024-03-06T07:29:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.254+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9000fe70 con 0x7f2aa00af960 2024-03-06T07:29:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.254+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa006cdf0 con 0x7f2aa00af960 2024-03-06T07:29:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.254+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa006d690 con 0x7f2aa00af960 2024-03-06T07:29:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.256+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa0069510 con 0x7f2aa00af960 2024-03-06T07:29:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.256+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a90015460 con 0x7f2aa00af960 2024-03-06T07:29:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.257+0000 7f2a9cc00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a8406b4d0 0x7f2a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.257+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a9008ab60 con 0x7f2aa00af960 2024-03-06T07:29:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.258+0000 7f2aa5200640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a8406b4d0 0x7f2a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.259+0000 7f2aa5200640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a8406b4d0 0x7f2a8406d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2a94033000 tx=0x7f2a94009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.262+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a90019030 con 0x7f2aa00af960 2024-03-06T07:29:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.487+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2aa00ce580 con 0x7f2aa00af960 2024-03-06T07:29:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.489+0000 7f2a9cc00640 1 -- 172.21.15.6:0/1671879094 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16597 (secure 0 0 0) 0x7f2a900592f0 con 0x7f2aa00af960 2024-03-06T07:29:13.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:13.491 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":35,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:28:41.851077+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:38.380358+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:39.297336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.491+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a8406b4d0 msgr2=0x7f2a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a8406b4d0 0x7f2a8406d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2a94033000 tx=0x7f2a94009030 comp rx=0 tx=0).stop 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 msgr2=0x7f2aa00b0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 0x7f2aa00b0f60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2a90011600 tx=0x7f2a90011ac0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 shutdown_connections 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a8406b4d0 0x7f2a8406d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00b14a0 0x7f2aa014f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00af960 0x7f2aa00b0f60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 --2- 172.21.15.6:0/1671879094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2a88003a50 0x7f2aa00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 >> 172.21.15.6:0/1671879094 conn(0x7f2aa006afb0 msgr2=0x7f2aa00ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 shutdown_connections 2024-03-06T07:29:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.492+0000 7f2aa7c00640 1 -- 172.21.15.6:0/1671879094 wait complete. 2024-03-06T07:29:13.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:13.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.720+0000 7fcb9a200640 1 Processor -- start 2024-03-06T07:29:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.721+0000 7fcb9a200640 1 -- start start 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.722+0000 7fcb9a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb9415f310 0x7fcb940cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.722+0000 7fcb9a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940cfdd0 0x7fcb940d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.722+0000 7fcb9a200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb940d02d0 con 0x7fcb940d2170 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.722+0000 7fcb9a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb940d2600 con 0x7fcb9415f310 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.722+0000 7fcb9a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940d05b0 con 0x7fcb940cfdd0 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.722+0000 7fcb93400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb9415f310 0x7fcb940cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb93400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb9415f310 0x7fcb940cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51490/0 (socket says 172.21.15.6:51490) 2024-03-06T07:29:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb93400640 1 -- 172.21.15.6:0/4003043073 learned_addr learned my addr 172.21.15.6:0/4003043073 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb8aa00640 1 --2- 172.21.15.6:0/4003043073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940cfdd0 0x7fcb940d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb93400640 1 -- 172.21.15.6:0/4003043073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940cfdd0 msgr2=0x7fcb940d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb93400640 1 --2- 172.21.15.6:0/4003043073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940cfdd0 0x7fcb940d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb93400640 1 -- 172.21.15.6:0/4003043073 >> v1:172.21.15.6:6789/0 conn(0x7fcb940d2170 legacy=0x7fcb940ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.723+0000 7fcb93400640 1 -- 172.21.15.6:0/4003043073 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb940d0890 con 0x7fcb9415f310 2024-03-06T07:29:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.725+0000 7fcb93400640 1 --2- 172.21.15.6:0/4003043073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9415f310 0x7fcb940cf890 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009e50 tx=0x7fcb8402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b271ae90c1336d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.725+0000 7fcb92a00640 1 -- 172.21.15.6:0/4003043073 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8402cb90 con 0x7fcb9415f310 2024-03-06T07:29:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.725+0000 7fcb92a00640 1 -- 172.21.15.6:0/4003043073 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8402cd30 con 0x7fcb9415f310 2024-03-06T07:29:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.725+0000 7fcb92a00640 1 -- 172.21.15.6:0/4003043073 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84034610 con 0x7fcb9415f310 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.725+0000 7fcb9a200640 1 -- 172.21.15.6:0/4003043073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9415f310 msgr2=0x7fcb940cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.725+0000 7fcb9a200640 1 --2- 172.21.15.6:0/4003043073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9415f310 0x7fcb940cf890 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009e50 tx=0x7fcb8402bff0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.726+0000 7fcb9a200640 1 -- 172.21.15.6:0/4003043073 shutdown_connections 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.726+0000 7fcb9a200640 1 --2- 172.21.15.6:0/4003043073 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940cfdd0 0x7fcb940d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.726+0000 7fcb9a200640 1 --2- 172.21.15.6:0/4003043073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9415f310 0x7fcb940cf890 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.726+0000 7fcb9a200640 1 -- 172.21.15.6:0/4003043073 >> 172.21.15.6:0/4003043073 conn(0x7fcb9409aa10 msgr2=0x7fcb9409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.726+0000 7fcb9a200640 1 -- 172.21.15.6:0/4003043073 shutdown_connections 2024-03-06T07:29:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.726+0000 7fcb9a200640 1 -- 172.21.15.6:0/4003043073 wait complete. 2024-03-06T07:29:13.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.727+0000 7fcb9a200640 1 Processor -- start 2024-03-06T07:29:13.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.728+0000 7fcb9a200640 1 -- start start 2024-03-06T07:29:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb9a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 0x7fcb9406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb9a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940d2170 0x7fcb9414f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb9a200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9415f310 0x7fcb940c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb9a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9415f6f0 con 0x7fcb940cfdd0 2024-03-06T07:29:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb9a200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb940cc920 con 0x7fcb9415f310 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb9a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9415f870 con 0x7fcb940d2170 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 0x7fcb9406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 0x7fcb9406dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51492/0 (socket says 172.21.15.6:51492) 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 -- 172.21.15.6:0/2960752228 learned_addr learned my addr 172.21.15.6:0/2960752228 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb8aa00640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940d2170 0x7fcb9414f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93e00640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9415f310 0x7fcb940c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 -- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940d2170 msgr2=0x7fcb9414f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940d2170 0x7fcb9414f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 -- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9415f310 msgr2=0x7fcb940c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9415f310 0x7fcb940c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb93400640 1 -- 172.21.15.6:0/2960752228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9406c160 con 0x7fcb940cfdd0 2024-03-06T07:29:13.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.729+0000 7fcb8aa00640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940d2170 0x7fcb9414f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.730+0000 7fcb93400640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 0x7fcb9406dd20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009f70 tx=0x7fcb84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.730+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84043020 con 0x7fcb940cfdd0 2024-03-06T07:29:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.730+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8403e450 con 0x7fcb940cfdd0 2024-03-06T07:29:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.730+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84009510 con 0x7fcb940cfdd0 2024-03-06T07:29:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.730+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9406c3d0 con 0x7fcb940cfdd0 2024-03-06T07:29:13.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.730+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9406c990 con 0x7fcb940cfdd0 2024-03-06T07:29:13.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.731+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9404f3a0 con 0x7fcb940cfdd0 2024-03-06T07:29:13.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.733+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb84046020 con 0x7fcb940cfdd0 2024-03-06T07:29:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.734+0000 7fcb8be00640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb7006b500 0x7fcb7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:13.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.734+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb840ac960 con 0x7fcb940cfdd0 2024-03-06T07:29:13.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.735+0000 7fcb8aa00640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb7006b500 0x7fcb7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:13.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.736+0000 7fcb8aa00640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb7006b500 0x7fcb7006d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcb9409b0c0 tx=0x7fcb7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:13.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.739+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb8404d030 con 0x7fcb940cfdd0 2024-03-06T07:29:13.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.977+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcb940cdc80 con 0x7fcb940cfdd0 2024-03-06T07:29:13.980 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:29:13.978+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:13.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.982+0000 7fcb8be00640 1 -- 172.21.15.6:0/2960752228 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcb8407b740 con 0x7fcb940cfdd0 2024-03-06T07:29:13.984 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:13.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.987+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb7006b500 msgr2=0x7fcb7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.987+0000 7fcb9a200640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb7006b500 0x7fcb7006d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcb9409b0c0 tx=0x7fcb7c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:13.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.987+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 msgr2=0x7fcb9406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.987+0000 7fcb9a200640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 0x7fcb9406dd20 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fcb84009f70 tx=0x7fcb84004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 shutdown_connections 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb7006b500 0x7fcb7006d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9415f310 0x7fcb940c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb940d2170 0x7fcb9414f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 --2- 172.21.15.6:0/2960752228 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb940cfdd0 0x7fcb9406dd20 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 >> 172.21.15.6:0/2960752228 conn(0x7fcb9409aa10 msgr2=0x7fcb940b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 shutdown_connections 2024-03-06T07:29:13.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:13.990+0000 7fcb9a200640 1 -- 172.21.15.6:0/2960752228 wait complete. 2024-03-06T07:29:14.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:14.062 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:29:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.201+0000 7f877a200640 1 Processor -- start 2024-03-06T07:29:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.201+0000 7f877a200640 1 -- start start 2024-03-06T07:29:14.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.203+0000 7f877a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8774147590 0x7f8774071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.203+0000 7f877a200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87740719c0 con 0x7f877406d8d0 2024-03-06T07:29:14.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.203+0000 7f877a200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f877406de40 con 0x7f87740b2250 2024-03-06T07:29:14.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.203+0000 7f877a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8774147970 con 0x7f8774147590 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.203+0000 7f8778e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8774147590 0x7f8774071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.204+0000 7f8778e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8774147590 0x7f8774071480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51518/0 (socket says 172.21.15.6:51518) 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.204+0000 7f8778e00640 1 -- 172.21.15.6:0/2664636474 learned_addr learned my addr 172.21.15.6:0/2664636474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.204+0000 7f8773e00640 1 --1- 172.21.15.6:0/2664636474 >> v1:172.21.15.6:6789/0 conn(0x7f877406d8d0 0x7f87740b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33080/0 (socket says 172.21.15.6:33080) 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.204+0000 7f8778e00640 1 -- 172.21.15.6:0/2664636474 >> v1:172.21.15.178:6789/0 conn(0x7f87740b2250 legacy=0x7f87740b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.204+0000 7f8778e00640 1 -- 172.21.15.6:0/2664636474 >> v1:172.21.15.6:6789/0 conn(0x7f877406d8d0 legacy=0x7f87740b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.204+0000 7f8778e00640 1 -- 172.21.15.6:0/2664636474 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877406e120 con 0x7f8774147590 2024-03-06T07:29:14.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.205+0000 7f8778e00640 1 --2- 172.21.15.6:0/2664636474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8774147590 0x7f8774071480 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8768009870 tx=0x7f876802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d9d3a65447d53e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f8773400640 1 -- 172.21.15.6:0/2664636474 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768034070 con 0x7f8774147590 2024-03-06T07:29:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f8773400640 1 -- 172.21.15.6:0/2664636474 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8768030860 con 0x7f8774147590 2024-03-06T07:29:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f8773400640 1 -- 172.21.15.6:0/2664636474 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768030c30 con 0x7f8774147590 2024-03-06T07:29:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 -- 172.21.15.6:0/2664636474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8774147590 msgr2=0x7f8774071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 --2- 172.21.15.6:0/2664636474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8774147590 0x7f8774071480 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8768009870 tx=0x7f876802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 -- 172.21.15.6:0/2664636474 shutdown_connections 2024-03-06T07:29:14.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 --2- 172.21.15.6:0/2664636474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8774147590 0x7f8774071480 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 -- 172.21.15.6:0/2664636474 >> 172.21.15.6:0/2664636474 conn(0x7f877406b3a0 msgr2=0x7f877406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 -- 172.21.15.6:0/2664636474 shutdown_connections 2024-03-06T07:29:14.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.206+0000 7f877a200640 1 -- 172.21.15.6:0/2664636474 wait complete. 2024-03-06T07:29:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.208+0000 7f877a200640 1 Processor -- start 2024-03-06T07:29:14.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.208+0000 7f877a200640 1 -- start start 2024-03-06T07:29:14.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f877a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f877406d8d0 0x7f87741f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f877a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 0x7f877415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f877a200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8774147590 0x7f87740cf390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f877a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b2b50 con 0x7f87740b2250 2024-03-06T07:29:14.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f877a200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b31b0 con 0x7f8774147590 2024-03-06T07:29:14.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f877a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b3d50 con 0x7f877406d8d0 2024-03-06T07:29:14.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f8773e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 0x7f877415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.209+0000 7f8773e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 0x7f877415fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51524/0 (socket says 172.21.15.6:51524) 2024-03-06T07:29:14.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 -- 172.21.15.6:0/338762838 learned_addr learned my addr 172.21.15.6:0/338762838 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:14.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 -- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f877406d8d0 msgr2=0x7f87741f3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8778e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f877406d8d0 0x7f87741f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8779800640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8774147590 0x7f87740cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f877406d8d0 0x7f87741f3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 -- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8774147590 msgr2=0x7f87740cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8774147590 0x7f87740cf390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 -- 172.21.15.6:0/338762838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87741f40a0 con 0x7f87740b2250 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8778e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f877406d8d0 0x7f87741f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8779800640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8774147590 0x7f87740cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.210+0000 7f8773e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 0x7f877415fd80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f876400c400 tx=0x7f876400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.211+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87640074e0 con 0x7f87740b2250 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.211+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8764007680 con 0x7f87740b2250 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.211+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8764004a00 con 0x7f87740b2250 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.211+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8774141950 con 0x7f87740b2250 2024-03-06T07:29:14.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.211+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87741e6d60 con 0x7f87740b2250 2024-03-06T07:29:14.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.212+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877404f330 con 0x7f87740b2250 2024-03-06T07:29:14.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.213+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f876401a020 con 0x7f87740b2250 2024-03-06T07:29:14.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.214+0000 7f8770c00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f875806b4d0 0x7f875806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.214+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8764089a80 con 0x7f87740b2250 2024-03-06T07:29:14.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.215+0000 7f8778e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f875806b4d0 0x7f875806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.216+0000 7f8778e00640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f875806b4d0 0x7f875806d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8768036000 tx=0x7f876803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.221+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87741f40a0 con 0x7f87740b2250 2024-03-06T07:29:14.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:14.465+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f87740b1c90 con 0x7f87740b2250 2024-03-06T07:29:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.037+0000 7f8770c00640 1 -- 172.21.15.6:0/338762838 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87640588a0 con 0x7f87740b2250 2024-03-06T07:29:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f875806b4d0 msgr2=0x7f875806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f875806b4d0 0x7f875806d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8768036000 tx=0x7f876803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 msgr2=0x7f877415fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 0x7f877415fd80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f876400c400 tx=0x7f876400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 shutdown_connections 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f875806b4d0 0x7f875806d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8774147590 0x7f87740cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87740b2250 0x7f877415fd80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 --2- 172.21.15.6:0/338762838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f877406d8d0 0x7f87741f3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 >> 172.21.15.6:0/338762838 conn(0x7f877406b3a0 msgr2=0x7f87740b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 shutdown_connections 2024-03-06T07:29:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.041+0000 7f877a200640 1 -- 172.21.15.6:0/338762838 wait complete. 2024-03-06T07:29:15.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:29:15.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.284+0000 7fa656000640 1 Processor -- start 2024-03-06T07:29:15.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.284+0000 7fa656000640 1 -- start start 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa656000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 0x7fa6500d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa656000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa65015e2f0 0x7fa6500d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa656000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6500d2a60 con 0x7fa6500d45b0 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa656000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6500b6b80 con 0x7fa65015e2f0 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa656000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500b6e20 con 0x7fa65015df10 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa655600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa6500d45b0 0x7fa6500ce950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33096/0 (socket says 172.21.15.6:33096) 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.286+0000 7fa655600640 1 -- 172.21.15.6:0/812577336 learned_addr learned my addr 172.21.15.6:0/812577336 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.287+0000 7fa64fe00640 1 --2- 172.21.15.6:0/812577336 >> v2:172.21.15.178:3300/0 conn(0x7fa65015e2f0 0x7fa6500d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.287+0000 7fa654c00640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 0x7fa6500d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.287+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813041283 0 0) 0x7fa6500d2a60 con 0x7fa6500d45b0 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.287+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638003520 con 0x7fa6500d45b0 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1029560902 0 0) 0x7fa638003520 con 0x7fa6500d45b0 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 msgr2=0x7fa6500d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 0x7fa6500d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 >> v2:172.21.15.178:3300/0 conn(0x7fa65015e2f0 msgr2=0x7fa6500d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 --2- 172.21.15.6:0/812577336 >> v2:172.21.15.178:3300/0 conn(0x7fa65015e2f0 0x7fa6500d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6500d2780 con 0x7fa6500d45b0 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fa648001a10 con 0x7fa6500d45b0 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 >> v1:172.21.15.6:6789/0 conn(0x7fa6500d45b0 legacy=0x7fa6500ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 0x7fa6380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa638003f80 0x7fa638004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500b6b80 con 0x7fa638004d40 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500b6e20 con 0x7fa638003f80 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63800baf0 con 0x7fa65015df10 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa655600640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa654c00640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 0x7fa6380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.288+0000 7fa64fe00640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa638003f80 0x7fa638004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.289+0000 7fa655600640 1 -- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 msgr2=0x7fa6380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.289+0000 7fa655600640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa65015df10 0x7fa6380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.289+0000 7fa655600640 1 -- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa638003f80 msgr2=0x7fa638004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.289+0000 7fa655600640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa638003f80 0x7fa638004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.289+0000 7fa655600640 1 -- 172.21.15.6:0/812577336 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa638003520 con 0x7fa638004d40 2024-03-06T07:29:15.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.290+0000 7fa655600640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6380051c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa6500d2c30 tx=0x7fa638005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55e610bb8f4ff4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.290+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64800a410 con 0x7fa638004d40 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.290+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa648034630 con 0x7fa638004d40 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.290+0000 7fa64f400640 1 -- 172.21.15.6:0/812577336 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648034a20 con 0x7fa638004d40 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa656000640 1 -- 172.21.15.6:0/812577336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 msgr2=0x7fa6380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa656000640 1 --2- 172.21.15.6:0/812577336 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6380051c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa6500d2c30 tx=0x7fa638005830 comp rx=0 tx=0).stop 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa655600640 1 -- 172.21.15.6:0/812577336 reap_dead start 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa656000640 1 -- 172.21.15.6:0/812577336 shutdown_connections 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa656000640 1 -- 172.21.15.6:0/812577336 >> 172.21.15.6:0/812577336 conn(0x7fa65009cfe0 msgr2=0x7fa6500d3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa656000640 1 -- 172.21.15.6:0/812577336 shutdown_connections 2024-03-06T07:29:15.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.291+0000 7fa656000640 1 -- 172.21.15.6:0/812577336 wait complete. 2024-03-06T07:29:15.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.293+0000 7fa656000640 1 Processor -- start 2024-03-06T07:29:15.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.293+0000 7fa656000640 1 -- start start 2024-03-06T07:29:15.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa656000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6500cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa656000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa65014ddc0 0x7fa65006bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa656000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa6500a17c0 0x7fa65006c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa656000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500b87b0 con 0x7fa638004d40 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa656000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500b8630 con 0x7fa65014ddc0 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa656000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500b8930 con 0x7fa6500a17c0 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa654c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6500cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa654c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6500cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51530/0 (socket says 172.21.15.6:51530) 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa654c00640 1 -- 172.21.15.6:0/570051970 learned_addr learned my addr 172.21.15.6:0/570051970 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.294+0000 7fa64fe00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa65014ddc0 0x7fa65006bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa655600640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa6500a17c0 0x7fa65006c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa654c00640 1 -- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa6500a17c0 msgr2=0x7fa65006c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa654c00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa6500a17c0 0x7fa65006c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa654c00640 1 -- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa65014ddc0 msgr2=0x7fa65006bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa654c00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa65014ddc0 0x7fa65006bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa654c00640 1 -- 172.21.15.6:0/570051970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65014e2f0 con 0x7fa638004d40 2024-03-06T07:29:15.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa654c00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6500cad60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa644011660 tx=0x7fa644011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400f960 con 0x7fa638004d40 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64400fb00 con 0x7fa638004d40 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65014e5d0 con 0x7fa638004d40 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644015460 con 0x7fa638004d40 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.295+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65014eb90 con 0x7fa638004d40 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.297+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6500cb2a0 con 0x7fa638004d40 2024-03-06T07:29:15.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.297+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa644015600 con 0x7fa638004d40 2024-03-06T07:29:15.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.298+0000 7fa64cc00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa63006b500 0x7fa63006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:15.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.298+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa64408abf0 con 0x7fa638004d40 2024-03-06T07:29:15.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.299+0000 7fa64fe00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa63006b500 0x7fa63006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:15.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.300+0000 7fa64fe00640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa63006b500 0x7fa63006d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa65006cd10 tx=0x7fa63c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:15.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.303+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa644019030 con 0x7fa638004d40 2024-03-06T07:29:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:15.502+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa65004f330 con 0x7fa638004d40 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.047+0000 7fa64cc00640 1 -- 172.21.15.6:0/570051970 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v37) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa644059970 con 0x7fa638004d40 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa63006b500 msgr2=0x7fa63006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa63006b500 0x7fa63006d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa65006cd10 tx=0x7fa63c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 msgr2=0x7fa6500cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6500cad60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa644011660 tx=0x7fa644011b20 comp rx=0 tx=0).stop 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 shutdown_connections 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa63006b500 0x7fa63006d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa65014ddc0 0x7fa65006bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa6500a17c0 0x7fa65006c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 --2- 172.21.15.6:0/570051970 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa638004d40 0x7fa6500cad60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.051+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 >> 172.21.15.6:0/570051970 conn(0x7fa65009cfe0 msgr2=0x7fa6501610c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.052+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 shutdown_connections 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.052+0000 7fa656000640 1 -- 172.21.15.6:0/570051970 wait complete. 2024-03-06T07:29:16.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:29:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.305+0000 7f190ca00640 1 Processor -- start 2024-03-06T07:29:16.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.305+0000 7f190ca00640 1 -- start start 2024-03-06T07:29:16.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.306+0000 7f190ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 0x7f19080aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.306+0000 7f190ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 0x7f190806cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.306+0000 7f190ca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19081531f0 con 0x7f19080b3640 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.306+0000 7f190ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19080d04d0 con 0x7f19080ae7b0 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.306+0000 7f190ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19080b2b80 con 0x7f190815cdc0 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.307+0000 7f1906800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 0x7f190806cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.307+0000 7f1905e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 0x7f19080aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.307+0000 7f1906800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 0x7f190806cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:57776/0 (socket says 172.21.15.6:57776) 2024-03-06T07:29:16.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.307+0000 7f1905e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 0x7f19080aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:34588/0 (socket says 172.21.15.6:34588) 2024-03-06T07:29:16.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.307+0000 7f1906800640 1 -- 172.21.15.6:0/2270242276 learned_addr learned my addr 172.21.15.6:0/2270242276 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:16.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.308+0000 7f1906800640 1 -- 172.21.15.6:0/2270242276 >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 msgr2=0x7f19080aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.308+0000 7f1906800640 1 --2- 172.21.15.6:0/2270242276 >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 0x7f19080aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.308+0000 7f1906800640 1 -- 172.21.15.6:0/2270242276 >> v1:172.21.15.178:6789/0 conn(0x7f19080b3640 legacy=0x7f19080aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.308+0000 7f1906800640 1 -- 172.21.15.6:0/2270242276 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190806ebd0 con 0x7f190815cdc0 2024-03-06T07:29:16.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.308+0000 7f1905e00640 1 --2- 172.21.15.6:0/2270242276 >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 0x7f19080aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:16.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.310+0000 7f1906800640 1 --2- 172.21.15.6:0/2270242276 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 0x7f190806cc70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1900009e50 tx=0x7f190002bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eb862043b760a8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.310+0000 7f1905400640 1 -- 172.21.15.6:0/2270242276 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190002c8c0 con 0x7f190815cdc0 2024-03-06T07:29:16.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f1905400640 1 -- 172.21.15.6:0/2270242276 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190002ca60 con 0x7f190815cdc0 2024-03-06T07:29:16.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f1905400640 1 -- 172.21.15.6:0/2270242276 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190002ce30 con 0x7f190815cdc0 2024-03-06T07:29:16.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 -- 172.21.15.6:0/2270242276 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 msgr2=0x7f190806cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 --2- 172.21.15.6:0/2270242276 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 0x7f190806cc70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1900009e50 tx=0x7f190002bda0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 -- 172.21.15.6:0/2270242276 shutdown_connections 2024-03-06T07:29:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 --2- 172.21.15.6:0/2270242276 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f190815cdc0 0x7f190806cc70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 --2- 172.21.15.6:0/2270242276 >> v2:172.21.15.178:3300/0 conn(0x7f19080ae7b0 0x7f19080aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 -- 172.21.15.6:0/2270242276 >> 172.21.15.6:0/2270242276 conn(0x7f190806afb0 msgr2=0x7f190806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 -- 172.21.15.6:0/2270242276 shutdown_connections 2024-03-06T07:29:16.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.311+0000 7f190ca00640 1 -- 172.21.15.6:0/2270242276 wait complete. 2024-03-06T07:29:16.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.313+0000 7f190ca00640 1 Processor -- start 2024-03-06T07:29:16.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.313+0000 7f190ca00640 1 -- start start 2024-03-06T07:29:16.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.314+0000 7f190ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 0x7f1908151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.314+0000 7f190ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19080b3640 0x7f19081569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.314+0000 7f190ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f190815cdc0 0x7f190814e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.314+0000 7f190ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19080b3120 con 0x7f19080ae7b0 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.314+0000 7f190ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19080b2e40 con 0x7f190815cdc0 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.314+0000 7f190ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190815d250 con 0x7f19080b3640 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 0x7f1908151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 0x7f1908151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51542/0 (socket says 172.21.15.6:51542) 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 -- 172.21.15.6:0/4046146134 learned_addr learned my addr 172.21.15.6:0/4046146134 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1906800640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f190815cdc0 0x7f190814e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f18fd400640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19080b3640 0x7f19081569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 -- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19080b3640 msgr2=0x7f19081569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19080b3640 0x7f19081569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 -- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f190815cdc0 msgr2=0x7f190814e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f190815cdc0 0x7f190814e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.315+0000 7f1905e00640 1 -- 172.21.15.6:0/4046146134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1908157b50 con 0x7f19080ae7b0 2024-03-06T07:29:16.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.316+0000 7f1905e00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 0x7f1908151cc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f18f800ebe0 tx=0x7f18f800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.316+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8017070 con 0x7f19080ae7b0 2024-03-06T07:29:16.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.316+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f800cc30 con 0x7f19080ae7b0 2024-03-06T07:29:16.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.316+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8013470 con 0x7f19080ae7b0 2024-03-06T07:29:16.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.316+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1908157590 con 0x7f19080ae7b0 2024-03-06T07:29:16.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.316+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190809c110 con 0x7f19080ae7b0 2024-03-06T07:29:16.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.317+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1908067950 con 0x7f19080ae7b0 2024-03-06T07:29:16.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.318+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18f801a070 con 0x7f19080ae7b0 2024-03-06T07:29:16.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.319+0000 7f18fea00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18e406b4d0 0x7f18e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.319+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f18f8089fe0 con 0x7f19080ae7b0 2024-03-06T07:29:16.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.320+0000 7f18fd400640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18e406b4d0 0x7f18e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.321+0000 7f18fd400640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18e406b4d0 0x7f18e406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1908159e60 tx=0x7f18f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.324+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1908067950 con 0x7f19080ae7b0 2024-03-06T07:29:16.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:16.524+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f190809bca0 con 0x7f19080ae7b0 2024-03-06T07:29:17.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.064+0000 7f18fea00640 1 -- 172.21.15.6:0/4046146134 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v38) v1 ==== 169+0+0 (secure 0 0 0) 0x7f18f8058fc0 con 0x7f19080ae7b0 2024-03-06T07:29:17.067 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.069+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18e406b4d0 msgr2=0x7f18e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18e406b4d0 0x7f18e406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1908159e60 tx=0x7f18f0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 msgr2=0x7f1908151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 0x7f1908151cc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f18f800ebe0 tx=0x7f18f800c520 comp rx=0 tx=0).stop 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 shutdown_connections 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18e406b4d0 0x7f18e406d990 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1908159e60 tx=0x7f18f0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f190815cdc0 0x7f190814e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19080b3640 0x7f19081569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 --2- 172.21.15.6:0/4046146134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19080ae7b0 0x7f1908151cc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.070+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 >> 172.21.15.6:0/4046146134 conn(0x7f190806afb0 msgr2=0x7f190815e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.071+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 shutdown_connections 2024-03-06T07:29:17.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.072+0000 7f190ca00640 1 -- 172.21.15.6:0/4046146134 wait complete. 2024-03-06T07:29:17.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.290+0000 7fa75c600640 1 Processor -- start 2024-03-06T07:29:17.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.290+0000 7fa75c600640 1 -- start start 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.292+0000 7fa75c600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa75415bf20 0x7fa7540acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.292+0000 7fa75c600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa754071e20 0x7fa7540ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.292+0000 7fa75c600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa754072df0 con 0x7fa754072520 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.292+0000 7fa75c600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7540cdab0 con 0x7fa754071e20 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.292+0000 7fa75c600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7540b4d70 con 0x7fa75415bf20 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.292+0000 7fa75a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa754071e20 0x7fa7540ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa75a600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa754071e20 0x7fa7540ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51558/0 (socket says 172.21.15.6:51558) 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa75a600640 1 -- 172.21.15.6:0/3564490957 learned_addr learned my addr 172.21.15.6:0/3564490957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:17.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa759200640 1 --1- 172.21.15.6:0/3564490957 >> v1:172.21.15.6:6789/0 conn(0x7fa754072520 0x7fa7540cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33098/0 (socket says 172.21.15.6:33098) 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa759c00640 1 --2- 172.21.15.6:0/3564490957 >> v2:172.21.15.178:3300/0 conn(0x7fa75415bf20 0x7fa7540acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa75a600640 1 -- 172.21.15.6:0/3564490957 >> v2:172.21.15.178:3300/0 conn(0x7fa75415bf20 msgr2=0x7fa7540acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa75a600640 1 --2- 172.21.15.6:0/3564490957 >> v2:172.21.15.178:3300/0 conn(0x7fa75415bf20 0x7fa7540acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa75a600640 1 -- 172.21.15.6:0/3564490957 >> v1:172.21.15.6:6789/0 conn(0x7fa754072520 legacy=0x7fa7540cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa75a600640 1 -- 172.21.15.6:0/3564490957 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7540b5050 con 0x7fa754071e20 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.293+0000 7fa753e00640 1 -- 172.21.15.6:0/3564490957 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219083659 0 0) 0x7fa754072df0 con 0x7fa754072520 2024-03-06T07:29:17.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.294+0000 7fa75a600640 1 --2- 172.21.15.6:0/3564490957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa754071e20 0x7fa7540ccad0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa75406ec40 tx=0x7fa74c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a1f48e42f9053d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa753e00640 1 -- 172.21.15.6:0/3564490957 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c02cb20 con 0x7fa754071e20 2024-03-06T07:29:17.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa753e00640 1 -- 172.21.15.6:0/3564490957 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c02ccc0 con 0x7fa754071e20 2024-03-06T07:29:17.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa753e00640 1 -- 172.21.15.6:0/3564490957 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c034610 con 0x7fa754071e20 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 -- 172.21.15.6:0/3564490957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa754071e20 msgr2=0x7fa7540ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 --2- 172.21.15.6:0/3564490957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa754071e20 0x7fa7540ccad0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa75406ec40 tx=0x7fa74c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 -- 172.21.15.6:0/3564490957 shutdown_connections 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 --2- 172.21.15.6:0/3564490957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa754071e20 0x7fa7540ccad0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 --2- 172.21.15.6:0/3564490957 >> v2:172.21.15.178:3300/0 conn(0x7fa75415bf20 0x7fa7540acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 -- 172.21.15.6:0/3564490957 >> 172.21.15.6:0/3564490957 conn(0x7fa75406b3d0 msgr2=0x7fa75406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 -- 172.21.15.6:0/3564490957 shutdown_connections 2024-03-06T07:29:17.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.295+0000 7fa75c600640 1 -- 172.21.15.6:0/3564490957 wait complete. 2024-03-06T07:29:17.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.297+0000 7fa75c600640 1 Processor -- start 2024-03-06T07:29:17.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.297+0000 7fa75c600640 1 -- start start 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75c600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa754071e20 0x7fa754154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75c600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa754072520 0x7fa75409ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75c600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 0x7fa7541495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75c600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540b4b10 con 0x7fa75415bf20 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75c600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540b4990 con 0x7fa754071e20 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75c600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7540ce6c0 con 0x7fa754072520 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.298+0000 7fa75a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 0x7fa7541495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 0x7fa7541495b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51564/0 (socket says 172.21.15.6:51564) 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 -- 172.21.15.6:0/6111829 learned_addr learned my addr 172.21.15.6:0/6111829 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 -- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa754072520 msgr2=0x7fa75409ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa754072520 0x7fa75409ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 -- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa754071e20 msgr2=0x7fa754154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa754071e20 0x7fa754154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 -- 172.21.15.6:0/6111829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754149af0 con 0x7fa75415bf20 2024-03-06T07:29:17.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa75a600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 0x7fa7541495b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa74c036040 tx=0x7fa74c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.299+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c043020 con 0x7fa75415bf20 2024-03-06T07:29:17.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.300+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c03e450 con 0x7fa75415bf20 2024-03-06T07:29:17.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.300+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c009510 con 0x7fa75415bf20 2024-03-06T07:29:17.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.300+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa754154ab0 con 0x7fa75415bf20 2024-03-06T07:29:17.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.300+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa754155070 con 0x7fa75415bf20 2024-03-06T07:29:17.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.301+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7540b3140 con 0x7fa75415bf20 2024-03-06T07:29:17.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.302+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa74c046020 con 0x7fa75415bf20 2024-03-06T07:29:17.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.303+0000 7fa751600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa73406b4d0 0x7fa73406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.303+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa74c0ac7f0 con 0x7fa75415bf20 2024-03-06T07:29:17.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.303+0000 7fa759c00640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa73406b4d0 0x7fa73406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.304+0000 7fa759c00640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa73406b4d0 0x7fa73406d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa7480098f0 tx=0x7fa748008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.307+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7540b3140 con 0x7fa75415bf20 2024-03-06T07:29:17.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.506+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa75404f330 con 0x7fa75415bf20 2024-03-06T07:29:17.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.512+0000 7fa751600640 1 -- 172.21.15.6:0/6111829 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa74c04d030 con 0x7fa75415bf20 2024-03-06T07:29:17.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa73406b4d0 msgr2=0x7fa73406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa73406b4d0 0x7fa73406d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa7480098f0 tx=0x7fa748008040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 msgr2=0x7fa7541495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 0x7fa7541495b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa74c036040 tx=0x7fa74c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 shutdown_connections 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa73406b4d0 0x7fa73406d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa75415bf20 0x7fa7541495b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa754072520 0x7fa75409ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 --2- 172.21.15.6:0/6111829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa754071e20 0x7fa754154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 >> 172.21.15.6:0/6111829 conn(0x7fa75406b3d0 msgr2=0x7fa7540b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 shutdown_connections 2024-03-06T07:29:17.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.515+0000 7fa75c600640 1 -- 172.21.15.6:0/6111829 wait complete. 2024-03-06T07:29:17.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:29:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.757+0000 7fc5b0800640 1 Processor -- start 2024-03-06T07:29:17.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.757+0000 7fc5b0800640 1 -- start start 2024-03-06T07:29:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5b0800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5b0800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5a80ae0a0 0x7fc5a80711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5b0800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a80ce9a0 con 0x7fc5a806e240 2024-03-06T07:29:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5b0800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5a80730d0 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5b0800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80cec80 con 0x7fc5a8161670 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5ad400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc5a806e240 0x7fc5a806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33100/0 (socket says 172.21.15.6:33100) 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5ad400640 1 -- 172.21.15.6:0/1259347412 learned_addr learned my addr 172.21.15.6:0/1259347412 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5ae800640 1 --2- 172.21.15.6:0/1259347412 >> v2:172.21.15.6:3300/0 conn(0x7fc5a80ae0a0 0x7fc5a80711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.759+0000 7fc5ade00640 1 --2- 172.21.15.6:0/1259347412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.760+0000 7fc5aca00640 1 -- 172.21.15.6:0/1259347412 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47115534 0 0) 0x7fc5a80ce9a0 con 0x7fc5a806e240 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.760+0000 7fc5ae800640 1 -- 172.21.15.6:0/1259347412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 msgr2=0x7fc5a8070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.760+0000 7fc5ae800640 1 --2- 172.21.15.6:0/1259347412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.760+0000 7fc5ae800640 1 -- 172.21.15.6:0/1259347412 >> v1:172.21.15.6:6789/0 conn(0x7fc5a806e240 legacy=0x7fc5a806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.760+0000 7fc5ae800640 1 -- 172.21.15.6:0/1259347412 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80cef60 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.761+0000 7fc5ae800640 1 --2- 172.21.15.6:0/1259347412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80711b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009d90 tx=0x7fc5a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e512d041580532e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.761+0000 7fc5aca00640 1 -- 172.21.15.6:0/1259347412 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4039070 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.761+0000 7fc5aca00640 1 -- 172.21.15.6:0/1259347412 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a402cc50 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.761+0000 7fc5aca00640 1 -- 172.21.15.6:0/1259347412 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4034440 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 -- 172.21.15.6:0/1259347412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 msgr2=0x7fc5a80711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 --2- 172.21.15.6:0/1259347412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80711b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009d90 tx=0x7fc5a402bea0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 -- 172.21.15.6:0/1259347412 shutdown_connections 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 --2- 172.21.15.6:0/1259347412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80711b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 --2- 172.21.15.6:0/1259347412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 -- 172.21.15.6:0/1259347412 >> 172.21.15.6:0/1259347412 conn(0x7fc5a806afb0 msgr2=0x7fc5a806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 -- 172.21.15.6:0/1259347412 shutdown_connections 2024-03-06T07:29:17.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.762+0000 7fc5b0800640 1 -- 172.21.15.6:0/1259347412 wait complete. 2024-03-06T07:29:17.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.764+0000 7fc5b0800640 1 Processor -- start 2024-03-06T07:29:17.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.764+0000 7fc5b0800640 1 -- start start 2024-03-06T07:29:17.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5b0800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5a806e240 0x7fc5a81f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5b0800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5b0800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5b0800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80ce600 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5b0800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80cf520 con 0x7fc5a806e240 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5b0800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a80ce780 con 0x7fc5a8161670 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5ad400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5ad400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51592/0 (socket says 172.21.15.6:51592) 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5ad400640 1 -- 172.21.15.6:0/3625130018 learned_addr learned my addr 172.21.15.6:0/3625130018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5ae800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.765+0000 7fc5ade00640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5a806e240 0x7fc5a81f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ad400640 1 -- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 msgr2=0x7fc5a8150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ad400640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ad400640 1 -- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5a806e240 msgr2=0x7fc5a81f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ad400640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5a806e240 0x7fc5a81f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ad400640 1 -- 172.21.15.6:0/3625130018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a8150cb0 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ade00640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5a806e240 0x7fc5a81f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:17.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5ad400640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80c9150 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc59400c500 tx=0x7fc59400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594018070 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc594007580 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc594004a20 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.766+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a81f4460 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.767+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a81f4a20 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.768+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a8069510 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.769+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc594004bc0 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.769+0000 7fc5a2000640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc58806b4d0 0x7fc58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.770+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc594088fe0 con 0x7fc5a80ae0a0 2024-03-06T07:29:17.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.770+0000 7fc5ade00640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc58806b4d0 0x7fc58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.771+0000 7fc5ade00640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc58806b4d0 0x7fc58806d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc59c006fd0 tx=0x7fc59c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:17.774+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc594057b60 con 0x7fc5a80ae0a0 2024-03-06T07:29:18.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.049+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fc5a80cf990 con 0x7fc5a80ae0a0 2024-03-06T07:29:18.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.085+0000 7fc5a2000640 1 -- 172.21.15.6:0/3625130018 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v124) v1 ==== 132+0+0 (secure 0 0 0) 0x7fc59405bc00 con 0x7fc5a80ae0a0 2024-03-06T07:29:18.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc58806b4d0 msgr2=0x7fc58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc58806b4d0 0x7fc58806d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc59c006fd0 tx=0x7fc59c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:18.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 msgr2=0x7fc5a80c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80c9150 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc59400c500 tx=0x7fc59400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 shutdown_connections 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc58806b4d0 0x7fc58806d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5a8161670 0x7fc5a8150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5a80ae0a0 0x7fc5a80c9150 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 --2- 172.21.15.6:0/3625130018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5a806e240 0x7fc5a81f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 >> 172.21.15.6:0/3625130018 conn(0x7fc5a806afb0 msgr2=0x7fc5a806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 shutdown_connections 2024-03-06T07:29:18.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.088+0000 7fc5b0800640 1 -- 172.21.15.6:0/3625130018 wait complete. 2024-03-06T07:29:18.111 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:29:18.111 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:29:18.111 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:29:18.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.315+0000 7f0c06600640 1 Processor -- start 2024-03-06T07:29:18.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.315+0000 7f0c06600640 1 -- start start 2024-03-06T07:29:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.317+0000 7f0c06600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0c00145300 0x7f0c000b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.317+0000 7f0c06600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000714d0 con 0x7f0c00070f10 2024-03-06T07:29:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.317+0000 7f0c06600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000717b0 con 0x7f0c000cd060 2024-03-06T07:29:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.317+0000 7f0c06600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c00071a90 con 0x7f0c00145300 2024-03-06T07:29:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.317+0000 7f0bffe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0c00070f10 0x7f0c000cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33110/0 (socket says 172.21.15.6:33110) 2024-03-06T07:29:18.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.317+0000 7f0bffe00640 1 -- 172.21.15.6:0/3905956197 learned_addr learned my addr 172.21.15.6:0/3905956197 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0c05200640 1 --2- 172.21.15.6:0/3905956197 >> v2:172.21.15.178:3300/0 conn(0x7f0c00145300 0x7f0c000b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029597641 0 0) 0x7f0c000714d0 con 0x7f0c00070f10 2024-03-06T07:29:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be8003520 con 0x7f0c00070f10 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 553109266 0 0) 0x7f0be8003520 con 0x7f0c00070f10 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 >> v2:172.21.15.178:3300/0 conn(0x7f0c00145300 msgr2=0x7f0c000b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 --2- 172.21.15.6:0/3905956197 >> v2:172.21.15.178:3300/0 conn(0x7f0c00145300 0x7f0c000b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 >> v1:172.21.15.178:6789/0 conn(0x7f0c000cd060 legacy=0x7f0c0015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c00071d70 con 0x7f0c00070f10 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0bec001670 con 0x7f0c00070f10 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 >> v1:172.21.15.6:6789/0 conn(0x7f0c00070f10 legacy=0x7f0c000cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0be8003a00 0x7f0be8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8004800 0x7f0be8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be8005210 0x7f0be800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00071a90 con 0x7f0be8005210 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be800c160 con 0x7f0be8003a00 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.318+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be800c360 con 0x7f0be8004800 2024-03-06T07:29:18.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05c00640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be8005210 0x7f0be800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05200640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0be8003a00 0x7f0be8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0bffe00640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8004800 0x7f0be8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05c00640 1 -- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8004800 msgr2=0x7f0be8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05c00640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8004800 0x7f0be8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05c00640 1 -- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0be8003a00 msgr2=0x7f0be8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05c00640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0be8003a00 0x7f0be8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05c00640 1 -- 172.21.15.6:0/3905956197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8003520 con 0x7f0be8005210 2024-03-06T07:29:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.319+0000 7f0c05200640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0be8003a00 0x7f0be8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:18.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.320+0000 7f0c05c00640 1 -- 172.21.15.6:0/3905956197 reap_dead start 2024-03-06T07:29:18.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.320+0000 7f0c05c00640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be8005210 0x7f0be800b9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c00072090 tx=0x7f0be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744f007b0c16d41a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8001a70 con 0x7f0be8005210 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf8003b70 con 0x7f0be8005210 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0bff400640 1 -- 172.21.15.6:0/3905956197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80050d0 con 0x7f0be8005210 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 -- 172.21.15.6:0/3905956197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be8005210 msgr2=0x7f0be800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be8005210 0x7f0be800b9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c00072090 tx=0x7f0be8005830 comp rx=0 tx=0).stop 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 -- 172.21.15.6:0/3905956197 shutdown_connections 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 --2- 172.21.15.6:0/3905956197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0be8005210 0x7f0be800b9c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 -- 172.21.15.6:0/3905956197 >> 172.21.15.6:0/3905956197 conn(0x7f0c0006b3d0 msgr2=0x7f0c0006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 -- 172.21.15.6:0/3905956197 shutdown_connections 2024-03-06T07:29:18.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.321+0000 7f0c06600640 1 -- 172.21.15.6:0/3905956197 wait complete. 2024-03-06T07:29:18.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.323+0000 7f0c06600640 1 Processor -- start 2024-03-06T07:29:18.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.323+0000 7f0c06600640 1 -- start start 2024-03-06T07:29:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.324+0000 7f0c06600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8005210 0x7f0c001411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.324+0000 7f0c06600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0015af00 0x7f0c0015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.324+0000 7f0c06600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 0x7f0c001f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.324+0000 7f0c06600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000ccad0 con 0x7f0c001f4fa0 2024-03-06T07:29:18.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.324+0000 7f0c06600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000cc950 con 0x7f0c0015af00 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.324+0000 7f0c06600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000ccc50 con 0x7f0be8005210 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 0x7f0c001f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 0x7f0c001f5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51610/0 (socket says 172.21.15.6:51610) 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 -- 172.21.15.6:0/572293048 learned_addr learned my addr 172.21.15.6:0/572293048 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0bffe00640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0015af00 0x7f0c0015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05200640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8005210 0x7f0c001411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 -- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8005210 msgr2=0x7f0c001411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8005210 0x7f0c001411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 -- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0015af00 msgr2=0x7f0c0015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0015af00 0x7f0c0015b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 -- 172.21.15.6:0/572293048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c001f5cd0 con 0x7f0c001f4fa0 2024-03-06T07:29:18.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.325+0000 7f0c05c00640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 0x7f0c001f5420 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8005040 tx=0x7f0bf80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.326+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8007db0 con 0x7f0c001f4fa0 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.326+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf8006d60 con 0x7f0c001f4fa0 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.326+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80056a0 con 0x7f0c001f4fa0 2024-03-06T07:29:18.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.326+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000aae50 con 0x7f0c001f4fa0 2024-03-06T07:29:18.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.326+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000ab410 con 0x7f0c001f4fa0 2024-03-06T07:29:18.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.327+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0004f330 con 0x7f0c001f4fa0 2024-03-06T07:29:18.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.328+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bf8007030 con 0x7f0c001f4fa0 2024-03-06T07:29:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.329+0000 7f0bfcc00640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0be006b500 0x7f0be006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.329+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0bf80aad30 con 0x7f0c001f4fa0 2024-03-06T07:29:18.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.329+0000 7f0c05200640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0be006b500 0x7f0be006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.330+0000 7f0c05200640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0be006b500 0x7f0be006d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009990 tx=0x7f0bf4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.333+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bf8049020 con 0x7f0c001f4fa0 2024-03-06T07:29:18.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.533+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c000ab5e0 con 0x7f0c001f4fa0 2024-03-06T07:29:18.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.535+0000 7f0bfcc00640 1 -- 172.21.15.6:0/572293048 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12946 (secure 0 0 0) 0x7f0bf803f070 con 0x7f0c001f4fa0 2024-03-06T07:29:18.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:18.537 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:17.061892+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/2554316293":"2024-03-07T07:29:13.979313+0000","172.21.15.6:6833/2554316293":"2024-03-07T07:29:13.979313+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0be006b500 msgr2=0x7f0be006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0be006b500 0x7f0be006d9c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009990 tx=0x7f0bf4008040 comp rx=0 tx=0).stop 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 msgr2=0x7f0c001f5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 0x7f0c001f5420 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8005040 tx=0x7f0bf80014a0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 shutdown_connections 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0be006b500 0x7f0be006d9c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c0015af00 0x7f0c0015b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c001f4fa0 0x7f0c001f5420 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 --2- 172.21.15.6:0/572293048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0be8005210 0x7f0c001411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 >> 172.21.15.6:0/572293048 conn(0x7f0c0006b3d0 msgr2=0x7f0c0015cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.537+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 shutdown_connections 2024-03-06T07:29:18.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.538+0000 7f0c06600640 1 -- 172.21.15.6:0/572293048 wait complete. 2024-03-06T07:29:18.557 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:18.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.777+0000 7f81fe400640 1 Processor -- start 2024-03-06T07:29:18.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.777+0000 7f81fe400640 1 -- start start 2024-03-06T07:29:18.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.778+0000 7f81fe400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f81f806da40 0x7f81f806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.779+0000 7f81fe400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f806d090 con 0x7f81f815e680 2024-03-06T07:29:18.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.779+0000 7f81fe400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f815eb80 con 0x7f81f80ccad0 2024-03-06T07:29:18.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.779+0000 7f81fe400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f8161c70 con 0x7f81f806da40 2024-03-06T07:29:18.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.779+0000 7f81f6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81f815e680 0x7f81f806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33112/0 (socket says 172.21.15.6:33112) 2024-03-06T07:29:18.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.779+0000 7f81f6a00640 1 -- 172.21.15.6:0/4034635913 learned_addr learned my addr 172.21.15.6:0/4034635913 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:18.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.779+0000 7f81f7400640 1 --2- 172.21.15.6:0/4034635913 >> v2:172.21.15.6:3300/0 conn(0x7f81f806da40 0x7f81f806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060206267 0 0) 0x7f81f806d090 con 0x7f81f815e680 2024-03-06T07:29:18.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81dc003520 con 0x7f81f815e680 2024-03-06T07:29:18.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896378389 0 0) 0x7f81f815eb80 con 0x7f81f80ccad0 2024-03-06T07:29:18.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f806d090 con 0x7f81f80ccad0 2024-03-06T07:29:18.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4821397 0 0) 0x7f81dc003520 con 0x7f81f815e680 2024-03-06T07:29:18.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f7400640 1 -- 172.21.15.6:0/4034635913 >> v1:172.21.15.178:6789/0 conn(0x7f81f80ccad0 legacy=0x7f81f8162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f7400640 1 -- 172.21.15.6:0/4034635913 >> v1:172.21.15.6:6789/0 conn(0x7f81f815e680 legacy=0x7f81f806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.780+0000 7f81f7400640 1 -- 172.21.15.6:0/4034635913 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f815ee60 con 0x7f81f806da40 2024-03-06T07:29:18.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.781+0000 7f81f7400640 1 --2- 172.21.15.6:0/4034635913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f806e340 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f81e80097a0 tx=0x7f81e802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec5005c3424e53c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.782+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8038070 con 0x7f81f806da40 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.782+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e8031500 con 0x7f81f806da40 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.782+0000 7f81f6000640 1 -- 172.21.15.6:0/4034635913 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0001670 con 0x7f81f806da40 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.782+0000 7f81fe400640 1 -- 172.21.15.6:0/4034635913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 msgr2=0x7f81f806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.782+0000 7f81fe400640 1 --2- 172.21.15.6:0/4034635913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f806e340 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f81e80097a0 tx=0x7f81e802b820 comp rx=0 tx=0).stop 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.783+0000 7f81fe400640 1 -- 172.21.15.6:0/4034635913 shutdown_connections 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.783+0000 7f81fe400640 1 --2- 172.21.15.6:0/4034635913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f806e340 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.783+0000 7f81fe400640 1 -- 172.21.15.6:0/4034635913 >> 172.21.15.6:0/4034635913 conn(0x7f81f806b3d0 msgr2=0x7f81f806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.783+0000 7f81fe400640 1 -- 172.21.15.6:0/4034635913 shutdown_connections 2024-03-06T07:29:18.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.783+0000 7f81fe400640 1 -- 172.21.15.6:0/4034635913 wait complete. 2024-03-06T07:29:18.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.785+0000 7f81fe400640 1 Processor -- start 2024-03-06T07:29:18.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.785+0000 7f81fe400640 1 -- start start 2024-03-06T07:29:18.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81fe400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f809b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81fe400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81f80ccad0 0x7f81f8153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81fe400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81f815e680 0x7f81f815baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81fe400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f806e880 con 0x7f81f806da40 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81fe400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f815f420 con 0x7f81f80ccad0 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81fe400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f806ea00 con 0x7f81f815e680 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f809b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f809b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51632/0 (socket says 172.21.15.6:51632) 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 -- 172.21.15.6:0/1964417506 learned_addr learned my addr 172.21.15.6:0/1964417506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f6a00640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81f80ccad0 0x7f81f8153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 -- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81f815e680 msgr2=0x7f81f815baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81f815e680 0x7f81f815baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 -- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81f80ccad0 msgr2=0x7f81f8153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81f80ccad0 0x7f81f8153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.786+0000 7f81f7400640 1 -- 172.21.15.6:0/1964417506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f815c610 con 0x7f81f806da40 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.787+0000 7f81f7400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f809b870 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000c00 tx=0x7f81e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.787+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8043070 con 0x7f81f806da40 2024-03-06T07:29:18.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.787+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e803f870 con 0x7f81f806da40 2024-03-06T07:29:18.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.787+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8038040 con 0x7f81f806da40 2024-03-06T07:29:18.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.787+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f815c8f0 con 0x7f81f806da40 2024-03-06T07:29:18.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.787+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f80ca710 con 0x7f81f806da40 2024-03-06T07:29:18.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.789+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f804f330 con 0x7f81f806da40 2024-03-06T07:29:18.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.789+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81e8046070 con 0x7f81f806da40 2024-03-06T07:29:18.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.790+0000 7f81ef400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81d406b500 0x7f81d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:18.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.790+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f81e80ab890 con 0x7f81f806da40 2024-03-06T07:29:18.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.791+0000 7f81f6a00640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81d406b500 0x7f81d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:18.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.792+0000 7f81f6a00640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81d406b500 0x7f81d406d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001380 tx=0x7f81e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:18.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.795+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81e804d070 con 0x7f81f806da40 2024-03-06T07:29:18.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.987+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81f806ba60 con 0x7f81f806da40 2024-03-06T07:29:18.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.988+0000 7f81ef400640 1 -- 172.21.15.6:0/1964417506 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81e8030070 con 0x7f81f806da40 2024-03-06T07:29:18.991 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:29:18.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81d406b500 msgr2=0x7f81d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81d406b500 0x7f81d406d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001380 tx=0x7f81e0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:18.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 msgr2=0x7f81f809b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f809b870 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000c00 tx=0x7f81e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 shutdown_connections 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81d406b500 0x7f81d406d9c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81f80ccad0 0x7f81f8153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81f815e680 0x7f81f815baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 --2- 172.21.15.6:0/1964417506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81f806da40 0x7f81f809b870 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 >> 172.21.15.6:0/1964417506 conn(0x7f81f806b3d0 msgr2=0x7f81f806de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 shutdown_connections 2024-03-06T07:29:18.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:18.991+0000 7f81fe400640 1 -- 172.21.15.6:0/1964417506 wait complete. 2024-03-06T07:29:19.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:29:19.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.222+0000 7f964be00640 1 Processor -- start 2024-03-06T07:29:19.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.222+0000 7f964be00640 1 -- start start 2024-03-06T07:29:19.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.223+0000 7f964be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 0x7f964c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.223+0000 7f964be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f964c0cce90 0x7f964c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.223+0000 7f964be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c0714d0 con 0x7f964c145300 2024-03-06T07:29:19.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.223+0000 7f964be00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c0cd300 con 0x7f964c0cce90 2024-03-06T07:29:19.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.223+0000 7f964be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0717b0 con 0x7f964c070f10 2024-03-06T07:29:19.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f964aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f964c145300 0x7f964c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33122/0 (socket says 172.21.15.6:33122) 2024-03-06T07:29:19.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f964aa00640 1 -- 172.21.15.6:0/952430197 learned_addr learned my addr 172.21.15.6:0/952430197 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f964a000640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 0x7f964c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f964b400640 1 --2- 172.21.15.6:0/952430197 >> v2:172.21.15.178:3300/0 conn(0x7f964c0cce90 0x7f964c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253034694 0 0) 0x7f964c0714d0 con 0x7f964c145300 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9630003520 con 0x7f964c145300 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.224+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190670230 0 0) 0x7f9630003520 con 0x7f964c145300 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 msgr2=0x7f964c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 0x7f964c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 >> v2:172.21.15.178:3300/0 conn(0x7f964c0cce90 msgr2=0x7f964c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 --2- 172.21.15.6:0/952430197 >> v2:172.21.15.178:3300/0 conn(0x7f964c0cce90 0x7f964c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c071a90 con 0x7f964c145300 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f963c001670 con 0x7f964c145300 2024-03-06T07:29:19.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 >> v1:172.21.15.6:6789/0 conn(0x7f964c145300 legacy=0x7f964c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 0x7f96300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9630003f30 0x7f9630004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9630004d40 0x7f96300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0cd300 con 0x7f9630003f30 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0717b0 con 0x7f9630004d40 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963000b9c0 con 0x7f964c070f10 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f964a000640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9630003f30 0x7f9630004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f964b400640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9630004d40 0x7f96300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.225+0000 7f964aa00640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 0x7f96300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.226+0000 7f964a000640 1 -- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 msgr2=0x7f96300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.226+0000 7f964a000640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f964c070f10 0x7f96300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.226+0000 7f964a000640 1 -- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9630004d40 msgr2=0x7f96300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.226+0000 7f964a000640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9630004d40 0x7f96300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.226+0000 7f964a000640 1 -- 172.21.15.6:0/952430197 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630003520 con 0x7f9630003f30 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.227+0000 7f964a000640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9630003f30 0x7f9630004800 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9638009ac0 tx=0x7f9630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40dc485778dd48f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.227+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638039070 con 0x7f9630003f30 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.227+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963802c6b0 con 0x7f9630003f30 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.227+0000 7f9649600640 1 -- 172.21.15.6:0/952430197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9638009690 con 0x7f9630003f30 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964be00640 1 -- 172.21.15.6:0/952430197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9630003f30 msgr2=0x7f9630004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964be00640 1 --2- 172.21.15.6:0/952430197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9630003f30 0x7f9630004800 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9638009ac0 tx=0x7f9630005830 comp rx=0 tx=0).stop 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964b400640 1 -- 172.21.15.6:0/952430197 reap_dead start 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964be00640 1 -- 172.21.15.6:0/952430197 shutdown_connections 2024-03-06T07:29:19.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964be00640 1 -- 172.21.15.6:0/952430197 >> 172.21.15.6:0/952430197 conn(0x7f964c06b3d0 msgr2=0x7f964c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964be00640 1 -- 172.21.15.6:0/952430197 shutdown_connections 2024-03-06T07:29:19.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.228+0000 7f964be00640 1 -- 172.21.15.6:0/952430197 wait complete. 2024-03-06T07:29:19.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.230+0000 7f964be00640 1 Processor -- start 2024-03-06T07:29:19.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.230+0000 7f964be00640 1 -- start start 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9630003f30 0x7f964c140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f964c06d550 0x7f964c13f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 0x7f964c140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0cc950 con 0x7f964c13fe80 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c1456e0 con 0x7f964c06d550 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0ccad0 con 0x7f9630003f30 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 0x7f964c140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 0x7f964c140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51662/0 (socket says 172.21.15.6:51662) 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 -- 172.21.15.6:0/2478727502 learned_addr learned my addr 172.21.15.6:0/2478727502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964aa00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9630003f30 0x7f964c140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964a000640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f964c06d550 0x7f964c13f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 -- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9630003f30 msgr2=0x7f964c140d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9630003f30 0x7f964c140d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 -- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f964c06d550 msgr2=0x7f964c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f964c06d550 0x7f964c13f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964b400640 1 -- 172.21.15.6:0/2478727502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c06dd30 con 0x7f964c13fe80 2024-03-06T07:29:19.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.231+0000 7f964a000640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f964c06d550 0x7f964c13f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:19.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.232+0000 7f964b400640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 0x7f964c140300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f964c0cefa0 tx=0x7f96440115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.232+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644011db0 con 0x7f964c13fe80 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.232+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964401b070 con 0x7f964c13fe80 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.232+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644003990 con 0x7f964c13fe80 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.232+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c0ae3c0 con 0x7f964c13fe80 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.232+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c0af140 con 0x7f964c13fe80 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.233+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c04f330 con 0x7f964c13fe80 2024-03-06T07:29:19.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.234+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9644005d10 con 0x7f964c13fe80 2024-03-06T07:29:19.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.235+0000 7f9642a00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f962c06b500 0x7f962c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.236+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f964408a440 con 0x7f964c13fe80 2024-03-06T07:29:19.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.236+0000 7f964aa00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f962c06b500 0x7f962c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.237+0000 7f964aa00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f962c06b500 0x7f962c06d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f963c001050 tx=0x7f963c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.241+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9644025020 con 0x7f964c13fe80 2024-03-06T07:29:19.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:19.516+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f964c0b0440 con 0x7f964c13fe80 2024-03-06T07:29:20.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.097+0000 7f9642a00640 1 -- 172.21.15.6:0/2478727502 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v126) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9644014350 con 0x7f964c13fe80 2024-03-06T07:29:20.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f962c06b500 msgr2=0x7f962c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f962c06b500 0x7f962c06d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f963c001050 tx=0x7f963c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 msgr2=0x7f964c140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 0x7f964c140300 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f964c0cefa0 tx=0x7f96440115a0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 shutdown_connections 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f962c06b500 0x7f962c06d9c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f964c06d550 0x7f964c13f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f964c13fe80 0x7f964c140300 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 --2- 172.21.15.6:0/2478727502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9630003f30 0x7f964c140d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 >> 172.21.15.6:0/2478727502 conn(0x7f964c06b3d0 msgr2=0x7f964c0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 shutdown_connections 2024-03-06T07:29:20.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.100+0000 7f964be00640 1 -- 172.21.15.6:0/2478727502 wait complete. 2024-03-06T07:29:20.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:29:20.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.347+0000 7f9d1e400640 1 Processor -- start 2024-03-06T07:29:20.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.347+0000 7f9d1e400640 1 -- start start 2024-03-06T07:29:20.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.348+0000 7f9d1e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 0x7f9d18169090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d1e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d180cf780 con 0x7f9d180d07d0 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d1e400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d180cfa60 con 0x7f9d180d4930 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d1e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180b3770 con 0x7f9d1815e6c0 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d17400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9d180d07d0 0x7f9d180b4f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33138/0 (socket says 172.21.15.6:33138) 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d17400640 1 -- 172.21.15.6:0/1398865057 learned_addr learned my addr 172.21.15.6:0/1398865057 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d17e00640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 0x7f9d18169090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.349+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452917506 0 0) 0x7f9d180cf780 con 0x7f9d180d07d0 2024-03-06T07:29:20.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cfc003520 con 0x7f9d180d07d0 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 298593455 0 0) 0x7f9cfc003520 con 0x7f9d180d07d0 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 msgr2=0x7f9d18169090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 0x7f9d18169090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 >> v1:172.21.15.178:6789/0 conn(0x7f9d180d4930 legacy=0x7f9d1815dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d180cfd40 con 0x7f9d180d07d0 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9d08001670 con 0x7f9d180d07d0 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 >> v1:172.21.15.6:6789/0 conn(0x7f9d180d07d0 legacy=0x7f9d180b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cfc0039e0 0x7f9cfc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 0x7f9cfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc004d40 0x7f9cfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180b3770 con 0x7f9cfc004d40 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc00bb60 con 0x7f9cfc0039e0 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc00bd60 con 0x7f9d1815e6c0 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.350+0000 7f9d17e00640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc004d40 0x7f9cfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d16a00640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 0x7f9cfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d17400640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cfc0039e0 0x7f9cfc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d17e00640 1 -- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 msgr2=0x7f9cfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d17e00640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d1815e6c0 0x7f9cfc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d17e00640 1 -- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cfc0039e0 msgr2=0x7f9cfc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d17e00640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9cfc0039e0 0x7f9cfc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.351+0000 7f9d17e00640 1 -- 172.21.15.6:0/1398865057 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc003520 con 0x7f9cfc004d40 2024-03-06T07:29:20.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.352+0000 7f9d17e00640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc004d40 0x7f9cfc0051c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009f90 tx=0x7f9cfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5953d0f33628631d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.352+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10039070 con 0x7f9cfc004d40 2024-03-06T07:29:20.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.352+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1002cbd0 con 0x7f9cfc004d40 2024-03-06T07:29:20.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.352+0000 7f9d16000640 1 -- 172.21.15.6:0/1398865057 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d10009c10 con 0x7f9cfc004d40 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d1e400640 1 -- 172.21.15.6:0/1398865057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc004d40 msgr2=0x7f9cfc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d1e400640 1 --2- 172.21.15.6:0/1398865057 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cfc004d40 0x7f9cfc0051c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009f90 tx=0x7f9cfc005830 comp rx=0 tx=0).stop 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d17e00640 1 -- 172.21.15.6:0/1398865057 reap_dead start 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d1e400640 1 -- 172.21.15.6:0/1398865057 shutdown_connections 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d1e400640 1 -- 172.21.15.6:0/1398865057 >> 172.21.15.6:0/1398865057 conn(0x7f9d180d3e10 msgr2=0x7f9d180d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d1e400640 1 -- 172.21.15.6:0/1398865057 shutdown_connections 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.353+0000 7f9d1e400640 1 -- 172.21.15.6:0/1398865057 wait complete. 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.355+0000 7f9d1e400640 1 Processor -- start 2024-03-06T07:29:20.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.355+0000 7f9d1e400640 1 -- start start 2024-03-06T07:29:20.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d1e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9cfc004d40 0x7f9d18156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d1e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 0x7f9d181f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d1e400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d18156880 0x7f9d181f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d1e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180d0d30 con 0x7f9d18154bb0 2024-03-06T07:29:20.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d1e400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180d0bb0 con 0x7f9d18156880 2024-03-06T07:29:20.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d1e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d180d0eb0 con 0x7f9cfc004d40 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 0x7f9d181f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 0x7f9d181f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51680/0 (socket says 172.21.15.6:51680) 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.356+0000 7f9d16a00640 1 -- 172.21.15.6:0/3844425182 learned_addr learned my addr 172.21.15.6:0/3844425182 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d17400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9cfc004d40 0x7f9d18156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d17e00640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d18156880 0x7f9d181f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d16a00640 1 -- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9cfc004d40 msgr2=0x7f9d18156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d16a00640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9cfc004d40 0x7f9d18156340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d16a00640 1 -- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d18156880 msgr2=0x7f9d181f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d16a00640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d18156880 0x7f9d181f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d16a00640 1 -- 172.21.15.6:0/3844425182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d18156e50 con 0x7f9d18154bb0 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d17e00640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d18156880 0x7f9d181f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d16a00640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 0x7f9d181f7560 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400e580 tx=0x7f9d0400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d040052e0 con 0x7f9d18154bb0 2024-03-06T07:29:20.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04018070 con 0x7f9d18154bb0 2024-03-06T07:29:20.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04003810 con 0x7f9d18154bb0 2024-03-06T07:29:20.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d181540d0 con 0x7f9d18154bb0 2024-03-06T07:29:20.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.357+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d180cc4d0 con 0x7f9d18154bb0 2024-03-06T07:29:20.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.359+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d18153ef0 con 0x7f9d18154bb0 2024-03-06T07:29:20.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.360+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d040039b0 con 0x7f9d18154bb0 2024-03-06T07:29:20.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.360+0000 7f9d0f400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9cf806b4d0 0x7f9cf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.361+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9d04089b60 con 0x7f9d18154bb0 2024-03-06T07:29:20.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.361+0000 7f9d17400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9cf806b4d0 0x7f9cf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.362+0000 7f9d17400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9cf806b4d0 0x7f9cf806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9d08001eb0 tx=0x7f9d08000f50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.365+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d18153ef0 con 0x7f9d18154bb0 2024-03-06T07:29:20.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.566+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9d1804f330 con 0x7f9d18154bb0 2024-03-06T07:29:20.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.573+0000 7f9d0f400640 1 -- 172.21.15.6:0/3844425182 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d04058cf0 con 0x7f9d18154bb0 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9cf806b4d0 msgr2=0x7f9cf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9cf806b4d0 0x7f9cf806d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9d08001eb0 tx=0x7f9d08000f50 comp rx=0 tx=0).stop 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 msgr2=0x7f9d181f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 0x7f9d181f7560 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9d0400e580 tx=0x7f9d0400ea40 comp rx=0 tx=0).stop 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 shutdown_connections 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9cf806b4d0 0x7f9cf806d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d18156880 0x7f9d181f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d18154bb0 0x7f9d181f7560 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 --2- 172.21.15.6:0/3844425182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9cfc004d40 0x7f9d18156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 >> 172.21.15.6:0/3844425182 conn(0x7f9d180d3e10 msgr2=0x7f9d180b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 shutdown_connections 2024-03-06T07:29:20.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:20.576+0000 7f9d1e400640 1 -- 172.21.15.6:0/3844425182 wait complete. 2024-03-06T07:29:20.598 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:20.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:20.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:20.838 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:20.838 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:20.838 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:20.898 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:20.898 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:29:20.898 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:20.954 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:20.954 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:20.954 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:21.013 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:21.013 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:29:21.013 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:21.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:21.068 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:29:21.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:21.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.297+0000 7fc3b0800640 1 Processor -- start 2024-03-06T07:29:21.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.297+0000 7fc3b0800640 1 -- start start 2024-03-06T07:29:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.299+0000 7fc3b0800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc3a80b3c90 0x7fc3a80cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.299+0000 7fc3b0800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 0x7fc3a80b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.299+0000 7fc3b0800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80b45b0 con 0x7fc3a806cb10 2024-03-06T07:29:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.299+0000 7fc3b0800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80ce160 con 0x7fc3a80b3c90 2024-03-06T07:29:21.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.299+0000 7fc3b0800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80b4860 con 0x7fc3a80cf5c0 2024-03-06T07:29:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.299+0000 7fc3ad400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 0x7fc3a80b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ade00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc3a80b3c90 0x7fc3a80cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ad400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 0x7fc3a80b4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:57882/0 (socket says 172.21.15.6:57882) 2024-03-06T07:29:21.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ad400640 1 -- 172.21.15.6:0/1021794082 learned_addr learned my addr 172.21.15.6:0/1021794082 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:21.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ad400640 1 -- 172.21.15.6:0/1021794082 >> v2:172.21.15.178:3300/0 conn(0x7fc3a80b3c90 msgr2=0x7fc3a80cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:21.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ad400640 1 --2- 172.21.15.6:0/1021794082 >> v2:172.21.15.178:3300/0 conn(0x7fc3a80b3c90 0x7fc3a80cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ad400640 1 -- 172.21.15.6:0/1021794082 >> v1:172.21.15.178:6789/0 conn(0x7fc3a806cb10 legacy=0x7fc3a806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.300+0000 7fc3ad400640 1 -- 172.21.15.6:0/1021794082 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80b4b40 con 0x7fc3a80cf5c0 2024-03-06T07:29:21.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.302+0000 7fc3ad400640 1 --2- 172.21.15.6:0/1021794082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 0x7fc3a80b4070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc398009870 tx=0x7fc39802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19187e07c992e1b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.302+0000 7fc3aca00640 1 -- 172.21.15.6:0/1021794082 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39802f830 con 0x7fc3a80cf5c0 2024-03-06T07:29:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3aca00640 1 -- 172.21.15.6:0/1021794082 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39802f9d0 con 0x7fc3a80cf5c0 2024-03-06T07:29:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3aca00640 1 -- 172.21.15.6:0/1021794082 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39802fda0 con 0x7fc3a80cf5c0 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3b0800640 1 -- 172.21.15.6:0/1021794082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 msgr2=0x7fc3a80b4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1021794082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 0x7fc3a80b4070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc398009870 tx=0x7fc39802ba10 comp rx=0 tx=0).stop 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3b0800640 1 -- 172.21.15.6:0/1021794082 shutdown_connections 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1021794082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a80cf5c0 0x7fc3a80b4070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1021794082 >> v2:172.21.15.178:3300/0 conn(0x7fc3a80b3c90 0x7fc3a80cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.303+0000 7fc3b0800640 1 -- 172.21.15.6:0/1021794082 >> 172.21.15.6:0/1021794082 conn(0x7fc3a806afb0 msgr2=0x7fc3a806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.304+0000 7fc3b0800640 1 -- 172.21.15.6:0/1021794082 shutdown_connections 2024-03-06T07:29:21.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.304+0000 7fc3b0800640 1 -- 172.21.15.6:0/1021794082 wait complete. 2024-03-06T07:29:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.305+0000 7fc3b0800640 1 Processor -- start 2024-03-06T07:29:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.305+0000 7fc3b0800640 1 -- start start 2024-03-06T07:29:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.306+0000 7fc3b0800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a806cb10 0x7fc3a8146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.306+0000 7fc3b0800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 0x7fc3a809c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.306+0000 7fc3b0800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3a80cf5c0 0x7fc3a8072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.306+0000 7fc3b0800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a815b340 con 0x7fc3a80b3c90 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.306+0000 7fc3b0800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80b5100 con 0x7fc3a80cf5c0 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.306+0000 7fc3b0800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a806d6b0 con 0x7fc3a806cb10 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 0x7fc3a809c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 0x7fc3a809c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51692/0 (socket says 172.21.15.6:51692) 2024-03-06T07:29:21.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 -- 172.21.15.6:0/1602324106 learned_addr learned my addr 172.21.15.6:0/1602324106 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ade00640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a806cb10 0x7fc3a8146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ae800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3a80cf5c0 0x7fc3a8072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 -- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a806cb10 msgr2=0x7fc3a8146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a806cb10 0x7fc3a8146130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 -- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3a80cf5c0 msgr2=0x7fc3a8072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3a80cf5c0 0x7fc3a8072160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ad400640 1 -- 172.21.15.6:0/1602324106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a8151820 con 0x7fc3a80b3c90 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.307+0000 7fc3ae800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3a80cf5c0 0x7fc3a8072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.308+0000 7fc3ad400640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 0x7fc3a809c430 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc39802ba50 tx=0x7fc398004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.308+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398047070 con 0x7fc3a80b3c90 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.308+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398037c60 con 0x7fc3a80b3c90 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.308+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398042630 con 0x7fc3a80b3c90 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.308+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a8146db0 con 0x7fc3a80b3c90 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.308+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a8151540 con 0x7fc3a80b3c90 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.309+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a8069510 con 0x7fc3a80b3c90 2024-03-06T07:29:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.310+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc39804a070 con 0x7fc3a80b3c90 2024-03-06T07:29:21.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.311+0000 7fc3a2000640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38c06b500 0x7fc38c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.311+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc3980abcd0 con 0x7fc3a80b3c90 2024-03-06T07:29:21.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.312+0000 7fc3ade00640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38c06b500 0x7fc38c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.313+0000 7fc3ade00640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38c06b500 0x7fc38c06d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009ad0 tx=0x7fc39c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.316+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc398033030 con 0x7fc3a80b3c90 2024-03-06T07:29:21.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:21.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:21.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.554+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3a806be30 con 0x7fc3a80b3c90 2024-03-06T07:29:21.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.556+0000 7fc3a2000640 1 -- 172.21.15.6:0/1602324106 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fc39807aee0 con 0x7fc3a80b3c90 2024-03-06T07:29:21.558 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:21.558 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4855,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/347534079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":347534079},{"type":"v1","addr":"172.21.15.178:6836","nonce":347534079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4858,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1112785999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1112785999},{"type":"v1","addr":"172.21.15.178:6837","nonce":1112785999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5121,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/3634145100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3634145100},{"type":"v1","addr":"172.21.15.6:6834","nonce":3634145100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2995787982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":2995787982},{"type":"v1","addr":"172.21.15.6:6835","nonce":2995787982}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-03-06T07:29:21.558 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 19 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38c06b500 msgr2=0x7fc38c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38c06b500 0x7fc38c06d9c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009ad0 tx=0x7fc39c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 msgr2=0x7fc3a809c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 0x7fc3a809c430 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc39802ba50 tx=0x7fc398004730 comp rx=0 tx=0).stop 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 shutdown_connections 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc38c06b500 0x7fc38c06d9c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3a80cf5c0 0x7fc3a8072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3a80b3c90 0x7fc3a809c430 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 --2- 172.21.15.6:0/1602324106 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3a806cb10 0x7fc3a8146130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 >> 172.21.15.6:0/1602324106 conn(0x7fc3a806afb0 msgr2=0x7fc3a806dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 shutdown_connections 2024-03-06T07:29:21.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.558+0000 7fc3b0800640 1 -- 172.21.15.6:0/1602324106 wait complete. 2024-03-06T07:29:21.580 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:29:21.580 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:29:21.580 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:29:21.580 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:21.580 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:29:21.580 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:29:21.581 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:29:21.581 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:21.581 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:29:21.581 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:29:21.581 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:29:21.581 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:21.581 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:29:21.581 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:29:21.582 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:29:21.582 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:29:21.602 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:29:21.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:29:21.603 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:29:21.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:29:21.643 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:29:21.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:29:21.645 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:29:21.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:29:21.648 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:29:21.649 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:21.652 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:29:21.652 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:21.653 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:29:21.653 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:21.658 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:29:21.658 DEBUG:teuthology.parallel:result is None 2024-03-06T07:29:21.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:29:21.800 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:29:21.800 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:29:21.806 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:29:21.807 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:29:21.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.840+0000 7fbc05400640 1 Processor -- start 2024-03-06T07:29:21.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.841+0000 7fbc05400640 1 -- start start 2024-03-06T07:29:21.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbc05400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbc000af580 0x7fbc0015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbc05400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0015cc40 con 0x7fbc000af1a0 2024-03-06T07:29:21.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbc05400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0015cf20 con 0x7fbc000cdc20 2024-03-06T07:29:21.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbc05400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc0015d200 con 0x7fbc000af580 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbbfe800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbc000af1a0 0x7fbc000b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33144/0 (socket says 172.21.15.6:33144) 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbbfe800640 1 -- 172.21.15.6:0/2148622083 learned_addr learned my addr 172.21.15.6:0/2148622083 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.842+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/2148622083 >> v2:172.21.15.178:3300/0 conn(0x7fbc000af580 0x7fbc0015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.843+0000 7fbbfde00640 1 -- 172.21.15.6:0/2148622083 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836190685 0 0) 0x7fbc0015cc40 con 0x7fbc000af1a0 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.843+0000 7fbbfde00640 1 -- 172.21.15.6:0/2148622083 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe4003520 con 0x7fbc000af1a0 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.843+0000 7fbbf7e00640 1 -- 172.21.15.6:0/2148622083 >> v1:172.21.15.178:6789/0 conn(0x7fbc000cdc20 legacy=0x7fbc000b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.843+0000 7fbbf7e00640 1 -- 172.21.15.6:0/2148622083 >> v1:172.21.15.6:6789/0 conn(0x7fbc000af1a0 legacy=0x7fbc000b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.843+0000 7fbbf7e00640 1 -- 172.21.15.6:0/2148622083 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0015d4e0 con 0x7fbc000af580 2024-03-06T07:29:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.844+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/2148622083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af580 0x7fbc0015c600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80097a0 tx=0x7fbbe802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f89b10e3367b1d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.844+0000 7fbbfde00640 1 -- 172.21.15.6:0/2148622083 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8035020 con 0x7fbc000af580 2024-03-06T07:29:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.844+0000 7fbbfde00640 1 -- 172.21.15.6:0/2148622083 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe8031830 con 0x7fbc000af580 2024-03-06T07:29:21.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.844+0000 7fbbfde00640 1 -- 172.21.15.6:0/2148622083 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8031be0 con 0x7fbc000af580 2024-03-06T07:29:21.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 -- 172.21.15.6:0/2148622083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af580 msgr2=0x7fbc0015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 --2- 172.21.15.6:0/2148622083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af580 0x7fbc0015c600 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80097a0 tx=0x7fbbe802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 -- 172.21.15.6:0/2148622083 shutdown_connections 2024-03-06T07:29:21.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 --2- 172.21.15.6:0/2148622083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af580 0x7fbc0015c600 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 -- 172.21.15.6:0/2148622083 >> 172.21.15.6:0/2148622083 conn(0x7fbc0006afb0 msgr2=0x7fbc0006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 -- 172.21.15.6:0/2148622083 shutdown_connections 2024-03-06T07:29:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.845+0000 7fbc05400640 1 -- 172.21.15.6:0/2148622083 wait complete. 2024-03-06T07:29:21.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.847+0000 7fbc05400640 1 Processor -- start 2024-03-06T07:29:21.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.847+0000 7fbc05400640 1 -- start start 2024-03-06T07:29:21.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbc05400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af1a0 0x7fbc0014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbc05400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc000af580 0x7fbc0009c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbc05400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 0x7fbc000b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbc05400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00155e70 con 0x7fbc000cdc20 2024-03-06T07:29:21.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbc05400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc000b3730 con 0x7fbc000af1a0 2024-03-06T07:29:21.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbc05400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00155ff0 con 0x7fbc000af580 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 0x7fbc000b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 0x7fbc000b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51758/0 (socket says 172.21.15.6:51758) 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 -- 172.21.15.6:0/442568765 learned_addr learned my addr 172.21.15.6:0/442568765 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbfe800640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af1a0 0x7fbc0014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbf7e00640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc000af580 0x7fbc0009c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 -- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc000af580 msgr2=0x7fbc0009c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc000af580 0x7fbc0009c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 -- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af1a0 msgr2=0x7fbc0014abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af1a0 0x7fbc0014abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.848+0000 7fbbff200640 1 -- 172.21.15.6:0/442568765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0014c890 con 0x7fbc000cdc20 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.849+0000 7fbbff200640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 0x7fbc000b0a00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c730 tx=0x7fbbf800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.849+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8016020 con 0x7fbc000cdc20 2024-03-06T07:29:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.849+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf8006d50 con 0x7fbc000cdc20 2024-03-06T07:29:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.849+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8003bb0 con 0x7fbc000cdc20 2024-03-06T07:29:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.849+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0014cb70 con 0x7fbc000cdc20 2024-03-06T07:29:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.849+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0014c4b0 con 0x7fbc000cdc20 2024-03-06T07:29:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.850+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0014d090 con 0x7fbc000cdc20 2024-03-06T07:29:21.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.851+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbf80050d0 con 0x7fbc000cdc20 2024-03-06T07:29:21.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.852+0000 7fbbf6a00640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbdc06b4d0 0x7fbbdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:21.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.852+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbbf8089b30 con 0x7fbc000cdc20 2024-03-06T07:29:21.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.852+0000 7fbbfe800640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbdc06b4d0 0x7fbbdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:21.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.853+0000 7fbbfe800640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbdc06b4d0 0x7fbbdc06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00011e0 tx=0x7fbbf0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:21.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:21.857+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbf8020020 con 0x7fbc000cdc20 2024-03-06T07:29:22.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.059+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbc00069510 con 0x7fbc000cdc20 2024-03-06T07:29:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.122+0000 7fbbf6a00640 1 -- 172.21.15.6:0/442568765 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v40) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbbf8058ca0 con 0x7fbc000cdc20 2024-03-06T07:29:22.125 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:29:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.125+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbdc06b4d0 msgr2=0x7fbbdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbdc06b4d0 0x7fbbdc06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00011e0 tx=0x7fbbf0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 msgr2=0x7fbc000b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 0x7fbc000b0a00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c730 tx=0x7fbbf800cb30 comp rx=0 tx=0).stop 2024-03-06T07:29:22.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 shutdown_connections 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbdc06b4d0 0x7fbbdc06d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc000cdc20 0x7fbc000b0a00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc000af580 0x7fbc0009c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 --2- 172.21.15.6:0/442568765 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc000af1a0 0x7fbc0014abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 >> 172.21.15.6:0/442568765 conn(0x7fbc0006afb0 msgr2=0x7fbc00155300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.126+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 shutdown_connections 2024-03-06T07:29:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.127+0000 7fbc05400640 1 -- 172.21.15.6:0/442568765 wait complete. 2024-03-06T07:29:22.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:29:22.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.347+0000 7fbe19800640 1 Processor -- start 2024-03-06T07:29:22.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.347+0000 7fbe19800640 1 -- start start 2024-03-06T07:29:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.349+0000 7fbe19800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe14070ee0 0x7fbe14162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.349+0000 7fbe19800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbe140712c0 0x7fbe1406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.349+0000 7fbe19800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe140ce090 con 0x7fbe14072da0 2024-03-06T07:29:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.349+0000 7fbe19800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe140ceca0 con 0x7fbe14070ee0 2024-03-06T07:29:22.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.349+0000 7fbe19800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe140ce370 con 0x7fbe140712c0 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe14070ee0 0x7fbe14162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe14070ee0 0x7fbe14162c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50504/0 (socket says 172.21.15.6:50504) 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 -- 172.21.15.6:0/1838844244 learned_addr learned my addr 172.21.15.6:0/1838844244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 -- 172.21.15.6:0/1838844244 >> v2:172.21.15.178:3300/0 conn(0x7fbe140712c0 msgr2=0x7fbe1406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 --2- 172.21.15.6:0/1838844244 >> v2:172.21.15.178:3300/0 conn(0x7fbe140712c0 0x7fbe1406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 -- 172.21.15.6:0/1838844244 >> v1:172.21.15.178:6789/0 conn(0x7fbe14072da0 legacy=0x7fbe1406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.350+0000 7fbe12a00640 1 -- 172.21.15.6:0/1838844244 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe140ce650 con 0x7fbe14070ee0 2024-03-06T07:29:22.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.351+0000 7fbe12a00640 1 --2- 172.21.15.6:0/1838844244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14070ee0 0x7fbe14162c40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbe1406f070 tx=0x7fbe0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dff770fa5e085b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe11600640 1 -- 172.21.15.6:0/1838844244 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04035020 con 0x7fbe14070ee0 2024-03-06T07:29:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe11600640 1 -- 172.21.15.6:0/1838844244 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe04031870 con 0x7fbe14070ee0 2024-03-06T07:29:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe11600640 1 -- 172.21.15.6:0/1838844244 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04031c40 con 0x7fbe14070ee0 2024-03-06T07:29:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe19800640 1 -- 172.21.15.6:0/1838844244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14070ee0 msgr2=0x7fbe14162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe19800640 1 --2- 172.21.15.6:0/1838844244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14070ee0 0x7fbe14162c40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbe1406f070 tx=0x7fbe0402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe19800640 1 -- 172.21.15.6:0/1838844244 shutdown_connections 2024-03-06T07:29:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe19800640 1 --2- 172.21.15.6:0/1838844244 >> v2:172.21.15.178:3300/0 conn(0x7fbe140712c0 0x7fbe1406e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe19800640 1 --2- 172.21.15.6:0/1838844244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14070ee0 0x7fbe14162c40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.352+0000 7fbe19800640 1 -- 172.21.15.6:0/1838844244 >> 172.21.15.6:0/1838844244 conn(0x7fbe1406afb0 msgr2=0x7fbe1406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.353+0000 7fbe19800640 1 -- 172.21.15.6:0/1838844244 shutdown_connections 2024-03-06T07:29:22.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.353+0000 7fbe19800640 1 -- 172.21.15.6:0/1838844244 wait complete. 2024-03-06T07:29:22.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.354+0000 7fbe19800640 1 Processor -- start 2024-03-06T07:29:22.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.354+0000 7fbe19800640 1 -- start start 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.355+0000 7fbe19800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe14070ee0 0x7fbe1409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.355+0000 7fbe19800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe140712c0 0x7fbe1414ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.355+0000 7fbe19800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 0x7fbe14160860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.355+0000 7fbe19800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140cdda0 con 0x7fbe14072da0 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.355+0000 7fbe19800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140cdc20 con 0x7fbe14070ee0 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.355+0000 7fbe19800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe140cf130 con 0x7fbe140712c0 2024-03-06T07:29:22.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 0x7fbe14160860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 0x7fbe14160860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50516/0 (socket says 172.21.15.6:50516) 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 -- 172.21.15.6:0/1355562780 learned_addr learned my addr 172.21.15.6:0/1355562780 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe12a00640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe14070ee0 0x7fbe1409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe12000640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe140712c0 0x7fbe1414ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 -- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe140712c0 msgr2=0x7fbe1414ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe140712c0 0x7fbe1414ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 -- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe14070ee0 msgr2=0x7fbe1409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe14070ee0 0x7fbe1409aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe13400640 1 -- 172.21.15.6:0/1355562780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe141504e0 con 0x7fbe14072da0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.356+0000 7fbe12000640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe140712c0 0x7fbe1414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.357+0000 7fbe13400640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 0x7fbe14160860 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b9a0 tx=0x7fbe0c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.357+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c019070 con 0x7fbe14072da0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.357+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c00cbc0 con 0x7fbe14072da0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.357+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c014440 con 0x7fbe14072da0 2024-03-06T07:29:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.357+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe14151be0 con 0x7fbe14072da0 2024-03-06T07:29:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.357+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe141521a0 con 0x7fbe14072da0 2024-03-06T07:29:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.358+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe14069510 con 0x7fbe14072da0 2024-03-06T07:29:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.359+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe0c01c070 con 0x7fbe14072da0 2024-03-06T07:29:22.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.360+0000 7fbe0aa00640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbdf406b500 0x7fbdf406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.361+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbe0c089ba0 con 0x7fbe14072da0 2024-03-06T07:29:22.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.361+0000 7fbe12a00640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbdf406b500 0x7fbdf406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.362+0000 7fbe12a00640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbdf406b500 0x7fbdf406d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbe04036000 tx=0x7fbe0403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.365+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe141504e0 con 0x7fbe14072da0 2024-03-06T07:29:22.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.638+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbe1404f3a0 con 0x7fbe14072da0 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.639+0000 7fbe0aa00640 1 -- 172.21.15.6:0/1355562780 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbe0c058d50 con 0x7fbe14072da0 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:29:22.641 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:29:22.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:29:22.644 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.645 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.646 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.647 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:29:22.648 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:29:22.649 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.651 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:29:22.652 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:29:22.653 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:22.654 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:29:22.655 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.656 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:22.656 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:29:22.656 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbdf406b500 msgr2=0x7fbdf406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbdf406b500 0x7fbdf406d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbe04036000 tx=0x7fbe0403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 msgr2=0x7fbe14160860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 0x7fbe14160860 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00b9a0 tx=0x7fbe0c00be60 comp rx=0 tx=0).stop 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 shutdown_connections 2024-03-06T07:29:22.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbdf406b500 0x7fbdf406d9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe14072da0 0x7fbe14160860 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe140712c0 0x7fbe1414ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 --2- 172.21.15.6:0/1355562780 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe14070ee0 0x7fbe1409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 >> 172.21.15.6:0/1355562780 conn(0x7fbe1406afb0 msgr2=0x7fbe140b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 shutdown_connections 2024-03-06T07:29:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.642+0000 7fbe19800640 1 -- 172.21.15.6:0/1355562780 wait complete. 2024-03-06T07:29:22.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:22.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.881+0000 7f46ff200640 1 Processor -- start 2024-03-06T07:29:22.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.881+0000 7f46ff200640 1 -- start start 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46ff200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f47000cd060 0x7f4700161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46ff200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4700162000 0x7f4700070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46ff200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47001627f0 con 0x7f470015efc0 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46ff200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470015f3a0 con 0x7f4700162000 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46ff200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f470006d190 con 0x7f47000cd060 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46fe800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4700162000 0x7f4700070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46fe800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4700162000 0x7f4700070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50526/0 (socket says 172.21.15.6:50526) 2024-03-06T07:29:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46fe800640 1 -- 172.21.15.6:0/1289301262 learned_addr learned my addr 172.21.15.6:0/1289301262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.883+0000 7f46fd400640 1 --2- 172.21.15.6:0/1289301262 >> v2:172.21.15.178:3300/0 conn(0x7f47000cd060 0x7f4700161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.884+0000 7f46fe800640 1 -- 172.21.15.6:0/1289301262 >> v2:172.21.15.178:3300/0 conn(0x7f47000cd060 msgr2=0x7f4700161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.884+0000 7f46fe800640 1 --2- 172.21.15.6:0/1289301262 >> v2:172.21.15.178:3300/0 conn(0x7f47000cd060 0x7f4700161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.884+0000 7f46fe800640 1 -- 172.21.15.6:0/1289301262 >> v1:172.21.15.178:6789/0 conn(0x7f470015efc0 legacy=0x7f47000cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.884+0000 7f46fe800640 1 -- 172.21.15.6:0/1289301262 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470006c860 con 0x7f4700162000 2024-03-06T07:29:22.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.885+0000 7f46fe800640 1 --2- 172.21.15.6:0/1289301262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4700162000 0x7f4700070c20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f47000b51d0 tx=0x7f46f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57beec841071b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.885+0000 7f46fca00640 1 -- 172.21.15.6:0/1289301262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f802ca80 con 0x7f4700162000 2024-03-06T07:29:22.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.885+0000 7f46fca00640 1 -- 172.21.15.6:0/1289301262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f802cc20 con 0x7f4700162000 2024-03-06T07:29:22.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.885+0000 7f46fca00640 1 -- 172.21.15.6:0/1289301262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f80343f0 con 0x7f4700162000 2024-03-06T07:29:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 -- 172.21.15.6:0/1289301262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4700162000 msgr2=0x7f4700070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 --2- 172.21.15.6:0/1289301262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4700162000 0x7f4700070c20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f47000b51d0 tx=0x7f46f802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 -- 172.21.15.6:0/1289301262 shutdown_connections 2024-03-06T07:29:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 --2- 172.21.15.6:0/1289301262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4700162000 0x7f4700070c20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 --2- 172.21.15.6:0/1289301262 >> v2:172.21.15.178:3300/0 conn(0x7f47000cd060 0x7f4700161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 -- 172.21.15.6:0/1289301262 >> 172.21.15.6:0/1289301262 conn(0x7f470006b3d0 msgr2=0x7f470006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 -- 172.21.15.6:0/1289301262 shutdown_connections 2024-03-06T07:29:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.886+0000 7f46ff200640 1 -- 172.21.15.6:0/1289301262 wait complete. 2024-03-06T07:29:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.888+0000 7f46ff200640 1 Processor -- start 2024-03-06T07:29:22.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.888+0000 7f46ff200640 1 -- start start 2024-03-06T07:29:22.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46ff200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 0x7f4700146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46ff200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f470015efc0 0x7f47001453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46ff200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4700162000 0x7f4700144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46ff200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4700162470 con 0x7f47000cd060 2024-03-06T07:29:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46ff200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470006ce20 con 0x7f4700162000 2024-03-06T07:29:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46ff200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470006d620 con 0x7f470015efc0 2024-03-06T07:29:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46fde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 0x7f4700146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46fde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 0x7f4700146ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50536/0 (socket says 172.21.15.6:50536) 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.889+0000 7f46fde00640 1 -- 172.21.15.6:0/4227884871 learned_addr learned my addr 172.21.15.6:0/4227884871 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fe800640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4700162000 0x7f4700144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fd400640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f470015efc0 0x7f47001453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fde00640 1 -- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f470015efc0 msgr2=0x7f47001453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fde00640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f470015efc0 0x7f47001453b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fde00640 1 -- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4700162000 msgr2=0x7f4700144930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fde00640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4700162000 0x7f4700144930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fde00640 1 -- 172.21.15.6:0/4227884871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700144e70 con 0x7f47000cd060 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46fde00640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 0x7f4700146ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b420 tx=0x7f46f000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0011830 con 0x7f47000cd060 2024-03-06T07:29:22.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f0016070 con 0x7f47000cd060 2024-03-06T07:29:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0011c10 con 0x7f47000cd060 2024-03-06T07:29:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.890+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4700145a20 con 0x7f47000cd060 2024-03-06T07:29:22.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.891+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4700145fe0 con 0x7f47000cd060 2024-03-06T07:29:22.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.892+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470004f330 con 0x7f47000cd060 2024-03-06T07:29:22.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.893+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46f001b460 con 0x7f47000cd060 2024-03-06T07:29:22.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.893+0000 7f46f6000640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f46e006b4d0 0x7f46e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.893+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f46f00899b0 con 0x7f47000cd060 2024-03-06T07:29:22.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.894+0000 7f46fd400640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f46e006b4d0 0x7f46e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.895+0000 7f46fd400640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f46e006b4d0 0x7f46e006d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f46ec006fd0 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:22.898+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f46f0020020 con 0x7f47000cd060 2024-03-06T07:29:23.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.096+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47000af2d0 con 0x7f47000cd060 2024-03-06T07:29:23.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.098+0000 7f46f6000640 1 -- 172.21.15.6:0/4227884871 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12819 (secure 0 0 0) 0x7f46f0058b70 con 0x7f47000cd060 2024-03-06T07:29:23.101 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:23.101 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":42,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:22.134726+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f46e006b4d0 msgr2=0x7f46e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f46e006b4d0 0x7f46e006d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f46ec006fd0 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-03-06T07:29:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 msgr2=0x7f4700146ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 0x7f4700146ac0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b420 tx=0x7f46f000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 shutdown_connections 2024-03-06T07:29:23.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f46e006b4d0 0x7f46e006d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4700162000 0x7f4700144930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f470015efc0 0x7f47001453b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 --2- 172.21.15.6:0/4227884871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47000cd060 0x7f4700146ac0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 >> 172.21.15.6:0/4227884871 conn(0x7f470006b3d0 msgr2=0x7f47000cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 shutdown_connections 2024-03-06T07:29:23.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.101+0000 7f46ff200640 1 -- 172.21.15.6:0/4227884871 wait complete. 2024-03-06T07:29:23.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:23.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:23.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.333+0000 7f8e43200640 1 Processor -- start 2024-03-06T07:29:23.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.333+0000 7f8e43200640 1 -- start start 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e43200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8e44145430 0x7f8e4415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e43200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 0x7f8e440b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e43200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e440b3fe0 con 0x7f8e440b2080 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e43200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4415fe40 con 0x7f8e44145430 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e43200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44160120 con 0x7f8e4415cc90 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e42800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8e440b2080 0x7f8e4406c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43370/0 (socket says 172.21.15.6:43370) 2024-03-06T07:29:23.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e42800640 1 -- 172.21.15.6:0/952241974 learned_addr learned my addr 172.21.15.6:0/952241974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e41400640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 0x7f8e440b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.335+0000 7f8e41e00640 1 --2- 172.21.15.6:0/952241974 >> v2:172.21.15.178:3300/0 conn(0x7f8e44145430 0x7f8e4415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912213558 0 0) 0x7f8e440b3fe0 con 0x7f8e440b2080 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e28003520 con 0x7f8e440b2080 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196150711 0 0) 0x7f8e28003520 con 0x7f8e440b2080 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 msgr2=0x7f8e440b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 0x7f8e440b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 >> v2:172.21.15.178:3300/0 conn(0x7f8e44145430 msgr2=0x7f8e4415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 --2- 172.21.15.6:0/952241974 >> v2:172.21.15.178:3300/0 conn(0x7f8e44145430 0x7f8e4415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44160400 con 0x7f8e440b2080 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8e3c0019f0 con 0x7f8e440b2080 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 >> v1:172.21.15.6:6789/0 conn(0x7f8e440b2080 legacy=0x7f8e4406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e280039b0 0x7f8e28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e28004800 0x7f8e28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 0x7f8e280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4415fe40 con 0x7f8e280039b0 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44160120 con 0x7f8e28004800 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2800b9c0 con 0x7f8e4415cc90 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.336+0000 7f8e41e00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e280039b0 0x7f8e28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e41400640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e28004800 0x7f8e28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e41e00640 1 -- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 msgr2=0x7f8e280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e42800640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 0x7f8e280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e41e00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e4415cc90 0x7f8e280051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e41e00640 1 -- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e28004800 msgr2=0x7f8e28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e41e00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e28004800 0x7f8e28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.337+0000 7f8e41e00640 1 -- 172.21.15.6:0/952241974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e28003520 con 0x7f8e280039b0 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.338+0000 7f8e41e00640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e280039b0 0x7f8e28003dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8e34009ac0 tx=0x7f8e28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debd25072ca2ffa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.338+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34039070 con 0x7f8e280039b0 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.338+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3402c6d0 con 0x7f8e280039b0 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.338+0000 7f8e40a00640 1 -- 172.21.15.6:0/952241974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34009690 con 0x7f8e280039b0 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e43200640 1 -- 172.21.15.6:0/952241974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e280039b0 msgr2=0x7f8e28003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e43200640 1 --2- 172.21.15.6:0/952241974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e280039b0 0x7f8e28003dd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8e34009ac0 tx=0x7f8e28005830 comp rx=0 tx=0).stop 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e42800640 1 -- 172.21.15.6:0/952241974 reap_dead start 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e43200640 1 -- 172.21.15.6:0/952241974 shutdown_connections 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e43200640 1 -- 172.21.15.6:0/952241974 >> 172.21.15.6:0/952241974 conn(0x7f8e4406b3a0 msgr2=0x7f8e4406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e43200640 1 -- 172.21.15.6:0/952241974 shutdown_connections 2024-03-06T07:29:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.339+0000 7f8e43200640 1 -- 172.21.15.6:0/952241974 wait complete. 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.341+0000 7f8e43200640 1 Processor -- start 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.341+0000 7f8e43200640 1 -- start start 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e43200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e280039b0 0x7f8e4415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e43200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 0x7f8e44141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e43200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e44141740 0x7f8e44071870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e43200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44145810 con 0x7f8e4415e7e0 2024-03-06T07:29:23.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e43200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4415d0d0 con 0x7f8e44141740 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e43200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e44145990 con 0x7f8e280039b0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 0x7f8e44141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 0x7f8e44141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50564/0 (socket says 172.21.15.6:50564) 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 -- 172.21.15.6:0/3900472694 learned_addr learned my addr 172.21.15.6:0/3900472694 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41e00640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e280039b0 0x7f8e4415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 -- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e280039b0 msgr2=0x7f8e4415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e280039b0 0x7f8e4415e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 -- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e44141740 msgr2=0x7f8e44071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e44141740 0x7f8e44071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.342+0000 7f8e41400640 1 -- 172.21.15.6:0/3900472694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44070b20 con 0x7f8e4415e7e0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.343+0000 7f8e41400640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 0x7f8e44141200 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8e30011630 tx=0x7f8e30011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.343+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000f920 con 0x7f8e4415e7e0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.343+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3000fac0 con 0x7f8e4415e7e0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.343+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000fe70 con 0x7f8e4415e7e0 2024-03-06T07:29:23.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.343+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e440ae750 con 0x7f8e4415e7e0 2024-03-06T07:29:23.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.343+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e44071070 con 0x7f8e4415e7e0 2024-03-06T07:29:23.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.344+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e44141b60 con 0x7f8e4415e7e0 2024-03-06T07:29:23.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.345+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e30015460 con 0x7f8e4415e7e0 2024-03-06T07:29:23.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.346+0000 7f8e3a000640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e2406b500 0x7f8e2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.346+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e3008a800 con 0x7f8e4415e7e0 2024-03-06T07:29:23.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.347+0000 7f8e41e00640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e2406b500 0x7f8e2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.348+0000 7f8e41e00640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e2406b500 0x7f8e2406d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e34036000 tx=0x7f8e34030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.350+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e44141b60 con 0x7f8e4415e7e0 2024-03-06T07:29:23.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.543+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e4404f330 con 0x7f8e4415e7e0 2024-03-06T07:29:23.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.544+0000 7f8e3a000640 1 -- 172.21.15.6:0/3900472694 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e30019030 con 0x7f8e4415e7e0 2024-03-06T07:29:23.546 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:29:23.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e2406b500 msgr2=0x7f8e2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e2406b500 0x7f8e2406d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e34036000 tx=0x7f8e34030000 comp rx=0 tx=0).stop 2024-03-06T07:29:23.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 msgr2=0x7f8e44141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 0x7f8e44141200 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8e30011630 tx=0x7f8e30011af0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 shutdown_connections 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e2406b500 0x7f8e2406d9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8e44141740 0x7f8e44071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8e4415e7e0 0x7f8e44141200 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 --2- 172.21.15.6:0/3900472694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8e280039b0 0x7f8e4415e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 >> 172.21.15.6:0/3900472694 conn(0x7f8e4406b3a0 msgr2=0x7f8e440b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 shutdown_connections 2024-03-06T07:29:23.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.547+0000 7f8e43200640 1 -- 172.21.15.6:0/3900472694 wait complete. 2024-03-06T07:29:23.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:29:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.780+0000 7f3e16a00640 1 Processor -- start 2024-03-06T07:29:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.780+0000 7f3e16a00640 1 -- start start 2024-03-06T07:29:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3e10145300 0x7f3e100b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e100cd060 0x7f3e1015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e100714d0 con 0x7f3e10070f10 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e100717b0 con 0x7f3e100cd060 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e10071a90 con 0x7f3e10145300 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e100cd060 0x7f3e1015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3e100cd060 0x7f3e1015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50580/0 (socket says 172.21.15.6:50580) 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.782+0000 7f3e16000640 1 -- 172.21.15.6:0/865844021 learned_addr learned my addr 172.21.15.6:0/865844021 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.783+0000 7f3e15600640 1 --2- 172.21.15.6:0/865844021 >> v2:172.21.15.178:3300/0 conn(0x7f3e10145300 0x7f3e100b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.783+0000 7f3e16000640 1 -- 172.21.15.6:0/865844021 >> v2:172.21.15.178:3300/0 conn(0x7f3e10145300 msgr2=0x7f3e100b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.783+0000 7f3e16000640 1 --2- 172.21.15.6:0/865844021 >> v2:172.21.15.178:3300/0 conn(0x7f3e10145300 0x7f3e100b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.783+0000 7f3e16000640 1 -- 172.21.15.6:0/865844021 >> v1:172.21.15.178:6789/0 conn(0x7f3e10070f10 legacy=0x7f3e100cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.783+0000 7f3e16000640 1 -- 172.21.15.6:0/865844021 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10071d70 con 0x7f3e100cd060 2024-03-06T07:29:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.784+0000 7f3e16000640 1 --2- 172.21.15.6:0/865844021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e1015c570 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3e100cf990 tx=0x7f3e0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7da6343e5ef476f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.784+0000 7f3e0fe00640 1 -- 172.21.15.6:0/865844021 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08035020 con 0x7f3e100cd060 2024-03-06T07:29:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.784+0000 7f3e0fe00640 1 -- 172.21.15.6:0/865844021 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e08031870 con 0x7f3e100cd060 2024-03-06T07:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.784+0000 7f3e0fe00640 1 -- 172.21.15.6:0/865844021 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08031c40 con 0x7f3e100cd060 2024-03-06T07:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 -- 172.21.15.6:0/865844021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 msgr2=0x7f3e1015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 --2- 172.21.15.6:0/865844021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e1015c570 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3e100cf990 tx=0x7f3e0802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 -- 172.21.15.6:0/865844021 shutdown_connections 2024-03-06T07:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 --2- 172.21.15.6:0/865844021 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e1015c570 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 --2- 172.21.15.6:0/865844021 >> v2:172.21.15.178:3300/0 conn(0x7f3e10145300 0x7f3e100b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 -- 172.21.15.6:0/865844021 >> 172.21.15.6:0/865844021 conn(0x7f3e1006b3d0 msgr2=0x7f3e1006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 -- 172.21.15.6:0/865844021 shutdown_connections 2024-03-06T07:29:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.785+0000 7f3e16a00640 1 -- 172.21.15.6:0/865844021 wait complete. 2024-03-06T07:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.787+0000 7f3e16a00640 1 Processor -- start 2024-03-06T07:29:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.787+0000 7f3e16a00640 1 -- start start 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e10070f10 0x7f3e101f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e101f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e16a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e10145300 0x7f3e1013f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e16a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1015cab0 con 0x7f3e100cd060 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e16a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e101456e0 con 0x7f3e10145300 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e16a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1015cc30 con 0x7f3e10070f10 2024-03-06T07:29:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e14c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e101f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e14c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e101f4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50590/0 (socket says 172.21.15.6:50590) 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.788+0000 7f3e14c00640 1 -- 172.21.15.6:0/1937663536 learned_addr learned my addr 172.21.15.6:0/1937663536 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e16000640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e10145300 0x7f3e1013f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e15600640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e10070f10 0x7f3e101f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e14c00640 1 -- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e10070f10 msgr2=0x7f3e101f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e14c00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e10070f10 0x7f3e101f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e14c00640 1 -- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e10145300 msgr2=0x7f3e1013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e14c00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e10145300 0x7f3e1013f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e14c00640 1 -- 172.21.15.6:0/1937663536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1015b310 con 0x7f3e100cd060 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e16000640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e10145300 0x7f3e1013f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e14c00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e101f4920 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00c400 tx=0x7f3dfc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.789+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0052d0 con 0x7f3e100cd060 2024-03-06T07:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.790+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc005490 con 0x7f3e100cd060 2024-03-06T07:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.790+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc007260 con 0x7f3e100cd060 2024-03-06T07:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.790+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1013fcf0 con 0x7f3e100cd060 2024-03-06T07:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.790+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e101402b0 con 0x7f3e100cd060 2024-03-06T07:29:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.791+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1004f330 con 0x7f3e100cd060 2024-03-06T07:29:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.792+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3dfc007400 con 0x7f3e100cd060 2024-03-06T07:29:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.793+0000 7f3e0d600640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3df006b4d0 0x7f3df006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.793+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3dfc089410 con 0x7f3e100cd060 2024-03-06T07:29:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.793+0000 7f3e15600640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3df006b4d0 0x7f3df006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.794+0000 7f3e15600640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3df006b4d0 0x7f3df006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e00007380 tx=0x7f3e0000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:23.797+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3dfc0585d0 con 0x7f3e100cd060 2024-03-06T07:29:24.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.000+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3e1015c570 con 0x7f3e100cd060 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.001+0000 7f3e0d600640 1 -- 172.21.15.6:0/1937663536 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3dfc057ed0 con 0x7f3e100cd060 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:29:24.003 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:29:24.004 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3df006b4d0 msgr2=0x7f3df006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3df006b4d0 0x7f3df006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e00007380 tx=0x7f3e0000b040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 msgr2=0x7f3e101f4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e101f4920 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc00c400 tx=0x7f3dfc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 shutdown_connections 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3df006b4d0 0x7f3df006d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3e10145300 0x7f3e1013f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e100cd060 0x7f3e101f4920 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 --2- 172.21.15.6:0/1937663536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3e10070f10 0x7f3e101f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 >> 172.21.15.6:0/1937663536 conn(0x7f3e1006b3d0 msgr2=0x7f3e1015d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.003+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 shutdown_connections 2024-03-06T07:29:24.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.004+0000 7f3e16a00640 1 -- 172.21.15.6:0/1937663536 wait complete. 2024-03-06T07:29:24.023 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:29:24.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:29:24.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.242+0000 7f0bcaa00640 1 Processor -- start 2024-03-06T07:29:24.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.243+0000 7f0bcaa00640 1 -- start start 2024-03-06T07:29:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.245+0000 7f0bcaa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc4144320 0x7f0bc406daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.245+0000 7f0bcaa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0bc406e030 0x7f0bc40720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.245+0000 7f0bcaa00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc40ae890 con 0x7f0bc40725f0 2024-03-06T07:29:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.245+0000 7f0bcaa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc406e470 con 0x7f0bc406e030 2024-03-06T07:29:24.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.245+0000 7f0bcaa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b3ea0 con 0x7f0bc4144320 2024-03-06T07:29:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0bc406e030 0x7f0bc40720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0bc406e030 0x7f0bc40720b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:41252/0 (socket says 172.21.15.6:41252) 2024-03-06T07:29:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 -- 172.21.15.6:0/112728831 learned_addr learned my addr 172.21.15.6:0/112728831 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc9600640 1 --2- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc4144320 0x7f0bc406daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 -- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc4144320 msgr2=0x7f0bc406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 --2- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc4144320 0x7f0bc406daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 -- 172.21.15.6:0/112728831 >> v1:172.21.15.178:6789/0 conn(0x7f0bc40725f0 legacy=0x7f0bc4070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.246+0000 7f0bc8c00640 1 -- 172.21.15.6:0/112728831 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc40b4180 con 0x7f0bc406e030 2024-03-06T07:29:24.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.248+0000 7f0bc8c00640 1 --2- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc406e030 0x7f0bc40720b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0bc406e5f0 tx=0x7f0bb002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4182738030d12ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.248+0000 7f0bc3e00640 1 -- 172.21.15.6:0/112728831 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0034070 con 0x7f0bc406e030 2024-03-06T07:29:24.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.248+0000 7f0bc3e00640 1 -- 172.21.15.6:0/112728831 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb0030840 con 0x7f0bc406e030 2024-03-06T07:29:24.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.248+0000 7f0bc3e00640 1 -- 172.21.15.6:0/112728831 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb0030c10 con 0x7f0bc406e030 2024-03-06T07:29:24.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 -- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc406e030 msgr2=0x7f0bc40720b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc406e030 0x7f0bc40720b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0bc406e5f0 tx=0x7f0bb002b980 comp rx=0 tx=0).stop 2024-03-06T07:29:24.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 -- 172.21.15.6:0/112728831 shutdown_connections 2024-03-06T07:29:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc406e030 0x7f0bc40720b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/112728831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc4144320 0x7f0bc406daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 -- 172.21.15.6:0/112728831 >> 172.21.15.6:0/112728831 conn(0x7f0bc40b17c0 msgr2=0x7f0bc40b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 -- 172.21.15.6:0/112728831 shutdown_connections 2024-03-06T07:29:24.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.249+0000 7f0bcaa00640 1 -- 172.21.15.6:0/112728831 wait complete. 2024-03-06T07:29:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.252+0000 7f0bcaa00640 1 Processor -- start 2024-03-06T07:29:24.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.252+0000 7f0bcaa00640 1 -- start start 2024-03-06T07:29:24.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bcaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 0x7f0bc41435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bcaa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc40725f0 0x7f0bc415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bcaa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc4144320 0x7f0bc41470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bcaa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b4a20 con 0x7f0bc406e030 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bcaa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b4740 con 0x7f0bc4144320 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bcaa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc40b4f40 con 0x7f0bc40725f0 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.253+0000 7f0bc9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 0x7f0bc41435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 0x7f0bc41435d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50606/0 (socket says 172.21.15.6:50606) 2024-03-06T07:29:24.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 -- 172.21.15.6:0/4059598456 learned_addr learned my addr 172.21.15.6:0/4059598456 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:24.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bca000640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc4144320 0x7f0bc41470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc8c00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc40725f0 0x7f0bc415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 -- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc40725f0 msgr2=0x7f0bc415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc40725f0 0x7f0bc415d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 -- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc4144320 msgr2=0x7f0bc41470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc4144320 0x7f0bc41470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 -- 172.21.15.6:0/4059598456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc41607a0 con 0x7f0bc406e030 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.254+0000 7f0bc9600640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 0x7f0bc41435d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800e8a0 tx=0x7f0bb800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.255+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8017070 con 0x7f0bc406e030 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.255+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb800c940 con 0x7f0bc406e030 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.255+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb800cd30 con 0x7f0bc406e030 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.255+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc4160a80 con 0x7f0bc406e030 2024-03-06T07:29:24.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.255+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc4161040 con 0x7f0bc406e030 2024-03-06T07:29:24.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.256+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc404f330 con 0x7f0bc406e030 2024-03-06T07:29:24.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.257+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bb801a070 con 0x7f0bc406e030 2024-03-06T07:29:24.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.259+0000 7f0bc1600640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b500 0x7f0ba406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:24.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.259+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0bb808a0e0 con 0x7f0bc406e030 2024-03-06T07:29:24.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.260+0000 7f0bc8c00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b500 0x7f0ba406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:24.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.261+0000 7f0bc8c00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b500 0x7f0ba406d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0036000 tx=0x7f0bb003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:24.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.266+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bb80592b0 con 0x7f0bc406e030 2024-03-06T07:29:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:24.469+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0bc40b2380 con 0x7f0bc406e030 2024-03-06T07:29:25.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.150+0000 7f0bc1600640 1 -- 172.21.15.6:0/4059598456 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0bb8058bb0 con 0x7f0bc406e030 2024-03-06T07:29:25.152 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b500 msgr2=0x7f0ba406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b500 0x7f0ba406d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0bb0036000 tx=0x7f0bb003b040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 msgr2=0x7f0bc41435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 0x7f0bc41435d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800e8a0 tx=0x7f0bb800ed60 comp rx=0 tx=0).stop 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 shutdown_connections 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ba406b500 0x7f0ba406d9c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bc4144320 0x7f0bc41470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bc40725f0 0x7f0bc415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 --2- 172.21.15.6:0/4059598456 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bc406e030 0x7f0bc41435d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.154+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 >> 172.21.15.6:0/4059598456 conn(0x7f0bc40b17c0 msgr2=0x7f0bc4162020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.155+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 shutdown_connections 2024-03-06T07:29:25.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.156+0000 7f0bcaa00640 1 -- 172.21.15.6:0/4059598456 wait complete. 2024-03-06T07:29:25.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:29:25.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.390+0000 7f01c2e00640 1 Processor -- start 2024-03-06T07:29:25.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.391+0000 7f01c2e00640 1 -- start start 2024-03-06T07:29:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.392+0000 7f01c2e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f01bc0d26f0 0x7f01bc0d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.392+0000 7f01c2e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01bc0d1920 0x7f01bc0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.392+0000 7f01c2e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01bc0cf4a0 con 0x7f01bc15e220 2024-03-06T07:29:25.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.392+0000 7f01c2e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc0b36b0 con 0x7f01bc0d1920 2024-03-06T07:29:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.392+0000 7f01c2e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01bc0cf780 con 0x7f01bc0d26f0 2024-03-06T07:29:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01bc0d1920 0x7f01bc0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c1a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f01bc15e220 0x7f01bc0d1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43382/0 (socket says 172.21.15.6:43382) 2024-03-06T07:29:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c1a00640 1 -- 172.21.15.6:0/432057815 learned_addr learned my addr 172.21.15.6:0/432057815 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:25.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c1000640 1 --2- 172.21.15.6:0/432057815 >> v2:172.21.15.178:3300/0 conn(0x7f01bc0d26f0 0x7f01bc0d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c2400640 1 -- 172.21.15.6:0/432057815 >> v2:172.21.15.178:3300/0 conn(0x7f01bc0d26f0 msgr2=0x7f01bc0d4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c2400640 1 --2- 172.21.15.6:0/432057815 >> v2:172.21.15.178:3300/0 conn(0x7f01bc0d26f0 0x7f01bc0d4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c2400640 1 -- 172.21.15.6:0/432057815 >> v1:172.21.15.6:6789/0 conn(0x7f01bc15e220 legacy=0x7f01bc0d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01c2400640 1 -- 172.21.15.6:0/432057815 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc0cfa60 con 0x7f01bc0d1920 2024-03-06T07:29:25.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.393+0000 7f01bbe00640 1 -- 172.21.15.6:0/432057815 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423927291 0 0) 0x7f01bc0cf4a0 con 0x7f01bc15e220 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.395+0000 7f01c2400640 1 --2- 172.21.15.6:0/432057815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc0d1920 0x7f01bc0b3e20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f01bc08d0d0 tx=0x7f01b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55849a2d933b37f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.395+0000 7f01bbe00640 1 -- 172.21.15.6:0/432057815 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4039070 con 0x7f01bc0d1920 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.395+0000 7f01bbe00640 1 -- 172.21.15.6:0/432057815 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b402cbf0 con 0x7f01bc0d1920 2024-03-06T07:29:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.395+0000 7f01bbe00640 1 -- 172.21.15.6:0/432057815 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4034440 con 0x7f01bc0d1920 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 -- 172.21.15.6:0/432057815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc0d1920 msgr2=0x7f01bc0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 --2- 172.21.15.6:0/432057815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc0d1920 0x7f01bc0b3e20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f01bc08d0d0 tx=0x7f01b402be40 comp rx=0 tx=0).stop 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 -- 172.21.15.6:0/432057815 shutdown_connections 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 --2- 172.21.15.6:0/432057815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc0d1920 0x7f01bc0b3e20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 --2- 172.21.15.6:0/432057815 >> v2:172.21.15.178:3300/0 conn(0x7f01bc0d26f0 0x7f01bc0d4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 -- 172.21.15.6:0/432057815 >> 172.21.15.6:0/432057815 conn(0x7f01bc0d3d50 msgr2=0x7f01bc0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 -- 172.21.15.6:0/432057815 shutdown_connections 2024-03-06T07:29:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.396+0000 7f01c2e00640 1 -- 172.21.15.6:0/432057815 wait complete. 2024-03-06T07:29:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.398+0000 7f01c2e00640 1 Processor -- start 2024-03-06T07:29:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.398+0000 7f01c2e00640 1 -- start start 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01bc0d1920 0x7f01bc0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01bc0d26f0 0x7f01bc1f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 0x7f01bc15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0b4360 con 0x7f01bc15e220 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0d1d50 con 0x7f01bc0d26f0 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01bc0b44e0 con 0x7f01bc0d1920 2024-03-06T07:29:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 0x7f01bc15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 0x7f01bc15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50620/0 (socket says 172.21.15.6:50620) 2024-03-06T07:29:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.399+0000 7f01c2400640 1 -- 172.21.15.6:0/3767441671 learned_addr learned my addr 172.21.15.6:0/3767441671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c1000640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01bc0d26f0 0x7f01bc1f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c1a00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01bc0d1920 0x7f01bc0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2400640 1 -- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01bc0d1920 msgr2=0x7f01bc0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2400640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01bc0d1920 0x7f01bc0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2400640 1 -- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01bc0d26f0 msgr2=0x7f01bc1f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2400640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01bc0d26f0 0x7f01bc1f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2400640 1 -- 172.21.15.6:0/3767441671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc1f4a00 con 0x7f01bc15e220 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c1a00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01bc0d1920 0x7f01bc0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2400640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 0x7f01bc15cc00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009e80 tx=0x7f01b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4039050 con 0x7f01bc15e220 2024-03-06T07:29:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b4045070 con 0x7f01bc15e220 2024-03-06T07:29:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.400+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc0cc070 con 0x7f01bc15e220 2024-03-06T07:29:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.401+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4009450 con 0x7f01bc15e220 2024-03-06T07:29:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.401+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01bc0cc630 con 0x7f01bc15e220 2024-03-06T07:29:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.402+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01bc04f330 con 0x7f01bc15e220 2024-03-06T07:29:25.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.403+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01b403e850 con 0x7f01bc15e220 2024-03-06T07:29:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.404+0000 7f01b9600640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f019c06b520 0x7f019c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.404+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f01b40ad810 con 0x7f01bc15e220 2024-03-06T07:29:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.405+0000 7f01c1a00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f019c06b520 0x7f019c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.406+0000 7f01c1a00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f019c06b520 0x7f019c06d9e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f01b0000f60 tx=0x7f01b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.408+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01bc1f4a00 con 0x7f01bc15e220 2024-03-06T07:29:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:25.603+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f01bc0d3b30 con 0x7f01bc15e220 2024-03-06T07:29:26.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.156+0000 7f01b9600640 1 -- 172.21.15.6:0/3767441671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f01b407c970 con 0x7f01bc15e220 2024-03-06T07:29:26.164 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:29:26.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.160+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f019c06b520 msgr2=0x7f019c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.160+0000 7f01c2e00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f019c06b520 0x7f019c06d9e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f01b0000f60 tx=0x7f01b0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:26.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.160+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 msgr2=0x7f01bc15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.160+0000 7f01c2e00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 0x7f01bc15cc00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009e80 tx=0x7f01b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.161+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 shutdown_connections 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.161+0000 7f01c2e00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f019c06b520 0x7f019c06d9e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.161+0000 7f01c2e00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01bc0d26f0 0x7f01bc1f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.161+0000 7f01c2e00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01bc15e220 0x7f01bc15cc00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.161+0000 7f01c2e00640 1 --2- 172.21.15.6:0/3767441671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01bc0d1920 0x7f01bc0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.161+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 >> 172.21.15.6:0/3767441671 conn(0x7f01bc0d3d50 msgr2=0x7f01bc0b4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.164+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 shutdown_connections 2024-03-06T07:29:26.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.164+0000 7f01c2e00640 1 -- 172.21.15.6:0/3767441671 wait complete. 2024-03-06T07:29:26.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:29:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.380+0000 7f030ae00640 1 Processor -- start 2024-03-06T07:29:26.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.380+0000 7f030ae00640 1 -- start start 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.381+0000 7f030ae00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0304072520 0x7f03040cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.381+0000 7f030ae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0304071e20 0x7f03040ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.381+0000 7f030ae00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0304072df0 con 0x7f030415bf20 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.381+0000 7f030ae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03040cdab0 con 0x7f0304071e20 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.381+0000 7f030ae00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03040b4d70 con 0x7f0304072520 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0304071e20 0x7f03040ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0304071e20 0x7f03040ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50622/0 (socket says 172.21.15.6:50622) 2024-03-06T07:29:26.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 -- 172.21.15.6:0/1380790888 learned_addr learned my addr 172.21.15.6:0/1380790888 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f02fbe00640 1 --2- 172.21.15.6:0/1380790888 >> v2:172.21.15.178:3300/0 conn(0x7f0304072520 0x7f03040cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 -- 172.21.15.6:0/1380790888 >> v2:172.21.15.178:3300/0 conn(0x7f0304072520 msgr2=0x7f03040cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 --2- 172.21.15.6:0/1380790888 >> v2:172.21.15.178:3300/0 conn(0x7f0304072520 0x7f03040cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 -- 172.21.15.6:0/1380790888 >> v1:172.21.15.178:6789/0 conn(0x7f030415bf20 legacy=0x7f03040acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f0308e00640 1 -- 172.21.15.6:0/1380790888 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040b5050 con 0x7f0304071e20 2024-03-06T07:29:26.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.382+0000 7f02fbe00640 1 --2- 172.21.15.6:0/1380790888 >> v2:172.21.15.178:3300/0 conn(0x7f0304072520 0x7f03040cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.383+0000 7f0308e00640 1 --2- 172.21.15.6:0/1380790888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f03040ce4f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0304070b90 tx=0x7f02fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1989e5b1f266d74d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.384+0000 7f0303400640 1 -- 172.21.15.6:0/1380790888 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc02ca80 con 0x7f0304071e20 2024-03-06T07:29:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.384+0000 7f0303400640 1 -- 172.21.15.6:0/1380790888 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc02cc20 con 0x7f0304071e20 2024-03-06T07:29:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.384+0000 7f0303400640 1 -- 172.21.15.6:0/1380790888 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0343f0 con 0x7f0304071e20 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.384+0000 7f030ae00640 1 -- 172.21.15.6:0/1380790888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 msgr2=0x7f03040ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.384+0000 7f030ae00640 1 --2- 172.21.15.6:0/1380790888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f03040ce4f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0304070b90 tx=0x7f02fc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.385+0000 7f030ae00640 1 -- 172.21.15.6:0/1380790888 shutdown_connections 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.385+0000 7f030ae00640 1 --2- 172.21.15.6:0/1380790888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f03040ce4f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.385+0000 7f030ae00640 1 --2- 172.21.15.6:0/1380790888 >> v2:172.21.15.178:3300/0 conn(0x7f0304072520 0x7f03040cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.385+0000 7f030ae00640 1 -- 172.21.15.6:0/1380790888 >> 172.21.15.6:0/1380790888 conn(0x7f030406b3d0 msgr2=0x7f030406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.385+0000 7f030ae00640 1 -- 172.21.15.6:0/1380790888 shutdown_connections 2024-03-06T07:29:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.385+0000 7f030ae00640 1 -- 172.21.15.6:0/1380790888 wait complete. 2024-03-06T07:29:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.386+0000 7f030ae00640 1 Processor -- start 2024-03-06T07:29:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.386+0000 7f030ae00640 1 -- start start 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.387+0000 7f030ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f030414b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.387+0000 7f030ae00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0304072520 0x7f030413db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.387+0000 7f030ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f030415bf20 0x7f0304070e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.387+0000 7f030ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040ad450 con 0x7f0304071e20 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.387+0000 7f030ae00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b4990 con 0x7f0304072520 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.387+0000 7f030ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f030409b100 con 0x7f030415bf20 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f030414b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f030414b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50628/0 (socket says 172.21.15.6:50628) 2024-03-06T07:29:26.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 -- 172.21.15.6:0/499787165 learned_addr learned my addr 172.21.15.6:0/499787165 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 -- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f030415bf20 msgr2=0x7f0304070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f030415bf20 0x7f0304070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 -- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0304072520 msgr2=0x7f030413db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0304072520 0x7f030413db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 -- 172.21.15.6:0/499787165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030414bf80 con 0x7f0304071e20 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.388+0000 7f0303e00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f030414b880 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b3f0 tx=0x7f02f400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.389+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4015020 con 0x7f0304071e20 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.389+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f4011830 con 0x7f0304071e20 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.389+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4011c20 con 0x7f0304071e20 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.389+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030414c260 con 0x7f0304071e20 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.389+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f030409ad90 con 0x7f0304071e20 2024-03-06T07:29:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.390+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030404f330 con 0x7f0304071e20 2024-03-06T07:29:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.391+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02f4018020 con 0x7f0304071e20 2024-03-06T07:29:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.392+0000 7f0300c00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02e006b500 0x7f02e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.392+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f02f4089e10 con 0x7f0304071e20 2024-03-06T07:29:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.393+0000 7f02fbe00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02e006b500 0x7f02e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.394+0000 7f02fbe00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02e006b500 0x7f02e006d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0304071a60 tx=0x7f02ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.397+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02f408d030 con 0x7f0304071e20 2024-03-06T07:29:26.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:26.632+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f030415f720 con 0x7f0304071e20 2024-03-06T07:29:27.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.211+0000 7f0300c00640 1 -- 172.21.15.6:0/499787165 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7f02f401f030 con 0x7f0304071e20 2024-03-06T07:29:27.216 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:29:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.217+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02e006b500 msgr2=0x7f02e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.217+0000 7f030ae00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02e006b500 0x7f02e006d9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0304071a60 tx=0x7f02ec008040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.217+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 msgr2=0x7f030414b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.217+0000 7f030ae00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f030414b880 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f02f400b3f0 tx=0x7f02f400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.221+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 shutdown_connections 2024-03-06T07:29:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.221+0000 7f030ae00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02e006b500 0x7f02e006d9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.221+0000 7f030ae00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0304072520 0x7f030413db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.221+0000 7f030ae00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f030415bf20 0x7f0304070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.221+0000 7f030ae00640 1 --2- 172.21.15.6:0/499787165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0304071e20 0x7f030414b880 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.221+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 >> 172.21.15.6:0/499787165 conn(0x7f030406b3d0 msgr2=0x7f03040ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.224+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 shutdown_connections 2024-03-06T07:29:27.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.224+0000 7f030ae00640 1 -- 172.21.15.6:0/499787165 wait complete. 2024-03-06T07:29:27.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.430+0000 7f78bbe00640 1 Processor -- start 2024-03-06T07:29:27.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.430+0000 7f78bbe00640 1 -- start start 2024-03-06T07:29:27.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.431+0000 7f78bbe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f78b40af580 0x7f78b415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.431+0000 7f78bbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78b40cdc20 0x7f78b40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.431+0000 7f78bbe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b415cc40 con 0x7f78b40af1a0 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.431+0000 7f78bbe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b415cf20 con 0x7f78b40cdc20 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.431+0000 7f78bbe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b415d200 con 0x7f78b40af580 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78b40cdc20 0x7f78b40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f78b40cdc20 0x7f78b40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50644/0 (socket says 172.21.15.6:50644) 2024-03-06T07:29:27.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 -- 172.21.15.6:0/2104501562 learned_addr learned my addr 172.21.15.6:0/2104501562 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 -- 172.21.15.6:0/2104501562 >> v2:172.21.15.178:3300/0 conn(0x7f78b40af580 msgr2=0x7f78b415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 --2- 172.21.15.6:0/2104501562 >> v2:172.21.15.178:3300/0 conn(0x7f78b40af580 0x7f78b415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 -- 172.21.15.6:0/2104501562 >> v1:172.21.15.178:6789/0 conn(0x7f78b40af1a0 legacy=0x7f78b40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.432+0000 7f78b9e00640 1 -- 172.21.15.6:0/2104501562 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b415d4e0 con 0x7f78b40cdc20 2024-03-06T07:29:27.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.433+0000 7f78b9e00640 1 --2- 172.21.15.6:0/2104501562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40cdc20 0x7f78b40b2dd0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f78b40b54c0 tx=0x7f78ac02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddfdb83a00a062f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.433+0000 7f78b3e00640 1 -- 172.21.15.6:0/2104501562 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac02c9f0 con 0x7f78b40cdc20 2024-03-06T07:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.433+0000 7f78b3e00640 1 -- 172.21.15.6:0/2104501562 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac02cb90 con 0x7f78b40cdc20 2024-03-06T07:29:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.433+0000 7f78b3e00640 1 -- 172.21.15.6:0/2104501562 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac0343f0 con 0x7f78b40cdc20 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 -- 172.21.15.6:0/2104501562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40cdc20 msgr2=0x7f78b40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 --2- 172.21.15.6:0/2104501562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40cdc20 0x7f78b40b2dd0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f78b40b54c0 tx=0x7f78ac02be50 comp rx=0 tx=0).stop 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 -- 172.21.15.6:0/2104501562 shutdown_connections 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 --2- 172.21.15.6:0/2104501562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40cdc20 0x7f78b40b2dd0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 --2- 172.21.15.6:0/2104501562 >> v2:172.21.15.178:3300/0 conn(0x7f78b40af580 0x7f78b415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 -- 172.21.15.6:0/2104501562 >> 172.21.15.6:0/2104501562 conn(0x7f78b406afb0 msgr2=0x7f78b406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 -- 172.21.15.6:0/2104501562 shutdown_connections 2024-03-06T07:29:27.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.434+0000 7f78bbe00640 1 -- 172.21.15.6:0/2104501562 wait complete. 2024-03-06T07:29:27.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.436+0000 7f78bbe00640 1 Processor -- start 2024-03-06T07:29:27.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.436+0000 7f78bbe00640 1 -- start start 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78bbe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78b40af1a0 0x7f78b40ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78bbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 0x7f78b41f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78bbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f78b40cdc20 0x7f78b41f55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78bbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40b36a0 con 0x7f78b40af580 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78bbe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40b3520 con 0x7f78b40af1a0 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78bbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b40b3820 con 0x7f78b40cdc20 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78b8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 0x7f78b41f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78b8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 0x7f78b41f50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50660/0 (socket says 172.21.15.6:50660) 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78b8a00640 1 -- 172.21.15.6:0/3228316801 learned_addr learned my addr 172.21.15.6:0/3228316801 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.437+0000 7f78b9400640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78b40af1a0 0x7f78b40ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b9e00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f78b40cdc20 0x7f78b41f55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b8a00640 1 -- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f78b40cdc20 msgr2=0x7f78b41f55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b8a00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f78b40cdc20 0x7f78b41f55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b8a00640 1 -- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78b40af1a0 msgr2=0x7f78b40ad570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b8a00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78b40af1a0 0x7f78b40ad570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b8a00640 1 -- 172.21.15.6:0/3228316801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4157410 con 0x7f78b40af580 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b9e00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f78b40cdc20 0x7f78b41f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b8a00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 0x7f78b41f50b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f78a000b450 tx=0x7f78a000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0015020 con 0x7f78b40af580 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0011890 con 0x7f78b40af580 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a0011c80 con 0x7f78b40af580 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b41576f0 con 0x7f78b40af580 2024-03-06T07:29:27.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.438+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b4154790 con 0x7f78b40af580 2024-03-06T07:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.440+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b404f3a0 con 0x7f78b40af580 2024-03-06T07:29:27.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.440+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78a0018020 con 0x7f78b40af580 2024-03-06T07:29:27.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.442+0000 7f78b1600640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f789406b520 0x7f789406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.442+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a008a410 con 0x7f78b40af580 2024-03-06T07:29:27.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.442+0000 7f78b9400640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f789406b520 0x7f789406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.444+0000 7f78b9400640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f789406b520 0x7f789406d9e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f78a8007fd0 tx=0x7f78a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.447+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78a001f020 con 0x7f78b40af580 2024-03-06T07:29:27.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.651+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78b40cd340 con 0x7f78b40af580 2024-03-06T07:29:27.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.653+0000 7f78b1600640 1 -- 172.21.15.6:0/3228316801 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16554 (secure 0 0 0) 0x7f78a000d040 con 0x7f78b40af580 2024-03-06T07:29:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:27.655 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:26.634750+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.655+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f789406b520 msgr2=0x7f789406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.655+0000 7f78bbe00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f789406b520 0x7f789406d9e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f78a8007fd0 tx=0x7f78a8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.655+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 msgr2=0x7f78b41f50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.655+0000 7f78bbe00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 0x7f78b41f50b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f78a000b450 tx=0x7f78a000b910 comp rx=0 tx=0).stop 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 shutdown_connections 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f789406b520 0x7f789406d9e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f78b40cdc20 0x7f78b41f55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f78b40af580 0x7f78b41f50b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 --2- 172.21.15.6:0/3228316801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78b40af1a0 0x7f78b40ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 >> 172.21.15.6:0/3228316801 conn(0x7f78b406afb0 msgr2=0x7f78b415bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 shutdown_connections 2024-03-06T07:29:27.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.656+0000 7f78bbe00640 1 -- 172.21.15.6:0/3228316801 wait complete. 2024-03-06T07:29:27.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:27.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.888+0000 7fbfd7000640 1 Processor -- start 2024-03-06T07:29:27.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.888+0000 7fbfd7000640 1 -- start start 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd7000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbfd0070f10 0x7fbfd00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd7000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfd00cce90 0x7fbfd015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd7000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd00714d0 con 0x7fbfd0145300 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd7000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd00cd300 con 0x7fbfd00cce90 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd7000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd00717b0 con 0x7fbfd0070f10 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd6600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfd00cce90 0x7fbfd015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd6600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfd00cce90 0x7fbfd015c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50672/0 (socket says 172.21.15.6:50672) 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd6600640 1 -- 172.21.15.6:0/3801413331 learned_addr learned my addr 172.21.15.6:0/3801413331 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.890+0000 7fbfd5200640 1 --2- 172.21.15.6:0/3801413331 >> v2:172.21.15.178:3300/0 conn(0x7fbfd0070f10 0x7fbfd00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.891+0000 7fbfd6600640 1 -- 172.21.15.6:0/3801413331 >> v2:172.21.15.178:3300/0 conn(0x7fbfd0070f10 msgr2=0x7fbfd00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.891+0000 7fbfd6600640 1 --2- 172.21.15.6:0/3801413331 >> v2:172.21.15.178:3300/0 conn(0x7fbfd0070f10 0x7fbfd00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.891+0000 7fbfd6600640 1 -- 172.21.15.6:0/3801413331 >> v1:172.21.15.178:6789/0 conn(0x7fbfd0145300 legacy=0x7fbfd00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.891+0000 7fbfd6600640 1 -- 172.21.15.6:0/3801413331 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd0071a90 con 0x7fbfd00cce90 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.892+0000 7fbfd6600640 1 --2- 172.21.15.6:0/3801413331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015c570 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00b5150 tx=0x7fbfc802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c823d9f832f1f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.892+0000 7fbfcfe00640 1 -- 172.21.15.6:0/3801413331 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc802ca80 con 0x7fbfd00cce90 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.892+0000 7fbfcfe00640 1 -- 172.21.15.6:0/3801413331 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc802cc20 con 0x7fbfd00cce90 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.892+0000 7fbfcfe00640 1 -- 172.21.15.6:0/3801413331 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80343f0 con 0x7fbfd00cce90 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 -- 172.21.15.6:0/3801413331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 msgr2=0x7fbfd015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3801413331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015c570 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00b5150 tx=0x7fbfc802bee0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 -- 172.21.15.6:0/3801413331 shutdown_connections 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3801413331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015c570 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3801413331 >> v2:172.21.15.178:3300/0 conn(0x7fbfd0070f10 0x7fbfd00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 -- 172.21.15.6:0/3801413331 >> 172.21.15.6:0/3801413331 conn(0x7fbfd006b3d0 msgr2=0x7fbfd006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 -- 172.21.15.6:0/3801413331 shutdown_connections 2024-03-06T07:29:27.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.893+0000 7fbfd7000640 1 -- 172.21.15.6:0/3801413331 wait complete. 2024-03-06T07:29:27.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.894+0000 7fbfd7000640 1 Processor -- start 2024-03-06T07:29:27.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 -- start start 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfd0070f10 0x7fbfd0137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfd0145300 0x7fbfd01541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd015cab0 con 0x7fbfd00cce90 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd01456e0 con 0x7fbfd0145300 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.895+0000 7fbfd7000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd015cc30 con 0x7fbfd0070f10 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50688/0 (socket says 172.21.15.6:50688) 2024-03-06T07:29:27.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 -- 172.21.15.6:0/3221082724 learned_addr learned my addr 172.21.15.6:0/3221082724 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:27.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd6600640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfd0145300 0x7fbfd01541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5c00640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfd0070f10 0x7fbfd0137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 -- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfd0070f10 msgr2=0x7fbfd0137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfd0070f10 0x7fbfd0137390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 -- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfd0145300 msgr2=0x7fbfd01541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfd0145300 0x7fbfd01541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd5200640 1 -- 172.21.15.6:0/3221082724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd015df50 con 0x7fbfd00cce90 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.896+0000 7fbfd6600640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfd0145300 0x7fbfd01541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.897+0000 7fbfd5200640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015d720 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00d800 tx=0x7fbfbc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.897+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc00b6c0 con 0x7fbfd00cce90 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.897+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc019070 con 0x7fbfd00cce90 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.897+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc00ba80 con 0x7fbfd00cce90 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.897+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd013c150 con 0x7fbfd00cce90 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.897+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd013c710 con 0x7fbfd00cce90 2024-03-06T07:29:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.898+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd004f330 con 0x7fbfd00cce90 2024-03-06T07:29:27.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.899+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfbc00bc20 con 0x7fbfd00cce90 2024-03-06T07:29:27.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.900+0000 7fbfcd600640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfb006b4d0 0x7fbfb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:27.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.900+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfbc08a6c0 con 0x7fbfd00cce90 2024-03-06T07:29:27.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.901+0000 7fbfd5c00640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfb006b4d0 0x7fbfb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:27.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.902+0000 7fbfd5c00640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfb006b4d0 0x7fbfb006d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfd015f430 tx=0x7fbfc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:27.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:27.905+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfbc059530 con 0x7fbfd00cce90 2024-03-06T07:29:28.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.108+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfd0157050 con 0x7fbfd00cce90 2024-03-06T07:29:28.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.112+0000 7fbfcd600640 1 -- 172.21.15.6:0/3221082724 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16554 (secure 0 0 0) 0x7fbfbc058e30 con 0x7fbfd00cce90 2024-03-06T07:29:28.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:28.114 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:26.634750+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfb006b4d0 msgr2=0x7fbfb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfb006b4d0 0x7fbfb006d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfd015f430 tx=0x7fbfc4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 msgr2=0x7fbfd015d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015d720 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00d800 tx=0x7fbfbc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 shutdown_connections 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfb006b4d0 0x7fbfb006d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfd0145300 0x7fbfd01541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfd00cce90 0x7fbfd015d720 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 --2- 172.21.15.6:0/3221082724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfd0070f10 0x7fbfd0137390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 >> 172.21.15.6:0/3221082724 conn(0x7fbfd006b3d0 msgr2=0x7fbfd006de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 shutdown_connections 2024-03-06T07:29:28.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.115+0000 7fbfd7000640 1 -- 172.21.15.6:0/3221082724 wait complete. 2024-03-06T07:29:28.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:29:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.359+0000 7f98a9200640 1 Processor -- start 2024-03-06T07:29:28.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.359+0000 7f98a9200640 1 -- start start 2024-03-06T07:29:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.361+0000 7f98a9200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f98a40cd1e0 0x7f98a40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.361+0000 7f98a9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a40b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.361+0000 7f98a9200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a4071c60 con 0x7f98a415afa0 2024-03-06T07:29:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.361+0000 7f98a9200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a40cfa10 con 0x7f98a40cd1e0 2024-03-06T07:29:28.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.361+0000 7f98a9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a4071f40 con 0x7f98a4070ce0 2024-03-06T07:29:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.362+0000 7f989be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f98a40cd1e0 0x7f98a40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.362+0000 7f98a2800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a40b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.362+0000 7f989be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f98a40cd1e0 0x7f98a40cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:41332/0 (socket says 172.21.15.6:41332) 2024-03-06T07:29:28.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.362+0000 7f989be00640 1 -- 172.21.15.6:0/1809697866 learned_addr learned my addr 172.21.15.6:0/1809697866 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.363+0000 7f989be00640 1 -- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 msgr2=0x7f98a40b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.363+0000 7f989be00640 1 --2- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a40b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.363+0000 7f989be00640 1 -- 172.21.15.6:0/1809697866 >> v1:172.21.15.178:6789/0 conn(0x7f98a415afa0 legacy=0x7f98a40ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.363+0000 7f989be00640 1 -- 172.21.15.6:0/1809697866 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4072220 con 0x7f98a40cd1e0 2024-03-06T07:29:28.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.364+0000 7f989be00640 1 --2- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a40cf3a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f988c009870 tx=0x7f988c02db00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf368202ca57cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.365+0000 7f98a1400640 1 -- 172.21.15.6:0/1809697866 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c02f8c0 con 0x7f98a40cd1e0 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.365+0000 7f98a1400640 1 -- 172.21.15.6:0/1809697866 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c02fa60 con 0x7f98a40cd1e0 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.365+0000 7f98a1400640 1 -- 172.21.15.6:0/1809697866 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c02fe30 con 0x7f98a40cd1e0 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.365+0000 7f98a9200640 1 -- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 msgr2=0x7f98a40cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.365+0000 7f98a9200640 1 --2- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a40cf3a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f988c009870 tx=0x7f988c02db00 comp rx=0 tx=0).stop 2024-03-06T07:29:28.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.366+0000 7f98a9200640 1 -- 172.21.15.6:0/1809697866 shutdown_connections 2024-03-06T07:29:28.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.366+0000 7f98a9200640 1 --2- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a40b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.366+0000 7f98a9200640 1 --2- 172.21.15.6:0/1809697866 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a40cf3a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.366+0000 7f98a9200640 1 -- 172.21.15.6:0/1809697866 >> 172.21.15.6:0/1809697866 conn(0x7f98a406b3d0 msgr2=0x7f98a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.366+0000 7f98a9200640 1 -- 172.21.15.6:0/1809697866 shutdown_connections 2024-03-06T07:29:28.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.366+0000 7f98a9200640 1 -- 172.21.15.6:0/1809697866 wait complete. 2024-03-06T07:29:28.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.368+0000 7f98a9200640 1 Processor -- start 2024-03-06T07:29:28.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.368+0000 7f98a9200640 1 -- start start 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a9200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a4146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 0x7f98a40cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40b4000 con 0x7f98a415afa0 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a9200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a4071150 con 0x7f98a40cd1e0 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40b4180 con 0x7f98a4070ce0 2024-03-06T07:29:28.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 0x7f98a40cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 0x7f98a40cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50700/0 (socket says 172.21.15.6:50700) 2024-03-06T07:29:28.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a2800640 1 -- 172.21.15.6:0/1295756048 learned_addr learned my addr 172.21.15.6:0/1295756048 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f98a1e00640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.369+0000 7f989be00640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a4146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f98a2800640 1 -- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 msgr2=0x7f98a409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f98a2800640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a409aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f98a2800640 1 -- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 msgr2=0x7f98a4146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f98a2800640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a4146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f98a2800640 1 -- 172.21.15.6:0/1295756048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a40ca400 con 0x7f98a415afa0 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f98a2800640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 0x7f98a40cb4a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f989c009dd0 tx=0x7f989c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c018020 con 0x7f98a415afa0 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c014480 con 0x7f98a415afa0 2024-03-06T07:29:28.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.370+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c014870 con 0x7f98a415afa0 2024-03-06T07:29:28.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.371+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a40ca6e0 con 0x7f98a415afa0 2024-03-06T07:29:28.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.371+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a40caca0 con 0x7f98a415afa0 2024-03-06T07:29:28.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.372+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a404f330 con 0x7f98a415afa0 2024-03-06T07:29:28.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.373+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f989c01b020 con 0x7f98a415afa0 2024-03-06T07:29:28.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.374+0000 7f989a000640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f988006b500 0x7f988006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.374+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f989c08ad80 con 0x7f98a415afa0 2024-03-06T07:29:28.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.375+0000 7f98a1e00640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f988006b500 0x7f988006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.376+0000 7f98a1e00640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f988006b500 0x7f988006d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9894007fd0 tx=0x7f9894009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.379+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f989c022030 con 0x7f98a415afa0 2024-03-06T07:29:28.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:28.618+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f98a41469c0 con 0x7f98a415afa0 2024-03-06T07:29:29.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.223+0000 7f989a000640 1 -- 172.21.15.6:0/1295756048 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f989c010040 con 0x7f98a415afa0 2024-03-06T07:29:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.226+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f988006b500 msgr2=0x7f988006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.226+0000 7f98a9200640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f988006b500 0x7f988006d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9894007fd0 tx=0x7f9894009040 comp rx=0 tx=0).stop 2024-03-06T07:29:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 msgr2=0x7f98a40cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 0x7f98a40cb4a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f989c009dd0 tx=0x7f989c00c490 comp rx=0 tx=0).stop 2024-03-06T07:29:29.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 shutdown_connections 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f988006b500 0x7f988006d9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98a40cd1e0 0x7f98a4146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98a415afa0 0x7f98a40cb4a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 --2- 172.21.15.6:0/1295756048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98a4070ce0 0x7f98a409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 >> 172.21.15.6:0/1295756048 conn(0x7f98a406b3d0 msgr2=0x7f98a4159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 shutdown_connections 2024-03-06T07:29:29.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.227+0000 7f98a9200640 1 -- 172.21.15.6:0/1295756048 wait complete. 2024-03-06T07:29:29.249 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:29:29.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:29:29.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.471+0000 7f5965a00640 1 Processor -- start 2024-03-06T07:29:29.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.471+0000 7f5965a00640 1 -- start start 2024-03-06T07:29:29.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.473+0000 7f5965a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f596006edc0 0x7f596006fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.473+0000 7f5965a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f596015f1c0 0x7f596015fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.473+0000 7f5965a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59600b1530 con 0x7f596006f9c0 2024-03-06T07:29:29.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.473+0000 7f5965a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59600cff10 con 0x7f596015f1c0 2024-03-06T07:29:29.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.473+0000 7f5965a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596006f5a0 con 0x7f596006edc0 2024-03-06T07:29:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f596006f9c0 0x7f59600d0c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43396/0 (socket says 172.21.15.6:43396) 2024-03-06T07:29:29.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f596015f1c0 0x7f596015fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595ea00640 1 -- 172.21.15.6:0/1859284079 learned_addr learned my addr 172.21.15.6:0/1859284079 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f5957e00640 1 --2- 172.21.15.6:0/1859284079 >> v2:172.21.15.178:3300/0 conn(0x7f596006edc0 0x7f596006fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595f400640 1 -- 172.21.15.6:0/1859284079 >> v2:172.21.15.178:3300/0 conn(0x7f596006edc0 msgr2=0x7f596006fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595f400640 1 --2- 172.21.15.6:0/1859284079 >> v2:172.21.15.178:3300/0 conn(0x7f596006edc0 0x7f596006fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595f400640 1 -- 172.21.15.6:0/1859284079 >> v1:172.21.15.6:6789/0 conn(0x7f596006f9c0 legacy=0x7f59600d0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595f400640 1 -- 172.21.15.6:0/1859284079 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596006f2c0 con 0x7f596015f1c0 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.474+0000 7f595e000640 1 -- 172.21.15.6:0/1859284079 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830009350 0 0) 0x7f5950001110 con 0x7f596006f9c0 2024-03-06T07:29:29.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.476+0000 7f595f400640 1 --2- 172.21.15.6:0/1859284079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596015f1c0 0x7f596015fab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5960071b70 tx=0x7f595802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8383b223e83bfb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.476+0000 7f595e000640 1 -- 172.21.15.6:0/1859284079 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958039070 con 0x7f596015f1c0 2024-03-06T07:29:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.476+0000 7f595e000640 1 -- 172.21.15.6:0/1859284079 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595802cc80 con 0x7f596015f1c0 2024-03-06T07:29:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.476+0000 7f595e000640 1 -- 172.21.15.6:0/1859284079 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958034660 con 0x7f596015f1c0 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.476+0000 7f5965a00640 1 -- 172.21.15.6:0/1859284079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596015f1c0 msgr2=0x7f596015fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.476+0000 7f5965a00640 1 --2- 172.21.15.6:0/1859284079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596015f1c0 0x7f596015fab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5960071b70 tx=0x7f595802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.477+0000 7f5965a00640 1 -- 172.21.15.6:0/1859284079 shutdown_connections 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.477+0000 7f5965a00640 1 --2- 172.21.15.6:0/1859284079 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596015f1c0 0x7f596015fab0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.477+0000 7f5965a00640 1 --2- 172.21.15.6:0/1859284079 >> v2:172.21.15.178:3300/0 conn(0x7f596006edc0 0x7f596006fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.477+0000 7f5965a00640 1 -- 172.21.15.6:0/1859284079 >> 172.21.15.6:0/1859284079 conn(0x7f59600b3820 msgr2=0x7f59600b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.477+0000 7f5965a00640 1 -- 172.21.15.6:0/1859284079 shutdown_connections 2024-03-06T07:29:29.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.477+0000 7f5965a00640 1 -- 172.21.15.6:0/1859284079 wait complete. 2024-03-06T07:29:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.478+0000 7f5965a00640 1 Processor -- start 2024-03-06T07:29:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.479+0000 7f5965a00640 1 -- start start 2024-03-06T07:29:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.479+0000 7f5965a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596006edc0 0x7f59601f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5965a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 0x7f59600b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5965a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596015f1c0 0x7f59600b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5965a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600d1330 con 0x7f596006f9c0 2024-03-06T07:29:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5965a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59600d0790 con 0x7f596006edc0 2024-03-06T07:29:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5965a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59600d0420 con 0x7f596015f1c0 2024-03-06T07:29:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 0x7f59600b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 0x7f59600b4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50714/0 (socket says 172.21.15.6:50714) 2024-03-06T07:29:29.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 -- 172.21.15.6:0/18978156 learned_addr learned my addr 172.21.15.6:0/18978156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:29.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 -- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596015f1c0 msgr2=0x7f59600b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f595f400640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596015f1c0 0x7f59600b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596015f1c0 0x7f59600b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 -- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596006edc0 msgr2=0x7f59601f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f595ea00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596006edc0 0x7f59601f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596006edc0 0x7f59601f3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.480+0000 7f5957e00640 1 -- 172.21.15.6:0/18978156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5960157290 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.481+0000 7f5957e00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 0x7f59600b4910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f594800e8a0 tx=0x7f594800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.481+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5948017070 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.481+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594800c940 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.481+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594800cd30 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.481+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5960157570 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.481+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59601f4300 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.482+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59600cf540 con 0x7f596006f9c0 2024-03-06T07:29:29.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.483+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f594801a070 con 0x7f596006f9c0 2024-03-06T07:29:29.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.484+0000 7f5956a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f593c06b4d0 0x7f593c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:29.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.484+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f594808a130 con 0x7f596006f9c0 2024-03-06T07:29:29.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.485+0000 7f595ea00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f593c06b4d0 0x7f593c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:29.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.486+0000 7f595ea00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f593c06b4d0 0x7f593c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5950000f60 tx=0x7f5950009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:29.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.489+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59600cf540 con 0x7f596006f9c0 2024-03-06T07:29:29.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:29.731+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f596004f330 con 0x7f596006f9c0 2024-03-06T07:29:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.233+0000 7f5956a00640 1 -- 172.21.15.6:0/18978156 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5948058f10 con 0x7f596006f9c0 2024-03-06T07:29:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f593c06b4d0 msgr2=0x7f593c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f593c06b4d0 0x7f593c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5950000f60 tx=0x7f5950009040 comp rx=0 tx=0).stop 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 msgr2=0x7f59600b4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 0x7f59600b4910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f594800e8a0 tx=0x7f594800ed60 comp rx=0 tx=0).stop 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 shutdown_connections 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f593c06b4d0 0x7f593c06d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596015f1c0 0x7f59600b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596006f9c0 0x7f59600b4910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 --2- 172.21.15.6:0/18978156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596006edc0 0x7f59601f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.236+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 >> 172.21.15.6:0/18978156 conn(0x7f59600b3820 msgr2=0x7f59600af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.237+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 shutdown_connections 2024-03-06T07:29:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.237+0000 7f5965a00640 1 -- 172.21.15.6:0/18978156 wait complete. 2024-03-06T07:29:30.261 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:29:30.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:30.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.480+0000 7f078b200640 1 Processor -- start 2024-03-06T07:29:30.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.481+0000 7f078b200640 1 -- start start 2024-03-06T07:29:30.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.482+0000 7f078b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f078415e0b0 0x7f07840cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.482+0000 7f078b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f07840d0e50 0x7f07840b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.482+0000 7f078b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07840b3710 con 0x7f07840d4a10 2024-03-06T07:29:30.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.482+0000 7f078b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07840b09e0 con 0x7f078415e0b0 2024-03-06T07:29:30.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.482+0000 7f078b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07840b39f0 con 0x7f07840d0e50 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f07840d4a10 0x7f07840d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43412/0 (socket says 172.21.15.6:43412) 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789400640 1 -- 172.21.15.6:0/4054486409 learned_addr learned my addr 172.21.15.6:0/4054486409 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789e00640 1 --2- 172.21.15.6:0/4054486409 >> v2:172.21.15.6:3300/0 conn(0x7f078415e0b0 0x7f07840cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f078a800640 1 --2- 172.21.15.6:0/4054486409 >> v2:172.21.15.178:3300/0 conn(0x7f07840d0e50 0x7f07840b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0788a00640 1 -- 172.21.15.6:0/4054486409 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130094603 0 0) 0x7f07840b3710 con 0x7f07840d4a10 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0788a00640 1 -- 172.21.15.6:0/4054486409 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f076c003520 con 0x7f07840d4a10 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789e00640 1 -- 172.21.15.6:0/4054486409 >> v2:172.21.15.178:3300/0 conn(0x7f07840d0e50 msgr2=0x7f07840b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789e00640 1 --2- 172.21.15.6:0/4054486409 >> v2:172.21.15.178:3300/0 conn(0x7f07840d0e50 0x7f07840b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789e00640 1 -- 172.21.15.6:0/4054486409 >> v1:172.21.15.6:6789/0 conn(0x7f07840d4a10 legacy=0x7f07840d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.483+0000 7f0789e00640 1 -- 172.21.15.6:0/4054486409 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07840b3cd0 con 0x7f078415e0b0 2024-03-06T07:29:30.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.485+0000 7f0789e00640 1 --2- 172.21.15.6:0/4054486409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078415e0b0 0x7f07840cf310 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f078408d6d0 tx=0x7f077802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa48d3fe7dc0543 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.485+0000 7f0788a00640 1 -- 172.21.15.6:0/4054486409 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778039070 con 0x7f078415e0b0 2024-03-06T07:29:30.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.485+0000 7f0788a00640 1 -- 172.21.15.6:0/4054486409 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077802cc70 con 0x7f078415e0b0 2024-03-06T07:29:30.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.485+0000 7f0788a00640 1 -- 172.21.15.6:0/4054486409 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778034640 con 0x7f078415e0b0 2024-03-06T07:29:30.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 -- 172.21.15.6:0/4054486409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078415e0b0 msgr2=0x7f07840cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 --2- 172.21.15.6:0/4054486409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078415e0b0 0x7f07840cf310 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f078408d6d0 tx=0x7f077802bec0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 -- 172.21.15.6:0/4054486409 shutdown_connections 2024-03-06T07:29:30.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 --2- 172.21.15.6:0/4054486409 >> v2:172.21.15.178:3300/0 conn(0x7f07840d0e50 0x7f07840b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 --2- 172.21.15.6:0/4054486409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f078415e0b0 0x7f07840cf310 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 -- 172.21.15.6:0/4054486409 >> 172.21.15.6:0/4054486409 conn(0x7f07840d3d30 msgr2=0x7f07840d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 -- 172.21.15.6:0/4054486409 shutdown_connections 2024-03-06T07:29:30.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.486+0000 7f078b200640 1 -- 172.21.15.6:0/4054486409 wait complete. 2024-03-06T07:29:30.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.488+0000 7f078b200640 1 Processor -- start 2024-03-06T07:29:30.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.488+0000 7f078b200640 1 -- start start 2024-03-06T07:29:30.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f078b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07840d0e50 0x7f07840cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f078b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 0x7f07840cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f078b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f078415e0b0 0x7f07840b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f078b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07840cfc60 con 0x7f07840d4a10 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f078b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078415e490 con 0x7f078415e0b0 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f078b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07840cf850 con 0x7f07840d0e50 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f0789400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 0x7f07840cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f0789400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 0x7f07840cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50740/0 (socket says 172.21.15.6:50740) 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.489+0000 7f0789400640 1 -- 172.21.15.6:0/2402155915 learned_addr learned my addr 172.21.15.6:0/2402155915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789e00640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07840d0e50 0x7f07840cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f078a800640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f078415e0b0 0x7f07840b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789400640 1 -- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07840d0e50 msgr2=0x7f07840cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789400640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07840d0e50 0x7f07840cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789400640 1 -- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f078415e0b0 msgr2=0x7f07840b7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789400640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f078415e0b0 0x7f07840b7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789400640 1 -- 172.21.15.6:0/2402155915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07840cb200 con 0x7f07840d4a10 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f0789400640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 0x7f07840cd9b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f077400c400 tx=0x7f077400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774017070 con 0x7f07840d4a10 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0774005d00 con 0x7f07840d4a10 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774007310 con 0x7f07840d4a10 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.490+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07840cb4e0 con 0x7f07840d4a10 2024-03-06T07:29:30.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.491+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07840cbaa0 con 0x7f07840d4a10 2024-03-06T07:29:30.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.492+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078404f330 con 0x7f07840d4a10 2024-03-06T07:29:30.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.493+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f077401a070 con 0x7f07840d4a10 2024-03-06T07:29:30.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.493+0000 7f077e000640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f076806b520 0x7f076806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.494+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0774089670 con 0x7f07840d4a10 2024-03-06T07:29:30.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.494+0000 7f0789e00640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f076806b520 0x7f076806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.495+0000 7f0789e00640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f076806b520 0x7f076806d9e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0778036000 tx=0x7f0778030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.498+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0774058560 con 0x7f07840d4a10 2024-03-06T07:29:30.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.734+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07840cf3a0 con 0x7f07840d4a10 2024-03-06T07:29:30.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.735+0000 7f077e000640 1 -- 172.21.15.6:0/2402155915 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f0774057e60 con 0x7f07840d4a10 2024-03-06T07:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:30.738 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:26.635978+0000","modified":"2024-03-06T07:29:29.733477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/590822743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":590822743},{"type":"v1","addr":"172.21.15.178:6837","nonce":590822743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5562,"qdb_cluster":[5562]},"id":3}]} 2024-03-06T07:29:30.738 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-06T07:29:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f076806b520 msgr2=0x7f076806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f076806b520 0x7f076806d9e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0778036000 tx=0x7f0778030000 comp rx=0 tx=0).stop 2024-03-06T07:29:30.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 msgr2=0x7f07840cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 0x7f07840cd9b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f077400c400 tx=0x7f077400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 shutdown_connections 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f076806b520 0x7f076806d9e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f078415e0b0 0x7f07840b7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07840d4a10 0x7f07840cd9b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 --2- 172.21.15.6:0/2402155915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07840d0e50 0x7f07840cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 >> 172.21.15.6:0/2402155915 conn(0x7f07840d3d30 msgr2=0x7f078406be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 shutdown_connections 2024-03-06T07:29:30.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.738+0000 7f078b200640 1 -- 172.21.15.6:0/2402155915 wait complete. 2024-03-06T07:29:30.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:30.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.961+0000 7fe242c00640 1 Processor -- start 2024-03-06T07:29:30.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.961+0000 7fe242c00640 1 -- start start 2024-03-06T07:29:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.963+0000 7fe242c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe23c15e200 0x7fe23c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.963+0000 7fe242c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c0afdf0 0x7fe23c0af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.963+0000 7fe242c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23c06bec0 con 0x7fe23c06afb0 2024-03-06T07:29:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.963+0000 7fe242c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe23c0b0670 con 0x7fe23c15e200 2024-03-06T07:29:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.963+0000 7fe242c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c0ae380 con 0x7fe23c0afdf0 2024-03-06T07:29:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe23c15e200 0x7fe23c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe23c15e200 0x7fe23c0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50756/0 (socket says 172.21.15.6:50756) 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 -- 172.21.15.6:0/3217123436 learned_addr learned my addr 172.21.15.6:0/3217123436 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23b400640 1 --2- 172.21.15.6:0/3217123436 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c0afdf0 0x7fe23c0af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 -- 172.21.15.6:0/3217123436 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c0afdf0 msgr2=0x7fe23c0af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 --2- 172.21.15.6:0/3217123436 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c0afdf0 0x7fe23c0af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 -- 172.21.15.6:0/3217123436 >> v1:172.21.15.178:6789/0 conn(0x7fe23c06afb0 legacy=0x7fe23c06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.964+0000 7fe23be00640 1 -- 172.21.15.6:0/3217123436 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c0ae660 con 0x7fe23c15e200 2024-03-06T07:29:30.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.965+0000 7fe23be00640 1 --2- 172.21.15.6:0/3217123436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c15e200 0x7fe23c0b3de0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe23c071cc0 tx=0x7fe22c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3558fbf874a9b95a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe23aa00640 1 -- 172.21.15.6:0/3217123436 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c035020 con 0x7fe23c15e200 2024-03-06T07:29:30.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe23aa00640 1 -- 172.21.15.6:0/3217123436 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c031640 con 0x7fe23c15e200 2024-03-06T07:29:30.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe23aa00640 1 -- 172.21.15.6:0/3217123436 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c02d040 con 0x7fe23c15e200 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe242c00640 1 -- 172.21.15.6:0/3217123436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c15e200 msgr2=0x7fe23c0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe242c00640 1 --2- 172.21.15.6:0/3217123436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c15e200 0x7fe23c0b3de0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe23c071cc0 tx=0x7fe22c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe242c00640 1 -- 172.21.15.6:0/3217123436 shutdown_connections 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe242c00640 1 --2- 172.21.15.6:0/3217123436 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c0afdf0 0x7fe23c0af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe242c00640 1 --2- 172.21.15.6:0/3217123436 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c15e200 0x7fe23c0b3de0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.966+0000 7fe242c00640 1 -- 172.21.15.6:0/3217123436 >> 172.21.15.6:0/3217123436 conn(0x7fe23c0b2ae0 msgr2=0x7fe23c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.967+0000 7fe242c00640 1 -- 172.21.15.6:0/3217123436 shutdown_connections 2024-03-06T07:29:30.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.967+0000 7fe242c00640 1 -- 172.21.15.6:0/3217123436 wait complete. 2024-03-06T07:29:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.968+0000 7fe242c00640 1 Processor -- start 2024-03-06T07:29:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.968+0000 7fe242c00640 1 -- start start 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.969+0000 7fe242c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c06afb0 0x7fe23c149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.969+0000 7fe242c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 0x7fe23c15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.969+0000 7fe242c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe23c15e200 0x7fe23c1e9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.969+0000 7fe242c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c0af680 con 0x7fe23c0afdf0 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.969+0000 7fe242c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c0aec20 con 0x7fe23c15e200 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.969+0000 7fe242c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23c06e2c0 con 0x7fe23c06afb0 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 0x7fe23c15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 0x7fe23c15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50768/0 (socket says 172.21.15.6:50768) 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 -- 172.21.15.6:0/3612700853 learned_addr learned my addr 172.21.15.6:0/3612700853 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23be00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c06afb0 0x7fe23c149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 -- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c06afb0 msgr2=0x7fe23c149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c06afb0 0x7fe23c149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe240c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe23c15e200 0x7fe23c1e9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 -- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe23c15e200 msgr2=0x7fe23c1e9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe23c15e200 0x7fe23c1e9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 -- 172.21.15.6:0/3612700853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23c15d2b0 con 0x7fe23c0afdf0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23be00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c06afb0 0x7fe23c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.970+0000 7fe23b400640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 0x7fe23c15bfc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe22400d7d0 tx=0x7fe22400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.971+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224004020 con 0x7fe23c0afdf0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.971+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2240041e0 con 0x7fe23c0afdf0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.971+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe23c15ca10 con 0x7fe23c0afdf0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.971+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2240043d0 con 0x7fe23c0afdf0 2024-03-06T07:29:30.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.971+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23c15cfd0 con 0x7fe23c0afdf0 2024-03-06T07:29:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.972+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23c070260 con 0x7fe23c0afdf0 2024-03-06T07:29:30.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.973+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe22401b020 con 0x7fe23c0afdf0 2024-03-06T07:29:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.974+0000 7fe233e00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe21806b4d0 0x7fe21806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.974+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe224089f80 con 0x7fe23c0afdf0 2024-03-06T07:29:30.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.975+0000 7fe23be00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe21806b4d0 0x7fe21806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.976+0000 7fe23be00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe21806b4d0 0x7fe21806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe22c036000 tx=0x7fe22c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:30.979+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe23c070260 con 0x7fe23c0afdf0 2024-03-06T07:29:31.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.179+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe23c04f3a0 con 0x7fe23c0afdf0 2024-03-06T07:29:31.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.182+0000 7fe233e00640 1 -- 172.21.15.6:0/3612700853 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fe224058e90 con 0x7fe23c0afdf0 2024-03-06T07:29:31.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:31.185 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:28.183720+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:31.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe21806b4d0 msgr2=0x7fe21806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe21806b4d0 0x7fe21806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe22c036000 tx=0x7fe22c008030 comp rx=0 tx=0).stop 2024-03-06T07:29:31.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 msgr2=0x7fe23c15bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 0x7fe23c15bfc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe22400d7d0 tx=0x7fe22400dc90 comp rx=0 tx=0).stop 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 shutdown_connections 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe21806b4d0 0x7fe21806d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe23c15e200 0x7fe23c1e9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe23c0afdf0 0x7fe23c15bfc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 --2- 172.21.15.6:0/3612700853 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe23c06afb0 0x7fe23c149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 >> 172.21.15.6:0/3612700853 conn(0x7fe23c0b2ae0 msgr2=0x7fe23c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 shutdown_connections 2024-03-06T07:29:31.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.185+0000 7fe242c00640 1 -- 172.21.15.6:0/3612700853 wait complete. 2024-03-06T07:29:31.207 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:29:31.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:31.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.421+0000 7f8130800640 1 Processor -- start 2024-03-06T07:29:31.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.421+0000 7f8130800640 1 -- start start 2024-03-06T07:29:31.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.422+0000 7f8130800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.422+0000 7f8130800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f81280d1480 0x7f81280b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.422+0000 7f8130800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81280b3940 con 0x7f812815e2e0 2024-03-06T07:29:31.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.422+0000 7f8130800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81280d4e70 con 0x7f81280d1480 2024-03-06T07:29:31.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.422+0000 7f8130800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81280b3c40 con 0x7f81280d2b40 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.423+0000 7f812e800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f81280d1480 0x7f81280b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.423+0000 7f812d400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.423+0000 7f812e800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f81280d1480 0x7f81280b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:41386/0 (socket says 172.21.15.6:41386) 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.423+0000 7f812e800640 1 -- 172.21.15.6:0/1639008214 learned_addr learned my addr 172.21.15.6:0/1639008214 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.424+0000 7f812e800640 1 -- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 msgr2=0x7f81280d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.424+0000 7f812e800640 1 --2- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.424+0000 7f812e800640 1 -- 172.21.15.6:0/1639008214 >> v1:172.21.15.178:6789/0 conn(0x7f812815e2e0 legacy=0x7f81280d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.424+0000 7f812e800640 1 -- 172.21.15.6:0/1639008214 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81280cf580 con 0x7f81280d1480 2024-03-06T07:29:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.425+0000 7f812e800640 1 --2- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81280d1480 0x7f81280b40d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8124009870 tx=0x7f812402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=364d25f32cb94de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f812ca00640 1 -- 172.21.15.6:0/1639008214 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124034070 con 0x7f81280d1480 2024-03-06T07:29:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f812ca00640 1 -- 172.21.15.6:0/1639008214 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8124030840 con 0x7f81280d1480 2024-03-06T07:29:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f812ca00640 1 -- 172.21.15.6:0/1639008214 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8124030c10 con 0x7f81280d1480 2024-03-06T07:29:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 -- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81280d1480 msgr2=0x7f81280b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 --2- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81280d1480 0x7f81280b40d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8124009870 tx=0x7f812402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 -- 172.21.15.6:0/1639008214 shutdown_connections 2024-03-06T07:29:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 --2- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81280d1480 0x7f81280b40d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 --2- 172.21.15.6:0/1639008214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 -- 172.21.15.6:0/1639008214 >> 172.21.15.6:0/1639008214 conn(0x7f81280d3f60 msgr2=0x7f81280d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 -- 172.21.15.6:0/1639008214 shutdown_connections 2024-03-06T07:29:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.426+0000 7f8130800640 1 -- 172.21.15.6:0/1639008214 wait complete. 2024-03-06T07:29:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.429+0000 7f8130800640 1 Processor -- start 2024-03-06T07:29:31.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.429+0000 7f8130800640 1 -- start start 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f8130800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f8130800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 0x7f81281f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f8130800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81281610d0 0x7f81280cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f8130800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81280cfe20 con 0x7f812815e2e0 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f8130800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81280cfb40 con 0x7f81281610d0 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f8130800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81280b4610 con 0x7f81280d2b40 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f812d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 0x7f81281f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f812d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 0x7f81281f4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50782/0 (socket says 172.21.15.6:50782) 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.430+0000 7f812d400640 1 -- 172.21.15.6:0/1807642985 learned_addr learned my addr 172.21.15.6:0/1807642985 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812de00640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812e800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81281610d0 0x7f81280cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812d400640 1 -- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 msgr2=0x7f81280c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812d400640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812d400640 1 -- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81281610d0 msgr2=0x7f81280cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812d400640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81281610d0 0x7f81280cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812d400640 1 -- 172.21.15.6:0/1807642985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81280c8190 con 0x7f812815e2e0 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f812d400640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 0x7f81281f4640 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f811800e980 tx=0x7f811800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.431+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8118017070 con 0x7f812815e2e0 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.432+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811800ca00 con 0x7f812815e2e0 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.432+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81280c8470 con 0x7f812815e2e0 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.432+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811800cdd0 con 0x7f812815e2e0 2024-03-06T07:29:31.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.432+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81280c8a30 con 0x7f812815e2e0 2024-03-06T07:29:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.433+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812804f3a0 con 0x7f812815e2e0 2024-03-06T07:29:31.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.434+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f811801a070 con 0x7f812815e2e0 2024-03-06T07:29:31.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.435+0000 7f8122000640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f810c06b4d0 0x7f810c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.435+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f811808a690 con 0x7f812815e2e0 2024-03-06T07:29:31.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.436+0000 7f812de00640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f810c06b4d0 0x7f810c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.437+0000 7f812de00640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f810c06b4d0 0x7f810c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f81280ca020 tx=0x7f811c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.440+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8118059470 con 0x7f812815e2e0 2024-03-06T07:29:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.628+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f81280d2750 con 0x7f810c06b4d0 2024-03-06T07:29:31.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.649+0000 7f8122000640 1 -- 172.21.15.6:0/1807642985 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247401 (secure 0 0 0) 0x7f81280d2750 con 0x7f810c06b4d0 2024-03-06T07:29:31.655 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:31.656 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:29:31.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f810c06b4d0 msgr2=0x7f810c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f810c06b4d0 0x7f810c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f81280ca020 tx=0x7f811c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 msgr2=0x7f81281f4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 0x7f81281f4640 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f811800e980 tx=0x7f811800ee40 comp rx=0 tx=0).stop 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 shutdown_connections 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f810c06b4d0 0x7f810c06d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81281610d0 0x7f81280cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f812815e2e0 0x7f81281f4640 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 --2- 172.21.15.6:0/1807642985 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81280d2b40 0x7f81280c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 >> 172.21.15.6:0/1807642985 conn(0x7f81280d3f60 msgr2=0x7f81280b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 shutdown_connections 2024-03-06T07:29:31.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.654+0000 7f8130800640 1 -- 172.21.15.6:0/1807642985 wait complete. 2024-03-06T07:29:31.675 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-03-06T07:29:31.480064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":169,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254568,"kb_used_data":5104,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720152,"statfs":{"total":773094113280,"available":772833435648,"internally_reserved":0,"allocated":5226496,"data_stored":2799602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001807"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180481+0000","last_change":"2024-03-06T07:29:26.180481+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180481+0000","last_undegraded":"2024-03-06T07:29:26.180481+0000","last_fullsized":"2024-03-06T07:29:26.180481+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.185266+0000","last_change":"2024-03-06T07:29:26.185266+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.185266+0000","last_undegraded":"2024-03-06T07:29:26.185266+0000","last_fullsized":"2024-03-06T07:29:26.185266+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.186354+0000","last_change":"2024-03-06T07:29:26.186354+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.186354+0000","last_undegraded":"2024-03-06T07:29:26.186354+0000","last_fullsized":"2024-03-06T07:29:26.186354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180082+0000","last_change":"2024-03-06T07:29:26.180082+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180082+0000","last_undegraded":"2024-03-06T07:29:26.180082+0000","last_fullsized":"2024-03-06T07:29:26.180082+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179026+0000","last_change":"2024-03-06T07:29:26.179026+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179026+0000","last_undegraded":"2024-03-06T07:29:26.179026+0000","last_fullsized":"2024-03-06T07:29:26.179026+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.165169+0000","last_change":"2024-03-06T07:29:26.165169+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.165169+0000","last_undegraded":"2024-03-06T07:29:26.165169+0000","last_fullsized":"2024-03-06T07:29:26.165169+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173991+0000","last_change":"2024-03-06T07:29:26.173991+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173991+0000","last_undegraded":"2024-03-06T07:29:26.173991+0000","last_fullsized":"2024-03-06T07:29:26.173991+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179661+0000","last_change":"2024-03-06T07:29:26.179661+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179661+0000","last_undegraded":"2024-03-06T07:29:26.179661+0000","last_fullsized":"2024-03-06T07:29:26.179661+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179688+0000","last_change":"2024-03-06T07:29:26.179688+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179688+0000","last_undegraded":"2024-03-06T07:29:26.179688+0000","last_fullsized":"2024-03-06T07:29:26.179688+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179130+0000","last_change":"2024-03-06T07:29:26.179130+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179130+0000","last_undegraded":"2024-03-06T07:29:26.179130+0000","last_fullsized":"2024-03-06T07:29:26.179130+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.177134+0000","last_change":"2024-03-06T07:29:26.177134+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.177134+0000","last_undegraded":"2024-03-06T07:29:26.177134+0000","last_fullsized":"2024-03-06T07:29:26.177134+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.186917+0000","last_change":"2024-03-06T07:29:26.186917+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.186917+0000","last_undegraded":"2024-03-06T07:29:26.186917+0000","last_fullsized":"2024-03-06T07:29:26.186917+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.236195+0000","last_change":"2024-03-06T07:29:27.234158+0000","last_active":"2024-03-06T07:29:27.236195+0000","last_peered":"2024-03-06T07:29:27.236195+0000","last_clean":"2024-03-06T07:29:27.236195+0000","last_became_active":"2024-03-06T07:29:27.233803+0000","last_became_peered":"2024-03-06T07:29:27.233803+0000","last_unstale":"2024-03-06T07:29:27.236195+0000","last_undegraded":"2024-03-06T07:29:27.236195+0000","last_fullsized":"2024-03-06T07:29:27.236195+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:56.608802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180437+0000","last_change":"2024-03-06T07:29:26.180437+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180437+0000","last_undegraded":"2024-03-06T07:29:26.180437+0000","last_fullsized":"2024-03-06T07:29:26.180437+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.167433+0000","last_change":"2024-03-06T07:29:26.167433+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.167433+0000","last_undegraded":"2024-03-06T07:29:26.167433+0000","last_fullsized":"2024-03-06T07:29:26.167433+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172611+0000","last_change":"2024-03-06T07:29:26.172611+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172611+0000","last_undegraded":"2024-03-06T07:29:26.172611+0000","last_fullsized":"2024-03-06T07:29:26.172611+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.187650+0000","last_change":"2024-03-06T07:29:26.187650+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.187650+0000","last_undegraded":"2024-03-06T07:29:26.187650+0000","last_fullsized":"2024-03-06T07:29:26.187650+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.189091+0000","last_change":"2024-03-06T07:29:26.189091+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.189091+0000","last_undegraded":"2024-03-06T07:29:26.189091+0000","last_fullsized":"2024-03-06T07:29:26.189091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171250+0000","last_change":"2024-03-06T07:29:26.171250+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171250+0000","last_undegraded":"2024-03-06T07:29:26.171250+0000","last_fullsized":"2024-03-06T07:29:26.171250+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.237778+0000","last_change":"2024-03-06T07:29:27.232657+0000","last_active":"2024-03-06T07:29:27.237778+0000","last_peered":"2024-03-06T07:29:27.237778+0000","last_clean":"2024-03-06T07:29:27.237778+0000","last_became_active":"2024-03-06T07:29:27.232358+0000","last_became_peered":"2024-03-06T07:29:27.232358+0000","last_unstale":"2024-03-06T07:29:27.237778+0000","last_undegraded":"2024-03-06T07:29:27.237778+0000","last_fullsized":"2024-03-06T07:29:27.237778+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:14.573067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164817+0000","last_change":"2024-03-06T07:29:26.164817+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164817+0000","last_undegraded":"2024-03-06T07:29:26.164817+0000","last_fullsized":"2024-03-06T07:29:26.164817+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176355+0000","last_change":"2024-03-06T07:29:26.176355+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176355+0000","last_undegraded":"2024-03-06T07:29:26.176355+0000","last_fullsized":"2024-03-06T07:29:26.176355+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.170468+0000","last_change":"2024-03-06T07:29:26.170468+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.170468+0000","last_undegraded":"2024-03-06T07:29:26.170468+0000","last_fullsized":"2024-03-06T07:29:26.170468+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171475+0000","last_change":"2024-03-06T07:29:26.171475+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171475+0000","last_undegraded":"2024-03-06T07:29:26.171475+0000","last_fullsized":"2024-03-06T07:29:26.171475+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.174866+0000","last_change":"2024-03-06T07:29:26.174866+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.174866+0000","last_undegraded":"2024-03-06T07:29:26.174866+0000","last_fullsized":"2024-03-06T07:29:26.174866+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.236949+0000","last_change":"2024-03-06T07:29:27.234582+0000","last_active":"2024-03-06T07:29:27.236949+0000","last_peered":"2024-03-06T07:29:27.236949+0000","last_clean":"2024-03-06T07:29:27.236949+0000","last_became_active":"2024-03-06T07:29:27.233810+0000","last_became_peered":"2024-03-06T07:29:27.233810+0000","last_unstale":"2024-03-06T07:29:27.236949+0000","last_undegraded":"2024-03-06T07:29:27.236949+0000","last_fullsized":"2024-03-06T07:29:27.236949+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:44.382794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.184051+0000","last_change":"2024-03-06T07:29:26.184051+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.184051+0000","last_undegraded":"2024-03-06T07:29:26.184051+0000","last_fullsized":"2024-03-06T07:29:26.184051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.183973+0000","last_change":"2024-03-06T07:29:26.183973+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.183973+0000","last_undegraded":"2024-03-06T07:29:26.183973+0000","last_fullsized":"2024-03-06T07:29:26.183973+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171100+0000","last_change":"2024-03-06T07:29:26.171100+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171100+0000","last_undegraded":"2024-03-06T07:29:26.171100+0000","last_fullsized":"2024-03-06T07:29:26.171100+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.182982+0000","last_change":"2024-03-06T07:29:26.182982+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.182982+0000","last_undegraded":"2024-03-06T07:29:26.182982+0000","last_fullsized":"2024-03-06T07:29:26.182982+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.166901+0000","last_change":"2024-03-06T07:29:26.166901+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.166901+0000","last_undegraded":"2024-03-06T07:29:26.166901+0000","last_fullsized":"2024-03-06T07:29:26.166901+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.233068+0000","last_change":"2024-03-06T07:29:27.227701+0000","last_active":"2024-03-06T07:29:27.233068+0000","last_peered":"2024-03-06T07:29:27.233068+0000","last_clean":"2024-03-06T07:29:27.233068+0000","last_became_active":"2024-03-06T07:29:27.225595+0000","last_became_peered":"2024-03-06T07:29:27.225595+0000","last_unstale":"2024-03-06T07:29:27.233068+0000","last_undegraded":"2024-03-06T07:29:27.233068+0000","last_fullsized":"2024-03-06T07:29:27.233068+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:55.302360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.227914+0000","last_change":"2024-03-06T07:29:27.222401+0000","last_active":"2024-03-06T07:29:27.227914+0000","last_peered":"2024-03-06T07:29:27.227914+0000","last_clean":"2024-03-06T07:29:27.227914+0000","last_became_active":"2024-03-06T07:29:27.221978+0000","last_became_peered":"2024-03-06T07:29:27.221978+0000","last_unstale":"2024-03-06T07:29:27.227914+0000","last_undegraded":"2024-03-06T07:29:27.227914+0000","last_fullsized":"2024-03-06T07:29:27.227914+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:43.908483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192462+0000","last_change":"2024-03-06T07:29:26.184227+0000","last_active":"2024-03-06T07:29:26.192462+0000","last_peered":"2024-03-06T07:29:26.192462+0000","last_clean":"2024-03-06T07:29:26.192462+0000","last_became_active":"2024-03-06T07:29:26.183923+0000","last_became_peered":"2024-03-06T07:29:26.183923+0000","last_unstale":"2024-03-06T07:29:26.192462+0000","last_undegraded":"2024-03-06T07:29:26.192462+0000","last_fullsized":"2024-03-06T07:29:26.192462+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:08.181893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173821+0000","last_change":"2024-03-06T07:29:26.173821+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173821+0000","last_undegraded":"2024-03-06T07:29:26.173821+0000","last_fullsized":"2024-03-06T07:29:26.173821+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.202304+0000","last_change":"2024-03-06T07:29:26.200719+0000","last_active":"2024-03-06T07:29:26.202304+0000","last_peered":"2024-03-06T07:29:26.202304+0000","last_clean":"2024-03-06T07:29:26.202304+0000","last_became_active":"2024-03-06T07:29:26.200438+0000","last_became_peered":"2024-03-06T07:29:26.200438+0000","last_unstale":"2024-03-06T07:29:26.202304+0000","last_undegraded":"2024-03-06T07:29:26.202304+0000","last_fullsized":"2024-03-06T07:29:26.202304+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:27.927651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.183386+0000","last_change":"2024-03-06T07:29:26.183386+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.183386+0000","last_undegraded":"2024-03-06T07:29:26.183386+0000","last_fullsized":"2024-03-06T07:29:26.183386+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.243346+0000","last_change":"2024-03-06T07:29:26.201813+0000","last_active":"2024-03-06T07:29:27.243346+0000","last_peered":"2024-03-06T07:29:27.243346+0000","last_clean":"2024-03-06T07:29:27.243346+0000","last_became_active":"2024-03-06T07:29:26.201360+0000","last_became_peered":"2024-03-06T07:29:26.201360+0000","last_unstale":"2024-03-06T07:29:27.243346+0000","last_undegraded":"2024-03-06T07:29:27.243346+0000","last_fullsized":"2024-03-06T07:29:27.243346+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.232720+0000","last_change":"2024-03-06T07:29:27.231046+0000","last_active":"2024-03-06T07:29:27.232720+0000","last_peered":"2024-03-06T07:29:27.232720+0000","last_clean":"2024-03-06T07:29:27.232720+0000","last_became_active":"2024-03-06T07:29:27.230774+0000","last_became_peered":"2024-03-06T07:29:27.230774+0000","last_unstale":"2024-03-06T07:29:27.232720+0000","last_undegraded":"2024-03-06T07:29:27.232720+0000","last_fullsized":"2024-03-06T07:29:27.232720+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:29.573395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.204067+0000","last_change":"2024-03-06T07:29:26.202950+0000","last_active":"2024-03-06T07:29:26.204067+0000","last_peered":"2024-03-06T07:29:26.204067+0000","last_clean":"2024-03-06T07:29:26.204067+0000","last_became_active":"2024-03-06T07:29:26.202564+0000","last_became_peered":"2024-03-06T07:29:26.202564+0000","last_unstale":"2024-03-06T07:29:26.204067+0000","last_undegraded":"2024-03-06T07:29:26.204067+0000","last_fullsized":"2024-03-06T07:29:26.204067+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.182043+0000","last_change":"2024-03-06T07:29:26.182043+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.182043+0000","last_undegraded":"2024-03-06T07:29:26.182043+0000","last_fullsized":"2024-03-06T07:29:26.182043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192624+0000","last_change":"2024-03-06T07:29:26.188354+0000","last_active":"2024-03-06T07:29:26.192624+0000","last_peered":"2024-03-06T07:29:26.192624+0000","last_clean":"2024-03-06T07:29:26.192624+0000","last_became_active":"2024-03-06T07:29:26.187378+0000","last_became_peered":"2024-03-06T07:29:26.187378+0000","last_unstale":"2024-03-06T07:29:26.192624+0000","last_undegraded":"2024-03-06T07:29:26.192624+0000","last_fullsized":"2024-03-06T07:29:26.192624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:41.649859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.178981+0000","last_change":"2024-03-06T07:29:26.178981+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.178981+0000","last_undegraded":"2024-03-06T07:29:26.178981+0000","last_fullsized":"2024-03-06T07:29:26.178981+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.205035+0000","last_change":"2024-03-06T07:29:26.204279+0000","last_active":"2024-03-06T07:29:26.205035+0000","last_peered":"2024-03-06T07:29:26.205035+0000","last_clean":"2024-03-06T07:29:26.205035+0000","last_became_active":"2024-03-06T07:29:26.203822+0000","last_became_peered":"2024-03-06T07:29:26.203822+0000","last_unstale":"2024-03-06T07:29:26.205035+0000","last_undegraded":"2024-03-06T07:29:26.205035+0000","last_fullsized":"2024-03-06T07:29:26.205035+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:11.492878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175828+0000","last_change":"2024-03-06T07:29:26.175828+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175828+0000","last_undegraded":"2024-03-06T07:29:26.175828+0000","last_fullsized":"2024-03-06T07:29:26.175828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.202004+0000","last_change":"2024-03-06T07:29:26.194000+0000","last_active":"2024-03-06T07:29:26.202004+0000","last_peered":"2024-03-06T07:29:26.202004+0000","last_clean":"2024-03-06T07:29:26.202004+0000","last_became_active":"2024-03-06T07:29:26.193684+0000","last_became_peered":"2024-03-06T07:29:26.193684+0000","last_unstale":"2024-03-06T07:29:26.202004+0000","last_undegraded":"2024-03-06T07:29:26.202004+0000","last_fullsized":"2024-03-06T07:29:26.202004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:29.750611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171753+0000","last_change":"2024-03-06T07:29:26.171753+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171753+0000","last_undegraded":"2024-03-06T07:29:26.171753+0000","last_fullsized":"2024-03-06T07:29:26.171753+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189388+0000","last_change":"2024-03-06T07:29:26.182870+0000","last_active":"2024-03-06T07:29:26.189388+0000","last_peered":"2024-03-06T07:29:26.189388+0000","last_clean":"2024-03-06T07:29:26.189388+0000","last_became_active":"2024-03-06T07:29:26.182537+0000","last_became_peered":"2024-03-06T07:29:26.182537+0000","last_unstale":"2024-03-06T07:29:26.189388+0000","last_undegraded":"2024-03-06T07:29:26.189388+0000","last_fullsized":"2024-03-06T07:29:26.189388+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:04.448614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.188604+0000","last_change":"2024-03-06T07:29:26.188604+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.188604+0000","last_undegraded":"2024-03-06T07:29:26.188604+0000","last_fullsized":"2024-03-06T07:29:26.188604+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.196320+0000","last_change":"2024-03-06T07:29:26.193099+0000","last_active":"2024-03-06T07:29:26.196320+0000","last_peered":"2024-03-06T07:29:26.196320+0000","last_clean":"2024-03-06T07:29:26.196320+0000","last_became_active":"2024-03-06T07:29:26.192805+0000","last_became_peered":"2024-03-06T07:29:26.192805+0000","last_unstale":"2024-03-06T07:29:26.196320+0000","last_undegraded":"2024-03-06T07:29:26.196320+0000","last_fullsized":"2024-03-06T07:29:26.196320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:43.625964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179348+0000","last_change":"2024-03-06T07:29:26.179348+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179348+0000","last_undegraded":"2024-03-06T07:29:26.179348+0000","last_fullsized":"2024-03-06T07:29:26.179348+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.190038+0000","last_change":"2024-03-06T07:29:26.186625+0000","last_active":"2024-03-06T07:29:26.190038+0000","last_peered":"2024-03-06T07:29:26.190038+0000","last_clean":"2024-03-06T07:29:26.190038+0000","last_became_active":"2024-03-06T07:29:26.186363+0000","last_became_peered":"2024-03-06T07:29:26.186363+0000","last_unstale":"2024-03-06T07:29:26.190038+0000","last_undegraded":"2024-03-06T07:29:26.190038+0000","last_fullsized":"2024-03-06T07:29:26.190038+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:53.612157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.166868+0000","last_change":"2024-03-06T07:29:26.166868+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.166868+0000","last_undegraded":"2024-03-06T07:29:26.166868+0000","last_fullsized":"2024-03-06T07:29:26.166868+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198735+0000","last_change":"2024-03-06T07:29:26.197651+0000","last_active":"2024-03-06T07:29:26.198735+0000","last_peered":"2024-03-06T07:29:26.198735+0000","last_clean":"2024-03-06T07:29:26.198735+0000","last_became_active":"2024-03-06T07:29:26.197161+0000","last_became_peered":"2024-03-06T07:29:26.197161+0000","last_unstale":"2024-03-06T07:29:26.198735+0000","last_undegraded":"2024-03-06T07:29:26.198735+0000","last_fullsized":"2024-03-06T07:29:26.198735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179847+0000","last_change":"2024-03-06T07:29:26.179847+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179847+0000","last_undegraded":"2024-03-06T07:29:26.179847+0000","last_fullsized":"2024-03-06T07:29:26.179847+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.248964+0000","last_change":"2024-03-06T07:29:26.200648+0000","last_active":"2024-03-06T07:29:27.248964+0000","last_peered":"2024-03-06T07:29:27.248964+0000","last_clean":"2024-03-06T07:29:27.248964+0000","last_became_active":"2024-03-06T07:29:26.200365+0000","last_became_peered":"2024-03-06T07:29:26.200365+0000","last_unstale":"2024-03-06T07:29:27.248964+0000","last_undegraded":"2024-03-06T07:29:27.248964+0000","last_fullsized":"2024-03-06T07:29:27.248964+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.411331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.170598+0000","last_change":"2024-03-06T07:29:26.170598+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.170598+0000","last_undegraded":"2024-03-06T07:29:26.170598+0000","last_fullsized":"2024-03-06T07:29:26.170598+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189203+0000","last_change":"2024-03-06T07:29:26.180918+0000","last_active":"2024-03-06T07:29:26.189203+0000","last_peered":"2024-03-06T07:29:26.189203+0000","last_clean":"2024-03-06T07:29:26.189203+0000","last_became_active":"2024-03-06T07:29:26.180570+0000","last_became_peered":"2024-03-06T07:29:26.180570+0000","last_unstale":"2024-03-06T07:29:26.189203+0000","last_undegraded":"2024-03-06T07:29:26.189203+0000","last_fullsized":"2024-03-06T07:29:26.189203+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.869827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175797+0000","last_change":"2024-03-06T07:29:26.175797+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175797+0000","last_undegraded":"2024-03-06T07:29:26.175797+0000","last_fullsized":"2024-03-06T07:29:26.175797+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198013+0000","last_change":"2024-03-06T07:29:26.194437+0000","last_active":"2024-03-06T07:29:26.198013+0000","last_peered":"2024-03-06T07:29:26.198013+0000","last_clean":"2024-03-06T07:29:26.198013+0000","last_became_active":"2024-03-06T07:29:26.194154+0000","last_became_peered":"2024-03-06T07:29:26.194154+0000","last_unstale":"2024-03-06T07:29:26.198013+0000","last_undegraded":"2024-03-06T07:29:26.198013+0000","last_fullsized":"2024-03-06T07:29:26.198013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.163731+0000","last_change":"2024-03-06T07:29:26.163731+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.163731+0000","last_undegraded":"2024-03-06T07:29:26.163731+0000","last_fullsized":"2024-03-06T07:29:26.163731+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192605+0000","last_change":"2024-03-06T07:29:26.185585+0000","last_active":"2024-03-06T07:29:26.192605+0000","last_peered":"2024-03-06T07:29:26.192605+0000","last_clean":"2024-03-06T07:29:26.192605+0000","last_became_active":"2024-03-06T07:29:26.180225+0000","last_became_peered":"2024-03-06T07:29:26.180225+0000","last_unstale":"2024-03-06T07:29:26.192605+0000","last_undegraded":"2024-03-06T07:29:26.192605+0000","last_fullsized":"2024-03-06T07:29:26.192605+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:24.806500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172162+0000","last_change":"2024-03-06T07:29:26.172162+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172162+0000","last_undegraded":"2024-03-06T07:29:26.172162+0000","last_fullsized":"2024-03-06T07:29:26.172162+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.197768+0000","last_change":"2024-03-06T07:29:26.194945+0000","last_active":"2024-03-06T07:29:26.197768+0000","last_peered":"2024-03-06T07:29:26.197768+0000","last_clean":"2024-03-06T07:29:26.197768+0000","last_became_active":"2024-03-06T07:29:26.194577+0000","last_became_peered":"2024-03-06T07:29:26.194577+0000","last_unstale":"2024-03-06T07:29:26.197768+0000","last_undegraded":"2024-03-06T07:29:26.197768+0000","last_fullsized":"2024-03-06T07:29:26.197768+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:30.583334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.228638+0000","last_change":"2024-03-06T07:29:27.217848+0000","last_active":"2024-03-06T07:29:27.228638+0000","last_peered":"2024-03-06T07:29:27.228638+0000","last_clean":"2024-03-06T07:29:27.228638+0000","last_became_active":"2024-03-06T07:29:27.217149+0000","last_became_peered":"2024-03-06T07:29:27.217149+0000","last_unstale":"2024-03-06T07:29:27.228638+0000","last_undegraded":"2024-03-06T07:29:27.228638+0000","last_fullsized":"2024-03-06T07:29:27.228638+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:52.822882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.247655+0000","last_change":"2024-03-06T07:29:26.188532+0000","last_active":"2024-03-06T07:29:27.247655+0000","last_peered":"2024-03-06T07:29:27.247655+0000","last_clean":"2024-03-06T07:29:27.247655+0000","last_became_active":"2024-03-06T07:29:26.188243+0000","last_became_peered":"2024-03-06T07:29:26.188243+0000","last_unstale":"2024-03-06T07:29:27.247655+0000","last_undegraded":"2024-03-06T07:29:27.247655+0000","last_fullsized":"2024-03-06T07:29:27.247655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:05.043428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176443+0000","last_change":"2024-03-06T07:29:26.176443+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176443+0000","last_undegraded":"2024-03-06T07:29:26.176443+0000","last_fullsized":"2024-03-06T07:29:26.176443+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.201430+0000","last_change":"2024-03-06T07:29:26.199005+0000","last_active":"2024-03-06T07:29:26.201430+0000","last_peered":"2024-03-06T07:29:26.201430+0000","last_clean":"2024-03-06T07:29:26.201430+0000","last_became_active":"2024-03-06T07:29:26.196517+0000","last_became_peered":"2024-03-06T07:29:26.196517+0000","last_unstale":"2024-03-06T07:29:26.201430+0000","last_undegraded":"2024-03-06T07:29:26.201430+0000","last_fullsized":"2024-03-06T07:29:26.201430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:45.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172544+0000","last_change":"2024-03-06T07:29:26.172544+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172544+0000","last_undegraded":"2024-03-06T07:29:26.172544+0000","last_fullsized":"2024-03-06T07:29:26.172544+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203430+0000","last_change":"2024-03-06T07:29:26.199717+0000","last_active":"2024-03-06T07:29:26.203430+0000","last_peered":"2024-03-06T07:29:26.203430+0000","last_clean":"2024-03-06T07:29:26.203430+0000","last_became_active":"2024-03-06T07:29:26.199436+0000","last_became_peered":"2024-03-06T07:29:26.199436+0000","last_unstale":"2024-03-06T07:29:26.203430+0000","last_undegraded":"2024-03-06T07:29:26.203430+0000","last_fullsized":"2024-03-06T07:29:26.203430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:58.852154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173271+0000","last_change":"2024-03-06T07:29:26.173271+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173271+0000","last_undegraded":"2024-03-06T07:29:26.173271+0000","last_fullsized":"2024-03-06T07:29:26.173271+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192409+0000","last_change":"2024-03-06T07:29:26.182768+0000","last_active":"2024-03-06T07:29:26.192409+0000","last_peered":"2024-03-06T07:29:26.192409+0000","last_clean":"2024-03-06T07:29:26.192409+0000","last_became_active":"2024-03-06T07:29:26.182489+0000","last_became_peered":"2024-03-06T07:29:26.182489+0000","last_unstale":"2024-03-06T07:29:26.192409+0000","last_undegraded":"2024-03-06T07:29:26.192409+0000","last_fullsized":"2024-03-06T07:29:26.192409+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.422880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164678+0000","last_change":"2024-03-06T07:29:26.164678+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164678+0000","last_undegraded":"2024-03-06T07:29:26.164678+0000","last_fullsized":"2024-03-06T07:29:26.164678+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192648+0000","last_change":"2024-03-06T07:29:26.188395+0000","last_active":"2024-03-06T07:29:26.192648+0000","last_peered":"2024-03-06T07:29:26.192648+0000","last_clean":"2024-03-06T07:29:26.192648+0000","last_became_active":"2024-03-06T07:29:26.187669+0000","last_became_peered":"2024-03-06T07:29:26.187669+0000","last_unstale":"2024-03-06T07:29:26.192648+0000","last_undegraded":"2024-03-06T07:29:26.192648+0000","last_fullsized":"2024-03-06T07:29:26.192648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:12.640396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.228102+0000","last_change":"2024-03-06T07:29:27.221492+0000","last_active":"2024-03-06T07:29:27.228102+0000","last_peered":"2024-03-06T07:29:27.228102+0000","last_clean":"2024-03-06T07:29:27.228102+0000","last_became_active":"2024-03-06T07:29:27.221203+0000","last_became_peered":"2024-03-06T07:29:27.221203+0000","last_unstale":"2024-03-06T07:29:27.228102+0000","last_undegraded":"2024-03-06T07:29:27.228102+0000","last_fullsized":"2024-03-06T07:29:27.228102+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:21.674792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203718+0000","last_change":"2024-03-06T07:29:26.200840+0000","last_active":"2024-03-06T07:29:26.203718+0000","last_peered":"2024-03-06T07:29:26.203718+0000","last_clean":"2024-03-06T07:29:26.203718+0000","last_became_active":"2024-03-06T07:29:26.200577+0000","last_became_peered":"2024-03-06T07:29:26.200577+0000","last_unstale":"2024-03-06T07:29:26.203718+0000","last_undegraded":"2024-03-06T07:29:26.203718+0000","last_fullsized":"2024-03-06T07:29:26.203718+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:40.088324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.232958+0000","last_change":"2024-03-06T07:29:27.226841+0000","last_active":"2024-03-06T07:29:27.232958+0000","last_peered":"2024-03-06T07:29:27.232958+0000","last_clean":"2024-03-06T07:29:27.232958+0000","last_became_active":"2024-03-06T07:29:27.226120+0000","last_became_peered":"2024-03-06T07:29:27.226120+0000","last_unstale":"2024-03-06T07:29:27.232958+0000","last_undegraded":"2024-03-06T07:29:27.232958+0000","last_fullsized":"2024-03-06T07:29:27.232958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:22.288331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.195993+0000","last_change":"2024-03-06T07:29:26.191147+0000","last_active":"2024-03-06T07:29:26.195993+0000","last_peered":"2024-03-06T07:29:26.195993+0000","last_clean":"2024-03-06T07:29:26.195993+0000","last_became_active":"2024-03-06T07:29:26.190641+0000","last_became_peered":"2024-03-06T07:29:26.190641+0000","last_unstale":"2024-03-06T07:29:26.195993+0000","last_undegraded":"2024-03-06T07:29:26.195993+0000","last_fullsized":"2024-03-06T07:29:26.195993+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:52.210630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.177373+0000","last_change":"2024-03-06T07:29:26.177373+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.177373+0000","last_undegraded":"2024-03-06T07:29:26.177373+0000","last_fullsized":"2024-03-06T07:29:26.177373+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198049+0000","last_change":"2024-03-06T07:29:26.193713+0000","last_active":"2024-03-06T07:29:26.198049+0000","last_peered":"2024-03-06T07:29:26.198049+0000","last_clean":"2024-03-06T07:29:26.198049+0000","last_became_active":"2024-03-06T07:29:26.193389+0000","last_became_peered":"2024-03-06T07:29:26.193389+0000","last_unstale":"2024-03-06T07:29:26.198049+0000","last_undegraded":"2024-03-06T07:29:26.198049+0000","last_fullsized":"2024-03-06T07:29:26.198049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:58.185331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172598+0000","last_change":"2024-03-06T07:29:26.172598+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172598+0000","last_undegraded":"2024-03-06T07:29:26.172598+0000","last_fullsized":"2024-03-06T07:29:26.172598+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198317+0000","last_change":"2024-03-06T07:29:26.196747+0000","last_active":"2024-03-06T07:29:26.198317+0000","last_peered":"2024-03-06T07:29:26.198317+0000","last_clean":"2024-03-06T07:29:26.198317+0000","last_became_active":"2024-03-06T07:29:26.196446+0000","last_became_peered":"2024-03-06T07:29:26.196446+0000","last_unstale":"2024-03-06T07:29:26.198317+0000","last_undegraded":"2024-03-06T07:29:26.198317+0000","last_fullsized":"2024-03-06T07:29:26.198317+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:18.921700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176362+0000","last_change":"2024-03-06T07:29:26.176362+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176362+0000","last_undegraded":"2024-03-06T07:29:26.176362+0000","last_fullsized":"2024-03-06T07:29:26.176362+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.200181+0000","last_change":"2024-03-06T07:29:26.199027+0000","last_active":"2024-03-06T07:29:26.200181+0000","last_peered":"2024-03-06T07:29:26.200181+0000","last_clean":"2024-03-06T07:29:26.200181+0000","last_became_active":"2024-03-06T07:29:26.198748+0000","last_became_peered":"2024-03-06T07:29:26.198748+0000","last_unstale":"2024-03-06T07:29:26.200181+0000","last_undegraded":"2024-03-06T07:29:26.200181+0000","last_fullsized":"2024-03-06T07:29:26.200181+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:42.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.174000+0000","last_change":"2024-03-06T07:29:26.174000+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.174000+0000","last_undegraded":"2024-03-06T07:29:26.174000+0000","last_fullsized":"2024-03-06T07:29:26.174000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.221797+0000","last_change":"2024-03-06T07:29:26.196655+0000","last_active":"2024-03-06T07:29:27.221797+0000","last_peered":"2024-03-06T07:29:27.221797+0000","last_clean":"2024-03-06T07:29:27.221797+0000","last_became_active":"2024-03-06T07:29:26.196370+0000","last_became_peered":"2024-03-06T07:29:26.196370+0000","last_unstale":"2024-03-06T07:29:27.221797+0000","last_undegraded":"2024-03-06T07:29:27.221797+0000","last_fullsized":"2024-03-06T07:29:27.221797+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.163810+0000","last_change":"2024-03-06T07:29:26.163810+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.163810+0000","last_undegraded":"2024-03-06T07:29:26.163810+0000","last_fullsized":"2024-03-06T07:29:26.163810+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198276+0000","last_change":"2024-03-06T07:29:26.195806+0000","last_active":"2024-03-06T07:29:26.198276+0000","last_peered":"2024-03-06T07:29:26.198276+0000","last_clean":"2024-03-06T07:29:26.198276+0000","last_became_active":"2024-03-06T07:29:26.195455+0000","last_became_peered":"2024-03-06T07:29:26.195455+0000","last_unstale":"2024-03-06T07:29:26.198276+0000","last_undegraded":"2024-03-06T07:29:26.198276+0000","last_fullsized":"2024-03-06T07:29:26.198276+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:20.660770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164058+0000","last_change":"2024-03-06T07:29:26.164058+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164058+0000","last_undegraded":"2024-03-06T07:29:26.164058+0000","last_fullsized":"2024-03-06T07:29:26.164058+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.201349+0000","last_change":"2024-03-06T07:29:26.199356+0000","last_active":"2024-03-06T07:29:26.201349+0000","last_peered":"2024-03-06T07:29:26.201349+0000","last_clean":"2024-03-06T07:29:26.201349+0000","last_became_active":"2024-03-06T07:29:26.199015+0000","last_became_peered":"2024-03-06T07:29:26.199015+0000","last_unstale":"2024-03-06T07:29:26.201349+0000","last_undegraded":"2024-03-06T07:29:26.201349+0000","last_fullsized":"2024-03-06T07:29:26.201349+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:21.561181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.178828+0000","last_change":"2024-03-06T07:29:26.178828+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.178828+0000","last_undegraded":"2024-03-06T07:29:26.178828+0000","last_fullsized":"2024-03-06T07:29:26.178828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.220161+0000","last_change":"2024-03-06T07:29:26.184910+0000","last_active":"2024-03-06T07:29:27.220161+0000","last_peered":"2024-03-06T07:29:27.220161+0000","last_clean":"2024-03-06T07:29:27.220161+0000","last_became_active":"2024-03-06T07:29:26.183588+0000","last_became_peered":"2024-03-06T07:29:26.183588+0000","last_unstale":"2024-03-06T07:29:27.220161+0000","last_undegraded":"2024-03-06T07:29:27.220161+0000","last_fullsized":"2024-03-06T07:29:27.220161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179731+0000","last_change":"2024-03-06T07:29:26.179731+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179731+0000","last_undegraded":"2024-03-06T07:29:26.179731+0000","last_fullsized":"2024-03-06T07:29:26.179731+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189832+0000","last_change":"2024-03-06T07:29:26.185660+0000","last_active":"2024-03-06T07:29:26.189832+0000","last_peered":"2024-03-06T07:29:26.189832+0000","last_clean":"2024-03-06T07:29:26.189832+0000","last_became_active":"2024-03-06T07:29:26.185316+0000","last_became_peered":"2024-03-06T07:29:26.185316+0000","last_unstale":"2024-03-06T07:29:26.189832+0000","last_undegraded":"2024-03-06T07:29:26.189832+0000","last_fullsized":"2024-03-06T07:29:26.189832+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:25.384746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175672+0000","last_change":"2024-03-06T07:29:26.175672+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175672+0000","last_undegraded":"2024-03-06T07:29:26.175672+0000","last_fullsized":"2024-03-06T07:29:26.175672+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203496+0000","last_change":"2024-03-06T07:29:26.200006+0000","last_active":"2024-03-06T07:29:26.203496+0000","last_peered":"2024-03-06T07:29:26.203496+0000","last_clean":"2024-03-06T07:29:26.203496+0000","last_became_active":"2024-03-06T07:29:26.198345+0000","last_became_peered":"2024-03-06T07:29:26.198345+0000","last_unstale":"2024-03-06T07:29:26.203496+0000","last_undegraded":"2024-03-06T07:29:26.203496+0000","last_fullsized":"2024-03-06T07:29:26.203496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.255706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":209,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.184437+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:29:26.184437+0000","last_peered":"2024-03-06T07:29:26.184437+0000","last_clean":"2024-03-06T07:29:26.184437+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:29:26.184437+0000","last_undegraded":"2024-03-06T07:29:26.184437+0000","last_fullsized":"2024-03-06T07:29:26.184437+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509468,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509469,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335436,"statfs":{"total":96636764160,"available":96599486464,"internally_reserved":0,"allocated":1089536,"data_stored":788026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542173,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":197658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":794293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:31.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:31.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.938+0000 7f0137200640 1 Processor -- start 2024-03-06T07:29:31.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.938+0000 7f0137200640 1 -- start start 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f0137200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01380cd060 0x7f01381667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f0137200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380714d0 con 0x7f0138070f10 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f0137200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380717b0 con 0x7f0138145300 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f0137200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0138071a90 con 0x7f01380cd060 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f0136800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f01380cd060 0x7f01381667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f012fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0138070f10 0x7f01380cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57560/0 (socket says 172.21.15.6:57560) 2024-03-06T07:29:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.940+0000 7f012fe00640 1 -- 172.21.15.6:0/1279060730 learned_addr learned my addr 172.21.15.6:0/1279060730 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:31.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.941+0000 7f0135400640 1 -- 172.21.15.6:0/1279060730 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297135635 0 0) 0x7f01380714d0 con 0x7f0138070f10 2024-03-06T07:29:31.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.941+0000 7f0136800640 1 -- 172.21.15.6:0/1279060730 >> v1:172.21.15.178:6789/0 conn(0x7f0138145300 legacy=0x7f01380b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.941+0000 7f0136800640 1 -- 172.21.15.6:0/1279060730 >> v1:172.21.15.6:6789/0 conn(0x7f0138070f10 legacy=0x7f01380cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.941+0000 7f0136800640 1 -- 172.21.15.6:0/1279060730 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0138071d70 con 0x7f01380cd060 2024-03-06T07:29:31.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.942+0000 7f0136800640 1 --2- 172.21.15.6:0/1279060730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01380cd060 0x7f01381667e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0130009d90 tx=0x7f013002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba847f0fb78800e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0135400640 1 -- 172.21.15.6:0/1279060730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130039070 con 0x7f01380cd060 2024-03-06T07:29:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0135400640 1 -- 172.21.15.6:0/1279060730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013002cc60 con 0x7f01380cd060 2024-03-06T07:29:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0135400640 1 -- 172.21.15.6:0/1279060730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130034660 con 0x7f01380cd060 2024-03-06T07:29:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0137200640 1 -- 172.21.15.6:0/1279060730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01380cd060 msgr2=0x7f01381667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0137200640 1 --2- 172.21.15.6:0/1279060730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01380cd060 0x7f01381667e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0130009d90 tx=0x7f013002beb0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0137200640 1 -- 172.21.15.6:0/1279060730 shutdown_connections 2024-03-06T07:29:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0137200640 1 --2- 172.21.15.6:0/1279060730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01380cd060 0x7f01381667e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0137200640 1 -- 172.21.15.6:0/1279060730 >> 172.21.15.6:0/1279060730 conn(0x7f013806b3d0 msgr2=0x7f013806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.943+0000 7f0137200640 1 -- 172.21.15.6:0/1279060730 shutdown_connections 2024-03-06T07:29:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.944+0000 7f0137200640 1 -- 172.21.15.6:0/1279060730 wait complete. 2024-03-06T07:29:31.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.945+0000 7f0137200640 1 Processor -- start 2024-03-06T07:29:31.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.945+0000 7f0137200640 1 -- start start 2024-03-06T07:29:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.946+0000 7f0137200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 0x7f013815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.946+0000 7f0137200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01380cd060 0x7f01381e57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.946+0000 7f0137200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0138145300 0x7f0138142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0137200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013815d700 con 0x7f0138070f10 2024-03-06T07:29:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0137200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013815d580 con 0x7f0138145300 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0137200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013815d880 con 0x7f01380cd060 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 0x7f013815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 0x7f013815b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43216/0 (socket says 172.21.15.6:43216) 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 -- 172.21.15.6:0/3146133280 learned_addr learned my addr 172.21.15.6:0/3146133280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f012fe00640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01380cd060 0x7f01381e57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 -- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01380cd060 msgr2=0x7f01381e57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0136800640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0138145300 0x7f0138142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01380cd060 0x7f01381e57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 -- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0138145300 msgr2=0x7f0138142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0138145300 0x7f0138142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.947+0000 7f0135e00640 1 -- 172.21.15.6:0/3146133280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01381e4420 con 0x7f0138070f10 2024-03-06T07:29:31.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.948+0000 7f0135e00640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 0x7f013815b6f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f012800c430 tx=0x7f012800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.948+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128017070 con 0x7f0138070f10 2024-03-06T07:29:31.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.948+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01280052d0 con 0x7f0138070f10 2024-03-06T07:29:31.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.948+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01381e4700 con 0x7f0138070f10 2024-03-06T07:29:31.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.948+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0128003830 con 0x7f0138070f10 2024-03-06T07:29:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.948+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01381e4cc0 con 0x7f0138070f10 2024-03-06T07:29:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.949+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0138140b20 con 0x7f0138070f10 2024-03-06T07:29:31.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.950+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f012801a070 con 0x7f0138070f10 2024-03-06T07:29:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.951+0000 7f012e000640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f011406b4d0 0x7f011406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.951+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01280896c0 con 0x7f0138070f10 2024-03-06T07:29:31.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.952+0000 7f012fe00640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f011406b4d0 0x7f011406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.953+0000 7f012fe00640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f011406b4d0 0x7f011406d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f01381f74a0 tx=0x7f0120007bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:31.956+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0138140b20 con 0x7f0138070f10 2024-03-06T07:29:32.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.151+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f013804f330 con 0x7f011406b4d0 2024-03-06T07:29:32.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.182+0000 7f012e000640 1 -- 172.21.15.6:0/3146133280 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247401 (secure 0 0 0) 0x7f013804f330 con 0x7f011406b4d0 2024-03-06T07:29:32.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.188+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f011406b4d0 msgr2=0x7f011406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.188+0000 7f0137200640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f011406b4d0 0x7f011406d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f01381f74a0 tx=0x7f0120007bb0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.188+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 msgr2=0x7f013815b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.188+0000 7f0137200640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 0x7f013815b6f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f012800c430 tx=0x7f012800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 shutdown_connections 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f011406b4d0 0x7f011406d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0138145300 0x7f0138142ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01380cd060 0x7f01381e57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 --2- 172.21.15.6:0/3146133280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0138070f10 0x7f013815b6f0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 >> 172.21.15.6:0/3146133280 conn(0x7f013806b3d0 msgr2=0x7f013815eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 shutdown_connections 2024-03-06T07:29:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.189+0000 7f0137200640 1 -- 172.21.15.6:0/3146133280 wait complete. 2024-03-06T07:29:32.213 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-03-06T07:29:31.480064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":169,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254568,"kb_used_data":5104,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720152,"statfs":{"total":773094113280,"available":772833435648,"internally_reserved":0,"allocated":5226496,"data_stored":2799602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001807"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180481+0000","last_change":"2024-03-06T07:29:26.180481+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180481+0000","last_undegraded":"2024-03-06T07:29:26.180481+0000","last_fullsized":"2024-03-06T07:29:26.180481+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.185266+0000","last_change":"2024-03-06T07:29:26.185266+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.185266+0000","last_undegraded":"2024-03-06T07:29:26.185266+0000","last_fullsized":"2024-03-06T07:29:26.185266+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.186354+0000","last_change":"2024-03-06T07:29:26.186354+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.186354+0000","last_undegraded":"2024-03-06T07:29:26.186354+0000","last_fullsized":"2024-03-06T07:29:26.186354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180082+0000","last_change":"2024-03-06T07:29:26.180082+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180082+0000","last_undegraded":"2024-03-06T07:29:26.180082+0000","last_fullsized":"2024-03-06T07:29:26.180082+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179026+0000","last_change":"2024-03-06T07:29:26.179026+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179026+0000","last_undegraded":"2024-03-06T07:29:26.179026+0000","last_fullsized":"2024-03-06T07:29:26.179026+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.165169+0000","last_change":"2024-03-06T07:29:26.165169+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.165169+0000","last_undegraded":"2024-03-06T07:29:26.165169+0000","last_fullsized":"2024-03-06T07:29:26.165169+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173991+0000","last_change":"2024-03-06T07:29:26.173991+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173991+0000","last_undegraded":"2024-03-06T07:29:26.173991+0000","last_fullsized":"2024-03-06T07:29:26.173991+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179661+0000","last_change":"2024-03-06T07:29:26.179661+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179661+0000","last_undegraded":"2024-03-06T07:29:26.179661+0000","last_fullsized":"2024-03-06T07:29:26.179661+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179688+0000","last_change":"2024-03-06T07:29:26.179688+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179688+0000","last_undegraded":"2024-03-06T07:29:26.179688+0000","last_fullsized":"2024-03-06T07:29:26.179688+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179130+0000","last_change":"2024-03-06T07:29:26.179130+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179130+0000","last_undegraded":"2024-03-06T07:29:26.179130+0000","last_fullsized":"2024-03-06T07:29:26.179130+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.177134+0000","last_change":"2024-03-06T07:29:26.177134+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.177134+0000","last_undegraded":"2024-03-06T07:29:26.177134+0000","last_fullsized":"2024-03-06T07:29:26.177134+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.186917+0000","last_change":"2024-03-06T07:29:26.186917+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.186917+0000","last_undegraded":"2024-03-06T07:29:26.186917+0000","last_fullsized":"2024-03-06T07:29:26.186917+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.236195+0000","last_change":"2024-03-06T07:29:27.234158+0000","last_active":"2024-03-06T07:29:27.236195+0000","last_peered":"2024-03-06T07:29:27.236195+0000","last_clean":"2024-03-06T07:29:27.236195+0000","last_became_active":"2024-03-06T07:29:27.233803+0000","last_became_peered":"2024-03-06T07:29:27.233803+0000","last_unstale":"2024-03-06T07:29:27.236195+0000","last_undegraded":"2024-03-06T07:29:27.236195+0000","last_fullsized":"2024-03-06T07:29:27.236195+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:56.608802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180437+0000","last_change":"2024-03-06T07:29:26.180437+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180437+0000","last_undegraded":"2024-03-06T07:29:26.180437+0000","last_fullsized":"2024-03-06T07:29:26.180437+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.167433+0000","last_change":"2024-03-06T07:29:26.167433+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.167433+0000","last_undegraded":"2024-03-06T07:29:26.167433+0000","last_fullsized":"2024-03-06T07:29:26.167433+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172611+0000","last_change":"2024-03-06T07:29:26.172611+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172611+0000","last_undegraded":"2024-03-06T07:29:26.172611+0000","last_fullsized":"2024-03-06T07:29:26.172611+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.187650+0000","last_change":"2024-03-06T07:29:26.187650+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.187650+0000","last_undegraded":"2024-03-06T07:29:26.187650+0000","last_fullsized":"2024-03-06T07:29:26.187650+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.189091+0000","last_change":"2024-03-06T07:29:26.189091+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.189091+0000","last_undegraded":"2024-03-06T07:29:26.189091+0000","last_fullsized":"2024-03-06T07:29:26.189091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171250+0000","last_change":"2024-03-06T07:29:26.171250+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171250+0000","last_undegraded":"2024-03-06T07:29:26.171250+0000","last_fullsized":"2024-03-06T07:29:26.171250+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.237778+0000","last_change":"2024-03-06T07:29:27.232657+0000","last_active":"2024-03-06T07:29:27.237778+0000","last_peered":"2024-03-06T07:29:27.237778+0000","last_clean":"2024-03-06T07:29:27.237778+0000","last_became_active":"2024-03-06T07:29:27.232358+0000","last_became_peered":"2024-03-06T07:29:27.232358+0000","last_unstale":"2024-03-06T07:29:27.237778+0000","last_undegraded":"2024-03-06T07:29:27.237778+0000","last_fullsized":"2024-03-06T07:29:27.237778+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:14.573067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164817+0000","last_change":"2024-03-06T07:29:26.164817+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164817+0000","last_undegraded":"2024-03-06T07:29:26.164817+0000","last_fullsized":"2024-03-06T07:29:26.164817+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176355+0000","last_change":"2024-03-06T07:29:26.176355+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176355+0000","last_undegraded":"2024-03-06T07:29:26.176355+0000","last_fullsized":"2024-03-06T07:29:26.176355+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.170468+0000","last_change":"2024-03-06T07:29:26.170468+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.170468+0000","last_undegraded":"2024-03-06T07:29:26.170468+0000","last_fullsized":"2024-03-06T07:29:26.170468+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171475+0000","last_change":"2024-03-06T07:29:26.171475+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171475+0000","last_undegraded":"2024-03-06T07:29:26.171475+0000","last_fullsized":"2024-03-06T07:29:26.171475+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.174866+0000","last_change":"2024-03-06T07:29:26.174866+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.174866+0000","last_undegraded":"2024-03-06T07:29:26.174866+0000","last_fullsized":"2024-03-06T07:29:26.174866+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.236949+0000","last_change":"2024-03-06T07:29:27.234582+0000","last_active":"2024-03-06T07:29:27.236949+0000","last_peered":"2024-03-06T07:29:27.236949+0000","last_clean":"2024-03-06T07:29:27.236949+0000","last_became_active":"2024-03-06T07:29:27.233810+0000","last_became_peered":"2024-03-06T07:29:27.233810+0000","last_unstale":"2024-03-06T07:29:27.236949+0000","last_undegraded":"2024-03-06T07:29:27.236949+0000","last_fullsized":"2024-03-06T07:29:27.236949+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:44.382794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.184051+0000","last_change":"2024-03-06T07:29:26.184051+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.184051+0000","last_undegraded":"2024-03-06T07:29:26.184051+0000","last_fullsized":"2024-03-06T07:29:26.184051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.183973+0000","last_change":"2024-03-06T07:29:26.183973+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.183973+0000","last_undegraded":"2024-03-06T07:29:26.183973+0000","last_fullsized":"2024-03-06T07:29:26.183973+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171100+0000","last_change":"2024-03-06T07:29:26.171100+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171100+0000","last_undegraded":"2024-03-06T07:29:26.171100+0000","last_fullsized":"2024-03-06T07:29:26.171100+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.182982+0000","last_change":"2024-03-06T07:29:26.182982+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.182982+0000","last_undegraded":"2024-03-06T07:29:26.182982+0000","last_fullsized":"2024-03-06T07:29:26.182982+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.166901+0000","last_change":"2024-03-06T07:29:26.166901+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.166901+0000","last_undegraded":"2024-03-06T07:29:26.166901+0000","last_fullsized":"2024-03-06T07:29:26.166901+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.233068+0000","last_change":"2024-03-06T07:29:27.227701+0000","last_active":"2024-03-06T07:29:27.233068+0000","last_peered":"2024-03-06T07:29:27.233068+0000","last_clean":"2024-03-06T07:29:27.233068+0000","last_became_active":"2024-03-06T07:29:27.225595+0000","last_became_peered":"2024-03-06T07:29:27.225595+0000","last_unstale":"2024-03-06T07:29:27.233068+0000","last_undegraded":"2024-03-06T07:29:27.233068+0000","last_fullsized":"2024-03-06T07:29:27.233068+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:55.302360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.227914+0000","last_change":"2024-03-06T07:29:27.222401+0000","last_active":"2024-03-06T07:29:27.227914+0000","last_peered":"2024-03-06T07:29:27.227914+0000","last_clean":"2024-03-06T07:29:27.227914+0000","last_became_active":"2024-03-06T07:29:27.221978+0000","last_became_peered":"2024-03-06T07:29:27.221978+0000","last_unstale":"2024-03-06T07:29:27.227914+0000","last_undegraded":"2024-03-06T07:29:27.227914+0000","last_fullsized":"2024-03-06T07:29:27.227914+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:43.908483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192462+0000","last_change":"2024-03-06T07:29:26.184227+0000","last_active":"2024-03-06T07:29:26.192462+0000","last_peered":"2024-03-06T07:29:26.192462+0000","last_clean":"2024-03-06T07:29:26.192462+0000","last_became_active":"2024-03-06T07:29:26.183923+0000","last_became_peered":"2024-03-06T07:29:26.183923+0000","last_unstale":"2024-03-06T07:29:26.192462+0000","last_undegraded":"2024-03-06T07:29:26.192462+0000","last_fullsized":"2024-03-06T07:29:26.192462+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:08.181893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173821+0000","last_change":"2024-03-06T07:29:26.173821+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173821+0000","last_undegraded":"2024-03-06T07:29:26.173821+0000","last_fullsized":"2024-03-06T07:29:26.173821+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.202304+0000","last_change":"2024-03-06T07:29:26.200719+0000","last_active":"2024-03-06T07:29:26.202304+0000","last_peered":"2024-03-06T07:29:26.202304+0000","last_clean":"2024-03-06T07:29:26.202304+0000","last_became_active":"2024-03-06T07:29:26.200438+0000","last_became_peered":"2024-03-06T07:29:26.200438+0000","last_unstale":"2024-03-06T07:29:26.202304+0000","last_undegraded":"2024-03-06T07:29:26.202304+0000","last_fullsized":"2024-03-06T07:29:26.202304+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:27.927651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.183386+0000","last_change":"2024-03-06T07:29:26.183386+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.183386+0000","last_undegraded":"2024-03-06T07:29:26.183386+0000","last_fullsized":"2024-03-06T07:29:26.183386+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.243346+0000","last_change":"2024-03-06T07:29:26.201813+0000","last_active":"2024-03-06T07:29:27.243346+0000","last_peered":"2024-03-06T07:29:27.243346+0000","last_clean":"2024-03-06T07:29:27.243346+0000","last_became_active":"2024-03-06T07:29:26.201360+0000","last_became_peered":"2024-03-06T07:29:26.201360+0000","last_unstale":"2024-03-06T07:29:27.243346+0000","last_undegraded":"2024-03-06T07:29:27.243346+0000","last_fullsized":"2024-03-06T07:29:27.243346+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.232720+0000","last_change":"2024-03-06T07:29:27.231046+0000","last_active":"2024-03-06T07:29:27.232720+0000","last_peered":"2024-03-06T07:29:27.232720+0000","last_clean":"2024-03-06T07:29:27.232720+0000","last_became_active":"2024-03-06T07:29:27.230774+0000","last_became_peered":"2024-03-06T07:29:27.230774+0000","last_unstale":"2024-03-06T07:29:27.232720+0000","last_undegraded":"2024-03-06T07:29:27.232720+0000","last_fullsized":"2024-03-06T07:29:27.232720+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:29.573395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.204067+0000","last_change":"2024-03-06T07:29:26.202950+0000","last_active":"2024-03-06T07:29:26.204067+0000","last_peered":"2024-03-06T07:29:26.204067+0000","last_clean":"2024-03-06T07:29:26.204067+0000","last_became_active":"2024-03-06T07:29:26.202564+0000","last_became_peered":"2024-03-06T07:29:26.202564+0000","last_unstale":"2024-03-06T07:29:26.204067+0000","last_undegraded":"2024-03-06T07:29:26.204067+0000","last_fullsized":"2024-03-06T07:29:26.204067+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.182043+0000","last_change":"2024-03-06T07:29:26.182043+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.182043+0000","last_undegraded":"2024-03-06T07:29:26.182043+0000","last_fullsized":"2024-03-06T07:29:26.182043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192624+0000","last_change":"2024-03-06T07:29:26.188354+0000","last_active":"2024-03-06T07:29:26.192624+0000","last_peered":"2024-03-06T07:29:26.192624+0000","last_clean":"2024-03-06T07:29:26.192624+0000","last_became_active":"2024-03-06T07:29:26.187378+0000","last_became_peered":"2024-03-06T07:29:26.187378+0000","last_unstale":"2024-03-06T07:29:26.192624+0000","last_undegraded":"2024-03-06T07:29:26.192624+0000","last_fullsized":"2024-03-06T07:29:26.192624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:41.649859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.178981+0000","last_change":"2024-03-06T07:29:26.178981+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.178981+0000","last_undegraded":"2024-03-06T07:29:26.178981+0000","last_fullsized":"2024-03-06T07:29:26.178981+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.205035+0000","last_change":"2024-03-06T07:29:26.204279+0000","last_active":"2024-03-06T07:29:26.205035+0000","last_peered":"2024-03-06T07:29:26.205035+0000","last_clean":"2024-03-06T07:29:26.205035+0000","last_became_active":"2024-03-06T07:29:26.203822+0000","last_became_peered":"2024-03-06T07:29:26.203822+0000","last_unstale":"2024-03-06T07:29:26.205035+0000","last_undegraded":"2024-03-06T07:29:26.205035+0000","last_fullsized":"2024-03-06T07:29:26.205035+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:11.492878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175828+0000","last_change":"2024-03-06T07:29:26.175828+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175828+0000","last_undegraded":"2024-03-06T07:29:26.175828+0000","last_fullsized":"2024-03-06T07:29:26.175828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.202004+0000","last_change":"2024-03-06T07:29:26.194000+0000","last_active":"2024-03-06T07:29:26.202004+0000","last_peered":"2024-03-06T07:29:26.202004+0000","last_clean":"2024-03-06T07:29:26.202004+0000","last_became_active":"2024-03-06T07:29:26.193684+0000","last_became_peered":"2024-03-06T07:29:26.193684+0000","last_unstale":"2024-03-06T07:29:26.202004+0000","last_undegraded":"2024-03-06T07:29:26.202004+0000","last_fullsized":"2024-03-06T07:29:26.202004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:29.750611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171753+0000","last_change":"2024-03-06T07:29:26.171753+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171753+0000","last_undegraded":"2024-03-06T07:29:26.171753+0000","last_fullsized":"2024-03-06T07:29:26.171753+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189388+0000","last_change":"2024-03-06T07:29:26.182870+0000","last_active":"2024-03-06T07:29:26.189388+0000","last_peered":"2024-03-06T07:29:26.189388+0000","last_clean":"2024-03-06T07:29:26.189388+0000","last_became_active":"2024-03-06T07:29:26.182537+0000","last_became_peered":"2024-03-06T07:29:26.182537+0000","last_unstale":"2024-03-06T07:29:26.189388+0000","last_undegraded":"2024-03-06T07:29:26.189388+0000","last_fullsized":"2024-03-06T07:29:26.189388+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:04.448614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.188604+0000","last_change":"2024-03-06T07:29:26.188604+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.188604+0000","last_undegraded":"2024-03-06T07:29:26.188604+0000","last_fullsized":"2024-03-06T07:29:26.188604+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.196320+0000","last_change":"2024-03-06T07:29:26.193099+0000","last_active":"2024-03-06T07:29:26.196320+0000","last_peered":"2024-03-06T07:29:26.196320+0000","last_clean":"2024-03-06T07:29:26.196320+0000","last_became_active":"2024-03-06T07:29:26.192805+0000","last_became_peered":"2024-03-06T07:29:26.192805+0000","last_unstale":"2024-03-06T07:29:26.196320+0000","last_undegraded":"2024-03-06T07:29:26.196320+0000","last_fullsized":"2024-03-06T07:29:26.196320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:43.625964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179348+0000","last_change":"2024-03-06T07:29:26.179348+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179348+0000","last_undegraded":"2024-03-06T07:29:26.179348+0000","last_fullsized":"2024-03-06T07:29:26.179348+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.190038+0000","last_change":"2024-03-06T07:29:26.186625+0000","last_active":"2024-03-06T07:29:26.190038+0000","last_peered":"2024-03-06T07:29:26.190038+0000","last_clean":"2024-03-06T07:29:26.190038+0000","last_became_active":"2024-03-06T07:29:26.186363+0000","last_became_peered":"2024-03-06T07:29:26.186363+0000","last_unstale":"2024-03-06T07:29:26.190038+0000","last_undegraded":"2024-03-06T07:29:26.190038+0000","last_fullsized":"2024-03-06T07:29:26.190038+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:53.612157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.166868+0000","last_change":"2024-03-06T07:29:26.166868+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.166868+0000","last_undegraded":"2024-03-06T07:29:26.166868+0000","last_fullsized":"2024-03-06T07:29:26.166868+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198735+0000","last_change":"2024-03-06T07:29:26.197651+0000","last_active":"2024-03-06T07:29:26.198735+0000","last_peered":"2024-03-06T07:29:26.198735+0000","last_clean":"2024-03-06T07:29:26.198735+0000","last_became_active":"2024-03-06T07:29:26.197161+0000","last_became_peered":"2024-03-06T07:29:26.197161+0000","last_unstale":"2024-03-06T07:29:26.198735+0000","last_undegraded":"2024-03-06T07:29:26.198735+0000","last_fullsized":"2024-03-06T07:29:26.198735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179847+0000","last_change":"2024-03-06T07:29:26.179847+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179847+0000","last_undegraded":"2024-03-06T07:29:26.179847+0000","last_fullsized":"2024-03-06T07:29:26.179847+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.248964+0000","last_change":"2024-03-06T07:29:26.200648+0000","last_active":"2024-03-06T07:29:27.248964+0000","last_peered":"2024-03-06T07:29:27.248964+0000","last_clean":"2024-03-06T07:29:27.248964+0000","last_became_active":"2024-03-06T07:29:26.200365+0000","last_became_peered":"2024-03-06T07:29:26.200365+0000","last_unstale":"2024-03-06T07:29:27.248964+0000","last_undegraded":"2024-03-06T07:29:27.248964+0000","last_fullsized":"2024-03-06T07:29:27.248964+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.411331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.170598+0000","last_change":"2024-03-06T07:29:26.170598+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.170598+0000","last_undegraded":"2024-03-06T07:29:26.170598+0000","last_fullsized":"2024-03-06T07:29:26.170598+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189203+0000","last_change":"2024-03-06T07:29:26.180918+0000","last_active":"2024-03-06T07:29:26.189203+0000","last_peered":"2024-03-06T07:29:26.189203+0000","last_clean":"2024-03-06T07:29:26.189203+0000","last_became_active":"2024-03-06T07:29:26.180570+0000","last_became_peered":"2024-03-06T07:29:26.180570+0000","last_unstale":"2024-03-06T07:29:26.189203+0000","last_undegraded":"2024-03-06T07:29:26.189203+0000","last_fullsized":"2024-03-06T07:29:26.189203+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.869827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175797+0000","last_change":"2024-03-06T07:29:26.175797+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175797+0000","last_undegraded":"2024-03-06T07:29:26.175797+0000","last_fullsized":"2024-03-06T07:29:26.175797+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198013+0000","last_change":"2024-03-06T07:29:26.194437+0000","last_active":"2024-03-06T07:29:26.198013+0000","last_peered":"2024-03-06T07:29:26.198013+0000","last_clean":"2024-03-06T07:29:26.198013+0000","last_became_active":"2024-03-06T07:29:26.194154+0000","last_became_peered":"2024-03-06T07:29:26.194154+0000","last_unstale":"2024-03-06T07:29:26.198013+0000","last_undegraded":"2024-03-06T07:29:26.198013+0000","last_fullsized":"2024-03-06T07:29:26.198013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.163731+0000","last_change":"2024-03-06T07:29:26.163731+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.163731+0000","last_undegraded":"2024-03-06T07:29:26.163731+0000","last_fullsized":"2024-03-06T07:29:26.163731+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192605+0000","last_change":"2024-03-06T07:29:26.185585+0000","last_active":"2024-03-06T07:29:26.192605+0000","last_peered":"2024-03-06T07:29:26.192605+0000","last_clean":"2024-03-06T07:29:26.192605+0000","last_became_active":"2024-03-06T07:29:26.180225+0000","last_became_peered":"2024-03-06T07:29:26.180225+0000","last_unstale":"2024-03-06T07:29:26.192605+0000","last_undegraded":"2024-03-06T07:29:26.192605+0000","last_fullsized":"2024-03-06T07:29:26.192605+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:24.806500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172162+0000","last_change":"2024-03-06T07:29:26.172162+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172162+0000","last_undegraded":"2024-03-06T07:29:26.172162+0000","last_fullsized":"2024-03-06T07:29:26.172162+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.197768+0000","last_change":"2024-03-06T07:29:26.194945+0000","last_active":"2024-03-06T07:29:26.197768+0000","last_peered":"2024-03-06T07:29:26.197768+0000","last_clean":"2024-03-06T07:29:26.197768+0000","last_became_active":"2024-03-06T07:29:26.194577+0000","last_became_peered":"2024-03-06T07:29:26.194577+0000","last_unstale":"2024-03-06T07:29:26.197768+0000","last_undegraded":"2024-03-06T07:29:26.197768+0000","last_fullsized":"2024-03-06T07:29:26.197768+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:30.583334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.228638+0000","last_change":"2024-03-06T07:29:27.217848+0000","last_active":"2024-03-06T07:29:27.228638+0000","last_peered":"2024-03-06T07:29:27.228638+0000","last_clean":"2024-03-06T07:29:27.228638+0000","last_became_active":"2024-03-06T07:29:27.217149+0000","last_became_peered":"2024-03-06T07:29:27.217149+0000","last_unstale":"2024-03-06T07:29:27.228638+0000","last_undegraded":"2024-03-06T07:29:27.228638+0000","last_fullsized":"2024-03-06T07:29:27.228638+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:52.822882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.247655+0000","last_change":"2024-03-06T07:29:26.188532+0000","last_active":"2024-03-06T07:29:27.247655+0000","last_peered":"2024-03-06T07:29:27.247655+0000","last_clean":"2024-03-06T07:29:27.247655+0000","last_became_active":"2024-03-06T07:29:26.188243+0000","last_became_peered":"2024-03-06T07:29:26.188243+0000","last_unstale":"2024-03-06T07:29:27.247655+0000","last_undegraded":"2024-03-06T07:29:27.247655+0000","last_fullsized":"2024-03-06T07:29:27.247655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:05.043428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176443+0000","last_change":"2024-03-06T07:29:26.176443+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176443+0000","last_undegraded":"2024-03-06T07:29:26.176443+0000","last_fullsized":"2024-03-06T07:29:26.176443+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.201430+0000","last_change":"2024-03-06T07:29:26.199005+0000","last_active":"2024-03-06T07:29:26.201430+0000","last_peered":"2024-03-06T07:29:26.201430+0000","last_clean":"2024-03-06T07:29:26.201430+0000","last_became_active":"2024-03-06T07:29:26.196517+0000","last_became_peered":"2024-03-06T07:29:26.196517+0000","last_unstale":"2024-03-06T07:29:26.201430+0000","last_undegraded":"2024-03-06T07:29:26.201430+0000","last_fullsized":"2024-03-06T07:29:26.201430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:45.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172544+0000","last_change":"2024-03-06T07:29:26.172544+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172544+0000","last_undegraded":"2024-03-06T07:29:26.172544+0000","last_fullsized":"2024-03-06T07:29:26.172544+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203430+0000","last_change":"2024-03-06T07:29:26.199717+0000","last_active":"2024-03-06T07:29:26.203430+0000","last_peered":"2024-03-06T07:29:26.203430+0000","last_clean":"2024-03-06T07:29:26.203430+0000","last_became_active":"2024-03-06T07:29:26.199436+0000","last_became_peered":"2024-03-06T07:29:26.199436+0000","last_unstale":"2024-03-06T07:29:26.203430+0000","last_undegraded":"2024-03-06T07:29:26.203430+0000","last_fullsized":"2024-03-06T07:29:26.203430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:58.852154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173271+0000","last_change":"2024-03-06T07:29:26.173271+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173271+0000","last_undegraded":"2024-03-06T07:29:26.173271+0000","last_fullsized":"2024-03-06T07:29:26.173271+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192409+0000","last_change":"2024-03-06T07:29:26.182768+0000","last_active":"2024-03-06T07:29:26.192409+0000","last_peered":"2024-03-06T07:29:26.192409+0000","last_clean":"2024-03-06T07:29:26.192409+0000","last_became_active":"2024-03-06T07:29:26.182489+0000","last_became_peered":"2024-03-06T07:29:26.182489+0000","last_unstale":"2024-03-06T07:29:26.192409+0000","last_undegraded":"2024-03-06T07:29:26.192409+0000","last_fullsized":"2024-03-06T07:29:26.192409+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.422880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164678+0000","last_change":"2024-03-06T07:29:26.164678+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164678+0000","last_undegraded":"2024-03-06T07:29:26.164678+0000","last_fullsized":"2024-03-06T07:29:26.164678+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192648+0000","last_change":"2024-03-06T07:29:26.188395+0000","last_active":"2024-03-06T07:29:26.192648+0000","last_peered":"2024-03-06T07:29:26.192648+0000","last_clean":"2024-03-06T07:29:26.192648+0000","last_became_active":"2024-03-06T07:29:26.187669+0000","last_became_peered":"2024-03-06T07:29:26.187669+0000","last_unstale":"2024-03-06T07:29:26.192648+0000","last_undegraded":"2024-03-06T07:29:26.192648+0000","last_fullsized":"2024-03-06T07:29:26.192648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:12.640396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.228102+0000","last_change":"2024-03-06T07:29:27.221492+0000","last_active":"2024-03-06T07:29:27.228102+0000","last_peered":"2024-03-06T07:29:27.228102+0000","last_clean":"2024-03-06T07:29:27.228102+0000","last_became_active":"2024-03-06T07:29:27.221203+0000","last_became_peered":"2024-03-06T07:29:27.221203+0000","last_unstale":"2024-03-06T07:29:27.228102+0000","last_undegraded":"2024-03-06T07:29:27.228102+0000","last_fullsized":"2024-03-06T07:29:27.228102+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:21.674792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203718+0000","last_change":"2024-03-06T07:29:26.200840+0000","last_active":"2024-03-06T07:29:26.203718+0000","last_peered":"2024-03-06T07:29:26.203718+0000","last_clean":"2024-03-06T07:29:26.203718+0000","last_became_active":"2024-03-06T07:29:26.200577+0000","last_became_peered":"2024-03-06T07:29:26.200577+0000","last_unstale":"2024-03-06T07:29:26.203718+0000","last_undegraded":"2024-03-06T07:29:26.203718+0000","last_fullsized":"2024-03-06T07:29:26.203718+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:40.088324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.232958+0000","last_change":"2024-03-06T07:29:27.226841+0000","last_active":"2024-03-06T07:29:27.232958+0000","last_peered":"2024-03-06T07:29:27.232958+0000","last_clean":"2024-03-06T07:29:27.232958+0000","last_became_active":"2024-03-06T07:29:27.226120+0000","last_became_peered":"2024-03-06T07:29:27.226120+0000","last_unstale":"2024-03-06T07:29:27.232958+0000","last_undegraded":"2024-03-06T07:29:27.232958+0000","last_fullsized":"2024-03-06T07:29:27.232958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:22.288331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.195993+0000","last_change":"2024-03-06T07:29:26.191147+0000","last_active":"2024-03-06T07:29:26.195993+0000","last_peered":"2024-03-06T07:29:26.195993+0000","last_clean":"2024-03-06T07:29:26.195993+0000","last_became_active":"2024-03-06T07:29:26.190641+0000","last_became_peered":"2024-03-06T07:29:26.190641+0000","last_unstale":"2024-03-06T07:29:26.195993+0000","last_undegraded":"2024-03-06T07:29:26.195993+0000","last_fullsized":"2024-03-06T07:29:26.195993+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:52.210630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.177373+0000","last_change":"2024-03-06T07:29:26.177373+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.177373+0000","last_undegraded":"2024-03-06T07:29:26.177373+0000","last_fullsized":"2024-03-06T07:29:26.177373+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198049+0000","last_change":"2024-03-06T07:29:26.193713+0000","last_active":"2024-03-06T07:29:26.198049+0000","last_peered":"2024-03-06T07:29:26.198049+0000","last_clean":"2024-03-06T07:29:26.198049+0000","last_became_active":"2024-03-06T07:29:26.193389+0000","last_became_peered":"2024-03-06T07:29:26.193389+0000","last_unstale":"2024-03-06T07:29:26.198049+0000","last_undegraded":"2024-03-06T07:29:26.198049+0000","last_fullsized":"2024-03-06T07:29:26.198049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:58.185331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172598+0000","last_change":"2024-03-06T07:29:26.172598+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172598+0000","last_undegraded":"2024-03-06T07:29:26.172598+0000","last_fullsized":"2024-03-06T07:29:26.172598+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198317+0000","last_change":"2024-03-06T07:29:26.196747+0000","last_active":"2024-03-06T07:29:26.198317+0000","last_peered":"2024-03-06T07:29:26.198317+0000","last_clean":"2024-03-06T07:29:26.198317+0000","last_became_active":"2024-03-06T07:29:26.196446+0000","last_became_peered":"2024-03-06T07:29:26.196446+0000","last_unstale":"2024-03-06T07:29:26.198317+0000","last_undegraded":"2024-03-06T07:29:26.198317+0000","last_fullsized":"2024-03-06T07:29:26.198317+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:18.921700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176362+0000","last_change":"2024-03-06T07:29:26.176362+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176362+0000","last_undegraded":"2024-03-06T07:29:26.176362+0000","last_fullsized":"2024-03-06T07:29:26.176362+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.200181+0000","last_change":"2024-03-06T07:29:26.199027+0000","last_active":"2024-03-06T07:29:26.200181+0000","last_peered":"2024-03-06T07:29:26.200181+0000","last_clean":"2024-03-06T07:29:26.200181+0000","last_became_active":"2024-03-06T07:29:26.198748+0000","last_became_peered":"2024-03-06T07:29:26.198748+0000","last_unstale":"2024-03-06T07:29:26.200181+0000","last_undegraded":"2024-03-06T07:29:26.200181+0000","last_fullsized":"2024-03-06T07:29:26.200181+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:42.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.174000+0000","last_change":"2024-03-06T07:29:26.174000+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.174000+0000","last_undegraded":"2024-03-06T07:29:26.174000+0000","last_fullsized":"2024-03-06T07:29:26.174000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.221797+0000","last_change":"2024-03-06T07:29:26.196655+0000","last_active":"2024-03-06T07:29:27.221797+0000","last_peered":"2024-03-06T07:29:27.221797+0000","last_clean":"2024-03-06T07:29:27.221797+0000","last_became_active":"2024-03-06T07:29:26.196370+0000","last_became_peered":"2024-03-06T07:29:26.196370+0000","last_unstale":"2024-03-06T07:29:27.221797+0000","last_undegraded":"2024-03-06T07:29:27.221797+0000","last_fullsized":"2024-03-06T07:29:27.221797+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.163810+0000","last_change":"2024-03-06T07:29:26.163810+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.163810+0000","last_undegraded":"2024-03-06T07:29:26.163810+0000","last_fullsized":"2024-03-06T07:29:26.163810+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198276+0000","last_change":"2024-03-06T07:29:26.195806+0000","last_active":"2024-03-06T07:29:26.198276+0000","last_peered":"2024-03-06T07:29:26.198276+0000","last_clean":"2024-03-06T07:29:26.198276+0000","last_became_active":"2024-03-06T07:29:26.195455+0000","last_became_peered":"2024-03-06T07:29:26.195455+0000","last_unstale":"2024-03-06T07:29:26.198276+0000","last_undegraded":"2024-03-06T07:29:26.198276+0000","last_fullsized":"2024-03-06T07:29:26.198276+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:20.660770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164058+0000","last_change":"2024-03-06T07:29:26.164058+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164058+0000","last_undegraded":"2024-03-06T07:29:26.164058+0000","last_fullsized":"2024-03-06T07:29:26.164058+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.201349+0000","last_change":"2024-03-06T07:29:26.199356+0000","last_active":"2024-03-06T07:29:26.201349+0000","last_peered":"2024-03-06T07:29:26.201349+0000","last_clean":"2024-03-06T07:29:26.201349+0000","last_became_active":"2024-03-06T07:29:26.199015+0000","last_became_peered":"2024-03-06T07:29:26.199015+0000","last_unstale":"2024-03-06T07:29:26.201349+0000","last_undegraded":"2024-03-06T07:29:26.201349+0000","last_fullsized":"2024-03-06T07:29:26.201349+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:21.561181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.178828+0000","last_change":"2024-03-06T07:29:26.178828+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.178828+0000","last_undegraded":"2024-03-06T07:29:26.178828+0000","last_fullsized":"2024-03-06T07:29:26.178828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.220161+0000","last_change":"2024-03-06T07:29:26.184910+0000","last_active":"2024-03-06T07:29:27.220161+0000","last_peered":"2024-03-06T07:29:27.220161+0000","last_clean":"2024-03-06T07:29:27.220161+0000","last_became_active":"2024-03-06T07:29:26.183588+0000","last_became_peered":"2024-03-06T07:29:26.183588+0000","last_unstale":"2024-03-06T07:29:27.220161+0000","last_undegraded":"2024-03-06T07:29:27.220161+0000","last_fullsized":"2024-03-06T07:29:27.220161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179731+0000","last_change":"2024-03-06T07:29:26.179731+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179731+0000","last_undegraded":"2024-03-06T07:29:26.179731+0000","last_fullsized":"2024-03-06T07:29:26.179731+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189832+0000","last_change":"2024-03-06T07:29:26.185660+0000","last_active":"2024-03-06T07:29:26.189832+0000","last_peered":"2024-03-06T07:29:26.189832+0000","last_clean":"2024-03-06T07:29:26.189832+0000","last_became_active":"2024-03-06T07:29:26.185316+0000","last_became_peered":"2024-03-06T07:29:26.185316+0000","last_unstale":"2024-03-06T07:29:26.189832+0000","last_undegraded":"2024-03-06T07:29:26.189832+0000","last_fullsized":"2024-03-06T07:29:26.189832+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:25.384746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175672+0000","last_change":"2024-03-06T07:29:26.175672+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175672+0000","last_undegraded":"2024-03-06T07:29:26.175672+0000","last_fullsized":"2024-03-06T07:29:26.175672+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203496+0000","last_change":"2024-03-06T07:29:26.200006+0000","last_active":"2024-03-06T07:29:26.203496+0000","last_peered":"2024-03-06T07:29:26.203496+0000","last_clean":"2024-03-06T07:29:26.203496+0000","last_became_active":"2024-03-06T07:29:26.198345+0000","last_became_peered":"2024-03-06T07:29:26.198345+0000","last_unstale":"2024-03-06T07:29:26.203496+0000","last_undegraded":"2024-03-06T07:29:26.203496+0000","last_fullsized":"2024-03-06T07:29:26.203496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.255706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":209,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.184437+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:29:26.184437+0000","last_peered":"2024-03-06T07:29:26.184437+0000","last_clean":"2024-03-06T07:29:26.184437+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:29:26.184437+0000","last_undegraded":"2024-03-06T07:29:26.184437+0000","last_fullsized":"2024-03-06T07:29:26.184437+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509468,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509469,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335436,"statfs":{"total":96636764160,"available":96599486464,"internally_reserved":0,"allocated":1089536,"data_stored":788026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542173,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":197658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":794293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:32.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:32.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.479+0000 7fc178a00640 1 Processor -- start 2024-03-06T07:29:32.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.480+0000 7fc178a00640 1 -- start start 2024-03-06T07:29:32.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.481+0000 7fc178a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc174071380 0x7fc1740ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.481+0000 7fc178a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1740ce9a0 con 0x7fc17406e240 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.481+0000 7fc178a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1740cec80 con 0x7fc174161670 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.481+0000 7fc178a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1740730d0 con 0x7fc174071380 2024-03-06T07:29:32.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc172800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc17406e240 0x7fc1740ad5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57570/0 (socket says 172.21.15.6:57570) 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc172800640 1 -- 172.21.15.6:0/1058500595 learned_addr learned my addr 172.21.15.6:0/1058500595 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc171400640 1 --2- 172.21.15.6:0/1058500595 >> v2:172.21.15.6:3300/0 conn(0x7fc174071380 0x7fc1740ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc171400640 1 -- 172.21.15.6:0/1058500595 >> v1:172.21.15.178:6789/0 conn(0x7fc174161670 legacy=0x7fc174070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc171400640 1 -- 172.21.15.6:0/1058500595 >> v1:172.21.15.6:6789/0 conn(0x7fc17406e240 legacy=0x7fc1740ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc171400640 1 -- 172.21.15.6:0/1058500595 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1740cef60 con 0x7fc174071380 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.482+0000 7fc170a00640 1 -- 172.21.15.6:0/1058500595 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884698779 0 0) 0x7fc1740ce9a0 con 0x7fc17406e240 2024-03-06T07:29:32.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.484+0000 7fc171400640 1 --2- 172.21.15.6:0/1058500595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174071380 0x7fc1740ad090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0097d0 tx=0x7fc15c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfbfdf4f3cd7fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.484+0000 7fc170a00640 1 -- 172.21.15.6:0/1058500595 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0315a0 con 0x7fc174071380 2024-03-06T07:29:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.484+0000 7fc170a00640 1 -- 172.21.15.6:0/1058500595 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c037070 con 0x7fc174071380 2024-03-06T07:29:32.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.484+0000 7fc170a00640 1 -- 172.21.15.6:0/1058500595 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c031960 con 0x7fc174071380 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.484+0000 7fc178a00640 1 -- 172.21.15.6:0/1058500595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174071380 msgr2=0x7fc1740ad090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.484+0000 7fc178a00640 1 --2- 172.21.15.6:0/1058500595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174071380 0x7fc1740ad090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0097d0 tx=0x7fc15c02b840 comp rx=0 tx=0).stop 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.485+0000 7fc178a00640 1 -- 172.21.15.6:0/1058500595 shutdown_connections 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.485+0000 7fc178a00640 1 --2- 172.21.15.6:0/1058500595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174071380 0x7fc1740ad090 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.485+0000 7fc178a00640 1 -- 172.21.15.6:0/1058500595 >> 172.21.15.6:0/1058500595 conn(0x7fc17406afb0 msgr2=0x7fc17406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.485+0000 7fc178a00640 1 -- 172.21.15.6:0/1058500595 shutdown_connections 2024-03-06T07:29:32.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.485+0000 7fc178a00640 1 -- 172.21.15.6:0/1058500595 wait complete. 2024-03-06T07:29:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.486+0000 7fc178a00640 1 Processor -- start 2024-03-06T07:29:32.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.487+0000 7fc178a00640 1 -- start start 2024-03-06T07:29:32.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.487+0000 7fc178a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc17406e240 0x7fc1740b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc178a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc174071380 0x7fc1740cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc178a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 0x7fc1740ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc178a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740ce600 con 0x7fc174161670 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc178a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740cf520 con 0x7fc17406e240 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc178a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740ce780 con 0x7fc174071380 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 0x7fc1740ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 0x7fc1740ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43244/0 (socket says 172.21.15.6:43244) 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 -- 172.21.15.6:0/2804055705 learned_addr learned my addr 172.21.15.6:0/2804055705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc171e00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc17406e240 0x7fc1740b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc171400640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc174071380 0x7fc1740cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 -- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc174071380 msgr2=0x7fc1740cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc174071380 0x7fc1740cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 -- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc17406e240 msgr2=0x7fc1740b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.488+0000 7fc172800640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc17406e240 0x7fc1740b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc172800640 1 -- 172.21.15.6:0/2804055705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1740b0ae0 con 0x7fc174161670 2024-03-06T07:29:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc171400640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc174071380 0x7fc1740cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc172800640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 0x7fc1740ccd30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00c480 tx=0x7fc16c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c018070 con 0x7fc174161670 2024-03-06T07:29:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16c0074e0 con 0x7fc174161670 2024-03-06T07:29:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c004a20 con 0x7fc174161670 2024-03-06T07:29:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1740b0dc0 con 0x7fc174161670 2024-03-06T07:29:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.489+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17409c310 con 0x7fc174161670 2024-03-06T07:29:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.490+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17404f3a0 con 0x7fc174161670 2024-03-06T07:29:32.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.491+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc16c01b070 con 0x7fc174161670 2024-03-06T07:29:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.492+0000 7fc16a000640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:32.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.492+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc16c089520 con 0x7fc174161670 2024-03-06T07:29:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.493+0000 7fc171e00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:32.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.494+0000 7fc171e00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc1740cc150 tx=0x7fc1640094d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:32.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.497+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc16c058390 con 0x7fc174161670 2024-03-06T07:29:32.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.692+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1740c9400 con 0x7fc15006b4d0 2024-03-06T07:29:32.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.711+0000 7fc16a000640 1 -- 172.21.15.6:0/2804055705 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247401 (secure 0 0 0) 0x7fc1740c9400 con 0x7fc15006b4d0 2024-03-06T07:29:32.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:32.717 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:29:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.715+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 msgr2=0x7fc15006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.715+0000 7fc178a00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc1740cc150 tx=0x7fc1640094d0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.715+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 msgr2=0x7fc1740ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.715+0000 7fc178a00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 0x7fc1740ccd30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00c480 tx=0x7fc16c00c940 comp rx=0 tx=0).stop 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 shutdown_connections 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc174161670 0x7fc1740ccd30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc174071380 0x7fc1740cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 --2- 172.21.15.6:0/2804055705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc17406e240 0x7fc1740b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 >> 172.21.15.6:0/2804055705 conn(0x7fc17406afb0 msgr2=0x7fc1740b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 shutdown_connections 2024-03-06T07:29:32.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:32.716+0000 7fc178a00640 1 -- 172.21.15.6:0/2804055705 wait complete. 2024-03-06T07:29:32.737 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-03-06T07:29:31.480064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":169,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126,"ondisk_log_size":126,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254568,"kb_used_data":5104,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720152,"statfs":{"total":773094113280,"available":772833435648,"internally_reserved":0,"allocated":5226496,"data_stored":2799602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001807"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180481+0000","last_change":"2024-03-06T07:29:26.180481+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180481+0000","last_undegraded":"2024-03-06T07:29:26.180481+0000","last_fullsized":"2024-03-06T07:29:26.180481+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.185266+0000","last_change":"2024-03-06T07:29:26.185266+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.185266+0000","last_undegraded":"2024-03-06T07:29:26.185266+0000","last_fullsized":"2024-03-06T07:29:26.185266+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.186354+0000","last_change":"2024-03-06T07:29:26.186354+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.186354+0000","last_undegraded":"2024-03-06T07:29:26.186354+0000","last_fullsized":"2024-03-06T07:29:26.186354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180082+0000","last_change":"2024-03-06T07:29:26.180082+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180082+0000","last_undegraded":"2024-03-06T07:29:26.180082+0000","last_fullsized":"2024-03-06T07:29:26.180082+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179026+0000","last_change":"2024-03-06T07:29:26.179026+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179026+0000","last_undegraded":"2024-03-06T07:29:26.179026+0000","last_fullsized":"2024-03-06T07:29:26.179026+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.165169+0000","last_change":"2024-03-06T07:29:26.165169+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.165169+0000","last_undegraded":"2024-03-06T07:29:26.165169+0000","last_fullsized":"2024-03-06T07:29:26.165169+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173991+0000","last_change":"2024-03-06T07:29:26.173991+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173991+0000","last_undegraded":"2024-03-06T07:29:26.173991+0000","last_fullsized":"2024-03-06T07:29:26.173991+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179661+0000","last_change":"2024-03-06T07:29:26.179661+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179661+0000","last_undegraded":"2024-03-06T07:29:26.179661+0000","last_fullsized":"2024-03-06T07:29:26.179661+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179688+0000","last_change":"2024-03-06T07:29:26.179688+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179688+0000","last_undegraded":"2024-03-06T07:29:26.179688+0000","last_fullsized":"2024-03-06T07:29:26.179688+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179130+0000","last_change":"2024-03-06T07:29:26.179130+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179130+0000","last_undegraded":"2024-03-06T07:29:26.179130+0000","last_fullsized":"2024-03-06T07:29:26.179130+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.177134+0000","last_change":"2024-03-06T07:29:26.177134+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.177134+0000","last_undegraded":"2024-03-06T07:29:26.177134+0000","last_fullsized":"2024-03-06T07:29:26.177134+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.186917+0000","last_change":"2024-03-06T07:29:26.186917+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.186917+0000","last_undegraded":"2024-03-06T07:29:26.186917+0000","last_fullsized":"2024-03-06T07:29:26.186917+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.236195+0000","last_change":"2024-03-06T07:29:27.234158+0000","last_active":"2024-03-06T07:29:27.236195+0000","last_peered":"2024-03-06T07:29:27.236195+0000","last_clean":"2024-03-06T07:29:27.236195+0000","last_became_active":"2024-03-06T07:29:27.233803+0000","last_became_peered":"2024-03-06T07:29:27.233803+0000","last_unstale":"2024-03-06T07:29:27.236195+0000","last_undegraded":"2024-03-06T07:29:27.236195+0000","last_fullsized":"2024-03-06T07:29:27.236195+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:56.608802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.180437+0000","last_change":"2024-03-06T07:29:26.180437+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.180437+0000","last_undegraded":"2024-03-06T07:29:26.180437+0000","last_fullsized":"2024-03-06T07:29:26.180437+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.167433+0000","last_change":"2024-03-06T07:29:26.167433+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.167433+0000","last_undegraded":"2024-03-06T07:29:26.167433+0000","last_fullsized":"2024-03-06T07:29:26.167433+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172611+0000","last_change":"2024-03-06T07:29:26.172611+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172611+0000","last_undegraded":"2024-03-06T07:29:26.172611+0000","last_fullsized":"2024-03-06T07:29:26.172611+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.187650+0000","last_change":"2024-03-06T07:29:26.187650+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.187650+0000","last_undegraded":"2024-03-06T07:29:26.187650+0000","last_fullsized":"2024-03-06T07:29:26.187650+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.189091+0000","last_change":"2024-03-06T07:29:26.189091+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.189091+0000","last_undegraded":"2024-03-06T07:29:26.189091+0000","last_fullsized":"2024-03-06T07:29:26.189091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171250+0000","last_change":"2024-03-06T07:29:26.171250+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171250+0000","last_undegraded":"2024-03-06T07:29:26.171250+0000","last_fullsized":"2024-03-06T07:29:26.171250+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.237778+0000","last_change":"2024-03-06T07:29:27.232657+0000","last_active":"2024-03-06T07:29:27.237778+0000","last_peered":"2024-03-06T07:29:27.237778+0000","last_clean":"2024-03-06T07:29:27.237778+0000","last_became_active":"2024-03-06T07:29:27.232358+0000","last_became_peered":"2024-03-06T07:29:27.232358+0000","last_unstale":"2024-03-06T07:29:27.237778+0000","last_undegraded":"2024-03-06T07:29:27.237778+0000","last_fullsized":"2024-03-06T07:29:27.237778+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:34:14.573067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164817+0000","last_change":"2024-03-06T07:29:26.164817+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164817+0000","last_undegraded":"2024-03-06T07:29:26.164817+0000","last_fullsized":"2024-03-06T07:29:26.164817+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176355+0000","last_change":"2024-03-06T07:29:26.176355+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176355+0000","last_undegraded":"2024-03-06T07:29:26.176355+0000","last_fullsized":"2024-03-06T07:29:26.176355+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.170468+0000","last_change":"2024-03-06T07:29:26.170468+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.170468+0000","last_undegraded":"2024-03-06T07:29:26.170468+0000","last_fullsized":"2024-03-06T07:29:26.170468+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171475+0000","last_change":"2024-03-06T07:29:26.171475+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171475+0000","last_undegraded":"2024-03-06T07:29:26.171475+0000","last_fullsized":"2024-03-06T07:29:26.171475+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.174866+0000","last_change":"2024-03-06T07:29:26.174866+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.174866+0000","last_undegraded":"2024-03-06T07:29:26.174866+0000","last_fullsized":"2024-03-06T07:29:26.174866+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.236949+0000","last_change":"2024-03-06T07:29:27.234582+0000","last_active":"2024-03-06T07:29:27.236949+0000","last_peered":"2024-03-06T07:29:27.236949+0000","last_clean":"2024-03-06T07:29:27.236949+0000","last_became_active":"2024-03-06T07:29:27.233810+0000","last_became_peered":"2024-03-06T07:29:27.233810+0000","last_unstale":"2024-03-06T07:29:27.236949+0000","last_undegraded":"2024-03-06T07:29:27.236949+0000","last_fullsized":"2024-03-06T07:29:27.236949+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:44.382794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.184051+0000","last_change":"2024-03-06T07:29:26.184051+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.184051+0000","last_undegraded":"2024-03-06T07:29:26.184051+0000","last_fullsized":"2024-03-06T07:29:26.184051+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.183973+0000","last_change":"2024-03-06T07:29:26.183973+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.183973+0000","last_undegraded":"2024-03-06T07:29:26.183973+0000","last_fullsized":"2024-03-06T07:29:26.183973+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171100+0000","last_change":"2024-03-06T07:29:26.171100+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171100+0000","last_undegraded":"2024-03-06T07:29:26.171100+0000","last_fullsized":"2024-03-06T07:29:26.171100+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.182982+0000","last_change":"2024-03-06T07:29:26.182982+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.182982+0000","last_undegraded":"2024-03-06T07:29:26.182982+0000","last_fullsized":"2024-03-06T07:29:26.182982+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.166901+0000","last_change":"2024-03-06T07:29:26.166901+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.166901+0000","last_undegraded":"2024-03-06T07:29:26.166901+0000","last_fullsized":"2024-03-06T07:29:26.166901+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.233068+0000","last_change":"2024-03-06T07:29:27.227701+0000","last_active":"2024-03-06T07:29:27.233068+0000","last_peered":"2024-03-06T07:29:27.233068+0000","last_clean":"2024-03-06T07:29:27.233068+0000","last_became_active":"2024-03-06T07:29:27.225595+0000","last_became_peered":"2024-03-06T07:29:27.225595+0000","last_unstale":"2024-03-06T07:29:27.233068+0000","last_undegraded":"2024-03-06T07:29:27.233068+0000","last_fullsized":"2024-03-06T07:29:27.233068+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:55.302360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.227914+0000","last_change":"2024-03-06T07:29:27.222401+0000","last_active":"2024-03-06T07:29:27.227914+0000","last_peered":"2024-03-06T07:29:27.227914+0000","last_clean":"2024-03-06T07:29:27.227914+0000","last_became_active":"2024-03-06T07:29:27.221978+0000","last_became_peered":"2024-03-06T07:29:27.221978+0000","last_unstale":"2024-03-06T07:29:27.227914+0000","last_undegraded":"2024-03-06T07:29:27.227914+0000","last_fullsized":"2024-03-06T07:29:27.227914+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:43.908483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192462+0000","last_change":"2024-03-06T07:29:26.184227+0000","last_active":"2024-03-06T07:29:26.192462+0000","last_peered":"2024-03-06T07:29:26.192462+0000","last_clean":"2024-03-06T07:29:26.192462+0000","last_became_active":"2024-03-06T07:29:26.183923+0000","last_became_peered":"2024-03-06T07:29:26.183923+0000","last_unstale":"2024-03-06T07:29:26.192462+0000","last_undegraded":"2024-03-06T07:29:26.192462+0000","last_fullsized":"2024-03-06T07:29:26.192462+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:08.181893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173821+0000","last_change":"2024-03-06T07:29:26.173821+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173821+0000","last_undegraded":"2024-03-06T07:29:26.173821+0000","last_fullsized":"2024-03-06T07:29:26.173821+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.202304+0000","last_change":"2024-03-06T07:29:26.200719+0000","last_active":"2024-03-06T07:29:26.202304+0000","last_peered":"2024-03-06T07:29:26.202304+0000","last_clean":"2024-03-06T07:29:26.202304+0000","last_became_active":"2024-03-06T07:29:26.200438+0000","last_became_peered":"2024-03-06T07:29:26.200438+0000","last_unstale":"2024-03-06T07:29:26.202304+0000","last_undegraded":"2024-03-06T07:29:26.202304+0000","last_fullsized":"2024-03-06T07:29:26.202304+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:27.927651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.183386+0000","last_change":"2024-03-06T07:29:26.183386+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.183386+0000","last_undegraded":"2024-03-06T07:29:26.183386+0000","last_fullsized":"2024-03-06T07:29:26.183386+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.243346+0000","last_change":"2024-03-06T07:29:26.201813+0000","last_active":"2024-03-06T07:29:27.243346+0000","last_peered":"2024-03-06T07:29:27.243346+0000","last_clean":"2024-03-06T07:29:27.243346+0000","last_became_active":"2024-03-06T07:29:26.201360+0000","last_became_peered":"2024-03-06T07:29:26.201360+0000","last_unstale":"2024-03-06T07:29:27.243346+0000","last_undegraded":"2024-03-06T07:29:27.243346+0000","last_fullsized":"2024-03-06T07:29:27.243346+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.232720+0000","last_change":"2024-03-06T07:29:27.231046+0000","last_active":"2024-03-06T07:29:27.232720+0000","last_peered":"2024-03-06T07:29:27.232720+0000","last_clean":"2024-03-06T07:29:27.232720+0000","last_became_active":"2024-03-06T07:29:27.230774+0000","last_became_peered":"2024-03-06T07:29:27.230774+0000","last_unstale":"2024-03-06T07:29:27.232720+0000","last_undegraded":"2024-03-06T07:29:27.232720+0000","last_fullsized":"2024-03-06T07:29:27.232720+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:29.573395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.204067+0000","last_change":"2024-03-06T07:29:26.202950+0000","last_active":"2024-03-06T07:29:26.204067+0000","last_peered":"2024-03-06T07:29:26.204067+0000","last_clean":"2024-03-06T07:29:26.204067+0000","last_became_active":"2024-03-06T07:29:26.202564+0000","last_became_peered":"2024-03-06T07:29:26.202564+0000","last_unstale":"2024-03-06T07:29:26.204067+0000","last_undegraded":"2024-03-06T07:29:26.204067+0000","last_fullsized":"2024-03-06T07:29:26.204067+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.182043+0000","last_change":"2024-03-06T07:29:26.182043+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.182043+0000","last_undegraded":"2024-03-06T07:29:26.182043+0000","last_fullsized":"2024-03-06T07:29:26.182043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192624+0000","last_change":"2024-03-06T07:29:26.188354+0000","last_active":"2024-03-06T07:29:26.192624+0000","last_peered":"2024-03-06T07:29:26.192624+0000","last_clean":"2024-03-06T07:29:26.192624+0000","last_became_active":"2024-03-06T07:29:26.187378+0000","last_became_peered":"2024-03-06T07:29:26.187378+0000","last_unstale":"2024-03-06T07:29:26.192624+0000","last_undegraded":"2024-03-06T07:29:26.192624+0000","last_fullsized":"2024-03-06T07:29:26.192624+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:41.649859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.178981+0000","last_change":"2024-03-06T07:29:26.178981+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.178981+0000","last_undegraded":"2024-03-06T07:29:26.178981+0000","last_fullsized":"2024-03-06T07:29:26.178981+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.205035+0000","last_change":"2024-03-06T07:29:26.204279+0000","last_active":"2024-03-06T07:29:26.205035+0000","last_peered":"2024-03-06T07:29:26.205035+0000","last_clean":"2024-03-06T07:29:26.205035+0000","last_became_active":"2024-03-06T07:29:26.203822+0000","last_became_peered":"2024-03-06T07:29:26.203822+0000","last_unstale":"2024-03-06T07:29:26.205035+0000","last_undegraded":"2024-03-06T07:29:26.205035+0000","last_fullsized":"2024-03-06T07:29:26.205035+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:11.492878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175828+0000","last_change":"2024-03-06T07:29:26.175828+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175828+0000","last_undegraded":"2024-03-06T07:29:26.175828+0000","last_fullsized":"2024-03-06T07:29:26.175828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.202004+0000","last_change":"2024-03-06T07:29:26.194000+0000","last_active":"2024-03-06T07:29:26.202004+0000","last_peered":"2024-03-06T07:29:26.202004+0000","last_clean":"2024-03-06T07:29:26.202004+0000","last_became_active":"2024-03-06T07:29:26.193684+0000","last_became_peered":"2024-03-06T07:29:26.193684+0000","last_unstale":"2024-03-06T07:29:26.202004+0000","last_undegraded":"2024-03-06T07:29:26.202004+0000","last_fullsized":"2024-03-06T07:29:26.202004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:29.750611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.171753+0000","last_change":"2024-03-06T07:29:26.171753+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.171753+0000","last_undegraded":"2024-03-06T07:29:26.171753+0000","last_fullsized":"2024-03-06T07:29:26.171753+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189388+0000","last_change":"2024-03-06T07:29:26.182870+0000","last_active":"2024-03-06T07:29:26.189388+0000","last_peered":"2024-03-06T07:29:26.189388+0000","last_clean":"2024-03-06T07:29:26.189388+0000","last_became_active":"2024-03-06T07:29:26.182537+0000","last_became_peered":"2024-03-06T07:29:26.182537+0000","last_unstale":"2024-03-06T07:29:26.189388+0000","last_undegraded":"2024-03-06T07:29:26.189388+0000","last_fullsized":"2024-03-06T07:29:26.189388+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:04:04.448614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.188604+0000","last_change":"2024-03-06T07:29:26.188604+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.188604+0000","last_undegraded":"2024-03-06T07:29:26.188604+0000","last_fullsized":"2024-03-06T07:29:26.188604+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.196320+0000","last_change":"2024-03-06T07:29:26.193099+0000","last_active":"2024-03-06T07:29:26.196320+0000","last_peered":"2024-03-06T07:29:26.196320+0000","last_clean":"2024-03-06T07:29:26.196320+0000","last_became_active":"2024-03-06T07:29:26.192805+0000","last_became_peered":"2024-03-06T07:29:26.192805+0000","last_unstale":"2024-03-06T07:29:26.196320+0000","last_undegraded":"2024-03-06T07:29:26.196320+0000","last_fullsized":"2024-03-06T07:29:26.196320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:43.625964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179348+0000","last_change":"2024-03-06T07:29:26.179348+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179348+0000","last_undegraded":"2024-03-06T07:29:26.179348+0000","last_fullsized":"2024-03-06T07:29:26.179348+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.190038+0000","last_change":"2024-03-06T07:29:26.186625+0000","last_active":"2024-03-06T07:29:26.190038+0000","last_peered":"2024-03-06T07:29:26.190038+0000","last_clean":"2024-03-06T07:29:26.190038+0000","last_became_active":"2024-03-06T07:29:26.186363+0000","last_became_peered":"2024-03-06T07:29:26.186363+0000","last_unstale":"2024-03-06T07:29:26.190038+0000","last_undegraded":"2024-03-06T07:29:26.190038+0000","last_fullsized":"2024-03-06T07:29:26.190038+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:10:53.612157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.166868+0000","last_change":"2024-03-06T07:29:26.166868+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.166868+0000","last_undegraded":"2024-03-06T07:29:26.166868+0000","last_fullsized":"2024-03-06T07:29:26.166868+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198735+0000","last_change":"2024-03-06T07:29:26.197651+0000","last_active":"2024-03-06T07:29:26.198735+0000","last_peered":"2024-03-06T07:29:26.198735+0000","last_clean":"2024-03-06T07:29:26.198735+0000","last_became_active":"2024-03-06T07:29:26.197161+0000","last_became_peered":"2024-03-06T07:29:26.197161+0000","last_unstale":"2024-03-06T07:29:26.198735+0000","last_undegraded":"2024-03-06T07:29:26.198735+0000","last_fullsized":"2024-03-06T07:29:26.198735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179847+0000","last_change":"2024-03-06T07:29:26.179847+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179847+0000","last_undegraded":"2024-03-06T07:29:26.179847+0000","last_fullsized":"2024-03-06T07:29:26.179847+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.248964+0000","last_change":"2024-03-06T07:29:26.200648+0000","last_active":"2024-03-06T07:29:27.248964+0000","last_peered":"2024-03-06T07:29:27.248964+0000","last_clean":"2024-03-06T07:29:27.248964+0000","last_became_active":"2024-03-06T07:29:26.200365+0000","last_became_peered":"2024-03-06T07:29:26.200365+0000","last_unstale":"2024-03-06T07:29:27.248964+0000","last_undegraded":"2024-03-06T07:29:27.248964+0000","last_fullsized":"2024-03-06T07:29:27.248964+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.411331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.170598+0000","last_change":"2024-03-06T07:29:26.170598+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.170598+0000","last_undegraded":"2024-03-06T07:29:26.170598+0000","last_fullsized":"2024-03-06T07:29:26.170598+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189203+0000","last_change":"2024-03-06T07:29:26.180918+0000","last_active":"2024-03-06T07:29:26.189203+0000","last_peered":"2024-03-06T07:29:26.189203+0000","last_clean":"2024-03-06T07:29:26.189203+0000","last_became_active":"2024-03-06T07:29:26.180570+0000","last_became_peered":"2024-03-06T07:29:26.180570+0000","last_unstale":"2024-03-06T07:29:26.189203+0000","last_undegraded":"2024-03-06T07:29:26.189203+0000","last_fullsized":"2024-03-06T07:29:26.189203+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.869827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175797+0000","last_change":"2024-03-06T07:29:26.175797+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175797+0000","last_undegraded":"2024-03-06T07:29:26.175797+0000","last_fullsized":"2024-03-06T07:29:26.175797+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198013+0000","last_change":"2024-03-06T07:29:26.194437+0000","last_active":"2024-03-06T07:29:26.198013+0000","last_peered":"2024-03-06T07:29:26.198013+0000","last_clean":"2024-03-06T07:29:26.198013+0000","last_became_active":"2024-03-06T07:29:26.194154+0000","last_became_peered":"2024-03-06T07:29:26.194154+0000","last_unstale":"2024-03-06T07:29:26.198013+0000","last_undegraded":"2024-03-06T07:29:26.198013+0000","last_fullsized":"2024-03-06T07:29:26.198013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.163731+0000","last_change":"2024-03-06T07:29:26.163731+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.163731+0000","last_undegraded":"2024-03-06T07:29:26.163731+0000","last_fullsized":"2024-03-06T07:29:26.163731+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192605+0000","last_change":"2024-03-06T07:29:26.185585+0000","last_active":"2024-03-06T07:29:26.192605+0000","last_peered":"2024-03-06T07:29:26.192605+0000","last_clean":"2024-03-06T07:29:26.192605+0000","last_became_active":"2024-03-06T07:29:26.180225+0000","last_became_peered":"2024-03-06T07:29:26.180225+0000","last_unstale":"2024-03-06T07:29:26.192605+0000","last_undegraded":"2024-03-06T07:29:26.192605+0000","last_fullsized":"2024-03-06T07:29:26.192605+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:24.806500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172162+0000","last_change":"2024-03-06T07:29:26.172162+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172162+0000","last_undegraded":"2024-03-06T07:29:26.172162+0000","last_fullsized":"2024-03-06T07:29:26.172162+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.197768+0000","last_change":"2024-03-06T07:29:26.194945+0000","last_active":"2024-03-06T07:29:26.197768+0000","last_peered":"2024-03-06T07:29:26.197768+0000","last_clean":"2024-03-06T07:29:26.197768+0000","last_became_active":"2024-03-06T07:29:26.194577+0000","last_became_peered":"2024-03-06T07:29:26.194577+0000","last_unstale":"2024-03-06T07:29:26.197768+0000","last_undegraded":"2024-03-06T07:29:26.197768+0000","last_fullsized":"2024-03-06T07:29:26.197768+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:06:30.583334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.228638+0000","last_change":"2024-03-06T07:29:27.217848+0000","last_active":"2024-03-06T07:29:27.228638+0000","last_peered":"2024-03-06T07:29:27.228638+0000","last_clean":"2024-03-06T07:29:27.228638+0000","last_became_active":"2024-03-06T07:29:27.217149+0000","last_became_peered":"2024-03-06T07:29:27.217149+0000","last_unstale":"2024-03-06T07:29:27.228638+0000","last_undegraded":"2024-03-06T07:29:27.228638+0000","last_fullsized":"2024-03-06T07:29:27.228638+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:52.822882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.247655+0000","last_change":"2024-03-06T07:29:26.188532+0000","last_active":"2024-03-06T07:29:27.247655+0000","last_peered":"2024-03-06T07:29:27.247655+0000","last_clean":"2024-03-06T07:29:27.247655+0000","last_became_active":"2024-03-06T07:29:26.188243+0000","last_became_peered":"2024-03-06T07:29:26.188243+0000","last_unstale":"2024-03-06T07:29:27.247655+0000","last_undegraded":"2024-03-06T07:29:27.247655+0000","last_fullsized":"2024-03-06T07:29:27.247655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:05.043428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176443+0000","last_change":"2024-03-06T07:29:26.176443+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176443+0000","last_undegraded":"2024-03-06T07:29:26.176443+0000","last_fullsized":"2024-03-06T07:29:26.176443+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.201430+0000","last_change":"2024-03-06T07:29:26.199005+0000","last_active":"2024-03-06T07:29:26.201430+0000","last_peered":"2024-03-06T07:29:26.201430+0000","last_clean":"2024-03-06T07:29:26.201430+0000","last_became_active":"2024-03-06T07:29:26.196517+0000","last_became_peered":"2024-03-06T07:29:26.196517+0000","last_unstale":"2024-03-06T07:29:26.201430+0000","last_undegraded":"2024-03-06T07:29:26.201430+0000","last_fullsized":"2024-03-06T07:29:26.201430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:45.086793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172544+0000","last_change":"2024-03-06T07:29:26.172544+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172544+0000","last_undegraded":"2024-03-06T07:29:26.172544+0000","last_fullsized":"2024-03-06T07:29:26.172544+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203430+0000","last_change":"2024-03-06T07:29:26.199717+0000","last_active":"2024-03-06T07:29:26.203430+0000","last_peered":"2024-03-06T07:29:26.203430+0000","last_clean":"2024-03-06T07:29:26.203430+0000","last_became_active":"2024-03-06T07:29:26.199436+0000","last_became_peered":"2024-03-06T07:29:26.199436+0000","last_unstale":"2024-03-06T07:29:26.203430+0000","last_undegraded":"2024-03-06T07:29:26.203430+0000","last_fullsized":"2024-03-06T07:29:26.203430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:58.852154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.173271+0000","last_change":"2024-03-06T07:29:26.173271+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.173271+0000","last_undegraded":"2024-03-06T07:29:26.173271+0000","last_fullsized":"2024-03-06T07:29:26.173271+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192409+0000","last_change":"2024-03-06T07:29:26.182768+0000","last_active":"2024-03-06T07:29:26.192409+0000","last_peered":"2024-03-06T07:29:26.192409+0000","last_clean":"2024-03-06T07:29:26.192409+0000","last_became_active":"2024-03-06T07:29:26.182489+0000","last_became_peered":"2024-03-06T07:29:26.182489+0000","last_unstale":"2024-03-06T07:29:26.192409+0000","last_undegraded":"2024-03-06T07:29:26.192409+0000","last_fullsized":"2024-03-06T07:29:26.192409+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.422880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164678+0000","last_change":"2024-03-06T07:29:26.164678+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164678+0000","last_undegraded":"2024-03-06T07:29:26.164678+0000","last_fullsized":"2024-03-06T07:29:26.164678+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.192648+0000","last_change":"2024-03-06T07:29:26.188395+0000","last_active":"2024-03-06T07:29:26.192648+0000","last_peered":"2024-03-06T07:29:26.192648+0000","last_clean":"2024-03-06T07:29:26.192648+0000","last_became_active":"2024-03-06T07:29:26.187669+0000","last_became_peered":"2024-03-06T07:29:26.187669+0000","last_unstale":"2024-03-06T07:29:26.192648+0000","last_undegraded":"2024-03-06T07:29:26.192648+0000","last_fullsized":"2024-03-06T07:29:26.192648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:12.640396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.228102+0000","last_change":"2024-03-06T07:29:27.221492+0000","last_active":"2024-03-06T07:29:27.228102+0000","last_peered":"2024-03-06T07:29:27.228102+0000","last_clean":"2024-03-06T07:29:27.228102+0000","last_became_active":"2024-03-06T07:29:27.221203+0000","last_became_peered":"2024-03-06T07:29:27.221203+0000","last_unstale":"2024-03-06T07:29:27.228102+0000","last_undegraded":"2024-03-06T07:29:27.228102+0000","last_fullsized":"2024-03-06T07:29:27.228102+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:21.674792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203718+0000","last_change":"2024-03-06T07:29:26.200840+0000","last_active":"2024-03-06T07:29:26.203718+0000","last_peered":"2024-03-06T07:29:26.203718+0000","last_clean":"2024-03-06T07:29:26.203718+0000","last_became_active":"2024-03-06T07:29:26.200577+0000","last_became_peered":"2024-03-06T07:29:26.200577+0000","last_unstale":"2024-03-06T07:29:26.203718+0000","last_undegraded":"2024-03-06T07:29:26.203718+0000","last_fullsized":"2024-03-06T07:29:26.203718+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:40.088324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.232958+0000","last_change":"2024-03-06T07:29:27.226841+0000","last_active":"2024-03-06T07:29:27.232958+0000","last_peered":"2024-03-06T07:29:27.232958+0000","last_clean":"2024-03-06T07:29:27.232958+0000","last_became_active":"2024-03-06T07:29:27.226120+0000","last_became_peered":"2024-03-06T07:29:27.226120+0000","last_unstale":"2024-03-06T07:29:27.232958+0000","last_undegraded":"2024-03-06T07:29:27.232958+0000","last_fullsized":"2024-03-06T07:29:27.232958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:22.288331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.195993+0000","last_change":"2024-03-06T07:29:26.191147+0000","last_active":"2024-03-06T07:29:26.195993+0000","last_peered":"2024-03-06T07:29:26.195993+0000","last_clean":"2024-03-06T07:29:26.195993+0000","last_became_active":"2024-03-06T07:29:26.190641+0000","last_became_peered":"2024-03-06T07:29:26.190641+0000","last_unstale":"2024-03-06T07:29:26.195993+0000","last_undegraded":"2024-03-06T07:29:26.195993+0000","last_fullsized":"2024-03-06T07:29:26.195993+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:52.210630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.177373+0000","last_change":"2024-03-06T07:29:26.177373+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.177373+0000","last_undegraded":"2024-03-06T07:29:26.177373+0000","last_fullsized":"2024-03-06T07:29:26.177373+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198049+0000","last_change":"2024-03-06T07:29:26.193713+0000","last_active":"2024-03-06T07:29:26.198049+0000","last_peered":"2024-03-06T07:29:26.198049+0000","last_clean":"2024-03-06T07:29:26.198049+0000","last_became_active":"2024-03-06T07:29:26.193389+0000","last_became_peered":"2024-03-06T07:29:26.193389+0000","last_unstale":"2024-03-06T07:29:26.198049+0000","last_undegraded":"2024-03-06T07:29:26.198049+0000","last_fullsized":"2024-03-06T07:29:26.198049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:58.185331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.172598+0000","last_change":"2024-03-06T07:29:26.172598+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.172598+0000","last_undegraded":"2024-03-06T07:29:26.172598+0000","last_fullsized":"2024-03-06T07:29:26.172598+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198317+0000","last_change":"2024-03-06T07:29:26.196747+0000","last_active":"2024-03-06T07:29:26.198317+0000","last_peered":"2024-03-06T07:29:26.198317+0000","last_clean":"2024-03-06T07:29:26.198317+0000","last_became_active":"2024-03-06T07:29:26.196446+0000","last_became_peered":"2024-03-06T07:29:26.196446+0000","last_unstale":"2024-03-06T07:29:26.198317+0000","last_undegraded":"2024-03-06T07:29:26.198317+0000","last_fullsized":"2024-03-06T07:29:26.198317+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:18.921700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.176362+0000","last_change":"2024-03-06T07:29:26.176362+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.176362+0000","last_undegraded":"2024-03-06T07:29:26.176362+0000","last_fullsized":"2024-03-06T07:29:26.176362+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.200181+0000","last_change":"2024-03-06T07:29:26.199027+0000","last_active":"2024-03-06T07:29:26.200181+0000","last_peered":"2024-03-06T07:29:26.200181+0000","last_clean":"2024-03-06T07:29:26.200181+0000","last_became_active":"2024-03-06T07:29:26.198748+0000","last_became_peered":"2024-03-06T07:29:26.198748+0000","last_unstale":"2024-03-06T07:29:26.200181+0000","last_undegraded":"2024-03-06T07:29:26.200181+0000","last_fullsized":"2024-03-06T07:29:26.200181+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:42.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.174000+0000","last_change":"2024-03-06T07:29:26.174000+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.174000+0000","last_undegraded":"2024-03-06T07:29:26.174000+0000","last_fullsized":"2024-03-06T07:29:26.174000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.221797+0000","last_change":"2024-03-06T07:29:26.196655+0000","last_active":"2024-03-06T07:29:27.221797+0000","last_peered":"2024-03-06T07:29:27.221797+0000","last_clean":"2024-03-06T07:29:27.221797+0000","last_became_active":"2024-03-06T07:29:26.196370+0000","last_became_peered":"2024-03-06T07:29:26.196370+0000","last_unstale":"2024-03-06T07:29:27.221797+0000","last_undegraded":"2024-03-06T07:29:27.221797+0000","last_fullsized":"2024-03-06T07:29:27.221797+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.163810+0000","last_change":"2024-03-06T07:29:26.163810+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.163810+0000","last_undegraded":"2024-03-06T07:29:26.163810+0000","last_fullsized":"2024-03-06T07:29:26.163810+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.198276+0000","last_change":"2024-03-06T07:29:26.195806+0000","last_active":"2024-03-06T07:29:26.198276+0000","last_peered":"2024-03-06T07:29:26.198276+0000","last_clean":"2024-03-06T07:29:26.198276+0000","last_became_active":"2024-03-06T07:29:26.195455+0000","last_became_peered":"2024-03-06T07:29:26.195455+0000","last_unstale":"2024-03-06T07:29:26.198276+0000","last_undegraded":"2024-03-06T07:29:26.198276+0000","last_fullsized":"2024-03-06T07:29:26.198276+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:20.660770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.164058+0000","last_change":"2024-03-06T07:29:26.164058+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.164058+0000","last_undegraded":"2024-03-06T07:29:26.164058+0000","last_fullsized":"2024-03-06T07:29:26.164058+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.201349+0000","last_change":"2024-03-06T07:29:26.199356+0000","last_active":"2024-03-06T07:29:26.201349+0000","last_peered":"2024-03-06T07:29:26.201349+0000","last_clean":"2024-03-06T07:29:26.201349+0000","last_became_active":"2024-03-06T07:29:26.199015+0000","last_became_peered":"2024-03-06T07:29:26.199015+0000","last_unstale":"2024-03-06T07:29:26.201349+0000","last_undegraded":"2024-03-06T07:29:26.201349+0000","last_fullsized":"2024-03-06T07:29:26.201349+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:21.561181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.178828+0000","last_change":"2024-03-06T07:29:26.178828+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.178828+0000","last_undegraded":"2024-03-06T07:29:26.178828+0000","last_fullsized":"2024-03-06T07:29:26.178828+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-03-06T07:29:27.220161+0000","last_change":"2024-03-06T07:29:26.184910+0000","last_active":"2024-03-06T07:29:27.220161+0000","last_peered":"2024-03-06T07:29:27.220161+0000","last_clean":"2024-03-06T07:29:27.220161+0000","last_became_active":"2024-03-06T07:29:26.183588+0000","last_became_peered":"2024-03-06T07:29:26.183588+0000","last_unstale":"2024-03-06T07:29:27.220161+0000","last_undegraded":"2024-03-06T07:29:27.220161+0000","last_fullsized":"2024-03-06T07:29:27.220161+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.179731+0000","last_change":"2024-03-06T07:29:26.179731+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.179731+0000","last_undegraded":"2024-03-06T07:29:26.179731+0000","last_fullsized":"2024-03-06T07:29:26.179731+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.189832+0000","last_change":"2024-03-06T07:29:26.185660+0000","last_active":"2024-03-06T07:29:26.189832+0000","last_peered":"2024-03-06T07:29:26.189832+0000","last_clean":"2024-03-06T07:29:26.189832+0000","last_became_active":"2024-03-06T07:29:26.185316+0000","last_became_peered":"2024-03-06T07:29:26.185316+0000","last_unstale":"2024-03-06T07:29:26.189832+0000","last_undegraded":"2024-03-06T07:29:26.189832+0000","last_fullsized":"2024-03-06T07:29:26.189832+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:53:25.384746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:29:26.175672+0000","last_change":"2024-03-06T07:29:26.175672+0000","last_active":"2024-03-06T07:29:26.150738+0000","last_peered":"2024-03-06T07:29:26.150738+0000","last_clean":"2024-03-06T07:29:26.150738+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:29:26.175672+0000","last_undegraded":"2024-03-06T07:29:26.175672+0000","last_fullsized":"2024-03-06T07:29:26.175672+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.203496+0000","last_change":"2024-03-06T07:29:26.200006+0000","last_active":"2024-03-06T07:29:26.203496+0000","last_peered":"2024-03-06T07:29:26.203496+0000","last_clean":"2024-03-06T07:29:26.203496+0000","last_became_active":"2024-03-06T07:29:26.198345+0000","last_became_peered":"2024-03-06T07:29:26.198345+0000","last_unstale":"2024-03-06T07:29:26.203496+0000","last_undegraded":"2024-03-06T07:29:26.203496+0000","last_fullsized":"2024-03-06T07:29:26.203496+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.255706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":209,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:29:26.184437+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:29:26.184437+0000","last_peered":"2024-03-06T07:29:26.184437+0000","last_clean":"2024-03-06T07:29:26.184437+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:29:26.184437+0000","last_undegraded":"2024-03-06T07:29:26.184437+0000","last_fullsized":"2024-03-06T07:29:26.184437+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509468,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509469,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335436,"statfs":{"total":96636764160,"available":96599486464,"internally_reserved":0,"allocated":1089536,"data_stored":788026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542173,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":197658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":507904,"data_stored":203925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1101824,"data_stored":794293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:32.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-03-06T07:29:32.740 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.180481+0000', 'last_change': '2024-03-06T07:29:26.180481+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.180481+0000', 'last_undegraded': '2024-03-06T07:29:26.180481+0000', 'last_fullsized': '2024-03-06T07:29:26.180481+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:29:32.740 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.185266+0000', 'last_change': '2024-03-06T07:29:26.185266+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.185266+0000', 'last_undegraded': '2024-03-06T07:29:26.185266+0000', 'last_fullsized': '2024-03-06T07:29:26.185266+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.186354+0000', 'last_change': '2024-03-06T07:29:26.186354+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.186354+0000', 'last_undegraded': '2024-03-06T07:29:26.186354+0000', 'last_fullsized': '2024-03-06T07:29:26.186354+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.180082+0000', 'last_change': '2024-03-06T07:29:26.180082+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.180082+0000', 'last_undegraded': '2024-03-06T07:29:26.180082+0000', 'last_fullsized': '2024-03-06T07:29:26.180082+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179026+0000', 'last_change': '2024-03-06T07:29:26.179026+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179026+0000', 'last_undegraded': '2024-03-06T07:29:26.179026+0000', 'last_fullsized': '2024-03-06T07:29:26.179026+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.165169+0000', 'last_change': '2024-03-06T07:29:26.165169+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.165169+0000', 'last_undegraded': '2024-03-06T07:29:26.165169+0000', 'last_fullsized': '2024-03-06T07:29:26.165169+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-03-06T07:29:32.741 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.173991+0000', 'last_change': '2024-03-06T07:29:26.173991+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.173991+0000', 'last_undegraded': '2024-03-06T07:29:26.173991+0000', 'last_fullsized': '2024-03-06T07:29:26.173991+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179661+0000', 'last_change': '2024-03-06T07:29:26.179661+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179661+0000', 'last_undegraded': '2024-03-06T07:29:26.179661+0000', 'last_fullsized': '2024-03-06T07:29:26.179661+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179688+0000', 'last_change': '2024-03-06T07:29:26.179688+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179688+0000', 'last_undegraded': '2024-03-06T07:29:26.179688+0000', 'last_fullsized': '2024-03-06T07:29:26.179688+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179130+0000', 'last_change': '2024-03-06T07:29:26.179130+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179130+0000', 'last_undegraded': '2024-03-06T07:29:26.179130+0000', 'last_fullsized': '2024-03-06T07:29:26.179130+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.177134+0000', 'last_change': '2024-03-06T07:29:26.177134+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.177134+0000', 'last_undegraded': '2024-03-06T07:29:26.177134+0000', 'last_fullsized': '2024-03-06T07:29:26.177134+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.186917+0000', 'last_change': '2024-03-06T07:29:26.186917+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.186917+0000', 'last_undegraded': '2024-03-06T07:29:26.186917+0000', 'last_fullsized': '2024-03-06T07:29:26.186917+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:29:32.742 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.180437+0000', 'last_change': '2024-03-06T07:29:26.180437+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.180437+0000', 'last_undegraded': '2024-03-06T07:29:26.180437+0000', 'last_fullsized': '2024-03-06T07:29:26.180437+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.167433+0000', 'last_change': '2024-03-06T07:29:26.167433+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.167433+0000', 'last_undegraded': '2024-03-06T07:29:26.167433+0000', 'last_fullsized': '2024-03-06T07:29:26.167433+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.172611+0000', 'last_change': '2024-03-06T07:29:26.172611+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.172611+0000', 'last_undegraded': '2024-03-06T07:29:26.172611+0000', 'last_fullsized': '2024-03-06T07:29:26.172611+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.187650+0000', 'last_change': '2024-03-06T07:29:26.187650+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.187650+0000', 'last_undegraded': '2024-03-06T07:29:26.187650+0000', 'last_fullsized': '2024-03-06T07:29:26.187650+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.189091+0000', 'last_change': '2024-03-06T07:29:26.189091+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.189091+0000', 'last_undegraded': '2024-03-06T07:29:26.189091+0000', 'last_fullsized': '2024-03-06T07:29:26.189091+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.171250+0000', 'last_change': '2024-03-06T07:29:26.171250+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.171250+0000', 'last_undegraded': '2024-03-06T07:29:26.171250+0000', 'last_fullsized': '2024-03-06T07:29:26.171250+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2024-03-06T07:29:32.743 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.164817+0000', 'last_change': '2024-03-06T07:29:26.164817+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.164817+0000', 'last_undegraded': '2024-03-06T07:29:26.164817+0000', 'last_fullsized': '2024-03-06T07:29:26.164817+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.176355+0000', 'last_change': '2024-03-06T07:29:26.176355+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.176355+0000', 'last_undegraded': '2024-03-06T07:29:26.176355+0000', 'last_fullsized': '2024-03-06T07:29:26.176355+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.170468+0000', 'last_change': '2024-03-06T07:29:26.170468+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.170468+0000', 'last_undegraded': '2024-03-06T07:29:26.170468+0000', 'last_fullsized': '2024-03-06T07:29:26.170468+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.171475+0000', 'last_change': '2024-03-06T07:29:26.171475+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.171475+0000', 'last_undegraded': '2024-03-06T07:29:26.171475+0000', 'last_fullsized': '2024-03-06T07:29:26.171475+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.174866+0000', 'last_change': '2024-03-06T07:29:26.174866+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.174866+0000', 'last_undegraded': '2024-03-06T07:29:26.174866+0000', 'last_fullsized': '2024-03-06T07:29:26.174866+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.184051+0000', 'last_change': '2024-03-06T07:29:26.184051+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.184051+0000', 'last_undegraded': '2024-03-06T07:29:26.184051+0000', 'last_fullsized': '2024-03-06T07:29:26.184051+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.744 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.183973+0000', 'last_change': '2024-03-06T07:29:26.183973+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.183973+0000', 'last_undegraded': '2024-03-06T07:29:26.183973+0000', 'last_fullsized': '2024-03-06T07:29:26.183973+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.171100+0000', 'last_change': '2024-03-06T07:29:26.171100+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.171100+0000', 'last_undegraded': '2024-03-06T07:29:26.171100+0000', 'last_fullsized': '2024-03-06T07:29:26.171100+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.182982+0000', 'last_change': '2024-03-06T07:29:26.182982+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.182982+0000', 'last_undegraded': '2024-03-06T07:29:26.182982+0000', 'last_fullsized': '2024-03-06T07:29:26.182982+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.166901+0000', 'last_change': '2024-03-06T07:29:26.166901+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.166901+0000', 'last_undegraded': '2024-03-06T07:29:26.166901+0000', 'last_fullsized': '2024-03-06T07:29:26.166901+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.173821+0000', 'last_change': '2024-03-06T07:29:26.173821+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.173821+0000', 'last_undegraded': '2024-03-06T07:29:26.173821+0000', 'last_fullsized': '2024-03-06T07:29:26.173821+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.183386+0000', 'last_change': '2024-03-06T07:29:26.183386+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.183386+0000', 'last_undegraded': '2024-03-06T07:29:26.183386+0000', 'last_fullsized': '2024-03-06T07:29:26.183386+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:29:32.745 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.182043+0000', 'last_change': '2024-03-06T07:29:26.182043+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.182043+0000', 'last_undegraded': '2024-03-06T07:29:26.182043+0000', 'last_fullsized': '2024-03-06T07:29:26.182043+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.178981+0000', 'last_change': '2024-03-06T07:29:26.178981+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.178981+0000', 'last_undegraded': '2024-03-06T07:29:26.178981+0000', 'last_fullsized': '2024-03-06T07:29:26.178981+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.175828+0000', 'last_change': '2024-03-06T07:29:26.175828+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.175828+0000', 'last_undegraded': '2024-03-06T07:29:26.175828+0000', 'last_fullsized': '2024-03-06T07:29:26.175828+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.171753+0000', 'last_change': '2024-03-06T07:29:26.171753+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.171753+0000', 'last_undegraded': '2024-03-06T07:29:26.171753+0000', 'last_fullsized': '2024-03-06T07:29:26.171753+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.188604+0000', 'last_change': '2024-03-06T07:29:26.188604+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.188604+0000', 'last_undegraded': '2024-03-06T07:29:26.188604+0000', 'last_fullsized': '2024-03-06T07:29:26.188604+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179348+0000', 'last_change': '2024-03-06T07:29:26.179348+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179348+0000', 'last_undegraded': '2024-03-06T07:29:26.179348+0000', 'last_fullsized': '2024-03-06T07:29:26.179348+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:29:32.746 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.166868+0000', 'last_change': '2024-03-06T07:29:26.166868+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.166868+0000', 'last_undegraded': '2024-03-06T07:29:26.166868+0000', 'last_fullsized': '2024-03-06T07:29:26.166868+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179847+0000', 'last_change': '2024-03-06T07:29:26.179847+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179847+0000', 'last_undegraded': '2024-03-06T07:29:26.179847+0000', 'last_fullsized': '2024-03-06T07:29:26.179847+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.170598+0000', 'last_change': '2024-03-06T07:29:26.170598+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.170598+0000', 'last_undegraded': '2024-03-06T07:29:26.170598+0000', 'last_fullsized': '2024-03-06T07:29:26.170598+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.175797+0000', 'last_change': '2024-03-06T07:29:26.175797+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.175797+0000', 'last_undegraded': '2024-03-06T07:29:26.175797+0000', 'last_fullsized': '2024-03-06T07:29:26.175797+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.163731+0000', 'last_change': '2024-03-06T07:29:26.163731+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.163731+0000', 'last_undegraded': '2024-03-06T07:29:26.163731+0000', 'last_fullsized': '2024-03-06T07:29:26.163731+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.172162+0000', 'last_change': '2024-03-06T07:29:26.172162+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.172162+0000', 'last_undegraded': '2024-03-06T07:29:26.172162+0000', 'last_fullsized': '2024-03-06T07:29:26.172162+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:29:32.747 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.176443+0000', 'last_change': '2024-03-06T07:29:26.176443+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.176443+0000', 'last_undegraded': '2024-03-06T07:29:26.176443+0000', 'last_fullsized': '2024-03-06T07:29:26.176443+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.172544+0000', 'last_change': '2024-03-06T07:29:26.172544+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.172544+0000', 'last_undegraded': '2024-03-06T07:29:26.172544+0000', 'last_fullsized': '2024-03-06T07:29:26.172544+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.173271+0000', 'last_change': '2024-03-06T07:29:26.173271+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.173271+0000', 'last_undegraded': '2024-03-06T07:29:26.173271+0000', 'last_fullsized': '2024-03-06T07:29:26.173271+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.164678+0000', 'last_change': '2024-03-06T07:29:26.164678+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.164678+0000', 'last_undegraded': '2024-03-06T07:29:26.164678+0000', 'last_fullsized': '2024-03-06T07:29:26.164678+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.177373+0000', 'last_change': '2024-03-06T07:29:26.177373+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.177373+0000', 'last_undegraded': '2024-03-06T07:29:26.177373+0000', 'last_fullsized': '2024-03-06T07:29:26.177373+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.748 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.172598+0000', 'last_change': '2024-03-06T07:29:26.172598+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.172598+0000', 'last_undegraded': '2024-03-06T07:29:26.172598+0000', 'last_fullsized': '2024-03-06T07:29:26.172598+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.176362+0000', 'last_change': '2024-03-06T07:29:26.176362+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.176362+0000', 'last_undegraded': '2024-03-06T07:29:26.176362+0000', 'last_fullsized': '2024-03-06T07:29:26.176362+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.174000+0000', 'last_change': '2024-03-06T07:29:26.174000+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.174000+0000', 'last_undegraded': '2024-03-06T07:29:26.174000+0000', 'last_fullsized': '2024-03-06T07:29:26.174000+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.163810+0000', 'last_change': '2024-03-06T07:29:26.163810+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.163810+0000', 'last_undegraded': '2024-03-06T07:29:26.163810+0000', 'last_fullsized': '2024-03-06T07:29:26.163810+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.164058+0000', 'last_change': '2024-03-06T07:29:26.164058+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.164058+0000', 'last_undegraded': '2024-03-06T07:29:26.164058+0000', 'last_fullsized': '2024-03-06T07:29:26.164058+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.178828+0000', 'last_change': '2024-03-06T07:29:26.178828+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.178828+0000', 'last_undegraded': '2024-03-06T07:29:26.178828+0000', 'last_fullsized': '2024-03-06T07:29:26.178828+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:29:32.749 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:29:32.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.179731+0000', 'last_change': '2024-03-06T07:29:26.179731+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.179731+0000', 'last_undegraded': '2024-03-06T07:29:26.179731+0000', 'last_fullsized': '2024-03-06T07:29:26.179731+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:29:32.750 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:29:32.750 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:29:26.175672+0000', 'last_change': '2024-03-06T07:29:26.175672+0000', 'last_active': '2024-03-06T07:29:26.150738+0000', 'last_peered': '2024-03-06T07:29:26.150738+0000', 'last_clean': '2024-03-06T07:29:26.150738+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:29:26.175672+0000', 'last_undegraded': '2024-03-06T07:29:26.175672+0000', 'last_fullsized': '2024-03-06T07:29:26.175672+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'last_clean_scrub_stamp': '2024-03-06T07:29:26.150738+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:29:32.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:29:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.002+0000 7f551da00640 1 Processor -- start 2024-03-06T07:29:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.002+0000 7f551da00640 1 -- start start 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f551da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f55181455e0 0x7f551815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f551da00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f551806daf0 0x7f551806df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f551da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5518070cd0 con 0x7f55180cc4a0 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f551da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55180cf690 con 0x7f55181455e0 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f551da00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5518070a80 con 0x7f551806daf0 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f550fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f55180cc4a0 0x7f55180cef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57578/0 (socket says 172.21.15.6:57578) 2024-03-06T07:29:33.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f550fe00640 1 -- 172.21.15.6:0/1319176987 learned_addr learned my addr 172.21.15.6:0/1319176987 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.004+0000 7f5517e00640 1 --2- 172.21.15.6:0/1319176987 >> v2:172.21.15.6:3300/0 conn(0x7f55181455e0 0x7f551815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f551d000640 1 --2- 172.21.15.6:0/1319176987 >> v2:172.21.15.178:3300/0 conn(0x7f551806daf0 0x7f551806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f5517400640 1 -- 172.21.15.6:0/1319176987 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022089750 0 0) 0x7f5518070cd0 con 0x7f55180cc4a0 2024-03-06T07:29:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f5517400640 1 -- 172.21.15.6:0/1319176987 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc003520 con 0x7f55180cc4a0 2024-03-06T07:29:33.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f5517e00640 1 -- 172.21.15.6:0/1319176987 >> v2:172.21.15.178:3300/0 conn(0x7f551806daf0 msgr2=0x7f551806df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f5517e00640 1 --2- 172.21.15.6:0/1319176987 >> v2:172.21.15.178:3300/0 conn(0x7f551806daf0 0x7f551806df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f5517e00640 1 -- 172.21.15.6:0/1319176987 >> v1:172.21.15.6:6789/0 conn(0x7f55180cc4a0 legacy=0x7f55180cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.005+0000 7f5517e00640 1 -- 172.21.15.6:0/1319176987 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5518070fb0 con 0x7f55181455e0 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.006+0000 7f5517e00640 1 --2- 172.21.15.6:0/1319176987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55181455e0 0x7f551815c2f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f55180ce810 tx=0x7f550802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196f83daeab58a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f5517400640 1 -- 172.21.15.6:0/1319176987 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080315a0 con 0x7f55181455e0 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f5517400640 1 -- 172.21.15.6:0/1319176987 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5508037070 con 0x7f55181455e0 2024-03-06T07:29:33.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f5517400640 1 -- 172.21.15.6:0/1319176987 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508031940 con 0x7f55181455e0 2024-03-06T07:29:33.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 -- 172.21.15.6:0/1319176987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55181455e0 msgr2=0x7f551815c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 --2- 172.21.15.6:0/1319176987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55181455e0 0x7f551815c2f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f55180ce810 tx=0x7f550802b840 comp rx=0 tx=0).stop 2024-03-06T07:29:33.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 -- 172.21.15.6:0/1319176987 shutdown_connections 2024-03-06T07:29:33.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 --2- 172.21.15.6:0/1319176987 >> v2:172.21.15.178:3300/0 conn(0x7f551806daf0 0x7f551806df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 --2- 172.21.15.6:0/1319176987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55181455e0 0x7f551815c2f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 -- 172.21.15.6:0/1319176987 >> 172.21.15.6:0/1319176987 conn(0x7f55180b17c0 msgr2=0x7f55180b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.007+0000 7f551da00640 1 -- 172.21.15.6:0/1319176987 shutdown_connections 2024-03-06T07:29:33.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.008+0000 7f551da00640 1 -- 172.21.15.6:0/1319176987 wait complete. 2024-03-06T07:29:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.010+0000 7f551da00640 1 Processor -- start 2024-03-06T07:29:33.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.010+0000 7f551da00640 1 -- start start 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.011+0000 7f551da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 0x7f5518137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.011+0000 7f551da00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55180cc4a0 0x7f5518142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.011+0000 7f551da00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55181455e0 0x7f55180b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.011+0000 7f551da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5518071570 con 0x7f551806daf0 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.011+0000 7f551da00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551806e460 con 0x7f55181455e0 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.011+0000 7f551da00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55181459c0 con 0x7f55180cc4a0 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 0x7f5518137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 0x7f5518137370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43248/0 (socket says 172.21.15.6:43248) 2024-03-06T07:29:33.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 -- 172.21.15.6:0/1633754507 learned_addr learned my addr 172.21.15.6:0/1633754507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:33.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 -- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55180cc4a0 msgr2=0x7f5518142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55180cc4a0 0x7f5518142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 -- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55181455e0 msgr2=0x7f55180b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55181455e0 0x7f55180b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 -- 172.21.15.6:0/1633754507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551813d050 con 0x7f551806daf0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.012+0000 7f5517e00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 0x7f5518137370 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5508000c00 tx=0x7f55080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.013+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080317a0 con 0x7f551806daf0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.013+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5508031940 con 0x7f551806daf0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.013+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508031d10 con 0x7f551806daf0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.013+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5518137b20 con 0x7f551806daf0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.013+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55181380e0 con 0x7f551806daf0 2024-03-06T07:29:33.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.014+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551804f330 con 0x7f551806daf0 2024-03-06T07:29:33.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.015+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5508032430 con 0x7f551806daf0 2024-03-06T07:29:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.016+0000 7f5514c00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54f406b4d0 0x7f54f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.016+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f550802f030 con 0x7f551806daf0 2024-03-06T07:29:33.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.017+0000 7f550fe00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54f406b4d0 0x7f54f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.018+0000 7f550fe00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54f406b4d0 0x7f54f406d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5500000f60 tx=0x7f55000078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.022+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f550804c060 con 0x7f551806daf0 2024-03-06T07:29:33.137 INFO:tasks.ceph.mds.b.smithi178.stderr:2024-03-06T07:29:33.135+0000 7fd03e400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:29:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.295+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f55180b2950 con 0x7f551806daf0 2024-03-06T07:29:33.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.296+0000 7f5514c00640 1 -- 172.21.15.6:0/1633754507 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1258 (secure 0 0 0) 0x7f550807bbd0 con 0x7f551806daf0 2024-03-06T07:29:33.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:33.299 INFO:teuthology.orchestra.run.smithi006.stdout:{"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":149,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":8,"num_up_osds":8,"osd_up_since":1709710036,"num_in_osds":8,"osd_in_since":1709710028,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":55},{"state_name":"active+clean","count":42}],"num_pgs":97,"num_pools":3,"num_objects":8,"data_bytes":592002,"bytes_used":260677632,"bytes_avail":772833435648,"bytes_total":773094113280,"inactive_pgs_ratio":0.56701028347015381,"write_bytes_sec":682,"read_op_per_sec":0,"write_op_per_sec":1},"fsmap":{"epoch":26,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":5562}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-06T07:28:27.458008+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54f406b4d0 msgr2=0x7f54f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54f406b4d0 0x7f54f406d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5500000f60 tx=0x7f55000078f0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 msgr2=0x7f5518137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 0x7f5518137370 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5508000c00 tx=0x7f55080046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 shutdown_connections 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54f406b4d0 0x7f54f406d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55181455e0 0x7f55180b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55180cc4a0 0x7f5518142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 --2- 172.21.15.6:0/1633754507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f551806daf0 0x7f5518137370 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 >> 172.21.15.6:0/1633754507 conn(0x7f55180b17c0 msgr2=0x7f5518073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 shutdown_connections 2024-03-06T07:29:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.299+0000 7f551da00640 1 -- 172.21.15.6:0/1633754507 wait complete. 2024-03-06T07:29:33.318 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:29:33.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.535+0000 7f5e3da00640 1 Processor -- start 2024-03-06T07:29:33.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.535+0000 7f5e3da00640 1 -- start start 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e3da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e3815e810 0x7f5e3815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e3da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3815ed10 con 0x7f5e380cd060 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e3da00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3815eff0 con 0x7f5e381552b0 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e3da00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3815f2d0 con 0x7f5e3815e810 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e3d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e3815e810 0x7f5e3815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e2f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5e380cd060 0x7f5e380b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57582/0 (socket says 172.21.15.6:57582) 2024-03-06T07:29:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.537+0000 7f5e2f400640 1 -- 172.21.15.6:0/2480380364 learned_addr learned my addr 172.21.15.6:0/2480380364 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.538+0000 7f5e3d000640 1 -- 172.21.15.6:0/2480380364 >> v1:172.21.15.178:6789/0 conn(0x7f5e381552b0 legacy=0x7f5e380cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.538+0000 7f5e3d000640 1 -- 172.21.15.6:0/2480380364 >> v1:172.21.15.6:6789/0 conn(0x7f5e380cd060 legacy=0x7f5e380b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.538+0000 7f5e3d000640 1 -- 172.21.15.6:0/2480380364 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3815f5b0 con 0x7f5e3815e810 2024-03-06T07:29:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.539+0000 7f5e3d000640 1 --2- 172.21.15.6:0/2480380364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3815b1d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097a0 tx=0x7f5e3002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a4a7292b24f7a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.539+0000 7f5e37400640 1 -- 172.21.15.6:0/2480380364 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002f8a0 con 0x7f5e3815e810 2024-03-06T07:29:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.539+0000 7f5e37400640 1 -- 172.21.15.6:0/2480380364 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3002fa40 con 0x7f5e3815e810 2024-03-06T07:29:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.539+0000 7f5e37400640 1 -- 172.21.15.6:0/2480380364 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002fe10 con 0x7f5e3815e810 2024-03-06T07:29:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 -- 172.21.15.6:0/2480380364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 msgr2=0x7f5e3815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 --2- 172.21.15.6:0/2480380364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3815b1d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097a0 tx=0x7f5e3002dae0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 -- 172.21.15.6:0/2480380364 shutdown_connections 2024-03-06T07:29:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 --2- 172.21.15.6:0/2480380364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3815b1d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 -- 172.21.15.6:0/2480380364 >> 172.21.15.6:0/2480380364 conn(0x7f5e3806b3d0 msgr2=0x7f5e3806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 -- 172.21.15.6:0/2480380364 shutdown_connections 2024-03-06T07:29:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.540+0000 7f5e3da00640 1 -- 172.21.15.6:0/2480380364 wait complete. 2024-03-06T07:29:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.542+0000 7f5e3da00640 1 Processor -- start 2024-03-06T07:29:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.542+0000 7f5e3da00640 1 -- start start 2024-03-06T07:29:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3da00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e380cd060 0x7f5e38148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3da00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e381552b0 0x7f5e3815c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3814c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3806dd30 con 0x7f5e3815e810 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3da00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e380b3760 con 0x7f5e381552b0 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3da00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3806deb0 con 0x7f5e380cd060 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3814c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3814c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43264/0 (socket says 172.21.15.6:43264) 2024-03-06T07:29:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.543+0000 7f5e3d000640 1 -- 172.21.15.6:0/3902442809 learned_addr learned my addr 172.21.15.6:0/3902442809 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e2f400640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e381552b0 0x7f5e3815c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e37e00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e380cd060 0x7f5e38148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e3d000640 1 -- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e380cd060 msgr2=0x7f5e38148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e3d000640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e380cd060 0x7f5e38148310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e3d000640 1 -- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e381552b0 msgr2=0x7f5e3815c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e3d000640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e381552b0 0x7f5e3815c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e3d000640 1 -- 172.21.15.6:0/3902442809 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3815cb90 con 0x7f5e3815e810 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.544+0000 7f5e3d000640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3814c830 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5e3002db20 tx=0x7f5e30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.545+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30049070 con 0x7f5e3815e810 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.545+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e30039400 con 0x7f5e3815e810 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.545+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30006e10 con 0x7f5e3815e810 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.545+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3815ce70 con 0x7f5e3815e810 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.545+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3815d430 con 0x7f5e3815e810 2024-03-06T07:29:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.546+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3804f330 con 0x7f5e3815e810 2024-03-06T07:29:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.547+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e300397f0 con 0x7f5e3815e810 2024-03-06T07:29:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.548+0000 7f5e34c00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e1406b4d0 0x7f5e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.548+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e300ae320 con 0x7f5e3815e810 2024-03-06T07:29:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.548+0000 7f5e37e00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e1406b4d0 0x7f5e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.550+0000 7f5e37e00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e1406b4d0 0x7f5e1406d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5e28007fd0 tx=0x7f5e28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.552+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e30035030 con 0x7f5e3815e810 2024-03-06T07:29:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.750+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e380cf4a0 con 0x7f5e1406b4d0 2024-03-06T07:29:33.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.769+0000 7f5e34c00640 1 -- 172.21.15.6:0/3902442809 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251960 (secure 0 0 0) 0x7f5e28056850 con 0x7f5e1406b4d0 2024-03-06T07:29:33.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:33.777 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e1406b4d0 msgr2=0x7f5e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e1406b4d0 0x7f5e1406d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5e28007fd0 tx=0x7f5e28009040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 msgr2=0x7f5e3814c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3814c830 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5e3002db20 tx=0x7f5e30004730 comp rx=0 tx=0).stop 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 shutdown_connections 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e1406b4d0 0x7f5e1406d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.775+0000 7f5e3da00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e381552b0 0x7f5e3815c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.776+0000 7f5e3da00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e3815e810 0x7f5e3814c830 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.776+0000 7f5e3da00640 1 --2- 172.21.15.6:0/3902442809 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e380cd060 0x7f5e38148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.776+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 >> 172.21.15.6:0/3902442809 conn(0x7f5e3806b3d0 msgr2=0x7f5e380cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.776+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 shutdown_connections 2024-03-06T07:29:33.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:33.776+0000 7f5e3da00640 1 -- 172.21.15.6:0/3902442809 wait complete. 2024-03-06T07:29:33.797 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-03-06T07:29:33.480812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259004,"kb_used_data":5384,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754715716,"statfs":{"total":773094113280,"available":772828893184,"internally_reserved":0,"allocated":5513216,"data_stored":2921556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.320405"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359873+0000","last_change":"2024-03-06T07:29:27.238818+0000","last_active":"2024-03-06T07:29:29.359873+0000","last_peered":"2024-03-06T07:29:29.359873+0000","last_clean":"2024-03-06T07:29:29.359873+0000","last_became_active":"2024-03-06T07:29:27.238533+0000","last_became_peered":"2024-03-06T07:29:27.238533+0000","last_unstale":"2024-03-06T07:29:29.359873+0000","last_undegraded":"2024-03-06T07:29:29.359873+0000","last_fullsized":"2024-03-06T07:29:29.359873+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:52.311496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197960+0000","last_change":"2024-03-06T07:29:27.232219+0000","last_active":"2024-03-06T07:29:29.197960+0000","last_peered":"2024-03-06T07:29:29.197960+0000","last_clean":"2024-03-06T07:29:29.197960+0000","last_became_active":"2024-03-06T07:29:27.231915+0000","last_became_peered":"2024-03-06T07:29:27.231915+0000","last_unstale":"2024-03-06T07:29:29.197960+0000","last_undegraded":"2024-03-06T07:29:29.197960+0000","last_fullsized":"2024-03-06T07:29:29.197960+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:04.309774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240247+0000","last_change":"2024-03-06T07:29:27.231359+0000","last_active":"2024-03-06T07:29:28.240247+0000","last_peered":"2024-03-06T07:29:28.240247+0000","last_clean":"2024-03-06T07:29:28.240247+0000","last_became_active":"2024-03-06T07:29:27.231038+0000","last_became_peered":"2024-03-06T07:29:27.231038+0000","last_unstale":"2024-03-06T07:29:28.240247+0000","last_undegraded":"2024-03-06T07:29:28.240247+0000","last_fullsized":"2024-03-06T07:29:28.240247+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:48.483161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193663+0000","last_change":"2024-03-06T07:29:27.219886+0000","last_active":"2024-03-06T07:29:28.193663+0000","last_peered":"2024-03-06T07:29:28.193663+0000","last_clean":"2024-03-06T07:29:28.193663+0000","last_became_active":"2024-03-06T07:29:27.219534+0000","last_became_peered":"2024-03-06T07:29:27.219534+0000","last_unstale":"2024-03-06T07:29:28.193663+0000","last_undegraded":"2024-03-06T07:29:28.193663+0000","last_fullsized":"2024-03-06T07:29:28.193663+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.040231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197432+0000","last_change":"2024-03-06T07:29:27.227793+0000","last_active":"2024-03-06T07:29:29.197432+0000","last_peered":"2024-03-06T07:29:29.197432+0000","last_clean":"2024-03-06T07:29:29.197432+0000","last_became_active":"2024-03-06T07:29:27.226210+0000","last_became_peered":"2024-03-06T07:29:27.226210+0000","last_unstale":"2024-03-06T07:29:29.197432+0000","last_undegraded":"2024-03-06T07:29:29.197432+0000","last_fullsized":"2024-03-06T07:29:29.197432+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:19.753745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.190958+0000","last_change":"2024-03-06T07:29:27.210023+0000","last_active":"2024-03-06T07:29:28.190958+0000","last_peered":"2024-03-06T07:29:28.190958+0000","last_clean":"2024-03-06T07:29:28.190958+0000","last_became_active":"2024-03-06T07:29:27.209281+0000","last_became_peered":"2024-03-06T07:29:27.209281+0000","last_unstale":"2024-03-06T07:29:28.190958+0000","last_undegraded":"2024-03-06T07:29:28.190958+0000","last_fullsized":"2024-03-06T07:29:28.190958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:46.291448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.357564+0000","last_change":"2024-03-06T07:29:27.226824+0000","last_active":"2024-03-06T07:29:29.357564+0000","last_peered":"2024-03-06T07:29:29.357564+0000","last_clean":"2024-03-06T07:29:29.357564+0000","last_became_active":"2024-03-06T07:29:27.226498+0000","last_became_peered":"2024-03-06T07:29:27.226498+0000","last_unstale":"2024-03-06T07:29:29.357564+0000","last_undegraded":"2024-03-06T07:29:29.357564+0000","last_fullsized":"2024-03-06T07:29:29.357564+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:55.566106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192296+0000","last_change":"2024-03-06T07:29:27.209995+0000","last_active":"2024-03-06T07:29:28.192296+0000","last_peered":"2024-03-06T07:29:28.192296+0000","last_clean":"2024-03-06T07:29:28.192296+0000","last_became_active":"2024-03-06T07:29:27.206041+0000","last_became_peered":"2024-03-06T07:29:27.206041+0000","last_unstale":"2024-03-06T07:29:28.192296+0000","last_undegraded":"2024-03-06T07:29:28.192296+0000","last_fullsized":"2024-03-06T07:29:28.192296+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:09.195915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194576+0000","last_change":"2024-03-06T07:29:27.226491+0000","last_active":"2024-03-06T07:29:28.194576+0000","last_peered":"2024-03-06T07:29:28.194576+0000","last_clean":"2024-03-06T07:29:28.194576+0000","last_became_active":"2024-03-06T07:29:27.225897+0000","last_became_peered":"2024-03-06T07:29:27.225897+0000","last_unstale":"2024-03-06T07:29:28.194576+0000","last_undegraded":"2024-03-06T07:29:28.194576+0000","last_fullsized":"2024-03-06T07:29:28.194576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:19.753745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193571+0000","last_change":"2024-03-06T07:29:27.210338+0000","last_active":"2024-03-06T07:29:28.193571+0000","last_peered":"2024-03-06T07:29:28.193571+0000","last_clean":"2024-03-06T07:29:28.193571+0000","last_became_active":"2024-03-06T07:29:27.207720+0000","last_became_peered":"2024-03-06T07:29:27.207720+0000","last_unstale":"2024-03-06T07:29:28.193571+0000","last_undegraded":"2024-03-06T07:29:28.193571+0000","last_fullsized":"2024-03-06T07:29:28.193571+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:41.530505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192842+0000","last_change":"2024-03-06T07:29:27.224388+0000","last_active":"2024-03-06T07:29:28.192842+0000","last_peered":"2024-03-06T07:29:28.192842+0000","last_clean":"2024-03-06T07:29:28.192842+0000","last_became_active":"2024-03-06T07:29:27.223984+0000","last_became_peered":"2024-03-06T07:29:27.223984+0000","last_unstale":"2024-03-06T07:29:28.192842+0000","last_undegraded":"2024-03-06T07:29:28.192842+0000","last_fullsized":"2024-03-06T07:29:28.192842+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:19.935722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197378+0000","last_change":"2024-03-06T07:29:27.229326+0000","last_active":"2024-03-06T07:29:29.197378+0000","last_peered":"2024-03-06T07:29:29.197378+0000","last_clean":"2024-03-06T07:29:29.197378+0000","last_became_active":"2024-03-06T07:29:27.228999+0000","last_became_peered":"2024-03-06T07:29:27.228999+0000","last_unstale":"2024-03-06T07:29:29.197378+0000","last_undegraded":"2024-03-06T07:29:29.197378+0000","last_fullsized":"2024-03-06T07:29:29.197378+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.040231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194301+0000","last_change":"2024-03-06T07:29:27.234158+0000","last_active":"2024-03-06T07:29:28.194301+0000","last_peered":"2024-03-06T07:29:28.194301+0000","last_clean":"2024-03-06T07:29:28.194301+0000","last_became_active":"2024-03-06T07:29:27.233803+0000","last_became_peered":"2024-03-06T07:29:27.233803+0000","last_unstale":"2024-03-06T07:29:28.194301+0000","last_undegraded":"2024-03-06T07:29:28.194301+0000","last_fullsized":"2024-03-06T07:29:28.194301+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:29.033715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192721+0000","last_change":"2024-03-06T07:29:27.221609+0000","last_active":"2024-03-06T07:29:28.192721+0000","last_peered":"2024-03-06T07:29:28.192721+0000","last_clean":"2024-03-06T07:29:28.192721+0000","last_became_active":"2024-03-06T07:29:27.221287+0000","last_became_peered":"2024-03-06T07:29:27.221287+0000","last_unstale":"2024-03-06T07:29:28.192721+0000","last_undegraded":"2024-03-06T07:29:28.192721+0000","last_fullsized":"2024-03-06T07:29:28.192721+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:13.654418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191093+0000","last_change":"2024-03-06T07:29:27.220611+0000","last_active":"2024-03-06T07:29:28.191093+0000","last_peered":"2024-03-06T07:29:28.191093+0000","last_clean":"2024-03-06T07:29:28.191093+0000","last_became_active":"2024-03-06T07:29:27.220033+0000","last_became_peered":"2024-03-06T07:29:27.220033+0000","last_unstale":"2024-03-06T07:29:28.191093+0000","last_undegraded":"2024-03-06T07:29:28.191093+0000","last_fullsized":"2024-03-06T07:29:28.191093+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.273553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358072+0000","last_change":"2024-03-06T07:29:27.226127+0000","last_active":"2024-03-06T07:29:29.358072+0000","last_peered":"2024-03-06T07:29:29.358072+0000","last_clean":"2024-03-06T07:29:29.358072+0000","last_became_active":"2024-03-06T07:29:27.225234+0000","last_became_peered":"2024-03-06T07:29:27.225234+0000","last_unstale":"2024-03-06T07:29:29.358072+0000","last_undegraded":"2024-03-06T07:29:29.358072+0000","last_fullsized":"2024-03-06T07:29:29.358072+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:34.153592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240623+0000","last_change":"2024-03-06T07:29:27.218384+0000","last_active":"2024-03-06T07:29:28.240623+0000","last_peered":"2024-03-06T07:29:28.240623+0000","last_clean":"2024-03-06T07:29:28.240623+0000","last_became_active":"2024-03-06T07:29:27.218081+0000","last_became_peered":"2024-03-06T07:29:27.218081+0000","last_unstale":"2024-03-06T07:29:28.240623+0000","last_undegraded":"2024-03-06T07:29:28.240623+0000","last_fullsized":"2024-03-06T07:29:28.240623+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:52.311496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240588+0000","last_change":"2024-03-06T07:29:27.233319+0000","last_active":"2024-03-06T07:29:28.240588+0000","last_peered":"2024-03-06T07:29:28.240588+0000","last_clean":"2024-03-06T07:29:28.240588+0000","last_became_active":"2024-03-06T07:29:27.232641+0000","last_became_peered":"2024-03-06T07:29:27.232641+0000","last_unstale":"2024-03-06T07:29:28.240588+0000","last_undegraded":"2024-03-06T07:29:28.240588+0000","last_fullsized":"2024-03-06T07:29:28.240588+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:02.131796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359784+0000","last_change":"2024-03-06T07:29:27.215755+0000","last_active":"2024-03-06T07:29:29.359784+0000","last_peered":"2024-03-06T07:29:29.359784+0000","last_clean":"2024-03-06T07:29:29.359784+0000","last_became_active":"2024-03-06T07:29:27.215473+0000","last_became_peered":"2024-03-06T07:29:27.215473+0000","last_unstale":"2024-03-06T07:29:29.359784+0000","last_undegraded":"2024-03-06T07:29:29.359784+0000","last_fullsized":"2024-03-06T07:29:29.359784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:35.893406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193974+0000","last_change":"2024-03-06T07:29:27.232657+0000","last_active":"2024-03-06T07:29:28.193974+0000","last_peered":"2024-03-06T07:29:28.193974+0000","last_clean":"2024-03-06T07:29:28.193974+0000","last_became_active":"2024-03-06T07:29:27.232358+0000","last_became_peered":"2024-03-06T07:29:27.232358+0000","last_unstale":"2024-03-06T07:29:28.193974+0000","last_undegraded":"2024-03-06T07:29:28.193974+0000","last_fullsized":"2024-03-06T07:29:28.193974+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:10.665455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191029+0000","last_change":"2024-03-06T07:29:27.203643+0000","last_active":"2024-03-06T07:29:28.191029+0000","last_peered":"2024-03-06T07:29:28.191029+0000","last_clean":"2024-03-06T07:29:28.191029+0000","last_became_active":"2024-03-06T07:29:27.203417+0000","last_became_peered":"2024-03-06T07:29:27.203417+0000","last_unstale":"2024-03-06T07:29:28.191029+0000","last_undegraded":"2024-03-06T07:29:28.191029+0000","last_fullsized":"2024-03-06T07:29:28.191029+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:20.006887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192897+0000","last_change":"2024-03-06T07:29:27.208247+0000","last_active":"2024-03-06T07:29:28.192897+0000","last_peered":"2024-03-06T07:29:28.192897+0000","last_clean":"2024-03-06T07:29:28.192897+0000","last_became_active":"2024-03-06T07:29:27.205384+0000","last_became_peered":"2024-03-06T07:29:27.205384+0000","last_unstale":"2024-03-06T07:29:28.192897+0000","last_undegraded":"2024-03-06T07:29:28.192897+0000","last_fullsized":"2024-03-06T07:29:28.192897+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:13.055064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358675+0000","last_change":"2024-03-06T07:29:27.217290+0000","last_active":"2024-03-06T07:29:29.358675+0000","last_peered":"2024-03-06T07:29:29.358675+0000","last_clean":"2024-03-06T07:29:29.358675+0000","last_became_active":"2024-03-06T07:29:27.216501+0000","last_became_peered":"2024-03-06T07:29:27.216501+0000","last_unstale":"2024-03-06T07:29:29.358675+0000","last_undegraded":"2024-03-06T07:29:29.358675+0000","last_fullsized":"2024-03-06T07:29:29.358675+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:16.663232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358217+0000","last_change":"2024-03-06T07:29:27.225865+0000","last_active":"2024-03-06T07:29:29.358217+0000","last_peered":"2024-03-06T07:29:29.358217+0000","last_clean":"2024-03-06T07:29:29.358217+0000","last_became_active":"2024-03-06T07:29:27.224862+0000","last_became_peered":"2024-03-06T07:29:27.224862+0000","last_unstale":"2024-03-06T07:29:29.358217+0000","last_undegraded":"2024-03-06T07:29:29.358217+0000","last_fullsized":"2024-03-06T07:29:29.358217+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:06.088588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358808+0000","last_change":"2024-03-06T07:29:27.225091+0000","last_active":"2024-03-06T07:29:29.358808+0000","last_peered":"2024-03-06T07:29:29.358808+0000","last_clean":"2024-03-06T07:29:29.358808+0000","last_became_active":"2024-03-06T07:29:27.224189+0000","last_became_peered":"2024-03-06T07:29:27.224189+0000","last_unstale":"2024-03-06T07:29:29.358808+0000","last_undegraded":"2024-03-06T07:29:29.358808+0000","last_fullsized":"2024-03-06T07:29:29.358808+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:30.764633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193880+0000","last_change":"2024-03-06T07:29:27.234582+0000","last_active":"2024-03-06T07:29:28.193880+0000","last_peered":"2024-03-06T07:29:28.193880+0000","last_clean":"2024-03-06T07:29:28.193880+0000","last_became_active":"2024-03-06T07:29:27.233810+0000","last_became_peered":"2024-03-06T07:29:27.233810+0000","last_unstale":"2024-03-06T07:29:28.193880+0000","last_undegraded":"2024-03-06T07:29:28.193880+0000","last_fullsized":"2024-03-06T07:29:28.193880+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:57.366619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240327+0000","last_change":"2024-03-06T07:29:27.222794+0000","last_active":"2024-03-06T07:29:28.240327+0000","last_peered":"2024-03-06T07:29:28.240327+0000","last_clean":"2024-03-06T07:29:28.240327+0000","last_became_active":"2024-03-06T07:29:27.222515+0000","last_became_peered":"2024-03-06T07:29:27.222515+0000","last_unstale":"2024-03-06T07:29:28.240327+0000","last_undegraded":"2024-03-06T07:29:28.240327+0000","last_fullsized":"2024-03-06T07:29:28.240327+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:21.095594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197900+0000","last_change":"2024-03-06T07:29:27.221671+0000","last_active":"2024-03-06T07:29:29.197900+0000","last_peered":"2024-03-06T07:29:29.197900+0000","last_clean":"2024-03-06T07:29:29.197900+0000","last_became_active":"2024-03-06T07:29:27.221331+0000","last_became_peered":"2024-03-06T07:29:27.221331+0000","last_unstale":"2024-03-06T07:29:29.197900+0000","last_undegraded":"2024-03-06T07:29:29.197900+0000","last_fullsized":"2024-03-06T07:29:29.197900+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.020983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358033+0000","last_change":"2024-03-06T07:29:27.223801+0000","last_active":"2024-03-06T07:29:29.358033+0000","last_peered":"2024-03-06T07:29:29.358033+0000","last_clean":"2024-03-06T07:29:29.358033+0000","last_became_active":"2024-03-06T07:29:27.223470+0000","last_became_peered":"2024-03-06T07:29:27.223470+0000","last_unstale":"2024-03-06T07:29:29.358033+0000","last_undegraded":"2024-03-06T07:29:29.358033+0000","last_fullsized":"2024-03-06T07:29:29.358033+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:10.599929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.239849+0000","last_change":"2024-03-06T07:29:27.226174+0000","last_active":"2024-03-06T07:29:28.239849+0000","last_peered":"2024-03-06T07:29:28.239849+0000","last_clean":"2024-03-06T07:29:28.239849+0000","last_became_active":"2024-03-06T07:29:27.225908+0000","last_became_peered":"2024-03-06T07:29:27.225908+0000","last_unstale":"2024-03-06T07:29:28.239849+0000","last_undegraded":"2024-03-06T07:29:28.239849+0000","last_fullsized":"2024-03-06T07:29:28.239849+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:30.251125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191536+0000","last_change":"2024-03-06T07:29:27.216498+0000","last_active":"2024-03-06T07:29:28.191536+0000","last_peered":"2024-03-06T07:29:28.191536+0000","last_clean":"2024-03-06T07:29:28.191536+0000","last_became_active":"2024-03-06T07:29:27.213938+0000","last_became_peered":"2024-03-06T07:29:27.213938+0000","last_unstale":"2024-03-06T07:29:28.191536+0000","last_undegraded":"2024-03-06T07:29:28.191536+0000","last_fullsized":"2024-03-06T07:29:28.191536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:44.006644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192845+0000","last_change":"2024-03-06T07:29:27.227701+0000","last_active":"2024-03-06T07:29:28.192845+0000","last_peered":"2024-03-06T07:29:28.192845+0000","last_clean":"2024-03-06T07:29:28.192845+0000","last_became_active":"2024-03-06T07:29:27.225595+0000","last_became_peered":"2024-03-06T07:29:27.225595+0000","last_unstale":"2024-03-06T07:29:28.192845+0000","last_undegraded":"2024-03-06T07:29:28.192845+0000","last_fullsized":"2024-03-06T07:29:28.192845+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:27.122537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.356669+0000","last_change":"2024-03-06T07:29:27.222401+0000","last_active":"2024-03-06T07:29:29.356669+0000","last_peered":"2024-03-06T07:29:29.356669+0000","last_clean":"2024-03-06T07:29:29.356669+0000","last_became_active":"2024-03-06T07:29:27.221978+0000","last_became_peered":"2024-03-06T07:29:27.221978+0000","last_unstale":"2024-03-06T07:29:29.356669+0000","last_undegraded":"2024-03-06T07:29:29.356669+0000","last_fullsized":"2024-03-06T07:29:29.356669+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:32.247068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191977+0000","last_change":"2024-03-06T07:29:26.184227+0000","last_active":"2024-03-06T07:29:28.191977+0000","last_peered":"2024-03-06T07:29:28.191977+0000","last_clean":"2024-03-06T07:29:28.191977+0000","last_became_active":"2024-03-06T07:29:26.183923+0000","last_became_peered":"2024-03-06T07:29:26.183923+0000","last_unstale":"2024-03-06T07:29:28.191977+0000","last_undegraded":"2024-03-06T07:29:28.191977+0000","last_fullsized":"2024-03-06T07:29:28.191977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:41.006960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359470+0000","last_change":"2024-03-06T07:29:27.219177+0000","last_active":"2024-03-06T07:29:29.359470+0000","last_peered":"2024-03-06T07:29:29.359470+0000","last_clean":"2024-03-06T07:29:29.359470+0000","last_became_active":"2024-03-06T07:29:27.218700+0000","last_became_peered":"2024-03-06T07:29:27.218700+0000","last_unstale":"2024-03-06T07:29:29.359470+0000","last_undegraded":"2024-03-06T07:29:29.359470+0000","last_fullsized":"2024-03-06T07:29:29.359470+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:41.102346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240751+0000","last_change":"2024-03-06T07:29:26.200719+0000","last_active":"2024-03-06T07:29:28.240751+0000","last_peered":"2024-03-06T07:29:28.240751+0000","last_clean":"2024-03-06T07:29:28.240751+0000","last_became_active":"2024-03-06T07:29:26.200438+0000","last_became_peered":"2024-03-06T07:29:26.200438+0000","last_unstale":"2024-03-06T07:29:28.240751+0000","last_undegraded":"2024-03-06T07:29:28.240751+0000","last_fullsized":"2024-03-06T07:29:28.240751+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197916+0000","last_change":"2024-03-06T07:29:27.222145+0000","last_active":"2024-03-06T07:29:29.197916+0000","last_peered":"2024-03-06T07:29:29.197916+0000","last_clean":"2024-03-06T07:29:29.197916+0000","last_became_active":"2024-03-06T07:29:27.221795+0000","last_became_peered":"2024-03-06T07:29:27.221795+0000","last_unstale":"2024-03-06T07:29:29.197916+0000","last_undegraded":"2024-03-06T07:29:29.197916+0000","last_fullsized":"2024-03-06T07:29:29.197916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:25.630352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.356799+0000","last_change":"2024-03-06T07:29:26.201813+0000","last_active":"2024-03-06T07:29:29.356799+0000","last_peered":"2024-03-06T07:29:29.356799+0000","last_clean":"2024-03-06T07:29:29.356799+0000","last_became_active":"2024-03-06T07:29:26.201360+0000","last_became_peered":"2024-03-06T07:29:26.201360+0000","last_unstale":"2024-03-06T07:29:29.356799+0000","last_undegraded":"2024-03-06T07:29:29.356799+0000","last_fullsized":"2024-03-06T07:29:29.356799+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.356503+0000","last_change":"2024-03-06T07:29:27.231046+0000","last_active":"2024-03-06T07:29:29.356503+0000","last_peered":"2024-03-06T07:29:29.356503+0000","last_clean":"2024-03-06T07:29:29.356503+0000","last_became_active":"2024-03-06T07:29:27.230774+0000","last_became_peered":"2024-03-06T07:29:27.230774+0000","last_unstale":"2024-03-06T07:29:29.356503+0000","last_undegraded":"2024-03-06T07:29:29.356503+0000","last_fullsized":"2024-03-06T07:29:29.356503+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:54.206839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359102+0000","last_change":"2024-03-06T07:29:26.202950+0000","last_active":"2024-03-06T07:29:29.359102+0000","last_peered":"2024-03-06T07:29:29.359102+0000","last_clean":"2024-03-06T07:29:29.359102+0000","last_became_active":"2024-03-06T07:29:26.202564+0000","last_became_peered":"2024-03-06T07:29:26.202564+0000","last_unstale":"2024-03-06T07:29:29.359102+0000","last_undegraded":"2024-03-06T07:29:29.359102+0000","last_fullsized":"2024-03-06T07:29:29.359102+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:31.233046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.239929+0000","last_change":"2024-03-06T07:29:27.236948+0000","last_active":"2024-03-06T07:29:28.239929+0000","last_peered":"2024-03-06T07:29:28.239929+0000","last_clean":"2024-03-06T07:29:28.239929+0000","last_became_active":"2024-03-06T07:29:27.235931+0000","last_became_peered":"2024-03-06T07:29:27.235931+0000","last_unstale":"2024-03-06T07:29:28.239929+0000","last_undegraded":"2024-03-06T07:29:28.239929+0000","last_fullsized":"2024-03-06T07:29:28.239929+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:32.673285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191146+0000","last_change":"2024-03-06T07:29:26.188354+0000","last_active":"2024-03-06T07:29:28.191146+0000","last_peered":"2024-03-06T07:29:28.191146+0000","last_clean":"2024-03-06T07:29:28.191146+0000","last_became_active":"2024-03-06T07:29:26.187378+0000","last_became_peered":"2024-03-06T07:29:26.187378+0000","last_unstale":"2024-03-06T07:29:28.191146+0000","last_undegraded":"2024-03-06T07:29:28.191146+0000","last_fullsized":"2024-03-06T07:29:28.191146+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:12.041042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193274+0000","last_change":"2024-03-06T07:29:27.224616+0000","last_active":"2024-03-06T07:29:28.193274+0000","last_peered":"2024-03-06T07:29:28.193274+0000","last_clean":"2024-03-06T07:29:28.193274+0000","last_became_active":"2024-03-06T07:29:27.223661+0000","last_became_peered":"2024-03-06T07:29:27.223661+0000","last_unstale":"2024-03-06T07:29:28.193274+0000","last_undegraded":"2024-03-06T07:29:28.193274+0000","last_fullsized":"2024-03-06T07:29:28.193274+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:52.966337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240090+0000","last_change":"2024-03-06T07:29:26.204279+0000","last_active":"2024-03-06T07:29:28.240090+0000","last_peered":"2024-03-06T07:29:28.240090+0000","last_clean":"2024-03-06T07:29:28.240090+0000","last_became_active":"2024-03-06T07:29:26.203822+0000","last_became_peered":"2024-03-06T07:29:26.203822+0000","last_unstale":"2024-03-06T07:29:28.240090+0000","last_undegraded":"2024-03-06T07:29:28.240090+0000","last_fullsized":"2024-03-06T07:29:28.240090+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192371+0000","last_change":"2024-03-06T07:29:27.215308+0000","last_active":"2024-03-06T07:29:28.192371+0000","last_peered":"2024-03-06T07:29:28.192371+0000","last_clean":"2024-03-06T07:29:28.192371+0000","last_became_active":"2024-03-06T07:29:27.212232+0000","last_became_peered":"2024-03-06T07:29:27.212232+0000","last_unstale":"2024-03-06T07:29:28.192371+0000","last_undegraded":"2024-03-06T07:29:28.192371+0000","last_fullsized":"2024-03-06T07:29:28.192371+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:42.663881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.196950+0000","last_change":"2024-03-06T07:29:26.194000+0000","last_active":"2024-03-06T07:29:29.196950+0000","last_peered":"2024-03-06T07:29:29.196950+0000","last_clean":"2024-03-06T07:29:29.196950+0000","last_became_active":"2024-03-06T07:29:26.193684+0000","last_became_peered":"2024-03-06T07:29:26.193684+0000","last_unstale":"2024-03-06T07:29:29.196950+0000","last_undegraded":"2024-03-06T07:29:29.196950+0000","last_fullsized":"2024-03-06T07:29:29.196950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:34.879384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358071+0000","last_change":"2024-03-06T07:29:27.217504+0000","last_active":"2024-03-06T07:29:29.358071+0000","last_peered":"2024-03-06T07:29:29.358071+0000","last_clean":"2024-03-06T07:29:29.358071+0000","last_became_active":"2024-03-06T07:29:27.216934+0000","last_became_peered":"2024-03-06T07:29:27.216934+0000","last_unstale":"2024-03-06T07:29:29.358071+0000","last_undegraded":"2024-03-06T07:29:29.358071+0000","last_fullsized":"2024-03-06T07:29:29.358071+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:59.041898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192441+0000","last_change":"2024-03-06T07:29:26.182870+0000","last_active":"2024-03-06T07:29:28.192441+0000","last_peered":"2024-03-06T07:29:28.192441+0000","last_clean":"2024-03-06T07:29:28.192441+0000","last_became_active":"2024-03-06T07:29:26.182537+0000","last_became_peered":"2024-03-06T07:29:26.182537+0000","last_unstale":"2024-03-06T07:29:28.192441+0000","last_undegraded":"2024-03-06T07:29:28.192441+0000","last_fullsized":"2024-03-06T07:29:28.192441+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:19.260089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240582+0000","last_change":"2024-03-06T07:29:27.233586+0000","last_active":"2024-03-06T07:29:28.240582+0000","last_peered":"2024-03-06T07:29:28.240582+0000","last_clean":"2024-03-06T07:29:28.240582+0000","last_became_active":"2024-03-06T07:29:27.233008+0000","last_became_peered":"2024-03-06T07:29:27.233008+0000","last_unstale":"2024-03-06T07:29:28.240582+0000","last_undegraded":"2024-03-06T07:29:28.240582+0000","last_fullsized":"2024-03-06T07:29:28.240582+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:55.566106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240640+0000","last_change":"2024-03-06T07:29:26.193099+0000","last_active":"2024-03-06T07:29:28.240640+0000","last_peered":"2024-03-06T07:29:28.240640+0000","last_clean":"2024-03-06T07:29:28.240640+0000","last_became_active":"2024-03-06T07:29:26.192805+0000","last_became_peered":"2024-03-06T07:29:26.192805+0000","last_unstale":"2024-03-06T07:29:28.240640+0000","last_undegraded":"2024-03-06T07:29:28.240640+0000","last_fullsized":"2024-03-06T07:29:28.240640+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359941+0000","last_change":"2024-03-06T07:29:27.233092+0000","last_active":"2024-03-06T07:29:29.359941+0000","last_peered":"2024-03-06T07:29:29.359941+0000","last_clean":"2024-03-06T07:29:29.359941+0000","last_became_active":"2024-03-06T07:29:27.232781+0000","last_became_peered":"2024-03-06T07:29:27.232781+0000","last_unstale":"2024-03-06T07:29:29.359941+0000","last_undegraded":"2024-03-06T07:29:29.359941+0000","last_fullsized":"2024-03-06T07:29:29.359941+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:40.080803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194292+0000","last_change":"2024-03-06T07:29:26.186625+0000","last_active":"2024-03-06T07:29:28.194292+0000","last_peered":"2024-03-06T07:29:28.194292+0000","last_clean":"2024-03-06T07:29:28.194292+0000","last_became_active":"2024-03-06T07:29:26.186363+0000","last_became_peered":"2024-03-06T07:29:26.186363+0000","last_unstale":"2024-03-06T07:29:28.194292+0000","last_undegraded":"2024-03-06T07:29:28.194292+0000","last_fullsized":"2024-03-06T07:29:28.194292+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:03.733713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191652+0000","last_change":"2024-03-06T07:29:27.224493+0000","last_active":"2024-03-06T07:29:28.191652+0000","last_peered":"2024-03-06T07:29:28.191652+0000","last_clean":"2024-03-06T07:29:28.191652+0000","last_became_active":"2024-03-06T07:29:27.224147+0000","last_became_peered":"2024-03-06T07:29:27.224147+0000","last_unstale":"2024-03-06T07:29:28.191652+0000","last_undegraded":"2024-03-06T07:29:28.191652+0000","last_fullsized":"2024-03-06T07:29:28.191652+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:05.148277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":17,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360444+0000","last_change":"2024-03-06T07:29:26.197651+0000","last_active":"2024-03-06T07:29:29.360444+0000","last_peered":"2024-03-06T07:29:29.360444+0000","last_clean":"2024-03-06T07:29:29.360444+0000","last_became_active":"2024-03-06T07:29:26.197161+0000","last_became_peered":"2024-03-06T07:29:26.197161+0000","last_unstale":"2024-03-06T07:29:29.360444+0000","last_undegraded":"2024-03-06T07:29:29.360444+0000","last_fullsized":"2024-03-06T07:29:29.360444+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:56.450035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360085+0000","last_change":"2024-03-06T07:29:27.230126+0000","last_active":"2024-03-06T07:29:29.360085+0000","last_peered":"2024-03-06T07:29:29.360085+0000","last_clean":"2024-03-06T07:29:29.360085+0000","last_became_active":"2024-03-06T07:29:27.229726+0000","last_became_peered":"2024-03-06T07:29:27.229726+0000","last_unstale":"2024-03-06T07:29:29.360085+0000","last_undegraded":"2024-03-06T07:29:29.360085+0000","last_fullsized":"2024-03-06T07:29:29.360085+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:59.335223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193894+0000","last_change":"2024-03-06T07:29:26.200648+0000","last_active":"2024-03-06T07:29:28.193894+0000","last_peered":"2024-03-06T07:29:28.193894+0000","last_clean":"2024-03-06T07:29:28.193894+0000","last_became_active":"2024-03-06T07:29:26.200365+0000","last_became_peered":"2024-03-06T07:29:26.200365+0000","last_unstale":"2024-03-06T07:29:28.193894+0000","last_undegraded":"2024-03-06T07:29:28.193894+0000","last_fullsized":"2024-03-06T07:29:28.193894+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.411331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358294+0000","last_change":"2024-03-06T07:29:27.217938+0000","last_active":"2024-03-06T07:29:29.358294+0000","last_peered":"2024-03-06T07:29:29.358294+0000","last_clean":"2024-03-06T07:29:29.358294+0000","last_became_active":"2024-03-06T07:29:27.217309+0000","last_became_peered":"2024-03-06T07:29:27.217309+0000","last_unstale":"2024-03-06T07:29:29.358294+0000","last_undegraded":"2024-03-06T07:29:29.358294+0000","last_fullsized":"2024-03-06T07:29:29.358294+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:48.496514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191206+0000","last_change":"2024-03-06T07:29:26.180918+0000","last_active":"2024-03-06T07:29:28.191206+0000","last_peered":"2024-03-06T07:29:28.191206+0000","last_clean":"2024-03-06T07:29:28.191206+0000","last_became_active":"2024-03-06T07:29:26.180570+0000","last_became_peered":"2024-03-06T07:29:26.180570+0000","last_unstale":"2024-03-06T07:29:28.191206+0000","last_undegraded":"2024-03-06T07:29:28.191206+0000","last_fullsized":"2024-03-06T07:29:28.191206+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:19.169731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193427+0000","last_change":"2024-03-06T07:29:27.220767+0000","last_active":"2024-03-06T07:29:28.193427+0000","last_peered":"2024-03-06T07:29:28.193427+0000","last_clean":"2024-03-06T07:29:28.193427+0000","last_became_active":"2024-03-06T07:29:27.220493+0000","last_became_peered":"2024-03-06T07:29:27.220493+0000","last_unstale":"2024-03-06T07:29:28.193427+0000","last_undegraded":"2024-03-06T07:29:28.193427+0000","last_fullsized":"2024-03-06T07:29:28.193427+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:20.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359155+0000","last_change":"2024-03-06T07:29:26.194437+0000","last_active":"2024-03-06T07:29:29.359155+0000","last_peered":"2024-03-06T07:29:29.359155+0000","last_clean":"2024-03-06T07:29:29.359155+0000","last_became_active":"2024-03-06T07:29:26.194154+0000","last_became_peered":"2024-03-06T07:29:26.194154+0000","last_unstale":"2024-03-06T07:29:29.359155+0000","last_undegraded":"2024-03-06T07:29:29.359155+0000","last_fullsized":"2024-03-06T07:29:29.359155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:08.796194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.190967+0000","last_change":"2024-03-06T07:29:27.222589+0000","last_active":"2024-03-06T07:29:28.190967+0000","last_peered":"2024-03-06T07:29:28.190967+0000","last_clean":"2024-03-06T07:29:28.190967+0000","last_became_active":"2024-03-06T07:29:27.222280+0000","last_became_peered":"2024-03-06T07:29:27.222280+0000","last_unstale":"2024-03-06T07:29:28.190967+0000","last_undegraded":"2024-03-06T07:29:28.190967+0000","last_fullsized":"2024-03-06T07:29:28.190967+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:17.154562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191810+0000","last_change":"2024-03-06T07:29:26.185585+0000","last_active":"2024-03-06T07:29:28.191810+0000","last_peered":"2024-03-06T07:29:28.191810+0000","last_clean":"2024-03-06T07:29:28.191810+0000","last_became_active":"2024-03-06T07:29:26.180225+0000","last_became_peered":"2024-03-06T07:29:26.180225+0000","last_unstale":"2024-03-06T07:29:28.191810+0000","last_undegraded":"2024-03-06T07:29:28.191810+0000","last_fullsized":"2024-03-06T07:29:28.191810+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:24.616330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359365+0000","last_change":"2024-03-06T07:29:27.232136+0000","last_active":"2024-03-06T07:29:29.359365+0000","last_peered":"2024-03-06T07:29:29.359365+0000","last_clean":"2024-03-06T07:29:29.359365+0000","last_became_active":"2024-03-06T07:29:27.231588+0000","last_became_peered":"2024-03-06T07:29:27.231588+0000","last_unstale":"2024-03-06T07:29:29.359365+0000","last_undegraded":"2024-03-06T07:29:29.359365+0000","last_fullsized":"2024-03-06T07:29:29.359365+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:02.131796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192468+0000","last_change":"2024-03-06T07:29:26.194945+0000","last_active":"2024-03-06T07:29:28.192468+0000","last_peered":"2024-03-06T07:29:28.192468+0000","last_clean":"2024-03-06T07:29:28.192468+0000","last_became_active":"2024-03-06T07:29:26.194577+0000","last_became_peered":"2024-03-06T07:29:26.194577+0000","last_unstale":"2024-03-06T07:29:28.192468+0000","last_undegraded":"2024-03-06T07:29:28.192468+0000","last_fullsized":"2024-03-06T07:29:28.192468+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:23.475086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194058+0000","last_change":"2024-03-06T07:29:27.217848+0000","last_active":"2024-03-06T07:29:28.194058+0000","last_peered":"2024-03-06T07:29:28.194058+0000","last_clean":"2024-03-06T07:29:28.194058+0000","last_became_active":"2024-03-06T07:29:27.217149+0000","last_became_peered":"2024-03-06T07:29:27.217149+0000","last_unstale":"2024-03-06T07:29:28.194058+0000","last_undegraded":"2024-03-06T07:29:28.194058+0000","last_fullsized":"2024-03-06T07:29:28.194058+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:23.354165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194197+0000","last_change":"2024-03-06T07:29:26.188532+0000","last_active":"2024-03-06T07:29:28.194197+0000","last_peered":"2024-03-06T07:29:28.194197+0000","last_clean":"2024-03-06T07:29:28.194197+0000","last_became_active":"2024-03-06T07:29:26.188243+0000","last_became_peered":"2024-03-06T07:29:26.188243+0000","last_unstale":"2024-03-06T07:29:28.194197+0000","last_undegraded":"2024-03-06T07:29:28.194197+0000","last_fullsized":"2024-03-06T07:29:28.194197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:05.043428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194587+0000","last_change":"2024-03-06T07:29:27.211086+0000","last_active":"2024-03-06T07:29:28.194587+0000","last_peered":"2024-03-06T07:29:28.194587+0000","last_clean":"2024-03-06T07:29:28.194587+0000","last_became_active":"2024-03-06T07:29:27.210737+0000","last_became_peered":"2024-03-06T07:29:27.210737+0000","last_unstale":"2024-03-06T07:29:28.194587+0000","last_undegraded":"2024-03-06T07:29:28.194587+0000","last_fullsized":"2024-03-06T07:29:28.194587+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:04.309774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240873+0000","last_change":"2024-03-06T07:29:26.199005+0000","last_active":"2024-03-06T07:29:28.240873+0000","last_peered":"2024-03-06T07:29:28.240873+0000","last_clean":"2024-03-06T07:29:28.240873+0000","last_became_active":"2024-03-06T07:29:26.196517+0000","last_became_peered":"2024-03-06T07:29:26.196517+0000","last_unstale":"2024-03-06T07:29:28.240873+0000","last_undegraded":"2024-03-06T07:29:28.240873+0000","last_fullsized":"2024-03-06T07:29:28.240873+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:20.660770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358862+0000","last_change":"2024-03-06T07:29:27.208624+0000","last_active":"2024-03-06T07:29:29.358862+0000","last_peered":"2024-03-06T07:29:29.358862+0000","last_clean":"2024-03-06T07:29:29.358862+0000","last_became_active":"2024-03-06T07:29:27.206474+0000","last_became_peered":"2024-03-06T07:29:27.206474+0000","last_unstale":"2024-03-06T07:29:29.358862+0000","last_undegraded":"2024-03-06T07:29:29.358862+0000","last_fullsized":"2024-03-06T07:29:29.358862+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:59.866176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.198040+0000","last_change":"2024-03-06T07:29:26.199717+0000","last_active":"2024-03-06T07:29:29.198040+0000","last_peered":"2024-03-06T07:29:29.198040+0000","last_clean":"2024-03-06T07:29:29.198040+0000","last_became_active":"2024-03-06T07:29:26.199436+0000","last_became_peered":"2024-03-06T07:29:26.199436+0000","last_unstale":"2024-03-06T07:29:29.198040+0000","last_undegraded":"2024-03-06T07:29:29.198040+0000","last_fullsized":"2024-03-06T07:29:29.198040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.422880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359170+0000","last_change":"2024-03-06T07:29:27.222774+0000","last_active":"2024-03-06T07:29:29.359170+0000","last_peered":"2024-03-06T07:29:29.359170+0000","last_clean":"2024-03-06T07:29:29.359170+0000","last_became_active":"2024-03-06T07:29:27.222387+0000","last_became_peered":"2024-03-06T07:29:27.222387+0000","last_unstale":"2024-03-06T07:29:29.359170+0000","last_undegraded":"2024-03-06T07:29:29.359170+0000","last_fullsized":"2024-03-06T07:29:29.359170+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:11.269728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191748+0000","last_change":"2024-03-06T07:29:26.182768+0000","last_active":"2024-03-06T07:29:28.191748+0000","last_peered":"2024-03-06T07:29:28.191748+0000","last_clean":"2024-03-06T07:29:28.191748+0000","last_became_active":"2024-03-06T07:29:26.182489+0000","last_became_peered":"2024-03-06T07:29:26.182489+0000","last_unstale":"2024-03-06T07:29:28.191748+0000","last_undegraded":"2024-03-06T07:29:28.191748+0000","last_fullsized":"2024-03-06T07:29:28.191748+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:51.952315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191475+0000","last_change":"2024-03-06T07:29:27.221898+0000","last_active":"2024-03-06T07:29:28.191475+0000","last_peered":"2024-03-06T07:29:28.191475+0000","last_clean":"2024-03-06T07:29:28.191475+0000","last_became_active":"2024-03-06T07:29:27.221636+0000","last_became_peered":"2024-03-06T07:29:27.221636+0000","last_unstale":"2024-03-06T07:29:28.191475+0000","last_undegraded":"2024-03-06T07:29:28.191475+0000","last_fullsized":"2024-03-06T07:29:28.191475+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:29.749556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191323+0000","last_change":"2024-03-06T07:29:26.188395+0000","last_active":"2024-03-06T07:29:28.191323+0000","last_peered":"2024-03-06T07:29:28.191323+0000","last_clean":"2024-03-06T07:29:28.191323+0000","last_became_active":"2024-03-06T07:29:26.187669+0000","last_became_peered":"2024-03-06T07:29:26.187669+0000","last_unstale":"2024-03-06T07:29:28.191323+0000","last_undegraded":"2024-03-06T07:29:28.191323+0000","last_fullsized":"2024-03-06T07:29:28.191323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:19.026209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.355952+0000","last_change":"2024-03-06T07:29:27.221492+0000","last_active":"2024-03-06T07:29:29.355952+0000","last_peered":"2024-03-06T07:29:29.355952+0000","last_clean":"2024-03-06T07:29:29.355952+0000","last_became_active":"2024-03-06T07:29:27.221203+0000","last_became_peered":"2024-03-06T07:29:27.221203+0000","last_unstale":"2024-03-06T07:29:29.355952+0000","last_undegraded":"2024-03-06T07:29:29.355952+0000","last_fullsized":"2024-03-06T07:29:29.355952+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:09.810216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197081+0000","last_change":"2024-03-06T07:29:26.200840+0000","last_active":"2024-03-06T07:29:29.197081+0000","last_peered":"2024-03-06T07:29:29.197081+0000","last_clean":"2024-03-06T07:29:29.197081+0000","last_became_active":"2024-03-06T07:29:26.200577+0000","last_became_peered":"2024-03-06T07:29:26.200577+0000","last_unstale":"2024-03-06T07:29:29.197081+0000","last_undegraded":"2024-03-06T07:29:29.197081+0000","last_fullsized":"2024-03-06T07:29:29.197081+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:08.181893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192995+0000","last_change":"2024-03-06T07:29:27.226841+0000","last_active":"2024-03-06T07:29:28.192995+0000","last_peered":"2024-03-06T07:29:28.192995+0000","last_clean":"2024-03-06T07:29:28.192995+0000","last_became_active":"2024-03-06T07:29:27.226120+0000","last_became_peered":"2024-03-06T07:29:27.226120+0000","last_unstale":"2024-03-06T07:29:28.192995+0000","last_undegraded":"2024-03-06T07:29:28.192995+0000","last_fullsized":"2024-03-06T07:29:28.192995+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:39.799373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193099+0000","last_change":"2024-03-06T07:29:26.191147+0000","last_active":"2024-03-06T07:29:28.193099+0000","last_peered":"2024-03-06T07:29:28.193099+0000","last_clean":"2024-03-06T07:29:28.193099+0000","last_became_active":"2024-03-06T07:29:26.190641+0000","last_became_peered":"2024-03-06T07:29:26.190641+0000","last_unstale":"2024-03-06T07:29:28.193099+0000","last_undegraded":"2024-03-06T07:29:28.193099+0000","last_fullsized":"2024-03-06T07:29:28.193099+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:09.406238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192348+0000","last_change":"2024-03-06T07:29:27.212821+0000","last_active":"2024-03-06T07:29:28.192348+0000","last_peered":"2024-03-06T07:29:28.192348+0000","last_clean":"2024-03-06T07:29:28.192348+0000","last_became_active":"2024-03-06T07:29:27.212249+0000","last_became_peered":"2024-03-06T07:29:27.212249+0000","last_unstale":"2024-03-06T07:29:28.192348+0000","last_undegraded":"2024-03-06T07:29:28.192348+0000","last_fullsized":"2024-03-06T07:29:28.192348+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:25.820523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.357267+0000","last_change":"2024-03-06T07:29:26.193713+0000","last_active":"2024-03-06T07:29:29.357267+0000","last_peered":"2024-03-06T07:29:29.357267+0000","last_clean":"2024-03-06T07:29:29.357267+0000","last_became_active":"2024-03-06T07:29:26.193389+0000","last_became_peered":"2024-03-06T07:29:26.193389+0000","last_unstale":"2024-03-06T07:29:29.357267+0000","last_undegraded":"2024-03-06T07:29:29.357267+0000","last_fullsized":"2024-03-06T07:29:29.357267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:28.559373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360892+0000","last_change":"2024-03-06T07:29:27.218961+0000","last_active":"2024-03-06T07:29:29.360892+0000","last_peered":"2024-03-06T07:29:29.360892+0000","last_clean":"2024-03-06T07:29:29.360892+0000","last_became_active":"2024-03-06T07:29:27.218344+0000","last_became_peered":"2024-03-06T07:29:27.218344+0000","last_unstale":"2024-03-06T07:29:29.360892+0000","last_undegraded":"2024-03-06T07:29:29.360892+0000","last_fullsized":"2024-03-06T07:29:29.360892+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:07.883849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"46'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191843+0000","last_change":"2024-03-06T07:29:26.196747+0000","last_active":"2024-03-06T07:29:28.191843+0000","last_peered":"2024-03-06T07:29:28.191843+0000","last_clean":"2024-03-06T07:29:28.191843+0000","last_became_active":"2024-03-06T07:29:26.196446+0000","last_became_peered":"2024-03-06T07:29:26.196446+0000","last_unstale":"2024-03-06T07:29:28.191843+0000","last_undegraded":"2024-03-06T07:29:28.191843+0000","last_fullsized":"2024-03-06T07:29:28.191843+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:40.516483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193814+0000","last_change":"2024-03-06T07:29:27.218206+0000","last_active":"2024-03-06T07:29:28.193814+0000","last_peered":"2024-03-06T07:29:28.193814+0000","last_clean":"2024-03-06T07:29:28.193814+0000","last_became_active":"2024-03-06T07:29:27.214524+0000","last_became_peered":"2024-03-06T07:29:27.214524+0000","last_unstale":"2024-03-06T07:29:28.193814+0000","last_undegraded":"2024-03-06T07:29:28.193814+0000","last_fullsized":"2024-03-06T07:29:28.193814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:25.630352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360705+0000","last_change":"2024-03-06T07:29:26.199027+0000","last_active":"2024-03-06T07:29:29.360705+0000","last_peered":"2024-03-06T07:29:29.360705+0000","last_clean":"2024-03-06T07:29:29.360705+0000","last_became_active":"2024-03-06T07:29:26.198748+0000","last_became_peered":"2024-03-06T07:29:26.198748+0000","last_unstale":"2024-03-06T07:29:29.360705+0000","last_undegraded":"2024-03-06T07:29:29.360705+0000","last_fullsized":"2024-03-06T07:29:29.360705+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:33.099509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360514+0000","last_change":"2024-03-06T07:29:27.235780+0000","last_active":"2024-03-06T07:29:29.360514+0000","last_peered":"2024-03-06T07:29:29.360514+0000","last_clean":"2024-03-06T07:29:29.360514+0000","last_became_active":"2024-03-06T07:29:27.235481+0000","last_became_peered":"2024-03-06T07:29:27.235481+0000","last_unstale":"2024-03-06T07:29:29.360514+0000","last_undegraded":"2024-03-06T07:29:29.360514+0000","last_fullsized":"2024-03-06T07:29:29.360514+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.355929+0000","last_change":"2024-03-06T07:29:26.196655+0000","last_active":"2024-03-06T07:29:29.355929+0000","last_peered":"2024-03-06T07:29:29.355929+0000","last_clean":"2024-03-06T07:29:29.355929+0000","last_became_active":"2024-03-06T07:29:26.196370+0000","last_became_peered":"2024-03-06T07:29:26.196370+0000","last_unstale":"2024-03-06T07:29:29.355929+0000","last_undegraded":"2024-03-06T07:29:29.355929+0000","last_fullsized":"2024-03-06T07:29:29.355929+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191187+0000","last_change":"2024-03-06T07:29:27.216567+0000","last_active":"2024-03-06T07:29:28.191187+0000","last_peered":"2024-03-06T07:29:28.191187+0000","last_clean":"2024-03-06T07:29:28.191187+0000","last_became_active":"2024-03-06T07:29:27.216186+0000","last_became_peered":"2024-03-06T07:29:27.216186+0000","last_unstale":"2024-03-06T07:29:28.191187+0000","last_undegraded":"2024-03-06T07:29:28.191187+0000","last_fullsized":"2024-03-06T07:29:28.191187+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:22.862112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359852+0000","last_change":"2024-03-06T07:29:26.195806+0000","last_active":"2024-03-06T07:29:29.359852+0000","last_peered":"2024-03-06T07:29:29.359852+0000","last_clean":"2024-03-06T07:29:29.359852+0000","last_became_active":"2024-03-06T07:29:26.195455+0000","last_became_peered":"2024-03-06T07:29:26.195455+0000","last_unstale":"2024-03-06T07:29:29.359852+0000","last_undegraded":"2024-03-06T07:29:29.359852+0000","last_fullsized":"2024-03-06T07:29:29.359852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:53.192817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191031+0000","last_change":"2024-03-06T07:29:27.222600+0000","last_active":"2024-03-06T07:29:28.191031+0000","last_peered":"2024-03-06T07:29:28.191031+0000","last_clean":"2024-03-06T07:29:28.191031+0000","last_became_active":"2024-03-06T07:29:27.222304+0000","last_became_peered":"2024-03-06T07:29:27.222304+0000","last_unstale":"2024-03-06T07:29:28.191031+0000","last_undegraded":"2024-03-06T07:29:28.191031+0000","last_fullsized":"2024-03-06T07:29:28.191031+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:07.910504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240126+0000","last_change":"2024-03-06T07:29:26.199356+0000","last_active":"2024-03-06T07:29:28.240126+0000","last_peered":"2024-03-06T07:29:28.240126+0000","last_clean":"2024-03-06T07:29:28.240126+0000","last_became_active":"2024-03-06T07:29:26.199015+0000","last_became_peered":"2024-03-06T07:29:26.199015+0000","last_unstale":"2024-03-06T07:29:28.240126+0000","last_undegraded":"2024-03-06T07:29:28.240126+0000","last_fullsized":"2024-03-06T07:29:28.240126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:42.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197144+0000","last_change":"2024-03-06T07:29:27.215173+0000","last_active":"2024-03-06T07:29:29.197144+0000","last_peered":"2024-03-06T07:29:29.197144+0000","last_clean":"2024-03-06T07:29:29.197144+0000","last_became_active":"2024-03-06T07:29:27.214896+0000","last_became_peered":"2024-03-06T07:29:27.214896+0000","last_unstale":"2024-03-06T07:29:29.197144+0000","last_undegraded":"2024-03-06T07:29:29.197144+0000","last_fullsized":"2024-03-06T07:29:29.197144+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:41.530505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.355963+0000","last_change":"2024-03-06T07:29:26.184910+0000","last_active":"2024-03-06T07:29:29.355963+0000","last_peered":"2024-03-06T07:29:29.355963+0000","last_clean":"2024-03-06T07:29:29.355963+0000","last_became_active":"2024-03-06T07:29:26.183588+0000","last_became_peered":"2024-03-06T07:29:26.183588+0000","last_unstale":"2024-03-06T07:29:29.355963+0000","last_undegraded":"2024-03-06T07:29:29.355963+0000","last_fullsized":"2024-03-06T07:29:29.355963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.239992+0000","last_change":"2024-03-06T07:29:27.234069+0000","last_active":"2024-03-06T07:29:28.239992+0000","last_peered":"2024-03-06T07:29:28.239992+0000","last_clean":"2024-03-06T07:29:28.239992+0000","last_became_active":"2024-03-06T07:29:27.233784+0000","last_became_peered":"2024-03-06T07:29:27.233784+0000","last_unstale":"2024-03-06T07:29:28.239992+0000","last_undegraded":"2024-03-06T07:29:28.239992+0000","last_fullsized":"2024-03-06T07:29:28.239992+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:54.594286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192999+0000","last_change":"2024-03-06T07:29:26.185660+0000","last_active":"2024-03-06T07:29:28.192999+0000","last_peered":"2024-03-06T07:29:28.192999+0000","last_clean":"2024-03-06T07:29:28.192999+0000","last_became_active":"2024-03-06T07:29:26.185316+0000","last_became_peered":"2024-03-06T07:29:26.185316+0000","last_unstale":"2024-03-06T07:29:28.192999+0000","last_undegraded":"2024-03-06T07:29:28.192999+0000","last_fullsized":"2024-03-06T07:29:28.192999+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:17.328622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194290+0000","last_change":"2024-03-06T07:29:27.220031+0000","last_active":"2024-03-06T07:29:28.194290+0000","last_peered":"2024-03-06T07:29:28.194290+0000","last_clean":"2024-03-06T07:29:28.194290+0000","last_became_active":"2024-03-06T07:29:27.218910+0000","last_became_peered":"2024-03-06T07:29:27.218910+0000","last_unstale":"2024-03-06T07:29:28.194290+0000","last_undegraded":"2024-03-06T07:29:28.194290+0000","last_fullsized":"2024-03-06T07:29:28.194290+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.020983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.198094+0000","last_change":"2024-03-06T07:29:26.200006+0000","last_active":"2024-03-06T07:29:29.198094+0000","last_peered":"2024-03-06T07:29:29.198094+0000","last_clean":"2024-03-06T07:29:29.198094+0000","last_became_active":"2024-03-06T07:29:26.198345+0000","last_became_peered":"2024-03-06T07:29:26.198345+0000","last_unstale":"2024-03-06T07:29:29.198094+0000","last_undegraded":"2024-03-06T07:29:29.198094+0000","last_fullsized":"2024-03-06T07:29:29.198094+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.869827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":213,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.357332+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:29:29.357332+0000","last_peered":"2024-03-06T07:29:29.357332+0000","last_clean":"2024-03-06T07:29:29.357332+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:29:29.357332+0000","last_undegraded":"2024-03-06T07:29:29.357332+0000","last_fullsized":"2024-03-06T07:29:29.357332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509469,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":548864,"data_stored":218634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509470,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":217070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542173,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36448,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335392,"statfs":{"total":96636764160,"available":96599441408,"internally_reserved":0,"allocated":1130496,"data_stored":807458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":217672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542174,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":217070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542173,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":218612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542174,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":217638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":807402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:36.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.037+0000 7f1115800640 1 Processor -- start 2024-03-06T07:29:37.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.037+0000 7f1115800640 1 -- start start 2024-03-06T07:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.039+0000 7f1115800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f11100ae900 0x7f111015cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.039+0000 7f1115800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11100aed30 con 0x7f111015bfb0 2024-03-06T07:29:37.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.039+0000 7f1115800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11100af000 con 0x7f111006afb0 2024-03-06T07:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.039+0000 7f1115800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11100b3de0 con 0x7f11100ae900 2024-03-06T07:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.040+0000 7f1106000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f111015bfb0 0x7f11100b4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57584/0 (socket says 172.21.15.6:57584) 2024-03-06T07:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.040+0000 7f1106000640 1 -- 172.21.15.6:0/4179575245 learned_addr learned my addr 172.21.15.6:0/4179575245 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:37.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.040+0000 7f110f400640 1 --2- 172.21.15.6:0/4179575245 >> v2:172.21.15.6:3300/0 conn(0x7f11100ae900 0x7f111015cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.040+0000 7f110f400640 1 -- 172.21.15.6:0/4179575245 >> v1:172.21.15.178:6789/0 conn(0x7f111006afb0 legacy=0x7f111006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.040+0000 7f110f400640 1 -- 172.21.15.6:0/4179575245 >> v1:172.21.15.6:6789/0 conn(0x7f111015bfb0 legacy=0x7f11100b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.041+0000 7f110f400640 1 -- 172.21.15.6:0/4179575245 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100af2e0 con 0x7f11100ae900 2024-03-06T07:29:37.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.041+0000 7f110e000640 1 -- 172.21.15.6:0/4179575245 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675284293 0 0) 0x7f11100aed30 con 0x7f111015bfb0 2024-03-06T07:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.042+0000 7f110f400640 1 --2- 172.21.15.6:0/4179575245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111015cda0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1108009d90 tx=0x7f110802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39535929b8b9023e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.042+0000 7f110e000640 1 -- 172.21.15.6:0/4179575245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108039070 con 0x7f11100ae900 2024-03-06T07:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.042+0000 7f110e000640 1 -- 172.21.15.6:0/4179575245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110802cd00 con 0x7f11100ae900 2024-03-06T07:29:37.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.042+0000 7f110e000640 1 -- 172.21.15.6:0/4179575245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108034660 con 0x7f11100ae900 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 -- 172.21.15.6:0/4179575245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 msgr2=0x7f111015cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 --2- 172.21.15.6:0/4179575245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111015cda0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1108009d90 tx=0x7f110802bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 -- 172.21.15.6:0/4179575245 shutdown_connections 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 --2- 172.21.15.6:0/4179575245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111015cda0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 -- 172.21.15.6:0/4179575245 >> 172.21.15.6:0/4179575245 conn(0x7f11100b2ae0 msgr2=0x7f11100b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 -- 172.21.15.6:0/4179575245 shutdown_connections 2024-03-06T07:29:37.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.043+0000 7f1115800640 1 -- 172.21.15.6:0/4179575245 wait complete. 2024-03-06T07:29:37.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.045+0000 7f1115800640 1 Processor -- start 2024-03-06T07:29:37.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.045+0000 7f1115800640 1 -- start start 2024-03-06T07:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.046+0000 7f1115800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111006afb0 0x7f11101534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.046+0000 7f1115800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111006f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.046+0000 7f1115800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111015bfb0 0x7f1110157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.046+0000 7f1115800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b4f40 con 0x7f11100ae900 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.046+0000 7f1115800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b4310 con 0x7f111006afb0 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.046+0000 7f1115800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b50c0 con 0x7f111015bfb0 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111006f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111006f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43278/0 (socket says 172.21.15.6:43278) 2024-03-06T07:29:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 -- 172.21.15.6:0/2941610158 learned_addr learned my addr 172.21.15.6:0/2941610158 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f110ea00640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111006afb0 0x7f11101534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f110f400640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111015bfb0 0x7f1110157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 -- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111015bfb0 msgr2=0x7f1110157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111015bfb0 0x7f1110157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 -- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111006afb0 msgr2=0x7f11101534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111006afb0 0x7f11101534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 -- 172.21.15.6:0/2941610158 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111015cd50 con 0x7f11100ae900 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.047+0000 7f1106000640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111006f360 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c3f0 tx=0x7f10f800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.048+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80074e0 con 0x7f11100ae900 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.048+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f80047e0 con 0x7f11100ae900 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.048+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8004c10 con 0x7f11100ae900 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.048+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111015d030 con 0x7f11100ae900 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.048+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111015d8d0 con 0x7f11100ae900 2024-03-06T07:29:37.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.049+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111015d200 con 0x7f11100ae900 2024-03-06T07:29:37.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.050+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10f8011070 con 0x7f11100ae900 2024-03-06T07:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.051+0000 7f1107400640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.051+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10f8089ea0 con 0x7f11100ae900 2024-03-06T07:29:37.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.052+0000 7f110ea00640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.053+0000 7f110ea00640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10ec06b4d0 0x7f10ec06d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1100000f60 tx=0x7f1100009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.056+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10f801a2b0 con 0x7f11100ae900 2024-03-06T07:29:37.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.253+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11100cea50 con 0x7f10ec06b4d0 2024-03-06T07:29:37.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.273+0000 7f1107400640 1 -- 172.21.15.6:0/2941610158 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251960 (secure 0 0 0) 0x7f11100cea50 con 0x7f10ec06b4d0 2024-03-06T07:29:37.278 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:37.280 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.278+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10ec06b4d0 msgr2=0x7f10ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.278+0000 7f1115800640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10ec06b4d0 0x7f10ec06d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1100000f60 tx=0x7f1100009040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.278+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 msgr2=0x7f111006f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.278+0000 7f1115800640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111006f360 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c3f0 tx=0x7f10f800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.278+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 shutdown_connections 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.278+0000 7f1115800640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10ec06b4d0 0x7f10ec06d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.279+0000 7f1115800640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111015bfb0 0x7f1110157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.279+0000 7f1115800640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f11100ae900 0x7f111006f360 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.279+0000 7f1115800640 1 --2- 172.21.15.6:0/2941610158 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111006afb0 0x7f11101534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.279+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 >> 172.21.15.6:0/2941610158 conn(0x7f11100b2ae0 msgr2=0x7f1110154180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.279+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 shutdown_connections 2024-03-06T07:29:37.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.279+0000 7f1115800640 1 -- 172.21.15.6:0/2941610158 wait complete. 2024-03-06T07:29:37.299 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-03-06T07:29:35.481888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259004,"kb_used_data":5384,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754715716,"statfs":{"total":773094113280,"available":772828893184,"internally_reserved":0,"allocated":5513216,"data_stored":2921556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.294846"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359873+0000","last_change":"2024-03-06T07:29:27.238818+0000","last_active":"2024-03-06T07:29:29.359873+0000","last_peered":"2024-03-06T07:29:29.359873+0000","last_clean":"2024-03-06T07:29:29.359873+0000","last_became_active":"2024-03-06T07:29:27.238533+0000","last_became_peered":"2024-03-06T07:29:27.238533+0000","last_unstale":"2024-03-06T07:29:29.359873+0000","last_undegraded":"2024-03-06T07:29:29.359873+0000","last_fullsized":"2024-03-06T07:29:29.359873+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:52.311496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197960+0000","last_change":"2024-03-06T07:29:27.232219+0000","last_active":"2024-03-06T07:29:29.197960+0000","last_peered":"2024-03-06T07:29:29.197960+0000","last_clean":"2024-03-06T07:29:29.197960+0000","last_became_active":"2024-03-06T07:29:27.231915+0000","last_became_peered":"2024-03-06T07:29:27.231915+0000","last_unstale":"2024-03-06T07:29:29.197960+0000","last_undegraded":"2024-03-06T07:29:29.197960+0000","last_fullsized":"2024-03-06T07:29:29.197960+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:04.309774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240247+0000","last_change":"2024-03-06T07:29:27.231359+0000","last_active":"2024-03-06T07:29:28.240247+0000","last_peered":"2024-03-06T07:29:28.240247+0000","last_clean":"2024-03-06T07:29:28.240247+0000","last_became_active":"2024-03-06T07:29:27.231038+0000","last_became_peered":"2024-03-06T07:29:27.231038+0000","last_unstale":"2024-03-06T07:29:28.240247+0000","last_undegraded":"2024-03-06T07:29:28.240247+0000","last_fullsized":"2024-03-06T07:29:28.240247+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:48.483161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193663+0000","last_change":"2024-03-06T07:29:27.219886+0000","last_active":"2024-03-06T07:29:28.193663+0000","last_peered":"2024-03-06T07:29:28.193663+0000","last_clean":"2024-03-06T07:29:28.193663+0000","last_became_active":"2024-03-06T07:29:27.219534+0000","last_became_peered":"2024-03-06T07:29:27.219534+0000","last_unstale":"2024-03-06T07:29:28.193663+0000","last_undegraded":"2024-03-06T07:29:28.193663+0000","last_fullsized":"2024-03-06T07:29:28.193663+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.040231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197432+0000","last_change":"2024-03-06T07:29:27.227793+0000","last_active":"2024-03-06T07:29:29.197432+0000","last_peered":"2024-03-06T07:29:29.197432+0000","last_clean":"2024-03-06T07:29:29.197432+0000","last_became_active":"2024-03-06T07:29:27.226210+0000","last_became_peered":"2024-03-06T07:29:27.226210+0000","last_unstale":"2024-03-06T07:29:29.197432+0000","last_undegraded":"2024-03-06T07:29:29.197432+0000","last_fullsized":"2024-03-06T07:29:29.197432+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:19.753745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.190958+0000","last_change":"2024-03-06T07:29:27.210023+0000","last_active":"2024-03-06T07:29:28.190958+0000","last_peered":"2024-03-06T07:29:28.190958+0000","last_clean":"2024-03-06T07:29:28.190958+0000","last_became_active":"2024-03-06T07:29:27.209281+0000","last_became_peered":"2024-03-06T07:29:27.209281+0000","last_unstale":"2024-03-06T07:29:28.190958+0000","last_undegraded":"2024-03-06T07:29:28.190958+0000","last_fullsized":"2024-03-06T07:29:28.190958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:46.291448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.357564+0000","last_change":"2024-03-06T07:29:27.226824+0000","last_active":"2024-03-06T07:29:29.357564+0000","last_peered":"2024-03-06T07:29:29.357564+0000","last_clean":"2024-03-06T07:29:29.357564+0000","last_became_active":"2024-03-06T07:29:27.226498+0000","last_became_peered":"2024-03-06T07:29:27.226498+0000","last_unstale":"2024-03-06T07:29:29.357564+0000","last_undegraded":"2024-03-06T07:29:29.357564+0000","last_fullsized":"2024-03-06T07:29:29.357564+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:55.566106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192296+0000","last_change":"2024-03-06T07:29:27.209995+0000","last_active":"2024-03-06T07:29:28.192296+0000","last_peered":"2024-03-06T07:29:28.192296+0000","last_clean":"2024-03-06T07:29:28.192296+0000","last_became_active":"2024-03-06T07:29:27.206041+0000","last_became_peered":"2024-03-06T07:29:27.206041+0000","last_unstale":"2024-03-06T07:29:28.192296+0000","last_undegraded":"2024-03-06T07:29:28.192296+0000","last_fullsized":"2024-03-06T07:29:28.192296+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:09.195915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194576+0000","last_change":"2024-03-06T07:29:27.226491+0000","last_active":"2024-03-06T07:29:28.194576+0000","last_peered":"2024-03-06T07:29:28.194576+0000","last_clean":"2024-03-06T07:29:28.194576+0000","last_became_active":"2024-03-06T07:29:27.225897+0000","last_became_peered":"2024-03-06T07:29:27.225897+0000","last_unstale":"2024-03-06T07:29:28.194576+0000","last_undegraded":"2024-03-06T07:29:28.194576+0000","last_fullsized":"2024-03-06T07:29:28.194576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:19.753745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193571+0000","last_change":"2024-03-06T07:29:27.210338+0000","last_active":"2024-03-06T07:29:28.193571+0000","last_peered":"2024-03-06T07:29:28.193571+0000","last_clean":"2024-03-06T07:29:28.193571+0000","last_became_active":"2024-03-06T07:29:27.207720+0000","last_became_peered":"2024-03-06T07:29:27.207720+0000","last_unstale":"2024-03-06T07:29:28.193571+0000","last_undegraded":"2024-03-06T07:29:28.193571+0000","last_fullsized":"2024-03-06T07:29:28.193571+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:41.530505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192842+0000","last_change":"2024-03-06T07:29:27.224388+0000","last_active":"2024-03-06T07:29:28.192842+0000","last_peered":"2024-03-06T07:29:28.192842+0000","last_clean":"2024-03-06T07:29:28.192842+0000","last_became_active":"2024-03-06T07:29:27.223984+0000","last_became_peered":"2024-03-06T07:29:27.223984+0000","last_unstale":"2024-03-06T07:29:28.192842+0000","last_undegraded":"2024-03-06T07:29:28.192842+0000","last_fullsized":"2024-03-06T07:29:28.192842+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:19.935722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197378+0000","last_change":"2024-03-06T07:29:27.229326+0000","last_active":"2024-03-06T07:29:29.197378+0000","last_peered":"2024-03-06T07:29:29.197378+0000","last_clean":"2024-03-06T07:29:29.197378+0000","last_became_active":"2024-03-06T07:29:27.228999+0000","last_became_peered":"2024-03-06T07:29:27.228999+0000","last_unstale":"2024-03-06T07:29:29.197378+0000","last_undegraded":"2024-03-06T07:29:29.197378+0000","last_fullsized":"2024-03-06T07:29:29.197378+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:20.040231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194301+0000","last_change":"2024-03-06T07:29:27.234158+0000","last_active":"2024-03-06T07:29:28.194301+0000","last_peered":"2024-03-06T07:29:28.194301+0000","last_clean":"2024-03-06T07:29:28.194301+0000","last_became_active":"2024-03-06T07:29:27.233803+0000","last_became_peered":"2024-03-06T07:29:27.233803+0000","last_unstale":"2024-03-06T07:29:28.194301+0000","last_undegraded":"2024-03-06T07:29:28.194301+0000","last_fullsized":"2024-03-06T07:29:28.194301+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:04:29.033715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192721+0000","last_change":"2024-03-06T07:29:27.221609+0000","last_active":"2024-03-06T07:29:28.192721+0000","last_peered":"2024-03-06T07:29:28.192721+0000","last_clean":"2024-03-06T07:29:28.192721+0000","last_became_active":"2024-03-06T07:29:27.221287+0000","last_became_peered":"2024-03-06T07:29:27.221287+0000","last_unstale":"2024-03-06T07:29:28.192721+0000","last_undegraded":"2024-03-06T07:29:28.192721+0000","last_fullsized":"2024-03-06T07:29:28.192721+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:13.654418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191093+0000","last_change":"2024-03-06T07:29:27.220611+0000","last_active":"2024-03-06T07:29:28.191093+0000","last_peered":"2024-03-06T07:29:28.191093+0000","last_clean":"2024-03-06T07:29:28.191093+0000","last_became_active":"2024-03-06T07:29:27.220033+0000","last_became_peered":"2024-03-06T07:29:27.220033+0000","last_unstale":"2024-03-06T07:29:28.191093+0000","last_undegraded":"2024-03-06T07:29:28.191093+0000","last_fullsized":"2024-03-06T07:29:28.191093+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:02.273553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358072+0000","last_change":"2024-03-06T07:29:27.226127+0000","last_active":"2024-03-06T07:29:29.358072+0000","last_peered":"2024-03-06T07:29:29.358072+0000","last_clean":"2024-03-06T07:29:29.358072+0000","last_became_active":"2024-03-06T07:29:27.225234+0000","last_became_peered":"2024-03-06T07:29:27.225234+0000","last_unstale":"2024-03-06T07:29:29.358072+0000","last_undegraded":"2024-03-06T07:29:29.358072+0000","last_fullsized":"2024-03-06T07:29:29.358072+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:34.153592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240623+0000","last_change":"2024-03-06T07:29:27.218384+0000","last_active":"2024-03-06T07:29:28.240623+0000","last_peered":"2024-03-06T07:29:28.240623+0000","last_clean":"2024-03-06T07:29:28.240623+0000","last_became_active":"2024-03-06T07:29:27.218081+0000","last_became_peered":"2024-03-06T07:29:27.218081+0000","last_unstale":"2024-03-06T07:29:28.240623+0000","last_undegraded":"2024-03-06T07:29:28.240623+0000","last_fullsized":"2024-03-06T07:29:28.240623+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:52.311496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240588+0000","last_change":"2024-03-06T07:29:27.233319+0000","last_active":"2024-03-06T07:29:28.240588+0000","last_peered":"2024-03-06T07:29:28.240588+0000","last_clean":"2024-03-06T07:29:28.240588+0000","last_became_active":"2024-03-06T07:29:27.232641+0000","last_became_peered":"2024-03-06T07:29:27.232641+0000","last_unstale":"2024-03-06T07:29:28.240588+0000","last_undegraded":"2024-03-06T07:29:28.240588+0000","last_fullsized":"2024-03-06T07:29:28.240588+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:02.131796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359784+0000","last_change":"2024-03-06T07:29:27.215755+0000","last_active":"2024-03-06T07:29:29.359784+0000","last_peered":"2024-03-06T07:29:29.359784+0000","last_clean":"2024-03-06T07:29:29.359784+0000","last_became_active":"2024-03-06T07:29:27.215473+0000","last_became_peered":"2024-03-06T07:29:27.215473+0000","last_unstale":"2024-03-06T07:29:29.359784+0000","last_undegraded":"2024-03-06T07:29:29.359784+0000","last_fullsized":"2024-03-06T07:29:29.359784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:35.893406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193974+0000","last_change":"2024-03-06T07:29:27.232657+0000","last_active":"2024-03-06T07:29:28.193974+0000","last_peered":"2024-03-06T07:29:28.193974+0000","last_clean":"2024-03-06T07:29:28.193974+0000","last_became_active":"2024-03-06T07:29:27.232358+0000","last_became_peered":"2024-03-06T07:29:27.232358+0000","last_unstale":"2024-03-06T07:29:28.193974+0000","last_undegraded":"2024-03-06T07:29:28.193974+0000","last_fullsized":"2024-03-06T07:29:28.193974+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:45:10.665455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191029+0000","last_change":"2024-03-06T07:29:27.203643+0000","last_active":"2024-03-06T07:29:28.191029+0000","last_peered":"2024-03-06T07:29:28.191029+0000","last_clean":"2024-03-06T07:29:28.191029+0000","last_became_active":"2024-03-06T07:29:27.203417+0000","last_became_peered":"2024-03-06T07:29:27.203417+0000","last_unstale":"2024-03-06T07:29:28.191029+0000","last_undegraded":"2024-03-06T07:29:28.191029+0000","last_fullsized":"2024-03-06T07:29:28.191029+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:27:20.006887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192897+0000","last_change":"2024-03-06T07:29:27.208247+0000","last_active":"2024-03-06T07:29:28.192897+0000","last_peered":"2024-03-06T07:29:28.192897+0000","last_clean":"2024-03-06T07:29:28.192897+0000","last_became_active":"2024-03-06T07:29:27.205384+0000","last_became_peered":"2024-03-06T07:29:27.205384+0000","last_unstale":"2024-03-06T07:29:28.192897+0000","last_undegraded":"2024-03-06T07:29:28.192897+0000","last_fullsized":"2024-03-06T07:29:28.192897+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:13.055064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358675+0000","last_change":"2024-03-06T07:29:27.217290+0000","last_active":"2024-03-06T07:29:29.358675+0000","last_peered":"2024-03-06T07:29:29.358675+0000","last_clean":"2024-03-06T07:29:29.358675+0000","last_became_active":"2024-03-06T07:29:27.216501+0000","last_became_peered":"2024-03-06T07:29:27.216501+0000","last_unstale":"2024-03-06T07:29:29.358675+0000","last_undegraded":"2024-03-06T07:29:29.358675+0000","last_fullsized":"2024-03-06T07:29:29.358675+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:16.663232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358217+0000","last_change":"2024-03-06T07:29:27.225865+0000","last_active":"2024-03-06T07:29:29.358217+0000","last_peered":"2024-03-06T07:29:29.358217+0000","last_clean":"2024-03-06T07:29:29.358217+0000","last_became_active":"2024-03-06T07:29:27.224862+0000","last_became_peered":"2024-03-06T07:29:27.224862+0000","last_unstale":"2024-03-06T07:29:29.358217+0000","last_undegraded":"2024-03-06T07:29:29.358217+0000","last_fullsized":"2024-03-06T07:29:29.358217+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:06.088588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358808+0000","last_change":"2024-03-06T07:29:27.225091+0000","last_active":"2024-03-06T07:29:29.358808+0000","last_peered":"2024-03-06T07:29:29.358808+0000","last_clean":"2024-03-06T07:29:29.358808+0000","last_became_active":"2024-03-06T07:29:27.224189+0000","last_became_peered":"2024-03-06T07:29:27.224189+0000","last_unstale":"2024-03-06T07:29:29.358808+0000","last_undegraded":"2024-03-06T07:29:29.358808+0000","last_fullsized":"2024-03-06T07:29:29.358808+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:23:30.764633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193880+0000","last_change":"2024-03-06T07:29:27.234582+0000","last_active":"2024-03-06T07:29:28.193880+0000","last_peered":"2024-03-06T07:29:28.193880+0000","last_clean":"2024-03-06T07:29:28.193880+0000","last_became_active":"2024-03-06T07:29:27.233810+0000","last_became_peered":"2024-03-06T07:29:27.233810+0000","last_unstale":"2024-03-06T07:29:28.193880+0000","last_undegraded":"2024-03-06T07:29:28.193880+0000","last_fullsized":"2024-03-06T07:29:28.193880+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:08:57.366619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240327+0000","last_change":"2024-03-06T07:29:27.222794+0000","last_active":"2024-03-06T07:29:28.240327+0000","last_peered":"2024-03-06T07:29:28.240327+0000","last_clean":"2024-03-06T07:29:28.240327+0000","last_became_active":"2024-03-06T07:29:27.222515+0000","last_became_peered":"2024-03-06T07:29:27.222515+0000","last_unstale":"2024-03-06T07:29:28.240327+0000","last_undegraded":"2024-03-06T07:29:28.240327+0000","last_fullsized":"2024-03-06T07:29:28.240327+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:21.095594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197900+0000","last_change":"2024-03-06T07:29:27.221671+0000","last_active":"2024-03-06T07:29:29.197900+0000","last_peered":"2024-03-06T07:29:29.197900+0000","last_clean":"2024-03-06T07:29:29.197900+0000","last_became_active":"2024-03-06T07:29:27.221331+0000","last_became_peered":"2024-03-06T07:29:27.221331+0000","last_unstale":"2024-03-06T07:29:29.197900+0000","last_undegraded":"2024-03-06T07:29:29.197900+0000","last_fullsized":"2024-03-06T07:29:29.197900+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.020983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358033+0000","last_change":"2024-03-06T07:29:27.223801+0000","last_active":"2024-03-06T07:29:29.358033+0000","last_peered":"2024-03-06T07:29:29.358033+0000","last_clean":"2024-03-06T07:29:29.358033+0000","last_became_active":"2024-03-06T07:29:27.223470+0000","last_became_peered":"2024-03-06T07:29:27.223470+0000","last_unstale":"2024-03-06T07:29:29.358033+0000","last_undegraded":"2024-03-06T07:29:29.358033+0000","last_fullsized":"2024-03-06T07:29:29.358033+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:51:10.599929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.239849+0000","last_change":"2024-03-06T07:29:27.226174+0000","last_active":"2024-03-06T07:29:28.239849+0000","last_peered":"2024-03-06T07:29:28.239849+0000","last_clean":"2024-03-06T07:29:28.239849+0000","last_became_active":"2024-03-06T07:29:27.225908+0000","last_became_peered":"2024-03-06T07:29:27.225908+0000","last_unstale":"2024-03-06T07:29:28.239849+0000","last_undegraded":"2024-03-06T07:29:28.239849+0000","last_fullsized":"2024-03-06T07:29:28.239849+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:02:30.251125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191536+0000","last_change":"2024-03-06T07:29:27.216498+0000","last_active":"2024-03-06T07:29:28.191536+0000","last_peered":"2024-03-06T07:29:28.191536+0000","last_clean":"2024-03-06T07:29:28.191536+0000","last_became_active":"2024-03-06T07:29:27.213938+0000","last_became_peered":"2024-03-06T07:29:27.213938+0000","last_unstale":"2024-03-06T07:29:28.191536+0000","last_undegraded":"2024-03-06T07:29:28.191536+0000","last_fullsized":"2024-03-06T07:29:28.191536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:12:44.006644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192845+0000","last_change":"2024-03-06T07:29:27.227701+0000","last_active":"2024-03-06T07:29:28.192845+0000","last_peered":"2024-03-06T07:29:28.192845+0000","last_clean":"2024-03-06T07:29:28.192845+0000","last_became_active":"2024-03-06T07:29:27.225595+0000","last_became_peered":"2024-03-06T07:29:27.225595+0000","last_unstale":"2024-03-06T07:29:28.192845+0000","last_undegraded":"2024-03-06T07:29:28.192845+0000","last_fullsized":"2024-03-06T07:29:28.192845+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:27.122537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.356669+0000","last_change":"2024-03-06T07:29:27.222401+0000","last_active":"2024-03-06T07:29:29.356669+0000","last_peered":"2024-03-06T07:29:29.356669+0000","last_clean":"2024-03-06T07:29:29.356669+0000","last_became_active":"2024-03-06T07:29:27.221978+0000","last_became_peered":"2024-03-06T07:29:27.221978+0000","last_unstale":"2024-03-06T07:29:29.356669+0000","last_undegraded":"2024-03-06T07:29:29.356669+0000","last_fullsized":"2024-03-06T07:29:29.356669+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:32.247068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191977+0000","last_change":"2024-03-06T07:29:26.184227+0000","last_active":"2024-03-06T07:29:28.191977+0000","last_peered":"2024-03-06T07:29:28.191977+0000","last_clean":"2024-03-06T07:29:28.191977+0000","last_became_active":"2024-03-06T07:29:26.183923+0000","last_became_peered":"2024-03-06T07:29:26.183923+0000","last_unstale":"2024-03-06T07:29:28.191977+0000","last_undegraded":"2024-03-06T07:29:28.191977+0000","last_fullsized":"2024-03-06T07:29:28.191977+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:41.006960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359470+0000","last_change":"2024-03-06T07:29:27.219177+0000","last_active":"2024-03-06T07:29:29.359470+0000","last_peered":"2024-03-06T07:29:29.359470+0000","last_clean":"2024-03-06T07:29:29.359470+0000","last_became_active":"2024-03-06T07:29:27.218700+0000","last_became_peered":"2024-03-06T07:29:27.218700+0000","last_unstale":"2024-03-06T07:29:29.359470+0000","last_undegraded":"2024-03-06T07:29:29.359470+0000","last_fullsized":"2024-03-06T07:29:29.359470+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:41.102346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240751+0000","last_change":"2024-03-06T07:29:26.200719+0000","last_active":"2024-03-06T07:29:28.240751+0000","last_peered":"2024-03-06T07:29:28.240751+0000","last_clean":"2024-03-06T07:29:28.240751+0000","last_became_active":"2024-03-06T07:29:26.200438+0000","last_became_peered":"2024-03-06T07:29:26.200438+0000","last_unstale":"2024-03-06T07:29:28.240751+0000","last_undegraded":"2024-03-06T07:29:28.240751+0000","last_fullsized":"2024-03-06T07:29:28.240751+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197916+0000","last_change":"2024-03-06T07:29:27.222145+0000","last_active":"2024-03-06T07:29:29.197916+0000","last_peered":"2024-03-06T07:29:29.197916+0000","last_clean":"2024-03-06T07:29:29.197916+0000","last_became_active":"2024-03-06T07:29:27.221795+0000","last_became_peered":"2024-03-06T07:29:27.221795+0000","last_unstale":"2024-03-06T07:29:29.197916+0000","last_undegraded":"2024-03-06T07:29:29.197916+0000","last_fullsized":"2024-03-06T07:29:29.197916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:25.630352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.356799+0000","last_change":"2024-03-06T07:29:26.201813+0000","last_active":"2024-03-06T07:29:29.356799+0000","last_peered":"2024-03-06T07:29:29.356799+0000","last_clean":"2024-03-06T07:29:29.356799+0000","last_became_active":"2024-03-06T07:29:26.201360+0000","last_became_peered":"2024-03-06T07:29:26.201360+0000","last_unstale":"2024-03-06T07:29:29.356799+0000","last_undegraded":"2024-03-06T07:29:29.356799+0000","last_fullsized":"2024-03-06T07:29:29.356799+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.356503+0000","last_change":"2024-03-06T07:29:27.231046+0000","last_active":"2024-03-06T07:29:29.356503+0000","last_peered":"2024-03-06T07:29:29.356503+0000","last_clean":"2024-03-06T07:29:29.356503+0000","last_became_active":"2024-03-06T07:29:27.230774+0000","last_became_peered":"2024-03-06T07:29:27.230774+0000","last_unstale":"2024-03-06T07:29:29.356503+0000","last_undegraded":"2024-03-06T07:29:29.356503+0000","last_fullsized":"2024-03-06T07:29:29.356503+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:54.206839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359102+0000","last_change":"2024-03-06T07:29:26.202950+0000","last_active":"2024-03-06T07:29:29.359102+0000","last_peered":"2024-03-06T07:29:29.359102+0000","last_clean":"2024-03-06T07:29:29.359102+0000","last_became_active":"2024-03-06T07:29:26.202564+0000","last_became_peered":"2024-03-06T07:29:26.202564+0000","last_unstale":"2024-03-06T07:29:29.359102+0000","last_undegraded":"2024-03-06T07:29:29.359102+0000","last_fullsized":"2024-03-06T07:29:29.359102+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:31.233046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.239929+0000","last_change":"2024-03-06T07:29:27.236948+0000","last_active":"2024-03-06T07:29:28.239929+0000","last_peered":"2024-03-06T07:29:28.239929+0000","last_clean":"2024-03-06T07:29:28.239929+0000","last_became_active":"2024-03-06T07:29:27.235931+0000","last_became_peered":"2024-03-06T07:29:27.235931+0000","last_unstale":"2024-03-06T07:29:28.239929+0000","last_undegraded":"2024-03-06T07:29:28.239929+0000","last_fullsized":"2024-03-06T07:29:28.239929+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:32.673285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191146+0000","last_change":"2024-03-06T07:29:26.188354+0000","last_active":"2024-03-06T07:29:28.191146+0000","last_peered":"2024-03-06T07:29:28.191146+0000","last_clean":"2024-03-06T07:29:28.191146+0000","last_became_active":"2024-03-06T07:29:26.187378+0000","last_became_peered":"2024-03-06T07:29:26.187378+0000","last_unstale":"2024-03-06T07:29:28.191146+0000","last_undegraded":"2024-03-06T07:29:28.191146+0000","last_fullsized":"2024-03-06T07:29:28.191146+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:12.041042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193274+0000","last_change":"2024-03-06T07:29:27.224616+0000","last_active":"2024-03-06T07:29:28.193274+0000","last_peered":"2024-03-06T07:29:28.193274+0000","last_clean":"2024-03-06T07:29:28.193274+0000","last_became_active":"2024-03-06T07:29:27.223661+0000","last_became_peered":"2024-03-06T07:29:27.223661+0000","last_unstale":"2024-03-06T07:29:28.193274+0000","last_undegraded":"2024-03-06T07:29:28.193274+0000","last_fullsized":"2024-03-06T07:29:28.193274+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:52.966337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240090+0000","last_change":"2024-03-06T07:29:26.204279+0000","last_active":"2024-03-06T07:29:28.240090+0000","last_peered":"2024-03-06T07:29:28.240090+0000","last_clean":"2024-03-06T07:29:28.240090+0000","last_became_active":"2024-03-06T07:29:26.203822+0000","last_became_peered":"2024-03-06T07:29:26.203822+0000","last_unstale":"2024-03-06T07:29:28.240090+0000","last_undegraded":"2024-03-06T07:29:28.240090+0000","last_fullsized":"2024-03-06T07:29:28.240090+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192371+0000","last_change":"2024-03-06T07:29:27.215308+0000","last_active":"2024-03-06T07:29:28.192371+0000","last_peered":"2024-03-06T07:29:28.192371+0000","last_clean":"2024-03-06T07:29:28.192371+0000","last_became_active":"2024-03-06T07:29:27.212232+0000","last_became_peered":"2024-03-06T07:29:27.212232+0000","last_unstale":"2024-03-06T07:29:28.192371+0000","last_undegraded":"2024-03-06T07:29:28.192371+0000","last_fullsized":"2024-03-06T07:29:28.192371+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:42.663881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.196950+0000","last_change":"2024-03-06T07:29:26.194000+0000","last_active":"2024-03-06T07:29:29.196950+0000","last_peered":"2024-03-06T07:29:29.196950+0000","last_clean":"2024-03-06T07:29:29.196950+0000","last_became_active":"2024-03-06T07:29:26.193684+0000","last_became_peered":"2024-03-06T07:29:26.193684+0000","last_unstale":"2024-03-06T07:29:29.196950+0000","last_undegraded":"2024-03-06T07:29:29.196950+0000","last_fullsized":"2024-03-06T07:29:29.196950+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:34.879384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358071+0000","last_change":"2024-03-06T07:29:27.217504+0000","last_active":"2024-03-06T07:29:29.358071+0000","last_peered":"2024-03-06T07:29:29.358071+0000","last_clean":"2024-03-06T07:29:29.358071+0000","last_became_active":"2024-03-06T07:29:27.216934+0000","last_became_peered":"2024-03-06T07:29:27.216934+0000","last_unstale":"2024-03-06T07:29:29.358071+0000","last_undegraded":"2024-03-06T07:29:29.358071+0000","last_fullsized":"2024-03-06T07:29:29.358071+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:59.041898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192441+0000","last_change":"2024-03-06T07:29:26.182870+0000","last_active":"2024-03-06T07:29:28.192441+0000","last_peered":"2024-03-06T07:29:28.192441+0000","last_clean":"2024-03-06T07:29:28.192441+0000","last_became_active":"2024-03-06T07:29:26.182537+0000","last_became_peered":"2024-03-06T07:29:26.182537+0000","last_unstale":"2024-03-06T07:29:28.192441+0000","last_undegraded":"2024-03-06T07:29:28.192441+0000","last_fullsized":"2024-03-06T07:29:28.192441+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:19.260089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240582+0000","last_change":"2024-03-06T07:29:27.233586+0000","last_active":"2024-03-06T07:29:28.240582+0000","last_peered":"2024-03-06T07:29:28.240582+0000","last_clean":"2024-03-06T07:29:28.240582+0000","last_became_active":"2024-03-06T07:29:27.233008+0000","last_became_peered":"2024-03-06T07:29:27.233008+0000","last_unstale":"2024-03-06T07:29:28.240582+0000","last_undegraded":"2024-03-06T07:29:28.240582+0000","last_fullsized":"2024-03-06T07:29:28.240582+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:55.566106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240640+0000","last_change":"2024-03-06T07:29:26.193099+0000","last_active":"2024-03-06T07:29:28.240640+0000","last_peered":"2024-03-06T07:29:28.240640+0000","last_clean":"2024-03-06T07:29:28.240640+0000","last_became_active":"2024-03-06T07:29:26.192805+0000","last_became_peered":"2024-03-06T07:29:26.192805+0000","last_unstale":"2024-03-06T07:29:28.240640+0000","last_undegraded":"2024-03-06T07:29:28.240640+0000","last_fullsized":"2024-03-06T07:29:28.240640+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:59.388905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359941+0000","last_change":"2024-03-06T07:29:27.233092+0000","last_active":"2024-03-06T07:29:29.359941+0000","last_peered":"2024-03-06T07:29:29.359941+0000","last_clean":"2024-03-06T07:29:29.359941+0000","last_became_active":"2024-03-06T07:29:27.232781+0000","last_became_peered":"2024-03-06T07:29:27.232781+0000","last_unstale":"2024-03-06T07:29:29.359941+0000","last_undegraded":"2024-03-06T07:29:29.359941+0000","last_fullsized":"2024-03-06T07:29:29.359941+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:40.080803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194292+0000","last_change":"2024-03-06T07:29:26.186625+0000","last_active":"2024-03-06T07:29:28.194292+0000","last_peered":"2024-03-06T07:29:28.194292+0000","last_clean":"2024-03-06T07:29:28.194292+0000","last_became_active":"2024-03-06T07:29:26.186363+0000","last_became_peered":"2024-03-06T07:29:26.186363+0000","last_unstale":"2024-03-06T07:29:28.194292+0000","last_undegraded":"2024-03-06T07:29:28.194292+0000","last_fullsized":"2024-03-06T07:29:28.194292+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:03.733713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191652+0000","last_change":"2024-03-06T07:29:27.224493+0000","last_active":"2024-03-06T07:29:28.191652+0000","last_peered":"2024-03-06T07:29:28.191652+0000","last_clean":"2024-03-06T07:29:28.191652+0000","last_became_active":"2024-03-06T07:29:27.224147+0000","last_became_peered":"2024-03-06T07:29:27.224147+0000","last_unstale":"2024-03-06T07:29:28.191652+0000","last_undegraded":"2024-03-06T07:29:28.191652+0000","last_fullsized":"2024-03-06T07:29:28.191652+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:05.148277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":17,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360444+0000","last_change":"2024-03-06T07:29:26.197651+0000","last_active":"2024-03-06T07:29:29.360444+0000","last_peered":"2024-03-06T07:29:29.360444+0000","last_clean":"2024-03-06T07:29:29.360444+0000","last_became_active":"2024-03-06T07:29:26.197161+0000","last_became_peered":"2024-03-06T07:29:26.197161+0000","last_unstale":"2024-03-06T07:29:29.360444+0000","last_undegraded":"2024-03-06T07:29:29.360444+0000","last_fullsized":"2024-03-06T07:29:29.360444+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:56.450035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360085+0000","last_change":"2024-03-06T07:29:27.230126+0000","last_active":"2024-03-06T07:29:29.360085+0000","last_peered":"2024-03-06T07:29:29.360085+0000","last_clean":"2024-03-06T07:29:29.360085+0000","last_became_active":"2024-03-06T07:29:27.229726+0000","last_became_peered":"2024-03-06T07:29:27.229726+0000","last_unstale":"2024-03-06T07:29:29.360085+0000","last_undegraded":"2024-03-06T07:29:29.360085+0000","last_fullsized":"2024-03-06T07:29:29.360085+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:59.335223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193894+0000","last_change":"2024-03-06T07:29:26.200648+0000","last_active":"2024-03-06T07:29:28.193894+0000","last_peered":"2024-03-06T07:29:28.193894+0000","last_clean":"2024-03-06T07:29:28.193894+0000","last_became_active":"2024-03-06T07:29:26.200365+0000","last_became_peered":"2024-03-06T07:29:26.200365+0000","last_unstale":"2024-03-06T07:29:28.193894+0000","last_undegraded":"2024-03-06T07:29:28.193894+0000","last_fullsized":"2024-03-06T07:29:28.193894+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:56.411331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358294+0000","last_change":"2024-03-06T07:29:27.217938+0000","last_active":"2024-03-06T07:29:29.358294+0000","last_peered":"2024-03-06T07:29:29.358294+0000","last_clean":"2024-03-06T07:29:29.358294+0000","last_became_active":"2024-03-06T07:29:27.217309+0000","last_became_peered":"2024-03-06T07:29:27.217309+0000","last_unstale":"2024-03-06T07:29:29.358294+0000","last_undegraded":"2024-03-06T07:29:29.358294+0000","last_fullsized":"2024-03-06T07:29:29.358294+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:48.496514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191206+0000","last_change":"2024-03-06T07:29:26.180918+0000","last_active":"2024-03-06T07:29:28.191206+0000","last_peered":"2024-03-06T07:29:28.191206+0000","last_clean":"2024-03-06T07:29:28.191206+0000","last_became_active":"2024-03-06T07:29:26.180570+0000","last_became_peered":"2024-03-06T07:29:26.180570+0000","last_unstale":"2024-03-06T07:29:28.191206+0000","last_undegraded":"2024-03-06T07:29:28.191206+0000","last_fullsized":"2024-03-06T07:29:28.191206+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:19.169731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193427+0000","last_change":"2024-03-06T07:29:27.220767+0000","last_active":"2024-03-06T07:29:28.193427+0000","last_peered":"2024-03-06T07:29:28.193427+0000","last_clean":"2024-03-06T07:29:28.193427+0000","last_became_active":"2024-03-06T07:29:27.220493+0000","last_became_peered":"2024-03-06T07:29:27.220493+0000","last_unstale":"2024-03-06T07:29:28.193427+0000","last_undegraded":"2024-03-06T07:29:28.193427+0000","last_fullsized":"2024-03-06T07:29:28.193427+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:20.183753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359155+0000","last_change":"2024-03-06T07:29:26.194437+0000","last_active":"2024-03-06T07:29:29.359155+0000","last_peered":"2024-03-06T07:29:29.359155+0000","last_clean":"2024-03-06T07:29:29.359155+0000","last_became_active":"2024-03-06T07:29:26.194154+0000","last_became_peered":"2024-03-06T07:29:26.194154+0000","last_unstale":"2024-03-06T07:29:29.359155+0000","last_undegraded":"2024-03-06T07:29:29.359155+0000","last_fullsized":"2024-03-06T07:29:29.359155+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:08.796194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.190967+0000","last_change":"2024-03-06T07:29:27.222589+0000","last_active":"2024-03-06T07:29:28.190967+0000","last_peered":"2024-03-06T07:29:28.190967+0000","last_clean":"2024-03-06T07:29:28.190967+0000","last_became_active":"2024-03-06T07:29:27.222280+0000","last_became_peered":"2024-03-06T07:29:27.222280+0000","last_unstale":"2024-03-06T07:29:28.190967+0000","last_undegraded":"2024-03-06T07:29:28.190967+0000","last_fullsized":"2024-03-06T07:29:28.190967+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:44:17.154562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191810+0000","last_change":"2024-03-06T07:29:26.185585+0000","last_active":"2024-03-06T07:29:28.191810+0000","last_peered":"2024-03-06T07:29:28.191810+0000","last_clean":"2024-03-06T07:29:28.191810+0000","last_became_active":"2024-03-06T07:29:26.180225+0000","last_became_peered":"2024-03-06T07:29:26.180225+0000","last_unstale":"2024-03-06T07:29:28.191810+0000","last_undegraded":"2024-03-06T07:29:28.191810+0000","last_fullsized":"2024-03-06T07:29:28.191810+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:24.616330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359365+0000","last_change":"2024-03-06T07:29:27.232136+0000","last_active":"2024-03-06T07:29:29.359365+0000","last_peered":"2024-03-06T07:29:29.359365+0000","last_clean":"2024-03-06T07:29:29.359365+0000","last_became_active":"2024-03-06T07:29:27.231588+0000","last_became_peered":"2024-03-06T07:29:27.231588+0000","last_unstale":"2024-03-06T07:29:29.359365+0000","last_undegraded":"2024-03-06T07:29:29.359365+0000","last_fullsized":"2024-03-06T07:29:29.359365+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:02.131796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192468+0000","last_change":"2024-03-06T07:29:26.194945+0000","last_active":"2024-03-06T07:29:28.192468+0000","last_peered":"2024-03-06T07:29:28.192468+0000","last_clean":"2024-03-06T07:29:28.192468+0000","last_became_active":"2024-03-06T07:29:26.194577+0000","last_became_peered":"2024-03-06T07:29:26.194577+0000","last_unstale":"2024-03-06T07:29:28.192468+0000","last_undegraded":"2024-03-06T07:29:28.192468+0000","last_fullsized":"2024-03-06T07:29:28.192468+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:23.475086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194058+0000","last_change":"2024-03-06T07:29:27.217848+0000","last_active":"2024-03-06T07:29:28.194058+0000","last_peered":"2024-03-06T07:29:28.194058+0000","last_clean":"2024-03-06T07:29:28.194058+0000","last_became_active":"2024-03-06T07:29:27.217149+0000","last_became_peered":"2024-03-06T07:29:27.217149+0000","last_unstale":"2024-03-06T07:29:28.194058+0000","last_undegraded":"2024-03-06T07:29:28.194058+0000","last_fullsized":"2024-03-06T07:29:28.194058+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:23.354165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194197+0000","last_change":"2024-03-06T07:29:26.188532+0000","last_active":"2024-03-06T07:29:28.194197+0000","last_peered":"2024-03-06T07:29:28.194197+0000","last_clean":"2024-03-06T07:29:28.194197+0000","last_became_active":"2024-03-06T07:29:26.188243+0000","last_became_peered":"2024-03-06T07:29:26.188243+0000","last_unstale":"2024-03-06T07:29:28.194197+0000","last_undegraded":"2024-03-06T07:29:28.194197+0000","last_fullsized":"2024-03-06T07:29:28.194197+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:05.043428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194587+0000","last_change":"2024-03-06T07:29:27.211086+0000","last_active":"2024-03-06T07:29:28.194587+0000","last_peered":"2024-03-06T07:29:28.194587+0000","last_clean":"2024-03-06T07:29:28.194587+0000","last_became_active":"2024-03-06T07:29:27.210737+0000","last_became_peered":"2024-03-06T07:29:27.210737+0000","last_unstale":"2024-03-06T07:29:28.194587+0000","last_undegraded":"2024-03-06T07:29:28.194587+0000","last_fullsized":"2024-03-06T07:29:28.194587+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:04.309774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240873+0000","last_change":"2024-03-06T07:29:26.199005+0000","last_active":"2024-03-06T07:29:28.240873+0000","last_peered":"2024-03-06T07:29:28.240873+0000","last_clean":"2024-03-06T07:29:28.240873+0000","last_became_active":"2024-03-06T07:29:26.196517+0000","last_became_peered":"2024-03-06T07:29:26.196517+0000","last_unstale":"2024-03-06T07:29:28.240873+0000","last_undegraded":"2024-03-06T07:29:28.240873+0000","last_fullsized":"2024-03-06T07:29:28.240873+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:51:20.660770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.358862+0000","last_change":"2024-03-06T07:29:27.208624+0000","last_active":"2024-03-06T07:29:29.358862+0000","last_peered":"2024-03-06T07:29:29.358862+0000","last_clean":"2024-03-06T07:29:29.358862+0000","last_became_active":"2024-03-06T07:29:27.206474+0000","last_became_peered":"2024-03-06T07:29:27.206474+0000","last_unstale":"2024-03-06T07:29:29.358862+0000","last_undegraded":"2024-03-06T07:29:29.358862+0000","last_fullsized":"2024-03-06T07:29:29.358862+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:59.866176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.198040+0000","last_change":"2024-03-06T07:29:26.199717+0000","last_active":"2024-03-06T07:29:29.198040+0000","last_peered":"2024-03-06T07:29:29.198040+0000","last_clean":"2024-03-06T07:29:29.198040+0000","last_became_active":"2024-03-06T07:29:26.199436+0000","last_became_peered":"2024-03-06T07:29:26.199436+0000","last_unstale":"2024-03-06T07:29:29.198040+0000","last_undegraded":"2024-03-06T07:29:29.198040+0000","last_fullsized":"2024-03-06T07:29:29.198040+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:07.422880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359170+0000","last_change":"2024-03-06T07:29:27.222774+0000","last_active":"2024-03-06T07:29:29.359170+0000","last_peered":"2024-03-06T07:29:29.359170+0000","last_clean":"2024-03-06T07:29:29.359170+0000","last_became_active":"2024-03-06T07:29:27.222387+0000","last_became_peered":"2024-03-06T07:29:27.222387+0000","last_unstale":"2024-03-06T07:29:29.359170+0000","last_undegraded":"2024-03-06T07:29:29.359170+0000","last_fullsized":"2024-03-06T07:29:29.359170+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:11.269728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191748+0000","last_change":"2024-03-06T07:29:26.182768+0000","last_active":"2024-03-06T07:29:28.191748+0000","last_peered":"2024-03-06T07:29:28.191748+0000","last_clean":"2024-03-06T07:29:28.191748+0000","last_became_active":"2024-03-06T07:29:26.182489+0000","last_became_peered":"2024-03-06T07:29:26.182489+0000","last_unstale":"2024-03-06T07:29:28.191748+0000","last_undegraded":"2024-03-06T07:29:28.191748+0000","last_fullsized":"2024-03-06T07:29:28.191748+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:51.952315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191475+0000","last_change":"2024-03-06T07:29:27.221898+0000","last_active":"2024-03-06T07:29:28.191475+0000","last_peered":"2024-03-06T07:29:28.191475+0000","last_clean":"2024-03-06T07:29:28.191475+0000","last_became_active":"2024-03-06T07:29:27.221636+0000","last_became_peered":"2024-03-06T07:29:27.221636+0000","last_unstale":"2024-03-06T07:29:28.191475+0000","last_undegraded":"2024-03-06T07:29:28.191475+0000","last_fullsized":"2024-03-06T07:29:28.191475+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:16:29.749556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191323+0000","last_change":"2024-03-06T07:29:26.188395+0000","last_active":"2024-03-06T07:29:28.191323+0000","last_peered":"2024-03-06T07:29:28.191323+0000","last_clean":"2024-03-06T07:29:28.191323+0000","last_became_active":"2024-03-06T07:29:26.187669+0000","last_became_peered":"2024-03-06T07:29:26.187669+0000","last_unstale":"2024-03-06T07:29:28.191323+0000","last_undegraded":"2024-03-06T07:29:28.191323+0000","last_fullsized":"2024-03-06T07:29:28.191323+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:20:19.026209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.355952+0000","last_change":"2024-03-06T07:29:27.221492+0000","last_active":"2024-03-06T07:29:29.355952+0000","last_peered":"2024-03-06T07:29:29.355952+0000","last_clean":"2024-03-06T07:29:29.355952+0000","last_became_active":"2024-03-06T07:29:27.221203+0000","last_became_peered":"2024-03-06T07:29:27.221203+0000","last_unstale":"2024-03-06T07:29:29.355952+0000","last_undegraded":"2024-03-06T07:29:29.355952+0000","last_fullsized":"2024-03-06T07:29:29.355952+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:09.810216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197081+0000","last_change":"2024-03-06T07:29:26.200840+0000","last_active":"2024-03-06T07:29:29.197081+0000","last_peered":"2024-03-06T07:29:29.197081+0000","last_clean":"2024-03-06T07:29:29.197081+0000","last_became_active":"2024-03-06T07:29:26.200577+0000","last_became_peered":"2024-03-06T07:29:26.200577+0000","last_unstale":"2024-03-06T07:29:29.197081+0000","last_undegraded":"2024-03-06T07:29:29.197081+0000","last_fullsized":"2024-03-06T07:29:29.197081+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:08.181893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192995+0000","last_change":"2024-03-06T07:29:27.226841+0000","last_active":"2024-03-06T07:29:28.192995+0000","last_peered":"2024-03-06T07:29:28.192995+0000","last_clean":"2024-03-06T07:29:28.192995+0000","last_became_active":"2024-03-06T07:29:27.226120+0000","last_became_peered":"2024-03-06T07:29:27.226120+0000","last_unstale":"2024-03-06T07:29:28.192995+0000","last_undegraded":"2024-03-06T07:29:28.192995+0000","last_fullsized":"2024-03-06T07:29:28.192995+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:39.799373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193099+0000","last_change":"2024-03-06T07:29:26.191147+0000","last_active":"2024-03-06T07:29:28.193099+0000","last_peered":"2024-03-06T07:29:28.193099+0000","last_clean":"2024-03-06T07:29:28.193099+0000","last_became_active":"2024-03-06T07:29:26.190641+0000","last_became_peered":"2024-03-06T07:29:26.190641+0000","last_unstale":"2024-03-06T07:29:28.193099+0000","last_undegraded":"2024-03-06T07:29:28.193099+0000","last_fullsized":"2024-03-06T07:29:28.193099+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:09.406238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192348+0000","last_change":"2024-03-06T07:29:27.212821+0000","last_active":"2024-03-06T07:29:28.192348+0000","last_peered":"2024-03-06T07:29:28.192348+0000","last_clean":"2024-03-06T07:29:28.192348+0000","last_became_active":"2024-03-06T07:29:27.212249+0000","last_became_peered":"2024-03-06T07:29:27.212249+0000","last_unstale":"2024-03-06T07:29:28.192348+0000","last_undegraded":"2024-03-06T07:29:28.192348+0000","last_fullsized":"2024-03-06T07:29:28.192348+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:25.820523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.357267+0000","last_change":"2024-03-06T07:29:26.193713+0000","last_active":"2024-03-06T07:29:29.357267+0000","last_peered":"2024-03-06T07:29:29.357267+0000","last_clean":"2024-03-06T07:29:29.357267+0000","last_became_active":"2024-03-06T07:29:26.193389+0000","last_became_peered":"2024-03-06T07:29:26.193389+0000","last_unstale":"2024-03-06T07:29:29.357267+0000","last_undegraded":"2024-03-06T07:29:29.357267+0000","last_fullsized":"2024-03-06T07:29:29.357267+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:28.559373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360892+0000","last_change":"2024-03-06T07:29:27.218961+0000","last_active":"2024-03-06T07:29:29.360892+0000","last_peered":"2024-03-06T07:29:29.360892+0000","last_clean":"2024-03-06T07:29:29.360892+0000","last_became_active":"2024-03-06T07:29:27.218344+0000","last_became_peered":"2024-03-06T07:29:27.218344+0000","last_unstale":"2024-03-06T07:29:29.360892+0000","last_undegraded":"2024-03-06T07:29:29.360892+0000","last_fullsized":"2024-03-06T07:29:29.360892+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:07.883849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"46'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191843+0000","last_change":"2024-03-06T07:29:26.196747+0000","last_active":"2024-03-06T07:29:28.191843+0000","last_peered":"2024-03-06T07:29:28.191843+0000","last_clean":"2024-03-06T07:29:28.191843+0000","last_became_active":"2024-03-06T07:29:26.196446+0000","last_became_peered":"2024-03-06T07:29:26.196446+0000","last_unstale":"2024-03-06T07:29:28.191843+0000","last_undegraded":"2024-03-06T07:29:28.191843+0000","last_fullsized":"2024-03-06T07:29:28.191843+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:40.516483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.193814+0000","last_change":"2024-03-06T07:29:27.218206+0000","last_active":"2024-03-06T07:29:28.193814+0000","last_peered":"2024-03-06T07:29:28.193814+0000","last_clean":"2024-03-06T07:29:28.193814+0000","last_became_active":"2024-03-06T07:29:27.214524+0000","last_became_peered":"2024-03-06T07:29:27.214524+0000","last_unstale":"2024-03-06T07:29:28.193814+0000","last_undegraded":"2024-03-06T07:29:28.193814+0000","last_fullsized":"2024-03-06T07:29:28.193814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:25.630352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360705+0000","last_change":"2024-03-06T07:29:26.199027+0000","last_active":"2024-03-06T07:29:29.360705+0000","last_peered":"2024-03-06T07:29:29.360705+0000","last_clean":"2024-03-06T07:29:29.360705+0000","last_became_active":"2024-03-06T07:29:26.198748+0000","last_became_peered":"2024-03-06T07:29:26.198748+0000","last_unstale":"2024-03-06T07:29:29.360705+0000","last_undegraded":"2024-03-06T07:29:29.360705+0000","last_fullsized":"2024-03-06T07:29:29.360705+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:33.099509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.360514+0000","last_change":"2024-03-06T07:29:27.235780+0000","last_active":"2024-03-06T07:29:29.360514+0000","last_peered":"2024-03-06T07:29:29.360514+0000","last_clean":"2024-03-06T07:29:29.360514+0000","last_became_active":"2024-03-06T07:29:27.235481+0000","last_became_peered":"2024-03-06T07:29:27.235481+0000","last_unstale":"2024-03-06T07:29:29.360514+0000","last_undegraded":"2024-03-06T07:29:29.360514+0000","last_fullsized":"2024-03-06T07:29:29.360514+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:46:55.194453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.355929+0000","last_change":"2024-03-06T07:29:26.196655+0000","last_active":"2024-03-06T07:29:29.355929+0000","last_peered":"2024-03-06T07:29:29.355929+0000","last_clean":"2024-03-06T07:29:29.355929+0000","last_became_active":"2024-03-06T07:29:26.196370+0000","last_became_peered":"2024-03-06T07:29:26.196370+0000","last_unstale":"2024-03-06T07:29:29.355929+0000","last_undegraded":"2024-03-06T07:29:29.355929+0000","last_fullsized":"2024-03-06T07:29:29.355929+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:27:23.656718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191187+0000","last_change":"2024-03-06T07:29:27.216567+0000","last_active":"2024-03-06T07:29:28.191187+0000","last_peered":"2024-03-06T07:29:28.191187+0000","last_clean":"2024-03-06T07:29:28.191187+0000","last_became_active":"2024-03-06T07:29:27.216186+0000","last_became_peered":"2024-03-06T07:29:27.216186+0000","last_unstale":"2024-03-06T07:29:28.191187+0000","last_undegraded":"2024-03-06T07:29:28.191187+0000","last_fullsized":"2024-03-06T07:29:28.191187+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:22.862112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.359852+0000","last_change":"2024-03-06T07:29:26.195806+0000","last_active":"2024-03-06T07:29:29.359852+0000","last_peered":"2024-03-06T07:29:29.359852+0000","last_clean":"2024-03-06T07:29:29.359852+0000","last_became_active":"2024-03-06T07:29:26.195455+0000","last_became_peered":"2024-03-06T07:29:26.195455+0000","last_unstale":"2024-03-06T07:29:29.359852+0000","last_undegraded":"2024-03-06T07:29:29.359852+0000","last_fullsized":"2024-03-06T07:29:29.359852+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:36:53.192817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.191031+0000","last_change":"2024-03-06T07:29:27.222600+0000","last_active":"2024-03-06T07:29:28.191031+0000","last_peered":"2024-03-06T07:29:28.191031+0000","last_clean":"2024-03-06T07:29:28.191031+0000","last_became_active":"2024-03-06T07:29:27.222304+0000","last_became_peered":"2024-03-06T07:29:27.222304+0000","last_unstale":"2024-03-06T07:29:28.191031+0000","last_undegraded":"2024-03-06T07:29:28.191031+0000","last_fullsized":"2024-03-06T07:29:28.191031+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:49:07.910504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.240126+0000","last_change":"2024-03-06T07:29:26.199356+0000","last_active":"2024-03-06T07:29:28.240126+0000","last_peered":"2024-03-06T07:29:28.240126+0000","last_clean":"2024-03-06T07:29:28.240126+0000","last_became_active":"2024-03-06T07:29:26.199015+0000","last_became_peered":"2024-03-06T07:29:26.199015+0000","last_unstale":"2024-03-06T07:29:28.240126+0000","last_undegraded":"2024-03-06T07:29:28.240126+0000","last_fullsized":"2024-03-06T07:29:28.240126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:42.894461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.197144+0000","last_change":"2024-03-06T07:29:27.215173+0000","last_active":"2024-03-06T07:29:29.197144+0000","last_peered":"2024-03-06T07:29:29.197144+0000","last_clean":"2024-03-06T07:29:29.197144+0000","last_became_active":"2024-03-06T07:29:27.214896+0000","last_became_peered":"2024-03-06T07:29:27.214896+0000","last_unstale":"2024-03-06T07:29:29.197144+0000","last_undegraded":"2024-03-06T07:29:29.197144+0000","last_fullsized":"2024-03-06T07:29:29.197144+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:41.530505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.355963+0000","last_change":"2024-03-06T07:29:26.184910+0000","last_active":"2024-03-06T07:29:29.355963+0000","last_peered":"2024-03-06T07:29:29.355963+0000","last_clean":"2024-03-06T07:29:29.355963+0000","last_became_active":"2024-03-06T07:29:26.183588+0000","last_became_peered":"2024-03-06T07:29:26.183588+0000","last_unstale":"2024-03-06T07:29:29.355963+0000","last_undegraded":"2024-03-06T07:29:29.355963+0000","last_fullsized":"2024-03-06T07:29:29.355963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:57.101764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.239992+0000","last_change":"2024-03-06T07:29:27.234069+0000","last_active":"2024-03-06T07:29:28.239992+0000","last_peered":"2024-03-06T07:29:28.239992+0000","last_clean":"2024-03-06T07:29:28.239992+0000","last_became_active":"2024-03-06T07:29:27.233784+0000","last_became_peered":"2024-03-06T07:29:27.233784+0000","last_unstale":"2024-03-06T07:29:28.239992+0000","last_undegraded":"2024-03-06T07:29:28.239992+0000","last_fullsized":"2024-03-06T07:29:28.239992+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:18:54.594286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.192999+0000","last_change":"2024-03-06T07:29:26.185660+0000","last_active":"2024-03-06T07:29:28.192999+0000","last_peered":"2024-03-06T07:29:28.192999+0000","last_clean":"2024-03-06T07:29:28.192999+0000","last_became_active":"2024-03-06T07:29:26.185316+0000","last_became_peered":"2024-03-06T07:29:26.185316+0000","last_unstale":"2024-03-06T07:29:28.192999+0000","last_undegraded":"2024-03-06T07:29:28.192999+0000","last_fullsized":"2024-03-06T07:29:28.192999+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:17.328622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:28.194290+0000","last_change":"2024-03-06T07:29:27.220031+0000","last_active":"2024-03-06T07:29:28.194290+0000","last_peered":"2024-03-06T07:29:28.194290+0000","last_clean":"2024-03-06T07:29:28.194290+0000","last_became_active":"2024-03-06T07:29:27.218910+0000","last_became_peered":"2024-03-06T07:29:27.218910+0000","last_unstale":"2024-03-06T07:29:28.194290+0000","last_undegraded":"2024-03-06T07:29:28.194290+0000","last_fullsized":"2024-03-06T07:29:28.194290+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:26.150738+0000","last_clean_scrub_stamp":"2024-03-06T07:29:26.150738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:42.020983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.198094+0000","last_change":"2024-03-06T07:29:26.200006+0000","last_active":"2024-03-06T07:29:29.198094+0000","last_peered":"2024-03-06T07:29:29.198094+0000","last_clean":"2024-03-06T07:29:29.198094+0000","last_became_active":"2024-03-06T07:29:26.198345+0000","last_became_peered":"2024-03-06T07:29:26.198345+0000","last_unstale":"2024-03-06T07:29:29.198094+0000","last_undegraded":"2024-03-06T07:29:29.198094+0000","last_fullsized":"2024-03-06T07:29:29.198094+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:29:25.136715+0000","last_clean_scrub_stamp":"2024-03-06T07:29:25.136715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:06.869827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":213,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:29:29.357332+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:29:29.357332+0000","last_peered":"2024-03-06T07:29:29.357332+0000","last_clean":"2024-03-06T07:29:29.357332+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:29:29.357332+0000","last_undegraded":"2024-03-06T07:29:29.357332+0000","last_fullsized":"2024-03-06T07:29:29.357332+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509469,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":548864,"data_stored":218634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509470,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":536576,"data_stored":217070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542173,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36448,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335392,"statfs":{"total":96636764160,"available":96599441408,"internally_reserved":0,"allocated":1130496,"data_stored":807458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":540672,"data_stored":217672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542174,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":217070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542173,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":218612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542174,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":217638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":807402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:29:37.302 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:29:37.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:29:37.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.574+0000 7f9b06c00640 1 Processor -- start 2024-03-06T07:29:37.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.575+0000 7f9b06c00640 1 -- start start 2024-03-06T07:29:37.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.576+0000 7f9b06c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9b0015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.576+0000 7f9b06c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b000b3a10 con 0x7f9b0015e640 2024-03-06T07:29:37.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.576+0000 7f9b06c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b000b3cf0 con 0x7f9b0015ef60 2024-03-06T07:29:37.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.576+0000 7f9b06c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0015c070 con 0x7f9b0006e020 2024-03-06T07:29:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9affe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9b0015e640 0x7f9b000b2f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57596/0 (socket says 172.21.15.6:57596) 2024-03-06T07:29:37.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9affe00640 1 -- 172.21.15.6:0/299307996 learned_addr learned my addr 172.21.15.6:0/299307996 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:37.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9aff400640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9b0015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115627691 0 0) 0x7f9b000b3a10 con 0x7f9b0015e640 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae0003520 con 0x7f9b0015e640 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029709214 0 0) 0x7f9ae0003520 con 0x7f9b0015e640 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 msgr2=0x7f9b0015ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9b0015ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 >> v1:172.21.15.178:6789/0 conn(0x7f9b0015ef60 legacy=0x7f9b00167bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000b3fd0 con 0x7f9b0015e640 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9af0001670 con 0x7f9b0015e640 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.577+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 >> v1:172.21.15.6:6789/0 conn(0x7f9b0015e640 legacy=0x7f9b000b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9afea00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9ae00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9afea00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0003fb0 0x7f9ae0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9afea00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ae0004d40 0x7f9ae00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0015c070 con 0x7f9ae0004d40 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae000bb00 con 0x7f9ae0003fb0 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae000bd00 con 0x7f9b0006e020 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9b04c00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ae0004d40 0x7f9ae00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9aff400640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0003fb0 0x7f9ae0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.578+0000 7f9affe00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9ae00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.579+0000 7f9b04c00640 1 -- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 msgr2=0x7f9ae00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.579+0000 7f9b04c00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9ae00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.579+0000 7f9b04c00640 1 -- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0003fb0 msgr2=0x7f9ae0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.579+0000 7f9b04c00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0003fb0 0x7f9ae0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.579+0000 7f9b04c00640 1 -- 172.21.15.6:0/299307996 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae0003520 con 0x7f9ae0004d40 2024-03-06T07:29:37.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.579+0000 7f9affe00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b0006e020 0x7f9ae00039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:37.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9b04c00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ae0004d40 0x7f9ae00051c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9af8001e50 tx=0x7f9ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc40fe1dccd5ef07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8001a70 con 0x7f9ae0004d40 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af8003b70 con 0x7f9ae0004d40 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9afea00640 1 -- 172.21.15.6:0/299307996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af80050d0 con 0x7f9ae0004d40 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9b06c00640 1 -- 172.21.15.6:0/299307996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ae0004d40 msgr2=0x7f9ae00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9b06c00640 1 --2- 172.21.15.6:0/299307996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ae0004d40 0x7f9ae00051c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9af8001e50 tx=0x7f9ae0005830 comp rx=0 tx=0).stop 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.580+0000 7f9b04c00640 1 -- 172.21.15.6:0/299307996 reap_dead start 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.581+0000 7f9b06c00640 1 -- 172.21.15.6:0/299307996 shutdown_connections 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.581+0000 7f9b06c00640 1 -- 172.21.15.6:0/299307996 >> 172.21.15.6:0/299307996 conn(0x7f9b0006b3d0 msgr2=0x7f9b0006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.581+0000 7f9b06c00640 1 -- 172.21.15.6:0/299307996 shutdown_connections 2024-03-06T07:29:37.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.581+0000 7f9b06c00640 1 -- 172.21.15.6:0/299307996 wait complete. 2024-03-06T07:29:37.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.582+0000 7f9b06c00640 1 Processor -- start 2024-03-06T07:29:37.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.583+0000 7f9b06c00640 1 -- start start 2024-03-06T07:29:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.583+0000 7f9b06c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0004d40 0x7f9b000cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9b06c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 0x7f9b000ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9b06c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b000b0f70 0x7f9b00149110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9b06c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000b36a0 con 0x7f9b000af980 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9b06c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0015f3f0 con 0x7f9ae0004d40 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9b06c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000ad900 con 0x7f9b000b0f70 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 0x7f9b000ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 0x7f9b000ccf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43302/0 (socket says 172.21.15.6:43302) 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 -- 172.21.15.6:0/793253537 learned_addr learned my addr 172.21.15.6:0/793253537 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9affe00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0004d40 0x7f9b000cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9b04c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b000b0f70 0x7f9b00149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 -- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b000b0f70 msgr2=0x7f9b00149110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b000b0f70 0x7f9b00149110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 -- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0004d40 msgr2=0x7f9b000cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0004d40 0x7f9b000cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.584+0000 7f9aff400640 1 -- 172.21.15.6:0/793253537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000afed0 con 0x7f9b000af980 2024-03-06T07:29:37.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.585+0000 7f9aff400640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 0x7f9b000ccf00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8011940 tx=0x7f9ae8011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.585+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae800faf0 con 0x7f9b000af980 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.585+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae800fc90 con 0x7f9b000af980 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.585+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8014440 con 0x7f9b000af980 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.585+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b000b01b0 con 0x7f9b000af980 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.585+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b000b0770 con 0x7f9b000af980 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.586+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b00068010 con 0x7f9b000af980 2024-03-06T07:29:37.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.587+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ae8012040 con 0x7f9b000af980 2024-03-06T07:29:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.588+0000 7f9af7e00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:37.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.589+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ae808a8a0 con 0x7f9b000af980 2024-03-06T07:29:37.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.589+0000 7f9affe00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:37.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.590+0000 7f9affe00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9adc06b4d0 0x7f9adc06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9af0001bf0 tx=0x7f9af0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:37.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.593+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ae8019030 con 0x7f9b000af980 2024-03-06T07:29:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.864+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9b000b2600 con 0x7f9b000af980 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.869+0000 7f9af7e00640 1 -- 172.21.15.6:0/793253537 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9ae8059680 con 0x7f9b000af980 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:29:37.871 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:29:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9adc06b4d0 msgr2=0x7f9adc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9adc06b4d0 0x7f9adc06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9af0001bf0 tx=0x7f9af0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 msgr2=0x7f9b000ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 0x7f9b000ccf00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8011940 tx=0x7f9ae8011e00 comp rx=0 tx=0).stop 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 shutdown_connections 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9adc06b4d0 0x7f9adc06d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b000b0f70 0x7f9b00149110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b000af980 0x7f9b000ccf00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 --2- 172.21.15.6:0/793253537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ae0004d40 0x7f9b000cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 >> 172.21.15.6:0/793253537 conn(0x7f9b0006b3d0 msgr2=0x7f9b00071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 shutdown_connections 2024-03-06T07:29:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:37.872+0000 7f9b06c00640 1 -- 172.21.15.6:0/793253537 wait complete. 2024-03-06T07:29:37.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.114+0000 7fdbd6a00640 1 Processor -- start 2024-03-06T07:29:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.114+0000 7fdbd6a00640 1 -- start start 2024-03-06T07:29:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdbd015efc0 0x7fdbd00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd6a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd00cce90 0x7fdbd0161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd6a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd01627f0 con 0x7fdbd0162000 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd6a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd015f3a0 con 0x7fdbd015efc0 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd6a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd006d190 con 0x7fdbd00cce90 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdbd015efc0 0x7fdbd00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdbd015efc0 0x7fdbd00cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43308/0 (socket says 172.21.15.6:43308) 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 -- 172.21.15.6:0/939310550 learned_addr learned my addr 172.21.15.6:0/939310550 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd4c00640 1 --2- 172.21.15.6:0/939310550 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd00cce90 0x7fdbd0161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 -- 172.21.15.6:0/939310550 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd00cce90 msgr2=0x7fdbd0161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 --2- 172.21.15.6:0/939310550 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd00cce90 0x7fdbd0161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 -- 172.21.15.6:0/939310550 >> v1:172.21.15.178:6789/0 conn(0x7fdbd0162000 legacy=0x7fdbd00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.116+0000 7fdbd5600640 1 -- 172.21.15.6:0/939310550 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd006c860 con 0x7fdbd015efc0 2024-03-06T07:29:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.118+0000 7fdbd5600640 1 --2- 172.21.15.6:0/939310550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd015efc0 0x7fdbd00cc950 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009870 tx=0x7fdbc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52dab9a7e071551 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.118+0000 7fdbcfe00640 1 -- 172.21.15.6:0/939310550 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4035020 con 0x7fdbd015efc0 2024-03-06T07:29:38.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.118+0000 7fdbcfe00640 1 -- 172.21.15.6:0/939310550 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc4031640 con 0x7fdbd015efc0 2024-03-06T07:29:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.118+0000 7fdbcfe00640 1 -- 172.21.15.6:0/939310550 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc402d040 con 0x7fdbd015efc0 2024-03-06T07:29:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 -- 172.21.15.6:0/939310550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd015efc0 msgr2=0x7fdbd00cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/939310550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd015efc0 0x7fdbd00cc950 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009870 tx=0x7fdbc402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 -- 172.21.15.6:0/939310550 shutdown_connections 2024-03-06T07:29:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/939310550 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd00cce90 0x7fdbd0161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/939310550 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd015efc0 0x7fdbd00cc950 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 -- 172.21.15.6:0/939310550 >> 172.21.15.6:0/939310550 conn(0x7fdbd006b3d0 msgr2=0x7fdbd006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 -- 172.21.15.6:0/939310550 shutdown_connections 2024-03-06T07:29:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.119+0000 7fdbd6a00640 1 -- 172.21.15.6:0/939310550 wait complete. 2024-03-06T07:29:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.121+0000 7fdbd6a00640 1 Processor -- start 2024-03-06T07:29:38.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.121+0000 7fdbd6a00640 1 -- start start 2024-03-06T07:29:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.122+0000 7fdbd6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 0x7fdbd01e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.122+0000 7fdbd6a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbd015efc0 0x7fdbd01f7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd6a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd0162000 0x7fdbd015d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd0071c30 con 0x7fdbd00cce90 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd6a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd006ce20 con 0x7fdbd015efc0 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd6a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd0162490 con 0x7fdbd0162000 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 0x7fdbd01e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 0x7fdbd01e95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43318/0 (socket says 172.21.15.6:43318) 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 -- 172.21.15.6:0/3152361843 learned_addr learned my addr 172.21.15.6:0/3152361843 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd4c00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbd015efc0 0x7fdbd01f7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 -- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd0162000 msgr2=0x7fdbd015d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd6000640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd0162000 0x7fdbd015d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd0162000 0x7fdbd015d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 -- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbd015efc0 msgr2=0x7fdbd01f7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbd015efc0 0x7fdbd01f7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd5600640 1 -- 172.21.15.6:0/3152361843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd01f8290 con 0x7fdbd00cce90 2024-03-06T07:29:38.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd4c00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbd015efc0 0x7fdbd01f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.123+0000 7fdbd6000640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd0162000 0x7fdbd015d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.124+0000 7fdbd5600640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 0x7fdbd01e95a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402b930 tx=0x7fdbc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.124+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc403aa60 con 0x7fdbd00cce90 2024-03-06T07:29:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.124+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc403ac00 con 0x7fdbd00cce90 2024-03-06T07:29:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.124+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40443f0 con 0x7fdbd00cce90 2024-03-06T07:29:38.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.124+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd01e9be0 con 0x7fdbd00cce90 2024-03-06T07:29:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.125+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd015c590 con 0x7fdbd00cce90 2024-03-06T07:29:38.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.126+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd004f330 con 0x7fdbd00cce90 2024-03-06T07:29:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.127+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbc404b060 con 0x7fdbd00cce90 2024-03-06T07:29:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.128+0000 7fdbcd600640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbb406b4d0 0x7fdbb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.128+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbc40abfe0 con 0x7fdbd00cce90 2024-03-06T07:29:38.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.129+0000 7fdbd4c00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbb406b4d0 0x7fdbb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.130+0000 7fdbd4c00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbb406b4d0 0x7fdbb406d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00cbf30 tx=0x7fdbc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.133+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbd01e9be0 con 0x7fdbd00cce90 2024-03-06T07:29:38.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.372+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbd00b25b0 con 0x7fdbd00cce90 2024-03-06T07:29:38.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.373+0000 7fdbcd600640 1 -- 172.21.15.6:0/3152361843 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fdbd01f8290 con 0x7fdbd00cce90 2024-03-06T07:29:38.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:38.375 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:26.635978+0000","modified":"2024-03-06T07:29:29.733477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/590822743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":590822743},{"type":"v1","addr":"172.21.15.178:6837","nonce":590822743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5562,"qdb_cluster":[5562]},"id":3}]} 2024-03-06T07:29:38.376 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.375+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbb406b4d0 msgr2=0x7fdbb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.375+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbb406b4d0 0x7fdbb406d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00cbf30 tx=0x7fdbc0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.375+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 msgr2=0x7fdbd01e95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.375+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 0x7fdbd01e95a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402b930 tx=0x7fdbc40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 shutdown_connections 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdbb406b4d0 0x7fdbb406d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbd015efc0 0x7fdbd01f7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbd0162000 0x7fdbd015d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 --2- 172.21.15.6:0/3152361843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbd00cce90 0x7fdbd01e95a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 >> 172.21.15.6:0/3152361843 conn(0x7fdbd006b3d0 msgr2=0x7fdbd00b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 shutdown_connections 2024-03-06T07:29:38.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:38.376+0000 7fdbd6a00640 1 -- 172.21.15.6:0/3152361843 wait complete. 2024-03-06T07:29:38.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:29:26.635978+0000', 'modified': '2024-03-06T07:29:29.733477+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5562}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5562': {'gid': 5562, 'name': 'b', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.178:6837/590822743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6835', 'nonce': 590822743}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 590822743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5562, 'qdb_cluster': [5562]} 2024-03-06T07:29:38.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:29:38.797 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:29:39.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.024+0000 7f8683e00640 1 Processor -- start 2024-03-06T07:29:39.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.024+0000 7f8683e00640 1 -- start start 2024-03-06T07:29:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8683e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8684145300 0x7f86840b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8683e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f868415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8683e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86840714d0 con 0x7f8684070f10 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8683e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86840717b0 con 0x7f8684145300 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8683e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8684071a90 con 0x7f86840cd060 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8682a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8684145300 0x7f86840b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8682a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8684145300 0x7f86840b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43322/0 (socket says 172.21.15.6:43322) 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8682a00640 1 -- 172.21.15.6:0/212415654 learned_addr learned my addr 172.21.15.6:0/212415654 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8683400640 1 --2- 172.21.15.6:0/212415654 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f868415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.026+0000 7f8682a00640 1 -- 172.21.15.6:0/212415654 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 msgr2=0x7f868415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.027+0000 7f8682a00640 1 --2- 172.21.15.6:0/212415654 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f868415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.027+0000 7f8682a00640 1 -- 172.21.15.6:0/212415654 >> v1:172.21.15.178:6789/0 conn(0x7f8684070f10 legacy=0x7f86840cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.027+0000 7f8682a00640 1 -- 172.21.15.6:0/212415654 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684071d70 con 0x7f8684145300 2024-03-06T07:29:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.028+0000 7f8682a00640 1 --2- 172.21.15.6:0/212415654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684145300 0x7f86840b49a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8674009870 tx=0x7f867402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cb7c662d7c2a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.028+0000 7f8681600640 1 -- 172.21.15.6:0/212415654 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740317c0 con 0x7f8684145300 2024-03-06T07:29:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.028+0000 7f8681600640 1 -- 172.21.15.6:0/212415654 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8674037070 con 0x7f8684145300 2024-03-06T07:29:39.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.028+0000 7f8681600640 1 -- 172.21.15.6:0/212415654 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674031b80 con 0x7f8684145300 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.028+0000 7f8683e00640 1 -- 172.21.15.6:0/212415654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684145300 msgr2=0x7f86840b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.028+0000 7f8683e00640 1 --2- 172.21.15.6:0/212415654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684145300 0x7f86840b49a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8674009870 tx=0x7f867402b860 comp rx=0 tx=0).stop 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.029+0000 7f8683e00640 1 -- 172.21.15.6:0/212415654 shutdown_connections 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.029+0000 7f8683e00640 1 --2- 172.21.15.6:0/212415654 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f868415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.029+0000 7f8683e00640 1 --2- 172.21.15.6:0/212415654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684145300 0x7f86840b49a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.029+0000 7f8683e00640 1 -- 172.21.15.6:0/212415654 >> 172.21.15.6:0/212415654 conn(0x7f868406b3d0 msgr2=0x7f868406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.029+0000 7f8683e00640 1 -- 172.21.15.6:0/212415654 shutdown_connections 2024-03-06T07:29:39.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.029+0000 7f8683e00640 1 -- 172.21.15.6:0/212415654 wait complete. 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.030+0000 7f8683e00640 1 Processor -- start 2024-03-06T07:29:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.031+0000 7f8683e00640 1 -- start start 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.031+0000 7f8683e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 0x7f868413b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.031+0000 7f8683e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f86841418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8683e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8684145300 0x7f8684141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8683e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868415cab0 con 0x7f8684070f10 2024-03-06T07:29:39.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8683e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86841456e0 con 0x7f8684145300 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8683e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868415cc30 con 0x7f86840cd060 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 0x7f868413b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 0x7f868413b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43328/0 (socket says 172.21.15.6:43328) 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 -- 172.21.15.6:0/3988026408 learned_addr learned my addr 172.21.15.6:0/3988026408 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8683400640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8684145300 0x7f8684141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 -- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 msgr2=0x7f86841418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f86841418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 -- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8684145300 msgr2=0x7f8684141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8684145300 0x7f8684141e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 -- 172.21.15.6:0/3988026408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868413cac0 con 0x7f8684070f10 2024-03-06T07:29:39.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.032+0000 7f8682a00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 0x7f868413b460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8674009870 tx=0x7f86740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.033+0000 7f867aa00640 1 -- 172.21.15.6:0/3988026408 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674032860 con 0x7f8684070f10 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.033+0000 7f867aa00640 1 -- 172.21.15.6:0/3988026408 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8674032a00 con 0x7f8684070f10 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.033+0000 7f867aa00640 1 -- 172.21.15.6:0/3988026408 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674032dd0 con 0x7f8684070f10 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.033+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868413cda0 con 0x7f8684070f10 2024-03-06T07:29:39.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.033+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868413d680 con 0x7f8684070f10 2024-03-06T07:29:39.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.035+0000 7f867aa00640 1 -- 172.21.15.6:0/3988026408 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f867404c060 con 0x7f8684070f10 2024-03-06T07:29:39.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.036+0000 7f867aa00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f866406b4d0 0x7f866406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.036+0000 7f867aa00640 1 -- 172.21.15.6:0/3988026408 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86740abba0 con 0x7f8684070f10 2024-03-06T07:29:39.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.037+0000 7f868c397740 1 Processor -- start 2024-03-06T07:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.037+0000 7f868c397740 1 -- start start 2024-03-06T07:29:39.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.037+0000 7f8682000640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f866406b4d0 0x7f866406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6a9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6d70 0x5642be6d9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f868c397740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 0x5642be6d9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f868c397740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642be613600 con 0x5642be6a9e70 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642be5bc930 con 0x5642be6d6d70 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642be60f6a0 con 0x5642be6d6990 2024-03-06T07:29:39.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 0x5642be6d9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 0x5642be6d9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43340/0 (socket says 172.21.15.6:43340) 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 -- 172.21.15.6:0/3194580610 learned_addr learned my addr 172.21.15.6:0/3194580610 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8682000640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f866406b4d0 0x7f866406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8684070de0 tx=0x7f86700079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8682a00640 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8682000640 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6d70 0x5642be6d9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 -- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 msgr2=0x5642be6a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6a9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 -- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6d70 msgr2=0x5642be6d9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6d70 0x5642be6d9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8683400640 1 -- 172.21.15.6:0/3194580610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642be3282d0 con 0x5642be6a9e70 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.038+0000 7f8682000640 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6d70 0x5642be6d9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.039+0000 7f8683400640 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 0x5642be6d9c60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f867c008fb0 tx=0x7f867c00d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfdac8f66a3bb88 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.039+0000 7f866f000640 1 -- 172.21.15.6:0/3194580610 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00e8c0 con 0x5642be6a9e70 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.039+0000 7f866f000640 1 -- 172.21.15.6:0/3194580610 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00ea60 con 0x5642be6a9e70 2024-03-06T07:29:39.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.039+0000 7f866f000640 1 -- 172.21.15.6:0/3194580610 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c00ee10 con 0x5642be6a9e70 2024-03-06T07:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 -- 172.21.15.6:0/3194580610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 msgr2=0x5642be6d9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 0x5642be6d9c60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f867c008fb0 tx=0x7f867c00d810 comp rx=0 tx=0).stop 2024-03-06T07:29:39.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 -- 172.21.15.6:0/3194580610 shutdown_connections 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6d70 0x5642be6d9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6a9e70 0x5642be6d9c60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 --2- 172.21.15.6:0/3194580610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 -- 172.21.15.6:0/3194580610 >> 172.21.15.6:0/3194580610 conn(0x5642be6abba0 msgr2=0x5642be6f4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 -- 172.21.15.6:0/3194580610 shutdown_connections 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 -- 172.21.15.6:0/3194580610 wait complete. 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.040+0000 7f868c397740 1 -- start start 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.041+0000 7f868c397740 1 Processor -- start 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be704eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6dc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642be6ae140 con 0x5642be6d6d70 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642be5cfd90 con 0x5642be6e2ad0 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642be6ad530 con 0x5642be6d6990 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f868c397740 10 client.0 _ll_register_callbacks cb 0x5642be60b350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be704eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be704eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43342/0 (socket says 172.21.15.6:43342) 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 -- 172.21.15.6:0/63977663 learned_addr learned my addr 172.21.15.6:0/63977663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682a00640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8683400640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6dc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 -- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 msgr2=0x5642be6d5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 -- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 msgr2=0x5642be6dc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.042+0000 7f8682000640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6dc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f8682000640 1 -- 172.21.15.6:0/63977663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642be3f26b0 con 0x5642be6d6d70 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f8682000640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be704eb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8670014f00 tx=0x7f8670003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0d2d9424249711 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f8663400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f8663400640 1 -- 172.21.15.6:0/63977663 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670024060 con 0x5642be6d6d70 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f867c00ea60 con 0x5642be6d6d70 2024-03-06T07:29:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f8663400640 1 -- 172.21.15.6:0/63977663 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8670007230 con 0x5642be6d6d70 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f8663400640 1 -- 172.21.15.6:0/63977663 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f867002a050 con 0x5642be6d6d70 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f868c397740 10 client.5721 fetch_fsmap learned FSMap version 26 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.043+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x5642be6e22d0 con 0x5642be6d6d70 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f8663400640 1 -- 172.21.15.6:0/63977663 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c00ea60 con 0x5642be6d6d70 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f8663400640 1 -- 172.21.15.6:0/63977663 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f8670007430 con 0x5642be6d6d70 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f868c397740 10 client.5721 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f868c397740 10 client.5721 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f868c397740 20 client.5721 populate_metadata read hostname 'smithi006' 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f868c397740 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6dfa20 0x5642be6de610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f868c397740 4 client.5721 mds_command: new command op to 5562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.044+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 --> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642be620b40 con 0x5642be6dfa20 2024-03-06T07:29:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.045+0000 7f8682a00640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6dfa20 0x5642be6de610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.046+0000 7f8682a00640 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6dfa20 0x5642be6de610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7299ffc9eea33bd server_cookie=cb26910c7562b3fc in_seq=0 out_seq=0 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.046+0000 7f8663400640 10 client.5721 ms_handle_connect on v2:172.21.15.178:6835/590822743 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f8663400640 1 -- 172.21.15.6:0/63977663 <== mds.0 v2:172.21.15.178:6835/590822743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5642be620b40 con 0x5642be6dfa20 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f8663400640 10 client.5721 handle_command_reply: tid=0 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 client.5721 shutdown 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 msgr2=0x5642be704eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be704eb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8670014f00 tx=0x7f8670003760 comp rx=0 tx=0).stop 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 shutdown_connections 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6dfa20 0x5642be6de610 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6dc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be704eb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 --2- 172.21.15.6:0/63977663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f8663400640 0 client.5721 ms_handle_reset on v2:172.21.15.178:6835/590822743 2024-03-06T07:29:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.048+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 >> 172.21.15.6:0/63977663 conn(0x5642be6a9e70 msgr2=0x5642be6d9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.049+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 shutdown_connections 2024-03-06T07:29:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.049+0000 7f868c397740 1 -- 172.21.15.6:0/63977663 wait complete. 2024-03-06T07:29:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.049+0000 7f868c397740 20 client.5721 trim_cache size 0 max 16384 2024-03-06T07:29:39.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.068+0000 7f868c397740 1 Processor -- start 2024-03-06T07:29:39.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.068+0000 7f868c397740 1 -- start start 2024-03-06T07:29:39.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f868c397740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be6ad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f868c397740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868413dba0 con 0x5642be6d6d70 2024-03-06T07:29:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868406dd20 con 0x5642be6e2ad0 2024-03-06T07:29:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868406e320 con 0x5642be6d6990 2024-03-06T07:29:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.069+0000 7f8682000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be6ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be6ad670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43358/0 (socket says 172.21.15.6:43358) 2024-03-06T07:29:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 -- 172.21.15.6:0/3001289422 learned_addr learned my addr 172.21.15.6:0/3001289422 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 -- 172.21.15.6:0/3001289422 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 msgr2=0x5642be6d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 -- 172.21.15.6:0/3001289422 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 msgr2=0x5642be6f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 -- 172.21.15.6:0/3001289422 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642be6b0550 con 0x5642be6d6d70 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f8682000640 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be6ad670 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8670028040 tx=0x7f867001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1259c19f3f0943 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f866e600640 1 -- 172.21.15.6:0/3001289422 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867000c790 con 0x5642be6d6d70 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f866e600640 1 -- 172.21.15.6:0/3001289422 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867000c930 con 0x5642be6d6d70 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.070+0000 7f866e600640 1 -- 172.21.15.6:0/3001289422 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867000cd00 con 0x5642be6d6d70 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 -- 172.21.15.6:0/3001289422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 msgr2=0x5642be6ad670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be6ad670 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8670028040 tx=0x7f867001d000 comp rx=0 tx=0).stop 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 -- 172.21.15.6:0/3001289422 shutdown_connections 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6d6d70 0x5642be6ad670 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 --2- 172.21.15.6:0/3001289422 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6990 0x5642be6d4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 -- 172.21.15.6:0/3001289422 >> 172.21.15.6:0/3001289422 conn(0x5642be6a9e70 msgr2=0x5642be708090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 -- 172.21.15.6:0/3001289422 shutdown_connections 2024-03-06T07:29:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.071+0000 7f868c397740 1 -- 172.21.15.6:0/3001289422 wait complete. 2024-03-06T07:29:39.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.072+0000 7f868c397740 1 -- start start 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.072+0000 7f868c397740 1 Processor -- start 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6990 0x5642be6fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6d70 0x5642be758480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642be6df8a0 con 0x5642be6e2ad0 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642be6e25b0 con 0x5642be6d6990 2024-03-06T07:29:39.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642be6dfe40 con 0x5642be6d6d70 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f868c397740 10 client.0 _ll_register_callbacks cb 0x7f8670003950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f8683400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f8683400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43370/0 (socket says 172.21.15.6:43370) 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.073+0000 7f8683400640 1 -- 172.21.15.6:0/809118615 learned_addr learned my addr 172.21.15.6:0/809118615 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8683400640 1 -- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6d70 msgr2=0x5642be758480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8683400640 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6d70 0x5642be758480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8683400640 1 -- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6990 msgr2=0x5642be6fe8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8683400640 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6990 0x5642be6fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8683400640 1 -- 172.21.15.6:0/809118615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5642be7184b0 con 0x5642be6e2ad0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8683400640 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f867c008f80 tx=0x7f867c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc26f4ca928a15fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8663400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8663400640 1 -- 172.21.15.6:0/809118615 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c025070 con 0x5642be6e2ad0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f8663400640 1 -- 172.21.15.6:0/809118615 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f867c003da0 con 0x5642be6e2ad0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.074+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f867000c930 con 0x5642be6e2ad0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f8663400640 1 -- 172.21.15.6:0/809118615 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f867c030050 con 0x5642be6e2ad0 2024-03-06T07:29:39.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 10 client.5727 fetch_fsmap learned FSMap version 26 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x5642be717930 con 0x5642be6e2ad0 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f8663400640 1 -- 172.21.15.6:0/809118615 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867000c930 con 0x5642be6e2ad0 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f8663400640 1 -- 172.21.15.6:0/809118615 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 26) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f867c033020 con 0x5642be6e2ad0 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 10 client.5727 fetch_fsmap finished waiting for FSMap version 26 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 10 client.5727 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 20 client.5727 populate_metadata read hostname 'smithi006' 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6af890 0x5642be6de570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 4 client.5727 mds_command: new command op to 5562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:29:39.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.075+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 --> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5642be61ae90 con 0x5642be6af890 2024-03-06T07:29:39.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.076+0000 7f8682a00640 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6af890 0x5642be6de570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.076+0000 7f8682a00640 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6af890 0x5642be6de570 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39e2e75dd7fcb9d9 server_cookie=11de12774da4d106 in_seq=0 out_seq=0 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.076+0000 7f8663400640 10 client.5727 ms_handle_connect on v2:172.21.15.178:6835/590822743 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f8663400640 1 -- 172.21.15.6:0/809118615 <== mds.0 v2:172.21.15.178:6835/590822743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5642be61ae90 con 0x5642be6af890 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f8663400640 10 client.5727 handle_command_reply: tid=0 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 client.5727 shutdown 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 msgr2=0x5642be6f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f867c008f80 tx=0x7f867c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 shutdown_connections 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:6835/590822743,v1:172.21.15.178:6837/590822743] conn(0x5642be6af890 0x5642be6de570 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5642be6e2ad0 0x5642be6f6a10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5642be6d6d70 0x5642be758480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 --2- 172.21.15.6:0/809118615 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5642be6d6990 0x5642be6fe8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 >> 172.21.15.6:0/809118615 conn(0x5642be6a9e70 msgr2=0x5642be6a4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f8663400640 0 client.5727 ms_handle_reset on v2:172.21.15.178:6835/590822743 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.077+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 shutdown_connections 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f868c397740 1 -- 172.21.15.6:0/809118615 wait complete. 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f868c397740 20 client.5727 trim_cache size 0 max 16384 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5562, 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:29:39.082 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 11.864476882, 2024-03-06T07:29:39.083 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 26, 2024-03-06T07:29:39.083 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 46, 2024-03-06T07:29:39.083 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 46, 2024-03-06T07:29:39.083 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 17.268129117000001 2024-03-06T07:29:39.083 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f866406b4d0 msgr2=0x7f866406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f866406b4d0 0x7f866406d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8684070de0 tx=0x7f86700079b0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 msgr2=0x7f868413b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 0x7f868413b460 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8674009870 tx=0x7f86740046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 shutdown_connections 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f866406b4d0 0x7f866406d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8684145300 0x7f8684141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86840cd060 0x7f86841418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 --2- 172.21.15.6:0/3988026408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8684070f10 0x7f868413b460 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 >> 172.21.15.6:0/3988026408 conn(0x7f868406b3d0 msgr2=0x7f868415e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.078+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 shutdown_connections 2024-03-06T07:29:39.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.079+0000 7f8683e00640 1 -- 172.21.15.6:0/3988026408 wait complete. 2024-03-06T07:29:39.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:29:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.317+0000 7f930cc00640 1 Processor -- start 2024-03-06T07:29:39.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.317+0000 7f930cc00640 1 -- start start 2024-03-06T07:29:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f930cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f930815e0b0 0x7f93080cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f930cc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f93080d4a10 0x7f93080d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f930cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f93080d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f930cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93080b09e0 con 0x7f930815e0b0 2024-03-06T07:29:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f930cc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93080b3710 con 0x7f93080d4a10 2024-03-06T07:29:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f930cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93080b39f0 con 0x7f93080d0c80 2024-03-06T07:29:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.319+0000 7f9307400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f930815e0b0 0x7f93080cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f930815e0b0 0x7f93080cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43380/0 (socket says 172.21.15.6:43380) 2024-03-06T07:29:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 -- 172.21.15.6:0/3572047068 learned_addr learned my addr 172.21.15.6:0/3572047068 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9306a00640 1 --2- 172.21.15.6:0/3572047068 >> v2:172.21.15.178:3300/0 conn(0x7f93080d4a10 0x7f93080d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307e00640 1 --2- 172.21.15.6:0/3572047068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f93080d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 -- 172.21.15.6:0/3572047068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 msgr2=0x7f93080d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 --2- 172.21.15.6:0/3572047068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f93080d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 -- 172.21.15.6:0/3572047068 >> v2:172.21.15.178:3300/0 conn(0x7f93080d4a10 msgr2=0x7f93080d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 --2- 172.21.15.6:0/3572047068 >> v2:172.21.15.178:3300/0 conn(0x7f93080d4a10 0x7f93080d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.320+0000 7f9307400640 1 -- 172.21.15.6:0/3572047068 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93080b3cd0 con 0x7f930815e0b0 2024-03-06T07:29:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.322+0000 7f9307400640 1 --2- 172.21.15.6:0/3572047068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f930815e0b0 0x7f93080cf310 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f930806b8c0 tx=0x7f92f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7011b1fd739039 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.322+0000 7f9306000640 1 -- 172.21.15.6:0/3572047068 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f802ca70 con 0x7f930815e0b0 2024-03-06T07:29:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.322+0000 7f9306000640 1 -- 172.21.15.6:0/3572047068 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f802cc10 con 0x7f930815e0b0 2024-03-06T07:29:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.322+0000 7f9306000640 1 -- 172.21.15.6:0/3572047068 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f80343f0 con 0x7f930815e0b0 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 -- 172.21.15.6:0/3572047068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f930815e0b0 msgr2=0x7f93080cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 --2- 172.21.15.6:0/3572047068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f930815e0b0 0x7f93080cf310 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f930806b8c0 tx=0x7f92f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 -- 172.21.15.6:0/3572047068 shutdown_connections 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 --2- 172.21.15.6:0/3572047068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f93080d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 --2- 172.21.15.6:0/3572047068 >> v2:172.21.15.178:3300/0 conn(0x7f93080d4a10 0x7f93080d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 --2- 172.21.15.6:0/3572047068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f930815e0b0 0x7f93080cf310 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 -- 172.21.15.6:0/3572047068 >> 172.21.15.6:0/3572047068 conn(0x7f93080d3d30 msgr2=0x7f93080d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 -- 172.21.15.6:0/3572047068 shutdown_connections 2024-03-06T07:29:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.323+0000 7f930cc00640 1 -- 172.21.15.6:0/3572047068 wait complete. 2024-03-06T07:29:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.325+0000 7f930cc00640 1 Processor -- start 2024-03-06T07:29:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.325+0000 7f930cc00640 1 -- start start 2024-03-06T07:29:39.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.326+0000 7f930cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f930809daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.326+0000 7f930cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 0x7f930815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f930cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930815e0b0 0x7f93080b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f930cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93080cfc60 con 0x7f93080d4a10 2024-03-06T07:29:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f930cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930815e490 con 0x7f930815e0b0 2024-03-06T07:29:39.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f930cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93080cf850 con 0x7f93080d0c80 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 0x7f930815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 0x7f930815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43396/0 (socket says 172.21.15.6:43396) 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 -- 172.21.15.6:0/3055281921 learned_addr learned my addr 172.21.15.6:0/3055281921 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9307e00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930815e0b0 0x7f93080b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9307400640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f930809daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 -- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 msgr2=0x7f930809daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f930809daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 -- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930815e0b0 msgr2=0x7f93080b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.327+0000 7f9306a00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930815e0b0 0x7f93080b8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f9306a00640 1 -- 172.21.15.6:0/3055281921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93080b8330 con 0x7f93080d4a10 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f9307400640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f930809daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f9306a00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 0x7f930815c090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f92f000d800 tx=0x7f92f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000b660 con 0x7f93080d4a10 2024-03-06T07:29:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f0019070 con 0x7f93080d4a10 2024-03-06T07:29:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f000ba20 con 0x7f93080d4a10 2024-03-06T07:29:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93080b7a90 con 0x7f93080d4a10 2024-03-06T07:29:39.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.328+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93080b8050 con 0x7f93080d4a10 2024-03-06T07:29:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.330+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930804f330 con 0x7f93080d4a10 2024-03-06T07:29:39.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.331+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92f000bbc0 con 0x7f93080d4a10 2024-03-06T07:29:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.332+0000 7f92ff400640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92e406b4d0 0x7f92e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.332+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92f008a620 con 0x7f93080d4a10 2024-03-06T07:29:39.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.332+0000 7f9307400640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92e406b4d0 0x7f92e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.333+0000 7f9307400640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92e406b4d0 0x7f92e406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92f8035000 tx=0x7f92f8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.337+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92f0059400 con 0x7f93080d4a10 2024-03-06T07:29:39.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.582+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9308161640 con 0x7f93080d4a10 2024-03-06T07:29:39.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.583+0000 7f92ff400640 1 -- 172.21.15.6:0/3055281921 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f92f0058d00 con 0x7f93080d4a10 2024-03-06T07:29:39.585 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.585+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92e406b4d0 msgr2=0x7f92e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.585+0000 7f930cc00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92e406b4d0 0x7f92e406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92f8035000 tx=0x7f92f8030000 comp rx=0 tx=0).stop 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 msgr2=0x7f930815c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 0x7f930815c090 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f92f000d800 tx=0x7f92f000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 shutdown_connections 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92e406b4d0 0x7f92e406d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930815e0b0 0x7f93080b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93080d4a10 0x7f930815c090 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 --2- 172.21.15.6:0/3055281921 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93080d0c80 0x7f930809daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 >> 172.21.15.6:0/3055281921 conn(0x7f93080d3d30 msgr2=0x7f93080d2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 shutdown_connections 2024-03-06T07:29:39.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:39.586+0000 7f930cc00640 1 -- 172.21.15.6:0/3055281921 wait complete. 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:29:39.607 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:39.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:39.607 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.661 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42478sec preferred_lft 42478sec 2024-03-06T07:29:39.662 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:39.663 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:29:39.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:39.664 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:29:39.756 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:29:39.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:39.760 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:29:39.815 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:29:39.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:39.816 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:29:39.816 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:29:39.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:29:39.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:29:39.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:29:39.816 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:29:40.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:40.009 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:29:40.062 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:40.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:40.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:29:40.155 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:40.155 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:40.210 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:29:40.211 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:29:40.211 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:40.264 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:29:40.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:40.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:40.407 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:40.407 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:40.443 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:29:40.443 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:29:40.443 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:29:40.443 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:29:40.443 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:29:40.446 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:29:40.446 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:29:40.446 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:29:40.446 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:29:40.446 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:29:40.450 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:29:40.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:40.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:40.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:40.540 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:29:40.620 INFO:teuthology.orchestra.run.smithi006.stdout:client5736 2024-03-06T07:29:40.620 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:40.620 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:40.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:29:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.903+0000 7f715ba00640 1 Processor -- start 2024-03-06T07:29:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.904+0000 7f715ba00640 1 -- start start 2024-03-06T07:29:40.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.905+0000 7f715ba00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7154072520 0x7f71540cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.905+0000 7f715ba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154071e20 0x7f71540ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.905+0000 7f715ba00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7154072df0 con 0x7f715415bf20 2024-03-06T07:29:40.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.905+0000 7f715ba00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71540cdab0 con 0x7f7154072520 2024-03-06T07:29:40.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.905+0000 7f715ba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71540b4d70 con 0x7f7154071e20 2024-03-06T07:29:40.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.906+0000 7f7153e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7154072520 0x7f71540cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.906+0000 7f7153e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7154072520 0x7f71540cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:38170/0 (socket says 172.21.15.6:38170) 2024-03-06T07:29:40.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.906+0000 7f7153e00640 1 -- 172.21.15.6:0/2617762542 learned_addr learned my addr 172.21.15.6:0/2617762542 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:40.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.906+0000 7f7159a00640 1 --2- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154071e20 0x7f71540ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.907+0000 7f7153e00640 1 -- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154071e20 msgr2=0x7f71540ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:40.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.907+0000 7f7153e00640 1 --2- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154071e20 0x7f71540ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.907+0000 7f7153e00640 1 -- 172.21.15.6:0/2617762542 >> v1:172.21.15.178:6789/0 conn(0x7f715415bf20 legacy=0x7f71540acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.907+0000 7f7153e00640 1 -- 172.21.15.6:0/2617762542 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71540b5050 con 0x7f7154072520 2024-03-06T07:29:40.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.908+0000 7f7153e00640 1 --2- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7154072520 0x7f71540cdfb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7144009870 tx=0x7f714402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f273000eee6c1ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.909+0000 7f7153400640 1 -- 172.21.15.6:0/2617762542 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144034070 con 0x7f7154072520 2024-03-06T07:29:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.909+0000 7f7153400640 1 -- 172.21.15.6:0/2617762542 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7144030840 con 0x7f7154072520 2024-03-06T07:29:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.909+0000 7f7153400640 1 -- 172.21.15.6:0/2617762542 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144030c10 con 0x7f7154072520 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 -- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7154072520 msgr2=0x7f71540cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 --2- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7154072520 0x7f71540cdfb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7144009870 tx=0x7f714402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 -- 172.21.15.6:0/2617762542 shutdown_connections 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 --2- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154071e20 0x7f71540ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 --2- 172.21.15.6:0/2617762542 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7154072520 0x7f71540cdfb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 -- 172.21.15.6:0/2617762542 >> 172.21.15.6:0/2617762542 conn(0x7f715406b3d0 msgr2=0x7f715406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 -- 172.21.15.6:0/2617762542 shutdown_connections 2024-03-06T07:29:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.910+0000 7f715ba00640 1 -- 172.21.15.6:0/2617762542 wait complete. 2024-03-06T07:29:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.912+0000 7f715ba00640 1 Processor -- start 2024-03-06T07:29:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.912+0000 7f715ba00640 1 -- start start 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f715ba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 0x7f715406e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f715ba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154072520 0x7f71540cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f715ba00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f715415bf20 0x7f715415a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f715ba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71540ad450 con 0x7f7154071e20 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f715ba00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71540b4990 con 0x7f715415bf20 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f715ba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715406d630 con 0x7f7154072520 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f7159000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 0x7f715406e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f7159000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 0x7f715406e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43410/0 (socket says 172.21.15.6:43410) 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f7159000640 1 -- 172.21.15.6:0/1527254372 learned_addr learned my addr 172.21.15.6:0/1527254372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f7159a00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f715415bf20 0x7f715415a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.913+0000 7f7153e00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154072520 0x7f71540cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7159000640 1 -- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154072520 msgr2=0x7f71540cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7159000640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154072520 0x7f71540cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7159000640 1 -- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f715415bf20 msgr2=0x7f715415a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7159000640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f715415bf20 0x7f715415a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7159000640 1 -- 172.21.15.6:0/1527254372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71540cba80 con 0x7f7154071e20 2024-03-06T07:29:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7159000640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 0x7f715406e640 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f714800c400 tx=0x7f714800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480052d0 con 0x7f7154071e20 2024-03-06T07:29:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7148005490 con 0x7f7154071e20 2024-03-06T07:29:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148007260 con 0x7f7154071e20 2024-03-06T07:29:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.914+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715406ca20 con 0x7f7154071e20 2024-03-06T07:29:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.915+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715406cfe0 con 0x7f7154071e20 2024-03-06T07:29:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.916+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715404f330 con 0x7f7154071e20 2024-03-06T07:29:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.917+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7148007400 con 0x7f7154071e20 2024-03-06T07:29:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.918+0000 7f7150c00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f713806b500 0x7f713806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.918+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71480896f0 con 0x7f7154071e20 2024-03-06T07:29:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.918+0000 7f7153e00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f713806b500 0x7f713806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:40.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.919+0000 7f7153e00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f713806b500 0x7f713806d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7144036000 tx=0x7f714403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:40.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:40.922+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7148058560 con 0x7f7154071e20 2024-03-06T07:29:41.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.126+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f7154150b90 con 0x7f7154071e20 2024-03-06T07:29:41.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.133+0000 7f7150c00640 1 -- 172.21.15.6:0/1527254372 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7148057e60 con 0x7f7154071e20 2024-03-06T07:29:41.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f713806b500 msgr2=0x7f713806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f713806b500 0x7f713806d9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7144036000 tx=0x7f714403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 msgr2=0x7f715406e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 0x7f715406e640 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f714800c400 tx=0x7f714800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 shutdown_connections 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f713806b500 0x7f713806d9c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f715415bf20 0x7f715415a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7154072520 0x7f71540cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 --2- 172.21.15.6:0/1527254372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7154071e20 0x7f715406e640 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 >> 172.21.15.6:0/1527254372 conn(0x7f715406b3d0 msgr2=0x7f71540cdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 shutdown_connections 2024-03-06T07:29:41.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:41.136+0000 7f715ba00640 1 -- 172.21.15.6:0/1527254372 wait complete. 2024-03-06T07:29:51.157 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:51.157 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> n = 51 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-06T07:29:51.220 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:29:51.221 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-06T07:29:51.222 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:29:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:Traceback (most recent call last): 2024-03-06T07:29:51.454 INFO:teuthology.orchestra.run.smithi006.stderr: File "", line 33, in 2024-03-06T07:29:51.454 INFO:teuthology.orchestra.run.smithi006.stderr:OSError: [Errno 28] No space left on device 2024-03-06T07:29:51.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:51 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:29:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:51.474 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:29:51.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:51.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:29:51.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:51.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:51.747 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:29:51.747 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:51.747 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.799 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:51.800 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:51.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:52.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.111+0000 7f73c6000640 1 Processor -- start 2024-03-06T07:29:52.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.111+0000 7f73c6000640 1 -- start start 2024-03-06T07:29:52.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.113+0000 7f73c6000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f73c00af1a0 0x7f73c015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.113+0000 7f73c6000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af580 0x7f73c00b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.113+0000 7f73c6000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c015cc40 con 0x7f73c00cdc20 2024-03-06T07:29:52.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.113+0000 7f73c6000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c015cf20 con 0x7f73c00af1a0 2024-03-06T07:29:52.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.113+0000 7f73c6000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c015d200 con 0x7f73c00af580 2024-03-06T07:29:52.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f73c00af1a0 0x7f73c015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f73c00af1a0 0x7f73c015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:47270/0 (socket says 172.21.15.6:47270) 2024-03-06T07:29:52.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 -- 172.21.15.6:0/488834209 learned_addr learned my addr 172.21.15.6:0/488834209 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73b7e00640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af580 0x7f73c00b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 -- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af580 msgr2=0x7f73c00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af580 0x7f73c00b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 -- 172.21.15.6:0/488834209 >> v1:172.21.15.178:6789/0 conn(0x7f73c00cdc20 legacy=0x7f73c00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73bf400640 1 -- 172.21.15.6:0/488834209 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c015d4e0 con 0x7f73c00af1a0 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.114+0000 7f73b7e00640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af580 0x7f73c00b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.116+0000 7f73bf400640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af1a0 0x7f73c015c600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009870 tx=0x7f73b002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9005625e631c810 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.116+0000 7f73bea00640 1 -- 172.21.15.6:0/488834209 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b0035020 con 0x7f73c00af1a0 2024-03-06T07:29:52.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.116+0000 7f73bea00640 1 -- 172.21.15.6:0/488834209 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b0031640 con 0x7f73c00af1a0 2024-03-06T07:29:52.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.116+0000 7f73bea00640 1 -- 172.21.15.6:0/488834209 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b002d040 con 0x7f73c00af1a0 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 -- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af1a0 msgr2=0x7f73c015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af1a0 0x7f73c015c600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f73b0009870 tx=0x7f73b002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 -- 172.21.15.6:0/488834209 shutdown_connections 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af580 0x7f73c00b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 --2- 172.21.15.6:0/488834209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af1a0 0x7f73c015c600 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 -- 172.21.15.6:0/488834209 >> 172.21.15.6:0/488834209 conn(0x7f73c006afb0 msgr2=0x7f73c006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 -- 172.21.15.6:0/488834209 shutdown_connections 2024-03-06T07:29:52.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.117+0000 7f73c6000640 1 -- 172.21.15.6:0/488834209 wait complete. 2024-03-06T07:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.119+0000 7f73c6000640 1 Processor -- start 2024-03-06T07:29:52.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.119+0000 7f73c6000640 1 -- start start 2024-03-06T07:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73c6000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af1a0 0x7f73c009d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73c6000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af580 0x7f73c014c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73c6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 0x7f73c00c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73c6000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00b3660 con 0x7f73c00cdc20 2024-03-06T07:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73c6000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00b34e0 con 0x7f73c00af580 2024-03-06T07:29:52.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73c6000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c00b37e0 con 0x7f73c00af1a0 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 0x7f73c00c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73bf400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af1a0 0x7f73c009d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73bfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 0x7f73c00c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38278/0 (socket says 172.21.15.6:38278) 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73b7e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af580 0x7f73c014c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.120+0000 7f73bfe00640 1 -- 172.21.15.6:0/886749048 learned_addr learned my addr 172.21.15.6:0/886749048 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.121+0000 7f73bfe00640 1 -- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af1a0 msgr2=0x7f73c009d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.121+0000 7f73bfe00640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af1a0 0x7f73c009d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.121+0000 7f73bfe00640 1 -- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af580 msgr2=0x7f73c014c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.121+0000 7f73bfe00640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af580 0x7f73c014c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.121+0000 7f73bfe00640 1 -- 172.21.15.6:0/886749048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c009d650 con 0x7f73c00cdc20 2024-03-06T07:29:52.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.121+0000 7f73bfe00640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 0x7f73c00c9830 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c6b0 tx=0x7f73b800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.122+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8003b70 con 0x7f73c00cdc20 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.122+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8003d30 con 0x7f73c00cdc20 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.122+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c006ccb0 con 0x7f73c00cdc20 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.122+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8005310 con 0x7f73c00cdc20 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.122+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c006d270 con 0x7f73c00cdc20 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.123+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c004f3a0 con 0x7f73c00cdc20 2024-03-06T07:29:52.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.124+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73b80054b0 con 0x7f73c00cdc20 2024-03-06T07:29:52.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.125+0000 7f73b7400640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f739c06b4d0 0x7f739c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.125+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73b8089fe0 con 0x7f73c00cdc20 2024-03-06T07:29:52.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.125+0000 7f73bf400640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f739c06b4d0 0x7f739c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.126+0000 7f73bf400640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f739c06b4d0 0x7f739c06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f73b0036000 tx=0x7f73b003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.130+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b8058750 con 0x7f73c00cdc20 2024-03-06T07:29:52.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.332+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c00b30d0 con 0x7f73c00cdc20 2024-03-06T07:29:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.335+0000 7f73b7400640 1 -- 172.21.15.6:0/886749048 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f73b805c7f0 con 0x7f73c00cdc20 2024-03-06T07:29:52.337 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:52.338 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:28.183720+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f739c06b4d0 msgr2=0x7f739c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f739c06b4d0 0x7f739c06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f73b0036000 tx=0x7f73b003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 msgr2=0x7f73c00c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 0x7f73c00c9830 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c6b0 tx=0x7f73b800cb70 comp rx=0 tx=0).stop 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 shutdown_connections 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f739c06b4d0 0x7f739c06d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73c00af580 0x7f73c014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73c00cdc20 0x7f73c00c9830 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 --2- 172.21.15.6:0/886749048 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73c00af1a0 0x7f73c009d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 >> 172.21.15.6:0/886749048 conn(0x7f73c006afb0 msgr2=0x7f73c0071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 shutdown_connections 2024-03-06T07:29:52.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.338+0000 7f73c6000640 1 -- 172.21.15.6:0/886749048 wait complete. 2024-03-06T07:29:52.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:24.525508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:25.659373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:52.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:52.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.582+0000 7fb273200640 1 Processor -- start 2024-03-06T07:29:52.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.582+0000 7fb273200640 1 -- start start 2024-03-06T07:29:52.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb273200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb274165d50 0x7fb2740b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb273200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2740b1b40 0x7fb2740b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb273200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb274166cb0 con 0x7fb274162280 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb273200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2740b4fd0 con 0x7fb274165d50 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb273200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2741661b0 con 0x7fb2740b1b40 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb271400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb274165d50 0x7fb2740b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb271e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb274162280 0x7fb2740cc430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51754/0 (socket says 172.21.15.6:51754) 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.584+0000 7fb271e00640 1 -- 172.21.15.6:0/1316397622 learned_addr learned my addr 172.21.15.6:0/1316397622 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb272800640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2740b1b40 0x7fb2740b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb271400640 1 -- 172.21.15.6:0/1316397622 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2740b1b40 msgr2=0x7fb2740b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb271400640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2740b1b40 0x7fb2740b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb271400640 1 -- 172.21.15.6:0/1316397622 >> v1:172.21.15.6:6789/0 conn(0x7fb274162280 legacy=0x7fb2740cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb271400640 1 -- 172.21.15.6:0/1316397622 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274166410 con 0x7fb274165d50 2024-03-06T07:29:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb272800640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2740b1b40 0x7fb2740b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.585+0000 7fb270a00640 1 -- 172.21.15.6:0/1316397622 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454552371 0 0) 0x7fb274166cb0 con 0x7fb274162280 2024-03-06T07:29:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.586+0000 7fb271400640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274165d50 0x7fb2740b4a90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb2600097d0 tx=0x7fb26002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17fd55819226649 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb270a00640 1 -- 172.21.15.6:0/1316397622 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260031780 con 0x7fb274165d50 2024-03-06T07:29:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb270a00640 1 -- 172.21.15.6:0/1316397622 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb260037070 con 0x7fb274165d50 2024-03-06T07:29:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb270a00640 1 -- 172.21.15.6:0/1316397622 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260031b40 con 0x7fb274165d50 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 -- 172.21.15.6:0/1316397622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274165d50 msgr2=0x7fb2740b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274165d50 0x7fb2740b4a90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb2600097d0 tx=0x7fb26002b820 comp rx=0 tx=0).stop 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 -- 172.21.15.6:0/1316397622 shutdown_connections 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2740b1b40 0x7fb2740b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 --2- 172.21.15.6:0/1316397622 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb274165d50 0x7fb2740b4a90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 -- 172.21.15.6:0/1316397622 >> 172.21.15.6:0/1316397622 conn(0x7fb27406b3a0 msgr2=0x7fb27406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 -- 172.21.15.6:0/1316397622 shutdown_connections 2024-03-06T07:29:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.587+0000 7fb273200640 1 -- 172.21.15.6:0/1316397622 wait complete. 2024-03-06T07:29:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.589+0000 7fb273200640 1 Processor -- start 2024-03-06T07:29:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.589+0000 7fb273200640 1 -- start start 2024-03-06T07:29:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.590+0000 7fb273200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 0x7fb2741607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.590+0000 7fb273200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb274162280 0x7fb2741635c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.590+0000 7fb273200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb274165d50 0x7fb274163b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.590+0000 7fb273200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb274071560 con 0x7fb2740b1b40 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.590+0000 7fb273200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2741669d0 con 0x7fb274162280 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.590+0000 7fb273200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740716e0 con 0x7fb274165d50 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 0x7fb2741607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 0x7fb2741607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38292/0 (socket says 172.21.15.6:38292) 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 -- 172.21.15.6:0/1318520311 learned_addr learned my addr 172.21.15.6:0/1318520311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271400640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb274162280 0x7fb2741635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb272800640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb274165d50 0x7fb274163b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 -- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb274165d50 msgr2=0x7fb274163b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb274165d50 0x7fb274163b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 -- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb274162280 msgr2=0x7fb2741635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb274162280 0x7fb2741635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 -- 172.21.15.6:0/1318520311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27414c7b0 con 0x7fb2740b1b40 2024-03-06T07:29:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.591+0000 7fb271e00640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 0x7fb2741607c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb26400c4d0 tx=0x7fb26400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.592+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264018070 con 0x7fb2740b1b40 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.592+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb264007500 con 0x7fb2740b1b40 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.592+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264004a20 con 0x7fb2740b1b40 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.592+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27415e0d0 con 0x7fb2740b1b40 2024-03-06T07:29:52.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.592+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27415e690 con 0x7fb2740b1b40 2024-03-06T07:29:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.593+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27404f330 con 0x7fb2740b1b40 2024-03-06T07:29:52.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.594+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb264004bc0 con 0x7fb2740b1b40 2024-03-06T07:29:52.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.595+0000 7fb26a000640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb25406b4d0 0x7fb25406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.595+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb264089540 con 0x7fb2740b1b40 2024-03-06T07:29:52.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.596+0000 7fb271400640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb25406b4d0 0x7fb25406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.597+0000 7fb271400640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb25406b4d0 0x7fb25406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb260036000 tx=0x7fb26003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.601+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb264058320 con 0x7fb2740b1b40 2024-03-06T07:29:52.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.799+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb27406e490 con 0x7fb2740b1b40 2024-03-06T07:29:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.800+0000 7fb26a000640 1 -- 172.21.15.6:0/1318520311 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb264057c20 con 0x7fb2740b1b40 2024-03-06T07:29:52.802 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:29:52.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.802+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb25406b4d0 msgr2=0x7fb25406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb25406b4d0 0x7fb25406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb260036000 tx=0x7fb26003d040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 msgr2=0x7fb2741607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 0x7fb2741607c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb26400c4d0 tx=0x7fb26400c990 comp rx=0 tx=0).stop 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 shutdown_connections 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb25406b4d0 0x7fb25406d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb274162280 0x7fb2741635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb274165d50 0x7fb274163b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 --2- 172.21.15.6:0/1318520311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2740b1b40 0x7fb2741607c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 >> 172.21.15.6:0/1318520311 conn(0x7fb27406b3a0 msgr2=0x7fb27415c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 shutdown_connections 2024-03-06T07:29:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:52.803+0000 7fb273200640 1 -- 172.21.15.6:0/1318520311 wait complete. 2024-03-06T07:29:52.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:53.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.051+0000 7f6451a00640 1 Processor -- start 2024-03-06T07:29:53.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.051+0000 7f6451a00640 1 -- start start 2024-03-06T07:29:53.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.053+0000 7f6451a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f644c162280 0x7f644c165d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.053+0000 7f6451a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f644c0b4a90 0x7f644c0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.053+0000 7f6451a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c166cb0 con 0x7f644c0b1b40 2024-03-06T07:29:53.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.053+0000 7f6451a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c0cc970 con 0x7f644c0b4a90 2024-03-06T07:29:53.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.053+0000 7f6451a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c166410 con 0x7f644c162280 2024-03-06T07:29:53.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6443400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f644c0b4a90 0x7f644c0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6451000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f644c0b1b40 0x7f644c071560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51764/0 (socket says 172.21.15.6:51764) 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6451000640 1 -- 172.21.15.6:0/1502214080 learned_addr learned my addr 172.21.15.6:0/1502214080 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f644be00640 1 --2- 172.21.15.6:0/1502214080 >> v2:172.21.15.178:3300/0 conn(0x7f644c162280 0x7f644c165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f644b400640 1 -- 172.21.15.6:0/1502214080 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170214667 0 0) 0x7f644c166cb0 con 0x7f644c0b1b40 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f644b400640 1 -- 172.21.15.6:0/1502214080 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6430003520 con 0x7f644c0b1b40 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6443400640 1 -- 172.21.15.6:0/1502214080 >> v2:172.21.15.178:3300/0 conn(0x7f644c162280 msgr2=0x7f644c165d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6443400640 1 --2- 172.21.15.6:0/1502214080 >> v2:172.21.15.178:3300/0 conn(0x7f644c162280 0x7f644c165d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6443400640 1 -- 172.21.15.6:0/1502214080 >> v1:172.21.15.6:6789/0 conn(0x7f644c0b1b40 legacy=0x7f644c071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.054+0000 7f6443400640 1 -- 172.21.15.6:0/1502214080 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c1666f0 con 0x7f644c0b4a90 2024-03-06T07:29:53.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.055+0000 7f6443400640 1 --2- 172.21.15.6:0/1502214080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c0cc430 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f644c0b2790 tx=0x7f643402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713fdda0eae12ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.056+0000 7f644b400640 1 -- 172.21.15.6:0/1502214080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64340315a0 con 0x7f644c0b4a90 2024-03-06T07:29:53.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.056+0000 7f644b400640 1 -- 172.21.15.6:0/1502214080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6434037070 con 0x7f644c0b4a90 2024-03-06T07:29:53.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.056+0000 7f644b400640 1 -- 172.21.15.6:0/1502214080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434031940 con 0x7f644c0b4a90 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.056+0000 7f6451a00640 1 -- 172.21.15.6:0/1502214080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 msgr2=0x7f644c0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.056+0000 7f6451a00640 1 --2- 172.21.15.6:0/1502214080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c0cc430 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f644c0b2790 tx=0x7f643402b840 comp rx=0 tx=0).stop 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.057+0000 7f6451a00640 1 -- 172.21.15.6:0/1502214080 shutdown_connections 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.057+0000 7f6451a00640 1 --2- 172.21.15.6:0/1502214080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c0cc430 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.057+0000 7f6451a00640 1 --2- 172.21.15.6:0/1502214080 >> v2:172.21.15.178:3300/0 conn(0x7f644c162280 0x7f644c165d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.057+0000 7f6451a00640 1 -- 172.21.15.6:0/1502214080 >> 172.21.15.6:0/1502214080 conn(0x7f644c06b3a0 msgr2=0x7f644c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.057+0000 7f6451a00640 1 -- 172.21.15.6:0/1502214080 shutdown_connections 2024-03-06T07:29:53.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.057+0000 7f6451a00640 1 -- 172.21.15.6:0/1502214080 wait complete. 2024-03-06T07:29:53.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.058+0000 7f6451a00640 1 Processor -- start 2024-03-06T07:29:53.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 -- start start 2024-03-06T07:29:53.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f644c0b1b40 0x7f644c1650c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c14fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f644c162280 0x7f644c15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c071c70 con 0x7f644c0b4a90 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0b4ef0 con 0x7f644c0b1b40 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.059+0000 7f6451a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c071df0 con 0x7f644c162280 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c14fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c14fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38316/0 (socket says 172.21.15.6:38316) 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 -- 172.21.15.6:0/1411969326 learned_addr learned my addr 172.21.15.6:0/1411969326 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f644be00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f644c0b1b40 0x7f644c1650c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6451000640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f644c162280 0x7f644c15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 -- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f644c162280 msgr2=0x7f644c15b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f644c162280 0x7f644c15b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 -- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f644c0b1b40 msgr2=0x7f644c1650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f644c0b1b40 0x7f644c1650c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.060+0000 7f6443400640 1 -- 172.21.15.6:0/1411969326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c165600 con 0x7f644c0b4a90 2024-03-06T07:29:53.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.061+0000 7f6443400640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c14fe80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f643402b880 tx=0x7f64340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.061+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434032630 con 0x7f644c0b4a90 2024-03-06T07:29:53.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.061+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64340327d0 con 0x7f644c0b4a90 2024-03-06T07:29:53.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.061+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6434032ba0 con 0x7f644c0b4a90 2024-03-06T07:29:53.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.061+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c15dfc0 con 0x7f644c0b4a90 2024-03-06T07:29:53.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.061+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c163f50 con 0x7f644c0b4a90 2024-03-06T07:29:53.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.062+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c0b13f0 con 0x7f644c0b4a90 2024-03-06T07:29:53.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.063+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f643404e430 con 0x7f644c0b4a90 2024-03-06T07:29:53.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.064+0000 7f6448c00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.064+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f643402f030 con 0x7f644c0b4a90 2024-03-06T07:29:53.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.065+0000 7f644be00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.066+0000 7f644be00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f642806b4d0 0x7f642806d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f643c009a30 tx=0x7f643c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.068+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f644c0b13f0 con 0x7f644c0b4a90 2024-03-06T07:29:53.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.268+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f644c161dc0 con 0x7f644c0b4a90 2024-03-06T07:29:53.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.269+0000 7f6448c00640 1 -- 172.21.15.6:0/1411969326 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6434040070 con 0x7f644c0b4a90 2024-03-06T07:29:53.272 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:29:53.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f642806b4d0 msgr2=0x7f642806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f642806b4d0 0x7f642806d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f643c009a30 tx=0x7f643c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 msgr2=0x7f644c14fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c14fe80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f643402b880 tx=0x7f64340046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 shutdown_connections 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f642806b4d0 0x7f642806d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f644c162280 0x7f644c15b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c0b4a90 0x7f644c14fe80 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 --2- 172.21.15.6:0/1411969326 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f644c0b1b40 0x7f644c1650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 >> 172.21.15.6:0/1411969326 conn(0x7f644c06b3a0 msgr2=0x7f644c06b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 shutdown_connections 2024-03-06T07:29:53.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.272+0000 7f6451a00640 1 -- 172.21.15.6:0/1411969326 wait complete. 2024-03-06T07:29:53.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.504+0000 7f75d3000640 1 Processor -- start 2024-03-06T07:29:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.504+0000 7f75d3000640 1 -- start start 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d3000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75cc06d8d0 0x7f75cc0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d3000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc06de40 con 0x7f75cc147590 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d3000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc06e120 con 0x7f75cc0b2080 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d3000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75cc147970 con 0x7f75cc06d8d0 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d1200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75cc06d8d0 0x7f75cc0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d1c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f75cc147590 0x7f75cc071480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51770/0 (socket says 172.21.15.6:51770) 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d1c00640 1 -- 172.21.15.6:0/722655966 learned_addr learned my addr 172.21.15.6:0/722655966 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.506+0000 7f75d1200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f75cc06d8d0 0x7f75cc0b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38318/0 (socket says 172.21.15.6:38318) 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.507+0000 7f75d1200640 1 -- 172.21.15.6:0/722655966 >> v1:172.21.15.178:6789/0 conn(0x7f75cc0b2080 legacy=0x7f75cc0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.507+0000 7f75d1200640 1 -- 172.21.15.6:0/722655966 >> v1:172.21.15.6:6789/0 conn(0x7f75cc147590 legacy=0x7f75cc071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.507+0000 7f75d1200640 1 -- 172.21.15.6:0/722655966 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0b2e40 con 0x7f75cc06d8d0 2024-03-06T07:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.508+0000 7f75d1200640 1 --2- 172.21.15.6:0/722655966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc06d8d0 0x7f75cc0b1b40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f75cc147af0 tx=0x7f75b802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b37853f4751eaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.508+0000 7f75cbe00640 1 -- 172.21.15.6:0/722655966 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8035020 con 0x7f75cc06d8d0 2024-03-06T07:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.508+0000 7f75cbe00640 1 -- 172.21.15.6:0/722655966 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8031810 con 0x7f75cc06d8d0 2024-03-06T07:29:53.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.508+0000 7f75cbe00640 1 -- 172.21.15.6:0/722655966 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8031be0 con 0x7f75cc06d8d0 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 -- 172.21.15.6:0/722655966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc06d8d0 msgr2=0x7f75cc0b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 --2- 172.21.15.6:0/722655966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc06d8d0 0x7f75cc0b1b40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f75cc147af0 tx=0x7f75b802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 -- 172.21.15.6:0/722655966 shutdown_connections 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 --2- 172.21.15.6:0/722655966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc06d8d0 0x7f75cc0b1b40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 -- 172.21.15.6:0/722655966 >> 172.21.15.6:0/722655966 conn(0x7f75cc06b3a0 msgr2=0x7f75cc06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 -- 172.21.15.6:0/722655966 shutdown_connections 2024-03-06T07:29:53.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.509+0000 7f75d3000640 1 -- 172.21.15.6:0/722655966 wait complete. 2024-03-06T07:29:53.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.511+0000 7f75d3000640 1 Processor -- start 2024-03-06T07:29:53.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.511+0000 7f75d3000640 1 -- start start 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d3000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75cc06d8d0 0x7f75cc15d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d3000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75cc0b2080 0x7f75cc0c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d3000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 0x7f75cc0c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d3000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b24b0 con 0x7f75cc147590 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d3000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b31b0 con 0x7f75cc0b2080 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d3000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0b3d50 con 0x7f75cc06d8d0 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 0x7f75cc0c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 0x7f75cc0c6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38326/0 (socket says 172.21.15.6:38326) 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.512+0000 7f75d2600640 1 -- 172.21.15.6:0/588069104 learned_addr learned my addr 172.21.15.6:0/588069104 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d1c00640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75cc06d8d0 0x7f75cc15d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d1200640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75cc0b2080 0x7f75cc0c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d2600640 1 -- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75cc06d8d0 msgr2=0x7f75cc15d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d2600640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75cc06d8d0 0x7f75cc15d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d2600640 1 -- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75cc0b2080 msgr2=0x7f75cc0c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d2600640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75cc0b2080 0x7f75cc0c6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d2600640 1 -- 172.21.15.6:0/588069104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc141ab0 con 0x7f75cc147590 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d2600640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 0x7f75cc0c6790 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c780 tx=0x7f75c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4004b60 con 0x7f75cc147590 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c4004d00 con 0x7f75cc147590 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c400aa90 con 0x7f75cc147590 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.513+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc141d90 con 0x7f75cc147590 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.514+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc142350 con 0x7f75cc147590 2024-03-06T07:29:53.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.515+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc04f330 con 0x7f75cc147590 2024-03-06T07:29:53.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.516+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75c400ac30 con 0x7f75cc147590 2024-03-06T07:29:53.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.517+0000 7f75c9600640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ac06b520 0x7f75ac06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.517+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75c408a510 con 0x7f75cc147590 2024-03-06T07:29:53.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.517+0000 7f75d1c00640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ac06b520 0x7f75ac06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.519+0000 7f75d1c00640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ac06b520 0x7f75ac06d9e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0b39d0 tx=0x7f75c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.521+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75c4020070 con 0x7f75cc147590 2024-03-06T07:29:53.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.724+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75cc146d10 con 0x7f75cc147590 2024-03-06T07:29:53.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.725+0000 7f75c9600640 1 -- 172.21.15.6:0/588069104 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f75c4059370 con 0x7f75cc147590 2024-03-06T07:29:53.727 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.727+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ac06b520 msgr2=0x7f75ac06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.727+0000 7f75d3000640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ac06b520 0x7f75ac06d9e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0b39d0 tx=0x7f75c0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 msgr2=0x7f75cc0c6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 0x7f75cc0c6790 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c780 tx=0x7f75c400cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 shutdown_connections 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ac06b520 0x7f75ac06d9e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75cc0b2080 0x7f75cc0c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75cc147590 0x7f75cc0c6790 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 --2- 172.21.15.6:0/588069104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75cc06d8d0 0x7f75cc15d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 >> 172.21.15.6:0/588069104 conn(0x7f75cc06b3a0 msgr2=0x7f75cc06b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 shutdown_connections 2024-03-06T07:29:53.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.728+0000 7f75d3000640 1 -- 172.21.15.6:0/588069104 wait complete. 2024-03-06T07:29:53.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:29:53.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.971+0000 7f9169e00640 1 Processor -- start 2024-03-06T07:29:53.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.972+0000 7f9169e00640 1 -- start start 2024-03-06T07:29:53.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.973+0000 7f9169e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f91640d4a10 0x7f91640d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.973+0000 7f9169e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91640b3710 con 0x7f91640d0c80 2024-03-06T07:29:53.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.973+0000 7f9169e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91640b39f0 con 0x7f916415e0b0 2024-03-06T07:29:53.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.973+0000 7f9169e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91640d10f0 con 0x7f91640d4a10 2024-03-06T07:29:53.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.973+0000 7f9169400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f91640d0c80 0x7f91640b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51784/0 (socket says 172.21.15.6:51784) 2024-03-06T07:29:53.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.973+0000 7f9169400640 1 -- 172.21.15.6:0/2245804463 learned_addr learned my addr 172.21.15.6:0/2245804463 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:53.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163e00640 1 --2- 172.21.15.6:0/2245804463 >> v2:172.21.15.178:3300/0 conn(0x7f91640d4a10 0x7f91640d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932182413 0 0) 0x7f91640b3710 con 0x7f91640d0c80 2024-03-06T07:29:53.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f914c003520 con 0x7f91640d0c80 2024-03-06T07:29:53.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2693018856 0 0) 0x7f914c003520 con 0x7f91640d0c80 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 >> v2:172.21.15.178:3300/0 conn(0x7f91640d4a10 msgr2=0x7f91640d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 --2- 172.21.15.6:0/2245804463 >> v2:172.21.15.178:3300/0 conn(0x7f91640d4a10 0x7f91640d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 >> v1:172.21.15.178:6789/0 conn(0x7f916415e0b0 legacy=0x7f91640cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640b3cd0 con 0x7f91640d0c80 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f915c001a10 con 0x7f91640d0c80 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 >> v1:172.21.15.6:6789/0 conn(0x7f91640d0c80 legacy=0x7f91640b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f914c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f914c004800 0x7f914c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f914c0051c0 0x7f914c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640d10f0 con 0x7f914c0039b0 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c00c060 con 0x7f914c004800 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.974+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c00c260 con 0x7f914c0051c0 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9168a00640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f914c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9169400640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f914c0051c0 0x7f914c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9168a00640 1 -- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f914c0051c0 msgr2=0x7f914c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9168a00640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f914c0051c0 0x7f914c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9168a00640 1 -- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f914c004800 msgr2=0x7f914c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9168a00640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f914c004800 0x7f914c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9168a00640 1 -- 172.21.15.6:0/2245804463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c003520 con 0x7f914c0039b0 2024-03-06T07:29:53.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.975+0000 7f9169400640 1 -- 172.21.15.6:0/2245804463 reap_dead start 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.976+0000 7f9168a00640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f914c003dd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9158001750 tx=0x7f914c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d019ac97cac111e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.976+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158001370 con 0x7f914c0039b0 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.976+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91580052d0 con 0x7f914c0039b0 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.976+0000 7f9163400640 1 -- 172.21.15.6:0/2245804463 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158003830 con 0x7f914c0039b0 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 -- 172.21.15.6:0/2245804463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 msgr2=0x7f914c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f914c003dd0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9158001750 tx=0x7f914c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 -- 172.21.15.6:0/2245804463 shutdown_connections 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 --2- 172.21.15.6:0/2245804463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f914c003dd0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 -- 172.21.15.6:0/2245804463 >> 172.21.15.6:0/2245804463 conn(0x7f91640d3d30 msgr2=0x7f91640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 -- 172.21.15.6:0/2245804463 shutdown_connections 2024-03-06T07:29:53.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.977+0000 7f9169e00640 1 -- 172.21.15.6:0/2245804463 wait complete. 2024-03-06T07:29:53.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.979+0000 7f9169e00640 1 Processor -- start 2024-03-06T07:29:53.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.979+0000 7f9169e00640 1 -- start start 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9169e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f916414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9169e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91640c7ac0 0x7f91640c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9169e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91641f7990 0x7f91641f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9169e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916415e490 con 0x7f914c0039b0 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9169e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640cfc60 con 0x7f91640c7ac0 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9169e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91640d0740 con 0x7f91641f7990 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9168a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f916414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9168a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f916414d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38348/0 (socket says 172.21.15.6:38348) 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.980+0000 7f9168a00640 1 -- 172.21.15.6:0/1112565639 learned_addr learned my addr 172.21.15.6:0/1112565639 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9168a00640 1 -- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91641f7990 msgr2=0x7f91641f7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9168a00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91641f7990 0x7f91641f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9168a00640 1 -- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91640c7ac0 msgr2=0x7f91640c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9168a00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91640c7ac0 0x7f91640c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9168a00640 1 -- 172.21.15.6:0/1112565639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916414e120 con 0x7f914c0039b0 2024-03-06T07:29:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9168a00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f916414d880 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9158005040 tx=0x7f9158001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158041070 con 0x7f914c0039b0 2024-03-06T07:29:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158005480 con 0x7f914c0039b0 2024-03-06T07:29:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158001370 con 0x7f914c0039b0 2024-03-06T07:29:53.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.981+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91640b7500 con 0x7f914c0039b0 2024-03-06T07:29:53.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.982+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91640ca950 con 0x7f914c0039b0 2024-03-06T07:29:53.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.983+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916404f330 con 0x7f914c0039b0 2024-03-06T07:29:53.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.984+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9158044070 con 0x7f914c0039b0 2024-03-06T07:29:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.984+0000 7f9160c00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f914806b4d0 0x7f914806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.985+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91580abcf0 con 0x7f914c0039b0 2024-03-06T07:29:53.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.985+0000 7f9163e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f914806b4d0 0x7f914806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.986+0000 7f9163e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f914806b4d0 0x7f914806d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9154006e80 tx=0x7f9154006dd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:53.989+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f915807aad0 con 0x7f914c0039b0 2024-03-06T07:29:54.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.206+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9164161640 con 0x7f914c0039b0 2024-03-06T07:29:54.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.206+0000 7f9160c00640 1 -- 172.21.15.6:0/1112565639 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) v1 ==== 80+0+0 (secure 0 0 0) 0x7f915807a3d0 con 0x7f914c0039b0 2024-03-06T07:29:54.209 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:29:54.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f914806b4d0 msgr2=0x7f914806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f914806b4d0 0x7f914806d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9154006e80 tx=0x7f9154006dd0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 msgr2=0x7f916414d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f916414d880 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9158005040 tx=0x7f9158001120 comp rx=0 tx=0).stop 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 shutdown_connections 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f914806b4d0 0x7f914806d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91640c7ac0 0x7f91640c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91641f7990 0x7f91641f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 --2- 172.21.15.6:0/1112565639 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f914c0039b0 0x7f916414d880 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.209+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 >> 172.21.15.6:0/1112565639 conn(0x7f91640d3d30 msgr2=0x7f91640cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.210+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 shutdown_connections 2024-03-06T07:29:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.210+0000 7f9169e00640 1 -- 172.21.15.6:0/1112565639 wait complete. 2024-03-06T07:29:54.231 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:29:54.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:54.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:29:54.410 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:29:54.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:54.410 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:29:54.447 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:29:54.447 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:54.484 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:54.484 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:29:54.484 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:54.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:29:54.529 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:29:54.529 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:54.574 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:29:54.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:54.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.783+0000 7fbaf7400640 1 Processor -- start 2024-03-06T07:29:54.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.783+0000 7fbaf7400640 1 -- start start 2024-03-06T07:29:54.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.785+0000 7fbaf7400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaf7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbaf006f140 0x7fbaf00b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaf7400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf015dc50 con 0x7fbaf006c500 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaf7400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf006c960 con 0x7fbaf006f140 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaf7400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00b3650 con 0x7fbaf015d870 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaefe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbaf006c500 0x7fbaf006ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51790/0 (socket says 172.21.15.6:51790) 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaefe00640 1 -- 172.21.15.6:0/1670600417 learned_addr learned my addr 172.21.15.6:0/1670600417 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:54.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaf5400640 1 --2- 172.21.15.6:0/1670600417 >> v2:172.21.15.6:3300/0 conn(0x7fbaf006f140 0x7fbaf00b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaf4a00640 1 --2- 172.21.15.6:0/1670600417 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaef400640 1 -- 172.21.15.6:0/1670600417 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774780861 0 0) 0x7fbaf015dc50 con 0x7fbaf006c500 2024-03-06T07:29:54.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.786+0000 7fbaef400640 1 -- 172.21.15.6:0/1670600417 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad4003440 con 0x7fbaf006c500 2024-03-06T07:29:54.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.787+0000 7fbaf5400640 1 -- 172.21.15.6:0/1670600417 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 msgr2=0x7fbaf006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:54.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.787+0000 7fbaf5400640 1 --2- 172.21.15.6:0/1670600417 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.787+0000 7fbaf5400640 1 -- 172.21.15.6:0/1670600417 >> v1:172.21.15.6:6789/0 conn(0x7fbaf006c500 legacy=0x7fbaf006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.787+0000 7fbaf5400640 1 -- 172.21.15.6:0/1670600417 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00b3930 con 0x7fbaf006f140 2024-03-06T07:29:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.788+0000 7fbaf5400640 1 --2- 172.21.15.6:0/1670600417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf00b4940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbae80097a0 tx=0x7fbae802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0644f3791ae516 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.788+0000 7fbaef400640 1 -- 172.21.15.6:0/1670600417 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8031780 con 0x7fbaf006f140 2024-03-06T07:29:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.788+0000 7fbaef400640 1 -- 172.21.15.6:0/1670600417 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae8037070 con 0x7fbaf006f140 2024-03-06T07:29:54.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.788+0000 7fbaef400640 1 -- 172.21.15.6:0/1670600417 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae8031b40 con 0x7fbaf006f140 2024-03-06T07:29:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 -- 172.21.15.6:0/1670600417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 msgr2=0x7fbaf00b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 --2- 172.21.15.6:0/1670600417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf00b4940 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbae80097a0 tx=0x7fbae802b820 comp rx=0 tx=0).stop 2024-03-06T07:29:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 -- 172.21.15.6:0/1670600417 shutdown_connections 2024-03-06T07:29:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 --2- 172.21.15.6:0/1670600417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf00b4940 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 --2- 172.21.15.6:0/1670600417 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 -- 172.21.15.6:0/1670600417 >> 172.21.15.6:0/1670600417 conn(0x7fbaf00714d0 msgr2=0x7fbaf00718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 -- 172.21.15.6:0/1670600417 shutdown_connections 2024-03-06T07:29:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.789+0000 7fbaf7400640 1 -- 172.21.15.6:0/1670600417 wait complete. 2024-03-06T07:29:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.791+0000 7fbaf7400640 1 Processor -- start 2024-03-06T07:29:54.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.791+0000 7fbaf7400640 1 -- start start 2024-03-06T07:29:54.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf7400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbaf006c500 0x7fbaf0146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf01e4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf7400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf01e52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf7400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00b41d0 con 0x7fbaf006f140 2024-03-06T07:29:54.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf7400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00b3ef0 con 0x7fbaf006c500 2024-03-06T07:29:54.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf7400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf00ce320 con 0x7fbaf015d870 2024-03-06T07:29:54.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf01e4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf01e4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38356/0 (socket says 172.21.15.6:38356) 2024-03-06T07:29:54.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 -- 172.21.15.6:0/2132879347 learned_addr learned my addr 172.21.15.6:0/2132879347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:54.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf4a00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbaf006c500 0x7fbaf0146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf5400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf01e52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 -- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 msgr2=0x7fbaf01e52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf01e52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 -- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbaf006c500 msgr2=0x7fbaf0146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:54.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbaf006c500 0x7fbaf0146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaefe00640 1 -- 172.21.15.6:0/2132879347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00adad0 con 0x7fbaf006f140 2024-03-06T07:29:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.792+0000 7fbaf5400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf01e52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.793+0000 7fbaefe00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf01e4d60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00cad0 tx=0x7fbadc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.793+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc017070 con 0x7fbaf006f140 2024-03-06T07:29:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.793+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc00ac30 con 0x7fbaf006f140 2024-03-06T07:29:54.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.793+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc012400 con 0x7fbaf006f140 2024-03-06T07:29:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.793+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf00addb0 con 0x7fbaf006f140 2024-03-06T07:29:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.793+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf00ae370 con 0x7fbaf006f140 2024-03-06T07:29:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.795+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf01572c0 con 0x7fbaf006f140 2024-03-06T07:29:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.795+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbadc01a070 con 0x7fbaf006f140 2024-03-06T07:29:54.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.796+0000 7fbaecc00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad006b4d0 0x7fbad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.796+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbadc089f10 con 0x7fbaf006f140 2024-03-06T07:29:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.797+0000 7fbaf4a00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad006b4d0 0x7fbad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.797+0000 7fbaf4a00640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad006b4d0 0x7fbad006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009b40 tx=0x7fbae0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:54.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:54.801+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbadc058740 con 0x7fbaf006f140 2024-03-06T07:29:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.031+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaf004f3a0 con 0x7fbaf006f140 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.033+0000 7fbaecc00640 1 -- 172.21.15.6:0/2132879347 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fbadc05c7e0 con 0x7fbaf006f140 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:26.635978+0000","modified":"2024-03-06T07:29:29.733477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/590822743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":590822743},{"type":"v1","addr":"172.21.15.178:6837","nonce":590822743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5562,"qdb_cluster":[5562]},"id":3}]} 2024-03-06T07:29:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-06T07:29:55.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad006b4d0 msgr2=0x7fbad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad006b4d0 0x7fbad006d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009b40 tx=0x7fbae0009190 comp rx=0 tx=0).stop 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 msgr2=0x7fbaf01e4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf01e4d60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fbadc00cad0 tx=0x7fbadc00cf90 comp rx=0 tx=0).stop 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 shutdown_connections 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad006b4d0 0x7fbad006d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbaf015d870 0x7fbaf01e52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbaf006f140 0x7fbaf01e4d60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 --2- 172.21.15.6:0/2132879347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbaf006c500 0x7fbaf0146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 >> 172.21.15.6:0/2132879347 conn(0x7fbaf00714d0 msgr2=0x7fbaf006d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.035+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 shutdown_connections 2024-03-06T07:29:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.036+0000 7fbaf7400640 1 -- 172.21.15.6:0/2132879347 wait complete. 2024-03-06T07:29:55.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:55.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.271+0000 7fcab8a00640 1 Processor -- start 2024-03-06T07:29:55.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.271+0000 7fcab8a00640 1 -- start start 2024-03-06T07:29:55.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.272+0000 7fcab8a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcab41621d0 0x7fcab4168d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcab8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab41627f0 con 0x7fcab40cd060 2024-03-06T07:29:55.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcab8a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab406c860 con 0x7fcab415efc0 2024-03-06T07:29:55.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcab8a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab415f3a0 con 0x7fcab41621d0 2024-03-06T07:29:55.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcaabe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcab40cd060 0x7fcab4161ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51800/0 (socket says 172.21.15.6:51800) 2024-03-06T07:29:55.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcab3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcab41621d0 0x7fcab4168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcaabe00640 1 -- 172.21.15.6:0/1947481881 learned_addr learned my addr 172.21.15.6:0/1947481881 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:55.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.273+0000 7fcab2a00640 1 -- 172.21.15.6:0/1947481881 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862910196 0 0) 0x7fcab41627f0 con 0x7fcab40cd060 2024-03-06T07:29:55.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.274+0000 7fcab2a00640 1 -- 172.21.15.6:0/1947481881 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca98003520 con 0x7fcab40cd060 2024-03-06T07:29:55.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.274+0000 7fcab3e00640 1 -- 172.21.15.6:0/1947481881 >> v1:172.21.15.178:6789/0 conn(0x7fcab415efc0 legacy=0x7fcab40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.274+0000 7fcab3e00640 1 -- 172.21.15.6:0/1947481881 >> v1:172.21.15.6:6789/0 conn(0x7fcab40cd060 legacy=0x7fcab4161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.274+0000 7fcab3e00640 1 -- 172.21.15.6:0/1947481881 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab406cb40 con 0x7fcab41621d0 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.275+0000 7fcab3e00640 1 --2- 172.21.15.6:0/1947481881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab4168d00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009d90 tx=0x7fcaac02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3ad9219a23427 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.275+0000 7fcab2a00640 1 -- 172.21.15.6:0/1947481881 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac039070 con 0x7fcab41621d0 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.275+0000 7fcab2a00640 1 -- 172.21.15.6:0/1947481881 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac02cd00 con 0x7fcab41621d0 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.275+0000 7fcab2a00640 1 -- 172.21.15.6:0/1947481881 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac034640 con 0x7fcab41621d0 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 -- 172.21.15.6:0/1947481881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 msgr2=0x7fcab4168d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 --2- 172.21.15.6:0/1947481881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab4168d00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009d90 tx=0x7fcaac02bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 -- 172.21.15.6:0/1947481881 shutdown_connections 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 --2- 172.21.15.6:0/1947481881 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab4168d00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 -- 172.21.15.6:0/1947481881 >> 172.21.15.6:0/1947481881 conn(0x7fcab406b3d0 msgr2=0x7fcab406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 -- 172.21.15.6:0/1947481881 shutdown_connections 2024-03-06T07:29:55.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.276+0000 7fcab8a00640 1 -- 172.21.15.6:0/1947481881 wait complete. 2024-03-06T07:29:55.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.278+0000 7fcab8a00640 1 Processor -- start 2024-03-06T07:29:55.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.278+0000 7fcab8a00640 1 -- start start 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcab40cd060 0x7fcab4157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab8a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab415efc0 0x7fcab415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab41e9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab4071c30 con 0x7fcab41621d0 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab8a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab406d190 con 0x7fcab415efc0 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab406d620 con 0x7fcab40cd060 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab41e9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab41e9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38372/0 (socket says 172.21.15.6:38372) 2024-03-06T07:29:55.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 -- 172.21.15.6:0/3496881685 learned_addr learned my addr 172.21.15.6:0/3496881685 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:55.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3400640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcab40cd060 0x7fcab4157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 -- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcab40cd060 msgr2=0x7fcab4157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcaabe00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab415efc0 0x7fcab415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcab40cd060 0x7fcab4157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 -- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab415efc0 msgr2=0x7fcab415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.279+0000 7fcab3e00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab415efc0 0x7fcab415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcab3e00640 1 -- 172.21.15.6:0/3496881685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab41e9dc0 con 0x7fcab41621d0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcab3400640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcab40cd060 0x7fcab4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcaabe00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab415efc0 0x7fcab415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcab3e00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab41e9880 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcaac037040 tx=0x7fcaac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac044020 con 0x7fcab41621d0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac03a040 con 0x7fcab41621d0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac039050 con 0x7fcab41621d0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab415ca30 con 0x7fcab41621d0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.280+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab415cff0 con 0x7fcab41621d0 2024-03-06T07:29:55.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.281+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab404f330 con 0x7fcab41621d0 2024-03-06T07:29:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.282+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcaac003bd0 con 0x7fcab41621d0 2024-03-06T07:29:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.283+0000 7fcaab400640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca9006b500 0x7fca9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.283+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcaac0ac810 con 0x7fcab41621d0 2024-03-06T07:29:55.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.284+0000 7fcab3400640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca9006b500 0x7fca9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.285+0000 7fcab3400640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca9006b500 0x7fca9006d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40015e0 tx=0x7fcaa4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.288+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcaac04e060 con 0x7fcab41621d0 2024-03-06T07:29:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.490+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcab40af2f0 con 0x7fcab41621d0 2024-03-06T07:29:55.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.492+0000 7fcaab400640 1 -- 172.21.15.6:0/3496881685 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fcaac07b680 con 0x7fcab41621d0 2024-03-06T07:29:55.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:55.495 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:28.183720+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca9006b500 msgr2=0x7fca9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca9006b500 0x7fca9006d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40015e0 tx=0x7fcaa4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 msgr2=0x7fcab41e9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab41e9880 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fcaac037040 tx=0x7fcaac004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 shutdown_connections 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca9006b500 0x7fca9006d9c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcab415efc0 0x7fcab415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcab41621d0 0x7fcab41e9880 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 --2- 172.21.15.6:0/3496881685 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcab40cd060 0x7fcab4157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.495+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 >> 172.21.15.6:0/3496881685 conn(0x7fcab406b3d0 msgr2=0x7fcab4071f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.496+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 shutdown_connections 2024-03-06T07:29:55.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.496+0000 7fcab8a00640 1 -- 172.21.15.6:0/3496881685 wait complete. 2024-03-06T07:29:55.517 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:29:24.525508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:29:25.659373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:29:55.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:29:55.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.733+0000 7fe5cba00640 1 Processor -- start 2024-03-06T07:29:55.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.734+0000 7fe5cba00640 1 -- start start 2024-03-06T07:29:55.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.735+0000 7fe5cba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c40cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.735+0000 7fe5cba00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe5c4070ce0 0x7fe5c40ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.735+0000 7fe5cba00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c4071c60 con 0x7fe5c40cd010 2024-03-06T07:29:55.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.735+0000 7fe5cba00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c40cfa10 con 0x7fe5c4070ce0 2024-03-06T07:29:55.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.735+0000 7fe5cba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4071f40 con 0x7fe5c415afa0 2024-03-06T07:29:55.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.736+0000 7fe5c3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe5c4070ce0 0x7fe5c40ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.736+0000 7fe5c9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c40cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.736+0000 7fe5c3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe5c4070ce0 0x7fe5c40ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:47364/0 (socket says 172.21.15.6:47364) 2024-03-06T07:29:55.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.736+0000 7fe5c9a00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe5c40cd010 0x7fe5c406c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:42202/0 (socket says 172.21.15.6:42202) 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.736+0000 7fe5c3e00640 1 -- 172.21.15.6:0/4040724785 learned_addr learned my addr 172.21.15.6:0/4040724785 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.737+0000 7fe5c3e00640 1 -- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 msgr2=0x7fe5c40cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.737+0000 7fe5c3e00640 1 --2- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c40cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.737+0000 7fe5c3e00640 1 -- 172.21.15.6:0/4040724785 >> v1:172.21.15.178:6789/0 conn(0x7fe5c40cd010 legacy=0x7fe5c406c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.737+0000 7fe5c3e00640 1 -- 172.21.15.6:0/4040724785 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c4072220 con 0x7fe5c4070ce0 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.737+0000 7fe5c9000640 1 --2- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c40cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:55.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.737+0000 7fe5c3400640 1 -- 172.21.15.6:0/4040724785 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157207213 0 0) 0x7fe5c4071c60 con 0x7fe5c40cd010 2024-03-06T07:29:55.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.739+0000 7fe5c3e00640 1 --2- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c40ccad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009870 tx=0x7fe5ac02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9d0338b40931b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.739+0000 7fe5c3400640 1 -- 172.21.15.6:0/4040724785 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac035020 con 0x7fe5c4070ce0 2024-03-06T07:29:55.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.739+0000 7fe5c3400640 1 -- 172.21.15.6:0/4040724785 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ac031640 con 0x7fe5c4070ce0 2024-03-06T07:29:55.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.739+0000 7fe5c3400640 1 -- 172.21.15.6:0/4040724785 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac02d040 con 0x7fe5c4070ce0 2024-03-06T07:29:55.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 -- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 msgr2=0x7fe5c40ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 --2- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c40ccad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac009870 tx=0x7fe5ac02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 -- 172.21.15.6:0/4040724785 shutdown_connections 2024-03-06T07:29:55.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 --2- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c40ccad0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 --2- 172.21.15.6:0/4040724785 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c40cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 -- 172.21.15.6:0/4040724785 >> 172.21.15.6:0/4040724785 conn(0x7fe5c406b3d0 msgr2=0x7fe5c406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 -- 172.21.15.6:0/4040724785 shutdown_connections 2024-03-06T07:29:55.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.740+0000 7fe5cba00640 1 -- 172.21.15.6:0/4040724785 wait complete. 2024-03-06T07:29:55.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.742+0000 7fe5cba00640 1 Processor -- start 2024-03-06T07:29:55.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.742+0000 7fe5cba00640 1 -- start start 2024-03-06T07:29:55.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5cba00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5cba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 0x7fe5c41e7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5cba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c41e7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5cba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40b4ad0 con 0x7fe5c40cd010 2024-03-06T07:29:55.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5cba00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4071140 con 0x7fe5c4070ce0 2024-03-06T07:29:55.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5cba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40b4c50 con 0x7fe5c415afa0 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 0x7fe5c41e7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 0x7fe5c41e7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38378/0 (socket says 172.21.15.6:38378) 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.743+0000 7fe5c3e00640 1 -- 172.21.15.6:0/3730826637 learned_addr learned my addr 172.21.15.6:0/3730826637 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c9a00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c41e7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c9000640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c3e00640 1 -- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 msgr2=0x7fe5c41e7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c3e00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c41e7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c3e00640 1 -- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 msgr2=0x7fe5c414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c3e00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c3e00640 1 -- 172.21.15.6:0/3730826637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c414cd40 con 0x7fe5c40cd010 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c9a00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c41e7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.744+0000 7fe5c3e00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 0x7fe5c41e7920 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac000c00 tx=0x7fe5ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.745+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac040070 con 0x7fe5c40cd010 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.745+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ac0399f0 con 0x7fe5c40cd010 2024-03-06T07:29:55.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.745+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac035040 con 0x7fe5c40cd010 2024-03-06T07:29:55.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.745+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c41f40e0 con 0x7fe5c40cd010 2024-03-06T07:29:55.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.745+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c41f46a0 con 0x7fe5c40cd010 2024-03-06T07:29:55.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.746+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c404f330 con 0x7fe5c40cd010 2024-03-06T07:29:55.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.747+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5ac04b060 con 0x7fe5c40cd010 2024-03-06T07:29:55.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.748+0000 7fe5c0c00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5a006b4d0 0x7fe5a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.748+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5ac0abb20 con 0x7fe5c40cd010 2024-03-06T07:29:55.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.749+0000 7fe5c9000640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5a006b4d0 0x7fe5a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.750+0000 7fe5c9000640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5a006b4d0 0x7fe5a006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40cc330 tx=0x7fe5b400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.753+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5ac07a880 con 0x7fe5c40cd010 2024-03-06T07:29:55.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.956+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5c4158fd0 con 0x7fe5c40cd010 2024-03-06T07:29:55.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.957+0000 7fe5c0c00640 1 -- 172.21.15.6:0/3730826637 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5ac07a180 con 0x7fe5c40cd010 2024-03-06T07:29:55.959 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:29:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5a006b4d0 msgr2=0x7fe5a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5a006b4d0 0x7fe5a006d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40cc330 tx=0x7fe5b400b040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 msgr2=0x7fe5c41e7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 0x7fe5c41e7920 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac000c00 tx=0x7fe5ac0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 shutdown_connections 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5a006b4d0 0x7fe5a006d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5c415afa0 0x7fe5c41e7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5c40cd010 0x7fe5c41e7920 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 --2- 172.21.15.6:0/3730826637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5c4070ce0 0x7fe5c414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 >> 172.21.15.6:0/3730826637 conn(0x7fe5c406b3d0 msgr2=0x7fe5c4158400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 shutdown_connections 2024-03-06T07:29:55.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:55.959+0000 7fe5cba00640 1 -- 172.21.15.6:0/3730826637 wait complete. 2024-03-06T07:29:55.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:29:56.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.201+0000 7f950d600640 1 Processor -- start 2024-03-06T07:29:56.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.201+0000 7f950d600640 1 -- start start 2024-03-06T07:29:56.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f950d600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 0x7f95080ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f950d600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9508072df0 con 0x7f950815bf20 2024-03-06T07:29:56.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f950d600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95080cdab0 con 0x7f9508072520 2024-03-06T07:29:56.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f950d600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95080b4d70 con 0x7f9508071e20 2024-03-06T07:29:56.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f9506a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f950815bf20 0x7f95080acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51812/0 (socket says 172.21.15.6:51812) 2024-03-06T07:29:56.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f9506a00640 1 -- 172.21.15.6:0/3232563768 learned_addr learned my addr 172.21.15.6:0/3232563768 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:56.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f9507400640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 0x7f95080ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.203+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444708368 0 0) 0x7f9508072df0 con 0x7f950815bf20 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec003520 con 0x7f950815bf20 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946985206 0 0) 0x7f94ec003520 con 0x7f950815bf20 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 msgr2=0x7f95080ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 0x7f95080ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 >> v1:172.21.15.178:6789/0 conn(0x7f9508072520 legacy=0x7f95080cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95080b5050 con 0x7f950815bf20 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f94f80019f0 con 0x7f950815bf20 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 >> v1:172.21.15.6:6789/0 conn(0x7f950815bf20 legacy=0x7f95080acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 0x7f94ec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94ec004060 0x7f94ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec004d40 0x7f94ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080b4d70 con 0x7f94ec004d40 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec00bc00 con 0x7f94ec004060 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec00be00 con 0x7f9508071e20 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.204+0000 7f9507400640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec004d40 0x7f94ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9506a00640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 0x7f94ec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9507400640 1 -- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 msgr2=0x7f94ec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9507400640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9508071e20 0x7f94ec0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9507400640 1 -- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94ec004060 msgr2=0x7f94ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9506000640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94ec004060 0x7f94ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9507400640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94ec004060 0x7f94ec004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.205+0000 7f9507400640 1 -- 172.21.15.6:0/3232563768 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec003520 con 0x7f94ec004d40 2024-03-06T07:29:56.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.206+0000 7f9507400640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec004d40 0x7f94ec0051c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f950000ba30 tx=0x7f94ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48237da2038d876c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.206+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500036020 con 0x7f94ec004d40 2024-03-06T07:29:56.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.206+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9500032640 con 0x7f94ec004d40 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.206+0000 7f9505600640 1 -- 172.21.15.6:0/3232563768 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950000b600 con 0x7f94ec004d40 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f950d600640 1 -- 172.21.15.6:0/3232563768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec004d40 msgr2=0x7f94ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f950d600640 1 --2- 172.21.15.6:0/3232563768 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94ec004d40 0x7f94ec0051c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f950000ba30 tx=0x7f94ec005830 comp rx=0 tx=0).stop 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f9507400640 1 -- 172.21.15.6:0/3232563768 reap_dead start 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f950d600640 1 -- 172.21.15.6:0/3232563768 shutdown_connections 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f950d600640 1 -- 172.21.15.6:0/3232563768 >> 172.21.15.6:0/3232563768 conn(0x7f950806b3d0 msgr2=0x7f950806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f950d600640 1 -- 172.21.15.6:0/3232563768 shutdown_connections 2024-03-06T07:29:56.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.207+0000 7f950d600640 1 -- 172.21.15.6:0/3232563768 wait complete. 2024-03-06T07:29:56.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.209+0000 7f950d600640 1 Processor -- start 2024-03-06T07:29:56.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.209+0000 7f950d600640 1 -- start start 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.210+0000 7f950d600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94ec004d40 0x7f950809abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.210+0000 7f950d600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95081e5610 0x7f95080ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.210+0000 7f950d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 0x7f9508149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.210+0000 7f950d600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080ad450 con 0x7f9508149180 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.210+0000 7f950d600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95080b4990 con 0x7f95081e5610 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.210+0000 7f950d600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9508149b40 con 0x7f94ec004d40 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 0x7f9508149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 0x7f9508149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38396/0 (socket says 172.21.15.6:38396) 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 -- 172.21.15.6:0/791061624 learned_addr learned my addr 172.21.15.6:0/791061624 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:56.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9506a00640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94ec004d40 0x7f950809abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9506000640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95081e5610 0x7f95080ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 -- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94ec004d40 msgr2=0x7f950809abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94ec004d40 0x7f950809abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 -- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95081e5610 msgr2=0x7f95080ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95081e5610 0x7f95080ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 -- 172.21.15.6:0/791061624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95081f7b20 con 0x7f9508149180 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.211+0000 7f9507400640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 0x7f9508149600 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f950000bb50 tx=0x7f9500004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.212+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500042070 con 0x7f9508149180 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.212+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9500004020 con 0x7f9508149180 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.212+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500036040 con 0x7f9508149180 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.212+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95081f7e00 con 0x7f9508149180 2024-03-06T07:29:56.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.212+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95081e5a90 con 0x7f9508149180 2024-03-06T07:29:56.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.213+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950804f330 con 0x7f9508149180 2024-03-06T07:29:56.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.214+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9500039020 con 0x7f9508149180 2024-03-06T07:29:56.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.215+0000 7f94fea00640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94e406b4d0 0x7f94e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.215+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95000ac130 con 0x7f9508149180 2024-03-06T07:29:56.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.215+0000 7f9506a00640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94e406b4d0 0x7f94e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.217+0000 7f9506a00640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94e406b4d0 0x7f94e406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9508156b90 tx=0x7f94f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.219+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f950007ae80 con 0x7f9508149180 2024-03-06T07:29:56.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.422+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f950806e4d0 con 0x7f9508149180 2024-03-06T07:29:56.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.423+0000 7f94fea00640 1 -- 172.21.15.6:0/791061624 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) v1 ==== 104+0+11 (secure 0 0 0) 0x7f950007a780 con 0x7f9508149180 2024-03-06T07:29:56.425 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:29:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94e406b4d0 msgr2=0x7f94e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94e406b4d0 0x7f94e406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9508156b90 tx=0x7f94f8009040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 msgr2=0x7f9508149600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 0x7f9508149600 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f950000bb50 tx=0x7f9500004730 comp rx=0 tx=0).stop 2024-03-06T07:29:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 shutdown_connections 2024-03-06T07:29:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94e406b4d0 0x7f94e406d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95081e5610 0x7f95080ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.425+0000 7f950d600640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9508149180 0x7f9508149600 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.426+0000 7f950d600640 1 --2- 172.21.15.6:0/791061624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94ec004d40 0x7f950809abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.426+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 >> 172.21.15.6:0/791061624 conn(0x7f950806b3d0 msgr2=0x7f95080b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.426+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 shutdown_connections 2024-03-06T07:29:56.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.426+0000 7f950d600640 1 -- 172.21.15.6:0/791061624 wait complete. 2024-03-06T07:29:56.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:29:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.663+0000 7f2d14800640 1 Processor -- start 2024-03-06T07:29:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.663+0000 7f2d14800640 1 -- start start 2024-03-06T07:29:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.664+0000 7f2d14800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2d0c15bf20 0x7f2d0c0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.664+0000 7f2d14800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 0x7f2d0c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.664+0000 7f2d14800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0c072df0 con 0x7f2d0c072520 2024-03-06T07:29:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.664+0000 7f2d14800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0c0cdab0 con 0x7f2d0c15bf20 2024-03-06T07:29:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.664+0000 7f2d14800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0b4d70 con 0x7f2d0c071e20 2024-03-06T07:29:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.665+0000 7f2d11400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d0c072520 0x7f2d0c0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51820/0 (socket says 172.21.15.6:51820) 2024-03-06T07:29:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.665+0000 7f2d11400640 1 -- 172.21.15.6:0/506695679 learned_addr learned my addr 172.21.15.6:0/506695679 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.665+0000 7f2d11e00640 1 --2- 172.21.15.6:0/506695679 >> v2:172.21.15.178:3300/0 conn(0x7f2d0c15bf20 0x7f2d0c0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.665+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050764967 0 0) 0x7f2d0c072df0 con 0x7f2d0c072520 2024-03-06T07:29:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.665+0000 7f2d12800640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 0x7f2d0c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.665+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf4003520 con 0x7f2d0c072520 2024-03-06T07:29:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1766674520 0 0) 0x7f2cf4003520 con 0x7f2d0c072520 2024-03-06T07:29:56.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 msgr2=0x7f2d0c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 0x7f2d0c0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 >> v2:172.21.15.178:3300/0 conn(0x7f2d0c15bf20 msgr2=0x7f2d0c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 --2- 172.21.15.6:0/506695679 >> v2:172.21.15.178:3300/0 conn(0x7f2d0c15bf20 0x7f2d0c0acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c0b5050 con 0x7f2d0c072520 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2cfc001670 con 0x7f2d0c072520 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 >> v1:172.21.15.6:6789/0 conn(0x7f2d0c072520 legacy=0x7f2d0c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2cf40039b0 0x7f2cf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cf4004800 0x7f2cf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 0x7f2cf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0cdab0 con 0x7f2cf4004800 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0b4d70 con 0x7f2cf40039b0 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf400b9c0 con 0x7f2d0c071e20 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.666+0000 7f2d11400640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cf4004800 0x7f2cf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d12800640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 0x7f2cf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d11e00640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2cf40039b0 0x7f2cf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d11400640 1 -- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 msgr2=0x7f2cf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d11400640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d0c071e20 0x7f2cf40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d11400640 1 -- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2cf40039b0 msgr2=0x7f2cf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d11400640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2cf40039b0 0x7f2cf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.667+0000 7f2d11400640 1 -- 172.21.15.6:0/506695679 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4003520 con 0x7f2cf4004800 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.668+0000 7f2d11400640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cf4004800 0x7f2cf4004c80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c160 tx=0x7f2cf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9018e17b6295bf8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.668+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc007c80 con 0x7f2cf4004800 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.668+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc030070 con 0x7f2cf4004800 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.668+0000 7f2d10a00640 1 -- 172.21.15.6:0/506695679 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc005d30 con 0x7f2cf4004800 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d14800640 1 -- 172.21.15.6:0/506695679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cf4004800 msgr2=0x7f2cf4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d14800640 1 --2- 172.21.15.6:0/506695679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cf4004800 0x7f2cf4004c80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c160 tx=0x7f2cf4005830 comp rx=0 tx=0).stop 2024-03-06T07:29:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d12800640 1 -- 172.21.15.6:0/506695679 reap_dead start 2024-03-06T07:29:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d14800640 1 -- 172.21.15.6:0/506695679 shutdown_connections 2024-03-06T07:29:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d14800640 1 -- 172.21.15.6:0/506695679 >> 172.21.15.6:0/506695679 conn(0x7f2d0c06b3d0 msgr2=0x7f2d0c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d14800640 1 -- 172.21.15.6:0/506695679 shutdown_connections 2024-03-06T07:29:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.669+0000 7f2d14800640 1 -- 172.21.15.6:0/506695679 wait complete. 2024-03-06T07:29:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.671+0000 7f2d14800640 1 Processor -- start 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.671+0000 7f2d14800640 1 -- start start 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d14800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2cf4004800 0x7f2d0c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d14800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d0c14d780 0x7f2d0c0af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d14800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 0x7f2d0c0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d14800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0b4b10 con 0x7f2d0c0ca120 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d14800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0b4990 con 0x7f2d0c14d780 2024-03-06T07:29:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d14800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0ce6c0 con 0x7f2cf4004800 2024-03-06T07:29:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 0x7f2d0c0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 0x7f2d0c0ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38410/0 (socket says 172.21.15.6:38410) 2024-03-06T07:29:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 -- 172.21.15.6:0/493004084 learned_addr learned my addr 172.21.15.6:0/493004084 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d11e00640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2cf4004800 0x7f2d0c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d11400640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d0c14d780 0x7f2d0c0af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 -- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2cf4004800 msgr2=0x7f2d0c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2cf4004800 0x7f2d0c06d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 -- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d0c14d780 msgr2=0x7f2d0c0af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d0c14d780 0x7f2d0c0af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.672+0000 7f2d12800640 1 -- 172.21.15.6:0/493004084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c06e760 con 0x7f2d0c0ca120 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.673+0000 7f2d12800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 0x7f2d0c0ca5a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c0cd670 tx=0x7f2d080115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.673+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08011e40 con 0x7f2d0c0ca120 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.673+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0801b070 con 0x7f2d0c0ca120 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.673+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0c0c9080 con 0x7f2d0c0ca120 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.673+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08003990 con 0x7f2d0c0ca120 2024-03-06T07:29:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.673+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0c0c9640 con 0x7f2d0c0ca120 2024-03-06T07:29:56.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.674+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0c04f330 con 0x7f2d0c0ca120 2024-03-06T07:29:56.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.675+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d08005d10 con 0x7f2d0c0ca120 2024-03-06T07:29:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.676+0000 7f2d06000640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2cf006b4d0 0x7f2cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.676+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d0808a4f0 con 0x7f2d0c0ca120 2024-03-06T07:29:56.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.677+0000 7f2d11e00640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2cf006b4d0 0x7f2cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.678+0000 7f2d11e00640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2cf006b4d0 0x7f2cf006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009a10 tx=0x7f2d00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.681+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d08014350 con 0x7f2d0c0ca120 2024-03-06T07:29:56.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.881+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2d0c071a90 con 0x7f2d0c0ca120 2024-03-06T07:29:56.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.882+0000 7f2d06000640 1 -- 172.21.15.6:0/493004084 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2d080593e0 con 0x7f2d0c0ca120 2024-03-06T07:29:56.884 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.884+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2cf006b4d0 msgr2=0x7f2cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.884+0000 7f2d14800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2cf006b4d0 0x7f2cf006d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d00009a10 tx=0x7f2d00008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.884+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 msgr2=0x7f2d0c0ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.884+0000 7f2d14800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 0x7f2d0c0ca5a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c0cd670 tx=0x7f2d080115a0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 shutdown_connections 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2cf006b4d0 0x7f2cf006d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d0c14d780 0x7f2d0c0af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d0c0ca120 0x7f2d0c0ca5a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 --2- 172.21.15.6:0/493004084 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2cf4004800 0x7f2d0c06d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 >> 172.21.15.6:0/493004084 conn(0x7f2d0c06b3d0 msgr2=0x7f2d0c157d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 shutdown_connections 2024-03-06T07:29:56.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:56.885+0000 7f2d14800640 1 -- 172.21.15.6:0/493004084 wait complete. 2024-03-06T07:29:56.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.125+0000 7f0b2d200640 1 Processor -- start 2024-03-06T07:29:57.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.125+0000 7f0b2d200640 1 -- start start 2024-03-06T07:29:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b2d200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 0x7f0b281667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b2d200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280714d0 con 0x7f0b28145300 2024-03-06T07:29:57.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b2d200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280717b0 con 0x7f0b28070f10 2024-03-06T07:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b2d200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28071a90 con 0x7f0b280cd060 2024-03-06T07:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b27400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b28145300 0x7f0b280b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51830/0 (socket says 172.21.15.6:51830) 2024-03-06T07:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b27400640 1 -- 172.21.15.6:0/3897171775 learned_addr learned my addr 172.21.15.6:0/3897171775 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 0x7f0b281667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140994422 0 0) 0x7f0b280714d0 con 0x7f0b28145300 2024-03-06T07:29:57.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.127+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b0c003520 con 0x7f0b28145300 2024-03-06T07:29:57.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1124216610 0 0) 0x7f0b0c003520 con 0x7f0b28145300 2024-03-06T07:29:57.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 msgr2=0x7f0b281667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 0x7f0b281667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 >> v1:172.21.15.178:6789/0 conn(0x7f0b28070f10 legacy=0x7f0b280cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28071d70 con 0x7f0b28145300 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0b10001670 con 0x7f0b28145300 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 >> v1:172.21.15.6:6789/0 conn(0x7f0b28145300 legacy=0x7f0b280b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 0x7f0b0c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b0c004d40 0x7f0b0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28071a90 con 0x7f0b0c003fb0 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c00bbb0 con 0x7f0b0c004d40 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c00bdb0 con 0x7f0b280cd060 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.128+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b0c004d40 0x7f0b0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b27400640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 0x7f0b0c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b26a00640 1 -- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 msgr2=0x7f0b0c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b280cd060 0x7f0b0c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b26a00640 1 -- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b0c004d40 msgr2=0x7f0b0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b0c004d40 0x7f0b0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b26a00640 1 -- 172.21.15.6:0/3897171775 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c003520 con 0x7f0b0c003fb0 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.129+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b0c004d40 0x7f0b0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.130+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b0c004800 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001e50 tx=0x7f0b0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96abec1987be9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.130+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b18001a70 con 0x7f0b0c003fb0 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.130+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b18003b70 con 0x7f0b0c003fb0 2024-03-06T07:29:57.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.130+0000 7f0b26000640 1 -- 172.21.15.6:0/3897171775 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b180050d0 con 0x7f0b0c003fb0 2024-03-06T07:29:57.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b2d200640 1 -- 172.21.15.6:0/3897171775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 msgr2=0x7f0b0c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3897171775 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b0c004800 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0b18001e50 tx=0x7f0b0c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:57.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b27e00640 1 -- 172.21.15.6:0/3897171775 reap_dead start 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b2d200640 1 -- 172.21.15.6:0/3897171775 shutdown_connections 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b2d200640 1 -- 172.21.15.6:0/3897171775 >> 172.21.15.6:0/3897171775 conn(0x7f0b2806b3d0 msgr2=0x7f0b2806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b2d200640 1 -- 172.21.15.6:0/3897171775 shutdown_connections 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.131+0000 7f0b2d200640 1 -- 172.21.15.6:0/3897171775 wait complete. 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.133+0000 7f0b2d200640 1 Processor -- start 2024-03-06T07:29:57.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.133+0000 7f0b2d200640 1 -- start start 2024-03-06T07:29:57.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b2d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b281500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b2d200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b28142410 0x7f0b2813b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b2d200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b28142890 0x7f0b2813b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b2d200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2815d700 con 0x7f0b0c003fb0 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b2d200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2815d580 con 0x7f0b28142890 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b2d200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2815d880 con 0x7f0b28142410 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b27400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b281500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b27400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b281500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38426/0 (socket says 172.21.15.6:38426) 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b27400640 1 -- 172.21.15.6:0/3598845975 learned_addr learned my addr 172.21.15.6:0/3598845975 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b28142410 0x7f0b2813b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.134+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b28142890 0x7f0b2813b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27400640 1 -- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b28142410 msgr2=0x7f0b2813b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27400640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b28142410 0x7f0b2813b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27400640 1 -- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b28142890 msgr2=0x7f0b2813b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27400640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b28142890 0x7f0b2813b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27400640 1 -- 172.21.15.6:0/3598845975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28140730 con 0x7f0b0c003fb0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27e00640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b28142890 0x7f0b2813b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b27400640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b281500d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000e780 tx=0x7f0b1000eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000c660 con 0x7f0b0c003fb0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b10018070 con 0x7f0b0c003fb0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000ca00 con 0x7f0b0c003fb0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b28140a10 con 0x7f0b0c003fb0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.135+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b28140fd0 con 0x7f0b0c003fb0 2024-03-06T07:29:57.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.137+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2804f330 con 0x7f0b0c003fb0 2024-03-06T07:29:57.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.138+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b1000cba0 con 0x7f0b0c003fb0 2024-03-06T07:29:57.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.139+0000 7f0b1f400640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b0406b4d0 0x7f0b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.139+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b1008a0d0 con 0x7f0b0c003fb0 2024-03-06T07:29:57.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.140+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b0406b4d0 0x7f0b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.141+0000 7f0b26a00640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b0406b4d0 0x7f0b0406d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0b18034000 tx=0x7f0b18009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.144+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b10058eb0 con 0x7f0b0c003fb0 2024-03-06T07:29:57.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.385+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b28144020 con 0x7f0b0c003fb0 2024-03-06T07:29:57.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.386+0000 7f0b1f400640 1 -- 172.21.15.6:0/3598845975 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f0b100587b0 con 0x7f0b0c003fb0 2024-03-06T07:29:57.389 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:57.389 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:26.635978+0000","modified":"2024-03-06T07:29:29.733477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/590822743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":590822743},{"type":"v1","addr":"172.21.15.178:6837","nonce":590822743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5562,"qdb_cluster":[5562]},"id":3}]} 2024-03-06T07:29:57.389 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-06T07:29:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b0406b4d0 msgr2=0x7f0b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b0406b4d0 0x7f0b0406d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0b18034000 tx=0x7f0b18009030 comp rx=0 tx=0).stop 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 msgr2=0x7f0b281500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b281500d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000e780 tx=0x7f0b1000eb80 comp rx=0 tx=0).stop 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 shutdown_connections 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b0406b4d0 0x7f0b0406d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b28142890 0x7f0b2813b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b28142410 0x7f0b2813b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 --2- 172.21.15.6:0/3598845975 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0c003fb0 0x7f0b281500d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 >> 172.21.15.6:0/3598845975 conn(0x7f0b2806b3d0 msgr2=0x7f0b2806d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 shutdown_connections 2024-03-06T07:29:57.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.389+0000 7f0b2d200640 1 -- 172.21.15.6:0/3598845975 wait complete. 2024-03-06T07:29:57.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:57.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.628+0000 7fece4e00640 1 Processor -- start 2024-03-06T07:29:57.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.628+0000 7fece4e00640 1 -- start start 2024-03-06T07:29:57.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fece4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece00d4a10 0x7fece00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fece4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fece00d0c80 0x7fece00b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fece4e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece00b3710 con 0x7fece015e0b0 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fece4e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece00b09e0 con 0x7fece00d0c80 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fece4e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece00cfa20 con 0x7fece00d4a10 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fecdfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fece00d0c80 0x7fece00b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.629+0000 7fecdfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fece00d0c80 0x7fece00b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38434/0 (socket says 172.21.15.6:38434) 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.630+0000 7fecdfe00640 1 -- 172.21.15.6:0/3521393102 learned_addr learned my addr 172.21.15.6:0/3521393102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.630+0000 7fecdfe00640 1 -- 172.21.15.6:0/3521393102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece00d4a10 msgr2=0x7fece00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.630+0000 7fecdfe00640 1 --2- 172.21.15.6:0/3521393102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece00d4a10 0x7fece00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.630+0000 7fecdfe00640 1 -- 172.21.15.6:0/3521393102 >> v1:172.21.15.178:6789/0 conn(0x7fece015e0b0 legacy=0x7fece00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.630+0000 7fecdfe00640 1 -- 172.21.15.6:0/3521393102 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece00b39f0 con 0x7fece00d0c80 2024-03-06T07:29:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.631+0000 7fecdfe00640 1 --2- 172.21.15.6:0/3521393102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d0c80 0x7fece00b44d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009870 tx=0x7fecd802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be83a2de6e67c06d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.631+0000 7fecde000640 1 -- 172.21.15.6:0/3521393102 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8035020 con 0x7fece00d0c80 2024-03-06T07:29:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.631+0000 7fecde000640 1 -- 172.21.15.6:0/3521393102 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd8031640 con 0x7fece00d0c80 2024-03-06T07:29:57.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.631+0000 7fecde000640 1 -- 172.21.15.6:0/3521393102 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd802d040 con 0x7fece00d0c80 2024-03-06T07:29:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 -- 172.21.15.6:0/3521393102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d0c80 msgr2=0x7fece00b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 --2- 172.21.15.6:0/3521393102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d0c80 0x7fece00b44d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009870 tx=0x7fecd802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 -- 172.21.15.6:0/3521393102 shutdown_connections 2024-03-06T07:29:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 --2- 172.21.15.6:0/3521393102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d0c80 0x7fece00b44d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 --2- 172.21.15.6:0/3521393102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece00d4a10 0x7fece00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 -- 172.21.15.6:0/3521393102 >> 172.21.15.6:0/3521393102 conn(0x7fece00d3d30 msgr2=0x7fece00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 -- 172.21.15.6:0/3521393102 shutdown_connections 2024-03-06T07:29:57.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.632+0000 7fece4e00640 1 -- 172.21.15.6:0/3521393102 wait complete. 2024-03-06T07:29:57.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.634+0000 7fece4e00640 1 Processor -- start 2024-03-06T07:29:57.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.634+0000 7fece4e00640 1 -- start start 2024-03-06T07:29:57.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fece4e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fece00d0c80 0x7fece01f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fece4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 0x7fece00b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fece4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece015e0b0 0x7fece01f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fece4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece015e490 con 0x7fece00d4a10 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fece4e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b3fb0 con 0x7fece00d0c80 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fece4e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece00cfc60 con 0x7fece015e0b0 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fecdea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 0x7fece00b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fecdea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 0x7fece00b87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38442/0 (socket says 172.21.15.6:38442) 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fecdea00640 1 -- 172.21.15.6:0/1790148800 learned_addr learned my addr 172.21.15.6:0/1790148800 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.635+0000 7fecdf400640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fece00d0c80 0x7fece01f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdfe00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece015e0b0 0x7fece01f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdea00640 1 -- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece015e0b0 msgr2=0x7fece01f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdea00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece015e0b0 0x7fece01f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdea00640 1 -- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fece00d0c80 msgr2=0x7fece01f4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdea00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fece00d0c80 0x7fece01f4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdea00640 1 -- 172.21.15.6:0/1790148800 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece01585e0 con 0x7fece00d4a10 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecdea00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 0x7fece00b87f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fecc800b480 tx=0x7fecc800b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8014070 con 0x7fece00d4a10 2024-03-06T07:29:57.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc8010870 con 0x7fece00d4a10 2024-03-06T07:29:57.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8010c60 con 0x7fece00d4a10 2024-03-06T07:29:57.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece01589c0 con 0x7fece00d4a10 2024-03-06T07:29:57.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.636+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece0158f80 con 0x7fece00d4a10 2024-03-06T07:29:57.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.637+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece004f330 con 0x7fece00d4a10 2024-03-06T07:29:57.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.638+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fecc8017070 con 0x7fece00d4a10 2024-03-06T07:29:57.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.639+0000 7fecd7400640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecbc06b4d0 0x7fecbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:57.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.639+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecc8089ff0 con 0x7fece00d4a10 2024-03-06T07:29:57.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.640+0000 7fecdf400640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecbc06b4d0 0x7fecbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:57.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.641+0000 7fecdf400640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecbc06b4d0 0x7fecbc06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fecd0005520 tx=0x7fecd0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:57.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.644+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fecc8058ea0 con 0x7fece00d4a10 2024-03-06T07:29:57.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.843+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fece00b0fc0 con 0x7fece00d4a10 2024-03-06T07:29:57.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.845+0000 7fecd7400640 1 -- 172.21.15.6:0/1790148800 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7fecc80587a0 con 0x7fece00d4a10 2024-03-06T07:29:57.848 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:57.848 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:28.183720+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:57.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecbc06b4d0 msgr2=0x7fecbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecbc06b4d0 0x7fecbc06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fecd0005520 tx=0x7fecd0009040 comp rx=0 tx=0).stop 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 msgr2=0x7fece00b87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 0x7fece00b87f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fecc800b480 tx=0x7fecc800b940 comp rx=0 tx=0).stop 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 shutdown_connections 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fecbc06b4d0 0x7fecbc06d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fece015e0b0 0x7fece01f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fece00d4a10 0x7fece00b87f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 --2- 172.21.15.6:0/1790148800 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fece00d0c80 0x7fece01f4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 >> 172.21.15.6:0/1790148800 conn(0x7fece00d3d30 msgr2=0x7fece0073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 shutdown_connections 2024-03-06T07:29:57.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:57.848+0000 7fece4e00640 1 -- 172.21.15.6:0/1790148800 wait complete. 2024-03-06T07:29:57.872 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:29:57.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.083+0000 7f8a77e00640 1 Processor -- start 2024-03-06T07:29:58.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.084+0000 7f8a77e00640 1 -- start start 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a780cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8a780d4a10 0x7f8a780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a780b3710 con 0x7f8a780d0c80 2024-03-06T07:29:58.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a780b09e0 con 0x7f8a780d4a10 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780b39f0 con 0x7f8a7815e0b0 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8a780d0c80 0x7f8a780d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51844/0 (socket says 172.21.15.6:51844) 2024-03-06T07:29:58.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.085+0000 7f8a77400640 1 -- 172.21.15.6:0/2231485365 learned_addr learned my addr 172.21.15.6:0/2231485365 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a76000640 1 --2- 172.21.15.6:0/2231485365 >> v2:172.21.15.178:3300/0 conn(0x7f8a780d4a10 0x7f8a780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a76a00640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a780cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523146708 0 0) 0x7f8a780b3710 con 0x7f8a780d0c80 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a58003520 con 0x7f8a780d0c80 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851939143 0 0) 0x7f8a58003520 con 0x7f8a780d0c80 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 msgr2=0x7f8a780cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a780cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 >> v2:172.21.15.178:3300/0 conn(0x7f8a780d4a10 msgr2=0x7f8a780d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 --2- 172.21.15.6:0/2231485365 >> v2:172.21.15.178:3300/0 conn(0x7f8a780d4a10 0x7f8a780d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780b3cd0 con 0x7f8a780d0c80 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8a700019f0 con 0x7f8a780d0c80 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 >> v1:172.21.15.6:6789/0 conn(0x7f8a780d0c80 legacy=0x7f8a780d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a580039b0 0x7f8a58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780b39f0 con 0x7f8a58004800 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5800b9c0 con 0x7f8a580039b0 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.086+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5800bb40 con 0x7f8a7815e0b0 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76000640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76a00640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a580039b0 0x7f8a58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76000640 1 -- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 msgr2=0x7f8a580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a77400640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76000640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76000640 1 -- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a580039b0 msgr2=0x7f8a58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76000640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a580039b0 0x7f8a58003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76000640 1 -- 172.21.15.6:0/2231485365 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a58003520 con 0x7f8a58004800 2024-03-06T07:29:58.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a76a00640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a580039b0 0x7f8a58003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.087+0000 7f8a77400640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815e0b0 0x7f8a580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.088+0000 7f8a76000640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009a90 tx=0x7f8a58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a673a75d4c25961 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60039070 con 0x7f8a58004800 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6002c6b0 con 0x7f8a58004800 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a75600640 1 -- 172.21.15.6:0/2231485365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a60009690 con 0x7f8a58004800 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a77e00640 1 -- 172.21.15.6:0/2231485365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 msgr2=0x7f8a58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a77e00640 1 --2- 172.21.15.6:0/2231485365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a58004c80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009a90 tx=0x7f8a58005830 comp rx=0 tx=0).stop 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a77400640 1 -- 172.21.15.6:0/2231485365 reap_dead start 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a77e00640 1 -- 172.21.15.6:0/2231485365 shutdown_connections 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a77e00640 1 -- 172.21.15.6:0/2231485365 >> 172.21.15.6:0/2231485365 conn(0x7f8a780d3d30 msgr2=0x7f8a780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.089+0000 7f8a77e00640 1 -- 172.21.15.6:0/2231485365 shutdown_connections 2024-03-06T07:29:58.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.090+0000 7f8a77e00640 1 -- 172.21.15.6:0/2231485365 wait complete. 2024-03-06T07:29:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.091+0000 7f8a77e00640 1 Processor -- start 2024-03-06T07:29:58.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.091+0000 7f8a77e00640 1 -- start start 2024-03-06T07:29:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.092+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a780cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.092+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a781f3840 0x7f8a781f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.092+0000 7f8a77e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815c970 0x7f8a780b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.092+0000 7f8a77e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780cfc60 con 0x7f8a58004800 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.092+0000 7f8a77e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7815e490 con 0x7f8a781f3840 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.092+0000 7f8a77e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780b54e0 con 0x7f8a7815c970 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a780cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a780cc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38456/0 (socket says 172.21.15.6:38456) 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 -- 172.21.15.6:0/3518164706 learned_addr learned my addr 172.21.15.6:0/3518164706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 -- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815c970 msgr2=0x7f8a780b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815c970 0x7f8a780b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 -- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a781f3840 msgr2=0x7f8a781f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76000640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a781f3840 0x7f8a781f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a781f3840 0x7f8a781f78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 -- 172.21.15.6:0/3518164706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a781f7e20 con 0x7f8a58004800 2024-03-06T07:29:58.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.093+0000 7f8a76a00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a780cc530 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8a680116d0 tx=0x7f8a68011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.094+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800f9c0 con 0x7f8a58004800 2024-03-06T07:29:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.094+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6800fb60 con 0x7f8a58004800 2024-03-06T07:29:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.094+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a781f4870 con 0x7f8a58004800 2024-03-06T07:29:58.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.094+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68015460 con 0x7f8a58004800 2024-03-06T07:29:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.094+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a781f3fd0 con 0x7f8a58004800 2024-03-06T07:29:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.095+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7804f330 con 0x7f8a58004800 2024-03-06T07:29:58.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.096+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a68015600 con 0x7f8a58004800 2024-03-06T07:29:58.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.097+0000 7f8a6f400640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8a5406b4d0 0x7f8a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.097+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a6808aa50 con 0x7f8a58004800 2024-03-06T07:29:58.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.098+0000 7f8a76000640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8a5406b4d0 0x7f8a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.099+0000 7f8a76000640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8a5406b4d0 0x7f8a5406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a60036000 tx=0x7f8a60030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.102+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a781f7e20 con 0x7f8a58004800 2024-03-06T07:29:58.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.345+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a780b2820 con 0x7f8a58004800 2024-03-06T07:29:58.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.346+0000 7f8a6f400640 1 -- 172.21.15.6:0/3518164706 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f8a68019030 con 0x7f8a58004800 2024-03-06T07:29:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:58.349 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:26.635978+0000","modified":"2024-03-06T07:29:29.733477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/590822743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":590822743},{"type":"v1","addr":"172.21.15.178:6837","nonce":590822743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5562,"qdb_cluster":[5562]},"id":3}]} 2024-03-06T07:29:58.349 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.348+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8a5406b4d0 msgr2=0x7f8a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.348+0000 7f8a77e00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8a5406b4d0 0x7f8a5406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a60036000 tx=0x7f8a60030000 comp rx=0 tx=0).stop 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.348+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 msgr2=0x7f8a780cc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.348+0000 7f8a77e00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a780cc530 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8a680116d0 tx=0x7f8a68011b90 comp rx=0 tx=0).stop 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 shutdown_connections 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8a5406b4d0 0x7f8a5406d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8a781f3840 0x7f8a781f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8a7815c970 0x7f8a780b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 --2- 172.21.15.6:0/3518164706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8a58004800 0x7f8a780cc530 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 >> 172.21.15.6:0/3518164706 conn(0x7f8a780d3d30 msgr2=0x7f8a7806cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 shutdown_connections 2024-03-06T07:29:58.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.349+0000 7f8a77e00640 1 -- 172.21.15.6:0/3518164706 wait complete. 2024-03-06T07:29:58.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.595+0000 7fb2ee800640 1 Processor -- start 2024-03-06T07:29:58.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.595+0000 7fb2ee800640 1 -- start start 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.597+0000 7fb2ee800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb2e8145300 0x7fb2e80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.597+0000 7fb2ee800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb2e80cd060 0x7fb2e815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.597+0000 7fb2ee800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80714d0 con 0x7fb2e8070f10 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.597+0000 7fb2ee800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e80717b0 con 0x7fb2e80cd060 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.597+0000 7fb2ee800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e8071a90 con 0x7fb2e8145300 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.598+0000 7fb2eca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb2e8070f10 0x7fb2e80cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51860/0 (socket says 172.21.15.6:51860) 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.598+0000 7fb2eca00640 1 -- 172.21.15.6:0/2945421121 learned_addr learned my addr 172.21.15.6:0/2945421121 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.598+0000 7fb2ede00640 1 --2- 172.21.15.6:0/2945421121 >> v2:172.21.15.6:3300/0 conn(0x7fb2e80cd060 0x7fb2e815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.598+0000 7fb2ed400640 1 --2- 172.21.15.6:0/2945421121 >> v2:172.21.15.178:3300/0 conn(0x7fb2e8145300 0x7fb2e80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.598+0000 7fb2e7e00640 1 -- 172.21.15.6:0/2945421121 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094500516 0 0) 0x7fb2e80714d0 con 0x7fb2e8070f10 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.599+0000 7fb2e7e00640 1 -- 172.21.15.6:0/2945421121 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0003520 con 0x7fb2e8070f10 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.599+0000 7fb2ede00640 1 -- 172.21.15.6:0/2945421121 >> v2:172.21.15.178:3300/0 conn(0x7fb2e8145300 msgr2=0x7fb2e80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.599+0000 7fb2ede00640 1 --2- 172.21.15.6:0/2945421121 >> v2:172.21.15.178:3300/0 conn(0x7fb2e8145300 0x7fb2e80b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.599+0000 7fb2ede00640 1 -- 172.21.15.6:0/2945421121 >> v1:172.21.15.6:6789/0 conn(0x7fb2e8070f10 legacy=0x7fb2e80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.599+0000 7fb2ede00640 1 -- 172.21.15.6:0/2945421121 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8071d70 con 0x7fb2e80cd060 2024-03-06T07:29:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.599+0000 7fb2ed400640 1 --2- 172.21.15.6:0/2945421121 >> v2:172.21.15.178:3300/0 conn(0x7fb2e8145300 0x7fb2e80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.600+0000 7fb2ede00640 1 --2- 172.21.15.6:0/2945421121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e80cd060 0x7fb2e815c570 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80cf990 tx=0x7fb2e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332682c9fe0fe428 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2e7e00640 1 -- 172.21.15.6:0/2945421121 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00315a0 con 0x7fb2e80cd060 2024-03-06T07:29:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2e7e00640 1 -- 172.21.15.6:0/2945421121 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e0037070 con 0x7fb2e80cd060 2024-03-06T07:29:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2e7e00640 1 -- 172.21.15.6:0/2945421121 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0031940 con 0x7fb2e80cd060 2024-03-06T07:29:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 -- 172.21.15.6:0/2945421121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e80cd060 msgr2=0x7fb2e815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 --2- 172.21.15.6:0/2945421121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e80cd060 0x7fb2e815c570 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80cf990 tx=0x7fb2e002b840 comp rx=0 tx=0).stop 2024-03-06T07:29:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 -- 172.21.15.6:0/2945421121 shutdown_connections 2024-03-06T07:29:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 --2- 172.21.15.6:0/2945421121 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e80cd060 0x7fb2e815c570 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 --2- 172.21.15.6:0/2945421121 >> v2:172.21.15.178:3300/0 conn(0x7fb2e8145300 0x7fb2e80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 -- 172.21.15.6:0/2945421121 >> 172.21.15.6:0/2945421121 conn(0x7fb2e806b3d0 msgr2=0x7fb2e806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.601+0000 7fb2ee800640 1 -- 172.21.15.6:0/2945421121 shutdown_connections 2024-03-06T07:29:58.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.602+0000 7fb2ee800640 1 -- 172.21.15.6:0/2945421121 wait complete. 2024-03-06T07:29:58.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.603+0000 7fb2ee800640 1 Processor -- start 2024-03-06T07:29:58.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.603+0000 7fb2ee800640 1 -- start start 2024-03-06T07:29:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.604+0000 7fb2ee800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2e8070f10 0x7fb2e81428e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.604+0000 7fb2ee800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2e80cd060 0x7fb2e80ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.604+0000 7fb2ee800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 0x7fb2e80c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.604+0000 7fb2ee800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e815cab0 con 0x7fb2e8145300 2024-03-06T07:29:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.604+0000 7fb2ee800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e81456e0 con 0x7fb2e80cd060 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.604+0000 7fb2ee800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e815cc30 con 0x7fb2e8070f10 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 0x7fb2e80c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 0x7fb2e80c66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38482/0 (socket says 172.21.15.6:38482) 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 -- 172.21.15.6:0/3562924579 learned_addr learned my addr 172.21.15.6:0/3562924579 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2eca00640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2e80cd060 0x7fb2e80ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ed400640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2e8070f10 0x7fb2e81428e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 -- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2e8070f10 msgr2=0x7fb2e81428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2e8070f10 0x7fb2e81428e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 -- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2e80cd060 msgr2=0x7fb2e80ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2e80cd060 0x7fb2e80ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.605+0000 7fb2ede00640 1 -- 172.21.15.6:0/3562924579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8142570 con 0x7fb2e8145300 2024-03-06T07:29:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.606+0000 7fb2ede00640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 0x7fb2e80c66c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0034040 tx=0x7fb2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.606+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0032630 con 0x7fb2e8145300 2024-03-06T07:29:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.606+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e00327d0 con 0x7fb2e8145300 2024-03-06T07:29:58.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.606+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0032b80 con 0x7fb2e8145300 2024-03-06T07:29:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.606+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e8142e20 con 0x7fb2e8145300 2024-03-06T07:29:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.606+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e80c51c0 con 0x7fb2e8145300 2024-03-06T07:29:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.607+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e80c5bb0 con 0x7fb2e8145300 2024-03-06T07:29:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.608+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2e004c020 con 0x7fb2e8145300 2024-03-06T07:29:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.609+0000 7fb2e5600640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.609+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2e002f020 con 0x7fb2e8145300 2024-03-06T07:29:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.610+0000 7fb2ed400640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.611+0000 7fb2ed400640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb2c806b4d0 0x7fb2c806d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc003fd0 tx=0x7fb2dc003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.613+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2e8142570 con 0x7fb2e8145300 2024-03-06T07:29:58.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.854+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2e80721e0 con 0x7fb2e8145300 2024-03-06T07:29:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.855+0000 7fb2e5600640 1 -- 172.21.15.6:0/3562924579 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fb2e0040070 con 0x7fb2e8145300 2024-03-06T07:29:58.858 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:58.858 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:29:26.635978+0000","modified":"2024-03-06T07:29:29.733477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6837/590822743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":590822743},{"type":"v1","addr":"172.21.15.178:6837","nonce":590822743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5562,"qdb_cluster":[5562]},"id":3}]} 2024-03-06T07:29:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 26 2024-03-06T07:29:58.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb2c806b4d0 msgr2=0x7fb2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb2c806b4d0 0x7fb2c806d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc003fd0 tx=0x7fb2dc003990 comp rx=0 tx=0).stop 2024-03-06T07:29:58.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 msgr2=0x7fb2e80c66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 0x7fb2e80c66c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0034040 tx=0x7fb2e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 shutdown_connections 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2e80cd060 0x7fb2e80ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb2e8145300 0x7fb2e80c66c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 --2- 172.21.15.6:0/3562924579 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2e8070f10 0x7fb2e81428e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 >> 172.21.15.6:0/3562924579 conn(0x7fb2e806b3d0 msgr2=0x7fb2e80cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.858+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 shutdown_connections 2024-03-06T07:29:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:58.859+0000 7fb2ee800640 1 -- 172.21.15.6:0/3562924579 wait complete. 2024-03-06T07:29:58.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:29:59.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.101+0000 7f842b200640 1 Processor -- start 2024-03-06T07:29:59.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.101+0000 7f842b200640 1 -- start start 2024-03-06T07:29:59.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.103+0000 7f842b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f842c0d2960 0x7f842c0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.103+0000 7f842b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c0d2dc0 con 0x7f842c0d0b00 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.103+0000 7f842b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c0d18c0 con 0x7f842c15e070 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.103+0000 7f842b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842c0d0f70 con 0x7f842c0d2960 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.103+0000 7f842a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f842c0d0b00 0x7f842c0b3630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51868/0 (socket says 172.21.15.6:51868) 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.103+0000 7f842a800640 1 -- 172.21.15.6:0/2428301257 learned_addr learned my addr 172.21.15.6:0/2428301257 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:59.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8421400640 1 --2- 172.21.15.6:0/2428301257 >> v2:172.21.15.178:3300/0 conn(0x7f842c0d2960 0x7f842c0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65590353 0 0) 0x7f842c0d2dc0 con 0x7f842c0d0b00 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8410003520 con 0x7f842c0d0b00 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725490337 0 0) 0x7f8410003520 con 0x7f842c0d0b00 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 >> v2:172.21.15.178:3300/0 conn(0x7f842c0d2960 msgr2=0x7f842c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 --2- 172.21.15.6:0/2428301257 >> v2:172.21.15.178:3300/0 conn(0x7f842c0d2960 0x7f842c0d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 >> v1:172.21.15.178:6789/0 conn(0x7f842c15e070 legacy=0x7f842c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0d1ba0 con 0x7f842c0d0b00 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8424001670 con 0x7f842c0d0b00 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 >> v1:172.21.15.6:6789/0 conn(0x7f842c0d0b00 legacy=0x7f842c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.104+0000 7f8429400640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8410003a00 0x7f8410003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429400640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8410004800 0x7f8410004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429400640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84100051c0 0x7f841000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0d0f70 con 0x7f8410003a00 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841000c0c0 con 0x7f8410004800 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841000c2c0 con 0x7f84100051c0 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429e00640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8410003a00 0x7f8410003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429e00640 1 -- 172.21.15.6:0/2428301257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84100051c0 msgr2=0x7f841000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429e00640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f84100051c0 0x7f841000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429e00640 1 -- 172.21.15.6:0/2428301257 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8410004800 msgr2=0x7f8410004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429e00640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8410004800 0x7f8410004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f8429e00640 1 -- 172.21.15.6:0/2428301257 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410003520 con 0x7f8410003a00 2024-03-06T07:29:59.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.105+0000 7f842a800640 1 -- 172.21.15.6:0/2428301257 reap_dead start 2024-03-06T07:29:59.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.106+0000 7f8429e00640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8410003a00 0x7f8410003e20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f841c001e50 tx=0x7f8410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d66f72010653a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c001a70 con 0x7f8410003a00 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841c003b70 con 0x7f8410003a00 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f8429400640 1 -- 172.21.15.6:0/2428301257 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841c0050d0 con 0x7f8410003a00 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f842b200640 1 -- 172.21.15.6:0/2428301257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8410003a00 msgr2=0x7f8410003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f842b200640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8410003a00 0x7f8410003e20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f841c001e50 tx=0x7f8410005830 comp rx=0 tx=0).stop 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f842b200640 1 -- 172.21.15.6:0/2428301257 shutdown_connections 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f842b200640 1 --2- 172.21.15.6:0/2428301257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8410003a00 0x7f8410003e20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f842b200640 1 -- 172.21.15.6:0/2428301257 >> 172.21.15.6:0/2428301257 conn(0x7f842c08d2a0 msgr2=0x7f842c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.107+0000 7f842b200640 1 -- 172.21.15.6:0/2428301257 shutdown_connections 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.108+0000 7f842b200640 1 -- 172.21.15.6:0/2428301257 wait complete. 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.109+0000 7f842b200640 1 Processor -- start 2024-03-06T07:29:59.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.109+0000 7f842b200640 1 -- start start 2024-03-06T07:29:59.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.110+0000 7f842b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8410003a00 0x7f842c15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.110+0000 7f842b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f842c0cdd20 0x7f842c0ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.110+0000 7f842b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 0x7f842c0b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.110+0000 7f842b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0d2440 con 0x7f842c154100 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.110+0000 7f842b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0d2160 con 0x7f842c0cdd20 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.110+0000 7f842b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c0d45b0 con 0x7f8410003a00 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 0x7f842c0b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 0x7f842c0b8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38492/0 (socket says 172.21.15.6:38492) 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 -- 172.21.15.6:0/2514442525 learned_addr learned my addr 172.21.15.6:0/2514442525 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f8429e00640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8410003a00 0x7f842c15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f8421400640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f842c0cdd20 0x7f842c0ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 -- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8410003a00 msgr2=0x7f842c15e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8410003a00 0x7f842c15e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 -- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f842c0cdd20 msgr2=0x7f842c0ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f842c0cdd20 0x7f842c0ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.111+0000 7f842a800640 1 -- 172.21.15.6:0/2514442525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0ce1a0 con 0x7f842c154100 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.112+0000 7f842a800640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 0x7f842c0b8d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f842400c390 tx=0x7f842400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.112+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424004bf0 con 0x7f842c154100 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.112+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424004d90 con 0x7f842c154100 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.112+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424007270 con 0x7f842c154100 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.112+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c0cd650 con 0x7f842c154100 2024-03-06T07:29:59.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.112+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c0b7110 con 0x7f842c154100 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.113+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c04f330 con 0x7f842c154100 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.114+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8424007410 con 0x7f842c154100 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.115+0000 7f8422a00640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f840806b4d0 0x7f840806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.115+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8424089610 con 0x7f842c154100 2024-03-06T07:29:59.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.116+0000 7f8429e00640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f840806b4d0 0x7f840806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.117+0000 7f8429e00640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f840806b4d0 0x7f840806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f841c034000 tx=0x7f841c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.120+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f842401b030 con 0x7f842c154100 2024-03-06T07:29:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.329+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f842c0ca810 con 0x7f842c154100 2024-03-06T07:29:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.332+0000 7f8422a00640 1 -- 172.21.15.6:0/2514442525 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16541 (secure 0 0 0) 0x7f842400f260 con 0x7f842c154100 2024-03-06T07:29:59.335 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:29:59.335 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":46,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:29:28.183720+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:29:24.525508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:29:25.659373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f840806b4d0 msgr2=0x7f840806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f840806b4d0 0x7f840806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f841c034000 tx=0x7f841c009030 comp rx=0 tx=0).stop 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 msgr2=0x7f842c0b8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 0x7f842c0b8d80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f842400c390 tx=0x7f842400c790 comp rx=0 tx=0).stop 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 shutdown_connections 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f840806b4d0 0x7f840806d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f842c0cdd20 0x7f842c0ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f842c154100 0x7f842c0b8d80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 --2- 172.21.15.6:0/2514442525 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8410003a00 0x7f842c15e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 >> 172.21.15.6:0/2514442525 conn(0x7f842c08d2a0 msgr2=0x7f842c15d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 shutdown_connections 2024-03-06T07:29:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.336+0000 7f842b200640 1 -- 172.21.15.6:0/2514442525 wait complete. 2024-03-06T07:29:59.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:29:59.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.579+0000 7f438aa00640 1 Processor -- start 2024-03-06T07:29:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.580+0000 7f438aa00640 1 -- start start 2024-03-06T07:29:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.581+0000 7f438aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43840afdf0 0x7f43840b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.581+0000 7f438aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f438406afb0 0x7f438406b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.581+0000 7f438aa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438406bec0 con 0x7f438415e200 2024-03-06T07:29:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.581+0000 7f438aa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43840b0670 con 0x7f438406afb0 2024-03-06T07:29:59.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.581+0000 7f438aa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43840ae380 con 0x7f43840afdf0 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.582+0000 7f4383e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f438415e200 0x7f43840af140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51884/0 (socket says 172.21.15.6:51884) 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.582+0000 7f4383e00640 1 -- 172.21.15.6:0/2510283356 learned_addr learned my addr 172.21.15.6:0/2510283356 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.582+0000 7f4388a00640 1 --2- 172.21.15.6:0/2510283356 >> v2:172.21.15.6:3300/0 conn(0x7f438406afb0 0x7f438406b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.582+0000 7f4383400640 1 --2- 172.21.15.6:0/2510283356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43840afdf0 0x7f43840b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.582+0000 7f4382a00640 1 -- 172.21.15.6:0/2510283356 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370523833 0 0) 0x7f438406bec0 con 0x7f438415e200 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.582+0000 7f4382a00640 1 -- 172.21.15.6:0/2510283356 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4368003520 con 0x7f438415e200 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.583+0000 7f4388a00640 1 -- 172.21.15.6:0/2510283356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43840afdf0 msgr2=0x7f43840b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.583+0000 7f4388a00640 1 --2- 172.21.15.6:0/2510283356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43840afdf0 0x7f43840b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.583+0000 7f4382a00640 1 -- 172.21.15.6:0/2510283356 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1054974826 0 0) 0x7f4368003520 con 0x7f438415e200 2024-03-06T07:29:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.583+0000 7f4388a00640 1 -- 172.21.15.6:0/2510283356 >> v1:172.21.15.6:6789/0 conn(0x7f438415e200 legacy=0x7f43840af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.583+0000 7f4388a00640 1 -- 172.21.15.6:0/2510283356 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43840ae660 con 0x7f438406afb0 2024-03-06T07:29:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.584+0000 7f4388a00640 1 --2- 172.21.15.6:0/2510283356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438406afb0 0x7f438406b430 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f437c009d90 tx=0x7f437c02bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59a05606f494a93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.584+0000 7f4382a00640 1 -- 172.21.15.6:0/2510283356 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c039070 con 0x7f438406afb0 2024-03-06T07:29:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.584+0000 7f4382a00640 1 -- 172.21.15.6:0/2510283356 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c02c8a0 con 0x7f438406afb0 2024-03-06T07:29:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.584+0000 7f4382a00640 1 -- 172.21.15.6:0/2510283356 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374001690 con 0x7f438406afb0 2024-03-06T07:29:59.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 -- 172.21.15.6:0/2510283356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438406afb0 msgr2=0x7f438406b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 --2- 172.21.15.6:0/2510283356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438406afb0 0x7f438406b430 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f437c009d90 tx=0x7f437c02bd70 comp rx=0 tx=0).stop 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 -- 172.21.15.6:0/2510283356 shutdown_connections 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 --2- 172.21.15.6:0/2510283356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438406afb0 0x7f438406b430 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 --2- 172.21.15.6:0/2510283356 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43840afdf0 0x7f43840b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 -- 172.21.15.6:0/2510283356 >> 172.21.15.6:0/2510283356 conn(0x7f43840b2ae0 msgr2=0x7f43840b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 -- 172.21.15.6:0/2510283356 shutdown_connections 2024-03-06T07:29:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.585+0000 7f438aa00640 1 -- 172.21.15.6:0/2510283356 wait complete. 2024-03-06T07:29:59.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.587+0000 7f438aa00640 1 Processor -- start 2024-03-06T07:29:59.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.587+0000 7f438aa00640 1 -- start start 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f438aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f438406afb0 0x7f4384142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f438aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43840afdf0 0x7f43841592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f438aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 0x7f4384159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f438aa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438406dbb0 con 0x7f438415e200 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f438aa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43840aec20 con 0x7f43840afdf0 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f438aa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438406dd30 con 0x7f438406afb0 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f4388a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 0x7f4384159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f4388a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 0x7f4384159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38512/0 (socket says 172.21.15.6:38512) 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.588+0000 7f4388a00640 1 -- 172.21.15.6:0/3173073230 learned_addr learned my addr 172.21.15.6:0/3173073230 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4383e00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f438406afb0 0x7f4384142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4383400640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43840afdf0 0x7f43841592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4388a00640 1 -- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f438406afb0 msgr2=0x7f4384142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4388a00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f438406afb0 0x7f4384142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4388a00640 1 -- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43840afdf0 msgr2=0x7f43841592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4388a00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43840afdf0 0x7f43841592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4388a00640 1 -- 172.21.15.6:0/3173073230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438415b890 con 0x7f438415e200 2024-03-06T07:29:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.589+0000 7f4388a00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 0x7f4384159800 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f437c009eb0 tx=0x7f437c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.590+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c039050 con 0x7f438415e200 2024-03-06T07:29:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.590+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c045070 con 0x7f438415e200 2024-03-06T07:29:59.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.590+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c009450 con 0x7f438415e200 2024-03-06T07:29:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.590+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438406e390 con 0x7f438415e200 2024-03-06T07:29:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.590+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438415be10 con 0x7f438415e200 2024-03-06T07:29:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.591+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438404f3a0 con 0x7f438415e200 2024-03-06T07:29:59.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.592+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f437c040670 con 0x7f438415e200 2024-03-06T07:29:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.593+0000 7f437be00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.593+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f437c0aca70 con 0x7f438415e200 2024-03-06T07:29:59.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.594+0000 7f4383e00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.595+0000 7f4383e00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f436006b4d0 0x7f436006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4374001eb0 tx=0x7f4374001c60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.598+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f437c07ba10 con 0x7f438415e200 2024-03-06T07:29:59.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.840+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4384073050 con 0x7f438415e200 2024-03-06T07:29:59.843 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:29:59.842+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:29:59.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.845+0000 7f437be00640 1 -- 172.21.15.6:0/3173073230 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) v1 ==== 161+0+0 (secure 0 0 0) 0x7f437c07b310 con 0x7f438415e200 2024-03-06T07:29:59.848 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f436006b4d0 msgr2=0x7f436006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f436006b4d0 0x7f436006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4374001eb0 tx=0x7f4374001c60 comp rx=0 tx=0).stop 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 msgr2=0x7f4384159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 0x7f4384159800 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f437c009eb0 tx=0x7f437c004a40 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 shutdown_connections 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43840afdf0 0x7f43841592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f438415e200 0x7f4384159800 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 --2- 172.21.15.6:0/3173073230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f438406afb0 0x7f4384142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.848+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 >> 172.21.15.6:0/3173073230 conn(0x7f43840b2ae0 msgr2=0x7f4384156580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.849+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 shutdown_connections 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:29:59.849+0000 7f438aa00640 1 -- 172.21.15.6:0/3173073230 wait complete. 2024-03-06T07:29:59.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:29:59.964 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:30:00.001 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:29:59.999+0000 7f7245000640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-03-06T07:30:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.090+0000 7f569ce00640 1 Processor -- start 2024-03-06T07:30:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.090+0000 7f569ce00640 1 -- start start 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f569ce00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f569815bf20 0x7f56980acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f569ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5698071e20 0x7f56980ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f569ce00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5698072df0 con 0x7f5698072520 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f569ce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56980cdab0 con 0x7f5698071e20 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f569ce00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56980b4d70 con 0x7f569815bf20 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f5696800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5698071e20 0x7f56980ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f5696800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5698071e20 0x7f56980ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38520/0 (socket says 172.21.15.6:38520) 2024-03-06T07:30:00.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f5696800640 1 -- 172.21.15.6:0/1541883001 learned_addr learned my addr 172.21.15.6:0/1541883001 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:00.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f5696800640 1 -- 172.21.15.6:0/1541883001 >> v2:172.21.15.178:3300/0 conn(0x7f569815bf20 msgr2=0x7f56980acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.092+0000 7f5695e00640 1 --2- 172.21.15.6:0/1541883001 >> v2:172.21.15.178:3300/0 conn(0x7f569815bf20 0x7f56980acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.093+0000 7f5696800640 1 --2- 172.21.15.6:0/1541883001 >> v2:172.21.15.178:3300/0 conn(0x7f569815bf20 0x7f56980acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.093+0000 7f5696800640 1 -- 172.21.15.6:0/1541883001 >> v1:172.21.15.178:6789/0 conn(0x7f5698072520 legacy=0x7f56980cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:00.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.093+0000 7f5696800640 1 -- 172.21.15.6:0/1541883001 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56980b5050 con 0x7f5698071e20 2024-03-06T07:30:00.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.094+0000 7f5696800640 1 --2- 172.21.15.6:0/1541883001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5698071e20 0x7f56980ccad0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f569806ec40 tx=0x7f569002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4df1316ef7825a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.094+0000 7f5695400640 1 -- 172.21.15.6:0/1541883001 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569002ca80 con 0x7f5698071e20 2024-03-06T07:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.094+0000 7f5695400640 1 -- 172.21.15.6:0/1541883001 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569002cc20 con 0x7f5698071e20 2024-03-06T07:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.094+0000 7f5695400640 1 -- 172.21.15.6:0/1541883001 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56900343f0 con 0x7f5698071e20 2024-03-06T07:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.094+0000 7f569ce00640 1 -- 172.21.15.6:0/1541883001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5698071e20 msgr2=0x7f56980ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.094+0000 7f569ce00640 1 --2- 172.21.15.6:0/1541883001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5698071e20 0x7f56980ccad0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f569806ec40 tx=0x7f569002bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.095+0000 7f569ce00640 1 -- 172.21.15.6:0/1541883001 shutdown_connections 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.095+0000 7f569ce00640 1 --2- 172.21.15.6:0/1541883001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5698071e20 0x7f56980ccad0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.095+0000 7f569ce00640 1 --2- 172.21.15.6:0/1541883001 >> v2:172.21.15.178:3300/0 conn(0x7f569815bf20 0x7f56980acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.095+0000 7f569ce00640 1 -- 172.21.15.6:0/1541883001 >> 172.21.15.6:0/1541883001 conn(0x7f569806b3d0 msgr2=0x7f569806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.095+0000 7f569ce00640 1 -- 172.21.15.6:0/1541883001 shutdown_connections 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.095+0000 7f569ce00640 1 -- 172.21.15.6:0/1541883001 wait complete. 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.096+0000 7f569ce00640 1 Processor -- start 2024-03-06T07:30:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.097+0000 7f569ce00640 1 -- start start 2024-03-06T07:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.097+0000 7f569ce00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5698071e20 0x7f5698149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.097+0000 7f569ce00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5698072520 0x7f5698155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f569ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 0x7f569806dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f569ce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980b4b10 con 0x7f569815bf20 2024-03-06T07:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f569ce00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980b4990 con 0x7f5698072520 2024-03-06T07:30:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f569ce00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56980ce6c0 con 0x7f5698071e20 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 0x7f569806dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 0x7f569806dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38534/0 (socket says 172.21.15.6:38534) 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 -- 172.21.15.6:0/4284477571 learned_addr learned my addr 172.21.15.6:0/4284477571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5695e00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5698071e20 0x7f5698149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f568fe00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5698072520 0x7f5698155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 -- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5698071e20 msgr2=0x7f5698149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5698071e20 0x7f5698149590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 -- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5698072520 msgr2=0x7f5698155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5698072520 0x7f5698155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f5696800640 1 -- 172.21.15.6:0/4284477571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5698156040 con 0x7f569815bf20 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.098+0000 7f568fe00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5698072520 0x7f5698155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.099+0000 7f5696800640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 0x7f569806dd60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5690009f40 tx=0x7f5690004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.099+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690043020 con 0x7f569815bf20 2024-03-06T07:30:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.099+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569003e450 con 0x7f569815bf20 2024-03-06T07:30:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.099+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5698155a80 con 0x7f569815bf20 2024-03-06T07:30:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.099+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569003e820 con 0x7f569815bf20 2024-03-06T07:30:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.099+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56980c9650 con 0x7f569815bf20 2024-03-06T07:30:00.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.100+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569804f330 con 0x7f569815bf20 2024-03-06T07:30:00.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.102+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5690046020 con 0x7f569815bf20 2024-03-06T07:30:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.103+0000 7f568e000640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f567406b4d0 0x7f567406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.103+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f56900ad450 con 0x7f569815bf20 2024-03-06T07:30:00.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.103+0000 7f5695e00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f567406b4d0 0x7f567406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.104+0000 7f5695e00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f567406b4d0 0x7f567406d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5688006fd0 tx=0x7f5688008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.106+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f569004d030 con 0x7f569815bf20 2024-03-06T07:30:00.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.352+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f569815f720 con 0x7f569815bf20 2024-03-06T07:30:00.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.849+0000 7f568e000640 1 -- 172.21.15.6:0/4284477571 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 104+0+0 (secure 0 0 0) 0x7f569007c370 con 0x7f569815bf20 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.852+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f567406b4d0 msgr2=0x7f567406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.852+0000 7f569ce00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f567406b4d0 0x7f567406d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5688006fd0 tx=0x7f5688008040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.852+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 msgr2=0x7f569806dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.852+0000 7f569ce00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 0x7f569806dd60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5690009f40 tx=0x7f5690004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 shutdown_connections 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f567406b4d0 0x7f567406d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5698072520 0x7f5698155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f569815bf20 0x7f569806dd60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 --2- 172.21.15.6:0/4284477571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5698071e20 0x7f5698149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 >> 172.21.15.6:0/4284477571 conn(0x7f569806b3d0 msgr2=0x7f569815a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 shutdown_connections 2024-03-06T07:30:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:00.853+0000 7f569ce00640 1 -- 172.21.15.6:0/4284477571 wait complete. 2024-03-06T07:30:00.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:30:01.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.106+0000 7f8104e00640 1 Processor -- start 2024-03-06T07:30:01.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.106+0000 7f8104e00640 1 -- start start 2024-03-06T07:30:01.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.107+0000 7f8104e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 0x7f81000d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f8104e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000d1e40 con 0x7f81000d23e0 2024-03-06T07:30:01.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f8104e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81000d2100 con 0x7f810015e180 2024-03-06T07:30:01.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f8104e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81000d1be0 con 0x7f81000d4610 2024-03-06T07:30:01.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f80fe800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81000d23e0 0x7f81000d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51898/0 (socket says 172.21.15.6:51898) 2024-03-06T07:30:01.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f80fe800640 1 -- 172.21.15.6:0/2506868744 learned_addr learned my addr 172.21.15.6:0/2506868744 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f80f5400640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 0x7f81000d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.108+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270261276 0 0) 0x7f81000d1e40 con 0x7f81000d23e0 2024-03-06T07:30:01.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e4003520 con 0x7f81000d23e0 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494168974 0 0) 0x7f80e4003520 con 0x7f81000d23e0 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 msgr2=0x7f81000d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 0x7f81000d4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 >> v1:172.21.15.178:6789/0 conn(0x7f810015e180 legacy=0x7f81000b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000ceed0 con 0x7f81000d23e0 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f80f8001670 con 0x7f81000d23e0 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 >> v1:172.21.15.6:6789/0 conn(0x7f81000d23e0 legacy=0x7f81000d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 0x7f80e40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80e4003fb0 0x7f80e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80e4004d40 0x7f80e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81000d1be0 con 0x7f80e4004d40 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e400bb00 con 0x7f80e4003fb0 2024-03-06T07:30:01.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.109+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e400bd00 con 0x7f81000d4610 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fe800640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80e4004d40 0x7f80e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fde00640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 0x7f80e40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80f5400640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80e4003fb0 0x7f80e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fe800640 1 -- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 msgr2=0x7f80e40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fe800640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81000d4610 0x7f80e40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fe800640 1 -- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80e4003fb0 msgr2=0x7f80e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fe800640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f80e4003fb0 0x7f80e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.110+0000 7f80fe800640 1 -- 172.21.15.6:0/2506868744 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e4003520 con 0x7f80e4004d40 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.111+0000 7f80fe800640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80e4004d40 0x7f80e40051c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c190 tx=0x7f80e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85b140cc599f196 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8007c60 con 0x7f80e4004d40 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8030070 con 0x7f80e4004d40 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f80fd400640 1 -- 172.21.15.6:0/2506868744 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8005db0 con 0x7f80e4004d40 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f8104e00640 1 -- 172.21.15.6:0/2506868744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80e4004d40 msgr2=0x7f80e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f8104e00640 1 --2- 172.21.15.6:0/2506868744 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80e4004d40 0x7f80e40051c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c190 tx=0x7f80e4005830 comp rx=0 tx=0).stop 2024-03-06T07:30:01.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f80fe800640 1 -- 172.21.15.6:0/2506868744 reap_dead start 2024-03-06T07:30:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f8104e00640 1 -- 172.21.15.6:0/2506868744 shutdown_connections 2024-03-06T07:30:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.112+0000 7f8104e00640 1 -- 172.21.15.6:0/2506868744 >> 172.21.15.6:0/2506868744 conn(0x7f810009d2b0 msgr2=0x7f81000d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.113+0000 7f8104e00640 1 -- 172.21.15.6:0/2506868744 shutdown_connections 2024-03-06T07:30:01.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.113+0000 7f8104e00640 1 -- 172.21.15.6:0/2506868744 wait complete. 2024-03-06T07:30:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.115+0000 7f8104e00640 1 Processor -- start 2024-03-06T07:30:01.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.115+0000 7f8104e00640 1 -- start start 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f8104e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80e4004d40 0x7f8100158fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f8104e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 0x7f8100158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f8104e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81000cd850 0x7f81000cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f8104e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81000b32a0 con 0x7f81000c75b0 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f8104e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81000d4a30 con 0x7f81000cd850 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f8104e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810015a030 con 0x7f80e4004d40 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 0x7f8100158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 0x7f8100158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38548/0 (socket says 172.21.15.6:38548) 2024-03-06T07:30:01.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 -- 172.21.15.6:0/3220296955 learned_addr learned my addr 172.21.15.6:0/3220296955 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80fde00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80e4004d40 0x7f8100158fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80fe800640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81000cd850 0x7f81000cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 -- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80e4004d40 msgr2=0x7f8100158fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80e4004d40 0x7f8100158fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 -- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81000cd850 msgr2=0x7f81000cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81000cd850 0x7f81000cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.117+0000 7f80f5400640 1 -- 172.21.15.6:0/3220296955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810015a310 con 0x7f81000c75b0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.118+0000 7f80f5400640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 0x7f8100158770 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f80e80115d0 tx=0x7f80e8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.118+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8015870 con 0x7f81000c75b0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.118+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e8015a10 con 0x7f81000c75b0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.118+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8015e00 con 0x7f81000c75b0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.118+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810015a5f0 con 0x7f81000c75b0 2024-03-06T07:30:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.118+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810015abb0 con 0x7f81000c75b0 2024-03-06T07:30:01.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.119+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8100068010 con 0x7f81000c75b0 2024-03-06T07:30:01.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.120+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80e800f480 con 0x7f81000c75b0 2024-03-06T07:30:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.121+0000 7f80f6a00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80dc06b4d0 0x7f80dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.121+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f80e808ac60 con 0x7f81000c75b0 2024-03-06T07:30:01.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.122+0000 7f80fde00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80dc06b4d0 0x7f80dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.123+0000 7f80fde00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80dc06b4d0 0x7f80dc06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f81000a1e60 tx=0x7f80f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.126+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80e8019030 con 0x7f81000c75b0 2024-03-06T07:30:01.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.328+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f81000d1580 con 0x7f81000c75b0 2024-03-06T07:30:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.853+0000 7f80f6a00640 1 -- 172.21.15.6:0/3220296955 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) v1 ==== 181+0+0 (secure 0 0 0) 0x7f80e80599e0 con 0x7f81000c75b0 2024-03-06T07:30:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:30:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.857+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80dc06b4d0 msgr2=0x7f80dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.857+0000 7f8104e00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80dc06b4d0 0x7f80dc06d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f81000a1e60 tx=0x7f80f0009040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.857+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 msgr2=0x7f8100158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.857+0000 7f8104e00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 0x7f8100158770 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f80e80115d0 tx=0x7f80e8011a90 comp rx=0 tx=0).stop 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 shutdown_connections 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f80dc06b4d0 0x7f80dc06d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81000cd850 0x7f81000cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81000c75b0 0x7f8100158770 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 --2- 172.21.15.6:0/3220296955 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80e4004d40 0x7f8100158fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 >> 172.21.15.6:0/3220296955 conn(0x7f810009d2b0 msgr2=0x7f81000b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 shutdown_connections 2024-03-06T07:30:01.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:01.860+0000 7f8104e00640 1 -- 172.21.15.6:0/3220296955 wait complete. 2024-03-06T07:30:01.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:30:02.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.102+0000 7f2148a00640 1 Processor -- start 2024-03-06T07:30:02.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.102+0000 7f2148a00640 1 -- start start 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.103+0000 7f2148a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 0x7f21440cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.103+0000 7f2148a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f21440d1ef0 0x7f214406d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.103+0000 7f2148a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214406ef30 con 0x7f21440d1a70 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.103+0000 7f2148a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214406f510 con 0x7f21440d1ef0 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.103+0000 7f2148a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440d2380 con 0x7f214415e210 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.104+0000 7f2141400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f21440d1a70 0x7f214406c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60480/0 (socket says 172.21.15.6:60480) 2024-03-06T07:30:02.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.104+0000 7f2141400640 1 -- 172.21.15.6:0/3528006367 learned_addr learned my addr 172.21.15.6:0/3528006367 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.104+0000 7f2142800640 1 --2- 172.21.15.6:0/3528006367 >> v2:172.21.15.178:3300/0 conn(0x7f21440d1ef0 0x7f214406d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.104+0000 7f2141e00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 0x7f21440cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.104+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407546162 0 0) 0x7f214406ef30 con 0x7f21440d1a70 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.104+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2128003520 con 0x7f21440d1a70 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2251908554 0 0) 0x7f2128003520 con 0x7f21440d1a70 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 msgr2=0x7f21440cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 0x7f21440cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 >> v2:172.21.15.178:3300/0 conn(0x7f21440d1ef0 msgr2=0x7f214406d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 --2- 172.21.15.6:0/3528006367 >> v2:172.21.15.178:3300/0 conn(0x7f21440d1ef0 0x7f214406d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214406fc60 con 0x7f21440d1a70 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f212c001670 con 0x7f21440d1a70 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 >> v1:172.21.15.6:6789/0 conn(0x7f21440d1a70 legacy=0x7f214406c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21280039b0 0x7f2128003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2128004800 0x7f2128004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 0x7f21280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214406f510 con 0x7f21280039b0 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440d2380 con 0x7f2128004800 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212800b9c0 con 0x7f214415e210 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.105+0000 7f2141e00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21280039b0 0x7f2128003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.106+0000 7f2141e00640 1 -- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 msgr2=0x7f21280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.106+0000 7f2141e00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f214415e210 0x7f21280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.106+0000 7f2141e00640 1 -- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2128004800 msgr2=0x7f2128004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.106+0000 7f2141400640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2128004800 0x7f2128004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.106+0000 7f2141e00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2128004800 0x7f2128004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.106+0000 7f2141e00640 1 -- 172.21.15.6:0/3528006367 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2128003520 con 0x7f21280039b0 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.107+0000 7f2141e00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21280039b0 0x7f2128003dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2134009fc0 tx=0x7f2128005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49232e7a254532 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.107+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134039070 con 0x7f21280039b0 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.107+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213402cc60 con 0x7f21280039b0 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.107+0000 7f2140a00640 1 -- 172.21.15.6:0/3528006367 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134009c10 con 0x7f21280039b0 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2148a00640 1 -- 172.21.15.6:0/3528006367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21280039b0 msgr2=0x7f2128003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2148a00640 1 --2- 172.21.15.6:0/3528006367 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f21280039b0 0x7f2128003dd0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2134009fc0 tx=0x7f2128005830 comp rx=0 tx=0).stop 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2142800640 1 -- 172.21.15.6:0/3528006367 reap_dead start 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2148a00640 1 -- 172.21.15.6:0/3528006367 shutdown_connections 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2148a00640 1 -- 172.21.15.6:0/3528006367 >> 172.21.15.6:0/3528006367 conn(0x7f214409aa10 msgr2=0x7f214409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2148a00640 1 -- 172.21.15.6:0/3528006367 shutdown_connections 2024-03-06T07:30:02.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.108+0000 7f2148a00640 1 -- 172.21.15.6:0/3528006367 wait complete. 2024-03-06T07:30:02.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.110+0000 7f2148a00640 1 Processor -- start 2024-03-06T07:30:02.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.110+0000 7f2148a00640 1 -- start start 2024-03-06T07:30:02.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2148a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21280039b0 0x7f21440b7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2148a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 0x7f2144160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2148a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21440b0240 0x7f21440b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2148a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440700c0 con 0x7f214415e5f0 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2148a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214406ff40 con 0x7f21440b0240 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2148a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214406f860 con 0x7f21280039b0 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2141400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 0x7f2144160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2141400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 0x7f2144160820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52220/0 (socket says 172.21.15.6:52220) 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2141400640 1 -- 172.21.15.6:0/235217554 learned_addr learned my addr 172.21.15.6:0/235217554 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2141400640 1 -- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21280039b0 msgr2=0x7f21440b7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2141e00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21280039b0 0x7f21440b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.111+0000 7f2141400640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21280039b0 0x7f21440b7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f2141400640 1 -- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21440b0240 msgr2=0x7f21440b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f2141400640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21440b0240 0x7f21440b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f2141400640 1 -- 172.21.15.6:0/235217554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214415eb60 con 0x7f214415e5f0 2024-03-06T07:30:02.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f2141400640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 0x7f2144160820 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f212c00d7e0 tx=0x7f212c00dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c019070 con 0x7f214415e5f0 2024-03-06T07:30:02.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c0074e0 con 0x7f214415e5f0 2024-03-06T07:30:02.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c004a00 con 0x7f214415e5f0 2024-03-06T07:30:02.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214415ee40 con 0x7f214415e5f0 2024-03-06T07:30:02.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.112+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214415f400 con 0x7f214415e5f0 2024-03-06T07:30:02.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.113+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2144068720 con 0x7f214415e5f0 2024-03-06T07:30:02.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.114+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f212c01c070 con 0x7f214415e5f0 2024-03-06T07:30:02.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.115+0000 7f213a000640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f212006b4d0 0x7f212006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.115+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f212c089b50 con 0x7f214415e5f0 2024-03-06T07:30:02.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.116+0000 7f2141e00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f212006b4d0 0x7f212006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.117+0000 7f2141e00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f212006b4d0 0x7f212006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2134036000 tx=0x7f213402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.120+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f212c058a90 con 0x7f214415e5f0 2024-03-06T07:30:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.326+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f214406c5d0 con 0x7f214415e5f0 2024-03-06T07:30:02.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.864+0000 7f213a000640 1 -- 172.21.15.6:0/235217554 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) v1 ==== 169+0+0 (secure 0 0 0) 0x7f212c058390 con 0x7f214415e5f0 2024-03-06T07:30:02.867 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:30:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.871+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f212006b4d0 msgr2=0x7f212006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.871+0000 7f2148a00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f212006b4d0 0x7f212006d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2134036000 tx=0x7f213402e000 comp rx=0 tx=0).stop 2024-03-06T07:30:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.871+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 msgr2=0x7f2144160820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.871+0000 7f2148a00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 0x7f2144160820 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f212c00d7e0 tx=0x7f212c00dca0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.874+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 shutdown_connections 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.874+0000 7f2148a00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f212006b4d0 0x7f212006d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.874+0000 7f2148a00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f21440b0240 0x7f21440b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.874+0000 7f2148a00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f214415e5f0 0x7f2144160820 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.874+0000 7f2148a00640 1 --2- 172.21.15.6:0/235217554 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f21280039b0 0x7f21440b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.874+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 >> 172.21.15.6:0/235217554 conn(0x7f214409aa10 msgr2=0x7f21440b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.875+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 shutdown_connections 2024-03-06T07:30:02.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:02.875+0000 7f2148a00640 1 -- 172.21.15.6:0/235217554 wait complete. 2024-03-06T07:30:02.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:30:03.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.096+0000 7f488fc00640 1 Processor -- start 2024-03-06T07:30:03.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.096+0000 7f488fc00640 1 -- start start 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.097+0000 7f488fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f48880cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.097+0000 7f488fc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4888070ce0 0x7f48880ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.097+0000 7f488fc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4888071c60 con 0x7f48880cd010 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.097+0000 7f488fc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48880cfa10 con 0x7f4888070ce0 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.097+0000 7f488fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4888071f40 con 0x7f488815afa0 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.098+0000 7f488dc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f48880cd010 0x7f488806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60494/0 (socket says 172.21.15.6:60494) 2024-03-06T07:30:03.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.098+0000 7f488dc00640 1 -- 172.21.15.6:0/1184009461 learned_addr learned my addr 172.21.15.6:0/1184009461 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.098+0000 7f4887e00640 1 --2- 172.21.15.6:0/1184009461 >> v2:172.21.15.178:3300/0 conn(0x7f4888070ce0 0x7f48880ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.098+0000 7f488d200640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f48880cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.098+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134986356 0 0) 0x7f4888071c60 con 0x7f48880cd010 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.098+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f486c003520 con 0x7f48880cd010 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812711002 0 0) 0x7f486c003520 con 0x7f48880cd010 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 msgr2=0x7f48880cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f48880cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 >> v2:172.21.15.178:3300/0 conn(0x7f4888070ce0 msgr2=0x7f48880ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 --2- 172.21.15.6:0/1184009461 >> v2:172.21.15.178:3300/0 conn(0x7f4888070ce0 0x7f48880ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4888072220 con 0x7f48880cd010 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f4880001670 con 0x7f48880cd010 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 >> v1:172.21.15.6:6789/0 conn(0x7f48880cd010 legacy=0x7f488806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c0039b0 0x7f486c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f486c004800 0x7f486c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f486c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48880cfa10 con 0x7f486c004800 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888071f40 con 0x7f486c0039b0 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486c00ba10 con 0x7f488815afa0 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.099+0000 7f4887e00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f486c004800 0x7f486c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f488dc00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f486c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f488d200640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c0039b0 0x7f486c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f4887e00640 1 -- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 msgr2=0x7f486c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f4887e00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f486c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f4887e00640 1 -- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c0039b0 msgr2=0x7f486c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f4887e00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c0039b0 0x7f486c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f4887e00640 1 -- 172.21.15.6:0/1184009461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c003520 con 0x7f486c004800 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f488dc00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f488815afa0 0x7f486c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.100+0000 7f488d200640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c0039b0 0x7f486c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.101+0000 7f4887e00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f486c004800 0x7f486c004c80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4874009650 tx=0x7f486c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628d75365fe030f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.101+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487403b070 con 0x7f486c004800 2024-03-06T07:30:03.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.101+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487402e920 con 0x7f486c004800 2024-03-06T07:30:03.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.101+0000 7f4887400640 1 -- 172.21.15.6:0/1184009461 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874009690 con 0x7f486c004800 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488fc00640 1 -- 172.21.15.6:0/1184009461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f486c004800 msgr2=0x7f486c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488fc00640 1 --2- 172.21.15.6:0/1184009461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f486c004800 0x7f486c004c80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4874009650 tx=0x7f486c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488dc00640 1 -- 172.21.15.6:0/1184009461 reap_dead start 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488fc00640 1 -- 172.21.15.6:0/1184009461 shutdown_connections 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488fc00640 1 -- 172.21.15.6:0/1184009461 >> 172.21.15.6:0/1184009461 conn(0x7f488806b3d0 msgr2=0x7f488806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488fc00640 1 -- 172.21.15.6:0/1184009461 shutdown_connections 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.102+0000 7f488fc00640 1 -- 172.21.15.6:0/1184009461 wait complete. 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.104+0000 7f488fc00640 1 Processor -- start 2024-03-06T07:30:03.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.104+0000 7f488fc00640 1 -- start start 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488fc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c004800 0x7f4888146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f48880b0620 0x7f4888149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 0x7f4888149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48880b4ad0 con 0x7f48880b0aa0 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488fc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888071140 con 0x7f486c004800 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48880b4c50 con 0x7f48880b0620 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 0x7f4888149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 0x7f4888149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52244/0 (socket says 172.21.15.6:52244) 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.105+0000 7f488dc00640 1 -- 172.21.15.6:0/4160454098 learned_addr learned my addr 172.21.15.6:0/4160454098 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f4887e00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f48880b0620 0x7f4888149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488d200640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c004800 0x7f4888146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488dc00640 1 -- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f48880b0620 msgr2=0x7f4888149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488dc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f48880b0620 0x7f4888149040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488dc00640 1 -- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c004800 msgr2=0x7f4888146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488dc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c004800 0x7f4888146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488dc00640 1 -- 172.21.15.6:0/4160454098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48880b1070 con 0x7f48880b0aa0 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f4887e00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f48880b0620 0x7f4888149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488d200640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c004800 0x7f4888146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:03.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488dc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 0x7f4888149580 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f488000d540 tx=0x7f488000da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800074e0 con 0x7f48880b0aa0 2024-03-06T07:30:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48800047e0 con 0x7f48880b0aa0 2024-03-06T07:30:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880004bd0 con 0x7f48880b0aa0 2024-03-06T07:30:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.106+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48880b1350 con 0x7f48880b0aa0 2024-03-06T07:30:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.107+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48881473c0 con 0x7f48880b0aa0 2024-03-06T07:30:03.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.108+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488804f330 con 0x7f48880b0aa0 2024-03-06T07:30:03.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.109+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f488001b020 con 0x7f48880b0aa0 2024-03-06T07:30:03.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.110+0000 7f4884c00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f486806b4d0 0x7f486806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.110+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f48800898b0 con 0x7f48880b0aa0 2024-03-06T07:30:03.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.110+0000 7f488d200640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f486806b4d0 0x7f486806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.111+0000 7f488d200640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f486806b4d0 0x7f486806d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4878007440 tx=0x7f487800d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.114+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4880058a20 con 0x7f48880b0aa0 2024-03-06T07:30:03.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.316+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f4888147590 con 0x7f48880b0aa0 2024-03-06T07:30:03.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.323+0000 7f4884c00640 1 -- 172.21.15.6:0/4160454098 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v14) v1 ==== 108+0+0 (secure 0 0 0) 0x7f4880058320 con 0x7f48880b0aa0 2024-03-06T07:30:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.325+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f486806b4d0 msgr2=0x7f486806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.325+0000 7f488fc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f486806b4d0 0x7f486806d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4878007440 tx=0x7f487800d040 comp rx=0 tx=0).stop 2024-03-06T07:30:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 msgr2=0x7f4888149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 0x7f4888149580 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f488000d540 tx=0x7f488000da00 comp rx=0 tx=0).stop 2024-03-06T07:30:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 shutdown_connections 2024-03-06T07:30:03.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f486806b4d0 0x7f486806d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f48880b0aa0 0x7f4888149580 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f48880b0620 0x7f4888149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 --2- 172.21.15.6:0/4160454098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f486c004800 0x7f4888146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 >> 172.21.15.6:0/4160454098 conn(0x7f488806b3d0 msgr2=0x7f488806c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 shutdown_connections 2024-03-06T07:30:03.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.326+0000 7f488fc00640 1 -- 172.21.15.6:0/4160454098 wait complete. 2024-03-06T07:30:03.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:30:03.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.562+0000 7f7385400640 1 Processor -- start 2024-03-06T07:30:03.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.563+0000 7f7385400640 1 -- start start 2024-03-06T07:30:03.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.564+0000 7f7385400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 0x7f73800b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.564+0000 7f7385400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7380071930 con 0x7f73800729d0 2024-03-06T07:30:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.564+0000 7f7385400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7380071c10 con 0x7f73800b3c20 2024-03-06T07:30:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.564+0000 7f7385400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738015bfb0 con 0x7f73800b2250 2024-03-06T07:30:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f7377e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73800729d0 0x7f73800b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60504/0 (socket says 172.21.15.6:60504) 2024-03-06T07:30:03.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f7377e00640 1 -- 172.21.15.6:0/1149121864 learned_addr learned my addr 172.21.15.6:0/1149121864 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f7384a00640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 0x7f73800b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595117698 0 0) 0x7f7380071930 con 0x7f73800729d0 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7364003520 con 0x7f73800729d0 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1887912065 0 0) 0x7f7364003520 con 0x7f73800729d0 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 msgr2=0x7f73800b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 0x7f73800b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 >> v1:172.21.15.178:6789/0 conn(0x7f73800b3c20 legacy=0x7f738015b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380071ef0 con 0x7f73800729d0 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.565+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7368001670 con 0x7f73800729d0 2024-03-06T07:30:03.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 >> v1:172.21.15.6:6789/0 conn(0x7f73800729d0 legacy=0x7f73800b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f7364003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 0x7f7364004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364004d40 0x7f73640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738015bfb0 con 0x7f73640039e0 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736400ba70 con 0x7f7364004d40 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736400de80 con 0x7f73800b2250 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737fe00640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f7364003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f7377e00640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 0x7f7364004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737fe00640 1 -- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 msgr2=0x7f7364004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737fe00640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800b2250 0x7f7364004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737fe00640 1 -- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364004d40 msgr2=0x7f73640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737fe00640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7364004d40 0x7f73640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.566+0000 7f737fe00640 1 -- 172.21.15.6:0/1149121864 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364003520 con 0x7f73640039e0 2024-03-06T07:30:03.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.567+0000 7f737fe00640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f7364003e20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7370001720 tx=0x7f7364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64fdd5a095b291bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370001370 con 0x7f73640039e0 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73700052d0 con 0x7f73640039e0 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f737f400640 1 -- 172.21.15.6:0/1149121864 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370003830 con 0x7f73640039e0 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f7385400640 1 -- 172.21.15.6:0/1149121864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 msgr2=0x7f7364003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f7385400640 1 --2- 172.21.15.6:0/1149121864 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f7364003e20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7370001720 tx=0x7f7364005830 comp rx=0 tx=0).stop 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f7384a00640 1 -- 172.21.15.6:0/1149121864 reap_dead start 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f7385400640 1 -- 172.21.15.6:0/1149121864 shutdown_connections 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.568+0000 7f7385400640 1 -- 172.21.15.6:0/1149121864 >> 172.21.15.6:0/1149121864 conn(0x7f738006b3a0 msgr2=0x7f738006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.569+0000 7f7385400640 1 -- 172.21.15.6:0/1149121864 shutdown_connections 2024-03-06T07:30:03.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.569+0000 7f7385400640 1 -- 172.21.15.6:0/1149121864 wait complete. 2024-03-06T07:30:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.570+0000 7f7385400640 1 Processor -- start 2024-03-06T07:30:03.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.570+0000 7f7385400640 1 -- start start 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.571+0000 7f7385400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f73801444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.571+0000 7f7385400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73801478d0 0x7f73801e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.571+0000 7f7385400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800ce280 0x7f73800ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.571+0000 7f7385400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b4000 con 0x7f73640039e0 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.571+0000 7f7385400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800724b0 con 0x7f73801478d0 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.571+0000 7f7385400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b4180 con 0x7f73800ce280 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f73801444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f73801444b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52264/0 (socket says 172.21.15.6:52264) 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 -- 172.21.15.6:0/4246474142 learned_addr learned my addr 172.21.15.6:0/4246474142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 -- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800ce280 msgr2=0x7f73800ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800ce280 0x7f73800ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 -- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73801478d0 msgr2=0x7f73801e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73801478d0 0x7f73801e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 -- 172.21.15.6:0/4246474142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73801e45d0 con 0x7f73640039e0 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.572+0000 7f737fe00640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f73801444b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7370001840 tx=0x7f7370001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.573+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370041070 con 0x7f73640039e0 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.573+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7370005480 con 0x7f73640039e0 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.573+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370007a30 con 0x7f73640039e0 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.573+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7380156a50 con 0x7f73640039e0 2024-03-06T07:30:03.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.573+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73801e48b0 con 0x7f73640039e0 2024-03-06T07:30:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.574+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738006bce0 con 0x7f73640039e0 2024-03-06T07:30:03.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.575+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7370044070 con 0x7f73640039e0 2024-03-06T07:30:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.576+0000 7f737cc00640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f735c06b4d0 0x7f735c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.576+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f73700aba90 con 0x7f73640039e0 2024-03-06T07:30:03.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.576+0000 7f7377e00640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f735c06b4d0 0x7f735c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:03.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.577+0000 7f7377e00640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f735c06b4d0 0x7f735c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7368001200 tx=0x7f7368000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:03.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.580+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f738006bce0 con 0x7f73640039e0 2024-03-06T07:30:03.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.780+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f738004f330 con 0x7f73640039e0 2024-03-06T07:30:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.786+0000 7f737cc00640 1 -- 172.21.15.6:0/4246474142 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f737007ab70 con 0x7f73640039e0 2024-03-06T07:30:03.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f735c06b4d0 msgr2=0x7f735c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f735c06b4d0 0x7f735c06d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7368001200 tx=0x7f7368000ed0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 msgr2=0x7f73801444b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f73801444b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f7370001840 tx=0x7f7370001120 comp rx=0 tx=0).stop 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 shutdown_connections 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f735c06b4d0 0x7f735c06d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73801478d0 0x7f73801e3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73800ce280 0x7f73800ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 --2- 172.21.15.6:0/4246474142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73640039e0 0x7f73801444b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 >> 172.21.15.6:0/4246474142 conn(0x7f738006b3a0 msgr2=0x7f738006d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 shutdown_connections 2024-03-06T07:30:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:03.789+0000 7f7385400640 1 -- 172.21.15.6:0/4246474142 wait complete. 2024-03-06T07:30:03.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:30:04.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.034+0000 7fd18be00640 1 Processor -- start 2024-03-06T07:30:04.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.034+0000 7fd18be00640 1 -- start start 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.035+0000 7fd18be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15e2a0 0x7fd18c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.035+0000 7fd18be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd18c0cf460 0x7fd18c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.035+0000 7fd18be00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd18c0d1120 con 0x7fd18c15dec0 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.035+0000 7fd18be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18c0b0ea0 con 0x7fd18c0cf460 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.035+0000 7fd18be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0d1400 con 0x7fd18c15e2a0 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.035+0000 7fd18b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd18c0cf460 0x7fd18c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd18c0cf460 0x7fd18c0b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52272/0 (socket says 172.21.15.6:52272) 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18b400640 1 -- 172.21.15.6:0/506357439 learned_addr learned my addr 172.21.15.6:0/506357439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18a000640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15e2a0 0x7fd18c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18b400640 1 -- 172.21.15.6:0/506357439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15e2a0 msgr2=0x7fd18c0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18b400640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15e2a0 0x7fd18c0cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18b400640 1 -- 172.21.15.6:0/506357439 >> v1:172.21.15.178:6789/0 conn(0x7fd18c15dec0 legacy=0x7fd18c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18b400640 1 -- 172.21.15.6:0/506357439 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c0b0b30 con 0x7fd18c0cf460 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.036+0000 7fd18a000640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15e2a0 0x7fd18c0cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.037+0000 7fd18b400640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c0cf460 0x7fd18c0b1330 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd184009e50 tx=0x7fd18402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c046d375d2b1962e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd189600640 1 -- 172.21.15.6:0/506357439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18402cb00 con 0x7fd18c0cf460 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd189600640 1 -- 172.21.15.6:0/506357439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18402cca0 con 0x7fd18c0cf460 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd189600640 1 -- 172.21.15.6:0/506357439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184034610 con 0x7fd18c0cf460 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 -- 172.21.15.6:0/506357439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c0cf460 msgr2=0x7fd18c0b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c0cf460 0x7fd18c0b1330 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd184009e50 tx=0x7fd18402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 -- 172.21.15.6:0/506357439 shutdown_connections 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c0cf460 0x7fd18c0b1330 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 --2- 172.21.15.6:0/506357439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15e2a0 0x7fd18c0cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 -- 172.21.15.6:0/506357439 >> 172.21.15.6:0/506357439 conn(0x7fd18c0d3d40 msgr2=0x7fd18c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 -- 172.21.15.6:0/506357439 shutdown_connections 2024-03-06T07:30:04.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.038+0000 7fd18be00640 1 -- 172.21.15.6:0/506357439 wait complete. 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.040+0000 7fd18be00640 1 Processor -- start 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.040+0000 7fd18be00640 1 -- start start 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.041+0000 7fd18be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd18c0cf460 0x7fd18c0c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.041+0000 7fd18be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15dec0 0x7fd18c0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.041+0000 7fd18be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 0x7fd18c14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.041+0000 7fd18be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0d0ec0 con 0x7fd18c15e2a0 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.041+0000 7fd18be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0d0d40 con 0x7fd18c0cf460 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.041+0000 7fd18be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd18c0b1870 con 0x7fd18c15dec0 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 0x7fd18c14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 0x7fd18c14d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52288/0 (socket says 172.21.15.6:52288) 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 -- 172.21.15.6:0/709176925 learned_addr learned my addr 172.21.15.6:0/709176925 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18a000640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15dec0 0x7fd18c0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 -- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15dec0 msgr2=0x7fd18c0c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15dec0 0x7fd18c0c7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18aa00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd18c0cf460 0x7fd18c0c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 -- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd18c0cf460 msgr2=0x7fd18c0c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd18c0cf460 0x7fd18c0c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18b400640 1 -- 172.21.15.6:0/709176925 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c151120 con 0x7fd18c15e2a0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.042+0000 7fd18aa00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd18c0cf460 0x7fd18c0c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.043+0000 7fd18b400640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 0x7fd18c14d1f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd184036040 tx=0x7fd184004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.043+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184043020 con 0x7fd18c15e2a0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.043+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18402ce50 con 0x7fd18c15e2a0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.043+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18c0c9e60 con 0x7fd18c15e2a0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.043+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184009510 con 0x7fd18c15e2a0 2024-03-06T07:30:04.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.043+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c0ca420 con 0x7fd18c15e2a0 2024-03-06T07:30:04.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.044+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c04f330 con 0x7fd18c15e2a0 2024-03-06T07:30:04.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.045+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd184046020 con 0x7fd18c15e2a0 2024-03-06T07:30:04.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.046+0000 7fd183400640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd16c06b4d0 0x7fd16c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.047+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd1840ac6b0 con 0x7fd18c15e2a0 2024-03-06T07:30:04.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.047+0000 7fd18aa00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd16c06b4d0 0x7fd16c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.048+0000 7fd18aa00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd16c06b4d0 0x7fd16c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0cad50 tx=0x7fd17c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.051+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd18404d030 con 0x7fd18c15e2a0 2024-03-06T07:30:04.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:04.323+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7fd18c0b14d0 con 0x7fd18c15e2a0 2024-03-06T07:30:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.325+0000 7fd183400640 1 -- 172.21.15.6:0/709176925 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v162) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd18407b790 con 0x7fd18c15e2a0 2024-03-06T07:30:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd16c06b4d0 msgr2=0x7fd16c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd16c06b4d0 0x7fd16c06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0cad50 tx=0x7fd17c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 msgr2=0x7fd18c14d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 0x7fd18c14d1f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd184036040 tx=0x7fd184004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 shutdown_connections 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd16c06b4d0 0x7fd16c06d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd18c15e2a0 0x7fd18c14d1f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd18c15dec0 0x7fd18c0c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 --2- 172.21.15.6:0/709176925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd18c0cf460 0x7fd18c0c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 >> 172.21.15.6:0/709176925 conn(0x7fd18c0d3d40 msgr2=0x7fd18c0ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 shutdown_connections 2024-03-06T07:30:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.329+0000 7fd18be00640 1 -- 172.21.15.6:0/709176925 wait complete. 2024-03-06T07:30:05.351 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:30:05.352 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:30:05.352 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:30:05.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.574+0000 7f87f3c00640 1 Processor -- start 2024-03-06T07:30:05.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.574+0000 7f87f3c00640 1 -- start start 2024-03-06T07:30:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87f3c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0af1a0 0x7f87ec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87f3c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87ec0cdc20 0x7f87ec0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87f3c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec15cc40 con 0x7f87ec0af580 2024-03-06T07:30:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87f3c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec15cf20 con 0x7f87ec0cdc20 2024-03-06T07:30:05.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87f3c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec15d200 con 0x7f87ec0af1a0 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87f1c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87ec0cdc20 0x7f87ec0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87ebe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f87ec0af580 0x7f87ec0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60508/0 (socket says 172.21.15.6:60508) 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.576+0000 7f87ebe00640 1 -- 172.21.15.6:0/13686051 learned_addr learned my addr 172.21.15.6:0/13686051 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.577+0000 7f87f1200640 1 --2- 172.21.15.6:0/13686051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0af1a0 0x7f87ec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.577+0000 7f87f1c00640 1 -- 172.21.15.6:0/13686051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0af1a0 msgr2=0x7f87ec15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.577+0000 7f87f1c00640 1 --2- 172.21.15.6:0/13686051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0af1a0 0x7f87ec15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.577+0000 7f87f1c00640 1 -- 172.21.15.6:0/13686051 >> v1:172.21.15.6:6789/0 conn(0x7f87ec0af580 legacy=0x7f87ec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.577+0000 7f87eb400640 1 -- 172.21.15.6:0/13686051 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274252189 0 0) 0x7f87ec15cc40 con 0x7f87ec0af580 2024-03-06T07:30:05.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.577+0000 7f87f1c00640 1 -- 172.21.15.6:0/13686051 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec15d4e0 con 0x7f87ec0cdc20 2024-03-06T07:30:05.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.578+0000 7f87f1c00640 1 --2- 172.21.15.6:0/13686051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0cdc20 0x7f87ec0b2dd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009d90 tx=0x7f87e402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ef0ee783bfb73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87eb400640 1 -- 172.21.15.6:0/13686051 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4039070 con 0x7f87ec0cdc20 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87eb400640 1 -- 172.21.15.6:0/13686051 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e402cbc0 con 0x7f87ec0cdc20 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87eb400640 1 -- 172.21.15.6:0/13686051 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4034440 con 0x7f87ec0cdc20 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 -- 172.21.15.6:0/13686051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0cdc20 msgr2=0x7f87ec0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 --2- 172.21.15.6:0/13686051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0cdc20 0x7f87ec0b2dd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009d90 tx=0x7f87e402be10 comp rx=0 tx=0).stop 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 -- 172.21.15.6:0/13686051 shutdown_connections 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 --2- 172.21.15.6:0/13686051 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0cdc20 0x7f87ec0b2dd0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 --2- 172.21.15.6:0/13686051 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0af1a0 0x7f87ec15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 -- 172.21.15.6:0/13686051 >> 172.21.15.6:0/13686051 conn(0x7f87ec06afb0 msgr2=0x7f87ec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.579+0000 7f87f3c00640 1 -- 172.21.15.6:0/13686051 shutdown_connections 2024-03-06T07:30:05.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.580+0000 7f87f3c00640 1 -- 172.21.15.6:0/13686051 wait complete. 2024-03-06T07:30:05.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.581+0000 7f87f3c00640 1 Processor -- start 2024-03-06T07:30:05.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.581+0000 7f87f3c00640 1 -- start start 2024-03-06T07:30:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.582+0000 7f87f3c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87ec0af1a0 0x7f87ec0b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.582+0000 7f87f3c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 0x7f87ec1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.582+0000 7f87f3c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0cdc20 0x7f87ec09d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.582+0000 7f87f3c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0b36a0 con 0x7f87ec0af580 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.582+0000 7f87f3c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0b3520 con 0x7f87ec0af1a0 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.582+0000 7f87f3c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0b3820 con 0x7f87ec0cdc20 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 0x7f87ec1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 0x7f87ec1572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52306/0 (socket says 172.21.15.6:52306) 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 -- 172.21.15.6:0/3096243483 learned_addr learned my addr 172.21.15.6:0/3096243483 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87f1c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0cdc20 0x7f87ec09d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87f1200640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87ec0af1a0 0x7f87ec0b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 -- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0cdc20 msgr2=0x7f87ec09d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0cdc20 0x7f87ec09d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 -- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87ec0af1a0 msgr2=0x7f87ec0b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87ec0af1a0 0x7f87ec0b08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.583+0000 7f87ebe00640 1 -- 172.21.15.6:0/3096243483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec1577e0 con 0x7f87ec0af580 2024-03-06T07:30:05.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.584+0000 7f87ebe00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 0x7f87ec1572a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f87d800c4d0 tx=0x7f87d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.584+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8018070 con 0x7f87ec0af580 2024-03-06T07:30:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.584+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d8007500 con 0x7f87ec0af580 2024-03-06T07:30:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.584+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec09c080 con 0x7f87ec0af580 2024-03-06T07:30:05.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.584+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d8004a20 con 0x7f87ec0af580 2024-03-06T07:30:05.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.584+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec09c640 con 0x7f87ec0af580 2024-03-06T07:30:05.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.585+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec069510 con 0x7f87ec0af580 2024-03-06T07:30:05.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.586+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87d801b070 con 0x7f87ec0af580 2024-03-06T07:30:05.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.587+0000 7f87e8c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87cc06b4d0 0x7f87cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:05.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.587+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f87d80892b0 con 0x7f87ec0af580 2024-03-06T07:30:05.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.588+0000 7f87f1200640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87cc06b4d0 0x7f87cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:05.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.589+0000 7f87f1200640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87cc06b4d0 0x7f87cc06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f87e0006fd0 tx=0x7f87e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:05.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.592+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87d8058390 con 0x7f87ec0af580 2024-03-06T07:30:05.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.792+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87ec0ce100 con 0x7f87ec0af580 2024-03-06T07:30:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.794+0000 7f87e8c00640 1 -- 172.21.15.6:0/3096243483 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+12948 (secure 0 0 0) 0x7f87d8057c90 con 0x7f87ec0af580 2024-03-06T07:30:05.796 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:05.797 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":49,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:02.854355+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:6837/590822743":"2024-03-07T07:29:59.842564+0000","172.21.15.178:6835/590822743":"2024-03-07T07:29:59.842564+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87cc06b4d0 msgr2=0x7f87cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87cc06b4d0 0x7f87cc06d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f87e0006fd0 tx=0x7f87e0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 msgr2=0x7f87ec1572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 0x7f87ec1572a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f87d800c4d0 tx=0x7f87d800c990 comp rx=0 tx=0).stop 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 shutdown_connections 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87cc06b4d0 0x7f87cc06d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87ec0cdc20 0x7f87ec09d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87ec0af580 0x7f87ec1572a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 --2- 172.21.15.6:0/3096243483 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87ec0af1a0 0x7f87ec0b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 >> 172.21.15.6:0/3096243483 conn(0x7f87ec06afb0 msgr2=0x7f87ec15ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 shutdown_connections 2024-03-06T07:30:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:05.797+0000 7f87f3c00640 1 -- 172.21.15.6:0/3096243483 wait complete. 2024-03-06T07:30:05.821 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:05.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.045+0000 7f56d4000640 1 Processor -- start 2024-03-06T07:30:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.046+0000 7f56d4000640 1 -- start start 2024-03-06T07:30:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d4000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc15bf20 0x7f56cc0acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d4000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f56cc071e20 0x7f56cc0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d4000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc072df0 con 0x7f56cc072520 2024-03-06T07:30:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d4000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0cdab0 con 0x7f56cc071e20 2024-03-06T07:30:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d4000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b4d70 con 0x7f56cc15bf20 2024-03-06T07:30:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d0c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f56cc072520 0x7f56cc0cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60522/0 (socket says 172.21.15.6:60522) 2024-03-06T07:30:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.047+0000 7f56d0c00640 1 -- 172.21.15.6:0/1912200170 learned_addr learned my addr 172.21.15.6:0/1912200170 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56d2000640 1 --2- 172.21.15.6:0/1912200170 >> v2:172.21.15.178:3300/0 conn(0x7f56cc071e20 0x7f56cc0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56d1600640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc15bf20 0x7f56cc0acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859777459 0 0) 0x7f56cc072df0 con 0x7f56cc072520 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b0003520 con 0x7f56cc072520 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246787311 0 0) 0x7f56b0003520 con 0x7f56cc072520 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc15bf20 msgr2=0x7f56cc0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc15bf20 0x7f56cc0acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 >> v2:172.21.15.178:3300/0 conn(0x7f56cc071e20 msgr2=0x7f56cc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 --2- 172.21.15.6:0/1912200170 >> v2:172.21.15.178:3300/0 conn(0x7f56cc071e20 0x7f56cc0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0b5050 con 0x7f56cc072520 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f56b8001670 con 0x7f56cc072520 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 >> v1:172.21.15.6:6789/0 conn(0x7f56cc072520 legacy=0x7f56cc0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56b00039b0 0x7f56b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b0004800 0x7f56b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56b00051c0 0x7f56b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0cdab0 con 0x7f56b00039b0 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b4d70 con 0x7f56b0004800 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b000bf30 con 0x7f56b00051c0 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.048+0000 7f56d1600640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc15bf20 0x7f56cc0acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d1600640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56b00039b0 0x7f56b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d2000640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56b00051c0 0x7f56b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d1600640 1 -- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56b00051c0 msgr2=0x7f56b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d1600640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56b00051c0 0x7f56b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d1600640 1 -- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b0004800 msgr2=0x7f56b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d1600640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b0004800 0x7f56b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.049+0000 7f56d1600640 1 -- 172.21.15.6:0/1912200170 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b0003520 con 0x7f56b00039b0 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.050+0000 7f56d1600640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56b00039b0 0x7f56b0003dd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f56bc006fd0 tx=0x7f56b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27405a1d46eff9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.050+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc039070 con 0x7f56b00039b0 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.050+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56bc034680 con 0x7f56b00039b0 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.050+0000 7f56cbe00640 1 -- 172.21.15.6:0/1912200170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00f040 con 0x7f56b00039b0 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d4000640 1 -- 172.21.15.6:0/1912200170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56b00039b0 msgr2=0x7f56b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d4000640 1 --2- 172.21.15.6:0/1912200170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56b00039b0 0x7f56b0003dd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f56bc006fd0 tx=0x7f56b0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d2000640 1 -- 172.21.15.6:0/1912200170 reap_dead start 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d4000640 1 -- 172.21.15.6:0/1912200170 shutdown_connections 2024-03-06T07:30:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d4000640 1 -- 172.21.15.6:0/1912200170 >> 172.21.15.6:0/1912200170 conn(0x7f56cc06b3d0 msgr2=0x7f56cc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d4000640 1 -- 172.21.15.6:0/1912200170 shutdown_connections 2024-03-06T07:30:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.051+0000 7f56d4000640 1 -- 172.21.15.6:0/1912200170 wait complete. 2024-03-06T07:30:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.053+0000 7f56d4000640 1 Processor -- start 2024-03-06T07:30:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.053+0000 7f56d4000640 1 -- start start 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d4000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b00039b0 0x7f56cc154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d4000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 0x7f56cc0ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d4000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc0ae880 0x7f56cc152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d4000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b4b10 con 0x7f56cc1f44c0 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d4000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0b4990 con 0x7f56b00039b0 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d4000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc0ce6c0 con 0x7f56cc0ae880 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 0x7f56cc0ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 0x7f56cc0ae340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52328/0 (socket says 172.21.15.6:52328) 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.054+0000 7f56d0c00640 1 -- 172.21.15.6:0/2143605491 learned_addr learned my addr 172.21.15.6:0/2143605491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d1600640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b00039b0 0x7f56cc154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d2000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc0ae880 0x7f56cc152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d0c00640 1 -- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc0ae880 msgr2=0x7f56cc152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d0c00640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc0ae880 0x7f56cc152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d0c00640 1 -- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b00039b0 msgr2=0x7f56cc154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d0c00640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b00039b0 0x7f56cc154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d0c00640 1 -- 172.21.15.6:0/2143605491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc1f54c0 con 0x7f56cc1f44c0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56d0c00640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 0x7f56cc0ae340 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c700 tx=0x7f56b800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8018070 con 0x7f56cc1f44c0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.055+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b80074e0 con 0x7f56cc1f44c0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.056+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8004a00 con 0x7f56cc1f44c0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.056+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc0c9110 con 0x7f56cc1f44c0 2024-03-06T07:30:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.056+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc1f4f00 con 0x7f56cc1f44c0 2024-03-06T07:30:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.057+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc04f330 con 0x7f56cc1f44c0 2024-03-06T07:30:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.058+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56b801b070 con 0x7f56cc1f44c0 2024-03-06T07:30:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.059+0000 7f56c9600640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56ac06b500 0x7f56ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.059+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f56b80893a0 con 0x7f56cc1f44c0 2024-03-06T07:30:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.059+0000 7f56d1600640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56ac06b500 0x7f56ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.061+0000 7f56d1600640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56ac06b500 0x7f56ac06d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f56bc036000 tx=0x7f56bc031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.063+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56b80585b0 con 0x7f56cc1f44c0 2024-03-06T07:30:06.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.264+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f56cc15a8e0 con 0x7f56cc1f44c0 2024-03-06T07:30:06.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.264+0000 7f56c9600640 1 -- 172.21.15.6:0/2143605491 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f56b8057eb0 con 0x7f56cc1f44c0 2024-03-06T07:30:06.267 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:30:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.267+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56ac06b500 msgr2=0x7f56ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56ac06b500 0x7f56ac06d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f56bc036000 tx=0x7f56bc031000 comp rx=0 tx=0).stop 2024-03-06T07:30:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 msgr2=0x7f56cc0ae340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 0x7f56cc0ae340 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c700 tx=0x7f56b800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 shutdown_connections 2024-03-06T07:30:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56ac06b500 0x7f56ac06d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56cc0ae880 0x7f56cc152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56cc1f44c0 0x7f56cc0ae340 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 --2- 172.21.15.6:0/2143605491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56b00039b0 0x7f56cc154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 >> 172.21.15.6:0/2143605491 conn(0x7f56cc06b3d0 msgr2=0x7f56cc06e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 shutdown_connections 2024-03-06T07:30:06.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.268+0000 7f56d4000640 1 -- 172.21.15.6:0/2143605491 wait complete. 2024-03-06T07:30:06.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:30:06.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.506+0000 7fd03b200640 1 Processor -- start 2024-03-06T07:30:06.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.506+0000 7fd03b200640 1 -- start start 2024-03-06T07:30:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.507+0000 7fd03b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd03c15dec0 0x7fd03c0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.507+0000 7fd03b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd03c0cf460 0x7fd03c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd03b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c0d1120 con 0x7fd03c15e2a0 2024-03-06T07:30:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd03b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03c0b0ea0 con 0x7fd03c15dec0 2024-03-06T07:30:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd03b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03c0d1400 con 0x7fd03c0cf460 2024-03-06T07:30:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd039e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd03c15dec0 0x7fd03c0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd039e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd03c15dec0 0x7fd03c0ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52342/0 (socket says 172.21.15.6:52342) 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd039e00640 1 -- 172.21.15.6:0/2908444914 learned_addr learned my addr 172.21.15.6:0/2908444914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.508+0000 7fd03a800640 1 --2- 172.21.15.6:0/2908444914 >> v2:172.21.15.178:3300/0 conn(0x7fd03c0cf460 0x7fd03c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.509+0000 7fd039e00640 1 -- 172.21.15.6:0/2908444914 >> v2:172.21.15.178:3300/0 conn(0x7fd03c0cf460 msgr2=0x7fd03c0b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.509+0000 7fd039e00640 1 --2- 172.21.15.6:0/2908444914 >> v2:172.21.15.178:3300/0 conn(0x7fd03c0cf460 0x7fd03c0b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.509+0000 7fd039e00640 1 -- 172.21.15.6:0/2908444914 >> v1:172.21.15.178:6789/0 conn(0x7fd03c15e2a0 legacy=0x7fd03c0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.509+0000 7fd039e00640 1 -- 172.21.15.6:0/2908444914 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0b0b30 con 0x7fd03c15dec0 2024-03-06T07:30:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.509+0000 7fd03a800640 1 --2- 172.21.15.6:0/2908444914 >> v2:172.21.15.178:3300/0 conn(0x7fd03c0cf460 0x7fd03c0b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.510+0000 7fd039e00640 1 --2- 172.21.15.6:0/2908444914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c0ce810 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd03c08c570 tx=0x7fd02c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965e46233526d68c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.510+0000 7fd038a00640 1 -- 172.21.15.6:0/2908444914 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c034070 con 0x7fd03c15dec0 2024-03-06T07:30:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.510+0000 7fd038a00640 1 -- 172.21.15.6:0/2908444914 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c030860 con 0x7fd03c15dec0 2024-03-06T07:30:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.510+0000 7fd038a00640 1 -- 172.21.15.6:0/2908444914 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c030c30 con 0x7fd03c15dec0 2024-03-06T07:30:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 -- 172.21.15.6:0/2908444914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 msgr2=0x7fd03c0ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 --2- 172.21.15.6:0/2908444914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c0ce810 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd03c08c570 tx=0x7fd02c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 -- 172.21.15.6:0/2908444914 shutdown_connections 2024-03-06T07:30:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 --2- 172.21.15.6:0/2908444914 >> v2:172.21.15.178:3300/0 conn(0x7fd03c0cf460 0x7fd03c0b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 --2- 172.21.15.6:0/2908444914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c0ce810 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 -- 172.21.15.6:0/2908444914 >> 172.21.15.6:0/2908444914 conn(0x7fd03c0d3d40 msgr2=0x7fd03c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 -- 172.21.15.6:0/2908444914 shutdown_connections 2024-03-06T07:30:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.511+0000 7fd03b200640 1 -- 172.21.15.6:0/2908444914 wait complete. 2024-03-06T07:30:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.513+0000 7fd03b200640 1 Processor -- start 2024-03-06T07:30:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.513+0000 7fd03b200640 1 -- start start 2024-03-06T07:30:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd03b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd03c0cf460 0x7fd03c0b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd03b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd03b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd03c15e2a0 0x7fd03c0c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd03b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0d0ec0 con 0x7fd03c15dec0 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd03b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0d0d40 con 0x7fd03c0cf460 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd03b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c0b1870 con 0x7fd03c15e2a0 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd039400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd039400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52348/0 (socket says 172.21.15.6:52348) 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.514+0000 7fd039400640 1 -- 172.21.15.6:0/134560779 learned_addr learned my addr 172.21.15.6:0/134560779 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd03a800640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd03c15e2a0 0x7fd03c0c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039e00640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd03c0cf460 0x7fd03c0b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039400640 1 -- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd03c15e2a0 msgr2=0x7fd03c0c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039400640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd03c15e2a0 0x7fd03c0c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039400640 1 -- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd03c0cf460 msgr2=0x7fd03c0b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039400640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd03c0cf460 0x7fd03c0b6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039400640 1 -- 172.21.15.6:0/134560779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0c8d80 con 0x7fd03c15dec0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd03a800640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd03c15e2a0 0x7fd03c0c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd039400640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c1573a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd02400c400 tx=0x7fd02400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024017070 con 0x7fd03c15dec0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0240052d0 con 0x7fd03c15dec0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024003830 con 0x7fd03c15dec0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.515+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c0c9060 con 0x7fd03c15dec0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.516+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c0c9620 con 0x7fd03c15dec0 2024-03-06T07:30:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.517+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03c04f330 con 0x7fd03c15dec0 2024-03-06T07:30:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.518+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd02401a070 con 0x7fd03c15dec0 2024-03-06T07:30:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.518+0000 7fd032000640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd01806b4d0 0x7fd01806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.519+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd024089300 con 0x7fd03c15dec0 2024-03-06T07:30:06.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.519+0000 7fd039e00640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd01806b4d0 0x7fd01806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.520+0000 7fd039e00640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd01806b4d0 0x7fd01806d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd02c036000 tx=0x7fd02c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.523+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0240583e0 con 0x7fd03c15dec0 2024-03-06T07:30:06.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:06.795+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fd03c06bc40 con 0x7fd03c15dec0 2024-03-06T07:30:07.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.336+0000 7fd032000640 1 -- 172.21.15.6:0/134560779 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v164) v1 ==== 147+0+0 (secure 0 0 0) 0x7fd024057ce0 con 0x7fd03c15dec0 2024-03-06T07:30:07.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd01806b4d0 msgr2=0x7fd01806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd01806b4d0 0x7fd01806d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd02c036000 tx=0x7fd02c03b040 comp rx=0 tx=0).stop 2024-03-06T07:30:07.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 msgr2=0x7fd03c1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c1573a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd02400c400 tx=0x7fd02400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 shutdown_connections 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd01806b4d0 0x7fd01806d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd03c15e2a0 0x7fd03c0c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd03c15dec0 0x7fd03c1573a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 --2- 172.21.15.6:0/134560779 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd03c0cf460 0x7fd03c0b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 >> 172.21.15.6:0/134560779 conn(0x7fd03c0d3d40 msgr2=0x7fd03c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 shutdown_connections 2024-03-06T07:30:07.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.341+0000 7fd03b200640 1 -- 172.21.15.6:0/134560779 wait complete. 2024-03-06T07:30:07.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:30:07.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.593+0000 7fa3bf000640 1 Processor -- start 2024-03-06T07:30:07.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.593+0000 7fa3bf000640 1 -- start start 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bf000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3b80729d0 0x7fa3b80b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bf000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa3b80b1b40 0x7fa3b80b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bf000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b8071930 con 0x7fa3b80b3c20 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bf000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b80b24e0 con 0x7fa3b80729d0 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bf000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b815bfb0 con 0x7fa3b80b1b40 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bd200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3b80729d0 0x7fa3b80b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bd200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3b80729d0 0x7fa3b80b4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52364/0 (socket says 172.21.15.6:52364) 2024-03-06T07:30:07.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.595+0000 7fa3bd200640 1 -- 172.21.15.6:0/396663198 learned_addr learned my addr 172.21.15.6:0/396663198 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:07.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.596+0000 7fa3bd200640 1 -- 172.21.15.6:0/396663198 >> v2:172.21.15.178:3300/0 conn(0x7fa3b80b1b40 msgr2=0x7fa3b80b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.596+0000 7fa3bd200640 1 --2- 172.21.15.6:0/396663198 >> v2:172.21.15.178:3300/0 conn(0x7fa3b80b1b40 0x7fa3b80b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.596+0000 7fa3bd200640 1 -- 172.21.15.6:0/396663198 >> v1:172.21.15.178:6789/0 conn(0x7fa3b80b3c20 legacy=0x7fa3b815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.596+0000 7fa3bd200640 1 -- 172.21.15.6:0/396663198 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8071c10 con 0x7fa3b80729d0 2024-03-06T07:30:07.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.597+0000 7fa3bd200640 1 --2- 172.21.15.6:0/396663198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80729d0 0x7fa3b80b4000 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80af660 tx=0x7fa3ac02e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546627412b9ef8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.597+0000 7fa3b7e00640 1 -- 172.21.15.6:0/396663198 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac02ec90 con 0x7fa3b80729d0 2024-03-06T07:30:07.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.597+0000 7fa3b7e00640 1 -- 172.21.15.6:0/396663198 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac02ee30 con 0x7fa3b80729d0 2024-03-06T07:30:07.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.597+0000 7fa3b7e00640 1 -- 172.21.15.6:0/396663198 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0365f0 con 0x7fa3b80729d0 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 -- 172.21.15.6:0/396663198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80729d0 msgr2=0x7fa3b80b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 --2- 172.21.15.6:0/396663198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80729d0 0x7fa3b80b4000 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa3b80af660 tx=0x7fa3ac02e0f0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 -- 172.21.15.6:0/396663198 shutdown_connections 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 --2- 172.21.15.6:0/396663198 >> v2:172.21.15.178:3300/0 conn(0x7fa3b80b1b40 0x7fa3b80b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 --2- 172.21.15.6:0/396663198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80729d0 0x7fa3b80b4000 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 -- 172.21.15.6:0/396663198 >> 172.21.15.6:0/396663198 conn(0x7fa3b806b3a0 msgr2=0x7fa3b806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 -- 172.21.15.6:0/396663198 shutdown_connections 2024-03-06T07:30:07.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.598+0000 7fa3bf000640 1 -- 172.21.15.6:0/396663198 wait complete. 2024-03-06T07:30:07.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.600+0000 7fa3bf000640 1 Processor -- start 2024-03-06T07:30:07.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.600+0000 7fa3bf000640 1 -- start start 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bf000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3b80729d0 0x7fa3b80c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 0x7fa3b81f8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bf000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3b80b3c20 0x7fa3b8155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b80724b0 con 0x7fa3b80b1b40 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bf000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b80721d0 con 0x7fa3b80b3c20 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bf000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b815c220 con 0x7fa3b80729d0 2024-03-06T07:30:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 0x7fa3b81f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 0x7fa3b81f8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52378/0 (socket says 172.21.15.6:52378) 2024-03-06T07:30:07.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.601+0000 7fa3bd200640 1 -- 172.21.15.6:0/2182899128 learned_addr learned my addr 172.21.15.6:0/2182899128 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bd200640 1 -- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3b80729d0 msgr2=0x7fa3b80c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bd200640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3b80729d0 0x7fa3b80c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bd200640 1 -- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3b80b3c20 msgr2=0x7fa3b8155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3be600640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3b80b3c20 0x7fa3b8155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bd200640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3b80b3c20 0x7fa3b8155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bd200640 1 -- 172.21.15.6:0/2182899128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8155e20 con 0x7fa3b80b1b40 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3be600640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3b80b3c20 0x7fa3b8155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bd200640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 0x7fa3b81f8480 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac038040 tx=0x7fa3ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac00e120 con 0x7fa3b80b1b40 2024-03-06T07:30:07.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ac040430 con 0x7fa3b80b1b40 2024-03-06T07:30:07.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac040820 con 0x7fa3b80b1b40 2024-03-06T07:30:07.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.602+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b8158c80 con 0x7fa3b80b1b40 2024-03-06T07:30:07.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.603+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b8159240 con 0x7fa3b80b1b40 2024-03-06T07:30:07.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.604+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b804f330 con 0x7fa3b80b1b40 2024-03-06T07:30:07.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.605+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3ac047030 con 0x7fa3b80b1b40 2024-03-06T07:30:07.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.606+0000 7fa3b5600640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa39c06b6f0 0x7fa39c06dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.606+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa3ac0abab0 con 0x7fa3b80b1b40 2024-03-06T07:30:07.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.607+0000 7fa3bdc00640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa39c06b6f0 0x7fa39c06dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.608+0000 7fa3bdc00640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa39c06b6f0 0x7fa39c06dbb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009650 tx=0x7fa3a8006ba0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.610+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3ac07aca0 con 0x7fa3b80b1b40 2024-03-06T07:30:07.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.810+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa3b806bca0 con 0x7fa3b80b1b40 2024-03-06T07:30:07.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.816+0000 7fa3b5600640 1 -- 172.21.15.6:0/2182899128 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa3ac07a5a0 con 0x7fa3b80b1b40 2024-03-06T07:30:07.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa39c06b6f0 msgr2=0x7fa39c06dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa39c06b6f0 0x7fa39c06dbb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009650 tx=0x7fa3a8006ba0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 msgr2=0x7fa3b81f8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 0x7fa3b81f8480 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac038040 tx=0x7fa3ac004ab0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 shutdown_connections 2024-03-06T07:30:07.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa39c06b6f0 0x7fa39c06dbb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3b80b3c20 0x7fa3b8155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3b80b1b40 0x7fa3b81f8480 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 --2- 172.21.15.6:0/2182899128 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3b80729d0 0x7fa3b80c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 >> 172.21.15.6:0/2182899128 conn(0x7fa3b806b3a0 msgr2=0x7fa3b806c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 shutdown_connections 2024-03-06T07:30:07.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:07.819+0000 7fa3bf000640 1 -- 172.21.15.6:0/2182899128 wait complete. 2024-03-06T07:30:07.841 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:07.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:07.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:08.087 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:08.088 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:08.088 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.145 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:08.145 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:30:08.145 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.201 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:08.201 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:08.201 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:08.259 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:30:08.260 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:08.313 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:08.313 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:30:08.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:08.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.537+0000 7ff26a400640 1 Processor -- start 2024-03-06T07:30:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.538+0000 7ff26a400640 1 -- start start 2024-03-06T07:30:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.539+0000 7ff26a400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 0x7ff26415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.539+0000 7ff26a400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26415cc40 con 0x7ff2640cdc20 2024-03-06T07:30:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.539+0000 7ff26a400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26415cf20 con 0x7ff2640af580 2024-03-06T07:30:08.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.539+0000 7ff26a400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26415d200 con 0x7ff2640af1a0 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.540+0000 7ff263e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff2640cdc20 0x7ff2640b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60528/0 (socket says 172.21.15.6:60528) 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.540+0000 7ff263e00640 1 -- 172.21.15.6:0/2799353722 learned_addr learned my addr 172.21.15.6:0/2799353722 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.540+0000 7ff263400640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 0x7ff26415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.540+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327489435 0 0) 0x7ff26415cc40 con 0x7ff2640cdc20 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.540+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248003520 con 0x7ff2640cdc20 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2378846249 0 0) 0x7ff248003520 con 0x7ff2640cdc20 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 msgr2=0x7ff26415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 0x7ff26415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 >> v1:172.21.15.178:6789/0 conn(0x7ff2640af580 legacy=0x7ff2640b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26415d4e0 con 0x7ff2640cdc20 2024-03-06T07:30:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff25c0019f0 con 0x7ff2640cdc20 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 >> v1:172.21.15.6:6789/0 conn(0x7ff2640cdc20 legacy=0x7ff2640b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248003a50 0x7ff248003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 0x7ff248004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff248004d40 0x7ff2480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26415d200 con 0x7ff248004d40 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24800bb50 con 0x7ff248003a50 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24800bd50 con 0x7ff2640af1a0 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff263e00640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff248004d40 0x7ff2480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.541+0000 7ff262a00640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 0x7ff248004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.542+0000 7ff263400640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248003a50 0x7ff248003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.542+0000 7ff263e00640 1 -- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 msgr2=0x7ff248004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.542+0000 7ff263e00640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2640af1a0 0x7ff248004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.542+0000 7ff263e00640 1 -- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248003a50 msgr2=0x7ff248003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.542+0000 7ff263e00640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248003a50 0x7ff248003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.542+0000 7ff263e00640 1 -- 172.21.15.6:0/2799353722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff248003520 con 0x7ff248004d40 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.543+0000 7ff263e00640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff248004d40 0x7ff2480051c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00a850 tx=0x7ff248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a98912602622cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.543+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c00a4a0 con 0x7ff248004d40 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.543+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c034650 con 0x7ff248004d40 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.543+0000 7ff262000640 1 -- 172.21.15.6:0/2799353722 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c034a40 con 0x7ff248004d40 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff26a400640 1 -- 172.21.15.6:0/2799353722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff248004d40 msgr2=0x7ff2480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff26a400640 1 --2- 172.21.15.6:0/2799353722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff248004d40 0x7ff2480051c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00a850 tx=0x7ff248005830 comp rx=0 tx=0).stop 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff263e00640 1 -- 172.21.15.6:0/2799353722 reap_dead start 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff26a400640 1 -- 172.21.15.6:0/2799353722 shutdown_connections 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff26a400640 1 -- 172.21.15.6:0/2799353722 >> 172.21.15.6:0/2799353722 conn(0x7ff26406afb0 msgr2=0x7ff26406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff26a400640 1 -- 172.21.15.6:0/2799353722 shutdown_connections 2024-03-06T07:30:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.544+0000 7ff26a400640 1 -- 172.21.15.6:0/2799353722 wait complete. 2024-03-06T07:30:08.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.546+0000 7ff26a400640 1 Processor -- start 2024-03-06T07:30:08.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.546+0000 7ff26a400640 1 -- start start 2024-03-06T07:30:08.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff26a400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248004d40 0x7ff26409cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff26a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 0x7ff2640c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff26a400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff26414ba20 0x7ff26414bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff26a400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26415c600 con 0x7ff2640c9150 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff26a400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640b36f0 con 0x7ff248004d40 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff26a400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26415c780 con 0x7ff26414ba20 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff262a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 0x7ff2640c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff262a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 0x7ff2640c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52398/0 (socket says 172.21.15.6:52398) 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff262a00640 1 -- 172.21.15.6:0/3732362951 learned_addr learned my addr 172.21.15.6:0/3732362951 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff263e00640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff26414ba20 0x7ff26414bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.547+0000 7ff262a00640 1 -- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff26414ba20 msgr2=0x7ff26414bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff262a00640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff26414ba20 0x7ff26414bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff263400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248004d40 0x7ff26409cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff262a00640 1 -- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248004d40 msgr2=0x7ff26409cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff262a00640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248004d40 0x7ff26409cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff262a00640 1 -- 172.21.15.6:0/3732362951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26414c3e0 con 0x7ff2640c9150 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff263e00640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff26414ba20 0x7ff26414bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:08.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff262a00640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 0x7ff2640c95b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff25000e550 tx=0x7ff25000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500052e0 con 0x7ff2640c9150 2024-03-06T07:30:08.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff250018070 con 0x7ff2640c9150 2024-03-06T07:30:08.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250003810 con 0x7ff2640c9150 2024-03-06T07:30:08.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640c9c10 con 0x7ff2640c9150 2024-03-06T07:30:08.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.548+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2640ca1d0 con 0x7ff2640c9150 2024-03-06T07:30:08.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.549+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26404f3a0 con 0x7ff2640c9150 2024-03-06T07:30:08.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.550+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2500039b0 con 0x7ff2640c9150 2024-03-06T07:30:08.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.551+0000 7ff25b400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff24406b4d0 0x7ff24406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:08.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.551+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff250089c20 con 0x7ff2640c9150 2024-03-06T07:30:08.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.552+0000 7ff263400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff24406b4d0 0x7ff24406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:08.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.553+0000 7ff263400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff24406b4d0 0x7ff24406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff254009950 tx=0x7ff254008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:08.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.556+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff250058d00 con 0x7ff2640c9150 2024-03-06T07:30:08.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.793+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2640b3050 con 0x7ff2640c9150 2024-03-06T07:30:08.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.795+0000 7ff25b400640 1 -- 172.21.15.6:0/3732362951 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3699 (secure 0 0 0) 0x7ff250058600 con 0x7ff2640c9150 2024-03-06T07:30:08.797 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:08.797 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3567932309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":3567932309},{"type":"v1","addr":"172.21.15.6:6835","nonce":3567932309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":4841,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/3579225212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3579225212},{"type":"v1","addr":"172.21.15.178:6836","nonce":3579225212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1997276286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1997276286},{"type":"v1","addr":"172.21.15.178:6837","nonce":1997276286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28},{"gid":5553,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6834/1017780573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1017780573},{"type":"v1","addr":"172.21.15.6:6834","nonce":1017780573}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-03-06T07:30:08.797 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 28 2024-03-06T07:30:08.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff24406b4d0 msgr2=0x7ff24406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff24406b4d0 0x7ff24406d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff254009950 tx=0x7ff254008040 comp rx=0 tx=0).stop 2024-03-06T07:30:08.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 msgr2=0x7ff2640c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:08.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 0x7ff2640c95b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff25000e550 tx=0x7ff25000ea10 comp rx=0 tx=0).stop 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 shutdown_connections 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff24406b4d0 0x7ff24406d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff26414ba20 0x7ff26414bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2640c9150 0x7ff2640c95b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 --2- 172.21.15.6:0/3732362951 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff248004d40 0x7ff26409cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 >> 172.21.15.6:0/3732362951 conn(0x7ff26406afb0 msgr2=0x7ff264073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 shutdown_connections 2024-03-06T07:30:08.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:08.797+0000 7ff26a400640 1 -- 172.21.15.6:0/3732362951 wait complete. 2024-03-06T07:30:08.819 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:30:08.819 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:30:08.819 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:30:08.819 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:08.819 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:30:08.819 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:30:08.820 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:08.820 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:30:08.820 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:30:08.820 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:30:08.820 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:08.820 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:30:08.821 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:30:08.821 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:30:08.821 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:30:08.837 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:30:08.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:30:08.843 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:30:08.844 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:30:08.845 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:30:08.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:30:08.885 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:30:08.885 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:08.893 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:30:08.893 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:08.899 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:30:08.899 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:09.059 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:30:09.059 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:30:09.078 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:30:11.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-06T07:30:14.837 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:30:14.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:30:14.846 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:30:14.846 DEBUG:teuthology.parallel:result is None 2024-03-06T07:30:14.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:30:15.019 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:30:15.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.045+0000 7f8f45800640 1 Processor -- start 2024-03-06T07:30:15.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.046+0000 7f8f45800640 1 -- start start 2024-03-06T07:30:15.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f45800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f400712c0 0x7f8f40162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f45800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400ce090 con 0x7f8f40070ee0 2024-03-06T07:30:15.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f45800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400ce370 con 0x7f8f40072da0 2024-03-06T07:30:15.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f45800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400ceca0 con 0x7f8f400712c0 2024-03-06T07:30:15.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f3ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8f40070ee0 0x7f8f4006e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34238/0 (socket says 172.21.15.6:34238) 2024-03-06T07:30:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f3ea00640 1 -- 172.21.15.6:0/2431407499 learned_addr learned my addr 172.21.15.6:0/2431407499 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.047+0000 7f8f3f400640 1 --2- 172.21.15.6:0/2431407499 >> v2:172.21.15.6:3300/0 conn(0x7f8f400712c0 0x7f8f40162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.048+0000 7f8f3e000640 1 -- 172.21.15.6:0/2431407499 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659429170 0 0) 0x7f8f400ce090 con 0x7f8f40070ee0 2024-03-06T07:30:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.048+0000 7f8f3e000640 1 -- 172.21.15.6:0/2431407499 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f24003520 con 0x7f8f40070ee0 2024-03-06T07:30:15.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.048+0000 7f8f3f400640 1 -- 172.21.15.6:0/2431407499 >> v1:172.21.15.178:6789/0 conn(0x7f8f40072da0 legacy=0x7f8f4006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.048+0000 7f8f3f400640 1 -- 172.21.15.6:0/2431407499 >> v1:172.21.15.6:6789/0 conn(0x7f8f40070ee0 legacy=0x7f8f4006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.048+0000 7f8f3f400640 1 -- 172.21.15.6:0/2431407499 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400ce650 con 0x7f8f400712c0 2024-03-06T07:30:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.049+0000 7f8f3f400640 1 --2- 172.21.15.6:0/2431407499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f400712c0 0x7f8f40162c40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8f400cee20 tx=0x7f8f3802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f28deec6be3d77a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.049+0000 7f8f3e000640 1 -- 172.21.15.6:0/2431407499 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38039070 con 0x7f8f400712c0 2024-03-06T07:30:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.049+0000 7f8f3e000640 1 -- 172.21.15.6:0/2431407499 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3802cce0 con 0x7f8f400712c0 2024-03-06T07:30:15.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.049+0000 7f8f3e000640 1 -- 172.21.15.6:0/2431407499 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38034640 con 0x7f8f400712c0 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 -- 172.21.15.6:0/2431407499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f400712c0 msgr2=0x7f8f40162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 --2- 172.21.15.6:0/2431407499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f400712c0 0x7f8f40162c40 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8f400cee20 tx=0x7f8f3802bf30 comp rx=0 tx=0).stop 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 -- 172.21.15.6:0/2431407499 shutdown_connections 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 --2- 172.21.15.6:0/2431407499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f400712c0 0x7f8f40162c40 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 -- 172.21.15.6:0/2431407499 >> 172.21.15.6:0/2431407499 conn(0x7f8f4006afb0 msgr2=0x7f8f4006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 -- 172.21.15.6:0/2431407499 shutdown_connections 2024-03-06T07:30:15.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.050+0000 7f8f45800640 1 -- 172.21.15.6:0/2431407499 wait complete. 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.052+0000 7f8f45800640 1 Processor -- start 2024-03-06T07:30:15.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.052+0000 7f8f45800640 1 -- start start 2024-03-06T07:30:15.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f45800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f40070ee0 0x7f8f401fb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f45800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f400712c0 0x7f8f400ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f45800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 0x7f8f40161d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f45800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400cdda0 con 0x7f8f40072da0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f45800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400cdc20 con 0x7f8f400712c0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f45800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4006e7d0 con 0x7f8f40070ee0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f3f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 0x7f8f40161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f3f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 0x7f8f40161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58766/0 (socket says 172.21.15.6:58766) 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f3f400640 1 -- 172.21.15.6:0/339526373 learned_addr learned my addr 172.21.15.6:0/339526373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f3ea00640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f40070ee0 0x7f8f401fb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.053+0000 7f8f37e00640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f400712c0 0x7f8f400ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f3f400640 1 -- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f40070ee0 msgr2=0x7f8f401fb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f3f400640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f40070ee0 0x7f8f401fb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f3f400640 1 -- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f400712c0 msgr2=0x7f8f400ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f3f400640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f400712c0 0x7f8f400ad240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f3f400640 1 -- 172.21.15.6:0/339526373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400adc20 con 0x7f8f40072da0 2024-03-06T07:30:15.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f3f400640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 0x7f8f40161d30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009e80 tx=0x7f8f38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38044020 con 0x7f8f40072da0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3803a040 con 0x7f8f40072da0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f400adf00 con 0x7f8f40072da0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.054+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38039050 con 0x7f8f40072da0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.055+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f400ae4c0 con 0x7f8f40072da0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.056+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4004f3a0 con 0x7f8f40072da0 2024-03-06T07:30:15.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.057+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f38003b90 con 0x7f8f40072da0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.058+0000 7f8f36a00640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8f1c06b4d0 0x7f8f1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.058+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8f380ac710 con 0x7f8f40072da0 2024-03-06T07:30:15.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.059+0000 7f8f3ea00640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8f1c06b4d0 0x7f8f1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.060+0000 7f8f3ea00640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8f1c06b4d0 0x7f8f1c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8f300011e0 tx=0x7f8f30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.063+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f3804e020 con 0x7f8f40072da0 2024-03-06T07:30:15.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:15.278+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8f40167140 con 0x7f8f40072da0 2024-03-06T07:30:16.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.560+0000 7f8f36a00640 1 -- 172.21.15.6:0/339526373 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8f3807b880 con 0x7f8f40072da0 2024-03-06T07:30:16.563 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:30:16.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8f1c06b4d0 msgr2=0x7f8f1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8f1c06b4d0 0x7f8f1c06d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8f300011e0 tx=0x7f8f30009040 comp rx=0 tx=0).stop 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 msgr2=0x7f8f40161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 0x7f8f40161d30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009e80 tx=0x7f8f38004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 shutdown_connections 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8f1c06b4d0 0x7f8f1c06d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f400712c0 0x7f8f400ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f40072da0 0x7f8f40161d30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 --2- 172.21.15.6:0/339526373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8f40070ee0 0x7f8f401fb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 >> 172.21.15.6:0/339526373 conn(0x7f8f4006afb0 msgr2=0x7f8f4006cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 shutdown_connections 2024-03-06T07:30:16.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.565+0000 7f8f45800640 1 -- 172.21.15.6:0/339526373 wait complete. 2024-03-06T07:30:16.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:30:16.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.810+0000 7f444be00640 1 Processor -- start 2024-03-06T07:30:16.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.810+0000 7f444be00640 1 -- start start 2024-03-06T07:30:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.811+0000 7f444be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c15efc0 0x7f444c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.811+0000 7f444be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f444c1621d0 0x7f444c070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.811+0000 7f444be00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f444c1627f0 con 0x7f444c0cce90 2024-03-06T07:30:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.811+0000 7f444be00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444c15f3a0 con 0x7f444c1621d0 2024-03-06T07:30:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.811+0000 7f444be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c06d190 con 0x7f444c15efc0 2024-03-06T07:30:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.812+0000 7f444b400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f444c1621d0 0x7f444c070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.812+0000 7f444aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c15efc0 0x7f444c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.812+0000 7f444b400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f444c1621d0 0x7f444c070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:57910/0 (socket says 172.21.15.6:57910) 2024-03-06T07:30:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.812+0000 7f444b400640 1 -- 172.21.15.6:0/2733222079 learned_addr learned my addr 172.21.15.6:0/2733222079 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.813+0000 7f444aa00640 1 -- 172.21.15.6:0/2733222079 >> v2:172.21.15.178:3300/0 conn(0x7f444c1621d0 msgr2=0x7f444c070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.813+0000 7f444aa00640 1 --2- 172.21.15.6:0/2733222079 >> v2:172.21.15.178:3300/0 conn(0x7f444c1621d0 0x7f444c070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.813+0000 7f444aa00640 1 -- 172.21.15.6:0/2733222079 >> v1:172.21.15.178:6789/0 conn(0x7f444c0cce90 legacy=0x7f444c161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.813+0000 7f444aa00640 1 -- 172.21.15.6:0/2733222079 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c06c860 con 0x7f444c15efc0 2024-03-06T07:30:16.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.814+0000 7f444aa00640 1 --2- 172.21.15.6:0/2733222079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c15efc0 0x7f444c0cc950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f443c0097a0 tx=0x7f443c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a86fbd1ddf52b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.815+0000 7f444a000640 1 -- 172.21.15.6:0/2733222079 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c035020 con 0x7f444c15efc0 2024-03-06T07:30:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.815+0000 7f444a000640 1 -- 172.21.15.6:0/2733222079 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c031810 con 0x7f444c15efc0 2024-03-06T07:30:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.815+0000 7f444a000640 1 -- 172.21.15.6:0/2733222079 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c031be0 con 0x7f444c15efc0 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 -- 172.21.15.6:0/2733222079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c15efc0 msgr2=0x7f444c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 --2- 172.21.15.6:0/2733222079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c15efc0 0x7f444c0cc950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f443c0097a0 tx=0x7f443c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 -- 172.21.15.6:0/2733222079 shutdown_connections 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 --2- 172.21.15.6:0/2733222079 >> v2:172.21.15.178:3300/0 conn(0x7f444c1621d0 0x7f444c070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 --2- 172.21.15.6:0/2733222079 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c15efc0 0x7f444c0cc950 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 -- 172.21.15.6:0/2733222079 >> 172.21.15.6:0/2733222079 conn(0x7f444c06b3d0 msgr2=0x7f444c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 -- 172.21.15.6:0/2733222079 shutdown_connections 2024-03-06T07:30:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.816+0000 7f444be00640 1 -- 172.21.15.6:0/2733222079 wait complete. 2024-03-06T07:30:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.818+0000 7f444be00640 1 Processor -- start 2024-03-06T07:30:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.818+0000 7f444be00640 1 -- start start 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f444c0cce90 0x7f444c145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 0x7f444c0b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c1621d0 0x7f444c158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c06d620 con 0x7f444c15efc0 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c06ce20 con 0x7f444c0cce90 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c0b0e10 con 0x7f444c1621d0 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f4443e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 0x7f444c0b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f4443e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 0x7f444c0b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58782/0 (socket says 172.21.15.6:58782) 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f4443e00640 1 -- 172.21.15.6:0/2934660094 learned_addr learned my addr 172.21.15.6:0/2934660094 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444aa00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f444c0cce90 0x7f444c145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.819+0000 7f444b400640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c1621d0 0x7f444c158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4443e00640 1 -- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c1621d0 msgr2=0x7f444c158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4443e00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c1621d0 0x7f444c158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4443e00640 1 -- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f444c0cce90 msgr2=0x7f444c145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4443e00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f444c0cce90 0x7f444c145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4443e00640 1 -- 172.21.15.6:0/2934660094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c0b10f0 con 0x7f444c15efc0 2024-03-06T07:30:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f444b400640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c1621d0 0x7f444c158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:16.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4443e00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 0x7f444c0b1f80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f443400c400 tx=0x7f443400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340052d0 con 0x7f444c15efc0 2024-03-06T07:30:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4434005490 con 0x7f444c15efc0 2024-03-06T07:30:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c0b13d0 con 0x7f444c15efc0 2024-03-06T07:30:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434007260 con 0x7f444c15efc0 2024-03-06T07:30:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.820+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444c070c80 con 0x7f444c15efc0 2024-03-06T07:30:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.821+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444c070e80 con 0x7f444c15efc0 2024-03-06T07:30:16.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.823+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4434007400 con 0x7f444c15efc0 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.824+0000 7f4442a00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f442806b4d0 0x7f442806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.824+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44340894a0 con 0x7f444c15efc0 2024-03-06T07:30:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.824+0000 7f444aa00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f442806b4d0 0x7f442806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.825+0000 7f444aa00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f442806b4d0 0x7f442806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f443c036000 tx=0x7f443c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:16.829+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f444c070e80 con 0x7f444c15efc0 2024-03-06T07:30:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.101+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f444c04f330 con 0x7f444c15efc0 2024-03-06T07:30:17.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.102+0000 7f4442a00640 1 -- 172.21.15.6:0/2934660094 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f44340585d0 con 0x7f444c15efc0 2024-03-06T07:30:17.104 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:30:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:30:17.107 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:30:17.108 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.110 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:30:17.111 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.113 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:30:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:30:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:30:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.116 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:30:17.118 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:17.119 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:30:17.119 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:30:17.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.104+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f442806b4d0 msgr2=0x7f442806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.104+0000 7f444be00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f442806b4d0 0x7f442806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f443c036000 tx=0x7f443c03c040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.104+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 msgr2=0x7f444c0b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.104+0000 7f444be00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 0x7f444c0b1f80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f443400c400 tx=0x7f443400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 shutdown_connections 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f442806b4d0 0x7f442806d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f444c1621d0 0x7f444c158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f444c15efc0 0x7f444c0b1f80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 --2- 172.21.15.6:0/2934660094 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f444c0cce90 0x7f444c145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 >> 172.21.15.6:0/2934660094 conn(0x7f444c06b3d0 msgr2=0x7f444c157e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 shutdown_connections 2024-03-06T07:30:17.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.105+0000 7f444be00640 1 -- 172.21.15.6:0/2934660094 wait complete. 2024-03-06T07:30:17.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:17.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.349+0000 7fa208e00640 1 Processor -- start 2024-03-06T07:30:17.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.350+0000 7fa208e00640 1 -- start start 2024-03-06T07:30:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa208e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa2040b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa208e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa20406afb0 0x7fa20406b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa208e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa20406bec0 con 0x7fa20415e200 2024-03-06T07:30:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa208e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2040b0670 con 0x7fa20406afb0 2024-03-06T07:30:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa208e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040ae380 con 0x7fa2040afdf0 2024-03-06T07:30:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa20406afb0 0x7fa20406b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa20406afb0 0x7fa20406b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58792/0 (socket says 172.21.15.6:58792) 2024-03-06T07:30:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 -- 172.21.15.6:0/1150742713 learned_addr learned my addr 172.21.15.6:0/1150742713 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/1150742713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa2040b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 -- 172.21.15.6:0/1150742713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 msgr2=0x7fa2040b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 --2- 172.21.15.6:0/1150742713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa2040b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 -- 172.21.15.6:0/1150742713 >> v1:172.21.15.178:6789/0 conn(0x7fa20415e200 legacy=0x7fa2040af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.352+0000 7fa202800640 1 -- 172.21.15.6:0/1150742713 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2040ae660 con 0x7fa20406afb0 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.354+0000 7fa202800640 1 --2- 172.21.15.6:0/1150742713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa20406b430 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa2040b3ba0 tx=0x7fa1fc02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99abcd2555e1e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.354+0000 7fa201400640 1 -- 172.21.15.6:0/1150742713 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc02ca30 con 0x7fa20406afb0 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.354+0000 7fa201400640 1 -- 172.21.15.6:0/1150742713 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1fc02cbd0 con 0x7fa20406afb0 2024-03-06T07:30:17.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.354+0000 7fa201400640 1 -- 172.21.15.6:0/1150742713 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0343f0 con 0x7fa20406afb0 2024-03-06T07:30:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 -- 172.21.15.6:0/1150742713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 msgr2=0x7fa20406b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 --2- 172.21.15.6:0/1150742713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa20406b430 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa2040b3ba0 tx=0x7fa1fc02be90 comp rx=0 tx=0).stop 2024-03-06T07:30:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 -- 172.21.15.6:0/1150742713 shutdown_connections 2024-03-06T07:30:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 --2- 172.21.15.6:0/1150742713 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa20406b430 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 --2- 172.21.15.6:0/1150742713 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa2040b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 -- 172.21.15.6:0/1150742713 >> 172.21.15.6:0/1150742713 conn(0x7fa2040b2ae0 msgr2=0x7fa2040b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 -- 172.21.15.6:0/1150742713 shutdown_connections 2024-03-06T07:30:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.355+0000 7fa208e00640 1 -- 172.21.15.6:0/1150742713 wait complete. 2024-03-06T07:30:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.358+0000 7fa208e00640 1 Processor -- start 2024-03-06T07:30:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.358+0000 7fa208e00640 1 -- start start 2024-03-06T07:30:17.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa208e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa2041424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa208e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa20415a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa208e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa20415e200 0x7fa2040b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa208e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20406dbb0 con 0x7fa20406afb0 2024-03-06T07:30:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa208e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2040aec20 con 0x7fa20415e200 2024-03-06T07:30:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa208e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa20406dd30 con 0x7fa2040afdf0 2024-03-06T07:30:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa201e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa2041424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa201e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa2041424f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58798/0 (socket says 172.21.15.6:58798) 2024-03-06T07:30:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.359+0000 7fa201e00640 1 -- 172.21.15.6:0/2428190676 learned_addr learned my addr 172.21.15.6:0/2428190676 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa202800640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa20415e200 0x7fa2040b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa201e00640 1 -- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 msgr2=0x7fa20415a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa201e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa20415a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa201e00640 1 -- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa20415e200 msgr2=0x7fa2040b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa201e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa20415e200 0x7fa2040b1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa201e00640 1 -- 172.21.15.6:0/2428190676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204142a30 con 0x7fa20406afb0 2024-03-06T07:30:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa201e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa2041424f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400b3f0 tx=0x7fa1f400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4015020 con 0x7fa20406afb0 2024-03-06T07:30:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f4011830 con 0x7fa20406afb0 2024-03-06T07:30:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.360+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20415ae20 con 0x7fa20406afb0 2024-03-06T07:30:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.361+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4011c20 con 0x7fa20406afb0 2024-03-06T07:30:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.361+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20415b3e0 con 0x7fa20406afb0 2024-03-06T07:30:17.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.362+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2040b4fc0 con 0x7fa20406afb0 2024-03-06T07:30:17.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.364+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1f4018020 con 0x7fa20406afb0 2024-03-06T07:30:17.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.364+0000 7fa1fa000640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1e006b4d0 0x7fa1e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.364+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa1f4089ba0 con 0x7fa20406afb0 2024-03-06T07:30:17.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.364+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1e006b4d0 0x7fa1e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.365+0000 7fa1fbe00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1e006b4d0 0x7fa1e006d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa20415d8b0 tx=0x7fa1ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.368+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2040b4fc0 con 0x7fa20406afb0 2024-03-06T07:30:17.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.575+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa20404f3a0 con 0x7fa20406afb0 2024-03-06T07:30:17.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.578+0000 7fa1fa000640 1 -- 172.21.15.6:0/2428190676 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+12819 (secure 0 0 0) 0x7fa1f401f030 con 0x7fa20406afb0 2024-03-06T07:30:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:17.580 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":54,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:16.557769+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1e006b4d0 msgr2=0x7fa1e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1e006b4d0 0x7fa1e006d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa20415d8b0 tx=0x7fa1ec008040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 msgr2=0x7fa2041424f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa2041424f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400b3f0 tx=0x7fa1f400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 shutdown_connections 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1e006b4d0 0x7fa1e006d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa20415e200 0x7fa2040b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2040afdf0 0x7fa20415a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 --2- 172.21.15.6:0/2428190676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa20406afb0 0x7fa2041424f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 >> 172.21.15.6:0/2428190676 conn(0x7fa2040b2ae0 msgr2=0x7fa2040b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 shutdown_connections 2024-03-06T07:30:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.581+0000 7fa208e00640 1 -- 172.21.15.6:0/2428190676 wait complete. 2024-03-06T07:30:17.604 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:17.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.819+0000 7f1ec7c00640 1 Processor -- start 2024-03-06T07:30:17.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.821+0000 7f1ec7c00640 1 -- start start 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.821+0000 7f1ec7c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00d1c00 0x7f1ec00d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.821+0000 7f1ec7c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ec00b0e10 0x7f1ec00d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.821+0000 7f1ec7c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec015c250 con 0x7f1ec00d2520 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ec7c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec00b2f50 con 0x7f1ec00b0e10 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ec7c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00b3510 con 0x7f1ec00d1c00 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ebfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ec00b0e10 0x7f1ec00d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ec5c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1ec00d2520 0x7f1ec0161e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34254/0 (socket says 172.21.15.6:34254) 2024-03-06T07:30:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ec5c00640 1 -- 172.21.15.6:0/1422901822 learned_addr learned my addr 172.21.15.6:0/1422901822 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ec5200640 1 --2- 172.21.15.6:0/1422901822 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00d1c00 0x7f1ec00d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ebfe00640 1 -- 172.21.15.6:0/1422901822 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00d1c00 msgr2=0x7f1ec00d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.822+0000 7f1ebfe00640 1 --2- 172.21.15.6:0/1422901822 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00d1c00 0x7f1ec00d13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.823+0000 7f1ebfe00640 1 -- 172.21.15.6:0/1422901822 >> v1:172.21.15.6:6789/0 conn(0x7f1ec00d2520 legacy=0x7f1ec0161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.823+0000 7f1ebfe00640 1 -- 172.21.15.6:0/1422901822 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec015f370 con 0x7f1ec00b0e10 2024-03-06T07:30:17.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.824+0000 7f1ebfe00640 1 --2- 172.21.15.6:0/1422901822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00b0e10 0x7f1ec00d1fe0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80097a0 tx=0x7f1ea802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48aed2eb7878cbba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.824+0000 7f1ebf400640 1 -- 172.21.15.6:0/1422901822 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8031780 con 0x7f1ec00b0e10 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.824+0000 7f1ebf400640 1 -- 172.21.15.6:0/1422901822 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea8037070 con 0x7f1ec00b0e10 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.824+0000 7f1ebf400640 1 -- 172.21.15.6:0/1422901822 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8031b40 con 0x7f1ec00b0e10 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 -- 172.21.15.6:0/1422901822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00b0e10 msgr2=0x7f1ec00d1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/1422901822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00b0e10 0x7f1ec00d1fe0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80097a0 tx=0x7f1ea802b820 comp rx=0 tx=0).stop 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 -- 172.21.15.6:0/1422901822 shutdown_connections 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/1422901822 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00b0e10 0x7f1ec00d1fe0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/1422901822 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00d1c00 0x7f1ec00d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 -- 172.21.15.6:0/1422901822 >> 172.21.15.6:0/1422901822 conn(0x7f1ec009aa10 msgr2=0x7f1ec009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 -- 172.21.15.6:0/1422901822 shutdown_connections 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.825+0000 7f1ec7c00640 1 -- 172.21.15.6:0/1422901822 wait complete. 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.826+0000 7f1ec7c00640 1 Processor -- start 2024-03-06T07:30:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.827+0000 7f1ec7c00640 1 -- start start 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec7c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00b0e10 0x7f1ec0151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec7c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ec00d1c00 0x7f1ec006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec7c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 0x7f1ec006d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec7c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00b31a0 con 0x7f1ec00d2520 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec7c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00cff80 con 0x7f1ec00d1c00 2024-03-06T07:30:17.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec7c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec00d1900 con 0x7f1ec00b0e10 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 0x7f1ec006d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 0x7f1ec006d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58812/0 (socket says 172.21.15.6:58812) 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 -- 172.21.15.6:0/2904581758 learned_addr learned my addr 172.21.15.6:0/2904581758 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ebfe00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ec00d1c00 0x7f1ec006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5200640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00b0e10 0x7f1ec0151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 -- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00b0e10 msgr2=0x7f1ec0151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00b0e10 0x7f1ec0151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 -- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ec00d1c00 msgr2=0x7f1ec006d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ec00d1c00 0x7f1ec006d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5c00640 1 -- 172.21.15.6:0/2904581758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec01605f0 con 0x7f1ec00d2520 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ec5200640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00b0e10 0x7f1ec0151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.828+0000 7f1ebfe00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ec00d1c00 0x7f1ec006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.829+0000 7f1ec5c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 0x7f1ec006d610 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c420 tx=0x7f1eb800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.829+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8017070 con 0x7f1ec00d2520 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.829+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8005ce0 con 0x7f1ec00d2520 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.829+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80072d0 con 0x7f1ec00d2520 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.829+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec01608d0 con 0x7f1ec00d2520 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.829+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec0160e90 con 0x7f1ec00d2520 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.831+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec004f330 con 0x7f1ec00d2520 2024-03-06T07:30:17.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.831+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1eb801a070 con 0x7f1ec00d2520 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.832+0000 7f1ebcc00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e9c06b4d0 0x7f1e9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.832+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1eb8089100 con 0x7f1ec00d2520 2024-03-06T07:30:17.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.833+0000 7f1ec5200640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e9c06b4d0 0x7f1e9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.834+0000 7f1ec5200640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e9c06b4d0 0x7f1e9c06d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0007330 tx=0x7f1eb000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:17.838+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1eb8057c80 con 0x7f1ec00d2520 2024-03-06T07:30:17.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-06T07:30:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.039+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ec00b2540 con 0x7f1ec00d2520 2024-03-06T07:30:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.041+0000 7f1ebcc00640 1 -- 172.21.15.6:0/2904581758 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1eb805bd20 con 0x7f1ec00d2520 2024-03-06T07:30:18.042 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:30:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e9c06b4d0 msgr2=0x7f1e9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e9c06b4d0 0x7f1e9c06d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0007330 tx=0x7f1eb000b040 comp rx=0 tx=0).stop 2024-03-06T07:30:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 msgr2=0x7f1ec006d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 0x7f1ec006d610 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800c420 tx=0x7f1eb800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 shutdown_connections 2024-03-06T07:30:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e9c06b4d0 0x7f1e9c06d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ec00d1c00 0x7f1ec006d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ec00d2520 0x7f1ec006d610 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 --2- 172.21.15.6:0/2904581758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ec00b0e10 0x7f1ec0151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 >> 172.21.15.6:0/2904581758 conn(0x7f1ec009aa10 msgr2=0x7f1ec006fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 shutdown_connections 2024-03-06T07:30:18.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.043+0000 7f1ec7c00640 1 -- 172.21.15.6:0/2904581758 wait complete. 2024-03-06T07:30:18.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:30:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.284+0000 7fc162e00640 1 Processor -- start 2024-03-06T07:30:18.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.285+0000 7fc162e00640 1 -- start start 2024-03-06T07:30:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.286+0000 7fc162e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c071380 0x7fc15c0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.286+0000 7fc162e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15c0ce9a0 con 0x7fc15c161670 2024-03-06T07:30:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.286+0000 7fc162e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15c0cec80 con 0x7fc15c06e240 2024-03-06T07:30:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.286+0000 7fc162e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0730d0 con 0x7fc15c071380 2024-03-06T07:30:18.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.286+0000 7fc15be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc15c161670 0x7fc15c070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34268/0 (socket says 172.21.15.6:34268) 2024-03-06T07:30:18.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.286+0000 7fc15be00640 1 -- 172.21.15.6:0/2301654507 learned_addr learned my addr 172.21.15.6:0/2301654507 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc153e00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c071380 0x7fc15c0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419113487 0 0) 0x7fc15c0ce9a0 con 0x7fc15c161670 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc140003520 con 0x7fc15c161670 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659393518 0 0) 0x7fc140003520 con 0x7fc15c161670 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c071380 msgr2=0x7fc15c0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c071380 0x7fc15c0ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 >> v1:172.21.15.178:6789/0 conn(0x7fc15c06e240 legacy=0x7fc15c0ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c0cef60 con 0x7fc15c161670 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc14c001670 con 0x7fc15c161670 2024-03-06T07:30:18.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 >> v1:172.21.15.6:6789/0 conn(0x7fc15c161670 legacy=0x7fc15c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc140003a50 0x7fc140003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc140004800 0x7fc140004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140005210 0x7fc14000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0730d0 con 0x7fc140003a50 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14000c140 con 0x7fc140005210 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14000c340 con 0x7fc140004800 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.287+0000 7fc153e00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c071380 0x7fc15c0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc15be00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc140003a50 0x7fc140003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc160e00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140005210 0x7fc14000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc15be00640 1 -- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc140004800 msgr2=0x7fc140004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc15be00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc140004800 0x7fc140004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc15be00640 1 -- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140005210 msgr2=0x7fc14000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc15be00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140005210 0x7fc14000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc15be00640 1 -- 172.21.15.6:0/2301654507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc140003520 con 0x7fc140003a50 2024-03-06T07:30:18.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.288+0000 7fc160e00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140005210 0x7fc14000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.289+0000 7fc15be00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc140003a50 0x7fc140003e70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00c0e0 tx=0x7fc140005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be71c11342d2af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c007c20 con 0x7fc140003a50 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c030070 con 0x7fc140003a50 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc15b400640 1 -- 172.21.15.6:0/2301654507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c005d30 con 0x7fc140003a50 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc162e00640 1 -- 172.21.15.6:0/2301654507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc140003a50 msgr2=0x7fc140003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc162e00640 1 --2- 172.21.15.6:0/2301654507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc140003a50 0x7fc140003e70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00c0e0 tx=0x7fc140005830 comp rx=0 tx=0).stop 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc160e00640 1 -- 172.21.15.6:0/2301654507 reap_dead start 2024-03-06T07:30:18.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc162e00640 1 -- 172.21.15.6:0/2301654507 shutdown_connections 2024-03-06T07:30:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc162e00640 1 -- 172.21.15.6:0/2301654507 >> 172.21.15.6:0/2301654507 conn(0x7fc15c06afb0 msgr2=0x7fc15c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc162e00640 1 -- 172.21.15.6:0/2301654507 shutdown_connections 2024-03-06T07:30:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.290+0000 7fc162e00640 1 -- 172.21.15.6:0/2301654507 wait complete. 2024-03-06T07:30:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.292+0000 7fc162e00640 1 Processor -- start 2024-03-06T07:30:18.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.292+0000 7fc162e00640 1 -- start start 2024-03-06T07:30:18.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc162e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140003a50 0x7fc15c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc162e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 0x7fc15c0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc162e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c09c090 0x7fc15c14d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc162e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0ce600 con 0x7fc15c157810 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc162e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0cf520 con 0x7fc140003a50 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc162e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc15c0ce780 con 0x7fc15c09c090 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc153e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 0x7fc15c0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc153e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 0x7fc15c0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58820/0 (socket says 172.21.15.6:58820) 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.293+0000 7fc153e00640 1 -- 172.21.15.6:0/68132829 learned_addr learned my addr 172.21.15.6:0/68132829 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc153e00640 1 -- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c09c090 msgr2=0x7fc15c14d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc153e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c09c090 0x7fc15c14d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc153e00640 1 -- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140003a50 msgr2=0x7fc15c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc15be00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140003a50 0x7fc15c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc153e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140003a50 0x7fc15c09aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc153e00640 1 -- 172.21.15.6:0/68132829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c09aff0 con 0x7fc15c157810 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc153e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 0x7fc15c0b0820 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e570 tx=0x7fc14400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14400f040 con 0x7fc15c157810 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc144004230 con 0x7fc15c157810 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144003610 con 0x7fc15c157810 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.294+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c09b2d0 con 0x7fc15c157810 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.295+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c09b890 con 0x7fc15c157810 2024-03-06T07:30:18.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.296+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15c04f3a0 con 0x7fc15c157810 2024-03-06T07:30:18.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.297+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1440037b0 con 0x7fc15c157810 2024-03-06T07:30:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.298+0000 7fc158c00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc13806b4d0 0x7fc13806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.298+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc144089b00 con 0x7fc15c157810 2024-03-06T07:30:18.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.298+0000 7fc15be00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc13806b4d0 0x7fc13806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.299+0000 7fc15be00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc13806b4d0 0x7fc13806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc14c034000 tx=0x7fc14c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.302+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc144017020 con 0x7fc15c157810 2024-03-06T07:30:18.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.506+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc15c06e720 con 0x7fc15c157810 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.507+0000 7fc158c00640 1 -- 172.21.15.6:0/68132829 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc144058c30 con 0x7fc15c157810 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:30:18.509 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc13806b4d0 msgr2=0x7fc13806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc13806b4d0 0x7fc13806d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc14c034000 tx=0x7fc14c007000 comp rx=0 tx=0).stop 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 msgr2=0x7fc15c0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 0x7fc15c0b0820 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e570 tx=0x7fc14400ea30 comp rx=0 tx=0).stop 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 shutdown_connections 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc13806b4d0 0x7fc13806d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc15c09c090 0x7fc15c14d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc15c157810 0x7fc15c0b0820 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 --2- 172.21.15.6:0/68132829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc140003a50 0x7fc15c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.509+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 >> 172.21.15.6:0/68132829 conn(0x7fc15c06afb0 msgr2=0x7fc15c0ad770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.510+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 shutdown_connections 2024-03-06T07:30:18.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.510+0000 7fc162e00640 1 -- 172.21.15.6:0/68132829 wait complete. 2024-03-06T07:30:18.530 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:30:18.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:30:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.754+0000 7feb26600640 1 Processor -- start 2024-03-06T07:30:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.755+0000 7feb26600640 1 -- start start 2024-03-06T07:30:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.756+0000 7feb26600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb2015e180 0x7feb200b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.756+0000 7feb26600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7feb200d2af0 0x7feb200d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.756+0000 7feb26600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200d1e40 con 0x7feb200d4610 2024-03-06T07:30:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.756+0000 7feb26600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200d1be0 con 0x7feb2015e180 2024-03-06T07:30:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.756+0000 7feb26600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200d2100 con 0x7feb200d2af0 2024-03-06T07:30:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feb200d4610 0x7feb200d23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34272/0 (socket says 172.21.15.6:34272) 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1ea00640 1 -- 172.21.15.6:0/3186772276 learned_addr learned my addr 172.21.15.6:0/3186772276 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7feb2015e180 0x7feb200b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1fe00640 1 --2- 172.21.15.6:0/3186772276 >> v2:172.21.15.178:3300/0 conn(0x7feb200d2af0 0x7feb200d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1f400640 1 -- 172.21.15.6:0/3186772276 >> v2:172.21.15.178:3300/0 conn(0x7feb200d2af0 msgr2=0x7feb200d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1f400640 1 --2- 172.21.15.6:0/3186772276 >> v2:172.21.15.178:3300/0 conn(0x7feb200d2af0 0x7feb200d4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1f400640 1 -- 172.21.15.6:0/3186772276 >> v1:172.21.15.6:6789/0 conn(0x7feb200d4610 legacy=0x7feb200d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1f400640 1 -- 172.21.15.6:0/3186772276 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200ceed0 con 0x7feb2015e180 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.757+0000 7feb1e000640 1 -- 172.21.15.6:0/3186772276 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441789933 0 0) 0x7feb200d1e40 con 0x7feb200d4610 2024-03-06T07:30:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.759+0000 7feb1f400640 1 --2- 172.21.15.6:0/3186772276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb200b35d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7feb2008db90 tx=0x7feb1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3047702e8346938 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.759+0000 7feb1e000640 1 -- 172.21.15.6:0/3186772276 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10035020 con 0x7feb2015e180 2024-03-06T07:30:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.759+0000 7feb1e000640 1 -- 172.21.15.6:0/3186772276 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10031830 con 0x7feb2015e180 2024-03-06T07:30:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.759+0000 7feb1e000640 1 -- 172.21.15.6:0/3186772276 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb10031c00 con 0x7feb2015e180 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 -- 172.21.15.6:0/3186772276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 msgr2=0x7feb200b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 --2- 172.21.15.6:0/3186772276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb200b35d0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7feb2008db90 tx=0x7feb1002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 -- 172.21.15.6:0/3186772276 shutdown_connections 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 --2- 172.21.15.6:0/3186772276 >> v2:172.21.15.178:3300/0 conn(0x7feb200d2af0 0x7feb200d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 --2- 172.21.15.6:0/3186772276 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb200b35d0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 -- 172.21.15.6:0/3186772276 >> 172.21.15.6:0/3186772276 conn(0x7feb2009d2b0 msgr2=0x7feb200d3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 -- 172.21.15.6:0/3186772276 shutdown_connections 2024-03-06T07:30:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.760+0000 7feb26600640 1 -- 172.21.15.6:0/3186772276 wait complete. 2024-03-06T07:30:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.762+0000 7feb26600640 1 Processor -- start 2024-03-06T07:30:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.762+0000 7feb26600640 1 -- start start 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.763+0000 7feb26600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb200d2af0 0x7feb2014d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.763+0000 7feb26600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb200d4610 0x7feb200ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.763+0000 7feb26600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb201f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.763+0000 7feb26600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200b32a0 con 0x7feb2015e180 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.763+0000 7feb26600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200cebc0 con 0x7feb200d2af0 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.763+0000 7feb26600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb200ca8e0 con 0x7feb200d4610 2024-03-06T07:30:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb201f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb201f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58832/0 (socket says 172.21.15.6:58832) 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 -- 172.21.15.6:0/3468490844 learned_addr learned my addr 172.21.15.6:0/3468490844 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1f400640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb200d2af0 0x7feb2014d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1ea00640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb200d4610 0x7feb200ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 -- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb200d4610 msgr2=0x7feb200ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb200d4610 0x7feb200ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 -- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb200d2af0 msgr2=0x7feb2014d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb200d2af0 0x7feb2014d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.764+0000 7feb1fe00640 1 -- 172.21.15.6:0/3468490844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb201f7200 con 0x7feb2015e180 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.765+0000 7feb1fe00640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb201f6cc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7feb18007d80 tx=0x7feb1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.765+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18014420 con 0x7feb2015e180 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.765+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18019070 con 0x7feb2015e180 2024-03-06T07:30:18.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.765+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18014800 con 0x7feb2015e180 2024-03-06T07:30:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.765+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb201f7590 con 0x7feb2015e180 2024-03-06T07:30:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.765+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201f7b50 con 0x7feb2015e180 2024-03-06T07:30:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.766+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb20068010 con 0x7feb2015e180 2024-03-06T07:30:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.767+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb180149a0 con 0x7feb2015e180 2024-03-06T07:30:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.768+0000 7feb17400640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feafc06b4d0 0x7feafc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.768+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb1808a6f0 con 0x7feb2015e180 2024-03-06T07:30:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.769+0000 7feb1f400640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feafc06b4d0 0x7feafc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.770+0000 7feb1f400640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feafc06b4d0 0x7feafc06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feb10036000 tx=0x7feb10008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.773+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb180598b0 con 0x7feb2015e180 2024-03-06T07:30:18.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:18.975+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7feb200d1e40 con 0x7feb2015e180 2024-03-06T07:30:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.586+0000 7feb17400640 1 -- 172.21.15.6:0/3468490844 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) v1 ==== 138+0+0 (secure 0 0 0) 0x7feb180591b0 con 0x7feb2015e180 2024-03-06T07:30:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:30:19.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feafc06b4d0 msgr2=0x7feafc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feafc06b4d0 0x7feafc06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feb10036000 tx=0x7feb10008030 comp rx=0 tx=0).stop 2024-03-06T07:30:19.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 msgr2=0x7feb201f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb201f6cc0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7feb18007d80 tx=0x7feb1800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 shutdown_connections 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feafc06b4d0 0x7feafc06d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb2015e180 0x7feb201f6cc0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb200d4610 0x7feb200ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 --2- 172.21.15.6:0/3468490844 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb200d2af0 0x7feb2014d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.590+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 >> 172.21.15.6:0/3468490844 conn(0x7feb2009d2b0 msgr2=0x7feb200cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.591+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 shutdown_connections 2024-03-06T07:30:19.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.591+0000 7feb26600640 1 -- 172.21.15.6:0/3468490844 wait complete. 2024-03-06T07:30:19.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.838+0000 7fa29b200640 1 Processor -- start 2024-03-06T07:30:19.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.839+0000 7fa29b200640 1 -- start start 2024-03-06T07:30:19.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.840+0000 7fa29b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa29c15e220 0x7fa29c0d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.840+0000 7fa29b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa29c0d1920 0x7fa29c0d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.840+0000 7fa29b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d2520 0x7fa29c0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.840+0000 7fa29b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c0d2990 con 0x7fa29c15e220 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.840+0000 7fa29b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c0b36b0 con 0x7fa29c0d1920 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.840+0000 7fa29b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0d1db0 con 0x7fa29c0d2520 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa29c15e220 0x7fa29c0d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa29c15e220 0x7fa29c0d4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58838/0 (socket says 172.21.15.6:58838) 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 -- 172.21.15.6:0/3047360467 learned_addr learned my addr 172.21.15.6:0/3047360467 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa29a800640 1 --2- 172.21.15.6:0/3047360467 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d2520 0x7fa29c0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa291400640 1 --2- 172.21.15.6:0/3047360467 >> v2:172.21.15.178:3300/0 conn(0x7fa29c0d1920 0x7fa29c0d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 -- 172.21.15.6:0/3047360467 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d2520 msgr2=0x7fa29c0d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 --2- 172.21.15.6:0/3047360467 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d2520 0x7fa29c0d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 -- 172.21.15.6:0/3047360467 >> v2:172.21.15.178:3300/0 conn(0x7fa29c0d1920 msgr2=0x7fa29c0d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 --2- 172.21.15.6:0/3047360467 >> v2:172.21.15.178:3300/0 conn(0x7fa29c0d1920 0x7fa29c0d1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.841+0000 7fa299e00640 1 -- 172.21.15.6:0/3047360467 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0cf4a0 con 0x7fa29c15e220 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.843+0000 7fa299e00640 1 --2- 172.21.15.6:0/3047360467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c15e220 0x7fa29c0d4870 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa29c08e020 tx=0x7fa28c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115157f2a184d703 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.843+0000 7fa299400640 1 -- 172.21.15.6:0/3047360467 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c02c9f0 con 0x7fa29c15e220 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.843+0000 7fa299400640 1 -- 172.21.15.6:0/3047360467 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28c02cb90 con 0x7fa29c15e220 2024-03-06T07:30:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.843+0000 7fa299400640 1 -- 172.21.15.6:0/3047360467 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28c0343f0 con 0x7fa29c15e220 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 -- 172.21.15.6:0/3047360467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c15e220 msgr2=0x7fa29c0d4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 --2- 172.21.15.6:0/3047360467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c15e220 0x7fa29c0d4870 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa29c08e020 tx=0x7fa28c02be50 comp rx=0 tx=0).stop 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 -- 172.21.15.6:0/3047360467 shutdown_connections 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 --2- 172.21.15.6:0/3047360467 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d2520 0x7fa29c0d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 --2- 172.21.15.6:0/3047360467 >> v2:172.21.15.178:3300/0 conn(0x7fa29c0d1920 0x7fa29c0d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 --2- 172.21.15.6:0/3047360467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c15e220 0x7fa29c0d4870 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 -- 172.21.15.6:0/3047360467 >> 172.21.15.6:0/3047360467 conn(0x7fa29c0d3d50 msgr2=0x7fa29c0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 -- 172.21.15.6:0/3047360467 shutdown_connections 2024-03-06T07:30:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.844+0000 7fa29b200640 1 -- 172.21.15.6:0/3047360467 wait complete. 2024-03-06T07:30:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.846+0000 7fa29b200640 1 Processor -- start 2024-03-06T07:30:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.846+0000 7fa29b200640 1 -- start start 2024-03-06T07:30:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa29b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d1920 0x7fa29c0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa29b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 0x7fa29c1578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa29b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c15e220 0x7fa29c15fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa29b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0b3900 con 0x7fa29c0d2520 2024-03-06T07:30:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa29b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0cfa60 con 0x7fa29c15e220 2024-03-06T07:30:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa29b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0d0c60 con 0x7fa29c0d1920 2024-03-06T07:30:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa291400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 0x7fa29c1578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa291400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 0x7fa29c1578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58852/0 (socket says 172.21.15.6:58852) 2024-03-06T07:30:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.847+0000 7fa291400640 1 -- 172.21.15.6:0/4248388361 learned_addr learned my addr 172.21.15.6:0/4248388361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa299e00640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d1920 0x7fa29c0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa29a800640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c15e220 0x7fa29c15fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa291400640 1 -- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d1920 msgr2=0x7fa29c0c7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa291400640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d1920 0x7fa29c0c7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa291400640 1 -- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c15e220 msgr2=0x7fa29c15fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa291400640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c15e220 0x7fa29c15fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa291400640 1 -- 172.21.15.6:0/4248388361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c1609d0 con 0x7fa29c0d2520 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa29a800640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c15e220 0x7fa29c15fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa291400640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 0x7fa29c1578e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa28400d830 tx=0x7fa28400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400b6c0 con 0x7fa29c0d2520 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa284019070 con 0x7fa29c0d2520 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c160cb0 con 0x7fa29c0d2520 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.848+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400ba80 con 0x7fa29c0d2520 2024-03-06T07:30:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.849+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c161270 con 0x7fa29c0d2520 2024-03-06T07:30:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.850+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c161440 con 0x7fa29c0d2520 2024-03-06T07:30:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.851+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa28401e460 con 0x7fa29c0d2520 2024-03-06T07:30:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.852+0000 7fa292a00640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa27806b4d0 0x7fa27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.852+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa28408a730 con 0x7fa29c0d2520 2024-03-06T07:30:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.852+0000 7fa299e00640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa27806b4d0 0x7fa27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.854+0000 7fa299e00640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa27806b4d0 0x7fa27806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa28c035000 tx=0x7fa28c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:19.857+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa284059780 con 0x7fa29c0d2520 2024-03-06T07:30:20.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.062+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa29c158480 con 0x7fa29c0d2520 2024-03-06T07:30:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.592+0000 7fa292a00640 1 -- 172.21.15.6:0/4248388361 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa284059080 con 0x7fa29c0d2520 2024-03-06T07:30:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:30:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.597+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa27806b4d0 msgr2=0x7fa27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.597+0000 7fa29b200640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa27806b4d0 0x7fa27806d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa28c035000 tx=0x7fa28c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:20.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.597+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 msgr2=0x7fa29c1578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.597+0000 7fa29b200640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 0x7fa29c1578e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa28400d830 tx=0x7fa28400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 shutdown_connections 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa27806b4d0 0x7fa27806d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa29c15e220 0x7fa29c15fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa29c0d2520 0x7fa29c1578e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 --2- 172.21.15.6:0/4248388361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa29c0d1920 0x7fa29c0c7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 >> 172.21.15.6:0/4248388361 conn(0x7fa29c0d3d50 msgr2=0x7fa29c0cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 shutdown_connections 2024-03-06T07:30:20.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.598+0000 7fa29b200640 1 -- 172.21.15.6:0/4248388361 wait complete. 2024-03-06T07:30:20.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:30:20.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.819+0000 7f9f72600640 1 Processor -- start 2024-03-06T07:30:20.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.820+0000 7f9f72600640 1 -- start start 2024-03-06T07:30:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f72600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9f6c1583e0 0x7f9f6c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f72600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f6c06e9d0 0x7f9f6c0ade40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f72600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c0b4780 con 0x7f9f6c0b4170 2024-03-06T07:30:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f72600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c0b4a60 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f72600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c0b4d40 con 0x7f9f6c1583e0 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f71c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f6c06e9d0 0x7f9f6c0ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f71c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f6c06e9d0 0x7f9f6c0ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58862/0 (socket says 172.21.15.6:58862) 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.821+0000 7f9f71c00640 1 -- 172.21.15.6:0/3454456846 learned_addr learned my addr 172.21.15.6:0/3454456846 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.822+0000 7f9f71200640 1 --2- 172.21.15.6:0/3454456846 >> v2:172.21.15.178:3300/0 conn(0x7f9f6c1583e0 0x7f9f6c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.822+0000 7f9f71c00640 1 -- 172.21.15.6:0/3454456846 >> v2:172.21.15.178:3300/0 conn(0x7f9f6c1583e0 msgr2=0x7f9f6c0b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.822+0000 7f9f71c00640 1 --2- 172.21.15.6:0/3454456846 >> v2:172.21.15.178:3300/0 conn(0x7f9f6c1583e0 0x7f9f6c0b3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.822+0000 7f9f71c00640 1 -- 172.21.15.6:0/3454456846 >> v1:172.21.15.178:6789/0 conn(0x7f9f6c0b4170 legacy=0x7f9f6c06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.822+0000 7f9f71c00640 1 -- 172.21.15.6:0/3454456846 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0b5020 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.823+0000 7f9f71c00640 1 --2- 172.21.15.6:0/3454456846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c0ade40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c06f170 tx=0x7f9f6402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da700747f81f635d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.823+0000 7f9f6b400640 1 -- 172.21.15.6:0/3454456846 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64035020 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.823+0000 7f9f6b400640 1 -- 172.21.15.6:0/3454456846 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64031870 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.823+0000 7f9f6b400640 1 -- 172.21.15.6:0/3454456846 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64031c40 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 -- 172.21.15.6:0/3454456846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 msgr2=0x7f9f6c0ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 --2- 172.21.15.6:0/3454456846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c0ade40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c06f170 tx=0x7f9f6402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:20.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 -- 172.21.15.6:0/3454456846 shutdown_connections 2024-03-06T07:30:20.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 --2- 172.21.15.6:0/3454456846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c0ade40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 --2- 172.21.15.6:0/3454456846 >> v2:172.21.15.178:3300/0 conn(0x7f9f6c1583e0 0x7f9f6c0b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 -- 172.21.15.6:0/3454456846 >> 172.21.15.6:0/3454456846 conn(0x7f9f6c06b6c0 msgr2=0x7f9f6c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 -- 172.21.15.6:0/3454456846 shutdown_connections 2024-03-06T07:30:20.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.824+0000 7f9f72600640 1 -- 172.21.15.6:0/3454456846 wait complete. 2024-03-06T07:30:20.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.826+0000 7f9f72600640 1 Processor -- start 2024-03-06T07:30:20.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.826+0000 7f9f72600640 1 -- start start 2024-03-06T07:30:20.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f72600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c144810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f72600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f6c0b4170 0x7f9f6c158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f72600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f6c1583e0 0x7f9f6c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f72600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c15e5e0 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f72600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1587c0 con 0x7f9f6c1583e0 2024-03-06T07:30:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f72600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c15e760 con 0x7f9f6c0b4170 2024-03-06T07:30:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f71200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f71200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c144810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58866/0 (socket says 172.21.15.6:58866) 2024-03-06T07:30:20.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.827+0000 7f9f71200640 1 -- 172.21.15.6:0/1710660674 learned_addr learned my addr 172.21.15.6:0/1710660674 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71c00640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f6c1583e0 0x7f9f6c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f6be00640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f6c0b4170 0x7f9f6c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71200640 1 -- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f6c0b4170 msgr2=0x7f9f6c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71200640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f6c0b4170 0x7f9f6c158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71200640 1 -- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f6c1583e0 msgr2=0x7f9f6c15fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71200640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f6c1583e0 0x7f9f6c15fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71200640 1 -- 172.21.15.6:0/1710660674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c144d50 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71c00640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f6c1583e0 0x7f9f6c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.828+0000 7f9f71200640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c144810 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c007e40 tx=0x7f9f5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.829+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c014620 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.829+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c019070 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.829+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c014a00 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.829+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c159190 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.829+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c159750 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.830+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c0cf230 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.831+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f5c014ba0 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.832+0000 7f9f68c00640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4c06b520 0x7f9f4c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.832+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9f5c08aa60 con 0x7f9f6c06e9d0 2024-03-06T07:30:20.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.833+0000 7f9f6be00640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4c06b520 0x7f9f4c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.834+0000 7f9f6be00640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4c06b520 0x7f9f4c06d9e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c155370 tx=0x7f9f58009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:20.838+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f6c0cf230 con 0x7f9f6c06e9d0 2024-03-06T07:30:21.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.086+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9f6c04f330 con 0x7f9f6c06e9d0 2024-03-06T07:30:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.633+0000 7f9f68c00640 1 -- 172.21.15.6:0/1710660674 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v32) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9f6c144d50 con 0x7f9f6c06e9d0 2024-03-06T07:30:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:30:21.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4c06b520 msgr2=0x7f9f4c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4c06b520 0x7f9f4c06d9e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c155370 tx=0x7f9f58009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 msgr2=0x7f9f6c144810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c144810 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c007e40 tx=0x7f9f5c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 shutdown_connections 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4c06b520 0x7f9f4c06d9e0 secure :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c155370 tx=0x7f9f58009040 comp rx=0 tx=0).stop 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f6c1583e0 0x7f9f6c15fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f6c0b4170 0x7f9f6c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 --2- 172.21.15.6:0/1710660674 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f6c06e9d0 0x7f9f6c144810 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.640+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 >> 172.21.15.6:0/1710660674 conn(0x7f9f6c06b6c0 msgr2=0x7f9f6c0ae5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.641+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 shutdown_connections 2024-03-06T07:30:21.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.641+0000 7f9f72600640 1 -- 172.21.15.6:0/1710660674 wait complete. 2024-03-06T07:30:21.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:21.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.858+0000 7fa37ea00640 1 Processor -- start 2024-03-06T07:30:21.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.859+0000 7fa37ea00640 1 -- start start 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa37ea00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa3780af580 0x7fa37815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa37ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa3780cdc20 0x7fa3780b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa37ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37815cc40 con 0x7fa3780af1a0 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa37ea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37815cf20 con 0x7fa3780cdc20 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa37ea00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37815d200 con 0x7fa3780af580 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa377e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa3780af1a0 0x7fa3780b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55648/0 (socket says 172.21.15.6:55648) 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa377e00640 1 -- 172.21.15.6:0/2488476246 learned_addr learned my addr 172.21.15.6:0/2488476246 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.860+0000 7fa37ca00640 1 --2- 172.21.15.6:0/2488476246 >> v2:172.21.15.6:3300/0 conn(0x7fa3780cdc20 0x7fa3780b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa377400640 1 --2- 172.21.15.6:0/2488476246 >> v2:172.21.15.178:3300/0 conn(0x7fa3780af580 0x7fa37815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa376a00640 1 -- 172.21.15.6:0/2488476246 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862582051 0 0) 0x7fa37815cc40 con 0x7fa3780af1a0 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa376a00640 1 -- 172.21.15.6:0/2488476246 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa358003520 con 0x7fa3780af1a0 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa37ca00640 1 -- 172.21.15.6:0/2488476246 >> v2:172.21.15.178:3300/0 conn(0x7fa3780af580 msgr2=0x7fa37815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa37ca00640 1 --2- 172.21.15.6:0/2488476246 >> v2:172.21.15.178:3300/0 conn(0x7fa3780af580 0x7fa37815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa37ca00640 1 -- 172.21.15.6:0/2488476246 >> v1:172.21.15.6:6789/0 conn(0x7fa3780af1a0 legacy=0x7fa3780b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa37ca00640 1 -- 172.21.15.6:0/2488476246 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37815d4e0 con 0x7fa3780cdc20 2024-03-06T07:30:21.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.861+0000 7fa377400640 1 --2- 172.21.15.6:0/2488476246 >> v2:172.21.15.178:3300/0 conn(0x7fa3780af580 0x7fa37815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.862+0000 7fa37ca00640 1 --2- 172.21.15.6:0/2488476246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780cdc20 0x7fa3780b2dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa3780b54c0 tx=0x7fa37002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c863aea0ef303b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.862+0000 7fa376a00640 1 -- 172.21.15.6:0/2488476246 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3700315a0 con 0x7fa3780cdc20 2024-03-06T07:30:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.862+0000 7fa376a00640 1 -- 172.21.15.6:0/2488476246 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa370037070 con 0x7fa3780cdc20 2024-03-06T07:30:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa376a00640 1 -- 172.21.15.6:0/2488476246 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370031940 con 0x7fa3780cdc20 2024-03-06T07:30:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 -- 172.21.15.6:0/2488476246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780cdc20 msgr2=0x7fa3780b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2488476246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780cdc20 0x7fa3780b2dd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa3780b54c0 tx=0x7fa37002b840 comp rx=0 tx=0).stop 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 -- 172.21.15.6:0/2488476246 shutdown_connections 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2488476246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780cdc20 0x7fa3780b2dd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2488476246 >> v2:172.21.15.178:3300/0 conn(0x7fa3780af580 0x7fa37815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 -- 172.21.15.6:0/2488476246 >> 172.21.15.6:0/2488476246 conn(0x7fa37806afb0 msgr2=0x7fa37806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 -- 172.21.15.6:0/2488476246 shutdown_connections 2024-03-06T07:30:21.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.863+0000 7fa37ea00640 1 -- 172.21.15.6:0/2488476246 wait complete. 2024-03-06T07:30:21.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.865+0000 7fa37ea00640 1 Processor -- start 2024-03-06T07:30:21.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.865+0000 7fa37ea00640 1 -- start start 2024-03-06T07:30:21.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3780af1a0 0x7fa37806cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 0x7fa37809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa37ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3780cdc20 0x7fa37809af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa37ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3780b36a0 con 0x7fa3780af580 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa37ea00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3780b3520 con 0x7fa3780af1a0 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa37ea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3780b3820 con 0x7fa3780cdc20 2024-03-06T07:30:21.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa377400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 0x7fa37809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa377400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 0x7fa37809aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33994/0 (socket says 172.21.15.6:33994) 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.866+0000 7fa377400640 1 -- 172.21.15.6:0/2265685995 learned_addr learned my addr 172.21.15.6:0/2265685995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:21.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377e00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3780af1a0 0x7fa37806cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377400640 1 -- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3780cdc20 msgr2=0x7fa37809af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377400640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3780cdc20 0x7fa37809af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377400640 1 -- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3780af1a0 msgr2=0x7fa37806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377400640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3780af1a0 0x7fa37806cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377400640 1 -- 172.21.15.6:0/2265685995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3780c92f0 con 0x7fa3780af580 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa377400640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 0x7fa37809aa10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa36000cf30 tx=0x7fa36000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.867+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36001a070 con 0x7fa3780af580 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.868+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36000d840 con 0x7fa3780af580 2024-03-06T07:30:21.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.868+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36000dc10 con 0x7fa3780af580 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.868+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3780c95d0 con 0x7fa3780af580 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.868+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37806d050 con 0x7fa3780af580 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.869+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa378069510 con 0x7fa3780af580 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.870+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa36001d070 con 0x7fa3780af580 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.871+0000 7fa36fe00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa35406b4d0 0x7fa35406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.871+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa36008aa50 con 0x7fa3780af580 2024-03-06T07:30:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.871+0000 7fa377e00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa35406b4d0 0x7fa35406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.872+0000 7fa377e00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa35406b4d0 0x7fa35406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa3680011e0 tx=0x7fa368009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:21.875+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3600591e0 con 0x7fa3780af580 2024-03-06T07:30:21.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:21.972 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:22.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.081+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa37804f3a0 con 0x7fa3780af580 2024-03-06T07:30:22.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.083+0000 7fa36fe00640 1 -- 172.21.15.6:0/2265685995 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16606 (secure 0 0 0) 0x7fa36005d280 con 0x7fa3780af580 2024-03-06T07:30:22.085 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:22.085 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":57,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:21.088482+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa35406b4d0 msgr2=0x7fa35406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa35406b4d0 0x7fa35406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa3680011e0 tx=0x7fa368009040 comp rx=0 tx=0).stop 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 msgr2=0x7fa37809aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 0x7fa37809aa10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa36000cf30 tx=0x7fa36000a7d0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 shutdown_connections 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa35406b4d0 0x7fa35406d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa3780cdc20 0x7fa37809af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa3780af580 0x7fa37809aa10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 --2- 172.21.15.6:0/2265685995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa3780af1a0 0x7fa37806cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.085+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 >> 172.21.15.6:0/2265685995 conn(0x7fa37806afb0 msgr2=0x7fa3780cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.086+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 shutdown_connections 2024-03-06T07:30:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.086+0000 7fa37ea00640 1 -- 172.21.15.6:0/2265685995 wait complete. 2024-03-06T07:30:22.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.358+0000 7fcf99600640 1 Processor -- start 2024-03-06T07:30:22.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.358+0000 7fcf99600640 1 -- start start 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.359+0000 7fcf99600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcf940af1a0 0x7fcf9415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.359+0000 7fcf99600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9415cc40 con 0x7fcf940af580 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.359+0000 7fcf99600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9415cf20 con 0x7fcf940cdc20 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.359+0000 7fcf99600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9415d200 con 0x7fcf940af1a0 2024-03-06T07:30:22.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.359+0000 7fcf8a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcf940af580 0x7fcf940b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55664/0 (socket says 172.21.15.6:55664) 2024-03-06T07:30:22.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.359+0000 7fcf8a000640 1 -- 172.21.15.6:0/3899438998 learned_addr learned my addr 172.21.15.6:0/3899438998 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92a00640 1 --2- 172.21.15.6:0/3899438998 >> v2:172.21.15.178:3300/0 conn(0x7fcf940af1a0 0x7fcf9415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632544598 0 0) 0x7fcf9415cc40 con 0x7fcf940af580 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf78003520 con 0x7fcf940af580 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030004878 0 0) 0x7fcf78003520 con 0x7fcf940af580 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 >> v2:172.21.15.178:3300/0 conn(0x7fcf940af1a0 msgr2=0x7fcf9415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 --2- 172.21.15.6:0/3899438998 >> v2:172.21.15.178:3300/0 conn(0x7fcf940af1a0 0x7fcf9415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 >> v1:172.21.15.178:6789/0 conn(0x7fcf940cdc20 legacy=0x7fcf940b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9415d4e0 con 0x7fcf940af580 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fcf7c001670 con 0x7fcf940af580 2024-03-06T07:30:22.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 >> v1:172.21.15.6:6789/0 conn(0x7fcf940af580 legacy=0x7fcf940b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf78003a20 0x7fcf78003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf78004800 0x7fcf78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf780052e0 0x7fcf7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415d200 con 0x7fcf78004800 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7800c230 con 0x7fcf780052e0 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.360+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7800c430 con 0x7fcf78003a20 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf8a000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf78004800 0x7fcf78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf93400640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf780052e0 0x7fcf7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf92a00640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf78003a20 0x7fcf78003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf8a000640 1 -- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf78003a20 msgr2=0x7fcf78003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf8a000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf78003a20 0x7fcf78003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf8a000640 1 -- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf780052e0 msgr2=0x7fcf7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf8a000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf780052e0 0x7fcf7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf8a000640 1 -- 172.21.15.6:0/3899438998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78003520 con 0x7fcf78004800 2024-03-06T07:30:22.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.361+0000 7fcf93400640 1 -- 172.21.15.6:0/3899438998 reap_dead start 2024-03-06T07:30:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.362+0000 7fcf8a000640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf78004800 0x7fcf78004c80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c001eb0 tx=0x7fcf78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3694146a0d40dfec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c007d30 con 0x7fcf78004800 2024-03-06T07:30:22.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c030070 con 0x7fcf78004800 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf92000640 1 -- 172.21.15.6:0/3899438998 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c005d70 con 0x7fcf78004800 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 -- 172.21.15.6:0/3899438998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf78004800 msgr2=0x7fcf78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf78004800 0x7fcf78004c80 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c001eb0 tx=0x7fcf78005830 comp rx=0 tx=0).stop 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 -- 172.21.15.6:0/3899438998 shutdown_connections 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 --2- 172.21.15.6:0/3899438998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf78004800 0x7fcf78004c80 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 -- 172.21.15.6:0/3899438998 >> 172.21.15.6:0/3899438998 conn(0x7fcf9406afb0 msgr2=0x7fcf9406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 -- 172.21.15.6:0/3899438998 shutdown_connections 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.363+0000 7fcf99600640 1 -- 172.21.15.6:0/3899438998 wait complete. 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.365+0000 7fcf99600640 1 Processor -- start 2024-03-06T07:30:22.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.365+0000 7fcf99600640 1 -- start start 2024-03-06T07:30:22.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf99600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf78004800 0x7fcf94158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf99600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 0x7fcf940cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf99600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf940ccc70 0x7fcf941f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf99600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415c990 con 0x7fcf9406e790 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf99600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415c810 con 0x7fcf78004800 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf99600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940b4c20 con 0x7fcf940ccc70 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf8a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 0x7fcf940cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf8a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 0x7fcf940cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34008/0 (socket says 172.21.15.6:34008) 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.366+0000 7fcf8a000640 1 -- 172.21.15.6:0/374586159 learned_addr learned my addr 172.21.15.6:0/374586159 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf93400640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf940ccc70 0x7fcf941f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf92a00640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf78004800 0x7fcf94158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf8a000640 1 -- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf940ccc70 msgr2=0x7fcf941f3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf8a000640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf940ccc70 0x7fcf941f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf8a000640 1 -- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf78004800 msgr2=0x7fcf94158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf8a000640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf78004800 0x7fcf94158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf8a000640 1 -- 172.21.15.6:0/374586159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9406d9f0 con 0x7fcf9406e790 2024-03-06T07:30:22.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.367+0000 7fcf8a000640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 0x7fcf940cc730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c001970 tx=0x7fcf7c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.368+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c046070 con 0x7fcf9406e790 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.368+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c000ed0 con 0x7fcf9406e790 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.368+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c004e10 con 0x7fcf9406e790 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.368+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf94158da0 con 0x7fcf9406e790 2024-03-06T07:30:22.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.368+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9406dcb0 con 0x7fcf9406e790 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.369+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf940cd080 con 0x7fcf9406e790 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.370+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf7c007070 con 0x7fcf9406e790 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.371+0000 7fcf8b400640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf7006b500 0x7fcf7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.371+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf7c0ab5c0 con 0x7fcf9406e790 2024-03-06T07:30:22.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.371+0000 7fcf92a00640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf7006b500 0x7fcf7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.373+0000 7fcf92a00640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf7006b500 0x7fcf7006d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcf94050c30 tx=0x7fcf8400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.375+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf940cd080 con 0x7fcf9406e790 2024-03-06T07:30:22.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.576+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf9415c1a0 con 0x7fcf9406e790 2024-03-06T07:30:22.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.579+0000 7fcf8b400640 1 -- 172.21.15.6:0/374586159 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16606 (secure 0 0 0) 0x7fcf7c079d50 con 0x7fcf9406e790 2024-03-06T07:30:22.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:22.581 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":57,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:21.088482+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf7006b500 msgr2=0x7fcf7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf7006b500 0x7fcf7006d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcf94050c30 tx=0x7fcf8400b040 comp rx=0 tx=0).stop 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 msgr2=0x7fcf940cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 0x7fcf940cc730 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c001970 tx=0x7fcf7c02e070 comp rx=0 tx=0).stop 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 shutdown_connections 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf7006b500 0x7fcf7006d9c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf940ccc70 0x7fcf941f3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf9406e790 0x7fcf940cc730 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 --2- 172.21.15.6:0/374586159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf78004800 0x7fcf94158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.581+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 >> 172.21.15.6:0/374586159 conn(0x7fcf9406afb0 msgr2=0x7fcf9415d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.582+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 shutdown_connections 2024-03-06T07:30:22.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.582+0000 7fcf99600640 1 -- 172.21.15.6:0/374586159 wait complete. 2024-03-06T07:30:22.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:30:22.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.804+0000 7f2915000640 1 Processor -- start 2024-03-06T07:30:22.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.804+0000 7f2915000640 1 -- start start 2024-03-06T07:30:22.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f2915000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f291015bf20 0x7f29100acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f2915000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2910071e20 0x7f29100ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f2915000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2910072df0 con 0x7f2910072520 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f2915000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29100cdab0 con 0x7f291015bf20 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f2915000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29100b4d70 con 0x7f2910071e20 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f290de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f291015bf20 0x7f29100acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f290de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f291015bf20 0x7f29100acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34016/0 (socket says 172.21.15.6:34016) 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f290de00640 1 -- 172.21.15.6:0/4131404169 learned_addr learned my addr 172.21.15.6:0/4131404169 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f290e800640 1 --2- 172.21.15.6:0/4131404169 >> v2:172.21.15.178:3300/0 conn(0x7f2910071e20 0x7f29100ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f290de00640 1 -- 172.21.15.6:0/4131404169 >> v2:172.21.15.178:3300/0 conn(0x7f2910071e20 msgr2=0x7f29100ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.806+0000 7f290de00640 1 --2- 172.21.15.6:0/4131404169 >> v2:172.21.15.178:3300/0 conn(0x7f2910071e20 0x7f29100ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.807+0000 7f290de00640 1 -- 172.21.15.6:0/4131404169 >> v1:172.21.15.178:6789/0 conn(0x7f2910072520 legacy=0x7f29100cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.807+0000 7f290de00640 1 -- 172.21.15.6:0/4131404169 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29100b5050 con 0x7f291015bf20 2024-03-06T07:30:22.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.808+0000 7f290de00640 1 --2- 172.21.15.6:0/4131404169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015bf20 0x7f29100acd40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f29100af200 tx=0x7f290002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4504b1ce5a591cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.808+0000 7f290ca00640 1 -- 172.21.15.6:0/4131404169 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900035020 con 0x7f291015bf20 2024-03-06T07:30:22.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.808+0000 7f290ca00640 1 -- 172.21.15.6:0/4131404169 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2900031870 con 0x7f291015bf20 2024-03-06T07:30:22.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.808+0000 7f290ca00640 1 -- 172.21.15.6:0/4131404169 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900031c40 con 0x7f291015bf20 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.808+0000 7f2915000640 1 -- 172.21.15.6:0/4131404169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015bf20 msgr2=0x7f29100acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.808+0000 7f2915000640 1 --2- 172.21.15.6:0/4131404169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015bf20 0x7f29100acd40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f29100af200 tx=0x7f290002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.809+0000 7f2915000640 1 -- 172.21.15.6:0/4131404169 shutdown_connections 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.809+0000 7f2915000640 1 --2- 172.21.15.6:0/4131404169 >> v2:172.21.15.178:3300/0 conn(0x7f2910071e20 0x7f29100ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.809+0000 7f2915000640 1 --2- 172.21.15.6:0/4131404169 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f291015bf20 0x7f29100acd40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.809+0000 7f2915000640 1 -- 172.21.15.6:0/4131404169 >> 172.21.15.6:0/4131404169 conn(0x7f291006b3d0 msgr2=0x7f291006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.809+0000 7f2915000640 1 -- 172.21.15.6:0/4131404169 shutdown_connections 2024-03-06T07:30:22.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.809+0000 7f2915000640 1 -- 172.21.15.6:0/4131404169 wait complete. 2024-03-06T07:30:22.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.810+0000 7f2915000640 1 Processor -- start 2024-03-06T07:30:22.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 -- start start 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2910071e20 0x7f29101f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 0x7f2910150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f291015bf20 0x7f29100b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100b4b10 con 0x7f2910072520 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100b4990 con 0x7f2910071e20 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.811+0000 7f2915000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29100ce6c0 con 0x7f291015bf20 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 0x7f2910150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 0x7f2910150f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34022/0 (socket says 172.21.15.6:34022) 2024-03-06T07:30:22.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 -- 172.21.15.6:0/2508013167 learned_addr learned my addr 172.21.15.6:0/2508013167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290de00640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2910071e20 0x7f29101f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290e800640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f291015bf20 0x7f29100b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 -- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f291015bf20 msgr2=0x7f29100b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f291015bf20 0x7f29100b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 -- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2910071e20 msgr2=0x7f29101f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2910071e20 0x7f29101f7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290d400640 1 -- 172.21.15.6:0/2508013167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29100b0b70 con 0x7f2910072520 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.812+0000 7f290de00640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2910071e20 0x7f29101f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.813+0000 7f290d400640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 0x7f2910150f20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00b3e0 tx=0x7f28fc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.813+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc015020 con 0x7f2910072520 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.813+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28fc011850 con 0x7f2910072520 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.813+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29100b0e50 con 0x7f2910072520 2024-03-06T07:30:22.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.813+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28fc011c40 con 0x7f2910072520 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.813+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29100b1410 con 0x7f2910072520 2024-03-06T07:30:22.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.814+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291004f330 con 0x7f2910072520 2024-03-06T07:30:22.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.816+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28fc018020 con 0x7f2910072520 2024-03-06T07:30:22.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.818+0000 7f2906000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28f006b520 0x7f28f006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.818+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28fc08a1b0 con 0x7f2910072520 2024-03-06T07:30:22.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.818+0000 7f290de00640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28f006b520 0x7f28f006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:22.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.819+0000 7f290de00640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28f006b520 0x7f28f006d9e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2900036000 tx=0x7f2900008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:22.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:22.822+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28fc01f020 con 0x7f2910072520 2024-03-06T07:30:23.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.063+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f291015f720 con 0x7f2910072520 2024-03-06T07:30:23.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.652+0000 7f2906000640 1 -- 172.21.15.6:0/2508013167 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7f28fc0172c0 con 0x7f2910072520 2024-03-06T07:30:23.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.655+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28f006b520 msgr2=0x7f28f006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.655+0000 7f2915000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28f006b520 0x7f28f006d9e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2900036000 tx=0x7f2900008030 comp rx=0 tx=0).stop 2024-03-06T07:30:23.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.655+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 msgr2=0x7f2910150f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.655+0000 7f2915000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 0x7f2910150f20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00b3e0 tx=0x7f28fc00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 shutdown_connections 2024-03-06T07:30:23.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28f006b520 0x7f28f006d9e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f291015bf20 0x7f29100b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2910072520 0x7f2910150f20 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 --2- 172.21.15.6:0/2508013167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2910071e20 0x7f29101f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 >> 172.21.15.6:0/2508013167 conn(0x7f291006b3d0 msgr2=0x7f29100b2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 shutdown_connections 2024-03-06T07:30:23.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.656+0000 7f2915000640 1 -- 172.21.15.6:0/2508013167 wait complete. 2024-03-06T07:30:23.675 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:30:23.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:30:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.871+0000 7f73a9c00640 1 Processor -- start 2024-03-06T07:30:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.871+0000 7f73a9c00640 1 -- start start 2024-03-06T07:30:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a9c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f73a40b2250 0x7f73a40b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a9c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a4071930 con 0x7f73a40729d0 2024-03-06T07:30:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a9c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a4071c10 con 0x7f73a40b3c20 2024-03-06T07:30:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a9c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a415bfb0 con 0x7f73a40b2250 2024-03-06T07:30:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f73a40729d0 0x7f73a40b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55666/0 (socket says 172.21.15.6:55666) 2024-03-06T07:30:23.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a3400640 1 -- 172.21.15.6:0/2756198792 learned_addr learned my addr 172.21.15.6:0/2756198792 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a9200640 1 --2- 172.21.15.6:0/2756198792 >> v2:172.21.15.178:3300/0 conn(0x7f73a40b2250 0x7f73a40b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.873+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924287346 0 0) 0x7f73a4071930 con 0x7f73a40729d0 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7388003520 con 0x7f73a40729d0 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1270148867 0 0) 0x7f7388003520 con 0x7f73a40729d0 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 >> v2:172.21.15.178:3300/0 conn(0x7f73a40b2250 msgr2=0x7f73a40b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 --2- 172.21.15.6:0/2756198792 >> v2:172.21.15.178:3300/0 conn(0x7f73a40b2250 0x7f73a40b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 >> v1:172.21.15.178:6789/0 conn(0x7f73a40b3c20 legacy=0x7f73a415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a4071ef0 con 0x7f73a40729d0 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7390001670 con 0x7f73a40729d0 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 >> v1:172.21.15.6:6789/0 conn(0x7f73a40729d0 legacy=0x7f73a40b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7388003a00 0x7f7388003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7388004800 0x7f7388004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f738800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a415bfb0 con 0x7f73880051c0 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738800c0c0 con 0x7f7388004800 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738800c2c0 con 0x7f7388003a00 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.874+0000 7f73a9200640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f738800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a3400640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7388004800 0x7f7388004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a3e00640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7388003a00 0x7f7388003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a9200640 1 -- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7388003a00 msgr2=0x7f7388003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a9200640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7388003a00 0x7f7388003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a9200640 1 -- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7388004800 msgr2=0x7f7388004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a9200640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7388004800 0x7f7388004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.875+0000 7f73a9200640 1 -- 172.21.15.6:0/2756198792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7388003520 con 0x7f73880051c0 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.876+0000 7f73a9200640 1 -- 172.21.15.6:0/2756198792 reap_dead start 2024-03-06T07:30:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.876+0000 7f73a9200640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f738800b9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f739c009fc0 tx=0x7f7388005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5959b676bd00fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.876+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c009c10 con 0x7f73880051c0 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.876+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c02cca0 con 0x7f73880051c0 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.876+0000 7f73a2a00640 1 -- 172.21.15.6:0/2756198792 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c003bd0 con 0x7f73880051c0 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 -- 172.21.15.6:0/2756198792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 msgr2=0x7f738800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f738800b9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f739c009fc0 tx=0x7f7388005830 comp rx=0 tx=0).stop 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 -- 172.21.15.6:0/2756198792 shutdown_connections 2024-03-06T07:30:23.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 --2- 172.21.15.6:0/2756198792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f738800b9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 -- 172.21.15.6:0/2756198792 >> 172.21.15.6:0/2756198792 conn(0x7f73a406b3a0 msgr2=0x7f73a406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 -- 172.21.15.6:0/2756198792 shutdown_connections 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.877+0000 7f73a9c00640 1 -- 172.21.15.6:0/2756198792 wait complete. 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.878+0000 7f73a9c00640 1 Processor -- start 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 -- start start 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f73a4156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73a4156670 0x7f73a41f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73a41f70d0 0x7f73a41f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40b4000 con 0x7f73880051c0 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40724b0 con 0x7f73a41f70d0 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.879+0000 7f73a9c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40b4180 con 0x7f73a4156670 2024-03-06T07:30:23.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f73a4156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f73a4156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34038/0 (socket says 172.21.15.6:34038) 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 -- 172.21.15.6:0/4189875522 learned_addr learned my addr 172.21.15.6:0/4189875522 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3400640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73a4156670 0x7f73a41f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a9200640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73a41f70d0 0x7f73a41f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 -- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73a4156670 msgr2=0x7f73a41f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73a4156670 0x7f73a41f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 -- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73a41f70d0 msgr2=0x7f73a41f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73a41f70d0 0x7f73a41f7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3e00640 1 -- 172.21.15.6:0/4189875522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a4156e40 con 0x7f73880051c0 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.880+0000 7f73a3400640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73a4156670 0x7f73a41f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.881+0000 7f73a3e00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f73a4156130 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f739400e4c0 tx=0x7f739400e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.881+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940135f0 con 0x7f73880051c0 2024-03-06T07:30:23.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.881+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739401a070 con 0x7f73880051c0 2024-03-06T07:30:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.881+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940139d0 con 0x7f73880051c0 2024-03-06T07:30:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.881+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a4157120 con 0x7f73880051c0 2024-03-06T07:30:23.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.881+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a41f4800 con 0x7f73880051c0 2024-03-06T07:30:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.882+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a404f330 con 0x7f73880051c0 2024-03-06T07:30:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.883+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7394013b70 con 0x7f73880051c0 2024-03-06T07:30:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.884+0000 7f739be00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f738406b500 0x7f738406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.884+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7394089ea0 con 0x7f73880051c0 2024-03-06T07:30:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.884+0000 7f73a3400640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f738406b500 0x7f738406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.885+0000 7f73a3400640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f738406b500 0x7f738406d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7390001eb0 tx=0x7f7390000f50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:23.888+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7394023020 con 0x7f73880051c0 2024-03-06T07:30:24.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.127+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f73a415c760 con 0x7f73880051c0 2024-03-06T07:30:24.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.655+0000 7f739be00640 1 -- 172.21.15.6:0/4189875522 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7394058c80 con 0x7f73880051c0 2024-03-06T07:30:24.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f738406b500 msgr2=0x7f738406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f738406b500 0x7f738406d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7390001eb0 tx=0x7f7390000f50 comp rx=0 tx=0).stop 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 msgr2=0x7f73a4156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f73a4156130 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f739400e4c0 tx=0x7f739400e980 comp rx=0 tx=0).stop 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 shutdown_connections 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f738406b500 0x7f738406d9c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f73a41f70d0 0x7f73a41f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73a4156670 0x7f73a41f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 --2- 172.21.15.6:0/4189875522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73880051c0 0x7f73a4156130 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 >> 172.21.15.6:0/4189875522 conn(0x7f73a406b3a0 msgr2=0x7f73a40b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 shutdown_connections 2024-03-06T07:30:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.659+0000 7f73a9c00640 1 -- 172.21.15.6:0/4189875522 wait complete. 2024-03-06T07:30:24.685 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:30:24.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.899+0000 7f9e2cc00640 1 Processor -- start 2024-03-06T07:30:24.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.899+0000 7f9e2cc00640 1 -- start start 2024-03-06T07:30:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.900+0000 7f9e2cc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9e280d0e50 0x7f9e28168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.900+0000 7f9e2cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280b3710 con 0x7f9e280d4a10 2024-03-06T07:30:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e2cc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280b39f0 con 0x7f9e2815e0b0 2024-03-06T07:30:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e2cc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280b09e0 con 0x7f9e280d0e50 2024-03-06T07:30:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e26a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9e280d4a10 0x7f9e280d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55680/0 (socket says 172.21.15.6:55680) 2024-03-06T07:30:24.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e26a00640 1 -- 172.21.15.6:0/1833124899 learned_addr learned my addr 172.21.15.6:0/1833124899 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e27e00640 1 --2- 172.21.15.6:0/1833124899 >> v2:172.21.15.178:3300/0 conn(0x7f9e280d0e50 0x7f9e28168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578286315 0 0) 0x7f9e280b3710 con 0x7f9e280d4a10 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.901+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c003520 con 0x7f9e280d4a10 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3412830948 0 0) 0x7f9e0c003520 con 0x7f9e280d4a10 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 >> v2:172.21.15.178:3300/0 conn(0x7f9e280d0e50 msgr2=0x7f9e28168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 --2- 172.21.15.6:0/1833124899 >> v2:172.21.15.178:3300/0 conn(0x7f9e280d0e50 0x7f9e28168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 >> v1:172.21.15.178:6789/0 conn(0x7f9e2815e0b0 legacy=0x7f9e280cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280b3cd0 con 0x7f9e280d4a10 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9e14001670 con 0x7f9e280d4a10 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 >> v1:172.21.15.6:6789/0 conn(0x7f9e280d4a10 legacy=0x7f9e280d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c0039b0 0x7f9e0c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c004800 0x7f9e0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e0c0051c0 0x7f9e0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c00c030 con 0x7f9e0c0039b0 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c00c1e0 con 0x7f9e0c0051c0 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c00c3e0 con 0x7f9e0c004800 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.902+0000 7f9e27400640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c0039b0 0x7f9e0c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27e00640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e0c0051c0 0x7f9e0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e26a00640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c004800 0x7f9e0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27400640 1 -- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c004800 msgr2=0x7f9e0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27400640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c004800 0x7f9e0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27400640 1 -- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e0c0051c0 msgr2=0x7f9e0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:24.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27400640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e0c0051c0 0x7f9e0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27400640 1 -- 172.21.15.6:0/1833124899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c003520 con 0x7f9e0c0039b0 2024-03-06T07:30:24.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.903+0000 7f9e27e00640 1 -- 172.21.15.6:0/1833124899 reap_dead start 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.905+0000 7f9e27400640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c0039b0 0x7f9e0c003dd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9e18001e50 tx=0x7f9e0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd265755cbcd6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.905+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e18001a70 con 0x7f9e0c0039b0 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.905+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18003b70 con 0x7f9e0c0039b0 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.905+0000 7f9e26000640 1 -- 172.21.15.6:0/1833124899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180050d0 con 0x7f9e0c0039b0 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.905+0000 7f9e2cc00640 1 -- 172.21.15.6:0/1833124899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c0039b0 msgr2=0x7f9e0c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.905+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c0039b0 0x7f9e0c003dd0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9e18001e50 tx=0x7f9e0c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.906+0000 7f9e2cc00640 1 -- 172.21.15.6:0/1833124899 shutdown_connections 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.906+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/1833124899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e0c0039b0 0x7f9e0c003dd0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.906+0000 7f9e2cc00640 1 -- 172.21.15.6:0/1833124899 >> 172.21.15.6:0/1833124899 conn(0x7f9e280d3d30 msgr2=0x7f9e280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:24.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.906+0000 7f9e2cc00640 1 -- 172.21.15.6:0/1833124899 shutdown_connections 2024-03-06T07:30:24.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.906+0000 7f9e2cc00640 1 -- 172.21.15.6:0/1833124899 wait complete. 2024-03-06T07:30:24.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.909+0000 7f9e2cc00640 1 Processor -- start 2024-03-06T07:30:24.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.909+0000 7f9e2cc00640 1 -- start start 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.910+0000 7f9e2cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c0039b0 0x7f9e280b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.910+0000 7f9e2cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 0x7f9e28160d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.910+0000 7f9e2cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e281612b0 0x7f9e281f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.910+0000 7f9e2cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280cfc60 con 0x7f9e280b6e90 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.910+0000 7f9e2cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2815e490 con 0x7f9e281612b0 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.910+0000 7f9e2cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280d0740 con 0x7f9e0c0039b0 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 0x7f9e28160d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 0x7f9e28160d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34050/0 (socket says 172.21.15.6:34050) 2024-03-06T07:30:24.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 -- 172.21.15.6:0/904997075 learned_addr learned my addr 172.21.15.6:0/904997075 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:24.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 -- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c0039b0 msgr2=0x7f9e280b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e27400640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c0039b0 0x7f9e280b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c0039b0 0x7f9e280b6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 -- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e281612b0 msgr2=0x7f9e281f6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e281612b0 0x7f9e281f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 -- 172.21.15.6:0/904997075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e280b9320 con 0x7f9e280b6e90 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.911+0000 7f9e26a00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 0x7f9e28160d70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400e780 tx=0x7f9e1400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.912+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400c6c0 con 0x7f9e280b6e90 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.912+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14018070 con 0x7f9e280b6e90 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.912+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e280b9600 con 0x7f9e280b6e90 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.912+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1400caa0 con 0x7f9e280b6e90 2024-03-06T07:30:24.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.912+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2806b9a0 con 0x7f9e280b6e90 2024-03-06T07:30:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.913+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2804f330 con 0x7f9e280b6e90 2024-03-06T07:30:24.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.914+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e1400cc40 con 0x7f9e280b6e90 2024-03-06T07:30:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.915+0000 7f9e1f400640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e0806b500 0x7f9e0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:24.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.915+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e14089de0 con 0x7f9e280b6e90 2024-03-06T07:30:24.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.916+0000 7f9e27400640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e0806b500 0x7f9e0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:24.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.917+0000 7f9e27400640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e0806b500 0x7f9e0806d9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9e18034000 tx=0x7f9e18009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:24.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:24.920+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e14058bc0 con 0x7f9e280b6e90 2024-03-06T07:30:25.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.160+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e280d2780 con 0x7f9e280b6e90 2024-03-06T07:30:25.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.161+0000 7f9e1f400640 1 -- 172.21.15.6:0/904997075 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f9e140584c0 con 0x7f9e280b6e90 2024-03-06T07:30:25.163 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:25.164 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:30:25.164 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e0806b500 msgr2=0x7f9e0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e0806b500 0x7f9e0806d9c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9e18034000 tx=0x7f9e18009030 comp rx=0 tx=0).stop 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 msgr2=0x7f9e28160d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 0x7f9e28160d70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400e780 tx=0x7f9e1400ec40 comp rx=0 tx=0).stop 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 shutdown_connections 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e0806b500 0x7f9e0806d9c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e281612b0 0x7f9e281f6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e280b6e90 0x7f9e28160d70 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 --2- 172.21.15.6:0/904997075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9e0c0039b0 0x7f9e280b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.164+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 >> 172.21.15.6:0/904997075 conn(0x7f9e280d3d30 msgr2=0x7f9e2815d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.165+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 shutdown_connections 2024-03-06T07:30:25.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.165+0000 7f9e2cc00640 1 -- 172.21.15.6:0/904997075 wait complete. 2024-03-06T07:30:25.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.395+0000 7f4b95e00640 1 Processor -- start 2024-03-06T07:30:25.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.395+0000 7f4b95e00640 1 -- start start 2024-03-06T07:30:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b95e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4b9015e200 0x7f4b900b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b95e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b900afdf0 0x7f4b900af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b95e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9006bec0 con 0x7f4b9006afb0 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b95e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b900b0670 con 0x7f4b900afdf0 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b95e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b900ae380 con 0x7f4b9015e200 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b900afdf0 0x7f4b900af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4b900afdf0 0x7f4b900af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34062/0 (socket says 172.21.15.6:34062) 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.396+0000 7f4b87e00640 1 -- 172.21.15.6:0/178185746 learned_addr learned my addr 172.21.15.6:0/178185746 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:25.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.397+0000 7f4b8ea00640 1 --2- 172.21.15.6:0/178185746 >> v2:172.21.15.178:3300/0 conn(0x7f4b9015e200 0x7f4b900b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.397+0000 7f4b87e00640 1 -- 172.21.15.6:0/178185746 >> v2:172.21.15.178:3300/0 conn(0x7f4b9015e200 msgr2=0x7f4b900b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.397+0000 7f4b87e00640 1 --2- 172.21.15.6:0/178185746 >> v2:172.21.15.178:3300/0 conn(0x7f4b9015e200 0x7f4b900b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.397+0000 7f4b87e00640 1 -- 172.21.15.6:0/178185746 >> v1:172.21.15.178:6789/0 conn(0x7f4b9006afb0 legacy=0x7f4b9006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.397+0000 7f4b87e00640 1 -- 172.21.15.6:0/178185746 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b900ae660 con 0x7f4b900afdf0 2024-03-06T07:30:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.399+0000 7f4b87e00640 1 --2- 172.21.15.6:0/178185746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b900afdf0 0x7f4b900af140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4b90071cc0 tx=0x7f4b7802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaea80cae7b4dec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.399+0000 7f4b8e000640 1 -- 172.21.15.6:0/178185746 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78034070 con 0x7f4b900afdf0 2024-03-06T07:30:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.399+0000 7f4b8e000640 1 -- 172.21.15.6:0/178185746 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b78030860 con 0x7f4b900afdf0 2024-03-06T07:30:25.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.399+0000 7f4b8e000640 1 -- 172.21.15.6:0/178185746 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78030c30 con 0x7f4b900afdf0 2024-03-06T07:30:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 -- 172.21.15.6:0/178185746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b900afdf0 msgr2=0x7f4b900af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 --2- 172.21.15.6:0/178185746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b900afdf0 0x7f4b900af140 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4b90071cc0 tx=0x7f4b7802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 -- 172.21.15.6:0/178185746 shutdown_connections 2024-03-06T07:30:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 --2- 172.21.15.6:0/178185746 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b900afdf0 0x7f4b900af140 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 --2- 172.21.15.6:0/178185746 >> v2:172.21.15.178:3300/0 conn(0x7f4b9015e200 0x7f4b900b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 -- 172.21.15.6:0/178185746 >> 172.21.15.6:0/178185746 conn(0x7f4b900b2ae0 msgr2=0x7f4b900b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 -- 172.21.15.6:0/178185746 shutdown_connections 2024-03-06T07:30:25.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.400+0000 7f4b95e00640 1 -- 172.21.15.6:0/178185746 wait complete. 2024-03-06T07:30:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.402+0000 7f4b95e00640 1 Processor -- start 2024-03-06T07:30:25.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.402+0000 7f4b95e00640 1 -- start start 2024-03-06T07:30:25.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b95e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b9006afb0 0x7f4b90158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b95e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4b900afdf0 0x7f4b9015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b95e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 0x7f4b9015c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b95e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900af680 con 0x7f4b9015e200 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b95e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b900aec20 con 0x7f4b9006afb0 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b95e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9006e2c0 con 0x7f4b900afdf0 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 0x7f4b9015c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b8f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 0x7f4b9015c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34070/0 (socket says 172.21.15.6:34070) 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b8f400640 1 -- 172.21.15.6:0/941495161 learned_addr learned my addr 172.21.15.6:0/941495161 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.403+0000 7f4b8ea00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b9006afb0 0x7f4b90158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b87e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4b900afdf0 0x7f4b9015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b8f400640 1 -- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4b900afdf0 msgr2=0x7f4b9015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b8f400640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4b900afdf0 0x7f4b9015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b8f400640 1 -- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b9006afb0 msgr2=0x7f4b90158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b8f400640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b9006afb0 0x7f4b90158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b8f400640 1 -- 172.21.15.6:0/941495161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9014cf70 con 0x7f4b9015e200 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b87e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4b900afdf0 0x7f4b9015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b8f400640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 0x7f4b9015c850 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c7a0 tx=0x7f4b8800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88003b70 con 0x7f4b9015e200 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88003d30 con 0x7f4b9015e200 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8800aa90 con 0x7f4b9015e200 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.404+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9015a340 con 0x7f4b9015e200 2024-03-06T07:30:25.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.405+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b90159580 con 0x7f4b9015e200 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.406+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9004f3a0 con 0x7f4b9015e200 2024-03-06T07:30:25.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.408+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b8800ac30 con 0x7f4b9015e200 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.409+0000 7f4b87400640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.409+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b8808a150 con 0x7f4b9015e200 2024-03-06T07:30:25.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.410+0000 7f4b8ea00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.411+0000 7f4b8ea00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4b80006fd0 tx=0x7f4b80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.414+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b88058fd0 con 0x7f4b9015e200 2024-03-06T07:30:25.584 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-06T07:30:25.582+0000 7fbac8e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:30:25.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.616+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b9006ee20 con 0x7f4b9015e200 2024-03-06T07:30:25.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.617+0000 7f4b87400640 1 -- 172.21.15.6:0/941495161 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f4b880588d0 con 0x7f4b9015e200 2024-03-06T07:30:25.620 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:25.620 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":58,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:22.633658+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.620+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4b6c06b4d0 msgr2=0x7f4b6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.620+0000 7f4b95e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4b80006fd0 tx=0x7f4b80008040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.620+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 msgr2=0x7f4b9015c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.620+0000 7f4b95e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 0x7f4b9015c850 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c7a0 tx=0x7f4b8800cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 shutdown_connections 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4b6c06b4d0 0x7f4b6c06d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b9015e200 0x7f4b9015c850 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4b900afdf0 0x7f4b9015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 --2- 172.21.15.6:0/941495161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b9006afb0 0x7f4b90158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 >> 172.21.15.6:0/941495161 conn(0x7f4b900b2ae0 msgr2=0x7f4b900cfd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 shutdown_connections 2024-03-06T07:30:25.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.621+0000 7f4b95e00640 1 -- 172.21.15.6:0/941495161 wait complete. 2024-03-06T07:30:25.642 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:30:25.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:25.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.859+0000 7f977f400640 1 Processor -- start 2024-03-06T07:30:25.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.859+0000 7f977f400640 1 -- start start 2024-03-06T07:30:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f977f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9778161670 0x7f9778070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f977f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97780ae0a0 0x7f97780711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f977f400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97780ce9a0 con 0x7f977806e240 2024-03-06T07:30:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f977f400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97780730d0 con 0x7f9778161670 2024-03-06T07:30:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f977f400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97780cec80 con 0x7f97780ae0a0 2024-03-06T07:30:25.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f9777e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f977806e240 0x7f977806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55686/0 (socket says 172.21.15.6:55686) 2024-03-06T07:30:25.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f9777e00640 1 -- 172.21.15.6:0/3535732635 learned_addr learned my addr 172.21.15.6:0/3535732635 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:25.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.861+0000 7f977ca00640 1 --2- 172.21.15.6:0/3535732635 >> v2:172.21.15.6:3300/0 conn(0x7f9778161670 0x7f9778070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f977d400640 1 --2- 172.21.15.6:0/3535732635 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97780ae0a0 0x7f97780711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f9777400640 1 -- 172.21.15.6:0/3535732635 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284595640 0 0) 0x7f97780ce9a0 con 0x7f977806e240 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f9777400640 1 -- 172.21.15.6:0/3535732635 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c003520 con 0x7f977806e240 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f977ca00640 1 -- 172.21.15.6:0/3535732635 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97780ae0a0 msgr2=0x7f97780711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f977ca00640 1 --2- 172.21.15.6:0/3535732635 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97780ae0a0 0x7f97780711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f977ca00640 1 -- 172.21.15.6:0/3535732635 >> v1:172.21.15.6:6789/0 conn(0x7f977806e240 legacy=0x7f977806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.862+0000 7f977ca00640 1 -- 172.21.15.6:0/3535732635 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97780cef60 con 0x7f9778161670 2024-03-06T07:30:25.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.864+0000 7f977ca00640 1 --2- 172.21.15.6:0/3535732635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9778161670 0x7f9778070c70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9768009d90 tx=0x7f976802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd8c7eea4fb3275 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.864+0000 7f9777400640 1 -- 172.21.15.6:0/3535732635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768039070 con 0x7f9778161670 2024-03-06T07:30:25.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.864+0000 7f9777400640 1 -- 172.21.15.6:0/3535732635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976802cc50 con 0x7f9778161670 2024-03-06T07:30:25.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.864+0000 7f9777400640 1 -- 172.21.15.6:0/3535732635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768034440 con 0x7f9778161670 2024-03-06T07:30:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 -- 172.21.15.6:0/3535732635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9778161670 msgr2=0x7f9778070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 --2- 172.21.15.6:0/3535732635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9778161670 0x7f9778070c70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9768009d90 tx=0x7f976802bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 -- 172.21.15.6:0/3535732635 shutdown_connections 2024-03-06T07:30:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 --2- 172.21.15.6:0/3535732635 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97780ae0a0 0x7f97780711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 --2- 172.21.15.6:0/3535732635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9778161670 0x7f9778070c70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 -- 172.21.15.6:0/3535732635 >> 172.21.15.6:0/3535732635 conn(0x7f977806afb0 msgr2=0x7f977806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 -- 172.21.15.6:0/3535732635 shutdown_connections 2024-03-06T07:30:25.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.865+0000 7f977f400640 1 -- 172.21.15.6:0/3535732635 wait complete. 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.868+0000 7f977f400640 1 Processor -- start 2024-03-06T07:30:25.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.868+0000 7f977f400640 1 -- start start 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 0x7f97781f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977f400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97780ae0a0 0x7f97780ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9778161670 0x7f97780cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977f400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780ce600 con 0x7f977806e240 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977f400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97780cf520 con 0x7f97780ae0a0 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977f400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97780ce780 con 0x7f9778161670 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 0x7f97781f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 0x7f97781f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34094/0 (socket says 172.21.15.6:34094) 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 -- 172.21.15.6:0/1139431167 learned_addr learned my addr 172.21.15.6:0/1139431167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f9777e00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97780ae0a0 0x7f97780ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 -- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9778161670 msgr2=0x7f97780cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9778161670 0x7f97780cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 -- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97780ae0a0 msgr2=0x7f97780ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97780ae0a0 0x7f97780ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.870+0000 7f977ca00640 1 -- 172.21.15.6:0/1139431167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778156670 con 0x7f977806e240 2024-03-06T07:30:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.871+0000 7f977ca00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 0x7f97781f71d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9768009eb0 tx=0x7f9768004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.871+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768039050 con 0x7f977806e240 2024-03-06T07:30:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.871+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9768045070 con 0x7f977806e240 2024-03-06T07:30:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.871+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97780cb1f0 con 0x7f977806e240 2024-03-06T07:30:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.871+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9768009450 con 0x7f977806e240 2024-03-06T07:30:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.871+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97780cb7b0 con 0x7f977806e240 2024-03-06T07:30:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.873+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9778069510 con 0x7f977806e240 2024-03-06T07:30:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.873+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f976803e610 con 0x7f977806e240 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.874+0000 7f9774c00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f975806b4d0 0x7f975806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.874+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97680acdc0 con 0x7f977806e240 2024-03-06T07:30:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.875+0000 7f9777e00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f975806b4d0 0x7f975806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.876+0000 7f9777e00640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f975806b4d0 0x7f975806d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f977815f7a0 tx=0x7f9764009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:25.879+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f976807bba0 con 0x7f977806e240 2024-03-06T07:30:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.075+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97780744d0 con 0x7f975806b4d0 2024-03-06T07:30:26.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.094+0000 7f9774c00640 1 -- 172.21.15.6:0/1139431167 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247619 (secure 0 0 0) 0x7f97780744d0 con 0x7f975806b4d0 2024-03-06T07:30:26.100 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:26.102 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:30:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f975806b4d0 msgr2=0x7f975806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f975806b4d0 0x7f975806d990 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f977815f7a0 tx=0x7f9764009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 msgr2=0x7f97781f71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 0x7f97781f71d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9768009eb0 tx=0x7f9768004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 shutdown_connections 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f975806b4d0 0x7f975806d990 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97780ae0a0 0x7f97780ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9778161670 0x7f97780cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 --2- 172.21.15.6:0/1139431167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f977806e240 0x7f97781f71d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 >> 172.21.15.6:0/1139431167 conn(0x7f977806afb0 msgr2=0x7f97780ad910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 shutdown_connections 2024-03-06T07:30:26.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.100+0000 7f977f400640 1 -- 172.21.15.6:0/1139431167 wait complete. 2024-03-06T07:30:26.121 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-06T07:30:25.499841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264180,"kb_used_data":6380,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754710540,"statfs":{"total":773094113280,"available":772823592960,"internally_reserved":0,"allocated":6533120,"data_stored":3373888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.909824"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.642928+0000","last_change":"2024-03-06T07:30:21.639251+0000","last_active":"2024-03-06T07:30:21.642928+0000","last_peered":"2024-03-06T07:30:21.642928+0000","last_clean":"2024-03-06T07:30:21.642928+0000","last_became_active":"2024-03-06T07:30:21.638950+0000","last_became_peered":"2024-03-06T07:30:21.638950+0000","last_unstale":"2024-03-06T07:30:21.642928+0000","last_undegraded":"2024-03-06T07:30:21.642928+0000","last_fullsized":"2024-03-06T07:30:21.642928+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:57.957584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605846+0000","last_change":"2024-03-06T07:30:20.605846+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605846+0000","last_undegraded":"2024-03-06T07:30:20.605846+0000","last_fullsized":"2024-03-06T07:30:20.605846+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.609643+0000","last_change":"2024-03-06T07:30:20.609643+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.609643+0000","last_undegraded":"2024-03-06T07:30:20.609643+0000","last_fullsized":"2024-03-06T07:30:20.609643+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.609392+0000","last_change":"2024-03-06T07:30:20.609392+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.609392+0000","last_undegraded":"2024-03-06T07:30:20.609392+0000","last_fullsized":"2024-03-06T07:30:20.609392+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.675095+0000","last_change":"2024-03-06T07:30:21.671393+0000","last_active":"2024-03-06T07:30:21.675095+0000","last_peered":"2024-03-06T07:30:21.675095+0000","last_clean":"2024-03-06T07:30:21.675095+0000","last_became_active":"2024-03-06T07:30:21.671107+0000","last_became_peered":"2024-03-06T07:30:21.671107+0000","last_unstale":"2024-03-06T07:30:21.675095+0000","last_undegraded":"2024-03-06T07:30:21.675095+0000","last_fullsized":"2024-03-06T07:30:21.675095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:49.030206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605107+0000","last_change":"2024-03-06T07:30:20.605107+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605107+0000","last_undegraded":"2024-03-06T07:30:20.605107+0000","last_fullsized":"2024-03-06T07:30:20.605107+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617084+0000","last_change":"2024-03-06T07:30:20.617084+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617084+0000","last_undegraded":"2024-03-06T07:30:20.617084+0000","last_fullsized":"2024-03-06T07:30:20.617084+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607822+0000","last_change":"2024-03-06T07:30:20.607822+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607822+0000","last_undegraded":"2024-03-06T07:30:20.607822+0000","last_fullsized":"2024-03-06T07:30:20.607822+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.621543+0000","last_change":"2024-03-06T07:30:20.621543+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.621543+0000","last_undegraded":"2024-03-06T07:30:20.621543+0000","last_fullsized":"2024-03-06T07:30:20.621543+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668500+0000","last_change":"2024-03-06T07:30:21.658602+0000","last_active":"2024-03-06T07:30:21.668500+0000","last_peered":"2024-03-06T07:30:21.668500+0000","last_clean":"2024-03-06T07:30:21.668500+0000","last_became_active":"2024-03-06T07:30:21.656608+0000","last_became_peered":"2024-03-06T07:30:21.656608+0000","last_unstale":"2024-03-06T07:30:21.668500+0000","last_undegraded":"2024-03-06T07:30:21.668500+0000","last_fullsized":"2024-03-06T07:30:21.668500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:06.554234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668477+0000","last_change":"2024-03-06T07:30:21.656749+0000","last_active":"2024-03-06T07:30:21.668477+0000","last_peered":"2024-03-06T07:30:21.668477+0000","last_clean":"2024-03-06T07:30:21.668477+0000","last_became_active":"2024-03-06T07:30:21.656479+0000","last_became_peered":"2024-03-06T07:30:21.656479+0000","last_unstale":"2024-03-06T07:30:21.668477+0000","last_undegraded":"2024-03-06T07:30:21.668477+0000","last_fullsized":"2024-03-06T07:30:21.668477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:25.712837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606399+0000","last_change":"2024-03-06T07:30:20.606399+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606399+0000","last_undegraded":"2024-03-06T07:30:20.606399+0000","last_fullsized":"2024-03-06T07:30:20.606399+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614576+0000","last_change":"2024-03-06T07:30:20.614576+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614576+0000","last_undegraded":"2024-03-06T07:30:20.614576+0000","last_fullsized":"2024-03-06T07:30:20.614576+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.620181+0000","last_change":"2024-03-06T07:30:20.620181+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.620181+0000","last_undegraded":"2024-03-06T07:30:20.620181+0000","last_fullsized":"2024-03-06T07:30:20.620181+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.616435+0000","last_change":"2024-03-06T07:30:20.616435+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.616435+0000","last_undegraded":"2024-03-06T07:30:20.616435+0000","last_fullsized":"2024-03-06T07:30:20.616435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.604507+0000","last_change":"2024-03-06T07:30:20.604507+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.604507+0000","last_undegraded":"2024-03-06T07:30:20.604507+0000","last_fullsized":"2024-03-06T07:30:20.604507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607582+0000","last_change":"2024-03-06T07:30:20.607582+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607582+0000","last_undegraded":"2024-03-06T07:30:20.607582+0000","last_fullsized":"2024-03-06T07:30:20.607582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606705+0000","last_change":"2024-03-06T07:30:20.606705+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606705+0000","last_undegraded":"2024-03-06T07:30:20.606705+0000","last_fullsized":"2024-03-06T07:30:20.606705+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.639713+0000","last_change":"2024-03-06T07:30:21.635273+0000","last_active":"2024-03-06T07:30:21.639713+0000","last_peered":"2024-03-06T07:30:21.639713+0000","last_clean":"2024-03-06T07:30:21.639713+0000","last_became_active":"2024-03-06T07:30:21.634938+0000","last_became_peered":"2024-03-06T07:30:21.634938+0000","last_unstale":"2024-03-06T07:30:21.639713+0000","last_undegraded":"2024-03-06T07:30:21.639713+0000","last_fullsized":"2024-03-06T07:30:21.639713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:28.549452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608995+0000","last_change":"2024-03-06T07:30:20.608995+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608995+0000","last_undegraded":"2024-03-06T07:30:20.608995+0000","last_fullsized":"2024-03-06T07:30:20.608995+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612095+0000","last_change":"2024-03-06T07:30:20.612095+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612095+0000","last_undegraded":"2024-03-06T07:30:20.612095+0000","last_fullsized":"2024-03-06T07:30:20.612095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612050+0000","last_change":"2024-03-06T07:30:20.612050+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612050+0000","last_undegraded":"2024-03-06T07:30:20.612050+0000","last_fullsized":"2024-03-06T07:30:20.612050+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.603712+0000","last_change":"2024-03-06T07:30:20.603712+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.603712+0000","last_undegraded":"2024-03-06T07:30:20.603712+0000","last_fullsized":"2024-03-06T07:30:20.603712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.618309+0000","last_change":"2024-03-06T07:30:20.618309+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.618309+0000","last_undegraded":"2024-03-06T07:30:20.618309+0000","last_fullsized":"2024-03-06T07:30:20.618309+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.648459+0000","last_change":"2024-03-06T07:30:21.646940+0000","last_active":"2024-03-06T07:30:21.648459+0000","last_peered":"2024-03-06T07:30:21.648459+0000","last_clean":"2024-03-06T07:30:21.648459+0000","last_became_active":"2024-03-06T07:30:21.644768+0000","last_became_peered":"2024-03-06T07:30:21.644768+0000","last_unstale":"2024-03-06T07:30:21.648459+0000","last_undegraded":"2024-03-06T07:30:21.648459+0000","last_fullsized":"2024-03-06T07:30:21.648459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:24.687045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610660+0000","last_change":"2024-03-06T07:30:20.610660+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610660+0000","last_undegraded":"2024-03-06T07:30:20.610660+0000","last_fullsized":"2024-03-06T07:30:20.610660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.613593+0000","last_change":"2024-03-06T07:30:20.613593+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.613593+0000","last_undegraded":"2024-03-06T07:30:20.613593+0000","last_fullsized":"2024-03-06T07:30:20.613593+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668154+0000","last_change":"2024-03-06T07:30:21.663886+0000","last_active":"2024-03-06T07:30:21.668154+0000","last_peered":"2024-03-06T07:30:21.668154+0000","last_clean":"2024-03-06T07:30:21.668154+0000","last_became_active":"2024-03-06T07:30:21.663511+0000","last_became_peered":"2024-03-06T07:30:21.663511+0000","last_unstale":"2024-03-06T07:30:21.668154+0000","last_undegraded":"2024-03-06T07:30:21.668154+0000","last_fullsized":"2024-03-06T07:30:21.668154+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:27.109206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.616277+0000","last_change":"2024-03-06T07:30:20.616277+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.616277+0000","last_undegraded":"2024-03-06T07:30:20.616277+0000","last_fullsized":"2024-03-06T07:30:20.616277+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.601987+0000","last_change":"2024-03-06T07:30:20.601987+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.601987+0000","last_undegraded":"2024-03-06T07:30:20.601987+0000","last_fullsized":"2024-03-06T07:30:20.601987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.648777+0000","last_change":"2024-03-06T07:30:21.645937+0000","last_active":"2024-03-06T07:30:21.648777+0000","last_peered":"2024-03-06T07:30:21.648777+0000","last_clean":"2024-03-06T07:30:21.648777+0000","last_became_active":"2024-03-06T07:30:21.644909+0000","last_became_peered":"2024-03-06T07:30:21.644909+0000","last_unstale":"2024-03-06T07:30:21.648777+0000","last_undegraded":"2024-03-06T07:30:21.648777+0000","last_fullsized":"2024-03-06T07:30:21.648777+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:18.572142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.603960+0000","last_change":"2024-03-06T07:30:20.603960+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.603960+0000","last_undegraded":"2024-03-06T07:30:20.603960+0000","last_fullsized":"2024-03-06T07:30:20.603960+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606790+0000","last_change":"2024-03-06T07:30:20.606790+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606790+0000","last_undegraded":"2024-03-06T07:30:20.606790+0000","last_fullsized":"2024-03-06T07:30:20.606790+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632214+0000","last_change":"2024-03-06T07:30:20.630369+0000","last_active":"2024-03-06T07:30:20.632214+0000","last_peered":"2024-03-06T07:30:20.632214+0000","last_clean":"2024-03-06T07:30:20.632214+0000","last_became_active":"2024-03-06T07:30:20.628580+0000","last_became_peered":"2024-03-06T07:30:20.628580+0000","last_unstale":"2024-03-06T07:30:20.632214+0000","last_undegraded":"2024-03-06T07:30:20.632214+0000","last_fullsized":"2024-03-06T07:30:20.632214+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:15.085610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.671038+0000","last_change":"2024-03-06T07:30:21.664516+0000","last_active":"2024-03-06T07:30:21.671038+0000","last_peered":"2024-03-06T07:30:21.671038+0000","last_clean":"2024-03-06T07:30:21.671038+0000","last_became_active":"2024-03-06T07:30:21.664224+0000","last_became_peered":"2024-03-06T07:30:21.664224+0000","last_unstale":"2024-03-06T07:30:21.671038+0000","last_undegraded":"2024-03-06T07:30:21.671038+0000","last_fullsized":"2024-03-06T07:30:21.671038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.768918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.629363+0000","last_change":"2024-03-06T07:30:20.625914+0000","last_active":"2024-03-06T07:30:20.629363+0000","last_peered":"2024-03-06T07:30:20.629363+0000","last_clean":"2024-03-06T07:30:20.629363+0000","last_became_active":"2024-03-06T07:30:20.625643+0000","last_became_peered":"2024-03-06T07:30:20.625643+0000","last_unstale":"2024-03-06T07:30:20.629363+0000","last_undegraded":"2024-03-06T07:30:20.629363+0000","last_fullsized":"2024-03-06T07:30:20.629363+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:15.048674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606274+0000","last_change":"2024-03-06T07:30:20.606274+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606274+0000","last_undegraded":"2024-03-06T07:30:20.606274+0000","last_fullsized":"2024-03-06T07:30:20.606274+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.619866+0000","last_change":"2024-03-06T07:30:20.617109+0000","last_active":"2024-03-06T07:30:20.619866+0000","last_peered":"2024-03-06T07:30:20.619866+0000","last_clean":"2024-03-06T07:30:20.619866+0000","last_became_active":"2024-03-06T07:30:20.615288+0000","last_became_peered":"2024-03-06T07:30:20.615288+0000","last_unstale":"2024-03-06T07:30:20.619866+0000","last_undegraded":"2024-03-06T07:30:20.619866+0000","last_fullsized":"2024-03-06T07:30:20.619866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:37.430833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.613038+0000","last_change":"2024-03-06T07:30:20.613038+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.613038+0000","last_undegraded":"2024-03-06T07:30:20.613038+0000","last_fullsized":"2024-03-06T07:30:20.613038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.631863+0000","last_change":"2024-03-06T07:30:20.626609+0000","last_active":"2024-03-06T07:30:20.631863+0000","last_peered":"2024-03-06T07:30:20.631863+0000","last_clean":"2024-03-06T07:30:20.631863+0000","last_became_active":"2024-03-06T07:30:20.625668+0000","last_became_peered":"2024-03-06T07:30:20.625668+0000","last_unstale":"2024-03-06T07:30:20.631863+0000","last_undegraded":"2024-03-06T07:30:20.631863+0000","last_fullsized":"2024-03-06T07:30:20.631863+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:37.430833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.611537+0000","last_change":"2024-03-06T07:30:20.611537+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.611537+0000","last_undegraded":"2024-03-06T07:30:20.611537+0000","last_fullsized":"2024-03-06T07:30:20.611537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633345+0000","last_change":"2024-03-06T07:30:20.629926+0000","last_active":"2024-03-06T07:30:20.633345+0000","last_peered":"2024-03-06T07:30:20.633345+0000","last_clean":"2024-03-06T07:30:20.633345+0000","last_became_active":"2024-03-06T07:30:20.629470+0000","last_became_peered":"2024-03-06T07:30:20.629470+0000","last_unstale":"2024-03-06T07:30:20.633345+0000","last_undegraded":"2024-03-06T07:30:20.633345+0000","last_fullsized":"2024-03-06T07:30:20.633345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:11.638813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.618124+0000","last_change":"2024-03-06T07:30:20.618124+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.618124+0000","last_undegraded":"2024-03-06T07:30:20.618124+0000","last_fullsized":"2024-03-06T07:30:20.618124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630744+0000","last_change":"2024-03-06T07:30:20.627675+0000","last_active":"2024-03-06T07:30:20.630744+0000","last_peered":"2024-03-06T07:30:20.630744+0000","last_clean":"2024-03-06T07:30:20.630744+0000","last_became_active":"2024-03-06T07:30:20.627383+0000","last_became_peered":"2024-03-06T07:30:20.627383+0000","last_unstale":"2024-03-06T07:30:20.630744+0000","last_undegraded":"2024-03-06T07:30:20.630744+0000","last_fullsized":"2024-03-06T07:30:20.630744+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.218902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614522+0000","last_change":"2024-03-06T07:30:20.614522+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614522+0000","last_undegraded":"2024-03-06T07:30:20.614522+0000","last_fullsized":"2024-03-06T07:30:20.614522+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633087+0000","last_change":"2024-03-06T07:30:20.629740+0000","last_active":"2024-03-06T07:30:20.633087+0000","last_peered":"2024-03-06T07:30:20.633087+0000","last_clean":"2024-03-06T07:30:20.633087+0000","last_became_active":"2024-03-06T07:30:20.629427+0000","last_became_peered":"2024-03-06T07:30:20.629427+0000","last_unstale":"2024-03-06T07:30:20.633087+0000","last_undegraded":"2024-03-06T07:30:20.633087+0000","last_fullsized":"2024-03-06T07:30:20.633087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.218902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617560+0000","last_change":"2024-03-06T07:30:20.617560+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617560+0000","last_undegraded":"2024-03-06T07:30:20.617560+0000","last_fullsized":"2024-03-06T07:30:20.617560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634979+0000","last_change":"2024-03-06T07:30:20.634196+0000","last_active":"2024-03-06T07:30:20.634979+0000","last_peered":"2024-03-06T07:30:20.634979+0000","last_clean":"2024-03-06T07:30:20.634979+0000","last_became_active":"2024-03-06T07:30:20.633405+0000","last_became_peered":"2024-03-06T07:30:20.633405+0000","last_unstale":"2024-03-06T07:30:20.634979+0000","last_undegraded":"2024-03-06T07:30:20.634979+0000","last_fullsized":"2024-03-06T07:30:20.634979+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:00.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.621452+0000","last_change":"2024-03-06T07:30:20.621452+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.621452+0000","last_undegraded":"2024-03-06T07:30:20.621452+0000","last_fullsized":"2024-03-06T07:30:20.621452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630055+0000","last_change":"2024-03-06T07:30:20.627513+0000","last_active":"2024-03-06T07:30:20.630055+0000","last_peered":"2024-03-06T07:30:20.630055+0000","last_clean":"2024-03-06T07:30:20.630055+0000","last_became_active":"2024-03-06T07:30:20.627220+0000","last_became_peered":"2024-03-06T07:30:20.627220+0000","last_unstale":"2024-03-06T07:30:20.630055+0000","last_undegraded":"2024-03-06T07:30:20.630055+0000","last_fullsized":"2024-03-06T07:30:20.630055+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:18.735168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605345+0000","last_change":"2024-03-06T07:30:20.605345+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605345+0000","last_undegraded":"2024-03-06T07:30:20.605345+0000","last_fullsized":"2024-03-06T07:30:20.605345+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630935+0000","last_change":"2024-03-06T07:30:20.628649+0000","last_active":"2024-03-06T07:30:20.630935+0000","last_peered":"2024-03-06T07:30:20.630935+0000","last_clean":"2024-03-06T07:30:20.630935+0000","last_became_active":"2024-03-06T07:30:20.628065+0000","last_became_peered":"2024-03-06T07:30:20.628065+0000","last_unstale":"2024-03-06T07:30:20.630935+0000","last_undegraded":"2024-03-06T07:30:20.630935+0000","last_fullsized":"2024-03-06T07:30:20.630935+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:30.404064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.667370+0000","last_change":"2024-03-06T07:30:21.652433+0000","last_active":"2024-03-06T07:30:21.667370+0000","last_peered":"2024-03-06T07:30:21.667370+0000","last_clean":"2024-03-06T07:30:21.667370+0000","last_became_active":"2024-03-06T07:30:21.651796+0000","last_became_peered":"2024-03-06T07:30:21.651796+0000","last_unstale":"2024-03-06T07:30:21.667370+0000","last_undegraded":"2024-03-06T07:30:21.667370+0000","last_fullsized":"2024-03-06T07:30:21.667370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:04.774122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630750+0000","last_change":"2024-03-06T07:30:20.628448+0000","last_active":"2024-03-06T07:30:20.630750+0000","last_peered":"2024-03-06T07:30:20.630750+0000","last_clean":"2024-03-06T07:30:20.630750+0000","last_became_active":"2024-03-06T07:30:20.628114+0000","last_became_peered":"2024-03-06T07:30:20.628114+0000","last_unstale":"2024-03-06T07:30:20.630750+0000","last_undegraded":"2024-03-06T07:30:20.630750+0000","last_fullsized":"2024-03-06T07:30:20.630750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:00.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608641+0000","last_change":"2024-03-06T07:30:20.608641+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608641+0000","last_undegraded":"2024-03-06T07:30:20.608641+0000","last_fullsized":"2024-03-06T07:30:20.608641+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633000+0000","last_change":"2024-03-06T07:30:20.629645+0000","last_active":"2024-03-06T07:30:20.633000+0000","last_peered":"2024-03-06T07:30:20.633000+0000","last_clean":"2024-03-06T07:30:20.633000+0000","last_became_active":"2024-03-06T07:30:20.628235+0000","last_became_peered":"2024-03-06T07:30:20.628235+0000","last_unstale":"2024-03-06T07:30:20.633000+0000","last_undegraded":"2024-03-06T07:30:20.633000+0000","last_fullsized":"2024-03-06T07:30:20.633000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.757187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607488+0000","last_change":"2024-03-06T07:30:20.607488+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607488+0000","last_undegraded":"2024-03-06T07:30:20.607488+0000","last_fullsized":"2024-03-06T07:30:20.607488+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630783+0000","last_change":"2024-03-06T07:30:20.627658+0000","last_active":"2024-03-06T07:30:20.630783+0000","last_peered":"2024-03-06T07:30:20.630783+0000","last_clean":"2024-03-06T07:30:20.630783+0000","last_became_active":"2024-03-06T07:30:20.627370+0000","last_became_peered":"2024-03-06T07:30:20.627370+0000","last_unstale":"2024-03-06T07:30:20.630783+0000","last_undegraded":"2024-03-06T07:30:20.630783+0000","last_fullsized":"2024-03-06T07:30:20.630783+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:37.737710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610618+0000","last_change":"2024-03-06T07:30:20.610618+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610618+0000","last_undegraded":"2024-03-06T07:30:20.610618+0000","last_fullsized":"2024-03-06T07:30:20.610618+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.629261+0000","last_change":"2024-03-06T07:30:20.626528+0000","last_active":"2024-03-06T07:30:20.629261+0000","last_peered":"2024-03-06T07:30:20.629261+0000","last_clean":"2024-03-06T07:30:20.629261+0000","last_became_active":"2024-03-06T07:30:20.626159+0000","last_became_peered":"2024-03-06T07:30:20.626159+0000","last_unstale":"2024-03-06T07:30:20.629261+0000","last_undegraded":"2024-03-06T07:30:20.629261+0000","last_fullsized":"2024-03-06T07:30:20.629261+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.777807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617892+0000","last_change":"2024-03-06T07:30:20.617892+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617892+0000","last_undegraded":"2024-03-06T07:30:20.617892+0000","last_fullsized":"2024-03-06T07:30:20.617892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633067+0000","last_change":"2024-03-06T07:30:20.628917+0000","last_active":"2024-03-06T07:30:20.633067+0000","last_peered":"2024-03-06T07:30:20.633067+0000","last_clean":"2024-03-06T07:30:20.633067+0000","last_became_active":"2024-03-06T07:30:20.625883+0000","last_became_peered":"2024-03-06T07:30:20.625883+0000","last_unstale":"2024-03-06T07:30:20.633067+0000","last_undegraded":"2024-03-06T07:30:20.633067+0000","last_fullsized":"2024-03-06T07:30:20.633067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:58.572467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.602149+0000","last_change":"2024-03-06T07:30:20.602149+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.602149+0000","last_undegraded":"2024-03-06T07:30:20.602149+0000","last_fullsized":"2024-03-06T07:30:20.602149+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628580+0000","last_change":"2024-03-06T07:30:20.624267+0000","last_active":"2024-03-06T07:30:20.628580+0000","last_peered":"2024-03-06T07:30:20.628580+0000","last_clean":"2024-03-06T07:30:20.628580+0000","last_became_active":"2024-03-06T07:30:20.623498+0000","last_became_peered":"2024-03-06T07:30:20.623498+0000","last_unstale":"2024-03-06T07:30:20.628580+0000","last_undegraded":"2024-03-06T07:30:20.628580+0000","last_fullsized":"2024-03-06T07:30:20.628580+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:11.638813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605234+0000","last_change":"2024-03-06T07:30:20.605234+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605234+0000","last_undegraded":"2024-03-06T07:30:20.605234+0000","last_fullsized":"2024-03-06T07:30:20.605234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628440+0000","last_change":"2024-03-06T07:30:20.625469+0000","last_active":"2024-03-06T07:30:20.628440+0000","last_peered":"2024-03-06T07:30:20.628440+0000","last_clean":"2024-03-06T07:30:20.628440+0000","last_became_active":"2024-03-06T07:30:20.625169+0000","last_became_peered":"2024-03-06T07:30:20.625169+0000","last_unstale":"2024-03-06T07:30:20.628440+0000","last_undegraded":"2024-03-06T07:30:20.628440+0000","last_fullsized":"2024-03-06T07:30:20.628440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:50.201205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610000+0000","last_change":"2024-03-06T07:30:20.610000+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610000+0000","last_undegraded":"2024-03-06T07:30:20.610000+0000","last_fullsized":"2024-03-06T07:30:20.610000+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.624471+0000","last_change":"2024-03-06T07:30:20.619107+0000","last_active":"2024-03-06T07:30:20.624471+0000","last_peered":"2024-03-06T07:30:20.624471+0000","last_clean":"2024-03-06T07:30:20.624471+0000","last_became_active":"2024-03-06T07:30:20.618101+0000","last_became_peered":"2024-03-06T07:30:20.618101+0000","last_unstale":"2024-03-06T07:30:20.624471+0000","last_undegraded":"2024-03-06T07:30:20.624471+0000","last_fullsized":"2024-03-06T07:30:20.624471+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:58.572467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608554+0000","last_change":"2024-03-06T07:30:20.608554+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608554+0000","last_undegraded":"2024-03-06T07:30:20.608554+0000","last_fullsized":"2024-03-06T07:30:20.608554+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632061+0000","last_change":"2024-03-06T07:30:20.626055+0000","last_active":"2024-03-06T07:30:20.632061+0000","last_peered":"2024-03-06T07:30:20.632061+0000","last_clean":"2024-03-06T07:30:20.632061+0000","last_became_active":"2024-03-06T07:30:20.625631+0000","last_became_peered":"2024-03-06T07:30:20.625631+0000","last_unstale":"2024-03-06T07:30:20.632061+0000","last_undegraded":"2024-03-06T07:30:20.632061+0000","last_fullsized":"2024-03-06T07:30:20.632061+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:23.173745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.604785+0000","last_change":"2024-03-06T07:30:20.604785+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.604785+0000","last_undegraded":"2024-03-06T07:30:20.604785+0000","last_fullsized":"2024-03-06T07:30:20.604785+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632549+0000","last_change":"2024-03-06T07:30:20.631068+0000","last_active":"2024-03-06T07:30:20.632549+0000","last_peered":"2024-03-06T07:30:20.632549+0000","last_clean":"2024-03-06T07:30:20.632549+0000","last_became_active":"2024-03-06T07:30:20.630723+0000","last_became_peered":"2024-03-06T07:30:20.630723+0000","last_unstale":"2024-03-06T07:30:20.632549+0000","last_undegraded":"2024-03-06T07:30:20.632549+0000","last_fullsized":"2024-03-06T07:30:20.632549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:53.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606205+0000","last_change":"2024-03-06T07:30:20.606205+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606205+0000","last_undegraded":"2024-03-06T07:30:20.606205+0000","last_fullsized":"2024-03-06T07:30:20.606205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632017+0000","last_change":"2024-03-06T07:30:20.630056+0000","last_active":"2024-03-06T07:30:20.632017+0000","last_peered":"2024-03-06T07:30:20.632017+0000","last_clean":"2024-03-06T07:30:20.632017+0000","last_became_active":"2024-03-06T07:30:20.629761+0000","last_became_peered":"2024-03-06T07:30:20.629761+0000","last_unstale":"2024-03-06T07:30:20.632017+0000","last_undegraded":"2024-03-06T07:30:20.632017+0000","last_fullsized":"2024-03-06T07:30:20.632017+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:15.540003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605142+0000","last_change":"2024-03-06T07:30:20.605142+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605142+0000","last_undegraded":"2024-03-06T07:30:20.605142+0000","last_fullsized":"2024-03-06T07:30:20.605142+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641952+0000","last_change":"2024-03-06T07:30:20.618404+0000","last_active":"2024-03-06T07:30:21.641952+0000","last_peered":"2024-03-06T07:30:21.641952+0000","last_clean":"2024-03-06T07:30:21.641952+0000","last_became_active":"2024-03-06T07:30:20.617575+0000","last_became_peered":"2024-03-06T07:30:20.617575+0000","last_unstale":"2024-03-06T07:30:21.641952+0000","last_undegraded":"2024-03-06T07:30:21.641952+0000","last_fullsized":"2024-03-06T07:30:21.641952+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:03.599842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605059+0000","last_change":"2024-03-06T07:30:20.605059+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605059+0000","last_undegraded":"2024-03-06T07:30:20.605059+0000","last_fullsized":"2024-03-06T07:30:20.605059+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634627+0000","last_change":"2024-03-06T07:30:20.633615+0000","last_active":"2024-03-06T07:30:20.634627+0000","last_peered":"2024-03-06T07:30:20.634627+0000","last_clean":"2024-03-06T07:30:20.634627+0000","last_became_active":"2024-03-06T07:30:20.633225+0000","last_became_peered":"2024-03-06T07:30:20.633225+0000","last_unstale":"2024-03-06T07:30:20.634627+0000","last_undegraded":"2024-03-06T07:30:20.634627+0000","last_fullsized":"2024-03-06T07:30:20.634627+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:15.085610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606321+0000","last_change":"2024-03-06T07:30:20.606321+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606321+0000","last_undegraded":"2024-03-06T07:30:20.606321+0000","last_fullsized":"2024-03-06T07:30:20.606321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.674444+0000","last_change":"2024-03-06T07:30:20.624699+0000","last_active":"2024-03-06T07:30:21.674444+0000","last_peered":"2024-03-06T07:30:21.674444+0000","last_clean":"2024-03-06T07:30:21.674444+0000","last_became_active":"2024-03-06T07:30:20.622946+0000","last_became_peered":"2024-03-06T07:30:20.622946+0000","last_unstale":"2024-03-06T07:30:21.674444+0000","last_undegraded":"2024-03-06T07:30:21.674444+0000","last_fullsized":"2024-03-06T07:30:21.674444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:50.448937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608839+0000","last_change":"2024-03-06T07:30:20.608839+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608839+0000","last_undegraded":"2024-03-06T07:30:20.608839+0000","last_fullsized":"2024-03-06T07:30:20.608839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.631221+0000","last_change":"2024-03-06T07:30:20.629443+0000","last_active":"2024-03-06T07:30:20.631221+0000","last_peered":"2024-03-06T07:30:20.631221+0000","last_clean":"2024-03-06T07:30:20.631221+0000","last_became_active":"2024-03-06T07:30:20.629144+0000","last_became_peered":"2024-03-06T07:30:20.629144+0000","last_unstale":"2024-03-06T07:30:20.631221+0000","last_undegraded":"2024-03-06T07:30:20.631221+0000","last_fullsized":"2024-03-06T07:30:20.631221+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:37.100598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.665902+0000","last_change":"2024-03-06T07:30:21.650700+0000","last_active":"2024-03-06T07:30:21.665902+0000","last_peered":"2024-03-06T07:30:21.665902+0000","last_clean":"2024-03-06T07:30:21.665902+0000","last_became_active":"2024-03-06T07:30:21.650363+0000","last_became_peered":"2024-03-06T07:30:21.650363+0000","last_unstale":"2024-03-06T07:30:21.665902+0000","last_undegraded":"2024-03-06T07:30:21.665902+0000","last_fullsized":"2024-03-06T07:30:21.665902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:03.131576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632815+0000","last_change":"2024-03-06T07:30:20.629663+0000","last_active":"2024-03-06T07:30:20.632815+0000","last_peered":"2024-03-06T07:30:20.632815+0000","last_clean":"2024-03-06T07:30:20.632815+0000","last_became_active":"2024-03-06T07:30:20.629365+0000","last_became_peered":"2024-03-06T07:30:20.629365+0000","last_unstale":"2024-03-06T07:30:20.632815+0000","last_undegraded":"2024-03-06T07:30:20.632815+0000","last_fullsized":"2024-03-06T07:30:20.632815+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:20.066237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612360+0000","last_change":"2024-03-06T07:30:20.612360+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612360+0000","last_undegraded":"2024-03-06T07:30:20.612360+0000","last_fullsized":"2024-03-06T07:30:20.612360+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633860+0000","last_change":"2024-03-06T07:30:20.631439+0000","last_active":"2024-03-06T07:30:20.633860+0000","last_peered":"2024-03-06T07:30:20.633860+0000","last_clean":"2024-03-06T07:30:20.633860+0000","last_became_active":"2024-03-06T07:30:20.631100+0000","last_became_peered":"2024-03-06T07:30:20.631100+0000","last_unstale":"2024-03-06T07:30:20.633860+0000","last_undegraded":"2024-03-06T07:30:20.633860+0000","last_fullsized":"2024-03-06T07:30:20.633860+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:37.100598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610052+0000","last_change":"2024-03-06T07:30:20.610052+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610052+0000","last_undegraded":"2024-03-06T07:30:20.610052+0000","last_fullsized":"2024-03-06T07:30:20.610052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641987+0000","last_change":"2024-03-06T07:30:20.623625+0000","last_active":"2024-03-06T07:30:21.641987+0000","last_peered":"2024-03-06T07:30:21.641987+0000","last_clean":"2024-03-06T07:30:21.641987+0000","last_became_active":"2024-03-06T07:30:20.623355+0000","last_became_peered":"2024-03-06T07:30:20.623355+0000","last_unstale":"2024-03-06T07:30:21.641987+0000","last_undegraded":"2024-03-06T07:30:21.641987+0000","last_fullsized":"2024-03-06T07:30:21.641987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:46.004625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.601988+0000","last_change":"2024-03-06T07:30:20.601988+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.601988+0000","last_undegraded":"2024-03-06T07:30:20.601988+0000","last_fullsized":"2024-03-06T07:30:20.601988+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634005+0000","last_change":"2024-03-06T07:30:20.632460+0000","last_active":"2024-03-06T07:30:20.634005+0000","last_peered":"2024-03-06T07:30:20.634005+0000","last_clean":"2024-03-06T07:30:20.634005+0000","last_became_active":"2024-03-06T07:30:20.630675+0000","last_became_peered":"2024-03-06T07:30:20.630675+0000","last_unstale":"2024-03-06T07:30:20.634005+0000","last_undegraded":"2024-03-06T07:30:20.634005+0000","last_fullsized":"2024-03-06T07:30:20.634005+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:18.735168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.670594+0000","last_change":"2024-03-06T07:30:21.661218+0000","last_active":"2024-03-06T07:30:21.670594+0000","last_peered":"2024-03-06T07:30:21.670594+0000","last_clean":"2024-03-06T07:30:21.670594+0000","last_became_active":"2024-03-06T07:30:21.660549+0000","last_became_peered":"2024-03-06T07:30:21.660549+0000","last_unstale":"2024-03-06T07:30:21.670594+0000","last_undegraded":"2024-03-06T07:30:21.670594+0000","last_fullsized":"2024-03-06T07:30:21.670594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:48.433580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641620+0000","last_change":"2024-03-06T07:30:20.609520+0000","last_active":"2024-03-06T07:30:21.641620+0000","last_peered":"2024-03-06T07:30:21.641620+0000","last_clean":"2024-03-06T07:30:21.641620+0000","last_became_active":"2024-03-06T07:30:20.608944+0000","last_became_peered":"2024-03-06T07:30:20.608944+0000","last_unstale":"2024-03-06T07:30:21.641620+0000","last_undegraded":"2024-03-06T07:30:21.641620+0000","last_fullsized":"2024-03-06T07:30:21.641620+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:54.924862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614665+0000","last_change":"2024-03-06T07:30:20.614665+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614665+0000","last_undegraded":"2024-03-06T07:30:20.614665+0000","last_fullsized":"2024-03-06T07:30:20.614665+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630868+0000","last_change":"2024-03-06T07:30:20.624382+0000","last_active":"2024-03-06T07:30:20.630868+0000","last_peered":"2024-03-06T07:30:20.630868+0000","last_clean":"2024-03-06T07:30:20.630868+0000","last_became_active":"2024-03-06T07:30:20.624100+0000","last_became_peered":"2024-03-06T07:30:20.624100+0000","last_unstale":"2024-03-06T07:30:20.630868+0000","last_undegraded":"2024-03-06T07:30:20.630868+0000","last_fullsized":"2024-03-06T07:30:20.630868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:22.937616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607340+0000","last_change":"2024-03-06T07:30:20.607340+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607340+0000","last_undegraded":"2024-03-06T07:30:20.607340+0000","last_fullsized":"2024-03-06T07:30:20.607340+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628428+0000","last_change":"2024-03-06T07:30:20.624654+0000","last_active":"2024-03-06T07:30:20.628428+0000","last_peered":"2024-03-06T07:30:20.628428+0000","last_clean":"2024-03-06T07:30:20.628428+0000","last_became_active":"2024-03-06T07:30:20.624112+0000","last_became_peered":"2024-03-06T07:30:20.624112+0000","last_unstale":"2024-03-06T07:30:20.628428+0000","last_undegraded":"2024-03-06T07:30:20.628428+0000","last_fullsized":"2024-03-06T07:30:20.628428+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:34.375041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.614301+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:30:20.614301+0000","last_peered":"2024-03-06T07:30:20.614301+0000","last_clean":"2024-03-06T07:30:20.614301+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:30:20.614301+0000","last_undegraded":"2024-03-06T07:30:20.614301+0000","last_fullsized":"2024-03-06T07:30:20.614301+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509479,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":279263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":279297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542183,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":862800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542184,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542183,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542184,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1257472,"data_stored":862800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:26.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:26.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.384+0000 7f5728600640 1 Processor -- start 2024-03-06T07:30:26.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.385+0000 7f5728600640 1 -- start start 2024-03-06T07:30:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.386+0000 7f5728600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 0x7f57200d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.386+0000 7f5728600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f57200d2760 0x7f57200b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.386+0000 7f5728600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200cea50 con 0x7f5720161b40 2024-03-06T07:30:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.386+0000 7f5728600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57200b5770 con 0x7f57200d2760 2024-03-06T07:30:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.386+0000 7f5728600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200cf900 con 0x7f57200cfd90 2024-03-06T07:30:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.387+0000 7f5725c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5720161b40 0x7f57200b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55694/0 (socket says 172.21.15.6:55694) 2024-03-06T07:30:26.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.387+0000 7f5725c00640 1 -- 172.21.15.6:0/647659794 learned_addr learned my addr 172.21.15.6:0/647659794 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.387+0000 7f5725200640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 0x7f57200d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.387+0000 7f5726600640 1 --2- 172.21.15.6:0/647659794 >> v2:172.21.15.178:3300/0 conn(0x7f57200d2760 0x7f57200b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.387+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897324418 0 0) 0x7f57200cea50 con 0x7f5720161b40 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.387+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708003520 con 0x7f5720161b40 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3595280744 0 0) 0x7f5708003520 con 0x7f5720161b40 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 msgr2=0x7f57200d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 0x7f57200d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 >> v2:172.21.15.178:3300/0 conn(0x7f57200d2760 msgr2=0x7f57200b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 --2- 172.21.15.6:0/647659794 >> v2:172.21.15.178:3300/0 conn(0x7f57200d2760 0x7f57200b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200cecf0 con 0x7f5720161b40 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5714001670 con 0x7f5720161b40 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 >> v1:172.21.15.6:6789/0 conn(0x7f5720161b40 legacy=0x7f57200b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5708003a20 0x7f5708003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 0x7f5708004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f57080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200b5770 con 0x7f5708004d40 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200cf900 con 0x7f5708003a20 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570800b9c0 con 0x7f57200cfd90 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f5726600640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f57080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f5725c00640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5708003a20 0x7f5708003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.388+0000 7f5725200640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 0x7f5708004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.389+0000 7f5726600640 1 -- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 msgr2=0x7f5708004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.389+0000 7f5726600640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200cfd90 0x7f5708004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.389+0000 7f5726600640 1 -- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5708003a20 msgr2=0x7f5708003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.389+0000 7f5726600640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5708003a20 0x7f5708003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.389+0000 7f5726600640 1 -- 172.21.15.6:0/647659794 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5708003520 con 0x7f5708004d40 2024-03-06T07:30:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.390+0000 7f5726600640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f57080051c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5718009fc0 tx=0x7f5708005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01615fa33a510ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.390+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718039070 con 0x7f5708004d40 2024-03-06T07:30:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.390+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571802cc60 con 0x7f5708004d40 2024-03-06T07:30:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.390+0000 7f571fe00640 1 -- 172.21.15.6:0/647659794 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718009c10 con 0x7f5708004d40 2024-03-06T07:30:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5728600640 1 -- 172.21.15.6:0/647659794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 msgr2=0x7f57080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5728600640 1 --2- 172.21.15.6:0/647659794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f57080051c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5718009fc0 tx=0x7f5708005830 comp rx=0 tx=0).stop 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5726600640 1 -- 172.21.15.6:0/647659794 reap_dead start 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5728600640 1 -- 172.21.15.6:0/647659794 shutdown_connections 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5728600640 1 -- 172.21.15.6:0/647659794 >> 172.21.15.6:0/647659794 conn(0x7f572009d220 msgr2=0x7f57200d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5728600640 1 -- 172.21.15.6:0/647659794 shutdown_connections 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.391+0000 7f5728600640 1 -- 172.21.15.6:0/647659794 wait complete. 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.393+0000 7f5728600640 1 Processor -- start 2024-03-06T07:30:26.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.393+0000 7f5728600640 1 -- start start 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5728600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f5720151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5728600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200b8e80 0x7f57200b9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5728600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f572015f6c0 0x7f572015fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5728600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200cf590 con 0x7f5708004d40 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5728600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200cf2b0 con 0x7f572015f6c0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5728600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200b2ff0 con 0x7f57200b8e80 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5725c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f5720151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5725c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f5720151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34112/0 (socket says 172.21.15.6:34112) 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.394+0000 7f5725c00640 1 -- 172.21.15.6:0/2209549351 learned_addr learned my addr 172.21.15.6:0/2209549351 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725200640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200b8e80 0x7f57200b9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725c00640 1 -- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200b8e80 msgr2=0x7f57200b9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725c00640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200b8e80 0x7f57200b9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725c00640 1 -- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f572015f6c0 msgr2=0x7f572015fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725c00640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f572015f6c0 0x7f572015fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725c00640 1 -- 172.21.15.6:0/2209549351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200cb210 con 0x7f5708004d40 2024-03-06T07:30:26.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5725c00640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f5720151360 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f571400e790 tx=0x7f571400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571400c8d0 con 0x7f5708004d40 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5714018070 con 0x7f5708004d40 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57200cb4f0 con 0x7f5708004d40 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.395+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571400ccb0 con 0x7f5708004d40 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.396+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57200cbab0 con 0x7f5708004d40 2024-03-06T07:30:26.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.397+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57200cafa0 con 0x7f5708004d40 2024-03-06T07:30:26.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.398+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f571401d460 con 0x7f5708004d40 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.399+0000 7f571d600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f570006b4d0 0x7f570006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.399+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f571408a120 con 0x7f5708004d40 2024-03-06T07:30:26.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.400+0000 7f5725200640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f570006b4d0 0x7f570006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.401+0000 7f5725200640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f570006b4d0 0x7f570006d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f57200b5ca0 tx=0x7f570c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.404+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5714058fa0 con 0x7f5708004d40 2024-03-06T07:30:26.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.597+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5720160060 con 0x7f570006b4d0 2024-03-06T07:30:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.616+0000 7f571d600640 1 -- 172.21.15.6:0/2209549351 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247619 (secure 0 0 0) 0x7f5720160060 con 0x7f570006b4d0 2024-03-06T07:30:26.621 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:26.622 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f570006b4d0 msgr2=0x7f570006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f570006b4d0 0x7f570006d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f57200b5ca0 tx=0x7f570c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 msgr2=0x7f5720151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f5720151360 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f571400e790 tx=0x7f571400ec50 comp rx=0 tx=0).stop 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 shutdown_connections 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f570006b4d0 0x7f570006d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f572015f6c0 0x7f572015fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57200b8e80 0x7f57200b9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 --2- 172.21.15.6:0/2209549351 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5708004d40 0x7f5720151360 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 >> 172.21.15.6:0/2209549351 conn(0x7f572009d220 msgr2=0x7f572006cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 shutdown_connections 2024-03-06T07:30:26.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.621+0000 7f5728600640 1 -- 172.21.15.6:0/2209549351 wait complete. 2024-03-06T07:30:26.641 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-06T07:30:25.499841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264180,"kb_used_data":6380,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754710540,"statfs":{"total":773094113280,"available":772823592960,"internally_reserved":0,"allocated":6533120,"data_stored":3373888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.909824"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.642928+0000","last_change":"2024-03-06T07:30:21.639251+0000","last_active":"2024-03-06T07:30:21.642928+0000","last_peered":"2024-03-06T07:30:21.642928+0000","last_clean":"2024-03-06T07:30:21.642928+0000","last_became_active":"2024-03-06T07:30:21.638950+0000","last_became_peered":"2024-03-06T07:30:21.638950+0000","last_unstale":"2024-03-06T07:30:21.642928+0000","last_undegraded":"2024-03-06T07:30:21.642928+0000","last_fullsized":"2024-03-06T07:30:21.642928+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:57.957584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605846+0000","last_change":"2024-03-06T07:30:20.605846+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605846+0000","last_undegraded":"2024-03-06T07:30:20.605846+0000","last_fullsized":"2024-03-06T07:30:20.605846+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.609643+0000","last_change":"2024-03-06T07:30:20.609643+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.609643+0000","last_undegraded":"2024-03-06T07:30:20.609643+0000","last_fullsized":"2024-03-06T07:30:20.609643+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.609392+0000","last_change":"2024-03-06T07:30:20.609392+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.609392+0000","last_undegraded":"2024-03-06T07:30:20.609392+0000","last_fullsized":"2024-03-06T07:30:20.609392+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.675095+0000","last_change":"2024-03-06T07:30:21.671393+0000","last_active":"2024-03-06T07:30:21.675095+0000","last_peered":"2024-03-06T07:30:21.675095+0000","last_clean":"2024-03-06T07:30:21.675095+0000","last_became_active":"2024-03-06T07:30:21.671107+0000","last_became_peered":"2024-03-06T07:30:21.671107+0000","last_unstale":"2024-03-06T07:30:21.675095+0000","last_undegraded":"2024-03-06T07:30:21.675095+0000","last_fullsized":"2024-03-06T07:30:21.675095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:49.030206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605107+0000","last_change":"2024-03-06T07:30:20.605107+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605107+0000","last_undegraded":"2024-03-06T07:30:20.605107+0000","last_fullsized":"2024-03-06T07:30:20.605107+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617084+0000","last_change":"2024-03-06T07:30:20.617084+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617084+0000","last_undegraded":"2024-03-06T07:30:20.617084+0000","last_fullsized":"2024-03-06T07:30:20.617084+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607822+0000","last_change":"2024-03-06T07:30:20.607822+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607822+0000","last_undegraded":"2024-03-06T07:30:20.607822+0000","last_fullsized":"2024-03-06T07:30:20.607822+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.621543+0000","last_change":"2024-03-06T07:30:20.621543+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.621543+0000","last_undegraded":"2024-03-06T07:30:20.621543+0000","last_fullsized":"2024-03-06T07:30:20.621543+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668500+0000","last_change":"2024-03-06T07:30:21.658602+0000","last_active":"2024-03-06T07:30:21.668500+0000","last_peered":"2024-03-06T07:30:21.668500+0000","last_clean":"2024-03-06T07:30:21.668500+0000","last_became_active":"2024-03-06T07:30:21.656608+0000","last_became_peered":"2024-03-06T07:30:21.656608+0000","last_unstale":"2024-03-06T07:30:21.668500+0000","last_undegraded":"2024-03-06T07:30:21.668500+0000","last_fullsized":"2024-03-06T07:30:21.668500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:06.554234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668477+0000","last_change":"2024-03-06T07:30:21.656749+0000","last_active":"2024-03-06T07:30:21.668477+0000","last_peered":"2024-03-06T07:30:21.668477+0000","last_clean":"2024-03-06T07:30:21.668477+0000","last_became_active":"2024-03-06T07:30:21.656479+0000","last_became_peered":"2024-03-06T07:30:21.656479+0000","last_unstale":"2024-03-06T07:30:21.668477+0000","last_undegraded":"2024-03-06T07:30:21.668477+0000","last_fullsized":"2024-03-06T07:30:21.668477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:25.712837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606399+0000","last_change":"2024-03-06T07:30:20.606399+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606399+0000","last_undegraded":"2024-03-06T07:30:20.606399+0000","last_fullsized":"2024-03-06T07:30:20.606399+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614576+0000","last_change":"2024-03-06T07:30:20.614576+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614576+0000","last_undegraded":"2024-03-06T07:30:20.614576+0000","last_fullsized":"2024-03-06T07:30:20.614576+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.620181+0000","last_change":"2024-03-06T07:30:20.620181+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.620181+0000","last_undegraded":"2024-03-06T07:30:20.620181+0000","last_fullsized":"2024-03-06T07:30:20.620181+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.616435+0000","last_change":"2024-03-06T07:30:20.616435+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.616435+0000","last_undegraded":"2024-03-06T07:30:20.616435+0000","last_fullsized":"2024-03-06T07:30:20.616435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.604507+0000","last_change":"2024-03-06T07:30:20.604507+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.604507+0000","last_undegraded":"2024-03-06T07:30:20.604507+0000","last_fullsized":"2024-03-06T07:30:20.604507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607582+0000","last_change":"2024-03-06T07:30:20.607582+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607582+0000","last_undegraded":"2024-03-06T07:30:20.607582+0000","last_fullsized":"2024-03-06T07:30:20.607582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606705+0000","last_change":"2024-03-06T07:30:20.606705+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606705+0000","last_undegraded":"2024-03-06T07:30:20.606705+0000","last_fullsized":"2024-03-06T07:30:20.606705+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.639713+0000","last_change":"2024-03-06T07:30:21.635273+0000","last_active":"2024-03-06T07:30:21.639713+0000","last_peered":"2024-03-06T07:30:21.639713+0000","last_clean":"2024-03-06T07:30:21.639713+0000","last_became_active":"2024-03-06T07:30:21.634938+0000","last_became_peered":"2024-03-06T07:30:21.634938+0000","last_unstale":"2024-03-06T07:30:21.639713+0000","last_undegraded":"2024-03-06T07:30:21.639713+0000","last_fullsized":"2024-03-06T07:30:21.639713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:28.549452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608995+0000","last_change":"2024-03-06T07:30:20.608995+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608995+0000","last_undegraded":"2024-03-06T07:30:20.608995+0000","last_fullsized":"2024-03-06T07:30:20.608995+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612095+0000","last_change":"2024-03-06T07:30:20.612095+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612095+0000","last_undegraded":"2024-03-06T07:30:20.612095+0000","last_fullsized":"2024-03-06T07:30:20.612095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612050+0000","last_change":"2024-03-06T07:30:20.612050+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612050+0000","last_undegraded":"2024-03-06T07:30:20.612050+0000","last_fullsized":"2024-03-06T07:30:20.612050+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.603712+0000","last_change":"2024-03-06T07:30:20.603712+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.603712+0000","last_undegraded":"2024-03-06T07:30:20.603712+0000","last_fullsized":"2024-03-06T07:30:20.603712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.618309+0000","last_change":"2024-03-06T07:30:20.618309+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.618309+0000","last_undegraded":"2024-03-06T07:30:20.618309+0000","last_fullsized":"2024-03-06T07:30:20.618309+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.648459+0000","last_change":"2024-03-06T07:30:21.646940+0000","last_active":"2024-03-06T07:30:21.648459+0000","last_peered":"2024-03-06T07:30:21.648459+0000","last_clean":"2024-03-06T07:30:21.648459+0000","last_became_active":"2024-03-06T07:30:21.644768+0000","last_became_peered":"2024-03-06T07:30:21.644768+0000","last_unstale":"2024-03-06T07:30:21.648459+0000","last_undegraded":"2024-03-06T07:30:21.648459+0000","last_fullsized":"2024-03-06T07:30:21.648459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:24.687045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610660+0000","last_change":"2024-03-06T07:30:20.610660+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610660+0000","last_undegraded":"2024-03-06T07:30:20.610660+0000","last_fullsized":"2024-03-06T07:30:20.610660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.613593+0000","last_change":"2024-03-06T07:30:20.613593+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.613593+0000","last_undegraded":"2024-03-06T07:30:20.613593+0000","last_fullsized":"2024-03-06T07:30:20.613593+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668154+0000","last_change":"2024-03-06T07:30:21.663886+0000","last_active":"2024-03-06T07:30:21.668154+0000","last_peered":"2024-03-06T07:30:21.668154+0000","last_clean":"2024-03-06T07:30:21.668154+0000","last_became_active":"2024-03-06T07:30:21.663511+0000","last_became_peered":"2024-03-06T07:30:21.663511+0000","last_unstale":"2024-03-06T07:30:21.668154+0000","last_undegraded":"2024-03-06T07:30:21.668154+0000","last_fullsized":"2024-03-06T07:30:21.668154+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:27.109206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.616277+0000","last_change":"2024-03-06T07:30:20.616277+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.616277+0000","last_undegraded":"2024-03-06T07:30:20.616277+0000","last_fullsized":"2024-03-06T07:30:20.616277+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.601987+0000","last_change":"2024-03-06T07:30:20.601987+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.601987+0000","last_undegraded":"2024-03-06T07:30:20.601987+0000","last_fullsized":"2024-03-06T07:30:20.601987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.648777+0000","last_change":"2024-03-06T07:30:21.645937+0000","last_active":"2024-03-06T07:30:21.648777+0000","last_peered":"2024-03-06T07:30:21.648777+0000","last_clean":"2024-03-06T07:30:21.648777+0000","last_became_active":"2024-03-06T07:30:21.644909+0000","last_became_peered":"2024-03-06T07:30:21.644909+0000","last_unstale":"2024-03-06T07:30:21.648777+0000","last_undegraded":"2024-03-06T07:30:21.648777+0000","last_fullsized":"2024-03-06T07:30:21.648777+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:18.572142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.603960+0000","last_change":"2024-03-06T07:30:20.603960+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.603960+0000","last_undegraded":"2024-03-06T07:30:20.603960+0000","last_fullsized":"2024-03-06T07:30:20.603960+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606790+0000","last_change":"2024-03-06T07:30:20.606790+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606790+0000","last_undegraded":"2024-03-06T07:30:20.606790+0000","last_fullsized":"2024-03-06T07:30:20.606790+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632214+0000","last_change":"2024-03-06T07:30:20.630369+0000","last_active":"2024-03-06T07:30:20.632214+0000","last_peered":"2024-03-06T07:30:20.632214+0000","last_clean":"2024-03-06T07:30:20.632214+0000","last_became_active":"2024-03-06T07:30:20.628580+0000","last_became_peered":"2024-03-06T07:30:20.628580+0000","last_unstale":"2024-03-06T07:30:20.632214+0000","last_undegraded":"2024-03-06T07:30:20.632214+0000","last_fullsized":"2024-03-06T07:30:20.632214+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:15.085610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.671038+0000","last_change":"2024-03-06T07:30:21.664516+0000","last_active":"2024-03-06T07:30:21.671038+0000","last_peered":"2024-03-06T07:30:21.671038+0000","last_clean":"2024-03-06T07:30:21.671038+0000","last_became_active":"2024-03-06T07:30:21.664224+0000","last_became_peered":"2024-03-06T07:30:21.664224+0000","last_unstale":"2024-03-06T07:30:21.671038+0000","last_undegraded":"2024-03-06T07:30:21.671038+0000","last_fullsized":"2024-03-06T07:30:21.671038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.768918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.629363+0000","last_change":"2024-03-06T07:30:20.625914+0000","last_active":"2024-03-06T07:30:20.629363+0000","last_peered":"2024-03-06T07:30:20.629363+0000","last_clean":"2024-03-06T07:30:20.629363+0000","last_became_active":"2024-03-06T07:30:20.625643+0000","last_became_peered":"2024-03-06T07:30:20.625643+0000","last_unstale":"2024-03-06T07:30:20.629363+0000","last_undegraded":"2024-03-06T07:30:20.629363+0000","last_fullsized":"2024-03-06T07:30:20.629363+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:15.048674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606274+0000","last_change":"2024-03-06T07:30:20.606274+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606274+0000","last_undegraded":"2024-03-06T07:30:20.606274+0000","last_fullsized":"2024-03-06T07:30:20.606274+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.619866+0000","last_change":"2024-03-06T07:30:20.617109+0000","last_active":"2024-03-06T07:30:20.619866+0000","last_peered":"2024-03-06T07:30:20.619866+0000","last_clean":"2024-03-06T07:30:20.619866+0000","last_became_active":"2024-03-06T07:30:20.615288+0000","last_became_peered":"2024-03-06T07:30:20.615288+0000","last_unstale":"2024-03-06T07:30:20.619866+0000","last_undegraded":"2024-03-06T07:30:20.619866+0000","last_fullsized":"2024-03-06T07:30:20.619866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:37.430833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.613038+0000","last_change":"2024-03-06T07:30:20.613038+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.613038+0000","last_undegraded":"2024-03-06T07:30:20.613038+0000","last_fullsized":"2024-03-06T07:30:20.613038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.631863+0000","last_change":"2024-03-06T07:30:20.626609+0000","last_active":"2024-03-06T07:30:20.631863+0000","last_peered":"2024-03-06T07:30:20.631863+0000","last_clean":"2024-03-06T07:30:20.631863+0000","last_became_active":"2024-03-06T07:30:20.625668+0000","last_became_peered":"2024-03-06T07:30:20.625668+0000","last_unstale":"2024-03-06T07:30:20.631863+0000","last_undegraded":"2024-03-06T07:30:20.631863+0000","last_fullsized":"2024-03-06T07:30:20.631863+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:37.430833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.611537+0000","last_change":"2024-03-06T07:30:20.611537+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.611537+0000","last_undegraded":"2024-03-06T07:30:20.611537+0000","last_fullsized":"2024-03-06T07:30:20.611537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633345+0000","last_change":"2024-03-06T07:30:20.629926+0000","last_active":"2024-03-06T07:30:20.633345+0000","last_peered":"2024-03-06T07:30:20.633345+0000","last_clean":"2024-03-06T07:30:20.633345+0000","last_became_active":"2024-03-06T07:30:20.629470+0000","last_became_peered":"2024-03-06T07:30:20.629470+0000","last_unstale":"2024-03-06T07:30:20.633345+0000","last_undegraded":"2024-03-06T07:30:20.633345+0000","last_fullsized":"2024-03-06T07:30:20.633345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:11.638813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.618124+0000","last_change":"2024-03-06T07:30:20.618124+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.618124+0000","last_undegraded":"2024-03-06T07:30:20.618124+0000","last_fullsized":"2024-03-06T07:30:20.618124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630744+0000","last_change":"2024-03-06T07:30:20.627675+0000","last_active":"2024-03-06T07:30:20.630744+0000","last_peered":"2024-03-06T07:30:20.630744+0000","last_clean":"2024-03-06T07:30:20.630744+0000","last_became_active":"2024-03-06T07:30:20.627383+0000","last_became_peered":"2024-03-06T07:30:20.627383+0000","last_unstale":"2024-03-06T07:30:20.630744+0000","last_undegraded":"2024-03-06T07:30:20.630744+0000","last_fullsized":"2024-03-06T07:30:20.630744+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.218902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614522+0000","last_change":"2024-03-06T07:30:20.614522+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614522+0000","last_undegraded":"2024-03-06T07:30:20.614522+0000","last_fullsized":"2024-03-06T07:30:20.614522+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633087+0000","last_change":"2024-03-06T07:30:20.629740+0000","last_active":"2024-03-06T07:30:20.633087+0000","last_peered":"2024-03-06T07:30:20.633087+0000","last_clean":"2024-03-06T07:30:20.633087+0000","last_became_active":"2024-03-06T07:30:20.629427+0000","last_became_peered":"2024-03-06T07:30:20.629427+0000","last_unstale":"2024-03-06T07:30:20.633087+0000","last_undegraded":"2024-03-06T07:30:20.633087+0000","last_fullsized":"2024-03-06T07:30:20.633087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.218902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617560+0000","last_change":"2024-03-06T07:30:20.617560+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617560+0000","last_undegraded":"2024-03-06T07:30:20.617560+0000","last_fullsized":"2024-03-06T07:30:20.617560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634979+0000","last_change":"2024-03-06T07:30:20.634196+0000","last_active":"2024-03-06T07:30:20.634979+0000","last_peered":"2024-03-06T07:30:20.634979+0000","last_clean":"2024-03-06T07:30:20.634979+0000","last_became_active":"2024-03-06T07:30:20.633405+0000","last_became_peered":"2024-03-06T07:30:20.633405+0000","last_unstale":"2024-03-06T07:30:20.634979+0000","last_undegraded":"2024-03-06T07:30:20.634979+0000","last_fullsized":"2024-03-06T07:30:20.634979+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:00.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.621452+0000","last_change":"2024-03-06T07:30:20.621452+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.621452+0000","last_undegraded":"2024-03-06T07:30:20.621452+0000","last_fullsized":"2024-03-06T07:30:20.621452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630055+0000","last_change":"2024-03-06T07:30:20.627513+0000","last_active":"2024-03-06T07:30:20.630055+0000","last_peered":"2024-03-06T07:30:20.630055+0000","last_clean":"2024-03-06T07:30:20.630055+0000","last_became_active":"2024-03-06T07:30:20.627220+0000","last_became_peered":"2024-03-06T07:30:20.627220+0000","last_unstale":"2024-03-06T07:30:20.630055+0000","last_undegraded":"2024-03-06T07:30:20.630055+0000","last_fullsized":"2024-03-06T07:30:20.630055+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:18.735168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605345+0000","last_change":"2024-03-06T07:30:20.605345+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605345+0000","last_undegraded":"2024-03-06T07:30:20.605345+0000","last_fullsized":"2024-03-06T07:30:20.605345+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630935+0000","last_change":"2024-03-06T07:30:20.628649+0000","last_active":"2024-03-06T07:30:20.630935+0000","last_peered":"2024-03-06T07:30:20.630935+0000","last_clean":"2024-03-06T07:30:20.630935+0000","last_became_active":"2024-03-06T07:30:20.628065+0000","last_became_peered":"2024-03-06T07:30:20.628065+0000","last_unstale":"2024-03-06T07:30:20.630935+0000","last_undegraded":"2024-03-06T07:30:20.630935+0000","last_fullsized":"2024-03-06T07:30:20.630935+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:30.404064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.667370+0000","last_change":"2024-03-06T07:30:21.652433+0000","last_active":"2024-03-06T07:30:21.667370+0000","last_peered":"2024-03-06T07:30:21.667370+0000","last_clean":"2024-03-06T07:30:21.667370+0000","last_became_active":"2024-03-06T07:30:21.651796+0000","last_became_peered":"2024-03-06T07:30:21.651796+0000","last_unstale":"2024-03-06T07:30:21.667370+0000","last_undegraded":"2024-03-06T07:30:21.667370+0000","last_fullsized":"2024-03-06T07:30:21.667370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:04.774122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630750+0000","last_change":"2024-03-06T07:30:20.628448+0000","last_active":"2024-03-06T07:30:20.630750+0000","last_peered":"2024-03-06T07:30:20.630750+0000","last_clean":"2024-03-06T07:30:20.630750+0000","last_became_active":"2024-03-06T07:30:20.628114+0000","last_became_peered":"2024-03-06T07:30:20.628114+0000","last_unstale":"2024-03-06T07:30:20.630750+0000","last_undegraded":"2024-03-06T07:30:20.630750+0000","last_fullsized":"2024-03-06T07:30:20.630750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:00.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608641+0000","last_change":"2024-03-06T07:30:20.608641+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608641+0000","last_undegraded":"2024-03-06T07:30:20.608641+0000","last_fullsized":"2024-03-06T07:30:20.608641+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633000+0000","last_change":"2024-03-06T07:30:20.629645+0000","last_active":"2024-03-06T07:30:20.633000+0000","last_peered":"2024-03-06T07:30:20.633000+0000","last_clean":"2024-03-06T07:30:20.633000+0000","last_became_active":"2024-03-06T07:30:20.628235+0000","last_became_peered":"2024-03-06T07:30:20.628235+0000","last_unstale":"2024-03-06T07:30:20.633000+0000","last_undegraded":"2024-03-06T07:30:20.633000+0000","last_fullsized":"2024-03-06T07:30:20.633000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.757187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607488+0000","last_change":"2024-03-06T07:30:20.607488+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607488+0000","last_undegraded":"2024-03-06T07:30:20.607488+0000","last_fullsized":"2024-03-06T07:30:20.607488+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630783+0000","last_change":"2024-03-06T07:30:20.627658+0000","last_active":"2024-03-06T07:30:20.630783+0000","last_peered":"2024-03-06T07:30:20.630783+0000","last_clean":"2024-03-06T07:30:20.630783+0000","last_became_active":"2024-03-06T07:30:20.627370+0000","last_became_peered":"2024-03-06T07:30:20.627370+0000","last_unstale":"2024-03-06T07:30:20.630783+0000","last_undegraded":"2024-03-06T07:30:20.630783+0000","last_fullsized":"2024-03-06T07:30:20.630783+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:37.737710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610618+0000","last_change":"2024-03-06T07:30:20.610618+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610618+0000","last_undegraded":"2024-03-06T07:30:20.610618+0000","last_fullsized":"2024-03-06T07:30:20.610618+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.629261+0000","last_change":"2024-03-06T07:30:20.626528+0000","last_active":"2024-03-06T07:30:20.629261+0000","last_peered":"2024-03-06T07:30:20.629261+0000","last_clean":"2024-03-06T07:30:20.629261+0000","last_became_active":"2024-03-06T07:30:20.626159+0000","last_became_peered":"2024-03-06T07:30:20.626159+0000","last_unstale":"2024-03-06T07:30:20.629261+0000","last_undegraded":"2024-03-06T07:30:20.629261+0000","last_fullsized":"2024-03-06T07:30:20.629261+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.777807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617892+0000","last_change":"2024-03-06T07:30:20.617892+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617892+0000","last_undegraded":"2024-03-06T07:30:20.617892+0000","last_fullsized":"2024-03-06T07:30:20.617892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633067+0000","last_change":"2024-03-06T07:30:20.628917+0000","last_active":"2024-03-06T07:30:20.633067+0000","last_peered":"2024-03-06T07:30:20.633067+0000","last_clean":"2024-03-06T07:30:20.633067+0000","last_became_active":"2024-03-06T07:30:20.625883+0000","last_became_peered":"2024-03-06T07:30:20.625883+0000","last_unstale":"2024-03-06T07:30:20.633067+0000","last_undegraded":"2024-03-06T07:30:20.633067+0000","last_fullsized":"2024-03-06T07:30:20.633067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:58.572467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.602149+0000","last_change":"2024-03-06T07:30:20.602149+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.602149+0000","last_undegraded":"2024-03-06T07:30:20.602149+0000","last_fullsized":"2024-03-06T07:30:20.602149+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628580+0000","last_change":"2024-03-06T07:30:20.624267+0000","last_active":"2024-03-06T07:30:20.628580+0000","last_peered":"2024-03-06T07:30:20.628580+0000","last_clean":"2024-03-06T07:30:20.628580+0000","last_became_active":"2024-03-06T07:30:20.623498+0000","last_became_peered":"2024-03-06T07:30:20.623498+0000","last_unstale":"2024-03-06T07:30:20.628580+0000","last_undegraded":"2024-03-06T07:30:20.628580+0000","last_fullsized":"2024-03-06T07:30:20.628580+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:11.638813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605234+0000","last_change":"2024-03-06T07:30:20.605234+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605234+0000","last_undegraded":"2024-03-06T07:30:20.605234+0000","last_fullsized":"2024-03-06T07:30:20.605234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628440+0000","last_change":"2024-03-06T07:30:20.625469+0000","last_active":"2024-03-06T07:30:20.628440+0000","last_peered":"2024-03-06T07:30:20.628440+0000","last_clean":"2024-03-06T07:30:20.628440+0000","last_became_active":"2024-03-06T07:30:20.625169+0000","last_became_peered":"2024-03-06T07:30:20.625169+0000","last_unstale":"2024-03-06T07:30:20.628440+0000","last_undegraded":"2024-03-06T07:30:20.628440+0000","last_fullsized":"2024-03-06T07:30:20.628440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:50.201205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610000+0000","last_change":"2024-03-06T07:30:20.610000+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610000+0000","last_undegraded":"2024-03-06T07:30:20.610000+0000","last_fullsized":"2024-03-06T07:30:20.610000+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.624471+0000","last_change":"2024-03-06T07:30:20.619107+0000","last_active":"2024-03-06T07:30:20.624471+0000","last_peered":"2024-03-06T07:30:20.624471+0000","last_clean":"2024-03-06T07:30:20.624471+0000","last_became_active":"2024-03-06T07:30:20.618101+0000","last_became_peered":"2024-03-06T07:30:20.618101+0000","last_unstale":"2024-03-06T07:30:20.624471+0000","last_undegraded":"2024-03-06T07:30:20.624471+0000","last_fullsized":"2024-03-06T07:30:20.624471+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:58.572467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608554+0000","last_change":"2024-03-06T07:30:20.608554+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608554+0000","last_undegraded":"2024-03-06T07:30:20.608554+0000","last_fullsized":"2024-03-06T07:30:20.608554+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632061+0000","last_change":"2024-03-06T07:30:20.626055+0000","last_active":"2024-03-06T07:30:20.632061+0000","last_peered":"2024-03-06T07:30:20.632061+0000","last_clean":"2024-03-06T07:30:20.632061+0000","last_became_active":"2024-03-06T07:30:20.625631+0000","last_became_peered":"2024-03-06T07:30:20.625631+0000","last_unstale":"2024-03-06T07:30:20.632061+0000","last_undegraded":"2024-03-06T07:30:20.632061+0000","last_fullsized":"2024-03-06T07:30:20.632061+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:23.173745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.604785+0000","last_change":"2024-03-06T07:30:20.604785+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.604785+0000","last_undegraded":"2024-03-06T07:30:20.604785+0000","last_fullsized":"2024-03-06T07:30:20.604785+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632549+0000","last_change":"2024-03-06T07:30:20.631068+0000","last_active":"2024-03-06T07:30:20.632549+0000","last_peered":"2024-03-06T07:30:20.632549+0000","last_clean":"2024-03-06T07:30:20.632549+0000","last_became_active":"2024-03-06T07:30:20.630723+0000","last_became_peered":"2024-03-06T07:30:20.630723+0000","last_unstale":"2024-03-06T07:30:20.632549+0000","last_undegraded":"2024-03-06T07:30:20.632549+0000","last_fullsized":"2024-03-06T07:30:20.632549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:53.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606205+0000","last_change":"2024-03-06T07:30:20.606205+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606205+0000","last_undegraded":"2024-03-06T07:30:20.606205+0000","last_fullsized":"2024-03-06T07:30:20.606205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632017+0000","last_change":"2024-03-06T07:30:20.630056+0000","last_active":"2024-03-06T07:30:20.632017+0000","last_peered":"2024-03-06T07:30:20.632017+0000","last_clean":"2024-03-06T07:30:20.632017+0000","last_became_active":"2024-03-06T07:30:20.629761+0000","last_became_peered":"2024-03-06T07:30:20.629761+0000","last_unstale":"2024-03-06T07:30:20.632017+0000","last_undegraded":"2024-03-06T07:30:20.632017+0000","last_fullsized":"2024-03-06T07:30:20.632017+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:15.540003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605142+0000","last_change":"2024-03-06T07:30:20.605142+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605142+0000","last_undegraded":"2024-03-06T07:30:20.605142+0000","last_fullsized":"2024-03-06T07:30:20.605142+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641952+0000","last_change":"2024-03-06T07:30:20.618404+0000","last_active":"2024-03-06T07:30:21.641952+0000","last_peered":"2024-03-06T07:30:21.641952+0000","last_clean":"2024-03-06T07:30:21.641952+0000","last_became_active":"2024-03-06T07:30:20.617575+0000","last_became_peered":"2024-03-06T07:30:20.617575+0000","last_unstale":"2024-03-06T07:30:21.641952+0000","last_undegraded":"2024-03-06T07:30:21.641952+0000","last_fullsized":"2024-03-06T07:30:21.641952+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:03.599842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605059+0000","last_change":"2024-03-06T07:30:20.605059+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605059+0000","last_undegraded":"2024-03-06T07:30:20.605059+0000","last_fullsized":"2024-03-06T07:30:20.605059+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634627+0000","last_change":"2024-03-06T07:30:20.633615+0000","last_active":"2024-03-06T07:30:20.634627+0000","last_peered":"2024-03-06T07:30:20.634627+0000","last_clean":"2024-03-06T07:30:20.634627+0000","last_became_active":"2024-03-06T07:30:20.633225+0000","last_became_peered":"2024-03-06T07:30:20.633225+0000","last_unstale":"2024-03-06T07:30:20.634627+0000","last_undegraded":"2024-03-06T07:30:20.634627+0000","last_fullsized":"2024-03-06T07:30:20.634627+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:15.085610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606321+0000","last_change":"2024-03-06T07:30:20.606321+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606321+0000","last_undegraded":"2024-03-06T07:30:20.606321+0000","last_fullsized":"2024-03-06T07:30:20.606321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.674444+0000","last_change":"2024-03-06T07:30:20.624699+0000","last_active":"2024-03-06T07:30:21.674444+0000","last_peered":"2024-03-06T07:30:21.674444+0000","last_clean":"2024-03-06T07:30:21.674444+0000","last_became_active":"2024-03-06T07:30:20.622946+0000","last_became_peered":"2024-03-06T07:30:20.622946+0000","last_unstale":"2024-03-06T07:30:21.674444+0000","last_undegraded":"2024-03-06T07:30:21.674444+0000","last_fullsized":"2024-03-06T07:30:21.674444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:50.448937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608839+0000","last_change":"2024-03-06T07:30:20.608839+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608839+0000","last_undegraded":"2024-03-06T07:30:20.608839+0000","last_fullsized":"2024-03-06T07:30:20.608839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.631221+0000","last_change":"2024-03-06T07:30:20.629443+0000","last_active":"2024-03-06T07:30:20.631221+0000","last_peered":"2024-03-06T07:30:20.631221+0000","last_clean":"2024-03-06T07:30:20.631221+0000","last_became_active":"2024-03-06T07:30:20.629144+0000","last_became_peered":"2024-03-06T07:30:20.629144+0000","last_unstale":"2024-03-06T07:30:20.631221+0000","last_undegraded":"2024-03-06T07:30:20.631221+0000","last_fullsized":"2024-03-06T07:30:20.631221+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:37.100598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.665902+0000","last_change":"2024-03-06T07:30:21.650700+0000","last_active":"2024-03-06T07:30:21.665902+0000","last_peered":"2024-03-06T07:30:21.665902+0000","last_clean":"2024-03-06T07:30:21.665902+0000","last_became_active":"2024-03-06T07:30:21.650363+0000","last_became_peered":"2024-03-06T07:30:21.650363+0000","last_unstale":"2024-03-06T07:30:21.665902+0000","last_undegraded":"2024-03-06T07:30:21.665902+0000","last_fullsized":"2024-03-06T07:30:21.665902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:03.131576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632815+0000","last_change":"2024-03-06T07:30:20.629663+0000","last_active":"2024-03-06T07:30:20.632815+0000","last_peered":"2024-03-06T07:30:20.632815+0000","last_clean":"2024-03-06T07:30:20.632815+0000","last_became_active":"2024-03-06T07:30:20.629365+0000","last_became_peered":"2024-03-06T07:30:20.629365+0000","last_unstale":"2024-03-06T07:30:20.632815+0000","last_undegraded":"2024-03-06T07:30:20.632815+0000","last_fullsized":"2024-03-06T07:30:20.632815+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:20.066237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612360+0000","last_change":"2024-03-06T07:30:20.612360+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612360+0000","last_undegraded":"2024-03-06T07:30:20.612360+0000","last_fullsized":"2024-03-06T07:30:20.612360+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633860+0000","last_change":"2024-03-06T07:30:20.631439+0000","last_active":"2024-03-06T07:30:20.633860+0000","last_peered":"2024-03-06T07:30:20.633860+0000","last_clean":"2024-03-06T07:30:20.633860+0000","last_became_active":"2024-03-06T07:30:20.631100+0000","last_became_peered":"2024-03-06T07:30:20.631100+0000","last_unstale":"2024-03-06T07:30:20.633860+0000","last_undegraded":"2024-03-06T07:30:20.633860+0000","last_fullsized":"2024-03-06T07:30:20.633860+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:37.100598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610052+0000","last_change":"2024-03-06T07:30:20.610052+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610052+0000","last_undegraded":"2024-03-06T07:30:20.610052+0000","last_fullsized":"2024-03-06T07:30:20.610052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641987+0000","last_change":"2024-03-06T07:30:20.623625+0000","last_active":"2024-03-06T07:30:21.641987+0000","last_peered":"2024-03-06T07:30:21.641987+0000","last_clean":"2024-03-06T07:30:21.641987+0000","last_became_active":"2024-03-06T07:30:20.623355+0000","last_became_peered":"2024-03-06T07:30:20.623355+0000","last_unstale":"2024-03-06T07:30:21.641987+0000","last_undegraded":"2024-03-06T07:30:21.641987+0000","last_fullsized":"2024-03-06T07:30:21.641987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:46.004625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.601988+0000","last_change":"2024-03-06T07:30:20.601988+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.601988+0000","last_undegraded":"2024-03-06T07:30:20.601988+0000","last_fullsized":"2024-03-06T07:30:20.601988+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634005+0000","last_change":"2024-03-06T07:30:20.632460+0000","last_active":"2024-03-06T07:30:20.634005+0000","last_peered":"2024-03-06T07:30:20.634005+0000","last_clean":"2024-03-06T07:30:20.634005+0000","last_became_active":"2024-03-06T07:30:20.630675+0000","last_became_peered":"2024-03-06T07:30:20.630675+0000","last_unstale":"2024-03-06T07:30:20.634005+0000","last_undegraded":"2024-03-06T07:30:20.634005+0000","last_fullsized":"2024-03-06T07:30:20.634005+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:18.735168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.670594+0000","last_change":"2024-03-06T07:30:21.661218+0000","last_active":"2024-03-06T07:30:21.670594+0000","last_peered":"2024-03-06T07:30:21.670594+0000","last_clean":"2024-03-06T07:30:21.670594+0000","last_became_active":"2024-03-06T07:30:21.660549+0000","last_became_peered":"2024-03-06T07:30:21.660549+0000","last_unstale":"2024-03-06T07:30:21.670594+0000","last_undegraded":"2024-03-06T07:30:21.670594+0000","last_fullsized":"2024-03-06T07:30:21.670594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:48.433580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641620+0000","last_change":"2024-03-06T07:30:20.609520+0000","last_active":"2024-03-06T07:30:21.641620+0000","last_peered":"2024-03-06T07:30:21.641620+0000","last_clean":"2024-03-06T07:30:21.641620+0000","last_became_active":"2024-03-06T07:30:20.608944+0000","last_became_peered":"2024-03-06T07:30:20.608944+0000","last_unstale":"2024-03-06T07:30:21.641620+0000","last_undegraded":"2024-03-06T07:30:21.641620+0000","last_fullsized":"2024-03-06T07:30:21.641620+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:54.924862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614665+0000","last_change":"2024-03-06T07:30:20.614665+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614665+0000","last_undegraded":"2024-03-06T07:30:20.614665+0000","last_fullsized":"2024-03-06T07:30:20.614665+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630868+0000","last_change":"2024-03-06T07:30:20.624382+0000","last_active":"2024-03-06T07:30:20.630868+0000","last_peered":"2024-03-06T07:30:20.630868+0000","last_clean":"2024-03-06T07:30:20.630868+0000","last_became_active":"2024-03-06T07:30:20.624100+0000","last_became_peered":"2024-03-06T07:30:20.624100+0000","last_unstale":"2024-03-06T07:30:20.630868+0000","last_undegraded":"2024-03-06T07:30:20.630868+0000","last_fullsized":"2024-03-06T07:30:20.630868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:22.937616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607340+0000","last_change":"2024-03-06T07:30:20.607340+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607340+0000","last_undegraded":"2024-03-06T07:30:20.607340+0000","last_fullsized":"2024-03-06T07:30:20.607340+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628428+0000","last_change":"2024-03-06T07:30:20.624654+0000","last_active":"2024-03-06T07:30:20.628428+0000","last_peered":"2024-03-06T07:30:20.628428+0000","last_clean":"2024-03-06T07:30:20.628428+0000","last_became_active":"2024-03-06T07:30:20.624112+0000","last_became_peered":"2024-03-06T07:30:20.624112+0000","last_unstale":"2024-03-06T07:30:20.628428+0000","last_undegraded":"2024-03-06T07:30:20.628428+0000","last_fullsized":"2024-03-06T07:30:20.628428+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:34.375041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.614301+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:30:20.614301+0000","last_peered":"2024-03-06T07:30:20.614301+0000","last_clean":"2024-03-06T07:30:20.614301+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:30:20.614301+0000","last_undegraded":"2024-03-06T07:30:20.614301+0000","last_fullsized":"2024-03-06T07:30:20.614301+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509479,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":279263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":279297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542183,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":862800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542184,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542183,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542184,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1257472,"data_stored":862800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:26.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.888+0000 7fea3f200640 1 Processor -- start 2024-03-06T07:30:26.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.888+0000 7fea3f200640 1 -- start start 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.889+0000 7fea3f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 0x7fea400b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.889+0000 7fea3f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fea40070f10 0x7fea400cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.889+0000 7fea3f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400714d0 con 0x7fea400cce90 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.889+0000 7fea3f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea400cd300 con 0x7fea40070f10 2024-03-06T07:30:26.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.889+0000 7fea3f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400717b0 con 0x7fea40145300 2024-03-06T07:30:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.890+0000 7fea3de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 0x7fea400b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.890+0000 7fea3de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 0x7fea400b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:37770/0 (socket says 172.21.15.6:37770) 2024-03-06T07:30:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.890+0000 7fea3de00640 1 -- 172.21.15.6:0/2681239275 learned_addr learned my addr 172.21.15.6:0/2681239275 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.890+0000 7fea37e00640 1 --2- 172.21.15.6:0/2681239275 >> v2:172.21.15.178:3300/0 conn(0x7fea40070f10 0x7fea400cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.891+0000 7fea3de00640 1 -- 172.21.15.6:0/2681239275 >> v2:172.21.15.178:3300/0 conn(0x7fea40070f10 msgr2=0x7fea400cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.891+0000 7fea3de00640 1 --2- 172.21.15.6:0/2681239275 >> v2:172.21.15.178:3300/0 conn(0x7fea40070f10 0x7fea400cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.891+0000 7fea3de00640 1 -- 172.21.15.6:0/2681239275 >> v1:172.21.15.178:6789/0 conn(0x7fea400cce90 legacy=0x7fea400cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.891+0000 7fea3de00640 1 -- 172.21.15.6:0/2681239275 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea40071a90 con 0x7fea40145300 2024-03-06T07:30:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.892+0000 7fea3de00640 1 --2- 172.21.15.6:0/2681239275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 0x7fea400b49a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea30009870 tx=0x7fea3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57c4178e36be1b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.893+0000 7fea3d400640 1 -- 172.21.15.6:0/2681239275 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30034070 con 0x7fea40145300 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.893+0000 7fea3d400640 1 -- 172.21.15.6:0/2681239275 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30030840 con 0x7fea40145300 2024-03-06T07:30:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.893+0000 7fea3d400640 1 -- 172.21.15.6:0/2681239275 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30030c10 con 0x7fea40145300 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.893+0000 7fea3f200640 1 -- 172.21.15.6:0/2681239275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 msgr2=0x7fea400b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 --2- 172.21.15.6:0/2681239275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 0x7fea400b49a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fea30009870 tx=0x7fea3002b980 comp rx=0 tx=0).stop 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 -- 172.21.15.6:0/2681239275 shutdown_connections 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 --2- 172.21.15.6:0/2681239275 >> v2:172.21.15.178:3300/0 conn(0x7fea40070f10 0x7fea400cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 --2- 172.21.15.6:0/2681239275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40145300 0x7fea400b49a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 -- 172.21.15.6:0/2681239275 >> 172.21.15.6:0/2681239275 conn(0x7fea4006b3d0 msgr2=0x7fea4006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 -- 172.21.15.6:0/2681239275 shutdown_connections 2024-03-06T07:30:26.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.894+0000 7fea3f200640 1 -- 172.21.15.6:0/2681239275 wait complete. 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.895+0000 7fea3f200640 1 Processor -- start 2024-03-06T07:30:26.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.896+0000 7fea3f200640 1 -- start start 2024-03-06T07:30:26.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.896+0000 7fea3f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40070f10 0x7fea401e5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.896+0000 7fea3f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 0x7fea401e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.896+0000 7fea3f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea40145300 0x7fea401e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.896+0000 7fea3f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea401456e0 con 0x7fea400cce90 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.896+0000 7fea3f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b4ee0 con 0x7fea40145300 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea3f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4015d580 con 0x7fea40070f10 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 0x7fea401e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 0x7fea401e72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34120/0 (socket says 172.21.15.6:34120) 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 -- 172.21.15.6:0/4250117195 learned_addr learned my addr 172.21.15.6:0/4250117195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea3de00640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40070f10 0x7fea401e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 -- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40070f10 msgr2=0x7fea401e5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40070f10 0x7fea401e5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 -- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea40145300 msgr2=0x7fea401e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea3e800640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea40145300 0x7fea401e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea40145300 0x7fea401e6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea37e00640 1 -- 172.21.15.6:0/4250117195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea401e4890 con 0x7fea400cce90 2024-03-06T07:30:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.897+0000 7fea3e800640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea40145300 0x7fea401e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.898+0000 7fea37e00640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 0x7fea401e72d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fea2800e9b0 tx=0x7fea2800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.898+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28017070 con 0x7fea400cce90 2024-03-06T07:30:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.898+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2800c9a0 con 0x7fea400cce90 2024-03-06T07:30:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.898+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea401e4b70 con 0x7fea400cce90 2024-03-06T07:30:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.898+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800cd90 con 0x7fea400cce90 2024-03-06T07:30:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.898+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea401e5130 con 0x7fea400cce90 2024-03-06T07:30:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.899+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea40067830 con 0x7fea400cce90 2024-03-06T07:30:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.901+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea2801a070 con 0x7fea400cce90 2024-03-06T07:30:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.902+0000 7fea36000640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea1c06b4d0 0x7fea1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.902+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea2808a5e0 con 0x7fea400cce90 2024-03-06T07:30:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.903+0000 7fea3de00640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea1c06b4d0 0x7fea1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.904+0000 7fea3de00640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea1c06b4d0 0x7fea1c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fea30036000 tx=0x7fea3003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:26.907+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea40067830 con 0x7fea400cce90 2024-03-06T07:30:27.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.099+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea4015e360 con 0x7fea1c06b4d0 2024-03-06T07:30:27.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.119+0000 7fea36000640 1 -- 172.21.15.6:0/4250117195 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247619 (secure 0 0 0) 0x7fea4015e360 con 0x7fea1c06b4d0 2024-03-06T07:30:27.125 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:27.126 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea1c06b4d0 msgr2=0x7fea1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea1c06b4d0 0x7fea1c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fea30036000 tx=0x7fea3003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 msgr2=0x7fea401e72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 0x7fea401e72d0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fea2800e9b0 tx=0x7fea2800ee70 comp rx=0 tx=0).stop 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 shutdown_connections 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea1c06b4d0 0x7fea1c06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea40145300 0x7fea401e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea400cce90 0x7fea401e72d0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 --2- 172.21.15.6:0/4250117195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea40070f10 0x7fea401e5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 >> 172.21.15.6:0/4250117195 conn(0x7fea4006b3d0 msgr2=0x7fea400cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 shutdown_connections 2024-03-06T07:30:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.125+0000 7fea3f200640 1 -- 172.21.15.6:0/4250117195 wait complete. 2024-03-06T07:30:27.146 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-06T07:30:25.499841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":162,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264180,"kb_used_data":6380,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754710540,"statfs":{"total":773094113280,"available":772823592960,"internally_reserved":0,"allocated":6533120,"data_stored":3373888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.909824"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.642928+0000","last_change":"2024-03-06T07:30:21.639251+0000","last_active":"2024-03-06T07:30:21.642928+0000","last_peered":"2024-03-06T07:30:21.642928+0000","last_clean":"2024-03-06T07:30:21.642928+0000","last_became_active":"2024-03-06T07:30:21.638950+0000","last_became_peered":"2024-03-06T07:30:21.638950+0000","last_unstale":"2024-03-06T07:30:21.642928+0000","last_undegraded":"2024-03-06T07:30:21.642928+0000","last_fullsized":"2024-03-06T07:30:21.642928+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:57.957584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605846+0000","last_change":"2024-03-06T07:30:20.605846+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605846+0000","last_undegraded":"2024-03-06T07:30:20.605846+0000","last_fullsized":"2024-03-06T07:30:20.605846+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.609643+0000","last_change":"2024-03-06T07:30:20.609643+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.609643+0000","last_undegraded":"2024-03-06T07:30:20.609643+0000","last_fullsized":"2024-03-06T07:30:20.609643+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.609392+0000","last_change":"2024-03-06T07:30:20.609392+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.609392+0000","last_undegraded":"2024-03-06T07:30:20.609392+0000","last_fullsized":"2024-03-06T07:30:20.609392+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.675095+0000","last_change":"2024-03-06T07:30:21.671393+0000","last_active":"2024-03-06T07:30:21.675095+0000","last_peered":"2024-03-06T07:30:21.675095+0000","last_clean":"2024-03-06T07:30:21.675095+0000","last_became_active":"2024-03-06T07:30:21.671107+0000","last_became_peered":"2024-03-06T07:30:21.671107+0000","last_unstale":"2024-03-06T07:30:21.675095+0000","last_undegraded":"2024-03-06T07:30:21.675095+0000","last_fullsized":"2024-03-06T07:30:21.675095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:19:49.030206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605107+0000","last_change":"2024-03-06T07:30:20.605107+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605107+0000","last_undegraded":"2024-03-06T07:30:20.605107+0000","last_fullsized":"2024-03-06T07:30:20.605107+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617084+0000","last_change":"2024-03-06T07:30:20.617084+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617084+0000","last_undegraded":"2024-03-06T07:30:20.617084+0000","last_fullsized":"2024-03-06T07:30:20.617084+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607822+0000","last_change":"2024-03-06T07:30:20.607822+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607822+0000","last_undegraded":"2024-03-06T07:30:20.607822+0000","last_fullsized":"2024-03-06T07:30:20.607822+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.621543+0000","last_change":"2024-03-06T07:30:20.621543+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.621543+0000","last_undegraded":"2024-03-06T07:30:20.621543+0000","last_fullsized":"2024-03-06T07:30:20.621543+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668500+0000","last_change":"2024-03-06T07:30:21.658602+0000","last_active":"2024-03-06T07:30:21.668500+0000","last_peered":"2024-03-06T07:30:21.668500+0000","last_clean":"2024-03-06T07:30:21.668500+0000","last_became_active":"2024-03-06T07:30:21.656608+0000","last_became_peered":"2024-03-06T07:30:21.656608+0000","last_unstale":"2024-03-06T07:30:21.668500+0000","last_undegraded":"2024-03-06T07:30:21.668500+0000","last_fullsized":"2024-03-06T07:30:21.668500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:14:06.554234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668477+0000","last_change":"2024-03-06T07:30:21.656749+0000","last_active":"2024-03-06T07:30:21.668477+0000","last_peered":"2024-03-06T07:30:21.668477+0000","last_clean":"2024-03-06T07:30:21.668477+0000","last_became_active":"2024-03-06T07:30:21.656479+0000","last_became_peered":"2024-03-06T07:30:21.656479+0000","last_unstale":"2024-03-06T07:30:21.668477+0000","last_undegraded":"2024-03-06T07:30:21.668477+0000","last_fullsized":"2024-03-06T07:30:21.668477+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:25.712837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606399+0000","last_change":"2024-03-06T07:30:20.606399+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606399+0000","last_undegraded":"2024-03-06T07:30:20.606399+0000","last_fullsized":"2024-03-06T07:30:20.606399+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614576+0000","last_change":"2024-03-06T07:30:20.614576+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614576+0000","last_undegraded":"2024-03-06T07:30:20.614576+0000","last_fullsized":"2024-03-06T07:30:20.614576+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.620181+0000","last_change":"2024-03-06T07:30:20.620181+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.620181+0000","last_undegraded":"2024-03-06T07:30:20.620181+0000","last_fullsized":"2024-03-06T07:30:20.620181+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.616435+0000","last_change":"2024-03-06T07:30:20.616435+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.616435+0000","last_undegraded":"2024-03-06T07:30:20.616435+0000","last_fullsized":"2024-03-06T07:30:20.616435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.604507+0000","last_change":"2024-03-06T07:30:20.604507+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.604507+0000","last_undegraded":"2024-03-06T07:30:20.604507+0000","last_fullsized":"2024-03-06T07:30:20.604507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607582+0000","last_change":"2024-03-06T07:30:20.607582+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607582+0000","last_undegraded":"2024-03-06T07:30:20.607582+0000","last_fullsized":"2024-03-06T07:30:20.607582+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606705+0000","last_change":"2024-03-06T07:30:20.606705+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606705+0000","last_undegraded":"2024-03-06T07:30:20.606705+0000","last_fullsized":"2024-03-06T07:30:20.606705+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.639713+0000","last_change":"2024-03-06T07:30:21.635273+0000","last_active":"2024-03-06T07:30:21.639713+0000","last_peered":"2024-03-06T07:30:21.639713+0000","last_clean":"2024-03-06T07:30:21.639713+0000","last_became_active":"2024-03-06T07:30:21.634938+0000","last_became_peered":"2024-03-06T07:30:21.634938+0000","last_unstale":"2024-03-06T07:30:21.639713+0000","last_undegraded":"2024-03-06T07:30:21.639713+0000","last_fullsized":"2024-03-06T07:30:21.639713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:35:28.549452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608995+0000","last_change":"2024-03-06T07:30:20.608995+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608995+0000","last_undegraded":"2024-03-06T07:30:20.608995+0000","last_fullsized":"2024-03-06T07:30:20.608995+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612095+0000","last_change":"2024-03-06T07:30:20.612095+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612095+0000","last_undegraded":"2024-03-06T07:30:20.612095+0000","last_fullsized":"2024-03-06T07:30:20.612095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612050+0000","last_change":"2024-03-06T07:30:20.612050+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612050+0000","last_undegraded":"2024-03-06T07:30:20.612050+0000","last_fullsized":"2024-03-06T07:30:20.612050+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.603712+0000","last_change":"2024-03-06T07:30:20.603712+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.603712+0000","last_undegraded":"2024-03-06T07:30:20.603712+0000","last_fullsized":"2024-03-06T07:30:20.603712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.618309+0000","last_change":"2024-03-06T07:30:20.618309+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.618309+0000","last_undegraded":"2024-03-06T07:30:20.618309+0000","last_fullsized":"2024-03-06T07:30:20.618309+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.648459+0000","last_change":"2024-03-06T07:30:21.646940+0000","last_active":"2024-03-06T07:30:21.648459+0000","last_peered":"2024-03-06T07:30:21.648459+0000","last_clean":"2024-03-06T07:30:21.648459+0000","last_became_active":"2024-03-06T07:30:21.644768+0000","last_became_peered":"2024-03-06T07:30:21.644768+0000","last_unstale":"2024-03-06T07:30:21.648459+0000","last_undegraded":"2024-03-06T07:30:21.648459+0000","last_fullsized":"2024-03-06T07:30:21.648459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:24.687045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610660+0000","last_change":"2024-03-06T07:30:20.610660+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610660+0000","last_undegraded":"2024-03-06T07:30:20.610660+0000","last_fullsized":"2024-03-06T07:30:20.610660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.613593+0000","last_change":"2024-03-06T07:30:20.613593+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.613593+0000","last_undegraded":"2024-03-06T07:30:20.613593+0000","last_fullsized":"2024-03-06T07:30:20.613593+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.668154+0000","last_change":"2024-03-06T07:30:21.663886+0000","last_active":"2024-03-06T07:30:21.668154+0000","last_peered":"2024-03-06T07:30:21.668154+0000","last_clean":"2024-03-06T07:30:21.668154+0000","last_became_active":"2024-03-06T07:30:21.663511+0000","last_became_peered":"2024-03-06T07:30:21.663511+0000","last_unstale":"2024-03-06T07:30:21.668154+0000","last_undegraded":"2024-03-06T07:30:21.668154+0000","last_fullsized":"2024-03-06T07:30:21.668154+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:27.109206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.616277+0000","last_change":"2024-03-06T07:30:20.616277+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.616277+0000","last_undegraded":"2024-03-06T07:30:20.616277+0000","last_fullsized":"2024-03-06T07:30:20.616277+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.601987+0000","last_change":"2024-03-06T07:30:20.601987+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.601987+0000","last_undegraded":"2024-03-06T07:30:20.601987+0000","last_fullsized":"2024-03-06T07:30:20.601987+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.648777+0000","last_change":"2024-03-06T07:30:21.645937+0000","last_active":"2024-03-06T07:30:21.648777+0000","last_peered":"2024-03-06T07:30:21.648777+0000","last_clean":"2024-03-06T07:30:21.648777+0000","last_became_active":"2024-03-06T07:30:21.644909+0000","last_became_peered":"2024-03-06T07:30:21.644909+0000","last_unstale":"2024-03-06T07:30:21.648777+0000","last_undegraded":"2024-03-06T07:30:21.648777+0000","last_fullsized":"2024-03-06T07:30:21.648777+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:18.572142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.603960+0000","last_change":"2024-03-06T07:30:20.603960+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.603960+0000","last_undegraded":"2024-03-06T07:30:20.603960+0000","last_fullsized":"2024-03-06T07:30:20.603960+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606790+0000","last_change":"2024-03-06T07:30:20.606790+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606790+0000","last_undegraded":"2024-03-06T07:30:20.606790+0000","last_fullsized":"2024-03-06T07:30:20.606790+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632214+0000","last_change":"2024-03-06T07:30:20.630369+0000","last_active":"2024-03-06T07:30:20.632214+0000","last_peered":"2024-03-06T07:30:20.632214+0000","last_clean":"2024-03-06T07:30:20.632214+0000","last_became_active":"2024-03-06T07:30:20.628580+0000","last_became_peered":"2024-03-06T07:30:20.628580+0000","last_unstale":"2024-03-06T07:30:20.632214+0000","last_undegraded":"2024-03-06T07:30:20.632214+0000","last_fullsized":"2024-03-06T07:30:20.632214+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:15.085610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.671038+0000","last_change":"2024-03-06T07:30:21.664516+0000","last_active":"2024-03-06T07:30:21.671038+0000","last_peered":"2024-03-06T07:30:21.671038+0000","last_clean":"2024-03-06T07:30:21.671038+0000","last_became_active":"2024-03-06T07:30:21.664224+0000","last_became_peered":"2024-03-06T07:30:21.664224+0000","last_unstale":"2024-03-06T07:30:21.671038+0000","last_undegraded":"2024-03-06T07:30:21.671038+0000","last_fullsized":"2024-03-06T07:30:21.671038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:53.768918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.629363+0000","last_change":"2024-03-06T07:30:20.625914+0000","last_active":"2024-03-06T07:30:20.629363+0000","last_peered":"2024-03-06T07:30:20.629363+0000","last_clean":"2024-03-06T07:30:20.629363+0000","last_became_active":"2024-03-06T07:30:20.625643+0000","last_became_peered":"2024-03-06T07:30:20.625643+0000","last_unstale":"2024-03-06T07:30:20.629363+0000","last_undegraded":"2024-03-06T07:30:20.629363+0000","last_fullsized":"2024-03-06T07:30:20.629363+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:15.048674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606274+0000","last_change":"2024-03-06T07:30:20.606274+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606274+0000","last_undegraded":"2024-03-06T07:30:20.606274+0000","last_fullsized":"2024-03-06T07:30:20.606274+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.619866+0000","last_change":"2024-03-06T07:30:20.617109+0000","last_active":"2024-03-06T07:30:20.619866+0000","last_peered":"2024-03-06T07:30:20.619866+0000","last_clean":"2024-03-06T07:30:20.619866+0000","last_became_active":"2024-03-06T07:30:20.615288+0000","last_became_peered":"2024-03-06T07:30:20.615288+0000","last_unstale":"2024-03-06T07:30:20.619866+0000","last_undegraded":"2024-03-06T07:30:20.619866+0000","last_fullsized":"2024-03-06T07:30:20.619866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:37.430833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.613038+0000","last_change":"2024-03-06T07:30:20.613038+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.613038+0000","last_undegraded":"2024-03-06T07:30:20.613038+0000","last_fullsized":"2024-03-06T07:30:20.613038+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.631863+0000","last_change":"2024-03-06T07:30:20.626609+0000","last_active":"2024-03-06T07:30:20.631863+0000","last_peered":"2024-03-06T07:30:20.631863+0000","last_clean":"2024-03-06T07:30:20.631863+0000","last_became_active":"2024-03-06T07:30:20.625668+0000","last_became_peered":"2024-03-06T07:30:20.625668+0000","last_unstale":"2024-03-06T07:30:20.631863+0000","last_undegraded":"2024-03-06T07:30:20.631863+0000","last_fullsized":"2024-03-06T07:30:20.631863+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:37.430833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.611537+0000","last_change":"2024-03-06T07:30:20.611537+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.611537+0000","last_undegraded":"2024-03-06T07:30:20.611537+0000","last_fullsized":"2024-03-06T07:30:20.611537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633345+0000","last_change":"2024-03-06T07:30:20.629926+0000","last_active":"2024-03-06T07:30:20.633345+0000","last_peered":"2024-03-06T07:30:20.633345+0000","last_clean":"2024-03-06T07:30:20.633345+0000","last_became_active":"2024-03-06T07:30:20.629470+0000","last_became_peered":"2024-03-06T07:30:20.629470+0000","last_unstale":"2024-03-06T07:30:20.633345+0000","last_undegraded":"2024-03-06T07:30:20.633345+0000","last_fullsized":"2024-03-06T07:30:20.633345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:11.638813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.618124+0000","last_change":"2024-03-06T07:30:20.618124+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.618124+0000","last_undegraded":"2024-03-06T07:30:20.618124+0000","last_fullsized":"2024-03-06T07:30:20.618124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630744+0000","last_change":"2024-03-06T07:30:20.627675+0000","last_active":"2024-03-06T07:30:20.630744+0000","last_peered":"2024-03-06T07:30:20.630744+0000","last_clean":"2024-03-06T07:30:20.630744+0000","last_became_active":"2024-03-06T07:30:20.627383+0000","last_became_peered":"2024-03-06T07:30:20.627383+0000","last_unstale":"2024-03-06T07:30:20.630744+0000","last_undegraded":"2024-03-06T07:30:20.630744+0000","last_fullsized":"2024-03-06T07:30:20.630744+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.218902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614522+0000","last_change":"2024-03-06T07:30:20.614522+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614522+0000","last_undegraded":"2024-03-06T07:30:20.614522+0000","last_fullsized":"2024-03-06T07:30:20.614522+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633087+0000","last_change":"2024-03-06T07:30:20.629740+0000","last_active":"2024-03-06T07:30:20.633087+0000","last_peered":"2024-03-06T07:30:20.633087+0000","last_clean":"2024-03-06T07:30:20.633087+0000","last_became_active":"2024-03-06T07:30:20.629427+0000","last_became_peered":"2024-03-06T07:30:20.629427+0000","last_unstale":"2024-03-06T07:30:20.633087+0000","last_undegraded":"2024-03-06T07:30:20.633087+0000","last_fullsized":"2024-03-06T07:30:20.633087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:06.218902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617560+0000","last_change":"2024-03-06T07:30:20.617560+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617560+0000","last_undegraded":"2024-03-06T07:30:20.617560+0000","last_fullsized":"2024-03-06T07:30:20.617560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634979+0000","last_change":"2024-03-06T07:30:20.634196+0000","last_active":"2024-03-06T07:30:20.634979+0000","last_peered":"2024-03-06T07:30:20.634979+0000","last_clean":"2024-03-06T07:30:20.634979+0000","last_became_active":"2024-03-06T07:30:20.633405+0000","last_became_peered":"2024-03-06T07:30:20.633405+0000","last_unstale":"2024-03-06T07:30:20.634979+0000","last_undegraded":"2024-03-06T07:30:20.634979+0000","last_fullsized":"2024-03-06T07:30:20.634979+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:00.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.621452+0000","last_change":"2024-03-06T07:30:20.621452+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.621452+0000","last_undegraded":"2024-03-06T07:30:20.621452+0000","last_fullsized":"2024-03-06T07:30:20.621452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630055+0000","last_change":"2024-03-06T07:30:20.627513+0000","last_active":"2024-03-06T07:30:20.630055+0000","last_peered":"2024-03-06T07:30:20.630055+0000","last_clean":"2024-03-06T07:30:20.630055+0000","last_became_active":"2024-03-06T07:30:20.627220+0000","last_became_peered":"2024-03-06T07:30:20.627220+0000","last_unstale":"2024-03-06T07:30:20.630055+0000","last_undegraded":"2024-03-06T07:30:20.630055+0000","last_fullsized":"2024-03-06T07:30:20.630055+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:18.735168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605345+0000","last_change":"2024-03-06T07:30:20.605345+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605345+0000","last_undegraded":"2024-03-06T07:30:20.605345+0000","last_fullsized":"2024-03-06T07:30:20.605345+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630935+0000","last_change":"2024-03-06T07:30:20.628649+0000","last_active":"2024-03-06T07:30:20.630935+0000","last_peered":"2024-03-06T07:30:20.630935+0000","last_clean":"2024-03-06T07:30:20.630935+0000","last_became_active":"2024-03-06T07:30:20.628065+0000","last_became_peered":"2024-03-06T07:30:20.628065+0000","last_unstale":"2024-03-06T07:30:20.630935+0000","last_undegraded":"2024-03-06T07:30:20.630935+0000","last_fullsized":"2024-03-06T07:30:20.630935+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:30.404064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.667370+0000","last_change":"2024-03-06T07:30:21.652433+0000","last_active":"2024-03-06T07:30:21.667370+0000","last_peered":"2024-03-06T07:30:21.667370+0000","last_clean":"2024-03-06T07:30:21.667370+0000","last_became_active":"2024-03-06T07:30:21.651796+0000","last_became_peered":"2024-03-06T07:30:21.651796+0000","last_unstale":"2024-03-06T07:30:21.667370+0000","last_undegraded":"2024-03-06T07:30:21.667370+0000","last_fullsized":"2024-03-06T07:30:21.667370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:04.774122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630750+0000","last_change":"2024-03-06T07:30:20.628448+0000","last_active":"2024-03-06T07:30:20.630750+0000","last_peered":"2024-03-06T07:30:20.630750+0000","last_clean":"2024-03-06T07:30:20.630750+0000","last_became_active":"2024-03-06T07:30:20.628114+0000","last_became_peered":"2024-03-06T07:30:20.628114+0000","last_unstale":"2024-03-06T07:30:20.630750+0000","last_undegraded":"2024-03-06T07:30:20.630750+0000","last_fullsized":"2024-03-06T07:30:20.630750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:00.719980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608641+0000","last_change":"2024-03-06T07:30:20.608641+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608641+0000","last_undegraded":"2024-03-06T07:30:20.608641+0000","last_fullsized":"2024-03-06T07:30:20.608641+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633000+0000","last_change":"2024-03-06T07:30:20.629645+0000","last_active":"2024-03-06T07:30:20.633000+0000","last_peered":"2024-03-06T07:30:20.633000+0000","last_clean":"2024-03-06T07:30:20.633000+0000","last_became_active":"2024-03-06T07:30:20.628235+0000","last_became_peered":"2024-03-06T07:30:20.628235+0000","last_unstale":"2024-03-06T07:30:20.633000+0000","last_undegraded":"2024-03-06T07:30:20.633000+0000","last_fullsized":"2024-03-06T07:30:20.633000+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:21:52.757187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607488+0000","last_change":"2024-03-06T07:30:20.607488+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607488+0000","last_undegraded":"2024-03-06T07:30:20.607488+0000","last_fullsized":"2024-03-06T07:30:20.607488+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630783+0000","last_change":"2024-03-06T07:30:20.627658+0000","last_active":"2024-03-06T07:30:20.630783+0000","last_peered":"2024-03-06T07:30:20.630783+0000","last_clean":"2024-03-06T07:30:20.630783+0000","last_became_active":"2024-03-06T07:30:20.627370+0000","last_became_peered":"2024-03-06T07:30:20.627370+0000","last_unstale":"2024-03-06T07:30:20.630783+0000","last_undegraded":"2024-03-06T07:30:20.630783+0000","last_fullsized":"2024-03-06T07:30:20.630783+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:37.737710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610618+0000","last_change":"2024-03-06T07:30:20.610618+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610618+0000","last_undegraded":"2024-03-06T07:30:20.610618+0000","last_fullsized":"2024-03-06T07:30:20.610618+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.629261+0000","last_change":"2024-03-06T07:30:20.626528+0000","last_active":"2024-03-06T07:30:20.629261+0000","last_peered":"2024-03-06T07:30:20.629261+0000","last_clean":"2024-03-06T07:30:20.629261+0000","last_became_active":"2024-03-06T07:30:20.626159+0000","last_became_peered":"2024-03-06T07:30:20.626159+0000","last_unstale":"2024-03-06T07:30:20.629261+0000","last_undegraded":"2024-03-06T07:30:20.629261+0000","last_fullsized":"2024-03-06T07:30:20.629261+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:59.777807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.617892+0000","last_change":"2024-03-06T07:30:20.617892+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.617892+0000","last_undegraded":"2024-03-06T07:30:20.617892+0000","last_fullsized":"2024-03-06T07:30:20.617892+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633067+0000","last_change":"2024-03-06T07:30:20.628917+0000","last_active":"2024-03-06T07:30:20.633067+0000","last_peered":"2024-03-06T07:30:20.633067+0000","last_clean":"2024-03-06T07:30:20.633067+0000","last_became_active":"2024-03-06T07:30:20.625883+0000","last_became_peered":"2024-03-06T07:30:20.625883+0000","last_unstale":"2024-03-06T07:30:20.633067+0000","last_undegraded":"2024-03-06T07:30:20.633067+0000","last_fullsized":"2024-03-06T07:30:20.633067+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:58.572467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.602149+0000","last_change":"2024-03-06T07:30:20.602149+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.602149+0000","last_undegraded":"2024-03-06T07:30:20.602149+0000","last_fullsized":"2024-03-06T07:30:20.602149+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628580+0000","last_change":"2024-03-06T07:30:20.624267+0000","last_active":"2024-03-06T07:30:20.628580+0000","last_peered":"2024-03-06T07:30:20.628580+0000","last_clean":"2024-03-06T07:30:20.628580+0000","last_became_active":"2024-03-06T07:30:20.623498+0000","last_became_peered":"2024-03-06T07:30:20.623498+0000","last_unstale":"2024-03-06T07:30:20.628580+0000","last_undegraded":"2024-03-06T07:30:20.628580+0000","last_fullsized":"2024-03-06T07:30:20.628580+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:11.638813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605234+0000","last_change":"2024-03-06T07:30:20.605234+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605234+0000","last_undegraded":"2024-03-06T07:30:20.605234+0000","last_fullsized":"2024-03-06T07:30:20.605234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628440+0000","last_change":"2024-03-06T07:30:20.625469+0000","last_active":"2024-03-06T07:30:20.628440+0000","last_peered":"2024-03-06T07:30:20.628440+0000","last_clean":"2024-03-06T07:30:20.628440+0000","last_became_active":"2024-03-06T07:30:20.625169+0000","last_became_peered":"2024-03-06T07:30:20.625169+0000","last_unstale":"2024-03-06T07:30:20.628440+0000","last_undegraded":"2024-03-06T07:30:20.628440+0000","last_fullsized":"2024-03-06T07:30:20.628440+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:50.201205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610000+0000","last_change":"2024-03-06T07:30:20.610000+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610000+0000","last_undegraded":"2024-03-06T07:30:20.610000+0000","last_fullsized":"2024-03-06T07:30:20.610000+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.624471+0000","last_change":"2024-03-06T07:30:20.619107+0000","last_active":"2024-03-06T07:30:20.624471+0000","last_peered":"2024-03-06T07:30:20.624471+0000","last_clean":"2024-03-06T07:30:20.624471+0000","last_became_active":"2024-03-06T07:30:20.618101+0000","last_became_peered":"2024-03-06T07:30:20.618101+0000","last_unstale":"2024-03-06T07:30:20.624471+0000","last_undegraded":"2024-03-06T07:30:20.624471+0000","last_fullsized":"2024-03-06T07:30:20.624471+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:51:58.572467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608554+0000","last_change":"2024-03-06T07:30:20.608554+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608554+0000","last_undegraded":"2024-03-06T07:30:20.608554+0000","last_fullsized":"2024-03-06T07:30:20.608554+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632061+0000","last_change":"2024-03-06T07:30:20.626055+0000","last_active":"2024-03-06T07:30:20.632061+0000","last_peered":"2024-03-06T07:30:20.632061+0000","last_clean":"2024-03-06T07:30:20.632061+0000","last_became_active":"2024-03-06T07:30:20.625631+0000","last_became_peered":"2024-03-06T07:30:20.625631+0000","last_unstale":"2024-03-06T07:30:20.632061+0000","last_undegraded":"2024-03-06T07:30:20.632061+0000","last_fullsized":"2024-03-06T07:30:20.632061+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:23.173745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.604785+0000","last_change":"2024-03-06T07:30:20.604785+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.604785+0000","last_undegraded":"2024-03-06T07:30:20.604785+0000","last_fullsized":"2024-03-06T07:30:20.604785+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632549+0000","last_change":"2024-03-06T07:30:20.631068+0000","last_active":"2024-03-06T07:30:20.632549+0000","last_peered":"2024-03-06T07:30:20.632549+0000","last_clean":"2024-03-06T07:30:20.632549+0000","last_became_active":"2024-03-06T07:30:20.630723+0000","last_became_peered":"2024-03-06T07:30:20.630723+0000","last_unstale":"2024-03-06T07:30:20.632549+0000","last_undegraded":"2024-03-06T07:30:20.632549+0000","last_fullsized":"2024-03-06T07:30:20.632549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:53.722583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606205+0000","last_change":"2024-03-06T07:30:20.606205+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606205+0000","last_undegraded":"2024-03-06T07:30:20.606205+0000","last_fullsized":"2024-03-06T07:30:20.606205+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632017+0000","last_change":"2024-03-06T07:30:20.630056+0000","last_active":"2024-03-06T07:30:20.632017+0000","last_peered":"2024-03-06T07:30:20.632017+0000","last_clean":"2024-03-06T07:30:20.632017+0000","last_became_active":"2024-03-06T07:30:20.629761+0000","last_became_peered":"2024-03-06T07:30:20.629761+0000","last_unstale":"2024-03-06T07:30:20.632017+0000","last_undegraded":"2024-03-06T07:30:20.632017+0000","last_fullsized":"2024-03-06T07:30:20.632017+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:15.540003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605142+0000","last_change":"2024-03-06T07:30:20.605142+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605142+0000","last_undegraded":"2024-03-06T07:30:20.605142+0000","last_fullsized":"2024-03-06T07:30:20.605142+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641952+0000","last_change":"2024-03-06T07:30:20.618404+0000","last_active":"2024-03-06T07:30:21.641952+0000","last_peered":"2024-03-06T07:30:21.641952+0000","last_clean":"2024-03-06T07:30:21.641952+0000","last_became_active":"2024-03-06T07:30:20.617575+0000","last_became_peered":"2024-03-06T07:30:20.617575+0000","last_unstale":"2024-03-06T07:30:21.641952+0000","last_undegraded":"2024-03-06T07:30:21.641952+0000","last_fullsized":"2024-03-06T07:30:21.641952+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:03.599842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.605059+0000","last_change":"2024-03-06T07:30:20.605059+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.605059+0000","last_undegraded":"2024-03-06T07:30:20.605059+0000","last_fullsized":"2024-03-06T07:30:20.605059+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634627+0000","last_change":"2024-03-06T07:30:20.633615+0000","last_active":"2024-03-06T07:30:20.634627+0000","last_peered":"2024-03-06T07:30:20.634627+0000","last_clean":"2024-03-06T07:30:20.634627+0000","last_became_active":"2024-03-06T07:30:20.633225+0000","last_became_peered":"2024-03-06T07:30:20.633225+0000","last_unstale":"2024-03-06T07:30:20.634627+0000","last_undegraded":"2024-03-06T07:30:20.634627+0000","last_fullsized":"2024-03-06T07:30:20.634627+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:15.085610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.606321+0000","last_change":"2024-03-06T07:30:20.606321+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.606321+0000","last_undegraded":"2024-03-06T07:30:20.606321+0000","last_fullsized":"2024-03-06T07:30:20.606321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.674444+0000","last_change":"2024-03-06T07:30:20.624699+0000","last_active":"2024-03-06T07:30:21.674444+0000","last_peered":"2024-03-06T07:30:21.674444+0000","last_clean":"2024-03-06T07:30:21.674444+0000","last_became_active":"2024-03-06T07:30:20.622946+0000","last_became_peered":"2024-03-06T07:30:20.622946+0000","last_unstale":"2024-03-06T07:30:21.674444+0000","last_undegraded":"2024-03-06T07:30:21.674444+0000","last_fullsized":"2024-03-06T07:30:21.674444+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:50.448937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.608839+0000","last_change":"2024-03-06T07:30:20.608839+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.608839+0000","last_undegraded":"2024-03-06T07:30:20.608839+0000","last_fullsized":"2024-03-06T07:30:20.608839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.631221+0000","last_change":"2024-03-06T07:30:20.629443+0000","last_active":"2024-03-06T07:30:20.631221+0000","last_peered":"2024-03-06T07:30:20.631221+0000","last_clean":"2024-03-06T07:30:20.631221+0000","last_became_active":"2024-03-06T07:30:20.629144+0000","last_became_peered":"2024-03-06T07:30:20.629144+0000","last_unstale":"2024-03-06T07:30:20.631221+0000","last_undegraded":"2024-03-06T07:30:20.631221+0000","last_fullsized":"2024-03-06T07:30:20.631221+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:37.100598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.665902+0000","last_change":"2024-03-06T07:30:21.650700+0000","last_active":"2024-03-06T07:30:21.665902+0000","last_peered":"2024-03-06T07:30:21.665902+0000","last_clean":"2024-03-06T07:30:21.665902+0000","last_became_active":"2024-03-06T07:30:21.650363+0000","last_became_peered":"2024-03-06T07:30:21.650363+0000","last_unstale":"2024-03-06T07:30:21.665902+0000","last_undegraded":"2024-03-06T07:30:21.665902+0000","last_fullsized":"2024-03-06T07:30:21.665902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:20:03.131576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.632815+0000","last_change":"2024-03-06T07:30:20.629663+0000","last_active":"2024-03-06T07:30:20.632815+0000","last_peered":"2024-03-06T07:30:20.632815+0000","last_clean":"2024-03-06T07:30:20.632815+0000","last_became_active":"2024-03-06T07:30:20.629365+0000","last_became_peered":"2024-03-06T07:30:20.629365+0000","last_unstale":"2024-03-06T07:30:20.632815+0000","last_undegraded":"2024-03-06T07:30:20.632815+0000","last_fullsized":"2024-03-06T07:30:20.632815+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:20.066237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.612360+0000","last_change":"2024-03-06T07:30:20.612360+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.612360+0000","last_undegraded":"2024-03-06T07:30:20.612360+0000","last_fullsized":"2024-03-06T07:30:20.612360+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.633860+0000","last_change":"2024-03-06T07:30:20.631439+0000","last_active":"2024-03-06T07:30:20.633860+0000","last_peered":"2024-03-06T07:30:20.633860+0000","last_clean":"2024-03-06T07:30:20.633860+0000","last_became_active":"2024-03-06T07:30:20.631100+0000","last_became_peered":"2024-03-06T07:30:20.631100+0000","last_unstale":"2024-03-06T07:30:20.633860+0000","last_undegraded":"2024-03-06T07:30:20.633860+0000","last_fullsized":"2024-03-06T07:30:20.633860+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:04:37.100598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.610052+0000","last_change":"2024-03-06T07:30:20.610052+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.610052+0000","last_undegraded":"2024-03-06T07:30:20.610052+0000","last_fullsized":"2024-03-06T07:30:20.610052+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641987+0000","last_change":"2024-03-06T07:30:20.623625+0000","last_active":"2024-03-06T07:30:21.641987+0000","last_peered":"2024-03-06T07:30:21.641987+0000","last_clean":"2024-03-06T07:30:21.641987+0000","last_became_active":"2024-03-06T07:30:20.623355+0000","last_became_peered":"2024-03-06T07:30:20.623355+0000","last_unstale":"2024-03-06T07:30:21.641987+0000","last_undegraded":"2024-03-06T07:30:21.641987+0000","last_fullsized":"2024-03-06T07:30:21.641987+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:46.004625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.601988+0000","last_change":"2024-03-06T07:30:20.601988+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.601988+0000","last_undegraded":"2024-03-06T07:30:20.601988+0000","last_fullsized":"2024-03-06T07:30:20.601988+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.634005+0000","last_change":"2024-03-06T07:30:20.632460+0000","last_active":"2024-03-06T07:30:20.634005+0000","last_peered":"2024-03-06T07:30:20.634005+0000","last_clean":"2024-03-06T07:30:20.634005+0000","last_became_active":"2024-03-06T07:30:20.630675+0000","last_became_peered":"2024-03-06T07:30:20.630675+0000","last_unstale":"2024-03-06T07:30:20.634005+0000","last_undegraded":"2024-03-06T07:30:20.634005+0000","last_fullsized":"2024-03-06T07:30:20.634005+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:18.735168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.670594+0000","last_change":"2024-03-06T07:30:21.661218+0000","last_active":"2024-03-06T07:30:21.670594+0000","last_peered":"2024-03-06T07:30:21.670594+0000","last_clean":"2024-03-06T07:30:21.670594+0000","last_became_active":"2024-03-06T07:30:21.660549+0000","last_became_peered":"2024-03-06T07:30:21.660549+0000","last_unstale":"2024-03-06T07:30:21.670594+0000","last_undegraded":"2024-03-06T07:30:21.670594+0000","last_fullsized":"2024-03-06T07:30:21.670594+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:48.433580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-06T07:30:21.641620+0000","last_change":"2024-03-06T07:30:20.609520+0000","last_active":"2024-03-06T07:30:21.641620+0000","last_peered":"2024-03-06T07:30:21.641620+0000","last_clean":"2024-03-06T07:30:21.641620+0000","last_became_active":"2024-03-06T07:30:20.608944+0000","last_became_peered":"2024-03-06T07:30:20.608944+0000","last_unstale":"2024-03-06T07:30:21.641620+0000","last_undegraded":"2024-03-06T07:30:21.641620+0000","last_fullsized":"2024-03-06T07:30:21.641620+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:54.924862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.614665+0000","last_change":"2024-03-06T07:30:20.614665+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.614665+0000","last_undegraded":"2024-03-06T07:30:20.614665+0000","last_fullsized":"2024-03-06T07:30:20.614665+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.630868+0000","last_change":"2024-03-06T07:30:20.624382+0000","last_active":"2024-03-06T07:30:20.630868+0000","last_peered":"2024-03-06T07:30:20.630868+0000","last_clean":"2024-03-06T07:30:20.630868+0000","last_became_active":"2024-03-06T07:30:20.624100+0000","last_became_peered":"2024-03-06T07:30:20.624100+0000","last_unstale":"2024-03-06T07:30:20.630868+0000","last_undegraded":"2024-03-06T07:30:20.630868+0000","last_fullsized":"2024-03-06T07:30:20.630868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:22.937616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-03-06T07:30:20.607340+0000","last_change":"2024-03-06T07:30:20.607340+0000","last_active":"2024-03-06T07:30:20.586658+0000","last_peered":"2024-03-06T07:30:20.586658+0000","last_clean":"2024-03-06T07:30:20.586658+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:30:20.607340+0000","last_undegraded":"2024-03-06T07:30:20.607340+0000","last_fullsized":"2024-03-06T07:30:20.607340+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.628428+0000","last_change":"2024-03-06T07:30:20.624654+0000","last_active":"2024-03-06T07:30:20.628428+0000","last_peered":"2024-03-06T07:30:20.628428+0000","last_clean":"2024-03-06T07:30:20.628428+0000","last_became_active":"2024-03-06T07:30:20.624112+0000","last_became_peered":"2024-03-06T07:30:20.624112+0000","last_unstale":"2024-03-06T07:30:20.628428+0000","last_undegraded":"2024-03-06T07:30:20.628428+0000","last_fullsized":"2024-03-06T07:30:20.628428+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:34.375041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":232,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-06T07:30:20.614301+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:30:20.614301+0000","last_peered":"2024-03-06T07:30:20.614301+0000","last_clean":"2024-03-06T07:30:20.614301+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:30:20.614301+0000","last_undegraded":"2024-03-06T07:30:20.614301+0000","last_fullsized":"2024-03-06T07:30:20.614301+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509479,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":279263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509480,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":679936,"data_stored":279297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542183,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":862800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542184,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542183,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":663552,"data_stored":272432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542184,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32412,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339428,"statfs":{"total":96636764160,"available":96603574272,"internally_reserved":0,"allocated":1257472,"data_stored":862800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:27.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-03-06T07:30:27.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.605846+0000', 'last_change': '2024-03-06T07:30:20.605846+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.605846+0000', 'last_undegraded': '2024-03-06T07:30:20.605846+0000', 'last_fullsized': '2024-03-06T07:30:20.605846+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3d is not active+clean 2024-03-06T07:30:27.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.609643+0000', 'last_change': '2024-03-06T07:30:20.609643+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.609643+0000', 'last_undegraded': '2024-03-06T07:30:20.609643+0000', 'last_fullsized': '2024-03-06T07:30:20.609643+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.150 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3c is not active+clean 2024-03-06T07:30:27.150 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.609392+0000', 'last_change': '2024-03-06T07:30:20.609392+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.609392+0000', 'last_undegraded': '2024-03-06T07:30:20.609392+0000', 'last_fullsized': '2024-03-06T07:30:20.609392+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.605107+0000', 'last_change': '2024-03-06T07:30:20.605107+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.605107+0000', 'last_undegraded': '2024-03-06T07:30:20.605107+0000', 'last_fullsized': '2024-03-06T07:30:20.605107+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.39 is not active+clean 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.617084+0000', 'last_change': '2024-03-06T07:30:20.617084+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.617084+0000', 'last_undegraded': '2024-03-06T07:30:20.617084+0000', 'last_fullsized': '2024-03-06T07:30:20.617084+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.607822+0000', 'last_change': '2024-03-06T07:30:20.607822+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.607822+0000', 'last_undegraded': '2024-03-06T07:30:20.607822+0000', 'last_fullsized': '2024-03-06T07:30:20.607822+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.37 is not active+clean 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.621543+0000', 'last_change': '2024-03-06T07:30:20.621543+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.621543+0000', 'last_undegraded': '2024-03-06T07:30:20.621543+0000', 'last_fullsized': '2024-03-06T07:30:20.621543+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.34 is not active+clean 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.606399+0000', 'last_change': '2024-03-06T07:30:20.606399+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.606399+0000', 'last_undegraded': '2024-03-06T07:30:20.606399+0000', 'last_fullsized': '2024-03-06T07:30:20.606399+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.33 is not active+clean 2024-03-06T07:30:27.151 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.614576+0000', 'last_change': '2024-03-06T07:30:20.614576+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.614576+0000', 'last_undegraded': '2024-03-06T07:30:20.614576+0000', 'last_fullsized': '2024-03-06T07:30:20.614576+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.32 is not active+clean 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.620181+0000', 'last_change': '2024-03-06T07:30:20.620181+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.620181+0000', 'last_undegraded': '2024-03-06T07:30:20.620181+0000', 'last_fullsized': '2024-03-06T07:30:20.620181+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.31 is not active+clean 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.616435+0000', 'last_change': '2024-03-06T07:30:20.616435+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.616435+0000', 'last_undegraded': '2024-03-06T07:30:20.616435+0000', 'last_fullsized': '2024-03-06T07:30:20.616435+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.604507+0000', 'last_change': '2024-03-06T07:30:20.604507+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.604507+0000', 'last_undegraded': '2024-03-06T07:30:20.604507+0000', 'last_fullsized': '2024-03-06T07:30:20.604507+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.607582+0000', 'last_change': '2024-03-06T07:30:20.607582+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.607582+0000', 'last_undegraded': '2024-03-06T07:30:20.607582+0000', 'last_fullsized': '2024-03-06T07:30:20.607582+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2e is not active+clean 2024-03-06T07:30:27.152 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.606705+0000', 'last_change': '2024-03-06T07:30:20.606705+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.606705+0000', 'last_undegraded': '2024-03-06T07:30:20.606705+0000', 'last_fullsized': '2024-03-06T07:30:20.606705+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.608995+0000', 'last_change': '2024-03-06T07:30:20.608995+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.608995+0000', 'last_undegraded': '2024-03-06T07:30:20.608995+0000', 'last_fullsized': '2024-03-06T07:30:20.608995+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2b is not active+clean 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.612095+0000', 'last_change': '2024-03-06T07:30:20.612095+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.612095+0000', 'last_undegraded': '2024-03-06T07:30:20.612095+0000', 'last_fullsized': '2024-03-06T07:30:20.612095+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.612050+0000', 'last_change': '2024-03-06T07:30:20.612050+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.612050+0000', 'last_undegraded': '2024-03-06T07:30:20.612050+0000', 'last_fullsized': '2024-03-06T07:30:20.612050+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.29 is not active+clean 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.603712+0000', 'last_change': '2024-03-06T07:30:20.603712+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.603712+0000', 'last_undegraded': '2024-03-06T07:30:20.603712+0000', 'last_fullsized': '2024-03-06T07:30:20.603712+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.28 is not active+clean 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.618309+0000', 'last_change': '2024-03-06T07:30:20.618309+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.618309+0000', 'last_undegraded': '2024-03-06T07:30:20.618309+0000', 'last_fullsized': '2024-03-06T07:30:20.618309+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.26 is not active+clean 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.610660+0000', 'last_change': '2024-03-06T07:30:20.610660+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.610660+0000', 'last_undegraded': '2024-03-06T07:30:20.610660+0000', 'last_fullsized': '2024-03-06T07:30:20.610660+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.153 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.613593+0000', 'last_change': '2024-03-06T07:30:20.613593+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.613593+0000', 'last_undegraded': '2024-03-06T07:30:20.613593+0000', 'last_fullsized': '2024-03-06T07:30:20.613593+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.616277+0000', 'last_change': '2024-03-06T07:30:20.616277+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.616277+0000', 'last_undegraded': '2024-03-06T07:30:20.616277+0000', 'last_fullsized': '2024-03-06T07:30:20.616277+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.22 is not active+clean 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.601987+0000', 'last_change': '2024-03-06T07:30:20.601987+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.601987+0000', 'last_undegraded': '2024-03-06T07:30:20.601987+0000', 'last_fullsized': '2024-03-06T07:30:20.601987+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.603960+0000', 'last_change': '2024-03-06T07:30:20.603960+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.603960+0000', 'last_undegraded': '2024-03-06T07:30:20.603960+0000', 'last_fullsized': '2024-03-06T07:30:20.603960+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.606790+0000', 'last_change': '2024-03-06T07:30:20.606790+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.606790+0000', 'last_undegraded': '2024-03-06T07:30:20.606790+0000', 'last_fullsized': '2024-03-06T07:30:20.606790+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.606274+0000', 'last_change': '2024-03-06T07:30:20.606274+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.606274+0000', 'last_undegraded': '2024-03-06T07:30:20.606274+0000', 'last_fullsized': '2024-03-06T07:30:20.606274+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.613038+0000', 'last_change': '2024-03-06T07:30:20.613038+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.613038+0000', 'last_undegraded': '2024-03-06T07:30:20.613038+0000', 'last_fullsized': '2024-03-06T07:30:20.613038+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.611537+0000', 'last_change': '2024-03-06T07:30:20.611537+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.611537+0000', 'last_undegraded': '2024-03-06T07:30:20.611537+0000', 'last_fullsized': '2024-03-06T07:30:20.611537+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.618124+0000', 'last_change': '2024-03-06T07:30:20.618124+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.618124+0000', 'last_undegraded': '2024-03-06T07:30:20.618124+0000', 'last_fullsized': '2024-03-06T07:30:20.618124+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.614522+0000', 'last_change': '2024-03-06T07:30:20.614522+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.614522+0000', 'last_undegraded': '2024-03-06T07:30:20.614522+0000', 'last_fullsized': '2024-03-06T07:30:20.614522+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.617560+0000', 'last_change': '2024-03-06T07:30:20.617560+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.617560+0000', 'last_undegraded': '2024-03-06T07:30:20.617560+0000', 'last_fullsized': '2024-03-06T07:30:20.617560+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-03-06T07:30:27.155 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.621452+0000', 'last_change': '2024-03-06T07:30:20.621452+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.621452+0000', 'last_undegraded': '2024-03-06T07:30:20.621452+0000', 'last_fullsized': '2024-03-06T07:30:20.621452+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.605345+0000', 'last_change': '2024-03-06T07:30:20.605345+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.605345+0000', 'last_undegraded': '2024-03-06T07:30:20.605345+0000', 'last_fullsized': '2024-03-06T07:30:20.605345+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.608641+0000', 'last_change': '2024-03-06T07:30:20.608641+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.608641+0000', 'last_undegraded': '2024-03-06T07:30:20.608641+0000', 'last_fullsized': '2024-03-06T07:30:20.608641+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.607488+0000', 'last_change': '2024-03-06T07:30:20.607488+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.607488+0000', 'last_undegraded': '2024-03-06T07:30:20.607488+0000', 'last_fullsized': '2024-03-06T07:30:20.607488+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.610618+0000', 'last_change': '2024-03-06T07:30:20.610618+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.610618+0000', 'last_undegraded': '2024-03-06T07:30:20.610618+0000', 'last_fullsized': '2024-03-06T07:30:20.610618+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-03-06T07:30:27.156 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.617892+0000', 'last_change': '2024-03-06T07:30:20.617892+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.617892+0000', 'last_undegraded': '2024-03-06T07:30:20.617892+0000', 'last_fullsized': '2024-03-06T07:30:20.617892+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.602149+0000', 'last_change': '2024-03-06T07:30:20.602149+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.602149+0000', 'last_undegraded': '2024-03-06T07:30:20.602149+0000', 'last_fullsized': '2024-03-06T07:30:20.602149+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.605234+0000', 'last_change': '2024-03-06T07:30:20.605234+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.605234+0000', 'last_undegraded': '2024-03-06T07:30:20.605234+0000', 'last_fullsized': '2024-03-06T07:30:20.605234+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.610000+0000', 'last_change': '2024-03-06T07:30:20.610000+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.610000+0000', 'last_undegraded': '2024-03-06T07:30:20.610000+0000', 'last_fullsized': '2024-03-06T07:30:20.610000+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.608554+0000', 'last_change': '2024-03-06T07:30:20.608554+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.608554+0000', 'last_undegraded': '2024-03-06T07:30:20.608554+0000', 'last_fullsized': '2024-03-06T07:30:20.608554+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.604785+0000', 'last_change': '2024-03-06T07:30:20.604785+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.604785+0000', 'last_undegraded': '2024-03-06T07:30:20.604785+0000', 'last_fullsized': '2024-03-06T07:30:20.604785+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-03-06T07:30:27.157 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.606205+0000', 'last_change': '2024-03-06T07:30:20.606205+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.606205+0000', 'last_undegraded': '2024-03-06T07:30:20.606205+0000', 'last_fullsized': '2024-03-06T07:30:20.606205+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.605142+0000', 'last_change': '2024-03-06T07:30:20.605142+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.605142+0000', 'last_undegraded': '2024-03-06T07:30:20.605142+0000', 'last_fullsized': '2024-03-06T07:30:20.605142+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.605059+0000', 'last_change': '2024-03-06T07:30:20.605059+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.605059+0000', 'last_undegraded': '2024-03-06T07:30:20.605059+0000', 'last_fullsized': '2024-03-06T07:30:20.605059+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.606321+0000', 'last_change': '2024-03-06T07:30:20.606321+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.606321+0000', 'last_undegraded': '2024-03-06T07:30:20.606321+0000', 'last_fullsized': '2024-03-06T07:30:20.606321+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.608839+0000', 'last_change': '2024-03-06T07:30:20.608839+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.608839+0000', 'last_undegraded': '2024-03-06T07:30:20.608839+0000', 'last_fullsized': '2024-03-06T07:30:20.608839+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.612360+0000', 'last_change': '2024-03-06T07:30:20.612360+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.612360+0000', 'last_undegraded': '2024-03-06T07:30:20.612360+0000', 'last_fullsized': '2024-03-06T07:30:20.612360+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:30:27.158 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.610052+0000', 'last_change': '2024-03-06T07:30:20.610052+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.610052+0000', 'last_undegraded': '2024-03-06T07:30:20.610052+0000', 'last_fullsized': '2024-03-06T07:30:20.610052+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.601988+0000', 'last_change': '2024-03-06T07:30:20.601988+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.601988+0000', 'last_undegraded': '2024-03-06T07:30:20.601988+0000', 'last_fullsized': '2024-03-06T07:30:20.601988+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.614665+0000', 'last_change': '2024-03-06T07:30:20.614665+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.614665+0000', 'last_undegraded': '2024-03-06T07:30:20.614665+0000', 'last_fullsized': '2024-03-06T07:30:20.614665+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-03-06T07:30:27.159 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:30:20.607340+0000', 'last_change': '2024-03-06T07:30:20.607340+0000', 'last_active': '2024-03-06T07:30:20.586658+0000', 'last_peered': '2024-03-06T07:30:20.586658+0000', 'last_clean': '2024-03-06T07:30:20.586658+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:30:20.607340+0000', 'last_undegraded': '2024-03-06T07:30:20.607340+0000', 'last_fullsized': '2024-03-06T07:30:20.607340+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'last_clean_scrub_stamp': '2024-03-06T07:30:20.586658+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:30:27.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:30:27.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.405+0000 7f7317000640 1 Processor -- start 2024-03-06T07:30:27.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.406+0000 7f7317000640 1 -- start start 2024-03-06T07:30:27.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f7317000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f731015f310 0x7f73100cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f7317000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f73100cfdd0 0x7f73100d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f7317000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100d02d0 con 0x7f73100d2170 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f7317000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100d25e0 con 0x7f731015f310 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f7317000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100d05b0 con 0x7f73100cfdd0 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f730fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f731015f310 0x7f73100cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.408+0000 7f730fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f731015f310 0x7f73100cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34132/0 (socket says 172.21.15.6:34132) 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.409+0000 7f730fe00640 1 -- 172.21.15.6:0/4230235916 learned_addr learned my addr 172.21.15.6:0/4230235916 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.409+0000 7f730f400640 1 --2- 172.21.15.6:0/4230235916 >> v2:172.21.15.178:3300/0 conn(0x7f73100cfdd0 0x7f73100d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.409+0000 7f730fe00640 1 -- 172.21.15.6:0/4230235916 >> v2:172.21.15.178:3300/0 conn(0x7f73100cfdd0 msgr2=0x7f73100d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.409+0000 7f730fe00640 1 --2- 172.21.15.6:0/4230235916 >> v2:172.21.15.178:3300/0 conn(0x7f73100cfdd0 0x7f73100d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.409+0000 7f730fe00640 1 -- 172.21.15.6:0/4230235916 >> v1:172.21.15.178:6789/0 conn(0x7f73100d2170 legacy=0x7f73100ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.409+0000 7f730fe00640 1 -- 172.21.15.6:0/4230235916 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100d0890 con 0x7f731015f310 2024-03-06T07:30:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.411+0000 7f730fe00640 1 --2- 172.21.15.6:0/4230235916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f731015f310 0x7f73100cf890 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f73100730e0 tx=0x7f730002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ced21c854f0fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.411+0000 7f730ea00640 1 -- 172.21.15.6:0/4230235916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300034070 con 0x7f731015f310 2024-03-06T07:30:27.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.411+0000 7f730ea00640 1 -- 172.21.15.6:0/4230235916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7300030860 con 0x7f731015f310 2024-03-06T07:30:27.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.411+0000 7f730ea00640 1 -- 172.21.15.6:0/4230235916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300030c30 con 0x7f731015f310 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 -- 172.21.15.6:0/4230235916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f731015f310 msgr2=0x7f73100cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 --2- 172.21.15.6:0/4230235916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f731015f310 0x7f73100cf890 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f73100730e0 tx=0x7f730002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 -- 172.21.15.6:0/4230235916 shutdown_connections 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 --2- 172.21.15.6:0/4230235916 >> v2:172.21.15.178:3300/0 conn(0x7f73100cfdd0 0x7f73100d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 --2- 172.21.15.6:0/4230235916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f731015f310 0x7f73100cf890 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 -- 172.21.15.6:0/4230235916 >> 172.21.15.6:0/4230235916 conn(0x7f731009aa10 msgr2=0x7f731009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 -- 172.21.15.6:0/4230235916 shutdown_connections 2024-03-06T07:30:27.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.412+0000 7f7317000640 1 -- 172.21.15.6:0/4230235916 wait complete. 2024-03-06T07:30:27.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.414+0000 7f7317000640 1 Processor -- start 2024-03-06T07:30:27.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.414+0000 7f7317000640 1 -- start start 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f7317000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73100cfdd0 0x7f73101f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f7317000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 0x7f73101f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f7317000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f731015f310 0x7f73101f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f7317000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731015f6f0 con 0x7f73100d2170 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f7317000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100cc920 con 0x7f731015f310 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f7317000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731015f870 con 0x7f73100cfdd0 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f730f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 0x7f73101f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f730f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 0x7f73101f6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34134/0 (socket says 172.21.15.6:34134) 2024-03-06T07:30:27.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.415+0000 7f730f400640 1 -- 172.21.15.6:0/2106561758 learned_addr learned my addr 172.21.15.6:0/2106561758 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730fe00640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73100cfdd0 0x7f73101f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f7315000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f731015f310 0x7f73101f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730f400640 1 -- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73100cfdd0 msgr2=0x7f73101f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730f400640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73100cfdd0 0x7f73101f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730f400640 1 -- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f731015f310 msgr2=0x7f73101f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730f400640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f731015f310 0x7f73101f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730f400640 1 -- 172.21.15.6:0/2106561758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73101564e0 con 0x7f73100d2170 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730fe00640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73100cfdd0 0x7f73101f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:27.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f730f400640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 0x7f73101f6d60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00d800 tx=0x7f72fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00b6c0 con 0x7f73100d2170 2024-03-06T07:30:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc019070 con 0x7f73100d2170 2024-03-06T07:30:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.416+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00ba80 con 0x7f73100d2170 2024-03-06T07:30:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.417+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73100c8200 con 0x7f73100d2170 2024-03-06T07:30:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.417+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731015cb30 con 0x7f73100d2170 2024-03-06T07:30:27.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.418+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731004f3a0 con 0x7f73100d2170 2024-03-06T07:30:27.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.419+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72fc00bc20 con 0x7f73100d2170 2024-03-06T07:30:27.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.419+0000 7f7307e00640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72f006b4d0 0x7f72f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.420+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72fc08a6c0 con 0x7f73100d2170 2024-03-06T07:30:27.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.420+0000 7f730fe00640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72f006b4d0 0x7f72f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.421+0000 7f730fe00640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72f006b4d0 0x7f72f006d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7300036000 tx=0x7f730003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.424+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72fc059530 con 0x7f73100d2170 2024-03-06T07:30:27.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.741+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f731015eb30 con 0x7f73100d2170 2024-03-06T07:30:27.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.741+0000 7f7307e00640 1 -- 172.21.15.6:0/2106561758 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1174 (secure 0 0 0) 0x7f72fc058e30 con 0x7f73100d2170 2024-03-06T07:30:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:27.744 INFO:teuthology.orchestra.run.smithi006.stdout:{"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1709710036,"num_in_osds":8,"osd_in_since":1709710028,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":270725120,"bytes_avail":772823388160,"bytes_total":773094113280,"write_bytes_sec":2076,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":36,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":5955}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-06T07:28:27.458008+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:30:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72f006b4d0 msgr2=0x7f72f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72f006b4d0 0x7f72f006d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7300036000 tx=0x7f730003b040 comp rx=0 tx=0).stop 2024-03-06T07:30:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 msgr2=0x7f73101f6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 0x7f73101f6d60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00d800 tx=0x7f72fc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 shutdown_connections 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72f006b4d0 0x7f72f006d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f731015f310 0x7f73101f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f73100d2170 0x7f73101f6d60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 --2- 172.21.15.6:0/2106561758 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f73100cfdd0 0x7f73101f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 >> 172.21.15.6:0/2106561758 conn(0x7f731009aa10 msgr2=0x7f73100b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 shutdown_connections 2024-03-06T07:30:27.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.744+0000 7f7317000640 1 -- 172.21.15.6:0/2106561758 wait complete. 2024-03-06T07:30:27.765 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:30:27.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:27.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.982+0000 7fb416600640 1 Processor -- start 2024-03-06T07:30:27.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.982+0000 7fb416600640 1 -- start start 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb416600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4100add50 0x7fb4100d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb416600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb41006afb0 0x7fb41006b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb416600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4100b5de0 con 0x7fb410145790 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb416600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4100d1b70 con 0x7fb41006afb0 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb416600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41009ef10 con 0x7fb4100add50 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb40fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb41006afb0 0x7fb41006b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb40fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb41006afb0 0x7fb41006b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34136/0 (socket says 172.21.15.6:34136) 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.984+0000 7fb40fe00640 1 -- 172.21.15.6:0/1624706972 learned_addr learned my addr 172.21.15.6:0/1624706972 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.985+0000 7fb40ea00640 1 --2- 172.21.15.6:0/1624706972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4100add50 0x7fb4100d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.985+0000 7fb40fe00640 1 -- 172.21.15.6:0/1624706972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4100add50 msgr2=0x7fb4100d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.985+0000 7fb40fe00640 1 --2- 172.21.15.6:0/1624706972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4100add50 0x7fb4100d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.985+0000 7fb40fe00640 1 -- 172.21.15.6:0/1624706972 >> v1:172.21.15.178:6789/0 conn(0x7fb410145790 legacy=0x7fb4101603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:27.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.985+0000 7fb40fe00640 1 -- 172.21.15.6:0/1624706972 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4100b6080 con 0x7fb41006afb0 2024-03-06T07:30:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.986+0000 7fb40fe00640 1 --2- 172.21.15.6:0/1624706972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb41006afb0 0x7fb41006b440 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb408009e50 tx=0x7fb40802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cd09212ef4956d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.986+0000 7fb40e000640 1 -- 172.21.15.6:0/1624706972 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40802ca70 con 0x7fb41006afb0 2024-03-06T07:30:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.986+0000 7fb40e000640 1 -- 172.21.15.6:0/1624706972 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40802cc10 con 0x7fb41006afb0 2024-03-06T07:30:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.986+0000 7fb40e000640 1 -- 172.21.15.6:0/1624706972 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4080343f0 con 0x7fb41006afb0 2024-03-06T07:30:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 -- 172.21.15.6:0/1624706972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb41006afb0 msgr2=0x7fb41006b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 --2- 172.21.15.6:0/1624706972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb41006afb0 0x7fb41006b440 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb408009e50 tx=0x7fb40802bed0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 -- 172.21.15.6:0/1624706972 shutdown_connections 2024-03-06T07:30:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 --2- 172.21.15.6:0/1624706972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb41006afb0 0x7fb41006b440 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 --2- 172.21.15.6:0/1624706972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb4100add50 0x7fb4100d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 -- 172.21.15.6:0/1624706972 >> 172.21.15.6:0/1624706972 conn(0x7fb41006c2a0 msgr2=0x7fb41006c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 -- 172.21.15.6:0/1624706972 shutdown_connections 2024-03-06T07:30:27.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.987+0000 7fb416600640 1 -- 172.21.15.6:0/1624706972 wait complete. 2024-03-06T07:30:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.989+0000 7fb416600640 1 Processor -- start 2024-03-06T07:30:27.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.989+0000 7fb416600640 1 -- start start 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb416600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb41006afb0 0x7fb41013ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb416600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 0x7fb41015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb416600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb410145790 0x7fb410141580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb416600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb410145b70 con 0x7fb4100add50 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb416600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41015fed0 con 0x7fb410145790 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb416600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100af720 con 0x7fb41006afb0 2024-03-06T07:30:27.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.990+0000 7fb40ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 0x7fb41015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 0x7fb41015b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34146/0 (socket says 172.21.15.6:34146) 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 -- 172.21.15.6:0/585297144 learned_addr learned my addr 172.21.15.6:0/585297144 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40f400640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb41006afb0 0x7fb41013ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40fe00640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb410145790 0x7fb410141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 -- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb41006afb0 msgr2=0x7fb41013ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb41006afb0 0x7fb41013ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 -- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb410145790 msgr2=0x7fb410141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb410145790 0x7fb410141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 -- 172.21.15.6:0/585297144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41013fe70 con 0x7fb4100add50 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.991+0000 7fb40ea00640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 0x7fb41015b2e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00d830 tx=0x7fb3fc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.992+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc00b6c0 con 0x7fb4100add50 2024-03-06T07:30:27.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.992+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3fc019070 con 0x7fb4100add50 2024-03-06T07:30:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.992+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc00ba80 con 0x7fb4100add50 2024-03-06T07:30:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.992+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb410140150 con 0x7fb4100add50 2024-03-06T07:30:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.992+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb410140710 con 0x7fb4100add50 2024-03-06T07:30:27.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.993+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41004f3a0 con 0x7fb4100add50 2024-03-06T07:30:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.995+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3fc01e460 con 0x7fb4100add50 2024-03-06T07:30:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.996+0000 7fb407400640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb3f006b520 0x7fb3f006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.996+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3fc08a7f0 con 0x7fb4100add50 2024-03-06T07:30:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.997+0000 7fb40f400640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb3f006b520 0x7fb3f006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:27.998+0000 7fb40f400640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb3f006b520 0x7fb3f006d9e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb400006fd0 tx=0x7fb400008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.001+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3fc059750 con 0x7fb4100add50 2024-03-06T07:30:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.217+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb41015bee0 con 0x7fb3f006b520 2024-03-06T07:30:28.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.236+0000 7fb407400640 1 -- 172.21.15.6:0/585297144 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251955 (secure 0 0 0) 0x7fb41015bee0 con 0x7fb3f006b520 2024-03-06T07:30:28.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:28.243 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb3f006b520 msgr2=0x7fb3f006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb3f006b520 0x7fb3f006d9e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb400006fd0 tx=0x7fb400008040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 msgr2=0x7fb41015b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 0x7fb41015b2e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00d830 tx=0x7fb3fc00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 shutdown_connections 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb3f006b520 0x7fb3f006d9e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb410145790 0x7fb410141580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb4100add50 0x7fb41015b2e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 --2- 172.21.15.6:0/585297144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb41006afb0 0x7fb41013ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 >> 172.21.15.6:0/585297144 conn(0x7fb41006c2a0 msgr2=0x7fb410144810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.241+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 shutdown_connections 2024-03-06T07:30:28.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:28.242+0000 7fb416600640 1 -- 172.21.15.6:0/585297144 wait complete. 2024-03-06T07:30:28.264 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-03-06T07:30:27.500479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264380,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754710340,"statfs":{"total":773094113280,"available":772823388160,"internally_reserved":0,"allocated":6758400,"data_stored":3469612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.904741"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650954+0000","last_change":"2024-03-06T07:30:21.639251+0000","last_active":"2024-03-06T07:30:22.650954+0000","last_peered":"2024-03-06T07:30:22.650954+0000","last_clean":"2024-03-06T07:30:22.650954+0000","last_became_active":"2024-03-06T07:30:21.638950+0000","last_became_peered":"2024-03-06T07:30:21.638950+0000","last_unstale":"2024-03-06T07:30:22.650954+0000","last_undegraded":"2024-03-06T07:30:22.650954+0000","last_fullsized":"2024-03-06T07:30:22.650954+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:46.747416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649085+0000","last_change":"2024-03-06T07:30:21.645700+0000","last_active":"2024-03-06T07:30:22.649085+0000","last_peered":"2024-03-06T07:30:22.649085+0000","last_clean":"2024-03-06T07:30:22.649085+0000","last_became_active":"2024-03-06T07:30:21.645396+0000","last_became_peered":"2024-03-06T07:30:21.645396+0000","last_unstale":"2024-03-06T07:30:22.649085+0000","last_undegraded":"2024-03-06T07:30:22.649085+0000","last_fullsized":"2024-03-06T07:30:22.649085+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:46.096729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874028+0000","last_change":"2024-03-06T07:30:21.664820+0000","last_active":"2024-03-06T07:30:22.874028+0000","last_peered":"2024-03-06T07:30:22.874028+0000","last_clean":"2024-03-06T07:30:22.874028+0000","last_became_active":"2024-03-06T07:30:21.659740+0000","last_became_peered":"2024-03-06T07:30:21.659740+0000","last_unstale":"2024-03-06T07:30:22.874028+0000","last_undegraded":"2024-03-06T07:30:22.874028+0000","last_fullsized":"2024-03-06T07:30:22.874028+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:47.037204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871307+0000","last_change":"2024-03-06T07:30:21.671963+0000","last_active":"2024-03-06T07:30:22.871307+0000","last_peered":"2024-03-06T07:30:22.871307+0000","last_clean":"2024-03-06T07:30:22.871307+0000","last_became_active":"2024-03-06T07:30:21.671541+0000","last_became_peered":"2024-03-06T07:30:21.671541+0000","last_unstale":"2024-03-06T07:30:22.871307+0000","last_undegraded":"2024-03-06T07:30:22.871307+0000","last_fullsized":"2024-03-06T07:30:22.871307+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.551735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650630+0000","last_change":"2024-03-06T07:30:21.671393+0000","last_active":"2024-03-06T07:30:22.650630+0000","last_peered":"2024-03-06T07:30:22.650630+0000","last_clean":"2024-03-06T07:30:22.650630+0000","last_became_active":"2024-03-06T07:30:21.671107+0000","last_became_peered":"2024-03-06T07:30:21.671107+0000","last_unstale":"2024-03-06T07:30:22.650630+0000","last_undegraded":"2024-03-06T07:30:22.650630+0000","last_fullsized":"2024-03-06T07:30:22.650630+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:15.531515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653562+0000","last_change":"2024-03-06T07:30:21.639455+0000","last_active":"2024-03-06T07:30:22.653562+0000","last_peered":"2024-03-06T07:30:22.653562+0000","last_clean":"2024-03-06T07:30:22.653562+0000","last_became_active":"2024-03-06T07:30:21.639169+0000","last_became_peered":"2024-03-06T07:30:21.639169+0000","last_unstale":"2024-03-06T07:30:22.653562+0000","last_undegraded":"2024-03-06T07:30:22.653562+0000","last_fullsized":"2024-03-06T07:30:22.653562+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:45.412635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868908+0000","last_change":"2024-03-06T07:30:21.664479+0000","last_active":"2024-03-06T07:30:22.868908+0000","last_peered":"2024-03-06T07:30:22.868908+0000","last_clean":"2024-03-06T07:30:22.868908+0000","last_became_active":"2024-03-06T07:30:21.664187+0000","last_became_peered":"2024-03-06T07:30:21.664187+0000","last_unstale":"2024-03-06T07:30:22.868908+0000","last_undegraded":"2024-03-06T07:30:22.868908+0000","last_fullsized":"2024-03-06T07:30:22.868908+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:46.693347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648518+0000","last_change":"2024-03-06T07:30:21.658244+0000","last_active":"2024-03-06T07:30:22.648518+0000","last_peered":"2024-03-06T07:30:22.648518+0000","last_clean":"2024-03-06T07:30:22.648518+0000","last_became_active":"2024-03-06T07:30:21.657945+0000","last_became_peered":"2024-03-06T07:30:21.657945+0000","last_unstale":"2024-03-06T07:30:22.648518+0000","last_undegraded":"2024-03-06T07:30:22.648518+0000","last_fullsized":"2024-03-06T07:30:22.648518+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:22.749385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.870079+0000","last_change":"2024-03-06T07:30:21.648774+0000","last_active":"2024-03-06T07:30:22.870079+0000","last_peered":"2024-03-06T07:30:22.870079+0000","last_clean":"2024-03-06T07:30:22.870079+0000","last_became_active":"2024-03-06T07:30:21.648225+0000","last_became_peered":"2024-03-06T07:30:21.648225+0000","last_unstale":"2024-03-06T07:30:22.870079+0000","last_undegraded":"2024-03-06T07:30:22.870079+0000","last_fullsized":"2024-03-06T07:30:22.870079+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:35.687120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646319+0000","last_change":"2024-03-06T07:30:21.658602+0000","last_active":"2024-03-06T07:30:22.646319+0000","last_peered":"2024-03-06T07:30:22.646319+0000","last_clean":"2024-03-06T07:30:22.646319+0000","last_became_active":"2024-03-06T07:30:21.656608+0000","last_became_peered":"2024-03-06T07:30:21.656608+0000","last_unstale":"2024-03-06T07:30:22.646319+0000","last_undegraded":"2024-03-06T07:30:22.646319+0000","last_fullsized":"2024-03-06T07:30:22.646319+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:01.731712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646507+0000","last_change":"2024-03-06T07:30:21.656749+0000","last_active":"2024-03-06T07:30:22.646507+0000","last_peered":"2024-03-06T07:30:22.646507+0000","last_clean":"2024-03-06T07:30:22.646507+0000","last_became_active":"2024-03-06T07:30:21.656479+0000","last_became_peered":"2024-03-06T07:30:21.656479+0000","last_unstale":"2024-03-06T07:30:22.646507+0000","last_undegraded":"2024-03-06T07:30:22.646507+0000","last_fullsized":"2024-03-06T07:30:22.646507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:17.732383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871659+0000","last_change":"2024-03-06T07:30:21.671007+0000","last_active":"2024-03-06T07:30:22.871659+0000","last_peered":"2024-03-06T07:30:22.871659+0000","last_clean":"2024-03-06T07:30:22.871659+0000","last_became_active":"2024-03-06T07:30:21.670719+0000","last_became_peered":"2024-03-06T07:30:21.670719+0000","last_unstale":"2024-03-06T07:30:22.871659+0000","last_undegraded":"2024-03-06T07:30:22.871659+0000","last_fullsized":"2024-03-06T07:30:22.871659+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:07.555539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660452+0000","last_change":"2024-03-06T07:30:21.648530+0000","last_active":"2024-03-06T07:30:22.660452+0000","last_peered":"2024-03-06T07:30:22.660452+0000","last_clean":"2024-03-06T07:30:22.660452+0000","last_became_active":"2024-03-06T07:30:21.648138+0000","last_became_peered":"2024-03-06T07:30:21.648138+0000","last_unstale":"2024-03-06T07:30:22.660452+0000","last_undegraded":"2024-03-06T07:30:22.660452+0000","last_fullsized":"2024-03-06T07:30:22.660452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:24.185476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869469+0000","last_change":"2024-03-06T07:30:21.652171+0000","last_active":"2024-03-06T07:30:22.869469+0000","last_peered":"2024-03-06T07:30:22.869469+0000","last_clean":"2024-03-06T07:30:22.869469+0000","last_became_active":"2024-03-06T07:30:21.651210+0000","last_became_peered":"2024-03-06T07:30:21.651210+0000","last_unstale":"2024-03-06T07:30:22.869469+0000","last_undegraded":"2024-03-06T07:30:22.869469+0000","last_fullsized":"2024-03-06T07:30:22.869469+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:29.003761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868924+0000","last_change":"2024-03-06T07:30:21.653860+0000","last_active":"2024-03-06T07:30:22.868924+0000","last_peered":"2024-03-06T07:30:22.868924+0000","last_clean":"2024-03-06T07:30:22.868924+0000","last_became_active":"2024-03-06T07:30:21.651008+0000","last_became_peered":"2024-03-06T07:30:21.651008+0000","last_unstale":"2024-03-06T07:30:22.868924+0000","last_undegraded":"2024-03-06T07:30:22.868924+0000","last_fullsized":"2024-03-06T07:30:22.868924+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:44.073218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653435+0000","last_change":"2024-03-06T07:30:21.657282+0000","last_active":"2024-03-06T07:30:22.653435+0000","last_peered":"2024-03-06T07:30:22.653435+0000","last_clean":"2024-03-06T07:30:22.653435+0000","last_became_active":"2024-03-06T07:30:21.654676+0000","last_became_peered":"2024-03-06T07:30:21.654676+0000","last_unstale":"2024-03-06T07:30:22.653435+0000","last_undegraded":"2024-03-06T07:30:22.653435+0000","last_fullsized":"2024-03-06T07:30:22.653435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:51.556506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651187+0000","last_change":"2024-03-06T07:30:21.654626+0000","last_active":"2024-03-06T07:30:22.651187+0000","last_peered":"2024-03-06T07:30:22.651187+0000","last_clean":"2024-03-06T07:30:22.651187+0000","last_became_active":"2024-03-06T07:30:21.654293+0000","last_became_peered":"2024-03-06T07:30:21.654293+0000","last_unstale":"2024-03-06T07:30:22.651187+0000","last_undegraded":"2024-03-06T07:30:22.651187+0000","last_fullsized":"2024-03-06T07:30:22.651187+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:51.388978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872868+0000","last_change":"2024-03-06T07:30:21.658100+0000","last_active":"2024-03-06T07:30:22.872868+0000","last_peered":"2024-03-06T07:30:22.872868+0000","last_clean":"2024-03-06T07:30:22.872868+0000","last_became_active":"2024-03-06T07:30:21.656111+0000","last_became_peered":"2024-03-06T07:30:21.656111+0000","last_unstale":"2024-03-06T07:30:22.872868+0000","last_undegraded":"2024-03-06T07:30:22.872868+0000","last_fullsized":"2024-03-06T07:30:22.872868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:23.468391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650808+0000","last_change":"2024-03-06T07:30:21.635273+0000","last_active":"2024-03-06T07:30:22.650808+0000","last_peered":"2024-03-06T07:30:22.650808+0000","last_clean":"2024-03-06T07:30:22.650808+0000","last_became_active":"2024-03-06T07:30:21.634938+0000","last_became_peered":"2024-03-06T07:30:21.634938+0000","last_unstale":"2024-03-06T07:30:22.650808+0000","last_undegraded":"2024-03-06T07:30:22.650808+0000","last_fullsized":"2024-03-06T07:30:22.650808+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:56.567717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649218+0000","last_change":"2024-03-06T07:30:21.640509+0000","last_active":"2024-03-06T07:30:22.649218+0000","last_peered":"2024-03-06T07:30:22.649218+0000","last_clean":"2024-03-06T07:30:22.649218+0000","last_became_active":"2024-03-06T07:30:21.640199+0000","last_became_peered":"2024-03-06T07:30:21.640199+0000","last_unstale":"2024-03-06T07:30:22.649218+0000","last_undegraded":"2024-03-06T07:30:22.649218+0000","last_fullsized":"2024-03-06T07:30:22.649218+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:00.156288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873082+0000","last_change":"2024-03-06T07:30:21.673682+0000","last_active":"2024-03-06T07:30:22.873082+0000","last_peered":"2024-03-06T07:30:22.873082+0000","last_clean":"2024-03-06T07:30:22.873082+0000","last_became_active":"2024-03-06T07:30:21.672381+0000","last_became_peered":"2024-03-06T07:30:21.672381+0000","last_unstale":"2024-03-06T07:30:22.873082+0000","last_undegraded":"2024-03-06T07:30:22.873082+0000","last_fullsized":"2024-03-06T07:30:22.873082+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:39.074474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648673+0000","last_change":"2024-03-06T07:30:21.633453+0000","last_active":"2024-03-06T07:30:22.648673+0000","last_peered":"2024-03-06T07:30:22.648673+0000","last_clean":"2024-03-06T07:30:22.648673+0000","last_became_active":"2024-03-06T07:30:21.632771+0000","last_became_peered":"2024-03-06T07:30:21.632771+0000","last_unstale":"2024-03-06T07:30:22.648673+0000","last_undegraded":"2024-03-06T07:30:22.648673+0000","last_fullsized":"2024-03-06T07:30:22.648673+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:38.963735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872321+0000","last_change":"2024-03-06T07:30:21.662230+0000","last_active":"2024-03-06T07:30:22.872321+0000","last_peered":"2024-03-06T07:30:22.872321+0000","last_clean":"2024-03-06T07:30:22.872321+0000","last_became_active":"2024-03-06T07:30:21.659240+0000","last_became_peered":"2024-03-06T07:30:21.659240+0000","last_unstale":"2024-03-06T07:30:22.872321+0000","last_undegraded":"2024-03-06T07:30:22.872321+0000","last_fullsized":"2024-03-06T07:30:22.872321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:35.687120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660249+0000","last_change":"2024-03-06T07:30:21.652868+0000","last_active":"2024-03-06T07:30:22.660249+0000","last_peered":"2024-03-06T07:30:22.660249+0000","last_clean":"2024-03-06T07:30:22.660249+0000","last_became_active":"2024-03-06T07:30:21.652069+0000","last_became_peered":"2024-03-06T07:30:21.652069+0000","last_unstale":"2024-03-06T07:30:22.660249+0000","last_undegraded":"2024-03-06T07:30:22.660249+0000","last_fullsized":"2024-03-06T07:30:22.660249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:17.298033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651046+0000","last_change":"2024-03-06T07:30:21.646940+0000","last_active":"2024-03-06T07:30:22.651046+0000","last_peered":"2024-03-06T07:30:22.651046+0000","last_clean":"2024-03-06T07:30:22.651046+0000","last_became_active":"2024-03-06T07:30:21.644768+0000","last_became_peered":"2024-03-06T07:30:21.644768+0000","last_unstale":"2024-03-06T07:30:22.651046+0000","last_undegraded":"2024-03-06T07:30:22.651046+0000","last_fullsized":"2024-03-06T07:30:22.651046+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:34.516723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871233+0000","last_change":"2024-03-06T07:30:21.662980+0000","last_active":"2024-03-06T07:30:22.871233+0000","last_peered":"2024-03-06T07:30:22.871233+0000","last_clean":"2024-03-06T07:30:22.871233+0000","last_became_active":"2024-03-06T07:30:21.662680+0000","last_became_peered":"2024-03-06T07:30:21.662680+0000","last_unstale":"2024-03-06T07:30:22.871233+0000","last_undegraded":"2024-03-06T07:30:22.871233+0000","last_fullsized":"2024-03-06T07:30:22.871233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:31.415796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653917+0000","last_change":"2024-03-06T07:30:21.654904+0000","last_active":"2024-03-06T07:30:22.653917+0000","last_peered":"2024-03-06T07:30:22.653917+0000","last_clean":"2024-03-06T07:30:22.653917+0000","last_became_active":"2024-03-06T07:30:21.653958+0000","last_became_peered":"2024-03-06T07:30:21.653958+0000","last_unstale":"2024-03-06T07:30:22.653917+0000","last_undegraded":"2024-03-06T07:30:22.653917+0000","last_fullsized":"2024-03-06T07:30:22.653917+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:54.366453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650784+0000","last_change":"2024-03-06T07:30:21.663886+0000","last_active":"2024-03-06T07:30:22.650784+0000","last_peered":"2024-03-06T07:30:22.650784+0000","last_clean":"2024-03-06T07:30:22.650784+0000","last_became_active":"2024-03-06T07:30:21.663511+0000","last_became_peered":"2024-03-06T07:30:21.663511+0000","last_unstale":"2024-03-06T07:30:22.650784+0000","last_undegraded":"2024-03-06T07:30:22.650784+0000","last_fullsized":"2024-03-06T07:30:22.650784+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:50.002026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649302+0000","last_change":"2024-03-06T07:30:21.671940+0000","last_active":"2024-03-06T07:30:22.649302+0000","last_peered":"2024-03-06T07:30:22.649302+0000","last_clean":"2024-03-06T07:30:22.649302+0000","last_became_active":"2024-03-06T07:30:21.671710+0000","last_became_peered":"2024-03-06T07:30:21.671710+0000","last_unstale":"2024-03-06T07:30:22.649302+0000","last_undegraded":"2024-03-06T07:30:22.649302+0000","last_fullsized":"2024-03-06T07:30:22.649302+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:20.419130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873042+0000","last_change":"2024-03-06T07:30:21.664779+0000","last_active":"2024-03-06T07:30:22.873042+0000","last_peered":"2024-03-06T07:30:22.873042+0000","last_clean":"2024-03-06T07:30:22.873042+0000","last_became_active":"2024-03-06T07:30:21.663664+0000","last_became_peered":"2024-03-06T07:30:21.663664+0000","last_unstale":"2024-03-06T07:30:22.873042+0000","last_undegraded":"2024-03-06T07:30:22.873042+0000","last_fullsized":"2024-03-06T07:30:22.873042+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:11.037035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650135+0000","last_change":"2024-03-06T07:30:21.645937+0000","last_active":"2024-03-06T07:30:22.650135+0000","last_peered":"2024-03-06T07:30:22.650135+0000","last_clean":"2024-03-06T07:30:22.650135+0000","last_became_active":"2024-03-06T07:30:21.644909+0000","last_became_peered":"2024-03-06T07:30:21.644909+0000","last_unstale":"2024-03-06T07:30:22.650135+0000","last_undegraded":"2024-03-06T07:30:22.650135+0000","last_fullsized":"2024-03-06T07:30:22.650135+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:42.919081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653474+0000","last_change":"2024-03-06T07:30:21.636575+0000","last_active":"2024-03-06T07:30:22.653474+0000","last_peered":"2024-03-06T07:30:22.653474+0000","last_clean":"2024-03-06T07:30:22.653474+0000","last_became_active":"2024-03-06T07:30:21.634856+0000","last_became_peered":"2024-03-06T07:30:21.634856+0000","last_unstale":"2024-03-06T07:30:22.653474+0000","last_undegraded":"2024-03-06T07:30:22.653474+0000","last_fullsized":"2024-03-06T07:30:22.653474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:07.555539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655221+0000","last_change":"2024-03-06T07:30:21.653801+0000","last_active":"2024-03-06T07:30:22.655221+0000","last_peered":"2024-03-06T07:30:22.655221+0000","last_clean":"2024-03-06T07:30:22.655221+0000","last_became_active":"2024-03-06T07:30:21.653476+0000","last_became_peered":"2024-03-06T07:30:21.653476+0000","last_unstale":"2024-03-06T07:30:22.655221+0000","last_undegraded":"2024-03-06T07:30:22.655221+0000","last_fullsized":"2024-03-06T07:30:22.655221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:35.655428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655282+0000","last_change":"2024-03-06T07:30:20.630369+0000","last_active":"2024-03-06T07:30:22.655282+0000","last_peered":"2024-03-06T07:30:22.655282+0000","last_clean":"2024-03-06T07:30:22.655282+0000","last_became_active":"2024-03-06T07:30:20.628580+0000","last_became_peered":"2024-03-06T07:30:20.628580+0000","last_unstale":"2024-03-06T07:30:22.655282+0000","last_undegraded":"2024-03-06T07:30:22.655282+0000","last_fullsized":"2024-03-06T07:30:22.655282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:01.989937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646633+0000","last_change":"2024-03-06T07:30:21.664516+0000","last_active":"2024-03-06T07:30:22.646633+0000","last_peered":"2024-03-06T07:30:22.646633+0000","last_clean":"2024-03-06T07:30:22.646633+0000","last_became_active":"2024-03-06T07:30:21.664224+0000","last_became_peered":"2024-03-06T07:30:21.664224+0000","last_unstale":"2024-03-06T07:30:22.646633+0000","last_undegraded":"2024-03-06T07:30:22.646633+0000","last_fullsized":"2024-03-06T07:30:22.646633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:05.832079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650905+0000","last_change":"2024-03-06T07:30:20.625914+0000","last_active":"2024-03-06T07:30:22.650905+0000","last_peered":"2024-03-06T07:30:22.650905+0000","last_clean":"2024-03-06T07:30:22.650905+0000","last_became_active":"2024-03-06T07:30:20.625643+0000","last_became_peered":"2024-03-06T07:30:20.625643+0000","last_unstale":"2024-03-06T07:30:22.650905+0000","last_undegraded":"2024-03-06T07:30:22.650905+0000","last_fullsized":"2024-03-06T07:30:22.650905+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:06.543808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653909+0000","last_change":"2024-03-06T07:30:21.644886+0000","last_active":"2024-03-06T07:30:22.653909+0000","last_peered":"2024-03-06T07:30:22.653909+0000","last_clean":"2024-03-06T07:30:22.653909+0000","last_became_active":"2024-03-06T07:30:21.643841+0000","last_became_peered":"2024-03-06T07:30:21.643841+0000","last_unstale":"2024-03-06T07:30:22.653909+0000","last_undegraded":"2024-03-06T07:30:22.653909+0000","last_fullsized":"2024-03-06T07:30:22.653909+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:43.678334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.654086+0000","last_change":"2024-03-06T07:30:20.617109+0000","last_active":"2024-03-06T07:30:22.654086+0000","last_peered":"2024-03-06T07:30:22.654086+0000","last_clean":"2024-03-06T07:30:22.654086+0000","last_became_active":"2024-03-06T07:30:20.615288+0000","last_became_peered":"2024-03-06T07:30:20.615288+0000","last_unstale":"2024-03-06T07:30:22.654086+0000","last_undegraded":"2024-03-06T07:30:22.654086+0000","last_fullsized":"2024-03-06T07:30:22.654086+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:24.398212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653973+0000","last_change":"2024-03-06T07:30:21.673120+0000","last_active":"2024-03-06T07:30:22.653973+0000","last_peered":"2024-03-06T07:30:22.653973+0000","last_clean":"2024-03-06T07:30:22.653973+0000","last_became_active":"2024-03-06T07:30:21.672837+0000","last_became_peered":"2024-03-06T07:30:21.672837+0000","last_unstale":"2024-03-06T07:30:22.653973+0000","last_undegraded":"2024-03-06T07:30:22.653973+0000","last_fullsized":"2024-03-06T07:30:22.653973+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:49.011453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871602+0000","last_change":"2024-03-06T07:30:20.626609+0000","last_active":"2024-03-06T07:30:22.871602+0000","last_peered":"2024-03-06T07:30:22.871602+0000","last_clean":"2024-03-06T07:30:22.871602+0000","last_became_active":"2024-03-06T07:30:20.625668+0000","last_became_peered":"2024-03-06T07:30:20.625668+0000","last_unstale":"2024-03-06T07:30:22.871602+0000","last_undegraded":"2024-03-06T07:30:22.871602+0000","last_fullsized":"2024-03-06T07:30:22.871602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:24.918578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648518+0000","last_change":"2024-03-06T07:30:21.634922+0000","last_active":"2024-03-06T07:30:22.648518+0000","last_peered":"2024-03-06T07:30:22.648518+0000","last_clean":"2024-03-06T07:30:22.648518+0000","last_became_active":"2024-03-06T07:30:21.634102+0000","last_became_peered":"2024-03-06T07:30:21.634102+0000","last_unstale":"2024-03-06T07:30:22.648518+0000","last_undegraded":"2024-03-06T07:30:22.648518+0000","last_fullsized":"2024-03-06T07:30:22.648518+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:40.537463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869351+0000","last_change":"2024-03-06T07:30:20.629926+0000","last_active":"2024-03-06T07:30:22.869351+0000","last_peered":"2024-03-06T07:30:22.869351+0000","last_clean":"2024-03-06T07:30:22.869351+0000","last_became_active":"2024-03-06T07:30:20.629470+0000","last_became_peered":"2024-03-06T07:30:20.629470+0000","last_unstale":"2024-03-06T07:30:22.869351+0000","last_undegraded":"2024-03-06T07:30:22.869351+0000","last_fullsized":"2024-03-06T07:30:22.869351+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:55.532286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649048+0000","last_change":"2024-03-06T07:30:21.640716+0000","last_active":"2024-03-06T07:30:22.649048+0000","last_peered":"2024-03-06T07:30:22.649048+0000","last_clean":"2024-03-06T07:30:22.649048+0000","last_became_active":"2024-03-06T07:30:21.640180+0000","last_became_peered":"2024-03-06T07:30:21.640180+0000","last_unstale":"2024-03-06T07:30:22.649048+0000","last_undegraded":"2024-03-06T07:30:22.649048+0000","last_fullsized":"2024-03-06T07:30:22.649048+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:09.590013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653660+0000","last_change":"2024-03-06T07:30:20.627675+0000","last_active":"2024-03-06T07:30:22.653660+0000","last_peered":"2024-03-06T07:30:22.653660+0000","last_clean":"2024-03-06T07:30:22.653660+0000","last_became_active":"2024-03-06T07:30:20.627383+0000","last_became_peered":"2024-03-06T07:30:20.627383+0000","last_unstale":"2024-03-06T07:30:22.653660+0000","last_undegraded":"2024-03-06T07:30:22.653660+0000","last_fullsized":"2024-03-06T07:30:22.653660+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:14.427677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660664+0000","last_change":"2024-03-06T07:30:21.657886+0000","last_active":"2024-03-06T07:30:22.660664+0000","last_peered":"2024-03-06T07:30:22.660664+0000","last_clean":"2024-03-06T07:30:22.660664+0000","last_became_active":"2024-03-06T07:30:21.656690+0000","last_became_peered":"2024-03-06T07:30:21.656690+0000","last_unstale":"2024-03-06T07:30:22.660664+0000","last_undegraded":"2024-03-06T07:30:22.660664+0000","last_fullsized":"2024-03-06T07:30:22.660664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:38.442564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871290+0000","last_change":"2024-03-06T07:30:20.629740+0000","last_active":"2024-03-06T07:30:22.871290+0000","last_peered":"2024-03-06T07:30:22.871290+0000","last_clean":"2024-03-06T07:30:22.871290+0000","last_became_active":"2024-03-06T07:30:20.629427+0000","last_became_peered":"2024-03-06T07:30:20.629427+0000","last_unstale":"2024-03-06T07:30:22.871290+0000","last_undegraded":"2024-03-06T07:30:22.871290+0000","last_fullsized":"2024-03-06T07:30:22.871290+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:14.427677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648584+0000","last_change":"2024-03-06T07:30:21.655791+0000","last_active":"2024-03-06T07:30:22.648584+0000","last_peered":"2024-03-06T07:30:22.648584+0000","last_clean":"2024-03-06T07:30:22.648584+0000","last_became_active":"2024-03-06T07:30:21.654685+0000","last_became_peered":"2024-03-06T07:30:21.654685+0000","last_unstale":"2024-03-06T07:30:22.648584+0000","last_undegraded":"2024-03-06T07:30:22.648584+0000","last_fullsized":"2024-03-06T07:30:22.648584+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:38.630816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868866+0000","last_change":"2024-03-06T07:30:20.634196+0000","last_active":"2024-03-06T07:30:22.868866+0000","last_peered":"2024-03-06T07:30:22.868866+0000","last_clean":"2024-03-06T07:30:22.868866+0000","last_became_active":"2024-03-06T07:30:20.633405+0000","last_became_peered":"2024-03-06T07:30:20.633405+0000","last_unstale":"2024-03-06T07:30:22.868866+0000","last_undegraded":"2024-03-06T07:30:22.868866+0000","last_fullsized":"2024-03-06T07:30:22.868866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.720652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.870474+0000","last_change":"2024-03-06T07:30:21.655531+0000","last_active":"2024-03-06T07:30:22.870474+0000","last_peered":"2024-03-06T07:30:22.870474+0000","last_clean":"2024-03-06T07:30:22.870474+0000","last_became_active":"2024-03-06T07:30:21.654818+0000","last_became_peered":"2024-03-06T07:30:21.654818+0000","last_unstale":"2024-03-06T07:30:22.870474+0000","last_undegraded":"2024-03-06T07:30:22.870474+0000","last_fullsized":"2024-03-06T07:30:22.870474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:55.049559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873858+0000","last_change":"2024-03-06T07:30:20.627513+0000","last_active":"2024-03-06T07:30:22.873858+0000","last_peered":"2024-03-06T07:30:22.873858+0000","last_clean":"2024-03-06T07:30:22.873858+0000","last_became_active":"2024-03-06T07:30:20.627220+0000","last_became_peered":"2024-03-06T07:30:20.627220+0000","last_unstale":"2024-03-06T07:30:22.873858+0000","last_undegraded":"2024-03-06T07:30:22.873858+0000","last_fullsized":"2024-03-06T07:30:22.873858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:55.532286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873699+0000","last_change":"2024-03-06T07:30:21.670188+0000","last_active":"2024-03-06T07:30:22.873699+0000","last_peered":"2024-03-06T07:30:22.873699+0000","last_clean":"2024-03-06T07:30:22.873699+0000","last_became_active":"2024-03-06T07:30:21.669962+0000","last_became_peered":"2024-03-06T07:30:21.669962+0000","last_unstale":"2024-03-06T07:30:22.873699+0000","last_undegraded":"2024-03-06T07:30:22.873699+0000","last_fullsized":"2024-03-06T07:30:22.873699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:21.619393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651018+0000","last_change":"2024-03-06T07:30:20.628649+0000","last_active":"2024-03-06T07:30:22.651018+0000","last_peered":"2024-03-06T07:30:22.651018+0000","last_clean":"2024-03-06T07:30:22.651018+0000","last_became_active":"2024-03-06T07:30:20.628065+0000","last_became_peered":"2024-03-06T07:30:20.628065+0000","last_unstale":"2024-03-06T07:30:22.651018+0000","last_undegraded":"2024-03-06T07:30:22.651018+0000","last_fullsized":"2024-03-06T07:30:22.651018+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:56.643347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646298+0000","last_change":"2024-03-06T07:30:21.652433+0000","last_active":"2024-03-06T07:30:22.646298+0000","last_peered":"2024-03-06T07:30:22.646298+0000","last_clean":"2024-03-06T07:30:22.646298+0000","last_became_active":"2024-03-06T07:30:21.651796+0000","last_became_peered":"2024-03-06T07:30:21.651796+0000","last_unstale":"2024-03-06T07:30:22.646298+0000","last_undegraded":"2024-03-06T07:30:22.646298+0000","last_fullsized":"2024-03-06T07:30:22.646298+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:19.746899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874259+0000","last_change":"2024-03-06T07:30:20.628448+0000","last_active":"2024-03-06T07:30:22.874259+0000","last_peered":"2024-03-06T07:30:22.874259+0000","last_clean":"2024-03-06T07:30:22.874259+0000","last_became_active":"2024-03-06T07:30:20.628114+0000","last_became_peered":"2024-03-06T07:30:20.628114+0000","last_unstale":"2024-03-06T07:30:22.874259+0000","last_undegraded":"2024-03-06T07:30:22.874259+0000","last_fullsized":"2024-03-06T07:30:22.874259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.720652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874204+0000","last_change":"2024-03-06T07:30:21.673522+0000","last_active":"2024-03-06T07:30:22.874204+0000","last_peered":"2024-03-06T07:30:22.874204+0000","last_clean":"2024-03-06T07:30:22.874204+0000","last_became_active":"2024-03-06T07:30:21.672490+0000","last_became_peered":"2024-03-06T07:30:21.672490+0000","last_unstale":"2024-03-06T07:30:22.874204+0000","last_undegraded":"2024-03-06T07:30:22.874204+0000","last_fullsized":"2024-03-06T07:30:22.874204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:01.932495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869896+0000","last_change":"2024-03-06T07:30:20.629645+0000","last_active":"2024-03-06T07:30:22.869896+0000","last_peered":"2024-03-06T07:30:22.869896+0000","last_clean":"2024-03-06T07:30:22.869896+0000","last_became_active":"2024-03-06T07:30:20.628235+0000","last_became_peered":"2024-03-06T07:30:20.628235+0000","last_unstale":"2024-03-06T07:30:22.869896+0000","last_undegraded":"2024-03-06T07:30:22.869896+0000","last_fullsized":"2024-03-06T07:30:22.869896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:27.152751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871611+0000","last_change":"2024-03-06T07:30:21.670083+0000","last_active":"2024-03-06T07:30:22.871611+0000","last_peered":"2024-03-06T07:30:22.871611+0000","last_clean":"2024-03-06T07:30:22.871611+0000","last_became_active":"2024-03-06T07:30:21.669474+0000","last_became_peered":"2024-03-06T07:30:21.669474+0000","last_unstale":"2024-03-06T07:30:22.871611+0000","last_undegraded":"2024-03-06T07:30:22.871611+0000","last_fullsized":"2024-03-06T07:30:22.871611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:51.556506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649148+0000","last_change":"2024-03-06T07:30:20.627658+0000","last_active":"2024-03-06T07:30:22.649148+0000","last_peered":"2024-03-06T07:30:22.649148+0000","last_clean":"2024-03-06T07:30:22.649148+0000","last_became_active":"2024-03-06T07:30:20.627370+0000","last_became_peered":"2024-03-06T07:30:20.627370+0000","last_unstale":"2024-03-06T07:30:22.649148+0000","last_undegraded":"2024-03-06T07:30:22.649148+0000","last_fullsized":"2024-03-06T07:30:22.649148+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:14.883601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873519+0000","last_change":"2024-03-06T07:30:21.654626+0000","last_active":"2024-03-06T07:30:22.873519+0000","last_peered":"2024-03-06T07:30:22.873519+0000","last_clean":"2024-03-06T07:30:22.873519+0000","last_became_active":"2024-03-06T07:30:21.654286+0000","last_became_peered":"2024-03-06T07:30:21.654286+0000","last_unstale":"2024-03-06T07:30:22.873519+0000","last_undegraded":"2024-03-06T07:30:22.873519+0000","last_fullsized":"2024-03-06T07:30:22.873519+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:41.911025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660041+0000","last_change":"2024-03-06T07:30:20.626528+0000","last_active":"2024-03-06T07:30:22.660041+0000","last_peered":"2024-03-06T07:30:22.660041+0000","last_clean":"2024-03-06T07:30:22.660041+0000","last_became_active":"2024-03-06T07:30:20.626159+0000","last_became_peered":"2024-03-06T07:30:20.626159+0000","last_unstale":"2024-03-06T07:30:22.660041+0000","last_undegraded":"2024-03-06T07:30:22.660041+0000","last_fullsized":"2024-03-06T07:30:22.660041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:39.715637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868820+0000","last_change":"2024-03-06T07:30:21.662241+0000","last_active":"2024-03-06T07:30:22.868820+0000","last_peered":"2024-03-06T07:30:22.868820+0000","last_clean":"2024-03-06T07:30:22.868820+0000","last_became_active":"2024-03-06T07:30:21.661945+0000","last_became_peered":"2024-03-06T07:30:21.661945+0000","last_unstale":"2024-03-06T07:30:22.868820+0000","last_undegraded":"2024-03-06T07:30:22.868820+0000","last_fullsized":"2024-03-06T07:30:22.868820+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:37.330026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871538+0000","last_change":"2024-03-06T07:30:20.628917+0000","last_active":"2024-03-06T07:30:22.871538+0000","last_peered":"2024-03-06T07:30:22.871538+0000","last_clean":"2024-03-06T07:30:22.871538+0000","last_became_active":"2024-03-06T07:30:20.625883+0000","last_became_peered":"2024-03-06T07:30:20.625883+0000","last_unstale":"2024-03-06T07:30:22.871538+0000","last_undegraded":"2024-03-06T07:30:22.871538+0000","last_fullsized":"2024-03-06T07:30:22.871538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:20.538163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872329+0000","last_change":"2024-03-06T07:30:21.662667+0000","last_active":"2024-03-06T07:30:22.872329+0000","last_peered":"2024-03-06T07:30:22.872329+0000","last_clean":"2024-03-06T07:30:22.872329+0000","last_became_active":"2024-03-06T07:30:21.661923+0000","last_became_peered":"2024-03-06T07:30:21.661923+0000","last_unstale":"2024-03-06T07:30:22.872329+0000","last_undegraded":"2024-03-06T07:30:22.872329+0000","last_fullsized":"2024-03-06T07:30:22.872329+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:55.049559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":22,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872200+0000","last_change":"2024-03-06T07:30:20.624267+0000","last_active":"2024-03-06T07:30:22.872200+0000","last_peered":"2024-03-06T07:30:22.872200+0000","last_clean":"2024-03-06T07:30:22.872200+0000","last_became_active":"2024-03-06T07:30:20.623498+0000","last_became_peered":"2024-03-06T07:30:20.623498+0000","last_unstale":"2024-03-06T07:30:22.872200+0000","last_undegraded":"2024-03-06T07:30:22.872200+0000","last_fullsized":"2024-03-06T07:30:22.872200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:49.009758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873141+0000","last_change":"2024-03-06T07:30:21.664390+0000","last_active":"2024-03-06T07:30:22.873141+0000","last_peered":"2024-03-06T07:30:22.873141+0000","last_clean":"2024-03-06T07:30:22.873141+0000","last_became_active":"2024-03-06T07:30:21.664123+0000","last_became_peered":"2024-03-06T07:30:21.664123+0000","last_unstale":"2024-03-06T07:30:22.873141+0000","last_undegraded":"2024-03-06T07:30:22.873141+0000","last_fullsized":"2024-03-06T07:30:22.873141+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:49.898109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651416+0000","last_change":"2024-03-06T07:30:20.625469+0000","last_active":"2024-03-06T07:30:22.651416+0000","last_peered":"2024-03-06T07:30:22.651416+0000","last_clean":"2024-03-06T07:30:22.651416+0000","last_became_active":"2024-03-06T07:30:20.625169+0000","last_became_peered":"2024-03-06T07:30:20.625169+0000","last_unstale":"2024-03-06T07:30:22.651416+0000","last_undegraded":"2024-03-06T07:30:22.651416+0000","last_fullsized":"2024-03-06T07:30:22.651416+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:44.400904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871404+0000","last_change":"2024-03-06T07:30:21.668360+0000","last_active":"2024-03-06T07:30:22.871404+0000","last_peered":"2024-03-06T07:30:22.871404+0000","last_clean":"2024-03-06T07:30:22.871404+0000","last_became_active":"2024-03-06T07:30:21.661641+0000","last_became_peered":"2024-03-06T07:30:21.661641+0000","last_unstale":"2024-03-06T07:30:22.871404+0000","last_undegraded":"2024-03-06T07:30:22.871404+0000","last_fullsized":"2024-03-06T07:30:22.871404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:15.895332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655180+0000","last_change":"2024-03-06T07:30:20.619107+0000","last_active":"2024-03-06T07:30:22.655180+0000","last_peered":"2024-03-06T07:30:22.655180+0000","last_clean":"2024-03-06T07:30:22.655180+0000","last_became_active":"2024-03-06T07:30:20.618101+0000","last_became_peered":"2024-03-06T07:30:20.618101+0000","last_unstale":"2024-03-06T07:30:22.655180+0000","last_undegraded":"2024-03-06T07:30:22.655180+0000","last_fullsized":"2024-03-06T07:30:22.655180+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:20.538163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649359+0000","last_change":"2024-03-06T07:30:21.657575+0000","last_active":"2024-03-06T07:30:22.649359+0000","last_peered":"2024-03-06T07:30:22.649359+0000","last_clean":"2024-03-06T07:30:22.649359+0000","last_became_active":"2024-03-06T07:30:21.657281+0000","last_became_peered":"2024-03-06T07:30:21.657281+0000","last_unstale":"2024-03-06T07:30:22.649359+0000","last_undegraded":"2024-03-06T07:30:22.649359+0000","last_fullsized":"2024-03-06T07:30:22.649359+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:42.061785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872218+0000","last_change":"2024-03-06T07:30:20.626055+0000","last_active":"2024-03-06T07:30:22.872218+0000","last_peered":"2024-03-06T07:30:22.872218+0000","last_clean":"2024-03-06T07:30:22.872218+0000","last_became_active":"2024-03-06T07:30:20.625631+0000","last_became_peered":"2024-03-06T07:30:20.625631+0000","last_unstale":"2024-03-06T07:30:22.872218+0000","last_undegraded":"2024-03-06T07:30:22.872218+0000","last_fullsized":"2024-03-06T07:30:22.872218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:30.885582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874173+0000","last_change":"2024-03-06T07:30:21.673813+0000","last_active":"2024-03-06T07:30:22.874173+0000","last_peered":"2024-03-06T07:30:22.874173+0000","last_clean":"2024-03-06T07:30:22.874173+0000","last_became_active":"2024-03-06T07:30:21.672892+0000","last_became_peered":"2024-03-06T07:30:21.672892+0000","last_unstale":"2024-03-06T07:30:22.874173+0000","last_undegraded":"2024-03-06T07:30:22.874173+0000","last_fullsized":"2024-03-06T07:30:22.874173+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:07.586968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653868+0000","last_change":"2024-03-06T07:30:20.631068+0000","last_active":"2024-03-06T07:30:22.653868+0000","last_peered":"2024-03-06T07:30:22.653868+0000","last_clean":"2024-03-06T07:30:22.653868+0000","last_became_active":"2024-03-06T07:30:20.630723+0000","last_became_peered":"2024-03-06T07:30:20.630723+0000","last_unstale":"2024-03-06T07:30:22.653868+0000","last_undegraded":"2024-03-06T07:30:22.653868+0000","last_fullsized":"2024-03-06T07:30:22.653868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:17.369684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871211+0000","last_change":"2024-03-06T07:30:21.669653+0000","last_active":"2024-03-06T07:30:22.871211+0000","last_peered":"2024-03-06T07:30:22.871211+0000","last_clean":"2024-03-06T07:30:22.871211+0000","last_became_active":"2024-03-06T07:30:21.669360+0000","last_became_peered":"2024-03-06T07:30:21.669360+0000","last_unstale":"2024-03-06T07:30:22.871211+0000","last_undegraded":"2024-03-06T07:30:22.871211+0000","last_fullsized":"2024-03-06T07:30:22.871211+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:38.749441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649397+0000","last_change":"2024-03-06T07:30:20.630056+0000","last_active":"2024-03-06T07:30:22.649397+0000","last_peered":"2024-03-06T07:30:22.649397+0000","last_clean":"2024-03-06T07:30:22.649397+0000","last_became_active":"2024-03-06T07:30:20.629761+0000","last_became_peered":"2024-03-06T07:30:20.629761+0000","last_unstale":"2024-03-06T07:30:22.649397+0000","last_undegraded":"2024-03-06T07:30:22.649397+0000","last_fullsized":"2024-03-06T07:30:22.649397+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:42.666603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653790+0000","last_change":"2024-03-06T07:30:21.657934+0000","last_active":"2024-03-06T07:30:22.653790+0000","last_peered":"2024-03-06T07:30:22.653790+0000","last_clean":"2024-03-06T07:30:22.653790+0000","last_became_active":"2024-03-06T07:30:21.657632+0000","last_became_peered":"2024-03-06T07:30:21.657632+0000","last_unstale":"2024-03-06T07:30:22.653790+0000","last_undegraded":"2024-03-06T07:30:22.653790+0000","last_fullsized":"2024-03-06T07:30:22.653790+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:33.363992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646093+0000","last_change":"2024-03-06T07:30:20.618404+0000","last_active":"2024-03-06T07:30:22.646093+0000","last_peered":"2024-03-06T07:30:22.646093+0000","last_clean":"2024-03-06T07:30:22.646093+0000","last_became_active":"2024-03-06T07:30:20.617575+0000","last_became_peered":"2024-03-06T07:30:20.617575+0000","last_unstale":"2024-03-06T07:30:22.646093+0000","last_undegraded":"2024-03-06T07:30:22.646093+0000","last_fullsized":"2024-03-06T07:30:22.646093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:03.599842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653559+0000","last_change":"2024-03-06T07:30:21.653592+0000","last_active":"2024-03-06T07:30:22.653559+0000","last_peered":"2024-03-06T07:30:22.653559+0000","last_clean":"2024-03-06T07:30:22.653559+0000","last_became_active":"2024-03-06T07:30:21.653210+0000","last_became_peered":"2024-03-06T07:30:21.653210+0000","last_unstale":"2024-03-06T07:30:22.653559+0000","last_undegraded":"2024-03-06T07:30:22.653559+0000","last_fullsized":"2024-03-06T07:30:22.653559+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:57.655078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.870590+0000","last_change":"2024-03-06T07:30:20.633615+0000","last_active":"2024-03-06T07:30:22.870590+0000","last_peered":"2024-03-06T07:30:22.870590+0000","last_clean":"2024-03-06T07:30:22.870590+0000","last_became_active":"2024-03-06T07:30:20.633225+0000","last_became_peered":"2024-03-06T07:30:20.633225+0000","last_unstale":"2024-03-06T07:30:22.870590+0000","last_undegraded":"2024-03-06T07:30:22.870590+0000","last_fullsized":"2024-03-06T07:30:22.870590+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:01.989937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871660+0000","last_change":"2024-03-06T07:30:21.663541+0000","last_active":"2024-03-06T07:30:22.871660+0000","last_peered":"2024-03-06T07:30:22.871660+0000","last_clean":"2024-03-06T07:30:22.871660+0000","last_became_active":"2024-03-06T07:30:21.663217+0000","last_became_peered":"2024-03-06T07:30:21.663217+0000","last_unstale":"2024-03-06T07:30:22.871660+0000","last_undegraded":"2024-03-06T07:30:22.871660+0000","last_fullsized":"2024-03-06T07:30:22.871660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:57.655078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646571+0000","last_change":"2024-03-06T07:30:20.624699+0000","last_active":"2024-03-06T07:30:22.646571+0000","last_peered":"2024-03-06T07:30:22.646571+0000","last_clean":"2024-03-06T07:30:22.646571+0000","last_became_active":"2024-03-06T07:30:20.622946+0000","last_became_peered":"2024-03-06T07:30:20.622946+0000","last_unstale":"2024-03-06T07:30:22.646571+0000","last_undegraded":"2024-03-06T07:30:22.646571+0000","last_fullsized":"2024-03-06T07:30:22.646571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:50.448937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872837+0000","last_change":"2024-03-06T07:30:21.669496+0000","last_active":"2024-03-06T07:30:22.872837+0000","last_peered":"2024-03-06T07:30:22.872837+0000","last_clean":"2024-03-06T07:30:22.872837+0000","last_became_active":"2024-03-06T07:30:21.669162+0000","last_became_peered":"2024-03-06T07:30:21.669162+0000","last_unstale":"2024-03-06T07:30:22.872837+0000","last_undegraded":"2024-03-06T07:30:22.872837+0000","last_fullsized":"2024-03-06T07:30:22.872837+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:45.412635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655321+0000","last_change":"2024-03-06T07:30:20.629443+0000","last_active":"2024-03-06T07:30:22.655321+0000","last_peered":"2024-03-06T07:30:22.655321+0000","last_clean":"2024-03-06T07:30:22.655321+0000","last_became_active":"2024-03-06T07:30:20.629144+0000","last_became_peered":"2024-03-06T07:30:20.629144+0000","last_unstale":"2024-03-06T07:30:22.655321+0000","last_undegraded":"2024-03-06T07:30:22.655321+0000","last_fullsized":"2024-03-06T07:30:22.655321+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:24.918578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646483+0000","last_change":"2024-03-06T07:30:21.650700+0000","last_active":"2024-03-06T07:30:22.646483+0000","last_peered":"2024-03-06T07:30:22.646483+0000","last_clean":"2024-03-06T07:30:22.646483+0000","last_became_active":"2024-03-06T07:30:21.650363+0000","last_became_peered":"2024-03-06T07:30:21.650363+0000","last_unstale":"2024-03-06T07:30:22.646483+0000","last_undegraded":"2024-03-06T07:30:22.646483+0000","last_fullsized":"2024-03-06T07:30:22.646483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:50.021489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660539+0000","last_change":"2024-03-06T07:30:20.629663+0000","last_active":"2024-03-06T07:30:22.660539+0000","last_peered":"2024-03-06T07:30:22.660539+0000","last_clean":"2024-03-06T07:30:22.660539+0000","last_became_active":"2024-03-06T07:30:20.629365+0000","last_became_peered":"2024-03-06T07:30:20.629365+0000","last_unstale":"2024-03-06T07:30:22.660539+0000","last_undegraded":"2024-03-06T07:30:22.660539+0000","last_fullsized":"2024-03-06T07:30:22.660539+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:10.578751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.654090+0000","last_change":"2024-03-06T07:30:21.663132+0000","last_active":"2024-03-06T07:30:22.654090+0000","last_peered":"2024-03-06T07:30:22.654090+0000","last_clean":"2024-03-06T07:30:22.654090+0000","last_became_active":"2024-03-06T07:30:21.662831+0000","last_became_peered":"2024-03-06T07:30:21.662831+0000","last_unstale":"2024-03-06T07:30:22.654090+0000","last_undegraded":"2024-03-06T07:30:22.654090+0000","last_fullsized":"2024-03-06T07:30:22.654090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:01.534240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871902+0000","last_change":"2024-03-06T07:30:20.631439+0000","last_active":"2024-03-06T07:30:22.871902+0000","last_peered":"2024-03-06T07:30:22.871902+0000","last_clean":"2024-03-06T07:30:22.871902+0000","last_became_active":"2024-03-06T07:30:20.631100+0000","last_became_peered":"2024-03-06T07:30:20.631100+0000","last_unstale":"2024-03-06T07:30:22.871902+0000","last_undegraded":"2024-03-06T07:30:22.871902+0000","last_fullsized":"2024-03-06T07:30:22.871902+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:17.369684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649483+0000","last_change":"2024-03-06T07:30:21.645566+0000","last_active":"2024-03-06T07:30:22.649483+0000","last_peered":"2024-03-06T07:30:22.649483+0000","last_clean":"2024-03-06T07:30:22.649483+0000","last_became_active":"2024-03-06T07:30:21.645268+0000","last_became_peered":"2024-03-06T07:30:21.645268+0000","last_unstale":"2024-03-06T07:30:22.649483+0000","last_undegraded":"2024-03-06T07:30:22.649483+0000","last_fullsized":"2024-03-06T07:30:22.649483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:55.464961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646076+0000","last_change":"2024-03-06T07:30:20.623625+0000","last_active":"2024-03-06T07:30:22.646076+0000","last_peered":"2024-03-06T07:30:22.646076+0000","last_clean":"2024-03-06T07:30:22.646076+0000","last_became_active":"2024-03-06T07:30:20.623355+0000","last_became_peered":"2024-03-06T07:30:20.623355+0000","last_unstale":"2024-03-06T07:30:22.646076+0000","last_undegraded":"2024-03-06T07:30:22.646076+0000","last_fullsized":"2024-03-06T07:30:22.646076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:46.004625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873708+0000","last_change":"2024-03-06T07:30:21.661152+0000","last_active":"2024-03-06T07:30:22.873708+0000","last_peered":"2024-03-06T07:30:22.873708+0000","last_clean":"2024-03-06T07:30:22.873708+0000","last_became_active":"2024-03-06T07:30:21.658623+0000","last_became_peered":"2024-03-06T07:30:21.658623+0000","last_unstale":"2024-03-06T07:30:22.873708+0000","last_undegraded":"2024-03-06T07:30:22.873708+0000","last_fullsized":"2024-03-06T07:30:22.873708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:34.085573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869510+0000","last_change":"2024-03-06T07:30:20.632460+0000","last_active":"2024-03-06T07:30:22.869510+0000","last_peered":"2024-03-06T07:30:22.869510+0000","last_clean":"2024-03-06T07:30:22.869510+0000","last_became_active":"2024-03-06T07:30:20.630675+0000","last_became_peered":"2024-03-06T07:30:20.630675+0000","last_unstale":"2024-03-06T07:30:22.869510+0000","last_undegraded":"2024-03-06T07:30:22.869510+0000","last_fullsized":"2024-03-06T07:30:22.869510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:04.820347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.645875+0000","last_change":"2024-03-06T07:30:21.661218+0000","last_active":"2024-03-06T07:30:22.645875+0000","last_peered":"2024-03-06T07:30:22.645875+0000","last_clean":"2024-03-06T07:30:22.645875+0000","last_became_active":"2024-03-06T07:30:21.660549+0000","last_became_peered":"2024-03-06T07:30:21.660549+0000","last_unstale":"2024-03-06T07:30:22.645875+0000","last_undegraded":"2024-03-06T07:30:22.645875+0000","last_fullsized":"2024-03-06T07:30:22.645875+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:12.650544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.645965+0000","last_change":"2024-03-06T07:30:20.609520+0000","last_active":"2024-03-06T07:30:22.645965+0000","last_peered":"2024-03-06T07:30:22.645965+0000","last_clean":"2024-03-06T07:30:22.645965+0000","last_became_active":"2024-03-06T07:30:20.608944+0000","last_became_peered":"2024-03-06T07:30:20.608944+0000","last_unstale":"2024-03-06T07:30:22.645965+0000","last_undegraded":"2024-03-06T07:30:22.645965+0000","last_fullsized":"2024-03-06T07:30:22.645965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:54.924862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648737+0000","last_change":"2024-03-06T07:30:21.655553+0000","last_active":"2024-03-06T07:30:22.648737+0000","last_peered":"2024-03-06T07:30:22.648737+0000","last_clean":"2024-03-06T07:30:22.648737+0000","last_became_active":"2024-03-06T07:30:21.654167+0000","last_became_peered":"2024-03-06T07:30:21.654167+0000","last_unstale":"2024-03-06T07:30:22.648737+0000","last_undegraded":"2024-03-06T07:30:22.648737+0000","last_fullsized":"2024-03-06T07:30:22.648737+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:41.993505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648778+0000","last_change":"2024-03-06T07:30:20.624382+0000","last_active":"2024-03-06T07:30:22.648778+0000","last_peered":"2024-03-06T07:30:22.648778+0000","last_clean":"2024-03-06T07:30:22.648778+0000","last_became_active":"2024-03-06T07:30:20.624100+0000","last_became_peered":"2024-03-06T07:30:20.624100+0000","last_unstale":"2024-03-06T07:30:22.648778+0000","last_undegraded":"2024-03-06T07:30:22.648778+0000","last_fullsized":"2024-03-06T07:30:22.648778+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:50.544775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648675+0000","last_change":"2024-03-06T07:30:21.639012+0000","last_active":"2024-03-06T07:30:22.648675+0000","last_peered":"2024-03-06T07:30:22.648675+0000","last_clean":"2024-03-06T07:30:22.648675+0000","last_became_active":"2024-03-06T07:30:21.638696+0000","last_became_peered":"2024-03-06T07:30:21.638696+0000","last_unstale":"2024-03-06T07:30:22.648675+0000","last_undegraded":"2024-03-06T07:30:22.648675+0000","last_fullsized":"2024-03-06T07:30:22.648675+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:50.623002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660271+0000","last_change":"2024-03-06T07:30:20.624654+0000","last_active":"2024-03-06T07:30:22.660271+0000","last_peered":"2024-03-06T07:30:22.660271+0000","last_clean":"2024-03-06T07:30:22.660271+0000","last_became_active":"2024-03-06T07:30:20.624112+0000","last_became_peered":"2024-03-06T07:30:20.624112+0000","last_unstale":"2024-03-06T07:30:22.660271+0000","last_undegraded":"2024-03-06T07:30:22.660271+0000","last_fullsized":"2024-03-06T07:30:22.660271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:56.720798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":236,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872608+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:30:22.872608+0000","last_peered":"2024-03-06T07:30:22.872608+0000","last_clean":"2024-03-06T07:30:22.872608+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:30:22.872608+0000","last_undegraded":"2024-03-06T07:30:22.872608+0000","last_fullsized":"2024-03-06T07:30:22.872608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509480,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":696320,"data_stored":285563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":285597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542184,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":876035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":688128,"data_stored":285507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":286159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":700416,"data_stored":287119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":285563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1298432,"data_stored":878069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:31.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:30:31.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.543+0000 7f2d5de00640 1 Processor -- start 2024-03-06T07:30:31.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.543+0000 7f2d5de00640 1 -- start start 2024-03-06T07:30:31.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d5de00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2d580d0e50 0x7f2d58168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d5de00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580b3710 con 0x7f2d580d4a10 2024-03-06T07:30:31.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d5de00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d580b39f0 con 0x7f2d5815e0b0 2024-03-06T07:30:31.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d5de00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d580b09e0 con 0x7f2d580d0e50 2024-03-06T07:30:31.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d57e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d580d4a10 0x7f2d580d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55696/0 (socket says 172.21.15.6:55696) 2024-03-06T07:30:31.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d57e00640 1 -- 172.21.15.6:0/2386939134 learned_addr learned my addr 172.21.15.6:0/2386939134 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:31.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.545+0000 7f2d5d400640 1 --2- 172.21.15.6:0/2386939134 >> v2:172.21.15.178:3300/0 conn(0x7f2d580d0e50 0x7f2d58168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021067325 0 0) 0x7f2d580b3710 con 0x7f2d580d4a10 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d40003520 con 0x7f2d580d4a10 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2500927107 0 0) 0x7f2d40003520 con 0x7f2d580d4a10 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 >> v2:172.21.15.178:3300/0 conn(0x7f2d580d0e50 msgr2=0x7f2d58168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 --2- 172.21.15.6:0/2386939134 >> v2:172.21.15.178:3300/0 conn(0x7f2d580d0e50 0x7f2d58168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 >> v1:172.21.15.178:6789/0 conn(0x7f2d5815e0b0 legacy=0x7f2d580cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580b3cd0 con 0x7f2d580d4a10 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2d44001670 con 0x7f2d580d4a10 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 >> v1:172.21.15.6:6789/0 conn(0x7f2d580d4a10 legacy=0x7f2d580d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d400039b0 0x7f2d40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d400051c0 0x7f2d4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c030 con 0x7f2d400039b0 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c1e0 con 0x7f2d40004800 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.546+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4000c3e0 con 0x7f2d400051c0 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d400039b0 0x7f2d40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d57e00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5d400640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d400051c0 0x7f2d4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5ca00640 1 -- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d400051c0 msgr2=0x7f2d4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d400051c0 0x7f2d4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5ca00640 1 -- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d40004800 msgr2=0x7f2d40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5ca00640 1 -- 172.21.15.6:0/2386939134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d40003520 con 0x7f2d400039b0 2024-03-06T07:30:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d57e00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d40004800 0x7f2d40004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:31.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.547+0000 7f2d5d400640 1 -- 172.21.15.6:0/2386939134 reap_dead start 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.548+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d400039b0 0x7f2d40003dd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001750 tx=0x7f2d40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c75d7082719f137 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c001370 con 0x7f2d400039b0 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c0052d0 con 0x7f2d400039b0 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d57400640 1 -- 172.21.15.6:0/2386939134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c003830 con 0x7f2d400039b0 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 -- 172.21.15.6:0/2386939134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d400039b0 msgr2=0x7f2d40003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d400039b0 0x7f2d40003dd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c001750 tx=0x7f2d40005830 comp rx=0 tx=0).stop 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 -- 172.21.15.6:0/2386939134 shutdown_connections 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2386939134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d400039b0 0x7f2d40003dd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 -- 172.21.15.6:0/2386939134 >> 172.21.15.6:0/2386939134 conn(0x7f2d580d3d30 msgr2=0x7f2d580d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 -- 172.21.15.6:0/2386939134 shutdown_connections 2024-03-06T07:30:31.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.549+0000 7f2d5de00640 1 -- 172.21.15.6:0/2386939134 wait complete. 2024-03-06T07:30:31.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.552+0000 7f2d5de00640 1 Processor -- start 2024-03-06T07:30:31.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.552+0000 7f2d5de00640 1 -- start start 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d5de00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d400039b0 0x7f2d580b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 0x7f2d581f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d5de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d5815b640 0x7f2d5815baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d5de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580cfc60 con 0x7f2d5815ffa0 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d5de00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5815e490 con 0x7f2d400039b0 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d5de00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580d0740 con 0x7f2d5815b640 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 0x7f2d581f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 0x7f2d581f4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34166/0 (socket says 172.21.15.6:34166) 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.554+0000 7f2d57e00640 1 -- 172.21.15.6:0/2352723338 learned_addr learned my addr 172.21.15.6:0/2352723338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d5d400640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d5815b640 0x7f2d5815baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d400039b0 0x7f2d580b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d57e00640 1 -- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d5815b640 msgr2=0x7f2d5815baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d57e00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d5815b640 0x7f2d5815baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d57e00640 1 -- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d400039b0 msgr2=0x7f2d580b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d57e00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d400039b0 0x7f2d580b84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d57e00640 1 -- 172.21.15.6:0/2352723338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d581f4a70 con 0x7f2d5815ffa0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d400039b0 0x7f2d580b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.555+0000 7f2d57e00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 0x7f2d581f4530 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400d7b0 tx=0x7f2d4400dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.556+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44019070 con 0x7f2d5815ffa0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.556+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d440074e0 con 0x7f2d5815ffa0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.556+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44004a20 con 0x7f2d5815ffa0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.556+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d58160440 con 0x7f2d5815ffa0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.556+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d58160a00 con 0x7f2d5815ffa0 2024-03-06T07:30:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.557+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5804f330 con 0x7f2d5815ffa0 2024-03-06T07:30:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.558+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d44004bc0 con 0x7f2d5815ffa0 2024-03-06T07:30:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.559+0000 7f2d54c00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2d3806b4d0 0x7f2d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.559+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d44089c80 con 0x7f2d5815ffa0 2024-03-06T07:30:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.560+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2d3806b4d0 0x7f2d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.561+0000 7f2d5ca00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2d3806b4d0 0x7f2d3806d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c033000 tx=0x7f2d4c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.564+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d44058a60 con 0x7f2d5815ffa0 2024-03-06T07:30:31.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.764+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d580b2020 con 0x7f2d3806b4d0 2024-03-06T07:30:31.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.783+0000 7f2d54c00640 1 -- 172.21.15.6:0/2352723338 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251919 (secure 0 0 0) 0x7f2d580b2020 con 0x7f2d3806b4d0 2024-03-06T07:30:31.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.789+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2d3806b4d0 msgr2=0x7f2d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.789+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2d3806b4d0 0x7f2d3806d990 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c033000 tx=0x7f2d4c009030 comp rx=0 tx=0).stop 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.789+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 msgr2=0x7f2d581f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 0x7f2d581f4530 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2d4400d7b0 tx=0x7f2d4400dc70 comp rx=0 tx=0).stop 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 shutdown_connections 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2d3806b4d0 0x7f2d3806d990 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2d5815b640 0x7f2d5815baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d5815ffa0 0x7f2d581f4530 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 --2- 172.21.15.6:0/2352723338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d400039b0 0x7f2d580b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 >> 172.21.15.6:0/2352723338 conn(0x7f2d580d3d30 msgr2=0x7f2d5815d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 shutdown_connections 2024-03-06T07:30:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:31.790+0000 7f2d5de00640 1 -- 172.21.15.6:0/2352723338 wait complete. 2024-03-06T07:30:31.812 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-03-06T07:30:31.502379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264380,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754710340,"statfs":{"total":773094113280,"available":772823388160,"internally_reserved":0,"allocated":6758400,"data_stored":3469612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.888653"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650954+0000","last_change":"2024-03-06T07:30:21.639251+0000","last_active":"2024-03-06T07:30:22.650954+0000","last_peered":"2024-03-06T07:30:22.650954+0000","last_clean":"2024-03-06T07:30:22.650954+0000","last_became_active":"2024-03-06T07:30:21.638950+0000","last_became_peered":"2024-03-06T07:30:21.638950+0000","last_unstale":"2024-03-06T07:30:22.650954+0000","last_undegraded":"2024-03-06T07:30:22.650954+0000","last_fullsized":"2024-03-06T07:30:22.650954+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:12:46.747416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649085+0000","last_change":"2024-03-06T07:30:21.645700+0000","last_active":"2024-03-06T07:30:22.649085+0000","last_peered":"2024-03-06T07:30:22.649085+0000","last_clean":"2024-03-06T07:30:22.649085+0000","last_became_active":"2024-03-06T07:30:21.645396+0000","last_became_peered":"2024-03-06T07:30:21.645396+0000","last_unstale":"2024-03-06T07:30:22.649085+0000","last_undegraded":"2024-03-06T07:30:22.649085+0000","last_fullsized":"2024-03-06T07:30:22.649085+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:46.096729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874028+0000","last_change":"2024-03-06T07:30:21.664820+0000","last_active":"2024-03-06T07:30:22.874028+0000","last_peered":"2024-03-06T07:30:22.874028+0000","last_clean":"2024-03-06T07:30:22.874028+0000","last_became_active":"2024-03-06T07:30:21.659740+0000","last_became_peered":"2024-03-06T07:30:21.659740+0000","last_unstale":"2024-03-06T07:30:22.874028+0000","last_undegraded":"2024-03-06T07:30:22.874028+0000","last_fullsized":"2024-03-06T07:30:22.874028+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:47.037204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871307+0000","last_change":"2024-03-06T07:30:21.671963+0000","last_active":"2024-03-06T07:30:22.871307+0000","last_peered":"2024-03-06T07:30:22.871307+0000","last_clean":"2024-03-06T07:30:22.871307+0000","last_became_active":"2024-03-06T07:30:21.671541+0000","last_became_peered":"2024-03-06T07:30:21.671541+0000","last_unstale":"2024-03-06T07:30:22.871307+0000","last_undegraded":"2024-03-06T07:30:22.871307+0000","last_fullsized":"2024-03-06T07:30:22.871307+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:01:16.551735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650630+0000","last_change":"2024-03-06T07:30:21.671393+0000","last_active":"2024-03-06T07:30:22.650630+0000","last_peered":"2024-03-06T07:30:22.650630+0000","last_clean":"2024-03-06T07:30:22.650630+0000","last_became_active":"2024-03-06T07:30:21.671107+0000","last_became_peered":"2024-03-06T07:30:21.671107+0000","last_unstale":"2024-03-06T07:30:22.650630+0000","last_undegraded":"2024-03-06T07:30:22.650630+0000","last_fullsized":"2024-03-06T07:30:22.650630+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:15.531515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653562+0000","last_change":"2024-03-06T07:30:21.639455+0000","last_active":"2024-03-06T07:30:22.653562+0000","last_peered":"2024-03-06T07:30:22.653562+0000","last_clean":"2024-03-06T07:30:22.653562+0000","last_became_active":"2024-03-06T07:30:21.639169+0000","last_became_peered":"2024-03-06T07:30:21.639169+0000","last_unstale":"2024-03-06T07:30:22.653562+0000","last_undegraded":"2024-03-06T07:30:22.653562+0000","last_fullsized":"2024-03-06T07:30:22.653562+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:45.412635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868908+0000","last_change":"2024-03-06T07:30:21.664479+0000","last_active":"2024-03-06T07:30:22.868908+0000","last_peered":"2024-03-06T07:30:22.868908+0000","last_clean":"2024-03-06T07:30:22.868908+0000","last_became_active":"2024-03-06T07:30:21.664187+0000","last_became_peered":"2024-03-06T07:30:21.664187+0000","last_unstale":"2024-03-06T07:30:22.868908+0000","last_undegraded":"2024-03-06T07:30:22.868908+0000","last_fullsized":"2024-03-06T07:30:22.868908+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:46.693347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648518+0000","last_change":"2024-03-06T07:30:21.658244+0000","last_active":"2024-03-06T07:30:22.648518+0000","last_peered":"2024-03-06T07:30:22.648518+0000","last_clean":"2024-03-06T07:30:22.648518+0000","last_became_active":"2024-03-06T07:30:21.657945+0000","last_became_peered":"2024-03-06T07:30:21.657945+0000","last_unstale":"2024-03-06T07:30:22.648518+0000","last_undegraded":"2024-03-06T07:30:22.648518+0000","last_fullsized":"2024-03-06T07:30:22.648518+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:22.749385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.870079+0000","last_change":"2024-03-06T07:30:21.648774+0000","last_active":"2024-03-06T07:30:22.870079+0000","last_peered":"2024-03-06T07:30:22.870079+0000","last_clean":"2024-03-06T07:30:22.870079+0000","last_became_active":"2024-03-06T07:30:21.648225+0000","last_became_peered":"2024-03-06T07:30:21.648225+0000","last_unstale":"2024-03-06T07:30:22.870079+0000","last_undegraded":"2024-03-06T07:30:22.870079+0000","last_fullsized":"2024-03-06T07:30:22.870079+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:35.687120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646319+0000","last_change":"2024-03-06T07:30:21.658602+0000","last_active":"2024-03-06T07:30:22.646319+0000","last_peered":"2024-03-06T07:30:22.646319+0000","last_clean":"2024-03-06T07:30:22.646319+0000","last_became_active":"2024-03-06T07:30:21.656608+0000","last_became_peered":"2024-03-06T07:30:21.656608+0000","last_unstale":"2024-03-06T07:30:22.646319+0000","last_undegraded":"2024-03-06T07:30:22.646319+0000","last_fullsized":"2024-03-06T07:30:22.646319+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:01.731712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646507+0000","last_change":"2024-03-06T07:30:21.656749+0000","last_active":"2024-03-06T07:30:22.646507+0000","last_peered":"2024-03-06T07:30:22.646507+0000","last_clean":"2024-03-06T07:30:22.646507+0000","last_became_active":"2024-03-06T07:30:21.656479+0000","last_became_peered":"2024-03-06T07:30:21.656479+0000","last_unstale":"2024-03-06T07:30:22.646507+0000","last_undegraded":"2024-03-06T07:30:22.646507+0000","last_fullsized":"2024-03-06T07:30:22.646507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:17.732383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871659+0000","last_change":"2024-03-06T07:30:21.671007+0000","last_active":"2024-03-06T07:30:22.871659+0000","last_peered":"2024-03-06T07:30:22.871659+0000","last_clean":"2024-03-06T07:30:22.871659+0000","last_became_active":"2024-03-06T07:30:21.670719+0000","last_became_peered":"2024-03-06T07:30:21.670719+0000","last_unstale":"2024-03-06T07:30:22.871659+0000","last_undegraded":"2024-03-06T07:30:22.871659+0000","last_fullsized":"2024-03-06T07:30:22.871659+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:07.555539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660452+0000","last_change":"2024-03-06T07:30:21.648530+0000","last_active":"2024-03-06T07:30:22.660452+0000","last_peered":"2024-03-06T07:30:22.660452+0000","last_clean":"2024-03-06T07:30:22.660452+0000","last_became_active":"2024-03-06T07:30:21.648138+0000","last_became_peered":"2024-03-06T07:30:21.648138+0000","last_unstale":"2024-03-06T07:30:22.660452+0000","last_undegraded":"2024-03-06T07:30:22.660452+0000","last_fullsized":"2024-03-06T07:30:22.660452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:24.185476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869469+0000","last_change":"2024-03-06T07:30:21.652171+0000","last_active":"2024-03-06T07:30:22.869469+0000","last_peered":"2024-03-06T07:30:22.869469+0000","last_clean":"2024-03-06T07:30:22.869469+0000","last_became_active":"2024-03-06T07:30:21.651210+0000","last_became_peered":"2024-03-06T07:30:21.651210+0000","last_unstale":"2024-03-06T07:30:22.869469+0000","last_undegraded":"2024-03-06T07:30:22.869469+0000","last_fullsized":"2024-03-06T07:30:22.869469+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:25:29.003761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868924+0000","last_change":"2024-03-06T07:30:21.653860+0000","last_active":"2024-03-06T07:30:22.868924+0000","last_peered":"2024-03-06T07:30:22.868924+0000","last_clean":"2024-03-06T07:30:22.868924+0000","last_became_active":"2024-03-06T07:30:21.651008+0000","last_became_peered":"2024-03-06T07:30:21.651008+0000","last_unstale":"2024-03-06T07:30:22.868924+0000","last_undegraded":"2024-03-06T07:30:22.868924+0000","last_fullsized":"2024-03-06T07:30:22.868924+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:44.073218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653435+0000","last_change":"2024-03-06T07:30:21.657282+0000","last_active":"2024-03-06T07:30:22.653435+0000","last_peered":"2024-03-06T07:30:22.653435+0000","last_clean":"2024-03-06T07:30:22.653435+0000","last_became_active":"2024-03-06T07:30:21.654676+0000","last_became_peered":"2024-03-06T07:30:21.654676+0000","last_unstale":"2024-03-06T07:30:22.653435+0000","last_undegraded":"2024-03-06T07:30:22.653435+0000","last_fullsized":"2024-03-06T07:30:22.653435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:51.556506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651187+0000","last_change":"2024-03-06T07:30:21.654626+0000","last_active":"2024-03-06T07:30:22.651187+0000","last_peered":"2024-03-06T07:30:22.651187+0000","last_clean":"2024-03-06T07:30:22.651187+0000","last_became_active":"2024-03-06T07:30:21.654293+0000","last_became_peered":"2024-03-06T07:30:21.654293+0000","last_unstale":"2024-03-06T07:30:22.651187+0000","last_undegraded":"2024-03-06T07:30:22.651187+0000","last_fullsized":"2024-03-06T07:30:22.651187+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:51.388978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872868+0000","last_change":"2024-03-06T07:30:21.658100+0000","last_active":"2024-03-06T07:30:22.872868+0000","last_peered":"2024-03-06T07:30:22.872868+0000","last_clean":"2024-03-06T07:30:22.872868+0000","last_became_active":"2024-03-06T07:30:21.656111+0000","last_became_peered":"2024-03-06T07:30:21.656111+0000","last_unstale":"2024-03-06T07:30:22.872868+0000","last_undegraded":"2024-03-06T07:30:22.872868+0000","last_fullsized":"2024-03-06T07:30:22.872868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:23.468391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650808+0000","last_change":"2024-03-06T07:30:21.635273+0000","last_active":"2024-03-06T07:30:22.650808+0000","last_peered":"2024-03-06T07:30:22.650808+0000","last_clean":"2024-03-06T07:30:22.650808+0000","last_became_active":"2024-03-06T07:30:21.634938+0000","last_became_peered":"2024-03-06T07:30:21.634938+0000","last_unstale":"2024-03-06T07:30:22.650808+0000","last_undegraded":"2024-03-06T07:30:22.650808+0000","last_fullsized":"2024-03-06T07:30:22.650808+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:56.567717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649218+0000","last_change":"2024-03-06T07:30:21.640509+0000","last_active":"2024-03-06T07:30:22.649218+0000","last_peered":"2024-03-06T07:30:22.649218+0000","last_clean":"2024-03-06T07:30:22.649218+0000","last_became_active":"2024-03-06T07:30:21.640199+0000","last_became_peered":"2024-03-06T07:30:21.640199+0000","last_unstale":"2024-03-06T07:30:22.649218+0000","last_undegraded":"2024-03-06T07:30:22.649218+0000","last_fullsized":"2024-03-06T07:30:22.649218+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:00.156288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873082+0000","last_change":"2024-03-06T07:30:21.673682+0000","last_active":"2024-03-06T07:30:22.873082+0000","last_peered":"2024-03-06T07:30:22.873082+0000","last_clean":"2024-03-06T07:30:22.873082+0000","last_became_active":"2024-03-06T07:30:21.672381+0000","last_became_peered":"2024-03-06T07:30:21.672381+0000","last_unstale":"2024-03-06T07:30:22.873082+0000","last_undegraded":"2024-03-06T07:30:22.873082+0000","last_fullsized":"2024-03-06T07:30:22.873082+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:39.074474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648673+0000","last_change":"2024-03-06T07:30:21.633453+0000","last_active":"2024-03-06T07:30:22.648673+0000","last_peered":"2024-03-06T07:30:22.648673+0000","last_clean":"2024-03-06T07:30:22.648673+0000","last_became_active":"2024-03-06T07:30:21.632771+0000","last_became_peered":"2024-03-06T07:30:21.632771+0000","last_unstale":"2024-03-06T07:30:22.648673+0000","last_undegraded":"2024-03-06T07:30:22.648673+0000","last_fullsized":"2024-03-06T07:30:22.648673+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:38.963735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872321+0000","last_change":"2024-03-06T07:30:21.662230+0000","last_active":"2024-03-06T07:30:22.872321+0000","last_peered":"2024-03-06T07:30:22.872321+0000","last_clean":"2024-03-06T07:30:22.872321+0000","last_became_active":"2024-03-06T07:30:21.659240+0000","last_became_peered":"2024-03-06T07:30:21.659240+0000","last_unstale":"2024-03-06T07:30:22.872321+0000","last_undegraded":"2024-03-06T07:30:22.872321+0000","last_fullsized":"2024-03-06T07:30:22.872321+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:35.687120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660249+0000","last_change":"2024-03-06T07:30:21.652868+0000","last_active":"2024-03-06T07:30:22.660249+0000","last_peered":"2024-03-06T07:30:22.660249+0000","last_clean":"2024-03-06T07:30:22.660249+0000","last_became_active":"2024-03-06T07:30:21.652069+0000","last_became_peered":"2024-03-06T07:30:21.652069+0000","last_unstale":"2024-03-06T07:30:22.660249+0000","last_undegraded":"2024-03-06T07:30:22.660249+0000","last_fullsized":"2024-03-06T07:30:22.660249+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:17.298033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651046+0000","last_change":"2024-03-06T07:30:21.646940+0000","last_active":"2024-03-06T07:30:22.651046+0000","last_peered":"2024-03-06T07:30:22.651046+0000","last_clean":"2024-03-06T07:30:22.651046+0000","last_became_active":"2024-03-06T07:30:21.644768+0000","last_became_peered":"2024-03-06T07:30:21.644768+0000","last_unstale":"2024-03-06T07:30:22.651046+0000","last_undegraded":"2024-03-06T07:30:22.651046+0000","last_fullsized":"2024-03-06T07:30:22.651046+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:34.516723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871233+0000","last_change":"2024-03-06T07:30:21.662980+0000","last_active":"2024-03-06T07:30:22.871233+0000","last_peered":"2024-03-06T07:30:22.871233+0000","last_clean":"2024-03-06T07:30:22.871233+0000","last_became_active":"2024-03-06T07:30:21.662680+0000","last_became_peered":"2024-03-06T07:30:21.662680+0000","last_unstale":"2024-03-06T07:30:22.871233+0000","last_undegraded":"2024-03-06T07:30:22.871233+0000","last_fullsized":"2024-03-06T07:30:22.871233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:31.415796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653917+0000","last_change":"2024-03-06T07:30:21.654904+0000","last_active":"2024-03-06T07:30:22.653917+0000","last_peered":"2024-03-06T07:30:22.653917+0000","last_clean":"2024-03-06T07:30:22.653917+0000","last_became_active":"2024-03-06T07:30:21.653958+0000","last_became_peered":"2024-03-06T07:30:21.653958+0000","last_unstale":"2024-03-06T07:30:22.653917+0000","last_undegraded":"2024-03-06T07:30:22.653917+0000","last_fullsized":"2024-03-06T07:30:22.653917+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:54.366453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650784+0000","last_change":"2024-03-06T07:30:21.663886+0000","last_active":"2024-03-06T07:30:22.650784+0000","last_peered":"2024-03-06T07:30:22.650784+0000","last_clean":"2024-03-06T07:30:22.650784+0000","last_became_active":"2024-03-06T07:30:21.663511+0000","last_became_peered":"2024-03-06T07:30:21.663511+0000","last_unstale":"2024-03-06T07:30:22.650784+0000","last_undegraded":"2024-03-06T07:30:22.650784+0000","last_fullsized":"2024-03-06T07:30:22.650784+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:50.002026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649302+0000","last_change":"2024-03-06T07:30:21.671940+0000","last_active":"2024-03-06T07:30:22.649302+0000","last_peered":"2024-03-06T07:30:22.649302+0000","last_clean":"2024-03-06T07:30:22.649302+0000","last_became_active":"2024-03-06T07:30:21.671710+0000","last_became_peered":"2024-03-06T07:30:21.671710+0000","last_unstale":"2024-03-06T07:30:22.649302+0000","last_undegraded":"2024-03-06T07:30:22.649302+0000","last_fullsized":"2024-03-06T07:30:22.649302+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:20.419130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873042+0000","last_change":"2024-03-06T07:30:21.664779+0000","last_active":"2024-03-06T07:30:22.873042+0000","last_peered":"2024-03-06T07:30:22.873042+0000","last_clean":"2024-03-06T07:30:22.873042+0000","last_became_active":"2024-03-06T07:30:21.663664+0000","last_became_peered":"2024-03-06T07:30:21.663664+0000","last_unstale":"2024-03-06T07:30:22.873042+0000","last_undegraded":"2024-03-06T07:30:22.873042+0000","last_fullsized":"2024-03-06T07:30:22.873042+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:11.037035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650135+0000","last_change":"2024-03-06T07:30:21.645937+0000","last_active":"2024-03-06T07:30:22.650135+0000","last_peered":"2024-03-06T07:30:22.650135+0000","last_clean":"2024-03-06T07:30:22.650135+0000","last_became_active":"2024-03-06T07:30:21.644909+0000","last_became_peered":"2024-03-06T07:30:21.644909+0000","last_unstale":"2024-03-06T07:30:22.650135+0000","last_undegraded":"2024-03-06T07:30:22.650135+0000","last_fullsized":"2024-03-06T07:30:22.650135+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:35:42.919081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653474+0000","last_change":"2024-03-06T07:30:21.636575+0000","last_active":"2024-03-06T07:30:22.653474+0000","last_peered":"2024-03-06T07:30:22.653474+0000","last_clean":"2024-03-06T07:30:22.653474+0000","last_became_active":"2024-03-06T07:30:21.634856+0000","last_became_peered":"2024-03-06T07:30:21.634856+0000","last_unstale":"2024-03-06T07:30:22.653474+0000","last_undegraded":"2024-03-06T07:30:22.653474+0000","last_fullsized":"2024-03-06T07:30:22.653474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:07.555539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655221+0000","last_change":"2024-03-06T07:30:21.653801+0000","last_active":"2024-03-06T07:30:22.655221+0000","last_peered":"2024-03-06T07:30:22.655221+0000","last_clean":"2024-03-06T07:30:22.655221+0000","last_became_active":"2024-03-06T07:30:21.653476+0000","last_became_peered":"2024-03-06T07:30:21.653476+0000","last_unstale":"2024-03-06T07:30:22.655221+0000","last_undegraded":"2024-03-06T07:30:22.655221+0000","last_fullsized":"2024-03-06T07:30:22.655221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:35.655428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655282+0000","last_change":"2024-03-06T07:30:20.630369+0000","last_active":"2024-03-06T07:30:22.655282+0000","last_peered":"2024-03-06T07:30:22.655282+0000","last_clean":"2024-03-06T07:30:22.655282+0000","last_became_active":"2024-03-06T07:30:20.628580+0000","last_became_peered":"2024-03-06T07:30:20.628580+0000","last_unstale":"2024-03-06T07:30:22.655282+0000","last_undegraded":"2024-03-06T07:30:22.655282+0000","last_fullsized":"2024-03-06T07:30:22.655282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:01.989937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646633+0000","last_change":"2024-03-06T07:30:21.664516+0000","last_active":"2024-03-06T07:30:22.646633+0000","last_peered":"2024-03-06T07:30:22.646633+0000","last_clean":"2024-03-06T07:30:22.646633+0000","last_became_active":"2024-03-06T07:30:21.664224+0000","last_became_peered":"2024-03-06T07:30:21.664224+0000","last_unstale":"2024-03-06T07:30:22.646633+0000","last_undegraded":"2024-03-06T07:30:22.646633+0000","last_fullsized":"2024-03-06T07:30:22.646633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:05.832079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.650905+0000","last_change":"2024-03-06T07:30:20.625914+0000","last_active":"2024-03-06T07:30:22.650905+0000","last_peered":"2024-03-06T07:30:22.650905+0000","last_clean":"2024-03-06T07:30:22.650905+0000","last_became_active":"2024-03-06T07:30:20.625643+0000","last_became_peered":"2024-03-06T07:30:20.625643+0000","last_unstale":"2024-03-06T07:30:22.650905+0000","last_undegraded":"2024-03-06T07:30:22.650905+0000","last_fullsized":"2024-03-06T07:30:22.650905+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:06.543808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653909+0000","last_change":"2024-03-06T07:30:21.644886+0000","last_active":"2024-03-06T07:30:22.653909+0000","last_peered":"2024-03-06T07:30:22.653909+0000","last_clean":"2024-03-06T07:30:22.653909+0000","last_became_active":"2024-03-06T07:30:21.643841+0000","last_became_peered":"2024-03-06T07:30:21.643841+0000","last_unstale":"2024-03-06T07:30:22.653909+0000","last_undegraded":"2024-03-06T07:30:22.653909+0000","last_fullsized":"2024-03-06T07:30:22.653909+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:43.678334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.654086+0000","last_change":"2024-03-06T07:30:20.617109+0000","last_active":"2024-03-06T07:30:22.654086+0000","last_peered":"2024-03-06T07:30:22.654086+0000","last_clean":"2024-03-06T07:30:22.654086+0000","last_became_active":"2024-03-06T07:30:20.615288+0000","last_became_peered":"2024-03-06T07:30:20.615288+0000","last_unstale":"2024-03-06T07:30:22.654086+0000","last_undegraded":"2024-03-06T07:30:22.654086+0000","last_fullsized":"2024-03-06T07:30:22.654086+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:24.398212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653973+0000","last_change":"2024-03-06T07:30:21.673120+0000","last_active":"2024-03-06T07:30:22.653973+0000","last_peered":"2024-03-06T07:30:22.653973+0000","last_clean":"2024-03-06T07:30:22.653973+0000","last_became_active":"2024-03-06T07:30:21.672837+0000","last_became_peered":"2024-03-06T07:30:21.672837+0000","last_unstale":"2024-03-06T07:30:22.653973+0000","last_undegraded":"2024-03-06T07:30:22.653973+0000","last_fullsized":"2024-03-06T07:30:22.653973+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:49.011453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871602+0000","last_change":"2024-03-06T07:30:20.626609+0000","last_active":"2024-03-06T07:30:22.871602+0000","last_peered":"2024-03-06T07:30:22.871602+0000","last_clean":"2024-03-06T07:30:22.871602+0000","last_became_active":"2024-03-06T07:30:20.625668+0000","last_became_peered":"2024-03-06T07:30:20.625668+0000","last_unstale":"2024-03-06T07:30:22.871602+0000","last_undegraded":"2024-03-06T07:30:22.871602+0000","last_fullsized":"2024-03-06T07:30:22.871602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:24.918578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648518+0000","last_change":"2024-03-06T07:30:21.634922+0000","last_active":"2024-03-06T07:30:22.648518+0000","last_peered":"2024-03-06T07:30:22.648518+0000","last_clean":"2024-03-06T07:30:22.648518+0000","last_became_active":"2024-03-06T07:30:21.634102+0000","last_became_peered":"2024-03-06T07:30:21.634102+0000","last_unstale":"2024-03-06T07:30:22.648518+0000","last_undegraded":"2024-03-06T07:30:22.648518+0000","last_fullsized":"2024-03-06T07:30:22.648518+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:40.537463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869351+0000","last_change":"2024-03-06T07:30:20.629926+0000","last_active":"2024-03-06T07:30:22.869351+0000","last_peered":"2024-03-06T07:30:22.869351+0000","last_clean":"2024-03-06T07:30:22.869351+0000","last_became_active":"2024-03-06T07:30:20.629470+0000","last_became_peered":"2024-03-06T07:30:20.629470+0000","last_unstale":"2024-03-06T07:30:22.869351+0000","last_undegraded":"2024-03-06T07:30:22.869351+0000","last_fullsized":"2024-03-06T07:30:22.869351+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:55.532286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649048+0000","last_change":"2024-03-06T07:30:21.640716+0000","last_active":"2024-03-06T07:30:22.649048+0000","last_peered":"2024-03-06T07:30:22.649048+0000","last_clean":"2024-03-06T07:30:22.649048+0000","last_became_active":"2024-03-06T07:30:21.640180+0000","last_became_peered":"2024-03-06T07:30:21.640180+0000","last_unstale":"2024-03-06T07:30:22.649048+0000","last_undegraded":"2024-03-06T07:30:22.649048+0000","last_fullsized":"2024-03-06T07:30:22.649048+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:49:09.590013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653660+0000","last_change":"2024-03-06T07:30:20.627675+0000","last_active":"2024-03-06T07:30:22.653660+0000","last_peered":"2024-03-06T07:30:22.653660+0000","last_clean":"2024-03-06T07:30:22.653660+0000","last_became_active":"2024-03-06T07:30:20.627383+0000","last_became_peered":"2024-03-06T07:30:20.627383+0000","last_unstale":"2024-03-06T07:30:22.653660+0000","last_undegraded":"2024-03-06T07:30:22.653660+0000","last_fullsized":"2024-03-06T07:30:22.653660+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:14.427677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660664+0000","last_change":"2024-03-06T07:30:21.657886+0000","last_active":"2024-03-06T07:30:22.660664+0000","last_peered":"2024-03-06T07:30:22.660664+0000","last_clean":"2024-03-06T07:30:22.660664+0000","last_became_active":"2024-03-06T07:30:21.656690+0000","last_became_peered":"2024-03-06T07:30:21.656690+0000","last_unstale":"2024-03-06T07:30:22.660664+0000","last_undegraded":"2024-03-06T07:30:22.660664+0000","last_fullsized":"2024-03-06T07:30:22.660664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:38.442564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871290+0000","last_change":"2024-03-06T07:30:20.629740+0000","last_active":"2024-03-06T07:30:22.871290+0000","last_peered":"2024-03-06T07:30:22.871290+0000","last_clean":"2024-03-06T07:30:22.871290+0000","last_became_active":"2024-03-06T07:30:20.629427+0000","last_became_peered":"2024-03-06T07:30:20.629427+0000","last_unstale":"2024-03-06T07:30:22.871290+0000","last_undegraded":"2024-03-06T07:30:22.871290+0000","last_fullsized":"2024-03-06T07:30:22.871290+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:59:14.427677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648584+0000","last_change":"2024-03-06T07:30:21.655791+0000","last_active":"2024-03-06T07:30:22.648584+0000","last_peered":"2024-03-06T07:30:22.648584+0000","last_clean":"2024-03-06T07:30:22.648584+0000","last_became_active":"2024-03-06T07:30:21.654685+0000","last_became_peered":"2024-03-06T07:30:21.654685+0000","last_unstale":"2024-03-06T07:30:22.648584+0000","last_undegraded":"2024-03-06T07:30:22.648584+0000","last_fullsized":"2024-03-06T07:30:22.648584+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:43:38.630816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868866+0000","last_change":"2024-03-06T07:30:20.634196+0000","last_active":"2024-03-06T07:30:22.868866+0000","last_peered":"2024-03-06T07:30:22.868866+0000","last_clean":"2024-03-06T07:30:22.868866+0000","last_became_active":"2024-03-06T07:30:20.633405+0000","last_became_peered":"2024-03-06T07:30:20.633405+0000","last_unstale":"2024-03-06T07:30:22.868866+0000","last_undegraded":"2024-03-06T07:30:22.868866+0000","last_fullsized":"2024-03-06T07:30:22.868866+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.720652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.870474+0000","last_change":"2024-03-06T07:30:21.655531+0000","last_active":"2024-03-06T07:30:22.870474+0000","last_peered":"2024-03-06T07:30:22.870474+0000","last_clean":"2024-03-06T07:30:22.870474+0000","last_became_active":"2024-03-06T07:30:21.654818+0000","last_became_peered":"2024-03-06T07:30:21.654818+0000","last_unstale":"2024-03-06T07:30:22.870474+0000","last_undegraded":"2024-03-06T07:30:22.870474+0000","last_fullsized":"2024-03-06T07:30:22.870474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:55.049559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873858+0000","last_change":"2024-03-06T07:30:20.627513+0000","last_active":"2024-03-06T07:30:22.873858+0000","last_peered":"2024-03-06T07:30:22.873858+0000","last_clean":"2024-03-06T07:30:22.873858+0000","last_became_active":"2024-03-06T07:30:20.627220+0000","last_became_peered":"2024-03-06T07:30:20.627220+0000","last_unstale":"2024-03-06T07:30:22.873858+0000","last_undegraded":"2024-03-06T07:30:22.873858+0000","last_fullsized":"2024-03-06T07:30:22.873858+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:55.532286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873699+0000","last_change":"2024-03-06T07:30:21.670188+0000","last_active":"2024-03-06T07:30:22.873699+0000","last_peered":"2024-03-06T07:30:22.873699+0000","last_clean":"2024-03-06T07:30:22.873699+0000","last_became_active":"2024-03-06T07:30:21.669962+0000","last_became_peered":"2024-03-06T07:30:21.669962+0000","last_unstale":"2024-03-06T07:30:22.873699+0000","last_undegraded":"2024-03-06T07:30:22.873699+0000","last_fullsized":"2024-03-06T07:30:22.873699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:21.619393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651018+0000","last_change":"2024-03-06T07:30:20.628649+0000","last_active":"2024-03-06T07:30:22.651018+0000","last_peered":"2024-03-06T07:30:22.651018+0000","last_clean":"2024-03-06T07:30:22.651018+0000","last_became_active":"2024-03-06T07:30:20.628065+0000","last_became_peered":"2024-03-06T07:30:20.628065+0000","last_unstale":"2024-03-06T07:30:22.651018+0000","last_undegraded":"2024-03-06T07:30:22.651018+0000","last_fullsized":"2024-03-06T07:30:22.651018+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:56.643347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646298+0000","last_change":"2024-03-06T07:30:21.652433+0000","last_active":"2024-03-06T07:30:22.646298+0000","last_peered":"2024-03-06T07:30:22.646298+0000","last_clean":"2024-03-06T07:30:22.646298+0000","last_became_active":"2024-03-06T07:30:21.651796+0000","last_became_peered":"2024-03-06T07:30:21.651796+0000","last_unstale":"2024-03-06T07:30:22.646298+0000","last_undegraded":"2024-03-06T07:30:22.646298+0000","last_fullsized":"2024-03-06T07:30:22.646298+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:44:19.746899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":17,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874259+0000","last_change":"2024-03-06T07:30:20.628448+0000","last_active":"2024-03-06T07:30:22.874259+0000","last_peered":"2024-03-06T07:30:22.874259+0000","last_clean":"2024-03-06T07:30:22.874259+0000","last_became_active":"2024-03-06T07:30:20.628114+0000","last_became_peered":"2024-03-06T07:30:20.628114+0000","last_unstale":"2024-03-06T07:30:22.874259+0000","last_undegraded":"2024-03-06T07:30:22.874259+0000","last_fullsized":"2024-03-06T07:30:22.874259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:26:16.720652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874204+0000","last_change":"2024-03-06T07:30:21.673522+0000","last_active":"2024-03-06T07:30:22.874204+0000","last_peered":"2024-03-06T07:30:22.874204+0000","last_clean":"2024-03-06T07:30:22.874204+0000","last_became_active":"2024-03-06T07:30:21.672490+0000","last_became_peered":"2024-03-06T07:30:21.672490+0000","last_unstale":"2024-03-06T07:30:22.874204+0000","last_undegraded":"2024-03-06T07:30:22.874204+0000","last_fullsized":"2024-03-06T07:30:22.874204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:01.932495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869896+0000","last_change":"2024-03-06T07:30:20.629645+0000","last_active":"2024-03-06T07:30:22.869896+0000","last_peered":"2024-03-06T07:30:22.869896+0000","last_clean":"2024-03-06T07:30:22.869896+0000","last_became_active":"2024-03-06T07:30:20.628235+0000","last_became_peered":"2024-03-06T07:30:20.628235+0000","last_unstale":"2024-03-06T07:30:22.869896+0000","last_undegraded":"2024-03-06T07:30:22.869896+0000","last_fullsized":"2024-03-06T07:30:22.869896+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:27.152751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871611+0000","last_change":"2024-03-06T07:30:21.670083+0000","last_active":"2024-03-06T07:30:22.871611+0000","last_peered":"2024-03-06T07:30:22.871611+0000","last_clean":"2024-03-06T07:30:22.871611+0000","last_became_active":"2024-03-06T07:30:21.669474+0000","last_became_peered":"2024-03-06T07:30:21.669474+0000","last_unstale":"2024-03-06T07:30:22.871611+0000","last_undegraded":"2024-03-06T07:30:22.871611+0000","last_fullsized":"2024-03-06T07:30:22.871611+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:51.556506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649148+0000","last_change":"2024-03-06T07:30:20.627658+0000","last_active":"2024-03-06T07:30:22.649148+0000","last_peered":"2024-03-06T07:30:22.649148+0000","last_clean":"2024-03-06T07:30:22.649148+0000","last_became_active":"2024-03-06T07:30:20.627370+0000","last_became_peered":"2024-03-06T07:30:20.627370+0000","last_unstale":"2024-03-06T07:30:22.649148+0000","last_undegraded":"2024-03-06T07:30:22.649148+0000","last_fullsized":"2024-03-06T07:30:22.649148+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:14.883601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873519+0000","last_change":"2024-03-06T07:30:21.654626+0000","last_active":"2024-03-06T07:30:22.873519+0000","last_peered":"2024-03-06T07:30:22.873519+0000","last_clean":"2024-03-06T07:30:22.873519+0000","last_became_active":"2024-03-06T07:30:21.654286+0000","last_became_peered":"2024-03-06T07:30:21.654286+0000","last_unstale":"2024-03-06T07:30:22.873519+0000","last_undegraded":"2024-03-06T07:30:22.873519+0000","last_fullsized":"2024-03-06T07:30:22.873519+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:41.911025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660041+0000","last_change":"2024-03-06T07:30:20.626528+0000","last_active":"2024-03-06T07:30:22.660041+0000","last_peered":"2024-03-06T07:30:22.660041+0000","last_clean":"2024-03-06T07:30:22.660041+0000","last_became_active":"2024-03-06T07:30:20.626159+0000","last_became_peered":"2024-03-06T07:30:20.626159+0000","last_unstale":"2024-03-06T07:30:22.660041+0000","last_undegraded":"2024-03-06T07:30:22.660041+0000","last_fullsized":"2024-03-06T07:30:22.660041+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:39.715637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.868820+0000","last_change":"2024-03-06T07:30:21.662241+0000","last_active":"2024-03-06T07:30:22.868820+0000","last_peered":"2024-03-06T07:30:22.868820+0000","last_clean":"2024-03-06T07:30:22.868820+0000","last_became_active":"2024-03-06T07:30:21.661945+0000","last_became_peered":"2024-03-06T07:30:21.661945+0000","last_unstale":"2024-03-06T07:30:22.868820+0000","last_undegraded":"2024-03-06T07:30:22.868820+0000","last_fullsized":"2024-03-06T07:30:22.868820+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:37.330026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871538+0000","last_change":"2024-03-06T07:30:20.628917+0000","last_active":"2024-03-06T07:30:22.871538+0000","last_peered":"2024-03-06T07:30:22.871538+0000","last_clean":"2024-03-06T07:30:22.871538+0000","last_became_active":"2024-03-06T07:30:20.625883+0000","last_became_peered":"2024-03-06T07:30:20.625883+0000","last_unstale":"2024-03-06T07:30:22.871538+0000","last_undegraded":"2024-03-06T07:30:22.871538+0000","last_fullsized":"2024-03-06T07:30:22.871538+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:20.538163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872329+0000","last_change":"2024-03-06T07:30:21.662667+0000","last_active":"2024-03-06T07:30:22.872329+0000","last_peered":"2024-03-06T07:30:22.872329+0000","last_clean":"2024-03-06T07:30:22.872329+0000","last_became_active":"2024-03-06T07:30:21.661923+0000","last_became_peered":"2024-03-06T07:30:21.661923+0000","last_unstale":"2024-03-06T07:30:22.872329+0000","last_undegraded":"2024-03-06T07:30:22.872329+0000","last_fullsized":"2024-03-06T07:30:22.872329+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:30:55.049559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":22,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872200+0000","last_change":"2024-03-06T07:30:20.624267+0000","last_active":"2024-03-06T07:30:22.872200+0000","last_peered":"2024-03-06T07:30:22.872200+0000","last_clean":"2024-03-06T07:30:22.872200+0000","last_became_active":"2024-03-06T07:30:20.623498+0000","last_became_peered":"2024-03-06T07:30:20.623498+0000","last_unstale":"2024-03-06T07:30:22.872200+0000","last_undegraded":"2024-03-06T07:30:22.872200+0000","last_fullsized":"2024-03-06T07:30:22.872200+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:49.009758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873141+0000","last_change":"2024-03-06T07:30:21.664390+0000","last_active":"2024-03-06T07:30:22.873141+0000","last_peered":"2024-03-06T07:30:22.873141+0000","last_clean":"2024-03-06T07:30:22.873141+0000","last_became_active":"2024-03-06T07:30:21.664123+0000","last_became_peered":"2024-03-06T07:30:21.664123+0000","last_unstale":"2024-03-06T07:30:22.873141+0000","last_undegraded":"2024-03-06T07:30:22.873141+0000","last_fullsized":"2024-03-06T07:30:22.873141+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:49.898109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.651416+0000","last_change":"2024-03-06T07:30:20.625469+0000","last_active":"2024-03-06T07:30:22.651416+0000","last_peered":"2024-03-06T07:30:22.651416+0000","last_clean":"2024-03-06T07:30:22.651416+0000","last_became_active":"2024-03-06T07:30:20.625169+0000","last_became_peered":"2024-03-06T07:30:20.625169+0000","last_unstale":"2024-03-06T07:30:22.651416+0000","last_undegraded":"2024-03-06T07:30:22.651416+0000","last_fullsized":"2024-03-06T07:30:22.651416+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:44.400904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871404+0000","last_change":"2024-03-06T07:30:21.668360+0000","last_active":"2024-03-06T07:30:22.871404+0000","last_peered":"2024-03-06T07:30:22.871404+0000","last_clean":"2024-03-06T07:30:22.871404+0000","last_became_active":"2024-03-06T07:30:21.661641+0000","last_became_peered":"2024-03-06T07:30:21.661641+0000","last_unstale":"2024-03-06T07:30:22.871404+0000","last_undegraded":"2024-03-06T07:30:22.871404+0000","last_fullsized":"2024-03-06T07:30:22.871404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:15.895332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655180+0000","last_change":"2024-03-06T07:30:20.619107+0000","last_active":"2024-03-06T07:30:22.655180+0000","last_peered":"2024-03-06T07:30:22.655180+0000","last_clean":"2024-03-06T07:30:22.655180+0000","last_became_active":"2024-03-06T07:30:20.618101+0000","last_became_peered":"2024-03-06T07:30:20.618101+0000","last_unstale":"2024-03-06T07:30:22.655180+0000","last_undegraded":"2024-03-06T07:30:22.655180+0000","last_fullsized":"2024-03-06T07:30:22.655180+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:20.538163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649359+0000","last_change":"2024-03-06T07:30:21.657575+0000","last_active":"2024-03-06T07:30:22.649359+0000","last_peered":"2024-03-06T07:30:22.649359+0000","last_clean":"2024-03-06T07:30:22.649359+0000","last_became_active":"2024-03-06T07:30:21.657281+0000","last_became_peered":"2024-03-06T07:30:21.657281+0000","last_unstale":"2024-03-06T07:30:22.649359+0000","last_undegraded":"2024-03-06T07:30:22.649359+0000","last_fullsized":"2024-03-06T07:30:22.649359+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:42.061785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872218+0000","last_change":"2024-03-06T07:30:20.626055+0000","last_active":"2024-03-06T07:30:22.872218+0000","last_peered":"2024-03-06T07:30:22.872218+0000","last_clean":"2024-03-06T07:30:22.872218+0000","last_became_active":"2024-03-06T07:30:20.625631+0000","last_became_peered":"2024-03-06T07:30:20.625631+0000","last_unstale":"2024-03-06T07:30:22.872218+0000","last_undegraded":"2024-03-06T07:30:22.872218+0000","last_fullsized":"2024-03-06T07:30:22.872218+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:14:30.885582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.874173+0000","last_change":"2024-03-06T07:30:21.673813+0000","last_active":"2024-03-06T07:30:22.874173+0000","last_peered":"2024-03-06T07:30:22.874173+0000","last_clean":"2024-03-06T07:30:22.874173+0000","last_became_active":"2024-03-06T07:30:21.672892+0000","last_became_peered":"2024-03-06T07:30:21.672892+0000","last_unstale":"2024-03-06T07:30:22.874173+0000","last_undegraded":"2024-03-06T07:30:22.874173+0000","last_fullsized":"2024-03-06T07:30:22.874173+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:07.586968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653868+0000","last_change":"2024-03-06T07:30:20.631068+0000","last_active":"2024-03-06T07:30:22.653868+0000","last_peered":"2024-03-06T07:30:22.653868+0000","last_clean":"2024-03-06T07:30:22.653868+0000","last_became_active":"2024-03-06T07:30:20.630723+0000","last_became_peered":"2024-03-06T07:30:20.630723+0000","last_unstale":"2024-03-06T07:30:22.653868+0000","last_undegraded":"2024-03-06T07:30:22.653868+0000","last_fullsized":"2024-03-06T07:30:22.653868+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:17.369684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871211+0000","last_change":"2024-03-06T07:30:21.669653+0000","last_active":"2024-03-06T07:30:22.871211+0000","last_peered":"2024-03-06T07:30:22.871211+0000","last_clean":"2024-03-06T07:30:22.871211+0000","last_became_active":"2024-03-06T07:30:21.669360+0000","last_became_peered":"2024-03-06T07:30:21.669360+0000","last_unstale":"2024-03-06T07:30:22.871211+0000","last_undegraded":"2024-03-06T07:30:22.871211+0000","last_fullsized":"2024-03-06T07:30:22.871211+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:38.749441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649397+0000","last_change":"2024-03-06T07:30:20.630056+0000","last_active":"2024-03-06T07:30:22.649397+0000","last_peered":"2024-03-06T07:30:22.649397+0000","last_clean":"2024-03-06T07:30:22.649397+0000","last_became_active":"2024-03-06T07:30:20.629761+0000","last_became_peered":"2024-03-06T07:30:20.629761+0000","last_unstale":"2024-03-06T07:30:22.649397+0000","last_undegraded":"2024-03-06T07:30:22.649397+0000","last_fullsized":"2024-03-06T07:30:22.649397+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:42.666603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653790+0000","last_change":"2024-03-06T07:30:21.657934+0000","last_active":"2024-03-06T07:30:22.653790+0000","last_peered":"2024-03-06T07:30:22.653790+0000","last_clean":"2024-03-06T07:30:22.653790+0000","last_became_active":"2024-03-06T07:30:21.657632+0000","last_became_peered":"2024-03-06T07:30:21.657632+0000","last_unstale":"2024-03-06T07:30:22.653790+0000","last_undegraded":"2024-03-06T07:30:22.653790+0000","last_fullsized":"2024-03-06T07:30:22.653790+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:33.363992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646093+0000","last_change":"2024-03-06T07:30:20.618404+0000","last_active":"2024-03-06T07:30:22.646093+0000","last_peered":"2024-03-06T07:30:22.646093+0000","last_clean":"2024-03-06T07:30:22.646093+0000","last_became_active":"2024-03-06T07:30:20.617575+0000","last_became_peered":"2024-03-06T07:30:20.617575+0000","last_unstale":"2024-03-06T07:30:22.646093+0000","last_undegraded":"2024-03-06T07:30:22.646093+0000","last_fullsized":"2024-03-06T07:30:22.646093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:22:03.599842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.653559+0000","last_change":"2024-03-06T07:30:21.653592+0000","last_active":"2024-03-06T07:30:22.653559+0000","last_peered":"2024-03-06T07:30:22.653559+0000","last_clean":"2024-03-06T07:30:22.653559+0000","last_became_active":"2024-03-06T07:30:21.653210+0000","last_became_peered":"2024-03-06T07:30:21.653210+0000","last_unstale":"2024-03-06T07:30:22.653559+0000","last_undegraded":"2024-03-06T07:30:22.653559+0000","last_fullsized":"2024-03-06T07:30:22.653559+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:57.655078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.870590+0000","last_change":"2024-03-06T07:30:20.633615+0000","last_active":"2024-03-06T07:30:22.870590+0000","last_peered":"2024-03-06T07:30:22.870590+0000","last_clean":"2024-03-06T07:30:22.870590+0000","last_became_active":"2024-03-06T07:30:20.633225+0000","last_became_peered":"2024-03-06T07:30:20.633225+0000","last_unstale":"2024-03-06T07:30:22.870590+0000","last_undegraded":"2024-03-06T07:30:22.870590+0000","last_fullsized":"2024-03-06T07:30:22.870590+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:01.989937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871660+0000","last_change":"2024-03-06T07:30:21.663541+0000","last_active":"2024-03-06T07:30:22.871660+0000","last_peered":"2024-03-06T07:30:22.871660+0000","last_clean":"2024-03-06T07:30:22.871660+0000","last_became_active":"2024-03-06T07:30:21.663217+0000","last_became_peered":"2024-03-06T07:30:21.663217+0000","last_unstale":"2024-03-06T07:30:22.871660+0000","last_undegraded":"2024-03-06T07:30:22.871660+0000","last_fullsized":"2024-03-06T07:30:22.871660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:24:57.655078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646571+0000","last_change":"2024-03-06T07:30:20.624699+0000","last_active":"2024-03-06T07:30:22.646571+0000","last_peered":"2024-03-06T07:30:22.646571+0000","last_clean":"2024-03-06T07:30:22.646571+0000","last_became_active":"2024-03-06T07:30:20.622946+0000","last_became_peered":"2024-03-06T07:30:20.622946+0000","last_unstale":"2024-03-06T07:30:22.646571+0000","last_undegraded":"2024-03-06T07:30:22.646571+0000","last_fullsized":"2024-03-06T07:30:22.646571+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:50.448937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872837+0000","last_change":"2024-03-06T07:30:21.669496+0000","last_active":"2024-03-06T07:30:22.872837+0000","last_peered":"2024-03-06T07:30:22.872837+0000","last_clean":"2024-03-06T07:30:22.872837+0000","last_became_active":"2024-03-06T07:30:21.669162+0000","last_became_peered":"2024-03-06T07:30:21.669162+0000","last_unstale":"2024-03-06T07:30:22.872837+0000","last_undegraded":"2024-03-06T07:30:22.872837+0000","last_fullsized":"2024-03-06T07:30:22.872837+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:45.412635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.655321+0000","last_change":"2024-03-06T07:30:20.629443+0000","last_active":"2024-03-06T07:30:22.655321+0000","last_peered":"2024-03-06T07:30:22.655321+0000","last_clean":"2024-03-06T07:30:22.655321+0000","last_became_active":"2024-03-06T07:30:20.629144+0000","last_became_peered":"2024-03-06T07:30:20.629144+0000","last_unstale":"2024-03-06T07:30:22.655321+0000","last_undegraded":"2024-03-06T07:30:22.655321+0000","last_fullsized":"2024-03-06T07:30:22.655321+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:48:24.918578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646483+0000","last_change":"2024-03-06T07:30:21.650700+0000","last_active":"2024-03-06T07:30:22.646483+0000","last_peered":"2024-03-06T07:30:22.646483+0000","last_clean":"2024-03-06T07:30:22.646483+0000","last_became_active":"2024-03-06T07:30:21.650363+0000","last_became_peered":"2024-03-06T07:30:21.650363+0000","last_unstale":"2024-03-06T07:30:22.646483+0000","last_undegraded":"2024-03-06T07:30:22.646483+0000","last_fullsized":"2024-03-06T07:30:22.646483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:50.021489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660539+0000","last_change":"2024-03-06T07:30:20.629663+0000","last_active":"2024-03-06T07:30:22.660539+0000","last_peered":"2024-03-06T07:30:22.660539+0000","last_clean":"2024-03-06T07:30:22.660539+0000","last_became_active":"2024-03-06T07:30:20.629365+0000","last_became_peered":"2024-03-06T07:30:20.629365+0000","last_unstale":"2024-03-06T07:30:22.660539+0000","last_undegraded":"2024-03-06T07:30:22.660539+0000","last_fullsized":"2024-03-06T07:30:22.660539+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:10.578751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.654090+0000","last_change":"2024-03-06T07:30:21.663132+0000","last_active":"2024-03-06T07:30:22.654090+0000","last_peered":"2024-03-06T07:30:22.654090+0000","last_clean":"2024-03-06T07:30:22.654090+0000","last_became_active":"2024-03-06T07:30:21.662831+0000","last_became_peered":"2024-03-06T07:30:21.662831+0000","last_unstale":"2024-03-06T07:30:22.654090+0000","last_undegraded":"2024-03-06T07:30:22.654090+0000","last_fullsized":"2024-03-06T07:30:22.654090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:01.534240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.871902+0000","last_change":"2024-03-06T07:30:20.631439+0000","last_active":"2024-03-06T07:30:22.871902+0000","last_peered":"2024-03-06T07:30:22.871902+0000","last_clean":"2024-03-06T07:30:22.871902+0000","last_became_active":"2024-03-06T07:30:20.631100+0000","last_became_peered":"2024-03-06T07:30:20.631100+0000","last_unstale":"2024-03-06T07:30:22.871902+0000","last_undegraded":"2024-03-06T07:30:22.871902+0000","last_fullsized":"2024-03-06T07:30:22.871902+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:17.369684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.649483+0000","last_change":"2024-03-06T07:30:21.645566+0000","last_active":"2024-03-06T07:30:22.649483+0000","last_peered":"2024-03-06T07:30:22.649483+0000","last_clean":"2024-03-06T07:30:22.649483+0000","last_became_active":"2024-03-06T07:30:21.645268+0000","last_became_peered":"2024-03-06T07:30:21.645268+0000","last_unstale":"2024-03-06T07:30:22.649483+0000","last_undegraded":"2024-03-06T07:30:22.649483+0000","last_fullsized":"2024-03-06T07:30:22.649483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:55.464961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.646076+0000","last_change":"2024-03-06T07:30:20.623625+0000","last_active":"2024-03-06T07:30:22.646076+0000","last_peered":"2024-03-06T07:30:22.646076+0000","last_clean":"2024-03-06T07:30:22.646076+0000","last_became_active":"2024-03-06T07:30:20.623355+0000","last_became_peered":"2024-03-06T07:30:20.623355+0000","last_unstale":"2024-03-06T07:30:22.646076+0000","last_undegraded":"2024-03-06T07:30:22.646076+0000","last_fullsized":"2024-03-06T07:30:22.646076+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:46.004625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.873708+0000","last_change":"2024-03-06T07:30:21.661152+0000","last_active":"2024-03-06T07:30:22.873708+0000","last_peered":"2024-03-06T07:30:22.873708+0000","last_clean":"2024-03-06T07:30:22.873708+0000","last_became_active":"2024-03-06T07:30:21.658623+0000","last_became_peered":"2024-03-06T07:30:21.658623+0000","last_unstale":"2024-03-06T07:30:22.873708+0000","last_undegraded":"2024-03-06T07:30:22.873708+0000","last_fullsized":"2024-03-06T07:30:22.873708+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:34.085573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.869510+0000","last_change":"2024-03-06T07:30:20.632460+0000","last_active":"2024-03-06T07:30:22.869510+0000","last_peered":"2024-03-06T07:30:22.869510+0000","last_clean":"2024-03-06T07:30:22.869510+0000","last_became_active":"2024-03-06T07:30:20.630675+0000","last_became_peered":"2024-03-06T07:30:20.630675+0000","last_unstale":"2024-03-06T07:30:22.869510+0000","last_undegraded":"2024-03-06T07:30:22.869510+0000","last_fullsized":"2024-03-06T07:30:22.869510+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:00:04.820347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.645875+0000","last_change":"2024-03-06T07:30:21.661218+0000","last_active":"2024-03-06T07:30:22.645875+0000","last_peered":"2024-03-06T07:30:22.645875+0000","last_clean":"2024-03-06T07:30:22.645875+0000","last_became_active":"2024-03-06T07:30:21.660549+0000","last_became_peered":"2024-03-06T07:30:21.660549+0000","last_unstale":"2024-03-06T07:30:22.645875+0000","last_undegraded":"2024-03-06T07:30:22.645875+0000","last_fullsized":"2024-03-06T07:30:22.645875+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:43:12.650544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.645965+0000","last_change":"2024-03-06T07:30:20.609520+0000","last_active":"2024-03-06T07:30:22.645965+0000","last_peered":"2024-03-06T07:30:22.645965+0000","last_clean":"2024-03-06T07:30:22.645965+0000","last_became_active":"2024-03-06T07:30:20.608944+0000","last_became_peered":"2024-03-06T07:30:20.608944+0000","last_unstale":"2024-03-06T07:30:22.645965+0000","last_undegraded":"2024-03-06T07:30:22.645965+0000","last_fullsized":"2024-03-06T07:30:22.645965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:58:54.924862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648737+0000","last_change":"2024-03-06T07:30:21.655553+0000","last_active":"2024-03-06T07:30:22.648737+0000","last_peered":"2024-03-06T07:30:22.648737+0000","last_clean":"2024-03-06T07:30:22.648737+0000","last_became_active":"2024-03-06T07:30:21.654167+0000","last_became_peered":"2024-03-06T07:30:21.654167+0000","last_unstale":"2024-03-06T07:30:22.648737+0000","last_undegraded":"2024-03-06T07:30:22.648737+0000","last_fullsized":"2024-03-06T07:30:22.648737+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:41.993505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648778+0000","last_change":"2024-03-06T07:30:20.624382+0000","last_active":"2024-03-06T07:30:22.648778+0000","last_peered":"2024-03-06T07:30:22.648778+0000","last_clean":"2024-03-06T07:30:22.648778+0000","last_became_active":"2024-03-06T07:30:20.624100+0000","last_became_peered":"2024-03-06T07:30:20.624100+0000","last_unstale":"2024-03-06T07:30:22.648778+0000","last_undegraded":"2024-03-06T07:30:22.648778+0000","last_fullsized":"2024-03-06T07:30:22.648778+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:50.544775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.648675+0000","last_change":"2024-03-06T07:30:21.639012+0000","last_active":"2024-03-06T07:30:22.648675+0000","last_peered":"2024-03-06T07:30:22.648675+0000","last_clean":"2024-03-06T07:30:22.648675+0000","last_became_active":"2024-03-06T07:30:21.638696+0000","last_became_peered":"2024-03-06T07:30:21.638696+0000","last_unstale":"2024-03-06T07:30:22.648675+0000","last_undegraded":"2024-03-06T07:30:22.648675+0000","last_fullsized":"2024-03-06T07:30:22.648675+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:20.586658+0000","last_clean_scrub_stamp":"2024-03-06T07:30:20.586658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:50.623002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.660271+0000","last_change":"2024-03-06T07:30:20.624654+0000","last_active":"2024-03-06T07:30:22.660271+0000","last_peered":"2024-03-06T07:30:22.660271+0000","last_clean":"2024-03-06T07:30:22.660271+0000","last_became_active":"2024-03-06T07:30:20.624112+0000","last_became_peered":"2024-03-06T07:30:20.624112+0000","last_unstale":"2024-03-06T07:30:22.660271+0000","last_undegraded":"2024-03-06T07:30:22.660271+0000","last_fullsized":"2024-03-06T07:30:22.660271+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:30:19.574927+0000","last_clean_scrub_stamp":"2024-03-06T07:30:19.574927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:55:56.720798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":236,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:30:22.872608+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:30:22.872608+0000","last_peered":"2024-03-06T07:30:22.872608+0000","last_clean":"2024-03-06T07:30:22.872608+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:30:22.872608+0000","last_undegraded":"2024-03-06T07:30:22.872608+0000","last_fullsized":"2024-03-06T07:30:22.872608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509480,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":696320,"data_stored":285563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":285597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":876035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":688128,"data_stored":285507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542185,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":286159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":700416,"data_stored":287119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":285563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1298432,"data_stored":878069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:30:31.816 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:30:31.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:30:32.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.078+0000 7f15e0c00640 1 Processor -- start 2024-03-06T07:30:32.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.079+0000 7f15e0c00640 1 -- start start 2024-03-06T07:30:32.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.080+0000 7f15e0c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15dc15ba20 0x7f15dc066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.080+0000 7f15e0c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f15dc0ce370 0x7f15dc0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.080+0000 7f15e0c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc15be00 con 0x7f15dc0afab0 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.080+0000 7f15e0c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15dc06f2c0 con 0x7f15dc15ba20 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15e0c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15dc0cece0 con 0x7f15dc0ce370 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15dc15ba20 0x7f15dc066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f15dc15ba20 0x7f15dc066d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52976/0 (socket says 172.21.15.6:52976) 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 -- 172.21.15.6:0/1589085392 learned_addr learned my addr 172.21.15.6:0/1589085392 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15dbe00640 1 --2- 172.21.15.6:0/1589085392 >> v2:172.21.15.178:3300/0 conn(0x7f15dc0ce370 0x7f15dc0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 -- 172.21.15.6:0/1589085392 >> v2:172.21.15.178:3300/0 conn(0x7f15dc0ce370 msgr2=0x7f15dc0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 --2- 172.21.15.6:0/1589085392 >> v2:172.21.15.178:3300/0 conn(0x7f15dc0ce370 0x7f15dc0ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 -- 172.21.15.6:0/1589085392 >> v1:172.21.15.178:6789/0 conn(0x7f15dc0afab0 legacy=0x7f15dc15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15db400640 1 -- 172.21.15.6:0/1589085392 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc06d4b0 con 0x7f15dc15ba20 2024-03-06T07:30:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.081+0000 7f15dbe00640 1 --2- 172.21.15.6:0/1589085392 >> v2:172.21.15.178:3300/0 conn(0x7f15dc0ce370 0x7f15dc0ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.083+0000 7f15db400640 1 --2- 172.21.15.6:0/1589085392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc066d00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f15dc06f440 tx=0x7f15cc02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774c9a20af053328 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.083+0000 7f15da000640 1 -- 172.21.15.6:0/1589085392 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc0317d0 con 0x7f15dc15ba20 2024-03-06T07:30:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.083+0000 7f15da000640 1 -- 172.21.15.6:0/1589085392 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15cc037070 con 0x7f15dc15ba20 2024-03-06T07:30:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.083+0000 7f15da000640 1 -- 172.21.15.6:0/1589085392 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc031b90 con 0x7f15dc15ba20 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 -- 172.21.15.6:0/1589085392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 msgr2=0x7f15dc066d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 --2- 172.21.15.6:0/1589085392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc066d00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f15dc06f440 tx=0x7f15cc02b870 comp rx=0 tx=0).stop 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 -- 172.21.15.6:0/1589085392 shutdown_connections 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 --2- 172.21.15.6:0/1589085392 >> v2:172.21.15.178:3300/0 conn(0x7f15dc0ce370 0x7f15dc0ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 --2- 172.21.15.6:0/1589085392 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc066d00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 -- 172.21.15.6:0/1589085392 >> 172.21.15.6:0/1589085392 conn(0x7f15dc0ccf70 msgr2=0x7f15dc0cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 -- 172.21.15.6:0/1589085392 shutdown_connections 2024-03-06T07:30:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.084+0000 7f15e0c00640 1 -- 172.21.15.6:0/1589085392 wait complete. 2024-03-06T07:30:32.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.086+0000 7f15e0c00640 1 Processor -- start 2024-03-06T07:30:32.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.086+0000 7f15e0c00640 1 -- start start 2024-03-06T07:30:32.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15e0c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f15dc0afab0 0x7f15dc13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15e0c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15dc0ce370 0x7f15dc0b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15e0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15e0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc15f360 con 0x7f15dc15ba20 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15e0c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc15e730 con 0x7f15dc0ce370 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15e0c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc15e420 con 0x7f15dc0afab0 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15dbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52990/0 (socket says 172.21.15.6:52990) 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15dbe00640 1 -- 172.21.15.6:0/180278564 learned_addr learned my addr 172.21.15.6:0/180278564 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15db400640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f15dc0afab0 0x7f15dc13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.087+0000 7f15daa00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15dc0ce370 0x7f15dc0b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15dbe00640 1 -- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f15dc0afab0 msgr2=0x7f15dc13f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15dbe00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f15dc0afab0 0x7f15dc13f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15dbe00640 1 -- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15dc0ce370 msgr2=0x7f15dc0b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15dbe00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15dc0ce370 0x7f15dc0b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15dbe00640 1 -- 172.21.15.6:0/180278564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0b0250 con 0x7f15dc15ba20 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15db400640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f15dc0afab0 0x7f15dc13f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15dbe00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc141f20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f15d400cef0 tx=0x7f15d400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.088+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d401a070 con 0x7f15dc15ba20 2024-03-06T07:30:32.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.089+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d400d860 con 0x7f15dc15ba20 2024-03-06T07:30:32.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.089+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d400dc50 con 0x7f15dc15ba20 2024-03-06T07:30:32.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.089+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc0b0530 con 0x7f15dc15ba20 2024-03-06T07:30:32.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.089+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc0b0af0 con 0x7f15dc15ba20 2024-03-06T07:30:32.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.090+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15dc04f330 con 0x7f15dc15ba20 2024-03-06T07:30:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.091+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15d401d070 con 0x7f15dc15ba20 2024-03-06T07:30:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.092+0000 7f15d3400640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f15bc06b4d0 0x7f15bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.093+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15d408ab70 con 0x7f15dc15ba20 2024-03-06T07:30:32.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.093+0000 7f15db400640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f15bc06b4d0 0x7f15bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.094+0000 7f15db400640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f15bc06b4d0 0x7f15bc06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f15cc036000 tx=0x7f15cc03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.097+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15d4059a10 con 0x7f15dc15ba20 2024-03-06T07:30:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.376+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f15dc067370 con 0x7f15dc15ba20 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.380+0000 7f15d3400640 1 -- 172.21.15.6:0/180278564 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f15d4059310 con 0x7f15dc15ba20 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:30:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:30:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f15bc06b4d0 msgr2=0x7f15bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f15bc06b4d0 0x7f15bc06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f15cc036000 tx=0x7f15cc03d040 comp rx=0 tx=0).stop 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 msgr2=0x7f15dc141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc141f20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f15d400cef0 tx=0x7f15d400a710 comp rx=0 tx=0).stop 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 shutdown_connections 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f15bc06b4d0 0x7f15bc06d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15dc0ce370 0x7f15dc0b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f15dc15ba20 0x7f15dc141f20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 --2- 172.21.15.6:0/180278564 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f15dc0afab0 0x7f15dc13f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 >> 172.21.15.6:0/180278564 conn(0x7f15dc0ccf70 msgr2=0x7f15dc0cee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 shutdown_connections 2024-03-06T07:30:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.383+0000 7f15e0c00640 1 -- 172.21.15.6:0/180278564 wait complete. 2024-03-06T07:30:32.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:32.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.616+0000 7f6b3e800640 1 Processor -- start 2024-03-06T07:30:32.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.616+0000 7f6b3e800640 1 -- start start 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3e800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6b38070f10 0x7f6b380cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3e800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6b380cce90 0x7f6b380cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3e800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380cd300 con 0x7f6b38070f10 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3e800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380714d0 con 0x7f6b380cce90 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3e800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380717b0 con 0x7f6b38145300 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6b38070f10 0x7f6b380cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6b38070f10 0x7f6b380cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53004/0 (socket says 172.21.15.6:53004) 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.618+0000 7f6b3ca00640 1 -- 172.21.15.6:0/3523773871 learned_addr learned my addr 172.21.15.6:0/3523773871 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:32.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3de00640 1 --2- 172.21.15.6:0/3523773871 >> v2:172.21.15.178:3300/0 conn(0x7f6b380cce90 0x7f6b380cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3d400640 1 --2- 172.21.15.6:0/3523773871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3ca00640 1 -- 172.21.15.6:0/3523773871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 msgr2=0x7f6b380b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3ca00640 1 --2- 172.21.15.6:0/3523773871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3ca00640 1 -- 172.21.15.6:0/3523773871 >> v2:172.21.15.178:3300/0 conn(0x7f6b380cce90 msgr2=0x7f6b380cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3ca00640 1 --2- 172.21.15.6:0/3523773871 >> v2:172.21.15.178:3300/0 conn(0x7f6b380cce90 0x7f6b380cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.619+0000 7f6b3ca00640 1 -- 172.21.15.6:0/3523773871 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b38071a90 con 0x7f6b38070f10 2024-03-06T07:30:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.620+0000 7f6b3ca00640 1 --2- 172.21.15.6:0/3523773871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b38070f10 0x7f6b380cc950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6b38072420 tx=0x7f6b2402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5949f81c2959afed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.620+0000 7f6b37e00640 1 -- 172.21.15.6:0/3523773871 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24035020 con 0x7f6b38070f10 2024-03-06T07:30:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.620+0000 7f6b37e00640 1 -- 172.21.15.6:0/3523773871 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2402d040 con 0x7f6b38070f10 2024-03-06T07:30:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.620+0000 7f6b37e00640 1 -- 172.21.15.6:0/3523773871 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24031a80 con 0x7f6b38070f10 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 -- 172.21.15.6:0/3523773871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b38070f10 msgr2=0x7f6b380cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 --2- 172.21.15.6:0/3523773871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b38070f10 0x7f6b380cc950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6b38072420 tx=0x7f6b2402b900 comp rx=0 tx=0).stop 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 -- 172.21.15.6:0/3523773871 shutdown_connections 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 --2- 172.21.15.6:0/3523773871 >> v2:172.21.15.178:3300/0 conn(0x7f6b380cce90 0x7f6b380cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 --2- 172.21.15.6:0/3523773871 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b38070f10 0x7f6b380cc950 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 --2- 172.21.15.6:0/3523773871 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 -- 172.21.15.6:0/3523773871 >> 172.21.15.6:0/3523773871 conn(0x7f6b3806b3d0 msgr2=0x7f6b3806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 -- 172.21.15.6:0/3523773871 shutdown_connections 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.621+0000 7f6b3e800640 1 -- 172.21.15.6:0/3523773871 wait complete. 2024-03-06T07:30:32.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.623+0000 7f6b3e800640 1 Processor -- start 2024-03-06T07:30:32.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.623+0000 7f6b3e800640 1 -- start start 2024-03-06T07:30:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3e800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b38070f10 0x7f6b3815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 0x7f6b381e5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3e800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b381456e0 con 0x7f6b380cce90 2024-03-06T07:30:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3e800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380b4ee0 con 0x7f6b38070f10 2024-03-06T07:30:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3e800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380cde70 con 0x7f6b38145300 2024-03-06T07:30:32.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 0x7f6b381e5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 0x7f6b381e5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53012/0 (socket says 172.21.15.6:53012) 2024-03-06T07:30:32.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.624+0000 7f6b3ca00640 1 -- 172.21.15.6:0/1777549333 learned_addr learned my addr 172.21.15.6:0/1777549333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3d400640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b38070f10 0x7f6b3815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3ca00640 1 -- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 msgr2=0x7f6b380c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3ca00640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3ca00640 1 -- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b38070f10 msgr2=0x7f6b3815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3ca00640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b38070f10 0x7f6b3815b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3ca00640 1 -- 172.21.15.6:0/1777549333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b381e5ee0 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3d400640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b38070f10 0x7f6b3815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3ca00640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 0x7f6b381e5490 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6b24000c00 tx=0x7f6b240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24040070 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b24039bf0 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24035040 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.625+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b381e61c0 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.626+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b381e6780 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.627+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3804f330 con 0x7f6b380cce90 2024-03-06T07:30:32.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.628+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b24038020 con 0x7f6b380cce90 2024-03-06T07:30:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.629+0000 7f6b35600640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6b1806b520 0x7f6b1806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.629+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b240abe30 con 0x7f6b380cce90 2024-03-06T07:30:32.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.629+0000 7f6b3d400640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6b1806b520 0x7f6b1806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.630+0000 7f6b3d400640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6b1806b520 0x7f6b1806d9e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006fd0 tx=0x7f6b2c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.638+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b2407ac90 con 0x7f6b380cce90 2024-03-06T07:30:32.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.878+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b3815be10 con 0x7f6b380cce90 2024-03-06T07:30:32.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.879+0000 7f6b35600640 1 -- 172.21.15.6:0/1777549333 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f6b2407a590 con 0x7f6b380cce90 2024-03-06T07:30:32.882 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:32.882 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:30:32.882 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6b1806b520 msgr2=0x7f6b1806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6b1806b520 0x7f6b1806d9e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006fd0 tx=0x7f6b2c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 msgr2=0x7f6b381e5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 0x7f6b381e5490 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6b24000c00 tx=0x7f6b240046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 shutdown_connections 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6b1806b520 0x7f6b1806d9e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6b38145300 0x7f6b380c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6b380cce90 0x7f6b381e5490 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 --2- 172.21.15.6:0/1777549333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b38070f10 0x7f6b3815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 >> 172.21.15.6:0/1777549333 conn(0x7f6b3806b3d0 msgr2=0x7f6b380b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 shutdown_connections 2024-03-06T07:30:32.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:32.883+0000 7f6b3e800640 1 -- 172.21.15.6:0/1777549333 wait complete. 2024-03-06T07:30:33.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:30:21.089049+0000', 'modified': '2024-03-06T07:30:24.129217+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5955}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5955': {'gid': 5955, 'name': 'a', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6835/1578886876', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 1578886876}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 1578886876}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5955, 'qdb_cluster': [5955]} 2024-03-06T07:30:33.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:30:33.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:30:33.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.535+0000 7ff3c5200640 1 Processor -- start 2024-03-06T07:30:33.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.536+0000 7ff3c5200640 1 -- start start 2024-03-06T07:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3c5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3c015bc60 0x7ff3c00af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3c5200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff3c01600c0 0x7ff3c00af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3c5200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00b0690 con 0x7ff3c015c040 2024-03-06T07:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3c5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00afe10 con 0x7ff3c015bc60 2024-03-06T07:30:33.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3c5200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00b00d0 con 0x7ff3c01600c0 2024-03-06T07:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3bde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3c015bc60 0x7ff3c00af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.537+0000 7ff3bde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3c015bc60 0x7ff3c00af2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53020/0 (socket says 172.21.15.6:53020) 2024-03-06T07:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.538+0000 7ff3bde00640 1 -- 172.21.15.6:0/4202427198 learned_addr learned my addr 172.21.15.6:0/4202427198 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.538+0000 7ff3be800640 1 --2- 172.21.15.6:0/4202427198 >> v2:172.21.15.178:3300/0 conn(0x7ff3c01600c0 0x7ff3c00af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.538+0000 7ff3bde00640 1 -- 172.21.15.6:0/4202427198 >> v2:172.21.15.178:3300/0 conn(0x7ff3c01600c0 msgr2=0x7ff3c00af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.538+0000 7ff3bde00640 1 --2- 172.21.15.6:0/4202427198 >> v2:172.21.15.178:3300/0 conn(0x7ff3c01600c0 0x7ff3c00af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.538+0000 7ff3bde00640 1 -- 172.21.15.6:0/4202427198 >> v1:172.21.15.178:6789/0 conn(0x7ff3c015c040 legacy=0x7ff3c015f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.538+0000 7ff3bde00640 1 -- 172.21.15.6:0/4202427198 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00b03b0 con 0x7ff3c015bc60 2024-03-06T07:30:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.541+0000 7ff3bde00640 1 --2- 172.21.15.6:0/4202427198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c015bc60 0x7ff3c00af2b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff3c015c440 tx=0x7ff3b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e376021323300df2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.541+0000 7ff3bca00640 1 -- 172.21.15.6:0/4202427198 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0035020 con 0x7ff3c015bc60 2024-03-06T07:30:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.541+0000 7ff3bca00640 1 -- 172.21.15.6:0/4202427198 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b0031870 con 0x7ff3c015bc60 2024-03-06T07:30:33.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.541+0000 7ff3bca00640 1 -- 172.21.15.6:0/4202427198 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0031c40 con 0x7ff3c015bc60 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 -- 172.21.15.6:0/4202427198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c015bc60 msgr2=0x7ff3c00af2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 --2- 172.21.15.6:0/4202427198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c015bc60 0x7ff3c00af2b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff3c015c440 tx=0x7ff3b002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 -- 172.21.15.6:0/4202427198 shutdown_connections 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 --2- 172.21.15.6:0/4202427198 >> v2:172.21.15.178:3300/0 conn(0x7ff3c01600c0 0x7ff3c00af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 --2- 172.21.15.6:0/4202427198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c015bc60 0x7ff3c00af2b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 -- 172.21.15.6:0/4202427198 >> 172.21.15.6:0/4202427198 conn(0x7ff3c009aa10 msgr2=0x7ff3c009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 -- 172.21.15.6:0/4202427198 shutdown_connections 2024-03-06T07:30:33.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.542+0000 7ff3c5200640 1 -- 172.21.15.6:0/4202427198 wait complete. 2024-03-06T07:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.544+0000 7ff3c5200640 1 Processor -- start 2024-03-06T07:30:33.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.544+0000 7ff3c5200640 1 -- start start 2024-03-06T07:30:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.545+0000 7ff3c5200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3c015bc60 0x7ff3c01f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.545+0000 7ff3c5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3c015c040 0x7ff3c01569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.545+0000 7ff3c5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 0x7ff3c0156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.545+0000 7ff3c5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c006f6f0 con 0x7ff3c01600c0 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.545+0000 7ff3c5200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c006f570 con 0x7ff3c015bc60 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.545+0000 7ff3c5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c006f870 con 0x7ff3c015c040 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 0x7ff3c0156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 0x7ff3c0156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53032/0 (socket says 172.21.15.6:53032) 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 -- 172.21.15.6:0/1314004157 learned_addr learned my addr 172.21.15.6:0/1314004157 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3bd400640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3c015c040 0x7ff3c01569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3bde00640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3c015bc60 0x7ff3c01f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 -- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3c015c040 msgr2=0x7ff3c01569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3c015c040 0x7ff3c01569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 -- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3c015bc60 msgr2=0x7ff3c01f7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3c015bc60 0x7ff3c01f7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.546+0000 7ff3be800640 1 -- 172.21.15.6:0/1314004157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0154310 con 0x7ff3c01600c0 2024-03-06T07:30:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.547+0000 7ff3be800640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 0x7ff3c0156ef0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009d30 tx=0x7ff3b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.547+0000 7ff3b6000640 1 -- 172.21.15.6:0/1314004157 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8018020 con 0x7ff3c01600c0 2024-03-06T07:30:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.547+0000 7ff3b6000640 1 -- 172.21.15.6:0/1314004157 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b8014480 con 0x7ff3c01600c0 2024-03-06T07:30:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.547+0000 7ff3b6000640 1 -- 172.21.15.6:0/1314004157 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8014870 con 0x7ff3c01600c0 2024-03-06T07:30:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.547+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c01545f0 con 0x7ff3c01600c0 2024-03-06T07:30:33.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.547+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c0154bb0 con 0x7ff3c01600c0 2024-03-06T07:30:33.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.550+0000 7ff3b6000640 1 -- 172.21.15.6:0/1314004157 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3b8005d10 con 0x7ff3c01600c0 2024-03-06T07:30:33.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.551+0000 7ff3b6000640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3a006b4d0 0x7ff3a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.551+0000 7ff3b6000640 1 -- 172.21.15.6:0/1314004157 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3b808ac20 con 0x7ff3c01600c0 2024-03-06T07:30:33.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.551+0000 7ff3bde00640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3a006b4d0 0x7ff3a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.552+0000 7ff3c7fcd740 1 Processor -- start 2024-03-06T07:30:33.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.552+0000 7ff3c7fcd740 1 -- start start 2024-03-06T07:30:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.552+0000 7ff3bde00640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3a006b4d0 0x7ff3a006d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0036000 tx=0x7ff3b003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa42750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa72540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa42c90 0x561d9aa72a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a9ac600 con 0x561d9aa6fb90 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a955930 con 0x561d9aa6f7b0 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d9a9a86a0 con 0x561d9aa42c90 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa72540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa72540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53034/0 (socket says 172.21.15.6:53034) 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 -- 172.21.15.6:0/4112872717 learned_addr learned my addr 172.21.15.6:0/4112872717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bde00640 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa42750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3be800640 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa42c90 0x561d9aa72a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 -- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa42c90 msgr2=0x561d9aa72a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa42c90 0x561d9aa72a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 -- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 msgr2=0x561d9aa42750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa42750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.553+0000 7ff3bd400640 1 -- 172.21.15.6:0/4112872717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d9a6c12d0 con 0x561d9aa6fb90 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.554+0000 7ff3bd400640 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa72540 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0014f0 tx=0x7ff3ac0019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d08b7027a2b60852 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.554+0000 7ff3aaa00640 1 -- 172.21.15.6:0/4112872717 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac019070 con 0x561d9aa6fb90 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.554+0000 7ff3aaa00640 1 -- 172.21.15.6:0/4112872717 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac0052e0 con 0x561d9aa6fb90 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.554+0000 7ff3aaa00640 1 -- 172.21.15.6:0/4112872717 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac003810 con 0x561d9aa6fb90 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/4112872717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 msgr2=0x561d9aa72540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa72540 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0014f0 tx=0x7ff3ac0019b0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/4112872717 shutdown_connections 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa42c90 0x561d9aa72a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa72540 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/4112872717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa42750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/4112872717 >> 172.21.15.6:0/4112872717 conn(0x561d9aa449c0 msgr2=0x561d9aa8d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/4112872717 shutdown_connections 2024-03-06T07:30:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.555+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/4112872717 wait complete. 2024-03-06T07:30:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.556+0000 7ff3c7fcd740 1 -- start start 2024-03-06T07:30:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.556+0000 7ff3c7fcd740 1 Processor -- start 2024-03-06T07:30:33.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 0x561d9aa87b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6fb90 0x561d9aa880b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9aa46f60 con 0x561d9aa6f7b0 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9a968d90 con 0x561d9aa6fb90 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d9aa46350 con 0x561d9aa8d5f0 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3c7fcd740 10 client.0 _ll_register_callbacks cb 0x561d9a9a4350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3bde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 0x561d9aa87b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3bde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 0x561d9aa87b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53044/0 (socket says 172.21.15.6:53044) 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.557+0000 7ff3bde00640 1 -- 172.21.15.6:0/829490390 learned_addr learned my addr 172.21.15.6:0/829490390 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bd400640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6fb90 0x561d9aa880b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3be800640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bde00640 1 -- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 msgr2=0x561d9aa74de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bde00640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bde00640 1 -- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6fb90 msgr2=0x561d9aa880b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bde00640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6fb90 0x561d9aa880b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bde00640 1 -- 172.21.15.6:0/829490390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d9a78b6b0 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3bde00640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 0x561d9aa87b70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0000c00 tx=0x7ff3b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7dbd19601d0249 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff39f400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff39f400640 1 -- 172.21.15.6:0/829490390 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0054060 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff39f400640 1 -- 172.21.15.6:0/829490390 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3b0047d00 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.558+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3ac0052e0 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff39f400640 1 -- 172.21.15.6:0/829490390 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3b0035040 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 10 client.6108 fetch_fsmap learned FSMap version 36 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x561d9aa85c00 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff39f400640 1 -- 172.21.15.6:0/829490390 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac0052e0 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff39f400640 1 -- 172.21.15.6:0/829490390 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 36) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff3b004e030 con 0x561d9aa6f7b0 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 10 client.6108 fetch_fsmap finished waiting for FSMap version 36 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 10 client.6108 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 20 client.6108 populate_metadata read hostname 'smithi006' 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aaa1390 0x561d9aa86cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 4 client.6108 mds_command: new command op to 5955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:30:33.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.559+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 --> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d9a9b9b40 con 0x561d9aaa1390 2024-03-06T07:30:33.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.560+0000 7ff3bd400640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aaa1390 0x561d9aa86cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.560+0000 7ff3bd400640 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aaa1390 0x561d9aa86cd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72f350cebcb106fe server_cookie=c7eb52d1356a40f8 in_seq=0 out_seq=0 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.560+0000 7ff39f400640 10 client.6108 ms_handle_connect on v2:172.21.15.6:6834/1578886876 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.561+0000 7ff39f400640 1 -- 172.21.15.6:0/829490390 <== mds.0 v2:172.21.15.6:6834/1578886876 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561d9a9b9b40 con 0x561d9aaa1390 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.561+0000 7ff39f400640 10 client.6108 handle_command_reply: tid=0 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.561+0000 7ff3c7fcd740 1 client.6108 shutdown 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.561+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 msgr2=0x561d9aa87b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.561+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 0x561d9aa87b70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0000c00 tx=0x7ff3b0004730 comp rx=0 tx=0).stop 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 shutdown_connections 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aaa1390 0x561d9aa86cd0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6fb90 0x561d9aa880b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/829490390 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6f7b0 0x561d9aa87b70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff39f400640 0 client.6108 ms_handle_reset on v2:172.21.15.6:6834/1578886876 2024-03-06T07:30:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 >> 172.21.15.6:0/829490390 conn(0x561d9aa42c90 msgr2=0x561d9aa72530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 shutdown_connections 2024-03-06T07:30:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/829490390 wait complete. 2024-03-06T07:30:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.562+0000 7ff3c7fcd740 20 client.6108 trim_cache size 0 max 16384 2024-03-06T07:30:33.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.580+0000 7ff3c7fcd740 1 Processor -- start 2024-03-06T07:30:33.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.580+0000 7ff3c7fcd740 1 -- start start 2024-03-06T07:30:33.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa84890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa74660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0158f50 con 0x561d9aa6fb90 2024-03-06T07:30:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0158350 con 0x561d9aa6f7b0 2024-03-06T07:30:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c0158950 con 0x561d9aa8d5f0 2024-03-06T07:30:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3bd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa74660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.581+0000 7ff3bd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa74660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53052/0 (socket says 172.21.15.6:53052) 2024-03-06T07:30:33.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 -- 172.21.15.6:0/2323524110 learned_addr learned my addr 172.21.15.6:0/2323524110 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 -- 172.21.15.6:0/2323524110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 msgr2=0x561d9aa74ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 -- 172.21.15.6:0/2323524110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 msgr2=0x561d9aa84890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa84890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 -- 172.21.15.6:0/2323524110 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d9aa90510 con 0x561d9aa6fb90 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3bd400640 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa74660 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0014c0 tx=0x7ff3ac00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fdec34c948e5cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3aa000640 1 -- 172.21.15.6:0/2323524110 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac019040 con 0x561d9aa6fb90 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3aa000640 1 -- 172.21.15.6:0/2323524110 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac01a040 con 0x561d9aa6fb90 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.582+0000 7ff3aa000640 1 -- 172.21.15.6:0/2323524110 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac025ae0 con 0x561d9aa6fb90 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/2323524110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 msgr2=0x561d9aa74660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa74660 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0014c0 tx=0x7ff3ac00a000 comp rx=0 tx=0).stop 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/2323524110 shutdown_connections 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa8d5f0 0x561d9aa74ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa6fb90 0x561d9aa74660 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/2323524110 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aa84890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/2323524110 >> 172.21.15.6:0/2323524110 conn(0x561d9aa42c90 msgr2=0x561d9aa8da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/2323524110 shutdown_connections 2024-03-06T07:30:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.583+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/2323524110 wait complete. 2024-03-06T07:30:33.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.584+0000 7ff3c7fcd740 1 -- start start 2024-03-06T07:30:33.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.584+0000 7ff3c7fcd740 1 Processor -- start 2024-03-06T07:30:33.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aafcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa6fb90 0x561d9aa5f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 0x561d9aa83490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9aaa1210 con 0x561d9aa8d5f0 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d9aa85ee0 con 0x561d9aa6f7b0 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d9aaa1780 con 0x561d9aa6fb90 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3c7fcd740 10 client.0 _ll_register_callbacks cb 0x7ff3b80095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3be800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 0x561d9aa83490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3be800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 0x561d9aa83490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53066/0 (socket says 172.21.15.6:53066) 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.585+0000 7ff3be800640 1 -- 172.21.15.6:0/972422012 learned_addr learned my addr 172.21.15.6:0/972422012 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3bde00640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aafcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3bd400640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa6fb90 0x561d9aa5f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3be800640 1 -- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa6fb90 msgr2=0x561d9aa5f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3be800640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa6fb90 0x561d9aa5f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3be800640 1 -- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 msgr2=0x561d9aafcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3be800640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aafcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3be800640 1 -- 172.21.15.6:0/972422012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d9aab04b0 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3bd400640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa6fb90 0x561d9aa5f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3bde00640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aafcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3be800640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 0x561d9aa83490 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80590a0 tx=0x7ff3b80593a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982cd4a96dca9d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff39f400640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff39f400640 1 -- 172.21.15.6:0/972422012 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8087060 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff39f400640 1 -- 172.21.15.6:0/972422012 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3b8061620 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.586+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3ac01a040 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff39f400640 1 -- 172.21.15.6:0/972422012 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3b805d2b0 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 10 client.6114 fetch_fsmap learned FSMap version 36 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x561d9aab0e10 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff39f400640 1 -- 172.21.15.6:0/972422012 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac01a040 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff39f400640 1 -- 172.21.15.6:0/972422012 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 36) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff3b80587f0 con 0x561d9aa8d5f0 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 10 client.6114 fetch_fsmap finished waiting for FSMap version 36 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 10 client.6114 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:30:33.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 20 client.6114 populate_metadata read hostname 'smithi006' 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aa84f00 0x561d9aa6d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 4 client.6114 mds_command: new command op to 5955 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 --> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561d9a9b3e90 con 0x561d9aa84f00 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.587+0000 7ff3bde00640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aa84f00 0x561d9aa6d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff3bde00640 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aa84f00 0x561d9aa6d410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55261eca1ec7a254 server_cookie=62164ebac4bdde44 in_seq=0 out_seq=0 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff39f400640 10 client.6114 ms_handle_connect on v2:172.21.15.6:6834/1578886876 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff39f400640 1 -- 172.21.15.6:0/972422012 <== mds.0 v2:172.21.15.6:6834/1578886876 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x561d9a9b3e90 con 0x561d9aa84f00 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff39f400640 10 client.6114 handle_command_reply: tid=0 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff3c7fcd740 1 client.6114 shutdown 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 msgr2=0x561d9aa83490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.588+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 0x561d9aa83490 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80590a0 tx=0x7ff3b80593a0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 shutdown_connections 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:6834/1578886876,v1:172.21.15.6:6835/1578886876] conn(0x561d9aa84f00 0x561d9aa6d410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x561d9aa8d5f0 0x561d9aa83490 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x561d9aa6fb90 0x561d9aa5f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 --2- 172.21.15.6:0/972422012 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x561d9aa6f7b0 0x561d9aafcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff39f400640 0 client.6114 ms_handle_reset on v2:172.21.15.6:6834/1578886876 2024-03-06T07:30:33.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 >> 172.21.15.6:0/972422012 conn(0x561d9aa42c90 msgr2=0x561d9aa3d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 shutdown_connections 2024-03-06T07:30:33.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 1 -- 172.21.15.6:0/972422012 wait complete. 2024-03-06T07:30:33.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.589+0000 7ff3c7fcd740 20 client.6114 trim_cache size 0 max 16384 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 5955, 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 11.953714577, 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 36, 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 58, 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 58, 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 18.567748039000001 2024-03-06T07:30:33.595 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:30:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3a006b4d0 msgr2=0x7ff3a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3a006b4d0 0x7ff3a006d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0036000 tx=0x7ff3b003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 msgr2=0x7ff3c0156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 0x7ff3c0156ef0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009d30 tx=0x7ff3b800c490 comp rx=0 tx=0).stop 2024-03-06T07:30:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 shutdown_connections 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3a006b4d0 0x7ff3a006d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c01600c0 0x7ff3c0156ef0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3c015c040 0x7ff3c01569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 --2- 172.21.15.6:0/1314004157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3c015bc60 0x7ff3c01f7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 >> 172.21.15.6:0/1314004157 conn(0x7ff3c009aa10 msgr2=0x7ff3c006d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 shutdown_connections 2024-03-06T07:30:33.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.590+0000 7ff3c5200640 1 -- 172.21.15.6:0/1314004157 wait complete. 2024-03-06T07:30:33.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.825+0000 7f97d5c00640 1 Processor -- start 2024-03-06T07:30:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.825+0000 7f97d5c00640 1 -- start start 2024-03-06T07:30:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.826+0000 7f97d5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97d00af580 0x7f97d015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.826+0000 7f97d5c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f97d00cdc20 0x7f97d00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.826+0000 7f97d5c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d015cc40 con 0x7f97d00af1a0 2024-03-06T07:30:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.826+0000 7f97d5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d015cf20 con 0x7f97d00af580 2024-03-06T07:30:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.826+0000 7f97d5c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d015d200 con 0x7f97d00cdc20 2024-03-06T07:30:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97ce000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97d00af580 0x7f97d015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97cea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f97d00af1a0 0x7f97d00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46090/0 (socket says 172.21.15.6:46090) 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97cea00640 1 -- 172.21.15.6:0/1211997645 learned_addr learned my addr 172.21.15.6:0/1211997645 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97ce000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f97d00af580 0x7f97d015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53082/0 (socket says 172.21.15.6:53082) 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97cf400640 1 --2- 172.21.15.6:0/1211997645 >> v2:172.21.15.178:3300/0 conn(0x7f97d00cdc20 0x7f97d00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97ce000640 1 -- 172.21.15.6:0/1211997645 >> v2:172.21.15.178:3300/0 conn(0x7f97d00cdc20 msgr2=0x7f97d00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97ce000640 1 --2- 172.21.15.6:0/1211997645 >> v2:172.21.15.178:3300/0 conn(0x7f97d00cdc20 0x7f97d00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97ce000640 1 -- 172.21.15.6:0/1211997645 >> v1:172.21.15.6:6789/0 conn(0x7f97d00af1a0 legacy=0x7f97d00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97ce000640 1 -- 172.21.15.6:0/1211997645 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d015d4e0 con 0x7f97d00af580 2024-03-06T07:30:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.827+0000 7f97cd600640 1 -- 172.21.15.6:0/1211997645 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890325526 0 0) 0x7f97d015cc40 con 0x7f97d00af1a0 2024-03-06T07:30:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.828+0000 7f97ce000640 1 --2- 172.21.15.6:0/1211997645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af580 0x7f97d015c600 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f97d006c0c0 tx=0x7f97b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5a0bdf3c1ea111 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97cd600640 1 -- 172.21.15.6:0/1211997645 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b80315a0 con 0x7f97d00af580 2024-03-06T07:30:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97cd600640 1 -- 172.21.15.6:0/1211997645 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8037070 con 0x7f97d00af580 2024-03-06T07:30:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97cd600640 1 -- 172.21.15.6:0/1211997645 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8031960 con 0x7f97d00af580 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97d5c00640 1 -- 172.21.15.6:0/1211997645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af580 msgr2=0x7f97d015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1211997645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af580 0x7f97d015c600 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f97d006c0c0 tx=0x7f97b802b840 comp rx=0 tx=0).stop 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97d5c00640 1 -- 172.21.15.6:0/1211997645 shutdown_connections 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1211997645 >> v2:172.21.15.178:3300/0 conn(0x7f97d00cdc20 0x7f97d00b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1211997645 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af580 0x7f97d015c600 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.829+0000 7f97d5c00640 1 -- 172.21.15.6:0/1211997645 >> 172.21.15.6:0/1211997645 conn(0x7f97d006afb0 msgr2=0x7f97d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.830+0000 7f97d5c00640 1 -- 172.21.15.6:0/1211997645 shutdown_connections 2024-03-06T07:30:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.830+0000 7f97d5c00640 1 -- 172.21.15.6:0/1211997645 wait complete. 2024-03-06T07:30:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.831+0000 7f97d5c00640 1 Processor -- start 2024-03-06T07:30:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.831+0000 7f97d5c00640 1 -- start start 2024-03-06T07:30:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97d5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 0x7f97d006d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97d5c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97d00af580 0x7f97d015e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97d5c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97d00cdc20 0x7f97d006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97d5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00b36a0 con 0x7f97d00af1a0 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97d5c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00b3520 con 0x7f97d00af580 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97d5c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00b3820 con 0x7f97d00cdc20 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.832+0000 7f97cea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 0x7f97d006d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 0x7f97d006d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53094/0 (socket says 172.21.15.6:53094) 2024-03-06T07:30:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 -- 172.21.15.6:0/1156711698 learned_addr learned my addr 172.21.15.6:0/1156711698 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 -- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97d00cdc20 msgr2=0x7f97d006dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97d00cdc20 0x7f97d006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 -- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97d00af580 msgr2=0x7f97d015e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97d00af580 0x7f97d015e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 -- 172.21.15.6:0/1156711698 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d01f7ed0 con 0x7f97d00af1a0 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.833+0000 7f97cea00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 0x7f97d006d970 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f97c000c4d0 tx=0x7f97c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.834+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0018070 con 0x7f97d00af1a0 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.834+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c00074e0 con 0x7f97d00af1a0 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.834+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0004a20 con 0x7f97d00af1a0 2024-03-06T07:30:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.834+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d006cbb0 con 0x7f97d00af1a0 2024-03-06T07:30:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.834+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d006d170 con 0x7f97d00af1a0 2024-03-06T07:30:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.835+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d0069510 con 0x7f97d00af1a0 2024-03-06T07:30:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.836+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97c0004bc0 con 0x7f97d00af1a0 2024-03-06T07:30:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.837+0000 7f97c6a00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f97ac06b4d0 0x7f97ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.837+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97c00894e0 con 0x7f97d00af1a0 2024-03-06T07:30:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.837+0000 7f97ce000640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f97ac06b4d0 0x7f97ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:33.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.839+0000 7f97ce000640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f97ac06b4d0 0x7f97ac06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f97b8036000 tx=0x7f97b8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:33.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:33.845+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97c0058350 con 0x7f97d00af1a0 2024-03-06T07:30:34.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.087+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f97d015f050 con 0x7f97d00af1a0 2024-03-06T07:30:34.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.088+0000 7f97c6a00640 1 -- 172.21.15.6:0/1156711698 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7f97c0057c50 con 0x7f97d00af1a0 2024-03-06T07:30:34.091 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f97ac06b4d0 msgr2=0x7f97ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f97ac06b4d0 0x7f97ac06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f97b8036000 tx=0x7f97b8030000 comp rx=0 tx=0).stop 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 msgr2=0x7f97d006d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 0x7f97d006d970 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f97c000c4d0 tx=0x7f97c000c990 comp rx=0 tx=0).stop 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 shutdown_connections 2024-03-06T07:30:34.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f97ac06b4d0 0x7f97ac06d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f97d00af580 0x7f97d015e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f97d00cdc20 0x7f97d006dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 --2- 172.21.15.6:0/1156711698 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97d00af1a0 0x7f97d006d970 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 >> 172.21.15.6:0/1156711698 conn(0x7f97d006afb0 msgr2=0x7f97d0072020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 shutdown_connections 2024-03-06T07:30:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:34.091+0000 7f97d5c00640 1 -- 172.21.15.6:0/1156711698 wait complete. 2024-03-06T07:30:34.112 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:34.113 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:34.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.113 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:30:34.162 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:34.162 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:34.162 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:34.162 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42423sec preferred_lft 42423sec 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:30:34.163 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:34.165 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:34.165 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:30:34.165 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:34.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.166 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:34.249 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:30:34.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.253 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:30:34.296 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:30:34.296 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:34.297 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:30:34.297 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:30:34.297 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:34.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.297 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:30:34.297 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:30:34.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:30:34.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:30:34.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:30:34.297 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:30:34.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.528 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:30:34.580 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:34.580 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:34.581 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:34.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:30:34.666 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:34.666 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:34.719 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:34.720 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:34.720 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:34.764 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:34.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:34.953 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:34.953 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:34.988 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:34.989 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:34.989 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:30:34.989 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:30:34.989 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:30:34.993 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:34.993 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:34.993 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:34.993 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:34.993 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:34.997 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:34.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:34.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:35.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:35.076 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:35.141 INFO:teuthology.orchestra.run.smithi006.stdout:client5075 2024-03-06T07:30:35.141 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:35.141 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:35.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:30:35.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.400+0000 7f3645400640 1 Processor -- start 2024-03-06T07:30:35.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.400+0000 7f3645400640 1 -- start start 2024-03-06T07:30:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.401+0000 7f3645400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 0x7f36400cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f3645400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3640072df0 con 0x7f3640071e20 2024-03-06T07:30:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f3645400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400cdab0 con 0x7f364015bf20 2024-03-06T07:30:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f3645400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36400b4d70 con 0x7f3640072520 2024-03-06T07:30:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f363f200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3640071e20 0x7f36400ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46092/0 (socket says 172.21.15.6:46092) 2024-03-06T07:30:35.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f363f200640 1 -- 172.21.15.6:0/281376103 learned_addr learned my addr 172.21.15.6:0/281376103 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f3637e00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 0x7f36400cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.402+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432775752 0 0) 0x7f3640072df0 con 0x7f3640071e20 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3624003520 con 0x7f3640071e20 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519518144 0 0) 0x7f3624003520 con 0x7f3640071e20 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 msgr2=0x7f36400cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 0x7f36400cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 >> v1:172.21.15.178:6789/0 conn(0x7f364015bf20 legacy=0x7f36400acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400b5050 con 0x7f3640071e20 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f36380019f0 con 0x7f3640071e20 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 >> v1:172.21.15.6:6789/0 conn(0x7f3640071e20 legacy=0x7f36400ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f3624003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3624004800 0x7f3624004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 0x7f36240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400b4d70 con 0x7f36240039e0 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362400baa0 con 0x7f3624004800 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362400bca0 con 0x7f3640072520 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.403+0000 7f363e800640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f3624003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f3637e00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3624004800 0x7f3624004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f363f200640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 0x7f36240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f363e800640 1 -- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 msgr2=0x7f36240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f363e800640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640072520 0x7f36240051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f363e800640 1 -- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3624004800 msgr2=0x7f3624004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f363e800640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3624004800 0x7f3624004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f363e800640 1 -- 172.21.15.6:0/281376103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3624003520 con 0x7f36240039e0 2024-03-06T07:30:35.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.404+0000 7f3637e00640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3624004800 0x7f3624004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.405+0000 7f363e800640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f3624003e00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3630001720 tx=0x7f3624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3111315dfb6392b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.405+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630001370 con 0x7f36240039e0 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.405+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36300052d0 con 0x7f36240039e0 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.405+0000 7f363de00640 1 -- 172.21.15.6:0/281376103 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630003830 con 0x7f36240039e0 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f3645400640 1 -- 172.21.15.6:0/281376103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 msgr2=0x7f3624003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f3645400640 1 --2- 172.21.15.6:0/281376103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f3624003e00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3630001720 tx=0x7f3624005830 comp rx=0 tx=0).stop 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f363f200640 1 -- 172.21.15.6:0/281376103 reap_dead start 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f3645400640 1 -- 172.21.15.6:0/281376103 shutdown_connections 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f3645400640 1 -- 172.21.15.6:0/281376103 >> 172.21.15.6:0/281376103 conn(0x7f364006b3d0 msgr2=0x7f364006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f3645400640 1 -- 172.21.15.6:0/281376103 shutdown_connections 2024-03-06T07:30:35.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.406+0000 7f3645400640 1 -- 172.21.15.6:0/281376103 wait complete. 2024-03-06T07:30:35.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.408+0000 7f3645400640 1 Processor -- start 2024-03-06T07:30:35.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.408+0000 7f3645400640 1 -- start start 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f3645400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f36400cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f3645400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640152350 0x7f36401f81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f3645400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36401483e0 0x7f364009ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f3645400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400ad450 con 0x7f36240039e0 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f3645400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400b4990 con 0x7f36401483e0 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f3645400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364006e9f0 con 0x7f3640152350 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f363e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f36400cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f363e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f36400cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53114/0 (socket says 172.21.15.6:53114) 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.409+0000 7f363e800640 1 -- 172.21.15.6:0/3122628517 learned_addr learned my addr 172.21.15.6:0/3122628517 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363e800640 1 -- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640152350 msgr2=0x7f36401f81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363e800640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640152350 0x7f36401f81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363e800640 1 -- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36401483e0 msgr2=0x7f364009ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363f200640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36401483e0 0x7f364009ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363e800640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36401483e0 0x7f364009ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363e800640 1 -- 172.21.15.6:0/3122628517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3640148890 con 0x7f36240039e0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f363e800640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f36400cabd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3630005040 tx=0x7f3630001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.410+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630041070 con 0x7f36240039e0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.411+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3630005480 con 0x7f36240039e0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.411+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630007a30 con 0x7f36240039e0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.411+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36400cb1b0 con 0x7f36240039e0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.411+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36400cb770 con 0x7f36240039e0 2024-03-06T07:30:35.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.412+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36400c8fe0 con 0x7f36240039e0 2024-03-06T07:30:35.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.414+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3630044070 con 0x7f36240039e0 2024-03-06T07:30:35.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.415+0000 7f3636a00640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361c06b500 0x7f361c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.415+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36300abef0 con 0x7f36240039e0 2024-03-06T07:30:35.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.415+0000 7f3637e00640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361c06b500 0x7f361c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.417+0000 7f3637e00640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361c06b500 0x7f361c06d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f364009d6c0 tx=0x7f3628008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.421+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f363007adf0 con 0x7f36240039e0 2024-03-06T07:30:35.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.626+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f364004f330 con 0x7f36240039e0 2024-03-06T07:30:35.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.632+0000 7f3636a00640 1 -- 172.21.15.6:0/3122628517 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7f363007a6f0 con 0x7f36240039e0 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361c06b500 msgr2=0x7f361c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361c06b500 0x7f361c06d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f364009d6c0 tx=0x7f3628008040 comp rx=0 tx=0).stop 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 msgr2=0x7f36400cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f36400cabd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3630005040 tx=0x7f3630001120 comp rx=0 tx=0).stop 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 shutdown_connections 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361c06b500 0x7f361c06d9c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36401483e0 0x7f364009ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3640152350 0x7f36401f81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 --2- 172.21.15.6:0/3122628517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36240039e0 0x7f36400cabd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.635+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 >> 172.21.15.6:0/3122628517 conn(0x7f364006b3d0 msgr2=0x7f3640157180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.636+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 shutdown_connections 2024-03-06T07:30:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.636+0000 7f3645400640 1 -- 172.21.15.6:0/3122628517 wait complete. 2024-03-06T07:30:35.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:30:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.863+0000 7f5971600640 1 Processor -- start 2024-03-06T07:30:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.863+0000 7f5971600640 1 -- start start 2024-03-06T07:30:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.864+0000 7f5971600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f596c06b6c0 0x7f596c0cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.864+0000 7f5971600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f596c0d00f0 0x7f596c0aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.864+0000 7f5971600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c071ff0 con 0x7f596c145860 2024-03-06T07:30:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.864+0000 7f5971600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c071da0 con 0x7f596c06b6c0 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.864+0000 7f5971600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c0722d0 con 0x7f596c0d00f0 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f596c06b6c0 0x7f596c0cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f596c06b6c0 0x7f596c0cfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53118/0 (socket says 172.21.15.6:53118) 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 -- 172.21.15.6:0/2512985142 learned_addr learned my addr 172.21.15.6:0/2512985142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5970c00640 1 --2- 172.21.15.6:0/2512985142 >> v2:172.21.15.178:3300/0 conn(0x7f596c0d00f0 0x7f596c0aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 -- 172.21.15.6:0/2512985142 >> v2:172.21.15.178:3300/0 conn(0x7f596c0d00f0 msgr2=0x7f596c0aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 --2- 172.21.15.6:0/2512985142 >> v2:172.21.15.178:3300/0 conn(0x7f596c0d00f0 0x7f596c0aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 -- 172.21.15.6:0/2512985142 >> v1:172.21.15.178:6789/0 conn(0x7f596c145860 legacy=0x7f596c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.865+0000 7f5963e00640 1 -- 172.21.15.6:0/2512985142 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c0725b0 con 0x7f596c06b6c0 2024-03-06T07:30:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.867+0000 7f5963e00640 1 --2- 172.21.15.6:0/2512985142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c06b6c0 0x7f596c0cfbb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f596c160a80 tx=0x7f595402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3318af8c66f85bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.867+0000 7f596b400640 1 -- 172.21.15.6:0/2512985142 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954035020 con 0x7f596c06b6c0 2024-03-06T07:30:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.867+0000 7f596b400640 1 -- 172.21.15.6:0/2512985142 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954031870 con 0x7f596c06b6c0 2024-03-06T07:30:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.867+0000 7f596b400640 1 -- 172.21.15.6:0/2512985142 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954031c40 con 0x7f596c06b6c0 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 -- 172.21.15.6:0/2512985142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c06b6c0 msgr2=0x7f596c0cfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 --2- 172.21.15.6:0/2512985142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c06b6c0 0x7f596c0cfbb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f596c160a80 tx=0x7f595402b910 comp rx=0 tx=0).stop 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 -- 172.21.15.6:0/2512985142 shutdown_connections 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 --2- 172.21.15.6:0/2512985142 >> v2:172.21.15.178:3300/0 conn(0x7f596c0d00f0 0x7f596c0aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 --2- 172.21.15.6:0/2512985142 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c06b6c0 0x7f596c0cfbb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 -- 172.21.15.6:0/2512985142 >> 172.21.15.6:0/2512985142 conn(0x7f596c0d0c00 msgr2=0x7f596c0d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:35.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 -- 172.21.15.6:0/2512985142 shutdown_connections 2024-03-06T07:30:35.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.868+0000 7f5971600640 1 -- 172.21.15.6:0/2512985142 wait complete. 2024-03-06T07:30:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.870+0000 7f5971600640 1 Processor -- start 2024-03-06T07:30:35.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.870+0000 7f5971600640 1 -- start start 2024-03-06T07:30:35.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5971600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596c06b6c0 0x7f596c0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5971600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596c0d00f0 0x7f596c1e8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5971600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 0x7f596c1e9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5971600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0af550 con 0x7f596c145860 2024-03-06T07:30:35.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5971600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0af3d0 con 0x7f596c0d00f0 2024-03-06T07:30:35.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5971600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0af6d0 con 0x7f596c06b6c0 2024-03-06T07:30:35.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5970c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 0x7f596c1e9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5970c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 0x7f596c1e9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53122/0 (socket says 172.21.15.6:53122) 2024-03-06T07:30:35.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5970c00640 1 -- 172.21.15.6:0/1198642681 learned_addr learned my addr 172.21.15.6:0/1198642681 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:35.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5963e00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596c0d00f0 0x7f596c1e8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f596be00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596c06b6c0 0x7f596c0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.871+0000 7f5970c00640 1 -- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596c06b6c0 msgr2=0x7f596c0cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5970c00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596c06b6c0 0x7f596c0cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5970c00640 1 -- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596c0d00f0 msgr2=0x7f596c1e8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:35.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5970c00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596c0d00f0 0x7f596c1e8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:35.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5970c00640 1 -- 172.21.15.6:0/1198642681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c1314b0 con 0x7f596c145860 2024-03-06T07:30:35.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5970c00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 0x7f596c1e9010 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5964007e40 tx=0x7f596400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964014620 con 0x7f596c145860 2024-03-06T07:30:35.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5964019070 con 0x7f596c145860 2024-03-06T07:30:35.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964014a20 con 0x7f596c145860 2024-03-06T07:30:35.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c131790 con 0x7f596c145860 2024-03-06T07:30:35.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.872+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c1e60e0 con 0x7f596c145860 2024-03-06T07:30:35.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.874+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c04f330 con 0x7f596c145860 2024-03-06T07:30:35.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.874+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5964014bc0 con 0x7f596c145860 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.876+0000 7f5968c00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f594806b500 0x7f594806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.876+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f596408aad0 con 0x7f596c145860 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.877+0000 7f596be00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f594806b500 0x7f594806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:35.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.878+0000 7f596be00640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f594806b500 0x7f594806d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f595c006fd0 tx=0x7f595c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:35.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:35.880+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5964059950 con 0x7f596c145860 2024-03-06T07:30:36.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.196+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f596c0cda40 con 0x7f596c145860 2024-03-06T07:30:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.201+0000 7f5968c00640 1 -- 172.21.15.6:0/1198642681 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f5964059250 con 0x7f596c145860 2024-03-06T07:30:36.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f594806b500 msgr2=0x7f594806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f594806b500 0x7f594806d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f595c006fd0 tx=0x7f595c008040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 msgr2=0x7f596c1e9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 0x7f596c1e9010 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f5964007e40 tx=0x7f596400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 shutdown_connections 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f594806b500 0x7f594806d9c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f596c0d00f0 0x7f596c1e8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f596c145860 0x7f596c1e9010 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 --2- 172.21.15.6:0/1198642681 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f596c06b6c0 0x7f596c0cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 >> 172.21.15.6:0/1198642681 conn(0x7f596c0d0c00 msgr2=0x7f596c0cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 shutdown_connections 2024-03-06T07:30:36.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:36.205+0000 7f5971600640 1 -- 172.21.15.6:0/1198642681 wait complete. 2024-03-06T07:30:46.227 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:46.227 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> n = 50 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:46.287 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-06T07:30:46.288 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-06T07:30:46.289 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:30:46.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:46.596 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:46.596 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:46.689 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:46.689 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:46.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.994+0000 7f91f5c00640 1 Processor -- start 2024-03-06T07:30:46.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.995+0000 7f91f5c00640 1 -- start start 2024-03-06T07:30:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.996+0000 7f91f5c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f91f015e200 0x7f91f00b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.996+0000 7f91f5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f91f00afdf0 0x7f91f00af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.996+0000 7f91f5c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f006bec0 con 0x7f91f006afb0 2024-03-06T07:30:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.996+0000 7f91f5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00b0670 con 0x7f91f00afdf0 2024-03-06T07:30:46.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.996+0000 7f91f5c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f00ae380 con 0x7f91f015e200 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ee000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f91f00afdf0 0x7f91f00af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ef400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f91f006afb0 0x7f91f006dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54632/0 (socket says 172.21.15.6:54632) 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ef400640 1 -- 172.21.15.6:0/3174331535 learned_addr learned my addr 172.21.15.6:0/3174331535 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91eea00640 1 --2- 172.21.15.6:0/3174331535 >> v2:172.21.15.178:3300/0 conn(0x7f91f015e200 0x7f91f00b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ee000640 1 -- 172.21.15.6:0/3174331535 >> v2:172.21.15.178:3300/0 conn(0x7f91f015e200 msgr2=0x7f91f00b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ee000640 1 --2- 172.21.15.6:0/3174331535 >> v2:172.21.15.178:3300/0 conn(0x7f91f015e200 0x7f91f00b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ee000640 1 -- 172.21.15.6:0/3174331535 >> v1:172.21.15.6:6789/0 conn(0x7f91f006afb0 legacy=0x7f91f006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ee000640 1 -- 172.21.15.6:0/3174331535 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00ae660 con 0x7f91f00afdf0 2024-03-06T07:30:47.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.997+0000 7f91ed600640 1 -- 172.21.15.6:0/3174331535 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891844214 0 0) 0x7f91f006bec0 con 0x7f91f006afb0 2024-03-06T07:30:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91ee000640 1 --2- 172.21.15.6:0/3174331535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00afdf0 0x7f91f00af140 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f91f0071cc0 tx=0x7f91dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae17819c5a27327 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91ed600640 1 -- 172.21.15.6:0/3174331535 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc035020 con 0x7f91f00afdf0 2024-03-06T07:30:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91ed600640 1 -- 172.21.15.6:0/3174331535 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc031830 con 0x7f91f00afdf0 2024-03-06T07:30:47.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91ed600640 1 -- 172.21.15.6:0/3174331535 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc031c00 con 0x7f91f00afdf0 2024-03-06T07:30:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91f5c00640 1 -- 172.21.15.6:0/3174331535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00afdf0 msgr2=0x7f91f00af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91f5c00640 1 --2- 172.21.15.6:0/3174331535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00afdf0 0x7f91f00af140 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f91f0071cc0 tx=0x7f91dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:46.999+0000 7f91f5c00640 1 -- 172.21.15.6:0/3174331535 shutdown_connections 2024-03-06T07:30:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.000+0000 7f91f5c00640 1 --2- 172.21.15.6:0/3174331535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f00afdf0 0x7f91f00af140 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.000+0000 7f91f5c00640 1 --2- 172.21.15.6:0/3174331535 >> v2:172.21.15.178:3300/0 conn(0x7f91f015e200 0x7f91f00b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.000+0000 7f91f5c00640 1 -- 172.21.15.6:0/3174331535 >> 172.21.15.6:0/3174331535 conn(0x7f91f00b2ae0 msgr2=0x7f91f00b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.000+0000 7f91f5c00640 1 -- 172.21.15.6:0/3174331535 shutdown_connections 2024-03-06T07:30:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.000+0000 7f91f5c00640 1 -- 172.21.15.6:0/3174331535 wait complete. 2024-03-06T07:30:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.001+0000 7f91f5c00640 1 Processor -- start 2024-03-06T07:30:47.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.001+0000 7f91f5c00640 1 -- start start 2024-03-06T07:30:47.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.002+0000 7f91f5c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91f006afb0 0x7f91f015e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.002+0000 7f91f5c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91f00afdf0 0x7f91f01f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.002+0000 7f91f5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 0x7f91f01f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.002+0000 7f91f5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00af680 con 0x7f91f015e200 2024-03-06T07:30:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.002+0000 7f91f5c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f00aec20 con 0x7f91f00afdf0 2024-03-06T07:30:47.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.002+0000 7f91f5c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f006e2c0 con 0x7f91f006afb0 2024-03-06T07:30:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 0x7f91f01f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 0x7f91f01f7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54668/0 (socket says 172.21.15.6:54668) 2024-03-06T07:30:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 -- 172.21.15.6:0/1554557972 learned_addr learned my addr 172.21.15.6:0/1554557972 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:47.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ee000640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91f00afdf0 0x7f91f01f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91eea00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91f006afb0 0x7f91f015e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 -- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91f006afb0 msgr2=0x7f91f015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91f006afb0 0x7f91f015e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 -- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91f00afdf0 msgr2=0x7f91f01f7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91f00afdf0 0x7f91f01f7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.003+0000 7f91ef400640 1 -- 172.21.15.6:0/1554557972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f0145810 con 0x7f91f015e200 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.004+0000 7f91ef400640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 0x7f91f01f7cb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c7a0 tx=0x7f91e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.004+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8018070 con 0x7f91f015e200 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.004+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e800a640 con 0x7f91f015e200 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.004+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800aab0 con 0x7f91f015e200 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.004+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f0145ab0 con 0x7f91f015e200 2024-03-06T07:30:47.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.004+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f0146070 con 0x7f91f015e200 2024-03-06T07:30:47.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.005+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f004f3a0 con 0x7f91f015e200 2024-03-06T07:30:47.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.007+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91e801b070 con 0x7f91f015e200 2024-03-06T07:30:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.009+0000 7f91e6a00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f91d006b4d0 0x7f91d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.009+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91e8089830 con 0x7f91f015e200 2024-03-06T07:30:47.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.010+0000 7f91eea00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f91d006b4d0 0x7f91d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.011+0000 7f91eea00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f91d006b4d0 0x7f91d006d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f91e0006fd0 tx=0x7f91e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.015+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91e80586a0 con 0x7f91f015e200 2024-03-06T07:30:47.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.214+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91f0157cc0 con 0x7f91f015e200 2024-03-06T07:30:47.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.216+0000 7f91e6a00640 1 -- 172.21.15.6:0/1554557972 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f91e8057fa0 con 0x7f91f015e200 2024-03-06T07:30:47.218 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:47.218 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":58,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:22.633658+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f91d006b4d0 msgr2=0x7f91d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f91d006b4d0 0x7f91d006d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f91e0006fd0 tx=0x7f91e0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 msgr2=0x7f91f01f7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 0x7f91f01f7cb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f91e800c7a0 tx=0x7f91e800cc60 comp rx=0 tx=0).stop 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 shutdown_connections 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f91d006b4d0 0x7f91d006d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f91f00afdf0 0x7f91f01f7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f91f015e200 0x7f91f01f7cb0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 --2- 172.21.15.6:0/1554557972 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91f006afb0 0x7f91f015e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 >> 172.21.15.6:0/1554557972 conn(0x7f91f00b2ae0 msgr2=0x7f91f01564a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.218+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 shutdown_connections 2024-03-06T07:30:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.219+0000 7f91f5c00640 1 -- 172.21.15.6:0/1554557972 wait complete. 2024-03-06T07:30:47.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:19.028665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:20.117374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:47.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:47.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.448+0000 7f54c3e00640 1 Processor -- start 2024-03-06T07:30:47.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.448+0000 7f54c3e00640 1 -- start start 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.451+0000 7f54c3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f54c406afb0 0x7f54c406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.451+0000 7f54c3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c40cf480 0x7f54c40cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c40b5800 con 0x7f54c415e0c0 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c406dd30 con 0x7f54c40cf480 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c40b5aa0 con 0x7f54c406afb0 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c40cf480 0x7f54c40cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54c40cf480 0x7f54c40cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54670/0 (socket says 172.21.15.6:54670) 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 -- 172.21.15.6:0/3068866319 learned_addr learned my addr 172.21.15.6:0/3068866319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:47.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c2000640 1 --2- 172.21.15.6:0/3068866319 >> v2:172.21.15.178:3300/0 conn(0x7f54c406afb0 0x7f54c406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 -- 172.21.15.6:0/3068866319 >> v2:172.21.15.178:3300/0 conn(0x7f54c406afb0 msgr2=0x7f54c406b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 --2- 172.21.15.6:0/3068866319 >> v2:172.21.15.178:3300/0 conn(0x7f54c406afb0 0x7f54c406b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 -- 172.21.15.6:0/3068866319 >> v1:172.21.15.6:6789/0 conn(0x7f54c415e0c0 legacy=0x7f54c4072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.452+0000 7f54c3400640 1 -- 172.21.15.6:0/3068866319 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40b5d80 con 0x7f54c40cf480 2024-03-06T07:30:47.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.454+0000 7f54c3400640 1 --2- 172.21.15.6:0/3068866319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c40cf480 0x7f54c40cf890 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f54c40b59d0 tx=0x7f54bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac47a4d97613b147 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.454+0000 7f54c1600640 1 -- 172.21.15.6:0/3068866319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc02cb00 con 0x7f54c40cf480 2024-03-06T07:30:47.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.454+0000 7f54c1600640 1 -- 172.21.15.6:0/3068866319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc02cca0 con 0x7f54c40cf480 2024-03-06T07:30:47.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.454+0000 7f54c1600640 1 -- 172.21.15.6:0/3068866319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc034610 con 0x7f54c40cf480 2024-03-06T07:30:47.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.454+0000 7f54c3e00640 1 -- 172.21.15.6:0/3068866319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c40cf480 msgr2=0x7f54c40cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 --2- 172.21.15.6:0/3068866319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c40cf480 0x7f54c40cf890 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f54c40b59d0 tx=0x7f54bc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 -- 172.21.15.6:0/3068866319 shutdown_connections 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 --2- 172.21.15.6:0/3068866319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c40cf480 0x7f54c40cf890 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 --2- 172.21.15.6:0/3068866319 >> v2:172.21.15.178:3300/0 conn(0x7f54c406afb0 0x7f54c406b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 -- 172.21.15.6:0/3068866319 >> 172.21.15.6:0/3068866319 conn(0x7f54c40d0c00 msgr2=0x7f54c40d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 -- 172.21.15.6:0/3068866319 shutdown_connections 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.455+0000 7f54c3e00640 1 -- 172.21.15.6:0/3068866319 wait complete. 2024-03-06T07:30:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.457+0000 7f54c3e00640 1 Processor -- start 2024-03-06T07:30:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.457+0000 7f54c3e00640 1 -- start start 2024-03-06T07:30:47.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54c406afb0 0x7f54c4142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54c40cf480 0x7f54c40af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 0x7f54c4140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c4072cc0 con 0x7f54c415e0c0 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40cef60 con 0x7f54c406afb0 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c4072e40 con 0x7f54c40cf480 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 0x7f54c4140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 0x7f54c4140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54680/0 (socket says 172.21.15.6:54680) 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c3400640 1 -- 172.21.15.6:0/874886541 learned_addr learned my addr 172.21.15.6:0/874886541 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c2a00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54c406afb0 0x7f54c4142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.458+0000 7f54c2000640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54c40cf480 0x7f54c40af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3400640 1 -- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54c40cf480 msgr2=0x7f54c40af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3400640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54c40cf480 0x7f54c40af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3400640 1 -- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54c406afb0 msgr2=0x7f54c4142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3400640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54c406afb0 0x7f54c4142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3400640 1 -- 172.21.15.6:0/874886541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c41430e0 con 0x7f54c415e0c0 2024-03-06T07:30:47.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c2a00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54c406afb0 0x7f54c4142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3400640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 0x7f54c4140970 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f54bc036040 tx=0x7f54bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc043020 con 0x7f54c415e0c0 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc02ce50 con 0x7f54c415e0c0 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c41433c0 con 0x7f54c415e0c0 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.459+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc009510 con 0x7f54c415e0c0 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.460+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c4160680 con 0x7f54c415e0c0 2024-03-06T07:30:47.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.461+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c404f330 con 0x7f54c415e0c0 2024-03-06T07:30:47.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.462+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54bc046020 con 0x7f54c415e0c0 2024-03-06T07:30:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.463+0000 7f54baa00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54a406b4d0 0x7f54a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.463+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54bc0ac810 con 0x7f54c415e0c0 2024-03-06T07:30:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.463+0000 7f54c2a00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54a406b4d0 0x7f54a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.465+0000 7f54c2a00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54a406b4d0 0x7f54a406d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54c415d8e0 tx=0x7f54b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.467+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54bc04d030 con 0x7f54c415e0c0 2024-03-06T07:30:47.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.716+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54c40b55a0 con 0x7f54c415e0c0 2024-03-06T07:30:47.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.716+0000 7f54baa00640 1 -- 172.21.15.6:0/874886541 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54bc07b720 con 0x7f54c415e0c0 2024-03-06T07:30:47.718 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:30:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54a406b4d0 msgr2=0x7f54a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54a406b4d0 0x7f54a406d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54c415d8e0 tx=0x7f54b4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 msgr2=0x7f54c4140970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 0x7f54c4140970 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f54bc036040 tx=0x7f54bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 shutdown_connections 2024-03-06T07:30:47.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54a406b4d0 0x7f54a406d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54c415e0c0 0x7f54c4140970 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54c40cf480 0x7f54c40af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 --2- 172.21.15.6:0/874886541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54c406afb0 0x7f54c4142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 >> 172.21.15.6:0/874886541 conn(0x7f54c40d0c00 msgr2=0x7f54c40b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 shutdown_connections 2024-03-06T07:30:47.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.719+0000 7f54c3e00640 1 -- 172.21.15.6:0/874886541 wait complete. 2024-03-06T07:30:47.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:47.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.956+0000 7feb7b200640 1 Processor -- start 2024-03-06T07:30:47.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.956+0000 7feb7b200640 1 -- start start 2024-03-06T07:30:47.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.959+0000 7feb7b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb7415e090 0x7feb740b5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.959+0000 7feb7b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7feb7406d1c0 0x7feb740b6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.959+0000 7feb7b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb740b1830 con 0x7feb740aea90 2024-03-06T07:30:47.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.959+0000 7feb7b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb740b1b10 con 0x7feb7406d1c0 2024-03-06T07:30:47.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.959+0000 7feb7b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb740b1df0 con 0x7feb7415e090 2024-03-06T07:30:47.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.960+0000 7feb79400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feb740aea90 0x7feb7406cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54658/0 (socket says 172.21.15.6:54658) 2024-03-06T07:30:47.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.960+0000 7feb79400640 1 -- 172.21.15.6:0/3247470339 learned_addr learned my addr 172.21.15.6:0/3247470339 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.960+0000 7feb7a800640 1 --2- 172.21.15.6:0/3247470339 >> v2:172.21.15.178:3300/0 conn(0x7feb7406d1c0 0x7feb740b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.960+0000 7feb79e00640 1 --2- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb7415e090 0x7feb740b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.960+0000 7feb78a00640 1 -- 172.21.15.6:0/3247470339 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788657119 0 0) 0x7feb740b1830 con 0x7feb740aea90 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.960+0000 7feb78a00640 1 -- 172.21.15.6:0/3247470339 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb5c003520 con 0x7feb740aea90 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.961+0000 7feb7a800640 1 -- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb7415e090 msgr2=0x7feb740b5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.961+0000 7feb7a800640 1 --2- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb7415e090 0x7feb740b5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.961+0000 7feb7a800640 1 -- 172.21.15.6:0/3247470339 >> v1:172.21.15.6:6789/0 conn(0x7feb740aea90 legacy=0x7feb7406cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.961+0000 7feb7a800640 1 -- 172.21.15.6:0/3247470339 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740b20d0 con 0x7feb7406d1c0 2024-03-06T07:30:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.963+0000 7feb7a800640 1 --2- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7406d1c0 0x7feb740b6540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feb70009d90 tx=0x7feb7002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=308540e23a5f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.963+0000 7feb78a00640 1 -- 172.21.15.6:0/3247470339 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70039070 con 0x7feb7406d1c0 2024-03-06T07:30:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb78a00640 1 -- 172.21.15.6:0/3247470339 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7002cc60 con 0x7feb7406d1c0 2024-03-06T07:30:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb78a00640 1 -- 172.21.15.6:0/3247470339 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70034660 con 0x7feb7406d1c0 2024-03-06T07:30:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 -- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7406d1c0 msgr2=0x7feb740b6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 --2- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7406d1c0 0x7feb740b6540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feb70009d90 tx=0x7feb7002beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 -- 172.21.15.6:0/3247470339 shutdown_connections 2024-03-06T07:30:47.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 --2- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7406d1c0 0x7feb740b6540 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 --2- 172.21.15.6:0/3247470339 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb7415e090 0x7feb740b5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 -- 172.21.15.6:0/3247470339 >> 172.21.15.6:0/3247470339 conn(0x7feb7409aa10 msgr2=0x7feb7409ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.964+0000 7feb7b200640 1 -- 172.21.15.6:0/3247470339 shutdown_connections 2024-03-06T07:30:47.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.965+0000 7feb7b200640 1 -- 172.21.15.6:0/3247470339 wait complete. 2024-03-06T07:30:47.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.967+0000 7feb7b200640 1 Processor -- start 2024-03-06T07:30:47.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.968+0000 7feb7b200640 1 -- start start 2024-03-06T07:30:47.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.969+0000 7feb7b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 0x7feb7414e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.969+0000 7feb7b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb740aea90 0x7feb7409f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.969+0000 7feb7b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7415e090 0x7feb7415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.969+0000 7feb7b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb740b2a50 con 0x7feb7406d1c0 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.969+0000 7feb7b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb740b28d0 con 0x7feb7415e090 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.969+0000 7feb7b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb740b2bd0 con 0x7feb740aea90 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 0x7feb7414e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 0x7feb7414e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54686/0 (socket says 172.21.15.6:54686) 2024-03-06T07:30:47.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 -- 172.21.15.6:0/3284944595 learned_addr learned my addr 172.21.15.6:0/3284944595 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb7a800640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7415e090 0x7feb7415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79400640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb740aea90 0x7feb7409f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 -- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb740aea90 msgr2=0x7feb7409f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb740aea90 0x7feb7409f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 -- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7415e090 msgr2=0x7feb7415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7415e090 0x7feb7415cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 -- 172.21.15.6:0/3284944595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7415bc70 con 0x7feb7406d1c0 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb7a800640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7415e090 0x7feb7415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79400640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb740aea90 0x7feb7409f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.970+0000 7feb79e00640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 0x7feb7414e730 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7feb6800e870 tx=0x7feb6800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.971+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68017070 con 0x7feb7406d1c0 2024-03-06T07:30:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.971+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6800c940 con 0x7feb7406d1c0 2024-03-06T07:30:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.971+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7415bf50 con 0x7feb7406d1c0 2024-03-06T07:30:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.971+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6800cd30 con 0x7feb7406d1c0 2024-03-06T07:30:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.971+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7415c510 con 0x7feb7406d1c0 2024-03-06T07:30:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.972+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7404f330 con 0x7feb7406d1c0 2024-03-06T07:30:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.973+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb6801a070 con 0x7feb7406d1c0 2024-03-06T07:30:47.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.974+0000 7feb6e000640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.974+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb6808a4b0 con 0x7feb7406d1c0 2024-03-06T07:30:47.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.975+0000 7feb79400640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.976+0000 7feb79400640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb5806b500 0x7feb5806d9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feb7415d950 tx=0x7feb64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:47.979+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb68059290 con 0x7feb7406d1c0 2024-03-06T07:30:48.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.178+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feb74067af0 con 0x7feb7406d1c0 2024-03-06T07:30:48.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.179+0000 7feb6e000640 1 -- 172.21.15.6:0/3284944595 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7feb68058b90 con 0x7feb7406d1c0 2024-03-06T07:30:48.181 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:30:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.181+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb5806b500 msgr2=0x7feb5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.181+0000 7feb7b200640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb5806b500 0x7feb5806d9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7feb7415d950 tx=0x7feb64009040 comp rx=0 tx=0).stop 2024-03-06T07:30:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.181+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 msgr2=0x7feb7414e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.181+0000 7feb7b200640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 0x7feb7414e730 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7feb6800e870 tx=0x7feb6800ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 shutdown_connections 2024-03-06T07:30:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb7415e090 0x7feb7415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb740aea90 0x7feb7409f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 --2- 172.21.15.6:0/3284944595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb7406d1c0 0x7feb7414e730 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 >> 172.21.15.6:0/3284944595 conn(0x7feb7409aa10 msgr2=0x7feb7406ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 shutdown_connections 2024-03-06T07:30:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.182+0000 7feb7b200640 1 -- 172.21.15.6:0/3284944595 wait complete. 2024-03-06T07:30:48.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:48.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.415+0000 7f4686400640 1 Processor -- start 2024-03-06T07:30:48.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.415+0000 7f4686400640 1 -- start start 2024-03-06T07:30:48.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.417+0000 7f4686400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f468015f810 0x7f46800b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.417+0000 7f4686400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468015fe40 con 0x7f4680145430 2024-03-06T07:30:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.417+0000 7f4686400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4680160120 con 0x7f46800b2080 2024-03-06T07:30:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.417+0000 7f4686400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46800b24d0 con 0x7f468015f810 2024-03-06T07:30:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.417+0000 7f4685000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4680145430 0x7f468015cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54664/0 (socket says 172.21.15.6:54664) 2024-03-06T07:30:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.417+0000 7f4685000640 1 -- 172.21.15.6:0/1490362068 learned_addr learned my addr 172.21.15.6:0/1490362068 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467fe00640 1 --2- 172.21.15.6:0/1490362068 >> v2:172.21.15.178:3300/0 conn(0x7f468015f810 0x7f46800b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846618905 0 0) 0x7f468015fe40 con 0x7f4680145430 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4664003520 con 0x7f4680145430 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393215246 0 0) 0x7f4664003520 con 0x7f4680145430 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 >> v2:172.21.15.178:3300/0 conn(0x7f468015f810 msgr2=0x7f46800b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 --2- 172.21.15.6:0/1490362068 >> v2:172.21.15.178:3300/0 conn(0x7f468015f810 0x7f46800b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 >> v1:172.21.15.178:6789/0 conn(0x7f46800b2080 legacy=0x7f46800b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680160400 con 0x7f4680145430 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f46700019f0 con 0x7f4680145430 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.418+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 >> v1:172.21.15.6:6789/0 conn(0x7f4680145430 legacy=0x7f468015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467f400640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4664003a00 0x7f4664003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467f400640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4664004800 0x7f4664004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467f400640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46640051c0 0x7f466400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46800b24d0 con 0x7f4664004800 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466400c0c0 con 0x7f46640051c0 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466400c2c0 con 0x7f4664003a00 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467fe00640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4664004800 0x7f4664004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467fe00640 1 -- 172.21.15.6:0/1490362068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4664003a00 msgr2=0x7f4664003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467fe00640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4664003a00 0x7f4664003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467fe00640 1 -- 172.21.15.6:0/1490362068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46640051c0 msgr2=0x7f466400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467fe00640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46640051c0 0x7f466400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f467fe00640 1 -- 172.21.15.6:0/1490362068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664003520 con 0x7f4664004800 2024-03-06T07:30:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.419+0000 7f4685a00640 1 -- 172.21.15.6:0/1490362068 reap_dead start 2024-03-06T07:30:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.420+0000 7f467fe00640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4664004800 0x7f4664004c80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4668009ac0 tx=0x7f4664005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9aff2b73616725 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668039070 con 0x7f4664004800 2024-03-06T07:30:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466802c710 con 0x7f4664004800 2024-03-06T07:30:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f467f400640 1 -- 172.21.15.6:0/1490362068 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668009690 con 0x7f4664004800 2024-03-06T07:30:48.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 -- 172.21.15.6:0/1490362068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4664004800 msgr2=0x7f4664004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4664004800 0x7f4664004c80 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4668009ac0 tx=0x7f4664005830 comp rx=0 tx=0).stop 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 -- 172.21.15.6:0/1490362068 shutdown_connections 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 --2- 172.21.15.6:0/1490362068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4664004800 0x7f4664004c80 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 -- 172.21.15.6:0/1490362068 >> 172.21.15.6:0/1490362068 conn(0x7f468006b3a0 msgr2=0x7f468006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 -- 172.21.15.6:0/1490362068 shutdown_connections 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.421+0000 7f4686400640 1 -- 172.21.15.6:0/1490362068 wait complete. 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.423+0000 7f4686400640 1 Processor -- start 2024-03-06T07:30:48.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.423+0000 7f4686400640 1 -- start start 2024-03-06T07:30:48.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.424+0000 7f4686400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4664004800 0x7f4680144780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.424+0000 7f4686400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 0x7f468013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.424+0000 7f4686400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f468015b850 0x7f468013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.424+0000 7f4686400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680145990 con 0x7f468015b3d0 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.424+0000 7f4686400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680145810 con 0x7f4664004800 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.424+0000 7f4686400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46800cde00 con 0x7f468015b850 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 0x7f468013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 0x7f468013faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54704/0 (socket says 172.21.15.6:54704) 2024-03-06T07:30:48.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 -- 172.21.15.6:0/669113071 learned_addr learned my addr 172.21.15.6:0/669113071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:48.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f4685000640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4664004800 0x7f4680144780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f4685a00640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f468015b850 0x7f468013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 -- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f468015b850 msgr2=0x7f468013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f468015b850 0x7f468013ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 -- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4664004800 msgr2=0x7f4680144780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4664004800 0x7f4680144780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.425+0000 7f467fe00640 1 -- 172.21.15.6:0/669113071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680135820 con 0x7f468015b3d0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.426+0000 7f467fe00640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 0x7f468013faa0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4668009be0 tx=0x7f4668004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.426+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668042020 con 0x7f468015b3d0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.426+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466803a040 con 0x7f468015b3d0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.426+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668039070 con 0x7f468015b3d0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.426+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468015a330 con 0x7f468015b3d0 2024-03-06T07:30:48.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.426+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468015a8f0 con 0x7f468015b3d0 2024-03-06T07:30:48.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.427+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468004f330 con 0x7f468015b3d0 2024-03-06T07:30:48.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.429+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4668045020 con 0x7f468015b3d0 2024-03-06T07:30:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.430+0000 7f467cc00640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465c06b4d0 0x7f465c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.430+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46680ab6a0 con 0x7f468015b3d0 2024-03-06T07:30:48.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.431+0000 7f4685000640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465c06b4d0 0x7f465c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.432+0000 7f4685000640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465c06b4d0 0x7f465c06d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4670005fd0 tx=0x7f4670001aa0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.435+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f466804c070 con 0x7f468015b3d0 2024-03-06T07:30:48.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.629+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f468015e330 con 0x7f468015b3d0 2024-03-06T07:30:48.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.629+0000 7f467cc00640 1 -- 172.21.15.6:0/669113071 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f466807a510 con 0x7f468015b3d0 2024-03-06T07:30:48.631 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.631+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465c06b4d0 msgr2=0x7f465c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.631+0000 7f4686400640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465c06b4d0 0x7f465c06d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4670005fd0 tx=0x7f4670001aa0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.631+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 msgr2=0x7f468013faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.631+0000 7f4686400640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 0x7f468013faa0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4668009be0 tx=0x7f4668004730 comp rx=0 tx=0).stop 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 shutdown_connections 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465c06b4d0 0x7f465c06d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f468015b850 0x7f468013ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f468015b3d0 0x7f468013faa0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 --2- 172.21.15.6:0/669113071 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4664004800 0x7f4680144780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 >> 172.21.15.6:0/669113071 conn(0x7f468006b3a0 msgr2=0x7f468006c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 shutdown_connections 2024-03-06T07:30:48.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.632+0000 7f4686400640 1 -- 172.21.15.6:0/669113071 wait complete. 2024-03-06T07:30:48.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:48.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.854+0000 7f7602a00640 1 Processor -- start 2024-03-06T07:30:48.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.854+0000 7f7602a00640 1 -- start start 2024-03-06T07:30:48.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f7602a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f7602a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f75fc0cdc20 0x7f75fc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f7602a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75fc15cc40 con 0x7f75fc0af1a0 2024-03-06T07:30:48.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f7602a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75fc15cf20 con 0x7f75fc0cdc20 2024-03-06T07:30:48.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f7602a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc15d200 con 0x7f75fc0af580 2024-03-06T07:30:48.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f7600a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f75fc0cdc20 0x7f75fc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.856+0000 7f75fb400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f7600a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f75fc0cdc20 0x7f75fc0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:54882/0 (socket says 172.21.15.6:54882) 2024-03-06T07:30:48.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f7600a00640 1 -- 172.21.15.6:0/213944710 learned_addr learned my addr 172.21.15.6:0/213944710 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:48.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f7600a00640 1 -- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 msgr2=0x7f75fc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f7600a00640 1 --2- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f7600a00640 1 -- 172.21.15.6:0/213944710 >> v1:172.21.15.178:6789/0 conn(0x7f75fc0af1a0 legacy=0x7f75fc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f7600a00640 1 -- 172.21.15.6:0/213944710 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc15d4e0 con 0x7f75fc0cdc20 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.857+0000 7f75fb400640 1 --2- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.859+0000 7f7600a00640 1 --2- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc0b2dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009e50 tx=0x7f75f402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9a23db1ae7ab70f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.859+0000 7f75faa00640 1 -- 172.21.15.6:0/213944710 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f402ca80 con 0x7f75fc0cdc20 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.859+0000 7f75faa00640 1 -- 172.21.15.6:0/213944710 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f402cc20 con 0x7f75fc0cdc20 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.859+0000 7f75faa00640 1 -- 172.21.15.6:0/213944710 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40343f0 con 0x7f75fc0cdc20 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 -- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 msgr2=0x7f75fc0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 --2- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc0b2dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009e50 tx=0x7f75f402bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 -- 172.21.15.6:0/213944710 shutdown_connections 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 --2- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc0b2dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 --2- 172.21.15.6:0/213944710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 -- 172.21.15.6:0/213944710 >> 172.21.15.6:0/213944710 conn(0x7f75fc06afb0 msgr2=0x7f75fc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 -- 172.21.15.6:0/213944710 shutdown_connections 2024-03-06T07:30:48.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.860+0000 7f7602a00640 1 -- 172.21.15.6:0/213944710 wait complete. 2024-03-06T07:30:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.862+0000 7f7602a00640 1 Processor -- start 2024-03-06T07:30:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.862+0000 7f7602a00640 1 -- start start 2024-03-06T07:30:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f7602a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 0x7f75fc157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f7602a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f7602a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f7602a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0b36a0 con 0x7f75fc0af1a0 2024-03-06T07:30:48.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f7602a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0b3520 con 0x7f75fc0cdc20 2024-03-06T07:30:48.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f7602a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75fc0b3820 con 0x7f75fc0af580 2024-03-06T07:30:48.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f75fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 0x7f75fc157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f75fbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 0x7f75fc157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54706/0 (socket says 172.21.15.6:54706) 2024-03-06T07:30:48.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.863+0000 7f75fbe00640 1 -- 172.21.15.6:0/3765497344 learned_addr learned my addr 172.21.15.6:0/3765497344 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:48.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fb400640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f7600a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fbe00640 1 -- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 msgr2=0x7f75fc1553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fbe00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc1553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fbe00640 1 -- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 msgr2=0x7f75fc149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fbe00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fbe00640 1 -- 172.21.15.6:0/3765497344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75fc06cd60 con 0x7f75fc0af1a0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.864+0000 7f75fbe00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 0x7f75fc157110 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c400 tx=0x7f75ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.865+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0052d0 con 0x7f75fc0af1a0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.865+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec005490 con 0x7f75fc0af1a0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.865+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec007260 con 0x7f75fc0af1a0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.865+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75fc06d040 con 0x7f75fc0af1a0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.865+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75fc06db20 con 0x7f75fc0af1a0 2024-03-06T07:30:48.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.866+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75fc069510 con 0x7f75fc0af1a0 2024-03-06T07:30:48.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.867+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75ec007400 con 0x7f75fc0af1a0 2024-03-06T07:30:48.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.869+0000 7f75f3e00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75dc06b4d0 0x7f75dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.869+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75ec089700 con 0x7f75fc0af1a0 2024-03-06T07:30:48.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.869+0000 7f75fb400640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75dc06b4d0 0x7f75dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.870+0000 7f75fb400640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75dc06b4d0 0x7f75dc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009560 tx=0x7f75e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:48.872+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75ec057e90 con 0x7f75fc0af1a0 2024-03-06T07:30:49.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.089+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f75fc04f3a0 con 0x7f75fc0af1a0 2024-03-06T07:30:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.090+0000 7f75f3e00640 1 -- 172.21.15.6:0/3765497344 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7f75ec05bf30 con 0x7f75fc0af1a0 2024-03-06T07:30:49.092 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.092+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75dc06b4d0 msgr2=0x7f75dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.092+0000 7f7602a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75dc06b4d0 0x7f75dc06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009560 tx=0x7f75e8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.092+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 msgr2=0x7f75fc157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.092+0000 7f7602a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 0x7f75fc157110 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c400 tx=0x7f75ec00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 shutdown_connections 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75dc06b4d0 0x7f75dc06d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75fc0cdc20 0x7f75fc149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75fc0af580 0x7f75fc1553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 --2- 172.21.15.6:0/3765497344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75fc0af1a0 0x7f75fc157110 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 >> 172.21.15.6:0/3765497344 conn(0x7f75fc06afb0 msgr2=0x7f75fc15c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 shutdown_connections 2024-03-06T07:30:49.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.093+0000 7f7602a00640 1 -- 172.21.15.6:0/3765497344 wait complete. 2024-03-06T07:30:49.112 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:49.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:49.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:49.300 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:30:49.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:49.300 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:49.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.566+0000 7fb26ec00640 1 Processor -- start 2024-03-06T07:30:49.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.566+0000 7fb26ec00640 1 -- start start 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.568+0000 7fb26ec00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb26815e2a0 0x7fb2680cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.568+0000 7fb26ec00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb2680b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.568+0000 7fb26ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2680d1120 con 0x7fb26815dec0 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.568+0000 7fb26ec00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2680b0ea0 con 0x7fb26815e2a0 2024-03-06T07:30:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.568+0000 7fb26ec00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680d1400 con 0x7fb2680cf460 2024-03-06T07:30:49.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.569+0000 7fb26d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb26815dec0 0x7fb2680ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54672/0 (socket says 172.21.15.6:54672) 2024-03-06T07:30:49.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.569+0000 7fb26d800640 1 -- 172.21.15.6:0/1723840193 learned_addr learned my addr 172.21.15.6:0/1723840193 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:49.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.569+0000 7fb26ce00640 1 --2- 172.21.15.6:0/1723840193 >> v2:172.21.15.6:3300/0 conn(0x7fb26815e2a0 0x7fb2680cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.569+0000 7fb26e200640 1 --2- 172.21.15.6:0/1723840193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb2680b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.570+0000 7fb26ce00640 1 -- 172.21.15.6:0/1723840193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 msgr2=0x7fb2680b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.570+0000 7fb26ce00640 1 --2- 172.21.15.6:0/1723840193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb2680b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.570+0000 7fb26ce00640 1 -- 172.21.15.6:0/1723840193 >> v1:172.21.15.6:6789/0 conn(0x7fb26815dec0 legacy=0x7fb2680ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.570+0000 7fb26ce00640 1 -- 172.21.15.6:0/1723840193 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680b0b30 con 0x7fb26815e2a0 2024-03-06T07:30:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.570+0000 7fb267e00640 1 -- 172.21.15.6:0/1723840193 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582459120 0 0) 0x7fb2680d1120 con 0x7fb26815dec0 2024-03-06T07:30:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.571+0000 7fb26ce00640 1 --2- 172.21.15.6:0/1723840193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815e2a0 0x7fb2680cef20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb2540097a0 tx=0x7fb25402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824da9cee85a1264 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.571+0000 7fb267e00640 1 -- 172.21.15.6:0/1723840193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254035020 con 0x7fb26815e2a0 2024-03-06T07:30:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.571+0000 7fb267e00640 1 -- 172.21.15.6:0/1723840193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb254031830 con 0x7fb26815e2a0 2024-03-06T07:30:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.571+0000 7fb267e00640 1 -- 172.21.15.6:0/1723840193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254031c00 con 0x7fb26815e2a0 2024-03-06T07:30:49.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 -- 172.21.15.6:0/1723840193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815e2a0 msgr2=0x7fb2680cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1723840193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815e2a0 0x7fb2680cef20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb2540097a0 tx=0x7fb25402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 -- 172.21.15.6:0/1723840193 shutdown_connections 2024-03-06T07:30:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1723840193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb2680b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1723840193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815e2a0 0x7fb2680cef20 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 -- 172.21.15.6:0/1723840193 >> 172.21.15.6:0/1723840193 conn(0x7fb2680d3d40 msgr2=0x7fb2680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 -- 172.21.15.6:0/1723840193 shutdown_connections 2024-03-06T07:30:49.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.572+0000 7fb26ec00640 1 -- 172.21.15.6:0/1723840193 wait complete. 2024-03-06T07:30:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.575+0000 7fb26ec00640 1 Processor -- start 2024-03-06T07:30:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.575+0000 7fb26ec00640 1 -- start start 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ec00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb268153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 0x7fb2680cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ec00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb26815e2a0 0x7fb2680cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680d0ec0 con 0x7fb26815dec0 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ec00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680d0d40 con 0x7fb26815e2a0 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ec00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2680b1870 con 0x7fb2680cf460 2024-03-06T07:30:49.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 0x7fb2680cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 0x7fb2680cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54728/0 (socket says 172.21.15.6:54728) 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.577+0000 7fb26ce00640 1 -- 172.21.15.6:0/1859804066 learned_addr learned my addr 172.21.15.6:0/1859804066 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26d800640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb268153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26e200640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb26815e2a0 0x7fb2680cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26ce00640 1 -- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 msgr2=0x7fb268153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26ce00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb268153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26ce00640 1 -- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb26815e2a0 msgr2=0x7fb2680cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26ce00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb26815e2a0 0x7fb2680cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26ce00640 1 -- 172.21.15.6:0/1859804066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2680cb7d0 con 0x7fb26815dec0 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26e200640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb26815e2a0 0x7fb2680cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:49.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.578+0000 7fb26ce00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 0x7fb2680cae10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb2540098c0 tx=0x7fb2540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.579+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254040070 con 0x7fb26815dec0 2024-03-06T07:30:49.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.579+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25403e070 con 0x7fb26815dec0 2024-03-06T07:30:49.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.579+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2680cbab0 con 0x7fb26815dec0 2024-03-06T07:30:49.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.579+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254035040 con 0x7fb26815dec0 2024-03-06T07:30:49.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.579+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2680cc070 con 0x7fb26815dec0 2024-03-06T07:30:49.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.580+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26804f330 con 0x7fb26815dec0 2024-03-06T07:30:49.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.581+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb254038020 con 0x7fb26815dec0 2024-03-06T07:30:49.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.582+0000 7fb265600640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:49.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.582+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2540ab8b0 con 0x7fb26815dec0 2024-03-06T07:30:49.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.582+0000 7fb26d800640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:49.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.584+0000 7fb26d800640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb24806b4d0 0x7fb24806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb25c000f60 tx=0x7fb25c007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:49.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.587+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb25407a6a0 con 0x7fb26815dec0 2024-03-06T07:30:49.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.828+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb268160700 con 0x7fb26815dec0 2024-03-06T07:30:49.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.829+0000 7fb265600640 1 -- 172.21.15.6:0/1859804066 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fb254079fa0 con 0x7fb26815dec0 2024-03-06T07:30:49.832 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:49.832 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:30:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb24806b4d0 msgr2=0x7fb24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb24806b4d0 0x7fb24806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb25c000f60 tx=0x7fb25c007b70 comp rx=0 tx=0).stop 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 msgr2=0x7fb2680cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 0x7fb2680cae10 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb2540098c0 tx=0x7fb2540046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 shutdown_connections 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb24806b4d0 0x7fb24806d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb26815e2a0 0x7fb2680cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb26815dec0 0x7fb2680cae10 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 --2- 172.21.15.6:0/1859804066 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb2680cf460 0x7fb268153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 >> 172.21.15.6:0/1859804066 conn(0x7fb2680d3d40 msgr2=0x7fb2680ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 shutdown_connections 2024-03-06T07:30:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:49.832+0000 7fb26ec00640 1 -- 172.21.15.6:0/1859804066 wait complete. 2024-03-06T07:30:49.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:30:50.083 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:30:50.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:50.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.297+0000 7fd5bb600640 1 Processor -- start 2024-03-06T07:30:50.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.298+0000 7fd5bb600640 1 -- start start 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.300+0000 7fd5bb600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b4162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.300+0000 7fd5bb600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5b4072da0 0x7fd5b40cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.300+0000 7fd5bb600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd5b40712c0 0x7fd5b406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.300+0000 7fd5bb600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b40ceca0 con 0x7fd5b4072da0 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.300+0000 7fd5bb600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b40ce090 con 0x7fd5b40712c0 2024-03-06T07:30:50.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.300+0000 7fd5bb600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b40ce370 con 0x7fd5b4070ee0 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5b4072da0 0x7fd5b40cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd5b4072da0 0x7fd5b40cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54730/0 (socket says 172.21.15.6:54730) 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 -- 172.21.15.6:0/2344291075 learned_addr learned my addr 172.21.15.6:0/2344291075 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b8c00640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b4162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b9600640 1 --2- 172.21.15.6:0/2344291075 >> v2:172.21.15.178:3300/0 conn(0x7fd5b40712c0 0x7fd5b406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 -- 172.21.15.6:0/2344291075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 msgr2=0x7fd5b4162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b4162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 -- 172.21.15.6:0/2344291075 >> v2:172.21.15.178:3300/0 conn(0x7fd5b40712c0 msgr2=0x7fd5b406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 --2- 172.21.15.6:0/2344291075 >> v2:172.21.15.178:3300/0 conn(0x7fd5b40712c0 0x7fd5b406e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b3e00640 1 -- 172.21.15.6:0/2344291075 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40ce650 con 0x7fd5b4072da0 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b8c00640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b4162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:50.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.301+0000 7fd5b9600640 1 --2- 172.21.15.6:0/2344291075 >> v2:172.21.15.178:3300/0 conn(0x7fd5b40712c0 0x7fd5b406e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.302+0000 7fd5b3e00640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b4072da0 0x7fd5b40cf130 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd5b406f040 tx=0x7fd5a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db82b75a579fc150 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5b3400640 1 -- 172.21.15.6:0/2344291075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0035020 con 0x7fd5b4072da0 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5b3400640 1 -- 172.21.15.6:0/2344291075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0031870 con 0x7fd5b4072da0 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5b3400640 1 -- 172.21.15.6:0/2344291075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0031c40 con 0x7fd5b4072da0 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 -- 172.21.15.6:0/2344291075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b4072da0 msgr2=0x7fd5b40cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b4072da0 0x7fd5b40cf130 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd5b406f040 tx=0x7fd5a002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 -- 172.21.15.6:0/2344291075 shutdown_connections 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 --2- 172.21.15.6:0/2344291075 >> v2:172.21.15.178:3300/0 conn(0x7fd5b40712c0 0x7fd5b406e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b4072da0 0x7fd5b40cf130 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 --2- 172.21.15.6:0/2344291075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b4162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.303+0000 7fd5bb600640 1 -- 172.21.15.6:0/2344291075 >> 172.21.15.6:0/2344291075 conn(0x7fd5b406afb0 msgr2=0x7fd5b406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.304+0000 7fd5bb600640 1 -- 172.21.15.6:0/2344291075 shutdown_connections 2024-03-06T07:30:50.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.304+0000 7fd5bb600640 1 -- 172.21.15.6:0/2344291075 wait complete. 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.305+0000 7fd5bb600640 1 Processor -- start 2024-03-06T07:30:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.305+0000 7fd5bb600640 1 -- start start 2024-03-06T07:30:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.306+0000 7fd5bb600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b406d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.306+0000 7fd5bb600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 0x7fd5b4150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.306+0000 7fd5bb600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5b4072da0 0x7fd5b414fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.306+0000 7fd5bb600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b40cdda0 con 0x7fd5b40712c0 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.306+0000 7fd5bb600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b40cdc20 con 0x7fd5b4072da0 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.306+0000 7fd5bb600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b406e600 con 0x7fd5b4070ee0 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 0x7fd5b4150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 0x7fd5b4150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54736/0 (socket says 172.21.15.6:54736) 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 -- 172.21.15.6:0/3196019565 learned_addr learned my addr 172.21.15.6:0/3196019565 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b9600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5b4072da0 0x7fd5b414fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 -- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 msgr2=0x7fd5b406d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b406d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 -- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5b4072da0 msgr2=0x7fd5b414fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5b4072da0 0x7fd5b414fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b3e00640 1 -- 172.21.15.6:0/3196019565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b409ab40 con 0x7fd5b40712c0 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.307+0000 7fd5b9600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5b4072da0 0x7fd5b414fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:50.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.308+0000 7fd5b3e00640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 0x7fd5b4150c20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd5a002b950 tx=0x7fd5a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.308+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0035040 con 0x7fd5b40712c0 2024-03-06T07:30:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.308+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0045480 con 0x7fd5b40712c0 2024-03-06T07:30:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.308+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0045870 con 0x7fd5b40712c0 2024-03-06T07:30:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.308+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b409ae20 con 0x7fd5b40712c0 2024-03-06T07:30:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.308+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b409b3e0 con 0x7fd5b40712c0 2024-03-06T07:30:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.309+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b406ea10 con 0x7fd5b40712c0 2024-03-06T07:30:50.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.310+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5a0038020 con 0x7fd5b40712c0 2024-03-06T07:30:50.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.312+0000 7fd5b0c00640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd59406b4d0 0x7fd59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.312+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5a00ac680 con 0x7fd5b40712c0 2024-03-06T07:30:50.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.313+0000 7fd5b8c00640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd59406b4d0 0x7fd59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.314+0000 7fd5b8c00640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd59406b4d0 0x7fd59406d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800a8b0 tx=0x7fd5a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.316+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5b406ea10 con 0x7fd5b40712c0 2024-03-06T07:30:50.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.552+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5b40b52d0 con 0x7fd5b40712c0 2024-03-06T07:30:50.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.553+0000 7fd5b0c00640 1 -- 172.21.15.6:0/3196019565 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd5a007b4f0 con 0x7fd5b40712c0 2024-03-06T07:30:50.555 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:50.556 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:30:50.556 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:30:50.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd59406b4d0 msgr2=0x7fd59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd59406b4d0 0x7fd59406d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800a8b0 tx=0x7fd5a8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 msgr2=0x7fd5b4150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 0x7fd5b4150c20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd5a002b950 tx=0x7fd5a00046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 shutdown_connections 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd59406b4d0 0x7fd59406d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5b4072da0 0x7fd5b414fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd5b40712c0 0x7fd5b4150c20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 --2- 172.21.15.6:0/3196019565 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5b4070ee0 0x7fd5b406d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 >> 172.21.15.6:0/3196019565 conn(0x7fd5b406afb0 msgr2=0x7fd5b40b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 shutdown_connections 2024-03-06T07:30:50.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:50.556+0000 7fd5bb600640 1 -- 172.21.15.6:0/3196019565 wait complete. 2024-03-06T07:30:50.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:30:50.920 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:30:50.920 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:30:50.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:30:51.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.165+0000 7fc039a00640 1 Processor -- start 2024-03-06T07:30:51.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.165+0000 7fc039a00640 1 -- start start 2024-03-06T07:30:51.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.166+0000 7fc039a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc03415efc0 0x7fc0340cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc039a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0341621d0 0x7fc034070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc039a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0341627f0 con 0x7fc0340cce90 2024-03-06T07:30:51.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc039a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03415f3a0 con 0x7fc03415efc0 2024-03-06T07:30:51.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc039a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03406d190 con 0x7fc0341621d0 2024-03-06T07:30:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc033e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc03415efc0 0x7fc0340cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc033e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc03415efc0 0x7fc0340cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:54922/0 (socket says 172.21.15.6:54922) 2024-03-06T07:30:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc033e00640 1 -- 172.21.15.6:0/3941776449 learned_addr learned my addr 172.21.15.6:0/3941776449 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.167+0000 7fc039000640 1 --2- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0341621d0 0x7fc034070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.168+0000 7fc033e00640 1 -- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0341621d0 msgr2=0x7fc034070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:51.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.168+0000 7fc033e00640 1 --2- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0341621d0 0x7fc034070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.168+0000 7fc033e00640 1 -- 172.21.15.6:0/3941776449 >> v1:172.21.15.178:6789/0 conn(0x7fc0340cce90 legacy=0x7fc034161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.168+0000 7fc033e00640 1 -- 172.21.15.6:0/3941776449 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03406c860 con 0x7fc03415efc0 2024-03-06T07:30:51.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.170+0000 7fc033e00640 1 --2- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc03415efc0 0x7fc0340cc950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc024009870 tx=0x7fc02402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d92cb00aa5197df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.170+0000 7fc033400640 1 -- 172.21.15.6:0/3941776449 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024034070 con 0x7fc03415efc0 2024-03-06T07:30:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.170+0000 7fc033400640 1 -- 172.21.15.6:0/3941776449 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024030840 con 0x7fc03415efc0 2024-03-06T07:30:51.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.170+0000 7fc033400640 1 -- 172.21.15.6:0/3941776449 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024030c10 con 0x7fc03415efc0 2024-03-06T07:30:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 -- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc03415efc0 msgr2=0x7fc0340cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 --2- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc03415efc0 0x7fc0340cc950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc024009870 tx=0x7fc02402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 -- 172.21.15.6:0/3941776449 shutdown_connections 2024-03-06T07:30:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 --2- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0341621d0 0x7fc034070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 --2- 172.21.15.6:0/3941776449 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc03415efc0 0x7fc0340cc950 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 -- 172.21.15.6:0/3941776449 >> 172.21.15.6:0/3941776449 conn(0x7fc03406b3d0 msgr2=0x7fc03406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 -- 172.21.15.6:0/3941776449 shutdown_connections 2024-03-06T07:30:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.171+0000 7fc039a00640 1 -- 172.21.15.6:0/3941776449 wait complete. 2024-03-06T07:30:51.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.172+0000 7fc039a00640 1 Processor -- start 2024-03-06T07:30:51.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.173+0000 7fc039a00640 1 -- start start 2024-03-06T07:30:51.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.173+0000 7fc039a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 0x7fc034157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc039a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc03415efc0 0x7fc034137560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc039a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0341621d0 0x7fc034131a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc039a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03406d620 con 0x7fc0340cce90 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc039a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03406ce20 con 0x7fc0341621d0 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc039a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc034144e30 con 0x7fc03415efc0 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 0x7fc034157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 0x7fc034157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54746/0 (socket says 172.21.15.6:54746) 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 -- 172.21.15.6:0/1075079705 learned_addr learned my addr 172.21.15.6:0/1075079705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc039000640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0341621d0 0x7fc034131a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc02be00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc03415efc0 0x7fc034137560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 -- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc03415efc0 msgr2=0x7fc034137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc03415efc0 0x7fc034137560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 -- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0341621d0 msgr2=0x7fc034131a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0341621d0 0x7fc034131a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.174+0000 7fc033e00640 1 -- 172.21.15.6:0/1075079705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0341364c0 con 0x7fc0340cce90 2024-03-06T07:30:51.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.175+0000 7fc033e00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 0x7fc034157f00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc02402b9c0 tx=0x7fc024004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.175+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024048070 con 0x7fc0340cce90 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.175+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc024030640 con 0x7fc0340cce90 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.175+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc024034040 con 0x7fc0340cce90 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.175+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0341367a0 con 0x7fc0340cce90 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.175+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc034136d60 con 0x7fc0340cce90 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.177+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03404f330 con 0x7fc0340cce90 2024-03-06T07:30:51.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.178+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc02404b070 con 0x7fc0340cce90 2024-03-06T07:30:51.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.179+0000 7fc030c00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc01006b4d0 0x7fc01006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.179+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0240ac090 con 0x7fc0340cce90 2024-03-06T07:30:51.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.179+0000 7fc02be00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc01006b4d0 0x7fc01006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.180+0000 7fc02be00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc01006b4d0 0x7fc01006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc034146f50 tx=0x7fc01c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.183+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc02407af10 con 0x7fc0340cce90 2024-03-06T07:30:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.422+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc0340af2f0 con 0x7fc0340cce90 2024-03-06T07:30:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.423+0000 7fc030c00640 1 -- 172.21.15.6:0/1075079705 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc024030810 con 0x7fc0340cce90 2024-03-06T07:30:51.425 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.425+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc01006b4d0 msgr2=0x7fc01006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.425+0000 7fc039a00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc01006b4d0 0x7fc01006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc034146f50 tx=0x7fc01c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.425+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 msgr2=0x7fc034157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.425+0000 7fc039a00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 0x7fc034157f00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc02402b9c0 tx=0x7fc024004730 comp rx=0 tx=0).stop 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 shutdown_connections 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc01006b4d0 0x7fc01006d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0341621d0 0x7fc034131a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc03415efc0 0x7fc034137560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 --2- 172.21.15.6:0/1075079705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0340cce90 0x7fc034157f00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 >> 172.21.15.6:0/1075079705 conn(0x7fc03406b3d0 msgr2=0x7fc0340b0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 shutdown_connections 2024-03-06T07:30:51.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:51.426+0000 7fc039a00640 1 -- 172.21.15.6:0/1075079705 wait complete. 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:30:51.454 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:51.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.455 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:30:51.503 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:30:51.503 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:51.504 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42406sec preferred_lft 42406sec 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.505 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:30:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:30:51.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.507 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:30:51.647 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:30:51.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.652 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:30:51.706 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:30:51.707 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:30:51.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.708 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:30:51.708 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:30:51.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:30:51.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:30:51.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:30:51.708 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:30:51.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.887 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:30:51.939 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:30:51.940 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:51.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:51.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:30:52.036 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:52.037 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:52.093 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:30:52.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:30:52.093 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:52.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:52.136 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:30:52.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:52.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:52.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:52.304 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:52.304 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:52.336 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:30:52.336 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:30:52.336 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:30:52.336 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:30:52.336 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:30:52.338 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:30:52.338 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:30:52.338 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:30:52.338 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:30:52.338 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:30:52.342 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:30:52.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:52.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:52.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:52.475 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:30:52.529 INFO:teuthology.orchestra.run.smithi006.stdout:client6189 2024-03-06T07:30:52.529 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:52.529 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:52.573 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:52.573 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:52.670 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:30:52.670 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> n = 75 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:52.671 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-06T07:30:52.672 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-06T07:30:52.673 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:30:53.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:30:53.767 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:30:53.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:53.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:30:54.041 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:30:54.042 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:54.042 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:54.099 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:54.099 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:54.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.403+0000 7f0f34600640 1 Processor -- start 2024-03-06T07:30:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.404+0000 7f0f34600640 1 -- start start 2024-03-06T07:30:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f34600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f2c0d0040 0x7f0f2c0d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f34600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c0af220 con 0x7f0f2c15fc60 2024-03-06T07:30:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f34600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c06ba00 con 0x7f0f2c145a90 2024-03-06T07:30:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f34600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0cd830 con 0x7f0f2c0d0040 2024-03-06T07:30:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f31200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0f2c0d0040 0x7f0f2c0d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f32600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0f2c15fc60 0x7f0f2c1600b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43112/0 (socket says 172.21.15.6:43112) 2024-03-06T07:30:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.406+0000 7f0f32600640 1 -- 172.21.15.6:0/2315345616 learned_addr learned my addr 172.21.15.6:0/2315345616 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.407+0000 7f0f31200640 1 -- 172.21.15.6:0/2315345616 >> v1:172.21.15.178:6789/0 conn(0x7f0f2c145a90 legacy=0x7f0f2c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.407+0000 7f0f2be00640 1 -- 172.21.15.6:0/2315345616 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482070132 0 0) 0x7f0f24001490 con 0x7f0f2c15fc60 2024-03-06T07:30:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.407+0000 7f0f31200640 1 -- 172.21.15.6:0/2315345616 >> v1:172.21.15.6:6789/0 conn(0x7f0f2c15fc60 legacy=0x7f0f2c1600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.407+0000 7f0f31200640 1 -- 172.21.15.6:0/2315345616 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c0cdb10 con 0x7f0f2c0d0040 2024-03-06T07:30:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.408+0000 7f0f31200640 1 --2- 172.21.15.6:0/2315345616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c0d04c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0f180097a0 tx=0x7f0f1802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b47189644227ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.409+0000 7f0f2be00640 1 -- 172.21.15.6:0/2315345616 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18034070 con 0x7f0f2c0d0040 2024-03-06T07:30:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.409+0000 7f0f2be00640 1 -- 172.21.15.6:0/2315345616 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f18030610 con 0x7f0f2c0d0040 2024-03-06T07:30:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.409+0000 7f0f2be00640 1 -- 172.21.15.6:0/2315345616 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f180309e0 con 0x7f0f2c0d0040 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.409+0000 7f0f34600640 1 -- 172.21.15.6:0/2315345616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c0d0040 msgr2=0x7f0f2c0d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.409+0000 7f0f34600640 1 --2- 172.21.15.6:0/2315345616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c0d04c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0f180097a0 tx=0x7f0f1802b960 comp rx=0 tx=0).stop 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.410+0000 7f0f34600640 1 -- 172.21.15.6:0/2315345616 shutdown_connections 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.410+0000 7f0f34600640 1 --2- 172.21.15.6:0/2315345616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c0d04c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.410+0000 7f0f34600640 1 -- 172.21.15.6:0/2315345616 >> 172.21.15.6:0/2315345616 conn(0x7f0f2c0d0d80 msgr2=0x7f0f2c0d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.410+0000 7f0f34600640 1 -- 172.21.15.6:0/2315345616 shutdown_connections 2024-03-06T07:30:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.410+0000 7f0f34600640 1 -- 172.21.15.6:0/2315345616 wait complete. 2024-03-06T07:30:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.411+0000 7f0f34600640 1 Processor -- start 2024-03-06T07:30:54.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.412+0000 7f0f34600640 1 -- start start 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.413+0000 7f0f34600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c06c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.413+0000 7f0f34600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f2c145a90 0x7f0f2c15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.413+0000 7f0f34600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 0x7f0f2c06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.413+0000 7f0f34600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c06b6c0 con 0x7f0f2c15fc60 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.413+0000 7f0f34600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0ce0d0 con 0x7f0f2c0d0040 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.413+0000 7f0f34600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c0ce5f0 con 0x7f0f2c145a90 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 0x7f0f2c06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 0x7f0f2c06eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49216/0 (socket says 172.21.15.6:49216) 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 -- 172.21.15.6:0/4088211262 learned_addr learned my addr 172.21.15.6:0/4088211262 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f31200640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f2c145a90 0x7f0f2c15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f31c00640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c06c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 -- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f2c145a90 msgr2=0x7f0f2c15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f2c145a90 0x7f0f2c15b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 -- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c0d0040 msgr2=0x7f0f2c06c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c06c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.414+0000 7f0f32600640 1 -- 172.21.15.6:0/4088211262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c09dd40 con 0x7f0f2c15fc60 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.415+0000 7f0f32600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 0x7f0f2c06eb70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400c780 tx=0x7f0f2400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.415+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24018070 con 0x7f0f2c15fc60 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.415+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2400a640 con 0x7f0f2c15fc60 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.415+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c09e020 con 0x7f0f2c15fc60 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.415+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2400aa30 con 0x7f0f2c15fc60 2024-03-06T07:30:54.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.415+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c09e5e0 con 0x7f0f2c15fc60 2024-03-06T07:30:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.416+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c09e820 con 0x7f0f2c15fc60 2024-03-06T07:30:54.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.417+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f2401b070 con 0x7f0f2c15fc60 2024-03-06T07:30:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.419+0000 7f0f29600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.419+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f240898e0 con 0x7f0f2c15fc60 2024-03-06T07:30:54.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.420+0000 7f0f31c00640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.421+0000 7f0f31c00640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0f20000f60 tx=0x7f0f20009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.423+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f2c09e820 con 0x7f0f2c15fc60 2024-03-06T07:30:54.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.655+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f2c1331b0 con 0x7f0f2c15fc60 2024-03-06T07:30:54.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.657+0000 7f0f29600640 1 -- 172.21.15.6:0/4088211262 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f0f2c0af220 con 0x7f0f2c15fc60 2024-03-06T07:30:54.659 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:54.659 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":58,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:22.633658+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.659+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f0c06b4d0 msgr2=0x7f0f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.659+0000 7f0f34600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0f20000f60 tx=0x7f0f20009040 comp rx=0 tx=0).stop 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.659+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 msgr2=0x7f0f2c06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.659+0000 7f0f34600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 0x7f0f2c06eb70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400c780 tx=0x7f0f2400cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 shutdown_connections 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f2c15fc60 0x7f0f2c06eb70 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f2c145a90 0x7f0f2c15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 --2- 172.21.15.6:0/4088211262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c0d0040 0x7f0f2c06c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 >> 172.21.15.6:0/4088211262 conn(0x7f0f2c0d0d80 msgr2=0x7f0f2c070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 shutdown_connections 2024-03-06T07:30:54.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.660+0000 7f0f34600640 1 -- 172.21.15.6:0/4088211262 wait complete. 2024-03-06T07:30:54.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:19.028665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:20.117374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:54.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:54.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.914+0000 7f17ecc00640 1 Processor -- start 2024-03-06T07:30:54.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.915+0000 7f17ecc00640 1 -- start start 2024-03-06T07:30:54.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.916+0000 7f17ecc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 0x7f17e80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.916+0000 7f17ecc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e80ceca0 con 0x7f17e815dee0 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.916+0000 7f17ecc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e80cef80 con 0x7f17e80d4860 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.916+0000 7f17ecc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cf5d0 con 0x7f17e80d0d40 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.917+0000 7f17e7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f17e815dee0 0x7f17e80d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43120/0 (socket says 172.21.15.6:43120) 2024-03-06T07:30:54.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.917+0000 7f17e7400640 1 -- 172.21.15.6:0/3170927695 learned_addr learned my addr 172.21.15.6:0/3170927695 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:54.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.917+0000 7f17e7e00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 0x7f17e80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.917+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193569077 0 0) 0x7f17e80ceca0 con 0x7f17e815dee0 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.917+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17cc003520 con 0x7f17e815dee0 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702719250 0 0) 0x7f17e80cef80 con 0x7f17e80d4860 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e80ceca0 con 0x7f17e80d4860 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 469703430 0 0) 0x7f17cc003520 con 0x7f17e815dee0 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 msgr2=0x7f17e80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 0x7f17e80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 >> v1:172.21.15.178:6789/0 conn(0x7f17e80d4860 legacy=0x7f17e80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80cf260 con 0x7f17e815dee0 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f17d8001690 con 0x7f17e815dee0 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 >> v1:172.21.15.6:6789/0 conn(0x7f17e815dee0 legacy=0x7f17e80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17cc003b70 0x7f17cc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 0x7f17cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cf5d0 con 0x7f17cc004800 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc00baa0 con 0x7f17cc003b70 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc00bca0 con 0x7f17e80d0d40 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.918+0000 7f17dfe00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17e7400640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17cc003b70 0x7f17cc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17e7e00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 0x7f17cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17dfe00640 1 -- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 msgr2=0x7f17cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17dfe00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e80d0d40 0x7f17cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17dfe00640 1 -- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17cc003b70 msgr2=0x7f17cc003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17dfe00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17cc003b70 0x7f17cc003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17dfe00640 1 -- 172.21.15.6:0/3170927695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc003520 con 0x7f17cc004800 2024-03-06T07:30:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.919+0000 7f17e7400640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17cc003b70 0x7f17cc003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17dfe00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17cc004c80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f17d00018a0 tx=0x7f17cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acda237297349517 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0037070 con 0x7f17cc004800 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d00052d0 con 0x7f17cc004800 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17e6a00640 1 -- 172.21.15.6:0/3170927695 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d00014f0 con 0x7f17cc004800 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17ecc00640 1 -- 172.21.15.6:0/3170927695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 msgr2=0x7f17cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17ecc00640 1 --2- 172.21.15.6:0/3170927695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17cc004c80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f17d00018a0 tx=0x7f17cc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.921+0000 7f17e7e00640 1 -- 172.21.15.6:0/3170927695 reap_dead start 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.922+0000 7f17ecc00640 1 -- 172.21.15.6:0/3170927695 shutdown_connections 2024-03-06T07:30:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.922+0000 7f17ecc00640 1 -- 172.21.15.6:0/3170927695 >> 172.21.15.6:0/3170927695 conn(0x7f17e80d3cd0 msgr2=0x7f17e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.922+0000 7f17ecc00640 1 -- 172.21.15.6:0/3170927695 shutdown_connections 2024-03-06T07:30:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.922+0000 7f17ecc00640 1 -- 172.21.15.6:0/3170927695 wait complete. 2024-03-06T07:30:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.924+0000 7f17ecc00640 1 Processor -- start 2024-03-06T07:30:54.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.924+0000 7f17ecc00640 1 -- start start 2024-03-06T07:30:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17ecc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17e814bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17ecc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e8149200 0x7f17e814aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17ecc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17e81487d0 0x7f17e80cff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17ecc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cfa60 con 0x7f17cc004800 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17ecc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e815e2c0 con 0x7f17e81487d0 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17ecc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cfbe0 con 0x7f17e8149200 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17e814bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17e7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17e814bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49226/0 (socket says 172.21.15.6:49226) 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17e7400640 1 -- 172.21.15.6:0/700016337 learned_addr learned my addr 172.21.15.6:0/700016337 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17dfe00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e8149200 0x7f17e814aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.925+0000 7f17e7e00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17e81487d0 0x7f17e80cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17e7400640 1 -- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e8149200 msgr2=0x7f17e814aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17e7400640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e8149200 0x7f17e814aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17e7400640 1 -- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17e81487d0 msgr2=0x7f17e80cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17e7400640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17e81487d0 0x7f17e80cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17e7400640 1 -- 172.21.15.6:0/700016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8149970 con 0x7f17cc004800 2024-03-06T07:30:54.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17e7400640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17e814bdc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f17d800e700 tx=0x7f17d800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d800c6c0 con 0x7f17cc004800 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d8018070 con 0x7f17cc004800 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d800ca60 con 0x7f17cc004800 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e8149c50 con 0x7f17cc004800 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.926+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e814a210 con 0x7f17cc004800 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.928+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e804f330 con 0x7f17cc004800 2024-03-06T07:30:54.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.928+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17d800cc00 con 0x7f17cc004800 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.929+0000 7f17df400640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17c406b4d0 0x7f17c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.930+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17d8089fe0 con 0x7f17cc004800 2024-03-06T07:30:54.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.930+0000 7f17dfe00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17c406b4d0 0x7f17c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.931+0000 7f17dfe00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17c406b4d0 0x7f17c406d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f17d0034000 tx=0x7f17d0031000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:54.934+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17d8058e20 con 0x7f17cc004800 2024-03-06T07:30:55.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.134+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17e806c760 con 0x7f17cc004800 2024-03-06T07:30:55.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.134+0000 7f17df400640 1 -- 172.21.15.6:0/700016337 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17d8058720 con 0x7f17cc004800 2024-03-06T07:30:55.137 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:30:55.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17c406b4d0 msgr2=0x7f17c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17c406b4d0 0x7f17c406d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f17d0034000 tx=0x7f17d0031000 comp rx=0 tx=0).stop 2024-03-06T07:30:55.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 msgr2=0x7f17e814bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17e814bdc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f17d800e700 tx=0x7f17d800ebc0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 shutdown_connections 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17c406b4d0 0x7f17c406d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17e81487d0 0x7f17e80cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17e8149200 0x7f17e814aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 --2- 172.21.15.6:0/700016337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17cc004800 0x7f17e814bdc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 >> 172.21.15.6:0/700016337 conn(0x7f17e80d3cd0 msgr2=0x7f17e806cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.137+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 shutdown_connections 2024-03-06T07:30:55.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.138+0000 7f17ecc00640 1 -- 172.21.15.6:0/700016337 wait complete. 2024-03-06T07:30:55.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.369+0000 7fae6ae00640 1 Processor -- start 2024-03-06T07:30:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.370+0000 7fae6ae00640 1 -- start start 2024-03-06T07:30:55.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.371+0000 7fae6ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 0x7fae64168dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.371+0000 7fae6ae00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640b2c00 con 0x7fae640d4890 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.371+0000 7fae6ae00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae64161ae0 con 0x7fae64161700 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.371+0000 7fae6ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b56e0 con 0x7fae640d24f0 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.372+0000 7fae69000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fae640d4890 0x7fae640d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43128/0 (socket says 172.21.15.6:43128) 2024-03-06T07:30:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.372+0000 7fae69000640 1 -- 172.21.15.6:0/1932284161 learned_addr learned my addr 172.21.15.6:0/1932284161 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.372+0000 7fae6a400640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 0x7fae64168dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.372+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371829469 0 0) 0x7fae640b2c00 con 0x7fae640d4890 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.372+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae4c003520 con 0x7fae640d4890 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789509916 0 0) 0x7fae64161ae0 con 0x7fae64161700 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae640b2c00 con 0x7fae64161700 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3838381661 0 0) 0x7fae4c003520 con 0x7fae640d4890 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 msgr2=0x7fae64168dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 0x7fae64168dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 >> v1:172.21.15.178:6789/0 conn(0x7fae64161700 legacy=0x7fae640b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640b2ea0 con 0x7fae640d4890 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fae50001670 con 0x7fae640d4890 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 >> v1:172.21.15.6:6789/0 conn(0x7fae640d4890 legacy=0x7fae640d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 0x7fae4c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c004120 0x7fae4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c004d40 0x7fae4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b56e0 con 0x7fae4c004d40 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c00bb00 con 0x7fae4c004120 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c00bd00 con 0x7fae640d24f0 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.373+0000 7fae6a400640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c004d40 0x7fae4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae69000640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c004120 0x7fae4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae6a400640 1 -- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 msgr2=0x7fae4c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae6a400640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640d24f0 0x7fae4c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae6a400640 1 -- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c004120 msgr2=0x7fae4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae6a400640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c004120 0x7fae4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae6a400640 1 -- 172.21.15.6:0/1932284161 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c003520 con 0x7fae4c004d40 2024-03-06T07:30:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.374+0000 7fae69000640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c004120 0x7fae4c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.375+0000 7fae6a400640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c004d40 0x7fae4c0051c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fae6408d0f0 tx=0x7fae4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d419a0326c659026 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.375+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c039070 con 0x7fae4c004d40 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.375+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c02cbd0 con 0x7fae4c004d40 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.375+0000 7fae63e00640 1 -- 172.21.15.6:0/1932284161 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c009c10 con 0x7fae4c004d40 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6ae00640 1 -- 172.21.15.6:0/1932284161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c004d40 msgr2=0x7fae4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6ae00640 1 --2- 172.21.15.6:0/1932284161 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c004d40 0x7fae4c0051c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fae6408d0f0 tx=0x7fae4c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6a400640 1 -- 172.21.15.6:0/1932284161 reap_dead start 2024-03-06T07:30:55.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6ae00640 1 -- 172.21.15.6:0/1932284161 shutdown_connections 2024-03-06T07:30:55.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6ae00640 1 -- 172.21.15.6:0/1932284161 >> 172.21.15.6:0/1932284161 conn(0x7fae640d3d70 msgr2=0x7fae640d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6ae00640 1 -- 172.21.15.6:0/1932284161 shutdown_connections 2024-03-06T07:30:55.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.376+0000 7fae6ae00640 1 -- 172.21.15.6:0/1932284161 wait complete. 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.378+0000 7fae6ae00640 1 Processor -- start 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.378+0000 7fae6ae00640 1 -- start start 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6ae00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae64154280 0x7fae641f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 0x7fae640cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640cfce0 0x7fae640cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b0d00 con 0x7fae64155880 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6ae00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b0a20 con 0x7fae64154280 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640b2230 con 0x7fae640cfce0 2024-03-06T07:30:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 0x7fae640cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 0x7fae640cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49240/0 (socket says 172.21.15.6:49240) 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 -- 172.21.15.6:0/3721816611 learned_addr learned my addr 172.21.15.6:0/3721816611 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69a00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae64154280 0x7fae641f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae6a400640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640cfce0 0x7fae640cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 -- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640cfce0 msgr2=0x7fae640cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640cfce0 0x7fae640cae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 -- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae64154280 msgr2=0x7fae641f7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae64154280 0x7fae641f7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69000640 1 -- 172.21.15.6:0/3721816611 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae641f8350 con 0x7fae64155880 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.379+0000 7fae69a00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae64154280 0x7fae641f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.380+0000 7fae69000640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 0x7fae640cf7a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fae5000dae0 tx=0x7fae50001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.380+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50019070 con 0x7fae64155880 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.380+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae50004cb0 con 0x7fae64155880 2024-03-06T07:30:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.380+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae50004e60 con 0x7fae64155880 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.380+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae64155e20 con 0x7fae64155880 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.380+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641563e0 con 0x7fae64155880 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.381+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6404f330 con 0x7fae64155880 2024-03-06T07:30:55.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.382+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae500072f0 con 0x7fae64155880 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.384+0000 7fae61600640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae4406b500 0x7fae4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.384+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae50089fd0 con 0x7fae64155880 2024-03-06T07:30:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.384+0000 7fae69a00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae4406b500 0x7fae4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.385+0000 7fae69a00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae4406b500 0x7fae4406d9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fae58007fd0 tx=0x7fae58009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.389+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae50058ee0 con 0x7fae64155880 2024-03-06T07:30:55.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.633+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae64161010 con 0x7fae64155880 2024-03-06T07:30:55.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.634+0000 7fae61600640 1 -- 172.21.15.6:0/3721816611 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae500587e0 con 0x7fae64155880 2024-03-06T07:30:55.636 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:30:55.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae4406b500 msgr2=0x7fae4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae4406b500 0x7fae4406d9c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fae58007fd0 tx=0x7fae58009040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 msgr2=0x7fae640cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 0x7fae640cf7a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fae5000dae0 tx=0x7fae50001e10 comp rx=0 tx=0).stop 2024-03-06T07:30:55.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 shutdown_connections 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae4406b500 0x7fae4406d9c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae640cfce0 0x7fae640cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae64155880 0x7fae640cf7a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 --2- 172.21.15.6:0/3721816611 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae64154280 0x7fae641f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 >> 172.21.15.6:0/3721816611 conn(0x7fae640d3d70 msgr2=0x7fae640d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 shutdown_connections 2024-03-06T07:30:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.637+0000 7fae6ae00640 1 -- 172.21.15.6:0/3721816611 wait complete. 2024-03-06T07:30:55.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.922+0000 7ff72f200640 1 Processor -- start 2024-03-06T07:30:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.922+0000 7ff72f200640 1 -- start start 2024-03-06T07:30:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.924+0000 7ff72f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff73015dee0 0x7ff7300d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.924+0000 7ff72f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7300ceca0 con 0x7ff7300d0d40 2024-03-06T07:30:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.924+0000 7ff72f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7300cef80 con 0x7ff7300d4860 2024-03-06T07:30:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.924+0000 7ff72f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7300cf5d0 con 0x7ff73015dee0 2024-03-06T07:30:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.924+0000 7ff72e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff7300d0d40 0x7ff7300b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43144/0 (socket says 172.21.15.6:43144) 2024-03-06T07:30:55.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.924+0000 7ff72e800640 1 -- 172.21.15.6:0/3867786104 learned_addr learned my addr 172.21.15.6:0/3867786104 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72de00640 1 --2- 172.21.15.6:0/3867786104 >> v2:172.21.15.178:3300/0 conn(0x7ff73015dee0 0x7ff7300d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228048422 0 0) 0x7ff7300ceca0 con 0x7ff7300d0d40 2024-03-06T07:30:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714003520 con 0x7ff7300d0d40 2024-03-06T07:30:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2002283731 0 0) 0x7ff714003520 con 0x7ff7300d0d40 2024-03-06T07:30:55.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 >> v2:172.21.15.178:3300/0 conn(0x7ff73015dee0 msgr2=0x7ff7300d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 --2- 172.21.15.6:0/3867786104 >> v2:172.21.15.178:3300/0 conn(0x7ff73015dee0 0x7ff7300d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 >> v1:172.21.15.178:6789/0 conn(0x7ff7300d4860 legacy=0x7ff7300d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300cf260 con 0x7ff7300d0d40 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff7280019f0 con 0x7ff7300d0d40 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.925+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 >> v1:172.21.15.6:6789/0 conn(0x7ff7300d0d40 legacy=0x7ff7300b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72d400640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7140039b0 0x7ff714003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72d400640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff714004800 0x7ff714004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72d400640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7140051c0 0x7ff71400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300cf5d0 con 0x7ff714004800 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71400c060 con 0x7ff7140051c0 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71400c260 con 0x7ff7140039b0 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff727e00640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff714004800 0x7ff714004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff727e00640 1 -- 172.21.15.6:0/3867786104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7140039b0 msgr2=0x7ff714003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff727e00640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7140039b0 0x7ff714003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff727e00640 1 -- 172.21.15.6:0/3867786104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7140051c0 msgr2=0x7ff71400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff727e00640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7140051c0 0x7ff71400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff727e00640 1 -- 172.21.15.6:0/3867786104 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff714003520 con 0x7ff714004800 2024-03-06T07:30:55.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.926+0000 7ff72e800640 1 -- 172.21.15.6:0/3867786104 reap_dead start 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.927+0000 7ff727e00640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff714004800 0x7ff714004c80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff718001750 tx=0x7ff714005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425fd7babb1a1617 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718001370 con 0x7ff714004800 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7180052d0 con 0x7ff714004800 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72d400640 1 -- 172.21.15.6:0/3867786104 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718003830 con 0x7ff714004800 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72f200640 1 -- 172.21.15.6:0/3867786104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff714004800 msgr2=0x7ff714004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72f200640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff714004800 0x7ff714004c80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff718001750 tx=0x7ff714005830 comp rx=0 tx=0).stop 2024-03-06T07:30:55.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72f200640 1 -- 172.21.15.6:0/3867786104 shutdown_connections 2024-03-06T07:30:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.928+0000 7ff72f200640 1 --2- 172.21.15.6:0/3867786104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff714004800 0x7ff714004c80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.929+0000 7ff72f200640 1 -- 172.21.15.6:0/3867786104 >> 172.21.15.6:0/3867786104 conn(0x7ff7300d3cd0 msgr2=0x7ff7300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.929+0000 7ff72f200640 1 -- 172.21.15.6:0/3867786104 shutdown_connections 2024-03-06T07:30:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.929+0000 7ff72f200640 1 -- 172.21.15.6:0/3867786104 wait complete. 2024-03-06T07:30:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.930+0000 7ff72f200640 1 Processor -- start 2024-03-06T07:30:55.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.931+0000 7ff72f200640 1 -- start start 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff714004800 0x7ff73015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 0x7ff7300cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff73015fa70 0x7ff73015fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300cfa60 con 0x7ff73009e1a0 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73015e2c0 con 0x7ff714004800 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300cfbe0 con 0x7ff73015fa70 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff727e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 0x7ff7300cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff727e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 0x7ff7300cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49254/0 (socket says 172.21.15.6:49254) 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff727e00640 1 -- 172.21.15.6:0/2683387584 learned_addr learned my addr 172.21.15.6:0/2683387584 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72de00640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff714004800 0x7ff73015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.932+0000 7ff72e800640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff73015fa70 0x7ff73015fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff727e00640 1 -- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff73015fa70 msgr2=0x7ff73015fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff727e00640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff73015fa70 0x7ff73015fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff727e00640 1 -- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff714004800 msgr2=0x7ff73015bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff727e00640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff714004800 0x7ff73015bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff727e00640 1 -- 172.21.15.6:0/2683387584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300b4b90 con 0x7ff73009e1a0 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff72e800640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff73015fa70 0x7ff73015fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff727e00640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 0x7ff7300cb7b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff718005040 tx=0x7ff718001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718041020 con 0x7ff73009e1a0 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff718005480 con 0x7ff73009e1a0 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718007a30 con 0x7ff73009e1a0 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.933+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7300b4e70 con 0x7ff73009e1a0 2024-03-06T07:30:55.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.934+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7300cbcf0 con 0x7ff73009e1a0 2024-03-06T07:30:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.935+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73004f330 con 0x7ff73009e1a0 2024-03-06T07:30:55.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.936+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff718044020 con 0x7ff73009e1a0 2024-03-06T07:30:55.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.937+0000 7ff726000640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff70c06b4d0 0x7ff70c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:55.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.937+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7180abb70 con 0x7ff73009e1a0 2024-03-06T07:30:55.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.937+0000 7ff72de00640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff70c06b4d0 0x7ff70c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:55.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.939+0000 7ff72de00640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff70c06b4d0 0x7ff70c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff730160890 tx=0x7ff720008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:55.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:55.943+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff71804b030 con 0x7ff73009e1a0 2024-03-06T07:30:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.142+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff73008d7d0 con 0x7ff73009e1a0 2024-03-06T07:30:56.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.142+0000 7ff726000640 1 -- 172.21.15.6:0/2683387584 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff7180303c0 con 0x7ff73009e1a0 2024-03-06T07:30:56.145 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff70c06b4d0 msgr2=0x7ff70c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff70c06b4d0 0x7ff70c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff730160890 tx=0x7ff720008040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 msgr2=0x7ff7300cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 0x7ff7300cb7b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff718005040 tx=0x7ff718001120 comp rx=0 tx=0).stop 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 shutdown_connections 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff70c06b4d0 0x7ff70c06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff73015fa70 0x7ff73015fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff73009e1a0 0x7ff7300cb7b0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 --2- 172.21.15.6:0/2683387584 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff714004800 0x7ff73015bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.145+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 >> 172.21.15.6:0/2683387584 conn(0x7ff7300d3cd0 msgr2=0x7ff7300d1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.146+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 shutdown_connections 2024-03-06T07:30:56.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.146+0000 7ff72f200640 1 -- 172.21.15.6:0/2683387584 wait complete. 2024-03-06T07:30:56.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:30:56.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.382+0000 7fd8d3e00640 1 Processor -- start 2024-03-06T07:30:56.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.382+0000 7fd8d3e00640 1 -- start start 2024-03-06T07:30:56.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.383+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.383+0000 7fd8d3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8d40d0c80 0x7fd8d40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.383+0000 7fd8d3e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40b3710 con 0x7fd8d415e0b0 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.383+0000 7fd8d3e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d40b09e0 con 0x7fd8d40d0c80 2024-03-06T07:30:56.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.383+0000 7fd8d3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40cfa20 con 0x7fd8d40d4a10 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.384+0000 7fd8d3400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8d40d0c80 0x7fd8d40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.384+0000 7fd8d3400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8d40d0c80 0x7fd8d40b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:47940/0 (socket says 172.21.15.6:47940) 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.384+0000 7fd8d3400640 1 -- 172.21.15.6:0/216821333 learned_addr learned my addr 172.21.15.6:0/216821333 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.384+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.385+0000 7fd8d3400640 1 -- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 msgr2=0x7fd8d40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.385+0000 7fd8d3400640 1 --2- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d40d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.385+0000 7fd8d3400640 1 -- 172.21.15.6:0/216821333 >> v1:172.21.15.178:6789/0 conn(0x7fd8d415e0b0 legacy=0x7fd8d40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.385+0000 7fd8d3400640 1 -- 172.21.15.6:0/216821333 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40b39f0 con 0x7fd8d40d0c80 2024-03-06T07:30:56.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.386+0000 7fd8d3400640 1 --2- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d40b44d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009e50 tx=0x7fd8cc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86f689d9abd07b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.386+0000 7fd8d2000640 1 -- 172.21.15.6:0/216821333 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc02cb00 con 0x7fd8d40d0c80 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d2000640 1 -- 172.21.15.6:0/216821333 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc02cca0 con 0x7fd8d40d0c80 2024-03-06T07:30:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d2000640 1 -- 172.21.15.6:0/216821333 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc034610 con 0x7fd8d40d0c80 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 -- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 msgr2=0x7fd8d40b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d40b44d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc009e50 tx=0x7fd8cc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 -- 172.21.15.6:0/216821333 shutdown_connections 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d40b44d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/216821333 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d40d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 -- 172.21.15.6:0/216821333 >> 172.21.15.6:0/216821333 conn(0x7fd8d40d3d30 msgr2=0x7fd8d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 -- 172.21.15.6:0/216821333 shutdown_connections 2024-03-06T07:30:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.387+0000 7fd8d3e00640 1 -- 172.21.15.6:0/216821333 wait complete. 2024-03-06T07:30:56.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.389+0000 7fd8d3e00640 1 Processor -- start 2024-03-06T07:30:56.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.389+0000 7fd8d3e00640 1 -- start start 2024-03-06T07:30:56.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.390+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d4154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.390+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d4160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.390+0000 7fd8d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 0x7fd8d4161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.390+0000 7fd8d3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d415e490 con 0x7fd8d415e0b0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.390+0000 7fd8d3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40b3fb0 con 0x7fd8d40d0c80 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.390+0000 7fd8d3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40cfc60 con 0x7fd8d40d4a10 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 0x7fd8d4161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 0x7fd8d4161200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49268/0 (socket says 172.21.15.6:49268) 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 -- 172.21.15.6:0/1388540730 learned_addr learned my addr 172.21.15.6:0/1388540730 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d2a00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d4154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8cbe00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d4160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 -- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 msgr2=0x7fd8d4160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d4160cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 -- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 msgr2=0x7fd8d4154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d4154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.391+0000 7fd8d3400640 1 -- 172.21.15.6:0/1388540730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40b8f20 con 0x7fd8d415e0b0 2024-03-06T07:30:56.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.392+0000 7fd8d3400640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 0x7fd8d4161200 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc036040 tx=0x7fd8cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.392+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc043020 con 0x7fd8d415e0b0 2024-03-06T07:30:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.392+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc02ce50 con 0x7fd8d415e0b0 2024-03-06T07:30:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.392+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc009510 con 0x7fd8d415e0b0 2024-03-06T07:30:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.392+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d40b9200 con 0x7fd8d415e0b0 2024-03-06T07:30:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.392+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d40b97c0 con 0x7fd8d415e0b0 2024-03-06T07:30:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.394+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d404f330 con 0x7fd8d415e0b0 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.394+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8cc046020 con 0x7fd8d415e0b0 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.395+0000 7fd8caa00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.395+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8cc0ac880 con 0x7fd8d415e0b0 2024-03-06T07:30:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.396+0000 7fd8d2a00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.397+0000 7fd8d2a00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8b006b4d0 0x7fd8b006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4000f60 tx=0x7fd8c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.400+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8cc04d030 con 0x7fd8d415e0b0 2024-03-06T07:30:56.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.628+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd8d406d720 con 0x7fd8d415e0b0 2024-03-06T07:30:56.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.629+0000 7fd8caa00640 1 -- 172.21.15.6:0/1388540730 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd8cc07b6f0 con 0x7fd8d415e0b0 2024-03-06T07:30:56.631 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8b006b4d0 msgr2=0x7fd8b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8b006b4d0 0x7fd8b006d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4000f60 tx=0x7fd8c4009040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 msgr2=0x7fd8d4161200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 0x7fd8d4161200 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc036040 tx=0x7fd8cc004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 shutdown_connections 2024-03-06T07:30:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd8b006b4d0 0x7fd8b006d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8d415e0b0 0x7fd8d4161200 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8d40d4a10 0x7fd8d4160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 --2- 172.21.15.6:0/1388540730 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8d40d0c80 0x7fd8d4154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 >> 172.21.15.6:0/1388540730 conn(0x7fd8d40d3d30 msgr2=0x7fd8d40b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 shutdown_connections 2024-03-06T07:30:56.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:56.632+0000 7fd8d3e00640 1 -- 172.21.15.6:0/1388540730 wait complete. 2024-03-06T07:30:56.654 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:30:56.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:30:56.876 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:30:56.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:30:56.876 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:30:56.915 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:30:56.915 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:56.957 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:56.957 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:30:56.957 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:56.997 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:30:56.998 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:30:56.998 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:30:57.048 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:30:57.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:57.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.254+0000 7fb90f200640 1 Processor -- start 2024-03-06T07:30:57.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.254+0000 7fb90f200640 1 -- start start 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.255+0000 7fb90f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb91015e0b0 0x7fb9100cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.255+0000 7fb90f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9100d4a10 0x7fb9100d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9100b3710 con 0x7fb9100d0c80 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9100b09e0 con 0x7fb9100d4a10 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9100b39f0 con 0x7fb91015e0b0 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9100d4a10 0x7fb9100d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9100d4a10 0x7fb9100d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49278/0 (socket says 172.21.15.6:49278) 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90d400640 1 -- 172.21.15.6:0/3369258407 learned_addr learned my addr 172.21.15.6:0/3369258407 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90e800640 1 --1- 172.21.15.6:0/3369258407 >> v1:172.21.15.6:6789/0 conn(0x7fb9100d0c80 0x7fb9100d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43150/0 (socket says 172.21.15.6:43150) 2024-03-06T07:30:57.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.256+0000 7fb90de00640 1 --2- 172.21.15.6:0/3369258407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb91015e0b0 0x7fb9100cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.257+0000 7fb90d400640 1 -- 172.21.15.6:0/3369258407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb91015e0b0 msgr2=0x7fb9100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.257+0000 7fb90d400640 1 --2- 172.21.15.6:0/3369258407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb91015e0b0 0x7fb9100cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.257+0000 7fb90d400640 1 -- 172.21.15.6:0/3369258407 >> v1:172.21.15.6:6789/0 conn(0x7fb9100d0c80 legacy=0x7fb9100d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.257+0000 7fb90d400640 1 -- 172.21.15.6:0/3369258407 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9100b3cd0 con 0x7fb9100d4a10 2024-03-06T07:30:57.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.257+0000 7fb90ca00640 1 -- 172.21.15.6:0/3369258407 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573737920 0 0) 0x7fb9100b3710 con 0x7fb9100d0c80 2024-03-06T07:30:57.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.258+0000 7fb90d400640 1 --2- 172.21.15.6:0/3369258407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d4a10 0x7fb9100d0740 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009e50 tx=0x7fb8f802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3d92da6bf61788 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.258+0000 7fb90ca00640 1 -- 172.21.15.6:0/3369258407 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f802cb00 con 0x7fb9100d4a10 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.258+0000 7fb90ca00640 1 -- 172.21.15.6:0/3369258407 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f802cca0 con 0x7fb9100d4a10 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.258+0000 7fb90ca00640 1 -- 172.21.15.6:0/3369258407 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8034610 con 0x7fb9100d4a10 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 -- 172.21.15.6:0/3369258407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d4a10 msgr2=0x7fb9100d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 --2- 172.21.15.6:0/3369258407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d4a10 0x7fb9100d0740 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009e50 tx=0x7fb8f802bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 -- 172.21.15.6:0/3369258407 shutdown_connections 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 --2- 172.21.15.6:0/3369258407 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d4a10 0x7fb9100d0740 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 --2- 172.21.15.6:0/3369258407 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb91015e0b0 0x7fb9100cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 -- 172.21.15.6:0/3369258407 >> 172.21.15.6:0/3369258407 conn(0x7fb9100d3d30 msgr2=0x7fb9100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 -- 172.21.15.6:0/3369258407 shutdown_connections 2024-03-06T07:30:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.259+0000 7fb90f200640 1 -- 172.21.15.6:0/3369258407 wait complete. 2024-03-06T07:30:57.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.261+0000 7fb90f200640 1 Processor -- start 2024-03-06T07:30:57.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.261+0000 7fb90f200640 1 -- start start 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 0x7fb9100a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9100d4a10 0x7fb9100b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb91015e0b0 0x7fb91015f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9100cfc60 con 0x7fb9100d0c80 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91015e490 con 0x7fb91015e0b0 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9100b54e0 con 0x7fb9100d4a10 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 0x7fb9100a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 0x7fb9100a13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49282/0 (socket says 172.21.15.6:49282) 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.262+0000 7fb90de00640 1 -- 172.21.15.6:0/2484935166 learned_addr learned my addr 172.21.15.6:0/2484935166 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90e800640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb91015e0b0 0x7fb91015f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90d400640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9100d4a10 0x7fb9100b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90de00640 1 -- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9100d4a10 msgr2=0x7fb9100b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90de00640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9100d4a10 0x7fb9100b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90de00640 1 -- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb91015e0b0 msgr2=0x7fb91015f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90de00640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb91015e0b0 0x7fb91015f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90de00640 1 -- 172.21.15.6:0/2484935166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9100cdc10 con 0x7fb9100d0c80 2024-03-06T07:30:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb90de00640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 0x7fb9100a13a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb90000d5f0 tx=0x7fb90000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.263+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9000125f0 con 0x7fb9100d0c80 2024-03-06T07:30:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.264+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb900019070 con 0x7fb9100d0c80 2024-03-06T07:30:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.264+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9000129d0 con 0x7fb9100d0c80 2024-03-06T07:30:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.264+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9100cb800 con 0x7fb9100d0c80 2024-03-06T07:30:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.264+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9100cbdc0 con 0x7fb9100d0c80 2024-03-06T07:30:57.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.265+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91004f330 con 0x7fb9100d0c80 2024-03-06T07:30:57.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.266+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb900012b70 con 0x7fb9100d0c80 2024-03-06T07:30:57.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.267+0000 7fb906000640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8ec06b4d0 0x7fb8ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.267+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb90008a6b0 con 0x7fb9100d0c80 2024-03-06T07:30:57.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.268+0000 7fb90d400640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8ec06b4d0 0x7fb8ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.269+0000 7fb90d400640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8ec06b4d0 0x7fb8ec06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8035000 tx=0x7fb8f8030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.272+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb900059490 con 0x7fb9100d0c80 2024-03-06T07:30:57.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.509+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb910161640 con 0x7fb9100d0c80 2024-03-06T07:30:57.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.510+0000 7fb906000640 1 -- 172.21.15.6:0/2484935166 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fb900058d90 con 0x7fb9100d0c80 2024-03-06T07:30:57.512 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:57.512 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:30:57.512 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8ec06b4d0 msgr2=0x7fb8ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8ec06b4d0 0x7fb8ec06d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8035000 tx=0x7fb8f8030000 comp rx=0 tx=0).stop 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 msgr2=0x7fb9100a13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 0x7fb9100a13a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fb90000d5f0 tx=0x7fb90000dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 shutdown_connections 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb8ec06b4d0 0x7fb8ec06d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb91015e0b0 0x7fb91015f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9100d4a10 0x7fb9100b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 --2- 172.21.15.6:0/2484935166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9100d0c80 0x7fb9100a13a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.512+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 >> 172.21.15.6:0/2484935166 conn(0x7fb9100d3d30 msgr2=0x7fb9100cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.513+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 shutdown_connections 2024-03-06T07:30:57.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.513+0000 7fb90f200640 1 -- 172.21.15.6:0/2484935166 wait complete. 2024-03-06T07:30:57.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:30:57.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.777+0000 7f5d16800640 1 Processor -- start 2024-03-06T07:30:57.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.777+0000 7f5d16800640 1 -- start start 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d16800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5d100d4a10 0x7f5d100d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d16800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5d100d0c80 0x7f5d100b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d16800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100b3710 con 0x7f5d1015e0b0 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d16800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d100d10f0 con 0x7f5d100d4a10 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d16800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d100b09e0 con 0x7f5d100d0c80 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d15400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5d1015e0b0 0x7f5d100cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43152/0 (socket says 172.21.15.6:43152) 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d15400640 1 -- 172.21.15.6:0/968574678 learned_addr learned my addr 172.21.15.6:0/968574678 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.779+0000 7f5d14a00640 1 --2- 172.21.15.6:0/968574678 >> v2:172.21.15.6:3300/0 conn(0x7f5d100d4a10 0x7f5d100d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d15e00640 1 --2- 172.21.15.6:0/968574678 >> v2:172.21.15.178:3300/0 conn(0x7f5d100d0c80 0x7f5d100b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d0fe00640 1 -- 172.21.15.6:0/968574678 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233580992 0 0) 0x7f5d100b3710 con 0x7f5d1015e0b0 2024-03-06T07:30:57.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d0fe00640 1 -- 172.21.15.6:0/968574678 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf8003520 con 0x7f5d1015e0b0 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d14a00640 1 -- 172.21.15.6:0/968574678 >> v2:172.21.15.178:3300/0 conn(0x7f5d100d0c80 msgr2=0x7f5d100b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d14a00640 1 --2- 172.21.15.6:0/968574678 >> v2:172.21.15.178:3300/0 conn(0x7f5d100d0c80 0x7f5d100b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d14a00640 1 -- 172.21.15.6:0/968574678 >> v1:172.21.15.6:6789/0 conn(0x7f5d1015e0b0 legacy=0x7f5d100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.780+0000 7f5d14a00640 1 -- 172.21.15.6:0/968574678 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100b39f0 con 0x7f5d100d4a10 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.781+0000 7f5d14a00640 1 --2- 172.21.15.6:0/968574678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d100d0740 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5d1008d6d0 tx=0x7f5cfc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06fd64bdaffea34 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.781+0000 7f5d0fe00640 1 -- 172.21.15.6:0/968574678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc035020 con 0x7f5d100d4a10 2024-03-06T07:30:57.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.781+0000 7f5d0fe00640 1 -- 172.21.15.6:0/968574678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc031830 con 0x7f5d100d4a10 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.781+0000 7f5d0fe00640 1 -- 172.21.15.6:0/968574678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc031be0 con 0x7f5d100d4a10 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 -- 172.21.15.6:0/968574678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 msgr2=0x7f5d100d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 --2- 172.21.15.6:0/968574678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d100d0740 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5d1008d6d0 tx=0x7f5cfc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 -- 172.21.15.6:0/968574678 shutdown_connections 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 --2- 172.21.15.6:0/968574678 >> v2:172.21.15.178:3300/0 conn(0x7f5d100d0c80 0x7f5d100b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 --2- 172.21.15.6:0/968574678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d100d0740 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 -- 172.21.15.6:0/968574678 >> 172.21.15.6:0/968574678 conn(0x7f5d100d3d30 msgr2=0x7f5d100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 -- 172.21.15.6:0/968574678 shutdown_connections 2024-03-06T07:30:57.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.782+0000 7f5d16800640 1 -- 172.21.15.6:0/968574678 wait complete. 2024-03-06T07:30:57.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.784+0000 7f5d16800640 1 Processor -- start 2024-03-06T07:30:57.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.784+0000 7f5d16800640 1 -- start start 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d16800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d100d0c80 0x7f5d100b9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d16800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d101f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d16800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d1015e0b0 0x7f5d1015cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d16800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100b3fb0 con 0x7f5d100d4a10 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d16800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100cfa20 con 0x7f5d1015e0b0 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d16800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1015e490 con 0x7f5d100d0c80 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d14a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d101f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d14a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d101f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49290/0 (socket says 172.21.15.6:49290) 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d14a00640 1 -- 172.21.15.6:0/1755076914 learned_addr learned my addr 172.21.15.6:0/1755076914 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.785+0000 7f5d15e00640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d1015e0b0 0x7f5d1015cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d15400640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d100d0c80 0x7f5d100b9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d14a00640 1 -- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d100d0c80 msgr2=0x7f5d100b9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d14a00640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d100d0c80 0x7f5d100b9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d14a00640 1 -- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d1015e0b0 msgr2=0x7f5d1015cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:57.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d14a00640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d1015e0b0 0x7f5d1015cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:57.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d14a00640 1 -- 172.21.15.6:0/1755076914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100cc110 con 0x7f5d100d4a10 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d14a00640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d101f4780 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0098c0 tx=0x7f5cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc041070 con 0x7f5d100d4a10 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc03b900 con 0x7f5d100d4a10 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc035040 con 0x7f5d100d4a10 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.786+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d100b6ad0 con 0x7f5d100d4a10 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.787+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d100b7090 con 0x7f5d100d4a10 2024-03-06T07:30:57.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.788+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1004f330 con 0x7f5d100d4a10 2024-03-06T07:30:57.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.790+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5cfc04c060 con 0x7f5d100d4a10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.791+0000 7f5d0d600640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cf006b4d0 0x7f5cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.791+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5cfc0abb50 con 0x7f5d100d4a10 2024-03-06T07:30:57.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.792+0000 7f5d15400640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cf006b4d0 0x7f5cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:57.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.793+0000 7f5d15400640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cf006b4d0 0x7f5cf006d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5d04000f60 tx=0x7f5d04009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:57.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:57.796+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cfc031800 con 0x7f5d100d4a10 2024-03-06T07:30:58.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.031+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d100b6390 con 0x7f5d100d4a10 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.033+0000 7f5d0d600640 1 -- 172.21.15.6:0/1755076914 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7f5cfc07a930 con 0x7f5d100d4a10 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":58,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:22.633658+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:30:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cf006b4d0 msgr2=0x7f5cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cf006b4d0 0x7f5cf006d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5d04000f60 tx=0x7f5d04009040 comp rx=0 tx=0).stop 2024-03-06T07:30:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 msgr2=0x7f5d101f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d101f4780 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0098c0 tx=0x7f5cfc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 shutdown_connections 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5cf006b4d0 0x7f5cf006d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d1015e0b0 0x7f5d1015cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5d100d4a10 0x7f5d101f4780 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 --2- 172.21.15.6:0/1755076914 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5d100d0c80 0x7f5d100b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 >> 172.21.15.6:0/1755076914 conn(0x7f5d100d3d30 msgr2=0x7f5d100b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 shutdown_connections 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.035+0000 7f5d16800640 1 -- 172.21.15.6:0/1755076914 wait complete. 2024-03-06T07:30:58.056 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:30:19.028665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:30:20.117374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:30:58.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:30:58.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.291+0000 7ff153e00640 1 Processor -- start 2024-03-06T07:30:58.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.291+0000 7ff153e00640 1 -- start start 2024-03-06T07:30:58.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.292+0000 7ff153e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1540d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.292+0000 7ff153e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff1540d4860 0x7ff1540d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.292+0000 7ff153e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540ceca0 con 0x7ff1540cfa60 2024-03-06T07:30:58.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.292+0000 7ff153e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540cf5d0 con 0x7ff1540d4860 2024-03-06T07:30:58.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.292+0000 7ff153e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cef80 con 0x7ff15415dee0 2024-03-06T07:30:58.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.293+0000 7ff153400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff1540cfa60 0x7ff1540b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43168/0 (socket says 172.21.15.6:43168) 2024-03-06T07:30:58.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.293+0000 7ff153400640 1 -- 172.21.15.6:0/2973591098 learned_addr learned my addr 172.21.15.6:0/2973591098 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.293+0000 7ff14a000640 1 --2- 172.21.15.6:0/2973591098 >> v2:172.21.15.178:3300/0 conn(0x7ff1540d4860 0x7ff1540d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.293+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847851921 0 0) 0x7ff1540ceca0 con 0x7ff1540cfa60 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.293+0000 7ff152a00640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1540d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.293+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138003520 con 0x7ff1540cfa60 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781520673 0 0) 0x7ff138003520 con 0x7ff1540cfa60 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 msgr2=0x7ff1540d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1540d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 >> v2:172.21.15.178:3300/0 conn(0x7ff1540d4860 msgr2=0x7ff1540d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 --2- 172.21.15.6:0/2973591098 >> v2:172.21.15.178:3300/0 conn(0x7ff1540d4860 0x7ff1540d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540cf260 con 0x7ff1540cfa60 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff14c001670 con 0x7ff1540cfa60 2024-03-06T07:30:58.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152a00640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1540d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 >> v1:172.21.15.6:6789/0 conn(0x7ff1540cfa60 legacy=0x7ff1540b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff138003ef0 0x7ff138004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cf5d0 con 0x7ff138003ef0 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540cef80 con 0x7ff138004d40 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13800b9f0 con 0x7ff15415dee0 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.294+0000 7ff14a000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff138003ef0 0x7ff138004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff153400640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff152a00640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff14a000640 1 -- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 msgr2=0x7ff1380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff14a000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff14a000640 1 -- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff138004d40 msgr2=0x7ff1380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff14a000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff138004d40 0x7ff1380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.299 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff14a000640 1 -- 172.21.15.6:0/2973591098 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138003520 con 0x7ff138003ef0 2024-03-06T07:30:58.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.295+0000 7ff152a00640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415dee0 0x7ff1380039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.296+0000 7ff14a000640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff138003ef0 0x7ff138004800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009a90 tx=0x7ff138005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5055efd332c46b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.297+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c039070 con 0x7ff138003ef0 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.297+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c02c6b0 con 0x7ff138003ef0 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.297+0000 7ff152000640 1 -- 172.21.15.6:0/2973591098 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c009690 con 0x7ff138003ef0 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153e00640 1 -- 172.21.15.6:0/2973591098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff138003ef0 msgr2=0x7ff138004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153e00640 1 --2- 172.21.15.6:0/2973591098 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff138003ef0 0x7ff138004800 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009a90 tx=0x7ff138005830 comp rx=0 tx=0).stop 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153400640 1 -- 172.21.15.6:0/2973591098 reap_dead start 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153e00640 1 -- 172.21.15.6:0/2973591098 shutdown_connections 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153e00640 1 -- 172.21.15.6:0/2973591098 >> 172.21.15.6:0/2973591098 conn(0x7ff1540d3cd0 msgr2=0x7ff1540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153e00640 1 -- 172.21.15.6:0/2973591098 shutdown_connections 2024-03-06T07:30:58.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.298+0000 7ff153e00640 1 -- 172.21.15.6:0/2973591098 wait complete. 2024-03-06T07:30:58.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.301+0000 7ff153e00640 1 Processor -- start 2024-03-06T07:30:58.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.301+0000 7ff153e00640 1 -- start start 2024-03-06T07:30:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff153e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 0x7ff15415e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff153e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff15414f3d0 0x7ff15406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff153e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415e980 0x7ff15406d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff153e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540d0b70 con 0x7ff15414eff0 2024-03-06T07:30:58.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff153e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15415e2c0 con 0x7ff15414f3d0 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff153e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540d0cf0 con 0x7ff15415e980 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 0x7ff15415e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 0x7ff15415e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49304/0 (socket says 172.21.15.6:49304) 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 -- 172.21.15.6:0/888535658 learned_addr learned my addr 172.21.15.6:0/888535658 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff14a000640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff15414f3d0 0x7ff15406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 -- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415e980 msgr2=0x7ff15406d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415e980 0x7ff15406d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 -- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff15414f3d0 msgr2=0x7ff15406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff15414f3d0 0x7ff15406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff152a00640 1 -- 172.21.15.6:0/888535658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15415ffa0 con 0x7ff15414eff0 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.303+0000 7ff14a000640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff15414f3d0 0x7ff15406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.304+0000 7ff152a00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 0x7ff15415e440 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff144009fc0 tx=0x7ff14400e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.304+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144015610 con 0x7ff15414eff0 2024-03-06T07:30:58.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.304+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14401a070 con 0x7ff15414eff0 2024-03-06T07:30:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.304+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15415ef00 con 0x7ff15414eff0 2024-03-06T07:30:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.304+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1440159b0 con 0x7ff15414eff0 2024-03-06T07:30:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.304+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15415f4c0 con 0x7ff15414eff0 2024-03-06T07:30:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.306+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15404f330 con 0x7ff15414eff0 2024-03-06T07:30:58.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.306+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff144012030 con 0x7ff15414eff0 2024-03-06T07:30:58.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.307+0000 7ff14b400640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff13006b500 0x7ff13006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.307+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff14408ade0 con 0x7ff15414eff0 2024-03-06T07:30:58.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.308+0000 7ff14a000640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff13006b500 0x7ff13006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.310+0000 7ff14a000640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff13006b500 0x7ff13006d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff13c036000 tx=0x7ff13c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.313+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff144010040 con 0x7ff15414eff0 2024-03-06T07:30:58.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.604+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff15415d950 con 0x7ff15414eff0 2024-03-06T07:30:58.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.605+0000 7ff14b400640 1 -- 172.21.15.6:0/888535658 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff144059bc0 con 0x7ff15414eff0 2024-03-06T07:30:58.607 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:30:58.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff13006b500 msgr2=0x7ff13006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff13006b500 0x7ff13006d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff13c036000 tx=0x7ff13c030000 comp rx=0 tx=0).stop 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 msgr2=0x7ff15415e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 0x7ff15415e440 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff144009fc0 tx=0x7ff14400e5d0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 shutdown_connections 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff13006b500 0x7ff13006d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff15414f3d0 0x7ff15406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff15415e980 0x7ff15406d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 --2- 172.21.15.6:0/888535658 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff15414eff0 0x7ff15415e440 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 >> 172.21.15.6:0/888535658 conn(0x7ff1540d3cd0 msgr2=0x7ff1540b7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 shutdown_connections 2024-03-06T07:30:58.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.608+0000 7ff153e00640 1 -- 172.21.15.6:0/888535658 wait complete. 2024-03-06T07:30:58.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:30:58.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.843+0000 7f5816200640 1 Processor -- start 2024-03-06T07:30:58.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.844+0000 7f5816200640 1 -- start start 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.845+0000 7f5816200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5810071e20 0x7f58100ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.845+0000 7f5816200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5810072df0 con 0x7f5810072520 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.845+0000 7f5816200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58100cdab0 con 0x7f581015bf20 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.845+0000 7f5816200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58100b4d70 con 0x7f5810071e20 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5810072520 0x7f58100cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43180/0 (socket says 172.21.15.6:43180) 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580ea00640 1 -- 172.21.15.6:0/111787695 learned_addr learned my addr 172.21.15.6:0/111787695 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:58.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580fe00640 1 --2- 172.21.15.6:0/111787695 >> v2:172.21.15.6:3300/0 conn(0x7f5810071e20 0x7f58100ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580fe00640 1 -- 172.21.15.6:0/111787695 >> v1:172.21.15.178:6789/0 conn(0x7f581015bf20 legacy=0x7f58100acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580fe00640 1 -- 172.21.15.6:0/111787695 >> v1:172.21.15.6:6789/0 conn(0x7f5810072520 legacy=0x7f58100cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580e000640 1 -- 172.21.15.6:0/111787695 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92964411 0 0) 0x7f5810072df0 con 0x7f5810072520 2024-03-06T07:30:58.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.846+0000 7f580fe00640 1 -- 172.21.15.6:0/111787695 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58100b5050 con 0x7f5810071e20 2024-03-06T07:30:58.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.847+0000 7f580fe00640 1 --2- 172.21.15.6:0/111787695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100ccad0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5808009d90 tx=0x7f580802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a51afc8a8c5ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f580e000640 1 -- 172.21.15.6:0/111787695 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808039070 con 0x7f5810071e20 2024-03-06T07:30:58.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f580e000640 1 -- 172.21.15.6:0/111787695 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580802cc60 con 0x7f5810071e20 2024-03-06T07:30:58.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f580e000640 1 -- 172.21.15.6:0/111787695 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808034660 con 0x7f5810071e20 2024-03-06T07:30:58.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f5816200640 1 -- 172.21.15.6:0/111787695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 msgr2=0x7f58100ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f5816200640 1 --2- 172.21.15.6:0/111787695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100ccad0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5808009d90 tx=0x7f580802beb0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f5816200640 1 -- 172.21.15.6:0/111787695 shutdown_connections 2024-03-06T07:30:58.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f5816200640 1 --2- 172.21.15.6:0/111787695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100ccad0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.848+0000 7f5816200640 1 -- 172.21.15.6:0/111787695 >> 172.21.15.6:0/111787695 conn(0x7f581006b3d0 msgr2=0x7f581006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.849+0000 7f5816200640 1 -- 172.21.15.6:0/111787695 shutdown_connections 2024-03-06T07:30:58.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.849+0000 7f5816200640 1 -- 172.21.15.6:0/111787695 wait complete. 2024-03-06T07:30:58.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.851+0000 7f5816200640 1 Processor -- start 2024-03-06T07:30:58.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.851+0000 7f5816200640 1 -- start start 2024-03-06T07:30:58.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f5816200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f5816200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5810072520 0x7f58101463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f5816200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f581015bf20 0x7f581009b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f5816200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58100ad450 con 0x7f5810071e20 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f5816200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58100b4990 con 0x7f581015bf20 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f5816200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58100cb810 con 0x7f5810072520 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f580f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f580f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49324/0 (socket says 172.21.15.6:49324) 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f580f400640 1 -- 172.21.15.6:0/967045391 learned_addr learned my addr 172.21.15.6:0/967045391 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f580f400640 1 -- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5810072520 msgr2=0x7f58101463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f580ea00640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5810072520 0x7f58101463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.853+0000 7f580fe00640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f581015bf20 0x7f581009b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580f400640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5810072520 0x7f58101463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580f400640 1 -- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f581015bf20 msgr2=0x7f581009b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580f400640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f581015bf20 0x7f581009b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580f400640 1 -- 172.21.15.6:0/967045391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581009ab40 con 0x7f5810071e20 2024-03-06T07:30:58.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580ea00640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5810072520 0x7f58101463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580fe00640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f581015bf20 0x7f581009b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f580f400640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100caf90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f580000c430 tx=0x7f580000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800011790 con 0x7f5810071e20 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800017070 con 0x7f5810071e20 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800011bb0 con 0x7f5810071e20 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581009ae20 con 0x7f5810071e20 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.854+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581009b3e0 con 0x7f5810071e20 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.856+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581004f330 con 0x7f5810071e20 2024-03-06T07:30:58.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.856+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f580001c460 con 0x7f5810071e20 2024-03-06T07:30:58.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.858+0000 7f5807400640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f57f006b500 0x7f57f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.858+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f580008a060 con 0x7f5810071e20 2024-03-06T07:30:58.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.859+0000 7f580ea00640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f57f006b500 0x7f57f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.860+0000 7f580ea00640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f57f006b500 0x7f57f006d9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f58101f74d0 tx=0x7f57fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:58.862+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5800058ed0 con 0x7f5810071e20 2024-03-06T07:30:59.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.064+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f581015f720 con 0x7f5810071e20 2024-03-06T07:30:59.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.065+0000 7f5807400640 1 -- 172.21.15.6:0/967045391 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) v1 ==== 104+0+11 (secure 0 0 0) 0x7f58000587d0 con 0x7f5810071e20 2024-03-06T07:30:59.068 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.067+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f57f006b500 msgr2=0x7f57f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.067+0000 7f5816200640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f57f006b500 0x7f57f006d9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f58101f74d0 tx=0x7f57fc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.067+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 msgr2=0x7f58100caf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.067+0000 7f5816200640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100caf90 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f580000c430 tx=0x7f580000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 shutdown_connections 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f57f006b500 0x7f57f006d9c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f581015bf20 0x7f581009b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5810072520 0x7f58101463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 --2- 172.21.15.6:0/967045391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5810071e20 0x7f58100caf90 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 >> 172.21.15.6:0/967045391 conn(0x7f581006b3d0 msgr2=0x7f58100adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 shutdown_connections 2024-03-06T07:30:59.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.068+0000 7f5816200640 1 -- 172.21.15.6:0/967045391 wait complete. 2024-03-06T07:30:59.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:30:59.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.307+0000 7f03f1000640 1 Processor -- start 2024-03-06T07:30:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.307+0000 7f03f1000640 1 -- start start 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.309+0000 7f03f1000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.309+0000 7f03f1000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f03ec06f370 0x7f03ec15c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.309+0000 7f03f1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03ec15c930 0x7f03ec0afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.309+0000 7f03f1000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec0af250 con 0x7f03ec15c930 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.309+0000 7f03f1000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec15cf50 con 0x7f03ec06f370 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.309+0000 7f03f1000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec15d230 con 0x7f03ec15c010 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03ec15c930 0x7f03ec0afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03ec15c930 0x7f03ec0afcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49332/0 (socket says 172.21.15.6:49332) 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 -- 172.21.15.6:0/1718949711 learned_addr learned my addr 172.21.15.6:0/1718949711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03e9e00640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03e9400640 1 --2- 172.21.15.6:0/1718949711 >> v2:172.21.15.178:3300/0 conn(0x7f03ec06f370 0x7f03ec15c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 -- 172.21.15.6:0/1718949711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 msgr2=0x7f03ec06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 -- 172.21.15.6:0/1718949711 >> v2:172.21.15.178:3300/0 conn(0x7f03ec06f370 msgr2=0x7f03ec15c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 --2- 172.21.15.6:0/1718949711 >> v2:172.21.15.178:3300/0 conn(0x7f03ec06f370 0x7f03ec15c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03ea800640 1 -- 172.21.15.6:0/1718949711 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0af4f0 con 0x7f03ec15c930 2024-03-06T07:30:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.310+0000 7f03e9e00640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.312+0000 7f03ea800640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec0afcf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0d2d80 tx=0x7f03e402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678add07fd40aec4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03e8a00640 1 -- 172.21.15.6:0/1718949711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4035020 con 0x7f03ec15c930 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03e8a00640 1 -- 172.21.15.6:0/1718949711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e402d040 con 0x7f03ec15c930 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03e8a00640 1 -- 172.21.15.6:0/1718949711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4031a80 con 0x7f03ec15c930 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 -- 172.21.15.6:0/1718949711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 msgr2=0x7f03ec0afcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec0afcf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f03ec0d2d80 tx=0x7f03e402b900 comp rx=0 tx=0).stop 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 -- 172.21.15.6:0/1718949711 shutdown_connections 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec0afcf0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 --2- 172.21.15.6:0/1718949711 >> v2:172.21.15.178:3300/0 conn(0x7f03ec06f370 0x7f03ec15c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 --2- 172.21.15.6:0/1718949711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 -- 172.21.15.6:0/1718949711 >> 172.21.15.6:0/1718949711 conn(0x7f03ec0732d0 msgr2=0x7f03ec09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 -- 172.21.15.6:0/1718949711 shutdown_connections 2024-03-06T07:30:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.313+0000 7f03f1000640 1 -- 172.21.15.6:0/1718949711 wait complete. 2024-03-06T07:30:59.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.315+0000 7f03f1000640 1 Processor -- start 2024-03-06T07:30:59.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.315+0000 7f03f1000640 1 -- start start 2024-03-06T07:30:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.316+0000 7f03f1000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03ec06f370 0x7f03ec1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.316+0000 7f03f1000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec0cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.316+0000 7f03f1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.316+0000 7f03f1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0b0230 con 0x7f03ec15c930 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.316+0000 7f03f1000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0d1ca0 con 0x7f03ec06f370 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.316+0000 7f03f1000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0b03b0 con 0x7f03ec15c010 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec14e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49348/0 (socket says 172.21.15.6:49348) 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 -- 172.21.15.6:0/3663614863 learned_addr learned my addr 172.21.15.6:0/3663614863 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03e9e00640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03ec06f370 0x7f03ec1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03e9400640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec0cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 -- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 msgr2=0x7f03ec0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec0cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 -- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03ec06f370 msgr2=0x7f03ec1506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03ec06f370 0x7f03ec1506a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.317+0000 7f03ea800640 1 -- 172.21.15.6:0/3663614863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec150be0 con 0x7f03ec15c930 2024-03-06T07:30:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.318+0000 7f03ea800640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec14e840 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f03e402b940 tx=0x7f03e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.318+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e403ac60 con 0x7f03ec15c930 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.318+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e403ae20 con 0x7f03ec15c930 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.318+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4044630 con 0x7f03ec15c930 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.318+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec1f3180 con 0x7f03ec15c930 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.318+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec1f3740 con 0x7f03ec15c930 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.319+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ec04f330 con 0x7f03ec15c930 2024-03-06T07:30:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.320+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03e4038020 con 0x7f03ec15c930 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.321+0000 7f03e2000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03c806b4d0 0x7f03c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.321+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03e40ac030 con 0x7f03ec15c930 2024-03-06T07:30:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.322+0000 7f03e9e00640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03c806b4d0 0x7f03c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.323+0000 7f03e9e00640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03c806b4d0 0x7f03c806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f03ec09fec0 tx=0x7f03dc003a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.327+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03e403f070 con 0x7f03ec15c930 2024-03-06T07:30:59.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.649+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03ec15b3a0 con 0x7f03ec15c930 2024-03-06T07:30:59.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.650+0000 7f03e2000640 1 -- 172.21.15.6:0/3663614863 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03e407af40 con 0x7f03ec15c930 2024-03-06T07:30:59.652 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03c806b4d0 msgr2=0x7f03c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03c806b4d0 0x7f03c806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f03ec09fec0 tx=0x7f03dc003a10 comp rx=0 tx=0).stop 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 msgr2=0x7f03ec14e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec14e840 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f03e402b940 tx=0x7f03e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 shutdown_connections 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03c806b4d0 0x7f03c806d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03ec15c930 0x7f03ec14e840 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f03ec15c010 0x7f03ec0cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 --2- 172.21.15.6:0/3663614863 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03ec06f370 0x7f03ec1506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 >> 172.21.15.6:0/3663614863 conn(0x7f03ec0732d0 msgr2=0x7f03ec06d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 shutdown_connections 2024-03-06T07:30:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.652+0000 7f03f1000640 1 -- 172.21.15.6:0/3663614863 wait complete. 2024-03-06T07:30:59.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.862+0000 7fedbbe00640 1 Processor -- start 2024-03-06T07:30:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.862+0000 7fedbbe00640 1 -- start start 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fedbc0d0c80 0x7fedbc0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbbe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc0b3710 con 0x7fedbc15e0b0 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbbe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedbc0b09e0 con 0x7fedbc0d0c80 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0cfa20 con 0x7fedbc0d4a10 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fedbc15e0b0 0x7fedbc0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43186/0 (socket says 172.21.15.6:43186) 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbaa00640 1 -- 172.21.15.6:0/3970025714 learned_addr learned my addr 172.21.15.6:0/3970025714 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:30:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.864+0000 7fedbb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fedbc0d0c80 0x7fedbc0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.865+0000 7fedba000640 1 --2- 172.21.15.6:0/3970025714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.865+0000 7fedbb400640 1 -- 172.21.15.6:0/3970025714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 msgr2=0x7fedbc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.865+0000 7fedbb400640 1 --2- 172.21.15.6:0/3970025714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.865+0000 7fedbb400640 1 -- 172.21.15.6:0/3970025714 >> v1:172.21.15.6:6789/0 conn(0x7fedbc15e0b0 legacy=0x7fedbc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.865+0000 7fedbb400640 1 -- 172.21.15.6:0/3970025714 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc0b39f0 con 0x7fedbc0d0c80 2024-03-06T07:30:59.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.865+0000 7fedb9600640 1 -- 172.21.15.6:0/3970025714 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154274244 0 0) 0x7fedbc0b3710 con 0x7fedbc15e0b0 2024-03-06T07:30:59.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.866+0000 7fedbb400640 1 --2- 172.21.15.6:0/3970025714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc0b44d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009d90 tx=0x7fedb402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bbe5cb3ce802f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.866+0000 7fedb9600640 1 -- 172.21.15.6:0/3970025714 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4039070 con 0x7fedbc0d0c80 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.866+0000 7fedb9600640 1 -- 172.21.15.6:0/3970025714 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb402cc50 con 0x7fedbc0d0c80 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.866+0000 7fedb9600640 1 -- 172.21.15.6:0/3970025714 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedb4034440 con 0x7fedbc0d0c80 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 -- 172.21.15.6:0/3970025714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 msgr2=0x7fedbc0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 --2- 172.21.15.6:0/3970025714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc0b44d0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009d90 tx=0x7fedb402bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 -- 172.21.15.6:0/3970025714 shutdown_connections 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 --2- 172.21.15.6:0/3970025714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc0b44d0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 --2- 172.21.15.6:0/3970025714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 -- 172.21.15.6:0/3970025714 >> 172.21.15.6:0/3970025714 conn(0x7fedbc0d3d30 msgr2=0x7fedbc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 -- 172.21.15.6:0/3970025714 shutdown_connections 2024-03-06T07:30:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.867+0000 7fedbbe00640 1 -- 172.21.15.6:0/3970025714 wait complete. 2024-03-06T07:30:59.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.869+0000 7fedbbe00640 1 Processor -- start 2024-03-06T07:30:59.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.869+0000 7fedbbe00640 1 -- start start 2024-03-06T07:30:59.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbbe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedbc15e0b0 0x7fedbc0b9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc15e490 con 0x7fedbc0d0c80 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbbe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0b3fb0 con 0x7fedbc15e0b0 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0cfc60 con 0x7fedbc0d4a10 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49364/0 (socket says 172.21.15.6:49364) 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 -- 172.21.15.6:0/1417020699 learned_addr learned my addr 172.21.15.6:0/1417020699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedba000640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbb400640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedbc15e0b0 0x7fedbc0b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 -- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 msgr2=0x7fedbc0b90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0b90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 -- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedbc15e0b0 msgr2=0x7fedbc0b9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedbc15e0b0 0x7fedbc0b9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbaa00640 1 -- 172.21.15.6:0/1417020699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc15e6b0 con 0x7fedbc0d0c80 2024-03-06T07:30:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.870+0000 7fedbb400640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedbc15e0b0 0x7fedbc0b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.871+0000 7fedbaa00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc15b290 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fedac00c430 tx=0x7fedac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.871+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0074e0 con 0x7fedbc0d0c80 2024-03-06T07:30:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.871+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac0047e0 con 0x7fedbc0d0c80 2024-03-06T07:30:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.871+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac004bd0 con 0x7fedbc0d0c80 2024-03-06T07:30:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.871+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc15e990 con 0x7fedbc0d0c80 2024-03-06T07:30:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.871+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc15ef50 con 0x7fedbc0d0c80 2024-03-06T07:30:59.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.872+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc04f330 con 0x7fedbc0d0c80 2024-03-06T07:30:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.873+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedac011070 con 0x7fedbc0d0c80 2024-03-06T07:30:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.874+0000 7fedb3400640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fed9c06b520 0x7fed9c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.874+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fedac089e30 con 0x7fedbc0d0c80 2024-03-06T07:30:59.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.875+0000 7fedba000640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fed9c06b520 0x7fed9c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.876+0000 7fedba000640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fed9c06b520 0x7fed9c06d9e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0a1d00 tx=0x7feda8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:30:59.879+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedac00e280 con 0x7fedbc0d0c80 2024-03-06T07:31:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.112+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedbc0b5480 con 0x7fedbc0d0c80 2024-03-06T07:31:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.113+0000 7fedb3400640 1 -- 172.21.15.6:0/1417020699 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fedac058c90 con 0x7fedbc0d0c80 2024-03-06T07:31:00.115 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:00.115 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:31:00.116 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:31:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fed9c06b520 msgr2=0x7fed9c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fed9c06b520 0x7fed9c06d9e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0a1d00 tx=0x7feda8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 msgr2=0x7fedbc15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc15b290 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fedac00c430 tx=0x7fedac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 shutdown_connections 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fed9c06b520 0x7fed9c06d9e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fedbc15e0b0 0x7fedbc0b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fedbc0d4a10 0x7fedbc0b90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 --2- 172.21.15.6:0/1417020699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fedbc0d0c80 0x7fedbc15b290 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 >> 172.21.15.6:0/1417020699 conn(0x7fedbc0d3d30 msgr2=0x7fedbc0cf100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.115+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 shutdown_connections 2024-03-06T07:31:00.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.116+0000 7fedbbe00640 1 -- 172.21.15.6:0/1417020699 wait complete. 2024-03-06T07:31:00.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:00.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.346+0000 7fa4b2400640 1 Processor -- start 2024-03-06T07:31:00.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.347+0000 7fa4b2400640 1 -- start start 2024-03-06T07:31:00.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.348+0000 7fa4b2400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa4ac0b0f90 0x7fa4ac0cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.348+0000 7fa4b2400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0cc730 0x7fa4ac0d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.348+0000 7fa4b2400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4ac0d05e0 0x7fa4ac15d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.348+0000 7fa4b2400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac0d0a50 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.348+0000 7fa4b2400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac0b1370 con 0x7fa4ac0b0f90 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.348+0000 7fa4b2400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0ccf80 con 0x7fa4ac0cc730 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4ac0d05e0 0x7fa4ac15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4ac0d05e0 0x7fa4ac15d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49376/0 (socket says 172.21.15.6:49376) 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 -- 172.21.15.6:0/1115568242 learned_addr learned my addr 172.21.15.6:0/1115568242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4abe00640 1 --2- 172.21.15.6:0/1115568242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0cc730 0x7fa4ac0d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1000640 1 --2- 172.21.15.6:0/1115568242 >> v2:172.21.15.178:3300/0 conn(0x7fa4ac0b0f90 0x7fa4ac0cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 -- 172.21.15.6:0/1115568242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0cc730 msgr2=0x7fa4ac0d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 --2- 172.21.15.6:0/1115568242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0cc730 0x7fa4ac0d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 -- 172.21.15.6:0/1115568242 >> v2:172.21.15.178:3300/0 conn(0x7fa4ac0b0f90 msgr2=0x7fa4ac0cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 --2- 172.21.15.6:0/1115568242 >> v2:172.21.15.178:3300/0 conn(0x7fa4ac0b0f90 0x7fa4ac0cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.349+0000 7fa4b1a00640 1 -- 172.21.15.6:0/1115568242 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0cd260 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.351+0000 7fa4b1a00640 1 --2- 172.21.15.6:0/1115568242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac15d750 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac071b80 tx=0x7fa4a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a90c72043ca94b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.351+0000 7fa4ab400640 1 -- 172.21.15.6:0/1115568242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a402c9e0 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.351+0000 7fa4ab400640 1 -- 172.21.15.6:0/1115568242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a402cb80 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.351+0000 7fa4ab400640 1 -- 172.21.15.6:0/1115568242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40343f0 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 -- 172.21.15.6:0/1115568242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 msgr2=0x7fa4ac15d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 --2- 172.21.15.6:0/1115568242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac15d750 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac071b80 tx=0x7fa4a402be40 comp rx=0 tx=0).stop 2024-03-06T07:31:00.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 -- 172.21.15.6:0/1115568242 shutdown_connections 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 --2- 172.21.15.6:0/1115568242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac15d750 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 --2- 172.21.15.6:0/1115568242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0cc730 0x7fa4ac0d00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 --2- 172.21.15.6:0/1115568242 >> v2:172.21.15.178:3300/0 conn(0x7fa4ac0b0f90 0x7fa4ac0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 -- 172.21.15.6:0/1115568242 >> 172.21.15.6:0/1115568242 conn(0x7fa4ac0d1ae0 msgr2=0x7fa4ac0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 -- 172.21.15.6:0/1115568242 shutdown_connections 2024-03-06T07:31:00.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.352+0000 7fa4b2400640 1 -- 172.21.15.6:0/1115568242 wait complete. 2024-03-06T07:31:00.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.354+0000 7fa4b2400640 1 Processor -- start 2024-03-06T07:31:00.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.354+0000 7fa4b2400640 1 -- start start 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b2400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0b0f90 0x7fa4ac0cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b2400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0cc730 0x7fa4ac0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b2400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac1561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b2400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0ccb70 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b2400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0cd820 con 0x7fa4ac0cc730 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b2400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0cccf0 con 0x7fa4ac0b0f90 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac1561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac1561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49388/0 (socket says 172.21.15.6:49388) 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.355+0000 7fa4b1a00640 1 -- 172.21.15.6:0/796509371 learned_addr learned my addr 172.21.15.6:0/796509371 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1000640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0b0f90 0x7fa4ac0cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1a00640 1 -- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0b0f90 msgr2=0x7fa4ac0cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1a00640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0b0f90 0x7fa4ac0cb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4abe00640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0cc730 0x7fa4ac0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1a00640 1 -- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0cc730 msgr2=0x7fa4ac0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1a00640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0cc730 0x7fa4ac0c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1a00640 1 -- 172.21.15.6:0/796509371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0c9400 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4abe00640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0cc730 0x7fa4ac0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.356+0000 7fa4b1a00640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac1561e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4036040 tx=0x7fa4a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.357+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4043020 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.357+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a403e450 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.357+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4009510 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.357+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac0c96e0 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.357+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac0c9ca0 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.358+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac04f330 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.359+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4a4046020 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.360+0000 7fa4a8c00640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa48c06b4d0 0x7fa48c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.360+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4a40acab0 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.360+0000 7fa4b1000640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa48c06b4d0 0x7fa48c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.361+0000 7fa4b1000640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa48c06b4d0 0x7fa48c06d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0006fd0 tx=0x7fa4a0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.364+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4a404d030 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.590+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4ac0cbc40 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.593+0000 7fa4a8c00640 1 -- 172.21.15.6:0/796509371 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fa4a407b920 con 0x7fa4ac0d05e0 2024-03-06T07:31:00.596 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:00.596 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":58,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:22.633658+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa48c06b4d0 msgr2=0x7fa48c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa48c06b4d0 0x7fa48c06d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0006fd0 tx=0x7fa4a0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 msgr2=0x7fa4ac1561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac1561e0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4036040 tx=0x7fa4a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 shutdown_connections 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa48c06b4d0 0x7fa48c06d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0cc730 0x7fa4ac0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4ac0d05e0 0x7fa4ac1561e0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 --2- 172.21.15.6:0/796509371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4ac0b0f90 0x7fa4ac0cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 >> 172.21.15.6:0/796509371 conn(0x7fa4ac0d1ae0 msgr2=0x7fa4ac06cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 shutdown_connections 2024-03-06T07:31:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.596+0000 7fa4b2400640 1 -- 172.21.15.6:0/796509371 wait complete. 2024-03-06T07:31:00.620 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:00.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:00.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.831+0000 7ff391400640 1 Processor -- start 2024-03-06T07:31:00.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.831+0000 7ff391400640 1 -- start start 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.834+0000 7ff391400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff38c0cdc20 0x7ff38c071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.834+0000 7ff391400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0b4860 con 0x7ff38c0b3c90 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.834+0000 7ff391400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38c0b4b40 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.834+0000 7ff391400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38c0b4e20 con 0x7ff38c0cdc20 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.834+0000 7ff38a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff38c0b3c90 0x7ff38c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43188/0 (socket says 172.21.15.6:43188) 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.834+0000 7ff38a800640 1 -- 172.21.15.6:0/43214716 learned_addr learned my addr 172.21.15.6:0/43214716 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:00.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.835+0000 7ff38b200640 1 --2- 172.21.15.6:0/43214716 >> v2:172.21.15.6:3300/0 conn(0x7ff38c0cdc20 0x7ff38c071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.835+0000 7ff389400640 1 -- 172.21.15.6:0/43214716 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505515607 0 0) 0x7ff38c0b4860 con 0x7ff38c0b3c90 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.835+0000 7ff389400640 1 -- 172.21.15.6:0/43214716 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff370005730 con 0x7ff38c0b3c90 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.835+0000 7ff38b200640 1 -- 172.21.15.6:0/43214716 >> v1:172.21.15.178:6789/0 conn(0x7ff38c0cf5c0 legacy=0x7ff38c071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.835+0000 7ff38b200640 1 -- 172.21.15.6:0/43214716 >> v1:172.21.15.6:6789/0 conn(0x7ff38c0b3c90 legacy=0x7ff38c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.835+0000 7ff38b200640 1 -- 172.21.15.6:0/43214716 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0b5100 con 0x7ff38c0cdc20 2024-03-06T07:31:00.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.836+0000 7ff38b200640 1 --2- 172.21.15.6:0/43214716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cdc20 0x7ff38c071760 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff384009d90 tx=0x7ff38402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a651ace7e0e4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff389400640 1 -- 172.21.15.6:0/43214716 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384039070 con 0x7ff38c0cdc20 2024-03-06T07:31:00.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff389400640 1 -- 172.21.15.6:0/43214716 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38402cd80 con 0x7ff38c0cdc20 2024-03-06T07:31:00.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff389400640 1 -- 172.21.15.6:0/43214716 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384034660 con 0x7ff38c0cdc20 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff391400640 1 -- 172.21.15.6:0/43214716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cdc20 msgr2=0x7ff38c071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff391400640 1 --2- 172.21.15.6:0/43214716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cdc20 0x7ff38c071760 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff384009d90 tx=0x7ff38402bfd0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff391400640 1 -- 172.21.15.6:0/43214716 shutdown_connections 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff391400640 1 --2- 172.21.15.6:0/43214716 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cdc20 0x7ff38c071760 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.837+0000 7ff391400640 1 -- 172.21.15.6:0/43214716 >> 172.21.15.6:0/43214716 conn(0x7ff38c06afb0 msgr2=0x7ff38c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.838+0000 7ff391400640 1 -- 172.21.15.6:0/43214716 shutdown_connections 2024-03-06T07:31:00.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.838+0000 7ff391400640 1 -- 172.21.15.6:0/43214716 wait complete. 2024-03-06T07:31:00.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.839+0000 7ff391400640 1 Processor -- start 2024-03-06T07:31:00.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.839+0000 7ff391400640 1 -- start start 2024-03-06T07:31:00.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.840+0000 7ff391400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff38c0b3c90 0x7ff38c150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.840+0000 7ff391400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff38c0cdc20 0x7ff38c149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.840+0000 7ff391400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 0x7ff38c06b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.840+0000 7ff391400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c071ca0 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.840+0000 7ff391400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c0ce060 con 0x7ff38c0cdc20 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.840+0000 7ff391400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38c071e20 con 0x7ff38c0b3c90 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 0x7ff38c06b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 0x7ff38c06b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49402/0 (socket says 172.21.15.6:49402) 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 -- 172.21.15.6:0/1875000028 learned_addr learned my addr 172.21.15.6:0/1875000028 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 -- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff38c0b3c90 msgr2=0x7ff38c150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff38c0b3c90 0x7ff38c150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 -- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff38c0cdc20 msgr2=0x7ff38c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff38c0cdc20 0x7ff38c149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 -- 172.21.15.6:0/1875000028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c14a3f0 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.841+0000 7ff38b200640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 0x7ff38c06b590 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff384009d90 tx=0x7ff384004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.842+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384039050 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.842+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff384045070 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.842+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384009450 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.842+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff38c151440 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.842+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff38c151a00 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.843+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff38c069510 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.844+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff38403e850 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.845+0000 7ff382a00640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff36c06b4d0 0x7ff36c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:00.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.845+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3840ad0f0 con 0x7ff38c0cf5c0 2024-03-06T07:31:00.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.846+0000 7ff38a800640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff36c06b4d0 0x7ff36c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:00.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.847+0000 7ff38a800640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff36c06b4d0 0x7ff36c06d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff37c001610 tx=0x7ff37c007930 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:00.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:00.850+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff38407c080 con 0x7ff38c0cf5c0 2024-03-06T07:31:01.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.092+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff38c09de90 con 0x7ff38c0cf5c0 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.093+0000 7ff382a00640 1 -- 172.21.15.6:0/1875000028 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7ff38407b980 con 0x7ff38c0cf5c0 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:01.095 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:31:01.096 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff36c06b4d0 msgr2=0x7ff36c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff36c06b4d0 0x7ff36c06d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff37c001610 tx=0x7ff37c007930 comp rx=0 tx=0).stop 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 msgr2=0x7ff38c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 0x7ff38c06b590 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff384009d90 tx=0x7ff384004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 shutdown_connections 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff36c06b4d0 0x7ff36c06d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff38c0cdc20 0x7ff38c149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff38c0cf5c0 0x7ff38c06b590 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 --2- 172.21.15.6:0/1875000028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff38c0b3c90 0x7ff38c150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 >> 172.21.15.6:0/1875000028 conn(0x7ff38c06afb0 msgr2=0x7ff38c070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 shutdown_connections 2024-03-06T07:31:01.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.096+0000 7ff391400640 1 -- 172.21.15.6:0/1875000028 wait complete. 2024-03-06T07:31:01.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:01.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.328+0000 7f4655000640 1 Processor -- start 2024-03-06T07:31:01.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.328+0000 7f4655000640 1 -- start start 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f4655000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f46500d1c30 0x7f46500d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f4655000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f46500cce40 0x7f46500cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f4655000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46500d02d0 con 0x7f465015f310 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f4655000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46500d25d0 con 0x7f46500d1c30 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f4655000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46500d05b0 con 0x7f46500cce40 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f464de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f465015f310 0x7f46500cf890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43200/0 (socket says 172.21.15.6:43200) 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f464de00640 1 -- 172.21.15.6:0/3760671726 learned_addr learned my addr 172.21.15.6:0/3760671726 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:01.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.330+0000 7f4647e00640 1 --2- 172.21.15.6:0/3760671726 >> v2:172.21.15.6:3300/0 conn(0x7f46500d1c30 0x7f46500d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f464e800640 1 --2- 172.21.15.6:0/3760671726 >> v2:172.21.15.178:3300/0 conn(0x7f46500cce40 0x7f46500cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f464d400640 1 -- 172.21.15.6:0/3760671726 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353528727 0 0) 0x7f46500d02d0 con 0x7f465015f310 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f464d400640 1 -- 172.21.15.6:0/3760671726 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4634003520 con 0x7f465015f310 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f4647e00640 1 -- 172.21.15.6:0/3760671726 >> v2:172.21.15.178:3300/0 conn(0x7f46500cce40 msgr2=0x7f46500cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f4647e00640 1 --2- 172.21.15.6:0/3760671726 >> v2:172.21.15.178:3300/0 conn(0x7f46500cce40 0x7f46500cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f4647e00640 1 -- 172.21.15.6:0/3760671726 >> v1:172.21.15.6:6789/0 conn(0x7f465015f310 legacy=0x7f46500cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.331+0000 7f4647e00640 1 -- 172.21.15.6:0/3760671726 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46500d0890 con 0x7f46500d1c30 2024-03-06T07:31:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.332+0000 7f4647e00640 1 --2- 172.21.15.6:0/3760671726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f46500d2090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f46500cc2c0 tx=0x7f463802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae686e61dae1119 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f464d400640 1 -- 172.21.15.6:0/3760671726 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638034070 con 0x7f46500d1c30 2024-03-06T07:31:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f464d400640 1 -- 172.21.15.6:0/3760671726 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638030610 con 0x7f46500d1c30 2024-03-06T07:31:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f464d400640 1 -- 172.21.15.6:0/3760671726 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380309c0 con 0x7f46500d1c30 2024-03-06T07:31:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 -- 172.21.15.6:0/3760671726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 msgr2=0x7f46500d2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 --2- 172.21.15.6:0/3760671726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f46500d2090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f46500cc2c0 tx=0x7f463802b960 comp rx=0 tx=0).stop 2024-03-06T07:31:01.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 -- 172.21.15.6:0/3760671726 shutdown_connections 2024-03-06T07:31:01.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 --2- 172.21.15.6:0/3760671726 >> v2:172.21.15.178:3300/0 conn(0x7f46500cce40 0x7f46500cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 --2- 172.21.15.6:0/3760671726 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f46500d2090 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 -- 172.21.15.6:0/3760671726 >> 172.21.15.6:0/3760671726 conn(0x7f465009aa10 msgr2=0x7f465009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.333+0000 7f4655000640 1 -- 172.21.15.6:0/3760671726 shutdown_connections 2024-03-06T07:31:01.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.334+0000 7f4655000640 1 -- 172.21.15.6:0/3760671726 wait complete. 2024-03-06T07:31:01.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.335+0000 7f4655000640 1 Processor -- start 2024-03-06T07:31:01.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.336+0000 7f4655000640 1 -- start start 2024-03-06T07:31:01.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.336+0000 7f4655000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46500cce40 0x7f465006c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.336+0000 7f4655000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f465015c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.336+0000 7f4655000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f465015f310 0x7f465014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.336+0000 7f4655000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500cffa0 con 0x7f46500d1c30 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4655000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500cc920 con 0x7f46500cce40 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4655000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465015f6f0 con 0x7f465015f310 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f465015c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f465015c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49416/0 (socket says 172.21.15.6:49416) 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 -- 172.21.15.6:0/2419608165 learned_addr learned my addr 172.21.15.6:0/2419608165 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f464de00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46500cce40 0x7f465006c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 -- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f465015f310 msgr2=0x7f465014dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f464e800640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f465015f310 0x7f465014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f465015f310 0x7f465014dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 -- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46500cce40 msgr2=0x7f465006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46500cce40 0x7f465006c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f4647e00640 1 -- 172.21.15.6:0/2419608165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465006cca0 con 0x7f46500d1c30 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f464de00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46500cce40 0x7f465006c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.337+0000 7f464e800640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f465015f310 0x7f465014dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.338+0000 7f4647e00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f465015c9d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f463802b9a0 tx=0x7f46380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.338+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638034040 con 0x7f46500d1c30 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.338+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638030410 con 0x7f46500d1c30 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.338+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638039c30 con 0x7f46500d1c30 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.338+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465006cf80 con 0x7f46500d1c30 2024-03-06T07:31:01.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.338+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465006d540 con 0x7f46500d1c30 2024-03-06T07:31:01.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.339+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465004f3a0 con 0x7f46500d1c30 2024-03-06T07:31:01.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.340+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4638037070 con 0x7f46500d1c30 2024-03-06T07:31:01.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.342+0000 7f4646000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.342+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46380ac260 con 0x7f46500d1c30 2024-03-06T07:31:01.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.342+0000 7f464de00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.343+0000 7f464de00640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f462c06b4d0 0x7f462c06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4640005880 tx=0x7f4640009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.346+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f463807b040 con 0x7f46500d1c30 2024-03-06T07:31:01.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.648+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46500b7ba0 con 0x7f46500d1c30 2024-03-06T07:31:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.650+0000 7f4646000640 1 -- 172.21.15.6:0/2419608165 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f463807a940 con 0x7f46500d1c30 2024-03-06T07:31:01.652 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:01.652 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:30:21.089049+0000","modified":"2024-03-06T07:30:24.129217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5955":{"gid":5955,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/1578886876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1578886876},{"type":"v1","addr":"172.21.15.6:6835","nonce":1578886876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5955,"qdb_cluster":[5955]},"id":4}]} 2024-03-06T07:31:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 36 2024-03-06T07:31:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f462c06b4d0 msgr2=0x7f462c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f462c06b4d0 0x7f462c06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4640005880 tx=0x7f4640009040 comp rx=0 tx=0).stop 2024-03-06T07:31:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 msgr2=0x7f465015c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f465015c9d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f463802b9a0 tx=0x7f46380046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 shutdown_connections 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f465015f310 0x7f465014dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46500d1c30 0x7f465015c9d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 --2- 172.21.15.6:0/2419608165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46500cce40 0x7f465006c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.653+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 >> 172.21.15.6:0/2419608165 conn(0x7f465009aa10 msgr2=0x7f465006ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.654+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 shutdown_connections 2024-03-06T07:31:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.654+0000 7f4655000640 1 -- 172.21.15.6:0/2419608165 wait complete. 2024-03-06T07:31:01.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.875+0000 7fc917200640 1 Processor -- start 2024-03-06T07:31:01.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.875+0000 7fc917200640 1 -- start start 2024-03-06T07:31:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.877+0000 7fc917200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc91015e030 0x7fc910161d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.877+0000 7fc917200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 0x7fc9100d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.877+0000 7fc917200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9100cea50 con 0x7fc9100cfd50 2024-03-06T07:31:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.877+0000 7fc917200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100cecf0 con 0x7fc91015e030 2024-03-06T07:31:01.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.877+0000 7fc917200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100cef90 con 0x7fc91015e410 2024-03-06T07:31:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.878+0000 7fc915200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 0x7fc9100d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.878+0000 7fc915200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 0x7fc9100d2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:53994/0 (socket says 172.21.15.6:53994) 2024-03-06T07:31:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.878+0000 7fc915200640 1 -- 172.21.15.6:0/1040359238 learned_addr learned my addr 172.21.15.6:0/1040359238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.878+0000 7fc90fe00640 1 --2- 172.21.15.6:0/1040359238 >> v2:172.21.15.178:3300/0 conn(0x7fc91015e030 0x7fc910161d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.879+0000 7fc915200640 1 -- 172.21.15.6:0/1040359238 >> v2:172.21.15.178:3300/0 conn(0x7fc91015e030 msgr2=0x7fc910161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.879+0000 7fc915200640 1 --2- 172.21.15.6:0/1040359238 >> v2:172.21.15.178:3300/0 conn(0x7fc91015e030 0x7fc910161d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.879+0000 7fc915200640 1 -- 172.21.15.6:0/1040359238 >> v1:172.21.15.178:6789/0 conn(0x7fc9100cfd50 legacy=0x7fc9100d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.879+0000 7fc915200640 1 -- 172.21.15.6:0/1040359238 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100cf270 con 0x7fc91015e410 2024-03-06T07:31:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.880+0000 7fc915200640 1 --2- 172.21.15.6:0/1040359238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 0x7fc9100d2ba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc908009e50 tx=0x7fc90802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89e626ab2253076c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.880+0000 7fc90f400640 1 -- 172.21.15.6:0/1040359238 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90802cb90 con 0x7fc91015e410 2024-03-06T07:31:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.880+0000 7fc90f400640 1 -- 172.21.15.6:0/1040359238 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90802cd30 con 0x7fc91015e410 2024-03-06T07:31:01.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.880+0000 7fc90f400640 1 -- 172.21.15.6:0/1040359238 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908034610 con 0x7fc91015e410 2024-03-06T07:31:01.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 -- 172.21.15.6:0/1040359238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 msgr2=0x7fc9100d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 --2- 172.21.15.6:0/1040359238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 0x7fc9100d2ba0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc908009e50 tx=0x7fc90802bff0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 -- 172.21.15.6:0/1040359238 shutdown_connections 2024-03-06T07:31:01.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 --2- 172.21.15.6:0/1040359238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e410 0x7fc9100d2ba0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 --2- 172.21.15.6:0/1040359238 >> v2:172.21.15.178:3300/0 conn(0x7fc91015e030 0x7fc910161d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 -- 172.21.15.6:0/1040359238 >> 172.21.15.6:0/1040359238 conn(0x7fc9100d3eb0 msgr2=0x7fc9100d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 -- 172.21.15.6:0/1040359238 shutdown_connections 2024-03-06T07:31:01.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.881+0000 7fc917200640 1 -- 172.21.15.6:0/1040359238 wait complete. 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.883+0000 7fc917200640 1 Processor -- start 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.883+0000 7fc917200640 1 -- start start 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc917200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e030 0x7fc91015b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc917200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc91015e410 0x7fc910158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc917200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 0x7fc9100b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc917200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc910162260 con 0x7fc910159bc0 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc917200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9100cf830 con 0x7fc91015e410 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc917200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9101623e0 con 0x7fc91015e030 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc915200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 0x7fc9100b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.884+0000 7fc915200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 0x7fc9100b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60946/0 (socket says 172.21.15.6:60946) 2024-03-06T07:31:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc915200640 1 -- 172.21.15.6:0/3541404000 learned_addr learned my addr 172.21.15.6:0/3541404000 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc907e00640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc91015e410 0x7fc910158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc90fe00640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e030 0x7fc91015b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc915200640 1 -- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e030 msgr2=0x7fc91015b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc915200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e030 0x7fc91015b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc915200640 1 -- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc91015e410 msgr2=0x7fc910158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc915200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc91015e410 0x7fc910158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.885+0000 7fc915200640 1 -- 172.21.15.6:0/3541404000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91015a130 con 0x7fc910159bc0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.886+0000 7fc915200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 0x7fc9100b1b20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc908009e50 tx=0x7fc908004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.886+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908043020 con 0x7fc910159bc0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.886+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90803e450 con 0x7fc910159bc0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.886+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908009510 con 0x7fc910159bc0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.886+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91015a410 con 0x7fc910159bc0 2024-03-06T07:31:01.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.886+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91015a9d0 con 0x7fc910159bc0 2024-03-06T07:31:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.887+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91004f3a0 con 0x7fc910159bc0 2024-03-06T07:31:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.888+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc908046020 con 0x7fc910159bc0 2024-03-06T07:31:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.889+0000 7fc90cc00640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8ec06b500 0x7fc8ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.889+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9080ac7e0 con 0x7fc910159bc0 2024-03-06T07:31:01.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.890+0000 7fc90fe00640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8ec06b500 0x7fc8ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.892+0000 7fc90fe00640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8ec06b500 0x7fc8ec06d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc900009ad0 tx=0x7fc900008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:01.894+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc90804d030 con 0x7fc910159bc0 2024-03-06T07:31:02.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.094+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9100ce140 con 0x7fc910159bc0 2024-03-06T07:31:02.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.096+0000 7fc90cc00640 1 -- 172.21.15.6:0/3541404000 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16593 (secure 0 0 0) 0x7fc90807b650 con 0x7fc910159bc0 2024-03-06T07:31:02.098 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:02.098 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":58,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:30:22.633658+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:30:19.028665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:30:20.117374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:02.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.098+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8ec06b500 msgr2=0x7fc8ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.098+0000 7fc917200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8ec06b500 0x7fc8ec06d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc900009ad0 tx=0x7fc900008040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.098+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 msgr2=0x7fc9100b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.098+0000 7fc917200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 0x7fc9100b1b20 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc908009e50 tx=0x7fc908004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:02.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 shutdown_connections 2024-03-06T07:31:02.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc8ec06b500 0x7fc8ec06d9c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc91015e410 0x7fc910158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc910159bc0 0x7fc9100b1b20 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 --2- 172.21.15.6:0/3541404000 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc91015e030 0x7fc91015b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 >> 172.21.15.6:0/3541404000 conn(0x7fc9100d3eb0 msgr2=0x7fc9100b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 shutdown_connections 2024-03-06T07:31:02.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.099+0000 7fc917200640 1 -- 172.21.15.6:0/3541404000 wait complete. 2024-03-06T07:31:02.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.325+0000 7fa7e3200640 1 Processor -- start 2024-03-06T07:31:02.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.326+0000 7fa7e3200640 1 -- start start 2024-03-06T07:31:02.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.327+0000 7fa7e3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 0x7fa7e4162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.327+0000 7fa7e3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e4147bc0 con 0x7fa7e406b6c0 2024-03-06T07:31:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.327+0000 7fa7e3200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7e40b3d70 con 0x7fa7e406ef20 2024-03-06T07:31:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.327+0000 7fa7e3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e4161f90 con 0x7fa7e41477e0 2024-03-06T07:31:02.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.328+0000 7fa7e2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa7e406b6c0 0x7fa7e40d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36448/0 (socket says 172.21.15.6:36448) 2024-03-06T07:31:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.328+0000 7fa7e2800640 1 -- 172.21.15.6:0/400698712 learned_addr learned my addr 172.21.15.6:0/400698712 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.328+0000 7fa7e1e00640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 0x7fa7e4162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.328+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664628050 0 0) 0x7fa7e4147bc0 con 0x7fa7e406b6c0 2024-03-06T07:31:02.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.328+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c8003520 con 0x7fa7e406b6c0 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915210194 0 0) 0x7fa7c8003520 con 0x7fa7e406b6c0 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 msgr2=0x7fa7e4162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 0x7fa7e4162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 >> v1:172.21.15.178:6789/0 conn(0x7fa7e406ef20 legacy=0x7fa7e406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e40b4050 con 0x7fa7e406b6c0 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fa7dc0019f0 con 0x7fa7e406b6c0 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 >> v1:172.21.15.6:6789/0 conn(0x7fa7e406b6c0 legacy=0x7fa7e40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 0x7fa7c80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7c8003f60 0x7fa7c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8004d40 0x7fa7c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e4161f90 con 0x7fa7c8003f60 2024-03-06T07:31:02.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c800bb50 con 0x7fa7c8004d40 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c800bd50 con 0x7fa7e41477e0 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.329+0000 7fa7e1400640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7c8003f60 0x7fa7c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.330+0000 7fa7e2800640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8004d40 0x7fa7c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.330+0000 7fa7e1400640 1 -- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 msgr2=0x7fa7c80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.330+0000 7fa7e1400640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e41477e0 0x7fa7c80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.330+0000 7fa7e1400640 1 -- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8004d40 msgr2=0x7fa7c80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.330+0000 7fa7e1400640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8004d40 0x7fa7c80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.330+0000 7fa7e1400640 1 -- 172.21.15.6:0/400698712 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c8003520 con 0x7fa7c8003f60 2024-03-06T07:31:02.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.331+0000 7fa7e1400640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7c8003f60 0x7fa7c8004800 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0001750 tx=0x7fa7c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebde852b0c7b4382 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.331+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0001370 con 0x7fa7c8003f60 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.331+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d00052d0 con 0x7fa7c8003f60 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.331+0000 7fa7e0a00640 1 -- 172.21.15.6:0/400698712 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0003830 con 0x7fa7c8003f60 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e3200640 1 -- 172.21.15.6:0/400698712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7c8003f60 msgr2=0x7fa7c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e3200640 1 --2- 172.21.15.6:0/400698712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7c8003f60 0x7fa7c8004800 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0001750 tx=0x7fa7c8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e2800640 1 -- 172.21.15.6:0/400698712 reap_dead start 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e3200640 1 -- 172.21.15.6:0/400698712 shutdown_connections 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e3200640 1 -- 172.21.15.6:0/400698712 >> 172.21.15.6:0/400698712 conn(0x7fa7e406c0c0 msgr2=0x7fa7e406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e3200640 1 -- 172.21.15.6:0/400698712 shutdown_connections 2024-03-06T07:31:02.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.332+0000 7fa7e3200640 1 -- 172.21.15.6:0/400698712 wait complete. 2024-03-06T07:31:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.335+0000 7fa7e3200640 1 Processor -- start 2024-03-06T07:31:02.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.335+0000 7fa7e3200640 1 -- start start 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8003f60 0x7fa7e41fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 0x7fa7e40abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e40cfa50 0x7fa7e40a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e406f8f0 con 0x7fa7e40d12a0 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e4161c20 con 0x7fa7c8003f60 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e406fa70 con 0x7fa7e40cfa50 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 0x7fa7e40abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 0x7fa7e40abe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60956/0 (socket says 172.21.15.6:60956) 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e1400640 1 -- 172.21.15.6:0/1825925663 learned_addr learned my addr 172.21.15.6:0/1825925663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e1e00640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8003f60 0x7fa7e41fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.337+0000 7fa7e2800640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e40cfa50 0x7fa7e40a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e1400640 1 -- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e40cfa50 msgr2=0x7fa7e40a9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e1400640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e40cfa50 0x7fa7e40a9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e1400640 1 -- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8003f60 msgr2=0x7fa7e41fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e1400640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8003f60 0x7fa7e41fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e1400640 1 -- 172.21.15.6:0/1825925663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e40aad60 con 0x7fa7e40d12a0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e1400640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 0x7fa7e40abe00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0001870 tx=0x7fa7d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0041070 con 0x7fa7e40d12a0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d0005480 con 0x7fa7e40d12a0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e40ab040 con 0x7fa7e40d12a0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.338+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0001370 con 0x7fa7e40d12a0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.339+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e40ab600 con 0x7fa7e40d12a0 2024-03-06T07:31:02.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.340+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e404f330 con 0x7fa7e40d12a0 2024-03-06T07:31:02.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.341+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7d0044070 con 0x7fa7e40d12a0 2024-03-06T07:31:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.342+0000 7fa7da000640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa7c406b4d0 0x7fa7c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.342+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d00abc60 con 0x7fa7e40d12a0 2024-03-06T07:31:02.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.343+0000 7fa7e1e00640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa7c406b4d0 0x7fa7c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.344+0000 7fa7e1e00640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa7c406b4d0 0x7fa7c406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009a50 tx=0x7fa7d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.346+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7d007ac90 con 0x7fa7e40d12a0 2024-03-06T07:31:02.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.590+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa7e40a9720 con 0x7fa7e40d12a0 2024-03-06T07:31:02.592 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:31:02.591+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:31:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.594+0000 7fa7da000640 1 -- 172.21.15.6:0/1825925663 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa7d007a590 con 0x7fa7e40d12a0 2024-03-06T07:31:02.596 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa7c406b4d0 msgr2=0x7fa7c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa7c406b4d0 0x7fa7c406d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009a50 tx=0x7fa7d4008040 comp rx=0 tx=0).stop 2024-03-06T07:31:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 msgr2=0x7fa7e40abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 0x7fa7e40abe00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa7d0001870 tx=0x7fa7d0001120 comp rx=0 tx=0).stop 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 shutdown_connections 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa7c406b4d0 0x7fa7c406d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa7e40cfa50 0x7fa7e40a9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7e40d12a0 0x7fa7e40abe00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 --2- 172.21.15.6:0/1825925663 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7c8003f60 0x7fa7e41fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 >> 172.21.15.6:0/1825925663 conn(0x7fa7e406c0c0 msgr2=0x7fa7e40adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.597+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 shutdown_connections 2024-03-06T07:31:02.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.598+0000 7fa7e3200640 1 -- 172.21.15.6:0/1825925663 wait complete. 2024-03-06T07:31:02.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:02.706 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:31:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.815+0000 7fd753400640 1 Processor -- start 2024-03-06T07:31:02.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.815+0000 7fd753400640 1 -- start start 2024-03-06T07:31:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.816+0000 7fd753400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd74c0cd1e0 0x7fd74c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.816+0000 7fd753400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 0x7fd74c0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd753400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c071c60 con 0x7fd74c15afa0 2024-03-06T07:31:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd753400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0cfa10 con 0x7fd74c0cd1e0 2024-03-06T07:31:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd753400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c071f40 con 0x7fd74c070ce0 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd750a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd74c15afa0 0x7fd74c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36464/0 (socket says 172.21.15.6:36464) 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd750a00640 1 -- 172.21.15.6:0/1499382523 learned_addr learned my addr 172.21.15.6:0/1499382523 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd743e00640 1 --2- 172.21.15.6:0/1499382523 >> v2:172.21.15.178:3300/0 conn(0x7fd74c0cd1e0 0x7fd74c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd751400640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 0x7fd74c0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.817+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705896453 0 0) 0x7fd74c071c60 con 0x7fd74c15afa0 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd730003520 con 0x7fd74c15afa0 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570622606 0 0) 0x7fd730003520 con 0x7fd74c15afa0 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 msgr2=0x7fd74c0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 0x7fd74c0b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 >> v2:172.21.15.178:3300/0 conn(0x7fd74c0cd1e0 msgr2=0x7fd74c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 --2- 172.21.15.6:0/1499382523 >> v2:172.21.15.178:3300/0 conn(0x7fd74c0cd1e0 0x7fd74c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c072220 con 0x7fd74c15afa0 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd73c001670 con 0x7fd74c15afa0 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 >> v1:172.21.15.6:6789/0 conn(0x7fd74c15afa0 legacy=0x7fd74c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7300039b0 0x7fd730003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd730004800 0x7fd730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 0x7fd7300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0cfa10 con 0x7fd7300039b0 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c071f40 con 0x7fd730004800 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73000b9c0 con 0x7fd74c070ce0 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.818+0000 7fd750a00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7300039b0 0x7fd730003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.819+0000 7fd743e00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd730004800 0x7fd730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.819+0000 7fd750a00640 1 -- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 msgr2=0x7fd7300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.819+0000 7fd750a00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd74c070ce0 0x7fd7300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.819+0000 7fd750a00640 1 -- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd730004800 msgr2=0x7fd730004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.819+0000 7fd750a00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd730004800 0x7fd730004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.819+0000 7fd750a00640 1 -- 172.21.15.6:0/1499382523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd730003520 con 0x7fd7300039b0 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.820+0000 7fd750a00640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7300039b0 0x7fd730003dd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c270 tx=0x7fd730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7d4bc0cffd6fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.820+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c007d30 con 0x7fd7300039b0 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.820+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c030070 con 0x7fd7300039b0 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.820+0000 7fd74be00640 1 -- 172.21.15.6:0/1499382523 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73c02cd60 con 0x7fd7300039b0 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd753400640 1 -- 172.21.15.6:0/1499382523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7300039b0 msgr2=0x7fd730003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd753400640 1 --2- 172.21.15.6:0/1499382523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7300039b0 0x7fd730003dd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c270 tx=0x7fd730005830 comp rx=0 tx=0).stop 2024-03-06T07:31:02.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd751400640 1 -- 172.21.15.6:0/1499382523 reap_dead start 2024-03-06T07:31:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd753400640 1 -- 172.21.15.6:0/1499382523 shutdown_connections 2024-03-06T07:31:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd753400640 1 -- 172.21.15.6:0/1499382523 >> 172.21.15.6:0/1499382523 conn(0x7fd74c06b3d0 msgr2=0x7fd74c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd753400640 1 -- 172.21.15.6:0/1499382523 shutdown_connections 2024-03-06T07:31:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.821+0000 7fd753400640 1 -- 172.21.15.6:0/1499382523 wait complete. 2024-03-06T07:31:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.823+0000 7fd753400640 1 Processor -- start 2024-03-06T07:31:02.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.823+0000 7fd753400640 1 -- start start 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd753400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7300039b0 0x7fd74c14bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd753400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd74c14d470 0x7fd74c142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd753400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 0x7fd74c14a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd753400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0cdab0 con 0x7fd74c14a3f0 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd753400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c071150 con 0x7fd74c14d470 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd753400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c0cdc30 con 0x7fd7300039b0 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.824+0000 7fd751400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 0x7fd74c14a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 0x7fd74c14a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60994/0 (socket says 172.21.15.6:60994) 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 -- 172.21.15.6:0/1875671534 learned_addr learned my addr 172.21.15.6:0/1875671534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd743e00640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd74c14d470 0x7fd74c142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 -- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7300039b0 msgr2=0x7fd74c14bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd750a00640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7300039b0 0x7fd74c14bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7300039b0 0x7fd74c14bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 -- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd74c14d470 msgr2=0x7fd74c142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd74c14d470 0x7fd74c142e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 -- 172.21.15.6:0/1875671534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c14c560 con 0x7fd74c14a3f0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.825+0000 7fd751400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 0x7fd74c14a870 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd74c15e900 tx=0x7fd74400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.826+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74400bce0 con 0x7fd74c14a3f0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.826+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74401a070 con 0x7fd74c14a3f0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.826+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744014640 con 0x7fd74c14a3f0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.826+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c14c840 con 0x7fd74c14a3f0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.826+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c14ce00 con 0x7fd74c14a3f0 2024-03-06T07:31:02.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.827+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd74c04f330 con 0x7fd74c14a3f0 2024-03-06T07:31:02.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.828+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd744005d10 con 0x7fd74c14a3f0 2024-03-06T07:31:02.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.829+0000 7fd749600640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd72806b4d0 0x7fd72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.829+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd74408a6c0 con 0x7fd74c14a3f0 2024-03-06T07:31:02.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.830+0000 7fd750a00640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd72806b4d0 0x7fd72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.831+0000 7fd750a00640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd72806b4d0 0x7fd72806d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd73c034000 tx=0x7fd73c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:02.834+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd744024020 con 0x7fd74c14a3f0 2024-03-06T07:31:03.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.073+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd74c072b90 con 0x7fd74c14a3f0 2024-03-06T07:31:03.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.597+0000 7fd749600640 1 -- 172.21.15.6:0/1875671534 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd744059440 con 0x7fd74c14a3f0 2024-03-06T07:31:03.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd72806b4d0 msgr2=0x7fd72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd72806b4d0 0x7fd72806d990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd73c034000 tx=0x7fd73c007000 comp rx=0 tx=0).stop 2024-03-06T07:31:03.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 msgr2=0x7fd74c14a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 0x7fd74c14a870 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd74c15e900 tx=0x7fd74400b480 comp rx=0 tx=0).stop 2024-03-06T07:31:03.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 shutdown_connections 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd72806b4d0 0x7fd72806d990 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd74c14d470 0x7fd74c142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd74c14a3f0 0x7fd74c14a870 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 --2- 172.21.15.6:0/1875671534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7300039b0 0x7fd74c14bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 >> 172.21.15.6:0/1875671534 conn(0x7fd74c06b3d0 msgr2=0x7fd74c0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 shutdown_connections 2024-03-06T07:31:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.601+0000 7fd753400640 1 -- 172.21.15.6:0/1875671534 wait complete. 2024-03-06T07:31:03.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:31:03.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.852+0000 7f8714400640 1 Processor -- start 2024-03-06T07:31:03.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.853+0000 7f8714400640 1 -- start start 2024-03-06T07:31:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.854+0000 7f8714400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c0aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.854+0000 7f8714400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f870c15cdc0 0x7f870c06cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.854+0000 7f8714400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c1531f0 con 0x7f870c0b3640 2024-03-06T07:31:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.854+0000 7f8714400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f870c0d04d0 con 0x7f870c15cdc0 2024-03-06T07:31:03.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.854+0000 7f8714400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c0b2b80 con 0x7f870c0ae7b0 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8712400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f870c15cdc0 0x7f870c06cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8711000640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f870c0b3640 0x7f870c0aeb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:52990/0 (socket says 172.21.15.6:52990) 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8711000640 1 -- 172.21.15.6:0/82819954 learned_addr learned my addr 172.21.15.6:0/82819954 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8711a00640 1 --2- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c0aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8712400640 1 -- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 msgr2=0x7f870c0aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8712400640 1 --2- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c0aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8712400640 1 -- 172.21.15.6:0/82819954 >> v1:172.21.15.178:6789/0 conn(0x7f870c0b3640 legacy=0x7f870c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.855+0000 7f8712400640 1 -- 172.21.15.6:0/82819954 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c06ebd0 con 0x7f870c15cdc0 2024-03-06T07:31:03.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.857+0000 7f8712400640 1 --2- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06cc70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8704009dc0 tx=0x7f870402be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd06d14261320ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.857+0000 7f870be00640 1 -- 172.21.15.6:0/82819954 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704039070 con 0x7f870c15cdc0 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.857+0000 7f870be00640 1 -- 172.21.15.6:0/82819954 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870402cc40 con 0x7f870c15cdc0 2024-03-06T07:31:03.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.857+0000 7f870be00640 1 -- 172.21.15.6:0/82819954 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704034440 con 0x7f870c15cdc0 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 -- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 msgr2=0x7f870c06cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 --2- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06cc70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8704009dc0 tx=0x7f870402be90 comp rx=0 tx=0).stop 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 -- 172.21.15.6:0/82819954 shutdown_connections 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 --2- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06cc70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 --2- 172.21.15.6:0/82819954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c0aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 -- 172.21.15.6:0/82819954 >> 172.21.15.6:0/82819954 conn(0x7f870c06afb0 msgr2=0x7f870c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 -- 172.21.15.6:0/82819954 shutdown_connections 2024-03-06T07:31:03.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.858+0000 7f8714400640 1 -- 172.21.15.6:0/82819954 wait complete. 2024-03-06T07:31:03.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.860+0000 7f8714400640 1 Processor -- start 2024-03-06T07:31:03.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.860+0000 7f8714400640 1 -- start start 2024-03-06T07:31:03.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8714400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8714400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 0x7f870c14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8714400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8714400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c0b3120 con 0x7f870c0b3640 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8714400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c0b2e40 con 0x7f870c15cdc0 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8714400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c15d250 con 0x7f870c0ae7b0 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.861+0000 7f8711000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 0x7f870c14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 0x7f870c14e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60998/0 (socket says 172.21.15.6:60998) 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 -- 172.21.15.6:0/2584450711 learned_addr learned my addr 172.21.15.6:0/2584450711 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711a00640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8712400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 -- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 msgr2=0x7f870c14daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c14daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 -- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 msgr2=0x7f870c06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 -- 172.21.15.6:0/2584450711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c155bd0 con 0x7f870c0b3640 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711a00640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.862+0000 7f8711000640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 0x7f870c14e000 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f86f800c430 tx=0x7f86f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.863+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8017070 con 0x7f870c0b3640 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.863+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f8005ce0 con 0x7f870c0b3640 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.863+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f80072f0 con 0x7f870c0b3640 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.863+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870c155eb0 con 0x7f870c0b3640 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.863+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c1e5b00 con 0x7f870c0b3640 2024-03-06T07:31:03.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.864+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f870c068010 con 0x7f870c0b3640 2024-03-06T07:31:03.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.865+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86f801a070 con 0x7f870c0b3640 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.866+0000 7f8709600640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f86ec06b4d0 0x7f86ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.866+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f86f8089720 con 0x7f870c0b3640 2024-03-06T07:31:03.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.867+0000 7f8711a00640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f86ec06b4d0 0x7f86ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:03.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.868+0000 7f8711a00640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f86ec06b4d0 0x7f86ec06d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f870c14fff0 tx=0x7f8700008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:03.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:03.871+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f86f80584a0 con 0x7f870c0b3640 2024-03-06T07:31:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.073+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f870c1e5cd0 con 0x7f870c0b3640 2024-03-06T07:31:04.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.785+0000 7f8709600640 1 -- 172.21.15.6:0/2584450711 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) v1 ==== 181+0+0 (secure 0 0 0) 0x7f86f8057da0 con 0x7f870c0b3640 2024-03-06T07:31:04.788 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:31:04.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f86ec06b4d0 msgr2=0x7f86ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f86ec06b4d0 0x7f86ec06d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f870c14fff0 tx=0x7f8700008040 comp rx=0 tx=0).stop 2024-03-06T07:31:04.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 msgr2=0x7f870c14e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 0x7f870c14e000 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f86f800c430 tx=0x7f86f800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 shutdown_connections 2024-03-06T07:31:04.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f86ec06b4d0 0x7f86ec06d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f870c15cdc0 0x7f870c06dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f870c0b3640 0x7f870c14e000 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 --2- 172.21.15.6:0/2584450711 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f870c0ae7b0 0x7f870c14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.789+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 >> 172.21.15.6:0/2584450711 conn(0x7f870c06afb0 msgr2=0x7f870c0b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.790+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 shutdown_connections 2024-03-06T07:31:04.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:04.791+0000 7f8714400640 1 -- 172.21.15.6:0/2584450711 wait complete. 2024-03-06T07:31:04.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:31:05.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.033+0000 7f873e400640 1 Processor -- start 2024-03-06T07:31:05.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.033+0000 7f873e400640 1 -- start start 2024-03-06T07:31:05.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f873e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87380ae7b0 0x7f87380aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f873e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380b3640 0x7f873815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f873e400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87381531f0 con 0x7f87380aeb90 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f873e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87380d04d0 con 0x7f87380ae7b0 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f873e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873815d300 con 0x7f87380b3640 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f8737400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87380ae7b0 0x7f87380aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f8737400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f87380ae7b0 0x7f87380aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32772/0 (socket says 172.21.15.6:32772) 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.035+0000 7f8737400640 1 -- 172.21.15.6:0/1382546252 learned_addr learned my addr 172.21.15.6:0/1382546252 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.036+0000 7f8736a00640 1 --2- 172.21.15.6:0/1382546252 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380b3640 0x7f873815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.036+0000 7f8737400640 1 -- 172.21.15.6:0/1382546252 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380b3640 msgr2=0x7f873815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.036+0000 7f8737400640 1 --2- 172.21.15.6:0/1382546252 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380b3640 0x7f873815cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.036+0000 7f8737400640 1 -- 172.21.15.6:0/1382546252 >> v1:172.21.15.178:6789/0 conn(0x7f87380aeb90 legacy=0x7f873806cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.036+0000 7f8737400640 1 -- 172.21.15.6:0/1382546252 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87380b2b80 con 0x7f87380ae7b0 2024-03-06T07:31:05.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.037+0000 7f8737400640 1 --2- 172.21.15.6:0/1382546252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380ae7b0 0x7f87380aced0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f87280098a0 tx=0x7f872802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dd757fad2587b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f8736000640 1 -- 172.21.15.6:0/1382546252 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728034070 con 0x7f87380ae7b0 2024-03-06T07:31:05.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f8736000640 1 -- 172.21.15.6:0/1382546252 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8728030840 con 0x7f87380ae7b0 2024-03-06T07:31:05.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f8736000640 1 -- 172.21.15.6:0/1382546252 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8728030c10 con 0x7f87380ae7b0 2024-03-06T07:31:05.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 -- 172.21.15.6:0/1382546252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380ae7b0 msgr2=0x7f87380aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 --2- 172.21.15.6:0/1382546252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380ae7b0 0x7f87380aced0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f87280098a0 tx=0x7f872802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 -- 172.21.15.6:0/1382546252 shutdown_connections 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 --2- 172.21.15.6:0/1382546252 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380b3640 0x7f873815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 --2- 172.21.15.6:0/1382546252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380ae7b0 0x7f87380aced0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 -- 172.21.15.6:0/1382546252 >> 172.21.15.6:0/1382546252 conn(0x7f873806afb0 msgr2=0x7f873806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 -- 172.21.15.6:0/1382546252 shutdown_connections 2024-03-06T07:31:05.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.038+0000 7f873e400640 1 -- 172.21.15.6:0/1382546252 wait complete. 2024-03-06T07:31:05.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.040+0000 7f873e400640 1 Processor -- start 2024-03-06T07:31:05.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.040+0000 7f873e400640 1 -- start start 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.041+0000 7f873e400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87380ae7b0 0x7f8738070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.041+0000 7f873e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380aeb90 0x7f87381447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f873e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380b3640 0x7f873815d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f873e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b2e40 con 0x7f87380b3640 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f873e400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873806eeb0 con 0x7f87380ae7b0 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f873e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87380b3120 con 0x7f87380aeb90 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87380ae7b0 0x7f8738070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380b3640 0x7f873815d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8736a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380aeb90 0x7f87381447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87380ae7b0 0x7f8738070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:49416/0 (socket says 172.21.15.6:49416) 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737400640 1 -- 172.21.15.6:0/1317526518 learned_addr learned my addr 172.21.15.6:0/1317526518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737e00640 1 -- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380aeb90 msgr2=0x7f87381447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737e00640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380aeb90 0x7f87381447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737e00640 1 -- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87380ae7b0 msgr2=0x7f8738070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737e00640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87380ae7b0 0x7f8738070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.042+0000 7f8737e00640 1 -- 172.21.15.6:0/1317526518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873809bcb0 con 0x7f87380b3640 2024-03-06T07:31:05.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.043+0000 7f8737e00640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380b3640 0x7f873815d480 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f873000c700 tx=0x7f873000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.043+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730003b70 con 0x7f87380b3640 2024-03-06T07:31:05.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.043+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730003d10 con 0x7f87380b3640 2024-03-06T07:31:05.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.043+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730005310 con 0x7f87380b3640 2024-03-06T07:31:05.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.043+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873809bf90 con 0x7f87380b3640 2024-03-06T07:31:05.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.043+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873809c550 con 0x7f87380b3640 2024-03-06T07:31:05.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.044+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8738068010 con 0x7f87380b3640 2024-03-06T07:31:05.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.045+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87300054b0 con 0x7f87380b3640 2024-03-06T07:31:05.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.046+0000 7f872f400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f871806b4d0 0x7f871806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.046+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f8730089d00 con 0x7f87380b3640 2024-03-06T07:31:05.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.047+0000 7f8737400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f871806b4d0 0x7f871806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.048+0000 7f8737400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f871806b4d0 0x7f871806d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f87280098a0 tx=0x7f8728008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.051+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8730058680 con 0x7f87380b3640 2024-03-06T07:31:05.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.250+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f87380b4670 con 0x7f87380b3640 2024-03-06T07:31:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.800+0000 7f872f400640 1 -- 172.21.15.6:0/1317526518 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) v1 ==== 169+0+0 (secure 0 0 0) 0x7f873005c720 con 0x7f87380b3640 2024-03-06T07:31:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.804+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f871806b4d0 msgr2=0x7f871806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.804+0000 7f873e400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f871806b4d0 0x7f871806d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f87280098a0 tx=0x7f8728008030 comp rx=0 tx=0).stop 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.804+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380b3640 msgr2=0x7f873815d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.804+0000 7f873e400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380b3640 0x7f873815d480 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f873000c700 tx=0x7f873000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.805+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 shutdown_connections 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.805+0000 7f873e400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f871806b4d0 0x7f871806d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.805+0000 7f873e400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87380b3640 0x7f873815d480 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.805+0000 7f873e400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87380aeb90 0x7f87381447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.805+0000 7f873e400640 1 --2- 172.21.15.6:0/1317526518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87380ae7b0 0x7f8738070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.805+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 >> 172.21.15.6:0/1317526518 conn(0x7f873806afb0 msgr2=0x7f873806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.806+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 shutdown_connections 2024-03-06T07:31:05.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:05.806+0000 7f873e400640 1 -- 172.21.15.6:0/1317526518 wait complete. 2024-03-06T07:31:05.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:31:06.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.042+0000 7f7303000640 1 Processor -- start 2024-03-06T07:31:06.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.042+0000 7f7303000640 1 -- start start 2024-03-06T07:31:06.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.044+0000 7f7303000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f72fc09b420 0x7f72fc0b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.044+0000 7f7303000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc0b6b10 con 0x7f72fc161f40 2024-03-06T07:31:06.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.044+0000 7f7303000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc06f0a0 con 0x7f72fc161b60 2024-03-06T07:31:06.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.044+0000 7f7303000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc0b68f0 con 0x7f72fc09b420 2024-03-06T07:31:06.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.045+0000 7f7301000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f72fc161f40 0x7f72fc06fb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36474/0 (socket says 172.21.15.6:36474) 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.045+0000 7f7301000640 1 -- 172.21.15.6:0/986865825 learned_addr learned my addr 172.21.15.6:0/986865825 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.045+0000 7f72f3e00640 1 --2- 172.21.15.6:0/986865825 >> v2:172.21.15.178:3300/0 conn(0x7f72fc09b420 0x7f72fc0b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.045+0000 7f72fb400640 1 -- 172.21.15.6:0/986865825 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119319827 0 0) 0x7f72fc0b6b10 con 0x7f72fc161f40 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.045+0000 7f72fb400640 1 -- 172.21.15.6:0/986865825 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e0003520 con 0x7f72fc161f40 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.046+0000 7f72fb400640 1 -- 172.21.15.6:0/986865825 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2007203820 0 0) 0x7f72e0003520 con 0x7f72fc161f40 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.046+0000 7f72f3e00640 1 -- 172.21.15.6:0/986865825 >> v1:172.21.15.178:6789/0 conn(0x7f72fc161b60 legacy=0x7f72fc0aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.046+0000 7f72f3e00640 1 -- 172.21.15.6:0/986865825 >> v1:172.21.15.6:6789/0 conn(0x7f72fc161f40 legacy=0x7f72fc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.046+0000 7f72f3e00640 1 -- 172.21.15.6:0/986865825 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc06f380 con 0x7f72fc09b420 2024-03-06T07:31:06.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.047+0000 7f72f3e00640 1 --2- 172.21.15.6:0/986865825 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc0b63b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f72e40097a0 tx=0x7f72e402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2b92793a8de9ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f72fb400640 1 -- 172.21.15.6:0/986865825 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4038070 con 0x7f72fc09b420 2024-03-06T07:31:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f72fb400640 1 -- 172.21.15.6:0/986865825 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4031500 con 0x7f72fc09b420 2024-03-06T07:31:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f72fb400640 1 -- 172.21.15.6:0/986865825 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40019f0 con 0x7f72fc09b420 2024-03-06T07:31:06.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f7303000640 1 -- 172.21.15.6:0/986865825 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 msgr2=0x7f72fc0b63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f7303000640 1 --2- 172.21.15.6:0/986865825 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc0b63b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f72e40097a0 tx=0x7f72e402b820 comp rx=0 tx=0).stop 2024-03-06T07:31:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f7303000640 1 -- 172.21.15.6:0/986865825 shutdown_connections 2024-03-06T07:31:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f7303000640 1 --2- 172.21.15.6:0/986865825 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc0b63b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.048+0000 7f7303000640 1 -- 172.21.15.6:0/986865825 >> 172.21.15.6:0/986865825 conn(0x7f72fc0b7900 msgr2=0x7f72fc09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.049+0000 7f7303000640 1 -- 172.21.15.6:0/986865825 shutdown_connections 2024-03-06T07:31:06.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.049+0000 7f7303000640 1 -- 172.21.15.6:0/986865825 wait complete. 2024-03-06T07:31:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.050+0000 7f7303000640 1 Processor -- start 2024-03-06T07:31:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.051+0000 7f7303000640 1 -- start start 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.051+0000 7f7303000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.051+0000 7f7303000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 0x7f72fc151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f7303000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72fc161f40 0x7f72fc151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f7303000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc162510 con 0x7f72fc161b60 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f7303000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc162390 con 0x7f72fc09b420 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f7303000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc158810 con 0x7f72fc161f40 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f72f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 0x7f72fc151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f72f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 0x7f72fc151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32788/0 (socket says 172.21.15.6:32788) 2024-03-06T07:31:06.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f72f3e00640 1 -- 172.21.15.6:0/849493671 learned_addr learned my addr 172.21.15.6:0/849493671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f72fbe00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.052+0000 7f7301000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72fc161f40 0x7f72fc151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72f3e00640 1 -- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72fc161f40 msgr2=0x7f72fc151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72f3e00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72fc161f40 0x7f72fc151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72f3e00640 1 -- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 msgr2=0x7f72fc156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72f3e00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72f3e00640 1 -- 172.21.15.6:0/849493671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc1e9070 con 0x7f72fc161b60 2024-03-06T07:31:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72fbe00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f7301000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72fc161f40 0x7f72fc151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:06.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.053+0000 7f72f3e00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 0x7f72fc151830 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f72e4036040 tx=0x7f72e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.054+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4043070 con 0x7f72fc161b60 2024-03-06T07:31:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.054+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e403f870 con 0x7f72fc161b60 2024-03-06T07:31:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.054+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc1e9350 con 0x7f72fc161b60 2024-03-06T07:31:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.054+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4038040 con 0x7f72fc161b60 2024-03-06T07:31:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.054+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc1e9910 con 0x7f72fc161b60 2024-03-06T07:31:06.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.055+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc04f330 con 0x7f72fc161b60 2024-03-06T07:31:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.056+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72e4046070 con 0x7f72fc161b60 2024-03-06T07:31:06.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.058+0000 7f72f8c00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72d806b4d0 0x7f72d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.058+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f72e40abc60 con 0x7f72fc161b60 2024-03-06T07:31:06.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.058+0000 7f72fbe00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72d806b4d0 0x7f72d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.060+0000 7f72fbe00640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72d806b4d0 0x7f72d806d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f72ec001700 tx=0x7f72ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.062+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72e4030070 con 0x7f72fc161b60 2024-03-06T07:31:06.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.262+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f72fc09b120 con 0x7f72fc161b60 2024-03-06T07:31:06.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.267+0000 7f72f8c00640 1 -- 172.21.15.6:0/849493671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v19) v1 ==== 101+0+0 (secure 0 0 0) 0x7f72e407aef0 con 0x7f72fc161b60 2024-03-06T07:31:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72d806b4d0 msgr2=0x7f72d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72d806b4d0 0x7f72d806d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f72ec001700 tx=0x7f72ec009040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 msgr2=0x7f72fc151830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 0x7f72fc151830 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f72e4036040 tx=0x7f72e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 shutdown_connections 2024-03-06T07:31:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f72d806b4d0 0x7f72d806d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72fc161f40 0x7f72fc151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f72fc161b60 0x7f72fc151830 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 --2- 172.21.15.6:0/849493671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f72fc09b420 0x7f72fc156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 >> 172.21.15.6:0/849493671 conn(0x7f72fc0b7900 msgr2=0x7f72fc15e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 shutdown_connections 2024-03-06T07:31:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.270+0000 7f7303000640 1 -- 172.21.15.6:0/849493671 wait complete. 2024-03-06T07:31:06.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:31:06.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.506+0000 7fc709800640 1 Processor -- start 2024-03-06T07:31:06.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.506+0000 7fc709800640 1 -- start start 2024-03-06T07:31:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.507+0000 7fc709800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040d4860 0x7fc7040d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.507+0000 7fc709800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc7040cfa60 0x7fc7040d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.507+0000 7fc709800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040ceca0 con 0x7fc70415dee0 2024-03-06T07:31:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.507+0000 7fc709800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040cf5d0 con 0x7fc7040cfa60 2024-03-06T07:31:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.507+0000 7fc709800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040cef80 con 0x7fc7040d4860 2024-03-06T07:31:06.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.508+0000 7fc708e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc7040cfa60 0x7fc7040d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.508+0000 7fc703400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040d4860 0x7fc7040d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.508+0000 7fc703e00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fc70415dee0 0x7fc7040d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:53010/0 (socket says 172.21.15.6:53010) 2024-03-06T07:31:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.508+0000 7fc708e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc7040cfa60 0x7fc7040d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:49442/0 (socket says 172.21.15.6:49442) 2024-03-06T07:31:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.508+0000 7fc703e00640 1 -- 172.21.15.6:0/691158567 learned_addr learned my addr 172.21.15.6:0/691158567 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.509+0000 7fc708e00640 1 -- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040d4860 msgr2=0x7fc7040d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.509+0000 7fc708e00640 1 --2- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040d4860 0x7fc7040d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.509+0000 7fc708e00640 1 -- 172.21.15.6:0/691158567 >> v1:172.21.15.178:6789/0 conn(0x7fc70415dee0 legacy=0x7fc7040d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.509+0000 7fc708e00640 1 -- 172.21.15.6:0/691158567 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040cf260 con 0x7fc7040cfa60 2024-03-06T07:31:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.511+0000 7fc708e00640 1 --2- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7040cfa60 0x7fc7040d0b70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0097a0 tx=0x7fc6fc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2789f66986f9e2da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc702a00640 1 -- 172.21.15.6:0/691158567 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc035020 con 0x7fc7040cfa60 2024-03-06T07:31:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc702a00640 1 -- 172.21.15.6:0/691158567 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc031810 con 0x7fc7040cfa60 2024-03-06T07:31:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc702a00640 1 -- 172.21.15.6:0/691158567 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc031be0 con 0x7fc7040cfa60 2024-03-06T07:31:06.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc709800640 1 -- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7040cfa60 msgr2=0x7fc7040d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc709800640 1 --2- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7040cfa60 0x7fc7040d0b70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0097a0 tx=0x7fc6fc02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc709800640 1 -- 172.21.15.6:0/691158567 shutdown_connections 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc709800640 1 --2- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7040cfa60 0x7fc7040d0b70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc709800640 1 --2- 172.21.15.6:0/691158567 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040d4860 0x7fc7040d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.512+0000 7fc709800640 1 -- 172.21.15.6:0/691158567 >> 172.21.15.6:0/691158567 conn(0x7fc7040d3cd0 msgr2=0x7fc7040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.513+0000 7fc709800640 1 -- 172.21.15.6:0/691158567 shutdown_connections 2024-03-06T07:31:06.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.513+0000 7fc709800640 1 -- 172.21.15.6:0/691158567 wait complete. 2024-03-06T07:31:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.515+0000 7fc709800640 1 Processor -- start 2024-03-06T07:31:06.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.515+0000 7fc709800640 1 -- start start 2024-03-06T07:31:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.516+0000 7fc709800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040cfa60 0x7fc7040cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.516+0000 7fc709800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 0x7fc70415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.516+0000 7fc709800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc70415dee0 0x7fc704160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.516+0000 7fc709800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040b3510 con 0x7fc7040d4860 2024-03-06T07:31:06.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.516+0000 7fc709800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70415e2c0 con 0x7fc70415dee0 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.516+0000 7fc709800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040b3690 con 0x7fc7040cfa60 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 0x7fc70415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 0x7fc70415ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32794/0 (socket says 172.21.15.6:32794) 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 -- 172.21.15.6:0/418076028 learned_addr learned my addr 172.21.15.6:0/418076028 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703e00640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040cfa60 0x7fc7040cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc708e00640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc70415dee0 0x7fc704160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 -- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040cfa60 msgr2=0x7fc7040cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040cfa60 0x7fc7040cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 -- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc70415dee0 msgr2=0x7fc704160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc70415dee0 0x7fc704160010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.517+0000 7fc703400640 1 -- 172.21.15.6:0/418076028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704160550 con 0x7fc7040d4860 2024-03-06T07:31:06.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.518+0000 7fc703400640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 0x7fc70415ee10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009d70 tx=0x7fc6ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.518+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec018020 con 0x7fc7040d4860 2024-03-06T07:31:06.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.518+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec014480 con 0x7fc7040d4860 2024-03-06T07:31:06.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.518+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec014850 con 0x7fc7040d4860 2024-03-06T07:31:06.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.518+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70415f530 con 0x7fc7040d4860 2024-03-06T07:31:06.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.518+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7040ccce0 con 0x7fc7040d4860 2024-03-06T07:31:06.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.519+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70404f330 con 0x7fc7040d4860 2024-03-06T07:31:06.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.520+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6ec005d10 con 0x7fc7040d4860 2024-03-06T07:31:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.521+0000 7fc6fbe00640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6e006b4d0 0x7fc6e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.521+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc6ec08acd0 con 0x7fc7040d4860 2024-03-06T07:31:06.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.522+0000 7fc703e00640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6e006b4d0 0x7fc6e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.523+0000 7fc703e00640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6e006b4d0 0x7fc6e006d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc70415ce50 tx=0x7fc6f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.526+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6ec022030 con 0x7fc7040d4860 2024-03-06T07:31:06.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.736+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7fc70408d7d0 con 0x7fc7040d4860 2024-03-06T07:31:06.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.743+0000 7fc6fbe00640 1 -- 172.21.15.6:0/418076028 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v20) v1 ==== 108+0+0 (secure 0 0 0) 0x7fc6ec010040 con 0x7fc7040d4860 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6e006b4d0 msgr2=0x7fc6e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6e006b4d0 0x7fc6e006d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc70415ce50 tx=0x7fc6f4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 msgr2=0x7fc70415ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 0x7fc70415ee10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009d70 tx=0x7fc6ec00c490 comp rx=0 tx=0).stop 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 shutdown_connections 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6e006b4d0 0x7fc6e006d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc70415dee0 0x7fc704160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7040d4860 0x7fc70415ee10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 --2- 172.21.15.6:0/418076028 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7040cfa60 0x7fc7040cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 >> 172.21.15.6:0/418076028 conn(0x7fc7040d3cd0 msgr2=0x7fc7040d18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 shutdown_connections 2024-03-06T07:31:06.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.746+0000 7fc709800640 1 -- 172.21.15.6:0/418076028 wait complete. 2024-03-06T07:31:06.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:31:06.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.982+0000 7f54e9400640 1 Processor -- start 2024-03-06T07:31:06.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.983+0000 7f54e9400640 1 -- start start 2024-03-06T07:31:06.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.984+0000 7f54e9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f54e40c8d20 0x7f54e40cf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.984+0000 7f54e9400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e40d20d0 con 0x7f54e4161930 2024-03-06T07:31:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.984+0000 7f54e9400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e40d4df0 con 0x7f54e40d2bd0 2024-03-06T07:31:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.984+0000 7f54e9400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e40d1df0 con 0x7f54e40c8d20 2024-03-06T07:31:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e2800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54e4161930 0x7f54e40d24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36486/0 (socket says 172.21.15.6:36486) 2024-03-06T07:31:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e2800640 1 -- 172.21.15.6:0/1979304366 learned_addr learned my addr 172.21.15.6:0/1979304366 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:06.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e3200640 1 --2- 172.21.15.6:0/1979304366 >> v2:172.21.15.6:3300/0 conn(0x7f54e40c8d20 0x7f54e40cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e1400640 1 -- 172.21.15.6:0/1979304366 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478112261 0 0) 0x7f54e40d20d0 con 0x7f54e4161930 2024-03-06T07:31:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e1400640 1 -- 172.21.15.6:0/1979304366 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c8003520 con 0x7f54e4161930 2024-03-06T07:31:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e3200640 1 -- 172.21.15.6:0/1979304366 >> v1:172.21.15.178:6789/0 conn(0x7f54e40d2bd0 legacy=0x7f54e40cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e3200640 1 -- 172.21.15.6:0/1979304366 >> v1:172.21.15.6:6789/0 conn(0x7f54e4161930 legacy=0x7f54e40d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.985+0000 7f54e3200640 1 -- 172.21.15.6:0/1979304366 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e40d4b10 con 0x7f54e40c8d20 2024-03-06T07:31:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.986+0000 7f54e3200640 1 --2- 172.21.15.6:0/1979304366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e40c8d20 0x7f54e40cf190 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009d90 tx=0x7f54dc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b377a1fc187b8c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e1400640 1 -- 172.21.15.6:0/1979304366 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc039070 con 0x7f54e40c8d20 2024-03-06T07:31:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e1400640 1 -- 172.21.15.6:0/1979304366 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc02cbd0 con 0x7f54e40c8d20 2024-03-06T07:31:06.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e1400640 1 -- 172.21.15.6:0/1979304366 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc034440 con 0x7f54e40c8d20 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e9400640 1 -- 172.21.15.6:0/1979304366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e40c8d20 msgr2=0x7f54e40cf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e9400640 1 --2- 172.21.15.6:0/1979304366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e40c8d20 0x7f54e40cf190 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009d90 tx=0x7f54dc02be20 comp rx=0 tx=0).stop 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e9400640 1 -- 172.21.15.6:0/1979304366 shutdown_connections 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e9400640 1 --2- 172.21.15.6:0/1979304366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e40c8d20 0x7f54e40cf190 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.987+0000 7f54e9400640 1 -- 172.21.15.6:0/1979304366 >> 172.21.15.6:0/1979304366 conn(0x7f54e40d3ee0 msgr2=0x7f54e40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.988+0000 7f54e9400640 1 -- 172.21.15.6:0/1979304366 shutdown_connections 2024-03-06T07:31:06.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.988+0000 7f54e9400640 1 -- 172.21.15.6:0/1979304366 wait complete. 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.989+0000 7f54e9400640 1 Processor -- start 2024-03-06T07:31:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.990+0000 7f54e9400640 1 -- start start 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.990+0000 7f54e9400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54e40c8d20 0x7f54e415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e9400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e40d2bd0 0x7f54e409dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 0x7f54e406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e9400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40d0f80 con 0x7f54e4161930 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e9400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40b4f30 con 0x7f54e40d2bd0 2024-03-06T07:31:06.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e9400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40d1100 con 0x7f54e40c8d20 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 0x7f54e406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 0x7f54e406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32810/0 (socket says 172.21.15.6:32810) 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 -- 172.21.15.6:0/453603253 learned_addr learned my addr 172.21.15.6:0/453603253 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e1e00640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e40d2bd0 0x7f54e409dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e2800640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54e40c8d20 0x7f54e415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 -- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54e40c8d20 msgr2=0x7f54e415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54e40c8d20 0x7f54e415c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 -- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e40d2bd0 msgr2=0x7f54e409dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e40d2bd0 0x7f54e409dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.991+0000 7f54e3200640 1 -- 172.21.15.6:0/453603253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e41f4780 con 0x7f54e4161930 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.992+0000 7f54e3200640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 0x7f54e406ba80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f54dc037040 tx=0x7f54dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.992+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc039050 con 0x7f54e4161930 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.992+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc045070 con 0x7f54e4161930 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.992+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e41f4a60 con 0x7f54e4161930 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.992+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc009450 con 0x7f54e4161930 2024-03-06T07:31:06.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.992+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e41f5020 con 0x7f54e4161930 2024-03-06T07:31:06.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.994+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e41f5510 con 0x7f54e4161930 2024-03-06T07:31:06.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.995+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54dc03e830 con 0x7f54e4161930 2024-03-06T07:31:06.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.995+0000 7f54daa00640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54c406b4d0 0x7f54c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:06.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.996+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f54dc0acc10 con 0x7f54e4161930 2024-03-06T07:31:06.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.996+0000 7f54e2800640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54c406b4d0 0x7f54c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:06.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:06.997+0000 7f54e2800640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54c406b4d0 0x7f54c406d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f54d4000f60 tx=0x7f54d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.000+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54dc032e20 con 0x7f54e4161930 2024-03-06T07:31:07.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.210+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f54e404f330 con 0x7f54e4161930 2024-03-06T07:31:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.215+0000 7f54daa00640 1 -- 172.21.15.6:0/453603253 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f54dc07bea0 con 0x7f54e4161930 2024-03-06T07:31:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54c406b4d0 msgr2=0x7f54c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54c406b4d0 0x7f54c406d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f54d4000f60 tx=0x7f54d4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 msgr2=0x7f54e406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 0x7f54e406ba80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f54dc037040 tx=0x7f54dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 shutdown_connections 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f54c406b4d0 0x7f54c406d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54e40d2bd0 0x7f54e409dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54e4161930 0x7f54e406ba80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 --2- 172.21.15.6:0/453603253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54e40c8d20 0x7f54e415c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 >> 172.21.15.6:0/453603253 conn(0x7f54e40d3ee0 msgr2=0x7f54e40d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 shutdown_connections 2024-03-06T07:31:07.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.218+0000 7f54e9400640 1 -- 172.21.15.6:0/453603253 wait complete. 2024-03-06T07:31:07.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:31:07.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.455+0000 7fceb9000640 1 Processor -- start 2024-03-06T07:31:07.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.455+0000 7fceb9000640 1 -- start start 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb9000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fceb415e0b0 0x7fceb40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fceb40d4a10 0x7fceb40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb9000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb40b3710 con 0x7fceb40d0c80 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb9000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb40d10f0 con 0x7fceb40d4a10 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb9000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb40b09e0 con 0x7fceb415e0b0 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fceb40d4a10 0x7fceb40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fceb40d0c80 0x7fceb40d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36502/0 (socket says 172.21.15.6:36502) 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fceb40d4a10 0x7fceb40d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32820/0 (socket says 172.21.15.6:32820) 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.456+0000 7fceb3e00640 1 -- 172.21.15.6:0/755065290 learned_addr learned my addr 172.21.15.6:0/755065290 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.457+0000 7fceb3400640 1 --2- 172.21.15.6:0/755065290 >> v2:172.21.15.178:3300/0 conn(0x7fceb415e0b0 0x7fceb40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.457+0000 7fceb2a00640 1 -- 172.21.15.6:0/755065290 >> v2:172.21.15.178:3300/0 conn(0x7fceb415e0b0 msgr2=0x7fceb40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.457+0000 7fceb2a00640 1 --2- 172.21.15.6:0/755065290 >> v2:172.21.15.178:3300/0 conn(0x7fceb415e0b0 0x7fceb40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.457+0000 7fceb2a00640 1 -- 172.21.15.6:0/755065290 >> v1:172.21.15.6:6789/0 conn(0x7fceb40d0c80 legacy=0x7fceb40d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.457+0000 7fceb2a00640 1 -- 172.21.15.6:0/755065290 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb40b39f0 con 0x7fceb40d4a10 2024-03-06T07:31:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.457+0000 7fceb2000640 1 -- 172.21.15.6:0/755065290 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030884595 0 0) 0x7fceb40b3710 con 0x7fceb40d0c80 2024-03-06T07:31:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.458+0000 7fceb2a00640 1 --2- 172.21.15.6:0/755065290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb40d0740 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fceb40b68c0 tx=0x7fcea402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8eb66a9e9977603 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.458+0000 7fceb2000640 1 -- 172.21.15.6:0/755065290 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4039070 con 0x7fceb40d4a10 2024-03-06T07:31:07.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.458+0000 7fceb2000640 1 -- 172.21.15.6:0/755065290 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea402cc60 con 0x7fceb40d4a10 2024-03-06T07:31:07.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.458+0000 7fceb2000640 1 -- 172.21.15.6:0/755065290 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4034660 con 0x7fceb40d4a10 2024-03-06T07:31:07.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 -- 172.21.15.6:0/755065290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 msgr2=0x7fceb40d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 --2- 172.21.15.6:0/755065290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb40d0740 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fceb40b68c0 tx=0x7fcea402beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 -- 172.21.15.6:0/755065290 shutdown_connections 2024-03-06T07:31:07.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 --2- 172.21.15.6:0/755065290 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb40d0740 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 --2- 172.21.15.6:0/755065290 >> v2:172.21.15.178:3300/0 conn(0x7fceb415e0b0 0x7fceb40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 -- 172.21.15.6:0/755065290 >> 172.21.15.6:0/755065290 conn(0x7fceb40d3d30 msgr2=0x7fceb40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 -- 172.21.15.6:0/755065290 shutdown_connections 2024-03-06T07:31:07.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.459+0000 7fceb9000640 1 -- 172.21.15.6:0/755065290 wait complete. 2024-03-06T07:31:07.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.461+0000 7fceb9000640 1 Processor -- start 2024-03-06T07:31:07.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.461+0000 7fceb9000640 1 -- start start 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fceb40d0c80 0x7fceb415b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb415e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb9000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fceb415e0b0 0x7fceb415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb415e490 con 0x7fceb40d4a10 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb9000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb40b3fb0 con 0x7fceb415e0b0 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb9000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb40cfc60 con 0x7fceb40d0c80 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb415e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb415e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32832/0 (socket says 172.21.15.6:32832) 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.462+0000 7fceb2a00640 1 -- 172.21.15.6:0/322684530 learned_addr learned my addr 172.21.15.6:0/322684530 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb3400640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fceb40d0c80 0x7fceb415b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb3e00640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fceb415e0b0 0x7fceb415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb2a00640 1 -- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fceb40d0c80 msgr2=0x7fceb415b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb2a00640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fceb40d0c80 0x7fceb415b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb2a00640 1 -- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fceb415e0b0 msgr2=0x7fceb415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb2a00640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fceb415e0b0 0x7fceb415c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb2a00640 1 -- 172.21.15.6:0/322684530 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb415f090 con 0x7fceb40d4a10 2024-03-06T07:31:07.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceb2a00640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb415e610 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009d90 tx=0x7fcea4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.463+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4039050 con 0x7fceb40d4a10 2024-03-06T07:31:07.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.464+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4045070 con 0x7fceb40d4a10 2024-03-06T07:31:07.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.464+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea403e850 con 0x7fceb40d4a10 2024-03-06T07:31:07.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.464+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb415f370 con 0x7fceb40d4a10 2024-03-06T07:31:07.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.464+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb415fcb0 con 0x7fceb40d4a10 2024-03-06T07:31:07.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.465+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb404f330 con 0x7fceb40d4a10 2024-03-06T07:31:07.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.466+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcea403e9f0 con 0x7fceb40d4a10 2024-03-06T07:31:07.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.467+0000 7fceab400640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fce9006b4d0 0x7fce9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.467+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcea40acf70 con 0x7fceb40d4a10 2024-03-06T07:31:07.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.468+0000 7fceb3400640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fce9006b4d0 0x7fce9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.469+0000 7fceb3400640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fce9006b4d0 0x7fce9006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fce9c006fd0 tx=0x7fce9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.472+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcea407c170 con 0x7fceb40d4a10 2024-03-06T07:31:07.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.746+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7fceb40b6300 con 0x7fceb40d4a10 2024-03-06T07:31:07.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.805+0000 7fceab400640 1 -- 172.21.15.6:0/322684530 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v214) v1 ==== 144+0+0 (secure 0 0 0) 0x7fcea407ba70 con 0x7fceb40d4a10 2024-03-06T07:31:07.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fce9006b4d0 msgr2=0x7fce9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fce9006b4d0 0x7fce9006d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fce9c006fd0 tx=0x7fce9c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:07.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 msgr2=0x7fceb415e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb415e610 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009d90 tx=0x7fcea4004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 shutdown_connections 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fce9006b4d0 0x7fce9006d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fceb415e0b0 0x7fceb415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fceb40d4a10 0x7fceb415e610 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 --2- 172.21.15.6:0/322684530 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fceb40d0c80 0x7fceb415b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 >> 172.21.15.6:0/322684530 conn(0x7fceb40d3d30 msgr2=0x7fceb40b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 shutdown_connections 2024-03-06T07:31:07.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:07.808+0000 7fceb9000640 1 -- 172.21.15.6:0/322684530 wait complete. 2024-03-06T07:31:07.832 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:31:07.832 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:31:07.833 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:31:07.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.053+0000 7f800d200640 1 Processor -- start 2024-03-06T07:31:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.054+0000 7f800d200640 1 -- start start 2024-03-06T07:31:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.055+0000 7f800d200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f80080d4a10 0x7f80080d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.055+0000 7f800d200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080b3710 con 0x7f800815e0b0 2024-03-06T07:31:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.055+0000 7f800d200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080b39f0 con 0x7f80080d0c80 2024-03-06T07:31:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.055+0000 7f800d200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080d10f0 con 0x7f80080d4a10 2024-03-06T07:31:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.055+0000 7f8007400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f800815e0b0 0x7f80080cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36510/0 (socket says 172.21.15.6:36510) 2024-03-06T07:31:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.055+0000 7f8007400640 1 -- 172.21.15.6:0/2494271331 learned_addr learned my addr 172.21.15.6:0/2494271331 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:08.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006a00640 1 --2- 172.21.15.6:0/2494271331 >> v2:172.21.15.178:3300/0 conn(0x7f80080d4a10 0x7f80080d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141558224 0 0) 0x7f80080b3710 con 0x7f800815e0b0 2024-03-06T07:31:08.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8003520 con 0x7f800815e0b0 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340744139 0 0) 0x7f7fe8003520 con 0x7f800815e0b0 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 >> v2:172.21.15.178:3300/0 conn(0x7f80080d4a10 msgr2=0x7f80080d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 --2- 172.21.15.6:0/2494271331 >> v2:172.21.15.178:3300/0 conn(0x7f80080d4a10 0x7f80080d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 >> v1:172.21.15.178:6789/0 conn(0x7f80080d0c80 legacy=0x7f80080b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080b3cd0 con 0x7f800815e0b0 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7ff8001690 con 0x7f800815e0b0 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 >> v1:172.21.15.6:6789/0 conn(0x7f800815e0b0 legacy=0x7f80080cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.056+0000 7f8006000640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fe80039b0 0x7f7fe8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8006000640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe8004800 0x7f7fe8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8006000640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fe80051c0 0x7f7fe800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080d10f0 con 0x7f7fe80051c0 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe800c060 con 0x7f7fe8004800 2024-03-06T07:31:08.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe800c260 con 0x7f7fe80039b0 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8007e00640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fe80051c0 0x7f7fe800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8006a00640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe8004800 0x7f7fe8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8007e00640 1 -- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fe80039b0 msgr2=0x7f7fe8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8007e00640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fe80039b0 0x7f7fe8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8007e00640 1 -- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe8004800 msgr2=0x7f7fe8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8007e00640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe8004800 0x7f7fe8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.057+0000 7f8007e00640 1 -- 172.21.15.6:0/2494271331 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8003520 con 0x7f7fe80051c0 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.058+0000 7f8007e00640 1 -- 172.21.15.6:0/2494271331 reap_dead start 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.058+0000 7f8007e00640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fe80051c0 0x7f7fe800b9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8000001e50 tx=0x7f7fe8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41940bbfb70b6dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000001a70 con 0x7f7fe80051c0 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8000003bb0 con 0x7f7fe80051c0 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f8006000640 1 -- 172.21.15.6:0/2494271331 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80000050d0 con 0x7f7fe80051c0 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 -- 172.21.15.6:0/2494271331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fe80051c0 msgr2=0x7f7fe800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fe80051c0 0x7f7fe800b9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8000001e50 tx=0x7f7fe8005830 comp rx=0 tx=0).stop 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 -- 172.21.15.6:0/2494271331 shutdown_connections 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 --2- 172.21.15.6:0/2494271331 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fe80051c0 0x7f7fe800b9c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 -- 172.21.15.6:0/2494271331 >> 172.21.15.6:0/2494271331 conn(0x7f80080d3d30 msgr2=0x7f80080d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 -- 172.21.15.6:0/2494271331 shutdown_connections 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.059+0000 7f800d200640 1 -- 172.21.15.6:0/2494271331 wait complete. 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.061+0000 7f800d200640 1 Processor -- start 2024-03-06T07:31:08.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.061+0000 7f800d200640 1 -- start start 2024-03-06T07:31:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f800d200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe80051c0 0x7f80080ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f800d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 0x7f80080c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f800d200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80080a1120 0x7f80080a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f800d200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800815e490 con 0x7f80080c7690 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f800d200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080cfc60 con 0x7f7fe80051c0 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f800d200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080d0740 con 0x7f80080a1120 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.062+0000 7f8006a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 0x7f80080c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 0x7f80080c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32850/0 (socket says 172.21.15.6:32850) 2024-03-06T07:31:08.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 -- 172.21.15.6:0/3462611689 learned_addr learned my addr 172.21.15.6:0/3462611689 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8007400640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe80051c0 0x7f80080ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 -- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80080a1120 msgr2=0x7f80080a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80080a1120 0x7f80080a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 -- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe80051c0 msgr2=0x7f80080ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe80051c0 0x7f80080ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 -- 172.21.15.6:0/3462611689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080c8270 con 0x7f80080c7690 2024-03-06T07:31:08.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8007400640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe80051c0 0x7f80080ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.063+0000 7f8006a00640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 0x7f80080c7b10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0011590 tx=0x7f7ff0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.064+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0004220 con 0x7f80080c7690 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.064+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff00043c0 con 0x7f80080c7690 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.064+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0015930 con 0x7f80080c7690 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.064+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80080c8550 con 0x7f80080c7690 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.064+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80080c8b10 con 0x7f80080c7690 2024-03-06T07:31:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.065+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800804f330 con 0x7f80080c7690 2024-03-06T07:31:08.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.066+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ff0015ad0 con 0x7f80080c7690 2024-03-06T07:31:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.067+0000 7f7fff400640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fe406b4d0 0x7f7fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.067+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7ff008a4b0 con 0x7f80080c7690 2024-03-06T07:31:08.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.067+0000 7f8007400640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fe406b4d0 0x7f7fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.068+0000 7f8007400640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fe406b4d0 0x7f7fe406d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7ff80011e0 tx=0x7f7ff8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.071+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ff0019030 con 0x7f80080c7690 2024-03-06T07:31:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.274+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80080b2be0 con 0x7f80080c7690 2024-03-06T07:31:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.276+0000 7f7fff400640 1 -- 172.21.15.6:0/3462611689 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+12946 (secure 0 0 0) 0x7f7ff0059620 con 0x7f80080c7690 2024-03-06T07:31:08.278 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:08.278 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":61,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:05.786815+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/1578886876":"2024-03-07T07:31:02.591762+0000","172.21.15.6:6834/1578886876":"2024-03-07T07:31:02.591762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fe406b4d0 msgr2=0x7f7fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fe406b4d0 0x7f7fe406d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7ff80011e0 tx=0x7f7ff8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 msgr2=0x7f80080c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 0x7f80080c7b10 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0011590 tx=0x7f7ff0011a50 comp rx=0 tx=0).stop 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 shutdown_connections 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fe406b4d0 0x7f7fe406d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80080a1120 0x7f80080a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f80080c7690 0x7f80080c7b10 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 --2- 172.21.15.6:0/3462611689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fe80051c0 0x7f80080ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.278+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 >> 172.21.15.6:0/3462611689 conn(0x7f80080d3d30 msgr2=0x7f80080b24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.279+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 shutdown_connections 2024-03-06T07:31:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.279+0000 7f800d200640 1 -- 172.21.15.6:0/3462611689 wait complete. 2024-03-06T07:31:08.301 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:08.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:08.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.527+0000 7f2815200640 1 Processor -- start 2024-03-06T07:31:08.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.527+0000 7f2815200640 1 -- start start 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.528+0000 7f2815200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f281015dee0 0x7f28100d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.528+0000 7f2815200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28100d0d40 0x7f28100b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.528+0000 7f2815200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28100ceca0 con 0x7f28100d4860 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.528+0000 7f2815200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28100cf5d0 con 0x7f28100d0d40 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.528+0000 7f2815200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28100cef80 con 0x7f281015dee0 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28100d0d40 0x7f28100b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f28100d4860 0x7f28100d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36526/0 (socket says 172.21.15.6:36526) 2024-03-06T07:31:08.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280ea00640 1 -- 172.21.15.6:0/3208335076 learned_addr learned my addr 172.21.15.6:0/3208335076 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:08.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280f400640 1 --2- 172.21.15.6:0/3208335076 >> v2:172.21.15.178:3300/0 conn(0x7f281015dee0 0x7f28100d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280fe00640 1 -- 172.21.15.6:0/3208335076 >> v2:172.21.15.178:3300/0 conn(0x7f281015dee0 msgr2=0x7f28100d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280fe00640 1 --2- 172.21.15.6:0/3208335076 >> v2:172.21.15.178:3300/0 conn(0x7f281015dee0 0x7f28100d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280fe00640 1 -- 172.21.15.6:0/3208335076 >> v1:172.21.15.6:6789/0 conn(0x7f28100d4860 legacy=0x7f28100d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280e000640 1 -- 172.21.15.6:0/3208335076 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973271018 0 0) 0x7f28100ceca0 con 0x7f28100d4860 2024-03-06T07:31:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.529+0000 7f280fe00640 1 -- 172.21.15.6:0/3208335076 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28100cf260 con 0x7f28100d0d40 2024-03-06T07:31:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.531+0000 7f280fe00640 1 --2- 172.21.15.6:0/3208335076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28100b3510 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f281008c6f0 tx=0x7f280802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551c793bac461d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.531+0000 7f280e000640 1 -- 172.21.15.6:0/3208335076 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808035020 con 0x7f28100d0d40 2024-03-06T07:31:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.531+0000 7f280e000640 1 -- 172.21.15.6:0/3208335076 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808031830 con 0x7f28100d0d40 2024-03-06T07:31:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.531+0000 7f280e000640 1 -- 172.21.15.6:0/3208335076 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808031c00 con 0x7f28100d0d40 2024-03-06T07:31:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 -- 172.21.15.6:0/3208335076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 msgr2=0x7f28100b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 --2- 172.21.15.6:0/3208335076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28100b3510 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f281008c6f0 tx=0x7f280802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 -- 172.21.15.6:0/3208335076 shutdown_connections 2024-03-06T07:31:08.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 --2- 172.21.15.6:0/3208335076 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28100b3510 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 --2- 172.21.15.6:0/3208335076 >> v2:172.21.15.178:3300/0 conn(0x7f281015dee0 0x7f28100d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 -- 172.21.15.6:0/3208335076 >> 172.21.15.6:0/3208335076 conn(0x7f28100d3cd0 msgr2=0x7f28100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 -- 172.21.15.6:0/3208335076 shutdown_connections 2024-03-06T07:31:08.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.532+0000 7f2815200640 1 -- 172.21.15.6:0/3208335076 wait complete. 2024-03-06T07:31:08.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.534+0000 7f2815200640 1 Processor -- start 2024-03-06T07:31:08.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.535+0000 7f2815200640 1 -- start start 2024-03-06T07:31:08.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f2815200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28101510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f2815200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28100d4860 0x7f28100ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f2815200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f281015dee0 0x7f28100b9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f2815200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100d1de0 con 0x7f28100d0d40 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f2815200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281015e2c0 con 0x7f281015dee0 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f2815200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100d1f60 con 0x7f28100d4860 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f280f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28101510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f280f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28101510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32874/0 (socket says 172.21.15.6:32874) 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.536+0000 7f280f400640 1 -- 172.21.15.6:0/2451510259 learned_addr learned my addr 172.21.15.6:0/2451510259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f280f400640 1 -- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28100d4860 msgr2=0x7f28100ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f280f400640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28100d4860 0x7f28100ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f280f400640 1 -- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f281015dee0 msgr2=0x7f28100b9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f280f400640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f281015dee0 0x7f28100b9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f280f400640 1 -- 172.21.15.6:0/2451510259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810162010 con 0x7f28100d0d40 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f280f400640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28101510b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2800007f20 tx=0x7f280000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2800014620 con 0x7f28100d0d40 2024-03-06T07:31:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2800019070 con 0x7f28100d0d40 2024-03-06T07:31:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2800014a00 con 0x7f28100d0d40 2024-03-06T07:31:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281015e670 con 0x7f28100d0d40 2024-03-06T07:31:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.537+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281015ec30 con 0x7f28100d0d40 2024-03-06T07:31:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.539+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281004f330 con 0x7f28100d0d40 2024-03-06T07:31:08.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.540+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2800014ba0 con 0x7f28100d0d40 2024-03-06T07:31:08.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.541+0000 7f2807400640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27ec06b4d0 0x7f27ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.541+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f280008a790 con 0x7f28100d0d40 2024-03-06T07:31:08.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.541+0000 7f280ea00640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27ec06b4d0 0x7f27ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.542+0000 7f280ea00640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27ec06b4d0 0x7f27ec06d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f27f8007fd0 tx=0x7f27f8007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.545+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28000599a0 con 0x7f28100d0d40 2024-03-06T07:31:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.745+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f281006d6d0 con 0x7f28100d0d40 2024-03-06T07:31:08.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.746+0000 7f2807400640 1 -- 172.21.15.6:0/2451510259 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28000592a0 con 0x7f28100d0d40 2024-03-06T07:31:08.748 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:31:08.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27ec06b4d0 msgr2=0x7f27ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27ec06b4d0 0x7f27ec06d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f27f8007fd0 tx=0x7f27f8007920 comp rx=0 tx=0).stop 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 msgr2=0x7f28101510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28101510b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2800007f20 tx=0x7f280000e570 comp rx=0 tx=0).stop 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 shutdown_connections 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27ec06b4d0 0x7f27ec06d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f281015dee0 0x7f28100b9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28100d4860 0x7f28100ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 --2- 172.21.15.6:0/2451510259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28100d0d40 0x7f28101510b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 >> 172.21.15.6:0/2451510259 conn(0x7f28100d3cd0 msgr2=0x7f28100b7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 shutdown_connections 2024-03-06T07:31:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.749+0000 7f2815200640 1 -- 172.21.15.6:0/2451510259 wait complete. 2024-03-06T07:31:08.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:31:08.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.986+0000 7fcf47e00640 1 Processor -- start 2024-03-06T07:31:08.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.987+0000 7fcf47e00640 1 -- start start 2024-03-06T07:31:08.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.988+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf4815e0b0 0x7fcf480cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.988+0000 7fcf47e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcf480d4a10 0x7fcf480d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:08.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.988+0000 7fcf47e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf480b3710 con 0x7fcf480d0c80 2024-03-06T07:31:08.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.988+0000 7fcf47e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf480b09e0 con 0x7fcf480d4a10 2024-03-06T07:31:08.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.988+0000 7fcf47e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480b39f0 con 0x7fcf4815e0b0 2024-03-06T07:31:08.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.989+0000 7fcf46000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcf480d4a10 0x7fcf480d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.989+0000 7fcf46000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcf480d4a10 0x7fcf480d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:49524/0 (socket says 172.21.15.6:49524) 2024-03-06T07:31:08.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.989+0000 7fcf46000640 1 -- 172.21.15.6:0/2594595113 learned_addr learned my addr 172.21.15.6:0/2594595113 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:08.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.989+0000 7fcf46a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf4815e0b0 0x7fcf480cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:08.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.990+0000 7fcf46a00640 1 -- 172.21.15.6:0/2594595113 >> v2:172.21.15.178:3300/0 conn(0x7fcf480d4a10 msgr2=0x7fcf480d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.990+0000 7fcf46a00640 1 --2- 172.21.15.6:0/2594595113 >> v2:172.21.15.178:3300/0 conn(0x7fcf480d4a10 0x7fcf480d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.990+0000 7fcf46a00640 1 -- 172.21.15.6:0/2594595113 >> v1:172.21.15.178:6789/0 conn(0x7fcf480d0c80 legacy=0x7fcf480d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.990+0000 7fcf46a00640 1 -- 172.21.15.6:0/2594595113 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf480b3cd0 con 0x7fcf4815e0b0 2024-03-06T07:31:08.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.992+0000 7fcf46a00640 1 --2- 172.21.15.6:0/2594595113 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf4815e0b0 0x7fcf480cf310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf380097a0 tx=0x7fcf3802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08614b0635bb2e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:08.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.992+0000 7fcf45600640 1 -- 172.21.15.6:0/2594595113 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38035020 con 0x7fcf4815e0b0 2024-03-06T07:31:08.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.992+0000 7fcf45600640 1 -- 172.21.15.6:0/2594595113 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38031830 con 0x7fcf4815e0b0 2024-03-06T07:31:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.992+0000 7fcf45600640 1 -- 172.21.15.6:0/2594595113 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38031c00 con 0x7fcf4815e0b0 2024-03-06T07:31:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.993+0000 7fcf47e00640 1 -- 172.21.15.6:0/2594595113 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf4815e0b0 msgr2=0x7fcf480cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:08.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.993+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2594595113 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf4815e0b0 0x7fcf480cf310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf380097a0 tx=0x7fcf3802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.993+0000 7fcf47e00640 1 -- 172.21.15.6:0/2594595113 shutdown_connections 2024-03-06T07:31:08.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.993+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2594595113 >> v2:172.21.15.178:3300/0 conn(0x7fcf480d4a10 0x7fcf480d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.993+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2594595113 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf4815e0b0 0x7fcf480cf310 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:08.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.993+0000 7fcf47e00640 1 -- 172.21.15.6:0/2594595113 >> 172.21.15.6:0/2594595113 conn(0x7fcf480d3d30 msgr2=0x7fcf480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:08.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.994+0000 7fcf47e00640 1 -- 172.21.15.6:0/2594595113 shutdown_connections 2024-03-06T07:31:08.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.994+0000 7fcf47e00640 1 -- 172.21.15.6:0/2594595113 wait complete. 2024-03-06T07:31:08.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.995+0000 7fcf47e00640 1 Processor -- start 2024-03-06T07:31:08.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.995+0000 7fcf47e00640 1 -- start start 2024-03-06T07:31:08.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.996+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 0x7fcf480b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.996+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf480d4a10 0x7fcf480d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.996+0000 7fcf47e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf4815e0b0 0x7fcf48159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.996+0000 7fcf47e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480cfc60 con 0x7fcf480d0c80 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.996+0000 7fcf47e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4815e490 con 0x7fcf4815e0b0 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.996+0000 7fcf47e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf480b54e0 con 0x7fcf480d4a10 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 0x7fcf480b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 0x7fcf480b8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32882/0 (socket says 172.21.15.6:32882) 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 -- 172.21.15.6:0/2174037508 learned_addr learned my addr 172.21.15.6:0/2174037508 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf47400640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf4815e0b0 0x7fcf48159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 -- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf480d4a10 msgr2=0x7fcf480d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46000640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf480d4a10 0x7fcf480d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf480d4a10 0x7fcf480d23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 -- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf4815e0b0 msgr2=0x7fcf48159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf4815e0b0 0x7fcf48159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46a00640 1 -- 172.21.15.6:0/2174037508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4815f2b0 con 0x7fcf480d0c80 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf47400640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf4815e0b0 0x7fcf48159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.997+0000 7fcf46000640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf480d4a10 0x7fcf480d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:09.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.998+0000 7fcf46a00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 0x7fcf480b8cd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcf3802b910 tx=0x7fcf380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.998+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38040070 con 0x7fcf480d0c80 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.998+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3803a930 con 0x7fcf480d0c80 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.998+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38035040 con 0x7fcf480d0c80 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.998+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf480b9540 con 0x7fcf480d0c80 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.998+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4815f520 con 0x7fcf480d0c80 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:08.999+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4804f330 con 0x7fcf480d0c80 2024-03-06T07:31:09.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.000+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf3804b060 con 0x7fcf480d0c80 2024-03-06T07:31:09.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.001+0000 7fcf3ea00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf2806b4d0 0x7fcf2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.001+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcf380ab740 con 0x7fcf480d0c80 2024-03-06T07:31:09.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.002+0000 7fcf46000640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf2806b4d0 0x7fcf2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.003+0000 7fcf46000640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf2806b4d0 0x7fcf2806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcf481f74c0 tx=0x7fcf34008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.006+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf3807a8c0 con 0x7fcf480d0c80 2024-03-06T07:31:09.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.271+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fcf480b2540 con 0x7fcf480d0c80 2024-03-06T07:31:09.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.819+0000 7fcf3ea00640 1 -- 172.21.15.6:0/2174037508 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v216) v1 ==== 143+0+0 (secure 0 0 0) 0x7fcf38031800 con 0x7fcf480d0c80 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.822+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf2806b4d0 msgr2=0x7fcf2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.822+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf2806b4d0 0x7fcf2806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fcf481f74c0 tx=0x7fcf34008040 comp rx=0 tx=0).stop 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 msgr2=0x7fcf480b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 0x7fcf480b8cd0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fcf3802b910 tx=0x7fcf380046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 shutdown_connections 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcf2806b4d0 0x7fcf2806d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf4815e0b0 0x7fcf48159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcf480d4a10 0x7fcf480d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 --2- 172.21.15.6:0/2174037508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcf480d0c80 0x7fcf480b8cd0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 >> 172.21.15.6:0/2174037508 conn(0x7fcf480d3d30 msgr2=0x7fcf480d18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 shutdown_connections 2024-03-06T07:31:09.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:09.823+0000 7fcf47e00640 1 -- 172.21.15.6:0/2174037508 wait complete. 2024-03-06T07:31:09.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:31:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.073+0000 7f3084e00640 1 Processor -- start 2024-03-06T07:31:10.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.074+0000 7f3084e00640 1 -- start start 2024-03-06T07:31:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.075+0000 7f3084e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3080072520 0x7f30800cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.075+0000 7f3084e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3080071e20 0x7f30800ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.075+0000 7f3084e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3080072df0 con 0x7f308015bf20 2024-03-06T07:31:10.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.075+0000 7f3084e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30800cdab0 con 0x7f3080071e20 2024-03-06T07:31:10.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.075+0000 7f3084e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30800b4d70 con 0x7f3080072520 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.076+0000 7f307e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3080071e20 0x7f30800ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.076+0000 7f307e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3080071e20 0x7f30800ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32896/0 (socket says 172.21.15.6:32896) 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.076+0000 7f307e800640 1 -- 172.21.15.6:0/3347851138 learned_addr learned my addr 172.21.15.6:0/3347851138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.076+0000 7f307d400640 1 --2- 172.21.15.6:0/3347851138 >> v2:172.21.15.178:3300/0 conn(0x7f3080072520 0x7f30800cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.077+0000 7f307e800640 1 -- 172.21.15.6:0/3347851138 >> v2:172.21.15.178:3300/0 conn(0x7f3080072520 msgr2=0x7f30800cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.077+0000 7f307e800640 1 --2- 172.21.15.6:0/3347851138 >> v2:172.21.15.178:3300/0 conn(0x7f3080072520 0x7f30800cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.077+0000 7f307e800640 1 -- 172.21.15.6:0/3347851138 >> v1:172.21.15.178:6789/0 conn(0x7f308015bf20 legacy=0x7f30800acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.077+0000 7f307e800640 1 -- 172.21.15.6:0/3347851138 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30800b5050 con 0x7f3080071e20 2024-03-06T07:31:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.078+0000 7f307e800640 1 --2- 172.21.15.6:0/3347851138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3080071e20 0x7f30800ce4f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3080070b90 tx=0x7f307802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb592d5aca0b468 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.078+0000 7f307ca00640 1 -- 172.21.15.6:0/3347851138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307802cb10 con 0x7f3080071e20 2024-03-06T07:31:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.078+0000 7f307ca00640 1 -- 172.21.15.6:0/3347851138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307802ccb0 con 0x7f3080071e20 2024-03-06T07:31:10.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.078+0000 7f307ca00640 1 -- 172.21.15.6:0/3347851138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307802ce60 con 0x7f3080071e20 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 -- 172.21.15.6:0/3347851138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3080071e20 msgr2=0x7f30800ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 --2- 172.21.15.6:0/3347851138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3080071e20 0x7f30800ce4f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3080070b90 tx=0x7f307802bf70 comp rx=0 tx=0).stop 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 -- 172.21.15.6:0/3347851138 shutdown_connections 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 --2- 172.21.15.6:0/3347851138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3080071e20 0x7f30800ce4f0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 --2- 172.21.15.6:0/3347851138 >> v2:172.21.15.178:3300/0 conn(0x7f3080072520 0x7f30800cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 -- 172.21.15.6:0/3347851138 >> 172.21.15.6:0/3347851138 conn(0x7f308006b3d0 msgr2=0x7f308006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 -- 172.21.15.6:0/3347851138 shutdown_connections 2024-03-06T07:31:10.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.079+0000 7f3084e00640 1 -- 172.21.15.6:0/3347851138 wait complete. 2024-03-06T07:31:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.080+0000 7f3084e00640 1 Processor -- start 2024-03-06T07:31:10.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.081+0000 7f3084e00640 1 -- start start 2024-03-06T07:31:10.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.081+0000 7f3084e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3080071e20 0x7f308009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f3084e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3080072520 0x7f308006dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f3084e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 0x7f3080156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f3084e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30800ad450 con 0x7f308015bf20 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f3084e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30800b4990 con 0x7f3080072520 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f3084e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3080144010 con 0x7f3080071e20 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 0x7f3080156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 0x7f3080156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32906/0 (socket says 172.21.15.6:32906) 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 -- 172.21.15.6:0/501320239 learned_addr learned my addr 172.21.15.6:0/501320239 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307de00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3080071e20 0x7f308009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 -- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3080071e20 msgr2=0x7f308009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3080071e20 0x7f308009aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307d400640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3080072520 0x7f308006dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 -- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3080072520 msgr2=0x7f308006dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3080072520 0x7f308006dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.082+0000 7f307e800640 1 -- 172.21.15.6:0/501320239 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308009c820 con 0x7f308015bf20 2024-03-06T07:31:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.083+0000 7f307e800640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 0x7f3080156fa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3078036040 tx=0x7f3078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.083+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078043020 con 0x7f308015bf20 2024-03-06T07:31:10.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.083+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307803e450 con 0x7f308015bf20 2024-03-06T07:31:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.083+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307802ce60 con 0x7f308015bf20 2024-03-06T07:31:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.083+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308009cb00 con 0x7f308015bf20 2024-03-06T07:31:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.083+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308009b0a0 con 0x7f308015bf20 2024-03-06T07:31:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.084+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308004f330 con 0x7f308015bf20 2024-03-06T07:31:10.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.085+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3078046020 con 0x7f308015bf20 2024-03-06T07:31:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.086+0000 7f3076000640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f305c06b500 0x7f305c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.086+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f30780ac530 con 0x7f308015bf20 2024-03-06T07:31:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.087+0000 7f307de00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f305c06b500 0x7f305c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.088+0000 7f307de00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f305c06b500 0x7f305c06d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3070000f60 tx=0x7f3070009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.091+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f307804d030 con 0x7f308015bf20 2024-03-06T07:31:10.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.292+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f308014abf0 con 0x7f308015bf20 2024-03-06T07:31:10.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.299+0000 7f3076000640 1 -- 172.21.15.6:0/501320239 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f307807b6a0 con 0x7f308015bf20 2024-03-06T07:31:10.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f305c06b500 msgr2=0x7f305c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f305c06b500 0x7f305c06d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3070000f60 tx=0x7f3070009040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 msgr2=0x7f3080156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 0x7f3080156fa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3078036040 tx=0x7f3078004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 shutdown_connections 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f305c06b500 0x7f305c06d9c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3080072520 0x7f308006dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f308015bf20 0x7f3080156fa0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 --2- 172.21.15.6:0/501320239 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3080071e20 0x7f308009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 >> 172.21.15.6:0/501320239 conn(0x7f308006b3d0 msgr2=0x7f30800716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 shutdown_connections 2024-03-06T07:31:10.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:10.302+0000 7f3084e00640 1 -- 172.21.15.6:0/501320239 wait complete. 2024-03-06T07:31:10.324 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:10.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:10.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:10.544 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:10.544 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:10.545 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:10.602 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:10.602 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:31:10.602 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:10.657 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:10.657 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:10.658 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:10.717 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:10.717 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:31:10.717 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:10.773 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:10.773 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:31:10.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:11.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.006+0000 7fd19de00640 1 Processor -- start 2024-03-06T07:31:11.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.006+0000 7fd19de00640 1 -- start start 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.007+0000 7fd19de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1980cd060 0x7fd198161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.007+0000 7fd19de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd198070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.007+0000 7fd19de00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1981627f0 con 0x7fd19815efc0 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.007+0000 7fd19de00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19815f3a0 con 0x7fd1980cd060 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.007+0000 7fd19de00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19806d190 con 0x7fd198162000 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1980cd060 0x7fd198161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1980cd060 0x7fd198161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32918/0 (socket says 172.21.15.6:32918) 2024-03-06T07:31:11.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 -- 172.21.15.6:0/3476795074 learned_addr learned my addr 172.21.15.6:0/3476795074 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:11.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd19d400640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd198070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 -- 172.21.15.6:0/3476795074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 msgr2=0x7fd198070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd198070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 -- 172.21.15.6:0/3476795074 >> v1:172.21.15.178:6789/0 conn(0x7fd19815efc0 legacy=0x7fd1980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd197e00640 1 -- 172.21.15.6:0/3476795074 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19806c860 con 0x7fd1980cd060 2024-03-06T07:31:11.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.008+0000 7fd19d400640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd198070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:11.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.010+0000 7fd197e00640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198161ac0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd180009e50 tx=0x7fd18002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcf4e6ca70f8953 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.010+0000 7fd197400640 1 -- 172.21.15.6:0/3476795074 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18002ca70 con 0x7fd1980cd060 2024-03-06T07:31:11.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.010+0000 7fd197400640 1 -- 172.21.15.6:0/3476795074 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18002cc10 con 0x7fd1980cd060 2024-03-06T07:31:11.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.010+0000 7fd197400640 1 -- 172.21.15.6:0/3476795074 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800343f0 con 0x7fd1980cd060 2024-03-06T07:31:11.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 -- 172.21.15.6:0/3476795074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 msgr2=0x7fd198161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198161ac0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fd180009e50 tx=0x7fd18002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 -- 172.21.15.6:0/3476795074 shutdown_connections 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd198070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 --2- 172.21.15.6:0/3476795074 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198161ac0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 -- 172.21.15.6:0/3476795074 >> 172.21.15.6:0/3476795074 conn(0x7fd19806b3d0 msgr2=0x7fd19806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 -- 172.21.15.6:0/3476795074 shutdown_connections 2024-03-06T07:31:11.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.011+0000 7fd19de00640 1 -- 172.21.15.6:0/3476795074 wait complete. 2024-03-06T07:31:11.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.013+0000 7fd19de00640 1 Processor -- start 2024-03-06T07:31:11.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.013+0000 7fd19de00640 1 -- start start 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19de00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd19815efc0 0x7fd1981413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd1981e8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19de00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1981624b0 con 0x7fd1980cd060 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19de00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19806ce20 con 0x7fd19815efc0 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19de00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19806d620 con 0x7fd198162000 2024-03-06T07:31:11.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.014+0000 7fd19ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32926/0 (socket says 172.21.15.6:32926) 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 -- 172.21.15.6:0/3783425011 learned_addr learned my addr 172.21.15.6:0/3783425011 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 -- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 msgr2=0x7fd1981e8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd1981e8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 -- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd19815efc0 msgr2=0x7fd1981413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd197e00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd19815efc0 0x7fd1981413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd19815efc0 0x7fd1981413a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 -- 172.21.15.6:0/3783425011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1981e7a50 con 0x7fd1980cd060 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.015+0000 7fd19ca00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198144710 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c430 tx=0x7fd18800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.016+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188011790 con 0x7fd1980cd060 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.016+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd188017070 con 0x7fd1980cd060 2024-03-06T07:31:11.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.016+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188011bb0 con 0x7fd1980cd060 2024-03-06T07:31:11.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.016+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd198143670 con 0x7fd1980cd060 2024-03-06T07:31:11.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.016+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd198143c30 con 0x7fd1980cd060 2024-03-06T07:31:11.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.017+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19804f330 con 0x7fd1980cd060 2024-03-06T07:31:11.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.018+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd18801c460 con 0x7fd1980cd060 2024-03-06T07:31:11.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.019+0000 7fd194c00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd17406b520 0x7fd17406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.019+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd188089da0 con 0x7fd1980cd060 2024-03-06T07:31:11.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.020+0000 7fd197e00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd17406b520 0x7fd17406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.021+0000 7fd197e00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd17406b520 0x7fd17406d9e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd180035000 tx=0x7fd180030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.024+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd188058f90 con 0x7fd1980cd060 2024-03-06T07:31:11.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.270+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1980cf490 con 0x7fd1980cd060 2024-03-06T07:31:11.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.271+0000 7fd194c00640 1 -- 172.21.15.6:0/3783425011 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fd188058890 con 0x7fd1980cd060 2024-03-06T07:31:11.273 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:11.274 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4058952040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4058952040},{"type":"v1","addr":"172.21.15.178:6836","nonce":4058952040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5009,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1857343709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1857343709},{"type":"v1","addr":"172.21.15.178:6837","nonce":1857343709}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/383450675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":383450675},{"type":"v1","addr":"172.21.15.6:6833","nonce":383450675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6309,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2662959432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2662959432},{"type":"v1","addr":"172.21.15.6:6835","nonce":2662959432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[]} 2024-03-06T07:31:11.274 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 38 2024-03-06T07:31:11.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd17406b520 msgr2=0x7fd17406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd17406b520 0x7fd17406d9e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd180035000 tx=0x7fd180030000 comp rx=0 tx=0).stop 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 msgr2=0x7fd198144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198144710 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c430 tx=0x7fd18800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 shutdown_connections 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd17406b520 0x7fd17406d9e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd19815efc0 0x7fd1981413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd198162000 0x7fd1981e8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 --2- 172.21.15.6:0/3783425011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1980cd060 0x7fd198144710 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 >> 172.21.15.6:0/3783425011 conn(0x7fd19806b3d0 msgr2=0x7fd198070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.274+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 shutdown_connections 2024-03-06T07:31:11.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.275+0000 7fd19de00640 1 -- 172.21.15.6:0/3783425011 wait complete. 2024-03-06T07:31:11.297 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:31:11.297 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:31:11.297 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:31:11.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:11.298 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:31:11.298 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:31:11.298 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:31:11.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:11.298 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:31:11.298 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:31:11.299 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:31:11.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:11.299 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:31:11.299 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:31:11.299 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:31:11.299 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:31:11.318 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:31:11.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:31:11.323 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:31:11.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:31:11.324 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:31:11.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:31:11.326 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:31:11.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:31:11.369 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:31:11.370 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:11.373 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:31:11.373 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:11.376 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:31:11.376 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:11.379 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:31:11.379 DEBUG:teuthology.parallel:result is None 2024-03-06T07:31:11.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:31:11.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:31:11.532 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:31:11.535 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:31:11.536 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:31:11.536 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:31:11.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.566+0000 7fb792800640 1 Processor -- start 2024-03-06T07:31:11.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.566+0000 7fb792800640 1 -- start start 2024-03-06T07:31:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.567+0000 7fb792800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb78c15e070 0x7fb78c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.567+0000 7fb792800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb78c0d0cd0 0x7fb78c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.567+0000 7fb792800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0d18c0 con 0x7fb78c0cf410 2024-03-06T07:31:11.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.567+0000 7fb792800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c0d1ba0 con 0x7fb78c0d0cd0 2024-03-06T07:31:11.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.567+0000 7fb792800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c0d1e80 con 0x7fb78c15e070 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb78c0d0cd0 0x7fb78c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb78c0d0cd0 0x7fb78c0cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32946/0 (socket says 172.21.15.6:32946) 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 -- 172.21.15.6:0/3938308737 learned_addr learned my addr 172.21.15.6:0/3938308737 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791400640 1 --2- 172.21.15.6:0/3938308737 >> v2:172.21.15.178:3300/0 conn(0x7fb78c15e070 0x7fb78c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 -- 172.21.15.6:0/3938308737 >> v2:172.21.15.178:3300/0 conn(0x7fb78c15e070 msgr2=0x7fb78c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 --2- 172.21.15.6:0/3938308737 >> v2:172.21.15.178:3300/0 conn(0x7fb78c15e070 0x7fb78c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 -- 172.21.15.6:0/3938308737 >> v1:172.21.15.178:6789/0 conn(0x7fb78c0cf410 legacy=0x7fb78c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791e00640 1 -- 172.21.15.6:0/3938308737 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0d2160 con 0x7fb78c0d0cd0 2024-03-06T07:31:11.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.568+0000 7fb791400640 1 --2- 172.21.15.6:0/3938308737 >> v2:172.21.15.178:3300/0 conn(0x7fb78c15e070 0x7fb78c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:11.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.569+0000 7fb791e00640 1 --2- 172.21.15.6:0/3938308737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0cf870 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb78c08d270 tx=0x7fb78402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43909dd288ba686 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb78be00640 1 -- 172.21.15.6:0/3938308737 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78402cb20 con 0x7fb78c0d0cd0 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb78be00640 1 -- 172.21.15.6:0/3938308737 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78402ccc0 con 0x7fb78c0d0cd0 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb78be00640 1 -- 172.21.15.6:0/3938308737 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784034610 con 0x7fb78c0d0cd0 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb792800640 1 -- 172.21.15.6:0/3938308737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c0d0cd0 msgr2=0x7fb78c0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb792800640 1 --2- 172.21.15.6:0/3938308737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0cf870 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb78c08d270 tx=0x7fb78402bf80 comp rx=0 tx=0).stop 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb792800640 1 -- 172.21.15.6:0/3938308737 shutdown_connections 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb792800640 1 --2- 172.21.15.6:0/3938308737 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0cf870 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb792800640 1 --2- 172.21.15.6:0/3938308737 >> v2:172.21.15.178:3300/0 conn(0x7fb78c15e070 0x7fb78c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.570+0000 7fb792800640 1 -- 172.21.15.6:0/3938308737 >> 172.21.15.6:0/3938308737 conn(0x7fb78c08d2a0 msgr2=0x7fb78c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.571+0000 7fb792800640 1 -- 172.21.15.6:0/3938308737 shutdown_connections 2024-03-06T07:31:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.571+0000 7fb792800640 1 -- 172.21.15.6:0/3938308737 wait complete. 2024-03-06T07:31:11.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.572+0000 7fb792800640 1 Processor -- start 2024-03-06T07:31:11.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.572+0000 7fb792800640 1 -- start start 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.573+0000 7fb792800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78c0cf410 0x7fb78c1f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.573+0000 7fb792800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.573+0000 7fb792800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 0x7fb78c15c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.573+0000 7fb792800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0d4730 con 0x7fb78c15e070 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.573+0000 7fb792800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0d45b0 con 0x7fb78c0d0cd0 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.573+0000 7fb792800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0b3630 con 0x7fb78c0cf410 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 0x7fb78c15c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 0x7fb78c15c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32962/0 (socket says 172.21.15.6:32962) 2024-03-06T07:31:11.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 -- 172.21.15.6:0/74548224 learned_addr learned my addr 172.21.15.6:0/74548224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791400640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78c0cf410 0x7fb78c1f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb790a00640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 -- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78c0cf410 msgr2=0x7fb78c1f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78c0cf410 0x7fb78c1f7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 -- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78c0d0cd0 msgr2=0x7fb78c0c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 -- 172.21.15.6:0/74548224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0c80b0 con 0x7fb78c15e070 2024-03-06T07:31:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791400640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78c0cf410 0x7fb78c1f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.574+0000 7fb791e00640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 0x7fb78c15c860 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb784009f40 tx=0x7fb784004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.575+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784043020 con 0x7fb78c15e070 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.575+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78403e450 con 0x7fb78c15e070 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.575+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784009510 con 0x7fb78c15e070 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.575+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c15f980 con 0x7fb78c15e070 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.575+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c15ff40 con 0x7fb78c15e070 2024-03-06T07:31:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.576+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c04f330 con 0x7fb78c15e070 2024-03-06T07:31:11.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.577+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb784005d10 con 0x7fb78c15e070 2024-03-06T07:31:11.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.578+0000 7fb789600640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76c06b500 0x7fb76c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:11.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.578+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb7840acae0 con 0x7fb78c15e070 2024-03-06T07:31:11.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.578+0000 7fb791400640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76c06b500 0x7fb76c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:11.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.579+0000 7fb791400640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76c06b500 0x7fb76c06d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb780006fd0 tx=0x7fb780008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:11.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.583+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb78407bbc0 con 0x7fb78c15e070 2024-03-06T07:31:11.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.793+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb78c06c910 con 0x7fb78c15e070 2024-03-06T07:31:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.842+0000 7fb789600640 1 -- 172.21.15.6:0/74548224 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb78407b4c0 con 0x7fb78c15e070 2024-03-06T07:31:11.845 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76c06b500 msgr2=0x7fb76c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76c06b500 0x7fb76c06d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb780006fd0 tx=0x7fb780008040 comp rx=0 tx=0).stop 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 msgr2=0x7fb78c15c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 0x7fb78c15c860 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb784009f40 tx=0x7fb784004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 shutdown_connections 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76c06b500 0x7fb76c06d9c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78c0d0cd0 0x7fb78c0c7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78c15e070 0x7fb78c15c860 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 --2- 172.21.15.6:0/74548224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78c0cf410 0x7fb78c1f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.845+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 >> 172.21.15.6:0/74548224 conn(0x7fb78c08d2a0 msgr2=0x7fb78c0b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.846+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 shutdown_connections 2024-03-06T07:31:11.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:11.846+0000 7fb792800640 1 -- 172.21.15.6:0/74548224 wait complete. 2024-03-06T07:31:11.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:31:12.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.078+0000 7f187ac00640 1 Processor -- start 2024-03-06T07:31:12.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.078+0000 7f187ac00640 1 -- start start 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f187ac00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f18740d4a10 0x7f18740d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f187ac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 0x7f18740b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f187ac00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18740b3710 con 0x7f187415e0b0 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f187ac00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18740b09e0 con 0x7f18740d4a10 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f187ac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740cfa20 con 0x7f18740d0c80 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f1879800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f187415e0b0 0x7f18740cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38916/0 (socket says 172.21.15.6:38916) 2024-03-06T07:31:12.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.080+0000 7f1879800640 1 -- 172.21.15.6:0/3348809520 learned_addr learned my addr 172.21.15.6:0/3348809520 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1878e00640 1 --2- 172.21.15.6:0/3348809520 >> v2:172.21.15.178:3300/0 conn(0x7f18740d4a10 0x7f18740d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f187a200640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 0x7f18740b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766007790 0 0) 0x7f18740b3710 con 0x7f187415e0b0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c003520 con 0x7f187415e0b0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4079372919 0 0) 0x7f185c003520 con 0x7f187415e0b0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 msgr2=0x7f18740b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 0x7f18740b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 >> v2:172.21.15.178:3300/0 conn(0x7f18740d4a10 msgr2=0x7f18740d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 --2- 172.21.15.6:0/3348809520 >> v2:172.21.15.178:3300/0 conn(0x7f18740d4a10 0x7f18740d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18740b39f0 con 0x7f187415e0b0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f1868001670 con 0x7f187415e0b0 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 >> v1:172.21.15.6:6789/0 conn(0x7f187415e0b0 legacy=0x7f18740cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f185c0039b0 0x7f185c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f185c004800 0x7f185c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 0x7f185c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740cfa20 con 0x7f185c004800 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c00b9c0 con 0x7f185c0039b0 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.081+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c00bb40 con 0x7f18740d0c80 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1878e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f185c004800 0x7f185c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f187a200640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 0x7f185c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1878e00640 1 -- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 msgr2=0x7f185c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1878e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18740d0c80 0x7f185c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1879800640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f185c0039b0 0x7f185c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1878e00640 1 -- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f185c0039b0 msgr2=0x7f185c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1878e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f185c0039b0 0x7f185c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.082+0000 7f1878e00640 1 -- 172.21.15.6:0/3348809520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c003520 con 0x7f185c004800 2024-03-06T07:31:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.083+0000 7f1878e00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f185c004800 0x7f185c004c80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1860009a90 tx=0x7f185c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19635d69af869a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860039070 con 0x7f185c004800 2024-03-06T07:31:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186002c6b0 con 0x7f185c004800 2024-03-06T07:31:12.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f1873e00640 1 -- 172.21.15.6:0/3348809520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860009690 con 0x7f185c004800 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f187ac00640 1 -- 172.21.15.6:0/3348809520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f185c004800 msgr2=0x7f185c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f187ac00640 1 --2- 172.21.15.6:0/3348809520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f185c004800 0x7f185c004c80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f1860009a90 tx=0x7f185c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f187a200640 1 -- 172.21.15.6:0/3348809520 reap_dead start 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f187ac00640 1 -- 172.21.15.6:0/3348809520 shutdown_connections 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f187ac00640 1 -- 172.21.15.6:0/3348809520 >> 172.21.15.6:0/3348809520 conn(0x7f18740d3d30 msgr2=0x7f18740d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.084+0000 7f187ac00640 1 -- 172.21.15.6:0/3348809520 shutdown_connections 2024-03-06T07:31:12.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.085+0000 7f187ac00640 1 -- 172.21.15.6:0/3348809520 wait complete. 2024-03-06T07:31:12.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.087+0000 7f187ac00640 1 Processor -- start 2024-03-06T07:31:12.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.087+0000 7f187ac00640 1 -- start start 2024-03-06T07:31:12.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.088+0000 7f187ac00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f185c004800 0x7f1874152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.088+0000 7f187ac00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18740b79a0 0x7f187406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.088+0000 7f187ac00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 0x7f187406bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.088+0000 7f187ac00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187415e490 con 0x7f18740b7e00 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.088+0000 7f187ac00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740b3fb0 con 0x7f18740b79a0 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.088+0000 7f187ac00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740cfc60 con 0x7f185c004800 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 0x7f187406bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 0x7f187406bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53972/0 (socket says 172.21.15.6:53972) 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 -- 172.21.15.6:0/562921405 learned_addr learned my addr 172.21.15.6:0/562921405 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f1879800640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f185c004800 0x7f1874152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f1878e00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18740b79a0 0x7f187406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 -- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f185c004800 msgr2=0x7f1874152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f185c004800 0x7f1874152060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 -- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18740b79a0 msgr2=0x7f187406b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18740b79a0 0x7f187406b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 -- 172.21.15.6:0/562921405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18741f7100 con 0x7f18740b7e00 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f1879800640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f185c004800 0x7f1874152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f1878e00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18740b79a0 0x7f187406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:12.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.089+0000 7f187a200640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 0x7f187406bee0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f187408e2b0 tx=0x7f186c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.090+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c01c070 con 0x7f18740b7e00 2024-03-06T07:31:12.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.090+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c017400 con 0x7f18740b7e00 2024-03-06T07:31:12.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.090+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c0177f0 con 0x7f18740b7e00 2024-03-06T07:31:12.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.090+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18741f73a0 con 0x7f18740b7e00 2024-03-06T07:31:12.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.090+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18741f7960 con 0x7f18740b7e00 2024-03-06T07:31:12.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.091+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187404f330 con 0x7f18740b7e00 2024-03-06T07:31:12.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.092+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f186c005d10 con 0x7f18740b7e00 2024-03-06T07:31:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.093+0000 7f1871600640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f185406b500 0x7f185406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.093+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f186c08a280 con 0x7f18740b7e00 2024-03-06T07:31:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.093+0000 7f1879800640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f185406b500 0x7f185406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.095+0000 7f1879800640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f185406b500 0x7f185406d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1868001050 tx=0x7f1868009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.097+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f186c012070 con 0x7f18740b7e00 2024-03-06T07:31:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.355+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f18740cf4a0 con 0x7f18740b7e00 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.356+0000 7f1871600640 1 -- 172.21.15.6:0/562921405 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f186c059440 con 0x7f18740b7e00 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.358 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.360 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.362 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.364 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:12.366 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:31:12.367 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:31:12.368 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:31:12.369 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:12.370 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:31:12.371 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f185406b500 msgr2=0x7f185406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f185406b500 0x7f185406d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1868001050 tx=0x7f1868009040 comp rx=0 tx=0).stop 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 msgr2=0x7f187406bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 0x7f187406bee0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f187408e2b0 tx=0x7f186c00f4a0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 shutdown_connections 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f185406b500 0x7f185406d9c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18740b79a0 0x7f187406b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18740b7e00 0x7f187406bee0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 --2- 172.21.15.6:0/562921405 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f185c004800 0x7f1874152060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.358+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 >> 172.21.15.6:0/562921405 conn(0x7f18740d3d30 msgr2=0x7f18740b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.359+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 shutdown_connections 2024-03-06T07:31:12.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.359+0000 7f187ac00640 1 -- 172.21.15.6:0/562921405 wait complete. 2024-03-06T07:31:12.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:12.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.591+0000 7f5551c00640 1 Processor -- start 2024-03-06T07:31:12.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.591+0000 7f5551c00640 1 -- start start 2024-03-06T07:31:12.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.592+0000 7f5551c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 0x7f554c070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f5551c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f554c15f360 0x7f554c0b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f5551c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c06f0a0 con 0x7f554c0b6230 2024-03-06T07:31:12.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f5551c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554c06fcb0 con 0x7f554c15f360 2024-03-06T07:31:12.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f5551c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c06f380 con 0x7f554c15df70 2024-03-06T07:31:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f554aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 0x7f554c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f554aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 0x7f554c070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:51632/0 (socket says 172.21.15.6:51632) 2024-03-06T07:31:12.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.593+0000 7f554aa00640 1 -- 172.21.15.6:0/1136160544 learned_addr learned my addr 172.21.15.6:0/1136160544 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.594+0000 7f554b400640 1 --2- 172.21.15.6:0/1136160544 >> v2:172.21.15.178:3300/0 conn(0x7f554c15f360 0x7f554c0b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.594+0000 7f554aa00640 1 -- 172.21.15.6:0/1136160544 >> v2:172.21.15.178:3300/0 conn(0x7f554c15f360 msgr2=0x7f554c0b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.594+0000 7f554aa00640 1 --2- 172.21.15.6:0/1136160544 >> v2:172.21.15.178:3300/0 conn(0x7f554c15f360 0x7f554c0b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.594+0000 7f554aa00640 1 -- 172.21.15.6:0/1136160544 >> v1:172.21.15.178:6789/0 conn(0x7f554c0b6230 legacy=0x7f554c161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.594+0000 7f554aa00640 1 -- 172.21.15.6:0/1136160544 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c06f660 con 0x7f554c15df70 2024-03-06T07:31:12.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.596+0000 7f554aa00640 1 --2- 172.21.15.6:0/1136160544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 0x7f554c070140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f553c009870 tx=0x7f553c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=995adf685499768e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.596+0000 7f5549600640 1 -- 172.21.15.6:0/1136160544 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c034070 con 0x7f554c15df70 2024-03-06T07:31:12.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.596+0000 7f5549600640 1 -- 172.21.15.6:0/1136160544 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553c030840 con 0x7f554c15df70 2024-03-06T07:31:12.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.596+0000 7f5549600640 1 -- 172.21.15.6:0/1136160544 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553c030c10 con 0x7f554c15df70 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 -- 172.21.15.6:0/1136160544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 msgr2=0x7f554c070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 --2- 172.21.15.6:0/1136160544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 0x7f554c070140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f553c009870 tx=0x7f553c02b980 comp rx=0 tx=0).stop 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 -- 172.21.15.6:0/1136160544 shutdown_connections 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 --2- 172.21.15.6:0/1136160544 >> v2:172.21.15.178:3300/0 conn(0x7f554c15f360 0x7f554c0b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 --2- 172.21.15.6:0/1136160544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c15df70 0x7f554c070140 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 -- 172.21.15.6:0/1136160544 >> 172.21.15.6:0/1136160544 conn(0x7f554c09aa10 msgr2=0x7f554c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 -- 172.21.15.6:0/1136160544 shutdown_connections 2024-03-06T07:31:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.597+0000 7f5551c00640 1 -- 172.21.15.6:0/1136160544 wait complete. 2024-03-06T07:31:12.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.599+0000 7f5551c00640 1 Processor -- start 2024-03-06T07:31:12.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.599+0000 7f5551c00640 1 -- start start 2024-03-06T07:31:12.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f5551c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c0b6230 0x7f554c1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f5551c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 0x7f554c15af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f5551c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f554c15b4d0 0x7f554c09df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f5551c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c070800 con 0x7f554c15df70 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f5551c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c070680 con 0x7f554c15b4d0 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f5551c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c1623d0 con 0x7f554c0b6230 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f554a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 0x7f554c15af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f554a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 0x7f554c15af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53976/0 (socket says 172.21.15.6:53976) 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f554a000640 1 -- 172.21.15.6:0/244719348 learned_addr learned my addr 172.21.15.6:0/244719348 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.601+0000 7f554aa00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c0b6230 0x7f554c1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554b400640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f554c15b4d0 0x7f554c09df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554a000640 1 -- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c0b6230 msgr2=0x7f554c1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554a000640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c0b6230 0x7f554c1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554a000640 1 -- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f554c15b4d0 msgr2=0x7f554c09df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554a000640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f554c15b4d0 0x7f554c09df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554a000640 1 -- 172.21.15.6:0/244719348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c15ba60 con 0x7f554c15df70 2024-03-06T07:31:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f554a000640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 0x7f554c15af90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f553400c430 tx=0x7f553400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55340052d0 con 0x7f554c15df70 2024-03-06T07:31:12.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5534005490 con 0x7f554c15df70 2024-03-06T07:31:12.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c15bd40 con 0x7f554c15df70 2024-03-06T07:31:12.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.602+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534007260 con 0x7f554c15df70 2024-03-06T07:31:12.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.603+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c09c110 con 0x7f554c15df70 2024-03-06T07:31:12.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.604+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554c09c310 con 0x7f554c15df70 2024-03-06T07:31:12.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.605+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5534007400 con 0x7f554c15df70 2024-03-06T07:31:12.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.606+0000 7f5542a00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f552806b4d0 0x7f552806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.606+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5534089390 con 0x7f554c15df70 2024-03-06T07:31:12.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.606+0000 7f554aa00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f552806b4d0 0x7f552806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.607+0000 7f554aa00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f552806b4d0 0x7f552806d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f553c036000 tx=0x7f553c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.610+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f554c09c310 con 0x7f554c15df70 2024-03-06T07:31:12.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.802+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f554c09b670 con 0x7f554c15df70 2024-03-06T07:31:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.804+0000 7f5542a00640 1 -- 172.21.15.6:0/244719348 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+12819 (secure 0 0 0) 0x7f5534058550 con 0x7f554c15df70 2024-03-06T07:31:12.806 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:12.806 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":65,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:11.855800+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f552806b4d0 msgr2=0x7f552806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f552806b4d0 0x7f552806d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f553c036000 tx=0x7f553c03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 msgr2=0x7f554c15af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 0x7f554c15af90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f553400c430 tx=0x7f553400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 shutdown_connections 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f552806b4d0 0x7f552806d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f554c15b4d0 0x7f554c09df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f554c15df70 0x7f554c15af90 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 --2- 172.21.15.6:0/244719348 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f554c0b6230 0x7f554c1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 >> 172.21.15.6:0/244719348 conn(0x7f554c09aa10 msgr2=0x7f554c06d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.806+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 shutdown_connections 2024-03-06T07:31:12.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:12.807+0000 7f5551c00640 1 -- 172.21.15.6:0/244719348 wait complete. 2024-03-06T07:31:12.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:12.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.032+0000 7f563cc00640 1 Processor -- start 2024-03-06T07:31:13.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.032+0000 7f563cc00640 1 -- start start 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f563cc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5638145430 0x7f563815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f563cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56380b2250 0x7f56380b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f563cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f563815fe40 con 0x7f563815cc90 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f563cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5638160120 con 0x7f56380b2250 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f563cc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5638160400 con 0x7f5638145430 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f5637e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f56380b2250 0x7f56380b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f5636a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f563815cc90 0x7f56380b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38930/0 (socket says 172.21.15.6:38930) 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.034+0000 7f5636a00640 1 -- 172.21.15.6:0/300425170 learned_addr learned my addr 172.21.15.6:0/300425170 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.035+0000 7f5637400640 1 --2- 172.21.15.6:0/300425170 >> v2:172.21.15.178:3300/0 conn(0x7f5638145430 0x7f563815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.035+0000 7f5637e00640 1 -- 172.21.15.6:0/300425170 >> v2:172.21.15.178:3300/0 conn(0x7f5638145430 msgr2=0x7f563815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.035+0000 7f5637e00640 1 --2- 172.21.15.6:0/300425170 >> v2:172.21.15.178:3300/0 conn(0x7f5638145430 0x7f563815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.035+0000 7f5637e00640 1 -- 172.21.15.6:0/300425170 >> v1:172.21.15.6:6789/0 conn(0x7f563815cc90 legacy=0x7f56380b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.035+0000 7f5637e00640 1 -- 172.21.15.6:0/300425170 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56381606e0 con 0x7f56380b2250 2024-03-06T07:31:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.036+0000 7f5637e00640 1 --2- 172.21.15.6:0/300425170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f56380b2fd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f56380b2670 tx=0x7f563002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7360c363deb3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.036+0000 7f5636000640 1 -- 172.21.15.6:0/300425170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630039070 con 0x7f56380b2250 2024-03-06T07:31:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.036+0000 7f5636000640 1 -- 172.21.15.6:0/300425170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563002cc70 con 0x7f56380b2250 2024-03-06T07:31:13.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.036+0000 7f5636000640 1 -- 172.21.15.6:0/300425170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630034660 con 0x7f56380b2250 2024-03-06T07:31:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 -- 172.21.15.6:0/300425170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 msgr2=0x7f56380b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 --2- 172.21.15.6:0/300425170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f56380b2fd0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f56380b2670 tx=0x7f563002bec0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 -- 172.21.15.6:0/300425170 shutdown_connections 2024-03-06T07:31:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 --2- 172.21.15.6:0/300425170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f56380b2fd0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 --2- 172.21.15.6:0/300425170 >> v2:172.21.15.178:3300/0 conn(0x7f5638145430 0x7f563815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 -- 172.21.15.6:0/300425170 >> 172.21.15.6:0/300425170 conn(0x7f563806b3a0 msgr2=0x7f563806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 -- 172.21.15.6:0/300425170 shutdown_connections 2024-03-06T07:31:13.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.037+0000 7f563cc00640 1 -- 172.21.15.6:0/300425170 wait complete. 2024-03-06T07:31:13.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.039+0000 7f563cc00640 1 Processor -- start 2024-03-06T07:31:13.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.039+0000 7f563cc00640 1 -- start start 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f563cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f563815b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f563cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5638145430 0x7f56380a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f563cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f563815cc90 0x7f563813df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f563cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638145990 con 0x7f56380b2250 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f563cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5638145810 con 0x7f5638145430 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f563cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56380cde00 con 0x7f563815cc90 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f563815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f563815b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53992/0 (socket says 172.21.15.6:53992) 2024-03-06T07:31:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 -- 172.21.15.6:0/883131130 learned_addr learned my addr 172.21.15.6:0/883131130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5636a00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5638145430 0x7f56380a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 -- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f563815cc90 msgr2=0x7f563813df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f563815cc90 0x7f563813df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 -- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5638145430 msgr2=0x7f56380a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5638145430 0x7f56380a9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.040+0000 7f5637400640 1 -- 172.21.15.6:0/883131130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56380aa160 con 0x7f56380b2250 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.041+0000 7f5637400640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f563815b740 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f562800d800 tx=0x7f562800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.041+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800b6c0 con 0x7f56380b2250 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.041+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628019070 con 0x7f56380b2250 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.041+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800ba80 con 0x7f56380b2250 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.041+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56381f29b0 con 0x7f56380b2250 2024-03-06T07:31:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.041+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56381f2f70 con 0x7f56380b2250 2024-03-06T07:31:13.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.042+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563804f330 con 0x7f56380b2250 2024-03-06T07:31:13.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.043+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f562800bc20 con 0x7f56380b2250 2024-03-06T07:31:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.045+0000 7f562f400640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f561806b4d0 0x7f561806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.045+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f562808a430 con 0x7f56380b2250 2024-03-06T07:31:13.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.046+0000 7f5636a00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f561806b4d0 0x7f561806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.047+0000 7f5636a00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f561806b4d0 0x7f561806d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5624000f60 tx=0x7f5624009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.049+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5628059560 con 0x7f56380b2250 2024-03-06T07:31:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.247+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f563806e240 con 0x7f56380b2250 2024-03-06T07:31:13.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.248+0000 7f562f400640 1 -- 172.21.15.6:0/883131130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5628058e60 con 0x7f56380b2250 2024-03-06T07:31:13.251 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:31:13.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f561806b4d0 msgr2=0x7f561806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f561806b4d0 0x7f561806d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5624000f60 tx=0x7f5624009040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 msgr2=0x7f563815b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f563815b740 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f562800d800 tx=0x7f562800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 shutdown_connections 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f561806b4d0 0x7f561806d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5638145430 0x7f56380a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f563815cc90 0x7f563813df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 --2- 172.21.15.6:0/883131130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f56380b2250 0x7f563815b740 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 >> 172.21.15.6:0/883131130 conn(0x7f563806b3a0 msgr2=0x7f563813fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 shutdown_connections 2024-03-06T07:31:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.251+0000 7f563cc00640 1 -- 172.21.15.6:0/883131130 wait complete. 2024-03-06T07:31:13.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:31:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.481+0000 7f88b2600640 1 Processor -- start 2024-03-06T07:31:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.482+0000 7f88b2600640 1 -- start start 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.483+0000 7f88b2600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88ac0cd060 0x7f88ac15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.483+0000 7f88b2600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac15e810 0x7f88ac0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.483+0000 7f88b2600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ac15ed10 con 0x7f88ac1552b0 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.483+0000 7f88b2600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ac15eff0 con 0x7f88ac0cd060 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.483+0000 7f88b2600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac15f2d0 con 0x7f88ac15e810 2024-03-06T07:31:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88ac0cd060 0x7f88ac15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f88ac0cd060 0x7f88ac15b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53996/0 (socket says 172.21.15.6:53996) 2024-03-06T07:31:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 -- 172.21.15.6:0/4222511571 learned_addr learned my addr 172.21.15.6:0/4222511571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 -- 172.21.15.6:0/4222511571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac15e810 msgr2=0x7f88ac0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 --2- 172.21.15.6:0/4222511571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac15e810 0x7f88ac0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 -- 172.21.15.6:0/4222511571 >> v1:172.21.15.178:6789/0 conn(0x7f88ac1552b0 legacy=0x7f88ac0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.484+0000 7f88a3e00640 1 -- 172.21.15.6:0/4222511571 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac15f5b0 con 0x7f88ac0cd060 2024-03-06T07:31:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.485+0000 7f88a3e00640 1 --2- 172.21.15.6:0/4222511571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac0cd060 0x7f88ac15b1d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0cf1d0 tx=0x7f889402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc858d5c160a711 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88abe00640 1 -- 172.21.15.6:0/4222511571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88940317c0 con 0x7f88ac0cd060 2024-03-06T07:31:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88abe00640 1 -- 172.21.15.6:0/4222511571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8894037070 con 0x7f88ac0cd060 2024-03-06T07:31:13.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88abe00640 1 -- 172.21.15.6:0/4222511571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894031b80 con 0x7f88ac0cd060 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 -- 172.21.15.6:0/4222511571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac0cd060 msgr2=0x7f88ac15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 --2- 172.21.15.6:0/4222511571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac0cd060 0x7f88ac15b1d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0cf1d0 tx=0x7f889402b860 comp rx=0 tx=0).stop 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 -- 172.21.15.6:0/4222511571 shutdown_connections 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 --2- 172.21.15.6:0/4222511571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac15e810 0x7f88ac0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 --2- 172.21.15.6:0/4222511571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac0cd060 0x7f88ac15b1d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 -- 172.21.15.6:0/4222511571 >> 172.21.15.6:0/4222511571 conn(0x7f88ac06b3d0 msgr2=0x7f88ac06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 -- 172.21.15.6:0/4222511571 shutdown_connections 2024-03-06T07:31:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.486+0000 7f88b2600640 1 -- 172.21.15.6:0/4222511571 wait complete. 2024-03-06T07:31:13.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.488+0000 7f88b2600640 1 Processor -- start 2024-03-06T07:31:13.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.488+0000 7f88b2600640 1 -- start start 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b2600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88ac0cd060 0x7f88ac144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b2600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac1552b0 0x7f88ac0a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 0x7f88ac144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b2600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac06dd30 con 0x7f88ac15e810 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b2600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac0b3760 con 0x7f88ac0cd060 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b2600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac06deb0 con 0x7f88ac1552b0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 0x7f88ac144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b1c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 0x7f88ac144480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54012/0 (socket says 172.21.15.6:54012) 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.489+0000 7f88b1c00640 1 -- 172.21.15.6:0/3744419261 learned_addr learned my addr 172.21.15.6:0/3744419261 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1c00640 1 -- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac1552b0 msgr2=0x7f88ac0a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1c00640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac1552b0 0x7f88ac0a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1200640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88ac0cd060 0x7f88ac144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1c00640 1 -- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88ac0cd060 msgr2=0x7f88ac144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1c00640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88ac0cd060 0x7f88ac144cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1c00640 1 -- 172.21.15.6:0/3744419261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac0aa450 con 0x7f88ac15e810 2024-03-06T07:31:13.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.490+0000 7f88b1c00640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 0x7f88ac144480 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f88a400b9e0 tx=0x7f88a400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.491+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4019070 con 0x7f88ac15e810 2024-03-06T07:31:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.491+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a400cb40 con 0x7f88ac15e810 2024-03-06T07:31:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.491+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4014440 con 0x7f88ac15e810 2024-03-06T07:31:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.491+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac1449c0 con 0x7f88ac15e810 2024-03-06T07:31:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.491+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ac145600 con 0x7f88ac15e810 2024-03-06T07:31:13.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.492+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ac04f330 con 0x7f88ac15e810 2024-03-06T07:31:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.493+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88a401c070 con 0x7f88ac15e810 2024-03-06T07:31:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.494+0000 7f88a9600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.494+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f88a4089b30 con 0x7f88ac15e810 2024-03-06T07:31:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.495+0000 7f88b1200640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.496+0000 7f88b1200640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f888806b4d0 0x7f888806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0b23c0 tx=0x7f889c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.498+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88a4058cf0 con 0x7f88ac15e810 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.697+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f88ac0cf480 con 0x7f88ac15e810 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.698+0000 7f88a9600640 1 -- 172.21.15.6:0/3744419261 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v65) v1 ==== 84+0+65 (secure 0 0 0) 0x7f88a40585f0 con 0x7f88ac15e810 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:13.700 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:31:13.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:31:13.701 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:31:13.701 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:31:13.701 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:31:13.701 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.700+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f888806b4d0 msgr2=0x7f888806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.700+0000 7f88b2600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f888806b4d0 0x7f888806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0b23c0 tx=0x7f889c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.700+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 msgr2=0x7f88ac144480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.700+0000 7f88b2600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 0x7f88ac144480 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f88a400b9e0 tx=0x7f88a400bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 shutdown_connections 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f888806b4d0 0x7f888806d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f88ac15e810 0x7f88ac144480 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f88ac1552b0 0x7f88ac0a9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 --2- 172.21.15.6:0/3744419261 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f88ac0cd060 0x7f88ac144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 >> 172.21.15.6:0/3744419261 conn(0x7f88ac06b3d0 msgr2=0x7f88ac0cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 shutdown_connections 2024-03-06T07:31:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.701+0000 7f88b2600640 1 -- 172.21.15.6:0/3744419261 wait complete. 2024-03-06T07:31:13.721 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:31:13.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:31:13.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.946+0000 7f90f5a00640 1 Processor -- start 2024-03-06T07:31:13.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.947+0000 7f90f5a00640 1 -- start start 2024-03-06T07:31:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.948+0000 7f90f5a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f90f00b4900 0x7f90f00cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.948+0000 7f90f5a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00d23c0 con 0x7f90f00d0d90 2024-03-06T07:31:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.948+0000 7f90f5a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f00d26a0 con 0x7f90f015e1f0 2024-03-06T07:31:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.948+0000 7f90f5a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f00d4c30 con 0x7f90f00b4900 2024-03-06T07:31:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.948+0000 7f90ef400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f90f00d0d90 0x7f90f00ce8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38944/0 (socket says 172.21.15.6:38944) 2024-03-06T07:31:13.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90ef400640 1 -- 172.21.15.6:0/94471115 learned_addr learned my addr 172.21.15.6:0/94471115 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187356583 0 0) 0x7f90f00d23c0 con 0x7f90f00d0d90 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d0003520 con 0x7f90f00d0d90 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90f5000640 1 --2- 172.21.15.6:0/94471115 >> v2:172.21.15.178:3300/0 conn(0x7f90f00b4900 0x7f90f00cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3800241117 0 0) 0x7f90d0003520 con 0x7f90f00d0d90 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 >> v2:172.21.15.178:3300/0 conn(0x7f90f00b4900 msgr2=0x7f90f00cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 --2- 172.21.15.6:0/94471115 >> v2:172.21.15.178:3300/0 conn(0x7f90f00b4900 0x7f90f00cefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 >> v1:172.21.15.178:6789/0 conn(0x7f90f015e1f0 legacy=0x7f90f00d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00d2980 con 0x7f90f00d0d90 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f90d80019f0 con 0x7f90f00d0d90 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.949+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 >> v1:172.21.15.6:6789/0 conn(0x7f90f00d0d90 legacy=0x7f90f00ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90eea00640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00039b0 0x7f90d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90eea00640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90d0004800 0x7f90d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90eea00640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90d00052e0 0x7f90d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00d4c30 con 0x7f90d00052e0 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d000c1b0 con 0x7f90d00039b0 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d000c3b0 con 0x7f90d0004800 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90f5000640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90d00052e0 0x7f90d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90efe00640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00039b0 0x7f90d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90ef400640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90d0004800 0x7f90d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90f5000640 1 -- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90d0004800 msgr2=0x7f90d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90f5000640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90d0004800 0x7f90d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90f5000640 1 -- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00039b0 msgr2=0x7f90d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90f5000640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00039b0 0x7f90d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.950+0000 7f90f5000640 1 -- 172.21.15.6:0/94471115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d0003520 con 0x7f90d00052e0 2024-03-06T07:31:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.951+0000 7f90f5000640 1 -- 172.21.15.6:0/94471115 reap_dead start 2024-03-06T07:31:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.952+0000 7f90f5000640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90d00052e0 0x7f90d000b9c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f90f00b4d90 tx=0x7f90d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5309a79bf190a2de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.952+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8036020 con 0x7f90d00052e0 2024-03-06T07:31:13.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.952+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8032630 con 0x7f90d00052e0 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.952+0000 7f90eea00640 1 -- 172.21.15.6:0/94471115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800b610 con 0x7f90d00052e0 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.952+0000 7f90f5a00640 1 -- 172.21.15.6:0/94471115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90d00052e0 msgr2=0x7f90d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.952+0000 7f90f5a00640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90d00052e0 0x7f90d000b9c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f90f00b4d90 tx=0x7f90d0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.953+0000 7f90f5a00640 1 -- 172.21.15.6:0/94471115 shutdown_connections 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.953+0000 7f90f5a00640 1 --2- 172.21.15.6:0/94471115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90d00052e0 0x7f90d000b9c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.953+0000 7f90f5a00640 1 -- 172.21.15.6:0/94471115 >> 172.21.15.6:0/94471115 conn(0x7f90f009d000 msgr2=0x7f90f00d3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.953+0000 7f90f5a00640 1 -- 172.21.15.6:0/94471115 shutdown_connections 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.953+0000 7f90f5a00640 1 -- 172.21.15.6:0/94471115 wait complete. 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.954+0000 7f90f5a00640 1 Processor -- start 2024-03-06T07:31:13.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 -- start start 2024-03-06T07:31:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00052e0 0x7f90f0154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 0x7f90f015ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90f00b9080 0x7f90f00b9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00d4660 con 0x7f90f015e5d0 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00b2da0 con 0x7f90d00052e0 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.955+0000 7f90f5a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f00d47e0 con 0x7f90f00b9080 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 0x7f90f015ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 0x7f90f015ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54044/0 (socket says 172.21.15.6:54044) 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 -- 172.21.15.6:0/3210608355 learned_addr learned my addr 172.21.15.6:0/3210608355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90efe00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00052e0 0x7f90f0154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 -- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90f00b9080 msgr2=0x7f90f00b9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90f5000640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90f00b9080 0x7f90f00b9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90f00b9080 0x7f90f00b9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 -- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00052e0 msgr2=0x7f90f0154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00052e0 0x7f90f0154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.956+0000 7f90ef400640 1 -- 172.21.15.6:0/3210608355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f00b8050 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.957+0000 7f90ef400640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 0x7f90f015ea50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f90d80079c0 tx=0x7f90d800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.957+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d800edf0 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.957+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8018070 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.957+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90d8013660 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.957+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f0154b50 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.957+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f00b85a0 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.958+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f00b2120 con 0x7f90f015e5d0 2024-03-06T07:31:13.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.959+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90d8013800 con 0x7f90f015e5d0 2024-03-06T07:31:13.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.960+0000 7f90e7e00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.961+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f90d808a4a0 con 0x7f90f015e5d0 2024-03-06T07:31:13.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.961+0000 7f90efe00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.962+0000 7f90efe00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f90cc06b4d0 0x7f90cc06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f90e0007fd0 tx=0x7f90e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:13.965+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90f00b2120 con 0x7f90f015e5d0 2024-03-06T07:31:14.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.165+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f90f00b2b80 con 0x7f90f015e5d0 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.870+0000 7f90e7e00640 1 -- 172.21.15.6:0/3210608355 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v66) v1 ==== 138+0+0 (secure 0 0 0) 0x7f90d8059660 con 0x7f90f015e5d0 2024-03-06T07:31:14.873 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:31:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f90cc06b4d0 msgr2=0x7f90cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f90cc06b4d0 0x7f90cc06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f90e0007fd0 tx=0x7f90e0009040 comp rx=0 tx=0).stop 2024-03-06T07:31:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 msgr2=0x7f90f015ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 0x7f90f015ea50 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f90d80079c0 tx=0x7f90d800e530 comp rx=0 tx=0).stop 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 shutdown_connections 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90f00b9080 0x7f90f00b9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90f015e5d0 0x7f90f015ea50 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 --2- 172.21.15.6:0/3210608355 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90d00052e0 0x7f90f0154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 >> 172.21.15.6:0/3210608355 conn(0x7f90f009d000 msgr2=0x7f90f006c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.874+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 shutdown_connections 2024-03-06T07:31:14.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:14.875+0000 7f90f5a00640 1 -- 172.21.15.6:0/3210608355 wait complete. 2024-03-06T07:31:14.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:31:15.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.120+0000 7f4331000640 1 Processor -- start 2024-03-06T07:31:15.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.120+0000 7f4331000640 1 -- start start 2024-03-06T07:31:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.122+0000 7f4331000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f432c06cfe0 0x7f432c0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.122+0000 7f4331000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f432c161470 0x7f432c0b5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.122+0000 7f4331000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c15f0e0 con 0x7f432c15e060 2024-03-06T07:31:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.122+0000 7f4331000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c15f3c0 con 0x7f432c06cfe0 2024-03-06T07:31:15.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.122+0000 7f4331000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c15f6a0 con 0x7f432c161470 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f432b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f432c15e060 0x7f432c06c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38960/0 (socket says 172.21.15.6:38960) 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f432b400640 1 -- 172.21.15.6:0/3198252750 learned_addr learned my addr 172.21.15.6:0/3198252750 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f4322a00640 1 --2- 172.21.15.6:0/3198252750 >> v2:172.21.15.6:3300/0 conn(0x7f432c06cfe0 0x7f432c0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f432be00640 1 --2- 172.21.15.6:0/3198252750 >> v2:172.21.15.178:3300/0 conn(0x7f432c161470 0x7f432c0b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f432aa00640 1 -- 172.21.15.6:0/3198252750 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067001402 0 0) 0x7f432c15f0e0 con 0x7f432c15e060 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f432aa00640 1 -- 172.21.15.6:0/3198252750 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4310003520 con 0x7f432c15e060 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f4322a00640 1 -- 172.21.15.6:0/3198252750 >> v2:172.21.15.178:3300/0 conn(0x7f432c161470 msgr2=0x7f432c0b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f4322a00640 1 --2- 172.21.15.6:0/3198252750 >> v2:172.21.15.178:3300/0 conn(0x7f432c161470 0x7f432c0b5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f4322a00640 1 -- 172.21.15.6:0/3198252750 >> v1:172.21.15.6:6789/0 conn(0x7f432c15e060 legacy=0x7f432c06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.123+0000 7f4322a00640 1 -- 172.21.15.6:0/3198252750 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c15f980 con 0x7f432c06cfe0 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.124+0000 7f4322a00640 1 --2- 172.21.15.6:0/3198252750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c06cfe0 0x7f432c0b5430 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f432c0b4850 tx=0x7f431402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf1862eeb7eab27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.125+0000 7f432aa00640 1 -- 172.21.15.6:0/3198252750 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314035020 con 0x7f432c06cfe0 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.125+0000 7f432aa00640 1 -- 172.21.15.6:0/3198252750 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314031830 con 0x7f432c06cfe0 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.125+0000 7f432aa00640 1 -- 172.21.15.6:0/3198252750 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314031be0 con 0x7f432c06cfe0 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.125+0000 7f4331000640 1 -- 172.21.15.6:0/3198252750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c06cfe0 msgr2=0x7f432c0b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.125+0000 7f4331000640 1 --2- 172.21.15.6:0/3198252750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c06cfe0 0x7f432c0b5430 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f432c0b4850 tx=0x7f431402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.126+0000 7f4331000640 1 -- 172.21.15.6:0/3198252750 shutdown_connections 2024-03-06T07:31:15.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.126+0000 7f4331000640 1 --2- 172.21.15.6:0/3198252750 >> v2:172.21.15.178:3300/0 conn(0x7f432c161470 0x7f432c0b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.126+0000 7f4331000640 1 --2- 172.21.15.6:0/3198252750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c06cfe0 0x7f432c0b5430 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.126+0000 7f4331000640 1 -- 172.21.15.6:0/3198252750 >> 172.21.15.6:0/3198252750 conn(0x7f432c09aa10 msgr2=0x7f432c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.126+0000 7f4331000640 1 -- 172.21.15.6:0/3198252750 shutdown_connections 2024-03-06T07:31:15.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.126+0000 7f4331000640 1 -- 172.21.15.6:0/3198252750 wait complete. 2024-03-06T07:31:15.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.128+0000 7f4331000640 1 Processor -- start 2024-03-06T07:31:15.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.128+0000 7f4331000640 1 -- start start 2024-03-06T07:31:15.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f4331000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432c06cfe0 0x7f432c09e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f4331000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f432c15e060 0x7f432c09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f4331000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 0x7f432c14b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f4331000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c15e440 con 0x7f432c161470 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f4331000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c15ff40 con 0x7f432c06cfe0 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f4331000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0b5eb0 con 0x7f432c15e060 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 0x7f432c14b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 0x7f432c14b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54064/0 (socket says 172.21.15.6:54064) 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 -- 172.21.15.6:0/3296851044 learned_addr learned my addr 172.21.15.6:0/3296851044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432b400640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432c06cfe0 0x7f432c09e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 -- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f432c15e060 msgr2=0x7f432c09cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f432c15e060 0x7f432c09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 -- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432c06cfe0 msgr2=0x7f432c09e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432c06cfe0 0x7f432c09e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.130+0000 7f432be00640 1 -- 172.21.15.6:0/3296851044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c09d8a0 con 0x7f432c161470 2024-03-06T07:31:15.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f432b400640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432c06cfe0 0x7f432c09e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f432be00640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 0x7f432c14b560 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f432400d5d0 tx=0x7f432400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43240125f0 con 0x7f432c161470 2024-03-06T07:31:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4324019070 con 0x7f432c161470 2024-03-06T07:31:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43240129f0 con 0x7f432c161470 2024-03-06T07:31:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c09de30 con 0x7f432c161470 2024-03-06T07:31:15.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.131+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c09e3f0 con 0x7f432c161470 2024-03-06T07:31:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.132+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c04f330 con 0x7f432c161470 2024-03-06T07:31:15.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.134+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4324012b90 con 0x7f432c161470 2024-03-06T07:31:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.135+0000 7f4323e00640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b500 0x7f430806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.135+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f432408a470 con 0x7f432c161470 2024-03-06T07:31:15.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.135+0000 7f432b400640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b500 0x7f430806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.137+0000 7f432b400640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b500 0x7f430806d9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f431c005c00 tx=0x7f431c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.140+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4324059440 con 0x7f432c161470 2024-03-06T07:31:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.340+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f432c09e5c0 con 0x7f432c161470 2024-03-06T07:31:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.876+0000 7f4323e00640 1 -- 172.21.15.6:0/3296851044 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v67) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4324058d40 con 0x7f432c161470 2024-03-06T07:31:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:31:15.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.881+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b500 msgr2=0x7f430806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.881+0000 7f4331000640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b500 0x7f430806d9c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f431c005c00 tx=0x7f431c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:15.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.881+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 msgr2=0x7f432c14b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.881+0000 7f4331000640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 0x7f432c14b560 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f432400d5d0 tx=0x7f432400da90 comp rx=0 tx=0).stop 2024-03-06T07:31:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 shutdown_connections 2024-03-06T07:31:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f430806b500 0x7f430806d9c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f432c161470 0x7f432c14b560 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f432c15e060 0x7f432c09cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 --2- 172.21.15.6:0/3296851044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f432c06cfe0 0x7f432c09e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 >> 172.21.15.6:0/3296851044 conn(0x7f432c09aa10 msgr2=0x7f432c0b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.882+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 shutdown_connections 2024-03-06T07:31:15.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:15.883+0000 7f4331000640 1 -- 172.21.15.6:0/3296851044 wait complete. 2024-03-06T07:31:15.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:31:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.103+0000 7f81d5800640 1 Processor -- start 2024-03-06T07:31:16.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.103+0000 7f81d5800640 1 -- start start 2024-03-06T07:31:16.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.104+0000 7f81d5800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f81d015bf20 0x7f81d00acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.104+0000 7f81d5800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f81d0071e20 0x7f81d00ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.104+0000 7f81d5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d0072df0 con 0x7f81d0072520 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.104+0000 7f81d5800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d00cdab0 con 0x7f81d0071e20 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.104+0000 7f81d5800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d00b4d70 con 0x7f81d015bf20 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81ce000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f81d0072520 0x7f81d00cdfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38974/0 (socket says 172.21.15.6:38974) 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81ce000640 1 -- 172.21.15.6:0/3182014335 learned_addr learned my addr 172.21.15.6:0/3182014335 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cf400640 1 --2- 172.21.15.6:0/3182014335 >> v2:172.21.15.6:3300/0 conn(0x7f81d0071e20 0x7f81d00ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cea00640 1 --2- 172.21.15.6:0/3182014335 >> v2:172.21.15.178:3300/0 conn(0x7f81d015bf20 0x7f81d00acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cd600640 1 -- 172.21.15.6:0/3182014335 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072395205 0 0) 0x7f81d0072df0 con 0x7f81d0072520 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cd600640 1 -- 172.21.15.6:0/3182014335 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b4003520 con 0x7f81d0072520 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cf400640 1 -- 172.21.15.6:0/3182014335 >> v2:172.21.15.178:3300/0 conn(0x7f81d015bf20 msgr2=0x7f81d00acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cf400640 1 --2- 172.21.15.6:0/3182014335 >> v2:172.21.15.178:3300/0 conn(0x7f81d015bf20 0x7f81d00acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cf400640 1 -- 172.21.15.6:0/3182014335 >> v1:172.21.15.6:6789/0 conn(0x7f81d0072520 legacy=0x7f81d00cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.105+0000 7f81cf400640 1 -- 172.21.15.6:0/3182014335 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d00b5050 con 0x7f81d0071e20 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.107+0000 7f81cf400640 1 --2- 172.21.15.6:0/3182014335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d00ccad0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f81d006ec40 tx=0x7f81c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c824174e532ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.107+0000 7f81cd600640 1 -- 172.21.15.6:0/3182014335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8039070 con 0x7f81d0071e20 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.107+0000 7f81cd600640 1 -- 172.21.15.6:0/3182014335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c802cc80 con 0x7f81d0071e20 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.107+0000 7f81cd600640 1 -- 172.21.15.6:0/3182014335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c8034640 con 0x7f81d0071e20 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.107+0000 7f81d5800640 1 -- 172.21.15.6:0/3182014335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 msgr2=0x7f81d00ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.107+0000 7f81d5800640 1 --2- 172.21.15.6:0/3182014335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d00ccad0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f81d006ec40 tx=0x7f81c802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.108+0000 7f81d5800640 1 -- 172.21.15.6:0/3182014335 shutdown_connections 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.108+0000 7f81d5800640 1 --2- 172.21.15.6:0/3182014335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d00ccad0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.108+0000 7f81d5800640 1 --2- 172.21.15.6:0/3182014335 >> v2:172.21.15.178:3300/0 conn(0x7f81d015bf20 0x7f81d00acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.108+0000 7f81d5800640 1 -- 172.21.15.6:0/3182014335 >> 172.21.15.6:0/3182014335 conn(0x7f81d006b3d0 msgr2=0x7f81d006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.108+0000 7f81d5800640 1 -- 172.21.15.6:0/3182014335 shutdown_connections 2024-03-06T07:31:16.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.108+0000 7f81d5800640 1 -- 172.21.15.6:0/3182014335 wait complete. 2024-03-06T07:31:16.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.109+0000 7f81d5800640 1 Processor -- start 2024-03-06T07:31:16.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.110+0000 7f81d5800640 1 -- start start 2024-03-06T07:31:16.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81d5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81d5800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81d0072520 0x7f81d015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81d5800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81d015bf20 0x7f81d015aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81d5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d00b4b10 con 0x7f81d0071e20 2024-03-06T07:31:16.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81d5800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d00b4990 con 0x7f81d0072520 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81d5800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d00ce6c0 con 0x7f81d015bf20 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d014c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54084/0 (socket says 172.21.15.6:54084) 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 -- 172.21.15.6:0/3302082835 learned_addr learned my addr 172.21.15.6:0/3302082835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81ce000640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81d0072520 0x7f81d015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 -- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81d015bf20 msgr2=0x7f81d015aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cf400640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81d015bf20 0x7f81d015aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81d015bf20 0x7f81d015aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 -- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81d0072520 msgr2=0x7f81d015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81d0072520 0x7f81d015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81cea00640 1 -- 172.21.15.6:0/3302082835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d01f76c0 con 0x7f81d0071e20 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.111+0000 7f81ce000640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81d0072520 0x7f81d015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.112+0000 7f81cea00640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d014c3e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f81c000d800 tx=0x7f81c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.112+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c000b6c0 con 0x7f81d0071e20 2024-03-06T07:31:16.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.112+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c0019070 con 0x7f81d0071e20 2024-03-06T07:31:16.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.112+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c000ba80 con 0x7f81d0071e20 2024-03-06T07:31:16.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.112+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d01f79a0 con 0x7f81d0071e20 2024-03-06T07:31:16.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.112+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d01f8240 con 0x7f81d0071e20 2024-03-06T07:31:16.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.113+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d004f330 con 0x7f81d0071e20 2024-03-06T07:31:16.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.114+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81c000bc20 con 0x7f81d0071e20 2024-03-06T07:31:16.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.115+0000 7f81c6a00640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81b006b4d0 0x7f81b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.115+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f81c008a6f0 con 0x7f81d0071e20 2024-03-06T07:31:16.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.116+0000 7f81ce000640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81b006b4d0 0x7f81b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.117+0000 7f81ce000640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81b006b4d0 0x7f81b006d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f81d01f7620 tx=0x7f81bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.121+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81c0059530 con 0x7f81d0071e20 2024-03-06T07:31:16.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.359+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f81d0156b60 con 0x7f81d0071e20 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.916+0000 7f81c6a00640 1 -- 172.21.15.6:0/3302082835 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) v1 ==== 174+0+0 (secure 0 0 0) 0x7f81c0058e30 con 0x7f81d0071e20 2024-03-06T07:31:16.918 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.923+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81b006b4d0 msgr2=0x7f81b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.923+0000 7f81d5800640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81b006b4d0 0x7f81b006d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f81d01f7620 tx=0x7f81bc009040 comp rx=0 tx=0).stop 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.923+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 msgr2=0x7f81d014c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.923+0000 7f81d5800640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d014c3e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f81c000d800 tx=0x7f81c000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.925+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 shutdown_connections 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.925+0000 7f81d5800640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81b006b4d0 0x7f81b006d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.925+0000 7f81d5800640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f81d0072520 0x7f81d015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.925+0000 7f81d5800640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f81d015bf20 0x7f81d015aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.925+0000 7f81d5800640 1 --2- 172.21.15.6:0/3302082835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f81d0071e20 0x7f81d014c3e0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.925+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 >> 172.21.15.6:0/3302082835 conn(0x7f81d006b3d0 msgr2=0x7f81d00b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.926+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 shutdown_connections 2024-03-06T07:31:16.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:16.926+0000 7f81d5800640 1 -- 172.21.15.6:0/3302082835 wait complete. 2024-03-06T07:31:16.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:17.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.139+0000 7f4823000640 1 Processor -- start 2024-03-06T07:31:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.140+0000 7f4823000640 1 -- start start 2024-03-06T07:31:17.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.141+0000 7f4823000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 0x7f481c0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.141+0000 7f4823000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0ce9a0 con 0x7f481c06e240 2024-03-06T07:31:17.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.141+0000 7f4823000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0cec80 con 0x7f481c161670 2024-03-06T07:31:17.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.141+0000 7f4823000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0730d0 con 0x7f481c071380 2024-03-06T07:31:17.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.141+0000 7f4821000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f481c06e240 0x7f481c0ad5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38988/0 (socket says 172.21.15.6:38988) 2024-03-06T07:31:17.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.141+0000 7f4821000640 1 -- 172.21.15.6:0/2157676792 learned_addr learned my addr 172.21.15.6:0/2157676792 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:17.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f4813400640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 0x7f481c0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061640878 0 0) 0x7f481c0ce9a0 con 0x7f481c06e240 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800003520 con 0x7f481c06e240 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374968623 0 0) 0x7f4800003520 con 0x7f481c06e240 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 msgr2=0x7f481c0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 0x7f481c0ad090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 >> v1:172.21.15.178:6789/0 conn(0x7f481c161670 legacy=0x7f481c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0cef60 con 0x7f481c06e240 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f48140019f0 con 0x7f481c06e240 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 >> v1:172.21.15.6:6789/0 conn(0x7f481c06e240 legacy=0x7f481c0ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4800003a50 0x7f4800003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.142+0000 7f481b400640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4800004800 0x7f4800004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481b400640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 0x7f4800005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0730d0 con 0x7f4800003a50 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480000bbb0 con 0x7f4800004800 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480000bdb0 con 0x7f481c071380 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481be00640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4800003a50 0x7f4800003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f4813400640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4800004800 0x7f4800004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f4821000640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 0x7f4800005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481be00640 1 -- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 msgr2=0x7f4800005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481be00640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f481c071380 0x7f4800005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481be00640 1 -- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4800004800 msgr2=0x7f4800004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481be00640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4800004800 0x7f4800004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.143+0000 7f481be00640 1 -- 172.21.15.6:0/2157676792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4800003520 con 0x7f4800003a50 2024-03-06T07:31:17.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.144+0000 7f481be00640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4800003a50 0x7f4800003e70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f480c001750 tx=0x7f4800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb5fd2dad2d8ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c001370 con 0x7f4800003a50 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480c0052d0 con 0x7f4800003a50 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f481b400640 1 -- 172.21.15.6:0/2157676792 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c003830 con 0x7f4800003a50 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4823000640 1 -- 172.21.15.6:0/2157676792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4800003a50 msgr2=0x7f4800003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4823000640 1 --2- 172.21.15.6:0/2157676792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4800003a50 0x7f4800003e70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f480c001750 tx=0x7f4800005830 comp rx=0 tx=0).stop 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4821000640 1 -- 172.21.15.6:0/2157676792 reap_dead start 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4823000640 1 -- 172.21.15.6:0/2157676792 shutdown_connections 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4823000640 1 -- 172.21.15.6:0/2157676792 >> 172.21.15.6:0/2157676792 conn(0x7f481c06afb0 msgr2=0x7f481c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4823000640 1 -- 172.21.15.6:0/2157676792 shutdown_connections 2024-03-06T07:31:17.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.145+0000 7f4823000640 1 -- 172.21.15.6:0/2157676792 wait complete. 2024-03-06T07:31:17.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.147+0000 7f4823000640 1 Processor -- start 2024-03-06T07:31:17.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.147+0000 7f4823000640 1 -- start start 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4823000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4800003a50 0x7f481c15dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4823000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 0x7f481c1e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4823000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f481c1ea0b0 0x7f481c0c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4823000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0ce600 con 0x7f481c0cb6c0 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4823000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0cf520 con 0x7f481c1ea0b0 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4823000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0ce780 con 0x7f4800003a50 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.148+0000 7f4813400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 0x7f481c1e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 0x7f481c1e9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54104/0 (socket says 172.21.15.6:54104) 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 -- 172.21.15.6:0/1367210911 learned_addr learned my addr 172.21.15.6:0/1367210911 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f481be00640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4800003a50 0x7f481c15dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4821000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f481c1ea0b0 0x7f481c0c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 -- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4800003a50 msgr2=0x7f481c15dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4800003a50 0x7f481c15dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 -- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f481c1ea0b0 msgr2=0x7f481c0c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f481c1ea0b0 0x7f481c0c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 -- 172.21.15.6:0/1367210911 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0cbf60 con 0x7f481c0cb6c0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4821000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f481c1ea0b0 0x7f481c0c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.149+0000 7f4813400640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 0x7f481c1e9b70 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4804011630 tx=0x7f4804011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.150+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480400f920 con 0x7f481c0cb6c0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.150+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480400fac0 con 0x7f481c0cb6c0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.150+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480400fe70 con 0x7f481c0cb6c0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.150+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c1f44c0 con 0x7f481c0cb6c0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.150+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c1f4a80 con 0x7f481c0cb6c0 2024-03-06T07:31:17.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.151+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c04f3a0 con 0x7f481c0cb6c0 2024-03-06T07:31:17.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.152+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4804015460 con 0x7f481c0cb6c0 2024-03-06T07:31:17.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.153+0000 7f4818c00640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f47f806b4d0 0x7f47f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.153+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f480408aa70 con 0x7f481c0cb6c0 2024-03-06T07:31:17.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.154+0000 7f481be00640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f47f806b4d0 0x7f47f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.155+0000 7f481be00640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f47f806b4d0 0x7f47f806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f480c033000 tx=0x7f480c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.158+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4804019030 con 0x7f481c0cb6c0 2024-03-06T07:31:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.349+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f481c0adbf0 con 0x7f481c0cb6c0 2024-03-06T07:31:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.352+0000 7f4818c00640 1 -- 172.21.15.6:0/1367210911 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f48040598f0 con 0x7f481c0cb6c0 2024-03-06T07:31:17.355 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:17.355 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:16.360875+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f47f806b4d0 msgr2=0x7f47f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f47f806b4d0 0x7f47f806d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f480c033000 tx=0x7f480c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 msgr2=0x7f481c1e9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 0x7f481c1e9b70 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4804011630 tx=0x7f4804011af0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 shutdown_connections 2024-03-06T07:31:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f47f806b4d0 0x7f47f806d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f481c1ea0b0 0x7f481c0c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f481c0cb6c0 0x7f481c1e9b70 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 --2- 172.21.15.6:0/1367210911 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4800003a50 0x7f481c15dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 >> 172.21.15.6:0/1367210911 conn(0x7f481c06afb0 msgr2=0x7f481c0ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 shutdown_connections 2024-03-06T07:31:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.355+0000 7f4823000640 1 -- 172.21.15.6:0/1367210911 wait complete. 2024-03-06T07:31:17.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.582+0000 7f01d8e00640 1 Processor -- start 2024-03-06T07:31:17.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.582+0000 7f01d8e00640 1 -- start start 2024-03-06T07:31:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.583+0000 7f01d8e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 0x7f01d40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.583+0000 7f01d8e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f01d40cd060 0x7f01d415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.583+0000 7f01d8e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40714d0 con 0x7f01d4070f10 2024-03-06T07:31:17.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.583+0000 7f01d8e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d40717b0 con 0x7f01d40cd060 2024-03-06T07:31:17.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.583+0000 7f01d8e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d4071a90 con 0x7f01d4145300 2024-03-06T07:31:17.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.584+0000 7f01d3400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 0x7f01d40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.584+0000 7f01d3400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 0x7f01d40b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:51730/0 (socket says 172.21.15.6:51730) 2024-03-06T07:31:17.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.584+0000 7f01d3400640 1 -- 172.21.15.6:0/3800494277 learned_addr learned my addr 172.21.15.6:0/3800494277 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.584+0000 7f01d3e00640 1 --2- 172.21.15.6:0/3800494277 >> v2:172.21.15.178:3300/0 conn(0x7f01d40cd060 0x7f01d415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.585+0000 7f01d3400640 1 -- 172.21.15.6:0/3800494277 >> v2:172.21.15.178:3300/0 conn(0x7f01d40cd060 msgr2=0x7f01d415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.585+0000 7f01d3400640 1 --2- 172.21.15.6:0/3800494277 >> v2:172.21.15.178:3300/0 conn(0x7f01d40cd060 0x7f01d415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.585+0000 7f01d3400640 1 -- 172.21.15.6:0/3800494277 >> v1:172.21.15.178:6789/0 conn(0x7f01d4070f10 legacy=0x7f01d40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.585+0000 7f01d3400640 1 -- 172.21.15.6:0/3800494277 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4071d70 con 0x7f01d4145300 2024-03-06T07:31:17.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.586+0000 7f01d3400640 1 --2- 172.21.15.6:0/3800494277 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 0x7f01d40b49a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009870 tx=0x7f01c402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edc186fec20b145 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d2a00640 1 -- 172.21.15.6:0/3800494277 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4034070 con 0x7f01d4145300 2024-03-06T07:31:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d2a00640 1 -- 172.21.15.6:0/3800494277 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c4030840 con 0x7f01d4145300 2024-03-06T07:31:17.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d2a00640 1 -- 172.21.15.6:0/3800494277 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4030c10 con 0x7f01d4145300 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 -- 172.21.15.6:0/3800494277 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 msgr2=0x7f01d40b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 --2- 172.21.15.6:0/3800494277 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 0x7f01d40b49a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01c4009870 tx=0x7f01c402b980 comp rx=0 tx=0).stop 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 -- 172.21.15.6:0/3800494277 shutdown_connections 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 --2- 172.21.15.6:0/3800494277 >> v2:172.21.15.178:3300/0 conn(0x7f01d40cd060 0x7f01d415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 --2- 172.21.15.6:0/3800494277 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4145300 0x7f01d40b49a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 -- 172.21.15.6:0/3800494277 >> 172.21.15.6:0/3800494277 conn(0x7f01d406b3d0 msgr2=0x7f01d406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 -- 172.21.15.6:0/3800494277 shutdown_connections 2024-03-06T07:31:17.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.587+0000 7f01d8e00640 1 -- 172.21.15.6:0/3800494277 wait complete. 2024-03-06T07:31:17.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.589+0000 7f01d8e00640 1 Processor -- start 2024-03-06T07:31:17.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.589+0000 7f01d8e00640 1 -- start start 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.590+0000 7f01d8e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4070f10 0x7f01d40b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.590+0000 7f01d8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 0x7f01d41e6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.590+0000 7f01d8e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01d4145300 0x7f01d41e6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.590+0000 7f01d8e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d415cab0 con 0x7f01d40cd060 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.590+0000 7f01d8e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d41456e0 con 0x7f01d4145300 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.590+0000 7f01d8e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d415cc30 con 0x7f01d4070f10 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 0x7f01d41e6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 0x7f01d41e6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54112/0 (socket says 172.21.15.6:54112) 2024-03-06T07:31:17.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 -- 172.21.15.6:0/872198103 learned_addr learned my addr 172.21.15.6:0/872198103 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01d3400640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4070f10 0x7f01d40b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01d3e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01d4145300 0x7f01d41e6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 -- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4070f10 msgr2=0x7f01d40b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4070f10 0x7f01d40b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 -- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01d4145300 msgr2=0x7f01d41e6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01d4145300 0x7f01d41e6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.591+0000 7f01cbe00640 1 -- 172.21.15.6:0/872198103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d41f7e90 con 0x7f01d40cd060 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.592+0000 7f01cbe00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 0x7f01d41e6800 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c400 tx=0x7f01bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.592+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc0052d0 con 0x7f01d40cd060 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.592+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc005490 con 0x7f01d40cd060 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.592+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc007260 con 0x7f01d40cd060 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.592+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d4130c70 con 0x7f01d40cd060 2024-03-06T07:31:17.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.592+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d4131230 con 0x7f01d40cd060 2024-03-06T07:31:17.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.593+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d4131720 con 0x7f01d40cd060 2024-03-06T07:31:17.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.594+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01bc007400 con 0x7f01d40cd060 2024-03-06T07:31:17.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.595+0000 7f01cb400640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01b006b4d0 0x7f01b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:17.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.596+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01bc089760 con 0x7f01d40cd060 2024-03-06T07:31:17.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.596+0000 7f01d3400640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01b006b4d0 0x7f01b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:17.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.597+0000 7f01d3400640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01b006b4d0 0x7f01b006d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f01c4036000 tx=0x7f01c403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:17.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.601+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01bc0585d0 con 0x7f01d40cd060 2024-03-06T07:31:17.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.792+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d404f330 con 0x7f01d40cd060 2024-03-06T07:31:17.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.794+0000 7f01cb400640 1 -- 172.21.15.6:0/872198103 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f01bc057ed0 con 0x7f01d40cd060 2024-03-06T07:31:17.797 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:17.797 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:16.360875+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.796+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01b006b4d0 msgr2=0x7f01b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.796+0000 7f01d8e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01b006b4d0 0x7f01b006d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f01c4036000 tx=0x7f01c403b040 comp rx=0 tx=0).stop 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.796+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 msgr2=0x7f01d41e6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.796+0000 7f01d8e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 0x7f01d41e6800 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00c400 tx=0x7f01bc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 shutdown_connections 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f01b006b4d0 0x7f01b006d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f01d4145300 0x7f01d41e6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f01d40cd060 0x7f01d41e6800 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 --2- 172.21.15.6:0/872198103 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f01d4070f10 0x7f01d40b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 >> 172.21.15.6:0/872198103 conn(0x7f01d406b3d0 msgr2=0x7f01d415ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 shutdown_connections 2024-03-06T07:31:17.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:17.797+0000 7f01d8e00640 1 -- 172.21.15.6:0/872198103 wait complete. 2024-03-06T07:31:17.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:31:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.024+0000 7f18cb200640 1 Processor -- start 2024-03-06T07:31:18.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.024+0000 7f18cb200640 1 -- start start 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18cb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18c415dee0 0x7f18c40d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18cb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c40d0d40 0x7f18c40b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18cb200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c40ceca0 con 0x7f18c40d4860 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18cb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c40cf5d0 con 0x7f18c415dee0 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18cb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40cef80 con 0x7f18c40d0d40 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18c9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18c415dee0 0x7f18c40d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18c9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18c415dee0 0x7f18c40d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54114/0 (socket says 172.21.15.6:54114) 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.026+0000 7f18c9e00640 1 -- 172.21.15.6:0/2564024801 learned_addr learned my addr 172.21.15.6:0/2564024801 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.027+0000 7f18ca800640 1 --2- 172.21.15.6:0/2564024801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c40d0d40 0x7f18c40b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.027+0000 7f18c9e00640 1 -- 172.21.15.6:0/2564024801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c40d0d40 msgr2=0x7f18c40b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.027+0000 7f18c9e00640 1 --2- 172.21.15.6:0/2564024801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c40d0d40 0x7f18c40b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.027+0000 7f18c9e00640 1 -- 172.21.15.6:0/2564024801 >> v1:172.21.15.178:6789/0 conn(0x7f18c40d4860 legacy=0x7f18c40d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.027+0000 7f18c9e00640 1 -- 172.21.15.6:0/2564024801 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c40cf260 con 0x7f18c415dee0 2024-03-06T07:31:18.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.028+0000 7f18c9e00640 1 --2- 172.21.15.6:0/2564024801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c415dee0 0x7f18c40d18a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f18c408c6f0 tx=0x7f18b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa2aa6da23c525f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.029+0000 7f18c8a00640 1 -- 172.21.15.6:0/2564024801 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8035020 con 0x7f18c415dee0 2024-03-06T07:31:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.029+0000 7f18c8a00640 1 -- 172.21.15.6:0/2564024801 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b8031830 con 0x7f18c415dee0 2024-03-06T07:31:18.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.029+0000 7f18c8a00640 1 -- 172.21.15.6:0/2564024801 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8031c00 con 0x7f18c415dee0 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 -- 172.21.15.6:0/2564024801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c415dee0 msgr2=0x7f18c40d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 --2- 172.21.15.6:0/2564024801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c415dee0 0x7f18c40d18a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f18c408c6f0 tx=0x7f18b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 -- 172.21.15.6:0/2564024801 shutdown_connections 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 --2- 172.21.15.6:0/2564024801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c40d0d40 0x7f18c40b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 --2- 172.21.15.6:0/2564024801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c415dee0 0x7f18c40d18a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 -- 172.21.15.6:0/2564024801 >> 172.21.15.6:0/2564024801 conn(0x7f18c40d3cd0 msgr2=0x7f18c40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 -- 172.21.15.6:0/2564024801 shutdown_connections 2024-03-06T07:31:18.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.030+0000 7f18cb200640 1 -- 172.21.15.6:0/2564024801 wait complete. 2024-03-06T07:31:18.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.032+0000 7f18cb200640 1 Processor -- start 2024-03-06T07:31:18.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.033+0000 7f18cb200640 1 -- start start 2024-03-06T07:31:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18cb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18c40d0d40 0x7f18c414c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18cb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 0x7f18c41e8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18cb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c415dee0 0x7f18c40c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18cb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40d1de0 con 0x7f18c40d4860 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18cb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c415e2c0 con 0x7f18c40d0d40 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18cb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40d1f60 con 0x7f18c415dee0 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 0x7f18c41e8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18c9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 0x7f18c41e8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54116/0 (socket says 172.21.15.6:54116) 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18c9400640 1 -- 172.21.15.6:0/3406818957 learned_addr learned my addr 172.21.15.6:0/3406818957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.034+0000 7f18c9e00640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18c40d0d40 0x7f18c414c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18ca800640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c415dee0 0x7f18c40c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18c9400640 1 -- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c415dee0 msgr2=0x7f18c40c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18c9400640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c415dee0 0x7f18c40c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18c9400640 1 -- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18c40d0d40 msgr2=0x7f18c414c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18c9400640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18c40d0d40 0x7f18c414c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18c9400640 1 -- 172.21.15.6:0/3406818957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c41e79e0 con 0x7f18c40d4860 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18ca800640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c415dee0 0x7f18c40c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18c9400640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 0x7f18c41e8a80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f18b400b3f0 tx=0x7f18b400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4015020 con 0x7f18c40d4860 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b4011830 con 0x7f18c40d4860 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c41e7cc0 con 0x7f18c40d4860 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b4011c20 con 0x7f18c40d4860 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.035+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c41e8280 con 0x7f18c40d4860 2024-03-06T07:31:18.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.037+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c404f330 con 0x7f18c40d4860 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.037+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18b4018020 con 0x7f18c40d4860 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.038+0000 7f18be000640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18a806b4d0 0x7f18a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.038+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18b4089d60 con 0x7f18c40d4860 2024-03-06T07:31:18.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.039+0000 7f18c9e00640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18a806b4d0 0x7f18a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.040+0000 7f18c9e00640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18a806b4d0 0x7f18a806d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f18b8036000 tx=0x7f18b803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.043+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18b401f030 con 0x7f18c40d4860 2024-03-06T07:31:18.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:31:18.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.302+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f18c408d7d0 con 0x7f18c40d4860 2024-03-06T07:31:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.929+0000 7f18be000640 1 -- 172.21.15.6:0/3406818957 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f18b40172c0 con 0x7f18c40d4860 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.932+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18a806b4d0 msgr2=0x7f18a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18a806b4d0 0x7f18a806d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f18b8036000 tx=0x7f18b803c040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 msgr2=0x7f18c41e8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 0x7f18c41e8a80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f18b400b3f0 tx=0x7f18b400b8b0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 shutdown_connections 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18a806b4d0 0x7f18a806d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18c415dee0 0x7f18c40c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18c40d4860 0x7f18c41e8a80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 --2- 172.21.15.6:0/3406818957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18c40d0d40 0x7f18c414c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 >> 172.21.15.6:0/3406818957 conn(0x7f18c40d3cd0 msgr2=0x7f18c40d2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 shutdown_connections 2024-03-06T07:31:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:18.933+0000 7f18cb200640 1 -- 172.21.15.6:0/3406818957 wait complete. 2024-03-06T07:31:18.955 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:31:18.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:31:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.176+0000 7f862f200640 1 Processor -- start 2024-03-06T07:31:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.176+0000 7f862f200640 1 -- start start 2024-03-06T07:31:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.177+0000 7f862f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f86280d24f0 0x7f8628168dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280b2c00 con 0x7f86280d4890 2024-03-06T07:31:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8628161ae0 con 0x7f8628161700 2024-03-06T07:31:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86280b56e0 con 0x7f86280d24f0 2024-03-06T07:31:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862d400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86280d4890 0x7f86280d1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39000/0 (socket says 172.21.15.6:39000) 2024-03-06T07:31:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862d400640 1 -- 172.21.15.6:0/3078466064 learned_addr learned my addr 172.21.15.6:0/3078466064 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:19.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862e800640 1 --2- 172.21.15.6:0/3078466064 >> v2:172.21.15.178:3300/0 conn(0x7f86280d24f0 0x7f8628168dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.178+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557824618 0 0) 0x7f86280b2c00 con 0x7f86280d4890 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8610003520 con 0x7f86280d4890 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426304587 0 0) 0x7f8610003520 con 0x7f86280d4890 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 >> v2:172.21.15.178:3300/0 conn(0x7f86280d24f0 msgr2=0x7f8628168dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 --2- 172.21.15.6:0/3078466064 >> v2:172.21.15.178:3300/0 conn(0x7f86280d24f0 0x7f8628168dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 >> v1:172.21.15.178:6789/0 conn(0x7f8628161700 legacy=0x7f86280b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280b2ea0 con 0x7f86280d4890 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8618001670 con 0x7f86280d4890 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 >> v1:172.21.15.6:6789/0 conn(0x7f86280d4890 legacy=0x7f86280d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86100039b0 0x7f8610003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f8610004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86280b56e0 con 0x7f8610004800 2024-03-06T07:31:19.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861000c060 con 0x7f86100051c0 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861000c260 con 0x7f86100039b0 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.179+0000 7f862d400640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f8610004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862e800640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862d400640 1 -- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86100039b0 msgr2=0x7f8610003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862de00640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86100039b0 0x7f8610003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862d400640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86100039b0 0x7f8610003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862d400640 1 -- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86100051c0 msgr2=0x7f861000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862d400640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862d400640 1 -- 172.21.15.6:0/3078466064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280b2ea0 con 0x7f8610004800 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862e800640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86100051c0 0x7f861000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862e800640 1 -- 172.21.15.6:0/3078466064 reap_dead start 2024-03-06T07:31:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.180+0000 7f862de00640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86100039b0 0x7f8610003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.181+0000 7f862d400640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f8610004c80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8618001050 tx=0x7f8610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd946e46415cddf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.182+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8618007d10 con 0x7f8610004800 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.182+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8618030070 con 0x7f8610004800 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.182+0000 7f862ca00640 1 -- 172.21.15.6:0/3078466064 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861802cd80 con 0x7f8610004800 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.182+0000 7f862f200640 1 -- 172.21.15.6:0/3078466064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 msgr2=0x7f8610004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.182+0000 7f862f200640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f8610004c80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8618001050 tx=0x7f8610005830 comp rx=0 tx=0).stop 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.183+0000 7f862f200640 1 -- 172.21.15.6:0/3078466064 shutdown_connections 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.183+0000 7f862f200640 1 --2- 172.21.15.6:0/3078466064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f8610004c80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.183+0000 7f862f200640 1 -- 172.21.15.6:0/3078466064 >> 172.21.15.6:0/3078466064 conn(0x7f86280d3d70 msgr2=0x7f86280d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.183+0000 7f862f200640 1 -- 172.21.15.6:0/3078466064 shutdown_connections 2024-03-06T07:31:19.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.183+0000 7f862f200640 1 -- 172.21.15.6:0/3078466064 wait complete. 2024-03-06T07:31:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.185+0000 7f862f200640 1 Processor -- start 2024-03-06T07:31:19.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.185+0000 7f862f200640 1 -- start start 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f862815c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8628147970 0x7f862815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86281f6de0 0x7f86281f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86280b0d00 con 0x7f8610004800 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86280b0a20 con 0x7f8628147970 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86280d1de0 con 0x7f86281f6de0 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f862815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f862815c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54126/0 (socket says 172.21.15.6:54126) 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862de00640 1 -- 172.21.15.6:0/1334798204 learned_addr learned my addr 172.21.15.6:0/1334798204 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862e800640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86281f6de0 0x7f86281f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.186+0000 7f862d400640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8628147970 0x7f862815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862de00640 1 -- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86281f6de0 msgr2=0x7f86281f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862de00640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86281f6de0 0x7f86281f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862de00640 1 -- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8628147970 msgr2=0x7f862815cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862de00640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8628147970 0x7f862815cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862de00640 1 -- 172.21.15.6:0/1334798204 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86281f77a0 con 0x7f8610004800 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862e800640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86281f6de0 0x7f86281f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862d400640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8628147970 0x7f862815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862de00640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f862815c880 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f861c00d400 tx=0x7f861c00d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c018070 con 0x7f8610004800 2024-03-06T07:31:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c003610 con 0x7f8610004800 2024-03-06T07:31:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c003a00 con 0x7f8610004800 2024-03-06T07:31:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.187+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86281f7a60 con 0x7f8610004800 2024-03-06T07:31:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.188+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8628147190 con 0x7f8610004800 2024-03-06T07:31:19.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.189+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862804f330 con 0x7f8610004800 2024-03-06T07:31:19.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.190+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f861c01b070 con 0x7f8610004800 2024-03-06T07:31:19.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.191+0000 7f8622000640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f860c06b500 0x7f860c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.192+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f861c089880 con 0x7f8610004800 2024-03-06T07:31:19.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.192+0000 7f862d400640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f860c06b500 0x7f860c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.193+0000 7f862d400640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f860c06b500 0x7f860c06d9c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8618034000 tx=0x7f8618009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.195+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f861c058140 con 0x7f8610004800 2024-03-06T07:31:19.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.434+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8628074590 con 0x7f8610004800 2024-03-06T07:31:19.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.938+0000 7f8622000640 1 -- 172.21.15.6:0/1334798204 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7f861c05c1e0 con 0x7f8610004800 2024-03-06T07:31:19.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f860c06b500 msgr2=0x7f860c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f860c06b500 0x7f860c06d9c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8618034000 tx=0x7f8618009030 comp rx=0 tx=0).stop 2024-03-06T07:31:19.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 msgr2=0x7f862815c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f862815c880 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f861c00d400 tx=0x7f861c00d8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 shutdown_connections 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f860c06b500 0x7f860c06d9c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8628147970 0x7f862815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86281f6de0 0x7f86281f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 --2- 172.21.15.6:0/1334798204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8610004800 0x7f862815c880 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 >> 172.21.15.6:0/1334798204 conn(0x7f86280d3d70 msgr2=0x7f86280b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 shutdown_connections 2024-03-06T07:31:19.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:19.942+0000 7f862f200640 1 -- 172.21.15.6:0/1334798204 wait complete. 2024-03-06T07:31:19.965 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:31:19.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.162+0000 7f100ae00640 1 Processor -- start 2024-03-06T07:31:20.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.162+0000 7f100ae00640 1 -- start start 2024-03-06T07:31:20.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.164+0000 7f100ae00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f10040d4a10 0x7f10040d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.164+0000 7f100ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d0c80 0x7f10040b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.164+0000 7f100ae00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10040b3710 con 0x7f100415e0b0 2024-03-06T07:31:20.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.164+0000 7f100ae00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10040b09e0 con 0x7f10040d4a10 2024-03-06T07:31:20.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.164+0000 7f100ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10040cfa20 con 0x7f10040d0c80 2024-03-06T07:31:20.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f100a400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d0c80 0x7f10040b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f1009a00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f100415e0b0 0x7f10040cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:50310/0 (socket says 172.21.15.6:50310) 2024-03-06T07:31:20.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f1009a00640 1 -- 172.21.15.6:0/3573675155 learned_addr learned my addr 172.21.15.6:0/3573675155 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:20.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f1009000640 1 --2- 172.21.15.6:0/3573675155 >> v2:172.21.15.178:3300/0 conn(0x7f10040d4a10 0x7f10040d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f100a400640 1 -- 172.21.15.6:0/3573675155 >> v2:172.21.15.178:3300/0 conn(0x7f10040d4a10 msgr2=0x7f10040d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:20.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f100a400640 1 --2- 172.21.15.6:0/3573675155 >> v2:172.21.15.178:3300/0 conn(0x7f10040d4a10 0x7f10040d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f1003e00640 1 -- 172.21.15.6:0/3573675155 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731045379 0 0) 0x7f10040b3710 con 0x7f100415e0b0 2024-03-06T07:31:20.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f100a400640 1 -- 172.21.15.6:0/3573675155 >> v1:172.21.15.178:6789/0 conn(0x7f100415e0b0 legacy=0x7f10040cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.165+0000 7f100a400640 1 -- 172.21.15.6:0/3573675155 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10040b39f0 con 0x7f10040d0c80 2024-03-06T07:31:20.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.167+0000 7f100a400640 1 --2- 172.21.15.6:0/3573675155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d0c80 0x7f10040b44d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009d90 tx=0x7f0ffc02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e88bc73e3f601297 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.167+0000 7f1003e00640 1 -- 172.21.15.6:0/3573675155 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc039070 con 0x7f10040d0c80 2024-03-06T07:31:20.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.167+0000 7f1003e00640 1 -- 172.21.15.6:0/3573675155 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc02cc60 con 0x7f10040d0c80 2024-03-06T07:31:20.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.167+0000 7f1003e00640 1 -- 172.21.15.6:0/3573675155 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc034660 con 0x7f10040d0c80 2024-03-06T07:31:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 -- 172.21.15.6:0/3573675155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d0c80 msgr2=0x7f10040b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 --2- 172.21.15.6:0/3573675155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d0c80 0x7f10040b44d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009d90 tx=0x7f0ffc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 -- 172.21.15.6:0/3573675155 shutdown_connections 2024-03-06T07:31:20.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 --2- 172.21.15.6:0/3573675155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d0c80 0x7f10040b44d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 --2- 172.21.15.6:0/3573675155 >> v2:172.21.15.178:3300/0 conn(0x7f10040d4a10 0x7f10040d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 -- 172.21.15.6:0/3573675155 >> 172.21.15.6:0/3573675155 conn(0x7f10040d3d30 msgr2=0x7f10040d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 -- 172.21.15.6:0/3573675155 shutdown_connections 2024-03-06T07:31:20.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.168+0000 7f100ae00640 1 -- 172.21.15.6:0/3573675155 wait complete. 2024-03-06T07:31:20.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.170+0000 7f100ae00640 1 Processor -- start 2024-03-06T07:31:20.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.170+0000 7f100ae00640 1 -- start start 2024-03-06T07:31:20.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.171+0000 7f100ae00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10040d0c80 0x7f1004160040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.171+0000 7f100ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d4a10 0x7f10041f4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.171+0000 7f100ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 0x7f1004160c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.171+0000 7f100ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100415e490 con 0x7f100415e0b0 2024-03-06T07:31:20.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.171+0000 7f100ae00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10040b3fb0 con 0x7f10040d0c80 2024-03-06T07:31:20.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.171+0000 7f100ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10040cfc60 con 0x7f10040d4a10 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 0x7f1004160c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 0x7f1004160c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54142/0 (socket says 172.21.15.6:54142) 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 -- 172.21.15.6:0/1293862688 learned_addr learned my addr 172.21.15.6:0/1293862688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f1009a00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10040d0c80 0x7f1004160040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f1009000640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d4a10 0x7f10041f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 -- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d4a10 msgr2=0x7f10041f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d4a10 0x7f10041f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 -- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10040d0c80 msgr2=0x7f1004160040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10040d0c80 0x7f1004160040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.172+0000 7f100a400640 1 -- 172.21.15.6:0/1293862688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1004160580 con 0x7f100415e0b0 2024-03-06T07:31:20.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.173+0000 7f100a400640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 0x7f1004160c00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009d90 tx=0x7f0ffc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.173+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc044020 con 0x7f100415e0b0 2024-03-06T07:31:20.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.173+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ffc03a040 con 0x7f100415e0b0 2024-03-06T07:31:20.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.173+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10041f4780 con 0x7f100415e0b0 2024-03-06T07:31:20.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.173+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc039050 con 0x7f100415e0b0 2024-03-06T07:31:20.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.173+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10040cb410 con 0x7f100415e0b0 2024-03-06T07:31:20.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.174+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f100404f330 con 0x7f100415e0b0 2024-03-06T07:31:20.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.175+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ffc003bb0 con 0x7f100415e0b0 2024-03-06T07:31:20.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.177+0000 7f1001600640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0fe406b4d0 0x7f0fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.177+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ffc0ac900 con 0x7f100415e0b0 2024-03-06T07:31:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.177+0000 7f1009a00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0fe406b4d0 0x7f0fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.179+0000 7f1009a00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0fe406b4d0 0x7f0fe406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8000f60 tx=0x7f0ff8007d20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.186+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ffc04e060 con 0x7f100415e0b0 2024-03-06T07:31:20.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.424+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10040008d0 con 0x7f100415e0b0 2024-03-06T07:31:20.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.425+0000 7f1001600640 1 -- 172.21.15.6:0/1293862688 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f0ffc0ac090 con 0x7f100415e0b0 2024-03-06T07:31:20.428 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:20.428 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:20.428 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0fe406b4d0 msgr2=0x7f0fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0fe406b4d0 0x7f0fe406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8000f60 tx=0x7f0ff8007d20 comp rx=0 tx=0).stop 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 msgr2=0x7f1004160c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 0x7f1004160c00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc009d90 tx=0x7f0ffc004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 shutdown_connections 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0fe406b4d0 0x7f0fe406d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100415e0b0 0x7f1004160c00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10040d4a10 0x7f10041f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 --2- 172.21.15.6:0/1293862688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10040d0c80 0x7f1004160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.427+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 >> 172.21.15.6:0/1293862688 conn(0x7f10040d3d30 msgr2=0x7f10040b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.428+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 shutdown_connections 2024-03-06T07:31:20.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.428+0000 7f100ae00640 1 -- 172.21.15.6:0/1293862688 wait complete. 2024-03-06T07:31:20.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:20.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.660+0000 7faee7e00640 1 Processor -- start 2024-03-06T07:31:20.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.660+0000 7faee7e00640 1 -- start start 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faee7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faee806ed90 0x7faee806bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faee7e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee80afb40 con 0x7faee806c4c0 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faee7e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee80afe20 con 0x7faee815c360 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faee7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee806c950 con 0x7faee806ed90 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faedfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faee806ed90 0x7faee806bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faee7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faee806c4c0 0x7faee815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39004/0 (socket says 172.21.15.6:39004) 2024-03-06T07:31:20.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.662+0000 7faee7400640 1 -- 172.21.15.6:0/2724677928 learned_addr learned my addr 172.21.15.6:0/2724677928 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.663+0000 7faedfe00640 1 -- 172.21.15.6:0/2724677928 >> v1:172.21.15.178:6789/0 conn(0x7faee815c360 legacy=0x7faee80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.663+0000 7faedfe00640 1 -- 172.21.15.6:0/2724677928 >> v1:172.21.15.6:6789/0 conn(0x7faee806c4c0 legacy=0x7faee815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.663+0000 7faedfe00640 1 -- 172.21.15.6:0/2724677928 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee806d560 con 0x7faee806ed90 2024-03-06T07:31:20.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.663+0000 7faee6000640 1 -- 172.21.15.6:0/2724677928 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461762980 0 0) 0x7faee80afb40 con 0x7faee806c4c0 2024-03-06T07:31:20.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.664+0000 7faedfe00640 1 --2- 172.21.15.6:0/2724677928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806ed90 0x7faee806bf80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7faed00097a0 tx=0x7faed002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283d475a7d3a9c3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee6000640 1 -- 172.21.15.6:0/2724677928 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0035020 con 0x7faee806ed90 2024-03-06T07:31:20.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee6000640 1 -- 172.21.15.6:0/2724677928 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0031830 con 0x7faee806ed90 2024-03-06T07:31:20.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee6000640 1 -- 172.21.15.6:0/2724677928 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0031c00 con 0x7faee806ed90 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 -- 172.21.15.6:0/2724677928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806ed90 msgr2=0x7faee806bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 --2- 172.21.15.6:0/2724677928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806ed90 0x7faee806bf80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7faed00097a0 tx=0x7faed002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 -- 172.21.15.6:0/2724677928 shutdown_connections 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 --2- 172.21.15.6:0/2724677928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806ed90 0x7faee806bf80 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 -- 172.21.15.6:0/2724677928 >> 172.21.15.6:0/2724677928 conn(0x7faee80ccf70 msgr2=0x7faee80cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 -- 172.21.15.6:0/2724677928 shutdown_connections 2024-03-06T07:31:20.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.665+0000 7faee7e00640 1 -- 172.21.15.6:0/2724677928 wait complete. 2024-03-06T07:31:20.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.667+0000 7faee7e00640 1 Processor -- start 2024-03-06T07:31:20.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.667+0000 7faee7e00640 1 -- start start 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 0x7faee815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee7e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee806ed90 0x7faee815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faee815c360 0x7faee80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee806d840 con 0x7faee806c4c0 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee7e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee806d310 con 0x7faee806ed90 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee80cea80 con 0x7faee815c360 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 0x7faee815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 0x7faee815c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54166/0 (socket says 172.21.15.6:54166) 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.668+0000 7faee6a00640 1 -- 172.21.15.6:0/30488945 learned_addr learned my addr 172.21.15.6:0/30488945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faedfe00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee806ed90 0x7faee815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee7400640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faee815c360 0x7faee80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee6a00640 1 -- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faee815c360 msgr2=0x7faee80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee6a00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faee815c360 0x7faee80b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee6a00640 1 -- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee806ed90 msgr2=0x7faee815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee6a00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee806ed90 0x7faee815ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee6a00640 1 -- 172.21.15.6:0/30488945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee8150be0 con 0x7faee806c4c0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.669+0000 7faee6a00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 0x7faee815c740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7faed800c400 tx=0x7faed800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.670+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8017070 con 0x7faee806c4c0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.670+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed80052d0 con 0x7faee806c4c0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.670+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee8150ec0 con 0x7faee806c4c0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.670+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8003830 con 0x7faee806c4c0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.670+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee8151760 con 0x7faee806c4c0 2024-03-06T07:31:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.671+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee81512b0 con 0x7faee806c4c0 2024-03-06T07:31:20.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.672+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faed801a070 con 0x7faee806c4c0 2024-03-06T07:31:20.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.673+0000 7faedea00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:20.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.673+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faed8089560 con 0x7faee806c4c0 2024-03-06T07:31:20.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.674+0000 7faedfe00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:20.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.675+0000 7faedfe00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faec406b4d0 0x7faec406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7faed0036000 tx=0x7faed003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:20.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.677+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faed80583e0 con 0x7faee806c4c0 2024-03-06T07:31:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.877+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee8151090 con 0x7faee806c4c0 2024-03-06T07:31:20.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.880+0000 7faedea00640 1 -- 172.21.15.6:0/30488945 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16596 (secure 0 0 0) 0x7faed8057ce0 con 0x7faee806c4c0 2024-03-06T07:31:20.882 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:20.882 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:17.916537+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:20.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faec406b4d0 msgr2=0x7faec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faec406b4d0 0x7faec406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7faed0036000 tx=0x7faed003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 msgr2=0x7faee815c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 0x7faee815c740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7faed800c400 tx=0x7faed800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 shutdown_connections 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faec406b4d0 0x7faec406d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faee806ed90 0x7faee815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faee815c360 0x7faee80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 --2- 172.21.15.6:0/30488945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faee806c4c0 0x7faee815c740 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 >> 172.21.15.6:0/30488945 conn(0x7faee80ccf70 msgr2=0x7faee80af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 shutdown_connections 2024-03-06T07:31:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:20.882+0000 7faee7e00640 1 -- 172.21.15.6:0/30488945 wait complete. 2024-03-06T07:31:20.904 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:31:20.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.119+0000 7fd32cc00640 1 Processor -- start 2024-03-06T07:31:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.119+0000 7fd32cc00640 1 -- start start 2024-03-06T07:31:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.121+0000 7fd32cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd32806c6d0 0x7fd328162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.121+0000 7fd32cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd328161a80 con 0x7fd32815e0e0 2024-03-06T07:31:21.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.121+0000 7fd32cc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32815efa0 con 0x7fd32815dd00 2024-03-06T07:31:21.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.121+0000 7fd32cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32815f280 con 0x7fd32806c6d0 2024-03-06T07:31:21.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd31fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd32806c6d0 0x7fd328162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd326800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd32815e0e0 0x7fd32815e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39010/0 (socket says 172.21.15.6:39010) 2024-03-06T07:31:21.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd326800640 1 -- 172.21.15.6:0/1646227911 learned_addr learned my addr 172.21.15.6:0/1646227911 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:21.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd325400640 1 -- 172.21.15.6:0/1646227911 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982641152 0 0) 0x7fd328161a80 con 0x7fd32815e0e0 2024-03-06T07:31:21.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd31fe00640 1 -- 172.21.15.6:0/1646227911 >> v1:172.21.15.178:6789/0 conn(0x7fd32815dd00 legacy=0x7fd32806bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd31fe00640 1 -- 172.21.15.6:0/1646227911 >> v1:172.21.15.6:6789/0 conn(0x7fd32815e0e0 legacy=0x7fd32815e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.122+0000 7fd31fe00640 1 -- 172.21.15.6:0/1646227911 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd328161d20 con 0x7fd32806c6d0 2024-03-06T07:31:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.124+0000 7fd31fe00640 1 --2- 172.21.15.6:0/1646227911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32806c6d0 0x7fd328162520 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd3180097a0 tx=0x7fd31802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e395388b25b2eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.124+0000 7fd325400640 1 -- 172.21.15.6:0/1646227911 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318035020 con 0x7fd32806c6d0 2024-03-06T07:31:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.124+0000 7fd325400640 1 -- 172.21.15.6:0/1646227911 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318031830 con 0x7fd32806c6d0 2024-03-06T07:31:21.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.124+0000 7fd325400640 1 -- 172.21.15.6:0/1646227911 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318031c00 con 0x7fd32806c6d0 2024-03-06T07:31:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 -- 172.21.15.6:0/1646227911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32806c6d0 msgr2=0x7fd328162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1646227911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32806c6d0 0x7fd328162520 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd3180097a0 tx=0x7fd31802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 -- 172.21.15.6:0/1646227911 shutdown_connections 2024-03-06T07:31:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1646227911 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32806c6d0 0x7fd328162520 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 -- 172.21.15.6:0/1646227911 >> 172.21.15.6:0/1646227911 conn(0x7fd3280714d0 msgr2=0x7fd3280718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 -- 172.21.15.6:0/1646227911 shutdown_connections 2024-03-06T07:31:21.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.125+0000 7fd32cc00640 1 -- 172.21.15.6:0/1646227911 wait complete. 2024-03-06T07:31:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.127+0000 7fd32cc00640 1 Processor -- start 2024-03-06T07:31:21.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.127+0000 7fd32cc00640 1 -- start start 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd32cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd32806c6d0 0x7fd328150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd32cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd32815dd00 0x7fd3281e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd32cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 0x7fd3281e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd32cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32806f280 con 0x7fd32815e0e0 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd32cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32815ec40 con 0x7fd32815dd00 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd32cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32806d250 con 0x7fd32806c6d0 2024-03-06T07:31:21.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd326800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 0x7fd3281e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd326800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 0x7fd3281e8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54180/0 (socket says 172.21.15.6:54180) 2024-03-06T07:31:21.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd326800640 1 -- 172.21.15.6:0/1687821097 learned_addr learned my addr 172.21.15.6:0/1687821097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.128+0000 7fd31fe00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd32815dd00 0x7fd3281e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd325e00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd32806c6d0 0x7fd328150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd326800640 1 -- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd32806c6d0 msgr2=0x7fd328150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd326800640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd32806c6d0 0x7fd328150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd326800640 1 -- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd32815dd00 msgr2=0x7fd3281e8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd326800640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd32815dd00 0x7fd3281e8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd326800640 1 -- 172.21.15.6:0/1687821097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3281e7650 con 0x7fd32815e0e0 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd325e00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd32806c6d0 0x7fd328150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd326800640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 0x7fd3281e8f80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd32000c700 tx=0x7fd32000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320018070 con 0x7fd32815e0e0 2024-03-06T07:31:21.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32000ce10 con 0x7fd32815e0e0 2024-03-06T07:31:21.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32000a860 con 0x7fd32815e0e0 2024-03-06T07:31:21.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.129+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3281e7930 con 0x7fd32815e0e0 2024-03-06T07:31:21.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.130+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3281e7ef0 con 0x7fd32815e0e0 2024-03-06T07:31:21.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.131+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32804f3a0 con 0x7fd32815e0e0 2024-03-06T07:31:21.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.132+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd32000aa00 con 0x7fd32815e0e0 2024-03-06T07:31:21.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.132+0000 7fd31e000640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd30406b4d0 0x7fd30406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.133+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd320089930 con 0x7fd32815e0e0 2024-03-06T07:31:21.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.133+0000 7fd325e00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd30406b4d0 0x7fd30406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.134+0000 7fd325e00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd30406b4d0 0x7fd30406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd310000f60 tx=0x7fd310009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.137+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd320058680 con 0x7fd32815e0e0 2024-03-06T07:31:21.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.470+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3280cf530 con 0x7fd30406b4d0 2024-03-06T07:31:21.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.488+0000 7fd31e000640 1 -- 172.21.15.6:0/1687821097 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247996 (secure 0 0 0) 0x7fd3280cf530 con 0x7fd30406b4d0 2024-03-06T07:31:21.494 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:21.496 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:31:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd30406b4d0 msgr2=0x7fd30406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd30406b4d0 0x7fd30406d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd310000f60 tx=0x7fd310009040 comp rx=0 tx=0).stop 2024-03-06T07:31:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 msgr2=0x7fd3281e8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 0x7fd3281e8f80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd32000c700 tx=0x7fd32000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 shutdown_connections 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd30406b4d0 0x7fd30406d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd32815dd00 0x7fd3281e8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd32815e0e0 0x7fd3281e8f80 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 --2- 172.21.15.6:0/1687821097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd32806c6d0 0x7fd328150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 >> 172.21.15.6:0/1687821097 conn(0x7fd3280714d0 msgr2=0x7fd32806bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.494+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 shutdown_connections 2024-03-06T07:31:21.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.495+0000 7fd32cc00640 1 -- 172.21.15.6:0/1687821097 wait complete. 2024-03-06T07:31:21.518 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-03-06T07:31:19.520997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592014,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":168,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265212,"kb_used_data":7440,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754709508,"statfs":{"total":773094113280,"available":772822536192,"internally_reserved":0,"allocated":7618560,"data_stored":3853242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1646,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.647151"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.909178+0000","last_change":"2024-03-06T07:31:15.909178+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.909178+0000","last_undegraded":"2024-03-06T07:31:15.909178+0000","last_fullsized":"2024-03-06T07:31:15.909178+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.897034+0000","last_change":"2024-03-06T07:31:15.897034+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.897034+0000","last_undegraded":"2024-03-06T07:31:15.897034+0000","last_fullsized":"2024-03-06T07:31:15.897034+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.889150+0000","last_change":"2024-03-06T07:31:15.889150+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.889150+0000","last_undegraded":"2024-03-06T07:31:15.889150+0000","last_fullsized":"2024-03-06T07:31:15.889150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.941593+0000","last_change":"2024-03-06T07:31:16.940027+0000","last_active":"2024-03-06T07:31:16.941593+0000","last_peered":"2024-03-06T07:31:16.941593+0000","last_clean":"2024-03-06T07:31:16.941593+0000","last_became_active":"2024-03-06T07:31:16.939339+0000","last_became_peered":"2024-03-06T07:31:16.939339+0000","last_unstale":"2024-03-06T07:31:16.941593+0000","last_undegraded":"2024-03-06T07:31:16.941593+0000","last_fullsized":"2024-03-06T07:31:16.941593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:53.522887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.940588+0000","last_change":"2024-03-06T07:31:16.934162+0000","last_active":"2024-03-06T07:31:16.940588+0000","last_peered":"2024-03-06T07:31:16.940588+0000","last_clean":"2024-03-06T07:31:16.940588+0000","last_became_active":"2024-03-06T07:31:16.933866+0000","last_became_peered":"2024-03-06T07:31:16.933866+0000","last_unstale":"2024-03-06T07:31:16.940588+0000","last_undegraded":"2024-03-06T07:31:16.940588+0000","last_fullsized":"2024-03-06T07:31:16.940588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:24.805790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.888498+0000","last_change":"2024-03-06T07:31:15.888498+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.888498+0000","last_undegraded":"2024-03-06T07:31:15.888498+0000","last_fullsized":"2024-03-06T07:31:15.888498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.905576+0000","last_change":"2024-03-06T07:31:15.905576+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.905576+0000","last_undegraded":"2024-03-06T07:31:15.905576+0000","last_fullsized":"2024-03-06T07:31:15.905576+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.889957+0000","last_change":"2024-03-06T07:31:15.889957+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.889957+0000","last_undegraded":"2024-03-06T07:31:15.889957+0000","last_fullsized":"2024-03-06T07:31:15.889957+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.935158+0000","last_change":"2024-03-06T07:31:16.931065+0000","last_active":"2024-03-06T07:31:16.935158+0000","last_peered":"2024-03-06T07:31:16.935158+0000","last_clean":"2024-03-06T07:31:16.935158+0000","last_became_active":"2024-03-06T07:31:16.930774+0000","last_became_peered":"2024-03-06T07:31:16.930774+0000","last_unstale":"2024-03-06T07:31:16.935158+0000","last_undegraded":"2024-03-06T07:31:16.935158+0000","last_fullsized":"2024-03-06T07:31:16.935158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:12.934078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.897280+0000","last_change":"2024-03-06T07:31:15.897280+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.897280+0000","last_undegraded":"2024-03-06T07:31:15.897280+0000","last_fullsized":"2024-03-06T07:31:15.897280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.909362+0000","last_change":"2024-03-06T07:31:15.909362+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.909362+0000","last_undegraded":"2024-03-06T07:31:15.909362+0000","last_fullsized":"2024-03-06T07:31:15.909362+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.935821+0000","last_change":"2024-03-06T07:31:16.922685+0000","last_active":"2024-03-06T07:31:16.935821+0000","last_peered":"2024-03-06T07:31:16.935821+0000","last_clean":"2024-03-06T07:31:16.935821+0000","last_became_active":"2024-03-06T07:31:16.922381+0000","last_became_peered":"2024-03-06T07:31:16.922381+0000","last_unstale":"2024-03-06T07:31:16.935821+0000","last_undegraded":"2024-03-06T07:31:16.935821+0000","last_fullsized":"2024-03-06T07:31:16.935821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:54.935260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.886642+0000","last_change":"2024-03-06T07:31:15.886642+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.886642+0000","last_undegraded":"2024-03-06T07:31:15.886642+0000","last_fullsized":"2024-03-06T07:31:15.886642+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.942704+0000","last_change":"2024-03-06T07:31:16.940311+0000","last_active":"2024-03-06T07:31:16.942704+0000","last_peered":"2024-03-06T07:31:16.942704+0000","last_clean":"2024-03-06T07:31:16.942704+0000","last_became_active":"2024-03-06T07:31:16.939993+0000","last_became_peered":"2024-03-06T07:31:16.939993+0000","last_unstale":"2024-03-06T07:31:16.942704+0000","last_undegraded":"2024-03-06T07:31:16.942704+0000","last_fullsized":"2024-03-06T07:31:16.942704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.357645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.886635+0000","last_change":"2024-03-06T07:31:15.886635+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.886635+0000","last_undegraded":"2024-03-06T07:31:15.886635+0000","last_fullsized":"2024-03-06T07:31:15.886635+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.895531+0000","last_change":"2024-03-06T07:31:15.895531+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.895531+0000","last_undegraded":"2024-03-06T07:31:15.895531+0000","last_fullsized":"2024-03-06T07:31:15.895531+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.928454+0000","last_change":"2024-03-06T07:31:16.926842+0000","last_active":"2024-03-06T07:31:16.928454+0000","last_peered":"2024-03-06T07:31:16.928454+0000","last_clean":"2024-03-06T07:31:16.928454+0000","last_became_active":"2024-03-06T07:31:16.925778+0000","last_became_peered":"2024-03-06T07:31:16.925778+0000","last_unstale":"2024-03-06T07:31:16.928454+0000","last_undegraded":"2024-03-06T07:31:16.928454+0000","last_fullsized":"2024-03-06T07:31:16.928454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:46.773320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.940014+0000","last_change":"2024-03-06T07:31:16.935681+0000","last_active":"2024-03-06T07:31:16.940014+0000","last_peered":"2024-03-06T07:31:16.940014+0000","last_clean":"2024-03-06T07:31:16.940014+0000","last_became_active":"2024-03-06T07:31:16.935372+0000","last_became_peered":"2024-03-06T07:31:16.935372+0000","last_unstale":"2024-03-06T07:31:16.940014+0000","last_undegraded":"2024-03-06T07:31:16.940014+0000","last_fullsized":"2024-03-06T07:31:16.940014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:24.370092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.893976+0000","last_change":"2024-03-06T07:31:15.893976+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.893976+0000","last_undegraded":"2024-03-06T07:31:15.893976+0000","last_fullsized":"2024-03-06T07:31:15.893976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.885536+0000","last_change":"2024-03-06T07:31:15.885536+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.885536+0000","last_undegraded":"2024-03-06T07:31:15.885536+0000","last_fullsized":"2024-03-06T07:31:15.885536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.899200+0000","last_change":"2024-03-06T07:31:15.899200+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.899200+0000","last_undegraded":"2024-03-06T07:31:15.899200+0000","last_fullsized":"2024-03-06T07:31:15.899200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.900379+0000","last_change":"2024-03-06T07:31:15.900379+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.900379+0000","last_undegraded":"2024-03-06T07:31:15.900379+0000","last_fullsized":"2024-03-06T07:31:15.900379+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.902008+0000","last_change":"2024-03-06T07:31:15.902008+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.902008+0000","last_undegraded":"2024-03-06T07:31:15.902008+0000","last_fullsized":"2024-03-06T07:31:15.902008+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.900498+0000","last_change":"2024-03-06T07:31:15.900498+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.900498+0000","last_undegraded":"2024-03-06T07:31:15.900498+0000","last_fullsized":"2024-03-06T07:31:15.900498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.900807+0000","last_change":"2024-03-06T07:31:15.900807+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.900807+0000","last_undegraded":"2024-03-06T07:31:15.900807+0000","last_fullsized":"2024-03-06T07:31:15.900807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.907302+0000","last_change":"2024-03-06T07:31:15.907302+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.907302+0000","last_undegraded":"2024-03-06T07:31:15.907302+0000","last_fullsized":"2024-03-06T07:31:15.907302+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.887976+0000","last_change":"2024-03-06T07:31:15.887976+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.887976+0000","last_undegraded":"2024-03-06T07:31:15.887976+0000","last_fullsized":"2024-03-06T07:31:15.887976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.899514+0000","last_change":"2024-03-06T07:31:15.899514+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.899514+0000","last_undegraded":"2024-03-06T07:31:15.899514+0000","last_fullsized":"2024-03-06T07:31:15.899514+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.889368+0000","last_change":"2024-03-06T07:31:15.889368+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.889368+0000","last_undegraded":"2024-03-06T07:31:15.889368+0000","last_fullsized":"2024-03-06T07:31:15.889368+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.904372+0000","last_change":"2024-03-06T07:31:15.904372+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.904372+0000","last_undegraded":"2024-03-06T07:31:15.904372+0000","last_fullsized":"2024-03-06T07:31:15.904372+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.900120+0000","last_change":"2024-03-06T07:31:15.900120+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.900120+0000","last_undegraded":"2024-03-06T07:31:15.900120+0000","last_fullsized":"2024-03-06T07:31:15.900120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.887838+0000","last_change":"2024-03-06T07:31:15.887838+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.887838+0000","last_undegraded":"2024-03-06T07:31:15.887838+0000","last_fullsized":"2024-03-06T07:31:15.887838+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.904890+0000","last_change":"2024-03-06T07:31:15.904890+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.904890+0000","last_undegraded":"2024-03-06T07:31:15.904890+0000","last_fullsized":"2024-03-06T07:31:15.904890+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.920829+0000","last_change":"2024-03-06T07:31:15.919730+0000","last_active":"2024-03-06T07:31:15.920829+0000","last_peered":"2024-03-06T07:31:15.920829+0000","last_clean":"2024-03-06T07:31:15.920829+0000","last_became_active":"2024-03-06T07:31:15.919105+0000","last_became_peered":"2024-03-06T07:31:15.919105+0000","last_unstale":"2024-03-06T07:31:15.920829+0000","last_undegraded":"2024-03-06T07:31:15.920829+0000","last_fullsized":"2024-03-06T07:31:15.920829+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:08.485177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.928433+0000","last_change":"2024-03-06T07:31:16.926586+0000","last_active":"2024-03-06T07:31:16.928433+0000","last_peered":"2024-03-06T07:31:16.928433+0000","last_clean":"2024-03-06T07:31:16.928433+0000","last_became_active":"2024-03-06T07:31:16.925428+0000","last_became_peered":"2024-03-06T07:31:16.925428+0000","last_unstale":"2024-03-06T07:31:16.928433+0000","last_undegraded":"2024-03-06T07:31:16.928433+0000","last_fullsized":"2024-03-06T07:31:16.928433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:23.448908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.903871+0000","last_change":"2024-03-06T07:31:15.900554+0000","last_active":"2024-03-06T07:31:15.903871+0000","last_peered":"2024-03-06T07:31:15.903871+0000","last_clean":"2024-03-06T07:31:15.903871+0000","last_became_active":"2024-03-06T07:31:15.900282+0000","last_became_peered":"2024-03-06T07:31:15.900282+0000","last_unstale":"2024-03-06T07:31:15.903871+0000","last_undegraded":"2024-03-06T07:31:15.903871+0000","last_fullsized":"2024-03-06T07:31:15.903871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:40.366521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.896283+0000","last_change":"2024-03-06T07:31:15.896283+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.896283+0000","last_undegraded":"2024-03-06T07:31:15.896283+0000","last_fullsized":"2024-03-06T07:31:15.896283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.919347+0000","last_change":"2024-03-06T07:31:15.917942+0000","last_active":"2024-03-06T07:31:15.919347+0000","last_peered":"2024-03-06T07:31:15.919347+0000","last_clean":"2024-03-06T07:31:15.919347+0000","last_became_active":"2024-03-06T07:31:15.917620+0000","last_became_peered":"2024-03-06T07:31:15.917620+0000","last_unstale":"2024-03-06T07:31:15.919347+0000","last_undegraded":"2024-03-06T07:31:15.919347+0000","last_fullsized":"2024-03-06T07:31:15.919347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:48.457513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.908932+0000","last_change":"2024-03-06T07:31:15.908932+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.908932+0000","last_undegraded":"2024-03-06T07:31:15.908932+0000","last_fullsized":"2024-03-06T07:31:15.908932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.916059+0000","last_change":"2024-03-06T07:31:15.910176+0000","last_active":"2024-03-06T07:31:15.916059+0000","last_peered":"2024-03-06T07:31:15.916059+0000","last_clean":"2024-03-06T07:31:15.916059+0000","last_became_active":"2024-03-06T07:31:15.906533+0000","last_became_peered":"2024-03-06T07:31:15.906533+0000","last_unstale":"2024-03-06T07:31:15.916059+0000","last_undegraded":"2024-03-06T07:31:15.916059+0000","last_fullsized":"2024-03-06T07:31:15.916059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:10.165124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.903430+0000","last_change":"2024-03-06T07:31:15.903430+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.903430+0000","last_undegraded":"2024-03-06T07:31:15.903430+0000","last_fullsized":"2024-03-06T07:31:15.903430+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.915105+0000","last_change":"2024-03-06T07:31:15.908836+0000","last_active":"2024-03-06T07:31:15.915105+0000","last_peered":"2024-03-06T07:31:15.915105+0000","last_clean":"2024-03-06T07:31:15.915105+0000","last_became_active":"2024-03-06T07:31:15.908445+0000","last_became_peered":"2024-03-06T07:31:15.908445+0000","last_unstale":"2024-03-06T07:31:15.915105+0000","last_undegraded":"2024-03-06T07:31:15.915105+0000","last_fullsized":"2024-03-06T07:31:15.915105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:30:18.406813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.898108+0000","last_change":"2024-03-06T07:31:15.898108+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.898108+0000","last_undegraded":"2024-03-06T07:31:15.898108+0000","last_fullsized":"2024-03-06T07:31:15.898108+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.913583+0000","last_change":"2024-03-06T07:31:15.905100+0000","last_active":"2024-03-06T07:31:15.913583+0000","last_peered":"2024-03-06T07:31:15.913583+0000","last_clean":"2024-03-06T07:31:15.913583+0000","last_became_active":"2024-03-06T07:31:15.904742+0000","last_became_peered":"2024-03-06T07:31:15.904742+0000","last_unstale":"2024-03-06T07:31:15.913583+0000","last_undegraded":"2024-03-06T07:31:15.913583+0000","last_fullsized":"2024-03-06T07:31:15.913583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:39:25.685588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.942677+0000","last_change":"2024-03-06T07:31:16.940268+0000","last_active":"2024-03-06T07:31:16.942677+0000","last_peered":"2024-03-06T07:31:16.942677+0000","last_clean":"2024-03-06T07:31:16.942677+0000","last_became_active":"2024-03-06T07:31:16.937294+0000","last_became_peered":"2024-03-06T07:31:16.937294+0000","last_unstale":"2024-03-06T07:31:16.942677+0000","last_undegraded":"2024-03-06T07:31:16.942677+0000","last_fullsized":"2024-03-06T07:31:16.942677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:41.977774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.903243+0000","last_change":"2024-03-06T07:31:15.910549+0000","last_active":"2024-03-06T07:31:16.903243+0000","last_peered":"2024-03-06T07:31:16.903243+0000","last_clean":"2024-03-06T07:31:16.903243+0000","last_became_active":"2024-03-06T07:31:15.910107+0000","last_became_peered":"2024-03-06T07:31:15.910107+0000","last_unstale":"2024-03-06T07:31:16.903243+0000","last_undegraded":"2024-03-06T07:31:16.903243+0000","last_fullsized":"2024-03-06T07:31:16.903243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:01.856760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.885734+0000","last_change":"2024-03-06T07:31:15.885734+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.885734+0000","last_undegraded":"2024-03-06T07:31:15.885734+0000","last_fullsized":"2024-03-06T07:31:15.885734+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.905219+0000","last_change":"2024-03-06T07:31:15.901210+0000","last_active":"2024-03-06T07:31:16.905219+0000","last_peered":"2024-03-06T07:31:16.905219+0000","last_clean":"2024-03-06T07:31:16.905219+0000","last_became_active":"2024-03-06T07:31:15.900702+0000","last_became_peered":"2024-03-06T07:31:15.900702+0000","last_unstale":"2024-03-06T07:31:16.905219+0000","last_undegraded":"2024-03-06T07:31:16.905219+0000","last_fullsized":"2024-03-06T07:31:16.905219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:12.002175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.906821+0000","last_change":"2024-03-06T07:31:15.906821+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.906821+0000","last_undegraded":"2024-03-06T07:31:15.906821+0000","last_fullsized":"2024-03-06T07:31:15.906821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.910253+0000","last_change":"2024-03-06T07:31:15.906775+0000","last_active":"2024-03-06T07:31:15.910253+0000","last_peered":"2024-03-06T07:31:15.910253+0000","last_clean":"2024-03-06T07:31:15.910253+0000","last_became_active":"2024-03-06T07:31:15.906477+0000","last_became_peered":"2024-03-06T07:31:15.906477+0000","last_unstale":"2024-03-06T07:31:15.910253+0000","last_undegraded":"2024-03-06T07:31:15.910253+0000","last_fullsized":"2024-03-06T07:31:15.910253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:52.508253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.907205+0000","last_change":"2024-03-06T07:31:15.907205+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.907205+0000","last_undegraded":"2024-03-06T07:31:15.907205+0000","last_fullsized":"2024-03-06T07:31:15.907205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.918928+0000","last_change":"2024-03-06T07:31:15.916683+0000","last_active":"2024-03-06T07:31:15.918928+0000","last_peered":"2024-03-06T07:31:15.918928+0000","last_clean":"2024-03-06T07:31:15.918928+0000","last_became_active":"2024-03-06T07:31:15.916395+0000","last_became_peered":"2024-03-06T07:31:15.916395+0000","last_unstale":"2024-03-06T07:31:15.918928+0000","last_undegraded":"2024-03-06T07:31:15.918928+0000","last_fullsized":"2024-03-06T07:31:15.918928+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:39.682427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.895468+0000","last_change":"2024-03-06T07:31:15.895468+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.895468+0000","last_undegraded":"2024-03-06T07:31:15.895468+0000","last_fullsized":"2024-03-06T07:31:15.895468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.917780+0000","last_change":"2024-03-06T07:31:15.913307+0000","last_active":"2024-03-06T07:31:15.917780+0000","last_peered":"2024-03-06T07:31:15.917780+0000","last_clean":"2024-03-06T07:31:15.917780+0000","last_became_active":"2024-03-06T07:31:15.913035+0000","last_became_peered":"2024-03-06T07:31:15.913035+0000","last_unstale":"2024-03-06T07:31:15.917780+0000","last_undegraded":"2024-03-06T07:31:15.917780+0000","last_fullsized":"2024-03-06T07:31:15.917780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:10.675122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.924220+0000","last_change":"2024-03-06T07:31:16.921205+0000","last_active":"2024-03-06T07:31:16.924220+0000","last_peered":"2024-03-06T07:31:16.924220+0000","last_clean":"2024-03-06T07:31:16.924220+0000","last_became_active":"2024-03-06T07:31:16.920894+0000","last_became_peered":"2024-03-06T07:31:16.920894+0000","last_unstale":"2024-03-06T07:31:16.924220+0000","last_undegraded":"2024-03-06T07:31:16.924220+0000","last_fullsized":"2024-03-06T07:31:16.924220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:41.460377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.956660+0000","last_change":"2024-03-06T07:31:15.900762+0000","last_active":"2024-03-06T07:31:16.956660+0000","last_peered":"2024-03-06T07:31:16.956660+0000","last_clean":"2024-03-06T07:31:16.956660+0000","last_became_active":"2024-03-06T07:31:15.900473+0000","last_became_peered":"2024-03-06T07:31:15.900473+0000","last_unstale":"2024-03-06T07:31:16.956660+0000","last_undegraded":"2024-03-06T07:31:16.956660+0000","last_fullsized":"2024-03-06T07:31:16.956660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:14.489235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.902276+0000","last_change":"2024-03-06T07:31:15.902276+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.902276+0000","last_undegraded":"2024-03-06T07:31:15.902276+0000","last_fullsized":"2024-03-06T07:31:15.902276+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.956372+0000","last_change":"2024-03-06T07:31:15.911227+0000","last_active":"2024-03-06T07:31:16.956372+0000","last_peered":"2024-03-06T07:31:16.956372+0000","last_clean":"2024-03-06T07:31:16.956372+0000","last_became_active":"2024-03-06T07:31:15.910935+0000","last_became_peered":"2024-03-06T07:31:15.910935+0000","last_unstale":"2024-03-06T07:31:16.956372+0000","last_undegraded":"2024-03-06T07:31:16.956372+0000","last_fullsized":"2024-03-06T07:31:16.956372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:56.202287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.902391+0000","last_change":"2024-03-06T07:31:15.902391+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.902391+0000","last_undegraded":"2024-03-06T07:31:15.902391+0000","last_fullsized":"2024-03-06T07:31:15.902391+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.913551+0000","last_change":"2024-03-06T07:31:15.903043+0000","last_active":"2024-03-06T07:31:15.913551+0000","last_peered":"2024-03-06T07:31:15.913551+0000","last_clean":"2024-03-06T07:31:15.913551+0000","last_became_active":"2024-03-06T07:31:15.902732+0000","last_became_peered":"2024-03-06T07:31:15.902732+0000","last_unstale":"2024-03-06T07:31:15.913551+0000","last_undegraded":"2024-03-06T07:31:15.913551+0000","last_fullsized":"2024-03-06T07:31:15.913551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:40:33.019233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.885877+0000","last_change":"2024-03-06T07:31:15.885877+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.885877+0000","last_undegraded":"2024-03-06T07:31:15.885877+0000","last_fullsized":"2024-03-06T07:31:15.885877+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.918791+0000","last_change":"2024-03-06T07:31:15.913969+0000","last_active":"2024-03-06T07:31:15.918791+0000","last_peered":"2024-03-06T07:31:15.918791+0000","last_clean":"2024-03-06T07:31:15.918791+0000","last_became_active":"2024-03-06T07:31:15.913653+0000","last_became_peered":"2024-03-06T07:31:15.913653+0000","last_unstale":"2024-03-06T07:31:15.918791+0000","last_undegraded":"2024-03-06T07:31:15.918791+0000","last_fullsized":"2024-03-06T07:31:15.918791+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:36.263297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.935196+0000","last_change":"2024-03-06T07:31:16.928682+0000","last_active":"2024-03-06T07:31:16.935196+0000","last_peered":"2024-03-06T07:31:16.935196+0000","last_clean":"2024-03-06T07:31:16.935196+0000","last_became_active":"2024-03-06T07:31:16.928384+0000","last_became_peered":"2024-03-06T07:31:16.928384+0000","last_unstale":"2024-03-06T07:31:16.935196+0000","last_undegraded":"2024-03-06T07:31:16.935196+0000","last_fullsized":"2024-03-06T07:31:16.935196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:59.429967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.917250+0000","last_change":"2024-03-06T07:31:15.914916+0000","last_active":"2024-03-06T07:31:15.917250+0000","last_peered":"2024-03-06T07:31:15.917250+0000","last_clean":"2024-03-06T07:31:15.917250+0000","last_became_active":"2024-03-06T07:31:15.914582+0000","last_became_peered":"2024-03-06T07:31:15.914582+0000","last_unstale":"2024-03-06T07:31:15.917250+0000","last_undegraded":"2024-03-06T07:31:15.917250+0000","last_fullsized":"2024-03-06T07:31:15.917250+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:59.312816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.896117+0000","last_change":"2024-03-06T07:31:15.896117+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.896117+0000","last_undegraded":"2024-03-06T07:31:15.896117+0000","last_fullsized":"2024-03-06T07:31:15.896117+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.917098+0000","last_change":"2024-03-06T07:31:15.912672+0000","last_active":"2024-03-06T07:31:15.917098+0000","last_peered":"2024-03-06T07:31:15.917098+0000","last_clean":"2024-03-06T07:31:15.917098+0000","last_became_active":"2024-03-06T07:31:15.912309+0000","last_became_peered":"2024-03-06T07:31:15.912309+0000","last_unstale":"2024-03-06T07:31:15.917098+0000","last_undegraded":"2024-03-06T07:31:15.917098+0000","last_fullsized":"2024-03-06T07:31:15.917098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:17:01.825331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.906149+0000","last_change":"2024-03-06T07:31:15.906149+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.906149+0000","last_undegraded":"2024-03-06T07:31:15.906149+0000","last_fullsized":"2024-03-06T07:31:15.906149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.919590+0000","last_change":"2024-03-06T07:31:15.917451+0000","last_active":"2024-03-06T07:31:15.919590+0000","last_peered":"2024-03-06T07:31:15.919590+0000","last_clean":"2024-03-06T07:31:15.919590+0000","last_became_active":"2024-03-06T07:31:15.917076+0000","last_became_peered":"2024-03-06T07:31:15.917076+0000","last_unstale":"2024-03-06T07:31:15.919590+0000","last_undegraded":"2024-03-06T07:31:15.919590+0000","last_fullsized":"2024-03-06T07:31:15.919590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:18.097526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.938055+0000","last_change":"2024-03-06T07:31:16.925969+0000","last_active":"2024-03-06T07:31:16.938055+0000","last_peered":"2024-03-06T07:31:16.938055+0000","last_clean":"2024-03-06T07:31:16.938055+0000","last_became_active":"2024-03-06T07:31:16.925618+0000","last_became_peered":"2024-03-06T07:31:16.925618+0000","last_unstale":"2024-03-06T07:31:16.938055+0000","last_undegraded":"2024-03-06T07:31:16.938055+0000","last_fullsized":"2024-03-06T07:31:16.938055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:56.377142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.918998+0000","last_change":"2024-03-06T07:31:15.915434+0000","last_active":"2024-03-06T07:31:15.918998+0000","last_peered":"2024-03-06T07:31:15.918998+0000","last_clean":"2024-03-06T07:31:15.918998+0000","last_became_active":"2024-03-06T07:31:15.915119+0000","last_became_peered":"2024-03-06T07:31:15.915119+0000","last_unstale":"2024-03-06T07:31:15.918998+0000","last_undegraded":"2024-03-06T07:31:15.918998+0000","last_fullsized":"2024-03-06T07:31:15.918998+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:25:51.924870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.939807+0000","last_change":"2024-03-06T07:31:16.934397+0000","last_active":"2024-03-06T07:31:16.939807+0000","last_peered":"2024-03-06T07:31:16.939807+0000","last_clean":"2024-03-06T07:31:16.939807+0000","last_became_active":"2024-03-06T07:31:16.934037+0000","last_became_peered":"2024-03-06T07:31:16.934037+0000","last_unstale":"2024-03-06T07:31:16.939807+0000","last_undegraded":"2024-03-06T07:31:16.939807+0000","last_fullsized":"2024-03-06T07:31:16.939807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:32.614453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.920815+0000","last_change":"2024-03-06T07:31:15.919406+0000","last_active":"2024-03-06T07:31:15.920815+0000","last_peered":"2024-03-06T07:31:15.920815+0000","last_clean":"2024-03-06T07:31:15.920815+0000","last_became_active":"2024-03-06T07:31:15.919150+0000","last_became_peered":"2024-03-06T07:31:15.919150+0000","last_unstale":"2024-03-06T07:31:15.920815+0000","last_undegraded":"2024-03-06T07:31:15.920815+0000","last_fullsized":"2024-03-06T07:31:15.920815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:57.176847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.888510+0000","last_change":"2024-03-06T07:31:15.888510+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.888510+0000","last_undegraded":"2024-03-06T07:31:15.888510+0000","last_fullsized":"2024-03-06T07:31:15.888510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.913168+0000","last_change":"2024-03-06T07:31:15.897274+0000","last_active":"2024-03-06T07:31:15.913168+0000","last_peered":"2024-03-06T07:31:15.913168+0000","last_clean":"2024-03-06T07:31:15.913168+0000","last_became_active":"2024-03-06T07:31:15.896726+0000","last_became_peered":"2024-03-06T07:31:15.896726+0000","last_unstale":"2024-03-06T07:31:15.913168+0000","last_undegraded":"2024-03-06T07:31:15.913168+0000","last_fullsized":"2024-03-06T07:31:15.913168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:20:45.482728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.884995+0000","last_change":"2024-03-06T07:31:15.884995+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.884995+0000","last_undegraded":"2024-03-06T07:31:15.884995+0000","last_fullsized":"2024-03-06T07:31:15.884995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.918306+0000","last_change":"2024-03-06T07:31:15.913905+0000","last_active":"2024-03-06T07:31:15.918306+0000","last_peered":"2024-03-06T07:31:15.918306+0000","last_clean":"2024-03-06T07:31:15.918306+0000","last_became_active":"2024-03-06T07:31:15.913284+0000","last_became_peered":"2024-03-06T07:31:15.913284+0000","last_unstale":"2024-03-06T07:31:15.918306+0000","last_undegraded":"2024-03-06T07:31:15.918306+0000","last_fullsized":"2024-03-06T07:31:15.918306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:44.892794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.885095+0000","last_change":"2024-03-06T07:31:15.885095+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.885095+0000","last_undegraded":"2024-03-06T07:31:15.885095+0000","last_fullsized":"2024-03-06T07:31:15.885095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.903968+0000","last_change":"2024-03-06T07:31:15.901323+0000","last_active":"2024-03-06T07:31:15.903968+0000","last_peered":"2024-03-06T07:31:15.903968+0000","last_clean":"2024-03-06T07:31:15.903968+0000","last_became_active":"2024-03-06T07:31:15.900934+0000","last_became_peered":"2024-03-06T07:31:15.900934+0000","last_unstale":"2024-03-06T07:31:15.903968+0000","last_undegraded":"2024-03-06T07:31:15.903968+0000","last_fullsized":"2024-03-06T07:31:15.903968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:54.426080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.885480+0000","last_change":"2024-03-06T07:31:15.885480+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.885480+0000","last_undegraded":"2024-03-06T07:31:15.885480+0000","last_fullsized":"2024-03-06T07:31:15.885480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.916004+0000","last_change":"2024-03-06T07:31:15.909680+0000","last_active":"2024-03-06T07:31:15.916004+0000","last_peered":"2024-03-06T07:31:15.916004+0000","last_clean":"2024-03-06T07:31:15.916004+0000","last_became_active":"2024-03-06T07:31:15.908456+0000","last_became_peered":"2024-03-06T07:31:15.908456+0000","last_unstale":"2024-03-06T07:31:15.916004+0000","last_undegraded":"2024-03-06T07:31:15.916004+0000","last_fullsized":"2024-03-06T07:31:15.916004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:02:10.821527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.906292+0000","last_change":"2024-03-06T07:31:15.906292+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.906292+0000","last_undegraded":"2024-03-06T07:31:15.906292+0000","last_fullsized":"2024-03-06T07:31:15.906292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.917723+0000","last_change":"2024-03-06T07:31:15.913471+0000","last_active":"2024-03-06T07:31:15.917723+0000","last_peered":"2024-03-06T07:31:15.917723+0000","last_clean":"2024-03-06T07:31:15.917723+0000","last_became_active":"2024-03-06T07:31:15.913173+0000","last_became_peered":"2024-03-06T07:31:15.913173+0000","last_unstale":"2024-03-06T07:31:15.917723+0000","last_undegraded":"2024-03-06T07:31:15.917723+0000","last_fullsized":"2024-03-06T07:31:15.917723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:00:33.233528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.905215+0000","last_change":"2024-03-06T07:31:15.905215+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.905215+0000","last_undegraded":"2024-03-06T07:31:15.905215+0000","last_fullsized":"2024-03-06T07:31:15.905215+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.910951+0000","last_change":"2024-03-06T07:31:15.904132+0000","last_active":"2024-03-06T07:31:16.910951+0000","last_peered":"2024-03-06T07:31:16.910951+0000","last_clean":"2024-03-06T07:31:16.910951+0000","last_became_active":"2024-03-06T07:31:15.903817+0000","last_became_peered":"2024-03-06T07:31:15.903817+0000","last_unstale":"2024-03-06T07:31:16.910951+0000","last_undegraded":"2024-03-06T07:31:16.910951+0000","last_fullsized":"2024-03-06T07:31:16.910951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:00.101871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.906025+0000","last_change":"2024-03-06T07:31:15.906025+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.906025+0000","last_undegraded":"2024-03-06T07:31:15.906025+0000","last_fullsized":"2024-03-06T07:31:15.906025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.914900+0000","last_change":"2024-03-06T07:31:15.910213+0000","last_active":"2024-03-06T07:31:15.914900+0000","last_peered":"2024-03-06T07:31:15.914900+0000","last_clean":"2024-03-06T07:31:15.914900+0000","last_became_active":"2024-03-06T07:31:15.909858+0000","last_became_peered":"2024-03-06T07:31:15.909858+0000","last_unstale":"2024-03-06T07:31:15.914900+0000","last_undegraded":"2024-03-06T07:31:15.914900+0000","last_fullsized":"2024-03-06T07:31:15.914900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:14.688923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.930066+0000","last_change":"2024-03-06T07:31:16.926862+0000","last_active":"2024-03-06T07:31:16.930066+0000","last_peered":"2024-03-06T07:31:16.930066+0000","last_clean":"2024-03-06T07:31:16.930066+0000","last_became_active":"2024-03-06T07:31:16.926344+0000","last_became_peered":"2024-03-06T07:31:16.926344+0000","last_unstale":"2024-03-06T07:31:16.930066+0000","last_undegraded":"2024-03-06T07:31:16.930066+0000","last_fullsized":"2024-03-06T07:31:16.930066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:10.875725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.911800+0000","last_change":"2024-03-06T07:31:15.916239+0000","last_active":"2024-03-06T07:31:16.911800+0000","last_peered":"2024-03-06T07:31:16.911800+0000","last_clean":"2024-03-06T07:31:16.911800+0000","last_became_active":"2024-03-06T07:31:15.915781+0000","last_became_peered":"2024-03-06T07:31:15.915781+0000","last_unstale":"2024-03-06T07:31:16.911800+0000","last_undegraded":"2024-03-06T07:31:16.911800+0000","last_fullsized":"2024-03-06T07:31:16.911800+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:50.813809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.891105+0000","last_change":"2024-03-06T07:31:15.891105+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.891105+0000","last_undegraded":"2024-03-06T07:31:15.891105+0000","last_fullsized":"2024-03-06T07:31:15.891105+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.903175+0000","last_change":"2024-03-06T07:31:15.907149+0000","last_active":"2024-03-06T07:31:16.903175+0000","last_peered":"2024-03-06T07:31:16.903175+0000","last_clean":"2024-03-06T07:31:16.903175+0000","last_became_active":"2024-03-06T07:31:15.906805+0000","last_became_peered":"2024-03-06T07:31:15.906805+0000","last_unstale":"2024-03-06T07:31:16.903175+0000","last_undegraded":"2024-03-06T07:31:16.903175+0000","last_fullsized":"2024-03-06T07:31:16.903175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:41.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.938127+0000","last_change":"2024-03-06T07:31:16.930343+0000","last_active":"2024-03-06T07:31:16.938127+0000","last_peered":"2024-03-06T07:31:16.938127+0000","last_clean":"2024-03-06T07:31:16.938127+0000","last_became_active":"2024-03-06T07:31:16.928375+0000","last_became_peered":"2024-03-06T07:31:16.928375+0000","last_unstale":"2024-03-06T07:31:16.938127+0000","last_undegraded":"2024-03-06T07:31:16.938127+0000","last_fullsized":"2024-03-06T07:31:16.938127+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:56.648903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.914685+0000","last_change":"2024-03-06T07:31:15.913686+0000","last_active":"2024-03-06T07:31:15.914685+0000","last_peered":"2024-03-06T07:31:15.914685+0000","last_clean":"2024-03-06T07:31:15.914685+0000","last_became_active":"2024-03-06T07:31:15.913337+0000","last_became_peered":"2024-03-06T07:31:15.913337+0000","last_unstale":"2024-03-06T07:31:15.914685+0000","last_undegraded":"2024-03-06T07:31:15.914685+0000","last_fullsized":"2024-03-06T07:31:15.914685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:07:40.879689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.889291+0000","last_change":"2024-03-06T07:31:15.889291+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.889291+0000","last_undegraded":"2024-03-06T07:31:15.889291+0000","last_fullsized":"2024-03-06T07:31:15.889291+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.913043+0000","last_change":"2024-03-06T07:31:15.898982+0000","last_active":"2024-03-06T07:31:15.913043+0000","last_peered":"2024-03-06T07:31:15.913043+0000","last_clean":"2024-03-06T07:31:15.913043+0000","last_became_active":"2024-03-06T07:31:15.898591+0000","last_became_peered":"2024-03-06T07:31:15.898591+0000","last_unstale":"2024-03-06T07:31:15.913043+0000","last_undegraded":"2024-03-06T07:31:15.913043+0000","last_fullsized":"2024-03-06T07:31:15.913043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:10.330197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-03-06T07:31:15.885808+0000","last_change":"2024-03-06T07:31:15.885808+0000","last_active":"2024-03-06T07:31:15.871084+0000","last_peered":"2024-03-06T07:31:15.871084+0000","last_clean":"2024-03-06T07:31:15.871084+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:31:15.885808+0000","last_undegraded":"2024-03-06T07:31:15.885808+0000","last_fullsized":"2024-03-06T07:31:15.885808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.916993+0000","last_change":"2024-03-06T07:31:15.911459+0000","last_active":"2024-03-06T07:31:15.916993+0000","last_peered":"2024-03-06T07:31:15.916993+0000","last_clean":"2024-03-06T07:31:15.916993+0000","last_became_active":"2024-03-06T07:31:15.911167+0000","last_became_peered":"2024-03-06T07:31:15.911167+0000","last_unstale":"2024-03-06T07:31:15.916993+0000","last_undegraded":"2024-03-06T07:31:15.916993+0000","last_fullsized":"2024-03-06T07:31:15.916993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:45.826298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":252,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-06T07:31:15.911244+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:31:15.911244+0000","last_peered":"2024-03-06T07:31:15.911244+0000","last_clean":"2024-03-06T07:31:15.911244+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:31:15.911244+0000","last_undegraded":"2024-03-06T07:31:15.911244+0000","last_fullsized":"2024-03-06T07:31:15.911244+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":1646,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509490,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":335632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":335632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542194,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36708,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335132,"statfs":{"total":96636764160,"available":96599175168,"internally_reserved":0,"allocated":1400832,"data_stored":926000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":329357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":806912,"data_stored":335632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542194,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36128,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335712,"statfs":{"total":96636764160,"available":96599769088,"internally_reserved":0,"allocated":806912,"data_stored":335632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542195,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":329357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542195,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":1400832,"data_stored":926000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:21.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:31:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.772+0000 7f1267e00640 1 Processor -- start 2024-03-06T07:31:21.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.772+0000 7f1267e00640 1 -- start start 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.773+0000 7f1267e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12680cd060 0x7f126815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.773+0000 7f1267e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f126815e810 0x7f12680b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.773+0000 7f1267e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126815ed10 con 0x7f12681552b0 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.773+0000 7f1267e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f126815eff0 con 0x7f126815e810 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.773+0000 7f1267e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126815f2d0 con 0x7f12680cd060 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1267400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f126815e810 0x7f12680b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1266a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f12681552b0 0x7f12680cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39014/0 (socket says 172.21.15.6:39014) 2024-03-06T07:31:21.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1266a00640 1 -- 172.21.15.6:0/2066095813 learned_addr learned my addr 172.21.15.6:0/2066095813 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f125e000640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12680cd060 0x7f126815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1267400640 1 -- 172.21.15.6:0/2066095813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12680cd060 msgr2=0x7f126815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1267400640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12680cd060 0x7f126815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1267400640 1 -- 172.21.15.6:0/2066095813 >> v1:172.21.15.6:6789/0 conn(0x7f12681552b0 legacy=0x7f12680cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1266000640 1 -- 172.21.15.6:0/2066095813 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266753116 0 0) 0x7f126815ed10 con 0x7f12681552b0 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f1267400640 1 -- 172.21.15.6:0/2066095813 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126815f5b0 con 0x7f126815e810 2024-03-06T07:31:21.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.774+0000 7f125e000640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12680cd060 0x7f126815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.775+0000 7f1267400640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f12680b3cb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1260009d90 tx=0x7f126002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4803294c6f0d6614 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1266000640 1 -- 172.21.15.6:0/2066095813 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260039070 con 0x7f126815e810 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1266000640 1 -- 172.21.15.6:0/2066095813 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126002cbc0 con 0x7f126815e810 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1266000640 1 -- 172.21.15.6:0/2066095813 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260034440 con 0x7f126815e810 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1267e00640 1 -- 172.21.15.6:0/2066095813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 msgr2=0x7f12680b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1267e00640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f12680b3cb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1260009d90 tx=0x7f126002be10 comp rx=0 tx=0).stop 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1267e00640 1 -- 172.21.15.6:0/2066095813 shutdown_connections 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1267e00640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f12680b3cb0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1267e00640 1 --2- 172.21.15.6:0/2066095813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12680cd060 0x7f126815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.776+0000 7f1267e00640 1 -- 172.21.15.6:0/2066095813 >> 172.21.15.6:0/2066095813 conn(0x7f126806b3d0 msgr2=0x7f126806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.777+0000 7f1267e00640 1 -- 172.21.15.6:0/2066095813 shutdown_connections 2024-03-06T07:31:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.777+0000 7f1267e00640 1 -- 172.21.15.6:0/2066095813 wait complete. 2024-03-06T07:31:21.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.778+0000 7f1267e00640 1 Processor -- start 2024-03-06T07:31:21.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.778+0000 7f1267e00640 1 -- start start 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12680cd060 0x7f12680aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12681552b0 0x7f1268157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f126814f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126806dd30 con 0x7f126815e810 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12680b3760 con 0x7f12680cd060 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126806deb0 con 0x7f12681552b0 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f126814f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f126814f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:54194/0 (socket says 172.21.15.6:54194) 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.780+0000 7f1267400640 1 -- 172.21.15.6:0/2053989679 learned_addr learned my addr 172.21.15.6:0/2053989679 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:21.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1266a00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12680cd060 0x7f12680aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f125e000640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12681552b0 0x7f1268157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1267400640 1 -- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12681552b0 msgr2=0x7f1268157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1267400640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12681552b0 0x7f1268157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1267400640 1 -- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12680cd060 msgr2=0x7f12680aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1267400640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12680cd060 0x7f12680aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1267400640 1 -- 172.21.15.6:0/2053989679 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268157bc0 con 0x7f126815e810 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f1267400640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f126814f480 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1260009d90 tx=0x7f1260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.781+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260044020 con 0x7f126815e810 2024-03-06T07:31:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.782+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126003a040 con 0x7f126815e810 2024-03-06T07:31:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.782+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1268157e40 con 0x7f126815e810 2024-03-06T07:31:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.782+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1260039050 con 0x7f126815e810 2024-03-06T07:31:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.782+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1268159b50 con 0x7f126815e810 2024-03-06T07:31:21.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.783+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126804f330 con 0x7f126815e810 2024-03-06T07:31:21.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.784+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1260003bd0 con 0x7f126815e810 2024-03-06T07:31:21.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.785+0000 7f125f400640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124406b4d0 0x7f124406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.785+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12600ac920 con 0x7f126815e810 2024-03-06T07:31:21.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.786+0000 7f1266a00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124406b4d0 0x7f124406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.787+0000 7f1266a00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124406b4d0 0x7f124406d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1258000f60 tx=0x7f1258007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.790+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f126004e060 con 0x7f126815e810 2024-03-06T07:31:21.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:21.987+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f126815a040 con 0x7f124406b4d0 2024-03-06T07:31:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.005+0000 7f125f400640 1 -- 172.21.15.6:0/2053989679 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252052 (secure 0 0 0) 0x7f126815a040 con 0x7f124406b4d0 2024-03-06T07:31:22.010 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:22.011 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124406b4d0 msgr2=0x7f124406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124406b4d0 0x7f124406d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1258000f60 tx=0x7f1258007b70 comp rx=0 tx=0).stop 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 msgr2=0x7f126814f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f126814f480 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1260009d90 tx=0x7f1260004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 shutdown_connections 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f124406b4d0 0x7f124406d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f126815e810 0x7f126814f480 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f12681552b0 0x7f1268157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 --2- 172.21.15.6:0/2053989679 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12680cd060 0x7f12680aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.010+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 >> 172.21.15.6:0/2053989679 conn(0x7f126806b3d0 msgr2=0x7f1268073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.011+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 shutdown_connections 2024-03-06T07:31:22.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.011+0000 7f1267e00640 1 -- 172.21.15.6:0/2053989679 wait complete. 2024-03-06T07:31:22.033 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-03-06T07:31:21.521617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":265452,"kb_used_data":7640,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754709268,"statfs":{"total":773094113280,"available":772822290432,"internally_reserved":0,"allocated":7823360,"data_stored":3945222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001965"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461760+0000","last_change":"2024-03-06T07:31:16.939803+0000","last_active":"2024-03-06T07:31:18.461760+0000","last_peered":"2024-03-06T07:31:18.461760+0000","last_clean":"2024-03-06T07:31:18.461760+0000","last_became_active":"2024-03-06T07:31:16.939489+0000","last_became_peered":"2024-03-06T07:31:16.939489+0000","last_unstale":"2024-03-06T07:31:18.461760+0000","last_undegraded":"2024-03-06T07:31:18.461760+0000","last_fullsized":"2024-03-06T07:31:18.461760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:13:14.899919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.936396+0000","last_change":"2024-03-06T07:31:16.947864+0000","last_active":"2024-03-06T07:31:17.936396+0000","last_peered":"2024-03-06T07:31:17.936396+0000","last_clean":"2024-03-06T07:31:17.936396+0000","last_became_active":"2024-03-06T07:31:16.947482+0000","last_became_peered":"2024-03-06T07:31:16.947482+0000","last_unstale":"2024-03-06T07:31:17.936396+0000","last_undegraded":"2024-03-06T07:31:17.936396+0000","last_fullsized":"2024-03-06T07:31:17.936396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:46.673404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931723+0000","last_change":"2024-03-06T07:31:16.923724+0000","last_active":"2024-03-06T07:31:17.931723+0000","last_peered":"2024-03-06T07:31:17.931723+0000","last_clean":"2024-03-06T07:31:17.931723+0000","last_became_active":"2024-03-06T07:31:16.922096+0000","last_became_peered":"2024-03-06T07:31:16.922096+0000","last_unstale":"2024-03-06T07:31:17.931723+0000","last_undegraded":"2024-03-06T07:31:17.931723+0000","last_fullsized":"2024-03-06T07:31:17.931723+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:50:29.491921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.941593+0000","last_change":"2024-03-06T07:31:16.940027+0000","last_active":"2024-03-06T07:31:16.941593+0000","last_peered":"2024-03-06T07:31:16.941593+0000","last_clean":"2024-03-06T07:31:16.941593+0000","last_became_active":"2024-03-06T07:31:16.939339+0000","last_became_peered":"2024-03-06T07:31:16.939339+0000","last_unstale":"2024-03-06T07:31:16.941593+0000","last_undegraded":"2024-03-06T07:31:16.941593+0000","last_fullsized":"2024-03-06T07:31:16.941593+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:55:53.522887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.940588+0000","last_change":"2024-03-06T07:31:16.934162+0000","last_active":"2024-03-06T07:31:16.940588+0000","last_peered":"2024-03-06T07:31:16.940588+0000","last_clean":"2024-03-06T07:31:16.940588+0000","last_became_active":"2024-03-06T07:31:16.933866+0000","last_became_peered":"2024-03-06T07:31:16.933866+0000","last_unstale":"2024-03-06T07:31:16.940588+0000","last_undegraded":"2024-03-06T07:31:16.940588+0000","last_fullsized":"2024-03-06T07:31:16.940588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:24.805790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935331+0000","last_change":"2024-03-06T07:31:16.935428+0000","last_active":"2024-03-06T07:31:17.935331+0000","last_peered":"2024-03-06T07:31:17.935331+0000","last_clean":"2024-03-06T07:31:17.935331+0000","last_became_active":"2024-03-06T07:31:16.935084+0000","last_became_peered":"2024-03-06T07:31:16.935084+0000","last_unstale":"2024-03-06T07:31:17.935331+0000","last_undegraded":"2024-03-06T07:31:17.935331+0000","last_fullsized":"2024-03-06T07:31:17.935331+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:55.440714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.939648+0000","last_change":"2024-03-06T07:31:16.947971+0000","last_active":"2024-03-06T07:31:17.939648+0000","last_peered":"2024-03-06T07:31:17.939648+0000","last_clean":"2024-03-06T07:31:17.939648+0000","last_became_active":"2024-03-06T07:31:16.946843+0000","last_became_peered":"2024-03-06T07:31:16.946843+0000","last_unstale":"2024-03-06T07:31:17.939648+0000","last_undegraded":"2024-03-06T07:31:17.939648+0000","last_fullsized":"2024-03-06T07:31:17.939648+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:14.374916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.934639+0000","last_change":"2024-03-06T07:31:16.945664+0000","last_active":"2024-03-06T07:31:17.934639+0000","last_peered":"2024-03-06T07:31:17.934639+0000","last_clean":"2024-03-06T07:31:17.934639+0000","last_became_active":"2024-03-06T07:31:16.944571+0000","last_became_peered":"2024-03-06T07:31:16.944571+0000","last_unstale":"2024-03-06T07:31:17.934639+0000","last_undegraded":"2024-03-06T07:31:17.934639+0000","last_fullsized":"2024-03-06T07:31:17.934639+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:02:37.277931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.935158+0000","last_change":"2024-03-06T07:31:16.931065+0000","last_active":"2024-03-06T07:31:16.935158+0000","last_peered":"2024-03-06T07:31:16.935158+0000","last_clean":"2024-03-06T07:31:16.935158+0000","last_became_active":"2024-03-06T07:31:16.930774+0000","last_became_peered":"2024-03-06T07:31:16.930774+0000","last_unstale":"2024-03-06T07:31:16.935158+0000","last_undegraded":"2024-03-06T07:31:16.935158+0000","last_fullsized":"2024-03-06T07:31:16.935158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:12.934078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.375025+0000","last_change":"2024-03-06T07:31:16.927988+0000","last_active":"2024-03-06T07:31:18.375025+0000","last_peered":"2024-03-06T07:31:18.375025+0000","last_clean":"2024-03-06T07:31:18.375025+0000","last_became_active":"2024-03-06T07:31:16.927030+0000","last_became_peered":"2024-03-06T07:31:16.927030+0000","last_unstale":"2024-03-06T07:31:18.375025+0000","last_undegraded":"2024-03-06T07:31:18.375025+0000","last_fullsized":"2024-03-06T07:31:18.375025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:47.357905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461189+0000","last_change":"2024-03-06T07:31:16.941723+0000","last_active":"2024-03-06T07:31:18.461189+0000","last_peered":"2024-03-06T07:31:18.461189+0000","last_clean":"2024-03-06T07:31:18.461189+0000","last_became_active":"2024-03-06T07:31:16.941250+0000","last_became_peered":"2024-03-06T07:31:16.941250+0000","last_unstale":"2024-03-06T07:31:18.461189+0000","last_undegraded":"2024-03-06T07:31:18.461189+0000","last_fullsized":"2024-03-06T07:31:18.461189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:05:54.158963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.935821+0000","last_change":"2024-03-06T07:31:16.922685+0000","last_active":"2024-03-06T07:31:16.935821+0000","last_peered":"2024-03-06T07:31:16.935821+0000","last_clean":"2024-03-06T07:31:16.935821+0000","last_became_active":"2024-03-06T07:31:16.922381+0000","last_became_peered":"2024-03-06T07:31:16.922381+0000","last_unstale":"2024-03-06T07:31:16.935821+0000","last_undegraded":"2024-03-06T07:31:16.935821+0000","last_fullsized":"2024-03-06T07:31:16.935821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:54.935260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929636+0000","last_change":"2024-03-06T07:31:16.935995+0000","last_active":"2024-03-06T07:31:17.929636+0000","last_peered":"2024-03-06T07:31:17.929636+0000","last_clean":"2024-03-06T07:31:17.929636+0000","last_became_active":"2024-03-06T07:31:16.935647+0000","last_became_peered":"2024-03-06T07:31:16.935647+0000","last_unstale":"2024-03-06T07:31:17.929636+0000","last_undegraded":"2024-03-06T07:31:17.929636+0000","last_fullsized":"2024-03-06T07:31:17.929636+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:26.827543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.942704+0000","last_change":"2024-03-06T07:31:16.940311+0000","last_active":"2024-03-06T07:31:16.942704+0000","last_peered":"2024-03-06T07:31:16.942704+0000","last_clean":"2024-03-06T07:31:16.942704+0000","last_became_active":"2024-03-06T07:31:16.939993+0000","last_became_peered":"2024-03-06T07:31:16.939993+0000","last_unstale":"2024-03-06T07:31:16.942704+0000","last_undegraded":"2024-03-06T07:31:16.942704+0000","last_fullsized":"2024-03-06T07:31:16.942704+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:39.357645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.930522+0000","last_change":"2024-03-06T07:31:16.934139+0000","last_active":"2024-03-06T07:31:17.930522+0000","last_peered":"2024-03-06T07:31:17.930522+0000","last_clean":"2024-03-06T07:31:17.930522+0000","last_became_active":"2024-03-06T07:31:16.933474+0000","last_became_peered":"2024-03-06T07:31:16.933474+0000","last_unstale":"2024-03-06T07:31:17.930522+0000","last_undegraded":"2024-03-06T07:31:17.930522+0000","last_fullsized":"2024-03-06T07:31:17.930522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:44.209373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374172+0000","last_change":"2024-03-06T07:31:16.939775+0000","last_active":"2024-03-06T07:31:18.374172+0000","last_peered":"2024-03-06T07:31:18.374172+0000","last_clean":"2024-03-06T07:31:18.374172+0000","last_became_active":"2024-03-06T07:31:16.939468+0000","last_became_peered":"2024-03-06T07:31:16.939468+0000","last_unstale":"2024-03-06T07:31:18.374172+0000","last_undegraded":"2024-03-06T07:31:18.374172+0000","last_fullsized":"2024-03-06T07:31:18.374172+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:54:21.274554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.928454+0000","last_change":"2024-03-06T07:31:16.926842+0000","last_active":"2024-03-06T07:31:16.928454+0000","last_peered":"2024-03-06T07:31:16.928454+0000","last_clean":"2024-03-06T07:31:16.928454+0000","last_became_active":"2024-03-06T07:31:16.925778+0000","last_became_peered":"2024-03-06T07:31:16.925778+0000","last_unstale":"2024-03-06T07:31:16.928454+0000","last_undegraded":"2024-03-06T07:31:16.928454+0000","last_fullsized":"2024-03-06T07:31:16.928454+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:25:46.773320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.940014+0000","last_change":"2024-03-06T07:31:16.935681+0000","last_active":"2024-03-06T07:31:16.940014+0000","last_peered":"2024-03-06T07:31:16.940014+0000","last_clean":"2024-03-06T07:31:16.940014+0000","last_became_active":"2024-03-06T07:31:16.935372+0000","last_became_peered":"2024-03-06T07:31:16.935372+0000","last_unstale":"2024-03-06T07:31:16.940014+0000","last_undegraded":"2024-03-06T07:31:16.940014+0000","last_fullsized":"2024-03-06T07:31:16.940014+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:24.370092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.936219+0000","last_change":"2024-03-06T07:31:16.943872+0000","last_active":"2024-03-06T07:31:17.936219+0000","last_peered":"2024-03-06T07:31:17.936219+0000","last_clean":"2024-03-06T07:31:17.936219+0000","last_became_active":"2024-03-06T07:31:16.943549+0000","last_became_peered":"2024-03-06T07:31:16.943549+0000","last_unstale":"2024-03-06T07:31:17.936219+0000","last_undegraded":"2024-03-06T07:31:17.936219+0000","last_fullsized":"2024-03-06T07:31:17.936219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:50.749388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929311+0000","last_change":"2024-03-06T07:31:16.938187+0000","last_active":"2024-03-06T07:31:17.929311+0000","last_peered":"2024-03-06T07:31:17.929311+0000","last_clean":"2024-03-06T07:31:17.929311+0000","last_became_active":"2024-03-06T07:31:16.937889+0000","last_became_peered":"2024-03-06T07:31:16.937889+0000","last_unstale":"2024-03-06T07:31:17.929311+0000","last_undegraded":"2024-03-06T07:31:17.929311+0000","last_fullsized":"2024-03-06T07:31:17.929311+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:22.662913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374439+0000","last_change":"2024-03-06T07:31:16.929007+0000","last_active":"2024-03-06T07:31:18.374439+0000","last_peered":"2024-03-06T07:31:18.374439+0000","last_clean":"2024-03-06T07:31:18.374439+0000","last_became_active":"2024-03-06T07:31:16.928733+0000","last_became_peered":"2024-03-06T07:31:16.928733+0000","last_unstale":"2024-03-06T07:31:18.374439+0000","last_undegraded":"2024-03-06T07:31:18.374439+0000","last_fullsized":"2024-03-06T07:31:18.374439+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:43.883560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374176+0000","last_change":"2024-03-06T07:31:16.924966+0000","last_active":"2024-03-06T07:31:18.374176+0000","last_peered":"2024-03-06T07:31:18.374176+0000","last_clean":"2024-03-06T07:31:18.374176+0000","last_became_active":"2024-03-06T07:31:16.924647+0000","last_became_peered":"2024-03-06T07:31:16.924647+0000","last_unstale":"2024-03-06T07:31:18.374176+0000","last_undegraded":"2024-03-06T07:31:18.374176+0000","last_fullsized":"2024-03-06T07:31:18.374176+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:40.521478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.460715+0000","last_change":"2024-03-06T07:31:16.936610+0000","last_active":"2024-03-06T07:31:18.460715+0000","last_peered":"2024-03-06T07:31:18.460715+0000","last_clean":"2024-03-06T07:31:18.460715+0000","last_became_active":"2024-03-06T07:31:16.936311+0000","last_became_peered":"2024-03-06T07:31:16.936311+0000","last_unstale":"2024-03-06T07:31:18.460715+0000","last_undegraded":"2024-03-06T07:31:18.460715+0000","last_fullsized":"2024-03-06T07:31:18.460715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:15.084951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.460497+0000","last_change":"2024-03-06T07:31:16.940778+0000","last_active":"2024-03-06T07:31:18.460497+0000","last_peered":"2024-03-06T07:31:18.460497+0000","last_clean":"2024-03-06T07:31:18.460497+0000","last_became_active":"2024-03-06T07:31:16.940449+0000","last_became_peered":"2024-03-06T07:31:16.940449+0000","last_unstale":"2024-03-06T07:31:18.460497+0000","last_undegraded":"2024-03-06T07:31:18.460497+0000","last_fullsized":"2024-03-06T07:31:18.460497+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:05.735454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.375010+0000","last_change":"2024-03-06T07:31:16.938688+0000","last_active":"2024-03-06T07:31:18.375010+0000","last_peered":"2024-03-06T07:31:18.375010+0000","last_clean":"2024-03-06T07:31:18.375010+0000","last_became_active":"2024-03-06T07:31:16.938310+0000","last_became_peered":"2024-03-06T07:31:16.938310+0000","last_unstale":"2024-03-06T07:31:18.375010+0000","last_undegraded":"2024-03-06T07:31:18.375010+0000","last_fullsized":"2024-03-06T07:31:18.375010+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:09:25.229376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.939951+0000","last_change":"2024-03-06T07:31:16.944213+0000","last_active":"2024-03-06T07:31:17.939951+0000","last_peered":"2024-03-06T07:31:17.939951+0000","last_clean":"2024-03-06T07:31:17.939951+0000","last_became_active":"2024-03-06T07:31:16.942346+0000","last_became_peered":"2024-03-06T07:31:16.942346+0000","last_unstale":"2024-03-06T07:31:17.939951+0000","last_undegraded":"2024-03-06T07:31:17.939951+0000","last_fullsized":"2024-03-06T07:31:17.939951+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:39.924176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931893+0000","last_change":"2024-03-06T07:31:16.935643+0000","last_active":"2024-03-06T07:31:17.931893+0000","last_peered":"2024-03-06T07:31:17.931893+0000","last_clean":"2024-03-06T07:31:17.931893+0000","last_became_active":"2024-03-06T07:31:16.935386+0000","last_became_peered":"2024-03-06T07:31:16.935386+0000","last_unstale":"2024-03-06T07:31:17.931893+0000","last_undegraded":"2024-03-06T07:31:17.931893+0000","last_fullsized":"2024-03-06T07:31:17.931893+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:01.027585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374744+0000","last_change":"2024-03-06T07:31:16.932549+0000","last_active":"2024-03-06T07:31:18.374744+0000","last_peered":"2024-03-06T07:31:18.374744+0000","last_clean":"2024-03-06T07:31:18.374744+0000","last_became_active":"2024-03-06T07:31:16.927603+0000","last_became_peered":"2024-03-06T07:31:16.927603+0000","last_unstale":"2024-03-06T07:31:18.374744+0000","last_undegraded":"2024-03-06T07:31:18.374744+0000","last_fullsized":"2024-03-06T07:31:18.374744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:27.521768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931845+0000","last_change":"2024-03-06T07:31:16.936988+0000","last_active":"2024-03-06T07:31:17.931845+0000","last_peered":"2024-03-06T07:31:17.931845+0000","last_clean":"2024-03-06T07:31:17.931845+0000","last_became_active":"2024-03-06T07:31:16.936602+0000","last_became_peered":"2024-03-06T07:31:16.936602+0000","last_unstale":"2024-03-06T07:31:17.931845+0000","last_undegraded":"2024-03-06T07:31:17.931845+0000","last_fullsized":"2024-03-06T07:31:17.931845+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:21:25.267067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.939994+0000","last_change":"2024-03-06T07:31:16.936234+0000","last_active":"2024-03-06T07:31:17.939994+0000","last_peered":"2024-03-06T07:31:17.939994+0000","last_clean":"2024-03-06T07:31:17.939994+0000","last_became_active":"2024-03-06T07:31:16.935936+0000","last_became_peered":"2024-03-06T07:31:16.935936+0000","last_unstale":"2024-03-06T07:31:17.939994+0000","last_undegraded":"2024-03-06T07:31:17.939994+0000","last_fullsized":"2024-03-06T07:31:17.939994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:28.354330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940101+0000","last_change":"2024-03-06T07:31:16.932111+0000","last_active":"2024-03-06T07:31:17.940101+0000","last_peered":"2024-03-06T07:31:17.940101+0000","last_clean":"2024-03-06T07:31:17.940101+0000","last_became_active":"2024-03-06T07:31:16.931527+0000","last_became_peered":"2024-03-06T07:31:16.931527+0000","last_unstale":"2024-03-06T07:31:17.940101+0000","last_undegraded":"2024-03-06T07:31:17.940101+0000","last_fullsized":"2024-03-06T07:31:17.940101+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:54.992946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.934793+0000","last_change":"2024-03-06T07:31:16.944825+0000","last_active":"2024-03-06T07:31:17.934793+0000","last_peered":"2024-03-06T07:31:17.934793+0000","last_clean":"2024-03-06T07:31:17.934793+0000","last_became_active":"2024-03-06T07:31:16.943660+0000","last_became_peered":"2024-03-06T07:31:16.943660+0000","last_unstale":"2024-03-06T07:31:17.934793+0000","last_undegraded":"2024-03-06T07:31:17.934793+0000","last_fullsized":"2024-03-06T07:31:17.934793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:04.874439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461483+0000","last_change":"2024-03-06T07:31:16.951046+0000","last_active":"2024-03-06T07:31:18.461483+0000","last_peered":"2024-03-06T07:31:18.461483+0000","last_clean":"2024-03-06T07:31:18.461483+0000","last_became_active":"2024-03-06T07:31:16.950729+0000","last_became_peered":"2024-03-06T07:31:16.950729+0000","last_unstale":"2024-03-06T07:31:18.461483+0000","last_undegraded":"2024-03-06T07:31:18.461483+0000","last_fullsized":"2024-03-06T07:31:18.461483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:46.628312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"68'4","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461965+0000","last_change":"2024-03-06T07:31:15.919730+0000","last_active":"2024-03-06T07:31:18.461965+0000","last_peered":"2024-03-06T07:31:18.461965+0000","last_clean":"2024-03-06T07:31:18.461965+0000","last_became_active":"2024-03-06T07:31:15.919105+0000","last_became_peered":"2024-03-06T07:31:15.919105+0000","last_unstale":"2024-03-06T07:31:18.461965+0000","last_undegraded":"2024-03-06T07:31:18.461965+0000","last_fullsized":"2024-03-06T07:31:18.461965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:37:00.058579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.928433+0000","last_change":"2024-03-06T07:31:16.926586+0000","last_active":"2024-03-06T07:31:16.928433+0000","last_peered":"2024-03-06T07:31:16.928433+0000","last_clean":"2024-03-06T07:31:16.928433+0000","last_became_active":"2024-03-06T07:31:16.925428+0000","last_became_peered":"2024-03-06T07:31:16.925428+0000","last_unstale":"2024-03-06T07:31:16.928433+0000","last_undegraded":"2024-03-06T07:31:16.928433+0000","last_fullsized":"2024-03-06T07:31:16.928433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:23.448908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929690+0000","last_change":"2024-03-06T07:31:15.900554+0000","last_active":"2024-03-06T07:31:17.929690+0000","last_peered":"2024-03-06T07:31:17.929690+0000","last_clean":"2024-03-06T07:31:17.929690+0000","last_became_active":"2024-03-06T07:31:15.900282+0000","last_became_peered":"2024-03-06T07:31:15.900282+0000","last_unstale":"2024-03-06T07:31:17.929690+0000","last_undegraded":"2024-03-06T07:31:17.929690+0000","last_fullsized":"2024-03-06T07:31:17.929690+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:49.734753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935544+0000","last_change":"2024-03-06T07:31:16.945929+0000","last_active":"2024-03-06T07:31:17.935544+0000","last_peered":"2024-03-06T07:31:17.935544+0000","last_clean":"2024-03-06T07:31:17.935544+0000","last_became_active":"2024-03-06T07:31:16.945626+0000","last_became_peered":"2024-03-06T07:31:16.945626+0000","last_unstale":"2024-03-06T07:31:17.935544+0000","last_undegraded":"2024-03-06T07:31:17.935544+0000","last_fullsized":"2024-03-06T07:31:17.935544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:15.703557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374810+0000","last_change":"2024-03-06T07:31:15.917942+0000","last_active":"2024-03-06T07:31:18.374810+0000","last_peered":"2024-03-06T07:31:18.374810+0000","last_clean":"2024-03-06T07:31:18.374810+0000","last_became_active":"2024-03-06T07:31:15.917620+0000","last_became_peered":"2024-03-06T07:31:15.917620+0000","last_unstale":"2024-03-06T07:31:18.374810+0000","last_undegraded":"2024-03-06T07:31:18.374810+0000","last_fullsized":"2024-03-06T07:31:18.374810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:26:27.278046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461312+0000","last_change":"2024-03-06T07:31:16.946454+0000","last_active":"2024-03-06T07:31:18.461312+0000","last_peered":"2024-03-06T07:31:18.461312+0000","last_clean":"2024-03-06T07:31:18.461312+0000","last_became_active":"2024-03-06T07:31:16.946068+0000","last_became_peered":"2024-03-06T07:31:16.946068+0000","last_unstale":"2024-03-06T07:31:18.461312+0000","last_undegraded":"2024-03-06T07:31:18.461312+0000","last_fullsized":"2024-03-06T07:31:18.461312+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:59.541317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935803+0000","last_change":"2024-03-06T07:31:15.910176+0000","last_active":"2024-03-06T07:31:17.935803+0000","last_peered":"2024-03-06T07:31:17.935803+0000","last_clean":"2024-03-06T07:31:17.935803+0000","last_became_active":"2024-03-06T07:31:15.906533+0000","last_became_peered":"2024-03-06T07:31:15.906533+0000","last_unstale":"2024-03-06T07:31:17.935803+0000","last_undegraded":"2024-03-06T07:31:17.935803+0000","last_fullsized":"2024-03-06T07:31:17.935803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:24.640339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940042+0000","last_change":"2024-03-06T07:31:16.929910+0000","last_active":"2024-03-06T07:31:17.940042+0000","last_peered":"2024-03-06T07:31:17.940042+0000","last_clean":"2024-03-06T07:31:17.940042+0000","last_became_active":"2024-03-06T07:31:16.929054+0000","last_became_peered":"2024-03-06T07:31:16.929054+0000","last_unstale":"2024-03-06T07:31:17.940042+0000","last_undegraded":"2024-03-06T07:31:17.940042+0000","last_fullsized":"2024-03-06T07:31:17.940042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:26.827543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931955+0000","last_change":"2024-03-06T07:31:15.908836+0000","last_active":"2024-03-06T07:31:17.931955+0000","last_peered":"2024-03-06T07:31:17.931955+0000","last_clean":"2024-03-06T07:31:17.931955+0000","last_became_active":"2024-03-06T07:31:15.908445+0000","last_became_peered":"2024-03-06T07:31:15.908445+0000","last_unstale":"2024-03-06T07:31:17.931955+0000","last_undegraded":"2024-03-06T07:31:17.931955+0000","last_fullsized":"2024-03-06T07:31:17.931955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:36.450971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374659+0000","last_change":"2024-03-06T07:31:16.937994+0000","last_active":"2024-03-06T07:31:18.374659+0000","last_peered":"2024-03-06T07:31:18.374659+0000","last_clean":"2024-03-06T07:31:18.374659+0000","last_became_active":"2024-03-06T07:31:16.937683+0000","last_became_peered":"2024-03-06T07:31:16.937683+0000","last_unstale":"2024-03-06T07:31:18.374659+0000","last_undegraded":"2024-03-06T07:31:18.374659+0000","last_fullsized":"2024-03-06T07:31:18.374659+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:47.725100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935614+0000","last_change":"2024-03-06T07:31:15.905100+0000","last_active":"2024-03-06T07:31:17.935614+0000","last_peered":"2024-03-06T07:31:17.935614+0000","last_clean":"2024-03-06T07:31:17.935614+0000","last_became_active":"2024-03-06T07:31:15.904742+0000","last_became_peered":"2024-03-06T07:31:15.904742+0000","last_unstale":"2024-03-06T07:31:17.935614+0000","last_undegraded":"2024-03-06T07:31:17.935614+0000","last_fullsized":"2024-03-06T07:31:17.935614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:55.804032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.942677+0000","last_change":"2024-03-06T07:31:16.940268+0000","last_active":"2024-03-06T07:31:16.942677+0000","last_peered":"2024-03-06T07:31:16.942677+0000","last_clean":"2024-03-06T07:31:16.942677+0000","last_became_active":"2024-03-06T07:31:16.937294+0000","last_became_peered":"2024-03-06T07:31:16.937294+0000","last_unstale":"2024-03-06T07:31:16.942677+0000","last_undegraded":"2024-03-06T07:31:16.942677+0000","last_fullsized":"2024-03-06T07:31:16.942677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:41.977774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.903243+0000","last_change":"2024-03-06T07:31:15.910549+0000","last_active":"2024-03-06T07:31:16.903243+0000","last_peered":"2024-03-06T07:31:16.903243+0000","last_clean":"2024-03-06T07:31:16.903243+0000","last_became_active":"2024-03-06T07:31:15.910107+0000","last_became_peered":"2024-03-06T07:31:15.910107+0000","last_unstale":"2024-03-06T07:31:16.903243+0000","last_undegraded":"2024-03-06T07:31:16.903243+0000","last_fullsized":"2024-03-06T07:31:16.903243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:01.856760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.930530+0000","last_change":"2024-03-06T07:31:16.938348+0000","last_active":"2024-03-06T07:31:17.930530+0000","last_peered":"2024-03-06T07:31:17.930530+0000","last_clean":"2024-03-06T07:31:17.930530+0000","last_became_active":"2024-03-06T07:31:16.938056+0000","last_became_peered":"2024-03-06T07:31:16.938056+0000","last_unstale":"2024-03-06T07:31:17.930530+0000","last_undegraded":"2024-03-06T07:31:17.930530+0000","last_fullsized":"2024-03-06T07:31:17.930530+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:36.611432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.905219+0000","last_change":"2024-03-06T07:31:15.901210+0000","last_active":"2024-03-06T07:31:16.905219+0000","last_peered":"2024-03-06T07:31:16.905219+0000","last_clean":"2024-03-06T07:31:16.905219+0000","last_became_active":"2024-03-06T07:31:15.900702+0000","last_became_peered":"2024-03-06T07:31:15.900702+0000","last_unstale":"2024-03-06T07:31:16.905219+0000","last_undegraded":"2024-03-06T07:31:16.905219+0000","last_fullsized":"2024-03-06T07:31:16.905219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:27:12.002175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940493+0000","last_change":"2024-03-06T07:31:16.936707+0000","last_active":"2024-03-06T07:31:17.940493+0000","last_peered":"2024-03-06T07:31:17.940493+0000","last_clean":"2024-03-06T07:31:17.940493+0000","last_became_active":"2024-03-06T07:31:16.936397+0000","last_became_peered":"2024-03-06T07:31:16.936397+0000","last_unstale":"2024-03-06T07:31:17.940493+0000","last_undegraded":"2024-03-06T07:31:17.940493+0000","last_fullsized":"2024-03-06T07:31:17.940493+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:45:36.611432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.375176+0000","last_change":"2024-03-06T07:31:15.906775+0000","last_active":"2024-03-06T07:31:18.375176+0000","last_peered":"2024-03-06T07:31:18.375176+0000","last_clean":"2024-03-06T07:31:18.375176+0000","last_became_active":"2024-03-06T07:31:15.906477+0000","last_became_peered":"2024-03-06T07:31:15.906477+0000","last_unstale":"2024-03-06T07:31:18.375176+0000","last_undegraded":"2024-03-06T07:31:18.375176+0000","last_fullsized":"2024-03-06T07:31:18.375176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:48.759764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461798+0000","last_change":"2024-03-06T07:31:16.954302+0000","last_active":"2024-03-06T07:31:18.461798+0000","last_peered":"2024-03-06T07:31:18.461798+0000","last_clean":"2024-03-06T07:31:18.461798+0000","last_became_active":"2024-03-06T07:31:16.954030+0000","last_became_peered":"2024-03-06T07:31:16.954030+0000","last_unstale":"2024-03-06T07:31:18.461798+0000","last_undegraded":"2024-03-06T07:31:18.461798+0000","last_fullsized":"2024-03-06T07:31:18.461798+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:44.874008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.936278+0000","last_change":"2024-03-06T07:31:15.916683+0000","last_active":"2024-03-06T07:31:17.936278+0000","last_peered":"2024-03-06T07:31:17.936278+0000","last_clean":"2024-03-06T07:31:17.936278+0000","last_became_active":"2024-03-06T07:31:15.916395+0000","last_became_peered":"2024-03-06T07:31:15.916395+0000","last_unstale":"2024-03-06T07:31:17.936278+0000","last_undegraded":"2024-03-06T07:31:17.936278+0000","last_fullsized":"2024-03-06T07:31:17.936278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:23.144468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935792+0000","last_change":"2024-03-06T07:31:16.942206+0000","last_active":"2024-03-06T07:31:17.935792+0000","last_peered":"2024-03-06T07:31:17.935792+0000","last_clean":"2024-03-06T07:31:17.935792+0000","last_became_active":"2024-03-06T07:31:16.941938+0000","last_became_peered":"2024-03-06T07:31:16.941938+0000","last_unstale":"2024-03-06T07:31:17.935792+0000","last_undegraded":"2024-03-06T07:31:17.935792+0000","last_fullsized":"2024-03-06T07:31:17.935792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:37.346212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"68'3","reported_seq":17,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931996+0000","last_change":"2024-03-06T07:31:15.913307+0000","last_active":"2024-03-06T07:31:17.931996+0000","last_peered":"2024-03-06T07:31:17.931996+0000","last_clean":"2024-03-06T07:31:17.931996+0000","last_became_active":"2024-03-06T07:31:15.913035+0000","last_became_peered":"2024-03-06T07:31:15.913035+0000","last_unstale":"2024-03-06T07:31:17.931996+0000","last_undegraded":"2024-03-06T07:31:17.931996+0000","last_fullsized":"2024-03-06T07:31:17.931996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:03.011737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.924220+0000","last_change":"2024-03-06T07:31:16.921205+0000","last_active":"2024-03-06T07:31:16.924220+0000","last_peered":"2024-03-06T07:31:16.924220+0000","last_clean":"2024-03-06T07:31:16.924220+0000","last_became_active":"2024-03-06T07:31:16.920894+0000","last_became_peered":"2024-03-06T07:31:16.920894+0000","last_unstale":"2024-03-06T07:31:16.924220+0000","last_undegraded":"2024-03-06T07:31:16.924220+0000","last_fullsized":"2024-03-06T07:31:16.924220+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:41.460377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"68'6","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.956660+0000","last_change":"2024-03-06T07:31:15.900762+0000","last_active":"2024-03-06T07:31:16.956660+0000","last_peered":"2024-03-06T07:31:16.956660+0000","last_clean":"2024-03-06T07:31:16.956660+0000","last_became_active":"2024-03-06T07:31:15.900473+0000","last_became_peered":"2024-03-06T07:31:15.900473+0000","last_unstale":"2024-03-06T07:31:16.956660+0000","last_undegraded":"2024-03-06T07:31:16.956660+0000","last_fullsized":"2024-03-06T07:31:16.956660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:14.489235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.460526+0000","last_change":"2024-03-06T07:31:16.939904+0000","last_active":"2024-03-06T07:31:18.460526+0000","last_peered":"2024-03-06T07:31:18.460526+0000","last_clean":"2024-03-06T07:31:18.460526+0000","last_became_active":"2024-03-06T07:31:16.939497+0000","last_became_peered":"2024-03-06T07:31:16.939497+0000","last_unstale":"2024-03-06T07:31:18.460526+0000","last_undegraded":"2024-03-06T07:31:18.460526+0000","last_fullsized":"2024-03-06T07:31:18.460526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:29:28.135724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"68'2","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.956372+0000","last_change":"2024-03-06T07:31:15.911227+0000","last_active":"2024-03-06T07:31:16.956372+0000","last_peered":"2024-03-06T07:31:16.956372+0000","last_clean":"2024-03-06T07:31:16.956372+0000","last_became_active":"2024-03-06T07:31:15.910935+0000","last_became_peered":"2024-03-06T07:31:15.910935+0000","last_unstale":"2024-03-06T07:31:16.956372+0000","last_undegraded":"2024-03-06T07:31:16.956372+0000","last_fullsized":"2024-03-06T07:31:16.956372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:56.202287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940164+0000","last_change":"2024-03-06T07:31:16.934005+0000","last_active":"2024-03-06T07:31:17.940164+0000","last_peered":"2024-03-06T07:31:17.940164+0000","last_clean":"2024-03-06T07:31:17.940164+0000","last_became_active":"2024-03-06T07:31:16.933637+0000","last_became_peered":"2024-03-06T07:31:16.933637+0000","last_unstale":"2024-03-06T07:31:17.940164+0000","last_undegraded":"2024-03-06T07:31:17.940164+0000","last_fullsized":"2024-03-06T07:31:17.940164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:24:22.662913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935693+0000","last_change":"2024-03-06T07:31:15.903043+0000","last_active":"2024-03-06T07:31:17.935693+0000","last_peered":"2024-03-06T07:31:17.935693+0000","last_clean":"2024-03-06T07:31:17.935693+0000","last_became_active":"2024-03-06T07:31:15.902732+0000","last_became_peered":"2024-03-06T07:31:15.902732+0000","last_unstale":"2024-03-06T07:31:17.935693+0000","last_undegraded":"2024-03-06T07:31:17.935693+0000","last_fullsized":"2024-03-06T07:31:17.935693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:07:27.633784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929438+0000","last_change":"2024-03-06T07:31:16.939633+0000","last_active":"2024-03-06T07:31:17.929438+0000","last_peered":"2024-03-06T07:31:17.929438+0000","last_clean":"2024-03-06T07:31:17.929438+0000","last_became_active":"2024-03-06T07:31:16.939343+0000","last_became_peered":"2024-03-06T07:31:16.939343+0000","last_unstale":"2024-03-06T07:31:17.929438+0000","last_undegraded":"2024-03-06T07:31:17.929438+0000","last_fullsized":"2024-03-06T07:31:17.929438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:32:14.374916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.939607+0000","last_change":"2024-03-06T07:31:15.913969+0000","last_active":"2024-03-06T07:31:17.939607+0000","last_peered":"2024-03-06T07:31:17.939607+0000","last_clean":"2024-03-06T07:31:17.939607+0000","last_became_active":"2024-03-06T07:31:15.913653+0000","last_became_peered":"2024-03-06T07:31:15.913653+0000","last_unstale":"2024-03-06T07:31:17.939607+0000","last_undegraded":"2024-03-06T07:31:17.939607+0000","last_fullsized":"2024-03-06T07:31:17.939607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:40.366521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.935196+0000","last_change":"2024-03-06T07:31:16.928682+0000","last_active":"2024-03-06T07:31:16.935196+0000","last_peered":"2024-03-06T07:31:16.935196+0000","last_clean":"2024-03-06T07:31:16.935196+0000","last_became_active":"2024-03-06T07:31:16.928384+0000","last_became_peered":"2024-03-06T07:31:16.928384+0000","last_unstale":"2024-03-06T07:31:16.935196+0000","last_undegraded":"2024-03-06T07:31:16.935196+0000","last_fullsized":"2024-03-06T07:31:16.935196+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:27:59.429967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"68'8","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.374201+0000","last_change":"2024-03-06T07:31:15.914916+0000","last_active":"2024-03-06T07:31:18.374201+0000","last_peered":"2024-03-06T07:31:18.374201+0000","last_clean":"2024-03-06T07:31:18.374201+0000","last_became_active":"2024-03-06T07:31:15.914582+0000","last_became_peered":"2024-03-06T07:31:15.914582+0000","last_unstale":"2024-03-06T07:31:18.374201+0000","last_undegraded":"2024-03-06T07:31:18.374201+0000","last_fullsized":"2024-03-06T07:31:18.374201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:32.016395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.375068+0000","last_change":"2024-03-06T07:31:16.942458+0000","last_active":"2024-03-06T07:31:18.375068+0000","last_peered":"2024-03-06T07:31:18.375068+0000","last_clean":"2024-03-06T07:31:18.375068+0000","last_became_active":"2024-03-06T07:31:16.940182+0000","last_became_peered":"2024-03-06T07:31:16.940182+0000","last_unstale":"2024-03-06T07:31:18.375068+0000","last_undegraded":"2024-03-06T07:31:18.375068+0000","last_fullsized":"2024-03-06T07:31:18.375068+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:06.585964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.936445+0000","last_change":"2024-03-06T07:31:15.912672+0000","last_active":"2024-03-06T07:31:17.936445+0000","last_peered":"2024-03-06T07:31:17.936445+0000","last_clean":"2024-03-06T07:31:17.936445+0000","last_became_active":"2024-03-06T07:31:15.912309+0000","last_became_peered":"2024-03-06T07:31:15.912309+0000","last_unstale":"2024-03-06T07:31:17.936445+0000","last_undegraded":"2024-03-06T07:31:17.936445+0000","last_fullsized":"2024-03-06T07:31:17.936445+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:12:37.948126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461998+0000","last_change":"2024-03-06T07:31:16.943649+0000","last_active":"2024-03-06T07:31:18.461998+0000","last_peered":"2024-03-06T07:31:18.461998+0000","last_clean":"2024-03-06T07:31:18.461998+0000","last_became_active":"2024-03-06T07:31:16.943275+0000","last_became_peered":"2024-03-06T07:31:16.943275+0000","last_unstale":"2024-03-06T07:31:18.461998+0000","last_undegraded":"2024-03-06T07:31:18.461998+0000","last_fullsized":"2024-03-06T07:31:18.461998+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:44.919951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.375280+0000","last_change":"2024-03-06T07:31:15.917451+0000","last_active":"2024-03-06T07:31:18.375280+0000","last_peered":"2024-03-06T07:31:18.375280+0000","last_clean":"2024-03-06T07:31:18.375280+0000","last_became_active":"2024-03-06T07:31:15.917076+0000","last_became_peered":"2024-03-06T07:31:15.917076+0000","last_unstale":"2024-03-06T07:31:18.375280+0000","last_undegraded":"2024-03-06T07:31:18.375280+0000","last_fullsized":"2024-03-06T07:31:18.375280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:45.523392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.938055+0000","last_change":"2024-03-06T07:31:16.925969+0000","last_active":"2024-03-06T07:31:16.938055+0000","last_peered":"2024-03-06T07:31:16.938055+0000","last_clean":"2024-03-06T07:31:16.938055+0000","last_became_active":"2024-03-06T07:31:16.925618+0000","last_became_peered":"2024-03-06T07:31:16.925618+0000","last_unstale":"2024-03-06T07:31:16.938055+0000","last_undegraded":"2024-03-06T07:31:16.938055+0000","last_fullsized":"2024-03-06T07:31:16.938055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:56.377142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"68'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.936127+0000","last_change":"2024-03-06T07:31:15.915434+0000","last_active":"2024-03-06T07:31:17.936127+0000","last_peered":"2024-03-06T07:31:17.936127+0000","last_clean":"2024-03-06T07:31:17.936127+0000","last_became_active":"2024-03-06T07:31:15.915119+0000","last_became_peered":"2024-03-06T07:31:15.915119+0000","last_unstale":"2024-03-06T07:31:17.936127+0000","last_undegraded":"2024-03-06T07:31:17.936127+0000","last_fullsized":"2024-03-06T07:31:17.936127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:13.329683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.939807+0000","last_change":"2024-03-06T07:31:16.934397+0000","last_active":"2024-03-06T07:31:16.939807+0000","last_peered":"2024-03-06T07:31:16.939807+0000","last_clean":"2024-03-06T07:31:16.939807+0000","last_became_active":"2024-03-06T07:31:16.934037+0000","last_became_peered":"2024-03-06T07:31:16.934037+0000","last_unstale":"2024-03-06T07:31:16.939807+0000","last_undegraded":"2024-03-06T07:31:16.939807+0000","last_fullsized":"2024-03-06T07:31:16.939807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:32.614453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461078+0000","last_change":"2024-03-06T07:31:15.919406+0000","last_active":"2024-03-06T07:31:18.461078+0000","last_peered":"2024-03-06T07:31:18.461078+0000","last_clean":"2024-03-06T07:31:18.461078+0000","last_became_active":"2024-03-06T07:31:15.919150+0000","last_became_peered":"2024-03-06T07:31:15.919150+0000","last_unstale":"2024-03-06T07:31:18.461078+0000","last_undegraded":"2024-03-06T07:31:18.461078+0000","last_fullsized":"2024-03-06T07:31:18.461078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:20:19.986293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931867+0000","last_change":"2024-03-06T07:31:16.922941+0000","last_active":"2024-03-06T07:31:17.931867+0000","last_peered":"2024-03-06T07:31:17.931867+0000","last_clean":"2024-03-06T07:31:17.931867+0000","last_became_active":"2024-03-06T07:31:16.922632+0000","last_became_peered":"2024-03-06T07:31:16.922632+0000","last_unstale":"2024-03-06T07:31:17.931867+0000","last_undegraded":"2024-03-06T07:31:17.931867+0000","last_fullsized":"2024-03-06T07:31:17.931867+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:22:45.333258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935900+0000","last_change":"2024-03-06T07:31:15.897274+0000","last_active":"2024-03-06T07:31:17.935900+0000","last_peered":"2024-03-06T07:31:17.935900+0000","last_clean":"2024-03-06T07:31:17.935900+0000","last_became_active":"2024-03-06T07:31:15.896726+0000","last_became_peered":"2024-03-06T07:31:15.896726+0000","last_unstale":"2024-03-06T07:31:17.935900+0000","last_undegraded":"2024-03-06T07:31:17.935900+0000","last_fullsized":"2024-03-06T07:31:17.935900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:29.925220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931689+0000","last_change":"2024-03-06T07:31:16.934694+0000","last_active":"2024-03-06T07:31:17.931689+0000","last_peered":"2024-03-06T07:31:17.931689+0000","last_clean":"2024-03-06T07:31:17.931689+0000","last_became_active":"2024-03-06T07:31:16.934468+0000","last_became_peered":"2024-03-06T07:31:16.934468+0000","last_unstale":"2024-03-06T07:31:17.931689+0000","last_undegraded":"2024-03-06T07:31:17.931689+0000","last_fullsized":"2024-03-06T07:31:17.931689+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:49.704941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940361+0000","last_change":"2024-03-06T07:31:15.913905+0000","last_active":"2024-03-06T07:31:17.940361+0000","last_peered":"2024-03-06T07:31:17.940361+0000","last_clean":"2024-03-06T07:31:17.940361+0000","last_became_active":"2024-03-06T07:31:15.913284+0000","last_became_peered":"2024-03-06T07:31:15.913284+0000","last_unstale":"2024-03-06T07:31:17.940361+0000","last_undegraded":"2024-03-06T07:31:17.940361+0000","last_fullsized":"2024-03-06T07:31:17.940361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:54.426080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.931659+0000","last_change":"2024-03-06T07:31:16.934369+0000","last_active":"2024-03-06T07:31:17.931659+0000","last_peered":"2024-03-06T07:31:17.931659+0000","last_clean":"2024-03-06T07:31:17.931659+0000","last_became_active":"2024-03-06T07:31:16.934101+0000","last_became_peered":"2024-03-06T07:31:16.934101+0000","last_unstale":"2024-03-06T07:31:17.931659+0000","last_undegraded":"2024-03-06T07:31:17.931659+0000","last_fullsized":"2024-03-06T07:31:17.931659+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:11:13.320364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929117+0000","last_change":"2024-03-06T07:31:15.901323+0000","last_active":"2024-03-06T07:31:17.929117+0000","last_peered":"2024-03-06T07:31:17.929117+0000","last_clean":"2024-03-06T07:31:17.929117+0000","last_became_active":"2024-03-06T07:31:15.900934+0000","last_became_peered":"2024-03-06T07:31:15.900934+0000","last_unstale":"2024-03-06T07:31:17.929117+0000","last_undegraded":"2024-03-06T07:31:17.929117+0000","last_fullsized":"2024-03-06T07:31:17.929117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:36.331577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929605+0000","last_change":"2024-03-06T07:31:16.927440+0000","last_active":"2024-03-06T07:31:17.929605+0000","last_peered":"2024-03-06T07:31:17.929605+0000","last_clean":"2024-03-06T07:31:17.929605+0000","last_became_active":"2024-03-06T07:31:16.927145+0000","last_became_peered":"2024-03-06T07:31:16.927145+0000","last_unstale":"2024-03-06T07:31:17.929605+0000","last_undegraded":"2024-03-06T07:31:17.929605+0000","last_fullsized":"2024-03-06T07:31:17.929605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:54.992946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.936825+0000","last_change":"2024-03-06T07:31:15.909680+0000","last_active":"2024-03-06T07:31:17.936825+0000","last_peered":"2024-03-06T07:31:17.936825+0000","last_clean":"2024-03-06T07:31:17.936825+0000","last_became_active":"2024-03-06T07:31:15.908456+0000","last_became_peered":"2024-03-06T07:31:15.908456+0000","last_unstale":"2024-03-06T07:31:17.936825+0000","last_undegraded":"2024-03-06T07:31:17.936825+0000","last_fullsized":"2024-03-06T07:31:17.936825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:48.636245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940183+0000","last_change":"2024-03-06T07:31:16.942911+0000","last_active":"2024-03-06T07:31:17.940183+0000","last_peered":"2024-03-06T07:31:17.940183+0000","last_clean":"2024-03-06T07:31:17.940183+0000","last_became_active":"2024-03-06T07:31:16.942607+0000","last_became_peered":"2024-03-06T07:31:16.942607+0000","last_unstale":"2024-03-06T07:31:17.940183+0000","last_undegraded":"2024-03-06T07:31:17.940183+0000","last_fullsized":"2024-03-06T07:31:17.940183+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:44.209373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"69'1","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.940683+0000","last_change":"2024-03-06T07:31:15.913471+0000","last_active":"2024-03-06T07:31:17.940683+0000","last_peered":"2024-03-06T07:31:17.940683+0000","last_clean":"2024-03-06T07:31:17.940683+0000","last_became_active":"2024-03-06T07:31:15.913173+0000","last_became_peered":"2024-03-06T07:31:15.913173+0000","last_unstale":"2024-03-06T07:31:17.940683+0000","last_undegraded":"2024-03-06T07:31:17.940683+0000","last_fullsized":"2024-03-06T07:31:17.940683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:36.331577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461049+0000","last_change":"2024-03-06T07:31:16.954697+0000","last_active":"2024-03-06T07:31:18.461049+0000","last_peered":"2024-03-06T07:31:18.461049+0000","last_clean":"2024-03-06T07:31:18.461049+0000","last_became_active":"2024-03-06T07:31:16.954080+0000","last_became_peered":"2024-03-06T07:31:16.954080+0000","last_unstale":"2024-03-06T07:31:18.461049+0000","last_undegraded":"2024-03-06T07:31:18.461049+0000","last_fullsized":"2024-03-06T07:31:18.461049+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:02:43.387237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.910951+0000","last_change":"2024-03-06T07:31:15.904132+0000","last_active":"2024-03-06T07:31:16.910951+0000","last_peered":"2024-03-06T07:31:16.910951+0000","last_clean":"2024-03-06T07:31:16.910951+0000","last_became_active":"2024-03-06T07:31:15.903817+0000","last_became_peered":"2024-03-06T07:31:15.903817+0000","last_unstale":"2024-03-06T07:31:16.910951+0000","last_undegraded":"2024-03-06T07:31:16.910951+0000","last_fullsized":"2024-03-06T07:31:16.910951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:00.101871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.461641+0000","last_change":"2024-03-06T07:31:16.941957+0000","last_active":"2024-03-06T07:31:18.461641+0000","last_peered":"2024-03-06T07:31:18.461641+0000","last_clean":"2024-03-06T07:31:18.461641+0000","last_became_active":"2024-03-06T07:31:16.940723+0000","last_became_peered":"2024-03-06T07:31:16.940723+0000","last_unstale":"2024-03-06T07:31:18.461641+0000","last_undegraded":"2024-03-06T07:31:18.461641+0000","last_fullsized":"2024-03-06T07:31:18.461641+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:27.760025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.929451+0000","last_change":"2024-03-06T07:31:15.910213+0000","last_active":"2024-03-06T07:31:17.929451+0000","last_peered":"2024-03-06T07:31:17.929451+0000","last_clean":"2024-03-06T07:31:17.929451+0000","last_became_active":"2024-03-06T07:31:15.909858+0000","last_became_peered":"2024-03-06T07:31:15.909858+0000","last_unstale":"2024-03-06T07:31:17.929451+0000","last_undegraded":"2024-03-06T07:31:17.929451+0000","last_fullsized":"2024-03-06T07:31:17.929451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:45.658770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.930066+0000","last_change":"2024-03-06T07:31:16.926862+0000","last_active":"2024-03-06T07:31:16.930066+0000","last_peered":"2024-03-06T07:31:16.930066+0000","last_clean":"2024-03-06T07:31:16.930066+0000","last_became_active":"2024-03-06T07:31:16.926344+0000","last_became_peered":"2024-03-06T07:31:16.926344+0000","last_unstale":"2024-03-06T07:31:16.930066+0000","last_undegraded":"2024-03-06T07:31:16.930066+0000","last_fullsized":"2024-03-06T07:31:16.930066+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:10.875725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.911800+0000","last_change":"2024-03-06T07:31:15.916239+0000","last_active":"2024-03-06T07:31:16.911800+0000","last_peered":"2024-03-06T07:31:16.911800+0000","last_clean":"2024-03-06T07:31:16.911800+0000","last_became_active":"2024-03-06T07:31:15.915781+0000","last_became_peered":"2024-03-06T07:31:15.915781+0000","last_unstale":"2024-03-06T07:31:16.911800+0000","last_undegraded":"2024-03-06T07:31:16.911800+0000","last_fullsized":"2024-03-06T07:31:16.911800+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:50.813809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935206+0000","last_change":"2024-03-06T07:31:16.945431+0000","last_active":"2024-03-06T07:31:17.935206+0000","last_peered":"2024-03-06T07:31:17.935206+0000","last_clean":"2024-03-06T07:31:17.935206+0000","last_became_active":"2024-03-06T07:31:16.945102+0000","last_became_peered":"2024-03-06T07:31:16.945102+0000","last_unstale":"2024-03-06T07:31:17.935206+0000","last_undegraded":"2024-03-06T07:31:17.935206+0000","last_fullsized":"2024-03-06T07:31:17.935206+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:40:41.381155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.903175+0000","last_change":"2024-03-06T07:31:15.907149+0000","last_active":"2024-03-06T07:31:16.903175+0000","last_peered":"2024-03-06T07:31:16.903175+0000","last_clean":"2024-03-06T07:31:16.903175+0000","last_became_active":"2024-03-06T07:31:15.906805+0000","last_became_peered":"2024-03-06T07:31:15.906805+0000","last_unstale":"2024-03-06T07:31:16.903175+0000","last_undegraded":"2024-03-06T07:31:16.903175+0000","last_fullsized":"2024-03-06T07:31:16.903175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:49:41.306996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-03-06T07:31:16.938127+0000","last_change":"2024-03-06T07:31:16.930343+0000","last_active":"2024-03-06T07:31:16.938127+0000","last_peered":"2024-03-06T07:31:16.938127+0000","last_clean":"2024-03-06T07:31:16.938127+0000","last_became_active":"2024-03-06T07:31:16.928375+0000","last_became_peered":"2024-03-06T07:31:16.928375+0000","last_unstale":"2024-03-06T07:31:16.938127+0000","last_undegraded":"2024-03-06T07:31:16.938127+0000","last_fullsized":"2024-03-06T07:31:16.938127+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:37:56.648903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.373936+0000","last_change":"2024-03-06T07:31:15.913686+0000","last_active":"2024-03-06T07:31:18.373936+0000","last_peered":"2024-03-06T07:31:18.373936+0000","last_clean":"2024-03-06T07:31:18.373936+0000","last_became_active":"2024-03-06T07:31:15.913337+0000","last_became_peered":"2024-03-06T07:31:15.913337+0000","last_unstale":"2024-03-06T07:31:18.373936+0000","last_undegraded":"2024-03-06T07:31:18.373936+0000","last_fullsized":"2024-03-06T07:31:18.373936+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:13.102146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.934508+0000","last_change":"2024-03-06T07:31:16.945137+0000","last_active":"2024-03-06T07:31:17.934508+0000","last_peered":"2024-03-06T07:31:17.934508+0000","last_clean":"2024-03-06T07:31:17.934508+0000","last_became_active":"2024-03-06T07:31:16.942621+0000","last_became_peered":"2024-03-06T07:31:16.942621+0000","last_unstale":"2024-03-06T07:31:17.934508+0000","last_undegraded":"2024-03-06T07:31:17.934508+0000","last_fullsized":"2024-03-06T07:31:17.934508+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:45.907428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.934950+0000","last_change":"2024-03-06T07:31:15.898982+0000","last_active":"2024-03-06T07:31:17.934950+0000","last_peered":"2024-03-06T07:31:17.934950+0000","last_clean":"2024-03-06T07:31:17.934950+0000","last_became_active":"2024-03-06T07:31:15.898591+0000","last_became_peered":"2024-03-06T07:31:15.898591+0000","last_unstale":"2024-03-06T07:31:17.934950+0000","last_undegraded":"2024-03-06T07:31:17.934950+0000","last_fullsized":"2024-03-06T07:31:17.934950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:49:04.072877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.932166+0000","last_change":"2024-03-06T07:31:16.934899+0000","last_active":"2024-03-06T07:31:17.932166+0000","last_peered":"2024-03-06T07:31:17.932166+0000","last_clean":"2024-03-06T07:31:17.932166+0000","last_became_active":"2024-03-06T07:31:16.934558+0000","last_became_peered":"2024-03-06T07:31:16.934558+0000","last_unstale":"2024-03-06T07:31:17.932166+0000","last_undegraded":"2024-03-06T07:31:17.932166+0000","last_fullsized":"2024-03-06T07:31:17.932166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:15.871084+0000","last_clean_scrub_stamp":"2024-03-06T07:31:15.871084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:10.993565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"68'2","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:17.935528+0000","last_change":"2024-03-06T07:31:15.911459+0000","last_active":"2024-03-06T07:31:17.935528+0000","last_peered":"2024-03-06T07:31:17.935528+0000","last_clean":"2024-03-06T07:31:17.935528+0000","last_became_active":"2024-03-06T07:31:15.911167+0000","last_became_peered":"2024-03-06T07:31:15.911167+0000","last_unstale":"2024-03-06T07:31:17.935528+0000","last_undegraded":"2024-03-06T07:31:17.935528+0000","last_fullsized":"2024-03-06T07:31:17.935528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:31:14.856450+0000","last_clean_scrub_stamp":"2024-03-06T07:31:14.856450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:35:43.281245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":257,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:31:18.460591+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:31:18.460591+0000","last_peered":"2024-03-06T07:31:18.460591+0000","last_clean":"2024-03-06T07:31:18.460591+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:31:18.460591+0000","last_undegraded":"2024-03-06T07:31:18.460591+0000","last_fullsized":"2024-03-06T07:31:18.460591+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509490,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":335632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":806912,"data_stored":335632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542195,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36740,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335100,"statfs":{"total":96636764160,"available":96599142400,"internally_reserved":0,"allocated":1433600,"data_stored":939691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542195,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":348799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542196,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":835584,"data_stored":348797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542195,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36164,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335676,"statfs":{"total":96636764160,"available":96599732224,"internally_reserved":0,"allocated":835584,"data_stored":348777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542196,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":839680,"data_stored":348819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542196,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1425408,"data_stored":939075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:31:22.038 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:31:22.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:31:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.316+0000 7fc5b9200640 1 Processor -- start 2024-03-06T07:31:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.316+0000 7fc5b9200640 1 -- start start 2024-03-06T07:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b9200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5b415b600 0x7fc5b415ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40b2c80 0x7fc5b40b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b9200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b415f840 con 0x7fc5b40cee80 2024-03-06T07:31:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b9200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b415f3b0 con 0x7fc5b415b600 2024-03-06T07:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b415ea80 con 0x7fc5b40b2c80 2024-03-06T07:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5a9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc5b415b600 0x7fc5b415ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc5b40cee80 0x7fc5b40cf260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47048/0 (socket says 172.21.15.6:47048) 2024-03-06T07:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.318+0000 7fc5b1e00640 1 -- 172.21.15.6:0/977678366 learned_addr learned my addr 172.21.15.6:0/977678366 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.319+0000 7fc5b2800640 1 --2- 172.21.15.6:0/977678366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40b2c80 0x7fc5b40b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.319+0000 7fc5a9400640 1 -- 172.21.15.6:0/977678366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40b2c80 msgr2=0x7fc5b40b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.319+0000 7fc5a9400640 1 --2- 172.21.15.6:0/977678366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40b2c80 0x7fc5b40b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.319+0000 7fc5a9400640 1 -- 172.21.15.6:0/977678366 >> v1:172.21.15.6:6789/0 conn(0x7fc5b40cee80 legacy=0x7fc5b40cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.319+0000 7fc5a9400640 1 -- 172.21.15.6:0/977678366 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b415ed60 con 0x7fc5b415b600 2024-03-06T07:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.320+0000 7fc5a9400640 1 --2- 172.21.15.6:0/977678366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b415ba60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc59c0097a0 tx=0x7fc59c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a641bd4e32e25a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.320+0000 7fc5b1400640 1 -- 172.21.15.6:0/977678366 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c031780 con 0x7fc5b415b600 2024-03-06T07:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.320+0000 7fc5b1400640 1 -- 172.21.15.6:0/977678366 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59c037070 con 0x7fc5b415b600 2024-03-06T07:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.320+0000 7fc5b1400640 1 -- 172.21.15.6:0/977678366 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59c031b40 con 0x7fc5b415b600 2024-03-06T07:31:22.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 -- 172.21.15.6:0/977678366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 msgr2=0x7fc5b415ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 --2- 172.21.15.6:0/977678366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b415ba60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc59c0097a0 tx=0x7fc59c02b820 comp rx=0 tx=0).stop 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 -- 172.21.15.6:0/977678366 shutdown_connections 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 --2- 172.21.15.6:0/977678366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40b2c80 0x7fc5b40b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 --2- 172.21.15.6:0/977678366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b415ba60 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 -- 172.21.15.6:0/977678366 >> 172.21.15.6:0/977678366 conn(0x7fc5b406b3d0 msgr2=0x7fc5b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 -- 172.21.15.6:0/977678366 shutdown_connections 2024-03-06T07:31:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.321+0000 7fc5b9200640 1 -- 172.21.15.6:0/977678366 wait complete. 2024-03-06T07:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.323+0000 7fc5b9200640 1 Processor -- start 2024-03-06T07:31:22.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.323+0000 7fc5b9200640 1 -- start start 2024-03-06T07:31:22.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b9200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5b40b2c80 0x7fc5b4143550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40cee80 0x7fc5b409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b40afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40ce930 con 0x7fc5b415b600 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b9200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b415bfa0 con 0x7fc5b40b2c80 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40ceab0 con 0x7fc5b40cee80 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b40afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b40afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48320/0 (socket says 172.21.15.6:48320) 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.324+0000 7fc5b2800640 1 -- 172.21.15.6:0/2648562397 learned_addr learned my addr 172.21.15.6:0/2648562397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5a9400640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40cee80 0x7fc5b409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b1e00640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5b40b2c80 0x7fc5b4143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b2800640 1 -- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40cee80 msgr2=0x7fc5b409aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b2800640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40cee80 0x7fc5b409aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b2800640 1 -- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5b40b2c80 msgr2=0x7fc5b4143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b2800640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5b40b2c80 0x7fc5b4143550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b2800640 1 -- 172.21.15.6:0/2648562397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b409cf70 con 0x7fc5b415b600 2024-03-06T07:31:22.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b1e00640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5b40b2c80 0x7fc5b4143550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:22.330 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5b2800640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b40afce0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac007e90 tx=0x7fc5ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac014420 con 0x7fc5b415b600 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.325+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac019070 con 0x7fc5b415b600 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.326+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b415a8d0 con 0x7fc5b415b600 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.326+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac014800 con 0x7fc5b415b600 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.326+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b409d1f0 con 0x7fc5b415b600 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.327+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b40acf60 con 0x7fc5b415b600 2024-03-06T07:31:22.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.328+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc5ac0149a0 con 0x7fc5b415b600 2024-03-06T07:31:22.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.329+0000 7fc5aaa00640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc59006b4d0 0x7fc59006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.330+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5ac08aa40 con 0x7fc5b415b600 2024-03-06T07:31:22.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.330+0000 7fc5b1e00640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc59006b4d0 0x7fc59006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.331+0000 7fc5b1e00640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc59006b4d0 0x7fc59006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4000f60 tx=0x7fc5a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.334+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5b40acf60 con 0x7fc5b415b600 2024-03-06T07:31:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.612+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc5b4068010 con 0x7fc5b415b600 2024-03-06T07:31:22.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.616+0000 7fc5aaa00640 1 -- 172.21.15.6:0/2648562397 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc5b409cf70 con 0x7fc5b415b600 2024-03-06T07:31:22.618 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:31:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:31:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:31:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:31:22.618 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:31:22.619 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:31:22.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.618+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc59006b4d0 msgr2=0x7fc59006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.618+0000 7fc5b9200640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc59006b4d0 0x7fc59006d990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4000f60 tx=0x7fc5a4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:22.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.618+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 msgr2=0x7fc5b40afce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.618+0000 7fc5b9200640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b40afce0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac007e90 tx=0x7fc5ac00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 shutdown_connections 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc59006b4d0 0x7fc59006d990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5b415b600 0x7fc5b40afce0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5b40cee80 0x7fc5b409aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 --2- 172.21.15.6:0/2648562397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5b40b2c80 0x7fc5b4143550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 >> 172.21.15.6:0/2648562397 conn(0x7fc5b406b3d0 msgr2=0x7fc5b406de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 shutdown_connections 2024-03-06T07:31:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.619+0000 7fc5b9200640 1 -- 172.21.15.6:0/2648562397 wait complete. 2024-03-06T07:31:22.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:22.855 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-06T07:31:22.853+0000 7f4703e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:31:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.856+0000 7fb6e7200640 1 Processor -- start 2024-03-06T07:31:22.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.856+0000 7fb6e7200640 1 -- start start 2024-03-06T07:31:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.858+0000 7fb6e7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d4a10 0x7fb6e80d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.858+0000 7fb6e7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e80b3710 con 0x7fb6e80d0c80 2024-03-06T07:31:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.858+0000 7fb6e7200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e80b39f0 con 0x7fb6e815e0b0 2024-03-06T07:31:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.858+0000 7fb6e7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80b09e0 con 0x7fb6e80d4a10 2024-03-06T07:31:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.858+0000 7fb6e6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb6e80d0c80 0x7fb6e80b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47064/0 (socket says 172.21.15.6:47064) 2024-03-06T07:31:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.858+0000 7fb6e6800640 1 -- 172.21.15.6:0/285045468 learned_addr learned my addr 172.21.15.6:0/285045468 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.859+0000 7fb6dd400640 1 --2- 172.21.15.6:0/285045468 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d4a10 0x7fb6e80d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.859+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702883410 0 0) 0x7fb6e80b3710 con 0x7fb6e80d0c80 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.859+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6cc003520 con 0x7fb6e80d0c80 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.859+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845278586 0 0) 0x7fb6e80b39f0 con 0x7fb6e815e0b0 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.859+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e80b3710 con 0x7fb6e815e0b0 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.860+0000 7fb6dd400640 1 -- 172.21.15.6:0/285045468 >> v1:172.21.15.178:6789/0 conn(0x7fb6e815e0b0 legacy=0x7fb6e80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.860+0000 7fb6dd400640 1 -- 172.21.15.6:0/285045468 >> v1:172.21.15.6:6789/0 conn(0x7fb6e80d0c80 legacy=0x7fb6e80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.860+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1432133762 0 0) 0x7fb6cc003520 con 0x7fb6e80d0c80 2024-03-06T07:31:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.860+0000 7fb6dd400640 1 -- 172.21.15.6:0/285045468 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e80b3cd0 con 0x7fb6e80d4a10 2024-03-06T07:31:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.861+0000 7fb6dd400640 1 --2- 172.21.15.6:0/285045468 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d4a10 0x7fb6e80d0740 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00097d0 tx=0x7fb6d002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae703991da21850 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0037070 con 0x7fb6e80d4a10 2024-03-06T07:31:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d0030510 con 0x7fb6e80d4a10 2024-03-06T07:31:22.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e5400640 1 -- 172.21.15.6:0/285045468 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00015a0 con 0x7fb6e80d4a10 2024-03-06T07:31:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e7200640 1 -- 172.21.15.6:0/285045468 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d4a10 msgr2=0x7fb6e80d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e7200640 1 --2- 172.21.15.6:0/285045468 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d4a10 0x7fb6e80d0740 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00097d0 tx=0x7fb6d002b800 comp rx=0 tx=0).stop 2024-03-06T07:31:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e7200640 1 -- 172.21.15.6:0/285045468 shutdown_connections 2024-03-06T07:31:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e7200640 1 --2- 172.21.15.6:0/285045468 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d4a10 0x7fb6e80d0740 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e7200640 1 -- 172.21.15.6:0/285045468 >> 172.21.15.6:0/285045468 conn(0x7fb6e80d3d30 msgr2=0x7fb6e80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:22.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.862+0000 7fb6e7200640 1 -- 172.21.15.6:0/285045468 shutdown_connections 2024-03-06T07:31:22.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.863+0000 7fb6e7200640 1 -- 172.21.15.6:0/285045468 wait complete. 2024-03-06T07:31:22.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.864+0000 7fb6e7200640 1 Processor -- start 2024-03-06T07:31:22.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.864+0000 7fb6e7200640 1 -- start start 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6e7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d0c80 0x7fb6e815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6e7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 0x7fb6e809daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6e7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6e815e0b0 0x7fb6e809f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6e7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e815e490 con 0x7fb6e80d4a10 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6e7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80cfa20 con 0x7fb6e815e0b0 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6e7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e80cfc60 con 0x7fb6e80d0c80 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.865+0000 7fb6dd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 0x7fb6e809daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 0x7fb6e809daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48324/0 (socket says 172.21.15.6:48324) 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 -- 172.21.15.6:0/2003105557 learned_addr learned my addr 172.21.15.6:0/2003105557 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 -- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d0c80 msgr2=0x7fb6e815c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d0c80 0x7fb6e815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 -- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6e815e0b0 msgr2=0x7fb6e809f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6e815e0b0 0x7fb6e809f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 -- 172.21.15.6:0/2003105557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e80a0400 con 0x7fb6e80d4a10 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.866+0000 7fb6dd400640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 0x7fb6e809daf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0035040 tx=0x7fb6d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.867+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0040020 con 0x7fb6e80d4a10 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.867+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d0038040 con 0x7fb6e80d4a10 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.867+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e80a06e0 con 0x7fb6e80d4a10 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.867+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d0037070 con 0x7fb6e80d4a10 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.867+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e80a0ca0 con 0x7fb6e80d4a10 2024-03-06T07:31:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.868+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e804f330 con 0x7fb6e80d4a10 2024-03-06T07:31:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.870+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6d0043020 con 0x7fb6e80d4a10 2024-03-06T07:31:22.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.871+0000 7fb6dea00640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6c406b4d0 0x7fb6c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:22.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.871+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6d00ab6d0 con 0x7fb6e80d4a10 2024-03-06T07:31:22.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.872+0000 7fb6e5e00640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6c406b4d0 0x7fb6c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:22.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.873+0000 7fb6e5e00640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6c406b4d0 0x7fb6c406d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8001460 tx=0x7fb6d8007cd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:22.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:22.876+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6d004a030 con 0x7fb6e80d4a10 2024-03-06T07:31:23.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.115+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6e80b6420 con 0x7fb6e80d4a10 2024-03-06T07:31:23.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.117+0000 7fb6dea00640 1 -- 172.21.15.6:0/2003105557 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fb6d007a540 con 0x7fb6e80d4a10 2024-03-06T07:31:23.119 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:23.119 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:23.119 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:23.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6c406b4d0 msgr2=0x7fb6c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6c406b4d0 0x7fb6c406d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8001460 tx=0x7fb6d8007cd0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 msgr2=0x7fb6e809daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 0x7fb6e809daf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0035040 tx=0x7fb6d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 shutdown_connections 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6c406b4d0 0x7fb6c406d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6e815e0b0 0x7fb6e809f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6e80d4a10 0x7fb6e809daf0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 --2- 172.21.15.6:0/2003105557 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6e80d0c80 0x7fb6e815c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 >> 172.21.15.6:0/2003105557 conn(0x7fb6e80d3d30 msgr2=0x7fb6e80cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 shutdown_connections 2024-03-06T07:31:23.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.119+0000 7fb6e7200640 1 -- 172.21.15.6:0/2003105557 wait complete. 2024-03-06T07:31:23.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:31:16.361554+0000', 'modified': '2024-03-06T07:31:19.436145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6402}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6402': {'gid': 6402, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6835/4173228924', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 4173228924}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 4173228924}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6402, 'qdb_cluster': [6402]} 2024-03-06T07:31:23.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:31:23.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:31:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.776+0000 7f2232200640 1 Processor -- start 2024-03-06T07:31:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.777+0000 7f2232200640 1 -- start start 2024-03-06T07:31:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f2232200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f222c15e2e0 0x7f222c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f2232200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f222c0b50e0 0x7f222c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f2232200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c0b3940 con 0x7f222c0d0d70 2024-03-06T07:31:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f2232200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222c0d4e70 con 0x7f222c0b50e0 2024-03-06T07:31:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f2232200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222c0b3c40 con 0x7f222c15e2e0 2024-03-06T07:31:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f222be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f222c0b50e0 0x7f222c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f222be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f222c0b50e0 0x7f222c0d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48332/0 (socket says 172.21.15.6:48332) 2024-03-06T07:31:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.778+0000 7f222be00640 1 -- 172.21.15.6:0/4140524306 learned_addr learned my addr 172.21.15.6:0/4140524306 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.779+0000 7f222be00640 1 -- 172.21.15.6:0/4140524306 >> v2:172.21.15.178:3300/0 conn(0x7f222c15e2e0 msgr2=0x7f222c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.779+0000 7f222be00640 1 --2- 172.21.15.6:0/4140524306 >> v2:172.21.15.178:3300/0 conn(0x7f222c15e2e0 0x7f222c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.779+0000 7f222be00640 1 -- 172.21.15.6:0/4140524306 >> v1:172.21.15.178:6789/0 conn(0x7f222c0d0d70 legacy=0x7f222c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.779+0000 7f222be00640 1 -- 172.21.15.6:0/4140524306 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0cf580 con 0x7f222c0b50e0 2024-03-06T07:31:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.780+0000 7f222be00640 1 --2- 172.21.15.6:0/4140524306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0b50e0 0x7f222c0d11d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f222c08d8b0 tx=0x7f222402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd2383831898a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.781+0000 7f222a000640 1 -- 172.21.15.6:0/4140524306 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222402ca80 con 0x7f222c0b50e0 2024-03-06T07:31:23.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.781+0000 7f222a000640 1 -- 172.21.15.6:0/4140524306 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222402cc20 con 0x7f222c0b50e0 2024-03-06T07:31:23.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.781+0000 7f222a000640 1 -- 172.21.15.6:0/4140524306 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240343f0 con 0x7f222c0b50e0 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.781+0000 7f2232200640 1 -- 172.21.15.6:0/4140524306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0b50e0 msgr2=0x7f222c0d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.781+0000 7f2232200640 1 --2- 172.21.15.6:0/4140524306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0b50e0 0x7f222c0d11d0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f222c08d8b0 tx=0x7f222402bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.782+0000 7f2232200640 1 -- 172.21.15.6:0/4140524306 shutdown_connections 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.782+0000 7f2232200640 1 --2- 172.21.15.6:0/4140524306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0b50e0 0x7f222c0d11d0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.782+0000 7f2232200640 1 --2- 172.21.15.6:0/4140524306 >> v2:172.21.15.178:3300/0 conn(0x7f222c15e2e0 0x7f222c0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.782+0000 7f2232200640 1 -- 172.21.15.6:0/4140524306 >> 172.21.15.6:0/4140524306 conn(0x7f222c0d3f60 msgr2=0x7f222c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.782+0000 7f2232200640 1 -- 172.21.15.6:0/4140524306 shutdown_connections 2024-03-06T07:31:23.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.782+0000 7f2232200640 1 -- 172.21.15.6:0/4140524306 wait complete. 2024-03-06T07:31:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.784+0000 7f2232200640 1 Processor -- start 2024-03-06T07:31:23.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.784+0000 7f2232200640 1 -- start start 2024-03-06T07:31:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f2232200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f222c0b50e0 0x7f222c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f2232200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 0x7f222c1f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f2232200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f222c15e2e0 0x7f222c0cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f2232200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0cfe20 con 0x7f222c0d0d70 2024-03-06T07:31:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f2232200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0cfb40 con 0x7f222c0b50e0 2024-03-06T07:31:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f2232200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0d2240 con 0x7f222c15e2e0 2024-03-06T07:31:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f222aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 0x7f222c1f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f222aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 0x7f222c1f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48338/0 (socket says 172.21.15.6:48338) 2024-03-06T07:31:23.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.785+0000 7f222aa00640 1 -- 172.21.15.6:0/3678686389 learned_addr learned my addr 172.21.15.6:0/3678686389 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222be00640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f222c15e2e0 0x7f222c0cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222b400640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f222c0b50e0 0x7f222c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222aa00640 1 -- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f222c15e2e0 msgr2=0x7f222c0cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222aa00640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f222c15e2e0 0x7f222c0cb030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222aa00640 1 -- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f222c0b50e0 msgr2=0x7f222c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222aa00640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f222c0b50e0 0x7f222c0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222aa00640 1 -- 172.21.15.6:0/3678686389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0cdbd0 con 0x7f222c0d0d70 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222b400640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f222c0b50e0 0x7f222c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f222aa00640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 0x7f222c1f69a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f221800c460 tx=0x7f221800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f2223400640 1 -- 172.21.15.6:0/3678686389 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218016020 con 0x7f222c0d0d70 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f2223400640 1 -- 172.21.15.6:0/3678686389 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2218012870 con 0x7f222c0d0d70 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.786+0000 7f2223400640 1 -- 172.21.15.6:0/3678686389 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218012c60 con 0x7f222c0d0d70 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.787+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c0cdeb0 con 0x7f222c0d0d70 2024-03-06T07:31:23.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.787+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c1f6ee0 con 0x7f222c0d0d70 2024-03-06T07:31:23.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.789+0000 7f2223400640 1 -- 172.21.15.6:0/3678686389 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2218019020 con 0x7f222c0d0d70 2024-03-06T07:31:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.790+0000 7f2223400640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f220c06b4d0 0x7f220c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.790+0000 7f2223400640 1 -- 172.21.15.6:0/3678686389 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2218089f20 con 0x7f222c0d0d70 2024-03-06T07:31:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.791+0000 7f222b400640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f220c06b4d0 0x7f220c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.791+0000 7f2234fb7740 1 Processor -- start 2024-03-06T07:31:23.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.791+0000 7f2234fb7740 1 -- start start 2024-03-06T07:31:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f222b400640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f220c06b4d0 0x7f220c06d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f221c00a870 tx=0x7f221c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0eb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 0x56522d0eba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f2234fb7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522d025600 con 0x56522d0bbc90 2024-03-06T07:31:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522cfce930 con 0x56522d0e87b0 2024-03-06T07:31:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x56522d0216a0 con 0x56522d0e8b90 2024-03-06T07:31:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f222be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 0x56522d0eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f222be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 0x56522d0eba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48352/0 (socket says 172.21.15.6:48352) 2024-03-06T07:31:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f222be00640 1 -- 172.21.15.6:0/3011071700 learned_addr learned my addr 172.21.15.6:0/3011071700 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f222aa00640 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.792+0000 7f222b400640 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f222be00640 1 -- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 msgr2=0x56522d0eb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f222be00640 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0eb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f222be00640 1 -- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 msgr2=0x56522d0bb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f222be00640 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f222be00640 1 -- 172.21.15.6:0/3011071700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56522cd3a2d0 con 0x56522d0bbc90 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f222be00640 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 0x56522d0eba80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2224009f40 tx=0x7f222402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fa0ba2da3e39dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f2217400640 1 -- 172.21.15.6:0/3011071700 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224045070 con 0x56522d0bbc90 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f2217400640 1 -- 172.21.15.6:0/3011071700 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222402caa0 con 0x56522d0bbc90 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.793+0000 7f2217400640 1 -- 172.21.15.6:0/3011071700 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240406b0 con 0x56522d0bbc90 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 -- 172.21.15.6:0/3011071700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 msgr2=0x56522d0eba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 0x56522d0eba80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2224009f40 tx=0x7f222402e000 comp rx=0 tx=0).stop 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 -- 172.21.15.6:0/3011071700 shutdown_connections 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0bbc90 0x56522d0eba80 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0eb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3011071700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 -- 172.21.15.6:0/3011071700 >> 172.21.15.6:0/3011071700 conn(0x56522d0bd9c0 msgr2=0x56522d106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 -- 172.21.15.6:0/3011071700 shutdown_connections 2024-03-06T07:31:23.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.794+0000 7f2234fb7740 1 -- 172.21.15.6:0/3011071700 wait complete. 2024-03-06T07:31:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.795+0000 7f2234fb7740 1 -- start start 2024-03-06T07:31:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.795+0000 7f2234fb7740 1 Processor -- start 2024-03-06T07:31:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0e1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0e3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522d0bff60 con 0x56522d0e8b90 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522cfe1d90 con 0x56522d0e0c50 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x56522d0bf350 con 0x56522d0e87b0 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f2234fb7740 10 client.0 _ll_register_callbacks cb 0x56522d01d350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f222aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0e3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f222aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0e3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48354/0 (socket says 172.21.15.6:48354) 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.796+0000 7f222aa00640 1 -- 172.21.15.6:0/1137292995 learned_addr learned my addr 172.21.15.6:0/1137292995 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222b400640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0e1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222be00640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222aa00640 1 -- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 msgr2=0x56522d0e1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222aa00640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0e1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222aa00640 1 -- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 msgr2=0x56522d0e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222aa00640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222aa00640 1 -- 172.21.15.6:0/1137292995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56522ce046b0 con 0x56522d0e8b90 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f222aa00640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0e3a50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f22180583a0 tx=0x7f2218060e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e7b6a03d7059ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f220be00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f220be00640 1 -- 172.21.15.6:0/1137292995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2218086060 con 0x56522d0e8b90 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.797+0000 7f220be00640 1 -- 172.21.15.6:0/1137292995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2218054d10 con 0x56522d0e8b90 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f222402caa0 con 0x56522d0e8b90 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f220be00640 1 -- 172.21.15.6:0/1137292995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2218075050 con 0x56522d0e8b90 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 10 client.6534 fetch_fsmap learned FSMap version 45 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x56522d0e1610 con 0x56522d0e8b90 2024-03-06T07:31:23.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f220be00640 1 -- 172.21.15.6:0/1137292995 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222402caa0 con 0x56522d0e8b90 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f220be00640 1 -- 172.21.15.6:0/1137292995 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 45) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2218057af0 con 0x56522d0e8b90 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 10 client.6534 fetch_fsmap finished waiting for FSMap version 45 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 10 client.6534 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 20 client.6534 populate_metadata read hostname 'smithi006' 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d0ea330 0x56522d0ea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 4 client.6534 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.798+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 --> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56522d032b40 con 0x56522d0ea330 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.799+0000 7f222b400640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d0ea330 0x56522d0ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.799+0000 7f222b400640 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d0ea330 0x56522d0ea710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cd172bc8241ff33 server_cookie=8f0aae31d03c4eb9 in_seq=0 out_seq=0 2024-03-06T07:31:23.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.799+0000 7f220be00640 10 client.6534 ms_handle_connect on v2:172.21.15.6:6834/4173228924 2024-03-06T07:31:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.800+0000 7f220be00640 1 -- 172.21.15.6:0/1137292995 <== mds.0 v2:172.21.15.6:6834/4173228924 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56522d032b40 con 0x56522d0ea330 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.800+0000 7f220be00640 10 client.6534 handle_command_reply: tid=0 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.800+0000 7f2234fb7740 1 client.6534 shutdown 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 msgr2=0x56522d0e3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0e3a50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f22180583a0 tx=0x7f2218060e70 comp rx=0 tx=0).stop 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 shutdown_connections 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d0ea330 0x56522d0ea710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0e3a50 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 --2- 172.21.15.6:0/1137292995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0e1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f220be00640 0 client.6534 ms_handle_reset on v2:172.21.15.6:6834/4173228924 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 >> 172.21.15.6:0/1137292995 conn(0x56522d0bbc90 msgr2=0x56522d1099c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 shutdown_connections 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 1 -- 172.21.15.6:0/1137292995 wait complete. 2024-03-06T07:31:23.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.801+0000 7f2234fb7740 20 client.6534 trim_cache size 0 max 16384 2024-03-06T07:31:23.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.819+0000 7f2234fb7740 1 Processor -- start 2024-03-06T07:31:23.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 -- start start 2024-03-06T07:31:23.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0dfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c200210 con 0x56522d0e8b90 2024-03-06T07:31:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c1ff3b0 con 0x56522d0e0c50 2024-03-06T07:31:23.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.820+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c0ca290 con 0x56522d0e87b0 2024-03-06T07:31:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48370/0 (socket says 172.21.15.6:48370) 2024-03-06T07:31:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 -- 172.21.15.6:0/2874739312 learned_addr learned my addr 172.21.15.6:0/2874739312 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222aa00640 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222b400640 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0dfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 -- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 msgr2=0x56522d0bf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 -- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 msgr2=0x56522d0dfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0dfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222be00640 1 -- 172.21.15.6:0/2874739312 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56522d0eb7f0 con 0x56522d0e8b90 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.821+0000 7f222aa00640 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.822+0000 7f222be00640 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0bf890 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2224040040 tx=0x7f2224009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd2afcaa2dc2463 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.822+0000 7f2216a00640 1 -- 172.21.15.6:0/2874739312 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2224045040 con 0x56522d0e8b90 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.822+0000 7f2216a00640 1 -- 172.21.15.6:0/2874739312 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222402c880 con 0x56522d0e8b90 2024-03-06T07:31:23.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.822+0000 7f2216a00640 1 -- 172.21.15.6:0/2874739312 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222402c560 con 0x56522d0e8b90 2024-03-06T07:31:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 -- 172.21.15.6:0/2874739312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 msgr2=0x56522d0bf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0bf890 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2224040040 tx=0x7f2224009000 comp rx=0 tx=0).stop 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 -- 172.21.15.6:0/2874739312 shutdown_connections 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 --2- 172.21.15.6:0/2874739312 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e0c50 0x56522d0dfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 -- 172.21.15.6:0/2874739312 >> 172.21.15.6:0/2874739312 conn(0x56522d0bbc90 msgr2=0x56522d106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 -- 172.21.15.6:0/2874739312 shutdown_connections 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.823+0000 7f2234fb7740 1 -- 172.21.15.6:0/2874739312 wait complete. 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.824+0000 7f2234fb7740 1 -- start start 2024-03-06T07:31:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.824+0000 7f2234fb7740 1 Processor -- start 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 0x56522d109a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x56522d0ed580 con 0x56522d0e0c50 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c150b90 con 0x56522d0e87b0 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x56522d0ed210 con 0x56522d0e8b90 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f2234fb7740 10 client.0 _ll_register_callbacks cb 0x7f22180581e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:31:23.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f222b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 0x56522d109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f222b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 0x56522d109a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48384/0 (socket says 172.21.15.6:48384) 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.825+0000 7f222b400640 1 -- 172.21.15.6:0/3728908479 learned_addr learned my addr 172.21.15.6:0/3728908479 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f222b400640 1 -- 172.21.15.6:0/3728908479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 msgr2=0x56522d0bf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f222b400640 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f222b400640 1 -- 172.21.15.6:0/3728908479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 msgr2=0x56522d0bf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f222b400640 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f222b400640 1 -- 172.21.15.6:0/3728908479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56522d12a300 con 0x56522d0e0c50 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f222b400640 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 0x56522d109a50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f221c006fd0 tx=0x7f221c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9dc33ca682b6a77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f220be00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f220be00640 1 -- 172.21.15.6:0/3728908479 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c026880 con 0x56522d0e0c50 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f220be00640 1 -- 172.21.15.6:0/3728908479 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f221c030af0 con 0x56522d0e0c50 2024-03-06T07:31:23.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.826+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f222402c880 con 0x56522d0e0c50 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f220be00640 1 -- 172.21.15.6:0/3728908479 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f221c030cf0 con 0x56522d0e0c50 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 10 client.6540 fetch_fsmap learned FSMap version 45 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x56522d11bcb0 con 0x56522d0e0c50 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f220be00640 1 -- 172.21.15.6:0/3728908479 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222402c880 con 0x56522d0e0c50 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f220be00640 1 -- 172.21.15.6:0/3728908479 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 45) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f221c037050 con 0x56522d0e0c50 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 10 client.6540 fetch_fsmap finished waiting for FSMap version 45 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 10 client.6540 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 20 client.6540 populate_metadata read hostname 'smithi006' 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d17eed0 0x56522d17f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 4 client.6540 mds_command: new command op to 6402 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:31:23.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.827+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 --> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56522d02ce90 con 0x56522d17eed0 2024-03-06T07:31:23.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.828+0000 7f222aa00640 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d17eed0 0x56522d17f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.828+0000 7f222aa00640 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d17eed0 0x56522d17f2b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8187123e0f635054 server_cookie=b16d626ac2fdf71e in_seq=0 out_seq=0 2024-03-06T07:31:23.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.828+0000 7f220be00640 10 client.6540 ms_handle_connect on v2:172.21.15.6:6834/4173228924 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.828+0000 7f220be00640 1 -- 172.21.15.6:0/3728908479 <== mds.0 v2:172.21.15.6:6834/4173228924 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56522d02ce90 con 0x56522d17eed0 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.828+0000 7f220be00640 10 client.6540 handle_command_reply: tid=0 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.828+0000 7f2234fb7740 1 client.6540 shutdown 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 msgr2=0x56522d109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 0x56522d109a50 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f221c006fd0 tx=0x7f221c00f000 comp rx=0 tx=0).stop 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 shutdown_connections 2024-03-06T07:31:23.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:6834/4173228924,v1:172.21.15.6:6835/4173228924] conn(0x56522d17eed0 0x56522d17f2b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56522d0e87b0 0x56522d0bf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56522d0e8b90 0x56522d0bf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 --2- 172.21.15.6:0/3728908479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56522d0e0c50 0x56522d109a50 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f220be00640 0 client.6540 ms_handle_reset on v2:172.21.15.6:6834/4173228924 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 >> 172.21.15.6:0/3728908479 conn(0x56522d0bbc90 msgr2=0x56522d0dfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 shutdown_connections 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 1 -- 172.21.15.6:0/3728908479 wait complete. 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.829+0000 7f2234fb7740 20 client.6540 trim_cache size 0 max 16384 2024-03-06T07:31:23.835 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6402, 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.9091736619999997, 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 45, 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 69, 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 69, 2024-03-06T07:31:23.836 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 12.289417351000001 2024-03-06T07:31:23.837 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f220c06b4d0 msgr2=0x7f220c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f220c06b4d0 0x7f220c06d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f221c00a870 tx=0x7f221c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 msgr2=0x7f222c1f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 0x7f222c1f69a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f221800c460 tx=0x7f221800c920 comp rx=0 tx=0).stop 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 shutdown_connections 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f220c06b4d0 0x7f220c06d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f222c15e2e0 0x7f222c0cb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f222c0d0d70 0x7f222c1f69a0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 --2- 172.21.15.6:0/3678686389 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f222c0b50e0 0x7f222c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 >> 172.21.15.6:0/3678686389 conn(0x7f222c0d3f60 msgr2=0x7f222c06c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 shutdown_connections 2024-03-06T07:31:23.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:23.830+0000 7f2232200640 1 -- 172.21.15.6:0/3678686389 wait complete. 2024-03-06T07:31:23.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:31:24.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.046+0000 7f0d09000640 1 Processor -- start 2024-03-06T07:31:24.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.047+0000 7f0d09000640 1 -- start start 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0d09000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d040af580 0x7f0d0415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0d09000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0d09000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0415cc40 con 0x7f0d040af1a0 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0d09000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0415cf20 con 0x7f0d040af580 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0d09000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0415d200 con 0x7f0d040cdc20 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0cfbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d040af580 0x7f0d0415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0cfbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0d040af580 0x7f0d0415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48390/0 (socket says 172.21.15.6:48390) 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0cfbe00640 1 -- 172.21.15.6:0/150919782 learned_addr learned my addr 172.21.15.6:0/150919782 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.050+0000 7f0d02800640 1 --2- 172.21.15.6:0/150919782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.051+0000 7f0cfbe00640 1 -- 172.21.15.6:0/150919782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 msgr2=0x7f0d040b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.051+0000 7f0cfbe00640 1 --2- 172.21.15.6:0/150919782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.051+0000 7f0cfbe00640 1 -- 172.21.15.6:0/150919782 >> v1:172.21.15.178:6789/0 conn(0x7f0d040af1a0 legacy=0x7f0d040b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.051+0000 7f0cfbe00640 1 -- 172.21.15.6:0/150919782 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0415d4e0 con 0x7f0d040af580 2024-03-06T07:31:24.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.052+0000 7f0cfbe00640 1 --2- 172.21.15.6:0/150919782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af580 0x7f0d0415c600 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009870 tx=0x7f0cec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a950604512815a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.052+0000 7f0d01400640 1 -- 172.21.15.6:0/150919782 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec035020 con 0x7f0d040af580 2024-03-06T07:31:24.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.052+0000 7f0d01400640 1 -- 172.21.15.6:0/150919782 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec031640 con 0x7f0d040af580 2024-03-06T07:31:24.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.052+0000 7f0d01400640 1 -- 172.21.15.6:0/150919782 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec02d040 con 0x7f0d040af580 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 -- 172.21.15.6:0/150919782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af580 msgr2=0x7f0d0415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 --2- 172.21.15.6:0/150919782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af580 0x7f0d0415c600 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009870 tx=0x7f0cec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 -- 172.21.15.6:0/150919782 shutdown_connections 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 --2- 172.21.15.6:0/150919782 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 --2- 172.21.15.6:0/150919782 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af580 0x7f0d0415c600 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 -- 172.21.15.6:0/150919782 >> 172.21.15.6:0/150919782 conn(0x7f0d0406afb0 msgr2=0x7f0d0406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 -- 172.21.15.6:0/150919782 shutdown_connections 2024-03-06T07:31:24.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.053+0000 7f0d09000640 1 -- 172.21.15.6:0/150919782 wait complete. 2024-03-06T07:31:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.055+0000 7f0d09000640 1 Processor -- start 2024-03-06T07:31:24.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.055+0000 7f0d09000640 1 -- start start 2024-03-06T07:31:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d09000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 0x7f0d0415c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d09000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d040af580 0x7f0d040ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d09000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d09000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d040b3720 con 0x7f0d040af1a0 2024-03-06T07:31:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d09000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d040b35a0 con 0x7f0d040af580 2024-03-06T07:31:24.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d09000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d04155e70 con 0x7f0d040cdc20 2024-03-06T07:31:24.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d01e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 0x7f0d0415c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d01e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 0x7f0d0415c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48402/0 (socket says 172.21.15.6:48402) 2024-03-06T07:31:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.056+0000 7f0d01e00640 1 -- 172.21.15.6:0/3657736490 learned_addr learned my addr 172.21.15.6:0/3657736490 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0cfbe00640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d040af580 0x7f0d040ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d02800640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d01e00640 1 -- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 msgr2=0x7f0d040cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d01e00640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d01e00640 1 -- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d040af580 msgr2=0x7f0d040ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d01e00640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d040af580 0x7f0d040ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d01e00640 1 -- 172.21.15.6:0/3657736490 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0409c7e0 con 0x7f0d040af1a0 2024-03-06T07:31:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.057+0000 7f0d01e00640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 0x7f0d0415c170 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400c450 tx=0x7f0cf400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.058+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf40052d0 con 0x7f0d040af1a0 2024-03-06T07:31:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.058+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf4005470 con 0x7f0d040af1a0 2024-03-06T07:31:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.058+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4012870 con 0x7f0d040af1a0 2024-03-06T07:31:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.058+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0409cac0 con 0x7f0d040af1a0 2024-03-06T07:31:24.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.058+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0409d080 con 0x7f0d040af1a0 2024-03-06T07:31:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.059+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d04069510 con 0x7f0d040af1a0 2024-03-06T07:31:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.060+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cf4012a10 con 0x7f0d040af1a0 2024-03-06T07:31:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.061+0000 7f0cfa000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ce006b4d0 0x7f0ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.062+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cf4089e10 con 0x7f0d040af1a0 2024-03-06T07:31:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.062+0000 7f0cfbe00640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ce006b4d0 0x7f0ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.063+0000 7f0cfbe00640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ce006b4d0 0x7f0ce006d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0cec036000 tx=0x7f0cec03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.065+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0cf4058c90 con 0x7f0d040af1a0 2024-03-06T07:31:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.306+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0d040adb40 con 0x7f0d040af1a0 2024-03-06T07:31:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.307+0000 7f0cfa000640 1 -- 172.21.15.6:0/3657736490 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0cf4058590 con 0x7f0d040af1a0 2024-03-06T07:31:24.309 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ce006b4d0 msgr2=0x7f0ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ce006b4d0 0x7f0ce006d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0cec036000 tx=0x7f0cec03c040 comp rx=0 tx=0).stop 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 msgr2=0x7f0d0415c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 0x7f0d0415c170 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f0cf400c450 tx=0x7f0cf400c910 comp rx=0 tx=0).stop 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 shutdown_connections 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0ce006b4d0 0x7f0ce006d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.309+0000 7f0d09000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d040af580 0x7f0d040ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.310+0000 7f0d09000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d040cdc20 0x7f0d040cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.310+0000 7f0d09000640 1 --2- 172.21.15.6:0/3657736490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d040af1a0 0x7f0d0415c170 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.310+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 >> 172.21.15.6:0/3657736490 conn(0x7f0d0406afb0 msgr2=0x7f0d040cee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.310+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 shutdown_connections 2024-03-06T07:31:24.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:24.310+0000 7f0d09000640 1 -- 172.21.15.6:0/3657736490 wait complete. 2024-03-06T07:31:24.330 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:31:24.331 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:24.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:24.332 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:31:24.383 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:31:24.384 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42373sec preferred_lft 42373sec 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:24.385 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:31:24.386 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:31:24.387 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:31:24.387 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:31:24.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:24.388 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:31:24.472 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:31:24.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:24.478 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:31:24.521 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:31:24.521 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:31:24.521 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:31:24.521 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:31:24.521 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:31:24.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:24.522 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:31:24.522 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:31:24.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:31:24.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:31:24.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:31:24.522 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:31:24.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:24.759 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:31:24.817 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:31:24.817 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:31:24.818 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:24.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:24.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:31:24.927 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:24.927 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:24.981 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:31:24.982 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:31:24.982 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:25.028 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:31:25.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:25.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:25.166 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:25.166 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:25.199 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:31:25.199 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:31:25.199 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:31:25.199 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:31:25.199 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:31:25.201 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:31:25.201 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:31:25.201 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:31:25.201 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:31:25.201 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:31:25.205 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:31:25.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:25.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:25.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:25.281 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:31:25.337 INFO:teuthology.orchestra.run.smithi006.stdout:client5276 2024-03-06T07:31:25.337 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:25.337 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:25.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:31:25.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.642+0000 7fd135200640 1 Processor -- start 2024-03-06T07:31:25.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.642+0000 7fd135200640 1 -- start start 2024-03-06T07:31:25.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.643+0000 7fd135200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd13015ef90 0x7fd1300ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.643+0000 7fd135200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1301618e0 0x7fd13006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.643+0000 7fd135200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1300ae0d0 con 0x7fd1300aef20 2024-03-06T07:31:25.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.643+0000 7fd135200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1300aece0 con 0x7fd1301618e0 2024-03-06T07:31:25.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.643+0000 7fd135200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300ae3b0 con 0x7fd13015ef90 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd1300aef20 0x7fd1300af300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47078/0 (socket says 172.21.15.6:47078) 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12f400640 1 -- 172.21.15.6:0/1958521912 learned_addr learned my addr 172.21.15.6:0/1958521912 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12fe00640 1 --2- 172.21.15.6:0/1958521912 >> v2:172.21.15.6:3300/0 conn(0x7fd1301618e0 0x7fd13006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12ea00640 1 --2- 172.21.15.6:0/1958521912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd13015ef90 0x7fd1300ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12fe00640 1 -- 172.21.15.6:0/1958521912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd13015ef90 msgr2=0x7fd1300ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12fe00640 1 --2- 172.21.15.6:0/1958521912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd13015ef90 0x7fd1300ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.644+0000 7fd12e000640 1 -- 172.21.15.6:0/1958521912 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664213985 0 0) 0x7fd1300ae0d0 con 0x7fd1300aef20 2024-03-06T07:31:25.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.645+0000 7fd12fe00640 1 -- 172.21.15.6:0/1958521912 >> v1:172.21.15.6:6789/0 conn(0x7fd1300aef20 legacy=0x7fd1300af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.645+0000 7fd12fe00640 1 -- 172.21.15.6:0/1958521912 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1300ae690 con 0x7fd1301618e0 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.646+0000 7fd12fe00640 1 --2- 172.21.15.6:0/1958521912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1301618e0 0x7fd13006e4a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd128009d90 tx=0x7fd12802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9909043fda0415f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.646+0000 7fd12e000640 1 -- 172.21.15.6:0/1958521912 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128039070 con 0x7fd1301618e0 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.646+0000 7fd12e000640 1 -- 172.21.15.6:0/1958521912 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd12802cbc0 con 0x7fd1301618e0 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.646+0000 7fd12e000640 1 -- 172.21.15.6:0/1958521912 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd128034440 con 0x7fd1301618e0 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 -- 172.21.15.6:0/1958521912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1301618e0 msgr2=0x7fd13006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 --2- 172.21.15.6:0/1958521912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1301618e0 0x7fd13006e4a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fd128009d90 tx=0x7fd12802be10 comp rx=0 tx=0).stop 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 -- 172.21.15.6:0/1958521912 shutdown_connections 2024-03-06T07:31:25.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 --2- 172.21.15.6:0/1958521912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1301618e0 0x7fd13006e4a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 --2- 172.21.15.6:0/1958521912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd13015ef90 0x7fd1300ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 -- 172.21.15.6:0/1958521912 >> 172.21.15.6:0/1958521912 conn(0x7fd1300723d0 msgr2=0x7fd13006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 -- 172.21.15.6:0/1958521912 shutdown_connections 2024-03-06T07:31:25.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.647+0000 7fd135200640 1 -- 172.21.15.6:0/1958521912 wait complete. 2024-03-06T07:31:25.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.650+0000 7fd135200640 1 Processor -- start 2024-03-06T07:31:25.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.650+0000 7fd135200640 1 -- start start 2024-03-06T07:31:25.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.651+0000 7fd135200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1300aef20 0x7fd13015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.651+0000 7fd135200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 0x7fd13015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.651+0000 7fd135200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1301618e0 0x7fd1300cec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.651+0000 7fd135200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13006e9e0 con 0x7fd13015ef90 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.651+0000 7fd135200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300addc0 con 0x7fd1300aef20 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.651+0000 7fd135200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd13006eb60 con 0x7fd1301618e0 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 0x7fd13015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 0x7fd13015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48418/0 (socket says 172.21.15.6:48418) 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 -- 172.21.15.6:0/163158396 learned_addr learned my addr 172.21.15.6:0/163158396 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12f400640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1300aef20 0x7fd13015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12fe00640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1301618e0 0x7fd1300cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 -- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1301618e0 msgr2=0x7fd1300cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1301618e0 0x7fd1300cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 -- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1300aef20 msgr2=0x7fd13015ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1300aef20 0x7fd13015ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.652+0000 7fd12ea00640 1 -- 172.21.15.6:0/163158396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd130156440 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.653+0000 7fd12ea00640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 0x7fd13015a920 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00d800 tx=0x7fd11c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.653+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c00b660 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.653+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c019070 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.653+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd130153b50 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.653+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c00ba20 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.653+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd130154110 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.654+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13004f330 con 0x7fd13015ef90 2024-03-06T07:31:25.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.655+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd11c00bbc0 con 0x7fd13015ef90 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.656+0000 7fd127400640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd11006b4d0 0x7fd11006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:25.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.656+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd11c08a7c0 con 0x7fd13015ef90 2024-03-06T07:31:25.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.657+0000 7fd12f400640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd11006b4d0 0x7fd11006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:25.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.658+0000 7fd12f400640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd11006b4d0 0x7fd11006d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd120000f60 tx=0x7fd120007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:25.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.666+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd11c01c360 con 0x7fd13015ef90 2024-03-06T07:31:25.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.866+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7fd13006c980 con 0x7fd13015ef90 2024-03-06T07:31:25.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.872+0000 7fd127400640 1 -- 172.21.15.6:0/163158396 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd11c059510 con 0x7fd13015ef90 2024-03-06T07:31:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd11006b4d0 msgr2=0x7fd11006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd11006b4d0 0x7fd11006d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd120000f60 tx=0x7fd120007b50 comp rx=0 tx=0).stop 2024-03-06T07:31:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 msgr2=0x7fd13015a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 0x7fd13015a920 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00d800 tx=0x7fd11c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 shutdown_connections 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd11006b4d0 0x7fd11006d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1301618e0 0x7fd1300cec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd13015ef90 0x7fd13015a920 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 --2- 172.21.15.6:0/163158396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1300aef20 0x7fd13015ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 >> 172.21.15.6:0/163158396 conn(0x7fd1300723d0 msgr2=0x7fd13006bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 shutdown_connections 2024-03-06T07:31:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:25.875+0000 7fd135200640 1 -- 172.21.15.6:0/163158396 wait complete. 2024-03-06T07:31:35.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:36.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.188+0000 7f7716c00640 1 Processor -- start 2024-03-06T07:31:36.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.188+0000 7f7716c00640 1 -- start start 2024-03-06T07:31:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7716c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7710145300 0x7f77100b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7716c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f77100cd060 0x7f771015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7716c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77100714d0 con 0x7f7710070f10 2024-03-06T07:31:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7716c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77100717b0 con 0x7f77100cd060 2024-03-06T07:31:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7716c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7710071a90 con 0x7f7710145300 2024-03-06T07:31:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7714e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7710070f10 0x7f77100cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51184/0 (socket says 172.21.15.6:51184) 2024-03-06T07:31:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7714e00640 1 -- 172.21.15.6:0/276872791 learned_addr learned my addr 172.21.15.6:0/276872791 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:36.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.190+0000 7f7716200640 1 --2- 172.21.15.6:0/276872791 >> v2:172.21.15.6:3300/0 conn(0x7f77100cd060 0x7f771015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.191+0000 7f7715800640 1 --2- 172.21.15.6:0/276872791 >> v2:172.21.15.178:3300/0 conn(0x7f7710145300 0x7f77100b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.191+0000 7f770fe00640 1 -- 172.21.15.6:0/276872791 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803387351 0 0) 0x7f77100714d0 con 0x7f7710070f10 2024-03-06T07:31:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.191+0000 7f7716200640 1 -- 172.21.15.6:0/276872791 >> v2:172.21.15.178:3300/0 conn(0x7f7710145300 msgr2=0x7f77100b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.191+0000 7f7716200640 1 --2- 172.21.15.6:0/276872791 >> v2:172.21.15.178:3300/0 conn(0x7f7710145300 0x7f77100b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.191+0000 7f7716200640 1 -- 172.21.15.6:0/276872791 >> v1:172.21.15.6:6789/0 conn(0x7f7710070f10 legacy=0x7f77100cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.191+0000 7f7716200640 1 -- 172.21.15.6:0/276872791 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710071d70 con 0x7f77100cd060 2024-03-06T07:31:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.192+0000 7f7716200640 1 --2- 172.21.15.6:0/276872791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771015c570 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f77100cf990 tx=0x7f770802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309685a911e8d2fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.192+0000 7f770fe00640 1 -- 172.21.15.6:0/276872791 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708039070 con 0x7f77100cd060 2024-03-06T07:31:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f770fe00640 1 -- 172.21.15.6:0/276872791 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770802cbf0 con 0x7f77100cd060 2024-03-06T07:31:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f770fe00640 1 -- 172.21.15.6:0/276872791 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708034440 con 0x7f77100cd060 2024-03-06T07:31:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 -- 172.21.15.6:0/276872791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 msgr2=0x7f771015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 --2- 172.21.15.6:0/276872791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771015c570 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f77100cf990 tx=0x7f770802be40 comp rx=0 tx=0).stop 2024-03-06T07:31:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 -- 172.21.15.6:0/276872791 shutdown_connections 2024-03-06T07:31:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 --2- 172.21.15.6:0/276872791 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771015c570 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 --2- 172.21.15.6:0/276872791 >> v2:172.21.15.178:3300/0 conn(0x7f7710145300 0x7f77100b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 -- 172.21.15.6:0/276872791 >> 172.21.15.6:0/276872791 conn(0x7f771006b3d0 msgr2=0x7f771006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 -- 172.21.15.6:0/276872791 shutdown_connections 2024-03-06T07:31:36.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.193+0000 7f7716c00640 1 -- 172.21.15.6:0/276872791 wait complete. 2024-03-06T07:31:36.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.195+0000 7f7716c00640 1 Processor -- start 2024-03-06T07:31:36.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.195+0000 7f7716c00640 1 -- start start 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7716c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7710070f10 0x7f771013d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7716c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771013ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7716c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7710145300 0x7f77100a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7716c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771015cab0 con 0x7f77100cd060 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7716c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101456e0 con 0x7f7710145300 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7716c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771015cc30 con 0x7f7710070f10 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7714e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771013ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7714e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771013ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44098/0 (socket says 172.21.15.6:44098) 2024-03-06T07:31:36.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.196+0000 7f7714e00640 1 -- 172.21.15.6:0/2817135030 learned_addr learned my addr 172.21.15.6:0/2817135030 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7716200640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7710145300 0x7f77100a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7714e00640 1 -- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7710070f10 msgr2=0x7f771013d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7714e00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7710070f10 0x7f771013d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7714e00640 1 -- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7710145300 msgr2=0x7f77100a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7714e00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7710145300 0x7f77100a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7714e00640 1 -- 172.21.15.6:0/2817135030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771013f8e0 con 0x7f77100cd060 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7716200640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7710145300 0x7f77100a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:36.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.197+0000 7f7714e00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771013ddd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f770000c500 tx=0x7f770000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.198+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700018070 con 0x7f77100cd060 2024-03-06T07:31:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.198+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700007580 con 0x7f77100cd060 2024-03-06T07:31:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.198+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700004a20 con 0x7f77100cd060 2024-03-06T07:31:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.198+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771013fbc0 con 0x7f77100cd060 2024-03-06T07:31:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.198+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7710140180 con 0x7f77100cd060 2024-03-06T07:31:36.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.199+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771004f330 con 0x7f77100cd060 2024-03-06T07:31:36.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.200+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7700004bc0 con 0x7f77100cd060 2024-03-06T07:31:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.201+0000 7f770d600640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76f406b4d0 0x7f76f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:36.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.201+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7700089510 con 0x7f77100cd060 2024-03-06T07:31:36.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.202+0000 7f7715800640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76f406b4d0 0x7f76f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:36.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.203+0000 7f7715800640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76f406b4d0 0x7f76f406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7704006fd0 tx=0x7f7704008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:36.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.206+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7700058380 con 0x7f77100cd060 2024-03-06T07:31:36.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.443+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f771015d4c0 con 0x7f77100cd060 2024-03-06T07:31:36.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.444+0000 7f770d600640 1 -- 172.21.15.6:0/2817135030 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f770005e080 con 0x7f77100cd060 2024-03-06T07:31:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:36.447 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:36.447 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76f406b4d0 msgr2=0x7f76f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76f406b4d0 0x7f76f406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7704006fd0 tx=0x7f7704008040 comp rx=0 tx=0).stop 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 msgr2=0x7f771013ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771013ddd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f770000c500 tx=0x7f770000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 shutdown_connections 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76f406b4d0 0x7f76f406d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7710145300 0x7f77100a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f77100cd060 0x7f771013ddd0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 --2- 172.21.15.6:0/2817135030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7710070f10 0x7f771013d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 >> 172.21.15.6:0/2817135030 conn(0x7f771006b3d0 msgr2=0x7f77100cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 shutdown_connections 2024-03-06T07:31:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36.447+0000 7f7716c00640 1 -- 172.21.15.6:0/2817135030 wait complete. 2024-03-06T07:31:36.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:36.679 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:31:36.679 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:31:36.679 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:36.679 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:36.729 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:31:36.729 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> n = 100 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:36.730 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:31:36.731 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:31:36.732 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-06T07:31:36.732 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-06T07:31:36.732 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi006.stderr:Traceback (most recent call last): 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi006.stderr: File "", line 26, in 2024-03-06T07:31:36.937 INFO:teuthology.orchestra.run.smithi006.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:31:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:36 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:31:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:31:36.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:37.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.211+0000 7fe634800640 1 Processor -- start 2024-03-06T07:31:37.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.211+0000 7fe634800640 1 -- start start 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.213+0000 7fe634800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 0x7fe62c071760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.213+0000 7fe634800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62c0b4860 con 0x7fe62c0b3c90 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.213+0000 7fe634800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62c0b4b40 con 0x7fe62c0cf5c0 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.213+0000 7fe634800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0b4e20 con 0x7fe62c0cdc20 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.213+0000 7fe631e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe62c0b3c90 0x7fe62c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51196/0 (socket says 172.21.15.6:51196) 2024-03-06T07:31:37.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.213+0000 7fe631e00640 1 -- 172.21.15.6:0/4119082010 learned_addr learned my addr 172.21.15.6:0/4119082010 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282286187 0 0) 0x7fe62c0b4860 con 0x7fe62c0b3c90 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe614003520 con 0x7fe62c0b3c90 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe632800640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 0x7fe62c071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255909231 0 0) 0x7fe614003520 con 0x7fe62c0b3c90 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 msgr2=0x7fe62c071760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 0x7fe62c071760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 >> v1:172.21.15.178:6789/0 conn(0x7fe62c0cf5c0 legacy=0x7fe62c071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c0b5100 con 0x7fe62c0b3c90 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fe620001670 con 0x7fe62c0b3c90 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 >> v1:172.21.15.6:6789/0 conn(0x7fe62c0b3c90 legacy=0x7fe62c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe614003a50 0x7fe614003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe614004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 0x7fe614005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0b4e20 con 0x7fe614004800 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61400de50 con 0x7fe614003a50 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61400e050 con 0x7fe62c0cdc20 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.214+0000 7fe631400640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe614004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631e00640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe614003a50 0x7fe614003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631400640 1 -- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 msgr2=0x7fe614005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631400640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c0cdc20 0x7fe614005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631400640 1 -- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe614003a50 msgr2=0x7fe614003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631400640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe614003a50 0x7fe614003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631400640 1 -- 172.21.15.6:0/4119082010 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614003520 con 0x7fe614004800 2024-03-06T07:31:37.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.215+0000 7fe631e00640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe614003a50 0x7fe614003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.217+0000 7fe631400640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe614004c80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe61c001750 tx=0x7fe614005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adb09dfb95f8c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.217+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c001370 con 0x7fe614004800 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.217+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c0052d0 con 0x7fe614004800 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.217+0000 7fe630a00640 1 -- 172.21.15.6:0/4119082010 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c003830 con 0x7fe614004800 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe634800640 1 -- 172.21.15.6:0/4119082010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 msgr2=0x7fe614004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe634800640 1 --2- 172.21.15.6:0/4119082010 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe614004c80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fe61c001750 tx=0x7fe614005830 comp rx=0 tx=0).stop 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe632800640 1 -- 172.21.15.6:0/4119082010 reap_dead start 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe634800640 1 -- 172.21.15.6:0/4119082010 shutdown_connections 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe634800640 1 -- 172.21.15.6:0/4119082010 >> 172.21.15.6:0/4119082010 conn(0x7fe62c06afb0 msgr2=0x7fe62c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe634800640 1 -- 172.21.15.6:0/4119082010 shutdown_connections 2024-03-06T07:31:37.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.218+0000 7fe634800640 1 -- 172.21.15.6:0/4119082010 wait complete. 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.221+0000 7fe634800640 1 Processor -- start 2024-03-06T07:31:37.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.221+0000 7fe634800640 1 -- start start 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe634800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe62c1447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe634800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe62c158550 0x7fe62c1589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe634800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c155150 0x7fe62c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe634800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c071760 con 0x7fe614004800 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe634800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0ce060 con 0x7fe62c158550 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe634800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0718e0 con 0x7fe62c155150 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe631e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe62c1447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe631e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe62c1447d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44118/0 (socket says 172.21.15.6:44118) 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe631e00640 1 -- 172.21.15.6:0/2628799001 learned_addr learned my addr 172.21.15.6:0/2628799001 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe631400640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe62c158550 0x7fe62c1589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.223+0000 7fe632800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c155150 0x7fe62c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe631e00640 1 -- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c155150 msgr2=0x7fe62c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe631e00640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c155150 0x7fe62c155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe631e00640 1 -- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe62c158550 msgr2=0x7fe62c1589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe631e00640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe62c158550 0x7fe62c1589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe631e00640 1 -- 172.21.15.6:0/2628799001 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c1f3030 con 0x7fe614004800 2024-03-06T07:31:37.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe631e00640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe62c1447d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe62000d830 tx=0x7fe62000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620019070 con 0x7fe614004800 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6200074e0 con 0x7fe614004800 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c1f3310 con 0x7fe614004800 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.224+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6200049e0 con 0x7fe614004800 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.225+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c1f38d0 con 0x7fe614004800 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.226+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62c069510 con 0x7fe614004800 2024-03-06T07:31:37.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.227+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe62001c070 con 0x7fe614004800 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.228+0000 7fe626000640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe61006b500 0x7fe61006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.228+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe620089cb0 con 0x7fe614004800 2024-03-06T07:31:37.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.229+0000 7fe631400640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe61006b500 0x7fe61006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.230+0000 7fe631400640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe61006b500 0x7fe61006d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe61c033000 tx=0x7fe61c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.233+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe620058b50 con 0x7fe614004800 2024-03-06T07:31:37.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.471+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe62c04f3a0 con 0x7fe614004800 2024-03-06T07:31:37.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.472+0000 7fe626000640 1 -- 172.21.15.6:0/2628799001 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe620058450 con 0x7fe614004800 2024-03-06T07:31:37.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:37.475 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:37.475 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.474+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe61006b500 msgr2=0x7fe61006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.474+0000 7fe634800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe61006b500 0x7fe61006d9c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe61c033000 tx=0x7fe61c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.474+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 msgr2=0x7fe62c1447d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.474+0000 7fe634800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe62c1447d0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe62000d830 tx=0x7fe62000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 shutdown_connections 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe61006b500 0x7fe61006d9c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe62c158550 0x7fe62c1589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe62c155150 0x7fe62c155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 --2- 172.21.15.6:0/2628799001 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe614004800 0x7fe62c1447d0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 >> 172.21.15.6:0/2628799001 conn(0x7fe62c06afb0 msgr2=0x7fe62c06d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 shutdown_connections 2024-03-06T07:31:37.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.475+0000 7fe634800640 1 -- 172.21.15.6:0/2628799001 wait complete. 2024-03-06T07:31:37.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:31:37.748 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:31:37.749 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:31:37.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.993+0000 7f5ddce00640 1 Processor -- start 2024-03-06T07:31:37.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.993+0000 7f5ddce00640 1 -- start start 2024-03-06T07:31:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5ddce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5dd80cd060 0x7f5dd815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5ddce00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd815ed10 con 0x7f5dd815e810 2024-03-06T07:31:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5ddce00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd815eff0 con 0x7f5dd81552b0 2024-03-06T07:31:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5ddce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd815f2d0 con 0x7f5dd80cd060 2024-03-06T07:31:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5dd7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5dd815e810 0x7f5dd80b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51204/0 (socket says 172.21.15.6:51204) 2024-03-06T07:31:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5dcea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5dd80cd060 0x7f5dd815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.995+0000 7f5dd7e00640 1 -- 172.21.15.6:0/3477484561 learned_addr learned my addr 172.21.15.6:0/3477484561 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:37.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.996+0000 7f5dd6a00640 1 -- 172.21.15.6:0/3477484561 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854177355 0 0) 0x7f5dd815ed10 con 0x7f5dd815e810 2024-03-06T07:31:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.996+0000 7f5dcea00640 1 -- 172.21.15.6:0/3477484561 >> v1:172.21.15.178:6789/0 conn(0x7f5dd81552b0 legacy=0x7f5dd80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.996+0000 7f5dcea00640 1 -- 172.21.15.6:0/3477484561 >> v1:172.21.15.6:6789/0 conn(0x7f5dd815e810 legacy=0x7f5dd80b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.996+0000 7f5dcea00640 1 -- 172.21.15.6:0/3477484561 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd815f5b0 con 0x7f5dd80cd060 2024-03-06T07:31:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.997+0000 7f5dcea00640 1 --2- 172.21.15.6:0/3477484561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd80cd060 0x7f5dd815b1d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00097a0 tx=0x7f5dc002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfaf0c0cbbc798b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.997+0000 7f5dd6a00640 1 -- 172.21.15.6:0/3477484561 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0035020 con 0x7f5dd80cd060 2024-03-06T07:31:37.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.997+0000 7f5dd6a00640 1 -- 172.21.15.6:0/3477484561 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0031830 con 0x7f5dd80cd060 2024-03-06T07:31:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.997+0000 7f5dd6a00640 1 -- 172.21.15.6:0/3477484561 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0031c00 con 0x7f5dd80cd060 2024-03-06T07:31:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 -- 172.21.15.6:0/3477484561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd80cd060 msgr2=0x7f5dd815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 --2- 172.21.15.6:0/3477484561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd80cd060 0x7f5dd815b1d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00097a0 tx=0x7f5dc002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 -- 172.21.15.6:0/3477484561 shutdown_connections 2024-03-06T07:31:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 --2- 172.21.15.6:0/3477484561 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd80cd060 0x7f5dd815b1d0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 -- 172.21.15.6:0/3477484561 >> 172.21.15.6:0/3477484561 conn(0x7f5dd806b3d0 msgr2=0x7f5dd806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 -- 172.21.15.6:0/3477484561 shutdown_connections 2024-03-06T07:31:38.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:37.998+0000 7f5ddce00640 1 -- 172.21.15.6:0/3477484561 wait complete. 2024-03-06T07:31:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.000+0000 7f5ddce00640 1 Processor -- start 2024-03-06T07:31:38.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.000+0000 7f5ddce00640 1 -- start start 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5ddce00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd80cd060 0x7f5dd8158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5ddce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 0x7f5dd814d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5ddce00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd815e810 0x7f5dd80ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5ddce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd806dd30 con 0x7f5dd81552b0 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5ddce00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80b3760 con 0x7f5dd80cd060 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5ddce00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd806deb0 con 0x7f5dd815e810 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.001+0000 7f5dcea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 0x7f5dd814d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 0x7f5dd814d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44130/0 (socket says 172.21.15.6:44130) 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 -- 172.21.15.6:0/4028341551 learned_addr learned my addr 172.21.15.6:0/4028341551 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:38.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dd7400640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd80cd060 0x7f5dd8158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dd7e00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd815e810 0x7f5dd80ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 -- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd815e810 msgr2=0x7f5dd80ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd815e810 0x7f5dd80ae290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 -- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd80cd060 msgr2=0x7f5dd8158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd80cd060 0x7f5dd8158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 -- 172.21.15.6:0/4028341551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd80ae7d0 con 0x7f5dd81552b0 2024-03-06T07:31:38.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dd7400640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd80cd060 0x7f5dd8158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.002+0000 7f5dcea00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 0x7f5dd814d9d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00097a0 tx=0x7f5dc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.003+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0035050 con 0x7f5dd81552b0 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.003+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0004230 con 0x7f5dd81552b0 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.003+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc003b740 con 0x7f5dd81552b0 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.003+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd8159500 con 0x7f5dd81552b0 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.003+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd8159ac0 con 0x7f5dd81552b0 2024-03-06T07:31:38.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.004+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd804f330 con 0x7f5dd81552b0 2024-03-06T07:31:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.005+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5dc0038020 con 0x7f5dd81552b0 2024-03-06T07:31:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.006+0000 7f5dcfe00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db406b4d0 0x7f5db406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.006+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5dc00abfe0 con 0x7f5dd81552b0 2024-03-06T07:31:38.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.007+0000 7f5dd7400640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db406b4d0 0x7f5db406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.008+0000 7f5dd7400640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db406b4d0 0x7f5db406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8007fd0 tx=0x7f5dc8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.011+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dc0040070 con 0x7f5dd81552b0 2024-03-06T07:31:38.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.326+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dd81570e0 con 0x7f5dd81552b0 2024-03-06T07:31:38.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.327+0000 7f5dcfe00640 1 -- 172.21.15.6:0/4028341551 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f5dc007ae90 con 0x7f5dd81552b0 2024-03-06T07:31:38.329 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:38.329 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:38.329 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:38.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db406b4d0 msgr2=0x7f5db406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db406b4d0 0x7f5db406d990 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8007fd0 tx=0x7f5dc8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:38.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 msgr2=0x7f5dd814d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 0x7f5dd814d9d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00097a0 tx=0x7f5dc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 shutdown_connections 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5db406b4d0 0x7f5db406d990 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dd815e810 0x7f5dd80ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dd81552b0 0x7f5dd814d9d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 --2- 172.21.15.6:0/4028341551 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dd80cd060 0x7f5dd8158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 >> 172.21.15.6:0/4028341551 conn(0x7f5dd806b3d0 msgr2=0x7f5dd806e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 shutdown_connections 2024-03-06T07:31:38.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:38.330+0000 7f5ddce00640 1 -- 172.21.15.6:0/4028341551 wait complete. 2024-03-06T07:31:38.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:38.583 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:31:38.583 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:31:38.584 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:31:43.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.820+0000 7f368d000640 1 Processor -- start 2024-03-06T07:31:43.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.821+0000 7f368d000640 1 -- start start 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.822+0000 7f368d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f36881600c0 0x7f3688166d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.822+0000 7f368d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36880b0690 con 0x7f368815c040 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.822+0000 7f368d000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36880afe10 con 0x7f368815bc60 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.822+0000 7f368d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36880b00d0 con 0x7f36881600c0 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.823+0000 7f3686800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f36881600c0 0x7f3688166d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.823+0000 7f3685400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f368815c040 0x7f368815f9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39408/0 (socket says 172.21.15.6:39408) 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.823+0000 7f3685400640 1 -- 172.21.15.6:0/3946279548 learned_addr learned my addr 172.21.15.6:0/3946279548 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.823+0000 7f3686800640 1 -- 172.21.15.6:0/3946279548 >> v1:172.21.15.178:6789/0 conn(0x7f368815bc60 legacy=0x7f36880af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.823+0000 7f3686800640 1 -- 172.21.15.6:0/3946279548 >> v1:172.21.15.6:6789/0 conn(0x7f368815c040 legacy=0x7f368815f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.823+0000 7f3686800640 1 -- 172.21.15.6:0/3946279548 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36880b03b0 con 0x7f36881600c0 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.825+0000 7f3686800640 1 --2- 172.21.15.6:0/3946279548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36881600c0 0x7f3688166d40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3680009d90 tx=0x7f368002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6cb4e2a3ff9119 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.825+0000 7f3684a00640 1 -- 172.21.15.6:0/3946279548 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680039070 con 0x7f36881600c0 2024-03-06T07:31:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.825+0000 7f3684a00640 1 -- 172.21.15.6:0/3946279548 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368002cc60 con 0x7f36881600c0 2024-03-06T07:31:43.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.825+0000 7f3684a00640 1 -- 172.21.15.6:0/3946279548 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680034660 con 0x7f36881600c0 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 -- 172.21.15.6:0/3946279548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36881600c0 msgr2=0x7f3688166d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 --2- 172.21.15.6:0/3946279548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36881600c0 0x7f3688166d40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3680009d90 tx=0x7f368002beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 -- 172.21.15.6:0/3946279548 shutdown_connections 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 --2- 172.21.15.6:0/3946279548 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36881600c0 0x7f3688166d40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 -- 172.21.15.6:0/3946279548 >> 172.21.15.6:0/3946279548 conn(0x7f368809aa10 msgr2=0x7f368809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 -- 172.21.15.6:0/3946279548 shutdown_connections 2024-03-06T07:31:43.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.826+0000 7f368d000640 1 -- 172.21.15.6:0/3946279548 wait complete. 2024-03-06T07:31:43.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.827+0000 7f368d000640 1 Processor -- start 2024-03-06T07:31:43.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.828+0000 7f368d000640 1 -- start start 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f368d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 0x7f36881e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f368d000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f368815c040 0x7f368806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f368d000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f36881600c0 0x7f368806c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f368d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880afb40 con 0x7f368815bc60 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f368d000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880af9c0 con 0x7f368815c040 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f368d000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3688070580 con 0x7f36881600c0 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 0x7f36881e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 0x7f36881e6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35686/0 (socket says 172.21.15.6:35686) 2024-03-06T07:31:43.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 -- 172.21.15.6:0/2681186894 learned_addr learned my addr 172.21.15.6:0/2681186894 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685400640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f368815c040 0x7f368806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3686800640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f36881600c0 0x7f368806c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 -- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f36881600c0 msgr2=0x7f368806c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f36881600c0 0x7f368806c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 -- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f368815c040 msgr2=0x7f368806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f368815c040 0x7f368806c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685e00640 1 -- 172.21.15.6:0/2681186894 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36881e7a40 con 0x7f368815bc60 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.829+0000 7f3685400640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f368815c040 0x7f368806c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.830+0000 7f3685e00640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 0x7f36881e6690 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f367800c400 tx=0x7f367800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.830+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36780052d0 con 0x7f368815bc60 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.830+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3678005490 con 0x7f368815bc60 2024-03-06T07:31:43.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.830+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3678007260 con 0x7f368815bc60 2024-03-06T07:31:43.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.830+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3688158170 con 0x7f368815bc60 2024-03-06T07:31:43.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.830+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36881e6170 con 0x7f368815bc60 2024-03-06T07:31:43.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.831+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368804f3a0 con 0x7f368815bc60 2024-03-06T07:31:43.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.832+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3678007400 con 0x7f368815bc60 2024-03-06T07:31:43.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.833+0000 7f367e000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f366806b4d0 0x7f366806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.833+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3678089790 con 0x7f368815bc60 2024-03-06T07:31:43.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.834+0000 7f3685400640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f366806b4d0 0x7f366806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.835+0000 7f3685400640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f366806b4d0 0x7f366806d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f368806e2a0 tx=0x7f3674009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:43.838+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3678058600 con 0x7f368815bc60 2024-03-06T07:31:44.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.082+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f368809b630 con 0x7f368815bc60 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.084+0000 7f367e000640 1 -- 172.21.15.6:0/2681186894 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f3678057f00 con 0x7f368815bc60 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:44.086 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f366806b4d0 msgr2=0x7f366806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f366806b4d0 0x7f366806d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f368806e2a0 tx=0x7f3674009040 comp rx=0 tx=0).stop 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 msgr2=0x7f36881e6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 0x7f36881e6690 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f367800c400 tx=0x7f367800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 shutdown_connections 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f366806b4d0 0x7f366806d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f368815c040 0x7f368806c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f36881600c0 0x7f368806c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 --2- 172.21.15.6:0/2681186894 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f368815bc60 0x7f36881e6690 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:44.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.087+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 >> 172.21.15.6:0/2681186894 conn(0x7f368809aa10 msgr2=0x7f368806fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.088+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 shutdown_connections 2024-03-06T07:31:44.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:44.088+0000 7f368d000640 1 -- 172.21.15.6:0/2681186894 wait complete. 2024-03-06T07:31:44.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:44.329 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:31:44.330 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:31:44.330 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:31:49.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:49.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.558+0000 7f5c76200640 1 Processor -- start 2024-03-06T07:31:49.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.558+0000 7f5c76200640 1 -- start start 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c76200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5c701477e0 0x7f5c70162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c76200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c7006ef20 0x7f5c7006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c76200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c70147bc0 con 0x7f5c7006b4f0 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c76200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c70161f90 con 0x7f5c701477e0 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c76200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70162960 con 0x7f5c7006ef20 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c74e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5c701477e0 0x7f5c70162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c75800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5c7006b4f0 0x7f5c700d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39422/0 (socket says 172.21.15.6:39422) 2024-03-06T07:31:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.560+0000 7f5c75800640 1 -- 172.21.15.6:0/1978428427 learned_addr learned my addr 172.21.15.6:0/1978428427 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.561+0000 7f5c6fe00640 1 --2- 172.21.15.6:0/1978428427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c7006ef20 0x7f5c7006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.561+0000 7f5c74e00640 1 -- 172.21.15.6:0/1978428427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c7006ef20 msgr2=0x7f5c7006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.561+0000 7f5c74e00640 1 --2- 172.21.15.6:0/1978428427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c7006ef20 0x7f5c7006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.561+0000 7f5c74e00640 1 -- 172.21.15.6:0/1978428427 >> v1:172.21.15.6:6789/0 conn(0x7f5c7006b4f0 legacy=0x7f5c700d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.561+0000 7f5c74e00640 1 -- 172.21.15.6:0/1978428427 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700b3d70 con 0x7f5c701477e0 2024-03-06T07:31:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.561+0000 7f5c6f400640 1 -- 172.21.15.6:0/1978428427 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578379127 0 0) 0x7f5c70147bc0 con 0x7f5c7006b4f0 2024-03-06T07:31:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.562+0000 7f5c74e00640 1 --2- 172.21.15.6:0/1978428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c701477e0 0x7f5c70162420 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5c640097a0 tx=0x7f5c6402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a28e4d9dcef07af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.562+0000 7f5c6f400640 1 -- 172.21.15.6:0/1978428427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64031780 con 0x7f5c701477e0 2024-03-06T07:31:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.562+0000 7f5c6f400640 1 -- 172.21.15.6:0/1978428427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64037070 con 0x7f5c701477e0 2024-03-06T07:31:49.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.562+0000 7f5c6f400640 1 -- 172.21.15.6:0/1978428427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64031b40 con 0x7f5c701477e0 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 -- 172.21.15.6:0/1978428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c701477e0 msgr2=0x7f5c70162420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 --2- 172.21.15.6:0/1978428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c701477e0 0x7f5c70162420 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5c640097a0 tx=0x7f5c6402b820 comp rx=0 tx=0).stop 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 -- 172.21.15.6:0/1978428427 shutdown_connections 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 --2- 172.21.15.6:0/1978428427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c7006ef20 0x7f5c7006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 --2- 172.21.15.6:0/1978428427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c701477e0 0x7f5c70162420 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 -- 172.21.15.6:0/1978428427 >> 172.21.15.6:0/1978428427 conn(0x7f5c7006c0c0 msgr2=0x7f5c7006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 -- 172.21.15.6:0/1978428427 shutdown_connections 2024-03-06T07:31:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.563+0000 7f5c76200640 1 -- 172.21.15.6:0/1978428427 wait complete. 2024-03-06T07:31:49.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.565+0000 7f5c76200640 1 Processor -- start 2024-03-06T07:31:49.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.565+0000 7f5c76200640 1 -- start start 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c76200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c7006b4f0 0x7f5c701f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c76200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 0x7f5c7013f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c76200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c701477e0 0x7f5c700b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c76200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c70161c20 con 0x7f5c7006ef20 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c76200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b4330 con 0x7f5c7006b4f0 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c76200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7006f8f0 con 0x7f5c701477e0 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.566+0000 7f5c6fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 0x7f5c7013f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 0x7f5c7013f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35708/0 (socket says 172.21.15.6:35708) 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 -- 172.21.15.6:0/3150561568 learned_addr learned my addr 172.21.15.6:0/3150561568 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c74e00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c7006b4f0 0x7f5c701f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 -- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c701477e0 msgr2=0x7f5c700b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c701477e0 0x7f5c700b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 -- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c7006b4f0 msgr2=0x7f5c701f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c7006b4f0 0x7f5c701f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 -- 172.21.15.6:0/3150561568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70153500 con 0x7f5c7006ef20 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c74e00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c7006b4f0 0x7f5c701f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.567+0000 7f5c6fe00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 0x7f5c7013f660 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00d830 tx=0x7f5c5c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.568+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c00b6c0 con 0x7f5c7006ef20 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.568+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c019070 con 0x7f5c7006ef20 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.568+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c00ba80 con 0x7f5c7006ef20 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.568+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c70161340 con 0x7f5c7006ef20 2024-03-06T07:31:49.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.568+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c70161060 con 0x7f5c7006ef20 2024-03-06T07:31:49.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.569+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7004f330 con 0x7f5c7006ef20 2024-03-06T07:31:49.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.570+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c5c01e460 con 0x7f5c7006ef20 2024-03-06T07:31:49.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.572+0000 7f5c6cc00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c5006b500 0x7f5c5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.572+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c5c08a940 con 0x7f5c7006ef20 2024-03-06T07:31:49.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.572+0000 7f5c74e00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c5006b500 0x7f5c5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.573+0000 7f5c74e00640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c5006b500 0x7f5c5006d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5c64036000 tx=0x7f5c64008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.576+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c5c059720 con 0x7f5c7006ef20 2024-03-06T07:31:49.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.833+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c700724b0 con 0x7f5c7006ef20 2024-03-06T07:31:49.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.834+0000 7f5c6cc00640 1 -- 172.21.15.6:0/3150561568 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f5c5c059020 con 0x7f5c7006ef20 2024-03-06T07:31:49.836 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:49.837 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:49.837 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.836+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c5006b500 msgr2=0x7f5c5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.836+0000 7f5c76200640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c5006b500 0x7f5c5006d9c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5c64036000 tx=0x7f5c64008030 comp rx=0 tx=0).stop 2024-03-06T07:31:49.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.836+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 msgr2=0x7f5c7013f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.836+0000 7f5c76200640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 0x7f5c7013f660 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c00d830 tx=0x7f5c5c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 shutdown_connections 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c5006b500 0x7f5c5006d9c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c701477e0 0x7f5c700b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c7006ef20 0x7f5c7013f660 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 --2- 172.21.15.6:0/3150561568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c7006b4f0 0x7f5c701f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 >> 172.21.15.6:0/3150561568 conn(0x7f5c7006c0c0 msgr2=0x7f5c700ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 shutdown_connections 2024-03-06T07:31:49.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:49.837+0000 7f5c76200640 1 -- 172.21.15.6:0/3150561568 wait complete. 2024-03-06T07:31:49.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:31:50.100 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:31:50.100 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:31:50.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:50.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.314+0000 7f3277200640 1 Processor -- start 2024-03-06T07:31:50.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.314+0000 7f3277200640 1 -- start start 2024-03-06T07:31:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.315+0000 7f3277200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f32780b2340 0x7f3278168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.315+0000 7f3277200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780d1120 con 0x7f327815dec0 2024-03-06T07:31:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.315+0000 7f3277200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780d1400 con 0x7f327815e2a0 2024-03-06T07:31:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.316+0000 7f3277200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780b0ea0 con 0x7f32780b2340 2024-03-06T07:31:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.316+0000 7f3275e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f327815dec0 0x7f32780ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:39426/0 (socket says 172.21.15.6:39426) 2024-03-06T07:31:50.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.316+0000 7f3275e00640 1 -- 172.21.15.6:0/703655787 learned_addr learned my addr 172.21.15.6:0/703655787 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.316+0000 7f3276800640 1 --2- 172.21.15.6:0/703655787 >> v2:172.21.15.178:3300/0 conn(0x7f32780b2340 0x7f3278168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.316+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515587691 0 0) 0x7f32780d1120 con 0x7f327815dec0 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f325c003520 con 0x7f327815dec0 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747070660 0 0) 0x7f325c003520 con 0x7f327815dec0 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 >> v2:172.21.15.178:3300/0 conn(0x7f32780b2340 msgr2=0x7f3278168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 --2- 172.21.15.6:0/703655787 >> v2:172.21.15.178:3300/0 conn(0x7f32780b2340 0x7f3278168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 >> v1:172.21.15.178:6789/0 conn(0x7f327815e2a0 legacy=0x7f32780cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780b0b30 con 0x7f327815dec0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f32680019f0 con 0x7f327815dec0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 >> v1:172.21.15.6:6789/0 conn(0x7f327815dec0 legacy=0x7f32780ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f325c0039b0 0x7f325c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f325c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780b0ea0 con 0x7f325c004800 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c00c060 con 0x7f325c0039b0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c00c260 con 0x7f325c0051c0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.317+0000 7f326d400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f325c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f3275e00640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f325c0039b0 0x7f325c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f3276800640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f326d400640 1 -- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f325c0051c0 msgr2=0x7f325c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f326d400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f326d400640 1 -- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f325c0039b0 msgr2=0x7f325c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f326d400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f325c0039b0 0x7f325c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f326d400640 1 -- 172.21.15.6:0/703655787 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c003520 con 0x7f325c004800 2024-03-06T07:31:50.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.318+0000 7f3276800640 1 -- 172.21.15.6:0/703655787 reap_dead start 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.319+0000 7f326d400640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f325c004c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3260001750 tx=0x7f325c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4f3bc0f07e7a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.319+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260001370 con 0x7f325c004800 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.319+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32600052d0 con 0x7f325c004800 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.319+0000 7f3275400640 1 -- 172.21.15.6:0/703655787 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3260003830 con 0x7f325c004800 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 -- 172.21.15.6:0/703655787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 msgr2=0x7f325c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f325c004c80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3260001750 tx=0x7f325c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 -- 172.21.15.6:0/703655787 shutdown_connections 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 --2- 172.21.15.6:0/703655787 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f325c004c80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 -- 172.21.15.6:0/703655787 >> 172.21.15.6:0/703655787 conn(0x7f32780d3d40 msgr2=0x7f32780d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 -- 172.21.15.6:0/703655787 shutdown_connections 2024-03-06T07:31:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.320+0000 7f3277200640 1 -- 172.21.15.6:0/703655787 wait complete. 2024-03-06T07:31:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.322+0000 7f3277200640 1 Processor -- start 2024-03-06T07:31:50.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.322+0000 7f3277200640 1 -- start start 2024-03-06T07:31:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3277200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f3278151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3277200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f327815a8a0 0x7f32780cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3277200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f327815a2e0 0x7f32780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3277200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780b14b0 con 0x7f325c004800 2024-03-06T07:31:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3277200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780b1330 con 0x7f327815a8a0 2024-03-06T07:31:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3277200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32780b1630 con 0x7f327815a2e0 2024-03-06T07:31:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3275e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f3278151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3275e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f3278151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35726/0 (socket says 172.21.15.6:35726) 2024-03-06T07:31:50.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.323+0000 7f3275e00640 1 -- 172.21.15.6:0/4021180458 learned_addr learned my addr 172.21.15.6:0/4021180458 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f326d400640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f327815a8a0 0x7f32780cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3276800640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f327815a2e0 0x7f32780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3275e00640 1 -- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f327815a2e0 msgr2=0x7f32780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3275e00640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f327815a2e0 0x7f32780cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3275e00640 1 -- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f327815a8a0 msgr2=0x7f32780cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3275e00640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f327815a8a0 0x7f32780cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3275e00640 1 -- 172.21.15.6:0/4021180458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3278151780 con 0x7f325c004800 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.324+0000 7f3275e00640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f3278151240 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3268001ff0 tx=0x7f326800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.325+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268019070 con 0x7f325c004800 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.325+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326800bb50 con 0x7f325c004800 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.325+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32680070d0 con 0x7f325c004800 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.325+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32780b6d40 con 0x7f325c004800 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.325+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32780b7300 con 0x7f325c004800 2024-03-06T07:31:50.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.326+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f327804f330 con 0x7f325c004800 2024-03-06T07:31:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.327+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f326801c070 con 0x7f325c004800 2024-03-06T07:31:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.328+0000 7f326ea00640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f325406b500 0x7f325406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.328+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f326808a320 con 0x7f325c004800 2024-03-06T07:31:50.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.328+0000 7f326d400640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f325406b500 0x7f325406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:50.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.329+0000 7f326d400640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f325406b500 0x7f325406d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3260033000 tx=0x7f3260009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:50.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.332+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3278151780 con 0x7f325c004800 2024-03-06T07:31:50.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.572+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32780b1b60 con 0x7f325c004800 2024-03-06T07:31:50.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.573+0000 7f326ea00640 1 -- 172.21.15.6:0/4021180458 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f3268059100 con 0x7f325c004800 2024-03-06T07:31:50.575 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:50.576 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f325406b500 msgr2=0x7f325406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f325406b500 0x7f325406d9c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3260033000 tx=0x7f3260009030 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 msgr2=0x7f3278151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f3278151240 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f3268001ff0 tx=0x7f326800b4a0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 shutdown_connections 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f325406b500 0x7f325406d9c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f327815a8a0 0x7f32780cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f327815a2e0 0x7f32780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 --2- 172.21.15.6:0/4021180458 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f325c004800 0x7f3278151240 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 >> 172.21.15.6:0/4021180458 conn(0x7f32780d3d40 msgr2=0x7f32780d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 shutdown_connections 2024-03-06T07:31:50.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:50.576+0000 7f3277200640 1 -- 172.21.15.6:0/4021180458 wait complete. 2024-03-06T07:31:50.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:31:50.826 INFO:teuthology.orchestra.run.smithi006.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:31:50.826 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:31:50.826 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:50.826 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> import uuid 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> n = 10 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = False 2024-03-06T07:31:50.878 DEBUG:teuthology.orchestra.run.smithi006:> if 0 > 0: 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> create_hard_links = True 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> dpath = os.path.dirname(path) 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> fnameprefix = os.path.basename(path) 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(n): 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> f.write(f"{i}") 2024-03-06T07:31:50.879 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> f.flush() 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(f.fileno()) 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> os.unlink(fpath) 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> if True: 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> if create_hard_links: 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> for j in range(0): 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> if False: 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> os.fsync(dirfd) 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> finally: 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> os.close(dirfd) 2024-03-06T07:31:50.880 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:31:51.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:31:51.133 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:31:51.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:51.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:31:51.411 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:31:51.411 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:51.411 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:51.475 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:51.475 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:51.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:51.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.750+0000 7f6d4d000640 1 Processor -- start 2024-03-06T07:31:51.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.751+0000 7f6d4d000640 1 -- start start 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d4d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d4815e070 0x7f6d480d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d4d000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d4d000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d480d18c0 con 0x7f6d480d05c0 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d4d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d480d1ba0 con 0x7f6d4815e070 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d4d000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d480d1e80 con 0x7f6d480cf410 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d47400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d4815e070 0x7f6d480d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d47400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6d4815e070 0x7f6d480d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35738/0 (socket says 172.21.15.6:35738) 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.752+0000 7f6d47400640 1 -- 172.21.15.6:0/1813805182 learned_addr learned my addr 172.21.15.6:0/1813805182 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.753+0000 7f6d3fe00640 1 --2- 172.21.15.6:0/1813805182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.753+0000 7f6d47400640 1 -- 172.21.15.6:0/1813805182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 msgr2=0x7f6d480cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.753+0000 7f6d47400640 1 --2- 172.21.15.6:0/1813805182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.753+0000 7f6d47400640 1 -- 172.21.15.6:0/1813805182 >> v1:172.21.15.178:6789/0 conn(0x7f6d480d05c0 legacy=0x7f6d480b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.753+0000 7f6d47400640 1 -- 172.21.15.6:0/1813805182 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d480d2160 con 0x7f6d4815e070 2024-03-06T07:31:51.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.754+0000 7f6d47400640 1 --2- 172.21.15.6:0/1813805182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d480d2960 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009870 tx=0x7f6d3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cca3594cc07e16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.754+0000 7f6d46a00640 1 -- 172.21.15.6:0/1813805182 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38034070 con 0x7f6d4815e070 2024-03-06T07:31:51.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.754+0000 7f6d46a00640 1 -- 172.21.15.6:0/1813805182 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d38030840 con 0x7f6d4815e070 2024-03-06T07:31:51.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.754+0000 7f6d46a00640 1 -- 172.21.15.6:0/1813805182 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38030c10 con 0x7f6d4815e070 2024-03-06T07:31:51.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 -- 172.21.15.6:0/1813805182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 msgr2=0x7f6d480d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 --2- 172.21.15.6:0/1813805182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d480d2960 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6d38009870 tx=0x7f6d3802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:51.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 -- 172.21.15.6:0/1813805182 shutdown_connections 2024-03-06T07:31:51.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 --2- 172.21.15.6:0/1813805182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 --2- 172.21.15.6:0/1813805182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d480d2960 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 -- 172.21.15.6:0/1813805182 >> 172.21.15.6:0/1813805182 conn(0x7f6d4808d2a0 msgr2=0x7f6d480d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 -- 172.21.15.6:0/1813805182 shutdown_connections 2024-03-06T07:31:51.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.755+0000 7f6d4d000640 1 -- 172.21.15.6:0/1813805182 wait complete. 2024-03-06T07:31:51.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.757+0000 7f6d4d000640 1 Processor -- start 2024-03-06T07:31:51.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.757+0000 7f6d4d000640 1 -- start start 2024-03-06T07:31:51.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d4d000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d4d000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d480d05c0 0x7f6d48160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d4d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d481f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d4d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d480d4730 con 0x7f6d4815e070 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d4d000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d480d45b0 con 0x7f6d480d05c0 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d4d000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d480d0a50 con 0x7f6d480cf410 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d47e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d481f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d47e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d481f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35752/0 (socket says 172.21.15.6:35752) 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.758+0000 7f6d47e00640 1 -- 172.21.15.6:0/2980874087 learned_addr learned my addr 172.21.15.6:0/2980874087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d3fe00640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d480d05c0 0x7f6d48160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47400640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47e00640 1 -- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 msgr2=0x7f6d480cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47e00640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47e00640 1 -- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d480d05c0 msgr2=0x7f6d48160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47e00640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d480d05c0 0x7f6d48160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47e00640 1 -- 172.21.15.6:0/2980874087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d480cbf20 con 0x7f6d4815e070 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d47e00640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d481f3a20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000c740 tx=0x7f6d4000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.759+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40016020 con 0x7f6d4815e070 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.760+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4000ce50 con 0x7f6d4815e070 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.760+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d40003970 con 0x7f6d4815e070 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.760+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d480cc200 con 0x7f6d4815e070 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.760+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d480cc7c0 con 0x7f6d4815e070 2024-03-06T07:31:51.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.761+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d480cab40 con 0x7f6d4815e070 2024-03-06T07:31:51.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.762+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d40003b10 con 0x7f6d4815e070 2024-03-06T07:31:51.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.763+0000 7f6d3f400640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6d2406b4d0 0x7f6d2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.763+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d40089dc0 con 0x7f6d4815e070 2024-03-06T07:31:51.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.764+0000 7f6d47400640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6d2406b4d0 0x7f6d2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.765+0000 7f6d47400640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6d2406b4d0 0x7f6d2406d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6d38036000 tx=0x7f6d3803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.772+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d480cab40 con 0x7f6d4815e070 2024-03-06T07:31:51.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.978+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d4804f330 con 0x7f6d4815e070 2024-03-06T07:31:51.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.980+0000 7f6d3f400640 1 -- 172.21.15.6:0/2980874087 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f6d40020020 con 0x7f6d4815e070 2024-03-06T07:31:51.983 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:51.983 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:17.916537+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.983+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6d2406b4d0 msgr2=0x7f6d2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.983+0000 7f6d4d000640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6d2406b4d0 0x7f6d2406d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6d38036000 tx=0x7f6d3803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.983+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 msgr2=0x7f6d481f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.983+0000 7f6d4d000640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d481f3a20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6d4000c740 tx=0x7f6d4000cc00 comp rx=0 tx=0).stop 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 shutdown_connections 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6d2406b4d0 0x7f6d2406d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d480d05c0 0x7f6d48160890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d4815e070 0x7f6d481f3a20 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 --2- 172.21.15.6:0/2980874087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6d480cf410 0x7f6d480cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 >> 172.21.15.6:0/2980874087 conn(0x7f6d4808d2a0 msgr2=0x7f6d4806d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 shutdown_connections 2024-03-06T07:31:51.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:51.984+0000 7f6d4d000640 1 -- 172.21.15.6:0/2980874087 wait complete. 2024-03-06T07:31:52.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:14.222223+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:15.396328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:52.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.226+0000 7f7615800640 1 Processor -- start 2024-03-06T07:31:52.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.226+0000 7f7615800640 1 -- start start 2024-03-06T07:31:52.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.228+0000 7f7615800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 0x7f76100cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.228+0000 7f7615800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7610072df0 con 0x7f761015bf20 2024-03-06T07:31:52.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.228+0000 7f7615800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100cdab0 con 0x7f7610071e20 2024-03-06T07:31:52.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.228+0000 7f7615800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76100b4d70 con 0x7f7610072520 2024-03-06T07:31:52.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.228+0000 7f760ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f761015bf20 0x7f76100acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44820/0 (socket says 172.21.15.6:44820) 2024-03-06T07:31:52.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.228+0000 7f760ea00640 1 -- 172.21.15.6:0/3409281198 learned_addr learned my addr 172.21.15.6:0/3409281198 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f7606000640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 0x7f76100cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473054928 0 0) 0x7f7610072df0 con 0x7f761015bf20 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f4003520 con 0x7f761015bf20 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1936592622 0 0) 0x7f75f4003520 con 0x7f761015bf20 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 msgr2=0x7f76100cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 0x7f76100cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 >> v1:172.21.15.178:6789/0 conn(0x7f7610071e20 legacy=0x7f76100ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100b5050 con 0x7f761015bf20 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f76000019f0 con 0x7f761015bf20 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 >> v1:172.21.15.6:6789/0 conn(0x7f761015bf20 legacy=0x7f76100acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75f40039e0 0x7f75f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75f4004800 0x7f75f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 0x7f75f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76100b4d70 con 0x7f75f40039e0 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400ba70 con 0x7f75f4004800 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.229+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400bc40 con 0x7f7610072520 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f760ea00640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75f40039e0 0x7f75f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f7606000640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75f4004800 0x7f75f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f760ea00640 1 -- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 msgr2=0x7f75f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f760ea00640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7610072520 0x7f75f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f760ea00640 1 -- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75f4004800 msgr2=0x7f75f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f760ea00640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75f4004800 0x7f75f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.230+0000 7f760ea00640 1 -- 172.21.15.6:0/3409281198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4003520 con 0x7f75f40039e0 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.231+0000 7f760ea00640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75f40039e0 0x7f75f4003dc0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f760000a780 tx=0x7f75f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15521e68af4cdce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.231+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000a3d0 con 0x7f75f40039e0 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600034630 con 0x7f75f40039e0 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f760e000640 1 -- 172.21.15.6:0/3409281198 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600034a20 con 0x7f75f40039e0 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f7615800640 1 -- 172.21.15.6:0/3409281198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75f40039e0 msgr2=0x7f75f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f7615800640 1 --2- 172.21.15.6:0/3409281198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f75f40039e0 0x7f75f4003dc0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f760000a780 tx=0x7f75f4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f760f400640 1 -- 172.21.15.6:0/3409281198 reap_dead start 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f7615800640 1 -- 172.21.15.6:0/3409281198 shutdown_connections 2024-03-06T07:31:52.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f7615800640 1 -- 172.21.15.6:0/3409281198 >> 172.21.15.6:0/3409281198 conn(0x7f761006b3d0 msgr2=0x7f761006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f7615800640 1 -- 172.21.15.6:0/3409281198 shutdown_connections 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.232+0000 7f7615800640 1 -- 172.21.15.6:0/3409281198 wait complete. 2024-03-06T07:31:52.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.234+0000 7f7615800640 1 Processor -- start 2024-03-06T07:31:52.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.234+0000 7f7615800640 1 -- start start 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7615800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75f40039e0 0x7f76100af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7615800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 0x7f761014f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7615800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76100ca160 0x7f76100ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7615800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76100ad450 con 0x7f76100b12c0 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7615800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76100b4990 con 0x7f76100ca160 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7615800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76101572d0 con 0x7f75f40039e0 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.235+0000 7f7606000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 0x7f761014f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 0x7f761014f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52992/0 (socket says 172.21.15.6:52992) 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 -- 172.21.15.6:0/2101471053 learned_addr learned my addr 172.21.15.6:0/2101471053 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f760f400640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76100ca160 0x7f76100ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 -- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75f40039e0 msgr2=0x7f76100af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f760ea00640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75f40039e0 0x7f76100af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75f40039e0 0x7f76100af7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 -- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76100ca160 msgr2=0x7f76100ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76100ca160 0x7f76100ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 -- 172.21.15.6:0/2101471053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761014fbc0 con 0x7f76100b12c0 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f760f400640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76100ca160 0x7f76100ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:52.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.236+0000 7f7606000640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 0x7f761014f680 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f75f8011610 tx=0x7f75f8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.237+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8004220 con 0x7f76100b12c0 2024-03-06T07:31:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.237+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f80043c0 con 0x7f76100b12c0 2024-03-06T07:31:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.237+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76100afdb0 con 0x7f76100b12c0 2024-03-06T07:31:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.237+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f800f8f0 con 0x7f76100b12c0 2024-03-06T07:31:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.237+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76100b0370 con 0x7f76100b12c0 2024-03-06T07:31:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.238+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76100cab20 con 0x7f76100b12c0 2024-03-06T07:31:52.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.239+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75f800fa90 con 0x7f76100b12c0 2024-03-06T07:31:52.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.240+0000 7f7607400640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ec06b4d0 0x7f75ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.240+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75f808a4a0 con 0x7f76100b12c0 2024-03-06T07:31:52.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.241+0000 7f760ea00640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ec06b4d0 0x7f75ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.242+0000 7f760ea00640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ec06b4d0 0x7f75ec06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7600034000 tx=0x7f760002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.244+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75f8019030 con 0x7f76100b12c0 2024-03-06T07:31:52.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.446+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f761004f330 con 0x7f76100b12c0 2024-03-06T07:31:52.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.446+0000 7f7607400640 1 -- 172.21.15.6:0/2101471053 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75f8059290 con 0x7f76100b12c0 2024-03-06T07:31:52.449 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ec06b4d0 msgr2=0x7f75ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ec06b4d0 0x7f75ec06d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7600034000 tx=0x7f760002f000 comp rx=0 tx=0).stop 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 msgr2=0x7f761014f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 0x7f761014f680 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f75f8011610 tx=0x7f75f8011ad0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 shutdown_connections 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f75ec06b4d0 0x7f75ec06d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76100ca160 0x7f76100ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76100b12c0 0x7f761014f680 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 --2- 172.21.15.6:0/2101471053 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75f40039e0 0x7f76100af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.449+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 >> 172.21.15.6:0/2101471053 conn(0x7f761006b3d0 msgr2=0x7f761015b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.450+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 shutdown_connections 2024-03-06T07:31:52.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.450+0000 7f7615800640 1 -- 172.21.15.6:0/2101471053 wait complete. 2024-03-06T07:31:52.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:52.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:52.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:52.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.712+0000 7fc26cc00640 1 Processor -- start 2024-03-06T07:31:52.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.712+0000 7fc26cc00640 1 -- start start 2024-03-06T07:31:52.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.714+0000 7fc26cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2680ccad0 0x7fc2680ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.714+0000 7fc26cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26806d090 con 0x7fc26815e680 2024-03-06T07:31:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.714+0000 7fc26cc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26815eb80 con 0x7fc26806da40 2024-03-06T07:31:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.714+0000 7fc26cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc26806ea50 con 0x7fc2680ccad0 2024-03-06T07:31:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc266800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2680ccad0 0x7fc2680ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc265400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc26815e680 0x7fc26806c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44822/0 (socket says 172.21.15.6:44822) 2024-03-06T07:31:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc265400640 1 -- 172.21.15.6:0/2630052015 learned_addr learned my addr 172.21.15.6:0/2630052015 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:52.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc264a00640 1 -- 172.21.15.6:0/2630052015 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994094004 0 0) 0x7fc26806d090 con 0x7fc26815e680 2024-03-06T07:31:52.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc264a00640 1 -- 172.21.15.6:0/2630052015 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc24c003520 con 0x7fc26815e680 2024-03-06T07:31:52.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc266800640 1 -- 172.21.15.6:0/2630052015 >> v1:172.21.15.178:6789/0 conn(0x7fc26806da40 legacy=0x7fc26806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc266800640 1 -- 172.21.15.6:0/2630052015 >> v1:172.21.15.6:6789/0 conn(0x7fc26815e680 legacy=0x7fc26806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.715+0000 7fc266800640 1 -- 172.21.15.6:0/2630052015 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26815ee60 con 0x7fc2680ccad0 2024-03-06T07:31:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.717+0000 7fc266800640 1 --2- 172.21.15.6:0/2630052015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2680ccad0 0x7fc2680ccf80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc260009d90 tx=0x7fc26002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b834ead2570d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.717+0000 7fc264a00640 1 -- 172.21.15.6:0/2630052015 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260039070 con 0x7fc2680ccad0 2024-03-06T07:31:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.717+0000 7fc264a00640 1 -- 172.21.15.6:0/2630052015 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26002cd00 con 0x7fc2680ccad0 2024-03-06T07:31:52.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.717+0000 7fc264a00640 1 -- 172.21.15.6:0/2630052015 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260034640 con 0x7fc2680ccad0 2024-03-06T07:31:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.717+0000 7fc26cc00640 1 -- 172.21.15.6:0/2630052015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2680ccad0 msgr2=0x7fc2680ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.717+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2630052015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2680ccad0 0x7fc2680ccf80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fc260009d90 tx=0x7fc26002bf50 comp rx=0 tx=0).stop 2024-03-06T07:31:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.718+0000 7fc26cc00640 1 -- 172.21.15.6:0/2630052015 shutdown_connections 2024-03-06T07:31:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.718+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2630052015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2680ccad0 0x7fc2680ccf80 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.718+0000 7fc26cc00640 1 -- 172.21.15.6:0/2630052015 >> 172.21.15.6:0/2630052015 conn(0x7fc26806b3d0 msgr2=0x7fc26806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.718+0000 7fc26cc00640 1 -- 172.21.15.6:0/2630052015 shutdown_connections 2024-03-06T07:31:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.718+0000 7fc26cc00640 1 -- 172.21.15.6:0/2630052015 wait complete. 2024-03-06T07:31:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.719+0000 7fc26cc00640 1 Processor -- start 2024-03-06T07:31:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.720+0000 7fc26cc00640 1 -- start start 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc26cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc26806da40 0x7fc26814a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc26cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2680ccad0 0x7fc26809cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc26cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 0x7fc2681f8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc26cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26815f420 con 0x7fc26815e680 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc26cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268161c70 con 0x7fc26806da40 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc26cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc268162100 con 0x7fc2680ccad0 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc266800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 0x7fc2681f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc266800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 0x7fc2681f8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53004/0 (socket says 172.21.15.6:53004) 2024-03-06T07:31:52.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc266800640 1 -- 172.21.15.6:0/2189834485 learned_addr learned my addr 172.21.15.6:0/2189834485 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc265400640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2680ccad0 0x7fc26809cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.721+0000 7fc265e00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc26806da40 0x7fc26814a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc266800640 1 -- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2680ccad0 msgr2=0x7fc26809cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc266800640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2680ccad0 0x7fc26809cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc266800640 1 -- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc26806da40 msgr2=0x7fc26814a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc266800640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc26806da40 0x7fc26814a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc266800640 1 -- 172.21.15.6:0/2189834485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26809d520 con 0x7fc26815e680 2024-03-06T07:31:52.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc265e00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc26806da40 0x7fc26814a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc266800640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 0x7fc2681f8160 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc260009eb0 tx=0x7fc260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260043020 con 0x7fc26815e680 2024-03-06T07:31:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc260003e10 con 0x7fc26815e680 2024-03-06T07:31:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.722+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2681f86a0 con 0x7fc26815e680 2024-03-06T07:31:52.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.723+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260039040 con 0x7fc26815e680 2024-03-06T07:31:52.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.723+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26809c220 con 0x7fc26815e680 2024-03-06T07:31:52.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.724+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26804f330 con 0x7fc26815e680 2024-03-06T07:31:52.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.725+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc260005d10 con 0x7fc26815e680 2024-03-06T07:31:52.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.726+0000 7fc25e000640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc24806b4d0 0x7fc24806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:52.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.726+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2600ac8b0 con 0x7fc26815e680 2024-03-06T07:31:52.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.726+0000 7fc265e00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc24806b4d0 0x7fc24806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:52.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.727+0000 7fc265e00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc24806b4d0 0x7fc24806d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc258001700 tx=0x7fc258009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:52.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.730+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc26004d070 con 0x7fc26815e680 2024-03-06T07:31:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.937+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc26806b840 con 0x7fc26815e680 2024-03-06T07:31:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.938+0000 7fc25e000640 1 -- 172.21.15.6:0/2189834485 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc26007b750 con 0x7fc26815e680 2024-03-06T07:31:52.940 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:31:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc24806b4d0 msgr2=0x7fc24806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc24806b4d0 0x7fc24806d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc258001700 tx=0x7fc258009040 comp rx=0 tx=0).stop 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 msgr2=0x7fc2681f8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 0x7fc2681f8160 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc260009eb0 tx=0x7fc260004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 shutdown_connections 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc24806b4d0 0x7fc24806d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc26815e680 0x7fc2681f8160 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2680ccad0 0x7fc26809cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 --2- 172.21.15.6:0/2189834485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc26806da40 0x7fc26814a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 >> 172.21.15.6:0/2189834485 conn(0x7fc26806b3d0 msgr2=0x7fc26815f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 shutdown_connections 2024-03-06T07:31:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:52.941+0000 7fc26cc00640 1 -- 172.21.15.6:0/2189834485 wait complete. 2024-03-06T07:31:52.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:53.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.178+0000 7f0ecfe00640 1 Processor -- start 2024-03-06T07:31:53.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.179+0000 7f0ecfe00640 1 -- start start 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.180+0000 7f0ecfe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0ed015dee0 0x7f0ed00d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.180+0000 7f0ecfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ed00d4860 0x7f0ed00d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.180+0000 7f0ecfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00ceca0 con 0x7f0ed00cfa60 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.180+0000 7f0ecfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed00cf5d0 con 0x7f0ed00d4860 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.180+0000 7f0ecfe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed00cef80 con 0x7f0ed015dee0 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ece000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ed00d4860 0x7f0ed00d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ecf400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0ed00cfa60 0x7f0ed00b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44838/0 (socket says 172.21.15.6:44838) 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ecf400640 1 -- 172.21.15.6:0/1632501979 learned_addr learned my addr 172.21.15.6:0/1632501979 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:53.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ecea00640 1 --2- 172.21.15.6:0/1632501979 >> v2:172.21.15.178:3300/0 conn(0x7f0ed015dee0 0x7f0ed00d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ece000640 1 -- 172.21.15.6:0/1632501979 >> v2:172.21.15.178:3300/0 conn(0x7f0ed015dee0 msgr2=0x7f0ed00d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ece000640 1 --2- 172.21.15.6:0/1632501979 >> v2:172.21.15.178:3300/0 conn(0x7f0ed015dee0 0x7f0ed00d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ece000640 1 -- 172.21.15.6:0/1632501979 >> v1:172.21.15.6:6789/0 conn(0x7f0ed00cfa60 legacy=0x7f0ed00b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ece000640 1 -- 172.21.15.6:0/1632501979 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed00cf260 con 0x7f0ed00d4860 2024-03-06T07:31:53.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.181+0000 7f0ecd600640 1 -- 172.21.15.6:0/1632501979 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197279801 0 0) 0x7f0ed00ceca0 con 0x7f0ed00cfa60 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.182+0000 7f0ece000640 1 --2- 172.21.15.6:0/1632501979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00d4860 0x7f0ed00d0630 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0ed008c6f0 tx=0x7f0ebc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b700cc074ea59993 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.183+0000 7f0ecd600640 1 -- 172.21.15.6:0/1632501979 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc035020 con 0x7f0ed00d4860 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.183+0000 7f0ecd600640 1 -- 172.21.15.6:0/1632501979 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc031830 con 0x7f0ed00d4860 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.183+0000 7f0ecd600640 1 -- 172.21.15.6:0/1632501979 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc031c00 con 0x7f0ed00d4860 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.183+0000 7f0ecfe00640 1 -- 172.21.15.6:0/1632501979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00d4860 msgr2=0x7f0ed00d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.183+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/1632501979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00d4860 0x7f0ed00d0630 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0ed008c6f0 tx=0x7f0ebc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.183+0000 7f0ecfe00640 1 -- 172.21.15.6:0/1632501979 shutdown_connections 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.184+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/1632501979 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00d4860 0x7f0ed00d0630 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.184+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/1632501979 >> v2:172.21.15.178:3300/0 conn(0x7f0ed015dee0 0x7f0ed00d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.184+0000 7f0ecfe00640 1 -- 172.21.15.6:0/1632501979 >> 172.21.15.6:0/1632501979 conn(0x7f0ed00d3cd0 msgr2=0x7f0ed00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.184+0000 7f0ecfe00640 1 -- 172.21.15.6:0/1632501979 shutdown_connections 2024-03-06T07:31:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.184+0000 7f0ecfe00640 1 -- 172.21.15.6:0/1632501979 wait complete. 2024-03-06T07:31:53.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.186+0000 7f0ecfe00640 1 Processor -- start 2024-03-06T07:31:53.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.187+0000 7f0ecfe00640 1 -- start start 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.188+0000 7f0ecfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 0x7f0ed015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.188+0000 7f0ecfe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ed00d4860 0x7f0ed00a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.188+0000 7f0ecfe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ed015dee0 0x7f0ed009fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.188+0000 7f0ecfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed00d1de0 con 0x7f0ed00cfa60 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.188+0000 7f0ecfe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed015e2c0 con 0x7f0ed015dee0 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.188+0000 7f0ecfe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed00d1f60 con 0x7f0ed00d4860 2024-03-06T07:31:53.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 0x7f0ed015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 0x7f0ed015c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53026/0 (socket says 172.21.15.6:53026) 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 -- 172.21.15.6:0/2679737172 learned_addr learned my addr 172.21.15.6:0/2679737172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ece000640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ed00d4860 0x7f0ed00a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecf400640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ed015dee0 0x7f0ed009fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 -- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ed00d4860 msgr2=0x7f0ed00a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ed00d4860 0x7f0ed00a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 -- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ed015dee0 msgr2=0x7f0ed009fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ed015dee0 0x7f0ed009fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 -- 172.21.15.6:0/2679737172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed015c780 con 0x7f0ed00cfa60 2024-03-06T07:31:53.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.189+0000 7f0ecea00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 0x7f0ed015c240 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000e870 tx=0x7f0ec000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.190+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0017070 con 0x7f0ed00cfa60 2024-03-06T07:31:53.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.190+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec000c940 con 0x7f0ed00cfa60 2024-03-06T07:31:53.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.190+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000cd30 con 0x7f0ed00cfa60 2024-03-06T07:31:53.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.190+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed00a0340 con 0x7f0ed00cfa60 2024-03-06T07:31:53.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.190+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed00a0900 con 0x7f0ed00cfa60 2024-03-06T07:31:53.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.191+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed004f330 con 0x7f0ed00cfa60 2024-03-06T07:31:53.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.192+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ec001a070 con 0x7f0ed00cfa60 2024-03-06T07:31:53.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.193+0000 7f0ec6a00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0eb006b4d0 0x7f0eb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.193+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ec008a060 con 0x7f0ed00cfa60 2024-03-06T07:31:53.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.194+0000 7f0ece000640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0eb006b4d0 0x7f0eb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.195+0000 7f0ece000640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0eb006b4d0 0x7f0eb006d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc036000 tx=0x7f0ebc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.203+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ec00587f0 con 0x7f0ed00cfa60 2024-03-06T07:31:53.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.404+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ed008d7d0 con 0x7f0ed00cfa60 2024-03-06T07:31:53.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.405+0000 7f0ec6a00640 1 -- 172.21.15.6:0/2679737172 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ec005c890 con 0x7f0ed00cfa60 2024-03-06T07:31:53.408 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:31:53.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0eb006b4d0 msgr2=0x7f0eb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0eb006b4d0 0x7f0eb006d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc036000 tx=0x7f0ebc008030 comp rx=0 tx=0).stop 2024-03-06T07:31:53.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 msgr2=0x7f0ed015c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 0x7f0ed015c240 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000e870 tx=0x7f0ec000ed30 comp rx=0 tx=0).stop 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 shutdown_connections 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0eb006b4d0 0x7f0eb006d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ed015dee0 0x7f0ed009fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ed00d4860 0x7f0ed00a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 --2- 172.21.15.6:0/2679737172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ed00cfa60 0x7f0ed015c240 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 >> 172.21.15.6:0/2679737172 conn(0x7f0ed00d3cd0 msgr2=0x7f0ed0160450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 shutdown_connections 2024-03-06T07:31:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.408+0000 7f0ecfe00640 1 -- 172.21.15.6:0/2679737172 wait complete. 2024-03-06T07:31:53.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:31:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.639+0000 7f2aa4e00640 1 Processor -- start 2024-03-06T07:31:53.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.640+0000 7f2aa4e00640 1 -- start start 2024-03-06T07:31:53.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.641+0000 7f2aa4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00b0f90 0x7f2aa00cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.641+0000 7f2aa4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2aa00d07b0 0x7f2aa015d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.641+0000 7f2aa4e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa00b1370 con 0x7f2aa00cc730 2024-03-06T07:31:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.641+0000 7f2aa4e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa00ccf80 con 0x7f2aa00d07b0 2024-03-06T07:31:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.641+0000 7f2aa4e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00cd260 con 0x7f2aa00b0f90 2024-03-06T07:31:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.642+0000 7f2a97e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2aa00cc730 0x7f2aa00d00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44840/0 (socket says 172.21.15.6:44840) 2024-03-06T07:31:53.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.642+0000 7f2a97e00640 1 -- 172.21.15.6:0/4249081043 learned_addr learned my addr 172.21.15.6:0/4249081043 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.642+0000 7f2a9f400640 1 --2- 172.21.15.6:0/4249081043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00b0f90 0x7f2aa00cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.642+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/4249081043 >> v2:172.21.15.6:3300/0 conn(0x7f2aa00d07b0 0x7f2aa015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.642+0000 7f2a9ea00640 1 -- 172.21.15.6:0/4249081043 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732830703 0 0) 0x7f2aa00b1370 con 0x7f2aa00cc730 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.642+0000 7f2a9ea00640 1 -- 172.21.15.6:0/4249081043 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a84003520 con 0x7f2aa00cc730 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.643+0000 7f2a9f400640 1 -- 172.21.15.6:0/4249081043 >> v2:172.21.15.6:3300/0 conn(0x7f2aa00d07b0 msgr2=0x7f2aa015d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.643+0000 7f2a9f400640 1 --2- 172.21.15.6:0/4249081043 >> v2:172.21.15.6:3300/0 conn(0x7f2aa00d07b0 0x7f2aa015d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.643+0000 7f2a9f400640 1 -- 172.21.15.6:0/4249081043 >> v1:172.21.15.6:6789/0 conn(0x7f2aa00cc730 legacy=0x7f2aa00d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.643+0000 7f2a9f400640 1 -- 172.21.15.6:0/4249081043 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa00cd540 con 0x7f2aa00b0f90 2024-03-06T07:31:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.644+0000 7f2a9f400640 1 --2- 172.21.15.6:0/4249081043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00b0f90 0x7f2aa00cf080 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2a900097a0 tx=0x7f2a9002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=582fecdd84113195 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.645+0000 7f2a9ea00640 1 -- 172.21.15.6:0/4249081043 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90031780 con 0x7f2aa00b0f90 2024-03-06T07:31:53.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.645+0000 7f2a9ea00640 1 -- 172.21.15.6:0/4249081043 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90037070 con 0x7f2aa00b0f90 2024-03-06T07:31:53.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.645+0000 7f2a9ea00640 1 -- 172.21.15.6:0/4249081043 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90031b20 con 0x7f2aa00b0f90 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.645+0000 7f2aa4e00640 1 -- 172.21.15.6:0/4249081043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00b0f90 msgr2=0x7f2aa00cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.645+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/4249081043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00b0f90 0x7f2aa00cf080 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2a900097a0 tx=0x7f2a9002b820 comp rx=0 tx=0).stop 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.646+0000 7f2aa4e00640 1 -- 172.21.15.6:0/4249081043 shutdown_connections 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.646+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/4249081043 >> v2:172.21.15.6:3300/0 conn(0x7f2aa00d07b0 0x7f2aa015d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.646+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/4249081043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00b0f90 0x7f2aa00cf080 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.646+0000 7f2aa4e00640 1 -- 172.21.15.6:0/4249081043 >> 172.21.15.6:0/4249081043 conn(0x7f2aa00d1ae0 msgr2=0x7f2aa00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.646+0000 7f2aa4e00640 1 -- 172.21.15.6:0/4249081043 shutdown_connections 2024-03-06T07:31:53.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.646+0000 7f2aa4e00640 1 -- 172.21.15.6:0/4249081043 wait complete. 2024-03-06T07:31:53.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.647+0000 7f2aa4e00640 1 Processor -- start 2024-03-06T07:31:53.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 -- start start 2024-03-06T07:31:53.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 0x7f2aa00cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00cc730 0x7f2aa00c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00d07b0 0x7f2aa01f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00cccf0 con 0x7f2aa00b0f90 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa00ccb70 con 0x7f2aa00cc730 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.648+0000 7f2aa4e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa015dc90 con 0x7f2aa00d07b0 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 0x7f2aa00cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 0x7f2aa00cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53048/0 (socket says 172.21.15.6:53048) 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 -- 172.21.15.6:0/514506127 learned_addr learned my addr 172.21.15.6:0/514506127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a97e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00cc730 0x7f2aa00c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9fe00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00d07b0 0x7f2aa01f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 -- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00d07b0 msgr2=0x7f2aa01f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00d07b0 0x7f2aa01f4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 -- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00cc730 msgr2=0x7f2aa00c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00cc730 0x7f2aa00c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a9f400640 1 -- 172.21.15.6:0/514506127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa01f4830 con 0x7f2aa00b0f90 2024-03-06T07:31:53.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.649+0000 7f2a97e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00cc730 0x7f2aa00c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:53.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.650+0000 7f2a9f400640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 0x7f2aa00cb160 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2a900098c0 tx=0x7f2a900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.650+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90006e10 con 0x7f2aa00b0f90 2024-03-06T07:31:53.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.650+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a90031780 con 0x7f2aa00b0f90 2024-03-06T07:31:53.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.650+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90031b50 con 0x7f2aa00b0f90 2024-03-06T07:31:53.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.650+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa01f4550 con 0x7f2aa00b0f90 2024-03-06T07:31:53.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.650+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa00c79b0 con 0x7f2aa00b0f90 2024-03-06T07:31:53.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.652+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa004f330 con 0x7f2aa00b0f90 2024-03-06T07:31:53.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.652+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a90031cf0 con 0x7f2aa00b0f90 2024-03-06T07:31:53.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.653+0000 7f2a97400640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a7c06b4d0 0x7f2a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.653+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a900ac760 con 0x7f2aa00b0f90 2024-03-06T07:31:53.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.654+0000 7f2a97e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a7c06b4d0 0x7f2a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.655+0000 7f2a97e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a7c06b4d0 0x7f2a7c06d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2aa006e2a0 tx=0x7f2a88009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.658+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a9002f050 con 0x7f2aa00b0f90 2024-03-06T07:31:53.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.871+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2aa00d2160 con 0x7f2aa00b0f90 2024-03-06T07:31:53.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.872+0000 7f2a97400640 1 -- 172.21.15.6:0/514506127 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2a9007b540 con 0x7f2aa00b0f90 2024-03-06T07:31:53.874 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:31:53.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.874+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a7c06b4d0 msgr2=0x7f2a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.874+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a7c06b4d0 0x7f2a7c06d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2aa006e2a0 tx=0x7f2a88009040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.874+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 msgr2=0x7f2aa00cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.874+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 0x7f2aa00cb160 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2a900098c0 tx=0x7f2a900046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 shutdown_connections 2024-03-06T07:31:53.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2a7c06b4d0 0x7f2a7c06d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2aa00cc730 0x7f2aa00c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2aa00d07b0 0x7f2aa01f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 --2- 172.21.15.6:0/514506127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2aa00b0f90 0x7f2aa00cb160 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 >> 172.21.15.6:0/514506127 conn(0x7f2aa00d1ae0 msgr2=0x7f2aa015d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 shutdown_connections 2024-03-06T07:31:53.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:53.875+0000 7f2aa4e00640 1 -- 172.21.15.6:0/514506127 wait complete. 2024-03-06T07:31:53.896 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:31:53.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:53.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:31:54.025 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:31:54.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:31:54.025 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:31:54.078 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:31:54.078 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:54.113 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:54.114 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:31:54.114 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:54.151 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:31:54.151 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:31:54.151 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:31:54.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:31:54.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.395+0000 7fba50a00640 1 Processor -- start 2024-03-06T07:31:54.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.396+0000 7fba50a00640 1 -- start start 2024-03-06T07:31:54.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba50a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 0x7fba4c0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba50a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fba4c0d0c80 0x7fba4c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba50a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4c0b3710 con 0x7fba4c15e0b0 2024-03-06T07:31:54.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba50a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4c0b09e0 con 0x7fba4c0d0c80 2024-03-06T07:31:54.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba50a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0cfa20 con 0x7fba4c0d4a10 2024-03-06T07:31:54.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba4a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fba4c15e0b0 0x7fba4c0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44844/0 (socket says 172.21.15.6:44844) 2024-03-06T07:31:54.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.397+0000 7fba4a800640 1 -- 172.21.15.6:0/1985360045 learned_addr learned my addr 172.21.15.6:0/1985360045 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:54.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba4b200640 1 --2- 172.21.15.6:0/1985360045 >> v2:172.21.15.178:3300/0 conn(0x7fba4c0d0c80 0x7fba4c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba41e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 0x7fba4c0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559357886 0 0) 0x7fba4c0b3710 con 0x7fba4c15e0b0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba30003520 con 0x7fba4c15e0b0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353782036 0 0) 0x7fba30003520 con 0x7fba4c15e0b0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 msgr2=0x7fba4c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 0x7fba4c0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 >> v2:172.21.15.178:3300/0 conn(0x7fba4c0d0c80 msgr2=0x7fba4c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 --2- 172.21.15.6:0/1985360045 >> v2:172.21.15.178:3300/0 conn(0x7fba4c0d0c80 0x7fba4c0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c0b39f0 con 0x7fba4c15e0b0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fba3c001670 con 0x7fba4c15e0b0 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 >> v1:172.21.15.6:6789/0 conn(0x7fba4c15e0b0 legacy=0x7fba4c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba300039b0 0x7fba30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba30004800 0x7fba30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 0x7fba300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0cfa20 con 0x7fba30004800 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3000b9c0 con 0x7fba300039b0 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.398+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3000bb40 con 0x7fba4c0d4a10 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba41e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba30004800 0x7fba30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba4a800640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba300039b0 0x7fba30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba4b200640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 0x7fba300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba41e00640 1 -- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 msgr2=0x7fba300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba41e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba4c0d4a10 0x7fba300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba41e00640 1 -- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba300039b0 msgr2=0x7fba30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba41e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba300039b0 0x7fba30003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.399+0000 7fba41e00640 1 -- 172.21.15.6:0/1985360045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30003520 con 0x7fba30004800 2024-03-06T07:31:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.400+0000 7fba41e00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba30004800 0x7fba30004c80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fba34009a90 tx=0x7fba30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac026930f48bc95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34039070 con 0x7fba30004800 2024-03-06T07:31:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3402c680 con 0x7fba30004800 2024-03-06T07:31:54.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba49e00640 1 -- 172.21.15.6:0/1985360045 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34009690 con 0x7fba30004800 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba50a00640 1 -- 172.21.15.6:0/1985360045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba30004800 msgr2=0x7fba30004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba50a00640 1 --2- 172.21.15.6:0/1985360045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba30004800 0x7fba30004c80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fba34009a90 tx=0x7fba30005830 comp rx=0 tx=0).stop 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba4b200640 1 -- 172.21.15.6:0/1985360045 reap_dead start 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba50a00640 1 -- 172.21.15.6:0/1985360045 shutdown_connections 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba50a00640 1 -- 172.21.15.6:0/1985360045 >> 172.21.15.6:0/1985360045 conn(0x7fba4c0d3d30 msgr2=0x7fba4c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba50a00640 1 -- 172.21.15.6:0/1985360045 shutdown_connections 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.401+0000 7fba50a00640 1 -- 172.21.15.6:0/1985360045 wait complete. 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.403+0000 7fba50a00640 1 Processor -- start 2024-03-06T07:31:54.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.403+0000 7fba50a00640 1 -- start start 2024-03-06T07:31:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.404+0000 7fba50a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba30004800 0x7fba4c152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.404+0000 7fba50a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba4c0b7380 0x7fba4c0b7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.404+0000 7fba50a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 0x7fba4c0b9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.404+0000 7fba50a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c15e490 con 0x7fba4c0b90f0 2024-03-06T07:31:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.404+0000 7fba50a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0b3fb0 con 0x7fba4c0b7380 2024-03-06T07:31:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.404+0000 7fba50a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4c0cfc60 con 0x7fba30004800 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 0x7fba4c0b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 0x7fba4c0b9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53054/0 (socket says 172.21.15.6:53054) 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 -- 172.21.15.6:0/2499052756 learned_addr learned my addr 172.21.15.6:0/2499052756 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba41e00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba4c0b7380 0x7fba4c0b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4a800640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba30004800 0x7fba4c152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 -- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba30004800 msgr2=0x7fba4c152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba30004800 0x7fba4c152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 -- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba4c0b7380 msgr2=0x7fba4c0b7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba4c0b7380 0x7fba4c0b7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 -- 172.21.15.6:0/2499052756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c1f7890 con 0x7fba4c0b90f0 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.405+0000 7fba4b200640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 0x7fba4c0b9570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0d3c80 tx=0x7fba440115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.406+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44011e70 con 0x7fba4c0b90f0 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.406+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4401b070 con 0x7fba4c0b90f0 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.406+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba44003990 con 0x7fba4c0b90f0 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.406+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4c1f7b70 con 0x7fba4c0b90f0 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.406+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4c1f8130 con 0x7fba4c0b90f0 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.407+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4c04f330 con 0x7fba4c0b90f0 2024-03-06T07:31:54.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.408+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba44005d10 con 0x7fba4c0b90f0 2024-03-06T07:31:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.409+0000 7fba43400640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba2806b500 0x7fba2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.409+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba4408a660 con 0x7fba4c0b90f0 2024-03-06T07:31:54.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.410+0000 7fba4a800640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba2806b500 0x7fba2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.411+0000 7fba4a800640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba2806b500 0x7fba2806d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fba3c001050 tx=0x7fba3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.418+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba44014350 con 0x7fba4c0b90f0 2024-03-06T07:31:54.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.774+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba4c0b5660 con 0x7fba4c0b90f0 2024-03-06T07:31:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.775+0000 7fba43400640 1 -- 172.21.15.6:0/2499052756 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fba440594d0 con 0x7fba4c0b90f0 2024-03-06T07:31:54.777 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:54.777 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:54.777 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:54.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba2806b500 msgr2=0x7fba2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba2806b500 0x7fba2806d9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fba3c001050 tx=0x7fba3c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 msgr2=0x7fba4c0b9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 0x7fba4c0b9570 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fba4c0d3c80 tx=0x7fba440115a0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 shutdown_connections 2024-03-06T07:31:54.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fba2806b500 0x7fba2806d9c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fba4c0b7380 0x7fba4c0b7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fba4c0b90f0 0x7fba4c0b9570 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 --2- 172.21.15.6:0/2499052756 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fba30004800 0x7fba4c152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 >> 172.21.15.6:0/2499052756 conn(0x7fba4c0d3d30 msgr2=0x7fba4c0b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 shutdown_connections 2024-03-06T07:31:54.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:54.778+0000 7fba50a00640 1 -- 172.21.15.6:0/2499052756 wait complete. 2024-03-06T07:31:54.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:55.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.017+0000 7f851c400640 1 Processor -- start 2024-03-06T07:31:55.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.017+0000 7f851c400640 1 -- start start 2024-03-06T07:31:55.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f851c400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 0x7f8514070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f851c400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f85140ae0a0 0x7f85140711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f851c400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85140ce9a0 con 0x7f851406e240 2024-03-06T07:31:55.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f851c400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85140730d0 con 0x7f85140ae0a0 2024-03-06T07:31:55.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f851c400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140cec80 con 0x7f8514161670 2024-03-06T07:31:55.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f8519a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 0x7f8514070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f8519a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 0x7f8514070c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:52934/0 (socket says 172.21.15.6:52934) 2024-03-06T07:31:55.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.019+0000 7f8519a00640 1 -- 172.21.15.6:0/2263791238 learned_addr learned my addr 172.21.15.6:0/2263791238 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:55.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.020+0000 7f851a400640 1 --2- 172.21.15.6:0/2263791238 >> v2:172.21.15.178:3300/0 conn(0x7f85140ae0a0 0x7f85140711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.020+0000 7f8519a00640 1 -- 172.21.15.6:0/2263791238 >> v2:172.21.15.178:3300/0 conn(0x7f85140ae0a0 msgr2=0x7f85140711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.020+0000 7f8519a00640 1 --2- 172.21.15.6:0/2263791238 >> v2:172.21.15.178:3300/0 conn(0x7f85140ae0a0 0x7f85140711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.020+0000 7f8519a00640 1 -- 172.21.15.6:0/2263791238 >> v1:172.21.15.178:6789/0 conn(0x7f851406e240 legacy=0x7f851406cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.020+0000 7f8519a00640 1 -- 172.21.15.6:0/2263791238 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85140cef60 con 0x7f8514161670 2024-03-06T07:31:55.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.020+0000 7f851a400640 1 --2- 172.21.15.6:0/2263791238 >> v2:172.21.15.178:3300/0 conn(0x7f85140ae0a0 0x7f85140711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.021+0000 7f8519a00640 1 --2- 172.21.15.6:0/2263791238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 0x7f8514070c70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8508009870 tx=0x7f850802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=488965db46d7d34e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.022+0000 7f8513e00640 1 -- 172.21.15.6:0/2263791238 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8508034070 con 0x7f8514161670 2024-03-06T07:31:55.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.022+0000 7f8513e00640 1 -- 172.21.15.6:0/2263791238 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8508030840 con 0x7f8514161670 2024-03-06T07:31:55.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.022+0000 7f8513e00640 1 -- 172.21.15.6:0/2263791238 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8508030c10 con 0x7f8514161670 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 -- 172.21.15.6:0/2263791238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 msgr2=0x7f8514070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 --2- 172.21.15.6:0/2263791238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 0x7f8514070c70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8508009870 tx=0x7f850802b980 comp rx=0 tx=0).stop 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 -- 172.21.15.6:0/2263791238 shutdown_connections 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 --2- 172.21.15.6:0/2263791238 >> v2:172.21.15.178:3300/0 conn(0x7f85140ae0a0 0x7f85140711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 --2- 172.21.15.6:0/2263791238 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8514161670 0x7f8514070c70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 -- 172.21.15.6:0/2263791238 >> 172.21.15.6:0/2263791238 conn(0x7f851406afb0 msgr2=0x7f851406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 -- 172.21.15.6:0/2263791238 shutdown_connections 2024-03-06T07:31:55.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.023+0000 7f851c400640 1 -- 172.21.15.6:0/2263791238 wait complete. 2024-03-06T07:31:55.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.025+0000 7f851c400640 1 Processor -- start 2024-03-06T07:31:55.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.025+0000 7f851c400640 1 -- start start 2024-03-06T07:31:55.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851c400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f851406e240 0x7f8514154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851c400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85140ae0a0 0x7f85140ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851c400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 0x7f85140cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851c400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140ce600 con 0x7f8514161670 2024-03-06T07:31:55.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851c400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140cf520 con 0x7f85140ae0a0 2024-03-06T07:31:55.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851c400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85140ce780 con 0x7f851406e240 2024-03-06T07:31:55.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 0x7f85140cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f8519000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85140ae0a0 0x7f85140ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f8519000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85140ae0a0 0x7f85140ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:51054/0 (socket says 172.21.15.6:51054) 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f8519000640 1 -- 172.21.15.6:0/558501452 learned_addr learned my addr 172.21.15.6:0/558501452 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f851a400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 0x7f85140cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53062/0 (socket says 172.21.15.6:53062) 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.026+0000 7f8519a00640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f851406e240 0x7f8514154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.027+0000 7f851a400640 1 -- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f851406e240 msgr2=0x7f8514154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.027+0000 7f851a400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f851406e240 0x7f8514154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.027+0000 7f851a400640 1 -- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85140ae0a0 msgr2=0x7f85140ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.027+0000 7f851a400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85140ae0a0 0x7f85140ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.027+0000 7f851a400640 1 -- 172.21.15.6:0/558501452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85141548c0 con 0x7f8514161670 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.027+0000 7f851a400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 0x7f85140cb6e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f850c007e60 tx=0x7f850c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.028+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c01a070 con 0x7f8514161670 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.028+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c003b90 con 0x7f8514161670 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.028+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c004030 con 0x7f8514161670 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.028+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85140c9900 con 0x7f8514161670 2024-03-06T07:31:55.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.028+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85140c9ec0 con 0x7f8514161670 2024-03-06T07:31:55.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.029+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8514069510 con 0x7f8514161670 2024-03-06T07:31:55.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.030+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f850c01d070 con 0x7f8514161670 2024-03-06T07:31:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.032+0000 7f8511600640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84f406b4d0 0x7f84f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.032+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f850c08a4b0 con 0x7f8514161670 2024-03-06T07:31:55.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.032+0000 7f8519a00640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84f406b4d0 0x7f84f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.033+0000 7f8519a00640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84f406b4d0 0x7f84f406d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8508036000 tx=0x7f850803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.036+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f850c010070 con 0x7f8514161670 2024-03-06T07:31:55.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.235+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85140b52d0 con 0x7f8514161670 2024-03-06T07:31:55.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.237+0000 7f8511600640 1 -- 172.21.15.6:0/558501452 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f850c058c40 con 0x7f8514161670 2024-03-06T07:31:55.239 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:55.240 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:17.916537+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.239+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84f406b4d0 msgr2=0x7f84f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.239+0000 7f851c400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84f406b4d0 0x7f84f406d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8508036000 tx=0x7f850803b040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.239+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 msgr2=0x7f85140cb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.239+0000 7f851c400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 0x7f85140cb6e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f850c007e60 tx=0x7f850c00d4a0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 shutdown_connections 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f84f406b4d0 0x7f84f406d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f85140ae0a0 0x7f85140ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8514161670 0x7f85140cb6e0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 --2- 172.21.15.6:0/558501452 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f851406e240 0x7f8514154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 >> 172.21.15.6:0/558501452 conn(0x7f851406afb0 msgr2=0x7f85140cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 shutdown_connections 2024-03-06T07:31:55.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.240+0000 7f851c400640 1 -- 172.21.15.6:0/558501452 wait complete. 2024-03-06T07:31:55.260 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:31:14.222223+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:31:15.396328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:31:55.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:31:55.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.475+0000 7f44d7200640 1 Processor -- start 2024-03-06T07:31:55.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.475+0000 7f44d7200640 1 -- start start 2024-03-06T07:31:55.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.477+0000 7f44d7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f44d8145430 0x7f44d815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.477+0000 7f44d7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d815fe40 con 0x7f44d815cc90 2024-03-06T07:31:55.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.477+0000 7f44d7200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d8160120 con 0x7f44d80b2250 2024-03-06T07:31:55.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.477+0000 7f44d7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d8160400 con 0x7f44d8145430 2024-03-06T07:31:55.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.477+0000 7f44cfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44d815cc90 0x7f44d80b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44852/0 (socket says 172.21.15.6:44852) 2024-03-06T07:31:55.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.478+0000 7f44cfe00640 1 -- 172.21.15.6:0/231812188 learned_addr learned my addr 172.21.15.6:0/231812188 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.478+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079645306 0 0) 0x7f44d815fe40 con 0x7f44d815cc90 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.478+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc003520 con 0x7f44d815cc90 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.478+0000 7f44d5e00640 1 --2- 172.21.15.6:0/231812188 >> v2:172.21.15.178:3300/0 conn(0x7f44d8145430 0x7f44d815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 394158782 0 0) 0x7f44bc003520 con 0x7f44d815cc90 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 >> v2:172.21.15.178:3300/0 conn(0x7f44d8145430 msgr2=0x7f44d815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 --2- 172.21.15.6:0/231812188 >> v2:172.21.15.178:3300/0 conn(0x7f44d8145430 0x7f44d815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 >> v1:172.21.15.178:6789/0 conn(0x7f44d80b2250 legacy=0x7f44d80b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d81606e0 con 0x7f44d815cc90 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f44c0001670 con 0x7f44d815cc90 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5e00640 1 --2- 172.21.15.6:0/231812188 >> v2:172.21.15.178:3300/0 conn(0x7f44d8145430 0x7f44d815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 >> v1:172.21.15.6:6789/0 conn(0x7f44d815cc90 legacy=0x7f44d80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44bc003a00 0x7f44bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc004800 0x7f44bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc005210 0x7f44bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8160400 con 0x7f44bc005210 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc00c160 con 0x7f44bc003a00 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc00c360 con 0x7f44bc004800 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d5e00640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44bc003a00 0x7f44bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44d6800640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc005210 0x7f44bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.479+0000 7f44cfe00640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc004800 0x7f44bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.480+0000 7f44d6800640 1 -- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc004800 msgr2=0x7f44bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.480+0000 7f44d6800640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc004800 0x7f44bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.480+0000 7f44d6800640 1 -- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44bc003a00 msgr2=0x7f44bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:55.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.480+0000 7f44d6800640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44bc003a00 0x7f44bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.480+0000 7f44d6800640 1 -- 172.21.15.6:0/231812188 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc003520 con 0x7f44bc005210 2024-03-06T07:31:55.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.480+0000 7f44cfe00640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc004800 0x7f44bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.481+0000 7f44d6800640 1 -- 172.21.15.6:0/231812188 reap_dead start 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.481+0000 7f44d6800640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc005210 0x7f44bc00b9c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f44d0001e20 tx=0x7f44bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32699c581d88f072 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.481+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0001a70 con 0x7f44bc005210 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.481+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0003b70 con 0x7f44bc005210 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.481+0000 7f44d5400640 1 -- 172.21.15.6:0/231812188 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00050d0 con 0x7f44bc005210 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 -- 172.21.15.6:0/231812188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc005210 msgr2=0x7f44bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc005210 0x7f44bc00b9c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f44d0001e20 tx=0x7f44bc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 -- 172.21.15.6:0/231812188 shutdown_connections 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 --2- 172.21.15.6:0/231812188 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44bc005210 0x7f44bc00b9c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 -- 172.21.15.6:0/231812188 >> 172.21.15.6:0/231812188 conn(0x7f44d806b3a0 msgr2=0x7f44d806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 -- 172.21.15.6:0/231812188 shutdown_connections 2024-03-06T07:31:55.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.482+0000 7f44d7200640 1 -- 172.21.15.6:0/231812188 wait complete. 2024-03-06T07:31:55.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.484+0000 7f44d7200640 1 Processor -- start 2024-03-06T07:31:55.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.484+0000 7f44d7200640 1 -- start start 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.485+0000 7f44d7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc005210 0x7f44d813d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.485+0000 7f44d7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d81449a0 0x7f44d8142850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.485+0000 7f44d7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 0x7f44d81efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.485+0000 7f44d7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8145990 con 0x7f44d81441f0 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.485+0000 7f44d7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d8145810 con 0x7f44d81449a0 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.485+0000 7f44d7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80cde00 con 0x7f44bc005210 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 0x7f44d81efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 0x7f44d81efb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53086/0 (socket says 172.21.15.6:53086) 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 -- 172.21.15.6:0/1054983506 learned_addr learned my addr 172.21.15.6:0/1054983506 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:55.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d5e00640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc005210 0x7f44d813d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44cfe00640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d81449a0 0x7f44d8142850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 -- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc005210 msgr2=0x7f44d813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc005210 0x7f44d813d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 -- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d81449a0 msgr2=0x7f44d8142850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d81449a0 0x7f44d8142850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 -- 172.21.15.6:0/1054983506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d8143150 con 0x7f44d81441f0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44cfe00640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d81449a0 0x7f44d8142850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d5e00640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc005210 0x7f44d813d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.486+0000 7f44d6800640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 0x7f44d81efb20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f44d0001f10 tx=0x7f44d00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.487+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0006b50 con 0x7f44d81441f0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.487+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0006cf0 con 0x7f44d81441f0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.487+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0007230 con 0x7f44d81441f0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.487+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d8143430 con 0x7f44d81441f0 2024-03-06T07:31:55.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.487+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81439f0 con 0x7f44d81441f0 2024-03-06T07:31:55.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.489+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d804f330 con 0x7f44d81441f0 2024-03-06T07:31:55.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.489+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44d00073d0 con 0x7f44d81441f0 2024-03-06T07:31:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.490+0000 7f44ce000640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b406b4d0 0x7f44b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.490+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44d00abd60 con 0x7f44d81441f0 2024-03-06T07:31:55.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.491+0000 7f44d5e00640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b406b4d0 0x7f44b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:55.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.492+0000 7f44d5e00640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b406b4d0 0x7f44b406d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f44c8006fa0 tx=0x7f44c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:55.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.499+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44d0033030 con 0x7f44d81441f0 2024-03-06T07:31:55.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.813+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44d8072a90 con 0x7f44d81441f0 2024-03-06T07:31:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.814+0000 7f44ce000640 1 -- 172.21.15.6:0/1054983506 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44d007abc0 con 0x7f44d81441f0 2024-03-06T07:31:55.816 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.816+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b406b4d0 msgr2=0x7f44b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.816+0000 7f44d7200640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b406b4d0 0x7f44b406d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f44c8006fa0 tx=0x7f44c8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.816+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 msgr2=0x7f44d81efb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.816+0000 7f44d7200640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 0x7f44d81efb20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f44d0001f10 tx=0x7f44d00014a0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 shutdown_connections 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b406b4d0 0x7f44b406d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d81449a0 0x7f44d8142850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d81441f0 0x7f44d81efb20 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 --2- 172.21.15.6:0/1054983506 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44bc005210 0x7f44d813d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 >> 172.21.15.6:0/1054983506 conn(0x7f44d806b3a0 msgr2=0x7f44d80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:55.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 shutdown_connections 2024-03-06T07:31:55.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:55.817+0000 7f44d7200640 1 -- 172.21.15.6:0/1054983506 wait complete. 2024-03-06T07:31:55.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:31:56.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.038+0000 7feb87800640 1 Processor -- start 2024-03-06T07:31:56.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.038+0000 7feb87800640 1 -- start start 2024-03-06T07:31:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.040+0000 7feb87800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 0x7feb8006e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.040+0000 7feb87800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8006d090 con 0x7feb800ccad0 2024-03-06T07:31:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.040+0000 7feb87800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8015eb80 con 0x7feb8015e680 2024-03-06T07:31:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.040+0000 7feb87800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb80161c70 con 0x7feb8006da40 2024-03-06T07:31:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb85800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7feb800ccad0 0x7feb80162100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44866/0 (socket says 172.21.15.6:44866) 2024-03-06T07:31:56.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb85800640 1 -- 172.21.15.6:0/2129693218 learned_addr learned my addr 172.21.15.6:0/2129693218 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb84e00640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 0x7feb8006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917542043 0 0) 0x7feb8006d090 con 0x7feb800ccad0 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb64003520 con 0x7feb800ccad0 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711735292 0 0) 0x7feb64003520 con 0x7feb800ccad0 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 msgr2=0x7feb8006e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb7f400640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 0x7feb8006e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.041+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 >> v1:172.21.15.178:6789/0 conn(0x7feb8015e680 legacy=0x7feb8006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8015ee60 con 0x7feb800ccad0 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7feb78001670 con 0x7feb800ccad0 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 >> v1:172.21.15.6:6789/0 conn(0x7feb800ccad0 legacy=0x7feb80162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 0x7feb640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb64003fb0 0x7feb64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64004d40 0x7feb640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb80161c70 con 0x7feb64003fb0 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6400bb00 con 0x7feb64004d40 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6400bd00 con 0x7feb8006da40 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb7fe00640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb64003fb0 0x7feb64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.042+0000 7feb84e00640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 0x7feb640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb85800640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64004d40 0x7feb640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb7fe00640 1 -- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 msgr2=0x7feb640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb7fe00640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb8006da40 0x7feb640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb7fe00640 1 -- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64004d40 msgr2=0x7feb640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb7fe00640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64004d40 0x7feb640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb7fe00640 1 -- 172.21.15.6:0/2129693218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb64003520 con 0x7feb64003fb0 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.043+0000 7feb85800640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64004d40 0x7feb640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.044+0000 7feb7fe00640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb64003fb0 0x7feb64004800 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7feb6c001e50 tx=0x7feb64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ff636377aefb6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.044+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c001a70 con 0x7feb64003fb0 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.044+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c003b70 con 0x7feb64003fb0 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.044+0000 7feb7f400640 1 -- 172.21.15.6:0/2129693218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0050d0 con 0x7feb64003fb0 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb87800640 1 -- 172.21.15.6:0/2129693218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb64003fb0 msgr2=0x7feb64004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb87800640 1 --2- 172.21.15.6:0/2129693218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb64003fb0 0x7feb64004800 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7feb6c001e50 tx=0x7feb64005830 comp rx=0 tx=0).stop 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb85800640 1 -- 172.21.15.6:0/2129693218 reap_dead start 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb87800640 1 -- 172.21.15.6:0/2129693218 shutdown_connections 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb87800640 1 -- 172.21.15.6:0/2129693218 >> 172.21.15.6:0/2129693218 conn(0x7feb8006b3d0 msgr2=0x7feb8006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb87800640 1 -- 172.21.15.6:0/2129693218 shutdown_connections 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.045+0000 7feb87800640 1 -- 172.21.15.6:0/2129693218 wait complete. 2024-03-06T07:31:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.047+0000 7feb87800640 1 Processor -- start 2024-03-06T07:31:56.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.047+0000 7feb87800640 1 -- start start 2024-03-06T07:31:56.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb87800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64003fb0 0x7feb8015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb87800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb800ae880 0x7feb801f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb87800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 0x7feb800b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb87800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb80162810 con 0x7feb801f4630 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb87800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8015f420 con 0x7feb64003fb0 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb87800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb80162990 con 0x7feb800ae880 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb85800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 0x7feb800b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb85800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 0x7feb800b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53094/0 (socket says 172.21.15.6:53094) 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.048+0000 7feb85800640 1 -- 172.21.15.6:0/3500044459 learned_addr learned my addr 172.21.15.6:0/3500044459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb7fe00640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb800ae880 0x7feb801f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb84e00640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64003fb0 0x7feb8015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb85800640 1 -- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb800ae880 msgr2=0x7feb801f7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb85800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb800ae880 0x7feb801f7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb85800640 1 -- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64003fb0 msgr2=0x7feb8015af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb85800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64003fb0 0x7feb8015af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb85800640 1 -- 172.21.15.6:0/3500044459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb801f7b60 con 0x7feb801f4630 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb85800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 0x7feb800b3a80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7feb8006bfd0 tx=0x7feb7800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.049+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb78007230 con 0x7feb801f4630 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.050+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb780073d0 con 0x7feb801f4630 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.050+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb801f7e40 con 0x7feb801f4630 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.050+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb780047e0 con 0x7feb801f4630 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.050+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb801f7850 con 0x7feb801f4630 2024-03-06T07:31:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.051+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8004f330 con 0x7feb801f4630 2024-03-06T07:31:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.052+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb7801b020 con 0x7feb801f4630 2024-03-06T07:31:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.053+0000 7feb7cc00640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb6006b4d0 0x7feb6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.053+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb78089ce0 con 0x7feb801f4630 2024-03-06T07:31:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.054+0000 7feb84e00640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb6006b4d0 0x7feb6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.055+0000 7feb84e00640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb6006b4d0 0x7feb6006d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7feb800af210 tx=0x7feb70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.058+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb7801b310 con 0x7feb801f4630 2024-03-06T07:31:56.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.257+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feb800cee40 con 0x7feb801f4630 2024-03-06T07:31:56.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.258+0000 7feb7cc00640 1 -- 172.21.15.6:0/3500044459 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7feb78058ac0 con 0x7feb801f4630 2024-03-06T07:31:56.260 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:31:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.260+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb6006b4d0 msgr2=0x7feb6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb6006b4d0 0x7feb6006d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7feb800af210 tx=0x7feb70008040 comp rx=0 tx=0).stop 2024-03-06T07:31:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 msgr2=0x7feb800b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 0x7feb800b3a80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7feb8006bfd0 tx=0x7feb7800dc00 comp rx=0 tx=0).stop 2024-03-06T07:31:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 shutdown_connections 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb6006b4d0 0x7feb6006d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb801f4630 0x7feb800b3a80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb800ae880 0x7feb801f7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 --2- 172.21.15.6:0/3500044459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb64003fb0 0x7feb8015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 >> 172.21.15.6:0/3500044459 conn(0x7feb8006b3d0 msgr2=0x7feb800ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 shutdown_connections 2024-03-06T07:31:56.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.261+0000 7feb87800640 1 -- 172.21.15.6:0/3500044459 wait complete. 2024-03-06T07:31:56.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:31:56.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.504+0000 7f422d800640 1 Processor -- start 2024-03-06T07:31:56.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.505+0000 7f422d800640 1 -- start start 2024-03-06T07:31:56.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.506+0000 7f422d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4228145430 0x7f422815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.506+0000 7f422d800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422815fe40 con 0x7f42280b2250 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.506+0000 7f422d800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4228160120 con 0x7f422815cc90 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.506+0000 7f422d800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4228160400 con 0x7f4228145430 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.507+0000 7f422ce00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f42280b2250 0x7f42280b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44870/0 (socket says 172.21.15.6:44870) 2024-03-06T07:31:56.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.507+0000 7f422ce00640 1 -- 172.21.15.6:0/4035482398 learned_addr learned my addr 172.21.15.6:0/4035482398 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.507+0000 7f4227e00640 1 --2- 172.21.15.6:0/4035482398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4228145430 0x7f422815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.507+0000 7f4227400640 1 -- 172.21.15.6:0/4035482398 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710955674 0 0) 0x7f422815fe40 con 0x7f42280b2250 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.507+0000 7f4227400640 1 -- 172.21.15.6:0/4035482398 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420c003520 con 0x7f42280b2250 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.508+0000 7f4227400640 1 -- 172.21.15.6:0/4035482398 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 586738001 0 0) 0x7f420c003520 con 0x7f42280b2250 2024-03-06T07:31:56.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.508+0000 7f4227e00640 1 -- 172.21.15.6:0/4035482398 >> v1:172.21.15.178:6789/0 conn(0x7f422815cc90 legacy=0x7f42280b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.508+0000 7f4227e00640 1 -- 172.21.15.6:0/4035482398 >> v1:172.21.15.6:6789/0 conn(0x7f42280b2250 legacy=0x7f42280b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.508+0000 7f4227e00640 1 -- 172.21.15.6:0/4035482398 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42281606e0 con 0x7f4228145430 2024-03-06T07:31:56.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.510+0000 7f4227e00640 1 --2- 172.21.15.6:0/4035482398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4228145430 0x7f422815f810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42180097d0 tx=0x7f421802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63464ae9aaa8d212 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.510+0000 7f4227400640 1 -- 172.21.15.6:0/4035482398 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218037070 con 0x7f4228145430 2024-03-06T07:31:56.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.510+0000 7f4227400640 1 -- 172.21.15.6:0/4035482398 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4218030510 con 0x7f4228145430 2024-03-06T07:31:56.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.510+0000 7f4227400640 1 -- 172.21.15.6:0/4035482398 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42200019f0 con 0x7f4228145430 2024-03-06T07:31:56.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 -- 172.21.15.6:0/4035482398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4228145430 msgr2=0x7f422815f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 --2- 172.21.15.6:0/4035482398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4228145430 0x7f422815f810 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42180097d0 tx=0x7f421802b800 comp rx=0 tx=0).stop 2024-03-06T07:31:56.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 -- 172.21.15.6:0/4035482398 shutdown_connections 2024-03-06T07:31:56.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 --2- 172.21.15.6:0/4035482398 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4228145430 0x7f422815f810 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 -- 172.21.15.6:0/4035482398 >> 172.21.15.6:0/4035482398 conn(0x7f422806b3a0 msgr2=0x7f422806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 -- 172.21.15.6:0/4035482398 shutdown_connections 2024-03-06T07:31:56.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.511+0000 7f422d800640 1 -- 172.21.15.6:0/4035482398 wait complete. 2024-03-06T07:31:56.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.513+0000 7f422d800640 1 Processor -- start 2024-03-06T07:31:56.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.513+0000 7f422d800640 1 -- start start 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f422d800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42280b2250 0x7f422815b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f422d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 0x7f4228070c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f422d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f422815cc90 0x7f42280ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f422d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4228145990 con 0x7f4228145430 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f422d800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4228145810 con 0x7f42280b2250 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f422d800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42280cde00 con 0x7f422815cc90 2024-03-06T07:31:56.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f421fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 0x7f4228070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f421fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 0x7f4228070c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53110/0 (socket says 172.21.15.6:53110) 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.514+0000 7f421fe00640 1 -- 172.21.15.6:0/614952024 learned_addr learned my addr 172.21.15.6:0/614952024 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f422ce00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f422815cc90 0x7f42280ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f4227e00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42280b2250 0x7f422815b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f421fe00640 1 -- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f422815cc90 msgr2=0x7f42280ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f421fe00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f422815cc90 0x7f42280ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f421fe00640 1 -- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42280b2250 msgr2=0x7f422815b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f421fe00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42280b2250 0x7f422815b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f421fe00640 1 -- 172.21.15.6:0/614952024 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42280adad0 con 0x7f4228145430 2024-03-06T07:31:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f421fe00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 0x7f4228070c30 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f421000c3c0 tx=0x7f421000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42100052d0 con 0x7f4228145430 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4210005490 con 0x7f4228145430 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210007260 con 0x7f4228145430 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.515+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42280addb0 con 0x7f4228145430 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.516+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42280ae370 con 0x7f4228145430 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.517+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42280ae570 con 0x7f4228145430 2024-03-06T07:31:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.518+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4210007400 con 0x7f4228145430 2024-03-06T07:31:56.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.518+0000 7f4224c00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f420406b4d0 0x7f420406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.519+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4210089700 con 0x7f4228145430 2024-03-06T07:31:56.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.519+0000 7f4227e00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f420406b4d0 0x7f420406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.520+0000 7f4227e00640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f420406b4d0 0x7f420406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4218034000 tx=0x7f4218008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.523+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42280ae570 con 0x7f4228145430 2024-03-06T07:31:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.834+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f422804f330 con 0x7f4228145430 2024-03-06T07:31:56.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.835+0000 7f4224c00640 1 -- 172.21.15.6:0/614952024 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4210058580 con 0x7f4228145430 2024-03-06T07:31:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:31:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f420406b4d0 msgr2=0x7f420406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f420406b4d0 0x7f420406d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4218034000 tx=0x7f4218008030 comp rx=0 tx=0).stop 2024-03-06T07:31:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 msgr2=0x7f4228070c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 0x7f4228070c30 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f421000c3c0 tx=0x7f421000c880 comp rx=0 tx=0).stop 2024-03-06T07:31:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 shutdown_connections 2024-03-06T07:31:56.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f420406b4d0 0x7f420406d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f422815cc90 0x7f42280ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4228145430 0x7f4228070c30 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 --2- 172.21.15.6:0/614952024 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42280b2250 0x7f422815b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 >> 172.21.15.6:0/614952024 conn(0x7f422806b3a0 msgr2=0x7f42280cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.837+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 shutdown_connections 2024-03-06T07:31:56.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:56.838+0000 7f422d800640 1 -- 172.21.15.6:0/614952024 wait complete. 2024-03-06T07:31:56.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:57.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.103+0000 7f18f6800640 1 Processor -- start 2024-03-06T07:31:57.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.104+0000 7f18f6800640 1 -- start start 2024-03-06T07:31:57.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18f01552b0 0x7f18f015b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f6800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f015e810 0x7f18f00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f6800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f18f00cce90 0x7f18f006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f6800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f015ed10 con 0x7f18f01552b0 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f6800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f015eff0 con 0x7f18f00cce90 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f6800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f015f2d0 con 0x7f18f015e810 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.106+0000 7f18f5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18f01552b0 0x7f18f015b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f18f01552b0 0x7f18f015b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53118/0 (socket says 172.21.15.6:53118) 2024-03-06T07:31:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 -- 172.21.15.6:0/1983957688 learned_addr learned my addr 172.21.15.6:0/1983957688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 -- 172.21.15.6:0/1983957688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f015e810 msgr2=0x7f18f00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5e00640 1 --2- 172.21.15.6:0/1983957688 >> v2:172.21.15.178:3300/0 conn(0x7f18f00cce90 0x7f18f006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f4a00640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f015e810 0x7f18f00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f015e810 0x7f18f00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 -- 172.21.15.6:0/1983957688 >> v2:172.21.15.178:3300/0 conn(0x7f18f00cce90 msgr2=0x7f18f006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 --2- 172.21.15.6:0/1983957688 >> v2:172.21.15.178:3300/0 conn(0x7f18f00cce90 0x7f18f006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5400640 1 -- 172.21.15.6:0/1983957688 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f015f5b0 con 0x7f18f01552b0 2024-03-06T07:31:57.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f5e00640 1 --2- 172.21.15.6:0/1983957688 >> v2:172.21.15.178:3300/0 conn(0x7f18f00cce90 0x7f18f006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:57.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.107+0000 7f18f4a00640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f015e810 0x7f18f00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:57.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.109+0000 7f18f5400640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f01552b0 0x7f18f015b1d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f18f006dc10 tx=0x7f18e402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b019f838366eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.109+0000 7f18efe00640 1 -- 172.21.15.6:0/1983957688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e403b070 con 0x7f18f01552b0 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.109+0000 7f18efe00640 1 -- 172.21.15.6:0/1983957688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e402edf0 con 0x7f18f01552b0 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.109+0000 7f18efe00640 1 -- 172.21.15.6:0/1983957688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4036660 con 0x7f18f01552b0 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 -- 172.21.15.6:0/1983957688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f01552b0 msgr2=0x7f18f015b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f01552b0 0x7f18f015b1d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f18f006dc10 tx=0x7f18e402e040 comp rx=0 tx=0).stop 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 -- 172.21.15.6:0/1983957688 shutdown_connections 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 --2- 172.21.15.6:0/1983957688 >> v2:172.21.15.178:3300/0 conn(0x7f18f00cce90 0x7f18f006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f015e810 0x7f18f00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 --2- 172.21.15.6:0/1983957688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f01552b0 0x7f18f015b1d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 -- 172.21.15.6:0/1983957688 >> 172.21.15.6:0/1983957688 conn(0x7f18f006b3d0 msgr2=0x7f18f006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 -- 172.21.15.6:0/1983957688 shutdown_connections 2024-03-06T07:31:57.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.110+0000 7f18f6800640 1 -- 172.21.15.6:0/1983957688 wait complete. 2024-03-06T07:31:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.112+0000 7f18f6800640 1 Processor -- start 2024-03-06T07:31:57.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.112+0000 7f18f6800640 1 -- start start 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 0x7f18f014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f6800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f01552b0 0x7f18f01f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f6800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18f015e810 0x7f18f01f0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f6800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f006dd30 con 0x7f18f00cce90 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f6800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00b3760 con 0x7f18f015e810 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f6800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f006deb0 con 0x7f18f01552b0 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 0x7f18f014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 0x7f18f014e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53134/0 (socket says 172.21.15.6:53134) 2024-03-06T07:31:57.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.113+0000 7f18f5400640 1 -- 172.21.15.6:0/3407811292 learned_addr learned my addr 172.21.15.6:0/3407811292 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f5400640 1 -- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f01552b0 msgr2=0x7f18f01f0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f5400640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f01552b0 0x7f18f01f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f5400640 1 -- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18f015e810 msgr2=0x7f18f01f0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f5400640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18f015e810 0x7f18f01f0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f5400640 1 -- 172.21.15.6:0/3407811292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f01f3d00 con 0x7f18f00cce90 2024-03-06T07:31:57.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f5400640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 0x7f18f014e900 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f18e402e080 tx=0x7f18e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e403b040 con 0x7f18f00cce90 2024-03-06T07:31:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4046070 con 0x7f18f00cce90 2024-03-06T07:31:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40418b0 con 0x7f18f00cce90 2024-03-06T07:31:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.114+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f015d9b0 con 0x7f18f00cce90 2024-03-06T07:31:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.115+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f015df70 con 0x7f18f00cce90 2024-03-06T07:31:57.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.116+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f004f330 con 0x7f18f00cce90 2024-03-06T07:31:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.117+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18e4041a50 con 0x7f18f00cce90 2024-03-06T07:31:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.117+0000 7f18ed600640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.118+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18e40ac9d0 con 0x7f18f00cce90 2024-03-06T07:31:57.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.118+0000 7f18f4a00640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.119+0000 7f18f4a00640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18d006b4d0 0x7f18d006d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f18dc006fd0 tx=0x7f18dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.122+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18e40353a0 con 0x7f18f00cce90 2024-03-06T07:31:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.362+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18f00cf4a0 con 0x7f18f00cce90 2024-03-06T07:31:57.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.363+0000 7f18ed600640 1 -- 172.21.15.6:0/3407811292 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f18e4081020 con 0x7f18f00cce90 2024-03-06T07:31:57.365 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:57.365 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:57.366 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:57.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.365+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18d006b4d0 msgr2=0x7f18d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.365+0000 7f18f6800640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18d006b4d0 0x7f18d006d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f18dc006fd0 tx=0x7f18dc008040 comp rx=0 tx=0).stop 2024-03-06T07:31:57.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.365+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 msgr2=0x7f18f014e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.365+0000 7f18f6800640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 0x7f18f014e900 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f18e402e080 tx=0x7f18e4004730 comp rx=0 tx=0).stop 2024-03-06T07:31:57.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 shutdown_connections 2024-03-06T07:31:57.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f18d006b4d0 0x7f18d006d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f18f015e810 0x7f18f01f0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f18f01552b0 0x7f18f01f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 --2- 172.21.15.6:0/3407811292 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f18f00cce90 0x7f18f014e900 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 >> 172.21.15.6:0/3407811292 conn(0x7f18f006b3d0 msgr2=0x7f18f0070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 shutdown_connections 2024-03-06T07:31:57.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.366+0000 7f18f6800640 1 -- 172.21.15.6:0/3407811292 wait complete. 2024-03-06T07:31:57.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:57.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.595+0000 7f2e93800640 1 Processor -- start 2024-03-06T07:31:57.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.595+0000 7f2e93800640 1 -- start start 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e93800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af1a0 0x7f2e8c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e93800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e8c0af580 0x7f2e8c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e93800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c15cc40 con 0x7f2e8c0cdc20 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e93800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e8c15cf20 con 0x7f2e8c0af580 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e93800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c15d200 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e8be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e8c0af580 0x7f2e8c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e8be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2e8c0af580 0x7f2e8c0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53142/0 (socket says 172.21.15.6:53142) 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e8be00640 1 -- 172.21.15.6:0/3625602109 learned_addr learned my addr 172.21.15.6:0/3625602109 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e90e00640 1 --2- 172.21.15.6:0/3625602109 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af1a0 0x7f2e8c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e8be00640 1 -- 172.21.15.6:0/3625602109 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af1a0 msgr2=0x7f2e8c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.598+0000 7f2e8be00640 1 --2- 172.21.15.6:0/3625602109 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af1a0 0x7f2e8c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.599+0000 7f2e8be00640 1 -- 172.21.15.6:0/3625602109 >> v1:172.21.15.178:6789/0 conn(0x7f2e8c0cdc20 legacy=0x7f2e8c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.599+0000 7f2e8be00640 1 -- 172.21.15.6:0/3625602109 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c15d4e0 con 0x7f2e8c0af580 2024-03-06T07:31:57.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.600+0000 7f2e8be00640 1 --2- 172.21.15.6:0/3625602109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af580 0x7f2e8c0b4c20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009870 tx=0x7f2e7802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864d974086d81cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.600+0000 7f2e8b400640 1 -- 172.21.15.6:0/3625602109 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e780317c0 con 0x7f2e8c0af580 2024-03-06T07:31:57.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.600+0000 7f2e8b400640 1 -- 172.21.15.6:0/3625602109 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78037070 con 0x7f2e8c0af580 2024-03-06T07:31:57.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.600+0000 7f2e8b400640 1 -- 172.21.15.6:0/3625602109 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78031b80 con 0x7f2e8c0af580 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 -- 172.21.15.6:0/3625602109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af580 msgr2=0x7f2e8c0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 --2- 172.21.15.6:0/3625602109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af580 0x7f2e8c0b4c20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009870 tx=0x7f2e7802b860 comp rx=0 tx=0).stop 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 -- 172.21.15.6:0/3625602109 shutdown_connections 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 --2- 172.21.15.6:0/3625602109 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af580 0x7f2e8c0b4c20 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 --2- 172.21.15.6:0/3625602109 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af1a0 0x7f2e8c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 -- 172.21.15.6:0/3625602109 >> 172.21.15.6:0/3625602109 conn(0x7f2e8c06afb0 msgr2=0x7f2e8c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 -- 172.21.15.6:0/3625602109 shutdown_connections 2024-03-06T07:31:57.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.601+0000 7f2e93800640 1 -- 172.21.15.6:0/3625602109 wait complete. 2024-03-06T07:31:57.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.603+0000 7f2e93800640 1 Processor -- start 2024-03-06T07:31:57.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.603+0000 7f2e93800640 1 -- start start 2024-03-06T07:31:57.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e93800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 0x7f2e8c150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e93800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af580 0x7f2e8c0c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e93800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e8c0cdc20 0x7f2e8c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e93800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0b3660 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e93800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0b34e0 con 0x7f2e8c0cdc20 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e93800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0b37e0 con 0x7f2e8c0af580 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 0x7f2e8c150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 0x7f2e8c150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53156/0 (socket says 172.21.15.6:53156) 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 -- 172.21.15.6:0/3601532807 learned_addr learned my addr 172.21.15.6:0/3601532807 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e8be00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af580 0x7f2e8c0c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e91800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e8c0cdc20 0x7f2e8c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 -- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af580 msgr2=0x7f2e8c0c4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af580 0x7f2e8c0c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 -- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e8c0cdc20 msgr2=0x7f2e8c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e8c0cdc20 0x7f2e8c15e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e90e00640 1 -- 172.21.15.6:0/3601532807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0c5640 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.604+0000 7f2e91800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e8c0cdc20 0x7f2e8c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:57.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.605+0000 7f2e90e00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 0x7f2e8c150120 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c7f0 tx=0x7f2e8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.605+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80015070 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.605+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e80004240 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.605+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80007620 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.605+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c0c5920 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.605+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8c0c5ee0 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.606+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8c04f3a0 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.608+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e800079d0 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.608+0000 7f2e88c00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.609+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e8008a680 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.609+0000 7f2e8be00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.610+0000 7f2e8be00640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2e78036000 tx=0x7f2e78008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.613+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e80059500 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.814+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e8c0acf50 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.816+0000 7f2e88c00640 1 -- 172.21.15.6:0/3601532807 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f2e80058e00 con 0x7f2e8c0af1a0 2024-03-06T07:31:57.818 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:57.818 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:17.916537+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e6c06b4d0 msgr2=0x7f2e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2e78036000 tx=0x7f2e78008030 comp rx=0 tx=0).stop 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 msgr2=0x7f2e8c150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 0x7f2e8c150120 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c7f0 tx=0x7f2e8000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 shutdown_connections 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2e8c0cdc20 0x7f2e8c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2e8c0af580 0x7f2e8c0c4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 --2- 172.21.15.6:0/3601532807 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2e8c0af1a0 0x7f2e8c150120 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.818+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 >> 172.21.15.6:0/3601532807 conn(0x7f2e8c06afb0 msgr2=0x7f2e8c0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.819+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 shutdown_connections 2024-03-06T07:31:57.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:57.819+0000 7f2e93800640 1 -- 172.21.15.6:0/3601532807 wait complete. 2024-03-06T07:31:57.839 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:31:57.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.055+0000 7f9b4c800640 1 Processor -- start 2024-03-06T07:31:58.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.055+0000 7f9b4c800640 1 -- start start 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4c800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9b440af1a0 0x7f9b4415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4c800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b440cdc20 0x7f9b440b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4c800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4415cc40 con 0x7f9b440af580 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4c800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4415cf20 con 0x7f9b440cdc20 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4c800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4415d200 con 0x7f9b440af1a0 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b440cdc20 0x7f9b440b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4a800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9b440cdc20 0x7f9b440b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53168/0 (socket says 172.21.15.6:53168) 2024-03-06T07:31:58.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.057+0000 7f9b4a800640 1 -- 172.21.15.6:0/156169872 learned_addr learned my addr 172.21.15.6:0/156169872 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.058+0000 7f9b4a800640 1 -- 172.21.15.6:0/156169872 >> v2:172.21.15.178:3300/0 conn(0x7f9b440af1a0 msgr2=0x7f9b4415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.058+0000 7f9b4a800640 1 --2- 172.21.15.6:0/156169872 >> v2:172.21.15.178:3300/0 conn(0x7f9b440af1a0 0x7f9b4415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.058+0000 7f9b4a800640 1 -- 172.21.15.6:0/156169872 >> v1:172.21.15.178:6789/0 conn(0x7f9b440af580 legacy=0x7f9b440b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.058+0000 7f9b4a800640 1 -- 172.21.15.6:0/156169872 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4415d4e0 con 0x7f9b440cdc20 2024-03-06T07:31:58.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.059+0000 7f9b4a800640 1 --2- 172.21.15.6:0/156169872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440cdc20 0x7f9b440b2dd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f9b44074960 tx=0x7f9b4002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a8ac54c0f37e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.059+0000 7f9b48a00640 1 -- 172.21.15.6:0/156169872 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4002c9f0 con 0x7f9b440cdc20 2024-03-06T07:31:58.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.059+0000 7f9b48a00640 1 -- 172.21.15.6:0/156169872 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4002cb90 con 0x7f9b440cdc20 2024-03-06T07:31:58.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.059+0000 7f9b48a00640 1 -- 172.21.15.6:0/156169872 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400343f0 con 0x7f9b440cdc20 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 -- 172.21.15.6:0/156169872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440cdc20 msgr2=0x7f9b440b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 --2- 172.21.15.6:0/156169872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440cdc20 0x7f9b440b2dd0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f9b44074960 tx=0x7f9b4002be50 comp rx=0 tx=0).stop 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 -- 172.21.15.6:0/156169872 shutdown_connections 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 --2- 172.21.15.6:0/156169872 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440cdc20 0x7f9b440b2dd0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 --2- 172.21.15.6:0/156169872 >> v2:172.21.15.178:3300/0 conn(0x7f9b440af1a0 0x7f9b4415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 -- 172.21.15.6:0/156169872 >> 172.21.15.6:0/156169872 conn(0x7f9b4406afb0 msgr2=0x7f9b4406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 -- 172.21.15.6:0/156169872 shutdown_connections 2024-03-06T07:31:58.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.060+0000 7f9b4c800640 1 -- 172.21.15.6:0/156169872 wait complete. 2024-03-06T07:31:58.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.062+0000 7f9b4c800640 1 Processor -- start 2024-03-06T07:31:58.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.062+0000 7f9b4c800640 1 -- start start 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 0x7f9b441f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4c800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b440af580 0x7f9b4409bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4c800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b440cdc20 0x7f9b440c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440b36a0 con 0x7f9b440af1a0 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4c800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440b3520 con 0x7f9b440af580 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4c800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440b3820 con 0x7f9b440cdc20 2024-03-06T07:31:58.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 0x7f9b441f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 0x7f9b441f2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53184/0 (socket says 172.21.15.6:53184) 2024-03-06T07:31:58.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b49e00640 1 -- 172.21.15.6:0/1490593703 learned_addr learned my addr 172.21.15.6:0/1490593703 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b49400640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b440af580 0x7f9b4409bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.063+0000 7f9b4a800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b440cdc20 0x7f9b440c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49e00640 1 -- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b440cdc20 msgr2=0x7f9b440c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49e00640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b440cdc20 0x7f9b440c5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49e00640 1 -- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b440af580 msgr2=0x7f9b4409bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49e00640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b440af580 0x7f9b4409bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49e00640 1 -- 172.21.15.6:0/1490593703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440c5e80 con 0x7f9b440af1a0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49400640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b440af580 0x7f9b4409bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b49e00640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 0x7f9b441f2680 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b4a0 tx=0x7f9b3800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38014070 con 0x7f9b440af1a0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b38010610 con 0x7f9b440af1a0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38010a00 con 0x7f9b440af1a0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b440c6400 con 0x7f9b440af1a0 2024-03-06T07:31:58.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.064+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b441f1e80 con 0x7f9b440af1a0 2024-03-06T07:31:58.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.066+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b44069510 con 0x7f9b440af1a0 2024-03-06T07:31:58.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.067+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b38017070 con 0x7f9b440af1a0 2024-03-06T07:31:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.067+0000 7f9b3e000640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b2806b4d0 0x7f9b2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.068+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b38089b90 con 0x7f9b440af1a0 2024-03-06T07:31:58.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.068+0000 7f9b49400640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b2806b4d0 0x7f9b2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.069+0000 7f9b49400640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b2806b4d0 0x7f9b2806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b34007fd0 tx=0x7f9b34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.072+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b38058aa0 con 0x7f9b440af1a0 2024-03-06T07:31:58.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.310+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b4409c3e0 con 0x7f9b440af1a0 2024-03-06T07:31:58.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.311+0000 7f9b3e000640 1 -- 172.21.15.6:0/1490593703 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f9b380583a0 con 0x7f9b440af1a0 2024-03-06T07:31:58.313 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:58.313 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:58.313 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.313+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b2806b4d0 msgr2=0x7f9b2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.313+0000 7f9b4c800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b2806b4d0 0x7f9b2806d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9b34007fd0 tx=0x7f9b34009040 comp rx=0 tx=0).stop 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.313+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 msgr2=0x7f9b441f2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.313+0000 7f9b4c800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 0x7f9b441f2680 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b4a0 tx=0x7f9b3800b960 comp rx=0 tx=0).stop 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 shutdown_connections 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9b2806b4d0 0x7f9b2806d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9b440af580 0x7f9b4409bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9b440cdc20 0x7f9b440c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 --2- 172.21.15.6:0/1490593703 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9b440af1a0 0x7f9b441f2680 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 >> 172.21.15.6:0/1490593703 conn(0x7f9b4406afb0 msgr2=0x7f9b440cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 shutdown_connections 2024-03-06T07:31:58.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.314+0000 7f9b4c800640 1 -- 172.21.15.6:0/1490593703 wait complete. 2024-03-06T07:31:58.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:58.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.542+0000 7fdc4f200640 1 Processor -- start 2024-03-06T07:31:58.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.542+0000 7fdc4f200640 1 -- start start 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.543+0000 7fdc4f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fdc500d4a10 0x7fdc500d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.543+0000 7fdc4f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc500d0c80 0x7fdc500b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.543+0000 7fdc4f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc500b3710 con 0x7fdc5015e0b0 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.543+0000 7fdc4f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc500d10f0 con 0x7fdc500d0c80 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.543+0000 7fdc4f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc500b09e0 con 0x7fdc500d4a10 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc500d0c80 0x7fdc500b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdc500d0c80 0x7fdc500b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53196/0 (socket says 172.21.15.6:53196) 2024-03-06T07:31:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 -- 172.21.15.6:0/1795083794 learned_addr learned my addr 172.21.15.6:0/1795083794 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc45400640 1 --2- 172.21.15.6:0/1795083794 >> v2:172.21.15.178:3300/0 conn(0x7fdc500d4a10 0x7fdc500d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 -- 172.21.15.6:0/1795083794 >> v2:172.21.15.178:3300/0 conn(0x7fdc500d4a10 msgr2=0x7fdc500d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 --2- 172.21.15.6:0/1795083794 >> v2:172.21.15.178:3300/0 conn(0x7fdc500d4a10 0x7fdc500d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 -- 172.21.15.6:0/1795083794 >> v1:172.21.15.178:6789/0 conn(0x7fdc5015e0b0 legacy=0x7fdc500cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:58.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.544+0000 7fdc4e800640 1 -- 172.21.15.6:0/1795083794 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc500b39f0 con 0x7fdc500d0c80 2024-03-06T07:31:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.545+0000 7fdc4e800640 1 --2- 172.21.15.6:0/1795083794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d0c80 0x7fdc500b44d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdc5008d6d0 tx=0x7fdc4802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e5365dff172ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4d400640 1 -- 172.21.15.6:0/1795083794 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4802ca40 con 0x7fdc500d0c80 2024-03-06T07:31:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4d400640 1 -- 172.21.15.6:0/1795083794 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4802cbe0 con 0x7fdc500d0c80 2024-03-06T07:31:58.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4d400640 1 -- 172.21.15.6:0/1795083794 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc480343f0 con 0x7fdc500d0c80 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4f200640 1 -- 172.21.15.6:0/1795083794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d0c80 msgr2=0x7fdc500b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4f200640 1 --2- 172.21.15.6:0/1795083794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d0c80 0x7fdc500b44d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdc5008d6d0 tx=0x7fdc4802bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4f200640 1 -- 172.21.15.6:0/1795083794 shutdown_connections 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4f200640 1 --2- 172.21.15.6:0/1795083794 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d0c80 0x7fdc500b44d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4f200640 1 --2- 172.21.15.6:0/1795083794 >> v2:172.21.15.178:3300/0 conn(0x7fdc500d4a10 0x7fdc500d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.546+0000 7fdc4f200640 1 -- 172.21.15.6:0/1795083794 >> 172.21.15.6:0/1795083794 conn(0x7fdc500d3d30 msgr2=0x7fdc500d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.547+0000 7fdc4f200640 1 -- 172.21.15.6:0/1795083794 shutdown_connections 2024-03-06T07:31:58.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.547+0000 7fdc4f200640 1 -- 172.21.15.6:0/1795083794 wait complete. 2024-03-06T07:31:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.548+0000 7fdc4f200640 1 Processor -- start 2024-03-06T07:31:58.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.548+0000 7fdc4f200640 1 -- start start 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.549+0000 7fdc4f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc500d0c80 0x7fdc500b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.549+0000 7fdc4f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 0x7fdc50150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.549+0000 7fdc4f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc5015e0b0 0x7fdc500b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.549+0000 7fdc4f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc500b3fb0 con 0x7fdc500d4a10 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.549+0000 7fdc4f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc500cfa20 con 0x7fdc500d0c80 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.549+0000 7fdc4f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5015e490 con 0x7fdc5015e0b0 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 0x7fdc50150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 0x7fdc50150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53210/0 (socket says 172.21.15.6:53210) 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 -- 172.21.15.6:0/570450443 learned_addr learned my addr 172.21.15.6:0/570450443 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 -- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc5015e0b0 msgr2=0x7fdc500b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc5015e0b0 0x7fdc500b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 -- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc500d0c80 msgr2=0x7fdc500b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc4de00640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc500d0c80 0x7fdc500b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc500d0c80 0x7fdc500b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 -- 172.21.15.6:0/570450443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc500b6b30 con 0x7fdc500d4a10 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc4de00640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc500d0c80 0x7fdc500b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.550+0000 7fdc45400640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 0x7fdc50150190 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c820 tx=0x7fdc3800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.551+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38015070 con 0x7fdc500d4a10 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.551+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc38004220 con 0x7fdc500d4a10 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.551+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc500b6e10 con 0x7fdc500d4a10 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.551+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3800ce10 con 0x7fdc500d4a10 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.551+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc500b73d0 con 0x7fdc500d4a10 2024-03-06T07:31:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.552+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5004f330 con 0x7fdc500d4a10 2024-03-06T07:31:58.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.553+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc38007830 con 0x7fdc500d4a10 2024-03-06T07:31:58.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.554+0000 7fdc46a00640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.554+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc3808a450 con 0x7fdc500d4a10 2024-03-06T07:31:58.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.554+0000 7fdc4de00640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.555+0000 7fdc4de00640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdc40003eb0 tx=0x7fdc40009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.558+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc38059230 con 0x7fdc500d4a10 2024-03-06T07:31:58.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.798+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc500cf1a0 con 0x7fdc500d4a10 2024-03-06T07:31:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.799+0000 7fdc46a00640 1 -- 172.21.15.6:0/570450443 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fdc38058b30 con 0x7fdc500d4a10 2024-03-06T07:31:58.802 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:58.802 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:31:16.361554+0000","modified":"2024-03-06T07:31:19.436145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6402},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6402":{"gid":6402,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/4173228924","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":4173228924},{"type":"v1","addr":"172.21.15.6:6835","nonce":4173228924}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6402,"qdb_cluster":[6402]},"id":5}]} 2024-03-06T07:31:58.802 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 45 2024-03-06T07:31:58.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc2c06b4d0 msgr2=0x7fdc2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdc40003eb0 tx=0x7fdc40009040 comp rx=0 tx=0).stop 2024-03-06T07:31:58.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 msgr2=0x7fdc50150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:58.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 0x7fdc50150190 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c820 tx=0x7fdc3800cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 shutdown_connections 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdc5015e0b0 0x7fdc500b86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdc500d4a10 0x7fdc50150190 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 --2- 172.21.15.6:0/570450443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdc500d0c80 0x7fdc500b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.802+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 >> 172.21.15.6:0/570450443 conn(0x7fdc500d3d30 msgr2=0x7fdc500b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.803+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 shutdown_connections 2024-03-06T07:31:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:58.803+0000 7fdc4f200640 1 -- 172.21.15.6:0/570450443 wait complete. 2024-03-06T07:31:58.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:31:59.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.021+0000 7f5f15200640 1 Processor -- start 2024-03-06T07:31:59.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.021+0000 7f5f15200640 1 -- start start 2024-03-06T07:31:59.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f15200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 0x7f5f100d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f15200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5f100d0c80 0x7f5f100b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f15200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f100b3710 con 0x7f5f1015e0b0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f15200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f100b09e0 con 0x7f5f100d0c80 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f15200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100cfa20 con 0x7f5f100d4a10 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f0f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5f1015e0b0 0x7f5f100cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44886/0 (socket says 172.21.15.6:44886) 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.023+0000 7f5f0f400640 1 -- 172.21.15.6:0/848732444 learned_addr learned my addr 172.21.15.6:0/848732444 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0ea00640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 0x7f5f100d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966686075 0 0) 0x7f5f100b3710 con 0x7f5f1015e0b0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/848732444 >> v2:172.21.15.178:3300/0 conn(0x7f5f100d0c80 0x7f5f100b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef4003520 con 0x7f5f1015e0b0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278669492 0 0) 0x7f5ef4003520 con 0x7f5f1015e0b0 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 msgr2=0x7f5f100d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 0x7f5f100d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 >> v2:172.21.15.178:3300/0 conn(0x7f5f100d0c80 msgr2=0x7f5f100b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 --2- 172.21.15.6:0/848732444 >> v2:172.21.15.178:3300/0 conn(0x7f5f100d0c80 0x7f5f100b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f100b39f0 con 0x7f5f1015e0b0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5f00001670 con 0x7f5f1015e0b0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 >> v1:172.21.15.6:6789/0 conn(0x7f5f1015e0b0 legacy=0x7f5f100cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 0x7f5ef40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4003f80 0x7f5ef4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef4004d40 0x7f5ef40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100cfa20 con 0x7f5ef4004d40 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef400b9f0 con 0x7f5ef4003f80 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef400bbf0 con 0x7f5f100d4a10 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.024+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/848732444 >> v2:172.21.15.178:3300/0 conn(0x7f5f100d0c80 0x7f5f100b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef4004d40 0x7f5ef40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0f400640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 0x7f5ef40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0ea00640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4003f80 0x7f5ef4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0fe00640 1 -- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 msgr2=0x7f5ef40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f100d4a10 0x7f5ef40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0fe00640 1 -- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4003f80 msgr2=0x7f5ef4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4003f80 0x7f5ef4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.025+0000 7f5f0fe00640 1 -- 172.21.15.6:0/848732444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4003520 con 0x7f5ef4004d40 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.026+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef4004d40 0x7f5ef40051c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5f080073e0 tx=0x7f5ef4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f9798f2552ac0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.026+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08039070 con 0x7f5ef4004d40 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0802cdc0 con 0x7f5ef4004d40 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f0e000640 1 -- 172.21.15.6:0/848732444 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08009c70 con 0x7f5ef4004d40 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f15200640 1 -- 172.21.15.6:0/848732444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef4004d40 msgr2=0x7f5ef40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f15200640 1 --2- 172.21.15.6:0/848732444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ef4004d40 0x7f5ef40051c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5f080073e0 tx=0x7f5ef4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f0fe00640 1 -- 172.21.15.6:0/848732444 reap_dead start 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f15200640 1 -- 172.21.15.6:0/848732444 shutdown_connections 2024-03-06T07:31:59.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f15200640 1 -- 172.21.15.6:0/848732444 >> 172.21.15.6:0/848732444 conn(0x7f5f100d3d30 msgr2=0x7f5f100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f15200640 1 -- 172.21.15.6:0/848732444 shutdown_connections 2024-03-06T07:31:59.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.027+0000 7f5f15200640 1 -- 172.21.15.6:0/848732444 wait complete. 2024-03-06T07:31:59.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.029+0000 7f5f15200640 1 Processor -- start 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.029+0000 7f5f15200640 1 -- start start 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f15200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4004d40 0x7f5f101515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f15200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f10151d70 0x7f5f101f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f15200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 0x7f5f1006c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f15200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1015e490 con 0x7f5f101f3f70 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f15200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100b3fb0 con 0x7f5ef4004d40 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f15200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f100cfc60 con 0x7f5f10151d70 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 0x7f5f1006c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 0x7f5f1006c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53234/0 (socket says 172.21.15.6:53234) 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f0fe00640 1 -- 172.21.15.6:0/2780534678 learned_addr learned my addr 172.21.15.6:0/2780534678 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f0ea00640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f10151d70 0x7f5f101f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.030+0000 7f5f0f400640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4004d40 0x7f5f101515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0fe00640 1 -- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f10151d70 msgr2=0x7f5f101f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f10151d70 0x7f5f101f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0fe00640 1 -- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4004d40 msgr2=0x7f5f101515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4004d40 0x7f5f101515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0fe00640 1 -- 172.21.15.6:0/2780534678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10151b30 con 0x7f5f101f3f70 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0ea00640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f10151d70 0x7f5f101f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0f400640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4004d40 0x7f5f101515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f0fe00640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 0x7f5f1006c070 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5f080073e0 tx=0x7f5f08004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08042020 con 0x7f5f101f3f70 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0803a040 con 0x7f5f101f3f70 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1015fc40 con 0x7f5f101f3f70 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f08039070 con 0x7f5f101f3f70 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.031+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f10160200 con 0x7f5f101f3f70 2024-03-06T07:31:59.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.032+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1004f330 con 0x7f5f101f3f70 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.033+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f08045020 con 0x7f5f101f3f70 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.035+0000 7f5f07400640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ef006b4d0 0x7f5ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.035+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f080abe50 con 0x7f5f101f3f70 2024-03-06T07:31:59.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.035+0000 7f5f0f400640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ef006b4d0 0x7f5ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.036+0000 7f5f0f400640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ef006b4d0 0x7f5ef006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5f00000c00 tx=0x7f5f00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.039+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f0807ade0 con 0x7f5f101f3f70 2024-03-06T07:31:59.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.235+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f100b2a10 con 0x7f5f101f3f70 2024-03-06T07:31:59.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.237+0000 7f5f07400640 1 -- 172.21.15.6:0/2780534678 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f5f0807a6e0 con 0x7f5f101f3f70 2024-03-06T07:31:59.240 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:31:59.240 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:31:17.916537+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:31:14.222223+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:31:15.396328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:31:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ef006b4d0 msgr2=0x7f5ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ef006b4d0 0x7f5ef006d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5f00000c00 tx=0x7f5f00009040 comp rx=0 tx=0).stop 2024-03-06T07:31:59.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 msgr2=0x7f5f1006c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 0x7f5f1006c070 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5f080073e0 tx=0x7f5f08004ab0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 shutdown_connections 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ef006b4d0 0x7f5ef006d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f101f3f70 0x7f5f1006c070 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f10151d70 0x7f5f101f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 --2- 172.21.15.6:0/2780534678 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5ef4004d40 0x7f5f101515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 >> 172.21.15.6:0/2780534678 conn(0x7f5f100d3d30 msgr2=0x7f5f100b8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 shutdown_connections 2024-03-06T07:31:59.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.240+0000 7f5f15200640 1 -- 172.21.15.6:0/2780534678 wait complete. 2024-03-06T07:31:59.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:31:59.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.460+0000 7f2f1f000640 1 Processor -- start 2024-03-06T07:31:59.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.461+0000 7f2f1f000640 1 -- start start 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.463+0000 7f2f1f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 0x7f2f180d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.463+0000 7f2f1f000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f18147bc0 con 0x7f2f18162420 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.463+0000 7f2f1f000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f180b3d70 con 0x7f2f181477e0 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.463+0000 7f2f1f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18161f90 con 0x7f2f1806b6c0 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.463+0000 7f2f1d200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2f18162420 0x7f2f1806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44896/0 (socket says 172.21.15.6:44896) 2024-03-06T07:31:59.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.463+0000 7f2f1d200640 1 -- 172.21.15.6:0/763047313 learned_addr learned my addr 172.21.15.6:0/763047313 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:31:59.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f1e600640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 0x7f2f180d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864679292 0 0) 0x7f2f18147bc0 con 0x7f2f18162420 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f00003520 con 0x7f2f18162420 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459127787 0 0) 0x7f2f00003520 con 0x7f2f18162420 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 msgr2=0x7f2f180d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 0x7f2f180d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 >> v1:172.21.15.178:6789/0 conn(0x7f2f181477e0 legacy=0x7f2f1806ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f180b4050 con 0x7f2f18162420 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2f04001670 con 0x7f2f18162420 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 >> v1:172.21.15.6:6789/0 conn(0x7f2f18162420 legacy=0x7f2f1806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f000039e0 0x7f2f00003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 0x7f2f00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f00004d40 0x7f2f000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18161f90 con 0x7f2f000039e0 2024-03-06T07:31:59.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0000bb20 con 0x7f2f00004d40 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.464+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0000bd20 con 0x7f2f1806b6c0 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f000039e0 0x7f2f00003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1d200640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 0x7f2f00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1dc00640 1 -- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 msgr2=0x7f2f00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1e600640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f00004d40 0x7f2f000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f1806b6c0 0x7f2f00004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1dc00640 1 -- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f00004d40 msgr2=0x7f2f000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f00004d40 0x7f2f000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1dc00640 1 -- 172.21.15.6:0/763047313 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f00003520 con 0x7f2f000039e0 2024-03-06T07:31:59.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.465+0000 7f2f1e600640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f00004d40 0x7f2f000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.467+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f000039e0 0x7f2f00003e00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c001750 tx=0x7f2f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3b2c103aa89480 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.467+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c001370 con 0x7f2f000039e0 2024-03-06T07:31:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.467+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c0052d0 con 0x7f2f000039e0 2024-03-06T07:31:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.467+0000 7f2f17e00640 1 -- 172.21.15.6:0/763047313 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c003830 con 0x7f2f000039e0 2024-03-06T07:31:59.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1f000640 1 -- 172.21.15.6:0/763047313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f000039e0 msgr2=0x7f2f00003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1f000640 1 --2- 172.21.15.6:0/763047313 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f000039e0 0x7f2f00003e00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c001750 tx=0x7f2f00005830 comp rx=0 tx=0).stop 2024-03-06T07:31:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1e600640 1 -- 172.21.15.6:0/763047313 reap_dead start 2024-03-06T07:31:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1f000640 1 -- 172.21.15.6:0/763047313 shutdown_connections 2024-03-06T07:31:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1f000640 1 -- 172.21.15.6:0/763047313 >> 172.21.15.6:0/763047313 conn(0x7f2f1806c0c0 msgr2=0x7f2f1806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1f000640 1 -- 172.21.15.6:0/763047313 shutdown_connections 2024-03-06T07:31:59.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.468+0000 7f2f1f000640 1 -- 172.21.15.6:0/763047313 wait complete. 2024-03-06T07:31:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.470+0000 7f2f1f000640 1 Processor -- start 2024-03-06T07:31:59.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.470+0000 7f2f1f000640 1 -- start start 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f000039e0 0x7f2f18142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1f000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f181f4c90 0x7f2f1813e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 0x7f2f1813e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f181628a0 con 0x7f2f181f5110 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1f000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18161c20 con 0x7f2f181f4c90 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1806f8f0 con 0x7f2f000039e0 2024-03-06T07:31:59.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 0x7f2f1813e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 0x7f2f1813e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53246/0 (socket says 172.21.15.6:53246) 2024-03-06T07:31:59.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1e600640 1 -- 172.21.15.6:0/2696782978 learned_addr learned my addr 172.21.15.6:0/2696782978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.471+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f000039e0 0x7f2f18142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1e600640 1 -- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f000039e0 msgr2=0x7f2f18142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1e600640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f000039e0 0x7f2f18142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1e600640 1 -- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f181f4c90 msgr2=0x7f2f1813e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1e600640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f181f4c90 0x7f2f1813e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1e600640 1 -- 172.21.15.6:0/2696782978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1815e560 con 0x7f2f181f5110 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1e600640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 0x7f2f1813e620 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2f1806bfe0 tx=0x7f2f1000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1001c070 con 0x7f2f181f5110 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1000fe70 con 0x7f2f181f5110 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1815e800 con 0x7f2f181f5110 2024-03-06T07:31:59.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f10017620 con 0x7f2f181f5110 2024-03-06T07:31:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.472+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1815edc0 con 0x7f2f181f5110 2024-03-06T07:31:59.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.474+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1804f330 con 0x7f2f181f5110 2024-03-06T07:31:59.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.475+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f1001f070 con 0x7f2f181f5110 2024-03-06T07:31:59.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.477+0000 7f2f15600640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ef806b500 0x7f2ef806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.477+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f1008a080 con 0x7f2f181f5110 2024-03-06T07:31:59.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.477+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ef806b500 0x7f2ef806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.479+0000 7f2f1dc00640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ef806b500 0x7f2ef806d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c033000 tx=0x7f2f0c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.482+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f10012070 con 0x7f2f181f5110 2024-03-06T07:31:59.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.714+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2f180724b0 con 0x7f2f181f5110 2024-03-06T07:31:59.718 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:31:59.716+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:31:59.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.719+0000 7f2f15600640 1 -- 172.21.15.6:0/2696782978 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2f100588a0 con 0x7f2f181f5110 2024-03-06T07:31:59.721 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:31:59.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.722+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ef806b500 msgr2=0x7f2ef806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.722+0000 7f2f1f000640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ef806b500 0x7f2ef806d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c033000 tx=0x7f2f0c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:59.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.722+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 msgr2=0x7f2f1813e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.722+0000 7f2f1f000640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 0x7f2f1813e620 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2f1806bfe0 tx=0x7f2f1000f4a0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 shutdown_connections 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ef806b500 0x7f2ef806d9c0 secure :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c033000 tx=0x7f2f0c009030 comp rx=0 tx=0).stop 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f181f4c90 0x7f2f1813e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f181f5110 0x7f2f1813e620 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 --2- 172.21.15.6:0/2696782978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f000039e0 0x7f2f18142bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 >> 172.21.15.6:0/2696782978 conn(0x7f2f1806c0c0 msgr2=0x7f2f18072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 shutdown_connections 2024-03-06T07:31:59.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.723+0000 7f2f1f000640 1 -- 172.21.15.6:0/2696782978 wait complete. 2024-03-06T07:31:59.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:31:59.807 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:31:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.930+0000 7ff69e000640 1 Processor -- start 2024-03-06T07:31:59.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.930+0000 7ff69e000640 1 -- start start 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.931+0000 7ff69e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.931+0000 7ff69e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff69815e680 0x7ff69806c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.931+0000 7ff69e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6980ccad0 0x7ff6980ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.931+0000 7ff69e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff698161c70 con 0x7ff6980ccad0 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.931+0000 7ff69e000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69815eb80 con 0x7ff69815e680 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.931+0000 7ff69e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69815ee60 con 0x7ff69806da40 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6980ccad0 0x7ff6980ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff6980ccad0 0x7ff6980ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53260/0 (socket says 172.21.15.6:53260) 2024-03-06T07:31:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 -- 172.21.15.6:0/3279667089 learned_addr learned my addr 172.21.15.6:0/3279667089 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697400640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff68fe00640 1 --2- 172.21.15.6:0/3279667089 >> v2:172.21.15.178:3300/0 conn(0x7ff69815e680 0x7ff69806c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 -- 172.21.15.6:0/3279667089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 msgr2=0x7ff69806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69806e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 -- 172.21.15.6:0/3279667089 >> v2:172.21.15.178:3300/0 conn(0x7ff69815e680 msgr2=0x7ff69806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 --2- 172.21.15.6:0/3279667089 >> v2:172.21.15.178:3300/0 conn(0x7ff69815e680 0x7ff69806c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697e00640 1 -- 172.21.15.6:0/3279667089 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69815f140 con 0x7ff6980ccad0 2024-03-06T07:31:59.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.932+0000 7ff697400640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69806e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.933+0000 7ff697e00640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff6980ccf80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff6980cd6c0 tx=0x7ff69002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1608c9dede0e6ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff696a00640 1 -- 172.21.15.6:0/3279667089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69002ca80 con 0x7ff6980ccad0 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff696a00640 1 -- 172.21.15.6:0/3279667089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69002cc20 con 0x7ff6980ccad0 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff696a00640 1 -- 172.21.15.6:0/3279667089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6900343f0 con 0x7ff6980ccad0 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 -- 172.21.15.6:0/3279667089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 msgr2=0x7ff6980ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff6980ccf80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7ff6980cd6c0 tx=0x7ff69002bee0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 -- 172.21.15.6:0/3279667089 shutdown_connections 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff6980ccf80 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 --2- 172.21.15.6:0/3279667089 >> v2:172.21.15.178:3300/0 conn(0x7ff69815e680 0x7ff69806c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 --2- 172.21.15.6:0/3279667089 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69806e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.934+0000 7ff69e000640 1 -- 172.21.15.6:0/3279667089 >> 172.21.15.6:0/3279667089 conn(0x7ff69806b3d0 msgr2=0x7ff69806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.935+0000 7ff69e000640 1 -- 172.21.15.6:0/3279667089 shutdown_connections 2024-03-06T07:31:59.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.935+0000 7ff69e000640 1 -- 172.21.15.6:0/3279667089 wait complete. 2024-03-06T07:31:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.936+0000 7ff69e000640 1 Processor -- start 2024-03-06T07:31:59.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.936+0000 7ff69e000640 1 -- start start 2024-03-06T07:31:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.937+0000 7ff69e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.937+0000 7ff69e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff69809bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.937+0000 7ff69e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff69815e680 0x7ff69814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.937+0000 7ff69e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69806ea00 con 0x7ff6980ccad0 2024-03-06T07:31:59.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.937+0000 7ff69e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69806e880 con 0x7ff69815e680 2024-03-06T07:31:59.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.937+0000 7ff69e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69806cec0 con 0x7ff69806da40 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff69809bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff69809bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53266/0 (socket says 172.21.15.6:53266) 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 -- 172.21.15.6:0/449845749 learned_addr learned my addr 172.21.15.6:0/449845749 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff697400640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff697e00640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff69815e680 0x7ff69814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 -- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 msgr2=0x7ff69815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69815a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 -- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff69815e680 msgr2=0x7ff69814e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff69815e680 0x7ff69814e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.938+0000 7ff68fe00640 1 -- 172.21.15.6:0/449845749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6980b36b0 con 0x7ff6980ccad0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.939+0000 7ff68fe00640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff69809bc00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff68000c7f0 tx=0x7ff68000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.939+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680015070 con 0x7ff6980ccad0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.939+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff680004220 con 0x7ff6980ccad0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.939+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff680007620 con 0x7ff6980ccad0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.939+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6980b3990 con 0x7ff6980ccad0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.939+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6980af550 con 0x7ff6980ccad0 2024-03-06T07:31:59.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.940+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69804f330 con 0x7ff6980ccad0 2024-03-06T07:31:59.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.941+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6800077c0 con 0x7ff6980ccad0 2024-03-06T07:31:59.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.942+0000 7ff68f400640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b4d0 0x7ff67406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.942+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff68008a480 con 0x7ff6980ccad0 2024-03-06T07:31:59.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.943+0000 7ff697400640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b4d0 0x7ff67406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.944+0000 7ff697400640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b4d0 0x7ff67406d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff688006b10 tx=0x7ff688008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:31:59.948+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff680059200 con 0x7ff6980ccad0 2024-03-06T07:32:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.191+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff69815ffa0 con 0x7ff6980ccad0 2024-03-06T07:32:00.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.723+0000 7ff68f400640 1 -- 172.21.15.6:0/449845749 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff680058b00 con 0x7ff6980ccad0 2024-03-06T07:32:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b4d0 msgr2=0x7ff67406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b4d0 0x7ff67406d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff688006b10 tx=0x7ff688008040 comp rx=0 tx=0).stop 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 msgr2=0x7ff69809bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff69809bc00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff68000c7f0 tx=0x7ff68000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 shutdown_connections 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b4d0 0x7ff67406d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff69815e680 0x7ff69814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6980ccad0 0x7ff69809bc00 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 --2- 172.21.15.6:0/449845749 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff69806da40 0x7ff69815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 >> 172.21.15.6:0/449845749 conn(0x7ff69806b3d0 msgr2=0x7ff698162000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.726+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 shutdown_connections 2024-03-06T07:32:00.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.727+0000 7ff69e000640 1 -- 172.21.15.6:0/449845749 wait complete. 2024-03-06T07:32:00.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.975+0000 7f8ed3200640 1 Processor -- start 2024-03-06T07:32:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.975+0000 7f8ed3200640 1 -- start start 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed415cc20 0x7f8ed4072450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ed40ccc20 0x7f8ed40cd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed4159b10 con 0x7f8ed4159730 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed415d080 con 0x7f8ed40ccc20 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4071690 con 0x7f8ed415cc20 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ed4159730 0x7f8ed40ceea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44898/0 (socket says 172.21.15.6:44898) 2024-03-06T07:32:00.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed1e00640 1 -- 172.21.15.6:0/72267733 learned_addr learned my addr 172.21.15.6:0/72267733 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.977+0000 7f8ed2800640 1 --2- 172.21.15.6:0/72267733 >> v2:172.21.15.6:3300/0 conn(0x7f8ed40ccc20 0x7f8ed40cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.978+0000 7f8ecbe00640 1 --2- 172.21.15.6:0/72267733 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed415cc20 0x7f8ed4072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.978+0000 7f8ed2800640 1 -- 172.21.15.6:0/72267733 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed415cc20 msgr2=0x7f8ed4072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.978+0000 7f8ed2800640 1 --2- 172.21.15.6:0/72267733 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed415cc20 0x7f8ed4072450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.978+0000 7f8ed1400640 1 -- 172.21.15.6:0/72267733 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38921611 0 0) 0x7f8ed4159b10 con 0x7f8ed4159730 2024-03-06T07:32:00.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.978+0000 7f8ed2800640 1 -- 172.21.15.6:0/72267733 >> v1:172.21.15.6:6789/0 conn(0x7f8ed4159730 legacy=0x7f8ed40ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.978+0000 7f8ed2800640 1 -- 172.21.15.6:0/72267733 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed40cd580 con 0x7f8ed40ccc20 2024-03-06T07:32:00.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.979+0000 7f8ed2800640 1 --2- 172.21.15.6:0/72267733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed40ccc20 0x7f8ed40cd040 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009d90 tx=0x7f8ecc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0c5e6c79e592c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed1400640 1 -- 172.21.15.6:0/72267733 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc039070 con 0x7f8ed40ccc20 2024-03-06T07:32:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed1400640 1 -- 172.21.15.6:0/72267733 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc02cc50 con 0x7f8ed40ccc20 2024-03-06T07:32:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed1400640 1 -- 172.21.15.6:0/72267733 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc034440 con 0x7f8ed40ccc20 2024-03-06T07:32:00.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed3200640 1 -- 172.21.15.6:0/72267733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed40ccc20 msgr2=0x7f8ed40cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed3200640 1 --2- 172.21.15.6:0/72267733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed40ccc20 0x7f8ed40cd040 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009d90 tx=0x7f8ecc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed3200640 1 -- 172.21.15.6:0/72267733 shutdown_connections 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed3200640 1 --2- 172.21.15.6:0/72267733 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed40ccc20 0x7f8ed40cd040 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed3200640 1 --2- 172.21.15.6:0/72267733 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed415cc20 0x7f8ed4072450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.980+0000 7f8ed3200640 1 -- 172.21.15.6:0/72267733 >> 172.21.15.6:0/72267733 conn(0x7f8ed406afb0 msgr2=0x7f8ed406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.981+0000 7f8ed3200640 1 -- 172.21.15.6:0/72267733 shutdown_connections 2024-03-06T07:32:00.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.981+0000 7f8ed3200640 1 -- 172.21.15.6:0/72267733 wait complete. 2024-03-06T07:32:00.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.982+0000 7f8ed3200640 1 Processor -- start 2024-03-06T07:32:00.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.982+0000 7f8ed3200640 1 -- start start 2024-03-06T07:32:00.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.983+0000 7f8ed3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed40ccc20 0x7f8ed4150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.983+0000 7f8ed3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed4159730 0x7f8ed40adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.983+0000 7f8ed3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 0x7f8ed40ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.983+0000 7f8ed3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4071f30 con 0x7f8ed415cc20 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.983+0000 7f8ed3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed4071c50 con 0x7f8ed4159730 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.983+0000 7f8ed3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed40cf6d0 con 0x7f8ed40ccc20 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 0x7f8ed40ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 0x7f8ed40ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53294/0 (socket says 172.21.15.6:53294) 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 -- 172.21.15.6:0/3983368388 learned_addr learned my addr 172.21.15.6:0/3983368388 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed1e00640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed40ccc20 0x7f8ed4150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ecbe00640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed4159730 0x7f8ed40adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 -- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed40ccc20 msgr2=0x7f8ed4150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed40ccc20 0x7f8ed4150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 -- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed4159730 msgr2=0x7f8ed40adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed4159730 0x7f8ed40adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed2800640 1 -- 172.21.15.6:0/3983368388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed4152b20 con 0x7f8ed415cc20 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ed1e00640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed40ccc20 0x7f8ed4150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.984+0000 7f8ecbe00640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed4159730 0x7f8ed40adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.985+0000 7f8ed2800640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 0x7f8ed40ae0c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009eb0 tx=0x7f8ecc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.985+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc039050 con 0x7f8ed415cc20 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.985+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc045070 con 0x7f8ed415cc20 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.985+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc03e610 con 0x7f8ed415cc20 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.985+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed4154290 con 0x7f8ed415cc20 2024-03-06T07:32:00.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.985+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed4154850 con 0x7f8ed415cc20 2024-03-06T07:32:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.987+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed40b5d20 con 0x7f8ed415cc20 2024-03-06T07:32:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.987+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ecc03e7b0 con 0x7f8ed415cc20 2024-03-06T07:32:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.988+0000 7f8eca000640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8eb006b500 0x7f8eb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.988+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8ecc0acde0 con 0x7f8ed415cc20 2024-03-06T07:32:00.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.989+0000 7f8ed1e00640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8eb006b500 0x7f8eb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.990+0000 7f8ed1e00640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8eb006b500 0x7f8eb006d9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4000f60 tx=0x7f8ec4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:00.993+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ed40b5d20 con 0x7f8ed415cc20 2024-03-06T07:32:01.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:01.194+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8ed404f330 con 0x7f8ed415cc20 2024-03-06T07:32:02.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.029+0000 7f8eca000640 1 -- 172.21.15.6:0/3983368388 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v71) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8ecc032e20 con 0x7f8ed415cc20 2024-03-06T07:32:02.032 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8eb006b500 msgr2=0x7f8eb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8eb006b500 0x7f8eb006d9c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4000f60 tx=0x7f8ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 msgr2=0x7f8ed40ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 0x7f8ed40ae0c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009eb0 tx=0x7f8ecc004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 shutdown_connections 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8eb006b500 0x7f8eb006d9c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ed4159730 0x7f8ed40adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ed415cc20 0x7f8ed40ae0c0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 --2- 172.21.15.6:0/3983368388 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ed40ccc20 0x7f8ed4150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.034+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 >> 172.21.15.6:0/3983368388 conn(0x7f8ed406afb0 msgr2=0x7f8ed406de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.035+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 shutdown_connections 2024-03-06T07:32:02.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.035+0000 7f8ed3200640 1 -- 172.21.15.6:0/3983368388 wait complete. 2024-03-06T07:32:02.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:02.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.270+0000 7feb37200640 1 Processor -- start 2024-03-06T07:32:02.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.270+0000 7feb37200640 1 -- start start 2024-03-06T07:32:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.272+0000 7feb37200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb3006e200 0x7feb300720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.272+0000 7feb37200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7feb300725f0 0x7feb300ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.272+0000 7feb37200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb300b3ea0 con 0x7feb30144320 2024-03-06T07:32:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.272+0000 7feb37200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300b4180 con 0x7feb300725f0 2024-03-06T07:32:02.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.272+0000 7feb37200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300b4460 con 0x7feb3006e200 2024-03-06T07:32:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7feb300725f0 0x7feb300ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb35400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb3006e200 0x7feb300720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7feb300725f0 0x7feb300ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:53006/0 (socket says 172.21.15.6:53006) 2024-03-06T07:32:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 -- 172.21.15.6:0/4076706310 learned_addr learned my addr 172.21.15.6:0/4076706310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:02.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb35e00640 1 --1- 172.21.15.6:0/4076706310 >> v1:172.21.15.178:6789/0 conn(0x7feb30144320 0x7feb3006daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:46628/0 (socket says 172.21.15.6:46628) 2024-03-06T07:32:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 -- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb3006e200 msgr2=0x7feb300720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 --2- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb3006e200 0x7feb300720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 -- 172.21.15.6:0/4076706310 >> v1:172.21.15.178:6789/0 conn(0x7feb30144320 legacy=0x7feb3006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.273+0000 7feb36800640 1 -- 172.21.15.6:0/4076706310 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb300b4740 con 0x7feb300725f0 2024-03-06T07:32:02.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.275+0000 7feb36800640 1 --2- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb300ad880 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009e50 tx=0x7feb2c02bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb61920bb7fe81d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.276+0000 7feb34a00640 1 -- 172.21.15.6:0/4076706310 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c02caf0 con 0x7feb300725f0 2024-03-06T07:32:02.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.276+0000 7feb34a00640 1 -- 172.21.15.6:0/4076706310 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2c02cc90 con 0x7feb300725f0 2024-03-06T07:32:02.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.276+0000 7feb34a00640 1 -- 172.21.15.6:0/4076706310 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c034610 con 0x7feb300725f0 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.276+0000 7feb37200640 1 -- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 msgr2=0x7feb300ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.276+0000 7feb37200640 1 --2- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb300ad880 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009e50 tx=0x7feb2c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.277+0000 7feb37200640 1 -- 172.21.15.6:0/4076706310 shutdown_connections 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.277+0000 7feb37200640 1 --2- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb300ad880 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.277+0000 7feb37200640 1 --2- 172.21.15.6:0/4076706310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb3006e200 0x7feb300720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.277+0000 7feb37200640 1 -- 172.21.15.6:0/4076706310 >> 172.21.15.6:0/4076706310 conn(0x7feb300b17c0 msgr2=0x7feb300b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.277+0000 7feb37200640 1 -- 172.21.15.6:0/4076706310 shutdown_connections 2024-03-06T07:32:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.277+0000 7feb37200640 1 -- 172.21.15.6:0/4076706310 wait complete. 2024-03-06T07:32:02.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.279+0000 7feb37200640 1 Processor -- start 2024-03-06T07:32:02.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.279+0000 7feb37200640 1 -- start start 2024-03-06T07:32:02.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.280+0000 7feb37200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 0x7feb301390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.280+0000 7feb37200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb3015e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.280+0000 7feb37200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb30144320 0x7feb30140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.280+0000 7feb37200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb30072aa0 con 0x7feb3006e200 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.280+0000 7feb37200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb300b4f40 con 0x7feb300725f0 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.280+0000 7feb37200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb30144700 con 0x7feb30144320 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 0x7feb301390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 0x7feb301390f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48150/0 (socket says 172.21.15.6:48150) 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 -- 172.21.15.6:0/3329494013 learned_addr learned my addr 172.21.15.6:0/3329494013 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35400640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb3015e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 -- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb30144320 msgr2=0x7feb30140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb30144320 0x7feb30140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 -- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 msgr2=0x7feb3015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb3015e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35e00640 1 -- 172.21.15.6:0/3329494013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30140f10 con 0x7feb3006e200 2024-03-06T07:32:02.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.281+0000 7feb35400640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb3015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.282+0000 7feb35e00640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 0x7feb301390f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7feb2400c3f0 tx=0x7feb2400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.282+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb240074e0 con 0x7feb3006e200 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.282+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb24007680 con 0x7feb3006e200 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.282+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24004a00 con 0x7feb3006e200 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.282+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb301411f0 con 0x7feb3006e200 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.282+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3013b190 con 0x7feb3006e200 2024-03-06T07:32:02.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.284+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3004f330 con 0x7feb3006e200 2024-03-06T07:32:02.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.284+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb2401a020 con 0x7feb3006e200 2024-03-06T07:32:02.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.285+0000 7feb2a000640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1406b4d0 0x7feb1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.285+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5357+0+0 (secure 0 0 0) 0x7feb240898a0 con 0x7feb3006e200 2024-03-06T07:32:02.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.286+0000 7feb35400640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1406b4d0 0x7feb1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.287+0000 7feb35400640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1406b4d0 0x7feb1406d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7feb301f7410 tx=0x7feb20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.290+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb24010300 con 0x7feb3006e200 2024-03-06T07:32:02.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:02.490+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feb300cbfd0 con 0x7feb3006e200 2024-03-06T07:32:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.033+0000 7feb2a000640 1 -- 172.21.15.6:0/3329494013 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v72) v1 ==== 169+0+0 (secure 0 0 0) 0x7feb240587e0 con 0x7feb3006e200 2024-03-06T07:32:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.037+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1406b4d0 msgr2=0x7feb1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.037+0000 7feb37200640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1406b4d0 0x7feb1406d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7feb301f7410 tx=0x7feb20008040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.037+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 msgr2=0x7feb301390f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.037+0000 7feb37200640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 0x7feb301390f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7feb2400c3f0 tx=0x7feb2400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.043+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 shutdown_connections 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.043+0000 7feb37200640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7feb1406b4d0 0x7feb1406d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.043+0000 7feb37200640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feb300725f0 0x7feb3015e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.043+0000 7feb37200640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7feb30144320 0x7feb30140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.043+0000 7feb37200640 1 --2- 172.21.15.6:0/3329494013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7feb3006e200 0x7feb301390f0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.043+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 >> 172.21.15.6:0/3329494013 conn(0x7feb300b17c0 msgr2=0x7feb3013a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.044+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 shutdown_connections 2024-03-06T07:32:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.045+0000 7feb37200640 1 -- 172.21.15.6:0/3329494013 wait complete. 2024-03-06T07:32:03.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:32:03.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.265+0000 7f93b8e00640 1 Processor -- start 2024-03-06T07:32:03.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.265+0000 7f93b8e00640 1 -- start start 2024-03-06T07:32:03.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.266+0000 7f93b8e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f93b4145430 0x7f93b415f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.266+0000 7f93b8e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93b415cc90 0x7f93b40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.266+0000 7f93b8e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b40b2080 0x7f93b406c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.266+0000 7f93b8e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b415fe40 con 0x7f93b415cc90 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.266+0000 7f93b8e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b4160120 con 0x7f93b4145430 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.266+0000 7f93b8e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4160400 con 0x7f93b40b2080 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93b415cc90 0x7f93b40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93b415cc90 0x7f93b40b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48158/0 (socket says 172.21.15.6:48158) 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 -- 172.21.15.6:0/490995338 learned_addr learned my addr 172.21.15.6:0/490995338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93b3400640 1 --2- 172.21.15.6:0/490995338 >> v2:172.21.15.178:3300/0 conn(0x7f93b4145430 0x7f93b415f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93b3e00640 1 --2- 172.21.15.6:0/490995338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b40b2080 0x7f93b406c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 -- 172.21.15.6:0/490995338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b40b2080 msgr2=0x7f93b406c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 --2- 172.21.15.6:0/490995338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b40b2080 0x7f93b406c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 -- 172.21.15.6:0/490995338 >> v2:172.21.15.178:3300/0 conn(0x7f93b4145430 msgr2=0x7f93b415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 --2- 172.21.15.6:0/490995338 >> v2:172.21.15.178:3300/0 conn(0x7f93b4145430 0x7f93b415f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.267+0000 7f93abe00640 1 -- 172.21.15.6:0/490995338 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b41606e0 con 0x7f93b415cc90 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.269+0000 7f93abe00640 1 --2- 172.21.15.6:0/490995338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b40b1b40 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f93b40b2670 tx=0x7f939c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b91a86c2162be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.269+0000 7f93b2a00640 1 -- 172.21.15.6:0/490995338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c035020 con 0x7f93b415cc90 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.269+0000 7f93b2a00640 1 -- 172.21.15.6:0/490995338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c031870 con 0x7f93b415cc90 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.269+0000 7f93b2a00640 1 -- 172.21.15.6:0/490995338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c031c40 con 0x7f93b415cc90 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 -- 172.21.15.6:0/490995338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 msgr2=0x7f93b40b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 --2- 172.21.15.6:0/490995338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b40b1b40 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f93b40b2670 tx=0x7f939c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 -- 172.21.15.6:0/490995338 shutdown_connections 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 --2- 172.21.15.6:0/490995338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b40b2080 0x7f93b406c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 --2- 172.21.15.6:0/490995338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b40b1b40 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 --2- 172.21.15.6:0/490995338 >> v2:172.21.15.178:3300/0 conn(0x7f93b4145430 0x7f93b415f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 -- 172.21.15.6:0/490995338 >> 172.21.15.6:0/490995338 conn(0x7f93b406b3a0 msgr2=0x7f93b406b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 -- 172.21.15.6:0/490995338 shutdown_connections 2024-03-06T07:32:03.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.270+0000 7f93b8e00640 1 -- 172.21.15.6:0/490995338 wait complete. 2024-03-06T07:32:03.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.271+0000 7f93b8e00640 1 Processor -- start 2024-03-06T07:32:03.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.272+0000 7f93b8e00640 1 -- start start 2024-03-06T07:32:03.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b8e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93b40b2080 0x7f93b414f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b8e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b4145430 0x7f93b4141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b4141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b8e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4145990 con 0x7f93b415cc90 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b8e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4145810 con 0x7f93b40b2080 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b8e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40cde00 con 0x7f93b4145430 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b4141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b4141890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48172/0 (socket says 172.21.15.6:48172) 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 -- 172.21.15.6:0/1643472130 learned_addr learned my addr 172.21.15.6:0/1643472130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93abe00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b4145430 0x7f93b4141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3400640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93b40b2080 0x7f93b414f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 -- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b4145430 msgr2=0x7f93b4141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b4145430 0x7f93b4141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 -- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93b40b2080 msgr2=0x7f93b414f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93b40b2080 0x7f93b414f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3e00640 1 -- 172.21.15.6:0/1643472130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4143340 con 0x7f93b415cc90 2024-03-06T07:32:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93b3400640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93b40b2080 0x7f93b414f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.273+0000 7f93abe00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b4145430 0x7f93b4141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.274+0000 7f93b3e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b4141890 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f93ac007e70 tx=0x7f93ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.274+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac014620 con 0x7f93b415cc90 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.274+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac019070 con 0x7f93b415cc90 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.274+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b414ff60 con 0x7f93b415cc90 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.274+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac014a20 con 0x7f93b415cc90 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.274+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b4143600 con 0x7f93b415cc90 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.276+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b404f330 con 0x7f93b415cc90 2024-03-06T07:32:03.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.276+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93ac014bc0 con 0x7f93b415cc90 2024-03-06T07:32:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.277+0000 7f93ab400640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f939006b4d0 0x7f939006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.277+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f93ac08a800 con 0x7f93b415cc90 2024-03-06T07:32:03.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.278+0000 7f93b3400640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f939006b4d0 0x7f939006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.279+0000 7f93b3400640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f939006b4d0 0x7f939006d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f93a4006fd0 tx=0x7f93a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.282+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f93ac059980 con 0x7f93b415cc90 2024-03-06T07:32:03.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.480+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f93b40cd680 con 0x7f93b415cc90 2024-03-06T07:32:03.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.485+0000 7f93ab400640 1 -- 172.21.15.6:0/1643472130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v24) v1 ==== 108+0+0 (secure 0 0 0) 0x7f93ac059280 con 0x7f93b415cc90 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f939006b4d0 msgr2=0x7f939006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f939006b4d0 0x7f939006d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f93a4006fd0 tx=0x7f93a4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 msgr2=0x7f93b4141890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b4141890 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f93ac007e70 tx=0x7f93ac00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 shutdown_connections 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f939006b4d0 0x7f939006d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93b415cc90 0x7f93b4141890 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93b4145430 0x7f93b4141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 --2- 172.21.15.6:0/1643472130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f93b40b2080 0x7f93b414f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 >> 172.21.15.6:0/1643472130 conn(0x7f93b406b3a0 msgr2=0x7f93b40ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.488+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 shutdown_connections 2024-03-06T07:32:03.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.489+0000 7f93b8e00640 1 -- 172.21.15.6:0/1643472130 wait complete. 2024-03-06T07:32:03.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:03.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.734+0000 7fde99400640 1 Processor -- start 2024-03-06T07:32:03.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.734+0000 7fde99400640 1 -- start start 2024-03-06T07:32:03.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.735+0000 7fde99400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fde940d4860 0x7fde940d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.735+0000 7fde99400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940cfa60 0x7fde940d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.735+0000 7fde99400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde940ceca0 con 0x7fde9415dee0 2024-03-06T07:32:03.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.735+0000 7fde99400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde940cf5d0 con 0x7fde940d4860 2024-03-06T07:32:03.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.735+0000 7fde99400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde940cef80 con 0x7fde940cfa60 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.736+0000 7fde93e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde9415dee0 0x7fde940d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51222/0 (socket says 172.21.15.6:51222) 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.736+0000 7fde93e00640 1 -- 172.21.15.6:0/2168429165 learned_addr learned my addr 172.21.15.6:0/2168429165 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.736+0000 7fde93400640 1 --2- 172.21.15.6:0/2168429165 >> v2:172.21.15.178:3300/0 conn(0x7fde940d4860 0x7fde940d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.736+0000 7fde98a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940cfa60 0x7fde940d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.736+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158447935 0 0) 0x7fde940ceca0 con 0x7fde9415dee0 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.736+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78003520 con 0x7fde9415dee0 2024-03-06T07:32:03.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1773470131 0 0) 0x7fde78003520 con 0x7fde9415dee0 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940cfa60 msgr2=0x7fde940d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940cfa60 0x7fde940d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 >> v2:172.21.15.178:3300/0 conn(0x7fde940d4860 msgr2=0x7fde940d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 --2- 172.21.15.6:0/2168429165 >> v2:172.21.15.178:3300/0 conn(0x7fde940d4860 0x7fde940d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940cf260 con 0x7fde9415dee0 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fde84001670 con 0x7fde9415dee0 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 >> v1:172.21.15.6:6789/0 conn(0x7fde9415dee0 legacy=0x7fde940d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde780039b0 0x7fde78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde780043a0 0x7fde78004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940cf5d0 con 0x7fde78004ef0 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940cef80 con 0x7fde780043a0 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7800bf00 con 0x7fde780039b0 2024-03-06T07:32:03.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.737+0000 7fde98a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde93400640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde780043a0 0x7fde78004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde93e00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde780039b0 0x7fde78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde98a00640 1 -- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde780039b0 msgr2=0x7fde78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde98a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde780039b0 0x7fde78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde98a00640 1 -- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde780043a0 msgr2=0x7fde78004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde98a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde780043a0 0x7fde78004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.738+0000 7fde98a00640 1 -- 172.21.15.6:0/2168429165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940cf260 con 0x7fde78004ef0 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.739+0000 7fde98a00640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde7800b9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fde940cfe40 tx=0x7fde78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a521f6ce1d9c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c039070 con 0x7fde78004ef0 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c02cb70 con 0x7fde78004ef0 2024-03-06T07:32:03.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde92a00640 1 -- 172.21.15.6:0/2168429165 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c009bd0 con 0x7fde78004ef0 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde99400640 1 -- 172.21.15.6:0/2168429165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 msgr2=0x7fde7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde99400640 1 --2- 172.21.15.6:0/2168429165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde7800b9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fde940cfe40 tx=0x7fde78005830 comp rx=0 tx=0).stop 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde98a00640 1 -- 172.21.15.6:0/2168429165 reap_dead start 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde99400640 1 -- 172.21.15.6:0/2168429165 shutdown_connections 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde99400640 1 -- 172.21.15.6:0/2168429165 >> 172.21.15.6:0/2168429165 conn(0x7fde940d3cd0 msgr2=0x7fde940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde99400640 1 -- 172.21.15.6:0/2168429165 shutdown_connections 2024-03-06T07:32:03.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.740+0000 7fde99400640 1 -- 172.21.15.6:0/2168429165 wait complete. 2024-03-06T07:32:03.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.742+0000 7fde99400640 1 Processor -- start 2024-03-06T07:32:03.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.742+0000 7fde99400640 1 -- start start 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde99400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde9415cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde99400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde9409e6e0 0x7fde940c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde99400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940c9ff0 0x7fde941f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde99400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde940d0b70 con 0x7fde78004ef0 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde99400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9415e2c0 con 0x7fde9409e6e0 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde99400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde940d0cf0 con 0x7fde940c9ff0 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde93e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde9415cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde93e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde9415cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48184/0 (socket says 172.21.15.6:48184) 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.743+0000 7fde93e00640 1 -- 172.21.15.6:0/2932864004 learned_addr learned my addr 172.21.15.6:0/2932864004 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde93e00640 1 -- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940c9ff0 msgr2=0x7fde941f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde93e00640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940c9ff0 0x7fde941f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde93e00640 1 -- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde9409e6e0 msgr2=0x7fde940c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde93e00640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde9409e6e0 0x7fde940c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde93e00640 1 -- 172.21.15.6:0/2932864004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9415d0c0 con 0x7fde78004ef0 2024-03-06T07:32:03.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde93e00640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde9415cb80 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fde8400e890 tx=0x7fde8400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8400c850 con 0x7fde78004ef0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde84018070 con 0x7fde78004ef0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde941f4e80 con 0x7fde78004ef0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.744+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8400cc30 con 0x7fde78004ef0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.745+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9409dee0 con 0x7fde78004ef0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.746+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9404f330 con 0x7fde78004ef0 2024-03-06T07:32:03.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.747+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde8401d460 con 0x7fde78004ef0 2024-03-06T07:32:03.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.748+0000 7fde8be00640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fde7006b500 0x7fde7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.748+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fde84089ed0 con 0x7fde78004ef0 2024-03-06T07:32:03.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.748+0000 7fde93400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fde7006b500 0x7fde7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.750+0000 7fde93400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fde7006b500 0x7fde7006d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fde94162600 tx=0x7fde7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.752+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde84058fb0 con 0x7fde78004ef0 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.963+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fde9415d800 con 0x7fde78004ef0 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.968+0000 7fde8be00640 1 -- 172.21.15.6:0/2932864004 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fde840588b0 con 0x7fde78004ef0 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fde7006b500 msgr2=0x7fde7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fde7006b500 0x7fde7006d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fde94162600 tx=0x7fde7c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 msgr2=0x7fde9415cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde9415cb80 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fde8400e890 tx=0x7fde8400ed50 comp rx=0 tx=0).stop 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 shutdown_connections 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fde7006b500 0x7fde7006d9c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde9409e6e0 0x7fde940c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde940c9ff0 0x7fde941f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 --2- 172.21.15.6:0/2932864004 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde78004ef0 0x7fde9415cb80 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 >> 172.21.15.6:0/2932864004 conn(0x7fde940d3cd0 msgr2=0x7fde940b7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 shutdown_connections 2024-03-06T07:32:03.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:03.971+0000 7fde99400640 1 -- 172.21.15.6:0/2932864004 wait complete. 2024-03-06T07:32:03.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:32:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.206+0000 7f504fa00640 1 Processor -- start 2024-03-06T07:32:04.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.207+0000 7f504fa00640 1 -- start start 2024-03-06T07:32:04.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.208+0000 7f504fa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5048072da0 0x7f5048162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.208+0000 7f504fa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.208+0000 7f504fa00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480ce090 con 0x7f5048070ee0 2024-03-06T07:32:04.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.208+0000 7f504fa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50480ceca0 con 0x7f5048072da0 2024-03-06T07:32:04.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.208+0000 7f504fa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50480ce370 con 0x7f50480712c0 2024-03-06T07:32:04.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.209+0000 7f504d000640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f5048070ee0 0x7f504806e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:46636/0 (socket says 172.21.15.6:46636) 2024-03-06T07:32:04.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.209+0000 7f504d000640 1 -- 172.21.15.6:0/3555139413 learned_addr learned my addr 172.21.15.6:0/3555139413 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:04.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.209+0000 7f504da00640 1 --2- 172.21.15.6:0/3555139413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.209+0000 7f5047e00640 1 --2- 172.21.15.6:0/3555139413 >> v2:172.21.15.178:3300/0 conn(0x7f5048072da0 0x7f5048162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f5047400640 1 -- 172.21.15.6:0/3555139413 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240036331 0 0) 0x7f50480ce090 con 0x7f5048070ee0 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f5047400640 1 -- 172.21.15.6:0/3555139413 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c003520 con 0x7f5048070ee0 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f504da00640 1 -- 172.21.15.6:0/3555139413 >> v2:172.21.15.178:3300/0 conn(0x7f5048072da0 msgr2=0x7f5048162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f504da00640 1 --2- 172.21.15.6:0/3555139413 >> v2:172.21.15.178:3300/0 conn(0x7f5048072da0 0x7f5048162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f504da00640 1 -- 172.21.15.6:0/3555139413 >> v1:172.21.15.178:6789/0 conn(0x7f5048070ee0 legacy=0x7f504806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f504da00640 1 -- 172.21.15.6:0/3555139413 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50480ce650 con 0x7f50480712c0 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.210+0000 7f5047e00640 1 --2- 172.21.15.6:0/3555139413 >> v2:172.21.15.178:3300/0 conn(0x7f5048072da0 0x7f5048162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:04.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.211+0000 7f504da00640 1 --2- 172.21.15.6:0/3555139413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504806e7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f50480cee20 tx=0x7f504002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ebe27917138aecf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.212+0000 7f5047400640 1 -- 172.21.15.6:0/3555139413 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040035020 con 0x7f50480712c0 2024-03-06T07:32:04.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.212+0000 7f5047400640 1 -- 172.21.15.6:0/3555139413 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040031810 con 0x7f50480712c0 2024-03-06T07:32:04.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.212+0000 7f5047400640 1 -- 172.21.15.6:0/3555139413 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040031be0 con 0x7f50480712c0 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 -- 172.21.15.6:0/3555139413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 msgr2=0x7f504806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 --2- 172.21.15.6:0/3555139413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504806e7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f50480cee20 tx=0x7f504002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 -- 172.21.15.6:0/3555139413 shutdown_connections 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 --2- 172.21.15.6:0/3555139413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504806e7d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 --2- 172.21.15.6:0/3555139413 >> v2:172.21.15.178:3300/0 conn(0x7f5048072da0 0x7f5048162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 -- 172.21.15.6:0/3555139413 >> 172.21.15.6:0/3555139413 conn(0x7f504806afb0 msgr2=0x7f504806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 -- 172.21.15.6:0/3555139413 shutdown_connections 2024-03-06T07:32:04.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.213+0000 7f504fa00640 1 -- 172.21.15.6:0/3555139413 wait complete. 2024-03-06T07:32:04.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.215+0000 7f504fa00640 1 Processor -- start 2024-03-06T07:32:04.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.215+0000 7f504fa00640 1 -- start start 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504fa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5048070ee0 0x7f504815a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504fa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504809bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 0x7f50480ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cdda0 con 0x7f5048072da0 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504fa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cdc20 con 0x7f5048070ee0 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504fa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cf130 con 0x7f50480712c0 2024-03-06T07:32:04.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 0x7f50480ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 0x7f50480ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48186/0 (socket says 172.21.15.6:48186) 2024-03-06T07:32:04.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504da00640 1 -- 172.21.15.6:0/3042969347 learned_addr learned my addr 172.21.15.6:0/3042969347 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:04.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f5047e00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504809bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.216+0000 7f504d000640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5048070ee0 0x7f504815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504da00640 1 -- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 msgr2=0x7f504809bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504da00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504809bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504da00640 1 -- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5048070ee0 msgr2=0x7f504815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504da00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5048070ee0 0x7f504815a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504da00640 1 -- 172.21.15.6:0/3042969347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504809c980 con 0x7f5048072da0 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504da00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 0x7f50480ade00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f504002bec0 tx=0x7f50400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040041070 con 0x7f5048072da0 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040031e40 con 0x7f5048072da0 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040035040 con 0x7f5048072da0 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.217+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504809cc60 con 0x7f5048072da0 2024-03-06T07:32:04.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.218+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504809d220 con 0x7f5048072da0 2024-03-06T07:32:04.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.219+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5048069510 con 0x7f5048072da0 2024-03-06T07:32:04.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.220+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5040038020 con 0x7f5048072da0 2024-03-06T07:32:04.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.220+0000 7f5044c00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f502406b4d0 0x7f502406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:04.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.221+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f50400abb10 con 0x7f5048072da0 2024-03-06T07:32:04.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.221+0000 7f504d000640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f502406b4d0 0x7f502406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:04.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.222+0000 7f504d000640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f502406b4d0 0x7f502406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5038005c00 tx=0x7f5038009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:04.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.225+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f504002f2c0 con 0x7f5048072da0 2024-03-06T07:32:04.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:04.506+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f50480b52d0 con 0x7f5048072da0 2024-03-06T07:32:05.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.054+0000 7f5044c00640 1 -- 172.21.15.6:0/3042969347 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v258) v1 ==== 140+0+0 (secure 0 0 0) 0x7f504007ac80 con 0x7f5048072da0 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f502406b4d0 msgr2=0x7f502406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f502406b4d0 0x7f502406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5038005c00 tx=0x7f5038009040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 msgr2=0x7f50480ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 0x7f50480ade00 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f504002bec0 tx=0x7f50400046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 shutdown_connections 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f502406b4d0 0x7f502406d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5048072da0 0x7f50480ade00 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50480712c0 0x7f504809bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 --2- 172.21.15.6:0/3042969347 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5048070ee0 0x7f504815a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 >> 172.21.15.6:0/3042969347 conn(0x7f504806afb0 msgr2=0x7f50480af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 shutdown_connections 2024-03-06T07:32:05.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.059+0000 7f504fa00640 1 -- 172.21.15.6:0/3042969347 wait complete. 2024-03-06T07:32:05.086 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:32:05.086 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:32:05.086 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:05.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:05.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.302+0000 7ff3fcc00640 1 Processor -- start 2024-03-06T07:32:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.302+0000 7ff3fcc00640 1 -- start start 2024-03-06T07:32:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.304+0000 7ff3fcc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff3f815df00 0x7ff3f806c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.304+0000 7ff3fcc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3f80b0d30 0x7ff3f80af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.304+0000 7ff3fcc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f80b6270 con 0x7ff3f80b1280 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.304+0000 7ff3fcc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f80aeb00 con 0x7ff3f80b0d30 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.304+0000 7ff3fcc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f80b6510 con 0x7ff3f815df00 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3efe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff3f80b1280 0x7ff3f806cdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51232/0 (socket says 172.21.15.6:51232) 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3efe00640 1 -- 172.21.15.6:0/1666834214 learned_addr learned my addr 172.21.15.6:0/1666834214 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:05.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f7e00640 1 --2- 172.21.15.6:0/1666834214 >> v2:172.21.15.6:3300/0 conn(0x7ff3f80b0d30 0x7ff3f80af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f7400640 1 --2- 172.21.15.6:0/1666834214 >> v2:172.21.15.178:3300/0 conn(0x7ff3f815df00 0x7ff3f806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f7e00640 1 -- 172.21.15.6:0/1666834214 >> v2:172.21.15.178:3300/0 conn(0x7ff3f815df00 msgr2=0x7ff3f806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f7e00640 1 --2- 172.21.15.6:0/1666834214 >> v2:172.21.15.178:3300/0 conn(0x7ff3f815df00 0x7ff3f806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f7e00640 1 -- 172.21.15.6:0/1666834214 >> v1:172.21.15.6:6789/0 conn(0x7ff3f80b1280 legacy=0x7ff3f806cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f7e00640 1 -- 172.21.15.6:0/1666834214 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f80b67f0 con 0x7ff3f80b0d30 2024-03-06T07:32:05.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.305+0000 7ff3f6a00640 1 -- 172.21.15.6:0/1666834214 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424364664 0 0) 0x7ff3f80b6270 con 0x7ff3f80b1280 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.306+0000 7ff3f7e00640 1 --2- 172.21.15.6:0/1666834214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b0d30 0x7ff3f80af270 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8074d80 tx=0x7ff3f002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d138992d70c1f6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.307+0000 7ff3f6a00640 1 -- 172.21.15.6:0/1666834214 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0039070 con 0x7ff3f80b0d30 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.307+0000 7ff3f6a00640 1 -- 172.21.15.6:0/1666834214 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f002cbe0 con 0x7ff3f80b0d30 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.307+0000 7ff3f6a00640 1 -- 172.21.15.6:0/1666834214 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0034440 con 0x7ff3f80b0d30 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.307+0000 7ff3fcc00640 1 -- 172.21.15.6:0/1666834214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b0d30 msgr2=0x7ff3f80af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.307+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/1666834214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b0d30 0x7ff3f80af270 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8074d80 tx=0x7ff3f002be30 comp rx=0 tx=0).stop 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.308+0000 7ff3fcc00640 1 -- 172.21.15.6:0/1666834214 shutdown_connections 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.308+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/1666834214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b0d30 0x7ff3f80af270 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.308+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/1666834214 >> v2:172.21.15.178:3300/0 conn(0x7ff3f815df00 0x7ff3f806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.308+0000 7ff3fcc00640 1 -- 172.21.15.6:0/1666834214 >> 172.21.15.6:0/1666834214 conn(0x7ff3f809aa10 msgr2=0x7ff3f809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.308+0000 7ff3fcc00640 1 -- 172.21.15.6:0/1666834214 shutdown_connections 2024-03-06T07:32:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.308+0000 7ff3fcc00640 1 -- 172.21.15.6:0/1666834214 wait complete. 2024-03-06T07:32:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.310+0000 7ff3fcc00640 1 Processor -- start 2024-03-06T07:32:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.310+0000 7ff3fcc00640 1 -- start start 2024-03-06T07:32:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3fcc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3f80b0d30 0x7ff3f814bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3fcc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 0x7ff3f80b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3fcc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3f815df00 0x7ff3f80b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3fcc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f815e460 con 0x7ff3f80b1280 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3fcc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f815e2e0 con 0x7ff3f815df00 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3fcc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f80af7b0 con 0x7ff3f80b0d30 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 0x7ff3f80b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 0x7ff3f80b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48212/0 (socket says 172.21.15.6:48212) 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.311+0000 7ff3efe00640 1 -- 172.21.15.6:0/3729535835 learned_addr learned my addr 172.21.15.6:0/3729535835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3f7e00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3f815df00 0x7ff3f80b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3f7400640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3f80b0d30 0x7ff3f814bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3efe00640 1 -- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3f80b0d30 msgr2=0x7ff3f814bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3efe00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3f80b0d30 0x7ff3f814bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3efe00640 1 -- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3f815df00 msgr2=0x7ff3f80b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3efe00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3f815df00 0x7ff3f80b26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3efe00640 1 -- 172.21.15.6:0/3729535835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f80b2bf0 con 0x7ff3f80b1280 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3f7400640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3f80b0d30 0x7ff3f814bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3efe00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 0x7ff3f80b3020 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000c4d0 tx=0x7ff3e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.312+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0018070 con 0x7ff3f80b1280 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.313+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e0007500 con 0x7ff3f80b1280 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.313+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0004a20 con 0x7ff3f80b1280 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.313+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f80b36b0 con 0x7ff3f80b1280 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.313+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f80b3c70 con 0x7ff3f80b1280 2024-03-06T07:32:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.314+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f804f330 con 0x7ff3f80b1280 2024-03-06T07:32:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.315+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3e0004bc0 con 0x7ff3f80b1280 2024-03-06T07:32:05.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.316+0000 7ff3ef400640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3d406b500 0x7ff3d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.316+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff3e0089210 con 0x7ff3f80b1280 2024-03-06T07:32:05.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.317+0000 7ff3f7400640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3d406b500 0x7ff3d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.318+0000 7ff3f7400640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3d406b500 0x7ff3d406d9c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8006fd0 tx=0x7ff3e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.321+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3e0058380 con 0x7ff3f80b1280 2024-03-06T07:32:05.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.523+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3f80b4e80 con 0x7ff3f80b1280 2024-03-06T07:32:05.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.524+0000 7ff3ef400640 1 -- 172.21.15.6:0/3729535835 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+12947 (secure 0 0 0) 0x7ff3e0057c80 con 0x7ff3f80b1280 2024-03-06T07:32:05.527 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:05.527 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":72,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:03.030274+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/4173228924":"2024-03-07T07:31:59.716514+0000","172.21.15.6:6834/4173228924":"2024-03-07T07:31:59.716514+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:05.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3d406b500 msgr2=0x7ff3d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3d406b500 0x7ff3d406d9c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8006fd0 tx=0x7ff3e8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 msgr2=0x7ff3f80b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 0x7ff3f80b3020 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000c4d0 tx=0x7ff3e000c990 comp rx=0 tx=0).stop 2024-03-06T07:32:05.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 shutdown_connections 2024-03-06T07:32:05.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff3d406b500 0x7ff3d406d9c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3f815df00 0x7ff3f80b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3f80b1280 0x7ff3f80b3020 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 --2- 172.21.15.6:0/3729535835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3f80b0d30 0x7ff3f814bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.527+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 >> 172.21.15.6:0/3729535835 conn(0x7ff3f809aa10 msgr2=0x7ff3f80b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.528+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 shutdown_connections 2024-03-06T07:32:05.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.528+0000 7ff3fcc00640 1 -- 172.21.15.6:0/3729535835 wait complete. 2024-03-06T07:32:05.549 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:05.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:05.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.772+0000 7f5a19800640 1 Processor -- start 2024-03-06T07:32:05.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.772+0000 7f5a19800640 1 -- start start 2024-03-06T07:32:05.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.774+0000 7f5a19800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5a14144640 0x7f5a140cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.774+0000 7f5a19800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cbd90 0x7f5a140cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.774+0000 7f5a19800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a1415f640 con 0x7f5a140cdae0 2024-03-06T07:32:05.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.774+0000 7f5a19800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a140cdf50 con 0x7f5a14144640 2024-03-06T07:32:05.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.774+0000 7f5a19800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1406e460 con 0x7f5a140cbd90 2024-03-06T07:32:05.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.775+0000 7f5a12a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5a14144640 0x7f5a140cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.775+0000 7f5a0a000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cbd90 0x7f5a140cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.775+0000 7f5a13400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f5a140cdae0 0x7f5a1406f190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:46644/0 (socket says 172.21.15.6:46644) 2024-03-06T07:32:05.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.775+0000 7f5a13400640 1 -- 172.21.15.6:0/1276339989 learned_addr learned my addr 172.21.15.6:0/1276339989 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:05.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.776+0000 7f5a12a00640 1 -- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cbd90 msgr2=0x7f5a140cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.776+0000 7f5a12a00640 1 --2- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cbd90 0x7f5a140cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.776+0000 7f5a12a00640 1 -- 172.21.15.6:0/1276339989 >> v1:172.21.15.178:6789/0 conn(0x7f5a140cdae0 legacy=0x7f5a1406f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.776+0000 7f5a12a00640 1 -- 172.21.15.6:0/1276339989 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1406e6b0 con 0x7f5a14144640 2024-03-06T07:32:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.776+0000 7f5a0a000640 1 --2- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cbd90 0x7f5a140cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.777+0000 7f5a12a00640 1 --2- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a14144640 0x7f5a140cb850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5a040097a0 tx=0x7f5a0402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aecabe514601788b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a12000640 1 -- 172.21.15.6:0/1276339989 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040315a0 con 0x7f5a14144640 2024-03-06T07:32:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a12000640 1 -- 172.21.15.6:0/1276339989 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a04037070 con 0x7f5a14144640 2024-03-06T07:32:05.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a12000640 1 -- 172.21.15.6:0/1276339989 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04031960 con 0x7f5a14144640 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a19800640 1 -- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a14144640 msgr2=0x7f5a140cb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a19800640 1 --2- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a14144640 0x7f5a140cb850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5a040097a0 tx=0x7f5a0402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a19800640 1 -- 172.21.15.6:0/1276339989 shutdown_connections 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a19800640 1 --2- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cbd90 0x7f5a140cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a19800640 1 --2- 172.21.15.6:0/1276339989 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a14144640 0x7f5a140cb850 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.778+0000 7f5a19800640 1 -- 172.21.15.6:0/1276339989 >> 172.21.15.6:0/1276339989 conn(0x7f5a140b2860 msgr2=0x7f5a140b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.779+0000 7f5a19800640 1 -- 172.21.15.6:0/1276339989 shutdown_connections 2024-03-06T07:32:05.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.779+0000 7f5a19800640 1 -- 172.21.15.6:0/1276339989 wait complete. 2024-03-06T07:32:05.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.780+0000 7f5a19800640 1 Processor -- start 2024-03-06T07:32:05.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.780+0000 7f5a19800640 1 -- start start 2024-03-06T07:32:05.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.781+0000 7f5a19800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a140cbd90 0x7f5a141f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.781+0000 7f5a19800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cdae0 0x7f5a14156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.781+0000 7f5a19800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 0x7f5a141f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.781+0000 7f5a19800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a14144a20 con 0x7f5a14144640 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.781+0000 7f5a19800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1406ec70 con 0x7f5a140cbd90 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.781+0000 7f5a19800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1406f8a0 con 0x7f5a140cdae0 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 0x7f5a141f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 0x7f5a141f5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48214/0 (socket says 172.21.15.6:48214) 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 -- 172.21.15.6:0/86491311 learned_addr learned my addr 172.21.15.6:0/86491311 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a0a000640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cdae0 0x7f5a14156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a12a00640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a140cbd90 0x7f5a141f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 -- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cdae0 msgr2=0x7f5a14156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cdae0 0x7f5a14156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 -- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a140cbd90 msgr2=0x7f5a141f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a140cbd90 0x7f5a141f4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a13400640 1 -- 172.21.15.6:0/86491311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a141576b0 con 0x7f5a14144640 2024-03-06T07:32:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.782+0000 7f5a0a000640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cdae0 0x7f5a14156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.783+0000 7f5a13400640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 0x7f5a141f5110 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00cae0 tx=0x7f5a0c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.783+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c017070 con 0x7f5a14144640 2024-03-06T07:32:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.783+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00ac40 con 0x7f5a14144640 2024-03-06T07:32:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.783+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c012400 con 0x7f5a14144640 2024-03-06T07:32:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.783+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a14157990 con 0x7f5a14144640 2024-03-06T07:32:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.783+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a14157f50 con 0x7f5a14144640 2024-03-06T07:32:05.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.784+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1404f330 con 0x7f5a14144640 2024-03-06T07:32:05.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.786+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a0c01a070 con 0x7f5a14144640 2024-03-06T07:32:05.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.787+0000 7f5a0b400640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59f006b4d0 0x7f59f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.787+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5a0c089e30 con 0x7f5a14144640 2024-03-06T07:32:05.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.787+0000 7f5a12a00640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59f006b4d0 0x7f59f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.789+0000 7f5a12a00640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59f006b4d0 0x7f59f006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5a04036000 tx=0x7f5a0403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.792+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a0c058e00 con 0x7f5a14144640 2024-03-06T07:32:05.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.990+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a1406dee0 con 0x7f5a14144640 2024-03-06T07:32:05.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.990+0000 7f5a0b400640 1 -- 172.21.15.6:0/86491311 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a0c058700 con 0x7f5a14144640 2024-03-06T07:32:05.993 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59f006b4d0 msgr2=0x7f59f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59f006b4d0 0x7f59f006d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5a04036000 tx=0x7f5a0403d040 comp rx=0 tx=0).stop 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 msgr2=0x7f5a141f5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 0x7f5a141f5110 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c00cae0 tx=0x7f5a0c00a060 comp rx=0 tx=0).stop 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 shutdown_connections 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59f006b4d0 0x7f59f006d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5a14144640 0x7f5a141f5110 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a140cdae0 0x7f5a14156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 --2- 172.21.15.6:0/86491311 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a140cbd90 0x7f5a141f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 >> 172.21.15.6:0/86491311 conn(0x7f5a140b2860 msgr2=0x7f5a140b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 shutdown_connections 2024-03-06T07:32:05.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:05.994+0000 7f5a19800640 1 -- 172.21.15.6:0/86491311 wait complete. 2024-03-06T07:32:06.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:32:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.235+0000 7f2b82400640 1 Processor -- start 2024-03-06T07:32:06.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.236+0000 7f2b82400640 1 -- start start 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b82400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2b7c0af1a0 0x7f2b7c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b82400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b7c0cdc20 0x7f2b7c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b82400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7c15cc40 con 0x7f2b7c0af580 2024-03-06T07:32:06.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b82400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b7c15cf20 con 0x7f2b7c0cdc20 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b82400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b7c15d200 con 0x7f2b7c0af1a0 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b7be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b7c0cdc20 0x7f2b7c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b7be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b7c0cdc20 0x7f2b7c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48216/0 (socket says 172.21.15.6:48216) 2024-03-06T07:32:06.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.237+0000 7f2b7be00640 1 -- 172.21.15.6:0/2225031575 learned_addr learned my addr 172.21.15.6:0/2225031575 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.238+0000 7f2b7b400640 1 --2- 172.21.15.6:0/2225031575 >> v2:172.21.15.178:3300/0 conn(0x7f2b7c0af1a0 0x7f2b7c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.238+0000 7f2b7be00640 1 -- 172.21.15.6:0/2225031575 >> v2:172.21.15.178:3300/0 conn(0x7f2b7c0af1a0 msgr2=0x7f2b7c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.238+0000 7f2b7be00640 1 --2- 172.21.15.6:0/2225031575 >> v2:172.21.15.178:3300/0 conn(0x7f2b7c0af1a0 0x7f2b7c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.238+0000 7f2b7be00640 1 -- 172.21.15.6:0/2225031575 >> v1:172.21.15.178:6789/0 conn(0x7f2b7c0af580 legacy=0x7f2b7c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.238+0000 7f2b7be00640 1 -- 172.21.15.6:0/2225031575 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c15d4e0 con 0x7f2b7c0cdc20 2024-03-06T07:32:06.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.239+0000 7f2b7be00640 1 --2- 172.21.15.6:0/2225031575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0cdc20 0x7f2b7c0b2dd0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c074960 tx=0x7f2b7402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c166a6f1ef1d2d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.239+0000 7f2b7aa00640 1 -- 172.21.15.6:0/2225031575 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7402c9f0 con 0x7f2b7c0cdc20 2024-03-06T07:32:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.239+0000 7f2b7aa00640 1 -- 172.21.15.6:0/2225031575 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7402cb90 con 0x7f2b7c0cdc20 2024-03-06T07:32:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.239+0000 7f2b7aa00640 1 -- 172.21.15.6:0/2225031575 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740343f0 con 0x7f2b7c0cdc20 2024-03-06T07:32:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 -- 172.21.15.6:0/2225031575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0cdc20 msgr2=0x7f2b7c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 --2- 172.21.15.6:0/2225031575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0cdc20 0x7f2b7c0b2dd0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c074960 tx=0x7f2b7402be50 comp rx=0 tx=0).stop 2024-03-06T07:32:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 -- 172.21.15.6:0/2225031575 shutdown_connections 2024-03-06T07:32:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 --2- 172.21.15.6:0/2225031575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0cdc20 0x7f2b7c0b2dd0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 --2- 172.21.15.6:0/2225031575 >> v2:172.21.15.178:3300/0 conn(0x7f2b7c0af1a0 0x7f2b7c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 -- 172.21.15.6:0/2225031575 >> 172.21.15.6:0/2225031575 conn(0x7f2b7c06afb0 msgr2=0x7f2b7c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 -- 172.21.15.6:0/2225031575 shutdown_connections 2024-03-06T07:32:06.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.240+0000 7f2b82400640 1 -- 172.21.15.6:0/2225031575 wait complete. 2024-03-06T07:32:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.242+0000 7f2b82400640 1 Processor -- start 2024-03-06T07:32:06.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.242+0000 7f2b82400640 1 -- start start 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b82400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0af1a0 0x7f2b7c1f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b82400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 0x7f2b7c159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b82400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c0cdc20 0x7f2b7c14f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b82400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b36a0 con 0x7f2b7c0af580 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b82400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b3520 con 0x7f2b7c0af1a0 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b82400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b3820 con 0x7f2b7c0cdc20 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 0x7f2b7c159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 0x7f2b7c159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48220/0 (socket says 172.21.15.6:48220) 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.243+0000 7f2b73e00640 1 -- 172.21.15.6:0/3410988534 learned_addr learned my addr 172.21.15.6:0/3410988534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b7b400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0af1a0 0x7f2b7c1f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b7be00640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c0cdc20 0x7f2b7c14f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73e00640 1 -- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c0cdc20 msgr2=0x7f2b7c14f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73e00640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c0cdc20 0x7f2b7c14f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73e00640 1 -- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0af1a0 msgr2=0x7f2b7c1f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73e00640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0af1a0 0x7f2b7c1f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73e00640 1 -- 172.21.15.6:0/3410988534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c0ca730 con 0x7f2b7c0af580 2024-03-06T07:32:06.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73e00640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 0x7f2b7c159040 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400c400 tx=0x7f2b6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64011790 con 0x7f2b7c0af580 2024-03-06T07:32:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64017070 con 0x7f2b7c0af580 2024-03-06T07:32:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c1f7e50 con 0x7f2b7c0af580 2024-03-06T07:32:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.244+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64011bb0 con 0x7f2b7c0af580 2024-03-06T07:32:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.245+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c1f7b60 con 0x7f2b7c0af580 2024-03-06T07:32:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.246+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c069510 con 0x7f2b7c0af580 2024-03-06T07:32:06.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.247+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b6401c460 con 0x7f2b7c0af580 2024-03-06T07:32:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.248+0000 7f2b73400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b5806b4d0 0x7f2b5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.248+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2b64089d60 con 0x7f2b7c0af580 2024-03-06T07:32:06.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.248+0000 7f2b7b400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b5806b4d0 0x7f2b5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:06.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.250+0000 7f2b7b400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b5806b4d0 0x7f2b5806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c006fd0 tx=0x7f2b6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:06.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.252+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b64058ed0 con 0x7f2b7c0af580 2024-03-06T07:32:06.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:06.519+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f2b7c0cf210 con 0x7f2b7c0af580 2024-03-06T07:32:07.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.066+0000 7f2b73400640 1 -- 172.21.15.6:0/3410988534 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v260) v1 ==== 135+0+0 (secure 0 0 0) 0x7f2b640587d0 con 0x7f2b7c0af580 2024-03-06T07:32:07.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b5806b4d0 msgr2=0x7f2b5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b5806b4d0 0x7f2b5806d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c006fd0 tx=0x7f2b6c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 msgr2=0x7f2b7c159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 0x7f2b7c159040 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400c400 tx=0x7f2b6400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 shutdown_connections 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b5806b4d0 0x7f2b5806d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c0cdc20 0x7f2b7c14f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0af580 0x7f2b7c159040 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 --2- 172.21.15.6:0/3410988534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0af1a0 0x7f2b7c1f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 >> 172.21.15.6:0/3410988534 conn(0x7f2b7c06afb0 msgr2=0x7f2b7c15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 shutdown_connections 2024-03-06T07:32:07.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.069+0000 7f2b82400640 1 -- 172.21.15.6:0/3410988534 wait complete. 2024-03-06T07:32:07.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:07.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.311+0000 7f6086000640 1 Processor -- start 2024-03-06T07:32:07.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.311+0000 7f6086000640 1 -- start start 2024-03-06T07:32:07.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f6086000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f60800b2250 0x7f60800b4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f6086000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6080071930 con 0x7f60800729d0 2024-03-06T07:32:07.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f6086000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6080071c10 con 0x7f60800b3c20 2024-03-06T07:32:07.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f6086000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f608015bfb0 con 0x7f60800b2250 2024-03-06T07:32:07.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f607fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f60800729d0 0x7f60800b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51234/0 (socket says 172.21.15.6:51234) 2024-03-06T07:32:07.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f607fe00640 1 -- 172.21.15.6:0/2336449316 learned_addr learned my addr 172.21.15.6:0/2336449316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:07.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.313+0000 7f6085600640 1 --2- 172.21.15.6:0/2336449316 >> v2:172.21.15.6:3300/0 conn(0x7f60800b2250 0x7f60800b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.314+0000 7f607f400640 1 -- 172.21.15.6:0/2336449316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490261609 0 0) 0x7f6080071930 con 0x7f60800729d0 2024-03-06T07:32:07.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.314+0000 7f607f400640 1 -- 172.21.15.6:0/2336449316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6068005730 con 0x7f60800729d0 2024-03-06T07:32:07.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.314+0000 7f6085600640 1 -- 172.21.15.6:0/2336449316 >> v1:172.21.15.178:6789/0 conn(0x7f60800b3c20 legacy=0x7f608015b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.314+0000 7f6085600640 1 -- 172.21.15.6:0/2336449316 >> v1:172.21.15.6:6789/0 conn(0x7f60800729d0 legacy=0x7f60800b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.314+0000 7f6085600640 1 -- 172.21.15.6:0/2336449316 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6080071ef0 con 0x7f60800b2250 2024-03-06T07:32:07.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.315+0000 7f6085600640 1 --2- 172.21.15.6:0/2336449316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f60800b4000 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f608015c130 tx=0x7f607802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50dc594fa166d905 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.316+0000 7f607f400640 1 -- 172.21.15.6:0/2336449316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078039070 con 0x7f60800b2250 2024-03-06T07:32:07.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.316+0000 7f607f400640 1 -- 172.21.15.6:0/2336449316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f607802cce0 con 0x7f60800b2250 2024-03-06T07:32:07.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.316+0000 7f607f400640 1 -- 172.21.15.6:0/2336449316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6078034660 con 0x7f60800b2250 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.316+0000 7f6086000640 1 -- 172.21.15.6:0/2336449316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 msgr2=0x7f60800b4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.316+0000 7f6086000640 1 --2- 172.21.15.6:0/2336449316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f60800b4000 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f608015c130 tx=0x7f607802bf30 comp rx=0 tx=0).stop 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.317+0000 7f6086000640 1 -- 172.21.15.6:0/2336449316 shutdown_connections 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.317+0000 7f6086000640 1 --2- 172.21.15.6:0/2336449316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f60800b4000 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.317+0000 7f6086000640 1 -- 172.21.15.6:0/2336449316 >> 172.21.15.6:0/2336449316 conn(0x7f608006b3a0 msgr2=0x7f608006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.317+0000 7f6086000640 1 -- 172.21.15.6:0/2336449316 shutdown_connections 2024-03-06T07:32:07.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.317+0000 7f6086000640 1 -- 172.21.15.6:0/2336449316 wait complete. 2024-03-06T07:32:07.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.318+0000 7f6086000640 1 Processor -- start 2024-03-06T07:32:07.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.319+0000 7f6086000640 1 -- start start 2024-03-06T07:32:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.319+0000 7f6086000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60800729d0 0x7f6080148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.319+0000 7f6086000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f6080155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.319+0000 7f6086000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60800b3c20 0x7f6080159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f6086000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608015c220 con 0x7f60800b2250 2024-03-06T07:32:07.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f6086000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60800724b0 con 0x7f60800b3c20 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f6086000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f608015c3a0 con 0x7f60800729d0 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f6080155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f6080155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48250/0 (socket says 172.21.15.6:48250) 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 -- 172.21.15.6:0/2022347717 learned_addr learned my addr 172.21.15.6:0/2022347717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f6085600640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60800b3c20 0x7f6080159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f6084c00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60800729d0 0x7f6080148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 -- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60800729d0 msgr2=0x7f6080148510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60800729d0 0x7f6080148510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 -- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60800b3c20 msgr2=0x7f6080159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60800b3c20 0x7f6080159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.320+0000 7f607fe00640 1 -- 172.21.15.6:0/2022347717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60800c9260 con 0x7f60800b2250 2024-03-06T07:32:07.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.321+0000 7f607fe00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f6080155a00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f606c00c500 tx=0x7f606c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.321+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c017070 con 0x7f60800b2250 2024-03-06T07:32:07.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.321+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606c005ce0 con 0x7f60800b2250 2024-03-06T07:32:07.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.321+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606c007330 con 0x7f60800b2250 2024-03-06T07:32:07.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.321+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60800c9540 con 0x7f60800b2250 2024-03-06T07:32:07.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.321+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60800c9b00 con 0x7f60800b2250 2024-03-06T07:32:07.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.323+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608004f330 con 0x7f60800b2250 2024-03-06T07:32:07.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.323+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f606c01a070 con 0x7f60800b2250 2024-03-06T07:32:07.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.324+0000 7f607cc00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.324+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f606c089330 con 0x7f60800b2250 2024-03-06T07:32:07.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.325+0000 7f6084c00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.326+0000 7f6084c00640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6074000f60 tx=0x7f6074009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.329+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f606c058600 con 0x7f60800b2250 2024-03-06T07:32:07.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.529+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f608015b060 con 0x7f60800b2250 2024-03-06T07:32:07.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.536+0000 7f607cc00640 1 -- 172.21.15.6:0/2022347717 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f606c057f00 con 0x7f60800b2250 2024-03-06T07:32:07.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f606006b4d0 msgr2=0x7f606006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f606006b4d0 0x7f606006d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6074000f60 tx=0x7f6074009040 comp rx=0 tx=0).stop 2024-03-06T07:32:07.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 msgr2=0x7f6080155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f6080155a00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f606c00c500 tx=0x7f606c00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 shutdown_connections 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f606006b4d0 0x7f606006d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f60800b3c20 0x7f6080159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60800b2250 0x7f6080155a00 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 --2- 172.21.15.6:0/2022347717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60800729d0 0x7f6080148510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 >> 172.21.15.6:0/2022347717 conn(0x7f608006b3a0 msgr2=0x7f608006e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 shutdown_connections 2024-03-06T07:32:07.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:07.539+0000 7f6086000640 1 -- 172.21.15.6:0/2022347717 wait complete. 2024-03-06T07:32:07.562 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:07.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:07.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:07.803 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:07.803 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:07.803 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:07.862 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:07.862 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:32:07.862 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:07.917 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:07.918 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:07.918 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:07.979 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:07.979 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:32:07.979 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:08.035 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:08.035 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:08.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.253+0000 7fd3d7200640 1 Processor -- start 2024-03-06T07:32:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.253+0000 7fd3d7200640 1 -- start start 2024-03-06T07:32:08.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.255+0000 7fd3d7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd3d8145300 0x7fd3d80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.255+0000 7fd3d7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80714d0 con 0x7fd3d8070f10 2024-03-06T07:32:08.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.255+0000 7fd3d7200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d80717b0 con 0x7fd3d80cd060 2024-03-06T07:32:08.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.255+0000 7fd3d7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3d8071a90 con 0x7fd3d8145300 2024-03-06T07:32:08.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.255+0000 7fd3d5400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3d8070f10 0x7fd3d80cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51244/0 (socket says 172.21.15.6:51244) 2024-03-06T07:32:08.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.255+0000 7fd3d5400640 1 -- 172.21.15.6:0/1328567376 learned_addr learned my addr 172.21.15.6:0/1328567376 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:08.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d5e00640 1 --2- 172.21.15.6:0/1328567376 >> v2:172.21.15.178:3300/0 conn(0x7fd3d8145300 0x7fd3d80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638968799 0 0) 0x7fd3d80714d0 con 0x7fd3d8070f10 2024-03-06T07:32:08.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc003520 con 0x7fd3d8070f10 2024-03-06T07:32:08.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3456492736 0 0) 0x7fd3bc003520 con 0x7fd3d8070f10 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 >> v2:172.21.15.178:3300/0 conn(0x7fd3d8145300 msgr2=0x7fd3d80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 --2- 172.21.15.6:0/1328567376 >> v2:172.21.15.178:3300/0 conn(0x7fd3d8145300 0x7fd3d80b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 >> v1:172.21.15.178:6789/0 conn(0x7fd3d80cd060 legacy=0x7fd3d815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d8071d70 con 0x7fd3d8070f10 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd3c4001670 con 0x7fd3d8070f10 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 >> v1:172.21.15.6:6789/0 conn(0x7fd3d8070f10 legacy=0x7fd3d80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3bc003a00 0x7fd3bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3bc004800 0x7fd3bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d8071a90 con 0x7fd3bc005210 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc00c160 con 0x7fd3bc004800 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.256+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc00c360 con 0x7fd3bc003a00 2024-03-06T07:32:08.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d6800640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d5e00640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3bc003a00 0x7fd3bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d6800640 1 -- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3bc003a00 msgr2=0x7fd3bc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d5400640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3bc004800 0x7fd3bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d6800640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3bc003a00 0x7fd3bc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d6800640 1 -- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3bc004800 msgr2=0x7fd3bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d6800640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3bc004800 0x7fd3bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d6800640 1 -- 172.21.15.6:0/1328567376 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc003520 con 0x7fd3bc005210 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d5400640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3bc004800 0x7fd3bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:08.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.257+0000 7fd3d5e00640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3bc003a00 0x7fd3bc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.258+0000 7fd3d6800640 1 -- 172.21.15.6:0/1328567376 reap_dead start 2024-03-06T07:32:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.258+0000 7fd3d6800640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3bc00b9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8072090 tx=0x7fd3bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39fa35b3cc02274 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0001a70 con 0x7fd3bc005210 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d0003b70 con 0x7fd3bc005210 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d4a00640 1 -- 172.21.15.6:0/1328567376 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d00050d0 con 0x7fd3bc005210 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 -- 172.21.15.6:0/1328567376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 msgr2=0x7fd3bc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3bc00b9c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd3d8072090 tx=0x7fd3bc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 -- 172.21.15.6:0/1328567376 shutdown_connections 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1328567376 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3bc00b9c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 -- 172.21.15.6:0/1328567376 >> 172.21.15.6:0/1328567376 conn(0x7fd3d806b3d0 msgr2=0x7fd3d806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 -- 172.21.15.6:0/1328567376 shutdown_connections 2024-03-06T07:32:08.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.259+0000 7fd3d7200640 1 -- 172.21.15.6:0/1328567376 wait complete. 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.261+0000 7fd3d7200640 1 Processor -- start 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.261+0000 7fd3d7200640 1 -- start start 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3d8138170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3d815c400 0x7fd3d815c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3d815cda0 0x7fd3d8139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80ccad0 con 0x7fd3bc005210 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80cc950 con 0x7fd3d815c400 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d80ccc50 con 0x7fd3d815cda0 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3d8138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3d8138170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48276/0 (socket says 172.21.15.6:48276) 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.262+0000 7fd3d5e00640 1 -- 172.21.15.6:0/1908321211 learned_addr learned my addr 172.21.15.6:0/1908321211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:08.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d5e00640 1 -- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3d815cda0 msgr2=0x7fd3d8139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d5e00640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3d815cda0 0x7fd3d8139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d5e00640 1 -- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3d815c400 msgr2=0x7fd3d815c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d5e00640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3d815c400 0x7fd3d815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d5e00640 1 -- 172.21.15.6:0/1908321211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d815d220 con 0x7fd3bc005210 2024-03-06T07:32:08.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d5e00640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3d8138170 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80108e0 tx=0x7fd3c8010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800f040 con 0x7fd3bc005210 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c800ec20 con 0x7fd3bc005210 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c801c400 con 0x7fd3bc005210 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d815b650 con 0x7fd3bc005210 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.263+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d81390f0 con 0x7fd3bc005210 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.264+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d804f330 con 0x7fd3bc005210 2024-03-06T07:32:08.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.265+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3c801c5a0 con 0x7fd3bc005210 2024-03-06T07:32:08.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.266+0000 7fd3ce000640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd3b806b520 0x7fd3b806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.266+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd3c808a660 con 0x7fd3bc005210 2024-03-06T07:32:08.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.267+0000 7fd3d5400640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd3b806b520 0x7fd3b806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.268+0000 7fd3d5400640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd3b806b520 0x7fd3b806d9e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4000c00 tx=0x7fd3c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.270+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3c8018030 con 0x7fd3bc005210 2024-03-06T07:32:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.508+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d8142980 con 0x7fd3bc005210 2024-03-06T07:32:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.509+0000 7fd3ce000640 1 -- 172.21.15.6:0/1908321211 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fd3c8059830 con 0x7fd3bc005210 2024-03-06T07:32:08.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:08.511 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5216,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/4240727567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":4240727567},{"type":"v1","addr":"172.21.15.178:6837","nonce":4240727567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4199630419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4199630419},{"type":"v1","addr":"172.21.15.178:6836","nonce":4199630419}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6393,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3348889255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3348889255},{"type":"v1","addr":"172.21.15.6:6833","nonce":3348889255}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":6708,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2320143457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2320143457},{"type":"v1","addr":"172.21.15.6:6835","nonce":2320143457}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[]} 2024-03-06T07:32:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 47 2024-03-06T07:32:08.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd3b806b520 msgr2=0x7fd3b806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd3b806b520 0x7fd3b806d9e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd3c4000c00 tx=0x7fd3c4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:08.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 msgr2=0x7fd3d8138170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3d8138170 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80108e0 tx=0x7fd3c8010da0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 shutdown_connections 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd3b806b520 0x7fd3b806d9e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd3d815c400 0x7fd3d815c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd3d815cda0 0x7fd3d8139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.511+0000 7fd3d7200640 1 --2- 172.21.15.6:0/1908321211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3bc005210 0x7fd3d8138170 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.512+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 >> 172.21.15.6:0/1908321211 conn(0x7fd3d806b3d0 msgr2=0x7fd3d81443f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.512+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 shutdown_connections 2024-03-06T07:32:08.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.512+0000 7fd3d7200640 1 -- 172.21.15.6:0/1908321211 wait complete. 2024-03-06T07:32:08.535 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:08.535 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:32:08.535 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:08.535 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:08.535 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:08.535 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:32:08.536 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:08.536 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:08.536 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:08.536 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:32:08.536 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:08.536 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:08.536 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:08.536 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:32:08.537 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:08.537 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:08.558 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:08.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:08.559 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:08.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:08.561 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:08.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:08.568 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:08.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:08.606 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:08.606 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:08.612 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:08.612 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:08.616 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:08.616 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:08.622 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:08.622 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:08.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:32:08.762 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:32:08.765 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:32:08.776 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:32:08.776 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:32:08.778 INFO:tasks.ceph.mds.d.smithi178.stderr:2024-03-06T07:32:08.777+0000 7f8f0e800640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:32:08.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.808+0000 7f7214800640 1 Processor -- start 2024-03-06T07:32:08.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.808+0000 7f7214800640 1 -- start start 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.809+0000 7f7214800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c15e2e0 0x7f720c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.809+0000 7f7214800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f720c0b50e0 0x7f720c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.809+0000 7f7214800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720c0b3940 con 0x7f720c0d0d70 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.809+0000 7f7214800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720c0d4e70 con 0x7f720c0b50e0 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.809+0000 7f7214800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0b3c40 con 0x7f720c15e2e0 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7212800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f720c0b50e0 0x7f720c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f720c0d0d70 0x7f720c0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51268/0 (socket says 172.21.15.6:51268) 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211400640 1 -- 172.21.15.6:0/929295094 learned_addr learned my addr 172.21.15.6:0/929295094 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:08.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211e00640 1 --2- 172.21.15.6:0/929295094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c15e2e0 0x7f720c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211e00640 1 -- 172.21.15.6:0/929295094 >> v2:172.21.15.178:3300/0 conn(0x7f720c0b50e0 msgr2=0x7f720c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211e00640 1 --2- 172.21.15.6:0/929295094 >> v2:172.21.15.178:3300/0 conn(0x7f720c0b50e0 0x7f720c0d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7210a00640 1 -- 172.21.15.6:0/929295094 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901282728 0 0) 0x7f720c0b3940 con 0x7f720c0d0d70 2024-03-06T07:32:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211e00640 1 -- 172.21.15.6:0/929295094 >> v1:172.21.15.6:6789/0 conn(0x7f720c0d0d70 legacy=0x7f720c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.810+0000 7f7211e00640 1 -- 172.21.15.6:0/929295094 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c0cf580 con 0x7f720c15e2e0 2024-03-06T07:32:08.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.811+0000 7f7211e00640 1 --2- 172.21.15.6:0/929295094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c15e2e0 0x7f720c0d0830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f72000097a0 tx=0x7f720002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd9b7a567e8f03e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7210a00640 1 -- 172.21.15.6:0/929295094 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200035020 con 0x7f720c15e2e0 2024-03-06T07:32:08.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7210a00640 1 -- 172.21.15.6:0/929295094 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7200031830 con 0x7f720c15e2e0 2024-03-06T07:32:08.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7210a00640 1 -- 172.21.15.6:0/929295094 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200031c00 con 0x7f720c15e2e0 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 -- 172.21.15.6:0/929295094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c15e2e0 msgr2=0x7f720c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 --2- 172.21.15.6:0/929295094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c15e2e0 0x7f720c0d0830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f72000097a0 tx=0x7f720002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 -- 172.21.15.6:0/929295094 shutdown_connections 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 --2- 172.21.15.6:0/929295094 >> v2:172.21.15.178:3300/0 conn(0x7f720c0b50e0 0x7f720c0d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 --2- 172.21.15.6:0/929295094 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c15e2e0 0x7f720c0d0830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 -- 172.21.15.6:0/929295094 >> 172.21.15.6:0/929295094 conn(0x7f720c0d3f60 msgr2=0x7f720c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 -- 172.21.15.6:0/929295094 shutdown_connections 2024-03-06T07:32:08.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.812+0000 7f7214800640 1 -- 172.21.15.6:0/929295094 wait complete. 2024-03-06T07:32:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.814+0000 7f7214800640 1 Processor -- start 2024-03-06T07:32:08.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.814+0000 7f7214800640 1 -- start start 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7214800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c0b50e0 0x7f720c160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7214800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 0x7f720c15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7214800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c15e2e0 0x7f720c0b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7214800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0cfe20 con 0x7f720c0d0d70 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7214800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0cfb40 con 0x7f720c15e2e0 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7214800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0d2240 con 0x7f720c0b50e0 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7211400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 0x7f720c15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7211400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 0x7f720c15c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48328/0 (socket says 172.21.15.6:48328) 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7211400640 1 -- 172.21.15.6:0/821416275 learned_addr learned my addr 172.21.15.6:0/821416275 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.815+0000 7f7211e00640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c0b50e0 0x7f720c160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7212800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c15e2e0 0x7f720c0b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7211400640 1 -- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c0b50e0 msgr2=0x7f720c160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7211400640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c0b50e0 0x7f720c160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7211400640 1 -- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c15e2e0 msgr2=0x7f720c0b8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7211400640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c15e2e0 0x7f720c0b8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7211400640 1 -- 172.21.15.6:0/821416275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c15e860 con 0x7f720c0d0d70 2024-03-06T07:32:08.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7211400640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 0x7f720c15c030 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c4d0 tx=0x7f71fc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc018070 con 0x7f720c0d0d70 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71fc007500 con 0x7f720c0d0d70 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720c15eb40 con 0x7f720c0d0d70 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.816+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc004a20 con 0x7f720c0d0d70 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.817+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720c15f100 con 0x7f720c0d0d70 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.818+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720c04f3a0 con 0x7f720c0d0d70 2024-03-06T07:32:08.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.818+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71fc01b070 con 0x7f720c0d0d70 2024-03-06T07:32:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.820+0000 7f7206000640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71f006b4d0 0x7f71f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.820+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f71fc089250 con 0x7f720c0d0d70 2024-03-06T07:32:08.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.820+0000 7f7211e00640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71f006b4d0 0x7f71f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.821+0000 7f7211e00640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71f006b4d0 0x7f71f006d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7200036000 tx=0x7f7200008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:08.823+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71fc057ce0 con 0x7f720c0d0d70 2024-03-06T07:32:09.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:09.033+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f720c0d29c0 con 0x7f720c0d0d70 2024-03-06T07:32:10.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.101+0000 7f7206000640 1 -- 172.21.15.6:0/821416275 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v77) v1 ==== 97+0+0 (secure 0 0 0) 0x7f71fc05bd80 con 0x7f720c0d0d70 2024-03-06T07:32:10.104 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71f006b4d0 msgr2=0x7f71f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71f006b4d0 0x7f71f006d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7200036000 tx=0x7f7200008030 comp rx=0 tx=0).stop 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 msgr2=0x7f720c15c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 0x7f720c15c030 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c4d0 tx=0x7f71fc00c990 comp rx=0 tx=0).stop 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 shutdown_connections 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71f006b4d0 0x7f71f006d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f720c15e2e0 0x7f720c0b8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720c0d0d70 0x7f720c15c030 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 --2- 172.21.15.6:0/821416275 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720c0b50e0 0x7f720c160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.106+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 >> 172.21.15.6:0/821416275 conn(0x7f720c0d3f60 msgr2=0x7f720c06c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.107+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 shutdown_connections 2024-03-06T07:32:10.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.107+0000 7f7214800640 1 -- 172.21.15.6:0/821416275 wait complete. 2024-03-06T07:32:10.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:32:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.345+0000 7fbbaf200640 1 Processor -- start 2024-03-06T07:32:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.345+0000 7fbbaf200640 1 -- start start 2024-03-06T07:32:10.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.346+0000 7fbbaf200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbba815efc0 0x7fbba80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.346+0000 7fbbaf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbba80cce90 0x7fbba8161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.346+0000 7fbbaf200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba81627f0 con 0x7fbba8162000 2024-03-06T07:32:10.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.346+0000 7fbbaf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba815f3a0 con 0x7fbba80cce90 2024-03-06T07:32:10.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.346+0000 7fbbaf200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba806d190 con 0x7fbba815efc0 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbba80cce90 0x7fbba8161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbba80cce90 0x7fbba8161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48340/0 (socket says 172.21.15.6:48340) 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 -- 172.21.15.6:0/715619802 learned_addr learned my addr 172.21.15.6:0/715619802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbade00640 1 --2- 172.21.15.6:0/715619802 >> v2:172.21.15.178:3300/0 conn(0x7fbba815efc0 0x7fbba80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 -- 172.21.15.6:0/715619802 >> v2:172.21.15.178:3300/0 conn(0x7fbba815efc0 msgr2=0x7fbba80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 --2- 172.21.15.6:0/715619802 >> v2:172.21.15.178:3300/0 conn(0x7fbba815efc0 0x7fbba80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 -- 172.21.15.6:0/715619802 >> v1:172.21.15.178:6789/0 conn(0x7fbba8162000 legacy=0x7fbba80ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbad400640 1 -- 172.21.15.6:0/715619802 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba806c860 con 0x7fbba80cce90 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.347+0000 7fbbade00640 1 --2- 172.21.15.6:0/715619802 >> v2:172.21.15.178:3300/0 conn(0x7fbba815efc0 0x7fbba80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:10.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.349+0000 7fbbad400640 1 --2- 172.21.15.6:0/715619802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba80cce90 0x7fbba8161ac0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fbba80cd770 tx=0x7fbb9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55709e0bd36f31b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.349+0000 7fbbaca00640 1 -- 172.21.15.6:0/715619802 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98035020 con 0x7fbba80cce90 2024-03-06T07:32:10.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.349+0000 7fbbaca00640 1 -- 172.21.15.6:0/715619802 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb98031870 con 0x7fbba80cce90 2024-03-06T07:32:10.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.349+0000 7fbbaca00640 1 -- 172.21.15.6:0/715619802 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98031c40 con 0x7fbba80cce90 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.349+0000 7fbbaf200640 1 -- 172.21.15.6:0/715619802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba80cce90 msgr2=0x7fbba8161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.349+0000 7fbbaf200640 1 --2- 172.21.15.6:0/715619802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba80cce90 0x7fbba8161ac0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fbba80cd770 tx=0x7fbb9802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.350+0000 7fbbaf200640 1 -- 172.21.15.6:0/715619802 shutdown_connections 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.350+0000 7fbbaf200640 1 --2- 172.21.15.6:0/715619802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba80cce90 0x7fbba8161ac0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.350+0000 7fbbaf200640 1 --2- 172.21.15.6:0/715619802 >> v2:172.21.15.178:3300/0 conn(0x7fbba815efc0 0x7fbba80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.350+0000 7fbbaf200640 1 -- 172.21.15.6:0/715619802 >> 172.21.15.6:0/715619802 conn(0x7fbba806b3d0 msgr2=0x7fbba806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.350+0000 7fbbaf200640 1 -- 172.21.15.6:0/715619802 shutdown_connections 2024-03-06T07:32:10.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.350+0000 7fbbaf200640 1 -- 172.21.15.6:0/715619802 wait complete. 2024-03-06T07:32:10.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.351+0000 7fbbaf200640 1 Processor -- start 2024-03-06T07:32:10.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 -- start start 2024-03-06T07:32:10.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbba80cce90 0x7fbba81e76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 0x7fbba815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbba8162000 0x7fbba80b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8071c30 con 0x7fbba815efc0 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba806ce20 con 0x7fbba80cce90 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.352+0000 7fbbaf200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba8162470 con 0x7fbba8162000 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 0x7fbba815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 0x7fbba815bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48346/0 (socket says 172.21.15.6:48346) 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 -- 172.21.15.6:0/4044676058 learned_addr learned my addr 172.21.15.6:0/4044676058 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 -- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbba8162000 msgr2=0x7fbba80b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbba8162000 0x7fbba80b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 -- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbba80cce90 msgr2=0x7fbba81e76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbba80cce90 0x7fbba81e76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.353+0000 7fbbad400640 1 -- 172.21.15.6:0/4044676058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba81e8000 con 0x7fbba815efc0 2024-03-06T07:32:10.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.354+0000 7fbbad400640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 0x7fbba815bdc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009960 tx=0x7fbb980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.354+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98040070 con 0x7fbba815efc0 2024-03-06T07:32:10.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.354+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9803a420 con 0x7fbba815efc0 2024-03-06T07:32:10.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.354+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98035040 con 0x7fbba815efc0 2024-03-06T07:32:10.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.354+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba81e82e0 con 0x7fbba815efc0 2024-03-06T07:32:10.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.354+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba8158c90 con 0x7fbba815efc0 2024-03-06T07:32:10.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.355+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba804f330 con 0x7fbba815efc0 2024-03-06T07:32:10.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.356+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb98038020 con 0x7fbba815efc0 2024-03-06T07:32:10.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.357+0000 7fbba2000640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb8c06b4d0 0x7fbb8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.357+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbb980abe10 con 0x7fbba815efc0 2024-03-06T07:32:10.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.358+0000 7fbbade00640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb8c06b4d0 0x7fbb8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.359+0000 7fbbade00640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb8c06b4d0 0x7fbb8c06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c006fd0 tx=0x7fbb9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.362+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb980af030 con 0x7fbba815efc0 2024-03-06T07:32:10.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.632+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbba815d970 con 0x7fbba815efc0 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.634+0000 7fbba2000640 1 -- 172.21.15.6:0/4044676058 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbb9807afd0 con 0x7fbba815efc0 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:32:10.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:32:10.637 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.638 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.639 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:32:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.642 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:10.644 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.645 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.646 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:32:10.647 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:10.649 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:32:10.650 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:10.650 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:10.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:10.650 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:32:10.650 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb8c06b4d0 msgr2=0x7fbb8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb8c06b4d0 0x7fbb8c06d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c006fd0 tx=0x7fbb9c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 msgr2=0x7fbba815bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 0x7fbba815bdc0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009960 tx=0x7fbb980046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 shutdown_connections 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb8c06b4d0 0x7fbb8c06d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbba8162000 0x7fbba80b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbba815efc0 0x7fbba815bdc0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 --2- 172.21.15.6:0/4044676058 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbba80cce90 0x7fbba81e76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.636+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 >> 172.21.15.6:0/4044676058 conn(0x7fbba806b3d0 msgr2=0x7fbba8070be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.637+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 shutdown_connections 2024-03-06T07:32:10.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.637+0000 7fbbaf200640 1 -- 172.21.15.6:0/4044676058 wait complete. 2024-03-06T07:32:10.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:10.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.875+0000 7f961d800640 1 Processor -- start 2024-03-06T07:32:10.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.876+0000 7f961d800640 1 -- start start 2024-03-06T07:32:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961d800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9618145300 0x7f96180b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96180cce90 0x7f96180cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96180cd300 con 0x7f96180cce90 2024-03-06T07:32:10.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961d800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96180714d0 con 0x7f9618145300 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961d800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96180717b0 con 0x7f9618070f10 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96180cce90 0x7f96180cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f96180cce90 0x7f96180cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48348/0 (socket says 172.21.15.6:48348) 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.877+0000 7f961ce00640 1 -- 172.21.15.6:0/2388355343 learned_addr learned my addr 172.21.15.6:0/2388355343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.878+0000 7f9617e00640 1 --2- 172.21.15.6:0/2388355343 >> v2:172.21.15.178:3300/0 conn(0x7f9618145300 0x7f96180b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.878+0000 7f961ce00640 1 -- 172.21.15.6:0/2388355343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 msgr2=0x7f96180cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.878+0000 7f961ce00640 1 --2- 172.21.15.6:0/2388355343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.878+0000 7f961ce00640 1 -- 172.21.15.6:0/2388355343 >> v2:172.21.15.178:3300/0 conn(0x7f9618145300 msgr2=0x7f96180b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.878+0000 7f961ce00640 1 --2- 172.21.15.6:0/2388355343 >> v2:172.21.15.178:3300/0 conn(0x7f9618145300 0x7f96180b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.878+0000 7f961ce00640 1 -- 172.21.15.6:0/2388355343 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618071a90 con 0x7f96180cce90 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.879+0000 7f961ce00640 1 --2- 172.21.15.6:0/2388355343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96180cce90 0x7f96180cd930 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f96180cf990 tx=0x7f961002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bee66a7b186559 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.879+0000 7f9616a00640 1 -- 172.21.15.6:0/2388355343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610035020 con 0x7f96180cce90 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.879+0000 7f9616a00640 1 -- 172.21.15.6:0/2388355343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961002d040 con 0x7f96180cce90 2024-03-06T07:32:10.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.879+0000 7f9616a00640 1 -- 172.21.15.6:0/2388355343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610031a80 con 0x7f96180cce90 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 -- 172.21.15.6:0/2388355343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96180cce90 msgr2=0x7f96180cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 --2- 172.21.15.6:0/2388355343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96180cce90 0x7f96180cd930 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f96180cf990 tx=0x7f961002b900 comp rx=0 tx=0).stop 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 -- 172.21.15.6:0/2388355343 shutdown_connections 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 --2- 172.21.15.6:0/2388355343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f96180cce90 0x7f96180cd930 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 --2- 172.21.15.6:0/2388355343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 --2- 172.21.15.6:0/2388355343 >> v2:172.21.15.178:3300/0 conn(0x7f9618145300 0x7f96180b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 -- 172.21.15.6:0/2388355343 >> 172.21.15.6:0/2388355343 conn(0x7f961806b3d0 msgr2=0x7f961806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 -- 172.21.15.6:0/2388355343 shutdown_connections 2024-03-06T07:32:10.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.880+0000 7f961d800640 1 -- 172.21.15.6:0/2388355343 wait complete. 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.882+0000 7f961d800640 1 Processor -- start 2024-03-06T07:32:10.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.882+0000 7f961d800640 1 -- start start 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.883+0000 7f961d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.883+0000 7f961d800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96180cce90 0x7f961815b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.883+0000 7f961d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 0x7f961813fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.883+0000 7f961d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96181456e0 con 0x7f9618145300 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.883+0000 7f961d800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96180b4ee0 con 0x7f96180cce90 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.883+0000 7f961d800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96180cde70 con 0x7f9618070f10 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 0x7f961813fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 0x7f961813fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48356/0 (socket says 172.21.15.6:48356) 2024-03-06T07:32:10.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 -- 172.21.15.6:0/4096734960 learned_addr learned my addr 172.21.15.6:0/4096734960 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f9617e00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f9617400640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96180cce90 0x7f961815b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 -- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 msgr2=0x7f96180ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 -- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96180cce90 msgr2=0x7f961815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96180cce90 0x7f961815b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 -- 172.21.15.6:0/4096734960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96180aeaa0 con 0x7f9618145300 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.884+0000 7f961ce00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 0x7f961813fa20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f961002b940 tx=0x7f96100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.885+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610040070 con 0x7f9618145300 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.885+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9610039cd0 con 0x7f9618145300 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.885+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610035040 con 0x7f9618145300 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.885+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96180ae4e0 con 0x7f9618145300 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.885+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961813cce0 con 0x7f9618145300 2024-03-06T07:32:10.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.886+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961804f330 con 0x7f9618145300 2024-03-06T07:32:10.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.887+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9610038020 con 0x7f9618145300 2024-03-06T07:32:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.888+0000 7f960fe00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.888+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f96100abcc0 con 0x7f9618145300 2024-03-06T07:32:10.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.889+0000 7f9617e00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.890+0000 7f9617e00640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95f406b4d0 0x7f95f406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9618144cf0 tx=0x7f9608007900 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:10.893+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96100b2080 con 0x7f9618145300 2024-03-06T07:32:11.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.092+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9618072470 con 0x7f9618145300 2024-03-06T07:32:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.094+0000 7f960fe00640 1 -- 172.21.15.6:0/4096734960 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+12820 (secure 0 0 0) 0x7f961007ae80 con 0x7f9618145300 2024-03-06T07:32:11.096 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:11.097 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:10.098880+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95f406b4d0 msgr2=0x7f95f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95f406b4d0 0x7f95f406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9618144cf0 tx=0x7f9608007900 comp rx=0 tx=0).stop 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 msgr2=0x7f961813fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 0x7f961813fa20 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f961002b940 tx=0x7f96100046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 shutdown_connections 2024-03-06T07:32:11.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f95f406b4d0 0x7f95f406d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f96180cce90 0x7f961815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9618145300 0x7f961813fa20 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 --2- 172.21.15.6:0/4096734960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9618070f10 0x7f96180ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 >> 172.21.15.6:0/4096734960 conn(0x7f961806b3d0 msgr2=0x7f961815ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 shutdown_connections 2024-03-06T07:32:11.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.097+0000 7f961d800640 1 -- 172.21.15.6:0/4096734960 wait complete. 2024-03-06T07:32:11.120 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:11.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:11.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.340+0000 7f3d67400640 1 Processor -- start 2024-03-06T07:32:11.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.340+0000 7f3d67400640 1 -- start start 2024-03-06T07:32:11.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d67400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3d6015afa0 0x7f3d600cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d67400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d600cd1e0 0x7f3d600b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d67400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d60071c60 con 0x7f3d60070ce0 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d67400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d600cfa10 con 0x7f3d600cd1e0 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d67400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d60071f40 con 0x7f3d6015afa0 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d65400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d600cd1e0 0x7f3d600b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d65400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d600cd1e0 0x7f3d600b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48364/0 (socket says 172.21.15.6:48364) 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d65400640 1 -- 172.21.15.6:0/3636028222 learned_addr learned my addr 172.21.15.6:0/3636028222 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.342+0000 7f3d64a00640 1 --2- 172.21.15.6:0/3636028222 >> v2:172.21.15.178:3300/0 conn(0x7f3d6015afa0 0x7f3d600cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.343+0000 7f3d65400640 1 -- 172.21.15.6:0/3636028222 >> v2:172.21.15.178:3300/0 conn(0x7f3d6015afa0 msgr2=0x7f3d600cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.343+0000 7f3d65400640 1 --2- 172.21.15.6:0/3636028222 >> v2:172.21.15.178:3300/0 conn(0x7f3d6015afa0 0x7f3d600cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.343+0000 7f3d65400640 1 -- 172.21.15.6:0/3636028222 >> v1:172.21.15.178:6789/0 conn(0x7f3d60070ce0 legacy=0x7f3d600ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.343+0000 7f3d65400640 1 -- 172.21.15.6:0/3636028222 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d60072220 con 0x7f3d600cd1e0 2024-03-06T07:32:11.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.344+0000 7f3d65400640 1 --2- 172.21.15.6:0/3636028222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d600cd1e0 0x7f3d600b3ac0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f3d6006bfa0 tx=0x7f3d5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b6309ee18c776b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.344+0000 7f3d5f400640 1 -- 172.21.15.6:0/3636028222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5802e070 con 0x7f3d600cd1e0 2024-03-06T07:32:11.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.344+0000 7f3d5f400640 1 -- 172.21.15.6:0/3636028222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d58037070 con 0x7f3d600cd1e0 2024-03-06T07:32:11.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.344+0000 7f3d5f400640 1 -- 172.21.15.6:0/3636028222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58031a00 con 0x7f3d600cd1e0 2024-03-06T07:32:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 -- 172.21.15.6:0/3636028222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d600cd1e0 msgr2=0x7f3d600b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 --2- 172.21.15.6:0/3636028222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d600cd1e0 0x7f3d600b3ac0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f3d6006bfa0 tx=0x7f3d5802b880 comp rx=0 tx=0).stop 2024-03-06T07:32:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 -- 172.21.15.6:0/3636028222 shutdown_connections 2024-03-06T07:32:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 --2- 172.21.15.6:0/3636028222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d600cd1e0 0x7f3d600b3ac0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 --2- 172.21.15.6:0/3636028222 >> v2:172.21.15.178:3300/0 conn(0x7f3d6015afa0 0x7f3d600cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 -- 172.21.15.6:0/3636028222 >> 172.21.15.6:0/3636028222 conn(0x7f3d6006b3d0 msgr2=0x7f3d6006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 -- 172.21.15.6:0/3636028222 shutdown_connections 2024-03-06T07:32:11.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.345+0000 7f3d67400640 1 -- 172.21.15.6:0/3636028222 wait complete. 2024-03-06T07:32:11.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.347+0000 7f3d67400640 1 Processor -- start 2024-03-06T07:32:11.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.347+0000 7f3d67400640 1 -- start start 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d67400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 0x7f3d6014b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d67400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d600cd1e0 0x7f3d60147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d67400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d6015afa0 0x7f3d60147c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d67400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d600b4000 con 0x7f3d60070ce0 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d67400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d60071170 con 0x7f3d6015afa0 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d67400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d600b4180 con 0x7f3d600cd1e0 2024-03-06T07:32:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.348+0000 7f3d64a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 0x7f3d6014b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 0x7f3d6014b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48374/0 (socket says 172.21.15.6:48374) 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 -- 172.21.15.6:0/321480933 learned_addr learned my addr 172.21.15.6:0/321480933 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 -- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d600cd1e0 msgr2=0x7f3d60147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d600cd1e0 0x7f3d60147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d65400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d6015afa0 0x7f3d60147c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 -- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d6015afa0 msgr2=0x7f3d60147c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d6015afa0 0x7f3d60147c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 -- 172.21.15.6:0/321480933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d60149560 con 0x7f3d60070ce0 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.349+0000 7f3d64a00640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 0x7f3d6014b240 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800d5f0 tx=0x7f3d4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.350+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480125f0 con 0x7f3d60070ce0 2024-03-06T07:32:11.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.350+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d48019070 con 0x7f3d60070ce0 2024-03-06T07:32:11.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.350+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480129d0 con 0x7f3d60070ce0 2024-03-06T07:32:11.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.350+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6014b780 con 0x7f3d60070ce0 2024-03-06T07:32:11.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.350+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d60154f00 con 0x7f3d60070ce0 2024-03-06T07:32:11.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.351+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6004f330 con 0x7f3d60070ce0 2024-03-06T07:32:11.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.352+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d48012b70 con 0x7f3d60070ce0 2024-03-06T07:32:11.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.353+0000 7f3d5cc00640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d3c06b4d0 0x7f3d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.353+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d4808a3a0 con 0x7f3d60070ce0 2024-03-06T07:32:11.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.354+0000 7f3d5fe00640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d3c06b4d0 0x7f3d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.355+0000 7f3d5fe00640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d3c06b4d0 0x7f3d3c06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3d60145f90 tx=0x7f3d50007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.358+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d480594d0 con 0x7f3d60070ce0 2024-03-06T07:32:11.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.557+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d6015a200 con 0x7f3d60070ce0 2024-03-06T07:32:11.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.557+0000 7f3d5cc00640 1 -- 172.21.15.6:0/321480933 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d48058dd0 con 0x7f3d60070ce0 2024-03-06T07:32:11.559 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d3c06b4d0 msgr2=0x7f3d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d3c06b4d0 0x7f3d3c06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3d60145f90 tx=0x7f3d50007d30 comp rx=0 tx=0).stop 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 msgr2=0x7f3d6014b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 0x7f3d6014b240 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800d5f0 tx=0x7f3d4800dab0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 shutdown_connections 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d3c06b4d0 0x7f3d3c06d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d6015afa0 0x7f3d60147c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d600cd1e0 0x7f3d60147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 --2- 172.21.15.6:0/321480933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d60070ce0 0x7f3d6014b240 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.560+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 >> 172.21.15.6:0/321480933 conn(0x7f3d6006b3d0 msgr2=0x7f3d600cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.561+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 shutdown_connections 2024-03-06T07:32:11.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.561+0000 7f3d67400640 1 -- 172.21.15.6:0/321480933 wait complete. 2024-03-06T07:32:11.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.801+0000 7fa2cf200640 1 Processor -- start 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.801+0000 7fa2cf200640 1 -- start start 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.802+0000 7fa2cf200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d4a10 0x7fa2d00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.802+0000 7fa2cf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2d00d0c80 0x7fa2d00b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.802+0000 7fa2cf200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d00b3710 con 0x7fa2d015e0b0 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.802+0000 7fa2cf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d00b09e0 con 0x7fa2d00d0c80 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.802+0000 7fa2cf200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00cfa20 con 0x7fa2d00d4a10 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.803+0000 7fa2ce800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2d00d0c80 0x7fa2d00b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.803+0000 7fa2ce800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2d00d0c80 0x7fa2d00b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48388/0 (socket says 172.21.15.6:48388) 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.803+0000 7fa2ce800640 1 -- 172.21.15.6:0/1213812811 learned_addr learned my addr 172.21.15.6:0/1213812811 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.803+0000 7fa2cd400640 1 --2- 172.21.15.6:0/1213812811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d4a10 0x7fa2d00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.803+0000 7fa2ce800640 1 -- 172.21.15.6:0/1213812811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d4a10 msgr2=0x7fa2d00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.803+0000 7fa2ce800640 1 --2- 172.21.15.6:0/1213812811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d4a10 0x7fa2d00d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.804+0000 7fa2ce800640 1 -- 172.21.15.6:0/1213812811 >> v1:172.21.15.178:6789/0 conn(0x7fa2d015e0b0 legacy=0x7fa2d00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.804+0000 7fa2ce800640 1 -- 172.21.15.6:0/1213812811 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00b39f0 con 0x7fa2d00d0c80 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.805+0000 7fa2ce800640 1 --2- 172.21.15.6:0/1213812811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d0c80 0x7fa2d00b44d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009e50 tx=0x7fa2c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c81bf7542a1e26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.805+0000 7fa2cca00640 1 -- 172.21.15.6:0/1213812811 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c802cb00 con 0x7fa2d00d0c80 2024-03-06T07:32:11.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cca00640 1 -- 172.21.15.6:0/1213812811 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c802cca0 con 0x7fa2d00d0c80 2024-03-06T07:32:11.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cca00640 1 -- 172.21.15.6:0/1213812811 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8034610 con 0x7fa2d00d0c80 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 -- 172.21.15.6:0/1213812811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d0c80 msgr2=0x7fa2d00b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1213812811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d0c80 0x7fa2d00b44d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009e50 tx=0x7fa2c802bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 -- 172.21.15.6:0/1213812811 shutdown_connections 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1213812811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d0c80 0x7fa2d00b44d0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1213812811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d4a10 0x7fa2d00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 -- 172.21.15.6:0/1213812811 >> 172.21.15.6:0/1213812811 conn(0x7fa2d00d3d30 msgr2=0x7fa2d00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 -- 172.21.15.6:0/1213812811 shutdown_connections 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.806+0000 7fa2cf200640 1 -- 172.21.15.6:0/1213812811 wait complete. 2024-03-06T07:32:11.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.808+0000 7fa2cf200640 1 Processor -- start 2024-03-06T07:32:11.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.808+0000 7fa2cf200640 1 -- start start 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cf200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d0c80 0x7fa2d014eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 0x7fa2d0160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cf200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d015e0b0 0x7fa2d006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d015e490 con 0x7fa2d00d4a10 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cf200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b3fb0 con 0x7fa2d015e0b0 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cf200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00cfc60 con 0x7fa2d00d0c80 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 0x7fa2d0160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 0x7fa2d0160eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:48404/0 (socket says 172.21.15.6:48404) 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.809+0000 7fa2cd400640 1 -- 172.21.15.6:0/1206111568 learned_addr learned my addr 172.21.15.6:0/1206111568 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2ce800640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d015e0b0 0x7fa2d006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cd400640 1 -- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d0c80 msgr2=0x7fa2d014eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cde00640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d0c80 0x7fa2d014eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cd400640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d0c80 0x7fa2d014eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cd400640 1 -- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d015e0b0 msgr2=0x7fa2d006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cd400640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d015e0b0 0x7fa2d006c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cd400640 1 -- 172.21.15.6:0/1206111568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d015fd00 con 0x7fa2d00d4a10 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2ce800640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d015e0b0 0x7fa2d006c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cde00640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d0c80 0x7fa2d014eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:11.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2cd400640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 0x7fa2d0160eb0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00e980 tx=0x7fa2bc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.810+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc017070 con 0x7fa2d00d4a10 2024-03-06T07:32:11.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.811+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc00c9a0 con 0x7fa2d00d4a10 2024-03-06T07:32:11.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.811+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc00cd70 con 0x7fa2d00d4a10 2024-03-06T07:32:11.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.811+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d015ffe0 con 0x7fa2d00d4a10 2024-03-06T07:32:11.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.811+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d0160990 con 0x7fa2d00d4a10 2024-03-06T07:32:11.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.812+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d004f330 con 0x7fa2d00d4a10 2024-03-06T07:32:11.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.813+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2bc01a070 con 0x7fa2d00d4a10 2024-03-06T07:32:11.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.814+0000 7fa2c6000640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.814+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa2bc08a160 con 0x7fa2d00d4a10 2024-03-06T07:32:11.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.814+0000 7fa2cde00640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.815+0000 7fa2cde00640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0001680 tx=0x7fa2c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:11.818+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2bc0593c0 con 0x7fa2d00d4a10 2024-03-06T07:32:12.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.012+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa2d00b1800 con 0x7fa2d00d4a10 2024-03-06T07:32:12.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.013+0000 7fa2c6000640 1 -- 172.21.15.6:0/1206111568 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa2bc058cc0 con 0x7fa2d00d4a10 2024-03-06T07:32:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:12.015 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:32:12.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:32:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:32:12.016 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:12.016 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 msgr2=0x7fa2b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0001680 tx=0x7fa2c0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 msgr2=0x7fa2d0160eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 0x7fa2d0160eb0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc00e980 tx=0x7fa2bc00ee40 comp rx=0 tx=0).stop 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 shutdown_connections 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa2b006b4d0 0x7fa2b006d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2d015e0b0 0x7fa2d006c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2d00d4a10 0x7fa2d0160eb0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 --2- 172.21.15.6:0/1206111568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2d00d0c80 0x7fa2d014eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.015+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 >> 172.21.15.6:0/1206111568 conn(0x7fa2d00d3d30 msgr2=0x7fa2d00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.016+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 shutdown_connections 2024-03-06T07:32:12.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.016+0000 7fa2cf200640 1 -- 172.21.15.6:0/1206111568 wait complete. 2024-03-06T07:32:12.037 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:32:12.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:32:12.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.261+0000 7f9339c00640 1 Processor -- start 2024-03-06T07:32:12.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.261+0000 7f9339c00640 1 -- start start 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.262+0000 7f9339c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93340cf8a0 0x7f93340cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.262+0000 7f9339c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933415e8c0 0x7f933415ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9339c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933415f5f0 con 0x7f9334072860 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9339c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340adf40 con 0x7f93340cf8a0 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9339c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93340ad8f0 con 0x7f933415e8c0 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9332000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93340cf8a0 0x7f93340cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9332a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9334072860 0x7f93340ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41678/0 (socket says 172.21.15.6:41678) 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9332a00640 1 -- 172.21.15.6:0/898704728 learned_addr learned my addr 172.21.15.6:0/898704728 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9332000640 1 -- 172.21.15.6:0/898704728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933415e8c0 msgr2=0x7f933415ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9333400640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933415e8c0 0x7f933415ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9332000640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933415e8c0 0x7f933415ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.263+0000 7f9332000640 1 -- 172.21.15.6:0/898704728 >> v1:172.21.15.6:6789/0 conn(0x7f9334072860 legacy=0x7f93340ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.264+0000 7f9331600640 1 -- 172.21.15.6:0/898704728 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238582362 0 0) 0x7f933415f5f0 con 0x7f9334072860 2024-03-06T07:32:12.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.264+0000 7f9332000640 1 -- 172.21.15.6:0/898704728 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340adbd0 con 0x7f93340cf8a0 2024-03-06T07:32:12.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.264+0000 7f9333400640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933415e8c0 0x7f933415ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:12.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.265+0000 7f9332000640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93340cf8a0 0x7f93340cfd30 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f93200097d0 tx=0x7f932002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1ca4afa68f1309 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.265+0000 7f9331600640 1 -- 172.21.15.6:0/898704728 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320031770 con 0x7f93340cf8a0 2024-03-06T07:32:12.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.265+0000 7f9331600640 1 -- 172.21.15.6:0/898704728 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9320037070 con 0x7f93340cf8a0 2024-03-06T07:32:12.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.265+0000 7f9331600640 1 -- 172.21.15.6:0/898704728 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320031b30 con 0x7f93340cf8a0 2024-03-06T07:32:12.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.265+0000 7f9339c00640 1 -- 172.21.15.6:0/898704728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93340cf8a0 msgr2=0x7f93340cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.265+0000 7f9339c00640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93340cf8a0 0x7f93340cfd30 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f93200097d0 tx=0x7f932002b810 comp rx=0 tx=0).stop 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.266+0000 7f9339c00640 1 -- 172.21.15.6:0/898704728 shutdown_connections 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.266+0000 7f9339c00640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f933415e8c0 0x7f933415ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.266+0000 7f9339c00640 1 --2- 172.21.15.6:0/898704728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93340cf8a0 0x7f93340cfd30 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.266+0000 7f9339c00640 1 -- 172.21.15.6:0/898704728 >> 172.21.15.6:0/898704728 conn(0x7f933406afb0 msgr2=0x7f933406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.266+0000 7f9339c00640 1 -- 172.21.15.6:0/898704728 shutdown_connections 2024-03-06T07:32:12.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.266+0000 7f9339c00640 1 -- 172.21.15.6:0/898704728 wait complete. 2024-03-06T07:32:12.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.267+0000 7f9339c00640 1 Processor -- start 2024-03-06T07:32:12.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.267+0000 7f9339c00640 1 -- start start 2024-03-06T07:32:12.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.268+0000 7f9339c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9334072860 0x7f9334157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.268+0000 7f9339c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93340cf8a0 0x7f9334152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.268+0000 7f9339c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 0x7f93340b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.268+0000 7f9339c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334071340 con 0x7f933415e8c0 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.268+0000 7f9339c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340ad570 con 0x7f9334072860 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.268+0000 7f9339c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93340714c0 con 0x7f93340cf8a0 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 0x7f93340b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 0x7f93340b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53422/0 (socket says 172.21.15.6:53422) 2024-03-06T07:32:12.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 -- 172.21.15.6:0/2677871655 learned_addr learned my addr 172.21.15.6:0/2677871655 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:12.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9332000640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93340cf8a0 0x7f9334152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9332a00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9334072860 0x7f9334157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 -- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93340cf8a0 msgr2=0x7f9334152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93340cf8a0 0x7f9334152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 -- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9334072860 msgr2=0x7f9334157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9334072860 0x7f9334157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 -- 172.21.15.6:0/2677871655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93341540d0 con 0x7f933415e8c0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9332a00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9334072860 0x7f9334157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.269+0000 7f9333400640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 0x7f93340b3130 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f932c00ccd0 tx=0x7f932c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.270+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c00f680 con 0x7f933415e8c0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.270+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c00f840 con 0x7f933415e8c0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.270+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c00fc30 con 0x7f933415e8c0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.270+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93341534c0 con 0x7f933415e8c0 2024-03-06T07:32:12.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.270+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9334153a80 con 0x7f933415e8c0 2024-03-06T07:32:12.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.271+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9334068010 con 0x7f933415e8c0 2024-03-06T07:32:12.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.272+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f932c00e3e0 con 0x7f933415e8c0 2024-03-06T07:32:12.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.273+0000 7f932aa00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931406b4d0 0x7f931406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:12.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.273+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f932c08ab70 con 0x7f933415e8c0 2024-03-06T07:32:12.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.273+0000 7f9332a00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931406b4d0 0x7f931406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:12.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.274+0000 7f9332a00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931406b4d0 0x7f931406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9324000f60 tx=0x7f9324009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:12.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.277+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f932c014030 con 0x7f933415e8c0 2024-03-06T07:32:12.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:12.477+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9334072d60 con 0x7f933415e8c0 2024-03-06T07:32:13.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.129+0000 7f932aa00640 1 -- 172.21.15.6:0/2677871655 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f932c059d30 con 0x7f933415e8c0 2024-03-06T07:32:13.131 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931406b4d0 msgr2=0x7f931406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931406b4d0 0x7f931406d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9324000f60 tx=0x7f9324009040 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 msgr2=0x7f93340b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 0x7f93340b3130 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f932c00ccd0 tx=0x7f932c00a650 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 shutdown_connections 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f931406b4d0 0x7f931406d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f933415e8c0 0x7f93340b3130 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93340cf8a0 0x7f9334152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 --2- 172.21.15.6:0/2677871655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9334072860 0x7f9334157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 >> 172.21.15.6:0/2677871655 conn(0x7f933406afb0 msgr2=0x7f93340ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.133+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 shutdown_connections 2024-03-06T07:32:13.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.134+0000 7f9339c00640 1 -- 172.21.15.6:0/2677871655 wait complete. 2024-03-06T07:32:13.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:32:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.378+0000 7f4dc2c00640 1 Processor -- start 2024-03-06T07:32:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.378+0000 7f4dc2c00640 1 -- start start 2024-03-06T07:32:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.379+0000 7f4dc2c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 0x7f4dbc0d4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.379+0000 7f4dc2c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc0d1d90 con 0x7f4dbc0d1920 2024-03-06T07:32:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.379+0000 7f4dc2c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc0cf4a0 con 0x7f4dbc15e220 2024-03-06T07:32:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.379+0000 7f4dc2c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0b36b0 con 0x7f4dbc0d26f0 2024-03-06T07:32:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.380+0000 7f4dc2200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4dbc0d1920 0x7f4dbc0b3e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41682/0 (socket says 172.21.15.6:41682) 2024-03-06T07:32:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.380+0000 7f4dc2200640 1 -- 172.21.15.6:0/3431258843 learned_addr learned my addr 172.21.15.6:0/3431258843 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.380+0000 7f4dc0e00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 0x7f4dbc0d4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.380+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169712928 0 0) 0x7f4dbc0d1d90 con 0x7f4dbc0d1920 2024-03-06T07:32:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.380+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da0003520 con 0x7f4dbc0d1920 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203509700 0 0) 0x7f4da0003520 con 0x7f4dbc0d1920 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 msgr2=0x7f4dbc0d4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 0x7f4dbc0d4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 >> v1:172.21.15.178:6789/0 conn(0x7f4dbc15e220 legacy=0x7f4dbc0d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc0cf780 con 0x7f4dbc0d1920 2024-03-06T07:32:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f4db4001670 con 0x7f4dbc0d1920 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 >> v1:172.21.15.6:6789/0 conn(0x7f4dbc0d1920 legacy=0x7f4dbc0b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 0x7f4da00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4da0003f20 0x7f4da0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4da0004d40 0x7f4da00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0b36b0 con 0x7f4da0004d40 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da000bad0 con 0x7f4da0003f20 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da000bcd0 con 0x7f4dbc0d26f0 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.381+0000 7f4dc2200640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4da0004d40 0x7f4da00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc1800640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 0x7f4da00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc0e00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4da0003f20 0x7f4da0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc2200640 1 -- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 msgr2=0x7f4da00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc2200640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc0d26f0 0x7f4da00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc2200640 1 -- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4da0003f20 msgr2=0x7f4da0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc2200640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4da0003f20 0x7f4da0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.382+0000 7f4dc2200640 1 -- 172.21.15.6:0/3431258843 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0003520 con 0x7f4da0004d40 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.383+0000 7f4dc2200640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4da0004d40 0x7f4da00051c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c190 tx=0x7f4da0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9966ab036132a268 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.383+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4007cc0 con 0x7f4da0004d40 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.383+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4030070 con 0x7f4da0004d40 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.383+0000 7f4dbbe00640 1 -- 172.21.15.6:0/3431258843 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4005d50 con 0x7f4da0004d40 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2c00640 1 -- 172.21.15.6:0/3431258843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4da0004d40 msgr2=0x7f4da00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/3431258843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4da0004d40 0x7f4da00051c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c190 tx=0x7f4da0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2200640 1 -- 172.21.15.6:0/3431258843 reap_dead start 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2c00640 1 -- 172.21.15.6:0/3431258843 shutdown_connections 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2c00640 1 -- 172.21.15.6:0/3431258843 >> 172.21.15.6:0/3431258843 conn(0x7f4dbc0d3d50 msgr2=0x7f4dbc0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2c00640 1 -- 172.21.15.6:0/3431258843 shutdown_connections 2024-03-06T07:32:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.384+0000 7f4dc2c00640 1 -- 172.21.15.6:0/3431258843 wait complete. 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.386+0000 7f4dc2c00640 1 Processor -- start 2024-03-06T07:32:13.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.386+0000 7f4dc2c00640 1 -- start start 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4dbc14e200 0x7f4dbc0ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc1573e0 0x7f4dbc1595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 0x7f4dbc0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0d4870 con 0x7f4dbc159ae0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0b3900 con 0x7f4dbc14e200 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0d49f0 con 0x7f4dbc1573e0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 0x7f4dbc0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 0x7f4dbc0cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53440/0 (socket says 172.21.15.6:53440) 2024-03-06T07:32:13.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.387+0000 7f4dc2200640 1 -- 172.21.15.6:0/141605069 learned_addr learned my addr 172.21.15.6:0/141605069 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc1800640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4dbc14e200 0x7f4dbc0ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc0e00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc1573e0 0x7f4dbc1595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2200640 1 -- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc1573e0 msgr2=0x7f4dbc1595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2200640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc1573e0 0x7f4dbc1595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2200640 1 -- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4dbc14e200 msgr2=0x7f4dbc0ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2200640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4dbc14e200 0x7f4dbc0ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2200640 1 -- 172.21.15.6:0/141605069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc15a420 con 0x7f4dbc159ae0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2200640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 0x7f4dbc0cbe70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c190 tx=0x7f4db40019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4007cc0 con 0x7f4dbc159ae0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db402cbd0 con 0x7f4dbc159ae0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db4004e30 con 0x7f4dbc159ae0 2024-03-06T07:32:13.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.388+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc15a700 con 0x7f4dbc159ae0 2024-03-06T07:32:13.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.389+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc15acc0 con 0x7f4dbc159ae0 2024-03-06T07:32:13.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.390+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc15af00 con 0x7f4dbc159ae0 2024-03-06T07:32:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.391+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4db4007070 con 0x7f4dbc159ae0 2024-03-06T07:32:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.392+0000 7f4db9600640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.392+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f4db40ab4f0 con 0x7f4dbc159ae0 2024-03-06T07:32:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.393+0000 7f4dc1800640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.394+0000 7f4dc1800640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4dac005240 tx=0x7f4dac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.396+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dbc15af00 con 0x7f4dbc159ae0 2024-03-06T07:32:13.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:13.597+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4dbc04f330 con 0x7f4dbc159ae0 2024-03-06T07:32:14.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.135+0000 7f4db9600640 1 -- 172.21.15.6:0/141605069 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4db407a530 con 0x7f4dbc159ae0 2024-03-06T07:32:14.139 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:32:14.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.139+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b4d0 msgr2=0x7f4d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.139+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4dac005240 tx=0x7f4dac009040 comp rx=0 tx=0).stop 2024-03-06T07:32:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.139+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 msgr2=0x7f4dbc0cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.139+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 0x7f4dbc0cbe70 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c190 tx=0x7f4db40019c0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.140+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 shutdown_connections 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.140+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b4d0 0x7f4d9c06d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.140+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dbc159ae0 0x7f4dbc0cbe70 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.140+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4dbc1573e0 0x7f4dbc1595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.140+0000 7f4dc2c00640 1 --2- 172.21.15.6:0/141605069 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4dbc14e200 0x7f4dbc0ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.140+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 >> 172.21.15.6:0/141605069 conn(0x7f4dbc0d3d50 msgr2=0x7f4dbc06d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.141+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 shutdown_connections 2024-03-06T07:32:14.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.141+0000 7f4dc2c00640 1 -- 172.21.15.6:0/141605069 wait complete. 2024-03-06T07:32:14.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:32:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.367+0000 7f9c70a00640 1 Processor -- start 2024-03-06T07:32:14.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.368+0000 7f9c70a00640 1 -- start start 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.369+0000 7f9c70a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c6c15e680 0x7f9c6c06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.369+0000 7f9c70a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c0ccad0 0x7f9c6c0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.369+0000 7f9c70a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c15eb80 con 0x7f9c6c06da40 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.369+0000 7f9c70a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6c06ea50 con 0x7f9c6c15e680 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.369+0000 7f9c70a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c161c70 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9c6c06da40 0x7f9c6c06e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41698/0 (socket says 172.21.15.6:41698) 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69e00640 1 -- 172.21.15.6:0/2802100080 learned_addr learned my addr 172.21.15.6:0/2802100080 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69400640 1 --2- 172.21.15.6:0/2802100080 >> v2:172.21.15.6:3300/0 conn(0x7f9c6c15e680 0x7f9c6c06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69400640 1 -- 172.21.15.6:0/2802100080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c0ccad0 msgr2=0x7f9c6c0ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c6a800640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c0ccad0 0x7f9c6c0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c68a00640 1 -- 172.21.15.6:0/2802100080 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855173595 0 0) 0x7f9c6c15eb80 con 0x7f9c6c06da40 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69400640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c0ccad0 0x7f9c6c0ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69400640 1 -- 172.21.15.6:0/2802100080 >> v1:172.21.15.6:6789/0 conn(0x7f9c6c06da40 legacy=0x7f9c6c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c69400640 1 -- 172.21.15.6:0/2802100080 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c15ee60 con 0x7f9c6c15e680 2024-03-06T07:32:14.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.370+0000 7f9c6a800640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c0ccad0 0x7f9c6c0ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.372+0000 7f9c69400640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c15e680 0x7f9c6c06c980 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009d90 tx=0x7f9c5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179050e3a91c67bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.372+0000 7f9c68a00640 1 -- 172.21.15.6:0/2802100080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c039070 con 0x7f9c6c15e680 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.372+0000 7f9c68a00640 1 -- 172.21.15.6:0/2802100080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c02cc60 con 0x7f9c6c15e680 2024-03-06T07:32:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.372+0000 7f9c68a00640 1 -- 172.21.15.6:0/2802100080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c034660 con 0x7f9c6c15e680 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 -- 172.21.15.6:0/2802100080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c15e680 msgr2=0x7f9c6c06c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c15e680 0x7f9c6c06c980 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009d90 tx=0x7f9c5c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 -- 172.21.15.6:0/2802100080 shutdown_connections 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c0ccad0 0x7f9c6c0ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2802100080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c15e680 0x7f9c6c06c980 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 -- 172.21.15.6:0/2802100080 >> 172.21.15.6:0/2802100080 conn(0x7f9c6c06b3d0 msgr2=0x7f9c6c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 -- 172.21.15.6:0/2802100080 shutdown_connections 2024-03-06T07:32:14.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.373+0000 7f9c70a00640 1 -- 172.21.15.6:0/2802100080 wait complete. 2024-03-06T07:32:14.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.375+0000 7f9c70a00640 1 Processor -- start 2024-03-06T07:32:14.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.375+0000 7f9c70a00640 1 -- start start 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c70a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c06da40 0x7f9c6c1f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c70a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 0x7f9c6c14a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c70a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c6c15e680 0x7f9c6c0b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c70a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c06cec0 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c70a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c15f420 con 0x7f9c6c15e680 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c70a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c06d040 con 0x7f9c6c06da40 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 0x7f9c6c14a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 0x7f9c6c14a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53460/0 (socket says 172.21.15.6:53460) 2024-03-06T07:32:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 -- 172.21.15.6:0/2609354042 learned_addr learned my addr 172.21.15.6:0/2609354042 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:14.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69e00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c06da40 0x7f9c6c1f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c6a800640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c6c15e680 0x7f9c6c0b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 -- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c06da40 msgr2=0x7f9c6c1f7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c06da40 0x7f9c6c1f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 -- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c6c15e680 msgr2=0x7f9c6c0b3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.376+0000 7f9c69400640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c6c15e680 0x7f9c6c0b3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c69400640 1 -- 172.21.15.6:0/2609354042 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c15c7d0 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c69400640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 0x7f9c6c14a4c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c037040 tx=0x7f9c5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c039050 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c03a040 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c15cab0 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c03e610 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.377+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c0b2ef0 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.378+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c04f330 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.380+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c5c047020 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.381+0000 7f9c62000640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c4806b4d0 0x7f9c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.381+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9c5c0acf70 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.381+0000 7f9c69e00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c4806b4d0 0x7f9c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.383+0000 7f9c69e00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c4806b4d0 0x7f9c4806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c54007fd0 tx=0x7f9c54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.385+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c5c04e020 con 0x7f9c6c0ccad0 2024-03-06T07:32:14.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:14.627+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9c6c0745e0 con 0x7f9c6c0ccad0 2024-03-06T07:32:15.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.181+0000 7f9c62000640 1 -- 172.21.15.6:0/2609354042 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v49) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9c5c07bec0 con 0x7f9c6c0ccad0 2024-03-06T07:32:15.184 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.186+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c4806b4d0 msgr2=0x7f9c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.186+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c4806b4d0 0x7f9c4806d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c54007fd0 tx=0x7f9c54009040 comp rx=0 tx=0).stop 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.186+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 msgr2=0x7f9c6c14a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.186+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 0x7f9c6c14a4c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c037040 tx=0x7f9c5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.187+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 shutdown_connections 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.187+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9c4806b4d0 0x7f9c4806d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.187+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c6c15e680 0x7f9c6c0b3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.187+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c6c0ccad0 0x7f9c6c14a4c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.187+0000 7f9c70a00640 1 --2- 172.21.15.6:0/2609354042 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c6c06da40 0x7f9c6c1f7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.187+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 >> 172.21.15.6:0/2609354042 conn(0x7f9c6c06b3d0 msgr2=0x7f9c6c15b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.188+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 shutdown_connections 2024-03-06T07:32:15.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.189+0000 7f9c70a00640 1 -- 172.21.15.6:0/2609354042 wait complete. 2024-03-06T07:32:15.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.404+0000 7fb6c4800640 1 Processor -- start 2024-03-06T07:32:15.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.404+0000 7fb6c4800640 1 -- start start 2024-03-06T07:32:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c4800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb6bc072da0 0x7fb6bc162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c4800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 0x7fb6bc06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c4800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6bc0ce090 con 0x7fb6bc070ee0 2024-03-06T07:32:15.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c4800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6bc0ceca0 con 0x7fb6bc072da0 2024-03-06T07:32:15.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c4800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0ce370 con 0x7fb6bc0712c0 2024-03-06T07:32:15.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb6bc070ee0 0x7fb6bc06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41710/0 (socket says 172.21.15.6:41710) 2024-03-06T07:32:15.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.406+0000 7fb6c1e00640 1 -- 172.21.15.6:0/4227741603 learned_addr learned my addr 172.21.15.6:0/4227741603 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c2800640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 0x7fb6bc06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c1400640 1 --2- 172.21.15.6:0/4227741603 >> v2:172.21.15.178:3300/0 conn(0x7fb6bc072da0 0x7fb6bc162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250305965 0 0) 0x7fb6bc0ce090 con 0x7fb6bc070ee0 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a8003520 con 0x7fb6bc070ee0 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3791827874 0 0) 0x7fb6a8003520 con 0x7fb6bc070ee0 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 msgr2=0x7fb6bc06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 0x7fb6bc06e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 >> v2:172.21.15.178:3300/0 conn(0x7fb6bc072da0 msgr2=0x7fb6bc162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 --2- 172.21.15.6:0/4227741603 >> v2:172.21.15.178:3300/0 conn(0x7fb6bc072da0 0x7fb6bc162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc0ce650 con 0x7fb6bc070ee0 2024-03-06T07:32:15.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb6b40019f0 con 0x7fb6bc070ee0 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 >> v1:172.21.15.6:6789/0 conn(0x7fb6bc070ee0 legacy=0x7fb6bc06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 0x7fb6a8003a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.407+0000 7fb6c0a00640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6a8004010 0x7fb6a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c0a00640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0ceca0 con 0x7fb6a8004010 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0ce370 con 0x7fb6a8004d40 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a800ba40 con 0x7fb6bc0712c0 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1400640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6a8004010 0x7fb6a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1e00640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 0x7fb6a8003a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c2800640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1400640 1 -- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 msgr2=0x7fb6a8003a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1400640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc0712c0 0x7fb6a8003a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1400640 1 -- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004d40 msgr2=0x7fb6a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1400640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c1400640 1 -- 172.21.15.6:0/4227741603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8003520 con 0x7fb6a8004010 2024-03-06T07:32:15.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.408+0000 7fb6c2800640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.409+0000 7fb6c1400640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6a8004010 0x7fb6a8004800 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009ac0 tx=0x7fb6a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4ed828d168dd29 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac039070 con 0x7fb6a8004010 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ac02c710 con 0x7fb6a8004010 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c0a00640 1 -- 172.21.15.6:0/4227741603 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac009690 con 0x7fb6a8004010 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c4800640 1 -- 172.21.15.6:0/4227741603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6a8004010 msgr2=0x7fb6a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c4800640 1 --2- 172.21.15.6:0/4227741603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6a8004010 0x7fb6a8004800 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac009ac0 tx=0x7fb6a8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c2800640 1 -- 172.21.15.6:0/4227741603 reap_dead start 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c4800640 1 -- 172.21.15.6:0/4227741603 shutdown_connections 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.410+0000 7fb6c4800640 1 -- 172.21.15.6:0/4227741603 >> 172.21.15.6:0/4227741603 conn(0x7fb6bc06afb0 msgr2=0x7fb6bc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.411+0000 7fb6c4800640 1 -- 172.21.15.6:0/4227741603 shutdown_connections 2024-03-06T07:32:15.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.411+0000 7fb6c4800640 1 -- 172.21.15.6:0/4227741603 wait complete. 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.412+0000 7fb6c4800640 1 Processor -- start 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.412+0000 7fb6c4800640 1 -- start start 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.413+0000 7fb6c4800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004010 0x7fb6bc15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.413+0000 7fb6c4800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc1fcc50 0x7fb6bc15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.413+0000 7fb6c4800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 0x7fb6bc0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.413+0000 7fb6c4800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0cdda0 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.413+0000 7fb6c4800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0cdc20 con 0x7fb6a8004010 2024-03-06T07:32:15.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.413+0000 7fb6c4800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc06e7d0 con 0x7fb6bc1fcc50 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 0x7fb6bc0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 0x7fb6bc0b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53486/0 (socket says 172.21.15.6:53486) 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 -- 172.21.15.6:0/53466736 learned_addr learned my addr 172.21.15.6:0/53466736 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 -- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc1fcc50 msgr2=0x7fb6bc15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc1fcc50 0x7fb6bc15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 -- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004010 msgr2=0x7fb6bc15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004010 0x7fb6bc15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 -- 172.21.15.6:0/53466736 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc15aa20 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6c2800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 0x7fb6bc0b4020 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc073180 tx=0x7fb6b8010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800f040 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b800e840 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.414+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b800ec10 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.415+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc1fc170 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.415+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc1fc730 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.416+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6bc069510 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.417+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6b80143f0 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.418+0000 7fb6b2000640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6a006b4d0 0x7fb6a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.418+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6b808a3f0 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.418+0000 7fb6c1e00640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6a006b4d0 0x7fb6a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.419+0000 7fb6c1e00640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6a006b4d0 0x7fb6a006d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4001730 tx=0x7fb6b4001250 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.422+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6b8018030 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.619+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6bc1fc930 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.622+0000 7fb6b2000640 1 -- 172.21.15.6:0/53466736 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16601 (secure 0 0 0) 0x7fb6bc1fc930 con 0x7fb6bc0b3ba0 2024-03-06T07:32:15.625 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:15.625 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":80,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:14.629262+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6a006b4d0 msgr2=0x7fb6a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6a006b4d0 0x7fb6a006d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4001730 tx=0x7fb6b4001250 comp rx=0 tx=0).stop 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 msgr2=0x7fb6bc0b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 0x7fb6bc0b4020 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc073180 tx=0x7fb6b8010c20 comp rx=0 tx=0).stop 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 shutdown_connections 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6a006b4d0 0x7fb6a006d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb6bc0b3ba0 0x7fb6bc0b4020 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb6bc1fcc50 0x7fb6bc15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 --2- 172.21.15.6:0/53466736 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6a8004010 0x7fb6bc15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 >> 172.21.15.6:0/53466736 conn(0x7fb6bc06afb0 msgr2=0x7fb6bc06dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 shutdown_connections 2024-03-06T07:32:15.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.625+0000 7fb6c4800640 1 -- 172.21.15.6:0/53466736 wait complete. 2024-03-06T07:32:15.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:15.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.865+0000 7f820f000640 1 Processor -- start 2024-03-06T07:32:15.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.866+0000 7f820f000640 1 -- start start 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.867+0000 7f820f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8208070f10 0x7f82080cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.867+0000 7f820f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82080cce90 0x7f820815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.867+0000 7f820f000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82080714d0 con 0x7f8208145300 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.867+0000 7f820f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82080cd300 con 0x7f8208070f10 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.867+0000 7f820f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82080717b0 con 0x7f82080cce90 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820dc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8208145300 0x7f82080b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41726/0 (socket says 172.21.15.6:41726) 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820dc00640 1 -- 172.21.15.6:0/3571498478 learned_addr learned my addr 172.21.15.6:0/3571498478 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:15.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820d200640 1 --2- 172.21.15.6:0/3571498478 >> v2:172.21.15.6:3300/0 conn(0x7f8208070f10 0x7f82080cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f8207e00640 1 -- 172.21.15.6:0/3571498478 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476816158 0 0) 0x7f82080714d0 con 0x7f8208145300 2024-03-06T07:32:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f8207e00640 1 -- 172.21.15.6:0/3571498478 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0003520 con 0x7f8208145300 2024-03-06T07:32:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820e600640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82080cce90 0x7f820815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820d200640 1 -- 172.21.15.6:0/3571498478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82080cce90 msgr2=0x7f820815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820d200640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82080cce90 0x7f820815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820d200640 1 -- 172.21.15.6:0/3571498478 >> v1:172.21.15.6:6789/0 conn(0x7f8208145300 legacy=0x7f82080b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820d200640 1 -- 172.21.15.6:0/3571498478 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8208071a90 con 0x7f8208070f10 2024-03-06T07:32:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.868+0000 7f820e600640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82080cce90 0x7f820815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.870+0000 7f820d200640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8208070f10 0x7f82080cc950 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009d90 tx=0x7f81f802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1673e7e3cd2cd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.870+0000 7f8207e00640 1 -- 172.21.15.6:0/3571498478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8039070 con 0x7f8208070f10 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.870+0000 7f8207e00640 1 -- 172.21.15.6:0/3571498478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f802cbc0 con 0x7f8208070f10 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.870+0000 7f8207e00640 1 -- 172.21.15.6:0/3571498478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8034440 con 0x7f8208070f10 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 -- 172.21.15.6:0/3571498478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8208070f10 msgr2=0x7f82080cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8208070f10 0x7f82080cc950 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009d90 tx=0x7f81f802be10 comp rx=0 tx=0).stop 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 -- 172.21.15.6:0/3571498478 shutdown_connections 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f82080cce90 0x7f820815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 --2- 172.21.15.6:0/3571498478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8208070f10 0x7f82080cc950 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 -- 172.21.15.6:0/3571498478 >> 172.21.15.6:0/3571498478 conn(0x7f820806b3d0 msgr2=0x7f820806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 -- 172.21.15.6:0/3571498478 shutdown_connections 2024-03-06T07:32:15.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.871+0000 7f820f000640 1 -- 172.21.15.6:0/3571498478 wait complete. 2024-03-06T07:32:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.873+0000 7f820f000640 1 Processor -- start 2024-03-06T07:32:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.873+0000 7f820f000640 1 -- start start 2024-03-06T07:32:15.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.874+0000 7f820f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8208070f10 0x7f820813dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.874+0000 7f820f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 0x7f820813f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.874+0000 7f820f000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8208145300 0x7f8208143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.874+0000 7f820f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820815cab0 con 0x7f82080cce90 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.874+0000 7f820f000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82081456e0 con 0x7f8208145300 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.874+0000 7f820f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820815cc30 con 0x7f8208070f10 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 0x7f820813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 0x7f820813f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53498/0 (socket says 172.21.15.6:53498) 2024-03-06T07:32:15.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 -- 172.21.15.6:0/1834339644 learned_addr learned my addr 172.21.15.6:0/1834339644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:15.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820dc00640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8208070f10 0x7f820813dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820e600640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8208145300 0x7f8208143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 -- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8208070f10 msgr2=0x7f820813dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8208070f10 0x7f820813dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 -- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8208145300 msgr2=0x7f8208143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8208145300 0x7f8208143310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820d200640 1 -- 172.21.15.6:0/1834339644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820813e5d0 con 0x7f82080cce90 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.875+0000 7f820dc00640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8208070f10 0x7f820813dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.876+0000 7f820d200640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 0x7f820813f670 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f81f8037040 tx=0x7f81f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.876+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8039050 con 0x7f82080cce90 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.876+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f8045070 con 0x7f82080cce90 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.876+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820813e8b0 con 0x7f82080cce90 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.876+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f8009450 con 0x7f82080cce90 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.876+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820813ee70 con 0x7f82080cce90 2024-03-06T07:32:15.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.877+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820804f330 con 0x7f82080cce90 2024-03-06T07:32:15.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.878+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81f803e850 con 0x7f82080cce90 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.879+0000 7f8205600640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81ec06b520 0x7f81ec06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:15.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.879+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81f80acf40 con 0x7f82080cce90 2024-03-06T07:32:15.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.880+0000 7f820dc00640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81ec06b520 0x7f81ec06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:15.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.881+0000 7f820dc00640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81ec06b520 0x7f81ec06d9e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f81fc005880 tx=0x7f81fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:15.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:15.884+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81f8032e20 con 0x7f82080cce90 2024-03-06T07:32:16.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.083+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82080cbd70 con 0x7f82080cce90 2024-03-06T07:32:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.086+0000 7f8205600640 1 -- 172.21.15.6:0/1834339644 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16601 (secure 0 0 0) 0x7f81f807bec0 con 0x7f82080cce90 2024-03-06T07:32:16.088 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:16.088 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":80,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:14.629262+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81ec06b520 msgr2=0x7f81ec06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81ec06b520 0x7f81ec06d9e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f81fc005880 tx=0x7f81fc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 msgr2=0x7f820813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 0x7f820813f670 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f81f8037040 tx=0x7f81f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 shutdown_connections 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81ec06b520 0x7f81ec06d9e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8208145300 0x7f8208143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.088+0000 7f820f000640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f82080cce90 0x7f820813f670 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.089+0000 7f820f000640 1 --2- 172.21.15.6:0/1834339644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8208070f10 0x7f820813dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.089+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 >> 172.21.15.6:0/1834339644 conn(0x7f820806b3d0 msgr2=0x7f8208072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.089+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 shutdown_connections 2024-03-06T07:32:16.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.089+0000 7f820f000640 1 -- 172.21.15.6:0/1834339644 wait complete. 2024-03-06T07:32:16.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:32:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.332+0000 7fb882000640 1 Processor -- start 2024-03-06T07:32:16.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.332+0000 7fb882000640 1 -- start start 2024-03-06T07:32:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb882000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb87c145430 0x7fb87c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb882000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0b2250 0x7fb87c0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb882000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c15fe40 con 0x7fb87c15cc90 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb882000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c160120 con 0x7fb87c145430 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb882000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c160400 con 0x7fb87c0b2250 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb880c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb87c145430 0x7fb87c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb880c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb87c145430 0x7fb87c15f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53500/0 (socket says 172.21.15.6:53500) 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.334+0000 7fb880c00640 1 -- 172.21.15.6:0/3938596096 learned_addr learned my addr 172.21.15.6:0/3938596096 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.335+0000 7fb881600640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0b2250 0x7fb87c0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.335+0000 7fb880c00640 1 -- 172.21.15.6:0/3938596096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0b2250 msgr2=0x7fb87c0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.335+0000 7fb880c00640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0b2250 0x7fb87c0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.335+0000 7fb880c00640 1 -- 172.21.15.6:0/3938596096 >> v1:172.21.15.6:6789/0 conn(0x7fb87c15cc90 legacy=0x7fb87c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.335+0000 7fb880c00640 1 -- 172.21.15.6:0/3938596096 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c1606e0 con 0x7fb87c145430 2024-03-06T07:32:16.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.335+0000 7fb881600640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0b2250 0x7fb87c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:16.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.336+0000 7fb880c00640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c145430 0x7fb87c15f810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009e50 tx=0x7fb86c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8736328fd546da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.336+0000 7fb87be00640 1 -- 172.21.15.6:0/3938596096 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c02cb00 con 0x7fb87c145430 2024-03-06T07:32:16.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb87be00640 1 -- 172.21.15.6:0/3938596096 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c02cca0 con 0x7fb87c145430 2024-03-06T07:32:16.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb87be00640 1 -- 172.21.15.6:0/3938596096 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c034610 con 0x7fb87c145430 2024-03-06T07:32:16.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 -- 172.21.15.6:0/3938596096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c145430 msgr2=0x7fb87c15f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c145430 0x7fb87c15f810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fb86c009e50 tx=0x7fb86c02bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 -- 172.21.15.6:0/3938596096 shutdown_connections 2024-03-06T07:32:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0b2250 0x7fb87c0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 --2- 172.21.15.6:0/3938596096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c145430 0x7fb87c15f810 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 -- 172.21.15.6:0/3938596096 >> 172.21.15.6:0/3938596096 conn(0x7fb87c06b3a0 msgr2=0x7fb87c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 -- 172.21.15.6:0/3938596096 shutdown_connections 2024-03-06T07:32:16.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.337+0000 7fb882000640 1 -- 172.21.15.6:0/3938596096 wait complete. 2024-03-06T07:32:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.339+0000 7fb882000640 1 Processor -- start 2024-03-06T07:32:16.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.339+0000 7fb882000640 1 -- start start 2024-03-06T07:32:16.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb882000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 0x7fb87c134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb882000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c145430 0x7fb87c1421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb882000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c15cc90 0x7fb87c143c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb882000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c145990 con 0x7fb87c0b2250 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb882000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c145810 con 0x7fb87c15cc90 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb882000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c0cde00 con 0x7fb87c145430 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb880c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 0x7fb87c134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb880c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 0x7fb87c134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53512/0 (socket says 172.21.15.6:53512) 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.340+0000 7fb880c00640 1 -- 172.21.15.6:0/2200085978 learned_addr learned my addr 172.21.15.6:0/2200085978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb880c00640 1 -- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c145430 msgr2=0x7fb87c1421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb880c00640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c145430 0x7fb87c1421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb880c00640 1 -- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c15cc90 msgr2=0x7fb87c143c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb880c00640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c15cc90 0x7fb87c143c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb880c00640 1 -- 172.21.15.6:0/2200085978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c13d530 con 0x7fb87c0b2250 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb880c00640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 0x7fb87c134c10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb86c000f80 tx=0x7fb86c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.341+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c043020 con 0x7fb87c0b2250 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.342+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c02ce50 con 0x7fb87c0b2250 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.342+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb86c03e690 con 0x7fb87c0b2250 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.342+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c13c920 con 0x7fb87c0b2250 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.342+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c13cee0 con 0x7fb87c0b2250 2024-03-06T07:32:16.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.343+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c04f330 con 0x7fb87c0b2250 2024-03-06T07:32:16.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.344+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb86c046020 con 0x7fb87c0b2250 2024-03-06T07:32:16.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.345+0000 7fb879600640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.345+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb86c0ac920 con 0x7fb87c0b2250 2024-03-06T07:32:16.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.345+0000 7fb873e00640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.347+0000 7fb873e00640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85806b4d0 0x7fb85806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb864000f60 tx=0x7fb8640079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.350+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb86c04d030 con 0x7fb87c0b2250 2024-03-06T07:32:16.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:16.590+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb87c06e240 con 0x7fb87c0b2250 2024-03-06T07:32:17.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.192+0000 7fb879600640 1 -- 172.21.15.6:0/2200085978 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v52) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb86c07b950 con 0x7fb87c0b2250 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.196+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85806b4d0 msgr2=0x7fb85806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.196+0000 7fb882000640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85806b4d0 0x7fb85806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb864000f60 tx=0x7fb8640079d0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.196+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 msgr2=0x7fb87c134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.196+0000 7fb882000640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 0x7fb87c134c10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb86c000f80 tx=0x7fb86c004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 shutdown_connections 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85806b4d0 0x7fb85806d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c15cc90 0x7fb87c143c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c145430 0x7fb87c1421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 --2- 172.21.15.6:0/2200085978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0b2250 0x7fb87c134c10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 >> 172.21.15.6:0/2200085978 conn(0x7fb87c06b3a0 msgr2=0x7fb87c15d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 shutdown_connections 2024-03-06T07:32:17.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.197+0000 7fb882000640 1 -- 172.21.15.6:0/2200085978 wait complete. 2024-03-06T07:32:17.220 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:32:17.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:32:17.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.433+0000 7f5778c00640 1 Processor -- start 2024-03-06T07:32:17.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.433+0000 7f5778c00640 1 -- start start 2024-03-06T07:32:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5778c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f577415bf20 0x7f57740acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5778c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5774072520 0x7f57740cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5778c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5774072df0 con 0x7f5774071e20 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5778c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57740cdab0 con 0x7f577415bf20 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5778c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57740b4d70 con 0x7f5774072520 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5771e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f577415bf20 0x7f57740acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5771e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f577415bf20 0x7f57740acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53516/0 (socket says 172.21.15.6:53516) 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.435+0000 7f5771e00640 1 -- 172.21.15.6:0/309780796 learned_addr learned my addr 172.21.15.6:0/309780796 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.436+0000 7f5771400640 1 --2- 172.21.15.6:0/309780796 >> v2:172.21.15.178:3300/0 conn(0x7f5774072520 0x7f57740cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.436+0000 7f5771e00640 1 -- 172.21.15.6:0/309780796 >> v2:172.21.15.178:3300/0 conn(0x7f5774072520 msgr2=0x7f57740cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.436+0000 7f5771e00640 1 --2- 172.21.15.6:0/309780796 >> v2:172.21.15.178:3300/0 conn(0x7f5774072520 0x7f57740cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.436+0000 7f5771e00640 1 -- 172.21.15.6:0/309780796 >> v1:172.21.15.178:6789/0 conn(0x7f5774071e20 legacy=0x7f57740ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.436+0000 7f5771e00640 1 -- 172.21.15.6:0/309780796 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740b5050 con 0x7f577415bf20 2024-03-06T07:32:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.437+0000 7f5771e00640 1 --2- 172.21.15.6:0/309780796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577415bf20 0x7f57740acd40 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5774070b90 tx=0x7f576402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7153bc1d9b793584 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.437+0000 7f5770a00640 1 -- 172.21.15.6:0/309780796 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764034070 con 0x7f577415bf20 2024-03-06T07:32:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.437+0000 7f5770a00640 1 -- 172.21.15.6:0/309780796 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5764030860 con 0x7f577415bf20 2024-03-06T07:32:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.437+0000 7f5770a00640 1 -- 172.21.15.6:0/309780796 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764030c30 con 0x7f577415bf20 2024-03-06T07:32:17.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 -- 172.21.15.6:0/309780796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577415bf20 msgr2=0x7f57740acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 --2- 172.21.15.6:0/309780796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577415bf20 0x7f57740acd40 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5774070b90 tx=0x7f576402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 -- 172.21.15.6:0/309780796 shutdown_connections 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 --2- 172.21.15.6:0/309780796 >> v2:172.21.15.178:3300/0 conn(0x7f5774072520 0x7f57740cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 --2- 172.21.15.6:0/309780796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f577415bf20 0x7f57740acd40 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 -- 172.21.15.6:0/309780796 >> 172.21.15.6:0/309780796 conn(0x7f577406b3d0 msgr2=0x7f577406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 -- 172.21.15.6:0/309780796 shutdown_connections 2024-03-06T07:32:17.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.438+0000 7f5778c00640 1 -- 172.21.15.6:0/309780796 wait complete. 2024-03-06T07:32:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.440+0000 7f5778c00640 1 Processor -- start 2024-03-06T07:32:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.440+0000 7f5778c00640 1 -- start start 2024-03-06T07:32:17.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5778c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5774071e20 0x7f5774149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5778c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 0x7f57740b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5778c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f577415bf20 0x7f57740b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5778c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740b4b10 con 0x7f5774072520 2024-03-06T07:32:17.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5778c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740b4990 con 0x7f5774071e20 2024-03-06T07:32:17.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5778c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57740cd1e0 con 0x7f577415bf20 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5771400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 0x7f57740b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5771400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 0x7f57740b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53520/0 (socket says 172.21.15.6:53520) 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.441+0000 7f5771400640 1 -- 172.21.15.6:0/2652220636 learned_addr learned my addr 172.21.15.6:0/2652220636 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771e00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5774071e20 0x7f5774149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5772800640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f577415bf20 0x7f57740b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771400640 1 -- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f577415bf20 msgr2=0x7f57740b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771400640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f577415bf20 0x7f57740b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771400640 1 -- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5774071e20 msgr2=0x7f5774149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771400640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5774071e20 0x7f5774149740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:17.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771400640 1 -- 172.21.15.6:0/2652220636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740b0c30 con 0x7f5774072520 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5772800640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f577415bf20 0x7f57740b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.442+0000 7f5771400640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 0x7f57740b01e0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f576000d800 tx=0x7f576000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.443+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000b6c0 con 0x7f5774072520 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.443+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5760019070 con 0x7f5774072520 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.443+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000ba80 con 0x7f5774072520 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.443+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57740b0950 con 0x7f5774072520 2024-03-06T07:32:17.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.443+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57741534c0 con 0x7f5774072520 2024-03-06T07:32:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.444+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577404f330 con 0x7f5774072520 2024-03-06T07:32:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.445+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f576000bc20 con 0x7f5774072520 2024-03-06T07:32:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.446+0000 7f576a000640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f575406b4d0 0x7f575406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:17.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.447+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f576008a6c0 con 0x7f5774072520 2024-03-06T07:32:17.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.447+0000 7f5771e00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f575406b4d0 0x7f575406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:17.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.448+0000 7f5771e00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f575406b4d0 0x7f575406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5764036000 tx=0x7f576403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:17.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.451+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57740b0c30 con 0x7f5774072520 2024-03-06T07:32:17.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:17.692+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f577415f720 con 0x7f5774072520 2024-03-06T07:32:18.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.196+0000 7f576a000640 1 -- 172.21.15.6:0/2652220636 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v53) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5760059530 con 0x7f5774072520 2024-03-06T07:32:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f575406b4d0 msgr2=0x7f575406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f575406b4d0 0x7f575406d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5764036000 tx=0x7f576403b040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 msgr2=0x7f57740b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 0x7f57740b01e0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f576000d800 tx=0x7f576000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 shutdown_connections 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f575406b4d0 0x7f575406d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f577415bf20 0x7f57740b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5774072520 0x7f57740b01e0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 --2- 172.21.15.6:0/2652220636 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5774071e20 0x7f5774149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 >> 172.21.15.6:0/2652220636 conn(0x7f577406b3d0 msgr2=0x7f577415a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 shutdown_connections 2024-03-06T07:32:18.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.200+0000 7f5778c00640 1 -- 172.21.15.6:0/2652220636 wait complete. 2024-03-06T07:32:18.223 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:32:18.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.442+0000 7f5eb6600640 1 Processor -- start 2024-03-06T07:32:18.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.442+0000 7f5eb6600640 1 -- start start 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.443+0000 7f5eb6600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5eb015e0b0 0x7f5eb00cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.443+0000 7f5eb6600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5eb00d4a10 0x7f5eb00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.443+0000 7f5eb6600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb00d0c80 0x7f5eb00d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.443+0000 7f5eb6600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00b09e0 con 0x7f5eb015e0b0 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.443+0000 7f5eb6600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00b3710 con 0x7f5eb00d4a10 2024-03-06T07:32:18.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.443+0000 7f5eb6600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00b39f0 con 0x7f5eb00d0c80 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5eb015e0b0 0x7f5eb00cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5eb015e0b0 0x7f5eb00cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53526/0 (socket says 172.21.15.6:53526) 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 -- 172.21.15.6:0/870295945 learned_addr learned my addr 172.21.15.6:0/870295945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 -- 172.21.15.6:0/870295945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb00d0c80 msgr2=0x7f5eb00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 --2- 172.21.15.6:0/870295945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb00d0c80 0x7f5eb00d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 -- 172.21.15.6:0/870295945 >> v2:172.21.15.178:3300/0 conn(0x7f5eb00d4a10 msgr2=0x7f5eb00d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 --2- 172.21.15.6:0/870295945 >> v2:172.21.15.178:3300/0 conn(0x7f5eb00d4a10 0x7f5eb00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.444+0000 7f5eb5200640 1 -- 172.21.15.6:0/870295945 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00b3cd0 con 0x7f5eb015e0b0 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.445+0000 7f5eb5200640 1 --2- 172.21.15.6:0/870295945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb015e0b0 0x7f5eb00cf310 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5eb006b8c0 tx=0x7f5ea402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9355bea0f757e036 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.445+0000 7f5eaf400640 1 -- 172.21.15.6:0/870295945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4035020 con 0x7f5eb015e0b0 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.445+0000 7f5eaf400640 1 -- 172.21.15.6:0/870295945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea4031870 con 0x7f5eb015e0b0 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.445+0000 7f5eaf400640 1 -- 172.21.15.6:0/870295945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4031c40 con 0x7f5eb015e0b0 2024-03-06T07:32:18.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 -- 172.21.15.6:0/870295945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb015e0b0 msgr2=0x7f5eb00cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 --2- 172.21.15.6:0/870295945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb015e0b0 0x7f5eb00cf310 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5eb006b8c0 tx=0x7f5ea402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 -- 172.21.15.6:0/870295945 shutdown_connections 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 --2- 172.21.15.6:0/870295945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb00d0c80 0x7f5eb00d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 --2- 172.21.15.6:0/870295945 >> v2:172.21.15.178:3300/0 conn(0x7f5eb00d4a10 0x7f5eb00d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 --2- 172.21.15.6:0/870295945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb015e0b0 0x7f5eb00cf310 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 -- 172.21.15.6:0/870295945 >> 172.21.15.6:0/870295945 conn(0x7f5eb00d3d30 msgr2=0x7f5eb00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 -- 172.21.15.6:0/870295945 shutdown_connections 2024-03-06T07:32:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.446+0000 7f5eb6600640 1 -- 172.21.15.6:0/870295945 wait complete. 2024-03-06T07:32:18.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.449+0000 7f5eb6600640 1 Processor -- start 2024-03-06T07:32:18.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.449+0000 7f5eb6600640 1 -- start start 2024-03-06T07:32:18.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eb6600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5eb00d0c80 0x7f5eb014f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eb6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 0x7f5eb00b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eb6600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb015e0b0 0x7f5eb00b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eb6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00cfc60 con 0x7f5eb00d4a10 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eb6600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb015e490 con 0x7f5eb00d0c80 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eb6600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00cf850 con 0x7f5eb015e0b0 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 0x7f5eb00b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 0x7f5eb00b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53540/0 (socket says 172.21.15.6:53540) 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.450+0000 7f5eafe00640 1 -- 172.21.15.6:0/892376686 learned_addr learned my addr 172.21.15.6:0/892376686 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eb5200640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5eb00d0c80 0x7f5eb014f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eb5c00640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb015e0b0 0x7f5eb00b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eafe00640 1 -- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb015e0b0 msgr2=0x7f5eb00b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eafe00640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb015e0b0 0x7f5eb00b24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eafe00640 1 -- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5eb00d0c80 msgr2=0x7f5eb014f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eafe00640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5eb00d0c80 0x7f5eb014f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eafe00640 1 -- 172.21.15.6:0/892376686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00caea0 con 0x7f5eb00d4a10 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eafe00640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 0x7f5eb00b1fa0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00b450 tx=0x7f5e9c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c015020 con 0x7f5eb00d4a10 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c011890 con 0x7f5eb00d4a10 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c011c80 con 0x7f5eb00d4a10 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.451+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb00cb180 con 0x7f5eb00d4a10 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.452+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb00cb740 con 0x7f5eb00d4a10 2024-03-06T07:32:18.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.453+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb004f330 con 0x7f5eb00d4a10 2024-03-06T07:32:18.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.454+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e9c018020 con 0x7f5eb00d4a10 2024-03-06T07:32:18.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.454+0000 7f5eacc00640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e9006b4d0 0x7f5e9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.455+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e9c08a400 con 0x7f5eb00d4a10 2024-03-06T07:32:18.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.455+0000 7f5eb5200640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e9006b4d0 0x7f5e9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.456+0000 7f5eb5200640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e9006b4d0 0x7f5e9006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4036000 tx=0x7f5ea403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.459+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e9c01f020 con 0x7f5eb00d4a10 2024-03-06T07:32:18.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.688+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eb00b5480 con 0x7f5eb00d4a10 2024-03-06T07:32:18.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.689+0000 7f5eacc00640 1 -- 172.21.15.6:0/892376686 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f5e9c00d040 con 0x7f5eb00d4a10 2024-03-06T07:32:18.691 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:18.691 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:18.691 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:18.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e9006b4d0 msgr2=0x7f5e9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e9006b4d0 0x7f5e9006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4036000 tx=0x7f5ea403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:18.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 msgr2=0x7f5eb00b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 0x7f5eb00b1fa0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00b450 tx=0x7f5e9c00b910 comp rx=0 tx=0).stop 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 shutdown_connections 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e9006b4d0 0x7f5e9006d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5eb015e0b0 0x7f5eb00b24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5eb00d4a10 0x7f5eb00b1fa0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 --2- 172.21.15.6:0/892376686 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5eb00d0c80 0x7f5eb014f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.691+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 >> 172.21.15.6:0/892376686 conn(0x7f5eb00d3d30 msgr2=0x7f5eb00cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.692+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 shutdown_connections 2024-03-06T07:32:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.692+0000 7f5eb6600640 1 -- 172.21.15.6:0/892376686 wait complete. 2024-03-06T07:32:18.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.925+0000 7f3b21e00640 1 Processor -- start 2024-03-06T07:32:18.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.926+0000 7f3b21e00640 1 -- start start 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.927+0000 7f3b21e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3b1c161670 0x7f3b1c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.927+0000 7f3b21e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b1c0ae0a0 0x7f3b1c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.927+0000 7f3b21e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1c0ce9a0 con 0x7f3b1c06e240 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.927+0000 7f3b21e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1c0730d0 con 0x7f3b1c0ae0a0 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.927+0000 7f3b21e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1c0cec80 con 0x7f3b1c161670 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b1c0ae0a0 0x7f3b1c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3b1c0ae0a0 0x7f3b1c0711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53556/0 (socket says 172.21.15.6:53556) 2024-03-06T07:32:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 -- 172.21.15.6:0/73685680 learned_addr learned my addr 172.21.15.6:0/73685680 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1aa00640 1 --2- 172.21.15.6:0/73685680 >> v2:172.21.15.178:3300/0 conn(0x7f3b1c161670 0x7f3b1c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 -- 172.21.15.6:0/73685680 >> v2:172.21.15.178:3300/0 conn(0x7f3b1c161670 msgr2=0x7f3b1c070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 --2- 172.21.15.6:0/73685680 >> v2:172.21.15.178:3300/0 conn(0x7f3b1c161670 0x7f3b1c070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 -- 172.21.15.6:0/73685680 >> v1:172.21.15.178:6789/0 conn(0x7f3b1c06e240 legacy=0x7f3b1c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1b400640 1 -- 172.21.15.6:0/73685680 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0cef60 con 0x7f3b1c0ae0a0 2024-03-06T07:32:18.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.928+0000 7f3b1aa00640 1 --2- 172.21.15.6:0/73685680 >> v2:172.21.15.178:3300/0 conn(0x7f3b1c161670 0x7f3b1c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.930+0000 7f3b1b400640 1 --2- 172.21.15.6:0/73685680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c0ae0a0 0x7f3b1c0711b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c073610 tx=0x7f3b1402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4bbff1c5201bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.930+0000 7f3b1a000640 1 -- 172.21.15.6:0/73685680 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1402e070 con 0x7f3b1c0ae0a0 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.930+0000 7f3b1a000640 1 -- 172.21.15.6:0/73685680 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14037070 con 0x7f3b1c0ae0a0 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.930+0000 7f3b1a000640 1 -- 172.21.15.6:0/73685680 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14031a00 con 0x7f3b1c0ae0a0 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 -- 172.21.15.6:0/73685680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c0ae0a0 msgr2=0x7f3b1c0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 --2- 172.21.15.6:0/73685680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c0ae0a0 0x7f3b1c0711b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c073610 tx=0x7f3b1402b880 comp rx=0 tx=0).stop 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 -- 172.21.15.6:0/73685680 shutdown_connections 2024-03-06T07:32:18.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 --2- 172.21.15.6:0/73685680 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c0ae0a0 0x7f3b1c0711b0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 --2- 172.21.15.6:0/73685680 >> v2:172.21.15.178:3300/0 conn(0x7f3b1c161670 0x7f3b1c070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 -- 172.21.15.6:0/73685680 >> 172.21.15.6:0/73685680 conn(0x7f3b1c06afb0 msgr2=0x7f3b1c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 -- 172.21.15.6:0/73685680 shutdown_connections 2024-03-06T07:32:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.931+0000 7f3b21e00640 1 -- 172.21.15.6:0/73685680 wait complete. 2024-03-06T07:32:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.932+0000 7f3b21e00640 1 Processor -- start 2024-03-06T07:32:18.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.933+0000 7f3b21e00640 1 -- start start 2024-03-06T07:32:18.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.933+0000 7f3b21e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b1c06e240 0x7f3b1c0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.933+0000 7f3b21e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3b1c0ae0a0 0x7f3b1c14b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b21e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 0x7f3b1c0b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b21e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0ce600 con 0x7f3b1c161670 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b21e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0cf520 con 0x7f3b1c06e240 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b21e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c0ce780 con 0x7f3b1c0ae0a0 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 0x7f3b1c0b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 0x7f3b1c0b0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53562/0 (socket says 172.21.15.6:53562) 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 -- 172.21.15.6:0/2624127945 learned_addr learned my addr 172.21.15.6:0/2624127945 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1aa00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b1c06e240 0x7f3b1c0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b12000640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3b1c0ae0a0 0x7f3b1c14b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 -- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3b1c0ae0a0 msgr2=0x7f3b1c14b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3b1c0ae0a0 0x7f3b1c14b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 -- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b1c06e240 msgr2=0x7f3b1c0cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:18.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b1c06e240 0x7f3b1c0cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.934+0000 7f3b1b400640 1 -- 172.21.15.6:0/2624127945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c09c560 con 0x7f3b1c161670 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.935+0000 7f3b1b400640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 0x7f3b1c0b0b80 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f3b140090b0 tx=0x7f3b140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.935+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14032880 con 0x7f3b1c161670 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.935+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14032a40 con 0x7f3b1c161670 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.935+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c09c840 con 0x7f3b1c161670 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.935+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14032e30 con 0x7f3b1c161670 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.935+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c09ce00 con 0x7f3b1c161670 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.936+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c069510 con 0x7f3b1c161670 2024-03-06T07:32:18.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.937+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b140453f0 con 0x7f3b1c161670 2024-03-06T07:32:18.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.938+0000 7f3b13e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3af806b4d0 0x7f3af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:18.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.938+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b140ac3d0 con 0x7f3b1c161670 2024-03-06T07:32:18.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.939+0000 7f3b1aa00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3af806b4d0 0x7f3af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:18.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.940+0000 7f3b1aa00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3af806b4d0 0x7f3af806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c007510 tx=0x7f3b0c0074a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:18.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:18.943+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b14040070 con 0x7f3b1c161670 2024-03-06T07:32:19.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.134+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b1c0b52d0 con 0x7f3b1c161670 2024-03-06T07:32:19.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.136+0000 7f3b13e00640 1 -- 172.21.15.6:0/2624127945 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f3b1407b400 con 0x7f3b1c161670 2024-03-06T07:32:19.139 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:19.139 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:16.181620+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:19.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3af806b4d0 msgr2=0x7f3af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3af806b4d0 0x7f3af806d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c007510 tx=0x7f3b0c0074a0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 msgr2=0x7f3b1c0b0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 0x7f3b1c0b0b80 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f3b140090b0 tx=0x7f3b140046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 shutdown_connections 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3af806b4d0 0x7f3af806d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3b1c161670 0x7f3b1c0b0b80 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3b1c0ae0a0 0x7f3b1c14b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 --2- 172.21.15.6:0/2624127945 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3b1c06e240 0x7f3b1c0cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 >> 172.21.15.6:0/2624127945 conn(0x7f3b1c06afb0 msgr2=0x7f3b1c0b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 shutdown_connections 2024-03-06T07:32:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.139+0000 7f3b21e00640 1 -- 172.21.15.6:0/2624127945 wait complete. 2024-03-06T07:32:19.159 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:32:19.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:19.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.380+0000 7f147ca00640 1 Processor -- start 2024-03-06T07:32:19.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.380+0000 7f147ca00640 1 -- start start 2024-03-06T07:32:19.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.381+0000 7f147ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.381+0000 7f147ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f14780af580 0x7f14780b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.381+0000 7f147ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14780cdc20 0x7f14780b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.381+0000 7f147ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147815cc40 con 0x7f14780cdc20 2024-03-06T07:32:19.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.381+0000 7f147ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147815cf20 con 0x7f14780af580 2024-03-06T07:32:19.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.381+0000 7f147ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147815d200 con 0x7f14780af1a0 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14780cdc20 0x7f14780b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14780cdc20 0x7f14780b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53572/0 (socket says 172.21.15.6:53572) 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 -- 172.21.15.6:0/2737105270 learned_addr learned my addr 172.21.15.6:0/2737105270 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1475e00640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f146fe00640 1 --2- 172.21.15.6:0/2737105270 >> v2:172.21.15.178:3300/0 conn(0x7f14780af580 0x7f14780b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 -- 172.21.15.6:0/2737105270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 msgr2=0x7f147815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 -- 172.21.15.6:0/2737105270 >> v2:172.21.15.178:3300/0 conn(0x7f14780af580 msgr2=0x7f14780b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.382+0000 7f1476800640 1 --2- 172.21.15.6:0/2737105270 >> v2:172.21.15.178:3300/0 conn(0x7f14780af580 0x7f14780b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.383+0000 7f1476800640 1 -- 172.21.15.6:0/2737105270 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147815d4e0 con 0x7f14780cdc20 2024-03-06T07:32:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.383+0000 7f1475e00640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.383+0000 7f146fe00640 1 --2- 172.21.15.6:0/2737105270 >> v2:172.21.15.178:3300/0 conn(0x7f14780af580 0x7f14780b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.384+0000 7f1476800640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14780b2dd0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f14780748c0 tx=0x7f147002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207cf9c7d7f1eeca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.384+0000 7f1475400640 1 -- 172.21.15.6:0/2737105270 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147002c9e0 con 0x7f14780cdc20 2024-03-06T07:32:19.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.384+0000 7f1475400640 1 -- 172.21.15.6:0/2737105270 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147002cb80 con 0x7f14780cdc20 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.384+0000 7f1475400640 1 -- 172.21.15.6:0/2737105270 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14700343f0 con 0x7f14780cdc20 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 -- 172.21.15.6:0/2737105270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 msgr2=0x7f14780b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14780b2dd0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f14780748c0 tx=0x7f147002be40 comp rx=0 tx=0).stop 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 -- 172.21.15.6:0/2737105270 shutdown_connections 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14780b2dd0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 --2- 172.21.15.6:0/2737105270 >> v2:172.21.15.178:3300/0 conn(0x7f14780af580 0x7f14780b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 --2- 172.21.15.6:0/2737105270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 -- 172.21.15.6:0/2737105270 >> 172.21.15.6:0/2737105270 conn(0x7f147806afb0 msgr2=0x7f147806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 -- 172.21.15.6:0/2737105270 shutdown_connections 2024-03-06T07:32:19.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.385+0000 7f147ca00640 1 -- 172.21.15.6:0/2737105270 wait complete. 2024-03-06T07:32:19.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.388+0000 7f147ca00640 1 Processor -- start 2024-03-06T07:32:19.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.388+0000 7f147ca00640 1 -- start start 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f147ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f147ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14780af580 0x7f147814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f147ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14781f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f147ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b3490 con 0x7f14780cdc20 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f147ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b3310 con 0x7f14780af580 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f147ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14780b3610 con 0x7f14780af1a0 2024-03-06T07:32:19.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f1476800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14781f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f1476800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14781f71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53578/0 (socket says 172.21.15.6:53578) 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.389+0000 7f1476800640 1 -- 172.21.15.6:0/1951767942 learned_addr learned my addr 172.21.15.6:0/1951767942 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1475e00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f146fe00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14780af580 0x7f147814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1476800640 1 -- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 msgr2=0x7f147815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1476800640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1476800640 1 -- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14780af580 msgr2=0x7f147814d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1476800640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14780af580 0x7f147814d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1476800640 1 -- 172.21.15.6:0/1951767942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147814d8b0 con 0x7f14780cdc20 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f1476800640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14781f71c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1470009f40 tx=0x7f1470004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.390+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470044020 con 0x7f14780cdc20 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.391+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1470040690 con 0x7f14780cdc20 2024-03-06T07:32:19.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.391+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470040a80 con 0x7f14780cdc20 2024-03-06T07:32:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.391+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147814db90 con 0x7f14780cdc20 2024-03-06T07:32:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.391+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14780afa90 con 0x7f14780cdc20 2024-03-06T07:32:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.392+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1478069510 con 0x7f14780cdc20 2024-03-06T07:32:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.393+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1470047020 con 0x7f14780cdc20 2024-03-06T07:32:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.394+0000 7f146e000640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f145406b4d0 0x7f145406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.394+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14700ace30 con 0x7f14780cdc20 2024-03-06T07:32:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.394+0000 7f1475e00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f145406b4d0 0x7f145406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.395+0000 7f1475e00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f145406b4d0 0x7f145406d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f146800a980 tx=0x7f1468008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.398+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f147004e030 con 0x7f14780cdc20 2024-03-06T07:32:19.555 INFO:tasks.ceph.mds.b.smithi178.stderr:2024-03-06T07:32:19.553+0000 7f5eae000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:19.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.594+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14780afc60 con 0x7f145406b4d0 2024-03-06T07:32:19.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.613+0000 7f146e000640 1 -- 172.21.15.6:0/1951767942 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252038 (secure 0 0 0) 0x7f14780afc60 con 0x7f145406b4d0 2024-03-06T07:32:19.617 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:19.621 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:32:19.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f145406b4d0 msgr2=0x7f145406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f145406b4d0 0x7f145406d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f146800a980 tx=0x7f1468008040 comp rx=0 tx=0).stop 2024-03-06T07:32:19.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 msgr2=0x7f14781f71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14781f71c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f1470009f40 tx=0x7f1470004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 shutdown_connections 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f145406b4d0 0x7f145406d990 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14780af580 0x7f147814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14780cdc20 0x7f14781f71c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 --2- 172.21.15.6:0/1951767942 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14780af1a0 0x7f147815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 >> 172.21.15.6:0/1951767942 conn(0x7f147806afb0 msgr2=0x7f14780cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 shutdown_connections 2024-03-06T07:32:19.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.620+0000 7f147ca00640 1 -- 172.21.15.6:0/1951767942 wait complete. 2024-03-06T07:32:19.644 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-03-06T07:32:19.544400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275000,"kb_used_data":8876,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754699720,"statfs":{"total":773094113280,"available":772812513280,"internally_reserved":0,"allocated":9089024,"data_stored":4491832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002035"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.213031+0000","last_change":"2024-03-06T07:32:15.209255+0000","last_active":"2024-03-06T07:32:15.213031+0000","last_peered":"2024-03-06T07:32:15.213031+0000","last_clean":"2024-03-06T07:32:15.213031+0000","last_became_active":"2024-03-06T07:32:15.208877+0000","last_became_peered":"2024-03-06T07:32:15.208877+0000","last_unstale":"2024-03-06T07:32:15.213031+0000","last_undegraded":"2024-03-06T07:32:15.213031+0000","last_fullsized":"2024-03-06T07:32:15.213031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:07.756219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200483+0000","last_change":"2024-03-06T07:32:15.212919+0000","last_active":"2024-03-06T07:32:16.200483+0000","last_peered":"2024-03-06T07:32:16.200483+0000","last_clean":"2024-03-06T07:32:16.200483+0000","last_became_active":"2024-03-06T07:32:15.212635+0000","last_became_peered":"2024-03-06T07:32:15.212635+0000","last_unstale":"2024-03-06T07:32:16.200483+0000","last_undegraded":"2024-03-06T07:32:16.200483+0000","last_fullsized":"2024-03-06T07:32:16.200483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:59.402463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218867+0000","last_change":"2024-03-06T07:32:15.213320+0000","last_active":"2024-03-06T07:32:15.218867+0000","last_peered":"2024-03-06T07:32:15.218867+0000","last_clean":"2024-03-06T07:32:15.218867+0000","last_became_active":"2024-03-06T07:32:15.212945+0000","last_became_peered":"2024-03-06T07:32:15.212945+0000","last_unstale":"2024-03-06T07:32:15.218867+0000","last_undegraded":"2024-03-06T07:32:15.218867+0000","last_fullsized":"2024-03-06T07:32:15.218867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:53.629535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198343+0000","last_change":"2024-03-06T07:32:15.191295+0000","last_active":"2024-03-06T07:32:16.198343+0000","last_peered":"2024-03-06T07:32:16.198343+0000","last_clean":"2024-03-06T07:32:16.198343+0000","last_became_active":"2024-03-06T07:32:15.190985+0000","last_became_peered":"2024-03-06T07:32:15.190985+0000","last_unstale":"2024-03-06T07:32:16.198343+0000","last_undegraded":"2024-03-06T07:32:16.198343+0000","last_fullsized":"2024-03-06T07:32:16.198343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:25.780590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.204670+0000","last_change":"2024-03-06T07:32:15.197293+0000","last_active":"2024-03-06T07:32:15.204670+0000","last_peered":"2024-03-06T07:32:15.204670+0000","last_clean":"2024-03-06T07:32:15.204670+0000","last_became_active":"2024-03-06T07:32:15.192219+0000","last_became_peered":"2024-03-06T07:32:15.192219+0000","last_unstale":"2024-03-06T07:32:15.204670+0000","last_undegraded":"2024-03-06T07:32:15.204670+0000","last_fullsized":"2024-03-06T07:32:15.204670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:36.488920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236140+0000","last_change":"2024-03-06T07:32:15.210767+0000","last_active":"2024-03-06T07:32:16.236140+0000","last_peered":"2024-03-06T07:32:16.236140+0000","last_clean":"2024-03-06T07:32:16.236140+0000","last_became_active":"2024-03-06T07:32:15.206434+0000","last_became_peered":"2024-03-06T07:32:15.206434+0000","last_unstale":"2024-03-06T07:32:16.236140+0000","last_undegraded":"2024-03-06T07:32:16.236140+0000","last_fullsized":"2024-03-06T07:32:16.236140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:33.623254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198181+0000","last_change":"2024-03-06T07:32:15.197471+0000","last_active":"2024-03-06T07:32:16.198181+0000","last_peered":"2024-03-06T07:32:16.198181+0000","last_clean":"2024-03-06T07:32:16.198181+0000","last_became_active":"2024-03-06T07:32:15.197200+0000","last_became_peered":"2024-03-06T07:32:15.197200+0000","last_unstale":"2024-03-06T07:32:16.198181+0000","last_undegraded":"2024-03-06T07:32:16.198181+0000","last_fullsized":"2024-03-06T07:32:16.198181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:45.983922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235580+0000","last_change":"2024-03-06T07:32:15.206010+0000","last_active":"2024-03-06T07:32:16.235580+0000","last_peered":"2024-03-06T07:32:16.235580+0000","last_clean":"2024-03-06T07:32:16.235580+0000","last_became_active":"2024-03-06T07:32:15.205581+0000","last_became_peered":"2024-03-06T07:32:15.205581+0000","last_unstale":"2024-03-06T07:32:16.235580+0000","last_undegraded":"2024-03-06T07:32:16.235580+0000","last_fullsized":"2024-03-06T07:32:16.235580+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:15.105303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236227+0000","last_change":"2024-03-06T07:32:15.201009+0000","last_active":"2024-03-06T07:32:16.236227+0000","last_peered":"2024-03-06T07:32:16.236227+0000","last_clean":"2024-03-06T07:32:16.236227+0000","last_became_active":"2024-03-06T07:32:15.200680+0000","last_became_peered":"2024-03-06T07:32:15.200680+0000","last_unstale":"2024-03-06T07:32:16.236227+0000","last_undegraded":"2024-03-06T07:32:16.236227+0000","last_fullsized":"2024-03-06T07:32:16.236227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:19.334348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.219609+0000","last_change":"2024-03-06T07:32:15.215897+0000","last_active":"2024-03-06T07:32:15.219609+0000","last_peered":"2024-03-06T07:32:15.219609+0000","last_clean":"2024-03-06T07:32:15.219609+0000","last_became_active":"2024-03-06T07:32:15.215623+0000","last_became_peered":"2024-03-06T07:32:15.215623+0000","last_unstale":"2024-03-06T07:32:15.219609+0000","last_undegraded":"2024-03-06T07:32:15.219609+0000","last_fullsized":"2024-03-06T07:32:15.219609+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.042864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200767+0000","last_change":"2024-03-06T07:32:15.200504+0000","last_active":"2024-03-06T07:32:16.200767+0000","last_peered":"2024-03-06T07:32:16.200767+0000","last_clean":"2024-03-06T07:32:16.200767+0000","last_became_active":"2024-03-06T07:32:15.197885+0000","last_became_peered":"2024-03-06T07:32:15.197885+0000","last_unstale":"2024-03-06T07:32:16.200767+0000","last_undegraded":"2024-03-06T07:32:16.200767+0000","last_fullsized":"2024-03-06T07:32:16.200767+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:57.737289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218828+0000","last_change":"2024-03-06T07:32:15.217030+0000","last_active":"2024-03-06T07:32:15.218828+0000","last_peered":"2024-03-06T07:32:15.218828+0000","last_clean":"2024-03-06T07:32:15.218828+0000","last_became_active":"2024-03-06T07:32:15.211487+0000","last_became_peered":"2024-03-06T07:32:15.211487+0000","last_unstale":"2024-03-06T07:32:15.218828+0000","last_undegraded":"2024-03-06T07:32:15.218828+0000","last_fullsized":"2024-03-06T07:32:15.218828+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:33.623254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.215796+0000","last_change":"2024-03-06T07:32:15.207961+0000","last_active":"2024-03-06T07:32:15.215796+0000","last_peered":"2024-03-06T07:32:15.215796+0000","last_clean":"2024-03-06T07:32:15.215796+0000","last_became_active":"2024-03-06T07:32:15.207626+0000","last_became_peered":"2024-03-06T07:32:15.207626+0000","last_unstale":"2024-03-06T07:32:15.215796+0000","last_undegraded":"2024-03-06T07:32:15.215796+0000","last_fullsized":"2024-03-06T07:32:15.215796+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:59.627690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200821+0000","last_change":"2024-03-06T07:32:15.194949+0000","last_active":"2024-03-06T07:32:16.200821+0000","last_peered":"2024-03-06T07:32:16.200821+0000","last_clean":"2024-03-06T07:32:16.200821+0000","last_became_active":"2024-03-06T07:32:15.192951+0000","last_became_peered":"2024-03-06T07:32:15.192951+0000","last_unstale":"2024-03-06T07:32:16.200821+0000","last_undegraded":"2024-03-06T07:32:16.200821+0000","last_fullsized":"2024-03-06T07:32:16.200821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:26.394546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197643+0000","last_change":"2024-03-06T07:32:15.199167+0000","last_active":"2024-03-06T07:32:16.197643+0000","last_peered":"2024-03-06T07:32:16.197643+0000","last_clean":"2024-03-06T07:32:16.197643+0000","last_became_active":"2024-03-06T07:32:15.196902+0000","last_became_peered":"2024-03-06T07:32:15.196902+0000","last_unstale":"2024-03-06T07:32:16.197643+0000","last_undegraded":"2024-03-06T07:32:16.197643+0000","last_fullsized":"2024-03-06T07:32:16.197643+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:01.789702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.221210+0000","last_change":"2024-03-06T07:32:15.219009+0000","last_active":"2024-03-06T07:32:15.221210+0000","last_peered":"2024-03-06T07:32:15.221210+0000","last_clean":"2024-03-06T07:32:15.221210+0000","last_became_active":"2024-03-06T07:32:15.218214+0000","last_became_peered":"2024-03-06T07:32:15.218214+0000","last_unstale":"2024-03-06T07:32:15.221210+0000","last_undegraded":"2024-03-06T07:32:15.221210+0000","last_fullsized":"2024-03-06T07:32:15.221210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:59.627690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.210340+0000","last_change":"2024-03-06T07:32:15.204451+0000","last_active":"2024-03-06T07:32:15.210340+0000","last_peered":"2024-03-06T07:32:15.210340+0000","last_clean":"2024-03-06T07:32:15.210340+0000","last_became_active":"2024-03-06T07:32:15.204089+0000","last_became_peered":"2024-03-06T07:32:15.204089+0000","last_unstale":"2024-03-06T07:32:15.210340+0000","last_undegraded":"2024-03-06T07:32:15.210340+0000","last_fullsized":"2024-03-06T07:32:15.210340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:02.825297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199714+0000","last_change":"2024-03-06T07:32:15.212985+0000","last_active":"2024-03-06T07:32:16.199714+0000","last_peered":"2024-03-06T07:32:16.199714+0000","last_clean":"2024-03-06T07:32:16.199714+0000","last_became_active":"2024-03-06T07:32:15.212708+0000","last_became_peered":"2024-03-06T07:32:15.212708+0000","last_unstale":"2024-03-06T07:32:16.199714+0000","last_undegraded":"2024-03-06T07:32:16.199714+0000","last_fullsized":"2024-03-06T07:32:16.199714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:03.994277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.201325+0000","last_change":"2024-03-06T07:32:15.196520+0000","last_active":"2024-03-06T07:32:15.201325+0000","last_peered":"2024-03-06T07:32:15.201325+0000","last_clean":"2024-03-06T07:32:15.201325+0000","last_became_active":"2024-03-06T07:32:15.193543+0000","last_became_peered":"2024-03-06T07:32:15.193543+0000","last_unstale":"2024-03-06T07:32:15.201325+0000","last_undegraded":"2024-03-06T07:32:15.201325+0000","last_fullsized":"2024-03-06T07:32:15.201325+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:43.466481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198491+0000","last_change":"2024-03-06T07:32:15.190490+0000","last_active":"2024-03-06T07:32:16.198491+0000","last_peered":"2024-03-06T07:32:16.198491+0000","last_clean":"2024-03-06T07:32:16.198491+0000","last_became_active":"2024-03-06T07:32:15.189029+0000","last_became_peered":"2024-03-06T07:32:15.189029+0000","last_unstale":"2024-03-06T07:32:16.198491+0000","last_undegraded":"2024-03-06T07:32:16.198491+0000","last_fullsized":"2024-03-06T07:32:16.198491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:23.488199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201656+0000","last_change":"2024-03-06T07:32:15.190779+0000","last_active":"2024-03-06T07:32:16.201656+0000","last_peered":"2024-03-06T07:32:16.201656+0000","last_clean":"2024-03-06T07:32:16.201656+0000","last_became_active":"2024-03-06T07:32:15.189840+0000","last_became_peered":"2024-03-06T07:32:15.189840+0000","last_unstale":"2024-03-06T07:32:16.201656+0000","last_undegraded":"2024-03-06T07:32:16.201656+0000","last_fullsized":"2024-03-06T07:32:16.201656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:41.646060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199770+0000","last_change":"2024-03-06T07:32:15.197498+0000","last_active":"2024-03-06T07:32:16.199770+0000","last_peered":"2024-03-06T07:32:16.199770+0000","last_clean":"2024-03-06T07:32:16.199770+0000","last_became_active":"2024-03-06T07:32:15.194902+0000","last_became_peered":"2024-03-06T07:32:15.194902+0000","last_unstale":"2024-03-06T07:32:16.199770+0000","last_undegraded":"2024-03-06T07:32:16.199770+0000","last_fullsized":"2024-03-06T07:32:16.199770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:50.313213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217483+0000","last_change":"2024-03-06T07:32:15.210971+0000","last_active":"2024-03-06T07:32:15.217483+0000","last_peered":"2024-03-06T07:32:15.217483+0000","last_clean":"2024-03-06T07:32:15.217483+0000","last_became_active":"2024-03-06T07:32:15.210383+0000","last_became_peered":"2024-03-06T07:32:15.210383+0000","last_unstale":"2024-03-06T07:32:15.217483+0000","last_undegraded":"2024-03-06T07:32:15.217483+0000","last_fullsized":"2024-03-06T07:32:15.217483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:22.726113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198808+0000","last_change":"2024-03-06T07:32:15.187112+0000","last_active":"2024-03-06T07:32:16.198808+0000","last_peered":"2024-03-06T07:32:16.198808+0000","last_clean":"2024-03-06T07:32:16.198808+0000","last_became_active":"2024-03-06T07:32:15.186775+0000","last_became_peered":"2024-03-06T07:32:15.186775+0000","last_unstale":"2024-03-06T07:32:16.198808+0000","last_undegraded":"2024-03-06T07:32:16.198808+0000","last_fullsized":"2024-03-06T07:32:16.198808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:45.616728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201180+0000","last_change":"2024-03-06T07:32:15.214462+0000","last_active":"2024-03-06T07:32:16.201180+0000","last_peered":"2024-03-06T07:32:16.201180+0000","last_clean":"2024-03-06T07:32:16.201180+0000","last_became_active":"2024-03-06T07:32:15.214192+0000","last_became_peered":"2024-03-06T07:32:15.214192+0000","last_unstale":"2024-03-06T07:32:16.201180+0000","last_undegraded":"2024-03-06T07:32:16.201180+0000","last_fullsized":"2024-03-06T07:32:16.201180+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:44.604593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.213255+0000","last_change":"2024-03-06T07:32:15.204063+0000","last_active":"2024-03-06T07:32:15.213255+0000","last_peered":"2024-03-06T07:32:15.213255+0000","last_clean":"2024-03-06T07:32:15.213255+0000","last_became_active":"2024-03-06T07:32:15.203792+0000","last_became_peered":"2024-03-06T07:32:15.203792+0000","last_unstale":"2024-03-06T07:32:15.213255+0000","last_undegraded":"2024-03-06T07:32:15.213255+0000","last_fullsized":"2024-03-06T07:32:15.213255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:41.113175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.212563+0000","last_change":"2024-03-06T07:32:15.210353+0000","last_active":"2024-03-06T07:32:15.212563+0000","last_peered":"2024-03-06T07:32:15.212563+0000","last_clean":"2024-03-06T07:32:15.212563+0000","last_became_active":"2024-03-06T07:32:15.210008+0000","last_became_peered":"2024-03-06T07:32:15.210008+0000","last_unstale":"2024-03-06T07:32:15.212563+0000","last_undegraded":"2024-03-06T07:32:15.212563+0000","last_fullsized":"2024-03-06T07:32:15.212563+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:41.113175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216925+0000","last_change":"2024-03-06T07:32:15.205793+0000","last_active":"2024-03-06T07:32:15.216925+0000","last_peered":"2024-03-06T07:32:15.216925+0000","last_clean":"2024-03-06T07:32:15.216925+0000","last_became_active":"2024-03-06T07:32:15.205371+0000","last_became_peered":"2024-03-06T07:32:15.205371+0000","last_unstale":"2024-03-06T07:32:15.216925+0000","last_undegraded":"2024-03-06T07:32:15.216925+0000","last_fullsized":"2024-03-06T07:32:15.216925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:11.769674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200861+0000","last_change":"2024-03-06T07:32:15.192171+0000","last_active":"2024-03-06T07:32:16.200861+0000","last_peered":"2024-03-06T07:32:16.200861+0000","last_clean":"2024-03-06T07:32:16.200861+0000","last_became_active":"2024-03-06T07:32:15.190338+0000","last_became_peered":"2024-03-06T07:32:15.190338+0000","last_unstale":"2024-03-06T07:32:16.200861+0000","last_undegraded":"2024-03-06T07:32:16.200861+0000","last_fullsized":"2024-03-06T07:32:16.200861+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:26.109997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199815+0000","last_change":"2024-03-06T07:32:15.188373+0000","last_active":"2024-03-06T07:32:16.199815+0000","last_peered":"2024-03-06T07:32:16.199815+0000","last_clean":"2024-03-06T07:32:16.199815+0000","last_became_active":"2024-03-06T07:32:15.187333+0000","last_became_peered":"2024-03-06T07:32:15.187333+0000","last_unstale":"2024-03-06T07:32:16.199815+0000","last_undegraded":"2024-03-06T07:32:16.199815+0000","last_fullsized":"2024-03-06T07:32:16.199815+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:17.082067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.199688+0000","last_change":"2024-03-06T07:32:15.190650+0000","last_active":"2024-03-06T07:32:15.199688+0000","last_peered":"2024-03-06T07:32:15.199688+0000","last_clean":"2024-03-06T07:32:15.199688+0000","last_became_active":"2024-03-06T07:32:15.190241+0000","last_became_peered":"2024-03-06T07:32:15.190241+0000","last_unstale":"2024-03-06T07:32:15.199688+0000","last_undegraded":"2024-03-06T07:32:15.199688+0000","last_fullsized":"2024-03-06T07:32:15.199688+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:19.073857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.208754+0000","last_change":"2024-03-06T07:32:15.200748+0000","last_active":"2024-03-06T07:32:15.208754+0000","last_peered":"2024-03-06T07:32:15.208754+0000","last_clean":"2024-03-06T07:32:15.208754+0000","last_became_active":"2024-03-06T07:32:15.200462+0000","last_became_peered":"2024-03-06T07:32:15.200462+0000","last_unstale":"2024-03-06T07:32:15.208754+0000","last_undegraded":"2024-03-06T07:32:15.208754+0000","last_fullsized":"2024-03-06T07:32:15.208754+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:09.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.212256+0000","last_change":"2024-03-06T07:32:15.209963+0000","last_active":"2024-03-06T07:32:15.212256+0000","last_peered":"2024-03-06T07:32:15.212256+0000","last_clean":"2024-03-06T07:32:15.212256+0000","last_became_active":"2024-03-06T07:32:15.206491+0000","last_became_peered":"2024-03-06T07:32:15.206491+0000","last_unstale":"2024-03-06T07:32:15.212256+0000","last_undegraded":"2024-03-06T07:32:15.212256+0000","last_fullsized":"2024-03-06T07:32:15.212256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:53.869423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.223466+0000","last_change":"2024-03-06T07:32:14.172703+0000","last_active":"2024-03-06T07:32:15.223466+0000","last_peered":"2024-03-06T07:32:15.223466+0000","last_clean":"2024-03-06T07:32:15.223466+0000","last_became_active":"2024-03-06T07:32:14.171546+0000","last_became_peered":"2024-03-06T07:32:14.171546+0000","last_unstale":"2024-03-06T07:32:15.223466+0000","last_undegraded":"2024-03-06T07:32:15.223466+0000","last_fullsized":"2024-03-06T07:32:15.223466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:22.511086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.214897+0000","last_change":"2024-03-06T07:32:15.213170+0000","last_active":"2024-03-06T07:32:15.214897+0000","last_peered":"2024-03-06T07:32:15.214897+0000","last_clean":"2024-03-06T07:32:15.214897+0000","last_became_active":"2024-03-06T07:32:15.212730+0000","last_became_peered":"2024-03-06T07:32:15.212730+0000","last_unstale":"2024-03-06T07:32:15.214897+0000","last_undegraded":"2024-03-06T07:32:15.214897+0000","last_fullsized":"2024-03-06T07:32:15.214897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:43.466481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.215681+0000","last_change":"2024-03-06T07:32:14.170974+0000","last_active":"2024-03-06T07:32:15.215681+0000","last_peered":"2024-03-06T07:32:15.215681+0000","last_clean":"2024-03-06T07:32:15.215681+0000","last_became_active":"2024-03-06T07:32:14.170643+0000","last_became_peered":"2024-03-06T07:32:14.170643+0000","last_unstale":"2024-03-06T07:32:15.215681+0000","last_undegraded":"2024-03-06T07:32:15.215681+0000","last_fullsized":"2024-03-06T07:32:15.215681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:26.115845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216952+0000","last_change":"2024-03-06T07:32:15.213103+0000","last_active":"2024-03-06T07:32:15.216952+0000","last_peered":"2024-03-06T07:32:15.216952+0000","last_clean":"2024-03-06T07:32:15.216952+0000","last_became_active":"2024-03-06T07:32:15.212881+0000","last_became_peered":"2024-03-06T07:32:15.212881+0000","last_unstale":"2024-03-06T07:32:15.216952+0000","last_undegraded":"2024-03-06T07:32:15.216952+0000","last_fullsized":"2024-03-06T07:32:15.216952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:15.105303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218948+0000","last_change":"2024-03-06T07:32:14.169744+0000","last_active":"2024-03-06T07:32:15.218948+0000","last_peered":"2024-03-06T07:32:15.218948+0000","last_clean":"2024-03-06T07:32:15.218948+0000","last_became_active":"2024-03-06T07:32:14.169467+0000","last_became_peered":"2024-03-06T07:32:14.169467+0000","last_unstale":"2024-03-06T07:32:15.218948+0000","last_undegraded":"2024-03-06T07:32:15.218948+0000","last_fullsized":"2024-03-06T07:32:15.218948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:13.533245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.214623+0000","last_change":"2024-03-06T07:32:15.208110+0000","last_active":"2024-03-06T07:32:15.214623+0000","last_peered":"2024-03-06T07:32:15.214623+0000","last_clean":"2024-03-06T07:32:15.214623+0000","last_became_active":"2024-03-06T07:32:15.207467+0000","last_became_peered":"2024-03-06T07:32:15.207467+0000","last_unstale":"2024-03-06T07:32:15.214623+0000","last_undegraded":"2024-03-06T07:32:15.214623+0000","last_fullsized":"2024-03-06T07:32:15.214623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:24.626631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218879+0000","last_change":"2024-03-06T07:32:14.169448+0000","last_active":"2024-03-06T07:32:15.218879+0000","last_peered":"2024-03-06T07:32:15.218879+0000","last_clean":"2024-03-06T07:32:15.218879+0000","last_became_active":"2024-03-06T07:32:14.169156+0000","last_became_peered":"2024-03-06T07:32:14.169156+0000","last_unstale":"2024-03-06T07:32:15.218879+0000","last_undegraded":"2024-03-06T07:32:15.218879+0000","last_fullsized":"2024-03-06T07:32:15.218879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:45.752210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199980+0000","last_change":"2024-03-06T07:32:15.188669+0000","last_active":"2024-03-06T07:32:16.199980+0000","last_peered":"2024-03-06T07:32:16.199980+0000","last_clean":"2024-03-06T07:32:16.199980+0000","last_became_active":"2024-03-06T07:32:15.187873+0000","last_became_peered":"2024-03-06T07:32:15.187873+0000","last_unstale":"2024-03-06T07:32:16.199980+0000","last_undegraded":"2024-03-06T07:32:16.199980+0000","last_fullsized":"2024-03-06T07:32:16.199980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:04.108562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.221291+0000","last_change":"2024-03-06T07:32:14.168081+0000","last_active":"2024-03-06T07:32:15.221291+0000","last_peered":"2024-03-06T07:32:15.221291+0000","last_clean":"2024-03-06T07:32:15.221291+0000","last_became_active":"2024-03-06T07:32:14.167753+0000","last_became_peered":"2024-03-06T07:32:14.167753+0000","last_unstale":"2024-03-06T07:32:15.221291+0000","last_undegraded":"2024-03-06T07:32:15.221291+0000","last_fullsized":"2024-03-06T07:32:15.221291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:58.103349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.200340+0000","last_change":"2024-03-06T07:32:15.192239+0000","last_active":"2024-03-06T07:32:15.200340+0000","last_peered":"2024-03-06T07:32:15.200340+0000","last_clean":"2024-03-06T07:32:15.200340+0000","last_became_active":"2024-03-06T07:32:15.191967+0000","last_became_peered":"2024-03-06T07:32:15.191967+0000","last_unstale":"2024-03-06T07:32:15.200340+0000","last_undegraded":"2024-03-06T07:32:15.200340+0000","last_fullsized":"2024-03-06T07:32:15.200340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:26.028733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.215851+0000","last_change":"2024-03-06T07:32:14.169365+0000","last_active":"2024-03-06T07:32:15.215851+0000","last_peered":"2024-03-06T07:32:15.215851+0000","last_clean":"2024-03-06T07:32:15.215851+0000","last_became_active":"2024-03-06T07:32:14.168747+0000","last_became_peered":"2024-03-06T07:32:14.168747+0000","last_unstale":"2024-03-06T07:32:15.215851+0000","last_undegraded":"2024-03-06T07:32:15.215851+0000","last_fullsized":"2024-03-06T07:32:15.215851+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:48.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.211395+0000","last_change":"2024-03-06T07:32:15.202616+0000","last_active":"2024-03-06T07:32:15.211395+0000","last_peered":"2024-03-06T07:32:15.211395+0000","last_clean":"2024-03-06T07:32:15.211395+0000","last_became_active":"2024-03-06T07:32:15.202326+0000","last_became_peered":"2024-03-06T07:32:15.202326+0000","last_unstale":"2024-03-06T07:32:15.211395+0000","last_undegraded":"2024-03-06T07:32:15.211395+0000","last_fullsized":"2024-03-06T07:32:15.211395+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:15.957458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.186957+0000","last_change":"2024-03-06T07:32:14.169483+0000","last_active":"2024-03-06T07:32:15.186957+0000","last_peered":"2024-03-06T07:32:15.186957+0000","last_clean":"2024-03-06T07:32:15.186957+0000","last_became_active":"2024-03-06T07:32:14.169193+0000","last_became_peered":"2024-03-06T07:32:14.169193+0000","last_unstale":"2024-03-06T07:32:15.186957+0000","last_undegraded":"2024-03-06T07:32:15.186957+0000","last_fullsized":"2024-03-06T07:32:15.186957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:10.754870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197905+0000","last_change":"2024-03-06T07:32:15.198297+0000","last_active":"2024-03-06T07:32:16.197905+0000","last_peered":"2024-03-06T07:32:16.197905+0000","last_clean":"2024-03-06T07:32:16.197905+0000","last_became_active":"2024-03-06T07:32:15.197613+0000","last_became_peered":"2024-03-06T07:32:15.197613+0000","last_unstale":"2024-03-06T07:32:16.197905+0000","last_undegraded":"2024-03-06T07:32:16.197905+0000","last_fullsized":"2024-03-06T07:32:16.197905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:09.413206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197838+0000","last_change":"2024-03-06T07:32:14.165392+0000","last_active":"2024-03-06T07:32:16.197838+0000","last_peered":"2024-03-06T07:32:16.197838+0000","last_clean":"2024-03-06T07:32:16.197838+0000","last_became_active":"2024-03-06T07:32:14.165096+0000","last_became_peered":"2024-03-06T07:32:14.165096+0000","last_unstale":"2024-03-06T07:32:16.197838+0000","last_undegraded":"2024-03-06T07:32:16.197838+0000","last_fullsized":"2024-03-06T07:32:16.197838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:24.765787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198077+0000","last_change":"2024-03-06T07:32:15.194238+0000","last_active":"2024-03-06T07:32:16.198077+0000","last_peered":"2024-03-06T07:32:16.198077+0000","last_clean":"2024-03-06T07:32:16.198077+0000","last_became_active":"2024-03-06T07:32:15.193905+0000","last_became_peered":"2024-03-06T07:32:15.193905+0000","last_unstale":"2024-03-06T07:32:16.198077+0000","last_undegraded":"2024-03-06T07:32:16.198077+0000","last_fullsized":"2024-03-06T07:32:16.198077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:06.881711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.198314+0000","last_change":"2024-03-06T07:32:14.171996+0000","last_active":"2024-03-06T07:32:15.198314+0000","last_peered":"2024-03-06T07:32:15.198314+0000","last_clean":"2024-03-06T07:32:15.198314+0000","last_became_active":"2024-03-06T07:32:14.169695+0000","last_became_peered":"2024-03-06T07:32:14.169695+0000","last_unstale":"2024-03-06T07:32:15.198314+0000","last_undegraded":"2024-03-06T07:32:15.198314+0000","last_fullsized":"2024-03-06T07:32:15.198314+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:42.577277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236748+0000","last_change":"2024-03-06T07:32:15.206844+0000","last_active":"2024-03-06T07:32:16.236748+0000","last_peered":"2024-03-06T07:32:16.236748+0000","last_clean":"2024-03-06T07:32:16.236748+0000","last_became_active":"2024-03-06T07:32:15.206264+0000","last_became_peered":"2024-03-06T07:32:15.206264+0000","last_unstale":"2024-03-06T07:32:16.236748+0000","last_undegraded":"2024-03-06T07:32:16.236748+0000","last_fullsized":"2024-03-06T07:32:16.236748+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:58.035445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.189698+0000","last_change":"2024-03-06T07:32:14.164636+0000","last_active":"2024-03-06T07:32:15.189698+0000","last_peered":"2024-03-06T07:32:15.189698+0000","last_clean":"2024-03-06T07:32:15.189698+0000","last_became_active":"2024-03-06T07:32:14.164289+0000","last_became_peered":"2024-03-06T07:32:14.164289+0000","last_unstale":"2024-03-06T07:32:15.189698+0000","last_undegraded":"2024-03-06T07:32:15.189698+0000","last_fullsized":"2024-03-06T07:32:15.189698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:10.564384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217907+0000","last_change":"2024-03-06T07:32:15.213656+0000","last_active":"2024-03-06T07:32:15.217907+0000","last_peered":"2024-03-06T07:32:15.217907+0000","last_clean":"2024-03-06T07:32:15.217907+0000","last_became_active":"2024-03-06T07:32:15.213389+0000","last_became_peered":"2024-03-06T07:32:15.213389+0000","last_unstale":"2024-03-06T07:32:15.217907+0000","last_undegraded":"2024-03-06T07:32:15.217907+0000","last_fullsized":"2024-03-06T07:32:15.217907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:19.334348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.223181+0000","last_change":"2024-03-06T07:32:14.165114+0000","last_active":"2024-03-06T07:32:15.223181+0000","last_peered":"2024-03-06T07:32:15.223181+0000","last_clean":"2024-03-06T07:32:15.223181+0000","last_became_active":"2024-03-06T07:32:14.164791+0000","last_became_peered":"2024-03-06T07:32:14.164791+0000","last_unstale":"2024-03-06T07:32:15.223181+0000","last_undegraded":"2024-03-06T07:32:15.223181+0000","last_fullsized":"2024-03-06T07:32:15.223181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:16.146640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.209745+0000","last_change":"2024-03-06T07:32:15.199594+0000","last_active":"2024-03-06T07:32:15.209745+0000","last_peered":"2024-03-06T07:32:15.209745+0000","last_clean":"2024-03-06T07:32:15.209745+0000","last_became_active":"2024-03-06T07:32:15.199267+0000","last_became_peered":"2024-03-06T07:32:15.199267+0000","last_unstale":"2024-03-06T07:32:15.209745+0000","last_undegraded":"2024-03-06T07:32:15.209745+0000","last_fullsized":"2024-03-06T07:32:15.209745+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:26.353270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236015+0000","last_change":"2024-03-06T07:32:14.174648+0000","last_active":"2024-03-06T07:32:16.236015+0000","last_peered":"2024-03-06T07:32:16.236015+0000","last_clean":"2024-03-06T07:32:16.236015+0000","last_became_active":"2024-03-06T07:32:14.173628+0000","last_became_peered":"2024-03-06T07:32:14.173628+0000","last_unstale":"2024-03-06T07:32:16.236015+0000","last_undegraded":"2024-03-06T07:32:16.236015+0000","last_fullsized":"2024-03-06T07:32:16.236015+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:22.511086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.204408+0000","last_change":"2024-03-06T07:32:15.203407+0000","last_active":"2024-03-06T07:32:15.204408+0000","last_peered":"2024-03-06T07:32:15.204408+0000","last_clean":"2024-03-06T07:32:15.204408+0000","last_became_active":"2024-03-06T07:32:15.202655+0000","last_became_peered":"2024-03-06T07:32:15.202655+0000","last_unstale":"2024-03-06T07:32:15.204408+0000","last_undegraded":"2024-03-06T07:32:15.204408+0000","last_fullsized":"2024-03-06T07:32:15.204408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:33.623254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201168+0000","last_change":"2024-03-06T07:32:14.163706+0000","last_active":"2024-03-06T07:32:16.201168+0000","last_peered":"2024-03-06T07:32:16.201168+0000","last_clean":"2024-03-06T07:32:16.201168+0000","last_became_active":"2024-03-06T07:32:14.161664+0000","last_became_peered":"2024-03-06T07:32:14.161664+0000","last_unstale":"2024-03-06T07:32:16.201168+0000","last_undegraded":"2024-03-06T07:32:16.201168+0000","last_fullsized":"2024-03-06T07:32:16.201168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:44.601924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.213216+0000","last_change":"2024-03-06T07:32:15.203637+0000","last_active":"2024-03-06T07:32:15.213216+0000","last_peered":"2024-03-06T07:32:15.213216+0000","last_clean":"2024-03-06T07:32:15.213216+0000","last_became_active":"2024-03-06T07:32:15.203331+0000","last_became_peered":"2024-03-06T07:32:15.203331+0000","last_unstale":"2024-03-06T07:32:15.213216+0000","last_undegraded":"2024-03-06T07:32:15.213216+0000","last_fullsized":"2024-03-06T07:32:15.213216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:53.869423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198344+0000","last_change":"2024-03-06T07:32:14.166677+0000","last_active":"2024-03-06T07:32:16.198344+0000","last_peered":"2024-03-06T07:32:16.198344+0000","last_clean":"2024-03-06T07:32:16.198344+0000","last_became_active":"2024-03-06T07:32:14.165397+0000","last_became_peered":"2024-03-06T07:32:14.165397+0000","last_unstale":"2024-03-06T07:32:16.198344+0000","last_undegraded":"2024-03-06T07:32:16.198344+0000","last_fullsized":"2024-03-06T07:32:16.198344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:47.069241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197707+0000","last_change":"2024-03-06T07:32:15.187239+0000","last_active":"2024-03-06T07:32:16.197707+0000","last_peered":"2024-03-06T07:32:16.197707+0000","last_clean":"2024-03-06T07:32:16.197707+0000","last_became_active":"2024-03-06T07:32:15.186439+0000","last_became_peered":"2024-03-06T07:32:15.186439+0000","last_unstale":"2024-03-06T07:32:16.197707+0000","last_undegraded":"2024-03-06T07:32:16.197707+0000","last_fullsized":"2024-03-06T07:32:16.197707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:43.178774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235720+0000","last_change":"2024-03-06T07:32:14.171528+0000","last_active":"2024-03-06T07:32:16.235720+0000","last_peered":"2024-03-06T07:32:16.235720+0000","last_clean":"2024-03-06T07:32:16.235720+0000","last_became_active":"2024-03-06T07:32:14.169630+0000","last_became_peered":"2024-03-06T07:32:14.169630+0000","last_unstale":"2024-03-06T07:32:16.235720+0000","last_undegraded":"2024-03-06T07:32:16.235720+0000","last_fullsized":"2024-03-06T07:32:16.235720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:42.577277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199410+0000","last_change":"2024-03-06T07:32:15.214251+0000","last_active":"2024-03-06T07:32:16.199410+0000","last_peered":"2024-03-06T07:32:16.199410+0000","last_clean":"2024-03-06T07:32:16.199410+0000","last_became_active":"2024-03-06T07:32:15.213894+0000","last_became_peered":"2024-03-06T07:32:15.213894+0000","last_unstale":"2024-03-06T07:32:16.199410+0000","last_undegraded":"2024-03-06T07:32:16.199410+0000","last_fullsized":"2024-03-06T07:32:16.199410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:25.580317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.220450+0000","last_change":"2024-03-06T07:32:14.169121+0000","last_active":"2024-03-06T07:32:15.220450+0000","last_peered":"2024-03-06T07:32:15.220450+0000","last_clean":"2024-03-06T07:32:15.220450+0000","last_became_active":"2024-03-06T07:32:14.168568+0000","last_became_peered":"2024-03-06T07:32:14.168568+0000","last_unstale":"2024-03-06T07:32:15.220450+0000","last_undegraded":"2024-03-06T07:32:15.220450+0000","last_fullsized":"2024-03-06T07:32:15.220450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:06.630348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216757+0000","last_change":"2024-03-06T07:32:15.208218+0000","last_active":"2024-03-06T07:32:15.216757+0000","last_peered":"2024-03-06T07:32:15.216757+0000","last_clean":"2024-03-06T07:32:15.216757+0000","last_became_active":"2024-03-06T07:32:15.207701+0000","last_became_peered":"2024-03-06T07:32:15.207701+0000","last_unstale":"2024-03-06T07:32:15.216757+0000","last_undegraded":"2024-03-06T07:32:15.216757+0000","last_fullsized":"2024-03-06T07:32:15.216757+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:53.002096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.223010+0000","last_change":"2024-03-06T07:32:14.169336+0000","last_active":"2024-03-06T07:32:15.223010+0000","last_peered":"2024-03-06T07:32:15.223010+0000","last_clean":"2024-03-06T07:32:15.223010+0000","last_became_active":"2024-03-06T07:32:14.169028+0000","last_became_peered":"2024-03-06T07:32:14.169028+0000","last_unstale":"2024-03-06T07:32:15.223010+0000","last_undegraded":"2024-03-06T07:32:15.223010+0000","last_fullsized":"2024-03-06T07:32:15.223010+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:58.271604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.220726+0000","last_change":"2024-03-06T07:32:15.211587+0000","last_active":"2024-03-06T07:32:15.220726+0000","last_peered":"2024-03-06T07:32:15.220726+0000","last_clean":"2024-03-06T07:32:15.220726+0000","last_became_active":"2024-03-06T07:32:15.211149+0000","last_became_peered":"2024-03-06T07:32:15.211149+0000","last_unstale":"2024-03-06T07:32:15.220726+0000","last_undegraded":"2024-03-06T07:32:15.220726+0000","last_fullsized":"2024-03-06T07:32:15.220726+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:14.548048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217185+0000","last_change":"2024-03-06T07:32:14.160976+0000","last_active":"2024-03-06T07:32:15.217185+0000","last_peered":"2024-03-06T07:32:15.217185+0000","last_clean":"2024-03-06T07:32:15.217185+0000","last_became_active":"2024-03-06T07:32:14.160658+0000","last_became_peered":"2024-03-06T07:32:14.160658+0000","last_unstale":"2024-03-06T07:32:15.217185+0000","last_undegraded":"2024-03-06T07:32:15.217185+0000","last_fullsized":"2024-03-06T07:32:15.217185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:14.090499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.209765+0000","last_change":"2024-03-06T07:32:15.199472+0000","last_active":"2024-03-06T07:32:15.209765+0000","last_peered":"2024-03-06T07:32:15.209765+0000","last_clean":"2024-03-06T07:32:15.209765+0000","last_became_active":"2024-03-06T07:32:15.199141+0000","last_became_peered":"2024-03-06T07:32:15.199141+0000","last_unstale":"2024-03-06T07:32:15.209765+0000","last_undegraded":"2024-03-06T07:32:15.209765+0000","last_fullsized":"2024-03-06T07:32:15.209765+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:16.743592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187956+0000","last_change":"2024-03-06T07:32:14.171346+0000","last_active":"2024-03-06T07:32:15.187956+0000","last_peered":"2024-03-06T07:32:15.187956+0000","last_clean":"2024-03-06T07:32:15.187956+0000","last_became_active":"2024-03-06T07:32:14.171055+0000","last_became_peered":"2024-03-06T07:32:14.171055+0000","last_unstale":"2024-03-06T07:32:15.187956+0000","last_undegraded":"2024-03-06T07:32:15.187956+0000","last_fullsized":"2024-03-06T07:32:15.187956+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:21.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201257+0000","last_change":"2024-03-06T07:32:15.194967+0000","last_active":"2024-03-06T07:32:16.201257+0000","last_peered":"2024-03-06T07:32:16.201257+0000","last_clean":"2024-03-06T07:32:16.201257+0000","last_became_active":"2024-03-06T07:32:15.192629+0000","last_became_peered":"2024-03-06T07:32:15.192629+0000","last_unstale":"2024-03-06T07:32:16.201257+0000","last_undegraded":"2024-03-06T07:32:16.201257+0000","last_fullsized":"2024-03-06T07:32:16.201257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:13.343774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197872+0000","last_change":"2024-03-06T07:32:14.174314+0000","last_active":"2024-03-06T07:32:16.197872+0000","last_peered":"2024-03-06T07:32:16.197872+0000","last_clean":"2024-03-06T07:32:16.197872+0000","last_became_active":"2024-03-06T07:32:14.173464+0000","last_became_peered":"2024-03-06T07:32:14.173464+0000","last_unstale":"2024-03-06T07:32:16.197872+0000","last_undegraded":"2024-03-06T07:32:16.197872+0000","last_fullsized":"2024-03-06T07:32:16.197872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:40.754642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236759+0000","last_change":"2024-03-06T07:32:15.191384+0000","last_active":"2024-03-06T07:32:16.236759+0000","last_peered":"2024-03-06T07:32:16.236759+0000","last_clean":"2024-03-06T07:32:16.236759+0000","last_became_active":"2024-03-06T07:32:15.190635+0000","last_became_peered":"2024-03-06T07:32:15.190635+0000","last_unstale":"2024-03-06T07:32:16.236759+0000","last_undegraded":"2024-03-06T07:32:16.236759+0000","last_fullsized":"2024-03-06T07:32:16.236759+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:37.173597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201466+0000","last_change":"2024-03-06T07:32:14.162795+0000","last_active":"2024-03-06T07:32:16.201466+0000","last_peered":"2024-03-06T07:32:16.201466+0000","last_clean":"2024-03-06T07:32:16.201466+0000","last_became_active":"2024-03-06T07:32:14.162519+0000","last_became_peered":"2024-03-06T07:32:14.162519+0000","last_unstale":"2024-03-06T07:32:16.201466+0000","last_undegraded":"2024-03-06T07:32:16.201466+0000","last_fullsized":"2024-03-06T07:32:16.201466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:43.782045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199238+0000","last_change":"2024-03-06T07:32:15.191585+0000","last_active":"2024-03-06T07:32:16.199238+0000","last_peered":"2024-03-06T07:32:16.199238+0000","last_clean":"2024-03-06T07:32:16.199238+0000","last_became_active":"2024-03-06T07:32:15.191226+0000","last_became_peered":"2024-03-06T07:32:15.191226+0000","last_unstale":"2024-03-06T07:32:16.199238+0000","last_undegraded":"2024-03-06T07:32:16.199238+0000","last_fullsized":"2024-03-06T07:32:16.199238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.844786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201305+0000","last_change":"2024-03-06T07:32:14.174695+0000","last_active":"2024-03-06T07:32:16.201305+0000","last_peered":"2024-03-06T07:32:16.201305+0000","last_clean":"2024-03-06T07:32:16.201305+0000","last_became_active":"2024-03-06T07:32:14.173489+0000","last_became_peered":"2024-03-06T07:32:14.173489+0000","last_unstale":"2024-03-06T07:32:16.201305+0000","last_undegraded":"2024-03-06T07:32:16.201305+0000","last_fullsized":"2024-03-06T07:32:16.201305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:55.435500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235724+0000","last_change":"2024-03-06T07:32:15.196921+0000","last_active":"2024-03-06T07:32:16.235724+0000","last_peered":"2024-03-06T07:32:16.235724+0000","last_clean":"2024-03-06T07:32:16.235724+0000","last_became_active":"2024-03-06T07:32:15.196136+0000","last_became_peered":"2024-03-06T07:32:15.196136+0000","last_unstale":"2024-03-06T07:32:16.235724+0000","last_undegraded":"2024-03-06T07:32:16.235724+0000","last_fullsized":"2024-03-06T07:32:16.235724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:43.466481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199240+0000","last_change":"2024-03-06T07:32:14.168845+0000","last_active":"2024-03-06T07:32:16.199240+0000","last_peered":"2024-03-06T07:32:16.199240+0000","last_clean":"2024-03-06T07:32:16.199240+0000","last_became_active":"2024-03-06T07:32:14.168391+0000","last_became_peered":"2024-03-06T07:32:14.168391+0000","last_unstale":"2024-03-06T07:32:16.199240+0000","last_undegraded":"2024-03-06T07:32:16.199240+0000","last_fullsized":"2024-03-06T07:32:16.199240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:31.532989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.210599+0000","last_change":"2024-03-06T07:32:15.202007+0000","last_active":"2024-03-06T07:32:15.210599+0000","last_peered":"2024-03-06T07:32:15.210599+0000","last_clean":"2024-03-06T07:32:15.210599+0000","last_became_active":"2024-03-06T07:32:15.197471+0000","last_became_peered":"2024-03-06T07:32:15.197471+0000","last_unstale":"2024-03-06T07:32:15.210599+0000","last_undegraded":"2024-03-06T07:32:15.210599+0000","last_fullsized":"2024-03-06T07:32:15.210599+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:15.957458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201312+0000","last_change":"2024-03-06T07:32:14.171237+0000","last_active":"2024-03-06T07:32:16.201312+0000","last_peered":"2024-03-06T07:32:16.201312+0000","last_clean":"2024-03-06T07:32:16.201312+0000","last_became_active":"2024-03-06T07:32:14.170248+0000","last_became_peered":"2024-03-06T07:32:14.170248+0000","last_unstale":"2024-03-06T07:32:16.201312+0000","last_undegraded":"2024-03-06T07:32:16.201312+0000","last_fullsized":"2024-03-06T07:32:16.201312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:03.829983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.220274+0000","last_change":"2024-03-06T07:32:15.215084+0000","last_active":"2024-03-06T07:32:15.220274+0000","last_peered":"2024-03-06T07:32:15.220274+0000","last_clean":"2024-03-06T07:32:15.220274+0000","last_became_active":"2024-03-06T07:32:15.214745+0000","last_became_peered":"2024-03-06T07:32:15.214745+0000","last_unstale":"2024-03-06T07:32:15.220274+0000","last_undegraded":"2024-03-06T07:32:15.220274+0000","last_fullsized":"2024-03-06T07:32:15.220274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:22.050942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200602+0000","last_change":"2024-03-06T07:32:14.162101+0000","last_active":"2024-03-06T07:32:16.200602+0000","last_peered":"2024-03-06T07:32:16.200602+0000","last_clean":"2024-03-06T07:32:16.200602+0000","last_became_active":"2024-03-06T07:32:14.161757+0000","last_became_peered":"2024-03-06T07:32:14.161757+0000","last_unstale":"2024-03-06T07:32:16.200602+0000","last_undegraded":"2024-03-06T07:32:16.200602+0000","last_fullsized":"2024-03-06T07:32:16.200602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:22.473395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216894+0000","last_change":"2024-03-06T07:32:15.201918+0000","last_active":"2024-03-06T07:32:15.216894+0000","last_peered":"2024-03-06T07:32:15.216894+0000","last_clean":"2024-03-06T07:32:15.216894+0000","last_became_active":"2024-03-06T07:32:15.201500+0000","last_became_peered":"2024-03-06T07:32:15.201500+0000","last_unstale":"2024-03-06T07:32:15.216894+0000","last_undegraded":"2024-03-06T07:32:15.216894+0000","last_fullsized":"2024-03-06T07:32:15.216894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:07.645152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187733+0000","last_change":"2024-03-06T07:32:14.169360+0000","last_active":"2024-03-06T07:32:15.187733+0000","last_peered":"2024-03-06T07:32:15.187733+0000","last_clean":"2024-03-06T07:32:15.187733+0000","last_became_active":"2024-03-06T07:32:14.169068+0000","last_became_peered":"2024-03-06T07:32:14.169068+0000","last_unstale":"2024-03-06T07:32:15.187733+0000","last_undegraded":"2024-03-06T07:32:15.187733+0000","last_fullsized":"2024-03-06T07:32:15.187733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:18.291932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.209664+0000","last_change":"2024-03-06T07:32:15.197704+0000","last_active":"2024-03-06T07:32:15.209664+0000","last_peered":"2024-03-06T07:32:15.209664+0000","last_clean":"2024-03-06T07:32:15.209664+0000","last_became_active":"2024-03-06T07:32:15.197353+0000","last_became_peered":"2024-03-06T07:32:15.197353+0000","last_unstale":"2024-03-06T07:32:15.209664+0000","last_undegraded":"2024-03-06T07:32:15.209664+0000","last_fullsized":"2024-03-06T07:32:15.209664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:50.560658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.191044+0000","last_change":"2024-03-06T07:32:14.164424+0000","last_active":"2024-03-06T07:32:15.191044+0000","last_peered":"2024-03-06T07:32:15.191044+0000","last_clean":"2024-03-06T07:32:15.191044+0000","last_became_active":"2024-03-06T07:32:14.164059+0000","last_became_peered":"2024-03-06T07:32:14.164059+0000","last_unstale":"2024-03-06T07:32:15.191044+0000","last_undegraded":"2024-03-06T07:32:15.191044+0000","last_fullsized":"2024-03-06T07:32:15.191044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:46.948960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235780+0000","last_change":"2024-03-06T07:32:15.207994+0000","last_active":"2024-03-06T07:32:16.235780+0000","last_peered":"2024-03-06T07:32:16.235780+0000","last_clean":"2024-03-06T07:32:16.235780+0000","last_became_active":"2024-03-06T07:32:15.207192+0000","last_became_peered":"2024-03-06T07:32:15.207192+0000","last_unstale":"2024-03-06T07:32:16.235780+0000","last_undegraded":"2024-03-06T07:32:16.235780+0000","last_fullsized":"2024-03-06T07:32:16.235780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.042864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187570+0000","last_change":"2024-03-06T07:32:14.168693+0000","last_active":"2024-03-06T07:32:15.187570+0000","last_peered":"2024-03-06T07:32:15.187570+0000","last_clean":"2024-03-06T07:32:15.187570+0000","last_became_active":"2024-03-06T07:32:14.168385+0000","last_became_peered":"2024-03-06T07:32:14.168385+0000","last_unstale":"2024-03-06T07:32:15.187570+0000","last_undegraded":"2024-03-06T07:32:15.187570+0000","last_fullsized":"2024-03-06T07:32:15.187570+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:06.741416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.203699+0000","last_change":"2024-03-06T07:32:15.201468+0000","last_active":"2024-03-06T07:32:15.203699+0000","last_peered":"2024-03-06T07:32:15.203699+0000","last_clean":"2024-03-06T07:32:15.203699+0000","last_became_active":"2024-03-06T07:32:15.201169+0000","last_became_peered":"2024-03-06T07:32:15.201169+0000","last_unstale":"2024-03-06T07:32:15.203699+0000","last_undegraded":"2024-03-06T07:32:15.203699+0000","last_fullsized":"2024-03-06T07:32:15.203699+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.042864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.184904+0000","last_change":"2024-03-06T07:32:14.169033+0000","last_active":"2024-03-06T07:32:15.184904+0000","last_peered":"2024-03-06T07:32:15.184904+0000","last_clean":"2024-03-06T07:32:15.184904+0000","last_became_active":"2024-03-06T07:32:14.168324+0000","last_became_peered":"2024-03-06T07:32:14.168324+0000","last_unstale":"2024-03-06T07:32:15.184904+0000","last_undegraded":"2024-03-06T07:32:15.184904+0000","last_fullsized":"2024-03-06T07:32:15.184904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:46.205058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216945+0000","last_change":"2024-03-06T07:32:15.209659+0000","last_active":"2024-03-06T07:32:15.216945+0000","last_peered":"2024-03-06T07:32:15.216945+0000","last_clean":"2024-03-06T07:32:15.216945+0000","last_became_active":"2024-03-06T07:32:15.206930+0000","last_became_peered":"2024-03-06T07:32:15.206930+0000","last_unstale":"2024-03-06T07:32:15.216945+0000","last_undegraded":"2024-03-06T07:32:15.216945+0000","last_fullsized":"2024-03-06T07:32:15.216945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:58.656525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.193609+0000","last_change":"2024-03-06T07:32:14.172371+0000","last_active":"2024-03-06T07:32:15.193609+0000","last_peered":"2024-03-06T07:32:15.193609+0000","last_clean":"2024-03-06T07:32:15.193609+0000","last_became_active":"2024-03-06T07:32:14.170178+0000","last_became_peered":"2024-03-06T07:32:14.170178+0000","last_unstale":"2024-03-06T07:32:15.193609+0000","last_undegraded":"2024-03-06T07:32:15.193609+0000","last_fullsized":"2024-03-06T07:32:15.193609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:26.735940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217454+0000","last_change":"2024-03-06T07:32:15.210744+0000","last_active":"2024-03-06T07:32:15.217454+0000","last_peered":"2024-03-06T07:32:15.217454+0000","last_clean":"2024-03-06T07:32:15.217454+0000","last_became_active":"2024-03-06T07:32:15.210085+0000","last_became_peered":"2024-03-06T07:32:15.210085+0000","last_unstale":"2024-03-06T07:32:15.217454+0000","last_undegraded":"2024-03-06T07:32:15.217454+0000","last_fullsized":"2024-03-06T07:32:15.217454+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:46.767014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187070+0000","last_change":"2024-03-06T07:32:14.173951+0000","last_active":"2024-03-06T07:32:15.187070+0000","last_peered":"2024-03-06T07:32:15.187070+0000","last_clean":"2024-03-06T07:32:15.187070+0000","last_became_active":"2024-03-06T07:32:14.173610+0000","last_became_peered":"2024-03-06T07:32:14.173610+0000","last_unstale":"2024-03-06T07:32:15.187070+0000","last_undegraded":"2024-03-06T07:32:15.187070+0000","last_fullsized":"2024-03-06T07:32:15.187070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:13.363225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200186+0000","last_change":"2024-03-06T07:32:15.199093+0000","last_active":"2024-03-06T07:32:16.200186+0000","last_peered":"2024-03-06T07:32:16.200186+0000","last_clean":"2024-03-06T07:32:16.200186+0000","last_became_active":"2024-03-06T07:32:15.196793+0000","last_became_peered":"2024-03-06T07:32:15.196793+0000","last_unstale":"2024-03-06T07:32:16.200186+0000","last_undegraded":"2024-03-06T07:32:16.200186+0000","last_fullsized":"2024-03-06T07:32:16.200186+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:20.980135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218808+0000","last_change":"2024-03-06T07:32:14.175070+0000","last_active":"2024-03-06T07:32:15.218808+0000","last_peered":"2024-03-06T07:32:15.218808+0000","last_clean":"2024-03-06T07:32:15.218808+0000","last_became_active":"2024-03-06T07:32:14.173917+0000","last_became_peered":"2024-03-06T07:32:14.173917+0000","last_unstale":"2024-03-06T07:32:15.218808+0000","last_undegraded":"2024-03-06T07:32:15.218808+0000","last_fullsized":"2024-03-06T07:32:15.218808+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:01.810493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":276,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.192159+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:32:15.192159+0000","last_peered":"2024-03-06T07:32:15.192159+0000","last_clean":"2024-03-06T07:32:15.192159+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:32:15.192159+0000","last_undegraded":"2024-03-06T07:32:15.192159+0000","last_fullsized":"2024-03-06T07:32:15.192159+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509502,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":991232,"data_stored":417804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":417416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542206,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1572864,"data_stored":1001360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542206,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36316,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335524,"statfs":{"total":96636764160,"available":96599576576,"internally_reserved":0,"allocated":987136,"data_stored":411616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335520,"statfs":{"total":96636764160,"available":96599572480,"internally_reserved":0,"allocated":1003520,"data_stored":418856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542206,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36300,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335540,"statfs":{"total":96636764160,"available":96599592960,"internally_reserved":0,"allocated":974848,"data_stored":410922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542207,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":412534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542207,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1572864,"data_stored":1001324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:19.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.902+0000 7efce7200640 1 Processor -- start 2024-03-06T07:32:19.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.902+0000 7efce7200640 1 -- start start 2024-03-06T07:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 0x7efce00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce00714d0 con 0x7efce0070f10 2024-03-06T07:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce7200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce00717b0 con 0x7efce00cd060 2024-03-06T07:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce0071a90 con 0x7efce0145300 2024-03-06T07:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce5400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efce0070f10 0x7efce00cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41740/0 (socket says 172.21.15.6:41740) 2024-03-06T07:32:19.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce5400640 1 -- 172.21.15.6:0/1194342359 learned_addr learned my addr 172.21.15.6:0/1194342359 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.904+0000 7efce5e00640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 0x7efce00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17170580 0 0) 0x7efce00714d0 con 0x7efce0070f10 2024-03-06T07:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc8003520 con 0x7efce0070f10 2024-03-06T07:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3670976712 0 0) 0x7efcc8003520 con 0x7efce0070f10 2024-03-06T07:32:19.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 msgr2=0x7efce00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 0x7efce00b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 >> v1:172.21.15.178:6789/0 conn(0x7efce00cd060 legacy=0x7efce015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0071d70 con 0x7efce0070f10 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7efcd0001670 con 0x7efce0070f10 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 >> v1:172.21.15.6:6789/0 conn(0x7efce0070f10 legacy=0x7efce00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 0x7efcc80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc8003f60 0x7efcc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8004d40 0x7efcc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0071a90 con 0x7efcc8003f60 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc800baa0 con 0x7efcc8004d40 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc800bca0 con 0x7efce0145300 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.905+0000 7efce5400640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc8003f60 0x7efcc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce5e00640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 0x7efcc80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce6800640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8004d40 0x7efcc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce5400640 1 -- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 msgr2=0x7efcc80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce5400640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce0145300 0x7efcc80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce5400640 1 -- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8004d40 msgr2=0x7efcc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:19.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce5400640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8004d40 0x7efcc80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce5400640 1 -- 172.21.15.6:0/1194342359 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc8003520 con 0x7efcc8003f60 2024-03-06T07:32:19.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.906+0000 7efce6800640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8004d40 0x7efcc80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:19.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.907+0000 7efce5400640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc8003f60 0x7efcc8004800 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7efcd000c240 tx=0x7efcc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655defc75edbd4be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0007d10 con 0x7efcc8003f60 2024-03-06T07:32:19.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0030070 con 0x7efcc8003f60 2024-03-06T07:32:19.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce4a00640 1 -- 172.21.15.6:0/1194342359 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd002cda0 con 0x7efcc8003f60 2024-03-06T07:32:19.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce7200640 1 -- 172.21.15.6:0/1194342359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc8003f60 msgr2=0x7efcc8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce7200640 1 --2- 172.21.15.6:0/1194342359 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc8003f60 0x7efcc8004800 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7efcd000c240 tx=0x7efcc8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce6800640 1 -- 172.21.15.6:0/1194342359 reap_dead start 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce7200640 1 -- 172.21.15.6:0/1194342359 shutdown_connections 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce7200640 1 -- 172.21.15.6:0/1194342359 >> 172.21.15.6:0/1194342359 conn(0x7efce006b3d0 msgr2=0x7efce006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce7200640 1 -- 172.21.15.6:0/1194342359 shutdown_connections 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.908+0000 7efce7200640 1 -- 172.21.15.6:0/1194342359 wait complete. 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.910+0000 7efce7200640 1 Processor -- start 2024-03-06T07:32:19.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.910+0000 7efce7200640 1 -- start start 2024-03-06T07:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8003f60 0x7efce015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce00b0160 0x7efce01400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 0x7efce0140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce00b4b20 con 0x7efce00b05e0 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce00b49a0 con 0x7efcc8003f60 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce00b4ca0 con 0x7efce00b0160 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.911+0000 7efce6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 0x7efce0140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 0x7efce0140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53594/0 (socket says 172.21.15.6:53594) 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 -- 172.21.15.6:0/3161440391 learned_addr learned my addr 172.21.15.6:0/3161440391 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce5e00640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8003f60 0x7efce015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce5400640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce00b0160 0x7efce01400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 -- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce00b0160 msgr2=0x7efce01400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce00b0160 0x7efce01400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 -- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8003f60 msgr2=0x7efce015b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8003f60 0x7efce015b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 -- 172.21.15.6:0/3161440391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce015b870 con 0x7efce00b05e0 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce5e00640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8003f60 0x7efce015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:19.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.912+0000 7efce6800640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 0x7efce0140610 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7efce00cefa0 tx=0x7efcdc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.913+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc006d50 con 0x7efce00b05e0 2024-03-06T07:32:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.913+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc003970 con 0x7efce00b05e0 2024-03-06T07:32:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.913+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc003da0 con 0x7efce00b05e0 2024-03-06T07:32:19.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.913+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce00b0bb0 con 0x7efce00b05e0 2024-03-06T07:32:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.913+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce013e200 con 0x7efce00b05e0 2024-03-06T07:32:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.914+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce015f4b0 con 0x7efce00b05e0 2024-03-06T07:32:19.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.915+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcdc01b020 con 0x7efce00b05e0 2024-03-06T07:32:19.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.916+0000 7efcda000640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcc406b4d0 0x7efcc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:19.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.916+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcdc089f60 con 0x7efce00b05e0 2024-03-06T07:32:19.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.917+0000 7efce5e00640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcc406b4d0 0x7efcc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:19.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.918+0000 7efce5e00640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcc406b4d0 0x7efcc406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009ad0 tx=0x7efcd4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:19.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:19.921+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efce015f4b0 con 0x7efce00b05e0 2024-03-06T07:32:20.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.116+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efce00b10d0 con 0x7efcc406b4d0 2024-03-06T07:32:20.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.135+0000 7efcda000640 1 -- 172.21.15.6:0/3161440391 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252038 (secure 0 0 0) 0x7efcd4013a20 con 0x7efcc406b4d0 2024-03-06T07:32:20.141 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:20.142 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:32:20.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcc406b4d0 msgr2=0x7efcc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcc406b4d0 0x7efcc406d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009ad0 tx=0x7efcd4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 msgr2=0x7efce0140610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 0x7efce0140610 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7efce00cefa0 tx=0x7efcdc00dc00 comp rx=0 tx=0).stop 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 shutdown_connections 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcc406b4d0 0x7efcc406d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efce00b05e0 0x7efce0140610 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efce00b0160 0x7efce01400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 --2- 172.21.15.6:0/3161440391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc8003f60 0x7efce015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 >> 172.21.15.6:0/3161440391 conn(0x7efce006b3d0 msgr2=0x7efce00ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 shutdown_connections 2024-03-06T07:32:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.140+0000 7efce7200640 1 -- 172.21.15.6:0/3161440391 wait complete. 2024-03-06T07:32:20.161 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-03-06T07:32:19.544400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":275000,"kb_used_data":8876,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754699720,"statfs":{"total":773094113280,"available":772812513280,"internally_reserved":0,"allocated":9089024,"data_stored":4491832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002035"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.213031+0000","last_change":"2024-03-06T07:32:15.209255+0000","last_active":"2024-03-06T07:32:15.213031+0000","last_peered":"2024-03-06T07:32:15.213031+0000","last_clean":"2024-03-06T07:32:15.213031+0000","last_became_active":"2024-03-06T07:32:15.208877+0000","last_became_peered":"2024-03-06T07:32:15.208877+0000","last_unstale":"2024-03-06T07:32:15.213031+0000","last_undegraded":"2024-03-06T07:32:15.213031+0000","last_fullsized":"2024-03-06T07:32:15.213031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:07.756219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200483+0000","last_change":"2024-03-06T07:32:15.212919+0000","last_active":"2024-03-06T07:32:16.200483+0000","last_peered":"2024-03-06T07:32:16.200483+0000","last_clean":"2024-03-06T07:32:16.200483+0000","last_became_active":"2024-03-06T07:32:15.212635+0000","last_became_peered":"2024-03-06T07:32:15.212635+0000","last_unstale":"2024-03-06T07:32:16.200483+0000","last_undegraded":"2024-03-06T07:32:16.200483+0000","last_fullsized":"2024-03-06T07:32:16.200483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:59.402463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218867+0000","last_change":"2024-03-06T07:32:15.213320+0000","last_active":"2024-03-06T07:32:15.218867+0000","last_peered":"2024-03-06T07:32:15.218867+0000","last_clean":"2024-03-06T07:32:15.218867+0000","last_became_active":"2024-03-06T07:32:15.212945+0000","last_became_peered":"2024-03-06T07:32:15.212945+0000","last_unstale":"2024-03-06T07:32:15.218867+0000","last_undegraded":"2024-03-06T07:32:15.218867+0000","last_fullsized":"2024-03-06T07:32:15.218867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:02:53.629535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198343+0000","last_change":"2024-03-06T07:32:15.191295+0000","last_active":"2024-03-06T07:32:16.198343+0000","last_peered":"2024-03-06T07:32:16.198343+0000","last_clean":"2024-03-06T07:32:16.198343+0000","last_became_active":"2024-03-06T07:32:15.190985+0000","last_became_peered":"2024-03-06T07:32:15.190985+0000","last_unstale":"2024-03-06T07:32:16.198343+0000","last_undegraded":"2024-03-06T07:32:16.198343+0000","last_fullsized":"2024-03-06T07:32:16.198343+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:25.780590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.204670+0000","last_change":"2024-03-06T07:32:15.197293+0000","last_active":"2024-03-06T07:32:15.204670+0000","last_peered":"2024-03-06T07:32:15.204670+0000","last_clean":"2024-03-06T07:32:15.204670+0000","last_became_active":"2024-03-06T07:32:15.192219+0000","last_became_peered":"2024-03-06T07:32:15.192219+0000","last_unstale":"2024-03-06T07:32:15.204670+0000","last_undegraded":"2024-03-06T07:32:15.204670+0000","last_fullsized":"2024-03-06T07:32:15.204670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:17:36.488920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236140+0000","last_change":"2024-03-06T07:32:15.210767+0000","last_active":"2024-03-06T07:32:16.236140+0000","last_peered":"2024-03-06T07:32:16.236140+0000","last_clean":"2024-03-06T07:32:16.236140+0000","last_became_active":"2024-03-06T07:32:15.206434+0000","last_became_peered":"2024-03-06T07:32:15.206434+0000","last_unstale":"2024-03-06T07:32:16.236140+0000","last_undegraded":"2024-03-06T07:32:16.236140+0000","last_fullsized":"2024-03-06T07:32:16.236140+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:33.623254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198181+0000","last_change":"2024-03-06T07:32:15.197471+0000","last_active":"2024-03-06T07:32:16.198181+0000","last_peered":"2024-03-06T07:32:16.198181+0000","last_clean":"2024-03-06T07:32:16.198181+0000","last_became_active":"2024-03-06T07:32:15.197200+0000","last_became_peered":"2024-03-06T07:32:15.197200+0000","last_unstale":"2024-03-06T07:32:16.198181+0000","last_undegraded":"2024-03-06T07:32:16.198181+0000","last_fullsized":"2024-03-06T07:32:16.198181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:45.983922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235580+0000","last_change":"2024-03-06T07:32:15.206010+0000","last_active":"2024-03-06T07:32:16.235580+0000","last_peered":"2024-03-06T07:32:16.235580+0000","last_clean":"2024-03-06T07:32:16.235580+0000","last_became_active":"2024-03-06T07:32:15.205581+0000","last_became_peered":"2024-03-06T07:32:15.205581+0000","last_unstale":"2024-03-06T07:32:16.235580+0000","last_undegraded":"2024-03-06T07:32:16.235580+0000","last_fullsized":"2024-03-06T07:32:16.235580+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:15.105303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236227+0000","last_change":"2024-03-06T07:32:15.201009+0000","last_active":"2024-03-06T07:32:16.236227+0000","last_peered":"2024-03-06T07:32:16.236227+0000","last_clean":"2024-03-06T07:32:16.236227+0000","last_became_active":"2024-03-06T07:32:15.200680+0000","last_became_peered":"2024-03-06T07:32:15.200680+0000","last_unstale":"2024-03-06T07:32:16.236227+0000","last_undegraded":"2024-03-06T07:32:16.236227+0000","last_fullsized":"2024-03-06T07:32:16.236227+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:19.334348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.219609+0000","last_change":"2024-03-06T07:32:15.215897+0000","last_active":"2024-03-06T07:32:15.219609+0000","last_peered":"2024-03-06T07:32:15.219609+0000","last_clean":"2024-03-06T07:32:15.219609+0000","last_became_active":"2024-03-06T07:32:15.215623+0000","last_became_peered":"2024-03-06T07:32:15.215623+0000","last_unstale":"2024-03-06T07:32:15.219609+0000","last_undegraded":"2024-03-06T07:32:15.219609+0000","last_fullsized":"2024-03-06T07:32:15.219609+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.042864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200767+0000","last_change":"2024-03-06T07:32:15.200504+0000","last_active":"2024-03-06T07:32:16.200767+0000","last_peered":"2024-03-06T07:32:16.200767+0000","last_clean":"2024-03-06T07:32:16.200767+0000","last_became_active":"2024-03-06T07:32:15.197885+0000","last_became_peered":"2024-03-06T07:32:15.197885+0000","last_unstale":"2024-03-06T07:32:16.200767+0000","last_undegraded":"2024-03-06T07:32:16.200767+0000","last_fullsized":"2024-03-06T07:32:16.200767+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:57.737289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218828+0000","last_change":"2024-03-06T07:32:15.217030+0000","last_active":"2024-03-06T07:32:15.218828+0000","last_peered":"2024-03-06T07:32:15.218828+0000","last_clean":"2024-03-06T07:32:15.218828+0000","last_became_active":"2024-03-06T07:32:15.211487+0000","last_became_peered":"2024-03-06T07:32:15.211487+0000","last_unstale":"2024-03-06T07:32:15.218828+0000","last_undegraded":"2024-03-06T07:32:15.218828+0000","last_fullsized":"2024-03-06T07:32:15.218828+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:33.623254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.215796+0000","last_change":"2024-03-06T07:32:15.207961+0000","last_active":"2024-03-06T07:32:15.215796+0000","last_peered":"2024-03-06T07:32:15.215796+0000","last_clean":"2024-03-06T07:32:15.215796+0000","last_became_active":"2024-03-06T07:32:15.207626+0000","last_became_peered":"2024-03-06T07:32:15.207626+0000","last_unstale":"2024-03-06T07:32:15.215796+0000","last_undegraded":"2024-03-06T07:32:15.215796+0000","last_fullsized":"2024-03-06T07:32:15.215796+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:59.627690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200821+0000","last_change":"2024-03-06T07:32:15.194949+0000","last_active":"2024-03-06T07:32:16.200821+0000","last_peered":"2024-03-06T07:32:16.200821+0000","last_clean":"2024-03-06T07:32:16.200821+0000","last_became_active":"2024-03-06T07:32:15.192951+0000","last_became_peered":"2024-03-06T07:32:15.192951+0000","last_unstale":"2024-03-06T07:32:16.200821+0000","last_undegraded":"2024-03-06T07:32:16.200821+0000","last_fullsized":"2024-03-06T07:32:16.200821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:30:26.394546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197643+0000","last_change":"2024-03-06T07:32:15.199167+0000","last_active":"2024-03-06T07:32:16.197643+0000","last_peered":"2024-03-06T07:32:16.197643+0000","last_clean":"2024-03-06T07:32:16.197643+0000","last_became_active":"2024-03-06T07:32:15.196902+0000","last_became_peered":"2024-03-06T07:32:15.196902+0000","last_unstale":"2024-03-06T07:32:16.197643+0000","last_undegraded":"2024-03-06T07:32:16.197643+0000","last_fullsized":"2024-03-06T07:32:16.197643+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:01.789702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.221210+0000","last_change":"2024-03-06T07:32:15.219009+0000","last_active":"2024-03-06T07:32:15.221210+0000","last_peered":"2024-03-06T07:32:15.221210+0000","last_clean":"2024-03-06T07:32:15.221210+0000","last_became_active":"2024-03-06T07:32:15.218214+0000","last_became_peered":"2024-03-06T07:32:15.218214+0000","last_unstale":"2024-03-06T07:32:15.221210+0000","last_undegraded":"2024-03-06T07:32:15.221210+0000","last_fullsized":"2024-03-06T07:32:15.221210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:42:59.627690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.210340+0000","last_change":"2024-03-06T07:32:15.204451+0000","last_active":"2024-03-06T07:32:15.210340+0000","last_peered":"2024-03-06T07:32:15.210340+0000","last_clean":"2024-03-06T07:32:15.210340+0000","last_became_active":"2024-03-06T07:32:15.204089+0000","last_became_peered":"2024-03-06T07:32:15.204089+0000","last_unstale":"2024-03-06T07:32:15.210340+0000","last_undegraded":"2024-03-06T07:32:15.210340+0000","last_fullsized":"2024-03-06T07:32:15.210340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:02.825297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199714+0000","last_change":"2024-03-06T07:32:15.212985+0000","last_active":"2024-03-06T07:32:16.199714+0000","last_peered":"2024-03-06T07:32:16.199714+0000","last_clean":"2024-03-06T07:32:16.199714+0000","last_became_active":"2024-03-06T07:32:15.212708+0000","last_became_peered":"2024-03-06T07:32:15.212708+0000","last_unstale":"2024-03-06T07:32:16.199714+0000","last_undegraded":"2024-03-06T07:32:16.199714+0000","last_fullsized":"2024-03-06T07:32:16.199714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:39:03.994277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.201325+0000","last_change":"2024-03-06T07:32:15.196520+0000","last_active":"2024-03-06T07:32:15.201325+0000","last_peered":"2024-03-06T07:32:15.201325+0000","last_clean":"2024-03-06T07:32:15.201325+0000","last_became_active":"2024-03-06T07:32:15.193543+0000","last_became_peered":"2024-03-06T07:32:15.193543+0000","last_unstale":"2024-03-06T07:32:15.201325+0000","last_undegraded":"2024-03-06T07:32:15.201325+0000","last_fullsized":"2024-03-06T07:32:15.201325+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:43.466481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198491+0000","last_change":"2024-03-06T07:32:15.190490+0000","last_active":"2024-03-06T07:32:16.198491+0000","last_peered":"2024-03-06T07:32:16.198491+0000","last_clean":"2024-03-06T07:32:16.198491+0000","last_became_active":"2024-03-06T07:32:15.189029+0000","last_became_peered":"2024-03-06T07:32:15.189029+0000","last_unstale":"2024-03-06T07:32:16.198491+0000","last_undegraded":"2024-03-06T07:32:16.198491+0000","last_fullsized":"2024-03-06T07:32:16.198491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:23.488199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201656+0000","last_change":"2024-03-06T07:32:15.190779+0000","last_active":"2024-03-06T07:32:16.201656+0000","last_peered":"2024-03-06T07:32:16.201656+0000","last_clean":"2024-03-06T07:32:16.201656+0000","last_became_active":"2024-03-06T07:32:15.189840+0000","last_became_peered":"2024-03-06T07:32:15.189840+0000","last_unstale":"2024-03-06T07:32:16.201656+0000","last_undegraded":"2024-03-06T07:32:16.201656+0000","last_fullsized":"2024-03-06T07:32:16.201656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:41.646060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199770+0000","last_change":"2024-03-06T07:32:15.197498+0000","last_active":"2024-03-06T07:32:16.199770+0000","last_peered":"2024-03-06T07:32:16.199770+0000","last_clean":"2024-03-06T07:32:16.199770+0000","last_became_active":"2024-03-06T07:32:15.194902+0000","last_became_peered":"2024-03-06T07:32:15.194902+0000","last_unstale":"2024-03-06T07:32:16.199770+0000","last_undegraded":"2024-03-06T07:32:16.199770+0000","last_fullsized":"2024-03-06T07:32:16.199770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:19:50.313213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217483+0000","last_change":"2024-03-06T07:32:15.210971+0000","last_active":"2024-03-06T07:32:15.217483+0000","last_peered":"2024-03-06T07:32:15.217483+0000","last_clean":"2024-03-06T07:32:15.217483+0000","last_became_active":"2024-03-06T07:32:15.210383+0000","last_became_peered":"2024-03-06T07:32:15.210383+0000","last_unstale":"2024-03-06T07:32:15.217483+0000","last_undegraded":"2024-03-06T07:32:15.217483+0000","last_fullsized":"2024-03-06T07:32:15.217483+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:22.726113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198808+0000","last_change":"2024-03-06T07:32:15.187112+0000","last_active":"2024-03-06T07:32:16.198808+0000","last_peered":"2024-03-06T07:32:16.198808+0000","last_clean":"2024-03-06T07:32:16.198808+0000","last_became_active":"2024-03-06T07:32:15.186775+0000","last_became_peered":"2024-03-06T07:32:15.186775+0000","last_unstale":"2024-03-06T07:32:16.198808+0000","last_undegraded":"2024-03-06T07:32:16.198808+0000","last_fullsized":"2024-03-06T07:32:16.198808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:45.616728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201180+0000","last_change":"2024-03-06T07:32:15.214462+0000","last_active":"2024-03-06T07:32:16.201180+0000","last_peered":"2024-03-06T07:32:16.201180+0000","last_clean":"2024-03-06T07:32:16.201180+0000","last_became_active":"2024-03-06T07:32:15.214192+0000","last_became_peered":"2024-03-06T07:32:15.214192+0000","last_unstale":"2024-03-06T07:32:16.201180+0000","last_undegraded":"2024-03-06T07:32:16.201180+0000","last_fullsized":"2024-03-06T07:32:16.201180+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:32:44.604593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.213255+0000","last_change":"2024-03-06T07:32:15.204063+0000","last_active":"2024-03-06T07:32:15.213255+0000","last_peered":"2024-03-06T07:32:15.213255+0000","last_clean":"2024-03-06T07:32:15.213255+0000","last_became_active":"2024-03-06T07:32:15.203792+0000","last_became_peered":"2024-03-06T07:32:15.203792+0000","last_unstale":"2024-03-06T07:32:15.213255+0000","last_undegraded":"2024-03-06T07:32:15.213255+0000","last_fullsized":"2024-03-06T07:32:15.213255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:41.113175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.212563+0000","last_change":"2024-03-06T07:32:15.210353+0000","last_active":"2024-03-06T07:32:15.212563+0000","last_peered":"2024-03-06T07:32:15.212563+0000","last_clean":"2024-03-06T07:32:15.212563+0000","last_became_active":"2024-03-06T07:32:15.210008+0000","last_became_peered":"2024-03-06T07:32:15.210008+0000","last_unstale":"2024-03-06T07:32:15.212563+0000","last_undegraded":"2024-03-06T07:32:15.212563+0000","last_fullsized":"2024-03-06T07:32:15.212563+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:21:41.113175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216925+0000","last_change":"2024-03-06T07:32:15.205793+0000","last_active":"2024-03-06T07:32:15.216925+0000","last_peered":"2024-03-06T07:32:15.216925+0000","last_clean":"2024-03-06T07:32:15.216925+0000","last_became_active":"2024-03-06T07:32:15.205371+0000","last_became_peered":"2024-03-06T07:32:15.205371+0000","last_unstale":"2024-03-06T07:32:15.216925+0000","last_undegraded":"2024-03-06T07:32:15.216925+0000","last_fullsized":"2024-03-06T07:32:15.216925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:11.769674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200861+0000","last_change":"2024-03-06T07:32:15.192171+0000","last_active":"2024-03-06T07:32:16.200861+0000","last_peered":"2024-03-06T07:32:16.200861+0000","last_clean":"2024-03-06T07:32:16.200861+0000","last_became_active":"2024-03-06T07:32:15.190338+0000","last_became_peered":"2024-03-06T07:32:15.190338+0000","last_unstale":"2024-03-06T07:32:16.200861+0000","last_undegraded":"2024-03-06T07:32:16.200861+0000","last_fullsized":"2024-03-06T07:32:16.200861+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:26.109997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199815+0000","last_change":"2024-03-06T07:32:15.188373+0000","last_active":"2024-03-06T07:32:16.199815+0000","last_peered":"2024-03-06T07:32:16.199815+0000","last_clean":"2024-03-06T07:32:16.199815+0000","last_became_active":"2024-03-06T07:32:15.187333+0000","last_became_peered":"2024-03-06T07:32:15.187333+0000","last_unstale":"2024-03-06T07:32:16.199815+0000","last_undegraded":"2024-03-06T07:32:16.199815+0000","last_fullsized":"2024-03-06T07:32:16.199815+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:17.082067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.199688+0000","last_change":"2024-03-06T07:32:15.190650+0000","last_active":"2024-03-06T07:32:15.199688+0000","last_peered":"2024-03-06T07:32:15.199688+0000","last_clean":"2024-03-06T07:32:15.199688+0000","last_became_active":"2024-03-06T07:32:15.190241+0000","last_became_peered":"2024-03-06T07:32:15.190241+0000","last_unstale":"2024-03-06T07:32:15.199688+0000","last_undegraded":"2024-03-06T07:32:15.199688+0000","last_fullsized":"2024-03-06T07:32:15.199688+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:19.073857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.208754+0000","last_change":"2024-03-06T07:32:15.200748+0000","last_active":"2024-03-06T07:32:15.208754+0000","last_peered":"2024-03-06T07:32:15.208754+0000","last_clean":"2024-03-06T07:32:15.208754+0000","last_became_active":"2024-03-06T07:32:15.200462+0000","last_became_peered":"2024-03-06T07:32:15.200462+0000","last_unstale":"2024-03-06T07:32:15.208754+0000","last_undegraded":"2024-03-06T07:32:15.208754+0000","last_fullsized":"2024-03-06T07:32:15.208754+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:09.252388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.212256+0000","last_change":"2024-03-06T07:32:15.209963+0000","last_active":"2024-03-06T07:32:15.212256+0000","last_peered":"2024-03-06T07:32:15.212256+0000","last_clean":"2024-03-06T07:32:15.212256+0000","last_became_active":"2024-03-06T07:32:15.206491+0000","last_became_peered":"2024-03-06T07:32:15.206491+0000","last_unstale":"2024-03-06T07:32:15.212256+0000","last_undegraded":"2024-03-06T07:32:15.212256+0000","last_fullsized":"2024-03-06T07:32:15.212256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:53.869423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.223466+0000","last_change":"2024-03-06T07:32:14.172703+0000","last_active":"2024-03-06T07:32:15.223466+0000","last_peered":"2024-03-06T07:32:15.223466+0000","last_clean":"2024-03-06T07:32:15.223466+0000","last_became_active":"2024-03-06T07:32:14.171546+0000","last_became_peered":"2024-03-06T07:32:14.171546+0000","last_unstale":"2024-03-06T07:32:15.223466+0000","last_undegraded":"2024-03-06T07:32:15.223466+0000","last_fullsized":"2024-03-06T07:32:15.223466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:22.511086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.214897+0000","last_change":"2024-03-06T07:32:15.213170+0000","last_active":"2024-03-06T07:32:15.214897+0000","last_peered":"2024-03-06T07:32:15.214897+0000","last_clean":"2024-03-06T07:32:15.214897+0000","last_became_active":"2024-03-06T07:32:15.212730+0000","last_became_peered":"2024-03-06T07:32:15.212730+0000","last_unstale":"2024-03-06T07:32:15.214897+0000","last_undegraded":"2024-03-06T07:32:15.214897+0000","last_fullsized":"2024-03-06T07:32:15.214897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:43.466481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.215681+0000","last_change":"2024-03-06T07:32:14.170974+0000","last_active":"2024-03-06T07:32:15.215681+0000","last_peered":"2024-03-06T07:32:15.215681+0000","last_clean":"2024-03-06T07:32:15.215681+0000","last_became_active":"2024-03-06T07:32:14.170643+0000","last_became_peered":"2024-03-06T07:32:14.170643+0000","last_unstale":"2024-03-06T07:32:15.215681+0000","last_undegraded":"2024-03-06T07:32:15.215681+0000","last_fullsized":"2024-03-06T07:32:15.215681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:26.115845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216952+0000","last_change":"2024-03-06T07:32:15.213103+0000","last_active":"2024-03-06T07:32:15.216952+0000","last_peered":"2024-03-06T07:32:15.216952+0000","last_clean":"2024-03-06T07:32:15.216952+0000","last_became_active":"2024-03-06T07:32:15.212881+0000","last_became_peered":"2024-03-06T07:32:15.212881+0000","last_unstale":"2024-03-06T07:32:15.216952+0000","last_undegraded":"2024-03-06T07:32:15.216952+0000","last_fullsized":"2024-03-06T07:32:15.216952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:15.105303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218948+0000","last_change":"2024-03-06T07:32:14.169744+0000","last_active":"2024-03-06T07:32:15.218948+0000","last_peered":"2024-03-06T07:32:15.218948+0000","last_clean":"2024-03-06T07:32:15.218948+0000","last_became_active":"2024-03-06T07:32:14.169467+0000","last_became_peered":"2024-03-06T07:32:14.169467+0000","last_unstale":"2024-03-06T07:32:15.218948+0000","last_undegraded":"2024-03-06T07:32:15.218948+0000","last_fullsized":"2024-03-06T07:32:15.218948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:13.533245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.214623+0000","last_change":"2024-03-06T07:32:15.208110+0000","last_active":"2024-03-06T07:32:15.214623+0000","last_peered":"2024-03-06T07:32:15.214623+0000","last_clean":"2024-03-06T07:32:15.214623+0000","last_became_active":"2024-03-06T07:32:15.207467+0000","last_became_peered":"2024-03-06T07:32:15.207467+0000","last_unstale":"2024-03-06T07:32:15.214623+0000","last_undegraded":"2024-03-06T07:32:15.214623+0000","last_fullsized":"2024-03-06T07:32:15.214623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:24.626631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218879+0000","last_change":"2024-03-06T07:32:14.169448+0000","last_active":"2024-03-06T07:32:15.218879+0000","last_peered":"2024-03-06T07:32:15.218879+0000","last_clean":"2024-03-06T07:32:15.218879+0000","last_became_active":"2024-03-06T07:32:14.169156+0000","last_became_peered":"2024-03-06T07:32:14.169156+0000","last_unstale":"2024-03-06T07:32:15.218879+0000","last_undegraded":"2024-03-06T07:32:15.218879+0000","last_fullsized":"2024-03-06T07:32:15.218879+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:45.752210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199980+0000","last_change":"2024-03-06T07:32:15.188669+0000","last_active":"2024-03-06T07:32:16.199980+0000","last_peered":"2024-03-06T07:32:16.199980+0000","last_clean":"2024-03-06T07:32:16.199980+0000","last_became_active":"2024-03-06T07:32:15.187873+0000","last_became_peered":"2024-03-06T07:32:15.187873+0000","last_unstale":"2024-03-06T07:32:16.199980+0000","last_undegraded":"2024-03-06T07:32:16.199980+0000","last_fullsized":"2024-03-06T07:32:16.199980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:04.108562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.221291+0000","last_change":"2024-03-06T07:32:14.168081+0000","last_active":"2024-03-06T07:32:15.221291+0000","last_peered":"2024-03-06T07:32:15.221291+0000","last_clean":"2024-03-06T07:32:15.221291+0000","last_became_active":"2024-03-06T07:32:14.167753+0000","last_became_peered":"2024-03-06T07:32:14.167753+0000","last_unstale":"2024-03-06T07:32:15.221291+0000","last_undegraded":"2024-03-06T07:32:15.221291+0000","last_fullsized":"2024-03-06T07:32:15.221291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:58.103349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.200340+0000","last_change":"2024-03-06T07:32:15.192239+0000","last_active":"2024-03-06T07:32:15.200340+0000","last_peered":"2024-03-06T07:32:15.200340+0000","last_clean":"2024-03-06T07:32:15.200340+0000","last_became_active":"2024-03-06T07:32:15.191967+0000","last_became_peered":"2024-03-06T07:32:15.191967+0000","last_unstale":"2024-03-06T07:32:15.200340+0000","last_undegraded":"2024-03-06T07:32:15.200340+0000","last_fullsized":"2024-03-06T07:32:15.200340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:26.028733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.215851+0000","last_change":"2024-03-06T07:32:14.169365+0000","last_active":"2024-03-06T07:32:15.215851+0000","last_peered":"2024-03-06T07:32:15.215851+0000","last_clean":"2024-03-06T07:32:15.215851+0000","last_became_active":"2024-03-06T07:32:14.168747+0000","last_became_peered":"2024-03-06T07:32:14.168747+0000","last_unstale":"2024-03-06T07:32:15.215851+0000","last_undegraded":"2024-03-06T07:32:15.215851+0000","last_fullsized":"2024-03-06T07:32:15.215851+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:38:48.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.211395+0000","last_change":"2024-03-06T07:32:15.202616+0000","last_active":"2024-03-06T07:32:15.211395+0000","last_peered":"2024-03-06T07:32:15.211395+0000","last_clean":"2024-03-06T07:32:15.211395+0000","last_became_active":"2024-03-06T07:32:15.202326+0000","last_became_peered":"2024-03-06T07:32:15.202326+0000","last_unstale":"2024-03-06T07:32:15.211395+0000","last_undegraded":"2024-03-06T07:32:15.211395+0000","last_fullsized":"2024-03-06T07:32:15.211395+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:15.957458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.186957+0000","last_change":"2024-03-06T07:32:14.169483+0000","last_active":"2024-03-06T07:32:15.186957+0000","last_peered":"2024-03-06T07:32:15.186957+0000","last_clean":"2024-03-06T07:32:15.186957+0000","last_became_active":"2024-03-06T07:32:14.169193+0000","last_became_peered":"2024-03-06T07:32:14.169193+0000","last_unstale":"2024-03-06T07:32:15.186957+0000","last_undegraded":"2024-03-06T07:32:15.186957+0000","last_fullsized":"2024-03-06T07:32:15.186957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:10.754870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197905+0000","last_change":"2024-03-06T07:32:15.198297+0000","last_active":"2024-03-06T07:32:16.197905+0000","last_peered":"2024-03-06T07:32:16.197905+0000","last_clean":"2024-03-06T07:32:16.197905+0000","last_became_active":"2024-03-06T07:32:15.197613+0000","last_became_peered":"2024-03-06T07:32:15.197613+0000","last_unstale":"2024-03-06T07:32:16.197905+0000","last_undegraded":"2024-03-06T07:32:16.197905+0000","last_fullsized":"2024-03-06T07:32:16.197905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:09.413206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197838+0000","last_change":"2024-03-06T07:32:14.165392+0000","last_active":"2024-03-06T07:32:16.197838+0000","last_peered":"2024-03-06T07:32:16.197838+0000","last_clean":"2024-03-06T07:32:16.197838+0000","last_became_active":"2024-03-06T07:32:14.165096+0000","last_became_peered":"2024-03-06T07:32:14.165096+0000","last_unstale":"2024-03-06T07:32:16.197838+0000","last_undegraded":"2024-03-06T07:32:16.197838+0000","last_fullsized":"2024-03-06T07:32:16.197838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:24.765787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198077+0000","last_change":"2024-03-06T07:32:15.194238+0000","last_active":"2024-03-06T07:32:16.198077+0000","last_peered":"2024-03-06T07:32:16.198077+0000","last_clean":"2024-03-06T07:32:16.198077+0000","last_became_active":"2024-03-06T07:32:15.193905+0000","last_became_peered":"2024-03-06T07:32:15.193905+0000","last_unstale":"2024-03-06T07:32:16.198077+0000","last_undegraded":"2024-03-06T07:32:16.198077+0000","last_fullsized":"2024-03-06T07:32:16.198077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:06.881711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.198314+0000","last_change":"2024-03-06T07:32:14.171996+0000","last_active":"2024-03-06T07:32:15.198314+0000","last_peered":"2024-03-06T07:32:15.198314+0000","last_clean":"2024-03-06T07:32:15.198314+0000","last_became_active":"2024-03-06T07:32:14.169695+0000","last_became_peered":"2024-03-06T07:32:14.169695+0000","last_unstale":"2024-03-06T07:32:15.198314+0000","last_undegraded":"2024-03-06T07:32:15.198314+0000","last_fullsized":"2024-03-06T07:32:15.198314+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:42.577277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236748+0000","last_change":"2024-03-06T07:32:15.206844+0000","last_active":"2024-03-06T07:32:16.236748+0000","last_peered":"2024-03-06T07:32:16.236748+0000","last_clean":"2024-03-06T07:32:16.236748+0000","last_became_active":"2024-03-06T07:32:15.206264+0000","last_became_peered":"2024-03-06T07:32:15.206264+0000","last_unstale":"2024-03-06T07:32:16.236748+0000","last_undegraded":"2024-03-06T07:32:16.236748+0000","last_fullsized":"2024-03-06T07:32:16.236748+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:58.035445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.189698+0000","last_change":"2024-03-06T07:32:14.164636+0000","last_active":"2024-03-06T07:32:15.189698+0000","last_peered":"2024-03-06T07:32:15.189698+0000","last_clean":"2024-03-06T07:32:15.189698+0000","last_became_active":"2024-03-06T07:32:14.164289+0000","last_became_peered":"2024-03-06T07:32:14.164289+0000","last_unstale":"2024-03-06T07:32:15.189698+0000","last_undegraded":"2024-03-06T07:32:15.189698+0000","last_fullsized":"2024-03-06T07:32:15.189698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:12:10.564384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217907+0000","last_change":"2024-03-06T07:32:15.213656+0000","last_active":"2024-03-06T07:32:15.217907+0000","last_peered":"2024-03-06T07:32:15.217907+0000","last_clean":"2024-03-06T07:32:15.217907+0000","last_became_active":"2024-03-06T07:32:15.213389+0000","last_became_peered":"2024-03-06T07:32:15.213389+0000","last_unstale":"2024-03-06T07:32:15.217907+0000","last_undegraded":"2024-03-06T07:32:15.217907+0000","last_fullsized":"2024-03-06T07:32:15.217907+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:19.334348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.223181+0000","last_change":"2024-03-06T07:32:14.165114+0000","last_active":"2024-03-06T07:32:15.223181+0000","last_peered":"2024-03-06T07:32:15.223181+0000","last_clean":"2024-03-06T07:32:15.223181+0000","last_became_active":"2024-03-06T07:32:14.164791+0000","last_became_peered":"2024-03-06T07:32:14.164791+0000","last_unstale":"2024-03-06T07:32:15.223181+0000","last_undegraded":"2024-03-06T07:32:15.223181+0000","last_fullsized":"2024-03-06T07:32:15.223181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:16.146640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.209745+0000","last_change":"2024-03-06T07:32:15.199594+0000","last_active":"2024-03-06T07:32:15.209745+0000","last_peered":"2024-03-06T07:32:15.209745+0000","last_clean":"2024-03-06T07:32:15.209745+0000","last_became_active":"2024-03-06T07:32:15.199267+0000","last_became_peered":"2024-03-06T07:32:15.199267+0000","last_unstale":"2024-03-06T07:32:15.209745+0000","last_undegraded":"2024-03-06T07:32:15.209745+0000","last_fullsized":"2024-03-06T07:32:15.209745+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:26.353270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236015+0000","last_change":"2024-03-06T07:32:14.174648+0000","last_active":"2024-03-06T07:32:16.236015+0000","last_peered":"2024-03-06T07:32:16.236015+0000","last_clean":"2024-03-06T07:32:16.236015+0000","last_became_active":"2024-03-06T07:32:14.173628+0000","last_became_peered":"2024-03-06T07:32:14.173628+0000","last_unstale":"2024-03-06T07:32:16.236015+0000","last_undegraded":"2024-03-06T07:32:16.236015+0000","last_fullsized":"2024-03-06T07:32:16.236015+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:22.511086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.204408+0000","last_change":"2024-03-06T07:32:15.203407+0000","last_active":"2024-03-06T07:32:15.204408+0000","last_peered":"2024-03-06T07:32:15.204408+0000","last_clean":"2024-03-06T07:32:15.204408+0000","last_became_active":"2024-03-06T07:32:15.202655+0000","last_became_peered":"2024-03-06T07:32:15.202655+0000","last_unstale":"2024-03-06T07:32:15.204408+0000","last_undegraded":"2024-03-06T07:32:15.204408+0000","last_fullsized":"2024-03-06T07:32:15.204408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:33.623254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201168+0000","last_change":"2024-03-06T07:32:14.163706+0000","last_active":"2024-03-06T07:32:16.201168+0000","last_peered":"2024-03-06T07:32:16.201168+0000","last_clean":"2024-03-06T07:32:16.201168+0000","last_became_active":"2024-03-06T07:32:14.161664+0000","last_became_peered":"2024-03-06T07:32:14.161664+0000","last_unstale":"2024-03-06T07:32:16.201168+0000","last_undegraded":"2024-03-06T07:32:16.201168+0000","last_fullsized":"2024-03-06T07:32:16.201168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:44.601924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.213216+0000","last_change":"2024-03-06T07:32:15.203637+0000","last_active":"2024-03-06T07:32:15.213216+0000","last_peered":"2024-03-06T07:32:15.213216+0000","last_clean":"2024-03-06T07:32:15.213216+0000","last_became_active":"2024-03-06T07:32:15.203331+0000","last_became_peered":"2024-03-06T07:32:15.203331+0000","last_unstale":"2024-03-06T07:32:15.213216+0000","last_undegraded":"2024-03-06T07:32:15.213216+0000","last_fullsized":"2024-03-06T07:32:15.213216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:53.869423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.198344+0000","last_change":"2024-03-06T07:32:14.166677+0000","last_active":"2024-03-06T07:32:16.198344+0000","last_peered":"2024-03-06T07:32:16.198344+0000","last_clean":"2024-03-06T07:32:16.198344+0000","last_became_active":"2024-03-06T07:32:14.165397+0000","last_became_peered":"2024-03-06T07:32:14.165397+0000","last_unstale":"2024-03-06T07:32:16.198344+0000","last_undegraded":"2024-03-06T07:32:16.198344+0000","last_fullsized":"2024-03-06T07:32:16.198344+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:41:47.069241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197707+0000","last_change":"2024-03-06T07:32:15.187239+0000","last_active":"2024-03-06T07:32:16.197707+0000","last_peered":"2024-03-06T07:32:16.197707+0000","last_clean":"2024-03-06T07:32:16.197707+0000","last_became_active":"2024-03-06T07:32:15.186439+0000","last_became_peered":"2024-03-06T07:32:15.186439+0000","last_unstale":"2024-03-06T07:32:16.197707+0000","last_undegraded":"2024-03-06T07:32:16.197707+0000","last_fullsized":"2024-03-06T07:32:16.197707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:43.178774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235720+0000","last_change":"2024-03-06T07:32:14.171528+0000","last_active":"2024-03-06T07:32:16.235720+0000","last_peered":"2024-03-06T07:32:16.235720+0000","last_clean":"2024-03-06T07:32:16.235720+0000","last_became_active":"2024-03-06T07:32:14.169630+0000","last_became_peered":"2024-03-06T07:32:14.169630+0000","last_unstale":"2024-03-06T07:32:16.235720+0000","last_undegraded":"2024-03-06T07:32:16.235720+0000","last_fullsized":"2024-03-06T07:32:16.235720+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:42.577277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199410+0000","last_change":"2024-03-06T07:32:15.214251+0000","last_active":"2024-03-06T07:32:16.199410+0000","last_peered":"2024-03-06T07:32:16.199410+0000","last_clean":"2024-03-06T07:32:16.199410+0000","last_became_active":"2024-03-06T07:32:15.213894+0000","last_became_peered":"2024-03-06T07:32:15.213894+0000","last_unstale":"2024-03-06T07:32:16.199410+0000","last_undegraded":"2024-03-06T07:32:16.199410+0000","last_fullsized":"2024-03-06T07:32:16.199410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:25.580317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.220450+0000","last_change":"2024-03-06T07:32:14.169121+0000","last_active":"2024-03-06T07:32:15.220450+0000","last_peered":"2024-03-06T07:32:15.220450+0000","last_clean":"2024-03-06T07:32:15.220450+0000","last_became_active":"2024-03-06T07:32:14.168568+0000","last_became_peered":"2024-03-06T07:32:14.168568+0000","last_unstale":"2024-03-06T07:32:15.220450+0000","last_undegraded":"2024-03-06T07:32:15.220450+0000","last_fullsized":"2024-03-06T07:32:15.220450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:06.630348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216757+0000","last_change":"2024-03-06T07:32:15.208218+0000","last_active":"2024-03-06T07:32:15.216757+0000","last_peered":"2024-03-06T07:32:15.216757+0000","last_clean":"2024-03-06T07:32:15.216757+0000","last_became_active":"2024-03-06T07:32:15.207701+0000","last_became_peered":"2024-03-06T07:32:15.207701+0000","last_unstale":"2024-03-06T07:32:15.216757+0000","last_undegraded":"2024-03-06T07:32:15.216757+0000","last_fullsized":"2024-03-06T07:32:15.216757+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:53.002096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.223010+0000","last_change":"2024-03-06T07:32:14.169336+0000","last_active":"2024-03-06T07:32:15.223010+0000","last_peered":"2024-03-06T07:32:15.223010+0000","last_clean":"2024-03-06T07:32:15.223010+0000","last_became_active":"2024-03-06T07:32:14.169028+0000","last_became_peered":"2024-03-06T07:32:14.169028+0000","last_unstale":"2024-03-06T07:32:15.223010+0000","last_undegraded":"2024-03-06T07:32:15.223010+0000","last_fullsized":"2024-03-06T07:32:15.223010+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:58.271604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.220726+0000","last_change":"2024-03-06T07:32:15.211587+0000","last_active":"2024-03-06T07:32:15.220726+0000","last_peered":"2024-03-06T07:32:15.220726+0000","last_clean":"2024-03-06T07:32:15.220726+0000","last_became_active":"2024-03-06T07:32:15.211149+0000","last_became_peered":"2024-03-06T07:32:15.211149+0000","last_unstale":"2024-03-06T07:32:15.220726+0000","last_undegraded":"2024-03-06T07:32:15.220726+0000","last_fullsized":"2024-03-06T07:32:15.220726+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:41:14.548048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217185+0000","last_change":"2024-03-06T07:32:14.160976+0000","last_active":"2024-03-06T07:32:15.217185+0000","last_peered":"2024-03-06T07:32:15.217185+0000","last_clean":"2024-03-06T07:32:15.217185+0000","last_became_active":"2024-03-06T07:32:14.160658+0000","last_became_peered":"2024-03-06T07:32:14.160658+0000","last_unstale":"2024-03-06T07:32:15.217185+0000","last_undegraded":"2024-03-06T07:32:15.217185+0000","last_fullsized":"2024-03-06T07:32:15.217185+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:31:14.090499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.209765+0000","last_change":"2024-03-06T07:32:15.199472+0000","last_active":"2024-03-06T07:32:15.209765+0000","last_peered":"2024-03-06T07:32:15.209765+0000","last_clean":"2024-03-06T07:32:15.209765+0000","last_became_active":"2024-03-06T07:32:15.199141+0000","last_became_peered":"2024-03-06T07:32:15.199141+0000","last_unstale":"2024-03-06T07:32:15.209765+0000","last_undegraded":"2024-03-06T07:32:15.209765+0000","last_fullsized":"2024-03-06T07:32:15.209765+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:16.743592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187956+0000","last_change":"2024-03-06T07:32:14.171346+0000","last_active":"2024-03-06T07:32:15.187956+0000","last_peered":"2024-03-06T07:32:15.187956+0000","last_clean":"2024-03-06T07:32:15.187956+0000","last_became_active":"2024-03-06T07:32:14.171055+0000","last_became_peered":"2024-03-06T07:32:14.171055+0000","last_unstale":"2024-03-06T07:32:15.187956+0000","last_undegraded":"2024-03-06T07:32:15.187956+0000","last_fullsized":"2024-03-06T07:32:15.187956+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:02:21.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201257+0000","last_change":"2024-03-06T07:32:15.194967+0000","last_active":"2024-03-06T07:32:16.201257+0000","last_peered":"2024-03-06T07:32:16.201257+0000","last_clean":"2024-03-06T07:32:16.201257+0000","last_became_active":"2024-03-06T07:32:15.192629+0000","last_became_peered":"2024-03-06T07:32:15.192629+0000","last_unstale":"2024-03-06T07:32:16.201257+0000","last_undegraded":"2024-03-06T07:32:16.201257+0000","last_fullsized":"2024-03-06T07:32:16.201257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:13.343774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.197872+0000","last_change":"2024-03-06T07:32:14.174314+0000","last_active":"2024-03-06T07:32:16.197872+0000","last_peered":"2024-03-06T07:32:16.197872+0000","last_clean":"2024-03-06T07:32:16.197872+0000","last_became_active":"2024-03-06T07:32:14.173464+0000","last_became_peered":"2024-03-06T07:32:14.173464+0000","last_unstale":"2024-03-06T07:32:16.197872+0000","last_undegraded":"2024-03-06T07:32:16.197872+0000","last_fullsized":"2024-03-06T07:32:16.197872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:00:40.754642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.236759+0000","last_change":"2024-03-06T07:32:15.191384+0000","last_active":"2024-03-06T07:32:16.236759+0000","last_peered":"2024-03-06T07:32:16.236759+0000","last_clean":"2024-03-06T07:32:16.236759+0000","last_became_active":"2024-03-06T07:32:15.190635+0000","last_became_peered":"2024-03-06T07:32:15.190635+0000","last_unstale":"2024-03-06T07:32:16.236759+0000","last_undegraded":"2024-03-06T07:32:16.236759+0000","last_fullsized":"2024-03-06T07:32:16.236759+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:37.173597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201466+0000","last_change":"2024-03-06T07:32:14.162795+0000","last_active":"2024-03-06T07:32:16.201466+0000","last_peered":"2024-03-06T07:32:16.201466+0000","last_clean":"2024-03-06T07:32:16.201466+0000","last_became_active":"2024-03-06T07:32:14.162519+0000","last_became_peered":"2024-03-06T07:32:14.162519+0000","last_unstale":"2024-03-06T07:32:16.201466+0000","last_undegraded":"2024-03-06T07:32:16.201466+0000","last_fullsized":"2024-03-06T07:32:16.201466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:20:43.782045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199238+0000","last_change":"2024-03-06T07:32:15.191585+0000","last_active":"2024-03-06T07:32:16.199238+0000","last_peered":"2024-03-06T07:32:16.199238+0000","last_clean":"2024-03-06T07:32:16.199238+0000","last_became_active":"2024-03-06T07:32:15.191226+0000","last_became_peered":"2024-03-06T07:32:15.191226+0000","last_unstale":"2024-03-06T07:32:16.199238+0000","last_undegraded":"2024-03-06T07:32:16.199238+0000","last_fullsized":"2024-03-06T07:32:16.199238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:04.844786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201305+0000","last_change":"2024-03-06T07:32:14.174695+0000","last_active":"2024-03-06T07:32:16.201305+0000","last_peered":"2024-03-06T07:32:16.201305+0000","last_clean":"2024-03-06T07:32:16.201305+0000","last_became_active":"2024-03-06T07:32:14.173489+0000","last_became_peered":"2024-03-06T07:32:14.173489+0000","last_unstale":"2024-03-06T07:32:16.201305+0000","last_undegraded":"2024-03-06T07:32:16.201305+0000","last_fullsized":"2024-03-06T07:32:16.201305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:55.435500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235724+0000","last_change":"2024-03-06T07:32:15.196921+0000","last_active":"2024-03-06T07:32:16.235724+0000","last_peered":"2024-03-06T07:32:16.235724+0000","last_clean":"2024-03-06T07:32:16.235724+0000","last_became_active":"2024-03-06T07:32:15.196136+0000","last_became_peered":"2024-03-06T07:32:15.196136+0000","last_unstale":"2024-03-06T07:32:16.235724+0000","last_undegraded":"2024-03-06T07:32:16.235724+0000","last_fullsized":"2024-03-06T07:32:16.235724+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:43.466481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.199240+0000","last_change":"2024-03-06T07:32:14.168845+0000","last_active":"2024-03-06T07:32:16.199240+0000","last_peered":"2024-03-06T07:32:16.199240+0000","last_clean":"2024-03-06T07:32:16.199240+0000","last_became_active":"2024-03-06T07:32:14.168391+0000","last_became_peered":"2024-03-06T07:32:14.168391+0000","last_unstale":"2024-03-06T07:32:16.199240+0000","last_undegraded":"2024-03-06T07:32:16.199240+0000","last_fullsized":"2024-03-06T07:32:16.199240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:07:31.532989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.210599+0000","last_change":"2024-03-06T07:32:15.202007+0000","last_active":"2024-03-06T07:32:15.210599+0000","last_peered":"2024-03-06T07:32:15.210599+0000","last_clean":"2024-03-06T07:32:15.210599+0000","last_became_active":"2024-03-06T07:32:15.197471+0000","last_became_peered":"2024-03-06T07:32:15.197471+0000","last_unstale":"2024-03-06T07:32:15.210599+0000","last_undegraded":"2024-03-06T07:32:15.210599+0000","last_fullsized":"2024-03-06T07:32:15.210599+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:15.957458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.201312+0000","last_change":"2024-03-06T07:32:14.171237+0000","last_active":"2024-03-06T07:32:16.201312+0000","last_peered":"2024-03-06T07:32:16.201312+0000","last_clean":"2024-03-06T07:32:16.201312+0000","last_became_active":"2024-03-06T07:32:14.170248+0000","last_became_peered":"2024-03-06T07:32:14.170248+0000","last_unstale":"2024-03-06T07:32:16.201312+0000","last_undegraded":"2024-03-06T07:32:16.201312+0000","last_fullsized":"2024-03-06T07:32:16.201312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:03.829983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.220274+0000","last_change":"2024-03-06T07:32:15.215084+0000","last_active":"2024-03-06T07:32:15.220274+0000","last_peered":"2024-03-06T07:32:15.220274+0000","last_clean":"2024-03-06T07:32:15.220274+0000","last_became_active":"2024-03-06T07:32:15.214745+0000","last_became_peered":"2024-03-06T07:32:15.214745+0000","last_unstale":"2024-03-06T07:32:15.220274+0000","last_undegraded":"2024-03-06T07:32:15.220274+0000","last_fullsized":"2024-03-06T07:32:15.220274+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:22.050942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200602+0000","last_change":"2024-03-06T07:32:14.162101+0000","last_active":"2024-03-06T07:32:16.200602+0000","last_peered":"2024-03-06T07:32:16.200602+0000","last_clean":"2024-03-06T07:32:16.200602+0000","last_became_active":"2024-03-06T07:32:14.161757+0000","last_became_peered":"2024-03-06T07:32:14.161757+0000","last_unstale":"2024-03-06T07:32:16.200602+0000","last_undegraded":"2024-03-06T07:32:16.200602+0000","last_fullsized":"2024-03-06T07:32:16.200602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:10:22.473395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216894+0000","last_change":"2024-03-06T07:32:15.201918+0000","last_active":"2024-03-06T07:32:15.216894+0000","last_peered":"2024-03-06T07:32:15.216894+0000","last_clean":"2024-03-06T07:32:15.216894+0000","last_became_active":"2024-03-06T07:32:15.201500+0000","last_became_peered":"2024-03-06T07:32:15.201500+0000","last_unstale":"2024-03-06T07:32:15.216894+0000","last_undegraded":"2024-03-06T07:32:15.216894+0000","last_fullsized":"2024-03-06T07:32:15.216894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:07.645152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187733+0000","last_change":"2024-03-06T07:32:14.169360+0000","last_active":"2024-03-06T07:32:15.187733+0000","last_peered":"2024-03-06T07:32:15.187733+0000","last_clean":"2024-03-06T07:32:15.187733+0000","last_became_active":"2024-03-06T07:32:14.169068+0000","last_became_peered":"2024-03-06T07:32:14.169068+0000","last_unstale":"2024-03-06T07:32:15.187733+0000","last_undegraded":"2024-03-06T07:32:15.187733+0000","last_fullsized":"2024-03-06T07:32:15.187733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:25:18.291932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.209664+0000","last_change":"2024-03-06T07:32:15.197704+0000","last_active":"2024-03-06T07:32:15.209664+0000","last_peered":"2024-03-06T07:32:15.209664+0000","last_clean":"2024-03-06T07:32:15.209664+0000","last_became_active":"2024-03-06T07:32:15.197353+0000","last_became_peered":"2024-03-06T07:32:15.197353+0000","last_unstale":"2024-03-06T07:32:15.209664+0000","last_undegraded":"2024-03-06T07:32:15.209664+0000","last_fullsized":"2024-03-06T07:32:15.209664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:50.560658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.191044+0000","last_change":"2024-03-06T07:32:14.164424+0000","last_active":"2024-03-06T07:32:15.191044+0000","last_peered":"2024-03-06T07:32:15.191044+0000","last_clean":"2024-03-06T07:32:15.191044+0000","last_became_active":"2024-03-06T07:32:14.164059+0000","last_became_peered":"2024-03-06T07:32:14.164059+0000","last_unstale":"2024-03-06T07:32:15.191044+0000","last_undegraded":"2024-03-06T07:32:15.191044+0000","last_fullsized":"2024-03-06T07:32:15.191044+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:46.948960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.235780+0000","last_change":"2024-03-06T07:32:15.207994+0000","last_active":"2024-03-06T07:32:16.235780+0000","last_peered":"2024-03-06T07:32:16.235780+0000","last_clean":"2024-03-06T07:32:16.235780+0000","last_became_active":"2024-03-06T07:32:15.207192+0000","last_became_peered":"2024-03-06T07:32:15.207192+0000","last_unstale":"2024-03-06T07:32:16.235780+0000","last_undegraded":"2024-03-06T07:32:16.235780+0000","last_fullsized":"2024-03-06T07:32:16.235780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.042864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187570+0000","last_change":"2024-03-06T07:32:14.168693+0000","last_active":"2024-03-06T07:32:15.187570+0000","last_peered":"2024-03-06T07:32:15.187570+0000","last_clean":"2024-03-06T07:32:15.187570+0000","last_became_active":"2024-03-06T07:32:14.168385+0000","last_became_peered":"2024-03-06T07:32:14.168385+0000","last_unstale":"2024-03-06T07:32:15.187570+0000","last_undegraded":"2024-03-06T07:32:15.187570+0000","last_fullsized":"2024-03-06T07:32:15.187570+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:06.741416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.203699+0000","last_change":"2024-03-06T07:32:15.201468+0000","last_active":"2024-03-06T07:32:15.203699+0000","last_peered":"2024-03-06T07:32:15.203699+0000","last_clean":"2024-03-06T07:32:15.203699+0000","last_became_active":"2024-03-06T07:32:15.201169+0000","last_became_peered":"2024-03-06T07:32:15.201169+0000","last_unstale":"2024-03-06T07:32:15.203699+0000","last_undegraded":"2024-03-06T07:32:15.203699+0000","last_fullsized":"2024-03-06T07:32:15.203699+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:12.042864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.184904+0000","last_change":"2024-03-06T07:32:14.169033+0000","last_active":"2024-03-06T07:32:15.184904+0000","last_peered":"2024-03-06T07:32:15.184904+0000","last_clean":"2024-03-06T07:32:15.184904+0000","last_became_active":"2024-03-06T07:32:14.168324+0000","last_became_peered":"2024-03-06T07:32:14.168324+0000","last_unstale":"2024-03-06T07:32:15.184904+0000","last_undegraded":"2024-03-06T07:32:15.184904+0000","last_fullsized":"2024-03-06T07:32:15.184904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:15:46.205058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.216945+0000","last_change":"2024-03-06T07:32:15.209659+0000","last_active":"2024-03-06T07:32:15.216945+0000","last_peered":"2024-03-06T07:32:15.216945+0000","last_clean":"2024-03-06T07:32:15.216945+0000","last_became_active":"2024-03-06T07:32:15.206930+0000","last_became_peered":"2024-03-06T07:32:15.206930+0000","last_unstale":"2024-03-06T07:32:15.216945+0000","last_undegraded":"2024-03-06T07:32:15.216945+0000","last_fullsized":"2024-03-06T07:32:15.216945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:58.656525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.193609+0000","last_change":"2024-03-06T07:32:14.172371+0000","last_active":"2024-03-06T07:32:15.193609+0000","last_peered":"2024-03-06T07:32:15.193609+0000","last_clean":"2024-03-06T07:32:15.193609+0000","last_became_active":"2024-03-06T07:32:14.170178+0000","last_became_peered":"2024-03-06T07:32:14.170178+0000","last_unstale":"2024-03-06T07:32:15.193609+0000","last_undegraded":"2024-03-06T07:32:15.193609+0000","last_fullsized":"2024-03-06T07:32:15.193609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:51:26.735940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.217454+0000","last_change":"2024-03-06T07:32:15.210744+0000","last_active":"2024-03-06T07:32:15.217454+0000","last_peered":"2024-03-06T07:32:15.217454+0000","last_clean":"2024-03-06T07:32:15.217454+0000","last_became_active":"2024-03-06T07:32:15.210085+0000","last_became_peered":"2024-03-06T07:32:15.210085+0000","last_unstale":"2024-03-06T07:32:15.217454+0000","last_undegraded":"2024-03-06T07:32:15.217454+0000","last_fullsized":"2024-03-06T07:32:15.217454+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:52:46.767014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.187070+0000","last_change":"2024-03-06T07:32:14.173951+0000","last_active":"2024-03-06T07:32:15.187070+0000","last_peered":"2024-03-06T07:32:15.187070+0000","last_clean":"2024-03-06T07:32:15.187070+0000","last_became_active":"2024-03-06T07:32:14.173610+0000","last_became_peered":"2024-03-06T07:32:14.173610+0000","last_unstale":"2024-03-06T07:32:15.187070+0000","last_undegraded":"2024-03-06T07:32:15.187070+0000","last_fullsized":"2024-03-06T07:32:15.187070+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:13.363225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:32:16.200186+0000","last_change":"2024-03-06T07:32:15.199093+0000","last_active":"2024-03-06T07:32:16.200186+0000","last_peered":"2024-03-06T07:32:16.200186+0000","last_clean":"2024-03-06T07:32:16.200186+0000","last_became_active":"2024-03-06T07:32:15.196793+0000","last_became_peered":"2024-03-06T07:32:15.196793+0000","last_unstale":"2024-03-06T07:32:16.200186+0000","last_undegraded":"2024-03-06T07:32:16.200186+0000","last_fullsized":"2024-03-06T07:32:16.200186+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:14.129907+0000","last_clean_scrub_stamp":"2024-03-06T07:32:14.129907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:20.980135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.218808+0000","last_change":"2024-03-06T07:32:14.175070+0000","last_active":"2024-03-06T07:32:15.218808+0000","last_peered":"2024-03-06T07:32:15.218808+0000","last_clean":"2024-03-06T07:32:15.218808+0000","last_became_active":"2024-03-06T07:32:14.173917+0000","last_became_peered":"2024-03-06T07:32:14.173917+0000","last_unstale":"2024-03-06T07:32:15.218808+0000","last_undegraded":"2024-03-06T07:32:15.218808+0000","last_fullsized":"2024-03-06T07:32:15.218808+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:13.115103+0000","last_clean_scrub_stamp":"2024-03-06T07:32:13.115103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:01.810493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":276,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-06T07:32:15.192159+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:32:15.192159+0000","last_peered":"2024-03-06T07:32:15.192159+0000","last_clean":"2024-03-06T07:32:15.192159+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:32:15.192159+0000","last_undegraded":"2024-03-06T07:32:15.192159+0000","last_fullsized":"2024-03-06T07:32:15.192159+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509502,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32148,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339692,"statfs":{"total":96636764160,"available":96603844608,"internally_reserved":0,"allocated":991232,"data_stored":417804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":417416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542206,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1572864,"data_stored":1001360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542206,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36316,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335524,"statfs":{"total":96636764160,"available":96599576576,"internally_reserved":0,"allocated":987136,"data_stored":411616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36320,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335520,"statfs":{"total":96636764160,"available":96599572480,"internally_reserved":0,"allocated":1003520,"data_stored":418856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542206,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36300,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335540,"statfs":{"total":96636764160,"available":96599592960,"internally_reserved":0,"allocated":974848,"data_stored":410922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542207,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":412534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542207,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32724,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339116,"statfs":{"total":96636764160,"available":96603254784,"internally_reserved":0,"allocated":1572864,"data_stored":1001324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:20.165 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:32:20.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:32:20.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.437+0000 7efd15800640 1 Processor -- start 2024-03-06T07:32:20.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.438+0000 7efd15800640 1 -- start start 2024-03-06T07:32:20.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd15800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd1015c370 0x7efd100adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd15800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1006da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd15800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd10156830 con 0x7efd100ad6f0 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd15800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1006ce60 con 0x7efd1015c370 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd15800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd100ae010 con 0x7efd1006d5d0 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd0e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd1015c370 0x7efd100adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efd1015c370 0x7efd100adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53602/0 (socket says 172.21.15.6:53602) 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0e000640 1 -- 172.21.15.6:0/2915292669 learned_addr learned my addr 172.21.15.6:0/2915292669 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.439+0000 7efd0ea00640 1 --1- 172.21.15.6:0/2915292669 >> v1:172.21.15.6:6789/0 conn(0x7efd100ad6f0 0x7efd1015f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41750/0 (socket says 172.21.15.6:41750) 2024-03-06T07:32:20.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0f400640 1 --2- 172.21.15.6:0/2915292669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1006da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0e000640 1 -- 172.21.15.6:0/2915292669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 msgr2=0x7efd1006da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0e000640 1 --2- 172.21.15.6:0/2915292669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1006da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0e000640 1 -- 172.21.15.6:0/2915292669 >> v1:172.21.15.6:6789/0 conn(0x7efd100ad6f0 legacy=0x7efd1015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0e000640 1 -- 172.21.15.6:0/2915292669 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10160120 con 0x7efd1015c370 2024-03-06T07:32:20.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.440+0000 7efd0d600640 1 -- 172.21.15.6:0/2915292669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168549007 0 0) 0x7efd10156830 con 0x7efd100ad6f0 2024-03-06T07:32:20.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.441+0000 7efd0e000640 1 --2- 172.21.15.6:0/2915292669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd1015c370 0x7efd100adad0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7efd1006cfe0 tx=0x7efcfc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c2222bdf9dc323 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd0d600640 1 -- 172.21.15.6:0/2915292669 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc034070 con 0x7efd1015c370 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd0d600640 1 -- 172.21.15.6:0/2915292669 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc030840 con 0x7efd1015c370 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd0d600640 1 -- 172.21.15.6:0/2915292669 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc030c10 con 0x7efd1015c370 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd15800640 1 -- 172.21.15.6:0/2915292669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd1015c370 msgr2=0x7efd100adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd15800640 1 --2- 172.21.15.6:0/2915292669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd1015c370 0x7efd100adad0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7efd1006cfe0 tx=0x7efcfc02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd15800640 1 -- 172.21.15.6:0/2915292669 shutdown_connections 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd15800640 1 --2- 172.21.15.6:0/2915292669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1006da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd15800640 1 --2- 172.21.15.6:0/2915292669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd1015c370 0x7efd100adad0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.442+0000 7efd15800640 1 -- 172.21.15.6:0/2915292669 >> 172.21.15.6:0/2915292669 conn(0x7efd1006b3d0 msgr2=0x7efd1006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.443+0000 7efd15800640 1 -- 172.21.15.6:0/2915292669 shutdown_connections 2024-03-06T07:32:20.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.443+0000 7efd15800640 1 -- 172.21.15.6:0/2915292669 wait complete. 2024-03-06T07:32:20.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.444+0000 7efd15800640 1 Processor -- start 2024-03-06T07:32:20.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.444+0000 7efd15800640 1 -- start start 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.445+0000 7efd15800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.445+0000 7efd15800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 0x7efd100cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.445+0000 7efd15800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd1014d850 0x7efd101f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.445+0000 7efd15800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1006d0b0 con 0x7efd100ad6f0 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.445+0000 7efd15800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd101606e0 con 0x7efd1014d850 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.445+0000 7efd15800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1006df40 con 0x7efd1006d5d0 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 0x7efd100cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 0x7efd100cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53610/0 (socket says 172.21.15.6:53610) 2024-03-06T07:32:20.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 -- 172.21.15.6:0/350740134 learned_addr learned my addr 172.21.15.6:0/350740134 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0ea00640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 -- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 msgr2=0x7efd1014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0f400640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd1014d850 0x7efd101f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1014ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 -- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd1014d850 msgr2=0x7efd101f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd1014d850 0x7efd101f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 -- 172.21.15.6:0/350740134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10144e40 con 0x7efd100ad6f0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.446+0000 7efd0e000640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 0x7efd100cefc0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0090b0 tx=0x7efcfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.447+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc048070 con 0x7efd100ad6f0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.447+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc030640 con 0x7efd100ad6f0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.447+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc034040 con 0x7efd100ad6f0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.447+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1014ddc0 con 0x7efd100ad6f0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.447+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1014e380 con 0x7efd100ad6f0 2024-03-06T07:32:20.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.448+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd10068010 con 0x7efd100ad6f0 2024-03-06T07:32:20.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.449+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcfc037070 con 0x7efd100ad6f0 2024-03-06T07:32:20.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.450+0000 7efd06a00640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcf006b4d0 0x7efcf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.450+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcfc0ac0a0 con 0x7efd100ad6f0 2024-03-06T07:32:20.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.451+0000 7efd0ea00640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcf006b4d0 0x7efcf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.452+0000 7efd0ea00640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcf006b4d0 0x7efcf006d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efd00000f60 tx=0x7efd00007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.455+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcfc07afb0 con 0x7efd100ad6f0 2024-03-06T07:32:20.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.727+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efd100b35b0 con 0x7efd100ad6f0 2024-03-06T07:32:20.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.732+0000 7efd06a00640 1 -- 172.21.15.6:0/350740134 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7efcfc07a8b0 con 0x7efd100ad6f0 2024-03-06T07:32:20.735 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:32:20.735 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:32:20.735 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:32:20.735 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:32:20.735 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:20.735 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:32:20.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcf006b4d0 msgr2=0x7efcf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcf006b4d0 0x7efcf006d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efd00000f60 tx=0x7efd00007c00 comp rx=0 tx=0).stop 2024-03-06T07:32:20.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 msgr2=0x7efd100cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 0x7efd100cefc0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0090b0 tx=0x7efcfc004730 comp rx=0 tx=0).stop 2024-03-06T07:32:20.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 shutdown_connections 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efcf006b4d0 0x7efcf006d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efd1014d850 0x7efd101f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efd100ad6f0 0x7efd100cefc0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 --2- 172.21.15.6:0/350740134 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efd1006d5d0 0x7efd1014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 >> 172.21.15.6:0/350740134 conn(0x7efd1006b3d0 msgr2=0x7efd1006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 shutdown_connections 2024-03-06T07:32:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.735+0000 7efd15800640 1 -- 172.21.15.6:0/350740134 wait complete. 2024-03-06T07:32:20.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:20.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.965+0000 7faf72200640 1 Processor -- start 2024-03-06T07:32:20.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.966+0000 7faf72200640 1 -- start start 2024-03-06T07:32:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.967+0000 7faf72200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7faf6c15e2a0 0x7faf6c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.967+0000 7faf72200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0d1120 con 0x7faf6c15dec0 2024-03-06T07:32:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.967+0000 7faf72200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0d1400 con 0x7faf6c0cf460 2024-03-06T07:32:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.967+0000 7faf72200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c0b0ea0 con 0x7faf6c15e2a0 2024-03-06T07:32:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.967+0000 7faf70e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faf6c15dec0 0x7faf6c0ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41754/0 (socket says 172.21.15.6:41754) 2024-03-06T07:32:20.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.967+0000 7faf70e00640 1 -- 172.21.15.6:0/2477879912 learned_addr learned my addr 172.21.15.6:0/2477879912 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6be00640 1 --2- 172.21.15.6:0/2477879912 >> v2:172.21.15.178:3300/0 conn(0x7faf6c15e2a0 0x7faf6c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448848147 0 0) 0x7faf6c0d1120 con 0x7faf6c15dec0 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf4c003520 con 0x7faf6c15dec0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580311877 0 0) 0x7faf4c003520 con 0x7faf6c15dec0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 >> v2:172.21.15.178:3300/0 conn(0x7faf6c15e2a0 msgr2=0x7faf6c0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 --2- 172.21.15.6:0/2477879912 >> v2:172.21.15.178:3300/0 conn(0x7faf6c15e2a0 0x7faf6c0cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 >> v1:172.21.15.178:6789/0 conn(0x7faf6c0cf460 legacy=0x7faf6c0b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0b0b30 con 0x7faf6c15dec0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7faf5c001670 con 0x7faf6c15dec0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 >> v1:172.21.15.6:6789/0 conn(0x7faf6c15dec0 legacy=0x7faf6c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf4c003a00 0x7faf4c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf4c004800 0x7faf4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c0051c0 0x7faf4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0b0ea0 con 0x7faf4c003a00 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.968+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c00c0c0 con 0x7faf4c0051c0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c00c2c0 con 0x7faf4c004800 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf70e00640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf4c003a00 0x7faf4c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf71800640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c0051c0 0x7faf4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf6be00640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf4c004800 0x7faf4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf70e00640 1 -- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf4c004800 msgr2=0x7faf4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf70e00640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf4c004800 0x7faf4c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf70e00640 1 -- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c0051c0 msgr2=0x7faf4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf70e00640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c0051c0 0x7faf4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf70e00640 1 -- 172.21.15.6:0/2477879912 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c003520 con 0x7faf4c003a00 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf71800640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c0051c0 0x7faf4c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.969+0000 7faf71800640 1 -- 172.21.15.6:0/2477879912 reap_dead start 2024-03-06T07:32:20.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.970+0000 7faf70e00640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf4c003a00 0x7faf4c003e20 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c190 tx=0x7faf4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c318424b4ef2e349 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c007c60 con 0x7faf4c003a00 2024-03-06T07:32:20.974 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c030070 con 0x7faf4c003a00 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf6b400640 1 -- 172.21.15.6:0/2477879912 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c005d90 con 0x7faf4c003a00 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf72200640 1 -- 172.21.15.6:0/2477879912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf4c003a00 msgr2=0x7faf4c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf72200640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf4c003a00 0x7faf4c003e20 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c190 tx=0x7faf4c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf72200640 1 -- 172.21.15.6:0/2477879912 shutdown_connections 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf72200640 1 --2- 172.21.15.6:0/2477879912 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf4c003a00 0x7faf4c003e20 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf72200640 1 -- 172.21.15.6:0/2477879912 >> 172.21.15.6:0/2477879912 conn(0x7faf6c0d3d40 msgr2=0x7faf6c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.971+0000 7faf72200640 1 -- 172.21.15.6:0/2477879912 shutdown_connections 2024-03-06T07:32:20.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.972+0000 7faf72200640 1 -- 172.21.15.6:0/2477879912 wait complete. 2024-03-06T07:32:20.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.973+0000 7faf72200640 1 Processor -- start 2024-03-06T07:32:20.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.973+0000 7faf72200640 1 -- start start 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.974+0000 7faf72200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c003a00 0x7faf6c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.974+0000 7faf72200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6c0ca120 0x7faf6c0b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.974+0000 7faf72200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 0x7faf6c1f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf72200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0ce990 con 0x7faf6c0b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf72200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0ce810 con 0x7faf4c003a00 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf72200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c0ceb10 con 0x7faf6c0ca120 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 0x7faf6c1f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 0x7faf6c1f71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53628/0 (socket says 172.21.15.6:53628) 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 -- 172.21.15.6:0/1479371582 learned_addr learned my addr 172.21.15.6:0/1479371582 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf6be00640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6c0ca120 0x7faf6c0b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf70e00640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c003a00 0x7faf6c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 -- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6c0ca120 msgr2=0x7faf6c0b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6c0ca120 0x7faf6c0b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 -- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c003a00 msgr2=0x7faf6c0ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c003a00 0x7faf6c0ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.975+0000 7faf71800640 1 -- 172.21.15.6:0/1479371582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0cc230 con 0x7faf6c0b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.976+0000 7faf71800640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 0x7faf6c1f71d0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0cf8d0 tx=0x7faf6400d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.976+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64018070 con 0x7faf6c0b9040 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.976+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6400de90 con 0x7faf6c0b9040 2024-03-06T07:32:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.976+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c0cc510 con 0x7faf6c0b9040 2024-03-06T07:32:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.976+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64003830 con 0x7faf6c0b9040 2024-03-06T07:32:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.976+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c0ccad0 con 0x7faf6c0b9040 2024-03-06T07:32:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.977+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c0d0af0 con 0x7faf6c0b9040 2024-03-06T07:32:20.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.978+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf6401b070 con 0x7faf6c0b9040 2024-03-06T07:32:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.979+0000 7faf68c00640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf4806b550 0x7faf4806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.979+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf640895f0 con 0x7faf6c0b9040 2024-03-06T07:32:20.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.980+0000 7faf70e00640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf4806b550 0x7faf4806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.981+0000 7faf70e00640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf4806b550 0x7faf4806da10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c160 tx=0x7faf5c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:20.984+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf6c0d0af0 con 0x7faf6c0b9040 2024-03-06T07:32:21.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.226+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf6c0cccd0 con 0x7faf6c0b9040 2024-03-06T07:32:21.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.228+0000 7faf68c00640 1 -- 172.21.15.6:0/1479371582 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7faf6c0cccd0 con 0x7faf6c0b9040 2024-03-06T07:32:21.230 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:21.230 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:21.230 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.230+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf4806b550 msgr2=0x7faf4806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.230+0000 7faf72200640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf4806b550 0x7faf4806da10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7faf5c00c160 tx=0x7faf5c009030 comp rx=0 tx=0).stop 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.230+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 msgr2=0x7faf6c1f71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.230+0000 7faf72200640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 0x7faf6c1f71d0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0cf8d0 tx=0x7faf6400d660 comp rx=0 tx=0).stop 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 shutdown_connections 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf4806b550 0x7faf4806da10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf6c0b9040 0x7faf6c1f71d0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6c0ca120 0x7faf6c0b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 --2- 172.21.15.6:0/1479371582 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf4c003a00 0x7faf6c0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 >> 172.21.15.6:0/1479371582 conn(0x7faf6c0d3d40 msgr2=0x7faf6c0d18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 shutdown_connections 2024-03-06T07:32:21.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.231+0000 7faf72200640 1 -- 172.21.15.6:0/1479371582 wait complete. 2024-03-06T07:32:21.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:14.630088+0000', 'modified': '2024-03-06T07:32:17.693833+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6801}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6801': {'gid': 6801, 'name': 'b', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.178:6836/1516947936', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 1516947936}, {'type': 'v1', 'addr': '172.21.15.178:6836', 'nonce': 1516947936}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6801, 'qdb_cluster': [6801]} 2024-03-06T07:32:21.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:21.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.884+0000 7f0977e00640 1 Processor -- start 2024-03-06T07:32:21.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.884+0000 7f0977e00640 1 -- start start 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0977e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09781552b0 0x7f097815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0977e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f09780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0977e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097815ed10 con 0x7f09780cce90 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0977e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097815eff0 con 0x7f09781552b0 2024-03-06T07:32:21.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0977e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097815f2d0 con 0x7f097815e810 2024-03-06T07:32:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0976a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09781552b0 0x7f097815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0976a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09781552b0 0x7f097815b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60030/0 (socket says 172.21.15.6:60030) 2024-03-06T07:32:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0976a00640 1 -- 172.21.15.6:0/3417780014 learned_addr learned my addr 172.21.15.6:0/3417780014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.886+0000 7f0976000640 1 --2- 172.21.15.6:0/3417780014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f09780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.887+0000 7f0976a00640 1 -- 172.21.15.6:0/3417780014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 msgr2=0x7f09780cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.887+0000 7f0976a00640 1 --2- 172.21.15.6:0/3417780014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f09780cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.887+0000 7f0976a00640 1 -- 172.21.15.6:0/3417780014 >> v1:172.21.15.178:6789/0 conn(0x7f09780cce90 legacy=0x7f097806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.887+0000 7f0975600640 1 -- 172.21.15.6:0/3417780014 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833706602 0 0) 0x7f097815ed10 con 0x7f09780cce90 2024-03-06T07:32:21.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.887+0000 7f0976a00640 1 -- 172.21.15.6:0/3417780014 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097815f5b0 con 0x7f09781552b0 2024-03-06T07:32:21.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.889+0000 7f0976a00640 1 --2- 172.21.15.6:0/3417780014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f097815b1d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0968009870 tx=0x7f096802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90e9835361ecf1a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.889+0000 7f0975600640 1 -- 172.21.15.6:0/3417780014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968035020 con 0x7f09781552b0 2024-03-06T07:32:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0975600640 1 -- 172.21.15.6:0/3417780014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0968031640 con 0x7f09781552b0 2024-03-06T07:32:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0975600640 1 -- 172.21.15.6:0/3417780014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096802d040 con 0x7f09781552b0 2024-03-06T07:32:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 -- 172.21.15.6:0/3417780014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 msgr2=0x7f097815b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 --2- 172.21.15.6:0/3417780014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f097815b1d0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f0968009870 tx=0x7f096802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 -- 172.21.15.6:0/3417780014 shutdown_connections 2024-03-06T07:32:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 --2- 172.21.15.6:0/3417780014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f09780cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 --2- 172.21.15.6:0/3417780014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f097815b1d0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 -- 172.21.15.6:0/3417780014 >> 172.21.15.6:0/3417780014 conn(0x7f097806b3d0 msgr2=0x7f097806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 -- 172.21.15.6:0/3417780014 shutdown_connections 2024-03-06T07:32:21.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.890+0000 7f0977e00640 1 -- 172.21.15.6:0/3417780014 wait complete. 2024-03-06T07:32:21.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.892+0000 7f0977e00640 1 Processor -- start 2024-03-06T07:32:21.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.892+0000 7f0977e00640 1 -- start start 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0977e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09780cce90 0x7f097813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0977e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f0978070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0977e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f0978071140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0977e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097806dd30 con 0x7f09781552b0 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0977e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09780b3760 con 0x7f09780cce90 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0977e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097806deb0 con 0x7f097815e810 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f0978070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f0978070c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60032/0 (socket says 172.21.15.6:60032) 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.893+0000 7f0976000640 1 -- 172.21.15.6:0/1249112973 learned_addr learned my addr 172.21.15.6:0/1249112973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976a00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09780cce90 0x7f097813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0977400640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f0978071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976000640 1 -- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 msgr2=0x7f0978071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976000640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f0978071140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976000640 1 -- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09780cce90 msgr2=0x7f097813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976000640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09780cce90 0x7f097813fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976000640 1 -- 172.21.15.6:0/1249112973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0978140690 con 0x7f09781552b0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0976000640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f0978070c00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f096000e960 tx=0x7f096000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f096ea00640 1 -- 172.21.15.6:0/1249112973 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960017070 con 0x7f09781552b0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f096ea00640 1 -- 172.21.15.6:0/1249112973 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096000ca00 con 0x7f09781552b0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.894+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0978140970 con 0x7f09781552b0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.895+0000 7f096ea00640 1 -- 172.21.15.6:0/1249112973 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096000cdf0 con 0x7f09781552b0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.895+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0978140f30 con 0x7f09781552b0 2024-03-06T07:32:21.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.897+0000 7f096ea00640 1 -- 172.21.15.6:0/1249112973 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f096001a070 con 0x7f09781552b0 2024-03-06T07:32:21.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.898+0000 7f096ea00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f095406b500 0x7f095406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.898+0000 7f096ea00640 1 -- 172.21.15.6:0/1249112973 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f096008a660 con 0x7f09781552b0 2024-03-06T07:32:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.898+0000 7f098082f740 1 Processor -- start 2024-03-06T07:32:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.898+0000 7f0976a00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f095406b500 0x7f095406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.898+0000 7f098082f740 1 -- start start 2024-03-06T07:32:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bb3990 0x564f02b86930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f098082f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02b86e70 0x564f02bb6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f098082f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f02af0600 con 0x564f02bb3d70 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f02a99930 con 0x564f02b86e70 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f02aec6a0 con 0x564f02bb3990 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60038/0 (socket says 172.21.15.6:60038) 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.899+0000 7f0976000640 1 -- 172.21.15.6:0/3235120195 learned_addr learned my addr 172.21.15.6:0/3235120195 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976a00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f095406b500 0x7f095406d9c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0968036000 tx=0x7f0968008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976a00640 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bb3990 0x564f02b86930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0977400640 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02b86e70 0x564f02bb6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976000640 1 -- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bb3990 msgr2=0x564f02b86930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976000640 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bb3990 0x564f02b86930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976000640 1 -- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02b86e70 msgr2=0x564f02bb6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976000640 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02b86e70 0x564f02bb6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976000640 1 -- 172.21.15.6:0/3235120195 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f028052d0 con 0x564f02bb3d70 2024-03-06T07:32:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0977400640 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02b86e70 0x564f02bb6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:21.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0976000640 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb6720 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f09600551a0 tx=0x7f0960059500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f8cdf18f4e30cb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0967400640 1 -- 172.21.15.6:0/3235120195 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960021070 con 0x564f02bb3d70 2024-03-06T07:32:21.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0967400640 1 -- 172.21.15.6:0/3235120195 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960061450 con 0x564f02bb3d70 2024-03-06T07:32:21.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.900+0000 7f0967400640 1 -- 172.21.15.6:0/3235120195 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960057040 con 0x564f02bb3d70 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 -- 172.21.15.6:0/3235120195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 msgr2=0x564f02bb6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb6720 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f09600551a0 tx=0x7f0960059500 comp rx=0 tx=0).stop 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 -- 172.21.15.6:0/3235120195 shutdown_connections 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02b86e70 0x564f02bb6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb6720 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 --2- 172.21.15.6:0/3235120195 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bb3990 0x564f02b86930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 -- 172.21.15.6:0/3235120195 >> 172.21.15.6:0/3235120195 conn(0x564f02b88ba0 msgr2=0x564f02bd1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 -- 172.21.15.6:0/3235120195 shutdown_connections 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.901+0000 7f098082f740 1 -- 172.21.15.6:0/3235120195 wait complete. 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.902+0000 7f098082f740 1 -- start start 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.902+0000 7f098082f740 1 Processor -- start 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 0x564f02ba0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3d70 0x564f02ba0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bb5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f02b8b140 con 0x564f02bb3990 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f02aacd90 con 0x564f02bb3d70 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f02b8a530 con 0x564f02bd1770 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f098082f740 10 client.0 _ll_register_callbacks cb 0x564f02ae8350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f0976a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 0x564f02ba0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f0976a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 0x564f02ba0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60054/0 (socket says 172.21.15.6:60054) 2024-03-06T07:32:21.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.903+0000 7f0976a00640 1 -- 172.21.15.6:0/2706285595 learned_addr learned my addr 172.21.15.6:0/2706285595 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976000640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3d70 0x564f02ba0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0977400640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bb5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976a00640 1 -- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 msgr2=0x564f02bb5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976a00640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bb5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976a00640 1 -- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3d70 msgr2=0x564f02ba0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976a00640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3d70 0x564f02ba0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976a00640 1 -- 172.21.15.6:0/2706285595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f028cf6b0 con 0x564f02bb3990 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f0976a00640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 0x564f02ba0730 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0968009990 tx=0x7f0968004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ee8faf1e7cc1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f095be00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f095be00640 1 -- 172.21.15.6:0/2706285595 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968054060 con 0x564f02bb3990 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f095be00640 1 -- 172.21.15.6:0/2706285595 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0968003bf0 con 0x564f02bb3990 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.904+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0960061450 con 0x564f02bb3990 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f095be00640 1 -- 172.21.15.6:0/2706285595 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f096805a050 con 0x564f02bb3990 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 10 client.6939 fetch_fsmap learned FSMap version 53 2024-03-06T07:32:21.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x564f02bb8b40 con 0x564f02bb3990 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f095be00640 1 -- 172.21.15.6:0/2706285595 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960061450 con 0x564f02bb3990 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f095be00640 1 -- 172.21.15.6:0/2706285595 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 53) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f096805d020 con 0x564f02bb3990 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 10 client.6939 fetch_fsmap finished waiting for FSMap version 53 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 10 client.6939 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 20 client.6939 populate_metadata read hostname 'smithi006' 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02bf2770 0x564f02bf4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 4 client.6939 mds_command: new command op to 6801 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:21.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.905+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 --> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f02afdb40 con 0x564f02bf2770 2024-03-06T07:32:21.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.906+0000 7f0976000640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02bf2770 0x564f02bf4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.906+0000 7f0976000640 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02bf2770 0x564f02bf4b60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afe96d904c8d5ee7 server_cookie=e24ab5a8249134d1 in_seq=0 out_seq=0 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.906+0000 7f095be00640 10 client.6939 ms_handle_connect on v2:172.21.15.178:6834/1516947936 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f095be00640 1 -- 172.21.15.6:0/2706285595 <== mds.0 v2:172.21.15.178:6834/1516947936 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564f02afdb40 con 0x564f02bf2770 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f095be00640 10 client.6939 handle_command_reply: tid=0 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 client.6939 shutdown 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 msgr2=0x564f02ba0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 0x564f02ba0730 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0968009990 tx=0x7f0968004730 comp rx=0 tx=0).stop 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 shutdown_connections 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02bf2770 0x564f02bf4b60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3d70 0x564f02ba0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bb5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 --2- 172.21.15.6:0/2706285595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3990 0x564f02ba0730 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 >> 172.21.15.6:0/2706285595 conn(0x564f02b86e70 msgr2=0x564f02b8b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f095be00640 0 client.6939 ms_handle_reset on v2:172.21.15.178:6834/1516947936 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 shutdown_connections 2024-03-06T07:32:21.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 1 -- 172.21.15.6:0/2706285595 wait complete. 2024-03-06T07:32:21.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.909+0000 7f098082f740 20 client.6939 trim_cache size 0 max 16384 2024-03-06T07:32:21.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.927+0000 7f098082f740 1 Processor -- start 2024-03-06T07:32:21.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.927+0000 7f098082f740 1 -- start start 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.928+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02ba5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.928+0000 7f098082f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bd3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.928+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.928+0000 7f098082f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978153da0 con 0x564f02bb3d70 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.928+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978142170 con 0x564f02bb3990 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.928+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09781537a0 con 0x564f02bd1770 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bd3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bd3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60060/0 (socket says 172.21.15.6:60060) 2024-03-06T07:32:21.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 -- 172.21.15.6:0/3639847858 learned_addr learned my addr 172.21.15.6:0/3639847858 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0977400640 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 -- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 msgr2=0x564f02bca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976a00640 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02ba5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 -- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 msgr2=0x564f02ba5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02ba5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 -- 172.21.15.6:0/3639847858 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f02bb60a0 con 0x564f02bb3d70 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.929+0000 7f0976000640 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bd3b70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f09600595f0 tx=0x7f096005f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfab48814da79c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f0966a00640 1 -- 172.21.15.6:0/3639847858 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960021040 con 0x564f02bb3d70 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f0966a00640 1 -- 172.21.15.6:0/3639847858 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960057040 con 0x564f02bb3d70 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f0966a00640 1 -- 172.21.15.6:0/3639847858 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960058190 con 0x564f02bb3d70 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 -- 172.21.15.6:0/3639847858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 msgr2=0x564f02bd3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bd3b70 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f09600595f0 tx=0x7f096005f050 comp rx=0 tx=0).stop 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 -- 172.21.15.6:0/3639847858 shutdown_connections 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bd3b70 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 --2- 172.21.15.6:0/3639847858 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02ba5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 -- 172.21.15.6:0/3639847858 >> 172.21.15.6:0/3639847858 conn(0x564f02b86e70 msgr2=0x564f02bb4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 -- 172.21.15.6:0/3639847858 shutdown_connections 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.930+0000 7f098082f740 1 -- 172.21.15.6:0/3639847858 wait complete. 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.931+0000 7f098082f740 1 -- start start 2024-03-06T07:32:21.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.931+0000 7f098082f740 1 Processor -- start 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02bb6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bd4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f02bb5a30 con 0x564f02bb3d70 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f02bb9120 con 0x564f02bb3990 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f02ba4ed0 con 0x564f02bd1770 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.932+0000 7f098082f740 10 client.0 _ll_register_callbacks cb 0x7f0960009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60074/0 (socket says 172.21.15.6:60074) 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 -- 172.21.15.6:0/420379156 learned_addr learned my addr 172.21.15.6:0/420379156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0977400640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bd4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976a00640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02bb6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 -- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 msgr2=0x564f02bd4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bd4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 -- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 msgr2=0x564f02bb6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02bb6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 -- 172.21.15.6:0/420379156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f02bca5e0 con 0x564f02bb3d70 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976a00640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02bb6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f0976000640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb71b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f09600595f0 tx=0x7f096008a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae102d8098216e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f095be00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.933+0000 7f095be00640 1 -- 172.21.15.6:0/420379156 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960021050 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f095be00640 1 -- 172.21.15.6:0/420379156 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0960058190 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0960057040 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f095be00640 1 -- 172.21.15.6:0/420379156 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0960080050 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 10 client.6945 fetch_fsmap learned FSMap version 53 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) v3 -- 0x564f02b86600 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f095be00640 1 -- 172.21.15.6:0/420379156 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960057040 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f095be00640 1 -- 172.21.15.6:0/420379156 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 53) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f0960054a90 con 0x564f02bb3d70 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 10 client.6945 fetch_fsmap finished waiting for FSMap version 53 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 10 client.6945 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 20 client.6945 populate_metadata read hostname 'smithi006' 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02c45150 0x564f02bd52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 4 client.6945 mds_command: new command op to 6801 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.934+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 --> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564f02af7e90 con 0x564f02c45150 2024-03-06T07:32:21.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.935+0000 7f0976a00640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02c45150 0x564f02bd52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f0976a00640 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02c45150 0x564f02bd52e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36be44f03047ba79 server_cookie=1d67d2b7a0d77101 in_seq=0 out_seq=0 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f095be00640 10 client.6945 ms_handle_connect on v2:172.21.15.178:6834/1516947936 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f095be00640 1 -- 172.21.15.6:0/420379156 <== mds.0 v2:172.21.15.178:6834/1516947936 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x564f02af7e90 con 0x564f02c45150 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f095be00640 10 client.6945 handle_command_reply: tid=0 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 client.6945 shutdown 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 msgr2=0x564f02bb71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb71b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f09600595f0 tx=0x7f096008a040 comp rx=0 tx=0).stop 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 shutdown_connections 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:6834/1516947936,v1:172.21.15.178:6836/1516947936] conn(0x564f02c45150 0x564f02bd52e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x564f02bd1770 0x564f02bd4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564f02bb3d70 0x564f02bb71b0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 --2- 172.21.15.6:0/420379156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x564f02bb3990 0x564f02bb6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 >> 172.21.15.6:0/420379156 conn(0x564f02b86e70 msgr2=0x564f02ba4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.936+0000 7f095be00640 0 client.6945 ms_handle_reset on v2:172.21.15.178:6834/1516947936 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 shutdown_connections 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f098082f740 1 -- 172.21.15.6:0/420379156 wait complete. 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f098082f740 20 client.6945 trim_cache size 0 max 16384 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 6801, 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:32:21.942 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.7534797050000002, 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 53, 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 81, 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 81, 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.157136361999999 2024-03-06T07:32:21.943 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f095406b500 msgr2=0x7f095406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f0977e00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f095406b500 0x7f095406d9c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0968036000 tx=0x7f0968008030 comp rx=0 tx=0).stop 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 msgr2=0x7f0978070c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.937+0000 7f0977e00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f0978070c00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f096000e960 tx=0x7f096000ee20 comp rx=0 tx=0).stop 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 shutdown_connections 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f095406b500 0x7f095406d9c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f097815e810 0x7f0978071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09781552b0 0x7f0978070c00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 --2- 172.21.15.6:0/1249112973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09780cce90 0x7f097813fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 >> 172.21.15.6:0/1249112973 conn(0x7f097806b3d0 msgr2=0x7f0978151f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 shutdown_connections 2024-03-06T07:32:21.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:21.938+0000 7f0977e00640 1 -- 172.21.15.6:0/1249112973 wait complete. 2024-03-06T07:32:21.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:22.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.163+0000 7f58da000640 1 Processor -- start 2024-03-06T07:32:22.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.163+0000 7f58da000640 1 -- start start 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.165+0000 7f58da000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58d4158460 0x7f58d40d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.165+0000 7f58da000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d40b1310 0x7f58d40b1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.165+0000 7f58da000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d40cff30 con 0x7f58d4158840 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.165+0000 7f58da000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d40b1cc0 con 0x7f58d4158460 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.165+0000 7f58da000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d41626d0 con 0x7f58d40b1310 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.165+0000 7f58d8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58d4158460 0x7f58d40d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58d4158460 0x7f58d40d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60084/0 (socket says 172.21.15.6:60084) 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d8c00640 1 -- 172.21.15.6:0/1207884088 learned_addr learned my addr 172.21.15.6:0/1207884088 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d9600640 1 --2- 172.21.15.6:0/1207884088 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d40b1310 0x7f58d40b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d8c00640 1 -- 172.21.15.6:0/1207884088 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d40b1310 msgr2=0x7f58d40b1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d8c00640 1 --2- 172.21.15.6:0/1207884088 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d40b1310 0x7f58d40b1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d8c00640 1 -- 172.21.15.6:0/1207884088 >> v1:172.21.15.178:6789/0 conn(0x7f58d4158840 legacy=0x7f58d4161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.166+0000 7f58d8c00640 1 -- 172.21.15.6:0/1207884088 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d41629b0 con 0x7f58d4158460 2024-03-06T07:32:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.168+0000 7f58d8c00640 1 --2- 172.21.15.6:0/1207884088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158460 0x7f58d40d09d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009870 tx=0x7f58c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5016f790e77ead5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.168+0000 7f58d3400640 1 -- 172.21.15.6:0/1207884088 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8034070 con 0x7f58d4158460 2024-03-06T07:32:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.168+0000 7f58d3400640 1 -- 172.21.15.6:0/1207884088 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c8030840 con 0x7f58d4158460 2024-03-06T07:32:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.168+0000 7f58d3400640 1 -- 172.21.15.6:0/1207884088 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8030c10 con 0x7f58d4158460 2024-03-06T07:32:22.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 -- 172.21.15.6:0/1207884088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158460 msgr2=0x7f58d40d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 --2- 172.21.15.6:0/1207884088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158460 0x7f58d40d09d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f58c8009870 tx=0x7f58c802b980 comp rx=0 tx=0).stop 2024-03-06T07:32:22.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 -- 172.21.15.6:0/1207884088 shutdown_connections 2024-03-06T07:32:22.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 --2- 172.21.15.6:0/1207884088 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d40b1310 0x7f58d40b1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 --2- 172.21.15.6:0/1207884088 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158460 0x7f58d40d09d0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 -- 172.21.15.6:0/1207884088 >> 172.21.15.6:0/1207884088 conn(0x7f58d40b5100 msgr2=0x7f58d40b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 -- 172.21.15.6:0/1207884088 shutdown_connections 2024-03-06T07:32:22.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.169+0000 7f58da000640 1 -- 172.21.15.6:0/1207884088 wait complete. 2024-03-06T07:32:22.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.171+0000 7f58da000640 1 Processor -- start 2024-03-06T07:32:22.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.171+0000 7f58da000640 1 -- start start 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.172+0000 7f58da000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58d40b1310 0x7f58d4153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.172+0000 7f58da000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d4158460 0x7f58d41e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.172+0000 7f58da000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 0x7f58d406c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.172+0000 7f58da000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40d0f10 con 0x7f58d4158840 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.172+0000 7f58da000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40d04b0 con 0x7f58d40b1310 2024-03-06T07:32:22.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.172+0000 7f58da000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d415e8c0 con 0x7f58d4158460 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 0x7f58d406c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 0x7f58d406c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60098/0 (socket says 172.21.15.6:60098) 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 -- 172.21.15.6:0/1952104214 learned_addr learned my addr 172.21.15.6:0/1952104214 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d8c00640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58d40b1310 0x7f58d4153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d3e00640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d4158460 0x7f58d41e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 -- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d4158460 msgr2=0x7f58d41e83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d4158460 0x7f58d41e83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 -- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58d40b1310 msgr2=0x7f58d4153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58d40b1310 0x7f58d4153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 -- 172.21.15.6:0/1952104214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d41e8e40 con 0x7f58d4158840 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.173+0000 7f58d9600640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 0x7f58d406c130 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f58cc007e90 tx=0x7f58cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.174+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc014420 con 0x7f58d4158840 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.174+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc019070 con 0x7f58d4158840 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.174+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc014800 con 0x7f58d4158840 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.174+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d41e9120 con 0x7f58d4158840 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.174+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d41e96e0 con 0x7f58d4158840 2024-03-06T07:32:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.175+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d404f330 con 0x7f58d4158840 2024-03-06T07:32:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.176+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58cc0149a0 con 0x7f58d4158840 2024-03-06T07:32:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.177+0000 7f58d0c00640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.177+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58cc08a9a0 con 0x7f58d4158840 2024-03-06T07:32:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.177+0000 7f58d8c00640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.178+0000 7f58d8c00640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f58b406b4d0 0x7f58b406d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f58c8036000 tx=0x7f58c803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.181+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58cc0598b0 con 0x7f58d4158840 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.422+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f58d4066f10 con 0x7f58d4158840 2024-03-06T07:32:22.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.423+0000 7f58d0c00640 1 -- 172.21.15.6:0/1952104214 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v53) v1 ==== 53+0+73 (secure 0 0 0) 0x7f58cc0591b0 con 0x7f58d4158840 2024-03-06T07:32:22.425 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f58b406b4d0 msgr2=0x7f58b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f58b406b4d0 0x7f58b406d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f58c8036000 tx=0x7f58c803b040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 msgr2=0x7f58d406c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 0x7f58d406c130 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f58cc007e90 tx=0x7f58cc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 shutdown_connections 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f58b406b4d0 0x7f58b406d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58d4158840 0x7f58d406c130 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58d4158460 0x7f58d41e83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 --2- 172.21.15.6:0/1952104214 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58d40b1310 0x7f58d4153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 >> 172.21.15.6:0/1952104214 conn(0x7f58d40b5100 msgr2=0x7f58d4072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 shutdown_connections 2024-03-06T07:32:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:22.426+0000 7f58da000640 1 -- 172.21.15.6:0/1952104214 wait complete. 2024-03-06T07:32:22.449 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:22.449 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:22.449 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:32:22.449 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:22.450 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:22.450 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:22.450 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:22.450 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:22.450 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:22.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:22.450 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:32:22.502 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:22.502 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:32:22.503 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42315sec preferred_lft 42315sec 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:32:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:32:22.505 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:22.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:22.506 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:22.601 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:32:22.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:22.606 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:32:22.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:22.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:22.654 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:32:22.654 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:22.654 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:32:22.655 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:32:22.655 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:22.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:22.655 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:32:22.655 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:32:22.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:32:22.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:32:22.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:32:22.655 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:32:22.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:22.885 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:32:22.939 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:22.939 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:22.939 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:22.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:22.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:32:23.034 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:23.034 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:23.087 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:23.087 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:23.087 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:23.144 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:23.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:23.292 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:23.292 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:23.326 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:23.326 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:23.326 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:32:23.326 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:32:23.326 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:32:23.328 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:23.328 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:23.328 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:23.328 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:23.333 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:23.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:23.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.416 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:23.493 INFO:teuthology.orchestra.run.smithi006.stdout:client6049 2024-03-06T07:32:23.493 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:23.493 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:23.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.552 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:32:23.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.623 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:32:23.705 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:23.706 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:23.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:23.765 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:32:24.157 INFO:teuthology.orchestra.run.smithi006.stderr:35+0 records in 2024-03-06T07:32:24.157 INFO:teuthology.orchestra.run.smithi006.stderr:35+0 records out 2024-03-06T07:32:24.157 INFO:teuthology.orchestra.run.smithi006.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.338631 s, 108 MB/s 2024-03-06T07:32:24.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:24.159 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:32:24.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.464+0000 7f9917e00640 1 Processor -- start 2024-03-06T07:32:24.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.464+0000 7f9917e00640 1 -- start start 2024-03-06T07:32:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9917e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 0x7f99181667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9917e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180714d0 con 0x7f9918070f10 2024-03-06T07:32:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9917e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99180717b0 con 0x7f9918145300 2024-03-06T07:32:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9917e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9918071a90 con 0x7f99180cd060 2024-03-06T07:32:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9916000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9918070f10 0x7f99180cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37816/0 (socket says 172.21.15.6:37816) 2024-03-06T07:32:24.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9916000640 1 -- 172.21.15.6:0/1058471020 learned_addr learned my addr 172.21.15.6:0/1058471020 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:24.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.466+0000 7f9917400640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 0x7f99181667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009813378 0 0) 0x7f99180714d0 con 0x7f9918070f10 2024-03-06T07:32:24.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98fc003520 con 0x7f9918070f10 2024-03-06T07:32:24.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3654827339 0 0) 0x7f98fc003520 con 0x7f9918070f10 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 msgr2=0x7f99181667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 0x7f99181667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 >> v1:172.21.15.178:6789/0 conn(0x7f9918145300 legacy=0x7f99180b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9918071d70 con 0x7f9918070f10 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9900001670 con 0x7f9918070f10 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 >> v1:172.21.15.6:6789/0 conn(0x7f9918070f10 legacy=0x7f99180cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 0x7f98fc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f98fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98fc004d40 0x7f98fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9918071a90 con 0x7f98fc003fb0 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc00bbb0 con 0x7f98fc004d40 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.467+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc00bdb0 con 0x7f99180cd060 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916000640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f98fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916a00640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 0x7f98fc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9917400640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98fc004d40 0x7f98fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916000640 1 -- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 msgr2=0x7f98fc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916000640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f99180cd060 0x7f98fc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916000640 1 -- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98fc004d40 msgr2=0x7f98fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916000640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98fc004d40 0x7f98fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.468+0000 7f9916000640 1 -- 172.21.15.6:0/1058471020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc003520 con 0x7f98fc003fb0 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.470+0000 7f9916000640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f98fc004800 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f990000c110 tx=0x7f98fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa3b3530a1916f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.470+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900007c40 con 0x7f98fc003fb0 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.470+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9900030070 con 0x7f98fc003fb0 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.470+0000 7f9915600640 1 -- 172.21.15.6:0/1058471020 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900005d10 con 0x7f98fc003fb0 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917e00640 1 -- 172.21.15.6:0/1058471020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 msgr2=0x7f98fc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917e00640 1 --2- 172.21.15.6:0/1058471020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f98fc004800 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f990000c110 tx=0x7f98fc005830 comp rx=0 tx=0).stop 2024-03-06T07:32:24.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917400640 1 -- 172.21.15.6:0/1058471020 reap_dead start 2024-03-06T07:32:24.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917e00640 1 -- 172.21.15.6:0/1058471020 shutdown_connections 2024-03-06T07:32:24.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917e00640 1 -- 172.21.15.6:0/1058471020 >> 172.21.15.6:0/1058471020 conn(0x7f991806b3d0 msgr2=0x7f991806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917e00640 1 -- 172.21.15.6:0/1058471020 shutdown_connections 2024-03-06T07:32:24.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.471+0000 7f9917e00640 1 -- 172.21.15.6:0/1058471020 wait complete. 2024-03-06T07:32:24.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.473+0000 7f9917e00640 1 Processor -- start 2024-03-06T07:32:24.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.473+0000 7f9917e00640 1 -- start start 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9917e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f99181f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9917e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f991813d0b0 0x7f991813d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9917e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f991806dd40 0x7f991806e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9917e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991815d700 con 0x7f98fc003fb0 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9917e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991815d580 con 0x7f991813d0b0 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9917e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991815d880 con 0x7f991806dd40 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.474+0000 7f9916a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f99181f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f99181f41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60122/0 (socket says 172.21.15.6:60122) 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 -- 172.21.15.6:0/694021305 learned_addr learned my addr 172.21.15.6:0/694021305 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:24.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9917400640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f991806dd40 0x7f991806e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916000640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f991813d0b0 0x7f991813d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 -- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f991806dd40 msgr2=0x7f991806e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f991806dd40 0x7f991806e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 -- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f991813d0b0 msgr2=0x7f991813d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f991813d0b0 0x7f991813d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 -- 172.21.15.6:0/694021305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991813b7a0 con 0x7f98fc003fb0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9917400640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f991806dd40 0x7f991806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916000640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f991813d0b0 0x7f991813d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.475+0000 7f9916a00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f99181f41a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f990800fb90 tx=0x7f990800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.476+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990800fe90 con 0x7f98fc003fb0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.476+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9908018070 con 0x7f98fc003fb0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.476+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9908003970 con 0x7f98fc003fb0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.476+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991813ba60 con 0x7f98fc003fb0 2024-03-06T07:32:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.476+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991813c020 con 0x7f98fc003fb0 2024-03-06T07:32:24.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.477+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99180b0b10 con 0x7f98fc003fb0 2024-03-06T07:32:24.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.478+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9908003b10 con 0x7f98fc003fb0 2024-03-06T07:32:24.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.479+0000 7f990ea00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98f406b6a0 0x7f98f406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.479+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f990808a460 con 0x7f98fc003fb0 2024-03-06T07:32:24.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.479+0000 7f9916000640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98f406b6a0 0x7f98f406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.480+0000 7f9916000640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98f406b6a0 0x7f98f406db60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9900034000 tx=0x7f9900009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.484+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99180b0b10 con 0x7f98fc003fb0 2024-03-06T07:32:24.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.727+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f991804f330 con 0x7f98fc003fb0 2024-03-06T07:32:24.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.728+0000 7f990ea00640 1 -- 172.21.15.6:0/694021305 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f9908059240 con 0x7f98fc003fb0 2024-03-06T07:32:24.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:24.731 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:24.731 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:24.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98f406b6a0 msgr2=0x7f98f406db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98f406b6a0 0x7f98f406db60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9900034000 tx=0x7f9900009030 comp rx=0 tx=0).stop 2024-03-06T07:32:24.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 msgr2=0x7f99181f41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f99181f41a0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f990800fb90 tx=0x7f990800d520 comp rx=0 tx=0).stop 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 shutdown_connections 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98f406b6a0 0x7f98f406db60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f991813d0b0 0x7f991813d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f991806dd40 0x7f991806e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 --2- 172.21.15.6:0/694021305 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98fc003fb0 0x7f99181f41a0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 >> 172.21.15.6:0/694021305 conn(0x7f991806b3d0 msgr2=0x7f99180b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 shutdown_connections 2024-03-06T07:32:24.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:24.731+0000 7f9917e00640 1 -- 172.21.15.6:0/694021305 wait complete. 2024-03-06T07:32:24.752 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:32:25.012 INFO:teuthology.orchestra.run.smithi178.stdout:{"message":"","return_code":0} 2024-03-06T07:32:25.013 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:32:25.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:32:25.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.234+0000 7fcc15800640 1 Processor -- start 2024-03-06T07:32:25.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.234+0000 7fcc15800640 1 -- start start 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc15800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcc100b4600 0x7fcc100b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc15800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcc10070a60 0x7fcc10070ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc15800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100b2b80 con 0x7fcc10144340 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc15800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc100cc780 con 0x7fcc10070a60 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc15800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc100b3790 con 0x7fcc100b4600 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc14e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcc10070a60 0x7fcc10070ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc14e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcc10070a60 0x7fcc10070ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60138/0 (socket says 172.21.15.6:60138) 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc14e00640 1 -- 172.21.15.6:0/2043307474 learned_addr learned my addr 172.21.15.6:0/2043307474 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc0f400640 1 --2- 172.21.15.6:0/2043307474 >> v2:172.21.15.178:3300/0 conn(0x7fcc100b4600 0x7fcc100b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc14e00640 1 -- 172.21.15.6:0/2043307474 >> v2:172.21.15.178:3300/0 conn(0x7fcc100b4600 msgr2=0x7fcc100b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc14e00640 1 --2- 172.21.15.6:0/2043307474 >> v2:172.21.15.178:3300/0 conn(0x7fcc100b4600 0x7fcc100b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.236+0000 7fcc14e00640 1 -- 172.21.15.6:0/2043307474 >> v1:172.21.15.178:6789/0 conn(0x7fcc10144340 legacy=0x7fcc100ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.237+0000 7fcc14e00640 1 -- 172.21.15.6:0/2043307474 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc100b2e60 con 0x7fcc10070a60 2024-03-06T07:32:25.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.238+0000 7fcc14e00640 1 --2- 172.21.15.6:0/2043307474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10070ed0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcc1006e450 tx=0x7fcc0802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c685d101dca58b0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.238+0000 7fcc0ea00640 1 -- 172.21.15.6:0/2043307474 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0802c9f0 con 0x7fcc10070a60 2024-03-06T07:32:25.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.238+0000 7fcc0ea00640 1 -- 172.21.15.6:0/2043307474 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0802cb90 con 0x7fcc10070a60 2024-03-06T07:32:25.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.238+0000 7fcc0ea00640 1 -- 172.21.15.6:0/2043307474 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc080343f0 con 0x7fcc10070a60 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 -- 172.21.15.6:0/2043307474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 msgr2=0x7fcc10070ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 --2- 172.21.15.6:0/2043307474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10070ed0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fcc1006e450 tx=0x7fcc0802be50 comp rx=0 tx=0).stop 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 -- 172.21.15.6:0/2043307474 shutdown_connections 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 --2- 172.21.15.6:0/2043307474 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10070ed0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 --2- 172.21.15.6:0/2043307474 >> v2:172.21.15.178:3300/0 conn(0x7fcc100b4600 0x7fcc100b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 -- 172.21.15.6:0/2043307474 >> 172.21.15.6:0/2043307474 conn(0x7fcc1006b190 msgr2=0x7fcc1006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 -- 172.21.15.6:0/2043307474 shutdown_connections 2024-03-06T07:32:25.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.239+0000 7fcc15800640 1 -- 172.21.15.6:0/2043307474 wait complete. 2024-03-06T07:32:25.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.240+0000 7fcc15800640 1 Processor -- start 2024-03-06T07:32:25.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.241+0000 7fcc15800640 1 -- start start 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.241+0000 7fcc15800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc15800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc100b4600 0x7fcc1015a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc15800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc10144340 0x7fcc1015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc15800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10144720 con 0x7fcc10070a60 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc15800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100b3420 con 0x7fcc10144340 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc15800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc100b3c20 con 0x7fcc100b4600 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60144/0 (socket says 172.21.15.6:60144) 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0fe00640 1 -- 172.21.15.6:0/4227264688 learned_addr learned my addr 172.21.15.6:0/4227264688 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0f400640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc100b4600 0x7fcc1015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc14e00640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc10144340 0x7fcc1015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0fe00640 1 -- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc100b4600 msgr2=0x7fcc1015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0fe00640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc100b4600 0x7fcc1015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.242+0000 7fcc0fe00640 1 -- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc10144340 msgr2=0x7fcc1015d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc0fe00640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc10144340 0x7fcc1015d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc0fe00640 1 -- 172.21.15.6:0/4227264688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1013ff10 con 0x7fcc10070a60 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc14e00640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc10144340 0x7fcc1015d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc0f400640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc100b4600 0x7fcc1015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc0fe00640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10141330 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000b420 tx=0x7fcc0000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00011830 con 0x7fcc10070a60 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00016070 con 0x7fcc10070a60 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00011c10 con 0x7fcc10070a60 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc101401f0 con 0x7fcc10070a60 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.243+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc101407b0 con 0x7fcc10070a60 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.244+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1004f330 con 0x7fcc10070a60 2024-03-06T07:32:25.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.245+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc0001b460 con 0x7fcc10070a60 2024-03-06T07:32:25.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.246+0000 7fcc07e00640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcbf006b4d0 0x7fcbf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.246+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc00089d10 con 0x7fcc10070a60 2024-03-06T07:32:25.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.247+0000 7fcc0f400640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcbf006b4d0 0x7fcbf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.248+0000 7fcc0f400640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcbf006b4d0 0x7fcbf006d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcc1006de30 tx=0x7fcbfc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.251+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc0008f020 con 0x7fcc10070a60 2024-03-06T07:32:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.527+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fcc100722e0 con 0x7fcc10070a60 2024-03-06T07:32:25.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.528+0000 7fcc07e00640 1 -- 172.21.15.6:0/4227264688 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1651 (secure 0 0 0) 0x7fcc00020020 con 0x7fcc10070a60 2024-03-06T07:32:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:25.530 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:32:25.530 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772812496896, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 281616384, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 281616384, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00036427180748432875, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "num_osds": 8, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_osds": 8, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "stats_by_class": { 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "ssd": { 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_bytes": 773094113280, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_avail_bytes": 772812496896, 2024-03-06T07:32:25.531 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_bytes": 281616384, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_bytes": 281616384, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "total_used_raw_ratio": 0.00036427180748432875 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "name": ".mgr", 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 1, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 590368, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 2, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 1160, 2024-03-06T07:32:25.532 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 1187840, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 1.61800551268243e-06, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367068610560 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_metadata", 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 12, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 2410, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 22, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 64, 2024-03-06T07:32:25.533 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 65536, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 8.9269406089442782e-08, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367068610560 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "cephfs_data", 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 13, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "stats": { 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "stored": 0, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "objects": 0, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "kb_used": 0, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "bytes_used": 0, 2024-03-06T07:32:25.534 INFO:teuthology.orchestra.run.smithi006.stdout: "percent_used": 0, 2024-03-06T07:32:25.535 INFO:teuthology.orchestra.run.smithi006.stdout: "max_avail": 367068610560 2024-03-06T07:32:25.535 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:25.535 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:25.535 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:32:25.535 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:32:25.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcbf006b4d0 msgr2=0x7fcbf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcbf006b4d0 0x7fcbf006d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcc1006de30 tx=0x7fcbfc008040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 msgr2=0x7fcc10141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10141330 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000b420 tx=0x7fcc0000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 shutdown_connections 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcbf006b4d0 0x7fcbf006d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc10144340 0x7fcc1015d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc100b4600 0x7fcc1015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 --2- 172.21.15.6:0/4227264688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc10070a60 0x7fcc10141330 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 >> 172.21.15.6:0/4227264688 conn(0x7fcc1006b190 msgr2=0x7fcc10070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 shutdown_connections 2024-03-06T07:32:25.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:25.531+0000 7fcc15800640 1 -- 172.21.15.6:0/4227264688 wait complete. 2024-03-06T07:32:25.553 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:32:25.553 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:25.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:25.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:25.790 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:25.790 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:25.790 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.850 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:25.850 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:25.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:26.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.136+0000 7fa95f400640 1 Processor -- start 2024-03-06T07:32:26.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.137+0000 7fa95f400640 1 -- start start 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.138+0000 7fa95f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af580 0x7fa95815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.138+0000 7fa95f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa9580cdc20 0x7fa9580b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.138+0000 7fa95f400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95815cc40 con 0x7fa9580af1a0 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.138+0000 7fa95f400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95815cf20 con 0x7fa9580cdc20 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.138+0000 7fa95f400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95815d200 con 0x7fa9580af580 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa9580cdc20 0x7fa9580b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa9580cdc20 0x7fa9580b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60156/0 (socket says 172.21.15.6:60156) 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 -- 172.21.15.6:0/3391094416 learned_addr learned my addr 172.21.15.6:0/3391094416 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:26.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa957e00640 1 --2- 172.21.15.6:0/3391094416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af580 0x7fa95815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 -- 172.21.15.6:0/3391094416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af580 msgr2=0x7fa95815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 --2- 172.21.15.6:0/3391094416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af580 0x7fa95815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 -- 172.21.15.6:0/3391094416 >> v1:172.21.15.6:6789/0 conn(0x7fa9580af1a0 legacy=0x7fa9580b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.139+0000 7fa95d400640 1 -- 172.21.15.6:0/3391094416 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95815d4e0 con 0x7fa9580cdc20 2024-03-06T07:32:26.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.140+0000 7fa95d400640 1 --2- 172.21.15.6:0/3391094416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580cdc20 0x7fa9580b2dd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa950009870 tx=0x7fa95002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59eea27f1c26efb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa957400640 1 -- 172.21.15.6:0/3391094416 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa950035020 con 0x7fa9580cdc20 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa957400640 1 -- 172.21.15.6:0/3391094416 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa950031640 con 0x7fa9580cdc20 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa957400640 1 -- 172.21.15.6:0/3391094416 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95002d040 con 0x7fa9580cdc20 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa95f400640 1 -- 172.21.15.6:0/3391094416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580cdc20 msgr2=0x7fa9580b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa95f400640 1 --2- 172.21.15.6:0/3391094416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580cdc20 0x7fa9580b2dd0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa950009870 tx=0x7fa95002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa95f400640 1 -- 172.21.15.6:0/3391094416 shutdown_connections 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa95f400640 1 --2- 172.21.15.6:0/3391094416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580cdc20 0x7fa9580b2dd0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa95f400640 1 --2- 172.21.15.6:0/3391094416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af580 0x7fa95815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.141+0000 7fa95f400640 1 -- 172.21.15.6:0/3391094416 >> 172.21.15.6:0/3391094416 conn(0x7fa95806afb0 msgr2=0x7fa95806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.142+0000 7fa95f400640 1 -- 172.21.15.6:0/3391094416 shutdown_connections 2024-03-06T07:32:26.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.142+0000 7fa95f400640 1 -- 172.21.15.6:0/3391094416 wait complete. 2024-03-06T07:32:26.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.143+0000 7fa95f400640 1 Processor -- start 2024-03-06T07:32:26.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.144+0000 7fa95f400640 1 -- start start 2024-03-06T07:32:26.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af1a0 0x7fa95809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 0x7fa9581f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95f400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9580cdc20 0x7fa9581f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95f400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580b36a0 con 0x7fa9580af580 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95f400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580b3520 con 0x7fa9580cdc20 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95f400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9580b3820 con 0x7fa9580af1a0 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 0x7fa9581f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 0x7fa9581f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60158/0 (socket says 172.21.15.6:60158) 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 -- 172.21.15.6:0/990802160 learned_addr learned my addr 172.21.15.6:0/990802160 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa95ca00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af1a0 0x7fa95809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 -- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af1a0 msgr2=0x7fa95809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af1a0 0x7fa95809aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 -- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9580cdc20 msgr2=0x7fa9581f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9580cdc20 0x7fa9581f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.145+0000 7fa957e00640 1 -- 172.21.15.6:0/990802160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa958156cc0 con 0x7fa9580af580 2024-03-06T07:32:26.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.146+0000 7fa957e00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 0x7fa9581f3e80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d830 tx=0x7fa94400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.146+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400b6c0 con 0x7fa9580af580 2024-03-06T07:32:26.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.146+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa944019070 con 0x7fa9580af580 2024-03-06T07:32:26.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.146+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94400ba80 con 0x7fa9580af580 2024-03-06T07:32:26.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.146+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa958156fa0 con 0x7fa9580af580 2024-03-06T07:32:26.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.146+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa958157560 con 0x7fa9580af580 2024-03-06T07:32:26.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.148+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa958069510 con 0x7fa9580af580 2024-03-06T07:32:26.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.148+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa94401e460 con 0x7fa9580af580 2024-03-06T07:32:26.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.149+0000 7fa954c00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa93806b4d0 0x7fa93806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.149+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa94408a8b0 con 0x7fa9580af580 2024-03-06T07:32:26.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.150+0000 7fa95ca00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa93806b4d0 0x7fa93806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.151+0000 7fa95ca00640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa93806b4d0 0x7fa93806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa948001d40 tx=0x7fa948009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.154+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa944059730 con 0x7fa9580af580 2024-03-06T07:32:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.354+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa95815f600 con 0x7fa9580af580 2024-03-06T07:32:26.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.357+0000 7fa954c00640 1 -- 172.21.15.6:0/990802160 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fa944059030 con 0x7fa9580af580 2024-03-06T07:32:26.359 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:26.360 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:16.181620+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.359+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa93806b4d0 msgr2=0x7fa93806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.359+0000 7fa95f400640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa93806b4d0 0x7fa93806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fa948001d40 tx=0x7fa948009040 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.359+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 msgr2=0x7fa9581f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.359+0000 7fa95f400640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 0x7fa9581f3e80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa94400d830 tx=0x7fa94400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 shutdown_connections 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa93806b4d0 0x7fa93806d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa9580cdc20 0x7fa9581f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa9580af580 0x7fa9581f3e80 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 --2- 172.21.15.6:0/990802160 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa9580af1a0 0x7fa95809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 >> 172.21.15.6:0/990802160 conn(0x7fa95806afb0 msgr2=0x7fa958071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 shutdown_connections 2024-03-06T07:32:26.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.360+0000 7fa95f400640 1 -- 172.21.15.6:0/990802160 wait complete. 2024-03-06T07:32:26.380 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:12.534039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:13.644895+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:26.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:26.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.606+0000 7fb610a00640 1 Processor -- start 2024-03-06T07:32:26.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.606+0000 7fb610a00640 1 -- start start 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.607+0000 7fb610a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c0d07d0 0x7fb60c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.607+0000 7fb610a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d4930 0x7fb60c15dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.607+0000 7fb610a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb60c0d0bb0 0x7fb60c15e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.607+0000 7fb610a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0b3770 con 0x7fb60c0d07d0 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb610a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb60c0cf780 con 0x7fb60c0d0bb0 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb610a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0cfa60 con 0x7fb60c0d4930 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c0d07d0 0x7fb60c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb60c0d07d0 0x7fb60c0b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60172/0 (socket says 172.21.15.6:60172) 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 -- 172.21.15.6:0/2311329524 learned_addr learned my addr 172.21.15.6:0/2311329524 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609400640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d4930 0x7fb60c15dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb60a800640 1 --2- 172.21.15.6:0/2311329524 >> v2:172.21.15.178:3300/0 conn(0x7fb60c0d0bb0 0x7fb60c15e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 -- 172.21.15.6:0/2311329524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d4930 msgr2=0x7fb60c15dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d4930 0x7fb60c15dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 -- 172.21.15.6:0/2311329524 >> v2:172.21.15.178:3300/0 conn(0x7fb60c0d0bb0 msgr2=0x7fb60c15e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:26.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 --2- 172.21.15.6:0/2311329524 >> v2:172.21.15.178:3300/0 conn(0x7fb60c0d0bb0 0x7fb60c15e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609e00640 1 -- 172.21.15.6:0/2311329524 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0cfd40 con 0x7fb60c0d07d0 2024-03-06T07:32:26.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.608+0000 7fb609400640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d4930 0x7fb60c15dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.610+0000 7fb609e00640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d07d0 0x7fb60c0b4f80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb60c08d2b0 tx=0x7fb5fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bf32087de23ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.610+0000 7fb608a00640 1 -- 172.21.15.6:0/2311329524 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc034070 con 0x7fb60c0d07d0 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.610+0000 7fb608a00640 1 -- 172.21.15.6:0/2311329524 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5fc030860 con 0x7fb60c0d07d0 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.610+0000 7fb608a00640 1 -- 172.21.15.6:0/2311329524 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc030c30 con 0x7fb60c0d07d0 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 -- 172.21.15.6:0/2311329524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d07d0 msgr2=0x7fb60c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d07d0 0x7fb60c0b4f80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb60c08d2b0 tx=0x7fb5fc02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 -- 172.21.15.6:0/2311329524 shutdown_connections 2024-03-06T07:32:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 --2- 172.21.15.6:0/2311329524 >> v2:172.21.15.178:3300/0 conn(0x7fb60c0d0bb0 0x7fb60c15e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d4930 0x7fb60c15dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 --2- 172.21.15.6:0/2311329524 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d07d0 0x7fb60c0b4f80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 -- 172.21.15.6:0/2311329524 >> 172.21.15.6:0/2311329524 conn(0x7fb60c0d3e10 msgr2=0x7fb60c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 -- 172.21.15.6:0/2311329524 shutdown_connections 2024-03-06T07:32:26.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.611+0000 7fb610a00640 1 -- 172.21.15.6:0/2311329524 wait complete. 2024-03-06T07:32:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.613+0000 7fb610a00640 1 Processor -- start 2024-03-06T07:32:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.613+0000 7fb610a00640 1 -- start start 2024-03-06T07:32:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb610a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d07d0 0x7fb60c1f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb610a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb60c0d0bb0 0x7fb60c0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb610a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 0x7fb60c1f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb610a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0d1d70 con 0x7fb60c0d4930 2024-03-06T07:32:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb610a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0d1bf0 con 0x7fb60c0d0bb0 2024-03-06T07:32:26.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb610a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c0d1ef0 con 0x7fb60c0d07d0 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb60a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 0x7fb60c1f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb60a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 0x7fb60c1f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60176/0 (socket says 172.21.15.6:60176) 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb60a800640 1 -- 172.21.15.6:0/744818097 learned_addr learned my addr 172.21.15.6:0/744818097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb609e00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d07d0 0x7fb60c1f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.614+0000 7fb609400640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb60c0d0bb0 0x7fb60c0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.615+0000 7fb60a800640 1 -- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d07d0 msgr2=0x7fb60c1f4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.615+0000 7fb60a800640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d07d0 0x7fb60c1f4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.615+0000 7fb60a800640 1 -- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb60c0d0bb0 msgr2=0x7fb60c0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.615+0000 7fb60a800640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb60c0d0bb0 0x7fb60c0b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.615+0000 7fb60a800640 1 -- 172.21.15.6:0/744818097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c0b2c80 con 0x7fb60c0d4930 2024-03-06T07:32:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.615+0000 7fb60a800640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 0x7fb60c1f37c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb60400c5a0 tx=0x7fb60400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.616+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604015070 con 0x7fb60c0d4930 2024-03-06T07:32:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.616+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb604004220 con 0x7fb60c0d4930 2024-03-06T07:32:26.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.616+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60c15f060 con 0x7fb60c0d4930 2024-03-06T07:32:26.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.616+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604003610 con 0x7fb60c0d4930 2024-03-06T07:32:26.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.616+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60c1f4070 con 0x7fb60c0d4930 2024-03-06T07:32:26.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.617+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60c04f330 con 0x7fb60c0d4930 2024-03-06T07:32:26.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.619+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6040037b0 con 0x7fb60c0d4930 2024-03-06T07:32:26.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.619+0000 7fb602000640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb5ec06b4d0 0x7fb5ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:26.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.620+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb604089fd0 con 0x7fb60c0d4930 2024-03-06T07:32:26.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.620+0000 7fb609e00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb5ec06b4d0 0x7fb5ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:26.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.621+0000 7fb609e00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb5ec06b4d0 0x7fb5ec06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc036000 tx=0x7fb5fc008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:26.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.624+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb604058760 con 0x7fb60c0d4930 2024-03-06T07:32:26.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.828+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb60c0ceda0 con 0x7fb60c0d4930 2024-03-06T07:32:26.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.829+0000 7fb602000640 1 -- 172.21.15.6:0/744818097 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb60405c800 con 0x7fb60c0d4930 2024-03-06T07:32:26.831 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.831+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb5ec06b4d0 msgr2=0x7fb5ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.831+0000 7fb610a00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb5ec06b4d0 0x7fb5ec06d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc036000 tx=0x7fb5fc008030 comp rx=0 tx=0).stop 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.831+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 msgr2=0x7fb60c1f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.831+0000 7fb610a00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 0x7fb60c1f37c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fb60400c5a0 tx=0x7fb60400ca60 comp rx=0 tx=0).stop 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 shutdown_connections 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb5ec06b4d0 0x7fb5ec06d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb60c0d0bb0 0x7fb60c0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb60c0d4930 0x7fb60c1f37c0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 --2- 172.21.15.6:0/744818097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb60c0d07d0 0x7fb60c1f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 >> 172.21.15.6:0/744818097 conn(0x7fb60c0d3e10 msgr2=0x7fb60c0b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 shutdown_connections 2024-03-06T07:32:26.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:26.832+0000 7fb610a00640 1 -- 172.21.15.6:0/744818097 wait complete. 2024-03-06T07:32:26.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.074+0000 7f2c0d000640 1 Processor -- start 2024-03-06T07:32:27.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.074+0000 7f2c0d000640 1 -- start start 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.075+0000 7f2c0d000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2c08145430 0x7f2c0815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.075+0000 7f2c0d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c080b2250 0x7f2c080b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.075+0000 7f2c0d000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0815fe40 con 0x7f2c0815cc90 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.075+0000 7f2c0d000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c08160120 con 0x7f2c080b2250 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.075+0000 7f2c0d000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c08160400 con 0x7f2c08145430 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c080b2250 0x7f2c080b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c080b2250 0x7f2c080b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60182/0 (socket says 172.21.15.6:60182) 2024-03-06T07:32:27.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 -- 172.21.15.6:0/2056963741 learned_addr learned my addr 172.21.15.6:0/2056963741 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 -- 172.21.15.6:0/2056963741 >> v2:172.21.15.178:3300/0 conn(0x7f2c08145430 msgr2=0x7f2c0815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 --2- 172.21.15.6:0/2056963741 >> v2:172.21.15.178:3300/0 conn(0x7f2c08145430 0x7f2c0815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 -- 172.21.15.6:0/2056963741 >> v1:172.21.15.178:6789/0 conn(0x7f2c0815cc90 legacy=0x7f2c080b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.076+0000 7f2c07e00640 1 -- 172.21.15.6:0/2056963741 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c081606e0 con 0x7f2c080b2250 2024-03-06T07:32:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.077+0000 7f2c07e00640 1 --2- 172.21.15.6:0/2056963741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c080b2fd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2c080b2670 tx=0x7f2c0002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a661748df6f8005 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.078+0000 7f2c06000640 1 -- 172.21.15.6:0/2056963741 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0002ca80 con 0x7f2c080b2250 2024-03-06T07:32:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.078+0000 7f2c06000640 1 -- 172.21.15.6:0/2056963741 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0002cc20 con 0x7f2c080b2250 2024-03-06T07:32:27.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.078+0000 7f2c06000640 1 -- 172.21.15.6:0/2056963741 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c000343f0 con 0x7f2c080b2250 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.078+0000 7f2c0d000640 1 -- 172.21.15.6:0/2056963741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 msgr2=0x7f2c080b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.078+0000 7f2c0d000640 1 --2- 172.21.15.6:0/2056963741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c080b2fd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2c080b2670 tx=0x7f2c0002bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.079+0000 7f2c0d000640 1 -- 172.21.15.6:0/2056963741 shutdown_connections 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.079+0000 7f2c0d000640 1 --2- 172.21.15.6:0/2056963741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c080b2fd0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.079+0000 7f2c0d000640 1 --2- 172.21.15.6:0/2056963741 >> v2:172.21.15.178:3300/0 conn(0x7f2c08145430 0x7f2c0815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.079+0000 7f2c0d000640 1 -- 172.21.15.6:0/2056963741 >> 172.21.15.6:0/2056963741 conn(0x7f2c0806b3a0 msgr2=0x7f2c0806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.079+0000 7f2c0d000640 1 -- 172.21.15.6:0/2056963741 shutdown_connections 2024-03-06T07:32:27.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.079+0000 7f2c0d000640 1 -- 172.21.15.6:0/2056963741 wait complete. 2024-03-06T07:32:27.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.080+0000 7f2c0d000640 1 Processor -- start 2024-03-06T07:32:27.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.081+0000 7f2c0d000640 1 -- start start 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.081+0000 7f2c0d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c0813bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c0d000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c08145430 0x7f2c08139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c0d000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c0815cc90 0x7f2c0813a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c0d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08145990 con 0x7f2c080b2250 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c0d000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c08145810 con 0x7f2c08145430 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c0d000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c080cde00 con 0x7f2c0815cc90 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c0813bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c0813bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60190/0 (socket says 172.21.15.6:60190) 2024-03-06T07:32:27.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 -- 172.21.15.6:0/43379391 learned_addr learned my addr 172.21.15.6:0/43379391 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:27.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c06a00640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c08145430 0x7f2c08139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07e00640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c0815cc90 0x7f2c0813a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 -- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c0815cc90 msgr2=0x7f2c0813a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c0815cc90 0x7f2c0813a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 -- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c08145430 msgr2=0x7f2c08139bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c08145430 0x7f2c08139bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.082+0000 7f2c07400640 1 -- 172.21.15.6:0/43379391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0813c4f0 con 0x7f2c080b2250 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.083+0000 7f2c07400640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c0813bc90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800b4a0 tx=0x7f2bf800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.083+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8014070 con 0x7f2c080b2250 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.083+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf8010610 con 0x7f2c080b2250 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.083+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8010a00 con 0x7f2c080b2250 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.083+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c0813c7d0 con 0x7f2c080b2250 2024-03-06T07:32:27.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.083+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c0813cd90 con 0x7f2c080b2250 2024-03-06T07:32:27.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.085+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0804f330 con 0x7f2c080b2250 2024-03-06T07:32:27.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.086+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bf8017070 con 0x7f2c080b2250 2024-03-06T07:32:27.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.087+0000 7f2bff400640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2be406b4d0 0x7f2be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.087+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bf8089c40 con 0x7f2c080b2250 2024-03-06T07:32:27.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.088+0000 7f2c06a00640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2be406b4d0 0x7f2be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.089+0000 7f2c06a00640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2be406b4d0 0x7f2be406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006fd0 tx=0x7f2bf0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.092+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bf8058ac0 con 0x7f2c080b2250 2024-03-06T07:32:27.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.295+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c081404b0 con 0x7f2c080b2250 2024-03-06T07:32:27.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.296+0000 7f2bff400640 1 -- 172.21.15.6:0/43379391 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2bf80583c0 con 0x7f2c080b2250 2024-03-06T07:32:27.298 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2be406b4d0 msgr2=0x7f2be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2be406b4d0 0x7f2be406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006fd0 tx=0x7f2bf0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 msgr2=0x7f2c0813bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c0813bc90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800b4a0 tx=0x7f2bf800b960 comp rx=0 tx=0).stop 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 shutdown_connections 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2be406b4d0 0x7f2be406d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c08145430 0x7f2c08139bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c0815cc90 0x7f2c0813a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 --2- 172.21.15.6:0/43379391 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c080b2250 0x7f2c0813bc90 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 >> 172.21.15.6:0/43379391 conn(0x7f2c0806b3a0 msgr2=0x7f2c080d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 shutdown_connections 2024-03-06T07:32:27.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.299+0000 7f2c0d000640 1 -- 172.21.15.6:0/43379391 wait complete. 2024-03-06T07:32:27.321 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:27.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.550+0000 7fc059000640 1 Processor -- start 2024-03-06T07:32:27.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.550+0000 7fc059000640 1 -- start start 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc059000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc05406ed40 0x7fc0540b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc059000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406f830 0x7fc0540ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc059000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05406dee0 con 0x7fc0540cc710 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc059000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05406e180 con 0x7fc05406ed40 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc059000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05406e460 con 0x7fc05406f830 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc052a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc0540cc710 0x7fc05406f120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37846/0 (socket says 172.21.15.6:37846) 2024-03-06T07:32:27.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc052a00640 1 -- 172.21.15.6:0/2872704122 learned_addr learned my addr 172.21.15.6:0/2872704122 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.552+0000 7fc053400640 1 --2- 172.21.15.6:0/2872704122 >> v2:172.21.15.178:3300/0 conn(0x7fc05406ed40 0x7fc0540b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc053e00640 1 --2- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406f830 0x7fc0540ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc052000640 1 -- 172.21.15.6:0/2872704122 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291751339 0 0) 0x7fc05406dee0 con 0x7fc0540cc710 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc052000640 1 -- 172.21.15.6:0/2872704122 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc038003440 con 0x7fc0540cc710 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc053400640 1 -- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406f830 msgr2=0x7fc0540ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc053400640 1 --2- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406f830 0x7fc0540ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc053400640 1 -- 172.21.15.6:0/2872704122 >> v1:172.21.15.6:6789/0 conn(0x7fc0540cc710 legacy=0x7fc05406f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc052000640 1 -- 172.21.15.6:0/2872704122 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828695085 0 0) 0x7fc038003440 con 0x7fc0540cc710 2024-03-06T07:32:27.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.553+0000 7fc053400640 1 -- 172.21.15.6:0/2872704122 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0540ce400 con 0x7fc05406ed40 2024-03-06T07:32:27.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.554+0000 7fc053400640 1 --2- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc05406ed40 0x7fc0540b3520 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc0440097a0 tx=0x7fc04402b660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5688ee439582c95f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.555+0000 7fc052000640 1 -- 172.21.15.6:0/2872704122 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc044037070 con 0x7fc05406ed40 2024-03-06T07:32:27.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.555+0000 7fc052000640 1 -- 172.21.15.6:0/2872704122 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc044030510 con 0x7fc05406ed40 2024-03-06T07:32:27.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.555+0000 7fc052000640 1 -- 172.21.15.6:0/2872704122 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04402be60 con 0x7fc05406ed40 2024-03-06T07:32:27.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 -- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc05406ed40 msgr2=0x7fc0540b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 --2- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc05406ed40 0x7fc0540b3520 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc0440097a0 tx=0x7fc04402b660 comp rx=0 tx=0).stop 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 -- 172.21.15.6:0/2872704122 shutdown_connections 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 --2- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406f830 0x7fc0540ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 --2- 172.21.15.6:0/2872704122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc05406ed40 0x7fc0540b3520 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 -- 172.21.15.6:0/2872704122 >> 172.21.15.6:0/2872704122 conn(0x7fc0540723d0 msgr2=0x7fc05406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 -- 172.21.15.6:0/2872704122 shutdown_connections 2024-03-06T07:32:27.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.556+0000 7fc059000640 1 -- 172.21.15.6:0/2872704122 wait complete. 2024-03-06T07:32:27.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.559+0000 7fc059000640 1 Processor -- start 2024-03-06T07:32:27.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.559+0000 7fc059000640 1 -- start start 2024-03-06T07:32:27.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.560+0000 7fc059000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406ed40 0x7fc05413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.560+0000 7fc059000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 0x7fc054156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.560+0000 7fc059000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0540cc710 0x7fc0541571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.560+0000 7fc059000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05406ea10 con 0x7fc05406f830 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.560+0000 7fc059000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054066d00 con 0x7fc0540cc710 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.560+0000 7fc059000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc054140f40 con 0x7fc05406ed40 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 0x7fc054156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 0x7fc054156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60196/0 (socket says 172.21.15.6:60196) 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 -- 172.21.15.6:0/303177689 learned_addr learned my addr 172.21.15.6:0/303177689 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc053400640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406ed40 0x7fc05413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc053e00640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0540cc710 0x7fc0541571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 -- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406ed40 msgr2=0x7fc05413e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406ed40 0x7fc05413e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 -- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0540cc710 msgr2=0x7fc0541571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0540cc710 0x7fc0541571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 -- 172.21.15.6:0/303177689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05413ec00 con 0x7fc05406f830 2024-03-06T07:32:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.561+0000 7fc052a00640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 0x7fc054156cb0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc04000caf0 tx=0x7fc040007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.562+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040017070 con 0x7fc05406f830 2024-03-06T07:32:27.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.562+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc040005ce0 con 0x7fc05406f830 2024-03-06T07:32:27.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.562+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400072b0 con 0x7fc05406f830 2024-03-06T07:32:27.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.562+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc054159930 con 0x7fc05406f830 2024-03-06T07:32:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.562+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0541414b0 con 0x7fc05406f830 2024-03-06T07:32:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.563+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05404f330 con 0x7fc05406f830 2024-03-06T07:32:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.564+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc04001a070 con 0x7fc05406f830 2024-03-06T07:32:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.565+0000 7fc04b400640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc03406b4d0 0x7fc03406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.566+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc040089980 con 0x7fc05406f830 2024-03-06T07:32:27.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.566+0000 7fc053400640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc03406b4d0 0x7fc03406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.567+0000 7fc053400640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc03406b4d0 0x7fc03406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc044034000 tx=0x7fc044008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.570+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc040058760 con 0x7fc05406f830 2024-03-06T07:32:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.771+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc0540cd2c0 con 0x7fc05406f830 2024-03-06T07:32:27.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.772+0000 7fc04b400640 1 -- 172.21.15.6:0/303177689 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc040058060 con 0x7fc05406f830 2024-03-06T07:32:27.774 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:32:27.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc03406b4d0 msgr2=0x7fc03406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc03406b4d0 0x7fc03406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc044034000 tx=0x7fc044008030 comp rx=0 tx=0).stop 2024-03-06T07:32:27.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 msgr2=0x7fc054156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 0x7fc054156cb0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc04000caf0 tx=0x7fc040007e50 comp rx=0 tx=0).stop 2024-03-06T07:32:27.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 shutdown_connections 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc03406b4d0 0x7fc03406d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0540cc710 0x7fc0541571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc05406f830 0x7fc054156cb0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 --2- 172.21.15.6:0/303177689 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc05406ed40 0x7fc05413e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.774+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 >> 172.21.15.6:0/303177689 conn(0x7fc0540723d0 msgr2=0x7fc054154fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.775+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 shutdown_connections 2024-03-06T07:32:27.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:27.775+0000 7fc059000640 1 -- 172.21.15.6:0/303177689 wait complete. 2024-03-06T07:32:27.797 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:28.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.018+0000 7f467e400640 1 Processor -- start 2024-03-06T07:32:28.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.018+0000 7f467e400640 1 -- start start 2024-03-06T07:32:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.020+0000 7f467e400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4678145300 0x7f46780b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.020+0000 7f467e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f46780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.020+0000 7f467e400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f46780cce90 0x7f46780cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.020+0000 7f467e400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46780cd300 con 0x7f4678145300 2024-03-06T07:32:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.020+0000 7f467e400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46780714d0 con 0x7f46780cce90 2024-03-06T07:32:28.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.020+0000 7f467e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46780717b0 con 0x7f4678070f10 2024-03-06T07:32:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4678145300 0x7f46780b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4678145300 0x7f46780b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60200/0 (socket says 172.21.15.6:60200) 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 -- 172.21.15.6:0/3058037888 learned_addr learned my addr 172.21.15.6:0/3058037888 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467da00640 1 --2- 172.21.15.6:0/3058037888 >> v2:172.21.15.178:3300/0 conn(0x7f46780cce90 0x7f46780cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f4677e00640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f46780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 -- 172.21.15.6:0/3058037888 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 msgr2=0x7f46780cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f46780cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 -- 172.21.15.6:0/3058037888 >> v2:172.21.15.178:3300/0 conn(0x7f46780cce90 msgr2=0x7f46780cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 --2- 172.21.15.6:0/3058037888 >> v2:172.21.15.178:3300/0 conn(0x7f46780cce90 0x7f46780cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f467d000640 1 -- 172.21.15.6:0/3058037888 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678071a90 con 0x7f4678145300 2024-03-06T07:32:28.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.021+0000 7f4677e00640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f46780cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:28.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.023+0000 7f467d000640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46780b49a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4678072420 tx=0x7f466c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ddfd9af2f2f22c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.023+0000 7f4677400640 1 -- 172.21.15.6:0/3058037888 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c035020 con 0x7f4678145300 2024-03-06T07:32:28.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.023+0000 7f4677400640 1 -- 172.21.15.6:0/3058037888 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c031870 con 0x7f4678145300 2024-03-06T07:32:28.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.023+0000 7f4677400640 1 -- 172.21.15.6:0/3058037888 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c031c40 con 0x7f4678145300 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.023+0000 7f467e400640 1 -- 172.21.15.6:0/3058037888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 msgr2=0x7f46780b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.023+0000 7f467e400640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46780b49a0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f4678072420 tx=0x7f466c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 -- 172.21.15.6:0/3058037888 shutdown_connections 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 --2- 172.21.15.6:0/3058037888 >> v2:172.21.15.178:3300/0 conn(0x7f46780cce90 0x7f46780cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f46780cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 --2- 172.21.15.6:0/3058037888 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46780b49a0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 -- 172.21.15.6:0/3058037888 >> 172.21.15.6:0/3058037888 conn(0x7f467806b3d0 msgr2=0x7f467806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 -- 172.21.15.6:0/3058037888 shutdown_connections 2024-03-06T07:32:28.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.024+0000 7f467e400640 1 -- 172.21.15.6:0/3058037888 wait complete. 2024-03-06T07:32:28.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.025+0000 7f467e400640 1 Processor -- start 2024-03-06T07:32:28.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.025+0000 7f467e400640 1 -- start start 2024-03-06T07:32:28.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.026+0000 7f467e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f4678136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.026+0000 7f467e400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46780cce90 0x7f46781e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.026+0000 7f467e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46781e9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.026+0000 7f467e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46781456e0 con 0x7f4678145300 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.026+0000 7f467e400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46780b4ee0 con 0x7f46780cce90 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.026+0000 7f467e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46780cde70 con 0x7f4678070f10 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46781e9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46781e9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60202/0 (socket says 172.21.15.6:60202) 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 -- 172.21.15.6:0/671688006 learned_addr learned my addr 172.21.15.6:0/671688006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467d000640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f4678136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 -- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 msgr2=0x7f4678136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f4678136480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f4677e00640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46780cce90 0x7f46781e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 -- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46780cce90 msgr2=0x7f46781e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46780cce90 0x7f46781e7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 -- 172.21.15.6:0/671688006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46781e8900 con 0x7f4678145300 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467d000640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f4678136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:28.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.027+0000 7f467da00640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46781e9370 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f467000efc0 tx=0x7f467000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.028+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670018020 con 0x7f4678145300 2024-03-06T07:32:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.028+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4670014480 con 0x7f4678145300 2024-03-06T07:32:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.028+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670014870 con 0x7f4678145300 2024-03-06T07:32:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.028+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46781e8be0 con 0x7f4678145300 2024-03-06T07:32:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.028+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46781e4700 con 0x7f4678145300 2024-03-06T07:32:28.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.029+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467804f330 con 0x7f4678145300 2024-03-06T07:32:28.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.030+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4670005d10 con 0x7f4678145300 2024-03-06T07:32:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.031+0000 7f4674c00640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465806b4d0 0x7f465806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.031+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f467008ad20 con 0x7f4678145300 2024-03-06T07:32:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.031+0000 7f467d000640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465806b4d0 0x7f465806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.033+0000 7f467d000640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465806b4d0 0x7f465806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f466c036000 tx=0x7f466c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.036+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4670022070 con 0x7f4678145300 2024-03-06T07:32:28.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.253+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f46781e7210 con 0x7f4678145300 2024-03-06T07:32:28.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.254+0000 7f4674c00640 1 -- 172.21.15.6:0/671688006 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4670010040 con 0x7f4678145300 2024-03-06T07:32:28.256 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465806b4d0 msgr2=0x7f465806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465806b4d0 0x7f465806d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f466c036000 tx=0x7f466c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 msgr2=0x7f46781e9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46781e9370 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f467000efc0 tx=0x7f467000c490 comp rx=0 tx=0).stop 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 shutdown_connections 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f465806b4d0 0x7f465806d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46780cce90 0x7f46781e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4678145300 0x7f46781e9370 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 --2- 172.21.15.6:0/671688006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4678070f10 0x7f4678136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 >> 172.21.15.6:0/671688006 conn(0x7f467806b3d0 msgr2=0x7f467815cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 shutdown_connections 2024-03-06T07:32:28.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.257+0000 7f467e400640 1 -- 172.21.15.6:0/671688006 wait complete. 2024-03-06T07:32:28.279 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:32:28.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:28.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:28.379 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:32:28.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:28.379 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:28.435 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:32:28.435 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:28.490 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:28.491 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:28.491 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:28.552 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:28.552 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:32:28.552 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:28.612 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:28.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.833+0000 7fe942000640 1 Processor -- start 2024-03-06T07:32:28.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.833+0000 7fe942000640 1 -- start start 2024-03-06T07:32:28.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.835+0000 7fe942000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe93c0af1a0 0x7fe93c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.835+0000 7fe942000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe93c0cdc20 0x7fe93c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.835+0000 7fe942000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c15cc40 con 0x7fe93c0af580 2024-03-06T07:32:28.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.835+0000 7fe942000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c15cf20 con 0x7fe93c0af1a0 2024-03-06T07:32:28.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.835+0000 7fe942000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c15d200 con 0x7fe93c0cdc20 2024-03-06T07:32:28.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.835+0000 7fe93b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe93c0af1a0 0x7fe93c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe93c0af1a0 0x7fe93c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60218/0 (socket says 172.21.15.6:60218) 2024-03-06T07:32:28.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93b400640 1 -- 172.21.15.6:0/1207095948 learned_addr learned my addr 172.21.15.6:0/1207095948 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:28.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93be00640 1 --2- 172.21.15.6:0/1207095948 >> v2:172.21.15.178:3300/0 conn(0x7fe93c0cdc20 0x7fe93c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93b400640 1 -- 172.21.15.6:0/1207095948 >> v2:172.21.15.178:3300/0 conn(0x7fe93c0cdc20 msgr2=0x7fe93c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93b400640 1 --2- 172.21.15.6:0/1207095948 >> v2:172.21.15.178:3300/0 conn(0x7fe93c0cdc20 0x7fe93c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93b400640 1 -- 172.21.15.6:0/1207095948 >> v1:172.21.15.178:6789/0 conn(0x7fe93c0af580 legacy=0x7fe93c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.836+0000 7fe93b400640 1 -- 172.21.15.6:0/1207095948 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c15d4e0 con 0x7fe93c0af1a0 2024-03-06T07:32:28.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.838+0000 7fe93b400640 1 --2- 172.21.15.6:0/1207095948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0af1a0 0x7fe93c15c600 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe93c06c0c0 tx=0x7fe92c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a401c56e50a1952 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.838+0000 7fe93aa00640 1 -- 172.21.15.6:0/1207095948 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c02e070 con 0x7fe93c0af1a0 2024-03-06T07:32:28.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.838+0000 7fe93aa00640 1 -- 172.21.15.6:0/1207095948 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c037070 con 0x7fe93c0af1a0 2024-03-06T07:32:28.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.838+0000 7fe93aa00640 1 -- 172.21.15.6:0/1207095948 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c031a00 con 0x7fe93c0af1a0 2024-03-06T07:32:28.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 -- 172.21.15.6:0/1207095948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0af1a0 msgr2=0x7fe93c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 --2- 172.21.15.6:0/1207095948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0af1a0 0x7fe93c15c600 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe93c06c0c0 tx=0x7fe92c02b880 comp rx=0 tx=0).stop 2024-03-06T07:32:28.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 -- 172.21.15.6:0/1207095948 shutdown_connections 2024-03-06T07:32:28.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 --2- 172.21.15.6:0/1207095948 >> v2:172.21.15.178:3300/0 conn(0x7fe93c0cdc20 0x7fe93c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 --2- 172.21.15.6:0/1207095948 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0af1a0 0x7fe93c15c600 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 -- 172.21.15.6:0/1207095948 >> 172.21.15.6:0/1207095948 conn(0x7fe93c06afb0 msgr2=0x7fe93c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 -- 172.21.15.6:0/1207095948 shutdown_connections 2024-03-06T07:32:28.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.839+0000 7fe942000640 1 -- 172.21.15.6:0/1207095948 wait complete. 2024-03-06T07:32:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.841+0000 7fe942000640 1 Processor -- start 2024-03-06T07:32:28.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.842+0000 7fe942000640 1 -- start start 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe942000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe93c0af1a0 0x7fe93c14f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe942000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe93c0af580 0x7fe93c150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe942000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 0x7fe93c0ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe942000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b36a0 con 0x7fe93c0cdc20 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe942000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b3520 con 0x7fe93c0af1a0 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe942000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c0b3820 con 0x7fe93c0af580 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe93be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 0x7fe93c0ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe93be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 0x7fe93c0ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60224/0 (socket says 172.21.15.6:60224) 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe93be00640 1 -- 172.21.15.6:0/1811504003 learned_addr learned my addr 172.21.15.6:0/1811504003 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe933e00640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe93c0af580 0x7fe93c150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.843+0000 7fe93b400640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe93c0af1a0 0x7fe93c14f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93be00640 1 -- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe93c0af580 msgr2=0x7fe93c150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93be00640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe93c0af580 0x7fe93c150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93be00640 1 -- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe93c0af1a0 msgr2=0x7fe93c14f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93be00640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe93c0af1a0 0x7fe93c14f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93be00640 1 -- 172.21.15.6:0/1811504003 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0c98f0 con 0x7fe93c0cdc20 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe933e00640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe93c0af580 0x7fe93c150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93b400640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe93c0af1a0 0x7fe93c14f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe93be00640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 0x7fe93c0ca990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe934007e60 tx=0x7fe93400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934014420 con 0x7fe93c0cdc20 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe934019070 con 0x7fe93c0cdc20 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934014820 con 0x7fe93c0cdc20 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.844+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c0c9bd0 con 0x7fe93c0cdc20 2024-03-06T07:32:28.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.845+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c0ca470 con 0x7fe93c0cdc20 2024-03-06T07:32:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.846+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c069510 con 0x7fe93c0cdc20 2024-03-06T07:32:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.847+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9340149c0 con 0x7fe93c0cdc20 2024-03-06T07:32:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.848+0000 7fe933400640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe91806b4d0 0x7fe91806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.848+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe93408aa70 con 0x7fe93c0cdc20 2024-03-06T07:32:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.849+0000 7fe93b400640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe91806b4d0 0x7fe91806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.850+0000 7fe93b400640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe91806b4d0 0x7fe91806d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe92c036000 tx=0x7fe92c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:28.854+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe9340598e0 con 0x7fe93c0cdc20 2024-03-06T07:32:29.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.097+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe93c04f3a0 con 0x7fe93c0cdc20 2024-03-06T07:32:29.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.098+0000 7fe933400640 1 -- 172.21.15.6:0/1811504003 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe9340591e0 con 0x7fe93c0cdc20 2024-03-06T07:32:29.101 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:29.101 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe91806b4d0 msgr2=0x7fe91806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe91806b4d0 0x7fe91806d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fe92c036000 tx=0x7fe92c03d040 comp rx=0 tx=0).stop 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 msgr2=0x7fe93c0ca990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 0x7fe93c0ca990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe934007e60 tx=0x7fe93400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 shutdown_connections 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe91806b4d0 0x7fe91806d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe93c0cdc20 0x7fe93c0ca990 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe93c0af580 0x7fe93c150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 --2- 172.21.15.6:0/1811504003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe93c0af1a0 0x7fe93c14f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.101+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 >> 172.21.15.6:0/1811504003 conn(0x7fe93c06afb0 msgr2=0x7fe93c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.102+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 shutdown_connections 2024-03-06T07:32:29.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.102+0000 7fe942000640 1 -- 172.21.15.6:0/1811504003 wait complete. 2024-03-06T07:32:29.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:29.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.347+0000 7fef63200640 1 Processor -- start 2024-03-06T07:32:29.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.347+0000 7fef63200640 1 -- start start 2024-03-06T07:32:29.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.349+0000 7fef63200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef641552b0 0x7fef6415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.349+0000 7fef63200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6415ed10 con 0x7fef6415e810 2024-03-06T07:32:29.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.349+0000 7fef63200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef6415eff0 con 0x7fef640cd060 2024-03-06T07:32:29.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.349+0000 7fef63200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6415f2d0 con 0x7fef641552b0 2024-03-06T07:32:29.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.349+0000 7fef59400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef6415e810 0x7fef640cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37856/0 (socket says 172.21.15.6:37856) 2024-03-06T07:32:29.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.349+0000 7fef59400640 1 -- 172.21.15.6:0/3400084978 learned_addr learned my addr 172.21.15.6:0/3400084978 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61e00640 1 --2- 172.21.15.6:0/3400084978 >> v2:172.21.15.178:3300/0 conn(0x7fef641552b0 0x7fef6415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079759029 0 0) 0x7fef6415ed10 con 0x7fef6415e810 2024-03-06T07:32:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef48003520 con 0x7fef6415e810 2024-03-06T07:32:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1619343058 0 0) 0x7fef48003520 con 0x7fef6415e810 2024-03-06T07:32:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 >> v2:172.21.15.178:3300/0 conn(0x7fef641552b0 msgr2=0x7fef6415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 --2- 172.21.15.6:0/3400084978 >> v2:172.21.15.178:3300/0 conn(0x7fef641552b0 0x7fef6415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 >> v1:172.21.15.178:6789/0 conn(0x7fef640cd060 legacy=0x7fef640b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6415f5b0 con 0x7fef6415e810 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fef4c001670 con 0x7fef6415e810 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 >> v1:172.21.15.6:6789/0 conn(0x7fef6415e810 legacy=0x7fef640cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef48003a00 0x7fef48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef48005290 0x7fef4800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6415f2d0 con 0x7fef48004800 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4800e3f0 con 0x7fef48003a00 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4800e5f0 con 0x7fef48005290 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.350+0000 7fef59400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef61e00640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef48003a00 0x7fef48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef59400640 1 -- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef48005290 msgr2=0x7fef4800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef59400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef48005290 0x7fef4800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef59400640 1 -- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef48003a00 msgr2=0x7fef48003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef59400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef48003a00 0x7fef48003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef59400640 1 -- 172.21.15.6:0/3400084978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48003520 con 0x7fef48004800 2024-03-06T07:32:29.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.351+0000 7fef62800640 1 -- 172.21.15.6:0/3400084978 reap_dead start 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.352+0000 7fef59400640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef48004c80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00c190 tx=0x7fef48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dc62020dde2149 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.352+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c007c60 con 0x7fef48004800 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.352+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c030070 con 0x7fef48004800 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.352+0000 7fef61400640 1 -- 172.21.15.6:0/3400084978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c005d50 con 0x7fef48004800 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 -- 172.21.15.6:0/3400084978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 msgr2=0x7fef48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef48004c80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00c190 tx=0x7fef48005830 comp rx=0 tx=0).stop 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 -- 172.21.15.6:0/3400084978 shutdown_connections 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 --2- 172.21.15.6:0/3400084978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef48004c80 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 -- 172.21.15.6:0/3400084978 >> 172.21.15.6:0/3400084978 conn(0x7fef6406b3d0 msgr2=0x7fef6406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 -- 172.21.15.6:0/3400084978 shutdown_connections 2024-03-06T07:32:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.353+0000 7fef63200640 1 -- 172.21.15.6:0/3400084978 wait complete. 2024-03-06T07:32:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.355+0000 7fef63200640 1 Processor -- start 2024-03-06T07:32:29.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.355+0000 7fef63200640 1 -- start start 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef63200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef64151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef63200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef64150dd0 0x7fef64147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef63200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef640b17c0 0x7fef640b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef63200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6406dd30 con 0x7fef48004800 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef63200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef640b3760 con 0x7fef64150dd0 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef63200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6406deb0 con 0x7fef640b17c0 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef61e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef64151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef61e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef64151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60240/0 (socket says 172.21.15.6:60240) 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef61e00640 1 -- 172.21.15.6:0/3776458510 learned_addr learned my addr 172.21.15.6:0/3776458510 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef62800640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef640b17c0 0x7fef640b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.356+0000 7fef59400640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef64150dd0 0x7fef64147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef61e00640 1 -- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef640b17c0 msgr2=0x7fef640b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef61e00640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef640b17c0 0x7fef640b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef61e00640 1 -- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef64150dd0 msgr2=0x7fef64147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef61e00640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef64150dd0 0x7fef64147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef61e00640 1 -- 172.21.15.6:0/3776458510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef641527f0 con 0x7fef48004800 2024-03-06T07:32:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef61e00640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef64151f70 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fef54009cb0 tx=0x7fef54007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54014610 con 0x7fef48004800 2024-03-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54019070 con 0x7fef48004800 2024-03-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef64152ad0 con 0x7fef48004800 2024-03-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54014a10 con 0x7fef48004800 2024-03-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.357+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef64153090 con 0x7fef48004800 2024-03-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.358+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6404f330 con 0x7fef48004800 2024-03-06T07:32:29.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.359+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef54014bb0 con 0x7fef48004800 2024-03-06T07:32:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.361+0000 7fef5aa00640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef4006b500 0x7fef4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.361+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef5408acf0 con 0x7fef48004800 2024-03-06T07:32:29.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.361+0000 7fef59400640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef4006b500 0x7fef4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.362+0000 7fef59400640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef4006b500 0x7fef4006d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fef4c034000 tx=0x7fef4c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.366+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef54059b90 con 0x7fef48004800 2024-03-06T07:32:29.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.567+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef640cf4a0 con 0x7fef48004800 2024-03-06T07:32:29.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.570+0000 7fef5aa00640 1 -- 172.21.15.6:0/3776458510 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16588 (secure 0 0 0) 0x7fef54059490 con 0x7fef48004800 2024-03-06T07:32:29.572 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:29.572 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:16.181620+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:29.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.572+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef4006b500 msgr2=0x7fef4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.572+0000 7fef63200640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef4006b500 0x7fef4006d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fef4c034000 tx=0x7fef4c009030 comp rx=0 tx=0).stop 2024-03-06T07:32:29.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.572+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 msgr2=0x7fef64151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.572+0000 7fef63200640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef64151f70 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fef54009cb0 tx=0x7fef54007700 comp rx=0 tx=0).stop 2024-03-06T07:32:29.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 shutdown_connections 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef4006b500 0x7fef4006d9c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef64150dd0 0x7fef64147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef640b17c0 0x7fef640b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 --2- 172.21.15.6:0/3776458510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef48004800 0x7fef64151f70 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 >> 172.21.15.6:0/3776458510 conn(0x7fef6406b3d0 msgr2=0x7fef64071600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 shutdown_connections 2024-03-06T07:32:29.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.573+0000 7fef63200640 1 -- 172.21.15.6:0/3776458510 wait complete. 2024-03-06T07:32:29.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:12.534039+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:13.644895+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:29.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.811+0000 7fababe00640 1 Processor -- start 2024-03-06T07:32:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.811+0000 7fababe00640 1 -- start start 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fababe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b3c20 0x7fabac0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fababe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fabac0b4000 0x7fabac15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fababe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabac071930 con 0x7fabac0b1b40 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fababe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabac0b4460 con 0x7fabac0b4000 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fababe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac071c10 con 0x7fabac0b3c20 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fabaa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fabac0b4000 0x7fabac15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fabaa000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fabac0b4000 0x7fabac15b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60246/0 (socket says 172.21.15.6:60246) 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.813+0000 7fabaa000640 1 -- 172.21.15.6:0/3104659197 learned_addr learned my addr 172.21.15.6:0/3104659197 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.814+0000 7fabaaa00640 1 --2- 172.21.15.6:0/3104659197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b3c20 0x7fabac0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.814+0000 7fabaa000640 1 -- 172.21.15.6:0/3104659197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b3c20 msgr2=0x7fabac0729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.814+0000 7fabaa000640 1 --2- 172.21.15.6:0/3104659197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b3c20 0x7fabac0729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.814+0000 7fabaa000640 1 -- 172.21.15.6:0/3104659197 >> v1:172.21.15.178:6789/0 conn(0x7fabac0b1b40 legacy=0x7fabac15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.814+0000 7fabaa000640 1 -- 172.21.15.6:0/3104659197 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac071ef0 con 0x7fabac0b4000 2024-03-06T07:32:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.815+0000 7fabaa000640 1 --2- 172.21.15.6:0/3104659197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b4000 0x7fabac15b8a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fab98009870 tx=0x7fab9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3494c05951e0c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.815+0000 7faba9600640 1 -- 172.21.15.6:0/3104659197 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98035020 con 0x7fabac0b4000 2024-03-06T07:32:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7faba9600640 1 -- 172.21.15.6:0/3104659197 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab98031640 con 0x7fabac0b4000 2024-03-06T07:32:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7faba9600640 1 -- 172.21.15.6:0/3104659197 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9802d040 con 0x7fabac0b4000 2024-03-06T07:32:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 -- 172.21.15.6:0/3104659197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b4000 msgr2=0x7fabac15b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 --2- 172.21.15.6:0/3104659197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b4000 0x7fabac15b8a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fab98009870 tx=0x7fab9802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 -- 172.21.15.6:0/3104659197 shutdown_connections 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 --2- 172.21.15.6:0/3104659197 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b4000 0x7fabac15b8a0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 --2- 172.21.15.6:0/3104659197 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b3c20 0x7fabac0729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 -- 172.21.15.6:0/3104659197 >> 172.21.15.6:0/3104659197 conn(0x7fabac06b3a0 msgr2=0x7fabac06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 -- 172.21.15.6:0/3104659197 shutdown_connections 2024-03-06T07:32:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.816+0000 7fababe00640 1 -- 172.21.15.6:0/3104659197 wait complete. 2024-03-06T07:32:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.818+0000 7fababe00640 1 Processor -- start 2024-03-06T07:32:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.818+0000 7fababe00640 1 -- start start 2024-03-06T07:32:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fababe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b1b40 0x7fabac146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fababe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 0x7fabac14ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fababe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fabac0b4000 0x7fabac06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fababe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac15bde0 con 0x7fabac0b3c20 2024-03-06T07:32:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fababe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabac0724b0 con 0x7fabac0b4000 2024-03-06T07:32:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fababe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabac15bf60 con 0x7fabac0b1b40 2024-03-06T07:32:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fabaa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 0x7fabac14ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fabaa000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 0x7fabac14ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60252/0 (socket says 172.21.15.6:60252) 2024-03-06T07:32:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.819+0000 7fabaa000640 1 -- 172.21.15.6:0/1074740602 learned_addr learned my addr 172.21.15.6:0/1074740602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fabaa000640 1 -- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b1b40 msgr2=0x7fabac146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fabaa000640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b1b40 0x7fabac146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fabaa000640 1 -- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fabac0b4000 msgr2=0x7fabac06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fabaa000640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fabac0b4000 0x7fabac06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fabaa000640 1 -- 172.21.15.6:0/1074740602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabac146740 con 0x7fabac0b3c20 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fabaa000640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 0x7fabac14ca30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fab9802b930 tx=0x7fab980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98040070 con 0x7fabac0b3c20 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab980399f0 con 0x7fabac0b3c20 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98035040 con 0x7fabac0b3c20 2024-03-06T07:32:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.820+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabac14d330 con 0x7fabac0b3c20 2024-03-06T07:32:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.821+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabac0cce00 con 0x7fabac0b3c20 2024-03-06T07:32:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.822+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabac04f330 con 0x7fabac0b3c20 2024-03-06T07:32:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.823+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab9804b060 con 0x7fabac0b3c20 2024-03-06T07:32:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.823+0000 7faba2a00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fab8c06b4d0 0x7fab8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.824+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab980abb90 con 0x7fabac0b3c20 2024-03-06T07:32:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.824+0000 7fabaaa00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fab8c06b4d0 0x7fab8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.825+0000 7fabaaa00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fab8c06b4d0 0x7fab8c06d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0098f0 tx=0x7fab9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:29.828+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab9807a970 con 0x7fabac0b3c20 2024-03-06T07:32:30.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.026+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabac070c00 con 0x7fabac0b3c20 2024-03-06T07:32:30.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.027+0000 7faba2a00640 1 -- 172.21.15.6:0/1074740602 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab9807a270 con 0x7fabac0b3c20 2024-03-06T07:32:30.029 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:32:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fab8c06b4d0 msgr2=0x7fab8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fab8c06b4d0 0x7fab8c06d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fab9c0098f0 tx=0x7fab9c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 msgr2=0x7fabac14ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 0x7fabac14ca30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fab9802b930 tx=0x7fab980046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 shutdown_connections 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fab8c06b4d0 0x7fab8c06d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fabac0b4000 0x7fabac06d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabac0b3c20 0x7fabac14ca30 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 --2- 172.21.15.6:0/1074740602 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fabac0b1b40 0x7fabac146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 >> 172.21.15.6:0/1074740602 conn(0x7fabac06b3a0 msgr2=0x7fabac14dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 shutdown_connections 2024-03-06T07:32:30.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.030+0000 7fababe00640 1 -- 172.21.15.6:0/1074740602 wait complete. 2024-03-06T07:32:30.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.281+0000 7fdcde000640 1 Processor -- start 2024-03-06T07:32:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.281+0000 7fdcde000640 1 -- start start 2024-03-06T07:32:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.282+0000 7fdcde000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcd8144560 0x7fdcd806e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.282+0000 7fdcde000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd8161d20 con 0x7fdcd806b6c0 2024-03-06T07:32:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.282+0000 7fdcde000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd8162000 con 0x7fdcd8147510 2024-03-06T07:32:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.282+0000 7fdcde000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd8162ae0 con 0x7fdcd8144560 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcd8144560 0x7fdcd806e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdcd8144560 0x7fdcd806e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60262/0 (socket says 172.21.15.6:60262) 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7400640 1 -- 172.21.15.6:0/590560073 learned_addr learned my addr 172.21.15.6:0/590560073 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7e00640 1 --1- 172.21.15.6:0/590560073 >> v1:172.21.15.6:6789/0 conn(0x7fdcd806b6c0 0x7fdcd8071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37872/0 (socket says 172.21.15.6:37872) 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7400640 1 -- 172.21.15.6:0/590560073 >> v1:172.21.15.178:6789/0 conn(0x7fdcd8147510 legacy=0x7fdcd806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7400640 1 -- 172.21.15.6:0/590560073 >> v1:172.21.15.6:6789/0 conn(0x7fdcd806b6c0 legacy=0x7fdcd8071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd7400640 1 -- 172.21.15.6:0/590560073 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd81622e0 con 0x7fdcd8144560 2024-03-06T07:32:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.283+0000 7fdcd6000640 1 -- 172.21.15.6:0/590560073 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557901496 0 0) 0x7fdcd8161d20 con 0x7fdcd806b6c0 2024-03-06T07:32:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.285+0000 7fdcd7400640 1 --2- 172.21.15.6:0/590560073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd8144560 0x7fdcd806e680 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009870 tx=0x7fdcc802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b550b6fb36fa5e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.285+0000 7fdcd6000640 1 -- 172.21.15.6:0/590560073 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8034070 con 0x7fdcd8144560 2024-03-06T07:32:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.285+0000 7fdcd6000640 1 -- 172.21.15.6:0/590560073 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8030860 con 0x7fdcd8144560 2024-03-06T07:32:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.285+0000 7fdcd6000640 1 -- 172.21.15.6:0/590560073 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8030c30 con 0x7fdcd8144560 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.285+0000 7fdcde000640 1 -- 172.21.15.6:0/590560073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd8144560 msgr2=0x7fdcd806e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.285+0000 7fdcde000640 1 --2- 172.21.15.6:0/590560073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd8144560 0x7fdcd806e680 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009870 tx=0x7fdcc802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.286+0000 7fdcde000640 1 -- 172.21.15.6:0/590560073 shutdown_connections 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.286+0000 7fdcde000640 1 --2- 172.21.15.6:0/590560073 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd8144560 0x7fdcd806e680 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.286+0000 7fdcde000640 1 -- 172.21.15.6:0/590560073 >> 172.21.15.6:0/590560073 conn(0x7fdcd80b2860 msgr2=0x7fdcd80b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.286+0000 7fdcde000640 1 -- 172.21.15.6:0/590560073 shutdown_connections 2024-03-06T07:32:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.286+0000 7fdcde000640 1 -- 172.21.15.6:0/590560073 wait complete. 2024-03-06T07:32:30.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.287+0000 7fdcde000640 1 Processor -- start 2024-03-06T07:32:30.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.287+0000 7fdcde000640 1 -- start start 2024-03-06T07:32:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.288+0000 7fdcde000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 0x7fdcd815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.288+0000 7fdcde000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcd8144560 0x7fdcd815f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.288+0000 7fdcde000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcd8147510 0x7fdcd8072380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.288+0000 7fdcde000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806ebc0 con 0x7fdcd806b6c0 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.288+0000 7fdcde000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806efd0 con 0x7fdcd8147510 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.288+0000 7fdcde000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806ed40 con 0x7fdcd8144560 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 0x7fdcd815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 0x7fdcd815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60276/0 (socket says 172.21.15.6:60276) 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 -- 172.21.15.6:0/3652988683 learned_addr learned my addr 172.21.15.6:0/3652988683 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd6a00640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcd8144560 0x7fdcd815f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7e00640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcd8147510 0x7fdcd8072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 -- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcd8144560 msgr2=0x7fdcd815f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcd8144560 0x7fdcd815f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 -- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcd8147510 msgr2=0x7fdcd8072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcd8147510 0x7fdcd8072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.289+0000 7fdcd7400640 1 -- 172.21.15.6:0/3652988683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd8144d90 con 0x7fdcd806b6c0 2024-03-06T07:32:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.290+0000 7fdcd7400640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 0x7fdcd815dad0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009870 tx=0x7fdcc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.290+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8048070 con 0x7fdcd806b6c0 2024-03-06T07:32:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.290+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8039d90 con 0x7fdcd806b6c0 2024-03-06T07:32:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.290+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8034040 con 0x7fdcd806b6c0 2024-03-06T07:32:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.290+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd8145070 con 0x7fdcd806b6c0 2024-03-06T07:32:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.290+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd8145630 con 0x7fdcd806b6c0 2024-03-06T07:32:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.292+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd804f330 con 0x7fdcd806b6c0 2024-03-06T07:32:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.292+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcc8037070 con 0x7fdcd806b6c0 2024-03-06T07:32:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.293+0000 7fdccf400640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdcb806b500 0x7fdcb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.293+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcc80ac050 con 0x7fdcd806b6c0 2024-03-06T07:32:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.294+0000 7fdcd6a00640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdcb806b500 0x7fdcb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.295+0000 7fdcd6a00640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdcb806b500 0x7fdcb806d9c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdcd81460c0 tx=0x7fdcc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.298+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcc807af60 con 0x7fdcd806b6c0 2024-03-06T07:32:30.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.499+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdcd8147bc0 con 0x7fdcd806b6c0 2024-03-06T07:32:30.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.500+0000 7fdccf400640 1 -- 172.21.15.6:0/3652988683 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdcc807a860 con 0x7fdcd806b6c0 2024-03-06T07:32:30.502 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.502+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdcb806b500 msgr2=0x7fdcb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.502+0000 7fdcde000640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdcb806b500 0x7fdcb806d9c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdcd81460c0 tx=0x7fdcc4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.502+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 msgr2=0x7fdcd815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.502+0000 7fdcde000640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 0x7fdcd815dad0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009870 tx=0x7fdcc8004730 comp rx=0 tx=0).stop 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 shutdown_connections 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdcb806b500 0x7fdcb806d9c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdcd8147510 0x7fdcd8072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdcd8144560 0x7fdcd815f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 --2- 172.21.15.6:0/3652988683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdcd806b6c0 0x7fdcd815dad0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 >> 172.21.15.6:0/3652988683 conn(0x7fdcd80b2860 msgr2=0x7fdcd80cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 shutdown_connections 2024-03-06T07:32:30.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.503+0000 7fdcde000640 1 -- 172.21.15.6:0/3652988683 wait complete. 2024-03-06T07:32:30.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:30.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.747+0000 7fa076400640 1 Processor -- start 2024-03-06T07:32:30.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.747+0000 7fa076400640 1 -- start start 2024-03-06T07:32:30.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.749+0000 7fa076400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa07015f810 0x7fa0700b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.749+0000 7fa076400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0700b2080 0x7fa0700b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.749+0000 7fa076400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07015fe40 con 0x7fa070145430 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.749+0000 7fa076400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0700b24d0 con 0x7fa0700b2080 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.749+0000 7fa076400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa070160120 con 0x7fa07015f810 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0700b2080 0x7fa0700b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa06fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa07015f810 0x7fa0700b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa0700b2080 0x7fa0700b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60282/0 (socket says 172.21.15.6:60282) 2024-03-06T07:32:30.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 -- 172.21.15.6:0/3252202008 learned_addr learned my addr 172.21.15.6:0/3252202008 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:30.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 -- 172.21.15.6:0/3252202008 >> v2:172.21.15.178:3300/0 conn(0x7fa07015f810 msgr2=0x7fa0700b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 --2- 172.21.15.6:0/3252202008 >> v2:172.21.15.178:3300/0 conn(0x7fa07015f810 0x7fa0700b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 -- 172.21.15.6:0/3252202008 >> v1:172.21.15.178:6789/0 conn(0x7fa070145430 legacy=0x7fa07015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.750+0000 7fa075a00640 1 -- 172.21.15.6:0/3252202008 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa070160400 con 0x7fa0700b2080 2024-03-06T07:32:30.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.751+0000 7fa075a00640 1 --2- 172.21.15.6:0/3252202008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0700b2080 0x7fa0700b2fd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa07006d8c0 tx=0x7fa06802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872e224f2ee6ae0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa06f400640 1 -- 172.21.15.6:0/3252202008 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06802cb20 con 0x7fa0700b2080 2024-03-06T07:32:30.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa06f400640 1 -- 172.21.15.6:0/3252202008 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06802ccc0 con 0x7fa0700b2080 2024-03-06T07:32:30.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa06f400640 1 -- 172.21.15.6:0/3252202008 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068034610 con 0x7fa0700b2080 2024-03-06T07:32:30.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 -- 172.21.15.6:0/3252202008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0700b2080 msgr2=0x7fa0700b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 --2- 172.21.15.6:0/3252202008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0700b2080 0x7fa0700b2fd0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fa07006d8c0 tx=0x7fa06802bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:30.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 -- 172.21.15.6:0/3252202008 shutdown_connections 2024-03-06T07:32:30.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 --2- 172.21.15.6:0/3252202008 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa0700b2080 0x7fa0700b2fd0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 --2- 172.21.15.6:0/3252202008 >> v2:172.21.15.178:3300/0 conn(0x7fa07015f810 0x7fa0700b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 -- 172.21.15.6:0/3252202008 >> 172.21.15.6:0/3252202008 conn(0x7fa07006b3a0 msgr2=0x7fa07006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.752+0000 7fa076400640 1 -- 172.21.15.6:0/3252202008 shutdown_connections 2024-03-06T07:32:30.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.753+0000 7fa076400640 1 -- 172.21.15.6:0/3252202008 wait complete. 2024-03-06T07:32:30.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.754+0000 7fa076400640 1 Processor -- start 2024-03-06T07:32:30.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.754+0000 7fa076400640 1 -- start start 2024-03-06T07:32:30.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.755+0000 7fa076400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0700b2080 0x7fa070070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.755+0000 7fa076400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa070145430 0x7fa07013e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.755+0000 7fa076400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 0x7fa0700ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.755+0000 7fa076400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070145990 con 0x7fa07015f810 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.755+0000 7fa076400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070145810 con 0x7fa0700b2080 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.755+0000 7fa076400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700cde00 con 0x7fa070145430 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 0x7fa0700ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 0x7fa0700ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60296/0 (socket says 172.21.15.6:60296) 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 -- 172.21.15.6:0/3884662154 learned_addr learned my addr 172.21.15.6:0/3884662154 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075000640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0700b2080 0x7fa070070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa06fe00640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa070145430 0x7fa07013e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 -- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa070145430 msgr2=0x7fa07013e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa070145430 0x7fa07013e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 -- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0700b2080 msgr2=0x7fa070070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0700b2080 0x7fa070070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 -- 172.21.15.6:0/3884662154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0700ae960 con 0x7fa07015f810 2024-03-06T07:32:30.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa06fe00640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa070145430 0x7fa07013e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:30.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.756+0000 7fa075a00640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 0x7fa0700ae420 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa068009f40 tx=0x7fa068004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.757+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068043020 con 0x7fa07015f810 2024-03-06T07:32:30.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.757+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06803e450 con 0x7fa07015f810 2024-03-06T07:32:30.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.757+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0700721b0 con 0x7fa07015f810 2024-03-06T07:32:30.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.757+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa068009510 con 0x7fa07015f810 2024-03-06T07:32:30.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.757+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0700713f0 con 0x7fa07015f810 2024-03-06T07:32:30.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.758+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07004f330 con 0x7fa07015f810 2024-03-06T07:32:30.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.759+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa068046020 con 0x7fa07015f810 2024-03-06T07:32:30.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.760+0000 7fa06cc00640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa05006b4d0 0x7fa05006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.760+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0680ac9a0 con 0x7fa07015f810 2024-03-06T07:32:30.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.760+0000 7fa075000640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa05006b4d0 0x7fa05006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.762+0000 7fa075000640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa05006b4d0 0x7fa05006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa064001640 tx=0x7fa064009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.764+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa06804d030 con 0x7fa07015f810 2024-03-06T07:32:30.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.964+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa07006e240 con 0x7fa07015f810 2024-03-06T07:32:30.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.965+0000 7fa06cc00640 1 -- 172.21.15.6:0/3884662154 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa06807b810 con 0x7fa07015f810 2024-03-06T07:32:30.967 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:32:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa05006b4d0 msgr2=0x7fa05006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa05006b4d0 0x7fa05006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa064001640 tx=0x7fa064009040 comp rx=0 tx=0).stop 2024-03-06T07:32:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 msgr2=0x7fa0700ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 0x7fa0700ae420 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa068009f40 tx=0x7fa068004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 shutdown_connections 2024-03-06T07:32:30.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa05006b4d0 0x7fa05006d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa07015f810 0x7fa0700ae420 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa070145430 0x7fa07013e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 --2- 172.21.15.6:0/3884662154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa0700b2080 0x7fa070070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 >> 172.21.15.6:0/3884662154 conn(0x7fa07006b3a0 msgr2=0x7fa07015d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 shutdown_connections 2024-03-06T07:32:30.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:30.968+0000 7fa076400640 1 -- 172.21.15.6:0/3884662154 wait complete. 2024-03-06T07:32:30.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:31.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.213+0000 7fe556000640 1 Processor -- start 2024-03-06T07:32:31.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.214+0000 7fe556000640 1 -- start start 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.215+0000 7fe556000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe5501477e0 0x7fe550162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.215+0000 7fe556000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe55006ef20 0x7fe55006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.215+0000 7fe556000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe550147bc0 con 0x7fe55006b4f0 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.215+0000 7fe556000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe550161f90 con 0x7fe55006ef20 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.215+0000 7fe556000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe550162960 con 0x7fe5501477e0 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe55006ef20 0x7fe55006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe55006ef20 0x7fe55006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60312/0 (socket says 172.21.15.6:60312) 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 -- 172.21.15.6:0/520360428 learned_addr learned my addr 172.21.15.6:0/520360428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe554c00640 1 --2- 172.21.15.6:0/520360428 >> v2:172.21.15.178:3300/0 conn(0x7fe5501477e0 0x7fe550162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 -- 172.21.15.6:0/520360428 >> v2:172.21.15.178:3300/0 conn(0x7fe5501477e0 msgr2=0x7fe550162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 --2- 172.21.15.6:0/520360428 >> v2:172.21.15.178:3300/0 conn(0x7fe5501477e0 0x7fe550162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 -- 172.21.15.6:0/520360428 >> v1:172.21.15.178:6789/0 conn(0x7fe55006b4f0 legacy=0x7fe5500d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.216+0000 7fe54fe00640 1 -- 172.21.15.6:0/520360428 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5500b3d70 con 0x7fe55006ef20 2024-03-06T07:32:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.218+0000 7fe54fe00640 1 --2- 172.21.15.6:0/520360428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006ef20 0x7fe55006afb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe5500b6030 tx=0x7fe53c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597d3bac73bde7c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.218+0000 7fe54f400640 1 -- 172.21.15.6:0/520360428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0317c0 con 0x7fe55006ef20 2024-03-06T07:32:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.218+0000 7fe54f400640 1 -- 172.21.15.6:0/520360428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c037070 con 0x7fe55006ef20 2024-03-06T07:32:31.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.218+0000 7fe54f400640 1 -- 172.21.15.6:0/520360428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c031b80 con 0x7fe55006ef20 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 -- 172.21.15.6:0/520360428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006ef20 msgr2=0x7fe55006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 --2- 172.21.15.6:0/520360428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006ef20 0x7fe55006afb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe5500b6030 tx=0x7fe53c02b860 comp rx=0 tx=0).stop 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 -- 172.21.15.6:0/520360428 shutdown_connections 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 --2- 172.21.15.6:0/520360428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006ef20 0x7fe55006afb0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 --2- 172.21.15.6:0/520360428 >> v2:172.21.15.178:3300/0 conn(0x7fe5501477e0 0x7fe550162420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 -- 172.21.15.6:0/520360428 >> 172.21.15.6:0/520360428 conn(0x7fe55006c0c0 msgr2=0x7fe55006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 -- 172.21.15.6:0/520360428 shutdown_connections 2024-03-06T07:32:31.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.219+0000 7fe556000640 1 -- 172.21.15.6:0/520360428 wait complete. 2024-03-06T07:32:31.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.221+0000 7fe556000640 1 Processor -- start 2024-03-06T07:32:31.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.221+0000 7fe556000640 1 -- start start 2024-03-06T07:32:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe556000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 0x7fe5500d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe556000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe55006ef20 0x7fe550138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe556000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5501477e0 0x7fe550139130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe556000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5500b4330 con 0x7fe55006b4f0 2024-03-06T07:32:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe556000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55006b920 con 0x7fe5501477e0 2024-03-06T07:32:31.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe556000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe550161c20 con 0x7fe55006ef20 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 0x7fe5500d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 0x7fe5500d09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60328/0 (socket says 172.21.15.6:60328) 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 -- 172.21.15.6:0/597023829 learned_addr learned my addr 172.21.15.6:0/597023829 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe54fe00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe55006ef20 0x7fe550138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 -- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe55006ef20 msgr2=0x7fe550138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe55006ef20 0x7fe550138bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 -- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5501477e0 msgr2=0x7fe550139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5501477e0 0x7fe550139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.222+0000 7fe554c00640 1 -- 172.21.15.6:0/597023829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5501336e0 con 0x7fe55006b4f0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.223+0000 7fe554c00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 0x7fe5500d09a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe54400d800 tx=0x7fe54400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.223+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400b6c0 con 0x7fe55006b4f0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.223+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544019070 con 0x7fe55006b4f0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.223+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54400bac0 con 0x7fe55006b4f0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.223+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5501ea110 con 0x7fe55006b4f0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.223+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe550133c60 con 0x7fe55006b4f0 2024-03-06T07:32:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.224+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5500cce70 con 0x7fe55006b4f0 2024-03-06T07:32:31.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.225+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe54400bc60 con 0x7fe55006b4f0 2024-03-06T07:32:31.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.227+0000 7fe54cc00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe53006b500 0x7fe53006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.227+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe54408a730 con 0x7fe55006b4f0 2024-03-06T07:32:31.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.227+0000 7fe54fe00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe53006b500 0x7fe53006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.228+0000 7fe54fe00640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe53006b500 0x7fe53006d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe53c036000 tx=0x7fe53c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.231+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5500cce70 con 0x7fe55006b4f0 2024-03-06T07:32:31.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.471+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe55004f330 con 0x7fe55006b4f0 2024-03-06T07:32:31.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.472+0000 7fe54cc00640 1 -- 172.21.15.6:0/597023829 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7fe544059510 con 0x7fe55006b4f0 2024-03-06T07:32:31.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:31.474 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:31.474 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.474+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe53006b500 msgr2=0x7fe53006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.474+0000 7fe556000640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe53006b500 0x7fe53006d9c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fe53c036000 tx=0x7fe53c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.474+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 msgr2=0x7fe5500d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.474+0000 7fe556000640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 0x7fe5500d09a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fe54400d800 tx=0x7fe54400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 shutdown_connections 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe53006b500 0x7fe53006d9c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5501477e0 0x7fe550139130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe55006ef20 0x7fe550138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 --2- 172.21.15.6:0/597023829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe55006b4f0 0x7fe5500d09a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 >> 172.21.15.6:0/597023829 conn(0x7fe55006c0c0 msgr2=0x7fe5500b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 shutdown_connections 2024-03-06T07:32:31.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.475+0000 7fe556000640 1 -- 172.21.15.6:0/597023829 wait complete. 2024-03-06T07:32:31.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.713+0000 7f3d93e00640 1 Processor -- start 2024-03-06T07:32:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.714+0000 7f3d93e00640 1 -- start start 2024-03-06T07:32:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d93e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 0x7f3d940b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d93e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d940d2dc0 con 0x7f3d9415e070 2024-03-06T07:32:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d93e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d940d18c0 con 0x7f3d940d2960 2024-03-06T07:32:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d93e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940d1ba0 con 0x7f3d940d0cd0 2024-03-06T07:32:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d92a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3d9415e070 0x7f3d940cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37882/0 (socket says 172.21.15.6:37882) 2024-03-06T07:32:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d92a00640 1 -- 172.21.15.6:0/2600849026 learned_addr learned my addr 172.21.15.6:0/2600849026 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.715+0000 7f3d93400640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 0x7f3d940b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713255970 0 0) 0x7f3d940d2dc0 con 0x7f3d9415e070 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d78003520 con 0x7f3d9415e070 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 74178003 0 0) 0x7f3d78003520 con 0x7f3d9415e070 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 msgr2=0x7f3d940b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 0x7f3d940b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 >> v1:172.21.15.178:6789/0 conn(0x7f3d940d2960 legacy=0x7f3d940d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d940d1e80 con 0x7f3d9415e070 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f3d7c001670 con 0x7f3d9415e070 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 >> v1:172.21.15.6:6789/0 conn(0x7f3d9415e070 legacy=0x7f3d940cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 0x7f3d78005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940d1ba0 con 0x7f3d780039e0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7800bb80 con 0x7f3d78004800 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7800bd80 con 0x7f3d940d0cd0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.716+0000 7f3d92a00640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d93400640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 0x7f3d78005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92a00640 1 -- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 msgr2=0x7f3d78005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92a00640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940d0cd0 0x7f3d78005250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92a00640 1 -- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d78004800 msgr2=0x7f3d78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92000640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92a00640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92a00640 1 -- 172.21.15.6:0/2600849026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78003520 con 0x7f3d780039e0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.717+0000 7f3d92000640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d78004800 0x7f3d78004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.718+0000 7f3d92a00640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00c0e0 tx=0x7f3d78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ce78263d2a53c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.718+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c007c20 con 0x7f3d780039e0 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.718+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c030070 con 0x7f3d780039e0 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.718+0000 7f3d91600640 1 -- 172.21.15.6:0/2600849026 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c005d30 con 0x7f3d780039e0 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93e00640 1 -- 172.21.15.6:0/2600849026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d780039e0 msgr2=0x7f3d78003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93e00640 1 --2- 172.21.15.6:0/2600849026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d780039e0 0x7f3d78003e00 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00c0e0 tx=0x7f3d78005830 comp rx=0 tx=0).stop 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93400640 1 -- 172.21.15.6:0/2600849026 reap_dead start 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93e00640 1 -- 172.21.15.6:0/2600849026 shutdown_connections 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93e00640 1 -- 172.21.15.6:0/2600849026 >> 172.21.15.6:0/2600849026 conn(0x7f3d9408d2a0 msgr2=0x7f3d940d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93e00640 1 -- 172.21.15.6:0/2600849026 shutdown_connections 2024-03-06T07:32:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.719+0000 7f3d93e00640 1 -- 172.21.15.6:0/2600849026 wait complete. 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.721+0000 7f3d93e00640 1 Processor -- start 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.721+0000 7f3d93e00640 1 -- start start 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d780039e0 0x7f3d94156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940c8f10 0x7f3d940b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 0x7f3d941600f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940d45b0 con 0x7f3d9415fc70 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940d2440 con 0x7f3d780039e0 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940d4730 con 0x7f3d940c8f10 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.722+0000 7f3d93400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 0x7f3d941600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 0x7f3d941600f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60344/0 (socket says 172.21.15.6:60344) 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 -- 172.21.15.6:0/4208444778 learned_addr learned my addr 172.21.15.6:0/4208444778 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d92a00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d780039e0 0x7f3d94156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d92000640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940c8f10 0x7f3d940b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 -- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940c8f10 msgr2=0x7f3d940b92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940c8f10 0x7f3d940b92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 -- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d780039e0 msgr2=0x7f3d94156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d780039e0 0x7f3d94156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 -- 172.21.15.6:0/4208444778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d940c9530 con 0x7f3d9415fc70 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.723+0000 7f3d93400640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 0x7f3d941600f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0098c0 tx=0x7f3d8c009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.724+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c019070 con 0x7f3d9415fc70 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.724+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c007d40 con 0x7f3d9415fc70 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.724+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d940b76f0 con 0x7f3d9415fc70 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.724+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8c014640 con 0x7f3d9415fc70 2024-03-06T07:32:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.724+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d940b7cb0 con 0x7f3d9415fc70 2024-03-06T07:32:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.725+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9404f330 con 0x7f3d9415fc70 2024-03-06T07:32:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.726+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d8c01c070 con 0x7f3d9415fc70 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.727+0000 7f3d8aa00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d7006b4d0 0x7f3d7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.727+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d8c089c70 con 0x7f3d9415fc70 2024-03-06T07:32:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.728+0000 7f3d92a00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d7006b4d0 0x7f3d7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.729+0000 7f3d92a00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d7006b4d0 0x7f3d7006d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c034000 tx=0x7f3d7c007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.732+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d8c058a50 con 0x7f3d9415fc70 2024-03-06T07:32:31.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.933+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d940b5340 con 0x7f3d9415fc70 2024-03-06T07:32:31.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.936+0000 7f3d8aa00640 1 -- 172.21.15.6:0/4208444778 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f3d8c058350 con 0x7f3d9415fc70 2024-03-06T07:32:31.939 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:31.939 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:16.181620+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:31.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d7006b4d0 msgr2=0x7f3d7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d7006b4d0 0x7f3d7006d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c034000 tx=0x7f3d7c007000 comp rx=0 tx=0).stop 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 msgr2=0x7f3d941600f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 0x7f3d941600f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c0098c0 tx=0x7f3d8c009d80 comp rx=0 tx=0).stop 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 shutdown_connections 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d7006b4d0 0x7f3d7006d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d9415fc70 0x7f3d941600f0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d940c8f10 0x7f3d940b92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 --2- 172.21.15.6:0/4208444778 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d780039e0 0x7f3d94156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 >> 172.21.15.6:0/4208444778 conn(0x7f3d9408d2a0 msgr2=0x7f3d940b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 shutdown_connections 2024-03-06T07:32:31.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:31.939+0000 7f3d93e00640 1 -- 172.21.15.6:0/4208444778 wait complete. 2024-03-06T07:32:31.961 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:31.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:32.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.180+0000 7f0e0a800640 1 Processor -- start 2024-03-06T07:32:32.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.181+0000 7f0e0a800640 1 -- start start 2024-03-06T07:32:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.182+0000 7f0e0a800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 0x7f0e040d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.182+0000 7f0e0a800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e040b3710 con 0x7f0e0415e0b0 2024-03-06T07:32:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.182+0000 7f0e0a800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e040b39f0 con 0x7f0e040d0c80 2024-03-06T07:32:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.182+0000 7f0e0a800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040b09e0 con 0x7f0e040d4a10 2024-03-06T07:32:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.183+0000 7f0e09400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0e0415e0b0 0x7f0e040cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33620/0 (socket says 172.21.15.6:33620) 2024-03-06T07:32:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.183+0000 7f0e09400640 1 -- 172.21.15.6:0/243206495 learned_addr learned my addr 172.21.15.6:0/243206495 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.183+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915938836 0 0) 0x7f0e040b3710 con 0x7f0e0415e0b0 2024-03-06T07:32:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.183+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dec003520 con 0x7f0e0415e0b0 2024-03-06T07:32:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.183+0000 7f0e08a00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 0x7f0e040d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 253431998 0 0) 0x7f0dec003520 con 0x7f0e0415e0b0 2024-03-06T07:32:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 msgr2=0x7f0e040d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 0x7f0e040d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 >> v1:172.21.15.178:6789/0 conn(0x7f0e040d0c80 legacy=0x7f0e040b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e040b3cd0 con 0x7f0e0415e0b0 2024-03-06T07:32:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0df8001670 con 0x7f0e0415e0b0 2024-03-06T07:32:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 >> v1:172.21.15.6:6789/0 conn(0x7f0e0415e0b0 legacy=0x7f0e040cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 0x7f0dec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0dec003f60 0x7f0dec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec004d40 0x7f0dec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec00ba70 con 0x7f0dec004d40 2024-03-06T07:32:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec00bc20 con 0x7f0dec003f60 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.184+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dec00be20 con 0x7f0e040d4a10 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec004d40 0x7f0dec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e08a00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0dec003f60 0x7f0dec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09400640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 0x7f0dec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09e00640 1 -- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 msgr2=0x7f0dec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040d4a10 0x7f0dec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09e00640 1 -- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0dec003f60 msgr2=0x7f0dec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0dec003f60 0x7f0dec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e09e00640 1 -- 172.21.15.6:0/243206495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dec003520 con 0x7f0dec004d40 2024-03-06T07:32:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.185+0000 7f0e08a00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0dec003f60 0x7f0dec004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.186+0000 7f0e09e00640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec004d40 0x7f0dec0051c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc001e50 tx=0x7f0dec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950784686cf0241e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.186+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc001a70 con 0x7f0dec004d40 2024-03-06T07:32:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc003b70 con 0x7f0dec004d40 2024-03-06T07:32:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e03e00640 1 -- 172.21.15.6:0/243206495 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0050d0 con 0x7f0dec004d40 2024-03-06T07:32:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e0a800640 1 -- 172.21.15.6:0/243206495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec004d40 msgr2=0x7f0dec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e0a800640 1 --2- 172.21.15.6:0/243206495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0dec004d40 0x7f0dec0051c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc001e50 tx=0x7f0dec005830 comp rx=0 tx=0).stop 2024-03-06T07:32:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e09e00640 1 -- 172.21.15.6:0/243206495 reap_dead start 2024-03-06T07:32:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e0a800640 1 -- 172.21.15.6:0/243206495 shutdown_connections 2024-03-06T07:32:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e0a800640 1 -- 172.21.15.6:0/243206495 >> 172.21.15.6:0/243206495 conn(0x7f0e040d3d30 msgr2=0x7f0e040d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e0a800640 1 -- 172.21.15.6:0/243206495 shutdown_connections 2024-03-06T07:32:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.187+0000 7f0e0a800640 1 -- 172.21.15.6:0/243206495 wait complete. 2024-03-06T07:32:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.189+0000 7f0e0a800640 1 Processor -- start 2024-03-06T07:32:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.189+0000 7f0e0a800640 1 -- start start 2024-03-06T07:32:32.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e0a800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e04157300 0x7f0e040cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e0a800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040cda30 0x7f0e040b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e0a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 0x7f0e040b6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e0a800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0415e490 con 0x7f0e040b8eb0 2024-03-06T07:32:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e0a800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040cfa20 con 0x7f0e04157300 2024-03-06T07:32:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e0a800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040cfc60 con 0x7f0e040cda30 2024-03-06T07:32:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.190+0000 7f0e09e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 0x7f0e040b6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 0x7f0e040b6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33680/0 (socket says 172.21.15.6:33680) 2024-03-06T07:32:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 -- 172.21.15.6:0/507607430 learned_addr learned my addr 172.21.15.6:0/507607430 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e08a00640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040cda30 0x7f0e040b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09400640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e04157300 0x7f0e040cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 -- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040cda30 msgr2=0x7f0e040b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040cda30 0x7f0e040b8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 -- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e04157300 msgr2=0x7f0e040cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e04157300 0x7f0e040cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 -- 172.21.15.6:0/507607430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e040cc7b0 con 0x7f0e040b8eb0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.191+0000 7f0e09e00640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 0x7f0e040b6f70 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc001f70 tx=0x7f0dfc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.192+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc001a70 con 0x7f0e040b8eb0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.192+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc003970 con 0x7f0e040b8eb0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.192+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc007db0 con 0x7f0e040b8eb0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.192+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e040cdeb0 con 0x7f0e040b8eb0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.192+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e040cc1f0 con 0x7f0e040b8eb0 2024-03-06T07:32:32.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.193+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0404f330 con 0x7f0e040b8eb0 2024-03-06T07:32:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.194+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0dfc041030 con 0x7f0e040b8eb0 2024-03-06T07:32:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.195+0000 7f0e01600640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0de406b4d0 0x7f0de406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.195+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0dfc0ab190 con 0x7f0e040b8eb0 2024-03-06T07:32:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.195+0000 7f0e09400640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0de406b4d0 0x7f0de406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.197+0000 7f0e09400640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0de406b4d0 0x7f0de406d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0df8005af0 tx=0x7f0df8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.200+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0e040cc7b0 con 0x7f0e040b8eb0 2024-03-06T07:32:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.447+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e040b09e0 con 0x7f0e040b8eb0 2024-03-06T07:32:32.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.448+0000 7f0e01600640 1 -- 172.21.15.6:0/507607430 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f0dfc033020 con 0x7f0e040b8eb0 2024-03-06T07:32:32.451 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:32.451 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:32.451 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:32.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0de406b4d0 msgr2=0x7f0de406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0de406b4d0 0x7f0de406d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0df8005af0 tx=0x7f0df8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:32.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 msgr2=0x7f0e040b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 0x7f0e040b6f70 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc001f70 tx=0x7f0dfc0014a0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 shutdown_connections 2024-03-06T07:32:32.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0de406b4d0 0x7f0de406d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0e040b8eb0 0x7f0e040b6f70 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0e040cda30 0x7f0e040b8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 --2- 172.21.15.6:0/507607430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0e04157300 0x7f0e040cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 >> 172.21.15.6:0/507607430 conn(0x7f0e040d3d30 msgr2=0x7f0e040b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 shutdown_connections 2024-03-06T07:32:32.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.451+0000 7f0e0a800640 1 -- 172.21.15.6:0/507607430 wait complete. 2024-03-06T07:32:32.473 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:32.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.692+0000 7f0aada00640 1 Processor -- start 2024-03-06T07:32:32.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.693+0000 7f0aada00640 1 -- start start 2024-03-06T07:32:32.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.694+0000 7f0aada00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0aa80d0e50 0x7f0aa8168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.694+0000 7f0aada00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa80b3710 con 0x7f0aa815e0b0 2024-03-06T07:32:32.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.694+0000 7f0aada00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa80b39f0 con 0x7f0aa80d4a10 2024-03-06T07:32:32.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.694+0000 7f0aada00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa80b09e0 con 0x7f0aa80d0e50 2024-03-06T07:32:32.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.694+0000 7f0aa7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0aa815e0b0 0x7f0aa80cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33626/0 (socket says 172.21.15.6:33626) 2024-03-06T07:32:32.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.694+0000 7f0aa7e00640 1 -- 172.21.15.6:0/538331841 learned_addr learned my addr 172.21.15.6:0/538331841 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aad000640 1 --2- 172.21.15.6:0/538331841 >> v2:172.21.15.178:3300/0 conn(0x7f0aa80d0e50 0x7f0aa8168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115391930 0 0) 0x7f0aa80b3710 con 0x7f0aa815e0b0 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a88003520 con 0x7f0aa815e0b0 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699144207 0 0) 0x7f0a88003520 con 0x7f0aa815e0b0 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 >> v2:172.21.15.178:3300/0 conn(0x7f0aa80d0e50 msgr2=0x7f0aa8168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 --2- 172.21.15.6:0/538331841 >> v2:172.21.15.178:3300/0 conn(0x7f0aa80d0e50 0x7f0aa8168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 >> v1:172.21.15.178:6789/0 conn(0x7f0aa80d4a10 legacy=0x7f0aa80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa80b3cd0 con 0x7f0aa815e0b0 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0a98001670 con 0x7f0aa815e0b0 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 >> v1:172.21.15.6:6789/0 conn(0x7f0aa815e0b0 legacy=0x7f0aa80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a880039b0 0x7f0a88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.695+0000 7f0aa6a00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a88004800 0x7f0a88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa6a00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a880051c0 0x7f0a8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8800c0e0 con 0x7f0a880051c0 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8800c290 con 0x7f0a880039b0 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8800c490 con 0x7f0a88004800 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aad000640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a880051c0 0x7f0a8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa7400640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a88004800 0x7f0a88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aad000640 1 -- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a88004800 msgr2=0x7f0a88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa7e00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a880039b0 0x7f0a88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aad000640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a88004800 0x7f0a88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aad000640 1 -- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a880039b0 msgr2=0x7f0a88003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aad000640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a880039b0 0x7f0a88003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aad000640 1 -- 172.21.15.6:0/538331841 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a88003520 con 0x7f0a880051c0 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa7400640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a88004800 0x7f0a88004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:32.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.696+0000 7f0aa7e00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a880039b0 0x7f0a88003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:32.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.697+0000 7f0aad000640 1 -- 172.21.15.6:0/538331841 reap_dead start 2024-03-06T07:32:32.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.697+0000 7f0aad000640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a880051c0 0x7f0a8800b9c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000ba00 tx=0x7f0a88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f197c54483076c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0036030 con 0x7f0a880051c0 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa0004020 con 0x7f0a880051c0 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aa6a00640 1 -- 172.21.15.6:0/538331841 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa000b600 con 0x7f0a880051c0 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 -- 172.21.15.6:0/538331841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a880051c0 msgr2=0x7f0a8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a880051c0 0x7f0a8800b9c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000ba00 tx=0x7f0a88005830 comp rx=0 tx=0).stop 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 -- 172.21.15.6:0/538331841 shutdown_connections 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 --2- 172.21.15.6:0/538331841 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a880051c0 0x7f0a8800b9c0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 -- 172.21.15.6:0/538331841 >> 172.21.15.6:0/538331841 conn(0x7f0aa80d3d30 msgr2=0x7f0aa80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 -- 172.21.15.6:0/538331841 shutdown_connections 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.698+0000 7f0aada00640 1 -- 172.21.15.6:0/538331841 wait complete. 2024-03-06T07:32:32.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.700+0000 7f0aada00640 1 Processor -- start 2024-03-06T07:32:32.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.700+0000 7f0aada00640 1 -- start start 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.701+0000 7f0aada00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a880051c0 0x7f0aa815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.701+0000 7f0aada00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 0x7f0aa80a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.701+0000 7f0aada00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aa80cd210 0x7f0aa80cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.701+0000 7f0aada00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa80cfc60 con 0x7f0aa80c7650 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.701+0000 7f0aada00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa815e490 con 0x7f0aa80cd210 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.701+0000 7f0aada00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa80d0740 con 0x7f0a880051c0 2024-03-06T07:32:32.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 0x7f0aa80a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 0x7f0aa80a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33702/0 (socket says 172.21.15.6:33702) 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 -- 172.21.15.6:0/3458731101 learned_addr learned my addr 172.21.15.6:0/3458731101 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7e00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a880051c0 0x7f0aa815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aad000640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aa80cd210 0x7f0aa80cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 -- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a880051c0 msgr2=0x7f0aa815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a880051c0 0x7f0aa815a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 -- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aa80cd210 msgr2=0x7f0aa80cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aa80cd210 0x7f0aa80cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 -- 172.21.15.6:0/3458731101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa80c7bc0 con 0x7f0aa80c7650 2024-03-06T07:32:32.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.702+0000 7f0aa7400640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 0x7f0aa80a1220 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000d8b0 tx=0x7f0a9000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.703+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90003da0 con 0x7f0aa80c7650 2024-03-06T07:32:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.703+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a90017070 con 0x7f0aa80c7650 2024-03-06T07:32:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.703+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900052b0 con 0x7f0aa80c7650 2024-03-06T07:32:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.703+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa80c7ea0 con 0x7f0aa80c7650 2024-03-06T07:32:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.703+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa80c8460 con 0x7f0aa80c7650 2024-03-06T07:32:32.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.704+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa804f330 con 0x7f0aa80c7650 2024-03-06T07:32:32.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.705+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a90005450 con 0x7f0aa80c7650 2024-03-06T07:32:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.706+0000 7f0a9fe00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a8406b4d0 0x7f0a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.706+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a9008a3e0 con 0x7f0aa80c7650 2024-03-06T07:32:32.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.707+0000 7f0aa7e00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a8406b4d0 0x7f0a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:32.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.708+0000 7f0aa7e00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a8406b4d0 0x7f0a8406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0a98001920 tx=0x7f0a98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:32.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.711+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a900591c0 con 0x7f0aa80c7650 2024-03-06T07:32:32.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.953+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aa815d3a0 con 0x7f0aa80c7650 2024-03-06T07:32:32.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.954+0000 7f0a9fe00640 1 -- 172.21.15.6:0/3458731101 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f0a90058ac0 con 0x7f0aa80c7650 2024-03-06T07:32:32.957 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:32.957 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":53,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:14.630088+0000","modified":"2024-03-06T07:32:17.693833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6801},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6801":{"gid":6801,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.178:6836/1516947936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1516947936},{"type":"v1","addr":"172.21.15.178:6836","nonce":1516947936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6801,"qdb_cluster":[6801]},"id":6}]} 2024-03-06T07:32:32.957 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 53 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a8406b4d0 msgr2=0x7f0a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a8406b4d0 0x7f0a8406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0a98001920 tx=0x7f0a98009040 comp rx=0 tx=0).stop 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 msgr2=0x7f0aa80a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 0x7f0aa80a1220 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000d8b0 tx=0x7f0a9000dd70 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 shutdown_connections 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a8406b4d0 0x7f0a8406d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aa80cd210 0x7f0aa80cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aa80c7650 0x7f0aa80a1220 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 --2- 172.21.15.6:0/3458731101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a880051c0 0x7f0aa815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.957+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 >> 172.21.15.6:0/3458731101 conn(0x7f0aa80d3d30 msgr2=0x7f0aa80b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.958+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 shutdown_connections 2024-03-06T07:32:32.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:32.958+0000 7f0aada00640 1 -- 172.21.15.6:0/3458731101 wait complete. 2024-03-06T07:32:32.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:33.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.191+0000 7f9307c00640 1 Processor -- start 2024-03-06T07:32:33.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.191+0000 7f9307c00640 1 -- start start 2024-03-06T07:32:33.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9307c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f930015f360 0x7f93000b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9307c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93000b6770 0x7f9300161cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9307c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930006f0a0 con 0x7f930015df70 2024-03-06T07:32:33.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9307c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930006fcb0 con 0x7f93000b6770 2024-03-06T07:32:33.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9307c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930006f380 con 0x7f930015f360 2024-03-06T07:32:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9305c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f93000b6770 0x7f9300161cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.193+0000 7f9305200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f930015df70 0x7f9300070140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33638/0 (socket says 172.21.15.6:33638) 2024-03-06T07:32:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f9305200640 1 -- 172.21.15.6:0/661800523 learned_addr learned my addr 172.21.15.6:0/661800523 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f92f7e00640 1 --2- 172.21.15.6:0/661800523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f930015f360 0x7f93000b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f9305c00640 1 -- 172.21.15.6:0/661800523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f930015f360 msgr2=0x7f93000b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f9305c00640 1 --2- 172.21.15.6:0/661800523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f930015f360 0x7f93000b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f9305c00640 1 -- 172.21.15.6:0/661800523 >> v1:172.21.15.6:6789/0 conn(0x7f930015df70 legacy=0x7f9300070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f9305c00640 1 -- 172.21.15.6:0/661800523 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930006f660 con 0x7f93000b6770 2024-03-06T07:32:33.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.194+0000 7f92ffe00640 1 -- 172.21.15.6:0/661800523 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451172976 0 0) 0x7f930006f0a0 con 0x7f930015df70 2024-03-06T07:32:33.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.195+0000 7f9305c00640 1 --2- 172.21.15.6:0/661800523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b6770 0x7f9300161cc0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009d90 tx=0x7f92f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2dc3d0546e6ed48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f92ffe00640 1 -- 172.21.15.6:0/661800523 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8039070 con 0x7f93000b6770 2024-03-06T07:32:33.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f92ffe00640 1 -- 172.21.15.6:0/661800523 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f802cc60 con 0x7f93000b6770 2024-03-06T07:32:33.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f92ffe00640 1 -- 172.21.15.6:0/661800523 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8034660 con 0x7f93000b6770 2024-03-06T07:32:33.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 -- 172.21.15.6:0/661800523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b6770 msgr2=0x7f9300161cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 --2- 172.21.15.6:0/661800523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b6770 0x7f9300161cc0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009d90 tx=0x7f92f802beb0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 -- 172.21.15.6:0/661800523 shutdown_connections 2024-03-06T07:32:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 --2- 172.21.15.6:0/661800523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b6770 0x7f9300161cc0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 --2- 172.21.15.6:0/661800523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f930015f360 0x7f93000b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 -- 172.21.15.6:0/661800523 >> 172.21.15.6:0/661800523 conn(0x7f930009aa10 msgr2=0x7f930009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 -- 172.21.15.6:0/661800523 shutdown_connections 2024-03-06T07:32:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.196+0000 7f9307c00640 1 -- 172.21.15.6:0/661800523 wait complete. 2024-03-06T07:32:33.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.198+0000 7f9307c00640 1 Processor -- start 2024-03-06T07:32:33.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.198+0000 7f9307c00640 1 -- start start 2024-03-06T07:32:33.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.199+0000 7f9307c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93000b6770 0x7f9300147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.199+0000 7f9307c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930015df70 0x7f93000b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.199+0000 7f9307c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 0x7f93001e8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9307c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9300162380 con 0x7f93000b1d50 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9307c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9300162200 con 0x7f930015df70 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9307c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9300162500 con 0x7f93000b6770 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 0x7f93001e8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 0x7f93001e8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33734/0 (socket says 172.21.15.6:33734) 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 -- 172.21.15.6:0/2772625343 learned_addr learned my addr 172.21.15.6:0/2772625343 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 -- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93000b6770 msgr2=0x7f9300147710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93000b6770 0x7f9300147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 -- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930015df70 msgr2=0x7f93000b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930015df70 0x7f93000b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.200+0000 7f9305c00640 1 -- 172.21.15.6:0/2772625343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93000b45d0 con 0x7f93000b1d50 2024-03-06T07:32:33.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.201+0000 7f9305c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 0x7f93001e8fc0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f92f8037040 tx=0x7f92f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.201+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8044020 con 0x7f93000b1d50 2024-03-06T07:32:33.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.201+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f803a040 con 0x7f93000b1d50 2024-03-06T07:32:33.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.201+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f8039050 con 0x7f93000b1d50 2024-03-06T07:32:33.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.201+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93000b2380 con 0x7f93000b1d50 2024-03-06T07:32:33.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.201+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93000b2940 con 0x7f93000b1d50 2024-03-06T07:32:33.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.202+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930004f3a0 con 0x7f93000b1d50 2024-03-06T07:32:33.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.203+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92f8003bb0 con 0x7f93000b1d50 2024-03-06T07:32:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.204+0000 7f92fd600640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92dc06b4d0 0x7f92dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.204+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92f80ac7b0 con 0x7f93000b1d50 2024-03-06T07:32:33.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.205+0000 7f9305200640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92dc06b4d0 0x7f92dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.206+0000 7f9305200640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92dc06b4d0 0x7f92dc06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f92e8007fd0 tx=0x7f92e8007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.211+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92f804e060 con 0x7f93000b1d50 2024-03-06T07:32:33.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.415+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f930006c220 con 0x7f93000b1d50 2024-03-06T07:32:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.417+0000 7f92fd600640 1 -- 172.21.15.6:0/2772625343 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f92f807b6c0 con 0x7f93000b1d50 2024-03-06T07:32:33.420 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:33.420 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":81,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:16.181620+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:12.534039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:13.644895+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92dc06b4d0 msgr2=0x7f92dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92dc06b4d0 0x7f92dc06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f92e8007fd0 tx=0x7f92e8007920 comp rx=0 tx=0).stop 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 msgr2=0x7f93001e8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 0x7f93001e8fc0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f92f8037040 tx=0x7f92f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 shutdown_connections 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92dc06b4d0 0x7f92dc06d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f930015df70 0x7f93000b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f93000b1d50 0x7f93001e8fc0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 --2- 172.21.15.6:0/2772625343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f93000b6770 0x7f9300147710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 >> 172.21.15.6:0/2772625343 conn(0x7f930009aa10 msgr2=0x7f930006cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 shutdown_connections 2024-03-06T07:32:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.420+0000 7f9307c00640 1 -- 172.21.15.6:0/2772625343 wait complete. 2024-03-06T07:32:33.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:33.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.663+0000 7f3fc1e00640 1 Processor -- start 2024-03-06T07:32:33.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.663+0000 7f3fc1e00640 1 -- start start 2024-03-06T07:32:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.664+0000 7f3fc1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc1552b0 0x7f3fbc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.664+0000 7f3fc1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3fbc15e810 0x7f3fbc0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.664+0000 7f3fc1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3fbc0cce90 0x7f3fbc06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.664+0000 7f3fc1e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc15ed10 con 0x7f3fbc0cce90 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.664+0000 7f3fc1e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc15eff0 con 0x7f3fbc15e810 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc15f2d0 con 0x7f3fbc1552b0 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3fbc0cce90 0x7f3fbc06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3fbc0cce90 0x7f3fbc06c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33742/0 (socket says 172.21.15.6:33742) 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 -- 172.21.15.6:0/4135705589 learned_addr learned my addr 172.21.15.6:0/4135705589 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc0a00640 1 --2- 172.21.15.6:0/4135705589 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc1552b0 0x7f3fbc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fb3e00640 1 --2- 172.21.15.6:0/4135705589 >> v2:172.21.15.178:3300/0 conn(0x7f3fbc15e810 0x7f3fbc0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 -- 172.21.15.6:0/4135705589 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc1552b0 msgr2=0x7f3fbc15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 --2- 172.21.15.6:0/4135705589 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc1552b0 0x7f3fbc15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 -- 172.21.15.6:0/4135705589 >> v2:172.21.15.178:3300/0 conn(0x7f3fbc15e810 msgr2=0x7f3fbc0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 --2- 172.21.15.6:0/4135705589 >> v2:172.21.15.178:3300/0 conn(0x7f3fbc15e810 0x7f3fbc0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.665+0000 7f3fc1400640 1 -- 172.21.15.6:0/4135705589 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc15f5b0 con 0x7f3fbc0cce90 2024-03-06T07:32:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.667+0000 7f3fc1400640 1 --2- 172.21.15.6:0/4135705589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc0cce90 0x7f3fbc06c7c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc069210 tx=0x7f3fb402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f416a8b42a23cc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.667+0000 7f3fbbe00640 1 -- 172.21.15.6:0/4135705589 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb402c9e0 con 0x7f3fbc0cce90 2024-03-06T07:32:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.667+0000 7f3fbbe00640 1 -- 172.21.15.6:0/4135705589 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb402cb80 con 0x7f3fbc0cce90 2024-03-06T07:32:33.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.667+0000 7f3fbbe00640 1 -- 172.21.15.6:0/4135705589 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb40343f0 con 0x7f3fbc0cce90 2024-03-06T07:32:33.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 -- 172.21.15.6:0/4135705589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc0cce90 msgr2=0x7f3fbc06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/4135705589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc0cce90 0x7f3fbc06c7c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc069210 tx=0x7f3fb402be40 comp rx=0 tx=0).stop 2024-03-06T07:32:33.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 -- 172.21.15.6:0/4135705589 shutdown_connections 2024-03-06T07:32:33.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/4135705589 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc0cce90 0x7f3fbc06c7c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/4135705589 >> v2:172.21.15.178:3300/0 conn(0x7f3fbc15e810 0x7f3fbc0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/4135705589 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc1552b0 0x7f3fbc15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 -- 172.21.15.6:0/4135705589 >> 172.21.15.6:0/4135705589 conn(0x7f3fbc06b3d0 msgr2=0x7f3fbc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 -- 172.21.15.6:0/4135705589 shutdown_connections 2024-03-06T07:32:33.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.668+0000 7f3fc1e00640 1 -- 172.21.15.6:0/4135705589 wait complete. 2024-03-06T07:32:33.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.670+0000 7f3fc1e00640 1 Processor -- start 2024-03-06T07:32:33.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.670+0000 7f3fc1e00640 1 -- start start 2024-03-06T07:32:33.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc0cce90 0x7f3fbc13a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fbc1552b0 0x7f3fbc14f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 0x7f3fbc0b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc06dd30 con 0x7f3fbc15e810 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0b3760 con 0x7f3fbc1552b0 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc06deb0 con 0x7f3fbc0cce90 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 0x7f3fbc0b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 0x7f3fbc0b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33750/0 (socket says 172.21.15.6:33750) 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1400640 1 -- 172.21.15.6:0/2446972253 learned_addr learned my addr 172.21.15.6:0/2446972253 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fb3e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fbc1552b0 0x7f3fbc14f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc0a00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc0cce90 0x7f3fbc13a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1400640 1 -- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc0cce90 msgr2=0x7f3fbc13a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.671+0000 7f3fc1400640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc0cce90 0x7f3fbc13a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fc1400640 1 -- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fbc1552b0 msgr2=0x7f3fbc14f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fc1400640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fbc1552b0 0x7f3fbc14f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fc1400640 1 -- 172.21.15.6:0/2446972253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc150f00 con 0x7f3fbc15e810 2024-03-06T07:32:33.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fb3e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fbc1552b0 0x7f3fbc14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:33.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fc1400640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 0x7f3fbc0b17c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009f40 tx=0x7f3fb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4043020 con 0x7f3fbc15e810 2024-03-06T07:32:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb403e450 con 0x7f3fbc15e810 2024-03-06T07:32:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc13a7e0 con 0x7f3fbc15e810 2024-03-06T07:32:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4009510 con 0x7f3fbc15e810 2024-03-06T07:32:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.672+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc150140 con 0x7f3fbc15e810 2024-03-06T07:32:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.674+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0cca40 con 0x7f3fbc15e810 2024-03-06T07:32:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.675+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb4046020 con 0x7f3fbc15e810 2024-03-06T07:32:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.675+0000 7f3fb9600640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f9806b4d0 0x7f3f9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.676+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fb40ace20 con 0x7f3fbc15e810 2024-03-06T07:32:33.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.676+0000 7f3fc0a00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f9806b4d0 0x7f3f9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.677+0000 7f3fc0a00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f9806b4d0 0x7f3f9806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3fac006fd0 tx=0x7f3fac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.681+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fbc0cca40 con 0x7f3fbc15e810 2024-03-06T07:32:33.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.924+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3fbc15c1b0 con 0x7f3fbc15e810 2024-03-06T07:32:33.927 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:32:33.926+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:33.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.929+0000 7f3fb9600640 1 -- 172.21.15.6:0/2446972253 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3fbc15c1b0 con 0x7f3fbc15e810 2024-03-06T07:32:33.931 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.932+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f9806b4d0 msgr2=0x7f3f9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.932+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f9806b4d0 0x7f3f9806d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3fac006fd0 tx=0x7f3fac008040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.932+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 msgr2=0x7f3fbc0b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.932+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 0x7f3fbc0b17c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009f40 tx=0x7f3fb4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 shutdown_connections 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f9806b4d0 0x7f3f9806d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fbc1552b0 0x7f3fbc14f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fbc15e810 0x7f3fbc0b17c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 --2- 172.21.15.6:0/2446972253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fbc0cce90 0x7f3fbc13a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 >> 172.21.15.6:0/2446972253 conn(0x7f3fbc06b3d0 msgr2=0x7f3fbc0cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 shutdown_connections 2024-03-06T07:32:33.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:33.933+0000 7f3fc1e00640 1 -- 172.21.15.6:0/2446972253 wait complete. 2024-03-06T07:32:33.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:33.959 INFO:tasks.ceph.mds.b.smithi178.stderr: -3857> 2024-03-06T07:32:19.553+0000 7f5eae000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:34.053 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:32:34.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.175+0000 7f5604e00640 1 Processor -- start 2024-03-06T07:32:34.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.175+0000 7f5604e00640 1 -- start start 2024-03-06T07:32:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f5604e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5600162280 0x7f5600165d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f5604e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 0x7f56000b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f5604e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5600166cb0 con 0x7f56000b4a90 2024-03-06T07:32:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f5604e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5600166410 con 0x7f5600162280 2024-03-06T07:32:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f5604e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56001666f0 con 0x7f56000b1b40 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f55ffe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 0x7f56000b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f55ffe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 0x7f56000b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:36496/0 (socket says 172.21.15.6:36496) 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f55ffe00640 1 -- 172.21.15.6:0/2378215473 learned_addr learned my addr 172.21.15.6:0/2378215473 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.177+0000 7f55ff400640 1 --2- 172.21.15.6:0/2378215473 >> v2:172.21.15.178:3300/0 conn(0x7f5600162280 0x7f5600165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.178+0000 7f55ffe00640 1 -- 172.21.15.6:0/2378215473 >> v2:172.21.15.178:3300/0 conn(0x7f5600162280 msgr2=0x7f5600165d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.178+0000 7f55ffe00640 1 --2- 172.21.15.6:0/2378215473 >> v2:172.21.15.178:3300/0 conn(0x7f5600162280 0x7f5600165d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.178+0000 7f55ffe00640 1 -- 172.21.15.6:0/2378215473 >> v1:172.21.15.178:6789/0 conn(0x7f56000b4a90 legacy=0x7f56000cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.178+0000 7f55ffe00640 1 -- 172.21.15.6:0/2378215473 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56001669d0 con 0x7f56000b1b40 2024-03-06T07:32:34.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.179+0000 7f55ffe00640 1 --2- 172.21.15.6:0/2378215473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 0x7f56000b1fd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009e50 tx=0x7f55f802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b64dd16284be0c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.180+0000 7f55fea00640 1 -- 172.21.15.6:0/2378215473 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f802ca70 con 0x7f56000b1b40 2024-03-06T07:32:34.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.180+0000 7f55fea00640 1 -- 172.21.15.6:0/2378215473 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f802cc10 con 0x7f56000b1b40 2024-03-06T07:32:34.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.180+0000 7f55fea00640 1 -- 172.21.15.6:0/2378215473 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f80343f0 con 0x7f56000b1b40 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.180+0000 7f5604e00640 1 -- 172.21.15.6:0/2378215473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 msgr2=0x7f56000b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.180+0000 7f5604e00640 1 --2- 172.21.15.6:0/2378215473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 0x7f56000b1fd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009e50 tx=0x7f55f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.181+0000 7f5604e00640 1 -- 172.21.15.6:0/2378215473 shutdown_connections 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.181+0000 7f5604e00640 1 --2- 172.21.15.6:0/2378215473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b1b40 0x7f56000b1fd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.181+0000 7f5604e00640 1 --2- 172.21.15.6:0/2378215473 >> v2:172.21.15.178:3300/0 conn(0x7f5600162280 0x7f5600165d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.181+0000 7f5604e00640 1 -- 172.21.15.6:0/2378215473 >> 172.21.15.6:0/2378215473 conn(0x7f560006b3a0 msgr2=0x7f560006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.181+0000 7f5604e00640 1 -- 172.21.15.6:0/2378215473 shutdown_connections 2024-03-06T07:32:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.181+0000 7f5604e00640 1 -- 172.21.15.6:0/2378215473 wait complete. 2024-03-06T07:32:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.182+0000 7f5604e00640 1 Processor -- start 2024-03-06T07:32:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.183+0000 7f5604e00640 1 -- start start 2024-03-06T07:32:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.183+0000 7f5604e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56000b1b40 0x7f56001f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.183+0000 7f5604e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b4a90 0x7f56001588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f5604e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 0x7f5600158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f5604e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56000716e0 con 0x7f5600162280 2024-03-06T07:32:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f5604e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5600071560 con 0x7f56000b1b40 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f5604e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5600071860 con 0x7f56000b4a90 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 0x7f5600158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 0x7f5600158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33764/0 (socket says 172.21.15.6:33764) 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 -- 172.21.15.6:0/411772297 learned_addr learned my addr 172.21.15.6:0/411772297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55f7e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b4a90 0x7f56001588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ff400640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56000b1b40 0x7f56001f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 -- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b4a90 msgr2=0x7f56001588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b4a90 0x7f56001588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 -- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56000b1b40 msgr2=0x7f56001f8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56000b1b40 0x7f56001f8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.184+0000 7f55ffe00640 1 -- 172.21.15.6:0/411772297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600164520 con 0x7f5600162280 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.185+0000 7f55ffe00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 0x7f5600158e00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f55f8036040 tx=0x7f55f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.185+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8043020 con 0x7f5600162280 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.185+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f803e450 con 0x7f5600162280 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.185+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8009510 con 0x7f5600162280 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.185+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56001647c0 con 0x7f5600162280 2024-03-06T07:32:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.185+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5600164d80 con 0x7f5600162280 2024-03-06T07:32:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.187+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560004f330 con 0x7f5600162280 2024-03-06T07:32:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.188+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55f8046020 con 0x7f5600162280 2024-03-06T07:32:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.188+0000 7f55f7400640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55dc06b4d0 0x7f55dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.189+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f55f80ac960 con 0x7f5600162280 2024-03-06T07:32:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.189+0000 7f55ff400640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55dc06b4d0 0x7f55dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.190+0000 7f55ff400640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55dc06b4d0 0x7f55dc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009970 tx=0x7f55f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.193+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55f804d030 con 0x7f5600162280 2024-03-06T07:32:34.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.437+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5600072a00 con 0x7f5600162280 2024-03-06T07:32:34.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.933+0000 7f55f7400640 1 -- 172.21.15.6:0/411772297 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 104+0+0 (secure 0 0 0) 0x7f55f80ac0a0 con 0x7f5600162280 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55dc06b4d0 msgr2=0x7f55dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55dc06b4d0 0x7f55dc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009970 tx=0x7f55f0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 msgr2=0x7f5600158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 0x7f5600158e00 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f55f8036040 tx=0x7f55f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 shutdown_connections 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f55dc06b4d0 0x7f55dc06d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5600162280 0x7f5600158e00 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f56000b4a90 0x7f56001588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 --2- 172.21.15.6:0/411772297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56000b1b40 0x7f56001f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.936+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 >> 172.21.15.6:0/411772297 conn(0x7f560006b3a0 msgr2=0x7f560015cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.937+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 shutdown_connections 2024-03-06T07:32:34.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:34.937+0000 7f5604e00640 1 -- 172.21.15.6:0/411772297 wait complete. 2024-03-06T07:32:34.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:35.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.185+0000 7f0d21800640 1 Processor -- start 2024-03-06T07:32:35.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.185+0000 7f0d21800640 1 -- start start 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.187+0000 7f0d21800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 0x7f0d1c0ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.187+0000 7f0d21800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0ae910 con 0x7f0d1c0cf8a0 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.187+0000 7f0d21800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c15f5f0 con 0x7f0d1c15efd0 2024-03-06T07:32:35.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.187+0000 7f0d21800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0adf40 con 0x7f0d1c072860 2024-03-06T07:32:35.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.187+0000 7f0d12000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0d1c0cf8a0 0x7f0d1c15e8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33652/0 (socket says 172.21.15.6:33652) 2024-03-06T07:32:35.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d12000640 1 -- 172.21.15.6:0/2723211873 learned_addr learned my addr 172.21.15.6:0/2723211873 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 0x7f0d1c0ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537344743 0 0) 0x7f0d1c0ae910 con 0x7f0d1c0cf8a0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d00005730 con 0x7f0d1c0cf8a0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3614535469 0 0) 0x7f0d00005730 con 0x7f0d1c0cf8a0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 msgr2=0x7f0d1c0ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1a000640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 0x7f0d1c0ae3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.188+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 >> v1:172.21.15.178:6789/0 conn(0x7f0d1c15efd0 legacy=0x7f0d1c071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c0ad8f0 con 0x7f0d1c0cf8a0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0d04001670 con 0x7f0d1c0cf8a0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 >> v1:172.21.15.6:6789/0 conn(0x7f0d1c0cf8a0 legacy=0x7f0d1c15e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d00005bf0 0x7f0d00006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 0x7f0d00006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d00006f50 0x7f0d000073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0adf40 con 0x7f0d00005bf0 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0000dd00 con 0x7f0d00006f50 2024-03-06T07:32:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0000df00 con 0x7f0d1c072860 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d00005bf0 0x7f0d00006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1aa00640 1 -- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 msgr2=0x7f0d00006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d1c072860 0x7f0d00006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1aa00640 1 -- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d00006f50 msgr2=0x7f0d000073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1b400640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d00006f50 0x7f0d000073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d00006f50 0x7f0d000073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1aa00640 1 -- 172.21.15.6:0/2723211873 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00005730 con 0x7f0d00005bf0 2024-03-06T07:32:35.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.189+0000 7f0d1b400640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d00006f50 0x7f0d000073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:35.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d00005bf0 0x7f0d00006010 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009ac0 tx=0x7f0d00007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e97282c51411574 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c039070 con 0x7f0d00005bf0 2024-03-06T07:32:35.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c02c630 con 0x7f0d00005bf0 2024-03-06T07:32:35.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d1a000640 1 -- 172.21.15.6:0/2723211873 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c009690 con 0x7f0d00005bf0 2024-03-06T07:32:35.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d21800640 1 -- 172.21.15.6:0/2723211873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d00005bf0 msgr2=0x7f0d00006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d21800640 1 --2- 172.21.15.6:0/2723211873 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d00005bf0 0x7f0d00006010 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c009ac0 tx=0x7f0d00007a40 comp rx=0 tx=0).stop 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.191+0000 7f0d1b400640 1 -- 172.21.15.6:0/2723211873 reap_dead start 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.192+0000 7f0d21800640 1 -- 172.21.15.6:0/2723211873 shutdown_connections 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.192+0000 7f0d21800640 1 -- 172.21.15.6:0/2723211873 >> 172.21.15.6:0/2723211873 conn(0x7f0d1c06afb0 msgr2=0x7f0d1c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.192+0000 7f0d21800640 1 -- 172.21.15.6:0/2723211873 shutdown_connections 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.192+0000 7f0d21800640 1 -- 172.21.15.6:0/2723211873 wait complete. 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.193+0000 7f0d21800640 1 Processor -- start 2024-03-06T07:32:35.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.194+0000 7f0d21800640 1 -- start start 2024-03-06T07:32:35.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d21800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d00005bf0 0x7f0d1c14de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d21800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d1c14e570 0x7f0d1c0afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d21800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 0x7f0d1c150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d21800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0ae3d0 con 0x7f0d1c14fed0 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d21800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0ad570 con 0x7f0d1c14e570 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d21800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0ae550 con 0x7f0d00005bf0 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 0x7f0d1c150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 0x7f0d1c150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33788/0 (socket says 172.21.15.6:33788) 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 -- 172.21.15.6:0/2988967308 learned_addr learned my addr 172.21.15.6:0/2988967308 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d00005bf0 0x7f0d1c14de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d12000640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d1c14e570 0x7f0d1c0afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 -- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d00005bf0 msgr2=0x7f0d1c14de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d00005bf0 0x7f0d1c14de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 -- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d1c14e570 msgr2=0x7f0d1c0afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.195+0000 7f0d1b400640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d1c14e570 0x7f0d1c0afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d1b400640 1 -- 172.21.15.6:0/2988967308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c14f200 con 0x7f0d1c14fed0 2024-03-06T07:32:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d00005bf0 0x7f0d1c14de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:35.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d1b400640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 0x7f0d1c150350 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0b1810 tx=0x7f0d1400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14006d50 con 0x7f0d1c14fed0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d14003970 con 0x7f0d1c14fed0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14003d60 con 0x7f0d1c14fed0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c14f4a0 con 0x7f0d1c14fed0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.196+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1579a0 con 0x7f0d1c14fed0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.197+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c157ba0 con 0x7f0d1c14fed0 2024-03-06T07:32:35.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.198+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d1401b020 con 0x7f0d1c14fed0 2024-03-06T07:32:35.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.199+0000 7f0d13400640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0cf806b500 0x7f0cf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.199+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0d14089c70 con 0x7f0d1c14fed0 2024-03-06T07:32:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.200+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0cf806b500 0x7f0cf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.201+0000 7f0d1aa00640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0cf806b500 0x7f0cf806d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c036000 tx=0x7f0d0c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.205+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0d1c157ba0 con 0x7f0d1c14fed0 2024-03-06T07:32:35.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:35.408+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0d1c068010 con 0x7f0d1c14fed0 2024-03-06T07:32:36.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.312+0000 7f0d13400640 1 -- 172.21.15.6:0/2988967308 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0d14058a80 con 0x7f0d1c14fed0 2024-03-06T07:32:36.314 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:36.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0cf806b500 msgr2=0x7f0cf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0cf806b500 0x7f0cf806d9c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c036000 tx=0x7f0d0c030000 comp rx=0 tx=0).stop 2024-03-06T07:32:36.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 msgr2=0x7f0d1c150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 0x7f0d1c150350 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0b1810 tx=0x7f0d1400dc00 comp rx=0 tx=0).stop 2024-03-06T07:32:36.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 shutdown_connections 2024-03-06T07:32:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0cf806b500 0x7f0cf806d9c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0d1c14e570 0x7f0d1c0afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0d1c14fed0 0x7f0d1c150350 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 --2- 172.21.15.6:0/2988967308 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0d00005bf0 0x7f0d1c14de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.316+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 >> 172.21.15.6:0/2988967308 conn(0x7f0d1c06afb0 msgr2=0x7f0d1c15a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.317+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 shutdown_connections 2024-03-06T07:32:36.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.319+0000 7f0d21800640 1 -- 172.21.15.6:0/2988967308 wait complete. 2024-03-06T07:32:36.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:36.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.558+0000 7f10e3a00640 1 Processor -- start 2024-03-06T07:32:36.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.558+0000 7f10e3a00640 1 -- start start 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e3a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f10dc15e1c0 0x7f10dc06f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e3a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f10dc162830 0x7f10dc161c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e3a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc0702d0 con 0x7f10dc162140 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e3a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc06c0c0 con 0x7f10dc15e1c0 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e3a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc0705b0 con 0x7f10dc162830 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f10dc15e1c0 0x7f10dc06f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e1000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f10dc15e1c0 0x7f10dc06f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33804/0 (socket says 172.21.15.6:33804) 2024-03-06T07:32:36.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.560+0000 7f10e1000640 1 -- 172.21.15.6:0/3371695303 learned_addr learned my addr 172.21.15.6:0/3371695303 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.561+0000 7f10d3e00640 1 --2- 172.21.15.6:0/3371695303 >> v2:172.21.15.178:3300/0 conn(0x7f10dc162830 0x7f10dc161c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.561+0000 7f10e1000640 1 -- 172.21.15.6:0/3371695303 >> v2:172.21.15.178:3300/0 conn(0x7f10dc162830 msgr2=0x7f10dc161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.561+0000 7f10e1000640 1 --2- 172.21.15.6:0/3371695303 >> v2:172.21.15.178:3300/0 conn(0x7f10dc162830 0x7f10dc161c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.561+0000 7f10e1000640 1 -- 172.21.15.6:0/3371695303 >> v1:172.21.15.178:6789/0 conn(0x7f10dc162140 legacy=0x7f10dc0b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.561+0000 7f10e1000640 1 -- 172.21.15.6:0/3371695303 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc06c310 con 0x7f10dc15e1c0 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.561+0000 7f10d3e00640 1 --2- 172.21.15.6:0/3371695303 >> v2:172.21.15.178:3300/0 conn(0x7f10dc162830 0x7f10dc161c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.562+0000 7f10e1000640 1 --2- 172.21.15.6:0/3371695303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc15e1c0 0x7f10dc06f930 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f10dc09bac0 tx=0x7f10cc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e0597d96a6d6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.562+0000 7f10dbe00640 1 -- 172.21.15.6:0/3371695303 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc02ca80 con 0x7f10dc15e1c0 2024-03-06T07:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.562+0000 7f10dbe00640 1 -- 172.21.15.6:0/3371695303 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc02cc20 con 0x7f10dc15e1c0 2024-03-06T07:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.562+0000 7f10dbe00640 1 -- 172.21.15.6:0/3371695303 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0343f0 con 0x7f10dc15e1c0 2024-03-06T07:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 -- 172.21.15.6:0/3371695303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc15e1c0 msgr2=0x7f10dc06f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3371695303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc15e1c0 0x7f10dc06f930 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f10dc09bac0 tx=0x7f10cc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 -- 172.21.15.6:0/3371695303 shutdown_connections 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3371695303 >> v2:172.21.15.178:3300/0 conn(0x7f10dc162830 0x7f10dc161c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3371695303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc15e1c0 0x7f10dc06f930 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 -- 172.21.15.6:0/3371695303 >> 172.21.15.6:0/3371695303 conn(0x7f10dc09aa10 msgr2=0x7f10dc09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 -- 172.21.15.6:0/3371695303 shutdown_connections 2024-03-06T07:32:36.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.563+0000 7f10e3a00640 1 -- 172.21.15.6:0/3371695303 wait complete. 2024-03-06T07:32:36.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.565+0000 7f10e3a00640 1 Processor -- start 2024-03-06T07:32:36.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.565+0000 7f10e3a00640 1 -- start start 2024-03-06T07:32:36.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e3a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10dc15e1c0 0x7f10dc1e8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e3a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 0x7f10dc15c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e3a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10dc162830 0x7f10dc148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e3a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06fff0 con 0x7f10dc162140 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e3a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06fe70 con 0x7f10dc162830 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e3a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc06db20 con 0x7f10dc15e1c0 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 0x7f10dc15c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 0x7f10dc15c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33814/0 (socket says 172.21.15.6:33814) 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10d3e00640 1 -- 172.21.15.6:0/3601566360 learned_addr learned my addr 172.21.15.6:0/3601566360 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e1a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10dc162830 0x7f10dc148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.566+0000 7f10e1000640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10dc15e1c0 0x7f10dc1e8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d3e00640 1 -- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10dc15e1c0 msgr2=0x7f10dc1e8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d3e00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10dc15e1c0 0x7f10dc1e8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d3e00640 1 -- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10dc162830 msgr2=0x7f10dc148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d3e00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10dc162830 0x7f10dc148900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d3e00640 1 -- 172.21.15.6:0/3601566360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc15c860 con 0x7f10dc162140 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10e1a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10dc162830 0x7f10dc148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10e1000640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10dc15e1c0 0x7f10dc1e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d3e00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 0x7f10dc15c0f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f10c400c7f0 tx=0x7f10c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4015070 con 0x7f10dc162140 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c40077c0 con 0x7f10dc162140 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4007b70 con 0x7f10dc162140 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc15cb40 con 0x7f10dc162140 2024-03-06T07:32:36.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.567+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc15d100 con 0x7f10dc162140 2024-03-06T07:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.569+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc068010 con 0x7f10dc162140 2024-03-06T07:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.569+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10c4018070 con 0x7f10dc162140 2024-03-06T07:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.570+0000 7f10d9600640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.570+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f10c408a180 con 0x7f10dc162140 2024-03-06T07:32:36.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.571+0000 7f10e1000640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.572+0000 7f10e1000640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10b806b4d0 0x7f10b806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f10cc035000 tx=0x7f10cc030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.575+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10c4059150 con 0x7f10dc162140 2024-03-06T07:32:36.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:36.773+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10dc09b670 con 0x7f10dc162140 2024-03-06T07:32:37.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.315+0000 7f10d9600640 1 -- 172.21.15.6:0/3601566360 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f10c4058a50 con 0x7f10dc162140 2024-03-06T07:32:37.318 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:37.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.320+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10b806b4d0 msgr2=0x7f10b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.320+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10b806b4d0 0x7f10b806d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f10cc035000 tx=0x7f10cc030000 comp rx=0 tx=0).stop 2024-03-06T07:32:37.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.320+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 msgr2=0x7f10dc15c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.320+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 0x7f10dc15c0f0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f10c400c7f0 tx=0x7f10c400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.322+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 shutdown_connections 2024-03-06T07:32:37.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.322+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.322+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f10dc162830 0x7f10dc148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.322+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f10dc162140 0x7f10dc15c0f0 secure :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f10c400c7f0 tx=0x7f10c400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.322+0000 7f10e3a00640 1 --2- 172.21.15.6:0/3601566360 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10dc15e1c0 0x7f10dc1e8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.322+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 >> 172.21.15.6:0/3601566360 conn(0x7f10dc09aa10 msgr2=0x7f10dc074050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.323+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 shutdown_connections 2024-03-06T07:32:37.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.323+0000 7f10e3a00640 1 -- 172.21.15.6:0/3601566360 wait complete. 2024-03-06T07:32:37.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.545+0000 7f6e76c00640 1 Processor -- start 2024-03-06T07:32:37.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.545+0000 7f6e76c00640 1 -- start start 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e76c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6e7015bf20 0x7f6e700acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e76c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e70072520 0x7f6e700cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e76c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70071e20 0x7f6e700ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e76c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e70072df0 con 0x7f6e70072520 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e76c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e700cdab0 con 0x7f6e7015bf20 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e76c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700b4d70 con 0x7f6e70071e20 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e70072520 0x7f6e700cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6e70072520 0x7f6e700cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33828/0 (socket says 172.21.15.6:33828) 2024-03-06T07:32:37.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6f400640 1 -- 172.21.15.6:0/1991703107 learned_addr learned my addr 172.21.15.6:0/1991703107 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e74c00640 1 --2- 172.21.15.6:0/1991703107 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70071e20 0x7f6e700ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6f400640 1 -- 172.21.15.6:0/1991703107 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70071e20 msgr2=0x7f6e700ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6fe00640 1 --2- 172.21.15.6:0/1991703107 >> v2:172.21.15.178:3300/0 conn(0x7f6e7015bf20 0x7f6e700acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6f400640 1 --2- 172.21.15.6:0/1991703107 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70071e20 0x7f6e700ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.547+0000 7f6e6f400640 1 -- 172.21.15.6:0/1991703107 >> v2:172.21.15.178:3300/0 conn(0x7f6e7015bf20 msgr2=0x7f6e700acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.548+0000 7f6e6f400640 1 --2- 172.21.15.6:0/1991703107 >> v2:172.21.15.178:3300/0 conn(0x7f6e7015bf20 0x7f6e700acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.548+0000 7f6e6f400640 1 -- 172.21.15.6:0/1991703107 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e700b5050 con 0x7f6e70072520 2024-03-06T07:32:37.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.548+0000 7f6e6fe00640 1 --2- 172.21.15.6:0/1991703107 >> v2:172.21.15.178:3300/0 conn(0x7f6e7015bf20 0x7f6e700acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.549+0000 7f6e6f400640 1 --2- 172.21.15.6:0/1991703107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e70072520 0x7f6e700cdfb0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6e7006ec40 tx=0x7f6e5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5faeff11351d184e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.549+0000 7f6e6ea00640 1 -- 172.21.15.6:0/1991703107 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c035020 con 0x7f6e70072520 2024-03-06T07:32:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.549+0000 7f6e6ea00640 1 -- 172.21.15.6:0/1991703107 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c031870 con 0x7f6e70072520 2024-03-06T07:32:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.549+0000 7f6e6ea00640 1 -- 172.21.15.6:0/1991703107 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c031c40 con 0x7f6e70072520 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 -- 172.21.15.6:0/1991703107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e70072520 msgr2=0x7f6e700cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 --2- 172.21.15.6:0/1991703107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e70072520 0x7f6e700cdfb0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6e7006ec40 tx=0x7f6e5c02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 -- 172.21.15.6:0/1991703107 shutdown_connections 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 --2- 172.21.15.6:0/1991703107 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70071e20 0x7f6e700ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 --2- 172.21.15.6:0/1991703107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e70072520 0x7f6e700cdfb0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 --2- 172.21.15.6:0/1991703107 >> v2:172.21.15.178:3300/0 conn(0x7f6e7015bf20 0x7f6e700acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 -- 172.21.15.6:0/1991703107 >> 172.21.15.6:0/1991703107 conn(0x7f6e7006b3d0 msgr2=0x7f6e7006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 -- 172.21.15.6:0/1991703107 shutdown_connections 2024-03-06T07:32:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.550+0000 7f6e76c00640 1 -- 172.21.15.6:0/1991703107 wait complete. 2024-03-06T07:32:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.552+0000 7f6e76c00640 1 Processor -- start 2024-03-06T07:32:37.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.552+0000 7f6e76c00640 1 -- start start 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e76c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e70071e20 0x7f6e7014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e76c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70072520 0x7f6e7009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e76c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 0x7f6e70154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e76c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700b4b10 con 0x7f6e7015bf20 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e76c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700b4990 con 0x7f6e70071e20 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e76c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700ad280 con 0x7f6e70072520 2024-03-06T07:32:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 0x7f6e70154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e74c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 0x7f6e70154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33838/0 (socket says 172.21.15.6:33838) 2024-03-06T07:32:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e74c00640 1 -- 172.21.15.6:0/795370172 learned_addr learned my addr 172.21.15.6:0/795370172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e6f400640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70072520 0x7f6e7009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e6fe00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e70071e20 0x7f6e7014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.553+0000 7f6e74c00640 1 -- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70072520 msgr2=0x7f6e7009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e74c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70072520 0x7f6e7009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e74c00640 1 -- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e70071e20 msgr2=0x7f6e7014fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e74c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e70071e20 0x7f6e7014fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e74c00640 1 -- 172.21.15.6:0/795370172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e701503c0 con 0x7f6e7015bf20 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e74c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 0x7f6e70154c10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800cd60 tx=0x7f6e6800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68003b90 con 0x7f6e7015bf20 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68003d50 con 0x7f6e7015bf20 2024-03-06T07:32:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68004260 con 0x7f6e7015bf20 2024-03-06T07:32:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e7009b4b0 con 0x7f6e7015bf20 2024-03-06T07:32:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.554+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7009ba70 con 0x7f6e7015bf20 2024-03-06T07:32:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.555+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e7004f330 con 0x7f6e7015bf20 2024-03-06T07:32:37.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.556+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e68004400 con 0x7f6e7015bf20 2024-03-06T07:32:37.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.557+0000 7f6e67e00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6e5006b4d0 0x7f6e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:37.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.557+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6e6808aa60 con 0x7f6e7015bf20 2024-03-06T07:32:37.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.558+0000 7f6e6fe00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6e5006b4d0 0x7f6e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:37.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.559+0000 7f6e6fe00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6e5006b4d0 0x7f6e5006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000a8d0 tx=0x7f6e60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:37.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.561+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e68059b40 con 0x7f6e7015bf20 2024-03-06T07:32:37.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.760+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6e7015a5c0 con 0x7f6e7015bf20 2024-03-06T07:32:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.767+0000 7f6e67e00640 1 -- 172.21.15.6:0/795370172 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e68059440 con 0x7f6e7015bf20 2024-03-06T07:32:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6e5006b4d0 msgr2=0x7f6e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6e5006b4d0 0x7f6e5006d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6e6000a8d0 tx=0x7f6e60008040 comp rx=0 tx=0).stop 2024-03-06T07:32:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 msgr2=0x7f6e70154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 0x7f6e70154c10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800cd60 tx=0x7f6e6800a660 comp rx=0 tx=0).stop 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 shutdown_connections 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6e5006b4d0 0x7f6e5006d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e7015bf20 0x7f6e70154c10 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6e70072520 0x7f6e7009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 --2- 172.21.15.6:0/795370172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e70071e20 0x7f6e7014fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 >> 172.21.15.6:0/795370172 conn(0x7f6e7006b3d0 msgr2=0x7f6e7006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 shutdown_connections 2024-03-06T07:32:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:37.770+0000 7f6e76c00640 1 -- 172.21.15.6:0/795370172 wait complete. 2024-03-06T07:32:37.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:32:38.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.009+0000 7f61e4a00640 1 Processor -- start 2024-03-06T07:32:38.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.009+0000 7f61e4a00640 1 -- start start 2024-03-06T07:32:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61e4a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f61e00cd060 0x7f61e01667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61e4a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e00714d0 con 0x7f61e0145300 2024-03-06T07:32:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61e4a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e00717b0 con 0x7f61e0070f10 2024-03-06T07:32:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61e4a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e0071a90 con 0x7f61e00cd060 2024-03-06T07:32:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61df400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61e0145300 0x7f61e00b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33660/0 (socket says 172.21.15.6:33660) 2024-03-06T07:32:38.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61df400640 1 -- 172.21.15.6:0/656876446 learned_addr learned my addr 172.21.15.6:0/656876446 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:38.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.011+0000 7f61dfe00640 1 --2- 172.21.15.6:0/656876446 >> v2:172.21.15.178:3300/0 conn(0x7f61e00cd060 0x7f61e01667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212069927 0 0) 0x7f61e00714d0 con 0x7f61e0145300 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c4003520 con 0x7f61e0145300 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1532031391 0 0) 0x7f61c4003520 con 0x7f61e0145300 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 >> v2:172.21.15.178:3300/0 conn(0x7f61e00cd060 msgr2=0x7f61e01667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 --2- 172.21.15.6:0/656876446 >> v2:172.21.15.178:3300/0 conn(0x7f61e00cd060 0x7f61e01667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 >> v1:172.21.15.178:6789/0 conn(0x7f61e0070f10 legacy=0x7f61e00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e0071d70 con 0x7f61e0145300 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f61d0001670 con 0x7f61e0145300 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 >> v1:172.21.15.6:6789/0 conn(0x7f61e0145300 legacy=0x7f61e00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c4004800 0x7f61c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c40051c0 0x7f61c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0071a90 con 0x7f61c40039b0 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c400c160 con 0x7f61c4004800 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.012+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c400c360 con 0x7f61c40051c0 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61df400640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61d7e00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c4004800 0x7f61c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61dfe00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c40051c0 0x7f61c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61df400640 1 -- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c40051c0 msgr2=0x7f61c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61df400640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c40051c0 0x7f61c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61df400640 1 -- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c4004800 msgr2=0x7f61c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61df400640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c4004800 0x7f61c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61df400640 1 -- 172.21.15.6:0/656876446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4003520 con 0x7f61c40039b0 2024-03-06T07:32:38.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.013+0000 7f61dfe00640 1 -- 172.21.15.6:0/656876446 reap_dead start 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.014+0000 7f61df400640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61c4003dd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c240 tx=0x7f61c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c999b4bfee6659 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.014+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0007d10 con 0x7f61c40039b0 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.014+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d0030070 con 0x7f61c40039b0 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.014+0000 7f61dea00640 1 -- 172.21.15.6:0/656876446 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0005d70 con 0x7f61c40039b0 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 -- 172.21.15.6:0/656876446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 msgr2=0x7f61c4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61c4003dd0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c240 tx=0x7f61c4005830 comp rx=0 tx=0).stop 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 -- 172.21.15.6:0/656876446 shutdown_connections 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 --2- 172.21.15.6:0/656876446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61c4003dd0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 -- 172.21.15.6:0/656876446 >> 172.21.15.6:0/656876446 conn(0x7f61e006b3d0 msgr2=0x7f61e006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 -- 172.21.15.6:0/656876446 shutdown_connections 2024-03-06T07:32:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.015+0000 7f61e4a00640 1 -- 172.21.15.6:0/656876446 wait complete. 2024-03-06T07:32:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.017+0000 7f61e4a00640 1 Processor -- start 2024-03-06T07:32:38.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.017+0000 7f61e4a00640 1 -- start start 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61e4a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61e006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61e4a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e013db40 0x7f61e013dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61e4a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e0140960 0x7f61e0140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61e4a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00ccad0 con 0x7f61c40039b0 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61e4a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00cc950 con 0x7f61e0140960 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61e4a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00ccc50 con 0x7f61e013db40 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61e006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61e006df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33842/0 (socket says 172.21.15.6:33842) 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.018+0000 7f61df400640 1 -- 172.21.15.6:0/4131279052 learned_addr learned my addr 172.21.15.6:0/4131279052 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61d7e00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e013db40 0x7f61e013dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61df400640 1 -- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e013db40 msgr2=0x7f61e013dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61df400640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e013db40 0x7f61e013dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61df400640 1 -- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e0140960 msgr2=0x7f61e0140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61df400640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e0140960 0x7f61e0140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61df400640 1 -- 172.21.15.6:0/4131279052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e01f4c60 con 0x7f61c40039b0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61df400640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61e006df00 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c360 tx=0x7f61d002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0045070 con 0x7f61c40039b0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d0004bf0 con 0x7f61c40039b0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.019+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0007d10 con 0x7f61c40039b0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.020+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e013cd80 con 0x7f61c40039b0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.020+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e013d340 con 0x7f61c40039b0 2024-03-06T07:32:38.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.021+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e004f330 con 0x7f61c40039b0 2024-03-06T07:32:38.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.022+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61d0007070 con 0x7f61c40039b0 2024-03-06T07:32:38.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.023+0000 7f61d7400640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.023+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f61d00aae00 con 0x7f61c40039b0 2024-03-06T07:32:38.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.023+0000 7f61d7e00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.024+0000 7f61d7e00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61bc06b4d0 0x7f61bc06d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f61c8007fd0 tx=0x7f61c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.027+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61d0079ee0 con 0x7f61c40039b0 2024-03-06T07:32:38.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.299+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f61e004f330 con 0x7f61c40039b0 2024-03-06T07:32:38.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.340+0000 7f61d7400640 1 -- 172.21.15.6:0/4131279052 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v289) v1 ==== 132+0+0 (secure 0 0 0) 0x7f61d00797e0 con 0x7f61c40039b0 2024-03-06T07:32:38.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61bc06b4d0 msgr2=0x7f61bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61bc06b4d0 0x7f61bc06d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f61c8007fd0 tx=0x7f61c8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 msgr2=0x7f61e006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61e006df00 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c360 tx=0x7f61d002e070 comp rx=0 tx=0).stop 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 shutdown_connections 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61bc06b4d0 0x7f61bc06d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e0140960 0x7f61e0140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e013db40 0x7f61e013dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 --2- 172.21.15.6:0/4131279052 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c40039b0 0x7f61e006df00 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 >> 172.21.15.6:0/4131279052 conn(0x7f61e006b3d0 msgr2=0x7f61e00cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 shutdown_connections 2024-03-06T07:32:38.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.343+0000 7f61e4a00640 1 -- 172.21.15.6:0/4131279052 wait complete. 2024-03-06T07:32:38.364 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:32:38.364 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:38.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:38.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.583+0000 7f7e73200640 1 Processor -- start 2024-03-06T07:32:38.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.583+0000 7f7e73200640 1 -- start start 2024-03-06T07:32:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e73200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e740cd060 0x7f7e7415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e73200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e740b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e73200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7415ed10 con 0x7f7e741552b0 2024-03-06T07:32:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e73200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7415eff0 con 0x7f7e740cd060 2024-03-06T07:32:38.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e73200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7415f2d0 con 0x7f7e7415e810 2024-03-06T07:32:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e71400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e740cd060 0x7f7e7415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e71400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e740cd060 0x7f7e7415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33844/0 (socket says 172.21.15.6:33844) 2024-03-06T07:32:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e71400640 1 -- 172.21.15.6:0/674795543 learned_addr learned my addr 172.21.15.6:0/674795543 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.585+0000 7f7e71e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7e741552b0 0x7f7e740cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33664/0 (socket says 172.21.15.6:33664) 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e72800640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e740b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e71400640 1 -- 172.21.15.6:0/674795543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 msgr2=0x7f7e740b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e71400640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e740b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e71400640 1 -- 172.21.15.6:0/674795543 >> v1:172.21.15.6:6789/0 conn(0x7f7e741552b0 legacy=0x7f7e740cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e71400640 1 -- 172.21.15.6:0/674795543 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7415f5b0 con 0x7f7e740cd060 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e72800640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e740b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.586+0000 7f7e70a00640 1 -- 172.21.15.6:0/674795543 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549275588 0 0) 0x7f7e7415ed10 con 0x7f7e741552b0 2024-03-06T07:32:38.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.587+0000 7f7e71400640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7415b1d0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000bb10 tx=0x7f7e6002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25100d39ed9cf72 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.587+0000 7f7e70a00640 1 -- 172.21.15.6:0/674795543 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6003b070 con 0x7f7e740cd060 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e70a00640 1 -- 172.21.15.6:0/674795543 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60036440 con 0x7f7e740cd060 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e70a00640 1 -- 172.21.15.6:0/674795543 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60036810 con 0x7f7e740cd060 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 -- 172.21.15.6:0/674795543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 msgr2=0x7f7e7415b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7415b1d0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000bb10 tx=0x7f7e6002e0c0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 -- 172.21.15.6:0/674795543 shutdown_connections 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e740b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 --2- 172.21.15.6:0/674795543 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7415b1d0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 -- 172.21.15.6:0/674795543 >> 172.21.15.6:0/674795543 conn(0x7f7e7406b3d0 msgr2=0x7f7e7406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 -- 172.21.15.6:0/674795543 shutdown_connections 2024-03-06T07:32:38.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.588+0000 7f7e73200640 1 -- 172.21.15.6:0/674795543 wait complete. 2024-03-06T07:32:38.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.590+0000 7f7e73200640 1 Processor -- start 2024-03-06T07:32:38.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.590+0000 7f7e73200640 1 -- start start 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e73200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7414f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e73200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e741552b0 0x7f7e74144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e73200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e7415c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e73200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7406dd30 con 0x7f7e740cd060 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e73200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740b3760 con 0x7f7e741552b0 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e73200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7406deb0 con 0x7f7e7415e810 2024-03-06T07:32:38.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7414f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7414f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33856/0 (socket says 172.21.15.6:33856) 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 -- 172.21.15.6:0/275380798 learned_addr learned my addr 172.21.15.6:0/275380798 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 -- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 msgr2=0x7f7e7415c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e7415c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 -- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e741552b0 msgr2=0x7f7e74144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71400640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e741552b0 0x7f7e74144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e741552b0 0x7f7e74144710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71e00640 1 -- 172.21.15.6:0/275380798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7415d350 con 0x7f7e740cd060 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.592+0000 7f7e71400640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e741552b0 0x7f7e74144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.593+0000 7f7e71e00640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7414f2e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c410 tx=0x7f7e6400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.593+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e640074e0 con 0x7f7e740cd060 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.593+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64007680 con 0x7f7e740cd060 2024-03-06T07:32:38.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.593+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e64004a00 con 0x7f7e740cd060 2024-03-06T07:32:38.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.593+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7415cab0 con 0x7f7e740cd060 2024-03-06T07:32:38.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.593+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7415d070 con 0x7f7e740cd060 2024-03-06T07:32:38.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.594+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7404f330 con 0x7f7e740cd060 2024-03-06T07:32:38.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.595+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e6401a020 con 0x7f7e740cd060 2024-03-06T07:32:38.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.596+0000 7f7e6a000640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e5406b500 0x7f7e5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:38.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.597+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7e640897a0 con 0x7f7e740cd060 2024-03-06T07:32:38.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.597+0000 7f7e71400640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e5406b500 0x7f7e5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:38.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.598+0000 7f7e71400640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e5406b500 0x7f7e5406d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7e60038000 tx=0x7f7e60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.600+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e64058880 con 0x7f7e740cd060 2024-03-06T07:32:38.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.800+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e7414dbd0 con 0x7f7e740cd060 2024-03-06T07:32:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.802+0000 7f7e6a000640 1 -- 172.21.15.6:0/275380798 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+12951 (secure 0 0 0) 0x7f7e64058180 con 0x7f7e740cd060 2024-03-06T07:32:38.804 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:38.804 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:37.312598+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:6836/1516947936":"2024-03-07T07:32:33.926496+0000","172.21.15.178:6834/1516947936":"2024-03-07T07:32:33.926496+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e5406b500 msgr2=0x7f7e5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e5406b500 0x7f7e5406d9c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7e60038000 tx=0x7f7e60008040 comp rx=0 tx=0).stop 2024-03-06T07:32:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 msgr2=0x7f7e7414f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7414f2e0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c410 tx=0x7f7e6400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 shutdown_connections 2024-03-06T07:32:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e5406b500 0x7f7e5406d9c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e741552b0 0x7f7e74144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e7415e810 0x7f7e7415c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 --2- 172.21.15.6:0/275380798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e740cd060 0x7f7e7414f2e0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 >> 172.21.15.6:0/275380798 conn(0x7f7e7406b3d0 msgr2=0x7f7e74154580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 shutdown_connections 2024-03-06T07:32:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:38.805+0000 7f7e73200640 1 -- 172.21.15.6:0/275380798 wait complete. 2024-03-06T07:32:38.827 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:38.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.045+0000 7f2b83200640 1 Processor -- start 2024-03-06T07:32:39.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.045+0000 7f2b83200640 1 -- start start 2024-03-06T07:32:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.047+0000 7f2b83200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c06d8d0 0x7f2b7c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.047+0000 7f2b83200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b7c0b2080 0x7f2b7c0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.047+0000 7f2b83200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7c06de40 con 0x7f2b7c147590 2024-03-06T07:32:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.047+0000 7f2b83200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b7c147970 con 0x7f2b7c0b2080 2024-03-06T07:32:39.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.047+0000 7f2b83200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b31b0 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.047+0000 7f2b82800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b7c0b2080 0x7f2b7c0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b82800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2b7c0b2080 0x7f2b7c0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33864/0 (socket says 172.21.15.6:33864) 2024-03-06T07:32:39.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b82800640 1 -- 172.21.15.6:0/2634499813 learned_addr learned my addr 172.21.15.6:0/2634499813 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b81400640 1 --2- 172.21.15.6:0/2634499813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c06d8d0 0x7f2b7c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b82800640 1 -- 172.21.15.6:0/2634499813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c06d8d0 msgr2=0x7f2b7c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b82800640 1 --2- 172.21.15.6:0/2634499813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c06d8d0 0x7f2b7c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b82800640 1 -- 172.21.15.6:0/2634499813 >> v1:172.21.15.178:6789/0 conn(0x7f2b7c147590 legacy=0x7f2b7c071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.048+0000 7f2b82800640 1 -- 172.21.15.6:0/2634499813 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c06e120 con 0x7f2b7c0b2080 2024-03-06T07:32:39.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.049+0000 7f2b82800640 1 --2- 172.21.15.6:0/2634499813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0b3640 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009e50 tx=0x7f2b7802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26dd94da9f7334c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b80a00640 1 -- 172.21.15.6:0/2634499813 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7802cb00 con 0x7f2b7c0b2080 2024-03-06T07:32:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b80a00640 1 -- 172.21.15.6:0/2634499813 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7802cca0 con 0x7f2b7c0b2080 2024-03-06T07:32:39.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b80a00640 1 -- 172.21.15.6:0/2634499813 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78034610 con 0x7f2b7c0b2080 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b83200640 1 -- 172.21.15.6:0/2634499813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0b2080 msgr2=0x7f2b7c0b3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b83200640 1 --2- 172.21.15.6:0/2634499813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0b3640 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009e50 tx=0x7f2b7802bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b83200640 1 -- 172.21.15.6:0/2634499813 shutdown_connections 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b83200640 1 --2- 172.21.15.6:0/2634499813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0b3640 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b83200640 1 --2- 172.21.15.6:0/2634499813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c06d8d0 0x7f2b7c0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.050+0000 7f2b83200640 1 -- 172.21.15.6:0/2634499813 >> 172.21.15.6:0/2634499813 conn(0x7f2b7c06b3a0 msgr2=0x7f2b7c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.051+0000 7f2b83200640 1 -- 172.21.15.6:0/2634499813 shutdown_connections 2024-03-06T07:32:39.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.051+0000 7f2b83200640 1 -- 172.21.15.6:0/2634499813 wait complete. 2024-03-06T07:32:39.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.052+0000 7f2b83200640 1 Processor -- start 2024-03-06T07:32:39.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.052+0000 7f2b83200640 1 -- start start 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.053+0000 7f2b83200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 0x7f2b7c1375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.053+0000 7f2b83200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.053+0000 7f2b83200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c147590 0x7f2b7c0c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.053+0000 7f2b83200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b3b80 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.053+0000 7f2b83200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b2b50 con 0x7f2b7c0b2080 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.053+0000 7f2b83200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c0b3d00 con 0x7f2b7c147590 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 0x7f2b7c1375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 0x7f2b7c1375d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33876/0 (socket says 172.21.15.6:33876) 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 -- 172.21.15.6:0/3153722694 learned_addr learned my addr 172.21.15.6:0/3153722694 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 -- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c147590 msgr2=0x7f2b7c0c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c147590 0x7f2b7c0c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 -- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0b2080 msgr2=0x7f2b7c0c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81400640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 -- 172.21.15.6:0/3153722694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7c0c65d0 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81400640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.054+0000 7f2b81e00640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 0x7f2b7c1375d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000c3c0 tx=0x7f2b7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.055+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b700052d0 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.055+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b70005490 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.055+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b70007260 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.055+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c0c5530 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.055+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c0c5af0 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.056+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c04f330 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.057+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b70007400 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.058+0000 7f2b76000640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b6006b4d0 0x7f2b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.058+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2b70089480 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.059+0000 7f2b81400640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b6006b4d0 0x7f2b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.060+0000 7f2b81400640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b6006b4d0 0x7f2b6006d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c1f70e0 tx=0x7f2b6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.063+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b70058560 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.264+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b7c0719b0 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.265+0000 7f2b76000640 1 -- 172.21.15.6:0/3153722694 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b70057e60 con 0x7f2b7c06d8d0 2024-03-06T07:32:39.267 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.267+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b6006b4d0 msgr2=0x7f2b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.267+0000 7f2b83200640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b6006b4d0 0x7f2b6006d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c1f70e0 tx=0x7f2b6c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.267+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 msgr2=0x7f2b7c1375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.267+0000 7f2b83200640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 0x7f2b7c1375d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000c3c0 tx=0x7f2b7000c880 comp rx=0 tx=0).stop 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 shutdown_connections 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2b6006b4d0 0x7f2b6006d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2b7c0b2080 0x7f2b7c0c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2b7c147590 0x7f2b7c0c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 --2- 172.21.15.6:0/3153722694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b7c06d8d0 0x7f2b7c1375d0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 >> 172.21.15.6:0/3153722694 conn(0x7f2b7c06b3a0 msgr2=0x7f2b7c0b01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 shutdown_connections 2024-03-06T07:32:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.268+0000 7f2b83200640 1 -- 172.21.15.6:0/3153722694 wait complete. 2024-03-06T07:32:39.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:32:39.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.500+0000 7ff245a00640 1 Processor -- start 2024-03-06T07:32:39.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.500+0000 7ff245a00640 1 -- start start 2024-03-06T07:32:39.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.502+0000 7ff245a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff24015f310 0x7ff2400cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.502+0000 7ff245a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff2400cfdd0 0x7ff2400d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.502+0000 7ff245a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400d02d0 con 0x7ff2400d2170 2024-03-06T07:32:39.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.502+0000 7ff245a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2400d25e0 con 0x7ff24015f310 2024-03-06T07:32:39.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.502+0000 7ff245a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2400d05b0 con 0x7ff2400cfdd0 2024-03-06T07:32:39.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff24015f310 0x7ff2400cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff24015f310 0x7ff2400cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33884/0 (socket says 172.21.15.6:33884) 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 -- 172.21.15.6:0/2339661445 learned_addr learned my addr 172.21.15.6:0/2339661445 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23e000640 1 --2- 172.21.15.6:0/2339661445 >> v2:172.21.15.178:3300/0 conn(0x7ff2400cfdd0 0x7ff2400d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 -- 172.21.15.6:0/2339661445 >> v2:172.21.15.178:3300/0 conn(0x7ff2400cfdd0 msgr2=0x7ff2400d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 --2- 172.21.15.6:0/2339661445 >> v2:172.21.15.178:3300/0 conn(0x7ff2400cfdd0 0x7ff2400d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 -- 172.21.15.6:0/2339661445 >> v1:172.21.15.178:6789/0 conn(0x7ff2400d2170 legacy=0x7ff2400ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23ea00640 1 -- 172.21.15.6:0/2339661445 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2400d0890 con 0x7ff24015f310 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.503+0000 7ff23e000640 1 --2- 172.21.15.6:0/2339661445 >> v2:172.21.15.178:3300/0 conn(0x7ff2400cfdd0 0x7ff2400d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:39.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.505+0000 7ff23ea00640 1 --2- 172.21.15.6:0/2339661445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24015f310 0x7ff2400cf890 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ff2400730e0 tx=0x7ff23002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bcd734b4b8ba7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.505+0000 7ff23d600640 1 -- 172.21.15.6:0/2339661445 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23002cb20 con 0x7ff24015f310 2024-03-06T07:32:39.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.505+0000 7ff23d600640 1 -- 172.21.15.6:0/2339661445 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23002ccc0 con 0x7ff24015f310 2024-03-06T07:32:39.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.505+0000 7ff23d600640 1 -- 172.21.15.6:0/2339661445 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230034610 con 0x7ff24015f310 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 -- 172.21.15.6:0/2339661445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24015f310 msgr2=0x7ff2400cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 --2- 172.21.15.6:0/2339661445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24015f310 0x7ff2400cf890 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ff2400730e0 tx=0x7ff23002bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 -- 172.21.15.6:0/2339661445 shutdown_connections 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 --2- 172.21.15.6:0/2339661445 >> v2:172.21.15.178:3300/0 conn(0x7ff2400cfdd0 0x7ff2400d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 --2- 172.21.15.6:0/2339661445 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff24015f310 0x7ff2400cf890 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 -- 172.21.15.6:0/2339661445 >> 172.21.15.6:0/2339661445 conn(0x7ff24009aa10 msgr2=0x7ff24009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 -- 172.21.15.6:0/2339661445 shutdown_connections 2024-03-06T07:32:39.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.506+0000 7ff245a00640 1 -- 172.21.15.6:0/2339661445 wait complete. 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.508+0000 7ff245a00640 1 Processor -- start 2024-03-06T07:32:39.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.508+0000 7ff245a00640 1 -- start start 2024-03-06T07:32:39.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff245a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 0x7ff2401f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff245a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2400d2170 0x7ff2400c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff245a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff24015f310 0x7ff24015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff245a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24015f6f0 con 0x7ff2400cfdd0 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff245a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400cc920 con 0x7ff24015f310 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff245a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24015f870 con 0x7ff2400d2170 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 0x7ff2401f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 0x7ff2401f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33886/0 (socket says 172.21.15.6:33886) 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 -- 172.21.15.6:0/1379634087 learned_addr learned my addr 172.21.15.6:0/1379634087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23e000640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2400d2170 0x7ff2400c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 -- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2400d2170 msgr2=0x7ff2400c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23f400640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff24015f310 0x7ff24015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2400d2170 0x7ff2400c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 -- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff24015f310 msgr2=0x7ff24015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:39.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff24015f310 0x7ff24015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23ea00640 1 -- 172.21.15.6:0/1379634087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24014bf50 con 0x7ff2400cfdd0 2024-03-06T07:32:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.509+0000 7ff23f400640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff24015f310 0x7ff24015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.510+0000 7ff23ea00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 0x7ff2401f6a90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ff230036040 tx=0x7ff230004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.510+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230043020 con 0x7ff2400cfdd0 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.510+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23003e450 con 0x7ff2400cfdd0 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.510+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff230009510 con 0x7ff2400cfdd0 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.510+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2401f6fd0 con 0x7ff2400cfdd0 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.510+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2400c76a0 con 0x7ff2400cfdd0 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.511+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2400c7490 con 0x7ff2400cfdd0 2024-03-06T07:32:39.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.513+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff230046020 con 0x7ff2400cfdd0 2024-03-06T07:32:39.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.513+0000 7ff236a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff22006b4d0 0x7ff22006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:39.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.514+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff2300aca10 con 0x7ff2400cfdd0 2024-03-06T07:32:39.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.514+0000 7ff23e000640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff22006b4d0 0x7ff22006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:39.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.515+0000 7ff23e000640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff22006b4d0 0x7ff22006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff240148e30 tx=0x7ff22c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:39.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.519+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff23004d030 con 0x7ff2400cfdd0 2024-03-06T07:32:39.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:39.795+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7ff2400b4660 con 0x7ff2400cfdd0 2024-03-06T07:32:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.353+0000 7ff236a00640 1 -- 172.21.15.6:0/1379634087 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v291) v1 ==== 137+0+0 (secure 0 0 0) 0x7ff23007ba60 con 0x7ff2400cfdd0 2024-03-06T07:32:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff22006b4d0 msgr2=0x7ff22006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff22006b4d0 0x7ff22006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff240148e30 tx=0x7ff22c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 msgr2=0x7ff2401f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 0x7ff2401f6a90 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ff230036040 tx=0x7ff230004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 shutdown_connections 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff22006b4d0 0x7ff22006d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff24015f310 0x7ff24015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff2400d2170 0x7ff2400c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 --2- 172.21.15.6:0/1379634087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff2400cfdd0 0x7ff2401f6a90 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.357+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 >> 172.21.15.6:0/1379634087 conn(0x7ff24009aa10 msgr2=0x7ff2400aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.358+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 shutdown_connections 2024-03-06T07:32:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.358+0000 7ff245a00640 1 -- 172.21.15.6:0/1379634087 wait complete. 2024-03-06T07:32:40.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:32:40.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.606+0000 7f3f60c00640 1 Processor -- start 2024-03-06T07:32:40.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.606+0000 7f3f60c00640 1 -- start start 2024-03-06T07:32:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.608+0000 7f3f60c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f5c15e680 0x7f3f5c06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.608+0000 7f3f60c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5c15eb80 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.608+0000 7f3f60c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5c15ee60 con 0x7f3f5c06da40 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.608+0000 7f3f60c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5c06ea50 con 0x7f3f5c15e680 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f53e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3f5c15e680 0x7f3f5c06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f5a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3f5c0ccad0 0x7f3f5c162100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33676/0 (socket says 172.21.15.6:33676) 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f5a800640 1 -- 172.21.15.6:0/268811793 learned_addr learned my addr 172.21.15.6:0/268811793 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f53e00640 1 -- 172.21.15.6:0/268811793 >> v1:172.21.15.178:6789/0 conn(0x7f3f5c06da40 legacy=0x7f3f5c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f53e00640 1 -- 172.21.15.6:0/268811793 >> v1:172.21.15.6:6789/0 conn(0x7f3f5c0ccad0 legacy=0x7f3f5c162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f53e00640 1 -- 172.21.15.6:0/268811793 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c15f140 con 0x7f3f5c15e680 2024-03-06T07:32:40.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.609+0000 7f3f59400640 1 -- 172.21.15.6:0/268811793 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389375697 0 0) 0x7f3f5c15eb80 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.610+0000 7f3f53e00640 1 --2- 172.21.15.6:0/268811793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c15e680 0x7f3f5c06c980 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3f440097a0 tx=0x7f3f4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6762aad0dfefc40f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f59400640 1 -- 172.21.15.6:0/268811793 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44035020 con 0x7f3f5c15e680 2024-03-06T07:32:40.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f59400640 1 -- 172.21.15.6:0/268811793 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44031830 con 0x7f3f5c15e680 2024-03-06T07:32:40.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f59400640 1 -- 172.21.15.6:0/268811793 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44031c00 con 0x7f3f5c15e680 2024-03-06T07:32:40.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 -- 172.21.15.6:0/268811793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c15e680 msgr2=0x7f3f5c06c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 --2- 172.21.15.6:0/268811793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c15e680 0x7f3f5c06c980 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3f440097a0 tx=0x7f3f4402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 -- 172.21.15.6:0/268811793 shutdown_connections 2024-03-06T07:32:40.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 --2- 172.21.15.6:0/268811793 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c15e680 0x7f3f5c06c980 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 -- 172.21.15.6:0/268811793 >> 172.21.15.6:0/268811793 conn(0x7f3f5c06b3d0 msgr2=0x7f3f5c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 -- 172.21.15.6:0/268811793 shutdown_connections 2024-03-06T07:32:40.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.611+0000 7f3f60c00640 1 -- 172.21.15.6:0/268811793 wait complete. 2024-03-06T07:32:40.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.613+0000 7f3f60c00640 1 Processor -- start 2024-03-06T07:32:40.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.613+0000 7f3f60c00640 1 -- start start 2024-03-06T07:32:40.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f60c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c06da40 0x7f3f5c14fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f60c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 0x7f3f5c0b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f60c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c15e680 0x7f3f5c0b3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f60c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c06cec0 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f60c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c161c70 con 0x7f3f5c15e680 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f60c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c06d040 con 0x7f3f5c06da40 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.614+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 0x7f3f5c0b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 0x7f3f5c0b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33900/0 (socket says 172.21.15.6:33900) 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 -- 172.21.15.6:0/1415495977 learned_addr learned my addr 172.21.15.6:0/1415495977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f59e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c06da40 0x7f3f5c14fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f5a800640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c15e680 0x7f3f5c0b3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 -- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c06da40 msgr2=0x7f3f5c14fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c06da40 0x7f3f5c14fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 -- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c15e680 msgr2=0x7f3f5c0b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c15e680 0x7f3f5c0b3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 -- 172.21.15.6:0/1415495977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c0ca360 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f59e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c06da40 0x7f3f5c14fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f5a800640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c15e680 0x7f3f5c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:40.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.615+0000 7f3f53e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 0x7f3f5c0b3a80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f3f44000c00 tx=0x7f3f440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.616+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44041070 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.616+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f44004230 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.616+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44035040 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.616+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c0c9cd0 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.616+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c14e020 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.617+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c04f330 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.618+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f44038020 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.619+0000 7f3f52000640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3806b4d0 0x7f3f3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:40.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.619+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3f440aba30 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.620+0000 7f3f59e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3806b4d0 0x7f3f3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:40.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.621+0000 7f3f59e00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3806b4d0 0x7f3f3806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c003eb0 tx=0x7f3f4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:40.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.624+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f4407ab10 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.824+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3f5c0cf370 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.831+0000 7f3f52000640 1 -- 172.21.15.6:0/1415495977 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3f4407a410 con 0x7f3f5c0ccad0 2024-03-06T07:32:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.833+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3806b4d0 msgr2=0x7f3f3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.833+0000 7f3f60c00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3806b4d0 0x7f3f3806d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c003eb0 tx=0x7f3f4c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.833+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 msgr2=0x7f3f5c0b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.833+0000 7f3f60c00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 0x7f3f5c0b3a80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f3f44000c00 tx=0x7f3f440046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 shutdown_connections 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3f3806b4d0 0x7f3f3806d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3f5c15e680 0x7f3f5c0b3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3f5c0ccad0 0x7f3f5c0b3a80 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 --2- 172.21.15.6:0/1415495977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3f5c06da40 0x7f3f5c14fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 >> 172.21.15.6:0/1415495977 conn(0x7f3f5c06b3d0 msgr2=0x7f3f5c06b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 shutdown_connections 2024-03-06T07:32:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:40.834+0000 7f3f60c00640 1 -- 172.21.15.6:0/1415495977 wait complete. 2024-03-06T07:32:40.915 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:40.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:41.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:41.158 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:41.158 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:41.158 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:41.218 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:41.218 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:32:41.218 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:41.274 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:41.274 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:41.274 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:41.333 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:41.334 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:32:41.334 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:41.389 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:41.389 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:32:41.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.618+0000 7fc851c00640 1 Processor -- start 2024-03-06T07:32:41.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.618+0000 7fc851c00640 1 -- start start 2024-03-06T07:32:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.620+0000 7fc851c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc84c070f10 0x7fc84c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.620+0000 7fc851c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc84c0cce90 0x7fc84c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.620+0000 7fc851c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c0714d0 con 0x7fc84c145300 2024-03-06T07:32:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.620+0000 7fc851c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84c0cd300 con 0x7fc84c0cce90 2024-03-06T07:32:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.620+0000 7fc851c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84c0717b0 con 0x7fc84c070f10 2024-03-06T07:32:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc84be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc84c145300 0x7fc84c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33682/0 (socket says 172.21.15.6:33682) 2024-03-06T07:32:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc84be00640 1 -- 172.21.15.6:0/3765441304 learned_addr learned my addr 172.21.15.6:0/3765441304 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc851200640 1 --2- 172.21.15.6:0/3765441304 >> v2:172.21.15.6:3300/0 conn(0x7fc84c0cce90 0x7fc84c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc843e00640 1 --2- 172.21.15.6:0/3765441304 >> v2:172.21.15.178:3300/0 conn(0x7fc84c070f10 0x7fc84c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc84b400640 1 -- 172.21.15.6:0/3765441304 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240790628 0 0) 0x7fc84c0714d0 con 0x7fc84c145300 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc84b400640 1 -- 172.21.15.6:0/3765441304 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc830003520 con 0x7fc84c145300 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc851200640 1 -- 172.21.15.6:0/3765441304 >> v2:172.21.15.178:3300/0 conn(0x7fc84c070f10 msgr2=0x7fc84c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc851200640 1 --2- 172.21.15.6:0/3765441304 >> v2:172.21.15.178:3300/0 conn(0x7fc84c070f10 0x7fc84c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc851200640 1 -- 172.21.15.6:0/3765441304 >> v1:172.21.15.6:6789/0 conn(0x7fc84c145300 legacy=0x7fc84c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.621+0000 7fc851200640 1 -- 172.21.15.6:0/3765441304 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c071a90 con 0x7fc84c0cce90 2024-03-06T07:32:41.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.623+0000 7fc851200640 1 --2- 172.21.15.6:0/3765441304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c0cce90 0x7fc84c15c570 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0b5150 tx=0x7fc84402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6b85707a5f82ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.623+0000 7fc84b400640 1 -- 172.21.15.6:0/3765441304 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844039070 con 0x7fc84c0cce90 2024-03-06T07:32:41.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.623+0000 7fc84b400640 1 -- 172.21.15.6:0/3765441304 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84402cbf0 con 0x7fc84c0cce90 2024-03-06T07:32:41.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.623+0000 7fc84b400640 1 -- 172.21.15.6:0/3765441304 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844034440 con 0x7fc84c0cce90 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 -- 172.21.15.6:0/3765441304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c0cce90 msgr2=0x7fc84c15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 --2- 172.21.15.6:0/3765441304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c0cce90 0x7fc84c15c570 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0b5150 tx=0x7fc84402be40 comp rx=0 tx=0).stop 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 -- 172.21.15.6:0/3765441304 shutdown_connections 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 --2- 172.21.15.6:0/3765441304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c0cce90 0x7fc84c15c570 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 --2- 172.21.15.6:0/3765441304 >> v2:172.21.15.178:3300/0 conn(0x7fc84c070f10 0x7fc84c0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 -- 172.21.15.6:0/3765441304 >> 172.21.15.6:0/3765441304 conn(0x7fc84c06b3d0 msgr2=0x7fc84c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 -- 172.21.15.6:0/3765441304 shutdown_connections 2024-03-06T07:32:41.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.624+0000 7fc851c00640 1 -- 172.21.15.6:0/3765441304 wait complete. 2024-03-06T07:32:41.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.626+0000 7fc851c00640 1 Processor -- start 2024-03-06T07:32:41.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.626+0000 7fc851c00640 1 -- start start 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 0x7fc84c136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc84c0cce90 0x7fc84c1f0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc84c145300 0x7fc84c0c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c15cab0 con 0x7fc84c070f10 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c1456e0 con 0x7fc84c0cce90 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c15cc30 con 0x7fc84c145300 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 0x7fc84c136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 0x7fc84c136d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33922/0 (socket says 172.21.15.6:33922) 2024-03-06T07:32:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 -- 172.21.15.6:0/3110823538 learned_addr learned my addr 172.21.15.6:0/3110823538 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc843e00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc84c0cce90 0x7fc84c1f0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc851200640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc84c145300 0x7fc84c0c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 -- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc84c145300 msgr2=0x7fc84c0c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc84c145300 0x7fc84c0c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 -- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc84c0cce90 msgr2=0x7fc84c1f0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc84c0cce90 0x7fc84c1f0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.627+0000 7fc84be00640 1 -- 172.21.15.6:0/3110823538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c0ac570 con 0x7fc84c070f10 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc843e00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc84c0cce90 0x7fc84c1f0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc84be00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 0x7fc84c136d60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00c390 tx=0x7fc83c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c017070 con 0x7fc84c070f10 2024-03-06T07:32:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c005ce0 con 0x7fc84c070f10 2024-03-06T07:32:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0072f0 con 0x7fc84c070f10 2024-03-06T07:32:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c0ac850 con 0x7fc84c070f10 2024-03-06T07:32:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.628+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c0c6040 con 0x7fc84c070f10 2024-03-06T07:32:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.629+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c04f330 con 0x7fc84c070f10 2024-03-06T07:32:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.631+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc83c01a070 con 0x7fc84c070f10 2024-03-06T07:32:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.631+0000 7fc848c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc82806b4d0 0x7fc82806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.632+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc83c089320 con 0x7fc84c070f10 2024-03-06T07:32:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.632+0000 7fc843e00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc82806b4d0 0x7fc82806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.633+0000 7fc843e00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc82806b4d0 0x7fc82806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc84c1f5aa0 tx=0x7fc834008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.636+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc83c058490 con 0x7fc84c070f10 2024-03-06T07:32:41.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.877+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc84c0cc140 con 0x7fc84c070f10 2024-03-06T07:32:41.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.878+0000 7fc848c00640 1 -- 172.21.15.6:0/3110823538 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fc83c057d90 con 0x7fc84c070f10 2024-03-06T07:32:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:41.880 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/2804318881","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":2804318881},{"type":"v1","addr":"172.21.15.178:6837","nonce":2804318881}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6124,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/4096635461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":4096635461},{"type":"v1","addr":"172.21.15.178:6836","nonce":4096635461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/2070873728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2070873728},{"type":"v1","addr":"172.21.15.6:6833","nonce":2070873728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6798,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/1809621641","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":1809621641},{"type":"v1","addr":"172.21.15.6:6835","nonce":1809621641}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48}],"filesystems":[]} 2024-03-06T07:32:41.881 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 55 2024-03-06T07:32:41.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.880+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc82806b4d0 msgr2=0x7fc82806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc82806b4d0 0x7fc82806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc84c1f5aa0 tx=0x7fc834008040 comp rx=0 tx=0).stop 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 msgr2=0x7fc84c136d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 0x7fc84c136d60 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fc83c00c390 tx=0x7fc83c00c850 comp rx=0 tx=0).stop 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 shutdown_connections 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc82806b4d0 0x7fc82806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc84c0cce90 0x7fc84c1f0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc84c145300 0x7fc84c0c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 --2- 172.21.15.6:0/3110823538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc84c070f10 0x7fc84c136d60 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 >> 172.21.15.6:0/3110823538 conn(0x7fc84c06b3d0 msgr2=0x7fc84c1445a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 shutdown_connections 2024-03-06T07:32:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:41.881+0000 7fc851c00640 1 -- 172.21.15.6:0/3110823538 wait complete. 2024-03-06T07:32:41.904 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:41.904 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:32:41.904 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:41.904 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:41.904 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:41.904 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:32:41.904 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:41.905 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:41.905 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:41.905 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:32:41.905 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:41.905 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:41.905 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:32:41.905 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:32:41.906 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:41.906 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:41.922 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:41.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:41.925 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:41.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:41.929 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:41.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:32:41.934 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:41.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:41.972 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:41.972 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:41.979 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:41.979 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:41.980 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:32:41.981 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:41.987 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:41.987 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:41.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:32:42.131 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:32:42.136 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:32:42.143 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:32:42.144 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:32:42.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.174+0000 7f8d63e00640 1 Processor -- start 2024-03-06T07:32:42.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.175+0000 7f8d63e00640 1 -- start start 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d63e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8d6415e070 0x7f8d640d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d63e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d640d0cd0 0x7f8d640cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d63e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640d18c0 con 0x7f8d640cf410 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d63e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d640d1ba0 con 0x7f8d6415e070 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d63e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d1e80 con 0x7f8d640d0cd0 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d5be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8d640cf410 0x7f8d640d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51112/0 (socket says 172.21.15.6:51112) 2024-03-06T07:32:42.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d5be00640 1 -- 172.21.15.6:0/2752870624 learned_addr learned my addr 172.21.15.6:0/2752870624 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:42.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.176+0000 7f8d62a00640 1 --2- 172.21.15.6:0/2752870624 >> v2:172.21.15.178:3300/0 conn(0x7f8d6415e070 0x7f8d640d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d63400640 1 --2- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d640d0cd0 0x7f8d640cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62000640 1 -- 172.21.15.6:0/2752870624 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349002348 0 0) 0x7f8d640d18c0 con 0x7f8d640cf410 2024-03-06T07:32:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62000640 1 -- 172.21.15.6:0/2752870624 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d48003520 con 0x7f8d640cf410 2024-03-06T07:32:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62a00640 1 -- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d640d0cd0 msgr2=0x7f8d640cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62a00640 1 --2- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d640d0cd0 0x7f8d640cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62000640 1 -- 172.21.15.6:0/2752870624 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2433841964 0 0) 0x7f8d48003520 con 0x7f8d640cf410 2024-03-06T07:32:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62a00640 1 -- 172.21.15.6:0/2752870624 >> v1:172.21.15.6:6789/0 conn(0x7f8d640cf410 legacy=0x7f8d640d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.177+0000 7f8d62a00640 1 -- 172.21.15.6:0/2752870624 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640d2160 con 0x7f8d6415e070 2024-03-06T07:32:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.178+0000 7f8d62a00640 1 --2- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d6415e070 0x7f8d640d2960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8d540097a0 tx=0x7f8d5402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9618bd3c0c356c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.178+0000 7f8d62000640 1 -- 172.21.15.6:0/2752870624 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54037070 con 0x7f8d6415e070 2024-03-06T07:32:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.178+0000 7f8d62000640 1 -- 172.21.15.6:0/2752870624 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d54030510 con 0x7f8d6415e070 2024-03-06T07:32:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.178+0000 7f8d62000640 1 -- 172.21.15.6:0/2752870624 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c001670 con 0x7f8d6415e070 2024-03-06T07:32:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 -- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d6415e070 msgr2=0x7f8d640d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 --2- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d6415e070 0x7f8d640d2960 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8d540097a0 tx=0x7f8d5402b790 comp rx=0 tx=0).stop 2024-03-06T07:32:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 -- 172.21.15.6:0/2752870624 shutdown_connections 2024-03-06T07:32:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 --2- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d640d0cd0 0x7f8d640cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 --2- 172.21.15.6:0/2752870624 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d6415e070 0x7f8d640d2960 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 -- 172.21.15.6:0/2752870624 >> 172.21.15.6:0/2752870624 conn(0x7f8d6408d2a0 msgr2=0x7f8d640d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 -- 172.21.15.6:0/2752870624 shutdown_connections 2024-03-06T07:32:42.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.179+0000 7f8d63e00640 1 -- 172.21.15.6:0/2752870624 wait complete. 2024-03-06T07:32:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.181+0000 7f8d63e00640 1 Processor -- start 2024-03-06T07:32:42.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.181+0000 7f8d63e00640 1 -- start start 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 0x7f8d641582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d640d0cd0 0x7f8d640b7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d6415e070 0x7f8d640b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d4730 con 0x7f8d640cf410 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640d45b0 con 0x7f8d640d0cd0 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d640b3630 con 0x7f8d6415e070 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d62a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 0x7f8d641582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d62a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 0x7f8d641582d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37862/0 (socket says 172.21.15.6:37862) 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d62a00640 1 -- 172.21.15.6:0/3787988487 learned_addr learned my addr 172.21.15.6:0/3787988487 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d63400640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d6415e070 0x7f8d640b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d5be00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d640d0cd0 0x7f8d640b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d62a00640 1 -- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d6415e070 msgr2=0x7f8d640b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.182+0000 7f8d62a00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d6415e070 0x7f8d640b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d62a00640 1 -- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d640d0cd0 msgr2=0x7f8d640b7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d62a00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d640d0cd0 0x7f8d640b7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d62a00640 1 -- 172.21.15.6:0/3787988487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640b7ab0 con 0x7f8d640cf410 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d63400640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d6415e070 0x7f8d640b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d5be00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d640d0cd0 0x7f8d640b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d62a00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 0x7f8d641582d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8d5402b7d0 tx=0x7f8d540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54042070 con 0x7f8d640cf410 2024-03-06T07:32:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5403e870 con 0x7f8d640cf410 2024-03-06T07:32:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d640cdac0 con 0x7f8d640cf410 2024-03-06T07:32:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54037040 con 0x7f8d640cf410 2024-03-06T07:32:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.183+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d640b6d70 con 0x7f8d640cf410 2024-03-06T07:32:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.184+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6404f330 con 0x7f8d640cf410 2024-03-06T07:32:42.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.185+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d54045070 con 0x7f8d640cf410 2024-03-06T07:32:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.187+0000 7f8d5aa00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.187+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8d540abd40 con 0x7f8d640cf410 2024-03-06T07:32:42.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.187+0000 7f8d5be00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:42.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.188+0000 7f8d5be00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d4006b4d0 0x7f8d4006d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8d640b4b60 tx=0x7f8d4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:42.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.192+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d5407b070 con 0x7f8d640cf410 2024-03-06T07:32:42.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:42.411+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8d640c75c0 con 0x7f8d640cf410 2024-03-06T07:32:43.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.399+0000 7f8d5aa00640 1 -- 172.21.15.6:0/3787988487 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8d5407a970 con 0x7f8d640cf410 2024-03-06T07:32:43.401 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:32:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d4006b4d0 msgr2=0x7f8d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d4006b4d0 0x7f8d4006d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8d640b4b60 tx=0x7f8d4c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 msgr2=0x7f8d641582d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 0x7f8d641582d0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f8d5402b7d0 tx=0x7f8d540046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 shutdown_connections 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d640d0cd0 0x7f8d640b7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d6415e070 0x7f8d640b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 --2- 172.21.15.6:0/3787988487 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d640cf410 0x7f8d641582d0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 >> 172.21.15.6:0/3787988487 conn(0x7f8d6408d2a0 msgr2=0x7f8d6406b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.403+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 shutdown_connections 2024-03-06T07:32:43.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.404+0000 7f8d63e00640 1 -- 172.21.15.6:0/3787988487 wait complete. 2024-03-06T07:32:43.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:32:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.631+0000 7f9184e00640 1 Processor -- start 2024-03-06T07:32:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.631+0000 7f9184e00640 1 -- start start 2024-03-06T07:32:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.633+0000 7f9184e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 0x7f918015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.633+0000 7f9184e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f91800cdc20 0x7f91800b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.633+0000 7f9184e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918015cc40 con 0x7f91800af1a0 2024-03-06T07:32:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.633+0000 7f9184e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918015cf20 con 0x7f91800cdc20 2024-03-06T07:32:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.633+0000 7f9184e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918015d200 con 0x7f91800af580 2024-03-06T07:32:43.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f917de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f91800af1a0 0x7f91800b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51118/0 (socket says 172.21.15.6:51118) 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f917de00640 1 -- 172.21.15.6:0/4195530598 learned_addr learned my addr 172.21.15.6:0/4195530598 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f917e800640 1 --2- 172.21.15.6:0/4195530598 >> v2:172.21.15.178:3300/0 conn(0x7f91800cdc20 0x7f91800b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f9177e00640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 0x7f918015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587781335 0 0) 0x7f918015cc40 con 0x7f91800af1a0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9164003520 con 0x7f91800af1a0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.634+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3409331944 0 0) 0x7f9164003520 con 0x7f91800af1a0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 msgr2=0x7f918015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 0x7f918015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 >> v2:172.21.15.178:3300/0 conn(0x7f91800cdc20 msgr2=0x7f91800b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 --2- 172.21.15.6:0/4195530598 >> v2:172.21.15.178:3300/0 conn(0x7f91800cdc20 0x7f91800b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918015d4e0 con 0x7f91800af1a0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9170001670 con 0x7f91800af1a0 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 >> v1:172.21.15.6:6789/0 conn(0x7f91800af1a0 legacy=0x7f91800b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 0x7f9164003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9164003fa0 0x7f9164004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f91640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918015cf20 con 0x7f9164004d40 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918015d200 con 0x7f9164003fa0 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916400ba70 con 0x7f91800af580 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917e800640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f91640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f917de00640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 0x7f9164003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.635+0000 7f9177e00640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9164003fa0 0x7f9164004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.636+0000 7f917e800640 1 -- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 msgr2=0x7f9164003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.636+0000 7f917e800640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f91800af580 0x7f9164003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.636+0000 7f917e800640 1 -- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9164003fa0 msgr2=0x7f9164004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.636+0000 7f917e800640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9164003fa0 0x7f9164004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.636+0000 7f917e800640 1 -- 172.21.15.6:0/4195530598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164003520 con 0x7f9164004d40 2024-03-06T07:32:43.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.637+0000 7f917e800640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f91640051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9178009fc0 tx=0x7f9164005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69692cc54a517450 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.637+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178039070 con 0x7f9164004d40 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.637+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917802cc60 con 0x7f9164004d40 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.637+0000 7f917d400640 1 -- 172.21.15.6:0/4195530598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178009c10 con 0x7f9164004d40 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f9184e00640 1 -- 172.21.15.6:0/4195530598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 msgr2=0x7f91640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f9184e00640 1 --2- 172.21.15.6:0/4195530598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f91640051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9178009fc0 tx=0x7f9164005830 comp rx=0 tx=0).stop 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f917e800640 1 -- 172.21.15.6:0/4195530598 reap_dead start 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f9184e00640 1 -- 172.21.15.6:0/4195530598 shutdown_connections 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f9184e00640 1 -- 172.21.15.6:0/4195530598 >> 172.21.15.6:0/4195530598 conn(0x7f918006afb0 msgr2=0x7f918006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f9184e00640 1 -- 172.21.15.6:0/4195530598 shutdown_connections 2024-03-06T07:32:43.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.638+0000 7f9184e00640 1 -- 172.21.15.6:0/4195530598 wait complete. 2024-03-06T07:32:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.640+0000 7f9184e00640 1 Processor -- start 2024-03-06T07:32:43.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.641+0000 7f9184e00640 1 -- start start 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.641+0000 7f9184e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f918009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.641+0000 7f9184e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f918009cce0 0x7f918014e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.641+0000 7f9184e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f918014e840 0x7f918014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f9184e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918015c990 con 0x7f9164004d40 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f9184e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918015c810 con 0x7f918014e840 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f9184e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91800b3310 con 0x7f918009cce0 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f918009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f918009c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37880/0 (socket says 172.21.15.6:37880) 2024-03-06T07:32:43.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 -- 172.21.15.6:0/506951855 learned_addr learned my addr 172.21.15.6:0/506951855 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 -- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f918009cce0 msgr2=0x7f918014e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f918009cce0 0x7f918014e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 -- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f918014e840 msgr2=0x7f918014ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f918014e840 0x7f918014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.642+0000 7f917de00640 1 -- 172.21.15.6:0/506951855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918009d480 con 0x7f9164004d40 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.643+0000 7f917de00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f918009c7a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f9170001eb0 tx=0x7f917000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.643+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917000c850 con 0x7f9164004d40 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.643+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917000c9f0 con 0x7f9164004d40 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.643+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917000cbe0 con 0x7f9164004d40 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.643+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918009d760 con 0x7f9164004d40 2024-03-06T07:32:43.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.643+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918014c550 con 0x7f9164004d40 2024-03-06T07:32:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.644+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9180069510 con 0x7f9164004d40 2024-03-06T07:32:43.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.645+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f917001a020 con 0x7f9164004d40 2024-03-06T07:32:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.646+0000 7f9176000640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f915c06b4d0 0x7f915c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.646+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9170089630 con 0x7f9164004d40 2024-03-06T07:32:43.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.646+0000 7f9177e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f915c06b4d0 0x7f915c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:43.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.648+0000 7f9177e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f915c06b4d0 0x7f915c06d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f918015af90 tx=0x7f9168008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:43.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.651+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9170021020 con 0x7f9164004d40 2024-03-06T07:32:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.922+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9180074460 con 0x7f9164004d40 2024-03-06T07:32:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.922+0000 7f9176000640 1 -- 172.21.15.6:0/506951855 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f91700192c0 con 0x7f9164004d40 2024-03-06T07:32:43.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:43.925 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:32:43.925 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.926 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:32:43.927 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:43.928 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:32:43.929 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.931 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:32:43.932 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.934 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.936 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:32:43.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:32:43.938 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f915c06b4d0 msgr2=0x7f915c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f915c06b4d0 0x7f915c06d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f918015af90 tx=0x7f9168008040 comp rx=0 tx=0).stop 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 msgr2=0x7f918009c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f918009c7a0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f9170001eb0 tx=0x7f917000ed50 comp rx=0 tx=0).stop 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 shutdown_connections 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f915c06b4d0 0x7f915c06d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f918014e840 0x7f918014ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f918009cce0 0x7f918014e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 --2- 172.21.15.6:0/506951855 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9164004d40 0x7f918009c7a0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.925+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 >> 172.21.15.6:0/506951855 conn(0x7f918006afb0 msgr2=0x7f91800adaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.926+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 shutdown_connections 2024-03-06T07:32:43.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:43.926+0000 7f9184e00640 1 -- 172.21.15.6:0/506951855 wait complete. 2024-03-06T07:32:43.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:44.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.167+0000 7f54b3200640 1 Processor -- start 2024-03-06T07:32:44.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.167+0000 7f54b3200640 1 -- start start 2024-03-06T07:32:44.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.169+0000 7f54b3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 0x7f54b40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.169+0000 7f54b3200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40714d0 con 0x7f54b4145300 2024-03-06T07:32:44.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.169+0000 7f54b3200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40717b0 con 0x7f54b40cce90 2024-03-06T07:32:44.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.169+0000 7f54b3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40cd300 con 0x7f54b4070f10 2024-03-06T07:32:44.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.169+0000 7f54b1e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54b4145300 0x7f54b40b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51126/0 (socket says 172.21.15.6:51126) 2024-03-06T07:32:44.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.169+0000 7f54b1e00640 1 -- 172.21.15.6:0/3611899384 learned_addr learned my addr 172.21.15.6:0/3611899384 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:44.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b1400640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 0x7f54b40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503526052 0 0) 0x7f54b40714d0 con 0x7f54b4145300 2024-03-06T07:32:44.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5498003520 con 0x7f54b4145300 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 227540033 0 0) 0x7f5498003520 con 0x7f54b4145300 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 msgr2=0x7f54b40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 0x7f54b40cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 >> v1:172.21.15.178:6789/0 conn(0x7f54b40cce90 legacy=0x7f54b415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4071a90 con 0x7f54b4145300 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f54a4001670 con 0x7f54b4145300 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 >> v1:172.21.15.6:6789/0 conn(0x7f54b4145300 legacy=0x7f54b40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f5498003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 0x7f5498004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5498004d40 0x7f54980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40cd300 con 0x7f54980039e0 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549800baa0 con 0x7f5498004d40 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.170+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549800bca0 con 0x7f54b4070f10 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1e00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f5498003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1400640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 0x7f5498004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1e00640 1 -- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 msgr2=0x7f5498004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1e00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b4070f10 0x7f5498004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1e00640 1 -- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5498004d40 msgr2=0x7f54980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1e00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5498004d40 0x7f54980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.171+0000 7f54b1e00640 1 -- 172.21.15.6:0/3611899384 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498003520 con 0x7f54980039e0 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.172+0000 7f54b1e00640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f5498003e00 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c0e0 tx=0x7f5498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6cdf9bf3e59f88f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.172+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4007c40 con 0x7f54980039e0 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4030070 con 0x7f54980039e0 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b0a00640 1 -- 172.21.15.6:0/3611899384 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4005d30 con 0x7f54980039e0 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b3200640 1 -- 172.21.15.6:0/3611899384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 msgr2=0x7f5498003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b3200640 1 --2- 172.21.15.6:0/3611899384 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f5498003e00 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c0e0 tx=0x7f5498005830 comp rx=0 tx=0).stop 2024-03-06T07:32:44.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b2800640 1 -- 172.21.15.6:0/3611899384 reap_dead start 2024-03-06T07:32:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b3200640 1 -- 172.21.15.6:0/3611899384 shutdown_connections 2024-03-06T07:32:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b3200640 1 -- 172.21.15.6:0/3611899384 >> 172.21.15.6:0/3611899384 conn(0x7f54b406b3d0 msgr2=0x7f54b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b3200640 1 -- 172.21.15.6:0/3611899384 shutdown_connections 2024-03-06T07:32:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.173+0000 7f54b3200640 1 -- 172.21.15.6:0/3611899384 wait complete. 2024-03-06T07:32:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.175+0000 7f54b3200640 1 Processor -- start 2024-03-06T07:32:44.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.175+0000 7f54b3200640 1 -- start start 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.176+0000 7f54b3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f54b40abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.176+0000 7f54b3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b415b880 0x7f54b40ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.176+0000 7f54b3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54b41e59a0 0x7f54b41e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.176+0000 7f54b3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40cc950 con 0x7f54980039e0 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.176+0000 7f54b3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b41456e0 con 0x7f54b41e59a0 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.176+0000 7f54b3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b40ccad0 con 0x7f54b415b880 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f54b40abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f54b40abdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37886/0 (socket says 172.21.15.6:37886) 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 -- 172.21.15.6:0/637208632 learned_addr learned my addr 172.21.15.6:0/637208632 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 -- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b415b880 msgr2=0x7f54b40ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b415b880 0x7f54b40ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 -- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54b41e59a0 msgr2=0x7f54b41e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54b41e59a0 0x7f54b41e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 -- 172.21.15.6:0/637208632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4141ad0 con 0x7f54980039e0 2024-03-06T07:32:44.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.177+0000 7f54b1e00640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f54b40abdb0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f54a402c480 tx=0x7f54b4071980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.178+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4045070 con 0x7f54980039e0 2024-03-06T07:32:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.178+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a4000ed0 con 0x7f54980039e0 2024-03-06T07:32:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.178+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4007c40 con 0x7f54980039e0 2024-03-06T07:32:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.178+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b40a9830 con 0x7f54980039e0 2024-03-06T07:32:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.178+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b40aed60 con 0x7f54980039e0 2024-03-06T07:32:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.179+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b404f330 con 0x7f54980039e0 2024-03-06T07:32:44.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.180+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54a4007070 con 0x7f54980039e0 2024-03-06T07:32:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.181+0000 7f54aa000640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f549406b500 0x7f549406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.181+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f54a40aaf00 con 0x7f54980039e0 2024-03-06T07:32:44.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.181+0000 7f54b1400640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f549406b500 0x7f549406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.182+0000 7f54b1400640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f549406b500 0x7f549406d9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f54b415f4b0 tx=0x7f54a0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.185+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54a4079af0 con 0x7f54980039e0 2024-03-06T07:32:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.388+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54b415be10 con 0x7f54980039e0 2024-03-06T07:32:44.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.390+0000 7f54aa000640 1 -- 172.21.15.6:0/637208632 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12820 (secure 0 0 0) 0x7f54a407db90 con 0x7f54980039e0 2024-03-06T07:32:44.392 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:44.392 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":89,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:43.397127+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f549406b500 msgr2=0x7f549406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f549406b500 0x7f549406d9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f54b415f4b0 tx=0x7f54a0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 msgr2=0x7f54b40abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f54b40abdb0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f54a402c480 tx=0x7f54b4071980 comp rx=0 tx=0).stop 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 shutdown_connections 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f549406b500 0x7f549406d9c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54b41e59a0 0x7f54b41e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54b415b880 0x7f54b40ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 --2- 172.21.15.6:0/637208632 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54980039e0 0x7f54b40abdb0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 >> 172.21.15.6:0/637208632 conn(0x7f54b406b3d0 msgr2=0x7f54b40b49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 shutdown_connections 2024-03-06T07:32:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.393+0000 7f54b3200640 1 -- 172.21.15.6:0/637208632 wait complete. 2024-03-06T07:32:44.416 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:44.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:44.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.640+0000 7f5f06200640 1 Processor -- start 2024-03-06T07:32:44.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.640+0000 7f5f06200640 1 -- start start 2024-03-06T07:32:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.641+0000 7f5f06200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f000b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.641+0000 7f5f06200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f000713f0 0x7f5f00071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.641+0000 7f5f06200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f00154ce0 con 0x7f5f00154900 2024-03-06T07:32:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.641+0000 7f5f06200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f00071d60 con 0x7f5f000713f0 2024-03-06T07:32:44.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.641+0000 7f5f06200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000b2bf0 con 0x7f5f00072040 2024-03-06T07:32:44.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f04e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5f00154900 0x7f5f000cf1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51142/0 (socket says 172.21.15.6:51142) 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f04e00640 1 -- 172.21.15.6:0/3221239441 learned_addr learned my addr 172.21.15.6:0/3221239441 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f05800640 1 --2- 172.21.15.6:0/3221239441 >> v2:172.21.15.6:3300/0 conn(0x7f5f000713f0 0x7f5f00071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5effe00640 1 --2- 172.21.15.6:0/3221239441 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f000b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f05800640 1 -- 172.21.15.6:0/3221239441 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 msgr2=0x7f5f000b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f05800640 1 --2- 172.21.15.6:0/3221239441 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f000b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f05800640 1 -- 172.21.15.6:0/3221239441 >> v1:172.21.15.6:6789/0 conn(0x7f5f00154900 legacy=0x7f5f000cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.642+0000 7f5f05800640 1 -- 172.21.15.6:0/3221239441 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000b2ed0 con 0x7f5f000713f0 2024-03-06T07:32:44.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.644+0000 7f5f05800640 1 --2- 172.21.15.6:0/3221239441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f000713f0 0x7f5f00071820 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009dc0 tx=0x7f5ef802bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c5d3168636f3c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.644+0000 7f5eff400640 1 -- 172.21.15.6:0/3221239441 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8039070 con 0x7f5f000713f0 2024-03-06T07:32:44.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.644+0000 7f5eff400640 1 -- 172.21.15.6:0/3221239441 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef802cba0 con 0x7f5f000713f0 2024-03-06T07:32:44.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.644+0000 7f5eff400640 1 -- 172.21.15.6:0/3221239441 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8034440 con 0x7f5f000713f0 2024-03-06T07:32:44.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 -- 172.21.15.6:0/3221239441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f000713f0 msgr2=0x7f5f00071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 --2- 172.21.15.6:0/3221239441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f000713f0 0x7f5f00071820 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009dc0 tx=0x7f5ef802bdf0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 -- 172.21.15.6:0/3221239441 shutdown_connections 2024-03-06T07:32:44.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 --2- 172.21.15.6:0/3221239441 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f000713f0 0x7f5f00071820 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 --2- 172.21.15.6:0/3221239441 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f000b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 -- 172.21.15.6:0/3221239441 >> 172.21.15.6:0/3221239441 conn(0x7f5f0006b3a0 msgr2=0x7f5f0006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 -- 172.21.15.6:0/3221239441 shutdown_connections 2024-03-06T07:32:44.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.645+0000 7f5f06200640 1 -- 172.21.15.6:0/3221239441 wait complete. 2024-03-06T07:32:44.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.647+0000 7f5f06200640 1 Processor -- start 2024-03-06T07:32:44.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.647+0000 7f5f06200640 1 -- start start 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f06200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f000713f0 0x7f5f001424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f06200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f0014e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f06200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 0x7f5f0015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f06200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000b3cf0 con 0x7f5f00154900 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f06200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000b3b70 con 0x7f5f000713f0 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f06200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000b1b40 con 0x7f5f00072040 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 0x7f5f0015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 0x7f5f0015be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37906/0 (socket says 172.21.15.6:37906) 2024-03-06T07:32:44.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 -- 172.21.15.6:0/3362849762 learned_addr learned my addr 172.21.15.6:0/3362849762 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 -- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 msgr2=0x7f5f0014e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f0014e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 -- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f000713f0 msgr2=0x7f5f001424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f000713f0 0x7f5f001424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.648+0000 7f5f05800640 1 -- 172.21.15.6:0/3362849762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0014ff80 con 0x7f5f00154900 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.649+0000 7f5f05800640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 0x7f5f0015be30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8037040 tx=0x7f5ef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.649+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8039050 con 0x7f5f00154900 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.649+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef8045070 con 0x7f5f00154900 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.649+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f00150260 con 0x7f5f00154900 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.649+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef803e850 con 0x7f5f00154900 2024-03-06T07:32:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.649+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f00150820 con 0x7f5f00154900 2024-03-06T07:32:44.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.650+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0004f330 con 0x7f5f00154900 2024-03-06T07:32:44.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.651+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ef803e9f0 con 0x7f5f00154900 2024-03-06T07:32:44.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.652+0000 7f5efcc00640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ee006b4d0 0x7f5ee006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:44.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.652+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5ef80acf40 con 0x7f5f00154900 2024-03-06T07:32:44.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.653+0000 7f5f04e00640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ee006b4d0 0x7f5ee006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:44.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.654+0000 7f5f04e00640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ee006b4d0 0x7f5ee006d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4000f60 tx=0x7f5ef4007a80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:44.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.657+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ef807c190 con 0x7f5f00154900 2024-03-06T07:32:44.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.847+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f0006c910 con 0x7f5f00154900 2024-03-06T07:32:44.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.848+0000 7f5efcc00640 1 -- 172.21.15.6:0/3362849762 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ef807ba90 con 0x7f5f00154900 2024-03-06T07:32:44.850 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ee006b4d0 msgr2=0x7f5ee006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ee006b4d0 0x7f5ee006d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4000f60 tx=0x7f5ef4007a80 comp rx=0 tx=0).stop 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 msgr2=0x7f5f0015be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 0x7f5f0015be30 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8037040 tx=0x7f5ef8004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 shutdown_connections 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5ee006b4d0 0x7f5ee006d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.850+0000 7f5f06200640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f00154900 0x7f5f0015be30 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.851+0000 7f5f06200640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f00072040 0x7f5f0014e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.851+0000 7f5f06200640 1 --2- 172.21.15.6:0/3362849762 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f000713f0 0x7f5f001424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:44.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.851+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 >> 172.21.15.6:0/3362849762 conn(0x7f5f0006b3a0 msgr2=0x7f5f001534e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:44.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.851+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 shutdown_connections 2024-03-06T07:32:44.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:44.851+0000 7f5f06200640 1 -- 172.21.15.6:0/3362849762 wait complete. 2024-03-06T07:32:44.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:32:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.075+0000 7f0016e00640 1 Processor -- start 2024-03-06T07:32:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.075+0000 7f0016e00640 1 -- start start 2024-03-06T07:32:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.077+0000 7f0016e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0010145300 0x7f00100b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.077+0000 7f0016e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010070f10 0x7f00100cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.077+0000 7f0016e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00100714d0 con 0x7f00100cce90 2024-03-06T07:32:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.077+0000 7f0016e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00100cd300 con 0x7f0010145300 2024-03-06T07:32:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.077+0000 7f0016e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00100717b0 con 0x7f0010070f10 2024-03-06T07:32:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0015000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010070f10 0x7f00100cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0015a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0010145300 0x7f00100b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0016400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f00100cce90 0x7f00100cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:50172/0 (socket says 172.21.15.6:50172) 2024-03-06T07:32:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0016400640 1 -- 172.21.15.6:0/3531783864 learned_addr learned my addr 172.21.15.6:0/3531783864 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:45.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0015000640 1 -- 172.21.15.6:0/3531783864 >> v2:172.21.15.178:3300/0 conn(0x7f0010145300 msgr2=0x7f00100b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0015000640 1 --2- 172.21.15.6:0/3531783864 >> v2:172.21.15.178:3300/0 conn(0x7f0010145300 0x7f00100b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0015000640 1 -- 172.21.15.6:0/3531783864 >> v1:172.21.15.178:6789/0 conn(0x7f00100cce90 legacy=0x7f00100cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f000fe00640 1 -- 172.21.15.6:0/3531783864 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359273661 0 0) 0x7f00100714d0 con 0x7f00100cce90 2024-03-06T07:32:45.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.078+0000 7f0015000640 1 -- 172.21.15.6:0/3531783864 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0010071a90 con 0x7f0010070f10 2024-03-06T07:32:45.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.080+0000 7f0015000640 1 --2- 172.21.15.6:0/3531783864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010070f10 0x7f00100cc950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efffc0097a0 tx=0x7efffc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c9d09ce0186052 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f000fe00640 1 -- 172.21.15.6:0/3531783864 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc035020 con 0x7f0010070f10 2024-03-06T07:32:45.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f000fe00640 1 -- 172.21.15.6:0/3531783864 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc031830 con 0x7f0010070f10 2024-03-06T07:32:45.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f000fe00640 1 -- 172.21.15.6:0/3531783864 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc031c00 con 0x7f0010070f10 2024-03-06T07:32:45.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f0016e00640 1 -- 172.21.15.6:0/3531783864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010070f10 msgr2=0x7f00100cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f0016e00640 1 --2- 172.21.15.6:0/3531783864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010070f10 0x7f00100cc950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efffc0097a0 tx=0x7efffc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f0016e00640 1 -- 172.21.15.6:0/3531783864 shutdown_connections 2024-03-06T07:32:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f0016e00640 1 --2- 172.21.15.6:0/3531783864 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010070f10 0x7f00100cc950 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f0016e00640 1 --2- 172.21.15.6:0/3531783864 >> v2:172.21.15.178:3300/0 conn(0x7f0010145300 0x7f00100b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.081+0000 7f0016e00640 1 -- 172.21.15.6:0/3531783864 >> 172.21.15.6:0/3531783864 conn(0x7f001006b3d0 msgr2=0x7f001006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.082+0000 7f0016e00640 1 -- 172.21.15.6:0/3531783864 shutdown_connections 2024-03-06T07:32:45.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.082+0000 7f0016e00640 1 -- 172.21.15.6:0/3531783864 wait complete. 2024-03-06T07:32:45.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.083+0000 7f0016e00640 1 Processor -- start 2024-03-06T07:32:45.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.084+0000 7f0016e00640 1 -- start start 2024-03-06T07:32:45.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.084+0000 7f0016e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0010070f10 0x7f0010142870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0016e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 0x7f00101e6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0016e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010145300 0x7f00100b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0016e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00101456e0 con 0x7f00100cce90 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0016e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00100b4ee0 con 0x7f0010070f10 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0016e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001015d580 con 0x7f0010145300 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0015000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 0x7f00101e6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0015000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 0x7f00101e6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37916/0 (socket says 172.21.15.6:37916) 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0015000640 1 -- 172.21.15.6:0/3083994480 learned_addr learned my addr 172.21.15.6:0/3083994480 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0015a00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0010070f10 0x7f0010142870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.085+0000 7f0016400640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010145300 0x7f00100b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0015000640 1 -- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010145300 msgr2=0x7f00100b1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0015000640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010145300 0x7f00100b1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0015000640 1 -- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0010070f10 msgr2=0x7f0010142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0015000640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0010070f10 0x7f0010142870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0015000640 1 -- 172.21.15.6:0/3083994480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00101e6980 con 0x7f00100cce90 2024-03-06T07:32:45.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0015000640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 0x7f00101e6440 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7efffc000c00 tx=0x7efffc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc035070 con 0x7f00100cce90 2024-03-06T07:32:45.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc004230 con 0x7f00100cce90 2024-03-06T07:32:45.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00101f6f60 con 0x7f00100cce90 2024-03-06T07:32:45.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.086+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc03a630 con 0x7f00100cce90 2024-03-06T07:32:45.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.087+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00101f7520 con 0x7f00100cce90 2024-03-06T07:32:45.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.088+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0010067830 con 0x7f00100cce90 2024-03-06T07:32:45.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.089+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efffc038020 con 0x7f00100cce90 2024-03-06T07:32:45.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.090+0000 7f000d600640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efff006b4d0 0x7efff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.090+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efffc0abb90 con 0x7f00100cce90 2024-03-06T07:32:45.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.091+0000 7f0015a00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efff006b4d0 0x7efff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.092+0000 7f0015a00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efff006b4d0 0x7efff006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f00100b2800 tx=0x7f0004008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.095+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0010067830 con 0x7f00100cce90 2024-03-06T07:32:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.299+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f001004f330 con 0x7f00100cce90 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.300+0000 7f000d600640 1 -- 172.21.15.6:0/3083994480 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7efffc040070 con 0x7f00100cce90 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:32:45.302 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efff006b4d0 msgr2=0x7efff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efff006b4d0 0x7efff006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f00100b2800 tx=0x7f0004008040 comp rx=0 tx=0).stop 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 msgr2=0x7f00101e6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 0x7f00101e6440 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7efffc000c00 tx=0x7efffc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 shutdown_connections 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efff006b4d0 0x7efff006d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0010145300 0x7f00100b1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f00100cce90 0x7f00101e6440 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 --2- 172.21.15.6:0/3083994480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0010070f10 0x7f0010142870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.302+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 >> 172.21.15.6:0/3083994480 conn(0x7f001006b3d0 msgr2=0x7f001015d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.303+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 shutdown_connections 2024-03-06T07:32:45.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.303+0000 7f0016e00640 1 -- 172.21.15.6:0/3083994480 wait complete. 2024-03-06T07:32:45.324 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:32:45.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:32:45.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.542+0000 7fb89e000640 1 Processor -- start 2024-03-06T07:32:45.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.542+0000 7fb89e000640 1 -- start start 2024-03-06T07:32:45.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb89e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb898144320 0x7fb89806daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb89e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb8980ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb89e000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8980b3ea0 con 0x7fb89806e030 2024-03-06T07:32:45.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb89e000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89806e4a0 con 0x7fb898144320 2024-03-06T07:32:45.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb89e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b4180 con 0x7fb8980727c0 2024-03-06T07:32:45.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb897e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb89806e030 0x7fb8980720b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51150/0 (socket says 172.21.15.6:51150) 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.544+0000 7fb897e00640 1 -- 172.21.15.6:0/3688808640 learned_addr learned my addr 172.21.15.6:0/3688808640 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb89d600640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb8980ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb89cc00640 1 --2- 172.21.15.6:0/3688808640 >> v2:172.21.15.178:3300/0 conn(0x7fb898144320 0x7fb89806daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402716495 0 0) 0x7fb8980b3ea0 con 0x7fb89806e030 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb87c003520 con 0x7fb89806e030 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231722693 0 0) 0x7fb87c003520 con 0x7fb89806e030 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 msgr2=0x7fb8980ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.545+0000 7fb897400640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb8980ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 >> v2:172.21.15.178:3300/0 conn(0x7fb898144320 msgr2=0x7fb89806daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 --2- 172.21.15.6:0/3688808640 >> v2:172.21.15.178:3300/0 conn(0x7fb898144320 0x7fb89806daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980b4460 con 0x7fb89806e030 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb884001690 con 0x7fb89806e030 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 >> v1:172.21.15.6:6789/0 conn(0x7fb89806e030 legacy=0x7fb8980720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0039b0 0x7fb87c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb87c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c004d40 0x7fb87c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89806e4a0 con 0x7fb87c0039b0 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b4180 con 0x7fb87c004d40 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c00b9c0 con 0x7fb8980727c0 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89cc00640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0039b0 0x7fb87c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb897e00640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb87c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89cc00640 1 -- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 msgr2=0x7fb87c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89d600640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c004d40 0x7fb87c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89cc00640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb87c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89cc00640 1 -- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c004d40 msgr2=0x7fb87c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89cc00640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c004d40 0x7fb87c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.546+0000 7fb89cc00640 1 -- 172.21.15.6:0/3688808640 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c003520 con 0x7fb87c0039b0 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.547+0000 7fb89d600640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb87c004d40 0x7fb87c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:45.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.547+0000 7fb897e00640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8980727c0 0x7fb87c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:45.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb89cc00640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0039b0 0x7fb87c003dd0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb88800e170 tx=0x7fb87c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c87e83ceb38e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888039070 con 0x7fb87c0039b0 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88802cc20 con 0x7fb87c0039b0 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb897400640 1 -- 172.21.15.6:0/3688808640 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888009c10 con 0x7fb87c0039b0 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb89e000640 1 -- 172.21.15.6:0/3688808640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0039b0 msgr2=0x7fb87c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb89e000640 1 --2- 172.21.15.6:0/3688808640 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb87c0039b0 0x7fb87c003dd0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb88800e170 tx=0x7fb87c005830 comp rx=0 tx=0).stop 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.548+0000 7fb89d600640 1 -- 172.21.15.6:0/3688808640 reap_dead start 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.549+0000 7fb89e000640 1 -- 172.21.15.6:0/3688808640 shutdown_connections 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.549+0000 7fb89e000640 1 -- 172.21.15.6:0/3688808640 >> 172.21.15.6:0/3688808640 conn(0x7fb8980b17c0 msgr2=0x7fb8980b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.549+0000 7fb89e000640 1 -- 172.21.15.6:0/3688808640 shutdown_connections 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.549+0000 7fb89e000640 1 -- 172.21.15.6:0/3688808640 wait complete. 2024-03-06T07:32:45.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.551+0000 7fb89e000640 1 Processor -- start 2024-03-06T07:32:45.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.551+0000 7fb89e000640 1 -- start start 2024-03-06T07:32:45.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0039b0 0x7fb898160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 0x7fb89815d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89813f680 0x7fb89813faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b4f40 con 0x7fb89815fdb0 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b4a20 con 0x7fb89813f680 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb898144700 con 0x7fb87c0039b0 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb897e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 0x7fb89815d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb897e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 0x7fb89815d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37928/0 (socket says 172.21.15.6:37928) 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb897e00640 1 -- 172.21.15.6:0/1319631818 learned_addr learned my addr 172.21.15.6:0/1319631818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb89d600640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89813f680 0x7fb89813faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb897e00640 1 -- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0039b0 msgr2=0x7fb898160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.552+0000 7fb897e00640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0039b0 0x7fb898160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb897e00640 1 -- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89813f680 msgr2=0x7fb89813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb897e00640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89813f680 0x7fb89813faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb897e00640 1 -- 172.21.15.6:0/1319631818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89815e660 con 0x7fb89815fdb0 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb89d600640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89813f680 0x7fb89813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:45.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb897e00640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 0x7fb89815d460 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb88400d8b0 tx=0x7fb88400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb884019070 con 0x7fb89815fdb0 2024-03-06T07:32:45.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8840074e0 con 0x7fb89815fdb0 2024-03-06T07:32:45.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb884004a00 con 0x7fb89815fdb0 2024-03-06T07:32:45.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb89815e940 con 0x7fb89815fdb0 2024-03-06T07:32:45.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.553+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89815ef00 con 0x7fb89815fdb0 2024-03-06T07:32:45.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.555+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89804f330 con 0x7fb89815fdb0 2024-03-06T07:32:45.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.556+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb884004ba0 con 0x7fb89815fdb0 2024-03-06T07:32:45.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.557+0000 7fb894c00640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb87806b4d0 0x7fb87806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:45.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.557+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb884089e10 con 0x7fb89815fdb0 2024-03-06T07:32:45.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.558+0000 7fb89cc00640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb87806b4d0 0x7fb87806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:45.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.559+0000 7fb89cc00640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb87806b4d0 0x7fb87806d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb888036000 tx=0x7fb88802e070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:45.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.560+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb884022020 con 0x7fb89815fdb0 2024-03-06T07:32:45.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:45.762+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb8980ae7b0 con 0x7fb89815fdb0 2024-03-06T07:32:46.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.402+0000 7fb894c00640 1 -- 172.21.15.6:0/1319631818 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb884058fc0 con 0x7fb89815fdb0 2024-03-06T07:32:46.404 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.405+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb87806b4d0 msgr2=0x7fb87806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.405+0000 7fb89e000640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb87806b4d0 0x7fb87806d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb888036000 tx=0x7fb88802e070 comp rx=0 tx=0).stop 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.405+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 msgr2=0x7fb89815d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.405+0000 7fb89e000640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 0x7fb89815d460 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb88400d8b0 tx=0x7fb88400dd70 comp rx=0 tx=0).stop 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 shutdown_connections 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb87806b4d0 0x7fb87806d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb89813f680 0x7fb89813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb89815fdb0 0x7fb89815d460 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 --2- 172.21.15.6:0/1319631818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb87c0039b0 0x7fb898160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 >> 172.21.15.6:0/1319631818 conn(0x7fb8980b17c0 msgr2=0x7fb8980ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 shutdown_connections 2024-03-06T07:32:46.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.406+0000 7fb89e000640 1 -- 172.21.15.6:0/1319631818 wait complete. 2024-03-06T07:32:46.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:32:46.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.639+0000 7f3209000640 1 Processor -- start 2024-03-06T07:32:46.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.639+0000 7f3209000640 1 -- start start 2024-03-06T07:32:46.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3209000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 0x7f3204166a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3209000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040b4ae0 con 0x7f32040cef80 2024-03-06T07:32:46.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3209000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32040b4dc0 con 0x7f3204145570 2024-03-06T07:32:46.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3209000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320406c9f0 con 0x7f32040cd430 2024-03-06T07:32:46.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3202a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f32040cef80 0x7f32040ccd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51158/0 (socket says 172.21.15.6:51158) 2024-03-06T07:32:46.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3202a00640 1 -- 172.21.15.6:0/1388921741 learned_addr learned my addr 172.21.15.6:0/1388921741 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.641+0000 7f3203e00640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 0x7f3204166a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183146306 0 0) 0x7f32040b4ae0 con 0x7f32040cef80 2024-03-06T07:32:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e8003520 con 0x7f32040cef80 2024-03-06T07:32:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3559083551 0 0) 0x7f31e8003520 con 0x7f32040cef80 2024-03-06T07:32:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 msgr2=0x7f3204166a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 0x7f3204166a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 >> v1:172.21.15.178:6789/0 conn(0x7f3204145570 legacy=0x7f32040b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320406cc40 con 0x7f32040cef80 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f31f0001670 con 0x7f32040cef80 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 >> v1:172.21.15.6:6789/0 conn(0x7f32040cef80 legacy=0x7f32040ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f31e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31e8004800 0x7f31e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 0x7f31e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320406c9f0 con 0x7f31e80039e0 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e800bad0 con 0x7f31e8004800 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.642+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e800bcd0 con 0x7f32040cd430 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203400640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f31e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203e00640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 0x7f31e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203400640 1 -- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 msgr2=0x7f31e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203400640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040cd430 0x7f31e80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203400640 1 -- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31e8004800 msgr2=0x7f31e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:46.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3202a00640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31e8004800 0x7f31e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203400640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31e8004800 0x7f31e8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.643+0000 7f3203400640 1 -- 172.21.15.6:0/1388921741 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e8003520 con 0x7f31e80039e0 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.644+0000 7f3203400640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f31e8003e00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f31f4001720 tx=0x7f31e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6be541cb82738e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.644+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4001370 con 0x7f31e80039e0 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.644+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f40052d0 con 0x7f31e80039e0 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.644+0000 7f3202000640 1 -- 172.21.15.6:0/1388921741 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4003830 con 0x7f31e80039e0 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3209000640 1 -- 172.21.15.6:0/1388921741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 msgr2=0x7f31e8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3209000640 1 --2- 172.21.15.6:0/1388921741 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f31e8003e00 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f31f4001720 tx=0x7f31e8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3203e00640 1 -- 172.21.15.6:0/1388921741 reap_dead start 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3209000640 1 -- 172.21.15.6:0/1388921741 shutdown_connections 2024-03-06T07:32:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3209000640 1 -- 172.21.15.6:0/1388921741 >> 172.21.15.6:0/1388921741 conn(0x7f320406b190 msgr2=0x7f320406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3209000640 1 -- 172.21.15.6:0/1388921741 shutdown_connections 2024-03-06T07:32:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.645+0000 7f3209000640 1 -- 172.21.15.6:0/1388921741 wait complete. 2024-03-06T07:32:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.647+0000 7f3209000640 1 Processor -- start 2024-03-06T07:32:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.647+0000 7f3209000640 1 -- start start 2024-03-06T07:32:46.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3209000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f320413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3209000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32040addf0 0x7f32040aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3209000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040ab380 0x7f3204163350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3209000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320415c110 con 0x7f31e80039e0 2024-03-06T07:32:46.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3209000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320406cfb0 con 0x7f32040addf0 2024-03-06T07:32:46.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3209000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3204164a70 con 0x7f32040ab380 2024-03-06T07:32:46.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3203400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f320413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3203400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f320413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37950/0 (socket says 172.21.15.6:37950) 2024-03-06T07:32:46.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.648+0000 7f3203400640 1 -- 172.21.15.6:0/1790925957 learned_addr learned my addr 172.21.15.6:0/1790925957 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3203400640 1 -- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040ab380 msgr2=0x7f3204163350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3203400640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040ab380 0x7f3204163350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3203400640 1 -- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32040addf0 msgr2=0x7f32040aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3203400640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32040addf0 0x7f32040aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3203400640 1 -- 172.21.15.6:0/1790925957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320413f810 con 0x7f31e80039e0 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3203400640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f320413f2d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f31f4005040 tx=0x7f31f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4041070 con 0x7f31e80039e0 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f4005480 con 0x7f31e80039e0 2024-03-06T07:32:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4007a30 con 0x7f31e80039e0 2024-03-06T07:32:46.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.649+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32040acd50 con 0x7f31e80039e0 2024-03-06T07:32:46.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.650+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32040ad310 con 0x7f31e80039e0 2024-03-06T07:32:46.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.651+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320404f330 con 0x7f31e80039e0 2024-03-06T07:32:46.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.652+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31f4044070 con 0x7f31e80039e0 2024-03-06T07:32:46.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.653+0000 7f31fb400640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31e406b500 0x7f31e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:46.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.654+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f31f40abaf0 con 0x7f31e80039e0 2024-03-06T07:32:46.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.654+0000 7f3202a00640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31e406b500 0x7f31e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:46.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.655+0000 7f3202a00640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31e406b500 0x7f31e406d9c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f31f0000ff0 tx=0x7f31f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:46.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.659+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f320413f810 con 0x7f31e80039e0 2024-03-06T07:32:46.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:46.870+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f32040b0300 con 0x7f31e80039e0 2024-03-06T07:32:47.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.408+0000 7f31fb400640 1 -- 172.21.15.6:0/1790925957 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7f31f407ab40 con 0x7f31e80039e0 2024-03-06T07:32:47.411 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31e406b500 msgr2=0x7f31e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31e406b500 0x7f31e406d9c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f31f0000ff0 tx=0x7f31f0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:47.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 msgr2=0x7f320413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f320413f2d0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f31f4005040 tx=0x7f31f4001120 comp rx=0 tx=0).stop 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 shutdown_connections 2024-03-06T07:32:47.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31e406b500 0x7f31e406d9c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f32040addf0 0x7f32040aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f32040ab380 0x7f3204163350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 --2- 172.21.15.6:0/1790925957 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31e80039e0 0x7f320413f2d0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.412+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 >> 172.21.15.6:0/1790925957 conn(0x7f320406b190 msgr2=0x7f32040b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.413+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 shutdown_connections 2024-03-06T07:32:47.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.413+0000 7f3209000640 1 -- 172.21.15.6:0/1790925957 wait complete. 2024-03-06T07:32:47.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:32:47.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.639+0000 7f9002a00640 1 Processor -- start 2024-03-06T07:32:47.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.639+0000 7f9002a00640 1 -- start start 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.640+0000 7f9002a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ffc06da40 0x7f8ffc06e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.640+0000 7f9002a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8ffc15e680 0x7f8ffc06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.640+0000 7f9002a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ffc15eb80 con 0x7f8ffc0ccad0 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.640+0000 7f9002a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc161c70 con 0x7f8ffc06da40 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.640+0000 7f9002a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ffc15ee60 con 0x7f8ffc15e680 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.640+0000 7f8ffbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ffc06da40 0x7f8ffc06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.641+0000 7f8ffbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ffc06da40 0x7f8ffc06e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37964/0 (socket says 172.21.15.6:37964) 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.641+0000 7f8ffbe00640 1 -- 172.21.15.6:0/596880796 learned_addr learned my addr 172.21.15.6:0/596880796 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.641+0000 7f8ffbe00640 1 -- 172.21.15.6:0/596880796 >> v2:172.21.15.178:3300/0 conn(0x7f8ffc15e680 msgr2=0x7f8ffc06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.641+0000 7f8ffbe00640 1 --2- 172.21.15.6:0/596880796 >> v2:172.21.15.178:3300/0 conn(0x7f8ffc15e680 0x7f8ffc06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.641+0000 7f8ffbe00640 1 -- 172.21.15.6:0/596880796 >> v1:172.21.15.178:6789/0 conn(0x7f8ffc0ccad0 legacy=0x7f8ffc162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:47.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.641+0000 7f8ffbe00640 1 -- 172.21.15.6:0/596880796 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc15f140 con 0x7f8ffc06da40 2024-03-06T07:32:47.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.642+0000 7f8ffbe00640 1 --2- 172.21.15.6:0/596880796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06da40 0x7f8ffc06e340 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc06bfd0 tx=0x7f8fec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e61d62e38edd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f8ffaa00640 1 -- 172.21.15.6:0/596880796 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec035020 con 0x7f8ffc06da40 2024-03-06T07:32:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f8ffaa00640 1 -- 172.21.15.6:0/596880796 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec031870 con 0x7f8ffc06da40 2024-03-06T07:32:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f8ffaa00640 1 -- 172.21.15.6:0/596880796 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec031c40 con 0x7f8ffc06da40 2024-03-06T07:32:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 -- 172.21.15.6:0/596880796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06da40 msgr2=0x7f8ffc06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 --2- 172.21.15.6:0/596880796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06da40 0x7f8ffc06e340 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc06bfd0 tx=0x7f8fec02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:47.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 -- 172.21.15.6:0/596880796 shutdown_connections 2024-03-06T07:32:47.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 --2- 172.21.15.6:0/596880796 >> v2:172.21.15.178:3300/0 conn(0x7f8ffc15e680 0x7f8ffc06c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 --2- 172.21.15.6:0/596880796 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc06da40 0x7f8ffc06e340 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 -- 172.21.15.6:0/596880796 >> 172.21.15.6:0/596880796 conn(0x7f8ffc06b3d0 msgr2=0x7f8ffc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:47.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 -- 172.21.15.6:0/596880796 shutdown_connections 2024-03-06T07:32:47.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.643+0000 7f9002a00640 1 -- 172.21.15.6:0/596880796 wait complete. 2024-03-06T07:32:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.645+0000 7f9002a00640 1 Processor -- start 2024-03-06T07:32:47.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.645+0000 7f9002a00640 1 -- start start 2024-03-06T07:32:47.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.646+0000 7f9002a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ffc06da40 0x7f8ffc156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.646+0000 7f9002a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ffc0ccad0 0x7f8ffc0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.646+0000 7f9002a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 0x7f8ffc15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.646+0000 7f9002a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc06ea00 con 0x7f8ffc15e680 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.646+0000 7f9002a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc06e880 con 0x7f8ffc0ccad0 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.646+0000 7f9002a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ffc06cec0 con 0x7f8ffc06da40 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 0x7f8ffc15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 0x7f8ffc15c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37968/0 (socket says 172.21.15.6:37968) 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 -- 172.21.15.6:0/1374337924 learned_addr learned my addr 172.21.15.6:0/1374337924 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f8ffbe00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ffc06da40 0x7f8ffc156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f8ffb400640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ffc0ccad0 0x7f8ffc0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 -- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ffc06da40 msgr2=0x7f8ffc156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ffc06da40 0x7f8ffc156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 -- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ffc0ccad0 msgr2=0x7f8ffc0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ffc0ccad0 0x7f8ffc0ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:47.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.647+0000 7f9000a00640 1 -- 172.21.15.6:0/1374337924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0aedd0 con 0x7f8ffc15e680 2024-03-06T07:32:47.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.648+0000 7f9000a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 0x7f8ffc15c4a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c780 tx=0x7f8ff400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.648+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4004b60 con 0x7f8ffc15e680 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.648+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4004d00 con 0x7f8ffc15e680 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.648+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc0af050 con 0x7f8ffc15e680 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.648+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff400aa90 con 0x7f8ffc15e680 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.648+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc0ad110 con 0x7f8ffc15e680 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.649+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ffc04f330 con 0x7f8ffc15e680 2024-03-06T07:32:47.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.651+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ff400ac30 con 0x7f8ffc15e680 2024-03-06T07:32:47.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.651+0000 7f8ff3e00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8fdc06b4d0 0x7f8fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.652+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8ff408a370 con 0x7f8ffc15e680 2024-03-06T07:32:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.652+0000 7f8ffbe00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8fdc06b4d0 0x7f8fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:47.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.654+0000 7f8ffbe00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8fdc06b4d0 0x7f8fdc06d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:47.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.657+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ff4020070 con 0x7f8ffc15e680 2024-03-06T07:32:47.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:47.890+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f8ffc06cab0 con 0x7f8ffc15e680 2024-03-06T07:32:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.463+0000 7f8ff3e00640 1 -- 172.21.15.6:0/1374337924 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v58) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8ffc0aedd0 con 0x7f8ffc15e680 2024-03-06T07:32:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.471+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8fdc06b4d0 msgr2=0x7f8fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.471+0000 7f9002a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8fdc06b4d0 0x7f8fdc06d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8fec036000 tx=0x7f8fec03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.471+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 msgr2=0x7f8ffc15c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.471+0000 7f9002a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 0x7f8ffc15c4a0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8ff400c780 tx=0x7f8ff400cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.473+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 shutdown_connections 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.473+0000 7f9002a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8fdc06b4d0 0x7f8fdc06d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.473+0000 7f9002a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ffc0ccad0 0x7f8ffc0ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.473+0000 7f9002a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ffc15e680 0x7f8ffc15c4a0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.473+0000 7f9002a00640 1 --2- 172.21.15.6:0/1374337924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ffc06da40 0x7f8ffc156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.473+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 >> 172.21.15.6:0/1374337924 conn(0x7f8ffc06b3d0 msgr2=0x7f8ffc15dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.474+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 shutdown_connections 2024-03-06T07:32:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.474+0000 7f9002a00640 1 -- 172.21.15.6:0/1374337924 wait complete. 2024-03-06T07:32:48.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:48.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.687+0000 7f6475600640 1 Processor -- start 2024-03-06T07:32:48.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.687+0000 7f6475600640 1 -- start start 2024-03-06T07:32:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f6475600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f647015c920 0x7f6470070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f6475600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64700cdad0 con 0x7f64700ce850 2024-03-06T07:32:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f6475600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6470071500 con 0x7f64700cd060 2024-03-06T07:32:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f6475600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64700cdd70 con 0x7f647015c920 2024-03-06T07:32:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f646f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f64700ce850 0x7f64700cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51166/0 (socket says 172.21.15.6:51166) 2024-03-06T07:32:48.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f646f400640 1 -- 172.21.15.6:0/3457891533 learned_addr learned my addr 172.21.15.6:0/3457891533 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455158813 0 0) 0x7f64700cdad0 con 0x7f64700ce850 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f646fe00640 1 --2- 172.21.15.6:0/3457891533 >> v2:172.21.15.178:3300/0 conn(0x7f647015c920 0x7f6470070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.689+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454003520 con 0x7f64700ce850 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525507837 0 0) 0x7f6454003520 con 0x7f64700ce850 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 >> v2:172.21.15.178:3300/0 conn(0x7f647015c920 msgr2=0x7f6470070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 --2- 172.21.15.6:0/3457891533 >> v2:172.21.15.178:3300/0 conn(0x7f647015c920 0x7f6470070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 >> v1:172.21.15.178:6789/0 conn(0x7f64700cd060 legacy=0x7f647006cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700ce050 con 0x7f64700ce850 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f645c001690 con 0x7f64700ce850 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 >> v1:172.21.15.6:6789/0 conn(0x7f64700ce850 legacy=0x7f64700cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64540039b0 0x7f6454003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f6454004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64540051c0 0x7f645400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700cdd70 con 0x7f6454004800 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c060 con 0x7f64540039b0 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645400c260 con 0x7f64540051c0 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.690+0000 7f646f400640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f6454004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.691+0000 7f646f400640 1 -- 172.21.15.6:0/3457891533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64540051c0 msgr2=0x7f645400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.691+0000 7f646f400640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64540051c0 0x7f645400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.691+0000 7f646f400640 1 -- 172.21.15.6:0/3457891533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64540039b0 msgr2=0x7f6454003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.691+0000 7f646f400640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64540039b0 0x7f6454003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.691+0000 7f646f400640 1 -- 172.21.15.6:0/3457891533 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454003520 con 0x7f6454004800 2024-03-06T07:32:48.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.691+0000 7f6474c00640 1 -- 172.21.15.6:0/3457891533 reap_dead start 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.692+0000 7f646f400640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f6454004c80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c210 tx=0x7f6454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5f96001d63fc67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.692+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c007d30 con 0x7f6454004800 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.692+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645c030070 con 0x7f6454004800 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.692+0000 7f646ea00640 1 -- 172.21.15.6:0/3457891533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645c005d30 con 0x7f6454004800 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.692+0000 7f6475600640 1 -- 172.21.15.6:0/3457891533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 msgr2=0x7f6454004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.692+0000 7f6475600640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f6454004c80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c210 tx=0x7f6454005830 comp rx=0 tx=0).stop 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.693+0000 7f6475600640 1 -- 172.21.15.6:0/3457891533 shutdown_connections 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.693+0000 7f6475600640 1 --2- 172.21.15.6:0/3457891533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f6454004c80 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.693+0000 7f6475600640 1 -- 172.21.15.6:0/3457891533 >> 172.21.15.6:0/3457891533 conn(0x7f647006b3d0 msgr2=0x7f647006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.693+0000 7f6475600640 1 -- 172.21.15.6:0/3457891533 shutdown_connections 2024-03-06T07:32:48.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.693+0000 7f6475600640 1 -- 172.21.15.6:0/3457891533 wait complete. 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.694+0000 7f6475600640 1 Processor -- start 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.694+0000 7f6475600640 1 -- start start 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f6475600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f647015fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f6475600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f647015d4d0 0x7f647006cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f6475600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647015d950 0x7f647006d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f6475600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700cc950 con 0x7f6454004800 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f6475600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470163790 con 0x7f647015d950 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f6475600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700ccad0 con 0x7f647015d4d0 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.695+0000 7f646fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f647015fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f647015fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37982/0 (socket says 172.21.15.6:37982) 2024-03-06T07:32:48.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 -- 172.21.15.6:0/755676206 learned_addr learned my addr 172.21.15.6:0/755676206 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 -- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f647015d4d0 msgr2=0x7f647006cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646f400640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f647015d4d0 0x7f647006cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f6474c00640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647015d950 0x7f647006d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f647015d4d0 0x7f647006cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 -- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647015d950 msgr2=0x7f647006d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647015d950 0x7f647006d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 -- 172.21.15.6:0/755676206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647015f020 con 0x7f6454004800 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f6474c00640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647015d950 0x7f647006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.696+0000 7f646fe00640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f647015fd60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f646000e860 tx=0x7f646000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.697+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6460017070 con 0x7f6454004800 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.697+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646000c940 con 0x7f6454004800 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.697+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646000cd10 con 0x7f6454004800 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.697+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64701602a0 con 0x7f6454004800 2024-03-06T07:32:48.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.697+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64701fc5c0 con 0x7f6454004800 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.698+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647004f330 con 0x7f6454004800 2024-03-06T07:32:48.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.699+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f646001a070 con 0x7f6454004800 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.700+0000 7f6467e00640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.700+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f646008a220 con 0x7f6454004800 2024-03-06T07:32:48.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.701+0000 7f646f400640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:48.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.702+0000 7f646f400640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f645006b4d0 0x7f645006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f645c034000 tx=0x7f645c039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:48.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.705+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f647015f020 con 0x7f6454004800 2024-03-06T07:32:48.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.906+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f647015aab0 con 0x7f6454004800 2024-03-06T07:32:48.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.909+0000 7f6467e00640 1 -- 172.21.15.6:0/755676206 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16549 (secure 0 0 0) 0x7f6460059090 con 0x7f6454004800 2024-03-06T07:32:48.912 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:48.912 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":92,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:47.892431+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:48.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f645006b4d0 msgr2=0x7f645006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f645006b4d0 0x7f645006d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f645c034000 tx=0x7f645c039040 comp rx=0 tx=0).stop 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 msgr2=0x7f647015fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f647015fd60 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f646000e860 tx=0x7f646000ed20 comp rx=0 tx=0).stop 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 shutdown_connections 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f645006b4d0 0x7f645006d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f647015d950 0x7f647006d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f647015d4d0 0x7f647006cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 --2- 172.21.15.6:0/755676206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6454004800 0x7f647015fd60 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 >> 172.21.15.6:0/755676206 conn(0x7f647006b3d0 msgr2=0x7f6470161e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 shutdown_connections 2024-03-06T07:32:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:48.912+0000 7f6475600640 1 -- 172.21.15.6:0/755676206 wait complete. 2024-03-06T07:32:48.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:49.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.146+0000 7f471f200640 1 Processor -- start 2024-03-06T07:32:49.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.146+0000 7f471f200640 1 -- start start 2024-03-06T07:32:49.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.148+0000 7f471f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47200b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.148+0000 7f471f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f47200cd060 0x7f472015c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.148+0000 7f471f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47200714d0 con 0x7f4720070f10 2024-03-06T07:32:49.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.148+0000 7f471f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47200717b0 con 0x7f47200cd060 2024-03-06T07:32:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.148+0000 7f471f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4720071a90 con 0x7f4720145300 2024-03-06T07:32:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.149+0000 7f471de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47200b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.149+0000 7f471e800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f47200cd060 0x7f472015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.149+0000 7f471de00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47200b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:49918/0 (socket says 172.21.15.6:49918) 2024-03-06T07:32:49.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.149+0000 7f471de00640 1 -- 172.21.15.6:0/3518876577 learned_addr learned my addr 172.21.15.6:0/3518876577 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:49.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.150+0000 7f471e800640 1 -- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 msgr2=0x7f47200b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.150+0000 7f471e800640 1 --2- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47200b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.150+0000 7f471e800640 1 -- 172.21.15.6:0/3518876577 >> v1:172.21.15.178:6789/0 conn(0x7f4720070f10 legacy=0x7f47200cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.150+0000 7f471e800640 1 -- 172.21.15.6:0/3518876577 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720071d70 con 0x7f47200cd060 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.151+0000 7f471e800640 1 --2- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47200cd060 0x7f472015c570 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4718009d90 tx=0x7f471802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac852025fdf17b23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.151+0000 7f471ca00640 1 -- 172.21.15.6:0/3518876577 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718039070 con 0x7f47200cd060 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.151+0000 7f471ca00640 1 -- 172.21.15.6:0/3518876577 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471802cc50 con 0x7f47200cd060 2024-03-06T07:32:49.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.151+0000 7f471ca00640 1 -- 172.21.15.6:0/3518876577 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718034440 con 0x7f47200cd060 2024-03-06T07:32:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 -- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47200cd060 msgr2=0x7f472015c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 --2- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47200cd060 0x7f472015c570 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4718009d90 tx=0x7f471802bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 -- 172.21.15.6:0/3518876577 shutdown_connections 2024-03-06T07:32:49.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 --2- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47200cd060 0x7f472015c570 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 --2- 172.21.15.6:0/3518876577 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47200b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 -- 172.21.15.6:0/3518876577 >> 172.21.15.6:0/3518876577 conn(0x7f472006b3d0 msgr2=0x7f472006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 -- 172.21.15.6:0/3518876577 shutdown_connections 2024-03-06T07:32:49.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.152+0000 7f471f200640 1 -- 172.21.15.6:0/3518876577 wait complete. 2024-03-06T07:32:49.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.154+0000 7f471f200640 1 Processor -- start 2024-03-06T07:32:49.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.155+0000 7f471f200640 1 -- start start 2024-03-06T07:32:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4720070f10 0x7f47201f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 0x7f47200c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47201f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472015cab0 con 0x7f47200cd060 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47201456e0 con 0x7f4720070f10 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472015cc30 con 0x7f4720145300 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 0x7f47200c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 0x7f47200c4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37992/0 (socket says 172.21.15.6:37992) 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.156+0000 7f471d400640 1 -- 172.21.15.6:0/2469962987 learned_addr learned my addr 172.21.15.6:0/2469962987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471de00640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4720070f10 0x7f47201f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471e800640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47201f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471d400640 1 -- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 msgr2=0x7f47201f5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471d400640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47201f5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471d400640 1 -- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4720070f10 msgr2=0x7f47201f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471d400640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4720070f10 0x7f47201f46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471d400640 1 -- 172.21.15.6:0/2469962987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47201e4e90 con 0x7f47200cd060 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471de00640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4720070f10 0x7f47201f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f471d400640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 0x7f47200c4cc0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c450 tx=0x7f470c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c0052d0 con 0x7f47200cd060 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c005470 con 0x7f47200cd060 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.157+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c012870 con 0x7f47200cd060 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.158+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47201e5130 con 0x7f47200cd060 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.158+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47201e59d0 con 0x7f47200cd060 2024-03-06T07:32:49.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.159+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472004f330 con 0x7f47200cd060 2024-03-06T07:32:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.160+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f470c012a10 con 0x7f47200cd060 2024-03-06T07:32:49.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.161+0000 7f4716000640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f470006b4d0 0x7f470006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.162+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f470c089db0 con 0x7f47200cd060 2024-03-06T07:32:49.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.162+0000 7f471de00640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f470006b4d0 0x7f470006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.163+0000 7f471de00640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f470006b4d0 0x7f470006d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4710009b80 tx=0x7f4710008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.166+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f470c058cc0 con 0x7f47200cd060 2024-03-06T07:32:49.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.360+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47200ade40 con 0x7f47200cd060 2024-03-06T07:32:49.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.363+0000 7f4716000640 1 -- 172.21.15.6:0/2469962987 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16549 (secure 0 0 0) 0x7f470c0585c0 con 0x7f47200cd060 2024-03-06T07:32:49.365 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:49.365 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":92,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:47.892431+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f470006b4d0 msgr2=0x7f470006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f470006b4d0 0x7f470006d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4710009b80 tx=0x7f4710008040 comp rx=0 tx=0).stop 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 msgr2=0x7f47200c4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 0x7f47200c4cc0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c450 tx=0x7f470c00c910 comp rx=0 tx=0).stop 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 shutdown_connections 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f470006b4d0 0x7f470006d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4720145300 0x7f47201f5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47200cd060 0x7f47200c4cc0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 --2- 172.21.15.6:0/2469962987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4720070f10 0x7f47201f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 >> 172.21.15.6:0/2469962987 conn(0x7f472006b3d0 msgr2=0x7f47200b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.365+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 shutdown_connections 2024-03-06T07:32:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.366+0000 7f471f200640 1 -- 172.21.15.6:0/2469962987 wait complete. 2024-03-06T07:32:49.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:32:49.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.607+0000 7fb02fe00640 1 Processor -- start 2024-03-06T07:32:49.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.608+0000 7fb02fe00640 1 -- start start 2024-03-06T07:32:49.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.609+0000 7fb02fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb0300d4860 0x7fb0300d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.609+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300cfa60 0x7fb0300d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.609+0000 7fb02fe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0300ceca0 con 0x7fb03015dee0 2024-03-06T07:32:49.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.609+0000 7fb02fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0300cf5d0 con 0x7fb0300d4860 2024-03-06T07:32:49.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.609+0000 7fb02fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300cef80 con 0x7fb0300cfa60 2024-03-06T07:32:49.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb0300d4860 0x7fb0300d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300cfa60 0x7fb0300d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb0300d4860 0x7fb0300d0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:34608/0 (socket says 172.21.15.6:34608) 2024-03-06T07:32:49.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02f400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300cfa60 0x7fb0300d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:49932/0 (socket says 172.21.15.6:49932) 2024-03-06T07:32:49.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 -- 172.21.15.6:0/1313425018 learned_addr learned my addr 172.21.15.6:0/1313425018 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:49.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02ea00640 1 --1- 172.21.15.6:0/1313425018 >> v1:172.21.15.178:6789/0 conn(0x7fb03015dee0 0x7fb0300d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:50226/0 (socket says 172.21.15.6:50226) 2024-03-06T07:32:49.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 -- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300cfa60 msgr2=0x7fb0300d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:49.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 --2- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300cfa60 0x7fb0300d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 -- 172.21.15.6:0/1313425018 >> v1:172.21.15.178:6789/0 conn(0x7fb03015dee0 legacy=0x7fb0300d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.610+0000 7fb02e000640 1 -- 172.21.15.6:0/1313425018 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300cf260 con 0x7fb0300d4860 2024-03-06T07:32:49.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.612+0000 7fb02e000640 1 --2- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300d4860 0x7fb0300d0630 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009870 tx=0x7fb01c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a25af8cf7bb9ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.612+0000 7fb02d600640 1 -- 172.21.15.6:0/1313425018 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c034070 con 0x7fb0300d4860 2024-03-06T07:32:49.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.612+0000 7fb02d600640 1 -- 172.21.15.6:0/1313425018 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c030840 con 0x7fb0300d4860 2024-03-06T07:32:49.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.612+0000 7fb02d600640 1 -- 172.21.15.6:0/1313425018 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c030c10 con 0x7fb0300d4860 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 -- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300d4860 msgr2=0x7fb0300d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 --2- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300d4860 0x7fb0300d0630 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009870 tx=0x7fb01c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 -- 172.21.15.6:0/1313425018 shutdown_connections 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 --2- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300cfa60 0x7fb0300d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 --2- 172.21.15.6:0/1313425018 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300d4860 0x7fb0300d0630 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 -- 172.21.15.6:0/1313425018 >> 172.21.15.6:0/1313425018 conn(0x7fb0300d3cd0 msgr2=0x7fb0300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 -- 172.21.15.6:0/1313425018 shutdown_connections 2024-03-06T07:32:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.613+0000 7fb02fe00640 1 -- 172.21.15.6:0/1313425018 wait complete. 2024-03-06T07:32:49.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.615+0000 7fb02fe00640 1 Processor -- start 2024-03-06T07:32:49.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.615+0000 7fb02fe00640 1 -- start start 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300cfa60 0x7fb0300c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300d4860 0x7fb030152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 0x7fb030153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300b3510 con 0x7fb03015dee0 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03015e2c0 con 0x7fb0300cfa60 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300b3690 con 0x7fb0300d4860 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 0x7fb030153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 0x7fb030153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37996/0 (socket says 172.21.15.6:37996) 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02f400640 1 -- 172.21.15.6:0/2371808541 learned_addr learned my addr 172.21.15.6:0/2371808541 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02ea00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300cfa60 0x7fb0300c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.616+0000 7fb02e000640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300d4860 0x7fb030152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02f400640 1 -- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300d4860 msgr2=0x7fb030152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02f400640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300d4860 0x7fb030152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02f400640 1 -- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300cfa60 msgr2=0x7fb0300c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02f400640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300cfa60 0x7fb0300c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02f400640 1 -- 172.21.15.6:0/2371808541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300b80c0 con 0x7fb03015dee0 2024-03-06T07:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02e000640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300d4860 0x7fb030152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:49.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02f400640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 0x7fb030153430 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fb02800e960 tx=0x7fb02800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb028017070 con 0x7fb03015dee0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02800c980 con 0x7fb03015dee0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02800cd70 con 0x7fb03015dee0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0300c7640 con 0x7fb03015dee0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.617+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0300c7c00 con 0x7fb03015dee0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.619+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03004f330 con 0x7fb03015dee0 2024-03-06T07:32:49.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.619+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb02801a070 con 0x7fb03015dee0 2024-03-06T07:32:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.620+0000 7fb027400640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb01006b4d0 0x7fb01006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:49.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.620+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb02808a170 con 0x7fb03015dee0 2024-03-06T07:32:49.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.621+0000 7fb02ea00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb01006b4d0 0x7fb01006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:49.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.622+0000 7fb02ea00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb01006b4d0 0x7fb01006d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb0200012e0 tx=0x7fb020005b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:49.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.625+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb028059080 con 0x7fb03015dee0 2024-03-06T07:32:49.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:49.874+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb03008d7d0 con 0x7fb03015dee0 2024-03-06T07:32:50.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.478+0000 7fb027400640 1 -- 172.21.15.6:0/2371808541 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb028058980 con 0x7fb03015dee0 2024-03-06T07:32:50.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb01006b4d0 msgr2=0x7fb01006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb01006b4d0 0x7fb01006d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb0200012e0 tx=0x7fb020005b70 comp rx=0 tx=0).stop 2024-03-06T07:32:50.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 msgr2=0x7fb030153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 0x7fb030153430 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fb02800e960 tx=0x7fb02800ee20 comp rx=0 tx=0).stop 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 shutdown_connections 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb01006b4d0 0x7fb01006d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb03015dee0 0x7fb030153430 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0300d4860 0x7fb030152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 --2- 172.21.15.6:0/2371808541 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0300cfa60 0x7fb0300c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.482+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 >> 172.21.15.6:0/2371808541 conn(0x7fb0300d3cd0 msgr2=0x7fb0300d18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.483+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 shutdown_connections 2024-03-06T07:32:50.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.483+0000 7fb02fe00640 1 -- 172.21.15.6:0/2371808541 wait complete. 2024-03-06T07:32:50.505 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:32:50.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.721+0000 7fe34c800640 1 Processor -- start 2024-03-06T07:32:50.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.721+0000 7fe34c800640 1 -- start start 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe34c800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3440cf500 0x7fe3440d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe34c800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe3440b43a0 0x7fe3440b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe34c800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34415e640 con 0x7fe34415e260 2024-03-06T07:32:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe34c800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440d4ca0 con 0x7fe3440cf500 2024-03-06T07:32:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe34c800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3440cfa70 con 0x7fe3440b43a0 2024-03-06T07:32:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe349400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3440cf500 0x7fe3440d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe349400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3440cf500 0x7fe3440d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38002/0 (socket says 172.21.15.6:38002) 2024-03-06T07:32:50.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.723+0000 7fe349400640 1 -- 172.21.15.6:0/1615237551 learned_addr learned my addr 172.21.15.6:0/1615237551 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:50.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.724+0000 7fe349400640 1 -- 172.21.15.6:0/1615237551 >> v2:172.21.15.178:3300/0 conn(0x7fe3440b43a0 msgr2=0x7fe3440b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.724+0000 7fe349400640 1 --2- 172.21.15.6:0/1615237551 >> v2:172.21.15.178:3300/0 conn(0x7fe3440b43a0 0x7fe3440b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.724+0000 7fe349400640 1 -- 172.21.15.6:0/1615237551 >> v1:172.21.15.178:6789/0 conn(0x7fe34415e260 legacy=0x7fe3440d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:50.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.724+0000 7fe349400640 1 -- 172.21.15.6:0/1615237551 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3440cfd50 con 0x7fe3440cf500 2024-03-06T07:32:50.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.725+0000 7fe349400640 1 --2- 172.21.15.6:0/1615237551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe3440d21c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe334009870 tx=0x7fe33402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516517b7c25d5d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.725+0000 7fe348a00640 1 -- 172.21.15.6:0/1615237551 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33402e070 con 0x7fe3440cf500 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.725+0000 7fe348a00640 1 -- 172.21.15.6:0/1615237551 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe334037070 con 0x7fe3440cf500 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.725+0000 7fe348a00640 1 -- 172.21.15.6:0/1615237551 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334031a00 con 0x7fe3440cf500 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 -- 172.21.15.6:0/1615237551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 msgr2=0x7fe3440d21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 --2- 172.21.15.6:0/1615237551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe3440d21c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe334009870 tx=0x7fe33402b880 comp rx=0 tx=0).stop 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 -- 172.21.15.6:0/1615237551 shutdown_connections 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 --2- 172.21.15.6:0/1615237551 >> v2:172.21.15.178:3300/0 conn(0x7fe3440b43a0 0x7fe3440b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 --2- 172.21.15.6:0/1615237551 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe3440d21c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 -- 172.21.15.6:0/1615237551 >> 172.21.15.6:0/1615237551 conn(0x7fe3440d3ee0 msgr2=0x7fe3440d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 -- 172.21.15.6:0/1615237551 shutdown_connections 2024-03-06T07:32:50.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.726+0000 7fe34c800640 1 -- 172.21.15.6:0/1615237551 wait complete. 2024-03-06T07:32:50.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.728+0000 7fe34c800640 1 Processor -- start 2024-03-06T07:32:50.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.728+0000 7fe34c800640 1 -- start start 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe34c800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3440b43a0 0x7fe34415e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe34c800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe34414c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe34c800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe34415e260 0x7fe34414cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe34c800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440b3e80 con 0x7fe3440cf500 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe34c800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440b3ba0 con 0x7fe3440b43a0 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe34c800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3440d2700 con 0x7fe34415e260 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe349400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe34414c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe349400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe34414c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38004/0 (socket says 172.21.15.6:38004) 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.729+0000 7fe349400640 1 -- 172.21.15.6:0/3984944026 learned_addr learned my addr 172.21.15.6:0/3984944026 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349e00640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3440b43a0 0x7fe34415e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349400640 1 -- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe34415e260 msgr2=0x7fe34414cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349400640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe34415e260 0x7fe34414cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349400640 1 -- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3440b43a0 msgr2=0x7fe34415e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349400640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3440b43a0 0x7fe34415e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349400640 1 -- 172.21.15.6:0/3984944026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3441616b0 con 0x7fe3440cf500 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.730+0000 7fe349400640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe34414c1d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe334009870 tx=0x7fe3340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.731+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334031ed0 con 0x7fe3440cf500 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.731+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe334032880 con 0x7fe3440cf500 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.731+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3440b7db0 con 0x7fe3440cf500 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.731+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334032c70 con 0x7fe3440cf500 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.731+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3440b8370 con 0x7fe3440cf500 2024-03-06T07:32:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.732+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34404f330 con 0x7fe3440cf500 2024-03-06T07:32:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.734+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe33404c060 con 0x7fe3440cf500 2024-03-06T07:32:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.735+0000 7fe33e000640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe32806b4d0 0x7fe32806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.735+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3340abbd0 con 0x7fe3440cf500 2024-03-06T07:32:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.736+0000 7fe349e00640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe32806b4d0 0x7fe32806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:50.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.737+0000 7fe349e00640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe32806b4d0 0x7fe32806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe338006fd0 tx=0x7fe338008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:50.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.740+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe334040070 con 0x7fe3440cf500 2024-03-06T07:32:50.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:50.990+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe344074ab0 con 0x7fe3440cf500 2024-03-06T07:32:51.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.489+0000 7fe33e000640 1 -- 172.21.15.6:0/3984944026 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe33407a360 con 0x7fe3440cf500 2024-03-06T07:32:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe32806b4d0 msgr2=0x7fe32806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe32806b4d0 0x7fe32806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe338006fd0 tx=0x7fe338008040 comp rx=0 tx=0).stop 2024-03-06T07:32:51.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 msgr2=0x7fe34414c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe34414c1d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe334009870 tx=0x7fe3340046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 shutdown_connections 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe32806b4d0 0x7fe32806d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe34415e260 0x7fe34414cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3440cf500 0x7fe34414c1d0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 --2- 172.21.15.6:0/3984944026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3440b43a0 0x7fe34415e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 >> 172.21.15.6:0/3984944026 conn(0x7fe3440d3ee0 msgr2=0x7fe3440d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 shutdown_connections 2024-03-06T07:32:51.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.492+0000 7fe34c800640 1 -- 172.21.15.6:0/3984944026 wait complete. 2024-03-06T07:32:51.511 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:32:51.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:51.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.715+0000 7f7bf2a00640 1 Processor -- start 2024-03-06T07:32:51.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.715+0000 7f7bf2a00640 1 -- start start 2024-03-06T07:32:51.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7bec15e0b0 0x7f7bec0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bec0b3710 con 0x7f7bec0d4a10 2024-03-06T07:32:51.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bec0b09e0 con 0x7f7bec15e0b0 2024-03-06T07:32:51.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec0b39f0 con 0x7f7bec0d0e50 2024-03-06T07:32:51.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.717+0000 7f7bf2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec0b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:49962/0 (socket says 172.21.15.6:49962) 2024-03-06T07:32:51.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf2000640 1 -- 172.21.15.6:0/371412259 learned_addr learned my addr 172.21.15.6:0/371412259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:51.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf0c00640 1 --1- 172.21.15.6:0/371412259 >> v1:172.21.15.178:6789/0 conn(0x7f7bec0d4a10 0x7f7bec0d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:50242/0 (socket says 172.21.15.6:50242) 2024-03-06T07:32:51.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf1600640 1 --2- 172.21.15.6:0/371412259 >> v2:172.21.15.178:3300/0 conn(0x7f7bec15e0b0 0x7f7bec0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf2000640 1 -- 172.21.15.6:0/371412259 >> v2:172.21.15.178:3300/0 conn(0x7f7bec15e0b0 msgr2=0x7f7bec0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:51.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf2000640 1 --2- 172.21.15.6:0/371412259 >> v2:172.21.15.178:3300/0 conn(0x7f7bec15e0b0 0x7f7bec0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf2000640 1 -- 172.21.15.6:0/371412259 >> v1:172.21.15.178:6789/0 conn(0x7f7bec0d4a10 legacy=0x7f7bec0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.718+0000 7f7bf2000640 1 -- 172.21.15.6:0/371412259 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec0b3cd0 con 0x7f7bec0d0e50 2024-03-06T07:32:51.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.720+0000 7f7bf2000640 1 --2- 172.21.15.6:0/371412259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec0b44d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4ef71a63461825b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.720+0000 7f7bebe00640 1 -- 172.21.15.6:0/371412259 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4034070 con 0x7f7bec0d0e50 2024-03-06T07:32:51.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.720+0000 7f7bebe00640 1 -- 172.21.15.6:0/371412259 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be4030840 con 0x7f7bec0d0e50 2024-03-06T07:32:51.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.720+0000 7f7bebe00640 1 -- 172.21.15.6:0/371412259 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be4030c10 con 0x7f7bec0d0e50 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 -- 172.21.15.6:0/371412259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 msgr2=0x7f7bec0b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/371412259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec0b44d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7be4009870 tx=0x7f7be402b980 comp rx=0 tx=0).stop 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 -- 172.21.15.6:0/371412259 shutdown_connections 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/371412259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec0b44d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/371412259 >> v2:172.21.15.178:3300/0 conn(0x7f7bec15e0b0 0x7f7bec0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 -- 172.21.15.6:0/371412259 >> 172.21.15.6:0/371412259 conn(0x7f7bec0d3d30 msgr2=0x7f7bec0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 -- 172.21.15.6:0/371412259 shutdown_connections 2024-03-06T07:32:51.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.721+0000 7f7bf2a00640 1 -- 172.21.15.6:0/371412259 wait complete. 2024-03-06T07:32:51.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.723+0000 7f7bf2a00640 1 Processor -- start 2024-03-06T07:32:51.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.723+0000 7f7bf2a00640 1 -- start start 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf2a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 0x7f7bec0cc290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf2a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bec15e0b0 0x7f7bec0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf2a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec0cfc60 con 0x7f7bec0d4a10 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf2a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec15e490 con 0x7f7bec15e0b0 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf2a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec0cf850 con 0x7f7bec0d0e50 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 0x7f7bec0cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 0x7f7bec0cc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38018/0 (socket says 172.21.15.6:38018) 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf0c00640 1 -- 172.21.15.6:0/204543958 learned_addr learned my addr 172.21.15.6:0/204543958 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.724+0000 7f7bf1600640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf2000640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bec15e0b0 0x7f7bec0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf0c00640 1 -- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 msgr2=0x7f7bec158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf0c00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf0c00640 1 -- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bec15e0b0 msgr2=0x7f7bec0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf0c00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bec15e0b0 0x7f7bec0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf0c00640 1 -- 172.21.15.6:0/204543958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec0b8690 con 0x7f7bec0d4a10 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf1600640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:51.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf0c00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 0x7f7bec0cc290 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8001eb0 tx=0x7f7bd800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd80074e0 con 0x7f7bec0d4a10 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd80076a0 con 0x7f7bec0d4a10 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8004a20 con 0x7f7bec0d4a10 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.725+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec0b8970 con 0x7f7bec0d4a10 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.726+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec0b8f30 con 0x7f7bec0d4a10 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.727+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bec04f330 con 0x7f7bec0d4a10 2024-03-06T07:32:51.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.728+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bd801a020 con 0x7f7bec0d4a10 2024-03-06T07:32:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.729+0000 7f7be9600640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bcc06b4d0 0x7f7bcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.729+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bd80899d0 con 0x7f7bec0d4a10 2024-03-06T07:32:51.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.730+0000 7f7bf1600640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bcc06b4d0 0x7f7bcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:51.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.731+0000 7f7bf1600640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bcc06b4d0 0x7f7bcc06d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009fd0 tx=0x7f7bdc00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:51.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:51.734+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bd8010300 con 0x7f7bec0d4a10 2024-03-06T07:32:52.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.109+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bec0d2bb0 con 0x7f7bec0d4a10 2024-03-06T07:32:52.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.110+0000 7f7be9600640 1 -- 172.21.15.6:0/204543958 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f7bd805e070 con 0x7f7bec0d4a10 2024-03-06T07:32:52.113 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:52.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/544102913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":544102913},{"type":"v1","addr":"172.21.15.178:6837","nonce":544102913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/833246901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":833246901},{"type":"v1","addr":"172.21.15.178:6836","nonce":833246901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3875786404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3875786404},{"type":"v1","addr":"172.21.15.6:6833","nonce":3875786404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:32:50.992363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7167},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7167":{"gid":7167,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/2859903799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2859903799},{"type":"v1","addr":"172.21.15.6:6835","nonce":2859903799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7167,"qdb_cluster":[7167]},"id":7}]} 2024-03-06T07:32:52.113 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 61 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bcc06b4d0 msgr2=0x7f7bcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bcc06b4d0 0x7f7bcc06d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009fd0 tx=0x7f7bdc00b040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 msgr2=0x7f7bec0cc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 0x7f7bec0cc290 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8001eb0 tx=0x7f7bd800c890 comp rx=0 tx=0).stop 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 shutdown_connections 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bcc06b4d0 0x7f7bcc06d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bec15e0b0 0x7f7bec0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bec0d4a10 0x7f7bec0cc290 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 --2- 172.21.15.6:0/204543958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bec0d0e50 0x7f7bec158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 >> 172.21.15.6:0/204543958 conn(0x7f7bec0d3d30 msgr2=0x7f7bec0b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 shutdown_connections 2024-03-06T07:32:52.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.114+0000 7f7bf2a00640 1 -- 172.21.15.6:0/204543958 wait complete. 2024-03-06T07:32:52.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.333+0000 7f0674800640 1 Processor -- start 2024-03-06T07:32:52.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.333+0000 7f0674800640 1 -- start start 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.334+0000 7f0674800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c0cfd90 0x7f066c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.334+0000 7f0674800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066c0d2760 0x7f066c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.334+0000 7f0674800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0cea50 con 0x7f066c161b40 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.334+0000 7f0674800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066c0b5770 con 0x7f066c0d2760 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.334+0000 7f0674800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0cf900 con 0x7f066c0cfd90 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066c0d2760 0x7f066c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f066c0d2760 0x7f066c0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56348/0 (socket says 172.21.15.6:56348) 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 -- 172.21.15.6:0/1915966030 learned_addr learned my addr 172.21.15.6:0/1915966030 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0671400640 1 --2- 172.21.15.6:0/1915966030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c0cfd90 0x7f066c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 -- 172.21.15.6:0/1915966030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c0cfd90 msgr2=0x7f066c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 --2- 172.21.15.6:0/1915966030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c0cfd90 0x7f066c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 -- 172.21.15.6:0/1915966030 >> v1:172.21.15.178:6789/0 conn(0x7f066c161b40 legacy=0x7f066c0b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.335+0000 7f0672800640 1 -- 172.21.15.6:0/1915966030 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0cecf0 con 0x7f066c0d2760 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.336+0000 7f0672800640 1 --2- 172.21.15.6:0/1915966030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0d2760 0x7f066c0b3740 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f0668009e50 tx=0x7f066802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1734d3c8bf275e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.337+0000 7f0670a00640 1 -- 172.21.15.6:0/1915966030 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066802ca70 con 0x7f066c0d2760 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.337+0000 7f0670a00640 1 -- 172.21.15.6:0/1915966030 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066802cc10 con 0x7f066c0d2760 2024-03-06T07:32:52.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.337+0000 7f0670a00640 1 -- 172.21.15.6:0/1915966030 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06680343f0 con 0x7f066c0d2760 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.337+0000 7f0674800640 1 -- 172.21.15.6:0/1915966030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0d2760 msgr2=0x7f066c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.337+0000 7f0674800640 1 --2- 172.21.15.6:0/1915966030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0d2760 0x7f066c0b3740 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f0668009e50 tx=0x7f066802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.338+0000 7f0674800640 1 -- 172.21.15.6:0/1915966030 shutdown_connections 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.338+0000 7f0674800640 1 --2- 172.21.15.6:0/1915966030 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0d2760 0x7f066c0b3740 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.338+0000 7f0674800640 1 --2- 172.21.15.6:0/1915966030 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c0cfd90 0x7f066c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.338+0000 7f0674800640 1 -- 172.21.15.6:0/1915966030 >> 172.21.15.6:0/1915966030 conn(0x7f066c09d220 msgr2=0x7f066c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.338+0000 7f0674800640 1 -- 172.21.15.6:0/1915966030 shutdown_connections 2024-03-06T07:32:52.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.338+0000 7f0674800640 1 -- 172.21.15.6:0/1915966030 wait complete. 2024-03-06T07:32:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.340+0000 7f0674800640 1 Processor -- start 2024-03-06T07:32:52.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.340+0000 7f0674800640 1 -- start start 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0674800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 0x7f066c0cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0674800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f066c0d2760 0x7f066c0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0674800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c161b40 0x7f066c0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0674800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0cf590 con 0x7f066c0cfd90 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0674800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0cf2b0 con 0x7f066c0d2760 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0674800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0b2ff0 con 0x7f066c161b40 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0671e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 0x7f066c0cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0671e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 0x7f066c0cbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56350/0 (socket says 172.21.15.6:56350) 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.342+0000 7f0671e00640 1 -- 172.21.15.6:0/3987523831 learned_addr learned my addr 172.21.15.6:0/3987523831 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671400640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f066c0d2760 0x7f066c0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0672800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c161b40 0x7f066c0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671e00640 1 -- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c161b40 msgr2=0x7f066c0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671e00640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c161b40 0x7f066c0cc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671e00640 1 -- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f066c0d2760 msgr2=0x7f066c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671e00640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f066c0d2760 0x7f066c0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671e00640 1 -- 172.21.15.6:0/3987523831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0cb060 con 0x7f066c0cfd90 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671400640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f066c0d2760 0x7f066c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0671e00640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 0x7f066c0cbe20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f066000c460 tx=0x7f066000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660016020 con 0x7f066c0cfd90 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0660012870 con 0x7f066c0cfd90 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660012c60 con 0x7f066c0cfd90 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.343+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c0cb340 con 0x7f066c0cfd90 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.344+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c0cd550 con 0x7f066c0cfd90 2024-03-06T07:32:52.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.345+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066c04f3a0 con 0x7f066c0cfd90 2024-03-06T07:32:52.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.346+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0660019020 con 0x7f066c0cfd90 2024-03-06T07:32:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.347+0000 7f0666000640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f065006b4d0 0x7f065006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.348+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0660089cd0 con 0x7f066c0cfd90 2024-03-06T07:32:52.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.348+0000 7f0671400640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f065006b4d0 0x7f065006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.349+0000 7f0671400640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f065006b4d0 0x7f065006d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f065c006fd0 tx=0x7f065c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.352+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0660020070 con 0x7f066c0cfd90 2024-03-06T07:32:52.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.554+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f066c0d1df0 con 0x7f066c0cfd90 2024-03-06T07:32:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.557+0000 7f0666000640 1 -- 172.21.15.6:0/3987523831 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16536 (secure 0 0 0) 0x7f0660058ab0 con 0x7f066c0cfd90 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:52.560 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":93,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:32:49.459105+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f065006b4d0 msgr2=0x7f065006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f065006b4d0 0x7f065006d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f065c006fd0 tx=0x7f065c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 msgr2=0x7f066c0cbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 0x7f066c0cbe20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f066000c460 tx=0x7f066000c920 comp rx=0 tx=0).stop 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 shutdown_connections 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f065006b4d0 0x7f065006d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f066c0d2760 0x7f066c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f066c161b40 0x7f066c0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 --2- 172.21.15.6:0/3987523831 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f066c0cfd90 0x7f066c0cbe20 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 >> 172.21.15.6:0/3987523831 conn(0x7f066c09d220 msgr2=0x7f066c0b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 shutdown_connections 2024-03-06T07:32:52.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.560+0000 7f0674800640 1 -- 172.21.15.6:0/3987523831 wait complete. 2024-03-06T07:32:52.580 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:32:52.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:52.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:52.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:52.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.793+0000 7f5e73e00640 1 Processor -- start 2024-03-06T07:32:52.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.794+0000 7f5e73e00640 1 -- start start 2024-03-06T07:32:52.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e73e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e6c0b3c90 0x7f5e6c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e73e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c0b4860 con 0x7f5e6c0cf5c0 2024-03-06T07:32:52.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e73e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c0b4b40 con 0x7f5e6c071050 2024-03-06T07:32:52.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e73e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c0ce160 con 0x7f5e6c0b3c90 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e71400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e6c0b3c90 0x7f5e6c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e71400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5e6c0b3c90 0x7f5e6c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56354/0 (socket says 172.21.15.6:56354) 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e71400640 1 -- 172.21.15.6:0/516477490 learned_addr learned my addr 172.21.15.6:0/516477490 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.796+0000 7f5e70a00640 1 --1- 172.21.15.6:0/516477490 >> v1:172.21.15.6:6789/0 conn(0x7f5e6c0cf5c0 0x7f5e6c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55204/0 (socket says 172.21.15.6:55204) 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.797+0000 7f5e71400640 1 -- 172.21.15.6:0/516477490 >> v1:172.21.15.178:6789/0 conn(0x7f5e6c071050 legacy=0x7f5e6c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.797+0000 7f5e71400640 1 -- 172.21.15.6:0/516477490 >> v1:172.21.15.6:6789/0 conn(0x7f5e6c0cf5c0 legacy=0x7f5e6c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.797+0000 7f5e71400640 1 -- 172.21.15.6:0/516477490 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0b4e20 con 0x7f5e6c0b3c90 2024-03-06T07:32:52.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.799+0000 7f5e71400640 1 --2- 172.21.15.6:0/516477490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c0cdc20 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009870 tx=0x7f5e6002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354b851d8cf34cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.799+0000 7f5e6be00640 1 -- 172.21.15.6:0/516477490 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6002f840 con 0x7f5e6c0b3c90 2024-03-06T07:32:52.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.799+0000 7f5e6be00640 1 -- 172.21.15.6:0/516477490 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6002f9e0 con 0x7f5e6c0b3c90 2024-03-06T07:32:52.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.799+0000 7f5e6be00640 1 -- 172.21.15.6:0/516477490 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6002fdb0 con 0x7f5e6c0b3c90 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 -- 172.21.15.6:0/516477490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c0b3c90 msgr2=0x7f5e6c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 --2- 172.21.15.6:0/516477490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c0cdc20 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009870 tx=0x7f5e6002ba20 comp rx=0 tx=0).stop 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 -- 172.21.15.6:0/516477490 shutdown_connections 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 --2- 172.21.15.6:0/516477490 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c0cdc20 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 -- 172.21.15.6:0/516477490 >> 172.21.15.6:0/516477490 conn(0x7f5e6c06afb0 msgr2=0x7f5e6c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 -- 172.21.15.6:0/516477490 shutdown_connections 2024-03-06T07:32:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.800+0000 7f5e73e00640 1 -- 172.21.15.6:0/516477490 wait complete. 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.803+0000 7f5e73e00640 1 Processor -- start 2024-03-06T07:32:52.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.803+0000 7f5e73e00640 1 -- start start 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e73e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 0x7f5e6c154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e73e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c1e45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e73e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e6c0cf5c0 0x7f5e6c1e4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e73e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c071ba0 con 0x7f5e6c071050 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e73e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c15b340 con 0x7f5e6c0b3c90 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e73e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c071d20 con 0x7f5e6c0cf5c0 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e71400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 0x7f5e6c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e71400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 0x7f5e6c154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56356/0 (socket says 172.21.15.6:56356) 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.805+0000 7f5e71400640 1 -- 172.21.15.6:0/1464323646 learned_addr learned my addr 172.21.15.6:0/1464323646 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:52.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e70a00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c1e45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e6c0cf5c0 0x7f5e6c1e4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71400640 1 -- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e6c0cf5c0 msgr2=0x7f5e6c1e4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71400640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e6c0cf5c0 0x7f5e6c1e4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71400640 1 -- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e6c0b3c90 msgr2=0x7f5e6c1e45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71400640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c1e45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:52.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71400640 1 -- 172.21.15.6:0/1464323646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c147550 con 0x7f5e6c071050 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.806+0000 7f5e71400640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 0x7f5e6c154380 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f5e60000c00 tx=0x7f5e60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.807+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60041400 con 0x7f5e6c071050 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.807+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e60046070 con 0x7f5e6c071050 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.807+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60041800 con 0x7f5e6c071050 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.807+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c147830 con 0x7f5e6c071050 2024-03-06T07:32:52.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.807+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c147df0 con 0x7f5e6c071050 2024-03-06T07:32:52.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.808+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c069510 con 0x7f5e6c071050 2024-03-06T07:32:52.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.809+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e60040070 con 0x7f5e6c071050 2024-03-06T07:32:52.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.810+0000 7f5e69600640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e4c06b4d0 0x7f5e4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:52.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.810+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e600acad0 con 0x7f5e6c071050 2024-03-06T07:32:52.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.810+0000 7f5e70a00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e4c06b4d0 0x7f5e4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:52.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.811+0000 7f5e70a00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e4c06b4d0 0x7f5e4c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c09cbf0 tx=0x7f5e58007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:52.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:52.815+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e60033030 con 0x7f5e6c071050 2024-03-06T07:32:53.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.011+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e6c04f3a0 con 0x7f5e4c06b4d0 2024-03-06T07:32:53.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.030+0000 7f5e69600640 1 -- 172.21.15.6:0/1464323646 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+250523 (secure 0 0 0) 0x7f5e6c04f3a0 con 0x7f5e4c06b4d0 2024-03-06T07:32:53.036 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:53.038 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:32:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e4c06b4d0 msgr2=0x7f5e4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e4c06b4d0 0x7f5e4c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c09cbf0 tx=0x7f5e58007ce0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 msgr2=0x7f5e6c154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 0x7f5e6c154380 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f5e60000c00 tx=0x7f5e60004730 comp rx=0 tx=0).stop 2024-03-06T07:32:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 shutdown_connections 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5e4c06b4d0 0x7f5e4c06d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5e6c0b3c90 0x7f5e6c1e45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5e6c0cf5c0 0x7f5e6c1e4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 --2- 172.21.15.6:0/1464323646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c071050 0x7f5e6c154380 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 >> 172.21.15.6:0/1464323646 conn(0x7f5e6c06afb0 msgr2=0x7f5e6c09cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 shutdown_connections 2024-03-06T07:32:53.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.037+0000 7f5e73e00640 1 -- 172.21.15.6:0/1464323646 wait complete. 2024-03-06T07:32:53.061 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-03-06T07:32:51.555873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":189,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276168,"kb_used_data":10076,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698552,"statfs":{"total":773094113280,"available":772811317248,"internally_reserved":0,"allocated":10317824,"data_stored":5032718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.150908"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.434548+0000","last_change":"2024-03-06T07:32:47.434548+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.434548+0000","last_undegraded":"2024-03-06T07:32:47.434548+0000","last_fullsized":"2024-03-06T07:32:47.434548+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047747+0000","last_change":"2024-03-06T07:32:48.477759+0000","last_active":"2024-03-06T07:32:50.047747+0000","last_peered":"2024-03-06T07:32:50.047747+0000","last_clean":"2024-03-06T07:32:50.047747+0000","last_became_active":"2024-03-06T07:32:48.477057+0000","last_became_peered":"2024-03-06T07:32:48.477057+0000","last_unstale":"2024-03-06T07:32:50.047747+0000","last_undegraded":"2024-03-06T07:32:50.047747+0000","last_fullsized":"2024-03-06T07:32:50.047747+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:09.016281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.432321+0000","last_change":"2024-03-06T07:32:47.432321+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.432321+0000","last_undegraded":"2024-03-06T07:32:47.432321+0000","last_fullsized":"2024-03-06T07:32:47.432321+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475024+0000","last_change":"2024-03-06T07:32:48.482309+0000","last_active":"2024-03-06T07:32:49.475024+0000","last_peered":"2024-03-06T07:32:49.475024+0000","last_clean":"2024-03-06T07:32:49.475024+0000","last_became_active":"2024-03-06T07:32:48.481368+0000","last_became_peered":"2024-03-06T07:32:48.481368+0000","last_unstale":"2024-03-06T07:32:49.475024+0000","last_undegraded":"2024-03-06T07:32:49.475024+0000","last_fullsized":"2024-03-06T07:32:49.475024+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:02.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048459+0000","last_change":"2024-03-06T07:32:48.475342+0000","last_active":"2024-03-06T07:32:50.048459+0000","last_peered":"2024-03-06T07:32:50.048459+0000","last_clean":"2024-03-06T07:32:50.048459+0000","last_became_active":"2024-03-06T07:32:48.475052+0000","last_became_peered":"2024-03-06T07:32:48.475052+0000","last_unstale":"2024-03-06T07:32:50.048459+0000","last_undegraded":"2024-03-06T07:32:50.048459+0000","last_fullsized":"2024-03-06T07:32:50.048459+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:24.300728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475116+0000","last_change":"2024-03-06T07:32:48.481391+0000","last_active":"2024-03-06T07:32:49.475116+0000","last_peered":"2024-03-06T07:32:49.475116+0000","last_clean":"2024-03-06T07:32:49.475116+0000","last_became_active":"2024-03-06T07:32:48.481067+0000","last_became_peered":"2024-03-06T07:32:48.481067+0000","last_unstale":"2024-03-06T07:32:49.475116+0000","last_undegraded":"2024-03-06T07:32:49.475116+0000","last_fullsized":"2024-03-06T07:32:49.475116+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:54.546027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474755+0000","last_change":"2024-03-06T07:32:48.480318+0000","last_active":"2024-03-06T07:32:49.474755+0000","last_peered":"2024-03-06T07:32:49.474755+0000","last_clean":"2024-03-06T07:32:49.474755+0000","last_became_active":"2024-03-06T07:32:48.478766+0000","last_became_peered":"2024-03-06T07:32:48.478766+0000","last_unstale":"2024-03-06T07:32:49.474755+0000","last_undegraded":"2024-03-06T07:32:49.474755+0000","last_fullsized":"2024-03-06T07:32:49.474755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:32.012886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046874+0000","last_change":"2024-03-06T07:32:48.476674+0000","last_active":"2024-03-06T07:32:50.046874+0000","last_peered":"2024-03-06T07:32:50.046874+0000","last_clean":"2024-03-06T07:32:50.046874+0000","last_became_active":"2024-03-06T07:32:48.472921+0000","last_became_peered":"2024-03-06T07:32:48.472921+0000","last_unstale":"2024-03-06T07:32:50.046874+0000","last_undegraded":"2024-03-06T07:32:50.046874+0000","last_fullsized":"2024-03-06T07:32:50.046874+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:57.480021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.051123+0000","last_change":"2024-03-06T07:32:48.488539+0000","last_active":"2024-03-06T07:32:50.051123+0000","last_peered":"2024-03-06T07:32:50.051123+0000","last_clean":"2024-03-06T07:32:50.051123+0000","last_became_active":"2024-03-06T07:32:48.488181+0000","last_became_peered":"2024-03-06T07:32:48.488181+0000","last_unstale":"2024-03-06T07:32:50.051123+0000","last_undegraded":"2024-03-06T07:32:50.051123+0000","last_fullsized":"2024-03-06T07:32:50.051123+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:54.546027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048514+0000","last_change":"2024-03-06T07:32:48.475586+0000","last_active":"2024-03-06T07:32:50.048514+0000","last_peered":"2024-03-06T07:32:50.048514+0000","last_clean":"2024-03-06T07:32:50.048514+0000","last_became_active":"2024-03-06T07:32:48.472161+0000","last_became_peered":"2024-03-06T07:32:48.472161+0000","last_unstale":"2024-03-06T07:32:50.048514+0000","last_undegraded":"2024-03-06T07:32:50.048514+0000","last_fullsized":"2024-03-06T07:32:50.048514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:02.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475612+0000","last_change":"2024-03-06T07:32:48.474439+0000","last_active":"2024-03-06T07:32:49.475612+0000","last_peered":"2024-03-06T07:32:49.475612+0000","last_clean":"2024-03-06T07:32:49.475612+0000","last_became_active":"2024-03-06T07:32:48.473837+0000","last_became_peered":"2024-03-06T07:32:48.473837+0000","last_unstale":"2024-03-06T07:32:49.475612+0000","last_undegraded":"2024-03-06T07:32:49.475612+0000","last_fullsized":"2024-03-06T07:32:49.475612+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:55.621564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050695+0000","last_change":"2024-03-06T07:32:48.473220+0000","last_active":"2024-03-06T07:32:50.050695+0000","last_peered":"2024-03-06T07:32:50.050695+0000","last_clean":"2024-03-06T07:32:50.050695+0000","last_became_active":"2024-03-06T07:32:48.472930+0000","last_became_peered":"2024-03-06T07:32:48.472930+0000","last_unstale":"2024-03-06T07:32:50.050695+0000","last_undegraded":"2024-03-06T07:32:50.050695+0000","last_fullsized":"2024-03-06T07:32:50.050695+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:12.461489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474705+0000","last_change":"2024-03-06T07:32:48.477921+0000","last_active":"2024-03-06T07:32:49.474705+0000","last_peered":"2024-03-06T07:32:49.474705+0000","last_clean":"2024-03-06T07:32:49.474705+0000","last_became_active":"2024-03-06T07:32:48.477410+0000","last_became_peered":"2024-03-06T07:32:48.477410+0000","last_unstale":"2024-03-06T07:32:49.474705+0000","last_undegraded":"2024-03-06T07:32:49.474705+0000","last_fullsized":"2024-03-06T07:32:49.474705+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:24.300728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.432015+0000","last_change":"2024-03-06T07:32:47.432015+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.432015+0000","last_undegraded":"2024-03-06T07:32:47.432015+0000","last_fullsized":"2024-03-06T07:32:47.432015+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474876+0000","last_change":"2024-03-06T07:32:48.479483+0000","last_active":"2024-03-06T07:32:49.474876+0000","last_peered":"2024-03-06T07:32:49.474876+0000","last_clean":"2024-03-06T07:32:49.474876+0000","last_became_active":"2024-03-06T07:32:48.478879+0000","last_became_peered":"2024-03-06T07:32:48.478879+0000","last_unstale":"2024-03-06T07:32:49.474876+0000","last_undegraded":"2024-03-06T07:32:49.474876+0000","last_fullsized":"2024-03-06T07:32:49.474876+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:50.841839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474977+0000","last_change":"2024-03-06T07:32:48.473952+0000","last_active":"2024-03-06T07:32:49.474977+0000","last_peered":"2024-03-06T07:32:49.474977+0000","last_clean":"2024-03-06T07:32:49.474977+0000","last_became_active":"2024-03-06T07:32:48.473130+0000","last_became_peered":"2024-03-06T07:32:48.473130+0000","last_unstale":"2024-03-06T07:32:49.474977+0000","last_undegraded":"2024-03-06T07:32:49.474977+0000","last_fullsized":"2024-03-06T07:32:49.474977+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:26.058500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047941+0000","last_change":"2024-03-06T07:32:48.468475+0000","last_active":"2024-03-06T07:32:50.047941+0000","last_peered":"2024-03-06T07:32:50.047941+0000","last_clean":"2024-03-06T07:32:50.047941+0000","last_became_active":"2024-03-06T07:32:48.468154+0000","last_became_peered":"2024-03-06T07:32:48.468154+0000","last_unstale":"2024-03-06T07:32:50.047941+0000","last_undegraded":"2024-03-06T07:32:50.047941+0000","last_fullsized":"2024-03-06T07:32:50.047941+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:21.501947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474896+0000","last_change":"2024-03-06T07:32:48.481353+0000","last_active":"2024-03-06T07:32:49.474896+0000","last_peered":"2024-03-06T07:32:49.474896+0000","last_clean":"2024-03-06T07:32:49.474896+0000","last_became_active":"2024-03-06T07:32:48.479874+0000","last_became_peered":"2024-03-06T07:32:48.479874+0000","last_unstale":"2024-03-06T07:32:49.474896+0000","last_undegraded":"2024-03-06T07:32:49.474896+0000","last_fullsized":"2024-03-06T07:32:49.474896+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:57.480021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.421903+0000","last_change":"2024-03-06T07:32:47.421903+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.421903+0000","last_undegraded":"2024-03-06T07:32:47.421903+0000","last_fullsized":"2024-03-06T07:32:47.421903+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050662+0000","last_change":"2024-03-06T07:32:48.489733+0000","last_active":"2024-03-06T07:32:50.050662+0000","last_peered":"2024-03-06T07:32:50.050662+0000","last_clean":"2024-03-06T07:32:50.050662+0000","last_became_active":"2024-03-06T07:32:48.488129+0000","last_became_peered":"2024-03-06T07:32:48.488129+0000","last_unstale":"2024-03-06T07:32:50.050662+0000","last_undegraded":"2024-03-06T07:32:50.050662+0000","last_fullsized":"2024-03-06T07:32:50.050662+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:32.012886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.426508+0000","last_change":"2024-03-06T07:32:47.426508+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.426508+0000","last_undegraded":"2024-03-06T07:32:47.426508+0000","last_fullsized":"2024-03-06T07:32:47.426508+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046695+0000","last_change":"2024-03-06T07:32:48.476584+0000","last_active":"2024-03-06T07:32:50.046695+0000","last_peered":"2024-03-06T07:32:50.046695+0000","last_clean":"2024-03-06T07:32:50.046695+0000","last_became_active":"2024-03-06T07:32:48.476309+0000","last_became_peered":"2024-03-06T07:32:48.476309+0000","last_unstale":"2024-03-06T07:32:50.046695+0000","last_undegraded":"2024-03-06T07:32:50.046695+0000","last_fullsized":"2024-03-06T07:32:50.046695+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:11.960503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046080+0000","last_change":"2024-03-06T07:32:48.478660+0000","last_active":"2024-03-06T07:32:50.046080+0000","last_peered":"2024-03-06T07:32:50.046080+0000","last_clean":"2024-03-06T07:32:50.046080+0000","last_became_active":"2024-03-06T07:32:48.475347+0000","last_became_peered":"2024-03-06T07:32:48.475347+0000","last_unstale":"2024-03-06T07:32:50.046080+0000","last_undegraded":"2024-03-06T07:32:50.046080+0000","last_fullsized":"2024-03-06T07:32:50.046080+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:50.841839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475213+0000","last_change":"2024-03-06T07:32:48.475228+0000","last_active":"2024-03-06T07:32:49.475213+0000","last_peered":"2024-03-06T07:32:49.475213+0000","last_clean":"2024-03-06T07:32:49.475213+0000","last_became_active":"2024-03-06T07:32:48.473605+0000","last_became_peered":"2024-03-06T07:32:48.473605+0000","last_unstale":"2024-03-06T07:32:49.475213+0000","last_undegraded":"2024-03-06T07:32:49.475213+0000","last_fullsized":"2024-03-06T07:32:49.475213+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:08.607260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049172+0000","last_change":"2024-03-06T07:32:48.468106+0000","last_active":"2024-03-06T07:32:50.049172+0000","last_peered":"2024-03-06T07:32:50.049172+0000","last_clean":"2024-03-06T07:32:50.049172+0000","last_became_active":"2024-03-06T07:32:48.467651+0000","last_became_peered":"2024-03-06T07:32:48.467651+0000","last_unstale":"2024-03-06T07:32:50.049172+0000","last_undegraded":"2024-03-06T07:32:50.049172+0000","last_fullsized":"2024-03-06T07:32:50.049172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:11.960503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047900+0000","last_change":"2024-03-06T07:32:48.475360+0000","last_active":"2024-03-06T07:32:50.047900+0000","last_peered":"2024-03-06T07:32:50.047900+0000","last_clean":"2024-03-06T07:32:50.047900+0000","last_became_active":"2024-03-06T07:32:48.474357+0000","last_became_peered":"2024-03-06T07:32:48.474357+0000","last_unstale":"2024-03-06T07:32:50.047900+0000","last_undegraded":"2024-03-06T07:32:50.047900+0000","last_fullsized":"2024-03-06T07:32:50.047900+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.225453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475631+0000","last_change":"2024-03-06T07:32:48.487211+0000","last_active":"2024-03-06T07:32:49.475631+0000","last_peered":"2024-03-06T07:32:49.475631+0000","last_clean":"2024-03-06T07:32:49.475631+0000","last_became_active":"2024-03-06T07:32:48.486949+0000","last_became_peered":"2024-03-06T07:32:48.486949+0000","last_unstale":"2024-03-06T07:32:49.475631+0000","last_undegraded":"2024-03-06T07:32:49.475631+0000","last_fullsized":"2024-03-06T07:32:49.475631+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:29.059277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048362+0000","last_change":"2024-03-06T07:32:48.477073+0000","last_active":"2024-03-06T07:32:50.048362+0000","last_peered":"2024-03-06T07:32:50.048362+0000","last_clean":"2024-03-06T07:32:50.048362+0000","last_became_active":"2024-03-06T07:32:48.476280+0000","last_became_peered":"2024-03-06T07:32:48.476280+0000","last_unstale":"2024-03-06T07:32:50.048362+0000","last_undegraded":"2024-03-06T07:32:50.048362+0000","last_fullsized":"2024-03-06T07:32:50.048362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:11.710892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047088+0000","last_change":"2024-03-06T07:32:48.474842+0000","last_active":"2024-03-06T07:32:50.047088+0000","last_peered":"2024-03-06T07:32:50.047088+0000","last_clean":"2024-03-06T07:32:50.047088+0000","last_became_active":"2024-03-06T07:32:48.474542+0000","last_became_peered":"2024-03-06T07:32:48.474542+0000","last_unstale":"2024-03-06T07:32:50.047088+0000","last_undegraded":"2024-03-06T07:32:50.047088+0000","last_fullsized":"2024-03-06T07:32:50.047088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:43.593642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.431177+0000","last_change":"2024-03-06T07:32:47.431177+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.431177+0000","last_undegraded":"2024-03-06T07:32:47.431177+0000","last_fullsized":"2024-03-06T07:32:47.431177+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474862+0000","last_change":"2024-03-06T07:32:48.471152+0000","last_active":"2024-03-06T07:32:49.474862+0000","last_peered":"2024-03-06T07:32:49.474862+0000","last_clean":"2024-03-06T07:32:49.474862+0000","last_became_active":"2024-03-06T07:32:48.470786+0000","last_became_peered":"2024-03-06T07:32:48.470786+0000","last_unstale":"2024-03-06T07:32:49.474862+0000","last_undegraded":"2024-03-06T07:32:49.474862+0000","last_fullsized":"2024-03-06T07:32:49.474862+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:11.710892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.429717+0000","last_change":"2024-03-06T07:32:47.429717+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.429717+0000","last_undegraded":"2024-03-06T07:32:47.429717+0000","last_fullsized":"2024-03-06T07:32:47.429717+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.431771+0000","last_change":"2024-03-06T07:32:47.431771+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.431771+0000","last_undegraded":"2024-03-06T07:32:47.431771+0000","last_fullsized":"2024-03-06T07:32:47.431771+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047815+0000","last_change":"2024-03-06T07:32:47.443466+0000","last_active":"2024-03-06T07:32:50.047815+0000","last_peered":"2024-03-06T07:32:50.047815+0000","last_clean":"2024-03-06T07:32:50.047815+0000","last_became_active":"2024-03-06T07:32:47.443209+0000","last_became_peered":"2024-03-06T07:32:47.443209+0000","last_unstale":"2024-03-06T07:32:50.047815+0000","last_undegraded":"2024-03-06T07:32:50.047815+0000","last_fullsized":"2024-03-06T07:32:50.047815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:05.073516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475223+0000","last_change":"2024-03-06T07:32:48.465838+0000","last_active":"2024-03-06T07:32:49.475223+0000","last_peered":"2024-03-06T07:32:49.475223+0000","last_clean":"2024-03-06T07:32:49.475223+0000","last_became_active":"2024-03-06T07:32:48.465431+0000","last_became_peered":"2024-03-06T07:32:48.465431+0000","last_unstale":"2024-03-06T07:32:49.475223+0000","last_undegraded":"2024-03-06T07:32:49.475223+0000","last_fullsized":"2024-03-06T07:32:49.475223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:43.593642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475018+0000","last_change":"2024-03-06T07:32:47.446298+0000","last_active":"2024-03-06T07:32:49.475018+0000","last_peered":"2024-03-06T07:32:49.475018+0000","last_clean":"2024-03-06T07:32:49.475018+0000","last_became_active":"2024-03-06T07:32:47.446022+0000","last_became_peered":"2024-03-06T07:32:47.446022+0000","last_unstale":"2024-03-06T07:32:49.475018+0000","last_undegraded":"2024-03-06T07:32:49.475018+0000","last_fullsized":"2024-03-06T07:32:49.475018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:45.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.429312+0000","last_change":"2024-03-06T07:32:47.429312+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.429312+0000","last_undegraded":"2024-03-06T07:32:47.429312+0000","last_fullsized":"2024-03-06T07:32:47.429312+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047377+0000","last_change":"2024-03-06T07:32:47.442753+0000","last_active":"2024-03-06T07:32:50.047377+0000","last_peered":"2024-03-06T07:32:50.047377+0000","last_clean":"2024-03-06T07:32:50.047377+0000","last_became_active":"2024-03-06T07:32:47.442432+0000","last_became_peered":"2024-03-06T07:32:47.442432+0000","last_unstale":"2024-03-06T07:32:50.047377+0000","last_undegraded":"2024-03-06T07:32:50.047377+0000","last_fullsized":"2024-03-06T07:32:50.047377+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:57.754771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.051016+0000","last_change":"2024-03-06T07:32:48.480970+0000","last_active":"2024-03-06T07:32:50.051016+0000","last_peered":"2024-03-06T07:32:50.051016+0000","last_clean":"2024-03-06T07:32:50.051016+0000","last_became_active":"2024-03-06T07:32:48.480572+0000","last_became_peered":"2024-03-06T07:32:48.480572+0000","last_unstale":"2024-03-06T07:32:50.051016+0000","last_undegraded":"2024-03-06T07:32:50.051016+0000","last_fullsized":"2024-03-06T07:32:50.051016+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:26.058500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046526+0000","last_change":"2024-03-06T07:32:47.448905+0000","last_active":"2024-03-06T07:32:50.046526+0000","last_peered":"2024-03-06T07:32:50.046526+0000","last_clean":"2024-03-06T07:32:50.046526+0000","last_became_active":"2024-03-06T07:32:47.448293+0000","last_became_peered":"2024-03-06T07:32:47.448293+0000","last_unstale":"2024-03-06T07:32:50.046526+0000","last_undegraded":"2024-03-06T07:32:50.046526+0000","last_fullsized":"2024-03-06T07:32:50.046526+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.523954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048159+0000","last_change":"2024-03-06T07:32:48.477826+0000","last_active":"2024-03-06T07:32:50.048159+0000","last_peered":"2024-03-06T07:32:50.048159+0000","last_clean":"2024-03-06T07:32:50.048159+0000","last_became_active":"2024-03-06T07:32:48.477514+0000","last_became_peered":"2024-03-06T07:32:48.477514+0000","last_unstale":"2024-03-06T07:32:50.048159+0000","last_undegraded":"2024-03-06T07:32:50.048159+0000","last_fullsized":"2024-03-06T07:32:50.048159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:05.662945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.441303+0000","last_change":"2024-03-06T07:32:47.435176+0000","last_active":"2024-03-06T07:32:47.441303+0000","last_peered":"2024-03-06T07:32:47.441303+0000","last_clean":"2024-03-06T07:32:47.441303+0000","last_became_active":"2024-03-06T07:32:47.433205+0000","last_became_peered":"2024-03-06T07:32:47.433205+0000","last_unstale":"2024-03-06T07:32:47.441303+0000","last_undegraded":"2024-03-06T07:32:47.441303+0000","last_fullsized":"2024-03-06T07:32:47.441303+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:35.083654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474917+0000","last_change":"2024-03-06T07:32:48.472777+0000","last_active":"2024-03-06T07:32:49.474917+0000","last_peered":"2024-03-06T07:32:49.474917+0000","last_clean":"2024-03-06T07:32:49.474917+0000","last_became_active":"2024-03-06T07:32:48.472455+0000","last_became_peered":"2024-03-06T07:32:48.472455+0000","last_unstale":"2024-03-06T07:32:49.474917+0000","last_undegraded":"2024-03-06T07:32:49.474917+0000","last_fullsized":"2024-03-06T07:32:49.474917+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:12.461489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046742+0000","last_change":"2024-03-06T07:32:47.449280+0000","last_active":"2024-03-06T07:32:50.046742+0000","last_peered":"2024-03-06T07:32:50.046742+0000","last_clean":"2024-03-06T07:32:50.046742+0000","last_became_active":"2024-03-06T07:32:47.447849+0000","last_became_peered":"2024-03-06T07:32:47.447849+0000","last_unstale":"2024-03-06T07:32:50.046742+0000","last_undegraded":"2024-03-06T07:32:50.046742+0000","last_fullsized":"2024-03-06T07:32:50.046742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:32.610977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474676+0000","last_change":"2024-03-06T07:32:48.470481+0000","last_active":"2024-03-06T07:32:49.474676+0000","last_peered":"2024-03-06T07:32:49.474676+0000","last_clean":"2024-03-06T07:32:49.474676+0000","last_became_active":"2024-03-06T07:32:48.469803+0000","last_became_peered":"2024-03-06T07:32:48.469803+0000","last_unstale":"2024-03-06T07:32:49.474676+0000","last_undegraded":"2024-03-06T07:32:49.474676+0000","last_fullsized":"2024-03-06T07:32:49.474676+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:40.747320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047331+0000","last_change":"2024-03-06T07:32:47.443161+0000","last_active":"2024-03-06T07:32:50.047331+0000","last_peered":"2024-03-06T07:32:50.047331+0000","last_clean":"2024-03-06T07:32:50.047331+0000","last_became_active":"2024-03-06T07:32:47.442877+0000","last_became_peered":"2024-03-06T07:32:47.442877+0000","last_unstale":"2024-03-06T07:32:50.047331+0000","last_undegraded":"2024-03-06T07:32:50.047331+0000","last_fullsized":"2024-03-06T07:32:50.047331+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:38.261444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047094+0000","last_change":"2024-03-06T07:32:48.484896+0000","last_active":"2024-03-06T07:32:50.047094+0000","last_peered":"2024-03-06T07:32:50.047094+0000","last_clean":"2024-03-06T07:32:50.047094+0000","last_became_active":"2024-03-06T07:32:48.484473+0000","last_became_peered":"2024-03-06T07:32:48.484473+0000","last_unstale":"2024-03-06T07:32:50.047094+0000","last_undegraded":"2024-03-06T07:32:50.047094+0000","last_fullsized":"2024-03-06T07:32:50.047094+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:16.329953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.422758+0000","last_change":"2024-03-06T07:32:47.421031+0000","last_active":"2024-03-06T07:32:47.422758+0000","last_peered":"2024-03-06T07:32:47.422758+0000","last_clean":"2024-03-06T07:32:47.422758+0000","last_became_active":"2024-03-06T07:32:47.420724+0000","last_became_peered":"2024-03-06T07:32:47.420724+0000","last_unstale":"2024-03-06T07:32:47.422758+0000","last_undegraded":"2024-03-06T07:32:47.422758+0000","last_fullsized":"2024-03-06T07:32:47.422758+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:58.611627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048203+0000","last_change":"2024-03-06T07:32:48.478123+0000","last_active":"2024-03-06T07:32:50.048203+0000","last_peered":"2024-03-06T07:32:50.048203+0000","last_clean":"2024-03-06T07:32:50.048203+0000","last_became_active":"2024-03-06T07:32:48.474915+0000","last_became_peered":"2024-03-06T07:32:48.474915+0000","last_unstale":"2024-03-06T07:32:50.048203+0000","last_undegraded":"2024-03-06T07:32:50.048203+0000","last_fullsized":"2024-03-06T07:32:50.048203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:40.486151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.450846+0000","last_change":"2024-03-06T07:32:47.448173+0000","last_active":"2024-03-06T07:32:47.450846+0000","last_peered":"2024-03-06T07:32:47.450846+0000","last_clean":"2024-03-06T07:32:47.450846+0000","last_became_active":"2024-03-06T07:32:47.445327+0000","last_became_peered":"2024-03-06T07:32:47.445327+0000","last_unstale":"2024-03-06T07:32:47.450846+0000","last_undegraded":"2024-03-06T07:32:47.450846+0000","last_fullsized":"2024-03-06T07:32:47.450846+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:48.215815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.051209+0000","last_change":"2024-03-06T07:32:48.482589+0000","last_active":"2024-03-06T07:32:50.051209+0000","last_peered":"2024-03-06T07:32:50.051209+0000","last_clean":"2024-03-06T07:32:50.051209+0000","last_became_active":"2024-03-06T07:32:48.481339+0000","last_became_peered":"2024-03-06T07:32:48.481339+0000","last_unstale":"2024-03-06T07:32:50.051209+0000","last_undegraded":"2024-03-06T07:32:50.051209+0000","last_fullsized":"2024-03-06T07:32:50.051209+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:08.607260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474944+0000","last_change":"2024-03-06T07:32:47.449558+0000","last_active":"2024-03-06T07:32:49.474944+0000","last_peered":"2024-03-06T07:32:49.474944+0000","last_clean":"2024-03-06T07:32:49.474944+0000","last_became_active":"2024-03-06T07:32:47.447476+0000","last_became_peered":"2024-03-06T07:32:47.447476+0000","last_unstale":"2024-03-06T07:32:49.474944+0000","last_undegraded":"2024-03-06T07:32:49.474944+0000","last_fullsized":"2024-03-06T07:32:49.474944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:57.754771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048641+0000","last_change":"2024-03-06T07:32:48.469589+0000","last_active":"2024-03-06T07:32:50.048641+0000","last_peered":"2024-03-06T07:32:50.048641+0000","last_clean":"2024-03-06T07:32:50.048641+0000","last_became_active":"2024-03-06T07:32:48.469119+0000","last_became_peered":"2024-03-06T07:32:48.469119+0000","last_unstale":"2024-03-06T07:32:50.048641+0000","last_undegraded":"2024-03-06T07:32:50.048641+0000","last_fullsized":"2024-03-06T07:32:50.048641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:50.841839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475933+0000","last_change":"2024-03-06T07:32:47.450101+0000","last_active":"2024-03-06T07:32:49.475933+0000","last_peered":"2024-03-06T07:32:49.475933+0000","last_clean":"2024-03-06T07:32:49.475933+0000","last_became_active":"2024-03-06T07:32:47.449794+0000","last_became_peered":"2024-03-06T07:32:47.449794+0000","last_unstale":"2024-03-06T07:32:49.475933+0000","last_undegraded":"2024-03-06T07:32:49.475933+0000","last_fullsized":"2024-03-06T07:32:49.475933+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.797839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475854+0000","last_change":"2024-03-06T07:32:48.474160+0000","last_active":"2024-03-06T07:32:49.475854+0000","last_peered":"2024-03-06T07:32:49.475854+0000","last_clean":"2024-03-06T07:32:49.475854+0000","last_became_active":"2024-03-06T07:32:48.473363+0000","last_became_peered":"2024-03-06T07:32:48.473363+0000","last_unstale":"2024-03-06T07:32:49.475854+0000","last_undegraded":"2024-03-06T07:32:49.475854+0000","last_fullsized":"2024-03-06T07:32:49.475854+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:35.227676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050794+0000","last_change":"2024-03-06T07:32:47.437099+0000","last_active":"2024-03-06T07:32:50.050794+0000","last_peered":"2024-03-06T07:32:50.050794+0000","last_clean":"2024-03-06T07:32:50.050794+0000","last_became_active":"2024-03-06T07:32:47.435874+0000","last_became_peered":"2024-03-06T07:32:47.435874+0000","last_unstale":"2024-03-06T07:32:50.050794+0000","last_undegraded":"2024-03-06T07:32:50.050794+0000","last_fullsized":"2024-03-06T07:32:50.050794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:45.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474893+0000","last_change":"2024-03-06T07:32:48.470464+0000","last_active":"2024-03-06T07:32:49.474893+0000","last_peered":"2024-03-06T07:32:49.474893+0000","last_clean":"2024-03-06T07:32:49.474893+0000","last_became_active":"2024-03-06T07:32:48.470163+0000","last_became_peered":"2024-03-06T07:32:48.470163+0000","last_unstale":"2024-03-06T07:32:49.474893+0000","last_undegraded":"2024-03-06T07:32:49.474893+0000","last_fullsized":"2024-03-06T07:32:49.474893+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:11.960503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.447503+0000","last_change":"2024-03-06T07:32:47.444849+0000","last_active":"2024-03-06T07:32:47.447503+0000","last_peered":"2024-03-06T07:32:47.447503+0000","last_clean":"2024-03-06T07:32:47.447503+0000","last_became_active":"2024-03-06T07:32:47.444559+0000","last_became_peered":"2024-03-06T07:32:47.444559+0000","last_unstale":"2024-03-06T07:32:47.447503+0000","last_undegraded":"2024-03-06T07:32:47.447503+0000","last_fullsized":"2024-03-06T07:32:47.447503+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:08:53.238492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.430968+0000","last_change":"2024-03-06T07:32:47.430968+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.430968+0000","last_undegraded":"2024-03-06T07:32:47.430968+0000","last_fullsized":"2024-03-06T07:32:47.430968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.440388+0000","last_change":"2024-03-06T07:32:47.436112+0000","last_active":"2024-03-06T07:32:47.440388+0000","last_peered":"2024-03-06T07:32:47.440388+0000","last_clean":"2024-03-06T07:32:47.440388+0000","last_became_active":"2024-03-06T07:32:47.435819+0000","last_became_peered":"2024-03-06T07:32:47.435819+0000","last_unstale":"2024-03-06T07:32:47.440388+0000","last_undegraded":"2024-03-06T07:32:47.440388+0000","last_fullsized":"2024-03-06T07:32:47.440388+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:59:49.340424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049032+0000","last_change":"2024-03-06T07:32:48.482978+0000","last_active":"2024-03-06T07:32:50.049032+0000","last_peered":"2024-03-06T07:32:50.049032+0000","last_clean":"2024-03-06T07:32:50.049032+0000","last_became_active":"2024-03-06T07:32:48.480934+0000","last_became_peered":"2024-03-06T07:32:48.480934+0000","last_unstale":"2024-03-06T07:32:50.049032+0000","last_undegraded":"2024-03-06T07:32:50.049032+0000","last_fullsized":"2024-03-06T07:32:50.049032+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:01.588274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047224+0000","last_change":"2024-03-06T07:32:47.435945+0000","last_active":"2024-03-06T07:32:50.047224+0000","last_peered":"2024-03-06T07:32:50.047224+0000","last_clean":"2024-03-06T07:32:50.047224+0000","last_became_active":"2024-03-06T07:32:47.435526+0000","last_became_peered":"2024-03-06T07:32:47.435526+0000","last_unstale":"2024-03-06T07:32:50.047224+0000","last_undegraded":"2024-03-06T07:32:50.047224+0000","last_fullsized":"2024-03-06T07:32:50.047224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:11.259835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.423052+0000","last_change":"2024-03-06T07:32:47.423052+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.423052+0000","last_undegraded":"2024-03-06T07:32:47.423052+0000","last_fullsized":"2024-03-06T07:32:47.423052+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.451112+0000","last_change":"2024-03-06T07:32:47.448669+0000","last_active":"2024-03-06T07:32:47.451112+0000","last_peered":"2024-03-06T07:32:47.451112+0000","last_clean":"2024-03-06T07:32:47.451112+0000","last_became_active":"2024-03-06T07:32:47.446355+0000","last_became_peered":"2024-03-06T07:32:47.446355+0000","last_unstale":"2024-03-06T07:32:47.451112+0000","last_undegraded":"2024-03-06T07:32:47.451112+0000","last_fullsized":"2024-03-06T07:32:47.451112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:26.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049326+0000","last_change":"2024-03-06T07:32:48.474379+0000","last_active":"2024-03-06T07:32:50.049326+0000","last_peered":"2024-03-06T07:32:50.049326+0000","last_clean":"2024-03-06T07:32:50.049326+0000","last_became_active":"2024-03-06T07:32:48.474089+0000","last_became_peered":"2024-03-06T07:32:48.474089+0000","last_unstale":"2024-03-06T07:32:50.049326+0000","last_undegraded":"2024-03-06T07:32:50.049326+0000","last_fullsized":"2024-03-06T07:32:50.049326+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:57.480021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050692+0000","last_change":"2024-03-06T07:32:47.443176+0000","last_active":"2024-03-06T07:32:50.050692+0000","last_peered":"2024-03-06T07:32:50.050692+0000","last_clean":"2024-03-06T07:32:50.050692+0000","last_became_active":"2024-03-06T07:32:47.442272+0000","last_became_peered":"2024-03-06T07:32:47.442272+0000","last_unstale":"2024-03-06T07:32:50.050692+0000","last_undegraded":"2024-03-06T07:32:50.050692+0000","last_fullsized":"2024-03-06T07:32:50.050692+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:58.102449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047072+0000","last_change":"2024-03-06T07:32:48.475030+0000","last_active":"2024-03-06T07:32:50.047072+0000","last_peered":"2024-03-06T07:32:50.047072+0000","last_clean":"2024-03-06T07:32:50.047072+0000","last_became_active":"2024-03-06T07:32:48.474739+0000","last_became_peered":"2024-03-06T07:32:48.474739+0000","last_unstale":"2024-03-06T07:32:50.047072+0000","last_undegraded":"2024-03-06T07:32:50.047072+0000","last_fullsized":"2024-03-06T07:32:50.047072+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:39.580526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048627+0000","last_change":"2024-03-06T07:32:47.445429+0000","last_active":"2024-03-06T07:32:50.048627+0000","last_peered":"2024-03-06T07:32:50.048627+0000","last_clean":"2024-03-06T07:32:50.048627+0000","last_became_active":"2024-03-06T07:32:47.445125+0000","last_became_peered":"2024-03-06T07:32:47.445125+0000","last_unstale":"2024-03-06T07:32:50.048627+0000","last_undegraded":"2024-03-06T07:32:50.048627+0000","last_fullsized":"2024-03-06T07:32:50.048627+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:59.869095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475902+0000","last_change":"2024-03-06T07:32:48.486226+0000","last_active":"2024-03-06T07:32:49.475902+0000","last_peered":"2024-03-06T07:32:49.475902+0000","last_clean":"2024-03-06T07:32:49.475902+0000","last_became_active":"2024-03-06T07:32:48.485713+0000","last_became_peered":"2024-03-06T07:32:48.485713+0000","last_unstale":"2024-03-06T07:32:49.475902+0000","last_undegraded":"2024-03-06T07:32:49.475902+0000","last_fullsized":"2024-03-06T07:32:49.475902+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:52.871094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047306+0000","last_change":"2024-03-06T07:32:47.449171+0000","last_active":"2024-03-06T07:32:50.047306+0000","last_peered":"2024-03-06T07:32:50.047306+0000","last_clean":"2024-03-06T07:32:50.047306+0000","last_became_active":"2024-03-06T07:32:47.448643+0000","last_became_peered":"2024-03-06T07:32:47.448643+0000","last_unstale":"2024-03-06T07:32:50.047306+0000","last_undegraded":"2024-03-06T07:32:50.047306+0000","last_fullsized":"2024-03-06T07:32:50.047306+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:14.429471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.424968+0000","last_change":"2024-03-06T07:32:47.424968+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.424968+0000","last_undegraded":"2024-03-06T07:32:47.424968+0000","last_fullsized":"2024-03-06T07:32:47.424968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047067+0000","last_change":"2024-03-06T07:32:47.446224+0000","last_active":"2024-03-06T07:32:50.047067+0000","last_peered":"2024-03-06T07:32:50.047067+0000","last_clean":"2024-03-06T07:32:50.047067+0000","last_became_active":"2024-03-06T07:32:47.443188+0000","last_became_peered":"2024-03-06T07:32:47.443188+0000","last_unstale":"2024-03-06T07:32:50.047067+0000","last_undegraded":"2024-03-06T07:32:50.047067+0000","last_fullsized":"2024-03-06T07:32:50.047067+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:38.261444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.425976+0000","last_change":"2024-03-06T07:32:47.425976+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.425976+0000","last_undegraded":"2024-03-06T07:32:47.425976+0000","last_fullsized":"2024-03-06T07:32:47.425976+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048292+0000","last_change":"2024-03-06T07:32:47.446946+0000","last_active":"2024-03-06T07:32:50.048292+0000","last_peered":"2024-03-06T07:32:50.048292+0000","last_clean":"2024-03-06T07:32:50.048292+0000","last_became_active":"2024-03-06T07:32:47.446516+0000","last_became_peered":"2024-03-06T07:32:47.446516+0000","last_unstale":"2024-03-06T07:32:50.048292+0000","last_undegraded":"2024-03-06T07:32:50.048292+0000","last_fullsized":"2024-03-06T07:32:50.048292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:34.048060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.422486+0000","last_change":"2024-03-06T07:32:47.422486+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.422486+0000","last_undegraded":"2024-03-06T07:32:47.422486+0000","last_fullsized":"2024-03-06T07:32:47.422486+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047323+0000","last_change":"2024-03-06T07:32:47.447401+0000","last_active":"2024-03-06T07:32:50.047323+0000","last_peered":"2024-03-06T07:32:50.047323+0000","last_clean":"2024-03-06T07:32:50.047323+0000","last_became_active":"2024-03-06T07:32:47.446563+0000","last_became_peered":"2024-03-06T07:32:47.446563+0000","last_unstale":"2024-03-06T07:32:50.047323+0000","last_undegraded":"2024-03-06T07:32:50.047323+0000","last_fullsized":"2024-03-06T07:32:50.047323+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:41.671563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047202+0000","last_change":"2024-03-06T07:32:48.472623+0000","last_active":"2024-03-06T07:32:50.047202+0000","last_peered":"2024-03-06T07:32:50.047202+0000","last_clean":"2024-03-06T07:32:50.047202+0000","last_became_active":"2024-03-06T07:32:48.470906+0000","last_became_peered":"2024-03-06T07:32:48.470906+0000","last_unstale":"2024-03-06T07:32:50.047202+0000","last_undegraded":"2024-03-06T07:32:50.047202+0000","last_fullsized":"2024-03-06T07:32:50.047202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:34.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046057+0000","last_change":"2024-03-06T07:32:47.444926+0000","last_active":"2024-03-06T07:32:50.046057+0000","last_peered":"2024-03-06T07:32:50.046057+0000","last_clean":"2024-03-06T07:32:50.046057+0000","last_became_active":"2024-03-06T07:32:47.444619+0000","last_became_peered":"2024-03-06T07:32:47.444619+0000","last_unstale":"2024-03-06T07:32:50.046057+0000","last_undegraded":"2024-03-06T07:32:50.046057+0000","last_fullsized":"2024-03-06T07:32:50.046057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.532542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048187+0000","last_change":"2024-03-06T07:32:48.477998+0000","last_active":"2024-03-06T07:32:50.048187+0000","last_peered":"2024-03-06T07:32:50.048187+0000","last_clean":"2024-03-06T07:32:50.048187+0000","last_became_active":"2024-03-06T07:32:48.477408+0000","last_became_peered":"2024-03-06T07:32:48.477408+0000","last_unstale":"2024-03-06T07:32:50.048187+0000","last_undegraded":"2024-03-06T07:32:50.048187+0000","last_fullsized":"2024-03-06T07:32:50.048187+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:01.194560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049544+0000","last_change":"2024-03-06T07:32:47.450076+0000","last_active":"2024-03-06T07:32:50.049544+0000","last_peered":"2024-03-06T07:32:50.049544+0000","last_clean":"2024-03-06T07:32:50.049544+0000","last_became_active":"2024-03-06T07:32:47.449766+0000","last_became_peered":"2024-03-06T07:32:47.449766+0000","last_unstale":"2024-03-06T07:32:50.049544+0000","last_undegraded":"2024-03-06T07:32:50.049544+0000","last_fullsized":"2024-03-06T07:32:50.049544+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:39.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.428543+0000","last_change":"2024-03-06T07:32:47.428543+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.428543+0000","last_undegraded":"2024-03-06T07:32:47.428543+0000","last_fullsized":"2024-03-06T07:32:47.428543+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048271+0000","last_change":"2024-03-06T07:32:47.445510+0000","last_active":"2024-03-06T07:32:50.048271+0000","last_peered":"2024-03-06T07:32:50.048271+0000","last_clean":"2024-03-06T07:32:50.048271+0000","last_became_active":"2024-03-06T07:32:47.445220+0000","last_became_peered":"2024-03-06T07:32:47.445220+0000","last_unstale":"2024-03-06T07:32:50.048271+0000","last_undegraded":"2024-03-06T07:32:50.048271+0000","last_fullsized":"2024-03-06T07:32:50.048271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:06.096609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.428464+0000","last_change":"2024-03-06T07:32:47.428464+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.428464+0000","last_undegraded":"2024-03-06T07:32:47.428464+0000","last_fullsized":"2024-03-06T07:32:47.428464+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047633+0000","last_change":"2024-03-06T07:32:47.448498+0000","last_active":"2024-03-06T07:32:50.047633+0000","last_peered":"2024-03-06T07:32:50.047633+0000","last_clean":"2024-03-06T07:32:50.047633+0000","last_became_active":"2024-03-06T07:32:47.448209+0000","last_became_peered":"2024-03-06T07:32:47.448209+0000","last_unstale":"2024-03-06T07:32:50.047633+0000","last_undegraded":"2024-03-06T07:32:50.047633+0000","last_fullsized":"2024-03-06T07:32:50.047633+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:20.798460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049512+0000","last_change":"2024-03-06T07:32:48.475439+0000","last_active":"2024-03-06T07:32:50.049512+0000","last_peered":"2024-03-06T07:32:50.049512+0000","last_clean":"2024-03-06T07:32:50.049512+0000","last_became_active":"2024-03-06T07:32:48.475144+0000","last_became_peered":"2024-03-06T07:32:48.475144+0000","last_unstale":"2024-03-06T07:32:50.049512+0000","last_undegraded":"2024-03-06T07:32:50.049512+0000","last_fullsized":"2024-03-06T07:32:50.049512+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:43.593642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048523+0000","last_change":"2024-03-06T07:32:47.442329+0000","last_active":"2024-03-06T07:32:50.048523+0000","last_peered":"2024-03-06T07:32:50.048523+0000","last_clean":"2024-03-06T07:32:50.048523+0000","last_became_active":"2024-03-06T07:32:47.441500+0000","last_became_peered":"2024-03-06T07:32:47.441500+0000","last_unstale":"2024-03-06T07:32:50.048523+0000","last_undegraded":"2024-03-06T07:32:50.048523+0000","last_fullsized":"2024-03-06T07:32:50.048523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.523954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.424495+0000","last_change":"2024-03-06T07:32:47.424495+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.424495+0000","last_undegraded":"2024-03-06T07:32:47.424495+0000","last_fullsized":"2024-03-06T07:32:47.424495+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047138+0000","last_change":"2024-03-06T07:32:47.445530+0000","last_active":"2024-03-06T07:32:50.047138+0000","last_peered":"2024-03-06T07:32:50.047138+0000","last_clean":"2024-03-06T07:32:50.047138+0000","last_became_active":"2024-03-06T07:32:47.442881+0000","last_became_peered":"2024-03-06T07:32:47.442881+0000","last_unstale":"2024-03-06T07:32:50.047138+0000","last_undegraded":"2024-03-06T07:32:50.047138+0000","last_fullsized":"2024-03-06T07:32:50.047138+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:59.869095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050367+0000","last_change":"2024-03-06T07:32:48.475397+0000","last_active":"2024-03-06T07:32:50.050367+0000","last_peered":"2024-03-06T07:32:50.050367+0000","last_clean":"2024-03-06T07:32:50.050367+0000","last_became_active":"2024-03-06T07:32:48.475132+0000","last_became_peered":"2024-03-06T07:32:48.475132+0000","last_unstale":"2024-03-06T07:32:50.050367+0000","last_undegraded":"2024-03-06T07:32:50.050367+0000","last_fullsized":"2024-03-06T07:32:50.050367+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:40.102797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047771+0000","last_change":"2024-03-06T07:32:47.434557+0000","last_active":"2024-03-06T07:32:50.047771+0000","last_peered":"2024-03-06T07:32:50.047771+0000","last_clean":"2024-03-06T07:32:50.047771+0000","last_became_active":"2024-03-06T07:32:47.433650+0000","last_became_peered":"2024-03-06T07:32:47.433650+0000","last_unstale":"2024-03-06T07:32:50.047771+0000","last_undegraded":"2024-03-06T07:32:50.047771+0000","last_fullsized":"2024-03-06T07:32:50.047771+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:32.610977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.421823+0000","last_change":"2024-03-06T07:32:47.421823+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.421823+0000","last_undegraded":"2024-03-06T07:32:47.421823+0000","last_fullsized":"2024-03-06T07:32:47.421823+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474923+0000","last_change":"2024-03-06T07:32:47.423971+0000","last_active":"2024-03-06T07:32:49.474923+0000","last_peered":"2024-03-06T07:32:49.474923+0000","last_clean":"2024-03-06T07:32:49.474923+0000","last_became_active":"2024-03-06T07:32:47.423700+0000","last_became_peered":"2024-03-06T07:32:47.423700+0000","last_unstale":"2024-03-06T07:32:49.474923+0000","last_undegraded":"2024-03-06T07:32:49.474923+0000","last_fullsized":"2024-03-06T07:32:49.474923+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:05.073516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047383+0000","last_change":"2024-03-06T07:32:48.474750+0000","last_active":"2024-03-06T07:32:50.047383+0000","last_peered":"2024-03-06T07:32:50.047383+0000","last_clean":"2024-03-06T07:32:50.047383+0000","last_became_active":"2024-03-06T07:32:48.472684+0000","last_became_peered":"2024-03-06T07:32:48.472684+0000","last_unstale":"2024-03-06T07:32:50.047383+0000","last_undegraded":"2024-03-06T07:32:50.047383+0000","last_fullsized":"2024-03-06T07:32:50.047383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:40.747320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-06T07:32:47.440816+0000","last_change":"2024-03-06T07:32:47.436392+0000","last_active":"2024-03-06T07:32:47.440816+0000","last_peered":"2024-03-06T07:32:47.440816+0000","last_clean":"2024-03-06T07:32:47.440816+0000","last_became_active":"2024-03-06T07:32:47.435756+0000","last_became_peered":"2024-03-06T07:32:47.435756+0000","last_unstale":"2024-03-06T07:32:47.440816+0000","last_undegraded":"2024-03-06T07:32:47.440816+0000","last_fullsized":"2024-03-06T07:32:47.440816+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:36.366920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-06T07:32:47.427054+0000","last_change":"2024-03-06T07:32:47.427054+0000","last_active":"2024-03-06T07:32:47.402268+0000","last_peered":"2024-03-06T07:32:47.402268+0000","last_clean":"2024-03-06T07:32:47.402268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:32:47.427054+0000","last_undegraded":"2024-03-06T07:32:47.427054+0000","last_fullsized":"2024-03-06T07:32:47.427054+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475229+0000","last_change":"2024-03-06T07:32:47.446682+0000","last_active":"2024-03-06T07:32:49.475229+0000","last_peered":"2024-03-06T07:32:49.475229+0000","last_clean":"2024-03-06T07:32:49.475229+0000","last_became_active":"2024-03-06T07:32:47.446061+0000","last_became_peered":"2024-03-06T07:32:47.446061+0000","last_unstale":"2024-03-06T07:32:49.475229+0000","last_undegraded":"2024-03-06T07:32:49.475229+0000","last_fullsized":"2024-03-06T07:32:49.475229+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:11.259835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":298,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475303+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:32:49.475303+0000","last_peered":"2024-03-06T07:32:49.475303+0000","last_clean":"2024-03-06T07:32:49.475303+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:32:49.475303+0000","last_undegraded":"2024-03-06T07:32:49.475303+0000","last_fullsized":"2024-03-06T07:32:49.475303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509508,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1105920,"data_stored":466387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509509,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1105920,"data_stored":466387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1744896,"data_stored":1076217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1150976,"data_stored":486389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1146880,"data_stored":485807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36476,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335364,"statfs":{"total":96636764160,"available":96599412736,"internally_reserved":0,"allocated":1163264,"data_stored":487473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":487371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542214,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1740800,"data_stored":1076687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:53.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:53.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.317+0000 7f8ec8c00640 1 Processor -- start 2024-03-06T07:32:53.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.317+0000 7f8ec8c00640 1 -- start start 2024-03-06T07:32:53.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec8c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4145300 0x7f8ec40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ec40cd060 0x7f8ec415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec8c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec40714d0 con 0x7f8ec4070f10 2024-03-06T07:32:53.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec8c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec40717b0 con 0x7f8ec40cd060 2024-03-06T07:32:53.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec8c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec4071a90 con 0x7f8ec4145300 2024-03-06T07:32:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8ec4070f10 0x7f8ec40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55206/0 (socket says 172.21.15.6:55206) 2024-03-06T07:32:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec2a00640 1 -- 172.21.15.6:0/4083594916 learned_addr learned my addr 172.21.15.6:0/4083594916 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.319+0000 7f8ec3e00640 1 --2- 172.21.15.6:0/4083594916 >> v2:172.21.15.6:3300/0 conn(0x7f8ec40cd060 0x7f8ec415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec3400640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4145300 0x7f8ec40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec2000640 1 -- 172.21.15.6:0/4083594916 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514494822 0 0) 0x7f8ec40714d0 con 0x7f8ec4070f10 2024-03-06T07:32:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec2000640 1 -- 172.21.15.6:0/4083594916 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea8003520 con 0x7f8ec4070f10 2024-03-06T07:32:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec3e00640 1 -- 172.21.15.6:0/4083594916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4145300 msgr2=0x7f8ec40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec3e00640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4145300 0x7f8ec40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec3e00640 1 -- 172.21.15.6:0/4083594916 >> v1:172.21.15.6:6789/0 conn(0x7f8ec4070f10 legacy=0x7f8ec40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec3e00640 1 -- 172.21.15.6:0/4083594916 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec4071d70 con 0x7f8ec40cd060 2024-03-06T07:32:53.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.320+0000 7f8ec3400640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4145300 0x7f8ec40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:53.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.321+0000 7f8ec3e00640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec415c570 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009d90 tx=0x7f8ebc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914c226a75a0d2de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.321+0000 7f8ec2000640 1 -- 172.21.15.6:0/4083594916 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc039070 con 0x7f8ec40cd060 2024-03-06T07:32:53.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.321+0000 7f8ec2000640 1 -- 172.21.15.6:0/4083594916 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc02cc50 con 0x7f8ec40cd060 2024-03-06T07:32:53.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.321+0000 7f8ec2000640 1 -- 172.21.15.6:0/4083594916 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc034440 con 0x7f8ec40cd060 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.322+0000 7f8ec8c00640 1 -- 172.21.15.6:0/4083594916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 msgr2=0x7f8ec415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.322+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec415c570 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009d90 tx=0x7f8ebc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.323+0000 7f8ec8c00640 1 -- 172.21.15.6:0/4083594916 shutdown_connections 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.323+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec415c570 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.323+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/4083594916 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4145300 0x7f8ec40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.323+0000 7f8ec8c00640 1 -- 172.21.15.6:0/4083594916 >> 172.21.15.6:0/4083594916 conn(0x7f8ec406b3d0 msgr2=0x7f8ec406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.323+0000 7f8ec8c00640 1 -- 172.21.15.6:0/4083594916 shutdown_connections 2024-03-06T07:32:53.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.323+0000 7f8ec8c00640 1 -- 172.21.15.6:0/4083594916 wait complete. 2024-03-06T07:32:53.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.327+0000 7f8ec8c00640 1 Processor -- start 2024-03-06T07:32:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.327+0000 7f8ec8c00640 1 -- start start 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.329+0000 7f8ec8c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4070f10 0x7f8ec41f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.329+0000 7f8ec8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec40ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.329+0000 7f8ec8c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ec4145300 0x7f8ec406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.329+0000 7f8ec8c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec415cab0 con 0x7f8ec40cd060 2024-03-06T07:32:53.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.329+0000 7f8ec8c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec41456e0 con 0x7f8ec4145300 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.329+0000 7f8ec8c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec415cc30 con 0x7f8ec4070f10 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec40ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec40ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56376/0 (socket says 172.21.15.6:56376) 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 -- 172.21.15.6:0/386343774 learned_addr learned my addr 172.21.15.6:0/386343774 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec3400640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4070f10 0x7f8ec41f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec3e00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ec4145300 0x7f8ec406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 -- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4070f10 msgr2=0x7f8ec41f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4070f10 0x7f8ec41f3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 -- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ec4145300 msgr2=0x7f8ec406d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ec4145300 0x7f8ec406d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.330+0000 7f8ec2a00640 1 -- 172.21.15.6:0/386343774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec40addc0 con 0x7f8ec40cd060 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.331+0000 7f8ec2a00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec40ad880 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000c430 tx=0x7f8eb000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.331+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0017070 con 0x7f8ec40cd060 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.331+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0005ce0 con 0x7f8ec40cd060 2024-03-06T07:32:53.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.331+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec40ae060 con 0x7f8ec40cd060 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.331+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb00072f0 con 0x7f8ec40cd060 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.331+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec40ae620 con 0x7f8ec40cd060 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.332+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec404f330 con 0x7f8ec40cd060 2024-03-06T07:32:53.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.334+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8eb001a070 con 0x7f8ec40cd060 2024-03-06T07:32:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.335+0000 7f8ebb400640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ea406b4d0 0x7f8ea406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.335+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8eb0089780 con 0x7f8ec40cd060 2024-03-06T07:32:53.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.336+0000 7f8ec3400640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ea406b4d0 0x7f8ea406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.337+0000 7f8ec3400640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ea406b4d0 0x7f8ea406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4144460 tx=0x7f8eb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.340+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8eb0058560 con 0x7f8ec40cd060 2024-03-06T07:32:53.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T07:32:53.395+0000 7f44c3200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:32:53.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.614+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ec4144ab0 con 0x7f8ea406b4d0 2024-03-06T07:32:53.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.632+0000 7f8ebb400640 1 -- 172.21.15.6:0/386343774 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252085 (secure 0 0 0) 0x7f8ec4144ab0 con 0x7f8ea406b4d0 2024-03-06T07:32:53.637 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:53.638 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ea406b4d0 msgr2=0x7f8ea406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ea406b4d0 0x7f8ea406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4144460 tx=0x7f8eb4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 msgr2=0x7f8ec40ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec40ad880 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000c430 tx=0x7f8eb000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 shutdown_connections 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8ea406b4d0 0x7f8ea406d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ec4145300 0x7f8ec406d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ec40cd060 0x7f8ec40ad880 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 --2- 172.21.15.6:0/386343774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ec4070f10 0x7f8ec41f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 >> 172.21.15.6:0/386343774 conn(0x7f8ec406b3d0 msgr2=0x7f8ec4073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 shutdown_connections 2024-03-06T07:32:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.636+0000 7f8ec8c00640 1 -- 172.21.15.6:0/386343774 wait complete. 2024-03-06T07:32:53.658 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-03-06T07:32:53.556757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":276244,"kb_used_data":10152,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754698476,"statfs":{"total":773094113280,"available":772811239424,"internally_reserved":0,"allocated":10395648,"data_stored":5071452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":272354864},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.143896"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474251+0000","last_change":"2024-03-06T07:32:48.476485+0000","last_active":"2024-03-06T07:32:49.474251+0000","last_peered":"2024-03-06T07:32:49.474251+0000","last_clean":"2024-03-06T07:32:49.474251+0000","last_became_active":"2024-03-06T07:32:48.476169+0000","last_became_peered":"2024-03-06T07:32:48.476169+0000","last_unstale":"2024-03-06T07:32:49.474251+0000","last_undegraded":"2024-03-06T07:32:49.474251+0000","last_fullsized":"2024-03-06T07:32:49.474251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:58:57.513247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047747+0000","last_change":"2024-03-06T07:32:48.477759+0000","last_active":"2024-03-06T07:32:50.047747+0000","last_peered":"2024-03-06T07:32:50.047747+0000","last_clean":"2024-03-06T07:32:50.047747+0000","last_became_active":"2024-03-06T07:32:48.477057+0000","last_became_peered":"2024-03-06T07:32:48.477057+0000","last_unstale":"2024-03-06T07:32:50.047747+0000","last_undegraded":"2024-03-06T07:32:50.047747+0000","last_fullsized":"2024-03-06T07:32:50.047747+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:15:09.016281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474026+0000","last_change":"2024-03-06T07:32:48.478460+0000","last_active":"2024-03-06T07:32:49.474026+0000","last_peered":"2024-03-06T07:32:49.474026+0000","last_clean":"2024-03-06T07:32:49.474026+0000","last_became_active":"2024-03-06T07:32:48.476810+0000","last_became_peered":"2024-03-06T07:32:48.476810+0000","last_unstale":"2024-03-06T07:32:49.474026+0000","last_undegraded":"2024-03-06T07:32:49.474026+0000","last_fullsized":"2024-03-06T07:32:49.474026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:44.651797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475024+0000","last_change":"2024-03-06T07:32:48.482309+0000","last_active":"2024-03-06T07:32:49.475024+0000","last_peered":"2024-03-06T07:32:49.475024+0000","last_clean":"2024-03-06T07:32:49.475024+0000","last_became_active":"2024-03-06T07:32:48.481368+0000","last_became_peered":"2024-03-06T07:32:48.481368+0000","last_unstale":"2024-03-06T07:32:49.475024+0000","last_undegraded":"2024-03-06T07:32:49.475024+0000","last_fullsized":"2024-03-06T07:32:49.475024+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:02.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048459+0000","last_change":"2024-03-06T07:32:48.475342+0000","last_active":"2024-03-06T07:32:50.048459+0000","last_peered":"2024-03-06T07:32:50.048459+0000","last_clean":"2024-03-06T07:32:50.048459+0000","last_became_active":"2024-03-06T07:32:48.475052+0000","last_became_peered":"2024-03-06T07:32:48.475052+0000","last_unstale":"2024-03-06T07:32:50.048459+0000","last_undegraded":"2024-03-06T07:32:50.048459+0000","last_fullsized":"2024-03-06T07:32:50.048459+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:24.300728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475116+0000","last_change":"2024-03-06T07:32:48.481391+0000","last_active":"2024-03-06T07:32:49.475116+0000","last_peered":"2024-03-06T07:32:49.475116+0000","last_clean":"2024-03-06T07:32:49.475116+0000","last_became_active":"2024-03-06T07:32:48.481067+0000","last_became_peered":"2024-03-06T07:32:48.481067+0000","last_unstale":"2024-03-06T07:32:49.475116+0000","last_undegraded":"2024-03-06T07:32:49.475116+0000","last_fullsized":"2024-03-06T07:32:49.475116+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:54.546027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474755+0000","last_change":"2024-03-06T07:32:48.480318+0000","last_active":"2024-03-06T07:32:49.474755+0000","last_peered":"2024-03-06T07:32:49.474755+0000","last_clean":"2024-03-06T07:32:49.474755+0000","last_became_active":"2024-03-06T07:32:48.478766+0000","last_became_peered":"2024-03-06T07:32:48.478766+0000","last_unstale":"2024-03-06T07:32:49.474755+0000","last_undegraded":"2024-03-06T07:32:49.474755+0000","last_fullsized":"2024-03-06T07:32:49.474755+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:32.012886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046874+0000","last_change":"2024-03-06T07:32:48.476674+0000","last_active":"2024-03-06T07:32:50.046874+0000","last_peered":"2024-03-06T07:32:50.046874+0000","last_clean":"2024-03-06T07:32:50.046874+0000","last_became_active":"2024-03-06T07:32:48.472921+0000","last_became_peered":"2024-03-06T07:32:48.472921+0000","last_unstale":"2024-03-06T07:32:50.046874+0000","last_undegraded":"2024-03-06T07:32:50.046874+0000","last_fullsized":"2024-03-06T07:32:50.046874+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:57.480021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.051123+0000","last_change":"2024-03-06T07:32:48.488539+0000","last_active":"2024-03-06T07:32:50.051123+0000","last_peered":"2024-03-06T07:32:50.051123+0000","last_clean":"2024-03-06T07:32:50.051123+0000","last_became_active":"2024-03-06T07:32:48.488181+0000","last_became_peered":"2024-03-06T07:32:48.488181+0000","last_unstale":"2024-03-06T07:32:50.051123+0000","last_undegraded":"2024-03-06T07:32:50.051123+0000","last_fullsized":"2024-03-06T07:32:50.051123+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:54.546027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048514+0000","last_change":"2024-03-06T07:32:48.475586+0000","last_active":"2024-03-06T07:32:50.048514+0000","last_peered":"2024-03-06T07:32:50.048514+0000","last_clean":"2024-03-06T07:32:50.048514+0000","last_became_active":"2024-03-06T07:32:48.472161+0000","last_became_peered":"2024-03-06T07:32:48.472161+0000","last_unstale":"2024-03-06T07:32:50.048514+0000","last_undegraded":"2024-03-06T07:32:50.048514+0000","last_fullsized":"2024-03-06T07:32:50.048514+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:02.196742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475612+0000","last_change":"2024-03-06T07:32:48.474439+0000","last_active":"2024-03-06T07:32:49.475612+0000","last_peered":"2024-03-06T07:32:49.475612+0000","last_clean":"2024-03-06T07:32:49.475612+0000","last_became_active":"2024-03-06T07:32:48.473837+0000","last_became_peered":"2024-03-06T07:32:48.473837+0000","last_unstale":"2024-03-06T07:32:49.475612+0000","last_undegraded":"2024-03-06T07:32:49.475612+0000","last_fullsized":"2024-03-06T07:32:49.475612+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:55.621564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050695+0000","last_change":"2024-03-06T07:32:48.473220+0000","last_active":"2024-03-06T07:32:50.050695+0000","last_peered":"2024-03-06T07:32:50.050695+0000","last_clean":"2024-03-06T07:32:50.050695+0000","last_became_active":"2024-03-06T07:32:48.472930+0000","last_became_peered":"2024-03-06T07:32:48.472930+0000","last_unstale":"2024-03-06T07:32:50.050695+0000","last_undegraded":"2024-03-06T07:32:50.050695+0000","last_fullsized":"2024-03-06T07:32:50.050695+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:12.461489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474705+0000","last_change":"2024-03-06T07:32:48.477921+0000","last_active":"2024-03-06T07:32:49.474705+0000","last_peered":"2024-03-06T07:32:49.474705+0000","last_clean":"2024-03-06T07:32:49.474705+0000","last_became_active":"2024-03-06T07:32:48.477410+0000","last_became_peered":"2024-03-06T07:32:48.477410+0000","last_unstale":"2024-03-06T07:32:49.474705+0000","last_undegraded":"2024-03-06T07:32:49.474705+0000","last_fullsized":"2024-03-06T07:32:49.474705+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:24.300728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045864+0000","last_change":"2024-03-06T07:32:48.490480+0000","last_active":"2024-03-06T07:32:50.045864+0000","last_peered":"2024-03-06T07:32:50.045864+0000","last_clean":"2024-03-06T07:32:50.045864+0000","last_became_active":"2024-03-06T07:32:48.490205+0000","last_became_peered":"2024-03-06T07:32:48.490205+0000","last_unstale":"2024-03-06T07:32:50.045864+0000","last_undegraded":"2024-03-06T07:32:50.045864+0000","last_fullsized":"2024-03-06T07:32:50.045864+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:14.385536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474876+0000","last_change":"2024-03-06T07:32:48.479483+0000","last_active":"2024-03-06T07:32:49.474876+0000","last_peered":"2024-03-06T07:32:49.474876+0000","last_clean":"2024-03-06T07:32:49.474876+0000","last_became_active":"2024-03-06T07:32:48.478879+0000","last_became_peered":"2024-03-06T07:32:48.478879+0000","last_unstale":"2024-03-06T07:32:49.474876+0000","last_undegraded":"2024-03-06T07:32:49.474876+0000","last_fullsized":"2024-03-06T07:32:49.474876+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:50.841839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474977+0000","last_change":"2024-03-06T07:32:48.473952+0000","last_active":"2024-03-06T07:32:49.474977+0000","last_peered":"2024-03-06T07:32:49.474977+0000","last_clean":"2024-03-06T07:32:49.474977+0000","last_became_active":"2024-03-06T07:32:48.473130+0000","last_became_peered":"2024-03-06T07:32:48.473130+0000","last_unstale":"2024-03-06T07:32:49.474977+0000","last_undegraded":"2024-03-06T07:32:49.474977+0000","last_fullsized":"2024-03-06T07:32:49.474977+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:26.058500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047941+0000","last_change":"2024-03-06T07:32:48.468475+0000","last_active":"2024-03-06T07:32:50.047941+0000","last_peered":"2024-03-06T07:32:50.047941+0000","last_clean":"2024-03-06T07:32:50.047941+0000","last_became_active":"2024-03-06T07:32:48.468154+0000","last_became_peered":"2024-03-06T07:32:48.468154+0000","last_unstale":"2024-03-06T07:32:50.047941+0000","last_undegraded":"2024-03-06T07:32:50.047941+0000","last_fullsized":"2024-03-06T07:32:50.047941+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:21.501947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474896+0000","last_change":"2024-03-06T07:32:48.481353+0000","last_active":"2024-03-06T07:32:49.474896+0000","last_peered":"2024-03-06T07:32:49.474896+0000","last_clean":"2024-03-06T07:32:49.474896+0000","last_became_active":"2024-03-06T07:32:48.479874+0000","last_became_peered":"2024-03-06T07:32:48.479874+0000","last_unstale":"2024-03-06T07:32:49.474896+0000","last_undegraded":"2024-03-06T07:32:49.474896+0000","last_fullsized":"2024-03-06T07:32:49.474896+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:57.480021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046039+0000","last_change":"2024-03-06T07:32:48.485481+0000","last_active":"2024-03-06T07:32:50.046039+0000","last_peered":"2024-03-06T07:32:50.046039+0000","last_clean":"2024-03-06T07:32:50.046039+0000","last_became_active":"2024-03-06T07:32:48.484228+0000","last_became_peered":"2024-03-06T07:32:48.484228+0000","last_unstale":"2024-03-06T07:32:50.046039+0000","last_undegraded":"2024-03-06T07:32:50.046039+0000","last_fullsized":"2024-03-06T07:32:50.046039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:26.274457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050662+0000","last_change":"2024-03-06T07:32:48.489733+0000","last_active":"2024-03-06T07:32:50.050662+0000","last_peered":"2024-03-06T07:32:50.050662+0000","last_clean":"2024-03-06T07:32:50.050662+0000","last_became_active":"2024-03-06T07:32:48.488129+0000","last_became_peered":"2024-03-06T07:32:48.488129+0000","last_unstale":"2024-03-06T07:32:50.050662+0000","last_undegraded":"2024-03-06T07:32:50.050662+0000","last_fullsized":"2024-03-06T07:32:50.050662+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:11:32.012886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045991+0000","last_change":"2024-03-06T07:32:48.486114+0000","last_active":"2024-03-06T07:32:50.045991+0000","last_peered":"2024-03-06T07:32:50.045991+0000","last_clean":"2024-03-06T07:32:50.045991+0000","last_became_active":"2024-03-06T07:32:48.484831+0000","last_became_peered":"2024-03-06T07:32:48.484831+0000","last_unstale":"2024-03-06T07:32:50.045991+0000","last_undegraded":"2024-03-06T07:32:50.045991+0000","last_fullsized":"2024-03-06T07:32:50.045991+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:57.898991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046695+0000","last_change":"2024-03-06T07:32:48.476584+0000","last_active":"2024-03-06T07:32:50.046695+0000","last_peered":"2024-03-06T07:32:50.046695+0000","last_clean":"2024-03-06T07:32:50.046695+0000","last_became_active":"2024-03-06T07:32:48.476309+0000","last_became_peered":"2024-03-06T07:32:48.476309+0000","last_unstale":"2024-03-06T07:32:50.046695+0000","last_undegraded":"2024-03-06T07:32:50.046695+0000","last_fullsized":"2024-03-06T07:32:50.046695+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:11.960503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046080+0000","last_change":"2024-03-06T07:32:48.478660+0000","last_active":"2024-03-06T07:32:50.046080+0000","last_peered":"2024-03-06T07:32:50.046080+0000","last_clean":"2024-03-06T07:32:50.046080+0000","last_became_active":"2024-03-06T07:32:48.475347+0000","last_became_peered":"2024-03-06T07:32:48.475347+0000","last_unstale":"2024-03-06T07:32:50.046080+0000","last_undegraded":"2024-03-06T07:32:50.046080+0000","last_fullsized":"2024-03-06T07:32:50.046080+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:50.841839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475213+0000","last_change":"2024-03-06T07:32:48.475228+0000","last_active":"2024-03-06T07:32:49.475213+0000","last_peered":"2024-03-06T07:32:49.475213+0000","last_clean":"2024-03-06T07:32:49.475213+0000","last_became_active":"2024-03-06T07:32:48.473605+0000","last_became_peered":"2024-03-06T07:32:48.473605+0000","last_unstale":"2024-03-06T07:32:49.475213+0000","last_undegraded":"2024-03-06T07:32:49.475213+0000","last_fullsized":"2024-03-06T07:32:49.475213+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:08.607260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049172+0000","last_change":"2024-03-06T07:32:48.468106+0000","last_active":"2024-03-06T07:32:50.049172+0000","last_peered":"2024-03-06T07:32:50.049172+0000","last_clean":"2024-03-06T07:32:50.049172+0000","last_became_active":"2024-03-06T07:32:48.467651+0000","last_became_peered":"2024-03-06T07:32:48.467651+0000","last_unstale":"2024-03-06T07:32:50.049172+0000","last_undegraded":"2024-03-06T07:32:50.049172+0000","last_fullsized":"2024-03-06T07:32:50.049172+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:11.960503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047900+0000","last_change":"2024-03-06T07:32:48.475360+0000","last_active":"2024-03-06T07:32:50.047900+0000","last_peered":"2024-03-06T07:32:50.047900+0000","last_clean":"2024-03-06T07:32:50.047900+0000","last_became_active":"2024-03-06T07:32:48.474357+0000","last_became_peered":"2024-03-06T07:32:48.474357+0000","last_unstale":"2024-03-06T07:32:50.047900+0000","last_undegraded":"2024-03-06T07:32:50.047900+0000","last_fullsized":"2024-03-06T07:32:50.047900+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:13:22.225453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475631+0000","last_change":"2024-03-06T07:32:48.487211+0000","last_active":"2024-03-06T07:32:49.475631+0000","last_peered":"2024-03-06T07:32:49.475631+0000","last_clean":"2024-03-06T07:32:49.475631+0000","last_became_active":"2024-03-06T07:32:48.486949+0000","last_became_peered":"2024-03-06T07:32:48.486949+0000","last_unstale":"2024-03-06T07:32:49.475631+0000","last_undegraded":"2024-03-06T07:32:49.475631+0000","last_fullsized":"2024-03-06T07:32:49.475631+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:29.059277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048362+0000","last_change":"2024-03-06T07:32:48.477073+0000","last_active":"2024-03-06T07:32:50.048362+0000","last_peered":"2024-03-06T07:32:50.048362+0000","last_clean":"2024-03-06T07:32:50.048362+0000","last_became_active":"2024-03-06T07:32:48.476280+0000","last_became_peered":"2024-03-06T07:32:48.476280+0000","last_unstale":"2024-03-06T07:32:50.048362+0000","last_undegraded":"2024-03-06T07:32:50.048362+0000","last_fullsized":"2024-03-06T07:32:50.048362+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:11.710892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047088+0000","last_change":"2024-03-06T07:32:48.474842+0000","last_active":"2024-03-06T07:32:50.047088+0000","last_peered":"2024-03-06T07:32:50.047088+0000","last_clean":"2024-03-06T07:32:50.047088+0000","last_became_active":"2024-03-06T07:32:48.474542+0000","last_became_peered":"2024-03-06T07:32:48.474542+0000","last_unstale":"2024-03-06T07:32:50.047088+0000","last_undegraded":"2024-03-06T07:32:50.047088+0000","last_fullsized":"2024-03-06T07:32:50.047088+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:43.593642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045409+0000","last_change":"2024-03-06T07:32:48.481504+0000","last_active":"2024-03-06T07:32:50.045409+0000","last_peered":"2024-03-06T07:32:50.045409+0000","last_clean":"2024-03-06T07:32:50.045409+0000","last_became_active":"2024-03-06T07:32:48.481235+0000","last_became_peered":"2024-03-06T07:32:48.481235+0000","last_unstale":"2024-03-06T07:32:50.045409+0000","last_undegraded":"2024-03-06T07:32:50.045409+0000","last_fullsized":"2024-03-06T07:32:50.045409+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:49.229819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474862+0000","last_change":"2024-03-06T07:32:48.471152+0000","last_active":"2024-03-06T07:32:49.474862+0000","last_peered":"2024-03-06T07:32:49.474862+0000","last_clean":"2024-03-06T07:32:49.474862+0000","last_became_active":"2024-03-06T07:32:48.470786+0000","last_became_peered":"2024-03-06T07:32:48.470786+0000","last_unstale":"2024-03-06T07:32:49.474862+0000","last_undegraded":"2024-03-06T07:32:49.474862+0000","last_fullsized":"2024-03-06T07:32:49.474862+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:11.710892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473147+0000","last_change":"2024-03-06T07:32:48.471815+0000","last_active":"2024-03-06T07:32:49.473147+0000","last_peered":"2024-03-06T07:32:49.473147+0000","last_clean":"2024-03-06T07:32:49.473147+0000","last_became_active":"2024-03-06T07:32:48.471491+0000","last_became_peered":"2024-03-06T07:32:48.471491+0000","last_unstale":"2024-03-06T07:32:49.473147+0000","last_undegraded":"2024-03-06T07:32:49.473147+0000","last_fullsized":"2024-03-06T07:32:49.473147+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:43.548019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046767+0000","last_change":"2024-03-06T07:32:48.480040+0000","last_active":"2024-03-06T07:32:50.046767+0000","last_peered":"2024-03-06T07:32:50.046767+0000","last_clean":"2024-03-06T07:32:50.046767+0000","last_became_active":"2024-03-06T07:32:48.479722+0000","last_became_peered":"2024-03-06T07:32:48.479722+0000","last_unstale":"2024-03-06T07:32:50.046767+0000","last_undegraded":"2024-03-06T07:32:50.046767+0000","last_fullsized":"2024-03-06T07:32:50.046767+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:55.323303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"92'4","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047815+0000","last_change":"2024-03-06T07:32:47.443466+0000","last_active":"2024-03-06T07:32:50.047815+0000","last_peered":"2024-03-06T07:32:50.047815+0000","last_clean":"2024-03-06T07:32:50.047815+0000","last_became_active":"2024-03-06T07:32:47.443209+0000","last_became_peered":"2024-03-06T07:32:47.443209+0000","last_unstale":"2024-03-06T07:32:50.047815+0000","last_undegraded":"2024-03-06T07:32:50.047815+0000","last_fullsized":"2024-03-06T07:32:50.047815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:05.073516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475223+0000","last_change":"2024-03-06T07:32:48.465838+0000","last_active":"2024-03-06T07:32:49.475223+0000","last_peered":"2024-03-06T07:32:49.475223+0000","last_clean":"2024-03-06T07:32:49.475223+0000","last_became_active":"2024-03-06T07:32:48.465431+0000","last_became_peered":"2024-03-06T07:32:48.465431+0000","last_unstale":"2024-03-06T07:32:49.475223+0000","last_undegraded":"2024-03-06T07:32:49.475223+0000","last_fullsized":"2024-03-06T07:32:49.475223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:43.593642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475018+0000","last_change":"2024-03-06T07:32:47.446298+0000","last_active":"2024-03-06T07:32:49.475018+0000","last_peered":"2024-03-06T07:32:49.475018+0000","last_clean":"2024-03-06T07:32:49.475018+0000","last_became_active":"2024-03-06T07:32:47.446022+0000","last_became_peered":"2024-03-06T07:32:47.446022+0000","last_unstale":"2024-03-06T07:32:49.475018+0000","last_undegraded":"2024-03-06T07:32:49.475018+0000","last_fullsized":"2024-03-06T07:32:49.475018+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:45.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473550+0000","last_change":"2024-03-06T07:32:48.483542+0000","last_active":"2024-03-06T07:32:49.473550+0000","last_peered":"2024-03-06T07:32:49.473550+0000","last_clean":"2024-03-06T07:32:49.473550+0000","last_became_active":"2024-03-06T07:32:48.482033+0000","last_became_peered":"2024-03-06T07:32:48.482033+0000","last_unstale":"2024-03-06T07:32:49.473550+0000","last_undegraded":"2024-03-06T07:32:49.473550+0000","last_fullsized":"2024-03-06T07:32:49.473550+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:21.915518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047377+0000","last_change":"2024-03-06T07:32:47.442753+0000","last_active":"2024-03-06T07:32:50.047377+0000","last_peered":"2024-03-06T07:32:50.047377+0000","last_clean":"2024-03-06T07:32:50.047377+0000","last_became_active":"2024-03-06T07:32:47.442432+0000","last_became_peered":"2024-03-06T07:32:47.442432+0000","last_unstale":"2024-03-06T07:32:50.047377+0000","last_undegraded":"2024-03-06T07:32:50.047377+0000","last_fullsized":"2024-03-06T07:32:50.047377+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:57.754771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.051016+0000","last_change":"2024-03-06T07:32:48.480970+0000","last_active":"2024-03-06T07:32:50.051016+0000","last_peered":"2024-03-06T07:32:50.051016+0000","last_clean":"2024-03-06T07:32:50.051016+0000","last_became_active":"2024-03-06T07:32:48.480572+0000","last_became_peered":"2024-03-06T07:32:48.480572+0000","last_unstale":"2024-03-06T07:32:50.051016+0000","last_undegraded":"2024-03-06T07:32:50.051016+0000","last_fullsized":"2024-03-06T07:32:50.051016+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:52:26.058500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046526+0000","last_change":"2024-03-06T07:32:47.448905+0000","last_active":"2024-03-06T07:32:50.046526+0000","last_peered":"2024-03-06T07:32:50.046526+0000","last_clean":"2024-03-06T07:32:50.046526+0000","last_became_active":"2024-03-06T07:32:47.448293+0000","last_became_peered":"2024-03-06T07:32:47.448293+0000","last_unstale":"2024-03-06T07:32:50.046526+0000","last_undegraded":"2024-03-06T07:32:50.046526+0000","last_fullsized":"2024-03-06T07:32:50.046526+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.523954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048159+0000","last_change":"2024-03-06T07:32:48.477826+0000","last_active":"2024-03-06T07:32:50.048159+0000","last_peered":"2024-03-06T07:32:50.048159+0000","last_clean":"2024-03-06T07:32:50.048159+0000","last_became_active":"2024-03-06T07:32:48.477514+0000","last_became_peered":"2024-03-06T07:32:48.477514+0000","last_unstale":"2024-03-06T07:32:50.048159+0000","last_undegraded":"2024-03-06T07:32:50.048159+0000","last_fullsized":"2024-03-06T07:32:50.048159+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:05.662945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473604+0000","last_change":"2024-03-06T07:32:47.435176+0000","last_active":"2024-03-06T07:32:49.473604+0000","last_peered":"2024-03-06T07:32:49.473604+0000","last_clean":"2024-03-06T07:32:49.473604+0000","last_became_active":"2024-03-06T07:32:47.433205+0000","last_became_peered":"2024-03-06T07:32:47.433205+0000","last_unstale":"2024-03-06T07:32:49.473604+0000","last_undegraded":"2024-03-06T07:32:49.473604+0000","last_fullsized":"2024-03-06T07:32:49.473604+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:22:13.371532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474917+0000","last_change":"2024-03-06T07:32:48.472777+0000","last_active":"2024-03-06T07:32:49.474917+0000","last_peered":"2024-03-06T07:32:49.474917+0000","last_clean":"2024-03-06T07:32:49.474917+0000","last_became_active":"2024-03-06T07:32:48.472455+0000","last_became_peered":"2024-03-06T07:32:48.472455+0000","last_unstale":"2024-03-06T07:32:49.474917+0000","last_undegraded":"2024-03-06T07:32:49.474917+0000","last_fullsized":"2024-03-06T07:32:49.474917+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:54:12.461489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046742+0000","last_change":"2024-03-06T07:32:47.449280+0000","last_active":"2024-03-06T07:32:50.046742+0000","last_peered":"2024-03-06T07:32:50.046742+0000","last_clean":"2024-03-06T07:32:50.046742+0000","last_became_active":"2024-03-06T07:32:47.447849+0000","last_became_peered":"2024-03-06T07:32:47.447849+0000","last_unstale":"2024-03-06T07:32:50.046742+0000","last_undegraded":"2024-03-06T07:32:50.046742+0000","last_fullsized":"2024-03-06T07:32:50.046742+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:32.610977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474676+0000","last_change":"2024-03-06T07:32:48.470481+0000","last_active":"2024-03-06T07:32:49.474676+0000","last_peered":"2024-03-06T07:32:49.474676+0000","last_clean":"2024-03-06T07:32:49.474676+0000","last_became_active":"2024-03-06T07:32:48.469803+0000","last_became_peered":"2024-03-06T07:32:48.469803+0000","last_unstale":"2024-03-06T07:32:49.474676+0000","last_undegraded":"2024-03-06T07:32:49.474676+0000","last_fullsized":"2024-03-06T07:32:49.474676+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:40.747320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047331+0000","last_change":"2024-03-06T07:32:47.443161+0000","last_active":"2024-03-06T07:32:50.047331+0000","last_peered":"2024-03-06T07:32:50.047331+0000","last_clean":"2024-03-06T07:32:50.047331+0000","last_became_active":"2024-03-06T07:32:47.442877+0000","last_became_peered":"2024-03-06T07:32:47.442877+0000","last_unstale":"2024-03-06T07:32:50.047331+0000","last_undegraded":"2024-03-06T07:32:50.047331+0000","last_fullsized":"2024-03-06T07:32:50.047331+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:38.261444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047094+0000","last_change":"2024-03-06T07:32:48.484896+0000","last_active":"2024-03-06T07:32:50.047094+0000","last_peered":"2024-03-06T07:32:50.047094+0000","last_clean":"2024-03-06T07:32:50.047094+0000","last_became_active":"2024-03-06T07:32:48.484473+0000","last_became_peered":"2024-03-06T07:32:48.484473+0000","last_unstale":"2024-03-06T07:32:50.047094+0000","last_undegraded":"2024-03-06T07:32:50.047094+0000","last_fullsized":"2024-03-06T07:32:50.047094+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:16.329953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473211+0000","last_change":"2024-03-06T07:32:47.421031+0000","last_active":"2024-03-06T07:32:49.473211+0000","last_peered":"2024-03-06T07:32:49.473211+0000","last_clean":"2024-03-06T07:32:49.473211+0000","last_became_active":"2024-03-06T07:32:47.420724+0000","last_became_peered":"2024-03-06T07:32:47.420724+0000","last_unstale":"2024-03-06T07:32:49.473211+0000","last_undegraded":"2024-03-06T07:32:49.473211+0000","last_fullsized":"2024-03-06T07:32:49.473211+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:37:56.884988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048203+0000","last_change":"2024-03-06T07:32:48.478123+0000","last_active":"2024-03-06T07:32:50.048203+0000","last_peered":"2024-03-06T07:32:50.048203+0000","last_clean":"2024-03-06T07:32:50.048203+0000","last_became_active":"2024-03-06T07:32:48.474915+0000","last_became_peered":"2024-03-06T07:32:48.474915+0000","last_unstale":"2024-03-06T07:32:50.048203+0000","last_undegraded":"2024-03-06T07:32:50.048203+0000","last_fullsized":"2024-03-06T07:32:50.048203+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:40.486151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474498+0000","last_change":"2024-03-06T07:32:47.448173+0000","last_active":"2024-03-06T07:32:49.474498+0000","last_peered":"2024-03-06T07:32:49.474498+0000","last_clean":"2024-03-06T07:32:49.474498+0000","last_became_active":"2024-03-06T07:32:47.445327+0000","last_became_peered":"2024-03-06T07:32:47.445327+0000","last_unstale":"2024-03-06T07:32:49.474498+0000","last_undegraded":"2024-03-06T07:32:49.474498+0000","last_fullsized":"2024-03-06T07:32:49.474498+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:31.886048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.051209+0000","last_change":"2024-03-06T07:32:48.482589+0000","last_active":"2024-03-06T07:32:50.051209+0000","last_peered":"2024-03-06T07:32:50.051209+0000","last_clean":"2024-03-06T07:32:50.051209+0000","last_became_active":"2024-03-06T07:32:48.481339+0000","last_became_peered":"2024-03-06T07:32:48.481339+0000","last_unstale":"2024-03-06T07:32:50.051209+0000","last_undegraded":"2024-03-06T07:32:50.051209+0000","last_fullsized":"2024-03-06T07:32:50.051209+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:46:08.607260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474944+0000","last_change":"2024-03-06T07:32:47.449558+0000","last_active":"2024-03-06T07:32:49.474944+0000","last_peered":"2024-03-06T07:32:49.474944+0000","last_clean":"2024-03-06T07:32:49.474944+0000","last_became_active":"2024-03-06T07:32:47.447476+0000","last_became_peered":"2024-03-06T07:32:47.447476+0000","last_unstale":"2024-03-06T07:32:49.474944+0000","last_undegraded":"2024-03-06T07:32:49.474944+0000","last_fullsized":"2024-03-06T07:32:49.474944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:40:57.754771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048641+0000","last_change":"2024-03-06T07:32:48.469589+0000","last_active":"2024-03-06T07:32:50.048641+0000","last_peered":"2024-03-06T07:32:50.048641+0000","last_clean":"2024-03-06T07:32:50.048641+0000","last_became_active":"2024-03-06T07:32:48.469119+0000","last_became_peered":"2024-03-06T07:32:48.469119+0000","last_unstale":"2024-03-06T07:32:50.048641+0000","last_undegraded":"2024-03-06T07:32:50.048641+0000","last_fullsized":"2024-03-06T07:32:50.048641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:50.841839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"92'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475933+0000","last_change":"2024-03-06T07:32:47.450101+0000","last_active":"2024-03-06T07:32:49.475933+0000","last_peered":"2024-03-06T07:32:49.475933+0000","last_clean":"2024-03-06T07:32:49.475933+0000","last_became_active":"2024-03-06T07:32:47.449794+0000","last_became_peered":"2024-03-06T07:32:47.449794+0000","last_unstale":"2024-03-06T07:32:49.475933+0000","last_undegraded":"2024-03-06T07:32:49.475933+0000","last_fullsized":"2024-03-06T07:32:49.475933+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:46.797839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475854+0000","last_change":"2024-03-06T07:32:48.474160+0000","last_active":"2024-03-06T07:32:49.475854+0000","last_peered":"2024-03-06T07:32:49.475854+0000","last_clean":"2024-03-06T07:32:49.475854+0000","last_became_active":"2024-03-06T07:32:48.473363+0000","last_became_peered":"2024-03-06T07:32:48.473363+0000","last_unstale":"2024-03-06T07:32:49.475854+0000","last_undegraded":"2024-03-06T07:32:49.475854+0000","last_fullsized":"2024-03-06T07:32:49.475854+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:35.227676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"92'6","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050794+0000","last_change":"2024-03-06T07:32:47.437099+0000","last_active":"2024-03-06T07:32:50.050794+0000","last_peered":"2024-03-06T07:32:50.050794+0000","last_clean":"2024-03-06T07:32:50.050794+0000","last_became_active":"2024-03-06T07:32:47.435874+0000","last_became_peered":"2024-03-06T07:32:47.435874+0000","last_unstale":"2024-03-06T07:32:50.050794+0000","last_undegraded":"2024-03-06T07:32:50.050794+0000","last_fullsized":"2024-03-06T07:32:50.050794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:45.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474893+0000","last_change":"2024-03-06T07:32:48.470464+0000","last_active":"2024-03-06T07:32:49.474893+0000","last_peered":"2024-03-06T07:32:49.474893+0000","last_clean":"2024-03-06T07:32:49.474893+0000","last_became_active":"2024-03-06T07:32:48.470163+0000","last_became_peered":"2024-03-06T07:32:48.470163+0000","last_unstale":"2024-03-06T07:32:49.474893+0000","last_undegraded":"2024-03-06T07:32:49.474893+0000","last_fullsized":"2024-03-06T07:32:49.474893+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:11.960503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046433+0000","last_change":"2024-03-06T07:32:47.444849+0000","last_active":"2024-03-06T07:32:50.046433+0000","last_peered":"2024-03-06T07:32:50.046433+0000","last_clean":"2024-03-06T07:32:50.046433+0000","last_became_active":"2024-03-06T07:32:47.444559+0000","last_became_peered":"2024-03-06T07:32:47.444559+0000","last_unstale":"2024-03-06T07:32:50.046433+0000","last_undegraded":"2024-03-06T07:32:50.046433+0000","last_fullsized":"2024-03-06T07:32:50.046433+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:14.074750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473507+0000","last_change":"2024-03-06T07:32:48.488504+0000","last_active":"2024-03-06T07:32:49.473507+0000","last_peered":"2024-03-06T07:32:49.473507+0000","last_clean":"2024-03-06T07:32:49.473507+0000","last_became_active":"2024-03-06T07:32:48.487842+0000","last_became_peered":"2024-03-06T07:32:48.487842+0000","last_unstale":"2024-03-06T07:32:49.473507+0000","last_undegraded":"2024-03-06T07:32:49.473507+0000","last_fullsized":"2024-03-06T07:32:49.473507+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:01:16.965332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046178+0000","last_change":"2024-03-06T07:32:47.436112+0000","last_active":"2024-03-06T07:32:50.046178+0000","last_peered":"2024-03-06T07:32:50.046178+0000","last_clean":"2024-03-06T07:32:50.046178+0000","last_became_active":"2024-03-06T07:32:47.435819+0000","last_became_peered":"2024-03-06T07:32:47.435819+0000","last_unstale":"2024-03-06T07:32:50.046178+0000","last_undegraded":"2024-03-06T07:32:50.046178+0000","last_fullsized":"2024-03-06T07:32:50.046178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:31.660821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049032+0000","last_change":"2024-03-06T07:32:48.482978+0000","last_active":"2024-03-06T07:32:50.049032+0000","last_peered":"2024-03-06T07:32:50.049032+0000","last_clean":"2024-03-06T07:32:50.049032+0000","last_became_active":"2024-03-06T07:32:48.480934+0000","last_became_peered":"2024-03-06T07:32:48.480934+0000","last_unstale":"2024-03-06T07:32:50.049032+0000","last_undegraded":"2024-03-06T07:32:50.049032+0000","last_fullsized":"2024-03-06T07:32:50.049032+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:57:01.588274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047224+0000","last_change":"2024-03-06T07:32:47.435945+0000","last_active":"2024-03-06T07:32:50.047224+0000","last_peered":"2024-03-06T07:32:50.047224+0000","last_clean":"2024-03-06T07:32:50.047224+0000","last_became_active":"2024-03-06T07:32:47.435526+0000","last_became_peered":"2024-03-06T07:32:47.435526+0000","last_unstale":"2024-03-06T07:32:50.047224+0000","last_undegraded":"2024-03-06T07:32:50.047224+0000","last_fullsized":"2024-03-06T07:32:50.047224+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:11.259835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045390+0000","last_change":"2024-03-06T07:32:48.486754+0000","last_active":"2024-03-06T07:32:50.045390+0000","last_peered":"2024-03-06T07:32:50.045390+0000","last_clean":"2024-03-06T07:32:50.045390+0000","last_became_active":"2024-03-06T07:32:48.485093+0000","last_became_peered":"2024-03-06T07:32:48.485093+0000","last_unstale":"2024-03-06T07:32:50.045390+0000","last_undegraded":"2024-03-06T07:32:50.045390+0000","last_fullsized":"2024-03-06T07:32:50.045390+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:36.097657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"92'8","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473321+0000","last_change":"2024-03-06T07:32:47.448669+0000","last_active":"2024-03-06T07:32:49.473321+0000","last_peered":"2024-03-06T07:32:49.473321+0000","last_clean":"2024-03-06T07:32:49.473321+0000","last_became_active":"2024-03-06T07:32:47.446355+0000","last_became_peered":"2024-03-06T07:32:47.446355+0000","last_unstale":"2024-03-06T07:32:49.473321+0000","last_undegraded":"2024-03-06T07:32:49.473321+0000","last_fullsized":"2024-03-06T07:32:49.473321+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:25.260453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049326+0000","last_change":"2024-03-06T07:32:48.474379+0000","last_active":"2024-03-06T07:32:50.049326+0000","last_peered":"2024-03-06T07:32:50.049326+0000","last_clean":"2024-03-06T07:32:50.049326+0000","last_became_active":"2024-03-06T07:32:48.474089+0000","last_became_peered":"2024-03-06T07:32:48.474089+0000","last_unstale":"2024-03-06T07:32:50.049326+0000","last_undegraded":"2024-03-06T07:32:50.049326+0000","last_fullsized":"2024-03-06T07:32:50.049326+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:57:57.480021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050692+0000","last_change":"2024-03-06T07:32:47.443176+0000","last_active":"2024-03-06T07:32:50.050692+0000","last_peered":"2024-03-06T07:32:50.050692+0000","last_clean":"2024-03-06T07:32:50.050692+0000","last_became_active":"2024-03-06T07:32:47.442272+0000","last_became_peered":"2024-03-06T07:32:47.442272+0000","last_unstale":"2024-03-06T07:32:50.050692+0000","last_undegraded":"2024-03-06T07:32:50.050692+0000","last_fullsized":"2024-03-06T07:32:50.050692+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:46:58.102449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047072+0000","last_change":"2024-03-06T07:32:48.475030+0000","last_active":"2024-03-06T07:32:50.047072+0000","last_peered":"2024-03-06T07:32:50.047072+0000","last_clean":"2024-03-06T07:32:50.047072+0000","last_became_active":"2024-03-06T07:32:48.474739+0000","last_became_peered":"2024-03-06T07:32:48.474739+0000","last_unstale":"2024-03-06T07:32:50.047072+0000","last_undegraded":"2024-03-06T07:32:50.047072+0000","last_fullsized":"2024-03-06T07:32:50.047072+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:39.580526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048627+0000","last_change":"2024-03-06T07:32:47.445429+0000","last_active":"2024-03-06T07:32:50.048627+0000","last_peered":"2024-03-06T07:32:50.048627+0000","last_clean":"2024-03-06T07:32:50.048627+0000","last_became_active":"2024-03-06T07:32:47.445125+0000","last_became_peered":"2024-03-06T07:32:47.445125+0000","last_unstale":"2024-03-06T07:32:50.048627+0000","last_undegraded":"2024-03-06T07:32:50.048627+0000","last_fullsized":"2024-03-06T07:32:50.048627+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:59.869095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475902+0000","last_change":"2024-03-06T07:32:48.486226+0000","last_active":"2024-03-06T07:32:49.475902+0000","last_peered":"2024-03-06T07:32:49.475902+0000","last_clean":"2024-03-06T07:32:49.475902+0000","last_became_active":"2024-03-06T07:32:48.485713+0000","last_became_peered":"2024-03-06T07:32:48.485713+0000","last_unstale":"2024-03-06T07:32:49.475902+0000","last_undegraded":"2024-03-06T07:32:49.475902+0000","last_fullsized":"2024-03-06T07:32:49.475902+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:09:52.871094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047306+0000","last_change":"2024-03-06T07:32:47.449171+0000","last_active":"2024-03-06T07:32:50.047306+0000","last_peered":"2024-03-06T07:32:50.047306+0000","last_clean":"2024-03-06T07:32:50.047306+0000","last_became_active":"2024-03-06T07:32:47.448643+0000","last_became_peered":"2024-03-06T07:32:47.448643+0000","last_unstale":"2024-03-06T07:32:50.047306+0000","last_undegraded":"2024-03-06T07:32:50.047306+0000","last_fullsized":"2024-03-06T07:32:50.047306+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:14.429471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045696+0000","last_change":"2024-03-06T07:32:48.493209+0000","last_active":"2024-03-06T07:32:50.045696+0000","last_peered":"2024-03-06T07:32:50.045696+0000","last_clean":"2024-03-06T07:32:50.045696+0000","last_became_active":"2024-03-06T07:32:48.492913+0000","last_became_peered":"2024-03-06T07:32:48.492913+0000","last_unstale":"2024-03-06T07:32:50.045696+0000","last_undegraded":"2024-03-06T07:32:50.045696+0000","last_fullsized":"2024-03-06T07:32:50.045696+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:56:27.141783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047067+0000","last_change":"2024-03-06T07:32:47.446224+0000","last_active":"2024-03-06T07:32:50.047067+0000","last_peered":"2024-03-06T07:32:50.047067+0000","last_clean":"2024-03-06T07:32:50.047067+0000","last_became_active":"2024-03-06T07:32:47.443188+0000","last_became_peered":"2024-03-06T07:32:47.443188+0000","last_unstale":"2024-03-06T07:32:50.047067+0000","last_undegraded":"2024-03-06T07:32:50.047067+0000","last_fullsized":"2024-03-06T07:32:50.047067+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:38.261444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046282+0000","last_change":"2024-03-06T07:32:48.487166+0000","last_active":"2024-03-06T07:32:50.046282+0000","last_peered":"2024-03-06T07:32:50.046282+0000","last_clean":"2024-03-06T07:32:50.046282+0000","last_became_active":"2024-03-06T07:32:48.484440+0000","last_became_peered":"2024-03-06T07:32:48.484440+0000","last_unstale":"2024-03-06T07:32:50.046282+0000","last_undegraded":"2024-03-06T07:32:50.046282+0000","last_fullsized":"2024-03-06T07:32:50.046282+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:25:31.928885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048292+0000","last_change":"2024-03-06T07:32:47.446946+0000","last_active":"2024-03-06T07:32:50.048292+0000","last_peered":"2024-03-06T07:32:50.048292+0000","last_clean":"2024-03-06T07:32:50.048292+0000","last_became_active":"2024-03-06T07:32:47.446516+0000","last_became_peered":"2024-03-06T07:32:47.446516+0000","last_unstale":"2024-03-06T07:32:50.048292+0000","last_undegraded":"2024-03-06T07:32:50.048292+0000","last_fullsized":"2024-03-06T07:32:50.048292+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:34.048060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045278+0000","last_change":"2024-03-06T07:32:48.495789+0000","last_active":"2024-03-06T07:32:50.045278+0000","last_peered":"2024-03-06T07:32:50.045278+0000","last_clean":"2024-03-06T07:32:50.045278+0000","last_became_active":"2024-03-06T07:32:48.495510+0000","last_became_peered":"2024-03-06T07:32:48.495510+0000","last_unstale":"2024-03-06T07:32:50.045278+0000","last_undegraded":"2024-03-06T07:32:50.045278+0000","last_fullsized":"2024-03-06T07:32:50.045278+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:59.625631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047323+0000","last_change":"2024-03-06T07:32:47.447401+0000","last_active":"2024-03-06T07:32:50.047323+0000","last_peered":"2024-03-06T07:32:50.047323+0000","last_clean":"2024-03-06T07:32:50.047323+0000","last_became_active":"2024-03-06T07:32:47.446563+0000","last_became_peered":"2024-03-06T07:32:47.446563+0000","last_unstale":"2024-03-06T07:32:50.047323+0000","last_undegraded":"2024-03-06T07:32:50.047323+0000","last_fullsized":"2024-03-06T07:32:50.047323+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:41.671563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047202+0000","last_change":"2024-03-06T07:32:48.472623+0000","last_active":"2024-03-06T07:32:50.047202+0000","last_peered":"2024-03-06T07:32:50.047202+0000","last_clean":"2024-03-06T07:32:50.047202+0000","last_became_active":"2024-03-06T07:32:48.470906+0000","last_became_peered":"2024-03-06T07:32:48.470906+0000","last_unstale":"2024-03-06T07:32:50.047202+0000","last_undegraded":"2024-03-06T07:32:50.047202+0000","last_fullsized":"2024-03-06T07:32:50.047202+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:34.090192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046057+0000","last_change":"2024-03-06T07:32:47.444926+0000","last_active":"2024-03-06T07:32:50.046057+0000","last_peered":"2024-03-06T07:32:50.046057+0000","last_clean":"2024-03-06T07:32:50.046057+0000","last_became_active":"2024-03-06T07:32:47.444619+0000","last_became_peered":"2024-03-06T07:32:47.444619+0000","last_unstale":"2024-03-06T07:32:50.046057+0000","last_undegraded":"2024-03-06T07:32:50.046057+0000","last_fullsized":"2024-03-06T07:32:50.046057+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:30.532542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048187+0000","last_change":"2024-03-06T07:32:48.477998+0000","last_active":"2024-03-06T07:32:50.048187+0000","last_peered":"2024-03-06T07:32:50.048187+0000","last_clean":"2024-03-06T07:32:50.048187+0000","last_became_active":"2024-03-06T07:32:48.477408+0000","last_became_peered":"2024-03-06T07:32:48.477408+0000","last_unstale":"2024-03-06T07:32:50.048187+0000","last_undegraded":"2024-03-06T07:32:50.048187+0000","last_fullsized":"2024-03-06T07:32:50.048187+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:01.194560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049544+0000","last_change":"2024-03-06T07:32:47.450076+0000","last_active":"2024-03-06T07:32:50.049544+0000","last_peered":"2024-03-06T07:32:50.049544+0000","last_clean":"2024-03-06T07:32:50.049544+0000","last_became_active":"2024-03-06T07:32:47.449766+0000","last_became_peered":"2024-03-06T07:32:47.449766+0000","last_unstale":"2024-03-06T07:32:50.049544+0000","last_undegraded":"2024-03-06T07:32:50.049544+0000","last_fullsized":"2024-03-06T07:32:50.049544+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:39.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474403+0000","last_change":"2024-03-06T07:32:48.475798+0000","last_active":"2024-03-06T07:32:49.474403+0000","last_peered":"2024-03-06T07:32:49.474403+0000","last_clean":"2024-03-06T07:32:49.474403+0000","last_became_active":"2024-03-06T07:32:48.473813+0000","last_became_peered":"2024-03-06T07:32:48.473813+0000","last_unstale":"2024-03-06T07:32:49.474403+0000","last_undegraded":"2024-03-06T07:32:49.474403+0000","last_fullsized":"2024-03-06T07:32:49.474403+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:59.010970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048271+0000","last_change":"2024-03-06T07:32:47.445510+0000","last_active":"2024-03-06T07:32:50.048271+0000","last_peered":"2024-03-06T07:32:50.048271+0000","last_clean":"2024-03-06T07:32:50.048271+0000","last_became_active":"2024-03-06T07:32:47.445220+0000","last_became_peered":"2024-03-06T07:32:47.445220+0000","last_unstale":"2024-03-06T07:32:50.048271+0000","last_undegraded":"2024-03-06T07:32:50.048271+0000","last_fullsized":"2024-03-06T07:32:50.048271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:06.096609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.473782+0000","last_change":"2024-03-06T07:32:48.472324+0000","last_active":"2024-03-06T07:32:49.473782+0000","last_peered":"2024-03-06T07:32:49.473782+0000","last_clean":"2024-03-06T07:32:49.473782+0000","last_became_active":"2024-03-06T07:32:48.471813+0000","last_became_peered":"2024-03-06T07:32:48.471813+0000","last_unstale":"2024-03-06T07:32:49.473782+0000","last_undegraded":"2024-03-06T07:32:49.473782+0000","last_fullsized":"2024-03-06T07:32:49.473782+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:14.022704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047633+0000","last_change":"2024-03-06T07:32:47.448498+0000","last_active":"2024-03-06T07:32:50.047633+0000","last_peered":"2024-03-06T07:32:50.047633+0000","last_clean":"2024-03-06T07:32:50.047633+0000","last_became_active":"2024-03-06T07:32:47.448209+0000","last_became_peered":"2024-03-06T07:32:47.448209+0000","last_unstale":"2024-03-06T07:32:50.047633+0000","last_undegraded":"2024-03-06T07:32:50.047633+0000","last_fullsized":"2024-03-06T07:32:50.047633+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:20.798460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.049512+0000","last_change":"2024-03-06T07:32:48.475439+0000","last_active":"2024-03-06T07:32:50.049512+0000","last_peered":"2024-03-06T07:32:50.049512+0000","last_clean":"2024-03-06T07:32:50.049512+0000","last_became_active":"2024-03-06T07:32:48.475144+0000","last_became_peered":"2024-03-06T07:32:48.475144+0000","last_unstale":"2024-03-06T07:32:50.049512+0000","last_undegraded":"2024-03-06T07:32:50.049512+0000","last_fullsized":"2024-03-06T07:32:50.049512+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:19:43.593642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.048523+0000","last_change":"2024-03-06T07:32:47.442329+0000","last_active":"2024-03-06T07:32:50.048523+0000","last_peered":"2024-03-06T07:32:50.048523+0000","last_clean":"2024-03-06T07:32:50.048523+0000","last_became_active":"2024-03-06T07:32:47.441500+0000","last_became_peered":"2024-03-06T07:32:47.441500+0000","last_unstale":"2024-03-06T07:32:50.048523+0000","last_undegraded":"2024-03-06T07:32:50.048523+0000","last_fullsized":"2024-03-06T07:32:50.048523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:14:30.523954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045958+0000","last_change":"2024-03-06T07:32:48.488649+0000","last_active":"2024-03-06T07:32:50.045958+0000","last_peered":"2024-03-06T07:32:50.045958+0000","last_clean":"2024-03-06T07:32:50.045958+0000","last_became_active":"2024-03-06T07:32:48.488344+0000","last_became_peered":"2024-03-06T07:32:48.488344+0000","last_unstale":"2024-03-06T07:32:50.045958+0000","last_undegraded":"2024-03-06T07:32:50.045958+0000","last_fullsized":"2024-03-06T07:32:50.045958+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:43:32.900051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047138+0000","last_change":"2024-03-06T07:32:47.445530+0000","last_active":"2024-03-06T07:32:50.047138+0000","last_peered":"2024-03-06T07:32:50.047138+0000","last_clean":"2024-03-06T07:32:50.047138+0000","last_became_active":"2024-03-06T07:32:47.442881+0000","last_became_peered":"2024-03-06T07:32:47.442881+0000","last_unstale":"2024-03-06T07:32:50.047138+0000","last_undegraded":"2024-03-06T07:32:50.047138+0000","last_fullsized":"2024-03-06T07:32:50.047138+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:59.869095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.050367+0000","last_change":"2024-03-06T07:32:48.475397+0000","last_active":"2024-03-06T07:32:50.050367+0000","last_peered":"2024-03-06T07:32:50.050367+0000","last_clean":"2024-03-06T07:32:50.050367+0000","last_became_active":"2024-03-06T07:32:48.475132+0000","last_became_peered":"2024-03-06T07:32:48.475132+0000","last_unstale":"2024-03-06T07:32:50.050367+0000","last_undegraded":"2024-03-06T07:32:50.050367+0000","last_fullsized":"2024-03-06T07:32:50.050367+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:40.102797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047771+0000","last_change":"2024-03-06T07:32:47.434557+0000","last_active":"2024-03-06T07:32:50.047771+0000","last_peered":"2024-03-06T07:32:50.047771+0000","last_clean":"2024-03-06T07:32:50.047771+0000","last_became_active":"2024-03-06T07:32:47.433650+0000","last_became_peered":"2024-03-06T07:32:47.433650+0000","last_unstale":"2024-03-06T07:32:50.047771+0000","last_undegraded":"2024-03-06T07:32:50.047771+0000","last_fullsized":"2024-03-06T07:32:50.047771+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:32.610977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046477+0000","last_change":"2024-03-06T07:32:48.490044+0000","last_active":"2024-03-06T07:32:50.046477+0000","last_peered":"2024-03-06T07:32:50.046477+0000","last_clean":"2024-03-06T07:32:50.046477+0000","last_became_active":"2024-03-06T07:32:48.489748+0000","last_became_peered":"2024-03-06T07:32:48.489748+0000","last_unstale":"2024-03-06T07:32:50.046477+0000","last_undegraded":"2024-03-06T07:32:50.046477+0000","last_fullsized":"2024-03-06T07:32:50.046477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:26.901895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.474923+0000","last_change":"2024-03-06T07:32:47.423971+0000","last_active":"2024-03-06T07:32:49.474923+0000","last_peered":"2024-03-06T07:32:49.474923+0000","last_clean":"2024-03-06T07:32:49.474923+0000","last_became_active":"2024-03-06T07:32:47.423700+0000","last_became_peered":"2024-03-06T07:32:47.423700+0000","last_unstale":"2024-03-06T07:32:49.474923+0000","last_undegraded":"2024-03-06T07:32:49.474923+0000","last_fullsized":"2024-03-06T07:32:49.474923+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:05.073516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.047383+0000","last_change":"2024-03-06T07:32:48.474750+0000","last_active":"2024-03-06T07:32:50.047383+0000","last_peered":"2024-03-06T07:32:50.047383+0000","last_clean":"2024-03-06T07:32:50.047383+0000","last_became_active":"2024-03-06T07:32:48.472684+0000","last_became_peered":"2024-03-06T07:32:48.472684+0000","last_unstale":"2024-03-06T07:32:50.047383+0000","last_undegraded":"2024-03-06T07:32:50.047383+0000","last_fullsized":"2024-03-06T07:32:50.047383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:57:40.747320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.045454+0000","last_change":"2024-03-06T07:32:47.436392+0000","last_active":"2024-03-06T07:32:50.045454+0000","last_peered":"2024-03-06T07:32:50.045454+0000","last_clean":"2024-03-06T07:32:50.045454+0000","last_became_active":"2024-03-06T07:32:47.435756+0000","last_became_peered":"2024-03-06T07:32:47.435756+0000","last_unstale":"2024-03-06T07:32:50.045454+0000","last_undegraded":"2024-03-06T07:32:50.045454+0000","last_fullsized":"2024-03-06T07:32:50.045454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:29.995646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:50.046961+0000","last_change":"2024-03-06T07:32:48.496683+0000","last_active":"2024-03-06T07:32:50.046961+0000","last_peered":"2024-03-06T07:32:50.046961+0000","last_clean":"2024-03-06T07:32:50.046961+0000","last_became_active":"2024-03-06T07:32:48.496473+0000","last_became_peered":"2024-03-06T07:32:48.496473+0000","last_unstale":"2024-03-06T07:32:50.046961+0000","last_undegraded":"2024-03-06T07:32:50.046961+0000","last_fullsized":"2024-03-06T07:32:50.046961+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:47.402268+0000","last_clean_scrub_stamp":"2024-03-06T07:32:47.402268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:43.379295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475229+0000","last_change":"2024-03-06T07:32:47.446682+0000","last_active":"2024-03-06T07:32:49.475229+0000","last_peered":"2024-03-06T07:32:49.475229+0000","last_clean":"2024-03-06T07:32:49.475229+0000","last_became_active":"2024-03-06T07:32:47.446061+0000","last_became_peered":"2024-03-06T07:32:47.446061+0000","last_unstale":"2024-03-06T07:32:49.475229+0000","last_undegraded":"2024-03-06T07:32:49.475229+0000","last_fullsized":"2024-03-06T07:32:49.475229+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:32:46.388265+0000","last_clean_scrub_stamp":"2024-03-06T07:32:46.388265+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:11.259835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":298,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:32:49.475303+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:32:49.475303+0000","last_peered":"2024-03-06T07:32:49.475303+0000","last_clean":"2024-03-06T07:32:49.475303+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:32:49.475303+0000","last_undegraded":"2024-03-06T07:32:49.475303+0000","last_fullsized":"2024-03-06T07:32:49.475303+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509509,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1146880,"data_stored":485771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509510,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1142784,"data_stored":485737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1744896,"data_stored":1076217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1150976,"data_stored":486389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36464,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335376,"statfs":{"total":96636764160,"available":96599425024,"internally_reserved":0,"allocated":1146880,"data_stored":485807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542213,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36476,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335364,"statfs":{"total":96636764160,"available":96599412736,"internally_reserved":0,"allocated":1163264,"data_stored":487473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339524,"statfs":{"total":96636764160,"available":96603672576,"internally_reserved":0,"allocated":1159168,"data_stored":487371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542214,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1740800,"data_stored":1076687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:53.662 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:32:53.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:32:53.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.926+0000 7f2163e00640 1 Processor -- start 2024-03-06T07:32:53.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.926+0000 7f2163e00640 1 -- start start 2024-03-06T07:32:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.927+0000 7f2163e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f215c0ad6f0 0x7f215c15c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.927+0000 7f2163e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c156830 con 0x7f215c15f960 2024-03-06T07:32:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.927+0000 7f2163e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215c160120 con 0x7f215c06d5d0 2024-03-06T07:32:53.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.927+0000 7f2163e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215c06ce60 con 0x7f215c0ad6f0 2024-03-06T07:32:53.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f2161400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f215c0ad6f0 0x7f215c15c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f2160a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f215c15f960 0x7f215c0adad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55220/0 (socket says 172.21.15.6:55220) 2024-03-06T07:32:53.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f2160a00640 1 -- 172.21.15.6:0/3867984541 learned_addr learned my addr 172.21.15.6:0/3867984541 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:53.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f2161400640 1 -- 172.21.15.6:0/3867984541 >> v1:172.21.15.178:6789/0 conn(0x7f215c06d5d0 legacy=0x7f215c06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:53.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f2161400640 1 -- 172.21.15.6:0/3867984541 >> v1:172.21.15.6:6789/0 conn(0x7f215c15f960 legacy=0x7f215c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f2161400640 1 -- 172.21.15.6:0/3867984541 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c160400 con 0x7f215c0ad6f0 2024-03-06T07:32:53.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.928+0000 7f215be00640 1 -- 172.21.15.6:0/3867984541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958354364 0 0) 0x7f215c156830 con 0x7f215c15f960 2024-03-06T07:32:53.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.930+0000 7f2161400640 1 --2- 172.21.15.6:0/3867984541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c15c370 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f215c06cfe0 tx=0x7f215002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb86408167d4be12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f215be00640 1 -- 172.21.15.6:0/3867984541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150035020 con 0x7f215c0ad6f0 2024-03-06T07:32:53.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f215be00640 1 -- 172.21.15.6:0/3867984541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150031830 con 0x7f215c0ad6f0 2024-03-06T07:32:53.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f215be00640 1 -- 172.21.15.6:0/3867984541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150031c00 con 0x7f215c0ad6f0 2024-03-06T07:32:53.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f2163e00640 1 -- 172.21.15.6:0/3867984541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 msgr2=0x7f215c15c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f2163e00640 1 --2- 172.21.15.6:0/3867984541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c15c370 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f215c06cfe0 tx=0x7f215002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f2163e00640 1 -- 172.21.15.6:0/3867984541 shutdown_connections 2024-03-06T07:32:53.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.931+0000 7f2163e00640 1 --2- 172.21.15.6:0/3867984541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c15c370 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.932+0000 7f2163e00640 1 -- 172.21.15.6:0/3867984541 >> 172.21.15.6:0/3867984541 conn(0x7f215c06b3d0 msgr2=0x7f215c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:53.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.932+0000 7f2163e00640 1 -- 172.21.15.6:0/3867984541 shutdown_connections 2024-03-06T07:32:53.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.932+0000 7f2163e00640 1 -- 172.21.15.6:0/3867984541 wait complete. 2024-03-06T07:32:53.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.933+0000 7f2163e00640 1 Processor -- start 2024-03-06T07:32:53.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.933+0000 7f2163e00640 1 -- start start 2024-03-06T07:32:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.934+0000 7f2163e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f215c06d5d0 0x7f215c14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.934+0000 7f2163e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c09c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.934+0000 7f2163e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f215c15f960 0x7f215c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.934+0000 7f2163e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c15fdf0 con 0x7f215c0ad6f0 2024-03-06T07:32:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.934+0000 7f2163e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215c06d0b0 con 0x7f215c06d5d0 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.934+0000 7f2163e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215c06e0f0 con 0x7f215c15f960 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c09c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c09c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56394/0 (socket says 172.21.15.6:56394) 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 -- 172.21.15.6:0/1748225162 learned_addr learned my addr 172.21.15.6:0/1748225162 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2161400640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f215c06d5d0 0x7f215c14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2161e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f215c15f960 0x7f215c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 -- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f215c15f960 msgr2=0x7f215c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f215c15f960 0x7f215c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 -- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f215c06d5d0 msgr2=0x7f215c14e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f215c06d5d0 0x7f215c14e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:53.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.935+0000 7f2160a00640 1 -- 172.21.15.6:0/1748225162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c09ad80 con 0x7f215c0ad6f0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.936+0000 7f2160a00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c09c6f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f214800c430 tx=0x7f214800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.936+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480074e0 con 0x7f215c0ad6f0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.936+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21480047e0 con 0x7f215c0ad6f0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.936+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148004bd0 con 0x7f215c0ad6f0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.936+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215c09b060 con 0x7f215c0ad6f0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.936+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215c09b620 con 0x7f215c0ad6f0 2024-03-06T07:32:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.937+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215c068010 con 0x7f215c0ad6f0 2024-03-06T07:32:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.939+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2148011070 con 0x7f215c0ad6f0 2024-03-06T07:32:53.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.940+0000 7f2159600640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f213c06b4d0 0x7f213c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:53.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.940+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2148089f20 con 0x7f215c0ad6f0 2024-03-06T07:32:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.941+0000 7f2161400640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f213c06b4d0 0x7f213c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:53.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.942+0000 7f2161400640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f213c06b4d0 0x7f213c06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2150036000 tx=0x7f215003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:53.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:53.945+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2148058d00 con 0x7f215c0ad6f0 2024-03-06T07:32:54.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.216+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f215c0b35b0 con 0x7f215c0ad6f0 2024-03-06T07:32:54.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.221+0000 7f2159600640 1 -- 172.21.15.6:0/1748225162 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2148058600 con 0x7f215c0ad6f0 2024-03-06T07:32:54.223 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:32:54.223 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:32:54.223 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:32:54.223 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:32:54.223 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:32:54.224 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f213c06b4d0 msgr2=0x7f213c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f213c06b4d0 0x7f213c06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2150036000 tx=0x7f215003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 msgr2=0x7f215c09c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c09c6f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f214800c430 tx=0x7f214800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 shutdown_connections 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f213c06b4d0 0x7f213c06d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f215c15f960 0x7f215c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f215c0ad6f0 0x7f215c09c6f0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 --2- 172.21.15.6:0/1748225162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f215c06d5d0 0x7f215c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.223+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 >> 172.21.15.6:0/1748225162 conn(0x7f215c06b3d0 msgr2=0x7f215c0b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.224+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 shutdown_connections 2024-03-06T07:32:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.224+0000 7f2163e00640 1 -- 172.21.15.6:0/1748225162 wait complete. 2024-03-06T07:32:54.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:54.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.463+0000 7f70f3e00640 1 Processor -- start 2024-03-06T07:32:54.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.463+0000 7f70f3e00640 1 -- start start 2024-03-06T07:32:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f70f4151b30 0x7f70f406f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70f40b4650 0x7f70f40b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f415c4a0 0x7f70f415c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f40b5000 con 0x7f70f40b4650 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f406e4b0 con 0x7f70f4151b30 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f406e790 con 0x7f70f415c4a0 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70ea000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70f40b4650 0x7f70f40b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70ea000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f70f40b4650 0x7f70f40b4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56396/0 (socket says 172.21.15.6:56396) 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70ea000640 1 -- 172.21.15.6:0/457580355 learned_addr learned my addr 172.21.15.6:0/457580355 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f3400640 1 --2- 172.21.15.6:0/457580355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f415c4a0 0x7f70f415c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.465+0000 7f70f2a00640 1 --2- 172.21.15.6:0/457580355 >> v2:172.21.15.178:3300/0 conn(0x7f70f4151b30 0x7f70f406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.466+0000 7f70ea000640 1 -- 172.21.15.6:0/457580355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f415c4a0 msgr2=0x7f70f415c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.466+0000 7f70ea000640 1 --2- 172.21.15.6:0/457580355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f415c4a0 0x7f70f415c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.466+0000 7f70ea000640 1 -- 172.21.15.6:0/457580355 >> v2:172.21.15.178:3300/0 conn(0x7f70f4151b30 msgr2=0x7f70f406f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.466+0000 7f70ea000640 1 --2- 172.21.15.6:0/457580355 >> v2:172.21.15.178:3300/0 conn(0x7f70f4151b30 0x7f70f406f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.466+0000 7f70ea000640 1 -- 172.21.15.6:0/457580355 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f406ea70 con 0x7f70f40b4650 2024-03-06T07:32:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.467+0000 7f70ea000640 1 --2- 172.21.15.6:0/457580355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f40b4650 0x7f70f40b4ac0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f70f40b6030 tx=0x7f70dc02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b26d731d9597a04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.467+0000 7f70f2000640 1 -- 172.21.15.6:0/457580355 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc03b070 con 0x7f70f40b4650 2024-03-06T07:32:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.467+0000 7f70f2000640 1 -- 172.21.15.6:0/457580355 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc02ee00 con 0x7f70f40b4650 2024-03-06T07:32:54.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.467+0000 7f70f2000640 1 -- 172.21.15.6:0/457580355 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc036660 con 0x7f70f40b4650 2024-03-06T07:32:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 -- 172.21.15.6:0/457580355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f40b4650 msgr2=0x7f70f40b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 --2- 172.21.15.6:0/457580355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f40b4650 0x7f70f40b4ac0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f70f40b6030 tx=0x7f70dc02e050 comp rx=0 tx=0).stop 2024-03-06T07:32:54.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 -- 172.21.15.6:0/457580355 shutdown_connections 2024-03-06T07:32:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 --2- 172.21.15.6:0/457580355 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f415c4a0 0x7f70f415c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 --2- 172.21.15.6:0/457580355 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f40b4650 0x7f70f40b4ac0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 --2- 172.21.15.6:0/457580355 >> v2:172.21.15.178:3300/0 conn(0x7f70f4151b30 0x7f70f406f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 -- 172.21.15.6:0/457580355 >> 172.21.15.6:0/457580355 conn(0x7f70f406c0c0 msgr2=0x7f70f406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 -- 172.21.15.6:0/457580355 shutdown_connections 2024-03-06T07:32:54.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.468+0000 7f70f3e00640 1 -- 172.21.15.6:0/457580355 wait complete. 2024-03-06T07:32:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.470+0000 7f70f3e00640 1 Processor -- start 2024-03-06T07:32:54.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.470+0000 7f70f3e00640 1 -- start start 2024-03-06T07:32:54.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.471+0000 7f70f3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f40b4650 0x7f70f41540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.471+0000 7f70f3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f4151b30 0x7f70f41545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.471+0000 7f70f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 0x7f70f40c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.471+0000 7f70f3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f4151f10 con 0x7f70f415c4a0 2024-03-06T07:32:54.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.471+0000 7f70f3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f415ce70 con 0x7f70f40b4650 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.471+0000 7f70f3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f406f7b0 con 0x7f70f4151b30 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 0x7f70f40c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 0x7f70f40c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56410/0 (socket says 172.21.15.6:56410) 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 -- 172.21.15.6:0/126018721 learned_addr learned my addr 172.21.15.6:0/126018721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70ea000640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f4151b30 0x7f70f41545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f2a00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f40b4650 0x7f70f41540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 -- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f4151b30 msgr2=0x7f70f41545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f4151b30 0x7f70f41545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 -- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f40b4650 msgr2=0x7f70f41540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f40b4650 0x7f70f41540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 -- 172.21.15.6:0/126018721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f40c5d80 con 0x7f70f415c4a0 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70ea000640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f4151b30 0x7f70f41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f2a00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f40b4650 0x7f70f41540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.472+0000 7f70f3400640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 0x7f70f40c6450 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f70ec007e70 tx=0x7f70ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.473+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec014620 con 0x7f70f415c4a0 2024-03-06T07:32:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.473+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec019070 con 0x7f70f415c4a0 2024-03-06T07:32:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.473+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec014a20 con 0x7f70f415c4a0 2024-03-06T07:32:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.473+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f40d05e0 con 0x7f70f415c4a0 2024-03-06T07:32:54.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.473+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f40d0ba0 con 0x7f70f415c4a0 2024-03-06T07:32:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.474+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f404f330 con 0x7f70f415c4a0 2024-03-06T07:32:54.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.475+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70ec014bc0 con 0x7f70f415c4a0 2024-03-06T07:32:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.476+0000 7f70eb400640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70d006b4d0 0x7f70d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.476+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70ec08aa90 con 0x7f70f415c4a0 2024-03-06T07:32:54.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.477+0000 7f70f2a00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70d006b4d0 0x7f70d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:54.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.478+0000 7f70f2a00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70d006b4d0 0x7f70d006d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006fd0 tx=0x7f70e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:54.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.481+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f70ec011260 con 0x7f70f415c4a0 2024-03-06T07:32:54.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.721+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f406e110 con 0x7f70f415c4a0 2024-03-06T07:32:54.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.723+0000 7f70eb400640 1 -- 172.21.15.6:0/126018721 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f70ec059900 con 0x7f70f415c4a0 2024-03-06T07:32:54.725 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:54.725 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":61,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/544102913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":544102913},{"type":"v1","addr":"172.21.15.178:6837","nonce":544102913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":6169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/833246901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":833246901},{"type":"v1","addr":"172.21.15.178:6836","nonce":833246901}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":7155,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3875786404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3875786404},{"type":"v1","addr":"172.21.15.6:6833","nonce":3875786404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:32:50.992363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7167},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7167":{"gid":7167,"name":"c","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6835/2859903799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2859903799},{"type":"v1","addr":"172.21.15.6:6835","nonce":2859903799}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7167,"qdb_cluster":[7167]},"id":7}]} 2024-03-06T07:32:54.725 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 61 2024-03-06T07:32:54.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70d006b4d0 msgr2=0x7f70d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70d006b4d0 0x7f70d006d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006fd0 tx=0x7f70e4008040 comp rx=0 tx=0).stop 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 msgr2=0x7f70f40c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 0x7f70f40c6450 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f70ec007e70 tx=0x7f70ec00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 shutdown_connections 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70d006b4d0 0x7f70d006d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f415c4a0 0x7f70f40c6450 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f4151b30 0x7f70f41545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 --2- 172.21.15.6:0/126018721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f40b4650 0x7f70f41540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 >> 172.21.15.6:0/126018721 conn(0x7f70f406c0c0 msgr2=0x7f70f406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 shutdown_connections 2024-03-06T07:32:54.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:54.725+0000 7f70f3e00640 1 -- 172.21.15.6:0/126018721 wait complete. 2024-03-06T07:32:55.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:47.892948+0000', 'modified': '2024-03-06T07:32:50.992363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7167}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7167': {'gid': 7167, 'name': 'c', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6835/2859903799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 2859903799}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 2859903799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7167, 'qdb_cluster': [7167]} 2024-03-06T07:32:55.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:32:55.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:32:55.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.368+0000 7f43f5a00640 1 Processor -- start 2024-03-06T07:32:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.369+0000 7f43f5a00640 1 -- start start 2024-03-06T07:32:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43f5a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43f00b4650 0x7f43f00b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43f5a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f015c4a0 0x7f43f015c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43f5a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f006e4b0 con 0x7f43f0151b30 2024-03-06T07:32:55.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43f5a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f00b5000 con 0x7f43f00b4650 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43f5a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f006e790 con 0x7f43f015c4a0 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43ef400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f43f00b4650 0x7f43f00b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43efe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f43f0151b30 0x7f43f006f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55224/0 (socket says 172.21.15.6:55224) 2024-03-06T07:32:55.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.370+0000 7f43efe00640 1 -- 172.21.15.6:0/2751048155 learned_addr learned my addr 172.21.15.6:0/2751048155 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.371+0000 7f43f5000640 1 --2- 172.21.15.6:0/2751048155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f015c4a0 0x7f43f015c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.371+0000 7f43ef400640 1 -- 172.21.15.6:0/2751048155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f015c4a0 msgr2=0x7f43f015c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.371+0000 7f43ef400640 1 --2- 172.21.15.6:0/2751048155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f015c4a0 0x7f43f015c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.371+0000 7f43ef400640 1 -- 172.21.15.6:0/2751048155 >> v1:172.21.15.6:6789/0 conn(0x7f43f0151b30 legacy=0x7f43f006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.371+0000 7f43ef400640 1 -- 172.21.15.6:0/2751048155 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f006ea70 con 0x7f43f00b4650 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.371+0000 7f43eea00640 1 -- 172.21.15.6:0/2751048155 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364568222 0 0) 0x7f43f006e4b0 con 0x7f43f0151b30 2024-03-06T07:32:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.372+0000 7f43ef400640 1 --2- 172.21.15.6:0/2751048155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f00b4650 0x7f43f00b4ac0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f43e00097a0 tx=0x7f43e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7834af6834fef141 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.372+0000 7f43eea00640 1 -- 172.21.15.6:0/2751048155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e00315a0 con 0x7f43f00b4650 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.372+0000 7f43eea00640 1 -- 172.21.15.6:0/2751048155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e0037070 con 0x7f43f00b4650 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.372+0000 7f43eea00640 1 -- 172.21.15.6:0/2751048155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0031960 con 0x7f43f00b4650 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 -- 172.21.15.6:0/2751048155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f00b4650 msgr2=0x7f43f00b4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 --2- 172.21.15.6:0/2751048155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f00b4650 0x7f43f00b4ac0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f43e00097a0 tx=0x7f43e002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 -- 172.21.15.6:0/2751048155 shutdown_connections 2024-03-06T07:32:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 --2- 172.21.15.6:0/2751048155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f015c4a0 0x7f43f015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 --2- 172.21.15.6:0/2751048155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f00b4650 0x7f43f00b4ac0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 -- 172.21.15.6:0/2751048155 >> 172.21.15.6:0/2751048155 conn(0x7f43f006c0c0 msgr2=0x7f43f006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 -- 172.21.15.6:0/2751048155 shutdown_connections 2024-03-06T07:32:55.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.373+0000 7f43f5a00640 1 -- 172.21.15.6:0/2751048155 wait complete. 2024-03-06T07:32:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.376+0000 7f43f5a00640 1 Processor -- start 2024-03-06T07:32:55.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.376+0000 7f43f5a00640 1 -- start start 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f00b4650 0x7f43f0149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43f0151b30 0x7f43f0147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 0x7f43f0139730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0151f10 con 0x7f43f015c4a0 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f015ce70 con 0x7f43f0151b30 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f015e8b0 con 0x7f43f00b4650 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 0x7f43f0139730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 0x7f43f0139730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56426/0 (socket says 172.21.15.6:56426) 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43f5000640 1 -- 172.21.15.6:0/951639244 learned_addr learned my addr 172.21.15.6:0/951639244 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.378+0000 7f43efe00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f00b4650 0x7f43f0149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5000640 1 -- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f00b4650 msgr2=0x7f43f0149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5000640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f00b4650 0x7f43f0149360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5000640 1 -- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43f0151b30 msgr2=0x7f43f0147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5000640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43f0151b30 0x7f43f0147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5000640 1 -- 172.21.15.6:0/951639244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0149ce0 con 0x7f43f015c4a0 2024-03-06T07:32:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43efe00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f00b4650 0x7f43f0149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5000640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 0x7f43f0139730 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f43e800cc30 tx=0x7f43e800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43e7e00640 1 -- 172.21.15.6:0/951639244 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e800ce40 con 0x7f43f015c4a0 2024-03-06T07:32:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43e7e00640 1 -- 172.21.15.6:0/951639244 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8003990 con 0x7f43f015c4a0 2024-03-06T07:32:55.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f0149fc0 con 0x7f43f015c4a0 2024-03-06T07:32:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43e7e00640 1 -- 172.21.15.6:0/951639244 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8003d40 con 0x7f43f015c4a0 2024-03-06T07:32:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.379+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f014a580 con 0x7f43f015c4a0 2024-03-06T07:32:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.382+0000 7f43e7e00640 1 -- 172.21.15.6:0/951639244 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43e8003ee0 con 0x7f43f015c4a0 2024-03-06T07:32:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.382+0000 7f43e7e00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43cc06b4d0 0x7f43cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.383+0000 7f43e7e00640 1 -- 172.21.15.6:0/951639244 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43e808a650 con 0x7f43f015c4a0 2024-03-06T07:32:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.383+0000 7f43efe00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43cc06b4d0 0x7f43cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.383+0000 7f43f9e23740 1 Processor -- start 2024-03-06T07:32:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.384+0000 7f43f9e23740 1 -- start start 2024-03-06T07:32:55.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.384+0000 7f43efe00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43cc06b4d0 0x7f43cc06d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f43d8000f60 tx=0x7f43d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529adda0 0x564552980d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 0x5645529b1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f9e23740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645528e6600 con 0x564552981280 2024-03-06T07:32:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56455288f930 con 0x5645529ae180 2024-03-06T07:32:55.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5645528e26a0 con 0x5645529adda0 2024-03-06T07:32:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 0x5645529b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 0x5645529b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56430/0 (socket says 172.21.15.6:56430) 2024-03-06T07:32:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 -- 172.21.15.6:0/322387209 learned_addr learned my addr 172.21.15.6:0/322387209 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43efe00640 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529adda0 0x564552980d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 -- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529adda0 msgr2=0x564552980d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529adda0 0x564552980d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43ef400640 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 -- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 msgr2=0x5645529b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.385+0000 7f43f5000640 1 -- 172.21.15.6:0/322387209 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645525fb2d0 con 0x564552981280 2024-03-06T07:32:55.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.386+0000 7f43f5000640 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 0x5645529b1070 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f43e8088bb0 tx=0x7f43e8058ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f52022964fb42d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.386+0000 7f43e4c00640 1 -- 172.21.15.6:0/322387209 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8070020 con 0x564552981280 2024-03-06T07:32:55.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.386+0000 7f43e4c00640 1 -- 172.21.15.6:0/322387209 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8061570 con 0x564552981280 2024-03-06T07:32:55.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.386+0000 7f43e4c00640 1 -- 172.21.15.6:0/322387209 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e801d040 con 0x564552981280 2024-03-06T07:32:55.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.386+0000 7f43f9e23740 1 -- 172.21.15.6:0/322387209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 msgr2=0x5645529b1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.386+0000 7f43f9e23740 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 0x5645529b1070 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f43e8088bb0 tx=0x7f43e8058ad0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 -- 172.21.15.6:0/322387209 shutdown_connections 2024-03-06T07:32:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x564552981280 0x5645529b1070 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 --2- 172.21.15.6:0/322387209 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529adda0 0x564552980d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 -- 172.21.15.6:0/322387209 >> 172.21.15.6:0/322387209 conn(0x564552982fb0 msgr2=0x5645529cbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 -- 172.21.15.6:0/322387209 shutdown_connections 2024-03-06T07:32:55.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 -- 172.21.15.6:0/322387209 wait complete. 2024-03-06T07:32:55.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.387+0000 7f43f9e23740 1 -- start start 2024-03-06T07:32:55.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.388+0000 7f43f9e23740 1 Processor -- start 2024-03-06T07:32:55.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529dc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x5645529c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x564552985550 con 0x5645529adda0 2024-03-06T07:32:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645528a2d90 con 0x5645529ae180 2024-03-06T07:32:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x564552984940 con 0x5645529c3150 2024-03-06T07:32:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f9e23740 10 client.0 _ll_register_callbacks cb 0x5645528de350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529dc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529dc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56434/0 (socket says 172.21.15.6:56434) 2024-03-06T07:32:55.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 -- 172.21.15.6:0/1371737961 learned_addr learned my addr 172.21.15.6:0/1371737961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43ef400640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 -- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 msgr2=0x5645529c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43f5000640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x5645529c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x5645529c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 -- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 msgr2=0x5645529c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43efe00640 1 -- 172.21.15.6:0/1371737961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645526c56b0 con 0x5645529adda0 2024-03-06T07:32:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.389+0000 7f43ef400640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43efe00640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529dc880 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005a70 tx=0x7f43d8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1380ea3a9b54cdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43dd600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:32:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43dd600640 1 -- 172.21.15.6:0/1371737961 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d8026060 con 0x5645529adda0 2024-03-06T07:32:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43dd600640 1 -- 172.21.15.6:0/1371737961 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43d8021a90 con 0x5645529adda0 2024-03-06T07:32:55.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43e8061570 con 0x5645529adda0 2024-03-06T07:32:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43dd600640 1 -- 172.21.15.6:0/1371737961 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43d8012410 con 0x5645529adda0 2024-03-06T07:32:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43f9e23740 10 client.7296 fetch_fsmap learned FSMap version 61 2024-03-06T07:32:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.390+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x5645529e5420 con 0x5645529adda0 2024-03-06T07:32:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43dd600640 1 -- 172.21.15.6:0/1371737961 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8061570 con 0x5645529adda0 2024-03-06T07:32:55.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43dd600640 1 -- 172.21.15.6:0/1371737961 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f43d80125b0 con 0x5645529adda0 2024-03-06T07:32:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43f9e23740 10 client.7296 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:32:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43f9e23740 10 client.7296 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43f9e23740 20 client.7296 populate_metadata read hostname 'smithi006' 2024-03-06T07:32:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x5645529e5bd0 0x5645529e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43f9e23740 4 client.7296 mds_command: new command op to 7167 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:32:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 --> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645528f3b40 con 0x5645529e5bd0 2024-03-06T07:32:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.391+0000 7f43ef400640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x5645529e5bd0 0x5645529e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.392+0000 7f43ef400640 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x5645529e5bd0 0x5645529e9d80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ac3f00940b43ba server_cookie=a94274f96f14376e in_seq=0 out_seq=0 2024-03-06T07:32:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.392+0000 7f43dd600640 10 client.7296 ms_handle_connect on v2:172.21.15.6:6834/2859903799 2024-03-06T07:32:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43dd600640 1 -- 172.21.15.6:0/1371737961 <== mds.0 v2:172.21.15.6:6834/2859903799 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5645528f3b40 con 0x5645529e5bd0 2024-03-06T07:32:55.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43dd600640 10 client.7296 handle_command_reply: tid=0 2024-03-06T07:32:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 client.7296 shutdown 2024-03-06T07:32:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 msgr2=0x5645529dc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529dc880 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005a70 tx=0x7f43d8000ed0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 shutdown_connections 2024-03-06T07:32:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x5645529e5bd0 0x5645529e9d80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x5645529c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1371737961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529dc880 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 >> 172.21.15.6:0/1371737961 conn(0x564552981280 msgr2=0x56455297c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43dd600640 0 client.7296 ms_handle_reset on v2:172.21.15.6:6834/2859903799 2024-03-06T07:32:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 shutdown_connections 2024-03-06T07:32:55.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 1 -- 172.21.15.6:0/1371737961 wait complete. 2024-03-06T07:32:55.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.393+0000 7f43f9e23740 20 client.7296 trim_cache size 0 max 16384 2024-03-06T07:32:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.412+0000 7f43f9e23740 1 Processor -- start 2024-03-06T07:32:55.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.412+0000 7f43f9e23740 1 -- start start 2024-03-06T07:32:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x564552984ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43f9e23740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0154cd0 con 0x5645529adda0 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01540d0 con 0x5645529ae180 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01546d0 con 0x5645529c3150 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56438/0 (socket says 172.21.15.6:56438) 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 -- 172.21.15.6:0/3076291165 learned_addr learned my addr 172.21.15.6:0/3076291165 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 -- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 msgr2=0x564552984ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x564552984ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43ef400640 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 -- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 msgr2=0x5645529c25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.413+0000 7f43efe00640 1 -- 172.21.15.6:0/3076291165 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645529c3b10 con 0x5645529adda0 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.414+0000 7f43efe00640 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529b11e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f43d800d640 tx=0x7f43d801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640968113afa76c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.414+0000 7f43dfe00640 1 -- 172.21.15.6:0/3076291165 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d8030070 con 0x5645529adda0 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.414+0000 7f43dfe00640 1 -- 172.21.15.6:0/3076291165 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8012a60 con 0x5645529adda0 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.414+0000 7f43dfe00640 1 -- 172.21.15.6:0/3076291165 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d8012270 con 0x5645529adda0 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 -- 172.21.15.6:0/3076291165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 msgr2=0x5645529b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529b11e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f43d800d640 tx=0x7f43d801e000 comp rx=0 tx=0).stop 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 -- 172.21.15.6:0/3076291165 shutdown_connections 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529ae180 0x5645529c25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529c3150 0x564552984ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 --2- 172.21.15.6:0/3076291165 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529adda0 0x5645529b11e0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 -- 172.21.15.6:0/3076291165 >> 172.21.15.6:0/3076291165 conn(0x564552981280 msgr2=0x5645529cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 -- 172.21.15.6:0/3076291165 shutdown_connections 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 -- 172.21.15.6:0/3076291165 wait complete. 2024-03-06T07:32:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.415+0000 7f43f9e23740 1 -- start start 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.416+0000 7f43f9e23740 1 Processor -- start 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529adda0 0x564552984ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529ae180 0x5645529c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 0x5645529cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645529e5a50 con 0x5645529c3150 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645529e5650 con 0x5645529adda0 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5645529e5fc0 con 0x5645529ae180 2024-03-06T07:32:55.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f9e23740 10 client.0 _ll_register_callbacks cb 0x7f43d80013d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 0x5645529cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 0x5645529cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56440/0 (socket says 172.21.15.6:56440) 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 -- 172.21.15.6:0/1302066299 learned_addr learned my addr 172.21.15.6:0/1302066299 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43efe00640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529adda0 0x564552984ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43ef400640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529ae180 0x5645529c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 -- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529ae180 msgr2=0x5645529c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529ae180 0x5645529c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 -- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529adda0 msgr2=0x564552984ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529adda0 0x564552984ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43f5000640 1 -- 172.21.15.6:0/1302066299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645529cd980 con 0x5645529c3150 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.417+0000 7f43ef400640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529ae180 0x5645529c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43f5000640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 0x5645529cd440 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f43e8088b80 tx=0x7f43e8088ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae916a70df9651f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43dd600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43dd600640 1 -- 172.21.15.6:0/1302066299 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e801d040 con 0x5645529c3150 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43dd600640 1 -- 172.21.15.6:0/1302066299 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43e80610e0 con 0x5645529c3150 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43d8012a60 con 0x5645529c3150 2024-03-06T07:32:55.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43dd600640 1 -- 172.21.15.6:0/1302066299 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43e8070040 con 0x5645529c3150 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43f9e23740 10 client.7302 fetch_fsmap learned FSMap version 61 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) v3 -- 0x56455297c300 con 0x5645529c3150 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43dd600640 1 -- 172.21.15.6:0/1302066299 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d8012a60 con 0x5645529c3150 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.418+0000 7f43dd600640 1 -- 172.21.15.6:0/1302066299 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 61) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f43e80582b0 con 0x5645529c3150 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43f9e23740 10 client.7302 fetch_fsmap finished waiting for FSMap version 61 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43f9e23740 10 client.7302 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43f9e23740 20 client.7302 populate_metadata read hostname 'smithi006' 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x564552a2f310 0x564552a2f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43f9e23740 4 client.7302 mds_command: new command op to 7167 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 --> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5645528ede90 con 0x564552a2f310 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43efe00640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x564552a2f310 0x564552a2f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43efe00640 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x564552a2f310 0x564552a2f6f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f6d6d1c9478c9de server_cookie=910ab62f6855588d in_seq=0 out_seq=0 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.419+0000 7f43dd600640 10 client.7302 ms_handle_connect on v2:172.21.15.6:6834/2859903799 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43dd600640 1 -- 172.21.15.6:0/1302066299 <== mds.0 v2:172.21.15.6:6834/2859903799 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5645528ede90 con 0x564552a2f310 2024-03-06T07:32:55.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43dd600640 10 client.7302 handle_command_reply: tid=0 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 client.7302 shutdown 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 msgr2=0x5645529cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 0x5645529cd440 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f43e8088b80 tx=0x7f43e8088ae0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 shutdown_connections 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:6834/2859903799,v1:172.21.15.6:6835/2859903799] conn(0x564552a2f310 0x564552a2f6f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5645529c3150 0x5645529cd440 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5645529ae180 0x5645529c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 --2- 172.21.15.6:0/1302066299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5645529adda0 0x564552984ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 >> 172.21.15.6:0/1302066299 conn(0x564552981280 msgr2=0x5645529cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43dd600640 0 client.7302 ms_handle_reset on v2:172.21.15.6:6834/2859903799 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 shutdown_connections 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 1 -- 172.21.15.6:0/1302066299 wait complete. 2024-03-06T07:32:55.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.420+0000 7f43f9e23740 20 client.7302 trim_cache size 0 max 16384 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7167, 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.9596585360000001, 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 61, 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 93, 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 93, 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.280594237000001 2024-03-06T07:32:55.428 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:32:55.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43cc06b4d0 msgr2=0x7f43cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43cc06b4d0 0x7f43cc06d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f43d8000f60 tx=0x7f43d8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:55.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 msgr2=0x7f43f0139730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 0x7f43f0139730 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f43e800cc30 tx=0x7f43e800a5c0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 shutdown_connections 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f43cc06b4d0 0x7f43cc06d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43f0151b30 0x7f43f0147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f43f015c4a0 0x7f43f0139730 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 --2- 172.21.15.6:0/951639244 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f43f00b4650 0x7f43f0149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 >> 172.21.15.6:0/951639244 conn(0x7f43f006c0c0 msgr2=0x7f43f014f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 shutdown_connections 2024-03-06T07:32:55.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.421+0000 7f43f5a00640 1 -- 172.21.15.6:0/951639244 wait complete. 2024-03-06T07:32:55.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:32:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.654+0000 7ffb39200640 1 Processor -- start 2024-03-06T07:32:55.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.654+0000 7ffb39200640 1 -- start start 2024-03-06T07:32:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb39200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb3415dec0 0x7ffb340ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb39200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ffb3415e2a0 0x7ffb340ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb39200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb340cf290 0x7ffb340d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb39200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340b0ea0 con 0x7ffb340cf290 2024-03-06T07:32:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb39200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340cf720 con 0x7ffb3415e2a0 2024-03-06T07:32:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb39200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d1120 con 0x7ffb3415dec0 2024-03-06T07:32:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb33e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb340cf290 0x7ffb340d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb33e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffb340cf290 0x7ffb340d1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56452/0 (socket says 172.21.15.6:56452) 2024-03-06T07:32:55.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.655+0000 7ffb33e00640 1 -- 172.21.15.6:0/2974278027 learned_addr learned my addr 172.21.15.6:0/2974278027 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb32a00640 1 --2- 172.21.15.6:0/2974278027 >> v2:172.21.15.178:3300/0 conn(0x7ffb3415e2a0 0x7ffb340ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb33e00640 1 -- 172.21.15.6:0/2974278027 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb3415dec0 msgr2=0x7ffb340ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb33400640 1 --2- 172.21.15.6:0/2974278027 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb3415dec0 0x7ffb340ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb33e00640 1 --2- 172.21.15.6:0/2974278027 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb3415dec0 0x7ffb340ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb33e00640 1 -- 172.21.15.6:0/2974278027 >> v2:172.21.15.178:3300/0 conn(0x7ffb3415e2a0 msgr2=0x7ffb340ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb33e00640 1 --2- 172.21.15.6:0/2974278027 >> v2:172.21.15.178:3300/0 conn(0x7ffb3415e2a0 0x7ffb340ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb33e00640 1 -- 172.21.15.6:0/2974278027 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340d1400 con 0x7ffb340cf290 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.656+0000 7ffb32a00640 1 --2- 172.21.15.6:0/2974278027 >> v2:172.21.15.178:3300/0 conn(0x7ffb3415e2a0 0x7ffb340ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.657+0000 7ffb33e00640 1 --2- 172.21.15.6:0/2974278027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb340cf290 0x7ffb340d1b20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ffb3408cab0 tx=0x7ffb2c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24242e00f60077a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.657+0000 7ffb32000640 1 -- 172.21.15.6:0/2974278027 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c035020 con 0x7ffb340cf290 2024-03-06T07:32:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.657+0000 7ffb32000640 1 -- 172.21.15.6:0/2974278027 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c031880 con 0x7ffb340cf290 2024-03-06T07:32:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.657+0000 7ffb32000640 1 -- 172.21.15.6:0/2974278027 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c031c50 con 0x7ffb340cf290 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 -- 172.21.15.6:0/2974278027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb340cf290 msgr2=0x7ffb340d1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 --2- 172.21.15.6:0/2974278027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb340cf290 0x7ffb340d1b20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ffb3408cab0 tx=0x7ffb2c02b920 comp rx=0 tx=0).stop 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 -- 172.21.15.6:0/2974278027 shutdown_connections 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 --2- 172.21.15.6:0/2974278027 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb340cf290 0x7ffb340d1b20 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 --2- 172.21.15.6:0/2974278027 >> v2:172.21.15.178:3300/0 conn(0x7ffb3415e2a0 0x7ffb340ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 --2- 172.21.15.6:0/2974278027 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb3415dec0 0x7ffb340ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 -- 172.21.15.6:0/2974278027 >> 172.21.15.6:0/2974278027 conn(0x7ffb340d3d40 msgr2=0x7ffb340d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 -- 172.21.15.6:0/2974278027 shutdown_connections 2024-03-06T07:32:55.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.658+0000 7ffb39200640 1 -- 172.21.15.6:0/2974278027 wait complete. 2024-03-06T07:32:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.660+0000 7ffb39200640 1 Processor -- start 2024-03-06T07:32:55.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.661+0000 7ffb39200640 1 -- start start 2024-03-06T07:32:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.662+0000 7ffb39200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb340cf290 0x7ffb340ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.662+0000 7ffb39200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb3415dec0 0x7ffb3415b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.662+0000 7ffb39200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 0x7ffb340cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.662+0000 7ffb39200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d0ec0 con 0x7ffb3415e2a0 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.662+0000 7ffb39200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d0d40 con 0x7ffb3415dec0 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.662+0000 7ffb39200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d2060 con 0x7ffb340cf290 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 0x7ffb340cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 0x7ffb340cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56468/0 (socket says 172.21.15.6:56468) 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 -- 172.21.15.6:0/1287857719 learned_addr learned my addr 172.21.15.6:0/1287857719 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33400640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb340cf290 0x7ffb340ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb32a00640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb3415dec0 0x7ffb3415b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 -- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb340cf290 msgr2=0x7ffb340ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb340cf290 0x7ffb340ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 -- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb3415dec0 msgr2=0x7ffb3415b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb3415dec0 0x7ffb3415b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 -- 172.21.15.6:0/1287857719 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3415e6c0 con 0x7ffb3415e2a0 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb32a00640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb3415dec0 0x7ffb3415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:55.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.663+0000 7ffb33e00640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 0x7ffb340cae10 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c02b960 tx=0x7ffb2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.664+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c040070 con 0x7ffb3415e2a0 2024-03-06T07:32:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.664+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c03a420 con 0x7ffb3415e2a0 2024-03-06T07:32:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.664+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c035040 con 0x7ffb3415e2a0 2024-03-06T07:32:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.664+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb340cb350 con 0x7ffb3415e2a0 2024-03-06T07:32:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.664+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb340cb8d0 con 0x7ffb3415e2a0 2024-03-06T07:32:55.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.666+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3404f330 con 0x7ffb3415e2a0 2024-03-06T07:32:55.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.666+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb2c038020 con 0x7ffb3415e2a0 2024-03-06T07:32:55.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.667+0000 7ffb2b400640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb1006b4d0 0x7ffb1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:55.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.668+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb2c0ac300 con 0x7ffb3415e2a0 2024-03-06T07:32:55.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.668+0000 7ffb33400640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb1006b4d0 0x7ffb1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:55.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.670+0000 7ffb33400640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb1006b4d0 0x7ffb1006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ffb34161510 tx=0x7ffb240079a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:55.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.672+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb3415e6c0 con 0x7ffb3415e2a0 2024-03-06T07:32:55.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.941+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffb340ca780 con 0x7ffb3415e2a0 2024-03-06T07:32:55.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.942+0000 7ffb2b400640 1 -- 172.21.15.6:0/1287857719 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffb2c07b0e0 con 0x7ffb3415e2a0 2024-03-06T07:32:55.944 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:32:55.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb1006b4d0 msgr2=0x7ffb1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb1006b4d0 0x7ffb1006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ffb34161510 tx=0x7ffb240079a0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 msgr2=0x7ffb340cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 0x7ffb340cae10 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c02b960 tx=0x7ffb2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 shutdown_connections 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffb1006b4d0 0x7ffb1006d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffb3415dec0 0x7ffb3415b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffb3415e2a0 0x7ffb340cae10 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 --2- 172.21.15.6:0/1287857719 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffb340cf290 0x7ffb340ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 >> 172.21.15.6:0/1287857719 conn(0x7ffb340d3d40 msgr2=0x7ffb3406c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 shutdown_connections 2024-03-06T07:32:55.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:55.945+0000 7ffb39200640 1 -- 172.21.15.6:0/1287857719 wait complete. 2024-03-06T07:32:55.966 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:32:55.966 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:32:55.966 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:32:55.966 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:32:55.966 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:55.966 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:32:55.967 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:32:55.967 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:32:55.967 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:55.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:55.967 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.014 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42281sec preferred_lft 42281sec 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:32:56.015 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:32:56.016 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:32:56.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.017 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:32:56.145 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:32:56.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.149 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:32:56.198 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:32:56.198 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:32:56.198 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:32:56.199 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:32:56.199 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:32:56.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.199 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:32:56.199 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:32:56.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:32:56.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:32:56.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:32:56.199 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:32:56.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.378 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:32:56.428 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:56.428 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:56.429 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:56.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:32:56.520 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:56.521 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:56.572 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:32:56.573 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:32:56.573 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:56.617 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:32:56.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:56.802 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:56.803 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:32:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:32:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:32:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:32:56.835 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:32:56.837 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:32:56.837 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:32:56.837 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:32:56.837 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:32:56.837 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:32:56.842 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:32:56.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:56.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:56.919 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:32:56.975 INFO:teuthology.orchestra.run.smithi006.stdout:client7311 2024-03-06T07:32:56.975 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:32:56.976 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:57.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:57.065 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:32:57.247 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.248 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:32:57.249 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:32:57.250 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-06T07:32:57.251 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.252 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:57.253 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:32:57.254 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:32:57.255 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:57.256 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:57.257 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:32:57.258 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:57.259 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:57.260 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:32:57.261 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.262 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_files = 6 2024-03-06T07:32:57.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:57.304 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.468 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.469 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.470 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:57.471 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:32:57.472 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:57.473 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.474 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.475 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:32:57.476 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:32:57.477 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_files = 6 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:57.478 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.479 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:57.480 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:32:57.481 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:32:57.482 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:57.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:57.484 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:32:57.485 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:57.486 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:32:57.487 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:57.488 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-06T07:32:57.489 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:57.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:57.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.734+0000 7fe0dfc00640 1 Processor -- start 2024-03-06T07:32:57.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.735+0000 7fe0dfc00640 1 -- start start 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.736+0000 7fe0dfc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe0d8070d80 0x7fe0d80cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.736+0000 7fe0dfc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d8161d80 con 0x7fe0d8071160 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.737+0000 7fe0dfc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d8162060 con 0x7fe0d815e830 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.737+0000 7fe0dfc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d8162340 con 0x7fe0d8070d80 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.737+0000 7fe0d7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe0d8071160 0x7fe0d80b6cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55230/0 (socket says 172.21.15.6:55230) 2024-03-06T07:32:57.739 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.737+0000 7fe0d7e00640 1 -- 172.21.15.6:0/3108524535 learned_addr learned my addr 172.21.15.6:0/3108524535 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.737+0000 7fe0dd200640 1 --2- 172.21.15.6:0/3108524535 >> v2:172.21.15.178:3300/0 conn(0x7fe0d8070d80 0x7fe0d80cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.737+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645930191 0 0) 0x7fe0d8161d80 con 0x7fe0d8071160 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c0003520 con 0x7fe0d8071160 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4133193743 0 0) 0x7fe0c0003520 con 0x7fe0d8071160 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 >> v2:172.21.15.178:3300/0 conn(0x7fe0d8070d80 msgr2=0x7fe0d80cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 --2- 172.21.15.6:0/3108524535 >> v2:172.21.15.178:3300/0 conn(0x7fe0d8070d80 0x7fe0d80cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 >> v1:172.21.15.178:6789/0 conn(0x7fe0d815e830 legacy=0x7fe0d815ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d8162620 con 0x7fe0d8071160 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fe0c8001670 con 0x7fe0d8071160 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 >> v1:172.21.15.6:6789/0 conn(0x7fe0d8071160 legacy=0x7fe0d80b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0c0003a00 0x7fe0c0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0c0004800 0x7fe0c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8162340 con 0x7fe0c00051c0 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c000c0c0 con 0x7fe0c0004800 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.738+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c000c2c0 con 0x7fe0c0003a00 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0ddc00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0dd200640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0c0003a00 0x7fe0c0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0d7e00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0c0004800 0x7fe0c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0ddc00640 1 -- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0c0003a00 msgr2=0x7fe0c0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0ddc00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0c0003a00 0x7fe0c0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0ddc00640 1 -- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0c0004800 msgr2=0x7fe0c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0ddc00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0c0004800 0x7fe0c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.739+0000 7fe0ddc00640 1 -- 172.21.15.6:0/3108524535 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c0003520 con 0x7fe0c00051c0 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.740+0000 7fe0ddc00640 1 -- 172.21.15.6:0/3108524535 reap_dead start 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.741+0000 7fe0ddc00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0c000b9c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0001e50 tx=0x7fe0c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c08bcaa531289aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.741+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0001a70 con 0x7fe0c00051c0 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.741+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d0003b90 con 0x7fe0c00051c0 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.741+0000 7fe0d7400640 1 -- 172.21.15.6:0/3108524535 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00050d0 con 0x7fe0c00051c0 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.741+0000 7fe0dfc00640 1 -- 172.21.15.6:0/3108524535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 msgr2=0x7fe0c000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.741+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0c000b9c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0001e50 tx=0x7fe0c0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.742+0000 7fe0dfc00640 1 -- 172.21.15.6:0/3108524535 shutdown_connections 2024-03-06T07:32:57.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.742+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/3108524535 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0c000b9c0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.742+0000 7fe0dfc00640 1 -- 172.21.15.6:0/3108524535 >> 172.21.15.6:0/3108524535 conn(0x7fe0d806e560 msgr2=0x7fe0d806e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.742+0000 7fe0dfc00640 1 -- 172.21.15.6:0/3108524535 shutdown_connections 2024-03-06T07:32:57.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.742+0000 7fe0dfc00640 1 -- 172.21.15.6:0/3108524535 wait complete. 2024-03-06T07:32:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.744+0000 7fe0dfc00640 1 Processor -- start 2024-03-06T07:32:57.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.745+0000 7fe0dfc00640 1 -- start start 2024-03-06T07:32:57.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.746+0000 7fe0dfc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0d81555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.746+0000 7fe0dfc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0d81e88d0 0x7fe0d81fbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.746+0000 7fe0dfc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0d814cc00 0x7fe0d806aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.746+0000 7fe0dfc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8071740 con 0x7fe0c00051c0 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dfc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d80715c0 con 0x7fe0d81e88d0 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dfc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81552b0 con 0x7fe0d814cc00 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0d81555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0d81555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56476/0 (socket says 172.21.15.6:56476) 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 -- 172.21.15.6:0/2318119096 learned_addr learned my addr 172.21.15.6:0/2318119096 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 -- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0d814cc00 msgr2=0x7fe0d806aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0ddc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0d814cc00 0x7fe0d806aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0d814cc00 0x7fe0d806aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0d7e00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0d81e88d0 0x7fe0d81fbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 -- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0d81e88d0 msgr2=0x7fe0d81fbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0d81e88d0 0x7fe0d81fbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.747+0000 7fe0dd200640 1 -- 172.21.15.6:0/2318119096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d81fc370 con 0x7fe0c00051c0 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.748+0000 7fe0dd200640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0d81555b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0118e0 tx=0x7fe0cc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.748+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00faf0 con 0x7fe0c00051c0 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.748+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00fc90 con 0x7fe0c00051c0 2024-03-06T07:32:57.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.748+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d81fc610 con 0x7fe0c00051c0 2024-03-06T07:32:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.748+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc014440 con 0x7fe0c00051c0 2024-03-06T07:32:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.748+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d81fcbd0 con 0x7fe0c00051c0 2024-03-06T07:32:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.750+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d80d0130 con 0x7fe0c00051c0 2024-03-06T07:32:57.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.750+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0cc012040 con 0x7fe0c00051c0 2024-03-06T07:32:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.752+0000 7fe0d4c00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.752+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0cc08a9a0 con 0x7fe0c00051c0 2024-03-06T07:32:57.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.753+0000 7fe0d7e00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.754+0000 7fe0d7e00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe0d81fbd90 tx=0x7fe0c8000f50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.756+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0d80d0130 con 0x7fe0c00051c0 2024-03-06T07:32:57.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.994+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0d8073fb0 con 0x7fe0c00051c0 2024-03-06T07:32:57.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.994+0000 7fe0d4c00640 1 -- 172.21.15.6:0/2318119096 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe0cc019030 con 0x7fe0c00051c0 2024-03-06T07:32:57.996 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:32:58.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe0bc06b4d0 msgr2=0x7fe0bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe0d81fbd90 tx=0x7fe0c8000f50 comp rx=0 tx=0).stop 2024-03-06T07:32:58.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 msgr2=0x7fe0d81555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0d81555b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc0118e0 tx=0x7fe0cc011da0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 shutdown_connections 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe0bc06b4d0 0x7fe0bc06d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0d81e88d0 0x7fe0d81fbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe0d814cc00 0x7fe0d806aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 --2- 172.21.15.6:0/2318119096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0c00051c0 0x7fe0d81555b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 >> 172.21.15.6:0/2318119096 conn(0x7fe0d806e560 msgr2=0x7fe0d80cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 shutdown_connections 2024-03-06T07:32:58.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:57.998+0000 7fe0dfc00640 1 -- 172.21.15.6:0/2318119096 wait complete. 2024-03-06T07:32:58.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:58.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:32:58.191 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:32:58.191 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:58.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:58.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:32:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:32:58.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.196 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:58.197 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:32:58.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:58.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:58.200 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:32:58.201 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:58.202 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:32:58.204 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:32:58.205 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:32:58.206 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:32:58.207 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:32:58.208 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:32:58.209 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:32:58.210 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:32:58.211 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:32:58.212 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_files = 6 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:32:58.213 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:32:58.246 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:58.246 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.247 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:58.247 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.247 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:32:58.247 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.248 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:32:58.248 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.266 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:32:58.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:32:58.267 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:32:58.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:32:58.268 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:58.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:32:58.269 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:58.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:32:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.503+0000 7fc0f3e00640 1 Processor -- start 2024-03-06T07:32:58.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.503+0000 7fc0f3e00640 1 -- start start 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc0ec072160 0x7fc0ec070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0ec0713f0 0x7fc0ec0af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec06e570 con 0x7fc0ec15e750 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0ec0ae2e0 con 0x7fc0ec0713f0 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f3e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0ec0ae530 con 0x7fc0ec072160 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0ec0713f0 0x7fc0ec0af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0ec0713f0 0x7fc0ec0af2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56490/0 (socket says 172.21.15.6:56490) 2024-03-06T07:32:58.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.506+0000 7fc0f1e00640 1 -- 172.21.15.6:0/2888762058 learned_addr learned my addr 172.21.15.6:0/2888762058 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.507+0000 7fc0f0a00640 1 --2- 172.21.15.6:0/2888762058 >> v2:172.21.15.178:3300/0 conn(0x7fc0ec072160 0x7fc0ec070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.507+0000 7fc0f1e00640 1 -- 172.21.15.6:0/2888762058 >> v2:172.21.15.178:3300/0 conn(0x7fc0ec072160 msgr2=0x7fc0ec070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.507+0000 7fc0f1e00640 1 --2- 172.21.15.6:0/2888762058 >> v2:172.21.15.178:3300/0 conn(0x7fc0ec072160 0x7fc0ec070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.507+0000 7fc0f1e00640 1 -- 172.21.15.6:0/2888762058 >> v1:172.21.15.6:6789/0 conn(0x7fc0ec15e750 legacy=0x7fc0ec071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.507+0000 7fc0f1e00640 1 -- 172.21.15.6:0/2888762058 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec0ae810 con 0x7fc0ec0713f0 2024-03-06T07:32:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.508+0000 7f5ca9a00640 1 Processor -- start 2024-03-06T07:32:58.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.508+0000 7fc0f1e00640 1 --2- 172.21.15.6:0/2888762058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec0713f0 0x7fc0ec0af2f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec04fcd0 tx=0x7fc0e402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91faf3471d0a08ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.509+0000 7f5ca9a00640 1 -- start start 2024-03-06T07:32:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.509+0000 7fc0ebe00640 1 -- 172.21.15.6:0/2888762058 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e402ea90 con 0x7fc0ec0713f0 2024-03-06T07:32:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.509+0000 7fc0ebe00640 1 -- 172.21.15.6:0/2888762058 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e402ec30 con 0x7fc0ec0713f0 2024-03-06T07:32:58.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.509+0000 7fc0ebe00640 1 -- 172.21.15.6:0/2888762058 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4009600 con 0x7fc0ec0713f0 2024-03-06T07:32:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 -- 172.21.15.6:0/2888762058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec0713f0 msgr2=0x7fc0ec0af2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/2888762058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec0713f0 0x7fc0ec0af2f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec04fcd0 tx=0x7fc0e402e0f0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 -- 172.21.15.6:0/2888762058 shutdown_connections 2024-03-06T07:32:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/2888762058 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec0713f0 0x7fc0ec0af2f0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/2888762058 >> v2:172.21.15.178:3300/0 conn(0x7fc0ec072160 0x7fc0ec070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 -- 172.21.15.6:0/2888762058 >> 172.21.15.6:0/2888762058 conn(0x7fc0ec0b1990 msgr2=0x7fc0ec0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 -- 172.21.15.6:0/2888762058 shutdown_connections 2024-03-06T07:32:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.510+0000 7fc0f3e00640 1 -- 172.21.15.6:0/2888762058 wait complete. 2024-03-06T07:32:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.511+0000 7f5ca9a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca4072af0 0x7f5ca41624f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.511+0000 7f5ca9a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca406caa0 con 0x7f5ca4161940 2024-03-06T07:32:58.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.511+0000 7f5ca9a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca40b3ca0 con 0x7f5ca4162a30 2024-03-06T07:32:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.511+0000 7f5ca9a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40b3f40 con 0x7f5ca4072af0 2024-03-06T07:32:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.511+0000 7f5ca2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ca4161940 0x7f5ca4161de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55254/0 (socket says 172.21.15.6:55254) 2024-03-06T07:32:58.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.511+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2056205262 learned_addr learned my addr 172.21.15.6:0/2056205262 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:32:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.512+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410761724 0 0) 0x7f5ca406caa0 con 0x7f5ca4161940 2024-03-06T07:32:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.512+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c88003520 con 0x7f5ca4161940 2024-03-06T07:32:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.512+0000 7f5ca2000640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca4072af0 0x7f5ca41624f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.512+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799181835 0 0) 0x7f5c88003520 con 0x7f5ca4161940 2024-03-06T07:32:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7fc0f3e00640 1 Processor -- start 2024-03-06T07:32:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7fc0f3e00640 1 -- start start 2024-03-06T07:32:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca4072af0 msgr2=0x7f5ca41624f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca4072af0 0x7f5ca41624f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 >> v1:172.21.15.178:6789/0 conn(0x7f5ca4162a30 legacy=0x7f5ca406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca406cd40 con 0x7f5ca4161940 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f5c94001670 con 0x7f5ca4161940 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 >> v1:172.21.15.6:6789/0 conn(0x7f5ca4161940 legacy=0x7f5ca4161de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c880039e0 0x7f5c88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c88004800 0x7f5c88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c880051c0 0x7f5c880141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40b3f40 con 0x7f5c880039e0 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88014900 con 0x7f5c88004800 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.513+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88014b00 con 0x7f5c880051c0 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c880039e0 0x7f5c88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2000640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c88004800 0x7f5c88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca3400640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c880051c0 0x7f5c880141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7fc0f3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0ec0713f0 0x7fc0ec1526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7fc0f3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0ec072160 0x7fc0ec15b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c880051c0 msgr2=0x7f5c880141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c880051c0 0x7f5c880141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c88004800 msgr2=0x7f5c88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 0x7fc0ec1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c88004800 0x7f5c88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2056205262 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca406cd40 con 0x7f5c880039e0 2024-03-06T07:32:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec0b2c90 con 0x7fc0ec15e750 2024-03-06T07:32:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec0aedd0 con 0x7fc0ec072160 2024-03-06T07:32:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec0b2e10 con 0x7fc0ec0713f0 2024-03-06T07:32:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.514+0000 7f5ca3400640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5c880051c0 0x7f5c880141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 0x7fc0ec1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 0x7fc0ec1599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56500/0 (socket says 172.21.15.6:56500) 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f1e00640 1 -- 172.21.15.6:0/51161747 learned_addr learned my addr 172.21.15.6:0/51161747 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f1400640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0ec0713f0 0x7fc0ec1526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.515+0000 7fc0f0a00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0ec072160 0x7fc0ec15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7fc0f1e00640 1 -- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0ec0713f0 msgr2=0x7fc0ec1526b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7fc0f1e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0ec0713f0 0x7fc0ec1526b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7fc0f1e00640 1 -- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0ec072160 msgr2=0x7fc0ec15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7fc0f1e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0ec072160 0x7fc0ec15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7fc0f1e00640 1 -- 172.21.15.6:0/51161747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec1591d0 con 0x7fc0ec15e750 2024-03-06T07:32:58.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c880039e0 0x7f5c88003dc0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400c210 tx=0x7f5c88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df200d0be5f45ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c94007d30 con 0x7f5c880039e0 2024-03-06T07:32:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c94030070 con 0x7f5c880039e0 2024-03-06T07:32:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7f5ca1600640 1 -- 172.21.15.6:0/2056205262 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9402cde0 con 0x7f5c880039e0 2024-03-06T07:32:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.516+0000 7fc0f1e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 0x7fc0ec1599b0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4038040 tx=0x7fc0e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e402ea90 con 0x7fc0ec15e750 2024-03-06T07:32:58.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec155ea0 con 0x7fc0ec15e750 2024-03-06T07:32:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2056205262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c880039e0 msgr2=0x7f5c88003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2056205262 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5c880039e0 0x7f5c88003dc0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400c210 tx=0x7f5c88005830 comp rx=0 tx=0).stop 2024-03-06T07:32:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca3400640 1 -- 172.21.15.6:0/2056205262 reap_dead start 2024-03-06T07:32:58.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec156460 con 0x7fc0ec15e750 2024-03-06T07:32:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e4045070 con 0x7fc0ec15e750 2024-03-06T07:32:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2056205262 shutdown_connections 2024-03-06T07:32:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2056205262 >> 172.21.15.6:0/2056205262 conn(0x7f5ca40b1990 msgr2=0x7f5ca40b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2056205262 shutdown_connections 2024-03-06T07:32:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.517+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2056205262 wait complete. 2024-03-06T07:32:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.518+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e400b720 con 0x7fc0ec15e750 2024-03-06T07:32:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.519+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ec09e0f0 con 0x7fc0ec15e750 2024-03-06T07:32:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.519+0000 7f5ca9a00640 1 Processor -- start 2024-03-06T07:32:58.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.519+0000 7f5ca9a00640 1 -- start start 2024-03-06T07:32:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0e4036800 con 0x7fc0ec15e750 2024-03-06T07:32:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7f5ca9a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 0x7f5ca415c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7f5ca9a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca40b1400 0x7f5ca4151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7f5ca9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca415de70 0x7f5ca415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7f5ca9a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca41624f0 con 0x7f5ca415de70 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7f5ca9a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40b4840 con 0x7f5c880039e0 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.520+0000 7f5ca9a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4162670 con 0x7f5ca40b1400 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.521+0000 7f5ca2a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 0x7f5ca415c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.521+0000 7f5ca2a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 0x7f5ca415c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:52500/0 (socket says 172.21.15.6:52500) 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.521+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2487973372 learned_addr learned my addr 172.21.15.6:0/2487973372 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.521+0000 7fc0e9600640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0cc06b4d0 0x7fc0cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0e40ad170 con 0x7fc0ec15e750 2024-03-06T07:32:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca40b1400 msgr2=0x7f5ca4151260 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:32:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca40b1400 0x7f5ca4151260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2487973372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca415de70 msgr2=0x7f5ca415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca415de70 0x7f5ca415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7f5ca2a00640 1 -- 172.21.15.6:0/2487973372 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca415c9f0 con 0x7f5c880039e0 2024-03-06T07:32:58.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7fc0f1400640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0cc06b4d0 0x7fc0cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.522+0000 7f5ca2a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 0x7f5ca415c3c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400c330 tx=0x7f5c9402e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.523+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c94045070 con 0x7f5c880039e0 2024-03-06T07:32:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.523+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c94004bf0 con 0x7f5c880039e0 2024-03-06T07:32:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.523+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c94007d10 con 0x7f5c880039e0 2024-03-06T07:32:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.523+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca40b03a0 con 0x7f5c880039e0 2024-03-06T07:32:58.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.524+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca415e830 con 0x7f5c880039e0 2024-03-06T07:32:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.524+0000 7fc0f1400640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0cc06b4d0 0x7fc0cc06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00016e0 tx=0x7fc0e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.525+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca409e0f0 con 0x7f5c880039e0 2024-03-06T07:32:58.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.526+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c94007070 con 0x7f5c880039e0 2024-03-06T07:32:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.527+0000 7f5c9aa00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c8406b4d0 0x7f5c8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.527+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c940abb60 con 0x7f5c880039e0 2024-03-06T07:32:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.528+0000 7f5ca2000640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c8406b4d0 0x7f5c8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.530+0000 7f5ca2000640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c8406b4d0 0x7f5c8406d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40cee40 tx=0x7f5c90008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.530+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc0e407bad0 con 0x7fc0ec15e750 2024-03-06T07:32:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.544+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c9407a520 con 0x7f5c880039e0 2024-03-06T07:32:58.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.552+0000 7ff7c4800640 1 Processor -- start 2024-03-06T07:32:58.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.553+0000 7ff7c4800640 1 -- start start 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.554+0000 7ff7c4800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7bc161940 0x7ff7bc072af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.554+0000 7ff7c4800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc06caa0 con 0x7ff7bc161de0 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.554+0000 7ff7c4800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7bc0b3ca0 con 0x7ff7bc162950 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.554+0000 7ff7c4800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7bc0b3f40 con 0x7ff7bc161940 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.555+0000 7ff7c1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7bc161940 0x7ff7bc072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.555+0000 7ff7c1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7bc161940 0x7ff7bc072af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56522/0 (socket says 172.21.15.6:56522) 2024-03-06T07:32:58.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.555+0000 7ff7c1e00640 1 -- 172.21.15.6:0/2737201743 learned_addr learned my addr 172.21.15.6:0/2737201743 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.555+0000 7ff7c1e00640 1 -- 172.21.15.6:0/2737201743 >> v1:172.21.15.178:6789/0 conn(0x7ff7bc162950 legacy=0x7ff7bc06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.555+0000 7ff7c1e00640 1 -- 172.21.15.6:0/2737201743 >> v1:172.21.15.6:6789/0 conn(0x7ff7bc161de0 legacy=0x7ff7bc162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.555+0000 7ff7c1e00640 1 -- 172.21.15.6:0/2737201743 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc06cd40 con 0x7ff7bc161940 2024-03-06T07:32:58.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.557+0000 7ff7c1e00640 1 --2- 172.21.15.6:0/2737201743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161940 0x7ff7bc072af0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009870 tx=0x7ff7b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2770480fdeb1ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.557+0000 7ff7c0a00640 1 -- 172.21.15.6:0/2737201743 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0035020 con 0x7ff7bc161940 2024-03-06T07:32:58.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.557+0000 7ff7c0a00640 1 -- 172.21.15.6:0/2737201743 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b0031870 con 0x7ff7bc161940 2024-03-06T07:32:58.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.557+0000 7ff7c0a00640 1 -- 172.21.15.6:0/2737201743 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0031c40 con 0x7ff7bc161940 2024-03-06T07:32:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.557+0000 7f50ed200640 1 Processor -- start 2024-03-06T07:32:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.557+0000 7f50ed200640 1 -- start start 2024-03-06T07:32:58.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 -- 172.21.15.6:0/2737201743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161940 msgr2=0x7ff7bc072af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 --2- 172.21.15.6:0/2737201743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161940 0x7ff7bc072af0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009870 tx=0x7ff7b002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 -- 172.21.15.6:0/2737201743 shutdown_connections 2024-03-06T07:32:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 --2- 172.21.15.6:0/2737201743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161940 0x7ff7bc072af0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 -- 172.21.15.6:0/2737201743 >> 172.21.15.6:0/2737201743 conn(0x7ff7bc0b1990 msgr2=0x7ff7bc0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 -- 172.21.15.6:0/2737201743 shutdown_connections 2024-03-06T07:32:58.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.558+0000 7ff7c4800640 1 -- 172.21.15.6:0/2737201743 wait complete. 2024-03-06T07:32:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50ed200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50e815f3e0 0x7f50e80cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50ed200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f50e806afc0 0x7f50e806b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50ed200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e80b3fb0 con 0x7f50e81479e0 2024-03-06T07:32:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50ed200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e80725c0 con 0x7f50e815f3e0 2024-03-06T07:32:58.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50ed200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e80728a0 con 0x7f50e806afc0 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50e5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50e815f3e0 0x7f50e80cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50e5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50e815f3e0 0x7f50e80cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56532/0 (socket says 172.21.15.6:56532) 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.559+0000 7f50e5400640 1 -- 172.21.15.6:0/3354017868 learned_addr learned my addr 172.21.15.6:0/3354017868 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7f50e6800640 1 --2- 172.21.15.6:0/3354017868 >> v2:172.21.15.178:3300/0 conn(0x7f50e806afc0 0x7f50e806b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7f50e5400640 1 -- 172.21.15.6:0/3354017868 >> v2:172.21.15.178:3300/0 conn(0x7f50e806afc0 msgr2=0x7f50e806b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7f50e5400640 1 --2- 172.21.15.6:0/3354017868 >> v2:172.21.15.178:3300/0 conn(0x7f50e806afc0 0x7f50e806b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7f50e5400640 1 -- 172.21.15.6:0/3354017868 >> v1:172.21.15.178:6789/0 conn(0x7f50e81479e0 legacy=0x7f50e8071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:58.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7f50e5400640 1 -- 172.21.15.6:0/3354017868 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8072b80 con 0x7f50e815f3e0 2024-03-06T07:32:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7ff7c4800640 1 Processor -- start 2024-03-06T07:32:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.560+0000 7ff7c4800640 1 -- start start 2024-03-06T07:32:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7ff7c4800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7bc161940 0x7ff7bc15e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7f50e5400640 1 --2- 172.21.15.6:0/3354017868 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e815f3e0 0x7f50e80cee00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f50e8073380 tx=0x7f50d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb7f167c3b46e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7ff7c4800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 0x7ff7bc13e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7ff7c4800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7bc162950 0x7ff7bc1e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7ff7c4800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc06d970 con 0x7ff7bc161de0 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7ff7c4800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc0b4840 con 0x7ff7bc161940 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.561+0000 7ff7c4800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc06daf0 con 0x7ff7bc162950 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 0x7ff7bc13e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7f50e4a00640 1 -- 172.21.15.6:0/3354017868 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4034070 con 0x7f50e815f3e0 2024-03-06T07:32:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 0x7ff7bc13e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56536/0 (socket says 172.21.15.6:56536) 2024-03-06T07:32:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 -- 172.21.15.6:0/3576645980 learned_addr learned my addr 172.21.15.6:0/3576645980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7f50e4a00640 1 -- 172.21.15.6:0/3354017868 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d4030860 con 0x7f50e815f3e0 2024-03-06T07:32:58.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7f50e4a00640 1 -- 172.21.15.6:0/3354017868 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4030c30 con 0x7f50e815f3e0 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c2800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7bc162950 0x7ff7bc1e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1e00640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7bc161940 0x7ff7bc15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 -- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7bc162950 msgr2=0x7ff7bc1e7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7bc162950 0x7ff7bc1e7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 -- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7bc161940 msgr2=0x7ff7bc15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7bc161940 0x7ff7bc15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1400640 1 -- 172.21.15.6:0/3576645980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc13ec40 con 0x7ff7bc161de0 2024-03-06T07:32:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c2800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7bc162950 0x7ff7bc1e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.562+0000 7ff7c1e00640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7bc161940 0x7ff7bc15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7ff7c1400640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 0x7ff7bc13e250 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00c440 tx=0x7ff7ac001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 -- 172.21.15.6:0/3354017868 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e815f3e0 msgr2=0x7f50e80cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 --2- 172.21.15.6:0/3354017868 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e815f3e0 0x7f50e80cee00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f50e8073380 tx=0x7f50d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 -- 172.21.15.6:0/3354017868 shutdown_connections 2024-03-06T07:32:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 --2- 172.21.15.6:0/3354017868 >> v2:172.21.15.178:3300/0 conn(0x7f50e806afc0 0x7f50e806b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 --2- 172.21.15.6:0/3354017868 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e815f3e0 0x7f50e80cee00 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 -- 172.21.15.6:0/3354017868 >> 172.21.15.6:0/3354017868 conn(0x7f50e806cb20 msgr2=0x7f50e80b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 -- 172.21.15.6:0/3354017868 shutdown_connections 2024-03-06T07:32:58.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac0052d0 con 0x7ff7bc161de0 2024-03-06T07:32:58.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc13ef20 con 0x7ff7bc161de0 2024-03-06T07:32:58.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc13f7c0 con 0x7ff7bc161de0 2024-03-06T07:32:58.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac005470 con 0x7ff7bc161de0 2024-03-06T07:32:58.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac007220 con 0x7ff7bc161de0 2024-03-06T07:32:58.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.563+0000 7f50ed200640 1 -- 172.21.15.6:0/3354017868 wait complete. 2024-03-06T07:32:58.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.565+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7bc09e0f0 con 0x7ff7bc161de0 2024-03-06T07:32:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.565+0000 7f50ed200640 1 Processor -- start 2024-03-06T07:32:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.566+0000 7f50ed200640 1 -- start start 2024-03-06T07:32:58.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.566+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7ac007480 con 0x7ff7bc161de0 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.566+0000 7f50ed200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e806afc0 0x7f50e8146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.566+0000 7f50ed200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 0x7f50e80b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50ed200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e815f3e0 0x7f50e8160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50ed200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8147f40 con 0x7f50e81479e0 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50ed200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8147dc0 con 0x7f50e815f3e0 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50ed200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e80b44b0 con 0x7f50e806afc0 2024-03-06T07:32:58.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7ff7b6000640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a006b4d0 0x7ff7a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50e5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 0x7f50e80b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7ac089760 con 0x7ff7bc161de0 2024-03-06T07:32:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50e5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 0x7f50e80b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56548/0 (socket says 172.21.15.6:56548) 2024-03-06T07:32:58.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50e5400640 1 -- 172.21.15.6:0/1092642201 learned_addr learned my addr 172.21.15.6:0/1092642201 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:32:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7ff7c1e00640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a006b4d0 0x7ff7a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.567+0000 7f50e5e00640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e806afc0 0x7f50e8146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e6800640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e815f3e0 0x7f50e8160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e5400640 1 -- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e806afc0 msgr2=0x7f50e8146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e5400640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e806afc0 0x7f50e8146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e5400640 1 -- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e815f3e0 msgr2=0x7f50e8160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e5400640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e815f3e0 0x7f50e8160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e5400640 1 -- 172.21.15.6:0/1092642201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e81500b0 con 0x7f50e81479e0 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.568+0000 7f50e5400640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 0x7f50e80b0d20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f50d4000c00 tx=0x7f50d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.569+0000 7ff7c1e00640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a006b4d0 0x7ff7a006d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0036000 tx=0x7ff7b003c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.573+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4034040 con 0x7f50e81479e0 2024-03-06T07:32:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.573+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d4039d90 con 0x7f50e81479e0 2024-03-06T07:32:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.573+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4043630 con 0x7f50e81479e0 2024-03-06T07:32:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.573+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7ac0584c0 con 0x7ff7bc161de0 2024-03-06T07:32:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.573+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e8150390 con 0x7f50e81479e0 2024-03-06T07:32:58.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.573+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e8150950 con 0x7f50e81479e0 2024-03-06T07:32:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.574+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e8162bc0 con 0x7f50e81479e0 2024-03-06T07:32:58.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.576+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50d4037070 con 0x7f50e81479e0 2024-03-06T07:32:58.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.577+0000 7f50de000640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c806b520 0x7f50c806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:58.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.577+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50d40ac310 con 0x7f50e81479e0 2024-03-06T07:32:58.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.578+0000 7f50e5e00640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c806b520 0x7f50c806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:58.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.579+0000 7f50e5e00640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c806b520 0x7f50c806d9e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f50d8004f60 tx=0x7f50d8007e30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:58.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.582+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50e8162bc0 con 0x7f50e81479e0 2024-03-06T07:32:58.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.852+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7ff7bc04f410 con 0x7ff7bc161de0 2024-03-06T07:32:58.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.862+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f50e80b3dd0 con 0x7f50e81479e0 2024-03-06T07:32:58.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.880+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fc0ec04f410 con 0x7fc0ec15e750 2024-03-06T07:32:58.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:58.898+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f5ca404f410 con 0x7f5c880039e0 2024-03-06T07:32:59.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.631+0000 7ff7b6000640 1 -- 172.21.15.6:0/3576645980 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7155 v62) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff7ac057dc0 con 0x7ff7bc161de0 2024-03-06T07:32:59.633 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 7155 2024-03-06T07:32:59.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.634+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a006b4d0 msgr2=0x7ff7a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.634+0000 7ff7c4800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a006b4d0 0x7ff7a006d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0036000 tx=0x7ff7b003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:59.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.634+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 msgr2=0x7ff7bc13e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.634+0000 7ff7c4800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 0x7ff7bc13e250 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac00c440 tx=0x7ff7ac001e70 comp rx=0 tx=0).stop 2024-03-06T07:32:59.637 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:32:59.635+0000 7f723ec00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 shutdown_connections 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a006b4d0 0x7ff7a006d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7bc162950 0x7ff7bc1e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7bc161de0 0x7ff7bc13e250 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 --2- 172.21.15.6:0/3576645980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7bc161940 0x7ff7bc15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 >> 172.21.15.6:0/3576645980 conn(0x7ff7bc0b1990 msgr2=0x7ff7bc070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 shutdown_connections 2024-03-06T07:32:59.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.635+0000 7ff7c4800640 1 -- 172.21.15.6:0/3576645980 wait complete. 2024-03-06T07:32:59.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.637+0000 7f50de000640 1 -- 172.21.15.6:0/1092642201 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7167 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f50e80b3dd0 con 0x7f50e81479e0 2024-03-06T07:32:59.640 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 7167 2024-03-06T07:32:59.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.642+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c806b520 msgr2=0x7f50c806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.642+0000 7f50ed200640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c806b520 0x7f50c806d9e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f50d8004f60 tx=0x7f50d8007e30 comp rx=0 tx=0).stop 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.642+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 msgr2=0x7f50e80b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.642+0000 7f50ed200640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 0x7f50e80b0d20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f50d4000c00 tx=0x7f50d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 shutdown_connections 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c806b520 0x7f50c806d9e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e815f3e0 0x7f50e8160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e81479e0 0x7f50e80b0d20 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 --2- 172.21.15.6:0/1092642201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e806afc0 0x7f50e8146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 >> 172.21.15.6:0/1092642201 conn(0x7f50e806cb20 msgr2=0x7f50e806fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.643+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 shutdown_connections 2024-03-06T07:32:59.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.644+0000 7f50ed200640 1 -- 172.21.15.6:0/1092642201 wait complete. 2024-03-06T07:32:59.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.646+0000 7fc0e9600640 1 -- 172.21.15.6:0/51161747 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5546 v64) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc0e407fb70 con 0x7fc0ec15e750 2024-03-06T07:32:59.648 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 5546 2024-03-06T07:32:59.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.651+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0cc06b4d0 msgr2=0x7fc0cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.651+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0cc06b4d0 0x7fc0cc06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00016e0 tx=0x7fc0e0009040 comp rx=0 tx=0).stop 2024-03-06T07:32:59.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.651+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 msgr2=0x7fc0ec1599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:59.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.651+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 0x7fc0ec1599b0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4038040 tx=0x7fc0e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:59.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.652+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 shutdown_connections 2024-03-06T07:32:59.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.652+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc0cc06b4d0 0x7fc0cc06d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.652+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0ec072160 0x7fc0ec15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.652+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0ec15e750 0x7fc0ec1599b0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.652+0000 7fc0f3e00640 1 --2- 172.21.15.6:0/51161747 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc0ec0713f0 0x7fc0ec1526b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.652+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 >> 172.21.15.6:0/51161747 conn(0x7fc0ec0b1990 msgr2=0x7fc0ec15f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.653+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 shutdown_connections 2024-03-06T07:32:59.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:32:59.653+0000 7fc0f3e00640 1 -- 172.21.15.6:0/51161747 wait complete. 2024-03-06T07:32:59.664 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:32:59.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:32:59.671 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:32:59.672 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:59.672 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:32:59.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:32:59.676 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:32:59.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:32:59.684 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:32:59.684 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:59.720 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:32:59.720 DEBUG:teuthology.parallel:result is None 2024-03-06T07:32:59.831 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:32:59.860 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:32:59.886 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:33:00.632 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:33:00.628+0000 7f723ec00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.639+0000 7f5c9aa00640 1 -- 172.21.15.6:0/2487973372 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6169 v65) v1 ==== 95+0+0 (secure 0 0 0) 0x7f5c9407e5c0 con 0x7f5c880039e0 2024-03-06T07:33:00.641 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6169 2024-03-06T07:33:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.645+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c8406b4d0 msgr2=0x7f5c8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.645+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c8406b4d0 0x7f5c8406d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5ca40cee40 tx=0x7f5c90008040 comp rx=0 tx=0).stop 2024-03-06T07:33:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.645+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 msgr2=0x7f5ca415c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.645+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 0x7f5ca415c3c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400c330 tx=0x7f5c9402e070 comp rx=0 tx=0).stop 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.646+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 shutdown_connections 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.646+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5c8406b4d0 0x7f5c8406d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.646+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ca415de70 0x7f5ca415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.646+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5ca40b1400 0x7f5ca4151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.646+0000 7f5ca9a00640 1 --2- 172.21.15.6:0/2487973372 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5c880039e0 0x7f5ca415c3c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.646+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 >> 172.21.15.6:0/2487973372 conn(0x7f5ca40b1990 msgr2=0x7f5ca4070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.647+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 shutdown_connections 2024-03-06T07:33:00.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.647+0000 7f5ca9a00640 1 -- 172.21.15.6:0/2487973372 wait complete. 2024-03-06T07:33:00.672 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:33:00.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:33:00.678 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:33:00.678 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:00.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:00.840 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:33:00.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.919+0000 7f3d92c00640 1 Processor -- start 2024-03-06T07:33:00.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.919+0000 7f3d92c00640 1 -- start start 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.920+0000 7f3d92c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d8c15ffa0 0x7f3d8c162b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.920+0000 7f3d92c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3d8c0b4b20 0x7f3d8c0b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d92c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0b3f40 con 0x7f3d8c160380 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d92c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c070e20 con 0x7f3d8c15ffa0 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d92c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c0b36a0 con 0x7f3d8c0b4b20 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d8c15ffa0 0x7f3d8c162b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3d8c15ffa0 0x7f3d8c162b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56588/0 (socket says 172.21.15.6:56588) 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 -- 172.21.15.6:0/891847304 learned_addr learned my addr 172.21.15.6:0/891847304 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8b400640 1 --1- 172.21.15.6:0/891847304 >> v1:172.21.15.6:6789/0 conn(0x7f3d8c160380 0x7f3d8c0b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55284/0 (socket says 172.21.15.6:55284) 2024-03-06T07:33:00.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d90c00640 1 --2- 172.21.15.6:0/891847304 >> v2:172.21.15.178:3300/0 conn(0x7f3d8c0b4b20 0x7f3d8c0b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 -- 172.21.15.6:0/891847304 >> v2:172.21.15.178:3300/0 conn(0x7f3d8c0b4b20 msgr2=0x7f3d8c0b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:00.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 --2- 172.21.15.6:0/891847304 >> v2:172.21.15.178:3300/0 conn(0x7f3d8c0b4b20 0x7f3d8c0b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 -- 172.21.15.6:0/891847304 >> v1:172.21.15.6:6789/0 conn(0x7f3d8c160380 legacy=0x7f3d8c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.921+0000 7f3d8be00640 1 -- 172.21.15.6:0/891847304 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c0b3980 con 0x7f3d8c15ffa0 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.923+0000 7f3d8be00640 1 --2- 172.21.15.6:0/891847304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c162b50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c072e50 tx=0x7f3d7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa02a87a8104047 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.923+0000 7f3d8aa00640 1 -- 172.21.15.6:0/891847304 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c034070 con 0x7f3d8c15ffa0 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.923+0000 7f3d8aa00640 1 -- 172.21.15.6:0/891847304 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c030860 con 0x7f3d8c15ffa0 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.923+0000 7f3d8aa00640 1 -- 172.21.15.6:0/891847304 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c030c30 con 0x7f3d8c15ffa0 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.923+0000 7f3d92c00640 1 -- 172.21.15.6:0/891847304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c15ffa0 msgr2=0x7f3d8c162b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.923+0000 7f3d92c00640 1 --2- 172.21.15.6:0/891847304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c162b50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c072e50 tx=0x7f3d7c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.924+0000 7f3d92c00640 1 -- 172.21.15.6:0/891847304 shutdown_connections 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.924+0000 7f3d92c00640 1 --2- 172.21.15.6:0/891847304 >> v2:172.21.15.178:3300/0 conn(0x7f3d8c0b4b20 0x7f3d8c0b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.924+0000 7f3d92c00640 1 --2- 172.21.15.6:0/891847304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c162b50 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.924+0000 7f3d92c00640 1 -- 172.21.15.6:0/891847304 >> 172.21.15.6:0/891847304 conn(0x7f3d8c06cf40 msgr2=0x7f3d8c06d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.924+0000 7f3d92c00640 1 -- 172.21.15.6:0/891847304 shutdown_connections 2024-03-06T07:33:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.924+0000 7f3d92c00640 1 -- 172.21.15.6:0/891847304 wait complete. 2024-03-06T07:33:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.926+0000 7f3d92c00640 1 Processor -- start 2024-03-06T07:33:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.926+0000 7f3d92c00640 1 -- start start 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d92c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d8c0b4b20 0x7f3d8c1596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d92c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d92c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 0x7f3d8c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d92c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c071430 con 0x7f3d8c147e00 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d92c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0712b0 con 0x7f3d8c15ffa0 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d92c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0715b0 con 0x7f3d8c0b4b20 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 0x7f3d8c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 0x7f3d8c15ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:56594/0 (socket says 172.21.15.6:56594) 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 -- 172.21.15.6:0/3189295505 learned_addr learned my addr 172.21.15.6:0/3189295505 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d8be00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d8c0b4b20 0x7f3d8c1596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d8b400640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 -- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d8c0b4b20 msgr2=0x7f3d8c1596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d8c0b4b20 0x7f3d8c1596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 -- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c15ffa0 msgr2=0x7f3d8c158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d90c00640 1 -- 172.21.15.6:0/3189295505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c159c80 con 0x7f3d8c147e00 2024-03-06T07:33:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.927+0000 7f3d8b400640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.928+0000 7f3d90c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 0x7f3d8c15ad20 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3d84007e40 tx=0x7f3d8400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.928+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84014620 con 0x7f3d8c147e00 2024-03-06T07:33:00.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.928+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d84019070 con 0x7f3d8c147e00 2024-03-06T07:33:00.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.928+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84014a20 con 0x7f3d8c147e00 2024-03-06T07:33:00.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.928+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c159f60 con 0x7f3d8c147e00 2024-03-06T07:33:00.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.928+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c15a520 con 0x7f3d8c147e00 2024-03-06T07:33:00.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.930+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c0692a0 con 0x7f3d8c147e00 2024-03-06T07:33:00.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.930+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d84014bc0 con 0x7f3d8c147e00 2024-03-06T07:33:00.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.931+0000 7f3d83e00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:00.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.931+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f3d8408ab80 con 0x7f3d8c147e00 2024-03-06T07:33:00.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.932+0000 7f3d8be00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:00.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.933+0000 7f3d8be00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c036000 tx=0x7f3d7c03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:00.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:00.936+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d84059950 con 0x7f3d8c147e00 2024-03-06T07:33:01.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.173+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8c163c00 con 0x7f3d8c147e00 2024-03-06T07:33:01.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.174+0000 7f3d83e00640 1 -- 172.21.15.6:0/3189295505 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4176 (secure 0 0 0) 0x7f3d84059250 con 0x7f3d8c147e00 2024-03-06T07:33:01.176 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:01.176 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":66,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:00.638471+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:33:01.177 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 66 2024-03-06T07:33:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.176+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d6c06b4d0 msgr2=0x7f3d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.176+0000 7f3d92c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c036000 tx=0x7f3d7c03b040 comp rx=0 tx=0).stop 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.176+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 msgr2=0x7f3d8c15ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.176+0000 7f3d92c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 0x7f3d8c15ad20 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f3d84007e40 tx=0x7f3d8400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 shutdown_connections 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3d8c15ffa0 0x7f3d8c158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3d8c147e00 0x7f3d8c15ad20 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 --2- 172.21.15.6:0/3189295505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3d8c0b4b20 0x7f3d8c1596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 >> 172.21.15.6:0/3189295505 conn(0x7f3d8c06cf40 msgr2=0x7f3d8c164400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 shutdown_connections 2024-03-06T07:33:01.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:01.177+0000 7f3d92c00640 1 -- 172.21.15.6:0/3189295505 wait complete. 2024-03-06T07:33:01.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:47.892948+0000', 'modified': '2024-03-06T07:33:00.638471+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7362}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7362': {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.6:6835/619677659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 619677659}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 619677659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:33:01.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7362:up:replay 2024-03-06T07:33:02.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:02.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.828+0000 7f17f5000640 1 Processor -- start 2024-03-06T07:33:02.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.828+0000 7f17f5000640 1 -- start start 2024-03-06T07:33:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.830+0000 7f17f5000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f0144820 0x7f17f00727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.830+0000 7f17f5000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f17f00b45b0 0x7f17f0070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.830+0000 7f17f5000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f00ae5c0 con 0x7f17f015f220 2024-03-06T07:33:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.830+0000 7f17f5000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f00af190 con 0x7f17f00b45b0 2024-03-06T07:33:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.830+0000 7f17f5000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00ae860 con 0x7f17f0144820 2024-03-06T07:33:02.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.831+0000 7f17efe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f17f00b45b0 0x7f17f0070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.831+0000 7f17ef400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f0144820 0x7f17f00727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.831+0000 7f17efe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f17f00b45b0 0x7f17f0070bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:42314/0 (socket says 172.21.15.6:42314) 2024-03-06T07:33:02.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.831+0000 7f17efe00640 1 -- 172.21.15.6:0/1708551714 learned_addr learned my addr 172.21.15.6:0/1708551714 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:02.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.832+0000 7f17efe00640 1 -- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f0144820 msgr2=0x7f17f00727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:02.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.832+0000 7f17efe00640 1 --2- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f0144820 0x7f17f00727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.832+0000 7f17efe00640 1 -- 172.21.15.6:0/1708551714 >> v1:172.21.15.178:6789/0 conn(0x7f17f015f220 legacy=0x7f17f00b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.832+0000 7f17efe00640 1 -- 172.21.15.6:0/1708551714 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f00aeb40 con 0x7f17f00b45b0 2024-03-06T07:33:02.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.832+0000 7f17ef400640 1 --2- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f0144820 0x7f17f00727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.833+0000 7f17efe00640 1 --2- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f00b45b0 0x7f17f0070bf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009e80 tx=0x7f17e802bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=face48b90edb83e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.833+0000 7f17ee000640 1 -- 172.21.15.6:0/1708551714 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e802cae0 con 0x7f17f00b45b0 2024-03-06T07:33:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.833+0000 7f17ee000640 1 -- 172.21.15.6:0/1708551714 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e802cc80 con 0x7f17f00b45b0 2024-03-06T07:33:02.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.833+0000 7f17ee000640 1 -- 172.21.15.6:0/1708551714 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8034610 con 0x7f17f00b45b0 2024-03-06T07:33:02.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 -- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f00b45b0 msgr2=0x7f17f0070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 --2- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f00b45b0 0x7f17f0070bf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009e80 tx=0x7f17e802bf40 comp rx=0 tx=0).stop 2024-03-06T07:33:02.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 -- 172.21.15.6:0/1708551714 shutdown_connections 2024-03-06T07:33:02.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 --2- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f00b45b0 0x7f17f0070bf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 --2- 172.21.15.6:0/1708551714 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f0144820 0x7f17f00727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 -- 172.21.15.6:0/1708551714 >> 172.21.15.6:0/1708551714 conn(0x7f17f00b1990 msgr2=0x7f17f00b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:02.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 -- 172.21.15.6:0/1708551714 shutdown_connections 2024-03-06T07:33:02.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.834+0000 7f17f5000640 1 -- 172.21.15.6:0/1708551714 wait complete. 2024-03-06T07:33:02.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.836+0000 7f17f5000640 1 Processor -- start 2024-03-06T07:33:02.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.836+0000 7f17f5000640 1 -- start start 2024-03-06T07:33:02.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.837+0000 7f17f5000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f00b45b0 0x7f17f01556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.837+0000 7f17f5000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f0144820 0x7f17f01675a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.837+0000 7f17f5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 0x7f17f0164620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.837+0000 7f17f5000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f00af620 con 0x7f17f015f220 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.837+0000 7f17f5000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f015f680 con 0x7f17f0144820 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.837+0000 7f17f5000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0071130 con 0x7f17f00b45b0 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 0x7f17f0164620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 0x7f17f0164620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44652/0 (socket says 172.21.15.6:44652) 2024-03-06T07:33:02.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 -- 172.21.15.6:0/1263237815 learned_addr learned my addr 172.21.15.6:0/1263237815 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17ef400640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f00b45b0 0x7f17f01556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17eea00640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f0144820 0x7f17f01675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 -- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f00b45b0 msgr2=0x7f17f01556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f00b45b0 0x7f17f01556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 -- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f0144820 msgr2=0x7f17f01675a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f0144820 0x7f17f01675a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 -- 172.21.15.6:0/1263237815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0168060 con 0x7f17f015f220 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.838+0000 7f17efe00640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 0x7f17f0164620 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f17e8036040 tx=0x7f17e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.839+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8043020 con 0x7f17f015f220 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.839+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e803e450 con 0x7f17f015f220 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.839+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f0168340 con 0x7f17f015f220 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.839+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8009510 con 0x7f17f015f220 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.839+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f0164100 con 0x7f17f015f220 2024-03-06T07:33:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.840+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f0163c10 con 0x7f17f015f220 2024-03-06T07:33:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.841+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17e8046020 con 0x7f17f015f220 2024-03-06T07:33:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.842+0000 7f17e7400640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17d006b4d0 0x7f17d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.842+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f17e80acbf0 con 0x7f17f015f220 2024-03-06T07:33:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.843+0000 7f17ef400640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17d006b4d0 0x7f17d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.844+0000 7f17ef400640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17d006b4d0 0x7f17d006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009560 tx=0x7f17e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:02.847+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17e804d030 con 0x7f17f015f220 2024-03-06T07:33:03.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.085+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17f004f330 con 0x7f17f015f220 2024-03-06T07:33:03.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.086+0000 7f17e7400640 1 -- 172.21.15.6:0/1263237815 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4967 (secure 0 0 0) 0x7f17e807ba40 con 0x7f17f015f220 2024-03-06T07:33:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:03.088 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":68,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:01.668068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:33:03.089 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 68 2024-03-06T07:33:03.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17d006b4d0 msgr2=0x7f17d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17d006b4d0 0x7f17d006d990 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009560 tx=0x7f17e0008040 comp rx=0 tx=0).stop 2024-03-06T07:33:03.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 msgr2=0x7f17f0164620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 0x7f17f0164620 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f17e8036040 tx=0x7f17e8004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 shutdown_connections 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f17d006b4d0 0x7f17d006d990 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17f0144820 0x7f17f01675a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17f015f220 0x7f17f0164620 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 --2- 172.21.15.6:0/1263237815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17f00b45b0 0x7f17f01556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 >> 172.21.15.6:0/1263237815 conn(0x7f17f00b1990 msgr2=0x7f17f006d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 shutdown_connections 2024-03-06T07:33:03.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:03.089+0000 7f17f5000640 1 -- 172.21.15.6:0/1263237815 wait complete. 2024-03-06T07:33:03.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:47.892948+0000', 'modified': '2024-03-06T07:33:01.668068+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7362}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7362': {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.6:6835/619677659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 619677659}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 619677659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:33:03.514 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7362:up:rejoin 2024-03-06T07:33:04.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:04.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.736+0000 7f0f43a00640 1 Processor -- start 2024-03-06T07:33:04.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.736+0000 7f0f43a00640 1 -- start start 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.737+0000 7f0f43a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 0x7f0f3c140790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.737+0000 7f0f43a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0f3c071d00 0x7f0f3c0b5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.737+0000 7f0f43a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c072290 con 0x7f0f3c0b53d0 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.737+0000 7f0f43a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3c0b5d50 con 0x7f0f3c071d00 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.737+0000 7f0f43a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c140cd0 con 0x7f0f3c0cedb0 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f3be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0f3c0b53d0 0x7f0f3c15e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57152/0 (socket says 172.21.15.6:57152) 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f3be00640 1 -- 172.21.15.6:0/417182155 learned_addr learned my addr 172.21.15.6:0/417182155 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f41a00640 1 --2- 172.21.15.6:0/417182155 >> v2:172.21.15.178:3300/0 conn(0x7f0f3c071d00 0x7f0f3c0b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672159955 0 0) 0x7f0f3c072290 con 0x7f0f3c0b53d0 2024-03-06T07:33:04.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f41000640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 0x7f0f3c140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f24003520 con 0x7f0f3c0b53d0 2024-03-06T07:33:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3073127111 0 0) 0x7f0f24003520 con 0x7f0f3c0b53d0 2024-03-06T07:33:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.738+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 msgr2=0x7f0f3c140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 0x7f0f3c140790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 >> v2:172.21.15.178:3300/0 conn(0x7f0f3c071d00 msgr2=0x7f0f3c0b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 --2- 172.21.15.6:0/417182155 >> v2:172.21.15.178:3300/0 conn(0x7f0f3c071d00 0x7f0f3c0b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c072570 con 0x7f0f3c0b53d0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0f2c001670 con 0x7f0f3c0b53d0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 >> v1:172.21.15.6:6789/0 conn(0x7f0f3c0b53d0 legacy=0x7f0f3c15e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f240039b0 0x7f0f24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f24004800 0x7f0f24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 0x7f0f240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c0b5d50 con 0x7f0f240039b0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c140cd0 con 0x7f0f24004800 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2400b9c0 con 0x7f0f3c0cedb0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f41000640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f240039b0 0x7f0f24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.739+0000 7f0f3be00640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f24004800 0x7f0f24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f41000640 1 -- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 msgr2=0x7f0f240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f41a00640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 0x7f0f240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f41000640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f3c0cedb0 0x7f0f240051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f41000640 1 -- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f24004800 msgr2=0x7f0f24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f41000640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f24004800 0x7f0f24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f41000640 1 -- 172.21.15.6:0/417182155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24003520 con 0x7f0f240039b0 2024-03-06T07:33:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.740+0000 7f0f3be00640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f24004800 0x7f0f24004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.741+0000 7f0f41000640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f240039b0 0x7f0f24003dd0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009ac0 tx=0x7f0f24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367c7dfdf476209c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.741+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30039070 con 0x7f0f240039b0 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.741+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3002c6b0 con 0x7f0f240039b0 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.741+0000 7f0f3b400640 1 -- 172.21.15.6:0/417182155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30009690 con 0x7f0f240039b0 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f43a00640 1 -- 172.21.15.6:0/417182155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f240039b0 msgr2=0x7f0f24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f43a00640 1 --2- 172.21.15.6:0/417182155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f240039b0 0x7f0f24003dd0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009ac0 tx=0x7f0f24005830 comp rx=0 tx=0).stop 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f41a00640 1 -- 172.21.15.6:0/417182155 reap_dead start 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f43a00640 1 -- 172.21.15.6:0/417182155 shutdown_connections 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f43a00640 1 -- 172.21.15.6:0/417182155 >> 172.21.15.6:0/417182155 conn(0x7f0f3c06cf40 msgr2=0x7f0f3c06d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f43a00640 1 -- 172.21.15.6:0/417182155 shutdown_connections 2024-03-06T07:33:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.742+0000 7f0f43a00640 1 -- 172.21.15.6:0/417182155 wait complete. 2024-03-06T07:33:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.744+0000 7f0f43a00640 1 Processor -- start 2024-03-06T07:33:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.744+0000 7f0f43a00640 1 -- start start 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f43a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f240039b0 0x7f0f3c0b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f43a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f3c159af0 0x7f0f3c0b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f43a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 0x7f0f3c0b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f43a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c140790 con 0x7f0f3c0b4920 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f43a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c072b30 con 0x7f0f3c159af0 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f43a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c140910 con 0x7f0f240039b0 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f41a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 0x7f0f3c0b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f41a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 0x7f0f3c0b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44662/0 (socket says 172.21.15.6:44662) 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.745+0000 7f0f41a00640 1 -- 172.21.15.6:0/234422242 learned_addr learned my addr 172.21.15.6:0/234422242 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f3be00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f3c159af0 0x7f0f3c0b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41000640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f240039b0 0x7f0f3c0b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41a00640 1 -- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f240039b0 msgr2=0x7f0f3c0b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f240039b0 0x7f0f3c0b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41a00640 1 -- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f3c159af0 msgr2=0x7f0f3c0b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f3c159af0 0x7f0f3c0b1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41a00640 1 -- 172.21.15.6:0/234422242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c134b40 con 0x7f0f3c0b4920 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f41a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 0x7f0f3c0b2bd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c074ef0 tx=0x7f0f3400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.746+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3401c070 con 0x7f0f3c0b4920 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.747+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f34017400 con 0x7f0f3c0b4920 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.747+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f340177f0 con 0x7f0f3c0b4920 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.747+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c134e20 con 0x7f0f3c0b4920 2024-03-06T07:33:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.747+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c1353e0 con 0x7f0f3c0b4920 2024-03-06T07:33:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.748+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c0b5810 con 0x7f0f3c0b4920 2024-03-06T07:33:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.749+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f34005d10 con 0x7f0f3c0b4920 2024-03-06T07:33:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.750+0000 7f0f38c00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f2006b500 0x7f0f2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.750+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0f3408a710 con 0x7f0f3c0b4920 2024-03-06T07:33:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.750+0000 7f0f41000640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f2006b500 0x7f0f2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.751+0000 7f0f41000640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f2006b500 0x7f0f2006d9c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0f30036000 tx=0x7f0f30030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:04.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.755+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f3c0b5810 con 0x7f0f3c0b4920 2024-03-06T07:33:04.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.992+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f3c0692a0 con 0x7f0f3c0b4920 2024-03-06T07:33:04.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.993+0000 7f0f38c00640 1 -- 172.21.15.6:0/234422242 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f0f34012070 con 0x7f0f3c0b4920 2024-03-06T07:33:04.996 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:04.996 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:04.996 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.995+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f2006b500 msgr2=0x7f0f2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.995+0000 7f0f43a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f2006b500 0x7f0f2006d9c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0f30036000 tx=0x7f0f30030000 comp rx=0 tx=0).stop 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.995+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 msgr2=0x7f0f3c0b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.995+0000 7f0f43a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 0x7f0f3c0b2bd0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c074ef0 tx=0x7f0f3400f4a0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 shutdown_connections 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f2006b500 0x7f0f2006d9c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f3c159af0 0x7f0f3c0b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f3c0b4920 0x7f0f3c0b2bd0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 --2- 172.21.15.6:0/234422242 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f240039b0 0x7f0f3c0b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 >> 172.21.15.6:0/234422242 conn(0x7f0f3c06cf40 msgr2=0x7f0f3c15e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:04.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 shutdown_connections 2024-03-06T07:33:04.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:04.996+0000 7f0f43a00640 1 -- 172.21.15.6:0/234422242 wait complete. 2024-03-06T07:33:05.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:32:47.892948+0000', 'modified': '2024-03-06T07:33:03.669632+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 97, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7362}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7362': {'gid': 7362, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/619677659', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 619677659}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 619677659}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7362, 'qdb_cluster': [7362]} 2024-03-06T07:33:05.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:33:05.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:33:05.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T07:33:05.655+0000 7f87da200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:05.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.660+0000 7fd0ae600640 1 Processor -- start 2024-03-06T07:33:05.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.660+0000 7fd0ae600640 1 -- start start 2024-03-06T07:33:05.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.661+0000 7fd0ae600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.661+0000 7fd0ae600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd0a80cf720 0x7fd0a80d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.661+0000 7fd0ae600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80cfbc0 con 0x7fd0a80d09d0 2024-03-06T07:33:05.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.661+0000 7fd0ae600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a8161e20 con 0x7fd0a80cf720 2024-03-06T07:33:05.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.661+0000 7fd0ae600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80b38f0 con 0x7fd0a80d2e30 2024-03-06T07:33:05.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.662+0000 7fd09fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd0a80cf720 0x7fd0a80d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.662+0000 7fd09fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd0a80cf720 0x7fd0a80d1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:42354/0 (socket says 172.21.15.6:42354) 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.662+0000 7fd09fe00640 1 -- 172.21.15.6:0/1119950221 learned_addr learned my addr 172.21.15.6:0/1119950221 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.662+0000 7fd0a7400640 1 --2- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.662+0000 7fd09fe00640 1 -- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 msgr2=0x7fd0a80d4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.662+0000 7fd09fe00640 1 --2- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80d4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.663+0000 7fd09fe00640 1 -- 172.21.15.6:0/1119950221 >> v1:172.21.15.178:6789/0 conn(0x7fd0a80d09d0 legacy=0x7fd0a80b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.663+0000 7fd09fe00640 1 -- 172.21.15.6:0/1119950221 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80cfea0 con 0x7fd0a80cf720 2024-03-06T07:33:05.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.664+0000 7fd09fe00640 1 --2- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80d1bd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd090009870 tx=0x7fd09002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5306da0e01918dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.664+0000 7fd0a6a00640 1 -- 172.21.15.6:0/1119950221 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090034070 con 0x7fd0a80cf720 2024-03-06T07:33:05.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.664+0000 7fd0a6a00640 1 -- 172.21.15.6:0/1119950221 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090030840 con 0x7fd0a80cf720 2024-03-06T07:33:05.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.664+0000 7fd0a6a00640 1 -- 172.21.15.6:0/1119950221 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090030c10 con 0x7fd0a80cf720 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 -- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 msgr2=0x7fd0a80d1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 --2- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80d1bd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd090009870 tx=0x7fd09002b980 comp rx=0 tx=0).stop 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 -- 172.21.15.6:0/1119950221 shutdown_connections 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 --2- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80d1bd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 --2- 172.21.15.6:0/1119950221 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80d4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 -- 172.21.15.6:0/1119950221 >> 172.21.15.6:0/1119950221 conn(0x7fd0a80d4000 msgr2=0x7fd0a80d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 -- 172.21.15.6:0/1119950221 shutdown_connections 2024-03-06T07:33:05.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.665+0000 7fd0ae600640 1 -- 172.21.15.6:0/1119950221 wait complete. 2024-03-06T07:33:05.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.667+0000 7fd0ae600640 1 Processor -- start 2024-03-06T07:33:05.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.667+0000 7fd0ae600640 1 -- start start 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0ae600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0ae600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 0x7fd0a81584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0ae600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0ae600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80b50e0 con 0x7fd0a80d09d0 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0ae600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80b3e20 con 0x7fd0a80cf720 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0ae600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80b5260 con 0x7fd0a80d2e30 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 0x7fd0a81584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 0x7fd0a81584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44664/0 (socket says 172.21.15.6:44664) 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd09fe00640 1 -- 172.21.15.6:0/5646081 learned_addr learned my addr 172.21.15.6:0/5646081 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0a7400640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.668+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09fe00640 1 -- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 msgr2=0x7fd0a80cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09fe00640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09fe00640 1 -- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 msgr2=0x7fd0a80c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09fe00640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09fe00640 1 -- 172.21.15.6:0/5646081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8158fc0 con 0x7fd0a80d09d0 2024-03-06T07:33:05.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09fe00640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 0x7fd0a81584e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd090000c00 tx=0x7fd090004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.669+0000 7fd09f400640 1 -- 172.21.15.6:0/5646081 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090048070 con 0x7fd0a80d09d0 2024-03-06T07:33:05.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.670+0000 7fd09f400640 1 -- 172.21.15.6:0/5646081 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090030640 con 0x7fd0a80d09d0 2024-03-06T07:33:05.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.670+0000 7fd09f400640 1 -- 172.21.15.6:0/5646081 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090034040 con 0x7fd0a80d09d0 2024-03-06T07:33:05.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.670+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a81592a0 con 0x7fd0a80d09d0 2024-03-06T07:33:05.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.670+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a8159b40 con 0x7fd0a80d09d0 2024-03-06T07:33:05.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.672+0000 7fd09f400640 1 -- 172.21.15.6:0/5646081 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd09004b070 con 0x7fd0a80d09d0 2024-03-06T07:33:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.673+0000 7fd09f400640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd08406b4d0 0x7fd08406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.673+0000 7fd09f400640 1 -- 172.21.15.6:0/5646081 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd0900ac1f0 con 0x7fd0a80d09d0 2024-03-06T07:33:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.673+0000 7fd0b13a1740 1 Processor -- start 2024-03-06T07:33:05.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.674+0000 7fd0a7400640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd08406b4d0 0x7fd08406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.674+0000 7fd0b13a1740 1 -- start start 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0a7400640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd08406b4d0 0x7fd08406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd098009a30 tx=0x7fd098008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf272750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 0x5619bf2a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619bf1dc600 con 0x5619bf272c90 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619bf185930 con 0x5619bf29f7b0 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619bf1d86a0 con 0x5619bf29fb90 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 0x5619bf2a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 0x5619bf2a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44672/0 (socket says 172.21.15.6:44672) 2024-03-06T07:33:05.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3397560434 learned_addr learned my addr 172.21.15.6:0/3397560434 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd0a7400640 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf272750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.675+0000 7fd09fe00640 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 msgr2=0x5619bf2a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 msgr2=0x5619bf272750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf272750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd0a7e00640 1 -- 172.21.15.6:0/3397560434 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619beef12d0 con 0x5619bf272c90 2024-03-06T07:33:05.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 0x5619bf2a2a80 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0001c40 tx=0x7fd0a0002100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfe7e35f226ce09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd097e00640 1 -- 172.21.15.6:0/3397560434 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0019070 con 0x5619bf272c90 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd097e00640 1 -- 172.21.15.6:0/3397560434 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0003b80 con 0x5619bf272c90 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.676+0000 7fd097e00640 1 -- 172.21.15.6:0/3397560434 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000e830 con 0x5619bf272c90 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3397560434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 msgr2=0x5619bf2a2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 0x5619bf2a2a80 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0001c40 tx=0x7fd0a0002100 comp rx=0 tx=0).stop 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3397560434 shutdown_connections 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf272c90 0x5619bf2a2a80 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3397560434 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf272750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3397560434 >> 172.21.15.6:0/3397560434 conn(0x5619bf2749c0 msgr2=0x5619bf2bd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3397560434 shutdown_connections 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.677+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3397560434 wait complete. 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.678+0000 7fd0b13a1740 1 -- start start 2024-03-06T07:33:05.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.678+0000 7fd0b13a1740 1 Processor -- start 2024-03-06T07:33:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29f7b0 0x5619bf2cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 0x5619bf28e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf28ea90 0x5619bf2a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619bf276f60 con 0x5619bf29fb90 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619bf198d90 con 0x5619bf28ea90 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619bf276350 con 0x5619bf29f7b0 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.679+0000 7fd0b13a1740 10 client.0 _ll_register_callbacks cb 0x5619bf1d4350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 0x5619bf28e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 0x5619bf28e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44684/0 (socket says 172.21.15.6:44684) 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 -- 172.21.15.6:0/306770096 learned_addr learned my addr 172.21.15.6:0/306770096 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd0a7400640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29f7b0 0x5619bf2cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf28ea90 0x5619bf2a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 -- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29f7b0 msgr2=0x5619bf2cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29f7b0 0x5619bf2cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 -- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf28ea90 msgr2=0x5619bf2a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf28ea90 0x5619bf2a4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 -- 172.21.15.6:0/306770096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619befbb6b0 con 0x5619bf29fb90 2024-03-06T07:33:05.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf28ea90 0x5619bf2a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd09fe00640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 0x5619bf28e550 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd09002b9c0 tx=0x7fd09007b110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac24122037b8684 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.680+0000 7fd094c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd094c00640 1 -- 172.21.15.6:0/306770096 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090079fa0 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd094c00640 1 -- 172.21.15.6:0/306770096 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd0900766a0 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd0a0003b80 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd094c00640 1 -- 172.21.15.6:0/306770096 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd09007eaa0 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 10 client.7395 fetch_fsmap learned FSMap version 69 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x5619bf28fb20 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd094c00640 1 -- 172.21.15.6:0/306770096 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0003b80 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd094c00640 1 -- 172.21.15.6:0/306770096 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 69) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd0900768a0 con 0x5619bf29fb90 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 10 client.7395 fetch_fsmap finished waiting for FSMap version 69 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 10 client.7395 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 20 client.7395 populate_metadata read hostname 'smithi006' 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.681+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf2908b0 0x5619bf291df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.682+0000 7fd0b13a1740 4 client.7395 mds_command: new command op to 7362 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:33:05.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.682+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 --> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619bf1e9b40 con 0x5619bf2908b0 2024-03-06T07:33:05.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.682+0000 7fd0a7400640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf2908b0 0x5619bf291df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.682+0000 7fd0a7400640 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf2908b0 0x5619bf291df0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33ca61fa88a43a55 server_cookie=b9f70347a81796c2 in_seq=0 out_seq=0 2024-03-06T07:33:05.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.682+0000 7fd094c00640 10 client.7395 ms_handle_connect on v2:172.21.15.6:6834/619677659 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.683+0000 7fd094c00640 1 -- 172.21.15.6:0/306770096 <== mds.0 v2:172.21.15.6:6834/619677659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5619bf1e9b40 con 0x5619bf2908b0 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.683+0000 7fd094c00640 10 client.7395 handle_command_reply: tid=0 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.683+0000 7fd0b13a1740 1 client.7395 shutdown 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 msgr2=0x5619bf28e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 0x5619bf28e550 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd09002b9c0 tx=0x7fd09007b110 comp rx=0 tx=0).stop 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 shutdown_connections 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf28ea90 0x5619bf2a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf2908b0 0x5619bf291df0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29fb90 0x5619bf28e550 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/306770096 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29f7b0 0x5619bf2cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 >> 172.21.15.6:0/306770096 conn(0x5619bf272c90 msgr2=0x5619bf2a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd094c00640 0 client.7395 ms_handle_reset on v2:172.21.15.6:6834/619677659 2024-03-06T07:33:05.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.684+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 shutdown_connections 2024-03-06T07:33:05.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.685+0000 7fd0b13a1740 1 -- 172.21.15.6:0/306770096 wait complete. 2024-03-06T07:33:05.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.685+0000 7fd0b13a1740 20 client.7395 trim_cache size 0 max 16384 2024-03-06T07:33:05.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.704+0000 7fd0b13a1740 1 Processor -- start 2024-03-06T07:33:05.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.704+0000 7fd0b13a1740 1 -- start start 2024-03-06T07:33:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.705+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 0x5619bf2b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.705+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf2a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.705+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.705+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a81ff8d0 con 0x5619bf28ea90 2024-03-06T07:33:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.705+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a81fed00 con 0x5619bf29f7b0 2024-03-06T07:33:05.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.705+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a81ff2d0 con 0x5619bf29fb90 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 0x5619bf2b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 0x5619bf2b89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44698/0 (socket says 172.21.15.6:44698) 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 -- 172.21.15.6:0/3161688534 learned_addr learned my addr 172.21.15.6:0/3161688534 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd09fe00640 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf2a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 -- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 msgr2=0x5619bf2a24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 -- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 msgr2=0x5619bf2a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf2a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 -- 172.21.15.6:0/3161688534 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619bf28df30 con 0x5619bf28ea90 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd09fe00640 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf2a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.706+0000 7fd0a7400640 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 0x5619bf2b89e0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x5619bf291d70 tx=0x7fd098014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c8c2e7f1c7c755 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd097400640 1 -- 172.21.15.6:0/3161688534 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098030070 con 0x5619bf28ea90 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd097400640 1 -- 172.21.15.6:0/3161688534 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09801a040 con 0x5619bf28ea90 2024-03-06T07:33:05.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd097400640 1 -- 172.21.15.6:0/3161688534 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09802a2d0 con 0x5619bf28ea90 2024-03-06T07:33:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3161688534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 msgr2=0x5619bf2b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 0x5619bf2b89e0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x5619bf291d70 tx=0x7fd098014000 comp rx=0 tx=0).stop 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3161688534 shutdown_connections 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29f7b0 0x5619bf2a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf29fb90 0x5619bf2a24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/3161688534 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf28ea90 0x5619bf2b89e0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.707+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3161688534 >> 172.21.15.6:0/3161688534 conn(0x5619bf272c90 msgr2=0x5619bf26d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.708+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3161688534 shutdown_connections 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.708+0000 7fd0b13a1740 1 -- 172.21.15.6:0/3161688534 wait complete. 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.708+0000 7fd0b13a1740 1 -- start start 2024-03-06T07:33:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.708+0000 7fd0b13a1740 1 Processor -- start 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.709+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf28ea90 0x5619bf2cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.709+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 0x5619bf2cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.709+0000 7fd0b13a1740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29fb90 0x5619bf277970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.709+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619bf290730 con 0x5619bf29f7b0 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.709+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619bf276f60 con 0x5619bf29fb90 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.709+0000 7fd0b13a1740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619bf290d30 con 0x5619bf28ea90 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd0b13a1740 10 client.0 _ll_register_callbacks cb 0x7fd0900095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 0x5619bf2cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 0x5619bf2cc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44700/0 (socket says 172.21.15.6:44700) 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 -- 172.21.15.6:0/839646743 learned_addr learned my addr 172.21.15.6:0/839646743 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd0a7400640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf28ea90 0x5619bf2cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 -- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf28ea90 msgr2=0x5619bf2cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf28ea90 0x5619bf2cbc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 -- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29fb90 msgr2=0x5619bf277970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29fb90 0x5619bf277970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29fb90 0x5619bf277970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 -- 172.21.15.6:0/839646743 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619bf291410 con 0x5619bf29f7b0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd0a7e00640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29fb90 0x5619bf277970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd09fe00640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 0x5619bf2cc150 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd0900772d0 tx=0x7fd0900aa540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3311ad6ccf140ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.710+0000 7fd094c00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd094c00640 1 -- 172.21.15.6:0/839646743 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09007a490 con 0x5619bf29f7b0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd094c00640 1 -- 172.21.15.6:0/839646743 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd0900768b0 con 0x5619bf29f7b0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd09801a040 con 0x5619bf29f7b0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd094c00640 1 -- 172.21.15.6:0/839646743 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd090083140 con 0x5619bf29f7b0 2024-03-06T07:33:05.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 10 client.7401 fetch_fsmap learned FSMap version 69 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x5619bf2a3c00 con 0x5619bf29f7b0 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd094c00640 1 -- 172.21.15.6:0/839646743 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09801a040 con 0x5619bf29f7b0 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd094c00640 1 -- 172.21.15.6:0/839646743 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 69) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd09008e430 con 0x5619bf29f7b0 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 10 client.7401 fetch_fsmap finished waiting for FSMap version 69 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 10 client.7401 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 20 client.7401 populate_metadata read hostname 'smithi006' 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf278050 0x5619bf278430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 4 client.7401 mds_command: new command op to 7362 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.711+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 --> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5619bf2777c0 con 0x5619bf278050 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.712+0000 7fd0a7400640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf278050 0x5619bf278430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.712+0000 7fd0a7400640 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf278050 0x5619bf278430 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ddb07c5765139f server_cookie=8070f827efa64135 in_seq=0 out_seq=0 2024-03-06T07:33:05.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.712+0000 7fd094c00640 10 client.7401 ms_handle_connect on v2:172.21.15.6:6834/619677659 2024-03-06T07:33:05.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd094c00640 1 -- 172.21.15.6:0/839646743 <== mds.0 v2:172.21.15.6:6834/619677659 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5619bf2777c0 con 0x5619bf278050 2024-03-06T07:33:05.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd094c00640 10 client.7401 handle_command_reply: tid=0 2024-03-06T07:33:05.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 client.7401 shutdown 2024-03-06T07:33:05.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 msgr2=0x5619bf2cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 0x5619bf2cc150 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fd0900772d0 tx=0x7fd0900aa540 comp rx=0 tx=0).stop 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 shutdown_connections 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5619bf29fb90 0x5619bf277970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:6834/619677659,v1:172.21.15.6:6835/619677659] conn(0x5619bf278050 0x5619bf278430 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5619bf29f7b0 0x5619bf2cc150 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 --2- 172.21.15.6:0/839646743 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5619bf28ea90 0x5619bf2cbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 >> 172.21.15.6:0/839646743 conn(0x5619bf272c90 msgr2=0x5619bf26d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd094c00640 0 client.7401 ms_handle_reset on v2:172.21.15.6:6834/619677659 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 shutdown_connections 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 1 -- 172.21.15.6:0/839646743 wait complete. 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.713+0000 7fd0b13a1740 20 client.7401 trim_cache size 0 max 16384 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:33:05.718 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7362, 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 5.0645465779999999, 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 69, 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 97, 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 97, 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 5.824769452 2024-03-06T07:33:05.719 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd08406b4d0 msgr2=0x7fd08406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd08406b4d0 0x7fd08406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd098009a30 tx=0x7fd098008040 comp rx=0 tx=0).stop 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 msgr2=0x7fd0a81584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 0x7fd0a81584e0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd090000c00 tx=0x7fd090004730 comp rx=0 tx=0).stop 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 shutdown_connections 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd08406b4d0 0x7fd08406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0a80d2e30 0x7fd0a80cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0a80d09d0 0x7fd0a81584e0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 --2- 172.21.15.6:0/5646081 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0a80cf720 0x7fd0a80c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 >> 172.21.15.6:0/5646081 conn(0x7fd0a80d4000 msgr2=0x7fd0a8074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 shutdown_connections 2024-03-06T07:33:05.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:05.714+0000 7fd0ae600640 1 -- 172.21.15.6:0/5646081 wait complete. 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> subdir = "delete_me" 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> size_unit = 1024 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> file_multiplier = 200 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> for i in range(0, file_multiplier): 2024-03-06T07:33:05.736 DEBUG:teuthology.orchestra.run.smithi006:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:33:05.737 DEBUG:teuthology.orchestra.run.smithi006:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:33:05.737 DEBUG:teuthology.orchestra.run.smithi006:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:33:05.737 DEBUG:teuthology.orchestra.run.smithi006:> f.write(size * '"'"'x'"'"') 2024-03-06T07:33:05.737 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:33:10.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:33:10.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_ops 2024-03-06T07:33:10.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:10.456 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:33:10.684 INFO:teuthology.orchestra.run.smithi178.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:33:10.685 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:33:10.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_files 2024-03-06T07:33:10.927 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:33:10.927 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:33:10.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:11.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.301+0000 7f66e9200640 1 Processor -- start 2024-03-06T07:33:11.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.301+0000 7f66e9200640 1 -- start start 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.302+0000 7f66e9200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f66e41457e0 0x7f66e40cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e9200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e4145bc0 con 0x7f66e415fba0 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e9200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e40cfb20 con 0x7f66e41457e0 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4160040 con 0x7f66e40b5870 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f66e41457e0 0x7f66e40cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e2800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f66e41457e0 0x7f66e40cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:42402/0 (socket says 172.21.15.6:42402) 2024-03-06T07:33:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.303+0000 7f66e1e00640 1 -- 172.21.15.6:0/257074994 learned_addr learned my addr 172.21.15.6:0/257074994 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.304+0000 7f66e1e00640 1 -- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 msgr2=0x7f66e40710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.304+0000 7f66e1e00640 1 --2- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e40710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.304+0000 7f66e1e00640 1 -- 172.21.15.6:0/257074994 >> v1:172.21.15.178:6789/0 conn(0x7f66e415fba0 legacy=0x7f66e40b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.304+0000 7f66e1e00640 1 -- 172.21.15.6:0/257074994 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e4160320 con 0x7f66e41457e0 2024-03-06T07:33:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.306+0000 7f66e1e00640 1 --2- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e41457e0 0x7f66e40cf5e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009870 tx=0x7f66d402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3d2ed12ea4c0623 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.306+0000 7f66e0a00640 1 -- 172.21.15.6:0/257074994 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4035020 con 0x7f66e41457e0 2024-03-06T07:33:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.306+0000 7f66e0a00640 1 -- 172.21.15.6:0/257074994 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d4031640 con 0x7f66e41457e0 2024-03-06T07:33:11.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.306+0000 7f66e0a00640 1 -- 172.21.15.6:0/257074994 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d402d040 con 0x7f66e41457e0 2024-03-06T07:33:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 -- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e41457e0 msgr2=0x7f66e40cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 --2- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e41457e0 0x7f66e40cf5e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f66d4009870 tx=0x7f66d402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 -- 172.21.15.6:0/257074994 shutdown_connections 2024-03-06T07:33:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 --2- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e40710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 --2- 172.21.15.6:0/257074994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e41457e0 0x7f66e40cf5e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 -- 172.21.15.6:0/257074994 >> 172.21.15.6:0/257074994 conn(0x7f66e406cf40 msgr2=0x7f66e406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 -- 172.21.15.6:0/257074994 shutdown_connections 2024-03-06T07:33:11.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.307+0000 7f66e9200640 1 -- 172.21.15.6:0/257074994 wait complete. 2024-03-06T07:33:11.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.309+0000 7f66e9200640 1 Processor -- start 2024-03-06T07:33:11.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.310+0000 7f66e9200640 1 -- start start 2024-03-06T07:33:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.311+0000 7f66e9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e41f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.311+0000 7f66e9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66e41457e0 0x7f66e41f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.311+0000 7f66e9200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e415fba0 0x7f66e4141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.311+0000 7f66e9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40cca70 con 0x7f66e41457e0 2024-03-06T07:33:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.311+0000 7f66e9200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41608e0 con 0x7f66e415fba0 2024-03-06T07:33:11.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.311+0000 7f66e9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40715f0 con 0x7f66e40b5870 2024-03-06T07:33:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.312+0000 7f66e1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66e41457e0 0x7f66e41f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.312+0000 7f66e2800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e415fba0 0x7f66e4141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.312+0000 7f66e1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e41f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.312+0000 7f66e1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e41f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:42160/0 (socket says 172.21.15.6:42160) 2024-03-06T07:33:11.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.312+0000 7f66e1e00640 1 -- 172.21.15.6:0/1818461342 learned_addr learned my addr 172.21.15.6:0/1818461342 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.312+0000 7f66e1e00640 1 -- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e415fba0 msgr2=0x7f66e4141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e1e00640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e415fba0 0x7f66e4141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e1e00640 1 -- 172.21.15.6:0/1818461342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66e41457e0 msgr2=0x7f66e41f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e1e00640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66e41457e0 0x7f66e41f6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e1e00640 1 -- 172.21.15.6:0/1818461342 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e4142730 con 0x7f66e40b5870 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e1400640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66e41457e0 0x7f66e41f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:11.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e2800640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e415fba0 0x7f66e4141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.313+0000 7f66e1e00640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e41f4160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f66d402b930 tx=0x7f66d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.314+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4040070 con 0x7f66e40b5870 2024-03-06T07:33:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.314+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d403e070 con 0x7f66e40b5870 2024-03-06T07:33:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.314+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4035040 con 0x7f66e40b5870 2024-03-06T07:33:11.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.314+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e4142a10 con 0x7f66e40b5870 2024-03-06T07:33:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.314+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e4142fd0 con 0x7f66e40b5870 2024-03-06T07:33:11.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.316+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e40692a0 con 0x7f66e40b5870 2024-03-06T07:33:11.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.317+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66d4038020 con 0x7f66e40b5870 2024-03-06T07:33:11.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.318+0000 7f66da000640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66c406b520 0x7f66c406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:11.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.318+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f66d40ac200 con 0x7f66e40b5870 2024-03-06T07:33:11.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.319+0000 7f66e1400640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66c406b520 0x7f66c406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:11.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.320+0000 7f66e1400640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66c406b520 0x7f66c406d9e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f66e40c5490 tx=0x7f66d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:11.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.333+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66d407a9f0 con 0x7f66e40b5870 2024-03-06T07:33:11.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.655+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66e406ed80 con 0x7f66e40b5870 2024-03-06T07:33:11.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.656+0000 7f66da000640 1 -- 172.21.15.6:0/1818461342 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f66d407ea90 con 0x7f66e40b5870 2024-03-06T07:33:11.658 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:11.658 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:11.659 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.659+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66c406b520 msgr2=0x7f66c406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.659+0000 7f66e9200640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66c406b520 0x7f66c406d9e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f66e40c5490 tx=0x7f66d0009040 comp rx=0 tx=0).stop 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.659+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 msgr2=0x7f66e41f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.659+0000 7f66e9200640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e41f4160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f66d402b930 tx=0x7f66d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 shutdown_connections 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66c406b520 0x7f66c406d9e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66e415fba0 0x7f66e4141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66e41457e0 0x7f66e41f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 --2- 172.21.15.6:0/1818461342 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66e40b5870 0x7f66e41f4160 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:11.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 >> 172.21.15.6:0/1818461342 conn(0x7f66e406cf40 msgr2=0x7f66e40aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:11.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 shutdown_connections 2024-03-06T07:33:11.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:11.660+0000 7f66e9200640 1 -- 172.21.15.6:0/1818461342 wait complete. 2024-03-06T07:33:11.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:11.942 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":23,"msgr_send_messages":23,"msgr_recv_bytes":15091,"msgr_send_bytes":4223179,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.028840462,"msgr_running_send_time":0.015398246,"msgr_running_recv_time":0.006811373,"msgr_running_fast_dispatch_time":0.003298479,"msgr_send_messages_queue_lat":{"avgcount":22,"sum":0.013466430,"avgtime":0.000612110},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6740,"msgr_send_messages":11802,"msgr_recv_bytes":2431940,"msgr_send_bytes":9979894,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.517407892,"msgr_running_send_time":1.030209740,"msgr_running_recv_time":1.014031123,"msgr_running_fast_dispatch_time":0.007156380,"msgr_send_messages_queue_lat":{"avgcount":11801,"sum":0.738216684,"avgtime":0.000062555},"msgr_handle_ack_lat":{"avgcount":13952,"sum":0.007829438,"avgtime":0.000000561},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":60800},"AsyncMessenger::Worker-2":{"msgr_recv_messages":41,"msgr_send_messages":37,"msgr_recv_bytes":117551,"msgr_send_bytes":1695056,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.031927542,"msgr_running_send_time":0.021622356,"msgr_running_recv_time":0.009111838,"msgr_running_fast_dispatch_time":0.003210029,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.012852509,"avgtime":0.000357014},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106000,"msgr_send_encrypted_bytes":6032},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":944,"complete_latency":{"avgcount":67,"sum":3.255955438,"avgtime":0.048596349}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000123054,"avgtime":0.000030763}},"mds":{"request":3203,"reply":3203,"reply_latency":{"avgcount":3203,"sum":847.600687447,"avgtime":0.264627126},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3205,"traverse_hit":3205,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2727,"root_rbytes":19239936,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":3009,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3325,"handle_client_caps_dirty":2851,"handle_client_cap_release":0,"process_request_cap_release":2},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":6533,"evex":0,"evtrm":0,"ev":6538,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":17836717,"rdpos":4195674,"jlat":{"avgcount":9,"sum":0.215986333,"avgtime":0.023998481},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3211,"dn+":3211,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":91464,"heap":248164},"mds_server":{"dispatch_client_request":3203,"dispatch_server_request":0,"handle_client_request":3203,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3084,"average_load":3084,"avg_session_uptime":950,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13299093,"buffer_anon_items":7946,"buffer_meta_bytes":80872,"buffer_meta_items":919,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11071609,"mds_co_items":112132,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":62,"op_send_bytes":12906879,"op_resend":0,"op_reply":62,"op_latency":{"avgcount":62,"sum":0.358996067,"avgtime":0.005790259},"op_inflight":0,"oplen_avg":{"avgcount":62,"sum":115},"op":62,"op_r":36,"op_w":26,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":9,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":13,"omap_rd":48,"omap_del":1},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":46608,"max":104857600,"get_started":0,"get":6797,"get_sum":1949401,"get_or_fail_fail":0,"get_or_fail_success":6797,"take":0,"take_sum":0,"put":6623,"put_sum":1902793,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":12908800,"put":62,"put_sum":12908800,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":62,"take_sum":62,"put":62,"put_sum":62,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":1061537,"max":3758096384,"get_started":0,"get":6534,"get_sum":13642264,"get_or_fail_fail":0,"get_or_fail_success":6534,"take":0,"take_sum":0,"put":9,"put_sum":12580727,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:11.944 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 15091, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 23, "msgr_running_fast_dispatch_time": 0.003298479, "msgr_running_recv_time": 0.006811373, "msgr_running_send_time": 0.015398246, "msgr_running_total_time": 0.028840462, "msgr_send_bytes": 4223179, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 23, "msgr_send_messages_queue_lat": { "avgcount": 22, "avgtime": 0.00061211, "sum": 0.01346643 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 13952, "avgtime": 5.61e-07, "sum": 0.007829438 }, "msgr_recv_bytes": 2431940, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 6740, "msgr_running_fast_dispatch_time": 0.00715638, "msgr_running_recv_time": 1.014031123, "msgr_running_send_time": 1.03020974, "msgr_running_total_time": 1.517407892, "msgr_send_bytes": 9979894, "msgr_send_encrypted_bytes": 60800, "msgr_send_messages": 11802, "msgr_send_messages_queue_lat": { "avgcount": 11801, "avgtime": 6.2555e-05, "sum": 0.738216684 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 117551, "msgr_recv_encrypted_bytes": 106000, "msgr_recv_messages": 41, "msgr_running_fast_dispatch_time": 0.003210029, "msgr_running_recv_time": 0.009111838, "msgr_running_send_time": 0.021622356, "msgr_running_total_time": 0.031927542, "msgr_send_bytes": 1695056, "msgr_send_encrypted_bytes": 6032, "msgr_send_messages": 37, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.000357014, "sum": 0.012852509 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 67, "avgtime": 0.048596349, "sum": 3.255955438 }, "queue_len": 944 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.0763e-05, "sum": 0.000123054 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3009, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3325, "handle_client_caps_dirty": 2851, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2, "q": 0, "reply": 3203, "reply_latency": { "avgcount": 3203, "avgtime": 0.264627126, "sum": 847.600687447 }, "request": 3203, "root_rbytes": 19239936, "root_rfiles": 2727, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3205, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3205, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6538, "evadd": 6533, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 9, "avgtime": 0.023998481, "sum": 0.215986333 }, "rdpos": 4195674, "replayed": 5, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 17836717 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3211, "dn+": 3211, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 91464 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3203, "dispatch_server_request": 0, "handle_client_request": 3203, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 3084, "avg_session_uptime": 950, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3084 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13299093, "buffer_anon_items": 7946, "buffer_meta_bytes": 80872, "buffer_meta_items": 919, "mds_co_bytes": 11071609, "mds_co_items": 112132, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 1, "omap_rd": 48, "omap_wr": 13, "op": 62, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 62, "avgtime": 0.005790259, "sum": 0.358996067 }, "op_pg": 0, "op_r": 36, "op_reply": 62, "op_resend": 0, "op_rmw": 0, "op_send": 62, "op_send_bytes": 12906879, "op_w": 26, "oplen_avg": { "avgcount": 62, "sum": 115 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 9, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6797, "get_or_fail_fail": 0, "get_or_fail_success": 6797, "get_started": 0, "get_sum": 1949401, "max": 104857600, "put": 6623, "put_sum": 1902793, "take": 0, "take_sum": 0, "val": 46608, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 62, "put_sum": 12908800, "take": 62, "take_sum": 12908800, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 62, "put_sum": 62, "take": 62, "take_sum": 62, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 6534, "get_or_fail_fail": 0, "get_or_fail_success": 6534, "get_started": 0, "get_sum": 13642264, "max": 3758096384, "put": 9, "put_sum": 12580727, "take": 0, "take_sum": 0, "val": 1061537, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:11.944 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/0, 0/0 2024-03-06T07:33:12.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:13.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.162+0000 7f710b200640 1 Processor -- start 2024-03-06T07:33:13.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.163+0000 7f710b200640 1 -- start start 2024-03-06T07:33:13.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f710b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f710c0b4b30 0x7f710c1623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f710b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0b50d0 con 0x7f710c0d1be0 2024-03-06T07:33:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f710b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0cf6a0 con 0x7f710c15f630 2024-03-06T07:33:13.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f710b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710c0cf450 con 0x7f710c0b4b30 2024-03-06T07:33:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f710a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f710c0d1be0 0x7f710c0d2070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41220/0 (socket says 172.21.15.6:41220) 2024-03-06T07:33:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f710a800640 1 -- 172.21.15.6:0/724719210 learned_addr learned my addr 172.21.15.6:0/724719210 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.164+0000 7f7109400640 1 --2- 172.21.15.6:0/724719210 >> v2:172.21.15.178:3300/0 conn(0x7f710c0b4b30 0x7f710c1623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795006895 0 0) 0x7f710c0b50d0 con 0x7f710c0d1be0 2024-03-06T07:33:13.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f0003520 con 0x7f710c0d1be0 2024-03-06T07:33:13.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3490909436 0 0) 0x7f70f0003520 con 0x7f710c0d1be0 2024-03-06T07:33:13.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 >> v2:172.21.15.178:3300/0 conn(0x7f710c0b4b30 msgr2=0x7f710c1623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 --2- 172.21.15.6:0/724719210 >> v2:172.21.15.178:3300/0 conn(0x7f710c0b4b30 0x7f710c1623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 >> v1:172.21.15.178:6789/0 conn(0x7f710c15f630 legacy=0x7f710c161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c0cf980 con 0x7f710c0d1be0 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f71040019f0 con 0x7f710c0d1be0 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 >> v1:172.21.15.6:6789/0 conn(0x7f710c0d1be0 legacy=0x7f710c0d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f00039e0 0x7f70f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f0004800 0x7f70f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f70f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0cf450 con 0x7f70f00052e0 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f000c1e0 con 0x7f70f00039e0 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.165+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f000c3e0 con 0x7f70f0004800 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f710a800640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f70f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f7109400640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f0004800 0x7f70f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f7109e00640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f00039e0 0x7f70f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f710a800640 1 -- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f0004800 msgr2=0x7f70f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f710a800640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70f0004800 0x7f70f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f710a800640 1 -- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f00039e0 msgr2=0x7f70f0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f710a800640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70f00039e0 0x7f70f0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.166+0000 7f710a800640 1 -- 172.21.15.6:0/724719210 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0003520 con 0x7f70f00052e0 2024-03-06T07:33:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.168+0000 7f710a800640 1 -- 172.21.15.6:0/724719210 reap_dead start 2024-03-06T07:33:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.168+0000 7f710a800640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f70f000b9c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f710400a810 tx=0x7f70f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea60f051db033041 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.168+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710400a460 con 0x7f70f00052e0 2024-03-06T07:33:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.168+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7104034650 con 0x7f70f00052e0 2024-03-06T07:33:13.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.168+0000 7f7108a00640 1 -- 172.21.15.6:0/724719210 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7104034a40 con 0x7f70f00052e0 2024-03-06T07:33:13.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.169+0000 7f710b200640 1 -- 172.21.15.6:0/724719210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 msgr2=0x7f70f000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.169+0000 7f710b200640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f70f000b9c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f710400a810 tx=0x7f70f0005830 comp rx=0 tx=0).stop 2024-03-06T07:33:13.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.169+0000 7f710b200640 1 -- 172.21.15.6:0/724719210 shutdown_connections 2024-03-06T07:33:13.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.169+0000 7f710b200640 1 --2- 172.21.15.6:0/724719210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f70f000b9c0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.169+0000 7f710b200640 1 -- 172.21.15.6:0/724719210 >> 172.21.15.6:0/724719210 conn(0x7f710c0d3e30 msgr2=0x7f710c0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.170+0000 7f710b200640 1 -- 172.21.15.6:0/724719210 shutdown_connections 2024-03-06T07:33:13.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.170+0000 7f710b200640 1 -- 172.21.15.6:0/724719210 wait complete. 2024-03-06T07:33:13.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.172+0000 7f710b200640 1 Processor -- start 2024-03-06T07:33:13.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.173+0000 7f710b200640 1 -- start start 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f710b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f710c1f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f710b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f710c159080 0x7f710c15b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f710b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f710c15bcd0 0x7f710c1f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f710b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0d2070 con 0x7f70f00052e0 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f710b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0d4c40 con 0x7f710c159080 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f710b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0d21f0 con 0x7f710c15bcd0 2024-03-06T07:33:13.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f7109e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f710c1f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.174+0000 7f7109e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f710c1f4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46160/0 (socket says 172.21.15.6:46160) 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 -- 172.21.15.6:0/2947490141 learned_addr learned my addr 172.21.15.6:0/2947490141 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f710a800640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f710c15bcd0 0x7f710c1f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109400640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f710c159080 0x7f710c15b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 -- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f710c15bcd0 msgr2=0x7f710c1f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f710c15bcd0 0x7f710c1f74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 -- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f710c159080 msgr2=0x7f710c15b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f710c159080 0x7f710c15b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 -- 172.21.15.6:0/2947490141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c15dd50 con 0x7f70f00052e0 2024-03-06T07:33:13.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.175+0000 7f7109e00640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f710c1f4970 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00e4e0 tx=0x7f70fc00e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.176+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0052e0 con 0x7f70f00052e0 2024-03-06T07:33:13.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.176+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc018070 con 0x7f70f00052e0 2024-03-06T07:33:13.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.176+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc003830 con 0x7f70f00052e0 2024-03-06T07:33:13.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.176+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c15e030 con 0x7f70f00052e0 2024-03-06T07:33:13.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.176+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c15e5f0 con 0x7f70f00052e0 2024-03-06T07:33:13.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.177+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c15c0e0 con 0x7f70f00052e0 2024-03-06T07:33:13.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.178+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70fc0039d0 con 0x7f70f00052e0 2024-03-06T07:33:13.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.179+0000 7f7102000640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70ec06b500 0x7f70ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:13.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.179+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f70fc089e90 con 0x7f70f00052e0 2024-03-06T07:33:13.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.180+0000 7f7109400640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70ec06b500 0x7f70ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:13.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.181+0000 7f7109400640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70ec06b500 0x7f70ec06d9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f710c0cd540 tx=0x7f70f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:13.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.184+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f710c15c0e0 con 0x7f70f00052e0 2024-03-06T07:33:13.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.488+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f710c04f330 con 0x7f70f00052e0 2024-03-06T07:33:13.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.489+0000 7f7102000640 1 -- 172.21.15.6:0/2947490141 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f70fc058bc0 con 0x7f70f00052e0 2024-03-06T07:33:13.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:13.492 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:13.492 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70ec06b500 msgr2=0x7f70ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70ec06b500 0x7f70ec06d9c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f710c0cd540 tx=0x7f70f8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 msgr2=0x7f710c1f4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f710c1f4970 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00e4e0 tx=0x7f70fc00e9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 shutdown_connections 2024-03-06T07:33:13.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f70ec06b500 0x7f70ec06d9c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f710c159080 0x7f710c15b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f710c15bcd0 0x7f710c1f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 --2- 172.21.15.6:0/2947490141 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f70f00052e0 0x7f710c1f4970 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 >> 172.21.15.6:0/2947490141 conn(0x7f710c0d3e30 msgr2=0x7f710c15aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 shutdown_connections 2024-03-06T07:33:13.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:13.493+0000 7f710b200640 1 -- 172.21.15.6:0/2947490141 wait complete. 2024-03-06T07:33:13.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:13.785 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":24,"msgr_send_messages":24,"msgr_recv_bytes":15337,"msgr_send_bytes":4223497,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.029050487,"msgr_running_send_time":0.015489089,"msgr_running_recv_time":0.006848764,"msgr_running_fast_dispatch_time":0.003342585,"msgr_send_messages_queue_lat":{"avgcount":23,"sum":0.013546119,"avgtime":0.000588961},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7600,"msgr_send_messages":13970,"msgr_recv_bytes":2726465,"msgr_send_bytes":16741987,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.807885893,"msgr_running_send_time":1.208553518,"msgr_running_recv_time":1.140504232,"msgr_running_fast_dispatch_time":0.009644117,"msgr_send_messages_queue_lat":{"avgcount":13969,"sum":0.878498044,"avgtime":0.000062889},"msgr_handle_ack_lat":{"avgcount":16594,"sum":0.009317953,"avgtime":0.000000561},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":60800},"AsyncMessenger::Worker-2":{"msgr_recv_messages":41,"msgr_send_messages":37,"msgr_recv_bytes":117551,"msgr_send_bytes":1695056,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.032030252,"msgr_running_send_time":0.021672446,"msgr_running_recv_time":0.009134573,"msgr_running_fast_dispatch_time":0.003210029,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.012852509,"avgtime":0.000357014},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106000,"msgr_send_encrypted_bytes":6032},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":914,"complete_latency":{"avgcount":74,"sum":4.811755206,"avgtime":0.065023719}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000123054,"avgtime":0.000030763}},"mds":{"request":4057,"reply":4056,"reply_latency":{"avgcount":4056,"sum":849.622768386,"avgtime":0.209473069},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4058,"inodes_top":0,"inodes_bottom":4058,"inodes_pin_tail":0,"inodes_pinned":4058,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":4059,"traverse_hit":4059,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3200,"root_rbytes":23306240,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":3636,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3499,"handle_client_caps_dirty":2996,"handle_client_cap_release":0,"process_request_cap_release":856},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":847,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":847,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7697,"evex":0,"evtrm":0,"ev":7702,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":21190687,"rdpos":4195674,"jlat":{"avgcount":11,"sum":0.335028372,"avgtime":0.030457124},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4058,"dn+":4058,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":98248,"heap":248164},"mds_server":{"dispatch_client_request":4058,"dispatch_server_request":0,"handle_client_request":4057,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":847,"sum":1.128331215,"avgtime":0.001332150},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3848,"average_load":3848,"avg_session_uptime":950,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13299093,"buffer_anon_items":7946,"buffer_meta_bytes":80872,"buffer_meta_items":919,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":11071609,"mds_co_items":112132,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":67,"op_send_bytes":17122665,"op_resend":0,"op_reply":67,"op_latency":{"avgcount":67,"sum":0.423995355,"avgtime":0.006328288},"op_inflight":0,"oplen_avg":{"avgcount":67,"sum":122},"op":67,"op_r":36,"op_w":31,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":9,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":16,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":252,"max":104857600,"get_started":0,"get":7658,"get_sum":2166682,"get_or_fail_fail":0,"get_or_fail_success":7658,"take":0,"take_sum":0,"put":7657,"put_sum":2166430,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":67,"take_sum":17124586,"put":67,"put_sum":17124586,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":67,"take_sum":67,"put":67,"put_sum":67,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":219167,"max":3758096384,"get_started":0,"get":7697,"get_sum":16994198,"get_or_fail_fail":0,"get_or_fail_success":7697,"take":0,"take_sum":0,"put":11,"put_sum":16775031,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:13.786 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 15337, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 24, "msgr_running_fast_dispatch_time": 0.003342585, "msgr_running_recv_time": 0.006848764, "msgr_running_send_time": 0.015489089, "msgr_running_total_time": 0.029050487, "msgr_send_bytes": 4223497, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 24, "msgr_send_messages_queue_lat": { "avgcount": 23, "avgtime": 0.000588961, "sum": 0.013546119 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 16594, "avgtime": 5.61e-07, "sum": 0.009317953 }, "msgr_recv_bytes": 2726465, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7600, "msgr_running_fast_dispatch_time": 0.009644117, "msgr_running_recv_time": 1.140504232, "msgr_running_send_time": 1.208553518, "msgr_running_total_time": 1.807885893, "msgr_send_bytes": 16741987, "msgr_send_encrypted_bytes": 60800, "msgr_send_messages": 13970, "msgr_send_messages_queue_lat": { "avgcount": 13969, "avgtime": 6.2889e-05, "sum": 0.878498044 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 117551, "msgr_recv_encrypted_bytes": 106000, "msgr_recv_messages": 41, "msgr_running_fast_dispatch_time": 0.003210029, "msgr_running_recv_time": 0.009134573, "msgr_running_send_time": 0.021672446, "msgr_running_total_time": 0.032030252, "msgr_send_bytes": 1695056, "msgr_send_encrypted_bytes": 6032, "msgr_send_messages": 37, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.000357014, "sum": 0.012852509 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.065023719, "sum": 4.811755206 }, "queue_len": 914 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.0763e-05, "sum": 0.000123054 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3636, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3499, "handle_client_caps_dirty": 2996, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4058, "inodes_bottom": 4058, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4058, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 856, "q": 0, "reply": 4056, "reply_latency": { "avgcount": 4056, "avgtime": 0.209473069, "sum": 849.622768386 }, "request": 4057, "root_rbytes": 23306240, "root_rfiles": 3200, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4059, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4059, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 847, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 847, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7702, "evadd": 7697, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.030457124, "sum": 0.335028372 }, "rdpos": 4195674, "replayed": 5, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 21190687 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4058, "dn+": 4058, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 98248 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4058, "dispatch_server_request": 0, "handle_client_request": 4057, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 847, "avgtime": 0.00133215, "sum": 1.128331215 } }, "mds_sessions": { "average_load": 3848, "avg_session_uptime": 950, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3848 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13299093, "buffer_anon_items": 7946, "buffer_meta_bytes": 80872, "buffer_meta_items": 919, "mds_co_bytes": 11071609, "mds_co_items": 112132, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 16, "op": 67, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 67, "avgtime": 0.006328288, "sum": 0.423995355 }, "op_pg": 0, "op_r": 36, "op_reply": 67, "op_resend": 0, "op_rmw": 0, "op_send": 67, "op_send_bytes": 17122665, "op_w": 31, "oplen_avg": { "avgcount": 67, "sum": 122 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 9, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7658, "get_or_fail_fail": 0, "get_or_fail_success": 7658, "get_started": 0, "get_sum": 2166682, "max": 104857600, "put": 7657, "put_sum": 2166430, "take": 0, "take_sum": 0, "val": 252, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 67, "put_sum": 17124586, "take": 67, "take_sum": 17124586, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 67, "put_sum": 67, "take": 67, "take_sum": 67, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 7697, "get_or_fail_fail": 0, "get_or_fail_success": 7697, "get_started": 0, "get_sum": 16994198, "max": 3758096384, "put": 11, "put_sum": 16775031, "take": 0, "take_sum": 0, "val": 219167, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:13.787 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/847, 0/847 2024-03-06T07:33:14.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:15.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.028+0000 7fea7ca00640 1 Processor -- start 2024-03-06T07:33:15.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.028+0000 7fea7ca00640 1 -- start start 2024-03-06T07:33:15.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.030+0000 7fea7ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fea780d27e0 0x7fea780b5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.030+0000 7fea7ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 0x7fea780b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.030+0000 7fea7ca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea780d08f0 con 0x7fea781617a0 2024-03-06T07:33:15.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.030+0000 7fea7ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea780d1220 con 0x7fea780d27e0 2024-03-06T07:33:15.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.030+0000 7fea7ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea780d0bd0 con 0x7fea780d2ce0 2024-03-06T07:33:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.031+0000 7fea77e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 0x7fea780b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.031+0000 7fea76a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fea780d27e0 0x7fea780b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.031+0000 7fea77e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 0x7fea780b3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:37508/0 (socket says 172.21.15.6:37508) 2024-03-06T07:33:15.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.031+0000 7fea77e00640 1 -- 172.21.15.6:0/1545948257 learned_addr learned my addr 172.21.15.6:0/1545948257 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.032+0000 7fea77e00640 1 -- 172.21.15.6:0/1545948257 >> v2:172.21.15.178:3300/0 conn(0x7fea780d27e0 msgr2=0x7fea780b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.032+0000 7fea77e00640 1 --2- 172.21.15.6:0/1545948257 >> v2:172.21.15.178:3300/0 conn(0x7fea780d27e0 0x7fea780b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.032+0000 7fea77e00640 1 -- 172.21.15.6:0/1545948257 >> v1:172.21.15.178:6789/0 conn(0x7fea781617a0 legacy=0x7fea780d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.032+0000 7fea77e00640 1 -- 172.21.15.6:0/1545948257 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea780d0eb0 con 0x7fea780d2ce0 2024-03-06T07:33:15.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.033+0000 7fea77e00640 1 --2- 172.21.15.6:0/1545948257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 0x7fea780b3800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea70009e50 tx=0x7fea7002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b7a03fa414f6848 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.034+0000 7fea76000640 1 -- 172.21.15.6:0/1545948257 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7002cb00 con 0x7fea780d2ce0 2024-03-06T07:33:15.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.034+0000 7fea76000640 1 -- 172.21.15.6:0/1545948257 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7002cca0 con 0x7fea780d2ce0 2024-03-06T07:33:15.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.034+0000 7fea76000640 1 -- 172.21.15.6:0/1545948257 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea70034610 con 0x7fea780d2ce0 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 -- 172.21.15.6:0/1545948257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 msgr2=0x7fea780b3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1545948257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 0x7fea780b3800 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea70009e50 tx=0x7fea7002bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 -- 172.21.15.6:0/1545948257 shutdown_connections 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1545948257 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea780d2ce0 0x7fea780b3800 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1545948257 >> v2:172.21.15.178:3300/0 conn(0x7fea780d27e0 0x7fea780b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 -- 172.21.15.6:0/1545948257 >> 172.21.15.6:0/1545948257 conn(0x7fea7809d1d0 msgr2=0x7fea780d3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 -- 172.21.15.6:0/1545948257 shutdown_connections 2024-03-06T07:33:15.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.035+0000 7fea7ca00640 1 -- 172.21.15.6:0/1545948257 wait complete. 2024-03-06T07:33:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.037+0000 7fea7ca00640 1 Processor -- start 2024-03-06T07:33:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.037+0000 7fea7ca00640 1 -- start start 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.038+0000 7fea7ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea780d27e0 0x7fea78154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.038+0000 7fea7ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 0x7fea780b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.038+0000 7fea7ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea781617a0 0x7fea780b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.038+0000 7fea7ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea780d1830 con 0x7fea780d2ce0 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.038+0000 7fea7ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea780d16b0 con 0x7fea780d27e0 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.038+0000 7fea7ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea780b2c50 con 0x7fea781617a0 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 0x7fea780b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 0x7fea780b1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46162/0 (socket says 172.21.15.6:46162) 2024-03-06T07:33:15.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 -- 172.21.15.6:0/1059257488 learned_addr learned my addr 172.21.15.6:0/1059257488 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:15.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea77400640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea780d27e0 0x7fea78154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea77e00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea781617a0 0x7fea780b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 -- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea781617a0 msgr2=0x7fea780b72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea781617a0 0x7fea780b72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 -- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea780d27e0 msgr2=0x7fea78154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea780d27e0 0x7fea78154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.039+0000 7fea76a00640 1 -- 172.21.15.6:0/1059257488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea780b2260 con 0x7fea780d2ce0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.040+0000 7fea76a00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 0x7fea780b1800 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fea6400e870 tx=0x7fea6400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.040+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64017070 con 0x7fea780d2ce0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.040+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6400c940 con 0x7fea780d2ce0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.040+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6400cd30 con 0x7fea780d2ce0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.040+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea780b2540 con 0x7fea780d2ce0 2024-03-06T07:33:15.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.040+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea780c8e90 con 0x7fea780d2ce0 2024-03-06T07:33:15.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.042+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7804f330 con 0x7fea780d2ce0 2024-03-06T07:33:15.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.044+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea6401a070 con 0x7fea780d2ce0 2024-03-06T07:33:15.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.045+0000 7fea6f400640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea5806b4d0 0x7fea5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:15.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.045+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fea6408a4a0 con 0x7fea780d2ce0 2024-03-06T07:33:15.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.046+0000 7fea77400640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea5806b4d0 0x7fea5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:15.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.047+0000 7fea77400640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea5806b4d0 0x7fea5806d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fea68001640 tx=0x7fea68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:15.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.055+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea64059300 con 0x7fea780d2ce0 2024-03-06T07:33:15.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.452+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea7808c0f0 con 0x7fea780d2ce0 2024-03-06T07:33:15.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.453+0000 7fea6f400640 1 -- 172.21.15.6:0/1059257488 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fea64058c00 con 0x7fea780d2ce0 2024-03-06T07:33:15.455 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:15.455 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea5806b4d0 msgr2=0x7fea5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea5806b4d0 0x7fea5806d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fea68001640 tx=0x7fea68009040 comp rx=0 tx=0).stop 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 msgr2=0x7fea780b1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 0x7fea780b1800 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fea6400e870 tx=0x7fea6400ed30 comp rx=0 tx=0).stop 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 shutdown_connections 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fea5806b4d0 0x7fea5806d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea781617a0 0x7fea780b72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea780d2ce0 0x7fea780b1800 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 --2- 172.21.15.6:0/1059257488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea780d27e0 0x7fea78154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:15.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 >> 172.21.15.6:0/1059257488 conn(0x7fea7809d1d0 msgr2=0x7fea780d0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:15.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 shutdown_connections 2024-03-06T07:33:15.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:15.457+0000 7fea7ca00640 1 -- 172.21.15.6:0/1059257488 wait complete. 2024-03-06T07:33:15.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:15.711 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":25,"msgr_send_messages":25,"msgr_recv_bytes":15583,"msgr_send_bytes":4223815,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.029153358,"msgr_running_send_time":0.015544074,"msgr_running_recv_time":0.006866296,"msgr_running_fast_dispatch_time":0.003362801,"msgr_send_messages_queue_lat":{"avgcount":24,"sum":0.013577982,"avgtime":0.000565749},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8264,"msgr_send_messages":15957,"msgr_recv_bytes":2957517,"msgr_send_bytes":18771576,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.061132697,"msgr_running_send_time":1.362245040,"msgr_running_recv_time":1.257670689,"msgr_running_fast_dispatch_time":0.010914353,"msgr_send_messages_queue_lat":{"avgcount":15956,"sum":0.977099457,"avgtime":0.000061237},"msgr_handle_ack_lat":{"avgcount":18556,"sum":0.010159048,"avgtime":0.000000547},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":62592},"AsyncMessenger::Worker-2":{"msgr_recv_messages":44,"msgr_send_messages":40,"msgr_recv_bytes":118315,"msgr_send_bytes":1696406,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.032481136,"msgr_running_send_time":0.021884172,"msgr_running_recv_time":0.009226169,"msgr_running_fast_dispatch_time":0.003301744,"msgr_send_messages_queue_lat":{"avgcount":39,"sum":0.013043219,"avgtime":0.000334441},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106272,"msgr_send_encrypted_bytes":6656},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":916,"complete_latency":{"avgcount":74,"sum":4.811755206,"avgtime":0.065023719}},"finisher-PurgeQueue":{"queue_len":91,"complete_latency":{"avgcount":12,"sum":0.000366131,"avgtime":0.000030510}},"mds":{"request":4707,"reply":4707,"reply_latency":{"avgcount":4707,"sum":851.305464061,"avgtime":0.180859456},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4709,"inodes_top":0,"inodes_bottom":4709,"inodes_pin_tail":0,"inodes_pinned":4709,"inodes_expired":0,"inodes_with_caps":3032,"caps":3032,"subtrees":2,"traverse":4709,"traverse_hit":4709,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2924,"root_rbytes":22495232,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":1,"ceph_cap_op_grant":4299,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3502,"handle_client_caps_dirty":2999,"handle_client_cap_release":1,"process_request_cap_release":1507},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1498,"num_strays_delayed":0,"num_strays_enqueuing":170,"strays_created":1498,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":8456,"evex":0,"evtrm":0,"ev":8461,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23491437,"rdpos":4195674,"jlat":{"avgcount":11,"sum":0.335028372,"avgtime":0.030457124},"replayed":5},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4709,"dn+":4709,"dn-":0,"cap":3032,"cap+":3202,"cap-":170,"rss":98248,"heap":248164},"mds_server":{"dispatch_client_request":4709,"dispatch_server_request":0,"handle_client_request":4708,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1498,"sum":2.811026890,"avgtime":0.001876519},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4408,"average_load":4408,"avg_session_uptime":955,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8550626,"buffer_anon_items":6096,"buffer_meta_bytes":1760,"buffer_meta_items":20,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14663613,"mds_co_items":183971,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":2,"op_laggy":0,"op_send":79,"op_send_bytes":18477711,"op_resend":0,"op_reply":77,"op_latency":{"avgcount":77,"sum":0.469994857,"avgtime":0.006103829},"op_inflight":0,"oplen_avg":{"avgcount":79,"sum":135},"op":79,"op_r":37,"op_w":42,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":9,"osdop_write":14,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":15,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":8326,"get_sum":2338572,"get_or_fail_fail":0,"get_or_fail_success":8326,"take":0,"take_sum":0,"put":8325,"put_sum":2338319,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":13938,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":79,"take_sum":18488823,"put":77,"put_sum":18474885,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":2,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":79,"take_sum":79,"put":77,"put_sum":77,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":3232,"max":3758096384,"get_started":0,"get":170,"get_sum":17170,"get_or_fail_fail":0,"get_or_fail_success":170,"take":0,"take_sum":0,"put":2,"put_sum":13938,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":1195613,"max":3758096384,"get_started":0,"get":8456,"get_sum":19294948,"get_or_fail_fail":0,"get_or_fail_success":8456,"take":0,"take_sum":0,"put":12,"put_sum":18099335,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:15.713 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 15583, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 25, "msgr_running_fast_dispatch_time": 0.003362801, "msgr_running_recv_time": 0.006866296, "msgr_running_send_time": 0.015544074, "msgr_running_total_time": 0.029153358, "msgr_send_bytes": 4223815, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 25, "msgr_send_messages_queue_lat": { "avgcount": 24, "avgtime": 0.000565749, "sum": 0.013577982 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18556, "avgtime": 5.47e-07, "sum": 0.010159048 }, "msgr_recv_bytes": 2957517, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 8264, "msgr_running_fast_dispatch_time": 0.010914353, "msgr_running_recv_time": 1.257670689, "msgr_running_send_time": 1.36224504, "msgr_running_total_time": 2.061132697, "msgr_send_bytes": 18771576, "msgr_send_encrypted_bytes": 62592, "msgr_send_messages": 15957, "msgr_send_messages_queue_lat": { "avgcount": 15956, "avgtime": 6.1237e-05, "sum": 0.977099457 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 118315, "msgr_recv_encrypted_bytes": 106272, "msgr_recv_messages": 44, "msgr_running_fast_dispatch_time": 0.003301744, "msgr_running_recv_time": 0.009226169, "msgr_running_send_time": 0.021884172, "msgr_running_total_time": 0.032481136, "msgr_send_bytes": 1696406, "msgr_send_encrypted_bytes": 6656, "msgr_send_messages": 40, "msgr_send_messages_queue_lat": { "avgcount": 39, "avgtime": 0.000334441, "sum": 0.013043219 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.065023719, "sum": 4.811755206 }, "queue_len": 916 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 12, "avgtime": 3.051e-05, "sum": 0.000366131 }, "queue_len": 91 }, "mds": { "caps": 3032, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4299, "ceph_cap_op_revoke": 1, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 3502, "handle_client_caps_dirty": 2999, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4709, "inodes_bottom": 4709, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4709, "inodes_top": 0, "inodes_with_caps": 3032, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1507, "q": 0, "reply": 4707, "reply_latency": { "avgcount": 4707, "avgtime": 0.180859456, "sum": 851.305464061 }, "request": 4707, "root_rbytes": 22495232, "root_rfiles": 2924, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4709, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4709, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1498, "num_strays_delayed": 0, "num_strays_enqueuing": 170, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1498, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 8461, "evadd": 8456, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.030457124, "sum": 0.335028372 }, "rdpos": 4195674, "replayed": 5, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23491437 }, "mds_mem": { "cap": 3032, "cap+": 3202, "cap-": 170, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4709, "dn+": 4709, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 98248 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4709, "dispatch_server_request": 0, "handle_client_request": 4708, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1498, "avgtime": 0.001876519, "sum": 2.81102689 } }, "mds_sessions": { "average_load": 4408, "avg_session_uptime": 955, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4408 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8550626, "buffer_anon_items": 6096, "buffer_meta_bytes": 1760, "buffer_meta_items": 20, "mds_co_bytes": 14663613, "mds_co_items": 183971, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 18, "op": 79, "op_Infinitylight": 0, "op_active": 2, "op_laggy": 0, "op_latency": { "avgcount": 77, "avgtime": 0.006103829, "sum": 0.469994857 }, "op_pg": 0, "op_r": 37, "op_reply": 77, "op_resend": 0, "op_rmw": 0, "op_send": 79, "op_send_bytes": 18477711, "op_w": 42, "oplen_avg": { "avgcount": 79, "sum": 135 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 15, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 9, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 14, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8326, "get_or_fail_fail": 0, "get_or_fail_success": 8326, "get_started": 0, "get_sum": 2338572, "max": 104857600, "put": 8325, "put_sum": 2338319, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 77, "put_sum": 18474885, "take": 79, "take_sum": 18488823, "val": 13938, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 77, "put_sum": 77, "take": 79, "take_sum": 79, "val": 2, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 170, "get_or_fail_fail": 0, "get_or_fail_success": 170, "get_started": 0, "get_sum": 17170, "max": 3758096384, "put": 2, "put_sum": 13938, "take": 0, "take_sum": 0, "val": 3232, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 8456, "get_or_fail_fail": 0, "get_or_fail_success": 8456, "get_started": 0, "get_sum": 19294948, "max": 3758096384, "put": 12, "put_sum": 18099335, "take": 0, "take_sum": 0, "val": 1195613, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:15.713 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1498, 0/1498 2024-03-06T07:33:16.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:16.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.932+0000 7f30c5200640 1 Processor -- start 2024-03-06T07:33:16.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.932+0000 7f30c5200640 1 -- start start 2024-03-06T07:33:16.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30c5200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f30c00b7d90 0x7f30c00b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30c5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f30c00b5fc0 0x7f30c00ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30c5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00b6410 con 0x7f30c01615c0 2024-03-06T07:33:16.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30c5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c00d2330 con 0x7f30c00b5fc0 2024-03-06T07:33:16.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30c5200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c00b66c0 con 0x7f30c00b7d90 2024-03-06T07:33:16.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30bf400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f30c01615c0 0x7f30c00b58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41232/0 (socket says 172.21.15.6:41232) 2024-03-06T07:33:16.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30bf400640 1 -- 172.21.15.6:0/838000694 learned_addr learned my addr 172.21.15.6:0/838000694 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:16.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.933+0000 7f30bfe00640 1 --2- 172.21.15.6:0/838000694 >> v2:172.21.15.6:3300/0 conn(0x7f30c00b5fc0 0x7f30c00ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30bea00640 1 --2- 172.21.15.6:0/838000694 >> v2:172.21.15.178:3300/0 conn(0x7f30c00b7d90 0x7f30c00b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30bfe00640 1 -- 172.21.15.6:0/838000694 >> v2:172.21.15.178:3300/0 conn(0x7f30c00b7d90 msgr2=0x7f30c00b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30bfe00640 1 --2- 172.21.15.6:0/838000694 >> v2:172.21.15.178:3300/0 conn(0x7f30c00b7d90 0x7f30c00b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30bfe00640 1 -- 172.21.15.6:0/838000694 >> v1:172.21.15.6:6789/0 conn(0x7f30c01615c0 legacy=0x7f30c00b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30bfe00640 1 -- 172.21.15.6:0/838000694 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c00d2590 con 0x7f30c00b5fc0 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30be000640 1 -- 172.21.15.6:0/838000694 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736740976 0 0) 0x7f30c00b6410 con 0x7f30c01615c0 2024-03-06T07:33:16.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.934+0000 7f30bea00640 1 --2- 172.21.15.6:0/838000694 >> v2:172.21.15.178:3300/0 conn(0x7f30c00b7d90 0x7f30c00b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.935+0000 7f30bfe00640 1 --2- 172.21.15.6:0/838000694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c00ce940 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f30c008cd80 tx=0x7f30b802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84decf38ec7423aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30be000640 1 -- 172.21.15.6:0/838000694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8039070 con 0x7f30c00b5fc0 2024-03-06T07:33:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30be000640 1 -- 172.21.15.6:0/838000694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b802cbe0 con 0x7f30c00b5fc0 2024-03-06T07:33:16.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30be000640 1 -- 172.21.15.6:0/838000694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8034440 con 0x7f30c00b5fc0 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 -- 172.21.15.6:0/838000694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 msgr2=0x7f30c00ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 --2- 172.21.15.6:0/838000694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c00ce940 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f30c008cd80 tx=0x7f30b802be30 comp rx=0 tx=0).stop 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 -- 172.21.15.6:0/838000694 shutdown_connections 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 --2- 172.21.15.6:0/838000694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c00ce940 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 --2- 172.21.15.6:0/838000694 >> v2:172.21.15.178:3300/0 conn(0x7f30c00b7d90 0x7f30c00b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 -- 172.21.15.6:0/838000694 >> 172.21.15.6:0/838000694 conn(0x7f30c00d3e30 msgr2=0x7f30c00d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.936+0000 7f30c5200640 1 -- 172.21.15.6:0/838000694 shutdown_connections 2024-03-06T07:33:16.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.937+0000 7f30c5200640 1 -- 172.21.15.6:0/838000694 wait complete. 2024-03-06T07:33:16.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.939+0000 7f30c5200640 1 Processor -- start 2024-03-06T07:33:16.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.939+0000 7f30c5200640 1 -- start start 2024-03-06T07:33:16.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30c5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c01f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30c5200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30c00b7d90 0x7f30c006c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30c5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f30c01615c0 0x7f30c00b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30c5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c01619a0 con 0x7f30c00b5fc0 2024-03-06T07:33:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30c5200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00d2b50 con 0x7f30c00b7d90 2024-03-06T07:33:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30c5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0161b20 con 0x7f30c01615c0 2024-03-06T07:33:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.940+0000 7f30bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c01f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c01f30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46180/0 (socket says 172.21.15.6:46180) 2024-03-06T07:33:16.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 -- 172.21.15.6:0/1741731511 learned_addr learned my addr 172.21.15.6:0/1741731511 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:16.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bea00640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30c00b7d90 0x7f30c006c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bfe00640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f30c01615c0 0x7f30c00b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 -- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f30c01615c0 msgr2=0x7f30c00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f30c01615c0 0x7f30c00b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 -- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30c00b7d90 msgr2=0x7f30c006c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30c00b7d90 0x7f30c006c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 -- 172.21.15.6:0/1741731511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c006cf40 con 0x7f30c00b5fc0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.941+0000 7f30bf400640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c01f30d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c4a0 tx=0x7f30b000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.942+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0018070 con 0x7f30c00b5fc0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.942+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b00074e0 con 0x7f30c00b5fc0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.942+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b0004a20 con 0x7f30c00b5fc0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.942+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c006d220 con 0x7f30c00b5fc0 2024-03-06T07:33:16.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.942+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c00b1860 con 0x7f30c00b5fc0 2024-03-06T07:33:16.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.945+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f30b001b070 con 0x7f30c00b5fc0 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.946+0000 7f30b7400640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f30a006b520 0x7f30a006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.946+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f30b00895c0 con 0x7f30c00b5fc0 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.946+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c004f330 con 0x7f30c00b5fc0 2024-03-06T07:33:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.947+0000 7f30bea00640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f30a006b520 0x7f30a006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:16.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.949+0000 7f30bea00640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f30a006b520 0x7f30a006d9e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f30c01f7ef0 tx=0x7f30ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:16.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:16.953+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f30b0058500 con 0x7f30c00b5fc0 2024-03-06T07:33:17.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.379+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c0160c90 con 0x7f30c00b5fc0 2024-03-06T07:33:17.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.380+0000 7f30b7400640 1 -- 172.21.15.6:0/1741731511 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f30b0057e00 con 0x7f30c00b5fc0 2024-03-06T07:33:17.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:17.382 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:17.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f30a006b520 msgr2=0x7f30a006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f30a006b520 0x7f30a006d9e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f30c01f7ef0 tx=0x7f30ac008040 comp rx=0 tx=0).stop 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 msgr2=0x7f30c01f30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c01f30d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f30b000c4a0 tx=0x7f30b000c960 comp rx=0 tx=0).stop 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 shutdown_connections 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f30a006b520 0x7f30a006d9e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f30c00b7d90 0x7f30c006c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f30c01615c0 0x7f30c00b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 --2- 172.21.15.6:0/1741731511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f30c00b5fc0 0x7f30c01f30d0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.383+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 >> 172.21.15.6:0/1741731511 conn(0x7f30c00d3e30 msgr2=0x7f30c00d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.384+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 shutdown_connections 2024-03-06T07:33:17.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:17.384+0000 7f30c5200640 1 -- 172.21.15.6:0/1741731511 wait complete. 2024-03-06T07:33:17.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:17.630 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":219,"msgr_send_messages":219,"msgr_recv_bytes":64851,"msgr_send_bytes":4287051,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.055739503,"msgr_running_send_time":0.029345457,"msgr_running_recv_time":0.012690412,"msgr_running_fast_dispatch_time":0.008838200,"msgr_send_messages_queue_lat":{"avgcount":218,"sum":0.045784307,"avgtime":0.000210019},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10635,"msgr_send_messages":18464,"msgr_recv_bytes":3790725,"msgr_send_bytes":22412767,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.347794632,"msgr_running_send_time":1.552023526,"msgr_running_recv_time":1.566150782,"msgr_running_fast_dispatch_time":0.017372722,"msgr_send_messages_queue_lat":{"avgcount":18463,"sum":1.310870940,"avgtime":0.000070999},"msgr_handle_ack_lat":{"avgcount":22182,"sum":0.011180241,"avgtime":0.000000504},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":62592},"AsyncMessenger::Worker-2":{"msgr_recv_messages":157,"msgr_send_messages":153,"msgr_recv_bytes":146993,"msgr_send_bytes":1779418,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.046616998,"msgr_running_send_time":0.027399850,"msgr_running_recv_time":0.012679891,"msgr_running_fast_dispatch_time":0.007277018,"msgr_send_messages_queue_lat":{"avgcount":152,"sum":0.025991659,"avgtime":0.000170997},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106272,"msgr_send_encrypted_bytes":6656},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1026,"complete_latency":{"avgcount":79,"sum":7.797815440,"avgtime":0.098706524}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":746,"sum":0.680747902,"avgtime":0.000912530}},"mds":{"request":4861,"reply":4861,"reply_latency":{"avgcount":4861,"sum":852.201001823,"avgtime":0.175313927},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4693,"inodes_top":0,"inodes_bottom":4693,"inodes_pin_tail":0,"inodes_pinned":4693,"inodes_expired":0,"inodes_with_caps":2509,"caps":2509,"subtrees":2,"traverse":4863,"traverse_hit":4863,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2299,"root_rbytes":17729536,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":26,"ceph_cap_op_grant":5543,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4802,"handle_client_caps_dirty":3000,"handle_client_cap_release":7,"process_request_cap_release":1660},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1222,"num_strays_delayed":105,"num_strays_enqueuing":0,"strays_created":1652,"strays_enqueued":430,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10171,"evex":0,"evtrm":0,"ev":10176,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26527256,"rdpos":4195674,"jlat":{"avgcount":13,"sum":1.341057389,"avgtime":0.103158260},"replayed":5},"mds_mem":{"ino":3044,"ino+":3214,"ino-":170,"dir":13,"dir+":13,"dir-":0,"dn":4693,"dn+":4863,"dn-":170,"cap":2407,"cap+":3202,"cap-":795,"rss":99144,"heap":248164},"mds_server":{"dispatch_client_request":4862,"dispatch_server_request":0,"handle_client_request":4861,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1652,"sum":3.706564652,"avgtime":0.002243683},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4511,"average_load":4511,"avg_session_uptime":955,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8550626,"buffer_anon_items":6096,"buffer_meta_bytes":1760,"buffer_meta_items":20,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14663613,"mds_co_items":183971,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":525,"op_send_bytes":22034237,"op_resend":0,"op_reply":525,"op_latency":{"avgcount":525,"sum":4.233953611,"avgtime":0.008064673},"op_inflight":0,"oplen_avg":{"avgcount":525,"sum":583},"op":525,"op_r":42,"op_w":483,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":14,"osdop_write":21,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":446,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":20,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":430,"pq_executed":430,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":150417,"max":104857600,"get_started":0,"get":11004,"get_sum":3008706,"get_or_fail_fail":0,"get_or_fail_success":11004,"take":0,"take_sum":0,"put":10235,"put_sum":2858289,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":525,"take_sum":22079588,"put":525,"put_sum":22079588,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":525,"take_sum":525,"put":525,"put_sum":525,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":430,"get_sum":43430,"get_or_fail_fail":0,"get_or_fail_success":430,"take":0,"take_sum":0,"put":6,"put_sum":43430,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":751879,"max":3758096384,"get_started":0,"get":10171,"get_sum":22330767,"get_or_fail_fail":0,"get_or_fail_success":10171,"take":0,"take_sum":0,"put":15,"put_sum":21578888,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:17.631 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 64851, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 219, "msgr_running_fast_dispatch_time": 0.0088382, "msgr_running_recv_time": 0.012690412, "msgr_running_send_time": 0.029345457, "msgr_running_total_time": 0.055739503, "msgr_send_bytes": 4287051, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 219, "msgr_send_messages_queue_lat": { "avgcount": 218, "avgtime": 0.000210019, "sum": 0.045784307 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22182, "avgtime": 5.04e-07, "sum": 0.011180241 }, "msgr_recv_bytes": 3790725, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10635, "msgr_running_fast_dispatch_time": 0.017372722, "msgr_running_recv_time": 1.566150782, "msgr_running_send_time": 1.552023526, "msgr_running_total_time": 2.347794632, "msgr_send_bytes": 22412767, "msgr_send_encrypted_bytes": 62592, "msgr_send_messages": 18464, "msgr_send_messages_queue_lat": { "avgcount": 18463, "avgtime": 7.0999e-05, "sum": 1.31087094 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 146993, "msgr_recv_encrypted_bytes": 106272, "msgr_recv_messages": 157, "msgr_running_fast_dispatch_time": 0.007277018, "msgr_running_recv_time": 0.012679891, "msgr_running_send_time": 0.02739985, "msgr_running_total_time": 0.046616998, "msgr_send_bytes": 1779418, "msgr_send_encrypted_bytes": 6656, "msgr_send_messages": 153, "msgr_send_messages_queue_lat": { "avgcount": 152, "avgtime": 0.000170997, "sum": 0.025991659 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 79, "avgtime": 0.098706524, "sum": 7.79781544 }, "queue_len": 1026 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 746, "avgtime": 0.00091253, "sum": 0.680747902 }, "queue_len": 0 }, "mds": { "caps": 2509, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5543, "ceph_cap_op_revoke": 26, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 7, "handle_client_caps": 4802, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4693, "inodes_bottom": 4693, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4693, "inodes_top": 0, "inodes_with_caps": 2509, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1660, "q": 0, "reply": 4861, "reply_latency": { "avgcount": 4861, "avgtime": 0.175313927, "sum": 852.201001823 }, "request": 4861, "root_rbytes": 17729536, "root_rfiles": 2299, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4863, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4863, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1222, "num_strays_delayed": 105, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1652, "strays_enqueued": 430, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10176, "evadd": 10171, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 13, "avgtime": 0.10315826, "sum": 1.341057389 }, "rdpos": 4195674, "replayed": 5, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26527256 }, "mds_mem": { "cap": 2407, "cap+": 3202, "cap-": 795, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4693, "dn+": 4863, "dn-": 170, "heap": 248164, "ino": 3044, "ino+": 3214, "ino-": 170, "rss": 99144 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4862, "dispatch_server_request": 0, "handle_client_request": 4861, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1652, "avgtime": 0.002243683, "sum": 3.706564652 } }, "mds_sessions": { "average_load": 4511, "avg_session_uptime": 955, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4511 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8550626, "buffer_anon_items": 6096, "buffer_meta_bytes": 1760, "buffer_meta_items": 20, "mds_co_bytes": 14663613, "mds_co_items": 183971, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 20, "op": 525, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 525, "avgtime": 0.008064673, "sum": 4.233953611 }, "op_pg": 0, "op_r": 42, "op_reply": 525, "op_resend": 0, "op_rmw": 0, "op_send": 525, "op_send_bytes": 22034237, "op_w": 483, "oplen_avg": { "avgcount": 525, "sum": 583 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 446, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 14, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 21, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 430, "pq_executed_ops": 430, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11004, "get_or_fail_fail": 0, "get_or_fail_success": 11004, "get_started": 0, "get_sum": 3008706, "max": 104857600, "put": 10235, "put_sum": 2858289, "take": 0, "take_sum": 0, "val": 150417, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 525, "put_sum": 22079588, "take": 525, "take_sum": 22079588, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 525, "put_sum": 525, "take": 525, "take_sum": 525, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 430, "get_or_fail_fail": 0, "get_or_fail_success": 430, "get_started": 0, "get_sum": 43430, "max": 3758096384, "put": 6, "put_sum": 43430, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 10171, "get_or_fail_fail": 0, "get_or_fail_success": 10171, "get_started": 0, "get_sum": 22330767, "max": 3758096384, "put": 15, "put_sum": 21578888, "take": 0, "take_sum": 0, "val": 751879, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:17.631 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1222, 430/1652 2024-03-06T07:33:18.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:18.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.863+0000 7fcc8fe00640 1 Processor -- start 2024-03-06T07:33:18.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.863+0000 7fcc8fe00640 1 -- start start 2024-03-06T07:33:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.865+0000 7fcc8fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcc901617a0 0x7fcc900b5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.865+0000 7fcc8fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc900d2ce0 0x7fcc900d20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.865+0000 7fcc8fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcc900d2610 0x7fcc90161eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.865+0000 7fcc8fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc900d1220 con 0x7fcc900d2610 2024-03-06T07:33:18.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.865+0000 7fcc8fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc900d08f0 con 0x7fcc901617a0 2024-03-06T07:33:18.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.865+0000 7fcc8fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900d0bd0 con 0x7fcc900d2ce0 2024-03-06T07:33:18.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcc900d2610 0x7fcc90161eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcc900d2610 0x7fcc90161eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46188/0 (socket says 172.21.15.6:46188) 2024-03-06T07:33:18.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 -- 172.21.15.6:0/2086557437 learned_addr learned my addr 172.21.15.6:0/2086557437 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8ea00640 1 --2- 172.21.15.6:0/2086557437 >> v2:172.21.15.178:3300/0 conn(0x7fcc901617a0 0x7fcc900b5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8e000640 1 --2- 172.21.15.6:0/2086557437 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc900d2ce0 0x7fcc900d20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 -- 172.21.15.6:0/2086557437 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc900d2ce0 msgr2=0x7fcc900d20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 --2- 172.21.15.6:0/2086557437 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc900d2ce0 0x7fcc900d20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 -- 172.21.15.6:0/2086557437 >> v2:172.21.15.178:3300/0 conn(0x7fcc901617a0 msgr2=0x7fcc900b5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 --2- 172.21.15.6:0/2086557437 >> v2:172.21.15.178:3300/0 conn(0x7fcc901617a0 0x7fcc900b5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.866+0000 7fcc8f400640 1 -- 172.21.15.6:0/2086557437 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900d0eb0 con 0x7fcc900d2610 2024-03-06T07:33:18.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.868+0000 7fcc8f400640 1 --2- 172.21.15.6:0/2086557437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc90161eb0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcc900b6a60 tx=0x7fcc8802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5586b55ca7c02090 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.868+0000 7fcc8d600640 1 -- 172.21.15.6:0/2086557437 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8802ca70 con 0x7fcc900d2610 2024-03-06T07:33:18.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.868+0000 7fcc8d600640 1 -- 172.21.15.6:0/2086557437 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8802cc10 con 0x7fcc900d2610 2024-03-06T07:33:18.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.868+0000 7fcc8d600640 1 -- 172.21.15.6:0/2086557437 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc880343f0 con 0x7fcc900d2610 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 -- 172.21.15.6:0/2086557437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 msgr2=0x7fcc90161eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/2086557437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc90161eb0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fcc900b6a60 tx=0x7fcc8802bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 -- 172.21.15.6:0/2086557437 shutdown_connections 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/2086557437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc90161eb0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/2086557437 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc900d2ce0 0x7fcc900d20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/2086557437 >> v2:172.21.15.178:3300/0 conn(0x7fcc901617a0 0x7fcc900b5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 -- 172.21.15.6:0/2086557437 >> 172.21.15.6:0/2086557437 conn(0x7fcc9009d1d0 msgr2=0x7fcc900d3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 -- 172.21.15.6:0/2086557437 shutdown_connections 2024-03-06T07:33:18.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.869+0000 7fcc8fe00640 1 -- 172.21.15.6:0/2086557437 wait complete. 2024-03-06T07:33:18.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.872+0000 7fcc8fe00640 1 Processor -- start 2024-03-06T07:33:18.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.872+0000 7fcc8fe00640 1 -- start start 2024-03-06T07:33:18.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.873+0000 7fcc8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc900c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.873+0000 7fcc8fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc900d2ce0 0x7fcc900b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.873+0000 7fcc8fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc901617a0 0x7fcc90158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.873+0000 7fcc8fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900d1830 con 0x7fcc900d2610 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.873+0000 7fcc8fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900d16b0 con 0x7fcc900d2ce0 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.873+0000 7fcc8fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc900b2c50 con 0x7fcc901617a0 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc900c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc900c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46192/0 (socket says 172.21.15.6:46192) 2024-03-06T07:33:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 -- 172.21.15.6:0/1591092805 learned_addr learned my addr 172.21.15.6:0/1591092805 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8e000640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc900d2ce0 0x7fcc900b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 -- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc901617a0 msgr2=0x7fcc90158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc901617a0 0x7fcc90158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 -- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc900d2ce0 msgr2=0x7fcc900b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc900d2ce0 0x7fcc900b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.874+0000 7fcc8ea00640 1 -- 172.21.15.6:0/1591092805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc900c9280 con 0x7fcc900d2610 2024-03-06T07:33:18.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.875+0000 7fcc8ea00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc900c83a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000e9e0 tx=0x7fcc8000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.875+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc80017070 con 0x7fcc900d2610 2024-03-06T07:33:18.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.875+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8000c9d0 con 0x7fcc900d2610 2024-03-06T07:33:18.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.875+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc900c9560 con 0x7fcc900d2610 2024-03-06T07:33:18.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.875+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8000cdc0 con 0x7fcc900d2610 2024-03-06T07:33:18.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.875+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9015ddd0 con 0x7fcc900d2610 2024-03-06T07:33:18.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.877+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9004f330 con 0x7fcc900d2610 2024-03-06T07:33:18.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.877+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc8001a070 con 0x7fcc900d2610 2024-03-06T07:33:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.879+0000 7fcc86a00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc7006b4d0 0x7fcc7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.879+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fcc8008a790 con 0x7fcc900d2610 2024-03-06T07:33:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.880+0000 7fcc8e000640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc7006b4d0 0x7fcc7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.881+0000 7fcc8e000640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc7006b4d0 0x7fcc7006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcc900b57f0 tx=0x7fcc7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:18.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:18.886+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc800594c0 con 0x7fcc900d2610 2024-03-06T07:33:19.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.253+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc90160f50 con 0x7fcc900d2610 2024-03-06T07:33:19.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.254+0000 7fcc86a00640 1 -- 172.21.15.6:0/1591092805 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fcc80058dc0 con 0x7fcc900d2610 2024-03-06T07:33:19.257 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:19.257 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:19.257 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:19.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.258+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc7006b4d0 msgr2=0x7fcc7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.258+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc7006b4d0 0x7fcc7006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcc900b57f0 tx=0x7fcc7c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:19.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.258+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 msgr2=0x7fcc900c83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:19.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.258+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc900c83a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fcc8000e9e0 tx=0x7fcc8000eea0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.261+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 shutdown_connections 2024-03-06T07:33:19.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.261+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcc7006b4d0 0x7fcc7006d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.261+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc900d2ce0 0x7fcc900b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.261+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcc901617a0 0x7fcc90158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.261+0000 7fcc8fe00640 1 --2- 172.21.15.6:0/1591092805 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcc900d2610 0x7fcc900c83a0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.261+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 >> 172.21.15.6:0/1591092805 conn(0x7fcc9009d1d0 msgr2=0x7fcc900b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.262+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 shutdown_connections 2024-03-06T07:33:19.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:19.262+0000 7fcc8fe00640 1 -- 172.21.15.6:0/1591092805 wait complete. 2024-03-06T07:33:19.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:19.627 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":614,"msgr_send_messages":618,"msgr_recv_bytes":165173,"msgr_send_bytes":4417117,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.117597825,"msgr_running_send_time":0.061546098,"msgr_running_recv_time":0.024657863,"msgr_running_fast_dispatch_time":0.023264187,"msgr_send_messages_queue_lat":{"avgcount":617,"sum":0.091742782,"avgtime":0.000148691},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11716,"msgr_send_messages":19785,"msgr_recv_bytes":4211277,"msgr_send_bytes":23123670,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.634304443,"msgr_running_send_time":1.729077110,"msgr_running_recv_time":1.693865955,"msgr_running_fast_dispatch_time":0.028140813,"msgr_send_messages_queue_lat":{"avgcount":19784,"sum":1.491193907,"avgtime":0.000075373},"msgr_handle_ack_lat":{"avgcount":23952,"sum":0.011880123,"avgtime":0.000000495},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":62592},"AsyncMessenger::Worker-2":{"msgr_recv_messages":494,"msgr_send_messages":492,"msgr_recv_bytes":232585,"msgr_send_bytes":3988741,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.100031219,"msgr_running_send_time":0.054165966,"msgr_running_recv_time":0.022250957,"msgr_running_fast_dispatch_time":0.021324246,"msgr_send_messages_queue_lat":{"avgcount":489,"sum":0.064045576,"avgtime":0.000130972},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106544,"msgr_send_encrypted_bytes":7280},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1822,"complete_latency":{"avgcount":87,"sum":9.485362425,"avgtime":0.109027154}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2356,"sum":1.268721310,"avgtime":0.000538506}},"mds":{"request":5508,"reply":5508,"reply_latency":{"avgcount":5508,"sum":854.850393095,"avgtime":0.155201596},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4953,"inodes_top":0,"inodes_bottom":4953,"inodes_pin_tail":0,"inodes_pinned":4953,"inodes_expired":0,"inodes_with_caps":1550,"caps":1550,"subtrees":2,"traverse":5510,"traverse_hit":5510,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1548,"root_rbytes":11962368,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":38,"ceph_cap_op_grant":5831,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5247,"handle_client_caps_dirty":3000,"handle_client_cap_release":509,"process_request_cap_release":2307},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":763,"num_strays_delayed":116,"num_strays_enqueuing":0,"strays_created":2299,"strays_enqueued":1536,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":12736,"evex":0,"evtrm":0,"ev":12741,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":30558724,"rdpos":4195674,"jlat":{"avgcount":17,"sum":2.943599374,"avgtime":0.173152904},"replayed":5},"mds_mem":{"ino":2658,"ino+":3214,"ino-":556,"dir":13,"dir+":13,"dir-":0,"dn":4954,"dn+":5510,"dn-":556,"cap":1550,"cap+":3202,"cap-":1652,"rss":99144,"heap":248164},"mds_server":{"dispatch_client_request":5509,"dispatch_server_request":0,"handle_client_request":5508,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2299,"sum":6.355955924,"avgtime":0.002764661},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5002,"average_load":5002,"avg_session_uptime":955,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8550626,"buffer_anon_items":6096,"buffer_meta_bytes":1760,"buffer_meta_items":20,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14663613,"mds_co_items":183971,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":1515,"op_send_bytes":25772558,"op_resend":0,"op_reply":1509,"op_latency":{"avgcount":1509,"sum":7.550917258,"avgtime":0.005003921},"op_inflight":5,"oplen_avg":{"avgcount":1515,"sum":1577},"op":1515,"op_r":48,"op_w":1467,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":20,"osdop_write":29,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1419,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":24,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":1396,"pq_executed":1396,"pq_item_in_journal":140},"throttle-msgr_dispatch_throttler-mds":{"val":253,"max":104857600,"get_started":0,"get":12817,"get_sum":3451950,"get_or_fail_fail":0,"get_or_fail_success":12817,"take":0,"take_sum":0,"put":12816,"put_sum":3451697,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1515,"take_sum":25929615,"put":1509,"put_sum":25929615,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1515,"take_sum":1515,"put":1509,"put_sum":1509,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1536,"get_sum":155136,"get_or_fail_fail":0,"get_or_fail_success":1536,"take":0,"take_sum":0,"put":12,"put_sum":155136,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":1198596,"max":3758096384,"get_started":0,"get":12736,"get_sum":26362235,"get_or_fail_fail":0,"get_or_fail_success":12736,"take":0,"take_sum":0,"put":17,"put_sum":25163639,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:19.629 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 165173, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 614, "msgr_running_fast_dispatch_time": 0.023264187, "msgr_running_recv_time": 0.024657863, "msgr_running_send_time": 0.061546098, "msgr_running_total_time": 0.117597825, "msgr_send_bytes": 4417117, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 618, "msgr_send_messages_queue_lat": { "avgcount": 617, "avgtime": 0.000148691, "sum": 0.091742782 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 23952, "avgtime": 4.95e-07, "sum": 0.011880123 }, "msgr_recv_bytes": 4211277, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11716, "msgr_running_fast_dispatch_time": 0.028140813, "msgr_running_recv_time": 1.693865955, "msgr_running_send_time": 1.72907711, "msgr_running_total_time": 2.634304443, "msgr_send_bytes": 23123670, "msgr_send_encrypted_bytes": 62592, "msgr_send_messages": 19785, "msgr_send_messages_queue_lat": { "avgcount": 19784, "avgtime": 7.5373e-05, "sum": 1.491193907 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 232585, "msgr_recv_encrypted_bytes": 106544, "msgr_recv_messages": 494, "msgr_running_fast_dispatch_time": 0.021324246, "msgr_running_recv_time": 0.022250957, "msgr_running_send_time": 0.054165966, "msgr_running_total_time": 0.100031219, "msgr_send_bytes": 3988741, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 492, "msgr_send_messages_queue_lat": { "avgcount": 489, "avgtime": 0.000130972, "sum": 0.064045576 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 87, "avgtime": 0.109027154, "sum": 9.485362425 }, "queue_len": 1822 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2356, "avgtime": 0.000538506, "sum": 1.26872131 }, "queue_len": 0 }, "mds": { "caps": 1550, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5831, "ceph_cap_op_revoke": 38, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 509, "handle_client_caps": 5247, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4953, "inodes_bottom": 4953, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4953, "inodes_top": 0, "inodes_with_caps": 1550, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2307, "q": 0, "reply": 5508, "reply_latency": { "avgcount": 5508, "avgtime": 0.155201596, "sum": 854.850393095 }, "request": 5508, "root_rbytes": 11962368, "root_rfiles": 1548, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5510, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5510, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 763, "num_strays_delayed": 116, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2299, "strays_enqueued": 1536, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 12741, "evadd": 12736, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 17, "avgtime": 0.173152904, "sum": 2.943599374 }, "rdpos": 4195674, "replayed": 5, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 30558724 }, "mds_mem": { "cap": 1550, "cap+": 3202, "cap-": 1652, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4954, "dn+": 5510, "dn-": 556, "heap": 248164, "ino": 2658, "ino+": 3214, "ino-": 556, "rss": 99144 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5509, "dispatch_server_request": 0, "handle_client_request": 5508, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2299, "avgtime": 0.002764661, "sum": 6.355955924 } }, "mds_sessions": { "average_load": 5002, "avg_session_uptime": 955, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5002 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8550626, "buffer_anon_items": 6096, "buffer_meta_bytes": 1760, "buffer_meta_items": 20, "mds_co_bytes": 14663613, "mds_co_items": 183971, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 24, "op": 1515, "op_Infinitylight": 5, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 1509, "avgtime": 0.005003921, "sum": 7.550917258 }, "op_pg": 0, "op_r": 48, "op_reply": 1509, "op_resend": 0, "op_rmw": 0, "op_send": 1515, "op_send_bytes": 25772558, "op_w": 1467, "oplen_avg": { "avgcount": 1515, "sum": 1577 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1419, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 20, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 29, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1396, "pq_executed_ops": 1396, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 140 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12817, "get_or_fail_fail": 0, "get_or_fail_success": 12817, "get_started": 0, "get_sum": 3451950, "max": 104857600, "put": 12816, "put_sum": 3451697, "take": 0, "take_sum": 0, "val": 253, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1509, "put_sum": 25929615, "take": 1515, "take_sum": 25929615, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1509, "put_sum": 1509, "take": 1515, "take_sum": 1515, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1536, "get_or_fail_fail": 0, "get_or_fail_success": 1536, "get_started": 0, "get_sum": 155136, "max": 3758096384, "put": 12, "put_sum": 155136, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 12736, "get_or_fail_fail": 0, "get_or_fail_success": 12736, "get_started": 0, "get_sum": 26362235, "max": 3758096384, "put": 17, "put_sum": 25163639, "take": 0, "take_sum": 0, "val": 1198596, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:19.629 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/763, 1396/2299 2024-03-06T07:33:20.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:20.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.862+0000 7fd888600640 1 Processor -- start 2024-03-06T07:33:20.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.862+0000 7fd888600640 1 -- start start 2024-03-06T07:33:20.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd888600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd888600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd88015fba0 0x7fd8800b5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd888600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8800b56a0 0x7fd88015c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd888600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8800b5ab0 con 0x7fd88015fba0 2024-03-06T07:33:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd888600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd880145bc0 con 0x7fd8800b56a0 2024-03-06T07:33:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd888600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800cfb20 con 0x7fd8801457e0 2024-03-06T07:33:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.863+0000 7fd885200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd88015fba0 0x7fd8800b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd88015fba0 0x7fd8800b5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46196/0 (socket says 172.21.15.6:46196) 2024-03-06T07:33:20.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 -- 172.21.15.6:0/4278383915 learned_addr learned my addr 172.21.15.6:0/4278383915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:20.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885c00640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd886600640 1 --2- 172.21.15.6:0/4278383915 >> v2:172.21.15.178:3300/0 conn(0x7fd8800b56a0 0x7fd88015c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 -- 172.21.15.6:0/4278383915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 msgr2=0x7fd8800cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 -- 172.21.15.6:0/4278383915 >> v2:172.21.15.178:3300/0 conn(0x7fd8800b56a0 msgr2=0x7fd88015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:20.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 --2- 172.21.15.6:0/4278383915 >> v2:172.21.15.178:3300/0 conn(0x7fd8800b56a0 0x7fd88015c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885200640 1 -- 172.21.15.6:0/4278383915 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880160040 con 0x7fd88015fba0 2024-03-06T07:33:20.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd886600640 1 --2- 172.21.15.6:0/4278383915 >> v2:172.21.15.178:3300/0 conn(0x7fd8800b56a0 0x7fd88015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.864+0000 7fd885c00640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:20.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.865+0000 7fd885200640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b5160 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd88006f0e0 tx=0x7fd86c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bb1896e7ac1e94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.866+0000 7fd87fe00640 1 -- 172.21.15.6:0/4278383915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c02e070 con 0x7fd88015fba0 2024-03-06T07:33:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.866+0000 7fd87fe00640 1 -- 172.21.15.6:0/4278383915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c037070 con 0x7fd88015fba0 2024-03-06T07:33:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.866+0000 7fd87fe00640 1 -- 172.21.15.6:0/4278383915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c031a00 con 0x7fd88015fba0 2024-03-06T07:33:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 -- 172.21.15.6:0/4278383915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 msgr2=0x7fd8800b5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b5160 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fd88006f0e0 tx=0x7fd86c02b880 comp rx=0 tx=0).stop 2024-03-06T07:33:20.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 -- 172.21.15.6:0/4278383915 shutdown_connections 2024-03-06T07:33:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 --2- 172.21.15.6:0/4278383915 >> v2:172.21.15.178:3300/0 conn(0x7fd8800b56a0 0x7fd88015c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b5160 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 --2- 172.21.15.6:0/4278383915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 -- 172.21.15.6:0/4278383915 >> 172.21.15.6:0/4278383915 conn(0x7fd88006cf40 msgr2=0x7fd88006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 -- 172.21.15.6:0/4278383915 shutdown_connections 2024-03-06T07:33:20.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.867+0000 7fd888600640 1 -- 172.21.15.6:0/4278383915 wait complete. 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.869+0000 7fd888600640 1 Processor -- start 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.870+0000 7fd888600640 1 -- start start 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.870+0000 7fd888600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8800b56a0 0x7fd8800b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.870+0000 7fd888600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd888600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd888600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8801608e0 con 0x7fd88015fba0 2024-03-06T07:33:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd888600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd880160600 con 0x7fd8800b56a0 2024-03-06T07:33:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd888600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800cca70 con 0x7fd8801457e0 2024-03-06T07:33:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd886600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd886600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46206/0 (socket says 172.21.15.6:46206) 2024-03-06T07:33:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd886600640 1 -- 172.21.15.6:0/3896184750 learned_addr learned my addr 172.21.15.6:0/3896184750 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd885c00640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8800b56a0 0x7fd8800b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.871+0000 7fd885200640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd886600640 1 -- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 msgr2=0x7fd8800c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd886600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd886600640 1 -- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8800b56a0 msgr2=0x7fd8800b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd886600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8800b56a0 0x7fd8800b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd886600640 1 -- 172.21.15.6:0/3896184750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8800b35e0 con 0x7fd88015fba0 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd886600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b4680 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd87800cef0 tx=0x7fd87800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87801a070 con 0x7fd88015fba0 2024-03-06T07:33:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87800d840 con 0x7fd88015fba0 2024-03-06T07:33:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87800dc30 con 0x7fd88015fba0 2024-03-06T07:33:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.872+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800b38c0 con 0x7fd88015fba0 2024-03-06T07:33:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.873+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800b4160 con 0x7fd88015fba0 2024-03-06T07:33:20.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.874+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800b3a90 con 0x7fd88015fba0 2024-03-06T07:33:20.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.876+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd87801d070 con 0x7fd88015fba0 2024-03-06T07:33:20.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.877+0000 7fd87d600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86006b4d0 0x7fd86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:20.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.877+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd87808ab60 con 0x7fd88015fba0 2024-03-06T07:33:20.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.879+0000 7fd885c00640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86006b4d0 0x7fd86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:20.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.881+0000 7fd885c00640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86006b4d0 0x7fd86006d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd87400ab60 tx=0x7fd874008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:20.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:20.886+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8780591c0 con 0x7fd88015fba0 2024-03-06T07:33:21.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.210+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8800692a0 con 0x7fd88015fba0 2024-03-06T07:33:21.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.211+0000 7fd87d600640 1 -- 172.21.15.6:0/3896184750 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fd87805d260 con 0x7fd88015fba0 2024-03-06T07:33:21.213 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:21.214 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:21.214 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:21.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86006b4d0 msgr2=0x7fd86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86006b4d0 0x7fd86006d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd87400ab60 tx=0x7fd874008040 comp rx=0 tx=0).stop 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 msgr2=0x7fd8800b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b4680 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fd87800cef0 tx=0x7fd87800a710 comp rx=0 tx=0).stop 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 shutdown_connections 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86006b4d0 0x7fd86006d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd88015fba0 0x7fd8800b4680 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8801457e0 0x7fd8800c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 --2- 172.21.15.6:0/3896184750 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd8800b56a0 0x7fd8800b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 >> 172.21.15.6:0/3896184750 conn(0x7fd88006cf40 msgr2=0x7fd8801436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 shutdown_connections 2024-03-06T07:33:21.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.215+0000 7fd888600640 1 -- 172.21.15.6:0/3896184750 wait complete. 2024-03-06T07:33:21.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:21.482 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":888,"msgr_send_messages":888,"msgr_recv_bytes":234761,"msgr_send_bytes":4505129,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.150219976,"msgr_running_send_time":0.076809731,"msgr_running_recv_time":0.031611474,"msgr_running_fast_dispatch_time":0.030569833,"msgr_send_messages_queue_lat":{"avgcount":887,"sum":0.111817664,"avgtime":0.000126062},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12829,"msgr_send_messages":22402,"msgr_recv_bytes":4640271,"msgr_send_bytes":29272197,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":2.938599597,"msgr_running_send_time":1.927088942,"msgr_running_recv_time":1.811738340,"msgr_running_fast_dispatch_time":0.037715289,"msgr_send_messages_queue_lat":{"avgcount":22400,"sum":1.781966463,"avgtime":0.000079552},"msgr_handle_ack_lat":{"avgcount":26487,"sum":0.012532784,"avgtime":0.000000473},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":64384},"AsyncMessenger::Worker-2":{"msgr_recv_messages":637,"msgr_send_messages":633,"msgr_recv_bytes":268875,"msgr_send_bytes":4035687,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.114314786,"msgr_running_send_time":0.058575810,"msgr_running_recv_time":0.026563897,"msgr_running_fast_dispatch_time":0.025382145,"msgr_send_messages_queue_lat":{"avgcount":632,"sum":0.083508012,"avgtime":0.000132132},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106544,"msgr_send_encrypted_bytes":7280},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1649,"complete_latency":{"avgcount":91,"sum":11.929459017,"avgtime":0.131092956}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3365,"sum":1.776982456,"avgtime":0.000528077}},"mds":{"request":6410,"reply":6409,"reply_latency":{"avgcount":6409,"sum":856.354767487,"avgtime":0.133617532},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4590,"inodes_top":0,"inodes_bottom":4590,"inodes_pin_tail":0,"inodes_pinned":4590,"inodes_expired":0,"inodes_with_caps":696,"caps":696,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":960,"root_rbytes":7525376,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":39,"ceph_cap_op_grant":6587,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5248,"handle_client_caps_dirty":3000,"handle_client_cap_release":515,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1208,"num_strays_delayed":514,"num_strays_enqueuing":0,"strays_created":3200,"strays_enqueued":1992,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":14728,"evex":0,"evtrm":0,"ev":14733,"evexg":0,"evexd":0,"segadd":16,"segex":0,"segtrm":0,"seg":18,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":34528903,"rdpos":4195674,"jlat":{"avgcount":21,"sum":4.706788774,"avgtime":0.224132798},"replayed":5},"mds_mem":{"ino":1415,"ino+":3214,"ino-":1799,"dir":13,"dir+":13,"dir-":0,"dn":4613,"dn+":6412,"dn-":1799,"cap":696,"cap+":3202,"cap-":2506,"rss":103740,"heap":248164},"mds_server":{"dispatch_client_request":6411,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.860330316,"avgtime":0.002456353},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5791,"average_load":5791,"avg_session_uptime":960,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13326565,"buffer_anon_items":6446,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12553582,"mds_co_items":185628,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":2128,"op_send_bytes":31003403,"op_resend":0,"op_reply":2128,"op_latency":{"avgcount":2128,"sum":9.657894163,"avgtime":0.004538484},"op_inflight":0,"oplen_avg":{"avgcount":2128,"sum":2194},"op":2128,"op_r":54,"op_w":2074,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":26,"osdop_write":39,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2010,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":28,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1992,"pq_executed":1992,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14347,"get_sum":3849122,"get_or_fail_fail":0,"get_or_fail_success":14347,"take":0,"take_sum":0,"put":14347,"put_sum":3849122,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2128,"take_sum":31206516,"put":2128,"put_sum":31206516,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2128,"take_sum":2128,"put":2128,"put_sum":2128,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1992,"get_sum":201192,"get_or_fail_fail":0,"get_or_fail_success":1992,"take":0,"take_sum":0,"put":18,"put_sum":201192,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":0,"max":3758096384,"get_started":0,"get":14728,"get_sum":30332414,"get_or_fail_fail":0,"get_or_fail_success":14728,"take":0,"take_sum":0,"put":21,"put_sum":30332414,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:21.484 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 234761, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 888, "msgr_running_fast_dispatch_time": 0.030569833, "msgr_running_recv_time": 0.031611474, "msgr_running_send_time": 0.076809731, "msgr_running_total_time": 0.150219976, "msgr_send_bytes": 4505129, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 888, "msgr_send_messages_queue_lat": { "avgcount": 887, "avgtime": 0.000126062, "sum": 0.111817664 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 26487, "avgtime": 4.73e-07, "sum": 0.012532784 }, "msgr_recv_bytes": 4640271, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 12829, "msgr_running_fast_dispatch_time": 0.037715289, "msgr_running_recv_time": 1.81173834, "msgr_running_send_time": 1.927088942, "msgr_running_total_time": 2.938599597, "msgr_send_bytes": 29272197, "msgr_send_encrypted_bytes": 64384, "msgr_send_messages": 22402, "msgr_send_messages_queue_lat": { "avgcount": 22400, "avgtime": 7.9552e-05, "sum": 1.781966463 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 268875, "msgr_recv_encrypted_bytes": 106544, "msgr_recv_messages": 637, "msgr_running_fast_dispatch_time": 0.025382145, "msgr_running_recv_time": 0.026563897, "msgr_running_send_time": 0.05857581, "msgr_running_total_time": 0.114314786, "msgr_send_bytes": 4035687, "msgr_send_encrypted_bytes": 7280, "msgr_send_messages": 633, "msgr_send_messages_queue_lat": { "avgcount": 632, "avgtime": 0.000132132, "sum": 0.083508012 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 91, "avgtime": 0.131092956, "sum": 11.929459017 }, "queue_len": 1649 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3365, "avgtime": 0.000528077, "sum": 1.776982456 }, "queue_len": 0 }, "mds": { "caps": 696, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6587, "ceph_cap_op_revoke": 39, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 515, "handle_client_caps": 5248, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4590, "inodes_bottom": 4590, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4590, "inodes_top": 0, "inodes_with_caps": 696, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.133617532, "sum": 856.354767487 }, "request": 6410, "root_rbytes": 7525376, "root_rfiles": 960, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1208, "num_strays_delayed": 514, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3200, "strays_enqueued": 1992, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 14733, "evadd": 14728, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 21, "avgtime": 0.224132798, "sum": 4.706788774 }, "rdpos": 4195674, "replayed": 5, "seg": 18, "segadd": 16, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 34528903 }, "mds_mem": { "cap": 696, "cap+": 3202, "cap-": 2506, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4613, "dn+": 6412, "dn-": 1799, "heap": 248164, "ino": 1415, "ino+": 3214, "ino-": 1799, "rss": 103740 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6411, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002456353, "sum": 7.860330316 } }, "mds_sessions": { "average_load": 5791, "avg_session_uptime": 960, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5791 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13326565, "buffer_anon_items": 6446, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 12553582, "mds_co_items": 185628, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 28, "op": 2128, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 2128, "avgtime": 0.004538484, "sum": 9.657894163 }, "op_pg": 0, "op_r": 54, "op_reply": 2128, "op_resend": 0, "op_rmw": 0, "op_send": 2128, "op_send_bytes": 31003403, "op_w": 2074, "oplen_avg": { "avgcount": 2128, "sum": 2194 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 2010, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 26, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 39, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1992, "pq_executed_ops": 1992, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14347, "get_or_fail_fail": 0, "get_or_fail_success": 14347, "get_started": 0, "get_sum": 3849122, "max": 104857600, "put": 14347, "put_sum": 3849122, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 2128, "put_sum": 31206516, "take": 2128, "take_sum": 31206516, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 2128, "put_sum": 2128, "take": 2128, "take_sum": 2128, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1992, "get_or_fail_fail": 0, "get_or_fail_success": 1992, "get_started": 0, "get_sum": 201192, "max": 3758096384, "put": 18, "put_sum": 201192, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 14728, "get_or_fail_fail": 0, "get_or_fail_success": 14728, "get_started": 0, "get_sum": 30332414, "max": 3758096384, "put": 21, "put_sum": 30332414, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:21.484 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1208, 1992/3200 2024-03-06T07:33:21.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:21.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.977+0000 7fe085c00640 1 Processor -- start 2024-03-06T07:33:22.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.978+0000 7fe085c00640 1 -- start start 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.981+0000 7fe085c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe080160380 0x7fe080162b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.981+0000 7fe085c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe0800b4b20 0x7fe0800b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe07ea00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fe08015ffa0 0x7fe0800b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:44740/0 (socket says 172.21.15.6:44740) 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe085c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0800b3f40 con 0x7fe08015ffa0 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe085c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800b54c0 con 0x7fe0800b4b20 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe085c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe080070e20 con 0x7fe080160380 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe07ea00640 1 -- 172.21.15.6:0/925999082 learned_addr learned my addr 172.21.15.6:0/925999082 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe07e000640 1 --2- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe080160380 0x7fe080162b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.982+0000 7fe07f400640 1 --2- 172.21.15.6:0/925999082 >> v2:172.21.15.178:3300/0 conn(0x7fe0800b4b20 0x7fe0800b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.983+0000 7fe07f400640 1 -- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe080160380 msgr2=0x7fe080162b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.983+0000 7fe07f400640 1 --2- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe080160380 0x7fe080162b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.983+0000 7fe07f400640 1 -- 172.21.15.6:0/925999082 >> v1:172.21.15.178:6789/0 conn(0x7fe08015ffa0 legacy=0x7fe0800b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.983+0000 7fe07f400640 1 -- 172.21.15.6:0/925999082 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0800b36a0 con 0x7fe0800b4b20 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.984+0000 7fe07f400640 1 --2- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0800b4b20 0x7fe0800b4f80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe080070fa0 tx=0x7fe07802be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c602eb233b2e3cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.985+0000 7fe07d600640 1 -- 172.21.15.6:0/925999082 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078039070 con 0x7fe0800b4b20 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.985+0000 7fe07d600640 1 -- 172.21.15.6:0/925999082 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07802cc40 con 0x7fe0800b4b20 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.985+0000 7fe07d600640 1 -- 172.21.15.6:0/925999082 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078034440 con 0x7fe0800b4b20 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.987+0000 7fe085c00640 1 -- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0800b4b20 msgr2=0x7fe0800b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.987+0000 7fe085c00640 1 --2- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0800b4b20 0x7fe0800b4f80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe080070fa0 tx=0x7fe07802be90 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.988+0000 7fe085c00640 1 -- 172.21.15.6:0/925999082 shutdown_connections 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.988+0000 7fe085c00640 1 --2- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0800b4b20 0x7fe0800b4f80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.988+0000 7fe085c00640 1 --2- 172.21.15.6:0/925999082 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe080160380 0x7fe080162b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.988+0000 7fe085c00640 1 -- 172.21.15.6:0/925999082 >> 172.21.15.6:0/925999082 conn(0x7fe08006cf40 msgr2=0x7fe08006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.988+0000 7fe085c00640 1 -- 172.21.15.6:0/925999082 shutdown_connections 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.988+0000 7fe085c00640 1 -- 172.21.15.6:0/925999082 wait complete. 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.991+0000 7fe085c00640 1 Processor -- start 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.991+0000 7fe085c00640 1 -- start start 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.992+0000 7fe085c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 0x7fe080146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.992+0000 7fe085c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe08015ffa0 0x7fe080146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.992+0000 7fe085c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe080147500 0x7fe080162860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.992+0000 7fe085c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800712b0 con 0x7fe0800b4b20 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.992+0000 7fe085c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800b3c60 con 0x7fe080147500 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.992+0000 7fe085c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe080071430 con 0x7fe08015ffa0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.993+0000 7fe07ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 0x7fe080146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.993+0000 7fe07f400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe080147500 0x7fe080162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.993+0000 7fe07ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 0x7fe080146800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52704/0 (socket says 172.21.15.6:52704) 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.993+0000 7fe07ea00640 1 -- 172.21.15.6:0/2519606193 learned_addr learned my addr 172.21.15.6:0/2519606193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.993+0000 7fe07e000640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe08015ffa0 0x7fe080146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07e000640 1 -- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe080147500 msgr2=0x7fe080162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07e000640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe080147500 0x7fe080162860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07e000640 1 -- 172.21.15.6:0/2519606193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 msgr2=0x7fe080146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07e000640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 0x7fe080146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07e000640 1 -- 172.21.15.6:0/2519606193 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0801617c0 con 0x7fe08015ffa0 2024-03-06T07:33:22.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07ea00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 0x7fe080146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe07e000640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe08015ffa0 0x7fe080146fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00e8a0 tx=0x7fe06c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.994+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c017070 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.995+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06c00c940 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.995+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c009e60 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.995+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe080161aa0 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.995+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe080162060 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.997+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0800692a0 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:21.998+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe06c01a070 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.000+0000 7fe076a00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe06006b520 0x7fe06006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.000+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe06c08a2c0 con 0x7fe08015ffa0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.001+0000 7fe07ea00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe06006b520 0x7fe06006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.001+0000 7fe07ea00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe06006b520 0x7fe06006d9e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe080072d60 tx=0x7fe070009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.007+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe06c058b20 con 0x7fe08015ffa0 2024-03-06T07:33:22.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.403+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe08006f010 con 0x7fe08015ffa0 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.405+0000 7fe076a00640 1 -- 172.21.15.6:0/2519606193 <== mon.2 v2:172.21.15.178:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fe06c05cbc0 con 0x7fe08015ffa0 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:22.407 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe06006b520 msgr2=0x7fe06006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe06006b520 0x7fe06006d9e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe080072d60 tx=0x7fe070009040 comp rx=0 tx=0).stop 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe08015ffa0 msgr2=0x7fe080146fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe08015ffa0 0x7fe080146fc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00e8a0 tx=0x7fe06c00ed60 comp rx=0 tx=0).stop 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 shutdown_connections 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe06006b520 0x7fe06006d9e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe080147500 0x7fe080162860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe08015ffa0 0x7fe080146fc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 --2- 172.21.15.6:0/2519606193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe0800b4b20 0x7fe080146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 >> 172.21.15.6:0/2519606193 conn(0x7fe08006cf40 msgr2=0x7fe08006d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 shutdown_connections 2024-03-06T07:33:22.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.407+0000 7fe085c00640 1 -- 172.21.15.6:0/2519606193 wait complete. 2024-03-06T07:33:22.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:33:22.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.666+0000 7ff98d400640 1 Processor -- start 2024-03-06T07:33:22.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.666+0000 7ff98d400640 1 -- start start 2024-03-06T07:33:22.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff98d400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816f3c0 0x7ff9880ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff98d400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff98816e7b0 0x7ff98816ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff98d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98816e520 con 0x7ff98816e140 2024-03-06T07:33:22.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff98d400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9880b4da0 con 0x7ff98816e7b0 2024-03-06T07:33:22.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff98d400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806d0f0 con 0x7ff98816f3c0 2024-03-06T07:33:22.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff987e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff98816e140 0x7ff9880b4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57190/0 (socket says 172.21.15.6:57190) 2024-03-06T07:33:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.668+0000 7ff987e00640 1 -- 172.21.15.6:0/946319744 learned_addr learned my addr 172.21.15.6:0/946319744 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.669+0000 7ff987400640 1 --2- 172.21.15.6:0/946319744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816f3c0 0x7ff9880ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.669+0000 7ff986a00640 1 -- 172.21.15.6:0/946319744 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025339828 0 0) 0x7ff98816e520 con 0x7ff98816e140 2024-03-06T07:33:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.669+0000 7ff98ca00640 1 --2- 172.21.15.6:0/946319744 >> v2:172.21.15.178:3300/0 conn(0x7ff98816e7b0 0x7ff98816ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.868+0000 7ff986a00640 1 -- 172.21.15.6:0/946319744 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96c003520 con 0x7ff98816e140 2024-03-06T07:33:22.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.869+0000 7ff987400640 1 -- 172.21.15.6:0/946319744 >> v2:172.21.15.178:3300/0 conn(0x7ff98816e7b0 msgr2=0x7ff98816ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:22.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.869+0000 7ff987400640 1 --2- 172.21.15.6:0/946319744 >> v2:172.21.15.178:3300/0 conn(0x7ff98816e7b0 0x7ff98816ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.869+0000 7ff987400640 1 -- 172.21.15.6:0/946319744 >> v1:172.21.15.6:6789/0 conn(0x7ff98816e140 legacy=0x7ff9880b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.869+0000 7ff987400640 1 -- 172.21.15.6:0/946319744 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9880b4ac0 con 0x7ff98816f3c0 2024-03-06T07:33:22.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.870+0000 7ff98ca00640 1 --2- 172.21.15.6:0/946319744 >> v2:172.21.15.178:3300/0 conn(0x7ff98816e7b0 0x7ff98816ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:22.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.872+0000 7ff987400640 1 --2- 172.21.15.6:0/946319744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816f3c0 0x7ff9880ad4d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff9880b4f20 tx=0x7ff97402c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65bcec8115a410e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.872+0000 7ff986a00640 1 -- 172.21.15.6:0/946319744 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974035020 con 0x7ff98816f3c0 2024-03-06T07:33:22.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.873+0000 7ff986a00640 1 -- 172.21.15.6:0/946319744 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff974008df0 con 0x7ff98816f3c0 2024-03-06T07:33:22.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.873+0000 7ff986a00640 1 -- 172.21.15.6:0/946319744 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97400a2f0 con 0x7ff98816f3c0 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.874+0000 7ff98d400640 1 -- 172.21.15.6:0/946319744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816f3c0 msgr2=0x7ff9880ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.874+0000 7ff98d400640 1 --2- 172.21.15.6:0/946319744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816f3c0 0x7ff9880ad4d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff9880b4f20 tx=0x7ff97402c940 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.877+0000 7ff98d400640 1 -- 172.21.15.6:0/946319744 shutdown_connections 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.877+0000 7ff98d400640 1 --2- 172.21.15.6:0/946319744 >> v2:172.21.15.178:3300/0 conn(0x7ff98816e7b0 0x7ff98816ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.877+0000 7ff98d400640 1 --2- 172.21.15.6:0/946319744 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816f3c0 0x7ff9880ad4d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.877+0000 7ff98d400640 1 -- 172.21.15.6:0/946319744 >> 172.21.15.6:0/946319744 conn(0x7ff98806afc0 msgr2=0x7ff98806b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.877+0000 7ff98d400640 1 -- 172.21.15.6:0/946319744 shutdown_connections 2024-03-06T07:33:22.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.877+0000 7ff98d400640 1 -- 172.21.15.6:0/946319744 wait complete. 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.880+0000 7ff98d400640 1 Processor -- start 2024-03-06T07:33:22.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.880+0000 7ff98d400640 1 -- start start 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98d400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816e140 0x7ff98809d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98d400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff98816e7b0 0x7ff9881668c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 0x7ff98815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806cd80 con 0x7ff98816f3c0 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98d400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9880b4860 con 0x7ff98816e7b0 2024-03-06T07:33:22.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98d400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806d580 con 0x7ff98816e140 2024-03-06T07:33:22.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 0x7ff98815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 0x7ff98815b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52708/0 (socket says 172.21.15.6:52708) 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.882+0000 7ff98ca00640 1 -- 172.21.15.6:0/3335967078 learned_addr learned my addr 172.21.15.6:0/3335967078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff987e00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816e140 0x7ff98809d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff987400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff98816e7b0 0x7ff9881668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff98ca00640 1 -- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816e140 msgr2=0x7ff98809d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff98ca00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816e140 0x7ff98809d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff98ca00640 1 -- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff98816e7b0 msgr2=0x7ff9881668c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff98ca00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff98816e7b0 0x7ff9881668c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.883+0000 7ff98ca00640 1 -- 172.21.15.6:0/3335967078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988166e00 con 0x7ff98816f3c0 2024-03-06T07:33:22.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.884+0000 7ff98ca00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 0x7ff98815b3d0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7ff98000ea20 tx=0x7ff98000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.884+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98000cd30 con 0x7ff98816f3c0 2024-03-06T07:33:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.884+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98001a480 con 0x7ff98816f3c0 2024-03-06T07:33:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.884+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98001a850 con 0x7ff98816f3c0 2024-03-06T07:33:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.884+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98809aa40 con 0x7ff98816f3c0 2024-03-06T07:33:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.884+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98809ad00 con 0x7ff98816f3c0 2024-03-06T07:33:22.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.886+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98804f330 con 0x7ff98816f3c0 2024-03-06T07:33:22.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:22.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.888+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff9800233f0 con 0x7ff98816f3c0 2024-03-06T07:33:22.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.889+0000 7ff97fe00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff96806b4d0 0x7ff96806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:22.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.889+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff98008ae50 con 0x7ff98816f3c0 2024-03-06T07:33:22.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.890+0000 7ff987e00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff96806b4d0 0x7ff96806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:22.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.891+0000 7ff987e00640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff96806b4d0 0x7ff96806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff978005880 tx=0x7ff978009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:22.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:22.898+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff988166e00 con 0x7ff98816f3c0 2024-03-06T07:33:23.147 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:33:23.148 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:33:23.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.159+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9880b3520 con 0x7ff98816f3c0 2024-03-06T07:33:23.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.159+0000 7ff97fe00640 1 -- 172.21.15.6:0/3335967078 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7ff9800595c0 con 0x7ff98816f3c0 2024-03-06T07:33:23.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:23.162 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:23.162 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:23.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff96806b4d0 msgr2=0x7ff96806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff96806b4d0 0x7ff96806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff978005880 tx=0x7ff978009040 comp rx=0 tx=0).stop 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 msgr2=0x7ff98815b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 0x7ff98815b3d0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7ff98000ea20 tx=0x7ff98000eee0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 shutdown_connections 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff96806b4d0 0x7ff96806d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff98816e7b0 0x7ff9881668c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff98816f3c0 0x7ff98815b3d0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 --2- 172.21.15.6:0/3335967078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff98816e140 0x7ff98809d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 >> 172.21.15.6:0/3335967078 conn(0x7ff98806afc0 msgr2=0x7ff98816d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 shutdown_connections 2024-03-06T07:33:23.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:23.162+0000 7ff98d400640 1 -- 172.21.15.6:0/3335967078 wait complete. 2024-03-06T07:33:23.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:23.434 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1403,"msgr_send_messages":1403,"msgr_recv_bytes":366534,"msgr_send_bytes":4681191,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.210895490,"msgr_running_send_time":0.102546510,"msgr_running_recv_time":0.044694765,"msgr_running_fast_dispatch_time":0.046460137,"msgr_send_messages_queue_lat":{"avgcount":1402,"sum":0.152706807,"avgtime":0.000108920},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13203,"msgr_send_messages":24121,"msgr_recv_bytes":4872810,"msgr_send_bytes":30602572,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.052996797,"msgr_running_send_time":1.995609701,"msgr_running_recv_time":1.834766425,"msgr_running_fast_dispatch_time":0.051691245,"msgr_send_messages_queue_lat":{"avgcount":24120,"sum":1.853510812,"avgtime":0.000076845},"msgr_handle_ack_lat":{"avgcount":27172,"sum":0.012806065,"avgtime":0.000000471},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":64384},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1014,"msgr_send_messages":1010,"msgr_recv_bytes":365505,"msgr_send_bytes":4171508,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.151358409,"msgr_running_send_time":0.070124840,"msgr_running_recv_time":0.035721559,"msgr_running_fast_dispatch_time":0.038324458,"msgr_send_messages_queue_lat":{"avgcount":1009,"sum":0.116124851,"avgtime":0.000115089},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106816,"msgr_send_encrypted_bytes":7904},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":133,"sum":12.752581966,"avgtime":0.095884074}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5398,"sum":1.919413579,"avgtime":0.000355578}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":856.710534720,"avgtime":0.133652189},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4420,"inodes_top":0,"inodes_bottom":4420,"inodes_pin_tail":0,"inodes_pinned":4420,"inodes_expired":0,"inodes_with_caps":16,"caps":16,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":41,"ceph_cap_op_grant":7262,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5250,"handle_client_caps_dirty":3000,"handle_client_cap_release":519,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":15,"num_strays_delayed":15,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3186,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15927,"evex":15929,"evtrm":15929,"ev":15931,"evexg":0,"evexd":15928,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":35247716,"wrpos":35251910,"rdpos":4195674,"jlat":{"avgcount":24,"sum":5.213867849,"avgtime":0.217244493},"replayed":5},"mds_mem":{"ino":1064,"ino+":3214,"ino-":2150,"dir":13,"dir+":13,"dir-":0,"dn":1062,"dn+":6412,"dn-":5350,"cap":16,"cap+":3202,"cap-":3186,"rss":103740,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.355767233,"avgtime":0.355767233},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.860330316,"avgtime":0.002456353},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":960,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13326565,"buffer_anon_items":6446,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12553582,"mds_co_items":185628,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3385,"op_send_bytes":31910931,"op_resend":0,"op_reply":3385,"op_latency":{"avgcount":3385,"sum":13.317854066,"avgtime":0.003934373},"op_inflight":0,"oplen_avg":{"avgcount":3385,"sum":3520},"op":3385,"op_r":57,"op_w":3328,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":25,"osdop_read":29,"osdop_write":45,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3211,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":50,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":49,"omap_rd":48,"omap_del":11},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3186,"pq_executed":3186,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15613,"get_sum":4196072,"get_or_fail_fail":0,"get_or_fail_success":15613,"take":0,"take_sum":0,"put":15613,"put_sum":4196072,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":32234638,"put":3385,"put_sum":32234638,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3385,"take_sum":3385,"put":3385,"put_sum":3385,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3186,"get_sum":321786,"get_or_fail_fail":0,"get_or_fail_success":3186,"take":0,"take_sum":0,"put":21,"put_sum":321786,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":0,"max":3758096384,"get_started":0,"get":15927,"get_sum":31055421,"get_or_fail_fail":0,"get_or_fail_success":15927,"take":0,"take_sum":0,"put":24,"put_sum":31055421,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:23.435 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 366534, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1403, "msgr_running_fast_dispatch_time": 0.046460137, "msgr_running_recv_time": 0.044694765, "msgr_running_send_time": 0.10254651, "msgr_running_total_time": 0.21089549, "msgr_send_bytes": 4681191, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1403, "msgr_send_messages_queue_lat": { "avgcount": 1402, "avgtime": 0.00010892, "sum": 0.152706807 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27172, "avgtime": 4.71e-07, "sum": 0.012806065 }, "msgr_recv_bytes": 4872810, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13203, "msgr_running_fast_dispatch_time": 0.051691245, "msgr_running_recv_time": 1.834766425, "msgr_running_send_time": 1.995609701, "msgr_running_total_time": 3.052996797, "msgr_send_bytes": 30602572, "msgr_send_encrypted_bytes": 64384, "msgr_send_messages": 24121, "msgr_send_messages_queue_lat": { "avgcount": 24120, "avgtime": 7.6845e-05, "sum": 1.853510812 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 365505, "msgr_recv_encrypted_bytes": 106816, "msgr_recv_messages": 1014, "msgr_running_fast_dispatch_time": 0.038324458, "msgr_running_recv_time": 0.035721559, "msgr_running_send_time": 0.07012484, "msgr_running_total_time": 0.151358409, "msgr_send_bytes": 4171508, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 1010, "msgr_send_messages_queue_lat": { "avgcount": 1009, "avgtime": 0.000115089, "sum": 0.116124851 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 133, "avgtime": 0.095884074, "sum": 12.752581966 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5398, "avgtime": 0.000355578, "sum": 1.919413579 }, "queue_len": 0 }, "mds": { "caps": 16, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7262, "ceph_cap_op_revoke": 41, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 519, "handle_client_caps": 5250, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4420, "inodes_bottom": 4420, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4420, "inodes_top": 0, "inodes_with_caps": 16, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.133652189, "sum": 856.71053472 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 15, "num_strays_delayed": 15, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3186, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 15931, "evadd": 15927, "evex": 15929, "evexd": 15928, "evexg": 0, "evlrg": 0, "evtrm": 15929, "expos": 35247716, "jlat": { "avgcount": 24, "avgtime": 0.217244493, "sum": 5.213867849 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 35251910 }, "mds_mem": { "cap": 16, "cap+": 3202, "cap-": 3186, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1062, "dn+": 6412, "dn-": 5350, "heap": 248164, "ino": 1064, "ino+": 3214, "ino-": 2150, "rss": 103740 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.355767233, "sum": 0.355767233 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002456353, "sum": 7.860330316 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 960, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13326565, "buffer_anon_items": 6446, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 12553582, "mds_co_items": 185628, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 11, "omap_rd": 48, "omap_wr": 49, "op": 3385, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3385, "avgtime": 0.003934373, "sum": 13.317854066 }, "op_pg": 0, "op_r": 57, "op_reply": 3385, "op_resend": 0, "op_rmw": 0, "op_send": 3385, "op_send_bytes": 31910931, "op_w": 3328, "oplen_avg": { "avgcount": 3385, "sum": 3520 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 25, "osdop_delete": 3211, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 29, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 50, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 45, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3186, "pq_executed_ops": 3186, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15613, "get_or_fail_fail": 0, "get_or_fail_success": 15613, "get_started": 0, "get_sum": 4196072, "max": 104857600, "put": 15613, "put_sum": 4196072, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3385, "put_sum": 32234638, "take": 3385, "take_sum": 32234638, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3385, "put_sum": 3385, "take": 3385, "take_sum": 3385, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3186, "get_or_fail_fail": 0, "get_or_fail_success": 3186, "get_started": 0, "get_sum": 321786, "max": 3758096384, "put": 21, "put_sum": 321786, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 15927, "get_or_fail_fail": 0, "get_or_fail_success": 15927, "get_started": 0, "get_sum": 31055421, "max": 3758096384, "put": 24, "put_sum": 31055421, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:23.436 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/15, 3186/3201 2024-03-06T07:33:24.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:24.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.661+0000 7ff092800640 1 Processor -- start 2024-03-06T07:33:24.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.661+0000 7ff092800640 1 -- start start 2024-03-06T07:33:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff092800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff08c15f630 0x7ff08c161ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff092800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff08c0b4b30 0x7ff08c1621e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff092800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0b50d0 con 0x7ff08c162720 2024-03-06T07:33:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff092800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0cf450 con 0x7ff08c0b4b30 2024-03-06T07:33:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff092800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0cf6a0 con 0x7ff08c15f630 2024-03-06T07:33:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff090a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff08c0b4b30 0x7ff08c1621e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff091e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff08c162720 0x7ff08c0b58b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57202/0 (socket says 172.21.15.6:57202) 2024-03-06T07:33:24.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.663+0000 7ff091e00640 1 -- 172.21.15.6:0/1745113819 learned_addr learned my addr 172.21.15.6:0/1745113819 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:24.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.664+0000 7ff091400640 1 --2- 172.21.15.6:0/1745113819 >> v2:172.21.15.178:3300/0 conn(0x7ff08c15f630 0x7ff08c161ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.664+0000 7ff090a00640 1 -- 172.21.15.6:0/1745113819 >> v2:172.21.15.178:3300/0 conn(0x7ff08c15f630 msgr2=0x7ff08c161ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.664+0000 7ff090a00640 1 --2- 172.21.15.6:0/1745113819 >> v2:172.21.15.178:3300/0 conn(0x7ff08c15f630 0x7ff08c161ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.664+0000 7ff090a00640 1 -- 172.21.15.6:0/1745113819 >> v1:172.21.15.6:6789/0 conn(0x7ff08c162720 legacy=0x7ff08c0b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.664+0000 7ff090a00640 1 -- 172.21.15.6:0/1745113819 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0cf980 con 0x7ff08c0b4b30 2024-03-06T07:33:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.664+0000 7ff08be00640 1 -- 172.21.15.6:0/1745113819 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227081902 0 0) 0x7ff08c0b50d0 con 0x7ff08c162720 2024-03-06T07:33:24.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.665+0000 7ff090a00640 1 --2- 172.21.15.6:0/1745113819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c0b4b30 0x7ff08c1621e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7ff0780097a0 tx=0x7ff07802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8a07e075f19d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff08be00640 1 -- 172.21.15.6:0/1745113819 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078038070 con 0x7ff08c0b4b30 2024-03-06T07:33:24.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff08be00640 1 -- 172.21.15.6:0/1745113819 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078031700 con 0x7ff08c0b4b30 2024-03-06T07:33:24.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff08be00640 1 -- 172.21.15.6:0/1745113819 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078031ad0 con 0x7ff08c0b4b30 2024-03-06T07:33:24.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 -- 172.21.15.6:0/1745113819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c0b4b30 msgr2=0x7ff08c1621e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 --2- 172.21.15.6:0/1745113819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c0b4b30 0x7ff08c1621e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7ff0780097a0 tx=0x7ff07802b7a0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 -- 172.21.15.6:0/1745113819 shutdown_connections 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 --2- 172.21.15.6:0/1745113819 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c0b4b30 0x7ff08c1621e0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 --2- 172.21.15.6:0/1745113819 >> v2:172.21.15.178:3300/0 conn(0x7ff08c15f630 0x7ff08c161ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 -- 172.21.15.6:0/1745113819 >> 172.21.15.6:0/1745113819 conn(0x7ff08c0d3e30 msgr2=0x7ff08c0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.666+0000 7ff092800640 1 -- 172.21.15.6:0/1745113819 shutdown_connections 2024-03-06T07:33:24.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.667+0000 7ff092800640 1 -- 172.21.15.6:0/1745113819 wait complete. 2024-03-06T07:33:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.668+0000 7ff092800640 1 Processor -- start 2024-03-06T07:33:24.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.669+0000 7ff092800640 1 -- start start 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff092800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff08c0b4b30 0x7ff08c15e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff092800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff08c15f630 0x7ff08c15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff092800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 0x7ff08c15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff092800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0d2bf0 con 0x7ff08c162720 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff092800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0d4c40 con 0x7ff08c0b4b30 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff092800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0d2d70 con 0x7ff08c15f630 2024-03-06T07:33:24.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff091e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 0x7ff08c15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff091e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 0x7ff08c15d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52730/0 (socket says 172.21.15.6:52730) 2024-03-06T07:33:24.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff091e00640 1 -- 172.21.15.6:0/2444738476 learned_addr learned my addr 172.21.15.6:0/2444738476 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:24.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff091400640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff08c0b4b30 0x7ff08c15e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff090a00640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff08c15f630 0x7ff08c15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff091e00640 1 -- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff08c15f630 msgr2=0x7ff08c15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.670+0000 7ff091e00640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff08c15f630 0x7ff08c15cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff091e00640 1 -- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff08c0b4b30 msgr2=0x7ff08c15e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff091e00640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff08c0b4b30 0x7ff08c15e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff091e00640 1 -- 172.21.15.6:0/2444738476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0c7970 con 0x7ff08c162720 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff091e00640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 0x7ff08c15d160 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff08400c780 tx=0x7ff08400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084018070 con 0x7ff08c162720 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08400ce30 con 0x7ff08c162720 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08400a840 con 0x7ff08c162720 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c1f4300 con 0x7ff08c162720 2024-03-06T07:33:24.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.671+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c0c7ec0 con 0x7ff08c162720 2024-03-06T07:33:24.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.673+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c04f330 con 0x7ff08c162720 2024-03-06T07:33:24.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.673+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff08400a9e0 con 0x7ff08c162720 2024-03-06T07:33:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.674+0000 7ff089600640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff06c06b4d0 0x7ff06c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:24.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.675+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7ff084089980 con 0x7ff08c162720 2024-03-06T07:33:24.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.675+0000 7ff091400640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff06c06b4d0 0x7ff06c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:24.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.676+0000 7ff091400640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff06c06b4d0 0x7ff06c06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff080006fd0 tx=0x7ff080008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:24.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:24.679+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff08c1f4300 con 0x7ff08c162720 2024-03-06T07:33:25.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.004+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08c0cac30 con 0x7ff08c162720 2024-03-06T07:33:25.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.004+0000 7ff089600640 1 -- 172.21.15.6:0/2444738476 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7ff0840586b0 con 0x7ff08c162720 2024-03-06T07:33:25.006 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:25.007 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:25.007 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff06c06b4d0 msgr2=0x7ff06c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff06c06b4d0 0x7ff06c06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff080006fd0 tx=0x7ff080008040 comp rx=0 tx=0).stop 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 msgr2=0x7ff08c15d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 0x7ff08c15d160 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff08400c780 tx=0x7ff08400cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 shutdown_connections 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff06c06b4d0 0x7ff06c06d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff08c162720 0x7ff08c15d160 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff08c15f630 0x7ff08c15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 --2- 172.21.15.6:0/2444738476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff08c0b4b30 0x7ff08c15e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 >> 172.21.15.6:0/2444738476 conn(0x7ff08c0d3e30 msgr2=0x7ff08c0d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 shutdown_connections 2024-03-06T07:33:25.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.007+0000 7ff092800640 1 -- 172.21.15.6:0/2444738476 wait complete. 2024-03-06T07:33:25.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:25.248 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1409,"msgr_send_messages":1409,"msgr_recv_bytes":368058,"msgr_send_bytes":4683147,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.211607376,"msgr_running_send_time":0.102828449,"msgr_running_recv_time":0.044867155,"msgr_running_fast_dispatch_time":0.046660868,"msgr_send_messages_queue_lat":{"avgcount":1408,"sum":0.152948699,"avgtime":0.000108628},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13216,"msgr_send_messages":24131,"msgr_recv_bytes":4878199,"msgr_send_bytes":30617330,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.055250848,"msgr_running_send_time":1.996730517,"msgr_running_recv_time":1.835327517,"msgr_running_fast_dispatch_time":0.052179073,"msgr_send_messages_queue_lat":{"avgcount":24130,"sum":1.854070223,"avgtime":0.000076836},"msgr_handle_ack_lat":{"avgcount":27175,"sum":0.012806767,"avgtime":0.000000471},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":64384},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1020,"msgr_send_messages":1016,"msgr_recv_bytes":367021,"msgr_send_bytes":4173546,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.152513899,"msgr_running_send_time":0.070328865,"msgr_running_recv_time":0.035855766,"msgr_running_fast_dispatch_time":0.038490446,"msgr_send_messages_queue_lat":{"avgcount":1015,"sum":0.116926391,"avgtime":0.000115198},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106816,"msgr_send_encrypted_bytes":7904},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":136,"sum":12.755562347,"avgtime":0.093790899}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5428,"sum":1.921599207,"avgtime":0.000354016}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":856.710534720,"avgtime":0.133652189},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":26,"inodes_top":0,"inodes_bottom":26,"inodes_pin_tail":0,"inodes_pinned":26,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":41,"ceph_cap_op_grant":7262,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5250,"handle_client_caps_dirty":3000,"handle_client_cap_release":520,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15942,"evex":15929,"evtrm":15929,"ev":18,"evexg":0,"evexd":15928,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":35247716,"wrpos":35261744,"rdpos":4195674,"jlat":{"avgcount":25,"sum":5.216418172,"avgtime":0.208656726},"replayed":5},"mds_mem":{"ino":28,"ino+":3214,"ino-":3186,"dir":13,"dir+":13,"dir-":0,"dn":26,"dn+":6412,"dn-":6386,"cap":1,"cap+":3202,"cap-":3201,"rss":103996,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.355767233,"avgtime":0.355767233},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.860330316,"avgtime":0.002456353},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":965,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":372453,"buffer_anon_items":138,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":103961,"mds_co_items":1335,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3407,"op_send_bytes":31922487,"op_resend":0,"op_reply":3407,"op_latency":{"avgcount":3407,"sum":13.400853154,"avgtime":0.003933329},"op_inflight":0,"oplen_avg":{"avgcount":3407,"sum":3544},"op":3407,"op_r":59,"op_w":3348,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":25,"osdop_read":31,"osdop_write":48,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":50,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15638,"get_sum":4202251,"get_or_fail_fail":0,"get_or_fail_success":15638,"take":0,"take_sum":0,"put":15638,"put_sum":4202251,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3407,"take_sum":32247709,"put":3407,"put_sum":32247709,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3407,"take_sum":3407,"put":3407,"put_sum":3407,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":0,"max":3758096384,"get_started":0,"get":15942,"get_sum":31065255,"get_or_fail_fail":0,"get_or_fail_success":15942,"take":0,"take_sum":0,"put":25,"put_sum":31065255,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:25.250 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 368058, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1409, "msgr_running_fast_dispatch_time": 0.046660868, "msgr_running_recv_time": 0.044867155, "msgr_running_send_time": 0.102828449, "msgr_running_total_time": 0.211607376, "msgr_send_bytes": 4683147, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1409, "msgr_send_messages_queue_lat": { "avgcount": 1408, "avgtime": 0.000108628, "sum": 0.152948699 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27175, "avgtime": 4.71e-07, "sum": 0.012806767 }, "msgr_recv_bytes": 4878199, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13216, "msgr_running_fast_dispatch_time": 0.052179073, "msgr_running_recv_time": 1.835327517, "msgr_running_send_time": 1.996730517, "msgr_running_total_time": 3.055250848, "msgr_send_bytes": 30617330, "msgr_send_encrypted_bytes": 64384, "msgr_send_messages": 24131, "msgr_send_messages_queue_lat": { "avgcount": 24130, "avgtime": 7.6836e-05, "sum": 1.854070223 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 367021, "msgr_recv_encrypted_bytes": 106816, "msgr_recv_messages": 1020, "msgr_running_fast_dispatch_time": 0.038490446, "msgr_running_recv_time": 0.035855766, "msgr_running_send_time": 0.070328865, "msgr_running_total_time": 0.152513899, "msgr_send_bytes": 4173546, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 1016, "msgr_send_messages_queue_lat": { "avgcount": 1015, "avgtime": 0.000115198, "sum": 0.116926391 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 136, "avgtime": 0.093790899, "sum": 12.755562347 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5428, "avgtime": 0.000354016, "sum": 1.921599207 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7262, "ceph_cap_op_revoke": 41, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 520, "handle_client_caps": 5250, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 26, "inodes_bottom": 26, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 26, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.133652189, "sum": 856.71053472 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 18, "evadd": 15942, "evex": 15929, "evexd": 15928, "evexg": 0, "evlrg": 0, "evtrm": 15929, "expos": 35247716, "jlat": { "avgcount": 25, "avgtime": 0.208656726, "sum": 5.216418172 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 35261744 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 26, "dn+": 6412, "dn-": 6386, "heap": 248164, "ino": 28, "ino+": 3214, "ino-": 3186, "rss": 103996 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.355767233, "sum": 0.355767233 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002456353, "sum": 7.860330316 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 965, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 372453, "buffer_anon_items": 138, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 103961, "mds_co_items": 1335, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 3407, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3407, "avgtime": 0.003933329, "sum": 13.400853154 }, "op_pg": 0, "op_r": 59, "op_reply": 3407, "op_resend": 0, "op_rmw": 0, "op_send": 3407, "op_send_bytes": 31922487, "op_w": 3348, "oplen_avg": { "avgcount": 3407, "sum": 3544 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 25, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 50, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15638, "get_or_fail_fail": 0, "get_or_fail_success": 15638, "get_started": 0, "get_sum": 4202251, "max": 104857600, "put": 15638, "put_sum": 4202251, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3407, "put_sum": 32247709, "take": 3407, "take_sum": 32247709, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3407, "put_sum": 3407, "take": 3407, "take_sum": 3407, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 15942, "get_or_fail_fail": 0, "get_or_fail_success": 15942, "get_started": 0, "get_sum": 31065255, "max": 3758096384, "put": 25, "put_sum": 31065255, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:25.250 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-06T07:33:25.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:25.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.454+0000 7f7bbea00640 1 Processor -- start 2024-03-06T07:33:25.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.454+0000 7f7bbea00640 1 -- start start 2024-03-06T07:33:25.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb815e570 0x7f7bb80b3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbea00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7bb80d2e20 0x7f7bb80cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbea00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb80b5120 con 0x7f7bb80b30b0 2024-03-06T07:33:25.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbea00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb815e950 con 0x7f7bb80d2e20 2024-03-06T07:33:25.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80b4080 con 0x7f7bb815e570 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7bb80d2e20 0x7f7bb80cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bb7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb815e570 0x7f7bb80b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7bb80d2e20 0x7f7bb80cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:46364/0 (socket says 172.21.15.6:46364) 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.456+0000 7f7bbca00640 1 -- 172.21.15.6:0/2997288310 learned_addr learned my addr 172.21.15.6:0/2997288310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.457+0000 7f7bbca00640 1 -- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb815e570 msgr2=0x7f7bb80b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.457+0000 7f7bbca00640 1 --2- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb815e570 0x7f7bb80b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.457+0000 7f7bbca00640 1 -- 172.21.15.6:0/2997288310 >> v1:172.21.15.178:6789/0 conn(0x7f7bb80b30b0 legacy=0x7f7bb80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.457+0000 7f7bbca00640 1 -- 172.21.15.6:0/2997288310 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb80b4360 con 0x7f7bb80d2e20 2024-03-06T07:33:25.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.457+0000 7f7bb7e00640 1 --2- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb815e570 0x7f7bb80b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:25.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.458+0000 7f7bbca00640 1 --2- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80cecb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0009e50 tx=0x7f7bb002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5b61efd38b94f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.459+0000 7f7bb6a00640 1 -- 172.21.15.6:0/2997288310 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb002cb00 con 0x7f7bb80d2e20 2024-03-06T07:33:25.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.459+0000 7f7bb6a00640 1 -- 172.21.15.6:0/2997288310 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb002cca0 con 0x7f7bb80d2e20 2024-03-06T07:33:25.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.459+0000 7f7bb6a00640 1 -- 172.21.15.6:0/2997288310 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0034610 con 0x7f7bb80d2e20 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.459+0000 7f7bbea00640 1 -- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb80d2e20 msgr2=0x7f7bb80cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.459+0000 7f7bbea00640 1 --2- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80cecb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0009e50 tx=0x7f7bb002bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.460+0000 7f7bbea00640 1 -- 172.21.15.6:0/2997288310 shutdown_connections 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.460+0000 7f7bbea00640 1 --2- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80cecb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.460+0000 7f7bbea00640 1 --2- 172.21.15.6:0/2997288310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb815e570 0x7f7bb80b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.460+0000 7f7bbea00640 1 -- 172.21.15.6:0/2997288310 >> 172.21.15.6:0/2997288310 conn(0x7f7bb8073070 msgr2=0x7f7bb809d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.460+0000 7f7bbea00640 1 -- 172.21.15.6:0/2997288310 shutdown_connections 2024-03-06T07:33:25.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.460+0000 7f7bbea00640 1 -- 172.21.15.6:0/2997288310 wait complete. 2024-03-06T07:33:25.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.461+0000 7f7bbea00640 1 Processor -- start 2024-03-06T07:33:25.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.462+0000 7f7bbea00640 1 -- start start 2024-03-06T07:33:25.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.462+0000 7f7bbea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb80b30b0 0x7f7bb8154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.462+0000 7f7bbea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.462+0000 7f7bbea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb815e570 0x7f7bb80b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bbea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80b4c00 con 0x7f7bb80d2e20 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bbea00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80b4920 con 0x7f7bb815e570 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bbea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb80b3ab0 con 0x7f7bb80b30b0 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52736/0 (socket says 172.21.15.6:52736) 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 -- 172.21.15.6:0/1311000373 learned_addr learned my addr 172.21.15.6:0/1311000373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7e00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb80b30b0 0x7f7bb8154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bbca00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb815e570 0x7f7bb80b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 -- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb80b30b0 msgr2=0x7f7bb8154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb80b30b0 0x7f7bb8154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 -- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb815e570 msgr2=0x7f7bb80b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb815e570 0x7f7bb80b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bb7400640 1 -- 172.21.15.6:0/1311000373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb80cd5a0 con 0x7f7bb80d2e20 2024-03-06T07:33:25.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.463+0000 7f7bbca00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb815e570 0x7f7bb80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:25.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.464+0000 7f7bb7400640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80ccbe0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7ba400c3c0 tx=0x7f7ba400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.464+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba40052d0 con 0x7f7bb80d2e20 2024-03-06T07:33:25.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.464+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba4005490 con 0x7f7bb80d2e20 2024-03-06T07:33:25.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.464+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4007260 con 0x7f7bb80d2e20 2024-03-06T07:33:25.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.464+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb80cd880 con 0x7f7bb80d2e20 2024-03-06T07:33:25.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.464+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb80cde40 con 0x7f7bb80d2e20 2024-03-06T07:33:25.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.465+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb80ce040 con 0x7f7bb80d2e20 2024-03-06T07:33:25.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.466+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ba4007400 con 0x7f7bb80d2e20 2024-03-06T07:33:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.467+0000 7f7bafe00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b9806b4d0 0x7f7b9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:25.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.467+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f7ba4089750 con 0x7f7bb80d2e20 2024-03-06T07:33:25.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.468+0000 7f7bb7e00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b9806b4d0 0x7f7b9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:25.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.469+0000 7f7bb7e00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b9806b4d0 0x7f7b9806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009560 tx=0x7f7ba8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:25.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.472+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ba4058520 con 0x7f7bb80d2e20 2024-03-06T07:33:25.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.711+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb8068070 con 0x7f7bb80d2e20 2024-03-06T07:33:25.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.712+0000 7f7bafe00640 1 -- 172.21.15.6:0/1311000373 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f7ba4057e20 con 0x7f7bb80d2e20 2024-03-06T07:33:25.714 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:25.715 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:25.715 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:25.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b9806b4d0 msgr2=0x7f7b9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b9806b4d0 0x7f7b9806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009560 tx=0x7f7ba8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:25.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 msgr2=0x7f7bb80ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:25.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80ccbe0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7ba400c3c0 tx=0x7f7ba400c880 comp rx=0 tx=0).stop 2024-03-06T07:33:25.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 shutdown_connections 2024-03-06T07:33:25.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b9806b4d0 0x7f7b9806d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bb815e570 0x7f7bb80b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bb80d2e20 0x7f7bb80ccbe0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 --2- 172.21.15.6:0/1311000373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7bb80b30b0 0x7f7bb8154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:25.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 >> 172.21.15.6:0/1311000373 conn(0x7f7bb8073070 msgr2=0x7f7bb80d2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:25.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 shutdown_connections 2024-03-06T07:33:25.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:25.715+0000 7f7bbea00640 1 -- 172.21.15.6:0/1311000373 wait complete. 2024-03-06T07:33:25.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:33:25.974 INFO:teuthology.orchestra.run.smithi006.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1409,"msgr_send_messages":1409,"msgr_recv_bytes":368058,"msgr_send_bytes":4683147,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.211607376,"msgr_running_send_time":0.102828449,"msgr_running_recv_time":0.044867155,"msgr_running_fast_dispatch_time":0.046660868,"msgr_send_messages_queue_lat":{"avgcount":1408,"sum":0.152948699,"avgtime":0.000108628},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000210,"avgtime":0.000000105},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":13218,"msgr_send_messages":24133,"msgr_recv_bytes":4878909,"msgr_send_bytes":30619587,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":3.055745783,"msgr_running_send_time":1.996952869,"msgr_running_recv_time":1.835428179,"msgr_running_fast_dispatch_time":0.052321922,"msgr_send_messages_queue_lat":{"avgcount":24132,"sum":1.854169372,"avgtime":0.000076834},"msgr_handle_ack_lat":{"avgcount":27176,"sum":0.012806950,"avgtime":0.000000471},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":66176},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1020,"msgr_send_messages":1016,"msgr_recv_bytes":367021,"msgr_send_bytes":4173546,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.152513899,"msgr_running_send_time":0.070328865,"msgr_running_recv_time":0.035855766,"msgr_running_fast_dispatch_time":0.038490446,"msgr_send_messages_queue_lat":{"avgcount":1015,"sum":0.116926391,"avgtime":0.000115198},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":106816,"msgr_send_encrypted_bytes":7904},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":137,"sum":12.755589143,"avgtime":0.093106490}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5428,"sum":1.921599207,"avgtime":0.000354016}},"mds":{"request":6410,"reply":6410,"reply_latency":{"avgcount":6410,"sum":856.710534720,"avgtime":0.133652189},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":26,"inodes_top":0,"inodes_bottom":26,"inodes_pin_tail":0,"inodes_pinned":26,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6412,"traverse_hit":6412,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":41,"ceph_cap_op_grant":7262,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":5250,"handle_client_caps_dirty":3000,"handle_client_cap_release":520,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":15942,"evex":15929,"evtrm":15929,"ev":18,"evexg":0,"evexd":15928,"segadd":18,"segex":19,"segtrm":19,"seg":1,"segmjr":1,"segexg":0,"segexd":18,"expos":35247716,"wrpos":35261744,"rdpos":4195674,"jlat":{"avgcount":25,"sum":5.216418172,"avgtime":0.208656726},"replayed":5},"mds_mem":{"ino":13,"ino+":3214,"ino-":3201,"dir":12,"dir+":13,"dir-":1,"dn":26,"dn+":6412,"dn-":6386,"cap":1,"cap+":3202,"cap-":3201,"rss":103996,"heap":248164},"mds_server":{"dispatch_client_request":6414,"dispatch_server_request":0,"handle_client_request":6410,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":846.705989876,"avgtime":0.264595621},"req_getattr_latency":{"avgcount":2,"sum":0.888896301,"avgtime":0.444448150},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.005801270,"avgtime":0.005801270},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.893749724,"avgtime":0.148958287},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.355767233,"avgtime":0.355767233},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":7.860330316,"avgtime":0.002456353},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5769,"average_load":5769,"avg_session_uptime":965,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":372453,"buffer_anon_items":138,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5672,"osdmap_items":92,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":103961,"mds_co_items":1335,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3408,"op_send_bytes":31922592,"op_resend":0,"op_reply":3408,"op_latency":{"avgcount":3408,"sum":13.403853121,"avgtime":0.003933055},"op_inflight":0,"oplen_avg":{"avgcount":3408,"sum":3546},"op":3408,"op_r":59,"op_w":3349,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":25,"osdop_read":31,"osdop_write":48,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3226,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":50,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":52,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15640,"get_sum":4202781,"get_or_fail_fail":0,"get_or_fail_success":15640,"take":0,"take_sum":0,"put":15640,"put_sum":4202781,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3408,"take_sum":32247814,"put":3408,"put_sum":32247814,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3408,"take_sum":3408,"put":3408,"put_sum":3408,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":23,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5581f45e0820":{"val":0,"max":3758096384,"get_started":0,"get":15942,"get_sum":31065255,"get_or_fail_fail":0,"get_or_fail_success":15942,"take":0,"take_sum":0,"put":25,"put_sum":31065255,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:33:25.976 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.05e-07, "sum": 2.1e-07 }, "msgr_recv_bytes": 368058, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1409, "msgr_running_fast_dispatch_time": 0.046660868, "msgr_running_recv_time": 0.044867155, "msgr_running_send_time": 0.102828449, "msgr_running_total_time": 0.211607376, "msgr_send_bytes": 4683147, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1409, "msgr_send_messages_queue_lat": { "avgcount": 1408, "avgtime": 0.000108628, "sum": 0.152948699 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 27176, "avgtime": 4.71e-07, "sum": 0.01280695 }, "msgr_recv_bytes": 4878909, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 13218, "msgr_running_fast_dispatch_time": 0.052321922, "msgr_running_recv_time": 1.835428179, "msgr_running_send_time": 1.996952869, "msgr_running_total_time": 3.055745783, "msgr_send_bytes": 30619587, "msgr_send_encrypted_bytes": 66176, "msgr_send_messages": 24133, "msgr_send_messages_queue_lat": { "avgcount": 24132, "avgtime": 7.6834e-05, "sum": 1.854169372 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 367021, "msgr_recv_encrypted_bytes": 106816, "msgr_recv_messages": 1020, "msgr_running_fast_dispatch_time": 0.038490446, "msgr_running_recv_time": 0.035855766, "msgr_running_send_time": 0.070328865, "msgr_running_total_time": 0.152513899, "msgr_send_bytes": 4173546, "msgr_send_encrypted_bytes": 7904, "msgr_send_messages": 1016, "msgr_send_messages_queue_lat": { "avgcount": 1015, "avgtime": 0.000115198, "sum": 0.116926391 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 137, "avgtime": 0.09310649, "sum": 12.755589143 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5428, "avgtime": 0.000354016, "sum": 1.921599207 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7262, "ceph_cap_op_revoke": 41, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 520, "handle_client_caps": 5250, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 26, "inodes_bottom": 26, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 26, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6410, "reply_latency": { "avgcount": 6410, "avgtime": 0.133652189, "sum": 856.71053472 }, "request": 6410, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6412, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 18, "evadd": 15942, "evex": 15929, "evexd": 15928, "evexg": 0, "evlrg": 0, "evtrm": 15929, "expos": 35247716, "jlat": { "avgcount": 25, "avgtime": 0.208656726, "sum": 5.216418172 }, "rdpos": 4195674, "replayed": 5, "seg": 1, "segadd": 18, "segex": 19, "segexd": 18, "segexg": 0, "segmjr": 1, "segtrm": 19, "wrpos": 35261744 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 26, "dn+": 6412, "dn-": 6386, "heap": 248164, "ino": 13, "ino+": 3214, "ino-": 3201, "rss": 103996 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6414, "dispatch_server_request": 0, "handle_client_request": 6410, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.264595621, "sum": 846.705989876 }, "req_getattr_latency": { "avgcount": 2, "avgtime": 0.44444815, "sum": 0.888896301 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00580127, "sum": 0.00580127 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.148958287, "sum": 0.893749724 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.355767233, "sum": 0.355767233 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.002456353, "sum": 7.860330316 } }, "mds_sessions": { "average_load": 5769, "avg_session_uptime": 965, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5769 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 372453, "buffer_anon_items": 138, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 103961, "mds_co_items": 1335, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5672, "osdmap_items": 92, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 52, "op": 3408, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3408, "avgtime": 0.003933055, "sum": 13.403853121 }, "op_pg": 0, "op_r": 59, "op_reply": 3408, "op_resend": 0, "op_rmw": 0, "op_send": 3408, "op_send_bytes": 31922592, "op_w": 3349, "oplen_avg": { "avgcount": 3408, "sum": 3546 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 25, "osdop_delete": 3226, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 50, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 48, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15640, "get_or_fail_fail": 0, "get_or_fail_success": 15640, "get_started": 0, "get_sum": 4202781, "max": 104857600, "put": 15640, "put_sum": 4202781, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3408, "put_sum": 32247814, "take": 3408, "take_sum": 32247814, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3408, "put_sum": 3408, "take": 3408, "take_sum": 3408, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 23, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5581f45e0820": { "get": 15942, "get_or_fail_fail": 0, "get_or_fail_success": 15942, "get_started": 0, "get_sum": 31065255, "max": 3758096384, "put": 25, "put_sum": 31065255, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:33:25.976 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:25.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:26.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:26.223 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:26.223 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:26.223 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:26.280 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:26.281 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:26.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:26.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.563+0000 7faf65200640 1 Processor -- start 2024-03-06T07:33:26.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.564+0000 7faf65200640 1 -- start start 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf65200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf600b46d0 0x7faf6015c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf65200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015cbc0 0x7faf600cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf65200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf600cdda0 con 0x7faf6015c2a0 2024-03-06T07:33:26.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf65200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6015d070 con 0x7faf600b46d0 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf65200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600ccd00 con 0x7faf6015cbc0 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf600b46d0 0x7faf6015c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf57e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faf600b46d0 0x7faf6015c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52738/0 (socket says 172.21.15.6:52738) 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.565+0000 7faf57e00640 1 -- 172.21.15.6:0/3659059006 learned_addr learned my addr 172.21.15.6:0/3659059006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.566+0000 7faf5e800640 1 --2- 172.21.15.6:0/3659059006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015cbc0 0x7faf600cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.566+0000 7faf57e00640 1 -- 172.21.15.6:0/3659059006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015cbc0 msgr2=0x7faf600cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:26.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.566+0000 7faf57e00640 1 --2- 172.21.15.6:0/3659059006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015cbc0 0x7faf600cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.566+0000 7faf57e00640 1 -- 172.21.15.6:0/3659059006 >> v1:172.21.15.178:6789/0 conn(0x7faf6015c2a0 legacy=0x7faf6009b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.566+0000 7faf57e00640 1 -- 172.21.15.6:0/3659059006 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600ccfe0 con 0x7faf600b46d0 2024-03-06T07:33:26.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.567+0000 7faf57e00640 1 --2- 172.21.15.6:0/3659059006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf6015c680 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf4802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cba47200079fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.567+0000 7faf5d400640 1 -- 172.21.15.6:0/3659059006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf48034070 con 0x7faf600b46d0 2024-03-06T07:33:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.567+0000 7faf5d400640 1 -- 172.21.15.6:0/3659059006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf48030840 con 0x7faf600b46d0 2024-03-06T07:33:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.567+0000 7faf5d400640 1 -- 172.21.15.6:0/3659059006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf48030c10 con 0x7faf600b46d0 2024-03-06T07:33:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 -- 172.21.15.6:0/3659059006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 msgr2=0x7faf6015c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 --2- 172.21.15.6:0/3659059006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf6015c680 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf4802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 -- 172.21.15.6:0/3659059006 shutdown_connections 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 --2- 172.21.15.6:0/3659059006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015cbc0 0x7faf600cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 --2- 172.21.15.6:0/3659059006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf6015c680 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 -- 172.21.15.6:0/3659059006 >> 172.21.15.6:0/3659059006 conn(0x7faf600b39f0 msgr2=0x7faf600b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 -- 172.21.15.6:0/3659059006 shutdown_connections 2024-03-06T07:33:26.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.568+0000 7faf65200640 1 -- 172.21.15.6:0/3659059006 wait complete. 2024-03-06T07:33:26.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.570+0000 7faf65200640 1 Processor -- start 2024-03-06T07:33:26.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.570+0000 7faf65200640 1 -- start start 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf65200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf601f8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf65200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015c2a0 0x7faf601f5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf65200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf6015cbc0 0x7faf600cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf65200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cd880 con 0x7faf600b46d0 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf65200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cd5a0 con 0x7faf6015cbc0 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf65200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600cc9f0 con 0x7faf6015c2a0 2024-03-06T07:33:26.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf601f8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf5de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf601f8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52754/0 (socket says 172.21.15.6:52754) 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.571+0000 7faf5de00640 1 -- 172.21.15.6:0/1772078144 learned_addr learned my addr 172.21.15.6:0/1772078144 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5e800640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf6015cbc0 0x7faf600cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf57e00640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015c2a0 0x7faf601f5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5de00640 1 -- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015c2a0 msgr2=0x7faf601f5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5de00640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015c2a0 0x7faf601f5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5de00640 1 -- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf6015cbc0 msgr2=0x7faf600cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5de00640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf6015cbc0 0x7faf600cbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5de00640 1 -- 172.21.15.6:0/1772078144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf601f55a0 con 0x7faf600b46d0 2024-03-06T07:33:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf5de00640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf601f8650 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7faf5000c3c0 tx=0x7faf5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf500052d0 con 0x7faf600b46d0 2024-03-06T07:33:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf50005490 con 0x7faf600b46d0 2024-03-06T07:33:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50007260 con 0x7faf600b46d0 2024-03-06T07:33:26.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.572+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf600b7690 con 0x7faf600b46d0 2024-03-06T07:33:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.573+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf601f4860 con 0x7faf600b46d0 2024-03-06T07:33:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.574+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf600b17e0 con 0x7faf600b46d0 2024-03-06T07:33:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.574+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf50007400 con 0x7faf600b46d0 2024-03-06T07:33:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.575+0000 7faf56000640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf3c06b4d0 0x7faf3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:26.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.576+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7faf50089790 con 0x7faf600b46d0 2024-03-06T07:33:26.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.576+0000 7faf57e00640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf3c06b4d0 0x7faf3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:26.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.577+0000 7faf57e00640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf3c06b4d0 0x7faf3c06d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf48036000 tx=0x7faf4803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:26.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.584+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf601f55a0 con 0x7faf600b46d0 2024-03-06T07:33:26.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.784+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf600b7bb0 con 0x7faf600b46d0 2024-03-06T07:33:26.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.786+0000 7faf56000640 1 -- 172.21.15.6:0/1772078144 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16793 (secure 0 0 0) 0x7faf50058560 con 0x7faf600b46d0 2024-03-06T07:33:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:26.788 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:00.629298+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:6836/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6835/2859903799":"2024-03-07T07:32:59.635590+0000","172.21.15.178:6834/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6834/2859903799":"2024-03-07T07:32:59.635590+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.788+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf3c06b4d0 msgr2=0x7faf3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.788+0000 7faf65200640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf3c06b4d0 0x7faf3c06d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf48036000 tx=0x7faf4803b040 comp rx=0 tx=0).stop 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.788+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 msgr2=0x7faf601f8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.788+0000 7faf65200640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf601f8650 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7faf5000c3c0 tx=0x7faf5000c880 comp rx=0 tx=0).stop 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 shutdown_connections 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf3c06b4d0 0x7faf3c06d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf6015cbc0 0x7faf600cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf6015c2a0 0x7faf601f5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 --2- 172.21.15.6:0/1772078144 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf600b46d0 0x7faf601f8650 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 >> 172.21.15.6:0/1772078144 conn(0x7faf600b39f0 msgr2=0x7faf600c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:26.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 shutdown_connections 2024-03-06T07:33:26.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:26.789+0000 7faf65200640 1 -- 172.21.15.6:0/1772078144 wait complete. 2024-03-06T07:33:26.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:45.819970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:46.926187+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:26.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:27.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.023+0000 7fe66fe00640 1 Processor -- start 2024-03-06T07:33:27.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.023+0000 7fe66fe00640 1 -- start start 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.024+0000 7fe66fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe6701617b0 0x7fe6700b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.024+0000 7fe66fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6700d23e0 0x7fe6700d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.024+0000 7fe66fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6700cea30 con 0x7fe6700d19f0 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.024+0000 7fe66fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6700d2d70 con 0x7fe6700d23e0 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.024+0000 7fe66fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6700d1f70 con 0x7fe6701617b0 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6700d23e0 0x7fe6700d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe667e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6700d19f0 0x7fe6700cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57206/0 (socket says 172.21.15.6:57206) 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6700d23e0 0x7fe6700d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52758/0 (socket says 172.21.15.6:52758) 2024-03-06T07:33:27.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe667e00640 1 -- 172.21.15.6:0/3866418252 learned_addr learned my addr 172.21.15.6:0/3866418252 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:27.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66ea00640 1 --2- 172.21.15.6:0/3866418252 >> v2:172.21.15.178:3300/0 conn(0x7fe6701617b0 0x7fe6700b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66f400640 1 -- 172.21.15.6:0/3866418252 >> v2:172.21.15.178:3300/0 conn(0x7fe6701617b0 msgr2=0x7fe6700b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66f400640 1 --2- 172.21.15.6:0/3866418252 >> v2:172.21.15.178:3300/0 conn(0x7fe6701617b0 0x7fe6700b62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66f400640 1 -- 172.21.15.6:0/3866418252 >> v1:172.21.15.6:6789/0 conn(0x7fe6700d19f0 legacy=0x7fe6700cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66f400640 1 -- 172.21.15.6:0/3866418252 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6700cecd0 con 0x7fe6700d23e0 2024-03-06T07:33:27.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.025+0000 7fe66e000640 1 -- 172.21.15.6:0/3866418252 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192652878 0 0) 0x7fe6700cea30 con 0x7fe6700d19f0 2024-03-06T07:33:27.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.027+0000 7fe66f400640 1 --2- 172.21.15.6:0/3866418252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d23e0 0x7fe6700d2830 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fe67008ccf0 tx=0x7fe66802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18829935ea274667 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.027+0000 7fe66e000640 1 -- 172.21.15.6:0/3866418252 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668039070 con 0x7fe6700d23e0 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.027+0000 7fe66e000640 1 -- 172.21.15.6:0/3866418252 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66802cc80 con 0x7fe6700d23e0 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.027+0000 7fe66e000640 1 -- 172.21.15.6:0/3866418252 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668034660 con 0x7fe6700d23e0 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.027+0000 7fe66fe00640 1 -- 172.21.15.6:0/3866418252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d23e0 msgr2=0x7fe6700d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.027+0000 7fe66fe00640 1 --2- 172.21.15.6:0/3866418252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d23e0 0x7fe6700d2830 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fe67008ccf0 tx=0x7fe66802bed0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.028+0000 7fe66fe00640 1 -- 172.21.15.6:0/3866418252 shutdown_connections 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.028+0000 7fe66fe00640 1 --2- 172.21.15.6:0/3866418252 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d23e0 0x7fe6700d2830 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.028+0000 7fe66fe00640 1 --2- 172.21.15.6:0/3866418252 >> v2:172.21.15.178:3300/0 conn(0x7fe6701617b0 0x7fe6700b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.028+0000 7fe66fe00640 1 -- 172.21.15.6:0/3866418252 >> 172.21.15.6:0/3866418252 conn(0x7fe6700d3eb0 msgr2=0x7fe6700d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.028+0000 7fe66fe00640 1 -- 172.21.15.6:0/3866418252 shutdown_connections 2024-03-06T07:33:27.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.028+0000 7fe66fe00640 1 -- 172.21.15.6:0/3866418252 wait complete. 2024-03-06T07:33:27.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.029+0000 7fe66fe00640 1 Processor -- start 2024-03-06T07:33:27.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.030+0000 7fe66fe00640 1 -- start start 2024-03-06T07:33:27.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.030+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 0x7fe6700c9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6700d23e0 0x7fe6701532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6701617b0 0x7fe6700c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6700b6810 con 0x7fe6700d19f0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6700cf290 con 0x7fe6700d23e0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6700b6d00 con 0x7fe6701617b0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 0x7fe6700c9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 0x7fe6700c9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52760/0 (socket says 172.21.15.6:52760) 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 -- 172.21.15.6:0/2047694427 learned_addr learned my addr 172.21.15.6:0/2047694427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe667e00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6700d23e0 0x7fe6701532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 -- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6701617b0 msgr2=0x7fe6700c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66f400640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6701617b0 0x7fe6700c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6701617b0 0x7fe6700c7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 -- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6700d23e0 msgr2=0x7fe6701532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6700d23e0 0x7fe6701532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe66ea00640 1 -- 172.21.15.6:0/2047694427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6700c85c0 con 0x7fe6700d19f0 2024-03-06T07:33:27.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.031+0000 7fe667e00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6700d23e0 0x7fe6701532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.032+0000 7fe66ea00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 0x7fe6700c9660 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe66000d620 tx=0x7fe66000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.032+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66000b620 con 0x7fe6700d19f0 2024-03-06T07:33:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.032+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe660019070 con 0x7fe6700d19f0 2024-03-06T07:33:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.032+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66000ba00 con 0x7fe6700d19f0 2024-03-06T07:33:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.032+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6700c88a0 con 0x7fe6700d19f0 2024-03-06T07:33:27.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.032+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6700c8e60 con 0x7fe6700d19f0 2024-03-06T07:33:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.034+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6700c9290 con 0x7fe6700d19f0 2024-03-06T07:33:27.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.035+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe66000bba0 con 0x7fe6700d19f0 2024-03-06T07:33:27.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.035+0000 7fe666a00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.035+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe66008a6f0 con 0x7fe6700d19f0 2024-03-06T07:33:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.036+0000 7fe667e00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.037+0000 7fe667e00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64c06b4d0 0x7fe64c06d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe6700d0050 tx=0x7fe658009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.045+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe6600594c0 con 0x7fe6700d19f0 2024-03-06T07:33:27.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.253+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe67004f330 con 0x7fe6700d19f0 2024-03-06T07:33:27.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.254+0000 7fe666a00640 1 -- 172.21.15.6:0/2047694427 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe660058dc0 con 0x7fe6700d19f0 2024-03-06T07:33:27.256 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64c06b4d0 msgr2=0x7fe64c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64c06b4d0 0x7fe64c06d990 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe6700d0050 tx=0x7fe658009040 comp rx=0 tx=0).stop 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 msgr2=0x7fe6700c9660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 0x7fe6700c9660 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fe66000d620 tx=0x7fe66000dae0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 shutdown_connections 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64c06b4d0 0x7fe64c06d990 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6700d23e0 0x7fe6701532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6701617b0 0x7fe6700c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 --2- 172.21.15.6:0/2047694427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6700d19f0 0x7fe6700c9660 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 >> 172.21.15.6:0/2047694427 conn(0x7fe6700d3eb0 msgr2=0x7fe67006bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 shutdown_connections 2024-03-06T07:33:27.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.258+0000 7fe66fe00640 1 -- 172.21.15.6:0/2047694427 wait complete. 2024-03-06T07:33:27.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:27.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.507+0000 7fbb30400640 1 Processor -- start 2024-03-06T07:33:27.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.507+0000 7fbb30400640 1 -- start start 2024-03-06T07:33:27.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb30400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb280aec40 0x7fbb280cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb30400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2815d500 con 0x7fbb280af020 2024-03-06T07:33:27.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb30400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb28071d70 con 0x7fbb280b58e0 2024-03-06T07:33:27.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb30400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb2815d070 con 0x7fbb280aec40 2024-03-06T07:33:27.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb2da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb280aec40 0x7fbb280cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb2da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbb280aec40 0x7fbb280cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52772/0 (socket says 172.21.15.6:52772) 2024-03-06T07:33:27.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.509+0000 7fbb2da00640 1 -- 172.21.15.6:0/1950013363 learned_addr learned my addr 172.21.15.6:0/1950013363 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:27.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.510+0000 7fbb2da00640 1 -- 172.21.15.6:0/1950013363 >> v1:172.21.15.178:6789/0 conn(0x7fbb280b58e0 legacy=0x7fbb28159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:27.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.510+0000 7fbb2da00640 1 -- 172.21.15.6:0/1950013363 >> v1:172.21.15.6:6789/0 conn(0x7fbb280af020 legacy=0x7fbb280b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.510+0000 7fbb2da00640 1 -- 172.21.15.6:0/1950013363 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2815c740 con 0x7fbb280aec40 2024-03-06T07:33:27.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.511+0000 7fbb2da00640 1 --2- 172.21.15.6:0/1950013363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280aec40 0x7fbb280cf730 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fbb2815d1f0 tx=0x7fbb1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9434f843f25cf339 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.511+0000 7fbb27e00640 1 -- 172.21.15.6:0/1950013363 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c035020 con 0x7fbb280aec40 2024-03-06T07:33:27.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.511+0000 7fbb27e00640 1 -- 172.21.15.6:0/1950013363 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c031640 con 0x7fbb280aec40 2024-03-06T07:33:27.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.511+0000 7fbb27e00640 1 -- 172.21.15.6:0/1950013363 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c02d040 con 0x7fbb280aec40 2024-03-06T07:33:27.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 -- 172.21.15.6:0/1950013363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280aec40 msgr2=0x7fbb280cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 --2- 172.21.15.6:0/1950013363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280aec40 0x7fbb280cf730 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fbb2815d1f0 tx=0x7fbb1c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 -- 172.21.15.6:0/1950013363 shutdown_connections 2024-03-06T07:33:27.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 --2- 172.21.15.6:0/1950013363 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280aec40 0x7fbb280cf730 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 -- 172.21.15.6:0/1950013363 >> 172.21.15.6:0/1950013363 conn(0x7fbb2806cfb0 msgr2=0x7fbb2806d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 -- 172.21.15.6:0/1950013363 shutdown_connections 2024-03-06T07:33:27.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.512+0000 7fbb30400640 1 -- 172.21.15.6:0/1950013363 wait complete. 2024-03-06T07:33:27.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.515+0000 7fbb30400640 1 Processor -- start 2024-03-06T07:33:27.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.515+0000 7fbb30400640 1 -- start start 2024-03-06T07:33:27.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb30400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbb280aec40 0x7fbb28147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb30400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 0x7fbb2815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb30400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbb280b58e0 0x7fbb280b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb30400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb280ccbb0 con 0x7fbb280af020 2024-03-06T07:33:27.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb30400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2815cd00 con 0x7fbb280aec40 2024-03-06T07:33:27.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb30400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb280cc810 con 0x7fbb280b58e0 2024-03-06T07:33:27.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb2d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 0x7fbb2815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb2d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 0x7fbb2815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52780/0 (socket says 172.21.15.6:52780) 2024-03-06T07:33:27.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.517+0000 7fbb2d000640 1 -- 172.21.15.6:0/3556681056 learned_addr learned my addr 172.21.15.6:0/3556681056 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:27.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2d000640 1 -- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbb280b58e0 msgr2=0x7fbb280b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2d000640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbb280b58e0 0x7fbb280b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2d000640 1 -- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbb280aec40 msgr2=0x7fbb28147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2da00640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbb280aec40 0x7fbb28147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2d000640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbb280aec40 0x7fbb28147640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2d000640 1 -- 172.21.15.6:0/3556681056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2815adf0 con 0x7fbb280af020 2024-03-06T07:33:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb2d000640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 0x7fbb2815fe30 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c440 tx=0x7fbb14001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb140052d0 con 0x7fbb280af020 2024-03-06T07:33:27.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb14005470 con 0x7fbb280af020 2024-03-06T07:33:27.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2815f0b0 con 0x7fbb280af020 2024-03-06T07:33:27.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.518+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14007260 con 0x7fbb280af020 2024-03-06T07:33:27.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.519+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2815f350 con 0x7fbb280af020 2024-03-06T07:33:27.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.520+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2815fb00 con 0x7fbb280af020 2024-03-06T07:33:27.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.521+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb14007400 con 0x7fbb280af020 2024-03-06T07:33:27.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.522+0000 7fbb25600640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb0806b500 0x7fbb0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:27.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.522+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fbb140896b0 con 0x7fbb280af020 2024-03-06T07:33:27.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.522+0000 7fbb2da00640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb0806b500 0x7fbb0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:27.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.524+0000 7fbb2da00640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb0806b500 0x7fbb0806d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c036000 tx=0x7fbb1c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:27.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.526+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbb14058510 con 0x7fbb280af020 2024-03-06T07:33:27.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.725+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb280cf820 con 0x7fbb280af020 2024-03-06T07:33:27.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.726+0000 7fbb25600640 1 -- 172.21.15.6:0/3556681056 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbb14057e10 con 0x7fbb280af020 2024-03-06T07:33:27.728 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:33:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb0806b500 msgr2=0x7fbb0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb0806b500 0x7fbb0806d9c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c036000 tx=0x7fbb1c03c040 comp rx=0 tx=0).stop 2024-03-06T07:33:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 msgr2=0x7fbb2815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 0x7fbb2815fe30 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c440 tx=0x7fbb14001e70 comp rx=0 tx=0).stop 2024-03-06T07:33:27.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 shutdown_connections 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbb0806b500 0x7fbb0806d9c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbb280b58e0 0x7fbb280b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbb280af020 0x7fbb2815fe30 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 --2- 172.21.15.6:0/3556681056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbb280aec40 0x7fbb28147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.728+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 >> 172.21.15.6:0/3556681056 conn(0x7fbb2806cfb0 msgr2=0x7fbb280b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.729+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 shutdown_connections 2024-03-06T07:33:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.729+0000 7fbb30400640 1 -- 172.21.15.6:0/3556681056 wait complete. 2024-03-06T07:33:27.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:27.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.997+0000 7f4dbf000640 1 Processor -- start 2024-03-06T07:33:28.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:27.998+0000 7f4dbf000640 1 -- start start 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbf000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4db80d19f0 0x7f4db80cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbf000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4db80d23e0 0x7f4db80d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbf000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db80cea30 con 0x7f4db81617b0 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbf000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80d2d70 con 0x7f4db80d23e0 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbf000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db80d1f70 con 0x7f4db80d19f0 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbe600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4db80d23e0 0x7f4db80d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbe600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4db80d23e0 0x7f4db80d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52782/0 (socket says 172.21.15.6:52782) 2024-03-06T07:33:28.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbe600640 1 -- 172.21.15.6:0/3377002533 learned_addr learned my addr 172.21.15.6:0/3377002533 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.000+0000 7f4dbd200640 1 --2- 172.21.15.6:0/3377002533 >> v2:172.21.15.178:3300/0 conn(0x7f4db80d19f0 0x7f4db80cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.001+0000 7f4dbe600640 1 -- 172.21.15.6:0/3377002533 >> v2:172.21.15.178:3300/0 conn(0x7f4db80d19f0 msgr2=0x7f4db80cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.001+0000 7f4dbe600640 1 --2- 172.21.15.6:0/3377002533 >> v2:172.21.15.178:3300/0 conn(0x7f4db80d19f0 0x7f4db80cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.001+0000 7f4dbe600640 1 -- 172.21.15.6:0/3377002533 >> v1:172.21.15.178:6789/0 conn(0x7f4db81617b0 legacy=0x7f4db80b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.001+0000 7f4dbe600640 1 -- 172.21.15.6:0/3377002533 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80cecd0 con 0x7f4db80d23e0 2024-03-06T07:33:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.002+0000 7f4dbe600640 1 --2- 172.21.15.6:0/3377002533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db80d23e0 0x7f4db80d2830 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f4db808d660 tx=0x7f4db002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b2a1ea1a7da80d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.002+0000 7f4db7e00640 1 -- 172.21.15.6:0/3377002533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0035020 con 0x7f4db80d23e0 2024-03-06T07:33:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.002+0000 7f4db7e00640 1 -- 172.21.15.6:0/3377002533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db0031870 con 0x7f4db80d23e0 2024-03-06T07:33:28.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.002+0000 7f4db7e00640 1 -- 172.21.15.6:0/3377002533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0031c40 con 0x7f4db80d23e0 2024-03-06T07:33:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.003+0000 7f4dbf000640 1 -- 172.21.15.6:0/3377002533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db80d23e0 msgr2=0x7f4db80d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.003+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3377002533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db80d23e0 0x7f4db80d2830 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f4db808d660 tx=0x7f4db002b910 comp rx=0 tx=0).stop 2024-03-06T07:33:28.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.003+0000 7f4dbf000640 1 -- 172.21.15.6:0/3377002533 shutdown_connections 2024-03-06T07:33:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.003+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3377002533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db80d23e0 0x7f4db80d2830 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.003+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3377002533 >> v2:172.21.15.178:3300/0 conn(0x7f4db80d19f0 0x7f4db80cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.003+0000 7f4dbf000640 1 -- 172.21.15.6:0/3377002533 >> 172.21.15.6:0/3377002533 conn(0x7f4db80d3eb0 msgr2=0x7f4db80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.004+0000 7f4dbf000640 1 -- 172.21.15.6:0/3377002533 shutdown_connections 2024-03-06T07:33:28.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.004+0000 7f4dbf000640 1 -- 172.21.15.6:0/3377002533 wait complete. 2024-03-06T07:33:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.005+0000 7f4dbf000640 1 Processor -- start 2024-03-06T07:33:28.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.006+0000 7f4dbf000640 1 -- start start 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.006+0000 7f4dbf000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4db80d19f0 0x7f4db815d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.006+0000 7f4dbf000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4db80d23e0 0x7f4db80cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.006+0000 7f4dbf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 0x7f4db80c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.006+0000 7f4dbf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80cfcf0 con 0x7f4db81617b0 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbf000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80cf290 con 0x7f4db80d19f0 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbf000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db80cd840 con 0x7f4db80d23e0 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 0x7f4db80c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 0x7f4db80c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52784/0 (socket says 172.21.15.6:52784) 2024-03-06T07:33:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 -- 172.21.15.6:0/3723935005 learned_addr learned my addr 172.21.15.6:0/3723935005 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbd200640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4db80d23e0 0x7f4db80cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbdc00640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4db80d19f0 0x7f4db815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 -- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4db80d23e0 msgr2=0x7f4db80cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4db80d23e0 0x7f4db80cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 -- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4db80d19f0 msgr2=0x7f4db815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4db80d19f0 0x7f4db815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbe600640 1 -- 172.21.15.6:0/3723935005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db80ccc30 con 0x7f4db81617b0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.007+0000 7f4dbdc00640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4db80d19f0 0x7f4db815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.008+0000 7f4dbe600640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 0x7f4db80c9940 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f4db002b950 tx=0x7f4db00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.008+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0040070 con 0x7f4db81617b0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.008+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db0030d00 con 0x7f4db81617b0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.008+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0035040 con 0x7f4db81617b0 2024-03-06T07:33:28.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.008+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db80ccf10 con 0x7f4db81617b0 2024-03-06T07:33:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.008+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db80cd4d0 con 0x7f4db81617b0 2024-03-06T07:33:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.009+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db804f330 con 0x7f4db81617b0 2024-03-06T07:33:28.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.010+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4db0038020 con 0x7f4db81617b0 2024-03-06T07:33:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.011+0000 7f4db5600640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b500 0x7f4d9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.011+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f4db00ac0d0 con 0x7f4db81617b0 2024-03-06T07:33:28.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.012+0000 7f4dbdc00640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b500 0x7f4d9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.013+0000 7f4dbdc00640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b500 0x7f4d9c06d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4da800a830 tx=0x7f4da8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.016+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4db007ae90 con 0x7f4db81617b0 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.214+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4db808d9d0 con 0x7f4db81617b0 2024-03-06T07:33:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.215+0000 7f4db5600640 1 -- 172.21.15.6:0/3723935005 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4db007a790 con 0x7f4db81617b0 2024-03-06T07:33:28.218 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b500 msgr2=0x7f4d9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b500 0x7f4d9c06d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4da800a830 tx=0x7f4da8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 msgr2=0x7f4db80c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 0x7f4db80c9940 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f4db002b950 tx=0x7f4db00046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 shutdown_connections 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4d9c06b500 0x7f4d9c06d9c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4db81617b0 0x7f4db80c9940 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4db80d23e0 0x7f4db80cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 --2- 172.21.15.6:0/3723935005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4db80d19f0 0x7f4db815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 >> 172.21.15.6:0/3723935005 conn(0x7f4db80d3eb0 msgr2=0x7f4db806d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 shutdown_connections 2024-03-06T07:33:28.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.218+0000 7f4dbf000640 1 -- 172.21.15.6:0/3723935005 wait complete. 2024-03-06T07:33:28.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:33:28.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.455+0000 7fb887200640 1 Processor -- start 2024-03-06T07:33:28.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.455+0000 7fb887200640 1 -- start start 2024-03-06T07:33:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb887200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb880070d40 0x7fb8800af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb887200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8801605e0 0x7fb8800afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb887200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8800703d0 con 0x7fb88015f7c0 2024-03-06T07:33:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb887200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb880160ba0 con 0x7fb880070d40 2024-03-06T07:33:28.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb887200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800d0640 con 0x7fb8801605e0 2024-03-06T07:33:28.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb885200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8801605e0 0x7fb8800afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb877e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb880070d40 0x7fb8800af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb885200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8801605e0 0x7fb8800afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:53822/0 (socket says 172.21.15.6:53822) 2024-03-06T07:33:28.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.457+0000 7fb885200640 1 -- 172.21.15.6:0/2043143835 learned_addr learned my addr 172.21.15.6:0/2043143835 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:28.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.458+0000 7fb877e00640 1 -- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8801605e0 msgr2=0x7fb8800afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.458+0000 7fb877e00640 1 --2- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8801605e0 0x7fb8800afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.458+0000 7fb877e00640 1 -- 172.21.15.6:0/2043143835 >> v1:172.21.15.178:6789/0 conn(0x7fb88015f7c0 legacy=0x7fb88015fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.458+0000 7fb877e00640 1 -- 172.21.15.6:0/2043143835 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88015fba0 con 0x7fb880070d40 2024-03-06T07:33:28.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.460+0000 7fb877e00640 1 --2- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800af6c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb8680097a0 tx=0x7fb86802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7adf6f8e5aadf0f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.460+0000 7fb87f400640 1 -- 172.21.15.6:0/2043143835 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868039070 con 0x7fb880070d40 2024-03-06T07:33:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.460+0000 7fb87f400640 1 -- 172.21.15.6:0/2043143835 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8680316f0 con 0x7fb880070d40 2024-03-06T07:33:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.460+0000 7fb87f400640 1 -- 172.21.15.6:0/2043143835 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868031ac0 con 0x7fb880070d40 2024-03-06T07:33:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.461+0000 7fb887200640 1 -- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 msgr2=0x7fb8800af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.461+0000 7fb887200640 1 --2- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800af6c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb8680097a0 tx=0x7fb86802b790 comp rx=0 tx=0).stop 2024-03-06T07:33:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.461+0000 7fb887200640 1 -- 172.21.15.6:0/2043143835 shutdown_connections 2024-03-06T07:33:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.461+0000 7fb887200640 1 --2- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb8801605e0 0x7fb8800afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.461+0000 7fb887200640 1 --2- 172.21.15.6:0/2043143835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800af6c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.462+0000 7fb887200640 1 -- 172.21.15.6:0/2043143835 >> 172.21.15.6:0/2043143835 conn(0x7fb8800b39d0 msgr2=0x7fb8800b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.462+0000 7fb887200640 1 -- 172.21.15.6:0/2043143835 shutdown_connections 2024-03-06T07:33:28.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.462+0000 7fb887200640 1 -- 172.21.15.6:0/2043143835 wait complete. 2024-03-06T07:33:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.465+0000 7fb887200640 1 Processor -- start 2024-03-06T07:33:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.466+0000 7fb887200640 1 -- start start 2024-03-06T07:33:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb887200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb887200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb88015f7c0 0x7fb880154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb887200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 0x7fb8800b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb887200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb880068900 con 0x7fb8801605e0 2024-03-06T07:33:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb887200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb880068780 con 0x7fb880070d40 2024-03-06T07:33:28.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb887200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800707f0 con 0x7fb88015f7c0 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb885200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 0x7fb8800b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb885200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 0x7fb8800b2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52790/0 (socket says 172.21.15.6:52790) 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb885200640 1 -- 172.21.15.6:0/2533853720 learned_addr learned my addr 172.21.15.6:0/2533853720 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb87fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.468+0000 7fb877e00640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb88015f7c0 0x7fb880154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.469+0000 7fb885200640 1 -- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb88015f7c0 msgr2=0x7fb880154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.469+0000 7fb885200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb88015f7c0 0x7fb880154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.469+0000 7fb885200640 1 -- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 msgr2=0x7fb8800b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.469+0000 7fb885200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.469+0000 7fb885200640 1 -- 172.21.15.6:0/2533853720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8800b5290 con 0x7fb8801605e0 2024-03-06T07:33:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.469+0000 7fb885200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 0x7fb8800b2510 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb87800efc0 tx=0x7fb87800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.470+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800cd20 con 0x7fb8801605e0 2024-03-06T07:33:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.470+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878003990 con 0x7fb8801605e0 2024-03-06T07:33:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.470+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8801e5de0 con 0x7fb8801605e0 2024-03-06T07:33:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.470+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878003d80 con 0x7fb8801605e0 2024-03-06T07:33:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.470+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8801e63a0 con 0x7fb8801605e0 2024-03-06T07:33:28.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.471+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8800a0030 con 0x7fb8801605e0 2024-03-06T07:33:28.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.473+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb878005d10 con 0x7fb8801605e0 2024-03-06T07:33:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.474+0000 7fb87cc00640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:28.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.474+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fb87808a640 con 0x7fb8801605e0 2024-03-06T07:33:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.475+0000 7fb87fe00640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:28.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.478+0000 7fb87fe00640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85c06b4d0 0x7fb85c06d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb870001780 tx=0x7fb870009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.481+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb878022020 con 0x7fb8801605e0 2024-03-06T07:33:28.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.694+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb88004f410 con 0x7fb8801605e0 2024-03-06T07:33:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.695+0000 7fb87cc00640 1 -- 172.21.15.6:0/2533853720 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v97) v1 ==== 80+0+242 (secure 0 0 0) 0x7fb878010040 con 0x7fb8801605e0 2024-03-06T07:33:28.697 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.178:6836/833246901 2024-03-07T07:33:00.629222+0000 2024-03-06T07:33:28.697 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6835/2859903799 2024-03-07T07:32:59.635590+0000 2024-03-06T07:33:28.697 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.178:6834/833246901 2024-03-07T07:33:00.629222+0000 2024-03-06T07:33:28.697 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6834/2859903799 2024-03-07T07:32:59.635590+0000 2024-03-06T07:33:28.697 INFO:teuthology.orchestra.run.smithi006.stderr:listed 4 entries 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.697+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85c06b4d0 msgr2=0x7fb85c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.697+0000 7fb887200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85c06b4d0 0x7fb85c06d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb870001780 tx=0x7fb870009040 comp rx=0 tx=0).stop 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.697+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 msgr2=0x7fb8800b2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.697+0000 7fb887200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 0x7fb8800b2510 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fb87800efc0 tx=0x7fb87800c490 comp rx=0 tx=0).stop 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 shutdown_connections 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb85c06b4d0 0x7fb85c06d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb8801605e0 0x7fb8800b2510 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb88015f7c0 0x7fb880154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 --2- 172.21.15.6:0/2533853720 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb880070d40 0x7fb8800b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 >> 172.21.15.6:0/2533853720 conn(0x7fb8800b39d0 msgr2=0x7fb8800d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 shutdown_connections 2024-03-06T07:33:28.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:28.698+0000 7fb887200640 1 -- 172.21.15.6:0/2533853720 wait complete. 2024-03-06T07:33:28.721 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:33:28.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:28.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:28.857 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:33:28.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:28.858 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:33:28.901 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:33:28.901 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:28.941 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:28.942 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:28.942 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:28.982 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:28.982 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:33:28.982 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:29.032 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:29.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.240+0000 7f102fc00640 1 Processor -- start 2024-03-06T07:33:29.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.240+0000 7f102fc00640 1 -- start start 2024-03-06T07:33:29.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.242+0000 7f102fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 0x7f10280b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.242+0000 7f102fc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102806ebb0 con 0x7f1028162460 2024-03-06T07:33:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.242+0000 7f102fc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102806ee90 con 0x7f102806e260 2024-03-06T07:33:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.242+0000 7f102fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102806f450 con 0x7f10280b4910 2024-03-06T07:33:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.243+0000 7f102dc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1028162460 0x7f102806b610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57224/0 (socket says 172.21.15.6:57224) 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.243+0000 7f102dc00640 1 -- 172.21.15.6:0/952810771 learned_addr learned my addr 172.21.15.6:0/952810771 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.243+0000 7f102d200640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 0x7f10280b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.243+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605986385 0 0) 0x7f102806ebb0 con 0x7f1028162460 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.243+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c003520 con 0x7f1028162460 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175621043 0 0) 0x7f100c003520 con 0x7f1028162460 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 msgr2=0x7f10280b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 0x7f10280b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 >> v1:172.21.15.178:6789/0 conn(0x7f102806e260 legacy=0x7f1028161d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102806f170 con 0x7f1028162460 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f10200019f0 con 0x7f1028162460 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 >> v1:172.21.15.6:6789/0 conn(0x7f1028162460 legacy=0x7f102806b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 0x7f100c003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c003f80 0x7f100c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100c004d40 0x7f100c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102806f450 con 0x7f100c004d40 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100c00bb80 con 0x7f100c003f80 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f100c00bd80 con 0x7f10280b4910 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f102dc00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100c004d40 0x7f100c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f102d200640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 0x7f100c003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.244+0000 7f1027e00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c003f80 0x7f100c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.245+0000 7f102dc00640 1 -- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 msgr2=0x7f100c003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.245+0000 7f102dc00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f10280b4910 0x7f100c003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.245+0000 7f102dc00640 1 -- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c003f80 msgr2=0x7f100c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.245+0000 7f102dc00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c003f80 0x7f100c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.245+0000 7f102dc00640 1 -- 172.21.15.6:0/952810771 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c003520 con 0x7f100c004d40 2024-03-06T07:33:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.245+0000 7f1027e00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c003f80 0x7f100c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:29.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.246+0000 7f102dc00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100c004d40 0x7f100c0051c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f102000a810 tx=0x7f100c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3850faeeb7d19327 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.246+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102000a460 con 0x7f100c004d40 2024-03-06T07:33:29.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.246+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1020034650 con 0x7f100c004d40 2024-03-06T07:33:29.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.246+0000 7f1027400640 1 -- 172.21.15.6:0/952810771 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020034a80 con 0x7f100c004d40 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102fc00640 1 -- 172.21.15.6:0/952810771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100c004d40 msgr2=0x7f100c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102fc00640 1 --2- 172.21.15.6:0/952810771 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f100c004d40 0x7f100c0051c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f102000a810 tx=0x7f100c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102dc00640 1 -- 172.21.15.6:0/952810771 reap_dead start 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102fc00640 1 -- 172.21.15.6:0/952810771 shutdown_connections 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102fc00640 1 -- 172.21.15.6:0/952810771 >> 172.21.15.6:0/952810771 conn(0x7f10280b25d0 msgr2=0x7f10280b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102fc00640 1 -- 172.21.15.6:0/952810771 shutdown_connections 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.247+0000 7f102fc00640 1 -- 172.21.15.6:0/952810771 wait complete. 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.249+0000 7f102fc00640 1 Processor -- start 2024-03-06T07:33:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.249+0000 7f102fc00640 1 -- start start 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102fc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c004d40 0x7f102814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1028152780 0x7f1028153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 0x7f10281f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102806e860 con 0x7f102814f8f0 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102fc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102806e6e0 con 0x7f100c004d40 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280b4cf0 con 0x7f1028152780 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 0x7f10281f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 0x7f10281f3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52810/0 (socket says 172.21.15.6:52810) 2024-03-06T07:33:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102dc00640 1 -- 172.21.15.6:0/2268386553 learned_addr learned my addr 172.21.15.6:0/2268386553 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102d200640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c004d40 0x7f102814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f102dc00640 1 -- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1028152780 msgr2=0x7f1028153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.250+0000 7f1027e00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1028152780 0x7f1028153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102dc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1028152780 0x7f1028153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102dc00640 1 -- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c004d40 msgr2=0x7f102814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102dc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c004d40 0x7f102814fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102dc00640 1 -- 172.21.15.6:0/2268386553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10281519c0 con 0x7f102814f8f0 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f1027e00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1028152780 0x7f1028153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102dc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 0x7f10281f3f40 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f102000a810 tx=0x7f102806f340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020001fb0 con 0x7f102814f8f0 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1020044070 con 0x7f102814f8f0 2024-03-06T07:33:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1028151ca0 con 0x7f102814f8f0 2024-03-06T07:33:29.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1020034a00 con 0x7f102814f8f0 2024-03-06T07:33:29.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.251+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10281f4480 con 0x7f102814f8f0 2024-03-06T07:33:29.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.253+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1028068070 con 0x7f102814f8f0 2024-03-06T07:33:29.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.253+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1020034ba0 con 0x7f102814f8f0 2024-03-06T07:33:29.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.254+0000 7f1024c00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f100406b500 0x7f100406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.254+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f10200ac5e0 con 0x7f102814f8f0 2024-03-06T07:33:29.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.255+0000 7f102d200640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f100406b500 0x7f100406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.256+0000 7f102d200640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f100406b500 0x7f100406d9c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f10280af1e0 tx=0x7f1018008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.260+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f102007b4c0 con 0x7f102814f8f0 2024-03-06T07:33:29.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.497+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f102815e1a0 con 0x7f102814f8f0 2024-03-06T07:33:29.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.499+0000 7f1024c00640 1 -- 172.21.15.6:0/2268386553 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f1020032e30 con 0x7f102814f8f0 2024-03-06T07:33:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:29.501 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:29.501 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:29.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f100406b500 msgr2=0x7f100406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f100406b500 0x7f100406d9c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f10280af1e0 tx=0x7f1018008040 comp rx=0 tx=0).stop 2024-03-06T07:33:29.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 msgr2=0x7f10281f3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 0x7f10281f3f40 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f102000a810 tx=0x7f102806f340 comp rx=0 tx=0).stop 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 shutdown_connections 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f100406b500 0x7f100406d9c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f102814f8f0 0x7f10281f3f40 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1028152780 0x7f1028153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 --2- 172.21.15.6:0/2268386553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f100c004d40 0x7f102814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 >> 172.21.15.6:0/2268386553 conn(0x7f10280b25d0 msgr2=0x7f10280af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.501+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 shutdown_connections 2024-03-06T07:33:29.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.502+0000 7f102fc00640 1 -- 172.21.15.6:0/2268386553 wait complete. 2024-03-06T07:33:29.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:29.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.740+0000 7f2c45000640 1 Processor -- start 2024-03-06T07:33:29.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.740+0000 7f2c45000640 1 -- start start 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c45000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c4015e1c0 0x7f2c4006cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c45000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c4006d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c45000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2c4015e5a0 0x7f2c400ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c45000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c400b17f0 con 0x7f2c4015e1c0 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c45000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c400d1180 con 0x7f2c4015e5a0 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c45000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400d1460 con 0x7f2c400d0ca0 2024-03-06T07:33:29.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c3de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c4015e1c0 0x7f2c4006cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c3de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2c4015e1c0 0x7f2c4006cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52812/0 (socket says 172.21.15.6:52812) 2024-03-06T07:33:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.743+0000 7f2c3de00640 1 -- 172.21.15.6:0/4242422613 learned_addr learned my addr 172.21.15.6:0/4242422613 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c35400640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c4006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3e800640 1 --2- 172.21.15.6:0/4242422613 >> v2:172.21.15.178:3300/0 conn(0x7f2c4015e5a0 0x7f2c400ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3de00640 1 -- 172.21.15.6:0/4242422613 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 msgr2=0x7f2c4006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3de00640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c4006d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3de00640 1 -- 172.21.15.6:0/4242422613 >> v2:172.21.15.178:3300/0 conn(0x7f2c4015e5a0 msgr2=0x7f2c400ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3de00640 1 --2- 172.21.15.6:0/4242422613 >> v2:172.21.15.178:3300/0 conn(0x7f2c4015e5a0 0x7f2c400ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3de00640 1 -- 172.21.15.6:0/4242422613 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400b1a40 con 0x7f2c4015e1c0 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c3e800640 1 --2- 172.21.15.6:0/4242422613 >> v2:172.21.15.178:3300/0 conn(0x7f2c4015e5a0 0x7f2c400ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.744+0000 7f2c35400640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c4006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.745+0000 7f2c3de00640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e1c0 0x7f2c4006cae0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2c40072180 tx=0x7f2c3002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509f80686cd1f101 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.745+0000 7f2c3d400640 1 -- 172.21.15.6:0/4242422613 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30034070 con 0x7f2c4015e1c0 2024-03-06T07:33:29.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.745+0000 7f2c3d400640 1 -- 172.21.15.6:0/4242422613 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c30030850 con 0x7f2c4015e1c0 2024-03-06T07:33:29.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.745+0000 7f2c3d400640 1 -- 172.21.15.6:0/4242422613 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30030c20 con 0x7f2c4015e1c0 2024-03-06T07:33:29.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 -- 172.21.15.6:0/4242422613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e1c0 msgr2=0x7f2c4006cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e1c0 0x7f2c4006cae0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2c40072180 tx=0x7f2c3002b990 comp rx=0 tx=0).stop 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 -- 172.21.15.6:0/4242422613 shutdown_connections 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 --2- 172.21.15.6:0/4242422613 >> v2:172.21.15.178:3300/0 conn(0x7f2c4015e5a0 0x7f2c400ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c4006d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 --2- 172.21.15.6:0/4242422613 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e1c0 0x7f2c4006cae0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.746+0000 7f2c45000640 1 -- 172.21.15.6:0/4242422613 >> 172.21.15.6:0/4242422613 conn(0x7f2c400d1eb0 msgr2=0x7f2c400d22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.747+0000 7f2c45000640 1 -- 172.21.15.6:0/4242422613 shutdown_connections 2024-03-06T07:33:29.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.747+0000 7f2c45000640 1 -- 172.21.15.6:0/4242422613 wait complete. 2024-03-06T07:33:29.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.749+0000 7f2c45000640 1 Processor -- start 2024-03-06T07:33:29.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.750+0000 7f2c45000640 1 -- start start 2024-03-06T07:33:29.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.751+0000 7f2c45000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c401f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.751+0000 7f2c45000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c4015e1c0 0x7f2c401f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.751+0000 7f2c45000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 0x7f2c400b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.751+0000 7f2c45000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400d1c60 con 0x7f2c4015e5a0 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.751+0000 7f2c45000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c400d1ae0 con 0x7f2c4015e1c0 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.751+0000 7f2c45000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4006c2e0 con 0x7f2c400d0ca0 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 0x7f2c400b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 0x7f2c400b2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52816/0 (socket says 172.21.15.6:52816) 2024-03-06T07:33:29.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 -- 172.21.15.6:0/2826294416 learned_addr learned my addr 172.21.15.6:0/2826294416 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3de00640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c401f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c35400640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c4015e1c0 0x7f2c401f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 -- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 msgr2=0x7f2c401f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c401f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 -- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c4015e1c0 msgr2=0x7f2c401f4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c4015e1c0 0x7f2c401f4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.752+0000 7f2c3e800640 1 -- 172.21.15.6:0/2826294416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400b5d50 con 0x7f2c4015e5a0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.753+0000 7f2c3e800640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 0x7f2c400b2140 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800cd80 tx=0x7f2c3800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.753+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3800f680 con 0x7f2c4015e5a0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.753+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3800f840 con 0x7f2c4015e5a0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.753+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3800fc30 con 0x7f2c4015e5a0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.753+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c400b6030 con 0x7f2c4015e5a0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.753+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c400b65f0 con 0x7f2c4015e5a0 2024-03-06T07:33:29.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.755+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c400692a0 con 0x7f2c4015e5a0 2024-03-06T07:33:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.755+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c3800e3e0 con 0x7f2c4015e5a0 2024-03-06T07:33:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.756+0000 7f2c36a00640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:29.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.756+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f2c3808b0d0 con 0x7f2c4015e5a0 2024-03-06T07:33:29.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.757+0000 7f2c3de00640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:29.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.758+0000 7f2c3de00640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036000 tx=0x7f2c3003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:29.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:29.761+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c38014030 con 0x7f2c4015e5a0 2024-03-06T07:33:30.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.115+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c400b7e30 con 0x7f2c4015e5a0 2024-03-06T07:33:30.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.117+0000 7f2c36a00640 1 -- 172.21.15.6:0/2826294416 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f2c38059e00 con 0x7f2c4015e5a0 2024-03-06T07:33:30.120 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:30.120 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:00.629298+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:6836/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6835/2859903799":"2024-03-07T07:32:59.635590+0000","172.21.15.178:6834/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6834/2859903799":"2024-03-07T07:32:59.635590+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:30.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c1c06b4d0 msgr2=0x7f2c1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036000 tx=0x7f2c3003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 msgr2=0x7f2c400b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 0x7f2c400b2140 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f2c3800cd80 tx=0x7f2c3800a650 comp rx=0 tx=0).stop 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 shutdown_connections 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2c1c06b4d0 0x7f2c1c06d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2c4015e1c0 0x7f2c401f4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2c4015e5a0 0x7f2c400b2140 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 --2- 172.21.15.6:0/2826294416 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2c400d0ca0 0x7f2c401f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 >> 172.21.15.6:0/2826294416 conn(0x7f2c400d1eb0 msgr2=0x7f2c400d2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 shutdown_connections 2024-03-06T07:33:30.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.121+0000 7f2c45000640 1 -- 172.21.15.6:0/2826294416 wait complete. 2024-03-06T07:33:30.143 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:32:45.819970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:32:46.926187+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:30.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:30.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.338+0000 7f1795000640 1 Processor -- start 2024-03-06T07:33:30.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.338+0000 7f1795000640 1 -- start start 2024-03-06T07:33:30.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.339+0000 7f1795000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17900aec40 0x7f17900cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.339+0000 7f1795000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f17900af020 0x7f17900b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f1795000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179015d500 con 0x7f1790073000 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f1795000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179015d070 con 0x7f17900aec40 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f1795000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1790071d70 con 0x7f17900af020 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17900aec40 0x7f17900cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f17900aec40 0x7f17900cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52830/0 (socket says 172.21.15.6:52830) 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 -- 172.21.15.6:0/3055349853 learned_addr learned my addr 172.21.15.6:0/3055349853 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f1787e00640 1 --2- 172.21.15.6:0/3055349853 >> v2:172.21.15.178:3300/0 conn(0x7f17900af020 0x7f17900b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 -- 172.21.15.6:0/3055349853 >> v2:172.21.15.178:3300/0 conn(0x7f17900af020 msgr2=0x7f17900b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 --2- 172.21.15.6:0/3055349853 >> v2:172.21.15.178:3300/0 conn(0x7f17900af020 0x7f17900b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 -- 172.21.15.6:0/3055349853 >> v1:172.21.15.178:6789/0 conn(0x7f1790073000 legacy=0x7f1790159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f178de00640 1 -- 172.21.15.6:0/3055349853 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179015c740 con 0x7f17900aec40 2024-03-06T07:33:30.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.340+0000 7f1787e00640 1 --2- 172.21.15.6:0/3055349853 >> v2:172.21.15.178:3300/0 conn(0x7f17900af020 0x7f17900b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.342+0000 7f178de00640 1 --2- 172.21.15.6:0/3055349853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17900aec40 0x7f17900cf730 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f179004fc90 tx=0x7f178002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9030be729fc0473 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.342+0000 7f178d400640 1 -- 172.21.15.6:0/3055349853 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178002e070 con 0x7f17900aec40 2024-03-06T07:33:30.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.342+0000 7f178d400640 1 -- 172.21.15.6:0/3055349853 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780037070 con 0x7f17900aec40 2024-03-06T07:33:30.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.342+0000 7f178d400640 1 -- 172.21.15.6:0/3055349853 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780031a00 con 0x7f17900aec40 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 -- 172.21.15.6:0/3055349853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17900aec40 msgr2=0x7f17900cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 --2- 172.21.15.6:0/3055349853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17900aec40 0x7f17900cf730 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f179004fc90 tx=0x7f178002b880 comp rx=0 tx=0).stop 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 -- 172.21.15.6:0/3055349853 shutdown_connections 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 --2- 172.21.15.6:0/3055349853 >> v2:172.21.15.178:3300/0 conn(0x7f17900af020 0x7f17900b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 --2- 172.21.15.6:0/3055349853 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f17900aec40 0x7f17900cf730 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 -- 172.21.15.6:0/3055349853 >> 172.21.15.6:0/3055349853 conn(0x7f179006cfb0 msgr2=0x7f179006d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 -- 172.21.15.6:0/3055349853 shutdown_connections 2024-03-06T07:33:30.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.343+0000 7f1795000640 1 -- 172.21.15.6:0/3055349853 wait complete. 2024-03-06T07:33:30.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.345+0000 7f1795000640 1 Processor -- start 2024-03-06T07:33:30.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.345+0000 7f1795000640 1 -- start start 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f1795000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 0x7f17900adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f1795000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17900aec40 0x7f17901e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f1795000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17900af020 0x7f17901e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f1795000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17900ccbb0 con 0x7f1790073000 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f1795000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179015cd00 con 0x7f17900af020 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f1795000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17900cc810 con 0x7f17900aec40 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f178de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 0x7f17900adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f178de00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 0x7f17900adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52840/0 (socket says 172.21.15.6:52840) 2024-03-06T07:33:30.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.346+0000 7f178de00640 1 -- 172.21.15.6:0/1777679335 learned_addr learned my addr 172.21.15.6:0/1777679335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f1787e00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17900aec40 0x7f17901e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178e800640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17900af020 0x7f17901e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178de00640 1 -- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17900aec40 msgr2=0x7f17901e5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178de00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17900aec40 0x7f17901e5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178de00640 1 -- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17900af020 msgr2=0x7f17901e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178de00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17900af020 0x7f17901e5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178de00640 1 -- 172.21.15.6:0/1777679335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1790157b40 con 0x7f1790073000 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f1787e00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17900aec40 0x7f17901e5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f178de00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 0x7f17900adb20 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f1780000c00 tx=0x7f17800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780032880 con 0x7f1790073000 2024-03-06T07:33:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780032a20 con 0x7f1790073000 2024-03-06T07:33:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.347+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17901e6590 con 0x7f1790073000 2024-03-06T07:33:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.348+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780032e10 con 0x7f1790073000 2024-03-06T07:33:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.348+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17901e6e30 con 0x7f1790073000 2024-03-06T07:33:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.349+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17900692a0 con 0x7f1790073000 2024-03-06T07:33:30.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.351+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f178004c060 con 0x7f1790073000 2024-03-06T07:33:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.352+0000 7f1786000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f176c06b4d0 0x7f176c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.352+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f17800abce0 con 0x7f1790073000 2024-03-06T07:33:30.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.353+0000 7f1787e00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f176c06b4d0 0x7f176c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.354+0000 7f1787e00640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f176c06b4d0 0x7f176c06d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1778006fd0 tx=0x7f1778008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.357+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1780040070 con 0x7f1790073000 2024-03-06T07:33:30.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.650+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f179006eee0 con 0x7f1790073000 2024-03-06T07:33:30.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.650+0000 7f1786000640 1 -- 172.21.15.6:0/1777679335 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f178007a3c0 con 0x7f1790073000 2024-03-06T07:33:30.653 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:33:30.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f176c06b4d0 msgr2=0x7f176c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f176c06b4d0 0x7f176c06d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1778006fd0 tx=0x7f1778008040 comp rx=0 tx=0).stop 2024-03-06T07:33:30.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 msgr2=0x7f17900adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 0x7f17900adb20 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f1780000c00 tx=0x7f17800046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 shutdown_connections 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f176c06b4d0 0x7f176c06d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17900af020 0x7f17901e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f17900aec40 0x7f17901e5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 --2- 172.21.15.6:0/1777679335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1790073000 0x7f17900adb20 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 >> 172.21.15.6:0/1777679335 conn(0x7f179006cfb0 msgr2=0x7f179015fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 shutdown_connections 2024-03-06T07:33:30.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.654+0000 7f1795000640 1 -- 172.21.15.6:0/1777679335 wait complete. 2024-03-06T07:33:30.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.891+0000 7fa217200640 1 Processor -- start 2024-03-06T07:33:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.891+0000 7fa217200640 1 -- start start 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa217200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2180d19f0 0x7fa2180cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa217200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa2180d23e0 0x7fa2180d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa217200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180cea30 con 0x7fa2181617b0 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa217200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2180d2d70 con 0x7fa2180d19f0 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa217200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2180d1f70 con 0x7fa2180d23e0 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa20d400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa2180d19f0 0x7fa2180cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa215e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa2181617b0 0x7fa2180b62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57230/0 (socket says 172.21.15.6:57230) 2024-03-06T07:33:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa215e00640 1 -- 172.21.15.6:0/1053162882 learned_addr learned my addr 172.21.15.6:0/1053162882 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.893+0000 7fa216800640 1 --2- 172.21.15.6:0/1053162882 >> v2:172.21.15.178:3300/0 conn(0x7fa2180d23e0 0x7fa2180d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.894+0000 7fa20d400640 1 -- 172.21.15.6:0/1053162882 >> v2:172.21.15.178:3300/0 conn(0x7fa2180d23e0 msgr2=0x7fa2180d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.894+0000 7fa20d400640 1 --2- 172.21.15.6:0/1053162882 >> v2:172.21.15.178:3300/0 conn(0x7fa2180d23e0 0x7fa2180d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.894+0000 7fa20d400640 1 -- 172.21.15.6:0/1053162882 >> v1:172.21.15.6:6789/0 conn(0x7fa2181617b0 legacy=0x7fa2180b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.894+0000 7fa20d400640 1 -- 172.21.15.6:0/1053162882 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180cecd0 con 0x7fa2180d19f0 2024-03-06T07:33:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.895+0000 7fa20d400640 1 --2- 172.21.15.6:0/1053162882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2180d19f0 0x7fa2180cf7b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa2180b9f40 tx=0x7fa20002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3b07863bed75d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.895+0000 7fa215400640 1 -- 172.21.15.6:0/1053162882 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200037070 con 0x7fa2180d19f0 2024-03-06T07:33:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.895+0000 7fa215400640 1 -- 172.21.15.6:0/1053162882 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa200030710 con 0x7fa2180d19f0 2024-03-06T07:33:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.895+0000 7fa215400640 1 -- 172.21.15.6:0/1053162882 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa200030ae0 con 0x7fa2180d19f0 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 -- 172.21.15.6:0/1053162882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2180d19f0 msgr2=0x7fa2180cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 --2- 172.21.15.6:0/1053162882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2180d19f0 0x7fa2180cf7b0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa2180b9f40 tx=0x7fa20002b710 comp rx=0 tx=0).stop 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 -- 172.21.15.6:0/1053162882 shutdown_connections 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 --2- 172.21.15.6:0/1053162882 >> v2:172.21.15.178:3300/0 conn(0x7fa2180d23e0 0x7fa2180d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 --2- 172.21.15.6:0/1053162882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2180d19f0 0x7fa2180cf7b0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 -- 172.21.15.6:0/1053162882 >> 172.21.15.6:0/1053162882 conn(0x7fa2180d3eb0 msgr2=0x7fa2180d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 -- 172.21.15.6:0/1053162882 shutdown_connections 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.896+0000 7fa217200640 1 -- 172.21.15.6:0/1053162882 wait complete. 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.898+0000 7fa217200640 1 Processor -- start 2024-03-06T07:33:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.898+0000 7fa217200640 1 -- start start 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa217200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2180d19f0 0x7fa2180b9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa217200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2180d23e0 0x7fa2180b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa217200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 0x7fa21815c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa217200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180cfcf0 con 0x7fa2181617b0 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa217200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180cf290 con 0x7fa2180d19f0 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa217200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2180b8a40 con 0x7fa2180d23e0 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa216800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 0x7fa21815c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa216800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 0x7fa21815c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52848/0 (socket says 172.21.15.6:52848) 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa216800640 1 -- 172.21.15.6:0/3554968322 learned_addr learned my addr 172.21.15.6:0/3554968322 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa20d400640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2180d23e0 0x7fa2180b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.899+0000 7fa215e00640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2180d19f0 0x7fa2180b9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa216800640 1 -- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2180d23e0 msgr2=0x7fa2180b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa216800640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2180d23e0 0x7fa2180b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa216800640 1 -- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2180d19f0 msgr2=0x7fa2180b9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa216800640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2180d19f0 0x7fa2180b9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa216800640 1 -- 172.21.15.6:0/3554968322 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2180c97c0 con 0x7fa2181617b0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa216800640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 0x7fa21815c440 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa210007e40 tx=0x7fa21000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210014620 con 0x7fa2181617b0 2024-03-06T07:33:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa210019070 con 0x7fa2181617b0 2024-03-06T07:33:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210014a20 con 0x7fa2181617b0 2024-03-06T07:33:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2180c9aa0 con 0x7fa2181617b0 2024-03-06T07:33:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.900+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2180ca060 con 0x7fa2181617b0 2024-03-06T07:33:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.902+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21804f330 con 0x7fa2181617b0 2024-03-06T07:33:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.903+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa210014bc0 con 0x7fa2181617b0 2024-03-06T07:33:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.903+0000 7fa20ea00640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1f406b4d0 0x7fa1f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.904+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fa21008ac90 con 0x7fa2181617b0 2024-03-06T07:33:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.904+0000 7fa215e00640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1f406b4d0 0x7fa1f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.905+0000 7fa215e00640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1f406b4d0 0x7fa1f406d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa208000f60 tx=0x7fa208007c60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:30.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:30.908+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2100599c0 con 0x7fa2181617b0 2024-03-06T07:33:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.103+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa21808d9d0 con 0x7fa2181617b0 2024-03-06T07:33:31.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.104+0000 7fa20ea00640 1 -- 172.21.15.6:0/3554968322 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa2100592c0 con 0x7fa2181617b0 2024-03-06T07:33:31.106 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1f406b4d0 msgr2=0x7fa1f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1f406b4d0 0x7fa1f406d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa208000f60 tx=0x7fa208007c60 comp rx=0 tx=0).stop 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 msgr2=0x7fa21815c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 0x7fa21815c440 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa210007e40 tx=0x7fa21000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 shutdown_connections 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa1f406b4d0 0x7fa1f406d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2181617b0 0x7fa21815c440 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa2180d23e0 0x7fa2180b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 --2- 172.21.15.6:0/3554968322 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa2180d19f0 0x7fa2180b9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 >> 172.21.15.6:0/3554968322 conn(0x7fa2180d3eb0 msgr2=0x7fa21815d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.107+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 shutdown_connections 2024-03-06T07:33:31.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.108+0000 7fa217200640 1 -- 172.21.15.6:0/3554968322 wait complete. 2024-03-06T07:33:31.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:33:31.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.354+0000 7f353cc00640 1 Processor -- start 2024-03-06T07:33:31.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.354+0000 7f353cc00640 1 -- start start 2024-03-06T07:33:31.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f353cc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35380aec40 0x7f35380cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f353cc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f35380af020 0x7f35380b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f353cc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353815d500 con 0x7f3538073000 2024-03-06T07:33:31.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f353cc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f353815d070 con 0x7f35380aec40 2024-03-06T07:33:31.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f353cc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3538071d70 con 0x7f35380af020 2024-03-06T07:33:31.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f3535e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35380aec40 0x7f35380cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f3535e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f35380aec40 0x7f35380cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52852/0 (socket says 172.21.15.6:52852) 2024-03-06T07:33:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.356+0000 7f3535e00640 1 -- 172.21.15.6:0/3184483428 learned_addr learned my addr 172.21.15.6:0/3184483428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.357+0000 7f3535400640 1 --2- 172.21.15.6:0/3184483428 >> v2:172.21.15.178:3300/0 conn(0x7f35380af020 0x7f35380b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.357+0000 7f3535e00640 1 -- 172.21.15.6:0/3184483428 >> v2:172.21.15.178:3300/0 conn(0x7f35380af020 msgr2=0x7f35380b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.357+0000 7f3535e00640 1 --2- 172.21.15.6:0/3184483428 >> v2:172.21.15.178:3300/0 conn(0x7f35380af020 0x7f35380b51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.357+0000 7f3535e00640 1 -- 172.21.15.6:0/3184483428 >> v1:172.21.15.6:6789/0 conn(0x7f3538073000 legacy=0x7f3538159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.357+0000 7f3535e00640 1 -- 172.21.15.6:0/3184483428 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353815c740 con 0x7f35380aec40 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.358+0000 7f3535e00640 1 --2- 172.21.15.6:0/3184483428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35380aec40 0x7f35380cf730 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f353804fc90 tx=0x7f352802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d921c35c0abdfbce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.358+0000 7f3534a00640 1 -- 172.21.15.6:0/3184483428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352802ca80 con 0x7f35380aec40 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.358+0000 7f3534a00640 1 -- 172.21.15.6:0/3184483428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352802cc20 con 0x7f35380aec40 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.358+0000 7f3534a00640 1 -- 172.21.15.6:0/3184483428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35280343f0 con 0x7f35380aec40 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 -- 172.21.15.6:0/3184483428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35380aec40 msgr2=0x7f35380cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 --2- 172.21.15.6:0/3184483428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35380aec40 0x7f35380cf730 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f353804fc90 tx=0x7f352802bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 -- 172.21.15.6:0/3184483428 shutdown_connections 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 --2- 172.21.15.6:0/3184483428 >> v2:172.21.15.178:3300/0 conn(0x7f35380af020 0x7f35380b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 --2- 172.21.15.6:0/3184483428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f35380aec40 0x7f35380cf730 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 -- 172.21.15.6:0/3184483428 >> 172.21.15.6:0/3184483428 conn(0x7f353806cfb0 msgr2=0x7f353806d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 -- 172.21.15.6:0/3184483428 shutdown_connections 2024-03-06T07:33:31.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.359+0000 7f353cc00640 1 -- 172.21.15.6:0/3184483428 wait complete. 2024-03-06T07:33:31.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.361+0000 7f353cc00640 1 Processor -- start 2024-03-06T07:33:31.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.361+0000 7f353cc00640 1 -- start start 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f353cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 0x7f35380adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f353cc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f35380aec40 0x7f3538156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f353cc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35380af020 0x7f3538156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f353cc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35380ccbb0 con 0x7f3538073000 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f353cc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353815cd00 con 0x7f35380af020 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f353cc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35380cc810 con 0x7f35380aec40 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 0x7f35380adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 0x7f35380adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52866/0 (socket says 172.21.15.6:52866) 2024-03-06T07:33:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535e00640 1 -- 172.21.15.6:0/3326449480 learned_addr learned my addr 172.21.15.6:0/3326449480 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535400640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f35380aec40 0x7f3538156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535e00640 1 -- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f35380aec40 msgr2=0x7f3538156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535e00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f35380aec40 0x7f3538156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3535e00640 1 -- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35380af020 msgr2=0x7f3538156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.362+0000 7f3536800640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35380af020 0x7f3538156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f3535e00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35380af020 0x7f3538156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f3535e00640 1 -- 172.21.15.6:0/3326449480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35381570d0 con 0x7f3538073000 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f3536800640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35380af020 0x7f3538156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f3535e00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 0x7f35380adb20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f3528036040 tx=0x7f3528004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528043020 con 0x7f3538073000 2024-03-06T07:33:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352803e450 con 0x7f3538073000 2024-03-06T07:33:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3538146fa0 con 0x7f3538073000 2024-03-06T07:33:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528009510 con 0x7f3538073000 2024-03-06T07:33:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.363+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3538147560 con 0x7f3538073000 2024-03-06T07:33:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.364+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3538147a60 con 0x7f3538073000 2024-03-06T07:33:31.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.366+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3528046020 con 0x7f3538073000 2024-03-06T07:33:31.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.367+0000 7f352e000640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f351806b4d0 0x7f351806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.367+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f35280acb60 con 0x7f3538073000 2024-03-06T07:33:31.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.367+0000 7f3535400640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f351806b4d0 0x7f351806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.369+0000 7f3535400640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f351806b4d0 0x7f351806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f353814d480 tx=0x7f3524008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.372+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3538147a60 con 0x7f3538073000 2024-03-06T07:33:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.569+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35380692a0 con 0x7f3538073000 2024-03-06T07:33:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.570+0000 7f352e000640 1 -- 172.21.15.6:0/3326449480 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7f352804d030 con 0x7f3538073000 2024-03-06T07:33:31.572 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f351806b4d0 msgr2=0x7f351806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f351806b4d0 0x7f351806d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f353814d480 tx=0x7f3524008040 comp rx=0 tx=0).stop 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 msgr2=0x7f35380adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 0x7f35380adb20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f3528036040 tx=0x7f3528004a40 comp rx=0 tx=0).stop 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 shutdown_connections 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f351806b4d0 0x7f351806d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f35380af020 0x7f3538156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f35380aec40 0x7f3538156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 --2- 172.21.15.6:0/3326449480 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3538073000 0x7f35380adb20 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.572+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 >> 172.21.15.6:0/3326449480 conn(0x7f353806cfb0 msgr2=0x7f35380b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.573+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 shutdown_connections 2024-03-06T07:33:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.573+0000 7f353cc00640 1 -- 172.21.15.6:0/3326449480 wait complete. 2024-03-06T07:33:31.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.787+0000 7f7207e00640 1 Processor -- start 2024-03-06T07:33:31.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.788+0000 7f7207e00640 1 -- start start 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.789+0000 7f7207e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f720815bcc0 0x7f72080aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.789+0000 7f7207e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7208071c30 0x7f72080ad510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.789+0000 7f7207e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720806d980 con 0x7f720806e700 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.789+0000 7f7207e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720815c0a0 con 0x7f720815bcc0 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.789+0000 7f7207e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080af230 con 0x7f7208071c30 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f720815bcc0 0x7f72080aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f720815bcc0 0x7f72080aecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52874/0 (socket says 172.21.15.6:52874) 2024-03-06T07:33:31.792 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 -- 172.21.15.6:0/2231146936 learned_addr learned my addr 172.21.15.6:0/2231146936 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 -- 172.21.15.6:0/2231146936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7208071c30 msgr2=0x7f72080ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 --2- 172.21.15.6:0/2231146936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7208071c30 0x7f72080ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 -- 172.21.15.6:0/2231146936 >> v1:172.21.15.6:6789/0 conn(0x7f720806e700 legacy=0x7f7208071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.790+0000 7f7206a00640 1 -- 172.21.15.6:0/2231146936 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720815f490 con 0x7f720815bcc0 2024-03-06T07:33:31.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.791+0000 7f7206a00640 1 --2- 172.21.15.6:0/2231146936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720815bcc0 0x7f72080aecf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7671ccac7a4116 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.791+0000 7f7206000640 1 -- 172.21.15.6:0/2231146936 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8034070 con 0x7f720815bcc0 2024-03-06T07:33:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7206000640 1 -- 172.21.15.6:0/2231146936 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8030840 con 0x7f720815bcc0 2024-03-06T07:33:31.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7206000640 1 -- 172.21.15.6:0/2231146936 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8030c10 con 0x7f720815bcc0 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 -- 172.21.15.6:0/2231146936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720815bcc0 msgr2=0x7f72080aecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 --2- 172.21.15.6:0/2231146936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720815bcc0 0x7f72080aecf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 -- 172.21.15.6:0/2231146936 shutdown_connections 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 --2- 172.21.15.6:0/2231146936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7208071c30 0x7f72080ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 --2- 172.21.15.6:0/2231146936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720815bcc0 0x7f72080aecf0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 -- 172.21.15.6:0/2231146936 >> 172.21.15.6:0/2231146936 conn(0x7f72080b1990 msgr2=0x7f72080b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 -- 172.21.15.6:0/2231146936 shutdown_connections 2024-03-06T07:33:31.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.792+0000 7f7207e00640 1 -- 172.21.15.6:0/2231146936 wait complete. 2024-03-06T07:33:31.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.794+0000 7f7207e00640 1 Processor -- start 2024-03-06T07:33:31.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.794+0000 7f7207e00640 1 -- start start 2024-03-06T07:33:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.795+0000 7f7207e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 0x7f72080b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7207e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7208071c30 0x7f7208149390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7207e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720815bcc0 0x7f720809b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7207e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720806e1e0 con 0x7f720806e700 2024-03-06T07:33:31.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7207e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720806df00 con 0x7f7208071c30 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7207e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080722c0 con 0x7f720815bcc0 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 0x7f72080b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 0x7f72080b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:52890/0 (socket says 172.21.15.6:52890) 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 -- 172.21.15.6:0/2765554037 learned_addr learned my addr 172.21.15.6:0/2765554037 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 -- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720815bcc0 msgr2=0x7f720809b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720815bcc0 0x7f720809b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 -- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7208071c30 msgr2=0x7f7208149390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7208071c30 0x7f7208149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.796+0000 7f7206a00640 1 -- 172.21.15.6:0/2765554037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720809acb0 con 0x7f720806e700 2024-03-06T07:33:31.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.797+0000 7f7206a00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 0x7f72080b3440 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.797+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8034040 con 0x7f720806e700 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.797+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8039d90 con 0x7f720806e700 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.797+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8043610 con 0x7f720806e700 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.797+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72080b3b70 con 0x7f720806e700 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.797+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72080b4130 con 0x7f720806e700 2024-03-06T07:33:31.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.798+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720804f330 con 0x7f720806e700 2024-03-06T07:33:31.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.800+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71f804a060 con 0x7f720806e700 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.801+0000 7f71fea00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71e406b4d0 0x7f71e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.801+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f71f80ac530 con 0x7f720806e700 2024-03-06T07:33:31.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.801+0000 7f71ffe00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71e406b4d0 0x7f71e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:31.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.803+0000 7f71ffe00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71e406b4d0 0x7f71e406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f71f0000f60 tx=0x7f71f00079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:31.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:31.806+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71f803e070 con 0x7f720806e700 2024-03-06T07:33:32.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.111+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f720809b690 con 0x7f720806e700 2024-03-06T07:33:32.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.112+0000 7f71fea00640 1 -- 172.21.15.6:0/2765554037 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f71f802d330 con 0x7f720806e700 2024-03-06T07:33:32.171 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:32.172 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:32.173 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71e406b4d0 msgr2=0x7f71e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71e406b4d0 0x7f71e406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f71f0000f60 tx=0x7f71f00079d0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 msgr2=0x7f72080b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 0x7f72080b3440 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 shutdown_connections 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f71e406b4d0 0x7f71e406d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7208071c30 0x7f7208149390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f720815bcc0 0x7f720809b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 --2- 172.21.15.6:0/2765554037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f720806e700 0x7f72080b3440 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 >> 172.21.15.6:0/2765554037 conn(0x7f72080b1990 msgr2=0x7f72080b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 shutdown_connections 2024-03-06T07:33:32.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.115+0000 7f7207e00640 1 -- 172.21.15.6:0/2765554037 wait complete. 2024-03-06T07:33:32.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.385+0000 7fbb01000640 1 Processor -- start 2024-03-06T07:33:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.385+0000 7fbb01000640 1 -- start start 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbb01000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbafc0ae990 0x7fbafc0715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbb01000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0ad8f0 con 0x7fbafc162420 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbb01000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0adbd0 con 0x7fbafc071b20 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbb01000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc071f90 con 0x7fbafc0ae990 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbaf3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbafc0ae990 0x7fbafc0715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbaf3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbafc0ae990 0x7fbafc0715e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57040/0 (socket says 172.21.15.6:57040) 2024-03-06T07:33:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.387+0000 7fbaf3e00640 1 -- 172.21.15.6:0/467301996 learned_addr learned my addr 172.21.15.6:0/467301996 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.388+0000 7fbaf3e00640 1 -- 172.21.15.6:0/467301996 >> v1:172.21.15.178:6789/0 conn(0x7fbafc071b20 legacy=0x7fbafc0b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.388+0000 7fbaf3e00640 1 -- 172.21.15.6:0/467301996 >> v1:172.21.15.6:6789/0 conn(0x7fbafc162420 legacy=0x7fbafc06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.388+0000 7fbafaa00640 1 -- 172.21.15.6:0/467301996 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449226050 0 0) 0x7fbafc0ad8f0 con 0x7fbafc162420 2024-03-06T07:33:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.388+0000 7fbaf3e00640 1 -- 172.21.15.6:0/467301996 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0adeb0 con 0x7fbafc0ae990 2024-03-06T07:33:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.389+0000 7fbaf3e00640 1 --2- 172.21.15.6:0/467301996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc0ae990 0x7fbafc0715e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009870 tx=0x7fbae402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e61abb7631c4303 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.389+0000 7fbafaa00640 1 -- 172.21.15.6:0/467301996 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4034070 con 0x7fbafc0ae990 2024-03-06T07:33:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.389+0000 7fbafaa00640 1 -- 172.21.15.6:0/467301996 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbae4030860 con 0x7fbafc0ae990 2024-03-06T07:33:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.389+0000 7fbafaa00640 1 -- 172.21.15.6:0/467301996 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae4030c30 con 0x7fbafc0ae990 2024-03-06T07:33:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 -- 172.21.15.6:0/467301996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc0ae990 msgr2=0x7fbafc0715e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 --2- 172.21.15.6:0/467301996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc0ae990 0x7fbafc0715e0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009870 tx=0x7fbae402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 -- 172.21.15.6:0/467301996 shutdown_connections 2024-03-06T07:33:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 --2- 172.21.15.6:0/467301996 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc0ae990 0x7fbafc0715e0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 -- 172.21.15.6:0/467301996 >> 172.21.15.6:0/467301996 conn(0x7fbafc0b1990 msgr2=0x7fbafc0b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 -- 172.21.15.6:0/467301996 shutdown_connections 2024-03-06T07:33:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.390+0000 7fbb01000640 1 -- 172.21.15.6:0/467301996 wait complete. 2024-03-06T07:33:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.392+0000 7fbb01000640 1 Processor -- start 2024-03-06T07:33:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.392+0000 7fbb01000640 1 -- start start 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbb01000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 0x7fbafc1f8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbb01000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbafc0ae990 0x7fbafc14fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbb01000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbafc162420 0x7fbafc09d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbb01000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0ae470 con 0x7fbafc071b20 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbb01000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc06ea50 con 0x7fbafc0ae990 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbb01000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0af380 con 0x7fbafc162420 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbafb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 0x7fbafc1f8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbafb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 0x7fbafc1f8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57048/0 (socket says 172.21.15.6:57048) 2024-03-06T07:33:32.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbafb400640 1 -- 172.21.15.6:0/1398457280 learned_addr learned my addr 172.21.15.6:0/1398457280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbaf3e00640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbafc0ae990 0x7fbafc14fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbafbe00640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbafc162420 0x7fbafc09d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.393+0000 7fbafb400640 1 -- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbafc162420 msgr2=0x7fbafc09d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbafb400640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbafc162420 0x7fbafc09d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbafb400640 1 -- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbafc0ae990 msgr2=0x7fbafc14fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbafb400640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbafc0ae990 0x7fbafc14fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbafb400640 1 -- 172.21.15.6:0/1398457280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0cd950 con 0x7fbafc071b20 2024-03-06T07:33:32.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbaf3e00640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbafc0ae990 0x7fbafc14fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbafbe00640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbafc162420 0x7fbafc09d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbafb400640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 0x7fbafc1f8730 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00c4b0 tx=0x7fbaec00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec017070 con 0x7fbafc071b20 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec005480 con 0x7fbafc071b20 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc0cdc30 con 0x7fbafc071b20 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec003830 con 0x7fbafc071b20 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.394+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc0ce4d0 con 0x7fbafc071b20 2024-03-06T07:33:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.396+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc0ce020 con 0x7fbafc071b20 2024-03-06T07:33:32.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.396+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbaec01a070 con 0x7fbafc071b20 2024-03-06T07:33:32.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.397+0000 7fbaf3400640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad806b500 0x7fbad806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.397+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fbaec089760 con 0x7fbafc071b20 2024-03-06T07:33:32.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.398+0000 7fbaf3e00640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad806b500 0x7fbad806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.399+0000 7fbaf3e00640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad806b500 0x7fbad806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbae4036000 tx=0x7fbae403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.402+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbaec058400 con 0x7fbafc071b20 2024-03-06T07:33:32.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.593+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbafc04f330 con 0x7fbafc071b20 2024-03-06T07:33:32.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.595+0000 7fbaf3400640 1 -- 172.21.15.6:0/1398457280 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16793 (secure 0 0 0) 0x7fbaec057d00 con 0x7fbafc071b20 2024-03-06T07:33:32.598 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:32.598 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:00.629298+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:6836/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6835/2859903799":"2024-03-07T07:32:59.635590+0000","172.21.15.178:6834/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6834/2859903799":"2024-03-07T07:32:59.635590+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad806b500 msgr2=0x7fbad806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad806b500 0x7fbad806d9c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fbae4036000 tx=0x7fbae403b040 comp rx=0 tx=0).stop 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 msgr2=0x7fbafc1f8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 0x7fbafc1f8730 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00c4b0 tx=0x7fbaec00c970 comp rx=0 tx=0).stop 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 shutdown_connections 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbad806b500 0x7fbad806d9c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbafc0ae990 0x7fbafc14fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbafc162420 0x7fbafc09d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 --2- 172.21.15.6:0/1398457280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbafc071b20 0x7fbafc1f8730 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 >> 172.21.15.6:0/1398457280 conn(0x7fbafc0b1990 msgr2=0x7fbafc06dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 shutdown_connections 2024-03-06T07:33:32.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.598+0000 7fbb01000640 1 -- 172.21.15.6:0/1398457280 wait complete. 2024-03-06T07:33:32.618 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:33:32.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:32.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.833+0000 7f0bfde00640 1 Processor -- start 2024-03-06T07:33:32.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.834+0000 7f0bfde00640 1 -- start start 2024-03-06T07:33:32.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.835+0000 7f0bfde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bf81615c0 0x7f0bf80b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.835+0000 7f0bfde00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0bf80b58b0 0x7f0bf80b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.835+0000 7f0bfde00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf80b66c0 con 0x7f0bf80b6d80 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.835+0000 7f0bfde00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80d2330 con 0x7f0bf81615c0 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.835+0000 7f0bfde00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf80d2590 con 0x7f0bf80b58b0 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.835+0000 7f0bfca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bf81615c0 0x7f0bf80b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bfca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0bf81615c0 0x7f0bf80b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57062/0 (socket says 172.21.15.6:57062) 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bfca00640 1 -- 172.21.15.6:0/3823672478 learned_addr learned my addr 172.21.15.6:0/3823672478 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bf7e00640 1 --2- 172.21.15.6:0/3823672478 >> v2:172.21.15.178:3300/0 conn(0x7f0bf80b58b0 0x7f0bf80b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bfca00640 1 -- 172.21.15.6:0/3823672478 >> v2:172.21.15.178:3300/0 conn(0x7f0bf80b58b0 msgr2=0x7f0bf80b5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bfca00640 1 --2- 172.21.15.6:0/3823672478 >> v2:172.21.15.178:3300/0 conn(0x7f0bf80b58b0 0x7f0bf80b5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bfca00640 1 -- 172.21.15.6:0/3823672478 >> v1:172.21.15.6:6789/0 conn(0x7f0bf80b6d80 legacy=0x7f0bf80b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.836+0000 7f0bfca00640 1 -- 172.21.15.6:0/3823672478 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf80d2870 con 0x7f0bf81615c0 2024-03-06T07:33:32.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.837+0000 7f0bfca00640 1 --2- 172.21.15.6:0/3823672478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf81615c0 0x7f0bf80b4bf0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f0bf808cd80 tx=0x7f0bec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6fed66d2d7181e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.838+0000 7f0bf7400640 1 -- 172.21.15.6:0/3823672478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec02ca80 con 0x7f0bf81615c0 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.838+0000 7f0bf7400640 1 -- 172.21.15.6:0/3823672478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec02cc20 con 0x7f0bf81615c0 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.838+0000 7f0bf7400640 1 -- 172.21.15.6:0/3823672478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec0343f0 con 0x7f0bf81615c0 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.838+0000 7f0bfde00640 1 -- 172.21.15.6:0/3823672478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf81615c0 msgr2=0x7f0bf80b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.838+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3823672478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf81615c0 0x7f0bf80b4bf0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f0bf808cd80 tx=0x7f0bec02bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.838+0000 7f0bfde00640 1 -- 172.21.15.6:0/3823672478 shutdown_connections 2024-03-06T07:33:32.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.839+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3823672478 >> v2:172.21.15.178:3300/0 conn(0x7f0bf80b58b0 0x7f0bf80b5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.839+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3823672478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf81615c0 0x7f0bf80b4bf0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.839+0000 7f0bfde00640 1 -- 172.21.15.6:0/3823672478 >> 172.21.15.6:0/3823672478 conn(0x7f0bf80d3e30 msgr2=0x7f0bf80d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:32.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.839+0000 7f0bfde00640 1 -- 172.21.15.6:0/3823672478 shutdown_connections 2024-03-06T07:33:32.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.839+0000 7f0bfde00640 1 -- 172.21.15.6:0/3823672478 wait complete. 2024-03-06T07:33:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.841+0000 7f0bfde00640 1 Processor -- start 2024-03-06T07:33:32.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.841+0000 7f0bfde00640 1 -- start start 2024-03-06T07:33:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.842+0000 7f0bfde00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bf80b58b0 0x7f0bf809db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.842+0000 7f0bfde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 0x7f0bf80b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.842+0000 7f0bfde00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bf81615c0 0x7f0bf809f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.842+0000 7f0bfde00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf8161b20 con 0x7f0bf80b6d80 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.842+0000 7f0bfde00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf81619a0 con 0x7f0bf81615c0 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.842+0000 7f0bfde00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf80b6270 con 0x7f0bf80b58b0 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 0x7f0bf80b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 0x7f0bf80b4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57078/0 (socket says 172.21.15.6:57078) 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 -- 172.21.15.6:0/3522318224 learned_addr learned my addr 172.21.15.6:0/3522318224 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bfca00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bf80b58b0 0x7f0bf809db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 -- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bf80b58b0 msgr2=0x7f0bf809db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bf80b58b0 0x7f0bf809db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 -- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bf81615c0 msgr2=0x7f0bf809f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bfd400640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bf81615c0 0x7f0bf809f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bf81615c0 0x7f0bf809f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bf7e00640 1 -- 172.21.15.6:0/3522318224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf809f6f0 con 0x7f0bf80b6d80 2024-03-06T07:33:32.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bfd400640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bf81615c0 0x7f0bf809f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.843+0000 7f0bfca00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bf80b58b0 0x7f0bf809db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:32.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.844+0000 7f0bf7e00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 0x7f0bf80b4480 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f0be400d5f0 tx=0x7f0be400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.844+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40125f0 con 0x7f0bf80b6d80 2024-03-06T07:33:32.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.844+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4019070 con 0x7f0bf80b6d80 2024-03-06T07:33:32.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.844+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf809e110 con 0x7f0bf80b6d80 2024-03-06T07:33:32.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.844+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be40129d0 con 0x7f0bf80b6d80 2024-03-06T07:33:32.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.844+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf809e6d0 con 0x7f0bf80b6d80 2024-03-06T07:33:32.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.845+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf804f330 con 0x7f0bf80b6d80 2024-03-06T07:33:32.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.847+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0be4012b70 con 0x7f0bf80b6d80 2024-03-06T07:33:32.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.848+0000 7f0bf4c00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0bd806b4d0 0x7f0bd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:32.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.848+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f0be408a6a0 con 0x7f0bf80b6d80 2024-03-06T07:33:32.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.849+0000 7f0bfca00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0bd806b4d0 0x7f0bd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:32.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.850+0000 7f0bfca00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0bd806b4d0 0x7f0bd806d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0bec035000 tx=0x7f0bec030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:32.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:32.856+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0be40593d0 con 0x7f0bf80b6d80 2024-03-06T07:33:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.097+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bf8160be0 con 0x7f0bf80b6d80 2024-03-06T07:33:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.098+0000 7f0bf4c00640 1 -- 172.21.15.6:0/3522318224 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f0be4058cd0 con 0x7f0bf80b6d80 2024-03-06T07:33:33.100 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:33.100 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:33.101 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0bd806b4d0 msgr2=0x7f0bd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0bd806b4d0 0x7f0bd806d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0bec035000 tx=0x7f0bec030000 comp rx=0 tx=0).stop 2024-03-06T07:33:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 msgr2=0x7f0bf80b4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 0x7f0bf80b4480 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f0be400d5f0 tx=0x7f0be400dab0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 shutdown_connections 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0bd806b4d0 0x7f0bd806d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0bf81615c0 0x7f0bf809f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0bf80b6d80 0x7f0bf80b4480 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 --2- 172.21.15.6:0/3522318224 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0bf80b58b0 0x7f0bf809db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 >> 172.21.15.6:0/3522318224 conn(0x7f0bf80d3e30 msgr2=0x7f0bf80ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.101+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 shutdown_connections 2024-03-06T07:33:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.102+0000 7f0bfde00640 1 -- 172.21.15.6:0/3522318224 wait complete. 2024-03-06T07:33:33.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.334+0000 7fbc9b200640 1 Processor -- start 2024-03-06T07:33:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.334+0000 7fbc9b200640 1 -- start start 2024-03-06T07:33:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc9b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc9c15be60 0x7fbc9c15fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc9b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c0cff00 0x7fbc9c15ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc9b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbc9c1604c0 0x7fbc9c0ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc9b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c0afda0 con 0x7fbc9c15be60 2024-03-06T07:33:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc9b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c06f840 con 0x7fbc9c1604c0 2024-03-06T07:33:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc9b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c06fdc0 con 0x7fbc9c0cff00 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc99e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc9c15be60 0x7fbc9c15fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc99e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbc9c15be60 0x7fbc9c15fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57090/0 (socket says 172.21.15.6:57090) 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc99e00640 1 -- 172.21.15.6:0/3941067706 learned_addr learned my addr 172.21.15.6:0/3941067706 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.336+0000 7fbc99400640 1 --2- 172.21.15.6:0/3941067706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c0cff00 0x7fbc9c15ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.337+0000 7fbc9a800640 1 --2- 172.21.15.6:0/3941067706 >> v2:172.21.15.178:3300/0 conn(0x7fbc9c1604c0 0x7fbc9c0ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.337+0000 7fbc99e00640 1 -- 172.21.15.6:0/3941067706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c0cff00 msgr2=0x7fbc9c15ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.337+0000 7fbc99e00640 1 --2- 172.21.15.6:0/3941067706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c0cff00 0x7fbc9c15ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.337+0000 7fbc99e00640 1 -- 172.21.15.6:0/3941067706 >> v2:172.21.15.178:3300/0 conn(0x7fbc9c1604c0 msgr2=0x7fbc9c0ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.337+0000 7fbc99e00640 1 --2- 172.21.15.6:0/3941067706 >> v2:172.21.15.178:3300/0 conn(0x7fbc9c1604c0 0x7fbc9c0ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.337+0000 7fbc99e00640 1 -- 172.21.15.6:0/3941067706 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c06fae0 con 0x7fbc9c15be60 2024-03-06T07:33:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.338+0000 7fbc99e00640 1 --2- 172.21.15.6:0/3941067706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c15be60 0x7fbc9c15fa40 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c15f970 tx=0x7fbc8c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9500dbba02b21626 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc98a00640 1 -- 172.21.15.6:0/3941067706 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c02e070 con 0x7fbc9c15be60 2024-03-06T07:33:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc98a00640 1 -- 172.21.15.6:0/3941067706 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8c037070 con 0x7fbc9c15be60 2024-03-06T07:33:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc98a00640 1 -- 172.21.15.6:0/3941067706 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c031a00 con 0x7fbc9c15be60 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 -- 172.21.15.6:0/3941067706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c15be60 msgr2=0x7fbc9c15fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 --2- 172.21.15.6:0/3941067706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c15be60 0x7fbc9c15fa40 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c15f970 tx=0x7fbc8c02b880 comp rx=0 tx=0).stop 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 -- 172.21.15.6:0/3941067706 shutdown_connections 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 --2- 172.21.15.6:0/3941067706 >> v2:172.21.15.178:3300/0 conn(0x7fbc9c1604c0 0x7fbc9c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 --2- 172.21.15.6:0/3941067706 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c0cff00 0x7fbc9c15ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 --2- 172.21.15.6:0/3941067706 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c15be60 0x7fbc9c15fa40 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 -- 172.21.15.6:0/3941067706 >> 172.21.15.6:0/3941067706 conn(0x7fbc9c0ccf30 msgr2=0x7fbc9c0cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 -- 172.21.15.6:0/3941067706 shutdown_connections 2024-03-06T07:33:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.339+0000 7fbc9b200640 1 -- 172.21.15.6:0/3941067706 wait complete. 2024-03-06T07:33:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.341+0000 7fbc9b200640 1 Processor -- start 2024-03-06T07:33:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.341+0000 7fbc9b200640 1 -- start start 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc9c0cff00 0x7fbc9c15ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c15be60 0x7fbc9c1576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 0x7fbc9c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c15c240 con 0x7fbc9c1604c0 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c06d510 con 0x7fbc9c0cff00 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0cee60 con 0x7fbc9c15be60 2024-03-06T07:33:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.342+0000 7fbc9a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 0x7fbc9c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 0x7fbc9c1530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57092/0 (socket says 172.21.15.6:57092) 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 -- 172.21.15.6:0/1492254897 learned_addr learned my addr 172.21.15.6:0/1492254897 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc99400640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c15be60 0x7fbc9c1576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 -- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c15be60 msgr2=0x7fbc9c1576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc99e00640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc9c0cff00 0x7fbc9c15ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c15be60 0x7fbc9c1576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 -- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc9c0cff00 msgr2=0x7fbc9c15ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc9c0cff00 0x7fbc9c15ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 -- 172.21.15.6:0/1492254897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c152580 con 0x7fbc9c1604c0 2024-03-06T07:33:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.343+0000 7fbc9a800640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 0x7fbc9c1530d0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbc94007e60 tx=0x7fbc9400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.344+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94014420 con 0x7fbc9c1604c0 2024-03-06T07:33:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.344+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc94019070 con 0x7fbc9c1604c0 2024-03-06T07:33:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.344+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94014820 con 0x7fbc9c1604c0 2024-03-06T07:33:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.344+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c15f1d0 con 0x7fbc9c1604c0 2024-03-06T07:33:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.344+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c15c610 con 0x7fbc9c1604c0 2024-03-06T07:33:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.345+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c0b1f30 con 0x7fbc9c1604c0 2024-03-06T07:33:33.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.346+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc940149c0 con 0x7fbc9c1604c0 2024-03-06T07:33:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.347+0000 7fbc92000640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.347+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fbc9408aba0 con 0x7fbc9c1604c0 2024-03-06T07:33:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.348+0000 7fbc99e00640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.349+0000 7fbc99e00640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c036000 tx=0x7fbc8c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.351+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc9c0b1f30 con 0x7fbc9c1604c0 2024-03-06T07:33:33.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.593+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc9c04f330 con 0x7fbc9c1604c0 2024-03-06T07:33:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.594+0000 7fbc92000640 1 -- 172.21.15.6:0/1492254897 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fbc940598e0 con 0x7fbc9c1604c0 2024-03-06T07:33:33.596 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:33.596 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":69,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:32:47.892948+0000","modified":"2024-03-06T07:33:03.669632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/619677659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":619677659},{"type":"v1","addr":"172.21.15.6:6835","nonce":619677659}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":7}]} 2024-03-06T07:33:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 69 2024-03-06T07:33:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc7c06b4d0 msgr2=0x7fbc7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c036000 tx=0x7fbc8c008030 comp rx=0 tx=0).stop 2024-03-06T07:33:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 msgr2=0x7fbc9c1530d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 0x7fbc9c1530d0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fbc94007e60 tx=0x7fbc9400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 shutdown_connections 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbc9c1604c0 0x7fbc9c1530d0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbc9c15be60 0x7fbc9c1576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 --2- 172.21.15.6:0/1492254897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc9c0cff00 0x7fbc9c15ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 >> 172.21.15.6:0/1492254897 conn(0x7fbc9c0ccf30 msgr2=0x7fbc9c06bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.596+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 shutdown_connections 2024-03-06T07:33:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.597+0000 7fbc9b200640 1 -- 172.21.15.6:0/1492254897 wait complete. 2024-03-06T07:33:33.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.835+0000 7f7b2dc00640 1 Processor -- start 2024-03-06T07:33:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.835+0000 7f7b2dc00640 1 -- start start 2024-03-06T07:33:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b2dc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 0x7f7b280d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b2dc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b280b5120 con 0x7f7b2815e570 2024-03-06T07:33:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b2dc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2815e950 con 0x7f7b280d2c50 2024-03-06T07:33:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b2dc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b4080 con 0x7f7b280b30b0 2024-03-06T07:33:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b26a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7b2815e570 0x7f7b280b3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58616/0 (socket says 172.21.15.6:58616) 2024-03-06T07:33:33.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b26a00640 1 -- 172.21.15.6:0/1501977303 learned_addr learned my addr 172.21.15.6:0/1501977303 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:33.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.838+0000 7f7b26000640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 0x7f7b280d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910010889 0 0) 0x7f7b280b5120 con 0x7f7b2815e570 2024-03-06T07:33:33.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c003520 con 0x7f7b2815e570 2024-03-06T07:33:33.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 492509868 0 0) 0x7f7b0c003520 con 0x7f7b2815e570 2024-03-06T07:33:33.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 msgr2=0x7f7b280d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 0x7f7b280d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 >> v1:172.21.15.178:6789/0 conn(0x7f7b280d2c50 legacy=0x7f7b280cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b280b4360 con 0x7f7b2815e570 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7b18001670 con 0x7f7b2815e570 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 >> v1:172.21.15.6:6789/0 conn(0x7f7b2815e570 legacy=0x7f7b280b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 0x7f7b0c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c003fb0 0x7f7b0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b0c004d40 0x7f7b0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b4080 con 0x7f7b0c004d40 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c00bb00 con 0x7f7b0c003fb0 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.839+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c00bd00 con 0x7f7b280b30b0 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b27400640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b0c004d40 0x7f7b0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b26a00640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 0x7f7b0c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b26000640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c003fb0 0x7f7b0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b27400640 1 -- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 msgr2=0x7f7b0c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b27400640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b280b30b0 0x7f7b0c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b27400640 1 -- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c003fb0 msgr2=0x7f7b0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:33.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b27400640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c003fb0 0x7f7b0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.840+0000 7f7b27400640 1 -- 172.21.15.6:0/1501977303 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c003520 con 0x7f7b0c004d40 2024-03-06T07:33:33.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.842+0000 7f7b27400640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b0c004d40 0x7f7b0c0051c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f7b20001e50 tx=0x7f7b0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e03f97875d3206 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20001a70 con 0x7f7b0c004d40 2024-03-06T07:33:33.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20003b70 con 0x7f7b0c004d40 2024-03-06T07:33:33.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b25600640 1 -- 172.21.15.6:0/1501977303 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200050d0 con 0x7f7b0c004d40 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b2dc00640 1 -- 172.21.15.6:0/1501977303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b0c004d40 msgr2=0x7f7b0c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/1501977303 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b0c004d40 0x7f7b0c0051c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f7b20001e50 tx=0x7f7b0c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b27400640 1 -- 172.21.15.6:0/1501977303 reap_dead start 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.845+0000 7f7b2dc00640 1 -- 172.21.15.6:0/1501977303 shutdown_connections 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.846+0000 7f7b2dc00640 1 -- 172.21.15.6:0/1501977303 >> 172.21.15.6:0/1501977303 conn(0x7f7b28073070 msgr2=0x7f7b2809d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.846+0000 7f7b2dc00640 1 -- 172.21.15.6:0/1501977303 shutdown_connections 2024-03-06T07:33:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.846+0000 7f7b2dc00640 1 -- 172.21.15.6:0/1501977303 wait complete. 2024-03-06T07:33:33.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.847+0000 7f7b2dc00640 1 Processor -- start 2024-03-06T07:33:33.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.848+0000 7f7b2dc00640 1 -- start start 2024-03-06T07:33:33.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.848+0000 7f7b2dc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c004d40 0x7f7b280ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.848+0000 7f7b2dc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b2806c550 0x7f7b281522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b2dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 0x7f7b28152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b2dc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b4c00 con 0x7f7b2806c9b0 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b2dc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280b4920 con 0x7f7b0c004d40 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b2dc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b280d2710 con 0x7f7b2806c550 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 0x7f7b28152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 0x7f7b28152810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57100/0 (socket says 172.21.15.6:57100) 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 -- 172.21.15.6:0/921936450 learned_addr learned my addr 172.21.15.6:0/921936450 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b26000640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b2806c550 0x7f7b281522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b26a00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c004d40 0x7f7b280ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 -- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b2806c550 msgr2=0x7f7b281522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b2806c550 0x7f7b281522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 -- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c004d40 msgr2=0x7f7b280ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c004d40 0x7f7b280ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:33.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.849+0000 7f7b27400640 1 -- 172.21.15.6:0/921936450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b280c8a20 con 0x7f7b2806c9b0 2024-03-06T07:33:33.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.850+0000 7f7b27400640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 0x7f7b28152810 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7b20001f70 tx=0x7f7b200014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.850+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20007db0 con 0x7f7b2806c9b0 2024-03-06T07:33:33.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.850+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20003970 con 0x7f7b2806c9b0 2024-03-06T07:33:33.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.850+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b280c9970 con 0x7f7b2806c9b0 2024-03-06T07:33:33.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.850+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20006d70 con 0x7f7b2806c9b0 2024-03-06T07:33:33.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.850+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b280c9f30 con 0x7f7b2806c9b0 2024-03-06T07:33:33.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.852+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b28068070 con 0x7f7b2806c9b0 2024-03-06T07:33:33.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.852+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b20041030 con 0x7f7b2806c9b0 2024-03-06T07:33:33.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.853+0000 7f7b1ea00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b0806b4d0 0x7f7b0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:33.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.853+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f7b200ab320 con 0x7f7b2806c9b0 2024-03-06T07:33:33.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.854+0000 7f7b26a00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b0806b4d0 0x7f7b0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:33.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.855+0000 7f7b26a00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b0806b4d0 0x7f7b0806d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7b18001050 tx=0x7f7b18009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:33.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:33.862+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b20033020 con 0x7f7b2806c9b0 2024-03-06T07:33:34.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.063+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b280d24f0 con 0x7f7b2806c9b0 2024-03-06T07:33:34.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.065+0000 7f7b1ea00640 1 -- 172.21.15.6:0/921936450 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16793 (secure 0 0 0) 0x7f7b2007a0f0 con 0x7f7b2806c9b0 2024-03-06T07:33:34.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:34.068 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:00.629298+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:32:45.819970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:32:46.926187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.178:6836/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6835/2859903799":"2024-03-07T07:32:59.635590+0000","172.21.15.178:6834/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6834/2859903799":"2024-03-07T07:32:59.635590+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b0806b4d0 msgr2=0x7f7b0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b0806b4d0 0x7f7b0806d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7b18001050 tx=0x7f7b18009040 comp rx=0 tx=0).stop 2024-03-06T07:33:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 msgr2=0x7f7b28152810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 0x7f7b28152810 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f7b20001f70 tx=0x7f7b200014a0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 shutdown_connections 2024-03-06T07:33:34.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b0806b4d0 0x7f7b0806d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b2806c9b0 0x7f7b28152810 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b2806c550 0x7f7b281522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 --2- 172.21.15.6:0/921936450 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b0c004d40 0x7f7b280ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 >> 172.21.15.6:0/921936450 conn(0x7f7b28073070 msgr2=0x7f7b28152ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 shutdown_connections 2024-03-06T07:33:34.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.068+0000 7f7b2dc00640 1 -- 172.21.15.6:0/921936450 wait complete. 2024-03-06T07:33:34.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:33:34.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.300+0000 7f13cd800640 1 Processor -- start 2024-03-06T07:33:34.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.301+0000 7f13cd800640 1 -- start start 2024-03-06T07:33:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13cd800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13c815a7f0 0x7f13c8071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13cd800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f13c80aceb0 0x7f13c80b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13cd800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80ad800 con 0x7f13c80ad330 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13cd800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c80adae0 con 0x7f13c815a7f0 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13cd800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c80addc0 con 0x7f13c80aceb0 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13c6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13c815a7f0 0x7f13c8071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f13c815a7f0 0x7f13c8071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57106/0 (socket says 172.21.15.6:57106) 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.303+0000 7f13c7400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f13c80ad330 0x7f13c806e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58618/0 (socket says 172.21.15.6:58618) 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6a00640 1 -- 172.21.15.6:0/950993762 learned_addr learned my addr 172.21.15.6:0/950993762 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6000640 1 --2- 172.21.15.6:0/950993762 >> v2:172.21.15.178:3300/0 conn(0x7f13c80aceb0 0x7f13c80b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6a00640 1 -- 172.21.15.6:0/950993762 >> v2:172.21.15.178:3300/0 conn(0x7f13c80aceb0 msgr2=0x7f13c80b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6a00640 1 --2- 172.21.15.6:0/950993762 >> v2:172.21.15.178:3300/0 conn(0x7f13c80aceb0 0x7f13c80b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6a00640 1 -- 172.21.15.6:0/950993762 >> v1:172.21.15.6:6789/0 conn(0x7f13c80ad330 legacy=0x7f13c806e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c5600640 1 -- 172.21.15.6:0/950993762 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405129403 0 0) 0x7f13c80ad800 con 0x7f13c80ad330 2024-03-06T07:33:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.304+0000 7f13c6a00640 1 -- 172.21.15.6:0/950993762 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c80ae0a0 con 0x7f13c815a7f0 2024-03-06T07:33:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.306+0000 7f13c6a00640 1 --2- 172.21.15.6:0/950993762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c815a7f0 0x7f13c8071970 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f13c8070fa0 tx=0x7f13b802df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc6494f48d74bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.306+0000 7f13c5600640 1 -- 172.21.15.6:0/950993762 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8034070 con 0x7f13c815a7f0 2024-03-06T07:33:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.306+0000 7f13c5600640 1 -- 172.21.15.6:0/950993762 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b802ed00 con 0x7f13c815a7f0 2024-03-06T07:33:34.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.306+0000 7f13c5600640 1 -- 172.21.15.6:0/950993762 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8038620 con 0x7f13c815a7f0 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 -- 172.21.15.6:0/950993762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c815a7f0 msgr2=0x7f13c8071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 --2- 172.21.15.6:0/950993762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c815a7f0 0x7f13c8071970 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f13c8070fa0 tx=0x7f13b802df50 comp rx=0 tx=0).stop 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 -- 172.21.15.6:0/950993762 shutdown_connections 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 --2- 172.21.15.6:0/950993762 >> v2:172.21.15.178:3300/0 conn(0x7f13c80aceb0 0x7f13c80b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 --2- 172.21.15.6:0/950993762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c815a7f0 0x7f13c8071970 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 -- 172.21.15.6:0/950993762 >> 172.21.15.6:0/950993762 conn(0x7f13c80b1990 msgr2=0x7f13c80b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 -- 172.21.15.6:0/950993762 shutdown_connections 2024-03-06T07:33:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.307+0000 7f13cd800640 1 -- 172.21.15.6:0/950993762 wait complete. 2024-03-06T07:33:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.309+0000 7f13cd800640 1 Processor -- start 2024-03-06T07:33:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.309+0000 7f13cd800640 1 -- start start 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13cd800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 0x7f13c809aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13cd800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c80ad330 0x7f13c809d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13cd800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13c815a7f0 0x7f13c81468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13cd800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80b3350 con 0x7f13c80aceb0 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13cd800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80b31d0 con 0x7f13c80ad330 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13cd800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c806e830 con 0x7f13c815a7f0 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 0x7f13c809aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 0x7f13c809aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57114/0 (socket says 172.21.15.6:57114) 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 -- 172.21.15.6:0/923989428 learned_addr learned my addr 172.21.15.6:0/923989428 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 -- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13c815a7f0 msgr2=0x7f13c81468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13c815a7f0 0x7f13c81468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 -- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c80ad330 msgr2=0x7f13c809d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c80ad330 0x7f13c809d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.310+0000 7f13c6a00640 1 -- 172.21.15.6:0/923989428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c809de80 con 0x7f13c80aceb0 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.311+0000 7f13c6a00640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 0x7f13c809aa40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f13b800bbf0 tx=0x7f13b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.311+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8004230 con 0x7f13c80aceb0 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.311+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b80043d0 con 0x7f13c80aceb0 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.311+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c809c8a0 con 0x7f13c80aceb0 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.311+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8003810 con 0x7f13c80aceb0 2024-03-06T07:33:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.311+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c809ce60 con 0x7f13c80aceb0 2024-03-06T07:33:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.312+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c809e0f0 con 0x7f13c80aceb0 2024-03-06T07:33:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.315+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13b803f070 con 0x7f13c80aceb0 2024-03-06T07:33:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.315+0000 7f13bea00640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a806b4d0 0x7f13a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.316+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f13b80ad6b0 con 0x7f13c80aceb0 2024-03-06T07:33:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.316+0000 7f13c6000640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a806b4d0 0x7f13a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.317+0000 7f13c6000640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a806b4d0 0x7f13a806d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f13b4006fd0 tx=0x7f13b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.319+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13b807c4f0 con 0x7f13c80aceb0 2024-03-06T07:33:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.554+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f13c80b42a0 con 0x7f13c80aceb0 2024-03-06T07:33:34.557 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:33:34.555+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:33:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.558+0000 7f13bea00640 1 -- 172.21.15.6:0/923989428 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v70) v1 ==== 161+0+0 (secure 0 0 0) 0x7f13b807bdf0 con 0x7f13c80aceb0 2024-03-06T07:33:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:33:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a806b4d0 msgr2=0x7f13a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a806b4d0 0x7f13a806d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f13b4006fd0 tx=0x7f13b4008040 comp rx=0 tx=0).stop 2024-03-06T07:33:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 msgr2=0x7f13c809aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 0x7f13c809aa40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f13b800bbf0 tx=0x7f13b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 shutdown_connections 2024-03-06T07:33:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f13a806b4d0 0x7f13a806d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f13c80ad330 0x7f13c809d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13c815a7f0 0x7f13c81468a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 --2- 172.21.15.6:0/923989428 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f13c80aceb0 0x7f13c809aa40 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.562+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 >> 172.21.15.6:0/923989428 conn(0x7f13c80b1990 msgr2=0x7f13c81e9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.565+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 shutdown_connections 2024-03-06T07:33:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.565+0000 7f13cd800640 1 -- 172.21.15.6:0/923989428 wait complete. 2024-03-06T07:33:34.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:33:34.656 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:33:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.771+0000 7f50eb000640 1 Processor -- start 2024-03-06T07:33:34.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.771+0000 7f50eb000640 1 -- start start 2024-03-06T07:33:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50eb000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e4072040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50eb000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50e40ad8b0 0x7f50e40add00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50eb000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e4071010 con 0x7f50e4072580 2024-03-06T07:33:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50eb000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e4071bb0 con 0x7f50e40ad8b0 2024-03-06T07:33:34.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50eb000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4071280 con 0x7f50e415e780 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50e9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50e40ad8b0 0x7f50e40add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50e9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50e40ad8b0 0x7f50e40add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57134/0 (socket says 172.21.15.6:57134) 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50e9000640 1 -- 172.21.15.6:0/1719370879 learned_addr learned my addr 172.21.15.6:0/1719370879 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.773+0000 7f50e3e00640 1 --2- 172.21.15.6:0/1719370879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e4072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.774+0000 7f50e9000640 1 -- 172.21.15.6:0/1719370879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 msgr2=0x7f50e4072040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.774+0000 7f50e9000640 1 --2- 172.21.15.6:0/1719370879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e4072040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.774+0000 7f50e9000640 1 -- 172.21.15.6:0/1719370879 >> v1:172.21.15.178:6789/0 conn(0x7f50e4072580 legacy=0x7f50e40b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.774+0000 7f50e9000640 1 -- 172.21.15.6:0/1719370879 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4071560 con 0x7f50e40ad8b0 2024-03-06T07:33:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.775+0000 7f50e9000640 1 --2- 172.21.15.6:0/1719370879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e40ad8b0 0x7f50e40add00 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009870 tx=0x7f50dc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ed88659b78b540 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.775+0000 7f50e2a00640 1 -- 172.21.15.6:0/1719370879 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0317c0 con 0x7f50e40ad8b0 2024-03-06T07:33:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.775+0000 7f50e2a00640 1 -- 172.21.15.6:0/1719370879 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc037070 con 0x7f50e40ad8b0 2024-03-06T07:33:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.775+0000 7f50e2a00640 1 -- 172.21.15.6:0/1719370879 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc031b80 con 0x7f50e40ad8b0 2024-03-06T07:33:34.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 -- 172.21.15.6:0/1719370879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e40ad8b0 msgr2=0x7f50e40add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 --2- 172.21.15.6:0/1719370879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e40ad8b0 0x7f50e40add00 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009870 tx=0x7f50dc02b860 comp rx=0 tx=0).stop 2024-03-06T07:33:34.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 -- 172.21.15.6:0/1719370879 shutdown_connections 2024-03-06T07:33:34.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 --2- 172.21.15.6:0/1719370879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e40ad8b0 0x7f50e40add00 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 --2- 172.21.15.6:0/1719370879 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e4072040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 -- 172.21.15.6:0/1719370879 >> 172.21.15.6:0/1719370879 conn(0x7f50e406cfb0 msgr2=0x7f50e406d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 -- 172.21.15.6:0/1719370879 shutdown_connections 2024-03-06T07:33:34.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.776+0000 7f50eb000640 1 -- 172.21.15.6:0/1719370879 wait complete. 2024-03-06T07:33:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.778+0000 7f50eb000640 1 Processor -- start 2024-03-06T07:33:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.778+0000 7f50eb000640 1 -- start start 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50eb000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e40ad8b0 0x7f50e41f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50eb000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e415fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50eb000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 0x7f50e40c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50eb000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40b5ea0 con 0x7f50e4160220 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50eb000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40b3080 con 0x7f50e40ad8b0 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50eb000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4160850 con 0x7f50e415e780 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50e9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 0x7f50e40c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50e9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 0x7f50e40c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57148/0 (socket says 172.21.15.6:57148) 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50e9000640 1 -- 172.21.15.6:0/952407433 learned_addr learned my addr 172.21.15.6:0/952407433 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.779+0000 7f50e3e00640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e40ad8b0 0x7f50e41f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e3400640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e415fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e9000640 1 -- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 msgr2=0x7f50e415fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e9000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e415fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e9000640 1 -- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e40ad8b0 msgr2=0x7f50e41f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:34.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e9000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e40ad8b0 0x7f50e41f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:34.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e9000640 1 -- 172.21.15.6:0/952407433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4160b30 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50e9000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 0x7f50e40c4f80 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f50dc02b8a0 tx=0x7f50dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.780+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc032650 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.781+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc0327f0 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.781+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc032be0 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.781+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e4160e10 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.781+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e41613d0 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.782+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e40692a0 con 0x7f50e4160220 2024-03-06T07:33:34.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.783+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50dc04c060 con 0x7f50e4160220 2024-03-06T07:33:34.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.784+0000 7f50dbe00640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c006b4d0 0x7f50c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:34.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.784+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f50dc0abef0 con 0x7f50e4160220 2024-03-06T07:33:34.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.784+0000 7f50e3e00640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c006b4d0 0x7f50c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.785+0000 7f50e3e00640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c006b4d0 0x7f50c006d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f50d40099c0 tx=0x7f50d4003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:34.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:34.788+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50dc040070 con 0x7f50e4160220 2024-03-06T07:33:35.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.029+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f50e41f1540 con 0x7f50e4160220 2024-03-06T07:33:35.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.562+0000 7f50dbe00640 1 -- 172.21.15.6:0/952407433 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v71) v1 ==== 104+0+0 (secure 0 0 0) 0x7f50dc07abd0 con 0x7f50e4160220 2024-03-06T07:33:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c006b4d0 msgr2=0x7f50c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c006b4d0 0x7f50c006d990 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f50d40099c0 tx=0x7f50d4003990 comp rx=0 tx=0).stop 2024-03-06T07:33:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 msgr2=0x7f50e40c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 0x7f50e40c4f80 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f50dc02b8a0 tx=0x7f50dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 shutdown_connections 2024-03-06T07:33:35.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f50c006b4d0 0x7f50c006d990 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50e4160220 0x7f50e40c4f80 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50e415e780 0x7f50e415fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 --2- 172.21.15.6:0/952407433 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e40ad8b0 0x7f50e41f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.568+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 >> 172.21.15.6:0/952407433 conn(0x7f50e406cfb0 msgr2=0x7f50e40b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.569+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 shutdown_connections 2024-03-06T07:33:35.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.569+0000 7f50eb000640 1 -- 172.21.15.6:0/952407433 wait complete. 2024-03-06T07:33:35.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:33:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.811+0000 7f5523e00640 1 Processor -- start 2024-03-06T07:33:35.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.811+0000 7f5523e00640 1 -- start start 2024-03-06T07:33:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f5523e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f55240cf6f0 0x7f55240acd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f5523e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55240b4c80 con 0x7f55241442e0 2024-03-06T07:33:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f5523e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55240cfad0 con 0x7f55240b3cf0 2024-03-06T07:33:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f5523e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55240b3860 con 0x7f55240cf6f0 2024-03-06T07:33:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f551a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f55241442e0 0x7f552415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58630/0 (socket says 172.21.15.6:58630) 2024-03-06T07:33:35.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f551a000640 1 -- 172.21.15.6:0/2974475345 learned_addr learned my addr 172.21.15.6:0/2974475345 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.813+0000 7f5522a00640 1 --2- 172.21.15.6:0/2974475345 >> v2:172.21.15.178:3300/0 conn(0x7f55240cf6f0 0x7f55240acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468108865 0 0) 0x7f55240b4c80 con 0x7f55241442e0 2024-03-06T07:33:35.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5508003520 con 0x7f55241442e0 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 994604057 0 0) 0x7f5508003520 con 0x7f55241442e0 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 >> v2:172.21.15.178:3300/0 conn(0x7f55240cf6f0 msgr2=0x7f55240acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 --2- 172.21.15.6:0/2974475345 >> v2:172.21.15.178:3300/0 conn(0x7f55240cf6f0 0x7f55240acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 >> v1:172.21.15.178:6789/0 conn(0x7f55240b3cf0 legacy=0x7f552415ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55240b4f20 con 0x7f55241442e0 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f550c0019f0 con 0x7f55241442e0 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 >> v1:172.21.15.6:6789/0 conn(0x7f55241442e0 legacy=0x7f552415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55080039b0 0x7f5508003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5508004800 0x7f5508004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55080052e0 0x7f550800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55240b3860 con 0x7f5508004800 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550800e3c0 con 0x7f55080052e0 2024-03-06T07:33:35.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550800e5c0 con 0x7f55080039b0 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.814+0000 7f551a000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5508004800 0x7f5508004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f5522a00640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55080039b0 0x7f5508003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f551a000640 1 -- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55080039b0 msgr2=0x7f5508003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f551a000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55080039b0 0x7f5508003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f551a000640 1 -- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55080052e0 msgr2=0x7f550800dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f551a000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f55080052e0 0x7f550800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f551a000640 1 -- 172.21.15.6:0/2974475345 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5508003520 con 0x7f5508004800 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f5522a00640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f55080039b0 0x7f5508003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:35.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.815+0000 7f5523400640 1 -- 172.21.15.6:0/2974475345 reap_dead start 2024-03-06T07:33:35.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.816+0000 7f551a000640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5508004800 0x7f5508004c80 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f55241446c0 tx=0x7f5508005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f675b8cf7bb0637 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.816+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c00a480 con 0x7f5508004800 2024-03-06T07:33:35.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.816+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c034630 con 0x7f5508004800 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.816+0000 7f5522000640 1 -- 172.21.15.6:0/2974475345 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c034aa0 con 0x7f5508004800 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 -- 172.21.15.6:0/2974475345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5508004800 msgr2=0x7f5508004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5508004800 0x7f5508004c80 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f55241446c0 tx=0x7f5508005830 comp rx=0 tx=0).stop 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 -- 172.21.15.6:0/2974475345 shutdown_connections 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 --2- 172.21.15.6:0/2974475345 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5508004800 0x7f5508004c80 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 -- 172.21.15.6:0/2974475345 >> 172.21.15.6:0/2974475345 conn(0x7f552406b3e0 msgr2=0x7f552406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 -- 172.21.15.6:0/2974475345 shutdown_connections 2024-03-06T07:33:35.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.817+0000 7f5523e00640 1 -- 172.21.15.6:0/2974475345 wait complete. 2024-03-06T07:33:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.819+0000 7f5523e00640 1 Processor -- start 2024-03-06T07:33:35.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.819+0000 7f5523e00640 1 -- start start 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.820+0000 7f5523e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5508004800 0x7f552409bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.820+0000 7f5523e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f552415a170 0x7f5524159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.820+0000 7f5523e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 0x7f55240b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.820+0000 7f5523e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55240acd70 con 0x7f55240b0940 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.820+0000 7f5523e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55240b5200 con 0x7f5508004800 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.820+0000 7f5523e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55240acef0 con 0x7f552415a170 2024-03-06T07:33:35.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 0x7f55240b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 0x7f55240b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57168/0 (socket says 172.21.15.6:57168) 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 -- 172.21.15.6:0/1855113191 learned_addr learned my addr 172.21.15.6:0/1855113191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f551a000640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f552415a170 0x7f5524159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5522a00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5508004800 0x7f552409bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 -- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f552415a170 msgr2=0x7f5524159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f552415a170 0x7f5524159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 -- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5508004800 msgr2=0x7f552409bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5508004800 0x7f552409bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.821+0000 7f5523400640 1 -- 172.21.15.6:0/1855113191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55240b2100 con 0x7f55240b0940 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.822+0000 7f5523400640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 0x7f55240b0dc0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f55240b43f0 tx=0x7f551c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.822+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c017070 con 0x7f55240b0940 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.822+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c007500 con 0x7f55240b0940 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.822+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55240b1b40 con 0x7f55240b0940 2024-03-06T07:33:35.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.822+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c005a00 con 0x7f55240b0940 2024-03-06T07:33:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.822+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552413d7c0 con 0x7f55240b0940 2024-03-06T07:33:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.823+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552404f330 con 0x7f55240b0940 2024-03-06T07:33:35.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.824+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f551c01a070 con 0x7f55240b0940 2024-03-06T07:33:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.825+0000 7f551be00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f550006b500 0x7f550006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:35.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.825+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f551c089450 con 0x7f55240b0940 2024-03-06T07:33:35.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.826+0000 7f5522a00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f550006b500 0x7f550006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:35.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.827+0000 7f5522a00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f550006b500 0x7f550006d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5524068720 tx=0x7f5514008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:35.829+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f551c058130 con 0x7f55240b0940 2024-03-06T07:33:36.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.036+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f55240727e0 con 0x7f55240b0940 2024-03-06T07:33:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.860+0000 7f551be00640 1 -- 172.21.15.6:0/1855113191 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v99) v1 ==== 181+0+0 (secure 0 0 0) 0x7f551c057a30 con 0x7f55240b0940 2024-03-06T07:33:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:33:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.865+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f550006b500 msgr2=0x7f550006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.865+0000 7f5523e00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f550006b500 0x7f550006d9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5524068720 tx=0x7f5514008040 comp rx=0 tx=0).stop 2024-03-06T07:33:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.865+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 msgr2=0x7f55240b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:36.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.865+0000 7f5523e00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 0x7f55240b0dc0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f55240b43f0 tx=0x7f551c00c820 comp rx=0 tx=0).stop 2024-03-06T07:33:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.866+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 shutdown_connections 2024-03-06T07:33:36.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.866+0000 7f5523e00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f550006b500 0x7f550006d9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.866+0000 7f5523e00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f55240b0940 0x7f55240b0dc0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.866+0000 7f5523e00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f552415a170 0x7f5524159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.866+0000 7f5523e00640 1 --2- 172.21.15.6:0/1855113191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5508004800 0x7f552409bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:36.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.866+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 >> 172.21.15.6:0/1855113191 conn(0x7f552406b3e0 msgr2=0x7f5524140ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:36.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.868+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 shutdown_connections 2024-03-06T07:33:36.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:36.869+0000 7f5523e00640 1 -- 172.21.15.6:0/1855113191 wait complete. 2024-03-06T07:33:36.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.109+0000 7f4642a00640 1 Processor -- start 2024-03-06T07:33:37.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.109+0000 7f4642a00640 1 -- start start 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.111+0000 7f4642a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f463c15e310 0x7f463c0d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.111+0000 7f4642a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c0cfce0 con 0x7f463c0d2ad0 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.111+0000 7f4642a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463c0cffc0 con 0x7f463c0d1ce0 2024-03-06T07:33:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.111+0000 7f4642a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0cfab0 con 0x7f463c15e310 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.111+0000 7f4642000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f463c0d2ad0 0x7f463c0b1160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58636/0 (socket says 172.21.15.6:58636) 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.111+0000 7f4642000640 1 -- 172.21.15.6:0/2117778457 learned_addr learned my addr 172.21.15.6:0/2117778457 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f4641600640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f463c15e310 0x7f463c0d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131438792 0 0) 0x7f463c0cfce0 con 0x7f463c0d2ad0 2024-03-06T07:33:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4624003520 con 0x7f463c0d2ad0 2024-03-06T07:33:37.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520862433 0 0) 0x7f4624003520 con 0x7f463c0d2ad0 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f463c15e310 msgr2=0x7f463c0d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f463c15e310 0x7f463c0d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 >> v1:172.21.15.178:6789/0 conn(0x7f463c0d1ce0 legacy=0x7f463c0d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0b0c40 con 0x7f463c0d2ad0 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f46340019f0 con 0x7f463c0d2ad0 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 >> v1:172.21.15.6:6789/0 conn(0x7f463c0d2ad0 legacy=0x7f463c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240039e0 0x7f4624003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46240043d0 0x7f46240048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46240050a0 0x7f462400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462400bfb0 con 0x7f46240050a0 2024-03-06T07:33:37.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462400c130 con 0x7f46240043d0 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.112+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462400c330 con 0x7f46240039e0 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4642000640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46240050a0 0x7f462400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4641600640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240039e0 0x7f4624003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4640c00640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46240043d0 0x7f46240048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4642000640 1 -- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240039e0 msgr2=0x7f4624003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4642000640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240039e0 0x7f4624003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4642000640 1 -- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46240043d0 msgr2=0x7f46240048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4642000640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46240043d0 0x7f46240048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4642000640 1 -- 172.21.15.6:0/2117778457 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4624003520 con 0x7f46240050a0 2024-03-06T07:33:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.113+0000 7f4641600640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240039e0 0x7f4624003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.114+0000 7f4642000640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46240050a0 0x7f462400b9c0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f463400a850 tx=0x7f4624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763ec600cd7980d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463400a4a0 con 0x7f46240050a0 2024-03-06T07:33:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634034650 con 0x7f46240050a0 2024-03-06T07:33:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f463be00640 1 -- 172.21.15.6:0/2117778457 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634034a80 con 0x7f46240050a0 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642a00640 1 -- 172.21.15.6:0/2117778457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46240050a0 msgr2=0x7f462400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642a00640 1 --2- 172.21.15.6:0/2117778457 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f46240050a0 0x7f462400b9c0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f463400a850 tx=0x7f4624005830 comp rx=0 tx=0).stop 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642000640 1 -- 172.21.15.6:0/2117778457 reap_dead start 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642a00640 1 -- 172.21.15.6:0/2117778457 shutdown_connections 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642a00640 1 -- 172.21.15.6:0/2117778457 >> 172.21.15.6:0/2117778457 conn(0x7f463c0d3f00 msgr2=0x7f463c0d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642a00640 1 -- 172.21.15.6:0/2117778457 shutdown_connections 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.115+0000 7f4642a00640 1 -- 172.21.15.6:0/2117778457 wait complete. 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.117+0000 7f4642a00640 1 Processor -- start 2024-03-06T07:33:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.117+0000 7f4642a00640 1 -- start start 2024-03-06T07:33:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.118+0000 7f4642a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240050a0 0x7f463c1f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.118+0000 7f4642a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f463c15b630 0x7f463c0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.118+0000 7f4642a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 0x7f463c0b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.118+0000 7f4642a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c15e6f0 con 0x7f463c0b3980 2024-03-06T07:33:37.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.118+0000 7f4642a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c0d21a0 con 0x7f463c15b630 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.118+0000 7f4642a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463c15e870 con 0x7f46240050a0 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 0x7f463c0b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 0x7f463c0b1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57182/0 (socket says 172.21.15.6:57182) 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 -- 172.21.15.6:0/1681763382 learned_addr learned my addr 172.21.15.6:0/1681763382 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4641600640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240050a0 0x7f463c1f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4640c00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f463c15b630 0x7f463c0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 -- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240050a0 msgr2=0x7f463c1f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240050a0 0x7f463c1f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 -- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f463c15b630 msgr2=0x7f463c0b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f463c15b630 0x7f463c0b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 -- 172.21.15.6:0/1681763382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c0b3e90 con 0x7f463c0b3980 2024-03-06T07:33:37.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.119+0000 7f4642000640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 0x7f463c0b1060 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f463400a970 tx=0x7f4634001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.120+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463400a480 con 0x7f463c0b3980 2024-03-06T07:33:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.120+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634044070 con 0x7f463c0b3980 2024-03-06T07:33:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.120+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634034a00 con 0x7f463c0b3980 2024-03-06T07:33:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.120+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c0b4170 con 0x7f463c0b3980 2024-03-06T07:33:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.120+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c1f75a0 con 0x7f463c0b3980 2024-03-06T07:33:37.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.121+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c04f330 con 0x7f463c0b3980 2024-03-06T07:33:37.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.122+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4634034ba0 con 0x7f463c0b3980 2024-03-06T07:33:37.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.123+0000 7f4639600640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f461c06b4d0 0x7f461c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:37.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.123+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5529+0+0 (secure 0 0 0) 0x7f46340ac400 con 0x7f463c0b3980 2024-03-06T07:33:37.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.123+0000 7f4641600640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f461c06b4d0 0x7f461c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.125+0000 7f4641600640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f461c06b4d0 0x7f461c06d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b1ca0 tx=0x7f4630008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:37.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.127+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f463403e350 con 0x7f463c0b3980 2024-03-06T07:33:37.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.327+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f463c1f7770 con 0x7f463c0b3980 2024-03-06T07:33:37.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.870+0000 7f4639600640 1 -- 172.21.15.6:0/1681763382 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v100) v1 ==== 169+0+0 (secure 0 0 0) 0x7f463407b320 con 0x7f463c0b3980 2024-03-06T07:33:37.872 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:33:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.874+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f461c06b4d0 msgr2=0x7f461c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.874+0000 7f4642a00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f461c06b4d0 0x7f461c06d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f463c0b1ca0 tx=0x7f4630008040 comp rx=0 tx=0).stop 2024-03-06T07:33:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.874+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 msgr2=0x7f463c0b1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.874+0000 7f4642a00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 0x7f463c0b1060 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f463400a970 tx=0x7f4634001dc0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.877+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 shutdown_connections 2024-03-06T07:33:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.877+0000 7f4642a00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f461c06b4d0 0x7f461c06d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.877+0000 7f4642a00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f463c15b630 0x7f463c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.877+0000 7f4642a00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f463c0b3980 0x7f463c0b1060 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.877+0000 7f4642a00640 1 --2- 172.21.15.6:0/1681763382 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f46240050a0 0x7f463c1f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.877+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 >> 172.21.15.6:0/1681763382 conn(0x7f463c0d3f00 msgr2=0x7f463c15eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:37.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.878+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 shutdown_connections 2024-03-06T07:33:37.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:37.878+0000 7f4642a00640 1 -- 172.21.15.6:0/1681763382 wait complete. 2024-03-06T07:33:37.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:37.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:38.053 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:33:38.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:38.055 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:33:38.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:33:38.057 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:38.058 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:33:38.059 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.060 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.061 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.062 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.063 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:33:38.064 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:33:38.065 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:33:38.065 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:33:38.065 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.065 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.065 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_files = 6 2024-03-06T07:33:38.065 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.072 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:38.073 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:33:38.074 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:38.075 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:33:38.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:33:38.077 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.078 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.079 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:33:38.080 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:33:38.081 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_files = 6 2024-03-06T07:33:38.082 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:38.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:38.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:38.294 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:33:38.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:33:38.296 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.297 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:33:38.298 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:33:38.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:33:38.300 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:33:38.301 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:38.302 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:33:38.303 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.305 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi006.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:33:38.306 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:33:38.307 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:38.308 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:33:38.309 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.310 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:38.311 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:33:38.312 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:38.313 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:38.314 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:33:38.315 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:33:38.316 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.317 INFO:teuthology.orchestra.run.smithi178.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:33:38.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:38.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:33:38.521 INFO:teuthology.orchestra.run.smithi178.stdout:[global] 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: chdir = "" 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: auth supported = cephx 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: filestore xattr use omap = true 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: auth debug = true 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on old message = true 2024-03-06T07:33:38.522 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on bug = true 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default size = 2 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd allow pg remap = true 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:38.523 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: mon warn on too few osds = false 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: osd default data pool replay window = 5 2024-03-06T07:33:38.524 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: mon allow pool delete = true 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: mon cluster log file level = debug 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: debug asserts on shutdown = true 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: mon health detail to clog = false 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: ms die on skipped message = False 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout:[osd] 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: osd journal size = 100 2024-03-06T07:33:38.525 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd scrub max interval = 600 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd recover clone overlap = true 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug shutdown = true 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug op order = true 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug trim objects = true 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd open classes on start = true 2024-03-06T07:33:38.526 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug pg log writeout = true 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: osd map max advance = 10 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: journal zero on create = true 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: bdev debug aio = true 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: osd debug misdirected ops = true 2024-03-06T07:33:38.527 INFO:teuthology.orchestra.run.smithi178.stdout: bdev async discard = True 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: bdev enable discard = True 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore allocator = bitmap 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore block size = 96636764160 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: bluestore fsck on mount = True 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluefs = 1/20 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: debug bluestore = 1/20 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: debug osd = 20 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: debug rocksdb = 4/10 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:38.528 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: osd objectstore = bluestore 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout:[mgr] 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: debug mgr = 20 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:38.529 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout:[mon] 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: debug mon = 20 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: debug paxos = 20 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: debug auth = 20 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: mon data avail warn = 5 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: mon osd prime pg temp = true 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.530 INFO:teuthology.orchestra.run.smithi178.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: # 1m isn't quite enough 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: mon op complaint time = 120 2024-03-06T07:33:38.531 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout:[client] 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: rgw cache enabled = true 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable ops log = true 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: rgw enable usage log = true 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: client mount timeout = 600 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: debug client = 20 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: pid file = "" 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.a] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.b] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mon.c] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.a] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.c] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.b] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mds.d] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout:[mds] 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds = 20 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: debug mds balancer = 20 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: debug ms = 1 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug frag = True 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: mds debug scatterstat = True 2024-03-06T07:33:38.533 INFO:teuthology.orchestra.run.smithi178.stdout: mds op complaint time = 180 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi178.stdout: mds verify scatter = True 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi178.stdout: osd op complaint time = 180 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi178.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi178.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:33:38.534 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-03-06T07:33:38.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:33:38.536 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.537 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.178,[v2:172.21.15.178:3301,v1:172.21.15.178:6790]" 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on skipped message = False 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = e57361cb-720d-49c4-a325-3e55e1c43024 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:33:38.538 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:33:38.539 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: bdev async discard = True 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: bdev enable discard = True 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore allocator = bitmap 2024-03-06T07:33:38.540 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-03-06T07:33:38.541 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-03-06T07:33:38.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:33:38.543 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:33:38.544 INFO:teuthology.orchestra.run.smithi006.stdout: mon op complaint time = 120 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: client mount timeout = 600 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: debug client = 20 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = "" 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.545 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.a] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.c] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.b] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mds.d] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout:[mds] 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds = 20 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout: debug mds balancer = 20 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug frag = True 2024-03-06T07:33:38.546 INFO:teuthology.orchestra.run.smithi006.stdout: mds debug scatterstat = True 2024-03-06T07:33:38.547 INFO:teuthology.orchestra.run.smithi006.stdout: mds op complaint time = 180 2024-03-06T07:33:38.547 INFO:teuthology.orchestra.run.smithi006.stdout: mds verify scatter = True 2024-03-06T07:33:38.547 INFO:teuthology.orchestra.run.smithi006.stdout: osd op complaint time = 180 2024-03-06T07:33:38.547 INFO:teuthology.orchestra.run.smithi006.stdout: rados mon op timeout = 900 2024-03-06T07:33:38.547 INFO:teuthology.orchestra.run.smithi006.stdout: rados osd op timeout = 900 2024-03-06T07:33:38.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.798+0000 7f60bf000640 1 Processor -- start 2024-03-06T07:33:38.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.798+0000 7f60bf000640 1 -- start start 2024-03-06T07:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.799+0000 7f60bf000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f60b815dee0 0x7f60b80d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.799+0000 7f60bf000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 0x7f60b80d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.799+0000 7f60bf000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b80ceca0 con 0x7f60b80cfa60 2024-03-06T07:33:38.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.799+0000 7f60bf000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b80cf5d0 con 0x7f60b815dee0 2024-03-06T07:33:38.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.799+0000 7f60bf000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80cef80 con 0x7f60b80d4860 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.800+0000 7f60be600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f60b80cfa60 0x7f60b80b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58646/0 (socket says 172.21.15.6:58646) 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.800+0000 7f60be600640 1 -- 172.21.15.6:0/3540545670 learned_addr learned my addr 172.21.15.6:0/3540545670 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.800+0000 7f60bdc00640 1 --2- 172.21.15.6:0/3540545670 >> v2:172.21.15.178:3300/0 conn(0x7f60b815dee0 0x7f60b80d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.800+0000 7f60bd200640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 0x7f60b80d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.800+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098694982 0 0) 0x7f60b80ceca0 con 0x7f60b80cfa60 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.800+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f609c003520 con 0x7f60b80cfa60 2024-03-06T07:33:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208565804 0 0) 0x7f609c003520 con 0x7f60b80cfa60 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 msgr2=0x7f60b80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 0x7f60b80d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 >> v2:172.21.15.178:3300/0 conn(0x7f60b815dee0 msgr2=0x7f60b80d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 --2- 172.21.15.6:0/3540545670 >> v2:172.21.15.178:3300/0 conn(0x7f60b815dee0 0x7f60b80d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b80cf260 con 0x7f60b80cfa60 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f60b00019f0 con 0x7f60b80cfa60 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 >> v1:172.21.15.6:6789/0 conn(0x7f60b80cfa60 legacy=0x7f60b80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 0x7f609c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c003f80 0x7f609c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c004d40 0x7f609c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80cf5d0 con 0x7f609c003f80 2024-03-06T07:33:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80cef80 con 0x7f609c004d40 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609c00b9c0 con 0x7f60b80d4860 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60bd200640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c003f80 0x7f609c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.801+0000 7f60bdc00640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 0x7f609c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.802+0000 7f60be600640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c004d40 0x7f609c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.802+0000 7f60bd200640 1 -- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 msgr2=0x7f609c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.802+0000 7f60bd200640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b80d4860 0x7f609c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.802+0000 7f60bd200640 1 -- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c004d40 msgr2=0x7f609c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.802+0000 7f60bd200640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c004d40 0x7f609c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.802+0000 7f60bd200640 1 -- 172.21.15.6:0/3540545670 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c003520 con 0x7f609c003f80 2024-03-06T07:33:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.803+0000 7f60bd200640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c003f80 0x7f609c004800 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009ac0 tx=0x7f609c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4b91de457d6271 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.803+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4039070 con 0x7f609c003f80 2024-03-06T07:33:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.803+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a402c680 con 0x7f609c003f80 2024-03-06T07:33:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.803+0000 7f60b7e00640 1 -- 172.21.15.6:0/3540545670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4009690 con 0x7f609c003f80 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60bf000640 1 -- 172.21.15.6:0/3540545670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c003f80 msgr2=0x7f609c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60bf000640 1 --2- 172.21.15.6:0/3540545670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f609c003f80 0x7f609c004800 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009ac0 tx=0x7f609c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60be600640 1 -- 172.21.15.6:0/3540545670 reap_dead start 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60bf000640 1 -- 172.21.15.6:0/3540545670 shutdown_connections 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60bf000640 1 -- 172.21.15.6:0/3540545670 >> 172.21.15.6:0/3540545670 conn(0x7f60b80d3cd0 msgr2=0x7f60b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60bf000640 1 -- 172.21.15.6:0/3540545670 shutdown_connections 2024-03-06T07:33:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.804+0000 7f60bf000640 1 -- 172.21.15.6:0/3540545670 wait complete. 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.806+0000 7f60bf000640 1 Processor -- start 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.806+0000 7f60bf000640 1 -- start start 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60bf000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c003f80 0x7f60b80c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60bf000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b8160de0 0x7f60b8157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60bf000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 0x7f60b815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60bf000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80d1de0 con 0x7f60b815a700 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60bf000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b815e2c0 con 0x7f609c003f80 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60bf000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80d1f60 con 0x7f60b8160de0 2024-03-06T07:33:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60be600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 0x7f60b815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60be600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 0x7f60b815abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57204/0 (socket says 172.21.15.6:57204) 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.807+0000 7f60be600640 1 -- 172.21.15.6:0/872842130 learned_addr learned my addr 172.21.15.6:0/872842130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60bdc00640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c003f80 0x7f60b80c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60bd200640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b8160de0 0x7f60b8157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60be600640 1 -- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b8160de0 msgr2=0x7f60b8157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60be600640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b8160de0 0x7f60b8157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60be600640 1 -- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c003f80 msgr2=0x7f60b80c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60be600640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c003f80 0x7f60b80c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60be600640 1 -- 172.21.15.6:0/872842130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b81581d0 con 0x7f60b815a700 2024-03-06T07:33:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60be600640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 0x7f60b815abb0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f60b00013d0 tx=0x7f60b000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b000ec10 con 0x7f60b815a700 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b0018070 con 0x7f60b815a700 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0013420 con 0x7f60b815a700 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.808+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b81584b0 con 0x7f60b815a700 2024-03-06T07:33:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.809+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b815eca0 con 0x7f60b815a700 2024-03-06T07:33:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.810+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b804f330 con 0x7f60b815a700 2024-03-06T07:33:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.811+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60b00135c0 con 0x7f60b815a700 2024-03-06T07:33:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.812+0000 7f60b5600640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f609806b500 0x7f609806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.812+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f60b008a380 con 0x7f60b815a700 2024-03-06T07:33:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.812+0000 7f60bdc00640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f609806b500 0x7f609806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.814+0000 7f60bdc00640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f609806b500 0x7f609806d9c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f60b8158c80 tx=0x7f60a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:38.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:38.816+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60b0059440 con 0x7f60b815a700 2024-03-06T07:33:39.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.015+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f60b815ee70 con 0x7f60b815a700 2024-03-06T07:33:39.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.022+0000 7f60b5600640 1 -- 172.21.15.6:0/872842130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f60b0058d40 con 0x7f60b815a700 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f609806b500 msgr2=0x7f609806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f609806b500 0x7f609806d9c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f60b8158c80 tx=0x7f60a8008040 comp rx=0 tx=0).stop 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 msgr2=0x7f60b815abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 0x7f60b815abb0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f60b00013d0 tx=0x7f60b000e530 comp rx=0 tx=0).stop 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 shutdown_connections 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f609806b500 0x7f609806d9c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f60b815a700 0x7f60b815abb0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f60b8160de0 0x7f60b8157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 --2- 172.21.15.6:0/872842130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f609c003f80 0x7f60b80c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 >> 172.21.15.6:0/872842130 conn(0x7f60b80d3cd0 msgr2=0x7f60b8161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 shutdown_connections 2024-03-06T07:33:39.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.025+0000 7f60bf000640 1 -- 172.21.15.6:0/872842130 wait complete. 2024-03-06T07:33:39.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:33:39.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.265+0000 7fb835400640 1 Processor -- start 2024-03-06T07:33:39.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.265+0000 7fb835400640 1 -- start start 2024-03-06T07:33:39.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.266+0000 7fb835400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb830145300 0x7fb8300b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.266+0000 7fb835400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300714d0 con 0x7fb8300cd060 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.266+0000 7fb835400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300717b0 con 0x7fb830070f10 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.266+0000 7fb835400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb830071a90 con 0x7fb830145300 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb82fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb830145300 0x7fb8300b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb834a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb8300cd060 0x7fb83015c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58652/0 (socket says 172.21.15.6:58652) 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb834a00640 1 -- 172.21.15.6:0/1513604260 learned_addr learned my addr 172.21.15.6:0/1513604260 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb82fe00640 1 -- 172.21.15.6:0/1513604260 >> v1:172.21.15.178:6789/0 conn(0x7fb830070f10 legacy=0x7fb8300cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:39.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb82fe00640 1 -- 172.21.15.6:0/1513604260 >> v1:172.21.15.6:6789/0 conn(0x7fb8300cd060 legacy=0x7fb83015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb82fe00640 1 -- 172.21.15.6:0/1513604260 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830071d70 con 0x7fb830145300 2024-03-06T07:33:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.267+0000 7fb82ea00640 1 -- 172.21.15.6:0/1513604260 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570009067 0 0) 0x7fb8300714d0 con 0x7fb8300cd060 2024-03-06T07:33:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.268+0000 7fb82fe00640 1 --2- 172.21.15.6:0/1513604260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830145300 0x7fb8300b49a0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb8200097a0 tx=0x7fb82002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1e1b131e91ed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.268+0000 7fb82ea00640 1 -- 172.21.15.6:0/1513604260 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8200315a0 con 0x7fb830145300 2024-03-06T07:33:39.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb82ea00640 1 -- 172.21.15.6:0/1513604260 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb820037070 con 0x7fb830145300 2024-03-06T07:33:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb82ea00640 1 -- 172.21.15.6:0/1513604260 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820031960 con 0x7fb830145300 2024-03-06T07:33:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 -- 172.21.15.6:0/1513604260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830145300 msgr2=0x7fb8300b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 --2- 172.21.15.6:0/1513604260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830145300 0x7fb8300b49a0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb8200097a0 tx=0x7fb82002b840 comp rx=0 tx=0).stop 2024-03-06T07:33:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 -- 172.21.15.6:0/1513604260 shutdown_connections 2024-03-06T07:33:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 --2- 172.21.15.6:0/1513604260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830145300 0x7fb8300b49a0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 -- 172.21.15.6:0/1513604260 >> 172.21.15.6:0/1513604260 conn(0x7fb83006b3d0 msgr2=0x7fb83006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 -- 172.21.15.6:0/1513604260 shutdown_connections 2024-03-06T07:33:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.269+0000 7fb835400640 1 -- 172.21.15.6:0/1513604260 wait complete. 2024-03-06T07:33:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.271+0000 7fb835400640 1 Processor -- start 2024-03-06T07:33:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.271+0000 7fb835400640 1 -- start start 2024-03-06T07:33:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb835400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 0x7fb830142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb835400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8300cd060 0x7fb83015b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb835400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb830145300 0x7fb830142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb835400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83015cc80 con 0x7fb830070f10 2024-03-06T07:33:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb835400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8301456e0 con 0x7fb8300cd060 2024-03-06T07:33:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb835400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83015ce00 con 0x7fb830145300 2024-03-06T07:33:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb82fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 0x7fb830142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb82fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 0x7fb830142340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57220/0 (socket says 172.21.15.6:57220) 2024-03-06T07:33:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.272+0000 7fb82fe00640 1 -- 172.21.15.6:0/4240421931 learned_addr learned my addr 172.21.15.6:0/4240421931 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82f400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8300cd060 0x7fb83015b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82fe00640 1 -- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb830145300 msgr2=0x7fb830142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82fe00640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb830145300 0x7fb830142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82fe00640 1 -- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8300cd060 msgr2=0x7fb83015b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82fe00640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8300cd060 0x7fb83015b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82fe00640 1 -- 172.21.15.6:0/4240421931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb830142880 con 0x7fb830070f10 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb82fe00640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 0x7fb830142340 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb820034040 tx=0x7fb8200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.273+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820004030 con 0x7fb830070f10 2024-03-06T07:33:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.274+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8200041f0 con 0x7fb830070f10 2024-03-06T07:33:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.274+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83015bc00 con 0x7fb830070f10 2024-03-06T07:33:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.274+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb820032640 con 0x7fb830070f10 2024-03-06T07:33:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.274+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83006e550 con 0x7fb830070f10 2024-03-06T07:33:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.275+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83004f330 con 0x7fb830070f10 2024-03-06T07:33:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.276+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb8200327e0 con 0x7fb830070f10 2024-03-06T07:33:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.276+0000 7fb827e00640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81006b4d0 0x7fb81006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.277+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fb82002f030 con 0x7fb830070f10 2024-03-06T07:33:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.277+0000 7fb82f400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81006b4d0 0x7fb81006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.278+0000 7fb82f400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81006b4d0 0x7fb81006d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb8301f7140 tx=0x7fb81c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.281+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb820041070 con 0x7fb830070f10 2024-03-06T07:33:39.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.555+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fb83006e790 con 0x7fb830070f10 2024-03-06T07:33:39.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.879+0000 7fb827e00640 1 -- 172.21.15.6:0/4240421931 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v347) v1 ==== 134+0+0 (secure 0 0 0) 0x7fb82007b4d0 con 0x7fb830070f10 2024-03-06T07:33:39.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81006b4d0 msgr2=0x7fb81006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81006b4d0 0x7fb81006d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb8301f7140 tx=0x7fb81c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:39.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 msgr2=0x7fb830142340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:39.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 0x7fb830142340 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb820034040 tx=0x7fb8200046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 shutdown_connections 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb81006b4d0 0x7fb81006d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8300cd060 0x7fb83015b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb830145300 0x7fb830142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 --2- 172.21.15.6:0/4240421931 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb830070f10 0x7fb830142340 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:39.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.884+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 >> 172.21.15.6:0/4240421931 conn(0x7fb83006b3d0 msgr2=0x7fb8300cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.885+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 shutdown_connections 2024-03-06T07:33:39.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:39.885+0000 7fb835400640 1 -- 172.21.15.6:0/4240421931 wait complete. 2024-03-06T07:33:39.912 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:33:39.912 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:33:39.912 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:33:39.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:40.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.114+0000 7f2644000640 1 Processor -- start 2024-03-06T07:33:40.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.114+0000 7f2644000640 1 -- start start 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.115+0000 7f2644000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.115+0000 7f2644000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f263c0b5060 0x7f263c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.115+0000 7f2644000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0cea50 con 0x7f263c0b54a0 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.115+0000 7f2644000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c0cf900 con 0x7f263c0b5060 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.115+0000 7f2644000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0cecf0 con 0x7f263c161b40 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f263c0b5060 0x7f263c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f263c0b5060 0x7f263c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57236/0 (socket says 172.21.15.6:57236) 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 -- 172.21.15.6:0/3886082427 learned_addr learned my addr 172.21.15.6:0/3886082427 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2642000640 1 --1- 172.21.15.6:0/3886082427 >> v1:172.21.15.6:6789/0 conn(0x7f263c0b54a0 0x7f263c0b3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58662/0 (socket says 172.21.15.6:58662) 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2641600640 1 --2- 172.21.15.6:0/3886082427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 -- 172.21.15.6:0/3886082427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 msgr2=0x7f263c0cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 --2- 172.21.15.6:0/3886082427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c0cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 -- 172.21.15.6:0/3886082427 >> v1:172.21.15.6:6789/0 conn(0x7f263c0b54a0 legacy=0x7f263c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.116+0000 7f2640c00640 1 -- 172.21.15.6:0/3886082427 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0cefd0 con 0x7f263c0b5060 2024-03-06T07:33:40.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.117+0000 7f2640c00640 1 --2- 172.21.15.6:0/3886082427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b5060 0x7f263c0d2220 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2628009870 tx=0x7f262802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d2325a2a89f55e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.118+0000 7f263be00640 1 -- 172.21.15.6:0/3886082427 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628034070 con 0x7f263c0b5060 2024-03-06T07:33:40.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.118+0000 7f263be00640 1 -- 172.21.15.6:0/3886082427 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2628030840 con 0x7f263c0b5060 2024-03-06T07:33:40.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.118+0000 7f263be00640 1 -- 172.21.15.6:0/3886082427 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628030c10 con 0x7f263c0b5060 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.118+0000 7f2644000640 1 -- 172.21.15.6:0/3886082427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b5060 msgr2=0x7f263c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.118+0000 7f2644000640 1 --2- 172.21.15.6:0/3886082427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b5060 0x7f263c0d2220 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f2628009870 tx=0x7f262802b980 comp rx=0 tx=0).stop 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.119+0000 7f2644000640 1 -- 172.21.15.6:0/3886082427 shutdown_connections 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.119+0000 7f2644000640 1 --2- 172.21.15.6:0/3886082427 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b5060 0x7f263c0d2220 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.119+0000 7f2644000640 1 --2- 172.21.15.6:0/3886082427 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c0cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.119+0000 7f2644000640 1 -- 172.21.15.6:0/3886082427 >> 172.21.15.6:0/3886082427 conn(0x7f263c09d220 msgr2=0x7f263c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.119+0000 7f2644000640 1 -- 172.21.15.6:0/3886082427 shutdown_connections 2024-03-06T07:33:40.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.119+0000 7f2644000640 1 -- 172.21.15.6:0/3886082427 wait complete. 2024-03-06T07:33:40.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.120+0000 7f2644000640 1 Processor -- start 2024-03-06T07:33:40.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.121+0000 7f2644000640 1 -- start start 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.121+0000 7f2644000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f263c0b5060 0x7f263c0ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2644000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 0x7f263c0b7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2644000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c15f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2644000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0b2ff0 con 0x7f263c0b54a0 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2644000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0cf590 con 0x7f263c0b5060 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2644000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0b3170 con 0x7f263c161b40 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2640c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 0x7f263c0b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2640c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 0x7f263c0b7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57244/0 (socket says 172.21.15.6:57244) 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2640c00640 1 -- 172.21.15.6:0/515624869 learned_addr learned my addr 172.21.15.6:0/515624869 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:40.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2642000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c15f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2641600640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f263c0b5060 0x7f263c0ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2640c00640 1 -- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 msgr2=0x7f263c15f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2640c00640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c15f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.122+0000 7f2640c00640 1 -- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f263c0b5060 msgr2=0x7f263c0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2640c00640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f263c0b5060 0x7f263c0ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2640c00640 1 -- 172.21.15.6:0/515624869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c14ef60 con 0x7f263c0b54a0 2024-03-06T07:33:40.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2640c00640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 0x7f263c0b7720 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2628000c00 tx=0x7f2628004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628048070 con 0x7f263c0b54a0 2024-03-06T07:33:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2628030640 con 0x7f263c0b54a0 2024-03-06T07:33:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c0b6c40 con 0x7f263c0b54a0 2024-03-06T07:33:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.123+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628034040 con 0x7f263c0b54a0 2024-03-06T07:33:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.124+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c15fc30 con 0x7f263c0b54a0 2024-03-06T07:33:40.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.125+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c04f3a0 con 0x7f263c0b54a0 2024-03-06T07:33:40.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.126+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f262804b070 con 0x7f263c0b54a0 2024-03-06T07:33:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.127+0000 7f2639600640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f261c06b4d0 0x7f261c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.127+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f26280ac080 con 0x7f263c0b54a0 2024-03-06T07:33:40.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.128+0000 7f2641600640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f261c06b4d0 0x7f261c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.129+0000 7f2641600640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f261c06b4d0 0x7f261c06d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f262c006fd0 tx=0x7f262c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.131+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f262807b260 con 0x7f263c0b54a0 2024-03-06T07:33:40.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.331+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f263c0d1df0 con 0x7f263c0b54a0 2024-03-06T07:33:40.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.333+0000 7f2639600640 1 -- 172.21.15.6:0/515624869 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13204 (secure 0 0 0) 0x7f262807ab60 con 0x7f263c0b54a0 2024-03-06T07:33:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:40.335 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:37.861763+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/619677659":"2024-03-07T07:33:34.556552+0000","172.21.15.178:6836/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6835/2859903799":"2024-03-07T07:32:59.635590+0000","172.21.15.6:6834/619677659":"2024-03-07T07:33:34.556552+0000","172.21.15.178:6834/833246901":"2024-03-07T07:33:00.629222+0000","172.21.15.6:6834/2859903799":"2024-03-07T07:32:59.635590+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.335+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f261c06b4d0 msgr2=0x7f261c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.335+0000 7f2644000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f261c06b4d0 0x7f261c06d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f262c006fd0 tx=0x7f262c008040 comp rx=0 tx=0).stop 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.335+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 msgr2=0x7f263c0b7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.335+0000 7f2644000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 0x7f263c0b7720 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f2628000c00 tx=0x7f2628004730 comp rx=0 tx=0).stop 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 shutdown_connections 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f261c06b4d0 0x7f261c06d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f263c161b40 0x7f263c15f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f263c0b54a0 0x7f263c0b7720 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 --2- 172.21.15.6:0/515624869 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f263c0b5060 0x7f263c0ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 >> 172.21.15.6:0/515624869 conn(0x7f263c09d220 msgr2=0x7f263c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 shutdown_connections 2024-03-06T07:33:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.336+0000 7f2644000640 1 -- 172.21.15.6:0/515624869 wait complete. 2024-03-06T07:33:40.358 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:40.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.587+0000 7f9ddfa00640 1 Processor -- start 2024-03-06T07:33:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.587+0000 7f9ddfa00640 1 -- start start 2024-03-06T07:33:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.589+0000 7f9ddfa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd80acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.589+0000 7f9ddfa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9dd8071e20 0x7f9dd80ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.589+0000 7f9ddfa00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd8072df0 con 0x7f9dd8072520 2024-03-06T07:33:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.589+0000 7f9ddfa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd80cdab0 con 0x7f9dd8071e20 2024-03-06T07:33:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.589+0000 7f9ddfa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80b4d70 con 0x7f9dd815bf20 2024-03-06T07:33:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.590+0000 7f9ddda00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9dd8071e20 0x7f9dd80ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.590+0000 7f9ddda00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9dd8071e20 0x7f9dd80ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:58380/0 (socket says 172.21.15.6:58380) 2024-03-06T07:33:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.590+0000 7f9ddda00640 1 -- 172.21.15.6:0/2274584570 learned_addr learned my addr 172.21.15.6:0/2274584570 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.590+0000 7f9ddd000640 1 --2- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd80acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.590+0000 7f9ddda00640 1 -- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 msgr2=0x7f9dd80acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.590+0000 7f9ddda00640 1 --2- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd80acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.591+0000 7f9ddda00640 1 -- 172.21.15.6:0/2274584570 >> v1:172.21.15.178:6789/0 conn(0x7f9dd8072520 legacy=0x7f9dd80cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.591+0000 7f9ddda00640 1 -- 172.21.15.6:0/2274584570 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd80b5050 con 0x7f9dd8071e20 2024-03-06T07:33:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.592+0000 7f9ddda00640 1 --2- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8071e20 0x7f9dd80ccad0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009e50 tx=0x7f9dd002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f971bdd8cf0abdbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.592+0000 7f9dd7400640 1 -- 172.21.15.6:0/2274584570 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd002cb00 con 0x7f9dd8071e20 2024-03-06T07:33:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9dd7400640 1 -- 172.21.15.6:0/2274584570 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd002cca0 con 0x7f9dd8071e20 2024-03-06T07:33:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9dd7400640 1 -- 172.21.15.6:0/2274584570 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0034610 con 0x7f9dd8071e20 2024-03-06T07:33:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 -- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8071e20 msgr2=0x7f9dd80ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8071e20 0x7f9dd80ccad0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009e50 tx=0x7f9dd002bf60 comp rx=0 tx=0).stop 2024-03-06T07:33:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 -- 172.21.15.6:0/2274584570 shutdown_connections 2024-03-06T07:33:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8071e20 0x7f9dd80ccad0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/2274584570 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd80acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 -- 172.21.15.6:0/2274584570 >> 172.21.15.6:0/2274584570 conn(0x7f9dd806b3d0 msgr2=0x7f9dd806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.593+0000 7f9ddfa00640 1 -- 172.21.15.6:0/2274584570 shutdown_connections 2024-03-06T07:33:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.594+0000 7f9ddfa00640 1 -- 172.21.15.6:0/2274584570 wait complete. 2024-03-06T07:33:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.595+0000 7f9ddfa00640 1 Processor -- start 2024-03-06T07:33:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.596+0000 7f9ddfa00640 1 -- start start 2024-03-06T07:33:40.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.596+0000 7f9ddfa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 0x7f9dd8149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddfa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8072520 0x7f9dd8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddfa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd81442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddfa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80b4b10 con 0x7f9dd8071e20 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddfa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80b4990 con 0x7f9dd8072520 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddfa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd80ce6c0 con 0x7f9dd815bf20 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 0x7f9dd8149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 0x7f9dd8149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57246/0 (socket says 172.21.15.6:57246) 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddd000640 1 -- 172.21.15.6:0/636141786 learned_addr learned my addr 172.21.15.6:0/636141786 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9dd7e00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8072520 0x7f9dd8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.597+0000 7f9ddda00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd81442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddd000640 1 -- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 msgr2=0x7f9dd81442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddd000640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd81442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddd000640 1 -- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8072520 msgr2=0x7f9dd8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddd000640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8072520 0x7f9dd8157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddd000640 1 -- 172.21.15.6:0/636141786 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd8149e80 con 0x7f9dd8071e20 2024-03-06T07:33:40.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9dd7e00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8072520 0x7f9dd8157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddd000640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 0x7f9dd8149940 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e870 tx=0x7f9dcc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc017070 con 0x7f9dd8071e20 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dcc00c940 con 0x7f9dd8071e20 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc00cd30 con 0x7f9dd8071e20 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd81587d0 con 0x7f9dd8071e20 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.598+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd8158d90 con 0x7f9dd8071e20 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.599+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd804f330 con 0x7f9dd8071e20 2024-03-06T07:33:40.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.601+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dcc01a070 con 0x7f9dd8071e20 2024-03-06T07:33:40.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.601+0000 7f9dd4c00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db806b4d0 0x7f9db806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:40.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.602+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9dcc089ec0 con 0x7f9dd8071e20 2024-03-06T07:33:40.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.602+0000 7f9dd7e00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db806b4d0 0x7f9db806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:40.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.603+0000 7f9dd7e00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db806b4d0 0x7f9db806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9dd815b4e0 tx=0x7f9dc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:40.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.606+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9dcc058ef0 con 0x7f9dd8071e20 2024-03-06T07:33:40.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.804+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9dd815f720 con 0x7f9dd8071e20 2024-03-06T07:33:40.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.805+0000 7f9dd4c00640 1 -- 172.21.15.6:0/636141786 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9dcc0587f0 con 0x7f9dd8071e20 2024-03-06T07:33:40.808 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:33:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db806b4d0 msgr2=0x7f9db806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db806b4d0 0x7f9db806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9dd815b4e0 tx=0x7f9dc4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 msgr2=0x7f9dd8149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 0x7f9dd8149940 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e870 tx=0x7f9dcc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 shutdown_connections 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9db806b4d0 0x7f9db806d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9dd8072520 0x7f9dd8157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9dd815bf20 0x7f9dd81442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 --2- 172.21.15.6:0/636141786 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9dd8071e20 0x7f9dd8149940 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 >> 172.21.15.6:0/636141786 conn(0x7f9dd806b3d0 msgr2=0x7f9dd815a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.808+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 shutdown_connections 2024-03-06T07:33:40.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:40.809+0000 7f9ddfa00640 1 -- 172.21.15.6:0/636141786 wait complete. 2024-03-06T07:33:40.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:33:41.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.047+0000 7f517d800640 1 Processor -- start 2024-03-06T07:33:41.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.047+0000 7f517d800640 1 -- start start 2024-03-06T07:33:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f517d800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5178070f10 0x7f51780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f517d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51780cce90 0x7f517815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f517d800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780714d0 con 0x7f5178145300 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f517d800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51780cd300 con 0x7f5178070f10 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f517d800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51780717b0 con 0x7f51780cce90 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f5177400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5178070f10 0x7f51780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f5177400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5178070f10 0x7f51780cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57260/0 (socket says 172.21.15.6:57260) 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f5177400640 1 -- 172.21.15.6:0/3115269895 learned_addr learned my addr 172.21.15.6:0/3115269895 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:41.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.049+0000 7f5177e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5178145300 0x7f51780b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58664/0 (socket says 172.21.15.6:58664) 2024-03-06T07:33:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.050+0000 7f5177400640 1 -- 172.21.15.6:0/3115269895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51780cce90 msgr2=0x7f517815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.050+0000 7f517ce00640 1 --2- 172.21.15.6:0/3115269895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51780cce90 0x7f517815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.050+0000 7f5177400640 1 --2- 172.21.15.6:0/3115269895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51780cce90 0x7f517815c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.050+0000 7f5177400640 1 -- 172.21.15.6:0/3115269895 >> v1:172.21.15.6:6789/0 conn(0x7f5178145300 legacy=0x7f51780b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.050+0000 7f5177400640 1 -- 172.21.15.6:0/3115269895 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5178071a90 con 0x7f5178070f10 2024-03-06T07:33:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.051+0000 7f5177400640 1 --2- 172.21.15.6:0/3115269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5178070f10 0x7f51780cc950 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f5164009870 tx=0x7f516402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7013159729923dda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.051+0000 7f5176a00640 1 -- 172.21.15.6:0/3115269895 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164035020 con 0x7f5178070f10 2024-03-06T07:33:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.051+0000 7f5176a00640 1 -- 172.21.15.6:0/3115269895 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5164031640 con 0x7f5178070f10 2024-03-06T07:33:41.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.051+0000 7f5176a00640 1 -- 172.21.15.6:0/3115269895 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516402d040 con 0x7f5178070f10 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 -- 172.21.15.6:0/3115269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5178070f10 msgr2=0x7f51780cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 --2- 172.21.15.6:0/3115269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5178070f10 0x7f51780cc950 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f5164009870 tx=0x7f516402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 -- 172.21.15.6:0/3115269895 shutdown_connections 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 --2- 172.21.15.6:0/3115269895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51780cce90 0x7f517815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 --2- 172.21.15.6:0/3115269895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5178070f10 0x7f51780cc950 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 -- 172.21.15.6:0/3115269895 >> 172.21.15.6:0/3115269895 conn(0x7f517806b3d0 msgr2=0x7f517806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 -- 172.21.15.6:0/3115269895 shutdown_connections 2024-03-06T07:33:41.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.052+0000 7f517d800640 1 -- 172.21.15.6:0/3115269895 wait complete. 2024-03-06T07:33:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.054+0000 7f517d800640 1 Processor -- start 2024-03-06T07:33:41.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.054+0000 7f517d800640 1 -- start start 2024-03-06T07:33:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.055+0000 7f517d800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5178070f10 0x7f51780b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.055+0000 7f517d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 0x7f5178131150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.055+0000 7f517d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5178145300 0x7f51781e5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.055+0000 7f517d800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517815cab0 con 0x7f51780cce90 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.055+0000 7f517d800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51781456e0 con 0x7f5178070f10 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.055+0000 7f517d800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517815cc30 con 0x7f5178145300 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 0x7f5178131150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 0x7f5178131150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57272/0 (socket says 172.21.15.6:57272) 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 -- 172.21.15.6:0/3106818446 learned_addr learned my addr 172.21.15.6:0/3106818446 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177e00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5178070f10 0x7f51780b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f517ce00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5178145300 0x7f51781e5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 -- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5178145300 msgr2=0x7f51781e5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5178145300 0x7f51781e5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 -- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5178070f10 msgr2=0x7f51780b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5178070f10 0x7f51780b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177400640 1 -- 172.21.15.6:0/3106818446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5178131c10 con 0x7f51780cce90 2024-03-06T07:33:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f517ce00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5178145300 0x7f51781e5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.056+0000 7f5177e00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5178070f10 0x7f51780b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:41.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.057+0000 7f5177400640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 0x7f5178131150 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5164000c00 tx=0x7f51640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.057+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164040070 con 0x7f51780cce90 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.057+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5164039bf0 con 0x7f51780cce90 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.057+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5178131ef0 con 0x7f51780cce90 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.057+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164035040 con 0x7f51780cce90 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.057+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51781324b0 con 0x7f51780cce90 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.058+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517804f330 con 0x7f51780cce90 2024-03-06T07:33:41.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.059+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f516404b060 con 0x7f51780cce90 2024-03-06T07:33:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.060+0000 7f516fe00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f515806b4d0 0x7f515806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:41.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.060+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f51640aba90 con 0x7f51780cce90 2024-03-06T07:33:41.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.061+0000 7f5177e00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f515806b4d0 0x7f515806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:41.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.062+0000 7f5177e00640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f515806b4d0 0x7f515806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5178131020 tx=0x7f5168009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:41.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.065+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5164031610 con 0x7f51780cce90 2024-03-06T07:33:41.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.343+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f5178074ab0 con 0x7f51780cce90 2024-03-06T07:33:41.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.891+0000 7f516fe00640 1 -- 172.21.15.6:0/3106818446 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v349) v1 ==== 145+0+0 (secure 0 0 0) 0x7f516407ab60 con 0x7f51780cce90 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f515806b4d0 msgr2=0x7f515806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f515806b4d0 0x7f515806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5178131020 tx=0x7f5168009040 comp rx=0 tx=0).stop 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 msgr2=0x7f5178131150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 0x7f5178131150 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f5164000c00 tx=0x7f51640046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 shutdown_connections 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f515806b4d0 0x7f515806d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5178145300 0x7f51781e5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.895+0000 7f517d800640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51780cce90 0x7f5178131150 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.896+0000 7f517d800640 1 --2- 172.21.15.6:0/3106818446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5178070f10 0x7f51780b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:41.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.896+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 >> 172.21.15.6:0/3106818446 conn(0x7f517806b3d0 msgr2=0x7f5178072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:41.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.896+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 shutdown_connections 2024-03-06T07:33:41.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:41.896+0000 7f517d800640 1 -- 172.21.15.6:0/3106818446 wait complete. 2024-03-06T07:33:41.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:33:42.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.146+0000 7effc4000640 1 Processor -- start 2024-03-06T07:33:42.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.147+0000 7effc4000640 1 -- start start 2024-03-06T07:33:42.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.148+0000 7effc4000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7effbc15afa0 0x7effbc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.148+0000 7effc4000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7effbc070ce0 0x7effbc0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.148+0000 7effc4000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc071c60 con 0x7effbc0cd010 2024-03-06T07:33:42.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.148+0000 7effc4000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effbc0cd480 con 0x7effbc15afa0 2024-03-06T07:33:42.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.148+0000 7effc4000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effbc0cfa10 con 0x7effbc070ce0 2024-03-06T07:33:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc1600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7effbc15afa0 0x7effbc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc2000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7effbc0cd010 0x7effbc06c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37824/0 (socket says 172.21.15.6:37824) 2024-03-06T07:33:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc2000640 1 -- 172.21.15.6:0/3599721511 learned_addr learned my addr 172.21.15.6:0/3599721511 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc0c00640 1 --2- 172.21.15.6:0/3599721511 >> v2:172.21.15.178:3300/0 conn(0x7effbc070ce0 0x7effbc0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc1600640 1 -- 172.21.15.6:0/3599721511 >> v2:172.21.15.178:3300/0 conn(0x7effbc070ce0 msgr2=0x7effbc0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:42.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc1600640 1 --2- 172.21.15.6:0/3599721511 >> v2:172.21.15.178:3300/0 conn(0x7effbc070ce0 0x7effbc0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc1600640 1 -- 172.21.15.6:0/3599721511 >> v1:172.21.15.6:6789/0 conn(0x7effbc0cd010 legacy=0x7effbc06c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc1600640 1 -- 172.21.15.6:0/3599721511 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc071f40 con 0x7effbc15afa0 2024-03-06T07:33:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.149+0000 7effc0c00640 1 --2- 172.21.15.6:0/3599721511 >> v2:172.21.15.178:3300/0 conn(0x7effbc070ce0 0x7effbc0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.151+0000 7effc1600640 1 --2- 172.21.15.6:0/3599721511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc15afa0 0x7effbc0cf3a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7effbc0b3650 tx=0x7effb002da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4c9245dafcce0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.151+0000 7effbbe00640 1 -- 172.21.15.6:0/3599721511 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0031870 con 0x7effbc15afa0 2024-03-06T07:33:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.151+0000 7effbbe00640 1 -- 172.21.15.6:0/3599721511 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0031a10 con 0x7effbc15afa0 2024-03-06T07:33:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.151+0000 7effbbe00640 1 -- 172.21.15.6:0/3599721511 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0031de0 con 0x7effbc15afa0 2024-03-06T07:33:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.151+0000 7effc4000640 1 -- 172.21.15.6:0/3599721511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc15afa0 msgr2=0x7effbc0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.151+0000 7effc4000640 1 --2- 172.21.15.6:0/3599721511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc15afa0 0x7effbc0cf3a0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7effbc0b3650 tx=0x7effb002da50 comp rx=0 tx=0).stop 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.152+0000 7effc4000640 1 -- 172.21.15.6:0/3599721511 shutdown_connections 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.152+0000 7effc4000640 1 --2- 172.21.15.6:0/3599721511 >> v2:172.21.15.178:3300/0 conn(0x7effbc070ce0 0x7effbc0ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.152+0000 7effc4000640 1 --2- 172.21.15.6:0/3599721511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc15afa0 0x7effbc0cf3a0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.152+0000 7effc4000640 1 -- 172.21.15.6:0/3599721511 >> 172.21.15.6:0/3599721511 conn(0x7effbc06b3d0 msgr2=0x7effbc06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.152+0000 7effc4000640 1 -- 172.21.15.6:0/3599721511 shutdown_connections 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.152+0000 7effc4000640 1 -- 172.21.15.6:0/3599721511 wait complete. 2024-03-06T07:33:42.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.153+0000 7effc4000640 1 Processor -- start 2024-03-06T07:33:42.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.154+0000 7effc4000640 1 -- start start 2024-03-06T07:33:42.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc4000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7effbc070ce0 0x7effbc1549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc4000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 0x7effbc0c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc4000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effbc15afa0 0x7effbc09c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc4000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc071170 con 0x7effbc0cd010 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc4000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc072500 con 0x7effbc15afa0 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc4000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b4ad0 con 0x7effbc070ce0 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 0x7effbc0c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 0x7effbc0c8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41406/0 (socket says 172.21.15.6:41406) 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc0c00640 1 -- 172.21.15.6:0/1284012502 learned_addr learned my addr 172.21.15.6:0/1284012502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc1600640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7effbc070ce0 0x7effbc1549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.155+0000 7effc2000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effbc15afa0 0x7effbc09c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc0c00640 1 -- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7effbc070ce0 msgr2=0x7effbc1549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc0c00640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7effbc070ce0 0x7effbc1549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc0c00640 1 -- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effbc15afa0 msgr2=0x7effbc09c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc0c00640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effbc15afa0 0x7effbc09c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc0c00640 1 -- 172.21.15.6:0/1284012502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc0cb3e0 con 0x7effbc0cd010 2024-03-06T07:33:42.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc0c00640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 0x7effbc0c8840 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7effa800c820 tx=0x7effa800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8015070 con 0x7effbc0cd010 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa8007a30 con 0x7effbc0cd010 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8007e00 con 0x7effbc0cd010 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effbc0cb6c0 con 0x7effbc0cd010 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.156+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc0cbc80 con 0x7effbc0cd010 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.157+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc04f330 con 0x7effbc0cd010 2024-03-06T07:33:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.158+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effa8018070 con 0x7effbc0cd010 2024-03-06T07:33:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.159+0000 7effb9600640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7eff9c06b4d0 0x7eff9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:42.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.159+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7effa8089f50 con 0x7effbc0cd010 2024-03-06T07:33:42.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.160+0000 7effc1600640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7eff9c06b4d0 0x7eff9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.161+0000 7effc1600640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7eff9c06b4d0 0x7eff9c06d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7effb0038000 tx=0x7effb003b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:42.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.164+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effa8059010 con 0x7effbc0cd010 2024-03-06T07:33:42.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.363+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7effbc0cef40 con 0x7effbc0cd010 2024-03-06T07:33:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.368+0000 7effb9600640 1 -- 172.21.15.6:0/1284012502 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7effa8058910 con 0x7effbc0cd010 2024-03-06T07:33:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7eff9c06b4d0 msgr2=0x7eff9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7eff9c06b4d0 0x7eff9c06d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7effb0038000 tx=0x7effb003b040 comp rx=0 tx=0).stop 2024-03-06T07:33:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 msgr2=0x7effbc0c8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 0x7effbc0c8840 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7effa800c820 tx=0x7effa800cc20 comp rx=0 tx=0).stop 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 shutdown_connections 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7eff9c06b4d0 0x7eff9c06d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effbc15afa0 0x7effbc09c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7effbc0cd010 0x7effbc0c8840 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 --2- 172.21.15.6:0/1284012502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7effbc070ce0 0x7effbc1549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 >> 172.21.15.6:0/1284012502 conn(0x7effbc06b3d0 msgr2=0x7effbc06c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 shutdown_connections 2024-03-06T07:33:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:42.371+0000 7effc4000640 1 -- 172.21.15.6:0/1284012502 wait complete. 2024-03-06T07:33:42.393 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:33:42.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:42.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:33:42.631 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:33:42.631 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:42.631 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:42.689 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:42.689 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:33:42.690 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:42.746 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:42.746 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:42.746 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:42.806 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:33:42.807 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:33:42.807 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:42.863 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:33:42.863 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:33:42.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.082+0000 7f9f26a00640 1 Processor -- start 2024-03-06T07:33:43.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.082+0000 7f9f26a00640 1 -- start start 2024-03-06T07:33:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.083+0000 7f9f26a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9f20145300 0x7f9f200b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.083+0000 7f9f26a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.083+0000 7f9f26a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200714d0 con 0x7f9f200cce90 2024-03-06T07:33:43.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.083+0000 7f9f26a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f200cd300 con 0x7f9f20145300 2024-03-06T07:33:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.083+0000 7f9f26a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200717b0 con 0x7f9f20070f10 2024-03-06T07:33:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.084+0000 7f9f25600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9f20145300 0x7f9f200b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.084+0000 7f9f24c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.084+0000 7f9f24c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:48890/0 (socket says 172.21.15.6:48890) 2024-03-06T07:33:43.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.084+0000 7f9f24c00640 1 -- 172.21.15.6:0/2916847982 learned_addr learned my addr 172.21.15.6:0/2916847982 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.085+0000 7f9f24c00640 1 -- 172.21.15.6:0/2916847982 >> v2:172.21.15.178:3300/0 conn(0x7f9f20145300 msgr2=0x7f9f200b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.085+0000 7f9f24c00640 1 --2- 172.21.15.6:0/2916847982 >> v2:172.21.15.178:3300/0 conn(0x7f9f20145300 0x7f9f200b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.085+0000 7f9f24c00640 1 -- 172.21.15.6:0/2916847982 >> v1:172.21.15.178:6789/0 conn(0x7f9f200cce90 legacy=0x7f9f200cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.085+0000 7f9f24c00640 1 -- 172.21.15.6:0/2916847982 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20071a90 con 0x7f9f20070f10 2024-03-06T07:33:43.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.086+0000 7f9f24c00640 1 --2- 172.21.15.6:0/2916847982 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200cc950 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009870 tx=0x7f9f0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf3dc9b20615c885 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f1fe00640 1 -- 172.21.15.6:0/2916847982 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c034070 con 0x7f9f20070f10 2024-03-06T07:33:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f1fe00640 1 -- 172.21.15.6:0/2916847982 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c030840 con 0x7f9f20070f10 2024-03-06T07:33:43.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f1fe00640 1 -- 172.21.15.6:0/2916847982 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c030c10 con 0x7f9f20070f10 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 -- 172.21.15.6:0/2916847982 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 msgr2=0x7f9f200cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 --2- 172.21.15.6:0/2916847982 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200cc950 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009870 tx=0x7f9f0c02b980 comp rx=0 tx=0).stop 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 -- 172.21.15.6:0/2916847982 shutdown_connections 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 --2- 172.21.15.6:0/2916847982 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200cc950 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 --2- 172.21.15.6:0/2916847982 >> v2:172.21.15.178:3300/0 conn(0x7f9f20145300 0x7f9f200b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 -- 172.21.15.6:0/2916847982 >> 172.21.15.6:0/2916847982 conn(0x7f9f2006b3d0 msgr2=0x7f9f2006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 -- 172.21.15.6:0/2916847982 shutdown_connections 2024-03-06T07:33:43.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.087+0000 7f9f26a00640 1 -- 172.21.15.6:0/2916847982 wait complete. 2024-03-06T07:33:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.089+0000 7f9f26a00640 1 Processor -- start 2024-03-06T07:33:43.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.089+0000 7f9f26a00640 1 -- start start 2024-03-06T07:33:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.090+0000 7f9f26a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.090+0000 7f9f26a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 0x7f9f2015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.090+0000 7f9f26a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f20145300 0x7f9f200c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.090+0000 7f9f26a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201456e0 con 0x7f9f200cce90 2024-03-06T07:33:43.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.090+0000 7f9f26a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f200b4ee0 con 0x7f9f20145300 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.090+0000 7f9f26a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2015d580 con 0x7f9f20070f10 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 0x7f9f2015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 0x7f9f2015b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41422/0 (socket says 172.21.15.6:41422) 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 -- 172.21.15.6:0/1453823364 learned_addr learned my addr 172.21.15.6:0/1453823364 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f25600640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f26000640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f20145300 0x7f9f200c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 -- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 msgr2=0x7f9f200aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 -- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f20145300 msgr2=0x7f9f200c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f20145300 0x7f9f200c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.091+0000 7f9f24c00640 1 -- 172.21.15.6:0/1453823364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f200aacf0 con 0x7f9f200cce90 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.092+0000 7f9f24c00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 0x7f9f2015b330 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c000c00 tx=0x7f9f0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.092+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c048070 con 0x7f9f200cce90 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.092+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c030640 con 0x7f9f200cce90 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.092+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c034040 con 0x7f9f200cce90 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.092+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f200aafd0 con 0x7f9f200cce90 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.092+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f200ab590 con 0x7f9f200cce90 2024-03-06T07:33:43.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.093+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2004f330 con 0x7f9f200cce90 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.094+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f0c04b070 con 0x7f9f200cce90 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.095+0000 7f9f1d600640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f0006b4d0 0x7f9f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.095+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9f0c0abe30 con 0x7f9f200cce90 2024-03-06T07:33:43.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.096+0000 7f9f25600640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f0006b4d0 0x7f9f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.097+0000 7f9f25600640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f0006b4d0 0x7f9f0006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009a30 tx=0x7f9f14008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.100+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f0c07aef0 con 0x7f9f200cce90 2024-03-06T07:33:43.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.327+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f200cca80 con 0x7f9f200cce90 2024-03-06T07:33:43.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.328+0000 7f9f1d600640 1 -- 172.21.15.6:0/1453823364 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f9f0c07a7f0 con 0x7f9f200cce90 2024-03-06T07:33:43.330 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:43.330 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3411431642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3411431642},{"type":"v1","addr":"172.21.15.178:6835","nonce":3411431642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":6289,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/242591515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":242591515},{"type":"v1","addr":"172.21.15.178:6837","nonce":242591515}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":7347,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/1571909203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":1571909203},{"type":"v1","addr":"172.21.15.6:6833","nonce":1571909203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":65},{"gid":7575,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3412500772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3412500772},{"type":"v1","addr":"172.21.15.6:6835","nonce":3412500772}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[]} 2024-03-06T07:33:43.330 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 71 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f0006b4d0 msgr2=0x7f9f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f0006b4d0 0x7f9f0006d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009a30 tx=0x7f9f14008040 comp rx=0 tx=0).stop 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 msgr2=0x7f9f2015b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 0x7f9f2015b330 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c000c00 tx=0x7f9f0c004730 comp rx=0 tx=0).stop 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 shutdown_connections 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f0006b4d0 0x7f9f0006d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f20145300 0x7f9f200c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f200cce90 0x7f9f2015b330 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 --2- 172.21.15.6:0/1453823364 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f20070f10 0x7f9f200aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.330+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 >> 172.21.15.6:0/1453823364 conn(0x7f9f2006b3d0 msgr2=0x7f9f200cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.331+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 shutdown_connections 2024-03-06T07:33:43.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.331+0000 7f9f26a00640 1 -- 172.21.15.6:0/1453823364 wait complete. 2024-03-06T07:33:43.352 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:33:43.353 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:33:43.353 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:33:43.353 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:43.353 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:33:43.353 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:33:43.353 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:33:43.353 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:43.354 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:33:43.354 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:33:43.354 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:33:43.354 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:43.354 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:33:43.354 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:33:43.354 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:33:43.355 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:33:43.372 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:33:43.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:33:43.375 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:33:43.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:33:43.379 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:33:43.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:33:43.380 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:33:43.380 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:33:43.422 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:33:43.422 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:43.427 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:33:43.427 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:43.428 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:33:43.428 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:43.432 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:33:43.433 DEBUG:teuthology.parallel:result is None 2024-03-06T07:33:43.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:33:43.578 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:33:43.580 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:33:43.582 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:33:43.590 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:33:43.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.617+0000 7f9f68200640 1 Processor -- start 2024-03-06T07:33:43.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.617+0000 7f9f68200640 1 -- start start 2024-03-06T07:33:43.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.618+0000 7f9f68200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9f60070ee0 0x7f9f60162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.618+0000 7f9f68200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f60072da0 0x7f9f600cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.618+0000 7f9f68200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600ce090 con 0x7f9f600712c0 2024-03-06T07:33:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.618+0000 7f9f68200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600ceca0 con 0x7f9f60072da0 2024-03-06T07:33:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.618+0000 7f9f68200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600ce370 con 0x7f9f60070ee0 2024-03-06T07:33:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f60072da0 0x7f9f600cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9f60072da0 0x7f9f600cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41478/0 (socket says 172.21.15.6:41478) 2024-03-06T07:33:43.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 -- 172.21.15.6:0/777729635 learned_addr learned my addr 172.21.15.6:0/777729635 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:43.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f65800640 1 --2- 172.21.15.6:0/777729635 >> v2:172.21.15.178:3300/0 conn(0x7f9f60070ee0 0x7f9f60162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 -- 172.21.15.6:0/777729635 >> v2:172.21.15.178:3300/0 conn(0x7f9f60070ee0 msgr2=0x7f9f60162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:43.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 --2- 172.21.15.6:0/777729635 >> v2:172.21.15.178:3300/0 conn(0x7f9f60070ee0 0x7f9f60162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 -- 172.21.15.6:0/777729635 >> v1:172.21.15.178:6789/0 conn(0x7f9f600712c0 legacy=0x7f9f6006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.619+0000 7f9f64e00640 1 -- 172.21.15.6:0/777729635 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600ce650 con 0x7f9f60072da0 2024-03-06T07:33:43.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.620+0000 7f9f64e00640 1 --2- 172.21.15.6:0/777729635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f60072da0 0x7f9f600cf130 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f9f60073610 tx=0x7f9f4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9c851f535e5a0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f5fe00640 1 -- 172.21.15.6:0/777729635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c034070 con 0x7f9f60072da0 2024-03-06T07:33:43.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f5fe00640 1 -- 172.21.15.6:0/777729635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c030860 con 0x7f9f60072da0 2024-03-06T07:33:43.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f5fe00640 1 -- 172.21.15.6:0/777729635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c030c30 con 0x7f9f60072da0 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 -- 172.21.15.6:0/777729635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f60072da0 msgr2=0x7f9f600cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 --2- 172.21.15.6:0/777729635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f60072da0 0x7f9f600cf130 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f9f60073610 tx=0x7f9f4c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 -- 172.21.15.6:0/777729635 shutdown_connections 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 --2- 172.21.15.6:0/777729635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f60072da0 0x7f9f600cf130 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 --2- 172.21.15.6:0/777729635 >> v2:172.21.15.178:3300/0 conn(0x7f9f60070ee0 0x7f9f60162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 -- 172.21.15.6:0/777729635 >> 172.21.15.6:0/777729635 conn(0x7f9f6006afb0 msgr2=0x7f9f6006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.621+0000 7f9f68200640 1 -- 172.21.15.6:0/777729635 shutdown_connections 2024-03-06T07:33:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.622+0000 7f9f68200640 1 -- 172.21.15.6:0/777729635 wait complete. 2024-03-06T07:33:43.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.623+0000 7f9f68200640 1 Processor -- start 2024-03-06T07:33:43.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.623+0000 7f9f68200640 1 -- start start 2024-03-06T07:33:43.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f68200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f60070ee0 0x7f9f6009c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f68200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 0x7f9f60161f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f68200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f60072da0 0x7f9f6009aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f68200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600cdda0 con 0x7f9f600712c0 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f68200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600cdc20 con 0x7f9f60072da0 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f68200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6006e7d0 con 0x7f9f60070ee0 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.624+0000 7f9f64e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 0x7f9f60161f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 0x7f9f60161f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41482/0 (socket says 172.21.15.6:41482) 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 -- 172.21.15.6:0/1887095610 learned_addr learned my addr 172.21.15.6:0/1887095610 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f66200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f60072da0 0x7f9f6009aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f65800640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f60070ee0 0x7f9f6009c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 -- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f60070ee0 msgr2=0x7f9f6009c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f60070ee0 0x7f9f6009c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 -- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f60072da0 msgr2=0x7f9f6009aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f60072da0 0x7f9f6009aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f64e00640 1 -- 172.21.15.6:0/1887095610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6009b3f0 con 0x7f9f600712c0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.625+0000 7f9f66200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f60072da0 0x7f9f6009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.626+0000 7f9f64e00640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 0x7f9f60161f60 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c000c00 tx=0x7f9f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.626+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c034040 con 0x7f9f600712c0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.626+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c03ad20 con 0x7f9f600712c0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.626+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c044610 con 0x7f9f600712c0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.626+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6009b6d0 con 0x7f9f600712c0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.626+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6009bc90 con 0x7f9f600712c0 2024-03-06T07:33:43.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.627+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f600b14b0 con 0x7f9f600712c0 2024-03-06T07:33:43.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.628+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f4c037070 con 0x7f9f600712c0 2024-03-06T07:33:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.629+0000 7f9f5d600640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4006b4d0 0x7f9f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.629+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9f4c0abfd0 con 0x7f9f600712c0 2024-03-06T07:33:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.629+0000 7f9f65800640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4006b4d0 0x7f9f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:43.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.630+0000 7f9f65800640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4006b4d0 0x7f9f4006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9f6009dcc0 tx=0x7f9f54009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:43.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.633+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f600b14b0 con 0x7f9f600712c0 2024-03-06T07:33:43.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.843+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9f600b52d0 con 0x7f9f600712c0 2024-03-06T07:33:43.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.916+0000 7f9f5d600640 1 -- 172.21.15.6:0/1887095610 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9f4c03f070 con 0x7f9f600712c0 2024-03-06T07:33:43.918 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:33:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.919+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4006b4d0 msgr2=0x7f9f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4006b4d0 0x7f9f4006d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9f6009dcc0 tx=0x7f9f54009190 comp rx=0 tx=0).stop 2024-03-06T07:33:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 msgr2=0x7f9f60161f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 0x7f9f60161f60 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c000c00 tx=0x7f9f4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 shutdown_connections 2024-03-06T07:33:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9f4006b4d0 0x7f9f4006d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f60072da0 0x7f9f6009aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9f600712c0 0x7f9f60161f60 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 --2- 172.21.15.6:0/1887095610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9f60070ee0 0x7f9f6009c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 >> 172.21.15.6:0/1887095610 conn(0x7f9f6006afb0 msgr2=0x7f9f6006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 shutdown_connections 2024-03-06T07:33:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:43.920+0000 7f9f68200640 1 -- 172.21.15.6:0/1887095610 wait complete. 2024-03-06T07:33:43.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:33:44.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.152+0000 7fc61aa00640 1 Processor -- start 2024-03-06T07:33:44.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.152+0000 7fc61aa00640 1 -- start start 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc61aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc614145300 0x7fc6140b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc61aa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc6140cd060 0x7fc61415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc61aa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6140714d0 con 0x7fc614070f10 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc61aa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6140717b0 con 0x7fc614145300 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc61aa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc614071a90 con 0x7fc6140cd060 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc619600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc614145300 0x7fc6140b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc618c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc614070f10 0x7fc6140cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37866/0 (socket says 172.21.15.6:37866) 2024-03-06T07:33:44.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.154+0000 7fc618c00640 1 -- 172.21.15.6:0/35992885 learned_addr learned my addr 172.21.15.6:0/35992885 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:44.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc61a000640 1 --2- 172.21.15.6:0/35992885 >> v2:172.21.15.178:3300/0 conn(0x7fc6140cd060 0x7fc61415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc619600640 1 -- 172.21.15.6:0/35992885 >> v2:172.21.15.178:3300/0 conn(0x7fc6140cd060 msgr2=0x7fc61415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc619600640 1 --2- 172.21.15.6:0/35992885 >> v2:172.21.15.178:3300/0 conn(0x7fc6140cd060 0x7fc61415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc619600640 1 -- 172.21.15.6:0/35992885 >> v1:172.21.15.6:6789/0 conn(0x7fc614070f10 legacy=0x7fc6140cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc619600640 1 -- 172.21.15.6:0/35992885 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614071d70 con 0x7fc614145300 2024-03-06T07:33:44.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc613e00640 1 -- 172.21.15.6:0/35992885 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757710175 0 0) 0x7fc6140714d0 con 0x7fc614070f10 2024-03-06T07:33:44.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.155+0000 7fc61a000640 1 --2- 172.21.15.6:0/35992885 >> v2:172.21.15.178:3300/0 conn(0x7fc6140cd060 0x7fc61415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.156+0000 7fc619600640 1 --2- 172.21.15.6:0/35992885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc614145300 0x7fc6140b49a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc6140cc500 tx=0x7fc60802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13dd98782317a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.156+0000 7fc613e00640 1 -- 172.21.15.6:0/35992885 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6080315a0 con 0x7fc614145300 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.156+0000 7fc613e00640 1 -- 172.21.15.6:0/35992885 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc608037070 con 0x7fc614145300 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.156+0000 7fc613e00640 1 -- 172.21.15.6:0/35992885 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608031960 con 0x7fc614145300 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 -- 172.21.15.6:0/35992885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc614145300 msgr2=0x7fc6140b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 --2- 172.21.15.6:0/35992885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc614145300 0x7fc6140b49a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fc6140cc500 tx=0x7fc60802b840 comp rx=0 tx=0).stop 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 -- 172.21.15.6:0/35992885 shutdown_connections 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 --2- 172.21.15.6:0/35992885 >> v2:172.21.15.178:3300/0 conn(0x7fc6140cd060 0x7fc61415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 --2- 172.21.15.6:0/35992885 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc614145300 0x7fc6140b49a0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 -- 172.21.15.6:0/35992885 >> 172.21.15.6:0/35992885 conn(0x7fc61406b3d0 msgr2=0x7fc61406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 -- 172.21.15.6:0/35992885 shutdown_connections 2024-03-06T07:33:44.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.157+0000 7fc61aa00640 1 -- 172.21.15.6:0/35992885 wait complete. 2024-03-06T07:33:44.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.159+0000 7fc61aa00640 1 Processor -- start 2024-03-06T07:33:44.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.159+0000 7fc61aa00640 1 -- start start 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc61aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc614070f10 0x7fc6140aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc61aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 0x7fc6141f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc61aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc614145300 0x7fc6141f5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc61aa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61415cab0 con 0x7fc6140cd060 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc61aa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6141456e0 con 0x7fc614070f10 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc61aa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61415cc30 con 0x7fc614145300 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc618c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 0x7fc6141f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc618c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 0x7fc6141f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41492/0 (socket says 172.21.15.6:41492) 2024-03-06T07:33:44.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.160+0000 7fc618c00640 1 -- 172.21.15.6:0/1908424198 learned_addr learned my addr 172.21.15.6:0/1908424198 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc619600640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc614070f10 0x7fc6140aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc61a000640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc614145300 0x7fc6141f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc618c00640 1 -- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc614145300 msgr2=0x7fc6141f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc618c00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc614145300 0x7fc6141f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc618c00640 1 -- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc614070f10 msgr2=0x7fc6140aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc618c00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc614070f10 0x7fc6140aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc618c00640 1 -- 172.21.15.6:0/1908424198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614140150 con 0x7fc6140cd060 2024-03-06T07:33:44.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc618c00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 0x7fc6141f4bd0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fc60000c4d0 tx=0x7fc60000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600018070 con 0x7fc6140cd060 2024-03-06T07:33:44.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc600007500 con 0x7fc6140cd060 2024-03-06T07:33:44.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.161+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61413f8b0 con 0x7fc6140cd060 2024-03-06T07:33:44.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.162+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600004a20 con 0x7fc6140cd060 2024-03-06T07:33:44.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.162+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61413fe70 con 0x7fc6140cd060 2024-03-06T07:33:44.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.163+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc61404f330 con 0x7fc6140cd060 2024-03-06T07:33:44.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.164+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60001b070 con 0x7fc6140cd060 2024-03-06T07:33:44.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.165+0000 7fc611600640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc5f406b550 0x7fc5f406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.165+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc600089300 con 0x7fc6140cd060 2024-03-06T07:33:44.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.166+0000 7fc619600640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc5f406b550 0x7fc5f406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.167+0000 7fc619600640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc5f406b550 0x7fc5f406da10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc608036000 tx=0x7fc608008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.170+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6000584b0 con 0x7fc6140cd060 2024-03-06T07:33:44.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.436+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc6140afd60 con 0x7fc6140cd060 2024-03-06T07:33:44.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.437+0000 7fc611600640 1 -- 172.21.15.6:0/1908424198 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc60005ce20 con 0x7fc6140cd060 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:33:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.442 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.444 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:33:44.446 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.447 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.448 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.449 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:33:44.451 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc5f406b550 msgr2=0x7fc5f406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc5f406b550 0x7fc5f406da10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc608036000 tx=0x7fc608008030 comp rx=0 tx=0).stop 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 msgr2=0x7fc6141f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 0x7fc6141f4bd0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fc60000c4d0 tx=0x7fc60000c990 comp rx=0 tx=0).stop 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 shutdown_connections 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc5f406b550 0x7fc5f406da10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc614145300 0x7fc6141f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6140cd060 0x7fc6141f4bd0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 --2- 172.21.15.6:0/1908424198 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc614070f10 0x7fc6140aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 >> 172.21.15.6:0/1908424198 conn(0x7fc61406b3d0 msgr2=0x7fc6140cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 shutdown_connections 2024-03-06T07:33:44.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.439+0000 7fc61aa00640 1 -- 172.21.15.6:0/1908424198 wait complete. 2024-03-06T07:33:44.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:44.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.678+0000 7f8224600640 1 Processor -- start 2024-03-06T07:33:44.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.679+0000 7f8224600640 1 -- start start 2024-03-06T07:33:44.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.680+0000 7f8224600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f821c0cf5c0 0x7f821c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.680+0000 7f8224600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f821c071050 0x7f821c071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.680+0000 7f8224600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f821c0b4860 con 0x7f821c0b3c90 2024-03-06T07:33:44.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.680+0000 7f8224600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821c0ce160 con 0x7f821c0cf5c0 2024-03-06T07:33:44.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.680+0000 7f8224600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f821c0b4b40 con 0x7f821c071050 2024-03-06T07:33:44.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f821c0cf5c0 0x7f821c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f821c0cf5c0 0x7f821c0cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41508/0 (socket says 172.21.15.6:41508) 2024-03-06T07:33:44.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 -- 172.21.15.6:0/234760969 learned_addr learned my addr 172.21.15.6:0/234760969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8222600640 1 --2- 172.21.15.6:0/234760969 >> v2:172.21.15.178:3300/0 conn(0x7f821c071050 0x7f821c071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 -- 172.21.15.6:0/234760969 >> v2:172.21.15.178:3300/0 conn(0x7f821c071050 msgr2=0x7f821c071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 --2- 172.21.15.6:0/234760969 >> v2:172.21.15.178:3300/0 conn(0x7f821c071050 0x7f821c071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 -- 172.21.15.6:0/234760969 >> v1:172.21.15.178:6789/0 conn(0x7f821c0b3c90 legacy=0x7f821c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.681+0000 7f8221200640 1 -- 172.21.15.6:0/234760969 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c0b4e20 con 0x7f821c0cf5c0 2024-03-06T07:33:44.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.682+0000 7f8221200640 1 --2- 172.21.15.6:0/234760969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c0cf5c0 0x7f821c0cdc20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f821c074860 tx=0x7f820802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aead9191cfac8a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f821be00640 1 -- 172.21.15.6:0/234760969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208034070 con 0x7f821c0cf5c0 2024-03-06T07:33:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f821be00640 1 -- 172.21.15.6:0/234760969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8208030860 con 0x7f821c0cf5c0 2024-03-06T07:33:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f821be00640 1 -- 172.21.15.6:0/234760969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208030c30 con 0x7f821c0cf5c0 2024-03-06T07:33:44.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f8224600640 1 -- 172.21.15.6:0/234760969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c0cf5c0 msgr2=0x7f821c0cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f8224600640 1 --2- 172.21.15.6:0/234760969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c0cf5c0 0x7f821c0cdc20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f821c074860 tx=0x7f820802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f8224600640 1 -- 172.21.15.6:0/234760969 shutdown_connections 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.683+0000 7f8224600640 1 --2- 172.21.15.6:0/234760969 >> v2:172.21.15.178:3300/0 conn(0x7f821c071050 0x7f821c071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.684+0000 7f8224600640 1 --2- 172.21.15.6:0/234760969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c0cf5c0 0x7f821c0cdc20 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.684+0000 7f8224600640 1 -- 172.21.15.6:0/234760969 >> 172.21.15.6:0/234760969 conn(0x7f821c06afb0 msgr2=0x7f821c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.684+0000 7f8224600640 1 -- 172.21.15.6:0/234760969 shutdown_connections 2024-03-06T07:33:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.684+0000 7f8224600640 1 -- 172.21.15.6:0/234760969 wait complete. 2024-03-06T07:33:44.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.685+0000 7f8224600640 1 Processor -- start 2024-03-06T07:33:44.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.685+0000 7f8224600640 1 -- start start 2024-03-06T07:33:44.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.686+0000 7f8224600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 0x7f821c139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.686+0000 7f8224600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f821c0b3c90 0x7f821c14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.686+0000 7f8224600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821c0cf5c0 0x7f821c152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.686+0000 7f8224600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c0719d0 con 0x7f821c071050 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.686+0000 7f8224600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c15b340 con 0x7f821c0b3c90 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.686+0000 7f8224600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821c071b50 con 0x7f821c0cf5c0 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 0x7f821c139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 0x7f821c139720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41518/0 (socket says 172.21.15.6:41518) 2024-03-06T07:33:44.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 -- 172.21.15.6:0/3020388889 learned_addr learned my addr 172.21.15.6:0/3020388889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:44.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221200640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f821c0b3c90 0x7f821c14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 -- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821c0cf5c0 msgr2=0x7f821c152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8222600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821c0cf5c0 0x7f821c152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821c0cf5c0 0x7f821c152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 -- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f821c0b3c90 msgr2=0x7f821c14fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f821c0b3c90 0x7f821c14fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 -- 172.21.15.6:0/3020388889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c158a20 con 0x7f821c071050 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.687+0000 7f8221c00640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 0x7f821c139720 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f821000c460 tx=0x7f821000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.688+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210016020 con 0x7f821c071050 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.688+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8210012870 con 0x7f821c071050 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.688+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210012c60 con 0x7f821c071050 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.688+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c158d00 con 0x7f821c071050 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.688+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c1592c0 con 0x7f821c071050 2024-03-06T07:33:44.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.689+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c069510 con 0x7f821c071050 2024-03-06T07:33:44.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.690+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8210019020 con 0x7f821c071050 2024-03-06T07:33:44.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.691+0000 7f8219600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81fc06b4d0 0x7f81fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:44.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.691+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8210089c70 con 0x7f821c071050 2024-03-06T07:33:44.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.692+0000 7f8221200640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81fc06b4d0 0x7f81fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:44.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.693+0000 7f8221200640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81fc06b4d0 0x7f81fc06d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8208036000 tx=0x7f8208008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:44.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.696+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8210020030 con 0x7f821c071050 2024-03-06T07:33:44.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.895+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f821c06d8d0 con 0x7f821c071050 2024-03-06T07:33:44.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.897+0000 7f8219600640 1 -- 172.21.15.6:0/3020388889 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12821 (secure 0 0 0) 0x7f8210058da0 con 0x7f821c071050 2024-03-06T07:33:44.900 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:44.900 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":104,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:43.928093+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81fc06b4d0 msgr2=0x7f81fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81fc06b4d0 0x7f81fc06d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8208036000 tx=0x7f8208008030 comp rx=0 tx=0).stop 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 msgr2=0x7f821c139720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 0x7f821c139720 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f821000c460 tx=0x7f821000c920 comp rx=0 tx=0).stop 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 shutdown_connections 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f81fc06b4d0 0x7f81fc06d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f821c0b3c90 0x7f821c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f821c0cf5c0 0x7f821c152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 --2- 172.21.15.6:0/3020388889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f821c071050 0x7f821c139720 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 >> 172.21.15.6:0/3020388889 conn(0x7f821c06afb0 msgr2=0x7f821c0ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 shutdown_connections 2024-03-06T07:33:44.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:44.900+0000 7f8224600640 1 -- 172.21.15.6:0/3020388889 wait complete. 2024-03-06T07:33:44.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:33:44.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:33:45.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.138+0000 7ff947200640 1 Processor -- start 2024-03-06T07:33:45.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.138+0000 7ff947200640 1 -- start start 2024-03-06T07:33:45.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.139+0000 7ff947200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff94815dee0 0x7ff9480d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.139+0000 7ff947200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480d4860 0x7ff9480d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.139+0000 7ff947200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9480ceca0 con 0x7ff9480cfa60 2024-03-06T07:33:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.139+0000 7ff947200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9480cf5d0 con 0x7ff94815dee0 2024-03-06T07:33:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.139+0000 7ff947200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9480cef80 con 0x7ff9480d4860 2024-03-06T07:33:45.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff945e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff94815dee0 0x7ff9480d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff946800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff9480cfa60 0x7ff9480b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37882/0 (socket says 172.21.15.6:37882) 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff946800640 1 -- 172.21.15.6:0/1972691268 learned_addr learned my addr 172.21.15.6:0/1972691268 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff945400640 1 --2- 172.21.15.6:0/1972691268 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480d4860 0x7ff9480d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff945e00640 1 -- 172.21.15.6:0/1972691268 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480d4860 msgr2=0x7ff9480d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff945e00640 1 --2- 172.21.15.6:0/1972691268 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480d4860 0x7ff9480d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff945e00640 1 -- 172.21.15.6:0/1972691268 >> v1:172.21.15.6:6789/0 conn(0x7ff9480cfa60 legacy=0x7ff9480b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.141+0000 7ff945e00640 1 -- 172.21.15.6:0/1972691268 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9480cf260 con 0x7ff94815dee0 2024-03-06T07:33:45.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.140+0000 7ff944a00640 1 -- 172.21.15.6:0/1972691268 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423910965 0 0) 0x7ff9480ceca0 con 0x7ff9480cfa60 2024-03-06T07:33:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.142+0000 7ff945e00640 1 --2- 172.21.15.6:0/1972691268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff94815dee0 0x7ff9480d18a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff938009d90 tx=0x7ff93802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2a74b33bfefe9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.142+0000 7ff944a00640 1 -- 172.21.15.6:0/1972691268 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938039070 con 0x7ff94815dee0 2024-03-06T07:33:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.142+0000 7ff944a00640 1 -- 172.21.15.6:0/1972691268 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff93802cc50 con 0x7ff94815dee0 2024-03-06T07:33:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.142+0000 7ff944a00640 1 -- 172.21.15.6:0/1972691268 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff938034440 con 0x7ff94815dee0 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.142+0000 7ff947200640 1 -- 172.21.15.6:0/1972691268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff94815dee0 msgr2=0x7ff9480d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.142+0000 7ff947200640 1 --2- 172.21.15.6:0/1972691268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff94815dee0 0x7ff9480d18a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff938009d90 tx=0x7ff93802bea0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.143+0000 7ff947200640 1 -- 172.21.15.6:0/1972691268 shutdown_connections 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.143+0000 7ff947200640 1 --2- 172.21.15.6:0/1972691268 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480d4860 0x7ff9480d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.143+0000 7ff947200640 1 --2- 172.21.15.6:0/1972691268 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff94815dee0 0x7ff9480d18a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.143+0000 7ff947200640 1 -- 172.21.15.6:0/1972691268 >> 172.21.15.6:0/1972691268 conn(0x7ff9480d3cd0 msgr2=0x7ff9480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.143+0000 7ff947200640 1 -- 172.21.15.6:0/1972691268 shutdown_connections 2024-03-06T07:33:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.143+0000 7ff947200640 1 -- 172.21.15.6:0/1972691268 wait complete. 2024-03-06T07:33:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.144+0000 7ff947200640 1 Processor -- start 2024-03-06T07:33:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.144+0000 7ff947200640 1 -- start start 2024-03-06T07:33:45.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.145+0000 7ff947200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480cfa60 0x7ff9480cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.145+0000 7ff947200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 0x7ff9481501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.145+0000 7ff947200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff94815dee0 0x7ff9480cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.145+0000 7ff947200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9480d1de0 con 0x7ff9480d4860 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.145+0000 7ff947200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94815e2c0 con 0x7ff94815dee0 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.145+0000 7ff947200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9480d1f60 con 0x7ff9480cfa60 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 0x7ff9481501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 0x7ff9481501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41532/0 (socket says 172.21.15.6:41532) 2024-03-06T07:33:45.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 -- 172.21.15.6:0/1748886660 learned_addr learned my addr 172.21.15.6:0/1748886660 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945e00640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480cfa60 0x7ff9480cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff946800640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff94815dee0 0x7ff9480cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 -- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480cfa60 msgr2=0x7ff9480cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480cfa60 0x7ff9480cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 -- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff94815dee0 msgr2=0x7ff9480cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff94815dee0 0x7ff9480cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.146+0000 7ff945400640 1 -- 172.21.15.6:0/1748886660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9480cb780 con 0x7ff9480d4860 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.147+0000 7ff945400640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 0x7ff9481501a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7ff93400d800 tx=0x7ff93400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.147+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93400b6c0 con 0x7ff9480d4860 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.147+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff934019070 con 0x7ff9480d4860 2024-03-06T07:33:45.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.147+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9480cdc30 con 0x7ff9480d4860 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.147+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93400ba80 con 0x7ff9480d4860 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.147+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9480cbce0 con 0x7ff9480d4860 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.148+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94804f330 con 0x7ff9480d4860 2024-03-06T07:33:45.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.149+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff93400bc20 con 0x7ff9480d4860 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.151+0000 7ff93e000640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.152+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff93408a250 con 0x7ff9480d4860 2024-03-06T07:33:45.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.152+0000 7ff945e00640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.153+0000 7ff945e00640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff92806b4d0 0x7ff92806d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff938036000 tx=0x7ff938030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.156+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9480cdc30 con 0x7ff9480d4860 2024-03-06T07:33:45.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.356+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff94808d7d0 con 0x7ff9480d4860 2024-03-06T07:33:45.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.356+0000 7ff93e000640 1 -- 172.21.15.6:0/1748886660 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff93405de20 con 0x7ff9480d4860 2024-03-06T07:33:45.359 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:33:45.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff92806b4d0 msgr2=0x7ff92806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff92806b4d0 0x7ff92806d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff938036000 tx=0x7ff938030000 comp rx=0 tx=0).stop 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 msgr2=0x7ff9481501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 0x7ff9481501a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7ff93400d800 tx=0x7ff93400dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 shutdown_connections 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff92806b4d0 0x7ff92806d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff94815dee0 0x7ff9480cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff9480d4860 0x7ff9481501a0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 --2- 172.21.15.6:0/1748886660 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff9480cfa60 0x7ff9480cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 >> 172.21.15.6:0/1748886660 conn(0x7ff9480d3cd0 msgr2=0x7ff9480b8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 shutdown_connections 2024-03-06T07:33:45.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.359+0000 7ff947200640 1 -- 172.21.15.6:0/1748886660 wait complete. 2024-03-06T07:33:45.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:33:45.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.597+0000 7fe66b400640 1 Processor -- start 2024-03-06T07:33:45.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.598+0000 7fe66b400640 1 -- start start 2024-03-06T07:33:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe66b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6640b34e0 0x7fe66415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe66b400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66415cc40 con 0x7fe6640af580 2024-03-06T07:33:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe66b400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66415cf20 con 0x7fe6640af1a0 2024-03-06T07:33:45.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe66b400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66415d200 con 0x7fe6640b34e0 2024-03-06T07:33:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe663e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6640af580 0x7fe6640b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37896/0 (socket says 172.21.15.6:37896) 2024-03-06T07:33:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe663e00640 1 -- 172.21.15.6:0/2063487808 learned_addr learned my addr 172.21.15.6:0/2063487808 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.599+0000 7fe669400640 1 --2- 172.21.15.6:0/2063487808 >> v2:172.21.15.6:3300/0 conn(0x7fe6640b34e0 0x7fe66415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.600+0000 7fe663400640 1 -- 172.21.15.6:0/2063487808 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292573682 0 0) 0x7fe66415cc40 con 0x7fe6640af580 2024-03-06T07:33:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.600+0000 7fe669400640 1 -- 172.21.15.6:0/2063487808 >> v1:172.21.15.178:6789/0 conn(0x7fe6640af1a0 legacy=0x7fe6640b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:45.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.600+0000 7fe669400640 1 -- 172.21.15.6:0/2063487808 >> v1:172.21.15.6:6789/0 conn(0x7fe6640af580 legacy=0x7fe6640b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.600+0000 7fe669400640 1 -- 172.21.15.6:0/2063487808 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66415d4e0 con 0x7fe6640b34e0 2024-03-06T07:33:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.601+0000 7fe669400640 1 --2- 172.21.15.6:0/2063487808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640b34e0 0x7fe66415c600 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0097a0 tx=0x7fe65c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fc9d59fd8bebe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.601+0000 7fe663400640 1 -- 172.21.15.6:0/2063487808 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0315a0 con 0x7fe6640b34e0 2024-03-06T07:33:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe663400640 1 -- 172.21.15.6:0/2063487808 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c037070 con 0x7fe6640b34e0 2024-03-06T07:33:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe663400640 1 -- 172.21.15.6:0/2063487808 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c031960 con 0x7fe6640b34e0 2024-03-06T07:33:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 -- 172.21.15.6:0/2063487808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640b34e0 msgr2=0x7fe66415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 --2- 172.21.15.6:0/2063487808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640b34e0 0x7fe66415c600 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0097a0 tx=0x7fe65c02b840 comp rx=0 tx=0).stop 2024-03-06T07:33:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 -- 172.21.15.6:0/2063487808 shutdown_connections 2024-03-06T07:33:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 --2- 172.21.15.6:0/2063487808 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640b34e0 0x7fe66415c600 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 -- 172.21.15.6:0/2063487808 >> 172.21.15.6:0/2063487808 conn(0x7fe66406afb0 msgr2=0x7fe66406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 -- 172.21.15.6:0/2063487808 shutdown_connections 2024-03-06T07:33:45.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.602+0000 7fe66b400640 1 -- 172.21.15.6:0/2063487808 wait complete. 2024-03-06T07:33:45.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.604+0000 7fe66b400640 1 Processor -- start 2024-03-06T07:33:45.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.604+0000 7fe66b400640 1 -- start start 2024-03-06T07:33:45.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe66b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 0x7fe6641f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe66b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6640af580 0x7fe66415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe66b400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6640b34e0 0x7fe66409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe66b400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe664155e70 con 0x7fe6640af1a0 2024-03-06T07:33:45.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe66b400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6640cde30 con 0x7fe6640b34e0 2024-03-06T07:33:45.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe66b400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe664155ff0 con 0x7fe6640af580 2024-03-06T07:33:45.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.605+0000 7fe668a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 0x7fe6641f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 0x7fe6641f7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41548/0 (socket says 172.21.15.6:41548) 2024-03-06T07:33:45.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 -- 172.21.15.6:0/925325695 learned_addr learned my addr 172.21.15.6:0/925325695 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:45.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe669400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6640b34e0 0x7fe66409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 -- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6640af580 msgr2=0x7fe66415aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe663e00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6640af580 0x7fe66415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6640af580 0x7fe66415aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 -- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6640b34e0 msgr2=0x7fe66409b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6640b34e0 0x7fe66409b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 -- 172.21.15.6:0/925325695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66409b170 con 0x7fe6640af1a0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe669400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6640b34e0 0x7fe66409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.606+0000 7fe668a00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 0x7fe6641f7d20 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fe65400c430 tx=0x7fe65400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.607+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654011790 con 0x7fe6640af1a0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.607+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe654017070 con 0x7fe6640af1a0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.607+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654011b70 con 0x7fe6640af1a0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.607+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66409abb0 con 0x7fe6640af1a0 2024-03-06T07:33:45.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.607+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6641f7520 con 0x7fe6640af1a0 2024-03-06T07:33:45.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.608+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66404f3a0 con 0x7fe6640af1a0 2024-03-06T07:33:45.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.609+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe654011d10 con 0x7fe6640af1a0 2024-03-06T07:33:45.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.610+0000 7fe660c00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64006b500 0x7fe64006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:45.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.610+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe654089d60 con 0x7fe6640af1a0 2024-03-06T07:33:45.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.610+0000 7fe663e00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64006b500 0x7fe64006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:45.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.611+0000 7fe663e00640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64006b500 0x7fe64006d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe664153dd0 tx=0x7fe64c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:45.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.614+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe654058f20 con 0x7fe6640af1a0 2024-03-06T07:33:45.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.818+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe66415fba0 con 0x7fe6640af1a0 2024-03-06T07:33:45.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.819+0000 7fe660c00640 1 -- 172.21.15.6:0/925325695 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe654058820 con 0x7fe6640af1a0 2024-03-06T07:33:45.821 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:45.821 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:33:45.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:45.822 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:33:45.822 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:33:45.822 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:45.822 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:33:45.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64006b500 msgr2=0x7fe64006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64006b500 0x7fe64006d9c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe664153dd0 tx=0x7fe64c009040 comp rx=0 tx=0).stop 2024-03-06T07:33:45.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 msgr2=0x7fe6641f7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:45.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 0x7fe6641f7d20 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fe65400c430 tx=0x7fe65400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 shutdown_connections 2024-03-06T07:33:45.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64006b500 0x7fe64006d9c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6640b34e0 0x7fe66409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6640af580 0x7fe66415aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 --2- 172.21.15.6:0/925325695 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6640af1a0 0x7fe6641f7d20 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:45.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 >> 172.21.15.6:0/925325695 conn(0x7fe66406afb0 msgr2=0x7fe6641603a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:45.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 shutdown_connections 2024-03-06T07:33:45.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:45.822+0000 7fe66b400640 1 -- 172.21.15.6:0/925325695 wait complete. 2024-03-06T07:33:45.842 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:33:45.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:33:46.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.064+0000 7f0423000640 1 Processor -- start 2024-03-06T07:33:46.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.064+0000 7f0423000640 1 -- start start 2024-03-06T07:33:46.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0423000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 0x7f041c0cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0423000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f041c15e360 0x7f041c0d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0423000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c0d17e0 con 0x7f041c0b2af0 2024-03-06T07:33:46.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0423000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041c0b2020 con 0x7f041c15e360 2024-03-06T07:33:46.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0423000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0afa30 con 0x7f041c15df80 2024-03-06T07:33:46.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0422600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f041c0b2af0 0x7f041c161cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37898/0 (socket says 172.21.15.6:37898) 2024-03-06T07:33:46.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.066+0000 7f0422600640 1 -- 172.21.15.6:0/1549377995 learned_addr learned my addr 172.21.15.6:0/1549377995 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:46.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f0421200640 1 --2- 172.21.15.6:0/1549377995 >> v2:172.21.15.178:3300/0 conn(0x7f041c15e360 0x7f041c0d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f0421c00640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 0x7f041c0cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251763557 0 0) 0x7f0414001490 con 0x7f041c0b2af0 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0404003520 con 0x7f041c0b2af0 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1230161177 0 0) 0x7f0404003520 con 0x7f041c0b2af0 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 msgr2=0x7f041c0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 0x7f041c0cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 >> v2:172.21.15.178:3300/0 conn(0x7f041c15e360 msgr2=0x7f041c0d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 --2- 172.21.15.6:0/1549377995 >> v2:172.21.15.178:3300/0 conn(0x7f041c15e360 0x7f041c0d12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0b1410 con 0x7f041c0b2af0 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0414001bc0 con 0x7f041c0b2af0 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 >> v1:172.21.15.6:6789/0 conn(0x7f041c0b2af0 legacy=0x7f041c161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 0x7f04040039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f0404004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0404004d40 0x7f04040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0b2020 con 0x7f0404003f80 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0afa30 con 0x7f0404004d40 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.067+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040400b9c0 con 0x7f041c15df80 2024-03-06T07:33:46.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421200640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f0404004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421c00640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 0x7f04040039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0422600640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0404004d40 0x7f04040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421200640 1 -- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 msgr2=0x7f04040039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421200640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15df80 0x7f04040039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421200640 1 -- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0404004d40 msgr2=0x7f04040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421200640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0404004d40 0x7f04040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0421200640 1 -- 172.21.15.6:0/1549377995 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404003520 con 0x7f0404003f80 2024-03-06T07:33:46.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.068+0000 7f0422600640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0404004d40 0x7f04040051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.069+0000 7f0421200640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f0404004800 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0408009af0 tx=0x7f0404005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb5c12938649039 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408039070 con 0x7f0404003f80 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040802c710 con 0x7f0404003f80 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f041be00640 1 -- 172.21.15.6:0/1549377995 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408009690 con 0x7f0404003f80 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0423000640 1 -- 172.21.15.6:0/1549377995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 msgr2=0x7f0404004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0423000640 1 --2- 172.21.15.6:0/1549377995 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f0404004800 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0408009af0 tx=0x7f0404005830 comp rx=0 tx=0).stop 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0422600640 1 -- 172.21.15.6:0/1549377995 reap_dead start 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0423000640 1 -- 172.21.15.6:0/1549377995 shutdown_connections 2024-03-06T07:33:46.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0423000640 1 -- 172.21.15.6:0/1549377995 >> 172.21.15.6:0/1549377995 conn(0x7f041c073100 msgr2=0x7f041c09aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0423000640 1 -- 172.21.15.6:0/1549377995 shutdown_connections 2024-03-06T07:33:46.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.070+0000 7f0423000640 1 -- 172.21.15.6:0/1549377995 wait complete. 2024-03-06T07:33:46.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.072+0000 7f0423000640 1 Processor -- start 2024-03-06T07:33:46.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.072+0000 7f0423000640 1 -- start start 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.073+0000 7f0423000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f041c0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.073+0000 7f0423000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15e740 0x7f041c0ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.073+0000 7f0423000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f041c14e000 0x7f041c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.073+0000 7f0423000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0b1cb0 con 0x7f0404003f80 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.073+0000 7f0423000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0b19d0 con 0x7f041c14e000 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.073+0000 7f0423000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c0b24b0 con 0x7f041c15e740 2024-03-06T07:33:46.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f041c0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f041c0cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41566/0 (socket says 172.21.15.6:41566) 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 -- 172.21.15.6:0/2228704200 learned_addr learned my addr 172.21.15.6:0/2228704200 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 -- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15e740 msgr2=0x7f041c0ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15e740 0x7f041c0ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 -- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f041c14e000 msgr2=0x7f041c14e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0422600640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f041c14e000 0x7f041c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f041c14e000 0x7f041c14e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 -- 172.21.15.6:0/2228704200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c1609c0 con 0x7f0404003f80 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0422600640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f041c14e000 0x7f041c14e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:46.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.074+0000 7f0421c00640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f041c0cc9f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f04100108c0 tx=0x7f0410010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.075+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041000f040 con 0x7f0404003f80 2024-03-06T07:33:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.075+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041000eb00 con 0x7f0404003f80 2024-03-06T07:33:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.075+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041001c400 con 0x7f0404003f80 2024-03-06T07:33:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.075+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c0cadc0 con 0x7f0404003f80 2024-03-06T07:33:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.075+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041c160f40 con 0x7f0404003f80 2024-03-06T07:33:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.076+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041c0afec0 con 0x7f0404003f80 2024-03-06T07:33:46.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.077+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f041001c5a0 con 0x7f0404003f80 2024-03-06T07:33:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.078+0000 7f0419600640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03fc06b500 0x7f03fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.078+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f041008a3f0 con 0x7f0404003f80 2024-03-06T07:33:46.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.079+0000 7f0421200640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03fc06b500 0x7f03fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:46.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.080+0000 7f0421200640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03fc06b500 0x7f03fc06d9c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0408036000 tx=0x7f040802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:46.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.082+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f041c0afec0 con 0x7f0404003f80 2024-03-06T07:33:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.282+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f041c04f330 con 0x7f0404003f80 2024-03-06T07:33:46.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.946+0000 7f0419600640 1 -- 172.21.15.6:0/2228704200 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0410018030 con 0x7f0404003f80 2024-03-06T07:33:46.949 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03fc06b500 msgr2=0x7f03fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03fc06b500 0x7f03fc06d9c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0408036000 tx=0x7f040802e000 comp rx=0 tx=0).stop 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 msgr2=0x7f041c0cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f041c0cc9f0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f04100108c0 tx=0x7f0410010d80 comp rx=0 tx=0).stop 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 shutdown_connections 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f03fc06b500 0x7f03fc06d9c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f041c14e000 0x7f041c14e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f041c15e740 0x7f041c0ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 --2- 172.21.15.6:0/2228704200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0404003f80 0x7f041c0cc9f0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:46.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.951+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 >> 172.21.15.6:0/2228704200 conn(0x7f041c073100 msgr2=0x7f041c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:46.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.952+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 shutdown_connections 2024-03-06T07:33:46.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:46.952+0000 7f0423000640 1 -- 172.21.15.6:0/2228704200 wait complete. 2024-03-06T07:33:46.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:33:47.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.178+0000 7fce17e00640 1 Processor -- start 2024-03-06T07:33:47.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.178+0000 7fce17e00640 1 -- start start 2024-03-06T07:33:47.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.179+0000 7fce17e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce1815e1c0 0x7fce180cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.179+0000 7fce17e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fce180d22a0 0x7fce180b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.179+0000 7fce17e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce180ce800 con 0x7fce180d0650 2024-03-06T07:33:47.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.179+0000 7fce17e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce180d2cb0 con 0x7fce1815e1c0 2024-03-06T07:33:47.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.179+0000 7fce17e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce180ceaa0 con 0x7fce180d22a0 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce1815e1c0 0x7fce180cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce1815e1c0 0x7fce180cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41574/0 (socket says 172.21.15.6:41574) 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 -- 172.21.15.6:0/2637074306 learned_addr learned my addr 172.21.15.6:0/2637074306 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce17400640 1 --2- 172.21.15.6:0/2637074306 >> v2:172.21.15.178:3300/0 conn(0x7fce180d22a0 0x7fce180b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 -- 172.21.15.6:0/2637074306 >> v2:172.21.15.178:3300/0 conn(0x7fce180d22a0 msgr2=0x7fce180b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 --2- 172.21.15.6:0/2637074306 >> v2:172.21.15.178:3300/0 conn(0x7fce180d22a0 0x7fce180b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 -- 172.21.15.6:0/2637074306 >> v1:172.21.15.178:6789/0 conn(0x7fce180d0650 legacy=0x7fce180d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:47.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.180+0000 7fce16000640 1 -- 172.21.15.6:0/2637074306 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce180ced80 con 0x7fce1815e1c0 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.182+0000 7fce16000640 1 --2- 172.21.15.6:0/2637074306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce1815e1c0 0x7fce180cf860 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fce180d03f0 tx=0x7fce0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87446e731e87459 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.182+0000 7fce15600640 1 -- 172.21.15.6:0/2637074306 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00035020 con 0x7fce1815e1c0 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.182+0000 7fce15600640 1 -- 172.21.15.6:0/2637074306 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00031870 con 0x7fce1815e1c0 2024-03-06T07:33:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.182+0000 7fce15600640 1 -- 172.21.15.6:0/2637074306 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00031c40 con 0x7fce1815e1c0 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 -- 172.21.15.6:0/2637074306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce1815e1c0 msgr2=0x7fce180cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 --2- 172.21.15.6:0/2637074306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce1815e1c0 0x7fce180cf860 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fce180d03f0 tx=0x7fce0002b910 comp rx=0 tx=0).stop 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 -- 172.21.15.6:0/2637074306 shutdown_connections 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 --2- 172.21.15.6:0/2637074306 >> v2:172.21.15.178:3300/0 conn(0x7fce180d22a0 0x7fce180b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 --2- 172.21.15.6:0/2637074306 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce1815e1c0 0x7fce180cf860 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 -- 172.21.15.6:0/2637074306 >> 172.21.15.6:0/2637074306 conn(0x7fce1808d2a0 msgr2=0x7fce180d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 -- 172.21.15.6:0/2637074306 shutdown_connections 2024-03-06T07:33:47.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.183+0000 7fce17e00640 1 -- 172.21.15.6:0/2637074306 wait complete. 2024-03-06T07:33:47.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.185+0000 7fce17e00640 1 Processor -- start 2024-03-06T07:33:47.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.185+0000 7fce17e00640 1 -- start start 2024-03-06T07:33:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 0x7fce18160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce17e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fce180d22a0 0x7fce18150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce17e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1815e1c0 0x7fce180cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce17e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce180d4a30 con 0x7fce180d0650 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce17e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce180cf340 con 0x7fce1815e1c0 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce17e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce180d4bb0 con 0x7fce180d22a0 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 0x7fce18160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 0x7fce18160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41582/0 (socket says 172.21.15.6:41582) 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 -- 172.21.15.6:0/4290229471 learned_addr learned my addr 172.21.15.6:0/4290229471 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 -- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fce180d22a0 msgr2=0x7fce18150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fce180d22a0 0x7fce18150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 -- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1815e1c0 msgr2=0x7fce180cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1815e1c0 0x7fce180cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.186+0000 7fce16a00640 1 -- 172.21.15.6:0/4290229471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce181516d0 con 0x7fce180d0650 2024-03-06T07:33:47.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.187+0000 7fce16a00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 0x7fce18160360 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fce0800b3f0 tx=0x7fce0800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.187+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08015020 con 0x7fce180d0650 2024-03-06T07:33:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.187+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce08011830 con 0x7fce180d0650 2024-03-06T07:33:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.187+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce08011c20 con 0x7fce180d0650 2024-03-06T07:33:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.187+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce181519b0 con 0x7fce180d0650 2024-03-06T07:33:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.187+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce18151f70 con 0x7fce180d0650 2024-03-06T07:33:47.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.188+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1804f330 con 0x7fce180d0650 2024-03-06T07:33:47.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.190+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce08018020 con 0x7fce180d0650 2024-03-06T07:33:47.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.190+0000 7fce0ea00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdf406b4d0 0x7fcdf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:47.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.191+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fce0808a100 con 0x7fce180d0650 2024-03-06T07:33:47.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.191+0000 7fce16000640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdf406b4d0 0x7fcdf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:47.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.193+0000 7fce16000640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdf406b4d0 0x7fcdf406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fce00036000 tx=0x7fce00030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:47.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.195+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce0801f030 con 0x7fce180d0650 2024-03-06T07:33:47.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.395+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fce1808d810 con 0x7fce180d0650 2024-03-06T07:33:47.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.953+0000 7fce0ea00640 1 -- 172.21.15.6:0/4290229471 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7fce080591d0 con 0x7fce180d0650 2024-03-06T07:33:47.955 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdf406b4d0 msgr2=0x7fcdf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdf406b4d0 0x7fcdf406d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fce00036000 tx=0x7fce00030000 comp rx=0 tx=0).stop 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 msgr2=0x7fce18160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 0x7fce18160360 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fce0800b3f0 tx=0x7fce0800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 shutdown_connections 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdf406b4d0 0x7fcdf406d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1815e1c0 0x7fce180cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fce180d22a0 0x7fce18150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 --2- 172.21.15.6:0/4290229471 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce180d0650 0x7fce18160360 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 >> 172.21.15.6:0/4290229471 conn(0x7fce1808d2a0 msgr2=0x7fce180b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 shutdown_connections 2024-03-06T07:33:47.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:47.961+0000 7fce17e00640 1 -- 172.21.15.6:0/4290229471 wait complete. 2024-03-06T07:33:47.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:33:48.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.180+0000 7fd977600640 1 Processor -- start 2024-03-06T07:33:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.180+0000 7fd977600640 1 -- start start 2024-03-06T07:33:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.182+0000 7fd977600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd970072520 0x7fd9700cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.182+0000 7fd977600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd970072df0 con 0x7fd970071e20 2024-03-06T07:33:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.182+0000 7fd977600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700cdab0 con 0x7fd97015bf20 2024-03-06T07:33:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.182+0000 7fd977600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9700b4d70 con 0x7fd970072520 2024-03-06T07:33:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.182+0000 7fd975600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd970071e20 0x7fd9700ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37914/0 (socket says 172.21.15.6:37914) 2024-03-06T07:33:48.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.182+0000 7fd975600640 1 -- 172.21.15.6:0/2712459688 learned_addr learned my addr 172.21.15.6:0/2712459688 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2712459688 >> v2:172.21.15.178:3300/0 conn(0x7fd970072520 0x7fd9700cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808971612 0 0) 0x7fd970072df0 con 0x7fd970071e20 2024-03-06T07:33:48.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd958003520 con 0x7fd970071e20 2024-03-06T07:33:48.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852354487 0 0) 0x7fd958003520 con 0x7fd970071e20 2024-03-06T07:33:48.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 >> v2:172.21.15.178:3300/0 conn(0x7fd970072520 msgr2=0x7fd9700cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 --2- 172.21.15.6:0/2712459688 >> v2:172.21.15.178:3300/0 conn(0x7fd970072520 0x7fd9700cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 >> v1:172.21.15.178:6789/0 conn(0x7fd97015bf20 legacy=0x7fd9700acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700b5050 con 0x7fd970071e20 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd9680019f0 con 0x7fd970071e20 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 >> v1:172.21.15.6:6789/0 conn(0x7fd970071e20 legacy=0x7fd9700ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd958003a00 0x7fd958003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd958004800 0x7fd958004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.183+0000 7fd96f400640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9580051c0 0x7fd95800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700b4d70 con 0x7fd958003a00 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95800c0c0 con 0x7fd958004800 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95800c2c0 con 0x7fd9580051c0 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2712459688 >> v2:172.21.15.178:3300/0 conn(0x7fd970072520 0x7fd9700cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd974c00640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd958003a00 0x7fd958003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd974c00640 1 -- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9580051c0 msgr2=0x7fd95800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd975600640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9580051c0 0x7fd95800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd958004800 0x7fd958004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd974c00640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9580051c0 0x7fd95800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd974c00640 1 -- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd958004800 msgr2=0x7fd958004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd974c00640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd958004800 0x7fd958004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd974c00640 1 -- 172.21.15.6:0/2712459688 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958003520 con 0x7fd958003a00 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd975600640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd9580051c0 0x7fd95800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd958004800 0x7fd958004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:48.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.184+0000 7fd975600640 1 -- 172.21.15.6:0/2712459688 reap_dead start 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.185+0000 7fd974c00640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd958003a00 0x7fd958003e20 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd964001750 tx=0x7fd958005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca182e1acac4a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964001370 con 0x7fd958003a00 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9640052d0 con 0x7fd958003a00 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd96f400640 1 -- 172.21.15.6:0/2712459688 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964003830 con 0x7fd958003a00 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd977600640 1 -- 172.21.15.6:0/2712459688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd958003a00 msgr2=0x7fd958003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd977600640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd958003a00 0x7fd958003e20 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd964001750 tx=0x7fd958005830 comp rx=0 tx=0).stop 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd977600640 1 -- 172.21.15.6:0/2712459688 shutdown_connections 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd977600640 1 --2- 172.21.15.6:0/2712459688 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd958003a00 0x7fd958003e20 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.186+0000 7fd977600640 1 -- 172.21.15.6:0/2712459688 >> 172.21.15.6:0/2712459688 conn(0x7fd97006b3d0 msgr2=0x7fd97006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.187+0000 7fd977600640 1 -- 172.21.15.6:0/2712459688 shutdown_connections 2024-03-06T07:33:48.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.187+0000 7fd977600640 1 -- 172.21.15.6:0/2712459688 wait complete. 2024-03-06T07:33:48.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.188+0000 7fd977600640 1 Processor -- start 2024-03-06T07:33:48.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.188+0000 7fd977600640 1 -- start start 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.189+0000 7fd977600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd958003a00 0x7fd97015e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.189+0000 7fd977600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 0x7fd9701563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.189+0000 7fd977600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd97015ed10 0x7fd9701e66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.189+0000 7fd977600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700b4b10 con 0x7fd970157910 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.189+0000 7fd977600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700b4990 con 0x7fd97015ed10 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.189+0000 7fd977600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700acd40 con 0x7fd958003a00 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 0x7fd9701563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 0x7fd9701563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41594/0 (socket says 172.21.15.6:41594) 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 -- 172.21.15.6:0/2631662482 learned_addr learned my addr 172.21.15.6:0/2631662482 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd974c00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd958003a00 0x7fd97015e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 -- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd958003a00 msgr2=0x7fd97015e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd958003a00 0x7fd97015e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 -- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd97015ed10 msgr2=0x7fd9701e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd97015ed10 0x7fd9701e66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 -- 172.21.15.6:0/2631662482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9701f8720 con 0x7fd970157910 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd974c00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd958003a00 0x7fd97015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.190+0000 7fd96fe00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 0x7fd9701563f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd95c010c00 tx=0x7fd95c00e540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.191+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00f040 con 0x7fd970157910 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.191+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c00ee50 con 0x7fd970157910 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.191+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c01c600 con 0x7fd970157910 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.191+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97015d5b0 con 0x7fd970157910 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.191+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97015d2a0 con 0x7fd970157910 2024-03-06T07:33:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.192+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97004f330 con 0x7fd970157910 2024-03-06T07:33:48.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.193+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd95c01c7a0 con 0x7fd970157910 2024-03-06T07:33:48.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.194+0000 7fd96cc00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd95006b500 0x7fd95006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:48.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.194+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd95c08a920 con 0x7fd970157910 2024-03-06T07:33:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.195+0000 7fd974c00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd95006b500 0x7fd95006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.196+0000 7fd974c00640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd95006b500 0x7fd95006d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd964033000 tx=0x7fd964009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:48.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.200+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd95c018030 con 0x7fd970157910 2024-03-06T07:33:48.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.439+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd970159160 con 0x7fd970157910 2024-03-06T07:33:48.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.972+0000 7fd96cc00640 1 -- 172.21.15.6:0/2631662482 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v74) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd95c0591a0 con 0x7fd970157910 2024-03-06T07:33:48.975 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:33:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.978+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd95006b500 msgr2=0x7fd95006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.978+0000 7fd977600640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd95006b500 0x7fd95006d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd964033000 tx=0x7fd964009030 comp rx=0 tx=0).stop 2024-03-06T07:33:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.978+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 msgr2=0x7fd9701563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:48.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.978+0000 7fd977600640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 0x7fd9701563f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd95c010c00 tx=0x7fd95c00e540 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.981+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 shutdown_connections 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.981+0000 7fd977600640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd95006b500 0x7fd95006d9c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.981+0000 7fd977600640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd97015ed10 0x7fd9701e66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.981+0000 7fd977600640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd970157910 0x7fd9701563f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.982+0000 7fd977600640 1 --2- 172.21.15.6:0/2631662482 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd958003a00 0x7fd97015e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:48.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.982+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 >> 172.21.15.6:0/2631662482 conn(0x7fd97006b3d0 msgr2=0x7fd9700ae590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.986+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 shutdown_connections 2024-03-06T07:33:48.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:48.986+0000 7fd977600640 1 -- 172.21.15.6:0/2631662482 wait complete. 2024-03-06T07:33:49.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:49.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.212+0000 7f27dfe00640 1 Processor -- start 2024-03-06T07:33:49.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.212+0000 7f27dfe00640 1 -- start start 2024-03-06T07:33:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.213+0000 7f27dfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f27e015bc10 0x7f27e015ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.213+0000 7f27dfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e015f3b0 con 0x7f27e0071990 2024-03-06T07:33:49.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.213+0000 7f27dfe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e006da90 con 0x7f27e00b3ed0 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.213+0000 7f27dfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e006dd70 con 0x7f27e015bc10 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27df400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f27e0071990 0x7f27e0071d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37918/0 (socket says 172.21.15.6:37918) 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27df400640 1 -- 172.21.15.6:0/841759616 learned_addr learned my addr 172.21.15.6:0/841759616 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27dea00640 1 --2- 172.21.15.6:0/841759616 >> v2:172.21.15.6:3300/0 conn(0x7f27e015bc10 0x7f27e015ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27dea00640 1 -- 172.21.15.6:0/841759616 >> v1:172.21.15.178:6789/0 conn(0x7f27e00b3ed0 legacy=0x7f27e00b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27dea00640 1 -- 172.21.15.6:0/841759616 >> v1:172.21.15.6:6789/0 conn(0x7f27e0071990 legacy=0x7f27e0071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27dea00640 1 -- 172.21.15.6:0/841759616 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e0070eb0 con 0x7f27e015bc10 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.214+0000 7f27dd600640 1 -- 172.21.15.6:0/841759616 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314422553 0 0) 0x7f27d8001110 con 0x7f27e0071990 2024-03-06T07:33:49.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.216+0000 7f27dea00640 1 --2- 172.21.15.6:0/841759616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e015ee70 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f27c80097a0 tx=0x7f27c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b0e53db272c2fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.216+0000 7f27dd600640 1 -- 172.21.15.6:0/841759616 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8035020 con 0x7f27e015bc10 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.216+0000 7f27dd600640 1 -- 172.21.15.6:0/841759616 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c8031830 con 0x7f27e015bc10 2024-03-06T07:33:49.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.216+0000 7f27dd600640 1 -- 172.21.15.6:0/841759616 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8031c00 con 0x7f27e015bc10 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.216+0000 7f27dfe00640 1 -- 172.21.15.6:0/841759616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 msgr2=0x7f27e015ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.216+0000 7f27dfe00640 1 --2- 172.21.15.6:0/841759616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e015ee70 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f27c80097a0 tx=0x7f27c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.217+0000 7f27dfe00640 1 -- 172.21.15.6:0/841759616 shutdown_connections 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.217+0000 7f27dfe00640 1 --2- 172.21.15.6:0/841759616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e015ee70 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.217+0000 7f27dfe00640 1 -- 172.21.15.6:0/841759616 >> 172.21.15.6:0/841759616 conn(0x7f27e006cc70 msgr2=0x7f27e006d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.217+0000 7f27dfe00640 1 -- 172.21.15.6:0/841759616 shutdown_connections 2024-03-06T07:33:49.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.217+0000 7f27dfe00640 1 -- 172.21.15.6:0/841759616 wait complete. 2024-03-06T07:33:49.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.219+0000 7f27dfe00640 1 Processor -- start 2024-03-06T07:33:49.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.219+0000 7f27dfe00640 1 -- start start 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dfe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f27e0071990 0x7f27e0154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dfe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e00b3ed0 0x7f27e01571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e0158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e006d760 con 0x7f27e015bc10 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dfe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e0071470 con 0x7f27e00b3ed0 2024-03-06T07:33:49.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dfe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00b4a50 con 0x7f27e0071990 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e0158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e0158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41618/0 (socket says 172.21.15.6:41618) 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 -- 172.21.15.6:0/407892700 learned_addr learned my addr 172.21.15.6:0/407892700 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27dea00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f27e0071990 0x7f27e0154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27de000640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e00b3ed0 0x7f27e01571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 -- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f27e0071990 msgr2=0x7f27e0154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f27e0071990 0x7f27e0154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 -- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e00b3ed0 msgr2=0x7f27e01571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.220+0000 7f27df400640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e00b3ed0 0x7f27e01571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27df400640 1 -- 172.21.15.6:0/407892700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e00c4e70 con 0x7f27e015bc10 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27de000640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e00b3ed0 0x7f27e01571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27df400640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e0158a90 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f27d800c780 tx=0x7f27d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8018070 con 0x7f27e015bc10 2024-03-06T07:33:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d80070d0 con 0x7f27e015bc10 2024-03-06T07:33:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d8007500 con 0x7f27e015bc10 2024-03-06T07:33:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e01549e0 con 0x7f27e015bc10 2024-03-06T07:33:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.221+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e0154fa0 con 0x7f27e015bc10 2024-03-06T07:33:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.222+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e004f330 con 0x7f27e015bc10 2024-03-06T07:33:49.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.223+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27d80047e0 con 0x7f27e015bc10 2024-03-06T07:33:49.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.224+0000 7f27d6a00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27bc06b4d0 0x7f27bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.224+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27d80894b0 con 0x7f27e015bc10 2024-03-06T07:33:49.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.225+0000 7f27dea00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27bc06b4d0 0x7f27bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.226+0000 7f27dea00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27bc06b4d0 0x7f27bc06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f27c8036000 tx=0x7f27c803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.229+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27e00c4e70 con 0x7f27e015bc10 2024-03-06T07:33:49.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.431+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27e00af490 con 0x7f27e015bc10 2024-03-06T07:33:49.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.432+0000 7f27d6a00640 1 -- 172.21.15.6:0/407892700 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f27e015f3b0 con 0x7f27e015bc10 2024-03-06T07:33:49.435 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:49.435 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":107,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:48.441347+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27bc06b4d0 msgr2=0x7f27bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27bc06b4d0 0x7f27bc06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f27c8036000 tx=0x7f27c803c040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 msgr2=0x7f27e0158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e0158a90 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f27d800c780 tx=0x7f27d800cc40 comp rx=0 tx=0).stop 2024-03-06T07:33:49.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 shutdown_connections 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27bc06b4d0 0x7f27bc06d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27e00b3ed0 0x7f27e01571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f27e015bc10 0x7f27e0158a90 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 --2- 172.21.15.6:0/407892700 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f27e0071990 0x7f27e0154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 >> 172.21.15.6:0/407892700 conn(0x7f27e006cc70 msgr2=0x7f27e01539b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 shutdown_connections 2024-03-06T07:33:49.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.435+0000 7f27dfe00640 1 -- 172.21.15.6:0/407892700 wait complete. 2024-03-06T07:33:49.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.663+0000 7f0606c00640 1 Processor -- start 2024-03-06T07:33:49.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.663+0000 7f0606c00640 1 -- start start 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f0606c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f06000af1a0 0x7f060015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f0606c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 0x7f06000b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f0606c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060015cc40 con 0x7f06000af580 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f0606c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060015cf20 con 0x7f06000af1a0 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f0606c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060015d200 con 0x7f06000cdc20 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f05f7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f06000af580 0x7f06000b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37930/0 (socket says 172.21.15.6:37930) 2024-03-06T07:33:49.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f05f7e00640 1 -- 172.21.15.6:0/1533952806 learned_addr learned my addr 172.21.15.6:0/1533952806 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f0604c00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 0x7f06000b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f05ffe00640 1 --2- 172.21.15.6:0/1533952806 >> v2:172.21.15.178:3300/0 conn(0x7f06000af1a0 0x7f060015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.665+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060165377 0 0) 0x7f060015cc40 con 0x7f06000af580 2024-03-06T07:33:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e4003520 con 0x7f06000af580 2024-03-06T07:33:49.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643997935 0 0) 0x7f05e4003520 con 0x7f06000af580 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 msgr2=0x7f06000b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 0x7f06000b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 >> v2:172.21.15.178:3300/0 conn(0x7f06000af1a0 msgr2=0x7f060015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 --2- 172.21.15.6:0/1533952806 >> v2:172.21.15.178:3300/0 conn(0x7f06000af1a0 0x7f060015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060015d4e0 con 0x7f06000af580 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f05e8001670 con 0x7f06000af580 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 >> v1:172.21.15.6:6789/0 conn(0x7f06000af580 legacy=0x7f06000b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 0x7f05e4003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05e4003fa0 0x7f05e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f05e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060015cf20 con 0x7f05e4004d40 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060015d200 con 0x7f05e4003fa0 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.666+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05e400ba70 con 0x7f06000cdc20 2024-03-06T07:33:49.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f0604c00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f05e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f05ffe00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 0x7f05e4003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f0604c00640 1 -- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 msgr2=0x7f05e4003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f05f7e00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05e4003fa0 0x7f05e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f0604c00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06000cdc20 0x7f05e4003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f0604c00640 1 -- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05e4003fa0 msgr2=0x7f05e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f0604c00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05e4003fa0 0x7f05e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f0604c00640 1 -- 172.21.15.6:0/1533952806 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4003520 con 0x7f05e4004d40 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.667+0000 7f05f7e00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05e4003fa0 0x7f05e4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.668+0000 7f0604c00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f05e40051c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009fc0 tx=0x7f05e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ae2f7e52a8d280 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.668+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8039070 con 0x7f05e4004d40 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.668+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f802cc20 con 0x7f05e4004d40 2024-03-06T07:33:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.668+0000 7f05ff400640 1 -- 172.21.15.6:0/1533952806 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8009c10 con 0x7f05e4004d40 2024-03-06T07:33:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0606c00640 1 -- 172.21.15.6:0/1533952806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 msgr2=0x7f05e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0606c00640 1 --2- 172.21.15.6:0/1533952806 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f05e40051c0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009fc0 tx=0x7f05e4005830 comp rx=0 tx=0).stop 2024-03-06T07:33:49.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0604c00640 1 -- 172.21.15.6:0/1533952806 reap_dead start 2024-03-06T07:33:49.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0606c00640 1 -- 172.21.15.6:0/1533952806 shutdown_connections 2024-03-06T07:33:49.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0606c00640 1 -- 172.21.15.6:0/1533952806 >> 172.21.15.6:0/1533952806 conn(0x7f060006afb0 msgr2=0x7f060006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0606c00640 1 -- 172.21.15.6:0/1533952806 shutdown_connections 2024-03-06T07:33:49.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.669+0000 7f0606c00640 1 -- 172.21.15.6:0/1533952806 wait complete. 2024-03-06T07:33:49.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.671+0000 7f0606c00640 1 Processor -- start 2024-03-06T07:33:49.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.671+0000 7f0606c00640 1 -- start start 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f0606c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f060009cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f0606c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06001f26d0 0x7f06001f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f0606c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06001f4670 0x7f06000ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f0606c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b3160 con 0x7f05e4004d40 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f0606c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b2fe0 con 0x7f06001f4670 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f0606c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b32e0 con 0x7f06001f26d0 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f05ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f060009cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f05ffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f060009cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41630/0 (socket says 172.21.15.6:41630) 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.672+0000 7f05ffe00640 1 -- 172.21.15.6:0/1355343055 learned_addr learned my addr 172.21.15.6:0/1355343055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:49.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05ffe00640 1 -- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06001f26d0 msgr2=0x7f06001f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05ffe00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06001f26d0 0x7f06001f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05ffe00640 1 -- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06001f4670 msgr2=0x7f06000ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05ffe00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06001f4670 0x7f06000ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05ffe00640 1 -- 172.21.15.6:0/1355343055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060009d1c0 con 0x7f05e4004d40 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05ffe00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f060009cc80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f05f00116a0 tx=0x7f05f0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000f940 con 0x7f05e4004d40 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f000fae0 con 0x7f05e4004d40 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000fe90 con 0x7f05e4004d40 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.673+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06001f3090 con 0x7f05e4004d40 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.674+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06001f3650 con 0x7f05e4004d40 2024-03-06T07:33:49.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.675+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0600069510 con 0x7f05e4004d40 2024-03-06T07:33:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.677+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05f0015460 con 0x7f05e4004d40 2024-03-06T07:33:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.678+0000 7f05fcc00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f05dc06b670 0x7f05dc06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.678+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05f008abe0 con 0x7f05e4004d40 2024-03-06T07:33:49.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.678+0000 7f05f7e00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f05dc06b670 0x7f05dc06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:49.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.680+0000 7f05f7e00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f05dc06b670 0x7f05dc06db30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f05e80017e0 tx=0x7f05e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:49.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.683+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05f0019030 con 0x7f05e4004d40 2024-03-06T07:33:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.958+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06000ccd30 con 0x7f05e4004d40 2024-03-06T07:33:49.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.961+0000 7f05fcc00640 1 -- 172.21.15.6:0/1355343055 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f05f0059a50 con 0x7f05e4004d40 2024-03-06T07:33:49.963 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:49.963 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":107,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:48.441347+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f05dc06b670 msgr2=0x7f05dc06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f05dc06b670 0x7f05dc06db30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f05e80017e0 tx=0x7f05e8009040 comp rx=0 tx=0).stop 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 msgr2=0x7f060009cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f060009cc80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f05f00116a0 tx=0x7f05f0011b60 comp rx=0 tx=0).stop 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 shutdown_connections 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f05dc06b670 0x7f05dc06db30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06001f4670 0x7f06000ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f06001f26d0 0x7f06001f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 --2- 172.21.15.6:0/1355343055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05e4004d40 0x7f060009cc80 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:49.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.963+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 >> 172.21.15.6:0/1355343055 conn(0x7f060006afb0 msgr2=0x7f06000b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:49.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.964+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 shutdown_connections 2024-03-06T07:33:49.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:49.964+0000 7f0606c00640 1 -- 172.21.15.6:0/1355343055 wait complete. 2024-03-06T07:33:49.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:33:50.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.187+0000 7ff7cb200640 1 Processor -- start 2024-03-06T07:33:50.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.187+0000 7ff7cb200640 1 -- start start 2024-03-06T07:33:50.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7cb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7cc0d4860 0x7ff7cc0d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7cb200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff7cc0cfa60 0x7ff7cc0d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7cb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7cc0ceca0 con 0x7ff7cc15dee0 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7cb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc0cf5d0 con 0x7ff7cc0d4860 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7cb200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7cc0cef80 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7c9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff7cc0d4860 0x7ff7cc0d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7c9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff7cc15dee0 0x7ff7cc0d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37934/0 (socket says 172.21.15.6:37934) 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.189+0000 7ff7c9e00640 1 -- 172.21.15.6:0/2741237867 learned_addr learned my addr 172.21.15.6:0/2741237867 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.190+0000 7ff7ca800640 1 --2- 172.21.15.6:0/2741237867 >> v2:172.21.15.178:3300/0 conn(0x7ff7cc0cfa60 0x7ff7cc0d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.190+0000 7ff7c9400640 1 -- 172.21.15.6:0/2741237867 >> v2:172.21.15.178:3300/0 conn(0x7ff7cc0cfa60 msgr2=0x7ff7cc0d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:50.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.190+0000 7ff7c9400640 1 --2- 172.21.15.6:0/2741237867 >> v2:172.21.15.178:3300/0 conn(0x7ff7cc0cfa60 0x7ff7cc0d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.190+0000 7ff7c9400640 1 -- 172.21.15.6:0/2741237867 >> v1:172.21.15.6:6789/0 conn(0x7ff7cc15dee0 legacy=0x7ff7cc0d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.190+0000 7ff7c9400640 1 -- 172.21.15.6:0/2741237867 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc0cf260 con 0x7ff7cc0d4860 2024-03-06T07:33:50.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.192+0000 7ff7c9400640 1 --2- 172.21.15.6:0/2741237867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc0d0630 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc06b2d0 tx=0x7ff7b402bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d18cd1eba239e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.192+0000 7ff7c8a00640 1 -- 172.21.15.6:0/2741237867 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4039070 con 0x7ff7cc0d4860 2024-03-06T07:33:50.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7c8a00640 1 -- 172.21.15.6:0/2741237867 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b402ca30 con 0x7ff7cc0d4860 2024-03-06T07:33:50.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7c8a00640 1 -- 172.21.15.6:0/2741237867 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b402ce00 con 0x7ff7cc0d4860 2024-03-06T07:33:50.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 -- 172.21.15.6:0/2741237867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0d4860 msgr2=0x7ff7cc0d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 --2- 172.21.15.6:0/2741237867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc0d0630 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc06b2d0 tx=0x7ff7b402bd00 comp rx=0 tx=0).stop 2024-03-06T07:33:50.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 -- 172.21.15.6:0/2741237867 shutdown_connections 2024-03-06T07:33:50.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 --2- 172.21.15.6:0/2741237867 >> v2:172.21.15.178:3300/0 conn(0x7ff7cc0cfa60 0x7ff7cc0d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 --2- 172.21.15.6:0/2741237867 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc0d0630 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 -- 172.21.15.6:0/2741237867 >> 172.21.15.6:0/2741237867 conn(0x7ff7cc0d3cd0 msgr2=0x7ff7cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:50.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 -- 172.21.15.6:0/2741237867 shutdown_connections 2024-03-06T07:33:50.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.193+0000 7ff7cb200640 1 -- 172.21.15.6:0/2741237867 wait complete. 2024-03-06T07:33:50.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.195+0000 7ff7cb200640 1 Processor -- start 2024-03-06T07:33:50.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.195+0000 7ff7cb200640 1 -- start start 2024-03-06T07:33:50.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.196+0000 7ff7cb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 0x7ff7cc0cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.196+0000 7ff7cb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc15c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.196+0000 7ff7cb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7cc15dee0 0x7ff7cc15c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.196+0000 7ff7cb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0b3510 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.196+0000 7ff7cb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc15e2c0 con 0x7ff7cc0d4860 2024-03-06T07:33:50.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.196+0000 7ff7cb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc0b3690 con 0x7ff7cc15dee0 2024-03-06T07:33:50.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 0x7ff7cc0cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 0x7ff7cc0cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41642/0 (socket says 172.21.15.6:41642) 2024-03-06T07:33:50.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 -- 172.21.15.6:0/373938560 learned_addr learned my addr 172.21.15.6:0/373938560 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9400640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7ca800640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7cc15dee0 0x7ff7cc15c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 -- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7cc15dee0 msgr2=0x7ff7cc15c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7cc15dee0 0x7ff7cc15c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 -- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7cc0d4860 msgr2=0x7ff7cc15c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc15c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 -- 172.21.15.6:0/373938560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc0b8670 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.197+0000 7ff7c9e00640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 0x7ff7cc0cb700 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c7b0 tx=0x7ff7bc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.198+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc018070 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.198+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc0072d0 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.198+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc0b8950 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.198+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0047e0 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.198+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc0b8f10 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.199+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc04f330 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.200+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7bc01b070 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.202+0000 7ff7c2000640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a806b4d0 0x7ff7a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.202+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7bc089640 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.202+0000 7ff7c9400640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a806b4d0 0x7ff7a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:50.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.203+0000 7ff7c9400640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a806b4d0 0x7ff7a806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4036000 tx=0x7ff7b4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:50.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.206+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7bc058570 con 0x7ff7cc0cfa60 2024-03-06T07:33:50.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:50.619+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff7cc08d7d0 con 0x7ff7cc0cfa60 2024-03-06T07:33:51.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.266+0000 7ff7c2000640 1 -- 172.21.15.6:0/373938560 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff7bc057e70 con 0x7ff7cc0cfa60 2024-03-06T07:33:51.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a806b4d0 msgr2=0x7ff7a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a806b4d0 0x7ff7a806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4036000 tx=0x7ff7b4030000 comp rx=0 tx=0).stop 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 msgr2=0x7ff7cc0cb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 0x7ff7cc0cb700 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00c7b0 tx=0x7ff7bc00cc70 comp rx=0 tx=0).stop 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 shutdown_connections 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff7a806b4d0 0x7ff7a806d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff7cc0d4860 0x7ff7cc15c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff7cc15dee0 0x7ff7cc15c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 --2- 172.21.15.6:0/373938560 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff7cc0cfa60 0x7ff7cc0cb700 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 >> 172.21.15.6:0/373938560 conn(0x7ff7cc0d3cd0 msgr2=0x7ff7cc06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.271+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 shutdown_connections 2024-03-06T07:33:51.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.272+0000 7ff7cb200640 1 -- 172.21.15.6:0/373938560 wait complete. 2024-03-06T07:33:51.294 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:33:51.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:33:51.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.505+0000 7f1ac5800640 1 Processor -- start 2024-03-06T07:33:51.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.505+0000 7f1ac5800640 1 -- start start 2024-03-06T07:33:51.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1ac5800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1ac00d4860 0x7f1ac00d0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1ac5800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ac00cfa60 0x7f1ac00d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1ac5800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00ceca0 con 0x7f1ac015dee0 2024-03-06T07:33:51.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1ac5800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac00cf5d0 con 0x7f1ac00cfa60 2024-03-06T07:33:51.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1ac5800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac00cef80 con 0x7f1ac00d4860 2024-03-06T07:33:51.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1abfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1ac015dee0 0x7f1ac00d18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37946/0 (socket says 172.21.15.6:37946) 2024-03-06T07:33:51.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.507+0000 7f1abfe00640 1 -- 172.21.15.6:0/2099652206 learned_addr learned my addr 172.21.15.6:0/2099652206 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:51.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1ac4e00640 1 --2- 172.21.15.6:0/2099652206 >> v2:172.21.15.6:3300/0 conn(0x7f1ac00cfa60 0x7f1ac00d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1abf400640 1 --2- 172.21.15.6:0/2099652206 >> v2:172.21.15.178:3300/0 conn(0x7f1ac00d4860 0x7f1ac00d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1abea00640 1 -- 172.21.15.6:0/2099652206 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282764356 0 0) 0x7f1ac00ceca0 con 0x7f1ac015dee0 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1abea00640 1 -- 172.21.15.6:0/2099652206 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa4003520 con 0x7f1ac015dee0 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1ac4e00640 1 -- 172.21.15.6:0/2099652206 >> v2:172.21.15.178:3300/0 conn(0x7f1ac00d4860 msgr2=0x7f1ac00d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1ac4e00640 1 --2- 172.21.15.6:0/2099652206 >> v2:172.21.15.178:3300/0 conn(0x7f1ac00d4860 0x7f1ac00d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1ac4e00640 1 -- 172.21.15.6:0/2099652206 >> v1:172.21.15.6:6789/0 conn(0x7f1ac015dee0 legacy=0x7f1ac00d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.508+0000 7f1ac4e00640 1 -- 172.21.15.6:0/2099652206 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00cf260 con 0x7f1ac00cfa60 2024-03-06T07:33:51.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.509+0000 7f1ac4e00640 1 --2- 172.21.15.6:0/2099652206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00cfa60 0x7f1ac00d0b70 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0162b70 tx=0x7f1ab802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baca29f7f1ecd67a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1abea00640 1 -- 172.21.15.6:0/2099652206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8039070 con 0x7f1ac00cfa60 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1abea00640 1 -- 172.21.15.6:0/2099652206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab802cc70 con 0x7f1ac00cfa60 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1abea00640 1 -- 172.21.15.6:0/2099652206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8034640 con 0x7f1ac00cfa60 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1ac5800640 1 -- 172.21.15.6:0/2099652206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00cfa60 msgr2=0x7f1ac00d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2099652206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00cfa60 0x7f1ac00d0b70 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0162b70 tx=0x7f1ab802bec0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1ac5800640 1 -- 172.21.15.6:0/2099652206 shutdown_connections 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2099652206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00cfa60 0x7f1ac00d0b70 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2099652206 >> v2:172.21.15.178:3300/0 conn(0x7f1ac00d4860 0x7f1ac00d0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.510+0000 7f1ac5800640 1 -- 172.21.15.6:0/2099652206 >> 172.21.15.6:0/2099652206 conn(0x7f1ac00d3cd0 msgr2=0x7f1ac00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.511+0000 7f1ac5800640 1 -- 172.21.15.6:0/2099652206 shutdown_connections 2024-03-06T07:33:51.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.511+0000 7f1ac5800640 1 -- 172.21.15.6:0/2099652206 wait complete. 2024-03-06T07:33:51.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.512+0000 7f1ac5800640 1 Processor -- start 2024-03-06T07:33:51.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.512+0000 7f1ac5800640 1 -- start start 2024-03-06T07:33:51.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1ac5800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ac00cfa60 0x7f1ac01521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1ac5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 0x7f1ac015e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1ac5800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ac015dee0 0x7f1ac015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1ac5800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b3510 con 0x7f1ac00d4860 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1ac5800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac015e2c0 con 0x7f1ac015dee0 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1ac5800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00b3690 con 0x7f1ac00cfa60 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1abf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 0x7f1ac015e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1abf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 0x7f1ac015e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41654/0 (socket says 172.21.15.6:41654) 2024-03-06T07:33:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.513+0000 7f1abf400640 1 -- 172.21.15.6:0/2889066259 learned_addr learned my addr 172.21.15.6:0/2889066259 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1abf400640 1 -- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ac00cfa60 msgr2=0x7f1ac01521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1abf400640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ac00cfa60 0x7f1ac01521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1abf400640 1 -- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ac015dee0 msgr2=0x7f1ac015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1ac4e00640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ac015dee0 0x7f1ac015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1abf400640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ac015dee0 0x7f1ac015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1abf400640 1 -- 172.21.15.6:0/2889066259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac0152700 con 0x7f1ac00d4860 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1ac4e00640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ac015dee0 0x7f1ac015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:51.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.514+0000 7f1abf400640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 0x7f1ac015e440 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00e9b0 tx=0x7f1aac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.515+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac017070 con 0x7f1ac00d4860 2024-03-06T07:33:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.515+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00c9a0 con 0x7f1ac00d4860 2024-03-06T07:33:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.515+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00cd90 con 0x7f1ac00d4860 2024-03-06T07:33:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.515+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac01f3f40 con 0x7f1ac00d4860 2024-03-06T07:33:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.515+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01f4500 con 0x7f1ac00d4860 2024-03-06T07:33:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.516+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac00b3330 con 0x7f1ac00d4860 2024-03-06T07:33:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.517+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aac01a070 con 0x7f1ac00d4860 2024-03-06T07:33:51.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.518+0000 7f1ab7e00640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1aa006b4d0 0x7f1aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:51.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.518+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1aac08a680 con 0x7f1ac00d4860 2024-03-06T07:33:51.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.519+0000 7f1abfe00640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1aa006b4d0 0x7f1aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.520+0000 7f1abfe00640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1aa006b4d0 0x7f1aa006d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0000f60 tx=0x7f1ab00078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.522+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ac00b3330 con 0x7f1ac00d4860 2024-03-06T07:33:51.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:51.762+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1ac008d7d0 con 0x7f1ac00d4860 2024-03-06T07:33:52.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.276+0000 7f1ab7e00640 1 -- 172.21.15.6:0/2889066259 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1aac059460 con 0x7f1ac00d4860 2024-03-06T07:33:52.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1aa006b4d0 msgr2=0x7f1aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1aa006b4d0 0x7f1aa006d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0000f60 tx=0x7f1ab00078f0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 msgr2=0x7f1ac015e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 0x7f1ac015e440 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f1aac00e9b0 tx=0x7f1aac00ee70 comp rx=0 tx=0).stop 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 shutdown_connections 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1aa006b4d0 0x7f1aa006d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ac015dee0 0x7f1ac015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ac00d4860 0x7f1ac015e440 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 --2- 172.21.15.6:0/2889066259 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ac00cfa60 0x7f1ac01521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 >> 172.21.15.6:0/2889066259 conn(0x7f1ac00d3cd0 msgr2=0x7f1ac00b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 shutdown_connections 2024-03-06T07:33:52.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.280+0000 7f1ac5800640 1 -- 172.21.15.6:0/2889066259 wait complete. 2024-03-06T07:33:52.303 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:33:52.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:52.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.512+0000 7fb703e00640 1 Processor -- start 2024-03-06T07:33:52.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.512+0000 7fb703e00640 1 -- start start 2024-03-06T07:33:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.514+0000 7fb703e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb70415e0b0 0x7fb7040cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.514+0000 7fb703e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 0x7fb7040b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.514+0000 7fb703e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7040b3710 con 0x7fb7040d4a10 2024-03-06T07:33:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.514+0000 7fb703e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7040b09e0 con 0x7fb70415e0b0 2024-03-06T07:33:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.514+0000 7fb703e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040b39f0 con 0x7fb7040d0e50 2024-03-06T07:33:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.515+0000 7fb702a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb70415e0b0 0x7fb7040cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.515+0000 7fb703400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 0x7fb7040b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.515+0000 7fb703400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 0x7fb7040b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47364/0 (socket says 172.21.15.6:47364) 2024-03-06T07:33:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.515+0000 7fb703400640 1 -- 172.21.15.6:0/3947184255 learned_addr learned my addr 172.21.15.6:0/3947184255 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.516+0000 7fb702a00640 1 -- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 msgr2=0x7fb7040b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.516+0000 7fb702a00640 1 --2- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 0x7fb7040b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.516+0000 7fb702a00640 1 -- 172.21.15.6:0/3947184255 >> v1:172.21.15.178:6789/0 conn(0x7fb7040d4a10 legacy=0x7fb7040d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.516+0000 7fb702a00640 1 -- 172.21.15.6:0/3947184255 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7040b3cd0 con 0x7fb70415e0b0 2024-03-06T07:33:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.516+0000 7fb703400640 1 --2- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 0x7fb7040b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:33:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.517+0000 7fb702a00640 1 --2- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415e0b0 0x7fb7040cf310 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40097a0 tx=0x7fb6f402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b0825fe5c1e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.518+0000 7fb702000640 1 -- 172.21.15.6:0/3947184255 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4037070 con 0x7fb70415e0b0 2024-03-06T07:33:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.518+0000 7fb702000640 1 -- 172.21.15.6:0/3947184255 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f4030710 con 0x7fb70415e0b0 2024-03-06T07:33:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.518+0000 7fb702000640 1 -- 172.21.15.6:0/3947184255 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4030ae0 con 0x7fb70415e0b0 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.518+0000 7fb703e00640 1 -- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415e0b0 msgr2=0x7fb7040cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.518+0000 7fb703e00640 1 --2- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415e0b0 0x7fb7040cf310 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40097a0 tx=0x7fb6f402b710 comp rx=0 tx=0).stop 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.519+0000 7fb703e00640 1 -- 172.21.15.6:0/3947184255 shutdown_connections 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.519+0000 7fb703e00640 1 --2- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d0e50 0x7fb7040b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.519+0000 7fb703e00640 1 --2- 172.21.15.6:0/3947184255 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415e0b0 0x7fb7040cf310 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.519+0000 7fb703e00640 1 -- 172.21.15.6:0/3947184255 >> 172.21.15.6:0/3947184255 conn(0x7fb7040d3d30 msgr2=0x7fb7040d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.519+0000 7fb703e00640 1 -- 172.21.15.6:0/3947184255 shutdown_connections 2024-03-06T07:33:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.519+0000 7fb703e00640 1 -- 172.21.15.6:0/3947184255 wait complete. 2024-03-06T07:33:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.521+0000 7fb703e00640 1 Processor -- start 2024-03-06T07:33:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.521+0000 7fb703e00640 1 -- start start 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb703e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 0x7fb70414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb703e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d4a10 0x7fb7040b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb703e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415f930 0x7fb70415fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb703e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040cfc60 con 0x7fb7040d0e50 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb703e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70415e490 con 0x7fb70415f930 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb703e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040cf850 con 0x7fb7040d4a10 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb702a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 0x7fb70414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb702a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 0x7fb70414aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49438/0 (socket says 172.21.15.6:49438) 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.522+0000 7fb702a00640 1 -- 172.21.15.6:0/3640357014 learned_addr learned my addr 172.21.15.6:0/3640357014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d4a10 0x7fb7040b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb702a00640 1 -- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d4a10 msgr2=0x7fb7040b68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb703400640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415f930 0x7fb70415fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb702a00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d4a10 0x7fb7040b68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb702a00640 1 -- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415f930 msgr2=0x7fb70415fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb702a00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415f930 0x7fb70415fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb702a00640 1 -- 172.21.15.6:0/3640357014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70414b960 con 0x7fb7040d0e50 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d4a10 0x7fb7040b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb702a00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 0x7fb70414aff0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40098c0 tx=0x7fb6f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4037040 con 0x7fb7040d0e50 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.523+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f4042070 con 0x7fb7040d0e50 2024-03-06T07:33:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.524+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70414bc40 con 0x7fb7040d0e50 2024-03-06T07:33:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.524+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f403ba30 con 0x7fb7040d0e50 2024-03-06T07:33:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.524+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70414c200 con 0x7fb7040d0e50 2024-03-06T07:33:52.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.525+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70404f330 con 0x7fb7040d0e50 2024-03-06T07:33:52.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.526+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb6f403a070 con 0x7fb7040d0e50 2024-03-06T07:33:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.527+0000 7fb6faa00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6e006b4d0 0x7fb6e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:52.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.528+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6f40ab680 con 0x7fb7040d0e50 2024-03-06T07:33:52.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.528+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6e006b4d0 0x7fb6e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:52.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.529+0000 7fb6fbe00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6e006b4d0 0x7fb6e006d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb70406d180 tx=0x7fb6ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:52.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.540+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb6f407a480 con 0x7fb7040d0e50 2024-03-06T07:33:52.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.783+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb70414b530 con 0x7fb7040d0e50 2024-03-06T07:33:52.786 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.784+0000 7fb6faa00640 1 -- 172.21.15.6:0/3640357014 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fb6f4079d80 con 0x7fb7040d0e50 2024-03-06T07:33:52.786 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:52.786 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5795,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3411419645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3411419645},{"type":"v1","addr":"172.21.15.6:6835","nonce":3411419645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5798,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/525628181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":525628181},{"type":"v1","addr":"172.21.15.178:6835","nonce":525628181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/319127923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":319127923},{"type":"v1","addr":"172.21.15.178:6837","nonce":319127923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:33:51.764337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7656},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7656":{"gid":7656,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6833/4114489444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4114489444},{"type":"v1","addr":"172.21.15.6:6833","nonce":4114489444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7656,"qdb_cluster":[7656]},"id":8}]} 2024-03-06T07:33:52.786 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6e006b4d0 msgr2=0x7fb6e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6e006b4d0 0x7fb6e006d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fb70406d180 tx=0x7fb6ec009040 comp rx=0 tx=0).stop 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 msgr2=0x7fb70414aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 0x7fb70414aff0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40098c0 tx=0x7fb6f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 shutdown_connections 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb6e006b4d0 0x7fb6e006d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb70415f930 0x7fb70415fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7040d4a10 0x7fb7040b68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 --2- 172.21.15.6:0/3640357014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7040d0e50 0x7fb70414aff0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:52.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 >> 172.21.15.6:0/3640357014 conn(0x7fb7040d3d30 msgr2=0x7fb7040b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:52.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 shutdown_connections 2024-03-06T07:33:52.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:52.787+0000 7fb703e00640 1 -- 172.21.15.6:0/3640357014 wait complete. 2024-03-06T07:33:52.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:53.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.015+0000 7f37d8a00640 1 Processor -- start 2024-03-06T07:33:53.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.016+0000 7f37d8a00640 1 -- start start 2024-03-06T07:33:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.017+0000 7f37d8a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f37d40d4a10 0x7f37d40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.017+0000 7f37d8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37d40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.017+0000 7f37d8a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d40b3710 con 0x7f37d415e0b0 2024-03-06T07:33:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.017+0000 7f37d8a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d40b09e0 con 0x7f37d40d4a10 2024-03-06T07:33:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.017+0000 7f37d8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40cfa20 con 0x7f37d40d0c80 2024-03-06T07:33:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.018+0000 7f37d3400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37d415e0b0 0x7f37d40cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46894/0 (socket says 172.21.15.6:46894) 2024-03-06T07:33:53.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.018+0000 7f37d3400640 1 -- 172.21.15.6:0/3236997285 learned_addr learned my addr 172.21.15.6:0/3236997285 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:53.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.018+0000 7f37d3e00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37d40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.018+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480709965 0 0) 0x7f37d40b3710 con 0x7f37d415e0b0 2024-03-06T07:33:53.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.018+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b8003520 con 0x7f37d415e0b0 2024-03-06T07:33:53.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.018+0000 7f37d2a00640 1 --2- 172.21.15.6:0/3236997285 >> v2:172.21.15.178:3300/0 conn(0x7f37d40d4a10 0x7f37d40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677978049 0 0) 0x7f37b8003520 con 0x7f37d415e0b0 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 msgr2=0x7f37d40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37d40b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 >> v2:172.21.15.178:3300/0 conn(0x7f37d40d4a10 msgr2=0x7f37d40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 --2- 172.21.15.6:0/3236997285 >> v2:172.21.15.178:3300/0 conn(0x7f37d40d4a10 0x7f37d40d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40b39f0 con 0x7f37d415e0b0 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f37c40019f0 con 0x7f37d415e0b0 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2a00640 1 --2- 172.21.15.6:0/3236997285 >> v2:172.21.15.178:3300/0 conn(0x7f37d40d4a10 0x7f37d40d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 >> v1:172.21.15.6:6789/0 conn(0x7f37d415e0b0 legacy=0x7f37d40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b80039b0 0x7f37b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37b8004d40 0x7f37b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40cfa20 con 0x7f37b8004d40 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b800b9c0 con 0x7f37b80039b0 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b800bbc0 con 0x7f37d40d0c80 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.019+0000 7f37d3e00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37b8004d40 0x7f37b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3400640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b80039b0 0x7f37b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d2a00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3e00640 1 -- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 msgr2=0x7f37b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3e00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3e00640 1 -- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b80039b0 msgr2=0x7f37b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3e00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b80039b0 0x7f37b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3e00640 1 -- 172.21.15.6:0/3236997285 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b8003520 con 0x7f37b8004d40 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d3400640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b80039b0 0x7f37b8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.020+0000 7f37d2a00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40d0c80 0x7f37b8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.021+0000 7f37d3e00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37b8004d40 0x7f37b80051c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009ac0 tx=0x7f37b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276f671467f3f52d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.021+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc039070 con 0x7f37b8004d40 2024-03-06T07:33:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.021+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc02c6b0 con 0x7f37b8004d40 2024-03-06T07:33:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.021+0000 7f37d2000640 1 -- 172.21.15.6:0/3236997285 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc009690 con 0x7f37b8004d40 2024-03-06T07:33:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d8a00640 1 -- 172.21.15.6:0/3236997285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37b8004d40 msgr2=0x7f37b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d8a00640 1 --2- 172.21.15.6:0/3236997285 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37b8004d40 0x7f37b80051c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009ac0 tx=0x7f37b8005830 comp rx=0 tx=0).stop 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d3e00640 1 -- 172.21.15.6:0/3236997285 reap_dead start 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d8a00640 1 -- 172.21.15.6:0/3236997285 shutdown_connections 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d8a00640 1 -- 172.21.15.6:0/3236997285 >> 172.21.15.6:0/3236997285 conn(0x7f37d40d3d30 msgr2=0x7f37d40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d8a00640 1 -- 172.21.15.6:0/3236997285 shutdown_connections 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.022+0000 7f37d8a00640 1 -- 172.21.15.6:0/3236997285 wait complete. 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.024+0000 7f37d8a00640 1 Processor -- start 2024-03-06T07:33:53.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.024+0000 7f37d8a00640 1 -- start start 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d8a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b8004d40 0x7f37d415b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40cae00 0x7f37d40b77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 0x7f37d406c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d415e490 con 0x7f37d40b7d30 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d8a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40b3fb0 con 0x7f37b8004d40 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40cfc60 con 0x7f37d40cae00 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 0x7f37d406c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 0x7f37d406c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49456/0 (socket says 172.21.15.6:49456) 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.025+0000 7f37d3e00640 1 -- 172.21.15.6:0/2092954147 learned_addr learned my addr 172.21.15.6:0/2092954147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d2a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40cae00 0x7f37d40b77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3400640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b8004d40 0x7f37d415b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3e00640 1 -- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40cae00 msgr2=0x7f37d40b77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3e00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40cae00 0x7f37d40b77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3e00640 1 -- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b8004d40 msgr2=0x7f37d415b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3e00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b8004d40 0x7f37d415b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3e00640 1 -- 172.21.15.6:0/2092954147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d40cb850 con 0x7f37d40b7d30 2024-03-06T07:33:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3400640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b8004d40 0x7f37d415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d3e00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 0x7f37d406c010 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009be0 tx=0x7f37cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc044070 con 0x7f37d40b7d30 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc03a040 con 0x7f37d40b7d30 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc039070 con 0x7f37d40b7d30 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.026+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d40cbb30 con 0x7f37d40b7d30 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.027+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d415b7f0 con 0x7f37d40b7d30 2024-03-06T07:33:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.028+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d404f330 con 0x7f37d40b7d30 2024-03-06T07:33:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.029+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37cc047070 con 0x7f37d40b7d30 2024-03-06T07:33:53.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.030+0000 7f37cb400640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37b406b4d0 0x7f37b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.030+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37cc0aab80 con 0x7f37d40b7d30 2024-03-06T07:33:53.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.030+0000 7f37d3400640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37b406b4d0 0x7f37b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.031+0000 7f37d3400640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37b406b4d0 0x7f37b406d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f37c4001b60 tx=0x7f37c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.039+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37cc079440 con 0x7f37d40b7d30 2024-03-06T07:33:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.244+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37d40b53e0 con 0x7f37d40b7d30 2024-03-06T07:33:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.247+0000 7f37cb400640 1 -- 172.21.15.6:0/2092954147 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16555 (secure 0 0 0) 0x7f37cc07d4e0 con 0x7f37d40b7d30 2024-03-06T07:33:53.249 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:53.249 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":108,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:49.990380+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:33:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37b406b4d0 msgr2=0x7f37b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37b406b4d0 0x7f37b406d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f37c4001b60 tx=0x7f37c4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 msgr2=0x7f37d406c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 0x7f37d406c010 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009be0 tx=0x7f37cc004730 comp rx=0 tx=0).stop 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 shutdown_connections 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37b406b4d0 0x7f37b406d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37d40b7d30 0x7f37d406c010 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37d40cae00 0x7f37d40b77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 --2- 172.21.15.6:0/2092954147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37b8004d40 0x7f37d415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 >> 172.21.15.6:0/2092954147 conn(0x7f37d40d3d30 msgr2=0x7f37d4160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.249+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 shutdown_connections 2024-03-06T07:33:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.250+0000 7f37d8a00640 1 -- 172.21.15.6:0/2092954147 wait complete. 2024-03-06T07:33:53.273 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:33:53.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:53.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:53.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:53.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.464+0000 7f1dbe200640 1 Processor -- start 2024-03-06T07:33:53.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.465+0000 7f1dbe200640 1 -- start start 2024-03-06T07:33:53.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.466+0000 7f1dbe200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1db815f310 0x7f1db80cf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.466+0000 7f1dbe200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db80d02d0 con 0x7f1db80d2340 2024-03-06T07:33:53.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.466+0000 7f1dbe200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db80d05b0 con 0x7f1db80cfdd0 2024-03-06T07:33:53.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.466+0000 7f1dbe200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db80d0890 con 0x7f1db815f310 2024-03-06T07:33:53.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.467+0000 7f1db7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1db815f310 0x7f1db80cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.467+0000 7f1db7e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1db80d2340 0x7f1db80cce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46904/0 (socket says 172.21.15.6:46904) 2024-03-06T07:33:53.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.467+0000 7f1db7e00640 1 -- 172.21.15.6:0/3600963823 learned_addr learned my addr 172.21.15.6:0/3600963823 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:53.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.467+0000 7f1db7400640 1 -- 172.21.15.6:0/3600963823 >> v1:172.21.15.178:6789/0 conn(0x7f1db80cfdd0 legacy=0x7f1db80d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:53.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.467+0000 7f1db7400640 1 -- 172.21.15.6:0/3600963823 >> v1:172.21.15.6:6789/0 conn(0x7f1db80d2340 legacy=0x7f1db80cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.467+0000 7f1db7400640 1 -- 172.21.15.6:0/3600963823 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db80cc640 con 0x7f1db815f310 2024-03-06T07:33:53.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.468+0000 7f1db7400640 1 --2- 172.21.15.6:0/3600963823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db815f310 0x7f1db80cf890 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f1da80097a0 tx=0x7f1da802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ab264880139a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.468+0000 7f1db6a00640 1 -- 172.21.15.6:0/3600963823 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8034070 con 0x7f1db815f310 2024-03-06T07:33:53.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.468+0000 7f1db6a00640 1 -- 172.21.15.6:0/3600963823 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da8030610 con 0x7f1db815f310 2024-03-06T07:33:53.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1db6a00640 1 -- 172.21.15.6:0/3600963823 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da80309e0 con 0x7f1db815f310 2024-03-06T07:33:53.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 -- 172.21.15.6:0/3600963823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db815f310 msgr2=0x7f1db80cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 --2- 172.21.15.6:0/3600963823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db815f310 0x7f1db80cf890 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f1da80097a0 tx=0x7f1da802b960 comp rx=0 tx=0).stop 2024-03-06T07:33:53.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 -- 172.21.15.6:0/3600963823 shutdown_connections 2024-03-06T07:33:53.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 --2- 172.21.15.6:0/3600963823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db815f310 0x7f1db80cf890 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 -- 172.21.15.6:0/3600963823 >> 172.21.15.6:0/3600963823 conn(0x7f1db809aa10 msgr2=0x7f1db809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 -- 172.21.15.6:0/3600963823 shutdown_connections 2024-03-06T07:33:53.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.469+0000 7f1dbe200640 1 -- 172.21.15.6:0/3600963823 wait complete. 2024-03-06T07:33:53.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.471+0000 7f1dbe200640 1 Processor -- start 2024-03-06T07:33:53.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.471+0000 7f1dbe200640 1 -- start start 2024-03-06T07:33:53.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.472+0000 7f1dbe200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80cfdd0 0x7f1db815b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.472+0000 7f1dbe200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 0x7f1db815b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.472+0000 7f1dbe200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1db815f310 0x7f1db806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.472+0000 7f1dbe200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db815f870 con 0x7f1db80d2340 2024-03-06T07:33:53.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.472+0000 7f1dbe200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db815f6f0 con 0x7f1db80cfdd0 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.472+0000 7f1dbe200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8070bd0 con 0x7f1db815f310 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 0x7f1db815b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 0x7f1db815b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49482/0 (socket says 172.21.15.6:49482) 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 -- 172.21.15.6:0/813713890 learned_addr learned my addr 172.21.15.6:0/813713890 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1db7e00640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1db815f310 0x7f1db806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1db7400640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80cfdd0 0x7f1db815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 -- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1db815f310 msgr2=0x7f1db806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1db815f310 0x7f1db806c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 -- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80cfdd0 msgr2=0x7f1db815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80cfdd0 0x7f1db815b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1dafe00640 1 -- 172.21.15.6:0/813713890 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db81f8fa0 con 0x7f1db80d2340 2024-03-06T07:33:53.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.473+0000 7f1db7400640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80cfdd0 0x7f1db815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:53.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.474+0000 7f1dafe00640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 0x7f1db815b760 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c430 tx=0x7f1da000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.474+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011790 con 0x7f1db80d2340 2024-03-06T07:33:53.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.474+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da0017070 con 0x7f1db80d2340 2024-03-06T07:33:53.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.474+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0011bb0 con 0x7f1db80d2340 2024-03-06T07:33:53.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.474+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db80b1560 con 0x7f1db80d2340 2024-03-06T07:33:53.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.474+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db81e4960 con 0x7f1db80d2340 2024-03-06T07:33:53.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.476+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db804f3a0 con 0x7f1db80d2340 2024-03-06T07:33:53.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.477+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1da001c460 con 0x7f1db80d2340 2024-03-06T07:33:53.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.477+0000 7f1daf400640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1d9406b500 0x7f1d9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:53.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.478+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1da0089f90 con 0x7f1db80d2340 2024-03-06T07:33:53.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.478+0000 7f1db7400640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1d9406b500 0x7f1d9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:53.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.480+0000 7f1db7400640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1d9406b500 0x7f1d9406d9c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1da8036000 tx=0x7f1da803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:53.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.482+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1da0058ea0 con 0x7f1db80d2340 2024-03-06T07:33:53.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.701+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1db80cbba0 con 0x7f1d9406b500 2024-03-06T07:33:53.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.723+0000 7f1daf400640 1 -- 172.21.15.6:0/813713890 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f1db80cbba0 con 0x7f1d9406b500 2024-03-06T07:33:53.728 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:53.730 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:33:53.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.728+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1d9406b500 msgr2=0x7f1d9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.728+0000 7f1dbe200640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1d9406b500 0x7f1d9406d9c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1da8036000 tx=0x7f1da803b040 comp rx=0 tx=0).stop 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.728+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 msgr2=0x7f1db815b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.728+0000 7f1dbe200640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 0x7f1db815b760 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f1da000c430 tx=0x7f1da000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 shutdown_connections 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1d9406b500 0x7f1d9406d9c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1db815f310 0x7f1db806c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db80d2340 0x7f1db815b760 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 --2- 172.21.15.6:0/813713890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80cfdd0 0x7f1db815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 >> 172.21.15.6:0/813713890 conn(0x7f1db809aa10 msgr2=0x7f1db806f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 shutdown_connections 2024-03-06T07:33:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:53.729+0000 7f1dbe200640 1 -- 172.21.15.6:0/813713890 wait complete. 2024-03-06T07:33:53.751 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-03-06T07:33:53.578658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":160,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311048,"kb_used_data":11656,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754663672,"statfs":{"total":773094113280,"available":772775600128,"internally_reserved":0,"allocated":11935744,"data_stored":5776892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002186"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004815+0000","last_change":"2024-03-06T07:33:49.020318+0000","last_active":"2024-03-06T07:33:50.004815+0000","last_peered":"2024-03-06T07:33:50.004815+0000","last_clean":"2024-03-06T07:33:50.004815+0000","last_became_active":"2024-03-06T07:33:49.020020+0000","last_became_peered":"2024-03-06T07:33:49.020020+0000","last_unstale":"2024-03-06T07:33:50.004815+0000","last_undegraded":"2024-03-06T07:33:50.004815+0000","last_fullsized":"2024-03-06T07:33:50.004815+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:19.945099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009186+0000","last_change":"2024-03-06T07:33:49.020369+0000","last_active":"2024-03-06T07:33:50.009186+0000","last_peered":"2024-03-06T07:33:50.009186+0000","last_clean":"2024-03-06T07:33:50.009186+0000","last_became_active":"2024-03-06T07:33:49.018641+0000","last_became_peered":"2024-03-06T07:33:49.018641+0000","last_unstale":"2024-03-06T07:33:50.009186+0000","last_undegraded":"2024-03-06T07:33:50.009186+0000","last_fullsized":"2024-03-06T07:33:50.009186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.585393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015913+0000","last_change":"2024-03-06T07:33:49.015358+0000","last_active":"2024-03-06T07:33:50.015913+0000","last_peered":"2024-03-06T07:33:50.015913+0000","last_clean":"2024-03-06T07:33:50.015913+0000","last_became_active":"2024-03-06T07:33:49.010587+0000","last_became_peered":"2024-03-06T07:33:49.010587+0000","last_unstale":"2024-03-06T07:33:50.015913+0000","last_undegraded":"2024-03-06T07:33:50.015913+0000","last_fullsized":"2024-03-06T07:33:50.015913+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.585393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003713+0000","last_change":"2024-03-06T07:33:49.055172+0000","last_active":"2024-03-06T07:33:50.003713+0000","last_peered":"2024-03-06T07:33:50.003713+0000","last_clean":"2024-03-06T07:33:50.003713+0000","last_became_active":"2024-03-06T07:33:49.054792+0000","last_became_peered":"2024-03-06T07:33:49.054792+0000","last_unstale":"2024-03-06T07:33:50.003713+0000","last_undegraded":"2024-03-06T07:33:50.003713+0000","last_fullsized":"2024-03-06T07:33:50.003713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:35.668313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013612+0000","last_change":"2024-03-06T07:33:49.010420+0000","last_active":"2024-03-06T07:33:50.013612+0000","last_peered":"2024-03-06T07:33:50.013612+0000","last_clean":"2024-03-06T07:33:50.013612+0000","last_became_active":"2024-03-06T07:33:49.009352+0000","last_became_peered":"2024-03-06T07:33:49.009352+0000","last_unstale":"2024-03-06T07:33:50.013612+0000","last_undegraded":"2024-03-06T07:33:50.013612+0000","last_fullsized":"2024-03-06T07:33:50.013612+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:55.090732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014891+0000","last_change":"2024-03-06T07:33:48.995368+0000","last_active":"2024-03-06T07:33:50.014891+0000","last_peered":"2024-03-06T07:33:50.014891+0000","last_clean":"2024-03-06T07:33:50.014891+0000","last_became_active":"2024-03-06T07:33:48.995046+0000","last_became_peered":"2024-03-06T07:33:48.995046+0000","last_unstale":"2024-03-06T07:33:50.014891+0000","last_undegraded":"2024-03-06T07:33:50.014891+0000","last_fullsized":"2024-03-06T07:33:50.014891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:59.038657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015807+0000","last_change":"2024-03-06T07:33:49.014249+0000","last_active":"2024-03-06T07:33:50.015807+0000","last_peered":"2024-03-06T07:33:50.015807+0000","last_clean":"2024-03-06T07:33:50.015807+0000","last_became_active":"2024-03-06T07:33:49.013284+0000","last_became_peered":"2024-03-06T07:33:49.013284+0000","last_unstale":"2024-03-06T07:33:50.015807+0000","last_undegraded":"2024-03-06T07:33:50.015807+0000","last_fullsized":"2024-03-06T07:33:50.015807+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:45.943465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014287+0000","last_change":"2024-03-06T07:33:49.004260+0000","last_active":"2024-03-06T07:33:50.014287+0000","last_peered":"2024-03-06T07:33:50.014287+0000","last_clean":"2024-03-06T07:33:50.014287+0000","last_became_active":"2024-03-06T07:33:49.003977+0000","last_became_peered":"2024-03-06T07:33:49.003977+0000","last_unstale":"2024-03-06T07:33:50.014287+0000","last_undegraded":"2024-03-06T07:33:50.014287+0000","last_fullsized":"2024-03-06T07:33:50.014287+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:20.617077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004847+0000","last_change":"2024-03-06T07:33:49.007697+0000","last_active":"2024-03-06T07:33:50.004847+0000","last_peered":"2024-03-06T07:33:50.004847+0000","last_clean":"2024-03-06T07:33:50.004847+0000","last_became_active":"2024-03-06T07:33:49.007394+0000","last_became_peered":"2024-03-06T07:33:49.007394+0000","last_unstale":"2024-03-06T07:33:50.004847+0000","last_undegraded":"2024-03-06T07:33:50.004847+0000","last_fullsized":"2024-03-06T07:33:50.004847+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:59.162236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004590+0000","last_change":"2024-03-06T07:33:49.016221+0000","last_active":"2024-03-06T07:33:50.004590+0000","last_peered":"2024-03-06T07:33:50.004590+0000","last_clean":"2024-03-06T07:33:50.004590+0000","last_became_active":"2024-03-06T07:33:49.015749+0000","last_became_peered":"2024-03-06T07:33:49.015749+0000","last_unstale":"2024-03-06T07:33:50.004590+0000","last_undegraded":"2024-03-06T07:33:50.004590+0000","last_fullsized":"2024-03-06T07:33:50.004590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:52.486280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009709+0000","last_change":"2024-03-06T07:33:49.019180+0000","last_active":"2024-03-06T07:33:50.009709+0000","last_peered":"2024-03-06T07:33:50.009709+0000","last_clean":"2024-03-06T07:33:50.009709+0000","last_became_active":"2024-03-06T07:33:49.018805+0000","last_became_peered":"2024-03-06T07:33:49.018805+0000","last_unstale":"2024-03-06T07:33:50.009709+0000","last_undegraded":"2024-03-06T07:33:50.009709+0000","last_fullsized":"2024-03-06T07:33:50.009709+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:44.767885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004136+0000","last_change":"2024-03-06T07:33:49.008352+0000","last_active":"2024-03-06T07:33:50.004136+0000","last_peered":"2024-03-06T07:33:50.004136+0000","last_clean":"2024-03-06T07:33:50.004136+0000","last_became_active":"2024-03-06T07:33:49.005977+0000","last_became_peered":"2024-03-06T07:33:49.005977+0000","last_unstale":"2024-03-06T07:33:50.004136+0000","last_undegraded":"2024-03-06T07:33:50.004136+0000","last_fullsized":"2024-03-06T07:33:50.004136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:04.492848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015641+0000","last_change":"2024-03-06T07:33:49.002349+0000","last_active":"2024-03-06T07:33:50.015641+0000","last_peered":"2024-03-06T07:33:50.015641+0000","last_clean":"2024-03-06T07:33:50.015641+0000","last_became_active":"2024-03-06T07:33:49.002038+0000","last_became_peered":"2024-03-06T07:33:49.002038+0000","last_unstale":"2024-03-06T07:33:50.015641+0000","last_undegraded":"2024-03-06T07:33:50.015641+0000","last_fullsized":"2024-03-06T07:33:50.015641+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:44.767885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014443+0000","last_change":"2024-03-06T07:33:49.010804+0000","last_active":"2024-03-06T07:33:50.014443+0000","last_peered":"2024-03-06T07:33:50.014443+0000","last_clean":"2024-03-06T07:33:50.014443+0000","last_became_active":"2024-03-06T07:33:49.010506+0000","last_became_peered":"2024-03-06T07:33:49.010506+0000","last_unstale":"2024-03-06T07:33:50.014443+0000","last_undegraded":"2024-03-06T07:33:50.014443+0000","last_fullsized":"2024-03-06T07:33:50.014443+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:32.529908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014561+0000","last_change":"2024-03-06T07:33:48.993683+0000","last_active":"2024-03-06T07:33:50.014561+0000","last_peered":"2024-03-06T07:33:50.014561+0000","last_clean":"2024-03-06T07:33:50.014561+0000","last_became_active":"2024-03-06T07:33:48.992983+0000","last_became_peered":"2024-03-06T07:33:48.992983+0000","last_unstale":"2024-03-06T07:33:50.014561+0000","last_undegraded":"2024-03-06T07:33:50.014561+0000","last_fullsized":"2024-03-06T07:33:50.014561+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:50.074022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003196+0000","last_change":"2024-03-06T07:33:49.016391+0000","last_active":"2024-03-06T07:33:50.003196+0000","last_peered":"2024-03-06T07:33:50.003196+0000","last_clean":"2024-03-06T07:33:50.003196+0000","last_became_active":"2024-03-06T07:33:49.008654+0000","last_became_peered":"2024-03-06T07:33:49.008654+0000","last_unstale":"2024-03-06T07:33:50.003196+0000","last_undegraded":"2024-03-06T07:33:50.003196+0000","last_fullsized":"2024-03-06T07:33:50.003196+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:31.402014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008895+0000","last_change":"2024-03-06T07:33:49.018884+0000","last_active":"2024-03-06T07:33:50.008895+0000","last_peered":"2024-03-06T07:33:50.008895+0000","last_clean":"2024-03-06T07:33:50.008895+0000","last_became_active":"2024-03-06T07:33:49.018513+0000","last_became_peered":"2024-03-06T07:33:49.018513+0000","last_unstale":"2024-03-06T07:33:50.008895+0000","last_undegraded":"2024-03-06T07:33:50.008895+0000","last_fullsized":"2024-03-06T07:33:50.008895+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:56.573442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011361+0000","last_change":"2024-03-06T07:33:49.012081+0000","last_active":"2024-03-06T07:33:50.011361+0000","last_peered":"2024-03-06T07:33:50.011361+0000","last_clean":"2024-03-06T07:33:50.011361+0000","last_became_active":"2024-03-06T07:33:49.011736+0000","last_became_peered":"2024-03-06T07:33:49.011736+0000","last_unstale":"2024-03-06T07:33:50.011361+0000","last_undegraded":"2024-03-06T07:33:50.011361+0000","last_fullsized":"2024-03-06T07:33:50.011361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:52.177813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016225+0000","last_change":"2024-03-06T07:33:49.015995+0000","last_active":"2024-03-06T07:33:50.016225+0000","last_peered":"2024-03-06T07:33:50.016225+0000","last_clean":"2024-03-06T07:33:50.016225+0000","last_became_active":"2024-03-06T07:33:49.015072+0000","last_became_peered":"2024-03-06T07:33:49.015072+0000","last_unstale":"2024-03-06T07:33:50.016225+0000","last_undegraded":"2024-03-06T07:33:50.016225+0000","last_fullsized":"2024-03-06T07:33:50.016225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:07.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009656+0000","last_change":"2024-03-06T07:33:49.009221+0000","last_active":"2024-03-06T07:33:50.009656+0000","last_peered":"2024-03-06T07:33:50.009656+0000","last_clean":"2024-03-06T07:33:50.009656+0000","last_became_active":"2024-03-06T07:33:49.008939+0000","last_became_peered":"2024-03-06T07:33:49.008939+0000","last_unstale":"2024-03-06T07:33:50.009656+0000","last_undegraded":"2024-03-06T07:33:50.009656+0000","last_fullsized":"2024-03-06T07:33:50.009656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:36.385628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009201+0000","last_change":"2024-03-06T07:33:49.022515+0000","last_active":"2024-03-06T07:33:50.009201+0000","last_peered":"2024-03-06T07:33:50.009201+0000","last_clean":"2024-03-06T07:33:50.009201+0000","last_became_active":"2024-03-06T07:33:49.021894+0000","last_became_peered":"2024-03-06T07:33:49.021894+0000","last_unstale":"2024-03-06T07:33:50.009201+0000","last_undegraded":"2024-03-06T07:33:50.009201+0000","last_fullsized":"2024-03-06T07:33:50.009201+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:07.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012448+0000","last_change":"2024-03-06T07:33:48.990851+0000","last_active":"2024-03-06T07:33:50.012448+0000","last_peered":"2024-03-06T07:33:50.012448+0000","last_clean":"2024-03-06T07:33:50.012448+0000","last_became_active":"2024-03-06T07:33:48.990512+0000","last_became_peered":"2024-03-06T07:33:48.990512+0000","last_unstale":"2024-03-06T07:33:50.012448+0000","last_undegraded":"2024-03-06T07:33:50.012448+0000","last_fullsized":"2024-03-06T07:33:50.012448+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:45.151748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007675+0000","last_change":"2024-03-06T07:33:49.008664+0000","last_active":"2024-03-06T07:33:50.007675+0000","last_peered":"2024-03-06T07:33:50.007675+0000","last_clean":"2024-03-06T07:33:50.007675+0000","last_became_active":"2024-03-06T07:33:49.008333+0000","last_became_peered":"2024-03-06T07:33:49.008333+0000","last_unstale":"2024-03-06T07:33:50.007675+0000","last_undegraded":"2024-03-06T07:33:50.007675+0000","last_fullsized":"2024-03-06T07:33:50.007675+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:57.867912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012970+0000","last_change":"2024-03-06T07:33:49.014087+0000","last_active":"2024-03-06T07:33:50.012970+0000","last_peered":"2024-03-06T07:33:50.012970+0000","last_clean":"2024-03-06T07:33:50.012970+0000","last_became_active":"2024-03-06T07:33:49.013709+0000","last_became_peered":"2024-03-06T07:33:49.013709+0000","last_unstale":"2024-03-06T07:33:50.012970+0000","last_undegraded":"2024-03-06T07:33:50.012970+0000","last_fullsized":"2024-03-06T07:33:50.012970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:05.545987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008406+0000","last_change":"2024-03-06T07:33:49.020112+0000","last_active":"2024-03-06T07:33:50.008406+0000","last_peered":"2024-03-06T07:33:50.008406+0000","last_clean":"2024-03-06T07:33:50.008406+0000","last_became_active":"2024-03-06T07:33:49.018460+0000","last_became_peered":"2024-03-06T07:33:49.018460+0000","last_unstale":"2024-03-06T07:33:50.008406+0000","last_undegraded":"2024-03-06T07:33:50.008406+0000","last_fullsized":"2024-03-06T07:33:50.008406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.744126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004438+0000","last_change":"2024-03-06T07:33:49.056192+0000","last_active":"2024-03-06T07:33:50.004438+0000","last_peered":"2024-03-06T07:33:50.004438+0000","last_clean":"2024-03-06T07:33:50.004438+0000","last_became_active":"2024-03-06T07:33:49.053993+0000","last_became_peered":"2024-03-06T07:33:49.053993+0000","last_unstale":"2024-03-06T07:33:50.004438+0000","last_undegraded":"2024-03-06T07:33:50.004438+0000","last_fullsized":"2024-03-06T07:33:50.004438+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:33.452860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015160+0000","last_change":"2024-03-06T07:33:49.010237+0000","last_active":"2024-03-06T07:33:50.015160+0000","last_peered":"2024-03-06T07:33:50.015160+0000","last_clean":"2024-03-06T07:33:50.015160+0000","last_became_active":"2024-03-06T07:33:49.009505+0000","last_became_peered":"2024-03-06T07:33:49.009505+0000","last_unstale":"2024-03-06T07:33:50.015160+0000","last_undegraded":"2024-03-06T07:33:50.015160+0000","last_fullsized":"2024-03-06T07:33:50.015160+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:58.958944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012191+0000","last_change":"2024-03-06T07:33:49.012052+0000","last_active":"2024-03-06T07:33:50.012191+0000","last_peered":"2024-03-06T07:33:50.012191+0000","last_clean":"2024-03-06T07:33:50.012191+0000","last_became_active":"2024-03-06T07:33:49.011771+0000","last_became_peered":"2024-03-06T07:33:49.011771+0000","last_unstale":"2024-03-06T07:33:50.012191+0000","last_undegraded":"2024-03-06T07:33:50.012191+0000","last_fullsized":"2024-03-06T07:33:50.012191+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:37.878950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007926+0000","last_change":"2024-03-06T07:33:49.020611+0000","last_active":"2024-03-06T07:33:50.007926+0000","last_peered":"2024-03-06T07:33:50.007926+0000","last_clean":"2024-03-06T07:33:50.007926+0000","last_became_active":"2024-03-06T07:33:49.019780+0000","last_became_peered":"2024-03-06T07:33:49.019780+0000","last_unstale":"2024-03-06T07:33:50.007926+0000","last_undegraded":"2024-03-06T07:33:50.007926+0000","last_fullsized":"2024-03-06T07:33:50.007926+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:20.617077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015569+0000","last_change":"2024-03-06T07:33:49.025501+0000","last_active":"2024-03-06T07:33:50.015569+0000","last_peered":"2024-03-06T07:33:50.015569+0000","last_clean":"2024-03-06T07:33:50.015569+0000","last_became_active":"2024-03-06T07:33:49.024990+0000","last_became_peered":"2024-03-06T07:33:49.024990+0000","last_unstale":"2024-03-06T07:33:50.015569+0000","last_undegraded":"2024-03-06T07:33:50.015569+0000","last_fullsized":"2024-03-06T07:33:50.015569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:13.905043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004037+0000","last_change":"2024-03-06T07:33:49.015786+0000","last_active":"2024-03-06T07:33:50.004037+0000","last_peered":"2024-03-06T07:33:50.004037+0000","last_clean":"2024-03-06T07:33:50.004037+0000","last_became_active":"2024-03-06T07:33:49.015522+0000","last_became_peered":"2024-03-06T07:33:49.015522+0000","last_unstale":"2024-03-06T07:33:50.004037+0000","last_undegraded":"2024-03-06T07:33:50.004037+0000","last_fullsized":"2024-03-06T07:33:50.004037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:21.060300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014607+0000","last_change":"2024-03-06T07:33:48.995977+0000","last_active":"2024-03-06T07:33:50.014607+0000","last_peered":"2024-03-06T07:33:50.014607+0000","last_clean":"2024-03-06T07:33:50.014607+0000","last_became_active":"2024-03-06T07:33:48.993196+0000","last_became_peered":"2024-03-06T07:33:48.993196+0000","last_unstale":"2024-03-06T07:33:50.014607+0000","last_undegraded":"2024-03-06T07:33:50.014607+0000","last_fullsized":"2024-03-06T07:33:50.014607+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:11.990488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015443+0000","last_change":"2024-03-06T07:33:48.994605+0000","last_active":"2024-03-06T07:33:50.015443+0000","last_peered":"2024-03-06T07:33:50.015443+0000","last_clean":"2024-03-06T07:33:50.015443+0000","last_became_active":"2024-03-06T07:33:48.994104+0000","last_became_peered":"2024-03-06T07:33:48.994104+0000","last_unstale":"2024-03-06T07:33:50.015443+0000","last_undegraded":"2024-03-06T07:33:50.015443+0000","last_fullsized":"2024-03-06T07:33:50.015443+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:10.752842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016284+0000","last_change":"2024-03-06T07:33:47.991600+0000","last_active":"2024-03-06T07:33:50.016284+0000","last_peered":"2024-03-06T07:33:50.016284+0000","last_clean":"2024-03-06T07:33:50.016284+0000","last_became_active":"2024-03-06T07:33:47.991111+0000","last_became_peered":"2024-03-06T07:33:47.991111+0000","last_unstale":"2024-03-06T07:33:50.016284+0000","last_undegraded":"2024-03-06T07:33:50.016284+0000","last_fullsized":"2024-03-06T07:33:50.016284+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:46.528329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011665+0000","last_change":"2024-03-06T07:33:49.005266+0000","last_active":"2024-03-06T07:33:50.011665+0000","last_peered":"2024-03-06T07:33:50.011665+0000","last_clean":"2024-03-06T07:33:50.011665+0000","last_became_active":"2024-03-06T07:33:49.004067+0000","last_became_peered":"2024-03-06T07:33:49.004067+0000","last_unstale":"2024-03-06T07:33:50.011665+0000","last_undegraded":"2024-03-06T07:33:50.011665+0000","last_fullsized":"2024-03-06T07:33:50.011665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:01.581000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016074+0000","last_change":"2024-03-06T07:33:47.994856+0000","last_active":"2024-03-06T07:33:50.016074+0000","last_peered":"2024-03-06T07:33:50.016074+0000","last_clean":"2024-03-06T07:33:50.016074+0000","last_became_active":"2024-03-06T07:33:47.994544+0000","last_became_peered":"2024-03-06T07:33:47.994544+0000","last_unstale":"2024-03-06T07:33:50.016074+0000","last_undegraded":"2024-03-06T07:33:50.016074+0000","last_fullsized":"2024-03-06T07:33:50.016074+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:11.807540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015091+0000","last_change":"2024-03-06T07:33:48.995351+0000","last_active":"2024-03-06T07:33:50.015091+0000","last_peered":"2024-03-06T07:33:50.015091+0000","last_clean":"2024-03-06T07:33:50.015091+0000","last_became_active":"2024-03-06T07:33:48.994782+0000","last_became_peered":"2024-03-06T07:33:48.994782+0000","last_unstale":"2024-03-06T07:33:50.015091+0000","last_undegraded":"2024-03-06T07:33:50.015091+0000","last_fullsized":"2024-03-06T07:33:50.015091+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:12.331929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013485+0000","last_change":"2024-03-06T07:33:47.988680+0000","last_active":"2024-03-06T07:33:50.013485+0000","last_peered":"2024-03-06T07:33:50.013485+0000","last_clean":"2024-03-06T07:33:50.013485+0000","last_became_active":"2024-03-06T07:33:47.987739+0000","last_became_peered":"2024-03-06T07:33:47.987739+0000","last_unstale":"2024-03-06T07:33:50.013485+0000","last_undegraded":"2024-03-06T07:33:50.013485+0000","last_fullsized":"2024-03-06T07:33:50.013485+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:59.128371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016263+0000","last_change":"2024-03-06T07:33:49.016229+0000","last_active":"2024-03-06T07:33:50.016263+0000","last_peered":"2024-03-06T07:33:50.016263+0000","last_clean":"2024-03-06T07:33:50.016263+0000","last_became_active":"2024-03-06T07:33:49.015698+0000","last_became_peered":"2024-03-06T07:33:49.015698+0000","last_unstale":"2024-03-06T07:33:50.016263+0000","last_undegraded":"2024-03-06T07:33:50.016263+0000","last_fullsized":"2024-03-06T07:33:50.016263+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:36.385628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013264+0000","last_change":"2024-03-06T07:33:47.988379+0000","last_active":"2024-03-06T07:33:50.013264+0000","last_peered":"2024-03-06T07:33:50.013264+0000","last_clean":"2024-03-06T07:33:50.013264+0000","last_became_active":"2024-03-06T07:33:47.988115+0000","last_became_peered":"2024-03-06T07:33:47.988115+0000","last_unstale":"2024-03-06T07:33:50.013264+0000","last_undegraded":"2024-03-06T07:33:50.013264+0000","last_fullsized":"2024-03-06T07:33:50.013264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:39.631996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003754+0000","last_change":"2024-03-06T07:33:49.054578+0000","last_active":"2024-03-06T07:33:50.003754+0000","last_peered":"2024-03-06T07:33:50.003754+0000","last_clean":"2024-03-06T07:33:50.003754+0000","last_became_active":"2024-03-06T07:33:49.054237+0000","last_became_peered":"2024-03-06T07:33:49.054237+0000","last_unstale":"2024-03-06T07:33:50.003754+0000","last_undegraded":"2024-03-06T07:33:50.003754+0000","last_fullsized":"2024-03-06T07:33:50.003754+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:00.041413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015464+0000","last_change":"2024-03-06T07:33:47.988192+0000","last_active":"2024-03-06T07:33:50.015464+0000","last_peered":"2024-03-06T07:33:50.015464+0000","last_clean":"2024-03-06T07:33:50.015464+0000","last_became_active":"2024-03-06T07:33:47.987831+0000","last_became_peered":"2024-03-06T07:33:47.987831+0000","last_unstale":"2024-03-06T07:33:50.015464+0000","last_undegraded":"2024-03-06T07:33:50.015464+0000","last_fullsized":"2024-03-06T07:33:50.015464+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:00.723758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008279+0000","last_change":"2024-03-06T07:33:49.013658+0000","last_active":"2024-03-06T07:33:50.008279+0000","last_peered":"2024-03-06T07:33:50.008279+0000","last_clean":"2024-03-06T07:33:50.008279+0000","last_became_active":"2024-03-06T07:33:49.013359+0000","last_became_peered":"2024-03-06T07:33:49.013359+0000","last_unstale":"2024-03-06T07:33:50.008279+0000","last_undegraded":"2024-03-06T07:33:50.008279+0000","last_fullsized":"2024-03-06T07:33:50.008279+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:32.529908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012772+0000","last_change":"2024-03-06T07:33:47.986784+0000","last_active":"2024-03-06T07:33:50.012772+0000","last_peered":"2024-03-06T07:33:50.012772+0000","last_clean":"2024-03-06T07:33:50.012772+0000","last_became_active":"2024-03-06T07:33:47.986447+0000","last_became_peered":"2024-03-06T07:33:47.986447+0000","last_unstale":"2024-03-06T07:33:50.012772+0000","last_undegraded":"2024-03-06T07:33:50.012772+0000","last_fullsized":"2024-03-06T07:33:50.012772+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:59.693081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003198+0000","last_change":"2024-03-06T07:33:49.008911+0000","last_active":"2024-03-06T07:33:50.003198+0000","last_peered":"2024-03-06T07:33:50.003198+0000","last_clean":"2024-03-06T07:33:50.003198+0000","last_became_active":"2024-03-06T07:33:49.006748+0000","last_became_peered":"2024-03-06T07:33:49.006748+0000","last_unstale":"2024-03-06T07:33:50.003198+0000","last_undegraded":"2024-03-06T07:33:50.003198+0000","last_fullsized":"2024-03-06T07:33:50.003198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:31.308345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007525+0000","last_change":"2024-03-06T07:33:47.985308+0000","last_active":"2024-03-06T07:33:50.007525+0000","last_peered":"2024-03-06T07:33:50.007525+0000","last_clean":"2024-03-06T07:33:50.007525+0000","last_became_active":"2024-03-06T07:33:47.984226+0000","last_became_peered":"2024-03-06T07:33:47.984226+0000","last_unstale":"2024-03-06T07:33:50.007525+0000","last_undegraded":"2024-03-06T07:33:50.007525+0000","last_fullsized":"2024-03-06T07:33:50.007525+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:25.494570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014665+0000","last_change":"2024-03-06T07:33:49.018940+0000","last_active":"2024-03-06T07:33:50.014665+0000","last_peered":"2024-03-06T07:33:50.014665+0000","last_clean":"2024-03-06T07:33:50.014665+0000","last_became_active":"2024-03-06T07:33:49.018645+0000","last_became_peered":"2024-03-06T07:33:49.018645+0000","last_unstale":"2024-03-06T07:33:50.014665+0000","last_undegraded":"2024-03-06T07:33:50.014665+0000","last_fullsized":"2024-03-06T07:33:50.014665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:17.902238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003401+0000","last_change":"2024-03-06T07:33:47.985494+0000","last_active":"2024-03-06T07:33:50.003401+0000","last_peered":"2024-03-06T07:33:50.003401+0000","last_clean":"2024-03-06T07:33:50.003401+0000","last_became_active":"2024-03-06T07:33:47.985232+0000","last_became_peered":"2024-03-06T07:33:47.985232+0000","last_unstale":"2024-03-06T07:33:50.003401+0000","last_undegraded":"2024-03-06T07:33:50.003401+0000","last_fullsized":"2024-03-06T07:33:50.003401+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:00.565493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016590+0000","last_change":"2024-03-06T07:33:49.014719+0000","last_active":"2024-03-06T07:33:50.016590+0000","last_peered":"2024-03-06T07:33:50.016590+0000","last_clean":"2024-03-06T07:33:50.016590+0000","last_became_active":"2024-03-06T07:33:49.014356+0000","last_became_peered":"2024-03-06T07:33:49.014356+0000","last_unstale":"2024-03-06T07:33:50.016590+0000","last_undegraded":"2024-03-06T07:33:50.016590+0000","last_fullsized":"2024-03-06T07:33:50.016590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.578502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013830+0000","last_change":"2024-03-06T07:33:47.986309+0000","last_active":"2024-03-06T07:33:50.013830+0000","last_peered":"2024-03-06T07:33:50.013830+0000","last_clean":"2024-03-06T07:33:50.013830+0000","last_became_active":"2024-03-06T07:33:47.986040+0000","last_became_peered":"2024-03-06T07:33:47.986040+0000","last_unstale":"2024-03-06T07:33:50.013830+0000","last_undegraded":"2024-03-06T07:33:50.013830+0000","last_fullsized":"2024-03-06T07:33:50.013830+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:36.359524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011038+0000","last_change":"2024-03-06T07:33:49.006283+0000","last_active":"2024-03-06T07:33:50.011038+0000","last_peered":"2024-03-06T07:33:50.011038+0000","last_clean":"2024-03-06T07:33:50.011038+0000","last_became_active":"2024-03-06T07:33:49.005896+0000","last_became_peered":"2024-03-06T07:33:49.005896+0000","last_unstale":"2024-03-06T07:33:50.011038+0000","last_undegraded":"2024-03-06T07:33:50.011038+0000","last_fullsized":"2024-03-06T07:33:50.011038+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:34.439233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011926+0000","last_change":"2024-03-06T07:33:47.991647+0000","last_active":"2024-03-06T07:33:50.011926+0000","last_peered":"2024-03-06T07:33:50.011926+0000","last_clean":"2024-03-06T07:33:50.011926+0000","last_became_active":"2024-03-06T07:33:47.991383+0000","last_became_peered":"2024-03-06T07:33:47.991383+0000","last_unstale":"2024-03-06T07:33:50.011926+0000","last_undegraded":"2024-03-06T07:33:50.011926+0000","last_fullsized":"2024-03-06T07:33:50.011926+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:15.664554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012352+0000","last_change":"2024-03-06T07:33:49.023569+0000","last_active":"2024-03-06T07:33:50.012352+0000","last_peered":"2024-03-06T07:33:50.012352+0000","last_clean":"2024-03-06T07:33:50.012352+0000","last_became_active":"2024-03-06T07:33:49.018365+0000","last_became_peered":"2024-03-06T07:33:49.018365+0000","last_unstale":"2024-03-06T07:33:50.012352+0000","last_undegraded":"2024-03-06T07:33:50.012352+0000","last_fullsized":"2024-03-06T07:33:50.012352+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:14.501377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004074+0000","last_change":"2024-03-06T07:33:47.982938+0000","last_active":"2024-03-06T07:33:50.004074+0000","last_peered":"2024-03-06T07:33:50.004074+0000","last_clean":"2024-03-06T07:33:50.004074+0000","last_became_active":"2024-03-06T07:33:47.982604+0000","last_became_peered":"2024-03-06T07:33:47.982604+0000","last_unstale":"2024-03-06T07:33:50.004074+0000","last_undegraded":"2024-03-06T07:33:50.004074+0000","last_fullsized":"2024-03-06T07:33:50.004074+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:50.759169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015383+0000","last_change":"2024-03-06T07:33:49.019238+0000","last_active":"2024-03-06T07:33:50.015383+0000","last_peered":"2024-03-06T07:33:50.015383+0000","last_clean":"2024-03-06T07:33:50.015383+0000","last_became_active":"2024-03-06T07:33:49.018964+0000","last_became_peered":"2024-03-06T07:33:49.018964+0000","last_unstale":"2024-03-06T07:33:50.015383+0000","last_undegraded":"2024-03-06T07:33:50.015383+0000","last_fullsized":"2024-03-06T07:33:50.015383+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:35.168326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004447+0000","last_change":"2024-03-06T07:33:47.984898+0000","last_active":"2024-03-06T07:33:50.004447+0000","last_peered":"2024-03-06T07:33:50.004447+0000","last_clean":"2024-03-06T07:33:50.004447+0000","last_became_active":"2024-03-06T07:33:47.984534+0000","last_became_peered":"2024-03-06T07:33:47.984534+0000","last_unstale":"2024-03-06T07:33:50.004447+0000","last_undegraded":"2024-03-06T07:33:50.004447+0000","last_fullsized":"2024-03-06T07:33:50.004447+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:43.071436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012643+0000","last_change":"2024-03-06T07:33:49.017384+0000","last_active":"2024-03-06T07:33:50.012643+0000","last_peered":"2024-03-06T07:33:50.012643+0000","last_clean":"2024-03-06T07:33:50.012643+0000","last_became_active":"2024-03-06T07:33:49.017100+0000","last_became_peered":"2024-03-06T07:33:49.017100+0000","last_unstale":"2024-03-06T07:33:50.012643+0000","last_undegraded":"2024-03-06T07:33:50.012643+0000","last_fullsized":"2024-03-06T07:33:50.012643+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:01.722730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014994+0000","last_change":"2024-03-06T07:33:47.988666+0000","last_active":"2024-03-06T07:33:50.014994+0000","last_peered":"2024-03-06T07:33:50.014994+0000","last_clean":"2024-03-06T07:33:50.014994+0000","last_became_active":"2024-03-06T07:33:47.987376+0000","last_became_peered":"2024-03-06T07:33:47.987376+0000","last_unstale":"2024-03-06T07:33:50.014994+0000","last_undegraded":"2024-03-06T07:33:50.014994+0000","last_fullsized":"2024-03-06T07:33:50.014994+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:54.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012732+0000","last_change":"2024-03-06T07:33:49.011247+0000","last_active":"2024-03-06T07:33:50.012732+0000","last_peered":"2024-03-06T07:33:50.012732+0000","last_clean":"2024-03-06T07:33:50.012732+0000","last_became_active":"2024-03-06T07:33:49.009708+0000","last_became_peered":"2024-03-06T07:33:49.009708+0000","last_unstale":"2024-03-06T07:33:50.012732+0000","last_undegraded":"2024-03-06T07:33:50.012732+0000","last_fullsized":"2024-03-06T07:33:50.012732+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:11.109818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009459+0000","last_change":"2024-03-06T07:33:47.984467+0000","last_active":"2024-03-06T07:33:50.009459+0000","last_peered":"2024-03-06T07:33:50.009459+0000","last_clean":"2024-03-06T07:33:50.009459+0000","last_became_active":"2024-03-06T07:33:47.983476+0000","last_became_peered":"2024-03-06T07:33:47.983476+0000","last_unstale":"2024-03-06T07:33:50.009459+0000","last_undegraded":"2024-03-06T07:33:50.009459+0000","last_fullsized":"2024-03-06T07:33:50.009459+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:14.299326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012050+0000","last_change":"2024-03-06T07:33:49.017308+0000","last_active":"2024-03-06T07:33:50.012050+0000","last_peered":"2024-03-06T07:33:50.012050+0000","last_clean":"2024-03-06T07:33:50.012050+0000","last_became_active":"2024-03-06T07:33:49.016740+0000","last_became_peered":"2024-03-06T07:33:49.016740+0000","last_unstale":"2024-03-06T07:33:50.012050+0000","last_undegraded":"2024-03-06T07:33:50.012050+0000","last_fullsized":"2024-03-06T07:33:50.012050+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:38.646550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013087+0000","last_change":"2024-03-06T07:33:47.976997+0000","last_active":"2024-03-06T07:33:50.013087+0000","last_peered":"2024-03-06T07:33:50.013087+0000","last_clean":"2024-03-06T07:33:50.013087+0000","last_became_active":"2024-03-06T07:33:47.976654+0000","last_became_peered":"2024-03-06T07:33:47.976654+0000","last_unstale":"2024-03-06T07:33:50.013087+0000","last_undegraded":"2024-03-06T07:33:50.013087+0000","last_fullsized":"2024-03-06T07:33:50.013087+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:15.859152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013728+0000","last_change":"2024-03-06T07:33:49.005374+0000","last_active":"2024-03-06T07:33:50.013728+0000","last_peered":"2024-03-06T07:33:50.013728+0000","last_clean":"2024-03-06T07:33:50.013728+0000","last_became_active":"2024-03-06T07:33:49.005153+0000","last_became_peered":"2024-03-06T07:33:49.005153+0000","last_unstale":"2024-03-06T07:33:50.013728+0000","last_undegraded":"2024-03-06T07:33:50.013728+0000","last_fullsized":"2024-03-06T07:33:50.013728+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:09.151965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004931+0000","last_change":"2024-03-06T07:33:47.990909+0000","last_active":"2024-03-06T07:33:50.004931+0000","last_peered":"2024-03-06T07:33:50.004931+0000","last_clean":"2024-03-06T07:33:50.004931+0000","last_became_active":"2024-03-06T07:33:47.987760+0000","last_became_peered":"2024-03-06T07:33:47.987760+0000","last_unstale":"2024-03-06T07:33:50.004931+0000","last_undegraded":"2024-03-06T07:33:50.004931+0000","last_fullsized":"2024-03-06T07:33:50.004931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:36.863443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015476+0000","last_change":"2024-03-06T07:33:49.007482+0000","last_active":"2024-03-06T07:33:50.015476+0000","last_peered":"2024-03-06T07:33:50.015476+0000","last_clean":"2024-03-06T07:33:50.015476+0000","last_became_active":"2024-03-06T07:33:49.007206+0000","last_became_peered":"2024-03-06T07:33:49.007206+0000","last_unstale":"2024-03-06T07:33:50.015476+0000","last_undegraded":"2024-03-06T07:33:50.015476+0000","last_fullsized":"2024-03-06T07:33:50.015476+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:37.203499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013673+0000","last_change":"2024-03-06T07:33:47.989436+0000","last_active":"2024-03-06T07:33:50.013673+0000","last_peered":"2024-03-06T07:33:50.013673+0000","last_clean":"2024-03-06T07:33:50.013673+0000","last_became_active":"2024-03-06T07:33:47.989129+0000","last_became_peered":"2024-03-06T07:33:47.989129+0000","last_unstale":"2024-03-06T07:33:50.013673+0000","last_undegraded":"2024-03-06T07:33:50.013673+0000","last_fullsized":"2024-03-06T07:33:50.013673+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:43.654988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015827+0000","last_change":"2024-03-06T07:33:49.056390+0000","last_active":"2024-03-06T07:33:50.015827+0000","last_peered":"2024-03-06T07:33:50.015827+0000","last_clean":"2024-03-06T07:33:50.015827+0000","last_became_active":"2024-03-06T07:33:49.056062+0000","last_became_peered":"2024-03-06T07:33:49.056062+0000","last_unstale":"2024-03-06T07:33:50.015827+0000","last_undegraded":"2024-03-06T07:33:50.015827+0000","last_fullsized":"2024-03-06T07:33:50.015827+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:47.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.010696+0000","last_change":"2024-03-06T07:33:47.983724+0000","last_active":"2024-03-06T07:33:50.010696+0000","last_peered":"2024-03-06T07:33:50.010696+0000","last_clean":"2024-03-06T07:33:50.010696+0000","last_became_active":"2024-03-06T07:33:47.983154+0000","last_became_peered":"2024-03-06T07:33:47.983154+0000","last_unstale":"2024-03-06T07:33:50.010696+0000","last_undegraded":"2024-03-06T07:33:50.010696+0000","last_fullsized":"2024-03-06T07:33:50.010696+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:00.870059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016631+0000","last_change":"2024-03-06T07:33:49.022820+0000","last_active":"2024-03-06T07:33:50.016631+0000","last_peered":"2024-03-06T07:33:50.016631+0000","last_clean":"2024-03-06T07:33:50.016631+0000","last_became_active":"2024-03-06T07:33:49.020120+0000","last_became_peered":"2024-03-06T07:33:49.020120+0000","last_unstale":"2024-03-06T07:33:50.016631+0000","last_undegraded":"2024-03-06T07:33:50.016631+0000","last_fullsized":"2024-03-06T07:33:50.016631+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:17.789144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015901+0000","last_change":"2024-03-06T07:33:47.989741+0000","last_active":"2024-03-06T07:33:50.015901+0000","last_peered":"2024-03-06T07:33:50.015901+0000","last_clean":"2024-03-06T07:33:50.015901+0000","last_became_active":"2024-03-06T07:33:47.988639+0000","last_became_peered":"2024-03-06T07:33:47.988639+0000","last_unstale":"2024-03-06T07:33:50.015901+0000","last_undegraded":"2024-03-06T07:33:50.015901+0000","last_fullsized":"2024-03-06T07:33:50.015901+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:56.579196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014656+0000","last_change":"2024-03-06T07:33:49.014635+0000","last_active":"2024-03-06T07:33:50.014656+0000","last_peered":"2024-03-06T07:33:50.014656+0000","last_clean":"2024-03-06T07:33:50.014656+0000","last_became_active":"2024-03-06T07:33:49.014314+0000","last_became_peered":"2024-03-06T07:33:49.014314+0000","last_unstale":"2024-03-06T07:33:50.014656+0000","last_undegraded":"2024-03-06T07:33:50.014656+0000","last_fullsized":"2024-03-06T07:33:50.014656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.744126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015182+0000","last_change":"2024-03-06T07:33:47.987544+0000","last_active":"2024-03-06T07:33:50.015182+0000","last_peered":"2024-03-06T07:33:50.015182+0000","last_clean":"2024-03-06T07:33:50.015182+0000","last_became_active":"2024-03-06T07:33:47.986492+0000","last_became_peered":"2024-03-06T07:33:47.986492+0000","last_unstale":"2024-03-06T07:33:50.015182+0000","last_undegraded":"2024-03-06T07:33:50.015182+0000","last_fullsized":"2024-03-06T07:33:50.015182+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:08.136458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003259+0000","last_change":"2024-03-06T07:33:49.006831+0000","last_active":"2024-03-06T07:33:50.003259+0000","last_peered":"2024-03-06T07:33:50.003259+0000","last_clean":"2024-03-06T07:33:50.003259+0000","last_became_active":"2024-03-06T07:33:49.006548+0000","last_became_peered":"2024-03-06T07:33:49.006548+0000","last_unstale":"2024-03-06T07:33:50.003259+0000","last_undegraded":"2024-03-06T07:33:50.003259+0000","last_fullsized":"2024-03-06T07:33:50.003259+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:04.859114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013205+0000","last_change":"2024-03-06T07:33:47.987714+0000","last_active":"2024-03-06T07:33:50.013205+0000","last_peered":"2024-03-06T07:33:50.013205+0000","last_clean":"2024-03-06T07:33:50.013205+0000","last_became_active":"2024-03-06T07:33:47.987418+0000","last_became_peered":"2024-03-06T07:33:47.987418+0000","last_unstale":"2024-03-06T07:33:50.013205+0000","last_undegraded":"2024-03-06T07:33:50.013205+0000","last_fullsized":"2024-03-06T07:33:50.013205+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:53.196457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014618+0000","last_change":"2024-03-06T07:33:49.012124+0000","last_active":"2024-03-06T07:33:50.014618+0000","last_peered":"2024-03-06T07:33:50.014618+0000","last_clean":"2024-03-06T07:33:50.014618+0000","last_became_active":"2024-03-06T07:33:49.010814+0000","last_became_peered":"2024-03-06T07:33:49.010814+0000","last_unstale":"2024-03-06T07:33:50.014618+0000","last_undegraded":"2024-03-06T07:33:50.014618+0000","last_fullsized":"2024-03-06T07:33:50.014618+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:33.676181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014632+0000","last_change":"2024-03-06T07:33:47.988212+0000","last_active":"2024-03-06T07:33:50.014632+0000","last_peered":"2024-03-06T07:33:50.014632+0000","last_clean":"2024-03-06T07:33:50.014632+0000","last_became_active":"2024-03-06T07:33:47.987770+0000","last_became_peered":"2024-03-06T07:33:47.987770+0000","last_unstale":"2024-03-06T07:33:50.014632+0000","last_undegraded":"2024-03-06T07:33:50.014632+0000","last_fullsized":"2024-03-06T07:33:50.014632+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:25.587698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012273+0000","last_change":"2024-03-06T07:33:49.007895+0000","last_active":"2024-03-06T07:33:50.012273+0000","last_peered":"2024-03-06T07:33:50.012273+0000","last_clean":"2024-03-06T07:33:50.012273+0000","last_became_active":"2024-03-06T07:33:49.007602+0000","last_became_peered":"2024-03-06T07:33:49.007602+0000","last_unstale":"2024-03-06T07:33:50.012273+0000","last_undegraded":"2024-03-06T07:33:50.012273+0000","last_fullsized":"2024-03-06T07:33:50.012273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:37.594251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004255+0000","last_change":"2024-03-06T07:33:47.990478+0000","last_active":"2024-03-06T07:33:50.004255+0000","last_peered":"2024-03-06T07:33:50.004255+0000","last_clean":"2024-03-06T07:33:50.004255+0000","last_became_active":"2024-03-06T07:33:47.990156+0000","last_became_peered":"2024-03-06T07:33:47.990156+0000","last_unstale":"2024-03-06T07:33:50.004255+0000","last_undegraded":"2024-03-06T07:33:50.004255+0000","last_fullsized":"2024-03-06T07:33:50.004255+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:33.423726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011531+0000","last_change":"2024-03-06T07:33:49.006209+0000","last_active":"2024-03-06T07:33:50.011531+0000","last_peered":"2024-03-06T07:33:50.011531+0000","last_clean":"2024-03-06T07:33:50.011531+0000","last_became_active":"2024-03-06T07:33:49.005513+0000","last_became_peered":"2024-03-06T07:33:49.005513+0000","last_unstale":"2024-03-06T07:33:50.011531+0000","last_undegraded":"2024-03-06T07:33:50.011531+0000","last_fullsized":"2024-03-06T07:33:50.011531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:54.664883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012374+0000","last_change":"2024-03-06T07:33:47.981443+0000","last_active":"2024-03-06T07:33:50.012374+0000","last_peered":"2024-03-06T07:33:50.012374+0000","last_clean":"2024-03-06T07:33:50.012374+0000","last_became_active":"2024-03-06T07:33:47.981169+0000","last_became_peered":"2024-03-06T07:33:47.981169+0000","last_unstale":"2024-03-06T07:33:50.012374+0000","last_undegraded":"2024-03-06T07:33:50.012374+0000","last_fullsized":"2024-03-06T07:33:50.012374+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.132652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004965+0000","last_change":"2024-03-06T07:33:49.022092+0000","last_active":"2024-03-06T07:33:50.004965+0000","last_peered":"2024-03-06T07:33:50.004965+0000","last_clean":"2024-03-06T07:33:50.004965+0000","last_became_active":"2024-03-06T07:33:49.021531+0000","last_became_peered":"2024-03-06T07:33:49.021531+0000","last_unstale":"2024-03-06T07:33:50.004965+0000","last_undegraded":"2024-03-06T07:33:50.004965+0000","last_fullsized":"2024-03-06T07:33:50.004965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:59.204169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011720+0000","last_change":"2024-03-06T07:33:47.988462+0000","last_active":"2024-03-06T07:33:50.011720+0000","last_peered":"2024-03-06T07:33:50.011720+0000","last_clean":"2024-03-06T07:33:50.011720+0000","last_became_active":"2024-03-06T07:33:47.988170+0000","last_became_peered":"2024-03-06T07:33:47.988170+0000","last_unstale":"2024-03-06T07:33:50.011720+0000","last_undegraded":"2024-03-06T07:33:50.011720+0000","last_fullsized":"2024-03-06T07:33:50.011720+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:59.693081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004495+0000","last_change":"2024-03-06T07:33:49.010220+0000","last_active":"2024-03-06T07:33:50.004495+0000","last_peered":"2024-03-06T07:33:50.004495+0000","last_clean":"2024-03-06T07:33:50.004495+0000","last_became_active":"2024-03-06T07:33:49.009891+0000","last_became_peered":"2024-03-06T07:33:49.009891+0000","last_unstale":"2024-03-06T07:33:50.004495+0000","last_undegraded":"2024-03-06T07:33:50.004495+0000","last_fullsized":"2024-03-06T07:33:50.004495+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:54.931396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004203+0000","last_change":"2024-03-06T07:33:47.989688+0000","last_active":"2024-03-06T07:33:50.004203+0000","last_peered":"2024-03-06T07:33:50.004203+0000","last_clean":"2024-03-06T07:33:50.004203+0000","last_became_active":"2024-03-06T07:33:47.989397+0000","last_became_peered":"2024-03-06T07:33:47.989397+0000","last_unstale":"2024-03-06T07:33:50.004203+0000","last_undegraded":"2024-03-06T07:33:50.004203+0000","last_fullsized":"2024-03-06T07:33:50.004203+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:53.649376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.010227+0000","last_change":"2024-03-06T07:33:49.012423+0000","last_active":"2024-03-06T07:33:50.010227+0000","last_peered":"2024-03-06T07:33:50.010227+0000","last_clean":"2024-03-06T07:33:50.010227+0000","last_became_active":"2024-03-06T07:33:49.012117+0000","last_became_peered":"2024-03-06T07:33:49.012117+0000","last_unstale":"2024-03-06T07:33:50.010227+0000","last_undegraded":"2024-03-06T07:33:50.010227+0000","last_fullsized":"2024-03-06T07:33:50.010227+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:38.385541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012452+0000","last_change":"2024-03-06T07:33:47.983772+0000","last_active":"2024-03-06T07:33:50.012452+0000","last_peered":"2024-03-06T07:33:50.012452+0000","last_clean":"2024-03-06T07:33:50.012452+0000","last_became_active":"2024-03-06T07:33:47.983340+0000","last_became_peered":"2024-03-06T07:33:47.983340+0000","last_unstale":"2024-03-06T07:33:50.012452+0000","last_undegraded":"2024-03-06T07:33:50.012452+0000","last_fullsized":"2024-03-06T07:33:50.012452+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:51.197766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007616+0000","last_change":"2024-03-06T07:33:49.017001+0000","last_active":"2024-03-06T07:33:50.007616+0000","last_peered":"2024-03-06T07:33:50.007616+0000","last_clean":"2024-03-06T07:33:50.007616+0000","last_became_active":"2024-03-06T07:33:49.016453+0000","last_became_peered":"2024-03-06T07:33:49.016453+0000","last_unstale":"2024-03-06T07:33:50.007616+0000","last_undegraded":"2024-03-06T07:33:50.007616+0000","last_fullsized":"2024-03-06T07:33:50.007616+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:33.095797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015719+0000","last_change":"2024-03-06T07:33:47.991027+0000","last_active":"2024-03-06T07:33:50.015719+0000","last_peered":"2024-03-06T07:33:50.015719+0000","last_clean":"2024-03-06T07:33:50.015719+0000","last_became_active":"2024-03-06T07:33:47.990666+0000","last_became_peered":"2024-03-06T07:33:47.990666+0000","last_unstale":"2024-03-06T07:33:50.015719+0000","last_undegraded":"2024-03-06T07:33:50.015719+0000","last_fullsized":"2024-03-06T07:33:50.015719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:40.015349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008689+0000","last_change":"2024-03-06T07:33:49.014461+0000","last_active":"2024-03-06T07:33:50.008689+0000","last_peered":"2024-03-06T07:33:50.008689+0000","last_clean":"2024-03-06T07:33:50.008689+0000","last_became_active":"2024-03-06T07:33:49.012167+0000","last_became_peered":"2024-03-06T07:33:49.012167+0000","last_unstale":"2024-03-06T07:33:50.008689+0000","last_undegraded":"2024-03-06T07:33:50.008689+0000","last_fullsized":"2024-03-06T07:33:50.008689+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:47.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008755+0000","last_change":"2024-03-06T07:33:47.993010+0000","last_active":"2024-03-06T07:33:50.008755+0000","last_peered":"2024-03-06T07:33:50.008755+0000","last_clean":"2024-03-06T07:33:50.008755+0000","last_became_active":"2024-03-06T07:33:47.992686+0000","last_became_peered":"2024-03-06T07:33:47.992686+0000","last_unstale":"2024-03-06T07:33:50.008755+0000","last_undegraded":"2024-03-06T07:33:50.008755+0000","last_fullsized":"2024-03-06T07:33:50.008755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:43.804662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014734+0000","last_change":"2024-03-06T07:33:49.020253+0000","last_active":"2024-03-06T07:33:50.014734+0000","last_peered":"2024-03-06T07:33:50.014734+0000","last_clean":"2024-03-06T07:33:50.014734+0000","last_became_active":"2024-03-06T07:33:49.017442+0000","last_became_peered":"2024-03-06T07:33:49.017442+0000","last_unstale":"2024-03-06T07:33:50.014734+0000","last_undegraded":"2024-03-06T07:33:50.014734+0000","last_fullsized":"2024-03-06T07:33:50.014734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:56.573442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013229+0000","last_change":"2024-03-06T07:33:47.985665+0000","last_active":"2024-03-06T07:33:50.013229+0000","last_peered":"2024-03-06T07:33:50.013229+0000","last_clean":"2024-03-06T07:33:50.013229+0000","last_became_active":"2024-03-06T07:33:47.985347+0000","last_became_peered":"2024-03-06T07:33:47.985347+0000","last_unstale":"2024-03-06T07:33:50.013229+0000","last_undegraded":"2024-03-06T07:33:50.013229+0000","last_fullsized":"2024-03-06T07:33:50.013229+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:01.117472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004227+0000","last_change":"2024-03-06T07:33:49.005688+0000","last_active":"2024-03-06T07:33:50.004227+0000","last_peered":"2024-03-06T07:33:50.004227+0000","last_clean":"2024-03-06T07:33:50.004227+0000","last_became_active":"2024-03-06T07:33:49.004361+0000","last_became_peered":"2024-03-06T07:33:49.004361+0000","last_unstale":"2024-03-06T07:33:50.004227+0000","last_undegraded":"2024-03-06T07:33:50.004227+0000","last_fullsized":"2024-03-06T07:33:50.004227+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:38.852254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003648+0000","last_change":"2024-03-06T07:33:47.989927+0000","last_active":"2024-03-06T07:33:50.003648+0000","last_peered":"2024-03-06T07:33:50.003648+0000","last_clean":"2024-03-06T07:33:50.003648+0000","last_became_active":"2024-03-06T07:33:47.989602+0000","last_became_peered":"2024-03-06T07:33:47.989602+0000","last_unstale":"2024-03-06T07:33:50.003648+0000","last_undegraded":"2024-03-06T07:33:50.003648+0000","last_fullsized":"2024-03-06T07:33:50.003648+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:51.162306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012582+0000","last_change":"2024-03-06T07:33:49.023480+0000","last_active":"2024-03-06T07:33:50.012582+0000","last_peered":"2024-03-06T07:33:50.012582+0000","last_clean":"2024-03-06T07:33:50.012582+0000","last_became_active":"2024-03-06T07:33:49.023176+0000","last_became_peered":"2024-03-06T07:33:49.023176+0000","last_unstale":"2024-03-06T07:33:50.012582+0000","last_undegraded":"2024-03-06T07:33:50.012582+0000","last_fullsized":"2024-03-06T07:33:50.012582+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:07.549515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015267+0000","last_change":"2024-03-06T07:33:47.982237+0000","last_active":"2024-03-06T07:33:50.015267+0000","last_peered":"2024-03-06T07:33:50.015267+0000","last_clean":"2024-03-06T07:33:50.015267+0000","last_became_active":"2024-03-06T07:33:47.981853+0000","last_became_peered":"2024-03-06T07:33:47.981853+0000","last_unstale":"2024-03-06T07:33:50.015267+0000","last_undegraded":"2024-03-06T07:33:50.015267+0000","last_fullsized":"2024-03-06T07:33:50.015267+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:21.031145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":325,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014884+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:33:50.014884+0000","last_peered":"2024-03-06T07:33:50.014884+0000","last_clean":"2024-03-06T07:33:50.014884+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:33:50.014884+0000","last_undegraded":"2024-03-06T07:33:50.014884+0000","last_fullsized":"2024-03-06T07:33:50.014884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509521,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1335296,"data_stored":573917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509522,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1343488,"data_stored":574041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542225,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1937408,"data_stored":1164901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40812,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331028,"statfs":{"total":96636764160,"available":96594972672,"internally_reserved":0,"allocated":1339392,"data_stored":573987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":574521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542225,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1335296,"data_stored":573917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542226,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1351680,"data_stored":575599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37248,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334592,"statfs":{"total":96636764160,"available":96598622208,"internally_reserved":0,"allocated":1949696,"data_stored":1166009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:53.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:33:53.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T07:33:53.925+0000 7fdc32600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:33:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.010+0000 7f0844a00640 1 Processor -- start 2024-03-06T07:33:54.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.011+0000 7f0844a00640 1 -- start start 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.012+0000 7f0844a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f084015efc0 0x7f08400cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.012+0000 7f0844a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08400cce90 0x7f0840161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.012+0000 7f0844a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0840162000 0x7f08400ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.012+0000 7f0844a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084015f3a0 con 0x7f08400cce90 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.012+0000 7f0844a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084006d190 con 0x7f0840162000 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.012+0000 7f0844a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08401627f0 con 0x7f084015efc0 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08400cce90 0x7f0840161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f08400cce90 0x7f0840161ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49492/0 (socket says 172.21.15.6:49492) 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 -- 172.21.15.6:0/3121472375 learned_addr learned my addr 172.21.15.6:0/3121472375 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f083e800640 1 --2- 172.21.15.6:0/3121472375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f084015efc0 0x7f08400cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f083f200640 1 --2- 172.21.15.6:0/3121472375 >> v2:172.21.15.178:3300/0 conn(0x7f0840162000 0x7f08400ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 -- 172.21.15.6:0/3121472375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f084015efc0 msgr2=0x7f08400cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 --2- 172.21.15.6:0/3121472375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f084015efc0 0x7f08400cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 -- 172.21.15.6:0/3121472375 >> v2:172.21.15.178:3300/0 conn(0x7f0840162000 msgr2=0x7f08400ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 --2- 172.21.15.6:0/3121472375 >> v2:172.21.15.178:3300/0 conn(0x7f0840162000 0x7f08400ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.013+0000 7f0835e00640 1 -- 172.21.15.6:0/3121472375 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084006c860 con 0x7f08400cce90 2024-03-06T07:33:54.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.015+0000 7f0835e00640 1 --2- 172.21.15.6:0/3121472375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08400cce90 0x7f0840161ac0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f08400b4bb0 tx=0x7f082802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dce8ad48ec0e9dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.015+0000 7f083de00640 1 -- 172.21.15.6:0/3121472375 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828035020 con 0x7f08400cce90 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.015+0000 7f083de00640 1 -- 172.21.15.6:0/3121472375 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0828031870 con 0x7f08400cce90 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.015+0000 7f083de00640 1 -- 172.21.15.6:0/3121472375 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828031c40 con 0x7f08400cce90 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 -- 172.21.15.6:0/3121472375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08400cce90 msgr2=0x7f0840161ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 --2- 172.21.15.6:0/3121472375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08400cce90 0x7f0840161ac0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f08400b4bb0 tx=0x7f082802b910 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 -- 172.21.15.6:0/3121472375 shutdown_connections 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 --2- 172.21.15.6:0/3121472375 >> v2:172.21.15.178:3300/0 conn(0x7f0840162000 0x7f08400ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 --2- 172.21.15.6:0/3121472375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f08400cce90 0x7f0840161ac0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 --2- 172.21.15.6:0/3121472375 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f084015efc0 0x7f08400cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 -- 172.21.15.6:0/3121472375 >> 172.21.15.6:0/3121472375 conn(0x7f084006b3d0 msgr2=0x7f084006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 -- 172.21.15.6:0/3121472375 shutdown_connections 2024-03-06T07:33:54.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.016+0000 7f0844a00640 1 -- 172.21.15.6:0/3121472375 wait complete. 2024-03-06T07:33:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.018+0000 7f0844a00640 1 Processor -- start 2024-03-06T07:33:54.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.018+0000 7f0844a00640 1 -- start start 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0844a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f08400cce90 0x7f08401e8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0844a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f084015efc0 0x7f0840144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0844a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 0x7f08400b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0844a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840162490 con 0x7f0840162000 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0844a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084006ce20 con 0x7f084015efc0 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0844a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084006d620 con 0x7f08400cce90 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f083f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 0x7f08400b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f083f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 0x7f08400b3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49500/0 (socket says 172.21.15.6:49500) 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f083f200640 1 -- 172.21.15.6:0/2889510827 learned_addr learned my addr 172.21.15.6:0/2889510827 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f083e800640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f08400cce90 0x7f08401e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.019+0000 7f0835e00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f084015efc0 0x7f0840144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f083f200640 1 -- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f08400cce90 msgr2=0x7f08401e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f083f200640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f08400cce90 0x7f08401e8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f083f200640 1 -- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f084015efc0 msgr2=0x7f0840144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f083f200640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f084015efc0 0x7f0840144340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f083f200640 1 -- 172.21.15.6:0/2889510827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08401432a0 con 0x7f0840162000 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f083f200640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 0x7f08400b3990 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f083800d5d0 tx=0x7f083800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380125f0 con 0x7f0840162000 2024-03-06T07:33:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0838019070 con 0x7f0840162000 2024-03-06T07:33:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08380129f0 con 0x7f0840162000 2024-03-06T07:33:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.020+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0840143580 con 0x7f0840162000 2024-03-06T07:33:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.021+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0840143b40 con 0x7f0840162000 2024-03-06T07:33:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.022+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08400cf150 con 0x7f0840162000 2024-03-06T07:33:54.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.022+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0838012b90 con 0x7f0840162000 2024-03-06T07:33:54.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.023+0000 7f0837400640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.023+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f083808a5b0 con 0x7f0840162000 2024-03-06T07:33:54.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.024+0000 7f083e800640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.025+0000 7f083e800640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f081c06b4d0 0x7f081c06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f08401467c0 tx=0x7f08300039b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.028+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08400cf150 con 0x7f0840162000 2024-03-06T07:33:54.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.227+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0840144880 con 0x7f081c06b4d0 2024-03-06T07:33:54.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.245+0000 7f0837400640 1 -- 172.21.15.6:0/2889510827 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252482 (secure 0 0 0) 0x7f0830004780 con 0x7f081c06b4d0 2024-03-06T07:33:54.249 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:54.252 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:33:54.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f081c06b4d0 msgr2=0x7f081c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f081c06b4d0 0x7f081c06d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f08401467c0 tx=0x7f08300039b0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 msgr2=0x7f08400b3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 0x7f08400b3990 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f083800d5d0 tx=0x7f083800da90 comp rx=0 tx=0).stop 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 shutdown_connections 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f084015efc0 0x7f0840144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0840162000 0x7f08400b3990 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 --2- 172.21.15.6:0/2889510827 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f08400cce90 0x7f08401e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 >> 172.21.15.6:0/2889510827 conn(0x7f084006b3d0 msgr2=0x7f0840074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 shutdown_connections 2024-03-06T07:33:54.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.251+0000 7f0844a00640 1 -- 172.21.15.6:0/2889510827 wait complete. 2024-03-06T07:33:54.281 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-03-06T07:33:53.578658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":160,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":311048,"kb_used_data":11656,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754663672,"statfs":{"total":773094113280,"available":772775600128,"internally_reserved":0,"allocated":11935744,"data_stored":5776892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002186"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004815+0000","last_change":"2024-03-06T07:33:49.020318+0000","last_active":"2024-03-06T07:33:50.004815+0000","last_peered":"2024-03-06T07:33:50.004815+0000","last_clean":"2024-03-06T07:33:50.004815+0000","last_became_active":"2024-03-06T07:33:49.020020+0000","last_became_peered":"2024-03-06T07:33:49.020020+0000","last_unstale":"2024-03-06T07:33:50.004815+0000","last_undegraded":"2024-03-06T07:33:50.004815+0000","last_fullsized":"2024-03-06T07:33:50.004815+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:19.945099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009186+0000","last_change":"2024-03-06T07:33:49.020369+0000","last_active":"2024-03-06T07:33:50.009186+0000","last_peered":"2024-03-06T07:33:50.009186+0000","last_clean":"2024-03-06T07:33:50.009186+0000","last_became_active":"2024-03-06T07:33:49.018641+0000","last_became_peered":"2024-03-06T07:33:49.018641+0000","last_unstale":"2024-03-06T07:33:50.009186+0000","last_undegraded":"2024-03-06T07:33:50.009186+0000","last_fullsized":"2024-03-06T07:33:50.009186+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.585393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015913+0000","last_change":"2024-03-06T07:33:49.015358+0000","last_active":"2024-03-06T07:33:50.015913+0000","last_peered":"2024-03-06T07:33:50.015913+0000","last_clean":"2024-03-06T07:33:50.015913+0000","last_became_active":"2024-03-06T07:33:49.010587+0000","last_became_peered":"2024-03-06T07:33:49.010587+0000","last_unstale":"2024-03-06T07:33:50.015913+0000","last_undegraded":"2024-03-06T07:33:50.015913+0000","last_fullsized":"2024-03-06T07:33:50.015913+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:07.585393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003713+0000","last_change":"2024-03-06T07:33:49.055172+0000","last_active":"2024-03-06T07:33:50.003713+0000","last_peered":"2024-03-06T07:33:50.003713+0000","last_clean":"2024-03-06T07:33:50.003713+0000","last_became_active":"2024-03-06T07:33:49.054792+0000","last_became_peered":"2024-03-06T07:33:49.054792+0000","last_unstale":"2024-03-06T07:33:50.003713+0000","last_undegraded":"2024-03-06T07:33:50.003713+0000","last_fullsized":"2024-03-06T07:33:50.003713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:35.668313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013612+0000","last_change":"2024-03-06T07:33:49.010420+0000","last_active":"2024-03-06T07:33:50.013612+0000","last_peered":"2024-03-06T07:33:50.013612+0000","last_clean":"2024-03-06T07:33:50.013612+0000","last_became_active":"2024-03-06T07:33:49.009352+0000","last_became_peered":"2024-03-06T07:33:49.009352+0000","last_unstale":"2024-03-06T07:33:50.013612+0000","last_undegraded":"2024-03-06T07:33:50.013612+0000","last_fullsized":"2024-03-06T07:33:50.013612+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:55.090732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014891+0000","last_change":"2024-03-06T07:33:48.995368+0000","last_active":"2024-03-06T07:33:50.014891+0000","last_peered":"2024-03-06T07:33:50.014891+0000","last_clean":"2024-03-06T07:33:50.014891+0000","last_became_active":"2024-03-06T07:33:48.995046+0000","last_became_peered":"2024-03-06T07:33:48.995046+0000","last_unstale":"2024-03-06T07:33:50.014891+0000","last_undegraded":"2024-03-06T07:33:50.014891+0000","last_fullsized":"2024-03-06T07:33:50.014891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:59.038657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015807+0000","last_change":"2024-03-06T07:33:49.014249+0000","last_active":"2024-03-06T07:33:50.015807+0000","last_peered":"2024-03-06T07:33:50.015807+0000","last_clean":"2024-03-06T07:33:50.015807+0000","last_became_active":"2024-03-06T07:33:49.013284+0000","last_became_peered":"2024-03-06T07:33:49.013284+0000","last_unstale":"2024-03-06T07:33:50.015807+0000","last_undegraded":"2024-03-06T07:33:50.015807+0000","last_fullsized":"2024-03-06T07:33:50.015807+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:45.943465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014287+0000","last_change":"2024-03-06T07:33:49.004260+0000","last_active":"2024-03-06T07:33:50.014287+0000","last_peered":"2024-03-06T07:33:50.014287+0000","last_clean":"2024-03-06T07:33:50.014287+0000","last_became_active":"2024-03-06T07:33:49.003977+0000","last_became_peered":"2024-03-06T07:33:49.003977+0000","last_unstale":"2024-03-06T07:33:50.014287+0000","last_undegraded":"2024-03-06T07:33:50.014287+0000","last_fullsized":"2024-03-06T07:33:50.014287+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:20.617077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004847+0000","last_change":"2024-03-06T07:33:49.007697+0000","last_active":"2024-03-06T07:33:50.004847+0000","last_peered":"2024-03-06T07:33:50.004847+0000","last_clean":"2024-03-06T07:33:50.004847+0000","last_became_active":"2024-03-06T07:33:49.007394+0000","last_became_peered":"2024-03-06T07:33:49.007394+0000","last_unstale":"2024-03-06T07:33:50.004847+0000","last_undegraded":"2024-03-06T07:33:50.004847+0000","last_fullsized":"2024-03-06T07:33:50.004847+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:35:59.162236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004590+0000","last_change":"2024-03-06T07:33:49.016221+0000","last_active":"2024-03-06T07:33:50.004590+0000","last_peered":"2024-03-06T07:33:50.004590+0000","last_clean":"2024-03-06T07:33:50.004590+0000","last_became_active":"2024-03-06T07:33:49.015749+0000","last_became_peered":"2024-03-06T07:33:49.015749+0000","last_unstale":"2024-03-06T07:33:50.004590+0000","last_undegraded":"2024-03-06T07:33:50.004590+0000","last_fullsized":"2024-03-06T07:33:50.004590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:50:52.486280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009709+0000","last_change":"2024-03-06T07:33:49.019180+0000","last_active":"2024-03-06T07:33:50.009709+0000","last_peered":"2024-03-06T07:33:50.009709+0000","last_clean":"2024-03-06T07:33:50.009709+0000","last_became_active":"2024-03-06T07:33:49.018805+0000","last_became_peered":"2024-03-06T07:33:49.018805+0000","last_unstale":"2024-03-06T07:33:50.009709+0000","last_undegraded":"2024-03-06T07:33:50.009709+0000","last_fullsized":"2024-03-06T07:33:50.009709+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:44.767885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004136+0000","last_change":"2024-03-06T07:33:49.008352+0000","last_active":"2024-03-06T07:33:50.004136+0000","last_peered":"2024-03-06T07:33:50.004136+0000","last_clean":"2024-03-06T07:33:50.004136+0000","last_became_active":"2024-03-06T07:33:49.005977+0000","last_became_peered":"2024-03-06T07:33:49.005977+0000","last_unstale":"2024-03-06T07:33:50.004136+0000","last_undegraded":"2024-03-06T07:33:50.004136+0000","last_fullsized":"2024-03-06T07:33:50.004136+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:04.492848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015641+0000","last_change":"2024-03-06T07:33:49.002349+0000","last_active":"2024-03-06T07:33:50.015641+0000","last_peered":"2024-03-06T07:33:50.015641+0000","last_clean":"2024-03-06T07:33:50.015641+0000","last_became_active":"2024-03-06T07:33:49.002038+0000","last_became_peered":"2024-03-06T07:33:49.002038+0000","last_unstale":"2024-03-06T07:33:50.015641+0000","last_undegraded":"2024-03-06T07:33:50.015641+0000","last_fullsized":"2024-03-06T07:33:50.015641+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:44.767885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014443+0000","last_change":"2024-03-06T07:33:49.010804+0000","last_active":"2024-03-06T07:33:50.014443+0000","last_peered":"2024-03-06T07:33:50.014443+0000","last_clean":"2024-03-06T07:33:50.014443+0000","last_became_active":"2024-03-06T07:33:49.010506+0000","last_became_peered":"2024-03-06T07:33:49.010506+0000","last_unstale":"2024-03-06T07:33:50.014443+0000","last_undegraded":"2024-03-06T07:33:50.014443+0000","last_fullsized":"2024-03-06T07:33:50.014443+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:32.529908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014561+0000","last_change":"2024-03-06T07:33:48.993683+0000","last_active":"2024-03-06T07:33:50.014561+0000","last_peered":"2024-03-06T07:33:50.014561+0000","last_clean":"2024-03-06T07:33:50.014561+0000","last_became_active":"2024-03-06T07:33:48.992983+0000","last_became_peered":"2024-03-06T07:33:48.992983+0000","last_unstale":"2024-03-06T07:33:50.014561+0000","last_undegraded":"2024-03-06T07:33:50.014561+0000","last_fullsized":"2024-03-06T07:33:50.014561+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:50.074022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003196+0000","last_change":"2024-03-06T07:33:49.016391+0000","last_active":"2024-03-06T07:33:50.003196+0000","last_peered":"2024-03-06T07:33:50.003196+0000","last_clean":"2024-03-06T07:33:50.003196+0000","last_became_active":"2024-03-06T07:33:49.008654+0000","last_became_peered":"2024-03-06T07:33:49.008654+0000","last_unstale":"2024-03-06T07:33:50.003196+0000","last_undegraded":"2024-03-06T07:33:50.003196+0000","last_fullsized":"2024-03-06T07:33:50.003196+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:31.402014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008895+0000","last_change":"2024-03-06T07:33:49.018884+0000","last_active":"2024-03-06T07:33:50.008895+0000","last_peered":"2024-03-06T07:33:50.008895+0000","last_clean":"2024-03-06T07:33:50.008895+0000","last_became_active":"2024-03-06T07:33:49.018513+0000","last_became_peered":"2024-03-06T07:33:49.018513+0000","last_unstale":"2024-03-06T07:33:50.008895+0000","last_undegraded":"2024-03-06T07:33:50.008895+0000","last_fullsized":"2024-03-06T07:33:50.008895+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:56.573442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011361+0000","last_change":"2024-03-06T07:33:49.012081+0000","last_active":"2024-03-06T07:33:50.011361+0000","last_peered":"2024-03-06T07:33:50.011361+0000","last_clean":"2024-03-06T07:33:50.011361+0000","last_became_active":"2024-03-06T07:33:49.011736+0000","last_became_peered":"2024-03-06T07:33:49.011736+0000","last_unstale":"2024-03-06T07:33:50.011361+0000","last_undegraded":"2024-03-06T07:33:50.011361+0000","last_fullsized":"2024-03-06T07:33:50.011361+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:52.177813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016225+0000","last_change":"2024-03-06T07:33:49.015995+0000","last_active":"2024-03-06T07:33:50.016225+0000","last_peered":"2024-03-06T07:33:50.016225+0000","last_clean":"2024-03-06T07:33:50.016225+0000","last_became_active":"2024-03-06T07:33:49.015072+0000","last_became_peered":"2024-03-06T07:33:49.015072+0000","last_unstale":"2024-03-06T07:33:50.016225+0000","last_undegraded":"2024-03-06T07:33:50.016225+0000","last_fullsized":"2024-03-06T07:33:50.016225+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:07.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009656+0000","last_change":"2024-03-06T07:33:49.009221+0000","last_active":"2024-03-06T07:33:50.009656+0000","last_peered":"2024-03-06T07:33:50.009656+0000","last_clean":"2024-03-06T07:33:50.009656+0000","last_became_active":"2024-03-06T07:33:49.008939+0000","last_became_peered":"2024-03-06T07:33:49.008939+0000","last_unstale":"2024-03-06T07:33:50.009656+0000","last_undegraded":"2024-03-06T07:33:50.009656+0000","last_fullsized":"2024-03-06T07:33:50.009656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:36.385628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009201+0000","last_change":"2024-03-06T07:33:49.022515+0000","last_active":"2024-03-06T07:33:50.009201+0000","last_peered":"2024-03-06T07:33:50.009201+0000","last_clean":"2024-03-06T07:33:50.009201+0000","last_became_active":"2024-03-06T07:33:49.021894+0000","last_became_peered":"2024-03-06T07:33:49.021894+0000","last_unstale":"2024-03-06T07:33:50.009201+0000","last_undegraded":"2024-03-06T07:33:50.009201+0000","last_fullsized":"2024-03-06T07:33:50.009201+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:07.573734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012448+0000","last_change":"2024-03-06T07:33:48.990851+0000","last_active":"2024-03-06T07:33:50.012448+0000","last_peered":"2024-03-06T07:33:50.012448+0000","last_clean":"2024-03-06T07:33:50.012448+0000","last_became_active":"2024-03-06T07:33:48.990512+0000","last_became_peered":"2024-03-06T07:33:48.990512+0000","last_unstale":"2024-03-06T07:33:50.012448+0000","last_undegraded":"2024-03-06T07:33:50.012448+0000","last_fullsized":"2024-03-06T07:33:50.012448+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:20:45.151748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007675+0000","last_change":"2024-03-06T07:33:49.008664+0000","last_active":"2024-03-06T07:33:50.007675+0000","last_peered":"2024-03-06T07:33:50.007675+0000","last_clean":"2024-03-06T07:33:50.007675+0000","last_became_active":"2024-03-06T07:33:49.008333+0000","last_became_peered":"2024-03-06T07:33:49.008333+0000","last_unstale":"2024-03-06T07:33:50.007675+0000","last_undegraded":"2024-03-06T07:33:50.007675+0000","last_fullsized":"2024-03-06T07:33:50.007675+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:56:57.867912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012970+0000","last_change":"2024-03-06T07:33:49.014087+0000","last_active":"2024-03-06T07:33:50.012970+0000","last_peered":"2024-03-06T07:33:50.012970+0000","last_clean":"2024-03-06T07:33:50.012970+0000","last_became_active":"2024-03-06T07:33:49.013709+0000","last_became_peered":"2024-03-06T07:33:49.013709+0000","last_unstale":"2024-03-06T07:33:50.012970+0000","last_undegraded":"2024-03-06T07:33:50.012970+0000","last_fullsized":"2024-03-06T07:33:50.012970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:05.545987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008406+0000","last_change":"2024-03-06T07:33:49.020112+0000","last_active":"2024-03-06T07:33:50.008406+0000","last_peered":"2024-03-06T07:33:50.008406+0000","last_clean":"2024-03-06T07:33:50.008406+0000","last_became_active":"2024-03-06T07:33:49.018460+0000","last_became_peered":"2024-03-06T07:33:49.018460+0000","last_unstale":"2024-03-06T07:33:50.008406+0000","last_undegraded":"2024-03-06T07:33:50.008406+0000","last_fullsized":"2024-03-06T07:33:50.008406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.744126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004438+0000","last_change":"2024-03-06T07:33:49.056192+0000","last_active":"2024-03-06T07:33:50.004438+0000","last_peered":"2024-03-06T07:33:50.004438+0000","last_clean":"2024-03-06T07:33:50.004438+0000","last_became_active":"2024-03-06T07:33:49.053993+0000","last_became_peered":"2024-03-06T07:33:49.053993+0000","last_unstale":"2024-03-06T07:33:50.004438+0000","last_undegraded":"2024-03-06T07:33:50.004438+0000","last_fullsized":"2024-03-06T07:33:50.004438+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:33.452860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015160+0000","last_change":"2024-03-06T07:33:49.010237+0000","last_active":"2024-03-06T07:33:50.015160+0000","last_peered":"2024-03-06T07:33:50.015160+0000","last_clean":"2024-03-06T07:33:50.015160+0000","last_became_active":"2024-03-06T07:33:49.009505+0000","last_became_peered":"2024-03-06T07:33:49.009505+0000","last_unstale":"2024-03-06T07:33:50.015160+0000","last_undegraded":"2024-03-06T07:33:50.015160+0000","last_fullsized":"2024-03-06T07:33:50.015160+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:58.958944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012191+0000","last_change":"2024-03-06T07:33:49.012052+0000","last_active":"2024-03-06T07:33:50.012191+0000","last_peered":"2024-03-06T07:33:50.012191+0000","last_clean":"2024-03-06T07:33:50.012191+0000","last_became_active":"2024-03-06T07:33:49.011771+0000","last_became_peered":"2024-03-06T07:33:49.011771+0000","last_unstale":"2024-03-06T07:33:50.012191+0000","last_undegraded":"2024-03-06T07:33:50.012191+0000","last_fullsized":"2024-03-06T07:33:50.012191+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:37.878950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007926+0000","last_change":"2024-03-06T07:33:49.020611+0000","last_active":"2024-03-06T07:33:50.007926+0000","last_peered":"2024-03-06T07:33:50.007926+0000","last_clean":"2024-03-06T07:33:50.007926+0000","last_became_active":"2024-03-06T07:33:49.019780+0000","last_became_peered":"2024-03-06T07:33:49.019780+0000","last_unstale":"2024-03-06T07:33:50.007926+0000","last_undegraded":"2024-03-06T07:33:50.007926+0000","last_fullsized":"2024-03-06T07:33:50.007926+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:07:20.617077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015569+0000","last_change":"2024-03-06T07:33:49.025501+0000","last_active":"2024-03-06T07:33:50.015569+0000","last_peered":"2024-03-06T07:33:50.015569+0000","last_clean":"2024-03-06T07:33:50.015569+0000","last_became_active":"2024-03-06T07:33:49.024990+0000","last_became_peered":"2024-03-06T07:33:49.024990+0000","last_unstale":"2024-03-06T07:33:50.015569+0000","last_undegraded":"2024-03-06T07:33:50.015569+0000","last_fullsized":"2024-03-06T07:33:50.015569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:13.905043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004037+0000","last_change":"2024-03-06T07:33:49.015786+0000","last_active":"2024-03-06T07:33:50.004037+0000","last_peered":"2024-03-06T07:33:50.004037+0000","last_clean":"2024-03-06T07:33:50.004037+0000","last_became_active":"2024-03-06T07:33:49.015522+0000","last_became_peered":"2024-03-06T07:33:49.015522+0000","last_unstale":"2024-03-06T07:33:50.004037+0000","last_undegraded":"2024-03-06T07:33:50.004037+0000","last_fullsized":"2024-03-06T07:33:50.004037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:21.060300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014607+0000","last_change":"2024-03-06T07:33:48.995977+0000","last_active":"2024-03-06T07:33:50.014607+0000","last_peered":"2024-03-06T07:33:50.014607+0000","last_clean":"2024-03-06T07:33:50.014607+0000","last_became_active":"2024-03-06T07:33:48.993196+0000","last_became_peered":"2024-03-06T07:33:48.993196+0000","last_unstale":"2024-03-06T07:33:50.014607+0000","last_undegraded":"2024-03-06T07:33:50.014607+0000","last_fullsized":"2024-03-06T07:33:50.014607+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:11.990488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015443+0000","last_change":"2024-03-06T07:33:48.994605+0000","last_active":"2024-03-06T07:33:50.015443+0000","last_peered":"2024-03-06T07:33:50.015443+0000","last_clean":"2024-03-06T07:33:50.015443+0000","last_became_active":"2024-03-06T07:33:48.994104+0000","last_became_peered":"2024-03-06T07:33:48.994104+0000","last_unstale":"2024-03-06T07:33:50.015443+0000","last_undegraded":"2024-03-06T07:33:50.015443+0000","last_fullsized":"2024-03-06T07:33:50.015443+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:10.752842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016284+0000","last_change":"2024-03-06T07:33:47.991600+0000","last_active":"2024-03-06T07:33:50.016284+0000","last_peered":"2024-03-06T07:33:50.016284+0000","last_clean":"2024-03-06T07:33:50.016284+0000","last_became_active":"2024-03-06T07:33:47.991111+0000","last_became_peered":"2024-03-06T07:33:47.991111+0000","last_unstale":"2024-03-06T07:33:50.016284+0000","last_undegraded":"2024-03-06T07:33:50.016284+0000","last_fullsized":"2024-03-06T07:33:50.016284+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:47:46.528329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011665+0000","last_change":"2024-03-06T07:33:49.005266+0000","last_active":"2024-03-06T07:33:50.011665+0000","last_peered":"2024-03-06T07:33:50.011665+0000","last_clean":"2024-03-06T07:33:50.011665+0000","last_became_active":"2024-03-06T07:33:49.004067+0000","last_became_peered":"2024-03-06T07:33:49.004067+0000","last_unstale":"2024-03-06T07:33:50.011665+0000","last_undegraded":"2024-03-06T07:33:50.011665+0000","last_fullsized":"2024-03-06T07:33:50.011665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:01.581000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016074+0000","last_change":"2024-03-06T07:33:47.994856+0000","last_active":"2024-03-06T07:33:50.016074+0000","last_peered":"2024-03-06T07:33:50.016074+0000","last_clean":"2024-03-06T07:33:50.016074+0000","last_became_active":"2024-03-06T07:33:47.994544+0000","last_became_peered":"2024-03-06T07:33:47.994544+0000","last_unstale":"2024-03-06T07:33:50.016074+0000","last_undegraded":"2024-03-06T07:33:50.016074+0000","last_fullsized":"2024-03-06T07:33:50.016074+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:49:11.807540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015091+0000","last_change":"2024-03-06T07:33:48.995351+0000","last_active":"2024-03-06T07:33:50.015091+0000","last_peered":"2024-03-06T07:33:50.015091+0000","last_clean":"2024-03-06T07:33:50.015091+0000","last_became_active":"2024-03-06T07:33:48.994782+0000","last_became_peered":"2024-03-06T07:33:48.994782+0000","last_unstale":"2024-03-06T07:33:50.015091+0000","last_undegraded":"2024-03-06T07:33:50.015091+0000","last_fullsized":"2024-03-06T07:33:50.015091+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:04:12.331929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013485+0000","last_change":"2024-03-06T07:33:47.988680+0000","last_active":"2024-03-06T07:33:50.013485+0000","last_peered":"2024-03-06T07:33:50.013485+0000","last_clean":"2024-03-06T07:33:50.013485+0000","last_became_active":"2024-03-06T07:33:47.987739+0000","last_became_peered":"2024-03-06T07:33:47.987739+0000","last_unstale":"2024-03-06T07:33:50.013485+0000","last_undegraded":"2024-03-06T07:33:50.013485+0000","last_fullsized":"2024-03-06T07:33:50.013485+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:59.128371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016263+0000","last_change":"2024-03-06T07:33:49.016229+0000","last_active":"2024-03-06T07:33:50.016263+0000","last_peered":"2024-03-06T07:33:50.016263+0000","last_clean":"2024-03-06T07:33:50.016263+0000","last_became_active":"2024-03-06T07:33:49.015698+0000","last_became_peered":"2024-03-06T07:33:49.015698+0000","last_unstale":"2024-03-06T07:33:50.016263+0000","last_undegraded":"2024-03-06T07:33:50.016263+0000","last_fullsized":"2024-03-06T07:33:50.016263+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:36.385628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013264+0000","last_change":"2024-03-06T07:33:47.988379+0000","last_active":"2024-03-06T07:33:50.013264+0000","last_peered":"2024-03-06T07:33:50.013264+0000","last_clean":"2024-03-06T07:33:50.013264+0000","last_became_active":"2024-03-06T07:33:47.988115+0000","last_became_peered":"2024-03-06T07:33:47.988115+0000","last_unstale":"2024-03-06T07:33:50.013264+0000","last_undegraded":"2024-03-06T07:33:50.013264+0000","last_fullsized":"2024-03-06T07:33:50.013264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:39.631996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003754+0000","last_change":"2024-03-06T07:33:49.054578+0000","last_active":"2024-03-06T07:33:50.003754+0000","last_peered":"2024-03-06T07:33:50.003754+0000","last_clean":"2024-03-06T07:33:50.003754+0000","last_became_active":"2024-03-06T07:33:49.054237+0000","last_became_peered":"2024-03-06T07:33:49.054237+0000","last_unstale":"2024-03-06T07:33:50.003754+0000","last_undegraded":"2024-03-06T07:33:50.003754+0000","last_fullsized":"2024-03-06T07:33:50.003754+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:41:00.041413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015464+0000","last_change":"2024-03-06T07:33:47.988192+0000","last_active":"2024-03-06T07:33:50.015464+0000","last_peered":"2024-03-06T07:33:50.015464+0000","last_clean":"2024-03-06T07:33:50.015464+0000","last_became_active":"2024-03-06T07:33:47.987831+0000","last_became_peered":"2024-03-06T07:33:47.987831+0000","last_unstale":"2024-03-06T07:33:50.015464+0000","last_undegraded":"2024-03-06T07:33:50.015464+0000","last_fullsized":"2024-03-06T07:33:50.015464+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:00.723758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008279+0000","last_change":"2024-03-06T07:33:49.013658+0000","last_active":"2024-03-06T07:33:50.008279+0000","last_peered":"2024-03-06T07:33:50.008279+0000","last_clean":"2024-03-06T07:33:50.008279+0000","last_became_active":"2024-03-06T07:33:49.013359+0000","last_became_peered":"2024-03-06T07:33:49.013359+0000","last_unstale":"2024-03-06T07:33:50.008279+0000","last_undegraded":"2024-03-06T07:33:50.008279+0000","last_fullsized":"2024-03-06T07:33:50.008279+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:23:32.529908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012772+0000","last_change":"2024-03-06T07:33:47.986784+0000","last_active":"2024-03-06T07:33:50.012772+0000","last_peered":"2024-03-06T07:33:50.012772+0000","last_clean":"2024-03-06T07:33:50.012772+0000","last_became_active":"2024-03-06T07:33:47.986447+0000","last_became_peered":"2024-03-06T07:33:47.986447+0000","last_unstale":"2024-03-06T07:33:50.012772+0000","last_undegraded":"2024-03-06T07:33:50.012772+0000","last_fullsized":"2024-03-06T07:33:50.012772+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:59.693081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003198+0000","last_change":"2024-03-06T07:33:49.008911+0000","last_active":"2024-03-06T07:33:50.003198+0000","last_peered":"2024-03-06T07:33:50.003198+0000","last_clean":"2024-03-06T07:33:50.003198+0000","last_became_active":"2024-03-06T07:33:49.006748+0000","last_became_peered":"2024-03-06T07:33:49.006748+0000","last_unstale":"2024-03-06T07:33:50.003198+0000","last_undegraded":"2024-03-06T07:33:50.003198+0000","last_fullsized":"2024-03-06T07:33:50.003198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:31.308345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007525+0000","last_change":"2024-03-06T07:33:47.985308+0000","last_active":"2024-03-06T07:33:50.007525+0000","last_peered":"2024-03-06T07:33:50.007525+0000","last_clean":"2024-03-06T07:33:50.007525+0000","last_became_active":"2024-03-06T07:33:47.984226+0000","last_became_peered":"2024-03-06T07:33:47.984226+0000","last_unstale":"2024-03-06T07:33:50.007525+0000","last_undegraded":"2024-03-06T07:33:50.007525+0000","last_fullsized":"2024-03-06T07:33:50.007525+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:25.494570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014665+0000","last_change":"2024-03-06T07:33:49.018940+0000","last_active":"2024-03-06T07:33:50.014665+0000","last_peered":"2024-03-06T07:33:50.014665+0000","last_clean":"2024-03-06T07:33:50.014665+0000","last_became_active":"2024-03-06T07:33:49.018645+0000","last_became_peered":"2024-03-06T07:33:49.018645+0000","last_unstale":"2024-03-06T07:33:50.014665+0000","last_undegraded":"2024-03-06T07:33:50.014665+0000","last_fullsized":"2024-03-06T07:33:50.014665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:32:17.902238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003401+0000","last_change":"2024-03-06T07:33:47.985494+0000","last_active":"2024-03-06T07:33:50.003401+0000","last_peered":"2024-03-06T07:33:50.003401+0000","last_clean":"2024-03-06T07:33:50.003401+0000","last_became_active":"2024-03-06T07:33:47.985232+0000","last_became_peered":"2024-03-06T07:33:47.985232+0000","last_unstale":"2024-03-06T07:33:50.003401+0000","last_undegraded":"2024-03-06T07:33:50.003401+0000","last_fullsized":"2024-03-06T07:33:50.003401+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:00.565493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016590+0000","last_change":"2024-03-06T07:33:49.014719+0000","last_active":"2024-03-06T07:33:50.016590+0000","last_peered":"2024-03-06T07:33:50.016590+0000","last_clean":"2024-03-06T07:33:50.016590+0000","last_became_active":"2024-03-06T07:33:49.014356+0000","last_became_peered":"2024-03-06T07:33:49.014356+0000","last_unstale":"2024-03-06T07:33:50.016590+0000","last_undegraded":"2024-03-06T07:33:50.016590+0000","last_fullsized":"2024-03-06T07:33:50.016590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:09.578502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013830+0000","last_change":"2024-03-06T07:33:47.986309+0000","last_active":"2024-03-06T07:33:50.013830+0000","last_peered":"2024-03-06T07:33:50.013830+0000","last_clean":"2024-03-06T07:33:50.013830+0000","last_became_active":"2024-03-06T07:33:47.986040+0000","last_became_peered":"2024-03-06T07:33:47.986040+0000","last_unstale":"2024-03-06T07:33:50.013830+0000","last_undegraded":"2024-03-06T07:33:50.013830+0000","last_fullsized":"2024-03-06T07:33:50.013830+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:23:36.359524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011038+0000","last_change":"2024-03-06T07:33:49.006283+0000","last_active":"2024-03-06T07:33:50.011038+0000","last_peered":"2024-03-06T07:33:50.011038+0000","last_clean":"2024-03-06T07:33:50.011038+0000","last_became_active":"2024-03-06T07:33:49.005896+0000","last_became_peered":"2024-03-06T07:33:49.005896+0000","last_unstale":"2024-03-06T07:33:50.011038+0000","last_undegraded":"2024-03-06T07:33:50.011038+0000","last_fullsized":"2024-03-06T07:33:50.011038+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:34.439233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011926+0000","last_change":"2024-03-06T07:33:47.991647+0000","last_active":"2024-03-06T07:33:50.011926+0000","last_peered":"2024-03-06T07:33:50.011926+0000","last_clean":"2024-03-06T07:33:50.011926+0000","last_became_active":"2024-03-06T07:33:47.991383+0000","last_became_peered":"2024-03-06T07:33:47.991383+0000","last_unstale":"2024-03-06T07:33:50.011926+0000","last_undegraded":"2024-03-06T07:33:50.011926+0000","last_fullsized":"2024-03-06T07:33:50.011926+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:23:15.664554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012352+0000","last_change":"2024-03-06T07:33:49.023569+0000","last_active":"2024-03-06T07:33:50.012352+0000","last_peered":"2024-03-06T07:33:50.012352+0000","last_clean":"2024-03-06T07:33:50.012352+0000","last_became_active":"2024-03-06T07:33:49.018365+0000","last_became_peered":"2024-03-06T07:33:49.018365+0000","last_unstale":"2024-03-06T07:33:50.012352+0000","last_undegraded":"2024-03-06T07:33:50.012352+0000","last_fullsized":"2024-03-06T07:33:50.012352+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:14.501377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":17,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004074+0000","last_change":"2024-03-06T07:33:47.982938+0000","last_active":"2024-03-06T07:33:50.004074+0000","last_peered":"2024-03-06T07:33:50.004074+0000","last_clean":"2024-03-06T07:33:50.004074+0000","last_became_active":"2024-03-06T07:33:47.982604+0000","last_became_peered":"2024-03-06T07:33:47.982604+0000","last_unstale":"2024-03-06T07:33:50.004074+0000","last_undegraded":"2024-03-06T07:33:50.004074+0000","last_fullsized":"2024-03-06T07:33:50.004074+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:10:50.759169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015383+0000","last_change":"2024-03-06T07:33:49.019238+0000","last_active":"2024-03-06T07:33:50.015383+0000","last_peered":"2024-03-06T07:33:50.015383+0000","last_clean":"2024-03-06T07:33:50.015383+0000","last_became_active":"2024-03-06T07:33:49.018964+0000","last_became_peered":"2024-03-06T07:33:49.018964+0000","last_unstale":"2024-03-06T07:33:50.015383+0000","last_undegraded":"2024-03-06T07:33:50.015383+0000","last_fullsized":"2024-03-06T07:33:50.015383+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:48:35.168326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004447+0000","last_change":"2024-03-06T07:33:47.984898+0000","last_active":"2024-03-06T07:33:50.004447+0000","last_peered":"2024-03-06T07:33:50.004447+0000","last_clean":"2024-03-06T07:33:50.004447+0000","last_became_active":"2024-03-06T07:33:47.984534+0000","last_became_peered":"2024-03-06T07:33:47.984534+0000","last_unstale":"2024-03-06T07:33:50.004447+0000","last_undegraded":"2024-03-06T07:33:50.004447+0000","last_fullsized":"2024-03-06T07:33:50.004447+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:43.071436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012643+0000","last_change":"2024-03-06T07:33:49.017384+0000","last_active":"2024-03-06T07:33:50.012643+0000","last_peered":"2024-03-06T07:33:50.012643+0000","last_clean":"2024-03-06T07:33:50.012643+0000","last_became_active":"2024-03-06T07:33:49.017100+0000","last_became_peered":"2024-03-06T07:33:49.017100+0000","last_unstale":"2024-03-06T07:33:50.012643+0000","last_undegraded":"2024-03-06T07:33:50.012643+0000","last_fullsized":"2024-03-06T07:33:50.012643+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:01.722730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014994+0000","last_change":"2024-03-06T07:33:47.988666+0000","last_active":"2024-03-06T07:33:50.014994+0000","last_peered":"2024-03-06T07:33:50.014994+0000","last_clean":"2024-03-06T07:33:50.014994+0000","last_became_active":"2024-03-06T07:33:47.987376+0000","last_became_peered":"2024-03-06T07:33:47.987376+0000","last_unstale":"2024-03-06T07:33:50.014994+0000","last_undegraded":"2024-03-06T07:33:50.014994+0000","last_fullsized":"2024-03-06T07:33:50.014994+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:06:54.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012732+0000","last_change":"2024-03-06T07:33:49.011247+0000","last_active":"2024-03-06T07:33:50.012732+0000","last_peered":"2024-03-06T07:33:50.012732+0000","last_clean":"2024-03-06T07:33:50.012732+0000","last_became_active":"2024-03-06T07:33:49.009708+0000","last_became_peered":"2024-03-06T07:33:49.009708+0000","last_unstale":"2024-03-06T07:33:50.012732+0000","last_undegraded":"2024-03-06T07:33:50.012732+0000","last_fullsized":"2024-03-06T07:33:50.012732+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:11.109818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.009459+0000","last_change":"2024-03-06T07:33:47.984467+0000","last_active":"2024-03-06T07:33:50.009459+0000","last_peered":"2024-03-06T07:33:50.009459+0000","last_clean":"2024-03-06T07:33:50.009459+0000","last_became_active":"2024-03-06T07:33:47.983476+0000","last_became_peered":"2024-03-06T07:33:47.983476+0000","last_unstale":"2024-03-06T07:33:50.009459+0000","last_undegraded":"2024-03-06T07:33:50.009459+0000","last_fullsized":"2024-03-06T07:33:50.009459+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:14.299326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012050+0000","last_change":"2024-03-06T07:33:49.017308+0000","last_active":"2024-03-06T07:33:50.012050+0000","last_peered":"2024-03-06T07:33:50.012050+0000","last_clean":"2024-03-06T07:33:50.012050+0000","last_became_active":"2024-03-06T07:33:49.016740+0000","last_became_peered":"2024-03-06T07:33:49.016740+0000","last_unstale":"2024-03-06T07:33:50.012050+0000","last_undegraded":"2024-03-06T07:33:50.012050+0000","last_fullsized":"2024-03-06T07:33:50.012050+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:38.646550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":22,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013087+0000","last_change":"2024-03-06T07:33:47.976997+0000","last_active":"2024-03-06T07:33:50.013087+0000","last_peered":"2024-03-06T07:33:50.013087+0000","last_clean":"2024-03-06T07:33:50.013087+0000","last_became_active":"2024-03-06T07:33:47.976654+0000","last_became_peered":"2024-03-06T07:33:47.976654+0000","last_unstale":"2024-03-06T07:33:50.013087+0000","last_undegraded":"2024-03-06T07:33:50.013087+0000","last_fullsized":"2024-03-06T07:33:50.013087+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:15.859152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013728+0000","last_change":"2024-03-06T07:33:49.005374+0000","last_active":"2024-03-06T07:33:50.013728+0000","last_peered":"2024-03-06T07:33:50.013728+0000","last_clean":"2024-03-06T07:33:50.013728+0000","last_became_active":"2024-03-06T07:33:49.005153+0000","last_became_peered":"2024-03-06T07:33:49.005153+0000","last_unstale":"2024-03-06T07:33:50.013728+0000","last_undegraded":"2024-03-06T07:33:50.013728+0000","last_fullsized":"2024-03-06T07:33:50.013728+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:09.151965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004931+0000","last_change":"2024-03-06T07:33:47.990909+0000","last_active":"2024-03-06T07:33:50.004931+0000","last_peered":"2024-03-06T07:33:50.004931+0000","last_clean":"2024-03-06T07:33:50.004931+0000","last_became_active":"2024-03-06T07:33:47.987760+0000","last_became_peered":"2024-03-06T07:33:47.987760+0000","last_unstale":"2024-03-06T07:33:50.004931+0000","last_undegraded":"2024-03-06T07:33:50.004931+0000","last_fullsized":"2024-03-06T07:33:50.004931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:36.863443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015476+0000","last_change":"2024-03-06T07:33:49.007482+0000","last_active":"2024-03-06T07:33:50.015476+0000","last_peered":"2024-03-06T07:33:50.015476+0000","last_clean":"2024-03-06T07:33:50.015476+0000","last_became_active":"2024-03-06T07:33:49.007206+0000","last_became_peered":"2024-03-06T07:33:49.007206+0000","last_unstale":"2024-03-06T07:33:50.015476+0000","last_undegraded":"2024-03-06T07:33:50.015476+0000","last_fullsized":"2024-03-06T07:33:50.015476+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:06:37.203499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013673+0000","last_change":"2024-03-06T07:33:47.989436+0000","last_active":"2024-03-06T07:33:50.013673+0000","last_peered":"2024-03-06T07:33:50.013673+0000","last_clean":"2024-03-06T07:33:50.013673+0000","last_became_active":"2024-03-06T07:33:47.989129+0000","last_became_peered":"2024-03-06T07:33:47.989129+0000","last_unstale":"2024-03-06T07:33:50.013673+0000","last_undegraded":"2024-03-06T07:33:50.013673+0000","last_fullsized":"2024-03-06T07:33:50.013673+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:43.654988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015827+0000","last_change":"2024-03-06T07:33:49.056390+0000","last_active":"2024-03-06T07:33:50.015827+0000","last_peered":"2024-03-06T07:33:50.015827+0000","last_clean":"2024-03-06T07:33:50.015827+0000","last_became_active":"2024-03-06T07:33:49.056062+0000","last_became_peered":"2024-03-06T07:33:49.056062+0000","last_unstale":"2024-03-06T07:33:50.015827+0000","last_undegraded":"2024-03-06T07:33:50.015827+0000","last_fullsized":"2024-03-06T07:33:50.015827+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:47.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.010696+0000","last_change":"2024-03-06T07:33:47.983724+0000","last_active":"2024-03-06T07:33:50.010696+0000","last_peered":"2024-03-06T07:33:50.010696+0000","last_clean":"2024-03-06T07:33:50.010696+0000","last_became_active":"2024-03-06T07:33:47.983154+0000","last_became_peered":"2024-03-06T07:33:47.983154+0000","last_unstale":"2024-03-06T07:33:50.010696+0000","last_undegraded":"2024-03-06T07:33:50.010696+0000","last_fullsized":"2024-03-06T07:33:50.010696+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:00.870059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.016631+0000","last_change":"2024-03-06T07:33:49.022820+0000","last_active":"2024-03-06T07:33:50.016631+0000","last_peered":"2024-03-06T07:33:50.016631+0000","last_clean":"2024-03-06T07:33:50.016631+0000","last_became_active":"2024-03-06T07:33:49.020120+0000","last_became_peered":"2024-03-06T07:33:49.020120+0000","last_unstale":"2024-03-06T07:33:50.016631+0000","last_undegraded":"2024-03-06T07:33:50.016631+0000","last_fullsized":"2024-03-06T07:33:50.016631+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:17.789144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015901+0000","last_change":"2024-03-06T07:33:47.989741+0000","last_active":"2024-03-06T07:33:50.015901+0000","last_peered":"2024-03-06T07:33:50.015901+0000","last_clean":"2024-03-06T07:33:50.015901+0000","last_became_active":"2024-03-06T07:33:47.988639+0000","last_became_peered":"2024-03-06T07:33:47.988639+0000","last_unstale":"2024-03-06T07:33:50.015901+0000","last_undegraded":"2024-03-06T07:33:50.015901+0000","last_fullsized":"2024-03-06T07:33:50.015901+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:56.579196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014656+0000","last_change":"2024-03-06T07:33:49.014635+0000","last_active":"2024-03-06T07:33:50.014656+0000","last_peered":"2024-03-06T07:33:50.014656+0000","last_clean":"2024-03-06T07:33:50.014656+0000","last_became_active":"2024-03-06T07:33:49.014314+0000","last_became_peered":"2024-03-06T07:33:49.014314+0000","last_unstale":"2024-03-06T07:33:50.014656+0000","last_undegraded":"2024-03-06T07:33:50.014656+0000","last_fullsized":"2024-03-06T07:33:50.014656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:22:22.744126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015182+0000","last_change":"2024-03-06T07:33:47.987544+0000","last_active":"2024-03-06T07:33:50.015182+0000","last_peered":"2024-03-06T07:33:50.015182+0000","last_clean":"2024-03-06T07:33:50.015182+0000","last_became_active":"2024-03-06T07:33:47.986492+0000","last_became_peered":"2024-03-06T07:33:47.986492+0000","last_unstale":"2024-03-06T07:33:50.015182+0000","last_undegraded":"2024-03-06T07:33:50.015182+0000","last_fullsized":"2024-03-06T07:33:50.015182+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:47:08.136458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003259+0000","last_change":"2024-03-06T07:33:49.006831+0000","last_active":"2024-03-06T07:33:50.003259+0000","last_peered":"2024-03-06T07:33:50.003259+0000","last_clean":"2024-03-06T07:33:50.003259+0000","last_became_active":"2024-03-06T07:33:49.006548+0000","last_became_peered":"2024-03-06T07:33:49.006548+0000","last_unstale":"2024-03-06T07:33:50.003259+0000","last_undegraded":"2024-03-06T07:33:50.003259+0000","last_fullsized":"2024-03-06T07:33:50.003259+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:26:04.859114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013205+0000","last_change":"2024-03-06T07:33:47.987714+0000","last_active":"2024-03-06T07:33:50.013205+0000","last_peered":"2024-03-06T07:33:50.013205+0000","last_clean":"2024-03-06T07:33:50.013205+0000","last_became_active":"2024-03-06T07:33:47.987418+0000","last_became_peered":"2024-03-06T07:33:47.987418+0000","last_unstale":"2024-03-06T07:33:50.013205+0000","last_undegraded":"2024-03-06T07:33:50.013205+0000","last_fullsized":"2024-03-06T07:33:50.013205+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:53.196457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014618+0000","last_change":"2024-03-06T07:33:49.012124+0000","last_active":"2024-03-06T07:33:50.014618+0000","last_peered":"2024-03-06T07:33:50.014618+0000","last_clean":"2024-03-06T07:33:50.014618+0000","last_became_active":"2024-03-06T07:33:49.010814+0000","last_became_peered":"2024-03-06T07:33:49.010814+0000","last_unstale":"2024-03-06T07:33:50.014618+0000","last_undegraded":"2024-03-06T07:33:50.014618+0000","last_fullsized":"2024-03-06T07:33:50.014618+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:33.676181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014632+0000","last_change":"2024-03-06T07:33:47.988212+0000","last_active":"2024-03-06T07:33:50.014632+0000","last_peered":"2024-03-06T07:33:50.014632+0000","last_clean":"2024-03-06T07:33:50.014632+0000","last_became_active":"2024-03-06T07:33:47.987770+0000","last_became_peered":"2024-03-06T07:33:47.987770+0000","last_unstale":"2024-03-06T07:33:50.014632+0000","last_undegraded":"2024-03-06T07:33:50.014632+0000","last_fullsized":"2024-03-06T07:33:50.014632+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:53:25.587698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012273+0000","last_change":"2024-03-06T07:33:49.007895+0000","last_active":"2024-03-06T07:33:50.012273+0000","last_peered":"2024-03-06T07:33:50.012273+0000","last_clean":"2024-03-06T07:33:50.012273+0000","last_became_active":"2024-03-06T07:33:49.007602+0000","last_became_peered":"2024-03-06T07:33:49.007602+0000","last_unstale":"2024-03-06T07:33:50.012273+0000","last_undegraded":"2024-03-06T07:33:50.012273+0000","last_fullsized":"2024-03-06T07:33:50.012273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:37.594251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004255+0000","last_change":"2024-03-06T07:33:47.990478+0000","last_active":"2024-03-06T07:33:50.004255+0000","last_peered":"2024-03-06T07:33:50.004255+0000","last_clean":"2024-03-06T07:33:50.004255+0000","last_became_active":"2024-03-06T07:33:47.990156+0000","last_became_peered":"2024-03-06T07:33:47.990156+0000","last_unstale":"2024-03-06T07:33:50.004255+0000","last_undegraded":"2024-03-06T07:33:50.004255+0000","last_fullsized":"2024-03-06T07:33:50.004255+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:33.423726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011531+0000","last_change":"2024-03-06T07:33:49.006209+0000","last_active":"2024-03-06T07:33:50.011531+0000","last_peered":"2024-03-06T07:33:50.011531+0000","last_clean":"2024-03-06T07:33:50.011531+0000","last_became_active":"2024-03-06T07:33:49.005513+0000","last_became_peered":"2024-03-06T07:33:49.005513+0000","last_unstale":"2024-03-06T07:33:50.011531+0000","last_undegraded":"2024-03-06T07:33:50.011531+0000","last_fullsized":"2024-03-06T07:33:50.011531+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:54.664883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012374+0000","last_change":"2024-03-06T07:33:47.981443+0000","last_active":"2024-03-06T07:33:50.012374+0000","last_peered":"2024-03-06T07:33:50.012374+0000","last_clean":"2024-03-06T07:33:50.012374+0000","last_became_active":"2024-03-06T07:33:47.981169+0000","last_became_peered":"2024-03-06T07:33:47.981169+0000","last_unstale":"2024-03-06T07:33:50.012374+0000","last_undegraded":"2024-03-06T07:33:50.012374+0000","last_fullsized":"2024-03-06T07:33:50.012374+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:03.132652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004965+0000","last_change":"2024-03-06T07:33:49.022092+0000","last_active":"2024-03-06T07:33:50.004965+0000","last_peered":"2024-03-06T07:33:50.004965+0000","last_clean":"2024-03-06T07:33:50.004965+0000","last_became_active":"2024-03-06T07:33:49.021531+0000","last_became_peered":"2024-03-06T07:33:49.021531+0000","last_unstale":"2024-03-06T07:33:50.004965+0000","last_undegraded":"2024-03-06T07:33:50.004965+0000","last_fullsized":"2024-03-06T07:33:50.004965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:59.204169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.011720+0000","last_change":"2024-03-06T07:33:47.988462+0000","last_active":"2024-03-06T07:33:50.011720+0000","last_peered":"2024-03-06T07:33:50.011720+0000","last_clean":"2024-03-06T07:33:50.011720+0000","last_became_active":"2024-03-06T07:33:47.988170+0000","last_became_peered":"2024-03-06T07:33:47.988170+0000","last_unstale":"2024-03-06T07:33:50.011720+0000","last_undegraded":"2024-03-06T07:33:50.011720+0000","last_fullsized":"2024-03-06T07:33:50.011720+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:45:59.693081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004495+0000","last_change":"2024-03-06T07:33:49.010220+0000","last_active":"2024-03-06T07:33:50.004495+0000","last_peered":"2024-03-06T07:33:50.004495+0000","last_clean":"2024-03-06T07:33:50.004495+0000","last_became_active":"2024-03-06T07:33:49.009891+0000","last_became_peered":"2024-03-06T07:33:49.009891+0000","last_unstale":"2024-03-06T07:33:50.004495+0000","last_undegraded":"2024-03-06T07:33:50.004495+0000","last_fullsized":"2024-03-06T07:33:50.004495+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:54.931396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004203+0000","last_change":"2024-03-06T07:33:47.989688+0000","last_active":"2024-03-06T07:33:50.004203+0000","last_peered":"2024-03-06T07:33:50.004203+0000","last_clean":"2024-03-06T07:33:50.004203+0000","last_became_active":"2024-03-06T07:33:47.989397+0000","last_became_peered":"2024-03-06T07:33:47.989397+0000","last_unstale":"2024-03-06T07:33:50.004203+0000","last_undegraded":"2024-03-06T07:33:50.004203+0000","last_fullsized":"2024-03-06T07:33:50.004203+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:53.649376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.010227+0000","last_change":"2024-03-06T07:33:49.012423+0000","last_active":"2024-03-06T07:33:50.010227+0000","last_peered":"2024-03-06T07:33:50.010227+0000","last_clean":"2024-03-06T07:33:50.010227+0000","last_became_active":"2024-03-06T07:33:49.012117+0000","last_became_peered":"2024-03-06T07:33:49.012117+0000","last_unstale":"2024-03-06T07:33:50.010227+0000","last_undegraded":"2024-03-06T07:33:50.010227+0000","last_fullsized":"2024-03-06T07:33:50.010227+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:38.385541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012452+0000","last_change":"2024-03-06T07:33:47.983772+0000","last_active":"2024-03-06T07:33:50.012452+0000","last_peered":"2024-03-06T07:33:50.012452+0000","last_clean":"2024-03-06T07:33:50.012452+0000","last_became_active":"2024-03-06T07:33:47.983340+0000","last_became_peered":"2024-03-06T07:33:47.983340+0000","last_unstale":"2024-03-06T07:33:50.012452+0000","last_undegraded":"2024-03-06T07:33:50.012452+0000","last_fullsized":"2024-03-06T07:33:50.012452+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:51.197766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.007616+0000","last_change":"2024-03-06T07:33:49.017001+0000","last_active":"2024-03-06T07:33:50.007616+0000","last_peered":"2024-03-06T07:33:50.007616+0000","last_clean":"2024-03-06T07:33:50.007616+0000","last_became_active":"2024-03-06T07:33:49.016453+0000","last_became_peered":"2024-03-06T07:33:49.016453+0000","last_unstale":"2024-03-06T07:33:50.007616+0000","last_undegraded":"2024-03-06T07:33:50.007616+0000","last_fullsized":"2024-03-06T07:33:50.007616+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:41:33.095797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015719+0000","last_change":"2024-03-06T07:33:47.991027+0000","last_active":"2024-03-06T07:33:50.015719+0000","last_peered":"2024-03-06T07:33:50.015719+0000","last_clean":"2024-03-06T07:33:50.015719+0000","last_became_active":"2024-03-06T07:33:47.990666+0000","last_became_peered":"2024-03-06T07:33:47.990666+0000","last_unstale":"2024-03-06T07:33:50.015719+0000","last_undegraded":"2024-03-06T07:33:50.015719+0000","last_fullsized":"2024-03-06T07:33:50.015719+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:40.015349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008689+0000","last_change":"2024-03-06T07:33:49.014461+0000","last_active":"2024-03-06T07:33:50.008689+0000","last_peered":"2024-03-06T07:33:50.008689+0000","last_clean":"2024-03-06T07:33:50.008689+0000","last_became_active":"2024-03-06T07:33:49.012167+0000","last_became_peered":"2024-03-06T07:33:49.012167+0000","last_unstale":"2024-03-06T07:33:50.008689+0000","last_undegraded":"2024-03-06T07:33:50.008689+0000","last_fullsized":"2024-03-06T07:33:50.008689+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:47.129082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.008755+0000","last_change":"2024-03-06T07:33:47.993010+0000","last_active":"2024-03-06T07:33:50.008755+0000","last_peered":"2024-03-06T07:33:50.008755+0000","last_clean":"2024-03-06T07:33:50.008755+0000","last_became_active":"2024-03-06T07:33:47.992686+0000","last_became_peered":"2024-03-06T07:33:47.992686+0000","last_unstale":"2024-03-06T07:33:50.008755+0000","last_undegraded":"2024-03-06T07:33:50.008755+0000","last_fullsized":"2024-03-06T07:33:50.008755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:43.804662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014734+0000","last_change":"2024-03-06T07:33:49.020253+0000","last_active":"2024-03-06T07:33:50.014734+0000","last_peered":"2024-03-06T07:33:50.014734+0000","last_clean":"2024-03-06T07:33:50.014734+0000","last_became_active":"2024-03-06T07:33:49.017442+0000","last_became_peered":"2024-03-06T07:33:49.017442+0000","last_unstale":"2024-03-06T07:33:50.014734+0000","last_undegraded":"2024-03-06T07:33:50.014734+0000","last_fullsized":"2024-03-06T07:33:50.014734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:48:56.573442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.013229+0000","last_change":"2024-03-06T07:33:47.985665+0000","last_active":"2024-03-06T07:33:50.013229+0000","last_peered":"2024-03-06T07:33:50.013229+0000","last_clean":"2024-03-06T07:33:50.013229+0000","last_became_active":"2024-03-06T07:33:47.985347+0000","last_became_peered":"2024-03-06T07:33:47.985347+0000","last_unstale":"2024-03-06T07:33:50.013229+0000","last_undegraded":"2024-03-06T07:33:50.013229+0000","last_fullsized":"2024-03-06T07:33:50.013229+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:58:01.117472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.004227+0000","last_change":"2024-03-06T07:33:49.005688+0000","last_active":"2024-03-06T07:33:50.004227+0000","last_peered":"2024-03-06T07:33:50.004227+0000","last_clean":"2024-03-06T07:33:50.004227+0000","last_became_active":"2024-03-06T07:33:49.004361+0000","last_became_peered":"2024-03-06T07:33:49.004361+0000","last_unstale":"2024-03-06T07:33:50.004227+0000","last_undegraded":"2024-03-06T07:33:50.004227+0000","last_fullsized":"2024-03-06T07:33:50.004227+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:38.852254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.003648+0000","last_change":"2024-03-06T07:33:47.989927+0000","last_active":"2024-03-06T07:33:50.003648+0000","last_peered":"2024-03-06T07:33:50.003648+0000","last_clean":"2024-03-06T07:33:50.003648+0000","last_became_active":"2024-03-06T07:33:47.989602+0000","last_became_peered":"2024-03-06T07:33:47.989602+0000","last_unstale":"2024-03-06T07:33:50.003648+0000","last_undegraded":"2024-03-06T07:33:50.003648+0000","last_fullsized":"2024-03-06T07:33:50.003648+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:51.162306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.012582+0000","last_change":"2024-03-06T07:33:49.023480+0000","last_active":"2024-03-06T07:33:50.012582+0000","last_peered":"2024-03-06T07:33:50.012582+0000","last_clean":"2024-03-06T07:33:50.012582+0000","last_became_active":"2024-03-06T07:33:49.023176+0000","last_became_peered":"2024-03-06T07:33:49.023176+0000","last_unstale":"2024-03-06T07:33:50.012582+0000","last_undegraded":"2024-03-06T07:33:50.012582+0000","last_fullsized":"2024-03-06T07:33:50.012582+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:47.946973+0000","last_clean_scrub_stamp":"2024-03-06T07:33:47.946973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:07.549515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.015267+0000","last_change":"2024-03-06T07:33:47.982237+0000","last_active":"2024-03-06T07:33:50.015267+0000","last_peered":"2024-03-06T07:33:50.015267+0000","last_clean":"2024-03-06T07:33:50.015267+0000","last_became_active":"2024-03-06T07:33:47.981853+0000","last_became_peered":"2024-03-06T07:33:47.981853+0000","last_unstale":"2024-03-06T07:33:50.015267+0000","last_undegraded":"2024-03-06T07:33:50.015267+0000","last_fullsized":"2024-03-06T07:33:50.015267+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:33:46.931466+0000","last_clean_scrub_stamp":"2024-03-06T07:33:46.931466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:09:21.031145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":325,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-06T07:33:50.014884+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:33:50.014884+0000","last_peered":"2024-03-06T07:33:50.014884+0000","last_clean":"2024-03-06T07:33:50.014884+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:33:50.014884+0000","last_undegraded":"2024-03-06T07:33:50.014884+0000","last_fullsized":"2024-03-06T07:33:50.014884+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509521,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1335296,"data_stored":573917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509522,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1343488,"data_stored":574041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542225,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41396,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330444,"statfs":{"total":96636764160,"available":96594374656,"internally_reserved":0,"allocated":1937408,"data_stored":1164901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542225,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40812,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331028,"statfs":{"total":96636764160,"available":96594972672,"internally_reserved":0,"allocated":1339392,"data_stored":573987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542226,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40816,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331024,"statfs":{"total":96636764160,"available":96594968576,"internally_reserved":0,"allocated":1343488,"data_stored":574521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542225,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1335296,"data_stored":573917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542226,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1351680,"data_stored":575599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542226,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37248,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334592,"statfs":{"total":96636764160,"available":96598622208,"internally_reserved":0,"allocated":1949696,"data_stored":1166009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:33:54.284 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:33:54.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:33:54.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.550+0000 7fb997e00640 1 Processor -- start 2024-03-06T07:33:54.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.550+0000 7fb997e00640 1 -- start start 2024-03-06T07:33:54.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb997e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb9980b4600 0x7fb9980b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb997e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 0x7fb998070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb997e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9980b2b80 con 0x7fb998144340 2024-03-06T07:33:54.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb997e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9980cc780 con 0x7fb9980b4600 2024-03-06T07:33:54.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb997e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980b3790 con 0x7fb998070a60 2024-03-06T07:33:54.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb996a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb998144340 0x7fb9980ce8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46910/0 (socket says 172.21.15.6:46910) 2024-03-06T07:33:54.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.553+0000 7fb996a00640 1 -- 172.21.15.6:0/3091226655 learned_addr learned my addr 172.21.15.6:0/3091226655 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661796176 0 0) 0x7fb9980b2b80 con 0x7fb998144340 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb997400640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 0x7fb998070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb97c003520 con 0x7fb998144340 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb996000640 1 --2- 172.21.15.6:0/3091226655 >> v2:172.21.15.178:3300/0 conn(0x7fb9980b4600 0x7fb9980b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837476786 0 0) 0x7fb97c003520 con 0x7fb998144340 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 msgr2=0x7fb998070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 0x7fb998070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 >> v2:172.21.15.178:3300/0 conn(0x7fb9980b4600 msgr2=0x7fb9980b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.554+0000 7fb995600640 1 --2- 172.21.15.6:0/3091226655 >> v2:172.21.15.178:3300/0 conn(0x7fb9980b4600 0x7fb9980b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9980b2e60 con 0x7fb998144340 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb988001670 con 0x7fb998144340 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb996000640 1 --2- 172.21.15.6:0/3091226655 >> v2:172.21.15.178:3300/0 conn(0x7fb9980b4600 0x7fb9980b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 >> v1:172.21.15.6:6789/0 conn(0x7fb998144340 legacy=0x7fb9980ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb97c0039b0 0x7fb97c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 0x7fb97c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb97c004d40 0x7fb97c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980b3790 con 0x7fb97c004d40 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c00ba70 con 0x7fb97c0039b0 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c00bc20 con 0x7fb998070a60 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb997400640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb97c004d40 0x7fb97c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb996a00640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb97c0039b0 0x7fb97c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.555+0000 7fb996000640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 0x7fb97c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.556+0000 7fb997400640 1 -- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 msgr2=0x7fb97c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.556+0000 7fb997400640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998070a60 0x7fb97c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.556+0000 7fb997400640 1 -- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb97c0039b0 msgr2=0x7fb97c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:54.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.556+0000 7fb997400640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb97c0039b0 0x7fb97c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.556+0000 7fb997400640 1 -- 172.21.15.6:0/3091226655 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c003520 con 0x7fb97c004d40 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.557+0000 7fb997400640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb97c004d40 0x7fb97c0051c0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb990009fc0 tx=0x7fb97c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a805f9f6a0a61e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.557+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990039070 con 0x7fb97c004d40 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.557+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99002cc20 con 0x7fb97c004d40 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.557+0000 7fb995600640 1 -- 172.21.15.6:0/3091226655 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990009c10 con 0x7fb97c004d40 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997e00640 1 -- 172.21.15.6:0/3091226655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb97c004d40 msgr2=0x7fb97c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997e00640 1 --2- 172.21.15.6:0/3091226655 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb97c004d40 0x7fb97c0051c0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fb990009fc0 tx=0x7fb97c005830 comp rx=0 tx=0).stop 2024-03-06T07:33:54.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997400640 1 -- 172.21.15.6:0/3091226655 reap_dead start 2024-03-06T07:33:54.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997e00640 1 -- 172.21.15.6:0/3091226655 shutdown_connections 2024-03-06T07:33:54.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997e00640 1 -- 172.21.15.6:0/3091226655 >> 172.21.15.6:0/3091226655 conn(0x7fb99806b190 msgr2=0x7fb99806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997e00640 1 -- 172.21.15.6:0/3091226655 shutdown_connections 2024-03-06T07:33:54.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.558+0000 7fb997e00640 1 -- 172.21.15.6:0/3091226655 wait complete. 2024-03-06T07:33:54.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.561+0000 7fb997e00640 1 Processor -- start 2024-03-06T07:33:54.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.561+0000 7fb997e00640 1 -- start start 2024-03-06T07:33:54.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb997e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb99815a270 0x7fb998141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb997e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 0x7fb9981654b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb997e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998142ba0 0x7fb9981659f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb997e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb998144720 con 0x7fb99815a650 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb997e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980b3420 con 0x7fb99815a270 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb997e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9980b3c20 con 0x7fb998142ba0 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb996000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 0x7fb9981654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb996000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 0x7fb9981654b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49528/0 (socket says 172.21.15.6:49528) 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb996000640 1 -- 172.21.15.6:0/2829290987 learned_addr learned my addr 172.21.15.6:0/2829290987 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.563+0000 7fb996a00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb99815a270 0x7fb998141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb997400640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998142ba0 0x7fb9981659f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb996000640 1 -- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998142ba0 msgr2=0x7fb9981659f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb996000640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998142ba0 0x7fb9981659f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb996000640 1 -- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb99815a270 msgr2=0x7fb998141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb996000640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb99815a270 0x7fb998141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb996000640 1 -- 172.21.15.6:0/2829290987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb998141b00 con 0x7fb99815a650 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb997400640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998142ba0 0x7fb9981659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb996000640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 0x7fb9981654b0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb9840107e0 tx=0x7fb984010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98401a020 con 0x7fb99815a650 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98400eb60 con 0x7fb99815a650 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98400f040 con 0x7fb99815a650 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb998141de0 con 0x7fb99815a650 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.564+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9981423a0 con 0x7fb99815a650 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.566+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99804f330 con 0x7fb99815a650 2024-03-06T07:33:54.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.567+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb98401d020 con 0x7fb99815a650 2024-03-06T07:33:54.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.568+0000 7fb98ea00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb97806b500 0x7fb97806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:54.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.568+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb98408abc0 con 0x7fb99815a650 2024-03-06T07:33:54.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.568+0000 7fb996a00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb97806b500 0x7fb97806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:54.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.569+0000 7fb996a00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb97806b500 0x7fb97806d9c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb988001050 tx=0x7fb988009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:54.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.577+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb984024020 con 0x7fb99815a650 2024-03-06T07:33:54.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.852+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb99813fd40 con 0x7fb99815a650 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.856+0000 7fb98ea00640 1 -- 172.21.15.6:0/2829290987 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb9840599a0 con 0x7fb99815a650 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:33:54.858 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb97806b500 msgr2=0x7fb97806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb97806b500 0x7fb97806d9c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb988001050 tx=0x7fb988009040 comp rx=0 tx=0).stop 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 msgr2=0x7fb9981654b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 0x7fb9981654b0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fb9840107e0 tx=0x7fb984010ca0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 shutdown_connections 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb97806b500 0x7fb97806d9c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb998142ba0 0x7fb9981659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb99815a650 0x7fb9981654b0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 --2- 172.21.15.6:0/2829290987 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb99815a270 0x7fb998141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.858+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 >> 172.21.15.6:0/2829290987 conn(0x7fb99806b190 msgr2=0x7fb998071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.859+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 shutdown_connections 2024-03-06T07:33:54.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:54.859+0000 7fb997e00640 1 -- 172.21.15.6:0/2829290987 wait complete. 2024-03-06T07:33:54.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:55.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.085+0000 7f8017e00640 1 Processor -- start 2024-03-06T07:33:55.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.086+0000 7f8017e00640 1 -- start start 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8017e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8018145300 0x7f80180b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8017e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f801815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8017e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80180714d0 con 0x7f8018070f10 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8017e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80180717b0 con 0x7f8018145300 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8017e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8018071a90 con 0x7f80180cd060 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8016a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8018145300 0x7f80180b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8016a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8018145300 0x7f80180b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49532/0 (socket says 172.21.15.6:49532) 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.087+0000 7f8016a00640 1 -- 172.21.15.6:0/3825715344 learned_addr learned my addr 172.21.15.6:0/3825715344 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.088+0000 7f8017400640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f801815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.088+0000 7f8016a00640 1 -- 172.21.15.6:0/3825715344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 msgr2=0x7f801815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.088+0000 7f8016a00640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f801815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.088+0000 7f8016a00640 1 -- 172.21.15.6:0/3825715344 >> v1:172.21.15.178:6789/0 conn(0x7f8018070f10 legacy=0x7f80180cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:55.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.088+0000 7f8016a00640 1 -- 172.21.15.6:0/3825715344 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8018071d70 con 0x7f8018145300 2024-03-06T07:33:55.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.088+0000 7f8017400640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f801815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.089+0000 7f8016a00640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018145300 0x7f80180b49a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f8008009870 tx=0x7f800802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa22d741023cc17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.089+0000 7f8016000640 1 -- 172.21.15.6:0/3825715344 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80080317c0 con 0x7f8018145300 2024-03-06T07:33:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.089+0000 7f8016000640 1 -- 172.21.15.6:0/3825715344 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8008037070 con 0x7f8018145300 2024-03-06T07:33:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.089+0000 7f8016000640 1 -- 172.21.15.6:0/3825715344 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008031b80 con 0x7f8018145300 2024-03-06T07:33:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 -- 172.21.15.6:0/3825715344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018145300 msgr2=0x7f80180b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018145300 0x7f80180b49a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f8008009870 tx=0x7f800802b860 comp rx=0 tx=0).stop 2024-03-06T07:33:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 -- 172.21.15.6:0/3825715344 shutdown_connections 2024-03-06T07:33:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f801815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 --2- 172.21.15.6:0/3825715344 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018145300 0x7f80180b49a0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 -- 172.21.15.6:0/3825715344 >> 172.21.15.6:0/3825715344 conn(0x7f801806b3d0 msgr2=0x7f801806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 -- 172.21.15.6:0/3825715344 shutdown_connections 2024-03-06T07:33:55.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.090+0000 7f8017e00640 1 -- 172.21.15.6:0/3825715344 wait complete. 2024-03-06T07:33:55.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.093+0000 7f8017e00640 1 Processor -- start 2024-03-06T07:33:55.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.093+0000 7f8017e00640 1 -- start start 2024-03-06T07:33:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 0x7f801815b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f80180c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8018145300 0x7f80180ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801815cab0 con 0x7f8018070f10 2024-03-06T07:33:55.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80181456e0 con 0x7f8018145300 2024-03-06T07:33:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f801815cc30 con 0x7f80180cd060 2024-03-06T07:33:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8016a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 0x7f801815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8016a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 0x7f801815b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49534/0 (socket says 172.21.15.6:49534) 2024-03-06T07:33:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8016a00640 1 -- 172.21.15.6:0/3826003020 learned_addr learned my addr 172.21.15.6:0/3826003020 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f800fe00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f80180c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.095+0000 7f8017400640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8018145300 0x7f80180ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f8016a00640 1 -- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 msgr2=0x7f80180c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f8016a00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f80180c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f8016a00640 1 -- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8018145300 msgr2=0x7f80180ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f8016a00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8018145300 0x7f80180ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f8016a00640 1 -- 172.21.15.6:0/3826003020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180c5bb0 con 0x7f8018070f10 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f800fe00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f80180c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f8016a00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 0x7f801815b430 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8008009870 tx=0x7f80080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008032860 con 0x7f8018070f10 2024-03-06T07:33:55.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.096+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8008032a00 con 0x7f8018070f10 2024-03-06T07:33:55.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.097+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008032df0 con 0x7f8018070f10 2024-03-06T07:33:55.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.097+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80180c5e90 con 0x7f8018070f10 2024-03-06T07:33:55.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.097+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80180c6450 con 0x7f8018070f10 2024-03-06T07:33:55.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.098+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801804f330 con 0x7f8018070f10 2024-03-06T07:33:55.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.099+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f800804c060 con 0x7f8018070f10 2024-03-06T07:33:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.100+0000 7f800ea00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:55.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.101+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80080abc10 con 0x7f8018070f10 2024-03-06T07:33:55.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.101+0000 7f800fe00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:55.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.102+0000 7f800fe00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f80181f6df0 tx=0x7f8000009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.105+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8008040070 con 0x7f8018070f10 2024-03-06T07:33:55.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.348+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80180ccac0 con 0x7f8018070f10 2024-03-06T07:33:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.349+0000 7f800ea00640 1 -- 172.21.15.6:0/3826003020 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f800807a4c0 con 0x7f8018070f10 2024-03-06T07:33:55.351 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:55.352 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5795,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3411419645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3411419645},{"type":"v1","addr":"172.21.15.6:6835","nonce":3411419645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5798,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/525628181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":525628181},{"type":"v1","addr":"172.21.15.178:6835","nonce":525628181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/319127923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":319127923},{"type":"v1","addr":"172.21.15.178:6837","nonce":319127923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:33:51.764337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7656},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7656":{"gid":7656,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6833/4114489444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4114489444},{"type":"v1","addr":"172.21.15.6:6833","nonce":4114489444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7656,"qdb_cluster":[7656]},"id":8}]} 2024-03-06T07:33:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ff406b4d0 msgr2=0x7f7ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f80181f6df0 tx=0x7f8000009040 comp rx=0 tx=0).stop 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 msgr2=0x7f801815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 0x7f801815b430 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f8008009870 tx=0x7f80080046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 shutdown_connections 2024-03-06T07:33:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8018145300 0x7f80180ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f80180cd060 0x7f80180c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 --2- 172.21.15.6:0/3826003020 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8018070f10 0x7f801815b430 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.352+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 >> 172.21.15.6:0/3826003020 conn(0x7f801806b3d0 msgr2=0x7f801815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.353+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 shutdown_connections 2024-03-06T07:33:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:55.353+0000 7f8017e00640 1 -- 172.21.15.6:0/3826003020 wait complete. 2024-03-06T07:33:55.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:48.441811+0000', 'modified': '2024-03-06T07:33:51.764337+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7656}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7656': {'gid': 7656, 'name': 'c', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6833/4114489444', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 4114489444}, {'type': 'v1', 'addr': '172.21.15.6:6833', 'nonce': 4114489444}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7656, 'qdb_cluster': [7656]} 2024-03-06T07:33:55.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:33:55.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:33:56.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.001+0000 7f502f600640 1 Processor -- start 2024-03-06T07:33:56.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.001+0000 7f502f600640 1 -- start start 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.002+0000 7f502f600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50280cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.002+0000 7f502f600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f50280cf5c0 0x7f50280b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.002+0000 7f502f600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f502806cb10 0x7f502806cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.002+0000 7f502f600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280ce160 con 0x7f502806cb10 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.002+0000 7f502f600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280b45b0 con 0x7f50280cf5c0 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.002+0000 7f502f600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50280b4860 con 0x7f50280b3c90 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f502806cb10 0x7f502806cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f502806cb10 0x7f502806cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49544/0 (socket says 172.21.15.6:49544) 2024-03-06T07:33:56.005 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 -- 172.21.15.6:0/2553805508 learned_addr learned my addr 172.21.15.6:0/2553805508 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502cc00640 1 --2- 172.21.15.6:0/2553805508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50280cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f501fe00640 1 --2- 172.21.15.6:0/2553805508 >> v2:172.21.15.178:3300/0 conn(0x7f50280cf5c0 0x7f50280b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 -- 172.21.15.6:0/2553805508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 msgr2=0x7f50280cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 --2- 172.21.15.6:0/2553805508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50280cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 -- 172.21.15.6:0/2553805508 >> v2:172.21.15.178:3300/0 conn(0x7f50280cf5c0 msgr2=0x7f50280b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 --2- 172.21.15.6:0/2553805508 >> v2:172.21.15.178:3300/0 conn(0x7f50280cf5c0 0x7f50280b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.003+0000 7f502d600640 1 -- 172.21.15.6:0/2553805508 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50280b4b40 con 0x7f502806cb10 2024-03-06T07:33:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.005+0000 7f502d600640 1 --2- 172.21.15.6:0/2553805508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f502806cf80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f502806c0e0 tx=0x7f502002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e88f2fd27eeb939 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.005+0000 7f5027e00640 1 -- 172.21.15.6:0/2553805508 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502002ed10 con 0x7f502806cb10 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.005+0000 7f5027e00640 1 -- 172.21.15.6:0/2553805508 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50200363f0 con 0x7f502806cb10 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.005+0000 7f5027e00640 1 -- 172.21.15.6:0/2553805508 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50200367c0 con 0x7f502806cb10 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 -- 172.21.15.6:0/2553805508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 msgr2=0x7f502806cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 --2- 172.21.15.6:0/2553805508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f502806cf80 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f502806c0e0 tx=0x7f502002e170 comp rx=0 tx=0).stop 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 -- 172.21.15.6:0/2553805508 shutdown_connections 2024-03-06T07:33:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 --2- 172.21.15.6:0/2553805508 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f502806cf80 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 --2- 172.21.15.6:0/2553805508 >> v2:172.21.15.178:3300/0 conn(0x7f50280cf5c0 0x7f50280b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 --2- 172.21.15.6:0/2553805508 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50280cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 -- 172.21.15.6:0/2553805508 >> 172.21.15.6:0/2553805508 conn(0x7f502806afb0 msgr2=0x7f502806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 -- 172.21.15.6:0/2553805508 shutdown_connections 2024-03-06T07:33:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.006+0000 7f502f600640 1 -- 172.21.15.6:0/2553805508 wait complete. 2024-03-06T07:33:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.008+0000 7f502f600640 1 Processor -- start 2024-03-06T07:33:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.008+0000 7f502f600640 1 -- start start 2024-03-06T07:33:56.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502f600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f50280ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502f600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50281f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502f600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50280cf5c0 0x7f5028153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502f600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502815b340 con 0x7f502806cb10 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502f600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280b5100 con 0x7f50280cf5c0 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502f600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502806d4c0 con 0x7f50280b3c90 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f50280ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f50280ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49558/0 (socket says 172.21.15.6:49558) 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 -- 172.21.15.6:0/1301601335 learned_addr learned my addr 172.21.15.6:0/1301601335 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f501fe00640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50281f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502d600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50280cf5c0 0x7f5028153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 -- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 msgr2=0x7f50281f4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50281f4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 -- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50280cf5c0 msgr2=0x7f5028153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.009+0000 7f502cc00640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50280cf5c0 0x7f5028153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f502cc00640 1 -- 172.21.15.6:0/1301601335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50280ca760 con 0x7f502806cb10 2024-03-06T07:33:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f502cc00640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f50280ca0f0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f501800d830 tx=0x7f501800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f5025600640 1 -- 172.21.15.6:0/1301601335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018018020 con 0x7f502806cb10 2024-03-06T07:33:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f5025600640 1 -- 172.21.15.6:0/1301601335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50180096a0 con 0x7f502806cb10 2024-03-06T07:33:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50280caa40 con 0x7f502806cb10 2024-03-06T07:33:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f5025600640 1 -- 172.21.15.6:0/1301601335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018009a70 con 0x7f502806cb10 2024-03-06T07:33:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.010+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50280cb000 con 0x7f502806cb10 2024-03-06T07:33:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.013+0000 7f5025600640 1 -- 172.21.15.6:0/1301601335 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f501801b020 con 0x7f502806cb10 2024-03-06T07:33:56.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.013+0000 7f5025600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.014+0000 7f5025600640 1 -- 172.21.15.6:0/1301601335 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f501808a760 con 0x7f502806cb10 2024-03-06T07:33:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.014+0000 7f50323b8740 1 Processor -- start 2024-03-06T07:33:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.014+0000 7f501fe00640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.014+0000 7f50323b8740 1 -- start start 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.015+0000 7f50323b8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fd9d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.015+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdcd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.015+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fd9dc90 0x55db7fdcda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.015+0000 7f50323b8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db7fd07600 con 0x55db7fdca7b0 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.015+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db7fcb0930 con 0x55db7fdcab90 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.015+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db7fd036a0 con 0x55db7fd9dc90 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f501fe00640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500406b4d0 0x7f500406d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5010006fd0 tx=0x7f5010008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fd9d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fd9d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49570/0 (socket says 172.21.15.6:49570) 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 -- 172.21.15.6:0/1046356507 learned_addr learned my addr 172.21.15.6:0/1046356507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f501fe00640 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdcd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502d600640 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fd9dc90 0x55db7fdcda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 -- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fd9dc90 msgr2=0x55db7fdcda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fd9dc90 0x55db7fdcda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 -- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 msgr2=0x55db7fdcd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdcd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.016+0000 7f502cc00640 1 -- 172.21.15.6:0/1046356507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db7fa1c2d0 con 0x55db7fdca7b0 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.017+0000 7f502cc00640 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fd9d750 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f5018061030 tx=0x7f50180588e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf304256f5e88373 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.017+0000 7f501d200640 1 -- 172.21.15.6:0/1046356507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501805cfa0 con 0x55db7fdca7b0 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.017+0000 7f501d200640 1 -- 172.21.15.6:0/1046356507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501805b050 con 0x55db7fdca7b0 2024-03-06T07:33:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.017+0000 7f501d200640 1 -- 172.21.15.6:0/1046356507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018061560 con 0x55db7fdca7b0 2024-03-06T07:33:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.017+0000 7f50323b8740 1 -- 172.21.15.6:0/1046356507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 msgr2=0x55db7fd9d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.017+0000 7f50323b8740 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fd9d750 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f5018061030 tx=0x7f50180588e0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 -- 172.21.15.6:0/1046356507 shutdown_connections 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdcd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fd9dc90 0x55db7fdcda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 --2- 172.21.15.6:0/1046356507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fd9d750 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 -- 172.21.15.6:0/1046356507 >> 172.21.15.6:0/1046356507 conn(0x55db7fd9f9c0 msgr2=0x55db7fde8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 -- 172.21.15.6:0/1046356507 shutdown_connections 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 -- 172.21.15.6:0/1046356507 wait complete. 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.018+0000 7f50323b8740 1 -- start start 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.019+0000 7f50323b8740 1 Processor -- start 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdd09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdf9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fde6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db7fda1f60 con 0x55db7fdca7b0 2024-03-06T07:33:56.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db7fcc3d90 con 0x55db7fdcab90 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db7fda1350 con 0x55db7fdf9fa0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f50323b8740 10 client.0 _ll_register_callbacks cb 0x55db7fcff350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdd09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdd09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49584/0 (socket says 172.21.15.6:49584) 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 -- 172.21.15.6:0/3821952571 learned_addr learned my addr 172.21.15.6:0/3821952571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f501fe00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdf9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502d600640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fde6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 -- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 msgr2=0x55db7fde6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fde6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 -- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 msgr2=0x55db7fdf9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdf9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f502cc00640 1 -- 172.21.15.6:0/3821952571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db7fae66b0 con 0x55db7fdca7b0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.020+0000 7f501fe00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdf9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f502cc00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdd09a0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f5018059010 tx=0x7f5018010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23420a30fc78bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f5015600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:33:56.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f5015600640 1 -- 172.21.15.6:0/3821952571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018086020 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f5015600640 1 -- 172.21.15.6:0/3821952571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5018024dc0 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f501805b050 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f5015600640 1 -- 172.21.15.6:0/3821952571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f501807f050 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f50323b8740 10 client.7806 fetch_fsmap learned FSMap version 77 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55db7fdcfec0 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.021+0000 7f5015600640 1 -- 172.21.15.6:0/3821952571 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501805b050 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f5015600640 1 -- 172.21.15.6:0/3821952571 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5018082020 con 0x55db7fdca7b0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f50323b8740 10 client.7806 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f50323b8740 10 client.7806 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f50323b8740 20 client.7806 populate_metadata read hostname 'smithi006' 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f50323b8740 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdf8c40 0x55db7fdf9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f50323b8740 4 client.7806 mds_command: new command op to 7656 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 --> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db7fd14b40 con 0x55db7fdf8c40 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f501fe00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdf8c40 0x55db7fdf9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f501fe00640 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdf8c40 0x55db7fdf9020 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=400505b1135e806e server_cookie=a85aef5db2c8c368 in_seq=0 out_seq=0 2024-03-06T07:33:56.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.022+0000 7f5015600640 10 client.7806 ms_handle_connect on v2:172.21.15.6:6832/4114489444 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.023+0000 7f5015600640 1 -- 172.21.15.6:0/3821952571 <== mds.0 v2:172.21.15.6:6832/4114489444 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55db7fd14b40 con 0x55db7fdf8c40 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.023+0000 7f5015600640 10 client.7806 handle_command_reply: tid=0 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.023+0000 7f50323b8740 1 client.7806 shutdown 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 msgr2=0x55db7fdd09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdd09a0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f5018059010 tx=0x7f5018010000 comp rx=0 tx=0).stop 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 shutdown_connections 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdf8c40 0x55db7fdf9020 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdf9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fde6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f5015600640 0 client.7806 ms_handle_reset on v2:172.21.15.6:6832/4114489444 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 --2- 172.21.15.6:0/3821952571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdd09a0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 >> 172.21.15.6:0/3821952571 conn(0x55db7fd9dc90 msgr2=0x55db7fdcd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 shutdown_connections 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 1 -- 172.21.15.6:0/3821952571 wait complete. 2024-03-06T07:33:56.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.024+0000 7f50323b8740 20 client.7806 trim_cache size 0 max 16384 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.042+0000 7f50323b8740 1 Processor -- start 2024-03-06T07:33:56.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.042+0000 7f50323b8740 1 -- start start 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.043+0000 7f50323b8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdf9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.043+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.043+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fdea660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.043+0000 7f50323b8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5028153360 con 0x55db7fdca7b0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.043+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50281f80e0 con 0x55db7fdcab90 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.043+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5028152d60 con 0x55db7fdf9fa0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdf9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdf9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49596/0 (socket says 172.21.15.6:49596) 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 -- 172.21.15.6:0/2239045647 learned_addr learned my addr 172.21.15.6:0/2239045647 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 -- 172.21.15.6:0/2239045647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 msgr2=0x55db7fdea660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fdea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 -- 172.21.15.6:0/2239045647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 msgr2=0x55db7fdea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 -- 172.21.15.6:0/2239045647 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db7fdf8690 con 0x55db7fdca7b0 2024-03-06T07:33:56.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.044+0000 7f502cc00640 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdf9210 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5018057100 tx=0x7f5018075000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f5cf4a162219cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f5017e00640 1 -- 172.21.15.6:0/2239045647 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180550f0 con 0x55db7fdca7b0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f5017e00640 1 -- 172.21.15.6:0/2239045647 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501805b070 con 0x55db7fdca7b0 2024-03-06T07:33:56.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f5017e00640 1 -- 172.21.15.6:0/2239045647 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180762f0 con 0x55db7fdca7b0 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 -- 172.21.15.6:0/2239045647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 msgr2=0x55db7fdf9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdf9210 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5018057100 tx=0x7f5018075000 comp rx=0 tx=0).stop 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 -- 172.21.15.6:0/2239045647 shutdown_connections 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdcab90 0x55db7fdea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdf9fa0 0x55db7fdea660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 --2- 172.21.15.6:0/2239045647 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdf9210 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 -- 172.21.15.6:0/2239045647 >> 172.21.15.6:0/2239045647 conn(0x55db7fd9dc90 msgr2=0x55db7fdcd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 -- 172.21.15.6:0/2239045647 shutdown_connections 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.045+0000 7f50323b8740 1 -- 172.21.15.6:0/2239045647 wait complete. 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.046+0000 7f50323b8740 1 -- start start 2024-03-06T07:33:56.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.046+0000 7f50323b8740 1 Processor -- start 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdcab90 0x55db7fdebe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdf9fa0 0x55db7fdec370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db7fda1f60 con 0x55db7fdca7b0 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db7fcc3d90 con 0x55db7fdf9fa0 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db7fde8670 con 0x55db7fdcab90 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f50323b8740 10 client.0 _ll_register_callbacks cb 0x7f501800b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:33:56.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.047+0000 7f502cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdea5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49606/0 (socket says 172.21.15.6:49606) 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 -- 172.21.15.6:0/523998187 learned_addr learned my addr 172.21.15.6:0/523998187 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 -- 172.21.15.6:0/523998187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdcab90 msgr2=0x55db7fdebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdcab90 0x55db7fdebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 -- 172.21.15.6:0/523998187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdf9fa0 msgr2=0x55db7fdec370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdf9fa0 0x55db7fdec370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 -- 172.21.15.6:0/523998187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db7fdeaaf0 con 0x55db7fdca7b0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f502cc00640 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdea5b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f50180552b0 tx=0x7f5018058cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a4b3d381a3e8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f5015600640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.048+0000 7f5015600640 1 -- 172.21.15.6:0/523998187 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180550f0 con 0x55db7fdca7b0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f5015600640 1 -- 172.21.15.6:0/523998187 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50180762f0 con 0x55db7fdca7b0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f501805b070 con 0x55db7fdca7b0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f5015600640 1 -- 172.21.15.6:0/523998187 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7f501804b050 con 0x55db7fdca7b0 2024-03-06T07:33:56.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 10 client.7812 fetch_fsmap learned FSMap version 77 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55db7fdeaaf0 con 0x55db7fdca7b0 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f5015600640 1 -- 172.21.15.6:0/523998187 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501805b070 con 0x55db7fdca7b0 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f5015600640 1 -- 172.21.15.6:0/523998187 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 77) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f5018082040 con 0x55db7fdca7b0 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 10 client.7812 fetch_fsmap finished waiting for FSMap version 77 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 10 client.7812 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 20 client.7812 populate_metadata read hostname 'smithi006' 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdce670 0x55db7fdcea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 4 client.7812 mds_command: new command op to 7656 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.049+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 --> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55db7fd0ee90 con 0x55db7fdce670 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.050+0000 7f501fe00640 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdce670 0x55db7fdcea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.050+0000 7f501fe00640 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdce670 0x55db7fdcea50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=876aba9ec25b63c3 server_cookie=bbd2b4839d4a8802 in_seq=0 out_seq=0 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.050+0000 7f5015600640 10 client.7812 ms_handle_connect on v2:172.21.15.6:6832/4114489444 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.050+0000 7f5015600640 1 -- 172.21.15.6:0/523998187 <== mds.0 v2:172.21.15.6:6832/4114489444 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55db7fd0ee90 con 0x55db7fdce670 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.050+0000 7f5015600640 10 client.7812 handle_command_reply: tid=0 2024-03-06T07:33:56.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.050+0000 7f50323b8740 1 client.7812 shutdown 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 msgr2=0x55db7fdea5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdea5b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f50180552b0 tx=0x7f5018058cd0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 shutdown_connections 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:6832/4114489444,v1:172.21.15.6:6833/4114489444] conn(0x55db7fdce670 0x55db7fdcea50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55db7fdf9fa0 0x55db7fdec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55db7fdcab90 0x55db7fdebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 --2- 172.21.15.6:0/523998187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55db7fdca7b0 0x55db7fdea5b0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f5015600640 0 client.7812 ms_handle_reset on v2:172.21.15.6:6832/4114489444 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 >> 172.21.15.6:0/523998187 conn(0x55db7fd9dc90 msgr2=0x55db7fda12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 shutdown_connections 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 1 -- 172.21.15.6:0/523998187 wait complete. 2024-03-06T07:33:56.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.051+0000 7f50323b8740 20 client.7812 trim_cache size 0 max 16384 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7656, 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 7.0761512209999999, 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 77, 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 108, 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 108, 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 12.469998895 2024-03-06T07:33:56.057 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500406b4d0 msgr2=0x7f500406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500406b4d0 0x7f500406d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f5010006fd0 tx=0x7f5010008040 comp rx=0 tx=0).stop 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 msgr2=0x7f50280ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f50280ca0f0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f501800d830 tx=0x7f501800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 shutdown_connections 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50280cf5c0 0x7f5028153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50280b3c90 0x7f50281f4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 --2- 172.21.15.6:0/1301601335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502806cb10 0x7f50280ca0f0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 >> 172.21.15.6:0/1301601335 conn(0x7f502806afb0 msgr2=0x7f5028156ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 shutdown_connections 2024-03-06T07:33:56.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.052+0000 7f502f600640 1 -- 172.21.15.6:0/1301601335 wait complete. 2024-03-06T07:33:56.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:33:56.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.276+0000 7fc1afe00640 1 Processor -- start 2024-03-06T07:33:56.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.276+0000 7fc1afe00640 1 -- start start 2024-03-06T07:33:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.277+0000 7fc1afe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1b00d4a10 0x7fc1b00d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.277+0000 7fc1afe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc1b00d0c80 0x7fc1b00b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.277+0000 7fc1afe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00b3710 con 0x7fc1b015e0b0 2024-03-06T07:33:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.277+0000 7fc1afe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b00d10f0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.277+0000 7fc1afe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b00b09e0 con 0x7fc1b00d0c80 2024-03-06T07:33:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.277+0000 7fc1a7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1b00d4a10 0x7fc1b00d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1a7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1b00d4a10 0x7fc1b00d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49608/0 (socket says 172.21.15.6:49608) 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1a7e00640 1 -- 172.21.15.6:0/395930425 learned_addr learned my addr 172.21.15.6:0/395930425 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1af400640 1 --2- 172.21.15.6:0/395930425 >> v2:172.21.15.178:3300/0 conn(0x7fc1b00d0c80 0x7fc1b00b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1a7e00640 1 -- 172.21.15.6:0/395930425 >> v2:172.21.15.178:3300/0 conn(0x7fc1b00d0c80 msgr2=0x7fc1b00b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1a7e00640 1 --2- 172.21.15.6:0/395930425 >> v2:172.21.15.178:3300/0 conn(0x7fc1b00d0c80 0x7fc1b00b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1a7e00640 1 -- 172.21.15.6:0/395930425 >> v1:172.21.15.178:6789/0 conn(0x7fc1b015e0b0 legacy=0x7fc1b00cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1a7e00640 1 -- 172.21.15.6:0/395930425 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00b39f0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.278+0000 7fc1af400640 1 --2- 172.21.15.6:0/395930425 >> v2:172.21.15.178:3300/0 conn(0x7fc1b00d0c80 0x7fc1b00b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.279+0000 7fc1a7e00640 1 --2- 172.21.15.6:0/395930425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00d0740 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fc1b006b8c0 tx=0x7fc19802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2919059f2f1caa6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1ae000640 1 -- 172.21.15.6:0/395930425 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198035020 con 0x7fc1b00d4a10 2024-03-06T07:33:56.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1ae000640 1 -- 172.21.15.6:0/395930425 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198031830 con 0x7fc1b00d4a10 2024-03-06T07:33:56.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1ae000640 1 -- 172.21.15.6:0/395930425 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198031c00 con 0x7fc1b00d4a10 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1afe00640 1 -- 172.21.15.6:0/395930425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 msgr2=0x7fc1b00d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1afe00640 1 --2- 172.21.15.6:0/395930425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00d0740 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fc1b006b8c0 tx=0x7fc19802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1afe00640 1 -- 172.21.15.6:0/395930425 shutdown_connections 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1afe00640 1 --2- 172.21.15.6:0/395930425 >> v2:172.21.15.178:3300/0 conn(0x7fc1b00d0c80 0x7fc1b00b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1afe00640 1 --2- 172.21.15.6:0/395930425 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00d0740 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.280+0000 7fc1afe00640 1 -- 172.21.15.6:0/395930425 >> 172.21.15.6:0/395930425 conn(0x7fc1b00d3d30 msgr2=0x7fc1b00d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.281+0000 7fc1afe00640 1 -- 172.21.15.6:0/395930425 shutdown_connections 2024-03-06T07:33:56.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.281+0000 7fc1afe00640 1 -- 172.21.15.6:0/395930425 wait complete. 2024-03-06T07:33:56.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.282+0000 7fc1afe00640 1 Processor -- start 2024-03-06T07:33:56.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.282+0000 7fc1afe00640 1 -- start start 2024-03-06T07:33:56.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.283+0000 7fc1afe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1b00d0c80 0x7fc1b00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.283+0000 7fc1afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1afe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1b015e0b0 0x7fc1b00b8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1afe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00b3fb0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1afe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00cfa20 con 0x7fc1b015e0b0 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1afe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b015e490 con 0x7fc1b00d0c80 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00b7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49612/0 (socket says 172.21.15.6:49612) 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 -- 172.21.15.6:0/1654565138 learned_addr learned my addr 172.21.15.6:0/1654565138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 -- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1b00d0c80 msgr2=0x7fc1b00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1aea00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1b00d0c80 0x7fc1b00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1af400640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1b015e0b0 0x7fc1b00b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1b00d0c80 0x7fc1b00cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 -- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1b015e0b0 msgr2=0x7fc1b00b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1b015e0b0 0x7fc1b00b8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1a7e00640 1 -- 172.21.15.6:0/1654565138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0151d00 con 0x7fc1b00d4a10 2024-03-06T07:33:56.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.284+0000 7fc1aea00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1b00d0c80 0x7fc1b00cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:56.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.285+0000 7fc1a7e00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00b7d10 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fc1980097f0 tx=0x7fc1980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.285+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198040070 con 0x7fc1b00d4a10 2024-03-06T07:33:56.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.285+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1980399c0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.285+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198035040 con 0x7fc1b00d4a10 2024-03-06T07:33:56.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.285+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b0151fe0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.285+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b01525a0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.286+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b004f330 con 0x7fc1b00d4a10 2024-03-06T07:33:56.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.289+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc19804b060 con 0x7fc1b00d4a10 2024-03-06T07:33:56.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.290+0000 7fc1a6a00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc18c06b4d0 0x7fc18c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:56.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.290+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1980ac0c0 con 0x7fc1b00d4a10 2024-03-06T07:33:56.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.290+0000 7fc1aea00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc18c06b4d0 0x7fc18c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:56.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.292+0000 7fc1aea00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc18c06b4d0 0x7fc18c06d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0003eb0 tx=0x7fc1a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:56.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.295+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc19807af20 con 0x7fc1b00d4a10 2024-03-06T07:33:56.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.533+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc1b00b0c10 con 0x7fc1b00d4a10 2024-03-06T07:33:56.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.534+0000 7fc1a6a00640 1 -- 172.21.15.6:0/1654565138 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc19807a820 con 0x7fc1b00d4a10 2024-03-06T07:33:56.536 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc18c06b4d0 msgr2=0x7fc18c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc18c06b4d0 0x7fc18c06d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0003eb0 tx=0x7fc1a0009040 comp rx=0 tx=0).stop 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 msgr2=0x7fc1b00b7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00b7d10 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fc1980097f0 tx=0x7fc1980046c0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 shutdown_connections 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc18c06b4d0 0x7fc18c06d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1b015e0b0 0x7fc1b00b8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1b00d4a10 0x7fc1b00b7d10 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 --2- 172.21.15.6:0/1654565138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1b00d0c80 0x7fc1b00cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:56.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 >> 172.21.15.6:0/1654565138 conn(0x7fc1b00d3d30 msgr2=0x7fc1b006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:56.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 shutdown_connections 2024-03-06T07:33:56.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:56.537+0000 7fc1afe00640 1 -- 172.21.15.6:0/1654565138 wait complete. 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:33:56.559 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:56.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:56.560 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:33:56.609 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:33:56.609 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:33:56.610 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42221sec preferred_lft 42221sec 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.611 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:33:56.612 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:33:56.612 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:33:56.612 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:33:56.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:56.612 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:33:56.695 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:33:56.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:56.700 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:33:56.750 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:33:56.750 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:33:56.750 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:33:56.750 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:33:56.750 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:33:56.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:56.751 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:33:56.751 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:33:56.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:33:56.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:33:56.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:33:56.751 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:33:56.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:56.934 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:33:56.983 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:33:56.983 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:33:56.984 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:56.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:56.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:33:57.074 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:33:57.074 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:57.120 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:33:57.120 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:33:57.121 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:57.214 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:33:57.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:57.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:57.358 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:57.358 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:33:57.391 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:33:57.392 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:33:57.396 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:33:57.396 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:33:57.396 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:33:57.396 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:33:57.396 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:33:57.401 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:33:57.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:57.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:33:57.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:57.481 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:33:57.542 INFO:teuthology.orchestra.run.smithi006.stdout:client7821 2024-03-06T07:33:57.542 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:57.542 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:57.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:57.588 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:33:57.696 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:57.696 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:57.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:57.747 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:33:57.947 INFO:teuthology.orchestra.run.smithi006.stderr:8+0 records in 2024-03-06T07:33:57.947 INFO:teuthology.orchestra.run.smithi006.stderr:8+0 records out 2024-03-06T07:33:57.947 INFO:teuthology.orchestra.run.smithi006.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.148544 s, 56.5 MB/s 2024-03-06T07:33:57.949 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:33:57.949 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:33:57.949 DEBUG:teuthology.orchestra.run.smithi006:> import stat 2024-03-06T07:33:57.949 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:33:57.949 DEBUG:teuthology.orchestra.run.smithi006:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:33:57.949 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:33:58.088 INFO:teuthology.orchestra.run.smithi006.stdout:1099511627777 2024-03-06T07:33:58.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:33:58.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:58.102 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:33:58.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:33:58.200 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi006:> import stat 2024-03-06T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi006:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:33:58.401 INFO:teuthology.orchestra.run.smithi006.stdout:1099511627777 2024-03-06T07:33:58.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:33:58.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:33:58.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.680+0000 7f4ed9000640 1 Processor -- start 2024-03-06T07:33:58.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.681+0000 7f4ed9000640 1 -- start start 2024-03-06T07:33:58.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed40cd060 0x7f4ed415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed415e810 0x7f4ed40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed9000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed415ed10 con 0x7f4ed41552b0 2024-03-06T07:33:58.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed9000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed415eff0 con 0x7f4ed415e810 2024-03-06T07:33:58.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed9000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed415f2d0 con 0x7f4ed40cd060 2024-03-06T07:33:58.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed415e810 0x7f4ed40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed415e810 0x7f4ed40b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49620/0 (socket says 172.21.15.6:49620) 2024-03-06T07:33:58.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.684+0000 7f4ed3e00640 1 -- 172.21.15.6:0/2386997014 learned_addr learned my addr 172.21.15.6:0/2386997014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.685+0000 7f4ed2a00640 1 --2- 172.21.15.6:0/2386997014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed40cd060 0x7f4ed415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.685+0000 7f4ed3e00640 1 -- 172.21.15.6:0/2386997014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed40cd060 msgr2=0x7f4ed415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.685+0000 7f4ed3e00640 1 --2- 172.21.15.6:0/2386997014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed40cd060 0x7f4ed415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.685+0000 7f4ed3e00640 1 -- 172.21.15.6:0/2386997014 >> v1:172.21.15.6:6789/0 conn(0x7f4ed41552b0 legacy=0x7f4ed40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.685+0000 7f4ed3e00640 1 -- 172.21.15.6:0/2386997014 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed415f5b0 con 0x7f4ed415e810 2024-03-06T07:33:58.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.686+0000 7f4ed3e00640 1 --2- 172.21.15.6:0/2386997014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed415e810 0x7f4ed40b3cb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009e50 tx=0x7f4ecc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644b73a619ff0f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.686+0000 7f4ed2000640 1 -- 172.21.15.6:0/2386997014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc02ca80 con 0x7f4ed415e810 2024-03-06T07:33:58.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.686+0000 7f4ed2000640 1 -- 172.21.15.6:0/2386997014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc02cc20 con 0x7f4ed415e810 2024-03-06T07:33:58.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.686+0000 7f4ed2000640 1 -- 172.21.15.6:0/2386997014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0343f0 con 0x7f4ed415e810 2024-03-06T07:33:58.689 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 -- 172.21.15.6:0/2386997014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed415e810 msgr2=0x7f4ed40b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 --2- 172.21.15.6:0/2386997014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed415e810 0x7f4ed40b3cb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009e50 tx=0x7f4ecc02bee0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 -- 172.21.15.6:0/2386997014 shutdown_connections 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 --2- 172.21.15.6:0/2386997014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed415e810 0x7f4ed40b3cb0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 --2- 172.21.15.6:0/2386997014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed40cd060 0x7f4ed415b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 -- 172.21.15.6:0/2386997014 >> 172.21.15.6:0/2386997014 conn(0x7f4ed406b3d0 msgr2=0x7f4ed406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 -- 172.21.15.6:0/2386997014 shutdown_connections 2024-03-06T07:33:58.690 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.687+0000 7f4ed9000640 1 -- 172.21.15.6:0/2386997014 wait complete. 2024-03-06T07:33:58.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.689+0000 7f4ed9000640 1 Processor -- start 2024-03-06T07:33:58.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.689+0000 7f4ed9000640 1 -- start start 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed9000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed40cd060 0x7f4ed414c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 0x7f4ed414fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed415e810 0x7f4ed41451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed406dd30 con 0x7f4ed41552b0 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed9000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40b3760 con 0x7f4ed40cd060 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed9000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed406deb0 con 0x7f4ed415e810 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 0x7f4ed414fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 0x7f4ed414fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49628/0 (socket says 172.21.15.6:49628) 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.690+0000 7f4ed2a00640 1 -- 172.21.15.6:0/3238346093 learned_addr learned my addr 172.21.15.6:0/3238346093 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed3400640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed40cd060 0x7f4ed414c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed3e00640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed415e810 0x7f4ed41451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed2a00640 1 -- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed415e810 msgr2=0x7f4ed41451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed2a00640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed415e810 0x7f4ed41451d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed2a00640 1 -- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed40cd060 msgr2=0x7f4ed414c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed2a00640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed40cd060 0x7f4ed414c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed2a00640 1 -- 172.21.15.6:0/3238346093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40ce750 con 0x7f4ed41552b0 2024-03-06T07:33:58.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed3400640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed40cd060 0x7f4ed414c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ed2a00640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 0x7f4ed414fa50 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d800 tx=0x7f4ec000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.691+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec000b6c0 con 0x7f4ed41552b0 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.692+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec0019070 con 0x7f4ed41552b0 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.692+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec000ba80 con 0x7f4ed41552b0 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.692+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed40cea30 con 0x7f4ed41552b0 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.692+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed4145710 con 0x7f4ed41552b0 2024-03-06T07:33:58.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.693+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed404f330 con 0x7f4ed41552b0 2024-03-06T07:33:58.696 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.694+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ec000bc20 con 0x7f4ed41552b0 2024-03-06T07:33:58.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.695+0000 7f4ecb400640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4eb406b500 0x7f4eb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:58.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.695+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ec008a6c0 con 0x7f4ed41552b0 2024-03-06T07:33:58.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.696+0000 7f4ed3400640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4eb406b500 0x7f4eb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:58.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.697+0000 7f4ed3400640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4eb406b500 0x7f4eb406d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4001680 tx=0x7f4ec4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:58.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.704+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ec0059530 con 0x7f4ed41552b0 2024-03-06T07:33:58.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.947+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ed40cf4a0 con 0x7f4ed41552b0 2024-03-06T07:33:58.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.948+0000 7f4ecb400640 1 -- 172.21.15.6:0/3238346093 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f4ec0058e30 con 0x7f4ed41552b0 2024-03-06T07:33:58.950 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:58.951 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5795,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3411419645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3411419645},{"type":"v1","addr":"172.21.15.6:6835","nonce":3411419645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5798,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/525628181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":525628181},{"type":"v1","addr":"172.21.15.178:6835","nonce":525628181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/319127923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":319127923},{"type":"v1","addr":"172.21.15.178:6837","nonce":319127923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:33:51.764337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7656},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7656":{"gid":7656,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6833/4114489444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4114489444},{"type":"v1","addr":"172.21.15.6:6833","nonce":4114489444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7656,"qdb_cluster":[7656]},"id":8}]} 2024-03-06T07:33:58.951 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-06T07:33:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4eb406b500 msgr2=0x7f4eb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4eb406b500 0x7f4eb406d9c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4001680 tx=0x7f4ec4009040 comp rx=0 tx=0).stop 2024-03-06T07:33:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 msgr2=0x7f4ed414fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 0x7f4ed414fa50 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d800 tx=0x7f4ec000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 shutdown_connections 2024-03-06T07:33:58.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4eb406b500 0x7f4eb406d9c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ed415e810 0x7f4ed41451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed41552b0 0x7f4ed414fa50 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 --2- 172.21.15.6:0/3238346093 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ed40cd060 0x7f4ed414c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 >> 172.21.15.6:0/3238346093 conn(0x7f4ed406b3d0 msgr2=0x7f4ed40cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 shutdown_connections 2024-03-06T07:33:58.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:58.951+0000 7f4ed9000640 1 -- 172.21.15.6:0/3238346093 wait complete. 2024-03-06T07:33:58.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:33:59.215 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:33:59.215 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:33:59.216 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:33:59.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.307+0000 7f880783df40 1 Processor -- start 2024-03-06T07:33:59.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.307+0000 7f880783df40 1 -- start start 2024-03-06T07:33:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.308+0000 7f880783df40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5639977cc660 0x563997811100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.308+0000 7f880783df40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977fa540 0x5639978962f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.308+0000 7f880783df40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 0x5639977f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.308+0000 7f880783df40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563997810d10 con 0x563997899830 2024-03-06T07:33:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.308+0000 7f880783df40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x563997770fd0 con 0x5639977cc660 2024-03-06T07:33:59.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.308+0000 7f880783df40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x563997813a90 con 0x5639977fa540 2024-03-06T07:33:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 0x5639977f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 0x5639977f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49630/0 (socket says 172.21.15.6:49630) 2024-03-06T07:33:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 -- 172.21.15.6:0/2990919986 learned_addr learned my addr 172.21.15.6:0/2990919986 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8803800640 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977fa540 0x5639978962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804200640 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5639977cc660 0x563997811100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 -- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977fa540 msgr2=0x5639978962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977fa540 0x5639978962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 -- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5639977cc660 msgr2=0x563997811100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5639977cc660 0x563997811100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.309+0000 7f8804c00640 1 -- 172.21.15.6:0/2990919986 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563997892400 con 0x563997899830 2024-03-06T07:33:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.310+0000 7f8804c00640 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 0x5639977f52c0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f87fc055040 tx=0x7f87fc04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9def49bd1a154f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.311+0000 7f8802400640 1 -- 172.21.15.6:0/2990919986 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc057040 con 0x563997899830 2024-03-06T07:33:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.311+0000 7f8802400640 1 -- 172.21.15.6:0/2990919986 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87fc062070 con 0x563997899830 2024-03-06T07:33:59.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.311+0000 7f8802400640 1 -- 172.21.15.6:0/2990919986 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc052c50 con 0x563997899830 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 -- 172.21.15.6:0/2990919986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 msgr2=0x5639977f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 0x5639977f52c0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f87fc055040 tx=0x7f87fc04c000 comp rx=0 tx=0).stop 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 -- 172.21.15.6:0/2990919986 shutdown_connections 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x563997899830 0x5639977f52c0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977fa540 0x5639978962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 --2- 172.21.15.6:0/2990919986 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5639977cc660 0x563997811100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 -- 172.21.15.6:0/2990919986 >> 172.21.15.6:0/2990919986 conn(0x5639977ca5c0 msgr2=0x563997813240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 -- 172.21.15.6:0/2990919986 shutdown_connections 2024-03-06T07:33:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.312+0000 7f880783df40 1 -- 172.21.15.6:0/2990919986 wait complete. 2024-03-06T07:33:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.313+0000 7f880783df40 1 Processor -- start 2024-03-06T07:33:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.313+0000 7f880783df40 1 -- start start 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f880783df40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977cc660 0x5639978851b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f880783df40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639977fa540 0x563997939390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f880783df40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563997899830 0x5639978843b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f880783df40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x563997892950 con 0x5639977fa540 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f880783df40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x563997810970 con 0x563997899830 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f880783df40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x563997939110 con 0x5639977cc660 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f8803800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639977fa540 0x563997939390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f8803800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639977fa540 0x563997939390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49636/0 (socket says 172.21.15.6:49636) 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.315+0000 7f8803800640 1 -- 172.21.15.6:0/316672960 learned_addr learned my addr 172.21.15.6:0/316672960 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:59.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8804200640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977cc660 0x5639978851b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8804c00640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563997899830 0x5639978843b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8803800640 1 -- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977cc660 msgr2=0x5639978851b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8803800640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5639977cc660 0x5639978851b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8803800640 1 -- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563997899830 msgr2=0x5639978843b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8803800640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x563997899830 0x5639978843b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8803800640 1 -- 172.21.15.6:0/316672960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563997811640 con 0x5639977fa540 2024-03-06T07:33:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8803800640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639977fa540 0x563997939390 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f87f4006fa0 tx=0x7f87f4003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8801000640 1 -- 172.21.15.6:0/316672960 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4012420 con 0x5639977fa540 2024-03-06T07:33:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8801000640 1 -- 172.21.15.6:0/316672960 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f4017070 con 0x5639977fa540 2024-03-06T07:33:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.316+0000 7f8801000640 1 -- 172.21.15.6:0/316672960 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f40127e0 con 0x5639977fa540 2024-03-06T07:33:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.317+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5639979398d0 con 0x5639977fa540 2024-03-06T07:33:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.317+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639978848f0 con 0x5639977fa540 2024-03-06T07:33:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.319+0000 7f8801000640 1 -- 172.21.15.6:0/316672960 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87f4012980 con 0x5639977fa540 2024-03-06T07:33:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.320+0000 7f8801000640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87e0069eb0 0x7f87e006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.320+0000 7f8801000640 1 -- 172.21.15.6:0/316672960 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87f4088f60 con 0x5639977fa540 2024-03-06T07:33:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.320+0000 7f880783df40 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x5639979389a0 0x56399789bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.320+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108) v9 -- 0x56399789c200 con 0x5639979389a0 2024-03-06T07:33:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.320+0000 7f8804c00640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x5639979389a0 0x56399789bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.320+0000 7f8804200640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87e0069eb0 0x7f87e006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.321+0000 7f8804c00640 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x5639979389a0 0x56399789bcc0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.321+0000 7f8804c00640 1 -- 172.21.15.6:0/316672960 <== osd.0 v2:172.21.15.6:6800/2712143005 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f87fc05a040 con 0x5639979389a0 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.321+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x5639979389a0 msgr2=0x56399789bcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.321+0000 7f880783df40 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x5639979389a0 0x56399789bcc0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.321+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87e0069eb0 msgr2=0x7f87e006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.321+0000 7f880783df40 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87e0069eb0 0x7f87e006c370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009740 tx=0x7f87ec00c040 comp rx=0 tx=0).stop 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639977fa540 msgr2=0x563997939390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f880783df40 1 --2- 172.21.15.6:0/316672960 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5639977fa540 0x563997939390 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f87f4006fa0 tx=0x7f87f4003890 comp rx=0 tx=0).stop 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f8804c00640 1 -- 172.21.15.6:0/316672960 reap_dead start 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 shutdown_connections 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 >> 172.21.15.6:0/316672960 conn(0x5639977ca5c0 msgr2=0x5639977b8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 shutdown_connections 2024-03-06T07:33:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.322+0000 7f880783df40 1 -- 172.21.15.6:0/316672960 wait complete. 2024-03-06T07:33:59.328 INFO:teuthology.orchestra.run.smithi006.stdout:Wfile_adir_1 2024-03-06T07:33:59.328 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1099511627776, 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "file_a", 2024-03-06T07:33:59.494 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 8 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1, 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "dir_1", 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 15 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:33:59.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:33:59.503 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:59.503 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:59.556 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:33:59.556 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:33:59.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:33:59.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.846+0000 7ffaf9c00640 1 Processor -- start 2024-03-06T07:33:59.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.846+0000 7ffaf9c00640 1 -- start start 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.847+0000 7ffaf9c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffaf415e0b0 0x7ffaf40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.847+0000 7ffaf9c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ffaf40d0e50 0x7ffaf40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.847+0000 7ffaf9c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40b3710 con 0x7ffaf40d4a10 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.847+0000 7ffaf9c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf40b09e0 con 0x7ffaf415e0b0 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.847+0000 7ffaf9c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf40b39f0 con 0x7ffaf40d0e50 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffaf415e0b0 0x7ffaf40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffaf415e0b0 0x7ffaf40cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49640/0 (socket says 172.21.15.6:49640) 2024-03-06T07:33:59.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf3e00640 1 -- 172.21.15.6:0/327148046 learned_addr learned my addr 172.21.15.6:0/327148046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf9200640 1 --2- 172.21.15.6:0/327148046 >> v2:172.21.15.178:3300/0 conn(0x7ffaf40d0e50 0x7ffaf40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf3e00640 1 -- 172.21.15.6:0/327148046 >> v2:172.21.15.178:3300/0 conn(0x7ffaf40d0e50 msgr2=0x7ffaf40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:33:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf3e00640 1 --2- 172.21.15.6:0/327148046 >> v2:172.21.15.178:3300/0 conn(0x7ffaf40d0e50 0x7ffaf40b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.848+0000 7ffaf3e00640 1 -- 172.21.15.6:0/327148046 >> v1:172.21.15.6:6789/0 conn(0x7ffaf40d4a10 legacy=0x7ffaf40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.849+0000 7ffaf3e00640 1 -- 172.21.15.6:0/327148046 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40b3cd0 con 0x7ffaf415e0b0 2024-03-06T07:33:59.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.850+0000 7ffaf3e00640 1 --2- 172.21.15.6:0/327148046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40cf310 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ffaf408d6d0 tx=0x7ffae402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de07551feeb4000 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.850+0000 7ffaf3400640 1 -- 172.21.15.6:0/327148046 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4037070 con 0x7ffaf415e0b0 2024-03-06T07:33:59.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.850+0000 7ffaf3400640 1 -- 172.21.15.6:0/327148046 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae4004230 con 0x7ffaf415e0b0 2024-03-06T07:33:59.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.850+0000 7ffaf3400640 1 -- 172.21.15.6:0/327148046 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4003610 con 0x7ffaf415e0b0 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 -- 172.21.15.6:0/327148046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf415e0b0 msgr2=0x7ffaf40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/327148046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40cf310 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ffaf408d6d0 tx=0x7ffae402b7e0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 -- 172.21.15.6:0/327148046 shutdown_connections 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/327148046 >> v2:172.21.15.178:3300/0 conn(0x7ffaf40d0e50 0x7ffaf40b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/327148046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40cf310 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 -- 172.21.15.6:0/327148046 >> 172.21.15.6:0/327148046 conn(0x7ffaf40d3d30 msgr2=0x7ffaf40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 -- 172.21.15.6:0/327148046 shutdown_connections 2024-03-06T07:33:59.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.851+0000 7ffaf9c00640 1 -- 172.21.15.6:0/327148046 wait complete. 2024-03-06T07:33:59.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.854+0000 7ffaf9c00640 1 Processor -- start 2024-03-06T07:33:59.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.854+0000 7ffaf9c00640 1 -- start start 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.855+0000 7ffaf9c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 0x7ffaf41611a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf9c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaf40d4a10 0x7ffaf40b8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf9c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf9c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40cfc60 con 0x7ffaf40d0e50 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf9c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf415e490 con 0x7ffaf415e0b0 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf9c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40cf850 con 0x7ffaf40d4a10 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 0x7ffaf41611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 0x7ffaf41611a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49646/0 (socket says 172.21.15.6:49646) 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 -- 172.21.15.6:0/60930717 learned_addr learned my addr 172.21.15.6:0/60930717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaebe00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaf40d4a10 0x7ffaf40b8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf9200640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 -- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaf40d4a10 msgr2=0x7ffaf40b8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaf40d4a10 0x7ffaf40b8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 -- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf415e0b0 msgr2=0x7ffaf40b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:33:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:33:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.856+0000 7ffaf3e00640 1 -- 172.21.15.6:0/60930717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf415e610 con 0x7ffaf40d0e50 2024-03-06T07:33:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.857+0000 7ffaf3e00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 0x7ffaf41611a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000c00 tx=0x7ffae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.857+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4041070 con 0x7ffaf40d0e50 2024-03-06T07:33:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.857+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae4030510 con 0x7ffaf40d0e50 2024-03-06T07:33:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.857+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae4037040 con 0x7ffaf40d0e50 2024-03-06T07:33:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.857+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf40b7500 con 0x7ffaf40d0e50 2024-03-06T07:33:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.857+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf40b7ac0 con 0x7ffaf40d0e50 2024-03-06T07:33:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.859+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf404f330 con 0x7ffaf40d0e50 2024-03-06T07:33:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.859+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffae4044070 con 0x7ffaf40d0e50 2024-03-06T07:33:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.860+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:33:59.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.860+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffae40abe00 con 0x7ffaf40d0e50 2024-03-06T07:33:59.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.861+0000 7ffaebe00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:33:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.862+0000 7ffaebe00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffad006b4d0 0x7ffad006d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffaf41601b0 tx=0x7ffadc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:33:59.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:33:59.866+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffae407abe0 con 0x7ffaf40d0e50 2024-03-06T07:34:00.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.082+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaf40cf380 con 0x7ffaf40d0e50 2024-03-06T07:34:00.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.085+0000 7ffaf0c00640 1 -- 172.21.15.6:0/60930717 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16555 (secure 0 0 0) 0x7ffae407a4e0 con 0x7ffaf40d0e50 2024-03-06T07:34:00.088 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:00.088 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":108,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:33:49.990380+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffad006b4d0 msgr2=0x7ffad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffad006b4d0 0x7ffad006d990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffaf41601b0 tx=0x7ffadc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 msgr2=0x7ffaf41611a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 0x7ffaf41611a0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000c00 tx=0x7ffae40046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 shutdown_connections 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaf415e0b0 0x7ffaf40b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaf40d4a10 0x7ffaf40b8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 --2- 172.21.15.6:0/60930717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffaf40d0e50 0x7ffaf41611a0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 >> 172.21.15.6:0/60930717 conn(0x7ffaf40d3d30 msgr2=0x7ffaf40b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.088+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 shutdown_connections 2024-03-06T07:34:00.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.089+0000 7ffaf9c00640 1 -- 172.21.15.6:0/60930717 wait complete. 2024-03-06T07:34:00.110 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:46.337716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:47.450154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:34:00.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:34:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.324+0000 7f1ed9800640 1 Processor -- start 2024-03-06T07:34:00.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.324+0000 7f1ed9800640 1 -- start start 2024-03-06T07:34:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.326+0000 7f1ed9800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.326+0000 7f1ed9800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1ed40672d0 0x7f1ed40ae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.326+0000 7f1ed9800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ed40ae780 0x7f1ed40b3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.326+0000 7f1ed9800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed406f020 con 0x7f1ed40ae780 2024-03-06T07:34:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.326+0000 7f1ed9800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ed406e6f0 con 0x7f1ed40672d0 2024-03-06T07:34:00.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.326+0000 7f1ed9800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed406e9d0 con 0x7f1ed40aefd0 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.327+0000 7f1ed3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ed40ae780 0x7f1ed40b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.327+0000 7f1ed3400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1ed40ae780 0x7f1ed40b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49658/0 (socket says 172.21.15.6:49658) 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.327+0000 7f1ed3400640 1 -- 172.21.15.6:0/1665709934 learned_addr learned my addr 172.21.15.6:0/1665709934 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.327+0000 7f1ed2a00640 1 --2- 172.21.15.6:0/1665709934 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.327+0000 7f1ed2000640 1 --2- 172.21.15.6:0/1665709934 >> v2:172.21.15.178:3300/0 conn(0x7f1ed40672d0 0x7f1ed40ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.327+0000 7f1ed3400640 1 -- 172.21.15.6:0/1665709934 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 msgr2=0x7f1ed40af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.328+0000 7f1ed3400640 1 --2- 172.21.15.6:0/1665709934 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.328+0000 7f1ed3400640 1 -- 172.21.15.6:0/1665709934 >> v2:172.21.15.178:3300/0 conn(0x7f1ed40672d0 msgr2=0x7f1ed40ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.328+0000 7f1ed3400640 1 --2- 172.21.15.6:0/1665709934 >> v2:172.21.15.178:3300/0 conn(0x7f1ed40672d0 0x7f1ed40ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.328+0000 7f1ed3400640 1 -- 172.21.15.6:0/1665709934 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed406ecb0 con 0x7f1ed40ae780 2024-03-06T07:34:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.330+0000 7f1ed3400640 1 --2- 172.21.15.6:0/1665709934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40b3440 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f1ed406d2b0 tx=0x7f1ecc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8676cf8baacbbea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.330+0000 7f1ed1600640 1 -- 172.21.15.6:0/1665709934 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc02ca70 con 0x7f1ed40ae780 2024-03-06T07:34:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.330+0000 7f1ed1600640 1 -- 172.21.15.6:0/1665709934 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc02cc10 con 0x7f1ed40ae780 2024-03-06T07:34:00.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.330+0000 7f1ed1600640 1 -- 172.21.15.6:0/1665709934 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0343f0 con 0x7f1ed40ae780 2024-03-06T07:34:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 -- 172.21.15.6:0/1665709934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 msgr2=0x7f1ed40b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 --2- 172.21.15.6:0/1665709934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40b3440 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f1ed406d2b0 tx=0x7f1ecc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 -- 172.21.15.6:0/1665709934 shutdown_connections 2024-03-06T07:34:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 --2- 172.21.15.6:0/1665709934 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40b3440 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 --2- 172.21.15.6:0/1665709934 >> v2:172.21.15.178:3300/0 conn(0x7f1ed40672d0 0x7f1ed40ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 --2- 172.21.15.6:0/1665709934 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 -- 172.21.15.6:0/1665709934 >> 172.21.15.6:0/1665709934 conn(0x7f1ed4072280 msgr2=0x7f1ed406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 -- 172.21.15.6:0/1665709934 shutdown_connections 2024-03-06T07:34:00.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.331+0000 7f1ed9800640 1 -- 172.21.15.6:0/1665709934 wait complete. 2024-03-06T07:34:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.333+0000 7f1ed9800640 1 Processor -- start 2024-03-06T07:34:00.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.333+0000 7f1ed9800640 1 -- start start 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed9800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ed40672d0 0x7f1ed41f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed9800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed9800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed406e490 con 0x7f1ed40ae780 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed9800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed406e310 con 0x7f1ed40672d0 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed9800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed415e1c0 con 0x7f1ed40aefd0 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49666/0 (socket says 172.21.15.6:49666) 2024-03-06T07:34:00.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.334+0000 7f1ed2000640 1 -- 172.21.15.6:0/207707981 learned_addr learned my addr 172.21.15.6:0/207707981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed3400640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2a00640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ed40672d0 0x7f1ed41f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2000640 1 -- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 msgr2=0x7f1ed40ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2000640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2000640 1 -- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ed40672d0 msgr2=0x7f1ed41f1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2000640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ed40672d0 0x7f1ed41f1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2000640 1 -- 172.21.15.6:0/207707981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed41f1560 con 0x7f1ed40ae780 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed3400640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed2000640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40ceab0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e870 tx=0x7f1ec000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0017070 con 0x7f1ed40ae780 2024-03-06T07:34:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec000ee60 con 0x7f1ed40ae780 2024-03-06T07:34:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000cb20 con 0x7f1ed40ae780 2024-03-06T07:34:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.335+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed41f71f0 con 0x7f1ed40ae780 2024-03-06T07:34:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.336+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed41f77b0 con 0x7f1ed40ae780 2024-03-06T07:34:00.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.337+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed404f330 con 0x7f1ed40ae780 2024-03-06T07:34:00.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.339+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ec001a070 con 0x7f1ed40ae780 2024-03-06T07:34:00.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.340+0000 7f1ecaa00640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.340+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ec0089f00 con 0x7f1ed40ae780 2024-03-06T07:34:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.340+0000 7f1ed2a00640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.342+0000 7f1ed2a00640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1eb406b4d0 0x7f1eb406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009a10 tx=0x7f1ec4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.345+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ec0058e10 con 0x7f1ed40ae780 2024-03-06T07:34:00.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.592+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ed40b1cf0 con 0x7f1ed40ae780 2024-03-06T07:34:00.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.593+0000 7f1ecaa00640 1 -- 172.21.15.6:0/207707981 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ec0058710 con 0x7f1ed40ae780 2024-03-06T07:34:00.595 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:34:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1eb406b4d0 msgr2=0x7f1eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1eb406b4d0 0x7f1eb406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009a10 tx=0x7f1ec4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 msgr2=0x7f1ed40ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40ceab0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000e870 tx=0x7f1ec000ed30 comp rx=0 tx=0).stop 2024-03-06T07:34:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 shutdown_connections 2024-03-06T07:34:00.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1eb406b4d0 0x7f1eb406d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1ed40aefd0 0x7f1ed40ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1ed40ae780 0x7f1ed40ceab0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 --2- 172.21.15.6:0/207707981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1ed40672d0 0x7f1ed41f1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 >> 172.21.15.6:0/207707981 conn(0x7f1ed4072280 msgr2=0x7f1ed4160500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 shutdown_connections 2024-03-06T07:34:00.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.596+0000 7f1ed9800640 1 -- 172.21.15.6:0/207707981 wait complete. 2024-03-06T07:34:00.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:34:00.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.845+0000 7f94cfe00640 1 Processor -- start 2024-03-06T07:34:00.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.845+0000 7f94cfe00640 1 -- start start 2024-03-06T07:34:00.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cfe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f94c80cf500 0x7f94c80d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cfe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c815e640 con 0x7f94c815e260 2024-03-06T07:34:00.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cfe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c80d4ca0 con 0x7f94c80b43a0 2024-03-06T07:34:00.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cfe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c80cfa70 con 0x7f94c80cf500 2024-03-06T07:34:00.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cd400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94c815e260 0x7f94c80d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46934/0 (socket says 172.21.15.6:46934) 2024-03-06T07:34:00.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cd400640 1 -- 172.21.15.6:0/1262923630 learned_addr learned my addr 172.21.15.6:0/1262923630 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:00.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94cca00640 1 --2- 172.21.15.6:0/1262923630 >> v2:172.21.15.178:3300/0 conn(0x7f94c80cf500 0x7f94c80d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.848+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713383698 0 0) 0x7f94c815e640 con 0x7f94c815e260 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94b0003520 con 0x7f94c815e260 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4107308245 0 0) 0x7f94b0003520 con 0x7f94c815e260 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 >> v2:172.21.15.178:3300/0 conn(0x7f94c80cf500 msgr2=0x7f94c80d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 --2- 172.21.15.6:0/1262923630 >> v2:172.21.15.178:3300/0 conn(0x7f94c80cf500 0x7f94c80d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 >> v1:172.21.15.178:6789/0 conn(0x7f94c80b43a0 legacy=0x7f94c80b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c80cfd50 con 0x7f94c815e260 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f94bc001670 con 0x7f94c815e260 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 >> v1:172.21.15.6:6789/0 conn(0x7f94c815e260 legacy=0x7f94c80d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94b0005210 0x7f94b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80cfa70 con 0x7f94b00039e0 2024-03-06T07:34:00.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b000c140 con 0x7f94b0004800 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b000c340 con 0x7f94b0005210 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.849+0000 7f94cd400640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cde00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94b0005210 0x7f94b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cd400640 1 -- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94b0005210 msgr2=0x7f94b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cca00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cd400640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94b0005210 0x7f94b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cd400640 1 -- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94b0004800 msgr2=0x7f94b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cd400640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cd400640 1 -- 172.21.15.6:0/1262923630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b0003520 con 0x7f94b00039e0 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cca00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cde00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94b0005210 0x7f94b000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.850+0000 7f94cde00640 1 -- 172.21.15.6:0/1262923630 reap_dead start 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.851+0000 7f94cd400640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94b0003e00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00c240 tx=0x7f94b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce46f97cf2327936 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.851+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc007d10 con 0x7f94b00039e0 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.851+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc030070 con 0x7f94b00039e0 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.851+0000 7f94c7e00640 1 -- 172.21.15.6:0/1262923630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02cda0 con 0x7f94b00039e0 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 -- 172.21.15.6:0/1262923630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 msgr2=0x7f94b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94b0003e00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00c240 tx=0x7f94b0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 -- 172.21.15.6:0/1262923630 shutdown_connections 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 --2- 172.21.15.6:0/1262923630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94b0003e00 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 -- 172.21.15.6:0/1262923630 >> 172.21.15.6:0/1262923630 conn(0x7f94c80d3ee0 msgr2=0x7f94c80d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 -- 172.21.15.6:0/1262923630 shutdown_connections 2024-03-06T07:34:00.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.852+0000 7f94cfe00640 1 -- 172.21.15.6:0/1262923630 wait complete. 2024-03-06T07:34:00.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.854+0000 7f94cfe00640 1 Processor -- start 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.854+0000 7f94cfe00640 1 -- start start 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.855+0000 7f94cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94c815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.855+0000 7f94cfe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94c814e070 0x7f94c8160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.855+0000 7f94cfe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94c814d3a0 0x7f94c814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.855+0000 7f94cfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b3e80 con 0x7f94b00039e0 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.855+0000 7f94cfe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80b3ba0 con 0x7f94c814d3a0 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.855+0000 7f94cfe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c80d1ab0 con 0x7f94c814e070 2024-03-06T07:34:00.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94c815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94c815fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49686/0 (socket says 172.21.15.6:49686) 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 -- 172.21.15.6:0/3432249274 learned_addr learned my addr 172.21.15.6:0/3432249274 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cca00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94c814e070 0x7f94c8160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cde00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94c814d3a0 0x7f94c814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 -- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94c814e070 msgr2=0x7f94c8160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94c814e070 0x7f94c8160b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 -- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94c814d3a0 msgr2=0x7f94c814d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94c814d3a0 0x7f94c814d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 -- 172.21.15.6:0/3432249274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c814dd40 con 0x7f94b00039e0 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cca00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94c814e070 0x7f94c8160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cde00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94c814d3a0 0x7f94c814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.856+0000 7f94cd400640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94c815fd40 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f94bc001970 tx=0x7f94bc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.857+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc004bf0 con 0x7f94b00039e0 2024-03-06T07:34:00.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.857+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc004d90 con 0x7f94b00039e0 2024-03-06T07:34:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.857+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc007290 con 0x7f94b00039e0 2024-03-06T07:34:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.857+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c8160350 con 0x7f94b00039e0 2024-03-06T07:34:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.857+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c80cb3c0 con 0x7f94b00039e0 2024-03-06T07:34:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.858+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c804f330 con 0x7f94b00039e0 2024-03-06T07:34:00.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.859+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94bc048020 con 0x7f94b00039e0 2024-03-06T07:34:00.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.861+0000 7f94c5600640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94a806b4d0 0x7f94a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:00.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.861+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94bc0ab730 con 0x7f94b00039e0 2024-03-06T07:34:00.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.862+0000 7f94cca00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94a806b4d0 0x7f94a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:00.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.863+0000 7f94cca00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94a806b4d0 0x7f94a806d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f94c815ce00 tx=0x7f94b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:00.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:00.865+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94c814dd40 con 0x7f94b00039e0 2024-03-06T07:34:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.066+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94c815a950 con 0x7f94b00039e0 2024-03-06T07:34:01.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.066+0000 7f94c5600640 1 -- 172.21.15.6:0/3432249274 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7f94bc034e90 con 0x7f94b00039e0 2024-03-06T07:34:01.068 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:34:01.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94a806b4d0 msgr2=0x7f94a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94a806b4d0 0x7f94a806d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f94c815ce00 tx=0x7f94b4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:01.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 msgr2=0x7f94c815fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94c815fd40 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f94bc001970 tx=0x7f94bc02e070 comp rx=0 tx=0).stop 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 shutdown_connections 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f94a806b4d0 0x7f94a806d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f94c814d3a0 0x7f94c814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f94c814e070 0x7f94c8160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 --2- 172.21.15.6:0/3432249274 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94b00039e0 0x7f94c815fd40 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 >> 172.21.15.6:0/3432249274 conn(0x7f94c80d3ee0 msgr2=0x7f94c806d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 shutdown_connections 2024-03-06T07:34:01.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.069+0000 7f94cfe00640 1 -- 172.21.15.6:0/3432249274 wait complete. 2024-03-06T07:34:01.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:34:01.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.300+0000 7f5a0b200640 1 Processor -- start 2024-03-06T07:34:01.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.301+0000 7f5a0b200640 1 -- start start 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.302+0000 7f5a0b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5a0c067410 0x7f5a0c0af600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.302+0000 7f5a0b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0aee70 con 0x7f5a0c15e070 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.302+0000 7f5a0b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a0c0b4f80 con 0x7f5a0c0b4200 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.302+0000 7f5a0b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c15e450 con 0x7f5a0c067410 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.303+0000 7f5a09e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5a0c15e070 0x7f5a0c06dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46944/0 (socket says 172.21.15.6:46944) 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.303+0000 7f5a09e00640 1 -- 172.21.15.6:0/3880406683 learned_addr learned my addr 172.21.15.6:0/3880406683 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:01.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.303+0000 7f5a0a800640 1 --2- 172.21.15.6:0/3880406683 >> v2:172.21.15.178:3300/0 conn(0x7f5a0c067410 0x7f5a0c0af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.303+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709640248 0 0) 0x7f5a0c0aee70 con 0x7f5a0c15e070 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.303+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0003520 con 0x7f5a0c15e070 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 234305751 0 0) 0x7f59f0003520 con 0x7f5a0c15e070 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 >> v2:172.21.15.178:3300/0 conn(0x7f5a0c067410 msgr2=0x7f5a0c0af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 --2- 172.21.15.6:0/3880406683 >> v2:172.21.15.178:3300/0 conn(0x7f5a0c067410 0x7f5a0c0af600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 >> v1:172.21.15.178:6789/0 conn(0x7f5a0c0b4200 legacy=0x7f5a0c066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0b5260 con 0x7f5a0c15e070 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f59fc001670 con 0x7f5a0c15e070 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 >> v1:172.21.15.6:6789/0 conn(0x7f5a0c15e070 legacy=0x7f5a0c06dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f59f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59f0004800 0x7f59f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59f00051c0 0x7f59f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c15e450 con 0x7f59f00039b0 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f000c160 con 0x7f59f00051c0 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f000c360 con 0x7f59f0004800 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.304+0000 7f5a09e00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f59f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a09e00640 1 -- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59f0004800 msgr2=0x7f59f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a0a800640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59f00051c0 0x7f59f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a09e00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59f0004800 0x7f59f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a09e00640 1 -- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59f00051c0 msgr2=0x7f59f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a09e00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59f00051c0 0x7f59f000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a09e00640 1 -- 172.21.15.6:0/3880406683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f0003520 con 0x7f59f00039b0 2024-03-06T07:34:01.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.305+0000 7f5a0a800640 1 -- 172.21.15.6:0/3880406683 reap_dead start 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.306+0000 7f5a09e00640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f59f0003dd0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c210 tx=0x7f59f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652b48b264b9951 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.306+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc007d30 con 0x7f59f00039b0 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.306+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc030070 con 0x7f59f00039b0 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.306+0000 7f5a08a00640 1 -- 172.21.15.6:0/3880406683 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc005d70 con 0x7f59f00039b0 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.306+0000 7f5a0b200640 1 -- 172.21.15.6:0/3880406683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 msgr2=0x7f59f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.306+0000 7f5a0b200640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f59f0003dd0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c210 tx=0x7f59f0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.307+0000 7f5a0b200640 1 -- 172.21.15.6:0/3880406683 shutdown_connections 2024-03-06T07:34:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.307+0000 7f5a0b200640 1 --2- 172.21.15.6:0/3880406683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f59f0003dd0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.307+0000 7f5a0b200640 1 -- 172.21.15.6:0/3880406683 >> 172.21.15.6:0/3880406683 conn(0x7f5a0c0729a0 msgr2=0x7f5a0c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.307+0000 7f5a0b200640 1 -- 172.21.15.6:0/3880406683 shutdown_connections 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.307+0000 7f5a0b200640 1 -- 172.21.15.6:0/3880406683 wait complete. 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.308+0000 7f5a0b200640 1 Processor -- start 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.308+0000 7f5a0b200640 1 -- start start 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.309+0000 7f5a0b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f5a0c14c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.309+0000 7f5a0b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a0c0c8f50 0x7f5a0c0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.309+0000 7f5a0b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a0c1e94e0 0x7f5a0c1e9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.309+0000 7f5a0b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0b4a00 con 0x7f59f00039b0 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.309+0000 7f5a0b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c06e2c0 con 0x7f5a0c1e94e0 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.309+0000 7f5a0b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c0b5540 con 0x7f5a0c0c8f50 2024-03-06T07:34:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f5a0c14c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f5a0c14c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49700/0 (socket says 172.21.15.6:49700) 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 -- 172.21.15.6:0/2176159583 learned_addr learned my addr 172.21.15.6:0/2176159583 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 -- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a0c0c8f50 msgr2=0x7f5a0c0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a0c0c8f50 0x7f5a0c0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 -- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a0c1e94e0 msgr2=0x7f5a0c1e9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a0c1e94e0 0x7f5a0c1e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 -- 172.21.15.6:0/2176159583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0c6900 con 0x7f59f00039b0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.310+0000 7f5a09e00640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f5a0c14c390 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c330 tx=0x7f59fc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.311+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc004bf0 con 0x7f59f00039b0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.311+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc000ed0 con 0x7f59f00039b0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.311+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c0c6bc0 con 0x7f59f00039b0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.311+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc007d10 con 0x7f59f00039b0 2024-03-06T07:34:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.311+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c14c8d0 con 0x7f59f00039b0 2024-03-06T07:34:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.312+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c06c540 con 0x7f59f00039b0 2024-03-06T07:34:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.314+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59fc048020 con 0x7f59f00039b0 2024-03-06T07:34:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.315+0000 7f5a02000640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59ec06b4d0 0x7f59ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.315+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59fc0ab3e0 con 0x7f59f00039b0 2024-03-06T07:34:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.315+0000 7f5a09400640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59ec06b4d0 0x7f59ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.316+0000 7f5a09400640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59ec06b4d0 0x7f59ec06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0cff30 tx=0x7f59f8007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.321+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5a0c06c540 con 0x7f59f00039b0 2024-03-06T07:34:01.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.561+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a0c15dbd0 con 0x7f59f00039b0 2024-03-06T07:34:01.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.562+0000 7f5a02000640 1 -- 172.21.15.6:0/2176159583 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a0c15dbd0 con 0x7f59f00039b0 2024-03-06T07:34:01.564 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59ec06b4d0 msgr2=0x7f59ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59ec06b4d0 0x7f59ec06d990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0cff30 tx=0x7f59f8007990 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 msgr2=0x7f5a0c14c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f5a0c14c390 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c330 tx=0x7f59fc02e070 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 shutdown_connections 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f59ec06b4d0 0x7f59ec06d990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5a0c1e94e0 0x7f5a0c1e9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5a0c0c8f50 0x7f5a0c0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 --2- 172.21.15.6:0/2176159583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59f00039b0 0x7f5a0c14c390 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.564+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 >> 172.21.15.6:0/2176159583 conn(0x7f5a0c0729a0 msgr2=0x7f5a0c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.565+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 shutdown_connections 2024-03-06T07:34:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.565+0000 7f5a0b200640 1 -- 172.21.15.6:0/2176159583 wait complete. 2024-03-06T07:34:01.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:34:01.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.796+0000 7f2885400640 1 Processor -- start 2024-03-06T07:34:01.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.797+0000 7f2885400640 1 -- start start 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2885400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f288015dec0 0x7f28800ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2885400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f288015e2a0 0x7f28800ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2885400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28800cf700 con 0x7f28800cf290 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2885400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28800b0ea0 con 0x7f288015e2a0 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2885400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28800d1120 con 0x7f288015dec0 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2877e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f288015e2a0 0x7f28800ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2877e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f288015e2a0 0x7f28800ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49716/0 (socket says 172.21.15.6:49716) 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.798+0000 7f2877e00640 1 -- 172.21.15.6:0/1483816989 learned_addr learned my addr 172.21.15.6:0/1483816989 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.799+0000 7f287fe00640 1 --2- 172.21.15.6:0/1483816989 >> v2:172.21.15.178:3300/0 conn(0x7f288015dec0 0x7f28800ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.799+0000 7f2877e00640 1 -- 172.21.15.6:0/1483816989 >> v2:172.21.15.178:3300/0 conn(0x7f288015dec0 msgr2=0x7f28800ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.799+0000 7f2877e00640 1 --2- 172.21.15.6:0/1483816989 >> v2:172.21.15.178:3300/0 conn(0x7f288015dec0 0x7f28800ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.799+0000 7f2877e00640 1 -- 172.21.15.6:0/1483816989 >> v1:172.21.15.178:6789/0 conn(0x7f28800cf290 legacy=0x7f28800d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:01.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.799+0000 7f2877e00640 1 -- 172.21.15.6:0/1483816989 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800d1400 con 0x7f288015e2a0 2024-03-06T07:34:01.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.800+0000 7f2877e00640 1 --2- 172.21.15.6:0/1483816989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f288015e2a0 0x7f28800ced50 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f288008c780 tx=0x7f286802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3eb1c3ff16e43f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.800+0000 7f287f400640 1 -- 172.21.15.6:0/1483816989 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868035020 con 0x7f288015e2a0 2024-03-06T07:34:01.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f287f400640 1 -- 172.21.15.6:0/1483816989 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2868031870 con 0x7f288015e2a0 2024-03-06T07:34:01.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f287f400640 1 -- 172.21.15.6:0/1483816989 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2868031c40 con 0x7f288015e2a0 2024-03-06T07:34:01.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 -- 172.21.15.6:0/1483816989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f288015e2a0 msgr2=0x7f28800ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 --2- 172.21.15.6:0/1483816989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f288015e2a0 0x7f28800ced50 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f288008c780 tx=0x7f286802b910 comp rx=0 tx=0).stop 2024-03-06T07:34:01.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 -- 172.21.15.6:0/1483816989 shutdown_connections 2024-03-06T07:34:01.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 --2- 172.21.15.6:0/1483816989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f288015e2a0 0x7f28800ced50 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 --2- 172.21.15.6:0/1483816989 >> v2:172.21.15.178:3300/0 conn(0x7f288015dec0 0x7f28800ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 -- 172.21.15.6:0/1483816989 >> 172.21.15.6:0/1483816989 conn(0x7f28800d3d40 msgr2=0x7f28800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:01.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 -- 172.21.15.6:0/1483816989 shutdown_connections 2024-03-06T07:34:01.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.801+0000 7f2885400640 1 -- 172.21.15.6:0/1483816989 wait complete. 2024-03-06T07:34:01.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.803+0000 7f2885400640 1 Processor -- start 2024-03-06T07:34:01.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.803+0000 7f2885400640 1 -- start start 2024-03-06T07:34:01.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.804+0000 7f2885400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 0x7f288015bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.804+0000 7f2885400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f288015dec0 0x7f28801f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.804+0000 7f2885400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f288015e2a0 0x7f28800d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.804+0000 7f2885400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800d0ec0 con 0x7f28800cf290 2024-03-06T07:34:01.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.804+0000 7f2885400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28800d0d40 con 0x7f288015dec0 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.804+0000 7f2885400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28800b2340 con 0x7f288015e2a0 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 0x7f288015bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 0x7f288015bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49732/0 (socket says 172.21.15.6:49732) 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 -- 172.21.15.6:0/636092728 learned_addr learned my addr 172.21.15.6:0/636092728 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f2877e00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f288015dec0 0x7f28801f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 -- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f288015e2a0 msgr2=0x7f28800d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f288015e2a0 0x7f28800d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 -- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f288015dec0 msgr2=0x7f28801f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f288015dec0 0x7f28801f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 -- 172.21.15.6:0/636092728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28800c8020 con 0x7f28800cf290 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f2877e00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f288015dec0 0x7f28801f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.805+0000 7f287fe00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 0x7f288015bfa0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f287000d800 tx=0x7f287000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.806+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000b6c0 con 0x7f28800cf290 2024-03-06T07:34:01.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.806+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2870019070 con 0x7f28800cf290 2024-03-06T07:34:01.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.806+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28800c82e0 con 0x7f28800cf290 2024-03-06T07:34:01.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.806+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000ba80 con 0x7f28800cf290 2024-03-06T07:34:01.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.806+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28800c8b80 con 0x7f28800cf290 2024-03-06T07:34:01.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.807+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28800c86d0 con 0x7f28800cf290 2024-03-06T07:34:01.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.808+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f287000bc20 con 0x7f28800cf290 2024-03-06T07:34:01.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.809+0000 7f287cc00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f285c06b4d0 0x7f285c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:01.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.809+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f287008a730 con 0x7f28800cf290 2024-03-06T07:34:01.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.810+0000 7f2877e00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f285c06b4d0 0x7f285c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:01.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.811+0000 7f2877e00640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f285c06b4d0 0x7f285c06d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2868036000 tx=0x7f2868008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:01.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:01.816+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2870059510 con 0x7f28800cf290 2024-03-06T07:34:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.028+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f28800c84b0 con 0x7f28800cf290 2024-03-06T07:34:02.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.029+0000 7f287cc00640 1 -- 172.21.15.6:0/636092728 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v108) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2870058e10 con 0x7f28800cf290 2024-03-06T07:34:02.031 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:34:02.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f285c06b4d0 msgr2=0x7f285c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f285c06b4d0 0x7f285c06d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2868036000 tx=0x7f2868008030 comp rx=0 tx=0).stop 2024-03-06T07:34:02.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 msgr2=0x7f288015bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 0x7f288015bfa0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f287000d800 tx=0x7f287000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 shutdown_connections 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f285c06b4d0 0x7f285c06d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f288015dec0 0x7f28801f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f288015e2a0 0x7f28800d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 --2- 172.21.15.6:0/636092728 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28800cf290 0x7f288015bfa0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 >> 172.21.15.6:0/636092728 conn(0x7f28800d3d40 msgr2=0x7f288015e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 shutdown_connections 2024-03-06T07:34:02.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.031+0000 7f2885400640 1 -- 172.21.15.6:0/636092728 wait complete. 2024-03-06T07:34:02.051 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:34:02.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:02.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:02.165 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:34:02.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:02.166 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:34:02.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.395+0000 7fca42000640 1 Processor -- start 2024-03-06T07:34:02.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.396+0000 7fca42000640 1 -- start start 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.397+0000 7fca42000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 0x7fca3c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.397+0000 7fca42000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fca3c0d0cd0 0x7fca3c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.397+0000 7fca42000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0d18c0 con 0x7fca3c0cf410 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.397+0000 7fca42000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c0d1ba0 con 0x7fca3c0d0cd0 2024-03-06T07:34:02.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.397+0000 7fca42000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0d1e80 con 0x7fca3c15e070 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 0x7fca3c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca41600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fca3c0d0cd0 0x7fca3c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 0x7fca3c0d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:50616/0 (socket says 172.21.15.6:50616) 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 -- 172.21.15.6:0/2892140346 learned_addr learned my addr 172.21.15.6:0/2892140346 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:02.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 -- 172.21.15.6:0/2892140346 >> v2:172.21.15.178:3300/0 conn(0x7fca3c0d0cd0 msgr2=0x7fca3c0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 --2- 172.21.15.6:0/2892140346 >> v2:172.21.15.178:3300/0 conn(0x7fca3c0d0cd0 0x7fca3c0cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 -- 172.21.15.6:0/2892140346 >> v1:172.21.15.178:6789/0 conn(0x7fca3c0cf410 legacy=0x7fca3c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.398+0000 7fca40c00640 1 -- 172.21.15.6:0/2892140346 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0d2160 con 0x7fca3c15e070 2024-03-06T07:34:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.400+0000 7fca40c00640 1 --2- 172.21.15.6:0/2892140346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 0x7fca3c0d2960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca30009870 tx=0x7fca3002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9e65ba13d7d3f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.401+0000 7fca3b400640 1 -- 172.21.15.6:0/2892140346 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30034070 con 0x7fca3c15e070 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.401+0000 7fca3b400640 1 -- 172.21.15.6:0/2892140346 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca30030840 con 0x7fca3c15e070 2024-03-06T07:34:02.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.401+0000 7fca3b400640 1 -- 172.21.15.6:0/2892140346 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30030c10 con 0x7fca3c15e070 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.401+0000 7fca42000640 1 -- 172.21.15.6:0/2892140346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 msgr2=0x7fca3c0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.401+0000 7fca42000640 1 --2- 172.21.15.6:0/2892140346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 0x7fca3c0d2960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca30009870 tx=0x7fca3002b980 comp rx=0 tx=0).stop 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.402+0000 7fca42000640 1 -- 172.21.15.6:0/2892140346 shutdown_connections 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.402+0000 7fca42000640 1 --2- 172.21.15.6:0/2892140346 >> v2:172.21.15.178:3300/0 conn(0x7fca3c0d0cd0 0x7fca3c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.402+0000 7fca42000640 1 --2- 172.21.15.6:0/2892140346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c15e070 0x7fca3c0d2960 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.402+0000 7fca42000640 1 -- 172.21.15.6:0/2892140346 >> 172.21.15.6:0/2892140346 conn(0x7fca3c08d2a0 msgr2=0x7fca3c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.402+0000 7fca42000640 1 -- 172.21.15.6:0/2892140346 shutdown_connections 2024-03-06T07:34:02.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.402+0000 7fca42000640 1 -- 172.21.15.6:0/2892140346 wait complete. 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.403+0000 7fca42000640 1 Processor -- start 2024-03-06T07:34:02.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.403+0000 7fca42000640 1 -- start start 2024-03-06T07:34:02.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.404+0000 7fca42000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 0x7fca3c0c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.404+0000 7fca42000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c0d0cd0 0x7fca3c0b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.404+0000 7fca42000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca3c15e070 0x7fca3c0b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.404+0000 7fca42000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0d4730 con 0x7fca3c0cf410 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.404+0000 7fca42000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0d45b0 con 0x7fca3c15e070 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.404+0000 7fca42000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0b3630 con 0x7fca3c0d0cd0 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 0x7fca3c0c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 0x7fca3c0c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36688/0 (socket says 172.21.15.6:36688) 2024-03-06T07:34:02.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 -- 172.21.15.6:0/1419328502 learned_addr learned my addr 172.21.15.6:0/1419328502 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 -- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c0d0cd0 msgr2=0x7fca3c0b91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c0d0cd0 0x7fca3c0b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 -- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca3c15e070 msgr2=0x7fca3c0b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca3c15e070 0x7fca3c0b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 -- 172.21.15.6:0/1419328502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0ca510 con 0x7fca3c0cf410 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.405+0000 7fca40c00640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 0x7fca3c0c9fd0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fca30000c00 tx=0x7fca300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.406+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30034040 con 0x7fca3c0cf410 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.406+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca30039d10 con 0x7fca3c0cf410 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.406+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca30043630 con 0x7fca3c0cf410 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.406+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c0b6ab0 con 0x7fca3c0cf410 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.406+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c0b7030 con 0x7fca3c0cf410 2024-03-06T07:34:02.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.407+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c04f330 con 0x7fca3c0cf410 2024-03-06T07:34:02.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.408+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca3004a060 con 0x7fca3c0cf410 2024-03-06T07:34:02.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.409+0000 7fca38c00640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca1c06b4d0 0x7fca1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:02.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.409+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca300ac190 con 0x7fca3c0cf410 2024-03-06T07:34:02.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.409+0000 7fca3be00640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca1c06b4d0 0x7fca1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:02.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.411+0000 7fca3be00640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca1c06b4d0 0x7fca1c06d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fca28000f60 tx=0x7fca280079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:02.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.413+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca3c0ca510 con 0x7fca3c0cf410 2024-03-06T07:34:02.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.650+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca3c0d1370 con 0x7fca3c0cf410 2024-03-06T07:34:02.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.651+0000 7fca38c00640 1 -- 172.21.15.6:0/1419328502 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fca3003e070 con 0x7fca3c0cf410 2024-03-06T07:34:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:02.654 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5795,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/3411419645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3411419645},{"type":"v1","addr":"172.21.15.6:6835","nonce":3411419645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5798,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/525628181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":525628181},{"type":"v1","addr":"172.21.15.178:6835","nonce":525628181}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73},{"gid":5801,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/319127923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":319127923},{"type":"v1","addr":"172.21.15.178:6837","nonce":319127923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:33:51.764337+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7656},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7656":{"gid":7656,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6833/4114489444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":4114489444},{"type":"v1","addr":"172.21.15.6:6833","nonce":4114489444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7656,"qdb_cluster":[7656]},"id":8}]} 2024-03-06T07:34:02.654 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 77 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.653+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca1c06b4d0 msgr2=0x7fca1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.653+0000 7fca42000640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca1c06b4d0 0x7fca1c06d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fca28000f60 tx=0x7fca280079d0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.653+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 msgr2=0x7fca3c0c9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.653+0000 7fca42000640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 0x7fca3c0c9fd0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fca30000c00 tx=0x7fca300046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 shutdown_connections 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca1c06b4d0 0x7fca1c06d990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca3c15e070 0x7fca3c0b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca3c0d0cd0 0x7fca3c0b91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 --2- 172.21.15.6:0/1419328502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca3c0cf410 0x7fca3c0c9fd0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 >> 172.21.15.6:0/1419328502 conn(0x7fca3c08d2a0 msgr2=0x7fca3c0d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 shutdown_connections 2024-03-06T07:34:02.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:02.654+0000 7fca42000640 1 -- 172.21.15.6:0/1419328502 wait complete. 2024-03-06T07:34:02.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:34:02.912 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:34:02.912 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:02.913 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:34:02.913 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:02.913 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:34:02.913 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:02.914 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:34:02.914 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:02.914 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:34:02.914 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:34:02.933 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:34:02.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:34:02.935 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:34:02.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:34:02.936 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:34:02.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:34:02.937 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:34:02.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:34:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.156+0000 7f128ec00640 1 Processor -- start 2024-03-06T07:34:03.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.156+0000 7f128ec00640 1 -- start start 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128ec00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 0x7f12880cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128ec00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f12880cce90 0x7f128815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128ec00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12880714d0 con 0x7f1288145300 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128ec00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12880cd300 con 0x7f12880cce90 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128ec00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12880717b0 con 0x7f1288070f10 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1288145300 0x7f12880b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43314/0 (socket says 172.21.15.6:43314) 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128d800640 1 -- 172.21.15.6:0/2389766418 learned_addr learned my addr 172.21.15.6:0/2389766418 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:03.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128e200640 1 --2- 172.21.15.6:0/2389766418 >> v2:172.21.15.178:3300/0 conn(0x7f12880cce90 0x7f128815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.158+0000 7f128ce00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 0x7f12880cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931849392 0 0) 0x7f12880714d0 con 0x7f1288145300 2024-03-06T07:34:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1270003520 con 0x7f1288145300 2024-03-06T07:34:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839035600 0 0) 0x7f1270003520 con 0x7f1288145300 2024-03-06T07:34:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 msgr2=0x7f12880cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 0x7f12880cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 >> v2:172.21.15.178:3300/0 conn(0x7f12880cce90 msgr2=0x7f128815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 --2- 172.21.15.6:0/2389766418 >> v2:172.21.15.178:3300/0 conn(0x7f12880cce90 0x7f128815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288071a90 con 0x7f1288145300 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f127c001670 con 0x7f1288145300 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 >> v1:172.21.15.6:6789/0 conn(0x7f1288145300 legacy=0x7f12880b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12700039b0 0x7f1270003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1270004800 0x7f1270004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 0x7f12700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12880cd300 con 0x7f1270004800 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12880717b0 con 0x7f12700039b0 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.159+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127000b9c0 con 0x7f1288070f10 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128ce00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1270004800 0x7f1270004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128d800640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12700039b0 0x7f1270003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128e200640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 0x7f12700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128ce00640 1 -- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 msgr2=0x7f12700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128ce00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1288070f10 0x7f12700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128ce00640 1 -- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12700039b0 msgr2=0x7f1270003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128ce00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12700039b0 0x7f1270003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128ce00640 1 -- 172.21.15.6:0/2389766418 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1270003520 con 0x7f1270004800 2024-03-06T07:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.160+0000 7f128d800640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12700039b0 0x7f1270003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.161+0000 7f128ce00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1270004800 0x7f1270004c80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f1274009ac0 tx=0x7f1270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17c4730465ddd4d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274039070 con 0x7f1270004800 2024-03-06T07:34:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127402c690 con 0x7f1270004800 2024-03-06T07:34:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f1287e00640 1 -- 172.21.15.6:0/2389766418 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274009690 con 0x7f1270004800 2024-03-06T07:34:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f8654800640 1 Processor -- start 2024-03-06T07:34:03.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f8654800640 1 -- start start 2024-03-06T07:34:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f128ec00640 1 -- 172.21.15.6:0/2389766418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1270004800 msgr2=0x7f1270004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f128ec00640 1 --2- 172.21.15.6:0/2389766418 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1270004800 0x7f1270004c80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f1274009ac0 tx=0x7f1270005830 comp rx=0 tx=0).stop 2024-03-06T07:34:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.162+0000 7f128e200640 1 -- 172.21.15.6:0/2389766418 reap_dead start 2024-03-06T07:34:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f128ec00640 1 -- 172.21.15.6:0/2389766418 shutdown_connections 2024-03-06T07:34:03.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f128ec00640 1 -- 172.21.15.6:0/2389766418 >> 172.21.15.6:0/2389766418 conn(0x7f128806b3d0 msgr2=0x7f128806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f128ec00640 1 -- 172.21.15.6:0/2389766418 shutdown_connections 2024-03-06T07:34:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f128ec00640 1 -- 172.21.15.6:0/2389766418 wait complete. 2024-03-06T07:34:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f272fc00640 1 Processor -- start 2024-03-06T07:34:03.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f272fc00640 1 -- start start 2024-03-06T07:34:03.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f8654800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0d2b40 0x7f864c0d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f8654800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0b3940 con 0x7f864c15e2e0 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f8654800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0cf580 con 0x7f864c0d1480 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.163+0000 7f8654800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0d4e70 con 0x7f864c0d2b40 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.164+0000 7f8651e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f864c15e2e0 0x7f864c0d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43328/0 (socket says 172.21.15.6:43328) 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.164+0000 7f8651e00640 1 -- 172.21.15.6:0/20693850 learned_addr learned my addr 172.21.15.6:0/20693850 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.164+0000 7f8651400640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0d2b40 0x7f864c0d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.164+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377739960 0 0) 0x7f864c0b3940 con 0x7f864c15e2e0 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.164+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8634003520 con 0x7f864c15e2e0 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1374121703 0 0) 0x7f8634003520 con 0x7f864c15e2e0 2024-03-06T07:34:03.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0d2b40 msgr2=0x7f864c0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272fc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f272815e0c0 0x7f27280d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e4a0 0x7f27280d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0d2b40 0x7f864c0d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 >> v1:172.21.15.178:6789/0 conn(0x7f864c0d1480 legacy=0x7f864c0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0cf860 con 0x7f864c15e2e0 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8640001670 con 0x7f864c15e2e0 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 >> v1:172.21.15.6:6789/0 conn(0x7f864c15e2e0 legacy=0x7f864c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8634003a50 0x7f8634003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86340043b0 0x7f8634004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8634005080 0x7f863400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863400bfb0 con 0x7f86340043b0 2024-03-06T07:34:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863400c130 con 0x7f8634005080 2024-03-06T07:34:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863400c330 con 0x7f8634003a50 2024-03-06T07:34:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272fc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27280d1160 con 0x7f27280d29f0 2024-03-06T07:34:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272fc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27280d0f10 con 0x7f272815e0c0 2024-03-06T07:34:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280d21f0 con 0x7f272815e4a0 2024-03-06T07:34:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f128ec00640 1 Processor -- start 2024-03-06T07:34:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272dc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f27280d29f0 0x7f27280ce9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43334/0 (socket says 172.21.15.6:43334) 2024-03-06T07:34:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f272dc00640 1 -- 172.21.15.6:0/2393421080 learned_addr learned my addr 172.21.15.6:0/2393421080 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.165+0000 7f128ec00640 1 -- start start 2024-03-06T07:34:03.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f8652800640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8634005080 0x7f863400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f272d200640 1 --2- 172.21.15.6:0/2393421080 >> v2:172.21.15.6:3300/0 conn(0x7f272815e0c0 0x7f27280d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727e00640 1 --2- 172.21.15.6:0/2393421080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e4a0 0x7f27280d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f8651e00640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8634003a50 0x7f8634003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727400640 1 -- 172.21.15.6:0/2393421080 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121756524 0 0) 0x7f27280d1160 con 0x7f27280d29f0 2024-03-06T07:34:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727400640 1 -- 172.21.15.6:0/2393421080 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710003520 con 0x7f27280d29f0 2024-03-06T07:34:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f8651400640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86340043b0 0x7f8634004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727e00640 1 -- 172.21.15.6:0/2393421080 >> v2:172.21.15.6:3300/0 conn(0x7f272815e0c0 msgr2=0x7f27280d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727400640 1 -- 172.21.15.6:0/2393421080 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1119317950 0 0) 0x7f2710003520 con 0x7f27280d29f0 2024-03-06T07:34:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f8652800640 1 -- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8634003a50 msgr2=0x7f8634003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f8652800640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8634003a50 0x7f8634003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f8652800640 1 -- 172.21.15.6:0/20693850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86340043b0 msgr2=0x7f8634004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727e00640 1 --2- 172.21.15.6:0/2393421080 >> v2:172.21.15.6:3300/0 conn(0x7f272815e0c0 0x7f27280d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727e00640 1 -- 172.21.15.6:0/2393421080 >> v1:172.21.15.6:6789/0 conn(0x7f27280d29f0 legacy=0x7f27280ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f2727e00640 1 -- 172.21.15.6:0/2393421080 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27280d1440 con 0x7f272815e4a0 2024-03-06T07:34:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f128ec00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1270004800 0x7f12880acf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f128ec00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 0x7f12881f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.166+0000 7f128ec00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12881f8e30 0x7f12880ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128ec00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12880cc950 con 0x7f12881f8710 2024-03-06T07:34:03.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128ec00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12881456e0 con 0x7f12881f8e30 2024-03-06T07:34:03.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128ec00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12880ccad0 con 0x7f1270004800 2024-03-06T07:34:03.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f8652800640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f86340043b0 0x7f8634004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f8652800640 1 -- 172.21.15.6:0/20693850 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0cf860 con 0x7f8634005080 2024-03-06T07:34:03.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f272d200640 1 --2- 172.21.15.6:0/2393421080 >> v2:172.21.15.6:3300/0 conn(0x7f272815e0c0 0x7f27280d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:03.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1270004800 0x7f12880acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 0x7f12881f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128d800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1270004800 0x7f12880acf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:50694/0 (socket says 172.21.15.6:50694) 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 0x7f12881f7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36732/0 (socket says 172.21.15.6:36732) 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128d800640 1 -- 172.21.15.6:0/1953524258 learned_addr learned my addr 172.21.15.6:0/1953524258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.167+0000 7f128e200640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12881f8e30 0x7f12880ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128ce00640 1 -- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1270004800 msgr2=0x7f12880acf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128ce00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1270004800 0x7f12880acf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128ce00640 1 -- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12881f8e30 msgr2=0x7f12880ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128ce00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12881f8e30 0x7f12880ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128ce00640 1 -- 172.21.15.6:0/1953524258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12881f7ad0 con 0x7f12881f8710 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128e200640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12881f8e30 0x7f12880ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f128ce00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 0x7f12881f7590 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f12700046a0 tx=0x7f1274004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f2727e00640 1 --2- 172.21.15.6:0/2393421080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e4a0 0x7f27280d24b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27140097a0 tx=0x7f271402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f0b02179b41446f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274042020 con 0x7f12881f8710 2024-03-06T07:34:03.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.168+0000 7f8652800640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8634005080 0x7f863400b9c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8648001f60 tx=0x7f8634005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8ca72712318461d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f127403a040 con 0x7f12881f8710 2024-03-06T07:34:03.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1274039070 con 0x7f12881f8710 2024-03-06T07:34:03.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12880ad510 con 0x7f12881f8710 2024-03-06T07:34:03.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f2727400640 1 -- 172.21.15.6:0/2393421080 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714038070 con 0x7f272815e4a0 2024-03-06T07:34:03.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f2727400640 1 -- 172.21.15.6:0/2393421080 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2714031500 con 0x7f272815e4a0 2024-03-06T07:34:03.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f2727400640 1 -- 172.21.15.6:0/2393421080 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27200019f0 con 0x7f272815e4a0 2024-03-06T07:34:03.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12880aaad0 con 0x7f12881f8710 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.169+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648001b30 con 0x7f8634005080 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864802d710 con 0x7f8634005080 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f8650a00640 1 -- 172.21.15.6:0/20693850 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864802db00 con 0x7f8634005080 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f272fc00640 1 -- 172.21.15.6:0/2393421080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e4a0 msgr2=0x7f27280d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f272fc00640 1 --2- 172.21.15.6:0/2393421080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e4a0 0x7f27280d24b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27140097a0 tx=0x7f271402b820 comp rx=0 tx=0).stop 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f8654800640 1 -- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8634005080 msgr2=0x7f863400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f8654800640 1 --2- 172.21.15.6:0/20693850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8634005080 0x7f863400b9c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8648001f60 tx=0x7f8634005830 comp rx=0 tx=0).stop 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f272fc00640 1 -- 172.21.15.6:0/2393421080 shutdown_connections 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f272fc00640 1 --2- 172.21.15.6:0/2393421080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e4a0 0x7f27280d24b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f272fc00640 1 --2- 172.21.15.6:0/2393421080 >> v2:172.21.15.6:3300/0 conn(0x7f272815e0c0 0x7f27280d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f272fc00640 1 -- 172.21.15.6:0/2393421080 >> 172.21.15.6:0/2393421080 conn(0x7f27280d3ed0 msgr2=0x7f27280d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.170+0000 7f8652800640 1 -- 172.21.15.6:0/20693850 reap_dead start 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f8654800640 1 -- 172.21.15.6:0/20693850 shutdown_connections 2024-03-06T07:34:03.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f8654800640 1 -- 172.21.15.6:0/20693850 >> 172.21.15.6:0/20693850 conn(0x7f864c0d3f60 msgr2=0x7f864c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.171+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128804f330 con 0x7f12881f8710 2024-03-06T07:34:03.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.172+0000 7f8654800640 1 -- 172.21.15.6:0/20693850 shutdown_connections 2024-03-06T07:34:03.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.172+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1274045020 con 0x7f12881f8710 2024-03-06T07:34:03.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.172+0000 7f8654800640 1 -- 172.21.15.6:0/20693850 wait complete. 2024-03-06T07:34:03.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.172+0000 7f272fc00640 1 -- 172.21.15.6:0/2393421080 shutdown_connections 2024-03-06T07:34:03.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.172+0000 7f1285600640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f126806b4d0 0x7f126806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.173+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12740ab170 con 0x7f12881f8710 2024-03-06T07:34:03.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.173+0000 7f128d800640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f126806b4d0 0x7f126806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.173+0000 7f272fc00640 1 -- 172.21.15.6:0/2393421080 wait complete. 2024-03-06T07:34:03.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.174+0000 7f272fc00640 1 Processor -- start 2024-03-06T07:34:03.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.174+0000 7f128d800640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f126806b4d0 0x7f126806d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f127c000fb0 tx=0x7f127c007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.174+0000 7f272fc00640 1 -- start start 2024-03-06T07:34:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f8654800640 1 Processor -- start 2024-03-06T07:34:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f8654800640 1 -- start start 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f272fc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27280d29f0 0x7f2728153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f272fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e0c0 0x7f27281e69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f272fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f272815e4a0 0x7f272815e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f272fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280c7d00 con 0x7f272815e4a0 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f272fc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280c7a20 con 0x7f27280d29f0 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.175+0000 7f272fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280d4b40 con 0x7f272815e0c0 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.176+0000 7f272dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f272815e4a0 0x7f272815e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.176+0000 7f272dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f272815e4a0 0x7f272815e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36734/0 (socket says 172.21.15.6:36734) 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.176+0000 7f272dc00640 1 -- 172.21.15.6:0/3700779803 learned_addr learned my addr 172.21.15.6:0/3700779803 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.176+0000 7f272d200640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27280d29f0 0x7f2728153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.176+0000 7f2727e00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e0c0 0x7f27281e69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f272d200640 1 -- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e0c0 msgr2=0x7f27281e69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f272d200640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e0c0 0x7f27281e69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f272d200640 1 -- 172.21.15.6:0/3700779803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f272815e4a0 msgr2=0x7f272815e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f272d200640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f272815e4a0 0x7f272815e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f272d200640 1 -- 172.21.15.6:0/3700779803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272815bc20 con 0x7f27280d29f0 2024-03-06T07:34:03.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f272d200640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27280d29f0 0x7f2728153fb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f271c006fd0 tx=0x7f271c00ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.177+0000 7f8654800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 0x7f864c0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8654800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f864c160250 0x7f864c0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8654800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0b76d0 0x7f864c0b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8654800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0cfe20 con 0x7f8634005080 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8654800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0b3c40 con 0x7f864c160250 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8654800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0d0f40 con 0x7f864c0b76d0 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8651e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 0x7f864c0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8651e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 0x7f864c0ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36736/0 (socket says 172.21.15.6:36736) 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8651e00640 1 -- 172.21.15.6:0/3945315075 learned_addr learned my addr 172.21.15.6:0/3945315075 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c00f040 con 0x7f27280d29f0 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c004030 con 0x7f27280d29f0 2024-03-06T07:34:03.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c0043e0 con 0x7f27280d29f0 2024-03-06T07:34:03.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272815bf00 con 0x7f27280d29f0 2024-03-06T07:34:03.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8651400640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f864c160250 0x7f864c0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272815c4c0 con 0x7f27280d29f0 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8652800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0b76d0 0x7f864c0b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8651e00640 1 -- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0b76d0 msgr2=0x7f864c0b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.178+0000 7f8651e00640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0b76d0 0x7f864c0b7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.179+0000 7f8651e00640 1 -- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f864c160250 msgr2=0x7f864c0c7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.179+0000 7f8651e00640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f864c160250 0x7f864c0c7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.179+0000 7f8651e00640 1 -- 172.21.15.6:0/3945315075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c1607b0 con 0x7f8634005080 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.179+0000 7f8652800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0b76d0 0x7f864c0b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.179+0000 7f8651e00640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 0x7f864c0ca2e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f86400072e0 tx=0x7f86400077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.180+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272804f3a0 con 0x7f27280d29f0 2024-03-06T07:34:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.180+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000fa00 con 0x7f8634005080 2024-03-06T07:34:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.180+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8640018070 con 0x7f8634005080 2024-03-06T07:34:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.180+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c0b6bf0 con 0x7f8634005080 2024-03-06T07:34:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.180+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c1539b0 con 0x7f8634005080 2024-03-06T07:34:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.180+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000fe40 con 0x7f8634005080 2024-03-06T07:34:03.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.181+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f127404c020 con 0x7f12881f8710 2024-03-06T07:34:03.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.183+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f271c00c6d0 con 0x7f27280d29f0 2024-03-06T07:34:03.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.182+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8628005290 con 0x7f8634005080 2024-03-06T07:34:03.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.183+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f864001d460 con 0x7f8634005080 2024-03-06T07:34:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.183+0000 7f2724c00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f270806b4d0 0x7f270806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.183+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f271c08a180 con 0x7f27280d29f0 2024-03-06T07:34:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.183+0000 7f8646000640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f863006b5a0 0x7f863006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.184+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f864008a4a0 con 0x7f8634005080 2024-03-06T07:34:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.184+0000 7f2727e00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f270806b4d0 0x7f270806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.184+0000 7f8651400640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f863006b5a0 0x7f863006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.186+0000 7f2727e00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f270806b4d0 0x7f270806d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2714035000 tx=0x7f2714008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.186+0000 7f8651400640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f863006b5a0 0x7f863006da60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f864c06d170 tx=0x7f863c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.192+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8640059300 con 0x7f8634005080 2024-03-06T07:34:03.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.199+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f271c017020 con 0x7f27280d29f0 2024-03-06T07:34:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.213+0000 7f4931600640 1 Processor -- start 2024-03-06T07:34:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.213+0000 7f4931600640 1 -- start start 2024-03-06T07:34:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4931600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f492c070ee0 0x7f492c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4931600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0712c0 0x7f492c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4931600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0ce090 con 0x7f492c072da0 2024-03-06T07:34:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4931600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c0ceca0 con 0x7f492c070ee0 2024-03-06T07:34:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4931600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0ce370 con 0x7f492c0712c0 2024-03-06T07:34:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4923e00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f492c072da0 0x7f492c06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:60962/0 (socket says 172.21.15.6:60962) 2024-03-06T07:34:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f4923e00640 1 -- 172.21.15.6:0/976579024 learned_addr learned my addr 172.21.15.6:0/976579024 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.215+0000 7f492b400640 1 --2- 172.21.15.6:0/976579024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0712c0 0x7f492c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492aa00640 1 --2- 172.21.15.6:0/976579024 >> v2:172.21.15.6:3300/0 conn(0x7f492c070ee0 0x7f492c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492a000640 1 -- 172.21.15.6:0/976579024 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802754413 0 0) 0x7f492c0ce090 con 0x7f492c072da0 2024-03-06T07:34:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492a000640 1 -- 172.21.15.6:0/976579024 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910003520 con 0x7f492c072da0 2024-03-06T07:34:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492b400640 1 -- 172.21.15.6:0/976579024 >> v2:172.21.15.6:3300/0 conn(0x7f492c070ee0 msgr2=0x7f492c162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492b400640 1 --2- 172.21.15.6:0/976579024 >> v2:172.21.15.6:3300/0 conn(0x7f492c070ee0 0x7f492c162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492b400640 1 -- 172.21.15.6:0/976579024 >> v1:172.21.15.178:6789/0 conn(0x7f492c072da0 legacy=0x7f492c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.216+0000 7f492b400640 1 -- 172.21.15.6:0/976579024 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0ce650 con 0x7f492c0712c0 2024-03-06T07:34:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.218+0000 7f492b400640 1 --2- 172.21.15.6:0/976579024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0712c0 0x7f492c06e7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f492c0cee20 tx=0x7f492402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2373e0828b9c07d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.218+0000 7f492a000640 1 -- 172.21.15.6:0/976579024 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924039070 con 0x7f492c0712c0 2024-03-06T07:34:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f492a000640 1 -- 172.21.15.6:0/976579024 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492402cc40 con 0x7f492c0712c0 2024-03-06T07:34:03.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f492a000640 1 -- 172.21.15.6:0/976579024 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924034440 con 0x7f492c0712c0 2024-03-06T07:34:03.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 -- 172.21.15.6:0/976579024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0712c0 msgr2=0x7f492c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 --2- 172.21.15.6:0/976579024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0712c0 0x7f492c06e7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f492c0cee20 tx=0x7f492402be90 comp rx=0 tx=0).stop 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 -- 172.21.15.6:0/976579024 shutdown_connections 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 --2- 172.21.15.6:0/976579024 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0712c0 0x7f492c06e7d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 --2- 172.21.15.6:0/976579024 >> v2:172.21.15.6:3300/0 conn(0x7f492c070ee0 0x7f492c162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 -- 172.21.15.6:0/976579024 >> 172.21.15.6:0/976579024 conn(0x7f492c06afb0 msgr2=0x7f492c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 -- 172.21.15.6:0/976579024 shutdown_connections 2024-03-06T07:34:03.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.219+0000 7f4931600640 1 -- 172.21.15.6:0/976579024 wait complete. 2024-03-06T07:34:03.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.221+0000 7f4931600640 1 Processor -- start 2024-03-06T07:34:03.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.221+0000 7f4931600640 1 -- start start 2024-03-06T07:34:03.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.222+0000 7f4931600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c070ee0 0x7f492c09aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.222+0000 7f4931600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c0712c0 0x7f492c0aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.222+0000 7f4931600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 0x7f492c14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.222+0000 7f4931600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0cdda0 con 0x7f492c072da0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.222+0000 7f4931600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0cdc20 con 0x7f492c0712c0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.222+0000 7f4931600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0cf130 con 0x7f492c070ee0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 0x7f492c14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 0x7f492c14fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36748/0 (socket says 172.21.15.6:36748) 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 -- 172.21.15.6:0/3221132225 learned_addr learned my addr 172.21.15.6:0/3221132225 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492aa00640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c070ee0 0x7f492c09aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f4923e00640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c0712c0 0x7f492c0aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 -- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c070ee0 msgr2=0x7f492c09aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c070ee0 0x7f492c09aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 -- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c0712c0 msgr2=0x7f492c0aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c0712c0 0x7f492c0aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f492b400640 1 -- 172.21.15.6:0/3221132225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0acec0 con 0x7f492c072da0 2024-03-06T07:34:03.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.223+0000 7f4923e00640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c0712c0 0x7f492c0aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.224+0000 7f492b400640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 0x7f492c14fcc0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e80 tx=0x7f4924004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.224+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924042020 con 0x7f492c072da0 2024-03-06T07:34:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.224+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492403a040 con 0x7f492c072da0 2024-03-06T07:34:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.224+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924039070 con 0x7f492c072da0 2024-03-06T07:34:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.224+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c0ad1a0 con 0x7f492c072da0 2024-03-06T07:34:03.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.224+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c0ad760 con 0x7f492c072da0 2024-03-06T07:34:03.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.226+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c069510 con 0x7f492c072da0 2024-03-06T07:34:03.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.227+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4924005d10 con 0x7f492c072da0 2024-03-06T07:34:03.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.228+0000 7f4922a00640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b550 0x7f490806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:03.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.228+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49240ac190 con 0x7f492c072da0 2024-03-06T07:34:03.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.228+0000 7f492aa00640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b550 0x7f490806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:03.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.229+0000 7f492aa00640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b550 0x7f490806da10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f491c009ad0 tx=0x7f491c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:03.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.236+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f492404c070 con 0x7f492c072da0 2024-03-06T07:34:03.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.469+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f12880b4b40 con 0x7f12881f8710 2024-03-06T07:34:03.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.474+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f8628005580 con 0x7f8634005080 2024-03-06T07:34:03.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.476+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f27281610f0 con 0x7f27280d29f0 2024-03-06T07:34:03.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:03.511+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f492c0b52d0 con 0x7f492c072da0 2024-03-06T07:34:04.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.351+0000 7f1285600640 1 -- 172.21.15.6:0/1953524258 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 5795 v78) v1 ==== 95+0+0 (secure 0 0 0) 0x7f1274079990 con 0x7f12881f8710 2024-03-06T07:34:04.353 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 5795 2024-03-06T07:34:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f126806b4d0 msgr2=0x7f126806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f126806b4d0 0x7f126806d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f127c000fb0 tx=0x7f127c007040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 msgr2=0x7f12881f7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 0x7f12881f7590 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f12700046a0 tx=0x7f1274004730 comp rx=0 tx=0).stop 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 shutdown_connections 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f126806b4d0 0x7f126806d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12881f8e30 0x7f12880ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12881f8710 0x7f12881f7590 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 --2- 172.21.15.6:0/1953524258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1270004800 0x7f12880acf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 >> 172.21.15.6:0/1953524258 conn(0x7f128806b3d0 msgr2=0x7f12880aee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 shutdown_connections 2024-03-06T07:34:04.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.355+0000 7f128ec00640 1 -- 172.21.15.6:0/1953524258 wait complete. 2024-03-06T07:34:04.379 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:34:04.377+0000 7f723ec00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:34:04.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.380+0000 7f4922a00640 1 -- 172.21.15.6:0/3221132225 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7656 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7f492407aff0 con 0x7f492c072da0 2024-03-06T07:34:04.382 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 7656 2024-03-06T07:34:04.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.380+0000 7f8646000640 1 -- 172.21.15.6:0/3945315075 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5801 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7f8640058c00 con 0x7f8634005080 2024-03-06T07:34:04.382 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 5801 2024-03-06T07:34:04.384 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:34:04.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b550 msgr2=0x7f490806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b550 0x7f490806da10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f491c009ad0 tx=0x7f491c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 msgr2=0x7f492c14fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 0x7f492c14fcc0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e80 tx=0x7f4924004a40 comp rx=0 tx=0).stop 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 shutdown_connections 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b550 0x7f490806da10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c0712c0 0x7f492c0aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c072da0 0x7f492c14fcc0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 --2- 172.21.15.6:0/3221132225 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c070ee0 0x7f492c09aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 >> 172.21.15.6:0/3221132225 conn(0x7f492c06afb0 msgr2=0x7f492c162c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f863006b5a0 msgr2=0x7f863006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.383+0000 7f8654800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f863006b5a0 0x7f863006da60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f864c06d170 tx=0x7f863c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 msgr2=0x7f864c0ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 0x7f864c0ca2e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f86400072e0 tx=0x7f86400077a0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 shutdown_connections 2024-03-06T07:34:04.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 shutdown_connections 2024-03-06T07:34:04.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f4931600640 1 -- 172.21.15.6:0/3221132225 wait complete. 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f863006b5a0 0x7f863006da60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f864c160250 0x7f864c0c7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f864c0b76d0 0x7f864c0b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 --2- 172.21.15.6:0/3945315075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8634005080 0x7f864c0ca2e0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 >> 172.21.15.6:0/3945315075 conn(0x7f864c0d3f60 msgr2=0x7f864c0b29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 shutdown_connections 2024-03-06T07:34:04.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.384+0000 7f8654800640 1 -- 172.21.15.6:0/3945315075 wait complete. 2024-03-06T07:34:04.390 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:34:04.390 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:04.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.390+0000 7f2724c00640 1 -- 172.21.15.6:0/3700779803 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5798 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7f271c058a90 con 0x7f27280d29f0 2024-03-06T07:34:04.393 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 5798 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.394+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f270806b4d0 msgr2=0x7f270806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.394+0000 7f272fc00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f270806b4d0 0x7f270806d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2714035000 tx=0x7f2714008030 comp rx=0 tx=0).stop 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.394+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27280d29f0 msgr2=0x7f2728153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.394+0000 7f272fc00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27280d29f0 0x7f2728153fb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f271c006fd0 tx=0x7f271c00ec70 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 shutdown_connections 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f270806b4d0 0x7f270806d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f272815e4a0 0x7f272815e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f272815e0c0 0x7f27281e69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 --2- 172.21.15.6:0/3700779803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f27280d29f0 0x7f2728153fb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 >> 172.21.15.6:0/3700779803 conn(0x7f27280d3ed0 msgr2=0x7f272806cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 shutdown_connections 2024-03-06T07:34:04.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.396+0000 7f272fc00640 1 -- 172.21.15.6:0/3700779803 wait complete. 2024-03-06T07:34:04.411 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:34:04.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:34:04.412 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:34:04.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:34:04.419 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:34:04.419 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:04.421 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:34:04.422 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:34:04.428 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:34:04.429 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:04.466 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:34:04.466 DEBUG:teuthology.parallel:result is None 2024-03-06T07:34:04.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:04.541 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:34:04.570 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:34:04.578 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:34:04.614 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:34:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.637+0000 7fd00ae00640 1 Processor -- start 2024-03-06T07:34:04.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.637+0000 7fd00ae00640 1 -- start start 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd00ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd00ae00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0040af580 0x7fd0040b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd00ae00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd0040cdc20 0x7fd0040b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd00ae00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00415cc40 con 0x7fd0040af580 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd00ae00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00415cf20 con 0x7fd0040cdc20 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd00ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00415d200 con 0x7fd0040af1a0 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fcffbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0040af580 0x7fd0040b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fcffbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd0040af580 0x7fd0040b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36790/0 (socket says 172.21.15.6:36790) 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fcffbe00640 1 -- 172.21.15.6:0/3054484836 learned_addr learned my addr 172.21.15.6:0/3054484836 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd003e00640 1 --2- 172.21.15.6:0/3054484836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.638+0000 7fd008e00640 1 --2- 172.21.15.6:0/3054484836 >> v2:172.21.15.178:3300/0 conn(0x7fd0040cdc20 0x7fd0040b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.639+0000 7fcffbe00640 1 -- 172.21.15.6:0/3054484836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 msgr2=0x7fd00415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.639+0000 7fcffbe00640 1 --2- 172.21.15.6:0/3054484836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.639+0000 7fcffbe00640 1 -- 172.21.15.6:0/3054484836 >> v2:172.21.15.178:3300/0 conn(0x7fd0040cdc20 msgr2=0x7fd0040b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.639+0000 7fcffbe00640 1 --2- 172.21.15.6:0/3054484836 >> v2:172.21.15.178:3300/0 conn(0x7fd0040cdc20 0x7fd0040b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.639+0000 7fcffbe00640 1 -- 172.21.15.6:0/3054484836 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00415d4e0 con 0x7fd0040af580 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.640+0000 7fcffbe00640 1 --2- 172.21.15.6:0/3054484836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040b4c20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fd0040cda30 tx=0x7fcfec02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2216a4c3710f596 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.640+0000 7fd003400640 1 -- 172.21.15.6:0/3054484836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0317d0 con 0x7fd0040af580 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.640+0000 7fd003400640 1 -- 172.21.15.6:0/3054484836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec037070 con 0x7fd0040af580 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.640+0000 7fd003400640 1 -- 172.21.15.6:0/3054484836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec031b90 con 0x7fd0040af580 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.640+0000 7fd00ae00640 1 -- 172.21.15.6:0/3054484836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 msgr2=0x7fd0040b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.640+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3054484836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040b4c20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fd0040cda30 tx=0x7fcfec02b870 comp rx=0 tx=0).stop 2024-03-06T07:34:04.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 -- 172.21.15.6:0/3054484836 shutdown_connections 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3054484836 >> v2:172.21.15.178:3300/0 conn(0x7fd0040cdc20 0x7fd0040b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3054484836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040b4c20 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3054484836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 -- 172.21.15.6:0/3054484836 >> 172.21.15.6:0/3054484836 conn(0x7fd00406afb0 msgr2=0x7fd00406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 -- 172.21.15.6:0/3054484836 shutdown_connections 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.641+0000 7fd00ae00640 1 -- 172.21.15.6:0/3054484836 wait complete. 2024-03-06T07:34:04.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.642+0000 7fd00ae00640 1 Processor -- start 2024-03-06T07:34:04.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.642+0000 7fd00ae00640 1 -- start start 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fd00ae00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fd00ae00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fd00ae00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040cdc20 0x7fd004152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fd00ae00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0040b3490 con 0x7fd0040af580 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fd00ae00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0040b3310 con 0x7fd0040cdc20 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fd00ae00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0040b3610 con 0x7fd0040af1a0 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.643+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36804/0 (socket says 172.21.15.6:36804) 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 -- 172.21.15.6:0/3341351568 learned_addr learned my addr 172.21.15.6:0/3341351568 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:04.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fd008e00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040cdc20 0x7fd004152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fd003e00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 -- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 msgr2=0x7fd00415a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 -- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040cdc20 msgr2=0x7fd004152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040cdc20 0x7fd004152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 -- 172.21.15.6:0/3341351568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0041526b0 con 0x7fd0040af580 2024-03-06T07:34:04.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fd003e00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.644+0000 7fcffbe00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040c9150 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000c00 tx=0x7fcfec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.645+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec032870 con 0x7fd0040af580 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.645+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec032a50 con 0x7fd0040af580 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.645+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec032e40 con 0x7fd0040af580 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.645+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd004153e50 con 0x7fd0040af580 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.645+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd004153b00 con 0x7fd0040af580 2024-03-06T07:34:04.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.646+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd004069510 con 0x7fd0040af580 2024-03-06T07:34:04.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.647+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcfec0453f0 con 0x7fd0040af580 2024-03-06T07:34:04.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.647+0000 7fd000c00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcfe006b4d0 0x7fcfe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:04.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.648+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcfec0ac160 con 0x7fd0040af580 2024-03-06T07:34:04.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.648+0000 7fd003e00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcfe006b4d0 0x7fcfe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:04.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.649+0000 7fd003e00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcfe006b4d0 0x7fcfe006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcff400ba50 tx=0x7fcff4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:04.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.652+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcfec040070 con 0x7fd0040af580 2024-03-06T07:34:04.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.883+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd00404f3a0 con 0x7fd0040af580 2024-03-06T07:34:04.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.884+0000 7fd000c00640 1 -- 172.21.15.6:0/3341351568 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+1802 (secure 0 0 0) 0x7fcfec031e90 con 0x7fd0040af580 2024-03-06T07:34:04.887 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:04.887 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":79,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:04.377968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:34:04.887 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 79 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcfe006b4d0 msgr2=0x7fcfe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcfe006b4d0 0x7fcfe006d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcff400ba50 tx=0x7fcff4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 msgr2=0x7fd0040c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040c9150 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000c00 tx=0x7fcfec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 shutdown_connections 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcfe006b4d0 0x7fcfe006d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040cdc20 0x7fd004152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd0040af580 0x7fd0040c9150 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 --2- 172.21.15.6:0/3341351568 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd0040af1a0 0x7fd00415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 >> 172.21.15.6:0/3341351568 conn(0x7fd00406afb0 msgr2=0x7fd0040acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 shutdown_connections 2024-03-06T07:34:04.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:04.887+0000 7fd00ae00640 1 -- 172.21.15.6:0/3341351568 wait complete. 2024-03-06T07:34:05.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:48.441811+0000', 'modified': '2024-03-06T07:34:04.377968+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:05.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-03-06T07:34:06.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:06.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.538+0000 7fdcaca00640 1 Processor -- start 2024-03-06T07:34:06.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.539+0000 7fdcaca00640 1 -- start start 2024-03-06T07:34:06.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdcaca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdca80d2bc0 0x7fdca80d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdcaca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80d1160 con 0x7fdca815e4a0 2024-03-06T07:34:06.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdcaca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80d1440 con 0x7fdca815e0c0 2024-03-06T07:34:06.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdcaca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca80d0f10 con 0x7fdca80d2bc0 2024-03-06T07:34:06.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdca5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdca80d2bc0 0x7fdca80d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdca6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdca815e4a0 0x7fdca80c8220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43360/0 (socket says 172.21.15.6:43360) 2024-03-06T07:34:06.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.540+0000 7fdca6800640 1 -- 172.21.15.6:0/2110507335 learned_addr learned my addr 172.21.15.6:0/2110507335 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:06.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.541+0000 7fdca5400640 1 -- 172.21.15.6:0/2110507335 >> v1:172.21.15.178:6789/0 conn(0x7fdca815e0c0 legacy=0x7fdca80d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:06.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.541+0000 7fdca5400640 1 -- 172.21.15.6:0/2110507335 >> v1:172.21.15.6:6789/0 conn(0x7fdca815e4a0 legacy=0x7fdca80c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.541+0000 7fdca5400640 1 -- 172.21.15.6:0/2110507335 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80d1720 con 0x7fdca80d2bc0 2024-03-06T07:34:06.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.541+0000 7fdca4a00640 1 -- 172.21.15.6:0/2110507335 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288540746 0 0) 0x7fdca80d1160 con 0x7fdca815e4a0 2024-03-06T07:34:06.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.542+0000 7fdca5400640 1 --2- 172.21.15.6:0/2110507335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca80d2bc0 0x7fdca80d1cb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdc940097a0 tx=0x7fdc9402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b9255493a26e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdca4a00640 1 -- 172.21.15.6:0/2110507335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94034070 con 0x7fdca80d2bc0 2024-03-06T07:34:06.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdca4a00640 1 -- 172.21.15.6:0/2110507335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94030610 con 0x7fdca80d2bc0 2024-03-06T07:34:06.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdca4a00640 1 -- 172.21.15.6:0/2110507335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc940309e0 con 0x7fdca80d2bc0 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 -- 172.21.15.6:0/2110507335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca80d2bc0 msgr2=0x7fdca80d1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2110507335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca80d2bc0 0x7fdca80d1cb0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fdc940097a0 tx=0x7fdc9402b960 comp rx=0 tx=0).stop 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 -- 172.21.15.6:0/2110507335 shutdown_connections 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2110507335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca80d2bc0 0x7fdca80d1cb0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 -- 172.21.15.6:0/2110507335 >> 172.21.15.6:0/2110507335 conn(0x7fdca80d3ed0 msgr2=0x7fdca80d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 -- 172.21.15.6:0/2110507335 shutdown_connections 2024-03-06T07:34:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.543+0000 7fdcaca00640 1 -- 172.21.15.6:0/2110507335 wait complete. 2024-03-06T07:34:06.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.545+0000 7fdcaca00640 1 Processor -- start 2024-03-06T07:34:06.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.545+0000 7fdcaca00640 1 -- start start 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdcaca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdca80d2bc0 0x7fdca80cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdcaca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 0x7fdca8159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdcaca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdca815e4a0 0x7fdca815f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdcaca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80c7d00 con 0x7fdca815e0c0 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdcaca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80d21f0 con 0x7fdca80d2bc0 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdcaca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80d4b40 con 0x7fdca815e4a0 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdca5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 0x7fdca8159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdca5400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 0x7fdca8159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36830/0 (socket says 172.21.15.6:36830) 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdca5400640 1 -- 172.21.15.6:0/2747021187 learned_addr learned my addr 172.21.15.6:0/2747021187 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdca5e00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdca80d2bc0 0x7fdca80cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdca6800640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdca815e4a0 0x7fdca815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.547+0000 7fdca5400640 1 -- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdca815e4a0 msgr2=0x7fdca815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca5400640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdca815e4a0 0x7fdca815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca5400640 1 -- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdca80d2bc0 msgr2=0x7fdca80cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca5400640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdca80d2bc0 0x7fdca80cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca5400640 1 -- 172.21.15.6:0/2747021187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80b7700 con 0x7fdca815e0c0 2024-03-06T07:34:06.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca6800640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdca815e4a0 0x7fdca815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:06.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca5e00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdca80d2bc0 0x7fdca80cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:06.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdca5400640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 0x7fdca8159f80 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdc940098c0 tx=0x7fdc940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94049070 con 0x7fdca815e0c0 2024-03-06T07:34:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc94039800 con 0x7fdca815e0c0 2024-03-06T07:34:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94034040 con 0x7fdca815e0c0 2024-03-06T07:34:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.548+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca80b79e0 con 0x7fdca815e0c0 2024-03-06T07:34:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.549+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80b6a90 con 0x7fdca815e0c0 2024-03-06T07:34:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.550+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca80b7ff0 con 0x7fdca815e0c0 2024-03-06T07:34:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.551+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc94037070 con 0x7fdca815e0c0 2024-03-06T07:34:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.552+0000 7fdc9e000640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.552+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdc940abc10 con 0x7fdca815e0c0 2024-03-06T07:34:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.552+0000 7fdca5e00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:06.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.553+0000 7fdca5e00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc8806b4d0 0x7fdc8806d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdc98003eb0 tx=0x7fdc98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:06.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.556+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc9407aac0 con 0x7fdca815e0c0 2024-03-06T07:34:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.797+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdca80b81e0 con 0x7fdca815e0c0 2024-03-06T07:34:06.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.798+0000 7fdc9e000640 1 -- 172.21.15.6:0/2747021187 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fdc9407a3c0 con 0x7fdca815e0c0 2024-03-06T07:34:06.800 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:06.800 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:05.410656+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:34:06.800 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 82 2024-03-06T07:34:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.800+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc8806b4d0 msgr2=0x7fdc8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.800+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc8806b4d0 0x7fdc8806d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fdc98003eb0 tx=0x7fdc98009040 comp rx=0 tx=0).stop 2024-03-06T07:34:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.800+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 msgr2=0x7fdca8159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:06.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.800+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 0x7fdca8159f80 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fdc940098c0 tx=0x7fdc940046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 shutdown_connections 2024-03-06T07:34:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdca815e4a0 0x7fdca815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdca815e0c0 0x7fdca8159f80 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 --2- 172.21.15.6:0/2747021187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdca80d2bc0 0x7fdca80cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:06.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 >> 172.21.15.6:0/2747021187 conn(0x7fdca80d3ed0 msgr2=0x7fdca80c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:06.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 shutdown_connections 2024-03-06T07:34:06.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:06.801+0000 7fdcaca00640 1 -- 172.21.15.6:0/2747021187 wait complete. 2024-03-06T07:34:07.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:48.441811+0000', 'modified': '2024-03-06T07:34:05.410656+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7932': {'gid': 7932, 'name': 'c', 'rank': 0, 'incarnation': 81, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.6:6835/3719236593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 3719236593}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 3719236593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:34:07.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7932:up:reconnect 2024-03-06T07:34:08.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:08.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.461+0000 7f4931c00640 1 Processor -- start 2024-03-06T07:34:08.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.461+0000 7f4931c00640 1 -- start start 2024-03-06T07:34:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.462+0000 7f4931c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 0x7f492c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.462+0000 7f4931c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f492c0cf460 0x7f492c0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.462+0000 7f4931c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0d1120 con 0x7f492c15dec0 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.462+0000 7f4931c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c0b0ea0 con 0x7f492c0cf460 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.462+0000 7f4931c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0d1400 con 0x7f492c15e2a0 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.463+0000 7f4923e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 0x7f492c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.463+0000 7f4931200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f492c0cf460 0x7f492c0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.463+0000 7f4923e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 0x7f492c0cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:50768/0 (socket says 172.21.15.6:50768) 2024-03-06T07:34:08.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.463+0000 7f4923e00640 1 -- 172.21.15.6:0/3426909697 learned_addr learned my addr 172.21.15.6:0/3426909697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.464+0000 7f4923e00640 1 -- 172.21.15.6:0/3426909697 >> v2:172.21.15.178:3300/0 conn(0x7f492c0cf460 msgr2=0x7f492c0b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.464+0000 7f4923e00640 1 --2- 172.21.15.6:0/3426909697 >> v2:172.21.15.178:3300/0 conn(0x7f492c0cf460 0x7f492c0b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.464+0000 7f4923e00640 1 -- 172.21.15.6:0/3426909697 >> v1:172.21.15.178:6789/0 conn(0x7f492c15dec0 legacy=0x7f492c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.464+0000 7f4923e00640 1 -- 172.21.15.6:0/3426909697 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0b0b30 con 0x7f492c15e2a0 2024-03-06T07:34:08.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.465+0000 7f4923e00640 1 --2- 172.21.15.6:0/3426909697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 0x7f492c0cef20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4914009870 tx=0x7f491402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6ce17f7cddaa4fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f492b400640 1 -- 172.21.15.6:0/3426909697 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914034070 con 0x7f492c15e2a0 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f492b400640 1 -- 172.21.15.6:0/3426909697 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4914030840 con 0x7f492c15e2a0 2024-03-06T07:34:08.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f492b400640 1 -- 172.21.15.6:0/3426909697 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914030c10 con 0x7f492c15e2a0 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 -- 172.21.15.6:0/3426909697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 msgr2=0x7f492c0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 --2- 172.21.15.6:0/3426909697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 0x7f492c0cef20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4914009870 tx=0x7f491402b980 comp rx=0 tx=0).stop 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 -- 172.21.15.6:0/3426909697 shutdown_connections 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 --2- 172.21.15.6:0/3426909697 >> v2:172.21.15.178:3300/0 conn(0x7f492c0cf460 0x7f492c0b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 --2- 172.21.15.6:0/3426909697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c15e2a0 0x7f492c0cef20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 -- 172.21.15.6:0/3426909697 >> 172.21.15.6:0/3426909697 conn(0x7f492c0d3d40 msgr2=0x7f492c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.466+0000 7f4931c00640 1 -- 172.21.15.6:0/3426909697 shutdown_connections 2024-03-06T07:34:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.467+0000 7f4931c00640 1 -- 172.21.15.6:0/3426909697 wait complete. 2024-03-06T07:34:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.468+0000 7f4931c00640 1 Processor -- start 2024-03-06T07:34:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.468+0000 7f4931c00640 1 -- start start 2024-03-06T07:34:08.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.469+0000 7f4931c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0cf460 0x7f492c14f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.469+0000 7f4931c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 0x7f492c0b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.469+0000 7f4931c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c15e2a0 0x7f492c0c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.469+0000 7f4931c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0d0ec0 con 0x7f492c15dec0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.469+0000 7f4931c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0d0d40 con 0x7f492c15e2a0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.469+0000 7f4931c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c0b1870 con 0x7f492c0cf460 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 0x7f492c0b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 0x7f492c0b91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36834/0 (socket says 172.21.15.6:36834) 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 -- 172.21.15.6:0/957200981 learned_addr learned my addr 172.21.15.6:0/957200981 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f492be00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0cf460 0x7f492c14f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4931200640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c15e2a0 0x7f492c0c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 -- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0cf460 msgr2=0x7f492c14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0cf460 0x7f492c14f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 -- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c15e2a0 msgr2=0x7f492c0c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c15e2a0 0x7f492c0c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.470+0000 7f4923e00640 1 -- 172.21.15.6:0/957200981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0b6ed0 con 0x7f492c15dec0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.471+0000 7f4923e00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 0x7f492c0b91e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f4914000c00 tx=0x7f49140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.471+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914034040 con 0x7f492c15dec0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.471+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4914039d90 con 0x7f492c15dec0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.471+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c14fd10 con 0x7f492c15dec0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.471+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914043630 con 0x7f492c15dec0 2024-03-06T07:34:08.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.471+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c1502d0 con 0x7f492c15dec0 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.472+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c04f330 con 0x7f492c15dec0 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.473+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f491404a060 con 0x7f492c15dec0 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.474+0000 7f4928c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b520 0x7f490806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:08.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.474+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f49140ac2a0 con 0x7f492c15dec0 2024-03-06T07:34:08.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.475+0000 7f492be00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b520 0x7f490806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:08.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.476+0000 7f492be00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b520 0x7f490806d9e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f491c001640 tx=0x7f491c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:08.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.478+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f491403e070 con 0x7f492c15dec0 2024-03-06T07:34:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.726+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f492c0748b0 con 0x7f492c15dec0 2024-03-06T07:34:08.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.727+0000 7f4928c00640 1 -- 172.21.15.6:0/957200981 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f491402d330 con 0x7f492c15dec0 2024-03-06T07:34:08.729 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:08.729 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:08.730 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:08.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b520 msgr2=0x7f490806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b520 0x7f490806d9e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f491c001640 tx=0x7f491c009040 comp rx=0 tx=0).stop 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 msgr2=0x7f492c0b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 0x7f492c0b91e0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f4914000c00 tx=0x7f49140046c0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 shutdown_connections 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f490806b520 0x7f490806d9e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f492c15e2a0 0x7f492c0c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f492c15dec0 0x7f492c0b91e0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 --2- 172.21.15.6:0/957200981 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f492c0cf460 0x7f492c14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 >> 172.21.15.6:0/957200981 conn(0x7f492c0d3d40 msgr2=0x7f492c06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 shutdown_connections 2024-03-06T07:34:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:08.730+0000 7f4931c00640 1 -- 172.21.15.6:0/957200981 wait complete. 2024-03-06T07:34:09.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:33:48.441811+0000', 'modified': '2024-03-06T07:34:08.409742+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 112, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7932': {'gid': 7932, 'name': 'c', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.6:6835/3719236593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6834', 'nonce': 3719236593}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 3719236593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7932, 'qdb_cluster': [7932]} 2024-03-06T07:34:09.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:34:09.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:34:09.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.377+0000 7f02dc800640 1 Processor -- start 2024-03-06T07:34:09.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.377+0000 7f02dc800640 1 -- start start 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.379+0000 7f02dc800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02d40b34e0 0x7f02d415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.379+0000 7f02dc800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d415cc40 con 0x7f02d40af580 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.379+0000 7f02dc800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d415cf20 con 0x7f02d40af1a0 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.379+0000 7f02dc800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d415d200 con 0x7f02d40b34e0 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02da800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02d40b34e0 0x7f02d415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02da800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f02d40b34e0 0x7f02d415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36848/0 (socket says 172.21.15.6:36848) 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02da800640 1 -- 172.21.15.6:0/379850042 learned_addr learned my addr 172.21.15.6:0/379850042 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02d9400640 1 --1- 172.21.15.6:0/379850042 >> v1:172.21.15.6:6789/0 conn(0x7f02d40af580 0x7f02d40b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43372/0 (socket says 172.21.15.6:43372) 2024-03-06T07:34:09.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02da800640 1 -- 172.21.15.6:0/379850042 >> v1:172.21.15.178:6789/0 conn(0x7f02d40af1a0 legacy=0x7f02d40b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02da800640 1 -- 172.21.15.6:0/379850042 >> v1:172.21.15.6:6789/0 conn(0x7f02d40af580 legacy=0x7f02d40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02da800640 1 -- 172.21.15.6:0/379850042 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d415d4e0 con 0x7f02d40b34e0 2024-03-06T07:34:09.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.380+0000 7f02d8a00640 1 -- 172.21.15.6:0/379850042 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627759091 0 0) 0x7f02d415cc40 con 0x7f02d40af580 2024-03-06T07:34:09.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.381+0000 7f02da800640 1 --2- 172.21.15.6:0/379850042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40b34e0 0x7f02d415c600 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009e50 tx=0x7f02d002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a452fefff9dd42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.382+0000 7f02d8a00640 1 -- 172.21.15.6:0/379850042 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d002ca80 con 0x7f02d40b34e0 2024-03-06T07:34:09.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.382+0000 7f02d8a00640 1 -- 172.21.15.6:0/379850042 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d002cc20 con 0x7f02d40b34e0 2024-03-06T07:34:09.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.382+0000 7f02d8a00640 1 -- 172.21.15.6:0/379850042 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d00343f0 con 0x7f02d40b34e0 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 -- 172.21.15.6:0/379850042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40b34e0 msgr2=0x7f02d415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 --2- 172.21.15.6:0/379850042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40b34e0 0x7f02d415c600 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009e50 tx=0x7f02d002bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 -- 172.21.15.6:0/379850042 shutdown_connections 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 --2- 172.21.15.6:0/379850042 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40b34e0 0x7f02d415c600 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 -- 172.21.15.6:0/379850042 >> 172.21.15.6:0/379850042 conn(0x7f02d406afb0 msgr2=0x7f02d406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 -- 172.21.15.6:0/379850042 shutdown_connections 2024-03-06T07:34:09.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.383+0000 7f02dc800640 1 -- 172.21.15.6:0/379850042 wait complete. 2024-03-06T07:34:09.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.386+0000 7f02dc800640 1 Processor -- start 2024-03-06T07:34:09.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.386+0000 7f02dc800640 1 -- start start 2024-03-06T07:34:09.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.387+0000 7f02dc800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 0x7f02d41f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.387+0000 7f02dc800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02d40af580 0x7f02d41f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.387+0000 7f02dc800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02d40b34e0 0x7f02d409b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.387+0000 7f02dc800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4155e70 con 0x7f02d40af1a0 2024-03-06T07:34:09.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.387+0000 7f02dc800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40cde30 con 0x7f02d40b34e0 2024-03-06T07:34:09.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.387+0000 7f02dc800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4155ff0 con 0x7f02d40af580 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 0x7f02d41f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 0x7f02d41f2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36850/0 (socket says 172.21.15.6:36850) 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 -- 172.21.15.6:0/2251626140 learned_addr learned my addr 172.21.15.6:0/2251626140 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02da800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02d40b34e0 0x7f02d409b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 -- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02d40af580 msgr2=0x7f02d41f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9400640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02d40af580 0x7f02d41f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02d40af580 0x7f02d41f49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 -- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02d40b34e0 msgr2=0x7f02d409b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02d40b34e0 0x7f02d409b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02d9e00640 1 -- 172.21.15.6:0/2251626140 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d409c080 con 0x7f02d40af1a0 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.388+0000 7f02da800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02d40b34e0 0x7f02d409b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.389+0000 7f02d9e00640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 0x7f02d41f2cb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f02c800c460 tx=0x7f02c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.389+0000 7f02ce000640 1 -- 172.21.15.6:0/2251626140 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8016020 con 0x7f02d40af1a0 2024-03-06T07:34:09.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.389+0000 7f02ce000640 1 -- 172.21.15.6:0/2251626140 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c8012870 con 0x7f02d40af1a0 2024-03-06T07:34:09.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.389+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d4152400 con 0x7f02d40af1a0 2024-03-06T07:34:09.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.389+0000 7f02ce000640 1 -- 172.21.15.6:0/2251626140 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8012c60 con 0x7f02d40af1a0 2024-03-06T07:34:09.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.389+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d409bc00 con 0x7f02d40af1a0 2024-03-06T07:34:09.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.392+0000 7f02ce000640 1 -- 172.21.15.6:0/2251626140 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02c8019020 con 0x7f02d40af1a0 2024-03-06T07:34:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.393+0000 7f02ce000640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02b806b4d0 0x7f02b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.393+0000 7f02ce000640 1 -- 172.21.15.6:0/2251626140 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f02c8089f50 con 0x7f02d40af1a0 2024-03-06T07:34:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.394+0000 7f02df56a740 1 Processor -- start 2024-03-06T07:34:09.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.394+0000 7f02df56a740 1 -- start start 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.394+0000 7f02d9400640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02b806b4d0 0x7f02b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02df56a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x56236262eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bfa0 0x56236265e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x56236265ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02df56a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x562362594600 con 0x56236265bbc0 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236253d930 con 0x56236262f0a0 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5623625906a0 con 0x56236265bfa0 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x56236262eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x56236262eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36856/0 (socket says 172.21.15.6:36856) 2024-03-06T07:34:09.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 -- 172.21.15.6:0/2172285954 learned_addr learned my addr 172.21.15.6:0/2172285954 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9400640 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bfa0 0x56236265e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02da800640 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x56236265ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 -- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bfa0 msgr2=0x56236265e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bfa0 0x56236265e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 -- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 msgr2=0x56236265ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x56236265ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9e00640 1 -- 172.21.15.6:0/2172285954 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623622a92d0 con 0x56236265bbc0 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02da800640 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x56236265ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.395+0000 7f02d9400640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02b806b4d0 0x7f02b806d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f02c4007fd0 tx=0x7f02c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.396+0000 7f02d9e00640 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x56236262eb60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f02c8088350 tx=0x7f02c805c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419ceea4c57c72f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.396+0000 7f02c2600640 1 -- 172.21.15.6:0/2172285954 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8086070 con 0x56236265bbc0 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.396+0000 7f02c2600640 1 -- 172.21.15.6:0/2172285954 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c80701d0 con 0x56236265bbc0 2024-03-06T07:34:09.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.396+0000 7f02c2600640 1 -- 172.21.15.6:0/2172285954 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8070030 con 0x56236265bbc0 2024-03-06T07:34:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 -- 172.21.15.6:0/2172285954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 msgr2=0x56236262eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x56236262eb60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f02c8088350 tx=0x7f02c805c930 comp rx=0 tx=0).stop 2024-03-06T07:34:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 -- 172.21.15.6:0/2172285954 shutdown_connections 2024-03-06T07:34:09.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x56236265ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bfa0 0x56236265e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 --2- 172.21.15.6:0/2172285954 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x56236262eb60 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 -- 172.21.15.6:0/2172285954 >> 172.21.15.6:0/2172285954 conn(0x562362630dd0 msgr2=0x562362679950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 -- 172.21.15.6:0/2172285954 shutdown_connections 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.397+0000 7f02df56a740 1 -- 172.21.15.6:0/2172285954 wait complete. 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.398+0000 7f02df56a740 1 -- start start 2024-03-06T07:34:09.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.399+0000 7f02df56a740 1 Processor -- start 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.400+0000 7f02df56a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x562362653d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.400+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362654830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.400+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236267a370 0x562362660410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.400+0000 7f02df56a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x562362633370 con 0x56236265bbc0 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562362550d90 con 0x56236265bfa0 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x562362632760 con 0x56236267a370 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x562362653d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02df56a740 10 client.0 _ll_register_callbacks cb 0x56236258c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x562362653d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36860/0 (socket says 172.21.15.6:36860) 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 -- 172.21.15.6:0/4091169310 learned_addr learned my addr 172.21.15.6:0/4091169310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9400640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362654830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02da800640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236267a370 0x562362660410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 -- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236267a370 msgr2=0x562362660410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236267a370 0x562362660410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 -- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 msgr2=0x562362654830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362654830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9e00640 1 -- 172.21.15.6:0/4091169310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623623736b0 con 0x56236265bbc0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02da800640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236267a370 0x562362660410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.401+0000 7f02d9400640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362654830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02d9e00640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x562362653d30 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f02c8060f30 tx=0x7f02c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21912c96fdc8ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02b6a00640 1 -- 172.21.15.6:0/4091169310 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8086070 con 0x56236265bbc0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02b6a00640 1 -- 172.21.15.6:0/4091169310 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f02c8054cd0 con 0x56236265bbc0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5623625a1b40 con 0x56236265bbc0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02b6a00640 1 -- 172.21.15.6:0/4091169310 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02c807c050 con 0x56236265bbc0 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02df56a740 10 client.7965 fetch_fsmap learned FSMap version 84 2024-03-06T07:34:09.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x562362655550 con 0x56236265bbc0 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02b6a00640 1 -- 172.21.15.6:0/4091169310 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5623625a1b40 con 0x56236265bbc0 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02b6a00640 1 -- 172.21.15.6:0/4091169310 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 84) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f02c8057ab0 con 0x56236265bbc0 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.402+0000 7f02df56a740 10 client.7965 fetch_fsmap finished waiting for FSMap version 84 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.403+0000 7f02df56a740 10 client.7965 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.403+0000 7f02df56a740 20 client.7965 populate_metadata read hostname 'smithi006' 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.403+0000 7f02df56a740 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236268d360 0x56236268d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.403+0000 7f02df56a740 4 client.7965 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.403+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 --> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56236259be90 con 0x56236268d360 2024-03-06T07:34:09.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.403+0000 7f02d9400640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236268d360 0x56236268d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.404+0000 7f02d9400640 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236268d360 0x56236268d740 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8973c8817b149278 server_cookie=cd7413c649af6c77 in_seq=0 out_seq=0 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.404+0000 7f02b6a00640 10 client.7965 ms_handle_connect on v2:172.21.15.6:6834/3719236593 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.404+0000 7f02b6a00640 1 -- 172.21.15.6:0/4091169310 <== mds.0 v2:172.21.15.6:6834/3719236593 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56236259be90 con 0x56236268d360 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.404+0000 7f02b6a00640 10 client.7965 handle_command_reply: tid=0 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 client.7965 shutdown 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 msgr2=0x562362653d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x562362653d30 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f02c8060f30 tx=0x7f02c8000bc0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 shutdown_connections 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236268d360 0x56236268d740 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362654830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236267a370 0x562362660410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 --2- 172.21.15.6:0/4091169310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x562362653d30 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02b6a00640 0 client.7965 ms_handle_reset on v2:172.21.15.6:6834/3719236593 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 >> 172.21.15.6:0/4091169310 conn(0x56236262f0a0 msgr2=0x56236267ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 shutdown_connections 2024-03-06T07:34:09.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 1 -- 172.21.15.6:0/4091169310 wait complete. 2024-03-06T07:34:09.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.405+0000 7f02df56a740 20 client.7965 trim_cache size 0 max 16384 2024-03-06T07:34:09.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.424+0000 7f02df56a740 1 Processor -- start 2024-03-06T07:34:09.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 -- start start 2024-03-06T07:34:09.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236262f0a0 0x562362634e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x5623626327c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362632d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d406dd20 con 0x56236265bbc0 2024-03-06T07:34:09.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d415f340 con 0x56236265bfa0 2024-03-06T07:34:09.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.425+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41537a0 con 0x56236262f0a0 2024-03-06T07:34:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x5623626327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x5623626327c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36872/0 (socket says 172.21.15.6:36872) 2024-03-06T07:34:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 -- 172.21.15.6:0/177472291 learned_addr learned my addr 172.21.15.6:0/177472291 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9e00640 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236262f0a0 0x562362634e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02da800640 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362632d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 -- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236262f0a0 msgr2=0x562362634e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236262f0a0 0x562362634e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 -- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 msgr2=0x562362632d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362632d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.426+0000 7f02d9400640 1 -- 172.21.15.6:0/177472291 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56236265c6f0 con 0x56236265bbc0 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.427+0000 7f02d9400640 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x5623626327c0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f02d40afa10 tx=0x7f02c4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586c9a1fa2444e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.427+0000 7f02c1c00640 1 -- 172.21.15.6:0/177472291 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c402a070 con 0x56236265bbc0 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.427+0000 7f02c1c00640 1 -- 172.21.15.6:0/177472291 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c401a040 con 0x56236265bbc0 2024-03-06T07:34:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.427+0000 7f02c1c00640 1 -- 172.21.15.6:0/177472291 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4019030 con 0x56236265bbc0 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 -- 172.21.15.6:0/177472291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 msgr2=0x5623626327c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x5623626327c0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f02d40afa10 tx=0x7f02c4013000 comp rx=0 tx=0).stop 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 -- 172.21.15.6:0/177472291 shutdown_connections 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236265bfa0 0x562362632d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bbc0 0x5623626327c0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 --2- 172.21.15.6:0/177472291 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236262f0a0 0x562362634e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 -- 172.21.15.6:0/177472291 >> 172.21.15.6:0/177472291 conn(0x56236265cca0 msgr2=0x562362656000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 -- 172.21.15.6:0/177472291 shutdown_connections 2024-03-06T07:34:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.428+0000 7f02df56a740 1 -- 172.21.15.6:0/177472291 wait complete. 2024-03-06T07:34:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.429+0000 7f02df56a740 1 -- start start 2024-03-06T07:34:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.429+0000 7f02df56a740 1 Processor -- start 2024-03-06T07:34:09.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x562362678950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bbc0 0x562362678e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 0x56236268be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x562362633370 con 0x56236265bfa0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562362550d90 con 0x56236262f0a0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x562362660c40 con 0x56236265bbc0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02df56a740 10 client.0 _ll_register_callbacks cb 0x7f02c8084070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02da800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 0x56236268be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02da800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 0x56236268be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36884/0 (socket says 172.21.15.6:36884) 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02da800640 1 -- 172.21.15.6:0/237372229 learned_addr learned my addr 172.21.15.6:0/237372229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02d9e00640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x562362678950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.430+0000 7f02d9400640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bbc0 0x562362678e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02da800640 1 -- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bbc0 msgr2=0x562362678e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02da800640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bbc0 0x562362678e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02da800640 1 -- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 msgr2=0x562362678950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02da800640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x562362678950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02da800640 1 -- 172.21.15.6:0/237372229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56236269d7e0 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02d9e00640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x562362678950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02da800640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 0x56236268be70 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f02d0036040 tx=0x7f02d0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b671d9810096f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.431+0000 7f02b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02b6a00640 1 -- 172.21.15.6:0/237372229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0044070 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02b6a00640 1 -- 172.21.15.6:0/237372229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f02d0049b20 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02c401a040 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02b6a00640 1 -- 172.21.15.6:0/237372229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02d0049d20 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 10 client.7971 fetch_fsmap learned FSMap version 84 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=2+}) v3 -- 0x56236269c7b0 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02b6a00640 1 -- 172.21.15.6:0/237372229 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c401a040 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02b6a00640 1 -- 172.21.15.6:0/237372229 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 84) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f02d0008040 con 0x56236265bfa0 2024-03-06T07:34:09.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 10 client.7971 fetch_fsmap finished waiting for FSMap version 84 2024-03-06T07:34:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 10 client.7971 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:34:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 20 client.7971 populate_metadata read hostname 'smithi006' 2024-03-06T07:34:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236267a370 0x5623626e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 4 client.7971 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:34:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.432+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 --> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56236251a860 con 0x56236267a370 2024-03-06T07:34:09.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.433+0000 7f02d9e00640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236267a370 0x5623626e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.433+0000 7f02b6a00640 10 client.7971 ms_handle_connect on v2:172.21.15.6:6834/3719236593 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.433+0000 7f02d9e00640 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236267a370 0x5623626e8d30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acfa3b410caa2269 server_cookie=460c0b4e6945bea4 in_seq=0 out_seq=0 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.433+0000 7f02b6a00640 1 -- 172.21.15.6:0/237372229 <== mds.0 v2:172.21.15.6:6834/3719236593 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x56236251a860 con 0x56236267a370 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02b6a00640 10 client.7971 handle_command_reply: tid=0 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 client.7971 shutdown 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 msgr2=0x56236268be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 0x56236268be70 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f02d0036040 tx=0x7f02d0004ab0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 shutdown_connections 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:6834/3719236593,v1:172.21.15.6:6835/3719236593] conn(0x56236267a370 0x5623626e8d30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x56236265bfa0 0x56236268be70 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x56236265bbc0 0x562362678e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 --2- 172.21.15.6:0/237372229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x56236262f0a0 0x562362678950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02b6a00640 0 client.7971 ms_handle_reset on v2:172.21.15.6:6834/3719236593 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 >> 172.21.15.6:0/237372229 conn(0x56236265cca0 msgr2=0x562362652e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 shutdown_connections 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 1 -- 172.21.15.6:0/237372229 wait complete. 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.434+0000 7f02df56a740 20 client.7971 trim_cache size 0 max 16384 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:34:09.439 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 7932, 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 4.0387425879999999, 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 84, 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 112, 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 112, 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 4.8180853629999998 2024-03-06T07:34:09.440 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02b806b4d0 msgr2=0x7f02b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02b806b4d0 0x7f02b806d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f02c4007fd0 tx=0x7f02c4009040 comp rx=0 tx=0).stop 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 msgr2=0x7f02d41f2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 0x7f02d41f2cb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f02c800c460 tx=0x7f02c800c920 comp rx=0 tx=0).stop 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 shutdown_connections 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f02b806b4d0 0x7f02b806d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f02d40b34e0 0x7f02d409b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f02d40af580 0x7f02d41f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 --2- 172.21.15.6:0/2251626140 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f02d40af1a0 0x7f02d41f2cb0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 >> 172.21.15.6:0/2251626140 conn(0x7f02d406afb0 msgr2=0x7f02d414a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 shutdown_connections 2024-03-06T07:34:09.442 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.435+0000 7f02dc800640 1 -- 172.21.15.6:0/2251626140 wait complete. 2024-03-06T07:34:09.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:34:09.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.666+0000 7fa5f8e00640 1 Processor -- start 2024-03-06T07:34:09.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.666+0000 7fa5f8e00640 1 -- start start 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.667+0000 7fa5f8e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa5f415e070 0x7fa5f40d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f8e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5f40d0cd0 0x7fa5f40cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f8e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f40d18c0 con 0x7fa5f40cf410 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f8e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40d1ba0 con 0x7fa5f40d0cd0 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f8e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5f40d1e80 con 0x7fa5f415e070 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa5f40cf410 0x7fa5f40d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43380/0 (socket says 172.21.15.6:43380) 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f2a00640 1 -- 172.21.15.6:0/1211824569 learned_addr learned my addr 172.21.15.6:0/1211824569 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f3e00640 1 --2- 172.21.15.6:0/1211824569 >> v2:172.21.15.6:3300/0 conn(0x7fa5f40d0cd0 0x7fa5f40cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.668+0000 7fa5f3400640 1 --2- 172.21.15.6:0/1211824569 >> v2:172.21.15.178:3300/0 conn(0x7fa5f415e070 0x7fa5f40d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.669+0000 7fa5f2000640 1 -- 172.21.15.6:0/1211824569 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564049790 0 0) 0x7fa5f40d18c0 con 0x7fa5f40cf410 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.669+0000 7fa5f2000640 1 -- 172.21.15.6:0/1211824569 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d8003520 con 0x7fa5f40cf410 2024-03-06T07:34:09.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.669+0000 7fa5f3e00640 1 -- 172.21.15.6:0/1211824569 >> v2:172.21.15.178:3300/0 conn(0x7fa5f415e070 msgr2=0x7fa5f40d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.669+0000 7fa5f3e00640 1 --2- 172.21.15.6:0/1211824569 >> v2:172.21.15.178:3300/0 conn(0x7fa5f415e070 0x7fa5f40d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.669+0000 7fa5f3e00640 1 -- 172.21.15.6:0/1211824569 >> v1:172.21.15.6:6789/0 conn(0x7fa5f40cf410 legacy=0x7fa5f40d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.669+0000 7fa5f3e00640 1 -- 172.21.15.6:0/1211824569 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f40d2160 con 0x7fa5f40d0cd0 2024-03-06T07:34:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.670+0000 7fa5f3e00640 1 --2- 172.21.15.6:0/1211824569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f40cf870 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fa5f408d270 tx=0x7fa5ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b84b5b6bd80f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.670+0000 7fa5f2000640 1 -- 172.21.15.6:0/1211824569 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec035020 con 0x7fa5f40d0cd0 2024-03-06T07:34:09.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.670+0000 7fa5f2000640 1 -- 172.21.15.6:0/1211824569 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec031830 con 0x7fa5f40d0cd0 2024-03-06T07:34:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.670+0000 7fa5f2000640 1 -- 172.21.15.6:0/1211824569 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec031be0 con 0x7fa5f40d0cd0 2024-03-06T07:34:09.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.671+0000 7fa5f8e00640 1 -- 172.21.15.6:0/1211824569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40d0cd0 msgr2=0x7fa5f40cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.671+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/1211824569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f40cf870 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fa5f408d270 tx=0x7fa5ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.671+0000 7fa5f8e00640 1 -- 172.21.15.6:0/1211824569 shutdown_connections 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.671+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/1211824569 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f40cf870 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.671+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/1211824569 >> v2:172.21.15.178:3300/0 conn(0x7fa5f415e070 0x7fa5f40d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.671+0000 7fa5f8e00640 1 -- 172.21.15.6:0/1211824569 >> 172.21.15.6:0/1211824569 conn(0x7fa5f408d2a0 msgr2=0x7fa5f40d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.672+0000 7fa5f8e00640 1 -- 172.21.15.6:0/1211824569 shutdown_connections 2024-03-06T07:34:09.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.672+0000 7fa5f8e00640 1 -- 172.21.15.6:0/1211824569 wait complete. 2024-03-06T07:34:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.673+0000 7fa5f8e00640 1 Processor -- start 2024-03-06T07:34:09.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.674+0000 7fa5f8e00640 1 -- start start 2024-03-06T07:34:09.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.674+0000 7fa5f8e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 0x7fa5f4154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.674+0000 7fa5f8e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f4160d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.674+0000 7fa5f8e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5f415e070 0x7fa5f40b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.674+0000 7fa5f8e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40d4730 con 0x7fa5f40cf410 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.674+0000 7fa5f8e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40d45b0 con 0x7fa5f40d0cd0 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f8e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f40b3630 con 0x7fa5f415e070 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 0x7fa5f4154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 0x7fa5f4154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36910/0 (socket says 172.21.15.6:36910) 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 -- 172.21.15.6:0/793431519 learned_addr learned my addr 172.21.15.6:0/793431519 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f2a00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f4160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5f415e070 0x7fa5f40b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 -- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5f415e070 msgr2=0x7fa5f40b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5f415e070 0x7fa5f40b0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 -- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5f40d0cd0 msgr2=0x7fa5f4160d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f4160d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f3400640 1 -- 172.21.15.6:0/793431519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f415ec10 con 0x7fa5f40cf410 2024-03-06T07:34:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.675+0000 7fa5f2a00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f4160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.676+0000 7fa5f3400640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 0x7fa5f4154ee0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400efc0 tx=0x7fa5e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.676+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4018020 con 0x7fa5f40cf410 2024-03-06T07:34:09.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.676+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e4014480 con 0x7fa5f40cf410 2024-03-06T07:34:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.676+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4014850 con 0x7fa5f40cf410 2024-03-06T07:34:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.676+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f415e930 con 0x7fa5f40cf410 2024-03-06T07:34:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.676+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f41f5650 con 0x7fa5f40cf410 2024-03-06T07:34:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.677+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5f404f330 con 0x7fa5f40cf410 2024-03-06T07:34:09.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.678+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5e4005d10 con 0x7fa5f40cf410 2024-03-06T07:34:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.680+0000 7fa5eb400640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.680+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa5e408af90 con 0x7fa5f40cf410 2024-03-06T07:34:09.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.681+0000 7fa5f2a00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:09.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.682+0000 7fa5f2a00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa5d006b4d0 0x7fa5d006d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4155420 tx=0x7fa5dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:09.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.684+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5e4022030 con 0x7fa5f40cf410 2024-03-06T07:34:09.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.926+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa5f40d1370 con 0x7fa5f40cf410 2024-03-06T07:34:09.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.927+0000 7fa5eb400640 1 -- 172.21.15.6:0/793431519 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v84) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa5e4010040 con 0x7fa5f40cf410 2024-03-06T07:34:09.929 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:34:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa5d006b4d0 msgr2=0x7fa5d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa5d006b4d0 0x7fa5d006d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4155420 tx=0x7fa5dc009040 comp rx=0 tx=0).stop 2024-03-06T07:34:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 msgr2=0x7fa5f4154ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 0x7fa5f4154ee0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fa5e400efc0 tx=0x7fa5e400c490 comp rx=0 tx=0).stop 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 shutdown_connections 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa5d006b4d0 0x7fa5d006d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5f40d0cd0 0x7fa5f4160d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5f415e070 0x7fa5f40b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 --2- 172.21.15.6:0/793431519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f40cf410 0x7fa5f4154ee0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 >> 172.21.15.6:0/793431519 conn(0x7fa5f408d2a0 msgr2=0x7fa5f40d2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 shutdown_connections 2024-03-06T07:34:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:09.930+0000 7fa5f8e00640 1 -- 172.21.15.6:0/793431519 wait complete. 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:34:09.951 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:09.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:09.952 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:10.000 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42207sec preferred_lft 42207sec 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:34:10.001 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:34:10.002 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:34:10.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.003 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:34:10.092 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:34:10.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.096 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:34:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:34:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:34:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:34:10.147 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:34:10.148 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:34:10.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.148 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:34:10.148 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:34:10.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:34:10.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:34:10.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:34:10.148 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:34:10.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.325 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:34:10.377 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:34:10.377 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:34:10.378 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:10.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:34:10.467 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:10.467 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:10.511 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:34:10.512 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:34:10.512 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:10.608 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:34:10.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:10.744 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:10.780 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:34:10.780 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:34:10.780 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:34:10.780 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:34:10.780 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:34:10.782 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:34:10.782 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:34:10.782 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:34:10.782 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:34:10.782 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:34:10.785 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:34:10.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:34:10.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.874 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:34:10.928 INFO:teuthology.orchestra.run.smithi006.stdout:client7983 2024-03-06T07:34:10.929 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:34:10.929 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:34:10.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:10.976 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:34:11.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.290+0000 7f2f7d600640 1 Processor -- start 2024-03-06T07:34:11.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.290+0000 7f2f7d600640 1 -- start start 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.291+0000 7f2f7d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2f780d4a10 0x7f2f780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.291+0000 7f2f7d600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780b3710 con 0x7f2f780d0c80 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f7d600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780b39f0 con 0x7f2f7815e0b0 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f7d600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f780d10f0 con 0x7f2f780d4a10 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f6f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2f780d4a10 0x7f2f780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f6f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2f780d4a10 0x7f2f780d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36912/0 (socket says 172.21.15.6:36912) 2024-03-06T07:34:11.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f6f400640 1 -- 172.21.15.6:0/2853742510 learned_addr learned my addr 172.21.15.6:0/2853742510 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f6f400640 1 -- 172.21.15.6:0/2853742510 >> v1:172.21.15.178:6789/0 conn(0x7f2f7815e0b0 legacy=0x7f2f780cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f6f400640 1 -- 172.21.15.6:0/2853742510 >> v1:172.21.15.6:6789/0 conn(0x7f2f780d0c80 legacy=0x7f2f780b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.292+0000 7f2f6f400640 1 -- 172.21.15.6:0/2853742510 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780b3cd0 con 0x7f2f780d4a10 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.293+0000 7f2f6f400640 1 --2- 172.21.15.6:0/2853742510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f780d4a10 0x7f2f780d0740 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009870 tx=0x7f2f6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b180cedc3a94ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f77400640 1 -- 172.21.15.6:0/2853742510 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60034070 con 0x7f2f780d4a10 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f77400640 1 -- 172.21.15.6:0/2853742510 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f60030860 con 0x7f2f780d4a10 2024-03-06T07:34:11.296 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f77400640 1 -- 172.21.15.6:0/2853742510 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60030c30 con 0x7f2f780d4a10 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f7d600640 1 -- 172.21.15.6:0/2853742510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f780d4a10 msgr2=0x7f2f780d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f7d600640 1 --2- 172.21.15.6:0/2853742510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f780d4a10 0x7f2f780d0740 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009870 tx=0x7f2f6002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f7d600640 1 -- 172.21.15.6:0/2853742510 shutdown_connections 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f7d600640 1 --2- 172.21.15.6:0/2853742510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f780d4a10 0x7f2f780d0740 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.294+0000 7f2f7d600640 1 -- 172.21.15.6:0/2853742510 >> 172.21.15.6:0/2853742510 conn(0x7f2f780d3d30 msgr2=0x7f2f780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.295+0000 7f2f7d600640 1 -- 172.21.15.6:0/2853742510 shutdown_connections 2024-03-06T07:34:11.297 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.295+0000 7f2f7d600640 1 -- 172.21.15.6:0/2853742510 wait complete. 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.296+0000 7f2f7d600640 1 Processor -- start 2024-03-06T07:34:11.298 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.296+0000 7f2f7d600640 1 -- start start 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.297+0000 7f2f7d600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f780d0c80 0x7f2f780b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.297+0000 7f2f7d600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f780d4a10 0x7f2f7815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.297+0000 7f2f7d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 0x7f2f7815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.297+0000 7f2f7d600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f780cfa20 con 0x7f2f7815e0b0 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7d600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7815e490 con 0x7f2f780d4a10 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7d600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f780cfc60 con 0x7f2f780d0c80 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 0x7f2f7815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 0x7f2f7815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36924/0 (socket says 172.21.15.6:36924) 2024-03-06T07:34:11.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 -- 172.21.15.6:0/898838494 learned_addr learned my addr 172.21.15.6:0/898838494 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:11.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f6f400640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f780d4a10 0x7f2f7815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.301 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f77e00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f780d0c80 0x7f2f780b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 -- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f780d0c80 msgr2=0x7f2f780b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f780d0c80 0x7f2f780b93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 -- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f780d4a10 msgr2=0x7f2f7815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f780d4a10 0x7f2f7815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.298+0000 7f2f7cc00640 1 -- 172.21.15.6:0/898838494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780cb8b0 con 0x7f2f7815e0b0 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.299+0000 7f2f7cc00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 0x7f2f7815c020 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c3f0 tx=0x7f2f7000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.299+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70017070 con 0x7f2f7815e0b0 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.299+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f700052d0 con 0x7f2f7815e0b0 2024-03-06T07:34:11.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.299+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70003850 con 0x7f2f7815e0b0 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.299+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f780cbb50 con 0x7f2f7815e0b0 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.299+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f780cc110 con 0x7f2f7815e0b0 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.300+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7804f330 con 0x7f2f7815e0b0 2024-03-06T07:34:11.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.302+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f7001a070 con 0x7f2f7815e0b0 2024-03-06T07:34:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.302+0000 7f2f74c00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f5406b4d0 0x7f2f5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:11.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.302+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f2f700895a0 con 0x7f2f7815e0b0 2024-03-06T07:34:11.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.303+0000 7f2f77e00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f5406b4d0 0x7f2f5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:11.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.304+0000 7f2f77e00640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f5406b4d0 0x7f2f5406d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2f68005520 tx=0x7f2f68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:11.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.307+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f70058390 con 0x7f2f7815e0b0 2024-03-06T07:34:11.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T07:34:11.399+0000 7f19f1400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:34:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.570+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f78161640 con 0x7f2f7815e0b0 2024-03-06T07:34:11.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.572+0000 7f2f74c00640 1 -- 172.21.15.6:0/898838494 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f2f70057c90 con 0x7f2f7815e0b0 2024-03-06T07:34:11.574 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:11.574 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:11.574 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.574+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f5406b4d0 msgr2=0x7f2f5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.574+0000 7f2f7d600640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f5406b4d0 0x7f2f5406d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2f68005520 tx=0x7f2f68009040 comp rx=0 tx=0).stop 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.574+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 msgr2=0x7f2f7815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.574+0000 7f2f7d600640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 0x7f2f7815c020 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c3f0 tx=0x7f2f7000c8b0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 shutdown_connections 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f5406b4d0 0x7f2f5406d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2f780d4a10 0x7f2f7815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2f7815e0b0 0x7f2f7815c020 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 --2- 172.21.15.6:0/898838494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2f780d0c80 0x7f2f780b93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 >> 172.21.15.6:0/898838494 conn(0x7f2f780d3d30 msgr2=0x7f2f780b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 shutdown_connections 2024-03-06T07:34:11.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:11.575+0000 7f2f7d600640 1 -- 172.21.15.6:0/898838494 wait complete. 2024-03-06T07:34:11.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:34:11.831 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:34:11.832 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:34:11.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.098+0000 7f0aa1e00640 1 Processor -- start 2024-03-06T07:34:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.098+0000 7f0aa1e00640 1 -- start start 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.099+0000 7f0aa1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c0d2960 0x7f0a9c0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.099+0000 7f0aa1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a9c0d0b00 0x7f0a9c0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.099+0000 7f0aa1e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0d2dc0 con 0x7f0a9c15e070 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.099+0000 7f0aa1e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9c0d18c0 con 0x7f0a9c0d0b00 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.099+0000 7f0aa1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0d1ba0 con 0x7f0a9c0d2960 2024-03-06T07:34:12.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.100+0000 7f0aa1400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a9c0d0b00 0x7f0a9c0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.100+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c0d2960 0x7f0a9c0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.100+0000 7f0aa1400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a9c0d0b00 0x7f0a9c0b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:36322/0 (socket says 172.21.15.6:36322) 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.100+0000 7f0aa1400640 1 -- 172.21.15.6:0/2427354301 learned_addr learned my addr 172.21.15.6:0/2427354301 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.101+0000 7f0aa1400640 1 -- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c0d2960 msgr2=0x7f0a9c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.101+0000 7f0aa1400640 1 --2- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c0d2960 0x7f0a9c0d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.101+0000 7f0aa1400640 1 -- 172.21.15.6:0/2427354301 >> v1:172.21.15.178:6789/0 conn(0x7f0a9c15e070 legacy=0x7f0a9c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.101+0000 7f0aa1400640 1 -- 172.21.15.6:0/2427354301 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0d1e80 con 0x7f0a9c0d0b00 2024-03-06T07:34:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.102+0000 7f0aa1400640 1 --2- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c0b3630 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009e50 tx=0x7f0a9402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=353186b738419168 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0a9b400640 1 -- 172.21.15.6:0/2427354301 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9402cb00 con 0x7f0a9c0d0b00 2024-03-06T07:34:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0a9b400640 1 -- 172.21.15.6:0/2427354301 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9402cca0 con 0x7f0a9c0d0b00 2024-03-06T07:34:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0a9b400640 1 -- 172.21.15.6:0/2427354301 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94034610 con 0x7f0a9c0d0b00 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0aa1e00640 1 -- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 msgr2=0x7f0a9c0b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c0b3630 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009e50 tx=0x7f0a9402bf60 comp rx=0 tx=0).stop 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0aa1e00640 1 -- 172.21.15.6:0/2427354301 shutdown_connections 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c0b3630 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/2427354301 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c0d2960 0x7f0a9c0d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.103+0000 7f0aa1e00640 1 -- 172.21.15.6:0/2427354301 >> 172.21.15.6:0/2427354301 conn(0x7f0a9c08d2a0 msgr2=0x7f0a9c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.104+0000 7f0aa1e00640 1 -- 172.21.15.6:0/2427354301 shutdown_connections 2024-03-06T07:34:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.104+0000 7f0aa1e00640 1 -- 172.21.15.6:0/2427354301 wait complete. 2024-03-06T07:34:12.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.105+0000 7f0aa1e00640 1 Processor -- start 2024-03-06T07:34:12.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.105+0000 7f0aa1e00640 1 -- start start 2024-03-06T07:34:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.106+0000 7f0aa1e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.106+0000 7f0aa1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 0x7f0a9c160890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.106+0000 7f0aa1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c15e070 0x7f0a9c150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.106+0000 7f0aa1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0d45b0 con 0x7f0a9c0d2960 2024-03-06T07:34:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.106+0000 7f0aa1e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0d2440 con 0x7f0a9c0d0b00 2024-03-06T07:34:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.106+0000 7f0aa1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0d4730 con 0x7f0a9c15e070 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 0x7f0a9c160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 0x7f0a9c160890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42250/0 (socket says 172.21.15.6:42250) 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 -- 172.21.15.6:0/380950760 learned_addr learned my addr 172.21.15.6:0/380950760 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0aa1400640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c15e070 0x7f0a9c150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0aa0a00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 -- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c15e070 msgr2=0x7f0a9c150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c15e070 0x7f0a9c150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 -- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 msgr2=0x7f0a9c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c14dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0a9be00640 1 -- 172.21.15.6:0/380950760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c160dd0 con 0x7f0a9c0d2960 2024-03-06T07:34:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.107+0000 7f0aa1400640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c15e070 0x7f0a9c150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.108+0000 7f0a9be00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 0x7f0a9c160890 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800e9b0 tx=0x7f0a8800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.108+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88017070 con 0x7f0a9c0d2960 2024-03-06T07:34:12.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.108+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8800c9a0 con 0x7f0a9c0d2960 2024-03-06T07:34:12.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.108+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800cd90 con 0x7f0a9c0d2960 2024-03-06T07:34:12.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.108+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c0b6a90 con 0x7f0a9c0d2960 2024-03-06T07:34:12.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.108+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c15fdb0 con 0x7f0a9c0d2960 2024-03-06T07:34:12.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.109+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9c068b30 con 0x7f0a9c0d2960 2024-03-06T07:34:12.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.110+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a8801a070 con 0x7f0a9c0d2960 2024-03-06T07:34:12.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.111+0000 7f0a98c00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a7c06b4d0 0x7f0a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:12.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.112+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0a8808a370 con 0x7f0a9c0d2960 2024-03-06T07:34:12.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.112+0000 7f0aa0a00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a7c06b4d0 0x7f0a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:12.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.114+0000 7f0aa0a00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a7c06b4d0 0x7f0a7c06d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0a90007fd0 tx=0x7f0a90009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:12.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.116+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a9c068b30 con 0x7f0a9c0d2960 2024-03-06T07:34:12.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.433+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a9c04f330 con 0x7f0a9c0d2960 2024-03-06T07:34:12.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.434+0000 7f0a98c00640 1 -- 172.21.15.6:0/380950760 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f0a880590c0 con 0x7f0a9c0d2960 2024-03-06T07:34:12.437 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:12.437 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:12.437 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:12.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.437+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a7c06b4d0 msgr2=0x7f0a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.437+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a7c06b4d0 0x7f0a7c06d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f0a90007fd0 tx=0x7f0a90009040 comp rx=0 tx=0).stop 2024-03-06T07:34:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.437+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 msgr2=0x7f0a9c160890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.437+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 0x7f0a9c160890 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800e9b0 tx=0x7f0a8800ee70 comp rx=0 tx=0).stop 2024-03-06T07:34:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 shutdown_connections 2024-03-06T07:34:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a7c06b4d0 0x7f0a7c06d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a9c15e070 0x7f0a9c150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a9c0d2960 0x7f0a9c160890 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 --2- 172.21.15.6:0/380950760 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a9c0d0b00 0x7f0a9c14dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 >> 172.21.15.6:0/380950760 conn(0x7f0a9c08d2a0 msgr2=0x7f0a9c06b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 shutdown_connections 2024-03-06T07:34:12.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:12.438+0000 7f0aa1e00640 1 -- 172.21.15.6:0/380950760 wait complete. 2024-03-06T07:34:12.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:34:12.696 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:34:12.697 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:34:23.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:23.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:42.697 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:34:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.802+0000 7ffbea239f40 1 Processor -- start 2024-03-06T07:34:42.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.802+0000 7ffbea239f40 1 -- start start 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbea239f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 0x556fb22a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbea239f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb2205c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbea239f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb22b13c0 0x556fb22ac540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbea239f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb2180fd0 con 0x556fb21dcb90 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbea239f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb2184f30 con 0x556fb2223680 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbea239f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fb22ad230 con 0x556fb22b13c0 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbe6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 0x556fb22a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbe6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 0x556fb22a6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59596/0 (socket says 172.21.15.6:59596) 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.803+0000 7ffbe6c00640 1 -- 172.21.15.6:0/3416732167 learned_addr learned my addr 172.21.15.6:0/3416732167 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6200640 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb2205c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe7600640 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb22b13c0 0x556fb22ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6c00640 1 -- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb22b13c0 msgr2=0x556fb22ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb22b13c0 0x556fb22ac540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6c00640 1 -- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 msgr2=0x556fb2205c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb2205c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6c00640 1 -- 172.21.15.6:0/3416732167 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fb2204330 con 0x556fb21dcb90 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 0x556fb22a6c30 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8001d90 tx=0x7ffbd800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c585ef40a610e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.804+0000 7ffbe4e00640 1 -- 172.21.15.6:0/3416732167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8012020 con 0x556fb21dcb90 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbe4e00640 1 -- 172.21.15.6:0/3416732167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd800e670 con 0x556fb21dcb90 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbe4e00640 1 -- 172.21.15.6:0/3416732167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd800eac0 con 0x556fb21dcb90 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 -- 172.21.15.6:0/3416732167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 msgr2=0x556fb22a6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 0x556fb22a6c30 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8001d90 tx=0x7ffbd800b6c0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 -- 172.21.15.6:0/3416732167 shutdown_connections 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb2205c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb22b13c0 0x556fb22ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 --2- 172.21.15.6:0/3416732167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb21dcb90 0x556fb22a6c30 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 -- 172.21.15.6:0/3416732167 >> 172.21.15.6:0/3416732167 conn(0x556fb21da5b0 msgr2=0x556fb22b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:42.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 -- 172.21.15.6:0/3416732167 shutdown_connections 2024-03-06T07:34:42.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.805+0000 7ffbea239f40 1 -- 172.21.15.6:0/3416732167 wait complete. 2024-03-06T07:34:42.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.806+0000 7ffbea239f40 1 Processor -- start 2024-03-06T07:34:42.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.806+0000 7ffbea239f40 1 -- start start 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbea239f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb21dcb90 0x556fb22b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbea239f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb22adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbea239f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 0x556fb22abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbea239f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb2203e00 con 0x556fb22b13c0 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbea239f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fb22a67a0 con 0x556fb2223680 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbea239f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fb2204090 con 0x556fb21dcb90 2024-03-06T07:34:42.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.807+0000 7ffbe7600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 0x556fb22abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 0x556fb22abc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59612/0 (socket says 172.21.15.6:59612) 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 learned_addr learned my addr 172.21.15.6:0/2851313929 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb21dcb90 0x556fb22b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb22adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb21dcb90 msgr2=0x556fb22b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x556fb21dcb90 0x556fb22b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 msgr2=0x556fb22adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556fb2223680 0x556fb22adab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.808+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fb22b4ac0 con 0x556fb22b13c0 2024-03-06T07:34:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.809+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 0x556fb22abc90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0003190 tx=0x7ffbe000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.809+0000 7ffbdf400640 1 -- 172.21.15.6:0/2851313929 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0015030 con 0x556fb22b13c0 2024-03-06T07:34:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.809+0000 7ffbdf400640 1 -- 172.21.15.6:0/2851313929 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0006020 con 0x556fb22b13c0 2024-03-06T07:34:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.809+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556fb234a560 con 0x556fb22b13c0 2024-03-06T07:34:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.809+0000 7ffbdf400640 1 -- 172.21.15.6:0/2851313929 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0005440 con 0x556fb22b13c0 2024-03-06T07:34:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.809+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fb234ab20 con 0x556fb22b13c0 2024-03-06T07:34:42.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.811+0000 7ffbdf400640 1 -- 172.21.15.6:0/2851313929 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbe0018070 con 0x556fb22b13c0 2024-03-06T07:34:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.812+0000 7ffbdf400640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffbc8069eb0 0x7ffbc806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.812+0000 7ffbdf400640 1 -- 172.21.15.6:0/2851313929 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ffbe0088b80 con 0x556fb22b13c0 2024-03-06T07:34:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.812+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x556fb221dc50 0x556fb221e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.813+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x556fb221e570 con 0x556fb221dc50 2024-03-06T07:34:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.813+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffbc8069eb0 0x7ffbc806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.813+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x556fb221dc50 0x556fb221e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.814+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffbc8069eb0 0x7ffbc806c370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8001d90 tx=0x7ffbd801b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.815+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x556fb221dc50 0x556fb221e030 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.816+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.816+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7ffbd405d040 0x7ffbd4006380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.816+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x556fb221e570 con 0x7ffbd405d040 2024-03-06T07:34:42.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.816+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7ffbd405d040 0x7ffbd4006380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.817+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7ffbd405d040 0x7ffbd4006380 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.817+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.817+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7ffbd8017040 0x7ffbd8025410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.817+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x556fb221e570 con 0x7ffbd8017040 2024-03-06T07:34:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.818+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7ffbd8017040 0x7ffbd8025410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.818+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7ffbd8017040 0x7ffbd8025410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.819+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.819+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7ffbd405c430 0x7ffbd4061950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.819+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x556fb221e570 con 0x7ffbd405c430 2024-03-06T07:34:42.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.819+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7ffbd405c430 0x7ffbd4061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.820+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7ffbd405c430 0x7ffbd4061950 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.821+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.5 v2:172.21.15.178:6800/1128394266 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd405c430 2024-03-06T07:34:42.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.821+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x556fb221e570 con 0x556fb221dc50 2024-03-06T07:34:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.822+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.822+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x556fb221e570 con 0x7ffbd8017040 2024-03-06T07:34:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.822+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.822+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7ffbd406d880 0x7ffbd405b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.822+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.823+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7ffbd406d880 0x7ffbd405b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.823+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7ffbd406d880 0x7ffbd405b340 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.824+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 1 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.824+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.825+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.825+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.825+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 2 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.825+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.826+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.826+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.826+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.826+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.826+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 3 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.826+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.827+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.827+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7ffbd405d420 0x7ffbd405d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.827+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.828+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7ffbd405d420 0x7ffbd405d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.829+0000 7ffbe6200640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7ffbd405d420 0x7ffbd405d880 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.830+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.830+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d040 2024-03-06T07:34:42.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.830+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.831+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.831+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 4 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.831+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.832+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.832+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405c430 2024-03-06T07:34:42.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.833+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.5 v2:172.21.15.178:6800/1128394266 2 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd405c430 2024-03-06T07:34:42.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.833+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.833+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.833+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.834+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.835+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.835+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.835+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.836+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.836+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.837+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 5 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.837+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d040 2024-03-06T07:34:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.837+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.837+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7ffbd8035920 0x7ffbd800d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.837+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8035920 2024-03-06T07:34:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.838+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7ffbd8035920 0x7ffbd800d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.839+0000 7ffbe7600640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7ffbd8035920 0x7ffbd800d3d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.839+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.6 v2:172.21.15.178:6824/196133891 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd8035920 2024-03-06T07:34:42.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.840+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.840+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 6 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.840+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405c430 2024-03-06T07:34:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.841+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.5 v2:172.21.15.178:6800/1128394266 3 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd405c430 2024-03-06T07:34:42.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.841+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.842+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 7 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.842+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7ffbd80163b0 0x7ffbd8016830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.842+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd80163b0 2024-03-06T07:34:42.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.842+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7ffbd80163b0 0x7ffbd8016830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:42.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.843+0000 7ffbe6c00640 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7ffbd80163b0 0x7ffbd8016830 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.844+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.2 v2:172.21.15.6:6822/1963740543 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd80163b0 2024-03-06T07:34:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.844+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8035920 2024-03-06T07:34:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.845+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.6 v2:172.21.15.178:6824/196133891 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd8035920 2024-03-06T07:34:42.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.845+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d040 2024-03-06T07:34:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.845+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.845+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.846+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.846+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.847+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.847+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.848+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.848+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.848+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.849+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d040 2024-03-06T07:34:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.849+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.849+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.850+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.850+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.851+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.851+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405c430 2024-03-06T07:34:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.852+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.5 v2:172.21.15.178:6800/1128394266 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd405c430 2024-03-06T07:34:42.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.852+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405c430 2024-03-06T07:34:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.853+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.5 v2:172.21.15.178:6800/1128394266 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd405c430 2024-03-06T07:34:42.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.853+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8035920 2024-03-06T07:34:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.854+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.6 v2:172.21.15.178:6824/196133891 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd8035920 2024-03-06T07:34:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.854+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d040 2024-03-06T07:34:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.855+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.855+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405c430 2024-03-06T07:34:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.856+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.5 v2:172.21.15.178:6800/1128394266 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd405c430 2024-03-06T07:34:42.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.856+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.858+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.858+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.859+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.859+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.860+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.860+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.861+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.861+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.862+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.862+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.863+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.863+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.864+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.864+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8017040 2024-03-06T07:34:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.865+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.3 v2:172.21.15.6:6816/1271003476 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd8017040 2024-03-06T07:34:42.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.865+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x556fb221dc50 2024-03-06T07:34:42.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.866+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.7 v2:172.21.15.178:6808/56642032 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x556fb221dc50 2024-03-06T07:34:42.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.866+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.867+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.867+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.868+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.868+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.870+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.870+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd80163b0 2024-03-06T07:34:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.871+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.2 v2:172.21.15.6:6822/1963740543 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd80163b0 2024-03-06T07:34:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.871+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd80163b0 2024-03-06T07:34:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.871+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.2 v2:172.21.15.6:6822/1963740543 3 ==== osd_op_reply(57 [pgnls start_epoch 114 out=81b] v108'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd80163b0 2024-03-06T07:34:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.871+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.873+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.873+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8035920 2024-03-06T07:34:42.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.874+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.6 v2:172.21.15.178:6824/196133891 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd8035920 2024-03-06T07:34:42.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.874+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.874+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 11 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.874+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d040 2024-03-06T07:34:42.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.875+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.1 v2:172.21.15.6:6808/1355309078 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd405d040 2024-03-06T07:34:42.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.875+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd405d420 2024-03-06T07:34:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.876+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 <== osd.4 v2:172.21.15.178:6809/2203214816 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=81b] v108'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7ffbd4052960 con 0x7ffbd405d420 2024-03-06T07:34:42.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.876+0000 7ffbe6200640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd8035920 2024-03-06T07:34:42.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.877+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 <== osd.6 v2:172.21.15.178:6824/196133891 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbe005b120 con 0x7ffbd8035920 2024-03-06T07:34:42.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.877+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x7ffbd406e040 con 0x7ffbd406d880 2024-03-06T07:34:42.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.878+0000 7ffbe6c00640 1 -- 172.21.15.6:0/2851313929 <== osd.0 v2:172.21.15.6:6800/2712143005 12 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7ffbd800e910 con 0x7ffbd406d880 2024-03-06T07:34:42.881 INFO:teuthology.orchestra.run.smithi006.stdout:10000000001.00000000 2024-03-06T07:34:42.881 INFO:teuthology.orchestra.run.smithi006.stdout:10000000001.00000001 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.878+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7ffbd406d880 msgr2=0x7ffbd405b340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.878+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7ffbd406d880 0x7ffbd405b340 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.878+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7ffbd405d040 msgr2=0x7ffbd4006380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.878+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7ffbd405d040 0x7ffbd4006380 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.878+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7ffbd80163b0 msgr2=0x7ffbd8016830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7ffbd80163b0 0x7ffbd8016830 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7ffbd8017040 msgr2=0x7ffbd8025410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7ffbd8017040 0x7ffbd8025410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7ffbd405d420 msgr2=0x7ffbd405d880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 reap_dead start 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7ffbd405d420 0x7ffbd405d880 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7ffbd405c430 msgr2=0x7ffbd4061950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7ffbd405c430 0x7ffbd4061950 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7ffbd8035920 msgr2=0x7ffbd800d3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7ffbd8035920 0x7ffbd800d3d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x556fb221dc50 msgr2=0x556fb221e030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x556fb221dc50 0x556fb221e030 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffbc8069eb0 msgr2=0x7ffbc806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffbc8069eb0 0x7ffbc806c370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8001d90 tx=0x7ffbd801b040 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbe7600640 1 -- 172.21.15.6:0/2851313929 reap_dead start 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 msgr2=0x556fb22abc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 0x556fb22abc90 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0003190 tx=0x7ffbe000c520 comp rx=0 tx=0).stop 2024-03-06T07:34:42.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 shutdown_connections 2024-03-06T07:34:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 --2- 172.21.15.6:0/2851313929 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x556fb22b13c0 0x556fb22abc90 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.879+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 >> 172.21.15.6:0/2851313929 conn(0x556fb21da5b0 msgr2=0x556fb22a32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.880+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 shutdown_connections 2024-03-06T07:34:42.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:42.880+0000 7ffbea239f40 1 -- 172.21.15.6:0/2851313929 wait complete. 2024-03-06T07:34:42.890 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:34:42.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.108+0000 7f9ef1e00640 1 Processor -- start 2024-03-06T07:34:43.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.108+0000 7f9ef1e00640 1 -- start start 2024-03-06T07:34:43.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.109+0000 7f9ef1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9eec0af1a0 0x7f9eec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9ef1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9eec0cdc20 0x7f9eec0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9ef1e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eec15cc40 con 0x7f9eec0af580 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9ef1e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec15cf20 con 0x7f9eec0cdc20 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9ef1e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec15d200 con 0x7f9eec0af1a0 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9eeb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9eec0cdc20 0x7f9eec0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9eeb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9eec0cdc20 0x7f9eec0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59622/0 (socket says 172.21.15.6:59622) 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9eeb400640 1 -- 172.21.15.6:0/54906049 learned_addr learned my addr 172.21.15.6:0/54906049 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9eeb400640 1 -- 172.21.15.6:0/54906049 >> v2:172.21.15.178:3300/0 conn(0x7f9eec0af1a0 msgr2=0x7f9eec15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.110+0000 7f9eeaa00640 1 --2- 172.21.15.6:0/54906049 >> v2:172.21.15.178:3300/0 conn(0x7f9eec0af1a0 0x7f9eec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.111+0000 7f9eeb400640 1 --2- 172.21.15.6:0/54906049 >> v2:172.21.15.178:3300/0 conn(0x7f9eec0af1a0 0x7f9eec15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.111+0000 7f9eeb400640 1 -- 172.21.15.6:0/54906049 >> v1:172.21.15.178:6789/0 conn(0x7f9eec0af580 legacy=0x7f9eec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.111+0000 7f9eeb400640 1 -- 172.21.15.6:0/54906049 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec15d4e0 con 0x7f9eec0cdc20 2024-03-06T07:34:43.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.111+0000 7f9eeaa00640 1 --2- 172.21.15.6:0/54906049 >> v2:172.21.15.178:3300/0 conn(0x7f9eec0af1a0 0x7f9eec15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:43.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.112+0000 7f9eeb400640 1 --2- 172.21.15.6:0/54906049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0cdc20 0x7f9eec0b2dd0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9eec074960 tx=0x7f9ee402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5936b558c89e58d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ee9600640 1 -- 172.21.15.6:0/54906049 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee402c9f0 con 0x7f9eec0cdc20 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ee9600640 1 -- 172.21.15.6:0/54906049 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee402cb90 con 0x7f9eec0cdc20 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ee9600640 1 -- 172.21.15.6:0/54906049 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee40343f0 con 0x7f9eec0cdc20 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ef1e00640 1 -- 172.21.15.6:0/54906049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0cdc20 msgr2=0x7f9eec0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/54906049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0cdc20 0x7f9eec0b2dd0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9eec074960 tx=0x7f9ee402be50 comp rx=0 tx=0).stop 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ef1e00640 1 -- 172.21.15.6:0/54906049 shutdown_connections 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/54906049 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0cdc20 0x7f9eec0b2dd0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/54906049 >> v2:172.21.15.178:3300/0 conn(0x7f9eec0af1a0 0x7f9eec15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.113+0000 7f9ef1e00640 1 -- 172.21.15.6:0/54906049 >> 172.21.15.6:0/54906049 conn(0x7f9eec06afb0 msgr2=0x7f9eec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.114+0000 7f9ef1e00640 1 -- 172.21.15.6:0/54906049 shutdown_connections 2024-03-06T07:34:43.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.114+0000 7f9ef1e00640 1 -- 172.21.15.6:0/54906049 wait complete. 2024-03-06T07:34:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.115+0000 7f9ef1e00640 1 Processor -- start 2024-03-06T07:34:43.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.115+0000 7f9ef1e00640 1 -- start start 2024-03-06T07:34:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.116+0000 7f9ef1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9eec0af1a0 0x7f9eec14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.116+0000 7f9ef1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 0x7f9eec1500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.116+0000 7f9ef1e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec0cdc20 0x7f9eec158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.116+0000 7f9ef1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0b36a0 con 0x7f9eec0af580 2024-03-06T07:34:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.116+0000 7f9ef1e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0b3520 con 0x7f9eec0cdc20 2024-03-06T07:34:43.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.116+0000 7f9ef1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0b3820 con 0x7f9eec0af1a0 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 0x7f9eec1500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 0x7f9eec1500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59630/0 (socket says 172.21.15.6:59630) 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 -- 172.21.15.6:0/572576641 learned_addr learned my addr 172.21.15.6:0/572576641 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eeb400640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec0cdc20 0x7f9eec158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eeaa00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9eec0af1a0 0x7f9eec14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 -- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9eec0af1a0 msgr2=0x7f9eec14e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9eec0af1a0 0x7f9eec14e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 -- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec0cdc20 msgr2=0x7f9eec158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec0cdc20 0x7f9eec158b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.117+0000 7f9eea000640 1 -- 172.21.15.6:0/572576641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec06db40 con 0x7f9eec0af580 2024-03-06T07:34:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.118+0000 7f9eea000640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 0x7f9eec1500d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c400 tx=0x7f9edc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.118+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc011790 con 0x7f9eec0af580 2024-03-06T07:34:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.118+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc017070 con 0x7f9eec0af580 2024-03-06T07:34:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.118+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc011bb0 con 0x7f9eec0af580 2024-03-06T07:34:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.118+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec06de20 con 0x7f9eec0af580 2024-03-06T07:34:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.118+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec0ca830 con 0x7f9eec0af580 2024-03-06T07:34:43.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.119+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec069510 con 0x7f9eec0af580 2024-03-06T07:34:43.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.121+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9edc01c460 con 0x7f9eec0af580 2024-03-06T07:34:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.122+0000 7f9ee3400640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ec806b4d0 0x7f9ec806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.122+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9edc08a050 con 0x7f9eec0af580 2024-03-06T07:34:43.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.123+0000 7f9eeaa00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ec806b4d0 0x7f9ec806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.124+0000 7f9eeaa00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ec806b4d0 0x7f9ec806d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4008850 tx=0x7f9ed4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.132+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9edc058ed0 con 0x7f9eec0af580 2024-03-06T07:34:43.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.372+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9eec04f3a0 con 0x7f9eec0af580 2024-03-06T07:34:43.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.373+0000 7f9ee3400640 1 -- 172.21.15.6:0/572576641 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f9edc0587d0 con 0x7f9eec0af580 2024-03-06T07:34:43.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:43.376 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:43.376 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ec806b4d0 msgr2=0x7f9ec806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ec806b4d0 0x7f9ec806d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4008850 tx=0x7f9ed4008040 comp rx=0 tx=0).stop 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 msgr2=0x7f9eec1500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 0x7f9eec1500d0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c400 tx=0x7f9edc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 shutdown_connections 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ec806b4d0 0x7f9ec806d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9eec0cdc20 0x7f9eec158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9eec0af580 0x7f9eec1500d0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 --2- 172.21.15.6:0/572576641 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9eec0af1a0 0x7f9eec14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 >> 172.21.15.6:0/572576641 conn(0x7f9eec06afb0 msgr2=0x7f9eec15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:43.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 shutdown_connections 2024-03-06T07:34:43.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.377+0000 7f9ef1e00640 1 -- 172.21.15.6:0/572576641 wait complete. 2024-03-06T07:34:43.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:34:43.633 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:34:43.633 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:34:43.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:43.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.853+0000 7fb06a200640 1 Processor -- start 2024-03-06T07:34:43.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.853+0000 7fb06a200640 1 -- start start 2024-03-06T07:34:43.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.855+0000 7fb06a200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb06415e0b0 0x7fb0640cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.855+0000 7fb06a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0640d4a10 0x7fb0640d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.855+0000 7fb06a200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0640d0c80 0x7fb0640d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.855+0000 7fb06a200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0640b09e0 con 0x7fb0640d0c80 2024-03-06T07:34:43.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.855+0000 7fb06a200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0640b3710 con 0x7fb06415e0b0 2024-03-06T07:34:43.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.855+0000 7fb06a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640b39f0 con 0x7fb0640d4a10 2024-03-06T07:34:43.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0640d0c80 0x7fb0640d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb0640d0c80 0x7fb0640d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59632/0 (socket says 172.21.15.6:59632) 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 -- 172.21.15.6:0/999727011 learned_addr learned my addr 172.21.15.6:0/999727011 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb068e00640 1 --2- 172.21.15.6:0/999727011 >> v2:172.21.15.178:3300/0 conn(0x7fb06415e0b0 0x7fb0640cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb05be00640 1 --2- 172.21.15.6:0/999727011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0640d4a10 0x7fb0640d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 -- 172.21.15.6:0/999727011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0640d4a10 msgr2=0x7fb0640d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 --2- 172.21.15.6:0/999727011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0640d4a10 0x7fb0640d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 -- 172.21.15.6:0/999727011 >> v2:172.21.15.178:3300/0 conn(0x7fb06415e0b0 msgr2=0x7fb0640cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 --2- 172.21.15.6:0/999727011 >> v2:172.21.15.178:3300/0 conn(0x7fb06415e0b0 0x7fb0640cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb069800640 1 -- 172.21.15.6:0/999727011 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0640b3cd0 con 0x7fb0640d0c80 2024-03-06T07:34:43.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.856+0000 7fb068e00640 1 --2- 172.21.15.6:0/999727011 >> v2:172.21.15.178:3300/0 conn(0x7fb06415e0b0 0x7fb0640cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:43.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.858+0000 7fb069800640 1 --2- 172.21.15.6:0/999727011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb0640d1900 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb0640b68c0 tx=0x7fb05c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc861429bb43534 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.858+0000 7fb063e00640 1 -- 172.21.15.6:0/999727011 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c02ca70 con 0x7fb0640d0c80 2024-03-06T07:34:43.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.858+0000 7fb063e00640 1 -- 172.21.15.6:0/999727011 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c02cc10 con 0x7fb0640d0c80 2024-03-06T07:34:43.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.858+0000 7fb063e00640 1 -- 172.21.15.6:0/999727011 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0343f0 con 0x7fb0640d0c80 2024-03-06T07:34:43.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 -- 172.21.15.6:0/999727011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 msgr2=0x7fb0640d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 --2- 172.21.15.6:0/999727011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb0640d1900 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb0640b68c0 tx=0x7fb05c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 -- 172.21.15.6:0/999727011 shutdown_connections 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 --2- 172.21.15.6:0/999727011 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb0640d1900 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 --2- 172.21.15.6:0/999727011 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0640d4a10 0x7fb0640d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 --2- 172.21.15.6:0/999727011 >> v2:172.21.15.178:3300/0 conn(0x7fb06415e0b0 0x7fb0640cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 -- 172.21.15.6:0/999727011 >> 172.21.15.6:0/999727011 conn(0x7fb0640d3d30 msgr2=0x7fb0640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.859+0000 7fb06a200640 1 -- 172.21.15.6:0/999727011 shutdown_connections 2024-03-06T07:34:43.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.860+0000 7fb06a200640 1 -- 172.21.15.6:0/999727011 wait complete. 2024-03-06T07:34:43.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.862+0000 7fb06a200640 1 Processor -- start 2024-03-06T07:34:43.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.862+0000 7fb06a200640 1 -- start start 2024-03-06T07:34:43.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb06a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb064160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb06a200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0640d4a10 0x7fb0640b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb06a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06415e0b0 0x7fb0640b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb06a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640cfc60 con 0x7fb0640d0c80 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb06a200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06415e490 con 0x7fb0640d4a10 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb06a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0640cf850 con 0x7fb06415e0b0 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb068e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb064160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.863+0000 7fb068e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb064160670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59644/0 (socket says 172.21.15.6:59644) 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 -- 172.21.15.6:0/3760804016 learned_addr learned my addr 172.21.15.6:0/3760804016 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb069800640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06415e0b0 0x7fb0640b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb05be00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0640d4a10 0x7fb0640b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 -- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06415e0b0 msgr2=0x7fb0640b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06415e0b0 0x7fb0640b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 -- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0640d4a10 msgr2=0x7fb0640b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0640d4a10 0x7fb0640b8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 -- 172.21.15.6:0/3760804016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06415fe80 con 0x7fb0640d0c80 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb069800640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06415e0b0 0x7fb0640b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb05be00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0640d4a10 0x7fb0640b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.864+0000 7fb068e00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb064160670 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fb05400d7d0 tx=0x7fb05400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.865+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054004030 con 0x7fb0640d0c80 2024-03-06T07:34:43.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.865+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0540041d0 con 0x7fb0640d0c80 2024-03-06T07:34:43.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.865+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05400b6c0 con 0x7fb0640d0c80 2024-03-06T07:34:43.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.865+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06415f020 con 0x7fb0640d0c80 2024-03-06T07:34:43.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.865+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06415f8c0 con 0x7fb0640d0c80 2024-03-06T07:34:43.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.866+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06404f330 con 0x7fb0640d0c80 2024-03-06T07:34:43.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.867+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb05400b860 con 0x7fb0640d0c80 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.868+0000 7fb061600640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04006b4d0 0x7fb04006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.868+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb05408a490 con 0x7fb0640d0c80 2024-03-06T07:34:43.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.869+0000 7fb05be00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04006b4d0 0x7fb04006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:43.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.870+0000 7fb05be00640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04006b4d0 0x7fb04006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb06415d090 tx=0x7fb04c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:43.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:43.872+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb054059270 con 0x7fb0640d0c80 2024-03-06T07:34:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.117+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0640ca5c0 con 0x7fb0640d0c80 2024-03-06T07:34:44.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.118+0000 7fb061600640 1 -- 172.21.15.6:0/3760804016 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fb054058b70 con 0x7fb0640d0c80 2024-03-06T07:34:44.121 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:44.121 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:44.121 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.121+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04006b4d0 msgr2=0x7fb04006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.121+0000 7fb06a200640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04006b4d0 0x7fb04006d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb06415d090 tx=0x7fb04c008040 comp rx=0 tx=0).stop 2024-03-06T07:34:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.121+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 msgr2=0x7fb064160670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.121+0000 7fb06a200640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb064160670 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fb05400d7d0 tx=0x7fb05400dc90 comp rx=0 tx=0).stop 2024-03-06T07:34:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 shutdown_connections 2024-03-06T07:34:44.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04006b4d0 0x7fb04006d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb0640d4a10 0x7fb0640b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06415e0b0 0x7fb0640b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 --2- 172.21.15.6:0/3760804016 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0640d0c80 0x7fb064160670 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 >> 172.21.15.6:0/3760804016 conn(0x7fb0640d3d30 msgr2=0x7fb0640cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:44.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 shutdown_connections 2024-03-06T07:34:44.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.122+0000 7fb06a200640 1 -- 172.21.15.6:0/3760804016 wait complete. 2024-03-06T07:34:44.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:34:44.396 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:34:44.396 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:44.397 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:34:44.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.481+0000 7f7b0763df40 1 Processor -- start 2024-03-06T07:34:44.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.481+0000 7f7b0763df40 1 -- start start 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.482+0000 7f7b0763df40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5573969241e0 0x5573969c5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.482+0000 7f7b0763df40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 0x5573968fca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.482+0000 7f7b0763df40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969ce330 0x5573969cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.482+0000 7f7b0763df40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x557396942bb0 con 0x557396925a90 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.482+0000 7f7b0763df40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55739689ffd0 con 0x5573969241e0 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.482+0000 7f7b0763df40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573968fc620 con 0x5573969ce330 2024-03-06T07:34:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 0x5573968fca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 0x5573968fca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59672/0 (socket says 172.21.15.6:59672) 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 -- 172.21.15.6:0/2063524412 learned_addr learned my addr 172.21.15.6:0/2063524412 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 -- 172.21.15.6:0/2063524412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969ce330 msgr2=0x5573969cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969ce330 0x5573969cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 -- 172.21.15.6:0/2063524412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5573969241e0 msgr2=0x5573969c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5573969241e0 0x5573969c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 -- 172.21.15.6:0/2063524412 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573969cc970 con 0x557396925a90 2024-03-06T07:34:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.483+0000 7f7b03600640 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 0x5573968fca40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7af001cfe0 tx=0x7f7af004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edebea027f10647 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b02200640 1 -- 172.21.15.6:0/2063524412 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0057070 con 0x557396925a90 2024-03-06T07:34:44.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b02200640 1 -- 172.21.15.6:0/2063524412 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0061070 con 0x557396925a90 2024-03-06T07:34:44.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b02200640 1 -- 172.21.15.6:0/2063524412 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0052b50 con 0x557396925a90 2024-03-06T07:34:44.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 -- 172.21.15.6:0/2063524412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 msgr2=0x5573968fca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 0x5573968fca40 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f7af001cfe0 tx=0x7f7af004c000 comp rx=0 tx=0).stop 2024-03-06T07:34:44.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 -- 172.21.15.6:0/2063524412 shutdown_connections 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969ce330 0x5573969cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x557396925a90 0x5573968fca40 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 --2- 172.21.15.6:0/2063524412 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5573969241e0 0x5573969c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 -- 172.21.15.6:0/2063524412 >> 172.21.15.6:0/2063524412 conn(0x557396924670 msgr2=0x55739693fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 -- 172.21.15.6:0/2063524412 shutdown_connections 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.484+0000 7f7b0763df40 1 -- 172.21.15.6:0/2063524412 wait complete. 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.485+0000 7f7b0763df40 1 Processor -- start 2024-03-06T07:34:44.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.485+0000 7f7b0763df40 1 -- start start 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b0763df40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969241e0 0x5573969c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b0763df40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x557396925a90 0x5573968f81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b0763df40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573969ce330 0x5573968f8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b0763df40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573969c5590 con 0x5573969ce330 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b0763df40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55739693dd80 con 0x557396925a90 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b0763df40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573968fcf80 con 0x5573969241e0 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b04a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573969ce330 0x5573968f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b04a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573969ce330 0x5573968f8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59688/0 (socket says 172.21.15.6:59688) 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.486+0000 7f7b04a00640 1 -- 172.21.15.6:0/4026355634 learned_addr learned my addr 172.21.15.6:0/4026355634 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:44.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b03600640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x557396925a90 0x5573968f81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04000640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969241e0 0x5573969c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04a00640 1 -- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969241e0 msgr2=0x5573969c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04a00640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5573969241e0 0x5573969c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04a00640 1 -- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x557396925a90 msgr2=0x5573968f81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04a00640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x557396925a90 0x5573968f81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04a00640 1 -- 172.21.15.6:0/4026355634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573969c6070 con 0x5573969ce330 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b03600640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x557396925a90 0x5573968f81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.487+0000 7f7b04a00640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573969ce330 0x5573968f8720 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0091d0 tx=0x7f7afc009690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.488+0000 7f7b00e00640 1 -- 172.21.15.6:0/4026355634 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc014070 con 0x5573969ce330 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.488+0000 7f7b00e00640 1 -- 172.21.15.6:0/4026355634 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc010800 con 0x5573969ce330 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.488+0000 7f7b00e00640 1 -- 172.21.15.6:0/4026355634 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc010bd0 con 0x5573969ce330 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.488+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5573968f9290 con 0x5573969ce330 2024-03-06T07:34:44.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.488+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573969c90c0 con 0x5573969ce330 2024-03-06T07:34:44.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.490+0000 7f7b00e00640 1 -- 172.21.15.6:0/4026355634 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7afc020350 con 0x5573969ce330 2024-03-06T07:34:44.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.491+0000 7f7b00e00640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ae4069eb0 0x7f7ae406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.491+0000 7f7b00e00640 1 -- 172.21.15.6:0/4026355634 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7afc088e70 con 0x5573969ce330 2024-03-06T07:34:44.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.491+0000 7f7b0763df40 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x5573969c80d0 0x5573968f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.491+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x5573968f9a90 con 0x5573969c80d0 2024-03-06T07:34:44.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.491+0000 7f7b03600640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x5573969c80d0 0x5573968f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.491+0000 7f7b04000640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ae4069eb0 0x7f7ae406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.492+0000 7f7b03600640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x5573969c80d0 0x5573968f8c60 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.492+0000 7f7b03600640 1 -- 172.21.15.6:0/4026355634 <== osd.2 v2:172.21.15.6:6822/1963740543 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7f7af005a070 con 0x5573969c80d0 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x5573969c80d0 msgr2=0x5573968f8c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x5573969c80d0 0x5573968f8c60 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b04000640 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ae4069eb0 0x7f7ae406c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7af4001cf0 tx=0x7f7af4001310 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ae4069eb0 msgr2=0x7f7ae406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7ae4069eb0 0x7f7ae406c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7af4001cf0 tx=0x7f7af4001310 comp rx=0 tx=0).stop 2024-03-06T07:34:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573969ce330 msgr2=0x5573968f8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 --2- 172.21.15.6:0/4026355634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5573969ce330 0x5573968f8720 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0091d0 tx=0x7f7afc009690 comp rx=0 tx=0).stop 2024-03-06T07:34:44.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b04a00640 1 -- 172.21.15.6:0/4026355634 reap_dead start 2024-03-06T07:34:44.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 shutdown_connections 2024-03-06T07:34:44.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 >> 172.21.15.6:0/4026355634 conn(0x557396924670 msgr2=0x5573969cb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:44.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 shutdown_connections 2024-03-06T07:34:44.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.493+0000 7f7b0763df40 1 -- 172.21.15.6:0/4026355634 wait complete. 2024-03-06T07:34:44.499 INFO:teuthology.orchestra.run.smithi006.stdout:] 10000000001 stray0 2024-03-06T07:34:44.499 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1536, 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "10000000001", 2024-03-06T07:34:44.587 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 13 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 256, 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "stray0", 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 5 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:34:44.588 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:44.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.811+0000 7f2435600640 1 Processor -- start 2024-03-06T07:34:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.812+0000 7f2435600640 1 -- start start 2024-03-06T07:34:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.813+0000 7f2435600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 0x7f24300ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.813+0000 7f2435600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24300ce9a0 con 0x7f2430161670 2024-03-06T07:34:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.813+0000 7f2435600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24300cec80 con 0x7f2430071380 2024-03-06T07:34:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.813+0000 7f2435600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24300730d0 con 0x7f243006e240 2024-03-06T07:34:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.814+0000 7f242ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2430161670 0x7f2430070c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55870/0 (socket says 172.21.15.6:55870) 2024-03-06T07:34:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.814+0000 7f242ea00640 1 -- 172.21.15.6:0/3607174784 learned_addr learned my addr 172.21.15.6:0/3607174784 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.814+0000 7f242f400640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 0x7f24300ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.814+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266105305 0 0) 0x7f24300ce9a0 con 0x7f2430161670 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.814+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2414003520 con 0x7f2430161670 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748214089 0 0) 0x7f2414003520 con 0x7f2430161670 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 msgr2=0x7f24300ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 0x7f24300ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 >> v1:172.21.15.178:6789/0 conn(0x7f2430071380 legacy=0x7f24300ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24300cef60 con 0x7f2430161670 2024-03-06T07:34:44.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2420001670 con 0x7f2430161670 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 >> v1:172.21.15.6:6789/0 conn(0x7f2430161670 legacy=0x7f2430070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2414003a50 0x7f2414003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414004800 0x7f2414004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 0x7f24140051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300730d0 con 0x7f2414003a50 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241400bad0 con 0x7f2414004800 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241400bcd0 con 0x7f243006e240 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.815+0000 7f242ea00640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2414003a50 0x7f2414003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f242f400640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 0x7f24140051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f2427e00640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414004800 0x7f2414004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f242ea00640 1 -- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 msgr2=0x7f24140051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f242ea00640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243006e240 0x7f24140051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f242ea00640 1 -- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414004800 msgr2=0x7f2414004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f242ea00640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414004800 0x7f2414004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f242ea00640 1 -- 172.21.15.6:0/3607174784 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414003520 con 0x7f2414003a50 2024-03-06T07:34:44.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.816+0000 7f2427e00640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414004800 0x7f2414004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.817+0000 7f242ea00640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2414003a50 0x7f2414003e30 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f242000c240 tx=0x7f2414005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78041930ca4ed17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.817+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420007d10 con 0x7f2414003a50 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420030070 con 0x7f2414003a50 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f242e000640 1 -- 172.21.15.6:0/3607174784 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242002cd80 con 0x7f2414003a50 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f2435600640 1 -- 172.21.15.6:0/3607174784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2414003a50 msgr2=0x7f2414003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f2435600640 1 --2- 172.21.15.6:0/3607174784 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2414003a50 0x7f2414003e30 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f242000c240 tx=0x7f2414005830 comp rx=0 tx=0).stop 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f242f400640 1 -- 172.21.15.6:0/3607174784 reap_dead start 2024-03-06T07:34:44.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f2435600640 1 -- 172.21.15.6:0/3607174784 shutdown_connections 2024-03-06T07:34:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f2435600640 1 -- 172.21.15.6:0/3607174784 >> 172.21.15.6:0/3607174784 conn(0x7f243006afb0 msgr2=0x7f243006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f2435600640 1 -- 172.21.15.6:0/3607174784 shutdown_connections 2024-03-06T07:34:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.818+0000 7f2435600640 1 -- 172.21.15.6:0/3607174784 wait complete. 2024-03-06T07:34:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.820+0000 7f2435600640 1 Processor -- start 2024-03-06T07:34:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.820+0000 7f2435600640 1 -- start start 2024-03-06T07:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f2435600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414003a50 0x7f2430159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f2435600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243015e180 0x7f243014c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f2435600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 0x7f243014c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f2435600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300ce600 con 0x7f243015e5e0 2024-03-06T07:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f2435600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300cf520 con 0x7f2414003a50 2024-03-06T07:34:44.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f2435600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24300ce780 con 0x7f243015e180 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.821+0000 7f242f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 0x7f243014c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 0x7f243014c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59704/0 (socket says 172.21.15.6:59704) 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 -- 172.21.15.6:0/1007134949 learned_addr learned my addr 172.21.15.6:0/1007134949 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f2427e00640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243015e180 0x7f243014c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242ea00640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414003a50 0x7f2430159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 -- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243015e180 msgr2=0x7f243014c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243015e180 0x7f243014c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 -- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414003a50 msgr2=0x7f2430159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414003a50 0x7f2430159940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 -- 172.21.15.6:0/1007134949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24301f7710 con 0x7f243015e5e0 2024-03-06T07:34:44.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.822+0000 7f242f400640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 0x7f243014c7d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f243006e6f0 tx=0x7f242800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.823+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428003990 con 0x7f243015e5e0 2024-03-06T07:34:44.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.823+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242801c070 con 0x7f243015e5e0 2024-03-06T07:34:44.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.823+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428003d50 con 0x7f243015e5e0 2024-03-06T07:34:44.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.823+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24301f79f0 con 0x7f243015e5e0 2024-03-06T07:34:44.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.823+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24301f7fb0 con 0x7f243015e5e0 2024-03-06T07:34:44.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.824+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2430069510 con 0x7f243015e5e0 2024-03-06T07:34:44.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.826+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2428004030 con 0x7f243015e5e0 2024-03-06T07:34:44.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.827+0000 7f2426a00640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f240c06b4d0 0x7f240c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:44.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.827+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f242808ac40 con 0x7f243015e5e0 2024-03-06T07:34:44.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.828+0000 7f242ea00640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f240c06b4d0 0x7f240c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:44.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.829+0000 7f242ea00640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f240c06b4d0 0x7f240c06d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2420034000 tx=0x7f2420007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:44.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:44.835+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2428010040 con 0x7f243015e5e0 2024-03-06T07:34:45.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.110+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f243006dff0 con 0x7f243015e5e0 2024-03-06T07:34:45.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.111+0000 7f2426a00640 1 -- 172.21.15.6:0/1007134949 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f243006dff0 con 0x7f243015e5e0 2024-03-06T07:34:45.113 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:45.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:45.113 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f240c06b4d0 msgr2=0x7f240c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f240c06b4d0 0x7f240c06d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2420034000 tx=0x7f2420007000 comp rx=0 tx=0).stop 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 msgr2=0x7f243014c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 0x7f243014c7d0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f243006e6f0 tx=0x7f242800f580 comp rx=0 tx=0).stop 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 shutdown_connections 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f240c06b4d0 0x7f240c06d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f243015e5e0 0x7f243014c7d0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f243015e180 0x7f243014c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.113+0000 7f2435600640 1 --2- 172.21.15.6:0/1007134949 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2414003a50 0x7f2430159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.114+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 >> 172.21.15.6:0/1007134949 conn(0x7f243006afb0 msgr2=0x7f24300cc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.114+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 shutdown_connections 2024-03-06T07:34:45.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.114+0000 7f2435600640 1 -- 172.21.15.6:0/1007134949 wait complete. 2024-03-06T07:34:45.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:34:45.379 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:34:45.379 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:34:45.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:45.379 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:34:45.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.643+0000 7f1309600640 1 Processor -- start 2024-03-06T07:34:45.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.643+0000 7f1309600640 1 -- start start 2024-03-06T07:34:45.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1309600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1304144c00 0x7f1304145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1309600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f13040cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1309600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13040d0910 con 0x7f13041457c0 2024-03-06T07:34:45.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1309600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040d0bf0 con 0x7f1304144c00 2024-03-06T07:34:45.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1309600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040d0ed0 con 0x7f130406b6c0 2024-03-06T07:34:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f12fbe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f13041457c0 0x7f130406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55886/0 (socket says 172.21.15.6:55886) 2024-03-06T07:34:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f12fbe00640 1 -- 172.21.15.6:0/1521782463 learned_addr learned my addr 172.21.15.6:0/1521782463 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1303e00640 1 --2- 172.21.15.6:0/1521782463 >> v2:172.21.15.178:3300/0 conn(0x7f1304144c00 0x7f1304145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.645+0000 7f1308c00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f13040cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383405039 0 0) 0x7f13040d0910 con 0x7f13041457c0 2024-03-06T07:34:45.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e8003520 con 0x7f13041457c0 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1467803217 0 0) 0x7f12e8003520 con 0x7f13041457c0 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 msgr2=0x7f13040cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f13040cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 >> v2:172.21.15.178:3300/0 conn(0x7f1304144c00 msgr2=0x7f1304145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 --2- 172.21.15.6:0/1521782463 >> v2:172.21.15.178:3300/0 conn(0x7f1304144c00 0x7f1304145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040d11b0 con 0x7f13041457c0 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f12ec001670 con 0x7f13041457c0 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1308c00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f13040cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 >> v1:172.21.15.6:6789/0 conn(0x7f13041457c0 legacy=0x7f130406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f12e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.646+0000 7f1303400640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040d0bf0 con 0x7f12e8004d40 2024-03-06T07:34:45.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040d0ed0 con 0x7f12e80039b0 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e800b9c0 con 0x7f130406b6c0 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1308c00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f12fbe00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f12e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1303e00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1308c00640 1 -- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 msgr2=0x7f12e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1308c00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f12e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1308c00640 1 -- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12e80039b0 msgr2=0x7f12e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1308c00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1308c00640 1 -- 172.21.15.6:0/1521782463 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e8003520 con 0x7f12e8004d40 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f12fbe00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f130406b6c0 0x7f12e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:45.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.647+0000 7f1303e00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:45.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1308c00640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00e350 tx=0x7f12e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1360a4ef0e564f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc039070 con 0x7f12e8004d40 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc02cd70 con 0x7f12e8004d40 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1303400640 1 -- 172.21.15.6:0/1521782463 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12fc009de0 con 0x7f12e8004d40 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1309600640 1 -- 172.21.15.6:0/1521782463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 msgr2=0x7f12e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1309600640 1 --2- 172.21.15.6:0/1521782463 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00e350 tx=0x7f12e8005830 comp rx=0 tx=0).stop 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.649+0000 7f1308c00640 1 -- 172.21.15.6:0/1521782463 reap_dead start 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.650+0000 7f1309600640 1 -- 172.21.15.6:0/1521782463 shutdown_connections 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.650+0000 7f1309600640 1 -- 172.21.15.6:0/1521782463 >> 172.21.15.6:0/1521782463 conn(0x7f130406c0c0 msgr2=0x7f130406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.650+0000 7f1309600640 1 -- 172.21.15.6:0/1521782463 shutdown_connections 2024-03-06T07:34:45.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.650+0000 7f1309600640 1 -- 172.21.15.6:0/1521782463 wait complete. 2024-03-06T07:34:45.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.652+0000 7f1309600640 1 Processor -- start 2024-03-06T07:34:45.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.652+0000 7f1309600640 1 -- start start 2024-03-06T07:34:45.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1309600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f1304143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1309600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304141ff0 0x7f130415d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1309600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13041402b0 0x7f13040b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1309600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040d19b0 con 0x7f12e8004d40 2024-03-06T07:34:45.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1309600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1304144fe0 con 0x7f1304141ff0 2024-03-06T07:34:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1309600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130406f820 con 0x7f13041402b0 2024-03-06T07:34:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1303e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f1304143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1303e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f1304143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59726/0 (socket says 172.21.15.6:59726) 2024-03-06T07:34:45.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.653+0000 7f1303e00640 1 -- 172.21.15.6:0/2097294769 learned_addr learned my addr 172.21.15.6:0/2097294769 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f12fbe00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304141ff0 0x7f130415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f1308c00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13041402b0 0x7f13040b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f1303e00640 1 -- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13041402b0 msgr2=0x7f13040b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f1303e00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13041402b0 0x7f13040b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f1303e00640 1 -- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304141ff0 msgr2=0x7f130415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f1303e00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304141ff0 0x7f130415d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f1303e00640 1 -- 172.21.15.6:0/2097294769 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304142480 con 0x7f12e8004d40 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.654+0000 7f12fbe00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304141ff0 0x7f130415d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.655+0000 7f1303e00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f1304143520 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f12f4011600 tx=0x7f12f4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.655+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f400f920 con 0x7f12e8004d40 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.655+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f400fac0 con 0x7f12e8004d40 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.655+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1304142760 con 0x7f12e8004d40 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.655+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f400fe90 con 0x7f12e8004d40 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.655+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1304142d20 con 0x7f12e8004d40 2024-03-06T07:34:45.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.656+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130404f330 con 0x7f12e8004d40 2024-03-06T07:34:45.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.657+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12f4015460 con 0x7f12e8004d40 2024-03-06T07:34:45.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.658+0000 7f1300c00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12e006b6a0 0x7f12e006db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:45.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.658+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f12f408abc0 con 0x7f12e8004d40 2024-03-06T07:34:45.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.659+0000 7f12fbe00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12e006b6a0 0x7f12e006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:45.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.660+0000 7f12fbe00640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12e006b6a0 0x7f12e006db60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f13040b5e40 tx=0x7f12ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:45.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.667+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12f4019030 con 0x7f12e8004d40 2024-03-06T07:34:45.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.913+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f130415ee00 con 0x7f12e8004d40 2024-03-06T07:34:45.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.914+0000 7f1300c00640 1 -- 172.21.15.6:0/2097294769 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f12f4059910 con 0x7f12e8004d40 2024-03-06T07:34:45.916 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:45.917 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:45.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12e006b6a0 msgr2=0x7f12e006db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12e006b6a0 0x7f12e006db60 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f13040b5e40 tx=0x7f12ec009040 comp rx=0 tx=0).stop 2024-03-06T07:34:45.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 msgr2=0x7f1304143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:45.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f1304143520 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f12f4011600 tx=0x7f12f4011ac0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 shutdown_connections 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f12e006b6a0 0x7f12e006db60 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1304141ff0 0x7f130415d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f13041402b0 0x7f13040b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 --2- 172.21.15.6:0/2097294769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f12e8004d40 0x7f1304143520 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 >> 172.21.15.6:0/2097294769 conn(0x7f130406c0c0 msgr2=0x7f13040b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 shutdown_connections 2024-03-06T07:34:45.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:45.917+0000 7f1309600640 1 -- 172.21.15.6:0/2097294769 wait complete. 2024-03-06T07:34:45.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:34:46.173 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:34:46.174 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:34:46.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:46.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.381+0000 7f92cde00640 1 Processor -- start 2024-03-06T07:34:46.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.381+0000 7f92cde00640 1 -- start start 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92cde00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80af580 0x7f92c815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92cde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92c80cdc20 0x7f92c80b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92cde00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c815cc40 con 0x7f92c80af1a0 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92cde00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c815cf20 con 0x7f92c80cdc20 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92cde00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c815d200 con 0x7f92c80af580 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92c7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92c80cdc20 0x7f92c80b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92c7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f92c80cdc20 0x7f92c80b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59742/0 (socket says 172.21.15.6:59742) 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92c7400640 1 -- 172.21.15.6:0/1126911013 learned_addr learned my addr 172.21.15.6:0/1126911013 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.383+0000 7f92bfe00640 1 --2- 172.21.15.6:0/1126911013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80af580 0x7f92c815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.384+0000 7f92c7400640 1 -- 172.21.15.6:0/1126911013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80af580 msgr2=0x7f92c815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.384+0000 7f92c7400640 1 --2- 172.21.15.6:0/1126911013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80af580 0x7f92c815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.384+0000 7f92c7400640 1 -- 172.21.15.6:0/1126911013 >> v1:172.21.15.178:6789/0 conn(0x7f92c80af1a0 legacy=0x7f92c80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:46.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.384+0000 7f92c7400640 1 -- 172.21.15.6:0/1126911013 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c815d4e0 con 0x7f92c80cdc20 2024-03-06T07:34:46.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.385+0000 7f92c7400640 1 --2- 172.21.15.6:0/1126911013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80cdc20 0x7f92c80b2dd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009e50 tx=0x7f92c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffde8580d4d9746 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:46.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.385+0000 7f92c6000640 1 -- 172.21.15.6:0/1126911013 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c002c9e0 con 0x7f92c80cdc20 2024-03-06T07:34:46.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.385+0000 7f92c6000640 1 -- 172.21.15.6:0/1126911013 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c002cb80 con 0x7f92c80cdc20 2024-03-06T07:34:46.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.385+0000 7f92c6000640 1 -- 172.21.15.6:0/1126911013 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c00343f0 con 0x7f92c80cdc20 2024-03-06T07:34:46.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 -- 172.21.15.6:0/1126911013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80cdc20 msgr2=0x7f92c80b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 --2- 172.21.15.6:0/1126911013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80cdc20 0x7f92c80b2dd0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f92c0009e50 tx=0x7f92c002be40 comp rx=0 tx=0).stop 2024-03-06T07:34:46.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 -- 172.21.15.6:0/1126911013 shutdown_connections 2024-03-06T07:34:46.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 --2- 172.21.15.6:0/1126911013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80cdc20 0x7f92c80b2dd0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 --2- 172.21.15.6:0/1126911013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80af580 0x7f92c815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 -- 172.21.15.6:0/1126911013 >> 172.21.15.6:0/1126911013 conn(0x7f92c806afb0 msgr2=0x7f92c806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:46.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 -- 172.21.15.6:0/1126911013 shutdown_connections 2024-03-06T07:34:46.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.386+0000 7f92cde00640 1 -- 172.21.15.6:0/1126911013 wait complete. 2024-03-06T07:34:46.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.388+0000 7f92cde00640 1 Processor -- start 2024-03-06T07:34:46.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.388+0000 7f92cde00640 1 -- start start 2024-03-06T07:34:46.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92cde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 0x7f92c809bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92cde00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f92c80af580 0x7f92c809ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92cde00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80cdc20 0x7f92c806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92cde00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c80b36a0 con 0x7f92c80af1a0 2024-03-06T07:34:46.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92cde00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c80b3520 con 0x7f92c80af580 2024-03-06T07:34:46.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92cde00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c80b3820 con 0x7f92c80cdc20 2024-03-06T07:34:46.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92c6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 0x7f92c809bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92c6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 0x7f92c809bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59752/0 (socket says 172.21.15.6:59752) 2024-03-06T07:34:46.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.389+0000 7f92c6a00640 1 -- 172.21.15.6:0/3750833155 learned_addr learned my addr 172.21.15.6:0/3750833155 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92bfe00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f92c80af580 0x7f92c809ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92c6a00640 1 -- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80cdc20 msgr2=0x7f92c806cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92c6a00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80cdc20 0x7f92c806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92c6a00640 1 -- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f92c80af580 msgr2=0x7f92c809ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92c6a00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f92c80af580 0x7f92c809ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92c6a00640 1 -- 172.21.15.6:0/3750833155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c80ca680 con 0x7f92c80af1a0 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92c6a00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 0x7f92c809bbe0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c460 tx=0x7f92b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8016020 con 0x7f92c80af1a0 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b8012870 con 0x7f92c80af1a0 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b8012c60 con 0x7f92c80af1a0 2024-03-06T07:34:46.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c80ca960 con 0x7f92c80af1a0 2024-03-06T07:34:46.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.390+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c80caf20 con 0x7f92c80af1a0 2024-03-06T07:34:46.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.392+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8069510 con 0x7f92c80af1a0 2024-03-06T07:34:46.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.393+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92b8019020 con 0x7f92c80af1a0 2024-03-06T07:34:46.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.393+0000 7f92bf400640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92a406b4d0 0x7f92a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:46.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.394+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f92b8089ff0 con 0x7f92c80af1a0 2024-03-06T07:34:46.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.394+0000 7f92bfe00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92a406b4d0 0x7f92a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:46.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.395+0000 7f92bfe00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92a406b4d0 0x7f92a406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f92c80cf6d0 tx=0x7f92b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:46.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.398+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f92b8020030 con 0x7f92c80af1a0 2024-03-06T07:34:46.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.642+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92c804f3a0 con 0x7f92c80af1a0 2024-03-06T07:34:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.643+0000 7f92bf400640 1 -- 172.21.15.6:0/3750833155 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f92b8058da0 con 0x7f92c80af1a0 2024-03-06T07:34:46.645 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:46.645 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:46.645 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92a406b4d0 msgr2=0x7f92a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92a406b4d0 0x7f92a406d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f92c80cf6d0 tx=0x7f92b0008040 comp rx=0 tx=0).stop 2024-03-06T07:34:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 msgr2=0x7f92c809bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 0x7f92c809bbe0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c460 tx=0x7f92b800c920 comp rx=0 tx=0).stop 2024-03-06T07:34:46.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 shutdown_connections 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f92a406b4d0 0x7f92a406d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f92c80af580 0x7f92c809ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f92c80cdc20 0x7f92c806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 --2- 172.21.15.6:0/3750833155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f92c80af1a0 0x7f92c809bbe0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 >> 172.21.15.6:0/3750833155 conn(0x7f92c806afb0 msgr2=0x7f92c80cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 shutdown_connections 2024-03-06T07:34:46.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:46.646+0000 7f92cde00640 1 -- 172.21.15.6:0/3750833155 wait complete. 2024-03-06T07:34:46.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:34:46.906 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:34:46.906 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:34:46.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:47.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.129+0000 7fd1b7200640 1 Processor -- start 2024-03-06T07:34:47.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.129+0000 7fd1b7200640 1 -- start start 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b8145300 0x7fd1b80b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1b8070f10 0x7fd1b80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd1b80cce90 0x7fd1b80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b80cd300 con 0x7fd1b8070f10 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b80714d0 con 0x7fd1b80cce90 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b80717b0 con 0x7fd1b8145300 2024-03-06T07:34:47.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1b8070f10 0x7fd1b80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b5400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1b8070f10 0x7fd1b80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59756/0 (socket says 172.21.15.6:59756) 2024-03-06T07:34:47.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b5400640 1 -- 172.21.15.6:0/2961406662 learned_addr learned my addr 172.21.15.6:0/2961406662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.131+0000 7fd1b5e00640 1 --2- 172.21.15.6:0/2961406662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b8145300 0x7fd1b80b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.132+0000 7fd1b6800640 1 --2- 172.21.15.6:0/2961406662 >> v2:172.21.15.178:3300/0 conn(0x7fd1b80cce90 0x7fd1b80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.132+0000 7fd1b5400640 1 -- 172.21.15.6:0/2961406662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b8145300 msgr2=0x7fd1b80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.132+0000 7fd1b5400640 1 --2- 172.21.15.6:0/2961406662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b8145300 0x7fd1b80b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.132+0000 7fd1b5400640 1 -- 172.21.15.6:0/2961406662 >> v2:172.21.15.178:3300/0 conn(0x7fd1b80cce90 msgr2=0x7fd1b80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.132+0000 7fd1b5400640 1 --2- 172.21.15.6:0/2961406662 >> v2:172.21.15.178:3300/0 conn(0x7fd1b80cce90 0x7fd1b80cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.132+0000 7fd1b5400640 1 -- 172.21.15.6:0/2961406662 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8071a90 con 0x7fd1b8070f10 2024-03-06T07:34:47.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.133+0000 7fd1b5400640 1 --2- 172.21.15.6:0/2961406662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b80cc950 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8072420 tx=0x7fd1a402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e85c67707665d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.133+0000 7fd1b4a00640 1 -- 172.21.15.6:0/2961406662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4035020 con 0x7fd1b8070f10 2024-03-06T07:34:47.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.133+0000 7fd1b4a00640 1 -- 172.21.15.6:0/2961406662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a402d040 con 0x7fd1b8070f10 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.133+0000 7fd1b4a00640 1 -- 172.21.15.6:0/2961406662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4031a80 con 0x7fd1b8070f10 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 -- 172.21.15.6:0/2961406662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 msgr2=0x7fd1b80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 --2- 172.21.15.6:0/2961406662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b80cc950 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8072420 tx=0x7fd1a402b900 comp rx=0 tx=0).stop 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 -- 172.21.15.6:0/2961406662 shutdown_connections 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 --2- 172.21.15.6:0/2961406662 >> v2:172.21.15.178:3300/0 conn(0x7fd1b80cce90 0x7fd1b80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 --2- 172.21.15.6:0/2961406662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b80cc950 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 --2- 172.21.15.6:0/2961406662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b8145300 0x7fd1b80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 -- 172.21.15.6:0/2961406662 >> 172.21.15.6:0/2961406662 conn(0x7fd1b806b3d0 msgr2=0x7fd1b806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 -- 172.21.15.6:0/2961406662 shutdown_connections 2024-03-06T07:34:47.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.134+0000 7fd1b7200640 1 -- 172.21.15.6:0/2961406662 wait complete. 2024-03-06T07:34:47.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.136+0000 7fd1b7200640 1 Processor -- start 2024-03-06T07:34:47.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.136+0000 7fd1b7200640 1 -- start start 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b81314c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b80cce90 0x7fd1b8130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b8145300 0x7fd1b8130e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b81456e0 con 0x7fd1b8070f10 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b80b4ee0 con 0x7fd1b8145300 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b80cde70 con 0x7fd1b80cce90 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b81314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b81314c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59766/0 (socket says 172.21.15.6:59766) 2024-03-06T07:34:47.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 -- 172.21.15.6:0/3710161518 learned_addr learned my addr 172.21.15.6:0/3710161518 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 -- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b80cce90 msgr2=0x7fd1b8130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b80cce90 0x7fd1b8130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 -- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b8145300 msgr2=0x7fd1b8130e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b8145300 0x7fd1b8130e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.137+0000 7fd1b5e00640 1 -- 172.21.15.6:0/3710161518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b80aa710 con 0x7fd1b8070f10 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.138+0000 7fd1b5e00640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b81314c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800d5f0 tx=0x7fd1a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.138+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80125f0 con 0x7fd1b8070f10 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.138+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a8019070 con 0x7fd1b8070f10 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.138+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80129d0 con 0x7fd1b8070f10 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.138+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b80aa9f0 con 0x7fd1b8070f10 2024-03-06T07:34:47.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.138+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b80aafb0 con 0x7fd1b8070f10 2024-03-06T07:34:47.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.139+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b804f330 con 0x7fd1b8070f10 2024-03-06T07:34:47.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.140+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1a8012b70 con 0x7fd1b8070f10 2024-03-06T07:34:47.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.142+0000 7fd1ae000640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd19806b4d0 0x7fd19806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.142+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd1a808a670 con 0x7fd1b8070f10 2024-03-06T07:34:47.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.143+0000 7fd1b5400640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd19806b4d0 0x7fd19806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.144+0000 7fd1b5400640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd19806b4d0 0x7fd19806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4036000 tx=0x7fd1a4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.146+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1a8059450 con 0x7fd1b8070f10 2024-03-06T07:34:47.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.385+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1b8074ab0 con 0x7fd1b8070f10 2024-03-06T07:34:47.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.386+0000 7fd1ae000640 1 -- 172.21.15.6:0/3710161518 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fd1a8058d50 con 0x7fd1b8070f10 2024-03-06T07:34:47.388 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:47.388 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:47.389 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.389+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd19806b4d0 msgr2=0x7fd19806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.389+0000 7fd1b7200640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd19806b4d0 0x7fd19806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4036000 tx=0x7fd1a4030000 comp rx=0 tx=0).stop 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.389+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 msgr2=0x7fd1b81314c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.389+0000 7fd1b7200640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b81314c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800d5f0 tx=0x7fd1a800dab0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 shutdown_connections 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd19806b4d0 0x7fd19806d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b8145300 0x7fd1b8130e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1b80cce90 0x7fd1b8130940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 --2- 172.21.15.6:0/3710161518 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1b8070f10 0x7fd1b81314c0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 >> 172.21.15.6:0/3710161518 conn(0x7fd1b806b3d0 msgr2=0x7fd1b815c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 shutdown_connections 2024-03-06T07:34:47.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.390+0000 7fd1b7200640 1 -- 172.21.15.6:0/3710161518 wait complete. 2024-03-06T07:34:47.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:34:47.646 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:34:47.646 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:47.647 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:34:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.753+0000 7f7c6d4f7f40 1 Processor -- start 2024-03-06T07:34:47.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.753+0000 7f7c6d4f7f40 1 -- start start 2024-03-06T07:34:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c6d4f7f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a21e0 0x55b4be8bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c6d4f7f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 0x55b4be8a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c6d4f7f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be8783f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c6d4f7f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4be8c0bb0 con 0x55b4be8a3a90 2024-03-06T07:34:47.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c6d4f7f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4be81dfd0 con 0x55b4be94c330 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c6d4f7f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4be87a620 con 0x55b4be8a21e0 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.754+0000 7f7c61c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 0x55b4be8a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 0x55b4be8a79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59788/0 (socket says 172.21.15.6:59788) 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 -- 172.21.15.6:0/1069407644 learned_addr learned my addr 172.21.15.6:0/1069407644 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c6a600640 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a21e0 0x55b4be8bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c6b000640 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be8783f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 -- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a21e0 msgr2=0x55b4be8bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a21e0 0x55b4be8bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 -- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 msgr2=0x55b4be8783f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be8783f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 -- 172.21.15.6:0/1069407644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4be946e90 con 0x55b4be8a3a90 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c6b000640 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be8783f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:47.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.755+0000 7f7c61c00640 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 0x55b4be8a79d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400b260 tx=0x7f7c5400b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e120af66cd3af40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.758+0000 7f7c69200640 1 -- 172.21.15.6:0/1069407644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400e5c0 con 0x55b4be8a3a90 2024-03-06T07:34:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.758+0000 7f7c69200640 1 -- 172.21.15.6:0/1069407644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c54006070 con 0x55b4be8a3a90 2024-03-06T07:34:47.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.758+0000 7f7c69200640 1 -- 172.21.15.6:0/1069407644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5400e9c0 con 0x55b4be8a3a90 2024-03-06T07:34:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/1069407644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 msgr2=0x55b4be8a79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 0x55b4be8a79d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400b260 tx=0x7f7c5400b720 comp rx=0 tx=0).stop 2024-03-06T07:34:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/1069407644 shutdown_connections 2024-03-06T07:34:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be8783f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a3a90 0x55b4be8a79d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/1069407644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a21e0 0x55b4be8bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/1069407644 >> 172.21.15.6:0/1069407644 conn(0x55b4be8a2670 msgr2=0x55b4be8a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/1069407644 shutdown_connections 2024-03-06T07:34:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.759+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/1069407644 wait complete. 2024-03-06T07:34:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.760+0000 7f7c6d4f7f40 1 Processor -- start 2024-03-06T07:34:47.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.760+0000 7f7c6d4f7f40 1 -- start start 2024-03-06T07:34:47.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6d4f7f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a21e0 0x55b4be8a6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6d4f7f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a3a90 0x55b4be93c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6d4f7f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be93fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6d4f7f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4be93a8a0 con 0x55b4be8a21e0 2024-03-06T07:34:47.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6d4f7f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4be8bea60 con 0x55b4be94c330 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6d4f7f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4be94b8c0 con 0x55b4be8a3a90 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a21e0 0x55b4be8a6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a21e0 0x55b4be8a6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59790/0 (socket says 172.21.15.6:59790) 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 -- 172.21.15.6:0/2294310459 learned_addr learned my addr 172.21.15.6:0/2294310459 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 -- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a3a90 msgr2=0x55b4be93c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b4be8a3a90 0x55b4be93c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 -- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 msgr2=0x55b4be93fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b4be94c330 0x55b4be93fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.762+0000 7f7c6a600640 1 -- 172.21.15.6:0/2294310459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4be9426f0 con 0x55b4be8a21e0 2024-03-06T07:34:47.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.763+0000 7f7c6a600640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a21e0 0x55b4be8a6900 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c007d80 tx=0x7f7c5c006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.764+0000 7f7c63400640 1 -- 172.21.15.6:0/2294310459 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c017070 con 0x55b4be8a21e0 2024-03-06T07:34:47.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.764+0000 7f7c63400640 1 -- 172.21.15.6:0/2294310459 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c012600 con 0x55b4be8a21e0 2024-03-06T07:34:47.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.764+0000 7f7c63400640 1 -- 172.21.15.6:0/2294310459 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c012a10 con 0x55b4be8a21e0 2024-03-06T07:34:47.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.764+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b4be93e460 con 0x55b4be8a21e0 2024-03-06T07:34:47.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.764+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4be93bf10 con 0x55b4be8a21e0 2024-03-06T07:34:47.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.766+0000 7f7c63400640 1 -- 172.21.15.6:0/2294310459 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c5c01a070 con 0x55b4be8a21e0 2024-03-06T07:34:47.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.767+0000 7f7c63400640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7c4806e2c0 0x7f7c48070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.767+0000 7f7c63400640 1 -- 172.21.15.6:0/2294310459 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7c5c088560 con 0x55b4be8a21e0 2024-03-06T07:34:47.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.768+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55b4be8a6e40 0x55b4be940050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:47.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.768+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b4be93c970 con 0x55b4be8a6e40 2024-03-06T07:34:47.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.768+0000 7f7c61c00640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7c4806e2c0 0x7f7c48070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.768+0000 7f7c6b000640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55b4be8a6e40 0x55b4be940050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:47.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6b000640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55b4be8a6e40 0x55b4be940050 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6b000640 1 -- 172.21.15.6:0/2294310459 <== osd.2 v2:172.21.15.6:6822/1963740543 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f7c64052640 con 0x55b4be8a6e40 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55b4be8a6e40 msgr2=0x55b4be940050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55b4be8a6e40 0x55b4be940050 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c61c00640 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7c4806e2c0 0x7f7c48070780 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55b4be8a6660 tx=0x7f7c54006000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7c4806e2c0 msgr2=0x7f7c48070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7c4806e2c0 0x7f7c48070780 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x55b4be8a6660 tx=0x7f7c54006000 comp rx=0 tx=0).stop 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a21e0 msgr2=0x55b4be8a6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:47.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 --2- 172.21.15.6:0/2294310459 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b4be8a21e0 0x55b4be8a6900 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c007d80 tx=0x7f7c5c006750 comp rx=0 tx=0).stop 2024-03-06T07:34:47.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6b000640 1 -- 172.21.15.6:0/2294310459 reap_dead start 2024-03-06T07:34:47.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 shutdown_connections 2024-03-06T07:34:47.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.769+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 >> 172.21.15.6:0/2294310459 conn(0x55b4be8a2670 msgr2=0x55b4be8c0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:47.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.770+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 shutdown_connections 2024-03-06T07:34:47.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:47.770+0000 7f7c6d4f7f40 1 -- 172.21.15.6:0/2294310459 wait complete. 2024-03-06T07:34:47.776 INFO:teuthology.orchestra.run.smithi006.stdout:W�file_cdir_2! 2024-03-06T07:34:47.776 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:34:47.862 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:34:47.862 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-06T07:34:47.862 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-06T07:34:47.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:34:47.862 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1099511628278, 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "file_c", 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 17 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1, 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "dir_2", 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 33 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:34:47.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:47.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:47.868 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:34:47.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:47.921 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:34:47.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:48.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.192+0000 7f3b02a00640 1 Processor -- start 2024-03-06T07:34:48.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.193+0000 7f3b02a00640 1 -- start start 2024-03-06T07:34:48.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.194+0000 7f3b02a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3afc15e070 0x7f3afc0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.194+0000 7f3b02a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3afc0d0cd0 0x7f3afc0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.194+0000 7f3b02a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3afc0d18c0 con 0x7f3afc0cf410 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.194+0000 7f3b02a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3afc0d1ba0 con 0x7f3afc0d0cd0 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.194+0000 7f3b02a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3afc0d1e80 con 0x7f3afc15e070 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.194+0000 7f3b02000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3afc0d0cd0 0x7f3afc0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b02000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3afc0d0cd0 0x7f3afc0cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59806/0 (socket says 172.21.15.6:59806) 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b02000640 1 -- 172.21.15.6:0/1728226107 learned_addr learned my addr 172.21.15.6:0/1728226107 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b01600640 1 --2- 172.21.15.6:0/1728226107 >> v2:172.21.15.178:3300/0 conn(0x7f3afc15e070 0x7f3afc0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b02000640 1 -- 172.21.15.6:0/1728226107 >> v2:172.21.15.178:3300/0 conn(0x7f3afc15e070 msgr2=0x7f3afc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b02000640 1 --2- 172.21.15.6:0/1728226107 >> v2:172.21.15.178:3300/0 conn(0x7f3afc15e070 0x7f3afc0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b02000640 1 -- 172.21.15.6:0/1728226107 >> v1:172.21.15.178:6789/0 conn(0x7f3afc0cf410 legacy=0x7f3afc0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.195+0000 7f3b02000640 1 -- 172.21.15.6:0/1728226107 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0d2160 con 0x7f3afc0d0cd0 2024-03-06T07:34:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.196+0000 7f3b02000640 1 --2- 172.21.15.6:0/1728226107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0d0cd0 0x7f3afc0cf870 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3afc08d270 tx=0x7f3af402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184912076959673b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.196+0000 7f3afbe00640 1 -- 172.21.15.6:0/1728226107 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af402ca80 con 0x7f3afc0d0cd0 2024-03-06T07:34:48.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3afbe00640 1 -- 172.21.15.6:0/1728226107 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af402cc20 con 0x7f3afc0d0cd0 2024-03-06T07:34:48.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3afbe00640 1 -- 172.21.15.6:0/1728226107 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af40343f0 con 0x7f3afc0d0cd0 2024-03-06T07:34:48.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 -- 172.21.15.6:0/1728226107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0d0cd0 msgr2=0x7f3afc0cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1728226107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0d0cd0 0x7f3afc0cf870 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f3afc08d270 tx=0x7f3af402bee0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 -- 172.21.15.6:0/1728226107 shutdown_connections 2024-03-06T07:34:48.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1728226107 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0d0cd0 0x7f3afc0cf870 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1728226107 >> v2:172.21.15.178:3300/0 conn(0x7f3afc15e070 0x7f3afc0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 -- 172.21.15.6:0/1728226107 >> 172.21.15.6:0/1728226107 conn(0x7f3afc08d2a0 msgr2=0x7f3afc0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:48.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 -- 172.21.15.6:0/1728226107 shutdown_connections 2024-03-06T07:34:48.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.197+0000 7f3b02a00640 1 -- 172.21.15.6:0/1728226107 wait complete. 2024-03-06T07:34:48.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.199+0000 7f3b02a00640 1 Processor -- start 2024-03-06T07:34:48.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.199+0000 7f3b02a00640 1 -- start start 2024-03-06T07:34:48.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b02a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 0x7f3afc0c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b02a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3afc0d0cd0 0x7f3afc0c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b02a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3afc15e070 0x7f3afc160ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b02a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc0d4730 con 0x7f3afc0cf410 2024-03-06T07:34:48.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b02a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc0d45b0 con 0x7f3afc15e070 2024-03-06T07:34:48.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b02a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc0b3630 con 0x7f3afc0d0cd0 2024-03-06T07:34:48.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b01600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 0x7f3afc0c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b01600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 0x7f3afc0c8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59822/0 (socket says 172.21.15.6:59822) 2024-03-06T07:34:48.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.200+0000 7f3b01600640 1 -- 172.21.15.6:0/1347709977 learned_addr learned my addr 172.21.15.6:0/1347709977 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b00c00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3afc0d0cd0 0x7f3afc0c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b02000640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3afc15e070 0x7f3afc160ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b01600640 1 -- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3afc0d0cd0 msgr2=0x7f3afc0c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b01600640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3afc0d0cd0 0x7f3afc0c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b01600640 1 -- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3afc15e070 msgr2=0x7f3afc160ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b01600640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3afc15e070 0x7f3afc160ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b01600640 1 -- 172.21.15.6:0/1347709977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0c7980 con 0x7f3afc0cf410 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b00c00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3afc0d0cd0 0x7f3afc0c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:48.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3b01600640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 0x7f3afc0c8700 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3af000d620 tx=0x7f3af000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:48.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.201+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af00125f0 con 0x7f3afc0cf410 2024-03-06T07:34:48.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.202+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af0019070 con 0x7f3afc0cf410 2024-03-06T07:34:48.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.202+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af00129d0 con 0x7f3afc0cf410 2024-03-06T07:34:48.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.202+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc0ca570 con 0x7f3afc0cf410 2024-03-06T07:34:48.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.202+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc0c7f00 con 0x7f3afc0cf410 2024-03-06T07:34:48.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.203+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc04f330 con 0x7f3afc0cf410 2024-03-06T07:34:48.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.204+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3af0012b70 con 0x7f3afc0cf410 2024-03-06T07:34:48.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.205+0000 7f3af9600640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3adc06b4d0 0x7f3adc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:48.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.205+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3af008a650 con 0x7f3afc0cf410 2024-03-06T07:34:48.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.205+0000 7f3b00c00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3adc06b4d0 0x7f3adc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:48.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.207+0000 7f3b00c00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3adc06b4d0 0x7f3adc06d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8007fd0 tx=0x7f3ae8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.212+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3af0059440 con 0x7f3afc0cf410 2024-03-06T07:34:48.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.456+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3afc0d1370 con 0x7f3afc0cf410 2024-03-06T07:34:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.457+0000 7f3af9600640 1 -- 172.21.15.6:0/1347709977 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f3af0058d40 con 0x7f3afc0cf410 2024-03-06T07:34:48.460 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:48.460 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3adc06b4d0 msgr2=0x7f3adc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3adc06b4d0 0x7f3adc06d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8007fd0 tx=0x7f3ae8009040 comp rx=0 tx=0).stop 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 msgr2=0x7f3afc0c8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 0x7f3afc0c8700 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f3af000d620 tx=0x7f3af000dae0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 shutdown_connections 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3adc06b4d0 0x7f3adc06d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3afc15e070 0x7f3afc160ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3afc0d0cd0 0x7f3afc0c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 --2- 172.21.15.6:0/1347709977 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3afc0cf410 0x7f3afc0c8700 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.460+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 >> 172.21.15.6:0/1347709977 conn(0x7f3afc08d2a0 msgr2=0x7f3afc06cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.461+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 shutdown_connections 2024-03-06T07:34:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:48.461+0000 7f3b02a00640 1 -- 172.21.15.6:0/1347709977 wait complete. 2024-03-06T07:34:48.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:34:48.714 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:34:48.714 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:34:48.714 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:34:53.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:53.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:34:53.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.934+0000 7efe02c00640 1 Processor -- start 2024-03-06T07:34:53.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.935+0000 7efe02c00640 1 -- start start 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.936+0000 7efe02c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.936+0000 7efe02c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7efdfc0b5060 0x7efdfc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.936+0000 7efe02c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdfc0b54a0 0x7efdfc0d2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.936+0000 7efe02c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdfc0cf900 con 0x7efdfc0b54a0 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.936+0000 7efe02c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdfc0cea50 con 0x7efdfc0b5060 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.936+0000 7efe02c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0cecf0 con 0x7efdfc161b40 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdfc0b54a0 0x7efdfc0d2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efdfc0b54a0 0x7efdfc0d2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60714/0 (socket says 172.21.15.6:60714) 2024-03-06T07:34:53.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 -- 172.21.15.6:0/2517891663 learned_addr learned my addr 172.21.15.6:0/2517891663 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efdfb400640 1 --2- 172.21.15.6:0/2517891663 >> v2:172.21.15.178:3300/0 conn(0x7efdfc0b5060 0x7efdfc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efdfbe00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 -- 172.21.15.6:0/2517891663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 msgr2=0x7efdfc0cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 -- 172.21.15.6:0/2517891663 >> v2:172.21.15.178:3300/0 conn(0x7efdfc0b5060 msgr2=0x7efdfc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 --2- 172.21.15.6:0/2517891663 >> v2:172.21.15.178:3300/0 conn(0x7efdfc0b5060 0x7efdfc0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efe00c00640 1 -- 172.21.15.6:0/2517891663 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc0cefd0 con 0x7efdfc0b54a0 2024-03-06T07:34:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.937+0000 7efdfbe00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.939+0000 7efe00c00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0d2760 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7efdfc06ba70 tx=0x7efdf402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4903dce015e495 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.939+0000 7efdfaa00640 1 -- 172.21.15.6:0/2517891663 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf402ca70 con 0x7efdfc0b54a0 2024-03-06T07:34:53.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.939+0000 7efdfaa00640 1 -- 172.21.15.6:0/2517891663 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf402cc10 con 0x7efdfc0b54a0 2024-03-06T07:34:53.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.939+0000 7efdfaa00640 1 -- 172.21.15.6:0/2517891663 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf40343f0 con 0x7efdfc0b54a0 2024-03-06T07:34:53.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 -- 172.21.15.6:0/2517891663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b54a0 msgr2=0x7efdfc0d2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0d2760 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7efdfc06ba70 tx=0x7efdf402bed0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 -- 172.21.15.6:0/2517891663 shutdown_connections 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0d2760 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 --2- 172.21.15.6:0/2517891663 >> v2:172.21.15.178:3300/0 conn(0x7efdfc0b5060 0x7efdfc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 --2- 172.21.15.6:0/2517891663 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 -- 172.21.15.6:0/2517891663 >> 172.21.15.6:0/2517891663 conn(0x7efdfc09d220 msgr2=0x7efdfc0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 -- 172.21.15.6:0/2517891663 shutdown_connections 2024-03-06T07:34:53.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.940+0000 7efe02c00640 1 -- 172.21.15.6:0/2517891663 wait complete. 2024-03-06T07:34:53.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.942+0000 7efe02c00640 1 Processor -- start 2024-03-06T07:34:53.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.942+0000 7efe02c00640 1 -- start start 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe02c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 0x7efdfc15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe02c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe02c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe02c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0b2ff0 con 0x7efdfc0b5060 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe02c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0cf590 con 0x7efdfc0b54a0 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe02c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0b3170 con 0x7efdfc161b40 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 0x7efdfc15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 0x7efdfc15b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60718/0 (socket says 172.21.15.6:60718) 2024-03-06T07:34:53.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 -- 172.21.15.6:0/2016966443 learned_addr learned my addr 172.21.15.6:0/2016966443 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfb400640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 -- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 msgr2=0x7efdfc0b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efe00c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0b6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 -- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc0b54a0 msgr2=0x7efdfc0b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.944+0000 7efdfbe00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efdfbe00640 1 -- 172.21.15.6:0/2016966443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc0b7810 con 0x7efdfc0b5060 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efdfb400640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efe00c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:53.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efdfbe00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 0x7efdfc15b600 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7efdec00c7f0 tx=0x7efdec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec015070 con 0x7efdfc0b5060 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec007820 con 0x7efdfc0b5060 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec007bd0 con 0x7efdfc0b5060 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc1f8360 con 0x7efdfc0b5060 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.945+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc1f75a0 con 0x7efdfc0b5060 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.946+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc04f3a0 con 0x7efdfc0b5060 2024-03-06T07:34:53.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.947+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdec018070 con 0x7efdfc0b5060 2024-03-06T07:34:53.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.948+0000 7efdf3e00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efddc06b520 0x7efddc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:53.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.948+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7efdec08a320 con 0x7efdfc0b5060 2024-03-06T07:34:53.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.949+0000 7efdfb400640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efddc06b520 0x7efddc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:53.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.950+0000 7efdfb400640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efddc06b520 0x7efddc06d9e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efde8006fd0 tx=0x7efde8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:53.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:53.953+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdfc0b7810 con 0x7efdfc0b5060 2024-03-06T07:34:54.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.191+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdfc06c6e0 con 0x7efdfc0b5060 2024-03-06T07:34:54.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.192+0000 7efdf3e00640 1 -- 172.21.15.6:0/2016966443 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7efdec059180 con 0x7efdfc0b5060 2024-03-06T07:34:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:54.194 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:54.194 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:54.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efddc06b520 msgr2=0x7efddc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efddc06b520 0x7efddc06d9e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efde8006fd0 tx=0x7efde8008040 comp rx=0 tx=0).stop 2024-03-06T07:34:54.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 msgr2=0x7efdfc15b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 0x7efdfc15b600 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7efdec00c7f0 tx=0x7efdec00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 shutdown_connections 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efddc06b520 0x7efddc06d9e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efdfc0b54a0 0x7efdfc0b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efdfc161b40 0x7efdfc0b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 --2- 172.21.15.6:0/2016966443 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efdfc0b5060 0x7efdfc15b600 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 >> 172.21.15.6:0/2016966443 conn(0x7efdfc09d220 msgr2=0x7efdfc06cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 shutdown_connections 2024-03-06T07:34:54.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.194+0000 7efe02c00640 1 -- 172.21.15.6:0/2016966443 wait complete. 2024-03-06T07:34:54.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:34:54.441 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:34:54.441 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:34:54.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.665+0000 7fbea0e00640 1 Processor -- start 2024-03-06T07:34:54.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.665+0000 7fbea0e00640 1 -- start start 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.666+0000 7fbea0e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe9c1552b0 0x7fbe9c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbea0e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c15ed10 con 0x7fbe9c0cd060 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbea0e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c15eff0 con 0x7fbe9c15e810 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbea0e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c15f2d0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbe9b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbe9c1552b0 0x7fbe9c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbe9be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbe9c0cd060 0x7fbe9c0b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43978/0 (socket says 172.21.15.6:43978) 2024-03-06T07:34:54.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbe9be00640 1 -- 172.21.15.6:0/1629920251 learned_addr learned my addr 172.21.15.6:0/1629920251 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.667+0000 7fbe9b400640 1 -- 172.21.15.6:0/1629920251 >> v1:172.21.15.178:6789/0 conn(0x7fbe9c15e810 legacy=0x7fbe9c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.668+0000 7fbe9b400640 1 -- 172.21.15.6:0/1629920251 >> v1:172.21.15.6:6789/0 conn(0x7fbe9c0cd060 legacy=0x7fbe9c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.668+0000 7fbe9b400640 1 -- 172.21.15.6:0/1629920251 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c15f5b0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.668+0000 7fbe9a000640 1 -- 172.21.15.6:0/1629920251 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369260901 0 0) 0x7fbe9c15ed10 con 0x7fbe9c0cd060 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.669+0000 7fbe9b400640 1 --2- 172.21.15.6:0/1629920251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c15b1d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fbe840097a0 tx=0x7fbe8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807e272f3286afee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.669+0000 7fbe9a000640 1 -- 172.21.15.6:0/1629920251 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe840315a0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.669+0000 7fbe9a000640 1 -- 172.21.15.6:0/1629920251 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe84037070 con 0x7fbe9c1552b0 2024-03-06T07:34:54.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.669+0000 7fbe9a000640 1 -- 172.21.15.6:0/1629920251 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84031960 con 0x7fbe9c1552b0 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.669+0000 7fbea0e00640 1 -- 172.21.15.6:0/1629920251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 msgr2=0x7fbe9c15b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.669+0000 7fbea0e00640 1 --2- 172.21.15.6:0/1629920251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c15b1d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fbe840097a0 tx=0x7fbe8402b840 comp rx=0 tx=0).stop 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.670+0000 7fbea0e00640 1 -- 172.21.15.6:0/1629920251 shutdown_connections 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.670+0000 7fbea0e00640 1 --2- 172.21.15.6:0/1629920251 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c15b1d0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.670+0000 7fbea0e00640 1 -- 172.21.15.6:0/1629920251 >> 172.21.15.6:0/1629920251 conn(0x7fbe9c06b3d0 msgr2=0x7fbe9c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.670+0000 7fbea0e00640 1 -- 172.21.15.6:0/1629920251 shutdown_connections 2024-03-06T07:34:54.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.670+0000 7fbea0e00640 1 -- 172.21.15.6:0/1629920251 wait complete. 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.671+0000 7fbea0e00640 1 Processor -- start 2024-03-06T07:34:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.672+0000 7fbea0e00640 1 -- start start 2024-03-06T07:34:54.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbea0e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe9c0cd060 0x7fbe9c1509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbea0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbea0e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe9c15e810 0x7fbe9c1e58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbea0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c06dd30 con 0x7fbe9c1552b0 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbea0e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0b3760 con 0x7fbe9c0cd060 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbea0e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c06deb0 con 0x7fbe9c15e810 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c0ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60738/0 (socket says 172.21.15.6:60738) 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 -- 172.21.15.6:0/2404810544 learned_addr learned my addr 172.21.15.6:0/2404810544 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9b400640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe9c0cd060 0x7fbe9c1509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 -- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe9c15e810 msgr2=0x7fbe9c1e58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe9c15e810 0x7fbe9c1e58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 -- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe9c0cd060 msgr2=0x7fbe9c1509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe9c0cd060 0x7fbe9c1509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.673+0000 7fbe9aa00640 1 -- 172.21.15.6:0/2404810544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0af290 con 0x7fbe9c1552b0 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.674+0000 7fbe9aa00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c0ad880 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00ba50 tx=0x7fbe8c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.674+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c019070 con 0x7fbe9c1552b0 2024-03-06T07:34:54.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.674+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c00cc20 con 0x7fbe9c1552b0 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.674+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c014440 con 0x7fbe9c1552b0 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.674+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c0ae1f0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.674+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c0ae7b0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.675+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c04f330 con 0x7fbe9c1552b0 2024-03-06T07:34:54.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.676+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe8c01c070 con 0x7fbe9c1552b0 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.677+0000 7fbe93400640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe7806b4d0 0x7fbe7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:54.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.677+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbe8c089ec0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.678+0000 7fbe9b400640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe7806b4d0 0x7fbe7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:54.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.679+0000 7fbe9b400640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe7806b4d0 0x7fbe7806d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbe84036000 tx=0x7fbe8403d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:54.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.682+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe8c058650 con 0x7fbe9c1552b0 2024-03-06T07:34:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.922+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe9c06e410 con 0x7fbe9c1552b0 2024-03-06T07:34:54.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.923+0000 7fbe93400640 1 -- 172.21.15.6:0/2404810544 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fbe8c05c6f0 con 0x7fbe9c1552b0 2024-03-06T07:34:54.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:54.925 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:54.925 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe7806b4d0 msgr2=0x7fbe7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe7806b4d0 0x7fbe7806d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbe84036000 tx=0x7fbe8403d040 comp rx=0 tx=0).stop 2024-03-06T07:34:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 msgr2=0x7fbe9c0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c0ad880 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00ba50 tx=0x7fbe8c00bf10 comp rx=0 tx=0).stop 2024-03-06T07:34:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 shutdown_connections 2024-03-06T07:34:54.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe7806b4d0 0x7fbe7806d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbe9c15e810 0x7fbe9c1e58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbe9c1552b0 0x7fbe9c0ad880 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 --2- 172.21.15.6:0/2404810544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbe9c0cd060 0x7fbe9c1509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 >> 172.21.15.6:0/2404810544 conn(0x7fbe9c06b3d0 msgr2=0x7fbe9c070bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 shutdown_connections 2024-03-06T07:34:54.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:54.925+0000 7fbea0e00640 1 -- 172.21.15.6:0/2404810544 wait complete. 2024-03-06T07:34:54.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:34:55.171 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:34:55.171 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:34:55.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.379+0000 7fa8ce400640 1 Processor -- start 2024-03-06T07:34:55.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.379+0000 7fa8ce400640 1 -- start start 2024-03-06T07:34:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8ce400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa8c8070f10 0x7fa8c80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8ce400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80714d0 con 0x7fa8c8145300 2024-03-06T07:34:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8ce400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80717b0 con 0x7fa8c80cce90 2024-03-06T07:34:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8ce400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c80cd300 con 0x7fa8c8070f10 2024-03-06T07:34:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8cd000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa8c8145300 0x7fa8c80b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43982/0 (socket says 172.21.15.6:43982) 2024-03-06T07:34:55.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8cd000640 1 -- 172.21.15.6:0/278927486 learned_addr learned my addr 172.21.15.6:0/278927486 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:34:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964014168 0 0) 0x7fa8c80714d0 con 0x7fa8c8145300 2024-03-06T07:34:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0003520 con 0x7fa8c8145300 2024-03-06T07:34:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.381+0000 7fa8c7e00640 1 --2- 172.21.15.6:0/278927486 >> v2:172.21.15.178:3300/0 conn(0x7fa8c8070f10 0x7fa8c80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266744494 0 0) 0x7fa8b0003520 con 0x7fa8c8145300 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 >> v2:172.21.15.178:3300/0 conn(0x7fa8c8070f10 msgr2=0x7fa8c80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 --2- 172.21.15.6:0/278927486 >> v2:172.21.15.178:3300/0 conn(0x7fa8c8070f10 0x7fa8c80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 >> v1:172.21.15.178:6789/0 conn(0x7fa8c80cce90 legacy=0x7fa8c815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8071a90 con 0x7fa8c8145300 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fa8bc001670 con 0x7fa8c8145300 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 >> v1:172.21.15.6:6789/0 conn(0x7fa8c8145300 legacy=0x7fa8c80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0003a00 0x7fa8b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0004800 0x7fa8b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0005290 0x7fa8b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80cd300 con 0x7fa8b0003a00 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000c1e0 con 0x7fa8b0004800 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b000c3e0 con 0x7fa8b0005290 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.382+0000 7fa8cd000640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0003a00 0x7fa8b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cda00640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0005290 0x7fa8b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8c7e00640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0004800 0x7fa8b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cd000640 1 -- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0005290 msgr2=0x7fa8b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cd000640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0005290 0x7fa8b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cd000640 1 -- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0004800 msgr2=0x7fa8b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cd000640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8b0004800 0x7fa8b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cd000640 1 -- 172.21.15.6:0/278927486 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0003520 con 0x7fa8b0003a00 2024-03-06T07:34:55.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.383+0000 7fa8cda00640 1 -- 172.21.15.6:0/278927486 reap_dead start 2024-03-06T07:34:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.384+0000 7fa8cd000640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0003a00 0x7fa8b0003e00 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c190 tx=0x7fa8b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7854d91033d0c79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.384+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc007c60 con 0x7fa8b0003a00 2024-03-06T07:34:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.384+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc030070 con 0x7fa8b0003a00 2024-03-06T07:34:55.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.384+0000 7fa8c7400640 1 -- 172.21.15.6:0/278927486 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc005d50 con 0x7fa8b0003a00 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 -- 172.21.15.6:0/278927486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0003a00 msgr2=0x7fa8b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0003a00 0x7fa8b0003e00 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c190 tx=0x7fa8b0005830 comp rx=0 tx=0).stop 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 -- 172.21.15.6:0/278927486 shutdown_connections 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 --2- 172.21.15.6:0/278927486 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8b0003a00 0x7fa8b0003e00 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 -- 172.21.15.6:0/278927486 >> 172.21.15.6:0/278927486 conn(0x7fa8c806b3d0 msgr2=0x7fa8c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 -- 172.21.15.6:0/278927486 shutdown_connections 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.385+0000 7fa8ce400640 1 -- 172.21.15.6:0/278927486 wait complete. 2024-03-06T07:34:55.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.387+0000 7fa8ce400640 1 Processor -- start 2024-03-06T07:34:55.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.387+0000 7fa8ce400640 1 -- start start 2024-03-06T07:34:55.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8ce400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0003a00 0x7fa8c806e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8ce400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 0x7fa8c81428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8ce400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8c815b120 0x7fa8c8142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8ce400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80cc950 con 0x7fa8c815aca0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8ce400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c81456e0 con 0x7fa8c815b120 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8ce400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80ccad0 con 0x7fa8b0003a00 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 0x7fa8c81428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 0x7fa8c81428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60750/0 (socket says 172.21.15.6:60750) 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.388+0000 7fa8c7e00640 1 -- 172.21.15.6:0/1371905683 learned_addr learned my addr 172.21.15.6:0/1371905683 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8cd000640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0003a00 0x7fa8c806e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c7e00640 1 -- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0003a00 msgr2=0x7fa8c806e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c7e00640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0003a00 0x7fa8c806e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c7e00640 1 -- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8c815b120 msgr2=0x7fa8c8142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8cda00640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8c815b120 0x7fa8c8142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c7e00640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8c815b120 0x7fa8c8142e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c7e00640 1 -- 172.21.15.6:0/1371905683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c806e7c0 con 0x7fa8c815aca0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c7e00640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 0x7fa8c81428f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400f530 tx=0x7fa8b400f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4019020 con 0x7fa8c815aca0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4014870 con 0x7fa8c815aca0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c806cb30 con 0x7fa8c815aca0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.389+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4014c60 con 0x7fa8c815aca0 2024-03-06T07:34:55.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.390+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c806d0f0 con 0x7fa8c815aca0 2024-03-06T07:34:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.391+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c804f330 con 0x7fa8c815aca0 2024-03-06T07:34:55.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.393+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8b401c020 con 0x7fa8c815aca0 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.393+0000 7fa8c4c00640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:55.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.394+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa8b408a990 con 0x7fa8c815aca0 2024-03-06T07:34:55.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.394+0000 7fa8cd000640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:55.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.396+0000 7fa8cd000640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8a806b4d0 0x7fa8a806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc034000 tx=0x7fa8bc039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.399+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8b4023020 con 0x7fa8c815aca0 2024-03-06T07:34:55.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.636+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8c8074ab0 con 0x7fa8c815aca0 2024-03-06T07:34:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.637+0000 7fa8c4c00640 1 -- 172.21.15.6:0/1371905683 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fa8b408a070 con 0x7fa8c815aca0 2024-03-06T07:34:55.640 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:55.640 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:55.640 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:55.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8a806b4d0 msgr2=0x7fa8a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8a806b4d0 0x7fa8a806d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc034000 tx=0x7fa8bc039040 comp rx=0 tx=0).stop 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 msgr2=0x7fa8c81428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 0x7fa8c81428f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400f530 tx=0x7fa8b400f9f0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 shutdown_connections 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8c815b120 0x7fa8c8142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa8c815aca0 0x7fa8c81428f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 --2- 172.21.15.6:0/1371905683 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa8b0003a00 0x7fa8c806e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 >> 172.21.15.6:0/1371905683 conn(0x7fa8c806b3d0 msgr2=0x7fa8c80724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 shutdown_connections 2024-03-06T07:34:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:55.640+0000 7fa8ce400640 1 -- 172.21.15.6:0/1371905683 wait complete. 2024-03-06T07:34:55.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:34:55.911 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:34:55.911 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:34:55.911 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:34:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.005+0000 7f1129586f40 1 Processor -- start 2024-03-06T07:34:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.005+0000 7f1129586f40 1 -- start start 2024-03-06T07:34:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.006+0000 7f1129586f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 0x55e5b40df6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.006+0000 7f1129586f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40e2540 0x55e5b40dc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.006+0000 7f1129586f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b4181830 0x55e5b40dd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.006+0000 7f1129586f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b40f8d10 con 0x55e5b40b4660 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.006+0000 7f1129586f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b4058fd0 con 0x55e5b40e2540 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.006+0000 7f1129586f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5b40fba90 con 0x55e5b4181830 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 0x55e5b40df6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 0x55e5b40df6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60766/0 (socket says 172.21.15.6:60766) 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 -- 172.21.15.6:0/2457599576 learned_addr learned my addr 172.21.15.6:0/2457599576 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f111de00640 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40e2540 0x55e5b40dc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1127200640 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b4181830 0x55e5b40dd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 -- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b4181830 msgr2=0x55e5b40dd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b4181830 0x55e5b40dd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 -- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40e2540 msgr2=0x55e5b40dc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40e2540 0x55e5b40dc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1126800640 1 -- 172.21.15.6:0/2457599576 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5b40dfc10 con 0x55e5b40b4660 2024-03-06T07:34:56.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.007+0000 7f1127200640 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b4181830 0x55e5b40dd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:56.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1126800640 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 0x55e5b40df6d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1118001d10 tx=0x7f111800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d298f42ea52dc973 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1125400640 1 -- 172.21.15.6:0/2457599576 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1118001e60 con 0x55e5b40b4660 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1125400640 1 -- 172.21.15.6:0/2457599576 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111800bd20 con 0x55e5b40b4660 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1125400640 1 -- 172.21.15.6:0/2457599576 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111800f640 con 0x55e5b40b4660 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 -- 172.21.15.6:0/2457599576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 msgr2=0x55e5b40df6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 0x55e5b40df6d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f1118001d10 tx=0x7f111800b5b0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 -- 172.21.15.6:0/2457599576 shutdown_connections 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40e2540 0x55e5b40dc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b4181830 0x55e5b40dd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 --2- 172.21.15.6:0/2457599576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b40b4660 0x55e5b40df6d0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.008+0000 7f1129586f40 1 -- 172.21.15.6:0/2457599576 >> 172.21.15.6:0/2457599576 conn(0x55e5b40b25c0 msgr2=0x55e5b40dbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.009+0000 7f1129586f40 1 -- 172.21.15.6:0/2457599576 shutdown_connections 2024-03-06T07:34:56.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.009+0000 7f1129586f40 1 -- 172.21.15.6:0/2457599576 wait complete. 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.009+0000 7f1129586f40 1 Processor -- start 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.009+0000 7f1129586f40 1 -- start start 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.010+0000 7f1129586f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40b4660 0x55e5b4176780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.010+0000 7f1129586f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b40e2540 0x55e5b40e0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.010+0000 7f1129586f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b4181830 0x55e5b41823e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.010+0000 7f1129586f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b417fec0 con 0x55e5b4181830 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.010+0000 7f1129586f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b41781c0 con 0x55e5b40b4660 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.010+0000 7f1129586f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5b417e7a0 con 0x55e5b40e2540 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b4181830 0x55e5b41823e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b4181830 0x55e5b41823e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60780/0 (socket says 172.21.15.6:60780) 2024-03-06T07:34:56.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 -- 172.21.15.6:0/344928381 learned_addr learned my addr 172.21.15.6:0/344928381 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f111de00640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b40e2540 0x55e5b40e0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1126800640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40b4660 0x55e5b4176780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 -- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b40e2540 msgr2=0x55e5b40e0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b40e2540 0x55e5b40e0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 -- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40b4660 msgr2=0x55e5b4176780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40b4660 0x55e5b4176780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 -- 172.21.15.6:0/344928381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5b40e1510 con 0x55e5b4181830 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f111de00640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55e5b40e2540 0x55e5b40e0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1126800640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55e5b40b4660 0x55e5b4176780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.011+0000 7f1127200640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b4181830 0x55e5b41823e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1120002560 tx=0x7f1120002a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.012+0000 7f111fe00640 1 -- 172.21.15.6:0/344928381 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120017020 con 0x55e5b4181830 2024-03-06T07:34:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.012+0000 7f111fe00640 1 -- 172.21.15.6:0/344928381 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120013870 con 0x55e5b4181830 2024-03-06T07:34:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.012+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e5b4185500 con 0x55e5b4181830 2024-03-06T07:34:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.012+0000 7f111fe00640 1 -- 172.21.15.6:0/344928381 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120013c60 con 0x55e5b4181830 2024-03-06T07:34:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.012+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5b4179f00 con 0x55e5b4181830 2024-03-06T07:34:56.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.014+0000 7f111fe00640 1 -- 172.21.15.6:0/344928381 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f112001a020 con 0x55e5b4181830 2024-03-06T07:34:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.015+0000 7f111fe00640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1104069eb0 0x7f110406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.015+0000 7f111fe00640 1 -- 172.21.15.6:0/344928381 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1120089630 con 0x55e5b4181830 2024-03-06T07:34:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.015+0000 7f1129586f40 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55e5b40dd410 0x55e5b40dd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.015+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v9 -- 0x55e5b4185a00 con 0x55e5b40dd410 2024-03-06T07:34:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.015+0000 7f1126800640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1104069eb0 0x7f110406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.018 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.015+0000 7f111de00640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55e5b40dd410 0x55e5b40dd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.016+0000 7f111de00640 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55e5b40dd410 0x55e5b40dd810 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.016+0000 7f111de00640 1 -- 172.21.15.6:0/344928381 <== osd.2 v2:172.21.15.6:6822/1963740543 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f1110052970 con 0x55e5b40dd410 2024-03-06T07:34:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55e5b40dd410 msgr2=0x55e5b40dd810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x55e5b40dd410 0x55e5b40dd810 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.019 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1104069eb0 msgr2=0x7f110406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1104069eb0 0x7f110406c370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f1118003040 tx=0x7f1118009000 comp rx=0 tx=0).stop 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b4181830 msgr2=0x55e5b41823e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 --2- 172.21.15.6:0/344928381 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55e5b4181830 0x55e5b41823e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f1120002560 tx=0x7f1120002a20 comp rx=0 tx=0).stop 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1127200640 1 -- 172.21.15.6:0/344928381 reap_dead start 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 shutdown_connections 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 >> 172.21.15.6:0/344928381 conn(0x55e5b40b25c0 msgr2=0x55e5b40dc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 shutdown_connections 2024-03-06T07:34:56.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.017+0000 7f1129586f40 1 -- 172.21.15.6:0/344928381 wait complete. 2024-03-06T07:34:56.023 INFO:teuthology.orchestra.run.smithi006.stdout:W�file_ddir_2, 2024-03-06T07:34:56.023 DEBUG:teuthology.orchestra.run.smithi006:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:34:56.112 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "ino": 1099511627777, 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "ancestors": [ 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1099511628278, 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "file_d", 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 27 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "dirino": 1, 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "dname": "dir_2", 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: "version": 44 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:34:56.113 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-06T07:34:56.114 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 17, 2024-03-06T07:34:56.114 INFO:teuthology.orchestra.run.smithi006.stdout: "old_pools": [] 2024-03-06T07:34:56.114 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:34:56.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:56.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:34:56.118 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:34:56.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:34:56.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.384+0000 7f3877200640 1 Processor -- start 2024-03-06T07:34:56.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.384+0000 7f3877200640 1 -- start start 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3877200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3877200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f38700af580 0x7f38700b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3877200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38700cdc20 0x7f38700b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3877200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387015cc40 con 0x7f38700cdc20 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3877200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f387015cf20 con 0x7f38700af580 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3877200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387015d200 con 0x7f38700af1a0 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3875200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38700cdc20 0x7f38700b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3875200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38700cdc20 0x7f38700b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60784/0 (socket says 172.21.15.6:60784) 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.386+0000 7f3875200640 1 -- 172.21.15.6:0/1712924662 learned_addr learned my addr 172.21.15.6:0/1712924662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:56.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f386f400640 1 --2- 172.21.15.6:0/1712924662 >> v2:172.21.15.178:3300/0 conn(0x7f38700af580 0x7f38700b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f386fe00640 1 --2- 172.21.15.6:0/1712924662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f3875200640 1 -- 172.21.15.6:0/1712924662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 msgr2=0x7f387015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f3875200640 1 --2- 172.21.15.6:0/1712924662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f3875200640 1 -- 172.21.15.6:0/1712924662 >> v2:172.21.15.178:3300/0 conn(0x7f38700af580 msgr2=0x7f38700b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f3875200640 1 --2- 172.21.15.6:0/1712924662 >> v2:172.21.15.178:3300/0 conn(0x7f38700af580 0x7f38700b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.387+0000 7f3875200640 1 -- 172.21.15.6:0/1712924662 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387015d4e0 con 0x7f38700cdc20 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.388+0000 7f3875200640 1 --2- 172.21.15.6:0/1712924662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700cdc20 0x7f38700b2dd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f38700748c0 tx=0x7f386802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842b57126217b451 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f386ea00640 1 -- 172.21.15.6:0/1712924662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386802c9e0 con 0x7f38700cdc20 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f386ea00640 1 -- 172.21.15.6:0/1712924662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386802cb80 con 0x7f38700cdc20 2024-03-06T07:34:56.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f386ea00640 1 -- 172.21.15.6:0/1712924662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38680343f0 con 0x7f38700cdc20 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 -- 172.21.15.6:0/1712924662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700cdc20 msgr2=0x7f38700b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 --2- 172.21.15.6:0/1712924662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700cdc20 0x7f38700b2dd0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f38700748c0 tx=0x7f386802be40 comp rx=0 tx=0).stop 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 -- 172.21.15.6:0/1712924662 shutdown_connections 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 --2- 172.21.15.6:0/1712924662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700cdc20 0x7f38700b2dd0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 --2- 172.21.15.6:0/1712924662 >> v2:172.21.15.178:3300/0 conn(0x7f38700af580 0x7f38700b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 --2- 172.21.15.6:0/1712924662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 -- 172.21.15.6:0/1712924662 >> 172.21.15.6:0/1712924662 conn(0x7f387006afb0 msgr2=0x7f387006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.389+0000 7f3877200640 1 -- 172.21.15.6:0/1712924662 shutdown_connections 2024-03-06T07:34:56.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.390+0000 7f3877200640 1 -- 172.21.15.6:0/1712924662 wait complete. 2024-03-06T07:34:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.392+0000 7f3877200640 1 Processor -- start 2024-03-06T07:34:56.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.393+0000 7f3877200640 1 -- start start 2024-03-06T07:34:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f3877200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387014b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f3877200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 0x7f38700b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f3877200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38700cdc20 0x7f38700b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f3877200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38700b3490 con 0x7f38700af580 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f3877200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38700b3310 con 0x7f38700cdc20 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f3877200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38700b3610 con 0x7f38700af1a0 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f386f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 0x7f38700b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f386f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 0x7f38700b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60794/0 (socket says 172.21.15.6:60794) 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.394+0000 7f386f400640 1 -- 172.21.15.6:0/1224656601 learned_addr learned my addr 172.21.15.6:0/1224656601 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:34:56.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386fe00640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387014b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386f400640 1 -- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 msgr2=0x7f387014b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386f400640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387014b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386f400640 1 -- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38700cdc20 msgr2=0x7f38700b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386f400640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38700cdc20 0x7f38700b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386f400640 1 -- 172.21.15.6:0/1224656601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387009d4b0 con 0x7f38700af580 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f386f400640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 0x7f38700b0410 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f385c00d830 tx=0x7f385c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c00b6c0 con 0x7f38700af580 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c019070 con 0x7f38700af580 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.395+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c00ba80 con 0x7f38700af580 2024-03-06T07:34:56.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.396+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387014e000 con 0x7f38700af580 2024-03-06T07:34:56.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.396+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387014e5c0 con 0x7f38700af580 2024-03-06T07:34:56.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.397+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387004f3a0 con 0x7f38700af580 2024-03-06T07:34:56.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.398+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f385c01e460 con 0x7f38700af580 2024-03-06T07:34:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.400+0000 7f3867e00640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f385006b550 0x7f385006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:34:56.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.400+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f385c08a8a0 con 0x7f38700af580 2024-03-06T07:34:56.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.400+0000 7f386fe00640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f385006b550 0x7f385006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:34:56.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.402+0000 7f386fe00640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f385006b550 0x7f385006da10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3860006fd0 tx=0x7f3860008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:34:56.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.404+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f385c059700 con 0x7f38700af580 2024-03-06T07:34:56.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.642+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38701552c0 con 0x7f38700af580 2024-03-06T07:34:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.643+0000 7f3867e00640 1 -- 172.21.15.6:0/1224656601 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f385c059000 con 0x7f38700af580 2024-03-06T07:34:56.645 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:34:56.645 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:34:56.645 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.645+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f385006b550 msgr2=0x7f385006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.645+0000 7f3877200640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f385006b550 0x7f385006da10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f3860006fd0 tx=0x7f3860008040 comp rx=0 tx=0).stop 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.645+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 msgr2=0x7f38700b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.645+0000 7f3877200640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 0x7f38700b0410 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f385c00d830 tx=0x7f385c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.645+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 shutdown_connections 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.645+0000 7f3877200640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f385006b550 0x7f385006da10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.646+0000 7f3877200640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38700cdc20 0x7f38700b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.646+0000 7f3877200640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38700af580 0x7f38700b0410 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.646+0000 7f3877200640 1 --2- 172.21.15.6:0/1224656601 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f38700af1a0 0x7f387014b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.646+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 >> 172.21.15.6:0/1224656601 conn(0x7f387006afb0 msgr2=0x7f38700cd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.646+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 shutdown_connections 2024-03-06T07:34:56.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:34:56.646+0000 7f3877200640 1 -- 172.21.15.6:0/1224656601 wait complete. 2024-03-06T07:34:56.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:34:56.893 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:34:56.893 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:34:56.893 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:35:01.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:02.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.122+0000 7fc7ea000640 1 Processor -- start 2024-03-06T07:35:02.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.123+0000 7fc7ea000640 1 -- start start 2024-03-06T07:35:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.124+0000 7fc7ea000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc7e4145300 0x7fc7e40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.124+0000 7fc7ea000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e4070f10 0x7fc7e40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.124+0000 7fc7ea000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7e40cce90 0x7fc7e40cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.124+0000 7fc7ea000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e40cd300 con 0x7fc7e40cce90 2024-03-06T07:35:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.124+0000 7fc7ea000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e40714d0 con 0x7fc7e4145300 2024-03-06T07:35:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.124+0000 7fc7ea000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40717b0 con 0x7fc7e4070f10 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7e40cce90 0x7fc7e40cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc7e40cce90 0x7fc7e40cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59620/0 (socket says 172.21.15.6:59620) 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 -- 172.21.15.6:0/1198818230 learned_addr learned my addr 172.21.15.6:0/1198818230 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e8c00640 1 --2- 172.21.15.6:0/1198818230 >> v2:172.21.15.178:3300/0 conn(0x7fc7e4145300 0x7fc7e40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 -- 172.21.15.6:0/1198818230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e4070f10 msgr2=0x7fc7e40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 --2- 172.21.15.6:0/1198818230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e4070f10 0x7fc7e40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 -- 172.21.15.6:0/1198818230 >> v2:172.21.15.178:3300/0 conn(0x7fc7e4145300 msgr2=0x7fc7e40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 --2- 172.21.15.6:0/1198818230 >> v2:172.21.15.178:3300/0 conn(0x7fc7e4145300 0x7fc7e40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e9600640 1 -- 172.21.15.6:0/1198818230 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e4071a90 con 0x7fc7e40cce90 2024-03-06T07:35:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.125+0000 7fc7e8c00640 1 --2- 172.21.15.6:0/1198818230 >> v2:172.21.15.178:3300/0 conn(0x7fc7e4145300 0x7fc7e40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.126+0000 7fc7e9600640 1 --2- 172.21.15.6:0/1198818230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e40cce90 0x7fc7e40cd930 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40cf990 tx=0x7fc7dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797efeca9b194c8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7e3400640 1 -- 172.21.15.6:0/1198818230 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc02ca70 con 0x7fc7e40cce90 2024-03-06T07:35:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7e3400640 1 -- 172.21.15.6:0/1198818230 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc02cc10 con 0x7fc7e40cce90 2024-03-06T07:35:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7e3400640 1 -- 172.21.15.6:0/1198818230 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0343f0 con 0x7fc7e40cce90 2024-03-06T07:35:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 -- 172.21.15.6:0/1198818230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e40cce90 msgr2=0x7fc7e40cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 --2- 172.21.15.6:0/1198818230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e40cce90 0x7fc7e40cd930 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fc7e40cf990 tx=0x7fc7dc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 -- 172.21.15.6:0/1198818230 shutdown_connections 2024-03-06T07:35:02.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 --2- 172.21.15.6:0/1198818230 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e40cce90 0x7fc7e40cd930 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 --2- 172.21.15.6:0/1198818230 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e4070f10 0x7fc7e40cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 --2- 172.21.15.6:0/1198818230 >> v2:172.21.15.178:3300/0 conn(0x7fc7e4145300 0x7fc7e40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.127+0000 7fc7ea000640 1 -- 172.21.15.6:0/1198818230 >> 172.21.15.6:0/1198818230 conn(0x7fc7e406b3d0 msgr2=0x7fc7e406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.128+0000 7fc7ea000640 1 -- 172.21.15.6:0/1198818230 shutdown_connections 2024-03-06T07:35:02.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.128+0000 7fc7ea000640 1 -- 172.21.15.6:0/1198818230 wait complete. 2024-03-06T07:35:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.130+0000 7fc7ea000640 1 Processor -- start 2024-03-06T07:35:02.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.130+0000 7fc7ea000640 1 -- start start 2024-03-06T07:35:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7ea000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7e4070f10 0x7fc7e406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7ea000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e40cce90 0x7fc7e40a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7ea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 0x7fc7e40a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7ea000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e41456e0 con 0x7fc7e4145300 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7ea000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40b4ee0 con 0x7fc7e4070f10 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7ea000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40cde70 con 0x7fc7e40cce90 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 0x7fc7e40a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 0x7fc7e40a99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59636/0 (socket says 172.21.15.6:59636) 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 -- 172.21.15.6:0/760112455 learned_addr learned my addr 172.21.15.6:0/760112455 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e8c00640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7e4070f10 0x7fc7e406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e3e00640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e40cce90 0x7fc7e40a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 -- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e40cce90 msgr2=0x7fc7e40a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e40cce90 0x7fc7e40a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 -- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7e4070f10 msgr2=0x7fc7e406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7e4070f10 0x7fc7e406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e9600640 1 -- 172.21.15.6:0/760112455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e406d0c0 con 0x7fc7e4145300 2024-03-06T07:35:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.132+0000 7fc7e8c00640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7e4070f10 0x7fc7e406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.133+0000 7fc7e9600640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 0x7fc7e40a99b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009f40 tx=0x7fc7dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.133+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc043020 con 0x7fc7e4145300 2024-03-06T07:35:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.133+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc03e450 con 0x7fc7e4145300 2024-03-06T07:35:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.133+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc03e820 con 0x7fc7e4145300 2024-03-06T07:35:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.133+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e406d3a0 con 0x7fc7e4145300 2024-03-06T07:35:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.133+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e406d960 con 0x7fc7e4145300 2024-03-06T07:35:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.135+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e415bb30 con 0x7fc7e4145300 2024-03-06T07:35:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.135+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7dc046020 con 0x7fc7e4145300 2024-03-06T07:35:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.136+0000 7fc7e0c00640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7c806b4d0 0x7fc7c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.136+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc7dc0acd00 con 0x7fc7e4145300 2024-03-06T07:35:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.137+0000 7fc7e8c00640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7c806b4d0 0x7fc7c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.138+0000 7fc7e8c00640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7c806b4d0 0x7fc7c806d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8006fd0 tx=0x7fc7d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.146+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7e415bb30 con 0x7fc7e4145300 2024-03-06T07:35:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.384+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7e4144ab0 con 0x7fc7e4145300 2024-03-06T07:35:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.385+0000 7fc7e0c00640 1 -- 172.21.15.6:0/760112455 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fc7dc04d030 con 0x7fc7e4145300 2024-03-06T07:35:02.387 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:02.387 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:02.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7c806b4d0 msgr2=0x7fc7c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7c806b4d0 0x7fc7c806d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8006fd0 tx=0x7fc7d8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:02.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 msgr2=0x7fc7e40a99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 0x7fc7e40a99b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009f40 tx=0x7fc7dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 shutdown_connections 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc7c806b4d0 0x7fc7c806d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc7e4145300 0x7fc7e40a99b0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc7e40cce90 0x7fc7e40a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 --2- 172.21.15.6:0/760112455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc7e4070f10 0x7fc7e406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 >> 172.21.15.6:0/760112455 conn(0x7fc7e406b3d0 msgr2=0x7fc7e415ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 shutdown_connections 2024-03-06T07:35:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.387+0000 7fc7ea000640 1 -- 172.21.15.6:0/760112455 wait complete. 2024-03-06T07:35:02.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:02.633 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:35:02.634 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:35:02.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:02.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.836+0000 7f5423e00640 1 Processor -- start 2024-03-06T07:35:02.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.836+0000 7f5423e00640 1 -- start start 2024-03-06T07:35:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.839+0000 7f5423e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5424145300 0x7f54240b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.839+0000 7f5423e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240714d0 con 0x7f54240cd060 2024-03-06T07:35:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.839+0000 7f5423e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240717b0 con 0x7f5424070f10 2024-03-06T07:35:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.839+0000 7f5423e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5424071a90 con 0x7f5424145300 2024-03-06T07:35:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.840+0000 7f5423400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f54240cd060 0x7f542415c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46374/0 (socket says 172.21.15.6:46374) 2024-03-06T07:35:02.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.840+0000 7f5423400640 1 -- 172.21.15.6:0/1437998669 learned_addr learned my addr 172.21.15.6:0/1437998669 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.840+0000 7f5422a00640 1 --2- 172.21.15.6:0/1437998669 >> v2:172.21.15.178:3300/0 conn(0x7f5424145300 0x7f54240b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.840+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144161673 0 0) 0x7f54240714d0 con 0x7f54240cd060 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.840+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5408003520 con 0x7f54240cd060 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 186079122 0 0) 0x7f5408003520 con 0x7f54240cd060 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 >> v2:172.21.15.178:3300/0 conn(0x7f5424145300 msgr2=0x7f54240b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 --2- 172.21.15.6:0/1437998669 >> v2:172.21.15.178:3300/0 conn(0x7f5424145300 0x7f54240b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 >> v1:172.21.15.178:6789/0 conn(0x7f5424070f10 legacy=0x7f54240cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424071d70 con 0x7f54240cd060 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f541c0019f0 con 0x7f54240cd060 2024-03-06T07:35:02.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 >> v1:172.21.15.6:6789/0 conn(0x7f54240cd060 legacy=0x7f542415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080039b0 0x7f5408003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424071a90 con 0x7f54080051c0 2024-03-06T07:35:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540800c0c0 con 0x7f54080039b0 2024-03-06T07:35:02.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540800c2c0 con 0x7f5408004800 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.841+0000 7f5423400640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5422a00640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080039b0 0x7f5408003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5422000640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5423400640 1 -- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5408004800 msgr2=0x7f5408004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5423400640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5408004800 0x7f5408004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5423400640 1 -- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080039b0 msgr2=0x7f5408003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5423400640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080039b0 0x7f5408003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.842+0000 7f5423400640 1 -- 172.21.15.6:0/1437998669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5408003520 con 0x7f54080051c0 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.843+0000 7f5423400640 1 -- 172.21.15.6:0/1437998669 reap_dead start 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.844+0000 7f5423400640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f541c00a780 tx=0x7f5408005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71f544e3735f758 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.844+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c00a3d0 con 0x7f54080051c0 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.844+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c034630 con 0x7f54080051c0 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.844+0000 7f5421600640 1 -- 172.21.15.6:0/1437998669 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541c034a20 con 0x7f54080051c0 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.844+0000 7f5423e00640 1 -- 172.21.15.6:0/1437998669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54080051c0 msgr2=0x7f540800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.844+0000 7f5423e00640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f541c00a780 tx=0x7f5408005830 comp rx=0 tx=0).stop 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.845+0000 7f5423e00640 1 -- 172.21.15.6:0/1437998669 shutdown_connections 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.845+0000 7f5423e00640 1 --2- 172.21.15.6:0/1437998669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54080051c0 0x7f540800b9c0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.845+0000 7f5423e00640 1 -- 172.21.15.6:0/1437998669 >> 172.21.15.6:0/1437998669 conn(0x7f542406b3d0 msgr2=0x7f542406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.845+0000 7f5423e00640 1 -- 172.21.15.6:0/1437998669 shutdown_connections 2024-03-06T07:35:02.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.845+0000 7f5423e00640 1 -- 172.21.15.6:0/1437998669 wait complete. 2024-03-06T07:35:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.847+0000 7f5423e00640 1 Processor -- start 2024-03-06T07:35:02.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.847+0000 7f5423e00640 1 -- start start 2024-03-06T07:35:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080051c0 0x7f542406da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 0x7f54240cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54240cb660 0x7f5424138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54240ccad0 con 0x7f5424139560 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54240cc950 con 0x7f54080051c0 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54240ccc50 con 0x7f54240cb660 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 0x7f54240cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 0x7f54240cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59648/0 (socket says 172.21.15.6:59648) 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 -- 172.21.15.6:0/2302407439 learned_addr learned my addr 172.21.15.6:0/2302407439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422a00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080051c0 0x7f542406da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5423400640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54240cb660 0x7f5424138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 -- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54240cb660 msgr2=0x7f5424138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54240cb660 0x7f5424138330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 -- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080051c0 msgr2=0x7f542406da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080051c0 0x7f542406da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:02.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.848+0000 7f5422000640 1 -- 172.21.15.6:0/2302407439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54240cbb10 con 0x7f5424139560 2024-03-06T07:35:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.849+0000 7f5422000640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 0x7f54240cb120 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f541000e560 tx=0x7f541000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.849+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54100052e0 con 0x7f5424139560 2024-03-06T07:35:02.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.849+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410018070 con 0x7f5424139560 2024-03-06T07:35:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.849+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410003830 con 0x7f5424139560 2024-03-06T07:35:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.849+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5424136210 con 0x7f5424139560 2024-03-06T07:35:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.849+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54241367d0 con 0x7f5424139560 2024-03-06T07:35:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.850+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f542404f330 con 0x7f5424139560 2024-03-06T07:35:02.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.851+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54100039d0 con 0x7f5424139560 2024-03-06T07:35:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.853+0000 7f541aa00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540406b4d0 0x7f540406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.853+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f541008a0d0 con 0x7f5424139560 2024-03-06T07:35:02.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.853+0000 7f5422a00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540406b4d0 0x7f540406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:02.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.854+0000 7f5422a00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540406b4d0 0x7f540406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5414009980 tx=0x7f5414008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:02.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:02.857+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5410058d90 con 0x7f5424139560 2024-03-06T07:35:03.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.109+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54240b4b20 con 0x7f5424139560 2024-03-06T07:35:03.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.110+0000 7f541aa00640 1 -- 172.21.15.6:0/2302407439 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f5410058690 con 0x7f5424139560 2024-03-06T07:35:03.113 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:03.113 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:03.113 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540406b4d0 msgr2=0x7f540406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540406b4d0 0x7f540406d990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f5414009980 tx=0x7f5414008040 comp rx=0 tx=0).stop 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 msgr2=0x7f54240cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 0x7f54240cb120 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f541000e560 tx=0x7f541000ea20 comp rx=0 tx=0).stop 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 shutdown_connections 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f540406b4d0 0x7f540406d990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f54240cb660 0x7f5424138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5424139560 0x7f54240cb120 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 --2- 172.21.15.6:0/2302407439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54080051c0 0x7f542406da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 >> 172.21.15.6:0/2302407439 conn(0x7f542406b3d0 msgr2=0x7f542415e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.113+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 shutdown_connections 2024-03-06T07:35:03.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.114+0000 7f5423e00640 1 -- 172.21.15.6:0/2302407439 wait complete. 2024-03-06T07:35:03.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:35:03.399 INFO:teuthology.orchestra.run.smithi006.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:35:03.399 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:35:03.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:03.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.615+0000 7f5250a00640 1 Processor -- start 2024-03-06T07:35:03.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.615+0000 7f5250a00640 1 -- start start 2024-03-06T07:35:03.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5250a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f524c152bc0 0x7f524c0cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5250a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c06c6d0 0x7f524c0ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5250a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f524c15c5e0 con 0x7f524c0af940 2024-03-06T07:35:03.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5250a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524c15cf10 con 0x7f524c152bc0 2024-03-06T07:35:03.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5250a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c15c8c0 con 0x7f524c06c6d0 2024-03-06T07:35:03.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5249e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f524c152bc0 0x7f524c0cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5249e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f524c152bc0 0x7f524c0cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59650/0 (socket says 172.21.15.6:59650) 2024-03-06T07:35:03.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.617+0000 7f5249e00640 1 -- 172.21.15.6:0/2152284621 learned_addr learned my addr 172.21.15.6:0/2152284621 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:03.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.618+0000 7f524a800640 1 --2- 172.21.15.6:0/2152284621 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c06c6d0 0x7f524c0ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.618+0000 7f5249e00640 1 -- 172.21.15.6:0/2152284621 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c06c6d0 msgr2=0x7f524c0ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.618+0000 7f5249e00640 1 --2- 172.21.15.6:0/2152284621 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c06c6d0 0x7f524c0ae5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.618+0000 7f5249e00640 1 -- 172.21.15.6:0/2152284621 >> v1:172.21.15.178:6789/0 conn(0x7f524c0af940 legacy=0x7f524c06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:03.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.618+0000 7f5249e00640 1 -- 172.21.15.6:0/2152284621 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c15cba0 con 0x7f524c152bc0 2024-03-06T07:35:03.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.619+0000 7f5249e00640 1 --2- 172.21.15.6:0/2152284621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0cfce0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f523c009e50 tx=0x7f523c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ff024e63e591e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.619+0000 7f5249400640 1 -- 172.21.15.6:0/2152284621 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c02ca70 con 0x7f524c152bc0 2024-03-06T07:35:03.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5249400640 1 -- 172.21.15.6:0/2152284621 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c02cc10 con 0x7f524c152bc0 2024-03-06T07:35:03.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5249400640 1 -- 172.21.15.6:0/2152284621 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c0343f0 con 0x7f524c152bc0 2024-03-06T07:35:03.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5250a00640 1 -- 172.21.15.6:0/2152284621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 msgr2=0x7f524c0cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5250a00640 1 --2- 172.21.15.6:0/2152284621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0cfce0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f523c009e50 tx=0x7f523c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5250a00640 1 -- 172.21.15.6:0/2152284621 shutdown_connections 2024-03-06T07:35:03.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5250a00640 1 --2- 172.21.15.6:0/2152284621 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c06c6d0 0x7f524c0ae5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5250a00640 1 --2- 172.21.15.6:0/2152284621 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0cfce0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.620+0000 7f5250a00640 1 -- 172.21.15.6:0/2152284621 >> 172.21.15.6:0/2152284621 conn(0x7f524c0714d0 msgr2=0x7f524c0718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.621+0000 7f5250a00640 1 -- 172.21.15.6:0/2152284621 shutdown_connections 2024-03-06T07:35:03.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.621+0000 7f5250a00640 1 -- 172.21.15.6:0/2152284621 wait complete. 2024-03-06T07:35:03.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.623+0000 7f5250a00640 1 Processor -- start 2024-03-06T07:35:03.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.624+0000 7f5250a00640 1 -- start start 2024-03-06T07:35:03.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f5250a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f524c06c6d0 0x7f524c0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f5250a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c0af940 0x7f524c15b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f5250a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f5250a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0aff00 con 0x7f524c152bc0 2024-03-06T07:35:03.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f5250a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524c0afd80 con 0x7f524c06c6d0 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f5250a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524c152fa0 con 0x7f524c0af940 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f524a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.625+0000 7f524a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59662/0 (socket says 172.21.15.6:59662) 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 -- 172.21.15.6:0/465752102 learned_addr learned my addr 172.21.15.6:0/465752102 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f5241400640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c0af940 0x7f524c15b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f5249e00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f524c06c6d0 0x7f524c0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 -- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c0af940 msgr2=0x7f524c15b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c0af940 0x7f524c15b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 -- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f524c06c6d0 msgr2=0x7f524c0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f524c06c6d0 0x7f524c0af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 -- 172.21.15.6:0/465752102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524c149540 con 0x7f524c152bc0 2024-03-06T07:35:03.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.626+0000 7f524a800640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0b4510 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f524400c5c0 tx=0x7f524400ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.627+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244015070 con 0x7f524c152bc0 2024-03-06T07:35:03.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.627+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5244004220 con 0x7f524c152bc0 2024-03-06T07:35:03.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.627+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5244003610 con 0x7f524c152bc0 2024-03-06T07:35:03.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.627+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524c0b3750 con 0x7f524c152bc0 2024-03-06T07:35:03.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.627+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524c0b3d10 con 0x7f524c152bc0 2024-03-06T07:35:03.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.628+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c04f3a0 con 0x7f524c152bc0 2024-03-06T07:35:03.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.629+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52440037b0 con 0x7f524c152bc0 2024-03-06T07:35:03.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.630+0000 7f5242a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f522806b4d0 0x7f522806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:03.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.630+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f524408a070 con 0x7f524c152bc0 2024-03-06T07:35:03.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.631+0000 7f5249e00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f522806b4d0 0x7f522806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:03.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.632+0000 7f5249e00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f522806b4d0 0x7f522806d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f523c035000 tx=0x7f523c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:03.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.639+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5244058ef0 con 0x7f524c152bc0 2024-03-06T07:35:03.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.883+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f524c072100 con 0x7f524c152bc0 2024-03-06T07:35:03.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.885+0000 7f5242a00640 1 -- 172.21.15.6:0/465752102 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f52440587f0 con 0x7f524c152bc0 2024-03-06T07:35:03.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:03.888 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:03.889 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:03.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.887+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f522806b4d0 msgr2=0x7f522806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.887+0000 7f5250a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f522806b4d0 0x7f522806d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f523c035000 tx=0x7f523c030000 comp rx=0 tx=0).stop 2024-03-06T07:35:03.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.887+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 msgr2=0x7f524c0b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:03.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.887+0000 7f5250a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0b4510 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f524400c5c0 tx=0x7f524400ca80 comp rx=0 tx=0).stop 2024-03-06T07:35:03.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.887+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 shutdown_connections 2024-03-06T07:35:03.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.887+0000 7f5250a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f522806b4d0 0x7f522806d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.888+0000 7f5250a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f524c152bc0 0x7f524c0b4510 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.888+0000 7f5250a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f524c0af940 0x7f524c15b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.888+0000 7f5250a00640 1 --2- 172.21.15.6:0/465752102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f524c06c6d0 0x7f524c0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.888+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 >> 172.21.15.6:0/465752102 conn(0x7f524c0714d0 msgr2=0x7f524c1515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.888+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 shutdown_connections 2024-03-06T07:35:03.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:03.888+0000 7f5250a00640 1 -- 172.21.15.6:0/465752102 wait complete. 2024-03-06T07:35:03.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-03-06T07:35:04.167 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:35:04.168 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:35:04.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.413+0000 7f89e3200640 1 Processor -- start 2024-03-06T07:35:04.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.414+0000 7f89e3200640 1 -- start start 2024-03-06T07:35:04.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.415+0000 7f89e3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.415+0000 7f89e3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f89e40d4a10 0x7f89e40d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.415+0000 7f89e3200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e40b3710 con 0x7f89e40d0c80 2024-03-06T07:35:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.415+0000 7f89e3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e40b09e0 con 0x7f89e40d4a10 2024-03-06T07:35:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.415+0000 7f89e3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40b39f0 con 0x7f89e415e0b0 2024-03-06T07:35:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.415+0000 7f89e1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f89e40d4a10 0x7f89e40d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.419 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f89e40d4a10 0x7f89e40d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59666/0 (socket says 172.21.15.6:59666) 2024-03-06T07:35:04.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1400640 1 -- 172.21.15.6:0/1999759682 learned_addr learned my addr 172.21.15.6:0/1999759682 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:04.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1e00640 1 --2- 172.21.15.6:0/1999759682 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1400640 1 -- 172.21.15.6:0/1999759682 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 msgr2=0x7f89e40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1400640 1 --2- 172.21.15.6:0/1999759682 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e40cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1400640 1 -- 172.21.15.6:0/1999759682 >> v1:172.21.15.178:6789/0 conn(0x7f89e40d0c80 legacy=0x7f89e40d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:04.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.416+0000 7f89e1400640 1 -- 172.21.15.6:0/1999759682 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e40b3cd0 con 0x7f89e40d4a10 2024-03-06T07:35:04.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.417+0000 7f89e1400640 1 --2- 172.21.15.6:0/1999759682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d4a10 0x7f89e40d0740 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009e50 tx=0x7f89d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65be3dec42af5ad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.418+0000 7f89e0a00640 1 -- 172.21.15.6:0/1999759682 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d402cb00 con 0x7f89e40d4a10 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.418+0000 7f89e0a00640 1 -- 172.21.15.6:0/1999759682 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89d402cca0 con 0x7f89e40d4a10 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.418+0000 7f89e0a00640 1 -- 172.21.15.6:0/1999759682 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4034610 con 0x7f89e40d4a10 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.418+0000 7f89e3200640 1 -- 172.21.15.6:0/1999759682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d4a10 msgr2=0x7f89e40d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.418+0000 7f89e3200640 1 --2- 172.21.15.6:0/1999759682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d4a10 0x7f89e40d0740 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009e50 tx=0x7f89d402bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.419+0000 7f89e3200640 1 -- 172.21.15.6:0/1999759682 shutdown_connections 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.419+0000 7f89e3200640 1 --2- 172.21.15.6:0/1999759682 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d4a10 0x7f89e40d0740 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.419+0000 7f89e3200640 1 --2- 172.21.15.6:0/1999759682 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e40cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.419+0000 7f89e3200640 1 -- 172.21.15.6:0/1999759682 >> 172.21.15.6:0/1999759682 conn(0x7f89e40d3d30 msgr2=0x7f89e40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.419+0000 7f89e3200640 1 -- 172.21.15.6:0/1999759682 shutdown_connections 2024-03-06T07:35:04.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.419+0000 7f89e3200640 1 -- 172.21.15.6:0/1999759682 wait complete. 2024-03-06T07:35:04.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.421+0000 7f89e3200640 1 Processor -- start 2024-03-06T07:35:04.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.422+0000 7f89e3200640 1 -- start start 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.423+0000 7f89e3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 0x7f89e4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.423+0000 7f89e3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e40d4a10 0x7f89e4159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.423+0000 7f89e3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e415f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.423+0000 7f89e3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40cfc60 con 0x7f89e40d0c80 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.423+0000 7f89e3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e415e490 con 0x7f89e40d4a10 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.423+0000 7f89e3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e40b54e0 con 0x7f89e415e0b0 2024-03-06T07:35:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 0x7f89e4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 0x7f89e4159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59674/0 (socket says 172.21.15.6:59674) 2024-03-06T07:35:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 -- 172.21.15.6:0/2736310721 learned_addr learned my addr 172.21.15.6:0/2736310721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 -- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 msgr2=0x7f89e415f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e415f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1400640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e40d4a10 0x7f89e4159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 -- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e40d4a10 msgr2=0x7f89e4159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e40d4a10 0x7f89e4159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 -- 172.21.15.6:0/2736310721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e415ff50 con 0x7f89e40d0c80 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.424+0000 7f89e1e00640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 0x7f89e4159e90 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00d5f0 tx=0x7f89cc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.425+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc0125f0 con 0x7f89e40d0c80 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.425+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89cc019070 con 0x7f89e40d0c80 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.425+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89cc0129d0 con 0x7f89e40d0c80 2024-03-06T07:35:04.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.425+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e40cc760 con 0x7f89e40d0c80 2024-03-06T07:35:04.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.425+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e40ccd20 con 0x7f89e40d0c80 2024-03-06T07:35:04.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.426+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e40cc550 con 0x7f89e40d0c80 2024-03-06T07:35:04.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.428+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89cc012b70 con 0x7f89e40d0c80 2024-03-06T07:35:04.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.429+0000 7f89da000640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89c006b4d0 0x7f89c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:04.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.429+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f89cc08a6b0 con 0x7f89e40d0c80 2024-03-06T07:35:04.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.430+0000 7f89e1400640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89c006b4d0 0x7f89c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:04.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.431+0000 7f89e1400640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89c006b4d0 0x7f89c006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f89d4035000 tx=0x7f89d4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:04.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.434+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89cc059490 con 0x7f89e40d0c80 2024-03-06T07:35:04.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.742+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89e40b4900 con 0x7f89e40d0c80 2024-03-06T07:35:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.743+0000 7f89da000640 1 -- 172.21.15.6:0/2736310721 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f89cc058d90 con 0x7f89e40d0c80 2024-03-06T07:35:04.746 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:04.746 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.746+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89c006b4d0 msgr2=0x7f89c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.746+0000 7f89e3200640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89c006b4d0 0x7f89c006d990 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f89d4035000 tx=0x7f89d4030000 comp rx=0 tx=0).stop 2024-03-06T07:35:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 msgr2=0x7f89e4159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 0x7f89e4159e90 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00d5f0 tx=0x7f89cc00dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 shutdown_connections 2024-03-06T07:35:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f89c006b4d0 0x7f89c006d990 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e40d4a10 0x7f89e4159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89e415e0b0 0x7f89e415f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 --2- 172.21.15.6:0/2736310721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89e40d0c80 0x7f89e4159e90 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 >> 172.21.15.6:0/2736310721 conn(0x7f89e40d3d30 msgr2=0x7f89e406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 shutdown_connections 2024-03-06T07:35:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:04.747+0000 7f89e3200640 1 -- 172.21.15.6:0/2736310721 wait complete. 2024-03-06T07:35:04.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-03-06T07:35:05.001 INFO:teuthology.orchestra.run.smithi006.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:35:05.002 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:35:05.002 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.095+0000 7f8d8e0d3f40 1 Processor -- start 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.095+0000 7f8d8e0d3f40 1 -- start start 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d8e0d3f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e74ab90 0x55b67e773bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d8e0d3f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 0x55b67e811ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d8e0d3f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e81aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d8e0d3f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67e6eefd0 con 0x55b67e791680 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d8e0d3f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67e74afa0 con 0x55b67e74ab90 2024-03-06T07:35:05.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d8e0d3f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b67e812d00 con 0x55b67e81f3c0 2024-03-06T07:35:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d83e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 0x55b67e811ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d83e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 0x55b67e811ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59682/0 (socket says 172.21.15.6:59682) 2024-03-06T07:35:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.096+0000 7f8d83e00640 1 -- 172.21.15.6:0/755872978 learned_addr learned my addr 172.21.15.6:0/755872978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:05.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e81aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d8b200640 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e74ab90 0x55b67e773bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d83e00640 1 -- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 msgr2=0x55b67e81aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d83e00640 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e81aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d83e00640 1 -- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e74ab90 msgr2=0x55b67e773bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d83e00640 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e74ab90 0x55b67e773bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d83e00640 1 -- 172.21.15.6:0/755872978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b67e81b860 con 0x55b67e791680 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d8b200640 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e74ab90 0x55b67e773bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d83e00640 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 0x55b67e811ce0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8d74000c00 tx=0x7f8d7404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9779ff7c3702f669 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d89e00640 1 -- 172.21.15.6:0/755872978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74057040 con 0x55b67e791680 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d89e00640 1 -- 172.21.15.6:0/755872978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d74062070 con 0x55b67e791680 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.097+0000 7f8d89e00640 1 -- 172.21.15.6:0/755872978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74052c30 con 0x55b67e791680 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/755872978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 msgr2=0x55b67e811ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 0x55b67e811ce0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8d74000c00 tx=0x7f8d7404c000 comp rx=0 tx=0).stop 2024-03-06T07:35:05.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/755872978 shutdown_connections 2024-03-06T07:35:05.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e81aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e791680 0x55b67e811ce0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/755872978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e74ab90 0x55b67e773bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/755872978 >> 172.21.15.6:0/755872978 conn(0x55b67e7485b0 msgr2=0x55b67e816420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/755872978 shutdown_connections 2024-03-06T07:35:05.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.098+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/755872978 wait complete. 2024-03-06T07:35:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.099+0000 7f8d8e0d3f40 1 Processor -- start 2024-03-06T07:35:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.099+0000 7f8d8e0d3f40 1 -- start start 2024-03-06T07:35:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8e0d3f40 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 0x55b67e81ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8e0d3f40 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e791680 0x55b67e815540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8e0d3f40 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e815a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8e0d3f40 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67e81bde0 con 0x55b67e74ab90 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8e0d3f40 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b67e81bb00 con 0x55b67e791680 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8e0d3f40 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b67e81c0c0 con 0x55b67e81f3c0 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 0x55b67e81ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 0x55b67e81ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59684/0 (socket says 172.21.15.6:59684) 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.100+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 learned_addr learned my addr 172.21.15.6:0/600782961 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e791680 0x55b67e815540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e815a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 msgr2=0x55b67e815a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x55b67e81f3c0 0x55b67e815a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e791680 msgr2=0x55b67e815540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e791680 0x55b67e815540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b67e8143e0 con 0x55b67e74ab90 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55b67e791680 0x55b67e815540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 0x55b67e81ae30 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0096a0 tx=0x7f8d7c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d88a00640 1 -- 172.21.15.6:0/600782961 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c006ac0 con 0x55b67e74ab90 2024-03-06T07:35:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d88a00640 1 -- 172.21.15.6:0/600782961 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c006c60 con 0x55b67e74ab90 2024-03-06T07:35:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d88a00640 1 -- 172.21.15.6:0/600782961 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c014660 con 0x55b67e74ab90 2024-03-06T07:35:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b67e748ed0 con 0x55b67e74ab90 2024-03-06T07:35:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.101+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b67e749450 con 0x55b67e74ab90 2024-03-06T07:35:05.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.105+0000 7f8d88a00640 1 -- 172.21.15.6:0/600782961 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d7c01d440 con 0x55b67e74ab90 2024-03-06T07:35:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.106+0000 7f8d88a00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d68069f00 0x7f8d6806c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.106+0000 7f8d88a00640 1 -- 172.21.15.6:0/600782961 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8d7c089260 con 0x55b67e74ab90 2024-03-06T07:35:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.107+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x55b67e803190 0x55b67e803fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.107+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.107+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d68069f00 0x7f8d6806c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.107+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x55b67e803190 0x55b67e803fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.108+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x55b67e803190 0x55b67e803fa0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.109+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d68069f00 0x7f8d6806c3c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8d74055040 tx=0x7f8d7404f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.109+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 1 ==== osd_op_reply(1 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.109+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f8d840128b0 0x7f8d84011330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.109+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.110+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f8d840128b0 0x7f8d84011330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.110+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f8d840128b0 0x7f8d84011330 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.111+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 1 ==== osd_op_reply(2 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c058d70 con 0x7f8d840128b0 2024-03-06T07:35:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.111+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7f8d7c057ca0 0x7f8d7c056900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.111+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.111+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7f8d7c057ca0 0x7f8d7c056900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.112+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7f8d7c057ca0 0x7f8d7c056900 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.112+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 1 ==== osd_op_reply(3 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.112+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7f8d74073410 0x7f8d74073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.112+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74073410 2024-03-06T07:35:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.113+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7f8d74073410 0x7f8d74073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.114+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7f8d74073410 0x7f8d74073850 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.115+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.5 v2:172.21.15.178:6800/1128394266 1 ==== osd_op_reply(4 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d74073410 2024-03-06T07:35:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.115+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.116+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 2 ==== osd_op_reply(5 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.116+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.116+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 2 ==== osd_op_reply(6 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.116+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7f8d74067910 0x7f8d740723a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.116+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.117+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7f8d74067910 0x7f8d740723a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.117+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7f8d74067910 0x7f8d740723a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.118+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 1 ==== osd_op_reply(7 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.118+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.119+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 3 ==== osd_op_reply(8 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.119+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.119+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 2 ==== osd_op_reply(9 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.119+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.120+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 3 ==== osd_op_reply(10 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.120+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.120+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 4 ==== osd_op_reply(11 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.120+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.121+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 3 ==== osd_op_reply(12 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.121+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.122+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 4 ==== osd_op_reply(13 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.122+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8d84015930 0x7f8d840213f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.122+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.123+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8d84015930 0x7f8d840213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.124+0000 7f8d83e00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8d84015930 0x7f8d840213f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.125+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 1 ==== osd_op_reply(14 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.125+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.125+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 2 ==== osd_op_reply(15 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d840128b0 2024-03-06T07:35:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.125+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.126+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 4 ==== osd_op_reply(16 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.126+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.127+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 2 ==== osd_op_reply(17 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.127+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74073410 2024-03-06T07:35:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.128+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.5 v2:172.21.15.178:6800/1128394266 2 ==== osd_op_reply(18 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d74073410 2024-03-06T07:35:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.128+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.128+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 5 ==== osd_op_reply(19 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.128+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.129+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 5 ==== osd_op_reply(20 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.129+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.130+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 6 ==== osd_op_reply(21 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.130+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.131+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 6 ==== osd_op_reply(22 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.131+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.131+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 5 ==== osd_op_reply(23 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.131+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.132+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 3 ==== osd_op_reply(24 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d840128b0 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.132+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f8d7c0872d0 0x7f8d7c05dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.132+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c0872d0 2024-03-06T07:35:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.132+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f8d7c0872d0 0x7f8d7c05dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.134+0000 7f8d8bc00640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f8d7c0872d0 0x7f8d7c05dac0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.134+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.6 v2:172.21.15.178:6824/196133891 1 ==== osd_op_reply(25 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d7c0872d0 2024-03-06T07:35:05.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.135+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.135+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 6 ==== osd_op_reply(26 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.135+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74073410 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.136+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.5 v2:172.21.15.178:6800/1128394266 3 ==== osd_op_reply(27 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d74073410 2024-03-06T07:35:05.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.136+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.137+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 7 ==== osd_op_reply(28 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.137+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f8d7c05fcc0 0x7f8d7c06f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.137+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c05fcc0 2024-03-06T07:35:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.137+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f8d7c05fcc0 0x7f8d7c06f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.137+0000 7f8d8b200640 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f8d7c05fcc0 0x7f8d7c06f0c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.138+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.2 v2:172.21.15.6:6822/1963740543 1 ==== osd_op_reply(29 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d7c05fcc0 2024-03-06T07:35:05.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.138+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c0872d0 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.139+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.6 v2:172.21.15.178:6824/196133891 2 ==== osd_op_reply(30 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d7c0872d0 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.139+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.139+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 4 ==== osd_op_reply(31 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d840128b0 2024-03-06T07:35:05.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.139+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.140+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 3 ==== osd_op_reply(32 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.140+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.141+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 8 ==== osd_op_reply(33 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.141+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.141+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 9 ==== osd_op_reply(34 [pgnls start_epoch 114 out=49b] v108'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.141+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.142+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 7 ==== osd_op_reply(35 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.142+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.142+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 5 ==== osd_op_reply(36 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d840128b0 2024-03-06T07:35:05.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.142+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.143+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 8 ==== osd_op_reply(37 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.143+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.144+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 4 ==== osd_op_reply(38 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.144+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74073410 2024-03-06T07:35:05.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.145+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.5 v2:172.21.15.178:6800/1128394266 4 ==== osd_op_reply(39 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d74073410 2024-03-06T07:35:05.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.145+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74073410 2024-03-06T07:35:05.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.145+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.5 v2:172.21.15.178:6800/1128394266 5 ==== osd_op_reply(40 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d74073410 2024-03-06T07:35:05.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.146+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c0872d0 2024-03-06T07:35:05.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.147+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.6 v2:172.21.15.178:6824/196133891 3 ==== osd_op_reply(41 [pgnls start_epoch 114 out=49b] v108'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d7c0872d0 2024-03-06T07:35:05.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.147+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.147+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 6 ==== osd_op_reply(42 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d840128b0 2024-03-06T07:35:05.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.147+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74073410 2024-03-06T07:35:05.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.148+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.5 v2:172.21.15.178:6800/1128394266 6 ==== osd_op_reply(43 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d74073410 2024-03-06T07:35:05.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.148+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.149+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 5 ==== osd_op_reply(44 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.149+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.150+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 7 ==== osd_op_reply(45 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.150+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.151+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 8 ==== osd_op_reply(46 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.151+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.152+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 9 ==== osd_op_reply(47 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.153+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.154+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 10 ==== osd_op_reply(48 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.154+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.155+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 6 ==== osd_op_reply(49 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.155+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.156+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 7 ==== osd_op_reply(50 [pgnls start_epoch 114 out=49b] v108'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.156+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c057ca0 2024-03-06T07:35:05.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.156+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.3 v2:172.21.15.6:6816/1271003476 9 ==== osd_op_reply(51 [pgnls start_epoch 114 out=49b] v108'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d7c057ca0 2024-03-06T07:35:05.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.156+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x55b67e803190 2024-03-06T07:35:05.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.157+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.7 v2:172.21.15.178:6808/56642032 11 ==== osd_op_reply(52 [pgnls start_epoch 114 out=49b] v108'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x55b67e803190 2024-03-06T07:35:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.157+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.158+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 10 ==== osd_op_reply(53 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.158+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.159+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 8 ==== osd_op_reply(54 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.159+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.160+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 9 ==== osd_op_reply(55 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.160+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c05fcc0 2024-03-06T07:35:05.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.161+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.2 v2:172.21.15.6:6822/1963740543 2 ==== osd_op_reply(56 [pgnls start_epoch 114 out=49b] v108'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d7c05fcc0 2024-03-06T07:35:05.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.161+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c05fcc0 2024-03-06T07:35:05.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.162+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.2 v2:172.21.15.6:6822/1963740543 3 ==== osd_op_reply(57 [pgnls start_epoch 114 out=49b] v114'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d7c05fcc0 2024-03-06T07:35:05.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.162+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.163+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 10 ==== osd_op_reply(58 [pgnls start_epoch 114 out=49b] v108'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.163+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c0872d0 2024-03-06T07:35:05.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.164+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.6 v2:172.21.15.178:6824/196133891 4 ==== osd_op_reply(59 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d7c0872d0 2024-03-06T07:35:05.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.164+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.165+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 11 ==== osd_op_reply(60 [pgnls start_epoch 114 out=49b] v108'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.165+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d840128b0 2024-03-06T07:35:05.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.165+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.1 v2:172.21.15.6:6808/1355309078 7 ==== osd_op_reply(61 [pgnls start_epoch 114 out=49b] v108'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d840128b0 2024-03-06T07:35:05.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.165+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d84015930 2024-03-06T07:35:05.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.167+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 <== osd.4 v2:172.21.15.178:6809/2203214816 11 ==== osd_op_reply(62 [pgnls start_epoch 114 out=49b] v114'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7405a040 con 0x7f8d84015930 2024-03-06T07:35:05.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.167+0000 7f8d83e00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d7c0872d0 2024-03-06T07:35:05.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.168+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 <== osd.6 v2:172.21.15.178:6824/196133891 5 ==== osd_op_reply(63 [pgnls start_epoch 114 out=49b] v108'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d8400e5c0 con 0x7f8d7c0872d0 2024-03-06T07:35:05.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.168+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 --> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 114 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e114) v9 -- 0x55b67e814810 con 0x7f8d74067910 2024-03-06T07:35:05.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8b200640 1 -- 172.21.15.6:0/600782961 <== osd.0 v2:172.21.15.6:6800/2712143005 12 ==== osd_op_reply(64 [pgnls start_epoch 114 out=49b] v108'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f8d7c087040 con 0x7f8d74067910 2024-03-06T07:35:05.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7f8d74067910 msgr2=0x7f8d740723a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6800/2712143005,v1:172.21.15.6:6801/2712143005] conn(0x7f8d74067910 0x7f8d740723a0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f8d840128b0 msgr2=0x7f8d84011330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6808/1355309078,v1:172.21.15.6:6809/1355309078] conn(0x7f8d840128b0 0x7f8d84011330 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f8d7c05fcc0 msgr2=0x7f8d7c06f0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6822/1963740543,v1:172.21.15.6:6825/1963740543] conn(0x7f8d7c05fcc0 0x7f8d7c06f0c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7f8d7c057ca0 msgr2=0x7f8d7c056900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:6816/1271003476,v1:172.21.15.6:6817/1271003476] conn(0x7f8d7c057ca0 0x7f8d7c056900 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8d84015930 msgr2=0x7f8d840213f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6809/2203214816,v1:172.21.15.178:6811/2203214816] conn(0x7f8d84015930 0x7f8d840213f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 reap_dead start 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7f8d74073410 msgr2=0x7f8d74073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6800/1128394266,v1:172.21.15.178:6801/1128394266] conn(0x7f8d74073410 0x7f8d74073850 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8bc00640 1 -- 172.21.15.6:0/600782961 reap_dead start 2024-03-06T07:35:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f8d7c0872d0 msgr2=0x7f8d7c05dac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f8d7c0872d0 0x7f8d7c05dac0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x55b67e803190 msgr2=0x55b67e803fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x55b67e803190 0x55b67e803fa0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d68069f00 msgr2=0x7f8d6806c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d68069f00 0x7f8d6806c3c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8d74055040 tx=0x7f8d7404f000 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 msgr2=0x55b67e81ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 0x55b67e81ae30 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0096a0 tx=0x7f8d7c009b60 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 shutdown_connections 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6824/196133891,v1:172.21.15.178:6825/196133891] conn(0x7f8d7c0872d0 0x7f8d7c05dac0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6808/56642032,v1:172.21.15.178:6810/56642032] conn(0x55b67e803190 0x55b67e803fa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d68069f00 0x7f8d6806c3c0 secure :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f8d74055040 tx=0x7f8d7404f000 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 --2- 172.21.15.6:0/600782961 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x55b67e74ab90 0x55b67e81ae30 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.169+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 >> 172.21.15.6:0/600782961 conn(0x55b67e7485b0 msgr2=0x55b67e816420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.170+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 shutdown_connections 2024-03-06T07:35:05.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.170+0000 7f8d8e0d3f40 1 -- 172.21.15.6:0/600782961 wait complete. 2024-03-06T07:35:05.177 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:35:05.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:05.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.391+0000 7fd213000640 1 Processor -- start 2024-03-06T07:35:05.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.391+0000 7fd213000640 1 -- start start 2024-03-06T07:35:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd213000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd20c0cd060 0x7fd20c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd213000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c15e810 0x7fd20c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd213000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd20c15ed10 con 0x7fd20c1552b0 2024-03-06T07:35:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd213000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20c15eff0 con 0x7fd20c0cd060 2024-03-06T07:35:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd213000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c15f2d0 con 0x7fd20c15e810 2024-03-06T07:35:05.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd211c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd20c1552b0 0x7fd20c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46380/0 (socket says 172.21.15.6:46380) 2024-03-06T07:35:05.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.393+0000 7fd211c00640 1 -- 172.21.15.6:0/720498522 learned_addr learned my addr 172.21.15.6:0/720498522 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd211200640 1 --2- 172.21.15.6:0/720498522 >> v2:172.21.15.178:3300/0 conn(0x7fd20c0cd060 0x7fd20c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd212600640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c15e810 0x7fd20c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768955924 0 0) 0x7fd20c15ed10 con 0x7fd20c1552b0 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1f4003520 con 0x7fd20c1552b0 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23871400 0 0) 0x7fd1f4003520 con 0x7fd20c1552b0 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c15e810 msgr2=0x7fd20c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c15e810 0x7fd20c0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 >> v2:172.21.15.178:3300/0 conn(0x7fd20c0cd060 msgr2=0x7fd20c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 --2- 172.21.15.6:0/720498522 >> v2:172.21.15.178:3300/0 conn(0x7fd20c0cd060 0x7fd20c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20c15f5b0 con 0x7fd20c1552b0 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd200001690 con 0x7fd20c1552b0 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 >> v1:172.21.15.6:6789/0 conn(0x7fd20c1552b0 legacy=0x7fd20c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1f40039b0 0x7fd1f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1f4004800 0x7fd1f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1f40051c0 0x7fd1f400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c15eff0 con 0x7fd1f40051c0 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c15f2d0 con 0x7fd1f40039b0 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f400e1f0 con 0x7fd1f4004800 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.394+0000 7fd212600640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c15e810 0x7fd20c0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd212600640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1f40051c0 0x7fd1f400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd211200640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1f4004800 0x7fd1f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd211c00640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1f40039b0 0x7fd1f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd212600640 1 -- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1f4004800 msgr2=0x7fd1f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd212600640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd1f4004800 0x7fd1f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd212600640 1 -- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1f40039b0 msgr2=0x7fd1f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd212600640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1f40039b0 0x7fd1f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd212600640 1 -- 172.21.15.6:0/720498522 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f4003520 con 0x7fd1f40051c0 2024-03-06T07:35:05.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.395+0000 7fd211c00640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1f40039b0 0x7fd1f4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.396+0000 7fd212600640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1f40051c0 0x7fd1f400dbd0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fd20c15ebf0 tx=0x7fd1f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6985fa2c0fed69b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd204032070 con 0x7fd1f40051c0 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd204030070 con 0x7fd1f40051c0 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd20be00640 1 -- 172.21.15.6:0/720498522 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd204009b50 con 0x7fd1f40051c0 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd213000640 1 -- 172.21.15.6:0/720498522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1f40051c0 msgr2=0x7fd1f400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd213000640 1 --2- 172.21.15.6:0/720498522 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1f40051c0 0x7fd1f400dbd0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fd20c15ebf0 tx=0x7fd1f4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd212600640 1 -- 172.21.15.6:0/720498522 reap_dead start 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd213000640 1 -- 172.21.15.6:0/720498522 shutdown_connections 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd213000640 1 -- 172.21.15.6:0/720498522 >> 172.21.15.6:0/720498522 conn(0x7fd20c06b3d0 msgr2=0x7fd20c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd213000640 1 -- 172.21.15.6:0/720498522 shutdown_connections 2024-03-06T07:35:05.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.397+0000 7fd213000640 1 -- 172.21.15.6:0/720498522 wait complete. 2024-03-06T07:35:05.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.399+0000 7fd213000640 1 Processor -- start 2024-03-06T07:35:05.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.400+0000 7fd213000640 1 -- start start 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd213000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 0x7fd20c14dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd213000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c14a930 0x7fd20c150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd213000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd20c151170 0x7fd20c0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd213000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c06dd30 con 0x7fd20c14a550 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd213000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c0b3760 con 0x7fd20c151170 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd213000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c06deb0 con 0x7fd20c14a930 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 0x7fd20c14dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 0x7fd20c14dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59708/0 (socket says 172.21.15.6:59708) 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 -- 172.21.15.6:0/3318284630 learned_addr learned my addr 172.21.15.6:0/3318284630 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd212600640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd20c151170 0x7fd20c0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211200640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c14a930 0x7fd20c150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 -- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c14a930 msgr2=0x7fd20c150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c14a930 0x7fd20c150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 -- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd20c151170 msgr2=0x7fd20c0af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.401+0000 7fd211c00640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd20c151170 0x7fd20c0af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd211c00640 1 -- 172.21.15.6:0/3318284630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20c14baf0 con 0x7fd20c14a550 2024-03-06T07:35:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd211c00640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 0x7fd20c14dcf0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd20000db10 tx=0x7fd200001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.405 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200019070 con 0x7fd20c14a550 2024-03-06T07:35:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd200004cb0 con 0x7fd20c14a550 2024-03-06T07:35:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20c14e5d0 con 0x7fd20c14a550 2024-03-06T07:35:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200004e60 con 0x7fd20c14a550 2024-03-06T07:35:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.402+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20c15a3f0 con 0x7fd20c14a550 2024-03-06T07:35:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.404+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20c04f330 con 0x7fd20c14a550 2024-03-06T07:35:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.405+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd200007310 con 0x7fd20c14a550 2024-03-06T07:35:05.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.406+0000 7fd209600640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd1ec06b500 0x7fd1ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.406+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd200089d30 con 0x7fd20c14a550 2024-03-06T07:35:05.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.406+0000 7fd211200640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd1ec06b500 0x7fd1ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:05.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.408+0000 7fd211200640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd1ec06b500 0x7fd1ec06d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd20c0b14b0 tx=0x7fd1f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:05.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.411+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd200058a80 con 0x7fd20c14a550 2024-03-06T07:35:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.800+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd20c0cecc0 con 0x7fd20c14a550 2024-03-06T07:35:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.801+0000 7fd209600640 1 -- 172.21.15.6:0/3318284630 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fd200058380 con 0x7fd20c14a550 2024-03-06T07:35:05.803 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:05.803 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.803+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd1ec06b500 msgr2=0x7fd1ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.803+0000 7fd213000640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd1ec06b500 0x7fd1ec06d9c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd20c0b14b0 tx=0x7fd1f8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 msgr2=0x7fd20c14dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 0x7fd20c14dcf0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fd20000db10 tx=0x7fd200001720 comp rx=0 tx=0).stop 2024-03-06T07:35:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 shutdown_connections 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd1ec06b500 0x7fd1ec06d9c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd20c151170 0x7fd20c0af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd20c14a930 0x7fd20c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 --2- 172.21.15.6:0/3318284630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd20c14a550 0x7fd20c14dcf0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 >> 172.21.15.6:0/3318284630 conn(0x7fd20c06b3d0 msgr2=0x7fd20c070bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 shutdown_connections 2024-03-06T07:35:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:05.804+0000 7fd213000640 1 -- 172.21.15.6:0/3318284630 wait complete. 2024-03-06T07:35:05.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:35:06.056 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:35:06.056 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:35:06.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:06.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.267+0000 7fcb97200640 1 Processor -- start 2024-03-06T07:35:06.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.268+0000 7fcb97200640 1 -- start start 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.269+0000 7fcb97200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcb9015bc80 0x7fcb9006ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.269+0000 7fcb97200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb9006ca90 0x7fcb9006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.269+0000 7fcb97200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9006d7f0 con 0x7fcb9006bf10 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.269+0000 7fcb97200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb900b60e0 con 0x7fcb9006ca90 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.269+0000 7fcb97200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb900b57f0 con 0x7fcb9015bc80 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb9006ca90 0x7fcb9006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb9006ca90 0x7fcb9006cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59722/0 (socket says 172.21.15.6:59722) 2024-03-06T07:35:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 -- 172.21.15.6:0/1954744861 learned_addr learned my addr 172.21.15.6:0/1954744861 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb8fe00640 1 --2- 172.21.15.6:0/1954744861 >> v2:172.21.15.178:3300/0 conn(0x7fcb9015bc80 0x7fcb9006ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 -- 172.21.15.6:0/1954744861 >> v2:172.21.15.178:3300/0 conn(0x7fcb9015bc80 msgr2=0x7fcb9006ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 --2- 172.21.15.6:0/1954744861 >> v2:172.21.15.178:3300/0 conn(0x7fcb9015bc80 0x7fcb9006ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 -- 172.21.15.6:0/1954744861 >> v1:172.21.15.178:6789/0 conn(0x7fcb9006bf10 legacy=0x7fcb900b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:06.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.270+0000 7fcb87e00640 1 -- 172.21.15.6:0/1954744861 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9006d510 con 0x7fcb9006ca90 2024-03-06T07:35:06.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.271+0000 7fcb87e00640 1 --2- 172.21.15.6:0/1954744861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006ca90 0x7fcb9006cef0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fcb90072b80 tx=0x7fcb7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca5aa1d8df2f2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb8f400640 1 -- 172.21.15.6:0/1954744861 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78035020 con 0x7fcb9006ca90 2024-03-06T07:35:06.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb8f400640 1 -- 172.21.15.6:0/1954744861 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78031870 con 0x7fcb9006ca90 2024-03-06T07:35:06.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb8f400640 1 -- 172.21.15.6:0/1954744861 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78031c40 con 0x7fcb9006ca90 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb97200640 1 -- 172.21.15.6:0/1954744861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006ca90 msgr2=0x7fcb9006cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb97200640 1 --2- 172.21.15.6:0/1954744861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006ca90 0x7fcb9006cef0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fcb90072b80 tx=0x7fcb7802b910 comp rx=0 tx=0).stop 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb97200640 1 -- 172.21.15.6:0/1954744861 shutdown_connections 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb97200640 1 --2- 172.21.15.6:0/1954744861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006ca90 0x7fcb9006cef0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb97200640 1 --2- 172.21.15.6:0/1954744861 >> v2:172.21.15.178:3300/0 conn(0x7fcb9015bc80 0x7fcb9006ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.272+0000 7fcb97200640 1 -- 172.21.15.6:0/1954744861 >> 172.21.15.6:0/1954744861 conn(0x7fcb9009aa10 msgr2=0x7fcb9009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.273+0000 7fcb97200640 1 -- 172.21.15.6:0/1954744861 shutdown_connections 2024-03-06T07:35:06.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.273+0000 7fcb97200640 1 -- 172.21.15.6:0/1954744861 wait complete. 2024-03-06T07:35:06.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.275+0000 7fcb97200640 1 Processor -- start 2024-03-06T07:35:06.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.276+0000 7fcb97200640 1 -- start start 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.277+0000 7fcb97200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 0x7fcb901f5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.277+0000 7fcb97200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb9006ca90 0x7fcb9015c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.277+0000 7fcb97200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9015bc80 0x7fcb900b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.277+0000 7fcb97200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9006f4a0 con 0x7fcb9006bf10 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.277+0000 7fcb97200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb900b5d70 con 0x7fcb9015bc80 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.277+0000 7fcb97200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9015f630 con 0x7fcb9006ca90 2024-03-06T07:35:06.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 0x7fcb901f5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 0x7fcb901f5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59732/0 (socket says 172.21.15.6:59732) 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 -- 172.21.15.6:0/2085976520 learned_addr learned my addr 172.21.15.6:0/2085976520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb95200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9015bc80 0x7fcb900b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 -- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb9006ca90 msgr2=0x7fcb9015c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb9006ca90 0x7fcb9015c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 -- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9015bc80 msgr2=0x7fcb900b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9015bc80 0x7fcb900b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 -- 172.21.15.6:0/2085976520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9015c5a0 con 0x7fcb9006bf10 2024-03-06T07:35:06.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb95200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9015bc80 0x7fcb900b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.278+0000 7fcb8fe00640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 0x7fcb901f5120 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c7f0 tx=0x7fcb8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.279+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80015070 con 0x7fcb9006bf10 2024-03-06T07:35:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.279+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb80004220 con 0x7fcb9006bf10 2024-03-06T07:35:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.279+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9015c870 con 0x7fcb9006bf10 2024-03-06T07:35:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.279+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb80003610 con 0x7fcb9006bf10 2024-03-06T07:35:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.279+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9015ce30 con 0x7fcb9006bf10 2024-03-06T07:35:06.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.280+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9004f3a0 con 0x7fcb9006bf10 2024-03-06T07:35:06.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.281+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb800039c0 con 0x7fcb9006bf10 2024-03-06T07:35:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.282+0000 7fcb8cc00640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.282+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fcb8008a2c0 con 0x7fcb9006bf10 2024-03-06T07:35:06.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.282+0000 7fcb87e00640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:06.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.284+0000 7fcb87e00640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcb78036000 tx=0x7fcb78030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:06.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.286+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb80059010 con 0x7fcb9006bf10 2024-03-06T07:35:06.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.525+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb9006d230 con 0x7fcb9006bf10 2024-03-06T07:35:06.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.526+0000 7fcb8cc00640 1 -- 172.21.15.6:0/2085976520 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fcb80058910 con 0x7fcb9006bf10 2024-03-06T07:35:06.528 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:06.528 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:06.528 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb6c06b500 msgr2=0x7fcb6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcb78036000 tx=0x7fcb78030000 comp rx=0 tx=0).stop 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 msgr2=0x7fcb901f5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 0x7fcb901f5120 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000c7f0 tx=0x7fcb8000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 shutdown_connections 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcb6c06b500 0x7fcb6c06d9c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb9015bc80 0x7fcb900b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb9006ca90 0x7fcb9015c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 --2- 172.21.15.6:0/2085976520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb9006bf10 0x7fcb901f5120 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 >> 172.21.15.6:0/2085976520 conn(0x7fcb9009aa10 msgr2=0x7fcb90161140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 shutdown_connections 2024-03-06T07:35:06.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:06.528+0000 7fcb97200640 1 -- 172.21.15.6:0/2085976520 wait complete. 2024-03-06T07:35:06.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:06.787 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:35:06.787 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:35:06.788 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:06.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:06.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:07.043 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:07.043 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:07.043 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:07.099 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:07.099 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:07.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:07.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.370+0000 7f3627200640 1 Processor -- start 2024-03-06T07:35:07.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.371+0000 7f3627200640 1 -- start start 2024-03-06T07:35:07.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.372+0000 7f3627200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f36280cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.372+0000 7f3627200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f36280d0e50 0x7f36280b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.372+0000 7f3627200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36280b3710 con 0x7f36280d4a10 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.372+0000 7f3627200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36280b09e0 con 0x7f36280d0e50 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.372+0000 7f3627200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36280b39f0 con 0x7f362815e0b0 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f36280d0e50 0x7f36280b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f36280d0e50 0x7f36280b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59748/0 (socket says 172.21.15.6:59748) 2024-03-06T07:35:07.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 -- 172.21.15.6:0/2018925722 learned_addr learned my addr 172.21.15.6:0/2018925722 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:07.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3625e00640 1 --2- 172.21.15.6:0/2018925722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f36280cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 -- 172.21.15.6:0/2018925722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 msgr2=0x7f36280cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:07.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 --2- 172.21.15.6:0/2018925722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f36280cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 -- 172.21.15.6:0/2018925722 >> v1:172.21.15.178:6789/0 conn(0x7f36280d4a10 legacy=0x7f36280d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:07.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.373+0000 7f3626800640 1 -- 172.21.15.6:0/2018925722 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36280b3cd0 con 0x7f36280d0e50 2024-03-06T07:35:07.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.374+0000 7f3626800640 1 --2- 172.21.15.6:0/2018925722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280b44d0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3620009e50 tx=0x7f362002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30f7e29d00d0d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.375+0000 7f3625400640 1 -- 172.21.15.6:0/2018925722 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362002ca70 con 0x7f36280d0e50 2024-03-06T07:35:07.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.375+0000 7f3625400640 1 -- 172.21.15.6:0/2018925722 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362002cc10 con 0x7f36280d0e50 2024-03-06T07:35:07.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.375+0000 7f3625400640 1 -- 172.21.15.6:0/2018925722 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36200343f0 con 0x7f36280d0e50 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.375+0000 7f3627200640 1 -- 172.21.15.6:0/2018925722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 msgr2=0x7f36280b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.375+0000 7f3627200640 1 --2- 172.21.15.6:0/2018925722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280b44d0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f3620009e50 tx=0x7f362002bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.376+0000 7f3627200640 1 -- 172.21.15.6:0/2018925722 shutdown_connections 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.376+0000 7f3627200640 1 --2- 172.21.15.6:0/2018925722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280b44d0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.376+0000 7f3627200640 1 --2- 172.21.15.6:0/2018925722 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f36280cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.376+0000 7f3627200640 1 -- 172.21.15.6:0/2018925722 >> 172.21.15.6:0/2018925722 conn(0x7f36280d3d30 msgr2=0x7f36280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.376+0000 7f3627200640 1 -- 172.21.15.6:0/2018925722 shutdown_connections 2024-03-06T07:35:07.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.376+0000 7f3627200640 1 -- 172.21.15.6:0/2018925722 wait complete. 2024-03-06T07:35:07.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.377+0000 7f3627200640 1 Processor -- start 2024-03-06T07:35:07.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.378+0000 7f3627200640 1 -- start start 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3627200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3627200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36280d4a10 0x7f36280cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3627200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f362814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3627200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36280cfc60 con 0x7f36280d0e50 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3627200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362815e490 con 0x7f36280d4a10 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3627200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36280cf850 con 0x7f362815e0b0 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280cdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59750/0 (socket says 172.21.15.6:59750) 2024-03-06T07:35:07.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 -- 172.21.15.6:0/1126000997 learned_addr learned my addr 172.21.15.6:0/1126000997 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 -- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 msgr2=0x7f362814f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f362814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 -- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36280d4a10 msgr2=0x7f36280cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36280d4a10 0x7f36280cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.379+0000 7f3625e00640 1 -- 172.21.15.6:0/1126000997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36280cca00 con 0x7f36280d0e50 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.380+0000 7f3625e00640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280cdaa0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f361800d5f0 tx=0x7f361800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.380+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36180125f0 con 0x7f36280d0e50 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.380+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618019070 con 0x7f36280d0e50 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.380+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36180129d0 con 0x7f36280d0e50 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.380+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36280ccce0 con 0x7f36280d0e50 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.380+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36280cd2a0 con 0x7f36280d0e50 2024-03-06T07:35:07.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.382+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362804f330 con 0x7f36280d0e50 2024-03-06T07:35:07.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.382+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3618012b70 con 0x7f36280d0e50 2024-03-06T07:35:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.383+0000 7f361ea00640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f360406b4d0 0x7f360406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.383+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f361808a690 con 0x7f36280d0e50 2024-03-06T07:35:07.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.384+0000 7f361d400640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f360406b4d0 0x7f360406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.385+0000 7f361d400640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f360406b4d0 0x7f360406d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f36280d4e30 tx=0x7f36100079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.392+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3618059480 con 0x7f36280d0e50 2024-03-06T07:35:07.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.601+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36280c6fa0 con 0x7f36280d0e50 2024-03-06T07:35:07.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.604+0000 7f361ea00640 1 -- 172.21.15.6:0/1126000997 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16775 (secure 0 0 0) 0x7f3618058d80 con 0x7f36280d0e50 2024-03-06T07:35:07.607 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:07.607 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":114,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:34:11.424305+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6833/4114489444":"2024-03-07T07:34:04.377501+0000","172.21.15.6:6832/4114489444":"2024-03-07T07:34:04.377501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:07.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f360406b4d0 msgr2=0x7f360406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f360406b4d0 0x7f360406d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f36280d4e30 tx=0x7f36100079b0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 msgr2=0x7f36280cdaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280cdaa0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f361800d5f0 tx=0x7f361800dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 shutdown_connections 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f360406b4d0 0x7f360406d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36280d4a10 0x7f36280cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f362815e0b0 0x7f362814f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 --2- 172.21.15.6:0/1126000997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36280d0e50 0x7f36280cdaa0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 >> 172.21.15.6:0/1126000997 conn(0x7f36280d3d30 msgr2=0x7f3628073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 shutdown_connections 2024-03-06T07:35:07.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.607+0000 7f3627200640 1 -- 172.21.15.6:0/1126000997 wait complete. 2024-03-06T07:35:07.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:46.337716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:47.450154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:07.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:07.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.843+0000 7f9c0ca00640 1 Processor -- start 2024-03-06T07:35:07.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.843+0000 7f9c0ca00640 1 -- start start 2024-03-06T07:35:07.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.845+0000 7f9c0ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c080afff0 0x7f9c0806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9c0ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c0815f0a0 0x7f9c0806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9c0ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9c0806c350 0x7f9c0806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9c0ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0806bde0 con 0x7f9c0815f0a0 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9c0ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0815ec10 con 0x7f9c0806c350 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9c0ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080cf650 con 0x7f9c080afff0 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c0815f0a0 0x7f9c0806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c0815f0a0 0x7f9c0806b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59760/0 (socket says 172.21.15.6:59760) 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 -- 172.21.15.6:0/231979721 learned_addr learned my addr 172.21.15.6:0/231979721 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9c07200640 1 --2- 172.21.15.6:0/231979721 >> v2:172.21.15.178:3300/0 conn(0x7f9c0806c350 0x7f9c0806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 -- 172.21.15.6:0/231979721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c080afff0 msgr2=0x7f9c0806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 --2- 172.21.15.6:0/231979721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c080afff0 0x7f9c0806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 -- 172.21.15.6:0/231979721 >> v2:172.21.15.178:3300/0 conn(0x7f9c0806c350 msgr2=0x7f9c0806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 --2- 172.21.15.6:0/231979721 >> v2:172.21.15.178:3300/0 conn(0x7f9c0806c350 0x7f9c0806c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.846+0000 7f9bffe00640 1 -- 172.21.15.6:0/231979721 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0806bb00 con 0x7f9c0815f0a0 2024-03-06T07:35:07.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.848+0000 7f9bffe00640 1 --2- 172.21.15.6:0/231979721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c0815f0a0 0x7f9c0806b4f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f9c0806d2e0 tx=0x7f9bf002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229d7a8a973af257 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.852+0000 7f9c05e00640 1 -- 172.21.15.6:0/231979721 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0035020 con 0x7f9c0815f0a0 2024-03-06T07:35:07.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.852+0000 7f9c05e00640 1 -- 172.21.15.6:0/231979721 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0031880 con 0x7f9c0815f0a0 2024-03-06T07:35:07.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.852+0000 7f9c05e00640 1 -- 172.21.15.6:0/231979721 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0031c50 con 0x7f9c0815f0a0 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 -- 172.21.15.6:0/231979721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c0815f0a0 msgr2=0x7f9c0806b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/231979721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c0815f0a0 0x7f9c0806b4f0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f9c0806d2e0 tx=0x7f9bf002b920 comp rx=0 tx=0).stop 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 -- 172.21.15.6:0/231979721 shutdown_connections 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/231979721 >> v2:172.21.15.178:3300/0 conn(0x7f9c0806c350 0x7f9c0806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/231979721 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c0815f0a0 0x7f9c0806b4f0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/231979721 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c080afff0 0x7f9c0806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 -- 172.21.15.6:0/231979721 >> 172.21.15.6:0/231979721 conn(0x7f9c080b2940 msgr2=0x7f9c080b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:07.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 -- 172.21.15.6:0/231979721 shutdown_connections 2024-03-06T07:35:07.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.853+0000 7f9c0ca00640 1 -- 172.21.15.6:0/231979721 wait complete. 2024-03-06T07:35:07.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.856+0000 7f9c0ca00640 1 Processor -- start 2024-03-06T07:35:07.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.856+0000 7f9c0ca00640 1 -- start start 2024-03-06T07:35:07.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9c0ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c0806c350 0x7f9c0814a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9c0ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 0x7f9c080d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9c0ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c0815f0a0 0x7f9c080d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9c0ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080b03d0 con 0x7f9c080afff0 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9c0ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0815e8a0 con 0x7f9c0815f0a0 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9c0ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080b0550 con 0x7f9c0806c350 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.857+0000 7f9bffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 0x7f9c080d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 0x7f9c080d0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59762/0 (socket says 172.21.15.6:59762) 2024-03-06T07:35:07.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 -- 172.21.15.6:0/2304923397 learned_addr learned my addr 172.21.15.6:0/2304923397 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9c06800640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c0806c350 0x7f9c0814a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9c07200640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c0815f0a0 0x7f9c080d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 -- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c0806c350 msgr2=0x7f9c0814a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c0806c350 0x7f9c0814a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 -- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c0815f0a0 msgr2=0x7f9c080d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:07.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c0815f0a0 0x7f9c080d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.858+0000 7f9bffe00640 1 -- 172.21.15.6:0/2304923397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0814aa10 con 0x7f9c080afff0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.859+0000 7f9bffe00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 0x7f9c080d0a10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f9bf002b960 tx=0x7f9bf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.859+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0040070 con 0x7f9c080afff0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.859+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf003a420 con 0x7f9c080afff0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.859+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c081590b0 con 0x7f9c080afff0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.859+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0035040 con 0x7f9c080afff0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.859+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c080b4370 con 0x7f9c080afff0 2024-03-06T07:35:07.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.860+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0804f330 con 0x7f9c080afff0 2024-03-06T07:35:07.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.861+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bf004b060 con 0x7f9c080afff0 2024-03-06T07:35:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.862+0000 7f9bfea00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9be406b4d0 0x7f9be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.862+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9bf00ac240 con 0x7f9c080afff0 2024-03-06T07:35:07.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.862+0000 7f9c06800640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9be406b4d0 0x7f9be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:07.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.863+0000 7f9c06800640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9be406b4d0 0x7f9be406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009890 tx=0x7f9bf8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:07.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:07.866+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c0814aa10 con 0x7f9c080afff0 2024-03-06T07:35:08.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.073+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c0806e6d0 con 0x7f9c080afff0 2024-03-06T07:35:08.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.074+0000 7f9bfea00640 1 -- 172.21.15.6:0/2304923397 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bf007b020 con 0x7f9c080afff0 2024-03-06T07:35:08.076 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:35:08.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9be406b4d0 msgr2=0x7f9be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9be406b4d0 0x7f9be406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009890 tx=0x7f9bf8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:08.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 msgr2=0x7f9c080d0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 0x7f9c080d0a10 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f9bf002b960 tx=0x7f9bf00046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 shutdown_connections 2024-03-06T07:35:08.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9be406b4d0 0x7f9be406d990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9c0815f0a0 0x7f9c080d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c080afff0 0x7f9c080d0a10 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 --2- 172.21.15.6:0/2304923397 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9c0806c350 0x7f9c0814a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 >> 172.21.15.6:0/2304923397 conn(0x7f9c080b2940 msgr2=0x7f9c080b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 shutdown_connections 2024-03-06T07:35:08.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.077+0000 7f9c0ca00640 1 -- 172.21.15.6:0/2304923397 wait complete. 2024-03-06T07:35:08.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.343+0000 7fa5c5a00640 1 Processor -- start 2024-03-06T07:35:08.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.343+0000 7fa5c5a00640 1 -- start start 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.344+0000 7fa5c5a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa5c00d4890 0x7fa5c00d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5c5a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5c00d2320 0x7fa5c00b1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5c5a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00b2c00 con 0x7fa5c0161700 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5c5a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00b56e0 con 0x7fa5c00d2320 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5c5a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00d2750 con 0x7fa5c00d4890 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5bfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa5c0161700 0x7fa5c00b4fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46392/0 (socket says 172.21.15.6:46392) 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5bfe00640 1 -- 172.21.15.6:0/617780789 learned_addr learned my addr 172.21.15.6:0/617780789 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:08.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5c5000640 1 --2- 172.21.15.6:0/617780789 >> v2:172.21.15.6:3300/0 conn(0x7fa5c00d2320 0x7fa5c00b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.345+0000 7fa5bf400640 1 --2- 172.21.15.6:0/617780789 >> v2:172.21.15.178:3300/0 conn(0x7fa5c00d4890 0x7fa5c00d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.346+0000 7fa5bea00640 1 -- 172.21.15.6:0/617780789 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210353984 0 0) 0x7fa5c00b2c00 con 0x7fa5c0161700 2024-03-06T07:35:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.346+0000 7fa5bea00640 1 -- 172.21.15.6:0/617780789 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4003520 con 0x7fa5c0161700 2024-03-06T07:35:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.346+0000 7fa5c5000640 1 -- 172.21.15.6:0/617780789 >> v2:172.21.15.178:3300/0 conn(0x7fa5c00d4890 msgr2=0x7fa5c00d1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:08.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.346+0000 7fa5c5000640 1 --2- 172.21.15.6:0/617780789 >> v2:172.21.15.178:3300/0 conn(0x7fa5c00d4890 0x7fa5c00d1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.346+0000 7fa5c5000640 1 -- 172.21.15.6:0/617780789 >> v1:172.21.15.6:6789/0 conn(0x7fa5c0161700 legacy=0x7fa5c00b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.346+0000 7fa5c5000640 1 -- 172.21.15.6:0/617780789 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0161ae0 con 0x7fa5c00d2320 2024-03-06T07:35:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.347+0000 7fa5c5000640 1 --2- 172.21.15.6:0/617780789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c00d2320 0x7fa5c00b1220 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa5c008c350 tx=0x7fa5b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf00aa870d7772c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.347+0000 7fa5bea00640 1 -- 172.21.15.6:0/617780789 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035020 con 0x7fa5c00d2320 2024-03-06T07:35:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.347+0000 7fa5bea00640 1 -- 172.21.15.6:0/617780789 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8031830 con 0x7fa5c00d2320 2024-03-06T07:35:08.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.347+0000 7fa5bea00640 1 -- 172.21.15.6:0/617780789 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8031be0 con 0x7fa5c00d2320 2024-03-06T07:35:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 -- 172.21.15.6:0/617780789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c00d2320 msgr2=0x7fa5c00b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/617780789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c00d2320 0x7fa5c00b1220 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa5c008c350 tx=0x7fa5b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 -- 172.21.15.6:0/617780789 shutdown_connections 2024-03-06T07:35:08.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/617780789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c00d2320 0x7fa5c00b1220 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/617780789 >> v2:172.21.15.178:3300/0 conn(0x7fa5c00d4890 0x7fa5c00d1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 -- 172.21.15.6:0/617780789 >> 172.21.15.6:0/617780789 conn(0x7fa5c00d3d70 msgr2=0x7fa5c00d4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 -- 172.21.15.6:0/617780789 shutdown_connections 2024-03-06T07:35:08.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.348+0000 7fa5c5a00640 1 -- 172.21.15.6:0/617780789 wait complete. 2024-03-06T07:35:08.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.350+0000 7fa5c5a00640 1 Processor -- start 2024-03-06T07:35:08.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.350+0000 7fa5c5a00640 1 -- start start 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5c00d2320 0x7fa5c00cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5c00d4890 0x7fa5c00cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 0x7fa5c00cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b0a20 con 0x7fa5c0161700 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b3180 con 0x7fa5c00d2320 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b0d00 con 0x7fa5c00d4890 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 0x7fa5c00cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 0x7fa5c00cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59778/0 (socket says 172.21.15.6:59778) 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.351+0000 7fa5c5000640 1 -- 172.21.15.6:0/3333902990 learned_addr learned my addr 172.21.15.6:0/3333902990 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5bf400640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5c00d4890 0x7fa5c00cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5bfe00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5c00d2320 0x7fa5c00cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5c5000640 1 -- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5c00d4890 msgr2=0x7fa5c00cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5c5000640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5c00d4890 0x7fa5c00cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5c5000640 1 -- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5c00d2320 msgr2=0x7fa5c00cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5c5000640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5c00d2320 0x7fa5c00cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5c5000640 1 -- 172.21.15.6:0/3333902990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00cb420 con 0x7fa5c0161700 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.352+0000 7fa5c5000640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 0x7fa5c00cb890 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009890 tx=0x7fa5b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.353+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8035040 con 0x7fa5c0161700 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.353+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b803b840 con 0x7fa5c0161700 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.353+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b803bc10 con 0x7fa5c0161700 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.353+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c01f3b50 con 0x7fa5c0161700 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.353+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c01f4110 con 0x7fa5c0161700 2024-03-06T07:35:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.354+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c004f330 con 0x7fa5c0161700 2024-03-06T07:35:08.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.355+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5b8038020 con 0x7fa5c0161700 2024-03-06T07:35:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.357+0000 7fa5b7e00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa59c06b4d0 0x7fa59c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.357+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fa5b80abf30 con 0x7fa5c0161700 2024-03-06T07:35:08.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.357+0000 7fa5bfe00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa59c06b4d0 0x7fa59c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.359+0000 7fa5bfe00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa59c06b4d0 0x7fa59c06d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0005c00 tx=0x7fa5b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.361+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5b807adb0 con 0x7fa5c0161700 2024-03-06T07:35:08.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.591+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa5c00b21d0 con 0x7fa5c0161700 2024-03-06T07:35:08.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.592+0000 7fa5b7e00640 1 -- 172.21.15.6:0/3333902990 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa5b807a6b0 con 0x7fa5c0161700 2024-03-06T07:35:08.595 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:35:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa59c06b4d0 msgr2=0x7fa59c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa59c06b4d0 0x7fa59c06d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0005c00 tx=0x7fa5b0009040 comp rx=0 tx=0).stop 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 msgr2=0x7fa5c00cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 0x7fa5c00cb890 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009890 tx=0x7fa5b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 shutdown_connections 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa59c06b4d0 0x7fa59c06d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5c0161700 0x7fa5c00cb890 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa5c00d4890 0x7fa5c00cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 --2- 172.21.15.6:0/3333902990 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa5c00d2320 0x7fa5c00cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 >> 172.21.15.6:0/3333902990 conn(0x7fa5c00d3d70 msgr2=0x7fa5c006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 shutdown_connections 2024-03-06T07:35:08.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.597+0000 7fa5c5a00640 1 -- 172.21.15.6:0/3333902990 wait complete. 2024-03-06T07:35:08.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:08.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.889+0000 7fe6a5a00640 1 Processor -- start 2024-03-06T07:35:08.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.889+0000 7fe6a5a00640 1 -- start start 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.890+0000 7fe6a5a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe6a015e0c0 0x7fe6a0072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.890+0000 7fe6a5a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe6a006b440 0x7fe6a00cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe6a5a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a00b5800 con 0x7fe6a006afb0 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe6a5a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a006dd30 con 0x7fe6a006b440 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe6a5a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a00b5aa0 con 0x7fe6a015e0c0 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe69f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6a006afb0 0x7fe6a00cd600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46404/0 (socket says 172.21.15.6:46404) 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe69f400640 1 -- 172.21.15.6:0/319225976 learned_addr learned my addr 172.21.15.6:0/319225976 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:08.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe6a5000640 1 --2- 172.21.15.6:0/319225976 >> v2:172.21.15.6:3300/0 conn(0x7fe6a006b440 0x7fe6a00cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe69fe00640 1 --2- 172.21.15.6:0/319225976 >> v2:172.21.15.178:3300/0 conn(0x7fe6a015e0c0 0x7fe6a0072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe69ea00640 1 -- 172.21.15.6:0/319225976 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658821724 0 0) 0x7fe6a00b5800 con 0x7fe6a006afb0 2024-03-06T07:35:08.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.891+0000 7fe69ea00640 1 -- 172.21.15.6:0/319225976 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe684003520 con 0x7fe6a006afb0 2024-03-06T07:35:08.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.892+0000 7fe6a5000640 1 -- 172.21.15.6:0/319225976 >> v2:172.21.15.178:3300/0 conn(0x7fe6a015e0c0 msgr2=0x7fe6a0072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:08.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.892+0000 7fe6a5000640 1 --2- 172.21.15.6:0/319225976 >> v2:172.21.15.178:3300/0 conn(0x7fe6a015e0c0 0x7fe6a0072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.892+0000 7fe6a5000640 1 -- 172.21.15.6:0/319225976 >> v1:172.21.15.6:6789/0 conn(0x7fe6a006afb0 legacy=0x7fe6a00cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.892+0000 7fe6a5000640 1 -- 172.21.15.6:0/319225976 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a00b5d80 con 0x7fe6a006b440 2024-03-06T07:35:08.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.893+0000 7fe6a5000640 1 --2- 172.21.15.6:0/319225976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a006b440 0x7fe6a00cdd10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe6a006d1f0 tx=0x7fe69802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19183c34a80eaf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.893+0000 7fe69ea00640 1 -- 172.21.15.6:0/319225976 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698039070 con 0x7fe6a006b440 2024-03-06T07:35:08.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.893+0000 7fe69ea00640 1 -- 172.21.15.6:0/319225976 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69802cc70 con 0x7fe6a006b440 2024-03-06T07:35:08.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.893+0000 7fe69ea00640 1 -- 172.21.15.6:0/319225976 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698034660 con 0x7fe6a006b440 2024-03-06T07:35:08.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 -- 172.21.15.6:0/319225976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a006b440 msgr2=0x7fe6a00cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/319225976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a006b440 0x7fe6a00cdd10 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fe6a006d1f0 tx=0x7fe69802bec0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 -- 172.21.15.6:0/319225976 shutdown_connections 2024-03-06T07:35:08.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/319225976 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a006b440 0x7fe6a00cdd10 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/319225976 >> v2:172.21.15.178:3300/0 conn(0x7fe6a015e0c0 0x7fe6a0072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 -- 172.21.15.6:0/319225976 >> 172.21.15.6:0/319225976 conn(0x7fe6a00d0c00 msgr2=0x7fe6a00d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:08.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 -- 172.21.15.6:0/319225976 shutdown_connections 2024-03-06T07:35:08.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.894+0000 7fe6a5a00640 1 -- 172.21.15.6:0/319225976 wait complete. 2024-03-06T07:35:08.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.897+0000 7fe6a5a00640 1 Processor -- start 2024-03-06T07:35:08.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.897+0000 7fe6a5a00640 1 -- start start 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6a006afb0 0x7fe6a00add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6a006b440 0x7fe6a014fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 0x7fe6a0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a006b850 con 0x7fe6a015e0c0 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a00cef60 con 0x7fe6a006afb0 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a00727b0 con 0x7fe6a006b440 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 0x7fe6a0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 0x7fe6a0159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59796/0 (socket says 172.21.15.6:59796) 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.899+0000 7fe6a5000640 1 -- 172.21.15.6:0/1439839119 learned_addr learned my addr 172.21.15.6:0/1439839119 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe69fe00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6a006afb0 0x7fe6a00add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe69f400640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6a006b440 0x7fe6a014fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe6a5000640 1 -- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6a006b440 msgr2=0x7fe6a014fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe6a5000640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6a006b440 0x7fe6a014fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe6a5000640 1 -- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6a006afb0 msgr2=0x7fe6a00add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe6a5000640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6a006afb0 0x7fe6a00add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:08.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe6a5000640 1 -- 172.21.15.6:0/1439839119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a015f4c0 con 0x7fe6a015e0c0 2024-03-06T07:35:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.900+0000 7fe6a5000640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 0x7fe6a0159b20 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe69802c4d0 tx=0x7fe698004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.901+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698039050 con 0x7fe6a015e0c0 2024-03-06T07:35:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.901+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe698045070 con 0x7fe6a015e0c0 2024-03-06T07:35:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.901+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698009450 con 0x7fe6a015e0c0 2024-03-06T07:35:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.901+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a015f760 con 0x7fe6a015e0c0 2024-03-06T07:35:08.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.901+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a015fd20 con 0x7fe6a015e0c0 2024-03-06T07:35:08.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.902+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a004f330 con 0x7fe6a015e0c0 2024-03-06T07:35:08.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.903+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe69803e850 con 0x7fe6a015e0c0 2024-03-06T07:35:08.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.904+0000 7fe697e00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe68006b4d0 0x7fe68006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:08.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.904+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe6980ad180 con 0x7fe6a015e0c0 2024-03-06T07:35:08.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.904+0000 7fe69fe00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe68006b4d0 0x7fe68006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:08.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.906+0000 7fe69fe00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe68006b4d0 0x7fe68006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe690009a30 tx=0x7fe690008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:08.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:08.908+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe69802ce20 con 0x7fe6a015e0c0 2024-03-06T07:35:09.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.226+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe6a006bc00 con 0x7fe6a015e0c0 2024-03-06T07:35:09.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.227+0000 7fe697e00640 1 -- 172.21.15.6:0/1439839119 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe69807bff0 con 0x7fe6a015e0c0 2024-03-06T07:35:09.229 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:35:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe68006b4d0 msgr2=0x7fe68006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe68006b4d0 0x7fe68006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe690009a30 tx=0x7fe690008040 comp rx=0 tx=0).stop 2024-03-06T07:35:09.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 msgr2=0x7fe6a0159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 0x7fe6a0159b20 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe69802c4d0 tx=0x7fe698004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 shutdown_connections 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe68006b4d0 0x7fe68006d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6a015e0c0 0x7fe6a0159b20 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe6a006b440 0x7fe6a014fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 --2- 172.21.15.6:0/1439839119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe6a006afb0 0x7fe6a00add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 >> 172.21.15.6:0/1439839119 conn(0x7fe6a00d0c00 msgr2=0x7fe6a00b4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 shutdown_connections 2024-03-06T07:35:09.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.229+0000 7fe6a5a00640 1 -- 172.21.15.6:0/1439839119 wait complete. 2024-03-06T07:35:09.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:35:09.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.459+0000 7f8ec2000640 1 Processor -- start 2024-03-06T07:35:09.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.460+0000 7f8ec2000640 1 -- start start 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.461+0000 7f8ec2000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8ebc15e200 0x7f8ebc0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.461+0000 7f8ec2000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ebc0afdf0 0x7f8ebc0af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.461+0000 7f8ec2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc06b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.461+0000 7f8ec2000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc06bec0 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.461+0000 7f8ec2000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ebc0b0670 con 0x7f8ebc15e200 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.461+0000 7f8ec2000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0ae380 con 0x7f8ebc06afb0 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ebc0afdf0 0x7f8ebc0af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8ebc0afdf0 0x7f8ebc0af140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59802/0 (socket says 172.21.15.6:59802) 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 -- 172.21.15.6:0/3762819846 learned_addr learned my addr 172.21.15.6:0/3762819846 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8ebb400640 1 --2- 172.21.15.6:0/3762819846 >> v2:172.21.15.178:3300/0 conn(0x7f8ebc15e200 0x7f8ebc0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8ebbe00640 1 --2- 172.21.15.6:0/3762819846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc06b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 -- 172.21.15.6:0/3762819846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 msgr2=0x7f8ebc06b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:09.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 --2- 172.21.15.6:0/3762819846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc06b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 -- 172.21.15.6:0/3762819846 >> v2:172.21.15.178:3300/0 conn(0x7f8ebc15e200 msgr2=0x7f8ebc0b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 --2- 172.21.15.6:0/3762819846 >> v2:172.21.15.178:3300/0 conn(0x7f8ebc15e200 0x7f8ebc0b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.462+0000 7f8eb3e00640 1 -- 172.21.15.6:0/3762819846 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc0ae660 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.463+0000 7f8eb3e00640 1 --2- 172.21.15.6:0/3762819846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc0af140 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc071cc0 tx=0x7f8ea402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4869ece5230fdae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.464+0000 7f8ebaa00640 1 -- 172.21.15.6:0/3762819846 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4035020 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.464+0000 7f8ebaa00640 1 -- 172.21.15.6:0/3762819846 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea402d040 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.464+0000 7f8ebaa00640 1 -- 172.21.15.6:0/3762819846 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4031a80 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.464+0000 7f8ec2000640 1 -- 172.21.15.6:0/3762819846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 msgr2=0x7f8ebc0af140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.464+0000 7f8ec2000640 1 --2- 172.21.15.6:0/3762819846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc0af140 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc071cc0 tx=0x7f8ea402b900 comp rx=0 tx=0).stop 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 -- 172.21.15.6:0/3762819846 shutdown_connections 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 --2- 172.21.15.6:0/3762819846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc06b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 --2- 172.21.15.6:0/3762819846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc0af140 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 --2- 172.21.15.6:0/3762819846 >> v2:172.21.15.178:3300/0 conn(0x7f8ebc15e200 0x7f8ebc0b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 -- 172.21.15.6:0/3762819846 >> 172.21.15.6:0/3762819846 conn(0x7f8ebc0b2ae0 msgr2=0x7f8ebc0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 -- 172.21.15.6:0/3762819846 shutdown_connections 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.465+0000 7f8ec2000640 1 -- 172.21.15.6:0/3762819846 wait complete. 2024-03-06T07:35:09.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.467+0000 7f8ec2000640 1 Processor -- start 2024-03-06T07:35:09.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.467+0000 7f8ec2000640 1 -- start start 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ec2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ec2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ec2000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ebc15e200 0x7f8ebc06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ec2000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0af680 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ec2000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc0aec20 con 0x7f8ebc15e200 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ec2000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ebc06dbb0 con 0x7f8ebc06afb0 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8eb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8eb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc06e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59808/0 (socket says 172.21.15.6:59808) 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8eb3e00640 1 -- 172.21.15.6:0/67498517 learned_addr learned my addr 172.21.15.6:0/67498517 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ebb400640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.468+0000 7f8ebbe00640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ebc15e200 0x7f8ebc06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3e00640 1 -- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 msgr2=0x7f8ebc147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3e00640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc147800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3e00640 1 -- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ebc15e200 msgr2=0x7f8ebc06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3e00640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ebc15e200 0x7f8ebc06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3e00640 1 -- 172.21.15.6:0/67498517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ebc153000 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8ebbe00640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ebc15e200 0x7f8ebc06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3e00640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc06e6f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097c0 tx=0x7f8ea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4040070 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea4039bf0 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4035040 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.469+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ebc1532e0 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.470+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ebc1538a0 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.471+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc04f3a0 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.472+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ea404b060 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.473+0000 7f8eb3400640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e9806b4d0 0x7f8e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.473+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8ea40abed0 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.473+0000 7f8ebb400640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e9806b4d0 0x7f8e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:09.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.475+0000 7f8ebb400640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e9806b4d0 0x7f8e9806d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8eac006fd0 tx=0x7f8eac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:09.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.483+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ea407ac20 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.695+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8ebc073050 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.695+0000 7f8eb3400640 1 -- 172.21.15.6:0/67498517 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v114) v1 ==== 80+0+120 (secure 0 0 0) 0x7f8ea407a520 con 0x7f8ebc0afdf0 2024-03-06T07:35:09.698 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6833/4114489444 2024-03-07T07:34:04.377501+0000 2024-03-06T07:35:09.698 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6:6832/4114489444 2024-03-07T07:34:04.377501+0000 2024-03-06T07:35:09.698 INFO:teuthology.orchestra.run.smithi006.stderr:listed 2 entries 2024-03-06T07:35:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e9806b4d0 msgr2=0x7f8e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e9806b4d0 0x7f8e9806d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8eac006fd0 tx=0x7f8eac008040 comp rx=0 tx=0).stop 2024-03-06T07:35:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 msgr2=0x7f8ebc06e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc06e6f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097c0 tx=0x7f8ea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 shutdown_connections 2024-03-06T07:35:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8e9806b4d0 0x7f8e9806d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8ebc15e200 0x7f8ebc06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8ebc0afdf0 0x7f8ebc06e6f0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 --2- 172.21.15.6:0/67498517 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8ebc06afb0 0x7f8ebc147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 >> 172.21.15.6:0/67498517 conn(0x7f8ebc0b2ae0 msgr2=0x7f8ebc15af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 shutdown_connections 2024-03-06T07:35:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:09.698+0000 7f8ec2000640 1 -- 172.21.15.6:0/67498517 wait complete. 2024-03-06T07:35:09.720 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:35:09.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:09.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:09.817 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi006.front.sepia.ceph.com 2024-03-06T07:35:09.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:09.817 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:35:09.866 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:35:09.866 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:09.922 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:09.922 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:09.922 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:09.976 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:09.976 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:35:09.976 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:10.039 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:10.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:10.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.330+0000 7f07e7200640 1 Processor -- start 2024-03-06T07:35:10.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.331+0000 7f07e7200640 1 -- start start 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07e81477e0 0x7f07e8162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f07e806ef20 0x7f07e806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e8147bc0 con 0x7f07e806b4f0 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e8161f90 con 0x7f07e81477e0 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e8162960 con 0x7f07e806ef20 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07e81477e0 0x7f07e8162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f07e806b4f0 0x7f07e80d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46408/0 (socket says 172.21.15.6:46408) 2024-03-06T07:35:10.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e5e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07e81477e0 0x7f07e8162420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59818/0 (socket says 172.21.15.6:59818) 2024-03-06T07:35:10.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e6800640 1 -- 172.21.15.6:0/1786235844 learned_addr learned my addr 172.21.15.6:0/1786235844 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:10.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.333+0000 7f07e5400640 1 --2- 172.21.15.6:0/1786235844 >> v2:172.21.15.178:3300/0 conn(0x7f07e806ef20 0x7f07e806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.334+0000 7f07e5e00640 1 -- 172.21.15.6:0/1786235844 >> v2:172.21.15.178:3300/0 conn(0x7f07e806ef20 msgr2=0x7f07e806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:10.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.334+0000 7f07e5e00640 1 --2- 172.21.15.6:0/1786235844 >> v2:172.21.15.178:3300/0 conn(0x7f07e806ef20 0x7f07e806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.334+0000 7f07e5e00640 1 -- 172.21.15.6:0/1786235844 >> v1:172.21.15.6:6789/0 conn(0x7f07e806b4f0 legacy=0x7f07e80d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.334+0000 7f07e5e00640 1 -- 172.21.15.6:0/1786235844 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e80b3d70 con 0x7f07e81477e0 2024-03-06T07:35:10.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.334+0000 7f07e5400640 1 --2- 172.21.15.6:0/1786235844 >> v2:172.21.15.178:3300/0 conn(0x7f07e806ef20 0x7f07e806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:10.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.335+0000 7f07e5e00640 1 --2- 172.21.15.6:0/1786235844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e81477e0 0x7f07e8162420 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f07e80b6030 tx=0x7f07d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5825ca155156cbbb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e4a00640 1 -- 172.21.15.6:0/1786235844 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d80315a0 con 0x7f07e81477e0 2024-03-06T07:35:10.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e4a00640 1 -- 172.21.15.6:0/1786235844 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d8037070 con 0x7f07e81477e0 2024-03-06T07:35:10.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e4a00640 1 -- 172.21.15.6:0/1786235844 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8031960 con 0x7f07e81477e0 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e7200640 1 -- 172.21.15.6:0/1786235844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e81477e0 msgr2=0x7f07e8162420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e7200640 1 --2- 172.21.15.6:0/1786235844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e81477e0 0x7f07e8162420 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f07e80b6030 tx=0x7f07d802b840 comp rx=0 tx=0).stop 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e7200640 1 -- 172.21.15.6:0/1786235844 shutdown_connections 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e7200640 1 --2- 172.21.15.6:0/1786235844 >> v2:172.21.15.178:3300/0 conn(0x7f07e806ef20 0x7f07e806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e7200640 1 --2- 172.21.15.6:0/1786235844 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e81477e0 0x7f07e8162420 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.336+0000 7f07e7200640 1 -- 172.21.15.6:0/1786235844 >> 172.21.15.6:0/1786235844 conn(0x7f07e806c0c0 msgr2=0x7f07e806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.337+0000 7f07e7200640 1 -- 172.21.15.6:0/1786235844 shutdown_connections 2024-03-06T07:35:10.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.337+0000 7f07e7200640 1 -- 172.21.15.6:0/1786235844 wait complete. 2024-03-06T07:35:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.338+0000 7f07e7200640 1 Processor -- start 2024-03-06T07:35:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.338+0000 7f07e7200640 1 -- start start 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.339+0000 7f07e7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 0x7f07e815d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.339+0000 7f07e7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07e806ef20 0x7f07e815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.339+0000 7f07e7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e81477e0 0x7f07e81354c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.339+0000 7f07e7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e80b4330 con 0x7f07e806b4f0 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.339+0000 7f07e7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e806b920 con 0x7f07e81477e0 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.339+0000 7f07e7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8161c20 con 0x7f07e806ef20 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 0x7f07e815d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 0x7f07e815d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59820/0 (socket says 172.21.15.6:59820) 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 -- 172.21.15.6:0/1859471499 learned_addr learned my addr 172.21.15.6:0/1859471499 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:10.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e6800640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e81477e0 0x7f07e81354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 -- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07e806ef20 msgr2=0x7f07e815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5400640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07e806ef20 0x7f07e815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07e806ef20 0x7f07e815ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 -- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e81477e0 msgr2=0x7f07e81354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e81477e0 0x7f07e81354c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e5e00640 1 -- 172.21.15.6:0/1859471499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8132370 con 0x7f07e806b4f0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.340+0000 7f07e6800640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e81477e0 0x7f07e81354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.341+0000 7f07e5e00640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 0x7f07e815d8b0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009890 tx=0x7f07d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.341+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8032840 con 0x7f07e806b4f0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.341+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d80329e0 con 0x7f07e806b4f0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.341+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e8132650 con 0x7f07e806b4f0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.341+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d8032dd0 con 0x7f07e806b4f0 2024-03-06T07:35:10.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.341+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e8132c10 con 0x7f07e806b4f0 2024-03-06T07:35:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.343+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e804f330 con 0x7f07e806b4f0 2024-03-06T07:35:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.343+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07d803b3f0 con 0x7f07e806b4f0 2024-03-06T07:35:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.344+0000 7f07de000640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f07c806b500 0x7f07c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.344+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f07d802f030 con 0x7f07e806b4f0 2024-03-06T07:35:10.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.345+0000 7f07e5400640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f07c806b500 0x7f07c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.346+0000 7f07e5400640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f07c806b500 0x7f07c806d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f07e81386e0 tx=0x7f07d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.349+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07d8040070 con 0x7f07e806b4f0 2024-03-06T07:35:10.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.621+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07e80724b0 con 0x7f07e806b4f0 2024-03-06T07:35:10.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.622+0000 7f07de000640 1 -- 172.21.15.6:0/1859471499 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f07d807b0b0 con 0x7f07e806b4f0 2024-03-06T07:35:10.624 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:10.625 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:10.625 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f07c806b500 msgr2=0x7f07c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f07c806b500 0x7f07c806d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f07e81386e0 tx=0x7f07d4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:10.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 msgr2=0x7f07e815d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 0x7f07e815d8b0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009890 tx=0x7f07d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 shutdown_connections 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f07c806b500 0x7f07c806d9c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07e81477e0 0x7f07e81354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f07e806ef20 0x7f07e815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 --2- 172.21.15.6:0/1859471499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07e806b4f0 0x7f07e815d8b0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 >> 172.21.15.6:0/1859471499 conn(0x7f07e806c0c0 msgr2=0x7f07e806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 shutdown_connections 2024-03-06T07:35:10.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.625+0000 7f07e7200640 1 -- 172.21.15.6:0/1859471499 wait complete. 2024-03-06T07:35:10.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:10.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.852+0000 7fe3f0200640 1 Processor -- start 2024-03-06T07:35:10.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.853+0000 7fe3f0200640 1 -- start start 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.854+0000 7fe3f0200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3e80af580 0x7fe3e815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.854+0000 7fe3f0200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e815cc40 con 0x7fe3e80cdc20 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.854+0000 7fe3f0200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e815cf20 con 0x7fe3e80af1a0 2024-03-06T07:35:10.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.854+0000 7fe3f0200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e815d200 con 0x7fe3e80af580 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3ece00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe3e80af580 0x7fe3e815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3ee200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe3e80cdc20 0x7fe3e80b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46418/0 (socket says 172.21.15.6:46418) 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3ee200640 1 -- 172.21.15.6:0/2690690127 learned_addr learned my addr 172.21.15.6:0/2690690127 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3e7e00640 1 -- 172.21.15.6:0/2690690127 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708301920 0 0) 0x7fe3e815cc40 con 0x7fe3e80cdc20 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3e7e00640 1 -- 172.21.15.6:0/2690690127 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3cc003520 con 0x7fe3e80cdc20 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3ece00640 1 -- 172.21.15.6:0/2690690127 >> v1:172.21.15.178:6789/0 conn(0x7fe3e80af1a0 legacy=0x7fe3e80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3ece00640 1 -- 172.21.15.6:0/2690690127 >> v1:172.21.15.6:6789/0 conn(0x7fe3e80cdc20 legacy=0x7fe3e80b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.855+0000 7fe3ece00640 1 -- 172.21.15.6:0/2690690127 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e815d4e0 con 0x7fe3e80af580 2024-03-06T07:35:10.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.858+0000 7fe3ece00640 1 --2- 172.21.15.6:0/2690690127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815c600 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40097a0 tx=0x7fe3d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f6344fd3f287fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.858+0000 7fe3e7e00640 1 -- 172.21.15.6:0/2690690127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4035020 con 0x7fe3e80af580 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.858+0000 7fe3e7e00640 1 -- 172.21.15.6:0/2690690127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d4031830 con 0x7fe3e80af580 2024-03-06T07:35:10.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.858+0000 7fe3e7e00640 1 -- 172.21.15.6:0/2690690127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4031be0 con 0x7fe3e80af580 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.858+0000 7fe3f0200640 1 -- 172.21.15.6:0/2690690127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 msgr2=0x7fe3e815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.858+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2690690127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815c600 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40097a0 tx=0x7fe3d402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.859+0000 7fe3f0200640 1 -- 172.21.15.6:0/2690690127 shutdown_connections 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.859+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2690690127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815c600 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.859+0000 7fe3f0200640 1 -- 172.21.15.6:0/2690690127 >> 172.21.15.6:0/2690690127 conn(0x7fe3e806afb0 msgr2=0x7fe3e806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.859+0000 7fe3f0200640 1 -- 172.21.15.6:0/2690690127 shutdown_connections 2024-03-06T07:35:10.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.859+0000 7fe3f0200640 1 -- 172.21.15.6:0/2690690127 wait complete. 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.860+0000 7fe3f0200640 1 Processor -- start 2024-03-06T07:35:10.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.861+0000 7fe3f0200640 1 -- start start 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe3e80af1a0 0x7fe3e809bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3f0200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e80cdc20 0x7fe3e81f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3f0200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e8155e70 con 0x7fe3e80af580 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3f0200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e80b3730 con 0x7fe3e80cdc20 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3f0200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e8155ff0 con 0x7fe3e80af1a0 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ece00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59830/0 (socket says 172.21.15.6:59830) 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ece00640 1 -- 172.21.15.6:0/2352970014 learned_addr learned my addr 172.21.15.6:0/2352970014 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ed800640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe3e80af1a0 0x7fe3e809bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ee200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e80cdc20 0x7fe3e81f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ece00640 1 -- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe3e80af1a0 msgr2=0x7fe3e809bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.862+0000 7fe3ece00640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe3e80af1a0 0x7fe3e809bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.863+0000 7fe3ece00640 1 -- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e80cdc20 msgr2=0x7fe3e81f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.863+0000 7fe3ece00640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e80cdc20 0x7fe3e81f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.863+0000 7fe3ece00640 1 -- 172.21.15.6:0/2352970014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e815b280 con 0x7fe3e80af580 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.863+0000 7fe3ed800640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe3e80af1a0 0x7fe3e809bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.863+0000 7fe3ee200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e80cdc20 0x7fe3e81f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:10.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.863+0000 7fe3ece00640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815a7c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40097a0 tx=0x7fe3d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.864+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4048070 con 0x7fe3e80af580 2024-03-06T07:35:10.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.864+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d4004450 con 0x7fe3e80af580 2024-03-06T07:35:10.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.864+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d4035070 con 0x7fe3e80af580 2024-03-06T07:35:10.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.864+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e815b560 con 0x7fe3e80af580 2024-03-06T07:35:10.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.865+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e815bb20 con 0x7fe3e80af580 2024-03-06T07:35:10.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.866+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e8069510 con 0x7fe3e80af580 2024-03-06T07:35:10.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.868+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3d404b070 con 0x7fe3e80af580 2024-03-06T07:35:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.869+0000 7fe3e5600640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3c806b4d0 0x7fe3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.869+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fe3d40abc70 con 0x7fe3e80af580 2024-03-06T07:35:10.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.870+0000 7fe3ed800640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3c806b4d0 0x7fe3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:10.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.871+0000 7fe3ed800640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3c806b4d0 0x7fe3c806d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8001d70 tx=0x7fe3d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:10.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:10.874+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3d4040070 con 0x7fe3e80af580 2024-03-06T07:35:11.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.074+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3e804f3a0 con 0x7fe3e80af580 2024-03-06T07:35:11.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.077+0000 7fe3e5600640 1 -- 172.21.15.6:0/2352970014 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16775 (secure 0 0 0) 0x7fe3d407a9c0 con 0x7fe3e80af580 2024-03-06T07:35:11.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:11.079 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":114,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:34:11.424305+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6833/4114489444":"2024-03-07T07:34:04.377501+0000","172.21.15.6:6832/4114489444":"2024-03-07T07:34:04.377501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:11.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3c806b4d0 msgr2=0x7fe3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3c806b4d0 0x7fe3c806d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8001d70 tx=0x7fe3d8009040 comp rx=0 tx=0).stop 2024-03-06T07:35:11.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 msgr2=0x7fe3e815a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815a7c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40097a0 tx=0x7fe3d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 shutdown_connections 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe3c806b4d0 0x7fe3c806d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3e80cdc20 0x7fe3e81f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe3e80af580 0x7fe3e815a7c0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 --2- 172.21.15.6:0/2352970014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe3e80af1a0 0x7fe3e809bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 >> 172.21.15.6:0/2352970014 conn(0x7fe3e806afb0 msgr2=0x7fe3e80cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 shutdown_connections 2024-03-06T07:35:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.080+0000 7fe3f0200640 1 -- 172.21.15.6:0/2352970014 wait complete. 2024-03-06T07:35:11.102 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:33:46.337716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:33:47.450154+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:11.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:11.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.359+0000 7f61bca00640 1 Processor -- start 2024-03-06T07:35:11.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.360+0000 7f61bca00640 1 -- start start 2024-03-06T07:35:11.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.361+0000 7f61bca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61b815dee0 0x7f61b80d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.361+0000 7f61bca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f61b80d0d40 0x7f61b80b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.361+0000 7f61bca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b80ceca0 con 0x7f61b80d4860 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.361+0000 7f61bca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b80cf5d0 con 0x7f61b815dee0 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.361+0000 7f61bca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b80cef80 con 0x7f61b80d0d40 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61b815dee0 0x7f61b80d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f61b815dee0 0x7f61b80d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59834/0 (socket says 172.21.15.6:59834) 2024-03-06T07:35:11.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 -- 172.21.15.6:0/2181051123 learned_addr learned my addr 172.21.15.6:0/2181051123 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:11.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7e00640 1 --2- 172.21.15.6:0/2181051123 >> v2:172.21.15.178:3300/0 conn(0x7f61b80d0d40 0x7f61b80b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 -- 172.21.15.6:0/2181051123 >> v2:172.21.15.178:3300/0 conn(0x7f61b80d0d40 msgr2=0x7f61b80b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 --2- 172.21.15.6:0/2181051123 >> v2:172.21.15.178:3300/0 conn(0x7f61b80d0d40 0x7f61b80b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 -- 172.21.15.6:0/2181051123 >> v1:172.21.15.178:6789/0 conn(0x7f61b80d4860 legacy=0x7f61b80d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7400640 1 -- 172.21.15.6:0/2181051123 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b80cf260 con 0x7f61b815dee0 2024-03-06T07:35:11.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.362+0000 7f61b7e00640 1 --2- 172.21.15.6:0/2181051123 >> v2:172.21.15.178:3300/0 conn(0x7f61b80d0d40 0x7f61b80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.363+0000 7f61b7400640 1 --2- 172.21.15.6:0/2181051123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b80d18a0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f61b806b2d0 tx=0x7f61a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c870237952a0d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61b6000640 1 -- 172.21.15.6:0/2181051123 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0035020 con 0x7f61b815dee0 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61b6000640 1 -- 172.21.15.6:0/2181051123 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a0031870 con 0x7f61b815dee0 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61b6000640 1 -- 172.21.15.6:0/2181051123 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0031c40 con 0x7f61b815dee0 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 -- 172.21.15.6:0/2181051123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 msgr2=0x7f61b80d18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 --2- 172.21.15.6:0/2181051123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b80d18a0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f61b806b2d0 tx=0x7f61a002b910 comp rx=0 tx=0).stop 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 -- 172.21.15.6:0/2181051123 shutdown_connections 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 --2- 172.21.15.6:0/2181051123 >> v2:172.21.15.178:3300/0 conn(0x7f61b80d0d40 0x7f61b80b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 --2- 172.21.15.6:0/2181051123 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b80d18a0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 -- 172.21.15.6:0/2181051123 >> 172.21.15.6:0/2181051123 conn(0x7f61b80d3cd0 msgr2=0x7f61b80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 -- 172.21.15.6:0/2181051123 shutdown_connections 2024-03-06T07:35:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.364+0000 7f61bca00640 1 -- 172.21.15.6:0/2181051123 wait complete. 2024-03-06T07:35:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.366+0000 7f61bca00640 1 Processor -- start 2024-03-06T07:35:11.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.366+0000 7f61bca00640 1 -- start start 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.367+0000 7f61bca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61b80d0d40 0x7f61b81485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.367+0000 7f61bca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61b80d4860 0x7f61b815fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.367+0000 7f61bca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b814a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.367+0000 7f61bca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80d1de0 con 0x7f61b815dee0 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.367+0000 7f61bca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b815e2c0 con 0x7f61b80d4860 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.367+0000 7f61bca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80d1f60 con 0x7f61b80d0d40 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b814a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b814a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59846/0 (socket says 172.21.15.6:59846) 2024-03-06T07:35:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 -- 172.21.15.6:0/2162609245 learned_addr learned my addr 172.21.15.6:0/2162609245 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b6a00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61b80d4860 0x7f61b815fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7400640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61b80d0d40 0x7f61b81485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 -- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61b80d0d40 msgr2=0x7f61b81485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61b80d0d40 0x7f61b81485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 -- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61b80d4860 msgr2=0x7f61b815fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61b80d4860 0x7f61b815fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 -- 172.21.15.6:0/2162609245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b81492e0 con 0x7f61b815dee0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b6a00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61b80d4860 0x7f61b815fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.368+0000 7f61b7e00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b814a380 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f61b000c7c0 tx=0x7f61b000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.369+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0015020 con 0x7f61b815dee0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.369+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b0004230 con 0x7f61b815dee0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.369+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0007620 con 0x7f61b815dee0 2024-03-06T07:35:11.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.369+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b81495c0 con 0x7f61b815dee0 2024-03-06T07:35:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.369+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b8149b80 con 0x7f61b815dee0 2024-03-06T07:35:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.370+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b804f330 con 0x7f61b815dee0 2024-03-06T07:35:11.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.371+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61b0018020 con 0x7f61b815dee0 2024-03-06T07:35:11.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.372+0000 7f61af400640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f619406b500 0x7f619406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.372+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f61b008a3e0 con 0x7f61b815dee0 2024-03-06T07:35:11.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.373+0000 7f61b7400640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f619406b500 0x7f619406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.376 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.374+0000 7f61b7400640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f619406b500 0x7f619406d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f61a0036000 tx=0x7f61a0008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.377+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61b001f020 con 0x7f61b815dee0 2024-03-06T07:35:11.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.606+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61b80b68a0 con 0x7f61b815dee0 2024-03-06T07:35:11.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.607+0000 7f61af400640 1 -- 172.21.15.6:0/2162609245 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61b00591f0 con 0x7f61b815dee0 2024-03-06T07:35:11.609 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f619406b500 msgr2=0x7f619406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f619406b500 0x7f619406d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f61a0036000 tx=0x7f61a0008030 comp rx=0 tx=0).stop 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 msgr2=0x7f61b814a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b814a380 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f61b000c7c0 tx=0x7f61b000cc80 comp rx=0 tx=0).stop 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 shutdown_connections 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f619406b500 0x7f619406d9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61b80d4860 0x7f61b815fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61b815dee0 0x7f61b814a380 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 --2- 172.21.15.6:0/2162609245 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61b80d0d40 0x7f61b81485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.609+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 >> 172.21.15.6:0/2162609245 conn(0x7f61b80d3cd0 msgr2=0x7f61b80b8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.610+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 shutdown_connections 2024-03-06T07:35:11.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.610+0000 7f61bca00640 1 -- 172.21.15.6:0/2162609245 wait complete. 2024-03-06T07:35:11.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:35:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.846+0000 7fae4be00640 1 Processor -- start 2024-03-06T07:35:11.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.846+0000 7fae4be00640 1 -- start start 2024-03-06T07:35:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.848+0000 7fae4be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fae4c06f540 0x7fae4c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.848+0000 7fae4be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fae4c0af930 0x7fae4c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.848+0000 7fae4be00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae4c06fe60 con 0x7fae4c0b4530 2024-03-06T07:35:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.848+0000 7fae4be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0b3520 con 0x7fae4c06f540 2024-03-06T07:35:11.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.848+0000 7fae4be00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae4c0b3770 con 0x7fae4c0af930 2024-03-06T07:35:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fae4c06f540 0x7fae4c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fae4c06f540 0x7fae4c06f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59860/0 (socket says 172.21.15.6:59860) 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 -- 172.21.15.6:0/1137189455 learned_addr learned my addr 172.21.15.6:0/1137189455 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4a000640 1 --2- 172.21.15.6:0/1137189455 >> v2:172.21.15.178:3300/0 conn(0x7fae4c0af930 0x7fae4c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 -- 172.21.15.6:0/1137189455 >> v2:172.21.15.178:3300/0 conn(0x7fae4c0af930 msgr2=0x7fae4c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 --2- 172.21.15.6:0/1137189455 >> v2:172.21.15.178:3300/0 conn(0x7fae4c0af930 0x7fae4c0cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 -- 172.21.15.6:0/1137189455 >> v1:172.21.15.178:6789/0 conn(0x7fae4c0b4530 legacy=0x7fae4c0b49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.849+0000 7fae4aa00640 1 -- 172.21.15.6:0/1137189455 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0b3a50 con 0x7fae4c06f540 2024-03-06T07:35:11.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.851+0000 7fae4aa00640 1 --2- 172.21.15.6:0/1137189455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c06f920 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0676f0 tx=0x7fae3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c1514558796a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.851+0000 7fae49600640 1 -- 172.21.15.6:0/1137189455 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c034070 con 0x7fae4c06f540 2024-03-06T07:35:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.851+0000 7fae49600640 1 -- 172.21.15.6:0/1137189455 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c030860 con 0x7fae4c06f540 2024-03-06T07:35:11.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.851+0000 7fae49600640 1 -- 172.21.15.6:0/1137189455 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c030c30 con 0x7fae4c06f540 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 -- 172.21.15.6:0/1137189455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 msgr2=0x7fae4c06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 --2- 172.21.15.6:0/1137189455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c06f920 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0676f0 tx=0x7fae3c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 -- 172.21.15.6:0/1137189455 shutdown_connections 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 --2- 172.21.15.6:0/1137189455 >> v2:172.21.15.178:3300/0 conn(0x7fae4c0af930 0x7fae4c0cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 --2- 172.21.15.6:0/1137189455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c06f920 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 -- 172.21.15.6:0/1137189455 >> 172.21.15.6:0/1137189455 conn(0x7fae4c0723d0 msgr2=0x7fae4c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 -- 172.21.15.6:0/1137189455 shutdown_connections 2024-03-06T07:35:11.854 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.852+0000 7fae4be00640 1 -- 172.21.15.6:0/1137189455 wait complete. 2024-03-06T07:35:11.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.855+0000 7fae4be00640 1 Processor -- start 2024-03-06T07:35:11.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.855+0000 7fae4be00640 1 -- start start 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.856+0000 7fae4be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c15a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c0af930 0x7fae4c141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae4c0b4530 0x7fae4c141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0cffd0 con 0x7fae4c06f540 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c0b4010 con 0x7fae4c0af930 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae4c15cf00 con 0x7fae4c0b4530 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c15a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c15a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60192/0 (socket says 172.21.15.6:60192) 2024-03-06T07:35:11.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 -- 172.21.15.6:0/339295774 learned_addr learned my addr 172.21.15.6:0/339295774 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4a000640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c0af930 0x7fae4c141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 -- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae4c0b4530 msgr2=0x7fae4c141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae4c0b4530 0x7fae4c141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 -- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c0af930 msgr2=0x7fae4c141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c0af930 0x7fae4c141050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4aa00640 1 -- 172.21.15.6:0/339295774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c1fc680 con 0x7fae4c06f540 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.857+0000 7fae4a000640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c0af930 0x7fae4c141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.858+0000 7fae4aa00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c15a3d0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fae3c000c00 tx=0x7fae3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.858+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c048070 con 0x7fae4c06f540 2024-03-06T07:35:11.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.858+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3c030660 con 0x7fae4c06f540 2024-03-06T07:35:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.858+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4c06d3d0 con 0x7fae4c06f540 2024-03-06T07:35:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.858+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3c034040 con 0x7fae4c06f540 2024-03-06T07:35:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.858+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae4c06d990 con 0x7fae4c06f540 2024-03-06T07:35:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.860+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae4c04f330 con 0x7fae4c06f540 2024-03-06T07:35:11.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.860+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae3c04b070 con 0x7fae4c06f540 2024-03-06T07:35:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.861+0000 7fae42a00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae2c06b4d0 0x7fae2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:11.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.861+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fae3c0ac3f0 con 0x7fae4c06f540 2024-03-06T07:35:11.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.862+0000 7fae4a000640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae2c06b4d0 0x7fae2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:11.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.863+0000 7fae4a000640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae2c06b4d0 0x7fae2c06d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fae4c148e90 tx=0x7fae38008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:11.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:11.866+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fae3c07b260 con 0x7fae4c06f540 2024-03-06T07:35:12.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.065+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae4c0cfd80 con 0x7fae4c06f540 2024-03-06T07:35:12.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.066+0000 7fae42a00640 1 -- 172.21.15.6:0/339295774 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v114) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae3c030830 con 0x7fae4c06f540 2024-03-06T07:35:12.068 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:35:12.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae2c06b4d0 msgr2=0x7fae2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae2c06b4d0 0x7fae2c06d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fae4c148e90 tx=0x7fae38008040 comp rx=0 tx=0).stop 2024-03-06T07:35:12.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 msgr2=0x7fae4c15a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c15a3d0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fae3c000c00 tx=0x7fae3c004730 comp rx=0 tx=0).stop 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 shutdown_connections 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fae2c06b4d0 0x7fae2c06d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fae4c0af930 0x7fae4c141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fae4c0b4530 0x7fae4c141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 --2- 172.21.15.6:0/339295774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fae4c06f540 0x7fae4c15a3d0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 >> 172.21.15.6:0/339295774 conn(0x7fae4c0723d0 msgr2=0x7fae4c0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 shutdown_connections 2024-03-06T07:35:12.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.068+0000 7fae4be00640 1 -- 172.21.15.6:0/339295774 wait complete. 2024-03-06T07:35:12.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:35:12.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.307+0000 7faddb200640 1 Processor -- start 2024-03-06T07:35:12.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.308+0000 7faddb200640 1 -- start start 2024-03-06T07:35:12.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.309+0000 7faddb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faddc15df70 0x7faddc0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.309+0000 7faddb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0d29e0 0x7faddc0cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.309+0000 7faddb200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0b6be0 con 0x7faddc15e350 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.309+0000 7faddb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faddc0b6ec0 con 0x7faddc15df70 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.309+0000 7faddb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0b71a0 con 0x7faddc0d29e0 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.309+0000 7fadd9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faddc15df70 0x7faddc0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd9e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faddc15df70 0x7faddc0cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60198/0 (socket says 172.21.15.6:60198) 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd9e00640 1 -- 172.21.15.6:0/385594206 learned_addr learned my addr 172.21.15.6:0/385594206 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd3e00640 1 --1- 172.21.15.6:0/385594206 >> v1:172.21.15.6:6789/0 conn(0x7faddc15e350 0x7faddc0d22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34668/0 (socket says 172.21.15.6:34668) 2024-03-06T07:35:12.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadda800640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0d29e0 0x7faddc0cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd9e00640 1 -- 172.21.15.6:0/385594206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0d29e0 msgr2=0x7faddc0cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd9e00640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0d29e0 0x7faddc0cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd9e00640 1 -- 172.21.15.6:0/385594206 >> v1:172.21.15.6:6789/0 conn(0x7faddc15e350 legacy=0x7faddc0d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadd9e00640 1 -- 172.21.15.6:0/385594206 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc0b7480 con 0x7faddc15df70 2024-03-06T07:35:12.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.310+0000 7fadda800640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0d29e0 0x7faddc0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.311+0000 7fadd9e00640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0cf3a0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009870 tx=0x7fadcc02ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d40fab5f51b07a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.312+0000 7fadd9400640 1 -- 172.21.15.6:0/385594206 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc02f830 con 0x7faddc15df70 2024-03-06T07:35:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.312+0000 7fadd9400640 1 -- 172.21.15.6:0/385594206 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc02f9d0 con 0x7faddc15df70 2024-03-06T07:35:12.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.312+0000 7fadd9400640 1 -- 172.21.15.6:0/385594206 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc02fda0 con 0x7faddc15df70 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.312+0000 7faddb200640 1 -- 172.21.15.6:0/385594206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 msgr2=0x7faddc0cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.312+0000 7faddb200640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0cf3a0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009870 tx=0x7fadcc02ba10 comp rx=0 tx=0).stop 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.313+0000 7faddb200640 1 -- 172.21.15.6:0/385594206 shutdown_connections 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.313+0000 7faddb200640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0d29e0 0x7faddc0cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.313+0000 7faddb200640 1 --2- 172.21.15.6:0/385594206 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0cf3a0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.313+0000 7faddb200640 1 -- 172.21.15.6:0/385594206 >> 172.21.15.6:0/385594206 conn(0x7faddc08d2a0 msgr2=0x7faddc0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.313+0000 7faddb200640 1 -- 172.21.15.6:0/385594206 shutdown_connections 2024-03-06T07:35:12.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.313+0000 7faddb200640 1 -- 172.21.15.6:0/385594206 wait complete. 2024-03-06T07:35:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.315+0000 7faddb200640 1 Processor -- start 2024-03-06T07:35:12.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.315+0000 7faddb200640 1 -- start start 2024-03-06T07:35:12.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7faddb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faddc0d29e0 0x7faddc1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7faddb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7faddb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0cc240 0x7faddc0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7faddb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0b4900 con 0x7faddc15df70 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7faddb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0d1d60 con 0x7faddc0d29e0 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7faddb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0b4a80 con 0x7faddc0cc240 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60200/0 (socket says 172.21.15.6:60200) 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 -- 172.21.15.6:0/1610521199 learned_addr learned my addr 172.21.15.6:0/1610521199 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd9e00640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faddc0d29e0 0x7faddc1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 -- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0cc240 msgr2=0x7faddc0ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0cc240 0x7faddc0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 -- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faddc0d29e0 msgr2=0x7faddc1553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faddc0d29e0 0x7faddc1553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.317+0000 7fadd3e00640 1 -- 172.21.15.6:0/1610521199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc0cb1a0 con 0x7faddc15df70 2024-03-06T07:35:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.318+0000 7fadd3e00640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0c8230 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c440 tx=0x7fadc400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.318+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc40074e0 con 0x7faddc15df70 2024-03-06T07:35:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.318+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc4007680 con 0x7faddc15df70 2024-03-06T07:35:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.318+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc0cb480 con 0x7faddc15df70 2024-03-06T07:35:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.318+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4004a20 con 0x7faddc15df70 2024-03-06T07:35:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.318+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc0cba40 con 0x7faddc15df70 2024-03-06T07:35:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.319+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc04f330 con 0x7faddc15df70 2024-03-06T07:35:12.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.321+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fadc401a020 con 0x7faddc15df70 2024-03-06T07:35:12.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.322+0000 7fadd2000640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fadb806b4d0 0x7fadb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.322+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fadc40899e0 con 0x7faddc15df70 2024-03-06T07:35:12.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.322+0000 7fadd9e00640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fadb806b4d0 0x7fadb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.323+0000 7fadd9e00640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fadb806b4d0 0x7fadb806d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fadcc036000 tx=0x7fadcc039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.326+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fadc4010300 con 0x7faddc15df70 2024-03-06T07:35:12.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.527+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faddc0d20b0 con 0x7faddc15df70 2024-03-06T07:35:12.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.528+0000 7fadd2000640 1 -- 172.21.15.6:0/1610521199 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v114) v1 ==== 100+0+11 (secure 0 0 0) 0x7fadc40587c0 con 0x7faddc15df70 2024-03-06T07:35:12.530 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fadb806b4d0 msgr2=0x7fadb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fadb806b4d0 0x7fadb806d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fadcc036000 tx=0x7fadcc039040 comp rx=0 tx=0).stop 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 msgr2=0x7faddc0c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0c8230 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fadc400c440 tx=0x7fadc400c900 comp rx=0 tx=0).stop 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 shutdown_connections 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fadb806b4d0 0x7fadb806d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faddc0cc240 0x7faddc0ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.530+0000 7faddb200640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faddc15df70 0x7faddc0c8230 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.531+0000 7faddb200640 1 --2- 172.21.15.6:0/1610521199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faddc0d29e0 0x7faddc1553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.531+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 >> 172.21.15.6:0/1610521199 conn(0x7faddc08d2a0 msgr2=0x7faddc0b7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.531+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 shutdown_connections 2024-03-06T07:35:12.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.531+0000 7faddb200640 1 -- 172.21.15.6:0/1610521199 wait complete. 2024-03-06T07:35:12.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.751+0000 7f44d7800640 1 Processor -- start 2024-03-06T07:35:12.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.751+0000 7f44d7800640 1 -- start start 2024-03-06T07:35:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d7800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44d009ac80 0x7f44d00b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d7800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d00b1440 0x7f44d006c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d7800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d00b60b0 con 0x7f44d009b060 2024-03-06T07:35:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d7800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d00b6390 con 0x7f44d009ac80 2024-03-06T07:35:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d7800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00b6670 con 0x7f44d00b1440 2024-03-06T07:35:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44d009ac80 0x7f44d00b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.753+0000 7f44d4e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44d009ac80 0x7f44d00b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60204/0 (socket says 172.21.15.6:60204) 2024-03-06T07:35:12.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44d4e00640 1 -- 172.21.15.6:0/1217616712 learned_addr learned my addr 172.21.15.6:0/1217616712 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44cfe00640 1 --2- 172.21.15.6:0/1217616712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d00b1440 0x7f44d006c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44d4e00640 1 -- 172.21.15.6:0/1217616712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d00b1440 msgr2=0x7f44d006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44d4e00640 1 --2- 172.21.15.6:0/1217616712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d00b1440 0x7f44d006c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44d4e00640 1 -- 172.21.15.6:0/1217616712 >> v1:172.21.15.6:6789/0 conn(0x7f44d009b060 legacy=0x7f44d006c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44d4e00640 1 -- 172.21.15.6:0/1217616712 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00b6950 con 0x7f44d009ac80 2024-03-06T07:35:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.754+0000 7f44cf400640 1 -- 172.21.15.6:0/1217616712 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472418245 0 0) 0x7f44d00b60b0 con 0x7f44d009b060 2024-03-06T07:35:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.756+0000 7f44d4e00640 1 --2- 172.21.15.6:0/1217616712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009ac80 0x7f44d00b5720 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009870 tx=0x7f44c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5179636aadd0a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.756+0000 7f44cf400640 1 -- 172.21.15.6:0/1217616712 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4035020 con 0x7f44d009ac80 2024-03-06T07:35:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.756+0000 7f44cf400640 1 -- 172.21.15.6:0/1217616712 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c4031640 con 0x7f44d009ac80 2024-03-06T07:35:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.756+0000 7f44cf400640 1 -- 172.21.15.6:0/1217616712 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c402d040 con 0x7f44d009ac80 2024-03-06T07:35:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 -- 172.21.15.6:0/1217616712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009ac80 msgr2=0x7f44d00b5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 --2- 172.21.15.6:0/1217616712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009ac80 0x7f44d00b5720 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009870 tx=0x7f44c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 -- 172.21.15.6:0/1217616712 shutdown_connections 2024-03-06T07:35:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 --2- 172.21.15.6:0/1217616712 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d00b1440 0x7f44d006c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 --2- 172.21.15.6:0/1217616712 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009ac80 0x7f44d00b5720 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 -- 172.21.15.6:0/1217616712 >> 172.21.15.6:0/1217616712 conn(0x7f44d00b3a10 msgr2=0x7f44d00b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 -- 172.21.15.6:0/1217616712 shutdown_connections 2024-03-06T07:35:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.757+0000 7f44d7800640 1 -- 172.21.15.6:0/1217616712 wait complete. 2024-03-06T07:35:12.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.759+0000 7f44d7800640 1 Processor -- start 2024-03-06T07:35:12.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.759+0000 7f44d7800640 1 -- start start 2024-03-06T07:35:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d7800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d009ac80 0x7f44d0145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d7800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 0x7f44d00cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d7800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d00b1440 0x7f44d01465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d7800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d006cd10 con 0x7f44d009b060 2024-03-06T07:35:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d7800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d00b5e70 con 0x7f44d00b1440 2024-03-06T07:35:12.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d7800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d006ce90 con 0x7f44d009ac80 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 0x7f44d00cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 0x7f44d00cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60210/0 (socket says 172.21.15.6:60210) 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 -- 172.21.15.6:0/3632639801 learned_addr learned my addr 172.21.15.6:0/3632639801 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d5800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d00b1440 0x7f44d01465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 -- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d009ac80 msgr2=0x7f44d0145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d009ac80 0x7f44d0145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 -- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d00b1440 msgr2=0x7f44d01465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d00b1440 0x7f44d01465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44cfe00640 1 -- 172.21.15.6:0/3632639801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d00cb2f0 con 0x7f44d009b060 2024-03-06T07:35:12.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.760+0000 7f44d5800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d00b1440 0x7f44d01465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.761+0000 7f44cfe00640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 0x7f44d00cc670 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00c820 tx=0x7f44bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.761+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc015070 con 0x7f44d009b060 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.761+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc004020 con 0x7f44d009b060 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.761+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0043f0 con 0x7f44d009b060 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.761+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d00cb5d0 con 0x7f44d009b060 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.761+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d00cbb90 con 0x7f44d009b060 2024-03-06T07:35:12.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.762+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d004f3a0 con 0x7f44d009b060 2024-03-06T07:35:12.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.764+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44bc007830 con 0x7f44d009b060 2024-03-06T07:35:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.764+0000 7f44ccc00640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b006b4d0 0x7f44b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:12.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.765+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f44bc08a570 con 0x7f44d009b060 2024-03-06T07:35:12.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.765+0000 7f44d4e00640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b006b4d0 0x7f44b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:12.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.767+0000 7f44d4e00640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b006b4d0 0x7f44b006d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f44c4036000 tx=0x7f44c4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:12.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:12.768+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44bc059340 con 0x7f44d009b060 2024-03-06T07:35:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.006+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44d0074c00 con 0x7f44d009b060 2024-03-06T07:35:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.007+0000 7f44ccc00640 1 -- 172.21.15.6:0/3632639801 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f44bc058c40 con 0x7f44d009b060 2024-03-06T07:35:13.009 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:13.010 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b006b4d0 msgr2=0x7f44b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b006b4d0 0x7f44b006d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f44c4036000 tx=0x7f44c4030000 comp rx=0 tx=0).stop 2024-03-06T07:35:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 msgr2=0x7f44d00cc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 0x7f44d00cc670 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00c820 tx=0x7f44bc00cce0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 shutdown_connections 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f44b006b4d0 0x7f44b006d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44d00b1440 0x7f44d01465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44d009b060 0x7f44d00cc670 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 --2- 172.21.15.6:0/3632639801 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44d009ac80 0x7f44d0145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 >> 172.21.15.6:0/3632639801 conn(0x7f44d00b3a10 msgr2=0x7f44d006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 shutdown_connections 2024-03-06T07:35:13.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.010+0000 7f44d7800640 1 -- 172.21.15.6:0/3632639801 wait complete. 2024-03-06T07:35:13.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.252+0000 7f3827200640 1 Processor -- start 2024-03-06T07:35:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.252+0000 7f3827200640 1 -- start start 2024-03-06T07:35:13.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3827200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f38280d4a10 0x7f38280d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3827200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f38280d0c80 0x7f38280b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3827200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38280b3710 con 0x7f382815e0b0 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3827200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38280d10f0 con 0x7f38280d0c80 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3827200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38280b09e0 con 0x7f38280d4a10 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3825e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f382815e0b0 0x7f38280cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34686/0 (socket says 172.21.15.6:34686) 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3825e00640 1 -- 172.21.15.6:0/1579484762 learned_addr learned my addr 172.21.15.6:0/1579484762 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3826800640 1 --2- 172.21.15.6:0/1579484762 >> v2:172.21.15.6:3300/0 conn(0x7f38280d0c80 0x7f38280b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3825400640 1 --2- 172.21.15.6:0/1579484762 >> v2:172.21.15.178:3300/0 conn(0x7f38280d4a10 0x7f38280d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.254+0000 7f3824a00640 1 -- 172.21.15.6:0/1579484762 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471382302 0 0) 0x7f38280b3710 con 0x7f382815e0b0 2024-03-06T07:35:13.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.255+0000 7f3824a00640 1 -- 172.21.15.6:0/1579484762 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c003520 con 0x7f382815e0b0 2024-03-06T07:35:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.255+0000 7f3826800640 1 -- 172.21.15.6:0/1579484762 >> v2:172.21.15.178:3300/0 conn(0x7f38280d4a10 msgr2=0x7f38280d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.255+0000 7f3826800640 1 --2- 172.21.15.6:0/1579484762 >> v2:172.21.15.178:3300/0 conn(0x7f38280d4a10 0x7f38280d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.255+0000 7f3826800640 1 -- 172.21.15.6:0/1579484762 >> v1:172.21.15.6:6789/0 conn(0x7f382815e0b0 legacy=0x7f38280cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.255+0000 7f3826800640 1 -- 172.21.15.6:0/1579484762 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280b39f0 con 0x7f38280d0c80 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.256+0000 7f3826800640 1 --2- 172.21.15.6:0/1579484762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d0c80 0x7f38280b44d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f382806b8c0 tx=0x7f382002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55415c78173780bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.256+0000 7f3824a00640 1 -- 172.21.15.6:0/1579484762 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820039070 con 0x7f38280d0c80 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.256+0000 7f3824a00640 1 -- 172.21.15.6:0/1579484762 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382002cc60 con 0x7f38280d0c80 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.256+0000 7f3824a00640 1 -- 172.21.15.6:0/1579484762 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820034640 con 0x7f38280d0c80 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.256+0000 7f3827200640 1 -- 172.21.15.6:0/1579484762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d0c80 msgr2=0x7f38280b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.256+0000 7f3827200640 1 --2- 172.21.15.6:0/1579484762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d0c80 0x7f38280b44d0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f382806b8c0 tx=0x7f382002beb0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.257+0000 7f3827200640 1 -- 172.21.15.6:0/1579484762 shutdown_connections 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.257+0000 7f3827200640 1 --2- 172.21.15.6:0/1579484762 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d0c80 0x7f38280b44d0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.257+0000 7f3827200640 1 --2- 172.21.15.6:0/1579484762 >> v2:172.21.15.178:3300/0 conn(0x7f38280d4a10 0x7f38280d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.257+0000 7f3827200640 1 -- 172.21.15.6:0/1579484762 >> 172.21.15.6:0/1579484762 conn(0x7f38280d3d30 msgr2=0x7f38280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.257+0000 7f3827200640 1 -- 172.21.15.6:0/1579484762 shutdown_connections 2024-03-06T07:35:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.257+0000 7f3827200640 1 -- 172.21.15.6:0/1579484762 wait complete. 2024-03-06T07:35:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.258+0000 7f3827200640 1 Processor -- start 2024-03-06T07:35:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.258+0000 7f3827200640 1 -- start start 2024-03-06T07:35:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.259+0000 7f3827200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38280d0c80 0x7f38281f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.259+0000 7f3827200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 0x7f38280b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.259+0000 7f3827200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382815e0b0 0x7f38280cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.259+0000 7f3827200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280b3fb0 con 0x7f38280d4a10 2024-03-06T07:35:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.259+0000 7f3827200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280cfa20 con 0x7f38280d0c80 2024-03-06T07:35:13.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.259+0000 7f3827200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382815e490 con 0x7f382815e0b0 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 0x7f38280b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 0x7f38280b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60232/0 (socket says 172.21.15.6:60232) 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 -- 172.21.15.6:0/2537245125 learned_addr learned my addr 172.21.15.6:0/2537245125 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 -- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382815e0b0 msgr2=0x7f38280cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382815e0b0 0x7f38280cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 -- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38280d0c80 msgr2=0x7f38281f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38280d0c80 0x7f38281f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 -- 172.21.15.6:0/2537245125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280b6af0 con 0x7f38280d4a10 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.260+0000 7f3825400640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 0x7f38280b8490 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f381400d7d0 tx=0x7f381400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.261+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381400b6c0 con 0x7f38280d4a10 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.261+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3814019070 con 0x7f38280d4a10 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.261+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381400ba80 con 0x7f38280d4a10 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.261+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38280b6dd0 con 0x7f38280d4a10 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.261+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38280b7390 con 0x7f38280d4a10 2024-03-06T07:35:13.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.262+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382804f330 con 0x7f38280d4a10 2024-03-06T07:35:13.266 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.264+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f381400bc20 con 0x7f38280d4a10 2024-03-06T07:35:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.265+0000 7f381e000640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b4d0 0x7f380806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.265+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f381408a790 con 0x7f38280d4a10 2024-03-06T07:35:13.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.265+0000 7f3825e00640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b4d0 0x7f380806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.266+0000 7f3825e00640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b4d0 0x7f380806d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3818000f60 tx=0x7f38180078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.270+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f381401c360 con 0x7f38280d4a10 2024-03-06T07:35:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.479+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38280cf780 con 0x7f38280d4a10 2024-03-06T07:35:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.481+0000 7f381e000640 1 -- 172.21.15.6:0/2537245125 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16775 (secure 0 0 0) 0x7f38140594e0 con 0x7f38280d4a10 2024-03-06T07:35:13.483 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:13.483 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":114,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:34:11.424305+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6833/4114489444":"2024-03-07T07:34:04.377501+0000","172.21.15.6:6832/4114489444":"2024-03-07T07:34:04.377501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:13.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b4d0 msgr2=0x7f380806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b4d0 0x7f380806d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3818000f60 tx=0x7f38180078f0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 msgr2=0x7f38280b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 0x7f38280b8490 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f381400d7d0 tx=0x7f381400dc90 comp rx=0 tx=0).stop 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 shutdown_connections 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f380806b4d0 0x7f380806d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f382815e0b0 0x7f38280cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38280d4a10 0x7f38280b8490 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 --2- 172.21.15.6:0/2537245125 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38280d0c80 0x7f38281f4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 >> 172.21.15.6:0/2537245125 conn(0x7f38280d3d30 msgr2=0x7f38280cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 shutdown_connections 2024-03-06T07:35:13.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.483+0000 7f3827200640 1 -- 172.21.15.6:0/2537245125 wait complete. 2024-03-06T07:35:13.504 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:35:13.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.710+0000 7fbf81400640 1 Processor -- start 2024-03-06T07:35:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.710+0000 7fbf81400640 1 -- start start 2024-03-06T07:35:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf81400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbf7c0d0cd0 0x7fbf7c0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf81400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7c0d2dc0 con 0x7fbf7c0d2960 2024-03-06T07:35:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf81400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf7c0d18c0 con 0x7fbf7c15e070 2024-03-06T07:35:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf81400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7c0d1ba0 con 0x7fbf7c0d0cd0 2024-03-06T07:35:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf73400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbf7c0d2960 0x7fbf7c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34690/0 (socket says 172.21.15.6:34690) 2024-03-06T07:35:13.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf73400640 1 -- 172.21.15.6:0/218777669 learned_addr learned my addr 172.21.15.6:0/218777669 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf80a00640 1 --2- 172.21.15.6:0/218777669 >> v2:172.21.15.178:3300/0 conn(0x7fbf7c0d0cd0 0x7fbf7c0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.712+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2887747484 0 0) 0x7fbf7c0d2dc0 con 0x7fbf7c0d2960 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf60003520 con 0x7fbf7c0d2960 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2763515895 0 0) 0x7fbf60003520 con 0x7fbf7c0d2960 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 >> v2:172.21.15.178:3300/0 conn(0x7fbf7c0d0cd0 msgr2=0x7fbf7c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 --2- 172.21.15.6:0/218777669 >> v2:172.21.15.178:3300/0 conn(0x7fbf7c0d0cd0 0x7fbf7c0b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 >> v1:172.21.15.178:6789/0 conn(0x7fbf7c15e070 legacy=0x7fbf7c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7c0d1e80 con 0x7fbf7c0d2960 2024-03-06T07:35:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fbf64001670 con 0x7fbf7c0d2960 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 >> v1:172.21.15.6:6789/0 conn(0x7fbf7c0d2960 legacy=0x7fbf7c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600039b0 0x7fbf60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0d1ba0 con 0x7fbf600051c0 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6000c110 con 0x7fbf60004800 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.713+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6000c310 con 0x7fbf600039b0 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf80a00640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf7be00640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600039b0 0x7fbf60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf73400640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf80a00640 1 -- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600039b0 msgr2=0x7fbf60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf80a00640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600039b0 0x7fbf60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf80a00640 1 -- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf60004800 msgr2=0x7fbf60004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf80a00640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf60004800 0x7fbf60004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.714+0000 7fbf80a00640 1 -- 172.21.15.6:0/218777669 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf60003520 con 0x7fbf600051c0 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.716+0000 7fbf80a00640 1 -- 172.21.15.6:0/218777669 reap_dead start 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.716+0000 7fbf80a00640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0d3be0 tx=0x7fbf60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6e48bec51b8893 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.716+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf74036020 con 0x7fbf600051c0 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.716+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf74032630 con 0x7fbf600051c0 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.716+0000 7fbf7b400640 1 -- 172.21.15.6:0/218777669 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400b610 con 0x7fbf600051c0 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 -- 172.21.15.6:0/218777669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf600051c0 msgr2=0x7fbf6000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0d3be0 tx=0x7fbf60005830 comp rx=0 tx=0).stop 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 -- 172.21.15.6:0/218777669 shutdown_connections 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 --2- 172.21.15.6:0/218777669 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf600051c0 0x7fbf6000b9c0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 -- 172.21.15.6:0/218777669 >> 172.21.15.6:0/218777669 conn(0x7fbf7c08d2a0 msgr2=0x7fbf7c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 -- 172.21.15.6:0/218777669 shutdown_connections 2024-03-06T07:35:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.717+0000 7fbf81400640 1 -- 172.21.15.6:0/218777669 wait complete. 2024-03-06T07:35:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.718+0000 7fbf81400640 1 Processor -- start 2024-03-06T07:35:13.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.719+0000 7fbf81400640 1 -- start start 2024-03-06T07:35:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.719+0000 7fbf81400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600051c0 0x7fbf7c149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf81400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 0x7fbf7c0ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf81400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf7c15f590 0x7fbf7c15fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf81400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0d45b0 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf81400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0d2440 con 0x7fbf7c15f590 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf81400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7c0d4730 con 0x7fbf600051c0 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 0x7fbf7c0ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 0x7fbf7c0ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60252/0 (socket says 172.21.15.6:60252) 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 -- 172.21.15.6:0/2782702282 learned_addr learned my addr 172.21.15.6:0/2782702282 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf7be00640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600051c0 0x7fbf7c149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf80a00640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf7c15f590 0x7fbf7c15fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 -- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600051c0 msgr2=0x7fbf7c149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600051c0 0x7fbf7c149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 -- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf7c15f590 msgr2=0x7fbf7c15fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf7c15f590 0x7fbf7c15fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf73400640 1 -- 172.21.15.6:0/2782702282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7c15e4f0 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.720+0000 7fbf80a00640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf7c15f590 0x7fbf7c15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.721+0000 7fbf73400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 0x7fbf7c0ca280 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fbf64007370 tx=0x7fbf64007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.721+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6401a070 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.721+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6400f940 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.721+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6400fd30 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.721+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf7c15e7d0 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.721+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7c15ed90 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.722+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7c04f330 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.723+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf6401d070 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.725+0000 7fbf78c00640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf5806b4d0 0x7fbf5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.725+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbf64089c80 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.725+0000 7fbf7be00640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf5806b4d0 0x7fbf5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:13.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.726+0000 7fbf7be00640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf5806b4d0 0x7fbf5806d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c000f80 tx=0x7fbf6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:13.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.729+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf640589d0 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.963+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf7c0c75c0 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.964+0000 7fbf78c00640 1 -- 172.21.15.6:0/2782702282 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7fbf640582d0 con 0x7fbf7c0c9e00 2024-03-06T07:35:13.966 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:13.966 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:13.966 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf5806b4d0 msgr2=0x7fbf5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf5806b4d0 0x7fbf5806d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c000f80 tx=0x7fbf6c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 msgr2=0x7fbf7c0ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 0x7fbf7c0ca280 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fbf64007370 tx=0x7fbf64007830 comp rx=0 tx=0).stop 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 shutdown_connections 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf5806b4d0 0x7fbf5806d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf7c15f590 0x7fbf7c15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf7c0c9e00 0x7fbf7c0ca280 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 --2- 172.21.15.6:0/2782702282 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf600051c0 0x7fbf7c149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 >> 172.21.15.6:0/2782702282 conn(0x7fbf7c08d2a0 msgr2=0x7fbf7c0ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.966+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 shutdown_connections 2024-03-06T07:35:13.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:13.967+0000 7fbf81400640 1 -- 172.21.15.6:0/2782702282 wait complete. 2024-03-06T07:35:13.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:14.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.183+0000 7f5dffa00640 1 Processor -- start 2024-03-06T07:35:14.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.184+0000 7f5dffa00640 1 -- start start 2024-03-06T07:35:14.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.185+0000 7f5dffa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df815e240 0x7f5df80b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.185+0000 7f5dffa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5df8162450 0x7f5df815f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.185+0000 7f5dffa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5df80b7260 con 0x7f5df81628e0 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.185+0000 7f5dffa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5df80b6ca0 con 0x7f5df8162450 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.185+0000 7f5dffa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df80b6f80 con 0x7f5df815e240 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5df8162450 0x7f5df815f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5df8162450 0x7f5df815f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60268/0 (socket says 172.21.15.6:60268) 2024-03-06T07:35:14.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 -- 172.21.15.6:0/3268825670 learned_addr learned my addr 172.21.15.6:0/3268825670 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfd000640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df815e240 0x7f5df80b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 -- 172.21.15.6:0/3268825670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df815e240 msgr2=0x7f5df80b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df815e240 0x7f5df80b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 -- 172.21.15.6:0/3268825670 >> v1:172.21.15.6:6789/0 conn(0x7f5df81628e0 legacy=0x7f5df8161d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfda00640 1 -- 172.21.15.6:0/3268825670 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df815e950 con 0x7f5df8162450 2024-03-06T07:35:14.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.186+0000 7f5dfd000640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df815e240 0x7f5df80b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:14.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.188+0000 7f5dfda00640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df8162450 0x7f5df815f150 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009e50 tx=0x7f5df002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220129b16cac4657 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.188+0000 7f5df7400640 1 -- 172.21.15.6:0/3268825670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df002cb00 con 0x7f5df8162450 2024-03-06T07:35:14.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.188+0000 7f5df7400640 1 -- 172.21.15.6:0/3268825670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df002cca0 con 0x7f5df8162450 2024-03-06T07:35:14.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5df7400640 1 -- 172.21.15.6:0/3268825670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0034610 con 0x7f5df8162450 2024-03-06T07:35:14.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 -- 172.21.15.6:0/3268825670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df8162450 msgr2=0x7f5df815f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df8162450 0x7f5df815f150 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f5df0009e50 tx=0x7f5df002bf60 comp rx=0 tx=0).stop 2024-03-06T07:35:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 -- 172.21.15.6:0/3268825670 shutdown_connections 2024-03-06T07:35:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df8162450 0x7f5df815f150 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 --2- 172.21.15.6:0/3268825670 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df815e240 0x7f5df80b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 -- 172.21.15.6:0/3268825670 >> 172.21.15.6:0/3268825670 conn(0x7f5df80b7450 msgr2=0x7f5df809aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 -- 172.21.15.6:0/3268825670 shutdown_connections 2024-03-06T07:35:14.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.189+0000 7f5dffa00640 1 -- 172.21.15.6:0/3268825670 wait complete. 2024-03-06T07:35:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.191+0000 7f5dffa00640 1 Processor -- start 2024-03-06T07:35:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.191+0000 7f5dffa00640 1 -- start start 2024-03-06T07:35:14.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dffa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 0x7f5df8157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dffa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5df8162450 0x7f5df80b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dffa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df81628e0 0x7f5df809df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dffa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df80b68e0 con 0x7f5df815e240 2024-03-06T07:35:14.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dffa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df815e620 con 0x7f5df8162450 2024-03-06T07:35:14.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dffa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df80b6a60 con 0x7f5df81628e0 2024-03-06T07:35:14.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dfd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 0x7f5df8157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dfd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 0x7f5df8157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60278/0 (socket says 172.21.15.6:60278) 2024-03-06T07:35:14.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.192+0000 7f5dfd000640 1 -- 172.21.15.6:0/623861408 learned_addr learned my addr 172.21.15.6:0/623861408 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dfd000640 1 -- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df81628e0 msgr2=0x7f5df809df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dfd000640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df81628e0 0x7f5df809df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dfd000640 1 -- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5df8162450 msgr2=0x7f5df80b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dfd000640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5df8162450 0x7f5df80b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dfd000640 1 -- 172.21.15.6:0/623861408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df81f4ce0 con 0x7f5df815e240 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dfd000640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 0x7f5df8157a70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c7f0 tx=0x7f5dec00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec015070 con 0x7f5df815e240 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dec007a30 con 0x7f5df815e240 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec007de0 con 0x7f5df815e240 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df80b1920 con 0x7f5df815e240 2024-03-06T07:35:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.193+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df80b15d0 con 0x7f5df815e240 2024-03-06T07:35:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.195+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df804f3a0 con 0x7f5df815e240 2024-03-06T07:35:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.196+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5dec018070 con 0x7f5df815e240 2024-03-06T07:35:14.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.196+0000 7f5df4c00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5dd806b500 0x7f5dd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.197+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5dec08a330 con 0x7f5df815e240 2024-03-06T07:35:14.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.197+0000 7f5df7e00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5dd806b500 0x7f5dd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.198+0000 7f5df7e00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5dd806b500 0x7f5dd806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5de4000f60 tx=0x7f5de40079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.206+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5dec059080 con 0x7f5df815e240 2024-03-06T07:35:14.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.493+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5df815cd80 con 0x7f5df815e240 2024-03-06T07:35:14.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.494+0000 7f5df4c00640 1 -- 172.21.15.6:0/623861408 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4978 (secure 0 0 0) 0x7f5dec058980 con 0x7f5df815e240 2024-03-06T07:35:14.497 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:14.497 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":84,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:33:48.441811+0000","modified":"2024-03-06T07:34:08.409742+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":4,"addr":"172.21.15.6:6835/3719236593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":3719236593},{"type":"v1","addr":"172.21.15.6:6835","nonce":3719236593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":8}]} 2024-03-06T07:35:14.497 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 84 2024-03-06T07:35:14.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5dd806b500 msgr2=0x7f5dd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5dd806b500 0x7f5dd806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5de4000f60 tx=0x7f5de40079d0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 msgr2=0x7f5df8157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 0x7f5df8157a70 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00c7f0 tx=0x7f5dec00cbf0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 shutdown_connections 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5dd806b500 0x7f5dd806d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5df8162450 0x7f5df80b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5df81628e0 0x7f5df809df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 --2- 172.21.15.6:0/623861408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5df815e240 0x7f5df8157a70 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 >> 172.21.15.6:0/623861408 conn(0x7f5df80b7450 msgr2=0x7f5df80d2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 shutdown_connections 2024-03-06T07:35:14.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.497+0000 7f5dffa00640 1 -- 172.21.15.6:0/623861408 wait complete. 2024-03-06T07:35:14.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:14.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.735+0000 7f1551400640 1 Processor -- start 2024-03-06T07:35:14.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.735+0000 7f1551400640 1 -- start start 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.737+0000 7f1551400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f154c06afb0 0x7f154c06b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.737+0000 7f1551400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c159cc0 con 0x7f154c1598e0 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.737+0000 7f1551400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0729b0 con 0x7f154c0cd760 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.738+0000 7f1551400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c06e3f0 con 0x7f154c06afb0 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.738+0000 7f1550a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f154c06afb0 0x7f154c06b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.738+0000 7f154be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f154c1598e0 0x7f154c06f620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34704/0 (socket says 172.21.15.6:34704) 2024-03-06T07:35:14.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.738+0000 7f154be00640 1 -- 172.21.15.6:0/949765962 learned_addr learned my addr 172.21.15.6:0/949765962 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:14.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.738+0000 7f1550a00640 1 -- 172.21.15.6:0/949765962 >> v1:172.21.15.178:6789/0 conn(0x7f154c0cd760 legacy=0x7f154c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.738+0000 7f1550a00640 1 -- 172.21.15.6:0/949765962 >> v1:172.21.15.6:6789/0 conn(0x7f154c1598e0 legacy=0x7f154c06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.739+0000 7f154b400640 1 -- 172.21.15.6:0/949765962 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603977024 0 0) 0x7f154c159cc0 con 0x7f154c1598e0 2024-03-06T07:35:14.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.739+0000 7f1550a00640 1 -- 172.21.15.6:0/949765962 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c15db40 con 0x7f154c06afb0 2024-03-06T07:35:14.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.740+0000 7f1550a00640 1 --2- 172.21.15.6:0/949765962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c06b440 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f154c06e570 tx=0x7f154402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801f268079b6eff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.740+0000 7f154b400640 1 -- 172.21.15.6:0/949765962 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544039070 con 0x7f154c06afb0 2024-03-06T07:35:14.743 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.740+0000 7f154b400640 1 -- 172.21.15.6:0/949765962 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154402cce0 con 0x7f154c06afb0 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.740+0000 7f154b400640 1 -- 172.21.15.6:0/949765962 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544034660 con 0x7f154c06afb0 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 -- 172.21.15.6:0/949765962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 msgr2=0x7f154c06b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 --2- 172.21.15.6:0/949765962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c06b440 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f154c06e570 tx=0x7f154402bf30 comp rx=0 tx=0).stop 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 -- 172.21.15.6:0/949765962 shutdown_connections 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 --2- 172.21.15.6:0/949765962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c06b440 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 -- 172.21.15.6:0/949765962 >> 172.21.15.6:0/949765962 conn(0x7f154c06c0c0 msgr2=0x7f154c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 -- 172.21.15.6:0/949765962 shutdown_connections 2024-03-06T07:35:14.744 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.741+0000 7f1551400640 1 -- 172.21.15.6:0/949765962 wait complete. 2024-03-06T07:35:14.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.743+0000 7f1551400640 1 Processor -- start 2024-03-06T07:35:14.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.743+0000 7f1551400640 1 -- start start 2024-03-06T07:35:14.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.744+0000 7f1551400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.744+0000 7f1551400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f154c0cd760 0x7f154c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.744+0000 7f1551400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f154c1598e0 0x7f154c158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.744+0000 7f1551400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c06dda0 con 0x7f154c06afb0 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.744+0000 7f1551400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c072f30 con 0x7f154c0cd760 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.744+0000 7f1551400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c06e080 con 0x7f154c1598e0 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60292/0 (socket says 172.21.15.6:60292) 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 -- 172.21.15.6:0/2514343578 learned_addr learned my addr 172.21.15.6:0/2514343578 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f1550a00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f154c1598e0 0x7f154c158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f1543e00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f154c0cd760 0x7f154c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 -- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f154c1598e0 msgr2=0x7f154c158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f154c1598e0 0x7f154c158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 -- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f154c0cd760 msgr2=0x7f154c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f154c0cd760 0x7f154c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f154be00640 1 -- 172.21.15.6:0/2514343578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0b5600 con 0x7f154c06afb0 2024-03-06T07:35:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.745+0000 7f1543e00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f154c0cd760 0x7f154c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:14.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.746+0000 7f154be00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c154a50 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c420 tx=0x7f153c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.746+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0074e0 con 0x7f154c06afb0 2024-03-06T07:35:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.746+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c0047e0 con 0x7f154c06afb0 2024-03-06T07:35:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.746+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c004bd0 con 0x7f154c06afb0 2024-03-06T07:35:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.746+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c1fb300 con 0x7f154c06afb0 2024-03-06T07:35:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.746+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c1fb8c0 con 0x7f154c06afb0 2024-03-06T07:35:14.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.747+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c1fbac0 con 0x7f154c06afb0 2024-03-06T07:35:14.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.749+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f153c011070 con 0x7f154c06afb0 2024-03-06T07:35:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.750+0000 7f1548c00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f152806b500 0x7f152806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.750+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f153c08a550 con 0x7f154c06afb0 2024-03-06T07:35:14.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.750+0000 7f1543e00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f152806b500 0x7f152806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:14.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.752+0000 7f1543e00640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f152806b500 0x7f152806d9c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f154c0b5a90 tx=0x7f1534009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:14.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.758+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f154c1fbac0 con 0x7f154c06afb0 2024-03-06T07:35:14.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.995+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f154c04f330 con 0x7f154c06afb0 2024-03-06T07:35:14.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:14.998+0000 7f1548c00640 1 -- 172.21.15.6:0/2514343578 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16775 (secure 0 0 0) 0x7f153c00e280 con 0x7f154c06afb0 2024-03-06T07:35:15.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:15.001 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":114,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:34:11.424305+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:33:46.337716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:33:47.450154+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":2}]},{"pgid":"17.30","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6833/4114489444":"2024-03-07T07:34:04.377501+0000","172.21.15.6:6832/4114489444":"2024-03-07T07:34:04.377501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f152806b500 msgr2=0x7f152806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f152806b500 0x7f152806d9c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f154c0b5a90 tx=0x7f1534009040 comp rx=0 tx=0).stop 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 msgr2=0x7f154c154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c154a50 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c420 tx=0x7f153c00c8e0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 shutdown_connections 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f152806b500 0x7f152806d9c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f154c0cd760 0x7f154c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f154c1598e0 0x7f154c158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 --2- 172.21.15.6:0/2514343578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f154c06afb0 0x7f154c154a50 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.000+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 >> 172.21.15.6:0/2514343578 conn(0x7f154c06c0c0 msgr2=0x7f154c06b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.001+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 shutdown_connections 2024-03-06T07:35:15.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.001+0000 7f1551400640 1 -- 172.21.15.6:0/2514343578 wait complete. 2024-03-06T07:35:15.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:35:15.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.236+0000 7f76d6600640 1 Processor -- start 2024-03-06T07:35:15.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.236+0000 7f76d6600640 1 -- start start 2024-03-06T07:35:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.237+0000 7f76d6600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f76d00af1a0 0x7f76d015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.237+0000 7f76d6600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 0x7f76d00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.237+0000 7f76d6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d015cc40 con 0x7f76d00af580 2024-03-06T07:35:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.237+0000 7f76d6600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d015cf20 con 0x7f76d00af1a0 2024-03-06T07:35:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.237+0000 7f76d6600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d015d200 con 0x7f76d00cdc20 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.238+0000 7f76cea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76d00af580 0x7f76d00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34710/0 (socket says 172.21.15.6:34710) 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.238+0000 7f76cea00640 1 -- 172.21.15.6:0/2760133115 learned_addr learned my addr 172.21.15.6:0/2760133115 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.238+0000 7f76cfe00640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 0x7f76d00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.238+0000 7f76cf400640 1 --2- 172.21.15.6:0/2760133115 >> v2:172.21.15.178:3300/0 conn(0x7f76d00af1a0 0x7f76d015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.238+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714878124 0 0) 0x7f76d015cc40 con 0x7f76d00af580 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.238+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b4003520 con 0x7f76d00af580 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377029306 0 0) 0x7f76b4003520 con 0x7f76d00af580 2024-03-06T07:35:15.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 msgr2=0x7f76d00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 0x7f76d00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 >> v2:172.21.15.178:3300/0 conn(0x7f76d00af1a0 msgr2=0x7f76d015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 --2- 172.21.15.6:0/2760133115 >> v2:172.21.15.178:3300/0 conn(0x7f76d00af1a0 0x7f76d015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d015d4e0 con 0x7f76d00af580 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f76bc001670 con 0x7f76d00af580 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 >> v1:172.21.15.6:6789/0 conn(0x7f76d00af580 legacy=0x7f76d00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 0x7f76b4003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76b4003fa0 0x7f76b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76b4004d40 0x7f76b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d015cf20 con 0x7f76b4004d40 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d015d200 con 0x7f76b4003fa0 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b400ba70 con 0x7f76d00cdc20 2024-03-06T07:35:15.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.239+0000 7f76cfe00640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76b4004d40 0x7f76b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cf400640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 0x7f76b4003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cea00640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76b4003fa0 0x7f76b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cfe00640 1 -- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 msgr2=0x7f76b4003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cfe00640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d00cdc20 0x7f76b4003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cfe00640 1 -- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76b4003fa0 msgr2=0x7f76b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cfe00640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76b4003fa0 0x7f76b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.240+0000 7f76cfe00640 1 -- 172.21.15.6:0/2760133115 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b4003520 con 0x7f76b4004d40 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.241+0000 7f76cfe00640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76b4004d40 0x7f76b40051c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009ac0 tx=0x7f76b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e28ec4188e46fa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.241+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8039070 con 0x7f76b4004d40 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.241+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c802c6b0 con 0x7f76b4004d40 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.241+0000 7f76ce000640 1 -- 172.21.15.6:0/2760133115 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8009690 con 0x7f76b4004d40 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76d6600640 1 -- 172.21.15.6:0/2760133115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76b4004d40 msgr2=0x7f76b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76d6600640 1 --2- 172.21.15.6:0/2760133115 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76b4004d40 0x7f76b40051c0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009ac0 tx=0x7f76b4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76cfe00640 1 -- 172.21.15.6:0/2760133115 reap_dead start 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76d6600640 1 -- 172.21.15.6:0/2760133115 shutdown_connections 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76d6600640 1 -- 172.21.15.6:0/2760133115 >> 172.21.15.6:0/2760133115 conn(0x7f76d006afb0 msgr2=0x7f76d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76d6600640 1 -- 172.21.15.6:0/2760133115 shutdown_connections 2024-03-06T07:35:15.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.242+0000 7f76d6600640 1 -- 172.21.15.6:0/2760133115 wait complete. 2024-03-06T07:35:15.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.244+0000 7f76d6600640 1 Processor -- start 2024-03-06T07:35:15.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.244+0000 7f76d6600640 1 -- start start 2024-03-06T07:35:15.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76d6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 0x7f76d009b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76d6600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d015ab40 0x7f76d0151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76d6600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76d01524c0 0x7f76d00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76d6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00b3160 con 0x7f76d014f430 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76d6600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00b2fe0 con 0x7f76d01524c0 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76d6600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d00b32e0 con 0x7f76d015ab40 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 0x7f76d009b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 0x7f76d009b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60310/0 (socket says 172.21.15.6:60310) 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.245+0000 7f76cf400640 1 -- 172.21.15.6:0/3470147006 learned_addr learned my addr 172.21.15.6:0/3470147006 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cea00640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d015ab40 0x7f76d0151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cf400640 1 -- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d015ab40 msgr2=0x7f76d0151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cf400640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d015ab40 0x7f76d0151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cfe00640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76d01524c0 0x7f76d00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cf400640 1 -- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76d01524c0 msgr2=0x7f76d00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cf400640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76d01524c0 0x7f76d00c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cf400640 1 -- 172.21.15.6:0/3470147006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d0152940 con 0x7f76d014f430 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.246+0000 7f76cf400640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 0x7f76d009b3a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f76c0007f60 tx=0x7f76c0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.247+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c001a020 con 0x7f76d014f430 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.247+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c0016470 con 0x7f76d014f430 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.247+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0016860 con 0x7f76d014f430 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.247+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d009ab40 con 0x7f76d014f430 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.247+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d00ca300 con 0x7f76d014f430 2024-03-06T07:35:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.248+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d0069510 con 0x7f76d014f430 2024-03-06T07:35:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.250+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76c0005d10 con 0x7f76d014f430 2024-03-06T07:35:15.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.251+0000 7f76c7400640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76b006b500 0x7f76b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.251+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f76c008b8e0 con 0x7f76d014f430 2024-03-06T07:35:15.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.251+0000 7f76cea00640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76b006b500 0x7f76b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.252+0000 7f76cea00640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76b006b500 0x7f76b006d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f76d015aa80 tx=0x7f76bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.254+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f76c0024030 con 0x7f76d014f430 2024-03-06T07:35:15.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.498+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f76d004f3a0 con 0x7f76d014f430 2024-03-06T07:35:15.502 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-03-06T07:35:15.500+0000 7f7241e00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:35:15.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.503+0000 7f76c7400640 1 -- 172.21.15.6:0/3470147006 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) v1 ==== 161+0+0 (secure 0 0 0) 0x7f76c005a740 con 0x7f76d014f430 2024-03-06T07:35:15.505 INFO:teuthology.orchestra.run.smithi006.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:35:15.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76b006b500 msgr2=0x7f76b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76b006b500 0x7f76b006d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f76d015aa80 tx=0x7f76bc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:15.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 msgr2=0x7f76d009b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 0x7f76d009b3a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f76c0007f60 tx=0x7f76c0007f90 comp rx=0 tx=0).stop 2024-03-06T07:35:15.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 shutdown_connections 2024-03-06T07:35:15.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f76b006b500 0x7f76b006d9c0 secure :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f76d015aa80 tx=0x7f76bc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:15.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76d01524c0 0x7f76d00c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76d015ab40 0x7f76d0151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 --2- 172.21.15.6:0/3470147006 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76d014f430 0x7f76d009b3a0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.510 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.507+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 >> 172.21.15.6:0/3470147006 conn(0x7f76d006afb0 msgr2=0x7f76d00b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.511 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.510+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 shutdown_connections 2024-03-06T07:35:15.512 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.510+0000 7f76d6600640 1 -- 172.21.15.6:0/3470147006 wait complete. 2024-03-06T07:35:15.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:35:15.537 INFO:tasks.ceph.mds.c.smithi006.stderr: -7503> 2024-03-06T07:34:11.399+0000 7f19f1400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:15.603 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:35:15.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.720+0000 7f3ff1000640 1 Processor -- start 2024-03-06T07:35:15.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.720+0000 7f3ff1000640 1 -- start start 2024-03-06T07:35:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3ff1000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3fec072520 0x7f3fec0cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3ff1000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 0x7f3fec0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3ff1000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fec072df0 con 0x7f3fec15bf20 2024-03-06T07:35:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3ff1000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fec0cdab0 con 0x7f3fec072520 2024-03-06T07:35:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3ff1000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b4d70 con 0x7f3fec071e20 2024-03-06T07:35:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3fe9e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3fec15bf20 0x7f3fec0acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34720/0 (socket says 172.21.15.6:34720) 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3fe9e00640 1 -- 172.21.15.6:0/2214707899 learned_addr learned my addr 172.21.15.6:0/2214707899 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/2214707899 >> v2:172.21.15.178:3300/0 conn(0x7f3fec072520 0x7f3fec0cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.722+0000 7f3fea800640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 0x7f3fec0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563663259 0 0) 0x7f3fec072df0 con 0x7f3fec15bf20 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd0003520 con 0x7f3fec15bf20 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740804053 0 0) 0x7f3fd0003520 con 0x7f3fec15bf20 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 msgr2=0x7f3fec0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 0x7f3fec0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 >> v2:172.21.15.178:3300/0 conn(0x7f3fec072520 msgr2=0x7f3fec0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 --2- 172.21.15.6:0/2214707899 >> v2:172.21.15.178:3300/0 conn(0x7f3fec072520 0x7f3fec0cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec0b5050 con 0x7f3fec15bf20 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f3fdc001670 con 0x7f3fec15bf20 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 >> v1:172.21.15.6:6789/0 conn(0x7f3fec15bf20 legacy=0x7f3fec0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 0x7f3fd00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fd0003f30 0x7f3fd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fd0004d40 0x7f3fd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0cdab0 con 0x7f3fd0003f30 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b4d70 con 0x7f3fd0004d40 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd000ba10 con 0x7f3fec071e20 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.723+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/2214707899 >> v2:172.21.15.178:3300/0 conn(0x7f3fec072520 0x7f3fec0cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:15.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fd0003f30 0x7f3fd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe9e00640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 0x7f3fd00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe3e00640 1 -- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 msgr2=0x7f3fd00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fec071e20 0x7f3fd00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe3e00640 1 -- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fd0004d40 msgr2=0x7f3fd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:15.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fd0004d40 0x7f3fd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.724+0000 7f3fe3e00640 1 -- 172.21.15.6:0/2214707899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0003520 con 0x7f3fd0003f30 2024-03-06T07:35:15.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.725+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fd0003f30 0x7f3fd0004800 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40098a0 tx=0x7f3fd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7b2b4376c10de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.725+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd400f040 con 0x7f3fd0003f30 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.725+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd4037600 con 0x7f3fd0003f30 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.725+0000 7f3fe9400640 1 -- 172.21.15.6:0/2214707899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40379b0 con 0x7f3fd0003f30 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3ff1000640 1 -- 172.21.15.6:0/2214707899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fd0003f30 msgr2=0x7f3fd0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3ff1000640 1 --2- 172.21.15.6:0/2214707899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fd0003f30 0x7f3fd0004800 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40098a0 tx=0x7f3fd0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3fea800640 1 -- 172.21.15.6:0/2214707899 reap_dead start 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3ff1000640 1 -- 172.21.15.6:0/2214707899 shutdown_connections 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3ff1000640 1 -- 172.21.15.6:0/2214707899 >> 172.21.15.6:0/2214707899 conn(0x7f3fec06b3d0 msgr2=0x7f3fec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3ff1000640 1 -- 172.21.15.6:0/2214707899 shutdown_connections 2024-03-06T07:35:15.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.726+0000 7f3ff1000640 1 -- 172.21.15.6:0/2214707899 wait complete. 2024-03-06T07:35:15.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.728+0000 7f3ff1000640 1 Processor -- start 2024-03-06T07:35:15.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.728+0000 7f3ff1000640 1 -- start start 2024-03-06T07:35:15.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3ff1000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fd0003f30 0x7f3fec155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3ff1000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 0x7f3fec14ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3ff1000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec157d50 0x7f3fec09cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3ff1000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0ad450 con 0x7f3fec09b6e0 2024-03-06T07:35:15.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3ff1000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec0b4990 con 0x7f3fec157d50 2024-03-06T07:35:15.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3ff1000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec154e90 con 0x7f3fd0003f30 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3fe3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 0x7f3fec14ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3fe3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 0x7f3fec14ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60348/0 (socket says 172.21.15.6:60348) 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.729+0000 7f3fe3e00640 1 -- 172.21.15.6:0/697855183 learned_addr learned my addr 172.21.15.6:0/697855183 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe9e00640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fd0003f30 0x7f3fec155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe3e00640 1 -- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fd0003f30 msgr2=0x7f3fec155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fd0003f30 0x7f3fec155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe3e00640 1 -- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec157d50 msgr2=0x7f3fec09cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec157d50 0x7f3fec09cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe3e00640 1 -- 172.21.15.6:0/697855183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fec155860 con 0x7f3fec09b6e0 2024-03-06T07:35:15.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe3e00640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 0x7f3fec14ed50 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f3fd402bff0 tx=0x7f3fd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.730+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4006dd0 con 0x7f3fec09b6e0 2024-03-06T07:35:15.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.731+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd402f9b0 con 0x7f3fec09b6e0 2024-03-06T07:35:15.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.731+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd402fda0 con 0x7f3fec09b6e0 2024-03-06T07:35:15.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.731+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec14f290 con 0x7f3fec09b6e0 2024-03-06T07:35:15.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.731+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec154840 con 0x7f3fec09b6e0 2024-03-06T07:35:15.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.732+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec1587b0 con 0x7f3fec09b6e0 2024-03-06T07:35:15.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.733+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fd4042440 con 0x7f3fec09b6e0 2024-03-06T07:35:15.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.734+0000 7f3fe2000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3fc806b550 0x7f3fc806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:15.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.734+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f3fd40ac5f0 con 0x7f3fec09b6e0 2024-03-06T07:35:15.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.735+0000 7f3fe9e00640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3fc806b550 0x7f3fc806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:15.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.736+0000 7f3fe9e00640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3fc806b550 0x7f3fc806da10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc000f60 tx=0x7f3fdc000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:15.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.738+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fd4033030 con 0x7f3fec09b6e0 2024-03-06T07:35:15.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:15.973+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fec04f330 con 0x7f3fec09b6e0 2024-03-06T07:35:16.509 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.506+0000 7f3fe2000640 1 -- 172.21.15.6:0/697855183 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3fd407b460 con 0x7f3fec09b6e0 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.510+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3fc806b550 msgr2=0x7f3fc806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.510+0000 7f3ff1000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3fc806b550 0x7f3fc806da10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc000f60 tx=0x7f3fdc000ed0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.510+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 msgr2=0x7f3fec14ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.510+0000 7f3ff1000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 0x7f3fec14ed50 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f3fd402bff0 tx=0x7f3fd4004730 comp rx=0 tx=0).stop 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 shutdown_connections 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3fc806b550 0x7f3fc806da10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec157d50 0x7f3fec09cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3fec09b6e0 0x7f3fec14ed50 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 --2- 172.21.15.6:0/697855183 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3fd0003f30 0x7f3fec155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.513 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 >> 172.21.15.6:0/697855183 conn(0x7f3fec06b3d0 msgr2=0x7f3fec0acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 shutdown_connections 2024-03-06T07:35:16.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.511+0000 7f3ff1000640 1 -- 172.21.15.6:0/697855183 wait complete. 2024-03-06T07:35:16.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:35:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.803+0000 7f4490c00640 1 Processor -- start 2024-03-06T07:35:16.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.804+0000 7f4490c00640 1 -- start start 2024-03-06T07:35:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.805+0000 7f4490c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 0x7f448c0d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.805+0000 7f4490c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f448c06b410 0x7f448c0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.805+0000 7f4490c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c0b5040 con 0x7f448c06afb0 2024-03-06T07:35:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.805+0000 7f4490c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0b52e0 con 0x7f448c06b410 2024-03-06T07:35:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.805+0000 7f4490c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0b55c0 con 0x7f448c145620 2024-03-06T07:35:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.806+0000 7f448b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 0x7f448c0d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.806+0000 7f448be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f448c06b410 0x7f448c0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.806+0000 7f448b400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 0x7f448c0d03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:52730/0 (socket says 172.21.15.6:52730) 2024-03-06T07:35:16.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.806+0000 7f448b400640 1 -- 172.21.15.6:0/60026156 learned_addr learned my addr 172.21.15.6:0/60026156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.807+0000 7f448be00640 1 -- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 msgr2=0x7f448c0d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.807+0000 7f448be00640 1 --2- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 0x7f448c0d03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.807+0000 7f448be00640 1 -- 172.21.15.6:0/60026156 >> v1:172.21.15.178:6789/0 conn(0x7f448c06afb0 legacy=0x7f448c15cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.807+0000 7f448be00640 1 -- 172.21.15.6:0/60026156 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c0b58a0 con 0x7f448c06b410 2024-03-06T07:35:16.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.807+0000 7f448b400640 1 --2- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 0x7f448c0d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:16.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.809+0000 7f448be00640 1 --2- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c06b410 0x7f448c0b4b00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f44840097a0 tx=0x7f448402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43bc6575fc620230 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.809+0000 7f448a000640 1 -- 172.21.15.6:0/60026156 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840315a0 con 0x7f448c06b410 2024-03-06T07:35:16.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f448a000640 1 -- 172.21.15.6:0/60026156 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4484037070 con 0x7f448c06b410 2024-03-06T07:35:16.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f448a000640 1 -- 172.21.15.6:0/60026156 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484031960 con 0x7f448c06b410 2024-03-06T07:35:16.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 -- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c06b410 msgr2=0x7f448c0b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 --2- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c06b410 0x7f448c0b4b00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f44840097a0 tx=0x7f448402b840 comp rx=0 tx=0).stop 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 -- 172.21.15.6:0/60026156 shutdown_connections 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 --2- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c06b410 0x7f448c0b4b00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 --2- 172.21.15.6:0/60026156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c145620 0x7f448c0d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 -- 172.21.15.6:0/60026156 >> 172.21.15.6:0/60026156 conn(0x7f448c0b27f0 msgr2=0x7f448c0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 -- 172.21.15.6:0/60026156 shutdown_connections 2024-03-06T07:35:16.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.810+0000 7f4490c00640 1 -- 172.21.15.6:0/60026156 wait complete. 2024-03-06T07:35:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.812+0000 7f4490c00640 1 Processor -- start 2024-03-06T07:35:16.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.812+0000 7f4490c00640 1 -- start start 2024-03-06T07:35:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f4490c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c06afb0 0x7f448c15b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f4490c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 0x7f448c0aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f4490c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c145620 0x7f448c1f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f4490c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c15c280 con 0x7f448c06b410 2024-03-06T07:35:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f4490c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c06b8a0 con 0x7f448c145620 2024-03-06T07:35:16.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f4490c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c15d1c0 con 0x7f448c06afb0 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.813+0000 7f448aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 0x7f448c0aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 0x7f448c0aaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60352/0 (socket says 172.21.15.6:60352) 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 -- 172.21.15.6:0/261220222 learned_addr learned my addr 172.21.15.6:0/261220222 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448b400640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c06afb0 0x7f448c15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448be00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c145620 0x7f448c1f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 -- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c06afb0 msgr2=0x7f448c15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c06afb0 0x7f448c15b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 -- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c145620 msgr2=0x7f448c1f4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:16.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c145620 0x7f448c1f4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 -- 172.21.15.6:0/261220222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c15bb60 con 0x7f448c06b410 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.814+0000 7f448aa00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 0x7f448c0aaed0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f447400c790 tx=0x7f447400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.815+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474016020 con 0x7f448c06b410 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.815+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474006d50 con 0x7f448c06b410 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.815+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c0ab990 con 0x7f448c06b410 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.815+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474003bb0 con 0x7f448c06b410 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.815+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c0abf50 con 0x7f448c06b410 2024-03-06T07:35:16.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.816+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448c062c00 con 0x7f448c06b410 2024-03-06T07:35:16.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.817+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f447400a850 con 0x7f448c06b410 2024-03-06T07:35:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.818+0000 7f4483400640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f446806b4d0 0x7f446806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:16.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.818+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f4474089f60 con 0x7f448c06b410 2024-03-06T07:35:16.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.819+0000 7f448b400640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f446806b4d0 0x7f446806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:16.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.820+0000 7f448b400640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f446806b4d0 0x7f446806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f448c1f4e70 tx=0x7f447c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:16.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:16.823+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f448c062c00 con 0x7f448c06b410 2024-03-06T07:35:17.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.017+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f448c0b20c0 con 0x7f448c06b410 2024-03-06T07:35:17.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.910+0000 7f4483400640 1 -- 172.21.15.6:0/261220222 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4474020020 con 0x7f448c06b410 2024-03-06T07:35:17.912 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f446806b4d0 msgr2=0x7f446806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f446806b4d0 0x7f446806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f448c1f4e70 tx=0x7f447c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 msgr2=0x7f448c0aaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 0x7f448c0aaed0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f447400c790 tx=0x7f447400cc50 comp rx=0 tx=0).stop 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 shutdown_connections 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f446806b4d0 0x7f446806d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f448c145620 0x7f448c1f4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f448c06b410 0x7f448c0aaed0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 --2- 172.21.15.6:0/261220222 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f448c06afb0 0x7f448c15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:17.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 >> 172.21.15.6:0/261220222 conn(0x7f448c0b27f0 msgr2=0x7f448c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:17.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 shutdown_connections 2024-03-06T07:35:17.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:17.914+0000 7f4490c00640 1 -- 172.21.15.6:0/261220222 wait complete. 2024-03-06T07:35:17.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:35:18.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.137+0000 7fe5d3e00640 1 Processor -- start 2024-03-06T07:35:18.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.137+0000 7fe5d3e00640 1 -- start start 2024-03-06T07:35:18.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5d4145570 0x7fe5d40cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe5d4071450 0x7fe5d406daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d3e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d4071c50 con 0x7fe5d406e030 2024-03-06T07:35:18.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d406e4a0 con 0x7fe5d4145570 2024-03-06T07:35:18.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d3e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d4071f30 con 0x7fe5d4071450 2024-03-06T07:35:18.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5d4145570 0x7fe5d40cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe5d4145570 0x7fe5d40cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60362/0 (socket says 172.21.15.6:60362) 2024-03-06T07:35:18.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.139+0000 7fe5d2a00640 1 -- 172.21.15.6:0/4144815889 learned_addr learned my addr 172.21.15.6:0/4144815889 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:18.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.140+0000 7fe5d2000640 1 --2- 172.21.15.6:0/4144815889 >> v2:172.21.15.178:3300/0 conn(0x7fe5d4071450 0x7fe5d406daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.140+0000 7fe5d2a00640 1 -- 172.21.15.6:0/4144815889 >> v2:172.21.15.178:3300/0 conn(0x7fe5d4071450 msgr2=0x7fe5d406daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.140+0000 7fe5d2a00640 1 --2- 172.21.15.6:0/4144815889 >> v2:172.21.15.178:3300/0 conn(0x7fe5d4071450 0x7fe5d406daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.140+0000 7fe5d2a00640 1 -- 172.21.15.6:0/4144815889 >> v1:172.21.15.178:6789/0 conn(0x7fe5d406e030 legacy=0x7fe5d40ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:18.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.140+0000 7fe5d2a00640 1 -- 172.21.15.6:0/4144815889 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d4072210 con 0x7fe5d4145570 2024-03-06T07:35:18.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.141+0000 7fe5d2a00640 1 --2- 172.21.15.6:0/4144815889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d40cf420 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fe5d406c3b0 tx=0x7fe5c402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5bb06d3dc9774e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.142+0000 7fe5d1600640 1 -- 172.21.15.6:0/4144815889 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c402c9f0 con 0x7fe5d4145570 2024-03-06T07:35:18.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.142+0000 7fe5d1600640 1 -- 172.21.15.6:0/4144815889 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c402cb90 con 0x7fe5d4145570 2024-03-06T07:35:18.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.142+0000 7fe5d1600640 1 -- 172.21.15.6:0/4144815889 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c40343f0 con 0x7fe5d4145570 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.142+0000 7fe5d3e00640 1 -- 172.21.15.6:0/4144815889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 msgr2=0x7fe5d40cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.142+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/4144815889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d40cf420 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fe5d406c3b0 tx=0x7fe5c402be50 comp rx=0 tx=0).stop 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.143+0000 7fe5d3e00640 1 -- 172.21.15.6:0/4144815889 shutdown_connections 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.143+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/4144815889 >> v2:172.21.15.178:3300/0 conn(0x7fe5d4071450 0x7fe5d406daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.143+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/4144815889 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d40cf420 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.143+0000 7fe5d3e00640 1 -- 172.21.15.6:0/4144815889 >> 172.21.15.6:0/4144815889 conn(0x7fe5d40b17c0 msgr2=0x7fe5d40b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.143+0000 7fe5d3e00640 1 -- 172.21.15.6:0/4144815889 shutdown_connections 2024-03-06T07:35:18.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.143+0000 7fe5d3e00640 1 -- 172.21.15.6:0/4144815889 wait complete. 2024-03-06T07:35:18.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.144+0000 7fe5d3e00640 1 Processor -- start 2024-03-06T07:35:18.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.145+0000 7fe5d3e00640 1 -- start start 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5d406e030 0x7fe5d413c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5d4071450 0x7fe5d415b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d41f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40718b0 con 0x7fe5d4145570 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d40727d0 con 0x7fe5d406e030 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d4071a30 con 0x7fe5d4071450 2024-03-06T07:35:18.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d41f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d41f3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60378/0 (socket says 172.21.15.6:60378) 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d3400640 1 -- 172.21.15.6:0/2008497516 learned_addr learned my addr 172.21.15.6:0/2008497516 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d2a00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5d406e030 0x7fe5d413c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.146+0000 7fe5d2000640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5d4071450 0x7fe5d415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3400640 1 -- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5d4071450 msgr2=0x7fe5d415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3400640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5d4071450 0x7fe5d415b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3400640 1 -- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5d406e030 msgr2=0x7fe5d413c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3400640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5d406e030 0x7fe5d413c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3400640 1 -- 172.21.15.6:0/2008497516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d41404d0 con 0x7fe5d4145570 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d2000640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5d4071450 0x7fe5d415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3400640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d41f3ff0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00c410 tx=0x7fe5cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc011790 con 0x7fe5d4145570 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5cc017070 con 0x7fe5d4145570 2024-03-06T07:35:18.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d41f4a70 con 0x7fe5d4145570 2024-03-06T07:35:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5cc011bb0 con 0x7fe5d4145570 2024-03-06T07:35:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.147+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d4140750 con 0x7fe5d4145570 2024-03-06T07:35:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.148+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d404f330 con 0x7fe5d4145570 2024-03-06T07:35:18.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.149+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5cc01c460 con 0x7fe5d4145570 2024-03-06T07:35:18.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.150+0000 7fe5cb400640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5b406b4d0 0x7fe5b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:18.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.151+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5501+0+0 (secure 0 0 0) 0x7fe5cc089f90 con 0x7fe5d4145570 2024-03-06T07:35:18.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.151+0000 7fe5d2a00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5b406b4d0 0x7fe5b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:18.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.152+0000 7fe5d2a00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5b406b4d0 0x7fe5b406d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4035000 tx=0x7fe5c4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:18.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.155+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5d41404d0 con 0x7fe5d4145570 2024-03-06T07:35:18.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.357+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe5d415d170 con 0x7fe5d4145570 2024-03-06T07:35:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.922+0000 7fe5cb400640 1 -- 172.21.15.6:0/2008497516 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe5cc058ed0 con 0x7fe5d4145570 2024-03-06T07:35:18.924 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' removed 2024-03-06T07:35:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.927+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5b406b4d0 msgr2=0x7fe5b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.927+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5b406b4d0 0x7fe5b406d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4035000 tx=0x7fe5c4030000 comp rx=0 tx=0).stop 2024-03-06T07:35:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.927+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 msgr2=0x7fe5d41f3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:18.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.927+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d41f3ff0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00c410 tx=0x7fe5cc00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 shutdown_connections 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe5b406b4d0 0x7fe5b406d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe5d4145570 0x7fe5d41f3ff0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe5d4071450 0x7fe5d415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 --2- 172.21.15.6:0/2008497516 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5d406e030 0x7fe5d413c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 >> 172.21.15.6:0/2008497516 conn(0x7fe5d40b17c0 msgr2=0x7fe5d40ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:18.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.928+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 shutdown_connections 2024-03-06T07:35:18.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:18.929+0000 7fe5d3e00640 1 -- 172.21.15.6:0/2008497516 wait complete. 2024-03-06T07:35:18.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:35:19.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.164+0000 7fb3a8200640 1 Processor -- start 2024-03-06T07:35:19.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.165+0000 7fb3a8200640 1 -- start start 2024-03-06T07:35:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a8200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb3a0161930 0x7fb3a00d24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a8200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00d20d0 con 0x7fb3a00cfa90 2024-03-06T07:35:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a8200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a00d4df0 con 0x7fb3a00d0f80 2024-03-06T07:35:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a8200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a00cfec0 con 0x7fb3a0161930 2024-03-06T07:35:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a6200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb3a00cfa90 0x7fb3a00c8420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34724/0 (socket says 172.21.15.6:34724) 2024-03-06T07:35:19.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a6200640 1 -- 172.21.15.6:0/3968626193 learned_addr learned my addr 172.21.15.6:0/3968626193 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:19.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.166+0000 7fb3a5800640 1 --2- 172.21.15.6:0/3968626193 >> v2:172.21.15.178:3300/0 conn(0x7fb3a0161930 0x7fb3a00d24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328796064 0 0) 0x7fb3a00d20d0 con 0x7fb3a00cfa90 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb384003520 con 0x7fb3a00cfa90 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643536245 0 0) 0x7fb384003520 con 0x7fb3a00cfa90 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 >> v2:172.21.15.178:3300/0 conn(0x7fb3a0161930 msgr2=0x7fb3a00d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 --2- 172.21.15.6:0/3968626193 >> v2:172.21.15.178:3300/0 conn(0x7fb3a0161930 0x7fb3a00d24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 >> v1:172.21.15.178:6789/0 conn(0x7fb3a00d0f80 legacy=0x7fb3a00c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00d1df0 con 0x7fb3a00cfa90 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb398001670 con 0x7fb3a00cfa90 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 >> v1:172.21.15.6:6789/0 conn(0x7fb3a00cfa90 legacy=0x7fb3a00c8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb384003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384004800 0x7fb384004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840052e0 0x7fb38400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38400c1d0 con 0x7fb384003a00 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38400c380 con 0x7fb384004800 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.167+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38400c580 con 0x7fb3840052e0 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a5800640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb384003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a5800640 1 -- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840052e0 msgr2=0x7fb38400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a5800640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840052e0 0x7fb38400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a5800640 1 -- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384004800 msgr2=0x7fb384004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a4e00640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384004800 0x7fb384004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a5800640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384004800 0x7fb384004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a5800640 1 -- 172.21.15.6:0/3968626193 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384003520 con 0x7fb384003a00 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a6200640 1 -- 172.21.15.6:0/3968626193 reap_dead start 2024-03-06T07:35:19.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.168+0000 7fb3a4e00640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384004800 0x7fb384004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.169+0000 7fb3a5800640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb384003e20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb390007360 tx=0x7fb384005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aed2a078535ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.169+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390039070 con 0x7fb384003a00 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39002ccd0 con 0x7fb384003a00 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb39fe00640 1 -- 172.21.15.6:0/3968626193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390009be0 con 0x7fb384003a00 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 -- 172.21.15.6:0/3968626193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 msgr2=0x7fb384003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb384003e20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fb390007360 tx=0x7fb384005830 comp rx=0 tx=0).stop 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 -- 172.21.15.6:0/3968626193 shutdown_connections 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 --2- 172.21.15.6:0/3968626193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb384003e20 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 -- 172.21.15.6:0/3968626193 >> 172.21.15.6:0/3968626193 conn(0x7fb3a00d3ee0 msgr2=0x7fb3a00d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 -- 172.21.15.6:0/3968626193 shutdown_connections 2024-03-06T07:35:19.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.170+0000 7fb3a8200640 1 -- 172.21.15.6:0/3968626193 wait complete. 2024-03-06T07:35:19.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.172+0000 7fb3a8200640 1 Processor -- start 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.172+0000 7fb3a8200640 1 -- start start 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a8200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb3a014b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a8200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a0154ff0 0x7fb3a00c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a8200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3a0155470 0x7fb3a00c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a8200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00b4f30 con 0x7fb384003a00 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a8200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a0161d10 con 0x7fb3a0154ff0 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a8200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a00d24c0 con 0x7fb3a0155470 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb3a014b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a5800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb3a014b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60396/0 (socket says 172.21.15.6:60396) 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.173+0000 7fb3a5800640 1 -- 172.21.15.6:0/453101520 learned_addr learned my addr 172.21.15.6:0/453101520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a4e00640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a0154ff0 0x7fb3a00c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a6200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3a0155470 0x7fb3a00c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a5800640 1 -- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3a0155470 msgr2=0x7fb3a00c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a5800640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3a0155470 0x7fb3a00c90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a5800640 1 -- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a0154ff0 msgr2=0x7fb3a00c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a5800640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a0154ff0 0x7fb3a00c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a5800640 1 -- 172.21.15.6:0/453101520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a014b830 con 0x7fb384003a00 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a4e00640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a0154ff0 0x7fb3a00c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb3a5800640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb3a014b2f0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb390007480 tx=0x7fb390004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.174+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390039050 con 0x7fb384003a00 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.175+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39003a040 con 0x7fb384003a00 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.175+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb390009be0 con 0x7fb384003a00 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.175+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a01559d0 con 0x7fb384003a00 2024-03-06T07:35:19.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.175+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a0155f90 con 0x7fb384003a00 2024-03-06T07:35:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.176+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a004f330 con 0x7fb384003a00 2024-03-06T07:35:19.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.177+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb390047020 con 0x7fb384003a00 2024-03-06T07:35:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.178+0000 7fb39d600640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb38006b4d0 0x7fb38006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.178+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb3900abeb0 con 0x7fb384003a00 2024-03-06T07:35:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.178+0000 7fb3a4e00640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb38006b4d0 0x7fb38006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.180+0000 7fb3a4e00640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb38006b4d0 0x7fb38006d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00c9c90 tx=0x7fb38c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.182+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb39004e020 con 0x7fb384003a00 2024-03-06T07:35:19.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.384+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb3a01568c0 con 0x7fb384003a00 2024-03-06T07:35:19.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.390+0000 7fb39d600640 1 -- 172.21.15.6:0/453101520 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb3a014b830 con 0x7fb384003a00 2024-03-06T07:35:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb38006b4d0 msgr2=0x7fb38006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb38006b4d0 0x7fb38006d990 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00c9c90 tx=0x7fb38c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 msgr2=0x7fb3a014b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb3a014b2f0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fb390007480 tx=0x7fb390004ab0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 shutdown_connections 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb38006b4d0 0x7fb38006d990 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3a0154ff0 0x7fb3a00c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3a0155470 0x7fb3a00c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 --2- 172.21.15.6:0/453101520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb384003a00 0x7fb3a014b2f0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 >> 172.21.15.6:0/453101520 conn(0x7fb3a00d3ee0 msgr2=0x7fb3a006ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 shutdown_connections 2024-03-06T07:35:19.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.393+0000 7fb3a8200640 1 -- 172.21.15.6:0/453101520 wait complete. 2024-03-06T07:35:19.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:35:19.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.634+0000 7f0b10000640 1 Processor -- start 2024-03-06T07:35:19.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.635+0000 7f0b10000640 1 -- start start 2024-03-06T07:35:19.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.636+0000 7f0b10000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 0x7f0b080d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.636+0000 7f0b10000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b080b5de0 con 0x7f0b08145790 2024-03-06T07:35:19.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b10000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b080b6080 con 0x7f0b080add50 2024-03-06T07:35:19.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b10000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080d1b70 con 0x7f0b0806b6c0 2024-03-06T07:35:19.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b0d600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b08145790 0x7f0b081603f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34726/0 (socket says 172.21.15.6:34726) 2024-03-06T07:35:19.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b0d600640 1 -- 172.21.15.6:0/4284380610 learned_addr learned my addr 172.21.15.6:0/4284380610 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b0e000640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 0x7f0b080d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519422663 0 0) 0x7f0b080b5de0 con 0x7f0b08145790 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.637+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af0003520 con 0x7f0b08145790 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839560622 0 0) 0x7f0af0003520 con 0x7f0b08145790 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 msgr2=0x7f0b080d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 0x7f0b080d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 >> v1:172.21.15.178:6789/0 conn(0x7f0b080add50 legacy=0x7f0b0806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0815fbf0 con 0x7f0b08145790 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0afc001670 con 0x7f0b08145790 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 >> v1:172.21.15.6:6789/0 conn(0x7f0b08145790 legacy=0x7f0b081603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af0003a50 0x7f0af0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0004800 0x7f0af0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 0x7f0af00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080d1b70 con 0x7f0af0003a50 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af000baf0 con 0x7f0af0004800 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af000bcf0 con 0x7f0b0806b6c0 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.638+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af0003a50 0x7f0af0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0e000640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 0x7f0af00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0cc00640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0004800 0x7f0af0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0d600640 1 -- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 msgr2=0x7f0af00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806b6c0 0x7f0af00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0d600640 1 -- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0004800 msgr2=0x7f0af0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:19.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0004800 0x7f0af0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.639+0000 7f0b0d600640 1 -- 172.21.15.6:0/4284380610 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af0003520 con 0x7f0af0003a50 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.640+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af0003a50 0x7f0af0003e70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c240 tx=0x7f0af0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d65246aa1194a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.640+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc007cf0 con 0x7f0af0003a50 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.640+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc030070 con 0x7f0af0003a50 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.640+0000 7f0b07e00640 1 -- 172.21.15.6:0/4284380610 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc005d30 con 0x7f0af0003a50 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b10000640 1 -- 172.21.15.6:0/4284380610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af0003a50 msgr2=0x7f0af0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b10000640 1 --2- 172.21.15.6:0/4284380610 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0af0003a50 0x7f0af0003e70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c240 tx=0x7f0af0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b0e000640 1 -- 172.21.15.6:0/4284380610 reap_dead start 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b10000640 1 -- 172.21.15.6:0/4284380610 shutdown_connections 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b10000640 1 -- 172.21.15.6:0/4284380610 >> 172.21.15.6:0/4284380610 conn(0x7f0b0806c2a0 msgr2=0x7f0b0806c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b10000640 1 -- 172.21.15.6:0/4284380610 shutdown_connections 2024-03-06T07:35:19.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.641+0000 7f0b10000640 1 -- 172.21.15.6:0/4284380610 wait complete. 2024-03-06T07:35:19.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.643+0000 7f0b10000640 1 Processor -- start 2024-03-06T07:35:19.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.643+0000 7f0b10000640 1 -- start start 2024-03-06T07:35:19.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b10000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0003a50 0x7f0b081f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b10000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806ec60 0x7f0b08158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b10000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 0x7f0b081588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b10000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b08145b70 con 0x7f0b0806f0e0 2024-03-06T07:35:19.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b10000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0809ef10 con 0x7f0af0003a50 2024-03-06T07:35:19.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b10000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b080d1630 con 0x7f0b0806ec60 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b0e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 0x7f0b081588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b0e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 0x7f0b081588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60416/0 (socket says 172.21.15.6:60416) 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b0e000640 1 -- 172.21.15.6:0/4109305424 learned_addr learned my addr 172.21.15.6:0/4109305424 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b0cc00640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806ec60 0x7f0b08158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.644+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0003a50 0x7f0b081f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b0e000640 1 -- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806ec60 msgr2=0x7f0b08158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b0e000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806ec60 0x7f0b08158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b0e000640 1 -- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0003a50 msgr2=0x7f0b081f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b0e000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0003a50 0x7f0b081f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b0e000640 1 -- 172.21.15.6:0/4109305424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0806dbc0 con 0x7f0b0806f0e0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b0e000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 0x7f0b081588c0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f0b0815d5d0 tx=0x7f0b00010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.645+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0000f040 con 0x7f0b0806f0e0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.646+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0000e8e0 con 0x7f0b0806f0e0 2024-03-06T07:35:19.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.646+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0000ecd0 con 0x7f0b0806f0e0 2024-03-06T07:35:19.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.646+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0806dea0 con 0x7f0b0806f0e0 2024-03-06T07:35:19.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.646+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0806e460 con 0x7f0b0806f0e0 2024-03-06T07:35:19.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.647+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0804f3a0 con 0x7f0b0806f0e0 2024-03-06T07:35:19.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.648+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b000143f0 con 0x7f0b0806f0e0 2024-03-06T07:35:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.649+0000 7f0b05600640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0aec06b4d0 0x7f0aec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:19.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.649+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0b0008a130 con 0x7f0b0806f0e0 2024-03-06T07:35:19.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.650+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0aec06b4d0 0x7f0aec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:19.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.651+0000 7f0b0d600640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0aec06b4d0 0x7f0aec06d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0afc034000 tx=0x7f0afc007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:19.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.653+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b00018030 con 0x7f0b0806f0e0 2024-03-06T07:35:19.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:19.928+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f0b080728d0 con 0x7f0b0806f0e0 2024-03-06T07:35:20.930 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.928+0000 7f0b05600640 1 -- 172.21.15.6:0/4109305424 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v425) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0b00059240 con 0x7f0b0806f0e0 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.932+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0aec06b4d0 msgr2=0x7f0aec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.932+0000 7f0b10000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0aec06b4d0 0x7f0aec06d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0afc034000 tx=0x7f0afc007000 comp rx=0 tx=0).stop 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.932+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 msgr2=0x7f0b081588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.932+0000 7f0b10000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 0x7f0b081588c0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f0b0815d5d0 tx=0x7f0b00010c20 comp rx=0 tx=0).stop 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 shutdown_connections 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0aec06b4d0 0x7f0aec06d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b0806f0e0 0x7f0b081588c0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b0806ec60 0x7f0b08158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 --2- 172.21.15.6:0/4109305424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0af0003a50 0x7f0b081f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 >> 172.21.15.6:0/4109305424 conn(0x7f0b0806c2a0 msgr2=0x7f0b0815d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 shutdown_connections 2024-03-06T07:35:20.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:20.933+0000 7f0b10000640 1 -- 172.21.15.6:0/4109305424 wait complete. 2024-03-06T07:35:20.956 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:35:20.957 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:35:20.957 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:35:20.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:21.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.182+0000 7faa12800640 1 Processor -- start 2024-03-06T07:35:21.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.183+0000 7faa12800640 1 -- start start 2024-03-06T07:35:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa12800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7faa0c145300 0x7faa0c0b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa12800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa0c0cd060 0x7faa0c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa12800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c0714d0 con 0x7faa0c070f10 2024-03-06T07:35:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa12800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0c0717b0 con 0x7faa0c0cd060 2024-03-06T07:35:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa12800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0c071a90 con 0x7faa0c145300 2024-03-06T07:35:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa11e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa0c0cd060 0x7faa0c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa11e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa0c0cd060 0x7faa0c15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60428/0 (socket says 172.21.15.6:60428) 2024-03-06T07:35:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.185+0000 7faa11e00640 1 -- 172.21.15.6:0/984690310 learned_addr learned my addr 172.21.15.6:0/984690310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.186+0000 7faa11400640 1 --2- 172.21.15.6:0/984690310 >> v2:172.21.15.178:3300/0 conn(0x7faa0c145300 0x7faa0c0b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.186+0000 7faa11e00640 1 -- 172.21.15.6:0/984690310 >> v2:172.21.15.178:3300/0 conn(0x7faa0c145300 msgr2=0x7faa0c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.186+0000 7faa11e00640 1 --2- 172.21.15.6:0/984690310 >> v2:172.21.15.178:3300/0 conn(0x7faa0c145300 0x7faa0c0b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.186+0000 7faa11e00640 1 -- 172.21.15.6:0/984690310 >> v1:172.21.15.178:6789/0 conn(0x7faa0c070f10 legacy=0x7faa0c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.186+0000 7faa11e00640 1 -- 172.21.15.6:0/984690310 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c071d70 con 0x7faa0c0cd060 2024-03-06T07:35:21.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.187+0000 7faa11e00640 1 --2- 172.21.15.6:0/984690310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c0cd060 0x7faa0c15c570 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0cf990 tx=0x7faa0402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fce2f01b3aa0c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.187+0000 7faa0be00640 1 -- 172.21.15.6:0/984690310 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0402c9f0 con 0x7faa0c0cd060 2024-03-06T07:35:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.187+0000 7faa0be00640 1 -- 172.21.15.6:0/984690310 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0402cb90 con 0x7faa0c0cd060 2024-03-06T07:35:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.187+0000 7faa0be00640 1 -- 172.21.15.6:0/984690310 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa040343f0 con 0x7faa0c0cd060 2024-03-06T07:35:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 -- 172.21.15.6:0/984690310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c0cd060 msgr2=0x7faa0c15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 --2- 172.21.15.6:0/984690310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c0cd060 0x7faa0c15c570 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0cf990 tx=0x7faa0402be50 comp rx=0 tx=0).stop 2024-03-06T07:35:21.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 -- 172.21.15.6:0/984690310 shutdown_connections 2024-03-06T07:35:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 --2- 172.21.15.6:0/984690310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c0cd060 0x7faa0c15c570 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 --2- 172.21.15.6:0/984690310 >> v2:172.21.15.178:3300/0 conn(0x7faa0c145300 0x7faa0c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 -- 172.21.15.6:0/984690310 >> 172.21.15.6:0/984690310 conn(0x7faa0c06b3d0 msgr2=0x7faa0c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 -- 172.21.15.6:0/984690310 shutdown_connections 2024-03-06T07:35:21.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.188+0000 7faa12800640 1 -- 172.21.15.6:0/984690310 wait complete. 2024-03-06T07:35:21.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.190+0000 7faa12800640 1 Processor -- start 2024-03-06T07:35:21.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.190+0000 7faa12800640 1 -- start start 2024-03-06T07:35:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa12800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 0x7faa0c1f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa12800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa0c0cd060 0x7faa0c13d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa12800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa0c145300 0x7faa0c13c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa12800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c15cab0 con 0x7faa0c070f10 2024-03-06T07:35:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa12800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c1456e0 con 0x7faa0c0cd060 2024-03-06T07:35:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa12800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c15cc30 con 0x7faa0c145300 2024-03-06T07:35:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa11400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 0x7faa0c1f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa11400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 0x7faa0c1f4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60436/0 (socket says 172.21.15.6:60436) 2024-03-06T07:35:21.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.191+0000 7faa11400640 1 -- 172.21.15.6:0/3089512802 learned_addr learned my addr 172.21.15.6:0/3089512802 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa11400640 1 -- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa0c145300 msgr2=0x7faa0c13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa11400640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa0c145300 0x7faa0c13c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa11400640 1 -- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa0c0cd060 msgr2=0x7faa0c13d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa11400640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa0c0cd060 0x7faa0c13d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa11400640 1 -- 172.21.15.6:0/3089512802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c1f4cd0 con 0x7faa0c070f10 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa11400640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 0x7faa0c1f4790 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7faa0000d620 tx=0x7faa0000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0000b620 con 0x7faa0c070f10 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa00019070 con 0x7faa0c070f10 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c140740 con 0x7faa0c070f10 2024-03-06T07:35:21.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.192+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0000ba00 con 0x7faa0c070f10 2024-03-06T07:35:21.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.193+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c140fe0 con 0x7faa0c070f10 2024-03-06T07:35:21.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.194+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0c04f330 con 0x7faa0c070f10 2024-03-06T07:35:21.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.195+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa0000bba0 con 0x7faa0c070f10 2024-03-06T07:35:21.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.196+0000 7faa09600640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.196+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7faa0008a390 con 0x7faa0c070f10 2024-03-06T07:35:21.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.196+0000 7faa10a00640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.197+0000 7faa10a00640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7faa0c1f1900 tx=0x7fa9f80079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.200+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa0c1f4cd0 con 0x7faa0c070f10 2024-03-06T07:35:21.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.400+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa0c0b2d80 con 0x7faa0c070f10 2024-03-06T07:35:21.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.402+0000 7faa09600640 1 -- 172.21.15.6:0/3089512802 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13084 (secure 0 0 0) 0x7faa000594a0 con 0x7faa0c070f10 2024-03-06T07:35:21.405 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:21.405 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":117,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:35:18.911590+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:6835/3719236593":"2024-03-07T07:35:15.500431+0000","172.21.15.6:6834/3719236593":"2024-03-07T07:35:15.500431+0000","172.21.15.6:6833/4114489444":"2024-03-07T07:34:04.377501+0000","172.21.15.6:6832/4114489444":"2024-03-07T07:34:04.377501+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9ec06b4d0 msgr2=0x7fa9ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7faa0c1f1900 tx=0x7fa9f80079b0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 msgr2=0x7faa0c1f4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 0x7faa0c1f4790 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7faa0000d620 tx=0x7faa0000dae0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 shutdown_connections 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa9ec06b4d0 0x7fa9ec06d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faa0c0cd060 0x7faa0c13d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faa0c145300 0x7faa0c13c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 --2- 172.21.15.6:0/3089512802 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa0c070f10 0x7faa0c1f4790 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.405+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 >> 172.21.15.6:0/3089512802 conn(0x7faa0c06b3d0 msgr2=0x7faa0c0b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.406+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 shutdown_connections 2024-03-06T07:35:21.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.406+0000 7faa12800640 1 -- 172.21.15.6:0/3089512802 wait complete. 2024-03-06T07:35:21.428 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:21.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:21.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.652+0000 7ff8f6400640 1 Processor -- start 2024-03-06T07:35:21.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.652+0000 7ff8f6400640 1 -- start start 2024-03-06T07:35:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f6400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff8f015f810 0x7ff8f00b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f6400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8f00b2080 0x7ff8f00b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f6400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f015fe40 con 0x7ff8f0145430 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f6400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f00b24d0 con 0x7ff8f00b2080 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f6400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f0160120 con 0x7ff8f015f810 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f5a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff8f00b2080 0x7ff8f00b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f5000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff8f0145430 0x7ff8f015cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34728/0 (socket says 172.21.15.6:34728) 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.654+0000 7ff8f5000640 1 -- 172.21.15.6:0/1317583216 learned_addr learned my addr 172.21.15.6:0/1317583216 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.655+0000 7ff8e7e00640 1 --2- 172.21.15.6:0/1317583216 >> v2:172.21.15.178:3300/0 conn(0x7ff8f015f810 0x7ff8f00b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.655+0000 7ff8f5a00640 1 -- 172.21.15.6:0/1317583216 >> v2:172.21.15.178:3300/0 conn(0x7ff8f015f810 msgr2=0x7ff8f00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:21.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.655+0000 7ff8f5a00640 1 --2- 172.21.15.6:0/1317583216 >> v2:172.21.15.178:3300/0 conn(0x7ff8f015f810 0x7ff8f00b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.655+0000 7ff8f5a00640 1 -- 172.21.15.6:0/1317583216 >> v1:172.21.15.6:6789/0 conn(0x7ff8f0145430 legacy=0x7ff8f015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.655+0000 7ff8f5a00640 1 -- 172.21.15.6:0/1317583216 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0160400 con 0x7ff8f00b2080 2024-03-06T07:35:21.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.656+0000 7ff8f5a00640 1 --2- 172.21.15.6:0/1317583216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f00b2080 0x7ff8f00b2fd0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff8f006d8c0 tx=0x7ff8e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883d140122e6a294 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.656+0000 7ff8efe00640 1 -- 172.21.15.6:0/1317583216 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8039070 con 0x7ff8f00b2080 2024-03-06T07:35:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.656+0000 7ff8efe00640 1 -- 172.21.15.6:0/1317583216 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e802cc80 con 0x7ff8f00b2080 2024-03-06T07:35:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.656+0000 7ff8efe00640 1 -- 172.21.15.6:0/1317583216 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8034660 con 0x7ff8f00b2080 2024-03-06T07:35:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 -- 172.21.15.6:0/1317583216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f00b2080 msgr2=0x7ff8f00b2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 --2- 172.21.15.6:0/1317583216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f00b2080 0x7ff8f00b2fd0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7ff8f006d8c0 tx=0x7ff8e802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 -- 172.21.15.6:0/1317583216 shutdown_connections 2024-03-06T07:35:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 --2- 172.21.15.6:0/1317583216 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f00b2080 0x7ff8f00b2fd0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 --2- 172.21.15.6:0/1317583216 >> v2:172.21.15.178:3300/0 conn(0x7ff8f015f810 0x7ff8f00b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 -- 172.21.15.6:0/1317583216 >> 172.21.15.6:0/1317583216 conn(0x7ff8f006b3a0 msgr2=0x7ff8f006b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 -- 172.21.15.6:0/1317583216 shutdown_connections 2024-03-06T07:35:21.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.657+0000 7ff8f6400640 1 -- 172.21.15.6:0/1317583216 wait complete. 2024-03-06T07:35:21.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.659+0000 7ff8f6400640 1 Processor -- start 2024-03-06T07:35:21.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.659+0000 7ff8f6400640 1 -- start start 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8f6400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff8f00b2080 0x7ff8f013d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8f6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 0x7ff8f0156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8f6400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f015f810 0x7ff8f0158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8f6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0145990 con 0x7ff8f0145430 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8f6400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0145810 con 0x7ff8f015f810 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8f6400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00cde00 con 0x7ff8f00b2080 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 0x7ff8f0156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8e7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 0x7ff8f0156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60448/0 (socket says 172.21.15.6:60448) 2024-03-06T07:35:21.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.660+0000 7ff8e7e00640 1 -- 172.21.15.6:0/3626762166 learned_addr learned my addr 172.21.15.6:0/3626762166 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8f5000640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff8f00b2080 0x7ff8f013d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8f5a00640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f015f810 0x7ff8f0158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8e7e00640 1 -- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff8f00b2080 msgr2=0x7ff8f013d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8e7e00640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff8f00b2080 0x7ff8f013d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8e7e00640 1 -- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f015f810 msgr2=0x7ff8f0158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8e7e00640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f015f810 0x7ff8f0158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8e7e00640 1 -- 172.21.15.6:0/3626762166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0156d50 con 0x7ff8f0145430 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8f5a00640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f015f810 0x7ff8f0158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:21.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8e7e00640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 0x7ff8f0156040 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800d5f0 tx=0x7ff8d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80125f0 con 0x7ff8f0145430 2024-03-06T07:35:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d8019070 con 0x7ff8f0145430 2024-03-06T07:35:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80129d0 con 0x7ff8f0145430 2024-03-06T07:35:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.661+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f0157030 con 0x7ff8f0145430 2024-03-06T07:35:21.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.662+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f0157b50 con 0x7ff8f0145430 2024-03-06T07:35:21.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.663+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f004f330 con 0x7ff8f0145430 2024-03-06T07:35:21.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.664+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8d8012b70 con 0x7ff8f0145430 2024-03-06T07:35:21.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.665+0000 7ff8ed600640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff8cc06b4d0 0x7ff8cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:21.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.665+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7ff8d808a2c0 con 0x7ff8f0145430 2024-03-06T07:35:21.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.666+0000 7ff8f5000640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff8cc06b4d0 0x7ff8cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:21.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.667+0000 7ff8f5000640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff8cc06b4d0 0x7ff8cc06d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0000f60 tx=0x7ff8e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:21.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.670+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8d8059470 con 0x7ff8f0145430 2024-03-06T07:35:21.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.869+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8f015ee80 con 0x7ff8f0145430 2024-03-06T07:35:21.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.870+0000 7ff8ed600640 1 -- 172.21.15.6:0/3626762166 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8d8058d70 con 0x7ff8f0145430 2024-03-06T07:35:21.872 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff8cc06b4d0 msgr2=0x7ff8cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff8cc06b4d0 0x7ff8cc06d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0000f60 tx=0x7ff8e0009040 comp rx=0 tx=0).stop 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 msgr2=0x7ff8f0156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 0x7ff8f0156040 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800d5f0 tx=0x7ff8d800dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 shutdown_connections 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff8cc06b4d0 0x7ff8cc06d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8f015f810 0x7ff8f0158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff8f0145430 0x7ff8f0156040 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 --2- 172.21.15.6:0/3626762166 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff8f00b2080 0x7ff8f013d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 >> 172.21.15.6:0/3626762166 conn(0x7ff8f006b3a0 msgr2=0x7ff8f00af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 shutdown_connections 2024-03-06T07:35:21.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:21.873+0000 7ff8f6400640 1 -- 172.21.15.6:0/3626762166 wait complete. 2024-03-06T07:35:21.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:35:22.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.110+0000 7f0fa6600640 1 Processor -- start 2024-03-06T07:35:22.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.110+0000 7f0fa6600640 1 -- start start 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0fa6600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0fa00af1a0 0x7f0fa015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0fa6600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 0x7f0fa00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0fa6600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa015cc40 con 0x7f0fa00af580 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0fa6600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa015cf20 con 0x7f0fa00af1a0 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0fa6600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa015d200 con 0x7f0fa00cdc20 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0f9ea00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0fa00af580 0x7f0fa00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56898/0 (socket says 172.21.15.6:56898) 2024-03-06T07:35:22.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.112+0000 7f0f9ea00640 1 -- 172.21.15.6:0/286050246 learned_addr learned my addr 172.21.15.6:0/286050246 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:22.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969786604 0 0) 0x7f0fa015cc40 con 0x7f0fa00af580 2024-03-06T07:35:22.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9f400640 1 --2- 172.21.15.6:0/286050246 >> v2:172.21.15.178:3300/0 conn(0x7f0fa00af1a0 0x7f0fa015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f84003520 con 0x7f0fa00af580 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9fe00640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 0x7f0fa00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 707254247 0 0) 0x7f0f84003520 con 0x7f0fa00af580 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 msgr2=0x7f0fa00b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 0x7f0fa00b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 >> v2:172.21.15.178:3300/0 conn(0x7f0fa00af1a0 msgr2=0x7f0fa015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 --2- 172.21.15.6:0/286050246 >> v2:172.21.15.178:3300/0 conn(0x7f0fa00af1a0 0x7f0fa015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa015d4e0 con 0x7f0fa00af580 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0f88001670 con 0x7f0fa00af580 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 >> v1:172.21.15.6:6789/0 conn(0x7f0fa00af580 legacy=0x7f0fa00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f84003a20 0x7f0f84003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f84004800 0x7f0f84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 0x7f0f840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa015cf20 con 0x7f0f84004800 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa015d200 con 0x7f0f84003a20 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.113+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8400ba40 con 0x7f0fa00cdc20 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f84004800 0x7f0f84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9f400640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f84003a20 0x7f0f84003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9ea00640 1 -- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 msgr2=0x7f0f840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fa00cdc20 0x7f0f840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9ea00640 1 -- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f84003a20 msgr2=0x7f0f84003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f84003a20 0x7f0f84003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.114+0000 7f0f9ea00640 1 -- 172.21.15.6:0/286050246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84003520 con 0x7f0f84004800 2024-03-06T07:35:22.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.115+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f84004800 0x7f0f84004c80 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0f88007fd0 tx=0x7f0f84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919d25288172d81d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.115+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88007d50 con 0x7f0f84004800 2024-03-06T07:35:22.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.115+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f88030070 con 0x7f0f84004800 2024-03-06T07:35:22.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.115+0000 7f0f9e000640 1 -- 172.21.15.6:0/286050246 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8802cd60 con 0x7f0f84004800 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0fa6600640 1 -- 172.21.15.6:0/286050246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f84004800 msgr2=0x7f0f84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0fa6600640 1 --2- 172.21.15.6:0/286050246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f84004800 0x7f0f84004c80 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0f88007fd0 tx=0x7f0f84005830 comp rx=0 tx=0).stop 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0f9fe00640 1 -- 172.21.15.6:0/286050246 reap_dead start 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0fa6600640 1 -- 172.21.15.6:0/286050246 shutdown_connections 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0fa6600640 1 -- 172.21.15.6:0/286050246 >> 172.21.15.6:0/286050246 conn(0x7f0fa006afb0 msgr2=0x7f0fa006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0fa6600640 1 -- 172.21.15.6:0/286050246 shutdown_connections 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.116+0000 7f0fa6600640 1 -- 172.21.15.6:0/286050246 wait complete. 2024-03-06T07:35:22.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.118+0000 7f0fa6600640 1 Processor -- start 2024-03-06T07:35:22.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.118+0000 7f0fa6600640 1 -- start start 2024-03-06T07:35:22.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0fa6600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f84004800 0x7f0fa006cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0fa6600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 0x7f0fa015dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0fa6600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0fa01507e0 0x7f0fa0159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0fa6600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00b31a0 con 0x7f0fa00caa20 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0fa6600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00b3020 con 0x7f0fa01507e0 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0fa6600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00b3320 con 0x7f0f84004800 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.119+0000 7f0f9ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 0x7f0fa015dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 0x7f0fa015dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59024/0 (socket says 172.21.15.6:59024) 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 -- 172.21.15.6:0/551810820 learned_addr learned my addr 172.21.15.6:0/551810820 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9f400640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f84004800 0x7f0fa006cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9fe00640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0fa01507e0 0x7f0fa0159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 -- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f84004800 msgr2=0x7f0fa006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f84004800 0x7f0fa006cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 -- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0fa01507e0 msgr2=0x7f0fa0159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.123 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0fa01507e0 0x7f0fa0159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 -- 172.21.15.6:0/551810820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa015e290 con 0x7f0fa00caa20 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f9ea00640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 0x7f0fa015dd50 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f0f8802c570 tx=0x7f0f88001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88007d50 con 0x7f0fa00caa20 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f88004bf0 con 0x7f0fa00caa20 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.120+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88001c90 con 0x7f0fa00caa20 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.121+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa006d1a0 con 0x7f0fa00caa20 2024-03-06T07:35:22.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.121+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa006d760 con 0x7f0fa00caa20 2024-03-06T07:35:22.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.122+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa0069510 con 0x7f0fa00caa20 2024-03-06T07:35:22.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.123+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f88048020 con 0x7f0fa00caa20 2024-03-06T07:35:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.124+0000 7f0f97400640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f7c06b500 0x7f0f7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.124+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0f880ab1e0 con 0x7f0fa00caa20 2024-03-06T07:35:22.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.124+0000 7f0f9f400640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f7c06b500 0x7f0f7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:22.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.125+0000 7f0f9f400640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f7c06b500 0x7f0f7c06d9c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0f90003a50 tx=0x7f0f9000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:22.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.128+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f8807a300 con 0x7f0fa00caa20 2024-03-06T07:35:22.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.399+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f0fa00cd240 con 0x7f0fa00caa20 2024-03-06T07:35:22.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.940+0000 7f0f97400640 1 -- 172.21.15.6:0/551810820 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v427) v1 ==== 143+0+0 (secure 0 0 0) 0x7f0f88079c00 con 0x7f0fa00caa20 2024-03-06T07:35:22.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f7c06b500 msgr2=0x7f0f7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f7c06b500 0x7f0f7c06d9c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f0f90003a50 tx=0x7f0f9000b040 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 msgr2=0x7f0fa015dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 0x7f0fa015dd50 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f0f8802c570 tx=0x7f0f88001ab0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 shutdown_connections 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f7c06b500 0x7f0f7c06d9c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0fa01507e0 0x7f0fa0159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fa00caa20 0x7f0fa015dd50 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 --2- 172.21.15.6:0/551810820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0f84004800 0x7f0fa006cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.943+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 >> 172.21.15.6:0/551810820 conn(0x7f0fa006afb0 msgr2=0x7f0fa0072150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.944+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 shutdown_connections 2024-03-06T07:35:22.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:22.944+0000 7f0fa6600640 1 -- 172.21.15.6:0/551810820 wait complete. 2024-03-06T07:35:22.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:35:23.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.190+0000 7fef7e000640 1 Processor -- start 2024-03-06T07:35:23.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.190+0000 7fef7e000640 1 -- start start 2024-03-06T07:35:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 0x7fef780b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fef780b2080 0x7fef780b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7e000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7815fe40 con 0x7fef78145430 2024-03-06T07:35:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7e000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef78160120 con 0x7fef780b2080 2024-03-06T07:35:23.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef78160400 con 0x7fef7815f810 2024-03-06T07:35:23.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7cc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef78145430 0x7fef7815cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56910/0 (socket says 172.21.15.6:56910) 2024-03-06T07:35:23.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.192+0000 7fef7cc00640 1 -- 172.21.15.6:0/217545366 learned_addr learned my addr 172.21.15.6:0/217545366 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:23.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef7d600640 1 --2- 172.21.15.6:0/217545366 >> v2:172.21.15.178:3300/0 conn(0x7fef780b2080 0x7fef780b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77e00640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 0x7fef780b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255515049 0 0) 0x7fef7815fe40 con 0x7fef78145430 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef60003520 con 0x7fef78145430 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740943294 0 0) 0x7fef60003520 con 0x7fef78145430 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 msgr2=0x7fef780b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 0x7fef780b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 >> v2:172.21.15.178:3300/0 conn(0x7fef780b2080 msgr2=0x7fef780b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 --2- 172.21.15.6:0/217545366 >> v2:172.21.15.178:3300/0 conn(0x7fef780b2080 0x7fef780b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef781606e0 con 0x7fef78145430 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fef6c001670 con 0x7fef78145430 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.193+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 >> v1:172.21.15.6:6789/0 conn(0x7fef78145430 legacy=0x7fef7815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77400640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef600039b0 0x7fef60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77400640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77400640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 0x7fef600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef78160120 con 0x7fef60004800 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef78160400 con 0x7fef600039b0 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6000bac0 con 0x7fef7815f810 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77e00640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef7cc00640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef600039b0 0x7fef60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef7d600640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 0x7fef600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77e00640 1 -- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 msgr2=0x7fef600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77e00640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef7815f810 0x7fef600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77e00640 1 -- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef600039b0 msgr2=0x7fef60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77e00640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef600039b0 0x7fef60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.194+0000 7fef77e00640 1 -- 172.21.15.6:0/217545366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef60003520 con 0x7fef60004800 2024-03-06T07:35:23.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef77e00640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef60004c80 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fef6400ba00 tx=0x7fef60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a087c815e2ba4e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64036020 con 0x7fef60004800 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef640325f0 con 0x7fef60004800 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef77400640 1 -- 172.21.15.6:0/217545366 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6400b600 con 0x7fef60004800 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef7e000640 1 -- 172.21.15.6:0/217545366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 msgr2=0x7fef60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef7e000640 1 --2- 172.21.15.6:0/217545366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef60004c80 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fef6400ba00 tx=0x7fef60005830 comp rx=0 tx=0).stop 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef7d600640 1 -- 172.21.15.6:0/217545366 reap_dead start 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.196+0000 7fef7e000640 1 -- 172.21.15.6:0/217545366 shutdown_connections 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.197+0000 7fef7e000640 1 -- 172.21.15.6:0/217545366 >> 172.21.15.6:0/217545366 conn(0x7fef7806b3a0 msgr2=0x7fef7806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.197+0000 7fef7e000640 1 -- 172.21.15.6:0/217545366 shutdown_connections 2024-03-06T07:35:23.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.197+0000 7fef7e000640 1 -- 172.21.15.6:0/217545366 wait complete. 2024-03-06T07:35:23.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.198+0000 7fef7e000640 1 Processor -- start 2024-03-06T07:35:23.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.199+0000 7fef7e000640 1 -- start start 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef781446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef78136ba0 0x7fef781394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef78139a10 0x7fef781f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef780cdf80 con 0x7fef60004800 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef780cde00 con 0x7fef78139a10 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef780b1b40 con 0x7fef78136ba0 2024-03-06T07:35:23.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef781446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef781446b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59048/0 (socket says 172.21.15.6:59048) 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 -- 172.21.15.6:0/3367294153 learned_addr learned my addr 172.21.15.6:0/3367294153 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 -- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef78136ba0 msgr2=0x7fef781394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef78136ba0 0x7fef781394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 -- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef78139a10 msgr2=0x7fef781f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef78139a10 0x7fef781f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 -- 172.21.15.6:0/3367294153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef781380b0 con 0x7fef60004800 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.200+0000 7fef7cc00640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef781446b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.201+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00ee60 con 0x7fef60004800 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.201+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c018070 con 0x7fef60004800 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.201+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef78137030 con 0x7fef60004800 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.201+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00ca80 con 0x7fef60004800 2024-03-06T07:35:23.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.201+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef781375f0 con 0x7fef60004800 2024-03-06T07:35:23.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.202+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7804f330 con 0x7fef60004800 2024-03-06T07:35:23.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.203+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef6c00cc20 con 0x7fef60004800 2024-03-06T07:35:23.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.204+0000 7fef74c00640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef5806b4d0 0x7fef5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:23.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.204+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fef6c089e60 con 0x7fef60004800 2024-03-06T07:35:23.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.205+0000 7fef77e00640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef5806b4d0 0x7fef5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:23.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.206+0000 7fef77e00640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef5806b4d0 0x7fef5806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fef600057e0 tx=0x7fef64032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:23.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.209+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef6c058ee0 con 0x7fef60004800 2024-03-06T07:35:23.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.409+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fef780cc8c0 con 0x7fef60004800 2024-03-06T07:35:23.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.416+0000 7fef74c00640 1 -- 172.21.15.6:0/3367294153 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7fef6c0587e0 con 0x7fef60004800 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.419+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef5806b4d0 msgr2=0x7fef5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.419+0000 7fef7e000640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef5806b4d0 0x7fef5806d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fef600057e0 tx=0x7fef64032000 comp rx=0 tx=0).stop 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.419+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 msgr2=0x7fef781446b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.419+0000 7fef7e000640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef781446b0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c00ec10 comp rx=0 tx=0).stop 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 shutdown_connections 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fef5806b4d0 0x7fef5806d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef78139a10 0x7fef781f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fef78136ba0 0x7fef781394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 --2- 172.21.15.6:0/3367294153 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef60004800 0x7fef781446b0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 >> 172.21.15.6:0/3367294153 conn(0x7fef7806b3a0 msgr2=0x7fef7806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 shutdown_connections 2024-03-06T07:35:23.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:23.420+0000 7fef7e000640 1 -- 172.21.15.6:0/3367294153 wait complete. 2024-03-06T07:35:23.442 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:35:23.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:23.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:35:23.671 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:35:23.671 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:23.671 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:23.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:23.732 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:35:23.732 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:23.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:23.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:23.786 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:23.786 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:23.786 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:23.839 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054202,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249800k,nr_inodes=812450,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499604k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:35:23.839 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-03-06T07:35:23.840 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:23.886 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053407,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249164k,nr_inodes=812291,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498332k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:35:23.886 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:35:23.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:24.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.133+0000 7f0a73000640 1 Processor -- start 2024-03-06T07:35:24.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.134+0000 7f0a73000640 1 -- start start 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.135+0000 7f0a73000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a6c161670 0x7f0a6c070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.135+0000 7f0a73000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.135+0000 7f0a73000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c0ce9a0 con 0x7f0a6c06e240 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.135+0000 7f0a73000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a6c0730d0 con 0x7f0a6c161670 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.135+0000 7f0a73000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0cec80 con 0x7f0a6c0ae0a0 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.136+0000 7f0a6be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0a6c161670 0x7f0a6c070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.136+0000 7f0a71000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.136+0000 7f0a6b400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f0a6c06e240 0x7f0a6c06cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:49602/0 (socket says 172.21.15.6:49602) 2024-03-06T07:35:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.136+0000 7f0a6b400640 1 -- 172.21.15.6:0/3008343022 learned_addr learned my addr 172.21.15.6:0/3008343022 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.137+0000 7f0a6aa00640 1 -- 172.21.15.6:0/3008343022 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779754585 0 0) 0x7f0a6c0ce9a0 con 0x7f0a6c06e240 2024-03-06T07:35:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.137+0000 7f0a6aa00640 1 -- 172.21.15.6:0/3008343022 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a50003520 con 0x7f0a6c06e240 2024-03-06T07:35:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.137+0000 7f0a6be00640 1 -- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 msgr2=0x7f0a6c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.137+0000 7f0a6be00640 1 --2- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.137+0000 7f0a6be00640 1 -- 172.21.15.6:0/3008343022 >> v1:172.21.15.178:6789/0 conn(0x7f0a6c06e240 legacy=0x7f0a6c06cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.137+0000 7f0a6be00640 1 -- 172.21.15.6:0/3008343022 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0cef60 con 0x7f0a6c161670 2024-03-06T07:35:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.139+0000 7f0a6be00640 1 --2- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c070c70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009d90 tx=0x7f0a5c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5997812a6d18806 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.139+0000 7f0a6aa00640 1 -- 172.21.15.6:0/3008343022 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c039070 con 0x7f0a6c161670 2024-03-06T07:35:24.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.139+0000 7f0a6aa00640 1 -- 172.21.15.6:0/3008343022 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c02cc50 con 0x7f0a6c161670 2024-03-06T07:35:24.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.139+0000 7f0a6aa00640 1 -- 172.21.15.6:0/3008343022 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c034440 con 0x7f0a6c161670 2024-03-06T07:35:24.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 -- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 msgr2=0x7f0a6c070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 --2- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c070c70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c009d90 tx=0x7f0a5c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 -- 172.21.15.6:0/3008343022 shutdown_connections 2024-03-06T07:35:24.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 --2- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 --2- 172.21.15.6:0/3008343022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c070c70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 -- 172.21.15.6:0/3008343022 >> 172.21.15.6:0/3008343022 conn(0x7f0a6c06afb0 msgr2=0x7f0a6c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 -- 172.21.15.6:0/3008343022 shutdown_connections 2024-03-06T07:35:24.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.140+0000 7f0a73000640 1 -- 172.21.15.6:0/3008343022 wait complete. 2024-03-06T07:35:24.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.142+0000 7f0a73000640 1 Processor -- start 2024-03-06T07:35:24.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.142+0000 7f0a73000640 1 -- start start 2024-03-06T07:35:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a73000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 0x7f0a6c09ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a73000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a73000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c0cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a73000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0ce600 con 0x7f0a6c06e240 2024-03-06T07:35:24.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a73000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0cf520 con 0x7f0a6c161670 2024-03-06T07:35:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a73000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0ce780 con 0x7f0a6c0ae0a0 2024-03-06T07:35:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 0x7f0a6c09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a6be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 0x7f0a6c09ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59056/0 (socket says 172.21.15.6:59056) 2024-03-06T07:35:24.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.143+0000 7f0a6be00640 1 -- 172.21.15.6:0/1933335055 learned_addr learned my addr 172.21.15.6:0/1933335055 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6b400640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a71000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c0cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6be00640 1 -- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 msgr2=0x7f0a6c0cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6be00640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6be00640 1 -- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 msgr2=0x7f0a6c0cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6be00640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c0cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6be00640 1 -- 172.21.15.6:0/1933335055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c09aa10 con 0x7f0a6c06e240 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6b400640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a71000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c0cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:24.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.144+0000 7f0a6be00640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 0x7f0a6c09ba70 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c02c430 tx=0x7f0a5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.145+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c039050 con 0x7f0a6c06e240 2024-03-06T07:35:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.145+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c045070 con 0x7f0a6c06e240 2024-03-06T07:35:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.145+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c009450 con 0x7f0a6c06e240 2024-03-06T07:35:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.145+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c09acb0 con 0x7f0a6c06e240 2024-03-06T07:35:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.145+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c09b270 con 0x7f0a6c06e240 2024-03-06T07:35:24.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.146+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c069510 con 0x7f0a6c06e240 2024-03-06T07:35:24.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.147+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a5c03e610 con 0x7f0a6c06e240 2024-03-06T07:35:24.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.149+0000 7f0a63e00640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4806b4d0 0x7f0a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.150+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0a5c0acb20 con 0x7f0a6c06e240 2024-03-06T07:35:24.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.150+0000 7f0a6b400640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4806b4d0 0x7f0a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.151+0000 7f0a6b400640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4806b4d0 0x7f0a4806d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c1f74e0 tx=0x7f0a54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.156+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a5c07bbd0 con 0x7f0a6c06e240 2024-03-06T07:35:24.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.400+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a6c04f3a0 con 0x7f0a6c06e240 2024-03-06T07:35:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.401+0000 7f0a63e00640 1 -- 172.21.15.6:0/1933335055 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f0a5c07b4d0 con 0x7f0a6c06e240 2024-03-06T07:35:24.403 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:24.403 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5906,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/130145943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":130145943},{"type":"v1","addr":"172.21.15.178:6837","nonce":130145943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":6649,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/3213576471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":3213576471},{"type":"v1","addr":"172.21.15.178:6835","nonce":3213576471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6833/3329038273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":3329038273},{"type":"v1","addr":"172.21.15.6:6833","nonce":3329038273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":80},{"gid":8280,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/2504383680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6834","nonce":2504383680},{"type":"v1","addr":"172.21.15.6:6835","nonce":2504383680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86}],"filesystems":[]} 2024-03-06T07:35:24.403 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 86 2024-03-06T07:35:24.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4806b4d0 msgr2=0x7f0a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4806b4d0 0x7f0a4806d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c1f74e0 tx=0x7f0a54009040 comp rx=0 tx=0).stop 2024-03-06T07:35:24.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 msgr2=0x7f0a6c09ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 0x7f0a6c09ba70 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c02c430 tx=0x7f0a5c004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:24.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 shutdown_connections 2024-03-06T07:35:24.406 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0a4806b4d0 0x7f0a4806d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0a6c161670 0x7f0a6c0cb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0a6c0ae0a0 0x7f0a6c0cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 --2- 172.21.15.6:0/1933335055 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a6c06e240 0x7f0a6c09ba70 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 >> 172.21.15.6:0/1933335055 conn(0x7f0a6c06afb0 msgr2=0x7f0a6c06b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 shutdown_connections 2024-03-06T07:35:24.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.404+0000 7f0a73000640 1 -- 172.21.15.6:0/1933335055 wait complete. 2024-03-06T07:35:24.427 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:35:24.428 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:35:24.428 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:35:24.428 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:24.428 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:35:24.428 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:35:24.429 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:35:24.429 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:24.429 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:35:24.429 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:35:24.429 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:35:24.429 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:24.429 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:35:24.430 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:35:24.430 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:35:24.430 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:24.447 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:35:24.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:35:24.451 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:35:24.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:35:24.453 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:35:24.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:35:24.461 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:35:24.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:35:24.500 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:35:24.500 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:24.506 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:35:24.506 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:24.508 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:35:24.509 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:24.515 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:35:24.515 DEBUG:teuthology.parallel:result is None 2024-03-06T07:35:24.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:35:24.658 INFO:tasks.ceph.mds.d.smithi178.stdout:starting mds.d at 2024-03-06T07:35:24.658 INFO:tasks.ceph.mds.b.smithi178.stdout:starting mds.b at 2024-03-06T07:35:24.667 INFO:tasks.ceph.mds.a.smithi006.stdout:starting mds.a at 2024-03-06T07:35:24.668 INFO:tasks.ceph.mds.c.smithi006.stdout:starting mds.c at 2024-03-06T07:35:24.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.710+0000 7fde1e000640 1 Processor -- start 2024-03-06T07:35:24.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.710+0000 7fde1e000640 1 -- start start 2024-03-06T07:35:24.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.711+0000 7fde1e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af1a0 0x7fde1815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.711+0000 7fde1e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fde180af580 0x7fde180b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.711+0000 7fde1e000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1815cc40 con 0x7fde180cdc20 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.711+0000 7fde1e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1815cf20 con 0x7fde180af580 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.711+0000 7fde1e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1815d200 con 0x7fde180af1a0 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.711+0000 7fde17e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde180cdc20 0x7fde180b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56936/0 (socket says 172.21.15.6:56936) 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde17e00640 1 -- 172.21.15.6:0/3579756646 learned_addr learned my addr 172.21.15.6:0/3579756646 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16a00640 1 --2- 172.21.15.6:0/3579756646 >> v2:172.21.15.6:3300/0 conn(0x7fde180af580 0x7fde180b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde17400640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af1a0 0x7fde1815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16000640 1 -- 172.21.15.6:0/3579756646 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311731364 0 0) 0x7fde1815cc40 con 0x7fde180cdc20 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16000640 1 -- 172.21.15.6:0/3579756646 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddfc003520 con 0x7fde180cdc20 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16a00640 1 -- 172.21.15.6:0/3579756646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af1a0 msgr2=0x7fde1815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16a00640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af1a0 0x7fde1815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16a00640 1 -- 172.21.15.6:0/3579756646 >> v1:172.21.15.6:6789/0 conn(0x7fde180cdc20 legacy=0x7fde180b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde16a00640 1 -- 172.21.15.6:0/3579756646 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1815d4e0 con 0x7fde180af580 2024-03-06T07:35:24.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.712+0000 7fde17400640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af1a0 0x7fde1815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.713+0000 7fde16a00640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180af580 0x7fde180b4c20 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fde040097a0 tx=0x7fde0402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb1e07165465d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.713+0000 7fde16000640 1 -- 172.21.15.6:0/3579756646 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04031780 con 0x7fde180af580 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde16000640 1 -- 172.21.15.6:0/3579756646 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04037070 con 0x7fde180af580 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde16000640 1 -- 172.21.15.6:0/3579756646 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04031b20 con 0x7fde180af580 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 -- 172.21.15.6:0/3579756646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180af580 msgr2=0x7fde180b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180af580 0x7fde180b4c20 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fde040097a0 tx=0x7fde0402b820 comp rx=0 tx=0).stop 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 -- 172.21.15.6:0/3579756646 shutdown_connections 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180af580 0x7fde180b4c20 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 --2- 172.21.15.6:0/3579756646 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af1a0 0x7fde1815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 -- 172.21.15.6:0/3579756646 >> 172.21.15.6:0/3579756646 conn(0x7fde1806afb0 msgr2=0x7fde1806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:24.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 -- 172.21.15.6:0/3579756646 shutdown_connections 2024-03-06T07:35:24.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.714+0000 7fde1e000640 1 -- 172.21.15.6:0/3579756646 wait complete. 2024-03-06T07:35:24.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.716+0000 7fde1e000640 1 Processor -- start 2024-03-06T07:35:24.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.716+0000 7fde1e000640 1 -- start start 2024-03-06T07:35:24.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.717+0000 7fde1e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde180af1a0 0x7fde1814f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.717+0000 7fde1e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af580 0x7fde1809d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.717+0000 7fde1e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 0x7fde18150380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.717+0000 7fde1e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180b3660 con 0x7fde180cdc20 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.717+0000 7fde1e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180b34e0 con 0x7fde180af1a0 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.717+0000 7fde1e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde180b37e0 con 0x7fde180af580 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 0x7fde18150380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 0x7fde18150380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59122/0 (socket says 172.21.15.6:59122) 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 -- 172.21.15.6:0/3465663964 learned_addr learned my addr 172.21.15.6:0/3465663964 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde16a00640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af580 0x7fde1809d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17400640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde180af1a0 0x7fde1814f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 -- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af580 msgr2=0x7fde1809d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af580 0x7fde1809d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 -- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde180af1a0 msgr2=0x7fde1814f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde180af1a0 0x7fde1814f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 -- 172.21.15.6:0/3465663964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1806eac0 con 0x7fde180cdc20 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.718+0000 7fde17e00640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 0x7fde18150380 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fde1000caa0 tx=0x7fde1000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.719+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10017070 con 0x7fde180cdc20 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.719+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1000ac00 con 0x7fde180cdc20 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.719+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10012400 con 0x7fde180cdc20 2024-03-06T07:35:24.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.719+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde180c91f0 con 0x7fde180cdc20 2024-03-06T07:35:24.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.719+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde180c97b0 con 0x7fde180cdc20 2024-03-06T07:35:24.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.720+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde18069510 con 0x7fde180cdc20 2024-03-06T07:35:24.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.721+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde1001a070 con 0x7fde180cdc20 2024-03-06T07:35:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.722+0000 7fde0f400640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddf806b4d0 0x7fddf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:24.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.722+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fde10089cc0 con 0x7fde180cdc20 2024-03-06T07:35:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.723+0000 7fde17400640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddf806b4d0 0x7fddf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:24.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.724+0000 7fde17400640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddf806b4d0 0x7fddf806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fde08009560 tx=0x7fde08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:24.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.727+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde10090080 con 0x7fde180cdc20 2024-03-06T07:35:24.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:24.938+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fde1804f3a0 con 0x7fde180cdc20 2024-03-06T07:35:25.975 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.973+0000 7fde0f400640 1 -- 172.21.15.6:0/3465663964 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v122) v1 ==== 97+0+0 (secure 0 0 0) 0x7fde10058780 con 0x7fde180cdc20 2024-03-06T07:35:25.975 INFO:teuthology.orchestra.run.smithi006.stderr: removed all blocklist entries 2024-03-06T07:35:25.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddf806b4d0 msgr2=0x7fddf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddf806b4d0 0x7fddf806d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fde08009560 tx=0x7fde08008040 comp rx=0 tx=0).stop 2024-03-06T07:35:25.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 msgr2=0x7fde18150380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:25.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 0x7fde18150380 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fde1000caa0 tx=0x7fde1000cf60 comp rx=0 tx=0).stop 2024-03-06T07:35:25.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 shutdown_connections 2024-03-06T07:35:25.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fddf806b4d0 0x7fddf806d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde180cdc20 0x7fde18150380 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fde180af580 0x7fde1809d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 --2- 172.21.15.6:0/3465663964 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde180af1a0 0x7fde1814f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:25.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 >> 172.21.15.6:0/3465663964 conn(0x7fde1806afb0 msgr2=0x7fde180cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:25.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 shutdown_connections 2024-03-06T07:35:25.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:25.977+0000 7fde1e000640 1 -- 172.21.15.6:0/3465663964 wait complete. 2024-03-06T07:35:26.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:35:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.215+0000 7f0c5e400640 1 Processor -- start 2024-03-06T07:35:26.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.215+0000 7f0c5e400640 1 -- start start 2024-03-06T07:35:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.216+0000 7f0c5e400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0c580af1a0 0x7f0c5815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c5e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c580b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c5e400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5815cc40 con 0x7f0c580cdc20 2024-03-06T07:35:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c5e400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c5815cf20 con 0x7f0c580af1a0 2024-03-06T07:35:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c5e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5815d200 con 0x7f0c580af580 2024-03-06T07:35:26.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c4fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c580b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c4fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c580b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:43150/0 (socket says 172.21.15.6:43150) 2024-03-06T07:35:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.217+0000 7f0c4fe00640 1 -- 172.21.15.6:0/1607979435 learned_addr learned my addr 172.21.15.6:0/1607979435 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.218+0000 7f0c57400640 1 --2- 172.21.15.6:0/1607979435 >> v2:172.21.15.178:3300/0 conn(0x7f0c580af1a0 0x7f0c5815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.218+0000 7f0c4fe00640 1 -- 172.21.15.6:0/1607979435 >> v2:172.21.15.178:3300/0 conn(0x7f0c580af1a0 msgr2=0x7f0c5815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.218+0000 7f0c4fe00640 1 --2- 172.21.15.6:0/1607979435 >> v2:172.21.15.178:3300/0 conn(0x7f0c580af1a0 0x7f0c5815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.218+0000 7f0c4fe00640 1 -- 172.21.15.6:0/1607979435 >> v1:172.21.15.178:6789/0 conn(0x7f0c580cdc20 legacy=0x7f0c580b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.218+0000 7f0c4fe00640 1 -- 172.21.15.6:0/1607979435 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5815d4e0 con 0x7f0c580af580 2024-03-06T07:35:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.220+0000 7f0c4fe00640 1 --2- 172.21.15.6:0/1607979435 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c580b4c20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009870 tx=0x7f0c4002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d582d065a8779956 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.220+0000 7f0c56a00640 1 -- 172.21.15.6:0/1607979435 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c40035020 con 0x7f0c580af580 2024-03-06T07:35:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.220+0000 7f0c56a00640 1 -- 172.21.15.6:0/1607979435 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c40031640 con 0x7f0c580af580 2024-03-06T07:35:26.222 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.220+0000 7f0c56a00640 1 -- 172.21.15.6:0/1607979435 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4002d040 con 0x7f0c580af580 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 -- 172.21.15.6:0/1607979435 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 msgr2=0x7f0c580b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 --2- 172.21.15.6:0/1607979435 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c580b4c20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0c40009870 tx=0x7f0c4002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 -- 172.21.15.6:0/1607979435 shutdown_connections 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 --2- 172.21.15.6:0/1607979435 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c580b4c20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 --2- 172.21.15.6:0/1607979435 >> v2:172.21.15.178:3300/0 conn(0x7f0c580af1a0 0x7f0c5815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 -- 172.21.15.6:0/1607979435 >> 172.21.15.6:0/1607979435 conn(0x7f0c5806afb0 msgr2=0x7f0c5806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 -- 172.21.15.6:0/1607979435 shutdown_connections 2024-03-06T07:35:26.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.221+0000 7f0c5e400640 1 -- 172.21.15.6:0/1607979435 wait complete. 2024-03-06T07:35:26.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.222+0000 7f0c5e400640 1 Processor -- start 2024-03-06T07:35:26.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.223+0000 7f0c5e400640 1 -- start start 2024-03-06T07:35:26.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c5e400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c580af1a0 0x7f0c58151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c5e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c581466d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c5e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 0x7f0c58157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c5e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b3660 con 0x7f0c580cdc20 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c5e400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b34e0 con 0x7f0c580af1a0 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c5e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580b37e0 con 0x7f0c580af580 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 0x7f0c58157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 0x7f0c58157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59126/0 (socket says 172.21.15.6:59126) 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 -- 172.21.15.6:0/152576408 learned_addr learned my addr 172.21.15.6:0/152576408 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c4fe00640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c581466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c580af1a0 0x7f0c58151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 -- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 msgr2=0x7f0c581466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c581466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 -- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c580af1a0 msgr2=0x7f0c58151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c580af1a0 0x7f0c58151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.224+0000 7f0c57e00640 1 -- 172.21.15.6:0/152576408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c58158cf0 con 0x7f0c580cdc20 2024-03-06T07:35:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.225+0000 7f0c57e00640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 0x7f0c58157390 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000c6b0 tx=0x7f0c5000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.225+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50003b70 con 0x7f0c580cdc20 2024-03-06T07:35:26.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.225+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c50003d30 con 0x7f0c580cdc20 2024-03-06T07:35:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.225+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c50005310 con 0x7f0c580cdc20 2024-03-06T07:35:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.225+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c581558c0 con 0x7f0c580cdc20 2024-03-06T07:35:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.225+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c58158f90 con 0x7f0c580cdc20 2024-03-06T07:35:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.226+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c58069510 con 0x7f0c580cdc20 2024-03-06T07:35:26.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.227+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c500054b0 con 0x7f0c580cdc20 2024-03-06T07:35:26.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.228+0000 7f0c4f400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0c3406b4d0 0x7f0c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.228+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0c50089d40 con 0x7f0c580cdc20 2024-03-06T07:35:26.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.229+0000 7f0c57400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0c3406b4d0 0x7f0c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.230+0000 7f0c57400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0c3406b4d0 0x7f0c3406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0c5815b640 tx=0x7f0c48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.232+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c50058f00 con 0x7f0c580cdc20 2024-03-06T07:35:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.505+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0c5804f3a0 con 0x7f0c580cdc20 2024-03-06T07:35:26.508 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.506+0000 7f0c4f400640 1 -- 172.21.15.6:0/152576408 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0c50058800 con 0x7f0c580cdc20 2024-03-06T07:35:26.508 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_dump": [ 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.a", 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSYrnBhAAzD0RJbNSE6r43EdChSNhow==", 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.509 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.b", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl6051KhAAexiPj9KJ7v2TgEUGonrijQ==", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.c", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhlSTGTGRAAzM3rt0FEOIQ0TvmghP1ozg==", 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.510 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mds.d", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQChGuhl3gILOxAAMxZvz7dv+zLY9FwWh44Heg==", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:35:26.511 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.0", 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCmGuhlfhvzORAAW/B9OiyfmLygk7+OGyr0Ng==", 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.1", 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCqGuhl1UxfGBAAHMH2sv16KFEl36d1r+8hhQ==", 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.512 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.2", 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCtGuhl6cfsMRAAsfwh48aOquQ33/bpFm7R9w==", 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.513 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.3", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCxGuhlqCyRARAAT2mloQ/K+GM/Md1FFizulw==", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.4", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC1Guhlh1lTKhAAtLdfxgmmADAr3k/UQhI4ww==", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.514 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.5", 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC4Guhltz6KMxAAgypFXykPHPu0EMuduPf0YA==", 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.6", 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC8GuhlK2KjABAAjlBDUHbq3wbIKdwqXNUZLQ==", 2024-03-06T07:35:26.515 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "osd.7", 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQC/Guhlh6OjGBAAXZYYwigBJuneU+mf7uLtfw==", 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow profile osd", 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile osd", 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.0", 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ==", 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow r", 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.1", 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlQ4UwJBAAd7wZdTdpR9CTPIxlGmjKjQ==", 2024-03-06T07:35:26.517 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.2", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlvuf/GBAATUqhw3/5f/nzKYsRlzbneQ==", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.3", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCiGuhlKhboLhAAMz0RrBcTzsMHE8HpsZLxHw==", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow r", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow rw", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow rwx" 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "client.admin", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCfGuhljTirMhAAEyWCPT/RdTHxzoDRy1TQxg==", 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.519 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": "allow *", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow *", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.x", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhllhX/NRAAwIzHY/W8n/lNlJyln/ew1g==", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: "entity": "mgr.y", 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "AQCgGuhl0YzxKRAAHv5NkqJq0H5uoNX4p64AfA==", 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: "caps": { 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: "mds": "allow *", 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: "mon": "allow profile mgr", 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": "allow *" 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-06T07:35:26.521 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0c3406b4d0 msgr2=0x7f0c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0c3406b4d0 0x7f0c3406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0c5815b640 tx=0x7f0c48008040 comp rx=0 tx=0).stop 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 msgr2=0x7f0c58157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 0x7f0c58157390 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f0c5000c6b0 tx=0x7f0c5000cb70 comp rx=0 tx=0).stop 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 shutdown_connections 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0c3406b4d0 0x7f0c3406d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0c580cdc20 0x7f0c58157390 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0c580af580 0x7f0c581466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 --2- 172.21.15.6:0/152576408 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c580af1a0 0x7f0c58151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 >> 172.21.15.6:0/152576408 conn(0x7f0c5806afb0 msgr2=0x7f0c580cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 shutdown_connections 2024-03-06T07:35:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.509+0000 7f0c5e400640 1 -- 172.21.15.6:0/152576408 wait complete. 2024-03-06T07:35:26.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:26.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.747+0000 7fa782000640 1 Processor -- start 2024-03-06T07:35:26.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.747+0000 7fa782000640 1 -- start start 2024-03-06T07:35:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa782000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fa77c0cd060 0x7fa77c161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa782000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa77c162000 0x7fa77c070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa782000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c1627f0 con 0x7fa77c15efc0 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa782000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c15f3a0 con 0x7fa77c162000 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa782000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c06d190 con 0x7fa77c0cd060 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa780c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa77c15efc0 0x7fa77c0cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56938/0 (socket says 172.21.15.6:56938) 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa780c00640 1 -- 172.21.15.6:0/1202646015 learned_addr learned my addr 172.21.15.6:0/1202646015 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.749+0000 7fa781600640 1 --2- 172.21.15.6:0/1202646015 >> v2:172.21.15.6:3300/0 conn(0x7fa77c162000 0x7fa77c070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.750+0000 7fa77be00640 1 --2- 172.21.15.6:0/1202646015 >> v2:172.21.15.178:3300/0 conn(0x7fa77c0cd060 0x7fa77c161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.750+0000 7fa781600640 1 -- 172.21.15.6:0/1202646015 >> v2:172.21.15.178:3300/0 conn(0x7fa77c0cd060 msgr2=0x7fa77c161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.750+0000 7fa781600640 1 --2- 172.21.15.6:0/1202646015 >> v2:172.21.15.178:3300/0 conn(0x7fa77c0cd060 0x7fa77c161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.750+0000 7fa781600640 1 -- 172.21.15.6:0/1202646015 >> v1:172.21.15.6:6789/0 conn(0x7fa77c15efc0 legacy=0x7fa77c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.750+0000 7fa781600640 1 -- 172.21.15.6:0/1202646015 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c06c860 con 0x7fa77c162000 2024-03-06T07:35:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.750+0000 7fa77b400640 1 -- 172.21.15.6:0/1202646015 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532276850 0 0) 0x7fa77c1627f0 con 0x7fa77c15efc0 2024-03-06T07:35:26.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.751+0000 7fa781600640 1 --2- 172.21.15.6:0/1202646015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c162000 0x7fa77c070c20 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0b51d0 tx=0x7fa77402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4466f4a2752d617 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.751+0000 7fa77b400640 1 -- 172.21.15.6:0/1202646015 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774039070 con 0x7fa77c162000 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.751+0000 7fa77b400640 1 -- 172.21.15.6:0/1202646015 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77402cc80 con 0x7fa77c162000 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa77b400640 1 -- 172.21.15.6:0/1202646015 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774034660 con 0x7fa77c162000 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 -- 172.21.15.6:0/1202646015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c162000 msgr2=0x7fa77c070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 --2- 172.21.15.6:0/1202646015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c162000 0x7fa77c070c20 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0b51d0 tx=0x7fa77402bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 -- 172.21.15.6:0/1202646015 shutdown_connections 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 --2- 172.21.15.6:0/1202646015 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c162000 0x7fa77c070c20 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 --2- 172.21.15.6:0/1202646015 >> v2:172.21.15.178:3300/0 conn(0x7fa77c0cd060 0x7fa77c161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 -- 172.21.15.6:0/1202646015 >> 172.21.15.6:0/1202646015 conn(0x7fa77c06b3d0 msgr2=0x7fa77c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 -- 172.21.15.6:0/1202646015 shutdown_connections 2024-03-06T07:35:26.755 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.752+0000 7fa782000640 1 -- 172.21.15.6:0/1202646015 wait complete. 2024-03-06T07:35:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.754+0000 7fa782000640 1 Processor -- start 2024-03-06T07:35:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.754+0000 7fa782000640 1 -- start start 2024-03-06T07:35:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa782000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa77c0cd060 0x7fa77c1404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa782000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 0x7fa77c140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa782000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa77c162000 0x7fa77c143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa782000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c162470 con 0x7fa77c15efc0 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa782000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c06ce20 con 0x7fa77c162000 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa782000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c06d620 con 0x7fa77c0cd060 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa77be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 0x7fa77c140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa77be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 0x7fa77c140d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59146/0 (socket says 172.21.15.6:59146) 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa77be00640 1 -- 172.21.15.6:0/3864828054 learned_addr learned my addr 172.21.15.6:0/3864828054 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.755+0000 7fa780c00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa77c0cd060 0x7fa77c1404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa77be00640 1 -- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa77c0cd060 msgr2=0x7fa77c1404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa781600640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa77c162000 0x7fa77c143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa77be00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa77c0cd060 0x7fa77c1404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa77be00640 1 -- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa77c162000 msgr2=0x7fa77c143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa77be00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa77c162000 0x7fa77c143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa77be00640 1 -- 172.21.15.6:0/3864828054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c141800 con 0x7fa77c15efc0 2024-03-06T07:35:26.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa780c00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa77c0cd060 0x7fa77c1404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa781600640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa77c162000 0x7fa77c143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa77be00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 0x7fa77c140d60 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa76800d800 tx=0x7fa76800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76800b6c0 con 0x7fa77c15efc0 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa768019070 con 0x7fa77c15efc0 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76800ba80 con 0x7fa77c15efc0 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c141ae0 con 0x7fa77c15efc0 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.756+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c1420a0 con 0x7fa77c15efc0 2024-03-06T07:35:26.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.757+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c04f330 con 0x7fa77c15efc0 2024-03-06T07:35:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.758+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa76800bc20 con 0x7fa77c15efc0 2024-03-06T07:35:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.759+0000 7fa778c00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa75c06b4d0 0x7fa75c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:26.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.759+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa76808a400 con 0x7fa77c15efc0 2024-03-06T07:35:26.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.760+0000 7fa780c00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa75c06b4d0 0x7fa75c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:26.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.761+0000 7fa780c00640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa75c06b4d0 0x7fa75c06d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa770000f60 tx=0x7fa770009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:26.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.764+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa768059530 con 0x7fa77c15efc0 2024-03-06T07:35:26.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.963+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa77c0ce600 con 0x7fa77c15efc0 2024-03-06T07:35:26.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.965+0000 7fa778c00640 1 -- 172.21.15.6:0/3864828054 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+12829 (secure 0 0 0) 0x7fa768058e30 con 0x7fa77c15efc0 2024-03-06T07:35:26.967 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:26.967 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":122,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:35:25.970334+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:26.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.967+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa75c06b4d0 msgr2=0x7fa75c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.967+0000 7fa782000640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa75c06b4d0 0x7fa75c06d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa770000f60 tx=0x7fa770009040 comp rx=0 tx=0).stop 2024-03-06T07:35:26.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.967+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 msgr2=0x7fa77c140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:26.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.967+0000 7fa782000640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 0x7fa77c140d60 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa76800d800 tx=0x7fa76800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 shutdown_connections 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa75c06b4d0 0x7fa75c06d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa77c162000 0x7fa77c143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa77c15efc0 0x7fa77c140d60 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 --2- 172.21.15.6:0/3864828054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa77c0cd060 0x7fa77c1404b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 >> 172.21.15.6:0/3864828054 conn(0x7fa77c06b3d0 msgr2=0x7fa77c0cdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 shutdown_connections 2024-03-06T07:35:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:26.968+0000 7fa782000640 1 -- 172.21.15.6:0/3864828054 wait complete. 2024-03-06T07:35:26.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:35:26.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:35:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.205+0000 7f207d000640 1 Processor -- start 2024-03-06T07:35:27.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.206+0000 7f207d000640 1 -- start start 2024-03-06T07:35:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.207+0000 7f207d000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f207815dee0 0x7f20780d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.207+0000 7f207d000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20780ceca0 con 0x7f20780d0d40 2024-03-06T07:35:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.207+0000 7f207d000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20780cef80 con 0x7f20780d4860 2024-03-06T07:35:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.207+0000 7f207d000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20780cf5d0 con 0x7f207815dee0 2024-03-06T07:35:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.207+0000 7f2077e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f20780d0d40 0x7f20780b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56954/0 (socket says 172.21.15.6:56954) 2024-03-06T07:35:27.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.207+0000 7f2077e00640 1 -- 172.21.15.6:0/303701444 learned_addr learned my addr 172.21.15.6:0/303701444 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382403352 0 0) 0x7f20780ceca0 con 0x7f20780d0d40 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2077400640 1 --2- 172.21.15.6:0/303701444 >> v2:172.21.15.178:3300/0 conn(0x7f207815dee0 0x7f20780d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2058003520 con 0x7f20780d0d40 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3654080432 0 0) 0x7f2058003520 con 0x7f20780d0d40 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 >> v2:172.21.15.178:3300/0 conn(0x7f207815dee0 msgr2=0x7f20780d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 --2- 172.21.15.6:0/303701444 >> v2:172.21.15.178:3300/0 conn(0x7f207815dee0 0x7f20780d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 >> v1:172.21.15.178:6789/0 conn(0x7f20780d4860 legacy=0x7f20780d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20780cf260 con 0x7f20780d0d40 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f20700019f0 con 0x7f20780d0d40 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2077400640 1 --2- 172.21.15.6:0/303701444 >> v2:172.21.15.178:3300/0 conn(0x7f207815dee0 0x7f20780d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 >> v1:172.21.15.6:6789/0 conn(0x7f20780d0d40 legacy=0x7f20780b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2058003a00 0x7f2058003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058004800 0x7f2058004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2058005210 0x7f205800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20780cf5d0 con 0x7f2058005210 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205800c160 con 0x7f2058003a00 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.208+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205800c360 con 0x7f2058004800 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077e00640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2058005210 0x7f205800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077400640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2058003a00 0x7f2058003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077e00640 1 -- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058004800 msgr2=0x7f2058004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077e00640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058004800 0x7f2058004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077e00640 1 -- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2058003a00 msgr2=0x7f2058003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077e00640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2058003a00 0x7f2058003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.209+0000 7f2077e00640 1 -- 172.21.15.6:0/303701444 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2058003520 con 0x7f2058005210 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.210+0000 7f2077e00640 1 -- 172.21.15.6:0/303701444 reap_dead start 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.210+0000 7f2077e00640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2058005210 0x7f205800b9c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f207000a790 tx=0x7f2058005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54cbaa29baf40bb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.210+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207000a3b0 con 0x7f2058005210 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.210+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2070034670 con 0x7f2058005210 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.210+0000 7f2076000640 1 -- 172.21.15.6:0/303701444 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070034a60 con 0x7f2058005210 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 -- 172.21.15.6:0/303701444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2058005210 msgr2=0x7f205800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2058005210 0x7f205800b9c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f207000a790 tx=0x7f2058005830 comp rx=0 tx=0).stop 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 -- 172.21.15.6:0/303701444 shutdown_connections 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 --2- 172.21.15.6:0/303701444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2058005210 0x7f205800b9c0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 -- 172.21.15.6:0/303701444 >> 172.21.15.6:0/303701444 conn(0x7f20780d3cd0 msgr2=0x7f20780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 -- 172.21.15.6:0/303701444 shutdown_connections 2024-03-06T07:35:27.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.211+0000 7f207d000640 1 -- 172.21.15.6:0/303701444 wait complete. 2024-03-06T07:35:27.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.213+0000 7f207d000640 1 Processor -- start 2024-03-06T07:35:27.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.213+0000 7f207d000640 1 -- start start 2024-03-06T07:35:27.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f207d000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058005210 0x7f20781583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f207d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 0x7f20781f7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f207d000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20781f7da0 0x7f20781f8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f207d000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20780cfa60 con 0x7f207815a1c0 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f207d000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207815e2c0 con 0x7f20781f7da0 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f207d000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20780cfbe0 con 0x7f2058005210 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 0x7f20781f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 0x7f20781f7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59166/0 (socket says 172.21.15.6:59166) 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 -- 172.21.15.6:0/1908390603 learned_addr learned my addr 172.21.15.6:0/1908390603 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2077400640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058005210 0x7f20781583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2077e00640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20781f7da0 0x7f20781f8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 -- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058005210 msgr2=0x7f20781583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058005210 0x7f20781583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 -- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20781f7da0 msgr2=0x7f20781f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20781f7da0 0x7f20781f8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2076a00640 1 -- 172.21.15.6:0/1908390603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20780b4460 con 0x7f207815a1c0 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2077400640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058005210 0x7f20781583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.214+0000 7f2077e00640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20781f7da0 0x7f20781f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.215+0000 7f2076a00640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 0x7f20781f7860 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f206000c500 tx=0x7f206000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.215+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20600052d0 con 0x7f207815a1c0 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.215+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2060005470 con 0x7f207815a1c0 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.215+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20780b4740 con 0x7f207815a1c0 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.215+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2060012850 con 0x7f207815a1c0 2024-03-06T07:35:27.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.215+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2078159870 con 0x7f207815a1c0 2024-03-06T07:35:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.216+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f207804f330 con 0x7f207815a1c0 2024-03-06T07:35:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.217+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20600129f0 con 0x7f207815a1c0 2024-03-06T07:35:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.218+0000 7f206f400640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f205406b4d0 0x7f205406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.218+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2060089bf0 con 0x7f207815a1c0 2024-03-06T07:35:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.219+0000 7f2077400640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f205406b4d0 0x7f205406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.220+0000 7f2077400640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f205406b4d0 0x7f205406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2068006fd0 tx=0x7f2068008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.222+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f20600182b0 con 0x7f207815a1c0 2024-03-06T07:35:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.421+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20780d0b00 con 0x7f207815a1c0 2024-03-06T07:35:27.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.422+0000 7f206f400640 1 -- 172.21.15.6:0/1908390603 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2060058db0 con 0x7f207815a1c0 2024-03-06T07:35:27.425 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:35:27.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f205406b4d0 msgr2=0x7f205406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f205406b4d0 0x7f205406d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2068006fd0 tx=0x7f2068008040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 msgr2=0x7f20781f7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 0x7f20781f7860 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f206000c500 tx=0x7f206000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 shutdown_connections 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f205406b4d0 0x7f205406d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20781f7da0 0x7f20781f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f207815a1c0 0x7f20781f7860 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 --2- 172.21.15.6:0/1908390603 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2058005210 0x7f20781583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 >> 172.21.15.6:0/1908390603 conn(0x7f20780d3cd0 msgr2=0x7f207806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 shutdown_connections 2024-03-06T07:35:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.425+0000 7f207d000640 1 -- 172.21.15.6:0/1908390603 wait complete. 2024-03-06T07:35:27.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:35:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.665+0000 7fb7a2200640 1 Processor -- start 2024-03-06T07:35:27.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.665+0000 7fb7a2200640 1 -- start start 2024-03-06T07:35:27.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb7a2200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb79c0af1a0 0x7fb79c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb7a2200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 0x7fb79c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb7a2200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79c15cc40 con 0x7fb79c0af580 2024-03-06T07:35:27.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb7a2200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79c15cf20 con 0x7fb79c0af1a0 2024-03-06T07:35:27.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb7a2200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c15d200 con 0x7fb79c0cdc20 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb79be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 0x7fb79c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb79be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 0x7fb79c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:43202/0 (socket says 172.21.15.6:43202) 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb79be00640 1 -- 172.21.15.6:0/2650887121 learned_addr learned my addr 172.21.15.6:0/2650887121 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb793e00640 1 --1- 172.21.15.6:0/2650887121 >> v1:172.21.15.178:6789/0 conn(0x7fb79c0af580 0x7fb79c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:49642/0 (socket says 172.21.15.6:49642) 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.667+0000 7fb79b400640 1 --2- 172.21.15.6:0/2650887121 >> v2:172.21.15.178:3300/0 conn(0x7fb79c0af1a0 0x7fb79c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.668+0000 7fb79be00640 1 -- 172.21.15.6:0/2650887121 >> v2:172.21.15.178:3300/0 conn(0x7fb79c0af1a0 msgr2=0x7fb79c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.668+0000 7fb79be00640 1 --2- 172.21.15.6:0/2650887121 >> v2:172.21.15.178:3300/0 conn(0x7fb79c0af1a0 0x7fb79c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.668+0000 7fb79be00640 1 -- 172.21.15.6:0/2650887121 >> v1:172.21.15.178:6789/0 conn(0x7fb79c0af580 legacy=0x7fb79c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.668+0000 7fb79be00640 1 -- 172.21.15.6:0/2650887121 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c15d4e0 con 0x7fb79c0cdc20 2024-03-06T07:35:27.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.669+0000 7fb79be00640 1 --2- 172.21.15.6:0/2650887121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 0x7fb79c0b2dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb794009e50 tx=0x7fb79402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b5bb2b8d521858 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.670+0000 7fb79aa00640 1 -- 172.21.15.6:0/2650887121 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79402ca80 con 0x7fb79c0cdc20 2024-03-06T07:35:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.670+0000 7fb79aa00640 1 -- 172.21.15.6:0/2650887121 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79402cc20 con 0x7fb79c0cdc20 2024-03-06T07:35:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.670+0000 7fb79aa00640 1 -- 172.21.15.6:0/2650887121 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7940343f0 con 0x7fb79c0cdc20 2024-03-06T07:35:27.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 -- 172.21.15.6:0/2650887121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 msgr2=0x7fb79c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 --2- 172.21.15.6:0/2650887121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 0x7fb79c0b2dd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb794009e50 tx=0x7fb79402bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 -- 172.21.15.6:0/2650887121 shutdown_connections 2024-03-06T07:35:27.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 --2- 172.21.15.6:0/2650887121 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0cdc20 0x7fb79c0b2dd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 --2- 172.21.15.6:0/2650887121 >> v2:172.21.15.178:3300/0 conn(0x7fb79c0af1a0 0x7fb79c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 -- 172.21.15.6:0/2650887121 >> 172.21.15.6:0/2650887121 conn(0x7fb79c06afb0 msgr2=0x7fb79c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 -- 172.21.15.6:0/2650887121 shutdown_connections 2024-03-06T07:35:27.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.671+0000 7fb7a2200640 1 -- 172.21.15.6:0/2650887121 wait complete. 2024-03-06T07:35:27.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.673+0000 7fb7a2200640 1 Processor -- start 2024-03-06T07:35:27.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.673+0000 7fb7a2200640 1 -- start start 2024-03-06T07:35:27.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb7a2200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0af1a0 0x7fb79c09b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb7a2200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 0x7fb79c148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb7a2200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb79c0cdc20 0x7fb79c148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb7a2200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c0b3720 con 0x7fb79c0af580 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb7a2200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c0b35a0 con 0x7fb79c0cdc20 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb7a2200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79c155e70 con 0x7fb79c0af1a0 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb793e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 0x7fb79c148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb793e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 0x7fb79c148230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59168/0 (socket says 172.21.15.6:59168) 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb793e00640 1 -- 172.21.15.6:0/3981886717 learned_addr learned my addr 172.21.15.6:0/3981886717 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.674+0000 7fb79b400640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0af1a0 0x7fb79c09b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb79be00640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb79c0cdc20 0x7fb79c148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793e00640 1 -- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0af1a0 msgr2=0x7fb79c09b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793e00640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0af1a0 0x7fb79c09b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793e00640 1 -- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb79c0cdc20 msgr2=0x7fb79c148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793e00640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb79c0cdc20 0x7fb79c148770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793e00640 1 -- 172.21.15.6:0/3981886717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c06cb50 con 0x7fb79c0af580 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793e00640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 0x7fb79c148230 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c470 tx=0x7fb784001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7840074e0 con 0x7fb79c0af580 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784007680 con 0x7fb79c0af580 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784004a20 con 0x7fb79c0af580 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.675+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb79c15edb0 con 0x7fb79c0af580 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.676+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb79c06cdf0 con 0x7fb79c0af580 2024-03-06T07:35:27.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.677+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79c069510 con 0x7fb79c0af580 2024-03-06T07:35:27.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.678+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb78401a020 con 0x7fb79c0af580 2024-03-06T07:35:27.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.679+0000 7fb793400640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb77806b4d0 0x7fb77806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:27.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.679+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb7840896a0 con 0x7fb79c0af580 2024-03-06T07:35:27.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.679+0000 7fb79b400640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb77806b4d0 0x7fb77806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:27.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.681+0000 7fb79b400640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb77806b4d0 0x7fb77806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009ad0 tx=0x7fb78c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:27.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.683+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb784010300 con 0x7fb79c0af580 2024-03-06T07:35:27.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.888+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb79c04f3a0 con 0x7fb79c0af580 2024-03-06T07:35:27.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.889+0000 7fb793400640 1 -- 172.21.15.6:0/3981886717 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v122) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb784058860 con 0x7fb79c0af580 2024-03-06T07:35:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:27.892 INFO:teuthology.orchestra.run.smithi006.stdout:[ 2024-03-06T07:35:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-06T07:35:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "poolnum": 1, 2024-03-06T07:35:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "poolname": ".mgr" 2024-03-06T07:35:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-06T07:35:27.892 INFO:teuthology.orchestra.run.smithi006.stdout:] 2024-03-06T07:35:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb77806b4d0 msgr2=0x7fb77806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb77806b4d0 0x7fb77806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009ad0 tx=0x7fb78c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:27.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 msgr2=0x7fb79c148230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 0x7fb79c148230 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c470 tx=0x7fb784001e70 comp rx=0 tx=0).stop 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 shutdown_connections 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb77806b4d0 0x7fb77806d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb79c0cdc20 0x7fb79c148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb79c0af580 0x7fb79c148230 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 --2- 172.21.15.6:0/3981886717 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb79c0af1a0 0x7fb79c09b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 >> 172.21.15.6:0/3981886717 conn(0x7fb79c06afb0 msgr2=0x7fb79c0ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 shutdown_connections 2024-03-06T07:35:27.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:27.892+0000 7fb7a2200640 1 -- 172.21.15.6:0/3981886717 wait complete. 2024-03-06T07:35:27.913 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:35:27.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:35:28.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.144+0000 7fc1c7e00640 1 Processor -- start 2024-03-06T07:35:28.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.145+0000 7fc1c7e00640 1 -- start start 2024-03-06T07:35:28.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.146+0000 7fc1c7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1c80d0740 0x7fc1c80cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.146+0000 7fc1c7e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc1c81627f0 0x7fc1c80d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.146+0000 7fc1c7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d4440 0x7fc1c80b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.146+0000 7fc1c7e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c80cf410 con 0x7fc1c80d0740 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.146+0000 7fc1c7e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c80ce840 con 0x7fc1c81627f0 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.146+0000 7fc1c7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80d1060 con 0x7fc1c80d4440 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1c80d0740 0x7fc1c80cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1c80d0740 0x7fc1c80cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59184/0 (socket says 172.21.15.6:59184) 2024-03-06T07:35:28.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 -- 172.21.15.6:0/2784570759 learned_addr learned my addr 172.21.15.6:0/2784570759 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c7400640 1 --2- 172.21.15.6:0/2784570759 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d4440 0x7fc1c80b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1bfe00640 1 --2- 172.21.15.6:0/2784570759 >> v2:172.21.15.178:3300/0 conn(0x7fc1c81627f0 0x7fc1c80d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 -- 172.21.15.6:0/2784570759 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d4440 msgr2=0x7fc1c80b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 --2- 172.21.15.6:0/2784570759 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d4440 0x7fc1c80b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 -- 172.21.15.6:0/2784570759 >> v2:172.21.15.178:3300/0 conn(0x7fc1c81627f0 msgr2=0x7fc1c80d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 --2- 172.21.15.6:0/2784570759 >> v2:172.21.15.178:3300/0 conn(0x7fc1c81627f0 0x7fc1c80d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.150 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.147+0000 7fc1c6a00640 1 -- 172.21.15.6:0/2784570759 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80ceae0 con 0x7fc1c80d0740 2024-03-06T07:35:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.149+0000 7fc1c6a00640 1 --2- 172.21.15.6:0/2784570759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c80d0740 0x7fc1c80cf8a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80c74d0 tx=0x7fc1b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02818ba3041b9c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.149+0000 7fc1c6000640 1 -- 172.21.15.6:0/2784570759 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8034070 con 0x7fc1c80d0740 2024-03-06T07:35:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.149+0000 7fc1c6000640 1 -- 172.21.15.6:0/2784570759 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8030860 con 0x7fc1c80d0740 2024-03-06T07:35:28.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.149+0000 7fc1c6000640 1 -- 172.21.15.6:0/2784570759 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8030c30 con 0x7fc1c80d0740 2024-03-06T07:35:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2784570759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c80d0740 msgr2=0x7fc1c80cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2784570759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c80d0740 0x7fc1c80cf8a0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80c74d0 tx=0x7fc1b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2784570759 shutdown_connections 2024-03-06T07:35:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2784570759 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d4440 0x7fc1c80b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2784570759 >> v2:172.21.15.178:3300/0 conn(0x7fc1c81627f0 0x7fc1c80d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2784570759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c80d0740 0x7fc1c80cf8a0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2784570759 >> 172.21.15.6:0/2784570759 conn(0x7fc1c808d2a0 msgr2=0x7fc1c80d3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2784570759 shutdown_connections 2024-03-06T07:35:28.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.150+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2784570759 wait complete. 2024-03-06T07:35:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.152+0000 7fc1c7e00640 1 Processor -- start 2024-03-06T07:35:28.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.152+0000 7fc1c7e00640 1 -- start start 2024-03-06T07:35:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d0740 0x7fc1c80cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1c80d4440 0x7fc1c80cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 0x7fc1c80cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8161760 con 0x7fc1c81627f0 2024-03-06T07:35:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c80cf0a0 con 0x7fc1c80d4440 2024-03-06T07:35:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81618e0 con 0x7fc1c80d0740 2024-03-06T07:35:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 0x7fc1c80cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 0x7fc1c80cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59196/0 (socket says 172.21.15.6:59196) 2024-03-06T07:35:28.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.153+0000 7fc1c7400640 1 -- 172.21.15.6:0/2041373710 learned_addr learned my addr 172.21.15.6:0/2041373710 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c6a00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d0740 0x7fc1c80cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1bfe00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1c80d4440 0x7fc1c80cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7400640 1 -- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d0740 msgr2=0x7fc1c80cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7400640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d0740 0x7fc1c80cbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7400640 1 -- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1c80d4440 msgr2=0x7fc1c80cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7400640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1c80d4440 0x7fc1c80cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:28.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7400640 1 -- 172.21.15.6:0/2041373710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c81f3ea0 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1bfe00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1c80d4440 0x7fc1c80cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7400640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 0x7fc1c80cc8d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0007d20 tx=0x7fc1c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0014420 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c0019070 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0014820 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.154+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c80b9660 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.155+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c81f4160 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.156+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c804f330 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.157+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1c00149c0 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.158+0000 7fc1bea00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc1a406b4d0 0x7fc1a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.158+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc1c008a6f0 con 0x7fc1c81627f0 2024-03-06T07:35:28.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.159+0000 7fc1c6a00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc1a406b4d0 0x7fc1a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:28.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.160+0000 7fc1c6a00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc1a406b4d0 0x7fc1a406d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8036000 tx=0x7fc1b803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:28.164 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.163+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc1c0059790 con 0x7fc1c81627f0 2024-03-06T07:35:28.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:28.370+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc1c80d0520 con 0x7fc1c81627f0 2024-03-06T07:35:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.000+0000 7fc1bea00640 1 -- 172.21.15.6:0/2041373710 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v123) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc1c0059090 con 0x7fc1c81627f0 2024-03-06T07:35:29.003 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_metadata' created 2024-03-06T07:35:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc1a406b4d0 msgr2=0x7fc1a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc1a406b4d0 0x7fc1a406d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8036000 tx=0x7fc1b803b040 comp rx=0 tx=0).stop 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 msgr2=0x7fc1c80cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 0x7fc1c80cc8d0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0007d20 tx=0x7fc1c000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 shutdown_connections 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc1a406b4d0 0x7fc1a406d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1c80d4440 0x7fc1c80cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1c81627f0 0x7fc1c80cc8d0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 --2- 172.21.15.6:0/2041373710 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1c80d0740 0x7fc1c80cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 >> 172.21.15.6:0/2041373710 conn(0x7fc1c808d2a0 msgr2=0x7fc1c815c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.005+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 shutdown_connections 2024-03-06T07:35:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.006+0000 7fc1c7e00640 1 -- 172.21.15.6:0/2041373710 wait complete. 2024-03-06T07:35:29.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:35:29.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.242+0000 7ff630c00640 1 Processor -- start 2024-03-06T07:35:29.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.242+0000 7ff630c00640 1 -- start start 2024-03-06T07:35:29.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.243+0000 7ff630c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff62c161320 0x7ff62c0d2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.243+0000 7ff630c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d48d0 0x7ff62c0d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.243+0000 7ff630c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff62c0d2be0 con 0x7ff62c0d0b50 2024-03-06T07:35:29.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.243+0000 7ff630c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62c0d0fc0 con 0x7ff62c161320 2024-03-06T07:35:29.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.243+0000 7ff630c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c0b3550 con 0x7ff62c0d48d0 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.244+0000 7ff62be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff62c0d0b50 0x7ff62c0ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56962/0 (socket says 172.21.15.6:56962) 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.244+0000 7ff62be00640 1 -- 172.21.15.6:0/3024542679 learned_addr learned my addr 172.21.15.6:0/3024542679 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.244+0000 7ff62b400640 1 --2- 172.21.15.6:0/3024542679 >> v2:172.21.15.178:3300/0 conn(0x7ff62c161320 0x7ff62c0d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.244+0000 7ff62aa00640 1 --2- 172.21.15.6:0/3024542679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d48d0 0x7ff62c0d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62a000640 1 -- 172.21.15.6:0/3024542679 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188363399 0 0) 0x7ff62c0d2be0 con 0x7ff62c0d0b50 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62a000640 1 -- 172.21.15.6:0/3024542679 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff610003520 con 0x7ff62c0d0b50 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62aa00640 1 -- 172.21.15.6:0/3024542679 >> v2:172.21.15.178:3300/0 conn(0x7ff62c161320 msgr2=0x7ff62c0d2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62aa00640 1 --2- 172.21.15.6:0/3024542679 >> v2:172.21.15.178:3300/0 conn(0x7ff62c161320 0x7ff62c0d2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62a000640 1 -- 172.21.15.6:0/3024542679 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438957911 0 0) 0x7ff610003520 con 0x7ff62c0d0b50 2024-03-06T07:35:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62aa00640 1 -- 172.21.15.6:0/3024542679 >> v1:172.21.15.6:6789/0 conn(0x7ff62c0d0b50 legacy=0x7ff62c0ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.245+0000 7ff62aa00640 1 -- 172.21.15.6:0/3024542679 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c0b3830 con 0x7ff62c0d48d0 2024-03-06T07:35:29.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.246+0000 7ff62aa00640 1 --2- 172.21.15.6:0/3024542679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d48d0 0x7ff62c0d0610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff6140097a0 tx=0x7ff61402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c21cee6aea6c4c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.247+0000 7ff62a000640 1 -- 172.21.15.6:0/3024542679 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614037070 con 0x7ff62c0d48d0 2024-03-06T07:35:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.247+0000 7ff62a000640 1 -- 172.21.15.6:0/3024542679 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff614030510 con 0x7ff62c0d48d0 2024-03-06T07:35:29.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.247+0000 7ff62a000640 1 -- 172.21.15.6:0/3024542679 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240019f0 con 0x7ff62c0d48d0 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 -- 172.21.15.6:0/3024542679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d48d0 msgr2=0x7ff62c0d0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 --2- 172.21.15.6:0/3024542679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d48d0 0x7ff62c0d0610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff6140097a0 tx=0x7ff61402b790 comp rx=0 tx=0).stop 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 -- 172.21.15.6:0/3024542679 shutdown_connections 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 --2- 172.21.15.6:0/3024542679 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d48d0 0x7ff62c0d0610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 --2- 172.21.15.6:0/3024542679 >> v2:172.21.15.178:3300/0 conn(0x7ff62c161320 0x7ff62c0d2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 -- 172.21.15.6:0/3024542679 >> 172.21.15.6:0/3024542679 conn(0x7ff62c0d3d40 msgr2=0x7ff62c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:29.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 -- 172.21.15.6:0/3024542679 shutdown_connections 2024-03-06T07:35:29.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.248+0000 7ff630c00640 1 -- 172.21.15.6:0/3024542679 wait complete. 2024-03-06T07:35:29.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.249+0000 7ff630c00640 1 Processor -- start 2024-03-06T07:35:29.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.250+0000 7ff630c00640 1 -- start start 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.250+0000 7ff630c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d0b50 0x7ff62c157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.250+0000 7ff630c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 0x7ff62c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff630c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff62c161320 0x7ff62c0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff630c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c0d2860 con 0x7ff62c0d48d0 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff630c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c0b3df0 con 0x7ff62c161320 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff630c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c161700 con 0x7ff62c0d0b50 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 0x7ff62c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 0x7ff62c0cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59208/0 (socket says 172.21.15.6:59208) 2024-03-06T07:35:29.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 -- 172.21.15.6:0/4085310060 learned_addr learned my addr 172.21.15.6:0/4085310060 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62be00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff62c161320 0x7ff62c0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62b400640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d0b50 0x7ff62c157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 -- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d0b50 msgr2=0x7ff62c157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d0b50 0x7ff62c157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 -- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff62c161320 msgr2=0x7ff62c0cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff62c161320 0x7ff62c0cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.251+0000 7ff62aa00640 1 -- 172.21.15.6:0/4085310060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c15a340 con 0x7ff62c0d48d0 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.252+0000 7ff62aa00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 0x7ff62c0cd8f0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff6140098c0 tx=0x7ff6140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.252+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614041020 con 0x7ff62c0d48d0 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.252+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61403c870 con 0x7ff62c0d48d0 2024-03-06T07:35:29.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.252+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62c159d80 con 0x7ff62c0d48d0 2024-03-06T07:35:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.252+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614037040 con 0x7ff62c0d48d0 2024-03-06T07:35:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.252+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c0cbb60 con 0x7ff62c0d48d0 2024-03-06T07:35:29.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.253+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62c08dda0 con 0x7ff62c0d48d0 2024-03-06T07:35:29.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.255+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff614044020 con 0x7ff62c0d48d0 2024-03-06T07:35:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.256+0000 7ff623400640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff60806b4d0 0x7ff60806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.256+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff6140abf60 con 0x7ff62c0d48d0 2024-03-06T07:35:29.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.256+0000 7ff62b400640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff60806b4d0 0x7ff60806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:29.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.257+0000 7ff62b400640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff60806b4d0 0x7ff60806d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009a10 tx=0x7ff61c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:29.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.260+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff62c08dda0 con 0x7ff62c0d48d0 2024-03-06T07:35:29.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:29.461+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff62c0d2130 con 0x7ff62c0d48d0 2024-03-06T07:35:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.006+0000 7ff623400640 1 -- 172.21.15.6:0/4085310060 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v124) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff61404b030 con 0x7ff62c0d48d0 2024-03-06T07:35:30.009 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'cephfs_data' created 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff60806b4d0 msgr2=0x7ff60806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff60806b4d0 0x7ff60806d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009a10 tx=0x7ff61c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 msgr2=0x7ff62c0cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 0x7ff62c0cd8f0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7ff6140098c0 tx=0x7ff6140046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 shutdown_connections 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff60806b4d0 0x7ff60806d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff62c161320 0x7ff62c0cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff62c0d48d0 0x7ff62c0cd8f0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 --2- 172.21.15.6:0/4085310060 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff62c0d0b50 0x7ff62c157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.010+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 >> 172.21.15.6:0/4085310060 conn(0x7ff62c0d3d40 msgr2=0x7ff62c06d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.011+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 shutdown_connections 2024-03-06T07:35:30.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.011+0000 7ff630c00640 1 -- 172.21.15.6:0/4085310060 wait complete. 2024-03-06T07:35:30.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:35:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.235+0000 7f89bd800640 1 Processor -- start 2024-03-06T07:35:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.235+0000 7f89bd800640 1 -- start start 2024-03-06T07:35:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.237+0000 7f89bd800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f89b80cf3a0 0x7f89b8070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.237+0000 7f89bd800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8071c60 con 0x7f89b815afa0 2024-03-06T07:35:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.237+0000 7f89bd800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8071f40 con 0x7f89b80cd1e0 2024-03-06T07:35:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.237+0000 7f89bd800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b80cfa10 con 0x7f89b80cf3a0 2024-03-06T07:35:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.237+0000 7f89b6a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f89b815afa0 0x7f89b80ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56976/0 (socket says 172.21.15.6:56976) 2024-03-06T07:35:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.237+0000 7f89b6a00640 1 -- 172.21.15.6:0/1752187370 learned_addr learned my addr 172.21.15.6:0/1752187370 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183817240 0 0) 0x7f89b8071c60 con 0x7f89b815afa0 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b7400640 1 --2- 172.21.15.6:0/1752187370 >> v2:172.21.15.178:3300/0 conn(0x7f89b80cf3a0 0x7f89b8070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f899c005730 con 0x7f89b815afa0 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1853556941 0 0) 0x7f899c005730 con 0x7f89b815afa0 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 >> v2:172.21.15.178:3300/0 conn(0x7f89b80cf3a0 msgr2=0x7f89b8070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 --2- 172.21.15.6:0/1752187370 >> v2:172.21.15.178:3300/0 conn(0x7f89b80cf3a0 0x7f89b8070ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 >> v1:172.21.15.178:6789/0 conn(0x7f89b80cd1e0 legacy=0x7f89b80b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b8072220 con 0x7f89b815afa0 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f89a80019f0 con 0x7f89b815afa0 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.238+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 >> v1:172.21.15.6:6789/0 conn(0x7f89b815afa0 legacy=0x7f89b80ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b5600640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f899c005bc0 0x7f899c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b5600640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c006a10 0x7f899c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b5600640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f899c0073d0 0x7f899c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80cfa10 con 0x7f899c0073d0 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899c00e270 con 0x7f899c006a10 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899c00e470 con 0x7f899c005bc0 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b7400640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f899c0073d0 0x7f899c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b6000640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c006a10 0x7f899c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b6a00640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f899c005bc0 0x7f899c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b7400640 1 -- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f899c005bc0 msgr2=0x7f899c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b7400640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f899c005bc0 0x7f899c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b7400640 1 -- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c006a10 msgr2=0x7f899c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b7400640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c006a10 0x7f899c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.239+0000 7f89b7400640 1 -- 172.21.15.6:0/1752187370 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899c005730 con 0x7f899c0073d0 2024-03-06T07:35:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.240+0000 7f89b6000640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c006a10 0x7f899c006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.240+0000 7f89b7400640 1 -- 172.21.15.6:0/1752187370 reap_dead start 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.241+0000 7f89b7400640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f899c0073d0 0x7f899c00dbd0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009ac0 tx=0x7f899c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1af85836e4865f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.241+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0009690 con 0x7f899c0073d0 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.241+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b0004020 con 0x7f899c0073d0 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.241+0000 7f89b5600640 1 -- 172.21.15.6:0/1752187370 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b0004410 con 0x7f899c0073d0 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 -- 172.21.15.6:0/1752187370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f899c0073d0 msgr2=0x7f899c00dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f899c0073d0 0x7f899c00dbd0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009ac0 tx=0x7f899c007a40 comp rx=0 tx=0).stop 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 -- 172.21.15.6:0/1752187370 shutdown_connections 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 --2- 172.21.15.6:0/1752187370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f899c0073d0 0x7f899c00dbd0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 -- 172.21.15.6:0/1752187370 >> 172.21.15.6:0/1752187370 conn(0x7f89b806b3d0 msgr2=0x7f89b806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 -- 172.21.15.6:0/1752187370 shutdown_connections 2024-03-06T07:35:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.242+0000 7f89bd800640 1 -- 172.21.15.6:0/1752187370 wait complete. 2024-03-06T07:35:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.243+0000 7f89bd800640 1 Processor -- start 2024-03-06T07:35:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.244+0000 7f89bd800640 1 -- start start 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.244+0000 7f89bd800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c0073d0 0x7f89b8152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89bd800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 0x7f89b814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89bd800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89b814f800 0x7f89b814fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89bd800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b8070ce0 con 0x7f89b814ba00 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89bd800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b80cf7d0 con 0x7f899c0073d0 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89bd800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b8070e60 con 0x7f89b814f800 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 0x7f89b814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 0x7f89b814c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59218/0 (socket says 172.21.15.6:59218) 2024-03-06T07:35:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 -- 172.21.15.6:0/4221267037 learned_addr learned my addr 172.21.15.6:0/4221267037 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6a00640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c0073d0 0x7f89b8152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 -- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89b814f800 msgr2=0x7f89b814fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b7400640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89b814f800 0x7f89b814fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89b814f800 0x7f89b814fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 -- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c0073d0 msgr2=0x7f89b8152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c0073d0 0x7f89b8152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6000640 1 -- 172.21.15.6:0/4221267037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b809bb70 con 0x7f89b814ba00 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.245+0000 7f89b6a00640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c0073d0 0x7f89b8152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.246+0000 7f89b6000640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 0x7f89b814c3c0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a400e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.246+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4019070 con 0x7f89b814ba00 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.246+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a40052e0 con 0x7f89b814ba00 2024-03-06T07:35:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.246+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4003830 con 0x7f89b814ba00 2024-03-06T07:35:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.246+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b8153200 con 0x7f89b814ba00 2024-03-06T07:35:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.246+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b814be80 con 0x7f89b814ba00 2024-03-06T07:35:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.247+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b804f330 con 0x7f89b814ba00 2024-03-06T07:35:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.248+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89a401c070 con 0x7f89b814ba00 2024-03-06T07:35:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.249+0000 7f89aea00640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f899806b4d0 0x7f899806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.249+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f89a4089730 con 0x7f89b814ba00 2024-03-06T07:35:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.250+0000 7f89b6a00640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f899806b4d0 0x7f899806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.251+0000 7f89b6a00640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f899806b4d0 0x7f899806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f89a8005c30 tx=0x7f89a80017b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.255+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89a4057fa0 con 0x7f89b814ba00 2024-03-06T07:35:30.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:30.495+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f89b8072e70 con 0x7f89b814ba00 2024-03-06T07:35:31.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.038+0000 7f89aea00640 1 -- 172.21.15.6:0/4221267037 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7f89a405c040 con 0x7f89b814ba00 2024-03-06T07:35:31.040 INFO:teuthology.orchestra.run.smithi006.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:35:31.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.046+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f899806b4d0 msgr2=0x7f899806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.046+0000 7f89bd800640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f899806b4d0 0x7f899806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f89a8005c30 tx=0x7f89a80017b0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.046+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 msgr2=0x7f89b814c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.046+0000 7f89bd800640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 0x7f89b814c3c0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f89a4000c00 tx=0x7f89a400e8f0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.047+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 shutdown_connections 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.047+0000 7f89bd800640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f899806b4d0 0x7f899806d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.047+0000 7f89bd800640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89b814f800 0x7f89b814fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.047+0000 7f89bd800640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f89b814ba00 0x7f89b814c3c0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.047+0000 7f89bd800640 1 --2- 172.21.15.6:0/4221267037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f899c0073d0 0x7f89b8152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.047+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 >> 172.21.15.6:0/4221267037 conn(0x7f89b806b3d0 msgr2=0x7f89b80ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.049+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 shutdown_connections 2024-03-06T07:35:31.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.051+0000 7f89bd800640 1 -- 172.21.15.6:0/4221267037 wait complete. 2024-03-06T07:35:31.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:31.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.274+0000 7f28ce000640 1 Processor -- start 2024-03-06T07:35:31.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.274+0000 7f28ce000640 1 -- start start 2024-03-06T07:35:31.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28ce000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c815c920 0x7f28c8070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28ce000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28c80ce850 0x7f28c80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28ce000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c80cdad0 con 0x7f28c8070f80 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28ce000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c8071500 con 0x7f28c80ce850 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28ce000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80cdd70 con 0x7f28c815c920 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28c7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28c80ce850 0x7f28c80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28cd600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f28c8070f80 0x7f28c8165e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56984/0 (socket says 172.21.15.6:56984) 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.276+0000 7f28cd600640 1 -- 172.21.15.6:0/1032566666 learned_addr learned my addr 172.21.15.6:0/1032566666 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:31.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28ccc00640 1 --2- 172.21.15.6:0/1032566666 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c815c920 0x7f28c8070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28c7400640 1 -- 172.21.15.6:0/1032566666 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574141992 0 0) 0x7f28c80cdad0 con 0x7f28c8070f80 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28c7400640 1 -- 172.21.15.6:0/1032566666 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b0003520 con 0x7f28c8070f80 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28c7e00640 1 -- 172.21.15.6:0/1032566666 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c815c920 msgr2=0x7f28c8070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28c7e00640 1 --2- 172.21.15.6:0/1032566666 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c815c920 0x7f28c8070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28c7e00640 1 -- 172.21.15.6:0/1032566666 >> v1:172.21.15.6:6789/0 conn(0x7f28c8070f80 legacy=0x7f28c8165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.277+0000 7f28c7e00640 1 -- 172.21.15.6:0/1032566666 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80ce050 con 0x7f28c80ce850 2024-03-06T07:35:31.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.278+0000 7f28c7e00640 1 --2- 172.21.15.6:0/1032566666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c80ce850 0x7f28c80cc950 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f28b40097a0 tx=0x7f28b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e875263aa470bcfd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.278+0000 7f28c7400640 1 -- 172.21.15.6:0/1032566666 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4035020 con 0x7f28c80ce850 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.278+0000 7f28c7400640 1 -- 172.21.15.6:0/1032566666 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b4031830 con 0x7f28c80ce850 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.278+0000 7f28c7400640 1 -- 172.21.15.6:0/1032566666 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b4031c00 con 0x7f28c80ce850 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 -- 172.21.15.6:0/1032566666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c80ce850 msgr2=0x7f28c80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 --2- 172.21.15.6:0/1032566666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c80ce850 0x7f28c80cc950 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f28b40097a0 tx=0x7f28b402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 -- 172.21.15.6:0/1032566666 shutdown_connections 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 --2- 172.21.15.6:0/1032566666 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c80ce850 0x7f28c80cc950 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 --2- 172.21.15.6:0/1032566666 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c815c920 0x7f28c8070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 -- 172.21.15.6:0/1032566666 >> 172.21.15.6:0/1032566666 conn(0x7f28c806b3d0 msgr2=0x7f28c806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 -- 172.21.15.6:0/1032566666 shutdown_connections 2024-03-06T07:35:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.279+0000 7f28ce000640 1 -- 172.21.15.6:0/1032566666 wait complete. 2024-03-06T07:35:31.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.281+0000 7f28ce000640 1 Processor -- start 2024-03-06T07:35:31.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.281+0000 7f28ce000640 1 -- start start 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ce000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28c8070f80 0x7f28c81eccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ce000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c80ce850 0x7f28c81f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ce000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 0x7f28c80b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ce000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80cce90 con 0x7f28c815c920 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ce000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8163790 con 0x7f28c8070f80 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ce000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c80cd010 con 0x7f28c80ce850 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28cd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 0x7f28c80b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28cd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 0x7f28c80b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59242/0 (socket says 172.21.15.6:59242) 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28cd600640 1 -- 172.21.15.6:0/3854149130 learned_addr learned my addr 172.21.15.6:0/3854149130 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.282+0000 7f28ccc00640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28c8070f80 0x7f28c81eccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28c7e00640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c80ce850 0x7f28c81f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28cd600640 1 -- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c80ce850 msgr2=0x7f28c81f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28cd600640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c80ce850 0x7f28c81f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28cd600640 1 -- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28c8070f80 msgr2=0x7f28c81eccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28cd600640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28c8070f80 0x7f28c81eccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28cd600640 1 -- 172.21.15.6:0/3854149130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80b3d40 con 0x7f28c815c920 2024-03-06T07:35:31.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28c7e00640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c80ce850 0x7f28c81f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28cd600640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 0x7f28c80b3520 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f28c000cb10 tx=0x7f28c000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0017070 con 0x7f28c815c920 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c000ac90 con 0x7f28c815c920 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0012640 con 0x7f28c815c920 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.283+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c80b4020 con 0x7f28c815c920 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.284+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c80b3a60 con 0x7f28c815c920 2024-03-06T07:35:31.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.285+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c80b4540 con 0x7f28c815c920 2024-03-06T07:35:31.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.286+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28c001a070 con 0x7f28c815c920 2024-03-06T07:35:31.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.286+0000 7f28c4c00640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28a806b4d0 0x7f28a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.287+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28c0089fe0 con 0x7f28c815c920 2024-03-06T07:35:31.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.287+0000 7f28ccc00640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28a806b4d0 0x7f28a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.290 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.288+0000 7f28ccc00640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28a806b4d0 0x7f28a806d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f28bc006fd0 tx=0x7f28bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.291+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f28c80b4540 con 0x7f28c815c920 2024-03-06T07:35:31.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.490+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28c804f330 con 0x7f28c815c920 2024-03-06T07:35:31.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.492+0000 7f28c4c00640 1 -- 172.21.15.6:0/3854149130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f28c0058dc0 con 0x7f28c815c920 2024-03-06T07:35:31.495 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:31.495 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":125,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:35:30.497868+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:28.426977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:29.515406+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:31.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28a806b4d0 msgr2=0x7f28a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28a806b4d0 0x7f28a806d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f28bc006fd0 tx=0x7f28bc008040 comp rx=0 tx=0).stop 2024-03-06T07:35:31.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 msgr2=0x7f28c80b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 0x7f28c80b3520 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f28c000cb10 tx=0x7f28c000a060 comp rx=0 tx=0).stop 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 shutdown_connections 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f28a806b4d0 0x7f28a806d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28c815c920 0x7f28c80b3520 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28c80ce850 0x7f28c81f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 --2- 172.21.15.6:0/3854149130 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28c8070f80 0x7f28c81eccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 >> 172.21.15.6:0/3854149130 conn(0x7f28c806b3d0 msgr2=0x7f28c81667a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 shutdown_connections 2024-03-06T07:35:31.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.495+0000 7f28ce000640 1 -- 172.21.15.6:0/3854149130 wait complete. 2024-03-06T07:35:31.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.714+0000 7fbbf9600640 1 Processor -- start 2024-03-06T07:35:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.714+0000 7fbbf9600640 1 -- start start 2024-03-06T07:35:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbf9600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbbf4145300 0x7fbbf40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbf9600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbbf40cd060 0x7fbbf415c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbf9600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf40714d0 con 0x7fbbf4070f10 2024-03-06T07:35:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbf9600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf40717b0 con 0x7fbbf40cd060 2024-03-06T07:35:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbf9600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf4071a90 con 0x7fbbf4145300 2024-03-06T07:35:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbf8c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbbf40cd060 0x7fbbf415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbeb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbbf4070f10 0x7fbbf40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:56992/0 (socket says 172.21.15.6:56992) 2024-03-06T07:35:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.716+0000 7fbbeb400640 1 -- 172.21.15.6:0/3587380797 learned_addr learned my addr 172.21.15.6:0/3587380797 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf3e00640 1 --2- 172.21.15.6:0/3587380797 >> v2:172.21.15.178:3300/0 conn(0x7fbbf4145300 0x7fbbf40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3587380797 >> v2:172.21.15.178:3300/0 conn(0x7fbbf4145300 msgr2=0x7fbbf40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf8c00640 1 --2- 172.21.15.6:0/3587380797 >> v2:172.21.15.178:3300/0 conn(0x7fbbf4145300 0x7fbbf40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3587380797 >> v1:172.21.15.6:6789/0 conn(0x7fbbf4070f10 legacy=0x7fbbf40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3587380797 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf4071d70 con 0x7fbbf40cd060 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf3400640 1 -- 172.21.15.6:0/3587380797 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936562306 0 0) 0x7fbbf40714d0 con 0x7fbbf4070f10 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.717+0000 7fbbf3e00640 1 --2- 172.21.15.6:0/3587380797 >> v2:172.21.15.178:3300/0 conn(0x7fbbf4145300 0x7fbbf40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.718+0000 7fbbf8c00640 1 --2- 172.21.15.6:0/3587380797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf40cd060 0x7fbbf415c570 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40cf990 tx=0x7fbbec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06ae797f36d2036 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.718+0000 7fbbf3400640 1 -- 172.21.15.6:0/3587380797 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec0315a0 con 0x7fbbf40cd060 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.718+0000 7fbbf3400640 1 -- 172.21.15.6:0/3587380797 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbec037070 con 0x7fbbf40cd060 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.718+0000 7fbbf3400640 1 -- 172.21.15.6:0/3587380797 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec031960 con 0x7fbbf40cd060 2024-03-06T07:35:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.718+0000 7fbbf9600640 1 -- 172.21.15.6:0/3587380797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf40cd060 msgr2=0x7fbbf415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.718+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3587380797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf40cd060 0x7fbbf415c570 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40cf990 tx=0x7fbbec02b840 comp rx=0 tx=0).stop 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.719+0000 7fbbf9600640 1 -- 172.21.15.6:0/3587380797 shutdown_connections 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.719+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3587380797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf40cd060 0x7fbbf415c570 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.719+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3587380797 >> v2:172.21.15.178:3300/0 conn(0x7fbbf4145300 0x7fbbf40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.719+0000 7fbbf9600640 1 -- 172.21.15.6:0/3587380797 >> 172.21.15.6:0/3587380797 conn(0x7fbbf406b3d0 msgr2=0x7fbbf406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.719+0000 7fbbf9600640 1 -- 172.21.15.6:0/3587380797 shutdown_connections 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.719+0000 7fbbf9600640 1 -- 172.21.15.6:0/3587380797 wait complete. 2024-03-06T07:35:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.720+0000 7fbbf9600640 1 Processor -- start 2024-03-06T07:35:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.720+0000 7fbbf9600640 1 -- start start 2024-03-06T07:35:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.721+0000 7fbbf9600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf4070f10 0x7fbbf40b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.721+0000 7fbbf9600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf40cd060 0x7fbbf40caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.721+0000 7fbbf9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 0x7fbbf40cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.721+0000 7fbbf9600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf415cab0 con 0x7fbbf4145300 2024-03-06T07:35:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.721+0000 7fbbf9600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf41456e0 con 0x7fbbf4070f10 2024-03-06T07:35:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.721+0000 7fbbf9600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf415cc30 con 0x7fbbf40cd060 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 0x7fbbf40cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 0x7fbbf40cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:59260/0 (socket says 172.21.15.6:59260) 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3622040676 learned_addr learned my addr 172.21.15.6:0/3622040676 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf3e00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf4070f10 0x7fbbf40b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf40cd060 msgr2=0x7fbbf40caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbeb400640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf40cd060 0x7fbbf40caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf40cd060 0x7fbbf40caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf4070f10 msgr2=0x7fbbf40b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf4070f10 0x7fbbf40b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf8c00640 1 -- 172.21.15.6:0/3622040676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf40c9460 con 0x7fbbf4145300 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.722+0000 7fbbf3e00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf4070f10 0x7fbbf40b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.723+0000 7fbbf8c00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 0x7fbbf40cb3f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fbbec034040 tx=0x7fbbec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.723+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec032840 con 0x7fbbf4145300 2024-03-06T07:35:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.723+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbec0329e0 con 0x7fbbf4145300 2024-03-06T07:35:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.723+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbec032dd0 con 0x7fbbf4145300 2024-03-06T07:35:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.723+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf40c9740 con 0x7fbbf4145300 2024-03-06T07:35:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.723+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf40c9d00 con 0x7fbbf4145300 2024-03-06T07:35:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.724+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf404f330 con 0x7fbbf4145300 2024-03-06T07:35:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.725+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbec04c020 con 0x7fbbf4145300 2024-03-06T07:35:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.726+0000 7fbbf0c00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd006b4d0 0x7fbbd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.726+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbec02f020 con 0x7fbbf4145300 2024-03-06T07:35:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.726+0000 7fbbf3e00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd006b4d0 0x7fbbd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.727+0000 7fbbf3e00640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd006b4d0 0x7fbbd006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4007f40 tx=0x7fbbe4007490 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.730+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbec040070 con 0x7fbbf4145300 2024-03-06T07:35:31.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.931+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbf40c9110 con 0x7fbbf4145300 2024-03-06T07:35:31.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.934+0000 7fbbf0c00640 1 -- 172.21.15.6:0/3622040676 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fbbec07ad70 con 0x7fbbf4145300 2024-03-06T07:35:31.936 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:31.936 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":125,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:35:30.497868+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:28.426977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:29.515406+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.936+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd006b4d0 msgr2=0x7fbbd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.936+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd006b4d0 0x7fbbd006d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4007f40 tx=0x7fbbe4007490 comp rx=0 tx=0).stop 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.936+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 msgr2=0x7fbbf40cb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.936+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 0x7fbbf40cb3f0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fbbec034040 tx=0x7fbbec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 shutdown_connections 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbbd006b4d0 0x7fbbd006d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbbf4145300 0x7fbbf40cb3f0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbbf40cd060 0x7fbbf40caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 --2- 172.21.15.6:0/3622040676 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbbf4070f10 0x7fbbf40b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:31.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 >> 172.21.15.6:0/3622040676 conn(0x7fbbf406b3d0 msgr2=0x7fbbf40724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:31.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 shutdown_connections 2024-03-06T07:35:31.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:31.937+0000 7fbbf9600640 1 -- 172.21.15.6:0/3622040676 wait complete. 2024-03-06T07:35:31.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:35:32.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.182+0000 7fd821c00640 1 Processor -- start 2024-03-06T07:35:32.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.183+0000 7fd821c00640 1 -- start start 2024-03-06T07:35:32.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.184+0000 7fd821c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 0x7fd81c15f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.184+0000 7fd821c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c15c350 con 0x7fd81c13ed70 2024-03-06T07:35:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.184+0000 7fd821c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c06e6a0 con 0x7fd81c15ce10 2024-03-06T07:35:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.184+0000 7fd821c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c15d2a0 con 0x7fd81c0c75b0 2024-03-06T07:35:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.185+0000 7fd81aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd81c13ed70 0x7fd81c06dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42776/0 (socket says 172.21.15.6:42776) 2024-03-06T07:35:32.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.185+0000 7fd81aa00640 1 -- 172.21.15.6:0/4152152260 learned_addr learned my addr 172.21.15.6:0/4152152260 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.185+0000 7fd81b400640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 0x7fd81c15f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.185+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147757353 0 0) 0x7fd81c15c350 con 0x7fd81c13ed70 2024-03-06T07:35:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.185+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd800003520 con 0x7fd81c13ed70 2024-03-06T07:35:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1428743463 0 0) 0x7fd800003520 con 0x7fd81c13ed70 2024-03-06T07:35:32.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 msgr2=0x7fd81c15f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 0x7fd81c15f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 >> v1:172.21.15.178:6789/0 conn(0x7fd81c15ce10 legacy=0x7fd81c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c06e980 con 0x7fd81c13ed70 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd80c001670 con 0x7fd81c13ed70 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 >> v1:172.21.15.6:6789/0 conn(0x7fd81c13ed70 legacy=0x7fd81c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8000039e0 0x7fd800003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 0x7fd800004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd800004d40 0x7fd8000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80000bb50 con 0x7fd8000039e0 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80000bd00 con 0x7fd800004d40 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80000bf00 con 0x7fd81c0c75b0 2024-03-06T07:35:32.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81aa00640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8000039e0 0x7fd800003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd813e00640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 0x7fd800004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.186+0000 7fd81b400640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd800004d40 0x7fd8000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.187+0000 7fd81aa00640 1 -- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 msgr2=0x7fd800004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.187+0000 7fd81aa00640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd81c0c75b0 0x7fd800004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.187+0000 7fd81aa00640 1 -- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd800004d40 msgr2=0x7fd8000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.187+0000 7fd81aa00640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd800004d40 0x7fd8000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.187+0000 7fd81aa00640 1 -- 172.21.15.6:0/4152152260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800003520 con 0x7fd8000039e0 2024-03-06T07:35:32.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.187+0000 7fd81b400640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd800004d40 0x7fd8000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.188+0000 7fd81aa00640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8000039e0 0x7fd800003e00 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00c210 tx=0x7fd800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa15bffe4bb5437 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.188+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c007d30 con 0x7fd8000039e0 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.188+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80c030070 con 0x7fd8000039e0 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.188+0000 7fd81a000640 1 -- 172.21.15.6:0/4152152260 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c02cd60 con 0x7fd8000039e0 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.188+0000 7fd821c00640 1 -- 172.21.15.6:0/4152152260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8000039e0 msgr2=0x7fd800003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.188+0000 7fd821c00640 1 --2- 172.21.15.6:0/4152152260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd8000039e0 0x7fd800003e00 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00c210 tx=0x7fd800005830 comp rx=0 tx=0).stop 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.189+0000 7fd81b400640 1 -- 172.21.15.6:0/4152152260 reap_dead start 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.189+0000 7fd821c00640 1 -- 172.21.15.6:0/4152152260 shutdown_connections 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.189+0000 7fd821c00640 1 -- 172.21.15.6:0/4152152260 >> 172.21.15.6:0/4152152260 conn(0x7fd81c0b2760 msgr2=0x7fd81c0b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.189+0000 7fd821c00640 1 -- 172.21.15.6:0/4152152260 shutdown_connections 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.189+0000 7fd821c00640 1 -- 172.21.15.6:0/4152152260 wait complete. 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.191+0000 7fd821c00640 1 Processor -- start 2024-03-06T07:35:32.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.191+0000 7fd821c00640 1 -- start start 2024-03-06T07:35:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd821c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8000039e0 0x7fd81c15ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd821c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 0x7fd81c0b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd821c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd81c0b5d00 0x7fd81c0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd821c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c06e360 con 0x7fd81c15b360 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd821c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c15c980 con 0x7fd81c0b5d00 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd821c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c06f1f0 con 0x7fd8000039e0 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd813e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 0x7fd81c0b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd813e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 0x7fd81c0b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39378/0 (socket says 172.21.15.6:39378) 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd813e00640 1 -- 172.21.15.6:0/3181901044 learned_addr learned my addr 172.21.15.6:0/3181901044 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd81b400640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd81c0b5d00 0x7fd81c0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.192+0000 7fd81aa00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8000039e0 0x7fd81c15ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd813e00640 1 -- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8000039e0 msgr2=0x7fd81c15ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd813e00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8000039e0 0x7fd81c15ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd813e00640 1 -- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd81c0b5d00 msgr2=0x7fd81c0b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd813e00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd81c0b5d00 0x7fd81c0b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd813e00640 1 -- 172.21.15.6:0/3181901044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c13db50 con 0x7fd81c15b360 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd81b400640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd81c0b5d00 0x7fd81c0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:32.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd813e00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 0x7fd81c0b57c0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e580 tx=0x7fd80400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8040052e0 con 0x7fd81c15b360 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd804018070 con 0x7fd81c15b360 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804003810 con 0x7fd81c15b360 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c13c340 con 0x7fd81c15b360 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.193+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c13c900 con 0x7fd81c15b360 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.195+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c04f330 con 0x7fd81c15b360 2024-03-06T07:35:32.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.196+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8040039b0 con 0x7fd81c15b360 2024-03-06T07:35:32.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.196+0000 7fd812a00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd7f806b4d0 0x7fd7f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:32.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.197+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd804089eb0 con 0x7fd81c15b360 2024-03-06T07:35:32.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.197+0000 7fd81aa00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd7f806b4d0 0x7fd7f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:32.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.198+0000 7fd81aa00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd7f806b4d0 0x7fd7f806d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd80c034000 tx=0x7fd80c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.201+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd804058d40 con 0x7fd81c15b360 2024-03-06T07:35:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:32.441+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd81c13e0d0 con 0x7fd81c15b360 2024-03-06T07:35:33.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.081+0000 7fd812a00640 1 -- 172.21.15.6:0/3181901044 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd804058640 con 0x7fd81c15b360 2024-03-06T07:35:33.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.084+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd7f806b4d0 msgr2=0x7fd7f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.084+0000 7fd821c00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd7f806b4d0 0x7fd7f806d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd80c034000 tx=0x7fd80c009030 comp rx=0 tx=0).stop 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.084+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 msgr2=0x7fd81c0b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.084+0000 7fd821c00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 0x7fd81c0b57c0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e580 tx=0x7fd80400ea40 comp rx=0 tx=0).stop 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 shutdown_connections 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd7f806b4d0 0x7fd7f806d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd81c0b5d00 0x7fd81c0b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd81c15b360 0x7fd81c0b57c0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 --2- 172.21.15.6:0/3181901044 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8000039e0 0x7fd81c15ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 >> 172.21.15.6:0/3181901044 conn(0x7fd81c0b2760 msgr2=0x7fd81c0cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 shutdown_connections 2024-03-06T07:35:33.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.085+0000 7fd821c00640 1 -- 172.21.15.6:0/3181901044 wait complete. 2024-03-06T07:35:33.107 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:35:33.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:35:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.331+0000 7ff378e00640 1 Processor -- start 2024-03-06T07:35:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.331+0000 7ff378e00640 1 -- start start 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.332+0000 7ff378e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3740d07d0 0x7ff3740b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.332+0000 7ff378e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff37415e6c0 0x7ff3740b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.332+0000 7ff378e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3740cf780 con 0x7ff3740d4930 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.332+0000 7ff378e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740b3770 con 0x7ff3740d07d0 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.332+0000 7ff378e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740cfa60 con 0x7ff37415e6c0 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3740d07d0 0x7ff3740b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3740d07d0 0x7ff3740b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39382/0 (socket says 172.21.15.6:39382) 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 -- 172.21.15.6:0/1755155254 learned_addr learned my addr 172.21.15.6:0/1755155254 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff372800640 1 --2- 172.21.15.6:0/1755155254 >> v2:172.21.15.178:3300/0 conn(0x7ff37415e6c0 0x7ff3740b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 -- 172.21.15.6:0/1755155254 >> v2:172.21.15.178:3300/0 conn(0x7ff37415e6c0 msgr2=0x7ff3740b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 --2- 172.21.15.6:0/1755155254 >> v2:172.21.15.178:3300/0 conn(0x7ff37415e6c0 0x7ff3740b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 -- 172.21.15.6:0/1755155254 >> v1:172.21.15.6:6789/0 conn(0x7ff3740d4930 legacy=0x7ff37415dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff371e00640 1 -- 172.21.15.6:0/1755155254 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740cfd40 con 0x7ff3740d07d0 2024-03-06T07:35:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.333+0000 7ff372800640 1 --2- 172.21.15.6:0/1755155254 >> v2:172.21.15.178:3300/0 conn(0x7ff37415e6c0 0x7ff3740b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.335+0000 7ff371e00640 1 --2- 172.21.15.6:0/1755155254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3740b4f80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff374067cb0 tx=0x7ff36402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=923a0c4c79151526 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.335+0000 7ff370a00640 1 -- 172.21.15.6:0/1755155254 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36402f850 con 0x7ff3740d07d0 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.335+0000 7ff370a00640 1 -- 172.21.15.6:0/1755155254 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36402f9f0 con 0x7ff3740d07d0 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.335+0000 7ff370a00640 1 -- 172.21.15.6:0/1755155254 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36402fdc0 con 0x7ff3740d07d0 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 -- 172.21.15.6:0/1755155254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 msgr2=0x7ff3740b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 --2- 172.21.15.6:0/1755155254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3740b4f80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff374067cb0 tx=0x7ff36402ba30 comp rx=0 tx=0).stop 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 -- 172.21.15.6:0/1755155254 shutdown_connections 2024-03-06T07:35:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 --2- 172.21.15.6:0/1755155254 >> v2:172.21.15.178:3300/0 conn(0x7ff37415e6c0 0x7ff3740b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 --2- 172.21.15.6:0/1755155254 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3740b4f80 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 -- 172.21.15.6:0/1755155254 >> 172.21.15.6:0/1755155254 conn(0x7ff3740d3e10 msgr2=0x7ff3740d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 -- 172.21.15.6:0/1755155254 shutdown_connections 2024-03-06T07:35:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.336+0000 7ff378e00640 1 -- 172.21.15.6:0/1755155254 wait complete. 2024-03-06T07:35:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.338+0000 7ff378e00640 1 Processor -- start 2024-03-06T07:35:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.338+0000 7ff378e00640 1 -- start start 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff378e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3741612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff378e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3740d4930 0x7ff3740b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff378e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff37415e6c0 0x7ff3741f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff378e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740d0d30 con 0x7ff3740d07d0 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff378e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740d0bb0 con 0x7ff3740d4930 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff378e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740d0eb0 con 0x7ff37415e6c0 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff371e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3741612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff371e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3741612f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39392/0 (socket says 172.21.15.6:39392) 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.339+0000 7ff371e00640 1 -- 172.21.15.6:0/3664532138 learned_addr learned my addr 172.21.15.6:0/3664532138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff371e00640 1 -- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff37415e6c0 msgr2=0x7ff3741f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff371e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff37415e6c0 0x7ff3741f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff371e00640 1 -- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3740d4930 msgr2=0x7ff3740b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff371e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3740d4930 0x7ff3740b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff371e00640 1 -- 172.21.15.6:0/3664532138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740b9690 con 0x7ff3740d07d0 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff371e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3741612f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff36402ba70 tx=0x7ff364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff364041400 con 0x7ff3740d07d0 2024-03-06T07:35:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff364046070 con 0x7ff3740d07d0 2024-03-06T07:35:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3741f4520 con 0x7ff3740d07d0 2024-03-06T07:35:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.340+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3640417e0 con 0x7ff3740d07d0 2024-03-06T07:35:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.341+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3741f4ae0 con 0x7ff3740d07d0 2024-03-06T07:35:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.342+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37404f330 con 0x7ff3740d07d0 2024-03-06T07:35:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.344+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff364040070 con 0x7ff3740d07d0 2024-03-06T07:35:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.345+0000 7ff36a000640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff35006b4d0 0x7ff35006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.345+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3640acc00 con 0x7ff3740d07d0 2024-03-06T07:35:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.345+0000 7ff371400640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff35006b4d0 0x7ff35006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.347+0000 7ff371400640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff35006b4d0 0x7ff35006d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff35c000f60 tx=0x7ff35c0079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.349+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff364033030 con 0x7ff3740d07d0 2024-03-06T07:35:33.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:33.594+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff3740d4320 con 0x7ff3740d07d0 2024-03-06T07:35:34.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.094+0000 7ff36a000640 1 -- 172.21.15.6:0/3664532138 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v92) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff36407b9e0 con 0x7ff3740d07d0 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff35006b4d0 msgr2=0x7ff35006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff35006b4d0 0x7ff35006d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff35c000f60 tx=0x7ff35c0079d0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 msgr2=0x7ff3741612f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3741612f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff36402ba70 tx=0x7ff364004730 comp rx=0 tx=0).stop 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 shutdown_connections 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff35006b4d0 0x7ff35006d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3740d4930 0x7ff3740b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff37415e6c0 0x7ff3741f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 --2- 172.21.15.6:0/3664532138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3740d07d0 0x7ff3741612f0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.099+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 >> 172.21.15.6:0/3664532138 conn(0x7ff3740d3e10 msgr2=0x7ff3740d19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.100+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 shutdown_connections 2024-03-06T07:35:34.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.100+0000 7ff378e00640 1 -- 172.21.15.6:0/3664532138 wait complete. 2024-03-06T07:35:34.125 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:35:34.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:34.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.334+0000 7f66f1e00640 1 Processor -- start 2024-03-06T07:35:34.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.335+0000 7f66f1e00640 1 -- start start 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.336+0000 7f66f1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f66ec070f10 0x7f66ec0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.336+0000 7f66f1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66ec0cce90 0x7f66ec15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.336+0000 7f66f1e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec0714d0 con 0x7f66ec145300 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.336+0000 7f66f1e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec0cd300 con 0x7f66ec0cce90 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.336+0000 7f66f1e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec0717b0 con 0x7f66ec070f10 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.337+0000 7f66f1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66ec0cce90 0x7f66ec15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.337+0000 7f66f1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66ec0cce90 0x7f66ec15c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39396/0 (socket says 172.21.15.6:39396) 2024-03-06T07:35:34.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.337+0000 7f66f1400640 1 -- 172.21.15.6:0/1841021264 learned_addr learned my addr 172.21.15.6:0/1841021264 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:34.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.337+0000 7f66ebe00640 1 --2- 172.21.15.6:0/1841021264 >> v2:172.21.15.178:3300/0 conn(0x7f66ec070f10 0x7f66ec0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.337+0000 7f66f1400640 1 -- 172.21.15.6:0/1841021264 >> v2:172.21.15.178:3300/0 conn(0x7f66ec070f10 msgr2=0x7f66ec0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.337+0000 7f66f1400640 1 --2- 172.21.15.6:0/1841021264 >> v2:172.21.15.178:3300/0 conn(0x7f66ec070f10 0x7f66ec0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.338+0000 7f66f1400640 1 -- 172.21.15.6:0/1841021264 >> v1:172.21.15.178:6789/0 conn(0x7f66ec145300 legacy=0x7f66ec0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.338+0000 7f66f1400640 1 -- 172.21.15.6:0/1841021264 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec071a90 con 0x7f66ec0cce90 2024-03-06T07:35:34.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.339+0000 7f66f1400640 1 --2- 172.21.15.6:0/1841021264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec0cce90 0x7f66ec15c570 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0b5150 tx=0x7f66e402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f7644c7bd7e5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.339+0000 7f66eb400640 1 -- 172.21.15.6:0/1841021264 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e402ca80 con 0x7f66ec0cce90 2024-03-06T07:35:34.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.339+0000 7f66eb400640 1 -- 172.21.15.6:0/1841021264 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e402cc20 con 0x7f66ec0cce90 2024-03-06T07:35:34.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.339+0000 7f66eb400640 1 -- 172.21.15.6:0/1841021264 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40343f0 con 0x7f66ec0cce90 2024-03-06T07:35:34.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 -- 172.21.15.6:0/1841021264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec0cce90 msgr2=0x7f66ec15c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1841021264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec0cce90 0x7f66ec15c570 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0b5150 tx=0x7f66e402bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 -- 172.21.15.6:0/1841021264 shutdown_connections 2024-03-06T07:35:34.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1841021264 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec0cce90 0x7f66ec15c570 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1841021264 >> v2:172.21.15.178:3300/0 conn(0x7f66ec070f10 0x7f66ec0cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 -- 172.21.15.6:0/1841021264 >> 172.21.15.6:0/1841021264 conn(0x7f66ec06b3d0 msgr2=0x7f66ec06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 -- 172.21.15.6:0/1841021264 shutdown_connections 2024-03-06T07:35:34.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.340+0000 7f66f1e00640 1 -- 172.21.15.6:0/1841021264 wait complete. 2024-03-06T07:35:34.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.342+0000 7f66f1e00640 1 Processor -- start 2024-03-06T07:35:34.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.342+0000 7f66f1e00640 1 -- start start 2024-03-06T07:35:34.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 0x7f66ec13ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66ec0cce90 0x7f66ec15ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66ec145300 0x7f66ec1e6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec15cab0 con 0x7f66ec070f10 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec1456e0 con 0x7f66ec145300 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec15cc30 con 0x7f66ec0cce90 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 0x7f66ec13ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 0x7f66ec13ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39408/0 (socket says 172.21.15.6:39408) 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f0a00640 1 -- 172.21.15.6:0/1662480523 learned_addr learned my addr 172.21.15.6:0/1662480523 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66ebe00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66ec0cce90 0x7f66ec15ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f0a00640 1 -- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66ec0cce90 msgr2=0x7f66ec15ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f0a00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66ec0cce90 0x7f66ec15ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f1400640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66ec145300 0x7f66ec1e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.343+0000 7f66f0a00640 1 -- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66ec145300 msgr2=0x7f66ec1e6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66f0a00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66ec145300 0x7f66ec1e6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66f0a00640 1 -- 172.21.15.6:0/1662480523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec13f250 con 0x7f66ec070f10 2024-03-06T07:35:34.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66ebe00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66ec0cce90 0x7f66ec15ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66f0a00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 0x7f66ec13ed10 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c400 tx=0x7f66e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0017070 con 0x7f66ec070f10 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00052d0 con 0x7f66ec070f10 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0003830 con 0x7f66ec070f10 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec13d130 con 0x7f66ec070f10 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.344+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec13d6f0 con 0x7f66ec070f10 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.346+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec04f330 con 0x7f66ec070f10 2024-03-06T07:35:34.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.346+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66e001a070 con 0x7f66ec070f10 2024-03-06T07:35:34.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.347+0000 7f66e8c00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66cc06b4d0 0x7f66cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.347+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66e00895e0 con 0x7f66ec070f10 2024-03-06T07:35:34.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.348+0000 7f66ebe00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66cc06b4d0 0x7f66cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.349+0000 7f66ebe00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66cc06b4d0 0x7f66cc06d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0af510 tx=0x7f66d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.352+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66e00583c0 con 0x7f66ec070f10 2024-03-06T07:35:34.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.631+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66ec15be10 con 0x7f66ec070f10 2024-03-06T07:35:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.633+0000 7f66e8c00640 1 -- 172.21.15.6:0/1662480523 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f66e0057cc0 con 0x7f66ec070f10 2024-03-06T07:35:34.635 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:34.635 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":92,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1518219931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1518219931},{"type":"v1","addr":"172.21.15.178:6837","nonce":1518219931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/1688442844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1688442844},{"type":"v1","addr":"172.21.15.178:6836","nonce":1688442844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6871,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:33.596622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:35:34.635 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 92 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.635+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66cc06b4d0 msgr2=0x7f66cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.635+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66cc06b4d0 0x7f66cc06d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0af510 tx=0x7f66d8008040 comp rx=0 tx=0).stop 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.635+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 msgr2=0x7f66ec13ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.635+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 0x7f66ec13ed10 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c400 tx=0x7f66e000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.635+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 shutdown_connections 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.635+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66cc06b4d0 0x7f66cc06d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.636+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66ec145300 0x7f66ec1e6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.636+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66ec0cce90 0x7f66ec15ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.636+0000 7f66f1e00640 1 --2- 172.21.15.6:0/1662480523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66ec070f10 0x7f66ec13ed10 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.636+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 >> 172.21.15.6:0/1662480523 conn(0x7f66ec06b3d0 msgr2=0x7f66ec15daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.636+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 shutdown_connections 2024-03-06T07:35:34.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.636+0000 7f66f1e00640 1 -- 172.21.15.6:0/1662480523 wait complete. 2024-03-06T07:35:34.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:35:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.871+0000 7ffaf0c00640 1 Processor -- start 2024-03-06T07:35:34.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.871+0000 7ffaf0c00640 1 -- start start 2024-03-06T07:35:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.873+0000 7ffaf0c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ffaec0b34e0 0x7ffaec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.873+0000 7ffaf0c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec15cc40 con 0x7ffaec0af580 2024-03-06T07:35:34.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.873+0000 7ffaf0c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec15cf20 con 0x7ffaec0af1a0 2024-03-06T07:35:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.873+0000 7ffaf0c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec15d200 con 0x7ffaec0b34e0 2024-03-06T07:35:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.873+0000 7ffae9400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffaec0af580 0x7ffaec0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42798/0 (socket says 172.21.15.6:42798) 2024-03-06T07:35:34.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.873+0000 7ffae9400640 1 -- 172.21.15.6:0/1430188963 learned_addr learned my addr 172.21.15.6:0/1430188963 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:34.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147019894 0 0) 0x7ffaec15cc40 con 0x7ffaec0af580 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0003520 con 0x7ffaec0af580 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffaea800640 1 --2- 172.21.15.6:0/1430188963 >> v2:172.21.15.178:3300/0 conn(0x7ffaec0b34e0 0x7ffaec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4069534742 0 0) 0x7ffad0003520 con 0x7ffaec0af580 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 >> v2:172.21.15.178:3300/0 conn(0x7ffaec0b34e0 msgr2=0x7ffaec15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 --2- 172.21.15.6:0/1430188963 >> v2:172.21.15.178:3300/0 conn(0x7ffaec0b34e0 0x7ffaec15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 >> v1:172.21.15.178:6789/0 conn(0x7ffaec0af1a0 legacy=0x7ffaec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec15d4e0 con 0x7ffaec0af580 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ffad4001670 con 0x7ffaec0af580 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.874+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 >> v1:172.21.15.6:6789/0 conn(0x7ffaec0af580 legacy=0x7ffaec0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae8a00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffad0003a70 0x7ffad0003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae8a00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffad0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae8a00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffad00052e0 0x7ffad000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15d200 con 0x7ffad0004800 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000c1b0 con 0x7ffad0003a70 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000c3b0 con 0x7ffad00052e0 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9400640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffad0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9e00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffad0003a70 0x7ffad0003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9400640 1 -- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffad00052e0 msgr2=0x7ffad000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffaea800640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffad00052e0 0x7ffad000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9400640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffad00052e0 0x7ffad000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9400640 1 -- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffad0003a70 msgr2=0x7ffad0003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9400640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffad0003a70 0x7ffad0003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9400640 1 -- 172.21.15.6:0/1430188963 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0003520 con 0x7ffad0004800 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffaea800640 1 -- 172.21.15.6:0/1430188963 reap_dead start 2024-03-06T07:35:34.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.875+0000 7ffae9e00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffad0003a70 0x7ffad0003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.877+0000 7ffae9400640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffad0004c80 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7ffad400c160 tx=0x7ffad0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae3f8576dfaed07 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.877+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4007c60 con 0x7ffad0004800 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.877+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4030070 con 0x7ffad0004800 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.877+0000 7ffae8a00640 1 -- 172.21.15.6:0/1430188963 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4005db0 con 0x7ffad0004800 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1430188963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 msgr2=0x7ffad0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffad0004c80 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7ffad400c160 tx=0x7ffad0005830 comp rx=0 tx=0).stop 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1430188963 shutdown_connections 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1430188963 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffad0004c80 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1430188963 >> 172.21.15.6:0/1430188963 conn(0x7ffaec06afb0 msgr2=0x7ffaec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1430188963 shutdown_connections 2024-03-06T07:35:34.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.878+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1430188963 wait complete. 2024-03-06T07:35:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.880+0000 7ffaf0c00640 1 Processor -- start 2024-03-06T07:35:34.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.880+0000 7ffaf0c00640 1 -- start start 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaf0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffaec0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaf0c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaec09c870 0x7ffaec09ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaf0c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaec09ac50 0x7ffaec09b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaf0c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15c600 con 0x7ffad0004800 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaf0c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0cde30 con 0x7ffaec09ac50 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaf0c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec15c780 con 0x7ffaec09c870 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffae9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffaec0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffae9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffaec0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39420/0 (socket says 172.21.15.6:39420) 2024-03-06T07:35:34.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffae9e00640 1 -- 172.21.15.6:0/1403946473 learned_addr learned my addr 172.21.15.6:0/1403946473 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffae9400640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaec09c870 0x7ffaec09ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.881+0000 7ffaea800640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaec09ac50 0x7ffaec09b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae9e00640 1 -- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaec09c870 msgr2=0x7ffaec09ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae9e00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaec09c870 0x7ffaec09ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae9e00640 1 -- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaec09ac50 msgr2=0x7ffaec09b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae9e00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaec09ac50 0x7ffaec09b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae9e00640 1 -- 172.21.15.6:0/1403946473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec06cf40 con 0x7ffad0004800 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae9e00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffaec0ca410 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00d460 tx=0x7ffadc00d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc005480 con 0x7ffad0004800 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc017070 con 0x7ffad0004800 2024-03-06T07:35:34.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc003870 con 0x7ffad0004800 2024-03-06T07:35:34.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec06d220 con 0x7ffad0004800 2024-03-06T07:35:34.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.882+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec06d7e0 con 0x7ffad0004800 2024-03-06T07:35:34.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.884+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec069510 con 0x7ffad0004800 2024-03-06T07:35:34.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.884+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffadc003a10 con 0x7ffad0004800 2024-03-06T07:35:34.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.885+0000 7ffae2000640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffac806b500 0x7ffac806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:34.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.885+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffadc089e50 con 0x7ffad0004800 2024-03-06T07:35:34.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.886+0000 7ffae9400640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffac806b500 0x7ffac806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:34.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.887+0000 7ffae9400640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffac806b500 0x7ffac806d9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0afa10 tx=0x7ffad4009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:34.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:34.890+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffadc058cd0 con 0x7ffad0004800 2024-03-06T07:35:35.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.091+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaec04f3a0 con 0x7ffad0004800 2024-03-06T07:35:35.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.092+0000 7ffae2000640 1 -- 172.21.15.6:0/1403946473 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16607 (secure 0 0 0) 0x7ffadc0585d0 con 0x7ffad0004800 2024-03-06T07:35:35.095 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:35.095 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":126,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:35:32.054872+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:28.426977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:29.515406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:35:35.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffac806b500 msgr2=0x7ffac806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffac806b500 0x7ffac806d9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0afa10 tx=0x7ffad4009030 comp rx=0 tx=0).stop 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 msgr2=0x7ffaec0ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffaec0ca410 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00d460 tx=0x7ffadc00d920 comp rx=0 tx=0).stop 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 shutdown_connections 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffac806b500 0x7ffac806d9c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffaec09ac50 0x7ffaec09b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffaec09c870 0x7ffaec09ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 --2- 172.21.15.6:0/1403946473 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffad0004800 0x7ffaec0ca410 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 >> 172.21.15.6:0/1403946473 conn(0x7ffaec06afb0 msgr2=0x7ffaec0cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 shutdown_connections 2024-03-06T07:35:35.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.095+0000 7ffaf0c00640 1 -- 172.21.15.6:0/1403946473 wait complete. 2024-03-06T07:35:35.118 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:35:35.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:35.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.327+0000 7fe219800640 1 Processor -- start 2024-03-06T07:35:35.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.327+0000 7fe219800640 1 -- start start 2024-03-06T07:35:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe219800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 0x7fe2140d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe219800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140b1370 con 0x7fe2140d05e0 2024-03-06T07:35:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe219800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2140ccf80 con 0x7fe2140b0f90 2024-03-06T07:35:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe219800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140d0a50 con 0x7fe2140cf080 2024-03-06T07:35:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe218e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2140d05e0 0x7fe21415d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42802/0 (socket says 172.21.15.6:42802) 2024-03-06T07:35:35.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe218e00640 1 -- 172.21.15.6:0/393145536 learned_addr learned my addr 172.21.15.6:0/393145536 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.329+0000 7fe213400640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 0x7fe2140d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678545538 0 0) 0x7fe2140b1370 con 0x7fe2140d05e0 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8003520 con 0x7fe2140d05e0 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348844689 0 0) 0x7fe1f8003520 con 0x7fe2140d05e0 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 msgr2=0x7fe2140d00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 0x7fe2140d00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 >> v1:172.21.15.178:6789/0 conn(0x7fe2140b0f90 legacy=0x7fe2140cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2140cd260 con 0x7fe2140d05e0 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fe20c001670 con 0x7fe2140d05e0 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 >> v1:172.21.15.6:6789/0 conn(0x7fe2140d05e0 legacy=0x7fe21415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe1f8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe1f8004390 0x7fe1f8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 0x7fe1f8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140d0a50 con 0x7fe1f80039e0 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f800ba70 con 0x7fe1f8004390 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.330+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f800bc70 con 0x7fe2140cf080 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.331+0000 7fe213e00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe1f8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.331+0000 7fe213400640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe1f8004390 0x7fe1f8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.331+0000 7fe218e00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 0x7fe1f8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.331+0000 7fe213e00640 1 -- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 msgr2=0x7fe1f8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.331+0000 7fe213e00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140cf080 0x7fe1f8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.332+0000 7fe213e00640 1 -- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe1f8004390 msgr2=0x7fe1f8004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.332+0000 7fe213e00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe1f8004390 0x7fe1f8004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.332+0000 7fe213e00640 1 -- 172.21.15.6:0/393145536 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8003520 con 0x7fe1f80039e0 2024-03-06T07:35:35.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.333+0000 7fe213e00640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe1f8003dc0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc001860 tx=0x7fe1f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e01c0e3525ff6f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.333+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc02f050 con 0x7fe1f80039e0 2024-03-06T07:35:35.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.333+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc033650 con 0x7fe1f80039e0 2024-03-06T07:35:35.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.333+0000 7fe212a00640 1 -- 172.21.15.6:0/393145536 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc001430 con 0x7fe1f80039e0 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe219800640 1 -- 172.21.15.6:0/393145536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 msgr2=0x7fe1f8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe219800640 1 --2- 172.21.15.6:0/393145536 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe1f8003dc0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc001860 tx=0x7fe1f8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe218e00640 1 -- 172.21.15.6:0/393145536 reap_dead start 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe219800640 1 -- 172.21.15.6:0/393145536 shutdown_connections 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe219800640 1 -- 172.21.15.6:0/393145536 >> 172.21.15.6:0/393145536 conn(0x7fe2140d1ae0 msgr2=0x7fe2140d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe219800640 1 -- 172.21.15.6:0/393145536 shutdown_connections 2024-03-06T07:35:35.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.334+0000 7fe219800640 1 -- 172.21.15.6:0/393145536 wait complete. 2024-03-06T07:35:35.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.336+0000 7fe219800640 1 Processor -- start 2024-03-06T07:35:35.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.336+0000 7fe219800640 1 -- start start 2024-03-06T07:35:35.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe219800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe21409de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe219800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe21409ba30 0x7fe2140c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe219800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140af960 0x7fe2140afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe219800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21406ffe0 con 0x7fe1f80039e0 2024-03-06T07:35:35.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe219800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2140cd820 con 0x7fe21409ba30 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe219800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe214070160 con 0x7fe2140af960 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe213e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe21409de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe213e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe21409de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39432/0 (socket says 172.21.15.6:39432) 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.337+0000 7fe213e00640 1 -- 172.21.15.6:0/4005458815 learned_addr learned my addr 172.21.15.6:0/4005458815 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213400640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe21409ba30 0x7fe2140c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe218e00640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140af960 0x7fe2140afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213e00640 1 -- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140af960 msgr2=0x7fe2140afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213e00640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140af960 0x7fe2140afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213e00640 1 -- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe21409ba30 msgr2=0x7fe2140c7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213e00640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe21409ba30 0x7fe2140c7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213e00640 1 -- 172.21.15.6:0/4005458815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21409bf30 con 0x7fe1f80039e0 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213400640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe21409ba30 0x7fe2140c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:35.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.338+0000 7fe213e00640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe21409de20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc02bdf0 tx=0x7fe1fc001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.339+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc042070 con 0x7fe1f80039e0 2024-03-06T07:35:35.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.339+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc0035f0 con 0x7fe1f80039e0 2024-03-06T07:35:35.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.339+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21409c210 con 0x7fe1f80039e0 2024-03-06T07:35:35.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.339+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc001430 con 0x7fe1f80039e0 2024-03-06T07:35:35.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.339+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21409c7d0 con 0x7fe1f80039e0 2024-03-06T07:35:35.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.340+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21404f330 con 0x7fe1f80039e0 2024-03-06T07:35:35.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.341+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1fc03b040 con 0x7fe1f80039e0 2024-03-06T07:35:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.342+0000 7fe20be00640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe1f006b4d0 0x7fe1f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.342+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1fc0ac970 con 0x7fe1f80039e0 2024-03-06T07:35:35.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.343+0000 7fe213400640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe1f006b4d0 0x7fe1f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.344+0000 7fe213400640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe1f006b4d0 0x7fe1f006d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe2140b0930 tx=0x7fe20400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.347+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1fc04c060 con 0x7fe1f80039e0 2024-03-06T07:35:35.436 INFO:tasks.ceph.mds.a.smithi006.stderr:2024-03-06T07:35:35.434+0000 7fd9c2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:35:35.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.589+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2140b0330 con 0x7fe1f006b4d0 2024-03-06T07:35:35.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.609+0000 7fe20be00640 1 -- 172.21.15.6:0/4005458815 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252422 (secure 0 0 0) 0x7fe2140b0330 con 0x7fe1f006b4d0 2024-03-06T07:35:35.615 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:35.616 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe1f006b4d0 msgr2=0x7fe1f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe1f006b4d0 0x7fe1f006d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe2140b0930 tx=0x7fe20400b040 comp rx=0 tx=0).stop 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 msgr2=0x7fe21409de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe21409de20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc02bdf0 tx=0x7fe1fc001230 comp rx=0 tx=0).stop 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 shutdown_connections 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe1f006b4d0 0x7fe1f006d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe21409ba30 0x7fe2140c7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe2140af960 0x7fe2140afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 --2- 172.21.15.6:0/4005458815 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe1f80039e0 0x7fe21409de20 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 >> 172.21.15.6:0/4005458815 conn(0x7fe2140d1ae0 msgr2=0x7fe21406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 shutdown_connections 2024-03-06T07:35:35.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.614+0000 7fe219800640 1 -- 172.21.15.6:0/4005458815 wait complete. 2024-03-06T07:35:35.635 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-03-06T07:35:33.617273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312700,"kb_used_data":13296,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754662020,"statfs":{"total":773094113280,"available":772773908480,"internally_reserved":0,"allocated":13615104,"data_stored":6546995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.612934"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.067897+0000","last_change":"2024-03-06T07:35:31.087859+0000","last_active":"2024-03-06T07:35:32.067897+0000","last_peered":"2024-03-06T07:35:32.067897+0000","last_clean":"2024-03-06T07:35:32.067897+0000","last_became_active":"2024-03-06T07:35:31.087497+0000","last_became_peered":"2024-03-06T07:35:31.087497+0000","last_unstale":"2024-03-06T07:35:32.067897+0000","last_undegraded":"2024-03-06T07:35:32.067897+0000","last_fullsized":"2024-03-06T07:35:32.067897+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:45.000491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086867+0000","last_change":"2024-03-06T07:35:31.085956+0000","last_active":"2024-03-06T07:35:31.086867+0000","last_peered":"2024-03-06T07:35:31.086867+0000","last_clean":"2024-03-06T07:35:31.086867+0000","last_became_active":"2024-03-06T07:35:31.085486+0000","last_became_peered":"2024-03-06T07:35:31.085486+0000","last_unstale":"2024-03-06T07:35:31.086867+0000","last_undegraded":"2024-03-06T07:35:31.086867+0000","last_fullsized":"2024-03-06T07:35:31.086867+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:07.280698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.070539+0000","last_change":"2024-03-06T07:35:31.064577+0000","last_active":"2024-03-06T07:35:31.070539+0000","last_peered":"2024-03-06T07:35:31.070539+0000","last_clean":"2024-03-06T07:35:31.070539+0000","last_became_active":"2024-03-06T07:35:31.064094+0000","last_became_peered":"2024-03-06T07:35:31.064094+0000","last_unstale":"2024-03-06T07:35:31.070539+0000","last_undegraded":"2024-03-06T07:35:31.070539+0000","last_fullsized":"2024-03-06T07:35:31.070539+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:58.512115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065495+0000","last_change":"2024-03-06T07:35:31.087422+0000","last_active":"2024-03-06T07:35:32.065495+0000","last_peered":"2024-03-06T07:35:32.065495+0000","last_clean":"2024-03-06T07:35:32.065495+0000","last_became_active":"2024-03-06T07:35:31.080944+0000","last_became_peered":"2024-03-06T07:35:31.080944+0000","last_unstale":"2024-03-06T07:35:32.065495+0000","last_undegraded":"2024-03-06T07:35:32.065495+0000","last_fullsized":"2024-03-06T07:35:32.065495+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:27.541279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092669+0000","last_change":"2024-03-06T07:35:31.087296+0000","last_active":"2024-03-06T07:35:31.092669+0000","last_peered":"2024-03-06T07:35:31.092669+0000","last_clean":"2024-03-06T07:35:31.092669+0000","last_became_active":"2024-03-06T07:35:31.087001+0000","last_became_peered":"2024-03-06T07:35:31.087001+0000","last_unstale":"2024-03-06T07:35:31.092669+0000","last_undegraded":"2024-03-06T07:35:31.092669+0000","last_fullsized":"2024-03-06T07:35:31.092669+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.732275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.089028+0000","last_change":"2024-03-06T07:35:31.085157+0000","last_active":"2024-03-06T07:35:31.089028+0000","last_peered":"2024-03-06T07:35:31.089028+0000","last_clean":"2024-03-06T07:35:31.089028+0000","last_became_active":"2024-03-06T07:35:31.084862+0000","last_became_peered":"2024-03-06T07:35:31.084862+0000","last_unstale":"2024-03-06T07:35:31.089028+0000","last_undegraded":"2024-03-06T07:35:31.089028+0000","last_fullsized":"2024-03-06T07:35:31.089028+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:31.641596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088269+0000","last_change":"2024-03-06T07:35:31.080243+0000","last_active":"2024-03-06T07:35:31.088269+0000","last_peered":"2024-03-06T07:35:31.088269+0000","last_clean":"2024-03-06T07:35:31.088269+0000","last_became_active":"2024-03-06T07:35:31.075467+0000","last_became_peered":"2024-03-06T07:35:31.075467+0000","last_unstale":"2024-03-06T07:35:31.088269+0000","last_undegraded":"2024-03-06T07:35:31.088269+0000","last_fullsized":"2024-03-06T07:35:31.088269+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:12.664361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.081817+0000","last_change":"2024-03-06T07:35:31.073269+0000","last_active":"2024-03-06T07:35:31.081817+0000","last_peered":"2024-03-06T07:35:31.081817+0000","last_clean":"2024-03-06T07:35:31.081817+0000","last_became_active":"2024-03-06T07:35:31.072948+0000","last_became_peered":"2024-03-06T07:35:31.072948+0000","last_unstale":"2024-03-06T07:35:31.081817+0000","last_undegraded":"2024-03-06T07:35:31.081817+0000","last_fullsized":"2024-03-06T07:35:31.081817+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:54.293725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082185+0000","last_change":"2024-03-06T07:35:31.076257+0000","last_active":"2024-03-06T07:35:31.082185+0000","last_peered":"2024-03-06T07:35:31.082185+0000","last_clean":"2024-03-06T07:35:31.082185+0000","last_became_active":"2024-03-06T07:35:31.075534+0000","last_became_peered":"2024-03-06T07:35:31.075534+0000","last_unstale":"2024-03-06T07:35:31.082185+0000","last_undegraded":"2024-03-06T07:35:31.082185+0000","last_fullsized":"2024-03-06T07:35:31.082185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.287892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085451+0000","last_change":"2024-03-06T07:35:31.082058+0000","last_active":"2024-03-06T07:35:31.085451+0000","last_peered":"2024-03-06T07:35:31.085451+0000","last_clean":"2024-03-06T07:35:31.085451+0000","last_became_active":"2024-03-06T07:35:31.081745+0000","last_became_peered":"2024-03-06T07:35:31.081745+0000","last_unstale":"2024-03-06T07:35:31.085451+0000","last_undegraded":"2024-03-06T07:35:31.085451+0000","last_fullsized":"2024-03-06T07:35:31.085451+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:45.685440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.066584+0000","last_change":"2024-03-06T07:35:31.059945+0000","last_active":"2024-03-06T07:35:31.066584+0000","last_peered":"2024-03-06T07:35:31.066584+0000","last_clean":"2024-03-06T07:35:31.066584+0000","last_became_active":"2024-03-06T07:35:31.059616+0000","last_became_peered":"2024-03-06T07:35:31.059616+0000","last_unstale":"2024-03-06T07:35:31.066584+0000","last_undegraded":"2024-03-06T07:35:31.066584+0000","last_fullsized":"2024-03-06T07:35:31.066584+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:40.753259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091272+0000","last_change":"2024-03-06T07:35:31.086607+0000","last_active":"2024-03-06T07:35:31.091272+0000","last_peered":"2024-03-06T07:35:31.091272+0000","last_clean":"2024-03-06T07:35:31.091272+0000","last_became_active":"2024-03-06T07:35:31.085704+0000","last_became_peered":"2024-03-06T07:35:31.085704+0000","last_unstale":"2024-03-06T07:35:31.091272+0000","last_undegraded":"2024-03-06T07:35:31.091272+0000","last_fullsized":"2024-03-06T07:35:31.091272+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:47.695177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.064788+0000","last_change":"2024-03-06T07:35:31.090324+0000","last_active":"2024-03-06T07:35:32.064788+0000","last_peered":"2024-03-06T07:35:32.064788+0000","last_clean":"2024-03-06T07:35:32.064788+0000","last_became_active":"2024-03-06T07:35:31.090027+0000","last_became_peered":"2024-03-06T07:35:31.090027+0000","last_unstale":"2024-03-06T07:35:32.064788+0000","last_undegraded":"2024-03-06T07:35:32.064788+0000","last_fullsized":"2024-03-06T07:35:32.064788+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:12.843143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.068266+0000","last_change":"2024-03-06T07:35:31.066242+0000","last_active":"2024-03-06T07:35:31.068266+0000","last_peered":"2024-03-06T07:35:31.068266+0000","last_clean":"2024-03-06T07:35:31.068266+0000","last_became_active":"2024-03-06T07:35:31.065736+0000","last_became_peered":"2024-03-06T07:35:31.065736+0000","last_unstale":"2024-03-06T07:35:31.068266+0000","last_undegraded":"2024-03-06T07:35:31.068266+0000","last_fullsized":"2024-03-06T07:35:31.068266+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:27.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.064963+0000","last_change":"2024-03-06T07:35:31.080181+0000","last_active":"2024-03-06T07:35:32.064963+0000","last_peered":"2024-03-06T07:35:32.064963+0000","last_clean":"2024-03-06T07:35:32.064963+0000","last_became_active":"2024-03-06T07:35:31.079846+0000","last_became_peered":"2024-03-06T07:35:31.079846+0000","last_unstale":"2024-03-06T07:35:32.064963+0000","last_undegraded":"2024-03-06T07:35:32.064963+0000","last_fullsized":"2024-03-06T07:35:32.064963+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:34.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.071517+0000","last_change":"2024-03-06T07:35:31.067684+0000","last_active":"2024-03-06T07:35:31.071517+0000","last_peered":"2024-03-06T07:35:31.071517+0000","last_clean":"2024-03-06T07:35:31.071517+0000","last_became_active":"2024-03-06T07:35:31.067414+0000","last_became_peered":"2024-03-06T07:35:31.067414+0000","last_unstale":"2024-03-06T07:35:31.071517+0000","last_undegraded":"2024-03-06T07:35:31.071517+0000","last_fullsized":"2024-03-06T07:35:31.071517+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:48.652921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088135+0000","last_change":"2024-03-06T07:35:31.081490+0000","last_active":"2024-03-06T07:35:31.088135+0000","last_peered":"2024-03-06T07:35:31.088135+0000","last_clean":"2024-03-06T07:35:31.088135+0000","last_became_active":"2024-03-06T07:35:31.080535+0000","last_became_peered":"2024-03-06T07:35:31.080535+0000","last_unstale":"2024-03-06T07:35:31.088135+0000","last_undegraded":"2024-03-06T07:35:31.088135+0000","last_fullsized":"2024-03-06T07:35:31.088135+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:06.762960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.084473+0000","last_change":"2024-03-06T07:35:31.081271+0000","last_active":"2024-03-06T07:35:31.084473+0000","last_peered":"2024-03-06T07:35:31.084473+0000","last_clean":"2024-03-06T07:35:31.084473+0000","last_became_active":"2024-03-06T07:35:31.080965+0000","last_became_peered":"2024-03-06T07:35:31.080965+0000","last_unstale":"2024-03-06T07:35:31.084473+0000","last_undegraded":"2024-03-06T07:35:31.084473+0000","last_fullsized":"2024-03-06T07:35:31.084473+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:33.051703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.076784+0000","last_change":"2024-03-06T07:35:31.074377+0000","last_active":"2024-03-06T07:35:31.076784+0000","last_peered":"2024-03-06T07:35:31.076784+0000","last_clean":"2024-03-06T07:35:31.076784+0000","last_became_active":"2024-03-06T07:35:31.074078+0000","last_became_peered":"2024-03-06T07:35:31.074078+0000","last_unstale":"2024-03-06T07:35:31.076784+0000","last_undegraded":"2024-03-06T07:35:31.076784+0000","last_fullsized":"2024-03-06T07:35:31.076784+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:53.730213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.076432+0000","last_change":"2024-03-06T07:35:31.070473+0000","last_active":"2024-03-06T07:35:31.076432+0000","last_peered":"2024-03-06T07:35:31.076432+0000","last_clean":"2024-03-06T07:35:31.076432+0000","last_became_active":"2024-03-06T07:35:31.068480+0000","last_became_peered":"2024-03-06T07:35:31.068480+0000","last_unstale":"2024-03-06T07:35:31.076432+0000","last_undegraded":"2024-03-06T07:35:31.076432+0000","last_fullsized":"2024-03-06T07:35:31.076432+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:42.146443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092594+0000","last_change":"2024-03-06T07:35:31.086064+0000","last_active":"2024-03-06T07:35:31.092594+0000","last_peered":"2024-03-06T07:35:31.092594+0000","last_clean":"2024-03-06T07:35:31.092594+0000","last_became_active":"2024-03-06T07:35:31.085705+0000","last_became_peered":"2024-03-06T07:35:31.085705+0000","last_unstale":"2024-03-06T07:35:31.092594+0000","last_undegraded":"2024-03-06T07:35:31.092594+0000","last_fullsized":"2024-03-06T07:35:31.092594+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:34.212984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.064519+0000","last_change":"2024-03-06T07:35:31.057959+0000","last_active":"2024-03-06T07:35:31.064519+0000","last_peered":"2024-03-06T07:35:31.064519+0000","last_clean":"2024-03-06T07:35:31.064519+0000","last_became_active":"2024-03-06T07:35:31.057669+0000","last_became_peered":"2024-03-06T07:35:31.057669+0000","last_unstale":"2024-03-06T07:35:31.064519+0000","last_undegraded":"2024-03-06T07:35:31.064519+0000","last_fullsized":"2024-03-06T07:35:31.064519+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:30:25.478431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.093034+0000","last_change":"2024-03-06T07:35:31.088914+0000","last_active":"2024-03-06T07:35:31.093034+0000","last_peered":"2024-03-06T07:35:31.093034+0000","last_clean":"2024-03-06T07:35:31.093034+0000","last_became_active":"2024-03-06T07:35:31.088319+0000","last_became_peered":"2024-03-06T07:35:31.088319+0000","last_unstale":"2024-03-06T07:35:31.093034+0000","last_undegraded":"2024-03-06T07:35:31.093034+0000","last_fullsized":"2024-03-06T07:35:31.093034+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:17.267852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086280+0000","last_change":"2024-03-06T07:35:31.085244+0000","last_active":"2024-03-06T07:35:31.086280+0000","last_peered":"2024-03-06T07:35:31.086280+0000","last_clean":"2024-03-06T07:35:31.086280+0000","last_became_active":"2024-03-06T07:35:31.084853+0000","last_became_peered":"2024-03-06T07:35:31.084853+0000","last_unstale":"2024-03-06T07:35:31.086280+0000","last_undegraded":"2024-03-06T07:35:31.086280+0000","last_fullsized":"2024-03-06T07:35:31.086280+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:01.603027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.075728+0000","last_change":"2024-03-06T07:35:31.074427+0000","last_active":"2024-03-06T07:35:31.075728+0000","last_peered":"2024-03-06T07:35:31.075728+0000","last_clean":"2024-03-06T07:35:31.075728+0000","last_became_active":"2024-03-06T07:35:31.073700+0000","last_became_peered":"2024-03-06T07:35:31.073700+0000","last_unstale":"2024-03-06T07:35:31.075728+0000","last_undegraded":"2024-03-06T07:35:31.075728+0000","last_fullsized":"2024-03-06T07:35:31.075728+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:41.233328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.075614+0000","last_change":"2024-03-06T07:35:31.069964+0000","last_active":"2024-03-06T07:35:31.075614+0000","last_peered":"2024-03-06T07:35:31.075614+0000","last_clean":"2024-03-06T07:35:31.075614+0000","last_became_active":"2024-03-06T07:35:31.068577+0000","last_became_peered":"2024-03-06T07:35:31.068577+0000","last_unstale":"2024-03-06T07:35:31.075614+0000","last_undegraded":"2024-03-06T07:35:31.075614+0000","last_fullsized":"2024-03-06T07:35:31.075614+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:34.958739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065939+0000","last_change":"2024-03-06T07:35:31.083376+0000","last_active":"2024-03-06T07:35:32.065939+0000","last_peered":"2024-03-06T07:35:32.065939+0000","last_clean":"2024-03-06T07:35:32.065939+0000","last_became_active":"2024-03-06T07:35:31.082803+0000","last_became_peered":"2024-03-06T07:35:31.082803+0000","last_unstale":"2024-03-06T07:35:32.065939+0000","last_undegraded":"2024-03-06T07:35:32.065939+0000","last_fullsized":"2024-03-06T07:35:32.065939+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:27.423888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088313+0000","last_change":"2024-03-06T07:35:31.086217+0000","last_active":"2024-03-06T07:35:31.088313+0000","last_peered":"2024-03-06T07:35:31.088313+0000","last_clean":"2024-03-06T07:35:31.088313+0000","last_became_active":"2024-03-06T07:35:31.084425+0000","last_became_peered":"2024-03-06T07:35:31.084425+0000","last_unstale":"2024-03-06T07:35:31.088313+0000","last_undegraded":"2024-03-06T07:35:31.088313+0000","last_fullsized":"2024-03-06T07:35:31.088313+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:45.043925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065031+0000","last_change":"2024-03-06T07:35:31.075053+0000","last_active":"2024-03-06T07:35:32.065031+0000","last_peered":"2024-03-06T07:35:32.065031+0000","last_clean":"2024-03-06T07:35:32.065031+0000","last_became_active":"2024-03-06T07:35:31.074742+0000","last_became_peered":"2024-03-06T07:35:31.074742+0000","last_unstale":"2024-03-06T07:35:32.065031+0000","last_undegraded":"2024-03-06T07:35:32.065031+0000","last_fullsized":"2024-03-06T07:35:32.065031+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:48.978175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065321+0000","last_change":"2024-03-06T07:35:31.073338+0000","last_active":"2024-03-06T07:35:32.065321+0000","last_peered":"2024-03-06T07:35:32.065321+0000","last_clean":"2024-03-06T07:35:32.065321+0000","last_became_active":"2024-03-06T07:35:31.072970+0000","last_became_peered":"2024-03-06T07:35:31.072970+0000","last_unstale":"2024-03-06T07:35:32.065321+0000","last_undegraded":"2024-03-06T07:35:32.065321+0000","last_fullsized":"2024-03-06T07:35:32.065321+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:02.547813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091197+0000","last_change":"2024-03-06T07:35:31.084869+0000","last_active":"2024-03-06T07:35:31.091197+0000","last_peered":"2024-03-06T07:35:31.091197+0000","last_clean":"2024-03-06T07:35:31.091197+0000","last_became_active":"2024-03-06T07:35:31.084464+0000","last_became_peered":"2024-03-06T07:35:31.084464+0000","last_unstale":"2024-03-06T07:35:31.091197+0000","last_undegraded":"2024-03-06T07:35:31.091197+0000","last_fullsized":"2024-03-06T07:35:31.091197+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:36.732723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.083452+0000","last_change":"2024-03-06T07:35:31.072115+0000","last_active":"2024-03-06T07:35:31.083452+0000","last_peered":"2024-03-06T07:35:31.083452+0000","last_clean":"2024-03-06T07:35:31.083452+0000","last_became_active":"2024-03-06T07:35:31.071843+0000","last_became_peered":"2024-03-06T07:35:31.071843+0000","last_unstale":"2024-03-06T07:35:31.083452+0000","last_undegraded":"2024-03-06T07:35:31.083452+0000","last_fullsized":"2024-03-06T07:35:31.083452+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:13.930044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.084802+0000","last_change":"2024-03-06T07:35:31.083736+0000","last_active":"2024-03-06T07:35:31.084802+0000","last_peered":"2024-03-06T07:35:31.084802+0000","last_clean":"2024-03-06T07:35:31.084802+0000","last_became_active":"2024-03-06T07:35:31.083299+0000","last_became_peered":"2024-03-06T07:35:31.083299+0000","last_unstale":"2024-03-06T07:35:31.084802+0000","last_undegraded":"2024-03-06T07:35:31.084802+0000","last_fullsized":"2024-03-06T07:35:31.084802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:07.272135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"125'4","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.090720+0000","last_change":"2024-03-06T07:35:30.049289+0000","last_active":"2024-03-06T07:35:31.090720+0000","last_peered":"2024-03-06T07:35:31.090720+0000","last_clean":"2024-03-06T07:35:31.090720+0000","last_became_active":"2024-03-06T07:35:30.048806+0000","last_became_peered":"2024-03-06T07:35:30.048806+0000","last_unstale":"2024-03-06T07:35:31.090720+0000","last_undegraded":"2024-03-06T07:35:31.090720+0000","last_fullsized":"2024-03-06T07:35:31.090720+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.626459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085704+0000","last_change":"2024-03-06T07:35:31.082036+0000","last_active":"2024-03-06T07:35:31.085704+0000","last_peered":"2024-03-06T07:35:31.085704+0000","last_clean":"2024-03-06T07:35:31.085704+0000","last_became_active":"2024-03-06T07:35:31.078269+0000","last_became_peered":"2024-03-06T07:35:31.078269+0000","last_unstale":"2024-03-06T07:35:31.085704+0000","last_undegraded":"2024-03-06T07:35:31.085704+0000","last_fullsized":"2024-03-06T07:35:31.085704+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:11.892738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082701+0000","last_change":"2024-03-06T07:35:30.036304+0000","last_active":"2024-03-06T07:35:31.082701+0000","last_peered":"2024-03-06T07:35:31.082701+0000","last_clean":"2024-03-06T07:35:31.082701+0000","last_became_active":"2024-03-06T07:35:30.035944+0000","last_became_peered":"2024-03-06T07:35:30.035944+0000","last_unstale":"2024-03-06T07:35:31.082701+0000","last_undegraded":"2024-03-06T07:35:31.082701+0000","last_fullsized":"2024-03-06T07:35:31.082701+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:33.525129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092824+0000","last_change":"2024-03-06T07:35:31.089213+0000","last_active":"2024-03-06T07:35:31.092824+0000","last_peered":"2024-03-06T07:35:31.092824+0000","last_clean":"2024-03-06T07:35:31.092824+0000","last_became_active":"2024-03-06T07:35:31.088881+0000","last_became_peered":"2024-03-06T07:35:31.088881+0000","last_unstale":"2024-03-06T07:35:31.092824+0000","last_undegraded":"2024-03-06T07:35:31.092824+0000","last_fullsized":"2024-03-06T07:35:31.092824+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.345461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.069640+0000","last_change":"2024-03-06T07:35:30.022256+0000","last_active":"2024-03-06T07:35:31.069640+0000","last_peered":"2024-03-06T07:35:31.069640+0000","last_clean":"2024-03-06T07:35:31.069640+0000","last_became_active":"2024-03-06T07:35:30.021875+0000","last_became_peered":"2024-03-06T07:35:30.021875+0000","last_unstale":"2024-03-06T07:35:31.069640+0000","last_undegraded":"2024-03-06T07:35:31.069640+0000","last_fullsized":"2024-03-06T07:35:31.069640+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:18.300977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085065+0000","last_change":"2024-03-06T07:35:31.076857+0000","last_active":"2024-03-06T07:35:31.085065+0000","last_peered":"2024-03-06T07:35:31.085065+0000","last_clean":"2024-03-06T07:35:31.085065+0000","last_became_active":"2024-03-06T07:35:31.074285+0000","last_became_peered":"2024-03-06T07:35:31.074285+0000","last_unstale":"2024-03-06T07:35:31.085065+0000","last_undegraded":"2024-03-06T07:35:31.085065+0000","last_fullsized":"2024-03-06T07:35:31.085065+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:39.309794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.068832+0000","last_change":"2024-03-06T07:35:30.033691+0000","last_active":"2024-03-06T07:35:31.068832+0000","last_peered":"2024-03-06T07:35:31.068832+0000","last_clean":"2024-03-06T07:35:31.068832+0000","last_became_active":"2024-03-06T07:35:30.031446+0000","last_became_peered":"2024-03-06T07:35:30.031446+0000","last_unstale":"2024-03-06T07:35:31.068832+0000","last_undegraded":"2024-03-06T07:35:31.068832+0000","last_fullsized":"2024-03-06T07:35:31.068832+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:02.436210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082185+0000","last_change":"2024-03-06T07:35:31.080017+0000","last_active":"2024-03-06T07:35:31.082185+0000","last_peered":"2024-03-06T07:35:31.082185+0000","last_clean":"2024-03-06T07:35:31.082185+0000","last_became_active":"2024-03-06T07:35:31.078787+0000","last_became_peered":"2024-03-06T07:35:31.078787+0000","last_unstale":"2024-03-06T07:35:31.082185+0000","last_undegraded":"2024-03-06T07:35:31.082185+0000","last_fullsized":"2024-03-06T07:35:31.082185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:51.504613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.069620+0000","last_change":"2024-03-06T07:35:30.031298+0000","last_active":"2024-03-06T07:35:31.069620+0000","last_peered":"2024-03-06T07:35:31.069620+0000","last_clean":"2024-03-06T07:35:31.069620+0000","last_became_active":"2024-03-06T07:35:30.030990+0000","last_became_peered":"2024-03-06T07:35:30.030990+0000","last_unstale":"2024-03-06T07:35:31.069620+0000","last_undegraded":"2024-03-06T07:35:31.069620+0000","last_fullsized":"2024-03-06T07:35:31.069620+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:01.532676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087999+0000","last_change":"2024-03-06T07:35:31.080931+0000","last_active":"2024-03-06T07:35:31.087999+0000","last_peered":"2024-03-06T07:35:31.087999+0000","last_clean":"2024-03-06T07:35:31.087999+0000","last_became_active":"2024-03-06T07:35:31.075129+0000","last_became_peered":"2024-03-06T07:35:31.075129+0000","last_unstale":"2024-03-06T07:35:31.087999+0000","last_undegraded":"2024-03-06T07:35:31.087999+0000","last_fullsized":"2024-03-06T07:35:31.087999+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:40.625424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.068239+0000","last_change":"2024-03-06T07:35:30.024560+0000","last_active":"2024-03-06T07:35:31.068239+0000","last_peered":"2024-03-06T07:35:31.068239+0000","last_clean":"2024-03-06T07:35:31.068239+0000","last_became_active":"2024-03-06T07:35:30.024329+0000","last_became_peered":"2024-03-06T07:35:30.024329+0000","last_unstale":"2024-03-06T07:35:31.068239+0000","last_undegraded":"2024-03-06T07:35:31.068239+0000","last_fullsized":"2024-03-06T07:35:31.068239+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:02.767220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.081791+0000","last_change":"2024-03-06T07:35:31.071215+0000","last_active":"2024-03-06T07:35:31.081791+0000","last_peered":"2024-03-06T07:35:31.081791+0000","last_clean":"2024-03-06T07:35:31.081791+0000","last_became_active":"2024-03-06T07:35:31.070890+0000","last_became_peered":"2024-03-06T07:35:31.070890+0000","last_unstale":"2024-03-06T07:35:31.081791+0000","last_undegraded":"2024-03-06T07:35:31.081791+0000","last_fullsized":"2024-03-06T07:35:31.081791+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:45.275788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.077092+0000","last_change":"2024-03-06T07:35:30.043286+0000","last_active":"2024-03-06T07:35:31.077092+0000","last_peered":"2024-03-06T07:35:31.077092+0000","last_clean":"2024-03-06T07:35:31.077092+0000","last_became_active":"2024-03-06T07:35:30.042372+0000","last_became_peered":"2024-03-06T07:35:30.042372+0000","last_unstale":"2024-03-06T07:35:31.077092+0000","last_undegraded":"2024-03-06T07:35:31.077092+0000","last_fullsized":"2024-03-06T07:35:31.077092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:56.768103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.062704+0000","last_change":"2024-03-06T07:35:31.059301+0000","last_active":"2024-03-06T07:35:31.062704+0000","last_peered":"2024-03-06T07:35:31.062704+0000","last_clean":"2024-03-06T07:35:31.062704+0000","last_became_active":"2024-03-06T07:35:31.058997+0000","last_became_peered":"2024-03-06T07:35:31.058997+0000","last_unstale":"2024-03-06T07:35:31.062704+0000","last_undegraded":"2024-03-06T07:35:31.062704+0000","last_fullsized":"2024-03-06T07:35:31.062704+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:11.383786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.044927+0000","last_change":"2024-03-06T07:35:30.037994+0000","last_active":"2024-03-06T07:35:31.044927+0000","last_peered":"2024-03-06T07:35:31.044927+0000","last_clean":"2024-03-06T07:35:31.044927+0000","last_became_active":"2024-03-06T07:35:30.037707+0000","last_became_peered":"2024-03-06T07:35:30.037707+0000","last_unstale":"2024-03-06T07:35:31.044927+0000","last_undegraded":"2024-03-06T07:35:31.044927+0000","last_fullsized":"2024-03-06T07:35:31.044927+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:26.526141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.066224+0000","last_change":"2024-03-06T07:35:31.068777+0000","last_active":"2024-03-06T07:35:32.066224+0000","last_peered":"2024-03-06T07:35:32.066224+0000","last_clean":"2024-03-06T07:35:32.066224+0000","last_became_active":"2024-03-06T07:35:31.068452+0000","last_became_peered":"2024-03-06T07:35:31.068452+0000","last_unstale":"2024-03-06T07:35:32.066224+0000","last_undegraded":"2024-03-06T07:35:32.066224+0000","last_fullsized":"2024-03-06T07:35:32.066224+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:35.032048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.066271+0000","last_change":"2024-03-06T07:35:30.038961+0000","last_active":"2024-03-06T07:35:31.066271+0000","last_peered":"2024-03-06T07:35:31.066271+0000","last_clean":"2024-03-06T07:35:31.066271+0000","last_became_active":"2024-03-06T07:35:30.037332+0000","last_became_peered":"2024-03-06T07:35:30.037332+0000","last_unstale":"2024-03-06T07:35:31.066271+0000","last_undegraded":"2024-03-06T07:35:31.066271+0000","last_fullsized":"2024-03-06T07:35:31.066271+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.970244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.084580+0000","last_change":"2024-03-06T07:35:31.074176+0000","last_active":"2024-03-06T07:35:31.084580+0000","last_peered":"2024-03-06T07:35:31.084580+0000","last_clean":"2024-03-06T07:35:31.084580+0000","last_became_active":"2024-03-06T07:35:31.073787+0000","last_became_peered":"2024-03-06T07:35:31.073787+0000","last_unstale":"2024-03-06T07:35:31.084580+0000","last_undegraded":"2024-03-06T07:35:31.084580+0000","last_fullsized":"2024-03-06T07:35:31.084580+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:01.999085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.054975+0000","last_change":"2024-03-06T07:35:30.039564+0000","last_active":"2024-03-06T07:35:31.054975+0000","last_peered":"2024-03-06T07:35:31.054975+0000","last_clean":"2024-03-06T07:35:31.054975+0000","last_became_active":"2024-03-06T07:35:30.038889+0000","last_became_peered":"2024-03-06T07:35:30.038889+0000","last_unstale":"2024-03-06T07:35:31.054975+0000","last_undegraded":"2024-03-06T07:35:31.054975+0000","last_fullsized":"2024-03-06T07:35:31.054975+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:46.222281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092803+0000","last_change":"2024-03-06T07:35:31.086401+0000","last_active":"2024-03-06T07:35:31.092803+0000","last_peered":"2024-03-06T07:35:31.092803+0000","last_clean":"2024-03-06T07:35:31.092803+0000","last_became_active":"2024-03-06T07:35:31.081501+0000","last_became_peered":"2024-03-06T07:35:31.081501+0000","last_unstale":"2024-03-06T07:35:31.092803+0000","last_undegraded":"2024-03-06T07:35:31.092803+0000","last_fullsized":"2024-03-06T07:35:31.092803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:25.782203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"125'3","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087111+0000","last_change":"2024-03-06T07:35:30.039183+0000","last_active":"2024-03-06T07:35:31.087111+0000","last_peered":"2024-03-06T07:35:31.087111+0000","last_clean":"2024-03-06T07:35:31.087111+0000","last_became_active":"2024-03-06T07:35:30.038896+0000","last_became_peered":"2024-03-06T07:35:30.038896+0000","last_unstale":"2024-03-06T07:35:31.087111+0000","last_undegraded":"2024-03-06T07:35:31.087111+0000","last_fullsized":"2024-03-06T07:35:31.087111+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:45.531697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085089+0000","last_change":"2024-03-06T07:35:31.078854+0000","last_active":"2024-03-06T07:35:31.085089+0000","last_peered":"2024-03-06T07:35:31.085089+0000","last_clean":"2024-03-06T07:35:31.085089+0000","last_became_active":"2024-03-06T07:35:31.078379+0000","last_became_peered":"2024-03-06T07:35:31.078379+0000","last_unstale":"2024-03-06T07:35:31.085089+0000","last_undegraded":"2024-03-06T07:35:31.085089+0000","last_fullsized":"2024-03-06T07:35:31.085089+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:29.181896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"125'6","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.093516+0000","last_change":"2024-03-06T07:35:30.044610+0000","last_active":"2024-03-06T07:35:31.093516+0000","last_peered":"2024-03-06T07:35:31.093516+0000","last_clean":"2024-03-06T07:35:31.093516+0000","last_became_active":"2024-03-06T07:35:30.043914+0000","last_became_peered":"2024-03-06T07:35:30.043914+0000","last_unstale":"2024-03-06T07:35:31.093516+0000","last_undegraded":"2024-03-06T07:35:31.093516+0000","last_fullsized":"2024-03-06T07:35:31.093516+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:47.055895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065272+0000","last_change":"2024-03-06T07:35:31.069880+0000","last_active":"2024-03-06T07:35:32.065272+0000","last_peered":"2024-03-06T07:35:32.065272+0000","last_clean":"2024-03-06T07:35:32.065272+0000","last_became_active":"2024-03-06T07:35:31.069612+0000","last_became_peered":"2024-03-06T07:35:31.069612+0000","last_unstale":"2024-03-06T07:35:32.065272+0000","last_undegraded":"2024-03-06T07:35:32.065272+0000","last_fullsized":"2024-03-06T07:35:32.065272+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:03.451348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065611+0000","last_change":"2024-03-06T07:35:30.043265+0000","last_active":"2024-03-06T07:35:32.065611+0000","last_peered":"2024-03-06T07:35:32.065611+0000","last_clean":"2024-03-06T07:35:32.065611+0000","last_became_active":"2024-03-06T07:35:30.042032+0000","last_became_peered":"2024-03-06T07:35:30.042032+0000","last_unstale":"2024-03-06T07:35:32.065611+0000","last_undegraded":"2024-03-06T07:35:32.065611+0000","last_fullsized":"2024-03-06T07:35:32.065611+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:14.134645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.065713+0000","last_change":"2024-03-06T07:35:31.062921+0000","last_active":"2024-03-06T07:35:31.065713+0000","last_peered":"2024-03-06T07:35:31.065713+0000","last_clean":"2024-03-06T07:35:31.065713+0000","last_became_active":"2024-03-06T07:35:31.062239+0000","last_became_peered":"2024-03-06T07:35:31.062239+0000","last_unstale":"2024-03-06T07:35:31.065713+0000","last_undegraded":"2024-03-06T07:35:31.065713+0000","last_fullsized":"2024-03-06T07:35:31.065713+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:45.000491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.043891+0000","last_change":"2024-03-06T07:35:30.042279+0000","last_active":"2024-03-06T07:35:31.043891+0000","last_peered":"2024-03-06T07:35:31.043891+0000","last_clean":"2024-03-06T07:35:31.043891+0000","last_became_active":"2024-03-06T07:35:30.042019+0000","last_became_peered":"2024-03-06T07:35:30.042019+0000","last_unstale":"2024-03-06T07:35:31.043891+0000","last_undegraded":"2024-03-06T07:35:31.043891+0000","last_fullsized":"2024-03-06T07:35:31.043891+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:47.963038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.093058+0000","last_change":"2024-03-06T07:35:31.089153+0000","last_active":"2024-03-06T07:35:31.093058+0000","last_peered":"2024-03-06T07:35:31.093058+0000","last_clean":"2024-03-06T07:35:31.093058+0000","last_became_active":"2024-03-06T07:35:31.088663+0000","last_became_peered":"2024-03-06T07:35:31.088663+0000","last_unstale":"2024-03-06T07:35:31.093058+0000","last_undegraded":"2024-03-06T07:35:31.093058+0000","last_fullsized":"2024-03-06T07:35:31.093058+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:27.590275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"125'8","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086143+0000","last_change":"2024-03-06T07:35:30.039434+0000","last_active":"2024-03-06T07:35:31.086143+0000","last_peered":"2024-03-06T07:35:31.086143+0000","last_clean":"2024-03-06T07:35:31.086143+0000","last_became_active":"2024-03-06T07:35:30.039088+0000","last_became_peered":"2024-03-06T07:35:30.039088+0000","last_unstale":"2024-03-06T07:35:31.086143+0000","last_undegraded":"2024-03-06T07:35:31.086143+0000","last_fullsized":"2024-03-06T07:35:31.086143+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:33.676045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082882+0000","last_change":"2024-03-06T07:35:31.080949+0000","last_active":"2024-03-06T07:35:31.082882+0000","last_peered":"2024-03-06T07:35:31.082882+0000","last_clean":"2024-03-06T07:35:31.082882+0000","last_became_active":"2024-03-06T07:35:31.078892+0000","last_became_peered":"2024-03-06T07:35:31.078892+0000","last_unstale":"2024-03-06T07:35:31.082882+0000","last_undegraded":"2024-03-06T07:35:31.082882+0000","last_fullsized":"2024-03-06T07:35:31.082882+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:08.438613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091952+0000","last_change":"2024-03-06T07:35:30.045104+0000","last_active":"2024-03-06T07:35:31.091952+0000","last_peered":"2024-03-06T07:35:31.091952+0000","last_clean":"2024-03-06T07:35:31.091952+0000","last_became_active":"2024-03-06T07:35:30.044457+0000","last_became_peered":"2024-03-06T07:35:30.044457+0000","last_unstale":"2024-03-06T07:35:31.091952+0000","last_undegraded":"2024-03-06T07:35:31.091952+0000","last_fullsized":"2024-03-06T07:35:31.091952+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:14.491709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086445+0000","last_change":"2024-03-06T07:35:31.079643+0000","last_active":"2024-03-06T07:35:31.086445+0000","last_peered":"2024-03-06T07:35:31.086445+0000","last_clean":"2024-03-06T07:35:31.086445+0000","last_became_active":"2024-03-06T07:35:31.079100+0000","last_became_peered":"2024-03-06T07:35:31.079100+0000","last_unstale":"2024-03-06T07:35:31.086445+0000","last_undegraded":"2024-03-06T07:35:31.086445+0000","last_fullsized":"2024-03-06T07:35:31.086445+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.216222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.074100+0000","last_change":"2024-03-06T07:35:30.033713+0000","last_active":"2024-03-06T07:35:31.074100+0000","last_peered":"2024-03-06T07:35:31.074100+0000","last_clean":"2024-03-06T07:35:31.074100+0000","last_became_active":"2024-03-06T07:35:30.033418+0000","last_became_peered":"2024-03-06T07:35:30.033418+0000","last_unstale":"2024-03-06T07:35:31.074100+0000","last_undegraded":"2024-03-06T07:35:31.074100+0000","last_fullsized":"2024-03-06T07:35:31.074100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:28.422330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.077324+0000","last_change":"2024-03-06T07:35:31.075594+0000","last_active":"2024-03-06T07:35:31.077324+0000","last_peered":"2024-03-06T07:35:31.077324+0000","last_clean":"2024-03-06T07:35:31.077324+0000","last_became_active":"2024-03-06T07:35:31.075305+0000","last_became_peered":"2024-03-06T07:35:31.075305+0000","last_unstale":"2024-03-06T07:35:31.077324+0000","last_undegraded":"2024-03-06T07:35:31.077324+0000","last_fullsized":"2024-03-06T07:35:31.077324+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:21.766248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.083717+0000","last_change":"2024-03-06T07:35:30.037748+0000","last_active":"2024-03-06T07:35:31.083717+0000","last_peered":"2024-03-06T07:35:31.083717+0000","last_clean":"2024-03-06T07:35:31.083717+0000","last_became_active":"2024-03-06T07:35:30.037440+0000","last_became_peered":"2024-03-06T07:35:30.037440+0000","last_unstale":"2024-03-06T07:35:31.083717+0000","last_undegraded":"2024-03-06T07:35:31.083717+0000","last_fullsized":"2024-03-06T07:35:31.083717+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.807072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.072137+0000","last_change":"2024-03-06T07:35:31.070382+0000","last_active":"2024-03-06T07:35:31.072137+0000","last_peered":"2024-03-06T07:35:31.072137+0000","last_clean":"2024-03-06T07:35:31.072137+0000","last_became_active":"2024-03-06T07:35:31.069769+0000","last_became_peered":"2024-03-06T07:35:31.069769+0000","last_unstale":"2024-03-06T07:35:31.072137+0000","last_undegraded":"2024-03-06T07:35:31.072137+0000","last_fullsized":"2024-03-06T07:35:31.072137+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:47.502208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.080590+0000","last_change":"2024-03-06T07:35:30.035027+0000","last_active":"2024-03-06T07:35:31.080590+0000","last_peered":"2024-03-06T07:35:31.080590+0000","last_clean":"2024-03-06T07:35:31.080590+0000","last_became_active":"2024-03-06T07:35:30.034569+0000","last_became_peered":"2024-03-06T07:35:30.034569+0000","last_unstale":"2024-03-06T07:35:31.080590+0000","last_undegraded":"2024-03-06T07:35:31.080590+0000","last_fullsized":"2024-03-06T07:35:31.080590+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:40.243018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065577+0000","last_change":"2024-03-06T07:35:31.082728+0000","last_active":"2024-03-06T07:35:32.065577+0000","last_peered":"2024-03-06T07:35:32.065577+0000","last_clean":"2024-03-06T07:35:32.065577+0000","last_became_active":"2024-03-06T07:35:31.082489+0000","last_became_peered":"2024-03-06T07:35:31.082489+0000","last_unstale":"2024-03-06T07:35:32.065577+0000","last_undegraded":"2024-03-06T07:35:32.065577+0000","last_fullsized":"2024-03-06T07:35:32.065577+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:00.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091631+0000","last_change":"2024-03-06T07:35:30.035046+0000","last_active":"2024-03-06T07:35:31.091631+0000","last_peered":"2024-03-06T07:35:31.091631+0000","last_clean":"2024-03-06T07:35:31.091631+0000","last_became_active":"2024-03-06T07:35:30.033003+0000","last_became_peered":"2024-03-06T07:35:30.033003+0000","last_unstale":"2024-03-06T07:35:31.091631+0000","last_undegraded":"2024-03-06T07:35:31.091631+0000","last_fullsized":"2024-03-06T07:35:31.091631+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:19.891102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087972+0000","last_change":"2024-03-06T07:35:31.078553+0000","last_active":"2024-03-06T07:35:31.087972+0000","last_peered":"2024-03-06T07:35:31.087972+0000","last_clean":"2024-03-06T07:35:31.087972+0000","last_became_active":"2024-03-06T07:35:31.074804+0000","last_became_peered":"2024-03-06T07:35:31.074804+0000","last_unstale":"2024-03-06T07:35:31.087972+0000","last_undegraded":"2024-03-06T07:35:31.087972+0000","last_fullsized":"2024-03-06T07:35:31.087972+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:02.368678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.053786+0000","last_change":"2024-03-06T07:35:30.027631+0000","last_active":"2024-03-06T07:35:31.053786+0000","last_peered":"2024-03-06T07:35:31.053786+0000","last_clean":"2024-03-06T07:35:31.053786+0000","last_became_active":"2024-03-06T07:35:30.027364+0000","last_became_peered":"2024-03-06T07:35:30.027364+0000","last_unstale":"2024-03-06T07:35:31.053786+0000","last_undegraded":"2024-03-06T07:35:31.053786+0000","last_fullsized":"2024-03-06T07:35:31.053786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:30.725663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.057391+0000","last_change":"2024-03-06T07:35:31.056358+0000","last_active":"2024-03-06T07:35:31.057391+0000","last_peered":"2024-03-06T07:35:31.057391+0000","last_clean":"2024-03-06T07:35:31.057391+0000","last_became_active":"2024-03-06T07:35:31.055546+0000","last_became_peered":"2024-03-06T07:35:31.055546+0000","last_unstale":"2024-03-06T07:35:31.057391+0000","last_undegraded":"2024-03-06T07:35:31.057391+0000","last_fullsized":"2024-03-06T07:35:31.057391+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:35.032048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.051788+0000","last_change":"2024-03-06T07:35:30.035330+0000","last_active":"2024-03-06T07:35:31.051788+0000","last_peered":"2024-03-06T07:35:31.051788+0000","last_clean":"2024-03-06T07:35:31.051788+0000","last_became_active":"2024-03-06T07:35:30.035058+0000","last_became_peered":"2024-03-06T07:35:30.035058+0000","last_unstale":"2024-03-06T07:35:31.051788+0000","last_undegraded":"2024-03-06T07:35:31.051788+0000","last_fullsized":"2024-03-06T07:35:31.051788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.575137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092794+0000","last_change":"2024-03-06T07:35:31.090318+0000","last_active":"2024-03-06T07:35:31.092794+0000","last_peered":"2024-03-06T07:35:31.092794+0000","last_clean":"2024-03-06T07:35:31.092794+0000","last_became_active":"2024-03-06T07:35:31.089852+0000","last_became_peered":"2024-03-06T07:35:31.089852+0000","last_unstale":"2024-03-06T07:35:31.092794+0000","last_undegraded":"2024-03-06T07:35:31.092794+0000","last_fullsized":"2024-03-06T07:35:31.092794+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:43.283452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.051974+0000","last_change":"2024-03-06T07:35:30.031608+0000","last_active":"2024-03-06T07:35:31.051974+0000","last_peered":"2024-03-06T07:35:31.051974+0000","last_clean":"2024-03-06T07:35:31.051974+0000","last_became_active":"2024-03-06T07:35:30.031279+0000","last_became_peered":"2024-03-06T07:35:30.031279+0000","last_unstale":"2024-03-06T07:35:31.051974+0000","last_undegraded":"2024-03-06T07:35:31.051974+0000","last_fullsized":"2024-03-06T07:35:31.051974+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:06.393220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.089142+0000","last_change":"2024-03-06T07:35:31.086338+0000","last_active":"2024-03-06T07:35:31.089142+0000","last_peered":"2024-03-06T07:35:31.089142+0000","last_clean":"2024-03-06T07:35:31.089142+0000","last_became_active":"2024-03-06T07:35:31.085137+0000","last_became_peered":"2024-03-06T07:35:31.085137+0000","last_unstale":"2024-03-06T07:35:31.089142+0000","last_undegraded":"2024-03-06T07:35:31.089142+0000","last_fullsized":"2024-03-06T07:35:31.089142+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:14.075792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.067782+0000","last_change":"2024-03-06T07:35:30.037407+0000","last_active":"2024-03-06T07:35:32.067782+0000","last_peered":"2024-03-06T07:35:32.067782+0000","last_clean":"2024-03-06T07:35:32.067782+0000","last_became_active":"2024-03-06T07:35:30.036918+0000","last_became_peered":"2024-03-06T07:35:30.036918+0000","last_unstale":"2024-03-06T07:35:32.067782+0000","last_undegraded":"2024-03-06T07:35:32.067782+0000","last_fullsized":"2024-03-06T07:35:32.067782+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:38.906760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.067183+0000","last_change":"2024-03-06T07:35:31.081206+0000","last_active":"2024-03-06T07:35:32.067183+0000","last_peered":"2024-03-06T07:35:32.067183+0000","last_clean":"2024-03-06T07:35:32.067183+0000","last_became_active":"2024-03-06T07:35:31.077539+0000","last_became_peered":"2024-03-06T07:35:31.077539+0000","last_unstale":"2024-03-06T07:35:32.067183+0000","last_undegraded":"2024-03-06T07:35:32.067183+0000","last_fullsized":"2024-03-06T07:35:32.067183+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:11.383786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.063069+0000","last_change":"2024-03-06T07:35:30.047979+0000","last_active":"2024-03-06T07:35:31.063069+0000","last_peered":"2024-03-06T07:35:31.063069+0000","last_clean":"2024-03-06T07:35:31.063069+0000","last_became_active":"2024-03-06T07:35:30.047634+0000","last_became_peered":"2024-03-06T07:35:30.047634+0000","last_unstale":"2024-03-06T07:35:31.063069+0000","last_undegraded":"2024-03-06T07:35:31.063069+0000","last_fullsized":"2024-03-06T07:35:31.063069+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:05.747823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.079802+0000","last_change":"2024-03-06T07:35:31.078809+0000","last_active":"2024-03-06T07:35:31.079802+0000","last_peered":"2024-03-06T07:35:31.079802+0000","last_clean":"2024-03-06T07:35:31.079802+0000","last_became_active":"2024-03-06T07:35:31.078494+0000","last_became_peered":"2024-03-06T07:35:31.078494+0000","last_unstale":"2024-03-06T07:35:31.079802+0000","last_undegraded":"2024-03-06T07:35:31.079802+0000","last_fullsized":"2024-03-06T07:35:31.079802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:44.825623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.045352+0000","last_change":"2024-03-06T07:35:30.041185+0000","last_active":"2024-03-06T07:35:31.045352+0000","last_peered":"2024-03-06T07:35:31.045352+0000","last_clean":"2024-03-06T07:35:31.045352+0000","last_became_active":"2024-03-06T07:35:30.040894+0000","last_became_peered":"2024-03-06T07:35:30.040894+0000","last_unstale":"2024-03-06T07:35:31.045352+0000","last_undegraded":"2024-03-06T07:35:31.045352+0000","last_fullsized":"2024-03-06T07:35:31.045352+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:59.309426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092631+0000","last_change":"2024-03-06T07:35:31.091436+0000","last_active":"2024-03-06T07:35:31.092631+0000","last_peered":"2024-03-06T07:35:31.092631+0000","last_clean":"2024-03-06T07:35:31.092631+0000","last_became_active":"2024-03-06T07:35:31.090996+0000","last_became_peered":"2024-03-06T07:35:31.090996+0000","last_unstale":"2024-03-06T07:35:31.092631+0000","last_undegraded":"2024-03-06T07:35:31.092631+0000","last_fullsized":"2024-03-06T07:35:31.092631+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:39.309794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.056340+0000","last_change":"2024-03-06T07:35:30.024276+0000","last_active":"2024-03-06T07:35:31.056340+0000","last_peered":"2024-03-06T07:35:31.056340+0000","last_clean":"2024-03-06T07:35:31.056340+0000","last_became_active":"2024-03-06T07:35:30.023889+0000","last_became_peered":"2024-03-06T07:35:30.023889+0000","last_unstale":"2024-03-06T07:35:31.056340+0000","last_undegraded":"2024-03-06T07:35:31.056340+0000","last_fullsized":"2024-03-06T07:35:31.056340+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:49.789801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087763+0000","last_change":"2024-03-06T07:35:31.079785+0000","last_active":"2024-03-06T07:35:31.087763+0000","last_peered":"2024-03-06T07:35:31.087763+0000","last_clean":"2024-03-06T07:35:31.087763+0000","last_became_active":"2024-03-06T07:35:31.079498+0000","last_became_peered":"2024-03-06T07:35:31.079498+0000","last_unstale":"2024-03-06T07:35:31.087763+0000","last_undegraded":"2024-03-06T07:35:31.087763+0000","last_fullsized":"2024-03-06T07:35:31.087763+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:44.399397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.046171+0000","last_change":"2024-03-06T07:35:30.047622+0000","last_active":"2024-03-06T07:35:31.046171+0000","last_peered":"2024-03-06T07:35:31.046171+0000","last_clean":"2024-03-06T07:35:31.046171+0000","last_became_active":"2024-03-06T07:35:30.047294+0000","last_became_peered":"2024-03-06T07:35:30.047294+0000","last_unstale":"2024-03-06T07:35:31.046171+0000","last_undegraded":"2024-03-06T07:35:31.046171+0000","last_fullsized":"2024-03-06T07:35:31.046171+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:26.408750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088189+0000","last_change":"2024-03-06T07:35:31.082980+0000","last_active":"2024-03-06T07:35:31.088189+0000","last_peered":"2024-03-06T07:35:31.088189+0000","last_clean":"2024-03-06T07:35:31.088189+0000","last_became_active":"2024-03-06T07:35:31.081956+0000","last_became_peered":"2024-03-06T07:35:31.081956+0000","last_unstale":"2024-03-06T07:35:31.088189+0000","last_undegraded":"2024-03-06T07:35:31.088189+0000","last_fullsized":"2024-03-06T07:35:31.088189+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:58.139075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.061453+0000","last_change":"2024-03-06T07:35:30.047816+0000","last_active":"2024-03-06T07:35:31.061453+0000","last_peered":"2024-03-06T07:35:31.061453+0000","last_clean":"2024-03-06T07:35:31.061453+0000","last_became_active":"2024-03-06T07:35:30.047450+0000","last_became_peered":"2024-03-06T07:35:30.047450+0000","last_unstale":"2024-03-06T07:35:31.061453+0000","last_undegraded":"2024-03-06T07:35:31.061453+0000","last_fullsized":"2024-03-06T07:35:31.061453+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:11.649223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.069130+0000","last_change":"2024-03-06T07:35:31.065889+0000","last_active":"2024-03-06T07:35:31.069130+0000","last_peered":"2024-03-06T07:35:31.069130+0000","last_clean":"2024-03-06T07:35:31.069130+0000","last_became_active":"2024-03-06T07:35:31.063697+0000","last_became_peered":"2024-03-06T07:35:31.063697+0000","last_unstale":"2024-03-06T07:35:31.069130+0000","last_undegraded":"2024-03-06T07:35:31.069130+0000","last_fullsized":"2024-03-06T07:35:31.069130+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:30.052004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.053782+0000","last_change":"2024-03-06T07:35:30.027364+0000","last_active":"2024-03-06T07:35:31.053782+0000","last_peered":"2024-03-06T07:35:31.053782+0000","last_clean":"2024-03-06T07:35:31.053782+0000","last_became_active":"2024-03-06T07:35:30.027059+0000","last_became_peered":"2024-03-06T07:35:30.027059+0000","last_unstale":"2024-03-06T07:35:31.053782+0000","last_undegraded":"2024-03-06T07:35:31.053782+0000","last_fullsized":"2024-03-06T07:35:31.053782+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:45.675494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.075733+0000","last_change":"2024-03-06T07:35:31.074790+0000","last_active":"2024-03-06T07:35:31.075733+0000","last_peered":"2024-03-06T07:35:31.075733+0000","last_clean":"2024-03-06T07:35:31.075733+0000","last_became_active":"2024-03-06T07:35:31.073549+0000","last_became_peered":"2024-03-06T07:35:31.073549+0000","last_unstale":"2024-03-06T07:35:31.075733+0000","last_undegraded":"2024-03-06T07:35:31.075733+0000","last_fullsized":"2024-03-06T07:35:31.075733+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:26.378562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.054683+0000","last_change":"2024-03-06T07:35:30.037878+0000","last_active":"2024-03-06T07:35:31.054683+0000","last_peered":"2024-03-06T07:35:31.054683+0000","last_clean":"2024-03-06T07:35:31.054683+0000","last_became_active":"2024-03-06T07:35:30.037610+0000","last_became_peered":"2024-03-06T07:35:30.037610+0000","last_unstale":"2024-03-06T07:35:31.054683+0000","last_undegraded":"2024-03-06T07:35:31.054683+0000","last_fullsized":"2024-03-06T07:35:31.054683+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:23.732107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.083185+0000","last_change":"2024-03-06T07:35:31.072067+0000","last_active":"2024-03-06T07:35:31.083185+0000","last_peered":"2024-03-06T07:35:31.083185+0000","last_clean":"2024-03-06T07:35:31.083185+0000","last_became_active":"2024-03-06T07:35:31.071608+0000","last_became_peered":"2024-03-06T07:35:31.071608+0000","last_unstale":"2024-03-06T07:35:31.083185+0000","last_undegraded":"2024-03-06T07:35:31.083185+0000","last_fullsized":"2024-03-06T07:35:31.083185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:14.075792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092134+0000","last_change":"2024-03-06T07:35:30.047830+0000","last_active":"2024-03-06T07:35:31.092134+0000","last_peered":"2024-03-06T07:35:31.092134+0000","last_clean":"2024-03-06T07:35:31.092134+0000","last_became_active":"2024-03-06T07:35:30.047425+0000","last_became_peered":"2024-03-06T07:35:30.047425+0000","last_unstale":"2024-03-06T07:35:31.092134+0000","last_undegraded":"2024-03-06T07:35:31.092134+0000","last_fullsized":"2024-03-06T07:35:31.092134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:39.610286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":355,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.058885+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:35:31.058885+0000","last_peered":"2024-03-06T07:35:31.058885+0000","last_clean":"2024-03-06T07:35:31.058885+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:35:31.058885+0000","last_undegraded":"2024-03-06T07:35:31.058885+0000","last_fullsized":"2024-03-06T07:35:31.058885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509541,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":673558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509542,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":672942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41628,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330212,"statfs":{"total":96636764160,"available":96594137088,"internally_reserved":0,"allocated":2170880,"data_stored":1265560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41008,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330832,"statfs":{"total":96636764160,"available":96594771968,"internally_reserved":0,"allocated":1540096,"data_stored":666167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542246,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":673054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41028,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330812,"statfs":{"total":96636764160,"available":96594751488,"internally_reserved":0,"allocated":1556480,"data_stored":673012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542246,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36848,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334992,"statfs":{"total":96636764160,"available":96599031808,"internally_reserved":0,"allocated":1540096,"data_stored":666167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37428,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334412,"statfs":{"total":96636764160,"available":96598437888,"internally_reserved":0,"allocated":2134016,"data_stored":1256535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:35.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:35:35.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.886+0000 7f758ca00640 1 Processor -- start 2024-03-06T07:35:35.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.886+0000 7f758ca00640 1 -- start start 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.888+0000 7f758ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 0x7f758815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.888+0000 7f758ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758815cc40 con 0x7f75880cdc20 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.888+0000 7f758ca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758815cf20 con 0x7f75880af580 2024-03-06T07:35:35.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.888+0000 7f758ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758815d200 con 0x7f75880af1a0 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.888+0000 7f7586800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f75880cdc20 0x7f75880b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42804/0 (socket says 172.21.15.6:42804) 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.888+0000 7f7586800640 1 -- 172.21.15.6:0/3546680767 learned_addr learned my addr 172.21.15.6:0/3546680767 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585e00640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 0x7f758815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598142766 0 0) 0x7f758815cc40 con 0x7f75880cdc20 2024-03-06T07:35:35.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c003520 con 0x7f75880cdc20 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3355991227 0 0) 0x7f756c003520 con 0x7f75880cdc20 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 msgr2=0x7f758815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 0x7f758815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 >> v1:172.21.15.178:6789/0 conn(0x7f75880af580 legacy=0x7f75880b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758815d4e0 con 0x7f75880cdc20 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7580001670 con 0x7f75880cdc20 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.889+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 >> v1:172.21.15.6:6789/0 conn(0x7f75880cdc20 legacy=0x7f75880b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585400640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f756c003a50 0x7f756c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585400640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 0x7f756c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585400640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f756c004d40 0x7f756c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758815d200 con 0x7f756c004d40 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c00bb50 con 0x7f756c003a50 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c00bd50 con 0x7f75880af1a0 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7586800640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f756c004d40 0x7f756c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7585e00640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f756c003a50 0x7f756c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7586800640 1 -- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 msgr2=0x7f756c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f757d400640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 0x7f756c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7586800640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f75880af1a0 0x7f756c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7586800640 1 -- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f756c003a50 msgr2=0x7f756c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7586800640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f756c003a50 0x7f756c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.890+0000 7f7586800640 1 -- 172.21.15.6:0/3546680767 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c003520 con 0x7f756c004d40 2024-03-06T07:35:35.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.891+0000 7f7586800640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f756c004d40 0x7f756c0051c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f758000c210 tx=0x7f756c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cede48ad8b752f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.892+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580007d30 con 0x7f756c004d40 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.892+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580030070 con 0x7f756c004d40 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.892+0000 7f7585400640 1 -- 172.21.15.6:0/3546680767 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758002cd60 con 0x7f756c004d40 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.892+0000 7f758ca00640 1 -- 172.21.15.6:0/3546680767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f756c004d40 msgr2=0x7f756c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.893+0000 7f758ca00640 1 --2- 172.21.15.6:0/3546680767 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f756c004d40 0x7f756c0051c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f758000c210 tx=0x7f756c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.893+0000 7f7586800640 1 -- 172.21.15.6:0/3546680767 reap_dead start 2024-03-06T07:35:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.893+0000 7f758ca00640 1 -- 172.21.15.6:0/3546680767 shutdown_connections 2024-03-06T07:35:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.893+0000 7f758ca00640 1 -- 172.21.15.6:0/3546680767 >> 172.21.15.6:0/3546680767 conn(0x7f758806afb0 msgr2=0x7f758806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.893+0000 7f758ca00640 1 -- 172.21.15.6:0/3546680767 shutdown_connections 2024-03-06T07:35:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.893+0000 7f758ca00640 1 -- 172.21.15.6:0/3546680767 wait complete. 2024-03-06T07:35:35.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.894+0000 7f758ca00640 1 Processor -- start 2024-03-06T07:35:35.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.895+0000 7f758ca00640 1 -- start start 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f758ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f756c004d40 0x7f7588156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f758ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75880b1610 0x7f75880cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f758ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 0x7f758814f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f758ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758815c600 con 0x7f7588150230 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f758ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880b36f0 con 0x7f75880b1610 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f758ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758815c780 con 0x7f756c004d40 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 0x7f758814f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 0x7f758814f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39448/0 (socket says 172.21.15.6:39448) 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 -- 172.21.15.6:0/2921230662 learned_addr learned my addr 172.21.15.6:0/2921230662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7585e00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f756c004d40 0x7f7588156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f757d400640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75880b1610 0x7f75880cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 -- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f756c004d40 msgr2=0x7f7588156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f756c004d40 0x7f7588156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.899 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 -- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75880b1610 msgr2=0x7f75880cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:35.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75880b1610 0x7f75880cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:35.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.896+0000 7f7586800640 1 -- 172.21.15.6:0/2921230662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588157d00 con 0x7f7588150230 2024-03-06T07:35:35.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.897+0000 7f7586800640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 0x7f758814f5f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f758002c570 tx=0x7f758002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.897+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580004bf0 con 0x7f7588150230 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.897+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580004d90 con 0x7f7588150230 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.897+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580007270 con 0x7f7588150230 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.897+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7588157fe0 con 0x7f7588150230 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.897+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75881578f0 con 0x7f7588150230 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.898+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7588069510 con 0x7f7588150230 2024-03-06T07:35:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.899+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7580007410 con 0x7f7588150230 2024-03-06T07:35:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.900+0000 7f757ea00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f756406b4d0 0x7f756406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.900+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75800ab360 con 0x7f7588150230 2024-03-06T07:35:35.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.901+0000 7f7585e00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f756406b4d0 0x7f756406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:35.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.902+0000 7f7585e00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f756406b4d0 0x7f756406d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7578007520 tx=0x7f757800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:35.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:35.909+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f758007a0b0 con 0x7f7588150230 2024-03-06T07:35:36.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.107+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 --> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f758804f3a0 con 0x7f756406b4d0 2024-03-06T07:35:36.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.126+0000 7f757ea00640 1 -- 172.21.15.6:0/2921230662 <== mgr.4110 v2:172.21.15.178:6832/3849979937 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252422 (secure 0 0 0) 0x7f758804f3a0 con 0x7f756406b4d0 2024-03-06T07:35:36.131 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:36.133 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-06T07:35:36.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f756406b4d0 msgr2=0x7f756406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f756406b4d0 0x7f756406d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7578007520 tx=0x7f757800b040 comp rx=0 tx=0).stop 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 msgr2=0x7f758814f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 0x7f758814f5f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f758002c570 tx=0x7f758002e070 comp rx=0 tx=0).stop 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 shutdown_connections 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f756406b4d0 0x7f756406d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f75880b1610 0x7f75880cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7588150230 0x7f758814f5f0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 --2- 172.21.15.6:0/2921230662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f756c004d40 0x7f7588156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 >> 172.21.15.6:0/2921230662 conn(0x7f758806afb0 msgr2=0x7f75880ad650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 shutdown_connections 2024-03-06T07:35:36.135 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.132+0000 7f758ca00640 1 -- 172.21.15.6:0/2921230662 wait complete. 2024-03-06T07:35:36.154 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-03-06T07:35:35.617978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312700,"kb_used_data":13296,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754662020,"statfs":{"total":773094113280,"available":772773908480,"internally_reserved":0,"allocated":13615104,"data_stored":6546995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002153"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.067897+0000","last_change":"2024-03-06T07:35:31.087859+0000","last_active":"2024-03-06T07:35:32.067897+0000","last_peered":"2024-03-06T07:35:32.067897+0000","last_clean":"2024-03-06T07:35:32.067897+0000","last_became_active":"2024-03-06T07:35:31.087497+0000","last_became_peered":"2024-03-06T07:35:31.087497+0000","last_unstale":"2024-03-06T07:35:32.067897+0000","last_undegraded":"2024-03-06T07:35:32.067897+0000","last_fullsized":"2024-03-06T07:35:32.067897+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:45.000491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086867+0000","last_change":"2024-03-06T07:35:31.085956+0000","last_active":"2024-03-06T07:35:31.086867+0000","last_peered":"2024-03-06T07:35:31.086867+0000","last_clean":"2024-03-06T07:35:31.086867+0000","last_became_active":"2024-03-06T07:35:31.085486+0000","last_became_peered":"2024-03-06T07:35:31.085486+0000","last_unstale":"2024-03-06T07:35:31.086867+0000","last_undegraded":"2024-03-06T07:35:31.086867+0000","last_fullsized":"2024-03-06T07:35:31.086867+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:25:07.280698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.070539+0000","last_change":"2024-03-06T07:35:31.064577+0000","last_active":"2024-03-06T07:35:31.070539+0000","last_peered":"2024-03-06T07:35:31.070539+0000","last_clean":"2024-03-06T07:35:31.070539+0000","last_became_active":"2024-03-06T07:35:31.064094+0000","last_became_peered":"2024-03-06T07:35:31.064094+0000","last_unstale":"2024-03-06T07:35:31.070539+0000","last_undegraded":"2024-03-06T07:35:31.070539+0000","last_fullsized":"2024-03-06T07:35:31.070539+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:58.512115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065495+0000","last_change":"2024-03-06T07:35:31.087422+0000","last_active":"2024-03-06T07:35:32.065495+0000","last_peered":"2024-03-06T07:35:32.065495+0000","last_clean":"2024-03-06T07:35:32.065495+0000","last_became_active":"2024-03-06T07:35:31.080944+0000","last_became_peered":"2024-03-06T07:35:31.080944+0000","last_unstale":"2024-03-06T07:35:32.065495+0000","last_undegraded":"2024-03-06T07:35:32.065495+0000","last_fullsized":"2024-03-06T07:35:32.065495+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:27.541279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092669+0000","last_change":"2024-03-06T07:35:31.087296+0000","last_active":"2024-03-06T07:35:31.092669+0000","last_peered":"2024-03-06T07:35:31.092669+0000","last_clean":"2024-03-06T07:35:31.092669+0000","last_became_active":"2024-03-06T07:35:31.087001+0000","last_became_peered":"2024-03-06T07:35:31.087001+0000","last_unstale":"2024-03-06T07:35:31.092669+0000","last_undegraded":"2024-03-06T07:35:31.092669+0000","last_fullsized":"2024-03-06T07:35:31.092669+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:51.732275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.089028+0000","last_change":"2024-03-06T07:35:31.085157+0000","last_active":"2024-03-06T07:35:31.089028+0000","last_peered":"2024-03-06T07:35:31.089028+0000","last_clean":"2024-03-06T07:35:31.089028+0000","last_became_active":"2024-03-06T07:35:31.084862+0000","last_became_peered":"2024-03-06T07:35:31.084862+0000","last_unstale":"2024-03-06T07:35:31.089028+0000","last_undegraded":"2024-03-06T07:35:31.089028+0000","last_fullsized":"2024-03-06T07:35:31.089028+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:31.641596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088269+0000","last_change":"2024-03-06T07:35:31.080243+0000","last_active":"2024-03-06T07:35:31.088269+0000","last_peered":"2024-03-06T07:35:31.088269+0000","last_clean":"2024-03-06T07:35:31.088269+0000","last_became_active":"2024-03-06T07:35:31.075467+0000","last_became_peered":"2024-03-06T07:35:31.075467+0000","last_unstale":"2024-03-06T07:35:31.088269+0000","last_undegraded":"2024-03-06T07:35:31.088269+0000","last_fullsized":"2024-03-06T07:35:31.088269+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:12.664361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.081817+0000","last_change":"2024-03-06T07:35:31.073269+0000","last_active":"2024-03-06T07:35:31.081817+0000","last_peered":"2024-03-06T07:35:31.081817+0000","last_clean":"2024-03-06T07:35:31.081817+0000","last_became_active":"2024-03-06T07:35:31.072948+0000","last_became_peered":"2024-03-06T07:35:31.072948+0000","last_unstale":"2024-03-06T07:35:31.081817+0000","last_undegraded":"2024-03-06T07:35:31.081817+0000","last_fullsized":"2024-03-06T07:35:31.081817+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:00:54.293725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082185+0000","last_change":"2024-03-06T07:35:31.076257+0000","last_active":"2024-03-06T07:35:31.082185+0000","last_peered":"2024-03-06T07:35:31.082185+0000","last_clean":"2024-03-06T07:35:31.082185+0000","last_became_active":"2024-03-06T07:35:31.075534+0000","last_became_peered":"2024-03-06T07:35:31.075534+0000","last_unstale":"2024-03-06T07:35:31.082185+0000","last_undegraded":"2024-03-06T07:35:31.082185+0000","last_fullsized":"2024-03-06T07:35:31.082185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:12.287892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085451+0000","last_change":"2024-03-06T07:35:31.082058+0000","last_active":"2024-03-06T07:35:31.085451+0000","last_peered":"2024-03-06T07:35:31.085451+0000","last_clean":"2024-03-06T07:35:31.085451+0000","last_became_active":"2024-03-06T07:35:31.081745+0000","last_became_peered":"2024-03-06T07:35:31.081745+0000","last_unstale":"2024-03-06T07:35:31.085451+0000","last_undegraded":"2024-03-06T07:35:31.085451+0000","last_fullsized":"2024-03-06T07:35:31.085451+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:45.685440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.066584+0000","last_change":"2024-03-06T07:35:31.059945+0000","last_active":"2024-03-06T07:35:31.066584+0000","last_peered":"2024-03-06T07:35:31.066584+0000","last_clean":"2024-03-06T07:35:31.066584+0000","last_became_active":"2024-03-06T07:35:31.059616+0000","last_became_peered":"2024-03-06T07:35:31.059616+0000","last_unstale":"2024-03-06T07:35:31.066584+0000","last_undegraded":"2024-03-06T07:35:31.066584+0000","last_fullsized":"2024-03-06T07:35:31.066584+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:40.753259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091272+0000","last_change":"2024-03-06T07:35:31.086607+0000","last_active":"2024-03-06T07:35:31.091272+0000","last_peered":"2024-03-06T07:35:31.091272+0000","last_clean":"2024-03-06T07:35:31.091272+0000","last_became_active":"2024-03-06T07:35:31.085704+0000","last_became_peered":"2024-03-06T07:35:31.085704+0000","last_unstale":"2024-03-06T07:35:31.091272+0000","last_undegraded":"2024-03-06T07:35:31.091272+0000","last_fullsized":"2024-03-06T07:35:31.091272+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:59:47.695177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.064788+0000","last_change":"2024-03-06T07:35:31.090324+0000","last_active":"2024-03-06T07:35:32.064788+0000","last_peered":"2024-03-06T07:35:32.064788+0000","last_clean":"2024-03-06T07:35:32.064788+0000","last_became_active":"2024-03-06T07:35:31.090027+0000","last_became_peered":"2024-03-06T07:35:31.090027+0000","last_unstale":"2024-03-06T07:35:32.064788+0000","last_undegraded":"2024-03-06T07:35:32.064788+0000","last_fullsized":"2024-03-06T07:35:32.064788+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:12.843143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.068266+0000","last_change":"2024-03-06T07:35:31.066242+0000","last_active":"2024-03-06T07:35:31.068266+0000","last_peered":"2024-03-06T07:35:31.068266+0000","last_clean":"2024-03-06T07:35:31.068266+0000","last_became_active":"2024-03-06T07:35:31.065736+0000","last_became_peered":"2024-03-06T07:35:31.065736+0000","last_unstale":"2024-03-06T07:35:31.068266+0000","last_undegraded":"2024-03-06T07:35:31.068266+0000","last_fullsized":"2024-03-06T07:35:31.068266+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:48:27.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.064963+0000","last_change":"2024-03-06T07:35:31.080181+0000","last_active":"2024-03-06T07:35:32.064963+0000","last_peered":"2024-03-06T07:35:32.064963+0000","last_clean":"2024-03-06T07:35:32.064963+0000","last_became_active":"2024-03-06T07:35:31.079846+0000","last_became_peered":"2024-03-06T07:35:31.079846+0000","last_unstale":"2024-03-06T07:35:32.064963+0000","last_undegraded":"2024-03-06T07:35:32.064963+0000","last_fullsized":"2024-03-06T07:35:32.064963+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:49:34.395241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.071517+0000","last_change":"2024-03-06T07:35:31.067684+0000","last_active":"2024-03-06T07:35:31.071517+0000","last_peered":"2024-03-06T07:35:31.071517+0000","last_clean":"2024-03-06T07:35:31.071517+0000","last_became_active":"2024-03-06T07:35:31.067414+0000","last_became_peered":"2024-03-06T07:35:31.067414+0000","last_unstale":"2024-03-06T07:35:31.071517+0000","last_undegraded":"2024-03-06T07:35:31.071517+0000","last_fullsized":"2024-03-06T07:35:31.071517+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:36:48.652921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088135+0000","last_change":"2024-03-06T07:35:31.081490+0000","last_active":"2024-03-06T07:35:31.088135+0000","last_peered":"2024-03-06T07:35:31.088135+0000","last_clean":"2024-03-06T07:35:31.088135+0000","last_became_active":"2024-03-06T07:35:31.080535+0000","last_became_peered":"2024-03-06T07:35:31.080535+0000","last_unstale":"2024-03-06T07:35:31.088135+0000","last_undegraded":"2024-03-06T07:35:31.088135+0000","last_fullsized":"2024-03-06T07:35:31.088135+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:06.762960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.084473+0000","last_change":"2024-03-06T07:35:31.081271+0000","last_active":"2024-03-06T07:35:31.084473+0000","last_peered":"2024-03-06T07:35:31.084473+0000","last_clean":"2024-03-06T07:35:31.084473+0000","last_became_active":"2024-03-06T07:35:31.080965+0000","last_became_peered":"2024-03-06T07:35:31.080965+0000","last_unstale":"2024-03-06T07:35:31.084473+0000","last_undegraded":"2024-03-06T07:35:31.084473+0000","last_fullsized":"2024-03-06T07:35:31.084473+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:37:33.051703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.076784+0000","last_change":"2024-03-06T07:35:31.074377+0000","last_active":"2024-03-06T07:35:31.076784+0000","last_peered":"2024-03-06T07:35:31.076784+0000","last_clean":"2024-03-06T07:35:31.076784+0000","last_became_active":"2024-03-06T07:35:31.074078+0000","last_became_peered":"2024-03-06T07:35:31.074078+0000","last_unstale":"2024-03-06T07:35:31.076784+0000","last_undegraded":"2024-03-06T07:35:31.076784+0000","last_fullsized":"2024-03-06T07:35:31.076784+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:53.730213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.076432+0000","last_change":"2024-03-06T07:35:31.070473+0000","last_active":"2024-03-06T07:35:31.076432+0000","last_peered":"2024-03-06T07:35:31.076432+0000","last_clean":"2024-03-06T07:35:31.076432+0000","last_became_active":"2024-03-06T07:35:31.068480+0000","last_became_peered":"2024-03-06T07:35:31.068480+0000","last_unstale":"2024-03-06T07:35:31.076432+0000","last_undegraded":"2024-03-06T07:35:31.076432+0000","last_fullsized":"2024-03-06T07:35:31.076432+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:42.146443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092594+0000","last_change":"2024-03-06T07:35:31.086064+0000","last_active":"2024-03-06T07:35:31.092594+0000","last_peered":"2024-03-06T07:35:31.092594+0000","last_clean":"2024-03-06T07:35:31.092594+0000","last_became_active":"2024-03-06T07:35:31.085705+0000","last_became_peered":"2024-03-06T07:35:31.085705+0000","last_unstale":"2024-03-06T07:35:31.092594+0000","last_undegraded":"2024-03-06T07:35:31.092594+0000","last_fullsized":"2024-03-06T07:35:31.092594+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:45:34.212984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.064519+0000","last_change":"2024-03-06T07:35:31.057959+0000","last_active":"2024-03-06T07:35:31.064519+0000","last_peered":"2024-03-06T07:35:31.064519+0000","last_clean":"2024-03-06T07:35:31.064519+0000","last_became_active":"2024-03-06T07:35:31.057669+0000","last_became_peered":"2024-03-06T07:35:31.057669+0000","last_unstale":"2024-03-06T07:35:31.064519+0000","last_undegraded":"2024-03-06T07:35:31.064519+0000","last_fullsized":"2024-03-06T07:35:31.064519+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:30:25.478431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.093034+0000","last_change":"2024-03-06T07:35:31.088914+0000","last_active":"2024-03-06T07:35:31.093034+0000","last_peered":"2024-03-06T07:35:31.093034+0000","last_clean":"2024-03-06T07:35:31.093034+0000","last_became_active":"2024-03-06T07:35:31.088319+0000","last_became_peered":"2024-03-06T07:35:31.088319+0000","last_unstale":"2024-03-06T07:35:31.093034+0000","last_undegraded":"2024-03-06T07:35:31.093034+0000","last_fullsized":"2024-03-06T07:35:31.093034+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:17.267852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086280+0000","last_change":"2024-03-06T07:35:31.085244+0000","last_active":"2024-03-06T07:35:31.086280+0000","last_peered":"2024-03-06T07:35:31.086280+0000","last_clean":"2024-03-06T07:35:31.086280+0000","last_became_active":"2024-03-06T07:35:31.084853+0000","last_became_peered":"2024-03-06T07:35:31.084853+0000","last_unstale":"2024-03-06T07:35:31.086280+0000","last_undegraded":"2024-03-06T07:35:31.086280+0000","last_fullsized":"2024-03-06T07:35:31.086280+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:19:01.603027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.075728+0000","last_change":"2024-03-06T07:35:31.074427+0000","last_active":"2024-03-06T07:35:31.075728+0000","last_peered":"2024-03-06T07:35:31.075728+0000","last_clean":"2024-03-06T07:35:31.075728+0000","last_became_active":"2024-03-06T07:35:31.073700+0000","last_became_peered":"2024-03-06T07:35:31.073700+0000","last_unstale":"2024-03-06T07:35:31.075728+0000","last_undegraded":"2024-03-06T07:35:31.075728+0000","last_fullsized":"2024-03-06T07:35:31.075728+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:03:41.233328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.075614+0000","last_change":"2024-03-06T07:35:31.069964+0000","last_active":"2024-03-06T07:35:31.075614+0000","last_peered":"2024-03-06T07:35:31.075614+0000","last_clean":"2024-03-06T07:35:31.075614+0000","last_became_active":"2024-03-06T07:35:31.068577+0000","last_became_peered":"2024-03-06T07:35:31.068577+0000","last_unstale":"2024-03-06T07:35:31.075614+0000","last_undegraded":"2024-03-06T07:35:31.075614+0000","last_fullsized":"2024-03-06T07:35:31.075614+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:30:34.958739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065939+0000","last_change":"2024-03-06T07:35:31.083376+0000","last_active":"2024-03-06T07:35:32.065939+0000","last_peered":"2024-03-06T07:35:32.065939+0000","last_clean":"2024-03-06T07:35:32.065939+0000","last_became_active":"2024-03-06T07:35:31.082803+0000","last_became_peered":"2024-03-06T07:35:31.082803+0000","last_unstale":"2024-03-06T07:35:32.065939+0000","last_undegraded":"2024-03-06T07:35:32.065939+0000","last_fullsized":"2024-03-06T07:35:32.065939+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:27.423888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088313+0000","last_change":"2024-03-06T07:35:31.086217+0000","last_active":"2024-03-06T07:35:31.088313+0000","last_peered":"2024-03-06T07:35:31.088313+0000","last_clean":"2024-03-06T07:35:31.088313+0000","last_became_active":"2024-03-06T07:35:31.084425+0000","last_became_peered":"2024-03-06T07:35:31.084425+0000","last_unstale":"2024-03-06T07:35:31.088313+0000","last_undegraded":"2024-03-06T07:35:31.088313+0000","last_fullsized":"2024-03-06T07:35:31.088313+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:14:45.043925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065031+0000","last_change":"2024-03-06T07:35:31.075053+0000","last_active":"2024-03-06T07:35:32.065031+0000","last_peered":"2024-03-06T07:35:32.065031+0000","last_clean":"2024-03-06T07:35:32.065031+0000","last_became_active":"2024-03-06T07:35:31.074742+0000","last_became_peered":"2024-03-06T07:35:31.074742+0000","last_unstale":"2024-03-06T07:35:32.065031+0000","last_undegraded":"2024-03-06T07:35:32.065031+0000","last_fullsized":"2024-03-06T07:35:32.065031+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:48.978175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065321+0000","last_change":"2024-03-06T07:35:31.073338+0000","last_active":"2024-03-06T07:35:32.065321+0000","last_peered":"2024-03-06T07:35:32.065321+0000","last_clean":"2024-03-06T07:35:32.065321+0000","last_became_active":"2024-03-06T07:35:31.072970+0000","last_became_peered":"2024-03-06T07:35:31.072970+0000","last_unstale":"2024-03-06T07:35:32.065321+0000","last_undegraded":"2024-03-06T07:35:32.065321+0000","last_fullsized":"2024-03-06T07:35:32.065321+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:02.547813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091197+0000","last_change":"2024-03-06T07:35:31.084869+0000","last_active":"2024-03-06T07:35:31.091197+0000","last_peered":"2024-03-06T07:35:31.091197+0000","last_clean":"2024-03-06T07:35:31.091197+0000","last_became_active":"2024-03-06T07:35:31.084464+0000","last_became_peered":"2024-03-06T07:35:31.084464+0000","last_unstale":"2024-03-06T07:35:31.091197+0000","last_undegraded":"2024-03-06T07:35:31.091197+0000","last_fullsized":"2024-03-06T07:35:31.091197+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:36.732723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.083452+0000","last_change":"2024-03-06T07:35:31.072115+0000","last_active":"2024-03-06T07:35:31.083452+0000","last_peered":"2024-03-06T07:35:31.083452+0000","last_clean":"2024-03-06T07:35:31.083452+0000","last_became_active":"2024-03-06T07:35:31.071843+0000","last_became_peered":"2024-03-06T07:35:31.071843+0000","last_unstale":"2024-03-06T07:35:31.083452+0000","last_undegraded":"2024-03-06T07:35:31.083452+0000","last_fullsized":"2024-03-06T07:35:31.083452+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:13.930044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.084802+0000","last_change":"2024-03-06T07:35:31.083736+0000","last_active":"2024-03-06T07:35:31.084802+0000","last_peered":"2024-03-06T07:35:31.084802+0000","last_clean":"2024-03-06T07:35:31.084802+0000","last_became_active":"2024-03-06T07:35:31.083299+0000","last_became_peered":"2024-03-06T07:35:31.083299+0000","last_unstale":"2024-03-06T07:35:31.084802+0000","last_undegraded":"2024-03-06T07:35:31.084802+0000","last_fullsized":"2024-03-06T07:35:31.084802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:07.272135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"125'4","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.090720+0000","last_change":"2024-03-06T07:35:30.049289+0000","last_active":"2024-03-06T07:35:31.090720+0000","last_peered":"2024-03-06T07:35:31.090720+0000","last_clean":"2024-03-06T07:35:31.090720+0000","last_became_active":"2024-03-06T07:35:30.048806+0000","last_became_peered":"2024-03-06T07:35:30.048806+0000","last_unstale":"2024-03-06T07:35:31.090720+0000","last_undegraded":"2024-03-06T07:35:31.090720+0000","last_fullsized":"2024-03-06T07:35:31.090720+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.626459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085704+0000","last_change":"2024-03-06T07:35:31.082036+0000","last_active":"2024-03-06T07:35:31.085704+0000","last_peered":"2024-03-06T07:35:31.085704+0000","last_clean":"2024-03-06T07:35:31.085704+0000","last_became_active":"2024-03-06T07:35:31.078269+0000","last_became_peered":"2024-03-06T07:35:31.078269+0000","last_unstale":"2024-03-06T07:35:31.085704+0000","last_undegraded":"2024-03-06T07:35:31.085704+0000","last_fullsized":"2024-03-06T07:35:31.085704+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:14:11.892738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082701+0000","last_change":"2024-03-06T07:35:30.036304+0000","last_active":"2024-03-06T07:35:31.082701+0000","last_peered":"2024-03-06T07:35:31.082701+0000","last_clean":"2024-03-06T07:35:31.082701+0000","last_became_active":"2024-03-06T07:35:30.035944+0000","last_became_peered":"2024-03-06T07:35:30.035944+0000","last_unstale":"2024-03-06T07:35:31.082701+0000","last_undegraded":"2024-03-06T07:35:31.082701+0000","last_fullsized":"2024-03-06T07:35:31.082701+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:52:33.525129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092824+0000","last_change":"2024-03-06T07:35:31.089213+0000","last_active":"2024-03-06T07:35:31.092824+0000","last_peered":"2024-03-06T07:35:31.092824+0000","last_clean":"2024-03-06T07:35:31.092824+0000","last_became_active":"2024-03-06T07:35:31.088881+0000","last_became_peered":"2024-03-06T07:35:31.088881+0000","last_unstale":"2024-03-06T07:35:31.092824+0000","last_undegraded":"2024-03-06T07:35:31.092824+0000","last_fullsized":"2024-03-06T07:35:31.092824+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:31.345461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.069640+0000","last_change":"2024-03-06T07:35:30.022256+0000","last_active":"2024-03-06T07:35:31.069640+0000","last_peered":"2024-03-06T07:35:31.069640+0000","last_clean":"2024-03-06T07:35:31.069640+0000","last_became_active":"2024-03-06T07:35:30.021875+0000","last_became_peered":"2024-03-06T07:35:30.021875+0000","last_unstale":"2024-03-06T07:35:31.069640+0000","last_undegraded":"2024-03-06T07:35:31.069640+0000","last_fullsized":"2024-03-06T07:35:31.069640+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:34:18.300977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085065+0000","last_change":"2024-03-06T07:35:31.076857+0000","last_active":"2024-03-06T07:35:31.085065+0000","last_peered":"2024-03-06T07:35:31.085065+0000","last_clean":"2024-03-06T07:35:31.085065+0000","last_became_active":"2024-03-06T07:35:31.074285+0000","last_became_peered":"2024-03-06T07:35:31.074285+0000","last_unstale":"2024-03-06T07:35:31.085065+0000","last_undegraded":"2024-03-06T07:35:31.085065+0000","last_fullsized":"2024-03-06T07:35:31.085065+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:39.309794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.068832+0000","last_change":"2024-03-06T07:35:30.033691+0000","last_active":"2024-03-06T07:35:31.068832+0000","last_peered":"2024-03-06T07:35:31.068832+0000","last_clean":"2024-03-06T07:35:31.068832+0000","last_became_active":"2024-03-06T07:35:30.031446+0000","last_became_peered":"2024-03-06T07:35:30.031446+0000","last_unstale":"2024-03-06T07:35:31.068832+0000","last_undegraded":"2024-03-06T07:35:31.068832+0000","last_fullsized":"2024-03-06T07:35:31.068832+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:02.436210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082185+0000","last_change":"2024-03-06T07:35:31.080017+0000","last_active":"2024-03-06T07:35:31.082185+0000","last_peered":"2024-03-06T07:35:31.082185+0000","last_clean":"2024-03-06T07:35:31.082185+0000","last_became_active":"2024-03-06T07:35:31.078787+0000","last_became_peered":"2024-03-06T07:35:31.078787+0000","last_unstale":"2024-03-06T07:35:31.082185+0000","last_undegraded":"2024-03-06T07:35:31.082185+0000","last_fullsized":"2024-03-06T07:35:31.082185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:51.504613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.069620+0000","last_change":"2024-03-06T07:35:30.031298+0000","last_active":"2024-03-06T07:35:31.069620+0000","last_peered":"2024-03-06T07:35:31.069620+0000","last_clean":"2024-03-06T07:35:31.069620+0000","last_became_active":"2024-03-06T07:35:30.030990+0000","last_became_peered":"2024-03-06T07:35:30.030990+0000","last_unstale":"2024-03-06T07:35:31.069620+0000","last_undegraded":"2024-03-06T07:35:31.069620+0000","last_fullsized":"2024-03-06T07:35:31.069620+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:12:01.532676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087999+0000","last_change":"2024-03-06T07:35:31.080931+0000","last_active":"2024-03-06T07:35:31.087999+0000","last_peered":"2024-03-06T07:35:31.087999+0000","last_clean":"2024-03-06T07:35:31.087999+0000","last_became_active":"2024-03-06T07:35:31.075129+0000","last_became_peered":"2024-03-06T07:35:31.075129+0000","last_unstale":"2024-03-06T07:35:31.087999+0000","last_undegraded":"2024-03-06T07:35:31.087999+0000","last_fullsized":"2024-03-06T07:35:31.087999+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:40.625424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.068239+0000","last_change":"2024-03-06T07:35:30.024560+0000","last_active":"2024-03-06T07:35:31.068239+0000","last_peered":"2024-03-06T07:35:31.068239+0000","last_clean":"2024-03-06T07:35:31.068239+0000","last_became_active":"2024-03-06T07:35:30.024329+0000","last_became_peered":"2024-03-06T07:35:30.024329+0000","last_unstale":"2024-03-06T07:35:31.068239+0000","last_undegraded":"2024-03-06T07:35:31.068239+0000","last_fullsized":"2024-03-06T07:35:31.068239+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:02.767220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.081791+0000","last_change":"2024-03-06T07:35:31.071215+0000","last_active":"2024-03-06T07:35:31.081791+0000","last_peered":"2024-03-06T07:35:31.081791+0000","last_clean":"2024-03-06T07:35:31.081791+0000","last_became_active":"2024-03-06T07:35:31.070890+0000","last_became_peered":"2024-03-06T07:35:31.070890+0000","last_unstale":"2024-03-06T07:35:31.081791+0000","last_undegraded":"2024-03-06T07:35:31.081791+0000","last_fullsized":"2024-03-06T07:35:31.081791+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:42:45.275788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.077092+0000","last_change":"2024-03-06T07:35:30.043286+0000","last_active":"2024-03-06T07:35:31.077092+0000","last_peered":"2024-03-06T07:35:31.077092+0000","last_clean":"2024-03-06T07:35:31.077092+0000","last_became_active":"2024-03-06T07:35:30.042372+0000","last_became_peered":"2024-03-06T07:35:30.042372+0000","last_unstale":"2024-03-06T07:35:31.077092+0000","last_undegraded":"2024-03-06T07:35:31.077092+0000","last_fullsized":"2024-03-06T07:35:31.077092+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:26:56.768103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.062704+0000","last_change":"2024-03-06T07:35:31.059301+0000","last_active":"2024-03-06T07:35:31.062704+0000","last_peered":"2024-03-06T07:35:31.062704+0000","last_clean":"2024-03-06T07:35:31.062704+0000","last_became_active":"2024-03-06T07:35:31.058997+0000","last_became_peered":"2024-03-06T07:35:31.058997+0000","last_unstale":"2024-03-06T07:35:31.062704+0000","last_undegraded":"2024-03-06T07:35:31.062704+0000","last_fullsized":"2024-03-06T07:35:31.062704+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:11.383786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.044927+0000","last_change":"2024-03-06T07:35:30.037994+0000","last_active":"2024-03-06T07:35:31.044927+0000","last_peered":"2024-03-06T07:35:31.044927+0000","last_clean":"2024-03-06T07:35:31.044927+0000","last_became_active":"2024-03-06T07:35:30.037707+0000","last_became_peered":"2024-03-06T07:35:30.037707+0000","last_unstale":"2024-03-06T07:35:31.044927+0000","last_undegraded":"2024-03-06T07:35:31.044927+0000","last_fullsized":"2024-03-06T07:35:31.044927+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:49:26.526141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.066224+0000","last_change":"2024-03-06T07:35:31.068777+0000","last_active":"2024-03-06T07:35:32.066224+0000","last_peered":"2024-03-06T07:35:32.066224+0000","last_clean":"2024-03-06T07:35:32.066224+0000","last_became_active":"2024-03-06T07:35:31.068452+0000","last_became_peered":"2024-03-06T07:35:31.068452+0000","last_unstale":"2024-03-06T07:35:32.066224+0000","last_undegraded":"2024-03-06T07:35:32.066224+0000","last_fullsized":"2024-03-06T07:35:32.066224+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:35.032048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.066271+0000","last_change":"2024-03-06T07:35:30.038961+0000","last_active":"2024-03-06T07:35:31.066271+0000","last_peered":"2024-03-06T07:35:31.066271+0000","last_clean":"2024-03-06T07:35:31.066271+0000","last_became_active":"2024-03-06T07:35:30.037332+0000","last_became_peered":"2024-03-06T07:35:30.037332+0000","last_unstale":"2024-03-06T07:35:31.066271+0000","last_undegraded":"2024-03-06T07:35:31.066271+0000","last_fullsized":"2024-03-06T07:35:31.066271+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:14:35.970244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.084580+0000","last_change":"2024-03-06T07:35:31.074176+0000","last_active":"2024-03-06T07:35:31.084580+0000","last_peered":"2024-03-06T07:35:31.084580+0000","last_clean":"2024-03-06T07:35:31.084580+0000","last_became_active":"2024-03-06T07:35:31.073787+0000","last_became_peered":"2024-03-06T07:35:31.073787+0000","last_unstale":"2024-03-06T07:35:31.084580+0000","last_undegraded":"2024-03-06T07:35:31.084580+0000","last_fullsized":"2024-03-06T07:35:31.084580+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:29:01.999085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.054975+0000","last_change":"2024-03-06T07:35:30.039564+0000","last_active":"2024-03-06T07:35:31.054975+0000","last_peered":"2024-03-06T07:35:31.054975+0000","last_clean":"2024-03-06T07:35:31.054975+0000","last_became_active":"2024-03-06T07:35:30.038889+0000","last_became_peered":"2024-03-06T07:35:30.038889+0000","last_unstale":"2024-03-06T07:35:31.054975+0000","last_undegraded":"2024-03-06T07:35:31.054975+0000","last_fullsized":"2024-03-06T07:35:31.054975+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:46.222281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092803+0000","last_change":"2024-03-06T07:35:31.086401+0000","last_active":"2024-03-06T07:35:31.092803+0000","last_peered":"2024-03-06T07:35:31.092803+0000","last_clean":"2024-03-06T07:35:31.092803+0000","last_became_active":"2024-03-06T07:35:31.081501+0000","last_became_peered":"2024-03-06T07:35:31.081501+0000","last_unstale":"2024-03-06T07:35:31.092803+0000","last_undegraded":"2024-03-06T07:35:31.092803+0000","last_fullsized":"2024-03-06T07:35:31.092803+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:53:25.782203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"125'3","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087111+0000","last_change":"2024-03-06T07:35:30.039183+0000","last_active":"2024-03-06T07:35:31.087111+0000","last_peered":"2024-03-06T07:35:31.087111+0000","last_clean":"2024-03-06T07:35:31.087111+0000","last_became_active":"2024-03-06T07:35:30.038896+0000","last_became_peered":"2024-03-06T07:35:30.038896+0000","last_unstale":"2024-03-06T07:35:31.087111+0000","last_undegraded":"2024-03-06T07:35:31.087111+0000","last_fullsized":"2024-03-06T07:35:31.087111+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:45.531697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.085089+0000","last_change":"2024-03-06T07:35:31.078854+0000","last_active":"2024-03-06T07:35:31.085089+0000","last_peered":"2024-03-06T07:35:31.085089+0000","last_clean":"2024-03-06T07:35:31.085089+0000","last_became_active":"2024-03-06T07:35:31.078379+0000","last_became_peered":"2024-03-06T07:35:31.078379+0000","last_unstale":"2024-03-06T07:35:31.085089+0000","last_undegraded":"2024-03-06T07:35:31.085089+0000","last_fullsized":"2024-03-06T07:35:31.085089+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:21:29.181896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"125'6","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.093516+0000","last_change":"2024-03-06T07:35:30.044610+0000","last_active":"2024-03-06T07:35:31.093516+0000","last_peered":"2024-03-06T07:35:31.093516+0000","last_clean":"2024-03-06T07:35:31.093516+0000","last_became_active":"2024-03-06T07:35:30.043914+0000","last_became_peered":"2024-03-06T07:35:30.043914+0000","last_unstale":"2024-03-06T07:35:31.093516+0000","last_undegraded":"2024-03-06T07:35:31.093516+0000","last_fullsized":"2024-03-06T07:35:31.093516+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:47.055895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065272+0000","last_change":"2024-03-06T07:35:31.069880+0000","last_active":"2024-03-06T07:35:32.065272+0000","last_peered":"2024-03-06T07:35:32.065272+0000","last_clean":"2024-03-06T07:35:32.065272+0000","last_became_active":"2024-03-06T07:35:31.069612+0000","last_became_peered":"2024-03-06T07:35:31.069612+0000","last_unstale":"2024-03-06T07:35:32.065272+0000","last_undegraded":"2024-03-06T07:35:32.065272+0000","last_fullsized":"2024-03-06T07:35:32.065272+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:03.451348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065611+0000","last_change":"2024-03-06T07:35:30.043265+0000","last_active":"2024-03-06T07:35:32.065611+0000","last_peered":"2024-03-06T07:35:32.065611+0000","last_clean":"2024-03-06T07:35:32.065611+0000","last_became_active":"2024-03-06T07:35:30.042032+0000","last_became_peered":"2024-03-06T07:35:30.042032+0000","last_unstale":"2024-03-06T07:35:32.065611+0000","last_undegraded":"2024-03-06T07:35:32.065611+0000","last_fullsized":"2024-03-06T07:35:32.065611+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:14.134645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.065713+0000","last_change":"2024-03-06T07:35:31.062921+0000","last_active":"2024-03-06T07:35:31.065713+0000","last_peered":"2024-03-06T07:35:31.065713+0000","last_clean":"2024-03-06T07:35:31.065713+0000","last_became_active":"2024-03-06T07:35:31.062239+0000","last_became_peered":"2024-03-06T07:35:31.062239+0000","last_unstale":"2024-03-06T07:35:31.065713+0000","last_undegraded":"2024-03-06T07:35:31.065713+0000","last_fullsized":"2024-03-06T07:35:31.065713+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:45.000491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.043891+0000","last_change":"2024-03-06T07:35:30.042279+0000","last_active":"2024-03-06T07:35:31.043891+0000","last_peered":"2024-03-06T07:35:31.043891+0000","last_clean":"2024-03-06T07:35:31.043891+0000","last_became_active":"2024-03-06T07:35:30.042019+0000","last_became_peered":"2024-03-06T07:35:30.042019+0000","last_unstale":"2024-03-06T07:35:31.043891+0000","last_undegraded":"2024-03-06T07:35:31.043891+0000","last_fullsized":"2024-03-06T07:35:31.043891+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:47.963038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.093058+0000","last_change":"2024-03-06T07:35:31.089153+0000","last_active":"2024-03-06T07:35:31.093058+0000","last_peered":"2024-03-06T07:35:31.093058+0000","last_clean":"2024-03-06T07:35:31.093058+0000","last_became_active":"2024-03-06T07:35:31.088663+0000","last_became_peered":"2024-03-06T07:35:31.088663+0000","last_unstale":"2024-03-06T07:35:31.093058+0000","last_undegraded":"2024-03-06T07:35:31.093058+0000","last_fullsized":"2024-03-06T07:35:31.093058+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:27.590275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"125'8","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086143+0000","last_change":"2024-03-06T07:35:30.039434+0000","last_active":"2024-03-06T07:35:31.086143+0000","last_peered":"2024-03-06T07:35:31.086143+0000","last_clean":"2024-03-06T07:35:31.086143+0000","last_became_active":"2024-03-06T07:35:30.039088+0000","last_became_peered":"2024-03-06T07:35:30.039088+0000","last_unstale":"2024-03-06T07:35:31.086143+0000","last_undegraded":"2024-03-06T07:35:31.086143+0000","last_fullsized":"2024-03-06T07:35:31.086143+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:08:33.676045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.082882+0000","last_change":"2024-03-06T07:35:31.080949+0000","last_active":"2024-03-06T07:35:31.082882+0000","last_peered":"2024-03-06T07:35:31.082882+0000","last_clean":"2024-03-06T07:35:31.082882+0000","last_became_active":"2024-03-06T07:35:31.078892+0000","last_became_peered":"2024-03-06T07:35:31.078892+0000","last_unstale":"2024-03-06T07:35:31.082882+0000","last_undegraded":"2024-03-06T07:35:31.082882+0000","last_fullsized":"2024-03-06T07:35:31.082882+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:08.438613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091952+0000","last_change":"2024-03-06T07:35:30.045104+0000","last_active":"2024-03-06T07:35:31.091952+0000","last_peered":"2024-03-06T07:35:31.091952+0000","last_clean":"2024-03-06T07:35:31.091952+0000","last_became_active":"2024-03-06T07:35:30.044457+0000","last_became_peered":"2024-03-06T07:35:30.044457+0000","last_unstale":"2024-03-06T07:35:31.091952+0000","last_undegraded":"2024-03-06T07:35:31.091952+0000","last_fullsized":"2024-03-06T07:35:31.091952+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:14.491709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.086445+0000","last_change":"2024-03-06T07:35:31.079643+0000","last_active":"2024-03-06T07:35:31.086445+0000","last_peered":"2024-03-06T07:35:31.086445+0000","last_clean":"2024-03-06T07:35:31.086445+0000","last_became_active":"2024-03-06T07:35:31.079100+0000","last_became_peered":"2024-03-06T07:35:31.079100+0000","last_unstale":"2024-03-06T07:35:31.086445+0000","last_undegraded":"2024-03-06T07:35:31.086445+0000","last_fullsized":"2024-03-06T07:35:31.086445+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:37:41.216222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.074100+0000","last_change":"2024-03-06T07:35:30.033713+0000","last_active":"2024-03-06T07:35:31.074100+0000","last_peered":"2024-03-06T07:35:31.074100+0000","last_clean":"2024-03-06T07:35:31.074100+0000","last_became_active":"2024-03-06T07:35:30.033418+0000","last_became_peered":"2024-03-06T07:35:30.033418+0000","last_unstale":"2024-03-06T07:35:31.074100+0000","last_undegraded":"2024-03-06T07:35:31.074100+0000","last_fullsized":"2024-03-06T07:35:31.074100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:28.422330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.077324+0000","last_change":"2024-03-06T07:35:31.075594+0000","last_active":"2024-03-06T07:35:31.077324+0000","last_peered":"2024-03-06T07:35:31.077324+0000","last_clean":"2024-03-06T07:35:31.077324+0000","last_became_active":"2024-03-06T07:35:31.075305+0000","last_became_peered":"2024-03-06T07:35:31.075305+0000","last_unstale":"2024-03-06T07:35:31.077324+0000","last_undegraded":"2024-03-06T07:35:31.077324+0000","last_fullsized":"2024-03-06T07:35:31.077324+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:21.766248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"125'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.083717+0000","last_change":"2024-03-06T07:35:30.037748+0000","last_active":"2024-03-06T07:35:31.083717+0000","last_peered":"2024-03-06T07:35:31.083717+0000","last_clean":"2024-03-06T07:35:31.083717+0000","last_became_active":"2024-03-06T07:35:30.037440+0000","last_became_peered":"2024-03-06T07:35:30.037440+0000","last_unstale":"2024-03-06T07:35:31.083717+0000","last_undegraded":"2024-03-06T07:35:31.083717+0000","last_fullsized":"2024-03-06T07:35:31.083717+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.807072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.072137+0000","last_change":"2024-03-06T07:35:31.070382+0000","last_active":"2024-03-06T07:35:31.072137+0000","last_peered":"2024-03-06T07:35:31.072137+0000","last_clean":"2024-03-06T07:35:31.072137+0000","last_became_active":"2024-03-06T07:35:31.069769+0000","last_became_peered":"2024-03-06T07:35:31.069769+0000","last_unstale":"2024-03-06T07:35:31.072137+0000","last_undegraded":"2024-03-06T07:35:31.072137+0000","last_fullsized":"2024-03-06T07:35:31.072137+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:47.502208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.080590+0000","last_change":"2024-03-06T07:35:30.035027+0000","last_active":"2024-03-06T07:35:31.080590+0000","last_peered":"2024-03-06T07:35:31.080590+0000","last_clean":"2024-03-06T07:35:31.080590+0000","last_became_active":"2024-03-06T07:35:30.034569+0000","last_became_peered":"2024-03-06T07:35:30.034569+0000","last_unstale":"2024-03-06T07:35:31.080590+0000","last_undegraded":"2024-03-06T07:35:31.080590+0000","last_fullsized":"2024-03-06T07:35:31.080590+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:40.243018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.065577+0000","last_change":"2024-03-06T07:35:31.082728+0000","last_active":"2024-03-06T07:35:32.065577+0000","last_peered":"2024-03-06T07:35:32.065577+0000","last_clean":"2024-03-06T07:35:32.065577+0000","last_became_active":"2024-03-06T07:35:31.082489+0000","last_became_peered":"2024-03-06T07:35:31.082489+0000","last_unstale":"2024-03-06T07:35:32.065577+0000","last_undegraded":"2024-03-06T07:35:32.065577+0000","last_fullsized":"2024-03-06T07:35:32.065577+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:00.324564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.091631+0000","last_change":"2024-03-06T07:35:30.035046+0000","last_active":"2024-03-06T07:35:31.091631+0000","last_peered":"2024-03-06T07:35:31.091631+0000","last_clean":"2024-03-06T07:35:31.091631+0000","last_became_active":"2024-03-06T07:35:30.033003+0000","last_became_peered":"2024-03-06T07:35:30.033003+0000","last_unstale":"2024-03-06T07:35:31.091631+0000","last_undegraded":"2024-03-06T07:35:31.091631+0000","last_fullsized":"2024-03-06T07:35:31.091631+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:19.891102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087972+0000","last_change":"2024-03-06T07:35:31.078553+0000","last_active":"2024-03-06T07:35:31.087972+0000","last_peered":"2024-03-06T07:35:31.087972+0000","last_clean":"2024-03-06T07:35:31.087972+0000","last_became_active":"2024-03-06T07:35:31.074804+0000","last_became_peered":"2024-03-06T07:35:31.074804+0000","last_unstale":"2024-03-06T07:35:31.087972+0000","last_undegraded":"2024-03-06T07:35:31.087972+0000","last_fullsized":"2024-03-06T07:35:31.087972+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:58:02.368678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.053786+0000","last_change":"2024-03-06T07:35:30.027631+0000","last_active":"2024-03-06T07:35:31.053786+0000","last_peered":"2024-03-06T07:35:31.053786+0000","last_clean":"2024-03-06T07:35:31.053786+0000","last_became_active":"2024-03-06T07:35:30.027364+0000","last_became_peered":"2024-03-06T07:35:30.027364+0000","last_unstale":"2024-03-06T07:35:31.053786+0000","last_undegraded":"2024-03-06T07:35:31.053786+0000","last_fullsized":"2024-03-06T07:35:31.053786+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:30.725663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.057391+0000","last_change":"2024-03-06T07:35:31.056358+0000","last_active":"2024-03-06T07:35:31.057391+0000","last_peered":"2024-03-06T07:35:31.057391+0000","last_clean":"2024-03-06T07:35:31.057391+0000","last_became_active":"2024-03-06T07:35:31.055546+0000","last_became_peered":"2024-03-06T07:35:31.055546+0000","last_unstale":"2024-03-06T07:35:31.057391+0000","last_undegraded":"2024-03-06T07:35:31.057391+0000","last_fullsized":"2024-03-06T07:35:31.057391+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:59:35.032048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.051788+0000","last_change":"2024-03-06T07:35:30.035330+0000","last_active":"2024-03-06T07:35:31.051788+0000","last_peered":"2024-03-06T07:35:31.051788+0000","last_clean":"2024-03-06T07:35:31.051788+0000","last_became_active":"2024-03-06T07:35:30.035058+0000","last_became_peered":"2024-03-06T07:35:30.035058+0000","last_unstale":"2024-03-06T07:35:31.051788+0000","last_undegraded":"2024-03-06T07:35:31.051788+0000","last_fullsized":"2024-03-06T07:35:31.051788+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:26.575137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092794+0000","last_change":"2024-03-06T07:35:31.090318+0000","last_active":"2024-03-06T07:35:31.092794+0000","last_peered":"2024-03-06T07:35:31.092794+0000","last_clean":"2024-03-06T07:35:31.092794+0000","last_became_active":"2024-03-06T07:35:31.089852+0000","last_became_peered":"2024-03-06T07:35:31.089852+0000","last_unstale":"2024-03-06T07:35:31.092794+0000","last_undegraded":"2024-03-06T07:35:31.092794+0000","last_fullsized":"2024-03-06T07:35:31.092794+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:43.283452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.051974+0000","last_change":"2024-03-06T07:35:30.031608+0000","last_active":"2024-03-06T07:35:31.051974+0000","last_peered":"2024-03-06T07:35:31.051974+0000","last_clean":"2024-03-06T07:35:31.051974+0000","last_became_active":"2024-03-06T07:35:30.031279+0000","last_became_peered":"2024-03-06T07:35:30.031279+0000","last_unstale":"2024-03-06T07:35:31.051974+0000","last_undegraded":"2024-03-06T07:35:31.051974+0000","last_fullsized":"2024-03-06T07:35:31.051974+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:06.393220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.089142+0000","last_change":"2024-03-06T07:35:31.086338+0000","last_active":"2024-03-06T07:35:31.089142+0000","last_peered":"2024-03-06T07:35:31.089142+0000","last_clean":"2024-03-06T07:35:31.089142+0000","last_became_active":"2024-03-06T07:35:31.085137+0000","last_became_peered":"2024-03-06T07:35:31.085137+0000","last_unstale":"2024-03-06T07:35:31.089142+0000","last_undegraded":"2024-03-06T07:35:31.089142+0000","last_fullsized":"2024-03-06T07:35:31.089142+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:14.075792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"125'2","reported_seq":16,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.067782+0000","last_change":"2024-03-06T07:35:30.037407+0000","last_active":"2024-03-06T07:35:32.067782+0000","last_peered":"2024-03-06T07:35:32.067782+0000","last_clean":"2024-03-06T07:35:32.067782+0000","last_became_active":"2024-03-06T07:35:30.036918+0000","last_became_peered":"2024-03-06T07:35:30.036918+0000","last_unstale":"2024-03-06T07:35:32.067782+0000","last_undegraded":"2024-03-06T07:35:32.067782+0000","last_fullsized":"2024-03-06T07:35:32.067782+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:38.906760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-03-06T07:35:32.067183+0000","last_change":"2024-03-06T07:35:31.081206+0000","last_active":"2024-03-06T07:35:32.067183+0000","last_peered":"2024-03-06T07:35:32.067183+0000","last_clean":"2024-03-06T07:35:32.067183+0000","last_became_active":"2024-03-06T07:35:31.077539+0000","last_became_peered":"2024-03-06T07:35:31.077539+0000","last_unstale":"2024-03-06T07:35:32.067183+0000","last_undegraded":"2024-03-06T07:35:32.067183+0000","last_fullsized":"2024-03-06T07:35:32.067183+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:06:11.383786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.063069+0000","last_change":"2024-03-06T07:35:30.047979+0000","last_active":"2024-03-06T07:35:31.063069+0000","last_peered":"2024-03-06T07:35:31.063069+0000","last_clean":"2024-03-06T07:35:31.063069+0000","last_became_active":"2024-03-06T07:35:30.047634+0000","last_became_peered":"2024-03-06T07:35:30.047634+0000","last_unstale":"2024-03-06T07:35:31.063069+0000","last_undegraded":"2024-03-06T07:35:31.063069+0000","last_fullsized":"2024-03-06T07:35:31.063069+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:12:05.747823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.079802+0000","last_change":"2024-03-06T07:35:31.078809+0000","last_active":"2024-03-06T07:35:31.079802+0000","last_peered":"2024-03-06T07:35:31.079802+0000","last_clean":"2024-03-06T07:35:31.079802+0000","last_became_active":"2024-03-06T07:35:31.078494+0000","last_became_peered":"2024-03-06T07:35:31.078494+0000","last_unstale":"2024-03-06T07:35:31.079802+0000","last_undegraded":"2024-03-06T07:35:31.079802+0000","last_fullsized":"2024-03-06T07:35:31.079802+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:56:44.825623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.045352+0000","last_change":"2024-03-06T07:35:30.041185+0000","last_active":"2024-03-06T07:35:31.045352+0000","last_peered":"2024-03-06T07:35:31.045352+0000","last_clean":"2024-03-06T07:35:31.045352+0000","last_became_active":"2024-03-06T07:35:30.040894+0000","last_became_peered":"2024-03-06T07:35:30.040894+0000","last_unstale":"2024-03-06T07:35:31.045352+0000","last_undegraded":"2024-03-06T07:35:31.045352+0000","last_fullsized":"2024-03-06T07:35:31.045352+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:38:59.309426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092631+0000","last_change":"2024-03-06T07:35:31.091436+0000","last_active":"2024-03-06T07:35:31.092631+0000","last_peered":"2024-03-06T07:35:31.092631+0000","last_clean":"2024-03-06T07:35:31.092631+0000","last_became_active":"2024-03-06T07:35:31.090996+0000","last_became_peered":"2024-03-06T07:35:31.090996+0000","last_unstale":"2024-03-06T07:35:31.092631+0000","last_undegraded":"2024-03-06T07:35:31.092631+0000","last_fullsized":"2024-03-06T07:35:31.092631+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:48:39.309794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.056340+0000","last_change":"2024-03-06T07:35:30.024276+0000","last_active":"2024-03-06T07:35:31.056340+0000","last_peered":"2024-03-06T07:35:31.056340+0000","last_clean":"2024-03-06T07:35:31.056340+0000","last_became_active":"2024-03-06T07:35:30.023889+0000","last_became_peered":"2024-03-06T07:35:30.023889+0000","last_unstale":"2024-03-06T07:35:31.056340+0000","last_undegraded":"2024-03-06T07:35:31.056340+0000","last_fullsized":"2024-03-06T07:35:31.056340+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:49.789801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.087763+0000","last_change":"2024-03-06T07:35:31.079785+0000","last_active":"2024-03-06T07:35:31.087763+0000","last_peered":"2024-03-06T07:35:31.087763+0000","last_clean":"2024-03-06T07:35:31.087763+0000","last_became_active":"2024-03-06T07:35:31.079498+0000","last_became_peered":"2024-03-06T07:35:31.079498+0000","last_unstale":"2024-03-06T07:35:31.087763+0000","last_undegraded":"2024-03-06T07:35:31.087763+0000","last_fullsized":"2024-03-06T07:35:31.087763+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:22:44.399397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.046171+0000","last_change":"2024-03-06T07:35:30.047622+0000","last_active":"2024-03-06T07:35:31.046171+0000","last_peered":"2024-03-06T07:35:31.046171+0000","last_clean":"2024-03-06T07:35:31.046171+0000","last_became_active":"2024-03-06T07:35:30.047294+0000","last_became_peered":"2024-03-06T07:35:30.047294+0000","last_unstale":"2024-03-06T07:35:31.046171+0000","last_undegraded":"2024-03-06T07:35:31.046171+0000","last_fullsized":"2024-03-06T07:35:31.046171+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:26.408750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.088189+0000","last_change":"2024-03-06T07:35:31.082980+0000","last_active":"2024-03-06T07:35:31.088189+0000","last_peered":"2024-03-06T07:35:31.088189+0000","last_clean":"2024-03-06T07:35:31.088189+0000","last_became_active":"2024-03-06T07:35:31.081956+0000","last_became_peered":"2024-03-06T07:35:31.081956+0000","last_unstale":"2024-03-06T07:35:31.088189+0000","last_undegraded":"2024-03-06T07:35:31.088189+0000","last_fullsized":"2024-03-06T07:35:31.088189+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:58.139075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.061453+0000","last_change":"2024-03-06T07:35:30.047816+0000","last_active":"2024-03-06T07:35:31.061453+0000","last_peered":"2024-03-06T07:35:31.061453+0000","last_clean":"2024-03-06T07:35:31.061453+0000","last_became_active":"2024-03-06T07:35:30.047450+0000","last_became_peered":"2024-03-06T07:35:30.047450+0000","last_unstale":"2024-03-06T07:35:31.061453+0000","last_undegraded":"2024-03-06T07:35:31.061453+0000","last_fullsized":"2024-03-06T07:35:31.061453+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:11.649223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.069130+0000","last_change":"2024-03-06T07:35:31.065889+0000","last_active":"2024-03-06T07:35:31.069130+0000","last_peered":"2024-03-06T07:35:31.069130+0000","last_clean":"2024-03-06T07:35:31.069130+0000","last_became_active":"2024-03-06T07:35:31.063697+0000","last_became_peered":"2024-03-06T07:35:31.063697+0000","last_unstale":"2024-03-06T07:35:31.069130+0000","last_undegraded":"2024-03-06T07:35:31.069130+0000","last_fullsized":"2024-03-06T07:35:31.069130+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:08:30.052004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.053782+0000","last_change":"2024-03-06T07:35:30.027364+0000","last_active":"2024-03-06T07:35:31.053782+0000","last_peered":"2024-03-06T07:35:31.053782+0000","last_clean":"2024-03-06T07:35:31.053782+0000","last_became_active":"2024-03-06T07:35:30.027059+0000","last_became_peered":"2024-03-06T07:35:30.027059+0000","last_unstale":"2024-03-06T07:35:31.053782+0000","last_undegraded":"2024-03-06T07:35:31.053782+0000","last_fullsized":"2024-03-06T07:35:31.053782+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:45.675494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.075733+0000","last_change":"2024-03-06T07:35:31.074790+0000","last_active":"2024-03-06T07:35:31.075733+0000","last_peered":"2024-03-06T07:35:31.075733+0000","last_clean":"2024-03-06T07:35:31.075733+0000","last_became_active":"2024-03-06T07:35:31.073549+0000","last_became_peered":"2024-03-06T07:35:31.073549+0000","last_unstale":"2024-03-06T07:35:31.075733+0000","last_undegraded":"2024-03-06T07:35:31.075733+0000","last_fullsized":"2024-03-06T07:35:31.075733+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:26.378562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.054683+0000","last_change":"2024-03-06T07:35:30.037878+0000","last_active":"2024-03-06T07:35:31.054683+0000","last_peered":"2024-03-06T07:35:31.054683+0000","last_clean":"2024-03-06T07:35:31.054683+0000","last_became_active":"2024-03-06T07:35:30.037610+0000","last_became_peered":"2024-03-06T07:35:30.037610+0000","last_unstale":"2024-03-06T07:35:31.054683+0000","last_undegraded":"2024-03-06T07:35:31.054683+0000","last_fullsized":"2024-03-06T07:35:31.054683+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:27:23.732107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.083185+0000","last_change":"2024-03-06T07:35:31.072067+0000","last_active":"2024-03-06T07:35:31.083185+0000","last_peered":"2024-03-06T07:35:31.083185+0000","last_clean":"2024-03-06T07:35:31.083185+0000","last_became_active":"2024-03-06T07:35:31.071608+0000","last_became_peered":"2024-03-06T07:35:31.071608+0000","last_unstale":"2024-03-06T07:35:31.083185+0000","last_undegraded":"2024-03-06T07:35:31.083185+0000","last_fullsized":"2024-03-06T07:35:31.083185+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:30.000959+0000","last_clean_scrub_stamp":"2024-03-06T07:35:30.000959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:19:14.075792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"125'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.092134+0000","last_change":"2024-03-06T07:35:30.047830+0000","last_active":"2024-03-06T07:35:31.092134+0000","last_peered":"2024-03-06T07:35:31.092134+0000","last_clean":"2024-03-06T07:35:31.092134+0000","last_became_active":"2024-03-06T07:35:30.047425+0000","last_became_peered":"2024-03-06T07:35:30.047425+0000","last_unstale":"2024-03-06T07:35:31.092134+0000","last_undegraded":"2024-03-06T07:35:31.092134+0000","last_fullsized":"2024-03-06T07:35:31.092134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:35:28.985822+0000","last_clean_scrub_stamp":"2024-03-06T07:35:28.985822+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:18:39.610286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":355,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:35:31.058885+0000","last_change":"2024-03-06T07:27:19.331230+0000","last_active":"2024-03-06T07:35:31.058885+0000","last_peered":"2024-03-06T07:35:31.058885+0000","last_clean":"2024-03-06T07:35:31.058885+0000","last_became_active":"2024-03-06T07:27:19.327789+0000","last_became_peered":"2024-03-06T07:27:19.327789+0000","last_unstale":"2024-03-06T07:35:31.058885+0000","last_undegraded":"2024-03-06T07:35:31.058885+0000","last_fullsized":"2024-03-06T07:35:31.058885+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:17.953672+0000","last_clean_scrub_stamp":"2024-03-06T07:27:17.953672+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:30:41.892633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":15,"seq":64424509541,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":673558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509542,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":672942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542245,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41628,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330212,"statfs":{"total":96636764160,"available":96594137088,"internally_reserved":0,"allocated":2170880,"data_stored":1265560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41008,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330832,"statfs":{"total":96636764160,"available":96594771968,"internally_reserved":0,"allocated":1540096,"data_stored":666167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542246,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":673054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41028,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330812,"statfs":{"total":96636764160,"available":96594751488,"internally_reserved":0,"allocated":1556480,"data_stored":673012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542246,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36848,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334992,"statfs":{"total":96636764160,"available":96599031808,"internally_reserved":0,"allocated":1540096,"data_stored":666167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37428,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334412,"statfs":{"total":96636764160,"available":96598437888,"internally_reserved":0,"allocated":2134016,"data_stored":1256535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:35:36.158 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:35:36.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:35:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.416+0000 7f5dc6000640 1 Processor -- start 2024-03-06T07:35:36.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.416+0000 7f5dc6000640 1 -- start start 2024-03-06T07:35:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc6000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f5dc0144ad0 0x7f5dc0145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc6000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc00d1870 0x7f5dc006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc6000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5dc01457c0 0x7f5dc006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc6000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc00d0860 con 0x7f5dc01457c0 2024-03-06T07:35:36.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc6000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc00d13e0 con 0x7f5dc0144ad0 2024-03-06T07:35:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc6000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00d0ab0 con 0x7f5dc00d1870 2024-03-06T07:35:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc5600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5dc01457c0 0x7f5dc006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc5600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5dc01457c0 0x7f5dc006b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39456/0 (socket says 172.21.15.6:39456) 2024-03-06T07:35:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc5600640 1 -- 172.21.15.6:0/3860012232 learned_addr learned my addr 172.21.15.6:0/3860012232 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dc4c00640 1 --2- 172.21.15.6:0/3860012232 >> v2:172.21.15.178:3300/0 conn(0x7f5dc0144ad0 0x7f5dc0145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.418+0000 7f5dbfe00640 1 --2- 172.21.15.6:0/3860012232 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc00d1870 0x7f5dc006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.419+0000 7f5dc5600640 1 -- 172.21.15.6:0/3860012232 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc00d1870 msgr2=0x7f5dc006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.419+0000 7f5dc5600640 1 --2- 172.21.15.6:0/3860012232 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc00d1870 0x7f5dc006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.419+0000 7f5dc5600640 1 -- 172.21.15.6:0/3860012232 >> v2:172.21.15.178:3300/0 conn(0x7f5dc0144ad0 msgr2=0x7f5dc0145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.419+0000 7f5dc5600640 1 --2- 172.21.15.6:0/3860012232 >> v2:172.21.15.178:3300/0 conn(0x7f5dc0144ad0 0x7f5dc0145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.419+0000 7f5dc5600640 1 -- 172.21.15.6:0/3860012232 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00d0d90 con 0x7f5dc01457c0 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.419+0000 7f5dc4c00640 1 --2- 172.21.15.6:0/3860012232 >> v2:172.21.15.178:3300/0 conn(0x7f5dc0144ad0 0x7f5dc0145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:36.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.420+0000 7f5dc5600640 1 --2- 172.21.15.6:0/3860012232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc01457c0 0x7f5dc006b4f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f5dc006c090 tx=0x7f5db802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a42044ddf611cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.420+0000 7f5dbf400640 1 -- 172.21.15.6:0/3860012232 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db802ca70 con 0x7f5dc01457c0 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.420+0000 7f5dbf400640 1 -- 172.21.15.6:0/3860012232 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db802cc10 con 0x7f5dc01457c0 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.420+0000 7f5dbf400640 1 -- 172.21.15.6:0/3860012232 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db80343f0 con 0x7f5dc01457c0 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 -- 172.21.15.6:0/3860012232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc01457c0 msgr2=0x7f5dc006b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 --2- 172.21.15.6:0/3860012232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc01457c0 0x7f5dc006b4f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f5dc006c090 tx=0x7f5db802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 -- 172.21.15.6:0/3860012232 shutdown_connections 2024-03-06T07:35:36.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 --2- 172.21.15.6:0/3860012232 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc01457c0 0x7f5dc006b4f0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 --2- 172.21.15.6:0/3860012232 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc00d1870 0x7f5dc006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 --2- 172.21.15.6:0/3860012232 >> v2:172.21.15.178:3300/0 conn(0x7f5dc0144ad0 0x7f5dc0145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 -- 172.21.15.6:0/3860012232 >> 172.21.15.6:0/3860012232 conn(0x7f5dc006c0c0 msgr2=0x7f5dc006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 -- 172.21.15.6:0/3860012232 shutdown_connections 2024-03-06T07:35:36.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.421+0000 7f5dc6000640 1 -- 172.21.15.6:0/3860012232 wait complete. 2024-03-06T07:35:36.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.423+0000 7f5dc6000640 1 Processor -- start 2024-03-06T07:35:36.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.423+0000 7f5dc6000640 1 -- start start 2024-03-06T07:35:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc6000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc0144ad0 0x7f5dc0139670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc6000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dc01457c0 0x7f5dc015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc6000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 0x7f5dc015d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.427 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc6000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00cd9a0 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc6000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00cd820 con 0x7f5dc01457c0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc6000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0144eb0 con 0x7f5dc0144ad0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 0x7f5dc015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc5600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 0x7f5dc015d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39466/0 (socket says 172.21.15.6:39466) 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc5600640 1 -- 172.21.15.6:0/2083732122 learned_addr learned my addr 172.21.15.6:0/2083732122 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dc4c00640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc0144ad0 0x7f5dc0139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.424+0000 7f5dbfe00640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dc01457c0 0x7f5dc015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc5600640 1 -- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc0144ad0 msgr2=0x7f5dc0139670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc5600640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc0144ad0 0x7f5dc0139670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc5600640 1 -- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dc01457c0 msgr2=0x7f5dc015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc5600640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dc01457c0 0x7f5dc015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc5600640 1 -- 172.21.15.6:0/2083732122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00c5fb0 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dbfe00640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dc01457c0 0x7f5dc015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc5600640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 0x7f5dc015d740 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f5db8036040 tx=0x7f5db8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8043020 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db803e450 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc0141260 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db8009510 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.425+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc0141820 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.427+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc004f330 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.428+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5db8046020 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.429+0000 7f5dbcc00640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5da006b500 0x7f5da006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.429+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5db80aca50 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.429+0000 7f5dc4c00640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5da006b500 0x7f5da006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.431+0000 7f5dc4c00640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5da006b500 0x7f5da006d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00cd740 tx=0x7f5db4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.433+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5db804d030 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.709+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5dc00722d0 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.713+0000 7f5dbcc00640 1 -- 172.21.15.6:0/2083732122 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5db807b830 con 0x7f5dc00c5ad0 2024-03-06T07:35:36.715 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-06T07:35:36.715 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCiGuhlsyN5DRAAQtL7zbCweBYVp24FZDizmQ== 2024-03-06T07:35:36.716 INFO:teuthology.orchestra.run.smithi006.stdout: caps mds = "allow" 2024-03-06T07:35:36.716 INFO:teuthology.orchestra.run.smithi006.stdout: caps mon = "allow r" 2024-03-06T07:35:36.716 INFO:teuthology.orchestra.run.smithi006.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:35:36.716 INFO:teuthology.orchestra.run.smithi006.stderr:updated caps for client.0 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5da006b500 msgr2=0x7f5da006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5da006b500 0x7f5da006d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00cd740 tx=0x7f5db4008040 comp rx=0 tx=0).stop 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 msgr2=0x7f5dc015d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 0x7f5dc015d740 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f5db8036040 tx=0x7f5db8004a40 comp rx=0 tx=0).stop 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 shutdown_connections 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5da006b500 0x7f5da006d9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5dc01457c0 0x7f5dc015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5dc00c5ad0 0x7f5dc015d740 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 --2- 172.21.15.6:0/2083732122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5dc0144ad0 0x7f5dc0139670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 >> 172.21.15.6:0/2083732122 conn(0x7f5dc006c0c0 msgr2=0x7f5dc00724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 shutdown_connections 2024-03-06T07:35:36.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.716+0000 7f5dc6000640 1 -- 172.21.15.6:0/2083732122 wait complete. 2024-03-06T07:35:36.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:36.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.945+0000 7f3637200640 1 Processor -- start 2024-03-06T07:35:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.946+0000 7f3637200640 1 -- start start 2024-03-06T07:35:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.948+0000 7f3637200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3630162420 0x7f363006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.948+0000 7f3637200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 0x7f36300d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.948+0000 7f3637200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630147bc0 con 0x7f36301477e0 2024-03-06T07:35:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.948+0000 7f3637200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363006b960 con 0x7f3630162420 2024-03-06T07:35:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.948+0000 7f3637200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3630161f90 con 0x7f363006b4f0 2024-03-06T07:35:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 0x7f36300d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 0x7f36300d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:36998/0 (socket says 172.21.15.6:36998) 2024-03-06T07:35:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 -- 172.21.15.6:0/3988349142 learned_addr learned my addr 172.21.15.6:0/3988349142 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3635400640 1 --2- 172.21.15.6:0/3988349142 >> v2:172.21.15.178:3300/0 conn(0x7f3630162420 0x7f363006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 -- 172.21.15.6:0/3988349142 >> v2:172.21.15.178:3300/0 conn(0x7f3630162420 msgr2=0x7f363006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 --2- 172.21.15.6:0/3988349142 >> v2:172.21.15.178:3300/0 conn(0x7f3630162420 0x7f363006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 -- 172.21.15.6:0/3988349142 >> v1:172.21.15.178:6789/0 conn(0x7f36301477e0 legacy=0x7f363006ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.949+0000 7f3636800640 1 -- 172.21.15.6:0/3988349142 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300b3d70 con 0x7f363006b4f0 2024-03-06T07:35:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.951+0000 7f3636800640 1 --2- 172.21.15.6:0/3988349142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 0x7f36300d0910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f362c009e50 tx=0x7f362c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ea0ca95c226c96f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3634a00640 1 -- 172.21.15.6:0/3988349142 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c02ca70 con 0x7f363006b4f0 2024-03-06T07:35:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3634a00640 1 -- 172.21.15.6:0/3988349142 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c02cc10 con 0x7f363006b4f0 2024-03-06T07:35:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3634a00640 1 -- 172.21.15.6:0/3988349142 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0343f0 con 0x7f363006b4f0 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3637200640 1 -- 172.21.15.6:0/3988349142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 msgr2=0x7f36300d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3637200640 1 --2- 172.21.15.6:0/3988349142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 0x7f36300d0910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f362c009e50 tx=0x7f362c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3637200640 1 -- 172.21.15.6:0/3988349142 shutdown_connections 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3637200640 1 --2- 172.21.15.6:0/3988349142 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f363006b4f0 0x7f36300d0910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3637200640 1 --2- 172.21.15.6:0/3988349142 >> v2:172.21.15.178:3300/0 conn(0x7f3630162420 0x7f363006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.952+0000 7f3637200640 1 -- 172.21.15.6:0/3988349142 >> 172.21.15.6:0/3988349142 conn(0x7f363006c0c0 msgr2=0x7f363006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.953+0000 7f3637200640 1 -- 172.21.15.6:0/3988349142 shutdown_connections 2024-03-06T07:35:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.953+0000 7f3637200640 1 -- 172.21.15.6:0/3988349142 wait complete. 2024-03-06T07:35:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.954+0000 7f3637200640 1 Processor -- start 2024-03-06T07:35:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.954+0000 7f3637200640 1 -- start start 2024-03-06T07:35:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.955+0000 7f3637200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f363006b4f0 0x7f363013d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.955+0000 7f3637200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 0x7f3630140360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.955+0000 7f3637200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3630162420 0x7f36301408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.955+0000 7f3637200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3630161c20 con 0x7f36301477e0 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.955+0000 7f3637200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36300b4330 con 0x7f363006b4f0 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.955+0000 7f3637200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36301628a0 con 0x7f3630162420 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 0x7f3630140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 0x7f3630140360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39476/0 (socket says 172.21.15.6:39476) 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 -- 172.21.15.6:0/382159919 learned_addr learned my addr 172.21.15.6:0/382159919 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635e00640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f363006b4f0 0x7f363013d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3636800640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3630162420 0x7f36301408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 -- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3630162420 msgr2=0x7f36301408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3630162420 0x7f36301408a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 -- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f363006b4f0 msgr2=0x7f363013d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f363006b4f0 0x7f363013d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.956+0000 7f3635400640 1 -- 172.21.15.6:0/382159919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630140ed0 con 0x7f36301477e0 2024-03-06T07:35:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.957+0000 7f3635400640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 0x7f3630140360 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f362000e870 tx=0x7f362000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.957+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620017070 con 0x7f36301477e0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.957+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362000c940 con 0x7f36301477e0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.957+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362000cd30 con 0x7f36301477e0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.957+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36301411b0 con 0x7f36301477e0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.957+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3630141770 con 0x7f36301477e0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.958+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363004f330 con 0x7f36301477e0 2024-03-06T07:35:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.959+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f362001a070 con 0x7f36301477e0 2024-03-06T07:35:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.961+0000 7f362a000640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.961+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f362008a520 con 0x7f36301477e0 2024-03-06T07:35:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.962+0000 7f3635e00640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:36.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.963+0000 7f3635e00640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361406b4d0 0x7f361406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3624000f60 tx=0x7f3624009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:36.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:36.964+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3620059300 con 0x7f36301477e0 2024-03-06T07:35:37.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.207+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36300724b0 con 0x7f36301477e0 2024-03-06T07:35:37.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.208+0000 7f362a000640 1 -- 172.21.15.6:0/382159919 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f3620058c00 con 0x7f36301477e0 2024-03-06T07:35:37.210 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:37.210 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":92,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1518219931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1518219931},{"type":"v1","addr":"172.21.15.178:6837","nonce":1518219931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/1688442844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1688442844},{"type":"v1","addr":"172.21.15.178:6836","nonce":1688442844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6871,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:33.596622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8355},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:35:37.210 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 92 2024-03-06T07:35:37.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361406b4d0 msgr2=0x7f361406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361406b4d0 0x7f361406d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3624000f60 tx=0x7f3624009040 comp rx=0 tx=0).stop 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 msgr2=0x7f3630140360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 0x7f3630140360 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f362000e870 tx=0x7f362000ed30 comp rx=0 tx=0).stop 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 shutdown_connections 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f361406b4d0 0x7f361406d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3630162420 0x7f36301408a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f36301477e0 0x7f3630140360 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 --2- 172.21.15.6:0/382159919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f363006b4f0 0x7f363013d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.210+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 >> 172.21.15.6:0/382159919 conn(0x7f363006c0c0 msgr2=0x7f36300b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.211+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 shutdown_connections 2024-03-06T07:35:37.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.211+0000 7f3637200640 1 -- 172.21.15.6:0/382159919 wait complete. 2024-03-06T07:35:37.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:33.596622+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8355}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:35:37.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:35:37.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:35:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.861+0000 7fbf47c00640 1 Processor -- start 2024-03-06T07:35:37.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.861+0000 7fbf47c00640 1 -- start start 2024-03-06T07:35:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.863+0000 7fbf47c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4006c6d0 0x7fbf40162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.863+0000 7fbf47c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbf4015e0e0 0x7fbf4015e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.863+0000 7fbf47c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf40161a80 con 0x7fbf4015dd00 2024-03-06T07:35:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.863+0000 7fbf47c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4015efa0 con 0x7fbf4015e0e0 2024-03-06T07:35:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.863+0000 7fbf47c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4015f280 con 0x7fbf4006c6d0 2024-03-06T07:35:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.864+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4006c6d0 0x7fbf40162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.864+0000 7fbf45200640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fbf4015dd00 0x7fbf4006bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:51318/0 (socket says 172.21.15.6:51318) 2024-03-06T07:35:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.864+0000 7fbf45200640 1 -- 172.21.15.6:0/1587189980 learned_addr learned my addr 172.21.15.6:0/1587189980 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.864+0000 7fbf45c00640 1 --2- 172.21.15.6:0/1587189980 >> v2:172.21.15.178:3300/0 conn(0x7fbf4015e0e0 0x7fbf4015e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.865+0000 7fbf3fe00640 1 -- 172.21.15.6:0/1587189980 >> v2:172.21.15.178:3300/0 conn(0x7fbf4015e0e0 msgr2=0x7fbf4015e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.865+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/1587189980 >> v2:172.21.15.178:3300/0 conn(0x7fbf4015e0e0 0x7fbf4015e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.865+0000 7fbf3fe00640 1 -- 172.21.15.6:0/1587189980 >> v1:172.21.15.178:6789/0 conn(0x7fbf4015dd00 legacy=0x7fbf4006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.865+0000 7fbf3fe00640 1 -- 172.21.15.6:0/1587189980 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40161d20 con 0x7fbf4006c6d0 2024-03-06T07:35:37.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.866+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/1587189980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4006c6d0 0x7fbf40162520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0097a0 tx=0x7fbf2c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f577b22f34e3b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.867+0000 7fbf3f400640 1 -- 172.21.15.6:0/1587189980 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c035020 con 0x7fbf4006c6d0 2024-03-06T07:35:37.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.867+0000 7fbf3f400640 1 -- 172.21.15.6:0/1587189980 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c031810 con 0x7fbf4006c6d0 2024-03-06T07:35:37.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.867+0000 7fbf3f400640 1 -- 172.21.15.6:0/1587189980 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c031be0 con 0x7fbf4006c6d0 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.867+0000 7fbf47c00640 1 -- 172.21.15.6:0/1587189980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4006c6d0 msgr2=0x7fbf40162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.867+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1587189980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4006c6d0 0x7fbf40162520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0097a0 tx=0x7fbf2c02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.868+0000 7fbf47c00640 1 -- 172.21.15.6:0/1587189980 shutdown_connections 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.868+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1587189980 >> v2:172.21.15.178:3300/0 conn(0x7fbf4015e0e0 0x7fbf4015e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.868+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1587189980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4006c6d0 0x7fbf40162520 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.868+0000 7fbf47c00640 1 -- 172.21.15.6:0/1587189980 >> 172.21.15.6:0/1587189980 conn(0x7fbf400714d0 msgr2=0x7fbf400718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.868+0000 7fbf47c00640 1 -- 172.21.15.6:0/1587189980 shutdown_connections 2024-03-06T07:35:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.868+0000 7fbf47c00640 1 -- 172.21.15.6:0/1587189980 wait complete. 2024-03-06T07:35:37.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.870+0000 7fbf47c00640 1 Processor -- start 2024-03-06T07:35:37.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.870+0000 7fbf47c00640 1 -- start start 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.871+0000 7fbf47c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 0x7fbf40157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.871+0000 7fbf47c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf4015dd00 0x7fbf40154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.871+0000 7fbf47c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4015e0e0 0x7fbf400cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.871+0000 7fbf47c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4015ea70 con 0x7fbf4006c6d0 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.871+0000 7fbf47c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4006f280 con 0x7fbf4015dd00 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.871+0000 7fbf47c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4015ebf0 con 0x7fbf4015e0e0 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 0x7fbf40157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 0x7fbf40157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39490/0 (socket says 172.21.15.6:39490) 2024-03-06T07:35:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 -- 172.21.15.6:0/1935672097 learned_addr learned my addr 172.21.15.6:0/1935672097 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4015e0e0 0x7fbf400cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf4015dd00 0x7fbf40154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 -- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4015e0e0 msgr2=0x7fbf400cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4015e0e0 0x7fbf400cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 -- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf4015dd00 msgr2=0x7fbf40154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf4015dd00 0x7fbf40154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45200640 1 -- 172.21.15.6:0/1935672097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40157fc0 con 0x7fbf4006c6d0 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.872+0000 7fbf45c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4015e0e0 0x7fbf400cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.873+0000 7fbf45200640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 0x7fbf40157a80 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400c400 tx=0x7fbf3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.873+0000 7fbf3cc00640 1 -- 172.21.15.6:0/1935672097 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34017070 con 0x7fbf4006c6d0 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.873+0000 7fbf3cc00640 1 -- 172.21.15.6:0/1935672097 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34005ce0 con 0x7fbf4006c6d0 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.873+0000 7fbf3cc00640 1 -- 172.21.15.6:0/1935672097 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34007310 con 0x7fbf4006c6d0 2024-03-06T07:35:37.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.873+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf40156cc0 con 0x7fbf4006c6d0 2024-03-06T07:35:37.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.873+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf40157280 con 0x7fbf4006c6d0 2024-03-06T07:35:37.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.876+0000 7fbf3cc00640 1 -- 172.21.15.6:0/1935672097 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf3401a070 con 0x7fbf4006c6d0 2024-03-06T07:35:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.877+0000 7fbf3cc00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf2006b4d0 0x7fbf2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.877+0000 7fbf3cc00640 1 -- 172.21.15.6:0/1935672097 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf34089610 con 0x7fbf4006c6d0 2024-03-06T07:35:37.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.877+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf2006b4d0 0x7fbf2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.879+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf2006b4d0 0x7fbf2006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c036000 tx=0x7fbf2c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.879+0000 7fbf4a86a740 1 Processor -- start 2024-03-06T07:35:37.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.879+0000 7fbf4a86a740 1 -- start start 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff60d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff63d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff60dc90 0x5557ff63da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557ff577600 con 0x5557ff63a7b0 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557ff520930 con 0x5557ff60dc90 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5557ff5736a0 con 0x5557ff63ab90 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf45200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff60d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf45200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff60d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39506/0 (socket says 172.21.15.6:39506) 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf45200640 1 -- 172.21.15.6:0/4166050357 learned_addr learned my addr 172.21.15.6:0/4166050357 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.881+0000 7fbf45c00640 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff60dc90 0x5557ff63da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff63d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf45200640 1 -- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 msgr2=0x5557ff63d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf45200640 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff63d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf45200640 1 -- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff60dc90 msgr2=0x5557ff63da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf45200640 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff60dc90 0x5557ff63da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf45200640 1 -- 172.21.15.6:0/4166050357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557ff28c2d0 con 0x5557ff63a7b0 2024-03-06T07:35:37.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf45200640 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff60d750 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fbf340070a0 tx=0x7fbf34057d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfd199bbb44a9de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf31600640 1 -- 172.21.15.6:0/4166050357 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34055e20 con 0x5557ff63a7b0 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf31600640 1 -- 172.21.15.6:0/4166050357 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34056080 con 0x5557ff63a7b0 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.882+0000 7fbf31600640 1 -- 172.21.15.6:0/4166050357 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340543f0 con 0x5557ff63a7b0 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4166050357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 msgr2=0x5557ff60d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff60d750 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fbf340070a0 tx=0x7fbf34057d80 comp rx=0 tx=0).stop 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4166050357 shutdown_connections 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff60dc90 0x5557ff63da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff63d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4166050357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff60d750 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.883+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4166050357 >> 172.21.15.6:0/4166050357 conn(0x5557ff60f9c0 msgr2=0x5557ff658540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.884+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4166050357 shutdown_connections 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.884+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4166050357 wait complete. 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.884+0000 7fbf4a86a740 1 -- start start 2024-03-06T07:35:37.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.885+0000 7fbf4a86a740 1 Processor -- start 2024-03-06T07:35:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.885+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff640160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.885+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63ab90 0x5557ff633490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6339d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557ff577600 con 0x5557ff63a7b0 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557ff533d90 con 0x5557ff63ab90 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5557ff611f60 con 0x5557ff641160 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf4a86a740 10 client.0 _ll_register_callbacks cb 0x5557ff56f350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff640160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff640160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39512/0 (socket says 172.21.15.6:39512) 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 -- 172.21.15.6:0/460537176 learned_addr learned my addr 172.21.15.6:0/460537176 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 -- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 msgr2=0x5557ff6339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63ab90 0x5557ff633490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45c00640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6339d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 -- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63ab90 msgr2=0x5557ff633490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63ab90 0x5557ff633490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf45200640 1 -- 172.21.15.6:0/460537176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557ff3566b0 con 0x5557ff63a7b0 2024-03-06T07:35:37.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.886+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63ab90 0x5557ff633490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf45200640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff640160 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbf3405c090 tx=0x7fbf3401e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b16213a078a7ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf26000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf26000640 1 -- 172.21.15.6:0/460537176 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340074b0 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf26000640 1 -- 172.21.15.6:0/460537176 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf34007650 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf34056080 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf26000640 1 -- 172.21.15.6:0/460537176 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf340571d0 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf4a86a740 10 client.8481 fetch_fsmap learned FSMap version 92 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5557ff63c2f0 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf26000640 1 -- 172.21.15.6:0/460537176 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34056080 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.887+0000 7fbf26000640 1 -- 172.21.15.6:0/460537176 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbf34057370 con 0x5557ff63a7b0 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf4a86a740 10 client.8481 fetch_fsmap finished waiting for FSMap version 92 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf4a86a740 10 client.8481 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf4a86a740 20 client.8481 populate_metadata read hostname 'smithi006' 2024-03-06T07:35:37.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff63ce70 0x5557ff63d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf4a86a740 4 client.8481 mds_command: new command op to 8355 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:35:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 --> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557ff584b40 con 0x5557ff63ce70 2024-03-06T07:35:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff63ce70 0x5557ff63d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.888+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff63ce70 0x5557ff63d250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96f3900fc5223b02 server_cookie=e8c4c9057bc3acfd in_seq=0 out_seq=0 2024-03-06T07:35:37.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.889+0000 7fbf26000640 10 client.8481 ms_handle_connect on v2:172.21.15.6:6832/2669115160 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.889+0000 7fbf26000640 1 -- 172.21.15.6:0/460537176 <== mds.0 v2:172.21.15.6:6832/2669115160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5557ff584b40 con 0x5557ff63ce70 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.889+0000 7fbf26000640 10 client.8481 handle_command_reply: tid=0 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.889+0000 7fbf4a86a740 1 client.8481 shutdown 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 msgr2=0x5557ff640160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff640160 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbf3405c090 tx=0x7fbf3401e040 comp rx=0 tx=0).stop 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 shutdown_connections 2024-03-06T07:35:37.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff63ce70 0x5557ff63d250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63ab90 0x5557ff633490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf26000640 0 client.8481 ms_handle_reset on v2:172.21.15.6:6832/2669115160 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/460537176 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63a7b0 0x5557ff640160 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 >> 172.21.15.6:0/460537176 conn(0x5557ff60dc90 msgr2=0x5557ff612450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 shutdown_connections 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 1 -- 172.21.15.6:0/460537176 wait complete. 2024-03-06T07:35:37.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.890+0000 7fbf4a86a740 20 client.8481 trim_cache size 0 max 16384 2024-03-06T07:35:37.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.908+0000 7fbf4a86a740 1 Processor -- start 2024-03-06T07:35:37.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 -- start start 2024-03-06T07:35:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff640000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 0x5557ff6345c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4015ab50 con 0x5557ff641160 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40159fa0 con 0x5557ff63a7b0 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.909+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4015a550 con 0x5557ff63ab90 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 0x5557ff6345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 0x5557ff6345c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39520/0 (socket says 172.21.15.6:39520) 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 -- 172.21.15.6:0/1595709383 learned_addr learned my addr 172.21.15.6:0/1595709383 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45200640 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff640000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 -- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 msgr2=0x5557ff640000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff640000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 -- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 msgr2=0x5557ff633800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.910+0000 7fbf45c00640 1 -- 172.21.15.6:0/1595709383 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557ff63edc0 con 0x5557ff641160 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.911+0000 7fbf45c00640 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 0x5557ff6345c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbf38014580 tx=0x7fbf38009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d084ca843a4693 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.911+0000 7fbf30c00640 1 -- 172.21.15.6:0/1595709383 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3801c070 con 0x5557ff641160 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.911+0000 7fbf30c00640 1 -- 172.21.15.6:0/1595709383 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38018660 con 0x5557ff641160 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.911+0000 7fbf30c00640 1 -- 172.21.15.6:0/1595709383 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38018a10 con 0x5557ff641160 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.911+0000 7fbf4a86a740 1 -- 172.21.15.6:0/1595709383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 msgr2=0x5557ff6345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.911+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 0x5557ff6345c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbf38014580 tx=0x7fbf38009750 comp rx=0 tx=0).stop 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 -- 172.21.15.6:0/1595709383 shutdown_connections 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff641160 0x5557ff6345c0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff63ab90 0x5557ff640000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/1595709383 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 -- 172.21.15.6:0/1595709383 >> 172.21.15.6:0/1595709383 conn(0x5557ff60dc90 msgr2=0x5557ff65ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 -- 172.21.15.6:0/1595709383 shutdown_connections 2024-03-06T07:35:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 -- 172.21.15.6:0/1595709383 wait complete. 2024-03-06T07:35:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 -- start start 2024-03-06T07:35:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.912+0000 7fbf4a86a740 1 Processor -- start 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.913+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.913+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 0x5557ff633740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.913+0000 7fbf4a86a740 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6bc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.913+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557ff577600 con 0x5557ff63ab90 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.913+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557ff533d90 con 0x5557ff63a7b0 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.913+0000 7fbf4a86a740 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x5557ff636130 con 0x5557ff641160 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf4a86a740 10 client.0 _ll_register_callbacks cb 0x7fbf3406e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 0x5557ff633740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 0x5557ff633740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39532/0 (socket says 172.21.15.6:39532) 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 -- 172.21.15.6:0/4064924915 learned_addr learned my addr 172.21.15.6:0/4064924915 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf45200640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf45c00640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6bc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 -- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 msgr2=0x5557ff6bc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6bc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 -- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 msgr2=0x5557ff633200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 -- 172.21.15.6:0/4064924915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557ff609050 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf3fe00640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 0x5557ff633740 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c034040 tx=0x7fbf2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7b872a1de197ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.914+0000 7fbf26000640 10 client.0 ms_handle_connect on v2:172.21.15.6:3300/0 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf26000640 1 -- 172.21.15.6:0/4064924915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c035040 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf26000640 1 -- 172.21.15.6:0/4064924915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf2c051da0 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf38018660 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf26000640 1 -- 172.21.15.6:0/4064924915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf2c005990 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 10 client.8487 fetch_fsmap learned FSMap version 92 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x5557ff6bc7d0 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf26000640 1 -- 172.21.15.6:0/4064924915 <== mon.0 v2:172.21.15.6:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38018660 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf26000640 1 -- 172.21.15.6:0/4064924915 <== mon.0 v2:172.21.15.6:3300/0 5 ==== fsmap(e 92) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fbf2c005b30 con 0x5557ff63ab90 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 10 client.8487 fetch_fsmap finished waiting for FSMap version 92 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 10 client.8487 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:35:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 20 client.8487 populate_metadata read hostname 'smithi006' 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff6aa5a0 0x5557ff6d3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 4 client.8487 mds_command: new command op to 8355 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.915+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 --> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5557ff57ee90 con 0x5557ff6aa5a0 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.916+0000 7fbf45200640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff6aa5a0 0x5557ff6d3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.916+0000 7fbf45200640 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff6aa5a0 0x5557ff6d3ce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af80bd619c447f3 server_cookie=4ca068c4bbfc2b7f in_seq=0 out_seq=0 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.916+0000 7fbf26000640 10 client.8487 ms_handle_connect on v2:172.21.15.6:6832/2669115160 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.916+0000 7fbf26000640 1 -- 172.21.15.6:0/4064924915 <== mds.0 v2:172.21.15.6:6832/2669115160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5557ff57ee90 con 0x5557ff6aa5a0 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.916+0000 7fbf26000640 10 client.8487 handle_command_reply: tid=0 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.916+0000 7fbf4a86a740 1 client.8487 shutdown 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 msgr2=0x5557ff633740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 0x5557ff633740 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c034040 tx=0x7fbf2c004730 comp rx=0 tx=0).stop 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 shutdown_connections 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:6832/2669115160,v1:172.21.15.6:6834/2669115160] conn(0x5557ff6aa5a0 0x5557ff6d3ce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x5557ff641160 0x5557ff6bc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x5557ff63ab90 0x5557ff633740 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 --2- 172.21.15.6:0/4064924915 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5557ff63a7b0 0x5557ff633200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf26000640 0 client.8487 ms_handle_reset on v2:172.21.15.6:6832/2669115160 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 >> 172.21.15.6:0/4064924915 conn(0x5557ff60dc90 msgr2=0x5557ff6bf320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 shutdown_connections 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 1 -- 172.21.15.6:0/4064924915 wait complete. 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.917+0000 7fbf4a86a740 20 client.8487 trim_cache size 0 max 16384 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_fsid": "e57361cb-720d-49c4-a325-3e55e1c43024", 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stdout: "whoami": 0, 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stdout: "id": 8355, 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stdout: "want_state": "up:active", 2024-03-06T07:35:37.923 INFO:teuthology.orchestra.run.smithi006.stdout: "state": "up:active", 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout: "fs_name": "cephfs", 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout: "rank_uptime": 6.876942895, 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout: "mdsmap_epoch": 92, 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch": 126, 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout: "osdmap_epoch_barrier": 126, 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout: "uptime": 13.246097560000001 2024-03-06T07:35:37.924 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf2006b4d0 msgr2=0x7fbf2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf2006b4d0 0x7fbf2006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c036000 tx=0x7fbf2c03c040 comp rx=0 tx=0).stop 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 msgr2=0x7fbf40157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 0x7fbf40157a80 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400c400 tx=0x7fbf3400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 shutdown_connections 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbf2006b4d0 0x7fbf2006d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbf4015dd00 0x7fbf40154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbf4015e0e0 0x7fbf400cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 --2- 172.21.15.6:0/1935672097 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbf4006c6d0 0x7fbf40157a80 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 >> 172.21.15.6:0/1935672097 conn(0x7fbf400714d0 msgr2=0x7fbf400b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 shutdown_connections 2024-03-06T07:35:37.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:37.918+0000 7fbf47c00640 1 -- 172.21.15.6:0/1935672097 wait complete. 2024-03-06T07:35:37.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:35:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.140+0000 7f09dde00640 1 Processor -- start 2024-03-06T07:35:38.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.141+0000 7f09dde00640 1 -- start start 2024-03-06T07:35:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dde00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09d815e0b0 0x7f09d80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dde00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09d80d0e50 0x7f09d80b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.144 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dde00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d80b3710 con 0x7f09d80d4a10 2024-03-06T07:35:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dde00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d80b09e0 con 0x7f09d80d0e50 2024-03-06T07:35:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dde00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d80b39f0 con 0x7f09d815e0b0 2024-03-06T07:35:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09d80d0e50 0x7f09d80b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dd400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f09d80d0e50 0x7f09d80b44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39536/0 (socket says 172.21.15.6:39536) 2024-03-06T07:35:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.142+0000 7f09dd400640 1 -- 172.21.15.6:0/1681322498 learned_addr learned my addr 172.21.15.6:0/1681322498 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:38.145 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.143+0000 7f09dd400640 1 -- 172.21.15.6:0/1681322498 >> v2:172.21.15.178:3300/0 conn(0x7f09d815e0b0 msgr2=0x7f09d80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.143+0000 7f09dd400640 1 --2- 172.21.15.6:0/1681322498 >> v2:172.21.15.178:3300/0 conn(0x7f09d815e0b0 0x7f09d80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.143+0000 7f09dd400640 1 -- 172.21.15.6:0/1681322498 >> v1:172.21.15.178:6789/0 conn(0x7f09d80d4a10 legacy=0x7f09d80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.143+0000 7f09dd400640 1 -- 172.21.15.6:0/1681322498 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d80b3cd0 con 0x7f09d80d0e50 2024-03-06T07:35:38.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.144+0000 7f09dd400640 1 --2- 172.21.15.6:0/1681322498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d80b44d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f09d80b68c0 tx=0x7f09d002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=503bf6ff60bb7d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.144+0000 7f09d7400640 1 -- 172.21.15.6:0/1681322498 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d002ca80 con 0x7f09d80d0e50 2024-03-06T07:35:38.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.144+0000 7f09d7400640 1 -- 172.21.15.6:0/1681322498 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d002cc20 con 0x7f09d80d0e50 2024-03-06T07:35:38.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.144+0000 7f09d7400640 1 -- 172.21.15.6:0/1681322498 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d00343f0 con 0x7f09d80d0e50 2024-03-06T07:35:38.147 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 -- 172.21.15.6:0/1681322498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 msgr2=0x7f09d80b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 --2- 172.21.15.6:0/1681322498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d80b44d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f09d80b68c0 tx=0x7f09d002bee0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 -- 172.21.15.6:0/1681322498 shutdown_connections 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 --2- 172.21.15.6:0/1681322498 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d80b44d0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 --2- 172.21.15.6:0/1681322498 >> v2:172.21.15.178:3300/0 conn(0x7f09d815e0b0 0x7f09d80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 -- 172.21.15.6:0/1681322498 >> 172.21.15.6:0/1681322498 conn(0x7f09d80d3d30 msgr2=0x7f09d80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 -- 172.21.15.6:0/1681322498 shutdown_connections 2024-03-06T07:35:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.145+0000 7f09dde00640 1 -- 172.21.15.6:0/1681322498 wait complete. 2024-03-06T07:35:38.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.147+0000 7f09dde00640 1 Processor -- start 2024-03-06T07:35:38.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.147+0000 7f09dde00640 1 -- start start 2024-03-06T07:35:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.148+0000 7f09dde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d815f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.148+0000 7f09dde00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d80d4a10 0x7f09d8161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.148+0000 7f09dde00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d815e0b0 0x7f09d81f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dde00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d80cfc60 con 0x7f09d80d0e50 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dde00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d815e490 con 0x7f09d815e0b0 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dde00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d80cf850 con 0x7f09d80d4a10 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d815f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d815f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39544/0 (socket says 172.21.15.6:39544) 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 -- 172.21.15.6:0/2709660478 learned_addr learned my addr 172.21.15.6:0/2709660478 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09d7e00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d80d4a10 0x7f09d8161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dd400640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d815e0b0 0x7f09d81f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 -- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d80d4a10 msgr2=0x7f09d8161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d80d4a10 0x7f09d8161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 -- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d815e0b0 msgr2=0x7f09d81f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d815e0b0 0x7f09d81f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dca00640 1 -- 172.21.15.6:0/2709660478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d815e610 con 0x7f09d80d0e50 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09d7e00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d80d4a10 0x7f09d8161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.149+0000 7f09dd400640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d815e0b0 0x7f09d81f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:38.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.150+0000 7f09dca00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d815f0b0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00b410 tx=0x7f09cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.150+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc015020 con 0x7f09d80d0e50 2024-03-06T07:35:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.150+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc011850 con 0x7f09d80d0e50 2024-03-06T07:35:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.150+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc011c40 con 0x7f09d80d0e50 2024-03-06T07:35:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.150+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d81f4560 con 0x7f09d80d0e50 2024-03-06T07:35:38.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.150+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d815eb90 con 0x7f09d80d0e50 2024-03-06T07:35:38.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.152+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d804f330 con 0x7f09d80d0e50 2024-03-06T07:35:38.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.152+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09cc018020 con 0x7f09d80d0e50 2024-03-06T07:35:38.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.154+0000 7f09d4c00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b806b4d0 0x7f09b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:38.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.154+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09cc08a1d0 con 0x7f09d80d0e50 2024-03-06T07:35:38.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.154+0000 7f09d7e00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b806b4d0 0x7f09b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:38.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.156+0000 7f09d7e00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b806b4d0 0x7f09b806d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f09d80d0d60 tx=0x7f09c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:38.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.158+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09cc01f020 con 0x7f09d80d0e50 2024-03-06T07:35:38.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.401+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f09d80b7370 con 0x7f09d80d0e50 2024-03-06T07:35:38.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.401+0000 7f09d4c00640 1 -- 172.21.15.6:0/2709660478 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7f09cc0172c0 con 0x7f09d80d0e50 2024-03-06T07:35:38.404 INFO:teuthology.orchestra.run.smithi006.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b806b4d0 msgr2=0x7f09b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b806b4d0 0x7f09b806d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f09d80d0d60 tx=0x7f09c4009040 comp rx=0 tx=0).stop 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 msgr2=0x7f09d815f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d815f0b0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00b410 tx=0x7f09cc00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 shutdown_connections 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b806b4d0 0x7f09b806d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d815e0b0 0x7f09d81f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d80d4a10 0x7f09d8161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 --2- 172.21.15.6:0/2709660478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d80d0e50 0x7f09d815f0b0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.404+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 >> 172.21.15.6:0/2709660478 conn(0x7f09d80d3d30 msgr2=0x7f09d80b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.405+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 shutdown_connections 2024-03-06T07:35:38.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:38.405+0000 7f09dde00640 1 -- 172.21.15.6:0/2709660478 wait complete. 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi006.front.sepia.ceph.com 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:35:38.425 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:38.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:38.426 DEBUG:teuthology.orchestra.run.smithi006:> ip addr 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 ::1/128 scope host 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e6 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f0 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:6b:fe:e7 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp7s0f1 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f0 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: inet 172.21.15.6/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft 42119sec preferred_lft 42119sec 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0c:c4:7a:bd:15:e5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: altname enp1s0f1 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff 2024-03-06T07:35:38.480 INFO:teuthology.orchestra.run.smithi006.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::7800:b8ff:fe49:b7a7/64 scope link proto kernel_ll 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 26:86:45:c5:9b:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::2486:45ff:fec5:9bfa/64 scope link proto kernel_ll 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: link/ether 0e:ed:c7:0c:1b:7d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: inet6 fe80::ced:c7ff:fe0c:1b7d/64 scope link proto kernel_ll 2024-03-06T07:35:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:35:38.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:38.483 DEBUG:teuthology.orchestra.run.smithi006:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:35:38.574 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-06T07:35:38.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:38.579 DEBUG:teuthology.orchestra.run.smithi006:> route 2024-03-06T07:35:38.633 INFO:teuthology.orchestra.run.smithi006.stdout:Kernel IP routing table 2024-03-06T07:35:38.633 INFO:teuthology.orchestra.run.smithi006.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:35:38.633 INFO:teuthology.orchestra.run.smithi006.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-06T07:35:38.633 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-06T07:35:38.633 INFO:teuthology.orchestra.run.smithi006.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:35:38.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:38.634 DEBUG:teuthology.orchestra.run.smithi006:> (cd / && exec bash -c ' 2024-03-06T07:35:38.634 DEBUG:teuthology.orchestra.run.smithi006:> set -e 2024-03-06T07:35:38.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-06T07:35:38.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-06T07:35:38.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-06T07:35:38.634 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:35:38.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:38.818 DEBUG:teuthology.orchestra.run.smithi006:> ip netns list 2024-03-06T07:35:38.866 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:35:38.866 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:35:38.867 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:38.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:38.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo ip link set brx.0 up 2024-03-06T07:35:38.954 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:38.955 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:39.005 INFO:teuthology.orchestra.run.smithi006.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:35:39.005 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:35:39.005 DEBUG:teuthology.orchestra.run.smithi006:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:39.055 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-06T07:35:39.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:39.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:39.186 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:39.186 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:39.219 INFO:teuthology.orchestra.run.smithi006.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-06T07:35:39.220 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-06T07:35:39.220 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: resolved to: "172.21.15.6:3300,172.21.15.178:3300,172.21.15.178:3301" 2024-03-06T07:35:39.220 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: trying mount with new device syntax: 0@e57361cb-720d-49c4-a325-3e55e1c43024.cephfs=/ 2024-03-06T07:35:39.220 INFO:teuthology.orchestra.run.smithi006.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.6:3300/172.21.15.178:3300/172.21.15.178:3301" will pass to kernel 2024-03-06T07:35:39.222 INFO:teuthology.orchestra.run.smithi006.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-06T07:35:39.222 INFO:teuthology.orchestra.run.smithi006.stdout: You just mounted a file system that supports labels which does not 2024-03-06T07:35:39.222 INFO:teuthology.orchestra.run.smithi006.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-06T07:35:39.222 INFO:teuthology.orchestra.run.smithi006.stdout: applications will generate AVC messages and not be allowed access to 2024-03-06T07:35:39.222 INFO:teuthology.orchestra.run.smithi006.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-06T07:35:39.225 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-06T07:35:39.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:39.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:35:39.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:39.299 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-06T07:35:39.345 INFO:teuthology.orchestra.run.smithi006.stdout:client6973 2024-03-06T07:35:39.345 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:39.345 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:39.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:35:39.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.699+0000 7f0482c00640 1 Processor -- start 2024-03-06T07:35:39.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.699+0000 7f0482c00640 1 -- start start 2024-03-06T07:35:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.701+0000 7f0482c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 0x7f047c0ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.701+0000 7f0482c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047c072df0 con 0x7f047c15bf20 2024-03-06T07:35:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.701+0000 7f0482c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f047c0cdab0 con 0x7f047c072520 2024-03-06T07:35:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.701+0000 7f0482c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0b4d70 con 0x7f047c071e20 2024-03-06T07:35:39.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.701+0000 7f047be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f047c15bf20 0x7f047c0acd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42818/0 (socket says 172.21.15.6:42818) 2024-03-06T07:35:39.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.701+0000 7f047be00640 1 -- 172.21.15.6:0/2101749357 learned_addr learned my addr 172.21.15.6:0/2101749357 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:39.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f0480c00640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 0x7f047c0ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792935751 0 0) 0x7f047c072df0 con 0x7f047c15bf20 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c003520 con 0x7f047c15bf20 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745095401 0 0) 0x7f045c003520 con 0x7f047c15bf20 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 msgr2=0x7f047c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 0x7f047c0ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 >> v1:172.21.15.178:6789/0 conn(0x7f047c072520 legacy=0x7f047c0cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c0b5050 con 0x7f047c15bf20 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f046c001670 con 0x7f047c15bf20 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.702+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 >> v1:172.21.15.6:6789/0 conn(0x7f047c15bf20 legacy=0x7f047c0acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047b400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c0039e0 0x7f045c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047b400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045c004800 0x7f045c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047b400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 0x7f045c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0b4d70 con 0x7f045c004800 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c00ba70 con 0x7f045c0039e0 2024-03-06T07:35:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c00bc70 con 0x7f047c071e20 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f0473400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045c004800 0x7f045c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f0480c00640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 0x7f045c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f047be00640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c0039e0 0x7f045c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.703+0000 7f0473400640 1 -- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 msgr2=0x7f045c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.704+0000 7f0473400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c071e20 0x7f045c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.704+0000 7f0473400640 1 -- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c0039e0 msgr2=0x7f045c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.704+0000 7f0473400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c0039e0 0x7f045c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.704+0000 7f0473400640 1 -- 172.21.15.6:0/2101749357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c003520 con 0x7f045c004800 2024-03-06T07:35:39.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.705+0000 7f0473400640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045c004800 0x7f045c004c80 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0464001720 tx=0x7f045c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518a3b3fce483abd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464001370 con 0x7f045c004800 2024-03-06T07:35:39.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04640052d0 con 0x7f045c004800 2024-03-06T07:35:39.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f047b400640 1 -- 172.21.15.6:0/2101749357 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464003830 con 0x7f045c004800 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0482c00640 1 -- 172.21.15.6:0/2101749357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045c004800 msgr2=0x7f045c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0482c00640 1 --2- 172.21.15.6:0/2101749357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f045c004800 0x7f045c004c80 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0464001720 tx=0x7f045c005830 comp rx=0 tx=0).stop 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0480c00640 1 -- 172.21.15.6:0/2101749357 reap_dead start 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0482c00640 1 -- 172.21.15.6:0/2101749357 shutdown_connections 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0482c00640 1 -- 172.21.15.6:0/2101749357 >> 172.21.15.6:0/2101749357 conn(0x7f047c06b3d0 msgr2=0x7f047c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0482c00640 1 -- 172.21.15.6:0/2101749357 shutdown_connections 2024-03-06T07:35:39.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.706+0000 7f0482c00640 1 -- 172.21.15.6:0/2101749357 wait complete. 2024-03-06T07:35:39.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.708+0000 7f0482c00640 1 Processor -- start 2024-03-06T07:35:39.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.708+0000 7f0482c00640 1 -- start start 2024-03-06T07:35:39.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0482c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c004800 0x7f047c0b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0482c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c0cb220 0x7f047c0c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0482c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 0x7f047c0c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0482c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0ad450 con 0x7f047c0cb6a0 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0482c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0b4990 con 0x7f045c004800 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0482c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047c0ca6b0 con 0x7f047c0cb220 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.709+0000 7f0480c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 0x7f047c0c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 0x7f047c0c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39556/0 (socket says 172.21.15.6:39556) 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 -- 172.21.15.6:0/3297583520 learned_addr learned my addr 172.21.15.6:0/3297583520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0473400640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c0cb220 0x7f047c0c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f047be00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c004800 0x7f047c0b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 -- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c0cb220 msgr2=0x7f047c0c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c0cb220 0x7f047c0c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 -- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c004800 msgr2=0x7f047c0b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c004800 0x7f047c0b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 -- 172.21.15.6:0/3297583520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c0b0e30 con 0x7f047c0cb6a0 2024-03-06T07:35:39.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.710+0000 7f0480c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 0x7f047c0c9f30 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f047c0cd670 tx=0x7f04740115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.711+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474003990 con 0x7f047c0cb6a0 2024-03-06T07:35:39.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.711+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047401b070 con 0x7f047c0cb6a0 2024-03-06T07:35:39.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.711+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c09abb0 con 0x7f047c0cb6a0 2024-03-06T07:35:39.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.711+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474003b70 con 0x7f047c0cb6a0 2024-03-06T07:35:39.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.711+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c09b130 con 0x7f047c0cb6a0 2024-03-06T07:35:39.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.712+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c04f330 con 0x7f047c0cb6a0 2024-03-06T07:35:39.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.713+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0474005d10 con 0x7f047c0cb6a0 2024-03-06T07:35:39.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.715+0000 7f0478c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f045806b4d0 0x7f045806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:39.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.715+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f047408a820 con 0x7f047c0cb6a0 2024-03-06T07:35:39.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.715+0000 7f047be00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f045806b4d0 0x7f045806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:39.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.717+0000 7f047be00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f045806b4d0 0x7f045806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f046c001050 tx=0x7f046c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:39.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:39.719+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0474025020 con 0x7f047c0cb6a0 2024-03-06T07:35:40.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.063+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f047c15f720 con 0x7f047c0cb6a0 2024-03-06T07:35:40.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.127+0000 7f0478c00640 1 -- 172.21.15.6:0/3297583520 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v93) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0474059730 con 0x7f047c0cb6a0 2024-03-06T07:35:40.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f045806b4d0 msgr2=0x7f045806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.132 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f045806b4d0 0x7f045806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f046c001050 tx=0x7f046c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 msgr2=0x7f047c0c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 0x7f047c0c9f30 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f047c0cd670 tx=0x7f04740115a0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 shutdown_connections 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f045806b4d0 0x7f045806d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f047c0cb6a0 0x7f047c0c9f30 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f047c0cb220 0x7f047c0c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 --2- 172.21.15.6:0/3297583520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f045c004800 0x7f047c0b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.130+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 >> 172.21.15.6:0/3297583520 conn(0x7f047c06b3d0 msgr2=0x7f047c0ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.131+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 shutdown_connections 2024-03-06T07:35:40.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.131+0000 7f0482c00640 1 -- 172.21.15.6:0/3297583520 wait complete. 2024-03-06T07:35:40.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.349+0000 7f5224e00640 1 Processor -- start 2024-03-06T07:35:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.350+0000 7f5224e00640 1 -- start start 2024-03-06T07:35:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f5224e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f522015e0b0 0x7f52200cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f5224e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f52200d4a10 0x7f52200d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f5224e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52200d0c80 0x7f52200d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f5224e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200b09e0 con 0x7f52200d0c80 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f5224e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52200b3710 con 0x7f52200d4a10 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f5224e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52200b39f0 con 0x7f522015e0b0 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f521fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52200d0c80 0x7f52200d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f521fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52200d0c80 0x7f52200d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39570/0 (socket says 172.21.15.6:39570) 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.351+0000 7f521fe00640 1 -- 172.21.15.6:0/1340185933 learned_addr learned my addr 172.21.15.6:0/1340185933 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f5217e00640 1 --2- 172.21.15.6:0/1340185933 >> v2:172.21.15.178:3300/0 conn(0x7f52200d4a10 0x7f52200d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f521f400640 1 --2- 172.21.15.6:0/1340185933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f522015e0b0 0x7f52200cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f521fe00640 1 -- 172.21.15.6:0/1340185933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f522015e0b0 msgr2=0x7f52200cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f521fe00640 1 --2- 172.21.15.6:0/1340185933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f522015e0b0 0x7f52200cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f521fe00640 1 -- 172.21.15.6:0/1340185933 >> v2:172.21.15.178:3300/0 conn(0x7f52200d4a10 msgr2=0x7f52200d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f521fe00640 1 --2- 172.21.15.6:0/1340185933 >> v2:172.21.15.178:3300/0 conn(0x7f52200d4a10 0x7f52200d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.352+0000 7f521fe00640 1 -- 172.21.15.6:0/1340185933 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200b3cd0 con 0x7f52200d0c80 2024-03-06T07:35:40.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.354+0000 7f521fe00640 1 --2- 172.21.15.6:0/1340185933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d0c80 0x7f52200d1900 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f52200b6920 tx=0x7f521802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11dde6ed37505b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.354+0000 7f521ea00640 1 -- 172.21.15.6:0/1340185933 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521802ca70 con 0x7f52200d0c80 2024-03-06T07:35:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f521ea00640 1 -- 172.21.15.6:0/1340185933 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521802cc10 con 0x7f52200d0c80 2024-03-06T07:35:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f521ea00640 1 -- 172.21.15.6:0/1340185933 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52180343f0 con 0x7f52200d0c80 2024-03-06T07:35:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 -- 172.21.15.6:0/1340185933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d0c80 msgr2=0x7f52200d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 --2- 172.21.15.6:0/1340185933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d0c80 0x7f52200d1900 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f52200b6920 tx=0x7f521802bed0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 -- 172.21.15.6:0/1340185933 shutdown_connections 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 --2- 172.21.15.6:0/1340185933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d0c80 0x7f52200d1900 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 --2- 172.21.15.6:0/1340185933 >> v2:172.21.15.178:3300/0 conn(0x7f52200d4a10 0x7f52200d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 --2- 172.21.15.6:0/1340185933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f522015e0b0 0x7f52200cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 -- 172.21.15.6:0/1340185933 >> 172.21.15.6:0/1340185933 conn(0x7f52200d3d30 msgr2=0x7f52200d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 -- 172.21.15.6:0/1340185933 shutdown_connections 2024-03-06T07:35:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.355+0000 7f5224e00640 1 -- 172.21.15.6:0/1340185933 wait complete. 2024-03-06T07:35:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.357+0000 7f5224e00640 1 Processor -- start 2024-03-06T07:35:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.357+0000 7f5224e00640 1 -- start start 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5224e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52200d0c80 0x7f52200a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5224e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 0x7f522015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5224e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f522015e0b0 0x7f52200cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5224e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52200cfc60 con 0x7f52200d4a10 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5224e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522015e490 con 0x7f522015e0b0 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5224e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52200cf850 con 0x7f52200d0c80 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5217e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 0x7f522015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.358+0000 7f5217e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 0x7f522015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39580/0 (socket says 172.21.15.6:39580) 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 -- 172.21.15.6:0/325455346 learned_addr learned my addr 172.21.15.6:0/325455346 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 -- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52200d0c80 msgr2=0x7f52200a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52200d0c80 0x7f52200a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 -- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f522015e0b0 msgr2=0x7f52200cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f522015e0b0 0x7f52200cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 -- 172.21.15.6:0/325455346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522015b370 con 0x7f52200d4a10 2024-03-06T07:35:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 0x7f522015ae30 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f520800d7d0 tx=0x7f520800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.359+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520800b6c0 con 0x7f52200d4a10 2024-03-06T07:35:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.360+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208019070 con 0x7f52200d4a10 2024-03-06T07:35:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.360+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520800ba80 con 0x7f52200d4a10 2024-03-06T07:35:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.360+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522015e610 con 0x7f52200d4a10 2024-03-06T07:35:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.360+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522015e8b0 con 0x7f52200d4a10 2024-03-06T07:35:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.361+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522004f330 con 0x7f52200d4a10 2024-03-06T07:35:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.362+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f520800bc20 con 0x7f52200d4a10 2024-03-06T07:35:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.363+0000 7f5217400640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51fc06b500 0x7f51fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.363+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f520808a710 con 0x7f52200d4a10 2024-03-06T07:35:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.364+0000 7f521f400640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51fc06b500 0x7f51fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.365+0000 7f521f400640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51fc06b500 0x7f51fc06d9c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f521000b7f0 tx=0x7f5210008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.368+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52080594f0 con 0x7f52200d4a10 2024-03-06T07:35:40.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.632+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52200b55f0 con 0x7f52200d4a10 2024-03-06T07:35:40.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.633+0000 7f5217400640 1 -- 172.21.15.6:0/325455346 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4992 (secure 0 0 0) 0x7f5208058df0 con 0x7f52200d4a10 2024-03-06T07:35:40.635 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:40.636 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1518219931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1518219931},{"type":"v1","addr":"172.21.15.178:6837","nonce":1518219931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/1688442844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1688442844},{"type":"v1","addr":"172.21.15.178:6836","nonce":1688442844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:40.133229+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:creating","state_seq":1,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:35:40.636 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 94 2024-03-06T07:35:40.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51fc06b500 msgr2=0x7f51fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51fc06b500 0x7f51fc06d9c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f521000b7f0 tx=0x7f5210008040 comp rx=0 tx=0).stop 2024-03-06T07:35:40.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 msgr2=0x7f522015ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:40.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 0x7f522015ae30 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f520800d7d0 tx=0x7f520800dc90 comp rx=0 tx=0).stop 2024-03-06T07:35:40.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 shutdown_connections 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51fc06b500 0x7f51fc06d9c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f522015e0b0 0x7f52200cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52200d4a10 0x7f522015ae30 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 --2- 172.21.15.6:0/325455346 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52200d0c80 0x7f52200a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 >> 172.21.15.6:0/325455346 conn(0x7f52200d3d30 msgr2=0x7f52200b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 shutdown_connections 2024-03-06T07:35:40.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:40.636+0000 7f5224e00640 1 -- 172.21.15.6:0/325455346 wait complete. 2024-03-06T07:35:40.656 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:35:45.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:45.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.870+0000 7f1f7a200640 1 Processor -- start 2024-03-06T07:35:45.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.871+0000 7f1f7a200640 1 -- start start 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.872+0000 7f1f7a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f7415b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.872+0000 7f1f7a200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f7415e810 0x7f1f740cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.872+0000 7f1f7a200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7415ed10 con 0x7f1f740cce90 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.872+0000 7f1f7a200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7415eff0 con 0x7f1f7415e810 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.872+0000 7f1f7a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7415f2d0 con 0x7f1f741552b0 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.873+0000 7f1f78e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f7415b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.873+0000 7f1f73e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f7415e810 0x7f1f740cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.873+0000 7f1f78e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f7415b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:39118/0 (socket says 172.21.15.6:39118) 2024-03-06T07:35:45.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.873+0000 7f1f78e00640 1 -- 172.21.15.6:0/3590546384 learned_addr learned my addr 172.21.15.6:0/3590546384 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:45.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.874+0000 7f1f73e00640 1 -- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 msgr2=0x7f1f7415b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:45.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.874+0000 7f1f73e00640 1 --2- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f7415b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.874+0000 7f1f73e00640 1 -- 172.21.15.6:0/3590546384 >> v1:172.21.15.178:6789/0 conn(0x7f1f740cce90 legacy=0x7f1f7406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.874+0000 7f1f73e00640 1 -- 172.21.15.6:0/3590546384 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7415f5b0 con 0x7f1f7415e810 2024-03-06T07:35:45.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.874+0000 7f1f78e00640 1 --2- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f7415b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:45.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.875+0000 7f1f73e00640 1 --2- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f7415e810 0x7f1f740cc950 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0097a0 tx=0x7f1f5c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2891a59341d4c267 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.876+0000 7f1f73400640 1 -- 172.21.15.6:0/3590546384 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c031780 con 0x7f1f7415e810 2024-03-06T07:35:45.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.876+0000 7f1f73400640 1 -- 172.21.15.6:0/3590546384 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f5c037070 con 0x7f1f7415e810 2024-03-06T07:35:45.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.876+0000 7f1f73400640 1 -- 172.21.15.6:0/3590546384 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c031b40 con 0x7f1f7415e810 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 -- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f7415e810 msgr2=0x7f1f740cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 --2- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f7415e810 0x7f1f740cc950 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0097a0 tx=0x7f1f5c02b820 comp rx=0 tx=0).stop 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 -- 172.21.15.6:0/3590546384 shutdown_connections 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 --2- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f7415e810 0x7f1f740cc950 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 --2- 172.21.15.6:0/3590546384 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f7415b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 -- 172.21.15.6:0/3590546384 >> 172.21.15.6:0/3590546384 conn(0x7f1f7406b3d0 msgr2=0x7f1f7406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 -- 172.21.15.6:0/3590546384 shutdown_connections 2024-03-06T07:35:45.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.877+0000 7f1f7a200640 1 -- 172.21.15.6:0/3590546384 wait complete. 2024-03-06T07:35:45.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.880+0000 7f1f7a200640 1 Processor -- start 2024-03-06T07:35:45.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.880+0000 7f1f7a200640 1 -- start start 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.881+0000 7f1f7a200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f740cce90 0x7f1f74152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.881+0000 7f1f7a200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f74147520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f7a200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 0x7f1f740b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f7a200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7406dd30 con 0x7f1f7415e810 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f7a200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f740b3760 con 0x7f1f740cce90 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f7a200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7406deb0 con 0x7f1f741552b0 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f79800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 0x7f1f740b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f79800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 0x7f1f740b08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32830/0 (socket says 172.21.15.6:32830) 2024-03-06T07:35:45.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f79800640 1 -- 172.21.15.6:0/630692156 learned_addr learned my addr 172.21.15.6:0/630692156 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:45.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f78e00640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f740cce90 0x7f1f74152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.882+0000 7f1f73e00640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f74147520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f79800640 1 -- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 msgr2=0x7f1f74147520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f79800640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f74147520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f79800640 1 -- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f740cce90 msgr2=0x7f1f74152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f79800640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f740cce90 0x7f1f74152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f79800640 1 -- 172.21.15.6:0/630692156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f741530a0 con 0x7f1f7415e810 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f79800640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 0x7f1f740b08f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00c3f0 tx=0x7f1f6c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0052d0 con 0x7f1f7415e810 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c005490 con 0x7f1f7415e810 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.883+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c007260 con 0x7f1f7415e810 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.884+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f74153370 con 0x7f1f7415e810 2024-03-06T07:35:45.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.884+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f74153930 con 0x7f1f7415e810 2024-03-06T07:35:45.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.885+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7404f330 con 0x7f1f7415e810 2024-03-06T07:35:45.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.886+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f6c007400 con 0x7f1f7415e810 2024-03-06T07:35:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.887+0000 7f1f70c00640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f5006b500 0x7f1f5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.887+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f6c0894f0 con 0x7f1f7415e810 2024-03-06T07:35:45.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.888+0000 7f1f78e00640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f5006b500 0x7f1f5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:45.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.889+0000 7f1f78e00640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f5006b500 0x7f1f5006d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1f741f2dc0 tx=0x7f1f6400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:45.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:45.894+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f6c057db0 con 0x7f1f7415e810 2024-03-06T07:35:46.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.207+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f740cf4a0 con 0x7f1f7415e810 2024-03-06T07:35:46.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.208+0000 7f1f70c00640 1 -- 172.21.15.6:0/630692156 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4995 (secure 0 0 0) 0x7f1f6c05be50 con 0x7f1f7415e810 2024-03-06T07:35:46.211 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:46.211 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":95,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1518219931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1518219931},{"type":"v1","addr":"172.21.15.178:6837","nonce":1518219931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/1688442844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1688442844},{"type":"v1","addr":"172.21.15.178:6836","nonce":1688442844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:41.136852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[6871,8355]},"id":9}]} 2024-03-06T07:35:46.211 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 95 2024-03-06T07:35:46.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f5006b500 msgr2=0x7f1f5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f5006b500 0x7f1f5006d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1f741f2dc0 tx=0x7f1f6400b040 comp rx=0 tx=0).stop 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 msgr2=0x7f1f740b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 0x7f1f740b08f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00c3f0 tx=0x7f1f6c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 shutdown_connections 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f5006b500 0x7f1f5006d9c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f7415e810 0x7f1f740b08f0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f741552b0 0x7f1f74147520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 --2- 172.21.15.6:0/630692156 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f740cce90 0x7f1f74152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 >> 172.21.15.6:0/630692156 conn(0x7f1f7406b3d0 msgr2=0x7f1f740cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 shutdown_connections 2024-03-06T07:35:46.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.211+0000 7f1f7a200640 1 -- 172.21.15.6:0/630692156 wait complete. 2024-03-06T07:35:46.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:46.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.443+0000 7fab1d400640 1 Processor -- start 2024-03-06T07:35:46.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.443+0000 7fab1d400640 1 -- start start 2024-03-06T07:35:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.444+0000 7fab1d400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fab180d2b40 0x7fab180d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.444+0000 7fab1d400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180d1480 0x7fab180b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.444+0000 7fab1d400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab180b3940 con 0x7fab1815e2e0 2024-03-06T07:35:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.444+0000 7fab1d400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab180d4e70 con 0x7fab180d2b40 2024-03-06T07:35:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.444+0000 7fab1d400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab180b3c40 con 0x7fab180d1480 2024-03-06T07:35:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.445+0000 7fab16800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fab1815e2e0 0x7fab180d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38434/0 (socket says 172.21.15.6:38434) 2024-03-06T07:35:46.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.445+0000 7fab16800640 1 -- 172.21.15.6:0/2503235935 learned_addr learned my addr 172.21.15.6:0/2503235935 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.445+0000 7fab15e00640 1 --2- 172.21.15.6:0/2503235935 >> v2:172.21.15.178:3300/0 conn(0x7fab180d2b40 0x7fab180d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.445+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548767726 0 0) 0x7fab180b3940 con 0x7fab1815e2e0 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.445+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faafc003520 con 0x7fab1815e2e0 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.445+0000 7fab17200640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180d1480 0x7fab180b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952656071 0 0) 0x7faafc003520 con 0x7fab1815e2e0 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180d1480 msgr2=0x7fab180b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180d1480 0x7fab180b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 >> v2:172.21.15.178:3300/0 conn(0x7fab180d2b40 msgr2=0x7fab180d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 --2- 172.21.15.6:0/2503235935 >> v2:172.21.15.178:3300/0 conn(0x7fab180d2b40 0x7fab180d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab180cf580 con 0x7fab1815e2e0 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fab08001670 con 0x7fab1815e2e0 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 >> v1:172.21.15.6:6789/0 conn(0x7fab1815e2e0 legacy=0x7fab180d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faafc003a20 0x7faafc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7faafc0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faafc004fc0 0x7faafc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab180b3c40 con 0x7faafc0043d0 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc00bf00 con 0x7faafc004fc0 2024-03-06T07:35:46.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc00c080 con 0x7faafc003a20 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab15e00640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7faafc0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab16800640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faafc003a20 0x7faafc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.446+0000 7fab17200640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faafc004fc0 0x7faafc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.447+0000 7fab15e00640 1 -- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faafc003a20 msgr2=0x7faafc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.447+0000 7fab15e00640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faafc003a20 0x7faafc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.447+0000 7fab15e00640 1 -- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faafc004fc0 msgr2=0x7faafc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.447+0000 7fab15e00640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faafc004fc0 0x7faafc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.447+0000 7fab15e00640 1 -- 172.21.15.6:0/2503235935 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc003520 con 0x7faafc0043d0 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab15e00640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7faafc0048a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fab04009a90 tx=0x7faafc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7260cfe79a614cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04039070 con 0x7faafc0043d0 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0402c6b0 con 0x7faafc0043d0 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab15400640 1 -- 172.21.15.6:0/2503235935 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04009690 con 0x7faafc0043d0 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab1d400640 1 -- 172.21.15.6:0/2503235935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 msgr2=0x7faafc0048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab1d400640 1 --2- 172.21.15.6:0/2503235935 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7faafc0048a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fab04009a90 tx=0x7faafc005830 comp rx=0 tx=0).stop 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.448+0000 7fab17200640 1 -- 172.21.15.6:0/2503235935 reap_dead start 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.449+0000 7fab1d400640 1 -- 172.21.15.6:0/2503235935 shutdown_connections 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.449+0000 7fab1d400640 1 -- 172.21.15.6:0/2503235935 >> 172.21.15.6:0/2503235935 conn(0x7fab180d3f60 msgr2=0x7fab180d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.449+0000 7fab1d400640 1 -- 172.21.15.6:0/2503235935 shutdown_connections 2024-03-06T07:35:46.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.449+0000 7fab1d400640 1 -- 172.21.15.6:0/2503235935 wait complete. 2024-03-06T07:35:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.452+0000 7fab1d400640 1 Processor -- start 2024-03-06T07:35:46.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.452+0000 7fab1d400640 1 -- start start 2024-03-06T07:35:46.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.453+0000 7fab1d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7fab180b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.453+0000 7fab1d400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab181537a0 0x7fab180c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.453+0000 7fab1d400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180b4ca0 0x7fab180cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab1d400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab180cfe20 con 0x7faafc0043d0 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab1d400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab180cfb40 con 0x7fab181537a0 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab1d400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab180b40d0 con 0x7fab180b4ca0 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7fab180b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7fab180b8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32848/0 (socket says 172.21.15.6:32848) 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 -- 172.21.15.6:0/4046292361 learned_addr learned my addr 172.21.15.6:0/4046292361 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab15e00640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab181537a0 0x7fab180c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 -- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180b4ca0 msgr2=0x7fab180cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180b4ca0 0x7fab180cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 -- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab181537a0 msgr2=0x7fab180c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab181537a0 0x7fab180c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.454+0000 7fab16800640 1 -- 172.21.15.6:0/4046292361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab180c9340 con 0x7faafc0043d0 2024-03-06T07:35:46.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.455+0000 7fab16800640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7fab180b8670 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fab08001050 tx=0x7fab0800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.455+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0800ee40 con 0x7faafc0043d0 2024-03-06T07:35:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.455+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08018070 con 0x7faafc0043d0 2024-03-06T07:35:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.455+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0800ca90 con 0x7faafc0043d0 2024-03-06T07:35:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.455+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab180c9620 con 0x7faafc0043d0 2024-03-06T07:35:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.455+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab180c9be0 con 0x7faafc0043d0 2024-03-06T07:35:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.456+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab18153ba0 con 0x7faafc0043d0 2024-03-06T07:35:46.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.457+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab0800cc30 con 0x7faafc0043d0 2024-03-06T07:35:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.459+0000 7fab0ea00640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faaf806b4d0 0x7faaf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:46.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.459+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab0808a050 con 0x7faafc0043d0 2024-03-06T07:35:46.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.460+0000 7fab15e00640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faaf806b4d0 0x7faaf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:46.463 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.461+0000 7fab15e00640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faaf806b4d0 0x7faaf806d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fab04036000 tx=0x7fab04030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:46.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.463+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fab18153ba0 con 0x7faafc0043d0 2024-03-06T07:35:46.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.810+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab1804f3a0 con 0x7faafc0043d0 2024-03-06T07:35:46.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.812+0000 7fab0ea00640 1 -- 172.21.15.6:0/4046292361 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4995 (secure 0 0 0) 0x7fab08058f60 con 0x7faafc0043d0 2024-03-06T07:35:46.814 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:46.814 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":95,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/1518219931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":1518219931},{"type":"v1","addr":"172.21.15.178:6837","nonce":1518219931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":6044,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6836/1688442844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6835","nonce":1688442844},{"type":"v1","addr":"172.21.15.178:6836","nonce":1688442844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:41.136852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[6871,8355]},"id":9}]} 2024-03-06T07:35:46.814 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 95 2024-03-06T07:35:46.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faaf806b4d0 msgr2=0x7faaf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faaf806b4d0 0x7faaf806d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fab04036000 tx=0x7fab04030000 comp rx=0 tx=0).stop 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 msgr2=0x7fab180b8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7fab180b8670 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fab08001050 tx=0x7fab0800ebf0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 shutdown_connections 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faaf806b4d0 0x7faaf806d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab181537a0 0x7fab180c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fab180b4ca0 0x7fab180cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 --2- 172.21.15.6:0/4046292361 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faafc0043d0 0x7fab180b8670 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 >> 172.21.15.6:0/4046292361 conn(0x7fab180d3f60 msgr2=0x7fab1806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 shutdown_connections 2024-03-06T07:35:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:46.815+0000 7fab1d400640 1 -- 172.21.15.6:0/4046292361 wait complete. 2024-03-06T07:35:46.837 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-03-06T07:35:46.838 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:35:46.838 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:46.867 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:35:46.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:35:47.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.067+0000 7f7c16c00640 1 Processor -- start 2024-03-06T07:35:47.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.068+0000 7f7c16c00640 1 -- start start 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7c1015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c16c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1015cc40 con 0x7f7c100af580 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c16c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1015cf20 con 0x7f7c100cdc20 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c16c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1015d200 con 0x7f7c100af1a0 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c0f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7c100af580 0x7f7c100b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38450/0 (socket says 172.21.15.6:38450) 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c0f400640 1 -- 172.21.15.6:0/1470949201 learned_addr learned my addr 172.21.15.6:0/1470949201 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:47.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7c1015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.070+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538599094 0 0) 0x7f7c1015cc40 con 0x7f7c100af580 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf4003520 con 0x7f7c100af580 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058367871 0 0) 0x7f7bf4003520 con 0x7f7c100af580 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 msgr2=0x7f7c1015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7c1015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 >> v1:172.21.15.178:6789/0 conn(0x7f7c100cdc20 legacy=0x7f7c100b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1015d4e0 con 0x7f7c100af580 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7bfc001670 con 0x7f7c100af580 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 >> v1:172.21.15.6:6789/0 conn(0x7f7c100af580 legacy=0x7f7c100b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7bf4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7bf4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bf4004d40 0x7f7bf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1015d200 con 0x7f7bf4003a50 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf400bba0 con 0x7f7bf4004d40 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.071+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf400bda0 con 0x7f7c100af1a0 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7bf4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0f400640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7bf4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 msgr2=0x7f7bf4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7bf4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bf4004d40 msgr2=0x7f7bf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7bf4004d40 0x7f7bf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.072+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1470949201 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4003520 con 0x7f7bf4003a50 2024-03-06T07:35:47.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.073+0000 7f7c0f400640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c100af1a0 0x7f7bf4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:47.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.074+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7bf4003e70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009ac0 tx=0x7f7bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16d91245af56bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.074+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00039070 con 0x7f7bf4003a50 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.074+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0002c6b0 con 0x7f7bf4003a50 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.074+0000 7f7c0ea00640 1 -- 172.21.15.6:0/1470949201 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00009690 con 0x7f7bf4003a50 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c16c00640 1 -- 172.21.15.6:0/1470949201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 msgr2=0x7f7bf4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1470949201 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7bf4003e70 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009ac0 tx=0x7f7bf4005830 comp rx=0 tx=0).stop 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c14c00640 1 -- 172.21.15.6:0/1470949201 reap_dead start 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c16c00640 1 -- 172.21.15.6:0/1470949201 shutdown_connections 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c16c00640 1 -- 172.21.15.6:0/1470949201 >> 172.21.15.6:0/1470949201 conn(0x7f7c1006afb0 msgr2=0x7f7c1006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c16c00640 1 -- 172.21.15.6:0/1470949201 shutdown_connections 2024-03-06T07:35:47.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.075+0000 7f7c16c00640 1 -- 172.21.15.6:0/1470949201 wait complete. 2024-03-06T07:35:47.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.077+0000 7f7c16c00640 1 Processor -- start 2024-03-06T07:35:47.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.077+0000 7f7c16c00640 1 -- start start 2024-03-06T07:35:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7c10154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c1015b620 0x7f7c10159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c16c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c10159d80 0x7f7c10157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c16c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1015c600 con 0x7f7bf4003a50 2024-03-06T07:35:47.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c16c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100b36f0 con 0x7f7c1015b620 2024-03-06T07:35:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c16c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1015c780 con 0x7f7c10159d80 2024-03-06T07:35:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7c10154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7c10154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32866/0 (socket says 172.21.15.6:32866) 2024-03-06T07:35:47.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.079+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1427849936 learned_addr learned my addr 172.21.15.6:0/1427849936 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:47.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0f400640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c1015b620 0x7f7c10159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c14c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c10159d80 0x7f7c10157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c10159d80 msgr2=0x7f7c10157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c10159d80 0x7f7c10157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c1015b620 msgr2=0x7f7c10159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c1015b620 0x7f7c10159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0fe00640 1 -- 172.21.15.6:0/1427849936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c101f2f40 con 0x7f7bf4003a50 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c14c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c10159d80 0x7f7c10157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0f400640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c1015b620 0x7f7c10159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.080+0000 7f7c0fe00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7c10154ed0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009ac0 tx=0x7f7c00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.081+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00039050 con 0x7f7bf4003a50 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.081+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c00045070 con 0x7f7bf4003a50 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.081+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c101f3220 con 0x7f7bf4003a50 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.081+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00003a50 con 0x7f7bf4003a50 2024-03-06T07:35:47.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.081+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c101f37e0 con 0x7f7bf4003a50 2024-03-06T07:35:47.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.082+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c10069510 con 0x7f7bf4003a50 2024-03-06T07:35:47.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.083+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c00003bf0 con 0x7f7bf4003a50 2024-03-06T07:35:47.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.084+0000 7f7c07e00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bf006b4d0 0x7f7bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:47.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.085+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c000ab910 con 0x7f7bf4003a50 2024-03-06T07:35:47.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.085+0000 7f7c0f400640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bf006b4d0 0x7f7bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:47.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.087+0000 7f7c0f400640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bf006b4d0 0x7f7bf006d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7c1006ef50 tx=0x7f7bfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:47.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.089+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c0007a730 con 0x7f7bf4003a50 2024-03-06T07:35:47.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:47.373+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f7c1004f3a0 con 0x7f7bf4003a50 2024-03-06T07:35:48.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.329+0000 7f7c07e00640 1 -- 172.21.15.6:0/1427849936 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6041 v96) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7c0007a030 con 0x7f7bf4003a50 2024-03-06T07:35:48.331 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6041 2024-03-06T07:35:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bf006b4d0 msgr2=0x7f7bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bf006b4d0 0x7f7bf006d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7c1006ef50 tx=0x7f7bfc009040 comp rx=0 tx=0).stop 2024-03-06T07:35:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 msgr2=0x7f7c10154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7c10154ed0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009ac0 tx=0x7f7c00004730 comp rx=0 tx=0).stop 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 shutdown_connections 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7bf006b4d0 0x7f7bf006d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c1015b620 0x7f7c10159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7c10159d80 0x7f7c10157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 --2- 172.21.15.6:0/1427849936 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7bf4003a50 0x7f7c10154ed0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.333+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 >> 172.21.15.6:0/1427849936 conn(0x7f7c1006afb0 msgr2=0x7f7c1006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.334+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 shutdown_connections 2024-03-06T07:35:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.334+0000 7f7c16c00640 1 -- 172.21.15.6:0/1427849936 wait complete. 2024-03-06T07:35:48.358 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:35:48.358 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:35:48.378 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:35:48.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:35:48.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.593+0000 7ff32ee00640 1 Processor -- start 2024-03-06T07:35:48.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.593+0000 7ff32ee00640 1 -- start start 2024-03-06T07:35:48.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32ee00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff32815e070 0x7ff3280d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3280cf410 0x7ff3280cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32ee00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3280d05c0 0x7ff3280d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3280d18c0 con 0x7ff3280cf410 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32ee00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3280d1ba0 con 0x7ff32815e070 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32ee00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3280d1e80 con 0x7ff3280d05c0 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3280cf410 0x7ff3280cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32d000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff3280cf410 0x7ff3280cf870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32880/0 (socket says 172.21.15.6:32880) 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32d000640 1 -- 172.21.15.6:0/380979319 learned_addr learned my addr 172.21.15.6:0/380979319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.595+0000 7ff32e400640 1 --2- 172.21.15.6:0/380979319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3280d05c0 0x7ff3280d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.596+0000 7ff32da00640 1 --2- 172.21.15.6:0/380979319 >> v2:172.21.15.178:3300/0 conn(0x7ff32815e070 0x7ff3280d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.596+0000 7ff32d000640 1 -- 172.21.15.6:0/380979319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3280d05c0 msgr2=0x7ff3280d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.596+0000 7ff32d000640 1 --2- 172.21.15.6:0/380979319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3280d05c0 0x7ff3280d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.596+0000 7ff32d000640 1 -- 172.21.15.6:0/380979319 >> v2:172.21.15.178:3300/0 conn(0x7ff32815e070 msgr2=0x7ff3280d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.596+0000 7ff32d000640 1 --2- 172.21.15.6:0/380979319 >> v2:172.21.15.178:3300/0 conn(0x7ff32815e070 0x7ff3280d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.596+0000 7ff32d000640 1 -- 172.21.15.6:0/380979319 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280d2160 con 0x7ff3280cf410 2024-03-06T07:35:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.597+0000 7ff32d000640 1 --2- 172.21.15.6:0/380979319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280cf410 0x7ff3280cf870 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff32808d270 tx=0x7ff31402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b236a24b78f6440 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.597+0000 7ff327e00640 1 -- 172.21.15.6:0/380979319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314035020 con 0x7ff3280cf410 2024-03-06T07:35:48.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.597+0000 7ff327e00640 1 -- 172.21.15.6:0/380979319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31402d040 con 0x7ff3280cf410 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.597+0000 7ff327e00640 1 -- 172.21.15.6:0/380979319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314031a80 con 0x7ff3280cf410 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 -- 172.21.15.6:0/380979319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280cf410 msgr2=0x7ff3280cf870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 --2- 172.21.15.6:0/380979319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280cf410 0x7ff3280cf870 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff32808d270 tx=0x7ff31402b900 comp rx=0 tx=0).stop 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 -- 172.21.15.6:0/380979319 shutdown_connections 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 --2- 172.21.15.6:0/380979319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff3280d05c0 0x7ff3280d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 --2- 172.21.15.6:0/380979319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280cf410 0x7ff3280cf870 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 --2- 172.21.15.6:0/380979319 >> v2:172.21.15.178:3300/0 conn(0x7ff32815e070 0x7ff3280d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 -- 172.21.15.6:0/380979319 >> 172.21.15.6:0/380979319 conn(0x7ff32808d2a0 msgr2=0x7ff3280d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 -- 172.21.15.6:0/380979319 shutdown_connections 2024-03-06T07:35:48.601 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.598+0000 7ff32ee00640 1 -- 172.21.15.6:0/380979319 wait complete. 2024-03-06T07:35:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.601+0000 7ff32ee00640 1 Processor -- start 2024-03-06T07:35:48.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.602+0000 7ff32ee00640 1 -- start start 2024-03-06T07:35:48.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.603+0000 7ff32ee00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3280cf410 0x7ff3281551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 0x7ff3280c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32ee00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff32815e070 0x7ff32815c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3280d4730 con 0x7ff3280d05c0 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32ee00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3280d45b0 con 0x7ff3280cf410 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32ee00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3280b3630 con 0x7ff32815e070 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 0x7ff3280c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 0x7ff3280c7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32890/0 (socket says 172.21.15.6:32890) 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 -- 172.21.15.6:0/1130835101 learned_addr learned my addr 172.21.15.6:0/1130835101 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32da00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3280cf410 0x7ff3281551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32e400640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff32815e070 0x7ff32815c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 -- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff32815e070 msgr2=0x7ff32815c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff32815e070 0x7ff32815c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 -- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3280cf410 msgr2=0x7ff3281551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:48.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.604+0000 7ff32d000640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3280cf410 0x7ff3281551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:48.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.605+0000 7ff32d000640 1 -- 172.21.15.6:0/1130835101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3280b1dd0 con 0x7ff3280d05c0 2024-03-06T07:35:48.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.605+0000 7ff32da00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3280cf410 0x7ff3281551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.605+0000 7ff32d000640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 0x7ff3280c7ca0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff3140097c0 tx=0x7ff3140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.605+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314040070 con 0x7ff3280d05c0 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.605+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff314039bf0 con 0x7ff3280d05c0 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.606+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3280b7ff0 con 0x7ff3280d05c0 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.606+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff314035040 con 0x7ff3280d05c0 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.606+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3280b85b0 con 0x7ff3280d05c0 2024-03-06T07:35:48.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.607+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32804f330 con 0x7ff3280d05c0 2024-03-06T07:35:48.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.609+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff31404b060 con 0x7ff3280d05c0 2024-03-06T07:35:48.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.610+0000 7ff325600640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff30806b4d0 0x7ff30806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:48.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.610+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3140abe20 con 0x7ff3280d05c0 2024-03-06T07:35:48.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.611+0000 7ff32da00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff30806b4d0 0x7ff30806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:48.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.612+0000 7ff32da00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff30806b4d0 0x7ff30806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff3280b2b40 tx=0x7ff31c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:48.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.615+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff31407ac00 con 0x7ff3280d05c0 2024-03-06T07:35:48.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:48.896+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7ff3280ce190 con 0x7ff3280d05c0 2024-03-06T07:35:49.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.333+0000 7ff325600640 1 -- 172.21.15.6:0/1130835101 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6044 v97) v1 ==== 95+0+0 (secure 0 0 0) 0x7ff31407a500 con 0x7ff3280d05c0 2024-03-06T07:35:49.336 INFO:teuthology.orchestra.run.smithi006.stderr:failed mds gid 6044 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff30806b4d0 msgr2=0x7ff30806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff30806b4d0 0x7ff30806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff3280b2b40 tx=0x7ff31c008040 comp rx=0 tx=0).stop 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 msgr2=0x7ff3280c7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 0x7ff3280c7ca0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff3140097c0 tx=0x7ff3140046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 shutdown_connections 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff30806b4d0 0x7ff30806d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff32815e070 0x7ff32815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3280d05c0 0x7ff3280c7ca0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 --2- 172.21.15.6:0/1130835101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff3280cf410 0x7ff3281551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.336+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 >> 172.21.15.6:0/1130835101 conn(0x7ff32808d2a0 msgr2=0x7ff32806cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.337+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 shutdown_connections 2024-03-06T07:35:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.337+0000 7ff32ee00640 1 -- 172.21.15.6:0/1130835101 wait complete. 2024-03-06T07:35:49.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:49.359 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:35:49.359 DEBUG:teuthology.orchestra.run.smithi006:> mkdir pin 2024-03-06T07:35:49.359 DEBUG:teuthology.orchestra.run.smithi006:> touch pin/placeholder 2024-03-06T07:35:49.359 DEBUG:teuthology.orchestra.run.smithi006:> ') 2024-03-06T07:35:49.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:49.430 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:35:49.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:49.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.744+0000 7f7b93200640 1 Processor -- start 2024-03-06T07:35:49.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.744+0000 7f7b93200640 1 -- start start 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b93200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b93200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b940cf460 0x7f7b940b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b93200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b940d1120 con 0x7f7b9415dec0 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b93200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b940b0ea0 con 0x7f7b940cf460 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b93200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b940d1400 con 0x7f7b9415e2a0 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b92800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b940cf460 0x7f7b940b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b92800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7b940cf460 0x7f7b940b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32902/0 (socket says 172.21.15.6:32902) 2024-03-06T07:35:49.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b92800640 1 -- 172.21.15.6:0/2263235769 learned_addr learned my addr 172.21.15.6:0/2263235769 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:49.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.746+0000 7f7b91400640 1 --2- 172.21.15.6:0/2263235769 >> v2:172.21.15.178:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.747+0000 7f7b92800640 1 -- 172.21.15.6:0/2263235769 >> v2:172.21.15.178:3300/0 conn(0x7f7b9415e2a0 msgr2=0x7f7b940cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.747+0000 7f7b92800640 1 --2- 172.21.15.6:0/2263235769 >> v2:172.21.15.178:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.747+0000 7f7b92800640 1 -- 172.21.15.6:0/2263235769 >> v1:172.21.15.178:6789/0 conn(0x7f7b9415dec0 legacy=0x7f7b940ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:49.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.747+0000 7f7b92800640 1 -- 172.21.15.6:0/2263235769 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940b0b30 con 0x7f7b940cf460 2024-03-06T07:35:49.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.748+0000 7f7b92800640 1 --2- 172.21.15.6:0/2263235769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b940cf460 0x7f7b940b1330 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7b940b5800 tx=0x7f7b8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de41d0ae6cd31269 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.749+0000 7f7b90a00640 1 -- 172.21.15.6:0/2263235769 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84035020 con 0x7f7b940cf460 2024-03-06T07:35:49.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.749+0000 7f7b90a00640 1 -- 172.21.15.6:0/2263235769 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b84031870 con 0x7f7b940cf460 2024-03-06T07:35:49.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.749+0000 7f7b90a00640 1 -- 172.21.15.6:0/2263235769 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84031c40 con 0x7f7b940cf460 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 -- 172.21.15.6:0/2263235769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b940cf460 msgr2=0x7f7b940b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 --2- 172.21.15.6:0/2263235769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b940cf460 0x7f7b940b1330 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7b940b5800 tx=0x7f7b8402b910 comp rx=0 tx=0).stop 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 -- 172.21.15.6:0/2263235769 shutdown_connections 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 --2- 172.21.15.6:0/2263235769 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b940cf460 0x7f7b940b1330 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 --2- 172.21.15.6:0/2263235769 >> v2:172.21.15.178:3300/0 conn(0x7f7b9415e2a0 0x7f7b940cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 -- 172.21.15.6:0/2263235769 >> 172.21.15.6:0/2263235769 conn(0x7f7b940d3d40 msgr2=0x7f7b940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 -- 172.21.15.6:0/2263235769 shutdown_connections 2024-03-06T07:35:49.752 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.750+0000 7f7b93200640 1 -- 172.21.15.6:0/2263235769 wait complete. 2024-03-06T07:35:49.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.752+0000 7f7b93200640 1 Processor -- start 2024-03-06T07:35:49.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.752+0000 7f7b93200640 1 -- start start 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b93200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b940cf460 0x7f7b9414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b93200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 0x7f7b940cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b93200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b9415e2a0 0x7f7b940b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b93200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940d0ec0 con 0x7f7b9415dec0 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b93200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940d0d40 con 0x7f7b940cf460 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b93200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940b1870 con 0x7f7b9415e2a0 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 0x7f7b940cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 0x7f7b940cc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:32908/0 (socket says 172.21.15.6:32908) 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 -- 172.21.15.6:0/368836393 learned_addr learned my addr 172.21.15.6:0/368836393 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91e00640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b940cf460 0x7f7b9414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 -- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b9415e2a0 msgr2=0x7f7b940b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b92800640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b9415e2a0 0x7f7b940b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.756 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b9415e2a0 0x7f7b940b72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 -- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b940cf460 msgr2=0x7f7b9414eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b940cf460 0x7f7b9414eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.753+0000 7f7b91400640 1 -- 172.21.15.6:0/368836393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940cdbe0 con 0x7f7b9415dec0 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.754+0000 7f7b91400640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 0x7f7b940cc080 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000d5f0 tx=0x7f7b8000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.754+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800125f0 con 0x7f7b9415dec0 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.754+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b80019070 con 0x7f7b9415dec0 2024-03-06T07:35:49.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.754+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b940ccb40 con 0x7f7b9415dec0 2024-03-06T07:35:49.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.754+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800129d0 con 0x7f7b9415dec0 2024-03-06T07:35:49.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.754+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b940cd3e0 con 0x7f7b9415dec0 2024-03-06T07:35:49.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.756+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9404f330 con 0x7f7b9415dec0 2024-03-06T07:35:49.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.757+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b80012b70 con 0x7f7b9415dec0 2024-03-06T07:35:49.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.758+0000 7f7b8a000640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:49.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.758+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b8008a5b0 con 0x7f7b9415dec0 2024-03-06T07:35:49.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.759+0000 7f7b91e00640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:49.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.760+0000 7f7b91e00640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b7406b500 0x7f7b7406d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c001db0 tx=0x7f7b8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:49.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:49.767+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b80059420 con 0x7f7b9415dec0 2024-03-06T07:35:50.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.020+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b941615b0 con 0x7f7b9415dec0 2024-03-06T07:35:50.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.021+0000 7f7b8a000640 1 -- 172.21.15.6:0/368836393 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3408 (secure 0 0 0) 0x7f7b80058d20 con 0x7f7b9415dec0 2024-03-06T07:35:50.024 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:50.024 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:41.136852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[6871,8355]},"id":9}]} 2024-03-06T07:35:50.024 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 97 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b7406b500 msgr2=0x7f7b7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b7406b500 0x7f7b7406d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c001db0 tx=0x7f7b8c009040 comp rx=0 tx=0).stop 2024-03-06T07:35:50.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 msgr2=0x7f7b940cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 0x7f7b940cc080 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000d5f0 tx=0x7f7b8000dab0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 shutdown_connections 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7b9415e2a0 0x7f7b940b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7b9415dec0 0x7f7b940cc080 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 --2- 172.21.15.6:0/368836393 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b940cf460 0x7f7b9414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 >> 172.21.15.6:0/368836393 conn(0x7f7b940d3d40 msgr2=0x7f7b940ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 shutdown_connections 2024-03-06T07:35:50.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:50.024+0000 7f7b93200640 1 -- 172.21.15.6:0/368836393 wait complete. 2024-03-06T07:35:50.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:35:50.344 INFO:teuthology.orchestra.run.smithi006.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:35:50.345 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:35:50.345 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:35:52.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:52.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.571+0000 7f9050200640 1 Processor -- start 2024-03-06T07:35:52.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.572+0000 7f9050200640 1 -- start start 2024-03-06T07:35:52.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.573+0000 7f9050200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f904815e000 0x7f90480ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.573+0000 7f9050200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90480cf090 con 0x7f9048161c80 2024-03-06T07:35:52.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.573+0000 7f9050200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90480d0650 con 0x7f90480b4450 2024-03-06T07:35:52.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.573+0000 7f9050200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904806ef40 con 0x7f904815e000 2024-03-06T07:35:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.574+0000 7f904e200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9048161c80 0x7f90481620b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36254/0 (socket says 172.21.15.6:36254) 2024-03-06T07:35:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.574+0000 7f904e200640 1 -- 172.21.15.6:0/2613738537 learned_addr learned my addr 172.21.15.6:0/2613738537 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.574+0000 7f904d800640 1 --2- 172.21.15.6:0/2613738537 >> v2:172.21.15.178:3300/0 conn(0x7f904815e000 0x7f90480ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.574+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531493026 0 0) 0x7f90480cf090 con 0x7f9048161c80 2024-03-06T07:35:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.574+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9030003520 con 0x7f9048161c80 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 398747927 0 0) 0x7f9030003520 con 0x7f9048161c80 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 >> v2:172.21.15.178:3300/0 conn(0x7f904815e000 msgr2=0x7f90480ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 --2- 172.21.15.6:0/2613738537 >> v2:172.21.15.178:3300/0 conn(0x7f904815e000 0x7f90480ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 >> v1:172.21.15.178:6789/0 conn(0x7f90480b4450 legacy=0x7f90480ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90480d0930 con 0x7f9048161c80 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f90400019f0 con 0x7f9048161c80 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 >> v1:172.21.15.6:6789/0 conn(0x7f9048161c80 legacy=0x7f90481620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f9030003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9030004800 0x7f9030004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90300051c0 0x7f903000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904806ef40 con 0x7f9030003a20 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903000c0d0 con 0x7f9030004800 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903000c2d0 con 0x7f90300051c0 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.575+0000 7f904d800640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f9030003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904ce00640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9030004800 0x7f9030004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904e200640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90300051c0 0x7f903000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904d800640 1 -- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90300051c0 msgr2=0x7f903000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904d800640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90300051c0 0x7f903000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904d800640 1 -- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9030004800 msgr2=0x7f9030004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:52.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904d800640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9030004800 0x7f9030004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904d800640 1 -- 172.21.15.6:0/2613738537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9030003520 con 0x7f9030003a20 2024-03-06T07:35:52.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.576+0000 7f904e200640 1 -- 172.21.15.6:0/2613738537 reap_dead start 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.578+0000 7f904d800640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f9030003e40 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f903c009ac0 tx=0x7f9030005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393eee98c04a013f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.578+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c039070 con 0x7f9030003a20 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.578+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c02c6b0 con 0x7f9030003a20 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.578+0000 7f9047e00640 1 -- 172.21.15.6:0/2613738537 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c009690 con 0x7f9030003a20 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 -- 172.21.15.6:0/2613738537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 msgr2=0x7f9030003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f9030003e40 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f903c009ac0 tx=0x7f9030005830 comp rx=0 tx=0).stop 2024-03-06T07:35:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 -- 172.21.15.6:0/2613738537 shutdown_connections 2024-03-06T07:35:52.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 --2- 172.21.15.6:0/2613738537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f9030003e40 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 -- 172.21.15.6:0/2613738537 >> 172.21.15.6:0/2613738537 conn(0x7f90480b38a0 msgr2=0x7f90480b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:52.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 -- 172.21.15.6:0/2613738537 shutdown_connections 2024-03-06T07:35:52.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.579+0000 7f9050200640 1 -- 172.21.15.6:0/2613738537 wait complete. 2024-03-06T07:35:52.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.582+0000 7f9050200640 1 Processor -- start 2024-03-06T07:35:52.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.582+0000 7f9050200640 1 -- start start 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f9050200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f90481e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f9050200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90481e8010 0x7f904814c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f9050200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f904814c570 0x7f904814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f9050200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90480d0280 con 0x7f9030003a20 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f9050200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904806f190 con 0x7f904814c570 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f9050200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90480d0400 con 0x7f90481e8010 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f90481e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f90481e7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60294/0 (socket says 172.21.15.6:60294) 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 -- 172.21.15.6:0/202341323 learned_addr learned my addr 172.21.15.6:0/202341323 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904ce00640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90481e8010 0x7f904814c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904e200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f904814c570 0x7f904814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 -- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90481e8010 msgr2=0x7f904814c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90481e8010 0x7f904814c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 -- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f904814c570 msgr2=0x7f904814c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f904814c570 0x7f904814c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.584+0000 7f904d800640 1 -- 172.21.15.6:0/202341323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9048160da0 con 0x7f9030003a20 2024-03-06T07:35:52.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.585+0000 7f904d800640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f90481e7ad0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f903c037040 tx=0x7f903c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.585+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c042020 con 0x7f9030003a20 2024-03-06T07:35:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.585+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c03a040 con 0x7f9030003a20 2024-03-06T07:35:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.585+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c039070 con 0x7f9030003a20 2024-03-06T07:35:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.585+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9048160ac0 con 0x7f9030003a20 2024-03-06T07:35:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.585+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904814a390 con 0x7f9030003a20 2024-03-06T07:35:52.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.587+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904804f3a0 con 0x7f9030003a20 2024-03-06T07:35:52.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.588+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f903c045020 con 0x7f9030003a20 2024-03-06T07:35:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.589+0000 7f9045600640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:52.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.589+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f903c0ab5b0 con 0x7f9030003a20 2024-03-06T07:35:52.592 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.590+0000 7f904ce00640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:52.593 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.591+0000 7f904ce00640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f902806b4d0 0x7f902806d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9048162b90 tx=0x7f9034009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:52.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.593+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f903c04c070 con 0x7f9030003a20 2024-03-06T07:35:52.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.871+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90481e83f0 con 0x7f9030003a20 2024-03-06T07:35:52.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.871+0000 7f9045600640 1 -- 172.21.15.6:0/202341323 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+3408 (secure 0 0 0) 0x7f903c07a430 con 0x7f9030003a20 2024-03-06T07:35:52.875 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:52.875 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":97,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:41.136852+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[6871,8355]},"id":9}]} 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 97 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f902806b4d0 msgr2=0x7f902806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f902806b4d0 0x7f902806d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9048162b90 tx=0x7f9034009040 comp rx=0 tx=0).stop 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 msgr2=0x7f90481e7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f90481e7ad0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f903c037040 tx=0x7f903c004730 comp rx=0 tx=0).stop 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 shutdown_connections 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f904814c570 0x7f904814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90481e8010 0x7f904814c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 --2- 172.21.15.6:0/202341323 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9030003a20 0x7f90481e7ad0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:52.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 >> 172.21.15.6:0/202341323 conn(0x7f90480b38a0 msgr2=0x7f90480cc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:52.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 shutdown_connections 2024-03-06T07:35:52.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:52.874+0000 7f9050200640 1 -- 172.21.15.6:0/202341323 wait complete. 2024-03-06T07:35:52.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:35:53.126 INFO:teuthology.orchestra.run.smithi006.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:35:53.126 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:35:53.127 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:35:53.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:53.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:35:55.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.346+0000 7fb389a00640 1 Processor -- start 2024-03-06T07:35:55.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.346+0000 7fb389a00640 1 -- start start 2024-03-06T07:35:55.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.349+0000 7fb389a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb3840cf5c0 0x7fb3840cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.349+0000 7fb389a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3840b4860 con 0x7fb384071050 2024-03-06T07:35:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.349+0000 7fb389a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3840b4b40 con 0x7fb3840b3c90 2024-03-06T07:35:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.349+0000 7fb389a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3840ce160 con 0x7fb3840cf5c0 2024-03-06T07:35:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.350+0000 7fb382000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb3840cf5c0 0x7fb3840cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.350+0000 7fb382a00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fb3840b3c90 0x7fb3840b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:39884/0 (socket says 172.21.15.6:39884) 2024-03-06T07:35:55.352 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.350+0000 7fb382a00640 1 -- 172.21.15.6:0/2405547997 learned_addr learned my addr 172.21.15.6:0/2405547997 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.351+0000 7fb382000640 1 -- 172.21.15.6:0/2405547997 >> v1:172.21.15.178:6789/0 conn(0x7fb3840b3c90 legacy=0x7fb3840b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.351+0000 7fb382000640 1 -- 172.21.15.6:0/2405547997 >> v1:172.21.15.6:6789/0 conn(0x7fb384071050 legacy=0x7fb384071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.351+0000 7fb382000640 1 -- 172.21.15.6:0/2405547997 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3840b4e20 con 0x7fb3840cf5c0 2024-03-06T07:35:55.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.352+0000 7fb382000640 1 --2- 172.21.15.6:0/2405547997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840cf5c0 0x7fb3840cdc20 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0097a0 tx=0x7fb36c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42750e011c3bcea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.353+0000 7fb381600640 1 -- 172.21.15.6:0/2405547997 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c02f810 con 0x7fb3840cf5c0 2024-03-06T07:35:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.353+0000 7fb381600640 1 -- 172.21.15.6:0/2405547997 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c02f9b0 con 0x7fb3840cf5c0 2024-03-06T07:35:55.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.353+0000 7fb381600640 1 -- 172.21.15.6:0/2405547997 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c02fd80 con 0x7fb3840cf5c0 2024-03-06T07:35:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.354+0000 7fb389a00640 1 -- 172.21.15.6:0/2405547997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840cf5c0 msgr2=0x7fb3840cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.354+0000 7fb389a00640 1 --2- 172.21.15.6:0/2405547997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840cf5c0 0x7fb3840cdc20 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0097a0 tx=0x7fb36c02b9f0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.354+0000 7fb389a00640 1 -- 172.21.15.6:0/2405547997 shutdown_connections 2024-03-06T07:35:55.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.354+0000 7fb389a00640 1 --2- 172.21.15.6:0/2405547997 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840cf5c0 0x7fb3840cdc20 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.354+0000 7fb389a00640 1 -- 172.21.15.6:0/2405547997 >> 172.21.15.6:0/2405547997 conn(0x7fb38406afb0 msgr2=0x7fb38406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.354+0000 7fb389a00640 1 -- 172.21.15.6:0/2405547997 shutdown_connections 2024-03-06T07:35:55.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.355+0000 7fb389a00640 1 -- 172.21.15.6:0/2405547997 wait complete. 2024-03-06T07:35:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.357+0000 7fb389a00640 1 Processor -- start 2024-03-06T07:35:55.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.357+0000 7fb389a00640 1 -- start start 2024-03-06T07:35:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb389a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384071050 0x7fb3841445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb389a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 0x7fb38409aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb389a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840cf5c0 0x7fb3840ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb389a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384071ba0 con 0x7fb3840b3c90 2024-03-06T07:35:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb389a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38415b340 con 0x7fb384071050 2024-03-06T07:35:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb389a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb384071d20 con 0x7fb3840cf5c0 2024-03-06T07:35:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb382000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 0x7fb38409aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb382000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 0x7fb38409aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60312/0 (socket says 172.21.15.6:60312) 2024-03-06T07:35:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb382000640 1 -- 172.21.15.6:0/3072749070 learned_addr learned my addr 172.21.15.6:0/3072749070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:55.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.358+0000 7fb382a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384071050 0x7fb3841445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb383400640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840cf5c0 0x7fb3840ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb382000640 1 -- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840cf5c0 msgr2=0x7fb3840ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb382000640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840cf5c0 0x7fb3840ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb382000640 1 -- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384071050 msgr2=0x7fb3841445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb382000640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384071050 0x7fb3841445d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb382000640 1 -- 172.21.15.6:0/3072749070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38409b360 con 0x7fb3840b3c90 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.359+0000 7fb382000640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 0x7fb38409aa10 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb36c02c000 tx=0x7fb36c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.360+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c046070 con 0x7fb3840b3c90 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.360+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c037c40 con 0x7fb3840b3c90 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.360+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c02fe60 con 0x7fb3840b3c90 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.360+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38409b640 con 0x7fb3840b3c90 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.360+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38409bc00 con 0x7fb3840b3c90 2024-03-06T07:35:55.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.361+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb384069510 con 0x7fb3840b3c90 2024-03-06T07:35:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.363+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb36c049070 con 0x7fb3840b3c90 2024-03-06T07:35:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.363+0000 7fb37aa00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb36006b500 0x7fb36006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:55.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.365+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb36c0abe00 con 0x7fb3840b3c90 2024-03-06T07:35:55.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.369+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb36c033030 con 0x7fb3840b3c90 2024-03-06T07:35:55.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.371+0000 7fb382a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb36006b500 0x7fb36006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.372+0000 7fb382a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb36006b500 0x7fb36006d9c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb374005c00 tx=0x7fb374009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:55.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.607+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb38404f3a0 con 0x7fb3840b3c90 2024-03-06T07:35:55.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.608+0000 7fb37aa00640 1 -- 172.21.15.6:0/3072749070 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+3409 (secure 0 0 0) 0x7fb36c07abe0 con 0x7fb3840b3c90 2024-03-06T07:35:55.610 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:55.610 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":98,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:54.676205+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:active","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355,6871]},"id":9}]} 2024-03-06T07:35:55.610 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 98 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb36006b500 msgr2=0x7fb36006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb36006b500 0x7fb36006d9c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb374005c00 tx=0x7fb374009040 comp rx=0 tx=0).stop 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 msgr2=0x7fb38409aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 0x7fb38409aa10 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb36c02c000 tx=0x7fb36c004730 comp rx=0 tx=0).stop 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 shutdown_connections 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb36006b500 0x7fb36006d9c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb3840cf5c0 0x7fb3840ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb3840b3c90 0x7fb38409aa10 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 --2- 172.21.15.6:0/3072749070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb384071050 0x7fb3841445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:55.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.610+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 >> 172.21.15.6:0/3072749070 conn(0x7fb38406afb0 msgr2=0x7fb38406cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:55.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.611+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 shutdown_connections 2024-03-06T07:35:55.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:55.611+0000 7fb389a00640 1 -- 172.21.15.6:0/3072749070 wait complete. 2024-03-06T07:35:55.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:35:55.836 INFO:teuthology.orchestra.run.smithi006.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:35:55.836 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:35:55.836 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:35:55.837 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:35:55.837 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> stdin-killer -- python3 -c ' 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> import fcntl 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> import sys 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> import time 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> import os 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:35:55.882 DEBUG:teuthology.orchestra.run.smithi006:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> while True: 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> try: 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> if os.read(0, 4096) == b"": 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> break 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> except BlockingIOError: 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> pass 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> time.sleep(2) 2024-03-06T07:35:55.883 DEBUG:teuthology.orchestra.run.smithi006:> ' 2024-03-06T07:35:55.929 DEBUG:teuthology.orchestra.run.smithi006:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:35:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:35:55.972 INFO:teuthology.orchestra.run.smithi006.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:35:56.048 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:35:56.973 DEBUG:teuthology.orchestra.run.smithi006:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:35:57.033 INFO:teuthology.orchestra.run.smithi006.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:35:57.033 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-06T07:35:57.033 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 28h/40d Inode: 2199023255552 Links: 2 2024-03-06T07:35:57.033 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:35:57.033 INFO:teuthology.orchestra.run.smithi006.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-06T07:35:57.033 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-06 07:35:56.042298781 +0000 2024-03-06T07:35:57.034 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-06 07:35:56.042298781 +0000 2024-03-06T07:35:57.034 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-06 07:35:56.042298781 +0000 2024-03-06T07:35:57.034 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-06 07:35:56.042298781 +0000 2024-03-06T07:35:57.034 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:35:57.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:35:57.034 DEBUG:teuthology.orchestra.run.smithi006:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:35:57.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:35:57.412 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:35:57.412 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:35:57.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:35:57.709 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:35:57.710 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:35:57.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:35:58.035 INFO:teuthology.orchestra.run.smithi006.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:35:58.036 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:35:58.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:35:58.049 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:35:58.280 INFO:teuthology.orchestra.run.smithi006.stdout:{"message":"","return_code":0} 2024-03-06T07:35:58.280 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:35:58.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:35:58.566 INFO:teuthology.orchestra.run.smithi006.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":378,"bytes":65362}},"duration":0.012224135000000001} 2024-03-06T07:35:58.566 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65362, "items": 378 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.012224135, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:35:58.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:35:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.791+0000 7f31d9a00640 1 Processor -- start 2024-03-06T07:35:58.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.791+0000 7f31d9a00640 1 -- start start 2024-03-06T07:35:58.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.793+0000 7f31d9a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f31d40d4a00 0x7f31d40d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.793+0000 7f31d9a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40d0080 con 0x7f31d415e280 2024-03-06T07:35:58.795 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.793+0000 7f31d9a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d40cf500 con 0x7f31d40d2990 2024-03-06T07:35:58.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.793+0000 7f31d9a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40cf7e0 con 0x7f31d40d4a00 2024-03-06T07:35:58.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.793+0000 7f31d2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31d415e280 0x7f31d40d2280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36272/0 (socket says 172.21.15.6:36272) 2024-03-06T07:35:58.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.793+0000 7f31d2a00640 1 -- 172.21.15.6:0/3006605477 learned_addr learned my addr 172.21.15.6:0/3006605477 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d3400640 1 --2- 172.21.15.6:0/3006605477 >> v2:172.21.15.178:3300/0 conn(0x7f31d40d4a00 0x7f31d40d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963705364 0 0) 0x7f31d40d0080 con 0x7f31d415e280 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b8003520 con 0x7f31d415e280 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2346418984 0 0) 0x7f31b8003520 con 0x7f31d415e280 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 >> v2:172.21.15.178:3300/0 conn(0x7f31d40d4a00 msgr2=0x7f31d40d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 --2- 172.21.15.6:0/3006605477 >> v2:172.21.15.178:3300/0 conn(0x7f31d40d4a00 0x7f31d40d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 >> v1:172.21.15.178:6789/0 conn(0x7f31d40d2990 legacy=0x7f31d40cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40cfac0 con 0x7f31d415e280 2024-03-06T07:35:58.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f31c4001670 con 0x7f31d415e280 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 >> v1:172.21.15.6:6789/0 conn(0x7f31d415e280 legacy=0x7f31d40d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.794+0000 7f31d1600640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b80039b0 0x7f31b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d1600640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31b8004800 0x7f31b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d1600640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40cf7e0 con 0x7f31b80051c0 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b800c160 con 0x7f31b80039b0 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b800c360 con 0x7f31b8004800 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d3400640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d2a00640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b80039b0 0x7f31b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.795+0000 7f31d2000640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31b8004800 0x7f31b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.796+0000 7f31d3400640 1 -- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31b8004800 msgr2=0x7f31b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.796+0000 7f31d3400640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31b8004800 0x7f31b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.796+0000 7f31d3400640 1 -- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b80039b0 msgr2=0x7f31b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.796+0000 7f31d3400640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31b80039b0 0x7f31b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.796+0000 7f31d3400640 1 -- 172.21.15.6:0/3006605477 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8003520 con 0x7f31b80051c0 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.797+0000 7f31d3400640 1 -- 172.21.15.6:0/3006605477 reap_dead start 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.797+0000 7f31d3400640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31b800b9c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009f90 tx=0x7f31b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa6bffec874c2c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.797+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc039070 con 0x7f31b80051c0 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.797+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc02cc20 con 0x7f31b80051c0 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.797+0000 7f31d1600640 1 -- 172.21.15.6:0/3006605477 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc009c10 con 0x7f31b80051c0 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 -- 172.21.15.6:0/3006605477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 msgr2=0x7f31b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31b800b9c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009f90 tx=0x7f31b8005830 comp rx=0 tx=0).stop 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 -- 172.21.15.6:0/3006605477 shutdown_connections 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 --2- 172.21.15.6:0/3006605477 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31b800b9c0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 -- 172.21.15.6:0/3006605477 >> 172.21.15.6:0/3006605477 conn(0x7f31d40d3ee0 msgr2=0x7f31d40d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:58.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 -- 172.21.15.6:0/3006605477 shutdown_connections 2024-03-06T07:35:58.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.798+0000 7f31d9a00640 1 -- 172.21.15.6:0/3006605477 wait complete. 2024-03-06T07:35:58.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.800+0000 7f31d9a00640 1 Processor -- start 2024-03-06T07:35:58.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.800+0000 7f31d9a00640 1 -- start start 2024-03-06T07:35:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31d40b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d9a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31d41f8190 0x7f31d409e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d9a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31d41f7510 0x7f31d41f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d9a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d415e660 con 0x7f31b80051c0 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d9a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40cb0e0 con 0x7f31d41f7510 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d9a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40d1cc0 con 0x7f31d41f8190 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31d40b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31d40b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60338/0 (socket says 172.21.15.6:60338) 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.802+0000 7f31d2a00640 1 -- 172.21.15.6:0/1397423270 learned_addr learned my addr 172.21.15.6:0/1397423270 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2000640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31d41f8190 0x7f31d409e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d3400640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31d41f7510 0x7f31d41f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2a00640 1 -- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31d41f8190 msgr2=0x7f31d409e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31d41f8190 0x7f31d409e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2a00640 1 -- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31d41f7510 msgr2=0x7f31d41f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31d41f7510 0x7f31d41f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2a00640 1 -- 172.21.15.6:0/1397423270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40b4870 con 0x7f31b80051c0 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d3400640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31d41f7510 0x7f31d41f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:58.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2000640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31d41f8190 0x7f31d409e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d2a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31d40b4330 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f31c4007340 tx=0x7f31c4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c400f940 con 0x7f31b80051c0 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4018070 con 0x7f31b80051c0 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d40a09a0 con 0x7f31b80051c0 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.803+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c400fd00 con 0x7f31b80051c0 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.804+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d40a0f60 con 0x7f31b80051c0 2024-03-06T07:35:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.805+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d4072a50 con 0x7f31b80051c0 2024-03-06T07:35:58.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.806+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31c401d460 con 0x7f31b80051c0 2024-03-06T07:35:58.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.807+0000 7f31caa00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31b406b500 0x7f31b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:58.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.807+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31c408a1d0 con 0x7f31b80051c0 2024-03-06T07:35:58.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.807+0000 7f31d2000640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31b406b500 0x7f31b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:58.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.809+0000 7f31d2000640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31b406b500 0x7f31b406d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f31c0000c00 tx=0x7f31c0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:58.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:58.811+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31d4072a50 con 0x7f31b80051c0 2024-03-06T07:35:59.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.051+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f31d40ccff0 con 0x7f31b80051c0 2024-03-06T07:35:59.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.305+0000 7f31caa00640 1 -- 172.21.15.6:0/1397423270 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v99) v1 ==== 105+0+0 (secure 0 0 0) 0x7f31c4059050 con 0x7f31b80051c0 2024-03-06T07:35:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31b406b500 msgr2=0x7f31b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31b406b500 0x7f31b406d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f31c0000c00 tx=0x7f31c0009040 comp rx=0 tx=0).stop 2024-03-06T07:35:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 msgr2=0x7f31d40b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31d40b4330 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f31c4007340 tx=0x7f31c4007800 comp rx=0 tx=0).stop 2024-03-06T07:35:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 shutdown_connections 2024-03-06T07:35:59.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f31b406b500 0x7f31b406d9c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f31d41f7510 0x7f31d41f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f31d41f8190 0x7f31d409e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 --2- 172.21.15.6:0/1397423270 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b80051c0 0x7f31d40b4330 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 >> 172.21.15.6:0/1397423270 conn(0x7f31d40d3ee0 msgr2=0x7f31d40d31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 shutdown_connections 2024-03-06T07:35:59.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.309+0000 7f31d9a00640 1 -- 172.21.15.6:0/1397423270 wait complete. 2024-03-06T07:35:59.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:35:59.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.543+0000 7fe94ea00640 1 Processor -- start 2024-03-06T07:35:59.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.544+0000 7fe94ea00640 1 -- start start 2024-03-06T07:35:59.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.545+0000 7fe94ea00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe948145300 0x7fe9480b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.545+0000 7fe94ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe948070f10 0x7fe9480cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.545+0000 7fe94ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9480714d0 con 0x7fe9480cce90 2024-03-06T07:35:59.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.545+0000 7fe94ea00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9480cd300 con 0x7fe948145300 2024-03-06T07:35:59.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.545+0000 7fe94ea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480717b0 con 0x7fe948070f10 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.546+0000 7fe94e000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe9480cce90 0x7fe9480cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36276/0 (socket says 172.21.15.6:36276) 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.546+0000 7fe94e000640 1 -- 172.21.15.6:0/3196756123 learned_addr learned my addr 172.21.15.6:0/3196756123 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.546+0000 7fe94d600640 1 --2- 172.21.15.6:0/3196756123 >> v2:172.21.15.178:3300/0 conn(0x7fe948145300 0x7fe9480b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.546+0000 7fe94cc00640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe948070f10 0x7fe9480cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.546+0000 7fe947e00640 1 -- 172.21.15.6:0/3196756123 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782826647 0 0) 0x7fe9480714d0 con 0x7fe9480cce90 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.546+0000 7fe947e00640 1 -- 172.21.15.6:0/3196756123 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe930003520 con 0x7fe9480cce90 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.547+0000 7fe94d600640 1 -- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe948070f10 msgr2=0x7fe9480cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.547+0000 7fe94d600640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe948070f10 0x7fe9480cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.547+0000 7fe947e00640 1 -- 172.21.15.6:0/3196756123 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 840420714 0 0) 0x7fe930003520 con 0x7fe9480cce90 2024-03-06T07:35:59.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.547+0000 7fe94d600640 1 -- 172.21.15.6:0/3196756123 >> v1:172.21.15.6:6789/0 conn(0x7fe9480cce90 legacy=0x7fe9480cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.547+0000 7fe94d600640 1 -- 172.21.15.6:0/3196756123 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe948071a90 con 0x7fe948145300 2024-03-06T07:35:59.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.547+0000 7fe94cc00640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe948070f10 0x7fe9480cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:35:59.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.548+0000 7fe94d600640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 0x7fe9480b49a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0097a0 tx=0x7fe93c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4236958b127c4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe947e00640 1 -- 172.21.15.6:0/3196756123 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c037070 con 0x7fe948145300 2024-03-06T07:35:59.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe947e00640 1 -- 172.21.15.6:0/3196756123 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c030510 con 0x7fe948145300 2024-03-06T07:35:59.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe947e00640 1 -- 172.21.15.6:0/3196756123 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9400019f0 con 0x7fe948145300 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe94ea00640 1 -- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 msgr2=0x7fe9480b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe94ea00640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 0x7fe9480b49a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0097a0 tx=0x7fe93c02b790 comp rx=0 tx=0).stop 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe94ea00640 1 -- 172.21.15.6:0/3196756123 shutdown_connections 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe94ea00640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe948070f10 0x7fe9480cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe94ea00640 1 --2- 172.21.15.6:0/3196756123 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 0x7fe9480b49a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.549+0000 7fe94ea00640 1 -- 172.21.15.6:0/3196756123 >> 172.21.15.6:0/3196756123 conn(0x7fe94806b3d0 msgr2=0x7fe94806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.550+0000 7fe94ea00640 1 -- 172.21.15.6:0/3196756123 shutdown_connections 2024-03-06T07:35:59.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.550+0000 7fe94ea00640 1 -- 172.21.15.6:0/3196756123 wait complete. 2024-03-06T07:35:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.551+0000 7fe94ea00640 1 Processor -- start 2024-03-06T07:35:59.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.552+0000 7fe94ea00640 1 -- start start 2024-03-06T07:35:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.552+0000 7fe94ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 0x7fe94813fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.552+0000 7fe94ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9480cce90 0x7fe9481408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 0x7fe94813c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9481456e0 con 0x7fe948070f10 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94ea00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480b4ee0 con 0x7fe948145300 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94ea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94815d580 con 0x7fe9480cce90 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 0x7fe94813fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 0x7fe94813fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60350/0 (socket says 172.21.15.6:60350) 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 -- 172.21.15.6:0/1545942905 learned_addr learned my addr 172.21.15.6:0/1545942905 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 -- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9480cce90 msgr2=0x7fe9481408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9480cce90 0x7fe9481408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 -- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 msgr2=0x7fe94813c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 0x7fe94813c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.553+0000 7fe94d600640 1 -- 172.21.15.6:0/1545942905 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94813e7e0 con 0x7fe948070f10 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.554+0000 7fe94d600640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 0x7fe94813fb60 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fe93c000c00 tx=0x7fe93c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.554+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c041020 con 0x7fe948070f10 2024-03-06T07:35:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.554+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c03c870 con 0x7fe948070f10 2024-03-06T07:35:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.554+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c037040 con 0x7fe948070f10 2024-03-06T07:35:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.554+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94813eac0 con 0x7fe948070f10 2024-03-06T07:35:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.554+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94813f080 con 0x7fe948070f10 2024-03-06T07:35:59.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.555+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe94804f330 con 0x7fe948070f10 2024-03-06T07:35:59.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.557+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe93c044020 con 0x7fe948070f10 2024-03-06T07:35:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.558+0000 7fe945600640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe92806b4d0 0x7fe92806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:35:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.559+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe93c0abbd0 con 0x7fe948070f10 2024-03-06T07:35:59.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.559+0000 7fe94cc00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe92806b4d0 0x7fe92806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:35:59.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.560+0000 7fe94cc00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe92806b4d0 0x7fe92806d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe934006e80 tx=0x7fe934006dd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:35:59.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.564+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe93c04b030 con 0x7fe948070f10 2024-03-06T07:35:59.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.857+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9480b4990 con 0x7fe948070f10 2024-03-06T07:35:59.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.858+0000 7fe945600640 1 -- 172.21.15.6:0/1545942905 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7fe93c07a9b0 con 0x7fe948070f10 2024-03-06T07:35:59.861 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:35:59.861 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:35:59.861 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe92806b4d0 msgr2=0x7fe92806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe92806b4d0 0x7fe92806d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe934006e80 tx=0x7fe934006dd0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 msgr2=0x7fe94813fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 0x7fe94813fb60 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fe93c000c00 tx=0x7fe93c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 shutdown_connections 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe92806b4d0 0x7fe92806d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.864 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe948145300 0x7fe94813c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9480cce90 0x7fe9481408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 --2- 172.21.15.6:0/1545942905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe948070f10 0x7fe94813fb60 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:35:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 >> 172.21.15.6:0/1545942905 conn(0x7fe94806b3d0 msgr2=0x7fe9480ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:35:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 shutdown_connections 2024-03-06T07:35:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:35:59.862+0000 7fe94ea00640 1 -- 172.21.15.6:0/1545942905 wait complete. 2024-03-06T07:36:00.052 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:00.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:00.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:01.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:01.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.309+0000 7fe671800640 1 Processor -- start 2024-03-06T07:36:01.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.310+0000 7fe671800640 1 -- start start 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.311+0000 7fe671800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe66c0af1a0 0x7fe66c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe671800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c15cc40 con 0x7fe66c0cdc20 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe671800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c15cf20 con 0x7fe66c0af580 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe671800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c15d200 con 0x7fe66c0af1a0 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe66c0af1a0 0x7fe66c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe66c0cdc20 0x7fe66c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:36288/0 (socket says 172.21.15.6:36288) 2024-03-06T07:36:01.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66b400640 1 -- 172.21.15.6:0/2703712710 learned_addr learned my addr 172.21.15.6:0/2703712710 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66aa00640 1 -- 172.21.15.6:0/2703712710 >> v1:172.21.15.178:6789/0 conn(0x7fe66c0af580 legacy=0x7fe66c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66aa00640 1 -- 172.21.15.6:0/2703712710 >> v1:172.21.15.6:6789/0 conn(0x7fe66c0cdc20 legacy=0x7fe66c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66a000640 1 -- 172.21.15.6:0/2703712710 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474627432 0 0) 0x7fe66c15cc40 con 0x7fe66c0cdc20 2024-03-06T07:36:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.312+0000 7fe66aa00640 1 -- 172.21.15.6:0/2703712710 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c15d4e0 con 0x7fe66c0af1a0 2024-03-06T07:36:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.314+0000 7fe66aa00640 1 --2- 172.21.15.6:0/2703712710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15c600 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0097a0 tx=0x7fe65c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a235070985f098d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.314+0000 7fe66a000640 1 -- 172.21.15.6:0/2703712710 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0315a0 con 0x7fe66c0af1a0 2024-03-06T07:36:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.314+0000 7fe66a000640 1 -- 172.21.15.6:0/2703712710 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c037070 con 0x7fe66c0af1a0 2024-03-06T07:36:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.314+0000 7fe66a000640 1 -- 172.21.15.6:0/2703712710 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c031960 con 0x7fe66c0af1a0 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.314+0000 7fe671800640 1 -- 172.21.15.6:0/2703712710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af1a0 msgr2=0x7fe66c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.315+0000 7fe671800640 1 --2- 172.21.15.6:0/2703712710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15c600 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0097a0 tx=0x7fe65c02b840 comp rx=0 tx=0).stop 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.315+0000 7fe671800640 1 -- 172.21.15.6:0/2703712710 shutdown_connections 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.315+0000 7fe671800640 1 --2- 172.21.15.6:0/2703712710 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15c600 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.315+0000 7fe671800640 1 -- 172.21.15.6:0/2703712710 >> 172.21.15.6:0/2703712710 conn(0x7fe66c06afb0 msgr2=0x7fe66c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.315+0000 7fe671800640 1 -- 172.21.15.6:0/2703712710 shutdown_connections 2024-03-06T07:36:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.315+0000 7fe671800640 1 -- 172.21.15.6:0/2703712710 wait complete. 2024-03-06T07:36:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.316+0000 7fe671800640 1 Processor -- start 2024-03-06T07:36:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.317+0000 7fe671800640 1 -- start start 2024-03-06T07:36:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe671800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe671800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 0x7fe66c06d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe671800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe66c0cdc20 0x7fe66c1f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe671800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c155e70 con 0x7fe66c0af580 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe671800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0b3730 con 0x7fe66c0af1a0 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe671800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c155ff0 con 0x7fe66c0cdc20 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 0x7fe66c06d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 0x7fe66c06d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60364/0 (socket says 172.21.15.6:60364) 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 -- 172.21.15.6:0/3975305317 learned_addr learned my addr 172.21.15.6:0/3975305317 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe66aa00640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe66b400640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe66c0cdc20 0x7fe66c1f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 -- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe66c0cdc20 msgr2=0x7fe66c1f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe66c0cdc20 0x7fe66c1f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 -- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe66c0af1a0 msgr2=0x7fe66c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe662000640 1 -- 172.21.15.6:0/3975305317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c1f3870 con 0x7fe66c0af580 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe66aa00640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.318+0000 7fe66b400640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe66c0cdc20 0x7fe66c1f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:01.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.319+0000 7fe662000640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 0x7fe66c06d970 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fe65400c400 tx=0x7fe65400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.319+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654011790 con 0x7fe66c0af580 2024-03-06T07:36:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.319+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe654017070 con 0x7fe66c0af580 2024-03-06T07:36:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.319+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c15b420 con 0x7fe66c0af580 2024-03-06T07:36:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.319+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654011bb0 con 0x7fe66c0af580 2024-03-06T07:36:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.319+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c15b9e0 con 0x7fe66c0af580 2024-03-06T07:36:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.321+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c04f3a0 con 0x7fe66c0af580 2024-03-06T07:36:01.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.321+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe65401c460 con 0x7fe66c0af580 2024-03-06T07:36:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.323+0000 7fe663400640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64806b4d0 0x7fe64806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.323+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe65408a0f0 con 0x7fe66c0af580 2024-03-06T07:36:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.323+0000 7fe66aa00640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64806b4d0 0x7fe64806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.325+0000 7fe66aa00640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64806b4d0 0x7fe64806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe65c036000 tx=0x7fe65c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.327+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe654058ed0 con 0x7fe66c0af580 2024-03-06T07:36:01.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.565+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe66c15f0c0 con 0x7fe66c0af580 2024-03-06T07:36:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.566+0000 7fe663400640 1 -- 172.21.15.6:0/3975305317 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7fe6540587d0 con 0x7fe66c0af580 2024-03-06T07:36:01.569 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:01.569 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:01.569 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:01.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64806b4d0 msgr2=0x7fe64806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64806b4d0 0x7fe64806d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fe65c036000 tx=0x7fe65c008030 comp rx=0 tx=0).stop 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 msgr2=0x7fe66c06d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 0x7fe66c06d970 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fe65400c400 tx=0x7fe65400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 shutdown_connections 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe64806b4d0 0x7fe64806d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe66c0cdc20 0x7fe66c1f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe66c0af580 0x7fe66c06d970 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 --2- 172.21.15.6:0/3975305317 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe66c0af1a0 0x7fe66c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 >> 172.21.15.6:0/3975305317 conn(0x7fe66c06afb0 msgr2=0x7fe66c0ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 shutdown_connections 2024-03-06T07:36:01.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:01.569+0000 7fe671800640 1 -- 172.21.15.6:0/3975305317 wait complete. 2024-03-06T07:36:01.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:01.792 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:02.054 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:02.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.035+0000 7f8d78c00640 1 Processor -- start 2024-03-06T07:36:03.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.035+0000 7f8d78c00640 1 -- start start 2024-03-06T07:36:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.037+0000 7f8d78c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d7406c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.037+0000 7f8d78c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f8d740b61b0 0x7f8d7406cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.037+0000 7f8d78c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8d7406cff0 0x7f8d741619e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.037+0000 7f8d78c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7415e2b0 con 0x7f8d7406cff0 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.037+0000 7f8d78c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d740b6970 con 0x7f8d740b61b0 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.037+0000 7f8d78c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740b6c50 con 0x7f8d7415ded0 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8d7406cff0 0x7f8d741619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8d7406cff0 0x7f8d741619e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33210/0 (socket says 172.21.15.6:33210) 2024-03-06T07:36:03.040 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 -- 172.21.15.6:0/1242629434 learned_addr learned my addr 172.21.15.6:0/1242629434 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d72a00640 1 --2- 172.21.15.6:0/1242629434 >> v2:172.21.15.178:3300/0 conn(0x7f8d740b61b0 0x7f8d7406cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73400640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d7406c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 -- 172.21.15.6:0/1242629434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 msgr2=0x7f8d7406c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d7406c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 -- 172.21.15.6:0/1242629434 >> v2:172.21.15.178:3300/0 conn(0x7f8d740b61b0 msgr2=0x7f8d7406cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 --2- 172.21.15.6:0/1242629434 >> v2:172.21.15.178:3300/0 conn(0x7f8d740b61b0 0x7f8d7406cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73e00640 1 -- 172.21.15.6:0/1242629434 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d740aeb30 con 0x7f8d7406cff0 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d73400640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d7406c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.038+0000 7f8d72a00640 1 --2- 172.21.15.6:0/1242629434 >> v2:172.21.15.178:3300/0 conn(0x7f8d740b61b0 0x7f8d7406cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.039+0000 7f8d73e00640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d7406cff0 0x7f8d741619e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8d74071720 tx=0x7f8d6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29c59bac31fb785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.039+0000 7f8d72000640 1 -- 172.21.15.6:0/1242629434 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c02ca70 con 0x7f8d7406cff0 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.040+0000 7f8d72000640 1 -- 172.21.15.6:0/1242629434 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6c02cc10 con 0x7f8d7406cff0 2024-03-06T07:36:03.042 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.040+0000 7f8d72000640 1 -- 172.21.15.6:0/1242629434 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6c0343f0 con 0x7f8d7406cff0 2024-03-06T07:36:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 -- 172.21.15.6:0/1242629434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d7406cff0 msgr2=0x7f8d741619e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d7406cff0 0x7f8d741619e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8d74071720 tx=0x7f8d6c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 -- 172.21.15.6:0/1242629434 shutdown_connections 2024-03-06T07:36:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d7406cff0 0x7f8d741619e0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1242629434 >> v2:172.21.15.178:3300/0 conn(0x7f8d740b61b0 0x7f8d7406cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1242629434 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d7406c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 -- 172.21.15.6:0/1242629434 >> 172.21.15.6:0/1242629434 conn(0x7f8d740731f0 msgr2=0x7f8d7409aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 -- 172.21.15.6:0/1242629434 shutdown_connections 2024-03-06T07:36:03.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.041+0000 7f8d78c00640 1 -- 172.21.15.6:0/1242629434 wait complete. 2024-03-06T07:36:03.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.045+0000 7f8d78c00640 1 Processor -- start 2024-03-06T07:36:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.045+0000 7f8d78c00640 1 -- start start 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d78c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d7406cff0 0x7f8d74160c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d78c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 0x7f8d7414a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d78c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d74149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d78c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d74161f20 con 0x7f8d740b61b0 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d78c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d740b6610 con 0x7f8d7406cff0 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d78c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d741620a0 con 0x7f8d7415ded0 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d72a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 0x7f8d7414a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d72a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 0x7f8d7414a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33216/0 (socket says 172.21.15.6:33216) 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.047+0000 7f8d72a00640 1 -- 172.21.15.6:0/1100632495 learned_addr learned my addr 172.21.15.6:0/1100632495 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:03.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d73e00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d74149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d73400640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d7406cff0 0x7f8d74160c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d72a00640 1 -- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 msgr2=0x7f8d74149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d72a00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d74149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d72a00640 1 -- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d7406cff0 msgr2=0x7f8d74160c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d72a00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d7406cff0 0x7f8d74160c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d72a00640 1 -- 172.21.15.6:0/1100632495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7414a760 con 0x7f8d740b61b0 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.048+0000 7f8d72a00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 0x7f8d7414a220 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c7f0 tx=0x7f8d6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.049+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60015070 con 0x7f8d740b61b0 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.049+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d60007a30 con 0x7f8d740b61b0 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.049+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60007e00 con 0x7f8d740b61b0 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.049+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7414aa00 con 0x7f8d740b61b0 2024-03-06T07:36:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.049+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7414b2a0 con 0x7f8d740b61b0 2024-03-06T07:36:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.050+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7404f330 con 0x7f8d740b61b0 2024-03-06T07:36:03.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.051+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d60018070 con 0x7f8d740b61b0 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.052+0000 7f8d6b400640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:03.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.053+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d6008a1c0 con 0x7f8d740b61b0 2024-03-06T07:36:03.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.053+0000 7f8d73400640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:03.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.054+0000 7f8d73400640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d5406b4d0 0x7f8d5406d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400a950 tx=0x7f8d64008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:03.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.057+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d60058fa0 con 0x7f8d740b61b0 2024-03-06T07:36:03.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.302+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d740cdec0 con 0x7f8d740b61b0 2024-03-06T07:36:03.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.303+0000 7f8d6b400640 1 -- 172.21.15.6:0/1100632495 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f8d600588a0 con 0x7f8d740b61b0 2024-03-06T07:36:03.305 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:03.305 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:03.305 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d5406b4d0 msgr2=0x7f8d5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d5406b4d0 0x7f8d5406d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8d6400a950 tx=0x7f8d64008040 comp rx=0 tx=0).stop 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 msgr2=0x7f8d7414a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 0x7f8d7414a220 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c7f0 tx=0x7f8d6000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 shutdown_connections 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f8d5406b4d0 0x7f8d5406d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8d7415ded0 0x7f8d74149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d740b61b0 0x7f8d7414a220 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 --2- 172.21.15.6:0/1100632495 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d7406cff0 0x7f8d74160c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.305+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 >> 172.21.15.6:0/1100632495 conn(0x7f8d740731f0 msgr2=0x7f8d7415e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:03.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.306+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 shutdown_connections 2024-03-06T07:36:03.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:03.306+0000 7f8d78c00640 1 -- 172.21.15.6:0/1100632495 wait complete. 2024-03-06T07:36:03.527 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:03.527 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:04.056 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:04.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:04.753 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.751+0000 7f8323c00640 1 Processor -- start 2024-03-06T07:36:04.754 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.752+0000 7f8323c00640 1 -- start start 2024-03-06T07:36:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8323c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0af1a0 0x7f831c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8323c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f831c0cdc20 0x7f831c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8323c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c15cc40 con 0x7f831c0af580 2024-03-06T07:36:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8323c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831c15cf20 con 0x7f831c0cdc20 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8323c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c15d200 con 0x7f831c0af1a0 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8321c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f831c0cdc20 0x7f831c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8321c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f831c0cdc20 0x7f831c0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33222/0 (socket says 172.21.15.6:33222) 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8321c00640 1 -- 172.21.15.6:0/2119905168 learned_addr learned my addr 172.21.15.6:0/2119905168 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.755+0000 7f8321200640 1 --2- 172.21.15.6:0/2119905168 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0af1a0 0x7f831c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.756+0000 7f8321c00640 1 -- 172.21.15.6:0/2119905168 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0af1a0 msgr2=0x7f831c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.756+0000 7f8321c00640 1 --2- 172.21.15.6:0/2119905168 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0af1a0 0x7f831c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.756+0000 7f8321c00640 1 -- 172.21.15.6:0/2119905168 >> v1:172.21.15.178:6789/0 conn(0x7f831c0af580 legacy=0x7f831c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.756+0000 7f8321c00640 1 -- 172.21.15.6:0/2119905168 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c15d4e0 con 0x7f831c0cdc20 2024-03-06T07:36:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.757+0000 7f8321c00640 1 --2- 172.21.15.6:0/2119905168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0cdc20 0x7f831c0b2dd0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f831402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9996bd7a088a9c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.757+0000 7f831b400640 1 -- 172.21.15.6:0/2119905168 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831402ca70 con 0x7f831c0cdc20 2024-03-06T07:36:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.757+0000 7f831b400640 1 -- 172.21.15.6:0/2119905168 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831402cc10 con 0x7f831c0cdc20 2024-03-06T07:36:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.757+0000 7f831b400640 1 -- 172.21.15.6:0/2119905168 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83140343f0 con 0x7f831c0cdc20 2024-03-06T07:36:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 -- 172.21.15.6:0/2119905168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0cdc20 msgr2=0x7f831c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 --2- 172.21.15.6:0/2119905168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0cdc20 0x7f831c0b2dd0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f8314009e50 tx=0x7f831402bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 -- 172.21.15.6:0/2119905168 shutdown_connections 2024-03-06T07:36:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 --2- 172.21.15.6:0/2119905168 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0cdc20 0x7f831c0b2dd0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 --2- 172.21.15.6:0/2119905168 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0af1a0 0x7f831c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 -- 172.21.15.6:0/2119905168 >> 172.21.15.6:0/2119905168 conn(0x7f831c06afb0 msgr2=0x7f831c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 -- 172.21.15.6:0/2119905168 shutdown_connections 2024-03-06T07:36:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.758+0000 7f8323c00640 1 -- 172.21.15.6:0/2119905168 wait complete. 2024-03-06T07:36:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.760+0000 7f8323c00640 1 Processor -- start 2024-03-06T07:36:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.760+0000 7f8323c00640 1 -- start start 2024-03-06T07:36:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f8323c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f831c0af1a0 0x7f831c151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f8323c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 0x7f831c14c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f8323c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0cdc20 0x7f831c09b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f8323c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0b36a0 con 0x7f831c0af580 2024-03-06T07:36:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f8323c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0b3520 con 0x7f831c0af1a0 2024-03-06T07:36:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f8323c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0b3820 con 0x7f831c0cdc20 2024-03-06T07:36:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f831be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 0x7f831c14c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f831be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 0x7f831c14c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33224/0 (socket says 172.21.15.6:33224) 2024-03-06T07:36:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.761+0000 7f831be00640 1 -- 172.21.15.6:0/1956532697 learned_addr learned my addr 172.21.15.6:0/1956532697 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f8321c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0cdc20 0x7f831c09b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f8321200640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f831c0af1a0 0x7f831c151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f831be00640 1 -- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0cdc20 msgr2=0x7f831c09b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f831be00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0cdc20 0x7f831c09b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f831be00640 1 -- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f831c0af1a0 msgr2=0x7f831c151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f831be00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f831c0af1a0 0x7f831c151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f831be00640 1 -- 172.21.15.6:0/1956532697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c14c7b0 con 0x7f831c0af580 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f831be00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 0x7f831c14c270 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f830800c400 tx=0x7f830800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.762+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83080052d0 con 0x7f831c0af580 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.763+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8308005490 con 0x7f831c0af580 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.763+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8308007260 con 0x7f831c0af580 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.763+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c09b960 con 0x7f831c0af580 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.763+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c1528f0 con 0x7f831c0af580 2024-03-06T07:36:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.764+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c069510 con 0x7f831c0af580 2024-03-06T07:36:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.765+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8308007400 con 0x7f831c0af580 2024-03-06T07:36:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.766+0000 7f8318c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82fc06b4d0 0x7f82fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.766+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83080897f0 con 0x7f831c0af580 2024-03-06T07:36:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.767+0000 7f8321200640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82fc06b4d0 0x7f82fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.768+0000 7f8321200640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82fc06b4d0 0x7f82fc06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8310006fd0 tx=0x7f8310008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:04.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:04.771+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f83080585d0 con 0x7f831c0af580 2024-03-06T07:36:05.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.077+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f831c15bd40 con 0x7f831c0af580 2024-03-06T07:36:05.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.078+0000 7f8318c00640 1 -- 172.21.15.6:0/1956532697 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f8308057ed0 con 0x7f831c0af580 2024-03-06T07:36:05.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:05.081 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:05.081 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82fc06b4d0 msgr2=0x7f82fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82fc06b4d0 0x7f82fc06d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8310006fd0 tx=0x7f8310008040 comp rx=0 tx=0).stop 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 msgr2=0x7f831c14c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 0x7f831c14c270 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f830800c400 tx=0x7f830800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 shutdown_connections 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f82fc06b4d0 0x7f82fc06d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f831c0cdc20 0x7f831c09b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f831c0af580 0x7f831c14c270 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 --2- 172.21.15.6:0/1956532697 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f831c0af1a0 0x7f831c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 >> 172.21.15.6:0/1956532697 conn(0x7f831c06afb0 msgr2=0x7f831c072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.081+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 shutdown_connections 2024-03-06T07:36:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:05.082+0000 7f8323c00640 1 -- 172.21.15.6:0/1956532697 wait complete. 2024-03-06T07:36:05.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:05.314 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:06.058 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:06.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.548+0000 7f1f4f800640 1 Processor -- start 2024-03-06T07:36:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.548+0000 7f1f4f800640 1 -- start start 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.549+0000 7f1f4f800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480cf5c0 0x7f1f480cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.549+0000 7f1f4f800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f48071050 0x7f1f480714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.549+0000 7f1f4f800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f480b4860 con 0x7f1f480b3c90 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.549+0000 7f1f4f800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f480ce160 con 0x7f1f48071050 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.549+0000 7f1f4f800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480b4b40 con 0x7f1f480cf5c0 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.550+0000 7f1f47e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480cf5c0 0x7f1f480cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.550+0000 7f1f4d800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f48071050 0x7f1f480714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.550+0000 7f1f4d800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1f48071050 0x7f1f480714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:47762/0 (socket says 172.21.15.6:47762) 2024-03-06T07:36:06.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.550+0000 7f1f4d800640 1 -- 172.21.15.6:0/1600016895 learned_addr learned my addr 172.21.15.6:0/1600016895 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.551+0000 7f1f4d800640 1 -- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480cf5c0 msgr2=0x7f1f480cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.551+0000 7f1f4d800640 1 --2- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480cf5c0 0x7f1f480cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.551+0000 7f1f4d800640 1 -- 172.21.15.6:0/1600016895 >> v1:172.21.15.178:6789/0 conn(0x7f1f480b3c90 legacy=0x7f1f480b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.551+0000 7f1f4d800640 1 -- 172.21.15.6:0/1600016895 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f480b4e20 con 0x7f1f48071050 2024-03-06T07:36:06.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.552+0000 7f1f4d800640 1 --2- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f480714e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009e50 tx=0x7f1f4002bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78a7799593a269b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.553+0000 7f1f47400640 1 -- 172.21.15.6:0/1600016895 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4002cb90 con 0x7f1f48071050 2024-03-06T07:36:06.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.553+0000 7f1f47400640 1 -- 172.21.15.6:0/1600016895 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4002cd30 con 0x7f1f48071050 2024-03-06T07:36:06.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.553+0000 7f1f47400640 1 -- 172.21.15.6:0/1600016895 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40034610 con 0x7f1f48071050 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.553+0000 7f1f4f800640 1 -- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 msgr2=0x7f1f480714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.553+0000 7f1f4f800640 1 --2- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f480714e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009e50 tx=0x7f1f4002bff0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.554+0000 7f1f4f800640 1 -- 172.21.15.6:0/1600016895 shutdown_connections 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.554+0000 7f1f4f800640 1 --2- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f480714e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.554+0000 7f1f4f800640 1 --2- 172.21.15.6:0/1600016895 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480cf5c0 0x7f1f480cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.554+0000 7f1f4f800640 1 -- 172.21.15.6:0/1600016895 >> 172.21.15.6:0/1600016895 conn(0x7f1f4806afb0 msgr2=0x7f1f4806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.554+0000 7f1f4f800640 1 -- 172.21.15.6:0/1600016895 shutdown_connections 2024-03-06T07:36:06.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.554+0000 7f1f4f800640 1 -- 172.21.15.6:0/1600016895 wait complete. 2024-03-06T07:36:06.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.557+0000 7f1f4f800640 1 Processor -- start 2024-03-06T07:36:06.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.557+0000 7f1f4f800640 1 -- start start 2024-03-06T07:36:06.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4f800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f481537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4f800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480b3c90 0x7f1f4809b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4f800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 0x7f1f480ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4f800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48071a20 con 0x7f1f480cf5c0 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4f800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4815b340 con 0x7f1f48071050 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4f800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48071ba0 con 0x7f1f480b3c90 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 0x7f1f480ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 0x7f1f480ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33234/0 (socket says 172.21.15.6:33234) 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 -- 172.21.15.6:0/4193522338 learned_addr learned my addr 172.21.15.6:0/4193522338 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f47e00640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480b3c90 0x7f1f4809b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4ce00640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f481537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 -- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480b3c90 msgr2=0x7f1f4809b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480b3c90 0x7f1f4809b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.562 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 -- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 msgr2=0x7f1f481537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.559+0000 7f1f4d800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f481537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f4d800640 1 -- 172.21.15.6:0/4193522338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4809c640 con 0x7f1f480cf5c0 2024-03-06T07:36:06.563 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f4d800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 0x7f1f480ce2e0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009e50 tx=0x7f1f40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40043020 con 0x7f1f480cf5c0 2024-03-06T07:36:06.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4003e450 con 0x7f1f480cf5c0 2024-03-06T07:36:06.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f4809c920 con 0x7f1f480cf5c0 2024-03-06T07:36:06.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40009510 con 0x7f1f480cf5c0 2024-03-06T07:36:06.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.560+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f4809cee0 con 0x7f1f480cf5c0 2024-03-06T07:36:06.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.561+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f48067830 con 0x7f1f480cf5c0 2024-03-06T07:36:06.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.564+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f40005d10 con 0x7f1f480cf5c0 2024-03-06T07:36:06.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.565+0000 7f1f44c00640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f2806b500 0x7f1f2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:06.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.565+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f400acb20 con 0x7f1f480cf5c0 2024-03-06T07:36:06.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.565+0000 7f1f4ce00640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f2806b500 0x7f1f2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:06.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.567+0000 7f1f4ce00640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f2806b500 0x7f1f2806d9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c001640 tx=0x7f1f3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:06.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.569+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f48067830 con 0x7f1f480cf5c0 2024-03-06T07:36:06.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.813+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f480cf380 con 0x7f1f480cf5c0 2024-03-06T07:36:06.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.814+0000 7f1f44c00640 1 -- 172.21.15.6:0/4193522338 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f1f4007b990 con 0x7f1f480cf5c0 2024-03-06T07:36:06.816 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:06.817 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:06.817 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:06.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f2806b500 msgr2=0x7f1f2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f2806b500 0x7f1f2806d9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c001640 tx=0x7f1f3c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:06.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 msgr2=0x7f1f480ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:06.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 0x7f1f480ce2e0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009e50 tx=0x7f1f40004a40 comp rx=0 tx=0).stop 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 shutdown_connections 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1f2806b500 0x7f1f2806d9c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f480cf5c0 0x7f1f480ce2e0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1f480b3c90 0x7f1f4809b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 --2- 172.21.15.6:0/4193522338 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f48071050 0x7f1f481537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 >> 172.21.15.6:0/4193522338 conn(0x7f1f4806afb0 msgr2=0x7f1f4806cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 shutdown_connections 2024-03-06T07:36:06.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:06.817+0000 7f1f4f800640 1 -- 172.21.15.6:0/4193522338 wait complete. 2024-03-06T07:36:07.039 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:07.039 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:08.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:08.059 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:08.268 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.266+0000 7f8811200640 1 Processor -- start 2024-03-06T07:36:08.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.267+0000 7f8811200640 1 -- start start 2024-03-06T07:36:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.268+0000 7f8811200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c1552b0 0x7f880c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.268+0000 7f8811200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880c15ed10 con 0x7f880c0cd060 2024-03-06T07:36:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.268+0000 7f8811200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f880c15eff0 con 0x7f880c15e810 2024-03-06T07:36:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.268+0000 7f8811200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880c15f2d0 con 0x7f880c1552b0 2024-03-06T07:36:08.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.268+0000 7f880be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f880c0cd060 0x7f880c0b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57940/0 (socket says 172.21.15.6:57940) 2024-03-06T07:36:08.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.268+0000 7f880be00640 1 -- 172.21.15.6:0/2968907414 learned_addr learned my addr 172.21.15.6:0/2968907414 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880b400640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c1552b0 0x7f880c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237242441 0 0) 0x7f880c15ed10 con 0x7f880c0cd060 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f0005730 con 0x7f880c0cd060 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803861402 0 0) 0x7f87f0005730 con 0x7f880c0cd060 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c1552b0 msgr2=0x7f880c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c1552b0 0x7f880c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 >> v1:172.21.15.178:6789/0 conn(0x7f880c15e810 legacy=0x7f880c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.269+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c15f5b0 con 0x7f880c0cd060 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f88040019f0 con 0x7f880c0cd060 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 >> v1:172.21.15.6:6789/0 conn(0x7f880c0cd060 legacy=0x7f880c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87f0005bf0 0x7f87f0006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87f0006a10 0x7f87f0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f00073d0 0x7f87f000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c15f2d0 con 0x7f87f00073d0 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f000e2d0 con 0x7f87f0006a10 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f000e4d0 con 0x7f87f0005bf0 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880be00640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f00073d0 0x7f87f000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880be00640 1 -- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87f0005bf0 msgr2=0x7f87f0006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880be00640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87f0005bf0 0x7f87f0006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880be00640 1 -- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87f0006a10 msgr2=0x7f87f0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:08.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880be00640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87f0006a10 0x7f87f0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.270+0000 7f880be00640 1 -- 172.21.15.6:0/2968907414 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0005730 con 0x7f87f00073d0 2024-03-06T07:36:08.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.271+0000 7f880be00640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f00073d0 0x7f87f000dbd0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f880c0cf120 tx=0x7f87f0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f22fcd3677fbe6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880400a3b0 con 0x7f87f00073d0 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8804034650 con 0x7f87f00073d0 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f880a000640 1 -- 172.21.15.6:0/2968907414 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804034a80 con 0x7f87f00073d0 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f8811200640 1 -- 172.21.15.6:0/2968907414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f00073d0 msgr2=0x7f87f000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f8811200640 1 --2- 172.21.15.6:0/2968907414 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87f00073d0 0x7f87f000dbd0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f880c0cf120 tx=0x7f87f0007a40 comp rx=0 tx=0).stop 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f880be00640 1 -- 172.21.15.6:0/2968907414 reap_dead start 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f8811200640 1 -- 172.21.15.6:0/2968907414 shutdown_connections 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.272+0000 7f8811200640 1 -- 172.21.15.6:0/2968907414 >> 172.21.15.6:0/2968907414 conn(0x7f880c06b3d0 msgr2=0x7f880c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.273+0000 7f8811200640 1 -- 172.21.15.6:0/2968907414 shutdown_connections 2024-03-06T07:36:08.275 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.273+0000 7f8811200640 1 -- 172.21.15.6:0/2968907414 wait complete. 2024-03-06T07:36:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.274+0000 7f8811200640 1 Processor -- start 2024-03-06T07:36:08.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.274+0000 7f8811200640 1 -- start start 2024-03-06T07:36:08.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.275+0000 7f8811200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 0x7f880c1e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.275+0000 7f8811200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c144610 0x7f880c1e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.275+0000 7f8811200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f880c1f4370 0x7f880c1f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.275+0000 7f8811200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c06dd30 con 0x7f880c144230 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.275+0000 7f8811200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c0b3760 con 0x7f880c1f4370 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.275+0000 7f8811200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f880c06deb0 con 0x7f880c144610 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 0x7f880c1e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 0x7f880c1e6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33256/0 (socket says 172.21.15.6:33256) 2024-03-06T07:36:08.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 -- 172.21.15.6:0/3985465373 learned_addr learned my addr 172.21.15.6:0/3985465373 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 -- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c144610 msgr2=0x7f880c1e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c144610 0x7f880c1e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 -- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f880c1f4370 msgr2=0x7f880c1f4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f880c1f4370 0x7f880c1f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 -- 172.21.15.6:0/3985465373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880c151780 con 0x7f880c144230 2024-03-06T07:36:08.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.276+0000 7f880b400640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 0x7f880c1e6850 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00e7e0 tx=0x7f87fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.277+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc017070 con 0x7f880c144230 2024-03-06T07:36:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.277+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87fc00c7e0 con 0x7f880c144230 2024-03-06T07:36:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.277+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc00cbb0 con 0x7f880c144230 2024-03-06T07:36:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.277+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c151a20 con 0x7f880c144230 2024-03-06T07:36:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.277+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c151fe0 con 0x7f880c144230 2024-03-06T07:36:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.278+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c04f330 con 0x7f880c144230 2024-03-06T07:36:08.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.279+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87fc01a070 con 0x7f880c144230 2024-03-06T07:36:08.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.281+0000 7f8803400640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87ec06b520 0x7f87ec06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.281+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87fc089ed0 con 0x7f880c144230 2024-03-06T07:36:08.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.281+0000 7f880aa00640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87ec06b520 0x7f87ec06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:08.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.282+0000 7f880aa00640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87ec06b520 0x7f87ec06d9e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f880c1516b0 tx=0x7f87f80079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:08.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.284+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87fc058ec0 con 0x7f880c144230 2024-03-06T07:36:08.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.527+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f880c0cf4a0 con 0x7f880c144230 2024-03-06T07:36:08.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.528+0000 7f8803400640 1 -- 172.21.15.6:0/3985465373 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f87fc0587c0 con 0x7f880c144230 2024-03-06T07:36:08.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:08.530 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:08.530 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.530+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87ec06b520 msgr2=0x7f87ec06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.530+0000 7f8811200640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87ec06b520 0x7f87ec06d9e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f880c1516b0 tx=0x7f87f80079b0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.530+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 msgr2=0x7f880c1e6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.530+0000 7f8811200640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 0x7f880c1e6850 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00e7e0 tx=0x7f87fc00eca0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 shutdown_connections 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f87ec06b520 0x7f87ec06d9e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f880c1f4370 0x7f880c1f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f880c144610 0x7f880c1e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 --2- 172.21.15.6:0/3985465373 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f880c144230 0x7f880c1e6850 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 >> 172.21.15.6:0/3985465373 conn(0x7f880c06b3d0 msgr2=0x7f880c0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 shutdown_connections 2024-03-06T07:36:08.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:08.531+0000 7f8811200640 1 -- 172.21.15.6:0/3985465373 wait complete. 2024-03-06T07:36:08.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:08.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:09.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.979+0000 7f3ec1e00640 1 Processor -- start 2024-03-06T07:36:09.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.979+0000 7f3ec1e00640 1 -- start start 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.980+0000 7f3ec1e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ebc0d4a10 0x7f3ebc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.981+0000 7f3ec1e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0b3710 con 0x7f3ebc15e0b0 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.981+0000 7f3ec1e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0b39f0 con 0x7f3ebc0d0c80 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.981+0000 7f3ec1e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ebc0d10f0 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.981+0000 7f3ec0a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3ebc15e0b0 0x7f3ebc0cf310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57948/0 (socket says 172.21.15.6:57948) 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.981+0000 7f3eb3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ebc0d4a10 0x7f3ebc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.981+0000 7f3ec0a00640 1 -- 172.21.15.6:0/2836549691 learned_addr learned my addr 172.21.15.6:0/2836549691 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:09.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.982+0000 7f3eb3e00640 1 -- 172.21.15.6:0/2836549691 >> v1:172.21.15.178:6789/0 conn(0x7f3ebc0d0c80 legacy=0x7f3ebc0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.982+0000 7f3eb3e00640 1 -- 172.21.15.6:0/2836549691 >> v1:172.21.15.6:6789/0 conn(0x7f3ebc15e0b0 legacy=0x7f3ebc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.982+0000 7f3eb3e00640 1 -- 172.21.15.6:0/2836549691 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc0b3cd0 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.983+0000 7f3eb3e00640 1 --2- 172.21.15.6:0/2836549691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0d0740 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097a0 tx=0x7f3ea402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3dca9b128d96e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.983+0000 7f3ebbe00640 1 -- 172.21.15.6:0/2836549691 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4034070 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.983+0000 7f3ebbe00640 1 -- 172.21.15.6:0/2836549691 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4030610 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.983+0000 7f3ebbe00640 1 -- 172.21.15.6:0/2836549691 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40309e0 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 -- 172.21.15.6:0/2836549691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 msgr2=0x7f3ebc0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/2836549691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0d0740 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097a0 tx=0x7f3ea402b960 comp rx=0 tx=0).stop 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 -- 172.21.15.6:0/2836549691 shutdown_connections 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/2836549691 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0d0740 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 -- 172.21.15.6:0/2836549691 >> 172.21.15.6:0/2836549691 conn(0x7f3ebc0d3d30 msgr2=0x7f3ebc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 -- 172.21.15.6:0/2836549691 shutdown_connections 2024-03-06T07:36:09.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.984+0000 7f3ec1e00640 1 -- 172.21.15.6:0/2836549691 wait complete. 2024-03-06T07:36:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.986+0000 7f3ec1e00640 1 Processor -- start 2024-03-06T07:36:09.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.986+0000 7f3ec1e00640 1 -- start start 2024-03-06T07:36:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3ec1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3ebc0d0c80 0x7f3ebc158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3ec1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3ec1e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ebc15e0b0 0x7f3ebc0b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3ec1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc0cfa20 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3ec1e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc15e490 con 0x7f3ebc15e0b0 2024-03-06T07:36:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3ec1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc0cfc60 con 0x7f3ebc0d0c80 2024-03-06T07:36:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3eb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3eb3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33264/0 (socket says 172.21.15.6:33264) 2024-03-06T07:36:09.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.987+0000 7f3eb3e00640 1 -- 172.21.15.6:0/1410134421 learned_addr learned my addr 172.21.15.6:0/1410134421 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3ec1400640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ebc15e0b0 0x7f3ebc0b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3ec0a00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3ebc0d0c80 0x7f3ebc158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb3e00640 1 -- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3ebc0d0c80 msgr2=0x7f3ebc158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb3e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3ebc0d0c80 0x7f3ebc158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb3e00640 1 -- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ebc15e0b0 msgr2=0x7f3ebc0b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb3e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ebc15e0b0 0x7f3ebc0b5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb3e00640 1 -- 172.21.15.6:0/1410134421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc1f84a0 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3ec1400640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ebc15e0b0 0x7f3ebc0b5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:09.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3ec0a00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3ebc0d0c80 0x7f3ebc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb3e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0b5a50 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f3ea402bf70 tx=0x7f3ea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4034040 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea4039a10 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4039e20 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.988+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ebc0b8980 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.989+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ebc0b8f40 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.990+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ebc04f330 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.992+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ea4037070 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.992+0000 7f3eb9600640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e9806b4d0 0x7f3e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:09.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.992+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ea40ac3a0 con 0x7f3ebc0d4a10 2024-03-06T07:36:09.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.993+0000 7f3ec0a00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e9806b4d0 0x7f3e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:09.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.994+0000 7f3ec0a00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e9806b4d0 0x7f3e9806d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3eac000f60 tx=0x7f3eac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:10.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:09.998+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ea407b180 con 0x7f3ebc0d4a10 2024-03-06T07:36:10.062 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:10.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.245+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ebc0b6640 con 0x7f3ebc0d4a10 2024-03-06T07:36:10.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.246+0000 7f3eb9600640 1 -- 172.21.15.6:0/1410134421 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f3ea407aa80 con 0x7f3ebc0d4a10 2024-03-06T07:36:10.248 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:10.249 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:10.249 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:10.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e9806b4d0 msgr2=0x7f3e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e9806b4d0 0x7f3e9806d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3eac000f60 tx=0x7f3eac009040 comp rx=0 tx=0).stop 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 msgr2=0x7f3ebc0b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0b5a50 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f3ea402bf70 tx=0x7f3ea40046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 shutdown_connections 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3e9806b4d0 0x7f3e9806d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3ebc15e0b0 0x7f3ebc0b5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ebc0d4a10 0x7f3ebc0b5a50 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 --2- 172.21.15.6:0/1410134421 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3ebc0d0c80 0x7f3ebc158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 >> 172.21.15.6:0/1410134421 conn(0x7f3ebc0d3d30 msgr2=0x7f3ebc06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.249+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 shutdown_connections 2024-03-06T07:36:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:10.250+0000 7f3ec1e00640 1 -- 172.21.15.6:0/1410134421 wait complete. 2024-03-06T07:36:10.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:10.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:11.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.693+0000 7f6f96c00640 1 Processor -- start 2024-03-06T07:36:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.693+0000 7f6f96c00640 1 -- start start 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f96c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 0x7f6f90145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f96c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6f9006b6c0 0x7f6f900cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f96c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f900d0910 con 0x7f6f901457c0 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f96c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f900d0bf0 con 0x7f6f9006b6c0 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f96c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900d0ed0 con 0x7f6f90144c00 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f94e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6f901457c0 0x7f6f9006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57960/0 (socket says 172.21.15.6:57960) 2024-03-06T07:36:11.697 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f94e00640 1 -- 172.21.15.6:0/1660904157 learned_addr learned my addr 172.21.15.6:0/1660904157 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f95800640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 0x7f6f90145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635493252 0 0) 0x7f6f900d0910 con 0x7f6f901457c0 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.695+0000 7f6f96200640 1 --2- 172.21.15.6:0/1660904157 >> v2:172.21.15.178:3300/0 conn(0x7f6f9006b6c0 0x7f6f900cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f78003520 con 0x7f6f901457c0 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944742299 0 0) 0x7f6f78003520 con 0x7f6f901457c0 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 msgr2=0x7f6f90145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 0x7f6f90145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 >> v2:172.21.15.178:3300/0 conn(0x7f6f9006b6c0 msgr2=0x7f6f900cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.699 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1660904157 >> v2:172.21.15.178:3300/0 conn(0x7f6f9006b6c0 0x7f6f900cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900d11b0 con 0x7f6f901457c0 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f6f7c001670 con 0x7f6f901457c0 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f96200640 1 --2- 172.21.15.6:0/1660904157 >> v2:172.21.15.178:3300/0 conn(0x7f6f9006b6c0 0x7f6f900cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 >> v1:172.21.15.6:6789/0 conn(0x7f6f901457c0 legacy=0x7f6f9006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 0x7f6f780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f78003f80 0x7f6f78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f78004d40 0x7f6f780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900d0bf0 con 0x7f6f78003f80 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900d0ed0 con 0x7f6f78004d40 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7800ba70 con 0x7f6f90144c00 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.696+0000 7f6f94e00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f78003f80 0x7f6f78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f95800640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 0x7f6f780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f96200640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f78004d40 0x7f6f780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f94e00640 1 -- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 msgr2=0x7f6f780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f94e00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f90144c00 0x7f6f780039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f94e00640 1 -- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f78004d40 msgr2=0x7f6f780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f94e00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f78004d40 0x7f6f780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f94e00640 1 -- 172.21.15.6:0/1660904157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78003520 con 0x7f6f78003f80 2024-03-06T07:36:11.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.697+0000 7f6f96200640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f78004d40 0x7f6f780051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.699+0000 7f6f94e00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f78003f80 0x7f6f78004800 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f6f90145c00 tx=0x7f6f78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19b8378cb3d5d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.699+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c007c80 con 0x7f6f78003f80 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.699+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7c030070 con 0x7f6f78003f80 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.699+0000 7f6f8fe00640 1 -- 172.21.15.6:0/1660904157 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c005d90 con 0x7f6f78003f80 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.699+0000 7f6f96c00640 1 -- 172.21.15.6:0/1660904157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f78003f80 msgr2=0x7f6f78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.699+0000 7f6f96c00640 1 --2- 172.21.15.6:0/1660904157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f78003f80 0x7f6f78004800 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f6f90145c00 tx=0x7f6f78005830 comp rx=0 tx=0).stop 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.700+0000 7f6f96200640 1 -- 172.21.15.6:0/1660904157 reap_dead start 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.700+0000 7f6f96c00640 1 -- 172.21.15.6:0/1660904157 shutdown_connections 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.700+0000 7f6f96c00640 1 -- 172.21.15.6:0/1660904157 >> 172.21.15.6:0/1660904157 conn(0x7f6f9006c0c0 msgr2=0x7f6f9006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:11.702 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.700+0000 7f6f96c00640 1 -- 172.21.15.6:0/1660904157 shutdown_connections 2024-03-06T07:36:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.700+0000 7f6f96c00640 1 -- 172.21.15.6:0/1660904157 wait complete. 2024-03-06T07:36:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.701+0000 7f6f96c00640 1 Processor -- start 2024-03-06T07:36:11.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.702+0000 7f6f96c00640 1 -- start start 2024-03-06T07:36:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.702+0000 7f6f96c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f78003f80 0x7f6f90141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.702+0000 7f6f96c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9013f970 0x7f6f900b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 0x7f6f900b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900d19b0 con 0x7f6f9013fdf0 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90144fe0 con 0x7f6f9013f970 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9006f820 con 0x7f6f78003f80 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 0x7f6f900b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 0x7f6f900b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33284/0 (socket says 172.21.15.6:33284) 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 -- 172.21.15.6:0/2520811671 learned_addr learned my addr 172.21.15.6:0/2520811671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f95800640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f78003f80 0x7f6f90141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f94e00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9013f970 0x7f6f900b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 -- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f78003f80 msgr2=0x7f6f90141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f78003f80 0x7f6f90141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 -- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9013f970 msgr2=0x7f6f900b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9013f970 0x7f6f900b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.703+0000 7f6f96200640 1 -- 172.21.15.6:0/2520811671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90140350 con 0x7f6f9013fdf0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.704+0000 7f6f96200640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 0x7f6f900b4f60 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800f410 tx=0x7f6f8800f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.704+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88004230 con 0x7f6f9013fdf0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.704+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f88019070 con 0x7f6f9013fdf0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.704+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88003610 con 0x7f6f9013fdf0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.704+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f900b3c60 con 0x7f6f9013fdf0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.704+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f900b4220 con 0x7f6f9013fdf0 2024-03-06T07:36:11.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.705+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9004f330 con 0x7f6f9013fdf0 2024-03-06T07:36:11.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.707+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f88012070 con 0x7f6f9013fdf0 2024-03-06T07:36:11.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.708+0000 7f6f8d600640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f7006b500 0x7f6f7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:11.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.708+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6f88089ca0 con 0x7f6f9013fdf0 2024-03-06T07:36:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.709+0000 7f6f95800640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f7006b500 0x7f6f7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:11.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.710+0000 7f6f95800640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f7006b500 0x7f6f7006d9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009fd0 tx=0x7f6f8400d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:11.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.714+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f88058ad0 con 0x7f6f9013fdf0 2024-03-06T07:36:11.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.951+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f900722f0 con 0x7f6f9013fdf0 2024-03-06T07:36:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.953+0000 7f6f8d600640 1 -- 172.21.15.6:0/2520811671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f6f880583d0 con 0x7f6f9013fdf0 2024-03-06T07:36:11.955 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:11.955 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:11.955 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f7006b500 msgr2=0x7f6f7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f7006b500 0x7f6f7006d9c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009fd0 tx=0x7f6f8400d040 comp rx=0 tx=0).stop 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 msgr2=0x7f6f900b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 0x7f6f900b4f60 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800f410 tx=0x7f6f8800f8d0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 shutdown_connections 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6f7006b500 0x7f6f7006d9c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9013f970 0x7f6f900b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6f9013fdf0 0x7f6f900b4f60 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 --2- 172.21.15.6:0/2520811671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6f78003f80 0x7f6f90141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 >> 172.21.15.6:0/2520811671 conn(0x7f6f9006c0c0 msgr2=0x7f6f900b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 shutdown_connections 2024-03-06T07:36:11.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:11.955+0000 7f6f96c00640 1 -- 172.21.15.6:0/2520811671 wait complete. 2024-03-06T07:36:12.064 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:12.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:12.177 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:13.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:13.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.439+0000 7f4f41c00640 1 Processor -- start 2024-03-06T07:36:13.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.439+0000 7f4f41c00640 1 -- start start 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f41c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 0x7f4f3c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f41c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4f3c0d0e50 0x7f4f3c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f41c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3c0b3710 con 0x7f4f3c0d4a10 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f41c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3c0b09e0 con 0x7f4f3c0d0e50 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f41c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0b39f0 con 0x7f4f3c15e0b0 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f3b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4f3c0d4a10 0x7f4f3c0d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45806/0 (socket says 172.21.15.6:45806) 2024-03-06T07:36:13.443 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f3b400640 1 -- 172.21.15.6:0/3854531078 learned_addr learned my addr 172.21.15.6:0/3854531078 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f41200640 1 --2- 172.21.15.6:0/3854531078 >> v2:172.21.15.178:3300/0 conn(0x7f4f3c0d0e50 0x7f4f3c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.441+0000 7f4f3be00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 0x7f4f3c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214060966 0 0) 0x7f4f3c0b3710 con 0x7f4f3c0d4a10 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f20003520 con 0x7f4f3c0d4a10 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699879314 0 0) 0x7f4f20003520 con 0x7f4f3c0d4a10 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 msgr2=0x7f4f3c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 0x7f4f3c0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 >> v2:172.21.15.178:3300/0 conn(0x7f4f3c0d0e50 msgr2=0x7f4f3c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 --2- 172.21.15.6:0/3854531078 >> v2:172.21.15.178:3300/0 conn(0x7f4f3c0d0e50 0x7f4f3c0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0b3cd0 con 0x7f4f3c0d4a10 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f4f28001670 con 0x7f4f3c0d4a10 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 >> v1:172.21.15.6:6789/0 conn(0x7f4f3c0d4a10 legacy=0x7f4f3c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 0x7f4f200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f20004d40 0x7f4f200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0b39f0 con 0x7f4f20003f30 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2000b9c0 con 0x7f4f20004d40 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.442+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2000bb40 con 0x7f4f3c15e0b0 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3b400640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3be00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 0x7f4f200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f41200640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f20004d40 0x7f4f200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3b400640 1 -- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 msgr2=0x7f4f200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3b400640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c15e0b0 0x7f4f200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3b400640 1 -- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f20004d40 msgr2=0x7f4f200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3b400640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f20004d40 0x7f4f200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.443+0000 7f4f3b400640 1 -- 172.21.15.6:0/3854531078 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20003520 con 0x7f4f20003f30 2024-03-06T07:36:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.444+0000 7f4f3b400640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f20004800 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4f28007fd0 tx=0x7f4f20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e33d7b2a260987 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28007d50 con 0x7f4f20003f30 2024-03-06T07:36:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f28030070 con 0x7f4f20003f30 2024-03-06T07:36:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f3aa00640 1 -- 172.21.15.6:0/3854531078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2802cd80 con 0x7f4f20003f30 2024-03-06T07:36:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41c00640 1 -- 172.21.15.6:0/3854531078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 msgr2=0x7f4f20004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41c00640 1 --2- 172.21.15.6:0/3854531078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f20004800 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4f28007fd0 tx=0x7f4f20005830 comp rx=0 tx=0).stop 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41200640 1 -- 172.21.15.6:0/3854531078 reap_dead start 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41c00640 1 -- 172.21.15.6:0/3854531078 shutdown_connections 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41c00640 1 -- 172.21.15.6:0/3854531078 >> 172.21.15.6:0/3854531078 conn(0x7f4f3c0d3d30 msgr2=0x7f4f3c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41c00640 1 -- 172.21.15.6:0/3854531078 shutdown_connections 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.445+0000 7f4f41c00640 1 -- 172.21.15.6:0/3854531078 wait complete. 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.447+0000 7f4f41c00640 1 Processor -- start 2024-03-06T07:36:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.447+0000 7f4f41c00640 1 -- start start 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.448+0000 7f4f41c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f3c152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.448+0000 7f4f41c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c0cb790 0x7f4f3c0b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.448+0000 7f4f41c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f3c0cbc10 0x7f4f3c0b4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.448+0000 7f4f41c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0cfc60 con 0x7f4f20003f30 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.448+0000 7f4f41c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c15e490 con 0x7f4f3c0cbc10 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.448+0000 7f4f41c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c0cf310 con 0x7f4f3c0cb790 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f3c152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f3c152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43554/0 (socket says 172.21.15.6:43554) 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 -- 172.21.15.6:0/1795602958 learned_addr learned my addr 172.21.15.6:0/1795602958 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:13.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3b400640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c0cb790 0x7f4f3c0b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f41200640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f3c0cbc10 0x7f4f3c0b4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 -- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c0cb790 msgr2=0x7f4f3c0b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c0cb790 0x7f4f3c0b4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 -- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f3c0cbc10 msgr2=0x7f4f3c0b4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f3c0cbc10 0x7f4f3c0b4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.449+0000 7f4f3be00640 1 -- 172.21.15.6:0/1795602958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c0cb450 con 0x7f4f20003f30 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.450+0000 7f4f3be00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f3c152140 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c011600 tx=0x7f4f2c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.450+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00f920 con 0x7f4f20003f30 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.450+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c00fac0 con 0x7f4f20003f30 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.450+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c15b830 con 0x7f4f20003f30 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.450+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c00fe70 con 0x7f4f20003f30 2024-03-06T07:36:13.453 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.450+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c0cb170 con 0x7f4f20003f30 2024-03-06T07:36:13.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.451+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c04f330 con 0x7f4f20003f30 2024-03-06T07:36:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.453+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f2c015460 con 0x7f4f20003f30 2024-03-06T07:36:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.455+0000 7f4f33e00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4f1c06b550 0x7f4f1c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.455+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4f2c08aad0 con 0x7f4f20003f30 2024-03-06T07:36:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.455+0000 7f4f3b400640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4f1c06b550 0x7f4f1c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.457+0000 7f4f3b400640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4f1c06b550 0x7f4f1c06da10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4f28034000 tx=0x7f4f28039040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:13.464 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.462+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4f2c019030 con 0x7f4f20003f30 2024-03-06T07:36:13.700 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.698+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3c0b1690 con 0x7f4f20003f30 2024-03-06T07:36:13.701 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.699+0000 7f4f33e00640 1 -- 172.21.15.6:0/1795602958 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f4f2c059980 con 0x7f4f20003f30 2024-03-06T07:36:13.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:13.702 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4f1c06b550 msgr2=0x7f4f1c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4f1c06b550 0x7f4f1c06da10 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4f28034000 tx=0x7f4f28039040 comp rx=0 tx=0).stop 2024-03-06T07:36:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 msgr2=0x7f4f3c152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f3c152140 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c011600 tx=0x7f4f2c011ac0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 shutdown_connections 2024-03-06T07:36:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4f1c06b550 0x7f4f1c06da10 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4f3c0cbc10 0x7f4f3c0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4f3c0cb790 0x7f4f3c0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 --2- 172.21.15.6:0/1795602958 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4f20003f30 0x7f4f3c152140 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:13.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 >> 172.21.15.6:0/1795602958 conn(0x7f4f3c0d3d30 msgr2=0x7f4f3c0b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:13.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 shutdown_connections 2024-03-06T07:36:13.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:13.702+0000 7f4f41c00640 1 -- 172.21.15.6:0/1795602958 wait complete. 2024-03-06T07:36:13.922 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:13.923 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:14.066 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:14.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:15.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.166+0000 7fb933c00640 1 Processor -- start 2024-03-06T07:36:15.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.166+0000 7fb933c00640 1 -- start start 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb933c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb92c158440 0x7fb92c0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb933c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 0x7fb92c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb933c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92c0b4b40 con 0x7fb92c158820 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb933c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0b4e20 con 0x7fb92c158440 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb933c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0b5100 con 0x7fb92c06c6d0 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb92be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb92c158820 0x7fb92c06bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45816/0 (socket says 172.21.15.6:45816) 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.170+0000 7fb92be00640 1 -- 172.21.15.6:0/3355289479 learned_addr learned my addr 172.21.15.6:0/3355289479 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:15.174 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb931c00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 0x7fb92c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb931200640 1 --2- 172.21.15.6:0/3355289479 >> v2:172.21.15.178:3300/0 conn(0x7fb92c158440 0x7fb92c0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729635494 0 0) 0x7fb92c0b4b40 con 0x7fb92c158820 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb914003520 con 0x7fb92c158820 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3374754752 0 0) 0x7fb914003520 con 0x7fb92c158820 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 msgr2=0x7fb92c0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 0x7fb92c0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 >> v2:172.21.15.178:3300/0 conn(0x7fb92c158440 msgr2=0x7fb92c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 --2- 172.21.15.6:0/3355289479 >> v2:172.21.15.178:3300/0 conn(0x7fb92c158440 0x7fb92c0b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0ae680 con 0x7fb92c158820 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb918001670 con 0x7fb92c158820 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.171+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 >> v1:172.21.15.6:6789/0 conn(0x7fb92c158820 legacy=0x7fb92c06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92b400640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 0x7fb914003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92b400640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914003fa0 0x7fb914004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92b400640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914004d40 0x7fb9140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0b4e20 con 0x7fb914003fa0 2024-03-06T07:36:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0b5100 con 0x7fb914004d40 2024-03-06T07:36:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91400b9f0 con 0x7fb92c06c6d0 2024-03-06T07:36:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92be00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914003fa0 0x7fb914004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb931c00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914004d40 0x7fb9140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92be00640 1 -- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 msgr2=0x7fb914003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92be00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c06c6d0 0x7fb914003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.172+0000 7fb92be00640 1 -- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914004d40 msgr2=0x7fb9140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.173+0000 7fb92be00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914004d40 0x7fb9140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.173+0000 7fb92be00640 1 -- 172.21.15.6:0/3355289479 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914003520 con 0x7fb914003fa0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.173+0000 7fb931c00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914004d40 0x7fb9140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb92be00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914003fa0 0x7fb914004800 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fb918007fd0 tx=0x7fb914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb413c2d02a93946 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918007d50 con 0x7fb914003fa0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb918030070 con 0x7fb914003fa0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb92b400640 1 -- 172.21.15.6:0/3355289479 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91802cd80 con 0x7fb914003fa0 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb933c00640 1 -- 172.21.15.6:0/3355289479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914003fa0 msgr2=0x7fb914004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb933c00640 1 --2- 172.21.15.6:0/3355289479 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb914003fa0 0x7fb914004800 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fb918007fd0 tx=0x7fb914005830 comp rx=0 tx=0).stop 2024-03-06T07:36:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.174+0000 7fb931c00640 1 -- 172.21.15.6:0/3355289479 reap_dead start 2024-03-06T07:36:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.175+0000 7fb933c00640 1 -- 172.21.15.6:0/3355289479 shutdown_connections 2024-03-06T07:36:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.175+0000 7fb933c00640 1 -- 172.21.15.6:0/3355289479 >> 172.21.15.6:0/3355289479 conn(0x7fb92c071490 msgr2=0x7fb92c0718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.175+0000 7fb933c00640 1 -- 172.21.15.6:0/3355289479 shutdown_connections 2024-03-06T07:36:15.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.175+0000 7fb933c00640 1 -- 172.21.15.6:0/3355289479 wait complete. 2024-03-06T07:36:15.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.177+0000 7fb933c00640 1 Processor -- start 2024-03-06T07:36:15.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.177+0000 7fb933c00640 1 -- start start 2024-03-06T07:36:15.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb933c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914003fa0 0x7fb92c1e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb933c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c14a130 0x7fb92c147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb933c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 0x7fb92c14ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb933c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0b4890 con 0x7fb92c147950 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb933c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0b4710 con 0x7fb914003fa0 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb933c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c15f270 con 0x7fb92c14a130 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 0x7fb92c14ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 0x7fb92c14ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43572/0 (socket says 172.21.15.6:43572) 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931c00640 1 -- 172.21.15.6:0/1177170908 learned_addr learned my addr 172.21.15.6:0/1177170908 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb92be00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c14a130 0x7fb92c147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931200640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914003fa0 0x7fb92c1e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931c00640 1 -- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c14a130 msgr2=0x7fb92c147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c14a130 0x7fb92c147410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.178+0000 7fb931c00640 1 -- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914003fa0 msgr2=0x7fb92c1e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb931c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914003fa0 0x7fb92c1e83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb931c00640 1 -- 172.21.15.6:0/1177170908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c147e30 con 0x7fb92c147950 2024-03-06T07:36:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb931c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 0x7fb92c14ef50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb92c15e5f0 tx=0x7fb92400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92401c070 con 0x7fb92c147950 2024-03-06T07:36:15.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92400fe90 con 0x7fb92c147950 2024-03-06T07:36:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924017620 con 0x7fb92c147950 2024-03-06T07:36:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c148110 con 0x7fb92c147950 2024-03-06T07:36:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.179+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c1486d0 con 0x7fb92c147950 2024-03-06T07:36:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.181+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c1488d0 con 0x7fb92c147950 2024-03-06T07:36:15.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.182+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb92401f070 con 0x7fb92c147950 2024-03-06T07:36:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.182+0000 7fb928c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb90c06b4d0 0x7fb90c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.183+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb92408a1a0 con 0x7fb92c147950 2024-03-06T07:36:15.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.183+0000 7fb931200640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb90c06b4d0 0x7fb90c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:15.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.184+0000 7fb931200640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb90c06b4d0 0x7fb90c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb9200047b0 tx=0x7fb920008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:15.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.187+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb92c1488d0 con 0x7fb92c147950 2024-03-06T07:36:15.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.422+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb92c072250 con 0x7fb92c147950 2024-03-06T07:36:15.425 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.423+0000 7fb928c00640 1 -- 172.21.15.6:0/1177170908 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7fb924012070 con 0x7fb92c147950 2024-03-06T07:36:15.426 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:15.426 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:15.426 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:15.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb90c06b4d0 msgr2=0x7fb90c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb90c06b4d0 0x7fb90c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fb9200047b0 tx=0x7fb920008040 comp rx=0 tx=0).stop 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 msgr2=0x7fb92c14ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 0x7fb92c14ef50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb92c15e5f0 tx=0x7fb92400f4a0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 shutdown_connections 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb90c06b4d0 0x7fb90c06d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb92c147950 0x7fb92c14ef50 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb92c14a130 0x7fb92c147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 --2- 172.21.15.6:0/1177170908 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb914003fa0 0x7fb92c1e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 >> 172.21.15.6:0/1177170908 conn(0x7fb92c071490 msgr2=0x7fb92c1607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 shutdown_connections 2024-03-06T07:36:15.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:15.426+0000 7fb933c00640 1 -- 172.21.15.6:0/1177170908 wait complete. 2024-03-06T07:36:15.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:15.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:16.068 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:16.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.880+0000 7f0331c00640 1 Processor -- start 2024-03-06T07:36:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.880+0000 7f0331c00640 1 -- start start 2024-03-06T07:36:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f0331c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f032c072da0 0x7f032c162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f0331c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f032c0712c0 0x7f032c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f0331c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c0ce090 con 0x7f032c070ee0 2024-03-06T07:36:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f0331c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032c0ceca0 con 0x7f032c0712c0 2024-03-06T07:36:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f0331c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032c0ce370 con 0x7f032c072da0 2024-03-06T07:36:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f032aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f032c070ee0 0x7f032c06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45822/0 (socket says 172.21.15.6:45822) 2024-03-06T07:36:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f032aa00640 1 -- 172.21.15.6:0/1085708576 learned_addr learned my addr 172.21.15.6:0/1085708576 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f032b400640 1 --2- 172.21.15.6:0/1085708576 >> v2:172.21.15.6:3300/0 conn(0x7f032c0712c0 0x7f032c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.882+0000 7f0323e00640 1 --2- 172.21.15.6:0/1085708576 >> v2:172.21.15.178:3300/0 conn(0x7f032c072da0 0x7f032c162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.883+0000 7f032a000640 1 -- 172.21.15.6:0/1085708576 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711106033 0 0) 0x7f032c0ce090 con 0x7f032c070ee0 2024-03-06T07:36:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.883+0000 7f032a000640 1 -- 172.21.15.6:0/1085708576 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f032c070ee0 2024-03-06T07:36:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.883+0000 7f032b400640 1 -- 172.21.15.6:0/1085708576 >> v2:172.21.15.178:3300/0 conn(0x7f032c072da0 msgr2=0x7f032c162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.883+0000 7f032b400640 1 --2- 172.21.15.6:0/1085708576 >> v2:172.21.15.178:3300/0 conn(0x7f032c072da0 0x7f032c162c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.883+0000 7f032b400640 1 -- 172.21.15.6:0/1085708576 >> v1:172.21.15.6:6789/0 conn(0x7f032c070ee0 legacy=0x7f032c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.886 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.883+0000 7f032b400640 1 -- 172.21.15.6:0/1085708576 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0ce650 con 0x7f032c0712c0 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.884+0000 7f032b400640 1 --2- 172.21.15.6:0/1085708576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c06e7d0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f032c06c570 tx=0x7f032402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e152bcf878f5a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.884+0000 7f032a000640 1 -- 172.21.15.6:0/1085708576 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324039070 con 0x7f032c0712c0 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.884+0000 7f032a000640 1 -- 172.21.15.6:0/1085708576 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032402cc80 con 0x7f032c0712c0 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.884+0000 7f032a000640 1 -- 172.21.15.6:0/1085708576 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0324034640 con 0x7f032c0712c0 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 -- 172.21.15.6:0/1085708576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 msgr2=0x7f032c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 --2- 172.21.15.6:0/1085708576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c06e7d0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f032c06c570 tx=0x7f032402bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 -- 172.21.15.6:0/1085708576 shutdown_connections 2024-03-06T07:36:16.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 --2- 172.21.15.6:0/1085708576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c06e7d0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 --2- 172.21.15.6:0/1085708576 >> v2:172.21.15.178:3300/0 conn(0x7f032c072da0 0x7f032c162c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 -- 172.21.15.6:0/1085708576 >> 172.21.15.6:0/1085708576 conn(0x7f032c06afb0 msgr2=0x7f032c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:16.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 -- 172.21.15.6:0/1085708576 shutdown_connections 2024-03-06T07:36:16.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.885+0000 7f0331c00640 1 -- 172.21.15.6:0/1085708576 wait complete. 2024-03-06T07:36:16.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.887+0000 7f0331c00640 1 Processor -- start 2024-03-06T07:36:16.889 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.887+0000 7f0331c00640 1 -- start start 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0331c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c070ee0 0x7f032c15e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0331c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0331c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f032c072da0 0x7f032c0b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0331c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0cdda0 con 0x7f032c0712c0 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0331c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0cdc20 con 0x7f032c070ee0 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0331c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0cf130 con 0x7f032c072da0 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0323e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0323e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c0b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43596/0 (socket says 172.21.15.6:43596) 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.888+0000 7f0323e00640 1 -- 172.21.15.6:0/1717806978 learned_addr learned my addr 172.21.15.6:0/1717806978 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f032aa00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c070ee0 0x7f032c15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f032b400640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f032c072da0 0x7f032c0b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f0323e00640 1 -- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f032c072da0 msgr2=0x7f032c0b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f0323e00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f032c072da0 0x7f032c0b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f0323e00640 1 -- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c070ee0 msgr2=0x7f032c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f0323e00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c070ee0 0x7f032c15e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f0323e00640 1 -- 172.21.15.6:0/1717806978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0b1020 con 0x7f032c0712c0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f032aa00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c070ee0 0x7f032c15e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f032b400640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f032c072da0 0x7f032c0b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.889+0000 7f0323e00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c0b01a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f031400e870 tx=0x7f031400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.890+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0314017070 con 0x7f032c0712c0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.890+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031400c940 con 0x7f032c0712c0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.890+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031400cd30 con 0x7f032c0712c0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.890+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c0b0d40 con 0x7f032c0712c0 2024-03-06T07:36:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.890+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c1f8560 con 0x7f032c0712c0 2024-03-06T07:36:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.891+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c069510 con 0x7f032c0712c0 2024-03-06T07:36:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.893+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f031401a070 con 0x7f032c0712c0 2024-03-06T07:36:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.894+0000 7f0322a00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f030806b4d0 0x7f030806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.894+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f031408a0c0 con 0x7f032c0712c0 2024-03-06T07:36:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.895+0000 7f032aa00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f030806b4d0 0x7f030806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.896+0000 7f032aa00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f030806b4d0 0x7f030806d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f031c005880 tx=0x7f031c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:16.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:16.899+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0314058ef0 con 0x7f032c0712c0 2024-03-06T07:36:17.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.136+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f032c0cc600 con 0x7f032c0712c0 2024-03-06T07:36:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.137+0000 7f0322a00640 1 -- 172.21.15.6:0/1717806978 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f03140587f0 con 0x7f032c0712c0 2024-03-06T07:36:17.139 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:17.139 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:17.140 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.139+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f030806b4d0 msgr2=0x7f030806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.139+0000 7f0331c00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f030806b4d0 0x7f030806d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f031c005880 tx=0x7f031c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.139+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 msgr2=0x7f032c0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.139+0000 7f0331c00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c0b01a0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f031400e870 tx=0x7f031400ed30 comp rx=0 tx=0).stop 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 shutdown_connections 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f030806b4d0 0x7f030806d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f032c072da0 0x7f032c0b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f032c0712c0 0x7f032c0b01a0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 --2- 172.21.15.6:0/1717806978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f032c070ee0 0x7f032c15e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:17.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 >> 172.21.15.6:0/1717806978 conn(0x7f032c06afb0 msgr2=0x7f032c0b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:17.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 shutdown_connections 2024-03-06T07:36:17.143 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:17.140+0000 7f0331c00640 1 -- 172.21.15.6:0/1717806978 wait complete. 2024-03-06T07:36:17.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:17.361 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:18.070 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:18.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:18.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.592+0000 7f1122e00640 1 Processor -- start 2024-03-06T07:36:18.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.592+0000 7f1122e00640 1 -- start start 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1122e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f111c070f10 0x7f111c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1122e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c15c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1122e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0714d0 con 0x7f111c145300 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1122e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c0cd300 con 0x7f111c070f10 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1122e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c0717b0 con 0x7f111c0cce90 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1121000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f111c070f10 0x7f111c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1121000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f111c070f10 0x7f111c0cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43602/0 (socket says 172.21.15.6:43602) 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1121000640 1 -- 172.21.15.6:0/47689653 learned_addr learned my addr 172.21.15.6:0/47689653 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.594+0000 7f1121a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f111c145300 0x7f111c0b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45828/0 (socket says 172.21.15.6:45828) 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.595+0000 7f1122400640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c15c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.595+0000 7f1121000640 1 -- 172.21.15.6:0/47689653 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 msgr2=0x7f111c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:18.597 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.595+0000 7f1121000640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c15c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.595+0000 7f1121000640 1 -- 172.21.15.6:0/47689653 >> v1:172.21.15.6:6789/0 conn(0x7f111c145300 legacy=0x7f111c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.595+0000 7f1121000640 1 -- 172.21.15.6:0/47689653 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c071a90 con 0x7f111c070f10 2024-03-06T07:36:18.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.595+0000 7f1122400640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:18.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.597+0000 7f1121000640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c070f10 0x7f111c0cc950 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f1108009870 tx=0x7f110802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed617a83ffe2bbc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.597+0000 7f111be00640 1 -- 172.21.15.6:0/47689653 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108035020 con 0x7f111c070f10 2024-03-06T07:36:18.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.597+0000 7f111be00640 1 -- 172.21.15.6:0/47689653 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1108031640 con 0x7f111c070f10 2024-03-06T07:36:18.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.597+0000 7f111be00640 1 -- 172.21.15.6:0/47689653 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110802d040 con 0x7f111c070f10 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.597+0000 7f1122e00640 1 -- 172.21.15.6:0/47689653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c070f10 msgr2=0x7f111c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.597+0000 7f1122e00640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c070f10 0x7f111c0cc950 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f1108009870 tx=0x7f110802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.598+0000 7f1122e00640 1 -- 172.21.15.6:0/47689653 shutdown_connections 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.598+0000 7f1122e00640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c15c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.598+0000 7f1122e00640 1 --2- 172.21.15.6:0/47689653 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c070f10 0x7f111c0cc950 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.598+0000 7f1122e00640 1 -- 172.21.15.6:0/47689653 >> 172.21.15.6:0/47689653 conn(0x7f111c06b3d0 msgr2=0x7f111c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.598+0000 7f1122e00640 1 -- 172.21.15.6:0/47689653 shutdown_connections 2024-03-06T07:36:18.600 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.598+0000 7f1122e00640 1 -- 172.21.15.6:0/47689653 wait complete. 2024-03-06T07:36:18.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.599+0000 7f1122e00640 1 Processor -- start 2024-03-06T07:36:18.602 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.600+0000 7f1122e00640 1 -- start start 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111c070f10 0x7f111c135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 0x7f111c06dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c15cab0 con 0x7f111c145300 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c1456e0 con 0x7f111c070f10 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c15cc30 con 0x7f111c0cce90 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 0x7f111c06dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 0x7f111c06dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43604/0 (socket says 172.21.15.6:43604) 2024-03-06T07:36:18.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 -- 172.21.15.6:0/4293647619 learned_addr learned my addr 172.21.15.6:0/4293647619 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:18.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1121000640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 -- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 msgr2=0x7f111c06d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c06d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 -- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111c070f10 msgr2=0x7f111c135bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111c070f10 0x7f111c135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.601+0000 7f1122400640 1 -- 172.21.15.6:0/4293647619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0aee50 con 0x7f111c145300 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.602+0000 7f1122400640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 0x7f111c06dbf0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f111400cef0 tx=0x7f111400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.602+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111401a070 con 0x7f111c145300 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.602+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111400d860 con 0x7f111c145300 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.602+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111400dc50 con 0x7f111c145300 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.602+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c0af130 con 0x7f111c145300 2024-03-06T07:36:18.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.602+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c1e5a20 con 0x7f111c145300 2024-03-06T07:36:18.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.604+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c04f330 con 0x7f111c145300 2024-03-06T07:36:18.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.605+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f111401d070 con 0x7f111c145300 2024-03-06T07:36:18.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.606+0000 7f1119600640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10fc06b4d0 0x7f10fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:18.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.607+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f111408abb0 con 0x7f111c145300 2024-03-06T07:36:18.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.607+0000 7f1121a00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10fc06b4d0 0x7f10fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:18.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.608+0000 7f1121a00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10fc06b4d0 0x7f10fc06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1110000f60 tx=0x7f1110007a00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:18.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.611+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1114059a10 con 0x7f111c145300 2024-03-06T07:36:18.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.875+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f111c0b2950 con 0x7f111c145300 2024-03-06T07:36:18.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.877+0000 7f1119600640 1 -- 172.21.15.6:0/4293647619 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f1114059310 con 0x7f111c145300 2024-03-06T07:36:18.879 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:18.879 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:18.879 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.879+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10fc06b4d0 msgr2=0x7f10fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.879+0000 7f1122e00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10fc06b4d0 0x7f10fc06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1110000f60 tx=0x7f1110007a00 comp rx=0 tx=0).stop 2024-03-06T07:36:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.879+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 msgr2=0x7f111c06dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:18.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.879+0000 7f1122e00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 0x7f111c06dbf0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f111400cef0 tx=0x7f111400a710 comp rx=0 tx=0).stop 2024-03-06T07:36:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 shutdown_connections 2024-03-06T07:36:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f10fc06b4d0 0x7f10fc06d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f111c145300 0x7f111c06dbf0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f111c0cce90 0x7f111c06d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 --2- 172.21.15.6:0/4293647619 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f111c070f10 0x7f111c135bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 >> 172.21.15.6:0/4293647619 conn(0x7f111c06b3d0 msgr2=0x7f111c15d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 shutdown_connections 2024-03-06T07:36:18.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:18.880+0000 7f1122e00640 1 -- 172.21.15.6:0/4293647619 wait complete. 2024-03-06T07:36:19.103 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:19.103 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:20.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:20.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.324+0000 7f05a3200640 1 Processor -- start 2024-03-06T07:36:20.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.324+0000 7f05a3200640 1 -- start start 2024-03-06T07:36:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.326+0000 7f05a3200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f05a415c920 0x7f05a4070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.326+0000 7f05a3200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f05a40cd060 0x7f05a406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.326+0000 7f05a3200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a40cdad0 con 0x7f05a40ce850 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.326+0000 7f05a3200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a4071500 con 0x7f05a40cd060 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.326+0000 7f05a3200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a40cdd70 con 0x7f05a415c920 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f05a40cd060 0x7f05a406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f05a40cd060 0x7f05a406cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43606/0 (socket says 172.21.15.6:43606) 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 -- 172.21.15.6:0/3203505992 learned_addr learned my addr 172.21.15.6:0/3203505992 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a1e00640 1 --2- 172.21.15.6:0/3203505992 >> v2:172.21.15.178:3300/0 conn(0x7f05a415c920 0x7f05a4070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 -- 172.21.15.6:0/3203505992 >> v2:172.21.15.178:3300/0 conn(0x7f05a415c920 msgr2=0x7f05a4070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 --2- 172.21.15.6:0/3203505992 >> v2:172.21.15.178:3300/0 conn(0x7f05a415c920 0x7f05a4070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 -- 172.21.15.6:0/3203505992 >> v1:172.21.15.178:6789/0 conn(0x7f05a40ce850 legacy=0x7f05a40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:20.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.327+0000 7f05a2800640 1 -- 172.21.15.6:0/3203505992 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a40ce050 con 0x7f05a40cd060 2024-03-06T07:36:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.328+0000 7f05a2800640 1 --2- 172.21.15.6:0/3203505992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a406cac0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f05a4072230 tx=0x7f059c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f89d74e470722d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.329+0000 7f05a0a00640 1 -- 172.21.15.6:0/3203505992 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c035020 con 0x7f05a40cd060 2024-03-06T07:36:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.329+0000 7f05a0a00640 1 -- 172.21.15.6:0/3203505992 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c031640 con 0x7f05a40cd060 2024-03-06T07:36:20.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.329+0000 7f05a0a00640 1 -- 172.21.15.6:0/3203505992 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c02d040 con 0x7f05a40cd060 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 -- 172.21.15.6:0/3203505992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 msgr2=0x7f05a406cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 --2- 172.21.15.6:0/3203505992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a406cac0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f05a4072230 tx=0x7f059c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 -- 172.21.15.6:0/3203505992 shutdown_connections 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 --2- 172.21.15.6:0/3203505992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a406cac0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 --2- 172.21.15.6:0/3203505992 >> v2:172.21.15.178:3300/0 conn(0x7f05a415c920 0x7f05a4070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 -- 172.21.15.6:0/3203505992 >> 172.21.15.6:0/3203505992 conn(0x7f05a406b3d0 msgr2=0x7f05a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 -- 172.21.15.6:0/3203505992 shutdown_connections 2024-03-06T07:36:20.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.330+0000 7f05a3200640 1 -- 172.21.15.6:0/3203505992 wait complete. 2024-03-06T07:36:20.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.333+0000 7f05a3200640 1 Processor -- start 2024-03-06T07:36:20.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.333+0000 7f05a3200640 1 -- start start 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.334+0000 7f05a3200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a40b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.334+0000 7f05a3200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f05a40ce850 0x7f05a41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a3200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05a415c920 0x7f05a41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a3200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4070f80 con 0x7f05a40cd060 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a3200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4163790 con 0x7f05a415c920 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a3200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4071100 con 0x7f05a40ce850 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a40b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a40b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43612/0 (socket says 172.21.15.6:43612) 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 -- 172.21.15.6:0/1991191919 learned_addr learned my addr 172.21.15.6:0/1991191919 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a2800640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05a415c920 0x7f05a41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 -- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f05a40ce850 msgr2=0x7f05a41541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1400640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f05a40ce850 0x7f05a41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f05a40ce850 0x7f05a41541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 -- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05a415c920 msgr2=0x7f05a41529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05a415c920 0x7f05a41529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a1e00640 1 -- 172.21.15.6:0/1991191919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a4151900 con 0x7f05a40cd060 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.335+0000 7f05a2800640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05a415c920 0x7f05a41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.336+0000 7f05a1e00640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a40b21b0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f0594007e40 tx=0x7f059400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.336+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594014620 con 0x7f05a40cd060 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.336+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0594019070 con 0x7f05a40cd060 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.336+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a4151be0 con 0x7f05a40cd060 2024-03-06T07:36:20.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.336+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594014a00 con 0x7f05a40cd060 2024-03-06T07:36:20.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.336+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a41521a0 con 0x7f05a40cd060 2024-03-06T07:36:20.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.337+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a404f330 con 0x7f05a40cd060 2024-03-06T07:36:20.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.338+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0594014ba0 con 0x7f05a40cd060 2024-03-06T07:36:20.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.339+0000 7f059a000640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f058406b4d0 0x7f058406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:20.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.339+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f059408ac60 con 0x7f05a40cd060 2024-03-06T07:36:20.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.340+0000 7f05a1400640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f058406b4d0 0x7f058406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:20.343 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.341+0000 7f05a1400640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f058406b4d0 0x7f058406d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f05a41fb6c0 tx=0x7f0590009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:20.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.344+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0594059970 con 0x7f05a40cd060 2024-03-06T07:36:20.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.589+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05a40cf3a0 con 0x7f05a40cd060 2024-03-06T07:36:20.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.589+0000 7f059a000640 1 -- 172.21.15.6:0/1991191919 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f0594059270 con 0x7f05a40cd060 2024-03-06T07:36:20.592 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:20.592 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:20.592 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f058406b4d0 msgr2=0x7f058406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f058406b4d0 0x7f058406d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f05a41fb6c0 tx=0x7f0590009040 comp rx=0 tx=0).stop 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 msgr2=0x7f05a40b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a40b21b0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f0594007e40 tx=0x7f059400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 shutdown_connections 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f058406b4d0 0x7f058406d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05a415c920 0x7f05a41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f05a40ce850 0x7f05a41541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 --2- 172.21.15.6:0/1991191919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f05a40cd060 0x7f05a40b21b0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.592+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 >> 172.21.15.6:0/1991191919 conn(0x7f05a406b3d0 msgr2=0x7f05a415bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.593+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 shutdown_connections 2024-03-06T07:36:20.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:20.593+0000 7f05a3200640 1 -- 172.21.15.6:0/1991191919 wait complete. 2024-03-06T07:36:20.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:20.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:21.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.044+0000 7efc70600640 1 Processor -- start 2024-03-06T07:36:22.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.044+0000 7efc70600640 1 -- start start 2024-03-06T07:36:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.046+0000 7efc70600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc680cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.046+0000 7efc70600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7efc680cf5c0 0x7efc680b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.046+0000 7efc70600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc680b45b0 con 0x7efc6806cb10 2024-03-06T07:36:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.046+0000 7efc70600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc680ce160 con 0x7efc680cf5c0 2024-03-06T07:36:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.046+0000 7efc70600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc680b4860 con 0x7efc680b3c90 2024-03-06T07:36:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc6e600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efc6806cb10 0x7efc6806cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43168/0 (socket says 172.21.15.6:43168) 2024-03-06T07:36:22.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc6e600640 1 -- 172.21.15.6:0/2433975137 learned_addr learned my addr 172.21.15.6:0/2433975137 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958545498 0 0) 0x7efc680b45b0 con 0x7efc6806cb10 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc6d200640 1 --2- 172.21.15.6:0/2433975137 >> v2:172.21.15.178:3300/0 conn(0x7efc680cf5c0 0x7efc680b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50005730 con 0x7efc6806cb10 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc6dc00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc680cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464372693 0 0) 0x7efc50005730 con 0x7efc6806cb10 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.047+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 msgr2=0x7efc680cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc680cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 >> v2:172.21.15.178:3300/0 conn(0x7efc680cf5c0 msgr2=0x7efc680b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 --2- 172.21.15.6:0/2433975137 >> v2:172.21.15.178:3300/0 conn(0x7efc680cf5c0 0x7efc680b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680b4b40 con 0x7efc6806cb10 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7efc60001670 con 0x7efc6806cb10 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 >> v1:172.21.15.6:6789/0 conn(0x7efc6806cb10 legacy=0x7efc6806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc50005c30 0x7efc50006050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc50006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc50006f50 0x7efc500073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680ce160 con 0x7efc50006f50 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680b4860 con 0x7efc50005c30 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc5000dc90 con 0x7efc680b3c90 2024-03-06T07:36:22.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc6e600640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc50006f50 0x7efc500073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc6dc00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc50005c30 0x7efc50006050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc6d200640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc50006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc6e600640 1 -- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 msgr2=0x7efc50006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.048+0000 7efc6e600640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc50006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.049+0000 7efc6e600640 1 -- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc50005c30 msgr2=0x7efc50006050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.049+0000 7efc6e600640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc50005c30 0x7efc50006050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.049+0000 7efc6e600640 1 -- 172.21.15.6:0/2433975137 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50005730 con 0x7efc50006f50 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.049+0000 7efc6dc00640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc50005c30 0x7efc50006050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.049+0000 7efc6d200640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc680b3c90 0x7efc50006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc6e600640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc50006f50 0x7efc500073d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7efc6000c160 tx=0x7efc50007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6878c51f7fc9cd03 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60007c80 con 0x7efc50006f50 2024-03-06T07:36:22.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60030070 con 0x7efc50006f50 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc67e00640 1 -- 172.21.15.6:0/2433975137 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60005d30 con 0x7efc50006f50 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc70600640 1 -- 172.21.15.6:0/2433975137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc50006f50 msgr2=0x7efc500073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc70600640 1 --2- 172.21.15.6:0/2433975137 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc50006f50 0x7efc500073d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7efc6000c160 tx=0x7efc50007a40 comp rx=0 tx=0).stop 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.050+0000 7efc6e600640 1 -- 172.21.15.6:0/2433975137 reap_dead start 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.051+0000 7efc70600640 1 -- 172.21.15.6:0/2433975137 shutdown_connections 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.051+0000 7efc70600640 1 -- 172.21.15.6:0/2433975137 >> 172.21.15.6:0/2433975137 conn(0x7efc6806afb0 msgr2=0x7efc6806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.051+0000 7efc70600640 1 -- 172.21.15.6:0/2433975137 shutdown_connections 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.051+0000 7efc70600640 1 -- 172.21.15.6:0/2433975137 wait complete. 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.052+0000 7efc70600640 1 Processor -- start 2024-03-06T07:36:22.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.053+0000 7efc70600640 1 -- start start 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.053+0000 7efc70600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc681f7330 0x7efc681e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.053+0000 7efc70600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc681f7710 0x7efc680c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.053+0000 7efc70600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 0x7efc680c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc70600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6815b340 con 0x7efc680c9690 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc70600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680b5100 con 0x7efc681f7330 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc70600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6806d690 con 0x7efc681f7710 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 0x7efc680c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 0x7efc680c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58690/0 (socket says 172.21.15.6:58690) 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 -- 172.21.15.6:0/587348180 learned_addr learned my addr 172.21.15.6:0/587348180 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6d200640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc681f7710 0x7efc680c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6dc00640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc681f7330 0x7efc681e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 -- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc681f7710 msgr2=0x7efc680c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc681f7710 0x7efc680c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 -- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc681f7330 msgr2=0x7efc681e01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc681f7330 0x7efc681e01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.054+0000 7efc6e600640 1 -- 172.21.15.6:0/587348180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc681f46c0 con 0x7efc680c9690 2024-03-06T07:36:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.055+0000 7efc6e600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 0x7efc680c9b10 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7efc6000c160 tx=0x7efc60000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.055+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60007c80 con 0x7efc680c9690 2024-03-06T07:36:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.055+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6002cc20 con 0x7efc680c9690 2024-03-06T07:36:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.055+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60001be0 con 0x7efc680c9690 2024-03-06T07:36:22.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.055+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc681f49a0 con 0x7efc680c9690 2024-03-06T07:36:22.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.055+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc681e0c80 con 0x7efc680c9690 2024-03-06T07:36:22.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.057+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc68069510 con 0x7efc680c9690 2024-03-06T07:36:22.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.058+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc60007070 con 0x7efc680c9690 2024-03-06T07:36:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.060+0000 7efc65600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efc4806b500 0x7efc4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.060+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efc600aad00 con 0x7efc680c9690 2024-03-06T07:36:22.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.060+0000 7efc6dc00640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efc4806b500 0x7efc4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:22.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.061+0000 7efc6dc00640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efc4806b500 0x7efc4806d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7efc5c003be0 tx=0x7efc5c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:22.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.064+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc60048030 con 0x7efc680c9690 2024-03-06T07:36:22.074 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.310+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc68069510 con 0x7efc680c9690 2024-03-06T07:36:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.311+0000 7efc65600640 1 -- 172.21.15.6:0/587348180 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7efc60079aa0 con 0x7efc680c9690 2024-03-06T07:36:22.313 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:22.313 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efc4806b500 msgr2=0x7efc4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efc4806b500 0x7efc4806d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7efc5c003be0 tx=0x7efc5c00b040 comp rx=0 tx=0).stop 2024-03-06T07:36:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 msgr2=0x7efc680c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 0x7efc680c9b10 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7efc6000c160 tx=0x7efc60000fe0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 shutdown_connections 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efc4806b500 0x7efc4806d9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efc680c9690 0x7efc680c9b10 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efc681f7710 0x7efc680c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 --2- 172.21.15.6:0/587348180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efc681f7330 0x7efc681e01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 >> 172.21.15.6:0/587348180 conn(0x7efc6806afb0 msgr2=0x7efc680ae450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 shutdown_connections 2024-03-06T07:36:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:22.314+0000 7efc70600640 1 -- 172.21.15.6:0/587348180 wait complete. 2024-03-06T07:36:22.535 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:22.535 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:23.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:23.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.763+0000 7fb597600640 1 Processor -- start 2024-03-06T07:36:23.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.763+0000 7fb597600640 1 -- start start 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.764+0000 7fb597600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb59015e2e0 0x7fb5900d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.764+0000 7fb597600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb5900d0d70 0x7fb5900d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.764+0000 7fb597600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5900b3940 con 0x7fb5900d1b30 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.764+0000 7fb597600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5900d4e70 con 0x7fb59015e2e0 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.764+0000 7fb597600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5900b3c40 con 0x7fb5900d0d70 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb59015e2e0 0x7fb5900d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb59015e2e0 0x7fb5900d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58706/0 (socket says 172.21.15.6:58706) 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 -- 172.21.15.6:0/540643576 learned_addr learned my addr 172.21.15.6:0/540643576 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:23.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb58fe00640 1 --2- 172.21.15.6:0/540643576 >> v2:172.21.15.178:3300/0 conn(0x7fb5900d0d70 0x7fb5900d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 -- 172.21.15.6:0/540643576 >> v2:172.21.15.178:3300/0 conn(0x7fb5900d0d70 msgr2=0x7fb5900d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:23.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 --2- 172.21.15.6:0/540643576 >> v2:172.21.15.178:3300/0 conn(0x7fb5900d0d70 0x7fb5900d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 -- 172.21.15.6:0/540643576 >> v1:172.21.15.178:6789/0 conn(0x7fb5900d1b30 legacy=0x7fb5900d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.765+0000 7fb594c00640 1 -- 172.21.15.6:0/540643576 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5900cf580 con 0x7fb59015e2e0 2024-03-06T07:36:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.768+0000 7fb594c00640 1 --2- 172.21.15.6:0/540643576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb59015e2e0 0x7fb5900d0830 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fb5900d3510 tx=0x7fb57802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d3f76ee6e5c8ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.768+0000 7fb58f400640 1 -- 172.21.15.6:0/540643576 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578034070 con 0x7fb59015e2e0 2024-03-06T07:36:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.768+0000 7fb58f400640 1 -- 172.21.15.6:0/540643576 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb578030860 con 0x7fb59015e2e0 2024-03-06T07:36:23.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.768+0000 7fb58f400640 1 -- 172.21.15.6:0/540643576 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578030c30 con 0x7fb59015e2e0 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 -- 172.21.15.6:0/540643576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb59015e2e0 msgr2=0x7fb5900d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 --2- 172.21.15.6:0/540643576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb59015e2e0 0x7fb5900d0830 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fb5900d3510 tx=0x7fb57802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 -- 172.21.15.6:0/540643576 shutdown_connections 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 --2- 172.21.15.6:0/540643576 >> v2:172.21.15.178:3300/0 conn(0x7fb5900d0d70 0x7fb5900d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 --2- 172.21.15.6:0/540643576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb59015e2e0 0x7fb5900d0830 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 -- 172.21.15.6:0/540643576 >> 172.21.15.6:0/540643576 conn(0x7fb5900d3f60 msgr2=0x7fb5900d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 -- 172.21.15.6:0/540643576 shutdown_connections 2024-03-06T07:36:23.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.769+0000 7fb597600640 1 -- 172.21.15.6:0/540643576 wait complete. 2024-03-06T07:36:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.772+0000 7fb597600640 1 Processor -- start 2024-03-06T07:36:23.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.772+0000 7fb597600640 1 -- start start 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb597600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 0x7fb5900b9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb597600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5900d1b30 0x7fb59014c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb597600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb59015e2e0 0x7fb5900c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb597600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5900cfe20 con 0x7fb5900d0d70 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb597600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5900cfb40 con 0x7fb5900d1b30 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb597600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5900b50e0 con 0x7fb59015e2e0 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb594c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 0x7fb5900b9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb594c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 0x7fb5900b9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58716/0 (socket says 172.21.15.6:58716) 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb594c00640 1 -- 172.21.15.6:0/3468725147 learned_addr learned my addr 172.21.15.6:0/3468725147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb58fe00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5900d1b30 0x7fb59014c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb594c00640 1 -- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb59015e2e0 msgr2=0x7fb5900c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.774+0000 7fb595600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb59015e2e0 0x7fb5900c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb594c00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb59015e2e0 0x7fb5900c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb594c00640 1 -- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5900d1b30 msgr2=0x7fb59014c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb594c00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5900d1b30 0x7fb59014c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb594c00640 1 -- 172.21.15.6:0/3468725147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59014c8c0 con 0x7fb5900d0d70 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb58fe00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5900d1b30 0x7fb59014c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb595600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb59015e2e0 0x7fb5900c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:23.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb594c00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 0x7fb5900b9250 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fb57802b9e0 tx=0x7fb5780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578034040 con 0x7fb5900d0d70 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb578039d90 con 0x7fb5900d0d70 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578043610 con 0x7fb5900d0d70 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.775+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5900b81b0 con 0x7fb5900d0d70 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.776+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5900b8770 con 0x7fb5900d0d70 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.777+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59004f3a0 con 0x7fb5900d0d70 2024-03-06T07:36:23.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.777+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb57804a060 con 0x7fb5900d0d70 2024-03-06T07:36:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.778+0000 7fb58cc00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb56c06b500 0x7fb56c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:23.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.778+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb5780ac350 con 0x7fb5900d0d70 2024-03-06T07:36:23.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.779+0000 7fb58fe00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb56c06b500 0x7fb56c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:23.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.780+0000 7fb58fe00640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb56c06b500 0x7fb56c06d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb590161a80 tx=0x7fb58000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:23.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:23.783+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb59014c8c0 con 0x7fb5900d0d70 2024-03-06T07:36:24.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.023+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb590068320 con 0x7fb5900d0d70 2024-03-06T07:36:24.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.024+0000 7fb58cc00640 1 -- 172.21.15.6:0/3468725147 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7fb57803e070 con 0x7fb5900d0d70 2024-03-06T07:36:24.026 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:24.026 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:24.026 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:24.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb56c06b500 msgr2=0x7fb56c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb56c06b500 0x7fb56c06d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb590161a80 tx=0x7fb58000b040 comp rx=0 tx=0).stop 2024-03-06T07:36:24.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 msgr2=0x7fb5900b9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:24.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 0x7fb5900b9250 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fb57802b9e0 tx=0x7fb5780046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 shutdown_connections 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb56c06b500 0x7fb56c06d9c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb5900d1b30 0x7fb59014c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb59015e2e0 0x7fb5900c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 --2- 172.21.15.6:0/3468725147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb5900d0d70 0x7fb5900b9250 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 >> 172.21.15.6:0/3468725147 conn(0x7fb5900d3f60 msgr2=0x7fb59006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 shutdown_connections 2024-03-06T07:36:24.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:24.027+0000 7fb597600640 1 -- 172.21.15.6:0/3468725147 wait complete. 2024-03-06T07:36:24.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:24.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:24.075 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:24.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:24.255 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:25.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:25.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.469+0000 7f570fa00640 1 Processor -- start 2024-03-06T07:36:25.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.469+0000 7f570fa00640 1 -- start start 2024-03-06T07:36:25.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.471+0000 7f570fa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f57080af580 0x7f570815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.471+0000 7f570fa00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f57080cdc20 0x7f57080b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.471+0000 7f570fa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570815cc40 con 0x7f57080af1a0 2024-03-06T07:36:25.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.471+0000 7f570fa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570815cf20 con 0x7f57080af580 2024-03-06T07:36:25.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.471+0000 7f570fa00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570815d200 con 0x7f57080cdc20 2024-03-06T07:36:25.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f5707e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f57080af580 0x7f570815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f570d000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f57080af1a0 0x7f57080b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43182/0 (socket says 172.21.15.6:43182) 2024-03-06T07:36:25.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f570d000640 1 -- 172.21.15.6:0/439904444 learned_addr learned my addr 172.21.15.6:0/439904444 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f570da00640 1 --2- 172.21.15.6:0/439904444 >> v2:172.21.15.178:3300/0 conn(0x7f57080cdc20 0x7f57080b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f5707e00640 1 -- 172.21.15.6:0/439904444 >> v2:172.21.15.178:3300/0 conn(0x7f57080cdc20 msgr2=0x7f57080b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f5707e00640 1 --2- 172.21.15.6:0/439904444 >> v2:172.21.15.178:3300/0 conn(0x7f57080cdc20 0x7f57080b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f5707e00640 1 -- 172.21.15.6:0/439904444 >> v1:172.21.15.6:6789/0 conn(0x7f57080af1a0 legacy=0x7f57080b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f5707e00640 1 -- 172.21.15.6:0/439904444 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570815d4e0 con 0x7f57080af580 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f5707400640 1 -- 172.21.15.6:0/439904444 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617752931 0 0) 0x7f570815cc40 con 0x7f57080af1a0 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.472+0000 7f570da00640 1 --2- 172.21.15.6:0/439904444 >> v2:172.21.15.178:3300/0 conn(0x7f57080cdc20 0x7f57080b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:25.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.473+0000 7f5707e00640 1 --2- 172.21.15.6:0/439904444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f570815c600 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f570806c0c0 tx=0x7f56f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b721783110c0b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f5707400640 1 -- 172.21.15.6:0/439904444 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f40315a0 con 0x7f57080af580 2024-03-06T07:36:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f5707400640 1 -- 172.21.15.6:0/439904444 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f4037070 con 0x7f57080af580 2024-03-06T07:36:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f5707400640 1 -- 172.21.15.6:0/439904444 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4031960 con 0x7f57080af580 2024-03-06T07:36:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 -- 172.21.15.6:0/439904444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 msgr2=0x7f570815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 --2- 172.21.15.6:0/439904444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f570815c600 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f570806c0c0 tx=0x7f56f402b840 comp rx=0 tx=0).stop 2024-03-06T07:36:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 -- 172.21.15.6:0/439904444 shutdown_connections 2024-03-06T07:36:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 --2- 172.21.15.6:0/439904444 >> v2:172.21.15.178:3300/0 conn(0x7f57080cdc20 0x7f57080b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 --2- 172.21.15.6:0/439904444 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f570815c600 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 -- 172.21.15.6:0/439904444 >> 172.21.15.6:0/439904444 conn(0x7f570806afb0 msgr2=0x7f570806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 -- 172.21.15.6:0/439904444 shutdown_connections 2024-03-06T07:36:25.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.474+0000 7f570fa00640 1 -- 172.21.15.6:0/439904444 wait complete. 2024-03-06T07:36:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.476+0000 7f570fa00640 1 Processor -- start 2024-03-06T07:36:25.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.476+0000 7f570fa00640 1 -- start start 2024-03-06T07:36:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.477+0000 7f570fa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57080af1a0 0x7f570806d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.477+0000 7f570fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f57081f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.477+0000 7f570fa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57080cdc20 0x7f57081566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.477+0000 7f570fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57080b36a0 con 0x7f57080af580 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.477+0000 7f570fa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57080b3520 con 0x7f57080af1a0 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.477+0000 7f570fa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57080b3820 con 0x7f57080cdc20 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f57081f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f57081f2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58736/0 (socket says 172.21.15.6:58736) 2024-03-06T07:36:25.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 -- 172.21.15.6:0/785163705 learned_addr learned my addr 172.21.15.6:0/785163705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f570d000640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57080af1a0 0x7f570806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f570da00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57080cdc20 0x7f57081566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 -- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57080cdc20 msgr2=0x7f57081566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57080cdc20 0x7f57081566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 -- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57080af1a0 msgr2=0x7f570806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57080af1a0 0x7f570806d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 -- 172.21.15.6:0/785163705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57081f2aa0 con 0x7f57080af580 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f570d000640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57080af1a0 0x7f570806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f570da00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57080cdc20 0x7f57081566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.478+0000 7f5707e00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f57081f2180 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f56f402be50 tx=0x7f56f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.479+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4032840 con 0x7f57080af580 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.479+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f40329e0 con 0x7f57080af580 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.479+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4032dd0 con 0x7f57080af580 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.479+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57081f2d40 con 0x7f57080af580 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.479+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57081f3300 con 0x7f57080af580 2024-03-06T07:36:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.480+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5708069510 con 0x7f57080af580 2024-03-06T07:36:25.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.482+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56f404c020 con 0x7f57080af580 2024-03-06T07:36:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.483+0000 7f5704c00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56e806b500 0x7f56e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.483+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56f402f020 con 0x7f57080af580 2024-03-06T07:36:25.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.484+0000 7f570d000640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56e806b500 0x7f56e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:25.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.485+0000 7f570d000640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56e806b500 0x7f56e806d9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f56f80012e0 tx=0x7f56f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:25.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.488+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56f4040070 con 0x7f57080af580 2024-03-06T07:36:25.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.728+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f570804f3a0 con 0x7f57080af580 2024-03-06T07:36:25.731 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.729+0000 7f5704c00640 1 -- 172.21.15.6:0/785163705 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f56f407a720 con 0x7f57080af580 2024-03-06T07:36:25.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:25.731 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:25.731 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:25.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.731+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56e806b500 msgr2=0x7f56e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.731+0000 7f570fa00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56e806b500 0x7f56e806d9c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f56f80012e0 tx=0x7f56f8009040 comp rx=0 tx=0).stop 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.731+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 msgr2=0x7f57081f2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.731+0000 7f570fa00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f57081f2180 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f56f402be50 tx=0x7f56f40046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.731+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 shutdown_connections 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.731+0000 7f570fa00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f56e806b500 0x7f56e806d9c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.732+0000 7f570fa00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f57080cdc20 0x7f57081566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.732+0000 7f570fa00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57080af580 0x7f57081f2180 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.732+0000 7f570fa00640 1 --2- 172.21.15.6:0/785163705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57080af1a0 0x7f570806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.732+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 >> 172.21.15.6:0/785163705 conn(0x7f570806afb0 msgr2=0x7f570806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.732+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 shutdown_connections 2024-03-06T07:36:25.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:25.732+0000 7f570fa00640 1 -- 172.21.15.6:0/785163705 wait complete. 2024-03-06T07:36:25.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:25.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:26.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.190+0000 7f5031e00640 1 Processor -- start 2024-03-06T07:36:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.190+0000 7f5031e00640 1 -- start start 2024-03-06T07:36:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.191+0000 7f5031e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f502c0af1a0 0x7f502c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f5031e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f502c0cdc20 0x7f502c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f5031e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502c15cc40 con 0x7f502c0af580 2024-03-06T07:36:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f5031e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502c15cf20 con 0x7f502c0cdc20 2024-03-06T07:36:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f5031e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502c15d200 con 0x7f502c0af1a0 2024-03-06T07:36:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f502a000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f502c0af580 0x7f502c0b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43184/0 (socket says 172.21.15.6:43184) 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f502a000640 1 -- 172.21.15.6:0/2720955920 learned_addr learned my addr 172.21.15.6:0/2720955920 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f502b400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f502c0cdc20 0x7f502c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.192+0000 7f502aa00640 1 --2- 172.21.15.6:0/2720955920 >> v2:172.21.15.178:3300/0 conn(0x7f502c0af1a0 0x7f502c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.193+0000 7f5029600640 1 -- 172.21.15.6:0/2720955920 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274969407 0 0) 0x7f502c15cc40 con 0x7f502c0af580 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.193+0000 7f502b400640 1 -- 172.21.15.6:0/2720955920 >> v2:172.21.15.178:3300/0 conn(0x7f502c0af1a0 msgr2=0x7f502c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.193+0000 7f502b400640 1 --2- 172.21.15.6:0/2720955920 >> v2:172.21.15.178:3300/0 conn(0x7f502c0af1a0 0x7f502c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.193+0000 7f502b400640 1 -- 172.21.15.6:0/2720955920 >> v1:172.21.15.6:6789/0 conn(0x7f502c0af580 legacy=0x7f502c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.193+0000 7f502b400640 1 -- 172.21.15.6:0/2720955920 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c15d4e0 con 0x7f502c0cdc20 2024-03-06T07:36:27.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.194+0000 7f502b400640 1 --2- 172.21.15.6:0/2720955920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdc20 0x7f502c0b2dd0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f502c074960 tx=0x7f502402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4c784b4fa9c701 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.195+0000 7f5029600640 1 -- 172.21.15.6:0/2720955920 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024039070 con 0x7f502c0cdc20 2024-03-06T07:36:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.195+0000 7f5029600640 1 -- 172.21.15.6:0/2720955920 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502402cbf0 con 0x7f502c0cdc20 2024-03-06T07:36:27.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.195+0000 7f5029600640 1 -- 172.21.15.6:0/2720955920 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024034440 con 0x7f502c0cdc20 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.195+0000 7f5031e00640 1 -- 172.21.15.6:0/2720955920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdc20 msgr2=0x7f502c0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.195+0000 7f5031e00640 1 --2- 172.21.15.6:0/2720955920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdc20 0x7f502c0b2dd0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f502c074960 tx=0x7f502402be40 comp rx=0 tx=0).stop 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.196+0000 7f5031e00640 1 -- 172.21.15.6:0/2720955920 shutdown_connections 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.196+0000 7f5031e00640 1 --2- 172.21.15.6:0/2720955920 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0cdc20 0x7f502c0b2dd0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.196+0000 7f5031e00640 1 --2- 172.21.15.6:0/2720955920 >> v2:172.21.15.178:3300/0 conn(0x7f502c0af1a0 0x7f502c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.196+0000 7f5031e00640 1 -- 172.21.15.6:0/2720955920 >> 172.21.15.6:0/2720955920 conn(0x7f502c06afb0 msgr2=0x7f502c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.196+0000 7f5031e00640 1 -- 172.21.15.6:0/2720955920 shutdown_connections 2024-03-06T07:36:27.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.196+0000 7f5031e00640 1 -- 172.21.15.6:0/2720955920 wait complete. 2024-03-06T07:36:27.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.198+0000 7f5031e00640 1 Processor -- start 2024-03-06T07:36:27.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.199+0000 7f5031e00640 1 -- start start 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.200+0000 7f5031e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f502c0af1a0 0x7f502c14dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.200+0000 7f5031e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 0x7f502c0acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.200+0000 7f5031e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f502c0cdc20 0x7f502c0ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.200+0000 7f5031e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0b36a0 con 0x7f502c0af580 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.200+0000 7f5031e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0b3520 con 0x7f502c0cdc20 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.200+0000 7f5031e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502c0b3820 con 0x7f502c0af1a0 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 0x7f502c0acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 0x7f502c0acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58756/0 (socket says 172.21.15.6:58756) 2024-03-06T07:36:27.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 -- 172.21.15.6:0/212326884 learned_addr learned my addr 172.21.15.6:0/212326884 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 -- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f502c0af1a0 msgr2=0x7f502c14dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502b400640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f502c0cdc20 0x7f502c0ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502aa00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f502c0af1a0 0x7f502c14dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f502c0af1a0 0x7f502c14dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 -- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f502c0cdc20 msgr2=0x7f502c0ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f502c0cdc20 0x7f502c0ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.201+0000 7f502a000640 1 -- 172.21.15.6:0/212326884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c09d6a0 con 0x7f502c0af580 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.202+0000 7f502a000640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 0x7f502c0acdb0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f501800c4e0 tx=0x7f501800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.202+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018018070 con 0x7f502c0af580 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.202+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5018007500 con 0x7f502c0af580 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.202+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018004a20 con 0x7f502c0af580 2024-03-06T07:36:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.202+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502c14e860 con 0x7f502c0af580 2024-03-06T07:36:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.202+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502c14ee20 con 0x7f502c0af580 2024-03-06T07:36:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.204+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502c069510 con 0x7f502c0af580 2024-03-06T07:36:27.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.204+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f501801b070 con 0x7f502c0af580 2024-03-06T07:36:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.206+0000 7f5023400640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500c06b4d0 0x7f500c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.206+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f50180895b0 con 0x7f502c0af580 2024-03-06T07:36:27.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.206+0000 7f502aa00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500c06b4d0 0x7f500c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:27.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.208+0000 7f502aa00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500c06b4d0 0x7f500c06d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f501c006fd0 tx=0x7f501c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:27.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.210+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50180584a0 con 0x7f502c0af580 2024-03-06T07:36:27.454 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.452+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f502c04f3a0 con 0x7f502c0af580 2024-03-06T07:36:27.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.453+0000 7f5023400640 1 -- 172.21.15.6:0/212326884 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f5018087e20 con 0x7f502c0af580 2024-03-06T07:36:27.455 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:27.455 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:27.455 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500c06b4d0 msgr2=0x7f500c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500c06b4d0 0x7f500c06d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f501c006fd0 tx=0x7f501c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 msgr2=0x7f502c0acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 0x7f502c0acdb0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f501800c4e0 tx=0x7f501800c9a0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 shutdown_connections 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f500c06b4d0 0x7f500c06d990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f502c0cdc20 0x7f502c0ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f502c0af580 0x7f502c0acdb0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 --2- 172.21.15.6:0/212326884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f502c0af1a0 0x7f502c14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.455+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 >> 172.21.15.6:0/212326884 conn(0x7f502c06afb0 msgr2=0x7f502c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.456+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 shutdown_connections 2024-03-06T07:36:27.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:27.456+0000 7f5031e00640 1 -- 172.21.15.6:0/212326884 wait complete. 2024-03-06T07:36:27.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:27.686 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:28.080 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:28.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.901+0000 7f7aa7200640 1 Processor -- start 2024-03-06T07:36:28.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.901+0000 7f7aa7200640 1 -- start start 2024-03-06T07:36:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.903+0000 7f7aa7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 0x7f7aa80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.903+0000 7f7aa7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80b3710 con 0x7f7aa80d0e50 2024-03-06T07:36:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.903+0000 7f7aa7200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa80b39f0 con 0x7f7aa80d4a10 2024-03-06T07:36:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.903+0000 7f7aa7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80b09e0 con 0x7f7aa815e0b0 2024-03-06T07:36:28.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.903+0000 7f7aa6800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7aa80d0e50 0x7f7aa80b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43190/0 (socket says 172.21.15.6:43190) 2024-03-06T07:36:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.903+0000 7f7aa6800640 1 -- 172.21.15.6:0/1751206922 learned_addr learned my addr 172.21.15.6:0/1751206922 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 0x7f7aa80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190598528 0 0) 0x7f7aa80b3710 con 0x7f7aa80d0e50 2024-03-06T07:36:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a8c003520 con 0x7f7aa80d0e50 2024-03-06T07:36:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 732955071 0 0) 0x7f7a8c003520 con 0x7f7aa80d0e50 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 msgr2=0x7f7aa80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 0x7f7aa80cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 >> v1:172.21.15.178:6789/0 conn(0x7f7aa80d4a10 legacy=0x7f7aa80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa80b3cd0 con 0x7f7aa80d0e50 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7aa00019f0 con 0x7f7aa80d0e50 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 >> v1:172.21.15.6:6789/0 conn(0x7f7aa80d0e50 legacy=0x7f7aa80b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8c0039e0 0x7f7a8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.904+0000 7f7aa4a00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a8c004800 0x7f7a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa4a00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 0x7f7a8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c00ba70 con 0x7f7a8c0039e0 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c00bbf0 con 0x7f7a8c004800 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c00bdf0 con 0x7f7aa815e0b0 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8c0039e0 0x7f7a8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5e00640 1 -- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 msgr2=0x7f7a8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7aa815e0b0 0x7f7a8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5400640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a8c004800 0x7f7a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5e00640 1 -- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a8c004800 msgr2=0x7f7a8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a8c004800 0x7f7a8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5e00640 1 -- 172.21.15.6:0/1751206922 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c003520 con 0x7f7a8c0039e0 2024-03-06T07:36:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.905+0000 7f7aa5400640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a8c004800 0x7f7a8c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.906+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8c0039e0 0x7f7a8c003e00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009a90 tx=0x7f7a8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cb8a529ca1200 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98039070 con 0x7f7a8c0039e0 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9802c6b0 con 0x7f7a8c0039e0 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa4a00640 1 -- 172.21.15.6:0/1751206922 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98009690 con 0x7f7a8c0039e0 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa7200640 1 -- 172.21.15.6:0/1751206922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8c0039e0 msgr2=0x7f7a8c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa7200640 1 --2- 172.21.15.6:0/1751206922 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7a8c0039e0 0x7f7a8c003e00 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009a90 tx=0x7f7a8c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa6800640 1 -- 172.21.15.6:0/1751206922 reap_dead start 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa7200640 1 -- 172.21.15.6:0/1751206922 shutdown_connections 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa7200640 1 -- 172.21.15.6:0/1751206922 >> 172.21.15.6:0/1751206922 conn(0x7f7aa80d3d30 msgr2=0x7f7aa80d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa7200640 1 -- 172.21.15.6:0/1751206922 shutdown_connections 2024-03-06T07:36:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.907+0000 7f7aa7200640 1 -- 172.21.15.6:0/1751206922 wait complete. 2024-03-06T07:36:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.909+0000 7f7aa7200640 1 Processor -- start 2024-03-06T07:36:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.909+0000 7f7aa7200640 1 -- start start 2024-03-06T07:36:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a8c0039e0 0x7f7aa815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7aa80cd6e0 0x7f7aa809e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 0x7f7aa81f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80cfc60 con 0x7f7aa80b8f70 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa815e490 con 0x7f7aa80cd6e0 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa80cf310 con 0x7f7a8c0039e0 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 0x7f7aa81f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa6800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 0x7f7aa81f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58772/0 (socket says 172.21.15.6:58772) 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.910+0000 7f7aa6800640 1 -- 172.21.15.6:0/4234685724 learned_addr learned my addr 172.21.15.6:0/4234685724 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a8c0039e0 0x7f7aa815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa5400640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7aa80cd6e0 0x7f7aa809e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa6800640 1 -- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a8c0039e0 msgr2=0x7f7aa815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa6800640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a8c0039e0 0x7f7aa815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa6800640 1 -- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7aa80cd6e0 msgr2=0x7f7aa809e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa6800640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7aa80cd6e0 0x7f7aa809e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa6800640 1 -- 172.21.15.6:0/4234685724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa815b8a0 con 0x7f7aa80b8f70 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a8c0039e0 0x7f7aa815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa5400640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7aa80cd6e0 0x7f7aa809e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7aa6800640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 0x7f7aa81f73a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f7aa808e2e0 tx=0x7f7aa000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.911+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000a5b0 con 0x7f7aa80b8f70 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.912+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa000a750 con 0x7f7aa80b8f70 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.912+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa000ab20 con 0x7f7aa80b8f70 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.912+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa815bb80 con 0x7f7aa80b8f70 2024-03-06T07:36:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.912+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa815c140 con 0x7f7aa80b8f70 2024-03-06T07:36:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.913+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa804f330 con 0x7f7aa80b8f70 2024-03-06T07:36:28.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.914+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aa000acc0 con 0x7f7aa80b8f70 2024-03-06T07:36:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.915+0000 7f7a9e000640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a8806b500 0x7f7a8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.915+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7aa00897c0 con 0x7f7aa80b8f70 2024-03-06T07:36:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.916+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a8806b500 0x7f7a8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:28.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.917+0000 7f7aa5e00640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a8806b500 0x7f7a8806d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7a98036000 tx=0x7f7a98030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:28.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:28.920+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7aa001a030 con 0x7f7aa80b8f70 2024-03-06T07:36:29.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.157+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aa806c790 con 0x7f7aa80b8f70 2024-03-06T07:36:29.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.157+0000 7f7a9e000640 1 -- 172.21.15.6:0/4234685724 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+3406 (secure 0 0 0) 0x7f7aa000f260 con 0x7f7aa80b8f70 2024-03-06T07:36:29.160 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:29.160 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":99,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:35:59.302163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:29.160 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 99 2024-03-06T07:36:29.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a8806b500 msgr2=0x7f7a8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a8806b500 0x7f7a8806d9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7a98036000 tx=0x7f7a98030000 comp rx=0 tx=0).stop 2024-03-06T07:36:29.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 msgr2=0x7f7aa81f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:29.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 0x7f7aa81f73a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f7aa808e2e0 tx=0x7f7aa000ccd0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 shutdown_connections 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7a8806b500 0x7f7a8806d9c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7aa80cd6e0 0x7f7aa809e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7aa80b8f70 0x7f7aa81f73a0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 --2- 172.21.15.6:0/4234685724 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7a8c0039e0 0x7f7aa815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.160+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 >> 172.21.15.6:0/4234685724 conn(0x7f7aa80d3d30 msgr2=0x7f7aa8073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.161+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 shutdown_connections 2024-03-06T07:36:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:29.161+0000 7f7aa7200640 1 -- 172.21.15.6:0/4234685724 wait complete. 2024-03-06T07:36:29.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:35:59.302163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:29.383 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:30.082 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:30.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.616+0000 7fc6e1200640 1 Processor -- start 2024-03-06T07:36:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.616+0000 7fc6e1200640 1 -- start start 2024-03-06T07:36:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.618+0000 7fc6e1200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc6dc06d0d0 0x7fc6dc1656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.618+0000 7fc6e1200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0b5e30 con 0x7fc6dc0b1440 2024-03-06T07:36:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.618+0000 7fc6e1200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0b60b0 con 0x7fc6dc09ac80 2024-03-06T07:36:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.618+0000 7fc6e1200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc0b6390 con 0x7fc6dc06d0d0 2024-03-06T07:36:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.618+0000 7fc6da800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6dc0b1440 0x7fc6dc167c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43202/0 (socket says 172.21.15.6:43202) 2024-03-06T07:36:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.618+0000 7fc6da800640 1 -- 172.21.15.6:0/38982200 learned_addr learned my addr 172.21.15.6:0/38982200 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:30.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/38982200 >> v2:172.21.15.178:3300/0 conn(0x7fc6dc06d0d0 0x7fc6dc1656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295768269 0 0) 0x7fc6d4001490 con 0x7fc6dc0b1440 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c0003520 con 0x7fc6dc0b1440 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1830404819 0 0) 0x7fc6c0003520 con 0x7fc6dc0b1440 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 >> v2:172.21.15.178:3300/0 conn(0x7fc6dc06d0d0 msgr2=0x7fc6dc1656d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 --2- 172.21.15.6:0/38982200 >> v2:172.21.15.178:3300/0 conn(0x7fc6dc06d0d0 0x7fc6dc1656d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 >> v1:172.21.15.178:6789/0 conn(0x7fc6dc09ac80 legacy=0x7fc6dc0b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0b6670 con 0x7fc6dc0b1440 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.619+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc6d4001bc0 con 0x7fc6dc0b1440 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 >> v1:172.21.15.6:6789/0 conn(0x7fc6dc0b1440 legacy=0x7fc6dc167c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6c0003a70 0x7fc6c0003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c0004800 0x7fc6c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0005310 0x7fc6c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0b6390 con 0x7fc6c0004800 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c000c250 con 0x7fc6c0003a70 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c000c450 con 0x7fc6c0005310 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c0004800 0x7fc6c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d9e00640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6c0003a70 0x7fc6c0003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6da800640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0005310 0x7fc6c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d3e00640 1 -- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0005310 msgr2=0x7fc6c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0005310 0x7fc6c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d3e00640 1 -- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6c0003a70 msgr2=0x7fc6c0003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.620+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6c0003a70 0x7fc6c0003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.621+0000 7fc6d3e00640 1 -- 172.21.15.6:0/38982200 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c0003520 con 0x7fc6c0004800 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.621+0000 7fc6da800640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0005310 0x7fc6c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:30.625 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.621+0000 7fc6da800640 1 -- 172.21.15.6:0/38982200 reap_dead start 2024-03-06T07:36:30.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.622+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c0004800 0x7fc6c0004c80 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009ac0 tx=0x7fc6c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15baddc9678290b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4039070 con 0x7fc6c0004800 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c402c6b0 con 0x7fc6c0004800 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6d9400640 1 -- 172.21.15.6:0/38982200 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4009690 con 0x7fc6c0004800 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6e1200640 1 -- 172.21.15.6:0/38982200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c0004800 msgr2=0x7fc6c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6e1200640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c0004800 0x7fc6c0004c80 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009ac0 tx=0x7fc6c0005830 comp rx=0 tx=0).stop 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6e1200640 1 -- 172.21.15.6:0/38982200 shutdown_connections 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6e1200640 1 --2- 172.21.15.6:0/38982200 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6c0004800 0x7fc6c0004c80 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.623+0000 7fc6e1200640 1 -- 172.21.15.6:0/38982200 >> 172.21.15.6:0/38982200 conn(0x7fc6dc0b3a10 msgr2=0x7fc6dc0b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.624+0000 7fc6e1200640 1 -- 172.21.15.6:0/38982200 shutdown_connections 2024-03-06T07:36:30.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.624+0000 7fc6e1200640 1 -- 172.21.15.6:0/38982200 wait complete. 2024-03-06T07:36:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.625+0000 7fc6e1200640 1 Processor -- start 2024-03-06T07:36:30.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.625+0000 7fc6e1200640 1 -- start start 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.626+0000 7fc6e1200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0004800 0x7fc6dc155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.626+0000 7fc6e1200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 0x7fc6dc1f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.626+0000 7fc6e1200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6dc15acd0 0x7fc6dc0af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.626+0000 7fc6e1200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc09b1e0 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.626+0000 7fc6e1200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc09b060 con 0x7fc6dc15acd0 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.626+0000 7fc6e1200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc09b360 con 0x7fc6c0004800 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 0x7fc6dc1f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 0x7fc6dc1f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58782/0 (socket says 172.21.15.6:58782) 2024-03-06T07:36:30.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 -- 172.21.15.6:0/2103614243 learned_addr learned my addr 172.21.15.6:0/2103614243 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6da800640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6dc15acd0 0x7fc6dc0af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 -- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0004800 msgr2=0x7fc6dc155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0004800 0x7fc6dc155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 -- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6dc15acd0 msgr2=0x7fc6dc0af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6dc15acd0 0x7fc6dc0af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.627+0000 7fc6d3e00640 1 -- 172.21.15.6:0/2103614243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0ce960 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.628+0000 7fc6d3e00640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 0x7fc6dc1f3e80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009ac0 tx=0x7fc6c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.628+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4039050 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.628+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4045070 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.628+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40044e0 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.628+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc1f4740 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.628+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc1f4d00 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.629+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc04f3a0 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.632 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.630+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6c4003a50 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.631+0000 7fc6d2000640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6b806b500 0x7fc6b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:30.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.631+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc6c40abe40 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.632+0000 7fc6d9e00640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6b806b500 0x7fc6b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.633+0000 7fc6d9e00640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6b806b500 0x7fc6b806d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007fd0 tx=0x7fc6cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:30.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.637+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6c404e020 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.922+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6dc0b0e00 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.923+0000 7fc6d2000640 1 -- 172.21.15.6:0/2103614243 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fc6c407abe0 con 0x7fc6dc0d08e0 2024-03-06T07:36:30.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:30.925 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6b806b500 msgr2=0x7fc6b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6b806b500 0x7fc6b806d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007fd0 tx=0x7fc6cc009040 comp rx=0 tx=0).stop 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 msgr2=0x7fc6dc1f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 0x7fc6dc1f3e80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009ac0 tx=0x7fc6c4004730 comp rx=0 tx=0).stop 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 shutdown_connections 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc6b806b500 0x7fc6b806d9c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6dc15acd0 0x7fc6dc0af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6dc0d08e0 0x7fc6dc1f3e80 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 --2- 172.21.15.6:0/2103614243 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc6c0004800 0x7fc6dc155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:30.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.925+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 >> 172.21.15.6:0/2103614243 conn(0x7fc6dc0b3a10 msgr2=0x7fc6dc1607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:30.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.926+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 shutdown_connections 2024-03-06T07:36:30.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:30.926+0000 7fc6e1200640 1 -- 172.21.15.6:0/2103614243 wait complete. 2024-03-06T07:36:31.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:31.150 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:32.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.380+0000 7f58bb200640 1 Processor -- start 2024-03-06T07:36:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.380+0000 7f58bb200640 1 -- start start 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.381+0000 7f58bb200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f58bc15e0b0 0x7f58bc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.381+0000 7f58bb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc0d4a10 0x7f58bc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.381+0000 7f58bb200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58bc0d0c80 0x7f58bc0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.381+0000 7f58bb200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0b09e0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.381+0000 7f58bb200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc0b3710 con 0x7f58bc15e0b0 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.381+0000 7f58bb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0b39f0 con 0x7f58bc0d4a10 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58bc0d0c80 0x7f58bc0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f58bc0d0c80 0x7f58bc0d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45414/0 (socket says 172.21.15.6:45414) 2024-03-06T07:36:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 -- 172.21.15.6:0/2137237210 learned_addr learned my addr 172.21.15.6:0/2137237210 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58b9400640 1 --2- 172.21.15.6:0/2137237210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc0d4a10 0x7f58bc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58b9e00640 1 --2- 172.21.15.6:0/2137237210 >> v2:172.21.15.178:3300/0 conn(0x7f58bc15e0b0 0x7f58bc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 -- 172.21.15.6:0/2137237210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc0d4a10 msgr2=0x7f58bc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 --2- 172.21.15.6:0/2137237210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc0d4a10 0x7f58bc0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 -- 172.21.15.6:0/2137237210 >> v2:172.21.15.178:3300/0 conn(0x7f58bc15e0b0 msgr2=0x7f58bc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 --2- 172.21.15.6:0/2137237210 >> v2:172.21.15.178:3300/0 conn(0x7f58bc15e0b0 0x7f58bc0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.382+0000 7f58ba800640 1 -- 172.21.15.6:0/2137237210 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc0b3cd0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.384+0000 7f58ba800640 1 --2- 172.21.15.6:0/2137237210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc0d1900 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b68c0 tx=0x7f58b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ff8abf9de0d465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58b8a00640 1 -- 172.21.15.6:0/2137237210 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b402ca70 con 0x7f58bc0d0c80 2024-03-06T07:36:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58b8a00640 1 -- 172.21.15.6:0/2137237210 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b402cc10 con 0x7f58bc0d0c80 2024-03-06T07:36:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58b8a00640 1 -- 172.21.15.6:0/2137237210 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b40343f0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 -- 172.21.15.6:0/2137237210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 msgr2=0x7f58bc0d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 --2- 172.21.15.6:0/2137237210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc0d1900 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b68c0 tx=0x7f58b402bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 -- 172.21.15.6:0/2137237210 shutdown_connections 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 --2- 172.21.15.6:0/2137237210 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc0d1900 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 --2- 172.21.15.6:0/2137237210 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc0d4a10 0x7f58bc0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 --2- 172.21.15.6:0/2137237210 >> v2:172.21.15.178:3300/0 conn(0x7f58bc15e0b0 0x7f58bc0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 -- 172.21.15.6:0/2137237210 >> 172.21.15.6:0/2137237210 conn(0x7f58bc0d3d30 msgr2=0x7f58bc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 -- 172.21.15.6:0/2137237210 shutdown_connections 2024-03-06T07:36:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.385+0000 7f58bb200640 1 -- 172.21.15.6:0/2137237210 wait complete. 2024-03-06T07:36:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.388+0000 7f58bb200640 1 Processor -- start 2024-03-06T07:36:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.388+0000 7f58bb200640 1 -- start start 2024-03-06T07:36:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58bb200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc1f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58bb200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58bc0d4a10 0x7f58bc0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58bb200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc15e0b0 0x7f58bc0cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58bb200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0cfc60 con 0x7f58bc0d0c80 2024-03-06T07:36:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58bb200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc15e490 con 0x7f58bc0d4a10 2024-03-06T07:36:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58bb200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc0cf850 con 0x7f58bc15e0b0 2024-03-06T07:36:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc1f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58b9e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc1f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45424/0 (socket says 172.21.15.6:45424) 2024-03-06T07:36:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.389+0000 7f58b9e00640 1 -- 172.21.15.6:0/243118065 learned_addr learned my addr 172.21.15.6:0/243118065 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58ba800640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc15e0b0 0x7f58bc0cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9400640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58bc0d4a10 0x7f58bc0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9e00640 1 -- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc15e0b0 msgr2=0x7f58bc0cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9e00640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc15e0b0 0x7f58bc0cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9e00640 1 -- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58bc0d4a10 msgr2=0x7f58bc0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9e00640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58bc0d4a10 0x7f58bc0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9e00640 1 -- 172.21.15.6:0/243118065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc1f36c0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b9e00640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc1f3180 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00d620 tx=0x7f58ac00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac00b620 con 0x7f58bc0d0c80 2024-03-06T07:36:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac019070 con 0x7f58bc0d0c80 2024-03-06T07:36:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac00ba00 con 0x7f58bc0d0c80 2024-03-06T07:36:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.390+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc0cb700 con 0x7f58bc0d0c80 2024-03-06T07:36:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.391+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc0cbcc0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.392+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc0cabb0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.393+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58ac00bba0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.394+0000 7f58b2000640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:32.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.394+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58ac08a660 con 0x7f58bc0d0c80 2024-03-06T07:36:32.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.394+0000 7f58b9400640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:32.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.396+0000 7f58b9400640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f589c06b4d0 0x7f589c06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b91f0 tx=0x7f58a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:32.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.398+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58bc0cabb0 con 0x7f58bc0d0c80 2024-03-06T07:36:32.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.640+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58bc15e610 con 0x7f58bc0d0c80 2024-03-06T07:36:32.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.641+0000 7f58b2000640 1 -- 172.21.15.6:0/243118065 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f58ac059400 con 0x7f58bc0d0c80 2024-03-06T07:36:32.644 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:32.644 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:32.644 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.644+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f589c06b4d0 msgr2=0x7f589c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.644+0000 7f58bb200640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f589c06b4d0 0x7f589c06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b91f0 tx=0x7f58a8008040 comp rx=0 tx=0).stop 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.644+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 msgr2=0x7f58bc1f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.644+0000 7f58bb200640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc1f3180 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f58ac00d620 tx=0x7f58ac00dae0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 shutdown_connections 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f589c06b4d0 0x7f589c06d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f58bc0d4a10 0x7f58bc0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58bc15e0b0 0x7f58bc0cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 --2- 172.21.15.6:0/243118065 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58bc0d0c80 0x7f58bc1f3180 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:32.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 >> 172.21.15.6:0/243118065 conn(0x7f58bc0d3d30 msgr2=0x7f58bc0b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:32.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 shutdown_connections 2024-03-06T07:36:32.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:32.645+0000 7f58bb200640 1 -- 172.21.15.6:0/243118065 wait complete. 2024-03-06T07:36:32.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:32.869 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:33.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:34.085 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:34.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.097+0000 7f98e3e00640 1 Processor -- start 2024-03-06T07:36:34.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.097+0000 7f98e3e00640 1 -- start start 2024-03-06T07:36:34.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.099+0000 7f98e3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 0x7f98e4168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.099+0000 7f98e3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40b3710 con 0x7f98e40d4a10 2024-03-06T07:36:34.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.099+0000 7f98e3e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40b39f0 con 0x7f98e415e0b0 2024-03-06T07:36:34.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.099+0000 7f98e3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40b09e0 con 0x7f98e40d0e50 2024-03-06T07:36:34.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.099+0000 7f98e2000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f98e40d4a10 0x7f98e40d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43516/0 (socket says 172.21.15.6:43516) 2024-03-06T07:36:34.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.099+0000 7f98e2000640 1 -- 172.21.15.6:0/1372168884 learned_addr learned my addr 172.21.15.6:0/1372168884 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e3400640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 0x7f98e4168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960449176 0 0) 0x7f98e40b3710 con 0x7f98e40d4a10 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c8003520 con 0x7f98e40d4a10 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133582694 0 0) 0x7f98c8003520 con 0x7f98e40d4a10 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 msgr2=0x7f98e4168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 0x7f98e4168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 >> v1:172.21.15.178:6789/0 conn(0x7f98e415e0b0 legacy=0x7f98e40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40b3cd0 con 0x7f98e40d4a10 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f98cc001690 con 0x7f98e40d4a10 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 >> v1:172.21.15.6:6789/0 conn(0x7f98e40d4a10 legacy=0x7f98e40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 0x7f98c80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8004010 0x7f98c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98c8004d40 0x7f98c80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c800bb50 con 0x7f98c8004d40 2024-03-06T07:36:34.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c800bd00 con 0x7f98c8004010 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.100+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c800bf00 con 0x7f98e40d0e50 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e3400640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98c8004d40 0x7f98c80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e2a00640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 0x7f98c80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e2000640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8004010 0x7f98c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e3400640 1 -- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 msgr2=0x7f98c80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e3400640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98e40d0e50 0x7f98c80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e3400640 1 -- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8004010 msgr2=0x7f98c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e3400640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98c8004010 0x7f98c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.101+0000 7f98e3400640 1 -- 172.21.15.6:0/1372168884 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40b3cd0 con 0x7f98c8004d40 2024-03-06T07:36:34.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.103+0000 7f98e3400640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98c8004d40 0x7f98c80051c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f98dc007330 tx=0x7f98c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b43da077a6bf09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.103+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc039070 con 0x7f98c8004d40 2024-03-06T07:36:34.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc02cca0 con 0x7f98c8004d40 2024-03-06T07:36:34.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e1600640 1 -- 172.21.15.6:0/1372168884 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc009be0 con 0x7f98c8004d40 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3e00640 1 -- 172.21.15.6:0/1372168884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98c8004d40 msgr2=0x7f98c80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3e00640 1 --2- 172.21.15.6:0/1372168884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98c8004d40 0x7f98c80051c0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f98dc007330 tx=0x7f98c8005830 comp rx=0 tx=0).stop 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3400640 1 -- 172.21.15.6:0/1372168884 reap_dead start 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3e00640 1 -- 172.21.15.6:0/1372168884 shutdown_connections 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3e00640 1 -- 172.21.15.6:0/1372168884 >> 172.21.15.6:0/1372168884 conn(0x7f98e40d3d30 msgr2=0x7f98e40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3e00640 1 -- 172.21.15.6:0/1372168884 shutdown_connections 2024-03-06T07:36:34.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.104+0000 7f98e3e00640 1 -- 172.21.15.6:0/1372168884 wait complete. 2024-03-06T07:36:34.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.106+0000 7f98e3e00640 1 Processor -- start 2024-03-06T07:36:34.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.106+0000 7f98e3e00640 1 -- start start 2024-03-06T07:36:34.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98c8004d40 0x7f98e40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 0x7f98e415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e41f4c70 0x7f98e40b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e415e490 con 0x7f98e41f4530 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cfa20 con 0x7f98e41f4c70 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cfc60 con 0x7f98c8004d40 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.107+0000 7f98e2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 0x7f98e415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 0x7f98e415c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45454/0 (socket says 172.21.15.6:45454) 2024-03-06T07:36:34.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 -- 172.21.15.6:0/2935050394 learned_addr learned my addr 172.21.15.6:0/2935050394 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:34.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e3400640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e41f4c70 0x7f98e40b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2a00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98c8004d40 0x7f98e40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 -- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98c8004d40 msgr2=0x7f98e40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98c8004d40 0x7f98e40cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 -- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e41f4c70 msgr2=0x7f98e40b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e41f4c70 0x7f98e40b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 -- 172.21.15.6:0/2935050394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40cb3e0 con 0x7f98e41f4530 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.108+0000 7f98e2000640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 0x7f98e415c2c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00e890 tx=0x7f98cc00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.109+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00c870 con 0x7f98e41f4530 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.109+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc018070 con 0x7f98e41f4530 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.109+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00cc70 con 0x7f98e41f4530 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.109+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e40cb6c0 con 0x7f98e41f4530 2024-03-06T07:36:34.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.109+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e40cbc80 con 0x7f98e41f4530 2024-03-06T07:36:34.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.111+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e404f330 con 0x7f98e41f4530 2024-03-06T07:36:34.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.111+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98cc01d460 con 0x7f98e41f4530 2024-03-06T07:36:34.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.112+0000 7f98daa00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98c006b500 0x7f98c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:34.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.112+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f98cc08a260 con 0x7f98e41f4530 2024-03-06T07:36:34.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.113+0000 7f98e2a00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98c006b500 0x7f98c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:34.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.114+0000 7f98e2a00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98c006b500 0x7f98c006d9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:34.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.117+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98cc058ff0 con 0x7f98e41f4530 2024-03-06T07:36:34.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.538+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98e40b2370 con 0x7f98e41f4530 2024-03-06T07:36:34.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.539+0000 7f98daa00640 1 -- 172.21.15.6:0/2935050394 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f98cc0588f0 con 0x7f98e41f4530 2024-03-06T07:36:34.542 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:34.542 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:34.542 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98c006b500 msgr2=0x7f98c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98c006b500 0x7f98c006d9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4009040 comp rx=0 tx=0).stop 2024-03-06T07:36:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 msgr2=0x7f98e415c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 0x7f98e415c2c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00e890 tx=0x7f98cc00ed50 comp rx=0 tx=0).stop 2024-03-06T07:36:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 shutdown_connections 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f98c006b500 0x7f98c006d9c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98e41f4c70 0x7f98e40b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98e41f4530 0x7f98e415c2c0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 --2- 172.21.15.6:0/2935050394 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f98c8004d40 0x7f98e40cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 >> 172.21.15.6:0/2935050394 conn(0x7f98e40d3d30 msgr2=0x7f98e40b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 shutdown_connections 2024-03-06T07:36:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:34.543+0000 7f98e3e00640 1 -- 172.21.15.6:0/2935050394 wait complete. 2024-03-06T07:36:34.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:34.763 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:35.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:36.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.041+0000 7f7f2be00640 1 Processor -- start 2024-03-06T07:36:36.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.042+0000 7f7f2be00640 1 -- start start 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.043+0000 7f7f2be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7f2c1552b0 0x7f7f2c15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.043+0000 7f7f2be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 0x7f7f2c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.043+0000 7f7f2be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2c15ed10 con 0x7f7f2c0cce90 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f2be00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2c15eff0 con 0x7f7f2c1552b0 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f2be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c15f2d0 con 0x7f7f2c15e810 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f2b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7f2c0cce90 0x7f7f2c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43526/0 (socket says 172.21.15.6:43526) 2024-03-06T07:36:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f2b400640 1 -- 172.21.15.6:0/1869284343 learned_addr learned my addr 172.21.15.6:0/1869284343 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:36.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f2aa00640 1 --2- 172.21.15.6:0/1869284343 >> v2:172.21.15.178:3300/0 conn(0x7f7f2c1552b0 0x7f7f2c15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f2a000640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 0x7f7f2c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086678688 0 0) 0x7f7f2c15ed10 con 0x7f7f2c0cce90 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.044+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f10003520 con 0x7f7f2c0cce90 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067559244 0 0) 0x7f7f10003520 con 0x7f7f2c0cce90 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 msgr2=0x7f7f2c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 0x7f7f2c0cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 >> v2:172.21.15.178:3300/0 conn(0x7f7f2c1552b0 msgr2=0x7f7f2c15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 --2- 172.21.15.6:0/1869284343 >> v2:172.21.15.178:3300/0 conn(0x7f7f2c1552b0 0x7f7f2c15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c15f5b0 con 0x7f7f2c0cce90 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7f24001670 con 0x7f7f2c0cce90 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 >> v1:172.21.15.6:6789/0 conn(0x7f7f2c0cce90 legacy=0x7f7f2c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f100039b0 0x7f7f10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 0x7f7f10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10004d40 0x7f7f100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c15eff0 con 0x7f7f10004d40 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c15f2d0 con 0x7f7f100039b0 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1000dbd0 con 0x7f7f2c15e810 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10004d40 0x7f7f100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.045+0000 7f7f2a000640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 0x7f7f10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.046+0000 7f7f2aa00640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f100039b0 0x7f7f10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.046+0000 7f7f2b400640 1 -- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 msgr2=0x7f7f10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.046+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c15e810 0x7f7f10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.046+0000 7f7f2b400640 1 -- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f100039b0 msgr2=0x7f7f10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.046+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f100039b0 0x7f7f10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.046+0000 7f7f2b400640 1 -- 172.21.15.6:0/1869284343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10003520 con 0x7f7f10004d40 2024-03-06T07:36:36.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.047+0000 7f7f2b400640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10004d40 0x7f7f100051c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f7f24007fd0 tx=0x7f7f10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235e512f7ccf6dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.047+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24007d50 con 0x7f7f10004d40 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.047+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f24030070 con 0x7f7f10004d40 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.047+0000 7f7f29600640 1 -- 172.21.15.6:0/1869284343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2402cd40 con 0x7f7f10004d40 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2be00640 1 -- 172.21.15.6:0/1869284343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10004d40 msgr2=0x7f7f100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2be00640 1 --2- 172.21.15.6:0/1869284343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f10004d40 0x7f7f100051c0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f7f24007fd0 tx=0x7f7f10005830 comp rx=0 tx=0).stop 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2b400640 1 -- 172.21.15.6:0/1869284343 reap_dead start 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2be00640 1 -- 172.21.15.6:0/1869284343 shutdown_connections 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2be00640 1 -- 172.21.15.6:0/1869284343 >> 172.21.15.6:0/1869284343 conn(0x7f7f2c06b3d0 msgr2=0x7f7f2c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2be00640 1 -- 172.21.15.6:0/1869284343 shutdown_connections 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.048+0000 7f7f2be00640 1 -- 172.21.15.6:0/1869284343 wait complete. 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.050+0000 7f7f2be00640 1 Processor -- start 2024-03-06T07:36:36.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.050+0000 7f7f2be00640 1 -- start start 2024-03-06T07:36:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.051+0000 7f7f2be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f10004d40 0x7f7f2c1599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.051+0000 7f7f2be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 0x7f7f2c1533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.051+0000 7f7f2be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c140010 0x7f7f2c140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.051+0000 7f7f2be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c06dd30 con 0x7f7f2c152f40 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.051+0000 7f7f2be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c0b3760 con 0x7f7f10004d40 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.051+0000 7f7f2be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c06deb0 con 0x7f7f2c140010 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 0x7f7f2c1533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 0x7f7f2c1533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45472/0 (socket says 172.21.15.6:45472) 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 -- 172.21.15.6:0/64981546 learned_addr learned my addr 172.21.15.6:0/64981546 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2aa00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f10004d40 0x7f7f2c1599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 -- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c140010 msgr2=0x7f7f2c140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c140010 0x7f7f2c140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 -- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f10004d40 msgr2=0x7f7f2c1599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f10004d40 0x7f7f2c1599c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 -- 172.21.15.6:0/64981546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c1409d0 con 0x7f7f2c152f40 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2aa00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f10004d40 0x7f7f2c1599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:36.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.052+0000 7f7f2a000640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 0x7f7f2c1533c0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f7f18007f30 tx=0x7f7f18011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.053+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1801a020 con 0x7f7f2c152f40 2024-03-06T07:36:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.053+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f18016470 con 0x7f7f2c152f40 2024-03-06T07:36:36.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.053+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18016860 con 0x7f7f2c152f40 2024-03-06T07:36:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.053+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c140c90 con 0x7f7f2c152f40 2024-03-06T07:36:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.053+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c141250 con 0x7f7f2c152f40 2024-03-06T07:36:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.054+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c04f330 con 0x7f7f2c152f40 2024-03-06T07:36:36.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.055+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f18005d10 con 0x7f7f2c152f40 2024-03-06T07:36:36.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.056+0000 7f7f22a00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:36.058 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.056+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f1808b000 con 0x7f7f2c152f40 2024-03-06T07:36:36.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.057+0000 7f7f2aa00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:36.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.058+0000 7f7f2aa00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c1f2f30 tx=0x7f7f1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:36.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.061+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7f18024030 con 0x7f7f2c152f40 2024-03-06T07:36:36.087 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:36.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.301+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f2c0cf4a0 con 0x7f7f2c152f40 2024-03-06T07:36:36.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.302+0000 7f7f22a00640 1 -- 172.21.15.6:0/64981546 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f7f18059ed0 con 0x7f7f2c152f40 2024-03-06T07:36:36.304 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:36.304 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:36.305 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7f0c06b500 msgr2=0x7f7f0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c1f2f30 tx=0x7f7f1c008040 comp rx=0 tx=0).stop 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 msgr2=0x7f7f2c1533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 0x7f7f2c1533c0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f7f18007f30 tx=0x7f7f18011660 comp rx=0 tx=0).stop 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 shutdown_connections 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7f0c06b500 0x7f7f0c06d9c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7f2c140010 0x7f7f2c140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7f2c152f40 0x7f7f2c1533c0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 --2- 172.21.15.6:0/64981546 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f10004d40 0x7f7f2c1599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:36.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.305+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 >> 172.21.15.6:0/64981546 conn(0x7f7f2c06b3d0 msgr2=0x7f7f2c071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:36.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.306+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 shutdown_connections 2024-03-06T07:36:36.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:36.306+0000 7f7f2be00640 1 -- 172.21.15.6:0/64981546 wait complete. 2024-03-06T07:36:36.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:36.536 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:37.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.763+0000 7f4ee7200640 1 Processor -- start 2024-03-06T07:36:37.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.763+0000 7f4ee7200640 1 -- start start 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.766+0000 7f4ee7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4ee8070f10 0x7f4ee80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.766+0000 7f4ee7200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ee80cce90 0x7f4ee815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.766+0000 7f4ee7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee80714d0 con 0x7f4ee8145300 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.766+0000 7f4ee7200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee80cd300 con 0x7f4ee80cce90 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.766+0000 7f4ee7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee80717b0 con 0x7f4ee8070f10 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ee80cce90 0x7f4ee815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ee80cce90 0x7f4ee815c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45476/0 (socket says 172.21.15.6:45476) 2024-03-06T07:36:37.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 -- 172.21.15.6:0/1055749476 learned_addr learned my addr 172.21.15.6:0/1055749476 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee5e00640 1 --1- 172.21.15.6:0/1055749476 >> v1:172.21.15.6:6789/0 conn(0x7f4ee8145300 0x7f4ee80b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43538/0 (socket says 172.21.15.6:43538) 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4edd400640 1 --2- 172.21.15.6:0/1055749476 >> v2:172.21.15.178:3300/0 conn(0x7f4ee8070f10 0x7f4ee80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 -- 172.21.15.6:0/1055749476 >> v2:172.21.15.178:3300/0 conn(0x7f4ee8070f10 msgr2=0x7f4ee80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 --2- 172.21.15.6:0/1055749476 >> v2:172.21.15.178:3300/0 conn(0x7f4ee8070f10 0x7f4ee80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 -- 172.21.15.6:0/1055749476 >> v1:172.21.15.6:6789/0 conn(0x7f4ee8145300 legacy=0x7f4ee80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.767+0000 7f4ee6800640 1 -- 172.21.15.6:0/1055749476 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8071a90 con 0x7f4ee80cce90 2024-03-06T07:36:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.768+0000 7f4ee6800640 1 --2- 172.21.15.6:0/1055749476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee815c570 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80b5150 tx=0x7f4ee002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50eff47e2fa17aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee5400640 1 -- 172.21.15.6:0/1055749476 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee002ca80 con 0x7f4ee80cce90 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee5400640 1 -- 172.21.15.6:0/1055749476 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee002cc20 con 0x7f4ee80cce90 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee5400640 1 -- 172.21.15.6:0/1055749476 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00343f0 con 0x7f4ee80cce90 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee7200640 1 -- 172.21.15.6:0/1055749476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 msgr2=0x7f4ee815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee7200640 1 --2- 172.21.15.6:0/1055749476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee815c570 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80b5150 tx=0x7f4ee002bee0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee7200640 1 -- 172.21.15.6:0/1055749476 shutdown_connections 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee7200640 1 --2- 172.21.15.6:0/1055749476 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee815c570 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee7200640 1 --2- 172.21.15.6:0/1055749476 >> v2:172.21.15.178:3300/0 conn(0x7f4ee8070f10 0x7f4ee80cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.769+0000 7f4ee7200640 1 -- 172.21.15.6:0/1055749476 >> 172.21.15.6:0/1055749476 conn(0x7f4ee806b3d0 msgr2=0x7f4ee806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.770+0000 7f4ee7200640 1 -- 172.21.15.6:0/1055749476 shutdown_connections 2024-03-06T07:36:37.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.770+0000 7f4ee7200640 1 -- 172.21.15.6:0/1055749476 wait complete. 2024-03-06T07:36:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.771+0000 7f4ee7200640 1 Processor -- start 2024-03-06T07:36:37.773 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.771+0000 7f4ee7200640 1 -- start start 2024-03-06T07:36:37.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.772+0000 7f4ee7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ee8070f10 0x7f4ee813b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee813bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ee8145300 0x7f4ee813c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee815cab0 con 0x7f4ee80cce90 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee81456e0 con 0x7f4ee8145300 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee815cc30 con 0x7f4ee8070f10 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee813bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45482/0 (socket says 172.21.15.6:45482) 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 -- 172.21.15.6:0/3609877588 learned_addr learned my addr 172.21.15.6:0/3609877588 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee5e00640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ee8070f10 0x7f4ee813b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4ee6800640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ee8145300 0x7f4ee813c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 -- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ee8070f10 msgr2=0x7f4ee813b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ee8070f10 0x7f4ee813b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 -- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ee8145300 msgr2=0x7f4ee813c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ee8145300 0x7f4ee813c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.773+0000 7f4edd400640 1 -- 172.21.15.6:0/3609877588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee813ec10 con 0x7f4ee80cce90 2024-03-06T07:36:37.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.774+0000 7f4edd400640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee813bd10 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000d800 tx=0x7f4ed000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.774+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000b6c0 con 0x7f4ee80cce90 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.774+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0019070 con 0x7f4ee80cce90 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.774+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee813eef0 con 0x7f4ee80cce90 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.774+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed000ba80 con 0x7f4ee80cce90 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.774+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee806ccf0 con 0x7f4ee80cce90 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.775+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee80cc950 con 0x7f4ee80cce90 2024-03-06T07:36:37.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.776+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ed000bc20 con 0x7f4ee80cce90 2024-03-06T07:36:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.777+0000 7f4edea00640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.777+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ed008a7a0 con 0x7f4ee80cce90 2024-03-06T07:36:37.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.778+0000 7f4ee5e00640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:37.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.779+0000 7f4ee5e00640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4ec406b4d0 0x7f4ec406d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80b1180 tx=0x7f4ed8007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:37.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:37.782+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ee80cc950 con 0x7f4ee80cce90 2024-03-06T07:36:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.023+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ee8067830 con 0x7f4ee80cce90 2024-03-06T07:36:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.024+0000 7f4edea00640 1 -- 172.21.15.6:0/3609877588 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f4ee8067830 con 0x7f4ee80cce90 2024-03-06T07:36:38.027 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:38.027 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:38.027 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4ec406b4d0 msgr2=0x7f4ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4ec406b4d0 0x7f4ec406d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80b1180 tx=0x7f4ed8007ce0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 msgr2=0x7f4ee813bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee813bd10 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f4ed000d800 tx=0x7f4ed000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 shutdown_connections 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f4ec406b4d0 0x7f4ec406d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4ee8145300 0x7f4ee813c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ee80cce90 0x7f4ee813bd10 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 --2- 172.21.15.6:0/3609877588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4ee8070f10 0x7f4ee813b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.027+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 >> 172.21.15.6:0/3609877588 conn(0x7f4ee806b3d0 msgr2=0x7f4ee80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.028+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 shutdown_connections 2024-03-06T07:36:38.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:38.028+0000 7f4ee7200640 1 -- 172.21.15.6:0/3609877588 wait complete. 2024-03-06T07:36:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:38.251 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:38.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:39.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.473+0000 7efe89c00640 1 Processor -- start 2024-03-06T07:36:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.473+0000 7efe89c00640 1 -- start start 2024-03-06T07:36:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe89c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7efe840cfce0 0x7efe8406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe89c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8415c5e0 con 0x7efe84152bc0 2024-03-06T07:36:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe89c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8415c8c0 con 0x7efe8406c500 2024-03-06T07:36:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe89c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe840d0120 con 0x7efe840cfce0 2024-03-06T07:36:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe82a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7efe84152bc0 0x7efe840af940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43548/0 (socket says 172.21.15.6:43548) 2024-03-06T07:36:39.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe82a00640 1 -- 172.21.15.6:0/869000319 learned_addr learned my addr 172.21.15.6:0/869000319 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:39.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888620642 0 0) 0x7efe8415c5e0 con 0x7efe84152bc0 2024-03-06T07:36:39.479 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.475+0000 7efe7be00640 1 --2- 172.21.15.6:0/869000319 >> v2:172.21.15.178:3300/0 conn(0x7efe840cfce0 0x7efe8406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe68003520 con 0x7efe84152bc0 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983536812 0 0) 0x7efe68003520 con 0x7efe84152bc0 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 >> v2:172.21.15.178:3300/0 conn(0x7efe840cfce0 msgr2=0x7efe8406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 --2- 172.21.15.6:0/869000319 >> v2:172.21.15.178:3300/0 conn(0x7efe840cfce0 0x7efe8406bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 >> v1:172.21.15.178:6789/0 conn(0x7efe8406c500 legacy=0x7efe840ae5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8415cba0 con 0x7efe84152bc0 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7efe74001670 con 0x7efe84152bc0 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 >> v1:172.21.15.6:6789/0 conn(0x7efe84152bc0 legacy=0x7efe840af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe68003a70 0x7efe68003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe68004800 0x7efe68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe680052e0 0x7efe6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840d0120 con 0x7efe68004800 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6800c1e0 con 0x7efe680052e0 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6800c3e0 con 0x7efe68003a70 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.476+0000 7efe7be00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe68004800 0x7efe68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe82a00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe68003a70 0x7efe68003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe83400640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe680052e0 0x7efe6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe7be00640 1 -- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe68003a70 msgr2=0x7efe68003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe7be00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe68003a70 0x7efe68003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe7be00640 1 -- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe680052e0 msgr2=0x7efe6800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe7be00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe680052e0 0x7efe6800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe7be00640 1 -- 172.21.15.6:0/869000319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68003520 con 0x7efe68004800 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe82a00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe68003a70 0x7efe68003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe83400640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe680052e0 0x7efe6800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:39.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.477+0000 7efe83400640 1 -- 172.21.15.6:0/869000319 reap_dead start 2024-03-06T07:36:39.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.478+0000 7efe7be00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe68004800 0x7efe68004c80 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00ba00 tx=0x7efe68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3777f59a6b5e194 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.478+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c036020 con 0x7efe68004800 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.478+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c032620 con 0x7efe68004800 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.478+0000 7efe82000640 1 -- 172.21.15.6:0/869000319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c00b600 con 0x7efe68004800 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 -- 172.21.15.6:0/869000319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe68004800 msgr2=0x7efe68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe68004800 0x7efe68004c80 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00ba00 tx=0x7efe68005830 comp rx=0 tx=0).stop 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 -- 172.21.15.6:0/869000319 shutdown_connections 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 --2- 172.21.15.6:0/869000319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe68004800 0x7efe68004c80 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 -- 172.21.15.6:0/869000319 >> 172.21.15.6:0/869000319 conn(0x7efe840714d0 msgr2=0x7efe840718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 -- 172.21.15.6:0/869000319 shutdown_connections 2024-03-06T07:36:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.479+0000 7efe89c00640 1 -- 172.21.15.6:0/869000319 wait complete. 2024-03-06T07:36:39.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.481+0000 7efe89c00640 1 Processor -- start 2024-03-06T07:36:39.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.481+0000 7efe89c00640 1 -- start start 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.482+0000 7efe89c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe68004800 0x7efe841f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.482+0000 7efe89c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 0x7efe840b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.482+0000 7efe89c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe84150d60 0x7efe841511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.482+0000 7efe89c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8415cf10 con 0x7efe84159540 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.482+0000 7efe89c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8406c970 con 0x7efe68004800 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.482+0000 7efe89c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe84152fa0 con 0x7efe84150d60 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 0x7efe840b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 0x7efe840b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45512/0 (socket says 172.21.15.6:45512) 2024-03-06T07:36:39.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 -- 172.21.15.6:0/531620415 learned_addr learned my addr 172.21.15.6:0/531620415 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:39.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe82a00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe68004800 0x7efe841f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe83400640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe84150d60 0x7efe841511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 -- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe84150d60 msgr2=0x7efe841511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe84150d60 0x7efe841511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 -- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe68004800 msgr2=0x7efe841f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe68004800 0x7efe841f7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 -- 172.21.15.6:0/531620415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84151720 con 0x7efe84159540 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.483+0000 7efe7be00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 0x7efe840b4b80 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7efe6c02bf70 tx=0x7efe6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.484+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c036040 con 0x7efe84159540 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.484+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c047680 con 0x7efe84159540 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.484+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c047a50 con 0x7efe84159540 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.484+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe841f7db0 con 0x7efe84159540 2024-03-06T07:36:39.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.484+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe841f8050 con 0x7efe84159540 2024-03-06T07:36:39.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.485+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe84072100 con 0x7efe84159540 2024-03-06T07:36:39.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.486+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe6c039020 con 0x7efe84159540 2024-03-06T07:36:39.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.487+0000 7efe7aa00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efe6006b4d0 0x7efe6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.488+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efe6c0acb10 con 0x7efe84159540 2024-03-06T07:36:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.488+0000 7efe82a00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efe6006b4d0 0x7efe6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:39.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.489+0000 7efe82a00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efe6006b4d0 0x7efe6006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efe74001eb0 tx=0x7efe74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:39.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.493+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efe6c07b8b0 con 0x7efe84159540 2024-03-06T07:36:39.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.837+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe8404f380 con 0x7efe84159540 2024-03-06T07:36:39.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.838+0000 7efe7aa00640 1 -- 172.21.15.6:0/531620415 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7efe6c045e40 con 0x7efe84159540 2024-03-06T07:36:39.840 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:39.840 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:39.841 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:39.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efe6006b4d0 msgr2=0x7efe6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efe6006b4d0 0x7efe6006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efe74001eb0 tx=0x7efe74009040 comp rx=0 tx=0).stop 2024-03-06T07:36:39.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 msgr2=0x7efe840b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:39.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 0x7efe840b4b80 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7efe6c02bf70 tx=0x7efe6c004730 comp rx=0 tx=0).stop 2024-03-06T07:36:39.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 shutdown_connections 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efe6006b4d0 0x7efe6006d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efe84150d60 0x7efe841511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efe84159540 0x7efe840b4b80 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 --2- 172.21.15.6:0/531620415 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efe68004800 0x7efe841f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 >> 172.21.15.6:0/531620415 conn(0x7efe840714d0 msgr2=0x7efe840cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 shutdown_connections 2024-03-06T07:36:39.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:39.841+0000 7efe89c00640 1 -- 172.21.15.6:0/531620415 wait complete. 2024-03-06T07:36:40.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:40.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:40.091 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:41.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:41.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.298+0000 7fc56ce00640 1 Processor -- start 2024-03-06T07:36:41.300 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.298+0000 7fc56ce00640 1 -- start start 2024-03-06T07:36:41.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.299+0000 7fc56ce00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc56815f450 0x7fc568070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.302 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.299+0000 7fc56ce00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc5680700c0 0x7fc5680cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.299+0000 7fc56ce00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc56806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.299+0000 7fc56ce00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5680d29e0 con 0x7fc56815f450 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.299+0000 7fc56ce00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5680d0710 con 0x7fc5680700c0 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.299+0000 7fc56ce00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680cd0c0 con 0x7fc5680d02e0 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc56815f450 0x7fc568070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc56815f450 0x7fc568070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45522/0 (socket says 172.21.15.6:45522) 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 -- 172.21.15.6:0/3652249184 learned_addr learned my addr 172.21.15.6:0/3652249184 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc55fe00640 1 --2- 172.21.15.6:0/3652249184 >> v2:172.21.15.178:3300/0 conn(0x7fc5680700c0 0x7fc5680cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567e00640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc56806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 -- 172.21.15.6:0/3652249184 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 msgr2=0x7fc56806ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc56806ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 -- 172.21.15.6:0/3652249184 >> v2:172.21.15.178:3300/0 conn(0x7fc5680700c0 msgr2=0x7fc5680cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 --2- 172.21.15.6:0/3652249184 >> v2:172.21.15.178:3300/0 conn(0x7fc5680700c0 0x7fc5680cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.303 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567400640 1 -- 172.21.15.6:0/3652249184 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680cd3a0 con 0x7fc56815f450 2024-03-06T07:36:41.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc55fe00640 1 --2- 172.21.15.6:0/3652249184 >> v2:172.21.15.178:3300/0 conn(0x7fc5680700c0 0x7fc5680cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:41.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.300+0000 7fc567e00640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc56806ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.302+0000 7fc567400640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc56815f450 0x7fc568070990 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc568071fc0 tx=0x7fc55802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6927bf7dbc7c46e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.302+0000 7fc566a00640 1 -- 172.21.15.6:0/3652249184 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558035020 con 0x7fc56815f450 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.302+0000 7fc566a00640 1 -- 172.21.15.6:0/3652249184 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc558031880 con 0x7fc56815f450 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.302+0000 7fc566a00640 1 -- 172.21.15.6:0/3652249184 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558031c50 con 0x7fc56815f450 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 -- 172.21.15.6:0/3652249184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc56815f450 msgr2=0x7fc568070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc56815f450 0x7fc568070990 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fc568071fc0 tx=0x7fc55802b920 comp rx=0 tx=0).stop 2024-03-06T07:36:41.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 -- 172.21.15.6:0/3652249184 shutdown_connections 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc56806ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 --2- 172.21.15.6:0/3652249184 >> v2:172.21.15.178:3300/0 conn(0x7fc5680700c0 0x7fc5680cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 --2- 172.21.15.6:0/3652249184 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc56815f450 0x7fc568070990 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 -- 172.21.15.6:0/3652249184 >> 172.21.15.6:0/3652249184 conn(0x7fc5680d1ae0 msgr2=0x7fc5680d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 -- 172.21.15.6:0/3652249184 shutdown_connections 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.303+0000 7fc56ce00640 1 -- 172.21.15.6:0/3652249184 wait complete. 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.304+0000 7fc56ce00640 1 Processor -- start 2024-03-06T07:36:41.306 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.305+0000 7fc56ce00640 1 -- start start 2024-03-06T07:36:41.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.305+0000 7fc56ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 0x7fc5680c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.305+0000 7fc56ce00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc568148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.305+0000 7fc56ce00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc56815f450 0x7fc5680c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc56ce00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680ae8b0 con 0x7fc5680700c0 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc56ce00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5680cd960 con 0x7fc56815f450 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc56ce00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56815f830 con 0x7fc5680d02e0 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 0x7fc5680c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 0x7fc5680c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45538/0 (socket says 172.21.15.6:45538) 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 -- 172.21.15.6:0/2081900280 learned_addr learned my addr 172.21.15.6:0/2081900280 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567e00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc56815f450 0x7fc5680c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc55fe00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc568148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 -- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 msgr2=0x7fc568148e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc568148e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 -- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc56815f450 msgr2=0x7fc5680c72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc56815f450 0x7fc5680c72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567400640 1 -- 172.21.15.6:0/2081900280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680c84c0 con 0x7fc5680700c0 2024-03-06T07:36:41.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc55fe00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc568148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:41.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.306+0000 7fc567e00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc56815f450 0x7fc5680c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:41.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.307+0000 7fc567400640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 0x7fc5680c7f80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fc558000c00 tx=0x7fc5580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.307+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558040070 con 0x7fc5680700c0 2024-03-06T07:36:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.307+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55803e070 con 0x7fc5680700c0 2024-03-06T07:36:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.307+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558035040 con 0x7fc5680700c0 2024-03-06T07:36:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.307+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56806dd10 con 0x7fc5680700c0 2024-03-06T07:36:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.307+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5680c6ab0 con 0x7fc5680700c0 2024-03-06T07:36:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.308+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56804f330 con 0x7fc5680700c0 2024-03-06T07:36:41.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.309+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc558038020 con 0x7fc5680700c0 2024-03-06T07:36:41.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.310+0000 7fc55f400640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc54406b4d0 0x7fc54406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:41.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.310+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc5580ac2f0 con 0x7fc5680700c0 2024-03-06T07:36:41.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.311+0000 7fc55fe00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc54406b4d0 0x7fc54406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:41.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.312+0000 7fc55fe00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc54406b4d0 0x7fc54406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc5680740d0 tx=0x7fc550008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:41.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.319+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc5680c84c0 con 0x7fc5680700c0 2024-03-06T07:36:41.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.732+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc56815ec70 con 0x7fc5680700c0 2024-03-06T07:36:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.733+0000 7fc55f400640 1 -- 172.21.15.6:0/2081900280 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fc5580443c0 con 0x7fc5680700c0 2024-03-06T07:36:41.735 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:41.735 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:41.735 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.735+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc54406b4d0 msgr2=0x7fc54406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.735+0000 7fc56ce00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc54406b4d0 0x7fc54406d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc5680740d0 tx=0x7fc550008040 comp rx=0 tx=0).stop 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.735+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 msgr2=0x7fc5680c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.735+0000 7fc56ce00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 0x7fc5680c7f80 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fc558000c00 tx=0x7fc5580046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 shutdown_connections 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc54406b4d0 0x7fc54406d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc56815f450 0x7fc5680c72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc5680d02e0 0x7fc568148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 --2- 172.21.15.6:0/2081900280 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc5680700c0 0x7fc5680c7f80 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 >> 172.21.15.6:0/2081900280 conn(0x7fc5680d1ae0 msgr2=0x7fc5681553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 shutdown_connections 2024-03-06T07:36:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:41.736+0000 7fc56ce00640 1 -- 172.21.15.6:0/2081900280 wait complete. 2024-03-06T07:36:41.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:41.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:42.094 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:42.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:43.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.184+0000 7f662ea00640 1 Processor -- start 2024-03-06T07:36:43.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.184+0000 7f662ea00640 1 -- start start 2024-03-06T07:36:43.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ea00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f662815a550 0x7f66280aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f662815a930 0x7f66280ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ea00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6628071640 con 0x7f66280ce9f0 2024-03-06T07:36:43.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ea00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6628071920 con 0x7f662815a550 2024-03-06T07:36:43.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6628071c00 con 0x7f662815a930 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f6627e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f662815a550 0x7f66280aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f66280ce9f0 0x7f66280b32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43598/0 (socket says 172.21.15.6:43598) 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.187+0000 7f662ca00640 1 -- 172.21.15.6:0/264911846 learned_addr learned my addr 172.21.15.6:0/264911846 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.188+0000 7f661f400640 1 --2- 172.21.15.6:0/264911846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f662815a930 0x7f66280ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.188+0000 7f6627e00640 1 -- 172.21.15.6:0/264911846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f662815a930 msgr2=0x7f66280ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.188+0000 7f6627e00640 1 --2- 172.21.15.6:0/264911846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f662815a930 0x7f66280ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.188+0000 7f6627e00640 1 -- 172.21.15.6:0/264911846 >> v1:172.21.15.6:6789/0 conn(0x7f66280ce9f0 legacy=0x7f66280b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.188+0000 7f6627e00640 1 -- 172.21.15.6:0/264911846 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6628071ee0 con 0x7f662815a550 2024-03-06T07:36:43.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.189+0000 7f6627e00640 1 --2- 172.21.15.6:0/264911846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a550 0x7f66280aefe0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f66180097a0 tx=0x7f661802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc07f7a42e0a353d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.189+0000 7f6627400640 1 -- 172.21.15.6:0/264911846 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618034070 con 0x7f662815a550 2024-03-06T07:36:43.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.189+0000 7f6627400640 1 -- 172.21.15.6:0/264911846 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6618030800 con 0x7f662815a550 2024-03-06T07:36:43.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.189+0000 7f6627400640 1 -- 172.21.15.6:0/264911846 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618030bd0 con 0x7f662815a550 2024-03-06T07:36:43.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 -- 172.21.15.6:0/264911846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a550 msgr2=0x7f66280aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 --2- 172.21.15.6:0/264911846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a550 0x7f66280aefe0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f66180097a0 tx=0x7f661802b940 comp rx=0 tx=0).stop 2024-03-06T07:36:43.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 -- 172.21.15.6:0/264911846 shutdown_connections 2024-03-06T07:36:43.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 --2- 172.21.15.6:0/264911846 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f662815a930 0x7f66280ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 --2- 172.21.15.6:0/264911846 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a550 0x7f66280aefe0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 -- 172.21.15.6:0/264911846 >> 172.21.15.6:0/264911846 conn(0x7f662806afb0 msgr2=0x7f662806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:43.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 -- 172.21.15.6:0/264911846 shutdown_connections 2024-03-06T07:36:43.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.190+0000 7f662ea00640 1 -- 172.21.15.6:0/264911846 wait complete. 2024-03-06T07:36:43.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.192+0000 7f662ea00640 1 Processor -- start 2024-03-06T07:36:43.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.192+0000 7f662ea00640 1 -- start start 2024-03-06T07:36:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ea00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66280ce9f0 0x7f66281506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ea00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f662815a550 0x7f662809b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 0x7f662809d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ea00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66280b52b0 con 0x7f662815a930 2024-03-06T07:36:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ea00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66280b5130 con 0x7f662815a550 2024-03-06T07:36:43.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ea00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66280ade50 con 0x7f66280ce9f0 2024-03-06T07:36:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.193+0000 7f662ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 0x7f662809d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 0x7f662809d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36948/0 (socket says 172.21.15.6:36948) 2024-03-06T07:36:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 -- 172.21.15.6:0/3150608537 learned_addr learned my addr 172.21.15.6:0/3150608537 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f6627e00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66280ce9f0 0x7f66281506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f661f400640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f662815a550 0x7f662809b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 -- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66280ce9f0 msgr2=0x7f66281506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66280ce9f0 0x7f66281506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 -- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f662815a550 msgr2=0x7f662809b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f662815a550 0x7f662809b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 -- 172.21.15.6:0/3150608537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662809ac60 con 0x7f662815a930 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f662ca00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 0x7f662809d1f0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f662000c450 tx=0x7f662000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.194+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620017070 con 0x7f662815a930 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.195+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6620005ce0 con 0x7f662815a930 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.195+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620007270 con 0x7f662815a930 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.195+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662809af40 con 0x7f662815a930 2024-03-06T07:36:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.195+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662809d730 con 0x7f662815a930 2024-03-06T07:36:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.196+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6628069510 con 0x7f662815a930 2024-03-06T07:36:43.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.197+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f662001a070 con 0x7f662815a930 2024-03-06T07:36:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.198+0000 7f6624c00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f660406b4d0 0x7f660406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.198+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6620089440 con 0x7f662815a930 2024-03-06T07:36:43.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.198+0000 7f6627e00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f660406b4d0 0x7f660406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:43.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.200+0000 7f6627e00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f660406b4d0 0x7f660406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6618036000 tx=0x7f661803b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:43.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.203+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6620057c20 con 0x7f662815a930 2024-03-06T07:36:43.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.442+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f662804f3a0 con 0x7f662815a930 2024-03-06T07:36:43.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.442+0000 7f6624c00640 1 -- 172.21.15.6:0/3150608537 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f662005bcc0 con 0x7f662815a930 2024-03-06T07:36:43.445 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:43.445 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:43.445 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:43.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f660406b4d0 msgr2=0x7f660406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f660406b4d0 0x7f660406d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6618036000 tx=0x7f661803b040 comp rx=0 tx=0).stop 2024-03-06T07:36:43.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 msgr2=0x7f662809d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:43.448 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 0x7f662809d1f0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f662000c450 tx=0x7f662000c910 comp rx=0 tx=0).stop 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 shutdown_connections 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f660406b4d0 0x7f660406d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f662815a550 0x7f662809b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f662815a930 0x7f662809d1f0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 --2- 172.21.15.6:0/3150608537 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66280ce9f0 0x7f66281506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 >> 172.21.15.6:0/3150608537 conn(0x7f662806afb0 msgr2=0x7f66280cdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 shutdown_connections 2024-03-06T07:36:43.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:43.446+0000 7f662ea00640 1 -- 172.21.15.6:0/3150608537 wait complete. 2024-03-06T07:36:43.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:43.670 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:44.096 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:44.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:44.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.903+0000 7fd894a00640 1 Processor -- start 2024-03-06T07:36:44.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.904+0000 7fd894a00640 1 -- start start 2024-03-06T07:36:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.905+0000 7fd894a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd890145300 0x7fd8900b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.905+0000 7fd894a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd890070f10 0x7fd8900cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.905+0000 7fd894a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd8900cce90 0x7fd8900cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.905+0000 7fd894a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8900cd300 con 0x7fd890145300 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.905+0000 7fd894a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8900714d0 con 0x7fd8900cce90 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.905+0000 7fd894a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900717b0 con 0x7fd890070f10 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd890145300 0x7fd8900b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd890145300 0x7fd8900b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36954/0 (socket says 172.21.15.6:36954) 2024-03-06T07:36:44.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 -- 172.21.15.6:0/3851713910 learned_addr learned my addr 172.21.15.6:0/3851713910 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd887e00640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd890070f10 0x7fd8900cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88f200640 1 --2- 172.21.15.6:0/3851713910 >> v2:172.21.15.178:3300/0 conn(0x7fd8900cce90 0x7fd8900cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 -- 172.21.15.6:0/3851713910 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd890070f10 msgr2=0x7fd8900cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd890070f10 0x7fd8900cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 -- 172.21.15.6:0/3851713910 >> v2:172.21.15.178:3300/0 conn(0x7fd8900cce90 msgr2=0x7fd8900cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 --2- 172.21.15.6:0/3851713910 >> v2:172.21.15.178:3300/0 conn(0x7fd8900cce90 0x7fd8900cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88e800640 1 -- 172.21.15.6:0/3851713910 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890071a90 con 0x7fd890145300 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd88f200640 1 --2- 172.21.15.6:0/3851713910 >> v2:172.21.15.178:3300/0 conn(0x7fd8900cce90 0x7fd8900cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:44.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.906+0000 7fd887e00640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd890070f10 0x7fd8900cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:44.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.907+0000 7fd88e800640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890145300 0x7fd8900b49a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd890072420 tx=0x7fd88002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4020282afddc6fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd88de00640 1 -- 172.21.15.6:0/3851713910 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880035020 con 0x7fd890145300 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd88de00640 1 -- 172.21.15.6:0/3851713910 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd880031870 con 0x7fd890145300 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd88de00640 1 -- 172.21.15.6:0/3851713910 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880031c40 con 0x7fd890145300 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 -- 172.21.15.6:0/3851713910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890145300 msgr2=0x7fd8900b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890145300 0x7fd8900b49a0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd890072420 tx=0x7fd88002b910 comp rx=0 tx=0).stop 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 -- 172.21.15.6:0/3851713910 shutdown_connections 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 --2- 172.21.15.6:0/3851713910 >> v2:172.21.15.178:3300/0 conn(0x7fd8900cce90 0x7fd8900cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd890070f10 0x7fd8900cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 --2- 172.21.15.6:0/3851713910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890145300 0x7fd8900b49a0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.908+0000 7fd894a00640 1 -- 172.21.15.6:0/3851713910 >> 172.21.15.6:0/3851713910 conn(0x7fd89006b3d0 msgr2=0x7fd89006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.909+0000 7fd894a00640 1 -- 172.21.15.6:0/3851713910 shutdown_connections 2024-03-06T07:36:44.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.909+0000 7fd894a00640 1 -- 172.21.15.6:0/3851713910 wait complete. 2024-03-06T07:36:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.910+0000 7fd894a00640 1 Processor -- start 2024-03-06T07:36:44.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.910+0000 7fd894a00640 1 -- start start 2024-03-06T07:36:44.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.911+0000 7fd894a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 0x7fd89013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.911+0000 7fd894a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8900cce90 0x7fd89013c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.911+0000 7fd894a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd890145300 0x7fd89013c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.911+0000 7fd894a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901456e0 con 0x7fd890070f10 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.911+0000 7fd894a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900b4ee0 con 0x7fd890145300 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.911+0000 7fd894a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8900cde70 con 0x7fd8900cce90 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 0x7fd89013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 0x7fd89013d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36958/0 (socket says 172.21.15.6:36958) 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 -- 172.21.15.6:0/4179475475 learned_addr learned my addr 172.21.15.6:0/4179475475 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd887e00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8900cce90 0x7fd89013c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88f200640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd890145300 0x7fd89013c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 -- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8900cce90 msgr2=0x7fd89013c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8900cce90 0x7fd89013c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 -- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd890145300 msgr2=0x7fd89013c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:44.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd890145300 0x7fd89013c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:44.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88e800640 1 -- 172.21.15.6:0/4179475475 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd890141710 con 0x7fd890070f10 2024-03-06T07:36:44.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.912+0000 7fd88f200640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd890145300 0x7fd89013c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:44.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.913+0000 7fd88e800640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 0x7fd89013d3e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd88002b950 tx=0x7fd8800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.913+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880040070 con 0x7fd890070f10 2024-03-06T07:36:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.913+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88003a420 con 0x7fd890070f10 2024-03-06T07:36:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.913+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880035040 con 0x7fd890070f10 2024-03-06T07:36:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.913+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89014c0d0 con 0x7fd890070f10 2024-03-06T07:36:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.913+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8901413a0 con 0x7fd890070f10 2024-03-06T07:36:44.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.915+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89004f330 con 0x7fd890070f10 2024-03-06T07:36:44.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.915+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd88004b060 con 0x7fd890070f10 2024-03-06T07:36:44.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.916+0000 7fd886a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86c06b4d0 0x7fd86c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:44.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.916+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd8800ac140 con 0x7fd890070f10 2024-03-06T07:36:44.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.917+0000 7fd887e00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86c06b4d0 0x7fd86c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:44.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.918+0000 7fd887e00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86c06b4d0 0x7fd86c06d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd8900721e0 tx=0x7fd878008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:44.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:44.921+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd88007af70 con 0x7fd890070f10 2024-03-06T07:36:45.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.160+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd890144a20 con 0x7fd890070f10 2024-03-06T07:36:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.161+0000 7fd886a00640 1 -- 172.21.15.6:0/4179475475 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fd88007a870 con 0x7fd890070f10 2024-03-06T07:36:45.164 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:45.164 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:45.164 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:45.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86c06b4d0 msgr2=0x7fd86c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86c06b4d0 0x7fd86c06d990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd8900721e0 tx=0x7fd878008040 comp rx=0 tx=0).stop 2024-03-06T07:36:45.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 msgr2=0x7fd89013d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 0x7fd89013d3e0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fd88002b950 tx=0x7fd8800046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 shutdown_connections 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd86c06b4d0 0x7fd86c06d990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd890145300 0x7fd89013c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd8900cce90 0x7fd89013c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 --2- 172.21.15.6:0/4179475475 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd890070f10 0x7fd89013d3e0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 >> 172.21.15.6:0/4179475475 conn(0x7fd89006b3d0 msgr2=0x7fd89015ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 shutdown_connections 2024-03-06T07:36:45.167 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:45.164+0000 7fd894a00640 1 -- 172.21.15.6:0/4179475475 wait complete. 2024-03-06T07:36:45.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:45.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:46.098 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:46.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:46.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.614+0000 7f0708a00640 1 Processor -- start 2024-03-06T07:36:46.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.615+0000 7f0708a00640 1 -- start start 2024-03-06T07:36:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.617+0000 7f0708a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f070415a550 0x7f07040aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0708a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f07040cebc0 0x7f07040ace40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0708a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0704071640 con 0x7f070415a930 2024-03-06T07:36:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0708a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0704071920 con 0x7f07040cebc0 2024-03-06T07:36:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0708a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0704071c00 con 0x7f070415a550 2024-03-06T07:36:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0701400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f070415a930 0x7f07040ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43604/0 (socket says 172.21.15.6:43604) 2024-03-06T07:36:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0701400640 1 -- 172.21.15.6:0/402546353 learned_addr learned my addr 172.21.15.6:0/402546353 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0702800640 1 --2- 172.21.15.6:0/402546353 >> v2:172.21.15.6:3300/0 conn(0x7f07040cebc0 0x7f07040ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.618+0000 7f0701e00640 1 --2- 172.21.15.6:0/402546353 >> v2:172.21.15.178:3300/0 conn(0x7f070415a550 0x7f07040aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.619+0000 7f0700a00640 1 -- 172.21.15.6:0/402546353 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264406301 0 0) 0x7f0704071640 con 0x7f070415a930 2024-03-06T07:36:46.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.619+0000 7f0700a00640 1 -- 172.21.15.6:0/402546353 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e8005730 con 0x7f070415a930 2024-03-06T07:36:46.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.619+0000 7f0702800640 1 -- 172.21.15.6:0/402546353 >> v2:172.21.15.178:3300/0 conn(0x7f070415a550 msgr2=0x7f07040aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:46.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.619+0000 7f0702800640 1 --2- 172.21.15.6:0/402546353 >> v2:172.21.15.178:3300/0 conn(0x7f070415a550 0x7f07040aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.619+0000 7f0702800640 1 -- 172.21.15.6:0/402546353 >> v1:172.21.15.6:6789/0 conn(0x7f070415a930 legacy=0x7f07040ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.619+0000 7f0702800640 1 -- 172.21.15.6:0/402546353 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704071ee0 con 0x7f07040cebc0 2024-03-06T07:36:46.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.620+0000 7f0702800640 1 --2- 172.21.15.6:0/402546353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07040cebc0 0x7f07040ace40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f0704074960 tx=0x7f06fc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c502404f19925e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.620+0000 7f0700a00640 1 -- 172.21.15.6:0/402546353 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc039070 con 0x7f07040cebc0 2024-03-06T07:36:46.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.620+0000 7f0700a00640 1 -- 172.21.15.6:0/402546353 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc02cd00 con 0x7f07040cebc0 2024-03-06T07:36:46.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.620+0000 7f0700a00640 1 -- 172.21.15.6:0/402546353 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc034640 con 0x7f07040cebc0 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.621+0000 7f0708a00640 1 -- 172.21.15.6:0/402546353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07040cebc0 msgr2=0x7f07040ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.621+0000 7f0708a00640 1 --2- 172.21.15.6:0/402546353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07040cebc0 0x7f07040ace40 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f0704074960 tx=0x7f06fc02bf50 comp rx=0 tx=0).stop 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.621+0000 7f0708a00640 1 -- 172.21.15.6:0/402546353 shutdown_connections 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.621+0000 7f0708a00640 1 --2- 172.21.15.6:0/402546353 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f07040cebc0 0x7f07040ace40 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.621+0000 7f0708a00640 1 --2- 172.21.15.6:0/402546353 >> v2:172.21.15.178:3300/0 conn(0x7f070415a550 0x7f07040aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.621+0000 7f0708a00640 1 -- 172.21.15.6:0/402546353 >> 172.21.15.6:0/402546353 conn(0x7f070406afb0 msgr2=0x7f070406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.622+0000 7f0708a00640 1 -- 172.21.15.6:0/402546353 shutdown_connections 2024-03-06T07:36:46.624 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.622+0000 7f0708a00640 1 -- 172.21.15.6:0/402546353 wait complete. 2024-03-06T07:36:46.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.624+0000 7f0708a00640 1 Processor -- start 2024-03-06T07:36:46.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.624+0000 7f0708a00640 1 -- start start 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.625+0000 7f0708a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07040cebc0 0x7f07041536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.625+0000 7f0708a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 0x7f070406cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0708a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f070415a930 0x7f070406d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0708a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040b52b0 con 0x7f070415a550 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0708a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040b5130 con 0x7f07040cebc0 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0708a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040cdc20 con 0x7f070415a930 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 0x7f070406cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 0x7f070406cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36980/0 (socket says 172.21.15.6:36980) 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 -- 172.21.15.6:0/2623259625 learned_addr learned my addr 172.21.15.6:0/2623259625 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:46.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0702800640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f070415a930 0x7f070406d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 -- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f070415a930 msgr2=0x7f070406d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f070415a930 0x7f070406d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701e00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07040cebc0 0x7f07041536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 -- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07040cebc0 msgr2=0x7f07041536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07040cebc0 0x7f07041536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.626+0000 7f0701400640 1 -- 172.21.15.6:0/2623259625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040ad120 con 0x7f070415a550 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.627+0000 7f0701400640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 0x7f070406cb10 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c390 tx=0x7f06f000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.627+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f0017070 con 0x7f070415a550 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.627+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f0005ce0 con 0x7f070415a550 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.627+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00072f0 con 0x7f070415a550 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.627+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07040ad400 con 0x7f070415a550 2024-03-06T07:36:46.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.627+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07040ad9c0 con 0x7f070415a550 2024-03-06T07:36:46.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.628+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07040adcf0 con 0x7f070415a550 2024-03-06T07:36:46.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.629+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06f001a070 con 0x7f070415a550 2024-03-06T07:36:46.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.631+0000 7f06fa000640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06e406b4d0 0x7f06e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:46.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.631+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f06f0089640 con 0x7f070415a550 2024-03-06T07:36:46.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.631+0000 7f0701e00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06e406b4d0 0x7f06e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:46.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.633+0000 7f0701e00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06e406b4d0 0x7f06e406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009a10 tx=0x7f06f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:46.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.636+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06f0058480 con 0x7f070415a550 2024-03-06T07:36:46.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.875+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0704069510 con 0x7f070415a550 2024-03-06T07:36:46.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.876+0000 7f06fa000640 1 -- 172.21.15.6:0/2623259625 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f06f0057d80 con 0x7f070415a550 2024-03-06T07:36:46.878 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:46.878 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:46.878 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:46.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06e406b4d0 msgr2=0x7f06e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06e406b4d0 0x7f06e406d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f06f4009a10 tx=0x7f06f4008040 comp rx=0 tx=0).stop 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 msgr2=0x7f070406cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 0x7f070406cb10 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f06f000c390 tx=0x7f06f000c850 comp rx=0 tx=0).stop 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 shutdown_connections 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f06e406b4d0 0x7f06e406d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f070415a930 0x7f070406d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f070415a550 0x7f070406cb10 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 --2- 172.21.15.6:0/2623259625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f07040cebc0 0x7f07041536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 >> 172.21.15.6:0/2623259625 conn(0x7f070406afb0 msgr2=0x7f070406d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 shutdown_connections 2024-03-06T07:36:46.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:46.878+0000 7f0708a00640 1 -- 172.21.15.6:0/2623259625 wait complete. 2024-03-06T07:36:47.100 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:47.100 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:48.100 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:48.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:48.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.323+0000 7f4474c00640 1 Processor -- start 2024-03-06T07:36:48.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.323+0000 7f4474c00640 1 -- start start 2024-03-06T07:36:48.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.325+0000 7f4474c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f447015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.325+0000 7f4474c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4470161ae0 con 0x7f4470147400 2024-03-06T07:36:48.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.325+0000 7f4474c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447015f080 con 0x7f447006b6c0 2024-03-06T07:36:48.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.325+0000 7f4474c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4470162110 con 0x7f44701625a0 2024-03-06T07:36:48.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.325+0000 7f446f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4470147400 0x7f447006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43608/0 (socket says 172.21.15.6:43608) 2024-03-06T07:36:48.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.325+0000 7f446f400640 1 -- 172.21.15.6:0/2199819903 learned_addr learned my addr 172.21.15.6:0/2199819903 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446fe00640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f447015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163323309 0 0) 0x7f4470161ae0 con 0x7f4470147400 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4454003520 con 0x7f4470147400 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3995703863 0 0) 0x7f4454003520 con 0x7f4470147400 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 msgr2=0x7f447015ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f447015ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 >> v1:172.21.15.178:6789/0 conn(0x7f447006b6c0 legacy=0x7f447015e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470161da0 con 0x7f4470147400 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f4460001670 con 0x7f4470147400 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 >> v1:172.21.15.6:6789/0 conn(0x7f4470147400 legacy=0x7f447006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f4454003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f4454004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.326+0000 7f446e000640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4454004d40 0x7f44540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4470162110 con 0x7f44540039e0 2024-03-06T07:36:48.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445400ba70 con 0x7f4454004d40 2024-03-06T07:36:48.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445400bc70 con 0x7f44701625a0 2024-03-06T07:36:48.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446f400640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f4454003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446f400640 1 -- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 msgr2=0x7f4454004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446ea00640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f4454004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446fe00640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4454004d40 0x7f44540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446f400640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f4454004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446f400640 1 -- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4454004d40 msgr2=0x7f44540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446f400640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4454004d40 0x7f44540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446f400640 1 -- 172.21.15.6:0/2199819903 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454003520 con 0x7f44540039e0 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446fe00640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4454004d40 0x7f44540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.327+0000 7f446ea00640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f44701625a0 0x7f4454004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.328+0000 7f446f400640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f4454003e00 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f446000c210 tx=0x7f4454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85bcfcaa5e169eb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460007cf0 con 0x7f44540039e0 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460030070 con 0x7f44540039e0 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f446e000640 1 -- 172.21.15.6:0/2199819903 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460005d30 con 0x7f44540039e0 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f4474c00640 1 -- 172.21.15.6:0/2199819903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 msgr2=0x7f4454003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f4474c00640 1 --2- 172.21.15.6:0/2199819903 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f4454003e00 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f446000c210 tx=0x7f4454005830 comp rx=0 tx=0).stop 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f446fe00640 1 -- 172.21.15.6:0/2199819903 reap_dead start 2024-03-06T07:36:48.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f4474c00640 1 -- 172.21.15.6:0/2199819903 shutdown_connections 2024-03-06T07:36:48.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f4474c00640 1 -- 172.21.15.6:0/2199819903 >> 172.21.15.6:0/2199819903 conn(0x7f447006c0c0 msgr2=0x7f447006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:48.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.329+0000 7f4474c00640 1 -- 172.21.15.6:0/2199819903 shutdown_connections 2024-03-06T07:36:48.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.330+0000 7f4474c00640 1 -- 172.21.15.6:0/2199819903 wait complete. 2024-03-06T07:36:48.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.331+0000 7f4474c00640 1 Processor -- start 2024-03-06T07:36:48.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.331+0000 7f4474c00640 1 -- start start 2024-03-06T07:36:48.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f4474c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f447015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f4474c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4470138590 0x7f4470134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f4474c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44700cf450 0x7f4470144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f4474c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447015ec00 con 0x7f44540039e0 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f4474c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447015ea80 con 0x7f44700cf450 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f4474c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f447015ed80 con 0x7f4470138590 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.332+0000 7f446f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f447015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f447015dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37000/0 (socket says 172.21.15.6:37000) 2024-03-06T07:36:48.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 -- 172.21.15.6:0/2814375634 learned_addr learned my addr 172.21.15.6:0/2814375634 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 -- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4470138590 msgr2=0x7f4470134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4470138590 0x7f4470134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 -- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44700cf450 msgr2=0x7f4470144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44700cf450 0x7f4470144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 -- 172.21.15.6:0/2814375634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4470135160 con 0x7f44540039e0 2024-03-06T07:36:48.336 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.333+0000 7f446f400640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f447015dba0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f446000c330 tx=0x7f4470161f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.334+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460007cf0 con 0x7f44540039e0 2024-03-06T07:36:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.334+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460004bf0 con 0x7f44540039e0 2024-03-06T07:36:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.334+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460001c30 con 0x7f44540039e0 2024-03-06T07:36:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.334+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4470135440 con 0x7f44540039e0 2024-03-06T07:36:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.334+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4470136060 con 0x7f44540039e0 2024-03-06T07:36:48.337 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.335+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4470135960 con 0x7f44540039e0 2024-03-06T07:36:48.338 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.336+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4460047020 con 0x7f44540039e0 2024-03-06T07:36:48.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.337+0000 7f4467400640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f445006b4d0 0x7f445006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:48.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.337+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f44600ab300 con 0x7f44540039e0 2024-03-06T07:36:48.339 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.338+0000 7f446ea00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f445006b4d0 0x7f445006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:48.340 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.339+0000 7f446ea00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f445006b4d0 0x7f445006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4470142a50 tx=0x7f445c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:48.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.346+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4470135960 con 0x7f44540039e0 2024-03-06T07:36:48.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.592+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f447004f330 con 0x7f44540039e0 2024-03-06T07:36:48.595 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.593+0000 7f4467400640 1 -- 172.21.15.6:0/2814375634 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f446007a130 con 0x7f44540039e0 2024-03-06T07:36:48.595 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:48.595 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:48.595 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.595+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f445006b4d0 msgr2=0x7f445006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.595+0000 7f4474c00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f445006b4d0 0x7f445006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4470142a50 tx=0x7f445c009040 comp rx=0 tx=0).stop 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.595+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 msgr2=0x7f447015dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.595+0000 7f4474c00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f447015dba0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f446000c330 tx=0x7f4470161f70 comp rx=0 tx=0).stop 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 shutdown_connections 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f445006b4d0 0x7f445006d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f44700cf450 0x7f4470144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4470138590 0x7f4470134a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 --2- 172.21.15.6:0/2814375634 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44540039e0 0x7f447015dba0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 >> 172.21.15.6:0/2814375634 conn(0x7f447006c0c0 msgr2=0x7f4470072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:48.598 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 shutdown_connections 2024-03-06T07:36:48.599 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:48.596+0000 7f4474c00640 1 -- 172.21.15.6:0/2814375634 wait complete. 2024-03-06T07:36:48.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:48.819 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:49.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:50.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.046+0000 7fb071c00640 1 Processor -- start 2024-03-06T07:36:50.048 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.046+0000 7fb071c00640 1 -- start start 2024-03-06T07:36:50.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb071c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb06c15e0b0 0x7fb06c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb071c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 0x7fb06c0b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb071c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c0b3710 con 0x7fb06c0d4a10 2024-03-06T07:36:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb071c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0b09e0 con 0x7fb06c15e0b0 2024-03-06T07:36:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb071c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0b39f0 con 0x7fb06c0d0e50 2024-03-06T07:36:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb06b400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb06c0d4a10 0x7fb06c0d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:43612/0 (socket says 172.21.15.6:43612) 2024-03-06T07:36:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb06b400640 1 -- 172.21.15.6:0/2214033430 learned_addr learned my addr 172.21.15.6:0/2214033430 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.048+0000 7fb06be00640 1 --2- 172.21.15.6:0/2214033430 >> v2:172.21.15.178:3300/0 conn(0x7fb06c15e0b0 0x7fb06c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb071200640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 0x7fb06c0b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664743002 0 0) 0x7fb06c0b3710 con 0x7fb06c0d4a10 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb050003520 con 0x7fb06c0d4a10 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1213544653 0 0) 0x7fb050003520 con 0x7fb06c0d4a10 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 msgr2=0x7fb06c0b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 0x7fb06c0b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 >> v2:172.21.15.178:3300/0 conn(0x7fb06c15e0b0 msgr2=0x7fb06c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 --2- 172.21.15.6:0/2214033430 >> v2:172.21.15.178:3300/0 conn(0x7fb06c15e0b0 0x7fb06c0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c0b3cd0 con 0x7fb06c0d4a10 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.049+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fb058001690 con 0x7fb06c0d4a10 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 >> v1:172.21.15.6:6789/0 conn(0x7fb06c0d4a10 legacy=0x7fb06c0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0500039b0 0x7fb050003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 0x7fb050004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb050004d40 0x7fb0500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0b39f0 con 0x7fb0500039b0 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05000b9c0 con 0x7fb050004d40 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05000bb40 con 0x7fb06c0d0e50 2024-03-06T07:36:50.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06be00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0500039b0 0x7fb050003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb071200640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb050004d40 0x7fb0500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06be00640 1 -- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 msgr2=0x7fb050004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06be00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb06c0d0e50 0x7fb050004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06be00640 1 -- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb050004d40 msgr2=0x7fb0500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:50.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06be00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb050004d40 0x7fb0500051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.050+0000 7fb06be00640 1 -- 172.21.15.6:0/2214033430 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb050003520 con 0x7fb0500039b0 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb06be00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0500039b0 0x7fb050003dd0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009af0 tx=0x7fb050005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5a3f2e746d7cea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c039070 con 0x7fb0500039b0 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05c02c710 con 0x7fb0500039b0 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb06aa00640 1 -- 172.21.15.6:0/2214033430 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c009690 con 0x7fb0500039b0 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb071c00640 1 -- 172.21.15.6:0/2214033430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0500039b0 msgr2=0x7fb050003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb071c00640 1 --2- 172.21.15.6:0/2214033430 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb0500039b0 0x7fb050003dd0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009af0 tx=0x7fb050005830 comp rx=0 tx=0).stop 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.052+0000 7fb071200640 1 -- 172.21.15.6:0/2214033430 reap_dead start 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.053+0000 7fb071c00640 1 -- 172.21.15.6:0/2214033430 shutdown_connections 2024-03-06T07:36:50.055 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.053+0000 7fb071c00640 1 -- 172.21.15.6:0/2214033430 >> 172.21.15.6:0/2214033430 conn(0x7fb06c0d3d30 msgr2=0x7fb06c0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.053+0000 7fb071c00640 1 -- 172.21.15.6:0/2214033430 shutdown_connections 2024-03-06T07:36:50.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.053+0000 7fb071c00640 1 -- 172.21.15.6:0/2214033430 wait complete. 2024-03-06T07:36:50.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.054+0000 7fb071c00640 1 Processor -- start 2024-03-06T07:36:50.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.055+0000 7fb071c00640 1 -- start start 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0500039b0 0x7fb06c1f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb06c153250 0x7fb06c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 0x7fb06c1601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0cfc60 con 0x7fb06c15fd10 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c15e490 con 0x7fb06c153250 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0cf850 con 0x7fb0500039b0 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 0x7fb06c1601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 0x7fb06c1601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37018/0 (socket says 172.21.15.6:37018) 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071200640 1 -- 172.21.15.6:0/102608461 learned_addr learned my addr 172.21.15.6:0/102608461 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb06be00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0500039b0 0x7fb06c1f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb06b400640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb06c153250 0x7fb06c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.056+0000 7fb071200640 1 -- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0500039b0 msgr2=0x7fb06c1f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071200640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0500039b0 0x7fb06c1f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071200640 1 -- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb06c153250 msgr2=0x7fb06c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071200640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb06c153250 0x7fb06c15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071200640 1 -- 172.21.15.6:0/102608461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c1536b0 con 0x7fb06c15fd10 2024-03-06T07:36:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071200640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 0x7fb06c1601a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0d3c80 tx=0x7fb06400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06400fe90 con 0x7fb06c15fd10 2024-03-06T07:36:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb064003990 con 0x7fb06c15fd10 2024-03-06T07:36:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064003d60 con 0x7fb06c15fd10 2024-03-06T07:36:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06c0cd1c0 con 0x7fb06c15fd10 2024-03-06T07:36:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.057+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c0cb210 con 0x7fb06c15fd10 2024-03-06T07:36:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.058+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c04f330 con 0x7fb06c15fd10 2024-03-06T07:36:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.059+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb064005d10 con 0x7fb06c15fd10 2024-03-06T07:36:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.060+0000 7fb063e00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04c06b670 0x7fb04c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.060+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb06408a550 con 0x7fb06c15fd10 2024-03-06T07:36:50.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.061+0000 7fb06be00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04c06b670 0x7fb04c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:50.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.062+0000 7fb06be00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04c06b670 0x7fb04c06db30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb05c036000 tx=0x7fb05c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:50.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.066+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb064012070 con 0x7fb06c15fd10 2024-03-06T07:36:50.101 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:50.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.358+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06c0d2a50 con 0x7fb06c15fd10 2024-03-06T07:36:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.359+0000 7fb063e00640 1 -- 172.21.15.6:0/102608461 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fb064059380 con 0x7fb06c15fd10 2024-03-06T07:36:50.361 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:50.361 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:50.361 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.361+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04c06b670 msgr2=0x7fb04c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.361+0000 7fb071c00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04c06b670 0x7fb04c06db30 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb05c036000 tx=0x7fb05c030000 comp rx=0 tx=0).stop 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.361+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 msgr2=0x7fb06c1601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.361+0000 7fb071c00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 0x7fb06c1601a0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0d3c80 tx=0x7fb06400f4a0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 shutdown_connections 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb04c06b670 0x7fb04c06db30 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb06c153250 0x7fb06c15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb06c15fd10 0x7fb06c1601a0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 --2- 172.21.15.6:0/102608461 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb0500039b0 0x7fb06c1f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 >> 172.21.15.6:0/102608461 conn(0x7fb06c0d3d30 msgr2=0x7fb06c0b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 shutdown_connections 2024-03-06T07:36:50.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:50.362+0000 7fb071c00640 1 -- 172.21.15.6:0/102608461 wait complete. 2024-03-06T07:36:50.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:50.582 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:51.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.817+0000 7fb1b6c00640 1 Processor -- start 2024-03-06T07:36:51.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.818+0000 7fb1b6c00640 1 -- start start 2024-03-06T07:36:51.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.819+0000 7fb1b6c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af1a0 0x7fb1b015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.819+0000 7fb1b6c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb1b00cdc20 0x7fb1b00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.819+0000 7fb1b6c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b015cc40 con 0x7fb1b00af580 2024-03-06T07:36:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.819+0000 7fb1b6c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1b015cf20 con 0x7fb1b00cdc20 2024-03-06T07:36:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.819+0000 7fb1b6c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b015d200 con 0x7fb1b00af1a0 2024-03-06T07:36:51.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.820+0000 7fb1afe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af1a0 0x7fb1b015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.820+0000 7fb1b4c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb1b00cdc20 0x7fb1b00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.820+0000 7fb1b4c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb1b00cdc20 0x7fb1b00b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:57812/0 (socket says 172.21.15.6:57812) 2024-03-06T07:36:51.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.820+0000 7fb1b4c00640 1 -- 172.21.15.6:0/3167425310 learned_addr learned my addr 172.21.15.6:0/3167425310 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:51.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.820+0000 7fb1a7e00640 1 --1- 172.21.15.6:0/3167425310 >> v1:172.21.15.178:6789/0 conn(0x7fb1b00af580 0x7fb1b00b4c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:34122/0 (socket says 172.21.15.6:34122) 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.821+0000 7fb1af400640 1 -- 172.21.15.6:0/3167425310 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471100946 0 0) 0x7fb1b015cc40 con 0x7fb1b00af580 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.821+0000 7fb1af400640 1 -- 172.21.15.6:0/3167425310 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb194003520 con 0x7fb1b00af580 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.821+0000 7fb1b4c00640 1 -- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af1a0 msgr2=0x7fb1b015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.821+0000 7fb1b4c00640 1 --2- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af1a0 0x7fb1b015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.821+0000 7fb1b4c00640 1 -- 172.21.15.6:0/3167425310 >> v1:172.21.15.178:6789/0 conn(0x7fb1b00af580 legacy=0x7fb1b00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.821+0000 7fb1b4c00640 1 -- 172.21.15.6:0/3167425310 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b015d4e0 con 0x7fb1b00cdc20 2024-03-06T07:36:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.823+0000 7fb1b4c00640 1 --2- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b00b2dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009e50 tx=0x7fb1a802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8277a020ed89498 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.823+0000 7fb1af400640 1 -- 172.21.15.6:0/3167425310 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a802ca80 con 0x7fb1b00cdc20 2024-03-06T07:36:51.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.823+0000 7fb1af400640 1 -- 172.21.15.6:0/3167425310 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a802cc20 con 0x7fb1b00cdc20 2024-03-06T07:36:51.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.823+0000 7fb1af400640 1 -- 172.21.15.6:0/3167425310 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a80343f0 con 0x7fb1b00cdc20 2024-03-06T07:36:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 -- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 msgr2=0x7fb1b00b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b00b2dd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009e50 tx=0x7fb1a802bee0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 -- 172.21.15.6:0/3167425310 shutdown_connections 2024-03-06T07:36:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b00b2dd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/3167425310 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af1a0 0x7fb1b015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 -- 172.21.15.6:0/3167425310 >> 172.21.15.6:0/3167425310 conn(0x7fb1b006afb0 msgr2=0x7fb1b006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:51.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 -- 172.21.15.6:0/3167425310 shutdown_connections 2024-03-06T07:36:51.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.824+0000 7fb1b6c00640 1 -- 172.21.15.6:0/3167425310 wait complete. 2024-03-06T07:36:51.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.826+0000 7fb1b6c00640 1 Processor -- start 2024-03-06T07:36:51.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.826+0000 7fb1b6c00640 1 -- start start 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.827+0000 7fb1b6c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 0x7fb1b00b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.827+0000 7fb1b6c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af580 0x7fb1b01f6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1b6c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b01f7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1b6c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b00b36a0 con 0x7fb1b00af1a0 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1b6c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b00b3520 con 0x7fb1b00cdc20 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1b6c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b00b3820 con 0x7fb1b00af580 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 0x7fb1b00b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 0x7fb1b00b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37034/0 (socket says 172.21.15.6:37034) 2024-03-06T07:36:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1afe00640 1 -- 172.21.15.6:0/1196520267 learned_addr learned my addr 172.21.15.6:0/1196520267 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1b4c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b01f7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.828+0000 7fb1a7e00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af580 0x7fb1b01f6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1afe00640 1 -- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af580 msgr2=0x7fb1b01f6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1afe00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af580 0x7fb1b01f6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1afe00640 1 -- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 msgr2=0x7fb1b01f7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1afe00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b01f7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1afe00640 1 -- 172.21.15.6:0/1196520267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b009ad10 con 0x7fb1b00af1a0 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1b4c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b01f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1a7e00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af580 0x7fb1b01f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.829+0000 7fb1afe00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 0x7fb1b00b0580 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000e9e0 tx=0x7fb1a000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.830+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0017070 con 0x7fb1b00af1a0 2024-03-06T07:36:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.830+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a000c9d0 con 0x7fb1b00af1a0 2024-03-06T07:36:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.830+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a000cdc0 con 0x7fb1b00af1a0 2024-03-06T07:36:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.830+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b01f7950 con 0x7fb1b00af1a0 2024-03-06T07:36:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.830+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b00afaa0 con 0x7fb1b00af1a0 2024-03-06T07:36:51.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.831+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b0069510 con 0x7fb1b00af1a0 2024-03-06T07:36:51.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.832+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1a001a070 con 0x7fb1b00af1a0 2024-03-06T07:36:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.833+0000 7fb1acc00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb18c06b520 0x7fb18c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.833+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1a008a530 con 0x7fb1b00af1a0 2024-03-06T07:36:51.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.834+0000 7fb1a7e00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb18c06b520 0x7fb18c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:51.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.835+0000 7fb1a7e00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb18c06b520 0x7fb18c06d9e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00ae1f0 tx=0x7fb198007b90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:51.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:51.838+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1a0059450 con 0x7fb1b00af1a0 2024-03-06T07:36:52.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.075+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1b004f3a0 con 0x7fb1b00af1a0 2024-03-06T07:36:52.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.076+0000 7fb1acc00640 1 -- 172.21.15.6:0/1196520267 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fb1a0058d50 con 0x7fb1b00af1a0 2024-03-06T07:36:52.078 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:52.078 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:52.079 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb18c06b520 msgr2=0x7fb18c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb18c06b520 0x7fb18c06d9e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00ae1f0 tx=0x7fb198007b90 comp rx=0 tx=0).stop 2024-03-06T07:36:52.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 msgr2=0x7fb1b00b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 0x7fb1b00b0580 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000e9e0 tx=0x7fb1a000eea0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 shutdown_connections 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb18c06b520 0x7fb18c06d9e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb1b00cdc20 0x7fb1b01f7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb1b00af580 0x7fb1b01f6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 --2- 172.21.15.6:0/1196520267 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb1b00af1a0 0x7fb1b00b0580 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 >> 172.21.15.6:0/1196520267 conn(0x7fb1b006afb0 msgr2=0x7fb1b00ae5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 shutdown_connections 2024-03-06T07:36:52.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:52.079+0000 7fb1b6c00640 1 -- 172.21.15.6:0/1196520267 wait complete. 2024-03-06T07:36:52.103 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:52.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:52.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:53.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:53.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.528+0000 7f7483a00640 1 Processor -- start 2024-03-06T07:36:53.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.529+0000 7f7483a00640 1 -- start start 2024-03-06T07:36:53.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.530+0000 7f7483a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f747c0b34e0 0x7f747c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.530+0000 7f7483a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c15cc40 con 0x7f747c0af580 2024-03-06T07:36:53.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.530+0000 7f7483a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c15cf20 con 0x7f747c0af1a0 2024-03-06T07:36:53.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.530+0000 7f7483a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c15d200 con 0x7f747c0b34e0 2024-03-06T07:36:53.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.531+0000 7f7473e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f747c0af580 0x7f747c0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55148/0 (socket says 172.21.15.6:55148) 2024-03-06T07:36:53.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.531+0000 7f7473e00640 1 -- 172.21.15.6:0/1175670413 learned_addr learned my addr 172.21.15.6:0/1175670413 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:53.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.531+0000 7f7481a00640 1 --2- 172.21.15.6:0/1175670413 >> v2:172.21.15.178:3300/0 conn(0x7f747c0b34e0 0x7f747c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.531+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909411872 0 0) 0x7f747c15cc40 con 0x7f747c0af580 2024-03-06T07:36:53.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.531+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c003520 con 0x7f747c0af580 2024-03-06T07:36:53.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539151251 0 0) 0x7f745c003520 con 0x7f747c0af580 2024-03-06T07:36:53.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 >> v2:172.21.15.178:3300/0 conn(0x7f747c0b34e0 msgr2=0x7f747c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 --2- 172.21.15.6:0/1175670413 >> v2:172.21.15.178:3300/0 conn(0x7f747c0b34e0 0x7f747c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 >> v1:172.21.15.178:6789/0 conn(0x7f747c0af1a0 legacy=0x7f747c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c15d4e0 con 0x7f747c0af580 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7464001670 con 0x7f747c0af580 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 >> v1:172.21.15.6:6789/0 conn(0x7f747c0af580 legacy=0x7f747c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c003a70 0x7f745c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c004800 0x7f745c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c0052e0 0x7f745c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c15d200 con 0x7f745c003a70 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745c00c1b0 con 0x7f745c004800 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745c00c3b0 con 0x7f745c0052e0 2024-03-06T07:36:53.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.532+0000 7f7481000640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c003a70 0x7f745c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7473e00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c004800 0x7f745c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481a00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c0052e0 0x7f745c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481000640 1 -- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c0052e0 msgr2=0x7f745c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481000640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f745c0052e0 0x7f745c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481000640 1 -- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c004800 msgr2=0x7f745c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481000640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c004800 0x7f745c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481000640 1 -- 172.21.15.6:0/1175670413 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745c003520 con 0x7f745c003a70 2024-03-06T07:36:53.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.533+0000 7f7481a00640 1 -- 172.21.15.6:0/1175670413 reap_dead start 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.534+0000 7f7481000640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c003a70 0x7f745c003e90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f746c001750 tx=0x7f745c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6faeff02d54ea70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.534+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c001370 con 0x7f745c003a70 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c0052d0 con 0x7f745c003a70 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f747be00640 1 -- 172.21.15.6:0/1175670413 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c003830 con 0x7f745c003a70 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 -- 172.21.15.6:0/1175670413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c003a70 msgr2=0x7f745c003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c003a70 0x7f745c003e90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f746c001750 tx=0x7f745c005830 comp rx=0 tx=0).stop 2024-03-06T07:36:53.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 -- 172.21.15.6:0/1175670413 shutdown_connections 2024-03-06T07:36:53.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 --2- 172.21.15.6:0/1175670413 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f745c003a70 0x7f745c003e90 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 -- 172.21.15.6:0/1175670413 >> 172.21.15.6:0/1175670413 conn(0x7f747c06afb0 msgr2=0x7f747c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:53.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 -- 172.21.15.6:0/1175670413 shutdown_connections 2024-03-06T07:36:53.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.535+0000 7f7483a00640 1 -- 172.21.15.6:0/1175670413 wait complete. 2024-03-06T07:36:53.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.537+0000 7f7483a00640 1 Processor -- start 2024-03-06T07:36:53.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.537+0000 7f7483a00640 1 -- start start 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7483a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c003a70 0x7f747c09d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7483a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f747c1581e0 0x7f747c158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7483a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 0x7f747c1f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7483a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c15c600 con 0x7f747c1f1900 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7483a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0cde30 con 0x7f745c003a70 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7483a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c15c780 con 0x7f747c1581e0 2024-03-06T07:36:53.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7481a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 0x7f747c1f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7481a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 0x7f747c1f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45706/0 (socket says 172.21.15.6:45706) 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.539+0000 7f7481a00640 1 -- 172.21.15.6:0/1881094637 learned_addr learned my addr 172.21.15.6:0/1881094637 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481000640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c003a70 0x7f747c09d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7473e00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f747c1581e0 0x7f747c158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481a00640 1 -- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f747c1581e0 msgr2=0x7f747c158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f747c1581e0 0x7f747c158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481a00640 1 -- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c003a70 msgr2=0x7f747c09d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c003a70 0x7f747c09d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481a00640 1 -- 172.21.15.6:0/1881094637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c1f5c50 con 0x7f747c1f1900 2024-03-06T07:36:53.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7481a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 0x7f747c1f1d80 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f747c0b3970 tx=0x7f74740115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.540+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474011e40 con 0x7f747c1f1900 2024-03-06T07:36:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.541+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747401b070 con 0x7f747c1f1900 2024-03-06T07:36:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.541+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474003990 con 0x7f747c1f1900 2024-03-06T07:36:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.541+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c1f2670 con 0x7f747c1f1900 2024-03-06T07:36:53.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.541+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c1f5170 con 0x7f747c1f1900 2024-03-06T07:36:53.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.542+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c069510 con 0x7f747c1f1900 2024-03-06T07:36:53.545 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.543+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7474005d10 con 0x7f747c1f1900 2024-03-06T07:36:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.545+0000 7f7479600640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f745806b4d0 0x7f745806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.545+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f747408a640 con 0x7f747c1f1900 2024-03-06T07:36:53.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.545+0000 7f7481000640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f745806b4d0 0x7f745806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:53.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.546+0000 7f7481000640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f745806b4d0 0x7f745806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f746c033000 tx=0x7f746c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:53.550 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.548+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f747401d310 con 0x7f747c1f1900 2024-03-06T07:36:53.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.791+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f747c04f3a0 con 0x7f747c1f1900 2024-03-06T07:36:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.792+0000 7f7479600640 1 -- 172.21.15.6:0/1881094637 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f7474059480 con 0x7f747c1f1900 2024-03-06T07:36:53.794 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:53.794 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:53.795 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:53.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f745806b4d0 msgr2=0x7f745806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f745806b4d0 0x7f745806d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f746c033000 tx=0x7f746c009030 comp rx=0 tx=0).stop 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 msgr2=0x7f747c1f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 0x7f747c1f1d80 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f747c0b3970 tx=0x7f74740115a0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 shutdown_connections 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f745806b4d0 0x7f745806d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f747c1f1900 0x7f747c1f1d80 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f747c1581e0 0x7f747c158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 --2- 172.21.15.6:0/1881094637 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f745c003a70 0x7f747c09d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 >> 172.21.15.6:0/1881094637 conn(0x7f747c06afb0 msgr2=0x7f747c15bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.795+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 shutdown_connections 2024-03-06T07:36:53.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:53.796+0000 7f7483a00640 1 -- 172.21.15.6:0/1881094637 wait complete. 2024-03-06T07:36:54.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:54.016 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:54.106 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:54.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:54.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:36:55.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.247+0000 7f09d9600640 1 Processor -- start 2024-03-06T07:36:55.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.248+0000 7f09d9600640 1 -- start start 2024-03-06T07:36:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d9600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f09d40add50 0x7f09d40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d9600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d406afb0 0x7f09d406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d9600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d40b5de0 con 0x7f09d4145790 2024-03-06T07:36:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d9600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d40d1b70 con 0x7f09d40add50 2024-03-06T07:36:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d9600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d409ef10 con 0x7f09d406afb0 2024-03-06T07:36:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f09d4145790 0x7f09d41603f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55150/0 (socket says 172.21.15.6:55150) 2024-03-06T07:36:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.249+0000 7f09d2a00640 1 -- 172.21.15.6:0/3678200505 learned_addr learned my addr 172.21.15.6:0/3678200505 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d3400640 1 --2- 172.21.15.6:0/3678200505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d406afb0 0x7f09d406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d2000640 1 --2- 172.21.15.6:0/3678200505 >> v2:172.21.15.178:3300/0 conn(0x7f09d40add50 0x7f09d40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d1600640 1 -- 172.21.15.6:0/3678200505 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769455372 0 0) 0x7f09d40b5de0 con 0x7f09d4145790 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d1600640 1 -- 172.21.15.6:0/3678200505 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8003520 con 0x7f09d4145790 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d1600640 1 -- 172.21.15.6:0/3678200505 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2038855583 0 0) 0x7f09b8003520 con 0x7f09d4145790 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d3400640 1 -- 172.21.15.6:0/3678200505 >> v2:172.21.15.178:3300/0 conn(0x7f09d40add50 msgr2=0x7f09d40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d3400640 1 --2- 172.21.15.6:0/3678200505 >> v2:172.21.15.178:3300/0 conn(0x7f09d40add50 0x7f09d40d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d3400640 1 -- 172.21.15.6:0/3678200505 >> v1:172.21.15.6:6789/0 conn(0x7f09d4145790 legacy=0x7f09d41603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.250+0000 7f09d3400640 1 -- 172.21.15.6:0/3678200505 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40b6080 con 0x7f09d406afb0 2024-03-06T07:36:55.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.252+0000 7f09d3400640 1 --2- 172.21.15.6:0/3678200505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d406afb0 0x7f09d406b460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0097a0 tx=0x7f09cc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fed0df490aa6ff43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.252+0000 7f09d1600640 1 -- 172.21.15.6:0/3678200505 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc037070 con 0x7f09d406afb0 2024-03-06T07:36:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.252+0000 7f09d1600640 1 -- 172.21.15.6:0/3678200505 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc030510 con 0x7f09d406afb0 2024-03-06T07:36:55.254 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.252+0000 7f09d1600640 1 -- 172.21.15.6:0/3678200505 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c40019f0 con 0x7f09d406afb0 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 -- 172.21.15.6:0/3678200505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d406afb0 msgr2=0x7f09d406b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 --2- 172.21.15.6:0/3678200505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d406afb0 0x7f09d406b460 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0097a0 tx=0x7f09cc02b790 comp rx=0 tx=0).stop 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 -- 172.21.15.6:0/3678200505 shutdown_connections 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 --2- 172.21.15.6:0/3678200505 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d406afb0 0x7f09d406b460 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 --2- 172.21.15.6:0/3678200505 >> v2:172.21.15.178:3300/0 conn(0x7f09d40add50 0x7f09d40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 -- 172.21.15.6:0/3678200505 >> 172.21.15.6:0/3678200505 conn(0x7f09d406c2a0 msgr2=0x7f09d406c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 -- 172.21.15.6:0/3678200505 shutdown_connections 2024-03-06T07:36:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.253+0000 7f09d9600640 1 -- 172.21.15.6:0/3678200505 wait complete. 2024-03-06T07:36:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.256+0000 7f09d9600640 1 Processor -- start 2024-03-06T07:36:55.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.256+0000 7f09d9600640 1 -- start start 2024-03-06T07:36:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.257+0000 7f09d9600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d406afb0 0x7f09d409e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.257+0000 7f09d9600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d40add50 0x7f09d413f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d9600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 0x7f09d413f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d9600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d4145b70 con 0x7f09d4145790 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d9600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d415fed0 con 0x7f09d406afb0 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d9600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d40af720 con 0x7f09d40add50 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 0x7f09d413f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 0x7f09d413f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45720/0 (socket says 172.21.15.6:45720) 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 -- 172.21.15.6:0/787697829 learned_addr learned my addr 172.21.15.6:0/787697829 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d2000640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d40add50 0x7f09d413f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d2a00640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d406afb0 0x7f09d409e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 -- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d40add50 msgr2=0x7f09d413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d40add50 0x7f09d413f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 -- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d406afb0 msgr2=0x7f09d409e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d406afb0 0x7f09d409e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.261 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.258+0000 7f09d3400640 1 -- 172.21.15.6:0/787697829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d413e420 con 0x7f09d4145790 2024-03-06T07:36:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.259+0000 7f09d3400640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 0x7f09d413f5b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0097a0 tx=0x7f09cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.259+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc041020 con 0x7f09d4145790 2024-03-06T07:36:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.259+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09cc03c870 con 0x7f09d4145790 2024-03-06T07:36:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.259+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc037040 con 0x7f09d4145790 2024-03-06T07:36:55.262 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.259+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d413fc10 con 0x7f09d4145790 2024-03-06T07:36:55.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.259+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d406da90 con 0x7f09d4145790 2024-03-06T07:36:55.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.261+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d40af300 con 0x7f09d4145790 2024-03-06T07:36:55.263 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.262+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09cc044020 con 0x7f09d4145790 2024-03-06T07:36:55.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.263+0000 7f09caa00640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b406b4d0 0x7f09b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:55.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.263+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f09cc0abc00 con 0x7f09d4145790 2024-03-06T07:36:55.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.264+0000 7f09d2a00640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b406b4d0 0x7f09b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:55.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.265+0000 7f09d2a00640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b406b4d0 0x7f09b406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f09c400e1b0 tx=0x7f09c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:55.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.269+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09d40af300 con 0x7f09d4145790 2024-03-06T07:36:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.640+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d404f3a0 con 0x7f09d4145790 2024-03-06T07:36:55.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.641+0000 7f09caa00640 1 -- 172.21.15.6:0/787697829 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f09cc04b030 con 0x7f09d4145790 2024-03-06T07:36:55.644 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:55.644 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:55.644 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b406b4d0 msgr2=0x7f09b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b406b4d0 0x7f09b406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f09c400e1b0 tx=0x7f09c4009040 comp rx=0 tx=0).stop 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 msgr2=0x7f09d413f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 0x7f09d413f5b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f09cc0097a0 tx=0x7f09cc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 shutdown_connections 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f09b406b4d0 0x7f09b406d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09d4145790 0x7f09d413f5b0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f09d40add50 0x7f09d413f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 --2- 172.21.15.6:0/787697829 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09d406afb0 0x7f09d409e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 >> 172.21.15.6:0/787697829 conn(0x7f09d406c2a0 msgr2=0x7f09d406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 shutdown_connections 2024-03-06T07:36:55.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:55.645+0000 7f09d9600640 1 -- 172.21.15.6:0/787697829 wait complete. 2024-03-06T07:36:55.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:55.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:56.108 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:56.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:57.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.090+0000 7f0adf200640 1 Processor -- start 2024-03-06T07:36:57.092 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.090+0000 7f0adf200640 1 -- start start 2024-03-06T07:36:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0adf200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f0ae0145300 0x7f0ae00b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0adf200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae0070f10 0x7f0ae00cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0adf200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ae00cce90 0x7f0ae00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0adf200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae00cd300 con 0x7f0ae00cce90 2024-03-06T07:36:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0adf200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae00714d0 con 0x7f0ae0145300 2024-03-06T07:36:57.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0adf200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00717b0 con 0x7f0ae0070f10 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0ade800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ae00cce90 0x7f0ae00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0ade800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ae00cce90 0x7f0ae00cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45722/0 (socket says 172.21.15.6:45722) 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.092+0000 7f0ade800640 1 -- 172.21.15.6:0/2614845664 learned_addr learned my addr 172.21.15.6:0/2614845664 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0adde00640 1 --2- 172.21.15.6:0/2614845664 >> v2:172.21.15.178:3300/0 conn(0x7f0ae0145300 0x7f0ae00b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ad7e00640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae0070f10 0x7f0ae00cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ade800640 1 -- 172.21.15.6:0/2614845664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae0070f10 msgr2=0x7f0ae00cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ade800640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae0070f10 0x7f0ae00cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ade800640 1 -- 172.21.15.6:0/2614845664 >> v2:172.21.15.178:3300/0 conn(0x7f0ae0145300 msgr2=0x7f0ae00b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ade800640 1 --2- 172.21.15.6:0/2614845664 >> v2:172.21.15.178:3300/0 conn(0x7f0ae0145300 0x7f0ae00b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ade800640 1 -- 172.21.15.6:0/2614845664 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0071a90 con 0x7f0ae00cce90 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.093+0000 7f0ad7e00640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae0070f10 0x7f0ae00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.094+0000 7f0ade800640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae00cce90 0x7f0ae00cd930 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00cf990 tx=0x7f0ad802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f73d3a85ccaca1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.094+0000 7f0add400640 1 -- 172.21.15.6:0/2614845664 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad802ca70 con 0x7f0ae00cce90 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.094+0000 7f0add400640 1 -- 172.21.15.6:0/2614845664 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad802cc10 con 0x7f0ae00cce90 2024-03-06T07:36:57.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.094+0000 7f0add400640 1 -- 172.21.15.6:0/2614845664 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad80343f0 con 0x7f0ae00cce90 2024-03-06T07:36:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 -- 172.21.15.6:0/2614845664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae00cce90 msgr2=0x7f0ae00cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae00cce90 0x7f0ae00cd930 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00cf990 tx=0x7f0ad802bed0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 -- 172.21.15.6:0/2614845664 shutdown_connections 2024-03-06T07:36:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae00cce90 0x7f0ae00cd930 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 --2- 172.21.15.6:0/2614845664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae0070f10 0x7f0ae00cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 --2- 172.21.15.6:0/2614845664 >> v2:172.21.15.178:3300/0 conn(0x7f0ae0145300 0x7f0ae00b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 -- 172.21.15.6:0/2614845664 >> 172.21.15.6:0/2614845664 conn(0x7f0ae006b3d0 msgr2=0x7f0ae006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 -- 172.21.15.6:0/2614845664 shutdown_connections 2024-03-06T07:36:57.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.095+0000 7f0adf200640 1 -- 172.21.15.6:0/2614845664 wait complete. 2024-03-06T07:36:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.098+0000 7f0adf200640 1 Processor -- start 2024-03-06T07:36:57.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.098+0000 7f0adf200640 1 -- start start 2024-03-06T07:36:57.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adf200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 0x7f0ae0133900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adf200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae00cce90 0x7f0ae00ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adf200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ae0145300 0x7f0ae01e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adf200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae01456e0 con 0x7f0ae0070f10 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adf200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00b4ee0 con 0x7f0ae0145300 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adf200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae00cde70 con 0x7f0ae00cce90 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 0x7f0ae0133900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 0x7f0ae0133900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45724/0 (socket says 172.21.15.6:45724) 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 -- 172.21.15.6:0/745864969 learned_addr learned my addr 172.21.15.6:0/745864969 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0ad7e00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae00cce90 0x7f0ae00ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0ade800640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ae0145300 0x7f0ae01e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 -- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae00cce90 msgr2=0x7f0ae00ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae00cce90 0x7f0ae00ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 -- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ae0145300 msgr2=0x7f0ae01e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ae0145300 0x7f0ae01e70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0adde00640 1 -- 172.21.15.6:0/745864969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0144830 con 0x7f0ae0070f10 2024-03-06T07:36:57.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.100+0000 7f0ad7e00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae00cce90 0x7f0ae00ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:36:57.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.101+0000 7f0adde00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 0x7f0ae0133900 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000d5f0 tx=0x7f0ad000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:57.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.101+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad00125f0 con 0x7f0ae0070f10 2024-03-06T07:36:57.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.101+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad0019070 con 0x7f0ae0070f10 2024-03-06T07:36:57.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.101+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad00129d0 con 0x7f0ae0070f10 2024-03-06T07:36:57.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.101+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae0133e40 con 0x7f0ae0070f10 2024-03-06T07:36:57.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.101+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae0131690 con 0x7f0ae0070f10 2024-03-06T07:36:57.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.103+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae004f330 con 0x7f0ae0070f10 2024-03-06T07:36:57.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.103+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ad0012b70 con 0x7f0ae0070f10 2024-03-06T07:36:57.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.105+0000 7f0ad6000640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0abc06b4d0 0x7f0abc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:57.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.105+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0ad008a610 con 0x7f0ae0070f10 2024-03-06T07:36:57.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.106+0000 7f0ad7e00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0abc06b4d0 0x7f0abc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:57.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.107+0000 7f0ad7e00640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0abc06b4d0 0x7f0abc06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8006fd0 tx=0x7f0ac8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:57.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.108+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ae0144830 con 0x7f0ae0070f10 2024-03-06T07:36:57.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.356+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae0132030 con 0x7f0ae0070f10 2024-03-06T07:36:57.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.357+0000 7f0ad6000640 1 -- 172.21.15.6:0/745864969 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f0ad0059450 con 0x7f0ae0070f10 2024-03-06T07:36:57.360 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:57.360 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:57.360 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:57.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0abc06b4d0 msgr2=0x7f0abc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0abc06b4d0 0x7f0abc06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8006fd0 tx=0x7f0ac8008040 comp rx=0 tx=0).stop 2024-03-06T07:36:57.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 msgr2=0x7f0ae0133900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:57.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 0x7f0ae0133900 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f0ad000d5f0 tx=0x7f0ad000dab0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 shutdown_connections 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0abc06b4d0 0x7f0abc06d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0ae0145300 0x7f0ae01e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0ae00cce90 0x7f0ae00ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 --2- 172.21.15.6:0/745864969 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ae0070f10 0x7f0ae0133900 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.361+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 >> 172.21.15.6:0/745864969 conn(0x7f0ae006b3d0 msgr2=0x7f0ae00b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:57.364 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.362+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 shutdown_connections 2024-03-06T07:36:57.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:57.362+0000 7f0adf200640 1 -- 172.21.15.6:0/745864969 wait complete. 2024-03-06T07:36:57.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:57.589 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:36:58.110 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:36:58.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:36:58.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.808+0000 7fa47d600640 1 Processor -- start 2024-03-06T07:36:58.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.808+0000 7fa47d600640 1 -- start start 2024-03-06T07:36:58.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa47d600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa478070f10 0x7fa4780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa47d600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa4780cce90 0x7fa47815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa47d600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4780714d0 con 0x7fa478145300 2024-03-06T07:36:58.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa47d600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4780cd300 con 0x7fa4780cce90 2024-03-06T07:36:58.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa47d600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4780717b0 con 0x7fa478070f10 2024-03-06T07:36:58.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa477e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa478145300 0x7fa4780b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55160/0 (socket says 172.21.15.6:55160) 2024-03-06T07:36:58.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa477e00640 1 -- 172.21.15.6:0/1757202816 learned_addr learned my addr 172.21.15.6:0/1757202816 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:36:58.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.810+0000 7fa47cc00640 1 --2- 172.21.15.6:0/1757202816 >> v2:172.21.15.6:3300/0 conn(0x7fa4780cce90 0x7fa47815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa477400640 1 --2- 172.21.15.6:0/1757202816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa478070f10 0x7fa4780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa476a00640 1 -- 172.21.15.6:0/1757202816 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503615147 0 0) 0x7fa4780714d0 con 0x7fa478145300 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa476a00640 1 -- 172.21.15.6:0/1757202816 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c003520 con 0x7fa478145300 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa47cc00640 1 -- 172.21.15.6:0/1757202816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa478070f10 msgr2=0x7fa4780cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa47cc00640 1 --2- 172.21.15.6:0/1757202816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa478070f10 0x7fa4780cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa47cc00640 1 -- 172.21.15.6:0/1757202816 >> v1:172.21.15.6:6789/0 conn(0x7fa478145300 legacy=0x7fa4780b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.811+0000 7fa47cc00640 1 -- 172.21.15.6:0/1757202816 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa478071a90 con 0x7fa4780cce90 2024-03-06T07:36:58.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.812+0000 7fa47cc00640 1 --2- 172.21.15.6:0/1757202816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4780cce90 0x7fa47815c570 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fa470009d90 tx=0x7fa47002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229fee43d92810d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.812+0000 7fa476a00640 1 -- 172.21.15.6:0/1757202816 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470039070 con 0x7fa4780cce90 2024-03-06T07:36:58.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa476a00640 1 -- 172.21.15.6:0/1757202816 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47002cc50 con 0x7fa4780cce90 2024-03-06T07:36:58.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa476a00640 1 -- 172.21.15.6:0/1757202816 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa470034440 con 0x7fa4780cce90 2024-03-06T07:36:58.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 -- 172.21.15.6:0/1757202816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4780cce90 msgr2=0x7fa47815c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 --2- 172.21.15.6:0/1757202816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4780cce90 0x7fa47815c570 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fa470009d90 tx=0x7fa47002bea0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 -- 172.21.15.6:0/1757202816 shutdown_connections 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 --2- 172.21.15.6:0/1757202816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa4780cce90 0x7fa47815c570 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 --2- 172.21.15.6:0/1757202816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa478070f10 0x7fa4780cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 -- 172.21.15.6:0/1757202816 >> 172.21.15.6:0/1757202816 conn(0x7fa47806b3d0 msgr2=0x7fa47806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 -- 172.21.15.6:0/1757202816 shutdown_connections 2024-03-06T07:36:58.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.813+0000 7fa47d600640 1 -- 172.21.15.6:0/1757202816 wait complete. 2024-03-06T07:36:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.815+0000 7fa47d600640 1 Processor -- start 2024-03-06T07:36:58.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.816+0000 7fa47d600640 1 -- start start 2024-03-06T07:36:58.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa47d600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 0x7fa47806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa47d600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4780cce90 0x7fa47806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa47d600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa478145300 0x7fa4780ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa47d600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47815cab0 con 0x7fa478070f10 2024-03-06T07:36:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa47d600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4781456e0 con 0x7fa478145300 2024-03-06T07:36:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa47d600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47815cc30 con 0x7fa4780cce90 2024-03-06T07:36:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa477e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 0x7fa47806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa477e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 0x7fa47806d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45736/0 (socket says 172.21.15.6:45736) 2024-03-06T07:36:58.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.817+0000 7fa477e00640 1 -- 172.21.15.6:0/3460927288 learned_addr learned my addr 172.21.15.6:0/3460927288 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:36:58.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477400640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4780cce90 0x7fa47806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477e00640 1 -- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4780cce90 msgr2=0x7fa47806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477e00640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4780cce90 0x7fa47806d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477e00640 1 -- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa478145300 msgr2=0x7fa4780ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:58.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa47cc00640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa478145300 0x7fa4780ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477e00640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa478145300 0x7fa4780ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477e00640 1 -- 172.21.15.6:0/3460927288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4780ae460 con 0x7fa478070f10 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa477e00640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 0x7fa47806d300 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c3c0 tx=0x7fa46800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.818+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468017070 con 0x7fa478070f10 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.819+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa468005ce0 con 0x7fa478070f10 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.819+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4780ae740 con 0x7fa478070f10 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.819+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4680072f0 con 0x7fa478070f10 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.819+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4780aed00 con 0x7fa478070f10 2024-03-06T07:36:58.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.820+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47804f330 con 0x7fa478070f10 2024-03-06T07:36:58.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.821+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa46801a070 con 0x7fa478070f10 2024-03-06T07:36:58.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.822+0000 7fa46fe00640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa45806b500 0x7fa45806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:36:58.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.822+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa4680896b0 con 0x7fa478070f10 2024-03-06T07:36:58.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.823+0000 7fa477400640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa45806b500 0x7fa45806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:36:58.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.824+0000 7fa477400640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa45806b500 0x7fa45806d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa4780af5f0 tx=0x7fa464008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:36:58.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:58.826+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa468058450 con 0x7fa478070f10 2024-03-06T07:36:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.224+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa47815be10 con 0x7fa478070f10 2024-03-06T07:36:59.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.225+0000 7fa46fe00640 1 -- 172.21.15.6:0/3460927288 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fa468057d50 con 0x7fa478070f10 2024-03-06T07:36:59.227 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:36:59.228 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:36:59.228 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa45806b500 msgr2=0x7fa45806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa45806b500 0x7fa45806d9c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fa4780af5f0 tx=0x7fa464008040 comp rx=0 tx=0).stop 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 msgr2=0x7fa47806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 0x7fa47806d300 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c3c0 tx=0x7fa46800c880 comp rx=0 tx=0).stop 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 shutdown_connections 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa45806b500 0x7fa45806d9c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa478145300 0x7fa4780ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa4780cce90 0x7fa47806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 --2- 172.21.15.6:0/3460927288 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa478070f10 0x7fa47806d300 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:36:59.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.227+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 >> 172.21.15.6:0/3460927288 conn(0x7fa47806b3d0 msgr2=0x7fa47815d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:36:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.228+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 shutdown_connections 2024-03-06T07:36:59.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:36:59.228+0000 7fa47d600640 1 -- 172.21.15.6:0/3460927288 wait complete. 2024-03-06T07:36:59.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:36:59.449 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:00.112 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:00.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:00.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.660+0000 7f4907000640 1 Processor -- start 2024-03-06T07:37:00.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.660+0000 7f4907000640 1 -- start start 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4907000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f490015e070 0x7f49000d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4907000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4907000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49000d18c0 con 0x7f49000cf410 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4907000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49000d1ba0 con 0x7f490015e070 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4907000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49000d1e80 con 0x7f49000d0cd0 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4905200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f49000cf410 0x7f49000d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:55168/0 (socket says 172.21.15.6:55168) 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4905200640 1 -- 172.21.15.6:0/289339253 learned_addr learned my addr 172.21.15.6:0/289339253 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:00.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4905c00640 1 --2- 172.21.15.6:0/289339253 >> v2:172.21.15.6:3300/0 conn(0x7f490015e070 0x7f49000d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.662+0000 7f4906600640 1 --2- 172.21.15.6:0/289339253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.663+0000 7f4905c00640 1 -- 172.21.15.6:0/289339253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 msgr2=0x7f49000cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.663+0000 7f4905c00640 1 --2- 172.21.15.6:0/289339253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.663+0000 7f4905c00640 1 -- 172.21.15.6:0/289339253 >> v1:172.21.15.6:6789/0 conn(0x7f49000cf410 legacy=0x7f49000d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.663+0000 7f4905c00640 1 -- 172.21.15.6:0/289339253 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49000d2160 con 0x7f490015e070 2024-03-06T07:37:00.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.663+0000 7f48ffe00640 1 -- 172.21.15.6:0/289339253 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913936471 0 0) 0x7f49000d18c0 con 0x7f49000cf410 2024-03-06T07:37:00.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.664+0000 7f4905c00640 1 --2- 172.21.15.6:0/289339253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f490015e070 0x7f49000d2960 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009d90 tx=0x7f48f002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbbf32405bfefa0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f48ffe00640 1 -- 172.21.15.6:0/289339253 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0039070 con 0x7f490015e070 2024-03-06T07:37:00.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f48ffe00640 1 -- 172.21.15.6:0/289339253 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f002cc50 con 0x7f490015e070 2024-03-06T07:37:00.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f48ffe00640 1 -- 172.21.15.6:0/289339253 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0034440 con 0x7f490015e070 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f4907000640 1 -- 172.21.15.6:0/289339253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f490015e070 msgr2=0x7f49000d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f4907000640 1 --2- 172.21.15.6:0/289339253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f490015e070 0x7f49000d2960 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009d90 tx=0x7f48f002bea0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f4907000640 1 -- 172.21.15.6:0/289339253 shutdown_connections 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f4907000640 1 --2- 172.21.15.6:0/289339253 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f4907000640 1 --2- 172.21.15.6:0/289339253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f490015e070 0x7f49000d2960 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.665+0000 7f4907000640 1 -- 172.21.15.6:0/289339253 >> 172.21.15.6:0/289339253 conn(0x7f490008d2a0 msgr2=0x7f49000d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.666+0000 7f4907000640 1 -- 172.21.15.6:0/289339253 shutdown_connections 2024-03-06T07:37:00.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.666+0000 7f4907000640 1 -- 172.21.15.6:0/289339253 wait complete. 2024-03-06T07:37:00.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.668+0000 7f4907000640 1 Processor -- start 2024-03-06T07:37:00.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.669+0000 7f4907000640 1 -- start start 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.670+0000 7f4907000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 0x7f490015ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.670+0000 7f4907000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.670+0000 7f4907000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f490015e070 0x7f49000cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.670+0000 7f4907000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49000d4730 con 0x7f49000cf410 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.670+0000 7f4907000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49000d45b0 con 0x7f490015e070 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.670+0000 7f4907000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49000b3630 con 0x7f49000d0cd0 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 0x7f490015ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 0x7f490015ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45752/0 (socket says 172.21.15.6:45752) 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 -- 172.21.15.6:0/360859553 learned_addr learned my addr 172.21.15.6:0/360859553 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 -- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 msgr2=0x7f49000b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 -- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f490015e070 msgr2=0x7f49000cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4906600640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f490015e070 0x7f49000cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f490015e070 0x7f49000cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 -- 172.21.15.6:0/360859553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490015e4f0 con 0x7f49000cf410 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.671+0000 7f4905c00640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 0x7f490015ca10 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f48f0037040 tx=0x7f48f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.672+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0039050 con 0x7f49000cf410 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.672+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f003a040 con 0x7f49000cf410 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.672+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490015e7c0 con 0x7f49000cf410 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.672+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f002ca50 con 0x7f49000cf410 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.672+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49000cc610 con 0x7f49000cf410 2024-03-06T07:37:00.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.673+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490004f330 con 0x7f49000cf410 2024-03-06T07:37:00.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.675+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48f0047020 con 0x7f49000cf410 2024-03-06T07:37:00.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.676+0000 7f48fd600640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f48e006b4d0 0x7f48e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:00.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.676+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f48f00ac3c0 con 0x7f49000cf410 2024-03-06T07:37:00.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.677+0000 7f4905200640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f48e006b4d0 0x7f48e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:00.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.678+0000 7f4905200640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f48e006b4d0 0x7f48e006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f48ec007fd0 tx=0x7f48ec007920 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:00.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.681+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48f004e020 con 0x7f49000cf410 2024-03-06T07:37:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.920+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49000c75c0 con 0x7f49000cf410 2024-03-06T07:37:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.921+0000 7f48fd600640 1 -- 172.21.15.6:0/360859553 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f490015e7c0 con 0x7f49000cf410 2024-03-06T07:37:00.924 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:00.924 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:00.924 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f48e006b4d0 msgr2=0x7f48e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f48e006b4d0 0x7f48e006d990 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f48ec007fd0 tx=0x7f48ec007920 comp rx=0 tx=0).stop 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 msgr2=0x7f490015ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 0x7f490015ca10 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f48f0037040 tx=0x7f48f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 shutdown_connections 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f48e006b4d0 0x7f48e006d990 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f490015e070 0x7f49000cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f49000d0cd0 0x7f49000b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 --2- 172.21.15.6:0/360859553 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f49000cf410 0x7f490015ca10 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 >> 172.21.15.6:0/360859553 conn(0x7f490008d2a0 msgr2=0x7f490006d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 shutdown_connections 2024-03-06T07:37:00.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:00.924+0000 7f4907000640 1 -- 172.21.15.6:0/360859553 wait complete. 2024-03-06T07:37:01.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:01.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:02.114 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:02.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:02.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.373+0000 7f51de400640 1 Processor -- start 2024-03-06T07:37:02.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.373+0000 7f51de400640 1 -- start start 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51de400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f51d80712c0 0x7f51d8162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51de400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80ce090 con 0x7f51d8072da0 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51de400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d80ce370 con 0x7f51d8070ee0 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51de400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80ceca0 con 0x7f51d80712c0 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51d7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f51d80712c0 0x7f51d8162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51cfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f51d8072da0 0x7f51d806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45124/0 (socket says 172.21.15.6:45124) 2024-03-06T07:37:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.375+0000 7f51cfe00640 1 -- 172.21.15.6:0/2546256652 learned_addr learned my addr 172.21.15.6:0/2546256652 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:02.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.376+0000 7f51d7e00640 1 -- 172.21.15.6:0/2546256652 >> v1:172.21.15.178:6789/0 conn(0x7f51d8070ee0 legacy=0x7f51d806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:02.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.376+0000 7f51d7e00640 1 -- 172.21.15.6:0/2546256652 >> v1:172.21.15.6:6789/0 conn(0x7f51d8072da0 legacy=0x7f51d806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.376+0000 7f51d7e00640 1 -- 172.21.15.6:0/2546256652 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80ce650 con 0x7f51d80712c0 2024-03-06T07:37:02.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.376+0000 7f51d6a00640 1 -- 172.21.15.6:0/2546256652 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462753590 0 0) 0x7f51d80ce090 con 0x7f51d8072da0 2024-03-06T07:37:02.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.377+0000 7f51d7e00640 1 --2- 172.21.15.6:0/2546256652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8162c40 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f51d80cee20 tx=0x7f51d002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5748a6556855d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.377+0000 7f51d6a00640 1 -- 172.21.15.6:0/2546256652 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0039070 con 0x7f51d80712c0 2024-03-06T07:37:02.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.377+0000 7f51d6a00640 1 -- 172.21.15.6:0/2546256652 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d002cc50 con 0x7f51d80712c0 2024-03-06T07:37:02.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.377+0000 7f51d6a00640 1 -- 172.21.15.6:0/2546256652 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0034440 con 0x7f51d80712c0 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 -- 172.21.15.6:0/2546256652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 msgr2=0x7f51d8162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 --2- 172.21.15.6:0/2546256652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8162c40 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f51d80cee20 tx=0x7f51d002bea0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 -- 172.21.15.6:0/2546256652 shutdown_connections 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 --2- 172.21.15.6:0/2546256652 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8162c40 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 -- 172.21.15.6:0/2546256652 >> 172.21.15.6:0/2546256652 conn(0x7f51d806afb0 msgr2=0x7f51d806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 -- 172.21.15.6:0/2546256652 shutdown_connections 2024-03-06T07:37:02.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.378+0000 7f51de400640 1 -- 172.21.15.6:0/2546256652 wait complete. 2024-03-06T07:37:02.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.380+0000 7f51de400640 1 Processor -- start 2024-03-06T07:37:02.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.380+0000 7f51de400640 1 -- start start 2024-03-06T07:37:02.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51de400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51d8070ee0 0x7f51d809bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51de400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51de400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51d8072da0 0x7f51d8162520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51de400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80cdda0 con 0x7f51d80712c0 2024-03-06T07:37:02.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51de400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80cdc20 con 0x7f51d8070ee0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51de400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d806e7d0 con 0x7f51d8072da0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35188/0 (socket says 172.21.15.6:35188) 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51cfe00640 1 -- 172.21.15.6:0/68280792 learned_addr learned my addr 172.21.15.6:0/68280792 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51d7e00640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51d8072da0 0x7f51d8162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51d7400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51d8070ee0 0x7f51d809bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.381+0000 7f51cfe00640 1 -- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51d8072da0 msgr2=0x7f51d8162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cfe00640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51d8072da0 0x7f51d8162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cfe00640 1 -- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51d8070ee0 msgr2=0x7f51d809bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cfe00640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51d8070ee0 0x7f51d809bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cfe00640 1 -- 172.21.15.6:0/68280792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d815de10 con 0x7f51d80712c0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51d7400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51d8070ee0 0x7f51d809bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cfe00640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8156830 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f51c000c430 tx=0x7f51c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c00074e0 con 0x7f51d80712c0 2024-03-06T07:37:02.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c00047e0 con 0x7f51d80712c0 2024-03-06T07:37:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d815e0f0 con 0x7f51d80712c0 2024-03-06T07:37:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0004bd0 con 0x7f51d80712c0 2024-03-06T07:37:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.382+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d815e6b0 con 0x7f51d80712c0 2024-03-06T07:37:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.383+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d804f3a0 con 0x7f51d80712c0 2024-03-06T07:37:02.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.384+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51c0011070 con 0x7f51d80712c0 2024-03-06T07:37:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.385+0000 7f51cf400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51b406b4d0 0x7f51b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:02.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.385+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51c0089ed0 con 0x7f51d80712c0 2024-03-06T07:37:02.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.386+0000 7f51d7400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51b406b4d0 0x7f51b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:02.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.387+0000 7f51d7400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51b406b4d0 0x7f51b406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f51c8003eb0 tx=0x7f51c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:02.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.390+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51c0058d00 con 0x7f51d80712c0 2024-03-06T07:37:02.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.624+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51d80af3c0 con 0x7f51d80712c0 2024-03-06T07:37:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.625+0000 7f51cf400640 1 -- 172.21.15.6:0/68280792 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f51c0058600 con 0x7f51d80712c0 2024-03-06T07:37:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:02.627 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:02.627 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51b406b4d0 msgr2=0x7f51b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51b406b4d0 0x7f51b406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f51c8003eb0 tx=0x7f51c8009040 comp rx=0 tx=0).stop 2024-03-06T07:37:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 msgr2=0x7f51d8156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:02.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8156830 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f51c000c430 tx=0x7f51c000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 shutdown_connections 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f51b406b4d0 0x7f51b406d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f51d8072da0 0x7f51d8162520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f51d80712c0 0x7f51d8156830 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 --2- 172.21.15.6:0/68280792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51d8070ee0 0x7f51d809bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 >> 172.21.15.6:0/68280792 conn(0x7f51d806afb0 msgr2=0x7f51d8164b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 shutdown_connections 2024-03-06T07:37:02.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:02.627+0000 7f51de400640 1 -- 172.21.15.6:0/68280792 wait complete. 2024-03-06T07:37:02.848 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:02.848 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:03.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.068+0000 7f0aebe00640 1 Processor -- start 2024-03-06T07:37:04.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.068+0000 7f0aebe00640 1 -- start start 2024-03-06T07:37:04.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aebe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0aec15c360 0x7f0aec06ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec06c690 0x7f0aec15c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aebe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aec0afb40 con 0x7f0aec0ce370 2024-03-06T07:37:04.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aebe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aec06d310 con 0x7f0aec15c360 2024-03-06T07:37:04.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aebe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec06d840 con 0x7f0aec06c690 2024-03-06T07:37:04.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aeaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0aec15c360 0x7f0aec06ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aeaa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0aec15c360 0x7f0aec06ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35194/0 (socket says 172.21.15.6:35194) 2024-03-06T07:37:04.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.070+0000 7f0aeaa00640 1 -- 172.21.15.6:0/4177393848 learned_addr learned my addr 172.21.15.6:0/4177393848 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:04.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.071+0000 7f0aeb400640 1 --2- 172.21.15.6:0/4177393848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec06c690 0x7f0aec15c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.071+0000 7f0aeaa00640 1 -- 172.21.15.6:0/4177393848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec06c690 msgr2=0x7f0aec15c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.071+0000 7f0aeaa00640 1 --2- 172.21.15.6:0/4177393848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec06c690 0x7f0aec15c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.071+0000 7f0aeaa00640 1 -- 172.21.15.6:0/4177393848 >> v1:172.21.15.178:6789/0 conn(0x7f0aec0ce370 legacy=0x7f0aec06bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.071+0000 7f0aeaa00640 1 -- 172.21.15.6:0/4177393848 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec0afe20 con 0x7f0aec15c360 2024-03-06T07:37:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.072+0000 7f0aeaa00640 1 --2- 172.21.15.6:0/4177393848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec15c360 0x7f0aec06ed90 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009870 tx=0x7f0adc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0c7d31b5af310a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.072+0000 7f0ae9600640 1 -- 172.21.15.6:0/4177393848 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc035020 con 0x7f0aec15c360 2024-03-06T07:37:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.072+0000 7f0ae9600640 1 -- 172.21.15.6:0/4177393848 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0adc031640 con 0x7f0aec15c360 2024-03-06T07:37:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.072+0000 7f0ae9600640 1 -- 172.21.15.6:0/4177393848 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc02d040 con 0x7f0aec15c360 2024-03-06T07:37:04.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.073+0000 7f0aebe00640 1 -- 172.21.15.6:0/4177393848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec15c360 msgr2=0x7f0aec06ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.073+0000 7f0aebe00640 1 --2- 172.21.15.6:0/4177393848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec15c360 0x7f0aec06ed90 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009870 tx=0x7f0adc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.073+0000 7f0aebe00640 1 -- 172.21.15.6:0/4177393848 shutdown_connections 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.073+0000 7f0aebe00640 1 --2- 172.21.15.6:0/4177393848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec06c690 0x7f0aec15c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.074+0000 7f0aebe00640 1 --2- 172.21.15.6:0/4177393848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec15c360 0x7f0aec06ed90 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.074+0000 7f0aebe00640 1 -- 172.21.15.6:0/4177393848 >> 172.21.15.6:0/4177393848 conn(0x7f0aec0ccf70 msgr2=0x7f0aec0cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.074+0000 7f0aebe00640 1 -- 172.21.15.6:0/4177393848 shutdown_connections 2024-03-06T07:37:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.074+0000 7f0aebe00640 1 -- 172.21.15.6:0/4177393848 wait complete. 2024-03-06T07:37:04.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.075+0000 7f0aebe00640 1 Processor -- start 2024-03-06T07:37:04.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.075+0000 7f0aebe00640 1 -- start start 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.076+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aec06c690 0x7f0aec148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.076+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 0x7f0aec0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec15c360 0x7f0aec0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aebe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec06ba60 con 0x7f0aec0ce370 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aebe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec06f2d0 con 0x7f0aec06c690 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aebe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec15bfa0 con 0x7f0aec15c360 2024-03-06T07:37:04.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 0x7f0aec0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aea000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 0x7f0aec0cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35200/0 (socket says 172.21.15.6:35200) 2024-03-06T07:37:04.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aea000640 1 -- 172.21.15.6:0/781478973 learned_addr learned my addr 172.21.15.6:0/781478973 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:04.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aeaa00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aec06c690 0x7f0aec148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.077+0000 7f0aeb400640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec15c360 0x7f0aec0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aea000640 1 -- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec15c360 msgr2=0x7f0aec0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aea000640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec15c360 0x7f0aec0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aea000640 1 -- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aec06c690 msgr2=0x7f0aec148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aea000640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aec06c690 0x7f0aec148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aea000640 1 -- 172.21.15.6:0/781478973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aec15d600 con 0x7f0aec0ce370 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aea000640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 0x7f0aec0cfad0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800c3c0 tx=0x7f0ad800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad80052d0 con 0x7f0aec0ce370 2024-03-06T07:37:04.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad8005490 con 0x7f0aec0ce370 2024-03-06T07:37:04.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8007260 con 0x7f0aec0ce370 2024-03-06T07:37:04.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec15d8e0 con 0x7f0aec0ce370 2024-03-06T07:37:04.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.078+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec0b49b0 con 0x7f0aec0ce370 2024-03-06T07:37:04.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.079+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aec1492f0 con 0x7f0aec0ce370 2024-03-06T07:37:04.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.080+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ad8007400 con 0x7f0aec0ce370 2024-03-06T07:37:04.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.081+0000 7f0ae2a00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0acc06b4d0 0x7f0acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:04.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.081+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0ad8089680 con 0x7f0aec0ce370 2024-03-06T07:37:04.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.082+0000 7f0aeaa00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0acc06b4d0 0x7f0acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.083+0000 7f0aeaa00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0acc06b4d0 0x7f0acc06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0adc036000 tx=0x7f0adc03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:04.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.086+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ad8057e60 con 0x7f0aec0ce370 2024-03-06T07:37:04.115 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:04.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.327+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0aec04f330 con 0x7f0aec0ce370 2024-03-06T07:37:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.328+0000 7f0ae2a00640 1 -- 172.21.15.6:0/781478973 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f0ad805ce20 con 0x7f0aec0ce370 2024-03-06T07:37:04.330 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:04.330 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:04.330 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.330+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0acc06b4d0 msgr2=0x7f0acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.330+0000 7f0aebe00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0acc06b4d0 0x7f0acc06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0adc036000 tx=0x7f0adc03c040 comp rx=0 tx=0).stop 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 msgr2=0x7f0aec0cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 0x7f0aec0cfad0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800c3c0 tx=0x7f0ad800c880 comp rx=0 tx=0).stop 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 shutdown_connections 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0acc06b4d0 0x7f0acc06d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0aec15c360 0x7f0aec0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.333 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0aec0ce370 0x7f0aec0cfad0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 --2- 172.21.15.6:0/781478973 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0aec06c690 0x7f0aec148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 >> 172.21.15.6:0/781478973 conn(0x7f0aec0ccf70 msgr2=0x7f0aec0b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 shutdown_connections 2024-03-06T07:37:04.334 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:04.331+0000 7f0aebe00640 1 -- 172.21.15.6:0/781478973 wait complete. 2024-03-06T07:37:04.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:04.555 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:05.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:05.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.792+0000 7f45d8800640 1 Processor -- start 2024-03-06T07:37:05.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.793+0000 7f45d8800640 1 -- start start 2024-03-06T07:37:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.794+0000 7f45d8800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.794+0000 7f45d8800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f45d006c500 0x7f45d00ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.794+0000 7f45d8800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d015c5e0 con 0x7f45d0152bc0 2024-03-06T07:37:05.796 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.794+0000 7f45d8800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d00d0120 con 0x7f45d006c500 2024-03-06T07:37:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.794+0000 7f45d8800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d015cf10 con 0x7f45d00cfce0 2024-03-06T07:37:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.795+0000 7f45d6800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f45d006c500 0x7f45d00ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.795+0000 7f45d5400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.795+0000 7f45d5e00640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7f45d0152bc0 0x7f45d00af940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:51440/0 (socket says 172.21.15.6:51440) 2024-03-06T07:37:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.795+0000 7f45d5e00640 1 -- 172.21.15.6:0/2679928186 learned_addr learned my addr 172.21.15.6:0/2679928186 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:05.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.795+0000 7f45d6800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f45d006c500 0x7f45d00ae5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:56968/0 (socket says 172.21.15.6:56968) 2024-03-06T07:37:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.796+0000 7f45d6800640 1 -- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 msgr2=0x7f45d006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.796+0000 7f45d6800640 1 --2- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.796+0000 7f45d6800640 1 -- 172.21.15.6:0/2679928186 >> v1:172.21.15.178:6789/0 conn(0x7f45d0152bc0 legacy=0x7f45d00af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.796+0000 7f45d6800640 1 -- 172.21.15.6:0/2679928186 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d015c8c0 con 0x7f45d006c500 2024-03-06T07:37:05.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.797+0000 7f45d6800640 1 --2- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d006c500 0x7f45d00ae5c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009d90 tx=0x7f45cc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=404c199b871b4200 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d4a00640 1 -- 172.21.15.6:0/2679928186 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc039070 con 0x7f45d006c500 2024-03-06T07:37:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d4a00640 1 -- 172.21.15.6:0/2679928186 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc02cc50 con 0x7f45d006c500 2024-03-06T07:37:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d4a00640 1 -- 172.21.15.6:0/2679928186 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc034440 con 0x7f45d006c500 2024-03-06T07:37:05.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 -- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d006c500 msgr2=0x7f45d00ae5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 --2- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d006c500 0x7f45d00ae5c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009d90 tx=0x7f45cc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 -- 172.21.15.6:0/2679928186 shutdown_connections 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 --2- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d006c500 0x7f45d00ae5c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 --2- 172.21.15.6:0/2679928186 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 -- 172.21.15.6:0/2679928186 >> 172.21.15.6:0/2679928186 conn(0x7f45d00714d0 msgr2=0x7f45d00718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 -- 172.21.15.6:0/2679928186 shutdown_connections 2024-03-06T07:37:05.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.798+0000 7f45d8800640 1 -- 172.21.15.6:0/2679928186 wait complete. 2024-03-06T07:37:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.800+0000 7f45d8800640 1 Processor -- start 2024-03-06T07:37:05.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.800+0000 7f45d8800640 1 -- start start 2024-03-06T07:37:05.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.801+0000 7f45d8800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 0x7f45d0140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.801+0000 7f45d8800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.801+0000 7f45d8800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d0152bc0 0x7f45d006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.801+0000 7f45d8800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d0152fa0 con 0x7f45d006c500 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.801+0000 7f45d8800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d015c290 con 0x7f45d0152bc0 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.801+0000 7f45d8800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d0153120 con 0x7f45d00cfce0 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 0x7f45d0140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 0x7f45d0140d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35208/0 (socket says 172.21.15.6:35208) 2024-03-06T07:37:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 -- 172.21.15.6:0/4050213675 learned_addr learned my addr 172.21.15.6:0/4050213675 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:05.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5400640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d6800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d0152bc0 0x7f45d006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 -- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 msgr2=0x7f45d006dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 -- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d0152bc0 msgr2=0x7f45d006e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d0152bc0 0x7f45d006e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.802+0000 7f45d5e00640 1 -- 172.21.15.6:0/4050213675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00b46f0 con 0x7f45d006c500 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.803+0000 7f45d5e00640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 0x7f45d0140d60 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c3c0 tx=0x7f45c400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.803+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c4017070 con 0x7f45d006c500 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.803+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c4005ce0 con 0x7f45d006c500 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.803+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c40072f0 con 0x7f45d006c500 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.803+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00b49d0 con 0x7f45d006c500 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.803+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d00b4f90 con 0x7f45d006c500 2024-03-06T07:37:05.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.804+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d004f3a0 con 0x7f45d006c500 2024-03-06T07:37:05.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.805+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45c401a070 con 0x7f45d006c500 2024-03-06T07:37:05.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.806+0000 7f45ca000640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45b406b500 0x7f45b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:05.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.806+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f45c4089500 con 0x7f45d006c500 2024-03-06T07:37:05.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.807+0000 7f45d5400640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45b406b500 0x7f45b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:05.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.808+0000 7f45d5400640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45b406b500 0x7f45b406d9c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f45d01f7520 tx=0x7f45c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:05.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:05.811+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45c40583d0 con 0x7f45d006c500 2024-03-06T07:37:06.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.059+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45d0072100 con 0x7f45d006c500 2024-03-06T07:37:06.062 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.061+0000 7f45ca000640 1 -- 172.21.15.6:0/4050213675 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f45c4057cd0 con 0x7f45d006c500 2024-03-06T07:37:06.063 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:06.063 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:06.063 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45b406b500 msgr2=0x7f45b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45b406b500 0x7f45b406d9c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f45d01f7520 tx=0x7f45c0008040 comp rx=0 tx=0).stop 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 msgr2=0x7f45d0140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 0x7f45d0140d60 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f45c400c3c0 tx=0x7f45c400c880 comp rx=0 tx=0).stop 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 shutdown_connections 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f45b406b500 0x7f45b406d9c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f45d0152bc0 0x7f45d006e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f45d00cfce0 0x7f45d006dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 --2- 172.21.15.6:0/4050213675 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45d006c500 0x7f45d0140d60 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.063+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 >> 172.21.15.6:0/4050213675 conn(0x7f45d00714d0 msgr2=0x7f45d00cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.064+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 shutdown_connections 2024-03-06T07:37:06.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:06.064+0000 7f45d8800640 1 -- 172.21.15.6:0/4050213675 wait complete. 2024-03-06T07:37:06.118 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:06.286 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:06.286 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:07.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:07.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.513+0000 7f86b1e00640 1 Processor -- start 2024-03-06T07:37:07.515 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.513+0000 7f86b1e00640 1 -- start start 2024-03-06T07:37:07.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.514+0000 7f86b1e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f86ac0d0cd0 0x7f86ac0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.514+0000 7f86b1e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac0d2dc0 con 0x7f86ac0d2960 2024-03-06T07:37:07.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.515+0000 7f86b1e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac0d18c0 con 0x7f86ac15e070 2024-03-06T07:37:07.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.515+0000 7f86b1e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0d1ba0 con 0x7f86ac0d0cd0 2024-03-06T07:37:07.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.515+0000 7f86a3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f86ac0d2960 0x7f86ac0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45140/0 (socket says 172.21.15.6:45140) 2024-03-06T07:37:07.517 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.515+0000 7f86a3e00640 1 -- 172.21.15.6:0/811277014 learned_addr learned my addr 172.21.15.6:0/811277014 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:07.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.515+0000 7f86b1400640 1 --2- 172.21.15.6:0/811277014 >> v2:172.21.15.178:3300/0 conn(0x7f86ac0d0cd0 0x7f86ac0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.515+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489849381 0 0) 0x7f86ac0d2dc0 con 0x7f86ac0d2960 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8690003520 con 0x7f86ac0d2960 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991522446 0 0) 0x7f8690003520 con 0x7f86ac0d2960 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 >> v2:172.21.15.178:3300/0 conn(0x7f86ac0d0cd0 msgr2=0x7f86ac0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 --2- 172.21.15.6:0/811277014 >> v2:172.21.15.178:3300/0 conn(0x7f86ac0d0cd0 0x7f86ac0b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 >> v1:172.21.15.178:6789/0 conn(0x7f86ac15e070 legacy=0x7f86ac0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac0d1e80 con 0x7f86ac0d2960 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f8694001670 con 0x7f86ac0d2960 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 >> v1:172.21.15.6:6789/0 conn(0x7f86ac0d2960 legacy=0x7f86ac0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f8690003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86900051c0 0x7f869000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0d1ba0 con 0x7f8690003a00 2024-03-06T07:37:07.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869000c0c0 con 0x7f8690004800 2024-03-06T07:37:07.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869000c2c0 con 0x7f86900051c0 2024-03-06T07:37:07.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.516+0000 7f86b0a00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f8690003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b1400640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86900051c0 0x7f869000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86a3e00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b0a00640 1 -- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86900051c0 msgr2=0x7f869000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b0a00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86900051c0 0x7f869000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b0a00640 1 -- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8690004800 msgr2=0x7f8690004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b0a00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b0a00640 1 -- 172.21.15.6:0/811277014 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8690003520 con 0x7f8690003a00 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.517+0000 7f86b1400640 1 -- 172.21.15.6:0/811277014 reap_dead start 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.518+0000 7f86b0a00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f8690003e20 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f869c001750 tx=0x7f8690005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acb425be1b7f862 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.519+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c001370 con 0x7f8690003a00 2024-03-06T07:37:07.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.519+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c0052d0 con 0x7f8690003a00 2024-03-06T07:37:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.519+0000 7f86abe00640 1 -- 172.21.15.6:0/811277014 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c003830 con 0x7f8690003a00 2024-03-06T07:37:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.519+0000 7f86b1e00640 1 -- 172.21.15.6:0/811277014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 msgr2=0x7f8690003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.519+0000 7f86b1e00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f8690003e20 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f869c001750 tx=0x7f8690005830 comp rx=0 tx=0).stop 2024-03-06T07:37:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.520+0000 7f86b1e00640 1 -- 172.21.15.6:0/811277014 shutdown_connections 2024-03-06T07:37:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.520+0000 7f86b1e00640 1 --2- 172.21.15.6:0/811277014 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f8690003e20 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.520+0000 7f86b1e00640 1 -- 172.21.15.6:0/811277014 >> 172.21.15.6:0/811277014 conn(0x7f86ac08d2a0 msgr2=0x7f86ac0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.520+0000 7f86b1e00640 1 -- 172.21.15.6:0/811277014 shutdown_connections 2024-03-06T07:37:07.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.520+0000 7f86b1e00640 1 -- 172.21.15.6:0/811277014 wait complete. 2024-03-06T07:37:07.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.523+0000 7f86b1e00640 1 Processor -- start 2024-03-06T07:37:07.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.523+0000 7f86b1e00640 1 -- start start 2024-03-06T07:37:07.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.524+0000 7f86b1e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f86ac0b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.527 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.524+0000 7f86b1e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86ac0b71b0 0x7f86ac0b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.524+0000 7f86b1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86ac0b2f30 0x7f86ac0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b1e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0d45b0 con 0x7f8690003a00 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b1e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0d2440 con 0x7f86ac0b71b0 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b1e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0d4730 con 0x7f86ac0b2f30 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f86ac0b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f86ac0b6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35232/0 (socket says 172.21.15.6:35232) 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 -- 172.21.15.6:0/2236999013 learned_addr learned my addr 172.21.15.6:0/2236999013 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86a3e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86ac0b71b0 0x7f86ac0b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b1400640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86ac0b2f30 0x7f86ac0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 -- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86ac0b2f30 msgr2=0x7f86ac0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86ac0b2f30 0x7f86ac0ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 -- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86ac0b71b0 msgr2=0x7f86ac0b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86ac0b71b0 0x7f86ac0b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86b0a00640 1 -- 172.21.15.6:0/2236999013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac0cbf40 con 0x7f8690003a00 2024-03-06T07:37:07.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.525+0000 7f86a3e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86ac0b71b0 0x7f86ac0b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:07.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.526+0000 7f86b0a00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f86ac0b6c70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f869c02bd00 tx=0x7f869c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.526+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c041020 con 0x7f8690003a00 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.526+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c005480 con 0x7f8690003a00 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.526+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac0cc220 con 0x7f8690003a00 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.526+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c001370 con 0x7f8690003a00 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.526+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac0b33e0 con 0x7f8690003a00 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.527+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac04f330 con 0x7f8690003a00 2024-03-06T07:37:07.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.528+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f869c044020 con 0x7f8690003a00 2024-03-06T07:37:07.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.530+0000 7f86a9600640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:07.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.530+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f869c0abcd0 con 0x7f8690003a00 2024-03-06T07:37:07.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.531+0000 7f86a3e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:07.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.532+0000 7f86a3e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f868806b500 0x7f868806d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0cdf70 tx=0x7f8694000f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:07.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.534+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f869c04b020 con 0x7f8690003a00 2024-03-06T07:37:07.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.774+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ac0b7590 con 0x7f8690003a00 2024-03-06T07:37:07.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.775+0000 7f86a9600640 1 -- 172.21.15.6:0/2236999013 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f869c07aba0 con 0x7f8690003a00 2024-03-06T07:37:07.777 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:07.777 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:07.777 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.778+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f868806b500 msgr2=0x7f868806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.778+0000 7f86b1e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f868806b500 0x7f868806d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0cdf70 tx=0x7f8694000f10 comp rx=0 tx=0).stop 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.778+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 msgr2=0x7f86ac0b6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.778+0000 7f86b1e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f86ac0b6c70 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f869c02bd00 tx=0x7f869c001120 comp rx=0 tx=0).stop 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 shutdown_connections 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f868806b500 0x7f868806d9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86ac0b71b0 0x7f86ac0b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f86ac0b2f30 0x7f86ac0ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 --2- 172.21.15.6:0/2236999013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8690003a00 0x7f86ac0b6c70 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:07.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 >> 172.21.15.6:0/2236999013 conn(0x7f86ac08d2a0 msgr2=0x7f86ac06cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:07.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 shutdown_connections 2024-03-06T07:37:07.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:07.779+0000 7f86b1e00640 1 -- 172.21.15.6:0/2236999013 wait complete. 2024-03-06T07:37:08.007 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:08.007 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:08.120 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:09.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.232+0000 7fc3b4e00640 1 Processor -- start 2024-03-06T07:37:09.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.233+0000 7fc3b4e00640 1 -- start start 2024-03-06T07:37:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.234+0000 7fc3b4e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc3b01477e0 0x7fc3b0162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.234+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 0x7fc3b006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.234+0000 7fc3b4e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b0147bc0 con 0x7fc3b006b4f0 2024-03-06T07:37:09.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.234+0000 7fc3b4e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0161f90 con 0x7fc3b01477e0 2024-03-06T07:37:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.234+0000 7fc3b4e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0162960 con 0x7fc3b006ef20 2024-03-06T07:37:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3afe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc3b006b4f0 0x7fc3b00d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45152/0 (socket says 172.21.15.6:45152) 2024-03-06T07:37:09.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3afe00640 1 -- 172.21.15.6:0/1620199812 learned_addr learned my addr 172.21.15.6:0/1620199812 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3af400640 1 --2- 172.21.15.6:0/1620199812 >> v2:172.21.15.178:3300/0 conn(0x7fc3b01477e0 0x7fc3b0162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3aea00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 0x7fc3b006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002414068 0 0) 0x7fc3b0147bc0 con 0x7fc3b006b4f0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc394003520 con 0x7fc3b006b4f0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1217177811 0 0) 0x7fc394003520 con 0x7fc3b006b4f0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 msgr2=0x7fc3b006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 0x7fc3b006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 >> v2:172.21.15.178:3300/0 conn(0x7fc3b01477e0 msgr2=0x7fc3b0162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.235+0000 7fc3ae000640 1 --2- 172.21.15.6:0/1620199812 >> v2:172.21.15.178:3300/0 conn(0x7fc3b01477e0 0x7fc3b0162420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b00b3d70 con 0x7fc3b006b4f0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fc3a80019f0 con 0x7fc3b006b4f0 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 >> v1:172.21.15.6:6789/0 conn(0x7fc3b006b4f0 legacy=0x7fc3b00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3940039b0 0x7fc394003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 0x7fc394004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc394004d40 0x7fc3940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0161f90 con 0x7fc394004d40 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0162960 con 0x7fc3940039b0 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39400b9f0 con 0x7fc3b006ef20 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3afe00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc394004d40 0x7fc3940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3aea00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 0x7fc394004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3af400640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3940039b0 0x7fc394003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3afe00640 1 -- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 msgr2=0x7fc394004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3afe00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b006ef20 0x7fc394004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3afe00640 1 -- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3940039b0 msgr2=0x7fc394003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.236+0000 7fc3afe00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3940039b0 0x7fc394003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.237+0000 7fc3afe00640 1 -- 172.21.15.6:0/1620199812 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc394003520 con 0x7fc394004d40 2024-03-06T07:37:09.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.238+0000 7fc3afe00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc394004d40 0x7fc3940051c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800a7d0 tx=0x7fc394005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79369b6a7b5230dd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.238+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a800a3f0 con 0x7fc394004d40 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.238+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a8034630 con 0x7fc394004d40 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.238+0000 7fc3ae000640 1 -- 172.21.15.6:0/1620199812 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a802ce60 con 0x7fc394004d40 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.238+0000 7fc3b4e00640 1 -- 172.21.15.6:0/1620199812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc394004d40 msgr2=0x7fc3940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.238+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/1620199812 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc394004d40 0x7fc3940051c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800a7d0 tx=0x7fc394005830 comp rx=0 tx=0).stop 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.239+0000 7fc3afe00640 1 -- 172.21.15.6:0/1620199812 reap_dead start 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.239+0000 7fc3b4e00640 1 -- 172.21.15.6:0/1620199812 shutdown_connections 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.239+0000 7fc3b4e00640 1 -- 172.21.15.6:0/1620199812 >> 172.21.15.6:0/1620199812 conn(0x7fc3b006c0c0 msgr2=0x7fc3b006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.239+0000 7fc3b4e00640 1 -- 172.21.15.6:0/1620199812 shutdown_connections 2024-03-06T07:37:09.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.239+0000 7fc3b4e00640 1 -- 172.21.15.6:0/1620199812 wait complete. 2024-03-06T07:37:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.241+0000 7fc3b4e00640 1 Processor -- start 2024-03-06T07:37:09.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.241+0000 7fc3b4e00640 1 -- start start 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b015d5f0 0x7fc3b015ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b013d7d0 0x7fc3b013e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3b4e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 0x7fc3b013c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3b4e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0161c20 con 0x7fc3b013e790 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3b4e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00b4330 con 0x7fc3b015d5f0 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3b4e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b006f8f0 con 0x7fc3b013d7d0 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 0x7fc3b013c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 0x7fc3b013c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35260/0 (socket says 172.21.15.6:35260) 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3afe00640 1 -- 172.21.15.6:0/3650241813 learned_addr learned my addr 172.21.15.6:0/3650241813 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3af400640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b015d5f0 0x7fc3b015ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.242+0000 7fc3aea00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b013d7d0 0x7fc3b013e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3afe00640 1 -- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b013d7d0 msgr2=0x7fc3b013e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3afe00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b013d7d0 0x7fc3b013e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3afe00640 1 -- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b015d5f0 msgr2=0x7fc3b015ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3afe00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b015d5f0 0x7fc3b015ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3afe00640 1 -- 172.21.15.6:0/3650241813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0140740 con 0x7fc3b013e790 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3aea00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b013d7d0 0x7fc3b013e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3afe00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 0x7fc3b013c520 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800a8f0 tx=0x7fc3a8001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8045070 con 0x7fc3b013e790 2024-03-06T07:37:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a803a040 con 0x7fc3b013e790 2024-03-06T07:37:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8001f40 con 0x7fc3b013e790 2024-03-06T07:37:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.243+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b01608a0 con 0x7fc3b013e790 2024-03-06T07:37:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.244+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b0160e60 con 0x7fc3b013e790 2024-03-06T07:37:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.245+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b004f330 con 0x7fc3b013e790 2024-03-06T07:37:09.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.246+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3a8048070 con 0x7fc3b013e790 2024-03-06T07:37:09.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.246+0000 7fc3a7400640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc39006b4d0 0x7fc39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:09.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.247+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc3a80abc40 con 0x7fc3b013e790 2024-03-06T07:37:09.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.247+0000 7fc3af400640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc39006b4d0 0x7fc39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:09.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.248+0000 7fc3af400640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc39006b4d0 0x7fc39006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009a90 tx=0x7fc3a0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:09.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.251+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3a807a390 con 0x7fc3b013e790 2024-03-06T07:37:09.490 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.488+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3b00724b0 con 0x7fc3b013e790 2024-03-06T07:37:09.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.489+0000 7fc3a7400640 1 -- 172.21.15.6:0/3650241813 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fc3a807e430 con 0x7fc3b013e790 2024-03-06T07:37:09.491 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:09.492 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:09.492 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:09.494 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.492+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc39006b4d0 msgr2=0x7fc39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.492+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc39006b4d0 0x7fc39006d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009a90 tx=0x7fc3a0008040 comp rx=0 tx=0).stop 2024-03-06T07:37:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.492+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 msgr2=0x7fc3b013c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.492+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 0x7fc3b013c520 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fc3a800a8f0 tx=0x7fc3a8001cf0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 shutdown_connections 2024-03-06T07:37:09.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc39006b4d0 0x7fc39006d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc3b013e790 0x7fc3b013c520 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc3b013d7d0 0x7fc3b013e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 --2- 172.21.15.6:0/3650241813 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc3b015d5f0 0x7fc3b015ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:09.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 >> 172.21.15.6:0/3650241813 conn(0x7fc3b006c0c0 msgr2=0x7fc3b00b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:09.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 shutdown_connections 2024-03-06T07:37:09.496 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:09.493+0000 7fc3b4e00640 1 -- 172.21.15.6:0/3650241813 wait complete. 2024-03-06T07:37:09.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:09.716 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:10.122 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:10.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.938+0000 7f8907000640 1 Processor -- start 2024-03-06T07:37:10.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.938+0000 7f8907000640 1 -- start start 2024-03-06T07:37:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8907000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8900145300 0x7f89000b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8907000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8900070f10 0x7f89000cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8907000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89000714d0 con 0x7f89000cce90 2024-03-06T07:37:10.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8907000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89000cd300 con 0x7f8900070f10 2024-03-06T07:37:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8907000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89000717b0 con 0x7f8900145300 2024-03-06T07:37:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8905200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8900070f10 0x7f89000cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8906600640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f89000cce90 0x7f89000cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45160/0 (socket says 172.21.15.6:45160) 2024-03-06T07:37:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.940+0000 7f8906600640 1 -- 172.21.15.6:0/1033353596 learned_addr learned my addr 172.21.15.6:0/1033353596 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f8905c00640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8900145300 0x7f89000b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f8905200640 1 -- 172.21.15.6:0/1033353596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8900145300 msgr2=0x7f89000b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f8905200640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8900145300 0x7f89000b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f8905200640 1 -- 172.21.15.6:0/1033353596 >> v1:172.21.15.6:6789/0 conn(0x7f89000cce90 legacy=0x7f89000cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f8905200640 1 -- 172.21.15.6:0/1033353596 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900071a90 con 0x7f8900070f10 2024-03-06T07:37:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f88ffe00640 1 -- 172.21.15.6:0/1033353596 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356983760 0 0) 0x7f89000714d0 con 0x7f89000cce90 2024-03-06T07:37:10.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.941+0000 7f8905c00640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8900145300 0x7f89000b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.942+0000 7f8905200640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f89000cc950 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0097a0 tx=0x7f88ec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5cc4a6bf73b4aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.942+0000 7f88ffe00640 1 -- 172.21.15.6:0/1033353596 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec035020 con 0x7f8900070f10 2024-03-06T07:37:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.942+0000 7f88ffe00640 1 -- 172.21.15.6:0/1033353596 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ec031810 con 0x7f8900070f10 2024-03-06T07:37:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.942+0000 7f88ffe00640 1 -- 172.21.15.6:0/1033353596 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ec031be0 con 0x7f8900070f10 2024-03-06T07:37:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 -- 172.21.15.6:0/1033353596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 msgr2=0x7f89000cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f89000cc950 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0097a0 tx=0x7f88ec02b8b0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 -- 172.21.15.6:0/1033353596 shutdown_connections 2024-03-06T07:37:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f89000cc950 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 --2- 172.21.15.6:0/1033353596 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f8900145300 0x7f89000b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 -- 172.21.15.6:0/1033353596 >> 172.21.15.6:0/1033353596 conn(0x7f890006b3d0 msgr2=0x7f890006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 -- 172.21.15.6:0/1033353596 shutdown_connections 2024-03-06T07:37:10.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.943+0000 7f8907000640 1 -- 172.21.15.6:0/1033353596 wait complete. 2024-03-06T07:37:10.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.945+0000 7f8907000640 1 Processor -- start 2024-03-06T07:37:10.947 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.945+0000 7f8907000640 1 -- start start 2024-03-06T07:37:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8907000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f8900142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8907000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89000cce90 0x7f89001e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8907000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8900145300 0x7f89001e67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8907000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89001456e0 con 0x7f8900070f10 2024-03-06T07:37:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8907000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000b4ee0 con 0x7f8900145300 2024-03-06T07:37:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8907000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890015d580 con 0x7f89000cce90 2024-03-06T07:37:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8905c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f8900142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8905c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f8900142f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35280/0 (socket says 172.21.15.6:35280) 2024-03-06T07:37:10.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.946+0000 7f8905c00640 1 -- 172.21.15.6:0/721051046 learned_addr learned my addr 172.21.15.6:0/721051046 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8905c00640 1 -- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89000cce90 msgr2=0x7f89001e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8905c00640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89000cce90 0x7f89001e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8905c00640 1 -- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8900145300 msgr2=0x7f89001e67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8905c00640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8900145300 0x7f89001e67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8905c00640 1 -- 172.21.15.6:0/721051046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900142ad0 con 0x7f8900070f10 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8905c00640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f8900142f30 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f88f400c7f0 tx=0x7f88f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4015070 con 0x7f8900070f10 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f4007820 con 0x7f8900070f10 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4007bd0 con 0x7f8900070f10 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.947+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89001e4ad0 con 0x7f8900070f10 2024-03-06T07:37:10.950 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.948+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89001e5090 con 0x7f8900070f10 2024-03-06T07:37:10.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.949+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890004f330 con 0x7f8900070f10 2024-03-06T07:37:10.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.950+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88f4018070 con 0x7f8900070f10 2024-03-06T07:37:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.950+0000 7f88fd600640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f88e006b4d0 0x7f88e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:10.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.951+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f88f408a2e0 con 0x7f8900070f10 2024-03-06T07:37:10.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.951+0000 7f8905200640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f88e006b4d0 0x7f88e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:10.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.952+0000 7f8905200640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f88e006b4d0 0x7f88e006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f88ec036000 tx=0x7f88ec030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:10.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:10.960+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88f40591b0 con 0x7f8900070f10 2024-03-06T07:37:11.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.199+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f890013e9e0 con 0x7f8900070f10 2024-03-06T07:37:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.200+0000 7f88fd600640 1 -- 172.21.15.6:0/721051046 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f88f4058ab0 con 0x7f8900070f10 2024-03-06T07:37:11.202 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:11.202 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:11.202 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:11.204 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f88e006b4d0 msgr2=0x7f88e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f88e006b4d0 0x7f88e006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f88ec036000 tx=0x7f88ec030000 comp rx=0 tx=0).stop 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 msgr2=0x7f8900142f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f8900142f30 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f88f400c7f0 tx=0x7f88f400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 shutdown_connections 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f88e006b4d0 0x7f88e006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8900145300 0x7f89001e67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f89000cce90 0x7f89001e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 --2- 172.21.15.6:0/721051046 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8900070f10 0x7f8900142f30 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 >> 172.21.15.6:0/721051046 conn(0x7f890006b3d0 msgr2=0x7f89000adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.202+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 shutdown_connections 2024-03-06T07:37:11.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:11.203+0000 7f8907000640 1 -- 172.21.15.6:0/721051046 wait complete. 2024-03-06T07:37:11.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:11.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:12.127 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:12.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.645+0000 7fee6e200640 1 Processor -- start 2024-03-06T07:37:12.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.646+0000 7fee6e200640 1 -- start start 2024-03-06T07:37:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.647+0000 7fee6e200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fee680d4a10 0x7fee680d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.647+0000 7fee6e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680b3710 con 0x7fee680d0c80 2024-03-06T07:37:12.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.647+0000 7fee6e200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee680b39f0 con 0x7fee6815e0b0 2024-03-06T07:37:12.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.647+0000 7fee6e200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee680d10f0 con 0x7fee680d4a10 2024-03-06T07:37:12.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.647+0000 7fee6d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fee680d0c80 0x7fee680b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35668/0 (socket says 172.21.15.6:35668) 2024-03-06T07:37:12.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.647+0000 7fee6d800640 1 -- 172.21.15.6:0/398599694 learned_addr learned my addr 172.21.15.6:0/398599694 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67e00640 1 --2- 172.21.15.6:0/398599694 >> v2:172.21.15.178:3300/0 conn(0x7fee680d4a10 0x7fee680d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642523189 0 0) 0x7fee680b3710 con 0x7fee680d0c80 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c003520 con 0x7fee680d0c80 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339024244 0 0) 0x7fee4c003520 con 0x7fee680d0c80 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 >> v2:172.21.15.178:3300/0 conn(0x7fee680d4a10 msgr2=0x7fee680d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 --2- 172.21.15.6:0/398599694 >> v2:172.21.15.178:3300/0 conn(0x7fee680d4a10 0x7fee680d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 >> v1:172.21.15.178:6789/0 conn(0x7fee6815e0b0 legacy=0x7fee680cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680b3cd0 con 0x7fee680d0c80 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fee60001690 con 0x7fee680d0c80 2024-03-06T07:37:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.648+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 >> v1:172.21.15.6:6789/0 conn(0x7fee680d0c80 legacy=0x7fee680b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67400640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67400640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee4c004800 0x7fee4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67400640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee4c0051c0 0x7fee4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680d10f0 con 0x7fee4c0039b0 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c00c060 con 0x7fee4c004800 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c00c260 con 0x7fee4c0051c0 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6ce00640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee67e00640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee4c004800 0x7fee4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6d800640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee4c0051c0 0x7fee4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6ce00640 1 -- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee4c0051c0 msgr2=0x7fee4c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6ce00640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee4c0051c0 0x7fee4c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6ce00640 1 -- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee4c004800 msgr2=0x7fee4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6ce00640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee4c004800 0x7fee4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6ce00640 1 -- 172.21.15.6:0/398599694 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c003520 con 0x7fee4c0039b0 2024-03-06T07:37:12.653 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6d800640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee4c0051c0 0x7fee4c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:12.654 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.649+0000 7fee6d800640 1 -- 172.21.15.6:0/398599694 reap_dead start 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.651+0000 7fee6ce00640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee4c003dd0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fee58001e50 tx=0x7fee4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1f4d155b80b360 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.651+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58001a70 con 0x7fee4c0039b0 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.651+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58003b70 con 0x7fee4c0039b0 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.651+0000 7fee67400640 1 -- 172.21.15.6:0/398599694 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580050d0 con 0x7fee4c0039b0 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 -- 172.21.15.6:0/398599694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 msgr2=0x7fee4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee4c003dd0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fee58001e50 tx=0x7fee4c005830 comp rx=0 tx=0).stop 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 -- 172.21.15.6:0/398599694 shutdown_connections 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 --2- 172.21.15.6:0/398599694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee4c003dd0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 -- 172.21.15.6:0/398599694 >> 172.21.15.6:0/398599694 conn(0x7fee680d3d30 msgr2=0x7fee680d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 -- 172.21.15.6:0/398599694 shutdown_connections 2024-03-06T07:37:12.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.652+0000 7fee6e200640 1 -- 172.21.15.6:0/398599694 wait complete. 2024-03-06T07:37:12.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.654+0000 7fee6e200640 1 Processor -- start 2024-03-06T07:37:12.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.654+0000 7fee6e200640 1 -- start start 2024-03-06T07:37:12.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee680b9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6e200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee6814c370 0x7fee680c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6e200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee680cce10 0x7fee680cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.657 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6815e490 con 0x7fee4c0039b0 2024-03-06T07:37:12.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6e200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680cfc60 con 0x7fee680cce10 2024-03-06T07:37:12.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6e200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee680d0740 con 0x7fee6814c370 2024-03-06T07:37:12.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee680b9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6ce00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee680b9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46580/0 (socket says 172.21.15.6:46580) 2024-03-06T07:37:12.658 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.655+0000 7fee6ce00640 1 -- 172.21.15.6:0/23216172 learned_addr learned my addr 172.21.15.6:0/23216172 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee67e00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee6814c370 0x7fee680c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6d800640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee680cce10 0x7fee680cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6ce00640 1 -- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee6814c370 msgr2=0x7fee680c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6ce00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee6814c370 0x7fee680c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6ce00640 1 -- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee680cce10 msgr2=0x7fee680cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6ce00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee680cce10 0x7fee680cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6ce00640 1 -- 172.21.15.6:0/23216172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680b8180 con 0x7fee4c0039b0 2024-03-06T07:37:12.659 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee6ce00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee680b9390 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fee58005040 tx=0x7fee580014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.656+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58003970 con 0x7fee4c0039b0 2024-03-06T07:37:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.657+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58006b50 con 0x7fee4c0039b0 2024-03-06T07:37:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.657+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee680b88b0 con 0x7fee4c0039b0 2024-03-06T07:37:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.657+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58007030 con 0x7fee4c0039b0 2024-03-06T07:37:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.657+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee680c7bd0 con 0x7fee4c0039b0 2024-03-06T07:37:12.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.658+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6804f330 con 0x7fee4c0039b0 2024-03-06T07:37:12.661 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.659+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee580071d0 con 0x7fee4c0039b0 2024-03-06T07:37:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.660+0000 7fee64c00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee4806b4d0 0x7fee4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.660+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fee580abf10 con 0x7fee4c0039b0 2024-03-06T07:37:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.661+0000 7fee67e00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee4806b4d0 0x7fee4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:12.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.662+0000 7fee67e00640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee4806b4d0 0x7fee4806d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fee680cda30 tx=0x7fee54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:12.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.665+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fee58033020 con 0x7fee4c0039b0 2024-03-06T07:37:12.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.906+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee680cf580 con 0x7fee4c0039b0 2024-03-06T07:37:12.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.908+0000 7fee64c00640 1 -- 172.21.15.6:0/23216172 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fee5807acb0 con 0x7fee4c0039b0 2024-03-06T07:37:12.910 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:12.910 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:12.910 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:12.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.910+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee4806b4d0 msgr2=0x7fee4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.910+0000 7fee6e200640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee4806b4d0 0x7fee4806d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fee680cda30 tx=0x7fee54008040 comp rx=0 tx=0).stop 2024-03-06T07:37:12.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 msgr2=0x7fee680b9390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:12.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee680b9390 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fee58005040 tx=0x7fee580014a0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 shutdown_connections 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fee4806b4d0 0x7fee4806d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fee680cce10 0x7fee680cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fee6814c370 0x7fee680c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 --2- 172.21.15.6:0/23216172 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fee4c0039b0 0x7fee680b9390 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 >> 172.21.15.6:0/23216172 conn(0x7fee680d3d30 msgr2=0x7fee680ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 shutdown_connections 2024-03-06T07:37:12.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:12.911+0000 7fee6e200640 1 -- 172.21.15.6:0/23216172 wait complete. 2024-03-06T07:37:13.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:13.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:14.126 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:14.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.353+0000 7f2fab800640 1 Processor -- start 2024-03-06T07:37:14.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.354+0000 7f2fab800640 1 -- start start 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.355+0000 7f2fab800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2fa40af1a0 0x7f2fa415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.355+0000 7f2fab800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2fa40af580 0x7f2fa40b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fab800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa415cc40 con 0x7f2fa40cdc20 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fab800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa415cf20 con 0x7f2fa40af580 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fab800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa415d200 con 0x7f2fa40af1a0 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fa9800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2fa40cdc20 0x7f2fa40b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35674/0 (socket says 172.21.15.6:35674) 2024-03-06T07:37:14.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fa9800640 1 -- 172.21.15.6:0/3394621533 learned_addr learned my addr 172.21.15.6:0/3394621533 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fa3e00640 1 --2- 172.21.15.6:0/3394621533 >> v2:172.21.15.6:3300/0 conn(0x7f2fa40af580 0x7f2fa40b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fa8e00640 1 --2- 172.21.15.6:0/3394621533 >> v2:172.21.15.178:3300/0 conn(0x7f2fa40af1a0 0x7f2fa415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.356+0000 7f2fa3e00640 1 -- 172.21.15.6:0/3394621533 >> v2:172.21.15.178:3300/0 conn(0x7f2fa40af1a0 msgr2=0x7f2fa415c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.357+0000 7f2fa3e00640 1 --2- 172.21.15.6:0/3394621533 >> v2:172.21.15.178:3300/0 conn(0x7f2fa40af1a0 0x7f2fa415c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.357+0000 7f2fa3e00640 1 -- 172.21.15.6:0/3394621533 >> v1:172.21.15.6:6789/0 conn(0x7f2fa40cdc20 legacy=0x7f2fa40b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.357+0000 7f2fa3e00640 1 -- 172.21.15.6:0/3394621533 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa415d4e0 con 0x7f2fa40af580 2024-03-06T07:37:14.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.357+0000 7f2fa3400640 1 -- 172.21.15.6:0/3394621533 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551524886 0 0) 0x7f2fa415cc40 con 0x7f2fa40cdc20 2024-03-06T07:37:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.358+0000 7f2fa3e00640 1 --2- 172.21.15.6:0/3394621533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40af580 0x7f2fa40b4c20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4074960 tx=0x7f2f9002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894be579865fd9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.358+0000 7f2fa3400640 1 -- 172.21.15.6:0/3394621533 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f900315a0 con 0x7f2fa40af580 2024-03-06T07:37:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.358+0000 7f2fa3400640 1 -- 172.21.15.6:0/3394621533 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f90037070 con 0x7f2fa40af580 2024-03-06T07:37:14.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.358+0000 7f2fa3400640 1 -- 172.21.15.6:0/3394621533 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f90031960 con 0x7f2fa40af580 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.358+0000 7f2fab800640 1 -- 172.21.15.6:0/3394621533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40af580 msgr2=0x7f2fa40b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.358+0000 7f2fab800640 1 --2- 172.21.15.6:0/3394621533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40af580 0x7f2fa40b4c20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4074960 tx=0x7f2f9002b840 comp rx=0 tx=0).stop 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.359+0000 7f2fab800640 1 -- 172.21.15.6:0/3394621533 shutdown_connections 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.359+0000 7f2fab800640 1 --2- 172.21.15.6:0/3394621533 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40af580 0x7f2fa40b4c20 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.359+0000 7f2fab800640 1 --2- 172.21.15.6:0/3394621533 >> v2:172.21.15.178:3300/0 conn(0x7f2fa40af1a0 0x7f2fa415c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.359+0000 7f2fab800640 1 -- 172.21.15.6:0/3394621533 >> 172.21.15.6:0/3394621533 conn(0x7f2fa406afb0 msgr2=0x7f2fa406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.359+0000 7f2fab800640 1 -- 172.21.15.6:0/3394621533 shutdown_connections 2024-03-06T07:37:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.359+0000 7f2fab800640 1 -- 172.21.15.6:0/3394621533 wait complete. 2024-03-06T07:37:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.361+0000 7f2fab800640 1 Processor -- start 2024-03-06T07:37:14.363 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.361+0000 7f2fab800640 1 -- start start 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fab800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fa40af1a0 0x7f2fa41f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fab800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fa40af580 0x7f2fa40cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fab800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 0x7f2fa406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fab800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40b3660 con 0x7f2fa40cdc20 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fab800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40b34e0 con 0x7f2fa40af580 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fab800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40b37e0 con 0x7f2fa40af1a0 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 0x7f2fa406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 0x7f2fa406e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46598/0 (socket says 172.21.15.6:46598) 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 -- 172.21.15.6:0/3996475828 learned_addr learned my addr 172.21.15.6:0/3996475828 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa8e00640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fa40af1a0 0x7f2fa41f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa3e00640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fa40af580 0x7f2fa40cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 -- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fa40af1a0 msgr2=0x7f2fa41f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.365 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fa40af1a0 0x7f2fa41f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 -- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fa40af580 msgr2=0x7f2fa40cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fa40af580 0x7f2fa40cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa9800640 1 -- 172.21.15.6:0/3996475828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa41f4020 con 0x7f2fa40cdc20 2024-03-06T07:37:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.362+0000 7f2fa3e00640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fa40af580 0x7f2fa40cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:14.366 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.363+0000 7f2fa9800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 0x7f2fa406e500 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00c780 tx=0x7f2f9c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.363+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c018070 con 0x7f2fa40cdc20 2024-03-06T07:37:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.363+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c00a640 con 0x7f2fa40cdc20 2024-03-06T07:37:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.363+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c00aa30 con 0x7f2fa40cdc20 2024-03-06T07:37:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.363+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa41f4300 con 0x7f2fa40cdc20 2024-03-06T07:37:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.363+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa41f48c0 con 0x7f2fa40cdc20 2024-03-06T07:37:14.367 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.364+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa404f3a0 con 0x7f2fa40cdc20 2024-03-06T07:37:14.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.366+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f9c01b070 con 0x7f2fa40cdc20 2024-03-06T07:37:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.367+0000 7f2fa0c00640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f8406b500 0x7f2f8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:14.371 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.367+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2f9c0897d0 con 0x7f2fa40cdc20 2024-03-06T07:37:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.368+0000 7f2fa8e00640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f8406b500 0x7f2f8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:14.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.369+0000 7f2fa8e00640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f8406b500 0x7f2f8406d9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2f98006fd0 tx=0x7f2f98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:14.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.372+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f9c058610 con 0x7f2fa40cdc20 2024-03-06T07:37:14.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.612+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fa40cb380 con 0x7f2fa40cdc20 2024-03-06T07:37:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.613+0000 7f2fa0c00640 1 -- 172.21.15.6:0/3996475828 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f2f9c057f10 con 0x7f2fa40cdc20 2024-03-06T07:37:14.615 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:14.615 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:14.615 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.615+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f8406b500 msgr2=0x7f2f8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.615+0000 7f2fab800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f8406b500 0x7f2f8406d9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2f98006fd0 tx=0x7f2f98008040 comp rx=0 tx=0).stop 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.615+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 msgr2=0x7f2fa406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.615+0000 7f2fab800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 0x7f2fa406e500 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00c780 tx=0x7f2f9c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.615+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 shutdown_connections 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.615+0000 7f2fab800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2f8406b500 0x7f2f8406d9c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.616+0000 7f2fab800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2fa40af580 0x7f2fa40cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.616+0000 7f2fab800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2fa40cdc20 0x7f2fa406e500 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.616+0000 7f2fab800640 1 --2- 172.21.15.6:0/3996475828 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2fa40af1a0 0x7f2fa41f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.616+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 >> 172.21.15.6:0/3996475828 conn(0x7f2fa406afb0 msgr2=0x7f2fa415aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.616+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 shutdown_connections 2024-03-06T07:37:14.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:14.616+0000 7f2fab800640 1 -- 172.21.15.6:0/3996475828 wait complete. 2024-03-06T07:37:14.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:14.837 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:15.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:16.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.065+0000 7fea1b200640 1 Processor -- start 2024-03-06T07:37:16.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.066+0000 7fea1b200640 1 -- start start 2024-03-06T07:37:16.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.067+0000 7fea1b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 0x7fea140d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.067+0000 7fea1b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea140d2dc0 con 0x7fea1415e070 2024-03-06T07:37:16.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.067+0000 7fea1b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea140d18c0 con 0x7fea140d0b00 2024-03-06T07:37:16.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.067+0000 7fea1b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea140d1ba0 con 0x7fea140d2960 2024-03-06T07:37:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.068+0000 7fea19e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fea1415e070 0x7fea140cf410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35676/0 (socket says 172.21.15.6:35676) 2024-03-06T07:37:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.068+0000 7fea19e00640 1 -- 172.21.15.6:0/2854317993 learned_addr learned my addr 172.21.15.6:0/2854317993 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.068+0000 7fea19400640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 0x7fea140d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.068+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650455352 0 0) 0x7fea140d2dc0 con 0x7fea1415e070 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.068+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9fc003520 con 0x7fea1415e070 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776945774 0 0) 0x7fea140d18c0 con 0x7fea140d0b00 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea140d2dc0 con 0x7fea140d0b00 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687155335 0 0) 0x7fe9fc003520 con 0x7fea1415e070 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 msgr2=0x7fea140d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 0x7fea140d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 >> v1:172.21.15.178:6789/0 conn(0x7fea140d0b00 legacy=0x7fea140b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea140d1e80 con 0x7fea1415e070 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fea080015f0 con 0x7fea1415e070 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 >> v1:172.21.15.6:6789/0 conn(0x7fea1415e070 legacy=0x7fea140cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9fc003b70 0x7fe9fc003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9fc004800 0x7fe9fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 0x7fe9fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140d1ba0 con 0x7fe9fc004800 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc00ba70 con 0x7fe9fc003b70 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc00bc70 con 0x7fea140d2960 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.069+0000 7fea19400640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9fc004800 0x7fe9fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19e00640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9fc003b70 0x7fe9fc003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea1a800640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 0x7fe9fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19400640 1 -- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 msgr2=0x7fe9fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19400640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fea140d2960 0x7fe9fc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19400640 1 -- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9fc003b70 msgr2=0x7fe9fc003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19400640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9fc003b70 0x7fe9fc003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19400640 1 -- 172.21.15.6:0/2854317993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc003520 con 0x7fe9fc004800 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.070+0000 7fea19e00640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe9fc003b70 0x7fe9fc003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.071+0000 7fea19400640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9fc004800 0x7fe9fc004c80 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fea04009ac0 tx=0x7fe9fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a884a50696d09a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.071+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04039070 con 0x7fe9fc004800 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.071+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0402c6b0 con 0x7fe9fc004800 2024-03-06T07:37:16.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.071+0000 7fea18a00640 1 -- 172.21.15.6:0/2854317993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea04009690 con 0x7fe9fc004800 2024-03-06T07:37:16.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1b200640 1 -- 172.21.15.6:0/2854317993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9fc004800 msgr2=0x7fe9fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1b200640 1 --2- 172.21.15.6:0/2854317993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe9fc004800 0x7fe9fc004c80 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fea04009ac0 tx=0x7fe9fc005830 comp rx=0 tx=0).stop 2024-03-06T07:37:16.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1a800640 1 -- 172.21.15.6:0/2854317993 reap_dead start 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1b200640 1 -- 172.21.15.6:0/2854317993 shutdown_connections 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1b200640 1 -- 172.21.15.6:0/2854317993 >> 172.21.15.6:0/2854317993 conn(0x7fea1408d2a0 msgr2=0x7fea140d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1b200640 1 -- 172.21.15.6:0/2854317993 shutdown_connections 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.072+0000 7fea1b200640 1 -- 172.21.15.6:0/2854317993 wait complete. 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.074+0000 7fea1b200640 1 Processor -- start 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.074+0000 7fea1b200640 1 -- start start 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.075+0000 7fea1b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9fc004800 0x7fea140c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.075+0000 7fea1b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea140c77e0 0x7fea14160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.075+0000 7fea1b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 0x7fea141f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.075+0000 7fea1b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140d45b0 con 0x7fea141f58c0 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.075+0000 7fea1b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea140d2440 con 0x7fea140c77e0 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.075+0000 7fea1b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea140d4730 con 0x7fe9fc004800 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 0x7fea141f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 0x7fea141f5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46612/0 (socket says 172.21.15.6:46612) 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 -- 172.21.15.6:0/1557836845 learned_addr learned my addr 172.21.15.6:0/1557836845 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea19e00640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9fc004800 0x7fea140c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea19400640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea140c77e0 0x7fea14160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 -- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9fc004800 msgr2=0x7fea140c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9fc004800 0x7fea140c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 -- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea140c77e0 msgr2=0x7fea14160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea140c77e0 0x7fea14160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea1a800640 1 -- 172.21.15.6:0/1557836845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea140c7ca0 con 0x7fea141f58c0 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.076+0000 7fea19e00640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9fc004800 0x7fea140c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.077+0000 7fea1a800640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 0x7fea141f5d70 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fea1408cf40 tx=0x7fea1000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.077+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10018070 con 0x7fea141f58c0 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.077+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea10003b70 con 0x7fea141f58c0 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.077+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10005070 con 0x7fea141f58c0 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.077+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea140c7f80 con 0x7fea141f58c0 2024-03-06T07:37:16.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.077+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea140c8540 con 0x7fea141f58c0 2024-03-06T07:37:16.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.078+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea140d1150 con 0x7fea141f58c0 2024-03-06T07:37:16.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.080+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea1001b070 con 0x7fea141f58c0 2024-03-06T07:37:16.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.081+0000 7fea0e000640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe9f806b4d0 0x7fe9f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:16.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.081+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea100899d0 con 0x7fea141f58c0 2024-03-06T07:37:16.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.082+0000 7fea19e00640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe9f806b4d0 0x7fe9f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:16.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.083+0000 7fea19e00640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe9f806b4d0 0x7fe9f806d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fea08001eb0 tx=0x7fea08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:16.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.085+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea140d1150 con 0x7fea141f58c0 2024-03-06T07:37:16.128 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:16.397 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.394+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea1404f330 con 0x7fea141f58c0 2024-03-06T07:37:16.398 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.396+0000 7fea0e000640 1 -- 172.21.15.6:0/1557836845 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fea10058120 con 0x7fea141f58c0 2024-03-06T07:37:16.398 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:16.398 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:16.398 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe9f806b4d0 msgr2=0x7fe9f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe9f806b4d0 0x7fe9f806d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fea08001eb0 tx=0x7fea08009040 comp rx=0 tx=0).stop 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 msgr2=0x7fea141f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 0x7fea141f5d70 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fea1408cf40 tx=0x7fea1000dc00 comp rx=0 tx=0).stop 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 shutdown_connections 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe9f806b4d0 0x7fe9f806d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fea140c77e0 0x7fea14160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fea141f58c0 0x7fea141f5d70 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 --2- 172.21.15.6:0/1557836845 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe9fc004800 0x7fea140c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.398+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 >> 172.21.15.6:0/1557836845 conn(0x7fea1408d2a0 msgr2=0x7fea1415d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.399+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 shutdown_connections 2024-03-06T07:37:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:16.399+0000 7fea1b200640 1 -- 172.21.15.6:0/1557836845 wait complete. 2024-03-06T07:37:16.621 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:16.622 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:17.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:17.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.849+0000 7f2adbe00640 1 Processor -- start 2024-03-06T07:37:17.851 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.849+0000 7f2adbe00640 1 -- start start 2024-03-06T07:37:17.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.850+0000 7f2adbe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f2adc0700c0 0x7f2adc070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.850+0000 7f2adbe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc0cd0c0 con 0x7f2adc0d04b0 2024-03-06T07:37:17.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.850+0000 7f2adbe00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2adc0cd3a0 con 0x7f2adc15f450 2024-03-06T07:37:17.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.850+0000 7f2adbe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2adc0d29e0 con 0x7f2adc0700c0 2024-03-06T07:37:17.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.850+0000 7f2ada000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2adc0d04b0 0x7f2adc0cde80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35690/0 (socket says 172.21.15.6:35690) 2024-03-06T07:37:17.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.850+0000 7f2ada000640 1 -- 172.21.15.6:0/1523114816 learned_addr learned my addr 172.21.15.6:0/1523114816 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2adb400640 1 --2- 172.21.15.6:0/1523114816 >> v2:172.21.15.178:3300/0 conn(0x7f2adc0700c0 0x7f2adc070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100521366 0 0) 0x7f2adc0cd0c0 con 0x7f2adc0d04b0 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac0003520 con 0x7f2adc0d04b0 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2180410649 0 0) 0x7f2ac0003520 con 0x7f2adc0d04b0 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 >> v2:172.21.15.178:3300/0 conn(0x7f2adc0700c0 msgr2=0x7f2adc070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 --2- 172.21.15.6:0/1523114816 >> v2:172.21.15.178:3300/0 conn(0x7f2adc0700c0 0x7f2adc070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 >> v1:172.21.15.178:6789/0 conn(0x7f2adc15f450 legacy=0x7f2adc0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc0cd680 con 0x7f2adc0d04b0 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2acc0019f0 con 0x7f2adc0d04b0 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 >> v1:172.21.15.6:6789/0 conn(0x7f2adc0d04b0 legacy=0x7f2adc0cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2ac0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.855 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.851+0000 7f2ad9600640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ac0004800 0x7f2ac0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2ad9600640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ac00051c0 0x7f2ac000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0d29e0 con 0x7f2ac0003a00 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac000c0c0 con 0x7f2ac00051c0 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac000c2c0 con 0x7f2ac0004800 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adaa00640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2ac0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adb400640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ac00051c0 0x7f2ac000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2ada000640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ac0004800 0x7f2ac0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adaa00640 1 -- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ac0004800 msgr2=0x7f2ac0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adaa00640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2ac0004800 0x7f2ac0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adaa00640 1 -- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ac00051c0 msgr2=0x7f2ac000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adaa00640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2ac00051c0 0x7f2ac000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adaa00640 1 -- 172.21.15.6:0/1523114816 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0003520 con 0x7f2ac0003a00 2024-03-06T07:37:17.856 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.852+0000 7f2adb400640 1 -- 172.21.15.6:0/1523114816 reap_dead start 2024-03-06T07:37:17.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2adaa00640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2ac0003e20 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4001750 tx=0x7f2ac0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5d45280d5e827 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.857 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4001370 con 0x7f2ac0003a00 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac40052d0 con 0x7f2ac0003a00 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2ad9600640 1 -- 172.21.15.6:0/1523114816 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4003830 con 0x7f2ac0003a00 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2adbe00640 1 -- 172.21.15.6:0/1523114816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 msgr2=0x7f2ac0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2adbe00640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2ac0003e20 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4001750 tx=0x7f2ac0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2adbe00640 1 -- 172.21.15.6:0/1523114816 shutdown_connections 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2adbe00640 1 --2- 172.21.15.6:0/1523114816 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2ac0003e20 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.854+0000 7f2adbe00640 1 -- 172.21.15.6:0/1523114816 >> 172.21.15.6:0/1523114816 conn(0x7f2adc0d1ae0 msgr2=0x7f2adc0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.855+0000 7f2adbe00640 1 -- 172.21.15.6:0/1523114816 shutdown_connections 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.855+0000 7f2adbe00640 1 -- 172.21.15.6:0/1523114816 wait complete. 2024-03-06T07:37:17.858 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.856+0000 7f2adbe00640 1 Processor -- start 2024-03-06T07:37:17.859 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.856+0000 7f2adbe00640 1 -- start start 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.857+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2adc152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.857+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2adc0af990 0x7f2adc0afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.857+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2adc0c7960 0x7f2adc0c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.857+0000 7f2adbe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc15f830 con 0x7f2ac0003a00 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.857+0000 7f2adbe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc0ae8b0 con 0x7f2adc0af990 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.857+0000 7f2adbe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2adc15f9b0 con 0x7f2adc0c7960 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2adc152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2adc152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46634/0 (socket says 172.21.15.6:46634) 2024-03-06T07:37:17.860 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 -- 172.21.15.6:0/4283094930 learned_addr learned my addr 172.21.15.6:0/4283094930 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:17.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 -- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2adc0c7960 msgr2=0x7f2adc0c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adb400640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2adc0c7960 0x7f2adc0c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2ada000640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2adc0af990 0x7f2adc0afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2adc0c7960 0x7f2adc0c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 -- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2adc0af990 msgr2=0x7f2adc0afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2adc0af990 0x7f2adc0afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 -- 172.21.15.6:0/4283094930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc0c9eb0 con 0x7f2ac0003a00 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.858+0000 7f2adaa00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2adc152210 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4034040 tx=0x7f2ac4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.859+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4041070 con 0x7f2ac0003a00 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.859+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac4005480 con 0x7f2ac0003a00 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.859+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2adc0c8320 con 0x7f2ac0003a00 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.859+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4007a30 con 0x7f2ac0003a00 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.859+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2adc1f74a0 con 0x7f2ac0003a00 2024-03-06T07:37:17.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.860+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2adc0b0350 con 0x7f2ac0003a00 2024-03-06T07:37:17.863 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.861+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ac4044070 con 0x7f2ac0003a00 2024-03-06T07:37:17.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.863+0000 7f2ad2a00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ab806b4d0 0x7f2ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:17.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.863+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ac40abcf0 con 0x7f2ac0003a00 2024-03-06T07:37:17.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.864+0000 7f2ada000640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ab806b4d0 0x7f2ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:17.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.865+0000 7f2ada000640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ab806b4d0 0x7f2ab806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2acc001330 tx=0x7f2acc0014f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:17.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:17.867+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2adc0c8320 con 0x7f2ac0003a00 2024-03-06T07:37:18.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.110+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2adc0c8500 con 0x7f2ac0003a00 2024-03-06T07:37:18.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.111+0000 7f2ad2a00640 1 -- 172.21.15.6:0/4283094930 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f2adc0c9eb0 con 0x7f2ac0003a00 2024-03-06T07:37:18.113 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:18.114 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:18.114 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:18.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ab806b4d0 msgr2=0x7f2ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ab806b4d0 0x7f2ab806d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2acc001330 tx=0x7f2acc0014f0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 msgr2=0x7f2adc152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2adc152210 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4034040 tx=0x7f2ac4001120 comp rx=0 tx=0).stop 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 shutdown_connections 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f2ab806b4d0 0x7f2ab806d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2adc0af990 0x7f2adc0afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f2adc0c7960 0x7f2adc0c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 --2- 172.21.15.6:0/4283094930 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2ac0003a00 0x7f2adc152210 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 >> 172.21.15.6:0/4283094930 conn(0x7f2adc0d1ae0 msgr2=0x7f2adc0d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 shutdown_connections 2024-03-06T07:37:18.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:18.114+0000 7f2adbe00640 1 -- 172.21.15.6:0/4283094930 wait complete. 2024-03-06T07:37:18.129 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:18.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:18.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:19.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.572+0000 7fbfd1400640 1 Processor -- start 2024-03-06T07:37:19.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.572+0000 7fbfd1400640 1 -- start start 2024-03-06T07:37:19.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfd1400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc15dec0 0x7fbfcc0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfd1400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfcc0cf460 0x7fbfcc0b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfd1400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfcc0d1120 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfd1400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfcc0b0ea0 con 0x7fbfcc0cf460 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfd1400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc0d1400 con 0x7fbfcc15dec0 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfd0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfcc0cf460 0x7fbfcc0b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.574+0000 7fbfcbe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc15dec0 0x7fbfcc0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.575+0000 7fbfd0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbfcc0cf460 0x7fbfcc0b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46636/0 (socket says 172.21.15.6:46636) 2024-03-06T07:37:19.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.575+0000 7fbfd0a00640 1 -- 172.21.15.6:0/3826289070 learned_addr learned my addr 172.21.15.6:0/3826289070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:19.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.575+0000 7fbfd0a00640 1 -- 172.21.15.6:0/3826289070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc15dec0 msgr2=0x7fbfcc0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:19.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.575+0000 7fbfd0a00640 1 --2- 172.21.15.6:0/3826289070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc15dec0 0x7fbfcc0ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.575+0000 7fbfd0a00640 1 -- 172.21.15.6:0/3826289070 >> v1:172.21.15.178:6789/0 conn(0x7fbfcc15e2a0 legacy=0x7fbfcc0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.575+0000 7fbfd0a00640 1 -- 172.21.15.6:0/3826289070 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc0b0b30 con 0x7fbfcc0cf460 2024-03-06T07:37:19.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.578+0000 7fbfd0a00640 1 --2- 172.21.15.6:0/3826289070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc0cf460 0x7fbfcc0b1330 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009e50 tx=0x7fbfc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa84b797d453ad0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.578+0000 7fbfcaa00640 1 -- 172.21.15.6:0/3826289070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc402cb00 con 0x7fbfcc0cf460 2024-03-06T07:37:19.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.578+0000 7fbfcaa00640 1 -- 172.21.15.6:0/3826289070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc402cca0 con 0x7fbfcc0cf460 2024-03-06T07:37:19.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.578+0000 7fbfcaa00640 1 -- 172.21.15.6:0/3826289070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4034610 con 0x7fbfcc0cf460 2024-03-06T07:37:19.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 -- 172.21.15.6:0/3826289070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc0cf460 msgr2=0x7fbfcc0b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 --2- 172.21.15.6:0/3826289070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc0cf460 0x7fbfcc0b1330 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009e50 tx=0x7fbfc402bf60 comp rx=0 tx=0).stop 2024-03-06T07:37:19.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 -- 172.21.15.6:0/3826289070 shutdown_connections 2024-03-06T07:37:19.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 --2- 172.21.15.6:0/3826289070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc0cf460 0x7fbfcc0b1330 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 --2- 172.21.15.6:0/3826289070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc15dec0 0x7fbfcc0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 -- 172.21.15.6:0/3826289070 >> 172.21.15.6:0/3826289070 conn(0x7fbfcc0d3d40 msgr2=0x7fbfcc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:19.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 -- 172.21.15.6:0/3826289070 shutdown_connections 2024-03-06T07:37:19.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.579+0000 7fbfd1400640 1 -- 172.21.15.6:0/3826289070 wait complete. 2024-03-06T07:37:19.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.581+0000 7fbfd1400640 1 Processor -- start 2024-03-06T07:37:19.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.581+0000 7fbfd1400640 1 -- start start 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd1400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc0cf460 0x7fbfcc1593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd1400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfcc15dec0 0x7fbfcc0c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd1400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 0x7fbfcc0b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd1400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc0d0ec0 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd1400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc0d0d40 con 0x7fbfcc15dec0 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd1400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfcc0b1870 con 0x7fbfcc0cf460 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 0x7fbfcc0b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 0x7fbfcc0b8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46644/0 (socket says 172.21.15.6:46644) 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.582+0000 7fbfd0a00640 1 -- 172.21.15.6:0/219344583 learned_addr learned my addr 172.21.15.6:0/219344583 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc0cf460 0x7fbfcc1593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfcb400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfcc15dec0 0x7fbfcc0c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfd0a00640 1 -- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc0cf460 msgr2=0x7fbfcc1593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfd0a00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc0cf460 0x7fbfcc1593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfd0a00640 1 -- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfcc15dec0 msgr2=0x7fbfcc0c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfd0a00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfcc15dec0 0x7fbfcc0c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfd0a00640 1 -- 172.21.15.6:0/219344583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfcc0b7750 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfcb400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfcc15dec0 0x7fbfcc0c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:19.587 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.583+0000 7fbfd0a00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 0x7fbfcc0b8bf0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009e50 tx=0x7fbfc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.584+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4043020 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.584+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc402ce50 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.584+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4009510 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.584+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfcc0b6b40 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.584+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfcc0b7100 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.585+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfcc04f330 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.586+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfc4046020 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.587+0000 7fbfc3e00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfac06b4d0 0x7fbfac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:19.589 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.587+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbfc40ac840 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.590 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.588+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfac06b4d0 0x7fbfac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:19.591 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.589+0000 7fbfcbe00640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfac06b4d0 0x7fbfac06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0b5030 tx=0x7fbfbc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:19.594 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.592+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfc404d030 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.845+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfcc06d2a0 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.846+0000 7fbfc3e00640 1 -- 172.21.15.6:0/219344583 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fbfc407b830 con 0x7fbfcc15e2a0 2024-03-06T07:37:19.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:19.850 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfac06b4d0 msgr2=0x7fbfac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfac06b4d0 0x7fbfac06d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc0b5030 tx=0x7fbfbc008040 comp rx=0 tx=0).stop 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 msgr2=0x7fbfcc0b8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 0x7fbfcc0b8bf0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4009e50 tx=0x7fbfc4004a40 comp rx=0 tx=0).stop 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 shutdown_connections 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbfac06b4d0 0x7fbfac06d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbfcc15dec0 0x7fbfcc0c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbfcc15e2a0 0x7fbfcc0b8bf0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 --2- 172.21.15.6:0/219344583 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbfcc0cf460 0x7fbfcc1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 >> 172.21.15.6:0/219344583 conn(0x7fbfcc0d3d40 msgr2=0x7fbfcc06c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.849+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 shutdown_connections 2024-03-06T07:37:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:19.850+0000 7fbfd1400640 1 -- 172.21.15.6:0/219344583 wait complete. 2024-03-06T07:37:20.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:20.080 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:20.132 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:21.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.303+0000 7faf83000640 1 Processor -- start 2024-03-06T07:37:21.305 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.303+0000 7faf83000640 1 -- start start 2024-03-06T07:37:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf83000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 0x7faf7c145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf83000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7faf7c06b6c0 0x7faf7c0cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf83000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c0d0910 con 0x7faf7c1457c0 2024-03-06T07:37:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf83000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0d0bf0 con 0x7faf7c06b6c0 2024-03-06T07:37:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf83000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0d0ed0 con 0x7faf7c144c00 2024-03-06T07:37:21.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf81200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7faf7c1457c0 0x7faf7c06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35692/0 (socket says 172.21.15.6:35692) 2024-03-06T07:37:21.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.305+0000 7faf81200640 1 -- 172.21.15.6:0/947321644 learned_addr learned my addr 172.21.15.6:0/947321644 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf81c00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 0x7faf7c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf82600640 1 --2- 172.21.15.6:0/947321644 >> v2:172.21.15.178:3300/0 conn(0x7faf7c06b6c0 0x7faf7c0cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656389632 0 0) 0x7faf7c0d0910 con 0x7faf7c1457c0 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf64003520 con 0x7faf7c1457c0 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756789577 0 0) 0x7faf64003520 con 0x7faf7c1457c0 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 msgr2=0x7faf7c145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 0x7faf7c145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 >> v2:172.21.15.178:3300/0 conn(0x7faf7c06b6c0 msgr2=0x7faf7c0cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 --2- 172.21.15.6:0/947321644 >> v2:172.21.15.178:3300/0 conn(0x7faf7c06b6c0 0x7faf7c0cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0d11b0 con 0x7faf7c1457c0 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7faf68001670 con 0x7faf7c1457c0 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 >> v1:172.21.15.6:6789/0 conn(0x7faf7c1457c0 legacy=0x7faf7c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf640039b0 0x7faf64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 0x7faf64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf64004d40 0x7faf640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0d0bf0 con 0x7faf640039b0 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0d0ed0 con 0x7faf64004d40 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.306+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6400b9c0 con 0x7faf7c144c00 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81c00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf640039b0 0x7faf64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81200640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 0x7faf64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf82600640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf64004d40 0x7faf640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81c00640 1 -- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 msgr2=0x7faf64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81c00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c144c00 0x7faf64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81c00640 1 -- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf64004d40 msgr2=0x7faf640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81c00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf64004d40 0x7faf640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf81c00640 1 -- 172.21.15.6:0/947321644 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf64003520 con 0x7faf640039b0 2024-03-06T07:37:21.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.307+0000 7faf82600640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf64004d40 0x7faf640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.308+0000 7faf81c00640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf640039b0 0x7faf64003dd0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7faf70009ac0 tx=0x7faf64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a077cb75b38d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70039070 con 0x7faf640039b0 2024-03-06T07:37:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7002c6b0 con 0x7faf640039b0 2024-03-06T07:37:21.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf7be00640 1 -- 172.21.15.6:0/947321644 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70009690 con 0x7faf640039b0 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf83000640 1 -- 172.21.15.6:0/947321644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf640039b0 msgr2=0x7faf64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf83000640 1 --2- 172.21.15.6:0/947321644 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf640039b0 0x7faf64003dd0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7faf70009ac0 tx=0x7faf64005830 comp rx=0 tx=0).stop 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf82600640 1 -- 172.21.15.6:0/947321644 reap_dead start 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf83000640 1 -- 172.21.15.6:0/947321644 shutdown_connections 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf83000640 1 -- 172.21.15.6:0/947321644 >> 172.21.15.6:0/947321644 conn(0x7faf7c06c0c0 msgr2=0x7faf7c06c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.309+0000 7faf83000640 1 -- 172.21.15.6:0/947321644 shutdown_connections 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.310+0000 7faf83000640 1 -- 172.21.15.6:0/947321644 wait complete. 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.311+0000 7faf83000640 1 Processor -- start 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.311+0000 7faf83000640 1 -- start start 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.312+0000 7faf83000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf640039b0 0x7faf7c140770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.312+0000 7faf83000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 0x7faf7c1f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.312+0000 7faf83000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c1f7420 0x7faf7c1f78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.312+0000 7faf83000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0d19b0 con 0x7faf7c157f30 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.312+0000 7faf83000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c144fe0 con 0x7faf640039b0 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.312+0000 7faf83000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c06f820 con 0x7faf7c1f7420 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 0x7faf7c1f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 0x7faf7c1f6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46648/0 (socket says 172.21.15.6:46648) 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 -- 172.21.15.6:0/3801330820 learned_addr learned my addr 172.21.15.6:0/3801330820 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf82600640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c1f7420 0x7faf7c1f78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81c00640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf640039b0 0x7faf7c140770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 -- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c1f7420 msgr2=0x7faf7c1f78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c1f7420 0x7faf7c1f78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 -- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf640039b0 msgr2=0x7faf7c140770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf640039b0 0x7faf7c140770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 -- 172.21.15.6:0/3801330820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0b48c0 con 0x7faf7c157f30 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81c00640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf640039b0 0x7faf7c140770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.313+0000 7faf81200640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 0x7faf7c1f6ee0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7faf6800e680 tx=0x7faf6800eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.314+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6800c6c0 con 0x7faf7c157f30 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.314+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf68018070 con 0x7faf7c157f30 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.314+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c0b4ba0 con 0x7faf7c157f30 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.314+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6800caa0 con 0x7faf7c157f30 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.314+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c1583b0 con 0x7faf7c157f30 2024-03-06T07:37:21.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.315+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c04f330 con 0x7faf7c157f30 2024-03-06T07:37:21.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.316+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf6800cc40 con 0x7faf7c157f30 2024-03-06T07:37:21.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.317+0000 7faf79600640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5c06b500 0x7faf5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:21.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.317+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faf6808a0d0 con 0x7faf7c157f30 2024-03-06T07:37:21.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.318+0000 7faf81c00640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5c06b500 0x7faf5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:21.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.319+0000 7faf81c00640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5c06b500 0x7faf5c06d9c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7faf70036000 tx=0x7faf70030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:21.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.322+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf68058f00 con 0x7faf7c157f30 2024-03-06T07:37:21.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.561+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7c0722f0 con 0x7faf7c157f30 2024-03-06T07:37:21.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.562+0000 7faf79600640 1 -- 172.21.15.6:0/3801330820 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7faf68058800 con 0x7faf7c157f30 2024-03-06T07:37:21.564 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:21.565 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:21.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5c06b500 msgr2=0x7faf5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5c06b500 0x7faf5c06d9c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7faf70036000 tx=0x7faf70030000 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 msgr2=0x7faf7c1f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 0x7faf7c1f6ee0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7faf6800e680 tx=0x7faf6800eb40 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 shutdown_connections 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7faf5c06b500 0x7faf5c06d9c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7faf7c1f7420 0x7faf7c1f78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faf7c157f30 0x7faf7c1f6ee0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 --2- 172.21.15.6:0/3801330820 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faf640039b0 0x7faf7c140770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 >> 172.21.15.6:0/3801330820 conn(0x7faf7c06c0c0 msgr2=0x7faf7c0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 shutdown_connections 2024-03-06T07:37:21.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:21.565+0000 7faf83000640 1 -- 172.21.15.6:0/3801330820 wait complete. 2024-03-06T07:37:21.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:21.787 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:22.134 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:22.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:23.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.036+0000 7f095b200640 1 Processor -- start 2024-03-06T07:37:23.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.037+0000 7f095b200640 1 -- start start 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.038+0000 7f095b200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f095415ad50 0x7f095415b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.038+0000 7f095b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09540ae790 con 0x7f0954071cc0 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f095b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09540b0760 con 0x7f09540b3d00 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f095b200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095406f050 con 0x7f095415ad50 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f0959200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0954071cc0 0x7f09540720c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48150/0 (socket says 172.21.15.6:48150) 2024-03-06T07:37:23.041 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f0959200640 1 -- 172.21.15.6:0/417791335 learned_addr learned my addr 172.21.15.6:0/417791335 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:23.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f0953e00640 1 --2- 172.21.15.6:0/417791335 >> v2:172.21.15.178:3300/0 conn(0x7f095415ad50 0x7f095415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.043 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826597700 0 0) 0x7f09540ae790 con 0x7f0954071cc0 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.039+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0938003520 con 0x7f0954071cc0 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206794223 0 0) 0x7f0938003520 con 0x7f0954071cc0 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 >> v2:172.21.15.178:3300/0 conn(0x7f095415ad50 msgr2=0x7f095415b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 --2- 172.21.15.6:0/417791335 >> v2:172.21.15.178:3300/0 conn(0x7f095415ad50 0x7f095415b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 >> v1:172.21.15.178:6789/0 conn(0x7f09540b3d00 legacy=0x7f095406e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095406f330 con 0x7f0954071cc0 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f094c001a10 con 0x7f0954071cc0 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 >> v1:172.21.15.6:6789/0 conn(0x7f0954071cc0 legacy=0x7f09540720c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0938003a20 0x7f0938003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0938004800 0x7f0938004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938005230 0x7f093800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.044 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095406f050 con 0x7f0938003a20 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093800c190 con 0x7f0938004800 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.040+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093800c390 con 0x7f0938005230 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0953e00640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0938003a20 0x7f0938003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f094b400640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0938004800 0x7f0938004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0953e00640 1 -- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938005230 msgr2=0x7f093800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0959200640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938005230 0x7f093800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0953e00640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938005230 0x7f093800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0953e00640 1 -- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0938004800 msgr2=0x7f0938004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0953e00640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0938004800 0x7f0938004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0953e00640 1 -- 172.21.15.6:0/417791335 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938003520 con 0x7f0938003a20 2024-03-06T07:37:23.045 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.041+0000 7f0959200640 1 -- 172.21.15.6:0/417791335 reap_dead start 2024-03-06T07:37:23.046 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.042+0000 7f0953e00640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0938003a20 0x7f0938003e40 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0944009ac0 tx=0x7f0938005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2790029fa0f90c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.042+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944039070 con 0x7f0938003a20 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094402c600 con 0x7f0938003a20 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f0953400640 1 -- 172.21.15.6:0/417791335 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944009690 con 0x7f0938003a20 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 -- 172.21.15.6:0/417791335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0938003a20 msgr2=0x7f0938003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0938003a20 0x7f0938003e40 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f0944009ac0 tx=0x7f0938005830 comp rx=0 tx=0).stop 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 -- 172.21.15.6:0/417791335 shutdown_connections 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 --2- 172.21.15.6:0/417791335 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0938003a20 0x7f0938003e40 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 -- 172.21.15.6:0/417791335 >> 172.21.15.6:0/417791335 conn(0x7f09540b2a00 msgr2=0x7f09540b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 -- 172.21.15.6:0/417791335 shutdown_connections 2024-03-06T07:37:23.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.043+0000 7f095b200640 1 -- 172.21.15.6:0/417791335 wait complete. 2024-03-06T07:37:23.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.045+0000 7f095b200640 1 Processor -- start 2024-03-06T07:37:23.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.045+0000 7f095b200640 1 -- start start 2024-03-06T07:37:23.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f095b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938003a20 0x7f095414c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.049 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f095b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 0x7f09540b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f095b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09540cfea0 0x7f09540d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f095b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095415b130 con 0x7f09540b4180 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f095b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095406f8f0 con 0x7f09540cfea0 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f095b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095415b2b0 con 0x7f0938003a20 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f094b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 0x7f09540b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f094b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 0x7f09540b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55442/0 (socket says 172.21.15.6:55442) 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.046+0000 7f094b400640 1 -- 172.21.15.6:0/3253312009 learned_addr learned my addr 172.21.15.6:0/3253312009 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f094b400640 1 -- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938003a20 msgr2=0x7f095414c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f094b400640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938003a20 0x7f095414c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f094b400640 1 -- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09540cfea0 msgr2=0x7f09540d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f0959200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09540cfea0 0x7f09540d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.050 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f094b400640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09540cfea0 0x7f09540d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f094b400640 1 -- 172.21.15.6:0/3253312009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095414cc60 con 0x7f09540b4180 2024-03-06T07:37:23.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f094b400640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 0x7f09540b4600 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e4d0 tx=0x7f093c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c0053a0 con 0x7f09540b4180 2024-03-06T07:37:23.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c018070 con 0x7f09540b4180 2024-03-06T07:37:23.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c003810 con 0x7f09540b4180 2024-03-06T07:37:23.051 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.047+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09540d0890 con 0x7f09540b4180 2024-03-06T07:37:23.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.048+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09541f4450 con 0x7f09540b4180 2024-03-06T07:37:23.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.049+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095404f3a0 con 0x7f09540b4180 2024-03-06T07:37:23.052 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.050+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f093c0039b0 con 0x7f09540b4180 2024-03-06T07:37:23.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.051+0000 7f0950c00640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f093006b4d0 0x7f093006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:23.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.051+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f093c089f40 con 0x7f09540b4180 2024-03-06T07:37:23.053 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.051+0000 7f0953e00640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f093006b4d0 0x7f093006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:23.054 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.052+0000 7f0953e00640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f093006b4d0 0x7f093006d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0944036000 tx=0x7f094402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:23.057 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.055+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f095414cc60 con 0x7f09540b4180 2024-03-06T07:37:23.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.305+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0954072a30 con 0x7f09540b4180 2024-03-06T07:37:23.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.306+0000 7f0950c00640 1 -- 172.21.15.6:0/3253312009 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f093c058d70 con 0x7f09540b4180 2024-03-06T07:37:23.308 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:23.309 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:23.309 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:23.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f093006b4d0 msgr2=0x7f093006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f093006b4d0 0x7f093006d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0944036000 tx=0x7f094402e000 comp rx=0 tx=0).stop 2024-03-06T07:37:23.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 msgr2=0x7f09540b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 0x7f09540b4600 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e4d0 tx=0x7f093c00e990 comp rx=0 tx=0).stop 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 shutdown_connections 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f093006b4d0 0x7f093006d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09540cfea0 0x7f09540d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f09540b4180 0x7f09540b4600 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 --2- 172.21.15.6:0/3253312009 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0938003a20 0x7f095414c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 >> 172.21.15.6:0/3253312009 conn(0x7f09540b2a00 msgr2=0x7f09540aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 shutdown_connections 2024-03-06T07:37:23.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:23.309+0000 7f095b200640 1 -- 172.21.15.6:0/3253312009 wait complete. 2024-03-06T07:37:23.531 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:23.583 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:24.136 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:24.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:24.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:24.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:24.793 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.791+0000 7f52f2000640 1 Processor -- start 2024-03-06T07:37:24.794 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.792+0000 7f52f2000640 1 -- start start 2024-03-06T07:37:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.794+0000 7f52f2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec15f450 0x7f52ec070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.794+0000 7f52f2000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52ec0d04b0 0x7f52ec0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52f2000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0cd0c0 con 0x7f52ec0700c0 2024-03-06T07:37:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52f2000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ec0d29e0 con 0x7f52ec0d04b0 2024-03-06T07:37:24.797 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52f2000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0cd3a0 con 0x7f52ec15f450 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52ebe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f52ec0700c0 0x7f52ec0cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48162/0 (socket says 172.21.15.6:48162) 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52f1600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f52ec0d04b0 0x7f52ec0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52ebe00640 1 -- 172.21.15.6:0/1702459941 learned_addr learned my addr 172.21.15.6:0/1702459941 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.795+0000 7f52f0c00640 1 --2- 172.21.15.6:0/1702459941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec15f450 0x7f52ec070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.796+0000 7f52f1600640 1 -- 172.21.15.6:0/1702459941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec15f450 msgr2=0x7f52ec070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.796+0000 7f52f1600640 1 --2- 172.21.15.6:0/1702459941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec15f450 0x7f52ec070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.796+0000 7f52f1600640 1 -- 172.21.15.6:0/1702459941 >> v1:172.21.15.6:6789/0 conn(0x7f52ec0700c0 legacy=0x7f52ec0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.796+0000 7f52eb400640 1 -- 172.21.15.6:0/1702459941 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888302495 0 0) 0x7f52ec0cd0c0 con 0x7f52ec0700c0 2024-03-06T07:37:24.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.796+0000 7f52f1600640 1 -- 172.21.15.6:0/1702459941 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0cd680 con 0x7f52ec0d04b0 2024-03-06T07:37:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.797+0000 7f52f1600640 1 --2- 172.21.15.6:0/1702459941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0d04b0 0x7f52ec0cde80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009d90 tx=0x7f52e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b049da0ac015e18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.797+0000 7f52eb400640 1 -- 172.21.15.6:0/1702459941 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4039070 con 0x7f52ec0d04b0 2024-03-06T07:37:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.797+0000 7f52eb400640 1 -- 172.21.15.6:0/1702459941 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e402cc60 con 0x7f52ec0d04b0 2024-03-06T07:37:24.799 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.797+0000 7f52eb400640 1 -- 172.21.15.6:0/1702459941 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4034660 con 0x7f52ec0d04b0 2024-03-06T07:37:24.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 -- 172.21.15.6:0/1702459941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0d04b0 msgr2=0x7f52ec0cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 --2- 172.21.15.6:0/1702459941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0d04b0 0x7f52ec0cde80 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009d90 tx=0x7f52e402beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 -- 172.21.15.6:0/1702459941 shutdown_connections 2024-03-06T07:37:24.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 --2- 172.21.15.6:0/1702459941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0d04b0 0x7f52ec0cde80 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 --2- 172.21.15.6:0/1702459941 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec15f450 0x7f52ec070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 -- 172.21.15.6:0/1702459941 >> 172.21.15.6:0/1702459941 conn(0x7f52ec0d1ae0 msgr2=0x7f52ec0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:24.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 -- 172.21.15.6:0/1702459941 shutdown_connections 2024-03-06T07:37:24.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.798+0000 7f52f2000640 1 -- 172.21.15.6:0/1702459941 wait complete. 2024-03-06T07:37:24.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.800+0000 7f52f2000640 1 Processor -- start 2024-03-06T07:37:24.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.800+0000 7f52f2000640 1 -- start start 2024-03-06T07:37:24.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 0x7f52ec1549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec0d04b0 0x7f52ec0c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f2000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52ec15f450 0x7f52ec157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f2000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec15f830 con 0x7f52ec0700c0 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f2000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0ae8b0 con 0x7f52ec15f450 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f2000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec15f9b0 con 0x7f52ec0d04b0 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.801+0000 7f52f0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 0x7f52ec1549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 0x7f52ec1549e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55468/0 (socket says 172.21.15.6:55468) 2024-03-06T07:37:24.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 -- 172.21.15.6:0/3050657453 learned_addr learned my addr 172.21.15.6:0/3050657453 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52ebe00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec0d04b0 0x7f52ec0c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 -- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec0d04b0 msgr2=0x7f52ec0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f1600640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52ec15f450 0x7f52ec157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec0d04b0 0x7f52ec0c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 -- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52ec15f450 msgr2=0x7f52ec157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52ec15f450 0x7f52ec157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 -- 172.21.15.6:0/3050657453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0c90f0 con 0x7f52ec0700c0 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52ebe00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec0d04b0 0x7f52ec0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:24.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.802+0000 7f52f0c00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 0x7f52ec1549e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f52e000d5f0 tx=0x7f52e000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.805+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e00125f0 con 0x7f52ec0700c0 2024-03-06T07:37:24.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.805+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e0019070 con 0x7f52ec0700c0 2024-03-06T07:37:24.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.805+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e00129d0 con 0x7f52ec0700c0 2024-03-06T07:37:24.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.805+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec0c9390 con 0x7f52ec0700c0 2024-03-06T07:37:24.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.805+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec0c60a0 con 0x7f52ec0700c0 2024-03-06T07:37:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.807+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec04f330 con 0x7f52ec0700c0 2024-03-06T07:37:24.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.807+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52e0012b70 con 0x7f52ec0700c0 2024-03-06T07:37:24.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.808+0000 7f52e8c00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f52cc06b500 0x7f52cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:24.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.808+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f52e008a570 con 0x7f52ec0700c0 2024-03-06T07:37:24.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.809+0000 7f52ebe00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f52cc06b500 0x7f52cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:24.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.810+0000 7f52ebe00640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f52cc06b500 0x7f52cc06d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f52ec09fb90 tx=0x7f52d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:24.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:24.818+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f52e0059440 con 0x7f52ec0700c0 2024-03-06T07:37:25.059 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.057+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52ec06cf90 con 0x7f52ec0700c0 2024-03-06T07:37:25.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.058+0000 7f52e8c00640 1 -- 172.21.15.6:0/3050657453 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f52e0058d40 con 0x7f52ec0700c0 2024-03-06T07:37:25.061 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:25.061 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:25.061 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:25.063 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f52cc06b500 msgr2=0x7f52cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f52cc06b500 0x7f52cc06d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f52ec09fb90 tx=0x7f52d8009040 comp rx=0 tx=0).stop 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 msgr2=0x7f52ec1549e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 0x7f52ec1549e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f52e000d5f0 tx=0x7f52e000dab0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 shutdown_connections 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f52cc06b500 0x7f52cc06d9c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52ec15f450 0x7f52ec157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f52ec0d04b0 0x7f52ec0c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 --2- 172.21.15.6:0/3050657453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f52ec0700c0 0x7f52ec1549e0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 >> 172.21.15.6:0/3050657453 conn(0x7f52ec0d1ae0 msgr2=0x7f52ec06c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 shutdown_connections 2024-03-06T07:37:25.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:25.061+0000 7f52f2000640 1 -- 172.21.15.6:0/3050657453 wait complete. 2024-03-06T07:37:25.283 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:25.283 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:26.138 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:26.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:26.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.516+0000 7fd733200640 1 Processor -- start 2024-03-06T07:37:26.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.516+0000 7fd733200640 1 -- start start 2024-03-06T07:37:26.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.517+0000 7fd733200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd7340d4a10 0x7fd7340d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.517+0000 7fd733200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd7340d0c80 0x7fd7340b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.517+0000 7fd733200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7340b3710 con 0x7fd73415e0b0 2024-03-06T07:37:26.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.517+0000 7fd733200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7340d10f0 con 0x7fd7340d0c80 2024-03-06T07:37:26.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.517+0000 7fd733200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7340b09e0 con 0x7fd7340d4a10 2024-03-06T07:37:26.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.518+0000 7fd72be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd7340d4a10 0x7fd7340d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.518+0000 7fd732800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd7340d0c80 0x7fd7340b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.518+0000 7fd72be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd7340d4a10 0x7fd7340d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:36604/0 (socket says 172.21.15.6:36604) 2024-03-06T07:37:26.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.518+0000 7fd72be00640 1 -- 172.21.15.6:0/2587844294 learned_addr learned my addr 172.21.15.6:0/2587844294 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:26.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.519+0000 7fd732800640 1 -- 172.21.15.6:0/2587844294 >> v2:172.21.15.178:3300/0 conn(0x7fd7340d4a10 msgr2=0x7fd7340d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:26.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.519+0000 7fd732800640 1 --2- 172.21.15.6:0/2587844294 >> v2:172.21.15.178:3300/0 conn(0x7fd7340d4a10 0x7fd7340d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.519+0000 7fd732800640 1 -- 172.21.15.6:0/2587844294 >> v1:172.21.15.178:6789/0 conn(0x7fd73415e0b0 legacy=0x7fd7340cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.519+0000 7fd732800640 1 -- 172.21.15.6:0/2587844294 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7340b39f0 con 0x7fd7340d0c80 2024-03-06T07:37:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.521+0000 7fd732800640 1 --2- 172.21.15.6:0/2587844294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd7340b44d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fd73406b8c0 tx=0x7fd72c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ce2f3011c0e178 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.521+0000 7fd731400640 1 -- 172.21.15.6:0/2587844294 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c039070 con 0x7fd7340d0c80 2024-03-06T07:37:26.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd731400640 1 -- 172.21.15.6:0/2587844294 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c02cc60 con 0x7fd7340d0c80 2024-03-06T07:37:26.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd731400640 1 -- 172.21.15.6:0/2587844294 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c034660 con 0x7fd7340d0c80 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd733200640 1 -- 172.21.15.6:0/2587844294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 msgr2=0x7fd7340b44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd733200640 1 --2- 172.21.15.6:0/2587844294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd7340b44d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fd73406b8c0 tx=0x7fd72c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd733200640 1 -- 172.21.15.6:0/2587844294 shutdown_connections 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd733200640 1 --2- 172.21.15.6:0/2587844294 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd7340b44d0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd733200640 1 --2- 172.21.15.6:0/2587844294 >> v2:172.21.15.178:3300/0 conn(0x7fd7340d4a10 0x7fd7340d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.522+0000 7fd733200640 1 -- 172.21.15.6:0/2587844294 >> 172.21.15.6:0/2587844294 conn(0x7fd7340d3d30 msgr2=0x7fd7340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.523+0000 7fd733200640 1 -- 172.21.15.6:0/2587844294 shutdown_connections 2024-03-06T07:37:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.523+0000 7fd733200640 1 -- 172.21.15.6:0/2587844294 wait complete. 2024-03-06T07:37:26.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.524+0000 7fd733200640 1 Processor -- start 2024-03-06T07:37:26.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.524+0000 7fd733200640 1 -- start start 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.525+0000 7fd733200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd734157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.525+0000 7fd733200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7340d4a10 0x7fd73415b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.525+0000 7fd733200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd73415e0b0 0x7fd73415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.525+0000 7fd733200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7340b3fb0 con 0x7fd7340d0c80 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.525+0000 7fd733200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7340cfa20 con 0x7fd73415e0b0 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.525+0000 7fd733200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73415e490 con 0x7fd7340d4a10 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd734157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd734157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55484/0 (socket says 172.21.15.6:55484) 2024-03-06T07:37:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 -- 172.21.15.6:0/1651251146 learned_addr learned my addr 172.21.15.6:0/1651251146 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:26.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd732800640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd73415e0b0 0x7fd73415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 -- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7340d4a10 msgr2=0x7fd73415b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd72be00640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7340d4a10 0x7fd73415b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7340d4a10 0x7fd73415b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 -- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd73415e0b0 msgr2=0x7fd73415b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd73415e0b0 0x7fd73415b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd731e00640 1 -- 172.21.15.6:0/1651251146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7340ccaa0 con 0x7fd7340d0c80 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.526+0000 7fd732800640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd73415e0b0 0x7fd73415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.527+0000 7fd731e00640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd734157ae0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fd72400c430 tx=0x7fd72400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.527+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724017070 con 0x7fd7340d0c80 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.527+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7240052d0 con 0x7fd7340d0c80 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.527+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7340ccd80 con 0x7fd7340d0c80 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.527+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724003850 con 0x7fd7340d0c80 2024-03-06T07:37:26.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.527+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7340cd340 con 0x7fd7340d0c80 2024-03-06T07:37:26.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.528+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7340d2c10 con 0x7fd7340d0c80 2024-03-06T07:37:26.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.530+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd72401a070 con 0x7fd7340d0c80 2024-03-06T07:37:26.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.531+0000 7fd72a000640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd71006b4d0 0x7fd71006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:26.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.531+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd724089740 con 0x7fd7340d0c80 2024-03-06T07:37:26.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.532+0000 7fd72be00640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd71006b4d0 0x7fd71006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:26.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.533+0000 7fd72be00640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd71006b4d0 0x7fd71006d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd7340b6830 tx=0x7fd71c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:26.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.536+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7340d2c10 con 0x7fd7340d0c80 2024-03-06T07:37:26.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.774+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd73404f330 con 0x7fd7340d0c80 2024-03-06T07:37:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.775+0000 7fd72a000640 1 -- 172.21.15.6:0/1651251146 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fd724058450 con 0x7fd7340d0c80 2024-03-06T07:37:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:26.778 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:26.778 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd71006b4d0 msgr2=0x7fd71006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd71006b4d0 0x7fd71006d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fd7340b6830 tx=0x7fd71c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 msgr2=0x7fd734157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd734157ae0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fd72400c430 tx=0x7fd72400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 shutdown_connections 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd71006b4d0 0x7fd71006d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd73415e0b0 0x7fd73415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd7340d4a10 0x7fd73415b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 --2- 172.21.15.6:0/1651251146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd7340d0c80 0x7fd734157ae0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:26.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 >> 172.21.15.6:0/1651251146 conn(0x7fd7340d3d30 msgr2=0x7fd7340cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:26.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 shutdown_connections 2024-03-06T07:37:26.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:26.778+0000 7fd733200640 1 -- 172.21.15.6:0/1651251146 wait complete. 2024-03-06T07:37:27.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:27.002 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:28.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:28.140 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.223+0000 7f0b7e200640 1 Processor -- start 2024-03-06T07:37:28.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.224+0000 7f0b7e200640 1 -- start start 2024-03-06T07:37:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b7e200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b780d4a10 0x7f0b780d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b7e200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780b3710 con 0x7f0b780d0c80 2024-03-06T07:37:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b7e200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b780b39f0 con 0x7f0b7815e0b0 2024-03-06T07:37:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b7e200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780d10f0 con 0x7f0b780d4a10 2024-03-06T07:37:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b77e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b780d4a10 0x7f0b780d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b7d800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0b780d0c80 0x7f0b780b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48178/0 (socket says 172.21.15.6:48178) 2024-03-06T07:37:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.226+0000 7f0b7d800640 1 -- 172.21.15.6:0/2698818125 learned_addr learned my addr 172.21.15.6:0/2698818125 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.227+0000 7f0b77e00640 1 -- 172.21.15.6:0/2698818125 >> v1:172.21.15.178:6789/0 conn(0x7f0b7815e0b0 legacy=0x7f0b780cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.227+0000 7f0b77e00640 1 -- 172.21.15.6:0/2698818125 >> v1:172.21.15.6:6789/0 conn(0x7f0b780d0c80 legacy=0x7f0b780b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.227+0000 7f0b77e00640 1 -- 172.21.15.6:0/2698818125 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780b3cd0 con 0x7f0b780d4a10 2024-03-06T07:37:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.228+0000 7f0b77e00640 1 --2- 172.21.15.6:0/2698818125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b780d4a10 0x7f0b780d0740 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f0b600097a0 tx=0x7f0b6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f59c56c1ffb0fae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.228+0000 7f0b77400640 1 -- 172.21.15.6:0/2698818125 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60035020 con 0x7f0b780d4a10 2024-03-06T07:37:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.228+0000 7f0b77400640 1 -- 172.21.15.6:0/2698818125 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b60031830 con 0x7f0b780d4a10 2024-03-06T07:37:28.230 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.228+0000 7f0b77400640 1 -- 172.21.15.6:0/2698818125 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b60031c00 con 0x7f0b780d4a10 2024-03-06T07:37:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 -- 172.21.15.6:0/2698818125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b780d4a10 msgr2=0x7f0b780d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 --2- 172.21.15.6:0/2698818125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b780d4a10 0x7f0b780d0740 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f0b600097a0 tx=0x7f0b6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 -- 172.21.15.6:0/2698818125 shutdown_connections 2024-03-06T07:37:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 --2- 172.21.15.6:0/2698818125 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b780d4a10 0x7f0b780d0740 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.231 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 -- 172.21.15.6:0/2698818125 >> 172.21.15.6:0/2698818125 conn(0x7f0b780d3d30 msgr2=0x7f0b780d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 -- 172.21.15.6:0/2698818125 shutdown_connections 2024-03-06T07:37:28.232 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.229+0000 7f0b7e200640 1 -- 172.21.15.6:0/2698818125 wait complete. 2024-03-06T07:37:28.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.231+0000 7f0b7e200640 1 Processor -- start 2024-03-06T07:37:28.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.231+0000 7f0b7e200640 1 -- start start 2024-03-06T07:37:28.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7e200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b780d0c80 0x7f0b780b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7e200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b780d4a10 0x7f0b7815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7e200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 0x7f0b7815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7e200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780cfa20 con 0x7f0b7815e0b0 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7e200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7815e490 con 0x7f0b780d0c80 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7e200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780cfc60 con 0x7f0b780d4a10 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 0x7f0b7815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7d800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 0x7f0b7815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55498/0 (socket says 172.21.15.6:55498) 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.232+0000 7f0b7d800640 1 -- 172.21.15.6:0/3583303927 learned_addr learned my addr 172.21.15.6:0/3583303927 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7ce00640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b780d0c80 0x7f0b780b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b77e00640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b780d4a10 0x7f0b7815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7d800640 1 -- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b780d4a10 msgr2=0x7f0b7815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7d800640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b780d4a10 0x7f0b7815bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7d800640 1 -- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b780d0c80 msgr2=0x7f0b780b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7d800640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b780d0c80 0x7f0b780b93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7d800640 1 -- 172.21.15.6:0/3583303927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780cb8b0 con 0x7f0b7815e0b0 2024-03-06T07:37:28.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7d800640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 0x7f0b7815c020 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000c3c0 tx=0x7f0b7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70017070 con 0x7f0b7815e0b0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b70005ce0 con 0x7f0b7815e0b0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70007310 con 0x7f0b7815e0b0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.233+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b780cbb50 con 0x7f0b7815e0b0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.234+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b780cc110 con 0x7f0b7815e0b0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.235+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7804f330 con 0x7f0b7815e0b0 2024-03-06T07:37:28.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.236+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b7001a070 con 0x7f0b7815e0b0 2024-03-06T07:37:28.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.236+0000 7f0b74c00640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b5406b4d0 0x7f0b5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:28.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.236+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0b700895e0 con 0x7f0b7815e0b0 2024-03-06T07:37:28.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.237+0000 7f0b7ce00640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b5406b4d0 0x7f0b5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:28.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.238+0000 7f0b7ce00640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b5406b4d0 0x7f0b5406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0b68005520 tx=0x7f0b68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:28.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.241+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b70058380 con 0x7f0b7815e0b0 2024-03-06T07:37:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.481+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b78161640 con 0x7f0b7815e0b0 2024-03-06T07:37:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.482+0000 7f0b74c00640 1 -- 172.21.15.6:0/3583303927 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f0b70057c80 con 0x7f0b7815e0b0 2024-03-06T07:37:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:28.485 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:28.485 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b5406b4d0 msgr2=0x7f0b5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b5406b4d0 0x7f0b5406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0b68005520 tx=0x7f0b68009040 comp rx=0 tx=0).stop 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 msgr2=0x7f0b7815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 0x7f0b7815c020 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000c3c0 tx=0x7f0b7000c880 comp rx=0 tx=0).stop 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 shutdown_connections 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0b5406b4d0 0x7f0b5406d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b7815e0b0 0x7f0b7815c020 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0b780d4a10 0x7f0b7815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 --2- 172.21.15.6:0/3583303927 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b780d0c80 0x7f0b780b93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:28.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.485+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 >> 172.21.15.6:0/3583303927 conn(0x7f0b780d3d30 msgr2=0x7f0b780b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.486+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 shutdown_connections 2024-03-06T07:37:28.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:28.486+0000 7f0b7e200640 1 -- 172.21.15.6:0/3583303927 wait complete. 2024-03-06T07:37:28.706 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:28.706 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:29.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.974+0000 7f50b9000640 1 Processor -- start 2024-03-06T07:37:29.976 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.974+0000 7f50b9000640 1 -- start start 2024-03-06T07:37:29.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.976+0000 7f50b9000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f50b40cd060 0x7f50b41667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.976+0000 7f50b9000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b40714d0 con 0x7f50b4070f10 2024-03-06T07:37:29.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.976+0000 7f50b9000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b40717b0 con 0x7f50b4145300 2024-03-06T07:37:29.978 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.976+0000 7f50b9000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b4071a90 con 0x7f50b40cd060 2024-03-06T07:37:29.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.976+0000 7f50b2a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f50b4070f10 0x7f50b40cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48186/0 (socket says 172.21.15.6:48186) 2024-03-06T07:37:29.979 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.976+0000 7f50b2a00640 1 -- 172.21.15.6:0/99355692 learned_addr learned my addr 172.21.15.6:0/99355692 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:29.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b3e00640 1 --2- 172.21.15.6:0/99355692 >> v2:172.21.15.178:3300/0 conn(0x7f50b40cd060 0x7f50b41667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.980 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715273658 0 0) 0x7f50b40714d0 con 0x7f50b4070f10 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098003520 con 0x7f50b4070f10 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418316693 0 0) 0x7f5098003520 con 0x7f50b4070f10 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 >> v2:172.21.15.178:3300/0 conn(0x7f50b40cd060 msgr2=0x7f50b41667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 --2- 172.21.15.6:0/99355692 >> v2:172.21.15.178:3300/0 conn(0x7f50b40cd060 0x7f50b41667e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 >> v1:172.21.15.178:6789/0 conn(0x7f50b4145300 legacy=0x7f50b40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4071d70 con 0x7f50b4070f10 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f509c001670 con 0x7f50b4070f10 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 >> v1:172.21.15.6:6789/0 conn(0x7f50b4070f10 legacy=0x7f50b40cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5098004800 0x7f5098004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50980051c0 0x7f509800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4071a90 con 0x7f50980039b0 2024-03-06T07:37:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.977+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509800c160 con 0x7f5098004800 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509800c360 con 0x7f50980051c0 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3400640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b2a00640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5098004800 0x7f5098004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3400640 1 -- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50980051c0 msgr2=0x7f509800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3e00640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50980051c0 0x7f509800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3400640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50980051c0 0x7f509800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3400640 1 -- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5098004800 msgr2=0x7f5098004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3400640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5098004800 0x7f5098004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3400640 1 -- 172.21.15.6:0/99355692 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098003520 con 0x7f50980039b0 2024-03-06T07:37:29.982 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.978+0000 7f50b3e00640 1 -- 172.21.15.6:0/99355692 reap_dead start 2024-03-06T07:37:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.979+0000 7f50b3400640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f50a4001e50 tx=0x7f5098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3653a7e08341737 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4001a70 con 0x7f50980039b0 2024-03-06T07:37:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4003b70 con 0x7f50980039b0 2024-03-06T07:37:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b2000640 1 -- 172.21.15.6:0/99355692 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a40050d0 con 0x7f50980039b0 2024-03-06T07:37:29.983 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b9000640 1 -- 172.21.15.6:0/99355692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 msgr2=0x7f5098003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b9000640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f50a4001e50 tx=0x7f5098005830 comp rx=0 tx=0).stop 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b9000640 1 -- 172.21.15.6:0/99355692 shutdown_connections 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b9000640 1 --2- 172.21.15.6:0/99355692 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f5098003dd0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b9000640 1 -- 172.21.15.6:0/99355692 >> 172.21.15.6:0/99355692 conn(0x7f50b406b3d0 msgr2=0x7f50b406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.980+0000 7f50b9000640 1 -- 172.21.15.6:0/99355692 shutdown_connections 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.981+0000 7f50b9000640 1 -- 172.21.15.6:0/99355692 wait complete. 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.982+0000 7f50b9000640 1 Processor -- start 2024-03-06T07:37:29.984 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.982+0000 7f50b9000640 1 -- start start 2024-03-06T07:37:29.986 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.983+0000 7f50b9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f50b41415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.983+0000 7f50b9000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50b413c130 0x7f50b41f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.983+0000 7f50b9000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50b406d8f0 0x7f50b413cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b9000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40ccad0 con 0x7f50980039b0 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b9000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40cc950 con 0x7f50b413c130 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b9000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40ccc50 con 0x7f50b406d8f0 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f50b41415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f50b41415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55506/0 (socket says 172.21.15.6:55506) 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 -- 172.21.15.6:0/2382670190 learned_addr learned my addr 172.21.15.6:0/2382670190 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b2a00640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50b413c130 0x7f50b41f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3e00640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50b406d8f0 0x7f50b413cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 -- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50b406d8f0 msgr2=0x7f50b413cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50b406d8f0 0x7f50b413cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 -- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50b413c130 msgr2=0x7f50b41f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50b413c130 0x7f50b41f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:29.987 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.984+0000 7f50b3400640 1 -- 172.21.15.6:0/2382670190 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b41f4470 con 0x7f50980039b0 2024-03-06T07:37:29.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.985+0000 7f50b3400640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f50b41415a0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f50a4001f70 tx=0x7f50a40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:29.988 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.985+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4003970 con 0x7f50980039b0 2024-03-06T07:37:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.985+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4006b50 con 0x7f50980039b0 2024-03-06T07:37:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.985+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4007030 con 0x7f50980039b0 2024-03-06T07:37:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.985+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b41f4750 con 0x7f50980039b0 2024-03-06T07:37:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.985+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b413fb60 con 0x7f50980039b0 2024-03-06T07:37:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.986+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b41f41e0 con 0x7f50980039b0 2024-03-06T07:37:29.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.988+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50a40071d0 con 0x7f50980039b0 2024-03-06T07:37:29.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.988+0000 7f50ab400640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:29.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.988+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f50a40abe90 con 0x7f50980039b0 2024-03-06T07:37:29.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.989+0000 7f50b2a00640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:29.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.990+0000 7f50b2a00640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f509c001eb0 tx=0x7f509c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:30.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:29.998+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f50a4033020 con 0x7f50980039b0 2024-03-06T07:37:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.262+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50b404f330 con 0x7f50980039b0 2024-03-06T07:37:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.263+0000 7f50ab400640 1 -- 172.21.15.6:0/2382670190 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f50a407acd0 con 0x7f50980039b0 2024-03-06T07:37:30.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:30.265 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f509006b4d0 msgr2=0x7f509006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f509006b4d0 0x7f509006d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f509c001eb0 tx=0x7f509c009040 comp rx=0 tx=0).stop 2024-03-06T07:37:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 msgr2=0x7f50b41415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f50b41415a0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f50a4001f70 tx=0x7f50a40014a0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 shutdown_connections 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f509006b4d0 0x7f509006d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50b413c130 0x7f50b41f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50b406d8f0 0x7f50b413cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 --2- 172.21.15.6:0/2382670190 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50980039b0 0x7f50b41415a0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 >> 172.21.15.6:0/2382670190 conn(0x7f50b406b3d0 msgr2=0x7f50b40cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 shutdown_connections 2024-03-06T07:37:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:30.267+0000 7f50b9000640 1 -- 172.21.15.6:0/2382670190 wait complete. 2024-03-06T07:37:30.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:30.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:31.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:31.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.704+0000 7f098ca00640 1 Processor -- start 2024-03-06T07:37:31.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.704+0000 7f098ca00640 1 -- start start 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f098ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f098815e2a0 0x7f09880cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f098ca00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880d1120 con 0x7f098815dec0 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f098ca00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880d1400 con 0x7f09880cf460 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f098ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09880b0ea0 con 0x7f098815e2a0 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f0986a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f098815e2a0 0x7f09880cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f0987400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f098815dec0 0x7f09880ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:48198/0 (socket says 172.21.15.6:48198) 2024-03-06T07:37:31.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.706+0000 7f0987400640 1 -- 172.21.15.6:0/3883594462 learned_addr learned my addr 172.21.15.6:0/3883594462 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.707+0000 7f0986a00640 1 -- 172.21.15.6:0/3883594462 >> v1:172.21.15.178:6789/0 conn(0x7f09880cf460 legacy=0x7f09880b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.707+0000 7f0986a00640 1 -- 172.21.15.6:0/3883594462 >> v1:172.21.15.6:6789/0 conn(0x7f098815dec0 legacy=0x7f09880ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.707+0000 7f0986a00640 1 -- 172.21.15.6:0/3883594462 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880b0b30 con 0x7f098815e2a0 2024-03-06T07:37:31.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.708+0000 7f0986a00640 1 --2- 172.21.15.6:0/3883594462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880cef20 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f09740097a0 tx=0x7f097402b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251d5ecfc2023fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.709+0000 7f0986000640 1 -- 172.21.15.6:0/3883594462 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974037070 con 0x7f098815e2a0 2024-03-06T07:37:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.709+0000 7f0986000640 1 -- 172.21.15.6:0/3883594462 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0974030710 con 0x7f098815e2a0 2024-03-06T07:37:31.711 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.709+0000 7f0986000640 1 -- 172.21.15.6:0/3883594462 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0974030ae0 con 0x7f098815e2a0 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.709+0000 7f098ca00640 1 -- 172.21.15.6:0/3883594462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 msgr2=0x7f09880cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.709+0000 7f098ca00640 1 --2- 172.21.15.6:0/3883594462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880cef20 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f09740097a0 tx=0x7f097402b710 comp rx=0 tx=0).stop 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.710+0000 7f098ca00640 1 -- 172.21.15.6:0/3883594462 shutdown_connections 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.710+0000 7f098ca00640 1 --2- 172.21.15.6:0/3883594462 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880cef20 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.710+0000 7f098ca00640 1 -- 172.21.15.6:0/3883594462 >> 172.21.15.6:0/3883594462 conn(0x7f09880d3d40 msgr2=0x7f09880d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.710+0000 7f098ca00640 1 -- 172.21.15.6:0/3883594462 shutdown_connections 2024-03-06T07:37:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.710+0000 7f098ca00640 1 -- 172.21.15.6:0/3883594462 wait complete. 2024-03-06T07:37:31.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.711+0000 7f098ca00640 1 Processor -- start 2024-03-06T07:37:31.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.712+0000 7f098ca00640 1 -- start start 2024-03-06T07:37:31.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.712+0000 7f098ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09880cf460 0x7f09880b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.712+0000 7f098ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f098815dec0 0x7f09881f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f098ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f098ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09880d0ec0 con 0x7f098815e2a0 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f098ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09880d0d40 con 0x7f09880cf460 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f098ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09880b1a40 con 0x7f098815dec0 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:55526/0 (socket says 172.21.15.6:55526) 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 -- 172.21.15.6:0/2470976054 learned_addr learned my addr 172.21.15.6:0/2470976054 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987400640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09880cf460 0x7f09880b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0986a00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f098815dec0 0x7f09881f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 -- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f098815dec0 msgr2=0x7f09881f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f098815dec0 0x7f09881f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 -- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09880cf460 msgr2=0x7f09880b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09880cf460 0x7f09880b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0987e00640 1 -- 172.21.15.6:0/2470976054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880cab80 con 0x7f098815e2a0 2024-03-06T07:37:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.713+0000 7f0986a00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f098815dec0 0x7f09881f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.714+0000 7f0987e00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880c7870 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f098000c700 tx=0x7f098000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.714+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980016020 con 0x7f098815e2a0 2024-03-06T07:37:31.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.714+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098000aa50 con 0x7f098815e2a0 2024-03-06T07:37:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.714+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09880ca8a0 con 0x7f098815e2a0 2024-03-06T07:37:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.714+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098000ce10 con 0x7f098815e2a0 2024-03-06T07:37:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.714+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09880b7c10 con 0x7f098815e2a0 2024-03-06T07:37:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.715+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09881f8020 con 0x7f098815e2a0 2024-03-06T07:37:31.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.716+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0980019020 con 0x7f098815e2a0 2024-03-06T07:37:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.717+0000 7f097f400640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f096806b4d0 0x7f096806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:31.719 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.717+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f098008a5e0 con 0x7f098815e2a0 2024-03-06T07:37:31.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.718+0000 7f0987400640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f096806b4d0 0x7f096806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:31.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.719+0000 7f0987400640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f096806b4d0 0x7f096806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0978000f60 tx=0x7f0978007c60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:31.724 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.722+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09881f8020 con 0x7f098815e2a0 2024-03-06T07:37:31.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.963+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09880b25e0 con 0x7f098815e2a0 2024-03-06T07:37:31.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.964+0000 7f097f400640 1 -- 172.21.15.6:0/2470976054 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f0980020020 con 0x7f098815e2a0 2024-03-06T07:37:31.966 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:31.966 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:31.966 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:31.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.966+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f096806b4d0 msgr2=0x7f096806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.966+0000 7f098ca00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f096806b4d0 0x7f096806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0978000f60 tx=0x7f0978007c60 comp rx=0 tx=0).stop 2024-03-06T07:37:31.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.966+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 msgr2=0x7f09880c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:31.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.966+0000 7f098ca00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880c7870 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f098000c700 tx=0x7f098000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 shutdown_connections 2024-03-06T07:37:31.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f096806b4d0 0x7f096806d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f098815e2a0 0x7f09880c7870 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f098815dec0 0x7f09881f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.969 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 --2- 172.21.15.6:0/2470976054 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f09880cf460 0x7f09880b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:31.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 >> 172.21.15.6:0/2470976054 conn(0x7f09880d3d40 msgr2=0x7f09880d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:31.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 shutdown_connections 2024-03-06T07:37:31.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:31.967+0000 7f098ca00640 1 -- 172.21.15.6:0/2470976054 wait complete. 2024-03-06T07:37:32.144 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:32.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:32.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:33.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.416+0000 7fb78b200640 1 Processor -- start 2024-03-06T07:37:33.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.416+0000 7fb78b200640 1 -- start start 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.417+0000 7fb78b200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb78415bfb0 0x7fb7840ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.417+0000 7fb78b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7840af000 con 0x7fb78415cda0 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.418+0000 7fb78b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7840af2e0 con 0x7fb78406afb0 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.418+0000 7fb78b200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7840b3de0 con 0x7fb78415bfb0 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.418+0000 7fb77be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb78415bfb0 0x7fb7840ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.418+0000 7fb789200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb78415cda0 0x7fb7840b4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54180/0 (socket says 172.21.15.6:54180) 2024-03-06T07:37:33.420 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.418+0000 7fb789200640 1 -- 172.21.15.6:0/1918564026 learned_addr learned my addr 172.21.15.6:0/1918564026 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.419+0000 7fb77be00640 1 -- 172.21.15.6:0/1918564026 >> v1:172.21.15.178:6789/0 conn(0x7fb78406afb0 legacy=0x7fb78406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.419+0000 7fb77be00640 1 -- 172.21.15.6:0/1918564026 >> v1:172.21.15.6:6789/0 conn(0x7fb78415cda0 legacy=0x7fb7840b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.419+0000 7fb77be00640 1 -- 172.21.15.6:0/1918564026 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7840b4030 con 0x7fb78415bfb0 2024-03-06T07:37:33.421 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.419+0000 7fb783400640 1 -- 172.21.15.6:0/1918564026 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789844017 0 0) 0x7fb7840af000 con 0x7fb78415cda0 2024-03-06T07:37:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.420+0000 7fb77be00640 1 --2- 172.21.15.6:0/1918564026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78415bfb0 0x7fb7840ae900 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0097a0 tx=0x7fb76c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee8227bf8bfa42a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.420+0000 7fb783400640 1 -- 172.21.15.6:0/1918564026 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c035020 con 0x7fb78415bfb0 2024-03-06T07:37:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.420+0000 7fb783400640 1 -- 172.21.15.6:0/1918564026 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c031830 con 0x7fb78415bfb0 2024-03-06T07:37:33.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.420+0000 7fb783400640 1 -- 172.21.15.6:0/1918564026 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c031c00 con 0x7fb78415bfb0 2024-03-06T07:37:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 -- 172.21.15.6:0/1918564026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78415bfb0 msgr2=0x7fb7840ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 --2- 172.21.15.6:0/1918564026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78415bfb0 0x7fb7840ae900 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0097a0 tx=0x7fb76c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 -- 172.21.15.6:0/1918564026 shutdown_connections 2024-03-06T07:37:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 --2- 172.21.15.6:0/1918564026 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78415bfb0 0x7fb7840ae900 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 -- 172.21.15.6:0/1918564026 >> 172.21.15.6:0/1918564026 conn(0x7fb7840b2ae0 msgr2=0x7fb7840b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:33.423 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 -- 172.21.15.6:0/1918564026 shutdown_connections 2024-03-06T07:37:33.424 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.421+0000 7fb78b200640 1 -- 172.21.15.6:0/1918564026 wait complete. 2024-03-06T07:37:33.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.424+0000 7fb78b200640 1 Processor -- start 2024-03-06T07:37:33.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.424+0000 7fb78b200640 1 -- start start 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb78b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 0x7fb7841e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb78b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78415bfb0 0x7fb7841f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb78b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78415cda0 0x7fb7841491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb78b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7840b4f40 con 0x7fb78406afb0 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb78b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78415d1d0 con 0x7fb78415cda0 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb78b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7840b50c0 con 0x7fb78415bfb0 2024-03-06T07:37:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb783e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 0x7fb7841e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb783e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 0x7fb7841e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44134/0 (socket says 172.21.15.6:44134) 2024-03-06T07:37:33.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb783e00640 1 -- 172.21.15.6:0/3090348371 learned_addr learned my addr 172.21.15.6:0/3090348371 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb77be00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78415bfb0 0x7fb7841f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb783e00640 1 -- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78415bfb0 msgr2=0x7fb7841f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.426+0000 7fb783e00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78415bfb0 0x7fb7841f7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb783e00640 1 -- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78415cda0 msgr2=0x7fb7841491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb783e00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78415cda0 0x7fb7841491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb783e00640 1 -- 172.21.15.6:0/3090348371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb784156e10 con 0x7fb78406afb0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb783e00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 0x7fb7841e8170 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fb77400c430 tx=0x7fb77400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774011790 con 0x7fb78406afb0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774017070 con 0x7fb78406afb0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774011bb0 con 0x7fb78406afb0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7841570f0 con 0x7fb78406afb0 2024-03-06T07:37:33.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.427+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7841e6b90 con 0x7fb78406afb0 2024-03-06T07:37:33.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.429+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78404f3a0 con 0x7fb78406afb0 2024-03-06T07:37:33.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.430+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb77401c460 con 0x7fb78406afb0 2024-03-06T07:37:33.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.431+0000 7fb780c00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76006b550 0x7fb76006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:33.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.431+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb77408a0f0 con 0x7fb78406afb0 2024-03-06T07:37:33.434 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.432+0000 7fb77be00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76006b550 0x7fb76006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:33.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.433+0000 7fb77be00640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76006b550 0x7fb76006da10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb76c036000 tx=0x7fb76c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:33.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.435+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb774058f10 con 0x7fb78406afb0 2024-03-06T07:37:33.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.673+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78415eea0 con 0x7fb78406afb0 2024-03-06T07:37:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.674+0000 7fb780c00640 1 -- 172.21.15.6:0/3090348371 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fb774058810 con 0x7fb78406afb0 2024-03-06T07:37:33.676 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:33.676 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76006b550 msgr2=0x7fb76006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76006b550 0x7fb76006da10 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb76c036000 tx=0x7fb76c008030 comp rx=0 tx=0).stop 2024-03-06T07:37:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 msgr2=0x7fb7841e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 0x7fb7841e8170 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fb77400c430 tx=0x7fb77400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 shutdown_connections 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb76006b550 0x7fb76006da10 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb78415cda0 0x7fb7841491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb78415bfb0 0x7fb7841f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 --2- 172.21.15.6:0/3090348371 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb78406afb0 0x7fb7841e8170 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 >> 172.21.15.6:0/3090348371 conn(0x7fb7840b2ae0 msgr2=0x7fb7840b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 shutdown_connections 2024-03-06T07:37:33.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:33.677+0000 7fb78b200640 1 -- 172.21.15.6:0/3090348371 wait complete. 2024-03-06T07:37:33.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:33.899 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:34.146 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:34.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:35.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.119+0000 7fad32600640 1 Processor -- start 2024-03-06T07:37:35.122 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.120+0000 7fad32600640 1 -- start start 2024-03-06T07:37:35.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.123+0000 7fad32600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad2c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad32600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0b4860 con 0x7fad2c0cf5c0 2024-03-06T07:37:35.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad32600640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0b4b40 con 0x7fad2c071050 2024-03-06T07:37:35.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad32600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0ce160 con 0x7fad2c0b3c90 2024-03-06T07:37:35.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad2aa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fad2c0cf5c0 0x7fad2c0b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54184/0 (socket says 172.21.15.6:54184) 2024-03-06T07:37:35.126 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad2aa00640 1 -- 172.21.15.6:0/370408544 learned_addr learned my addr 172.21.15.6:0/370408544 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:35.127 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad2b400640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad2c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.124+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810422938 0 0) 0x7fad2c0b4860 con 0x7fad2c0cf5c0 2024-03-06T07:37:35.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad10003520 con 0x7fad2c0cf5c0 2024-03-06T07:37:35.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4227750269 0 0) 0x7fad10003520 con 0x7fad2c0cf5c0 2024-03-06T07:37:35.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 msgr2=0x7fad2c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad2c0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 >> v1:172.21.15.178:6789/0 conn(0x7fad2c071050 legacy=0x7fad2c0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0b4e20 con 0x7fad2c0cf5c0 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fad18001670 con 0x7fad2c0cf5c0 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 >> v1:172.21.15.6:6789/0 conn(0x7fad2c0cf5c0 legacy=0x7fad2c0b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad10003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad10004020 0x7fad10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0ce160 con 0x7fad10004d40 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1000bc10 con 0x7fad10004020 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.125+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1000be10 con 0x7fad2c0b3c90 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2be00640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2b400640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad10003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2aa00640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad10004020 0x7fad10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2be00640 1 -- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 msgr2=0x7fad10003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2be00640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad10003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2be00640 1 -- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad10004020 msgr2=0x7fad10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2be00640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad10004020 0x7fad10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.129 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2be00640 1 -- 172.21.15.6:0/370408544 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad10003520 con 0x7fad10004d40 2024-03-06T07:37:35.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.126+0000 7fad2b400640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0b3c90 0x7fad10003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:35.130 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.127+0000 7fad2be00640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad100051c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fad24001e50 tx=0x7fad10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94526262882b3ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24001a70 con 0x7fad10004d40 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad24003b70 con 0x7fad10004d40 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad2a000640 1 -- 172.21.15.6:0/370408544 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad240050d0 con 0x7fad10004d40 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad32600640 1 -- 172.21.15.6:0/370408544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 msgr2=0x7fad100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad32600640 1 --2- 172.21.15.6:0/370408544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad100051c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fad24001e50 tx=0x7fad10005830 comp rx=0 tx=0).stop 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad2be00640 1 -- 172.21.15.6:0/370408544 reap_dead start 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad32600640 1 -- 172.21.15.6:0/370408544 shutdown_connections 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad32600640 1 -- 172.21.15.6:0/370408544 >> 172.21.15.6:0/370408544 conn(0x7fad2c06afb0 msgr2=0x7fad2c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.128+0000 7fad32600640 1 -- 172.21.15.6:0/370408544 shutdown_connections 2024-03-06T07:37:35.131 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.129+0000 7fad32600640 1 -- 172.21.15.6:0/370408544 wait complete. 2024-03-06T07:37:35.133 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.131+0000 7fad32600640 1 Processor -- start 2024-03-06T07:37:35.134 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.132+0000 7fad32600640 1 -- start start 2024-03-06T07:37:35.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.133+0000 7fad32600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad2c1f8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.133+0000 7fad32600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad2c1482b0 0x7fad2c1f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.133+0000 7fad32600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0c9150 0x7fad2c0ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.136 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.133+0000 7fad32600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0cdc20 con 0x7fad10004d40 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.133+0000 7fad32600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c15b340 con 0x7fad2c1482b0 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.133+0000 7fad32600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0cdda0 con 0x7fad2c0c9150 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad2c1f8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad2c1f8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44144/0 (socket says 172.21.15.6:44144) 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 -- 172.21.15.6:0/1082633494 learned_addr learned my addr 172.21.15.6:0/1082633494 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 -- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0c9150 msgr2=0x7fad2c0ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0c9150 0x7fad2c0ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 -- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad2c1482b0 msgr2=0x7fad2c1f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2aa00640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad2c1482b0 0x7fad2c1f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad2c1482b0 0x7fad2c1f3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 -- 172.21.15.6:0/1082633494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0c99e0 con 0x7fad10004d40 2024-03-06T07:37:35.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.134+0000 7fad2b400640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad2c1f8400 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fad1c011650 tx=0x7fad1c011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.135+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00f9c0 con 0x7fad10004d40 2024-03-06T07:37:35.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.135+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c00fb60 con 0x7fad10004d40 2024-03-06T07:37:35.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.135+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c015460 con 0x7fad10004d40 2024-03-06T07:37:35.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.135+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c0c9cc0 con 0x7fad10004d40 2024-03-06T07:37:35.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.135+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c1f7ee0 con 0x7fad10004d40 2024-03-06T07:37:35.138 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.136+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c069510 con 0x7fad10004d40 2024-03-06T07:37:35.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.138+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad1c015600 con 0x7fad10004d40 2024-03-06T07:37:35.140 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.139+0000 7fad23400640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fad0c06b500 0x7fad0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:35.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.139+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad1c08ab30 con 0x7fad10004d40 2024-03-06T07:37:35.141 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.139+0000 7fad2aa00640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fad0c06b500 0x7fad0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:35.142 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.140+0000 7fad2aa00640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fad0c06b500 0x7fad0c06d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1521c0 tx=0x7fad18009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:35.146 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.144+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad1c019030 con 0x7fad10004d40 2024-03-06T07:37:35.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.379+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad2c15a650 con 0x7fad10004d40 2024-03-06T07:37:35.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.380+0000 7fad23400640 1 -- 172.21.15.6:0/1082633494 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fad2c15a650 con 0x7fad10004d40 2024-03-06T07:37:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:35.383 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:35.383 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fad0c06b500 msgr2=0x7fad0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fad0c06b500 0x7fad0c06d9c0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fad2c1521c0 tx=0x7fad18009040 comp rx=0 tx=0).stop 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 msgr2=0x7fad2c1f8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad2c1f8400 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fad1c011650 tx=0x7fad1c011b10 comp rx=0 tx=0).stop 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 shutdown_connections 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fad0c06b500 0x7fad0c06d9c0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fad2c1482b0 0x7fad2c1f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fad2c0c9150 0x7fad2c0ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 --2- 172.21.15.6:0/1082633494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fad10004d40 0x7fad2c1f8400 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:35.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 >> 172.21.15.6:0/1082633494 conn(0x7fad2c06afb0 msgr2=0x7fad2c155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:35.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 shutdown_connections 2024-03-06T07:37:35.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:35.383+0000 7fad32600640 1 -- 172.21.15.6:0/1082633494 wait complete. 2024-03-06T07:37:35.603 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:35.603 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:36.148 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:36.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.829+0000 7fdf1f200640 1 Processor -- start 2024-03-06T07:37:36.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.829+0000 7fdf1f200640 1 -- start start 2024-03-06T07:37:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 0x7fdf200b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fdf20070f10 0x7fdf200cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200714d0 con 0x7fdf200cce90 2024-03-06T07:37:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf200cd300 con 0x7fdf20070f10 2024-03-06T07:37:36.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200717b0 con 0x7fdf20145300 2024-03-06T07:37:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdf200cce90 0x7fdf200cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54196/0 (socket says 172.21.15.6:54196) 2024-03-06T07:37:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1e800640 1 -- 172.21.15.6:0/1654931154 learned_addr learned my addr 172.21.15.6:0/1654931154 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf1de00640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 0x7fdf200b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.831+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1654931154 >> v2:172.21.15.178:3300/0 conn(0x7fdf20070f10 0x7fdf200cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784877008 0 0) 0x7fdf200714d0 con 0x7fdf200cce90 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf04003520 con 0x7fdf200cce90 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1814639428 0 0) 0x7fdf04003520 con 0x7fdf200cce90 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 msgr2=0x7fdf200b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 0x7fdf200b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 >> v2:172.21.15.178:3300/0 conn(0x7fdf20070f10 msgr2=0x7fdf200cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 --2- 172.21.15.6:0/1654931154 >> v2:172.21.15.178:3300/0 conn(0x7fdf20070f10 0x7fdf200cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf20071a90 con 0x7fdf200cce90 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fdf180019f0 con 0x7fdf200cce90 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 >> v1:172.21.15.6:6789/0 conn(0x7fdf200cce90 legacy=0x7fdf200cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf040039b0 0x7fdf04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf04004800 0x7fdf04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 0x7fdf040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200cd300 con 0x7fdf04004800 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200717b0 con 0x7fdf040039b0 2024-03-06T07:37:36.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.832+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0400b9c0 con 0x7fdf20145300 2024-03-06T07:37:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf04004800 0x7fdf04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf1de00640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf040039b0 0x7fdf04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf17e00640 1 -- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 msgr2=0x7fdf040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:36.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf20145300 0x7fdf040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf17e00640 1 -- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf040039b0 msgr2=0x7fdf04003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:36.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf040039b0 0x7fdf04003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.833+0000 7fdf17e00640 1 -- 172.21.15.6:0/1654931154 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf04003520 con 0x7fdf04004800 2024-03-06T07:37:36.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.835+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf04004800 0x7fdf04004c80 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009ac0 tx=0x7fdf04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bea05b152aac53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:36.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08039070 con 0x7fdf04004800 2024-03-06T07:37:36.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0802c6d0 con 0x7fdf04004800 2024-03-06T07:37:36.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1d400640 1 -- 172.21.15.6:0/1654931154 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08009690 con 0x7fdf04004800 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1f200640 1 -- 172.21.15.6:0/1654931154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf04004800 msgr2=0x7fdf04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1654931154 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf04004800 0x7fdf04004c80 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009ac0 tx=0x7fdf04005830 comp rx=0 tx=0).stop 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1e800640 1 -- 172.21.15.6:0/1654931154 reap_dead start 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1f200640 1 -- 172.21.15.6:0/1654931154 shutdown_connections 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1f200640 1 -- 172.21.15.6:0/1654931154 >> 172.21.15.6:0/1654931154 conn(0x7fdf2006b3d0 msgr2=0x7fdf2006b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1f200640 1 -- 172.21.15.6:0/1654931154 shutdown_connections 2024-03-06T07:37:36.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.836+0000 7fdf1f200640 1 -- 172.21.15.6:0/1654931154 wait complete. 2024-03-06T07:37:36.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.838+0000 7fdf1f200640 1 Processor -- start 2024-03-06T07:37:36.840 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.838+0000 7fdf1f200640 1 -- start start 2024-03-06T07:37:36.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf1f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf04004800 0x7fdf2015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf1f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 0x7fdf20140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf1f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf2013c210 0x7fdf2006d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf1f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2015d580 con 0x7fdf20141230 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf1f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf201456e0 con 0x7fdf04004800 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf1f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2015d700 con 0x7fdf2013c210 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 0x7fdf20140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf17e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 0x7fdf20140390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44166/0 (socket says 172.21.15.6:44166) 2024-03-06T07:37:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.839+0000 7fdf17e00640 1 -- 172.21.15.6:0/1140057147 learned_addr learned my addr 172.21.15.6:0/1140057147 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf1de00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf04004800 0x7fdf2015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf1e800640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf2013c210 0x7fdf2006d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf17e00640 1 -- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf2013c210 msgr2=0x7fdf2006d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf2013c210 0x7fdf2006d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf17e00640 1 -- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf04004800 msgr2=0x7fdf2015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:36.843 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf04004800 0x7fdf2015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf17e00640 1 -- 172.21.15.6:0/1140057147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf20140c50 con 0x7fdf20141230 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf1de00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf04004800 0x7fdf2015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.840+0000 7fdf17e00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 0x7fdf20140390 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fdf08037040 tx=0x7fdf08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.841+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08042020 con 0x7fdf20141230 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.841+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0803a040 con 0x7fdf20141230 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.841+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08039070 con 0x7fdf20141230 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.841+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf200ad9b0 con 0x7fdf20141230 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.841+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf200adf70 con 0x7fdf20141230 2024-03-06T07:37:36.844 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.842+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2004f330 con 0x7fdf20141230 2024-03-06T07:37:36.845 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.843+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf08045020 con 0x7fdf20141230 2024-03-06T07:37:36.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.844+0000 7fdf16000640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdefc06b4d0 0x7fdefc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:36.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.844+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdf080ab760 con 0x7fdf20141230 2024-03-06T07:37:36.846 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.844+0000 7fdf1de00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdefc06b4d0 0x7fdefc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:36.847 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.846+0000 7fdf1de00640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdefc06b4d0 0x7fdefc06d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006ce0 tx=0x7fdf10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:36.850 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:36.848+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf0804c070 con 0x7fdf20141230 2024-03-06T07:37:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.110+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf200ce050 con 0x7fdf20141230 2024-03-06T07:37:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.111+0000 7fdf16000640 1 -- 172.21.15.6:0/1140057147 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fdf0807a590 con 0x7fdf20141230 2024-03-06T07:37:37.114 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:37.114 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:37.114 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdefc06b4d0 msgr2=0x7fdefc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdefc06b4d0 0x7fdefc06d990 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf10006ce0 tx=0x7fdf10008040 comp rx=0 tx=0).stop 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 msgr2=0x7fdf20140390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 0x7fdf20140390 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fdf08037040 tx=0x7fdf08004730 comp rx=0 tx=0).stop 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 shutdown_connections 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdefc06b4d0 0x7fdefc06d990 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdf2013c210 0x7fdf2006d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdf20141230 0x7fdf20140390 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 --2- 172.21.15.6:0/1140057147 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdf04004800 0x7fdf2015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.114+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 >> 172.21.15.6:0/1140057147 conn(0x7fdf2006b3d0 msgr2=0x7fdf2015c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.115+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 shutdown_connections 2024-03-06T07:37:37.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:37.115+0000 7fdf1f200640 1 -- 172.21.15.6:0/1140057147 wait complete. 2024-03-06T07:37:37.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:37.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:38.150 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:38.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:38.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.562+0000 7facf5200640 1 Processor -- start 2024-03-06T07:37:38.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.563+0000 7facf5200640 1 -- start start 2024-03-06T07:37:38.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.565+0000 7facf5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facf00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7facf5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf00b4860 con 0x7facf00b3c90 2024-03-06T07:37:38.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7facf5200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf00b4b40 con 0x7facf0071050 2024-03-06T07:37:38.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7facf5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf00ce160 con 0x7facf00cf5c0 2024-03-06T07:37:38.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7facede00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7facf00b3c90 0x7facf00b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:54210/0 (socket says 172.21.15.6:54210) 2024-03-06T07:37:38.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7facede00640 1 -- 172.21.15.6:0/268672924 learned_addr learned my addr 172.21.15.6:0/268672924 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:38.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7faced400640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facf00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077420273 0 0) 0x7facf00b4860 con 0x7facf00b3c90 2024-03-06T07:37:38.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.566+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd4003520 con 0x7facf00b3c90 2024-03-06T07:37:38.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268581666 0 0) 0x7facd4003520 con 0x7facf00b3c90 2024-03-06T07:37:38.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 msgr2=0x7facf00cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:38.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facf00cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 >> v1:172.21.15.178:6789/0 conn(0x7facf0071050 legacy=0x7facf00714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf00b4e20 con 0x7facf00b3c90 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7face0001670 con 0x7facf00b3c90 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 >> v1:172.21.15.6:6789/0 conn(0x7facf00b3c90 legacy=0x7facf00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facd4003a50 0x7facd4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4004d40 0x7facd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf00ce160 con 0x7facd4003a50 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd400bb90 con 0x7facd4004d40 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd400bd90 con 0x7facf00cf5c0 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.567+0000 7facede00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facd4003a50 0x7facd4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7faced400640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7facede00640 1 -- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 msgr2=0x7facd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:38.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7facede00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facd4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7facede00640 1 -- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4004d40 msgr2=0x7facd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:38.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7facede00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4004d40 0x7facd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7facede00640 1 -- 172.21.15.6:0/268672924 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4003520 con 0x7facd4003a50 2024-03-06T07:37:38.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.568+0000 7faced400640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf00cf5c0 0x7facd4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:38.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.569+0000 7facede00640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facd4003a50 0x7facd4003e70 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7face000c210 tx=0x7facd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8974e6d68defe9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.569+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0007d10 con 0x7facd4003a50 2024-03-06T07:37:38.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.569+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face0030070 con 0x7facd4003a50 2024-03-06T07:37:38.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.569+0000 7faceca00640 1 -- 172.21.15.6:0/268672924 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face0005d30 con 0x7facd4003a50 2024-03-06T07:37:38.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facf5200640 1 -- 172.21.15.6:0/268672924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facd4003a50 msgr2=0x7facd4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facf5200640 1 --2- 172.21.15.6:0/268672924 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facd4003a50 0x7facd4003e70 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7face000c210 tx=0x7facd4005830 comp rx=0 tx=0).stop 2024-03-06T07:37:38.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facee800640 1 -- 172.21.15.6:0/268672924 reap_dead start 2024-03-06T07:37:38.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facf5200640 1 -- 172.21.15.6:0/268672924 shutdown_connections 2024-03-06T07:37:38.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facf5200640 1 -- 172.21.15.6:0/268672924 >> 172.21.15.6:0/268672924 conn(0x7facf006afb0 msgr2=0x7facf006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:38.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facf5200640 1 -- 172.21.15.6:0/268672924 shutdown_connections 2024-03-06T07:37:38.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.570+0000 7facf5200640 1 -- 172.21.15.6:0/268672924 wait complete. 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.572+0000 7facf5200640 1 Processor -- start 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.572+0000 7facf5200640 1 -- start start 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facf5200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4003a50 0x7facf01e4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facf5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf009aa10 0x7facf009bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facf5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 0x7facf0155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facf5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf00cdc20 con 0x7facf01553f0 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facf5200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf015b340 con 0x7facd4003a50 2024-03-06T07:37:38.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facf5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf00cdda0 con 0x7facf009aa10 2024-03-06T07:37:38.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facee800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 0x7facf0155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facee800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 0x7facf0155880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44176/0 (socket says 172.21.15.6:44176) 2024-03-06T07:37:38.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facee800640 1 -- 172.21.15.6:0/1177886950 learned_addr learned my addr 172.21.15.6:0/1177886950 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.573+0000 7facede00640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4003a50 0x7facf01e4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7faced400640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf009aa10 0x7facf009bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facee800640 1 -- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf009aa10 msgr2=0x7facf009bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facee800640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf009aa10 0x7facf009bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facee800640 1 -- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4003a50 msgr2=0x7facf01e4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facee800640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4003a50 0x7facf01e4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facee800640 1 -- 172.21.15.6:0/1177886950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf009cbe0 con 0x7facf01553f0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7faced400640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf009aa10 0x7facf009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facee800640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 0x7facf0155880 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7facf00cf9a0 tx=0x7face800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8017070 con 0x7facf01553f0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8005d40 con 0x7facf01553f0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face800a850 con 0x7facf01553f0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.574+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf01e5530 con 0x7facf01553f0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.575+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf009b180 con 0x7facf01553f0 2024-03-06T07:37:38.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.576+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf004f3a0 con 0x7facf01553f0 2024-03-06T07:37:38.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.578+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7face800a9f0 con 0x7facf01553f0 2024-03-06T07:37:38.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.579+0000 7face6000640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7facd006b4d0 0x7facd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.579+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7face80894b0 con 0x7facf01553f0 2024-03-06T07:37:38.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.579+0000 7facede00640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7facd006b4d0 0x7facd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:38.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.580+0000 7facede00640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7facd006b4d0 0x7facd006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7face0034000 tx=0x7face0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:38.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.583+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7facf009cbe0 con 0x7facf01553f0 2024-03-06T07:37:38.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.820+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facf009c480 con 0x7facf01553f0 2024-03-06T07:37:38.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.821+0000 7face6000640 1 -- 172.21.15.6:0/1177886950 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7face8058250 con 0x7facf01553f0 2024-03-06T07:37:38.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:38.824 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:38.824 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:38.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7facd006b4d0 msgr2=0x7facd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7facd006b4d0 0x7facd006d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7face0034000 tx=0x7face0009030 comp rx=0 tx=0).stop 2024-03-06T07:37:38.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 msgr2=0x7facf0155880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 0x7facf0155880 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7facf00cf9a0 tx=0x7face800cb90 comp rx=0 tx=0).stop 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 shutdown_connections 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7facd006b4d0 0x7facd006d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7facf01553f0 0x7facf0155880 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7facf009aa10 0x7facf009bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 --2- 172.21.15.6:0/1177886950 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7facd4003a50 0x7facf01e4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 >> 172.21.15.6:0/1177886950 conn(0x7facf006afb0 msgr2=0x7facf00ae160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 shutdown_connections 2024-03-06T07:37:38.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:38.824+0000 7facf5200640 1 -- 172.21.15.6:0/1177886950 wait complete. 2024-03-06T07:37:39.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:39.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:40.047 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:40.152 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:40.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.265+0000 7f50b1a00640 1 Processor -- start 2024-03-06T07:37:40.267 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.265+0000 7f50b1a00640 1 -- start start 2024-03-06T07:37:40.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50b1a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac15e0b0 0x7f50ac0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50b1a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50ac0d4a10 0x7f50ac0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50b1a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f50ac0d0c80 0x7f50ac0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.269 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50b1a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ac0b09e0 con 0x7f50ac0d4a10 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50b1a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ac0b3710 con 0x7f50ac0d0c80 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50b1a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac0b39f0 con 0x7f50ac15e0b0 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50ab400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50ac0d4a10 0x7f50ac0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50ab400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f50ac0d4a10 0x7f50ac0d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44192/0 (socket says 172.21.15.6:44192) 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.267+0000 7f50ab400640 1 -- 172.21.15.6:0/1739326795 learned_addr learned my addr 172.21.15.6:0/1739326795 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50abe00640 1 --2- 172.21.15.6:0/1739326795 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac15e0b0 0x7f50ac0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.270 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50b1000640 1 --2- 172.21.15.6:0/1739326795 >> v2:172.21.15.178:3300/0 conn(0x7f50ac0d0c80 0x7f50ac0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50ab400640 1 -- 172.21.15.6:0/1739326795 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac15e0b0 msgr2=0x7f50ac0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50ab400640 1 --2- 172.21.15.6:0/1739326795 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac15e0b0 0x7f50ac0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50ab400640 1 -- 172.21.15.6:0/1739326795 >> v2:172.21.15.178:3300/0 conn(0x7f50ac0d0c80 msgr2=0x7f50ac0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50ab400640 1 --2- 172.21.15.6:0/1739326795 >> v2:172.21.15.178:3300/0 conn(0x7f50ac0d0c80 0x7f50ac0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.268+0000 7f50ab400640 1 -- 172.21.15.6:0/1739326795 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac0b3cd0 con 0x7f50ac0d4a10 2024-03-06T07:37:40.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.269+0000 7f50ab400640 1 --2- 172.21.15.6:0/1739326795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0d0740 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0d06c0 tx=0x7f509802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1210f5ffe704ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.269+0000 7f50aaa00640 1 -- 172.21.15.6:0/1739326795 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098035020 con 0x7f50ac0d4a10 2024-03-06T07:37:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.269+0000 7f50aaa00640 1 -- 172.21.15.6:0/1739326795 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5098031870 con 0x7f50ac0d4a10 2024-03-06T07:37:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.269+0000 7f50aaa00640 1 -- 172.21.15.6:0/1739326795 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098031c40 con 0x7f50ac0d4a10 2024-03-06T07:37:40.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 -- 172.21.15.6:0/1739326795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 msgr2=0x7f50ac0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 --2- 172.21.15.6:0/1739326795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0d0740 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f50ac0d06c0 tx=0x7f509802b910 comp rx=0 tx=0).stop 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 -- 172.21.15.6:0/1739326795 shutdown_connections 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 --2- 172.21.15.6:0/1739326795 >> v2:172.21.15.178:3300/0 conn(0x7f50ac0d0c80 0x7f50ac0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 --2- 172.21.15.6:0/1739326795 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0d0740 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 --2- 172.21.15.6:0/1739326795 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac15e0b0 0x7f50ac0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 -- 172.21.15.6:0/1739326795 >> 172.21.15.6:0/1739326795 conn(0x7f50ac0d3d30 msgr2=0x7f50ac0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 -- 172.21.15.6:0/1739326795 shutdown_connections 2024-03-06T07:37:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.270+0000 7f50b1a00640 1 -- 172.21.15.6:0/1739326795 wait complete. 2024-03-06T07:37:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.272+0000 7f50b1a00640 1 Processor -- start 2024-03-06T07:37:40.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.272+0000 7f50b1a00640 1 -- start start 2024-03-06T07:37:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50b1a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac0d0c80 0x7f50ac15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50b1a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50b1a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50ac15e0b0 0x7f50ac0cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50b1a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac0cfc60 con 0x7f50ac0d4a10 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50b1a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac15e490 con 0x7f50ac15e0b0 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50b1a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac0cf850 con 0x7f50ac0d0c80 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.273+0000 7f50ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44196/0 (socket says 172.21.15.6:44196) 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 -- 172.21.15.6:0/3779474587 learned_addr learned my addr 172.21.15.6:0/3779474587 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50abe00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac0d0c80 0x7f50ac15c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50b1000640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50ac15e0b0 0x7f50ac0cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 -- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac0d0c80 msgr2=0x7f50ac15c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac0d0c80 0x7f50ac15c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 -- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50ac15e0b0 msgr2=0x7f50ac0cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50ac15e0b0 0x7f50ac0cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 -- 172.21.15.6:0/3779474587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac0b7500 con 0x7f50ac0d4a10 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50abe00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac0d0c80 0x7f50ac15c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:40.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.274+0000 7f50ab400640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0cae00 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f5098000c00 tx=0x7f50980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.275+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098035040 con 0x7f50ac0d4a10 2024-03-06T07:37:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.275+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5098030d00 con 0x7f50ac0d4a10 2024-03-06T07:37:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.275+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac0b77e0 con 0x7f50ac0d4a10 2024-03-06T07:37:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.275+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50980456a0 con 0x7f50ac0d4a10 2024-03-06T07:37:40.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.275+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac0b7da0 con 0x7f50ac0d4a10 2024-03-06T07:37:40.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.276+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ac04f330 con 0x7f50ac0d4a10 2024-03-06T07:37:40.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.277+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5098038020 con 0x7f50ac0d4a10 2024-03-06T07:37:40.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.279+0000 7f50a3e00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f508c06b4d0 0x7f508c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:40.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.279+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f50980ac630 con 0x7f50ac0d4a10 2024-03-06T07:37:40.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.280+0000 7f50abe00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f508c06b4d0 0x7f508c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:40.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.281+0000 7f50abe00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f508c06b4d0 0x7f508c06d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f509c006fd0 tx=0x7f509c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:40.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.283+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f509807b460 con 0x7f50ac0d4a10 2024-03-06T07:37:40.560 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.558+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50ac0b5360 con 0x7f50ac0d4a10 2024-03-06T07:37:40.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.559+0000 7f50a3e00640 1 -- 172.21.15.6:0/3779474587 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f509807ad60 con 0x7f50ac0d4a10 2024-03-06T07:37:40.561 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:40.562 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:40.562 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f508c06b4d0 msgr2=0x7f508c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f508c06b4d0 0x7f508c06d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f509c006fd0 tx=0x7f509c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 msgr2=0x7f50ac0cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0cae00 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f5098000c00 tx=0x7f50980046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 shutdown_connections 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f508c06b4d0 0x7f508c06d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50ac15e0b0 0x7f50ac0cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f50ac0d4a10 0x7f50ac0cae00 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 --2- 172.21.15.6:0/3779474587 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f50ac0d0c80 0x7f50ac15c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 >> 172.21.15.6:0/3779474587 conn(0x7f50ac0d3d30 msgr2=0x7f50ac0b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 shutdown_connections 2024-03-06T07:37:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:40.562+0000 7f50b1a00640 1 -- 172.21.15.6:0/3779474587 wait complete. 2024-03-06T07:37:40.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:40.785 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:41.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.008+0000 7f9efba00640 1 Processor -- start 2024-03-06T07:37:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.008+0000 7f9efba00640 1 -- start start 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9efba00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ef415dd00 0x7f9ef4162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9efba00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9ef406c450 0x7f9ef415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9efba00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef415ed30 con 0x7f9ef406bfc0 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9efba00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef4161a80 con 0x7f9ef415dd00 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9efba00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef415efa0 con 0x7f9ef406c450 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9ef9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ef415dd00 0x7f9ef4162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9ef9000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ef415dd00 0x7f9ef4162520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37460/0 (socket says 172.21.15.6:37460) 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9ef9000640 1 -- 172.21.15.6:0/3372451896 learned_addr learned my addr 172.21.15.6:0/3372451896 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9ef9a00640 1 --2- 172.21.15.6:0/3372451896 >> v2:172.21.15.178:3300/0 conn(0x7f9ef406c450 0x7f9ef415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.010+0000 7f9ef9000640 1 -- 172.21.15.6:0/3372451896 >> v2:172.21.15.178:3300/0 conn(0x7f9ef406c450 msgr2=0x7f9ef415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:42.013 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.011+0000 7f9ef9000640 1 --2- 172.21.15.6:0/3372451896 >> v2:172.21.15.178:3300/0 conn(0x7f9ef406c450 0x7f9ef415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.011+0000 7f9ef9000640 1 -- 172.21.15.6:0/3372451896 >> v1:172.21.15.178:6789/0 conn(0x7f9ef406bfc0 legacy=0x7f9ef415e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:42.014 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.011+0000 7f9ef9000640 1 -- 172.21.15.6:0/3372451896 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef415f280 con 0x7f9ef415dd00 2024-03-06T07:37:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.012+0000 7f9ef9000640 1 --2- 172.21.15.6:0/3372451896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef415dd00 0x7f9ef4162520 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9ef415f560 tx=0x7f9edc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0ae4199813bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.012+0000 7f9ef3e00640 1 -- 172.21.15.6:0/3372451896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc02e070 con 0x7f9ef415dd00 2024-03-06T07:37:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.012+0000 7f9ef3e00640 1 -- 172.21.15.6:0/3372451896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc037070 con 0x7f9ef415dd00 2024-03-06T07:37:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.012+0000 7f9ef3e00640 1 -- 172.21.15.6:0/3372451896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc031a00 con 0x7f9ef415dd00 2024-03-06T07:37:42.015 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 -- 172.21.15.6:0/3372451896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef415dd00 msgr2=0x7f9ef4162520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 --2- 172.21.15.6:0/3372451896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef415dd00 0x7f9ef4162520 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9ef415f560 tx=0x7f9edc02b880 comp rx=0 tx=0).stop 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 -- 172.21.15.6:0/3372451896 shutdown_connections 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 --2- 172.21.15.6:0/3372451896 >> v2:172.21.15.178:3300/0 conn(0x7f9ef406c450 0x7f9ef415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 --2- 172.21.15.6:0/3372451896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef415dd00 0x7f9ef4162520 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 -- 172.21.15.6:0/3372451896 >> 172.21.15.6:0/3372451896 conn(0x7f9ef40714d0 msgr2=0x7f9ef40718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 -- 172.21.15.6:0/3372451896 shutdown_connections 2024-03-06T07:37:42.016 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.013+0000 7f9efba00640 1 -- 172.21.15.6:0/3372451896 wait complete. 2024-03-06T07:37:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.015+0000 7f9efba00640 1 Processor -- start 2024-03-06T07:37:42.017 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.015+0000 7f9efba00640 1 -- start start 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9efba00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 0x7f9ef4158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9efba00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ef406c450 0x7f9ef40af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9efba00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ef415dd00 0x7f9ef40af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9efba00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef406c870 con 0x7f9ef406bfc0 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9efba00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4162000 con 0x7f9ef406c450 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9efba00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef406f280 con 0x7f9ef415dd00 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9ef9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 0x7f9ef4158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9ef9000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 0x7f9ef4158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37464/0 (socket says 172.21.15.6:37464) 2024-03-06T07:37:42.020 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9ef9000640 1 -- 172.21.15.6:0/3048327521 learned_addr learned my addr 172.21.15.6:0/3048327521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9eebe00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ef406c450 0x7f9ef40af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.016+0000 7f9ef9a00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ef415dd00 0x7f9ef40af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef9000640 1 -- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ef415dd00 msgr2=0x7f9ef40af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef9000640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ef415dd00 0x7f9ef40af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef9000640 1 -- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ef406c450 msgr2=0x7f9ef40af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef9000640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ef406c450 0x7f9ef40af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef9000640 1 -- 172.21.15.6:0/3048327521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef41586c0 con 0x7f9ef406bfc0 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef9000640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 0x7f9ef4158180 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f9edc000c00 tx=0x7f9edc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc032880 con 0x7f9ef406bfc0 2024-03-06T07:37:42.021 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc032a20 con 0x7f9ef406bfc0 2024-03-06T07:37:42.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc032e10 con 0x7f9ef406bfc0 2024-03-06T07:37:42.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.017+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef40c6e80 con 0x7f9ef406bfc0 2024-03-06T07:37:42.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.018+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef40c7440 con 0x7f9ef406bfc0 2024-03-06T07:37:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.019+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef404f3a0 con 0x7f9ef406bfc0 2024-03-06T07:37:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.020+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9edc04c060 con 0x7f9ef406bfc0 2024-03-06T07:37:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.020+0000 7f9ef1600640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ed006b4d0 0x7f9ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.021+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9edc0abe10 con 0x7f9ef406bfc0 2024-03-06T07:37:42.023 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.021+0000 7f9eebe00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ed006b4d0 0x7f9ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:42.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.022+0000 7f9eebe00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ed006b4d0 0x7f9ed006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40afc00 tx=0x7f9ee4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:42.031 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.029+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9edc0af030 con 0x7f9ef406bfc0 2024-03-06T07:37:42.154 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:42.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.314+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ef40724c0 con 0x7f9ef406bfc0 2024-03-06T07:37:42.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.315+0000 7f9ef1600640 1 -- 172.21.15.6:0/3048327521 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f9edc040070 con 0x7f9ef406bfc0 2024-03-06T07:37:42.317 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:42.317 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:42.318 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.317+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ed006b4d0 msgr2=0x7f9ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.317+0000 7f9efba00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ed006b4d0 0x7f9ed006d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40afc00 tx=0x7f9ee4009040 comp rx=0 tx=0).stop 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.317+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 msgr2=0x7f9ef4158180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.317+0000 7f9efba00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 0x7f9ef4158180 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f9edc000c00 tx=0x7f9edc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 shutdown_connections 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9ed006b4d0 0x7f9ed006d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ef406c450 0x7f9ef40af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ef415dd00 0x7f9ef40af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 --2- 172.21.15.6:0/3048327521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ef406bfc0 0x7f9ef4158180 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 >> 172.21.15.6:0/3048327521 conn(0x7f9ef40714d0 msgr2=0x7f9ef4159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 shutdown_connections 2024-03-06T07:37:42.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:42.318+0000 7f9efba00640 1 -- 172.21.15.6:0/3048327521 wait complete. 2024-03-06T07:37:42.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:42.541 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:43.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:43.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.767+0000 7f2413e00640 1 Processor -- start 2024-03-06T07:37:43.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.767+0000 7f2413e00640 1 -- start start 2024-03-06T07:37:43.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2413e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f24140d0e50 0x7f2414168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2413e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140b3710 con 0x7f24140d4a10 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2413e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140b39f0 con 0x7f241415e0b0 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2413e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24140b09e0 con 0x7f24140d0e50 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2412000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f24140d4a10 0x7f24140d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34702/0 (socket says 172.21.15.6:34702) 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2412000640 1 -- 172.21.15.6:0/2443219316 learned_addr learned my addr 172.21.15.6:0/2443219316 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.769+0000 7f2413400640 1 --2- 172.21.15.6:0/2443219316 >> v2:172.21.15.6:3300/0 conn(0x7f24140d0e50 0x7f2414168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227517082 0 0) 0x7f24140b3710 con 0x7f24140d4a10 2024-03-06T07:37:43.772 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f8003520 con 0x7f24140d4a10 2024-03-06T07:37:43.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 264853447 0 0) 0x7f23f8003520 con 0x7f24140d4a10 2024-03-06T07:37:43.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 >> v2:172.21.15.6:3300/0 conn(0x7f24140d0e50 msgr2=0x7f2414168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 --2- 172.21.15.6:0/2443219316 >> v2:172.21.15.6:3300/0 conn(0x7f24140d0e50 0x7f2414168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 >> v1:172.21.15.178:6789/0 conn(0x7f241415e0b0 legacy=0x7f24140cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140b3cd0 con 0x7f24140d4a10 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f2400001670 con 0x7f24140d4a10 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 >> v1:172.21.15.6:6789/0 conn(0x7f24140d4a10 legacy=0x7f24140d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f8003a00 0x7f23f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f23f8004800 0x7f23f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23f80051c0 0x7f23f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f800c090 con 0x7f23f80051c0 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f800c240 con 0x7f23f8003a00 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.770+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f800c440 con 0x7f23f8004800 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2413400640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23f80051c0 0x7f23f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2412000640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f23f8004800 0x7f23f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2412a00640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f8003a00 0x7f23f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2413400640 1 -- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f23f8004800 msgr2=0x7f23f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2413400640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f23f8004800 0x7f23f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2413400640 1 -- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f8003a00 msgr2=0x7f23f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2413400640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f8003a00 0x7f23f8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2413400640 1 -- 172.21.15.6:0/2443219316 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8003520 con 0x7f23f80051c0 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.771+0000 7f2412000640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f23f8004800 0x7f23f8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.772+0000 7f2413400640 1 -- 172.21.15.6:0/2443219316 reap_dead start 2024-03-06T07:37:43.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.772+0000 7f2413400640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23f80051c0 0x7f23f800b9c0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f240c009f90 tx=0x7f23f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b7bf23f5f01c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c009c10 con 0x7f23f80051c0 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c02cc20 con 0x7f23f80051c0 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2411600640 1 -- 172.21.15.6:0/2443219316 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c003990 con 0x7f23f80051c0 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2413e00640 1 -- 172.21.15.6:0/2443219316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23f80051c0 msgr2=0x7f23f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2413e00640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23f80051c0 0x7f23f800b9c0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f240c009f90 tx=0x7f23f8005830 comp rx=0 tx=0).stop 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2413e00640 1 -- 172.21.15.6:0/2443219316 shutdown_connections 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2413e00640 1 --2- 172.21.15.6:0/2443219316 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f23f80051c0 0x7f23f800b9c0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2413e00640 1 -- 172.21.15.6:0/2443219316 >> 172.21.15.6:0/2443219316 conn(0x7f24140d3d30 msgr2=0x7f24140d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.773+0000 7f2413e00640 1 -- 172.21.15.6:0/2443219316 shutdown_connections 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.774+0000 7f2413e00640 1 -- 172.21.15.6:0/2443219316 wait complete. 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.775+0000 7f2413e00640 1 Processor -- start 2024-03-06T07:37:43.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.775+0000 7f2413e00640 1 -- start start 2024-03-06T07:37:43.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.776+0000 7f2413e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f80051c0 0x7f24141593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.776+0000 7f2413e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 0x7f24140a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.776+0000 7f2413e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24140a0a60 0x7f24140cc980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.776+0000 7f2413e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24140cfc60 con 0x7f241415b200 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.776+0000 7f2413e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241415e490 con 0x7f23f80051c0 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.776+0000 7f2413e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24140d0740 con 0x7f24140a0a60 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 0x7f24140a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 0x7f24140a0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37488/0 (socket says 172.21.15.6:37488) 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 -- 172.21.15.6:0/1987875725 learned_addr learned my addr 172.21.15.6:0/1987875725 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 -- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24140a0a60 msgr2=0x7f24140cc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24140a0a60 0x7f24140cc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 -- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f80051c0 msgr2=0x7f24141593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f80051c0 0x7f24141593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.777+0000 7f2412000640 1 -- 172.21.15.6:0/1987875725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24140cd720 con 0x7f241415b200 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.778+0000 7f2412000640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 0x7f24140a0520 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f240000e700 tx=0x7f240000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.781 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.778+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240000c660 con 0x7f241415b200 2024-03-06T07:37:43.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.778+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2400018070 con 0x7f241415b200 2024-03-06T07:37:43.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.778+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240000ca00 con 0x7f241415b200 2024-03-06T07:37:43.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.778+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241409ddf0 con 0x7f241415b200 2024-03-06T07:37:43.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.778+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241409e3b0 con 0x7f241415b200 2024-03-06T07:37:43.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.779+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241404f330 con 0x7f241415b200 2024-03-06T07:37:43.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.781+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f240000cba0 con 0x7f241415b200 2024-03-06T07:37:43.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.782+0000 7f240aa00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f23f406b4d0 0x7f23f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:43.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.782+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2400089fe0 con 0x7f241415b200 2024-03-06T07:37:43.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.783+0000 7f2412a00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f23f406b4d0 0x7f23f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:43.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.783+0000 7f2412a00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f23f406b4d0 0x7f23f406d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2404007fd0 tx=0x7f2404009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:43.789 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:43.787+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2400058dc0 con 0x7f241415b200 2024-03-06T07:37:44.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.030+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24140a0eb0 con 0x7f241415b200 2024-03-06T07:37:44.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.031+0000 7f240aa00640 1 -- 172.21.15.6:0/1987875725 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f24000586c0 con 0x7f241415b200 2024-03-06T07:37:44.034 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:44.034 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:44.034 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f23f406b4d0 msgr2=0x7f23f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.037 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f23f406b4d0 0x7f23f406d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2404007fd0 tx=0x7f2404009040 comp rx=0 tx=0).stop 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 msgr2=0x7f24140a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 0x7f24140a0520 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f240000e700 tx=0x7f240000ebc0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 shutdown_connections 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f23f406b4d0 0x7f23f406d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f24140a0a60 0x7f24140cc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f241415b200 0x7f24140a0520 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 --2- 172.21.15.6:0/1987875725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f23f80051c0 0x7f24141593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 >> 172.21.15.6:0/1987875725 conn(0x7f24140d3d30 msgr2=0x7f24140cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 shutdown_connections 2024-03-06T07:37:44.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:44.035+0000 7f2413e00640 1 -- 172.21.15.6:0/1987875725 wait complete. 2024-03-06T07:37:44.156 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:44.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:44.259 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:45.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:45.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.483+0000 7fc4dfc00640 1 Processor -- start 2024-03-06T07:37:45.486 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.483+0000 7fc4dfc00640 1 -- start start 2024-03-06T07:37:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.485+0000 7fc4dfc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc4d80af1a0 0x7fc4d815c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.485+0000 7fc4dfc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4d80cdc20 0x7fc4d80b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.485+0000 7fc4dfc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d815cc40 con 0x7fc4d80af580 2024-03-06T07:37:45.487 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.485+0000 7fc4dfc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d815cf20 con 0x7fc4d80cdc20 2024-03-06T07:37:45.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.485+0000 7fc4dfc00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d815d200 con 0x7fc4d80af1a0 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.486+0000 7fc4ddc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4d80cdc20 0x7fc4d80b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.486+0000 7fc4ddc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc4d80cdc20 0x7fc4d80b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37492/0 (socket says 172.21.15.6:37492) 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.486+0000 7fc4ddc00640 1 -- 172.21.15.6:0/821445193 learned_addr learned my addr 172.21.15.6:0/821445193 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.486+0000 7fc4dd200640 1 --2- 172.21.15.6:0/821445193 >> v2:172.21.15.178:3300/0 conn(0x7fc4d80af1a0 0x7fc4d815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.486+0000 7fc4ddc00640 1 -- 172.21.15.6:0/821445193 >> v2:172.21.15.178:3300/0 conn(0x7fc4d80af1a0 msgr2=0x7fc4d815c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.486+0000 7fc4ddc00640 1 --2- 172.21.15.6:0/821445193 >> v2:172.21.15.178:3300/0 conn(0x7fc4d80af1a0 0x7fc4d815c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.487+0000 7fc4ddc00640 1 -- 172.21.15.6:0/821445193 >> v1:172.21.15.178:6789/0 conn(0x7fc4d80af580 legacy=0x7fc4d80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.489 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.487+0000 7fc4ddc00640 1 -- 172.21.15.6:0/821445193 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d815d4e0 con 0x7fc4d80cdc20 2024-03-06T07:37:45.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.489+0000 7fc4ddc00640 1 --2- 172.21.15.6:0/821445193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80cdc20 0x7fc4d80b2dd0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8074960 tx=0x7fc4d002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8890e9fba1d88da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:45.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.489+0000 7fc4d7e00640 1 -- 172.21.15.6:0/821445193 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d002ca80 con 0x7fc4d80cdc20 2024-03-06T07:37:45.491 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.489+0000 7fc4d7e00640 1 -- 172.21.15.6:0/821445193 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d002cc20 con 0x7fc4d80cdc20 2024-03-06T07:37:45.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.489+0000 7fc4d7e00640 1 -- 172.21.15.6:0/821445193 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d00343f0 con 0x7fc4d80cdc20 2024-03-06T07:37:45.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 -- 172.21.15.6:0/821445193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80cdc20 msgr2=0x7fc4d80b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.492 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/821445193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80cdc20 0x7fc4d80b2dd0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8074960 tx=0x7fc4d002bee0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 -- 172.21.15.6:0/821445193 shutdown_connections 2024-03-06T07:37:45.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/821445193 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80cdc20 0x7fc4d80b2dd0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/821445193 >> v2:172.21.15.178:3300/0 conn(0x7fc4d80af1a0 0x7fc4d815c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 -- 172.21.15.6:0/821445193 >> 172.21.15.6:0/821445193 conn(0x7fc4d806afb0 msgr2=0x7fc4d806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:45.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 -- 172.21.15.6:0/821445193 shutdown_connections 2024-03-06T07:37:45.493 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.490+0000 7fc4dfc00640 1 -- 172.21.15.6:0/821445193 wait complete. 2024-03-06T07:37:45.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.493+0000 7fc4dfc00640 1 Processor -- start 2024-03-06T07:37:45.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.493+0000 7fc4dfc00640 1 -- start start 2024-03-06T07:37:45.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dfc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 0x7fc4d814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dfc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc4d80af580 0x7fc4d809aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dfc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d80cdc20 0x7fc4d815d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dfc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80b36a0 con 0x7fc4d80af1a0 2024-03-06T07:37:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dfc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80b3520 con 0x7fc4d80cdc20 2024-03-06T07:37:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dfc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80b3820 con 0x7fc4d80af580 2024-03-06T07:37:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 0x7fc4d814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dd200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 0x7fc4d814e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37506/0 (socket says 172.21.15.6:37506) 2024-03-06T07:37:45.498 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.495+0000 7fc4dd200640 1 -- 172.21.15.6:0/2456402918 learned_addr learned my addr 172.21.15.6:0/2456402918 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4dd200640 1 -- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc4d80af580 msgr2=0x7fc4d809aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4dd200640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc4d80af580 0x7fc4d809aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4dd200640 1 -- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d80cdc20 msgr2=0x7fc4d815d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4dd200640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d80cdc20 0x7fc4d815d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4dd200640 1 -- 172.21.15.6:0/2456402918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d814eab0 con 0x7fc4d80af1a0 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4dd200640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 0x7fc4d814e170 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000e960 tx=0x7fc4c000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.496+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0017070 con 0x7fc4d80af1a0 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.497+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c000ca00 con 0x7fc4d80af1a0 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.497+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d814ed90 con 0x7fc4d80af1a0 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.497+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c000cdf0 con 0x7fc4d80af1a0 2024-03-06T07:37:45.499 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.497+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d814f350 con 0x7fc4d80af1a0 2024-03-06T07:37:45.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.498+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d8069510 con 0x7fc4d80af1a0 2024-03-06T07:37:45.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.499+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4c001a070 con 0x7fc4d80af1a0 2024-03-06T07:37:45.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.500+0000 7fc4d5600640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc4b406b4d0 0x7fc4b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:45.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.500+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc4c008a680 con 0x7fc4d80af1a0 2024-03-06T07:37:45.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.501+0000 7fc4cfe00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc4b406b4d0 0x7fc4b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:45.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.502+0000 7fc4cfe00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc4b406b4d0 0x7fc4b406d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8000f60 tx=0x7fc4c80079d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:45.507 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.505+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc4c00594b0 con 0x7fc4d80af1a0 2024-03-06T07:37:45.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.743+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4d804f3a0 con 0x7fc4d80af1a0 2024-03-06T07:37:45.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.744+0000 7fc4d5600640 1 -- 172.21.15.6:0/2456402918 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fc4c005de20 con 0x7fc4d80af1a0 2024-03-06T07:37:45.746 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:45.746 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:45.747 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:45.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.746+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc4b406b4d0 msgr2=0x7fc4b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.746+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc4b406b4d0 0x7fc4b406d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8000f60 tx=0x7fc4c80079d0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.746+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 msgr2=0x7fc4d814e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.746+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 0x7fc4d814e170 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000e960 tx=0x7fc4c000ee20 comp rx=0 tx=0).stop 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 shutdown_connections 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc4b406b4d0 0x7fc4b406d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d80cdc20 0x7fc4d815d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc4d80af580 0x7fc4d809aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 --2- 172.21.15.6:0/2456402918 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc4d80af1a0 0x7fc4d814e170 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 >> 172.21.15.6:0/2456402918 conn(0x7fc4d806afb0 msgr2=0x7fc4d815e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:45.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 shutdown_connections 2024-03-06T07:37:45.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:45.747+0000 7fc4dfc00640 1 -- 172.21.15.6:0/2456402918 wait complete. 2024-03-06T07:37:45.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:45.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:46.158 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:46.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:47.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.203+0000 7f6ad0e00640 1 Processor -- start 2024-03-06T07:37:47.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.203+0000 7f6ad0e00640 1 -- start start 2024-03-06T07:37:47.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.205+0000 7f6ad0e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6acc0b2340 0x7f6acc168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.205+0000 7f6ad0e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc0d1120 con 0x7f6acc15e2a0 2024-03-06T07:37:47.207 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.205+0000 7f6ad0e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc0d1400 con 0x7f6acc15dec0 2024-03-06T07:37:47.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.205+0000 7f6ad0e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc0b0ea0 con 0x7f6acc0b2340 2024-03-06T07:37:47.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.205+0000 7f6acaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6acc15e2a0 0x7f6acc0cef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34712/0 (socket says 172.21.15.6:34712) 2024-03-06T07:37:47.208 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.205+0000 7f6acaa00640 1 -- 172.21.15.6:0/1745935974 learned_addr learned my addr 172.21.15.6:0/1745935974 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:47.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6acbe00640 1 --2- 172.21.15.6:0/1745935974 >> v2:172.21.15.178:3300/0 conn(0x7f6acc0b2340 0x7f6acc168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.209 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363271144 0 0) 0x7f6acc0d1120 con 0x7f6acc15e2a0 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab0003520 con 0x7f6acc15e2a0 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 982801429 0 0) 0x7f6ab0003520 con 0x7f6acc15e2a0 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 >> v2:172.21.15.178:3300/0 conn(0x7f6acc0b2340 msgr2=0x7f6acc168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 --2- 172.21.15.6:0/1745935974 >> v2:172.21.15.178:3300/0 conn(0x7f6acc0b2340 0x7f6acc168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 >> v1:172.21.15.178:6789/0 conn(0x7f6acc15dec0 legacy=0x7f6acc0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0b0b30 con 0x7f6acc15e2a0 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f6ab8001670 con 0x7f6acc15e2a0 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 >> v1:172.21.15.6:6789/0 conn(0x7f6acc15e2a0 legacy=0x7f6acc0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.206+0000 7f6aca000640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab00039b0 0x7f6ab0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6aca000640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ab0004800 0x7f6ab0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6aca000640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00051c0 0x7f6ab000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b0ea0 con 0x7f6ab00039b0 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab000c060 con 0x7f6ab0004800 2024-03-06T07:37:47.210 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab000c260 con 0x7f6ab00051c0 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acb400640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab00039b0 0x7f6ab0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acaa00640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ab0004800 0x7f6ab0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acb400640 1 -- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00051c0 msgr2=0x7f6ab000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acb400640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00051c0 0x7f6ab000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acb400640 1 -- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ab0004800 msgr2=0x7f6ab0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acb400640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ab0004800 0x7f6ab0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acb400640 1 -- 172.21.15.6:0/1745935974 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0003520 con 0x7f6ab00039b0 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acbe00640 1 -- 172.21.15.6:0/1745935974 reap_dead start 2024-03-06T07:37:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.207+0000 7f6acaa00640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6ab0004800 0x7f6ab0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:47.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.208+0000 7f6acb400640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab00039b0 0x7f6ab0003dd0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f6abc001750 tx=0x7f6ab0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9c69147afeaa08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc001370 con 0x7f6ab00039b0 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc0052d0 con 0x7f6ab00039b0 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6aca000640 1 -- 172.21.15.6:0/1745935974 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc003830 con 0x7f6ab00039b0 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6ad0e00640 1 -- 172.21.15.6:0/1745935974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab00039b0 msgr2=0x7f6ab0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab00039b0 0x7f6ab0003dd0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f6abc001750 tx=0x7f6ab0005830 comp rx=0 tx=0).stop 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6ad0e00640 1 -- 172.21.15.6:0/1745935974 shutdown_connections 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/1745935974 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6ab00039b0 0x7f6ab0003dd0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.209+0000 7f6ad0e00640 1 -- 172.21.15.6:0/1745935974 >> 172.21.15.6:0/1745935974 conn(0x7f6acc0d3d40 msgr2=0x7f6acc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.210+0000 7f6ad0e00640 1 -- 172.21.15.6:0/1745935974 shutdown_connections 2024-03-06T07:37:47.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.210+0000 7f6ad0e00640 1 -- 172.21.15.6:0/1745935974 wait complete. 2024-03-06T07:37:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.213+0000 7f6ad0e00640 1 Processor -- start 2024-03-06T07:37:47.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.214+0000 7f6ad0e00640 1 -- start start 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6ad0e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00039b0 0x7f6acc1f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6ad0e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 0x7f6acc0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6ad0e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6acc0b9480 0x7f6acc0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6ad0e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b14b0 con 0x7f6acc0b9040 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6ad0e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b1330 con 0x7f6acc0b9480 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6ad0e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b1630 con 0x7f6ab00039b0 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6acaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 0x7f6acc0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6acaa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 0x7f6acc0c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37518/0 (socket says 172.21.15.6:37518) 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6acaa00640 1 -- 172.21.15.6:0/3553996341 learned_addr learned my addr 172.21.15.6:0/3553996341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6acb400640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00039b0 0x7f6acc1f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6acbe00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6acc0b9480 0x7f6acc0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.216+0000 7f6acaa00640 1 -- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00039b0 msgr2=0x7f6acc1f4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6acaa00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00039b0 0x7f6acc1f4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6acaa00640 1 -- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6acc0b9480 msgr2=0x7f6acc0c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6acaa00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6acc0b9480 0x7f6acc0c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6acaa00640 1 -- 172.21.15.6:0/3553996341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0b5460 con 0x7f6acc0b9040 2024-03-06T07:37:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6acbe00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6acc0b9480 0x7f6acc0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:47.220 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6acaa00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 0x7f6acc0c9310 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800d7a0 tx=0x7f6ab800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab8019070 con 0x7f6acc0b9040 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab800de50 con 0x7f6acc0b9040 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80047e0 con 0x7f6acc0b9040 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc0b5740 con 0x7f6acc0b9040 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.217+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc0b5d00 con 0x7f6acc0b9040 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.219+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc04f330 con 0x7f6acc0b9040 2024-03-06T07:37:47.221 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.220+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ab801c070 con 0x7f6acc0b9040 2024-03-06T07:37:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.221+0000 7f6ac3400640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6aac06b4d0 0x7f6aac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.221+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6ab8089d50 con 0x7f6acc0b9040 2024-03-06T07:37:47.223 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.222+0000 7f6acb400640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6aac06b4d0 0x7f6aac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:47.225 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.223+0000 7f6acb400640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6aac06b4d0 0x7f6aac06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6abc033000 tx=0x7f6abc009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:47.227 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.226+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ab8058af0 con 0x7f6acc0b9040 2024-03-06T07:37:47.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.468+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6acc06c510 con 0x7f6acc0b9040 2024-03-06T07:37:47.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.468+0000 7f6ac3400640 1 -- 172.21.15.6:0/3553996341 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f6ab80583f0 con 0x7f6acc0b9040 2024-03-06T07:37:47.471 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:47.471 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:47.471 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:47.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6aac06b4d0 msgr2=0x7f6aac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6aac06b4d0 0x7f6aac06d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6abc033000 tx=0x7f6abc009030 comp rx=0 tx=0).stop 2024-03-06T07:37:47.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 msgr2=0x7f6acc0c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:47.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 0x7f6acc0c9310 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800d7a0 tx=0x7f6ab800dc60 comp rx=0 tx=0).stop 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 shutdown_connections 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6aac06b4d0 0x7f6aac06d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6acc0b9480 0x7f6acc0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6acc0b9040 0x7f6acc0c9310 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 --2- 172.21.15.6:0/3553996341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6ab00039b0 0x7f6acc1f4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 >> 172.21.15.6:0/3553996341 conn(0x7f6acc0d3d40 msgr2=0x7f6acc15d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 shutdown_connections 2024-03-06T07:37:47.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:47.471+0000 7f6ad0e00640 1 -- 172.21.15.6:0/3553996341 wait complete. 2024-03-06T07:37:47.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:47.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:48.160 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:48.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:48.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.925+0000 7f874f200640 1 Processor -- start 2024-03-06T07:37:48.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.925+0000 7f874f200640 1 -- start start 2024-03-06T07:37:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.927+0000 7f874f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 0x7f87480b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.927+0000 7f874f200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f874809b120 0x7f87480b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.927+0000 7f874f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480b6060 con 0x7f874815f760 2024-03-06T07:37:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.927+0000 7f874f200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874815d9b0 con 0x7f874809b120 2024-03-06T07:37:48.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.927+0000 7f874f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874815dc90 con 0x7f87480b5740 2024-03-06T07:37:48.931 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.927+0000 7f874e800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f874815f760 0x7f874815ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34722/0 (socket says 172.21.15.6:34722) 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874e800640 1 -- 172.21.15.6:0/1085839781 learned_addr learned my addr 172.21.15.6:0/1085839781 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874d400640 1 --2- 172.21.15.6:0/1085839781 >> v2:172.21.15.178:3300/0 conn(0x7f874809b120 0x7f87480b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874de00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 0x7f87480b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158921759 0 0) 0x7f87480b6060 con 0x7f874815f760 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8730003520 con 0x7f874815f760 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1594372844 0 0) 0x7f8730003520 con 0x7f874815f760 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 msgr2=0x7f87480b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 0x7f87480b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 >> v2:172.21.15.178:3300/0 conn(0x7f874809b120 msgr2=0x7f87480b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.928+0000 7f874ca00640 1 --2- 172.21.15.6:0/1085839781 >> v2:172.21.15.178:3300/0 conn(0x7f874809b120 0x7f87480b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874815df70 con 0x7f874815f760 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f87440019f0 con 0x7f874815f760 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 >> v1:172.21.15.6:6789/0 conn(0x7f874815f760 legacy=0x7f874815ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87300039b0 0x7f8730003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8730004800 0x7f8730004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 0x7f87300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874815d9b0 con 0x7f87300039b0 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874815dc90 con 0x7f8730004800 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873000ba10 con 0x7f87480b5740 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874de00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87300039b0 0x7f8730003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874de00640 1 -- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 msgr2=0x7f87300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874de00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87480b5740 0x7f87300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874d400640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8730004800 0x7f8730004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874de00640 1 -- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8730004800 msgr2=0x7f8730004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874de00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8730004800 0x7f8730004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.929+0000 7f874de00640 1 -- 172.21.15.6:0/1085839781 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8730003520 con 0x7f87300039b0 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874de00640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87300039b0 0x7f8730003dd0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f873c009af0 tx=0x7f8730005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a294c6be20b760e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c039070 con 0x7f87300039b0 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c02c710 con 0x7f87300039b0 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874ca00640 1 -- 172.21.15.6:0/1085839781 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c009690 con 0x7f87300039b0 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874f200640 1 -- 172.21.15.6:0/1085839781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87300039b0 msgr2=0x7f8730003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874f200640 1 --2- 172.21.15.6:0/1085839781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87300039b0 0x7f8730003dd0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f873c009af0 tx=0x7f8730005830 comp rx=0 tx=0).stop 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874e800640 1 -- 172.21.15.6:0/1085839781 reap_dead start 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.931+0000 7f874f200640 1 -- 172.21.15.6:0/1085839781 shutdown_connections 2024-03-06T07:37:48.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.932+0000 7f874f200640 1 -- 172.21.15.6:0/1085839781 >> 172.21.15.6:0/1085839781 conn(0x7f87480731f0 msgr2=0x7f874809aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:48.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.932+0000 7f874f200640 1 -- 172.21.15.6:0/1085839781 shutdown_connections 2024-03-06T07:37:48.936 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.932+0000 7f874f200640 1 -- 172.21.15.6:0/1085839781 wait complete. 2024-03-06T07:37:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.933+0000 7f874f200640 1 Processor -- start 2024-03-06T07:37:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 -- start start 2024-03-06T07:37:48.938 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87300039b0 0x7f87481556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87480b0950 0x7f874815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 0x7f8748156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87480b1bd0 con 0x7f87480b0dd0 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874815d660 con 0x7f87480b0950 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.934+0000 7f874f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480b1d50 con 0x7f87300039b0 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 0x7f8748156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 0x7f8748156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37534/0 (socket says 172.21.15.6:37534) 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 -- 172.21.15.6:0/2283803064 learned_addr learned my addr 172.21.15.6:0/2283803064 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874de00640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87300039b0 0x7f87481556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874d400640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87480b0950 0x7f874815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 -- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87300039b0 msgr2=0x7f87481556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87300039b0 0x7f87481556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 -- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87480b0950 msgr2=0x7f874815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87480b0950 0x7f874815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874e800640 1 -- 172.21.15.6:0/2283803064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748154640 con 0x7f87480b0dd0 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.935+0000 7f874d400640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87480b0950 0x7f874815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.936+0000 7f874e800640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 0x7f8748156610 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2670 tx=0x7f874400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.936+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874400a610 con 0x7f87480b0dd0 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.936+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874400a7d0 con 0x7f87480b0dd0 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.936+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8748154920 con 0x7f87480b0dd0 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.936+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874400abc0 con 0x7f87480b0dd0 2024-03-06T07:37:48.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.936+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8748154ee0 con 0x7f87480b0dd0 2024-03-06T07:37:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.938+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874804f330 con 0x7f87480b0dd0 2024-03-06T07:37:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.938+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8744011070 con 0x7f87480b0dd0 2024-03-06T07:37:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.939+0000 7f8742000640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f872c06b500 0x7f872c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:48.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.939+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8744089890 con 0x7f87480b0dd0 2024-03-06T07:37:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.940+0000 7f874de00640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f872c06b500 0x7f872c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:48.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.941+0000 7f874de00640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f872c06b500 0x7f872c06d9c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f873c036000 tx=0x7f873c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:48.946 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:48.944+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8744019060 con 0x7f87480b0dd0 2024-03-06T07:37:49.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.279+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87480708d0 con 0x7f87480b0dd0 2024-03-06T07:37:49.282 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.280+0000 7f8742000640 1 -- 172.21.15.6:0/2283803064 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f8744011e40 con 0x7f87480b0dd0 2024-03-06T07:37:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:49.282 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:49.282 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f872c06b500 msgr2=0x7f872c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f872c06b500 0x7f872c06d9c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f873c036000 tx=0x7f873c030000 comp rx=0 tx=0).stop 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 msgr2=0x7f8748156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 0x7f8748156610 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2670 tx=0x7f874400cc10 comp rx=0 tx=0).stop 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 shutdown_connections 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f872c06b500 0x7f872c06d9c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f87480b0950 0x7f874815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f87480b0dd0 0x7f8748156610 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 --2- 172.21.15.6:0/2283803064 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f87300039b0 0x7f87481556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:49.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.282+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 >> 172.21.15.6:0/2283803064 conn(0x7f87480731f0 msgr2=0x7f87480b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:49.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.283+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 shutdown_connections 2024-03-06T07:37:49.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:49.283+0000 7f874f200640 1 -- 172.21.15.6:0/2283803064 wait complete. 2024-03-06T07:37:49.505 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:49.505 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:50.161 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:50.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.731+0000 7f7650600640 1 Processor -- start 2024-03-06T07:37:50.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.731+0000 7f7650600640 1 -- start start 2024-03-06T07:37:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f7650600640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f764815df70 0x7f7648070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f7650600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f764815f360 0x7f76480b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f7650600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764806f0a0 con 0x7f76480b6230 2024-03-06T07:37:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f7650600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764806fcb0 con 0x7f764815f360 2024-03-06T07:37:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f7650600640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764806f380 con 0x7f764815df70 2024-03-06T07:37:50.735 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f764d200640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f76480b6230 0x7f7648161cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34730/0 (socket says 172.21.15.6:34730) 2024-03-06T07:37:50.736 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.733+0000 7f764d200640 1 -- 172.21.15.6:0/2206319327 learned_addr learned my addr 172.21.15.6:0/2206319327 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f764e600640 1 --2- 172.21.15.6:0/2206319327 >> v2:172.21.15.6:3300/0 conn(0x7f764815f360 0x7f76480b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f764dc00640 1 --2- 172.21.15.6:0/2206319327 >> v2:172.21.15.178:3300/0 conn(0x7f764815df70 0x7f7648070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846533925 0 0) 0x7f764806f0a0 con 0x7f76480b6230 2024-03-06T07:37:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630003520 con 0x7f76480b6230 2024-03-06T07:37:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392752095 0 0) 0x7f7630003520 con 0x7f76480b6230 2024-03-06T07:37:50.737 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 >> v2:172.21.15.178:3300/0 conn(0x7f764815df70 msgr2=0x7f7648070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 --2- 172.21.15.6:0/2206319327 >> v2:172.21.15.178:3300/0 conn(0x7f764815df70 0x7f7648070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 >> v2:172.21.15.6:3300/0 conn(0x7f764815f360 msgr2=0x7f76480b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 --2- 172.21.15.6:0/2206319327 >> v2:172.21.15.6:3300/0 conn(0x7f764815f360 0x7f76480b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764806f660 con 0x7f76480b6230 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f7634001670 con 0x7f76480b6230 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.734+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 >> v1:172.21.15.6:6789/0 conn(0x7f76480b6230 legacy=0x7f7648161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f7647e00640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7630003a20 0x7f7630003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f7647e00640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7630004800 0x7f7630004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f7647e00640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76300051c0 0x7f763000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764806fcb0 con 0x7f76300051c0 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764806f380 con 0x7f7630003a20 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763000bfe0 con 0x7f7630004800 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f764e600640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76300051c0 0x7f763000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f764d200640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7630004800 0x7f7630004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.738 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.735+0000 7f764dc00640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7630003a20 0x7f7630003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764e600640 1 -- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7630004800 msgr2=0x7f7630004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764e600640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7630004800 0x7f7630004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764e600640 1 -- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7630003a20 msgr2=0x7f7630003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764e600640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7630003a20 0x7f7630003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764e600640 1 -- 172.21.15.6:0/2206319327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7630003520 con 0x7f76300051c0 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764d200640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7630004800 0x7f7630004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:50.740 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.736+0000 7f764dc00640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7630003a20 0x7f7630003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.737+0000 7f764e600640 1 -- 172.21.15.6:0/2206319327 reap_dead start 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.737+0000 7f764e600640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76300051c0 0x7f763000b9c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f7640009f90 tx=0x7f7630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd66699127a2c807 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.737+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640039070 con 0x7f76300051c0 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.737+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764002cc20 con 0x7f76300051c0 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.737+0000 7f7647e00640 1 -- 172.21.15.6:0/2206319327 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640009c10 con 0x7f76300051c0 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 -- 172.21.15.6:0/2206319327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76300051c0 msgr2=0x7f763000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76300051c0 0x7f763000b9c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f7640009f90 tx=0x7f7630005830 comp rx=0 tx=0).stop 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 -- 172.21.15.6:0/2206319327 shutdown_connections 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 --2- 172.21.15.6:0/2206319327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f76300051c0 0x7f763000b9c0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 -- 172.21.15.6:0/2206319327 >> 172.21.15.6:0/2206319327 conn(0x7f764809aa10 msgr2=0x7f764809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 -- 172.21.15.6:0/2206319327 shutdown_connections 2024-03-06T07:37:50.741 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.738+0000 7f7650600640 1 -- 172.21.15.6:0/2206319327 wait complete. 2024-03-06T07:37:50.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.740+0000 7f7650600640 1 Processor -- start 2024-03-06T07:37:50.742 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.740+0000 7f7650600640 1 -- start start 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.741+0000 7f7650600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76300051c0 0x7f7648147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.741+0000 7f7650600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f764815c660 0x7f76480b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.741+0000 7f7650600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 0x7f76480b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.741+0000 7f7650600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480b67d0 con 0x7f764815cac0 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.741+0000 7f7650600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480b6650 con 0x7f764815c660 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.741+0000 7f7650600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480b6950 con 0x7f76300051c0 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 0x7f76480b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 0x7f76480b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:37556/0 (socket says 172.21.15.6:37556) 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 -- 172.21.15.6:0/1307568520 learned_addr learned my addr 172.21.15.6:0/1307568520 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764d200640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f764815c660 0x7f76480b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764dc00640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76300051c0 0x7f7648147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 -- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76300051c0 msgr2=0x7f7648147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76300051c0 0x7f7648147670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 -- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f764815c660 msgr2=0x7f76480b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:50.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f764815c660 0x7f76480b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:50.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764e600640 1 -- 172.21.15.6:0/1307568520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764815a6a0 con 0x7f764815cac0 2024-03-06T07:37:50.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.742+0000 7f764d200640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f764815c660 0x7f76480b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:37:50.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.743+0000 7f764e600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 0x7f76480b35b0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f764000e230 tx=0x7f7640009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:50.746 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.743+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640039040 con 0x7f764815cac0 2024-03-06T07:37:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.743+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764002ca20 con 0x7f764815cac0 2024-03-06T07:37:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.743+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764003f680 con 0x7f764815cac0 2024-03-06T07:37:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.743+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764815a980 con 0x7f764815cac0 2024-03-06T07:37:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.743+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764815af40 con 0x7f764815cac0 2024-03-06T07:37:50.747 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.744+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764804f3a0 con 0x7f764815cac0 2024-03-06T07:37:50.748 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.746+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7640005d10 con 0x7f764815cac0 2024-03-06T07:37:50.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.747+0000 7f7645600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f762806b4d0 0x7f762806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:50.749 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.747+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76400ac210 con 0x7f764815cac0 2024-03-06T07:37:50.750 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.748+0000 7f764dc00640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f762806b4d0 0x7f762806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:50.751 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.749+0000 7f764dc00640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f762806b4d0 0x7f762806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f763c009990 tx=0x7f763c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:50.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.757+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f764007afb0 con 0x7f764815cac0 2024-03-06T07:37:50.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.994+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f764806caa0 con 0x7f764815cac0 2024-03-06T07:37:50.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.996+0000 7f7645600640 1 -- 172.21.15.6:0/1307568520 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f764007a8b0 con 0x7f764815cac0 2024-03-06T07:37:50.998 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:50.998 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:50.998 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:51.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f762806b4d0 msgr2=0x7f762806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f762806b4d0 0x7f762806d990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f763c009990 tx=0x7f763c008040 comp rx=0 tx=0).stop 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 msgr2=0x7f76480b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 0x7f76480b35b0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f764000e230 tx=0x7f7640009750 comp rx=0 tx=0).stop 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 shutdown_connections 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f762806b4d0 0x7f762806d990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f764815c660 0x7f76480b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f764815cac0 0x7f76480b35b0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 --2- 172.21.15.6:0/1307568520 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f76300051c0 0x7f7648147670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.998+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 >> 172.21.15.6:0/1307568520 conn(0x7f764809aa10 msgr2=0x7f76480aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.999+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 shutdown_connections 2024-03-06T07:37:51.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:50.999+0000 7f7650600640 1 -- 172.21.15.6:0/1307568520 wait complete. 2024-03-06T07:37:51.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:51.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:52.163 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:52.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:52.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.452+0000 7fafc8a00640 1 Processor -- start 2024-03-06T07:37:52.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.454+0000 7fafc8a00640 1 -- start start 2024-03-06T07:37:52.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 0x7fafc40cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc8a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fafc40d2930 0x7fafc40b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc8a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc40cea50 con 0x7fafc40b5060 2024-03-06T07:37:52.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc8a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc40cf900 con 0x7fafc40d2930 2024-03-06T07:37:52.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40cecf0 con 0x7fafc4161b40 2024-03-06T07:37:52.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 0x7fafc40cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.457 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc1e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 0x7fafc40cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3301/0 says I am v2:172.21.15.6:47980/0 (socket says 172.21.15.6:47980) 2024-03-06T07:37:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc2800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fafc40d2930 0x7fafc40b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.455+0000 7fafc1e00640 1 -- 172.21.15.6:0/2304656429 learned_addr learned my addr 172.21.15.6:0/2304656429 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.456+0000 7fafc1e00640 1 -- 172.21.15.6:0/2304656429 >> v2:172.21.15.178:3300/0 conn(0x7fafc40d2930 msgr2=0x7fafc40b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.456+0000 7fafc1e00640 1 --2- 172.21.15.6:0/2304656429 >> v2:172.21.15.178:3300/0 conn(0x7fafc40d2930 0x7fafc40b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.456+0000 7fafc1e00640 1 -- 172.21.15.6:0/2304656429 >> v1:172.21.15.178:6789/0 conn(0x7fafc40b5060 legacy=0x7fafc40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.458 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.456+0000 7fafc1e00640 1 -- 172.21.15.6:0/2304656429 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40cefd0 con 0x7fafc4161b40 2024-03-06T07:37:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.458+0000 7fafc1e00640 1 --2- 172.21.15.6:0/2304656429 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 0x7fafc40cfd90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009e50 tx=0x7fafb402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8828179cdfae8197 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc0a00640 1 -- 172.21.15.6:0/2304656429 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb402cb00 con 0x7fafc4161b40 2024-03-06T07:37:52.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc0a00640 1 -- 172.21.15.6:0/2304656429 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb402cca0 con 0x7fafc4161b40 2024-03-06T07:37:52.460 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc0a00640 1 -- 172.21.15.6:0/2304656429 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4034610 con 0x7fafc4161b40 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc8a00640 1 -- 172.21.15.6:0/2304656429 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 msgr2=0x7fafc40cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2304656429 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 0x7fafc40cfd90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009e50 tx=0x7fafb402bf60 comp rx=0 tx=0).stop 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc8a00640 1 -- 172.21.15.6:0/2304656429 shutdown_connections 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2304656429 >> v2:172.21.15.178:3300/0 conn(0x7fafc40d2930 0x7fafc40b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2304656429 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc4161b40 0x7fafc40cfd90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.459+0000 7fafc8a00640 1 -- 172.21.15.6:0/2304656429 >> 172.21.15.6:0/2304656429 conn(0x7fafc409d220 msgr2=0x7fafc40d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:52.461 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.460+0000 7fafc8a00640 1 -- 172.21.15.6:0/2304656429 shutdown_connections 2024-03-06T07:37:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.460+0000 7fafc8a00640 1 -- 172.21.15.6:0/2304656429 wait complete. 2024-03-06T07:37:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.461+0000 7fafc8a00640 1 Processor -- start 2024-03-06T07:37:52.462 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.462+0000 7fafc8a00640 1 -- start start 2024-03-06T07:37:52.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.462+0000 7fafc8a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc40b5060 0x7fafc41f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.465 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc8a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafc40d2930 0x7fafc415d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc8a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 0x7fafc40ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc8a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40b2ff0 con 0x7fafc4161b40 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc8a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40cf590 con 0x7fafc40d2930 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc8a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc40b3170 con 0x7fafc40b5060 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 0x7fafc40ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 0x7fafc40ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33854/0 (socket says 172.21.15.6:33854) 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 -- 172.21.15.6:0/2158060453 learned_addr learned my addr 172.21.15.6:0/2158060453 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc1e00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc40b5060 0x7fafc41f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc1400640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafc40d2930 0x7fafc415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 -- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc40b5060 msgr2=0x7fafc41f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc40b5060 0x7fafc41f50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.466 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 -- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafc40d2930 msgr2=0x7fafc415d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafc40d2930 0x7fafc415d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc2800640 1 -- 172.21.15.6:0/2158060453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40cc020 con 0x7fafc4161b40 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc1e00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc40b5060 0x7fafc41f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.463+0000 7fafc1400640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafc40d2930 0x7fafc415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.464+0000 7fafc2800640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 0x7fafc40ccde0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00e930 tx=0x7fafbc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.464+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc017070 con 0x7fafc4161b40 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.464+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc00c980 con 0x7fafc4161b40 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.464+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc40cc300 con 0x7fafc4161b40 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.464+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc00cd70 con 0x7fafc4161b40 2024-03-06T07:37:52.467 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.464+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc415dcb0 con 0x7fafc4161b40 2024-03-06T07:37:52.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.465+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc404f3a0 con 0x7fafc4161b40 2024-03-06T07:37:52.468 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.466+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafbc01a070 con 0x7fafc4161b40 2024-03-06T07:37:52.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.468+0000 7fafba000640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafa006b4d0 0x7fafa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:52.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.469+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fafbc08a610 con 0x7fafc4161b40 2024-03-06T07:37:52.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.469+0000 7fafc1e00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafa006b4d0 0x7fafa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:52.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.470+0000 7fafc1e00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafa006b4d0 0x7fafa006d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fafb4035000 tx=0x7fafb4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:52.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.473+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafbc059500 con 0x7fafc4161b40 2024-03-06T07:37:52.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.715+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafc40d1df0 con 0x7fafc4161b40 2024-03-06T07:37:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.716+0000 7fafba000640 1 -- 172.21.15.6:0/2158060453 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fafbc058e00 con 0x7fafc4161b40 2024-03-06T07:37:52.717 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:52.717 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:52.718 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.718+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafa006b4d0 msgr2=0x7fafa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.718+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafa006b4d0 0x7fafa006d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fafb4035000 tx=0x7fafb4030000 comp rx=0 tx=0).stop 2024-03-06T07:37:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.718+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 msgr2=0x7fafc40ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:52.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.718+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 0x7fafc40ccde0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fafbc00e930 tx=0x7fafbc00edf0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 shutdown_connections 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fafa006b4d0 0x7fafa006d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fafc40d2930 0x7fafc415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fafc4161b40 0x7fafc40ccde0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 --2- 172.21.15.6:0/2158060453 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fafc40b5060 0x7fafc41f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 >> 172.21.15.6:0/2158060453 conn(0x7fafc409d220 msgr2=0x7fafc40b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 shutdown_connections 2024-03-06T07:37:52.721 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:52.719+0000 7fafc8a00640 1 -- 172.21.15.6:0/2158060453 wait complete. 2024-03-06T07:37:52.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:52.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:53.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:54.165 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:54.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.164+0000 7fd477200640 1 Processor -- start 2024-03-06T07:37:54.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.164+0000 7fd477200640 1 -- start start 2024-03-06T07:37:54.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd477200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fd478145300 0x7fd4780b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd477200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd478070f10 0x7fd4780cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd477200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780714d0 con 0x7fd4780cce90 2024-03-06T07:37:54.168 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd477200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4780cd300 con 0x7fd478070f10 2024-03-06T07:37:54.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd477200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4780717b0 con 0x7fd478145300 2024-03-06T07:37:54.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd475400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd478070f10 0x7fd4780cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd476800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd4780cce90 0x7fd4780cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59610/0 (socket says 172.21.15.6:59610) 2024-03-06T07:37:54.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.166+0000 7fd476800640 1 -- 172.21.15.6:0/2958346349 learned_addr learned my addr 172.21.15.6:0/2958346349 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:54.169 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.167+0000 7fd475e00640 1 --2- 172.21.15.6:0/2958346349 >> v2:172.21.15.178:3300/0 conn(0x7fd478145300 0x7fd4780b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.167+0000 7fd475400640 1 -- 172.21.15.6:0/2958346349 >> v2:172.21.15.178:3300/0 conn(0x7fd478145300 msgr2=0x7fd4780b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:54.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.167+0000 7fd475400640 1 --2- 172.21.15.6:0/2958346349 >> v2:172.21.15.178:3300/0 conn(0x7fd478145300 0x7fd4780b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.167+0000 7fd475400640 1 -- 172.21.15.6:0/2958346349 >> v1:172.21.15.6:6789/0 conn(0x7fd4780cce90 legacy=0x7fd4780cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.167+0000 7fd475400640 1 -- 172.21.15.6:0/2958346349 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478071a90 con 0x7fd478070f10 2024-03-06T07:37:54.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.167+0000 7fd474a00640 1 -- 172.21.15.6:0/2958346349 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937133817 0 0) 0x7fd4780714d0 con 0x7fd4780cce90 2024-03-06T07:37:54.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.168+0000 7fd475400640 1 --2- 172.21.15.6:0/2958346349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd478070f10 0x7fd4780cc950 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd4780cf990 tx=0x7fd46002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18724e4c0f1be0b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:54.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.168+0000 7fd474a00640 1 -- 172.21.15.6:0/2958346349 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4600315a0 con 0x7fd478070f10 2024-03-06T07:37:54.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.168+0000 7fd474a00640 1 -- 172.21.15.6:0/2958346349 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460037070 con 0x7fd478070f10 2024-03-06T07:37:54.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.168+0000 7fd474a00640 1 -- 172.21.15.6:0/2958346349 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460031960 con 0x7fd478070f10 2024-03-06T07:37:54.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 -- 172.21.15.6:0/2958346349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd478070f10 msgr2=0x7fd4780cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 --2- 172.21.15.6:0/2958346349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd478070f10 0x7fd4780cc950 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd4780cf990 tx=0x7fd46002b840 comp rx=0 tx=0).stop 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 -- 172.21.15.6:0/2958346349 shutdown_connections 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 --2- 172.21.15.6:0/2958346349 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd478070f10 0x7fd4780cc950 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 --2- 172.21.15.6:0/2958346349 >> v2:172.21.15.178:3300/0 conn(0x7fd478145300 0x7fd4780b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 -- 172.21.15.6:0/2958346349 >> 172.21.15.6:0/2958346349 conn(0x7fd47806b3d0 msgr2=0x7fd47806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 -- 172.21.15.6:0/2958346349 shutdown_connections 2024-03-06T07:37:54.172 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.169+0000 7fd477200640 1 -- 172.21.15.6:0/2958346349 wait complete. 2024-03-06T07:37:54.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.171+0000 7fd477200640 1 Processor -- start 2024-03-06T07:37:54.173 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.171+0000 7fd477200640 1 -- start start 2024-03-06T07:37:54.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd477200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd478070f10 0x7fd47815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd477200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 0x7fd4781e6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd477200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd478145300 0x7fd4781e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd477200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4781456e0 con 0x7fd4780cce90 2024-03-06T07:37:54.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd477200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780b4ee0 con 0x7fd478070f10 2024-03-06T07:37:54.175 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd477200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47815d580 con 0x7fd478145300 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.172+0000 7fd475400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 0x7fd4781e6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 0x7fd4781e6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33868/0 (socket says 172.21.15.6:33868) 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 -- 172.21.15.6:0/238284913 learned_addr learned my addr 172.21.15.6:0/238284913 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475e00640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd478070f10 0x7fd47815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd476800640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd478145300 0x7fd4781e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 -- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd478145300 msgr2=0x7fd4781e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd478145300 0x7fd4781e6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 -- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd478070f10 msgr2=0x7fd47815b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd478070f10 0x7fd47815b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.176 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 -- 172.21.15.6:0/238284913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4781f4030 con 0x7fd4780cce90 2024-03-06T07:37:54.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.173+0000 7fd475400640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 0x7fd4781e6a60 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd460000c00 tx=0x7fd4600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:54.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.174+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460004230 con 0x7fd4780cce90 2024-03-06T07:37:54.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.174+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4600043d0 con 0x7fd4780cce90 2024-03-06T07:37:54.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.174+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460003850 con 0x7fd4780cce90 2024-03-06T07:37:54.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.174+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47815b9e0 con 0x7fd4780cce90 2024-03-06T07:37:54.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.174+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47806d680 con 0x7fd4780cce90 2024-03-06T07:37:54.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.175+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47804f330 con 0x7fd4780cce90 2024-03-06T07:37:54.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.176+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4600039f0 con 0x7fd4780cce90 2024-03-06T07:37:54.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.177+0000 7fd46e000640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd45406b4d0 0x7fd45406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:54.179 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.177+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd46002f030 con 0x7fd4780cce90 2024-03-06T07:37:54.180 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.178+0000 7fd475e00640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd45406b4d0 0x7fd45406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:54.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.179+0000 7fd475e00640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd45406b4d0 0x7fd45406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd4700099c0 tx=0x7fd470003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:54.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.182+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4781f4030 con 0x7fd4780cce90 2024-03-06T07:37:54.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.444+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4780b1a10 con 0x7fd4780cce90 2024-03-06T07:37:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.445+0000 7fd46e000640 1 -- 172.21.15.6:0/238284913 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fd460041070 con 0x7fd4780cce90 2024-03-06T07:37:54.447 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:54.447 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:54.447 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:54.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd45406b4d0 msgr2=0x7fd45406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd45406b4d0 0x7fd45406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd4700099c0 tx=0x7fd470003990 comp rx=0 tx=0).stop 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 msgr2=0x7fd4781e6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 0x7fd4781e6a60 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fd460000c00 tx=0x7fd4600046c0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 shutdown_connections 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd45406b4d0 0x7fd45406d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd478145300 0x7fd4781e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd4780cce90 0x7fd4781e6a60 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 --2- 172.21.15.6:0/238284913 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd478070f10 0x7fd47815b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 >> 172.21.15.6:0/238284913 conn(0x7fd47806b3d0 msgr2=0x7fd4780b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 shutdown_connections 2024-03-06T07:37:54.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:54.447+0000 7fd477200640 1 -- 172.21.15.6:0/238284913 wait complete. 2024-03-06T07:37:54.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:54.458 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:37:54.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:54.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:55.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.900+0000 7f2602a00640 1 Processor -- start 2024-03-06T07:37:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.901+0000 7f2602a00640 1 -- start start 2024-03-06T07:37:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f2602a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f25fc0d2340 0x7f25fc0d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f2602a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc06c420 con 0x7f25fc15e240 2024-03-06T07:37:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f2602a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0cf430 con 0x7f25fc15de60 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f2602a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc06d8b0 con 0x7f25fc0d2340 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f2600a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f25fc0d2340 0x7f25fc0d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f25fb400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f25fc15e240 0x7f25fc0d1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59616/0 (socket says 172.21.15.6:59616) 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.902+0000 7f25fb400640 1 -- 172.21.15.6:0/393959147 learned_addr learned my addr 172.21.15.6:0/393959147 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.903+0000 7f2600a00640 1 -- 172.21.15.6:0/393959147 >> v1:172.21.15.178:6789/0 conn(0x7f25fc15de60 legacy=0x7f25fc06d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.903+0000 7f2600a00640 1 -- 172.21.15.6:0/393959147 >> v1:172.21.15.6:6789/0 conn(0x7f25fc15e240 legacy=0x7f25fc0d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.903+0000 7f2600a00640 1 -- 172.21.15.6:0/393959147 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc06c6c0 con 0x7f25fc0d2340 2024-03-06T07:37:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.903+0000 7f25faa00640 1 -- 172.21.15.6:0/393959147 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460372702 0 0) 0x7f25fc06c420 con 0x7f25fc15e240 2024-03-06T07:37:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.905+0000 7f2600a00640 1 --2- 172.21.15.6:0/393959147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc0d2340 0x7f25fc0d0df0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f25f40097a0 tx=0x7f25f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5f9cfd00fc3a42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.905+0000 7f25faa00640 1 -- 172.21.15.6:0/393959147 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4035020 con 0x7f25fc0d2340 2024-03-06T07:37:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.905+0000 7f25faa00640 1 -- 172.21.15.6:0/393959147 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f4031830 con 0x7f25fc0d2340 2024-03-06T07:37:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.905+0000 7f25faa00640 1 -- 172.21.15.6:0/393959147 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f4031c00 con 0x7f25fc0d2340 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 -- 172.21.15.6:0/393959147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc0d2340 msgr2=0x7f25fc0d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 --2- 172.21.15.6:0/393959147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc0d2340 0x7f25fc0d0df0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f25f40097a0 tx=0x7f25f402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 -- 172.21.15.6:0/393959147 shutdown_connections 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 --2- 172.21.15.6:0/393959147 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc0d2340 0x7f25fc0d0df0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 -- 172.21.15.6:0/393959147 >> 172.21.15.6:0/393959147 conn(0x7f25fc09aa10 msgr2=0x7f25fc09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 -- 172.21.15.6:0/393959147 shutdown_connections 2024-03-06T07:37:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.906+0000 7f2602a00640 1 -- 172.21.15.6:0/393959147 wait complete. 2024-03-06T07:37:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.909+0000 7f2602a00640 1 Processor -- start 2024-03-06T07:37:55.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.909+0000 7f2602a00640 1 -- start start 2024-03-06T07:37:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f2602a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f25fc0d2340 0x7f25fc15f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f2602a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 0x7f25fc15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f2602a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25fc15e240 0x7f25fc0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f2602a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0d1330 con 0x7f25fc15de60 2024-03-06T07:37:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f2602a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc06cc80 con 0x7f25fc15e240 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f2602a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0d14b0 con 0x7f25fc0d2340 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f25fb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 0x7f25fc15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f25fb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 0x7f25fc15b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33874/0 (socket says 172.21.15.6:33874) 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.911+0000 7f25fb400640 1 -- 172.21.15.6:0/3483521159 learned_addr learned my addr 172.21.15.6:0/3483521159 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f2600a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25fc15e240 0x7f25fc0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fbe00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f25fc0d2340 0x7f25fc15f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fb400640 1 -- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f25fc0d2340 msgr2=0x7f25fc15f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fb400640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f25fc0d2340 0x7f25fc15f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fb400640 1 -- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25fc15e240 msgr2=0x7f25fc0b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fb400640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25fc15e240 0x7f25fc0b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fb400640 1 -- 172.21.15.6:0/3483521159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc15fdb0 con 0x7f25fc15de60 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f2600a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25fc15e240 0x7f25fc0b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fbe00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f25fc0d2340 0x7f25fc15f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25fb400640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 0x7f25fc15b2f0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f25e800cb10 tx=0x7f25e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8017070 con 0x7f25fc15de60 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e800ac70 con 0x7f25fc15de60 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.912+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc160090 con 0x7f25fc15de60 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.913+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8012620 con 0x7f25fc15de60 2024-03-06T07:37:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.913+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc160650 con 0x7f25fc15de60 2024-03-06T07:37:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.914+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc04f3a0 con 0x7f25fc15de60 2024-03-06T07:37:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.915+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25e801a070 con 0x7f25fc15de60 2024-03-06T07:37:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.916+0000 7f25f3e00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f25dc06b4d0 0x7f25dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.916+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f25e8089ff0 con 0x7f25fc15de60 2024-03-06T07:37:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.917+0000 7f25fbe00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f25dc06b4d0 0x7f25dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.918+0000 7f25fbe00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f25dc06b4d0 0x7f25dc06d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f25ec003eb0 tx=0x7f25ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:55.920+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25e800ae20 con 0x7f25fc15de60 2024-03-06T07:37:56.167 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:56.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.410+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25fc0b2980 con 0x7f25fc15de60 2024-03-06T07:37:56.413 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.411+0000 7f25f3e00640 1 -- 172.21.15.6:0/3483521159 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7f25e8058e30 con 0x7f25fc15de60 2024-03-06T07:37:56.414 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:56.414 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:56.414 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:56.416 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.413+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f25dc06b4d0 msgr2=0x7f25dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f25dc06b4d0 0x7f25dc06d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f25ec003eb0 tx=0x7f25ec009040 comp rx=0 tx=0).stop 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 msgr2=0x7f25fc15b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 0x7f25fc15b2f0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f25e800cb10 tx=0x7f25e800a060 comp rx=0 tx=0).stop 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 shutdown_connections 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f25dc06b4d0 0x7f25dc06d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25fc15e240 0x7f25fc0b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f25fc15de60 0x7f25fc15b2f0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.417 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 --2- 172.21.15.6:0/3483521159 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f25fc0d2340 0x7f25fc15f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 >> 172.21.15.6:0/3483521159 conn(0x7f25fc09aa10 msgr2=0x7f25fc0b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 shutdown_connections 2024-03-06T07:37:56.418 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:56.414+0000 7f2602a00640 1 -- 172.21.15.6:0/3483521159 wait complete. 2024-03-06T07:37:56.640 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:56.640 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:57.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:57.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.863+0000 7fcdf9a00640 1 Processor -- start 2024-03-06T07:37:57.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.863+0000 7fcdf9a00640 1 -- start start 2024-03-06T07:37:57.867 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.865+0000 7fcdf9a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcdf4151b30 0x7fcdf406f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.868 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.865+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 0x7fcdf40b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf9a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf406e4b0 con 0x7fcdf415c4a0 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf9a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf40b4fd0 con 0x7fcdf4151b30 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf9a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf406e790 con 0x7fcdf40b4650 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf9000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcdf415c4a0 0x7fcdf415c930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:59620/0 (socket says 172.21.15.6:59620) 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf9000640 1 -- 172.21.15.6:0/637134798 learned_addr learned my addr 172.21.15.6:0/637134798 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf3e00640 1 --2- 172.21.15.6:0/637134798 >> v2:172.21.15.178:3300/0 conn(0x7fcdf4151b30 0x7fcdf406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf3400640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 0x7fcdf40b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.866+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269367191 0 0) 0x7fcdf406e4b0 con 0x7fcdf415c4a0 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8003520 con 0x7fcdf415c4a0 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1024100749 0 0) 0x7fcdd8003520 con 0x7fcdf415c4a0 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 msgr2=0x7fcdf40b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 0x7fcdf40b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 >> v2:172.21.15.178:3300/0 conn(0x7fcdf4151b30 msgr2=0x7fcdf406f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 --2- 172.21.15.6:0/637134798 >> v2:172.21.15.178:3300/0 conn(0x7fcdf4151b30 0x7fcdf406f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf406ea70 con 0x7fcdf415c4a0 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fcdec0019f0 con 0x7fcdf415c4a0 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 >> v1:172.21.15.6:6789/0 conn(0x7fcdf415c4a0 legacy=0x7fcdf415c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 0x7fcdd80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdd8003f30 0x7fcdd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd8004d40 0x7fcdd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40b4fd0 con 0x7fcdd8003f30 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf406e790 con 0x7fcdd8004d40 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd800ba10 con 0x7fcdf40b4650 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.867+0000 7fcdf3400640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdd8003f30 0x7fcdd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf3400640 1 -- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 msgr2=0x7fcdd80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf3400640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdf40b4650 0x7fcdd80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf3400640 1 -- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd8004d40 msgr2=0x7fcdd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf9000640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd8004d40 0x7fcdd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf3400640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd8004d40 0x7fcdd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf3400640 1 -- 172.21.15.6:0/637134798 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8003520 con 0x7fcdd8003f30 2024-03-06T07:37:57.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.868+0000 7fcdf9000640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdd8004d40 0x7fcdd80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:57.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.869+0000 7fcdf3400640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdd8003f30 0x7fcdd8004800 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009af0 tx=0x7fcdd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9779d0e51f49d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.869+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0039070 con 0x7fcdd8003f30 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.869+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde002c710 con 0x7fcdd8003f30 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.869+0000 7fcdf2a00640 1 -- 172.21.15.6:0/637134798 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0009690 con 0x7fcdd8003f30 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9a00640 1 -- 172.21.15.6:0/637134798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdd8003f30 msgr2=0x7fcdd8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/637134798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdd8003f30 0x7fcdd8004800 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009af0 tx=0x7fcdd8005830 comp rx=0 tx=0).stop 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9000640 1 -- 172.21.15.6:0/637134798 reap_dead start 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9a00640 1 -- 172.21.15.6:0/637134798 shutdown_connections 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9a00640 1 -- 172.21.15.6:0/637134798 >> 172.21.15.6:0/637134798 conn(0x7fcdf406c0c0 msgr2=0x7fcdf406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9a00640 1 -- 172.21.15.6:0/637134798 shutdown_connections 2024-03-06T07:37:57.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.870+0000 7fcdf9a00640 1 -- 172.21.15.6:0/637134798 wait complete. 2024-03-06T07:37:57.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.872+0000 7fcdf9a00640 1 Processor -- start 2024-03-06T07:37:57.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.872+0000 7fcdf9a00640 1 -- start start 2024-03-06T07:37:57.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdd8003f30 0x7fcdf40d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 0x7fcdf4168560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf9a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdf40cfac0 0x7fcdf406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf9a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf406f7b0 con 0x7fcdf413e750 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf9a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4151f10 con 0x7fcdf40cfac0 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf9a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40d0e70 con 0x7fcdd8003f30 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.873+0000 7fcdf3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 0x7fcdf4168560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 0x7fcdf4168560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33878/0 (socket says 172.21.15.6:33878) 2024-03-06T07:37:57.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 -- 172.21.15.6:0/3272785507 learned_addr learned my addr 172.21.15.6:0/3272785507 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:57.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 -- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdd8003f30 msgr2=0x7fcdf40d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdd8003f30 0x7fcdf40d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 -- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdf40cfac0 msgr2=0x7fcdf406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf9000640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdf40cfac0 0x7fcdf406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdf40cfac0 0x7fcdf406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 -- 172.21.15.6:0/3272785507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf4168aa0 con 0x7fcdf413e750 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf9000640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdf40cfac0 0x7fcdf406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.874+0000 7fcdf3400640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 0x7fcdf4168560 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcde0037040 tx=0x7fcde0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.875+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0042020 con 0x7fcdf413e750 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.875+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde003a040 con 0x7fcdf413e750 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.875+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0039070 con 0x7fcdf413e750 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.875+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf413ebd0 con 0x7fcdf413e750 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.875+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf40d0b90 con 0x7fcdf413e750 2024-03-06T07:37:57.878 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.876+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf406f270 con 0x7fcdf413e750 2024-03-06T07:37:57.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.877+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcde0045020 con 0x7fcdf413e750 2024-03-06T07:37:57.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.878+0000 7fcdebe00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdd406b500 0x7fcdd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:57.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.878+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcde00ab330 con 0x7fcdf413e750 2024-03-06T07:37:57.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.879+0000 7fcdf3e00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdd406b500 0x7fcdd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:57.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.880+0000 7fcdf3e00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdd406b500 0x7fcdd406d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcde4006e80 tx=0x7fcde4006dd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:57.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:57.883+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf406f270 con 0x7fcdf413e750 2024-03-06T07:37:58.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.121+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdf404f330 con 0x7fcdf413e750 2024-03-06T07:37:58.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.122+0000 7fcdebe00640 1 -- 172.21.15.6:0/3272785507 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fcde004c020 con 0x7fcdf413e750 2024-03-06T07:37:58.125 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:58.125 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdd406b500 msgr2=0x7fcdd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdd406b500 0x7fcdd406d9c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcde4006e80 tx=0x7fcde4006dd0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 msgr2=0x7fcdf4168560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 0x7fcdf4168560 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fcde0037040 tx=0x7fcde0004730 comp rx=0 tx=0).stop 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 shutdown_connections 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcdd406b500 0x7fcdd406d9c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcdf40cfac0 0x7fcdf406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcdf413e750 0x7fcdf4168560 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 --2- 172.21.15.6:0/3272785507 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcdd8003f30 0x7fcdf40d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 >> 172.21.15.6:0/3272785507 conn(0x7fcdf406c0c0 msgr2=0x7fcdf415ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 shutdown_connections 2024-03-06T07:37:58.128 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:58.125+0000 7fcdf9a00640 1 -- 172.21.15.6:0/3272785507 wait complete. 2024-03-06T07:37:58.169 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:37:58.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:37:58.347 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:37:59.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:37:59.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.568+0000 7fbcbd000640 1 Processor -- start 2024-03-06T07:37:59.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.568+0000 7fbcbd000640 1 -- start start 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.570+0000 7fbcbd000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbcb815efc0 0x7fbcb80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.570+0000 7fbcbd000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb81621d0 0x7fbcb8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.570+0000 7fbcbd000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb81627f0 con 0x7fbcb80cce90 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.570+0000 7fbcbd000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb815f3a0 con 0x7fbcb815efc0 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.570+0000 7fbcbd000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb806d190 con 0x7fbcb81621d0 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.570+0000 7fbcb7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbcb815efc0 0x7fbcb80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbcb815efc0 0x7fbcb80cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33894/0 (socket says 172.21.15.6:33894) 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7400640 1 -- 172.21.15.6:0/1295631337 learned_addr learned my addr 172.21.15.6:0/1295631337 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7e00640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb81621d0 0x7fbcb8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7400640 1 -- 172.21.15.6:0/1295631337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb81621d0 msgr2=0x7fbcb8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7400640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb81621d0 0x7fbcb8070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7400640 1 -- 172.21.15.6:0/1295631337 >> v1:172.21.15.178:6789/0 conn(0x7fbcb80cce90 legacy=0x7fbcb8161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7400640 1 -- 172.21.15.6:0/1295631337 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb806c860 con 0x7fbcb815efc0 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.571+0000 7fbcb7e00640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb81621d0 0x7fbcb8070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:37:59.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.572+0000 7fbcb7400640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb815efc0 0x7fbcb80cc950 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009870 tx=0x7fbca802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f7bcc4bd361dae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcb6a00640 1 -- 172.21.15.6:0/1295631337 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca8035020 con 0x7fbcb815efc0 2024-03-06T07:37:59.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcb6a00640 1 -- 172.21.15.6:0/1295631337 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca8031640 con 0x7fbcb815efc0 2024-03-06T07:37:59.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcb6a00640 1 -- 172.21.15.6:0/1295631337 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca802d040 con 0x7fbcb815efc0 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcbd000640 1 -- 172.21.15.6:0/1295631337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb815efc0 msgr2=0x7fbcb80cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcbd000640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb815efc0 0x7fbcb80cc950 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fbca8009870 tx=0x7fbca802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcbd000640 1 -- 172.21.15.6:0/1295631337 shutdown_connections 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcbd000640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb81621d0 0x7fbcb8070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcbd000640 1 --2- 172.21.15.6:0/1295631337 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb815efc0 0x7fbcb80cc950 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.573+0000 7fbcbd000640 1 -- 172.21.15.6:0/1295631337 >> 172.21.15.6:0/1295631337 conn(0x7fbcb806b3d0 msgr2=0x7fbcb806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.574+0000 7fbcbd000640 1 -- 172.21.15.6:0/1295631337 shutdown_connections 2024-03-06T07:37:59.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.574+0000 7fbcbd000640 1 -- 172.21.15.6:0/1295631337 wait complete. 2024-03-06T07:37:59.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.576+0000 7fbcbd000640 1 Processor -- start 2024-03-06T07:37:59.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.577+0000 7fbcbd000640 1 -- start start 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcbd000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb80cce90 0x7fbcb81ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcbd000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbcb815efc0 0x7fbcb81e76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcbd000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 0x7fbcb8146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcbd000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb806d620 con 0x7fbcb81621d0 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcbd000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb806ce20 con 0x7fbcb815efc0 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcbd000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb81e7c10 con 0x7fbcb80cce90 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.578+0000 7fbcb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 0x7fbcb8146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 0x7fbcb8146b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33904/0 (socket says 172.21.15.6:33904) 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 -- 172.21.15.6:0/4126436656 learned_addr learned my addr 172.21.15.6:0/4126436656 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7400640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb80cce90 0x7fbcb81ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcafe00640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbcb815efc0 0x7fbcb81e76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 -- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb80cce90 msgr2=0x7fbcb81ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb80cce90 0x7fbcb81ea0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 -- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbcb815efc0 msgr2=0x7fbcb81e76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbcb815efc0 0x7fbcb81e76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 -- 172.21.15.6:0/4126436656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb8147750 con 0x7fbcb81621d0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.579+0000 7fbcb7e00640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 0x7fbcb8146b60 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000cd50 tx=0x7fbcb000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.580+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb000f680 con 0x7fbcb81621d0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.580+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb000f840 con 0x7fbcb81621d0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.580+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb000fc30 con 0x7fbcb81621d0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.580+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb81599c0 con 0x7fbcb81621d0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.580+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb815a7d0 con 0x7fbcb81621d0 2024-03-06T07:37:59.583 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.581+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb815a9d0 con 0x7fbcb81621d0 2024-03-06T07:37:59.585 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.583+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcb000e3e0 con 0x7fbcb81621d0 2024-03-06T07:37:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.584+0000 7fbcaf400640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc9406b4d0 0x7fbc9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:37:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.584+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbcb008b010 con 0x7fbcb81621d0 2024-03-06T07:37:59.586 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.585+0000 7fbcb7400640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc9406b4d0 0x7fbc9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:37:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.586+0000 7fbcb7400640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc9406b4d0 0x7fbc9406d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbca8036000 tx=0x7fbca803c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:37:59.596 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.594+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcb815a9d0 con 0x7fbcb81621d0 2024-03-06T07:37:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.862+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcb804f330 con 0x7fbcb81621d0 2024-03-06T07:37:59.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.863+0000 7fbcaf400640 1 -- 172.21.15.6:0/4126436656 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3456 (secure 0 0 0) 0x7fbcb8147750 con 0x7fbcb81621d0 2024-03-06T07:37:59.865 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:37:59.866 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":100,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:36:29.312449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:37:59.866 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 100 2024-03-06T07:37:59.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc9406b4d0 msgr2=0x7fbc9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc9406b4d0 0x7fbc9406d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fbca8036000 tx=0x7fbca803c040 comp rx=0 tx=0).stop 2024-03-06T07:37:59.869 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 msgr2=0x7fbcb8146b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 0x7fbcb8146b60 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000cd50 tx=0x7fbcb000a650 comp rx=0 tx=0).stop 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 shutdown_connections 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbc9406b4d0 0x7fbc9406d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbcb815efc0 0x7fbcb81e76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcb81621d0 0x7fbcb8146b60 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 --2- 172.21.15.6:0/4126436656 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbcb80cce90 0x7fbcb81ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 >> 172.21.15.6:0/4126436656 conn(0x7fbcb806b3d0 msgr2=0x7fbcb80ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 shutdown_connections 2024-03-06T07:37:59.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:37:59.867+0000 7fbcbd000640 1 -- 172.21.15.6:0/4126436656 wait complete. 2024-03-06T07:38:00.092 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 100, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:36:29.312449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:00.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:00.171 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:01.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:01.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.313+0000 7ffa56c00640 1 Processor -- start 2024-03-06T07:38:01.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.313+0000 7ffa56c00640 1 -- start start 2024-03-06T07:38:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa56c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffa500b5060 0x7ffa500d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa56c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa500cea50 con 0x7ffa500b54a0 2024-03-06T07:38:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa56c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa500cf900 con 0x7ffa500b5060 2024-03-06T07:38:01.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa56c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500cecf0 con 0x7ffa50161b40 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa4f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffa500b5060 0x7ffa500d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa4f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffa500b5060 0x7ffa500d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33908/0 (socket says 172.21.15.6:33908) 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.315+0000 7ffa4f400640 1 -- 172.21.15.6:0/2811165092 learned_addr learned my addr 172.21.15.6:0/2811165092 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.316+0000 7ffa4fe00640 1 --2- 172.21.15.6:0/2811165092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.316+0000 7ffa4f400640 1 -- 172.21.15.6:0/2811165092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 msgr2=0x7ffa500cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.316+0000 7ffa4f400640 1 --2- 172.21.15.6:0/2811165092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.316+0000 7ffa4f400640 1 -- 172.21.15.6:0/2811165092 >> v1:172.21.15.178:6789/0 conn(0x7ffa500b54a0 legacy=0x7ffa500b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:01.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.316+0000 7ffa4f400640 1 -- 172.21.15.6:0/2811165092 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500cefd0 con 0x7ffa500b5060 2024-03-06T07:38:01.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.317+0000 7ffa4f400640 1 --2- 172.21.15.6:0/2811165092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b5060 0x7ffa500d2220 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009870 tx=0x7ffa3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca1831d2ccacdb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.317+0000 7ffa4ea00640 1 -- 172.21.15.6:0/2811165092 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38035020 con 0x7ffa500b5060 2024-03-06T07:38:01.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa4ea00640 1 -- 172.21.15.6:0/2811165092 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa38031640 con 0x7ffa500b5060 2024-03-06T07:38:01.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa4ea00640 1 -- 172.21.15.6:0/2811165092 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3802d040 con 0x7ffa500b5060 2024-03-06T07:38:01.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 -- 172.21.15.6:0/2811165092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b5060 msgr2=0x7ffa500d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 --2- 172.21.15.6:0/2811165092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b5060 0x7ffa500d2220 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009870 tx=0x7ffa3802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 -- 172.21.15.6:0/2811165092 shutdown_connections 2024-03-06T07:38:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 --2- 172.21.15.6:0/2811165092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b5060 0x7ffa500d2220 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 --2- 172.21.15.6:0/2811165092 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 -- 172.21.15.6:0/2811165092 >> 172.21.15.6:0/2811165092 conn(0x7ffa5009d220 msgr2=0x7ffa500d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.318+0000 7ffa56c00640 1 -- 172.21.15.6:0/2811165092 shutdown_connections 2024-03-06T07:38:01.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.319+0000 7ffa56c00640 1 -- 172.21.15.6:0/2811165092 wait complete. 2024-03-06T07:38:01.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.321+0000 7ffa56c00640 1 Processor -- start 2024-03-06T07:38:01.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.322+0000 7ffa56c00640 1 -- start start 2024-03-06T07:38:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.323+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffa500b5060 0x7ffa500c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.323+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 0x7ffa5015a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.323+0000 7ffa56c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.323+0000 7ffa56c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500b2ff0 con 0x7ffa500b54a0 2024-03-06T07:38:01.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.323+0000 7ffa56c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500cf590 con 0x7ffa500b5060 2024-03-06T07:38:01.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.323+0000 7ffa56c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500b3170 con 0x7ffa50161b40 2024-03-06T07:38:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 0x7ffa5015a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 0x7ffa5015a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33918/0 (socket says 172.21.15.6:33918) 2024-03-06T07:38:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 -- 172.21.15.6:0/3854417410 learned_addr learned my addr 172.21.15.6:0/3854417410 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa54c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4fe00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffa500b5060 0x7ffa500c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 -- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 msgr2=0x7ffa500cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 -- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffa500b5060 msgr2=0x7ffa500c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffa500b5060 0x7ffa500c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.324+0000 7ffa4f400640 1 -- 172.21.15.6:0/3854417410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5015fb80 con 0x7ffa500b54a0 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.325+0000 7ffa4f400640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 0x7ffa5015a8d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802b930 tx=0x7ffa380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.325+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3803aa60 con 0x7ffa500b54a0 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.325+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3803ac20 con 0x7ffa500b54a0 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.325+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa380443f0 con 0x7ffa500b54a0 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.325+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5015fe60 con 0x7ffa500b54a0 2024-03-06T07:38:01.328 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.325+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa50160420 con 0x7ffa500b54a0 2024-03-06T07:38:01.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.326+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa500b1410 con 0x7ffa500b54a0 2024-03-06T07:38:01.329 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.327+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa38038020 con 0x7ffa500b54a0 2024-03-06T07:38:01.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.328+0000 7ffa47e00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffa2c06b4d0 0x7ffa2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:01.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.328+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffa380abeb0 con 0x7ffa500b54a0 2024-03-06T07:38:01.331 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.329+0000 7ffa4fe00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffa2c06b4d0 0x7ffa2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:01.332 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.330+0000 7ffa4fe00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffa2c06b4d0 0x7ffa2c06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ffa400099c0 tx=0x7ffa40003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:01.335 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.333+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffa500b1410 con 0x7ffa500b54a0 2024-03-06T07:38:01.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.573+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa5004f3a0 con 0x7ffa500b54a0 2024-03-06T07:38:01.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.574+0000 7ffa47e00640 1 -- 172.21.15.6:0/3854417410 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7ffa3803f070 con 0x7ffa500b54a0 2024-03-06T07:38:01.576 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:01.576 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:01.576 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:01.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffa2c06b4d0 msgr2=0x7ffa2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffa2c06b4d0 0x7ffa2c06d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ffa400099c0 tx=0x7ffa40003990 comp rx=0 tx=0).stop 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 msgr2=0x7ffa5015a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 0x7ffa5015a8d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802b930 tx=0x7ffa380046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 shutdown_connections 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffa2c06b4d0 0x7ffa2c06d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffa50161b40 0x7ffa500cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffa500b54a0 0x7ffa5015a8d0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 --2- 172.21.15.6:0/3854417410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffa500b5060 0x7ffa500c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 >> 172.21.15.6:0/3854417410 conn(0x7ffa5009d220 msgr2=0x7ffa5006bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.576+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 shutdown_connections 2024-03-06T07:38:01.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:01.577+0000 7ffa56c00640 1 -- 172.21.15.6:0/3854417410 wait complete. 2024-03-06T07:38:01.798 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:01.798 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:02.174 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:02.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:03.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.020+0000 7ff4d2000640 1 Processor -- start 2024-03-06T07:38:03.022 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.020+0000 7ff4d2000640 1 -- start start 2024-03-06T07:38:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.022+0000 7ff4d2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af1a0 0x7ff4cc15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.022+0000 7ff4d2000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff4cc0cdc20 0x7ff4cc0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.022+0000 7ff4d2000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc15cc40 con 0x7ff4cc0af580 2024-03-06T07:38:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.022+0000 7ff4d2000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4cc15cf20 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.024 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.022+0000 7ff4d2000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc15d200 con 0x7ff4cc0af1a0 2024-03-06T07:38:03.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff4cc0cdc20 0x7ff4cc0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7ff4cc0cdc20 0x7ff4cc0b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.6:59384/0 (socket says 172.21.15.6:59384) 2024-03-06T07:38:03.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 -- 172.21.15.6:0/2050484602 learned_addr learned my addr 172.21.15.6:0/2050484602 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:03.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cb400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af1a0 0x7ff4cc15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 -- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af1a0 msgr2=0x7ff4cc15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:03.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 --2- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af1a0 0x7ff4cc15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 -- 172.21.15.6:0/2050484602 >> v1:172.21.15.178:6789/0 conn(0x7ff4cc0af580 legacy=0x7ff4cc0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.026 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.023+0000 7ff4cbe00640 1 -- 172.21.15.6:0/2050484602 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc15d4e0 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.025+0000 7ff4cbe00640 1 --2- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0b2dd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009870 tx=0x7ff4c402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2fd003454400387 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.027 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.025+0000 7ff4caa00640 1 -- 172.21.15.6:0/2050484602 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4035020 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.025+0000 7ff4caa00640 1 -- 172.21.15.6:0/2050484602 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c4031640 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.028 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.025+0000 7ff4caa00640 1 -- 172.21.15.6:0/2050484602 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c402d040 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 -- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0cdc20 msgr2=0x7ff4cc0b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 --2- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0b2dd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009870 tx=0x7ff4c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 -- 172.21.15.6:0/2050484602 shutdown_connections 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 --2- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0b2dd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 --2- 172.21.15.6:0/2050484602 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af1a0 0x7ff4cc15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 -- 172.21.15.6:0/2050484602 >> 172.21.15.6:0/2050484602 conn(0x7ff4cc06afb0 msgr2=0x7ff4cc06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 -- 172.21.15.6:0/2050484602 shutdown_connections 2024-03-06T07:38:03.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.026+0000 7ff4d2000640 1 -- 172.21.15.6:0/2050484602 wait complete. 2024-03-06T07:38:03.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.028+0000 7ff4d2000640 1 Processor -- start 2024-03-06T07:38:03.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.029+0000 7ff4d2000640 1 -- start start 2024-03-06T07:38:03.032 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4d2000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0af1a0 0x7ff4cc0af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4d2000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af580 0x7ff4cc0ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4d2000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4d2000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0b36a0 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4d2000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0b3520 con 0x7ff4cc0af1a0 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4d2000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0b3820 con 0x7ff4cc0af580 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4cbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45922/0 (socket says 172.21.15.6:45922) 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.030+0000 7ff4cbe00640 1 -- 172.21.15.6:0/3852824588 learned_addr learned my addr 172.21.15.6:0/3852824588 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cb400640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0af1a0 0x7ff4cc0af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.033 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4c2a00640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af580 0x7ff4cc0ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.034 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cbe00640 1 -- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af580 msgr2=0x7ff4cc0ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cbe00640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af580 0x7ff4cc0ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cbe00640 1 -- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0af1a0 msgr2=0x7ff4cc0af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cbe00640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0af1a0 0x7ff4cc0af9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cbe00640 1 -- 172.21.15.6:0/3852824588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc0affc0 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4c2a00640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af580 0x7ff4cc0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.031+0000 7ff4cbe00640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0cee40 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009870 tx=0x7ff4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.032+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4040070 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.032+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c4039970 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.032+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4035040 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.032+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc0b02a0 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.032+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc0b0860 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.033+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc04f3a0 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.034+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4c404b060 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.035+0000 7ff4c3e00640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.036+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff4c40aba80 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.038 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.036+0000 7ff4cb400640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:03.039 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.037+0000 7ff4cb400640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0113e0 tx=0x7ff4bc00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:03.047 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.045+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4c40ade20 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.291 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.288+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4cc158100 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.290+0000 7ff4c3e00640 1 -- 172.21.15.6:0/3852824588 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7ff4c407a820 con 0x7ff4cc0cdc20 2024-03-06T07:38:03.292 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:03.292 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:03.292 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.292+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff4a806b4d0 msgr2=0x7ff4a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.292+0000 7ff4d2000640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff4a806b4d0 0x7ff4a806d990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0113e0 tx=0x7ff4bc00b040 comp rx=0 tx=0).stop 2024-03-06T07:38:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.292+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 msgr2=0x7ff4cc0cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:03.294 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.292+0000 7ff4d2000640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0cee40 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009870 tx=0x7ff4c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.292+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 shutdown_connections 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff4a806b4d0 0x7ff4a806d990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff4cc0cdc20 0x7ff4cc0cee40 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff4cc0af580 0x7ff4cc0ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 --2- 172.21.15.6:0/3852824588 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4cc0af1a0 0x7ff4cc0af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 >> 172.21.15.6:0/3852824588 conn(0x7ff4cc06afb0 msgr2=0x7ff4cc158470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 shutdown_connections 2024-03-06T07:38:03.295 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:03.293+0000 7ff4d2000640 1 -- 172.21.15.6:0/3852824588 wait complete. 2024-03-06T07:38:03.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:03.517 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:04.176 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:04.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.762+0000 7f2239200640 1 Processor -- start 2024-03-06T07:38:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.762+0000 7f2239200640 1 -- start start 2024-03-06T07:38:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f2239200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22340d4a10 0x7f22340d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f2239200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340b3710 con 0x7f22340d0c80 2024-03-06T07:38:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f2239200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340b39f0 con 0x7f223415e0b0 2024-03-06T07:38:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f2239200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340d10f0 con 0x7f22340d4a10 2024-03-06T07:38:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f222be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f22340d4a10 0x7f22340d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f2233e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f22340d0c80 0x7f22340b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41978/0 (socket says 172.21.15.6:41978) 2024-03-06T07:38:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.764+0000 7f2233e00640 1 -- 172.21.15.6:0/1687186104 learned_addr learned my addr 172.21.15.6:0/1687186104 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.765+0000 7f222be00640 1 -- 172.21.15.6:0/1687186104 >> v1:172.21.15.178:6789/0 conn(0x7f223415e0b0 legacy=0x7f22340cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.765+0000 7f222be00640 1 -- 172.21.15.6:0/1687186104 >> v1:172.21.15.6:6789/0 conn(0x7f22340d0c80 legacy=0x7f22340b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.765+0000 7f222be00640 1 -- 172.21.15.6:0/1687186104 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340b3cd0 con 0x7f22340d4a10 2024-03-06T07:38:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.766+0000 7f222be00640 1 --2- 172.21.15.6:0/1687186104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f22340d0740 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097a0 tx=0x7f221c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f370088eb2e0d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.766+0000 7f2232a00640 1 -- 172.21.15.6:0/1687186104 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c035020 con 0x7f22340d4a10 2024-03-06T07:38:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.766+0000 7f2232a00640 1 -- 172.21.15.6:0/1687186104 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c031830 con 0x7f22340d4a10 2024-03-06T07:38:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.766+0000 7f2232a00640 1 -- 172.21.15.6:0/1687186104 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c031c00 con 0x7f22340d4a10 2024-03-06T07:38:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 -- 172.21.15.6:0/1687186104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 msgr2=0x7f22340d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 --2- 172.21.15.6:0/1687186104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f22340d0740 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097a0 tx=0x7f221c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 -- 172.21.15.6:0/1687186104 shutdown_connections 2024-03-06T07:38:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 --2- 172.21.15.6:0/1687186104 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f22340d0740 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 -- 172.21.15.6:0/1687186104 >> 172.21.15.6:0/1687186104 conn(0x7f22340d3d30 msgr2=0x7f22340d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 -- 172.21.15.6:0/1687186104 shutdown_connections 2024-03-06T07:38:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.767+0000 7f2239200640 1 -- 172.21.15.6:0/1687186104 wait complete. 2024-03-06T07:38:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.769+0000 7f2239200640 1 Processor -- start 2024-03-06T07:38:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.769+0000 7f2239200640 1 -- start start 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2239200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22340d0c80 0x7f22340b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2239200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f223415bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2239200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f223415e0b0 0x7f223415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2239200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340cfa20 con 0x7f22340d4a10 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2239200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223415e490 con 0x7f22340d0c80 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2239200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340cfc60 con 0x7f223415e0b0 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f222be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f223415bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f222be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f223415bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45954/0 (socket says 172.21.15.6:45954) 2024-03-06T07:38:04.774 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f222be00640 1 -- 172.21.15.6:0/2538907439 learned_addr learned my addr 172.21.15.6:0/2538907439 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.770+0000 7f2233400640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22340d0c80 0x7f22340b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f2233e00640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f223415e0b0 0x7f223415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222be00640 1 -- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f223415e0b0 msgr2=0x7f223415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222be00640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f223415e0b0 0x7f223415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222be00640 1 -- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22340d0c80 msgr2=0x7f22340b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222be00640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22340d0c80 0x7f22340b93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222be00640 1 -- 172.21.15.6:0/2538907439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340cb8b0 con 0x7f22340d4a10 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f2233e00640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f223415e0b0 0x7f223415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222be00640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f223415bae0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f221c02bee0 tx=0x7f221c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c041070 con 0x7f22340d4a10 2024-03-06T07:38:04.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c03aa30 con 0x7f22340d4a10 2024-03-06T07:38:04.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22340cbb50 con 0x7f22340d4a10 2024-03-06T07:38:04.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.771+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c035040 con 0x7f22340d4a10 2024-03-06T07:38:04.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.772+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22340cc110 con 0x7f22340d4a10 2024-03-06T07:38:04.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.773+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223404f330 con 0x7f22340d4a10 2024-03-06T07:38:04.776 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.774+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f221c038020 con 0x7f22340d4a10 2024-03-06T07:38:04.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.775+0000 7f222b400640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f221006b4d0 0x7f221006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:04.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.776+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f221c0abd90 con 0x7f22340d4a10 2024-03-06T07:38:04.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.776+0000 7f2233400640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f221006b4d0 0x7f221006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:04.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.777+0000 7f2233400640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f221006b4d0 0x7f221006d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2224004ea0 tx=0x7f2224009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:04.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:04.780+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f221c07abd0 con 0x7f22340d4a10 2024-03-06T07:38:05.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.022+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2234161640 con 0x7f22340d4a10 2024-03-06T07:38:05.025 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.024+0000 7f222b400640 1 -- 172.21.15.6:0/2538907439 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f221c07a4d0 con 0x7f22340d4a10 2024-03-06T07:38:05.026 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:05.026 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:05.026 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:05.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f221006b4d0 msgr2=0x7f221006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f221006b4d0 0x7f221006d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2224004ea0 tx=0x7f2224009040 comp rx=0 tx=0).stop 2024-03-06T07:38:05.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 msgr2=0x7f223415bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:05.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f223415bae0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f221c02bee0 tx=0x7f221c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.029 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 shutdown_connections 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f221006b4d0 0x7f221006d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f223415e0b0 0x7f223415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f22340d4a10 0x7f223415bae0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 --2- 172.21.15.6:0/2538907439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f22340d0c80 0x7f22340b93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 >> 172.21.15.6:0/2538907439 conn(0x7f22340d3d30 msgr2=0x7f22340b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 shutdown_connections 2024-03-06T07:38:05.030 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:05.027+0000 7f2239200640 1 -- 172.21.15.6:0/2538907439 wait complete. 2024-03-06T07:38:05.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:05.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:06.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:06.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.469+0000 7f59a3e00640 1 Processor -- start 2024-03-06T07:38:06.471 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.469+0000 7f59a3e00640 1 -- start start 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f59a3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f59a40d0e50 0x7f59a4168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f59a3e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b3710 con 0x7f59a40d4a10 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f59a3e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b39f0 con 0x7f59a415e0b0 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f59a3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a40b09e0 con 0x7f59a40d0e50 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f599be00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59a40d4a10 0x7f59a40d0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41980/0 (socket says 172.21.15.6:41980) 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f599be00640 1 -- 172.21.15.6:0/2920755502 learned_addr learned my addr 172.21.15.6:0/2920755502 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:06.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f59a3400640 1 --2- 172.21.15.6:0/2920755502 >> v2:172.21.15.6:3300/0 conn(0x7f59a40d0e50 0x7f59a4168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.471+0000 7f59a2000640 1 -- 172.21.15.6:0/2920755502 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167808696 0 0) 0x7f59a40b3710 con 0x7f59a40d4a10 2024-03-06T07:38:06.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.472+0000 7f59a2000640 1 -- 172.21.15.6:0/2920755502 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5988003520 con 0x7f59a40d4a10 2024-03-06T07:38:06.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.472+0000 7f59a3400640 1 -- 172.21.15.6:0/2920755502 >> v1:172.21.15.178:6789/0 conn(0x7f59a415e0b0 legacy=0x7f59a40cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:06.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.472+0000 7f59a3400640 1 -- 172.21.15.6:0/2920755502 >> v1:172.21.15.6:6789/0 conn(0x7f59a40d4a10 legacy=0x7f59a40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.472+0000 7f59a3400640 1 -- 172.21.15.6:0/2920755502 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a40b3cd0 con 0x7f59a40d0e50 2024-03-06T07:38:06.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.473+0000 7f59a3400640 1 --2- 172.21.15.6:0/2920755502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4168c50 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f599c009d90 tx=0x7f599c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66cd6c7955e4f7ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.473+0000 7f59a2000640 1 -- 172.21.15.6:0/2920755502 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c039070 con 0x7f59a40d0e50 2024-03-06T07:38:06.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.473+0000 7f59a2000640 1 -- 172.21.15.6:0/2920755502 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c02cd00 con 0x7f59a40d0e50 2024-03-06T07:38:06.475 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.473+0000 7f59a2000640 1 -- 172.21.15.6:0/2920755502 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c034640 con 0x7f59a40d0e50 2024-03-06T07:38:06.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 -- 172.21.15.6:0/2920755502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 msgr2=0x7f59a4168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 --2- 172.21.15.6:0/2920755502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4168c50 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f599c009d90 tx=0x7f599c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:38:06.476 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 -- 172.21.15.6:0/2920755502 shutdown_connections 2024-03-06T07:38:06.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 --2- 172.21.15.6:0/2920755502 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4168c50 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 -- 172.21.15.6:0/2920755502 >> 172.21.15.6:0/2920755502 conn(0x7f59a40d3d30 msgr2=0x7f59a40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:06.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 -- 172.21.15.6:0/2920755502 shutdown_connections 2024-03-06T07:38:06.477 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.474+0000 7f59a3e00640 1 -- 172.21.15.6:0/2920755502 wait complete. 2024-03-06T07:38:06.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.476+0000 7f59a3e00640 1 Processor -- start 2024-03-06T07:38:06.478 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.476+0000 7f59a3e00640 1 -- start start 2024-03-06T07:38:06.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4161140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.480 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59a40d4a10 0x7f59a4150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59a415e0b0 0x7f59a40cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a415e490 con 0x7f59a40d0e50 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40cfa20 con 0x7f59a415e0b0 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a40cfc60 con 0x7f59a40d4a10 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a2a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4161140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45964/0 (socket says 172.21.15.6:45964) 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.477+0000 7f59a2a00640 1 -- 172.21.15.6:0/111542157 learned_addr learned my addr 172.21.15.6:0/111542157 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f599be00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59a40d4a10 0x7f59a4150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a3400640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59a415e0b0 0x7f59a40cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a2a00640 1 -- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59a40d4a10 msgr2=0x7f59a4150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a2a00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59a40d4a10 0x7f59a4150100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a2a00640 1 -- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59a415e0b0 msgr2=0x7f59a40cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a2a00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59a415e0b0 0x7f59a40cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a2a00640 1 -- 172.21.15.6:0/111542157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a4161680 con 0x7f59a40d0e50 2024-03-06T07:38:06.481 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a3400640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59a415e0b0 0x7f59a40cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a2a00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4161140 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f599400c3c0 tx=0x7f599400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59940052d0 con 0x7f59a40d0e50 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5994005490 con 0x7f59a40d0e50 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5994007260 con 0x7f59a40d0e50 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.478+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a40cb170 con 0x7f59a40d0e50 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.479+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a40cb730 con 0x7f59a40d0e50 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.480+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a404f330 con 0x7f59a40d0e50 2024-03-06T07:38:06.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.481+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5994007400 con 0x7f59a40d0e50 2024-03-06T07:38:06.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.481+0000 7f599b400640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f598006b4d0 0x7f598006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:06.483 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.482+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f59940897c0 con 0x7f59a40d0e50 2024-03-06T07:38:06.484 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.482+0000 7f599be00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f598006b4d0 0x7f598006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:06.485 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.483+0000 7f599be00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f598006b4d0 0x7f598006d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f598c0011e0 tx=0x7f598c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:06.488 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.486+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59a4161680 con 0x7f59a40d0e50 2024-03-06T07:38:06.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.727+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59a40d0ac0 con 0x7f59a40d0e50 2024-03-06T07:38:06.730 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.728+0000 7f599b400640 1 -- 172.21.15.6:0/111542157 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f5994058560 con 0x7f59a40d0e50 2024-03-06T07:38:06.730 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:06.731 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:06.731 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f598006b4d0 msgr2=0x7f598006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f598006b4d0 0x7f598006d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f598c0011e0 tx=0x7f598c009040 comp rx=0 tx=0).stop 2024-03-06T07:38:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 msgr2=0x7f59a4161140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4161140 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f599400c3c0 tx=0x7f599400c880 comp rx=0 tx=0).stop 2024-03-06T07:38:06.733 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 shutdown_connections 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f598006b4d0 0x7f598006d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59a415e0b0 0x7f59a40cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f59a40d4a10 0x7f59a4150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 --2- 172.21.15.6:0/111542157 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59a40d0e50 0x7f59a4161140 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 >> 172.21.15.6:0/111542157 conn(0x7f59a40d3d30 msgr2=0x7f59a40b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 shutdown_connections 2024-03-06T07:38:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:06.731+0000 7f59a3e00640 1 -- 172.21.15.6:0/111542157 wait complete. 2024-03-06T07:38:06.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:06.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:07.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:08.179 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:08.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.186+0000 7fe24fa00640 1 Processor -- start 2024-03-06T07:38:08.189 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.186+0000 7fe24fa00640 1 -- start start 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.188+0000 7fe24fa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe248162c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.188+0000 7fe24fa00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2480ce090 con 0x7fe2480712c0 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.188+0000 7fe24fa00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2480ce370 con 0x7fe248070ee0 2024-03-06T07:38:08.190 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.188+0000 7fe24fa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480ceca0 con 0x7fe248072da0 2024-03-06T07:38:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.189+0000 7fe24da00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe2480712c0 0x7fe24806cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41986/0 (socket says 172.21.15.6:41986) 2024-03-06T07:38:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.189+0000 7fe24da00640 1 -- 172.21.15.6:0/3652726014 learned_addr learned my addr 172.21.15.6:0/3652726014 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:08.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.189+0000 7fe247e00640 1 --2- 172.21.15.6:0/3652726014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe248162c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.189+0000 7fe247400640 1 -- 172.21.15.6:0/3652726014 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589503763 0 0) 0x7fe2480ce090 con 0x7fe2480712c0 2024-03-06T07:38:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.189+0000 7fe247400640 1 -- 172.21.15.6:0/3652726014 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe230003520 con 0x7fe2480712c0 2024-03-06T07:38:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.190+0000 7fe247e00640 1 -- 172.21.15.6:0/3652726014 >> v1:172.21.15.178:6789/0 conn(0x7fe248070ee0 legacy=0x7fe24806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.190+0000 7fe247400640 1 -- 172.21.15.6:0/3652726014 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552865288 0 0) 0x7fe230003520 con 0x7fe2480712c0 2024-03-06T07:38:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.190+0000 7fe247e00640 1 -- 172.21.15.6:0/3652726014 >> v1:172.21.15.6:6789/0 conn(0x7fe2480712c0 legacy=0x7fe24806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.190+0000 7fe247e00640 1 -- 172.21.15.6:0/3652726014 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480ce650 con 0x7fe248072da0 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.191+0000 7fe247e00640 1 --2- 172.21.15.6:0/3652726014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe248162c40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe2480cee20 tx=0x7fe23402b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e546167523c6ec5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:08.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe247400640 1 -- 172.21.15.6:0/3652726014 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234039070 con 0x7fe248072da0 2024-03-06T07:38:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe247400640 1 -- 172.21.15.6:0/3652726014 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234030070 con 0x7fe248072da0 2024-03-06T07:38:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe247400640 1 -- 172.21.15.6:0/3652726014 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2400019f0 con 0x7fe248072da0 2024-03-06T07:38:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe24fa00640 1 -- 172.21.15.6:0/3652726014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 msgr2=0x7fe248162c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe24fa00640 1 --2- 172.21.15.6:0/3652726014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe248162c40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe2480cee20 tx=0x7fe23402b7f0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe24fa00640 1 -- 172.21.15.6:0/3652726014 shutdown_connections 2024-03-06T07:38:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe24fa00640 1 --2- 172.21.15.6:0/3652726014 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe248162c40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.192+0000 7fe24fa00640 1 -- 172.21.15.6:0/3652726014 >> 172.21.15.6:0/3652726014 conn(0x7fe24806afb0 msgr2=0x7fe24806b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.193+0000 7fe24fa00640 1 -- 172.21.15.6:0/3652726014 shutdown_connections 2024-03-06T07:38:08.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.193+0000 7fe24fa00640 1 -- 172.21.15.6:0/3652726014 wait complete. 2024-03-06T07:38:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.194+0000 7fe24fa00640 1 Processor -- start 2024-03-06T07:38:08.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.194+0000 7fe24fa00640 1 -- start start 2024-03-06T07:38:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.195+0000 7fe24fa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe248070ee0 0x7fe2480b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.195+0000 7fe24fa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 0x7fe24815ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.195+0000 7fe24fa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe24815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.198 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.195+0000 7fe24fa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480cdda0 con 0x7fe2480712c0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.195+0000 7fe24fa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480cdc20 con 0x7fe248070ee0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.195+0000 7fe24fa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24806e7d0 con 0x7fe248072da0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 0x7fe24815ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 0x7fe24815ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45976/0 (socket says 172.21.15.6:45976) 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 -- 172.21.15.6:0/2501856070 learned_addr learned my addr 172.21.15.6:0/2501856070 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe24da00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe24815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe24d000640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe248070ee0 0x7fe2480b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 -- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 msgr2=0x7fe24815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe24815f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 -- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe248070ee0 msgr2=0x7fe2480b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe248070ee0 0x7fe2480b3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 -- 172.21.15.6:0/2501856070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24809be60 con 0x7fe2480712c0 2024-03-06T07:38:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.196+0000 7fe247e00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 0x7fe24815ee50 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fe234037040 tx=0x7fe2340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:08.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.197+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234044070 con 0x7fe2480712c0 2024-03-06T07:38:08.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.197+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234034680 con 0x7fe2480712c0 2024-03-06T07:38:08.200 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.197+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24809c3d0 con 0x7fe2480712c0 2024-03-06T07:38:08.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.197+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234039040 con 0x7fe2480712c0 2024-03-06T07:38:08.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.197+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24815f8d0 con 0x7fe2480712c0 2024-03-06T07:38:08.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.198+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe248069510 con 0x7fe2480712c0 2024-03-06T07:38:08.201 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.199+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe234047070 con 0x7fe2480712c0 2024-03-06T07:38:08.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.200+0000 7fe244c00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe22806b4d0 0x7fe22806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:08.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.200+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe2340abfe0 con 0x7fe2480712c0 2024-03-06T07:38:08.202 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.201+0000 7fe24d000640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe22806b4d0 0x7fe22806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:08.203 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.202+0000 7fe24d000640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe22806b4d0 0x7fe22806d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe23c000f60 tx=0x7fe23c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:08.206 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.204+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe23407af90 con 0x7fe2480712c0 2024-03-06T07:38:08.469 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.467+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe24804f3a0 con 0x7fe2480712c0 2024-03-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.468+0000 7fe244c00640 1 -- 172.21.15.6:0/2501856070 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fe23407a890 con 0x7fe2480712c0 2024-03-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:08.470 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:08.471 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:08.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe22806b4d0 msgr2=0x7fe22806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe22806b4d0 0x7fe22806d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe23c000f60 tx=0x7fe23c009040 comp rx=0 tx=0).stop 2024-03-06T07:38:08.473 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 msgr2=0x7fe24815ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 0x7fe24815ee50 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fe234037040 tx=0x7fe2340046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 shutdown_connections 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe22806b4d0 0x7fe22806d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe248072da0 0x7fe24815f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe2480712c0 0x7fe24815ee50 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 --2- 172.21.15.6:0/2501856070 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe248070ee0 0x7fe2480b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 >> 172.21.15.6:0/2501856070 conn(0x7fe24806afb0 msgr2=0x7fe248164d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 shutdown_connections 2024-03-06T07:38:08.474 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:08.471+0000 7fe24fa00640 1 -- 172.21.15.6:0/2501856070 wait complete. 2024-03-06T07:38:08.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:08.693 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:09.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:09.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.927+0000 7fc19ee00640 1 Processor -- start 2024-03-06T07:38:09.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.927+0000 7fc19ee00640 1 -- start start 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.928+0000 7fc19ee00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc198145300 0x7fc1980b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.928+0000 7fc19ee00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc198070f10 0x7fc1980cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19ee00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1980714d0 con 0x7fc1980cce90 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19ee00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1980cd300 con 0x7fc198145300 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19ee00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980717b0 con 0x7fc198070f10 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc198145300 0x7fc1980b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc198145300 0x7fc1980b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45982/0 (socket says 172.21.15.6:45982) 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 -- 172.21.15.6:0/2290476223 learned_addr learned my addr 172.21.15.6:0/2290476223 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19d000640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc198070f10 0x7fc1980cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.932 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 -- 172.21.15.6:0/2290476223 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc198070f10 msgr2=0x7fc1980cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc198070f10 0x7fc1980cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 -- 172.21.15.6:0/2290476223 >> v1:172.21.15.178:6789/0 conn(0x7fc1980cce90 legacy=0x7fc1980cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19da00640 1 -- 172.21.15.6:0/2290476223 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc198071a90 con 0x7fc198145300 2024-03-06T07:38:09.933 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.929+0000 7fc19d000640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc198070f10 0x7fc1980cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:09.934 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.931+0000 7fc19da00640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198145300 0x7fc1980b49a0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009870 tx=0x7fc18c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e6d618f2495b7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.931+0000 7fc197e00640 1 -- 172.21.15.6:0/2290476223 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c035020 con 0x7fc198145300 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.931+0000 7fc197e00640 1 -- 172.21.15.6:0/2290476223 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c031640 con 0x7fc198145300 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.931+0000 7fc197e00640 1 -- 172.21.15.6:0/2290476223 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c02d040 con 0x7fc198145300 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 -- 172.21.15.6:0/2290476223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198145300 msgr2=0x7fc1980b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198145300 0x7fc1980b49a0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009870 tx=0x7fc18c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 -- 172.21.15.6:0/2290476223 shutdown_connections 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc198070f10 0x7fc1980cc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 --2- 172.21.15.6:0/2290476223 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198145300 0x7fc1980b49a0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 -- 172.21.15.6:0/2290476223 >> 172.21.15.6:0/2290476223 conn(0x7fc19806b3d0 msgr2=0x7fc19806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 -- 172.21.15.6:0/2290476223 shutdown_connections 2024-03-06T07:38:09.935 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.932+0000 7fc19ee00640 1 -- 172.21.15.6:0/2290476223 wait complete. 2024-03-06T07:38:09.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.935+0000 7fc19ee00640 1 Processor -- start 2024-03-06T07:38:09.937 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.935+0000 7fc19ee00640 1 -- start start 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19ee00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 0x7fc198140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19ee00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1980cce90 0x7fc19806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19ee00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc198145300 0x7fc1981f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19ee00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1981456e0 con 0x7fc198070f10 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19ee00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1980b4ee0 con 0x7fc198145300 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19ee00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19815d580 con 0x7fc1980cce90 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 0x7fc198140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.940 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 0x7fc198140ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45996/0 (socket says 172.21.15.6:45996) 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19da00640 1 -- 172.21.15.6:0/414811836 learned_addr learned my addr 172.21.15.6:0/414811836 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19d000640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1980cce90 0x7fc19806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.937+0000 7fc19e400640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc198145300 0x7fc1981f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19da00640 1 -- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1980cce90 msgr2=0x7fc19806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19da00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1980cce90 0x7fc19806d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19da00640 1 -- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc198145300 msgr2=0x7fc1981f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19da00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc198145300 0x7fc1981f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19da00640 1 -- 172.21.15.6:0/414811836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1981f43a0 con 0x7fc198070f10 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19e400640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc198145300 0x7fc1981f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:09.941 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19da00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 0x7fc198140ee0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc18c02b930 tx=0x7fc18c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c040070 con 0x7fc198070f10 2024-03-06T07:38:09.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c039c10 con 0x7fc198070f10 2024-03-06T07:38:09.942 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c035040 con 0x7fc198070f10 2024-03-06T07:38:09.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.938+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1981f4680 con 0x7fc198070f10 2024-03-06T07:38:09.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.939+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19806cbd0 con 0x7fc198070f10 2024-03-06T07:38:09.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.940+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19804f330 con 0x7fc198070f10 2024-03-06T07:38:09.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.941+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc18c038020 con 0x7fc198070f10 2024-03-06T07:38:09.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.941+0000 7fc195600640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc17806b4d0 0x7fc17806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:09.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.942+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc18c0abcf0 con 0x7fc198070f10 2024-03-06T07:38:09.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.942+0000 7fc19d000640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc17806b4d0 0x7fc17806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:09.945 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.943+0000 7fc19d000640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc17806b4d0 0x7fc17806d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc1980b4e90 tx=0x7fc184008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:09.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:09.946+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc18c0b0070 con 0x7fc198070f10 2024-03-06T07:38:10.182 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:10.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.222+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1980cbe00 con 0x7fc198070f10 2024-03-06T07:38:10.224 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.222+0000 7fc195600640 1 -- 172.21.15.6:0/414811836 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fc18c07abc0 con 0x7fc198070f10 2024-03-06T07:38:10.225 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:10.225 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:10.225 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:10.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.225+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc17806b4d0 msgr2=0x7fc17806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.225+0000 7fc19ee00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc17806b4d0 0x7fc17806d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc1980b4e90 tx=0x7fc184008040 comp rx=0 tx=0).stop 2024-03-06T07:38:10.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 msgr2=0x7fc198140ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:10.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 0x7fc198140ee0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc18c02b930 tx=0x7fc18c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 shutdown_connections 2024-03-06T07:38:10.228 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc17806b4d0 0x7fc17806d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc198145300 0x7fc1981f39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1980cce90 0x7fc19806d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 --2- 172.21.15.6:0/414811836 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc198070f10 0x7fc198140ee0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:10.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 >> 172.21.15.6:0/414811836 conn(0x7fc19806b3d0 msgr2=0x7fc1980ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:10.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 shutdown_connections 2024-03-06T07:38:10.229 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:10.226+0000 7fc19ee00640 1 -- 172.21.15.6:0/414811836 wait complete. 2024-03-06T07:38:10.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:10.450 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:11.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:11.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.678+0000 7fbec6a00640 1 Processor -- start 2024-03-06T07:38:11.680 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.678+0000 7fbec6a00640 1 -- start start 2024-03-06T07:38:11.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbec6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbec00b3c90 0x7fbec00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbec6a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fbec00cf5c0 0x7fbec00b4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbec6a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec00b45b0 con 0x7fbec006cb10 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbec6a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00ce160 con 0x7fbec00b3c90 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbec6a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec00b4860 con 0x7fbec00cf5c0 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbebfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbec00b3c90 0x7fbec00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbebfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbec00b3c90 0x7fbec00cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46002/0 (socket says 172.21.15.6:46002) 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbec4a00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbec006cb10 0x7fbec006cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41994/0 (socket says 172.21.15.6:41994) 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbebfe00640 1 -- 172.21.15.6:0/748058573 learned_addr learned my addr 172.21.15.6:0/748058573 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.681+0000 7fbeb7e00640 1 --2- 172.21.15.6:0/748058573 >> v2:172.21.15.178:3300/0 conn(0x7fbec00cf5c0 0x7fbec00b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.682+0000 7fbebfe00640 1 -- 172.21.15.6:0/748058573 >> v2:172.21.15.178:3300/0 conn(0x7fbec00cf5c0 msgr2=0x7fbec00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.682+0000 7fbebfe00640 1 --2- 172.21.15.6:0/748058573 >> v2:172.21.15.178:3300/0 conn(0x7fbec00cf5c0 0x7fbec00b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.682+0000 7fbebfe00640 1 -- 172.21.15.6:0/748058573 >> v1:172.21.15.6:6789/0 conn(0x7fbec006cb10 legacy=0x7fbec006cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.685 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.682+0000 7fbebfe00640 1 -- 172.21.15.6:0/748058573 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec00b4b40 con 0x7fbec00b3c90 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.683+0000 7fbebfe00640 1 --2- 172.21.15.6:0/748058573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec00cdc20 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fbec00b2aa0 tx=0x7fbeb002dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9a835cabf4b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.683+0000 7fbebf400640 1 -- 172.21.15.6:0/748058573 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb003b070 con 0x7fbec00b3c90 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.683+0000 7fbebf400640 1 -- 172.21.15.6:0/748058573 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb002ea20 con 0x7fbec00b3c90 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.683+0000 7fbebf400640 1 -- 172.21.15.6:0/748058573 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeb002edf0 con 0x7fbec00b3c90 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 -- 172.21.15.6:0/748058573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 msgr2=0x7fbec00cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 --2- 172.21.15.6:0/748058573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec00cdc20 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fbec00b2aa0 tx=0x7fbeb002dc70 comp rx=0 tx=0).stop 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 -- 172.21.15.6:0/748058573 shutdown_connections 2024-03-06T07:38:11.686 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 --2- 172.21.15.6:0/748058573 >> v2:172.21.15.178:3300/0 conn(0x7fbec00cf5c0 0x7fbec00b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 --2- 172.21.15.6:0/748058573 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec00cdc20 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 -- 172.21.15.6:0/748058573 >> 172.21.15.6:0/748058573 conn(0x7fbec006afb0 msgr2=0x7fbec006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:11.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 -- 172.21.15.6:0/748058573 shutdown_connections 2024-03-06T07:38:11.687 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.684+0000 7fbec6a00640 1 -- 172.21.15.6:0/748058573 wait complete. 2024-03-06T07:38:11.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.686+0000 7fbec6a00640 1 Processor -- start 2024-03-06T07:38:11.688 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.686+0000 7fbec6a00640 1 -- start start 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbec6a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbec006cb10 0x7fbec009b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbec6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec01497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbec6a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbec00cf5c0 0x7fbec00cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbec6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec015b340 con 0x7fbec00b3c90 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbec6a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec00b5100 con 0x7fbec006cb10 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbec6a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec006d690 con 0x7fbec00cf5c0 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbeb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec01497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbeb7e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec01497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46008/0 (socket says 172.21.15.6:46008) 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.687+0000 7fbeb7e00640 1 -- 172.21.15.6:0/1060743705 learned_addr learned my addr 172.21.15.6:0/1060743705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbebfe00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbec006cb10 0x7fbec009b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbec4a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbec00cf5c0 0x7fbec00cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbeb7e00640 1 -- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbec00cf5c0 msgr2=0x7fbec00cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.691 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbeb7e00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbec00cf5c0 0x7fbec00cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbeb7e00640 1 -- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbec006cb10 msgr2=0x7fbec009b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbeb7e00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbec006cb10 0x7fbec009b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbeb7e00640 1 -- 172.21.15.6:0/1060743705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec009bc60 con 0x7fbec00b3c90 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbebfe00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbec006cb10 0x7fbec009b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbec4a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbec00cf5c0 0x7fbec00cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbeb7e00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec01497b0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fbea800d5f0 tx=0x7fbea800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea80125f0 con 0x7fbec00b3c90 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea8019070 con 0x7fbec00b3c90 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec009bf40 con 0x7fbec00b3c90 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.688+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea80129d0 con 0x7fbec00b3c90 2024-03-06T07:38:11.692 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.689+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec009c500 con 0x7fbec00b3c90 2024-03-06T07:38:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.690+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec0069510 con 0x7fbec00b3c90 2024-03-06T07:38:11.693 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.691+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbea8012b70 con 0x7fbec00b3c90 2024-03-06T07:38:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.691+0000 7fbebcc00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe9c06b4d0 0x7fbe9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.692+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbea808a640 con 0x7fbec00b3c90 2024-03-06T07:38:11.694 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.692+0000 7fbebfe00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe9c06b4d0 0x7fbe9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:11.695 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.693+0000 7fbebfe00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe9c06b4d0 0x7fbe9c06d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0038000 tx=0x7fbeb0032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:11.698 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.696+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbea8059470 con 0x7fbec00b3c90 2024-03-06T07:38:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.936+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbec004f3a0 con 0x7fbec00b3c90 2024-03-06T07:38:11.939 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.937+0000 7fbebcc00640 1 -- 172.21.15.6:0/1060743705 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fbea8058d70 con 0x7fbec00b3c90 2024-03-06T07:38:11.940 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:11.940 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:11.940 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.940+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe9c06b4d0 msgr2=0x7fbe9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.940+0000 7fbec6a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe9c06b4d0 0x7fbe9c06d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0038000 tx=0x7fbeb0032000 comp rx=0 tx=0).stop 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.940+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 msgr2=0x7fbec01497b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.940+0000 7fbec6a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec01497b0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fbea800d5f0 tx=0x7fbea800dab0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.940+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 shutdown_connections 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.940+0000 7fbec6a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fbe9c06b4d0 0x7fbe9c06d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.941+0000 7fbec6a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fbec00cf5c0 0x7fbec00cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.941+0000 7fbec6a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbec00b3c90 0x7fbec01497b0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.941+0000 7fbec6a00640 1 --2- 172.21.15.6:0/1060743705 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbec006cb10 0x7fbec009b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:11.943 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.941+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 >> 172.21.15.6:0/1060743705 conn(0x7fbec006afb0 msgr2=0x7fbec00acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.941+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 shutdown_connections 2024-03-06T07:38:11.944 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:11.941+0000 7fbec6a00640 1 -- 172.21.15.6:0/1060743705 wait complete. 2024-03-06T07:38:12.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:12.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:13.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.393+0000 7fb748800640 1 Processor -- start 2024-03-06T07:38:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.393+0000 7fb748800640 1 -- start start 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.395+0000 7fb748800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb7400cfd90 0x7fb7400d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.395+0000 7fb748800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7400d2760 0x7fb7400b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.395+0000 7fb748800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7400cea50 con 0x7fb740161b40 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.395+0000 7fb748800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7400b5770 con 0x7fb7400cfd90 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.395+0000 7fb748800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400cf900 con 0x7fb7400d2760 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fb740161b40 0x7fb7400b5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46876/0 (socket says 172.21.15.6:46876) 2024-03-06T07:38:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745e00640 1 -- 172.21.15.6:0/2774927562 learned_addr learned my addr 172.21.15.6:0/2774927562 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745400640 1 --2- 172.21.15.6:0/2774927562 >> v2:172.21.15.6:3300/0 conn(0x7fb7400cfd90 0x7fb7400d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb746800640 1 --2- 172.21.15.6:0/2774927562 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7400d2760 0x7fb7400b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745400640 1 -- 172.21.15.6:0/2774927562 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7400d2760 msgr2=0x7fb7400b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745400640 1 --2- 172.21.15.6:0/2774927562 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7400d2760 0x7fb7400b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745400640 1 -- 172.21.15.6:0/2774927562 >> v1:172.21.15.6:6789/0 conn(0x7fb740161b40 legacy=0x7fb7400b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.396+0000 7fb745400640 1 -- 172.21.15.6:0/2774927562 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400cecf0 con 0x7fb7400cfd90 2024-03-06T07:38:13.402 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.397+0000 7fb745400640 1 --2- 172.21.15.6:0/2774927562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400cfd90 0x7fb7400d2220 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fb7300097a0 tx=0x7fb73002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b330db09cc4fcb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.398+0000 7fb744a00640 1 -- 172.21.15.6:0/2774927562 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730035020 con 0x7fb7400cfd90 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.398+0000 7fb744a00640 1 -- 172.21.15.6:0/2774927562 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb730031810 con 0x7fb7400cfd90 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.398+0000 7fb744a00640 1 -- 172.21.15.6:0/2774927562 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730031be0 con 0x7fb7400cfd90 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.398+0000 7fb748800640 1 -- 172.21.15.6:0/2774927562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400cfd90 msgr2=0x7fb7400d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 --2- 172.21.15.6:0/2774927562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400cfd90 0x7fb7400d2220 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fb7300097a0 tx=0x7fb73002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 -- 172.21.15.6:0/2774927562 shutdown_connections 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 --2- 172.21.15.6:0/2774927562 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb7400d2760 0x7fb7400b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 --2- 172.21.15.6:0/2774927562 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400cfd90 0x7fb7400d2220 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.403 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 -- 172.21.15.6:0/2774927562 >> 172.21.15.6:0/2774927562 conn(0x7fb74009d220 msgr2=0x7fb7400d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:13.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 -- 172.21.15.6:0/2774927562 shutdown_connections 2024-03-06T07:38:13.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.399+0000 7fb748800640 1 -- 172.21.15.6:0/2774927562 wait complete. 2024-03-06T07:38:13.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.400+0000 7fb748800640 1 Processor -- start 2024-03-06T07:38:13.404 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.401+0000 7fb748800640 1 -- start start 2024-03-06T07:38:13.407 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb748800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7400cfd90 0x7fb7400b7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb748800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 0x7fb7400cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb748800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb740161b40 0x7fb7400b9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb748800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400cf590 con 0x7fb7400d2760 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb748800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400cf2b0 con 0x7fb7400cfd90 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb748800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400b2ff0 con 0x7fb740161b40 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.402+0000 7fb745400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 0x7fb7400cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 0x7fb7400cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46140/0 (socket says 172.21.15.6:46140) 2024-03-06T07:38:13.408 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 -- 172.21.15.6:0/784231642 learned_addr learned my addr 172.21.15.6:0/784231642 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745e00640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7400cfd90 0x7fb7400b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 -- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb740161b40 msgr2=0x7fb7400b9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb746800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb740161b40 0x7fb7400b9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb740161b40 0x7fb7400b9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 -- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7400cfd90 msgr2=0x7fb7400b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7400cfd90 0x7fb7400b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745400640 1 -- 172.21.15.6:0/784231642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400ccf60 con 0x7fb7400d2760 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.403+0000 7fb745e00640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7400cfd90 0x7fb7400b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.404+0000 7fb745400640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 0x7fb7400cca20 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fb7300096f0 tx=0x7fb7300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.404+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730040070 con 0x7fb7400d2760 2024-03-06T07:38:13.409 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.404+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb730039e90 con 0x7fb7400d2760 2024-03-06T07:38:13.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.404+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730035040 con 0x7fb7400d2760 2024-03-06T07:38:13.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.404+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7400cd1e0 con 0x7fb7400d2760 2024-03-06T07:38:13.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.404+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7400cd7a0 con 0x7fb7400d2760 2024-03-06T07:38:13.410 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.405+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74004f3a0 con 0x7fb7400d2760 2024-03-06T07:38:13.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.406+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb73004b060 con 0x7fb7400d2760 2024-03-06T07:38:13.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.408+0000 7fb73a000640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb72406b4d0 0x7fb72406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:13.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.408+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb7300abf60 con 0x7fb7400d2760 2024-03-06T07:38:13.411 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.408+0000 7fb745e00640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb72406b4d0 0x7fb72406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:13.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.409+0000 7fb745e00640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb72406b4d0 0x7fb72406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb734000f60 tx=0x7fb734009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:13.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.412+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7400ccf60 con 0x7fb7400d2760 2024-03-06T07:38:13.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.649+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7400d1610 con 0x7fb7400d2760 2024-03-06T07:38:13.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.650+0000 7fb73a000640 1 -- 172.21.15.6:0/784231642 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fb73004a2c0 con 0x7fb7400d2760 2024-03-06T07:38:13.652 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:13.653 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:13.653 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb72406b4d0 msgr2=0x7fb72406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb72406b4d0 0x7fb72406d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb734000f60 tx=0x7fb734009040 comp rx=0 tx=0).stop 2024-03-06T07:38:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 msgr2=0x7fb7400cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 0x7fb7400cca20 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fb7300096f0 tx=0x7fb7300046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 shutdown_connections 2024-03-06T07:38:13.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb72406b4d0 0x7fb72406d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb740161b40 0x7fb7400b9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb7400d2760 0x7fb7400cca20 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 --2- 172.21.15.6:0/784231642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb7400cfd90 0x7fb7400b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.652+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 >> 172.21.15.6:0/784231642 conn(0x7fb74009d220 msgr2=0x7fb7400b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.653+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 shutdown_connections 2024-03-06T07:38:13.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:13.653+0000 7fb748800640 1 -- 172.21.15.6:0/784231642 wait complete. 2024-03-06T07:38:13.873 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:13.873 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:14.186 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:14.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:15.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.103+0000 7f252b200640 1 Processor -- start 2024-03-06T07:38:15.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.103+0000 7f252b200640 1 -- start start 2024-03-06T07:38:15.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.105+0000 7f252b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c0cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.105+0000 7f252b200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c0714d0 con 0x7f252c0cce90 2024-03-06T07:38:15.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.105+0000 7f252b200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c0717b0 con 0x7f252c145300 2024-03-06T07:38:15.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.105+0000 7f252b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0cd300 con 0x7f252c070f10 2024-03-06T07:38:15.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.106+0000 7f252a800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f252c0cce90 0x7f252c15c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46882/0 (socket says 172.21.15.6:46882) 2024-03-06T07:38:15.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.106+0000 7f252a800640 1 -- 172.21.15.6:0/2479293821 learned_addr learned my addr 172.21.15.6:0/2479293821 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:15.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.106+0000 7f2529400640 1 --2- 172.21.15.6:0/2479293821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c0cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841758563 0 0) 0x7f252c0714d0 con 0x7f252c0cce90 2024-03-06T07:38:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2510003520 con 0x7f252c0cce90 2024-03-06T07:38:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729396996 0 0) 0x7f252c0717b0 con 0x7f252c145300 2024-03-06T07:38:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f252c0714d0 con 0x7f252c145300 2024-03-06T07:38:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2529400640 1 -- 172.21.15.6:0/2479293821 >> v1:172.21.15.178:6789/0 conn(0x7f252c145300 legacy=0x7f252c0b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2529400640 1 -- 172.21.15.6:0/2479293821 >> v1:172.21.15.6:6789/0 conn(0x7f252c0cce90 legacy=0x7f252c15c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.107+0000 7f2529400640 1 -- 172.21.15.6:0/2479293821 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c071a90 con 0x7f252c070f10 2024-03-06T07:38:15.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.108+0000 7f2529400640 1 --2- 172.21.15.6:0/2479293821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c0cc950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f25140097d0 tx=0x7f251402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67bfc1829d7569e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:15.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.109+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 <== mon.2 v2:172.21.15.178:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514035020 con 0x7f252c070f10 2024-03-06T07:38:15.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.109+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 <== mon.2 v2:172.21.15.178:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2514031890 con 0x7f252c070f10 2024-03-06T07:38:15.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.109+0000 7f2528a00640 1 -- 172.21.15.6:0/2479293821 <== mon.2 v2:172.21.15.178:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2514031c60 con 0x7f252c070f10 2024-03-06T07:38:15.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.109+0000 7f252b200640 1 -- 172.21.15.6:0/2479293821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 msgr2=0x7f252c0cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.111 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.109+0000 7f252b200640 1 --2- 172.21.15.6:0/2479293821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c0cc950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f25140097d0 tx=0x7f251402b930 comp rx=0 tx=0).stop 2024-03-06T07:38:15.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.110+0000 7f252b200640 1 -- 172.21.15.6:0/2479293821 shutdown_connections 2024-03-06T07:38:15.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.110+0000 7f252b200640 1 --2- 172.21.15.6:0/2479293821 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c0cc950 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.110+0000 7f252b200640 1 -- 172.21.15.6:0/2479293821 >> 172.21.15.6:0/2479293821 conn(0x7f252c06b3d0 msgr2=0x7f252c06b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:15.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.110+0000 7f252b200640 1 -- 172.21.15.6:0/2479293821 shutdown_connections 2024-03-06T07:38:15.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.110+0000 7f252b200640 1 -- 172.21.15.6:0/2479293821 wait complete. 2024-03-06T07:38:15.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.111+0000 7f252b200640 1 Processor -- start 2024-03-06T07:38:15.114 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.112+0000 7f252b200640 1 -- start start 2024-03-06T07:38:15.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.112+0000 7f252b200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.112+0000 7f252b200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f252c0cce90 0x7f252c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.112+0000 7f252b200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 0x7f252c15e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.112+0000 7f252b200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c15cc80 con 0x7f252c145300 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.112+0000 7f252b200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1456e0 con 0x7f252c0cce90 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252b200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c15ce00 con 0x7f252c070f10 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 0x7f252c15e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 0x7f252c15e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46144/0 (socket says 172.21.15.6:46144) 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 -- 172.21.15.6:0/232081182 learned_addr learned my addr 172.21.15.6:0/232081182 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 -- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 msgr2=0x7f252c141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 -- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f252c0cce90 msgr2=0x7f252c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f252c0cce90 0x7f252c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.116 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.113+0000 7f252a800640 1 -- 172.21.15.6:0/232081182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c1f4480 con 0x7f252c145300 2024-03-06T07:38:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.114+0000 7f252a800640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 0x7f252c15e7c0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f252400cbc0 tx=0x7f252400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.114+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524017070 con 0x7f252c145300 2024-03-06T07:38:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.114+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f252400ace0 con 0x7f252c145300 2024-03-06T07:38:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.114+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524012600 con 0x7f252c145300 2024-03-06T07:38:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.114+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252c141250 con 0x7f252c145300 2024-03-06T07:38:15.117 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.114+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c141810 con 0x7f252c145300 2024-03-06T07:38:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.116+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c04f330 con 0x7f252c145300 2024-03-06T07:38:15.118 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.116+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f252401a070 con 0x7f252c145300 2024-03-06T07:38:15.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.117+0000 7f2522000640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f250806b4d0 0x7f250806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:15.119 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.117+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2524089f90 con 0x7f252c145300 2024-03-06T07:38:15.120 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.118+0000 7f2529e00640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f250806b4d0 0x7f250806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:15.121 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.119+0000 7f2529e00640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f250806b4d0 0x7f250806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f251c0010e0 tx=0x7f251c007950 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:15.124 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.122+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2524058770 con 0x7f252c145300 2024-03-06T07:38:15.374 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.372+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f252c0cbd60 con 0x7f252c145300 2024-03-06T07:38:15.375 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.373+0000 7f2522000640 1 -- 172.21.15.6:0/232081182 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f252405c810 con 0x7f252c145300 2024-03-06T07:38:15.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:15.375 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:15.376 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:15.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.375+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f250806b4d0 msgr2=0x7f250806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.375+0000 7f252b200640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f250806b4d0 0x7f250806d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f251c0010e0 tx=0x7f251c007950 comp rx=0 tx=0).stop 2024-03-06T07:38:15.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.375+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 msgr2=0x7f252c15e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:15.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.375+0000 7f252b200640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 0x7f252c15e7c0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f252400cbc0 tx=0x7f252400a430 comp rx=0 tx=0).stop 2024-03-06T07:38:15.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 shutdown_connections 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f250806b4d0 0x7f250806d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f252c0cce90 0x7f252c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f252c145300 0x7f252c15e7c0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 --2- 172.21.15.6:0/232081182 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f252c070f10 0x7f252c141d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 >> 172.21.15.6:0/232081182 conn(0x7f252c06b3d0 msgr2=0x7f252c0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 shutdown_connections 2024-03-06T07:38:15.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:15.376+0000 7f252b200640 1 -- 172.21.15.6:0/232081182 wait complete. 2024-03-06T07:38:15.597 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:15.597 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:16.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:16.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.822+0000 7fc2cfe00640 1 Processor -- start 2024-03-06T07:38:16.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.822+0000 7fc2cfe00640 1 -- start start 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d006c8b0 0x7fc2d006ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cfe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fc2d0066d00 0x7fc2d015ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cfe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2d00ce370 0x7fc2d00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cfe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d015e470 con 0x7fc2d00ce370 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cfe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d015ea00 con 0x7fc2d0066d00 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cfe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00cf1b0 con 0x7fc2d006c8b0 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cf400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2d00ce370 0x7fc2d00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cf400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc2d00ce370 0x7fc2d00ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46154/0 (socket says 172.21.15.6:46154) 2024-03-06T07:38:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.824+0000 7fc2cf400640 1 -- 172.21.15.6:0/1950450229 learned_addr learned my addr 172.21.15.6:0/1950450229 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2c7e00640 1 --2- 172.21.15.6:0/1950450229 >> v2:172.21.15.178:3300/0 conn(0x7fc2d0066d00 0x7fc2d015ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cea00640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d006c8b0 0x7fc2d006ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cf400640 1 -- 172.21.15.6:0/1950450229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d006c8b0 msgr2=0x7fc2d006ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:16.827 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cf400640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d006c8b0 0x7fc2d006ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cf400640 1 -- 172.21.15.6:0/1950450229 >> v2:172.21.15.178:3300/0 conn(0x7fc2d0066d00 msgr2=0x7fc2d015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cf400640 1 --2- 172.21.15.6:0/1950450229 >> v2:172.21.15.178:3300/0 conn(0x7fc2d0066d00 0x7fc2d015ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cf400640 1 -- 172.21.15.6:0/1950450229 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d015f3d0 con 0x7fc2d00ce370 2024-03-06T07:38:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2cea00640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d006c8b0 0x7fc2d006ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:16.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.825+0000 7fc2c7e00640 1 --2- 172.21.15.6:0/1950450229 >> v2:172.21.15.178:3300/0 conn(0x7fc2d0066d00 0x7fc2d015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.826+0000 7fc2cf400640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00ce7c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006d2c0 tx=0x7fc2c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=110421d0087f893b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.826+0000 7fc2ce000640 1 -- 172.21.15.6:0/1950450229 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c802ca80 con 0x7fc2d00ce370 2024-03-06T07:38:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.826+0000 7fc2ce000640 1 -- 172.21.15.6:0/1950450229 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c802cc20 con 0x7fc2d00ce370 2024-03-06T07:38:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.826+0000 7fc2ce000640 1 -- 172.21.15.6:0/1950450229 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c80343f0 con 0x7fc2d00ce370 2024-03-06T07:38:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 -- 172.21.15.6:0/1950450229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 msgr2=0x7fc2d00ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00ce7c0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006d2c0 tx=0x7fc2c802bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 -- 172.21.15.6:0/1950450229 shutdown_connections 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00ce7c0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/1950450229 >> v2:172.21.15.178:3300/0 conn(0x7fc2d0066d00 0x7fc2d015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/1950450229 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d006c8b0 0x7fc2d006ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 -- 172.21.15.6:0/1950450229 >> 172.21.15.6:0/1950450229 conn(0x7fc2d00ccf70 msgr2=0x7fc2d00cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 -- 172.21.15.6:0/1950450229 shutdown_connections 2024-03-06T07:38:16.830 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.827+0000 7fc2cfe00640 1 -- 172.21.15.6:0/1950450229 wait complete. 2024-03-06T07:38:16.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.829+0000 7fc2cfe00640 1 Processor -- start 2024-03-06T07:38:16.831 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.829+0000 7fc2cfe00640 1 -- start start 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0066d00 0x7fc2d01e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2d006c8b0 0x7fc2d0153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cfe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00ced00 con 0x7fc2d00ce370 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cfe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d015e690 con 0x7fc2d006c8b0 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cfe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00cee80 con 0x7fc2d0066d00 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.830+0000 7fc2cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00b3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46160/0 (socket says 172.21.15.6:46160) 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 -- 172.21.15.6:0/2427616083 learned_addr learned my addr 172.21.15.6:0/2427616083 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cea00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0066d00 0x7fc2d01e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.833 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2c7e00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2d006c8b0 0x7fc2d0153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 -- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0066d00 msgr2=0x7fc2d01e8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.834 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0066d00 0x7fc2d01e8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 -- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2d006c8b0 msgr2=0x7fc2d0153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2d006c8b0 0x7fc2d0153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 -- 172.21.15.6:0/2427616083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d00af350 con 0x7fc2d00ce370 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.831+0000 7fc2cf400640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00b3160 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8036040 tx=0x7fc2c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.832+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8043020 con 0x7fc2d00ce370 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.832+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c803e450 con 0x7fc2d00ce370 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.832+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8009510 con 0x7fc2d00ce370 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.832+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d00af630 con 0x7fc2d00ce370 2024-03-06T07:38:16.835 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.832+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d00afbf0 con 0x7fc2d00ce370 2024-03-06T07:38:16.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.833+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d004f330 con 0x7fc2d00ce370 2024-03-06T07:38:16.836 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.834+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2c8005d10 con 0x7fc2d00ce370 2024-03-06T07:38:16.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.835+0000 7fc2c6a00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2ac06b4d0 0x7fc2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:16.837 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.836+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc2c80ace20 con 0x7fc2d00ce370 2024-03-06T07:38:16.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.836+0000 7fc2cea00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2ac06b4d0 0x7fc2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:16.839 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.837+0000 7fc2cea00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2ac06b4d0 0x7fc2ac06d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000a8d0 tx=0x7fc2c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:16.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:16.839+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc2c807bc50 con 0x7fc2d00ce370 2024-03-06T07:38:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.103+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d00cdde0 con 0x7fc2d00ce370 2024-03-06T07:38:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.104+0000 7fc2c6a00640 1 -- 172.21.15.6:0/2427616083 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fc2c807b550 con 0x7fc2d00ce370 2024-03-06T07:38:17.106 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:17.106 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:17.107 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2ac06b4d0 msgr2=0x7fc2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2ac06b4d0 0x7fc2ac06d990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000a8d0 tx=0x7fc2c0008040 comp rx=0 tx=0).stop 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 msgr2=0x7fc2d00b3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00b3160 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8036040 tx=0x7fc2c8004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 shutdown_connections 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc2ac06b4d0 0x7fc2ac06d990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc2d006c8b0 0x7fc2d0153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc2d00ce370 0x7fc2d00b3160 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.106+0000 7fc2cfe00640 1 --2- 172.21.15.6:0/2427616083 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc2d0066d00 0x7fc2d01e8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:17.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.107+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 >> 172.21.15.6:0/2427616083 conn(0x7fc2d00ccf70 msgr2=0x7fc2d006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:17.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.107+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 shutdown_connections 2024-03-06T07:38:17.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:17.107+0000 7fc2cfe00640 1 -- 172.21.15.6:0/2427616083 wait complete. 2024-03-06T07:38:17.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:17.328 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:18.189 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:18.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.563+0000 7f5f91800640 1 Processor -- start 2024-03-06T07:38:18.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.564+0000 7f5f91800640 1 -- start start 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.565+0000 7f5f91800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.565+0000 7f5f91800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f8c0d04b0 0x7f5f8c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.565+0000 7f5f91800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8c0cd0c0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.565+0000 7f5f91800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c0d29e0 con 0x7f5f8c0d04b0 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.565+0000 7f5f91800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0cd3a0 con 0x7f5f8c15f450 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f8c0d04b0 0x7f5f8c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5f8c0d04b0 0x7f5f8c0cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46162/0 (socket says 172.21.15.6:46162) 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 -- 172.21.15.6:0/165597152 learned_addr learned my addr 172.21.15.6:0/165597152 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:18.568 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f8be00640 1 --2- 172.21.15.6:0/165597152 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 -- 172.21.15.6:0/165597152 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 msgr2=0x7f5f8c070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 --2- 172.21.15.6:0/165597152 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 -- 172.21.15.6:0/165597152 >> v1:172.21.15.178:6789/0 conn(0x7f5f8c0700c0 legacy=0x7f5f8c0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:18.569 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.566+0000 7f5f90e00640 1 -- 172.21.15.6:0/165597152 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0cd680 con 0x7f5f8c0d04b0 2024-03-06T07:38:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.568+0000 7f5f90e00640 1 --2- 172.21.15.6:0/165597152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c0cde80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009e50 tx=0x7f5f8402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157badc1ffaad7c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.568+0000 7f5f8b400640 1 -- 172.21.15.6:0/165597152 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8402ca70 con 0x7f5f8c0d04b0 2024-03-06T07:38:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.568+0000 7f5f8b400640 1 -- 172.21.15.6:0/165597152 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8402cc10 con 0x7f5f8c0d04b0 2024-03-06T07:38:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f8b400640 1 -- 172.21.15.6:0/165597152 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f840343f0 con 0x7f5f8c0d04b0 2024-03-06T07:38:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f91800640 1 -- 172.21.15.6:0/165597152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0d04b0 msgr2=0x7f5f8c0cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f91800640 1 --2- 172.21.15.6:0/165597152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c0cde80 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009e50 tx=0x7f5f8402bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f91800640 1 -- 172.21.15.6:0/165597152 shutdown_connections 2024-03-06T07:38:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f91800640 1 --2- 172.21.15.6:0/165597152 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c0cde80 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f91800640 1 --2- 172.21.15.6:0/165597152 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.569+0000 7f5f91800640 1 -- 172.21.15.6:0/165597152 >> 172.21.15.6:0/165597152 conn(0x7f5f8c0d1ae0 msgr2=0x7f5f8c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.570+0000 7f5f91800640 1 -- 172.21.15.6:0/165597152 shutdown_connections 2024-03-06T07:38:18.573 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.570+0000 7f5f91800640 1 -- 172.21.15.6:0/165597152 wait complete. 2024-03-06T07:38:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.571+0000 7f5f91800640 1 Processor -- start 2024-03-06T07:38:18.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.571+0000 7f5f91800640 1 -- start start 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.572+0000 7f5f91800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 0x7f5f8c0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.572+0000 7f5f91800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c1f4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.572+0000 7f5f91800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.572+0000 7f5f91800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c15f830 con 0x7f5f8c0700c0 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.572+0000 7f5f91800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0ae8b0 con 0x7f5f8c0d04b0 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.572+0000 7f5f91800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c15f9b0 con 0x7f5f8c15f450 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 0x7f5f8c0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 0x7f5f8c0c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46174/0 (socket says 172.21.15.6:46174) 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 -- 172.21.15.6:0/3893528258 learned_addr learned my addr 172.21.15.6:0/3893528258 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f90e00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f83e00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c1f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 -- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 msgr2=0x7f5f8c06d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c06d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 -- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f8c0d04b0 msgr2=0x7f5f8c1f4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c1f4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f8be00640 1 -- 172.21.15.6:0/3893528258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0ca940 con 0x7f5f8c0700c0 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f90e00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.573+0000 7f5f83e00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.574+0000 7f5f8be00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 0x7f5f8c0c7ce0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00d5f0 tx=0x7f5f7c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.577 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.574+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0125f0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.574+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c019070 con 0x7f5f8c0700c0 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.574+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c0cac20 con 0x7f5f8c0700c0 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.574+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0129d0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.574+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c0cb1e0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.575+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c04f330 con 0x7f5f8c0700c0 2024-03-06T07:38:18.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.576+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f7c012b70 con 0x7f5f8c0700c0 2024-03-06T07:38:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.577+0000 7f5f88c00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5f6806b4d0 0x7f5f6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:18.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.577+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f7c08a690 con 0x7f5f8c0700c0 2024-03-06T07:38:18.580 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.578+0000 7f5f83e00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5f6806b4d0 0x7f5f6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:18.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.579+0000 7f5f83e00640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5f6806b4d0 0x7f5f6806d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c1f4cb0 tx=0x7f5f74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:18.584 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.582+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f7c0594d0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.822+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f8c15e3b0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.825 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.823+0000 7f5f88c00640 1 -- 172.21.15.6:0/3893528258 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f5f7c058dd0 con 0x7f5f8c0700c0 2024-03-06T07:38:18.825 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:18.825 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:18.825 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5f6806b4d0 msgr2=0x7f5f6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5f6806b4d0 0x7f5f6806d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c1f4cb0 tx=0x7f5f74009040 comp rx=0 tx=0).stop 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 msgr2=0x7f5f8c0c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 0x7f5f8c0c7ce0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00d5f0 tx=0x7f5f7c00dab0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 shutdown_connections 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f5f6806b4d0 0x7f5f6806d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5f8c0d04b0 0x7f5f8c1f4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.828 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5f8c15f450 0x7f5f8c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.825+0000 7f5f91800640 1 --2- 172.21.15.6:0/3893528258 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5f8c0700c0 0x7f5f8c0c7ce0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.826+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 >> 172.21.15.6:0/3893528258 conn(0x7f5f8c0d1ae0 msgr2=0x7f5f8c0b71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.826+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 shutdown_connections 2024-03-06T07:38:18.829 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:18.826+0000 7f5f91800640 1 -- 172.21.15.6:0/3893528258 wait complete. 2024-03-06T07:38:19.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:19.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:20.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:20.191 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:20.271 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.269+0000 7f044e400640 1 Processor -- start 2024-03-06T07:38:20.272 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.270+0000 7f044e400640 1 -- start start 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044e400640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f044815e0b0 0x7f04480cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f04480d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044e400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480b3710 con 0x7f04480d0c80 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044e400640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04480b09e0 con 0x7f044815e0b0 2024-03-06T07:38:20.273 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04480b39f0 con 0x7f04480d4a10 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044da00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f04480d0c80 0x7f04480d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:46888/0 (socket says 172.21.15.6:46888) 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.271+0000 7f044da00640 1 -- 172.21.15.6:0/2643372218 learned_addr learned my addr 172.21.15.6:0/2643372218 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447e00640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f04480d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f044d000640 1 --2- 172.21.15.6:0/2643372218 >> v2:172.21.15.178:3300/0 conn(0x7f044815e0b0 0x7f04480cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021663217 0 0) 0x7f04480b3710 con 0x7f04480d0c80 2024-03-06T07:38:20.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042c003520 con 0x7f04480d0c80 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690711132 0 0) 0x7f042c003520 con 0x7f04480d0c80 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 msgr2=0x7f04480d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f04480d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 >> v2:172.21.15.178:3300/0 conn(0x7f044815e0b0 msgr2=0x7f04480cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 --2- 172.21.15.6:0/2643372218 >> v2:172.21.15.178:3300/0 conn(0x7f044815e0b0 0x7f04480cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480b3cd0 con 0x7f04480d0c80 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0440001670 con 0x7f04480d0c80 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 >> v1:172.21.15.6:6789/0 conn(0x7f04480d0c80 legacy=0x7f04480d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c0039b0 0x7f042c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0043a0 0x7f042c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.276 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f042c004f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480b39f0 con 0x7f042c0039b0 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c005690 con 0x7f042c0043a0 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.272+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042c00ba40 con 0x7f04480d4a10 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044d000640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c0039b0 0x7f042c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f0447e00640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0043a0 0x7f042c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044da00640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f042c004f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044d000640 1 -- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 msgr2=0x7f042c004f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044d000640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f042c004f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044d000640 1 -- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0043a0 msgr2=0x7f042c004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044d000640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0043a0 0x7f042c004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044d000640 1 -- 172.21.15.6:0/2643372218 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480b3cd0 con 0x7f042c0039b0 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f044da00640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480d4a10 0x7f042c004f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:20.277 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.273+0000 7f0447e00640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0043a0 0x7f042c004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:20.278 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.274+0000 7f044d000640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c0039b0 0x7f042c003dd0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f0438009ac0 tx=0x7f042c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9663021e22e1b31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.275+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438039070 con 0x7f042c0039b0 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.275+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043802c710 con 0x7f042c0039b0 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.275+0000 7f0447400640 1 -- 172.21.15.6:0/2643372218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438009690 con 0x7f042c0039b0 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.275+0000 7f044e400640 1 -- 172.21.15.6:0/2643372218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c0039b0 msgr2=0x7f042c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.275+0000 7f044e400640 1 --2- 172.21.15.6:0/2643372218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f042c0039b0 0x7f042c003dd0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f0438009ac0 tx=0x7f042c005830 comp rx=0 tx=0).stop 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.275+0000 7f044da00640 1 -- 172.21.15.6:0/2643372218 reap_dead start 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.276+0000 7f044e400640 1 -- 172.21.15.6:0/2643372218 shutdown_connections 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.276+0000 7f044e400640 1 -- 172.21.15.6:0/2643372218 >> 172.21.15.6:0/2643372218 conn(0x7f04480d3d30 msgr2=0x7f04480d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.276+0000 7f044e400640 1 -- 172.21.15.6:0/2643372218 shutdown_connections 2024-03-06T07:38:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.276+0000 7f044e400640 1 -- 172.21.15.6:0/2643372218 wait complete. 2024-03-06T07:38:20.280 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.279+0000 7f044e400640 1 Processor -- start 2024-03-06T07:38:20.281 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.279+0000 7f044e400640 1 -- start start 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044e400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0039b0 0x7f044815c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044e400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480cae00 0x7f04480cb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044e400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 0x7f04480b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044e400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480cfc60 con 0x7f04480b8960 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044e400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044815e490 con 0x7f042c0039b0 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044e400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04480b54e0 con 0x7f04480cae00 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 0x7f04480b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 0x7f04480b8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46182/0 (socket says 172.21.15.6:46182) 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044da00640 1 -- 172.21.15.6:0/325625250 learned_addr learned my addr 172.21.15.6:0/325625250 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f044d000640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0039b0 0x7f044815c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.281+0000 7f0447e00640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480cae00 0x7f04480cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.284 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044da00640 1 -- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480cae00 msgr2=0x7f04480cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044da00640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480cae00 0x7f04480cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044da00640 1 -- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0039b0 msgr2=0x7f044815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044da00640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0039b0 0x7f044815c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044da00640 1 -- 172.21.15.6:0/325625250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480cb970 con 0x7f04480b8960 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f0447e00640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480cae00 0x7f04480cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044d000640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0039b0 0x7f044815c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:20.285 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044da00640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 0x7f04480b8de0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f0440000c00 tx=0x7f044000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0440007230 con 0x7f04480b8960 2024-03-06T07:38:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04400073f0 con 0x7f04480b8960 2024-03-06T07:38:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04400047e0 con 0x7f04480b8960 2024-03-06T07:38:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04480cbc50 con 0x7f04480b8960 2024-03-06T07:38:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.282+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480cc210 con 0x7f04480b8960 2024-03-06T07:38:20.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.284+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044804f330 con 0x7f04480b8960 2024-03-06T07:38:20.287 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.285+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f044001b020 con 0x7f04480b8960 2024-03-06T07:38:20.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.286+0000 7f0444c00640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f042406b4d0 0x7f042406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:20.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.286+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0440089d60 con 0x7f04480b8960 2024-03-06T07:38:20.288 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.286+0000 7f044d000640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f042406b4d0 0x7f042406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.287+0000 7f044d000640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f042406b4d0 0x7f042406d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0438036000 tx=0x7f0438030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.291+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0440058c60 con 0x7f04480b8960 2024-03-06T07:38:20.537 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.535+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04480b2c00 con 0x7f04480b8960 2024-03-06T07:38:20.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.536+0000 7f0444c00640 1 -- 172.21.15.6:0/325625250 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f0440058560 con 0x7f04480b8960 2024-03-06T07:38:20.538 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:20.538 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:20.538 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.538+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f042406b4d0 msgr2=0x7f042406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.538+0000 7f044e400640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f042406b4d0 0x7f042406d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0438036000 tx=0x7f0438030000 comp rx=0 tx=0).stop 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.538+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 msgr2=0x7f04480b8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.538+0000 7f044e400640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 0x7f04480b8de0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f0440000c00 tx=0x7f044000dc00 comp rx=0 tx=0).stop 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 shutdown_connections 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f042406b4d0 0x7f042406d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f04480b8960 0x7f04480b8de0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f04480cae00 0x7f04480cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 --2- 172.21.15.6:0/325625250 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f042c0039b0 0x7f044815c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 >> 172.21.15.6:0/325625250 conn(0x7f04480d3d30 msgr2=0x7f04480d20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:20.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 shutdown_connections 2024-03-06T07:38:20.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:20.539+0000 7f044e400640 1 -- 172.21.15.6:0/325625250 wait complete. 2024-03-06T07:38:20.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:20.760 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:21.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:21.989 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.987+0000 7fe262800640 1 Processor -- start 2024-03-06T07:38:21.990 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.987+0000 7fe262800640 1 -- start start 2024-03-06T07:38:21.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe262800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe25c15f450 0x7fe25c070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.991 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe262800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0700c0 0x7fe25c0cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe262800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fe25c0d02e0 0x7fe25c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe262800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25c0d29e0 con 0x7fe25c15f450 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe262800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25c0cd0c0 con 0x7fe25c0d02e0 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe262800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0cd3a0 con 0x7fe25c0700c0 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe261400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe25c15f450 0x7fe25c070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe261400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fe25c15f450 0x7fe25c070990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45682/0 (socket says 172.21.15.6:45682) 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.989+0000 7fe261400640 1 -- 172.21.15.6:0/1117913723 learned_addr learned my addr 172.21.15.6:0/1117913723 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe260a00640 1 --2- 172.21.15.6:0/1117913723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0700c0 0x7fe25c0cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.992 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe261e00640 1 --2- 172.21.15.6:0/1117913723 >> v2:172.21.15.178:3300/0 conn(0x7fe25c0d02e0 0x7fe25c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe261400640 1 -- 172.21.15.6:0/1117913723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0700c0 msgr2=0x7fe25c0cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe261400640 1 --2- 172.21.15.6:0/1117913723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0700c0 0x7fe25c0cfda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe261400640 1 -- 172.21.15.6:0/1117913723 >> v2:172.21.15.178:3300/0 conn(0x7fe25c0d02e0 msgr2=0x7fe25c06ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:21.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe261400640 1 --2- 172.21.15.6:0/1117913723 >> v2:172.21.15.178:3300/0 conn(0x7fe25c0d02e0 0x7fe25c06ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.993 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.990+0000 7fe261400640 1 -- 172.21.15.6:0/1117913723 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c0cd680 con 0x7fe25c15f450 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.991+0000 7fe261400640 1 --2- 172.21.15.6:0/1117913723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c15f450 0x7fe25c070990 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fe25c06b9c0 tx=0x7fe25002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631493d02baf2f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.991+0000 7fe25be00640 1 -- 172.21.15.6:0/1117913723 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250035020 con 0x7fe25c15f450 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.991+0000 7fe25be00640 1 -- 172.21.15.6:0/1117913723 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250031870 con 0x7fe25c15f450 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.991+0000 7fe25be00640 1 -- 172.21.15.6:0/1117913723 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250031c40 con 0x7fe25c15f450 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 -- 172.21.15.6:0/1117913723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c15f450 msgr2=0x7fe25c070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 --2- 172.21.15.6:0/1117913723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c15f450 0x7fe25c070990 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fe25c06b9c0 tx=0x7fe25002b910 comp rx=0 tx=0).stop 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 -- 172.21.15.6:0/1117913723 shutdown_connections 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 --2- 172.21.15.6:0/1117913723 >> v2:172.21.15.178:3300/0 conn(0x7fe25c0d02e0 0x7fe25c06ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.994 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 --2- 172.21.15.6:0/1117913723 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0700c0 0x7fe25c0cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 --2- 172.21.15.6:0/1117913723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c15f450 0x7fe25c070990 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 -- 172.21.15.6:0/1117913723 >> 172.21.15.6:0/1117913723 conn(0x7fe25c0d1ae0 msgr2=0x7fe25c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:21.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 -- 172.21.15.6:0/1117913723 shutdown_connections 2024-03-06T07:38:21.995 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.992+0000 7fe262800640 1 -- 172.21.15.6:0/1117913723 wait complete. 2024-03-06T07:38:21.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.994+0000 7fe262800640 1 Processor -- start 2024-03-06T07:38:21.996 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.994+0000 7fe262800640 1 -- start start 2024-03-06T07:38:21.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe262800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 0x7fe25c14c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.997 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe262800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0d02e0 0x7fe25c0c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe262800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe25c15f450 0x7fe25c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe262800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c15f830 con 0x7fe25c0700c0 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe262800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c0ae8b0 con 0x7fe25c15f450 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe262800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c15f9b0 con 0x7fe25c0d02e0 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 0x7fe25c14c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 0x7fe25c14c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45684/0 (socket says 172.21.15.6:45684) 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 -- 172.21.15.6:0/2738448854 learned_addr learned my addr 172.21.15.6:0/2738448854 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 -- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0d02e0 msgr2=0x7fe25c0c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0d02e0 0x7fe25c0c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 -- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe25c15f450 msgr2=0x7fe25c0c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe25c15f450 0x7fe25c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:21.998 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.995+0000 7fe261400640 1 -- 172.21.15.6:0/2738448854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c14cd40 con 0x7fe25c0700c0 2024-03-06T07:38:21.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.996+0000 7fe261400640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 0x7fe25c14c3e0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe250000c00 tx=0x7fe2500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:21.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.996+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25003ad60 con 0x7fe25c0700c0 2024-03-06T07:38:21.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.996+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2500443f0 con 0x7fe25c0700c0 2024-03-06T07:38:21.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.996+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500447e0 con 0x7fe25c0700c0 2024-03-06T07:38:21.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.996+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c14d020 con 0x7fe25c0700c0 2024-03-06T07:38:21.999 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.996+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c0c6410 con 0x7fe25c0700c0 2024-03-06T07:38:22.000 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.997+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c04f330 con 0x7fe25c0700c0 2024-03-06T07:38:22.001 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:21.999+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe25004b060 con 0x7fe25c0700c0 2024-03-06T07:38:22.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.000+0000 7fe259600640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe23c06b4d0 0x7fe23c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:22.002 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.000+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe2500ac420 con 0x7fe25c0700c0 2024-03-06T07:38:22.003 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.001+0000 7fe260a00640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe23c06b4d0 0x7fe23c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:22.004 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.002+0000 7fe260a00640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe23c06b4d0 0x7fe23c06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0c8100 tx=0x7fe248008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:22.006 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.004+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe25003f070 con 0x7fe25c0700c0 2024-03-06T07:38:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:22.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.250+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe25c0c65e0 con 0x7fe25c0700c0 2024-03-06T07:38:22.253 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.251+0000 7fe259600640 1 -- 172.21.15.6:0/2738448854 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fe25002f320 con 0x7fe25c0700c0 2024-03-06T07:38:22.253 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:22.253 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:22.254 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:22.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe23c06b4d0 msgr2=0x7fe23c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:22.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe23c06b4d0 0x7fe23c06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0c8100 tx=0x7fe248008040 comp rx=0 tx=0).stop 2024-03-06T07:38:22.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 msgr2=0x7fe25c14c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:22.256 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 0x7fe25c14c3e0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fe250000c00 tx=0x7fe2500046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 shutdown_connections 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fe23c06b4d0 0x7fe23c06d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe25c15f450 0x7fe25c0c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fe25c0d02e0 0x7fe25c0c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 --2- 172.21.15.6:0/2738448854 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe25c0700c0 0x7fe25c14c3e0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 >> 172.21.15.6:0/2738448854 conn(0x7fe25c0d1ae0 msgr2=0x7fe25c15cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 shutdown_connections 2024-03-06T07:38:22.257 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:22.254+0000 7fe262800640 1 -- 172.21.15.6:0/2738448854 wait complete. 2024-03-06T07:38:22.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:22.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:23.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:23.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.701+0000 7efcc3e00640 1 Processor -- start 2024-03-06T07:38:23.704 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.702+0000 7efcc3e00640 1 -- start start 2024-03-06T07:38:23.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc3e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efcc415e0b0 0x7efcc40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc3e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7efcc40d0e50 0x7efcc40b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc3e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc40b3710 con 0x7efcc40d4a10 2024-03-06T07:38:23.705 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc3e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40b09e0 con 0x7efcc415e0b0 2024-03-06T07:38:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc3e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc40b39f0 con 0x7efcc40d0e50 2024-03-06T07:38:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efcc415e0b0 0x7efcc40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efcc415e0b0 0x7efcc40cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45694/0 (socket says 172.21.15.6:45694) 2024-03-06T07:38:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.703+0000 7efcc2a00640 1 -- 172.21.15.6:0/1329419170 learned_addr learned my addr 172.21.15.6:0/1329419170 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.704+0000 7efcc3400640 1 --2- 172.21.15.6:0/1329419170 >> v2:172.21.15.178:3300/0 conn(0x7efcc40d0e50 0x7efcc40b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.704+0000 7efcc2a00640 1 -- 172.21.15.6:0/1329419170 >> v2:172.21.15.178:3300/0 conn(0x7efcc40d0e50 msgr2=0x7efcc40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.704+0000 7efcc2a00640 1 --2- 172.21.15.6:0/1329419170 >> v2:172.21.15.178:3300/0 conn(0x7efcc40d0e50 0x7efcc40b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.704+0000 7efcc2a00640 1 -- 172.21.15.6:0/1329419170 >> v1:172.21.15.178:6789/0 conn(0x7efcc40d4a10 legacy=0x7efcc40d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.704+0000 7efcc2a00640 1 -- 172.21.15.6:0/1329419170 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc40b3cd0 con 0x7efcc415e0b0 2024-03-06T07:38:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.705+0000 7efcc2a00640 1 --2- 172.21.15.6:0/1329419170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc40cf310 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efcc408d6d0 tx=0x7efcb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c876b622b187ab96 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.705+0000 7efcc1600640 1 -- 172.21.15.6:0/1329419170 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4035020 con 0x7efcc415e0b0 2024-03-06T07:38:23.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.705+0000 7efcc1600640 1 -- 172.21.15.6:0/1329419170 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb4031870 con 0x7efcc415e0b0 2024-03-06T07:38:23.708 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.705+0000 7efcc1600640 1 -- 172.21.15.6:0/1329419170 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4031c40 con 0x7efcc415e0b0 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 -- 172.21.15.6:0/1329419170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 msgr2=0x7efcc40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 --2- 172.21.15.6:0/1329419170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc40cf310 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efcc408d6d0 tx=0x7efcb402b910 comp rx=0 tx=0).stop 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 -- 172.21.15.6:0/1329419170 shutdown_connections 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 --2- 172.21.15.6:0/1329419170 >> v2:172.21.15.178:3300/0 conn(0x7efcc40d0e50 0x7efcc40b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 --2- 172.21.15.6:0/1329419170 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc40cf310 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 -- 172.21.15.6:0/1329419170 >> 172.21.15.6:0/1329419170 conn(0x7efcc40d3d30 msgr2=0x7efcc40d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 -- 172.21.15.6:0/1329419170 shutdown_connections 2024-03-06T07:38:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.706+0000 7efcc3e00640 1 -- 172.21.15.6:0/1329419170 wait complete. 2024-03-06T07:38:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.708+0000 7efcc3e00640 1 Processor -- start 2024-03-06T07:38:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.708+0000 7efcc3e00640 1 -- start start 2024-03-06T07:38:23.712 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc40d0e50 0x7efcc415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efcc40d4a10 0x7efcc40b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc41f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40cfc60 con 0x7efcc415e0b0 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc415e490 con 0x7efcc40d0e50 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc40cf850 con 0x7efcc40d4a10 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc41f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc41f3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45700/0 (socket says 172.21.15.6:45700) 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.709+0000 7efcc3400640 1 -- 172.21.15.6:0/668980616 learned_addr learned my addr 172.21.15.6:0/668980616 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc2000640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efcc40d4a10 0x7efcc40b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc2a00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc40d0e50 0x7efcc415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3400640 1 -- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efcc40d4a10 msgr2=0x7efcc40b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3400640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efcc40d4a10 0x7efcc40b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3400640 1 -- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc40d0e50 msgr2=0x7efcc415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3400640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc40d0e50 0x7efcc415ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.713 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3400640 1 -- 172.21.15.6:0/668980616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc4160520 con 0x7efcc415e0b0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc2a00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc40d0e50 0x7efcc415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3400640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc41f3940 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7efcbc007e30 tx=0x7efcbc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc014620 con 0x7efcc415e0b0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc019070 con 0x7efcc415e0b0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc014a00 con 0x7efcc415e0b0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.710+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc41f4390 con 0x7efcc415e0b0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.711+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc415f7e0 con 0x7efcc415e0b0 2024-03-06T07:38:23.714 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.712+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc404f330 con 0x7efcc415e0b0 2024-03-06T07:38:23.715 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.713+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcbc014ba0 con 0x7efcc415e0b0 2024-03-06T07:38:23.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.714+0000 7efcbaa00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efca406b500 0x7efca406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:23.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.714+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcbc08aa90 con 0x7efcc415e0b0 2024-03-06T07:38:23.716 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.715+0000 7efcc2a00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efca406b500 0x7efca406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.716+0000 7efcc2a00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efca406b500 0x7efca406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7efcb4036000 tx=0x7efcb403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.718+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcbc059960 con 0x7efcc415e0b0 2024-03-06T07:38:23.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.962+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcc40b44d0 con 0x7efcc415e0b0 2024-03-06T07:38:23.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.963+0000 7efcbaa00640 1 -- 172.21.15.6:0/668980616 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7efcbc059260 con 0x7efcc415e0b0 2024-03-06T07:38:23.965 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:23.965 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:23.965 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efca406b500 msgr2=0x7efca406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efca406b500 0x7efca406d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7efcb4036000 tx=0x7efcb403c040 comp rx=0 tx=0).stop 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 msgr2=0x7efcc41f3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc41f3940 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7efcbc007e30 tx=0x7efcbc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 shutdown_connections 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7efca406b500 0x7efca406d9c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efcc415e0b0 0x7efcc41f3940 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7efcc40d4a10 0x7efcc40b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 --2- 172.21.15.6:0/668980616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcc40d0e50 0x7efcc415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.965+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 >> 172.21.15.6:0/668980616 conn(0x7efcc40d3d30 msgr2=0x7efcc40b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.966+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 shutdown_connections 2024-03-06T07:38:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:23.966+0000 7efcc3e00640 1 -- 172.21.15.6:0/668980616 wait complete. 2024-03-06T07:38:24.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:24.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:24.195 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:24.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:24.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:25.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:25.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.424+0000 7f90a9200640 1 Processor -- start 2024-03-06T07:38:25.426 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.424+0000 7f90a9200640 1 -- start start 2024-03-06T07:38:25.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a9200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f90a4145300 0x7f90a40b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a9200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90a4070f10 0x7f90a40cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.429 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a9200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a40714d0 con 0x7f90a40cce90 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a9200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a40cd300 con 0x7f90a4070f10 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a9200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a40717b0 con 0x7f90a4145300 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90a4070f10 0x7f90a40cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a2a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f90a4070f10 0x7f90a40cc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45708/0 (socket says 172.21.15.6:45708) 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a2a00640 1 -- 172.21.15.6:0/483870571 learned_addr learned my addr 172.21.15.6:0/483870571 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.426+0000 7f90a3400640 1 --2- 172.21.15.6:0/483870571 >> v2:172.21.15.178:3300/0 conn(0x7f90a4145300 0x7f90a40b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.427+0000 7f90a2a00640 1 -- 172.21.15.6:0/483870571 >> v2:172.21.15.178:3300/0 conn(0x7f90a4145300 msgr2=0x7f90a40b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.427+0000 7f90a2a00640 1 --2- 172.21.15.6:0/483870571 >> v2:172.21.15.178:3300/0 conn(0x7f90a4145300 0x7f90a40b49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.427+0000 7f90a2a00640 1 -- 172.21.15.6:0/483870571 >> v1:172.21.15.178:6789/0 conn(0x7f90a40cce90 legacy=0x7f90a40cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.427+0000 7f90a2a00640 1 -- 172.21.15.6:0/483870571 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a4071a90 con 0x7f90a4070f10 2024-03-06T07:38:25.430 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.427+0000 7f90a3400640 1 --2- 172.21.15.6:0/483870571 >> v2:172.21.15.178:3300/0 conn(0x7f90a4145300 0x7f90a40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.428+0000 7f90a2a00640 1 --2- 172.21.15.6:0/483870571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a40cc950 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f90a40cf990 tx=0x7f909002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fb3eabbfb3e509 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a2000640 1 -- 172.21.15.6:0/483870571 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090035020 con 0x7f90a4070f10 2024-03-06T07:38:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a2000640 1 -- 172.21.15.6:0/483870571 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090031870 con 0x7f90a4070f10 2024-03-06T07:38:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a2000640 1 -- 172.21.15.6:0/483870571 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090031c40 con 0x7f90a4070f10 2024-03-06T07:38:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a9200640 1 -- 172.21.15.6:0/483870571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 msgr2=0x7f90a40cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.431 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a9200640 1 --2- 172.21.15.6:0/483870571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a40cc950 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f90a40cf990 tx=0x7f909002b910 comp rx=0 tx=0).stop 2024-03-06T07:38:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a9200640 1 -- 172.21.15.6:0/483870571 shutdown_connections 2024-03-06T07:38:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a9200640 1 --2- 172.21.15.6:0/483870571 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a40cc950 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a9200640 1 --2- 172.21.15.6:0/483870571 >> v2:172.21.15.178:3300/0 conn(0x7f90a4145300 0x7f90a40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.432 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.429+0000 7f90a9200640 1 -- 172.21.15.6:0/483870571 >> 172.21.15.6:0/483870571 conn(0x7f90a406b3d0 msgr2=0x7f90a406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.430+0000 7f90a9200640 1 -- 172.21.15.6:0/483870571 shutdown_connections 2024-03-06T07:38:25.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.430+0000 7f90a9200640 1 -- 172.21.15.6:0/483870571 wait complete. 2024-03-06T07:38:25.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.431+0000 7f90a9200640 1 Processor -- start 2024-03-06T07:38:25.433 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 -- start start 2024-03-06T07:38:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a41e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.435 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90a40cce90 0x7f90a41e7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90a4145300 0x7f90a41301b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a41456e0 con 0x7f90a4070f10 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a40b4ee0 con 0x7f90a4145300 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.432+0000 7f90a9200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a415d580 con 0x7f90a40cce90 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a41e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a41e6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45722/0 (socket says 172.21.15.6:45722) 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 -- 172.21.15.6:0/967967087 learned_addr learned my addr 172.21.15.6:0/967967087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 -- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90a40cce90 msgr2=0x7f90a41e7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90a40cce90 0x7f90a41e7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 -- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90a4145300 msgr2=0x7f90a41301b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90a4145300 0x7f90a41301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.433+0000 7f90a3400640 1 -- 172.21.15.6:0/967967087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a41e4500 con 0x7f90a4070f10 2024-03-06T07:38:25.436 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.434+0000 7f90a3400640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a41e6ec0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f909400d7d0 tx=0x7f909400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.434+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094004030 con 0x7f90a4070f10 2024-03-06T07:38:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.434+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90940041d0 con 0x7f90a4070f10 2024-03-06T07:38:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.434+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909400b6c0 con 0x7f90a4070f10 2024-03-06T07:38:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.434+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a41434b0 con 0x7f90a4070f10 2024-03-06T07:38:25.437 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.434+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a412e650 con 0x7f90a4070f10 2024-03-06T07:38:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.435+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a404f330 con 0x7f90a4070f10 2024-03-06T07:38:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.436+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f909400b860 con 0x7f90a4070f10 2024-03-06T07:38:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.438+0000 7f909b400640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f908406b4d0 0x7f908406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.438+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f909408a510 con 0x7f90a4070f10 2024-03-06T07:38:25.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.438+0000 7f90a2a00640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f908406b4d0 0x7f908406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:25.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.439+0000 7f90a2a00640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f908406b4d0 0x7f908406d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9090036000 tx=0x7f9090030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:25.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.443+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90a41e4500 con 0x7f90a4070f10 2024-03-06T07:38:25.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.722+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a40b2950 con 0x7f90a4070f10 2024-03-06T07:38:25.725 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.723+0000 7f909b400640 1 -- 172.21.15.6:0/967967087 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f9094059340 con 0x7f90a4070f10 2024-03-06T07:38:25.726 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:25.726 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:25.726 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f908406b4d0 msgr2=0x7f908406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f908406b4d0 0x7f908406d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f9090036000 tx=0x7f9090030000 comp rx=0 tx=0).stop 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 msgr2=0x7f90a41e6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a41e6ec0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f909400d7d0 tx=0x7f909400dc90 comp rx=0 tx=0).stop 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 shutdown_connections 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f908406b4d0 0x7f908406d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f90a4145300 0x7f90a41301b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f90a40cce90 0x7f90a41e7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 --2- 172.21.15.6:0/967967087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f90a4070f10 0x7f90a41e6ec0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:25.728 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.725+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 >> 172.21.15.6:0/967967087 conn(0x7f90a406b3d0 msgr2=0x7f90a415ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:25.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.726+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 shutdown_connections 2024-03-06T07:38:25.729 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:25.726+0000 7f90a9200640 1 -- 172.21.15.6:0/967967087 wait complete. 2024-03-06T07:38:25.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:25.948 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:26.197 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:26.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:27.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.179+0000 7f148f000640 1 Processor -- start 2024-03-06T07:38:27.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.179+0000 7f148f000640 1 -- start start 2024-03-06T07:38:27.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f148806afb0 0x7f148806b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14880cf130 0x7f14880cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148f000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880d0910 con 0x7f1488144c00 2024-03-06T07:38:27.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f148806b950 con 0x7f14880cf130 2024-03-06T07:38:27.183 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14880d0bf0 con 0x7f148806afb0 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14880cf130 0x7f14880cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148e600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f14880cf130 0x7f14880cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45724/0 (socket says 172.21.15.6:45724) 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148e600640 1 -- 172.21.15.6:0/2163745122 learned_addr learned my addr 172.21.15.6:0/2163745122 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.181+0000 7f148d200640 1 --2- 172.21.15.6:0/2163745122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f148806afb0 0x7f148806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.182+0000 7f148e600640 1 -- 172.21.15.6:0/2163745122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f148806afb0 msgr2=0x7f148806b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.182+0000 7f148e600640 1 --2- 172.21.15.6:0/2163745122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f148806afb0 0x7f148806b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.184 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.182+0000 7f148e600640 1 -- 172.21.15.6:0/2163745122 >> v1:172.21.15.178:6789/0 conn(0x7f1488144c00 legacy=0x7f1488145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:27.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.182+0000 7f148e600640 1 -- 172.21.15.6:0/2163745122 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14880d0ed0 con 0x7f14880cf130 2024-03-06T07:38:27.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.183+0000 7f148e600640 1 --2- 172.21.15.6:0/2163745122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880cf130 0x7f14880cf5b0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f1480009e50 tx=0x7f148002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc353cac25729d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:27.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.183+0000 7f1487e00640 1 -- 172.21.15.6:0/2163745122 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148002ca80 con 0x7f14880cf130 2024-03-06T07:38:27.185 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.183+0000 7f1487e00640 1 -- 172.21.15.6:0/2163745122 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148002cc20 con 0x7f14880cf130 2024-03-06T07:38:27.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.183+0000 7f1487e00640 1 -- 172.21.15.6:0/2163745122 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14800343f0 con 0x7f14880cf130 2024-03-06T07:38:27.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 -- 172.21.15.6:0/2163745122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880cf130 msgr2=0x7f14880cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 --2- 172.21.15.6:0/2163745122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880cf130 0x7f14880cf5b0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f1480009e50 tx=0x7f148002bee0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.186 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 -- 172.21.15.6:0/2163745122 shutdown_connections 2024-03-06T07:38:27.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 --2- 172.21.15.6:0/2163745122 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f14880cf130 0x7f14880cf5b0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 --2- 172.21.15.6:0/2163745122 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f148806afb0 0x7f148806b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 -- 172.21.15.6:0/2163745122 >> 172.21.15.6:0/2163745122 conn(0x7f148806c0c0 msgr2=0x7f148806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:27.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 -- 172.21.15.6:0/2163745122 shutdown_connections 2024-03-06T07:38:27.187 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.184+0000 7f148f000640 1 -- 172.21.15.6:0/2163745122 wait complete. 2024-03-06T07:38:27.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.186+0000 7f148f000640 1 Processor -- start 2024-03-06T07:38:27.188 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.187+0000 7f148f000640 1 -- start start 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148f000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f148806afb0 0x7f148815d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14880cf130 0x7f14880b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 0x7f14880b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488144fe0 con 0x7f1488144c00 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148f000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880d1490 con 0x7f148806afb0 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1488145990 con 0x7f14880cf130 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 0x7f14880b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148e600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 0x7f14880b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45728/0 (socket says 172.21.15.6:45728) 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148e600640 1 -- 172.21.15.6:0/3222703491 learned_addr learned my addr 172.21.15.6:0/3222703491 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.188+0000 7f148d200640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14880cf130 0x7f14880b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148dc00640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f148806afb0 0x7f148815d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148e600640 1 -- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14880cf130 msgr2=0x7f14880b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148e600640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14880cf130 0x7f14880b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148e600640 1 -- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f148806afb0 msgr2=0x7f148815d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148e600640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f148806afb0 0x7f148815d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148e600640 1 -- 172.21.15.6:0/3222703491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148813b140 con 0x7f1488144c00 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148d200640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14880cf130 0x7f14880b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148dc00640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f148806afb0 0x7f148815d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:27.192 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148e600640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 0x7f14880b1710 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f1480036040 tx=0x7f1480004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480043020 con 0x7f1488144c00 2024-03-06T07:38:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148003e450 con 0x7f1488144c00 2024-03-06T07:38:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480009510 con 0x7f1488144c00 2024-03-06T07:38:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.189+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1488138130 con 0x7f1488144c00 2024-03-06T07:38:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.190+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148813ae60 con 0x7f1488144c00 2024-03-06T07:38:27.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.191+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148806ca00 con 0x7f1488144c00 2024-03-06T07:38:27.194 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.192+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1480046020 con 0x7f1488144c00 2024-03-06T07:38:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.193+0000 7f1485600640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f146806b4d0 0x7f146806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.193+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f14800acb60 con 0x7f1488144c00 2024-03-06T07:38:27.195 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.193+0000 7f148dc00640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f146806b4d0 0x7f146806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:27.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.194+0000 7f148dc00640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f146806b4d0 0x7f146806d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1488131120 tx=0x7f147c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:27.205 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.203+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f148806ca00 con 0x7f1488144c00 2024-03-06T07:38:27.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.444+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f148804f330 con 0x7f1488144c00 2024-03-06T07:38:27.447 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.445+0000 7f1485600640 1 -- 172.21.15.6:0/3222703491 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f148004d030 con 0x7f1488144c00 2024-03-06T07:38:27.447 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:27.448 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:27.448 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:27.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.447+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f146806b4d0 msgr2=0x7f146806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.447+0000 7f148f000640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f146806b4d0 0x7f146806d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1488131120 tx=0x7f147c008040 comp rx=0 tx=0).stop 2024-03-06T07:38:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.447+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 msgr2=0x7f14880b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.447+0000 7f148f000640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 0x7f14880b1710 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f1480036040 tx=0x7f1480004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:27.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 shutdown_connections 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f146806b4d0 0x7f146806d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1488144c00 0x7f14880b1710 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f14880cf130 0x7f14880b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 --2- 172.21.15.6:0/3222703491 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f148806afb0 0x7f148815d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 >> 172.21.15.6:0/3222703491 conn(0x7f148806c0c0 msgr2=0x7f1488143610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 shutdown_connections 2024-03-06T07:38:27.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:27.448+0000 7f148f000640 1 -- 172.21.15.6:0/3222703491 wait complete. 2024-03-06T07:38:27.673 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:27.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:28.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.904+0000 7f9d69e00640 1 Processor -- start 2024-03-06T07:38:28.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.904+0000 7f9d69e00640 1 -- start start 2024-03-06T07:38:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.905+0000 7f9d69e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9d6415e0b0 0x7f9d640cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.905+0000 7f9d69e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 0x7f9d640d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d69e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d640b3710 con 0x7f9d640d0c80 2024-03-06T07:38:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d69e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d640b09e0 con 0x7f9d6415e0b0 2024-03-06T07:38:28.908 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d69e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640b39f0 con 0x7f9d640d4a10 2024-03-06T07:38:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d69400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9d640d0c80 0x7f9d640d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37372/0 (socket says 172.21.15.6:37372) 2024-03-06T07:38:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d69400640 1 -- 172.21.15.6:0/2809130940 learned_addr learned my addr 172.21.15.6:0/2809130940 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d63e00640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 0x7f9d640d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d68a00640 1 --2- 172.21.15.6:0/2809130940 >> v2:172.21.15.178:3300/0 conn(0x7f9d6415e0b0 0x7f9d640cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.906+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992134103 0 0) 0x7f9d640b3710 con 0x7f9d640d0c80 2024-03-06T07:38:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d4c003520 con 0x7f9d640d0c80 2024-03-06T07:38:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491876062 0 0) 0x7f9d4c003520 con 0x7f9d640d0c80 2024-03-06T07:38:28.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 msgr2=0x7f9d640d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 0x7f9d640d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 >> v2:172.21.15.178:3300/0 conn(0x7f9d6415e0b0 msgr2=0x7f9d640cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 --2- 172.21.15.6:0/2809130940 >> v2:172.21.15.178:3300/0 conn(0x7f9d6415e0b0 0x7f9d640cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640b3cd0 con 0x7f9d640d0c80 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9d5c0019f0 con 0x7f9d640d0c80 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 >> v1:172.21.15.6:6789/0 conn(0x7f9d640d0c80 legacy=0x7f9d640d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 0x7f9d4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640b39f0 con 0x7f9d4c004d40 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c00b9c0 con 0x7f9d4c0039b0 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.907+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c00bb40 con 0x7f9d640d4a10 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d68a00640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d63e00640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 0x7f9d4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d69400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d69400640 1 -- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 msgr2=0x7f9d4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d69400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640d4a10 0x7f9d4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d69400640 1 -- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d4c0039b0 msgr2=0x7f9d4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d69400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d4c0039b0 0x7f9d4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.908+0000 7f9d69400640 1 -- 172.21.15.6:0/2809130940 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c003520 con 0x7f9d4c004d40 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.910+0000 7f9d69400640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d4c0051c0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00a7d0 tx=0x7f9d4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a383b6f9ac6659 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.910+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c00a3f0 con 0x7f9d4c004d40 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.910+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c034630 con 0x7f9d4c004d40 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.910+0000 7f9d63400640 1 -- 172.21.15.6:0/2809130940 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c02ce60 con 0x7f9d4c004d40 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69e00640 1 -- 172.21.15.6:0/2809130940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 msgr2=0x7f9d4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69e00640 1 --2- 172.21.15.6:0/2809130940 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d4c0051c0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00a7d0 tx=0x7f9d4c005830 comp rx=0 tx=0).stop 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69400640 1 -- 172.21.15.6:0/2809130940 reap_dead start 2024-03-06T07:38:28.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69e00640 1 -- 172.21.15.6:0/2809130940 shutdown_connections 2024-03-06T07:38:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69e00640 1 -- 172.21.15.6:0/2809130940 >> 172.21.15.6:0/2809130940 conn(0x7f9d640d3d30 msgr2=0x7f9d640d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69e00640 1 -- 172.21.15.6:0/2809130940 shutdown_connections 2024-03-06T07:38:28.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.911+0000 7f9d69e00640 1 -- 172.21.15.6:0/2809130940 wait complete. 2024-03-06T07:38:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.913+0000 7f9d69e00640 1 Processor -- start 2024-03-06T07:38:28.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.913+0000 7f9d69e00640 1 -- start start 2024-03-06T07:38:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d69e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d640b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d69e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d640b6860 0x7f9d6415f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d69e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640b1030 0x7f9d64160c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d69e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640cfc60 con 0x7f9d4c004d40 2024-03-06T07:38:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d69e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6415e490 con 0x7f9d640b6860 2024-03-06T07:38:28.917 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d69e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d640b54e0 con 0x7f9d640b1030 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d640b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d640b71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45752/0 (socket says 172.21.15.6:45752) 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 -- 172.21.15.6:0/3688301662 learned_addr learned my addr 172.21.15.6:0/3688301662 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 -- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640b1030 msgr2=0x7f9d64160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640b1030 0x7f9d64160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d63e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d640b6860 0x7f9d6415f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 -- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d640b6860 msgr2=0x7f9d6415f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d640b6860 0x7f9d6415f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.914+0000 7f9d68a00640 1 -- 172.21.15.6:0/3688301662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d640a1b50 con 0x7f9d4c004d40 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.915+0000 7f9d68a00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d640b71d0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f9d580119f0 tx=0x7f9d58011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.915+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800faf0 con 0x7f9d4c004d40 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.915+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5800fc90 con 0x7f9d4c004d40 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.915+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d640b7860 con 0x7f9d4c004d40 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.915+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d58014440 con 0x7f9d4c004d40 2024-03-06T07:38:28.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.915+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6415e8f0 con 0x7f9d4c004d40 2024-03-06T07:38:28.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.916+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6404f330 con 0x7f9d4c004d40 2024-03-06T07:38:28.919 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.917+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d58012040 con 0x7f9d4c004d40 2024-03-06T07:38:28.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.919+0000 7f9d60c00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d4406b500 0x7f9d4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:28.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.919+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9d5808a800 con 0x7f9d4c004d40 2024-03-06T07:38:28.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.920+0000 7f9d63e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d4406b500 0x7f9d4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:28.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.921+0000 7f9d63e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d4406b500 0x7f9d4406d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9d640ca9b0 tx=0x7f9d50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:28.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:28.924+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d58019030 con 0x7f9d4c004d40 2024-03-06T07:38:29.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.159+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d640b3080 con 0x7f9d4c004d40 2024-03-06T07:38:29.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.160+0000 7f9d60c00640 1 -- 172.21.15.6:0/3688301662 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f9d580596d0 con 0x7f9d4c004d40 2024-03-06T07:38:29.163 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:29.163 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:29.163 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:29.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d4406b500 msgr2=0x7f9d4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d4406b500 0x7f9d4406d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9d640ca9b0 tx=0x7f9d50008040 comp rx=0 tx=0).stop 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 msgr2=0x7f9d640b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d640b71d0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f9d580119f0 tx=0x7f9d58011eb0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 shutdown_connections 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9d4406b500 0x7f9d4406d9c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d640b6860 0x7f9d6415f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9d640b1030 0x7f9d64160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 --2- 172.21.15.6:0/3688301662 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9d4c004d40 0x7f9d640b71d0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 >> 172.21.15.6:0/3688301662 conn(0x7f9d640d3d30 msgr2=0x7f9d6406cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 shutdown_connections 2024-03-06T07:38:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:29.163+0000 7f9d69e00640 1 -- 172.21.15.6:0/3688301662 wait complete. 2024-03-06T07:38:29.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:29.384 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:30.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:30.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.605+0000 7f433be00640 1 Processor -- start 2024-03-06T07:38:30.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.605+0000 7f433be00640 1 -- start start 2024-03-06T07:38:30.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.607+0000 7f433be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 0x7f433c168cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.607+0000 7f433be00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c0d1120 con 0x7f433c15e2a0 2024-03-06T07:38:30.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.607+0000 7f433be00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c0d1400 con 0x7f433c15dec0 2024-03-06T07:38:30.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.607+0000 7f433be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0b0ea0 con 0x7f433c0b2340 2024-03-06T07:38:30.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.607+0000 7f4332000640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f433c15e2a0 0x7f433c0cef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:37386/0 (socket says 172.21.15.6:37386) 2024-03-06T07:38:30.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.607+0000 7f4332000640 1 -- 172.21.15.6:0/3682192988 learned_addr learned my addr 172.21.15.6:0/3682192988 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433b400640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 0x7f433c168cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720394716 0 0) 0x7f433c0d1120 con 0x7f433c15e2a0 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4320003520 con 0x7f433c15e2a0 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2341674456 0 0) 0x7f4320003520 con 0x7f433c15e2a0 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 msgr2=0x7f433c168cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 0x7f433c168cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 >> v1:172.21.15.178:6789/0 conn(0x7f433c15dec0 legacy=0x7f433c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0b0b30 con 0x7f433c15e2a0 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f4324001670 con 0x7f433c15e2a0 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 >> v1:172.21.15.6:6789/0 conn(0x7f433c15e2a0 legacy=0x7f433c0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43200039e0 0x7f4320003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 0x7f4320004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4320004d40 0x7f43200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0b0ea0 con 0x7f4320004d40 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432000bb50 con 0x7f43200039e0 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.608+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432000bd50 con 0x7f433c0b2340 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f433b400640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4320004d40 0x7f43200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f4332000640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 0x7f4320004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f433aa00640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43200039e0 0x7f4320003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f433b400640 1 -- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 msgr2=0x7f4320004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f433b400640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f433c0b2340 0x7f4320004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f433b400640 1 -- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43200039e0 msgr2=0x7f4320003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:30.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.609+0000 7f433b400640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f43200039e0 0x7f4320003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.610+0000 7f433b400640 1 -- 172.21.15.6:0/3682192988 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4320003520 con 0x7f4320004d40 2024-03-06T07:38:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.611+0000 7f433b400640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4320004d40 0x7f43200051c0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f433400ba10 tx=0x7f4320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d77c565eea2f36c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.611+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334036020 con 0x7f4320004d40 2024-03-06T07:38:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.611+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4334032630 con 0x7f4320004d40 2024-03-06T07:38:30.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.611+0000 7f433a000640 1 -- 172.21.15.6:0/3682192988 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433400b610 con 0x7f4320004d40 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433be00640 1 -- 172.21.15.6:0/3682192988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4320004d40 msgr2=0x7f43200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433be00640 1 --2- 172.21.15.6:0/3682192988 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4320004d40 0x7f43200051c0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f433400ba10 tx=0x7f4320005830 comp rx=0 tx=0).stop 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433b400640 1 -- 172.21.15.6:0/3682192988 reap_dead start 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433be00640 1 -- 172.21.15.6:0/3682192988 shutdown_connections 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433be00640 1 -- 172.21.15.6:0/3682192988 >> 172.21.15.6:0/3682192988 conn(0x7f433c0d3d40 msgr2=0x7f433c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433be00640 1 -- 172.21.15.6:0/3682192988 shutdown_connections 2024-03-06T07:38:30.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.612+0000 7f433be00640 1 -- 172.21.15.6:0/3682192988 wait complete. 2024-03-06T07:38:30.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.614+0000 7f433be00640 1 Processor -- start 2024-03-06T07:38:30.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.614+0000 7f433be00640 1 -- start start 2024-03-06T07:38:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4320004d40 0x7f433c0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f433c0cc4a0 0x7f433c153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 0x7f433c09fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0d0d40 con 0x7f433c09f940 2024-03-06T07:38:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0cf630 con 0x7f433c0cc4a0 2024-03-06T07:38:30.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0d0ec0 con 0x7f4320004d40 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 0x7f433c09fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 0x7f433c09fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45764/0 (socket says 172.21.15.6:45764) 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.615+0000 7f433b400640 1 -- 172.21.15.6:0/1750663366 learned_addr learned my addr 172.21.15.6:0/1750663366 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433aa00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4320004d40 0x7f433c0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f4332000640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f433c0cc4a0 0x7f433c153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433b400640 1 -- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4320004d40 msgr2=0x7f433c0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433b400640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4320004d40 0x7f433c0cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433b400640 1 -- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f433c0cc4a0 msgr2=0x7f433c153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433b400640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f433c0cc4a0 0x7f433c153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433b400640 1 -- 172.21.15.6:0/1750663366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c0cdd50 con 0x7f433c09f940 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f4332000640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f433c0cc4a0 0x7f433c153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:30.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433b400640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 0x7f433c09fdc0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f4334035040 tx=0x7f4334004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334041070 con 0x7f433c09f940 2024-03-06T07:38:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433403a8f0 con 0x7f433c09f940 2024-03-06T07:38:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334036040 con 0x7f433c09f940 2024-03-06T07:38:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.616+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c09ef00 con 0x7f433c09f940 2024-03-06T07:38:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.617+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c0cbca0 con 0x7f433c09f940 2024-03-06T07:38:30.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.618+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c04f330 con 0x7f433c09f940 2024-03-06T07:38:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.619+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4334039020 con 0x7f433c09f940 2024-03-06T07:38:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.619+0000 7f4333400640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:30.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.620+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f43340ac1e0 con 0x7f433c09f940 2024-03-06T07:38:30.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.620+0000 7f433aa00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:30.623 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.622+0000 7f433aa00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f431806b4d0 0x7f431806d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f432c007fd0 tx=0x7f432c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:30.626 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.624+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f433402fe20 con 0x7f433c09f940 2024-03-06T07:38:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.920+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f433c15c570 con 0x7f433c09f940 2024-03-06T07:38:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.920+0000 7f4333400640 1 -- 172.21.15.6:0/1750663366 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f433407b010 con 0x7f433c09f940 2024-03-06T07:38:30.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:30.923 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:30.923 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f431806b4d0 msgr2=0x7f431806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f431806b4d0 0x7f431806d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f432c007fd0 tx=0x7f432c009040 comp rx=0 tx=0).stop 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 msgr2=0x7f433c09fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 0x7f433c09fdc0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f4334035040 tx=0x7f4334004730 comp rx=0 tx=0).stop 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 shutdown_connections 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f431806b4d0 0x7f431806d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f433c0cc4a0 0x7f433c153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f433c09f940 0x7f433c09fdc0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 --2- 172.21.15.6:0/1750663366 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4320004d40 0x7f433c0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 >> 172.21.15.6:0/1750663366 conn(0x7f433c0d3d40 msgr2=0x7f433c15e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.924+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 shutdown_connections 2024-03-06T07:38:30.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:30.925+0000 7f433be00640 1 -- 172.21.15.6:0/1750663366 wait complete. 2024-03-06T07:38:31.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:31.149 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:32.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.376+0000 7f7fd0a00640 1 Processor -- start 2024-03-06T07:38:32.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.376+0000 7f7fd0a00640 1 -- start start 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fd0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7fcc06c8b0 0x7f7fcc06ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fd0a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7fcc0ce370 0x7f7fcc0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fd0a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc0cf1b0 con 0x7f7fcc066d00 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fd0a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc15e470 con 0x7f7fcc06c8b0 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fd0a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc15ea00 con 0x7f7fcc0ce370 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fcb400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7fcc06c8b0 0x7f7fcc06ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fcaa00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7fcc066d00 0x7f7fcc15ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47934/0 (socket says 172.21.15.6:47934) 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.378+0000 7f7fcaa00640 1 -- 172.21.15.6:0/3608666759 learned_addr learned my addr 172.21.15.6:0/3608666759 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fcbe00640 1 --2- 172.21.15.6:0/3608666759 >> v2:172.21.15.178:3300/0 conn(0x7f7fcc0ce370 0x7f7fcc0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fcb400640 1 -- 172.21.15.6:0/3608666759 >> v2:172.21.15.178:3300/0 conn(0x7f7fcc0ce370 msgr2=0x7f7fcc0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fcb400640 1 --2- 172.21.15.6:0/3608666759 >> v2:172.21.15.178:3300/0 conn(0x7f7fcc0ce370 0x7f7fcc0ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fcb400640 1 -- 172.21.15.6:0/3608666759 >> v1:172.21.15.6:6789/0 conn(0x7f7fcc066d00 legacy=0x7f7fcc15ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fcb400640 1 -- 172.21.15.6:0/3608666759 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc06d420 con 0x7f7fcc06c8b0 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fcbe00640 1 --2- 172.21.15.6:0/3608666759 >> v2:172.21.15.178:3300/0 conn(0x7f7fcc0ce370 0x7f7fcc0ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:32.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.379+0000 7f7fca000640 1 -- 172.21.15.6:0/3608666759 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270681502 0 0) 0x7f7fcc0cf1b0 con 0x7f7fcc066d00 2024-03-06T07:38:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.380+0000 7f7fcb400640 1 --2- 172.21.15.6:0/3608666759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc06c8b0 0x7f7fcc06ef10 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc071370 tx=0x7f7fbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a05f0524fb26e2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:32.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.380+0000 7f7fca000640 1 -- 172.21.15.6:0/3608666759 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc035020 con 0x7f7fcc06c8b0 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.380+0000 7f7fca000640 1 -- 172.21.15.6:0/3608666759 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc031830 con 0x7f7fcc06c8b0 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.380+0000 7f7fca000640 1 -- 172.21.15.6:0/3608666759 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc031c00 con 0x7f7fcc06c8b0 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3608666759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc06c8b0 msgr2=0x7f7fcc06ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3608666759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc06c8b0 0x7f7fcc06ef10 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc071370 tx=0x7f7fbc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3608666759 shutdown_connections 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3608666759 >> v2:172.21.15.178:3300/0 conn(0x7f7fcc0ce370 0x7f7fcc0ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3608666759 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc06c8b0 0x7f7fcc06ef10 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3608666759 >> 172.21.15.6:0/3608666759 conn(0x7f7fcc0ccf70 msgr2=0x7f7fcc0cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3608666759 shutdown_connections 2024-03-06T07:38:32.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.381+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3608666759 wait complete. 2024-03-06T07:38:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.383+0000 7f7fd0a00640 1 Processor -- start 2024-03-06T07:38:32.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.383+0000 7f7fd0a00640 1 -- start start 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fd0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 0x7f7fcc158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fd0a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fcc06c8b0 0x7f7fcc14acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fd0a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fcc0ce370 0x7f7fcc06b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fd0a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0cee60 con 0x7f7fcc066d00 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fd0a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc0cece0 con 0x7f7fcc0ce370 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fd0a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc06afb0 con 0x7f7fcc06c8b0 2024-03-06T07:38:32.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fcb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 0x7f7fcc158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fcb400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 0x7f7fcc158780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50940/0 (socket says 172.21.15.6:50940) 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.384+0000 7f7fcb400640 1 -- 172.21.15.6:0/3950219013 learned_addr learned my addr 172.21.15.6:0/3950219013 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcaa00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fcc06c8b0 0x7f7fcc14acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcbe00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fcc0ce370 0x7f7fcc06b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcb400640 1 -- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fcc06c8b0 msgr2=0x7f7fcc14acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcb400640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fcc06c8b0 0x7f7fcc14acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcb400640 1 -- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fcc0ce370 msgr2=0x7f7fcc06b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcb400640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fcc0ce370 0x7f7fcc06b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.385+0000 7f7fcb400640 1 -- 172.21.15.6:0/3950219013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc14b1f0 con 0x7f7fcc066d00 2024-03-06T07:38:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.386+0000 7f7fcb400640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 0x7f7fcc158780 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009890 tx=0x7f7fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.386+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc035050 con 0x7f7fcc066d00 2024-03-06T07:38:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.386+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc004230 con 0x7f7fcc066d00 2024-03-06T07:38:32.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.386+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc03b800 con 0x7f7fcc066d00 2024-03-06T07:38:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.386+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc15baa0 con 0x7f7fcc066d00 2024-03-06T07:38:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.386+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc15c060 con 0x7f7fcc066d00 2024-03-06T07:38:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.387+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc04f330 con 0x7f7fcc066d00 2024-03-06T07:38:32.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.389+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fbc038020 con 0x7f7fcc066d00 2024-03-06T07:38:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.389+0000 7f7fc3400640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fac06b4d0 0x7f7fac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:32.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.390+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7fbc0ac0f0 con 0x7f7fcc066d00 2024-03-06T07:38:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.390+0000 7f7fcaa00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fac06b4d0 0x7f7fac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:32.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.391+0000 7f7fcaa00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fac06b4d0 0x7f7fac06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0cc7e0 tx=0x7f7fb8007b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:32.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.397+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fcc14b1f0 con 0x7f7fcc066d00 2024-03-06T07:38:32.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.648+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fcc0cd560 con 0x7f7fcc066d00 2024-03-06T07:38:32.651 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.649+0000 7f7fc3400640 1 -- 172.21.15.6:0/3950219013 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f7fbc040070 con 0x7f7fcc066d00 2024-03-06T07:38:32.651 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:32.651 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:32.652 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:32.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.652+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fac06b4d0 msgr2=0x7f7fac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.652+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fac06b4d0 0x7f7fac06d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0cc7e0 tx=0x7f7fb8007b50 comp rx=0 tx=0).stop 2024-03-06T07:38:32.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 msgr2=0x7f7fcc158780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:32.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 0x7f7fcc158780 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009890 tx=0x7f7fbc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.655 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 shutdown_connections 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7fac06b4d0 0x7f7fac06d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7fcc0ce370 0x7f7fcc06b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7fcc06c8b0 0x7f7fcc14acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 --2- 172.21.15.6:0/3950219013 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7fcc066d00 0x7f7fcc158780 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 >> 172.21.15.6:0/3950219013 conn(0x7f7fcc0ccf70 msgr2=0x7f7fcc162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 shutdown_connections 2024-03-06T07:38:32.656 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:32.653+0000 7f7fd0a00640 1 -- 172.21.15.6:0/3950219013 wait complete. 2024-03-06T07:38:32.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:32.883 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:33.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:34.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.146+0000 7f696f200640 1 Processor -- start 2024-03-06T07:38:34.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.146+0000 7f696f200640 1 -- start start 2024-03-06T07:38:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696f200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f69700700c0 0x7f6970070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696f200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700cd0c0 con 0x7f697015f450 2024-03-06T07:38:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696f200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700cd3a0 con 0x7f69700d04b0 2024-03-06T07:38:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696f200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69700d29e0 con 0x7f69700700c0 2024-03-06T07:38:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696de00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f697015f450 0x7f69700cfda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47940/0 (socket says 172.21.15.6:47940) 2024-03-06T07:38:34.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696de00640 1 -- 172.21.15.6:0/3525533826 learned_addr learned my addr 172.21.15.6:0/3525533826 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.148+0000 7f696e800640 1 --2- 172.21.15.6:0/3525533826 >> v2:172.21.15.6:3300/0 conn(0x7f69700700c0 0x7f6970070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.149+0000 7f696ca00640 1 -- 172.21.15.6:0/3525533826 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907380924 0 0) 0x7f69700cd0c0 con 0x7f697015f450 2024-03-06T07:38:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.149+0000 7f696ca00640 1 -- 172.21.15.6:0/3525533826 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6954003520 con 0x7f697015f450 2024-03-06T07:38:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.149+0000 7f696e800640 1 -- 172.21.15.6:0/3525533826 >> v1:172.21.15.178:6789/0 conn(0x7f69700d04b0 legacy=0x7f69700cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.149+0000 7f696e800640 1 -- 172.21.15.6:0/3525533826 >> v1:172.21.15.6:6789/0 conn(0x7f697015f450 legacy=0x7f69700cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.149+0000 7f696e800640 1 -- 172.21.15.6:0/3525533826 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700cd680 con 0x7f69700700c0 2024-03-06T07:38:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.150+0000 7f696e800640 1 --2- 172.21.15.6:0/3525533826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700700c0 0x7f6970070990 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f69680097a0 tx=0x7f696802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c769bcf4c20bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.150+0000 7f696ca00640 1 -- 172.21.15.6:0/3525533826 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968034070 con 0x7f69700700c0 2024-03-06T07:38:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696ca00640 1 -- 172.21.15.6:0/3525533826 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6968030610 con 0x7f69700700c0 2024-03-06T07:38:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696ca00640 1 -- 172.21.15.6:0/3525533826 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680309c0 con 0x7f69700700c0 2024-03-06T07:38:34.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 -- 172.21.15.6:0/3525533826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700700c0 msgr2=0x7f6970070990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 --2- 172.21.15.6:0/3525533826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700700c0 0x7f6970070990 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f69680097a0 tx=0x7f696802b960 comp rx=0 tx=0).stop 2024-03-06T07:38:34.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 -- 172.21.15.6:0/3525533826 shutdown_connections 2024-03-06T07:38:34.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 --2- 172.21.15.6:0/3525533826 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700700c0 0x7f6970070990 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 -- 172.21.15.6:0/3525533826 >> 172.21.15.6:0/3525533826 conn(0x7f69700d1ae0 msgr2=0x7f69700d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:34.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 -- 172.21.15.6:0/3525533826 shutdown_connections 2024-03-06T07:38:34.154 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.151+0000 7f696f200640 1 -- 172.21.15.6:0/3525533826 wait complete. 2024-03-06T07:38:34.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.153+0000 7f696f200640 1 Processor -- start 2024-03-06T07:38:34.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.153+0000 7f696f200640 1 -- start start 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696f200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69700700c0 0x7f6970153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696f200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 0x7f69700c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696f200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f697015f450 0x7f69700b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696f200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697015f830 con 0x7f69700d04b0 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696f200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69700ae8b0 con 0x7f69700700c0 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696f200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697015f9b0 con 0x7f697015f450 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 0x7f69700c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 0x7f69700c6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50960/0 (socket says 172.21.15.6:50960) 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.154+0000 7f696d400640 1 -- 172.21.15.6:0/509817448 learned_addr learned my addr 172.21.15.6:0/509817448 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696de00640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69700700c0 0x7f6970153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:34.158 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696e800640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f697015f450 0x7f69700b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696d400640 1 -- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f697015f450 msgr2=0x7f69700b0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696d400640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f697015f450 0x7f69700b0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696d400640 1 -- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69700700c0 msgr2=0x7f6970153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696d400640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69700700c0 0x7f6970153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696d400640 1 -- 172.21.15.6:0/509817448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69701531b0 con 0x7f69700d04b0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696de00640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69700700c0 0x7f6970153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696e800640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f697015f450 0x7f69700b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.155+0000 7f696d400640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 0x7f69700c6e40 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f696000c830 tx=0x7f696000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.156+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960003b70 con 0x7f69700d04b0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.156+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960003d10 con 0x7f69700d04b0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.156+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000aa70 con 0x7f69700d04b0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.156+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6970153490 con 0x7f69700d04b0 2024-03-06T07:38:34.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.156+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69700c8d40 con 0x7f69700d04b0 2024-03-06T07:38:34.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.157+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697004f330 con 0x7f69700d04b0 2024-03-06T07:38:34.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.158+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f696000ac10 con 0x7f69700d04b0 2024-03-06T07:38:34.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.159+0000 7f6966000640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694c06b4d0 0x7f694c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:34.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.159+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f696008a2b0 con 0x7f69700d04b0 2024-03-06T07:38:34.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.159+0000 7f696de00640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694c06b4d0 0x7f694c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:34.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.161+0000 7f696de00640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694c06b4d0 0x7f694c06d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f69580011e0 tx=0x7f6958009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:34.165 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.164+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6960059110 con 0x7f69700d04b0 2024-03-06T07:38:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:34.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.439+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69700cde20 con 0x7f69700d04b0 2024-03-06T07:38:34.441 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.439+0000 7f6966000640 1 -- 172.21.15.6:0/509817448 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f6960058a10 con 0x7f69700d04b0 2024-03-06T07:38:34.442 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:34.442 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:34.442 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:34.444 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694c06b4d0 msgr2=0x7f694c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694c06b4d0 0x7f694c06d990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f69580011e0 tx=0x7f6958009040 comp rx=0 tx=0).stop 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 msgr2=0x7f69700c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 0x7f69700c6e40 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f696000c830 tx=0x7f696000ccf0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 shutdown_connections 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694c06b4d0 0x7f694c06d990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f697015f450 0x7f69700b0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69700d04b0 0x7f69700c6e40 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 --2- 172.21.15.6:0/509817448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69700700c0 0x7f6970153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 >> 172.21.15.6:0/509817448 conn(0x7f69700d1ae0 msgr2=0x7f697006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 shutdown_connections 2024-03-06T07:38:34.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:34.442+0000 7f696f200640 1 -- 172.21.15.6:0/509817448 wait complete. 2024-03-06T07:38:34.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:34.663 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:35.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:35.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.888+0000 7fca9f000640 1 Processor -- start 2024-03-06T07:38:35.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.888+0000 7fca9f000640 1 -- start start 2024-03-06T07:38:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.890+0000 7fca9f000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca9815e2e0 0x7fca980d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.890+0000 7fca9f000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fca980d0d70 0x7fca980d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.890+0000 7fca9f000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca980b3940 con 0x7fca980d1b30 2024-03-06T07:38:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.890+0000 7fca9f000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca980d4e70 con 0x7fca9815e2e0 2024-03-06T07:38:35.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.890+0000 7fca9f000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca980b3c40 con 0x7fca980d0d70 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca9815e2e0 0x7fca980d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fca9815e2e0 0x7fca980d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50972/0 (socket says 172.21.15.6:50972) 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 -- 172.21.15.6:0/1942891705 learned_addr learned my addr 172.21.15.6:0/1942891705 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97400640 1 --2- 172.21.15.6:0/1942891705 >> v2:172.21.15.178:3300/0 conn(0x7fca980d0d70 0x7fca980d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 -- 172.21.15.6:0/1942891705 >> v2:172.21.15.178:3300/0 conn(0x7fca980d0d70 msgr2=0x7fca980d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 --2- 172.21.15.6:0/1942891705 >> v2:172.21.15.178:3300/0 conn(0x7fca980d0d70 0x7fca980d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 -- 172.21.15.6:0/1942891705 >> v1:172.21.15.178:6789/0 conn(0x7fca980d1b30 legacy=0x7fca980d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97e00640 1 -- 172.21.15.6:0/1942891705 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca980cf580 con 0x7fca9815e2e0 2024-03-06T07:38:35.894 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.891+0000 7fca97400640 1 --2- 172.21.15.6:0/1942891705 >> v2:172.21.15.178:3300/0 conn(0x7fca980d0d70 0x7fca980d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:35.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.893+0000 7fca97e00640 1 --2- 172.21.15.6:0/1942891705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca9815e2e0 0x7fca980d0830 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fca980d3510 tx=0x7fca8802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d67ea352d006e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca96a00640 1 -- 172.21.15.6:0/1942891705 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88035020 con 0x7fca9815e2e0 2024-03-06T07:38:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca96a00640 1 -- 172.21.15.6:0/1942891705 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca88031870 con 0x7fca9815e2e0 2024-03-06T07:38:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca96a00640 1 -- 172.21.15.6:0/1942891705 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88031c40 con 0x7fca9815e2e0 2024-03-06T07:38:35.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca9f000640 1 -- 172.21.15.6:0/1942891705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca9815e2e0 msgr2=0x7fca980d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca9f000640 1 --2- 172.21.15.6:0/1942891705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca9815e2e0 0x7fca980d0830 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fca980d3510 tx=0x7fca8802b910 comp rx=0 tx=0).stop 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca9f000640 1 -- 172.21.15.6:0/1942891705 shutdown_connections 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca9f000640 1 --2- 172.21.15.6:0/1942891705 >> v2:172.21.15.178:3300/0 conn(0x7fca980d0d70 0x7fca980d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca9f000640 1 --2- 172.21.15.6:0/1942891705 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca9815e2e0 0x7fca980d0830 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.894+0000 7fca9f000640 1 -- 172.21.15.6:0/1942891705 >> 172.21.15.6:0/1942891705 conn(0x7fca980d3f60 msgr2=0x7fca980d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.895+0000 7fca9f000640 1 -- 172.21.15.6:0/1942891705 shutdown_connections 2024-03-06T07:38:35.897 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.895+0000 7fca9f000640 1 -- 172.21.15.6:0/1942891705 wait complete. 2024-03-06T07:38:35.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.896+0000 7fca9f000640 1 Processor -- start 2024-03-06T07:38:35.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.896+0000 7fca9f000640 1 -- start start 2024-03-06T07:38:35.900 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.897+0000 7fca9f000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 0x7fca981f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.897+0000 7fca9f000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca980d1b30 0x7fca980b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.897+0000 7fca9f000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca9815e2e0 0x7fca98159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.897+0000 7fca9f000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca980cfe20 con 0x7fca980d0d70 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.897+0000 7fca9f000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca980cfb40 con 0x7fca9815e2e0 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.897+0000 7fca9f000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca980b50e0 con 0x7fca980d1b30 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 0x7fca981f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 0x7fca981f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50984/0 (socket says 172.21.15.6:50984) 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 -- 172.21.15.6:0/933991980 learned_addr learned my addr 172.21.15.6:0/933991980 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca9d000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca9815e2e0 0x7fca98159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97400640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca980d1b30 0x7fca980b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 -- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca980d1b30 msgr2=0x7fca980b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca980d1b30 0x7fca980b72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 -- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca9815e2e0 msgr2=0x7fca98159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca9815e2e0 0x7fca98159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.898+0000 7fca97e00640 1 -- 172.21.15.6:0/933991980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca981f4980 con 0x7fca980d0d70 2024-03-06T07:38:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.899+0000 7fca97e00640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 0x7fca981f4440 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fca8802b950 tx=0x7fca880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.899+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88040070 con 0x7fca980d0d70 2024-03-06T07:38:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.899+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8803a420 con 0x7fca980d0d70 2024-03-06T07:38:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.899+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca980b8670 con 0x7fca980d0d70 2024-03-06T07:38:35.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.899+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88035040 con 0x7fca980d0d70 2024-03-06T07:38:35.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.899+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca980b8c30 con 0x7fca980d0d70 2024-03-06T07:38:35.903 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.900+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9804f3a0 con 0x7fca980d0d70 2024-03-06T07:38:35.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.902+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca8804b060 con 0x7fca980d0d70 2024-03-06T07:38:35.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.903+0000 7fca8fe00640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca7806b4d0 0x7fca7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:35.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.903+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fca880ac1c0 con 0x7fca980d0d70 2024-03-06T07:38:35.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.904+0000 7fca97400640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca7806b4d0 0x7fca7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:35.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.905+0000 7fca97400640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca7806b4d0 0x7fca7806d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fca980cdf40 tx=0x7fca84008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:35.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:35.908+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca8807aff0 con 0x7fca980d0d70 2024-03-06T07:38:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.148+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca9806c6c0 con 0x7fca980d0d70 2024-03-06T07:38:36.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.149+0000 7fca8fe00640 1 -- 172.21.15.6:0/933991980 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fca8807a8f0 con 0x7fca980d0d70 2024-03-06T07:38:36.152 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:36.152 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:36.152 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca7806b4d0 msgr2=0x7fca7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca7806b4d0 0x7fca7806d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fca980cdf40 tx=0x7fca84008040 comp rx=0 tx=0).stop 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 msgr2=0x7fca981f4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 0x7fca981f4440 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fca8802b950 tx=0x7fca880046c0 comp rx=0 tx=0).stop 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 shutdown_connections 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fca7806b4d0 0x7fca7806d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca9815e2e0 0x7fca98159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fca980d1b30 0x7fca980b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 --2- 172.21.15.6:0/933991980 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fca980d0d70 0x7fca981f4440 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 >> 172.21.15.6:0/933991980 conn(0x7fca980d3f60 msgr2=0x7fca9806ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:36.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.152+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 shutdown_connections 2024-03-06T07:38:36.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:36.153+0000 7fca9f000640 1 -- 172.21.15.6:0/933991980 wait complete. 2024-03-06T07:38:36.206 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:36.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:36.377 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:37.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.601+0000 7f61ea400640 1 Processor -- start 2024-03-06T07:38:37.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.602+0000 7f61ea400640 1 -- start start 2024-03-06T07:38:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.603+0000 7f61ea400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 0x7f61e4162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.603+0000 7f61ea400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e4161a80 con 0x7f61e406c420 2024-03-06T07:38:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.603+0000 7f61ea400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e415efa0 con 0x7f61e406bfc0 2024-03-06T07:38:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.603+0000 7f61ea400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e415f280 con 0x7f61e415dd00 2024-03-06T07:38:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.604+0000 7f61e3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f61e406c420 0x7f61e415e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47954/0 (socket says 172.21.15.6:47954) 2024-03-06T07:38:37.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.604+0000 7f61e3e00640 1 -- 172.21.15.6:0/2409844233 learned_addr learned my addr 172.21.15.6:0/2409844233 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.604+0000 7f61e3400640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 0x7f61e4162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.604+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488944052 0 0) 0x7f61e4161a80 con 0x7f61e406c420 2024-03-06T07:38:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.604+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c8003520 con 0x7f61e406c420 2024-03-06T07:38:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3415078275 0 0) 0x7f61c8003520 con 0x7f61e406c420 2024-03-06T07:38:37.608 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 msgr2=0x7f61e4162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 0x7f61e4162520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 >> v1:172.21.15.178:6789/0 conn(0x7f61e406bfc0 legacy=0x7f61e415e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4161d20 con 0x7f61e406c420 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f61dc001a10 con 0x7f61e406c420 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 >> v1:172.21.15.6:6789/0 conn(0x7f61e406c420 legacy=0x7f61e415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c8003a50 0x7f61c8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c8004800 0x7f61c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 0x7f61c8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e415f280 con 0x7f61c8003a50 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c800bb80 con 0x7f61c8004800 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c800bd80 con 0x7f61e415dd00 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e3400640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c8003a50 0x7f61c8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e3e00640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 0x7f61c8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e3400640 1 -- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 msgr2=0x7f61c8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.606+0000 7f61e3400640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61e415dd00 0x7f61c8005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.605+0000 7f61e2a00640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c8004800 0x7f61c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.606+0000 7f61e3400640 1 -- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c8004800 msgr2=0x7f61c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.606+0000 7f61e3400640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c8004800 0x7f61c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.606+0000 7f61e3400640 1 -- 172.21.15.6:0/2409844233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8003520 con 0x7f61c8003a50 2024-03-06T07:38:37.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.606+0000 7f61e2a00640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61c8004800 0x7f61c8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.607+0000 7f61e3400640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c8003a50 0x7f61c8003e70 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009ac0 tx=0x7f61c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e015a65dda7ba82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.607+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4039070 con 0x7f61c8003a50 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.607+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d402c6b0 con 0x7f61c8003a50 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.607+0000 7f61e2000640 1 -- 172.21.15.6:0/2409844233 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4009690 con 0x7f61c8003a50 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.607+0000 7f61ea400640 1 -- 172.21.15.6:0/2409844233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c8003a50 msgr2=0x7f61c8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.607+0000 7f61ea400640 1 --2- 172.21.15.6:0/2409844233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61c8003a50 0x7f61c8003e70 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009ac0 tx=0x7f61c8005830 comp rx=0 tx=0).stop 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.608+0000 7f61e3e00640 1 -- 172.21.15.6:0/2409844233 reap_dead start 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.608+0000 7f61ea400640 1 -- 172.21.15.6:0/2409844233 shutdown_connections 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.608+0000 7f61ea400640 1 -- 172.21.15.6:0/2409844233 >> 172.21.15.6:0/2409844233 conn(0x7f61e40714d0 msgr2=0x7f61e40718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.608+0000 7f61ea400640 1 -- 172.21.15.6:0/2409844233 shutdown_connections 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.608+0000 7f61ea400640 1 -- 172.21.15.6:0/2409844233 wait complete. 2024-03-06T07:38:37.612 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.609+0000 7f61ea400640 1 Processor -- start 2024-03-06T07:38:37.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 -- start start 2024-03-06T07:38:37.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c8003a50 0x7f61e4157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e4141920 0x7f61e41449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 0x7f61e40af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e406f280 con 0x7f61e40af2e0 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e406c890 con 0x7f61e4141920 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.610+0000 7f61ea400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4162520 con 0x7f61c8003a50 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 0x7f61e40af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 0x7f61e40af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50990/0 (socket says 172.21.15.6:50990) 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 -- 172.21.15.6:0/3922530227 learned_addr learned my addr 172.21.15.6:0/3922530227 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c8003a50 0x7f61e4157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e2a00640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e4141920 0x7f61e41449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 -- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c8003a50 msgr2=0x7f61e4157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c8003a50 0x7f61e4157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 -- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e4141920 msgr2=0x7f61e41449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e4141920 0x7f61e41449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3e00640 1 -- 172.21.15.6:0/3922530227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4144f20 con 0x7f61e40af2e0 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.611+0000 7f61e3400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c8003a50 0x7f61e4157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.612+0000 7f61e3e00640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 0x7f61e40af760 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f61e415f590 tx=0x7f61dc007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.612+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00eb60 con 0x7f61e40af2e0 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.612+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc00ed20 con 0x7f61e40af2e0 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.612+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc014670 con 0x7f61e40af2e0 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.612+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e41584f0 con 0x7f61e40af2e0 2024-03-06T07:38:37.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.612+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e4158ab0 con 0x7f61e40af2e0 2024-03-06T07:38:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.613+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e40afca0 con 0x7f61e40af2e0 2024-03-06T07:38:37.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.615+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61dc014810 con 0x7f61e40af2e0 2024-03-06T07:38:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.615+0000 7f61db400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61c006b500 0x7f61c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.616+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f61dc089fc0 con 0x7f61e40af2e0 2024-03-06T07:38:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.616+0000 7f61e3400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61c006b500 0x7f61c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:37.619 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.617+0000 7f61e3400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61c006b500 0x7f61c006d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f61d4036000 tx=0x7f61d4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.625+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61e4144f20 con 0x7f61e40af2e0 2024-03-06T07:38:37.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.863+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e404f3a0 con 0x7f61e40af2e0 2024-03-06T07:38:37.866 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.864+0000 7f61db400640 1 -- 172.21.15.6:0/3922530227 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f61dc058d60 con 0x7f61e40af2e0 2024-03-06T07:38:37.867 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:37.867 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:37.867 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.867+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61c006b500 msgr2=0x7f61c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.867+0000 7f61ea400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61c006b500 0x7f61c006d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f61d4036000 tx=0x7f61d4030000 comp rx=0 tx=0).stop 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.867+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 msgr2=0x7f61e40af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.867+0000 7f61ea400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 0x7f61e40af760 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f61e415f590 tx=0x7f61dc007b70 comp rx=0 tx=0).stop 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 shutdown_connections 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f61c006b500 0x7f61c006d9c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61e4141920 0x7f61e41449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61e40af2e0 0x7f61e40af760 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 --2- 172.21.15.6:0/3922530227 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f61c8003a50 0x7f61e4157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:37.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 >> 172.21.15.6:0/3922530227 conn(0x7f61e40714d0 msgr2=0x7f61e40ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:37.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 shutdown_connections 2024-03-06T07:38:37.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:37.868+0000 7f61ea400640 1 -- 172.21.15.6:0/3922530227 wait complete. 2024-03-06T07:38:38.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:38.089 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:38.209 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:39.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:39.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.310+0000 7fcb0fe00640 1 Processor -- start 2024-03-06T07:38:39.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.310+0000 7fcb0fe00640 1 -- start start 2024-03-06T07:38:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fcb1015e0b0 0x7fcb100cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcb100d4a10 0x7fcb100d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0fe00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb100b3710 con 0x7fcb100d0c80 2024-03-06T07:38:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100d10f0 con 0x7fcb1015e0b0 2024-03-06T07:38:39.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb100b09e0 con 0x7fcb100d4a10 2024-03-06T07:38:39.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0f400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fcb100d0c80 0x7fcb100d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47958/0 (socket says 172.21.15.6:47958) 2024-03-06T07:38:39.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0f400640 1 -- 172.21.15.6:0/2493619664 learned_addr learned my addr 172.21.15.6:0/2493619664 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:39.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.312+0000 7fcb0ea00640 1 --2- 172.21.15.6:0/2493619664 >> v2:172.21.15.6:3300/0 conn(0x7fcb1015e0b0 0x7fcb100cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.313+0000 7fcb0e000640 1 --2- 172.21.15.6:0/2493619664 >> v2:172.21.15.178:3300/0 conn(0x7fcb100d4a10 0x7fcb100d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.313+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2493619664 >> v2:172.21.15.178:3300/0 conn(0x7fcb100d4a10 msgr2=0x7fcb100d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.313+0000 7fcb0ea00640 1 --2- 172.21.15.6:0/2493619664 >> v2:172.21.15.178:3300/0 conn(0x7fcb100d4a10 0x7fcb100d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.313+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2493619664 >> v1:172.21.15.6:6789/0 conn(0x7fcb100d0c80 legacy=0x7fcb100d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.313+0000 7fcb0d600640 1 -- 172.21.15.6:0/2493619664 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267255691 0 0) 0x7fcb100b3710 con 0x7fcb100d0c80 2024-03-06T07:38:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.313+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2493619664 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100b39f0 con 0x7fcb1015e0b0 2024-03-06T07:38:39.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.314+0000 7fcb0ea00640 1 --2- 172.21.15.6:0/2493619664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb1015e0b0 0x7fcb100cf310 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fcb1006b8c0 tx=0x7fcb0002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefe2dc126e6c412 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0d600640 1 -- 172.21.15.6:0/2493619664 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00039070 con 0x7fcb1015e0b0 2024-03-06T07:38:39.317 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0d600640 1 -- 172.21.15.6:0/2493619664 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0002cc60 con 0x7fcb1015e0b0 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0d600640 1 -- 172.21.15.6:0/2493619664 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00034660 con 0x7fcb1015e0b0 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2493619664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb1015e0b0 msgr2=0x7fcb100cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2493619664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb1015e0b0 0x7fcb100cf310 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fcb1006b8c0 tx=0x7fcb0002beb0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2493619664 shutdown_connections 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2493619664 >> v2:172.21.15.178:3300/0 conn(0x7fcb100d4a10 0x7fcb100d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2493619664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb1015e0b0 0x7fcb100cf310 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2493619664 >> 172.21.15.6:0/2493619664 conn(0x7fcb100d3d30 msgr2=0x7fcb100d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2493619664 shutdown_connections 2024-03-06T07:38:39.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.315+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2493619664 wait complete. 2024-03-06T07:38:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.317+0000 7fcb0fe00640 1 Processor -- start 2024-03-06T07:38:39.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.317+0000 7fcb0fe00640 1 -- start start 2024-03-06T07:38:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 0x7fcb1015f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb100d4a10 0x7fcb100b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb1015e0b0 0x7fcb10153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1015e490 con 0x7fcb100d0c80 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100b3fb0 con 0x7fcb1015e0b0 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100cfc60 con 0x7fcb100d4a10 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 0x7fcb1015f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 0x7fcb1015f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51002/0 (socket says 172.21.15.6:51002) 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.318+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2334744919 learned_addr learned my addr 172.21.15.6:0/2334744919 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0e000640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb100d4a10 0x7fcb100b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0f400640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb1015e0b0 0x7fcb10153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb100d4a10 msgr2=0x7fcb100b6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0ea00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb100d4a10 0x7fcb100b6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb1015e0b0 msgr2=0x7fcb10153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0ea00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb1015e0b0 0x7fcb10153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0ea00640 1 -- 172.21.15.6:0/2334744919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb100b70a0 con 0x7fcb100d0c80 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0e000640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb100d4a10 0x7fcb100b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:39.322 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.319+0000 7fcb0ea00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 0x7fcb1015f2c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009d90 tx=0x7fcb00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.320+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00039050 con 0x7fcb100d0c80 2024-03-06T07:38:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.320+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0003a040 con 0x7fcb100d0c80 2024-03-06T07:38:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.320+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0003e610 con 0x7fcb100d0c80 2024-03-06T07:38:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.320+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb100b7380 con 0x7fcb100d0c80 2024-03-06T07:38:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.320+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb100b7940 con 0x7fcb100d0c80 2024-03-06T07:38:39.323 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.321+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1004f330 con 0x7fcb100d0c80 2024-03-06T07:38:39.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.322+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb00047020 con 0x7fcb100d0c80 2024-03-06T07:38:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.323+0000 7fcb06a00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcaf006b500 0x7fcaf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:39.325 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.324+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcb000ac650 con 0x7fcb100d0c80 2024-03-06T07:38:39.326 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.324+0000 7fcb0e000640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcaf006b500 0x7fcaf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:39.327 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.325+0000 7fcb0e000640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcaf006b500 0x7fcaf006d9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fcb100b8c90 tx=0x7fcafc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:39.330 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.328+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb0004e020 con 0x7fcb100d0c80 2024-03-06T07:38:39.574 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.572+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb100b9b30 con 0x7fcb100d0c80 2024-03-06T07:38:39.575 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.573+0000 7fcb06a00640 1 -- 172.21.15.6:0/2334744919 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fcb00032e20 con 0x7fcb100d0c80 2024-03-06T07:38:39.575 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:39.575 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:39.576 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.575+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcaf006b500 msgr2=0x7fcaf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.575+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcaf006b500 0x7fcaf006d9c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fcb100b8c90 tx=0x7fcafc008040 comp rx=0 tx=0).stop 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.575+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 msgr2=0x7fcb1015f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.575+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 0x7fcb1015f2c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009d90 tx=0x7fcb00004a40 comp rx=0 tx=0).stop 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 shutdown_connections 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcaf006b500 0x7fcaf006d9c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcb1015e0b0 0x7fcb10153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcb100d4a10 0x7fcb100b6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 --2- 172.21.15.6:0/2334744919 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcb100d0c80 0x7fcb1015f2c0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:39.578 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 >> 172.21.15.6:0/2334744919 conn(0x7fcb100d3d30 msgr2=0x7fcb1015d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:39.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 shutdown_connections 2024-03-06T07:38:39.579 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:39.576+0000 7fcb0fe00640 1 -- 172.21.15.6:0/2334744919 wait complete. 2024-03-06T07:38:39.797 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:39.798 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:40.211 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:40.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:41.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.071+0000 7f5836e00640 1 Processor -- start 2024-03-06T07:38:41.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.071+0000 7f5836e00640 1 -- start start 2024-03-06T07:38:41.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.073+0000 7f5836e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f583015f450 0x7f5830070990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.073+0000 7f5836e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300cd0c0 con 0x7f58300d04b0 2024-03-06T07:38:41.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.073+0000 7f5836e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300cd3a0 con 0x7f58300700c0 2024-03-06T07:38:41.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.073+0000 7f5836e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58300d29e0 con 0x7f583015f450 2024-03-06T07:38:41.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.073+0000 7f5836400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f58300d04b0 0x7f58300cde80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47966/0 (socket says 172.21.15.6:47966) 2024-03-06T07:38:41.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.073+0000 7f5836400640 1 -- 172.21.15.6:0/360398059 learned_addr learned my addr 172.21.15.6:0/360398059 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:41.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f5835a00640 1 --2- 172.21.15.6:0/360398059 >> v2:172.21.15.178:3300/0 conn(0x7f583015f450 0x7f5830070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171854907 0 0) 0x7f58300cd0c0 con 0x7f58300d04b0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5818003520 con 0x7f58300d04b0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254824299 0 0) 0x7f5818003520 con 0x7f58300d04b0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 >> v2:172.21.15.178:3300/0 conn(0x7f583015f450 msgr2=0x7f5830070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 --2- 172.21.15.6:0/360398059 >> v2:172.21.15.178:3300/0 conn(0x7f583015f450 0x7f5830070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 >> v1:172.21.15.178:6789/0 conn(0x7f58300700c0 legacy=0x7f58300cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58300cd680 con 0x7f58300d04b0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f58280019f0 con 0x7f58300d04b0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 >> v1:172.21.15.6:6789/0 conn(0x7f58300d04b0 legacy=0x7f58300cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5818003a00 0x7f5818003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5818004800 0x7f5818004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58180051c0 0x7f581800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58300d29e0 con 0x7f58180051c0 2024-03-06T07:38:41.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.074+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581800c0c0 con 0x7f5818004800 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581800c2c0 con 0x7f5818003a00 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5836400640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58180051c0 0x7f581800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5835a00640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5818003a00 0x7f5818003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5835000640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5818004800 0x7f5818004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5836400640 1 -- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5818003a00 msgr2=0x7f5818003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5836400640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f5818003a00 0x7f5818003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5836400640 1 -- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5818004800 msgr2=0x7f5818004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5836400640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5818004800 0x7f5818004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5836400640 1 -- 172.21.15.6:0/360398059 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5818003520 con 0x7f58180051c0 2024-03-06T07:38:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.075+0000 7f5835000640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5818004800 0x7f5818004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:41.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.076+0000 7f5836400640 1 -- 172.21.15.6:0/360398059 reap_dead start 2024-03-06T07:38:41.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.076+0000 7f5836400640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58180051c0 0x7f581800b9c0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f582800a7b0 tx=0x7f5818005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f36e99eff6e4368 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582800a3d0 con 0x7f58180051c0 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828034650 con 0x7f58180051c0 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f582fe00640 1 -- 172.21.15.6:0/360398059 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828034a40 con 0x7f58180051c0 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f5836e00640 1 -- 172.21.15.6:0/360398059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58180051c0 msgr2=0x7f581800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f5836e00640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58180051c0 0x7f581800b9c0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f582800a7b0 tx=0x7f5818005830 comp rx=0 tx=0).stop 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f5836e00640 1 -- 172.21.15.6:0/360398059 shutdown_connections 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f5836e00640 1 --2- 172.21.15.6:0/360398059 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58180051c0 0x7f581800b9c0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f5836e00640 1 -- 172.21.15.6:0/360398059 >> 172.21.15.6:0/360398059 conn(0x7f58300d1ae0 msgr2=0x7f58300d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.077+0000 7f5836e00640 1 -- 172.21.15.6:0/360398059 shutdown_connections 2024-03-06T07:38:41.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.078+0000 7f5836e00640 1 -- 172.21.15.6:0/360398059 wait complete. 2024-03-06T07:38:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.079+0000 7f5836e00640 1 Processor -- start 2024-03-06T07:38:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.079+0000 7f5836e00640 1 -- start start 2024-03-06T07:38:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.080+0000 7f5836e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58180051c0 0x7f583015b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.080+0000 7f5836e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5830150d50 0x7f58300cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.080+0000 7f5836e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 0x7f5830157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.080+0000 7f5836e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583015f830 con 0x7f58301578f0 2024-03-06T07:38:41.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.080+0000 7f5836e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58300ae8b0 con 0x7f5830150d50 2024-03-06T07:38:41.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.080+0000 7f5836e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583015f9b0 con 0x7f58180051c0 2024-03-06T07:38:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 0x7f5830157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 0x7f5830157d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51012/0 (socket says 172.21.15.6:51012) 2024-03-06T07:38:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 -- 172.21.15.6:0/2106217297 learned_addr learned my addr 172.21.15.6:0/2106217297 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5835a00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58180051c0 0x7f583015b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5835000640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5830150d50 0x7f58300cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 -- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58180051c0 msgr2=0x7f583015b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58180051c0 0x7f583015b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 -- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5830150d50 msgr2=0x7f58300cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5830150d50 0x7f58300cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 -- 172.21.15.6:0/2106217297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583015bae0 con 0x7f58301578f0 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5835000640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5830150d50 0x7f58300cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.081+0000 7f5836400640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 0x7f5830157d70 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f582802cbd0 tx=0x7f5828001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.082+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828043070 con 0x7f58301578f0 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.082+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828001930 con 0x7f58301578f0 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.082+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583015bdc0 con 0x7f58301578f0 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.082+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828039040 con 0x7f58301578f0 2024-03-06T07:38:41.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.082+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5830151430 con 0x7f58301578f0 2024-03-06T07:38:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.083+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583004f330 con 0x7f58301578f0 2024-03-06T07:38:41.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.084+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5828046070 con 0x7f58301578f0 2024-03-06T07:38:41.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.085+0000 7f582d600640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f581006b4d0 0x7f581006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:41.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.085+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58280ab5f0 con 0x7f58301578f0 2024-03-06T07:38:41.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.086+0000 7f5835a00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f581006b4d0 0x7f581006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:41.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.087+0000 7f5835a00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f581006b4d0 0x7f581006d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5824009990 tx=0x7f5824008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:41.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.090+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f582807a420 con 0x7f58301578f0 2024-03-06T07:38:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.453+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f583006cc40 con 0x7f58301578f0 2024-03-06T07:38:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.453+0000 7f582d600640 1 -- 172.21.15.6:0/2106217297 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f5828079d20 con 0x7f58301578f0 2024-03-06T07:38:41.456 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:41.456 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f581006b4d0 msgr2=0x7f581006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f581006b4d0 0x7f581006d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5824009990 tx=0x7f5824008040 comp rx=0 tx=0).stop 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 msgr2=0x7f5830157d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 0x7f5830157d70 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f582802cbd0 tx=0x7f5828001cf0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 shutdown_connections 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f581006b4d0 0x7f581006d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5830150d50 0x7f58300cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58301578f0 0x7f5830157d70 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 --2- 172.21.15.6:0/2106217297 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f58180051c0 0x7f583015b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.456+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 >> 172.21.15.6:0/2106217297 conn(0x7f58300d1ae0 msgr2=0x7f5830158b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.457+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 shutdown_connections 2024-03-06T07:38:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:41.457+0000 7f5836e00640 1 -- 172.21.15.6:0/2106217297 wait complete. 2024-03-06T07:38:41.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:41.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:42.213 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:42.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:42.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.908+0000 7f2986a00640 1 Processor -- start 2024-03-06T07:38:42.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.908+0000 7f2986a00640 1 -- start start 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2986a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f298015e0b0 0x7f29800cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2986a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f29800d4a10 0x7f29800d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2986a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2986a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29800b09e0 con 0x7f29800d4a10 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2986a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29800b3710 con 0x7f298015e0b0 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2986a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29800b39f0 con 0x7f29800d0c80 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2984c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f29800d4a10 0x7f29800d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2984c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f29800d4a10 0x7f29800d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49940/0 (socket says 172.21.15.6:49940) 2024-03-06T07:38:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.910+0000 7f2984c00640 1 -- 172.21.15.6:0/4236545509 learned_addr learned my addr 172.21.15.6:0/4236545509 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2985600640 1 --2- 172.21.15.6:0/4236545509 >> v2:172.21.15.178:3300/0 conn(0x7f298015e0b0 0x7f29800cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2986000640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2984c00640 1 -- 172.21.15.6:0/4236545509 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 msgr2=0x7f29800d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2984c00640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2984c00640 1 -- 172.21.15.6:0/4236545509 >> v2:172.21.15.178:3300/0 conn(0x7f298015e0b0 msgr2=0x7f29800cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2984c00640 1 --2- 172.21.15.6:0/4236545509 >> v2:172.21.15.178:3300/0 conn(0x7f298015e0b0 0x7f29800cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2984c00640 1 -- 172.21.15.6:0/4236545509 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29800b3cd0 con 0x7f29800d4a10 2024-03-06T07:38:42.914 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.911+0000 7f2986000640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:42.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.912+0000 7f2984c00640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29800d4a10 0x7f29800d0740 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f29800b68c0 tx=0x7f296c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e3be7e5251e15a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.912+0000 7f297fe00640 1 -- 172.21.15.6:0/4236545509 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c035020 con 0x7f29800d4a10 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.912+0000 7f297fe00640 1 -- 172.21.15.6:0/4236545509 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296c031870 con 0x7f29800d4a10 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.912+0000 7f297fe00640 1 -- 172.21.15.6:0/4236545509 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296c031c40 con 0x7f29800d4a10 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 -- 172.21.15.6:0/4236545509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29800d4a10 msgr2=0x7f29800d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29800d4a10 0x7f29800d0740 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f29800b68c0 tx=0x7f296c02b910 comp rx=0 tx=0).stop 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 -- 172.21.15.6:0/4236545509 shutdown_connections 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 --2- 172.21.15.6:0/4236545509 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f29800d4a10 0x7f29800d0740 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 --2- 172.21.15.6:0/4236545509 >> v2:172.21.15.178:3300/0 conn(0x7f298015e0b0 0x7f29800cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 -- 172.21.15.6:0/4236545509 >> 172.21.15.6:0/4236545509 conn(0x7f29800d3d30 msgr2=0x7f29800d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 -- 172.21.15.6:0/4236545509 shutdown_connections 2024-03-06T07:38:42.916 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.913+0000 7f2986a00640 1 -- 172.21.15.6:0/4236545509 wait complete. 2024-03-06T07:38:42.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.916+0000 7f2986a00640 1 Processor -- start 2024-03-06T07:38:42.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.916+0000 7f2986a00640 1 -- start start 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.917+0000 7f2986a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.917+0000 7f2986a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29800d4a10 0x7f29800cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.917+0000 7f2986a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 0x7f29800cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.917+0000 7f2986a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29800cfc60 con 0x7f298015e0b0 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.917+0000 7f2986a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298015e490 con 0x7f29800d4a10 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.917+0000 7f2986a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29800cf850 con 0x7f29800d0c80 2024-03-06T07:38:42.921 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 0x7f29800cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 0x7f29800cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49952/0 (socket says 172.21.15.6:49952) 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 -- 172.21.15.6:0/4268738387 learned_addr learned my addr 172.21.15.6:0/4268738387 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2985600640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2984c00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29800d4a10 0x7f29800cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 -- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 msgr2=0x7f29800a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 -- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29800d4a10 msgr2=0x7f29800cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29800d4a10 0x7f29800cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.918+0000 7f2986000640 1 -- 172.21.15.6:0/4268738387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298014de20 con 0x7f298015e0b0 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.919+0000 7f2986000640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 0x7f29800cdb50 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f297800cf20 tx=0x7f297800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.919+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297801a070 con 0x7f298015e0b0 2024-03-06T07:38:42.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.919+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297800d840 con 0x7f298015e0b0 2024-03-06T07:38:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.919+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297800dc30 con 0x7f298015e0b0 2024-03-06T07:38:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.919+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298014e090 con 0x7f298015e0b0 2024-03-06T07:38:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.919+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298014e650 con 0x7f298015e0b0 2024-03-06T07:38:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.921+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298004f330 con 0x7f298015e0b0 2024-03-06T07:38:42.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.921+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f297801d070 con 0x7f298015e0b0 2024-03-06T07:38:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.922+0000 7f297d600640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f296006b4d0 0x7f296006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:42.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.922+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f297808aba0 con 0x7f298015e0b0 2024-03-06T07:38:42.925 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.923+0000 7f2985600640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f296006b4d0 0x7f296006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:42.926 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.924+0000 7f2985600640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f296006b4d0 0x7f296006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2974006fd0 tx=0x7f2974008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:42.929 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:42.927+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2978059940 con 0x7f298015e0b0 2024-03-06T07:38:43.177 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.175+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29800745e0 con 0x7f298015e0b0 2024-03-06T07:38:43.178 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.176+0000 7f297d600640 1 -- 172.21.15.6:0/4268738387 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f2978059240 con 0x7f298015e0b0 2024-03-06T07:38:43.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:43.179 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:43.179 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:43.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f296006b4d0 msgr2=0x7f296006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:43.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f296006b4d0 0x7f296006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2974006fd0 tx=0x7f2974008040 comp rx=0 tx=0).stop 2024-03-06T07:38:43.181 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 msgr2=0x7f29800cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 0x7f29800cdb50 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f297800cf20 tx=0x7f297800a7d0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 shutdown_connections 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f296006b4d0 0x7f296006d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f29800d4a10 0x7f29800cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f298015e0b0 0x7f29800cdb50 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 --2- 172.21.15.6:0/4268738387 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f29800d0c80 0x7f29800a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 >> 172.21.15.6:0/4268738387 conn(0x7f29800d3d30 msgr2=0x7f29800b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 shutdown_connections 2024-03-06T07:38:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:43.179+0000 7f2986a00640 1 -- 172.21.15.6:0/4268738387 wait complete. 2024-03-06T07:38:43.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:43.404 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:44.215 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:44.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:44.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.628+0000 7fb9b5c00640 1 Processor -- start 2024-03-06T07:38:44.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.628+0000 7fb9b5c00640 1 -- start start 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015dec0 0x7fb9b00ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5c00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fb9b015e2a0 0x7fb9b00ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9b00cf290 0x7fb9b00d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b00b0ea0 con 0x7fb9b00cf290 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5c00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b00cf720 con 0x7fb9b015e2a0 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b00d1120 con 0x7fb9b015dec0 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9b00cf290 0x7fb9b00d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fb9b00cf290 0x7fb9b00d1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49958/0 (socket says 172.21.15.6:49958) 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9b5200640 1 -- 172.21.15.6:0/693134370 learned_addr learned my addr 172.21.15.6:0/693134370 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9afe00640 1 --2- 172.21.15.6:0/693134370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015dec0 0x7fb9b00ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.630+0000 7fb9af400640 1 --2- 172.21.15.6:0/693134370 >> v2:172.21.15.178:3300/0 conn(0x7fb9b015e2a0 0x7fb9b00ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.633 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.631+0000 7fb9b5200640 1 -- 172.21.15.6:0/693134370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015dec0 msgr2=0x7fb9b00ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.631+0000 7fb9b5200640 1 --2- 172.21.15.6:0/693134370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015dec0 0x7fb9b00ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.631+0000 7fb9b5200640 1 -- 172.21.15.6:0/693134370 >> v2:172.21.15.178:3300/0 conn(0x7fb9b015e2a0 msgr2=0x7fb9b00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:44.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.631+0000 7fb9b5200640 1 --2- 172.21.15.6:0/693134370 >> v2:172.21.15.178:3300/0 conn(0x7fb9b015e2a0 0x7fb9b00ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.631+0000 7fb9b5200640 1 -- 172.21.15.6:0/693134370 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b00d1400 con 0x7fb9b00cf290 2024-03-06T07:38:44.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.632+0000 7fb9b5200640 1 --2- 172.21.15.6:0/693134370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b00d1b20 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fb9b008cab0 tx=0x7fb9a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b6fd34607723ece server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.632+0000 7fb9aea00640 1 -- 172.21.15.6:0/693134370 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a802ca70 con 0x7fb9b00cf290 2024-03-06T07:38:44.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.632+0000 7fb9aea00640 1 -- 172.21.15.6:0/693134370 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a802cc10 con 0x7fb9b00cf290 2024-03-06T07:38:44.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.632+0000 7fb9aea00640 1 -- 172.21.15.6:0/693134370 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a80343f0 con 0x7fb9b00cf290 2024-03-06T07:38:44.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 -- 172.21.15.6:0/693134370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 msgr2=0x7fb9b00d1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/693134370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b00d1b20 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fb9b008cab0 tx=0x7fb9a802bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 -- 172.21.15.6:0/693134370 shutdown_connections 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/693134370 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b00d1b20 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/693134370 >> v2:172.21.15.178:3300/0 conn(0x7fb9b015e2a0 0x7fb9b00ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/693134370 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015dec0 0x7fb9b00ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 -- 172.21.15.6:0/693134370 >> 172.21.15.6:0/693134370 conn(0x7fb9b00d3d40 msgr2=0x7fb9b00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 -- 172.21.15.6:0/693134370 shutdown_connections 2024-03-06T07:38:44.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.633+0000 7fb9b5c00640 1 -- 172.21.15.6:0/693134370 wait complete. 2024-03-06T07:38:44.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.635+0000 7fb9b5c00640 1 Processor -- start 2024-03-06T07:38:44.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.635+0000 7fb9b5c00640 1 -- start start 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9b5c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b015a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9b5c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b015dec0 0x7fb9b015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9b5c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015e2a0 0x7fb9b01601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9b5c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b00d0ec0 con 0x7fb9b00cf290 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9b5c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b00d0d40 con 0x7fb9b015dec0 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9b5c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b00d2060 con 0x7fb9b015e2a0 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b015a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9afe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b015a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49974/0 (socket says 172.21.15.6:49974) 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.636+0000 7fb9afe00640 1 -- 172.21.15.6:0/3080929171 learned_addr learned my addr 172.21.15.6:0/3080929171 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9af400640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b015dec0 0x7fb9b015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9b5200640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015e2a0 0x7fb9b01601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9afe00640 1 -- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015e2a0 msgr2=0x7fb9b01601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9afe00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015e2a0 0x7fb9b01601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9afe00640 1 -- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b015dec0 msgr2=0x7fb9b015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9afe00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b015dec0 0x7fb9b015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9afe00640 1 -- 172.21.15.6:0/3080929171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b015aba0 con 0x7fb9b00cf290 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9af400640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b015dec0 0x7fb9b015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9afe00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b015a660 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000d830 tx=0x7fb9a000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a000b6c0 con 0x7fb9b00cf290 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.637+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a0019070 con 0x7fb9b00cf290 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.638+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a000ba80 con 0x7fb9b00cf290 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.638+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b0161460 con 0x7fb9b00cf290 2024-03-06T07:38:44.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.638+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b00c7910 con 0x7fb9b00cf290 2024-03-06T07:38:44.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.639+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b004f330 con 0x7fb9b00cf290 2024-03-06T07:38:44.642 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.640+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9a001e460 con 0x7fb9b00cf290 2024-03-06T07:38:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.641+0000 7fb9a7e00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb99006b4d0 0x7fb99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.641+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb9a008a9a0 con 0x7fb9b00cf290 2024-03-06T07:38:44.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.641+0000 7fb9af400640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb99006b4d0 0x7fb99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:44.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.643+0000 7fb9af400640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb99006b4d0 0x7fb99006d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb99c006fd0 tx=0x7fb99c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:44.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.650+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9a0059740 con 0x7fb9b00cf290 2024-03-06T07:38:44.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.889+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9b015c860 con 0x7fb9b00cf290 2024-03-06T07:38:44.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.890+0000 7fb9a7e00640 1 -- 172.21.15.6:0/3080929171 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fb9a0059040 con 0x7fb9b00cf290 2024-03-06T07:38:44.892 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:44.892 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:44.892 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:44.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.892+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb99006b4d0 msgr2=0x7fb99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.892+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb99006b4d0 0x7fb99006d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fb99c006fd0 tx=0x7fb99c008040 comp rx=0 tx=0).stop 2024-03-06T07:38:44.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.892+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 msgr2=0x7fb9b015a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:44.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.892+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b015a660 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000d830 tx=0x7fb9a000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 shutdown_connections 2024-03-06T07:38:44.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fb99006b4d0 0x7fb99006d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb9b015dec0 0x7fb9b015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fb9b015e2a0 0x7fb9b01601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 --2- 172.21.15.6:0/3080929171 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb9b00cf290 0x7fb9b015a660 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:44.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 >> 172.21.15.6:0/3080929171 conn(0x7fb9b00d3d40 msgr2=0x7fb9b006ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:44.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 shutdown_connections 2024-03-06T07:38:44.896 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:44.893+0000 7fb9b5c00640 1 -- 172.21.15.6:0/3080929171 wait complete. 2024-03-06T07:38:45.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:45.115 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:46.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:46.217 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.342+0000 7f7237e00640 1 Processor -- start 2024-03-06T07:38:46.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.342+0000 7f7237e00640 1 -- start start 2024-03-06T07:38:46.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f723815efc0 0x7f72380cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:46.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f72380cce90 0x7f7238161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:46.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72381627f0 con 0x7f7238162000 2024-03-06T07:38:46.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723815f3a0 con 0x7f723815efc0 2024-03-06T07:38:46.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723806d190 con 0x7f72380cce90 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7238162000 0x7f72380ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53936/0 (socket says 172.21.15.6:53936) 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7236a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f723815efc0 0x7f72380cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.344+0000 7f7237400640 1 -- 172.21.15.6:0/1001501817 learned_addr learned my addr 172.21.15.6:0/1001501817 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.345+0000 7f7236000640 1 --2- 172.21.15.6:0/1001501817 >> v2:172.21.15.178:3300/0 conn(0x7f72380cce90 0x7f7238161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.345+0000 7f7236a00640 1 -- 172.21.15.6:0/1001501817 >> v2:172.21.15.178:3300/0 conn(0x7f72380cce90 msgr2=0x7f7238161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.345+0000 7f7236a00640 1 --2- 172.21.15.6:0/1001501817 >> v2:172.21.15.178:3300/0 conn(0x7f72380cce90 0x7f7238161ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.345+0000 7f7236a00640 1 -- 172.21.15.6:0/1001501817 >> v1:172.21.15.6:6789/0 conn(0x7f7238162000 legacy=0x7f72380ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.345+0000 7f7236a00640 1 -- 172.21.15.6:0/1001501817 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723806c860 con 0x7f723815efc0 2024-03-06T07:38:46.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.346+0000 7f7236a00640 1 --2- 172.21.15.6:0/1001501817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f72380cc950 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f72380b51d0 tx=0x7f722802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ac8f952f5bbf26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:46.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.346+0000 7f7235600640 1 -- 172.21.15.6:0/1001501817 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228035020 con 0x7f723815efc0 2024-03-06T07:38:46.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.346+0000 7f7235600640 1 -- 172.21.15.6:0/1001501817 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7228031830 con 0x7f723815efc0 2024-03-06T07:38:46.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.346+0000 7f7235600640 1 -- 172.21.15.6:0/1001501817 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228031c00 con 0x7f723815efc0 2024-03-06T07:38:46.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 -- 172.21.15.6:0/1001501817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 msgr2=0x7f72380cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 --2- 172.21.15.6:0/1001501817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f72380cc950 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f72380b51d0 tx=0x7f722802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 -- 172.21.15.6:0/1001501817 shutdown_connections 2024-03-06T07:38:46.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 --2- 172.21.15.6:0/1001501817 >> v2:172.21.15.178:3300/0 conn(0x7f72380cce90 0x7f7238161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 --2- 172.21.15.6:0/1001501817 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f72380cc950 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 -- 172.21.15.6:0/1001501817 >> 172.21.15.6:0/1001501817 conn(0x7f723806b3d0 msgr2=0x7f723806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:46.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 -- 172.21.15.6:0/1001501817 shutdown_connections 2024-03-06T07:38:46.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.347+0000 7f7237e00640 1 -- 172.21.15.6:0/1001501817 wait complete. 2024-03-06T07:38:46.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.349+0000 7f7237e00640 1 Processor -- start 2024-03-06T07:38:46.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.349+0000 7f7237e00640 1 -- start start 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7237e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72380cce90 0x7f7238143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7237e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f7238145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7237e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7238162000 0x7f7238140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7237e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7238071c30 con 0x7f723815efc0 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7237e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723806ce20 con 0x7f7238162000 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7237e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7238162470 con 0x7f72380cce90 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7236000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f7238145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7236000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f7238145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49984/0 (socket says 172.21.15.6:49984) 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7236000640 1 -- 172.21.15.6:0/1721289035 learned_addr learned my addr 172.21.15.6:0/1721289035 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.351+0000 7f7236a00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72380cce90 0x7f7238143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:46.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7237400640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7238162000 0x7f7238140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7236000640 1 -- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72380cce90 msgr2=0x7f7238143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7236000640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72380cce90 0x7f7238143060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7236000640 1 -- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7238162000 msgr2=0x7f7238140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7236000640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7238162000 0x7f7238140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7236000640 1 -- 172.21.15.6:0/1721289035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72381458b0 con 0x7f723815efc0 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7237400640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7238162000 0x7f7238140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:46.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f7236000640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f7238145160 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f722000d800 tx=0x7f722000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:46.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722000b6c0 con 0x7f723815efc0 2024-03-06T07:38:46.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7220019070 con 0x7f723815efc0 2024-03-06T07:38:46.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.352+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722000ba80 con 0x7f723815efc0 2024-03-06T07:38:46.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.353+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7238145b90 con 0x7f723815efc0 2024-03-06T07:38:46.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.353+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7238146150 con 0x7f723815efc0 2024-03-06T07:38:46.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.354+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723804f330 con 0x7f723815efc0 2024-03-06T07:38:46.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.356+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f722000bc20 con 0x7f723815efc0 2024-03-06T07:38:46.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.357+0000 7f722ea00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f721406b4d0 0x7f721406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:46.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.357+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f722008a730 con 0x7f723815efc0 2024-03-06T07:38:46.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.357+0000 7f7236a00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f721406b4d0 0x7f721406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.359+0000 7f7236a00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f721406b4d0 0x7f721406d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7228036000 tx=0x7f7228008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:46.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.366+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7220059560 con 0x7f723815efc0 2024-03-06T07:38:46.616 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.614+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72380af2f0 con 0x7f723815efc0 2024-03-06T07:38:46.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.615+0000 7f722ea00640 1 -- 172.21.15.6:0/1721289035 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f7220058e60 con 0x7f723815efc0 2024-03-06T07:38:46.618 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:46.618 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:46.618 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:46.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f721406b4d0 msgr2=0x7f721406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f721406b4d0 0x7f721406d990 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7228036000 tx=0x7f7228008030 comp rx=0 tx=0).stop 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 msgr2=0x7f7238145160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f7238145160 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f722000d800 tx=0x7f722000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 shutdown_connections 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f721406b4d0 0x7f721406d990 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7238162000 0x7f7238140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f723815efc0 0x7f7238145160 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 --2- 172.21.15.6:0/1721289035 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f72380cce90 0x7f7238143060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 >> 172.21.15.6:0/1721289035 conn(0x7f723806b3d0 msgr2=0x7f72380ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 shutdown_connections 2024-03-06T07:38:46.621 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:46.618+0000 7f7237e00640 1 -- 172.21.15.6:0/1721289035 wait complete. 2024-03-06T07:38:46.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:46.840 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:47.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:48.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.066+0000 7fa048400640 1 Processor -- start 2024-03-06T07:38:48.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.066+0000 7fa048400640 1 -- start start 2024-03-06T07:38:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.068+0000 7fa048400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 0x7fa04015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.068+0000 7fa048400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04015cc40 con 0x7fa0400cdc20 2024-03-06T07:38:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.068+0000 7fa048400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04015cf20 con 0x7fa0400af1a0 2024-03-06T07:38:48.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.068+0000 7fa048400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015d200 con 0x7fa0400af580 2024-03-06T07:38:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.068+0000 7fa046400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa0400cdc20 0x7fa0400b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53952/0 (socket says 172.21.15.6:53952) 2024-03-06T07:38:48.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.068+0000 7fa046400640 1 -- 172.21.15.6:0/220520933 learned_addr learned my addr 172.21.15.6:0/220520933 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:48.072 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa045000640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 0x7fa04015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277221008 0 0) 0x7fa04015cc40 con 0x7fa0400cdc20 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa028003520 con 0x7fa0400cdc20 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604377316 0 0) 0x7fa028003520 con 0x7fa0400cdc20 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 msgr2=0x7fa04015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 0x7fa04015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 >> v1:172.21.15.178:6789/0 conn(0x7fa0400af1a0 legacy=0x7fa0400b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04015d4e0 con 0x7fa0400cdc20 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fa0380019f0 con 0x7fa0400cdc20 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.069+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 >> v1:172.21.15.6:6789/0 conn(0x7fa0400cdc20 legacy=0x7fa0400b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa03fe00640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 0x7fa028003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa03fe00640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa028003f90 0x7fa028004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa03fe00640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa028004d40 0x7fa0280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015d200 con 0x7fa028004d40 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02800bb80 con 0x7fa028003f90 2024-03-06T07:38:48.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02800bd80 con 0x7fa0400af580 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa046400640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa028004d40 0x7fa0280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa045a00640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 0x7fa028003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa045000640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa028003f90 0x7fa028004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa046400640 1 -- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 msgr2=0x7fa028003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa046400640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa0400af580 0x7fa028003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa046400640 1 -- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa028003f90 msgr2=0x7fa028004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa046400640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa028003f90 0x7fa028004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.070+0000 7fa046400640 1 -- 172.21.15.6:0/220520933 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa028003520 con 0x7fa028004d40 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa046400640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa028004d40 0x7fa0280051c0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fa03800a7a0 tx=0x7fa028005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3c3af01b959dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03800a3f0 con 0x7fa028004d40 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa038034650 con 0x7fa028004d40 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa03fe00640 1 -- 172.21.15.6:0/220520933 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa038034a80 con 0x7fa028004d40 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa048400640 1 -- 172.21.15.6:0/220520933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa028004d40 msgr2=0x7fa0280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa048400640 1 --2- 172.21.15.6:0/220520933 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa028004d40 0x7fa0280051c0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fa03800a7a0 tx=0x7fa028005830 comp rx=0 tx=0).stop 2024-03-06T07:38:48.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.072+0000 7fa046400640 1 -- 172.21.15.6:0/220520933 reap_dead start 2024-03-06T07:38:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.073+0000 7fa048400640 1 -- 172.21.15.6:0/220520933 shutdown_connections 2024-03-06T07:38:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.073+0000 7fa048400640 1 -- 172.21.15.6:0/220520933 >> 172.21.15.6:0/220520933 conn(0x7fa04006afb0 msgr2=0x7fa04006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.073+0000 7fa048400640 1 -- 172.21.15.6:0/220520933 shutdown_connections 2024-03-06T07:38:48.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.073+0000 7fa048400640 1 -- 172.21.15.6:0/220520933 wait complete. 2024-03-06T07:38:48.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.076+0000 7fa048400640 1 Processor -- start 2024-03-06T07:38:48.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.076+0000 7fa048400640 1 -- start start 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa048400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa028004d40 0x7fa0401564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa048400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa040156bc0 0x7fa04009be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa048400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 0x7fa04015d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa048400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015c600 con 0x7fa04009b620 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa048400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0400b36f0 con 0x7fa040156bc0 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa048400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04015c780 con 0x7fa028004d40 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 0x7fa04015d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 0x7fa04015d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50014/0 (socket says 172.21.15.6:50014) 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 -- 172.21.15.6:0/111110432 learned_addr learned my addr 172.21.15.6:0/111110432 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa045a00640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa028004d40 0x7fa0401564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.080 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa045000640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa040156bc0 0x7fa04009be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 -- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa028004d40 msgr2=0x7fa0401564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa028004d40 0x7fa0401564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 -- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa040156bc0 msgr2=0x7fa04009be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.077+0000 7fa046400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa040156bc0 0x7fa04009be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa046400640 1 -- 172.21.15.6:0/111110432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04009c350 con 0x7fa04009b620 2024-03-06T07:38:48.081 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa046400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 0x7fa04015d700 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa03800a8c0 tx=0x7fa038001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa038045070 con 0x7fa04009b620 2024-03-06T07:38:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03803a040 con 0x7fa04009b620 2024-03-06T07:38:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa038034a00 con 0x7fa04009b620 2024-03-06T07:38:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04009c5f0 con 0x7fa04009b620 2024-03-06T07:38:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.078+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04009ae20 con 0x7fa04009b620 2024-03-06T07:38:48.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.080+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa040069510 con 0x7fa04009b620 2024-03-06T07:38:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.081+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa038048070 con 0x7fa04009b620 2024-03-06T07:38:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.081+0000 7fa03d600640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa02006b500 0x7fa02006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:48.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.082+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0380abdb0 con 0x7fa04009b620 2024-03-06T07:38:48.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.082+0000 7fa045a00640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa02006b500 0x7fa02006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:48.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.083+0000 7fa045a00640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa02006b500 0x7fa02006d9c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa034005540 tx=0x7fa034007c50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:48.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.086+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa0380b0070 con 0x7fa04009b620 2024-03-06T07:38:48.219 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:48.341 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.339+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa04004f3a0 con 0x7fa04009b620 2024-03-06T07:38:48.342 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.340+0000 7fa03d600640 1 -- 172.21.15.6:0/111110432 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fa03807a590 con 0x7fa04009b620 2024-03-06T07:38:48.342 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:48.342 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:48.343 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.342+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa02006b500 msgr2=0x7fa02006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.342+0000 7fa048400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa02006b500 0x7fa02006d9c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa034005540 tx=0x7fa034007c50 comp rx=0 tx=0).stop 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.342+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 msgr2=0x7fa04015d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.342+0000 7fa048400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 0x7fa04015d700 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa03800a8c0 tx=0x7fa038001cf0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 shutdown_connections 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fa02006b500 0x7fa02006d9c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa040156bc0 0x7fa04009be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa04009b620 0x7fa04015d700 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 --2- 172.21.15.6:0/111110432 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fa028004d40 0x7fa0401564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:48.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 >> 172.21.15.6:0/111110432 conn(0x7fa04006afb0 msgr2=0x7fa0401578b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:48.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 shutdown_connections 2024-03-06T07:38:48.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:48.343+0000 7fa048400640 1 -- 172.21.15.6:0/111110432 wait complete. 2024-03-06T07:38:48.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:48.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:49.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:49.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.796+0000 7fc17aa00640 1 Processor -- start 2024-03-06T07:38:49.798 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.796+0000 7fc17aa00640 1 -- start start 2024-03-06T07:38:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc17aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1740af1a0 0x7fc17415c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc17aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af580 0x7fc1740b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc17aa00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17415cc40 con 0x7fc1740cdc20 2024-03-06T07:38:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc17aa00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc17415cf20 con 0x7fc1740af1a0 2024-03-06T07:38:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc17aa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc17415d200 con 0x7fc1740af580 2024-03-06T07:38:49.800 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc173e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1740af1a0 0x7fc17415c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc173e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc1740af1a0 0x7fc17415c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50028/0 (socket says 172.21.15.6:50028) 2024-03-06T07:38:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc173e00640 1 -- 172.21.15.6:0/4053558071 learned_addr learned my addr 172.21.15.6:0/4053558071 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.798+0000 7fc16b400640 1 --2- 172.21.15.6:0/4053558071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af580 0x7fc1740b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.799+0000 7fc173e00640 1 -- 172.21.15.6:0/4053558071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af580 msgr2=0x7fc1740b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.799+0000 7fc173e00640 1 --2- 172.21.15.6:0/4053558071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af580 0x7fc1740b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.801 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.799+0000 7fc173e00640 1 -- 172.21.15.6:0/4053558071 >> v1:172.21.15.178:6789/0 conn(0x7fc1740cdc20 legacy=0x7fc1740b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.802 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.799+0000 7fc173e00640 1 -- 172.21.15.6:0/4053558071 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17415d4e0 con 0x7fc1740af1a0 2024-03-06T07:38:49.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.801+0000 7fc173e00640 1 --2- 172.21.15.6:0/4053558071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740af1a0 0x7fc17415c600 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc164009870 tx=0x7fc16402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=979538a52d20f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.803 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.801+0000 7fc173400640 1 -- 172.21.15.6:0/4053558071 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc164035020 con 0x7fc1740af1a0 2024-03-06T07:38:49.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.802+0000 7fc173400640 1 -- 172.21.15.6:0/4053558071 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc164031640 con 0x7fc1740af1a0 2024-03-06T07:38:49.804 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.802+0000 7fc173400640 1 -- 172.21.15.6:0/4053558071 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16402d040 con 0x7fc1740af1a0 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.802+0000 7fc17aa00640 1 -- 172.21.15.6:0/4053558071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740af1a0 msgr2=0x7fc17415c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.802+0000 7fc17aa00640 1 --2- 172.21.15.6:0/4053558071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740af1a0 0x7fc17415c600 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc164009870 tx=0x7fc16402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.803+0000 7fc17aa00640 1 -- 172.21.15.6:0/4053558071 shutdown_connections 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.803+0000 7fc17aa00640 1 --2- 172.21.15.6:0/4053558071 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af580 0x7fc1740b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.803+0000 7fc17aa00640 1 --2- 172.21.15.6:0/4053558071 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740af1a0 0x7fc17415c600 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.803+0000 7fc17aa00640 1 -- 172.21.15.6:0/4053558071 >> 172.21.15.6:0/4053558071 conn(0x7fc17406afb0 msgr2=0x7fc17406b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.803+0000 7fc17aa00640 1 -- 172.21.15.6:0/4053558071 shutdown_connections 2024-03-06T07:38:49.805 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.803+0000 7fc17aa00640 1 -- 172.21.15.6:0/4053558071 wait complete. 2024-03-06T07:38:49.806 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.805+0000 7fc17aa00640 1 Processor -- start 2024-03-06T07:38:49.807 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.805+0000 7fc17aa00640 1 -- start start 2024-03-06T07:38:49.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc17aa00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af1a0 0x7fc17414ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc17aa00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1740af580 0x7fc1740ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.808 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc17aa00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 0x7fc1740ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc17aa00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740b3660 con 0x7fc1740cdc20 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc17aa00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740b34e0 con 0x7fc1740af580 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc17aa00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1740b37e0 con 0x7fc1740af1a0 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc178a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 0x7fc1740ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc178a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 0x7fc1740ce640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50042/0 (socket says 172.21.15.6:50042) 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc178a00640 1 -- 172.21.15.6:0/2688243838 learned_addr learned my addr 172.21.15.6:0/2688243838 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc173e00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af1a0 0x7fc17414ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.809 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.806+0000 7fc16b400640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1740af580 0x7fc1740ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc178a00640 1 -- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af1a0 msgr2=0x7fc17414ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc178a00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af1a0 0x7fc17414ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc178a00640 1 -- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1740af580 msgr2=0x7fc1740ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc178a00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1740af580 0x7fc1740ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc178a00640 1 -- 172.21.15.6:0/2688243838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17414f460 con 0x7fc1740cdc20 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc173e00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af1a0 0x7fc17414ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc178a00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 0x7fc1740ce640 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00c6b0 tx=0x7fc16c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.810 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.807+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c003b70 con 0x7fc1740cdc20 2024-03-06T07:38:49.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.808+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16c003d30 con 0x7fc1740cdc20 2024-03-06T07:38:49.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.808+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16c005310 con 0x7fc1740cdc20 2024-03-06T07:38:49.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.808+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17414f740 con 0x7fc1740cdc20 2024-03-06T07:38:49.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.808+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17414fd00 con 0x7fc1740cdc20 2024-03-06T07:38:49.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.809+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc174069510 con 0x7fc1740cdc20 2024-03-06T07:38:49.812 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.810+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc16c0054b0 con 0x7fc1740cdc20 2024-03-06T07:38:49.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.811+0000 7fc170c00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:49.813 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.811+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc16c08a140 con 0x7fc1740cdc20 2024-03-06T07:38:49.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.812+0000 7fc173e00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:49.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.813+0000 7fc173e00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc164036000 tx=0x7fc16403c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:49.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:49.816+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc16c058ee0 con 0x7fc1740cdc20 2024-03-06T07:38:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.059+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc17404f3a0 con 0x7fc1740cdc20 2024-03-06T07:38:50.061 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.059+0000 7fc170c00640 1 -- 172.21.15.6:0/2688243838 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fc16c0587e0 con 0x7fc1740cdc20 2024-03-06T07:38:50.062 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:50.062 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:50.062 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 msgr2=0x7fc15006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc164036000 tx=0x7fc16403c040 comp rx=0 tx=0).stop 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 msgr2=0x7fc1740ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 0x7fc1740ce640 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00c6b0 tx=0x7fc16c00cb70 comp rx=0 tx=0).stop 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 shutdown_connections 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc15006b4d0 0x7fc15006d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc1740af580 0x7fc1740ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc1740cdc20 0x7fc1740ce640 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 --2- 172.21.15.6:0/2688243838 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc1740af1a0 0x7fc17414ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:50.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 >> 172.21.15.6:0/2688243838 conn(0x7fc17406afb0 msgr2=0x7fc1740adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:50.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 shutdown_connections 2024-03-06T07:38:50.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:50.064+0000 7fc17aa00640 1 -- 172.21.15.6:0/2688243838 wait complete. 2024-03-06T07:38:50.221 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:50.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:50.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:51.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:51.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.512+0000 7ff698a00640 1 Processor -- start 2024-03-06T07:38:51.514 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.512+0000 7ff698a00640 1 -- start start 2024-03-06T07:38:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.513+0000 7ff698a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 0x7ff694161c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff698a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6940cff30 con 0x7ff694158460 2024-03-06T07:38:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff698a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6941626d0 con 0x7ff694162150 2024-03-06T07:38:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff698a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941629b0 con 0x7ff694158840 2024-03-06T07:38:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff692800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff694158460 0x7ff6940d09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53966/0 (socket says 172.21.15.6:53966) 2024-03-06T07:38:51.516 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff692800640 1 -- 172.21.15.6:0/2298754043 learned_addr learned my addr 172.21.15.6:0/2298754043 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff691e00640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 0x7ff694161c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533858527 0 0) 0x7ff6940cff30 con 0x7ff694158460 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.514+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff678003520 con 0x7ff694158460 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451397114 0 0) 0x7ff678003520 con 0x7ff694158460 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 msgr2=0x7ff694161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 0x7ff694161c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 >> v1:172.21.15.178:6789/0 conn(0x7ff694162150 legacy=0x7ff6940b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.518 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6940d01d0 con 0x7ff694158460 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7ff684001670 con 0x7ff694158460 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 >> v1:172.21.15.6:6789/0 conn(0x7ff694158460 legacy=0x7ff6940d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 0x7ff6780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678003f60 0x7ff678004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff678004d40 0x7ff6780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941629b0 con 0x7ff678004d40 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67800bb50 con 0x7ff678003f60 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67800bd50 con 0x7ff694158840 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.515+0000 7ff693200640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff678004d40 0x7ff6780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff691e00640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678003f60 0x7ff678004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff692800640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 0x7ff6780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff693200640 1 -- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 msgr2=0x7ff6780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff693200640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694158840 0x7ff6780039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff693200640 1 -- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678003f60 msgr2=0x7ff678004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:51.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff693200640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678003f60 0x7ff678004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.520 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.516+0000 7ff693200640 1 -- 172.21.15.6:0/2298754043 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678003520 con 0x7ff678004d40 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.517+0000 7ff693200640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff678004d40 0x7ff6780051c0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff68c001e50 tx=0x7ff678005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35231f4555285465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.517+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c001a70 con 0x7ff678004d40 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.517+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c003b70 con 0x7ff678004d40 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.517+0000 7ff691400640 1 -- 172.21.15.6:0/2298754043 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c0050d0 con 0x7ff678004d40 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff698a00640 1 -- 172.21.15.6:0/2298754043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff678004d40 msgr2=0x7ff6780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff698a00640 1 --2- 172.21.15.6:0/2298754043 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff678004d40 0x7ff6780051c0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff68c001e50 tx=0x7ff678005830 comp rx=0 tx=0).stop 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff693200640 1 -- 172.21.15.6:0/2298754043 reap_dead start 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff698a00640 1 -- 172.21.15.6:0/2298754043 shutdown_connections 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff698a00640 1 -- 172.21.15.6:0/2298754043 >> 172.21.15.6:0/2298754043 conn(0x7ff6940b5100 msgr2=0x7ff6940b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff698a00640 1 -- 172.21.15.6:0/2298754043 shutdown_connections 2024-03-06T07:38:51.521 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.518+0000 7ff698a00640 1 -- 172.21.15.6:0/2298754043 wait complete. 2024-03-06T07:38:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.520+0000 7ff698a00640 1 Processor -- start 2024-03-06T07:38:51.522 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.520+0000 7ff698a00640 1 -- start start 2024-03-06T07:38:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff698a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678004d40 0x7ff69409d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff698a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694142e50 0x7ff6940aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff698a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 0x7ff6940ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff698a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940b1a20 con 0x7ff6940af370 2024-03-06T07:38:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff698a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69415e8c0 con 0x7ff678004d40 2024-03-06T07:38:51.523 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff698a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940b1ba0 con 0x7ff694142e50 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff693200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 0x7ff6940ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff693200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 0x7ff6940ae1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50066/0 (socket says 172.21.15.6:50066) 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff693200640 1 -- 172.21.15.6:0/3714450410 learned_addr learned my addr 172.21.15.6:0/3714450410 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff692800640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678004d40 0x7ff69409d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.521+0000 7ff691e00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694142e50 0x7ff6940aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.522+0000 7ff693200640 1 -- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694142e50 msgr2=0x7ff6940aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.524 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.522+0000 7ff693200640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694142e50 0x7ff6940aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.522+0000 7ff693200640 1 -- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678004d40 msgr2=0x7ff69409d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.522+0000 7ff693200640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678004d40 0x7ff69409d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.522+0000 7ff693200640 1 -- 172.21.15.6:0/3714450410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69409de30 con 0x7ff6940af370 2024-03-06T07:38:51.525 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.522+0000 7ff693200640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 0x7ff6940ae1b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02c060 tx=0x7ff68c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.523+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c007db0 con 0x7ff6940af370 2024-03-06T07:38:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.523+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c003970 con 0x7ff6940af370 2024-03-06T07:38:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.523+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c02f660 con 0x7ff6940af370 2024-03-06T07:38:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.523+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff694150270 con 0x7ff6940af370 2024-03-06T07:38:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.523+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff694150830 con 0x7ff6940af370 2024-03-06T07:38:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.524+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69404f330 con 0x7ff6940af370 2024-03-06T07:38:51.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.526+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff68c02f800 con 0x7ff6940af370 2024-03-06T07:38:51.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.527+0000 7ff68aa00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b500 0x7ff67406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:51.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.527+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff68c0ab930 con 0x7ff6940af370 2024-03-06T07:38:51.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.527+0000 7ff692800640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b500 0x7ff67406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:51.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.529+0000 7ff692800640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b500 0x7ff67406d9c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff684001cd0 tx=0x7ff684007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:51.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.537+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff69409de30 con 0x7ff6940af370 2024-03-06T07:38:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.784+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff694071570 con 0x7ff6940af370 2024-03-06T07:38:51.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.785+0000 7ff68aa00640 1 -- 172.21.15.6:0/3714450410 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7ff68c033020 con 0x7ff6940af370 2024-03-06T07:38:51.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:51.788 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:51.788 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:51.790 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b500 msgr2=0x7ff67406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b500 0x7ff67406d9c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff684001cd0 tx=0x7ff684007040 comp rx=0 tx=0).stop 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 msgr2=0x7ff6940ae1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 0x7ff6940ae1b0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff68c02c060 tx=0x7ff68c0014a0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 shutdown_connections 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ff67406b500 0x7ff67406d9c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff6940af370 0x7ff6940ae1b0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ff694142e50 0x7ff6940aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 --2- 172.21.15.6:0/3714450410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff678004d40 0x7ff69409d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.788+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 >> 172.21.15.6:0/3714450410 conn(0x7ff6940b5100 msgr2=0x7ff6940b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.789+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 shutdown_connections 2024-03-06T07:38:51.791 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:51.789+0000 7ff698a00640 1 -- 172.21.15.6:0/3714450410 wait complete. 2024-03-06T07:38:52.009 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:52.010 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:52.223 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:53.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:53.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.233+0000 7f410fe00640 1 Processor -- start 2024-03-06T07:38:53.236 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.234+0000 7f410fe00640 1 -- start start 2024-03-06T07:38:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.236+0000 7f410fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f411006c8b0 0x7f411006ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410fe00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f4110066d00 0x7f411015ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410fe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41100ce370 0x7f41100ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410fe00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411015e470 con 0x7f41100ce370 2024-03-06T07:38:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410fe00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411015ea00 con 0x7f4110066d00 2024-03-06T07:38:53.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100cf1b0 con 0x7f411006c8b0 2024-03-06T07:38:53.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41100ce370 0x7f41100ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410f400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f41100ce370 0x7f41100ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51610/0 (socket says 172.21.15.6:51610) 2024-03-06T07:38:53.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410f400640 1 -- 172.21.15.6:0/2267055284 learned_addr learned my addr 172.21.15.6:0/2267055284 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:53.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f410ea00640 1 --2- 172.21.15.6:0/2267055284 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f411006c8b0 0x7f411006ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.240 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.237+0000 7f4107e00640 1 --2- 172.21.15.6:0/2267055284 >> v2:172.21.15.178:3300/0 conn(0x7f4110066d00 0x7f411015ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.238+0000 7f410f400640 1 -- 172.21.15.6:0/2267055284 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f411006c8b0 msgr2=0x7f411006ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.238+0000 7f410f400640 1 --2- 172.21.15.6:0/2267055284 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f411006c8b0 0x7f411006ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.238+0000 7f410f400640 1 -- 172.21.15.6:0/2267055284 >> v2:172.21.15.178:3300/0 conn(0x7f4110066d00 msgr2=0x7f411015ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.238+0000 7f410f400640 1 --2- 172.21.15.6:0/2267055284 >> v2:172.21.15.178:3300/0 conn(0x7f4110066d00 0x7f411015ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.238+0000 7f410f400640 1 -- 172.21.15.6:0/2267055284 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411015f3d0 con 0x7f41100ce370 2024-03-06T07:38:53.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.239+0000 7f410f400640 1 --2- 172.21.15.6:0/2267055284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41100ce370 0x7f41100ce7c0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f411006d2c0 tx=0x7f410802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd347fddced8f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:53.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.239+0000 7f410e000640 1 -- 172.21.15.6:0/2267055284 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410802ca70 con 0x7f41100ce370 2024-03-06T07:38:53.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.239+0000 7f410e000640 1 -- 172.21.15.6:0/2267055284 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410802cc10 con 0x7f41100ce370 2024-03-06T07:38:53.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.239+0000 7f410e000640 1 -- 172.21.15.6:0/2267055284 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41080343f0 con 0x7f41100ce370 2024-03-06T07:38:53.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 -- 172.21.15.6:0/2267055284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41100ce370 msgr2=0x7f41100ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 --2- 172.21.15.6:0/2267055284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41100ce370 0x7f41100ce7c0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f411006d2c0 tx=0x7f410802bed0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 -- 172.21.15.6:0/2267055284 shutdown_connections 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 --2- 172.21.15.6:0/2267055284 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f41100ce370 0x7f41100ce7c0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 --2- 172.21.15.6:0/2267055284 >> v2:172.21.15.178:3300/0 conn(0x7f4110066d00 0x7f411015ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 --2- 172.21.15.6:0/2267055284 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f411006c8b0 0x7f411006ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 -- 172.21.15.6:0/2267055284 >> 172.21.15.6:0/2267055284 conn(0x7f41100ccf70 msgr2=0x7f41100cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 -- 172.21.15.6:0/2267055284 shutdown_connections 2024-03-06T07:38:53.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.240+0000 7f410fe00640 1 -- 172.21.15.6:0/2267055284 wait complete. 2024-03-06T07:38:53.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.242+0000 7f410fe00640 1 Processor -- start 2024-03-06T07:38:53.244 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.242+0000 7f410fe00640 1 -- start start 2024-03-06T07:38:53.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 0x7f41100afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.245 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410fe00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411006c8b0 0x7f41100ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410fe00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41100ce370 0x7f4110143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410fe00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100ced00 con 0x7f4110066d00 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410fe00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411015e690 con 0x7f411006c8b0 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410fe00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100cee80 con 0x7f41100ce370 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.243+0000 7f410ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 0x7f41100afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 0x7f41100afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51612/0 (socket says 172.21.15.6:51612) 2024-03-06T07:38:53.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 -- 172.21.15.6:0/2220404883 learned_addr learned my addr 172.21.15.6:0/2220404883 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:53.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f4107e00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411006c8b0 0x7f41100ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 -- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41100ce370 msgr2=0x7f4110143d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410f400640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41100ce370 0x7f4110143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41100ce370 0x7f4110143d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 -- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411006c8b0 msgr2=0x7f41100ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411006c8b0 0x7f41100ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 -- 172.21.15.6:0/2220404883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41100b3ed0 con 0x7f4110066d00 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f4107e00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411006c8b0 0x7f41100ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410f400640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41100ce370 0x7f4110143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.244+0000 7f410ea00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 0x7f41100afa90 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f410000d5f0 tx=0x7f410000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.245+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000125f0 con 0x7f4110066d00 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.245+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4100019070 con 0x7f4110066d00 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.245+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41100b41b0 con 0x7f4110066d00 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.245+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000129d0 con 0x7f4110066d00 2024-03-06T07:38:53.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.245+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41100b4770 con 0x7f4110066d00 2024-03-06T07:38:53.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.246+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411004f330 con 0x7f4110066d00 2024-03-06T07:38:53.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.248+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4100012b70 con 0x7f4110066d00 2024-03-06T07:38:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.249+0000 7f4106a00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40ec06b500 0x7f40ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.249+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f410008a6a0 con 0x7f4110066d00 2024-03-06T07:38:53.251 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.249+0000 7f4107e00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40ec06b500 0x7f40ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:53.252 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.251+0000 7f4107e00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40ec06b500 0x7f40ec06d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f40f8006fd0 tx=0x7f40f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:53.259 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.258+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f41000593b0 con 0x7f4110066d00 2024-03-06T07:38:53.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.499+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41100cdde0 con 0x7f4110066d00 2024-03-06T07:38:53.502 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.500+0000 7f4106a00640 1 -- 172.21.15.6:0/2220404883 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f4100058cb0 con 0x7f4110066d00 2024-03-06T07:38:53.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:53.502 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:53.503 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.502+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40ec06b500 msgr2=0x7f40ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.502+0000 7f410fe00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40ec06b500 0x7f40ec06d9c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f40f8006fd0 tx=0x7f40f8008040 comp rx=0 tx=0).stop 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.502+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 msgr2=0x7f41100afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.502+0000 7f410fe00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 0x7f41100afa90 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f410000d5f0 tx=0x7f410000dab0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 shutdown_connections 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f40ec06b500 0x7f40ec06d9c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f411006c8b0 0x7f41100ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f41100ce370 0x7f4110143d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 --2- 172.21.15.6:0/2220404883 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4110066d00 0x7f41100afa90 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 >> 172.21.15.6:0/2220404883 conn(0x7f41100ccf70 msgr2=0x7f411006afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 shutdown_connections 2024-03-06T07:38:53.506 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:53.503+0000 7f410fe00640 1 -- 172.21.15.6:0/2220404883 wait complete. 2024-03-06T07:38:53.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:53.725 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:54.225 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:54.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:54.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:54.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:38:54.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.949+0000 7f37f0a00640 1 Processor -- start 2024-03-06T07:38:54.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.949+0000 7f37f0a00640 1 -- start start 2024-03-06T07:38:54.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37f0a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37ec0af580 0x7f37ec15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37f0a00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec15cc40 con 0x7f37ec0cdc20 2024-03-06T07:38:54.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37f0a00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec15cf20 con 0x7f37ec0af1a0 2024-03-06T07:38:54.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37f0a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ec15d200 con 0x7f37ec0af580 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37ea800640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f37ec0cdc20 0x7f37ec0b2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58584/0 (socket says 172.21.15.6:58584) 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37e9400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f37ec0af580 0x7f37ec15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.951+0000 7f37ea800640 1 -- 172.21.15.6:0/4027649778 learned_addr learned my addr 172.21.15.6:0/4027649778 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.952+0000 7f37e8a00640 1 -- 172.21.15.6:0/4027649778 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288648520 0 0) 0x7f37ec15cc40 con 0x7f37ec0cdc20 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.952+0000 7f37e8a00640 1 -- 172.21.15.6:0/4027649778 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d0003520 con 0x7f37ec0cdc20 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.952+0000 7f37e9400640 1 -- 172.21.15.6:0/4027649778 >> v1:172.21.15.178:6789/0 conn(0x7f37ec0af1a0 legacy=0x7f37ec0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.952+0000 7f37e9400640 1 -- 172.21.15.6:0/4027649778 >> v1:172.21.15.6:6789/0 conn(0x7f37ec0cdc20 legacy=0x7f37ec0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.952+0000 7f37e9400640 1 -- 172.21.15.6:0/4027649778 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec15d4e0 con 0x7f37ec0af580 2024-03-06T07:38:54.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.953+0000 7f37e9400640 1 --2- 172.21.15.6:0/4027649778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0af580 0x7f37ec15c600 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f37d80097a0 tx=0x7f37d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3d429e96ce063a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37e8a00640 1 -- 172.21.15.6:0/4027649778 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d8035020 con 0x7f37ec0af580 2024-03-06T07:38:54.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37e8a00640 1 -- 172.21.15.6:0/4027649778 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d8031830 con 0x7f37ec0af580 2024-03-06T07:38:54.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37e8a00640 1 -- 172.21.15.6:0/4027649778 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d8031be0 con 0x7f37ec0af580 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 -- 172.21.15.6:0/4027649778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0af580 msgr2=0x7f37ec15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 --2- 172.21.15.6:0/4027649778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0af580 0x7f37ec15c600 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f37d80097a0 tx=0x7f37d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 -- 172.21.15.6:0/4027649778 shutdown_connections 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 --2- 172.21.15.6:0/4027649778 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0af580 0x7f37ec15c600 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 -- 172.21.15.6:0/4027649778 >> 172.21.15.6:0/4027649778 conn(0x7f37ec06afb0 msgr2=0x7f37ec06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 -- 172.21.15.6:0/4027649778 shutdown_connections 2024-03-06T07:38:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.954+0000 7f37f0a00640 1 -- 172.21.15.6:0/4027649778 wait complete. 2024-03-06T07:38:54.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.956+0000 7f37f0a00640 1 Processor -- start 2024-03-06T07:38:54.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.956+0000 7f37f0a00640 1 -- start start 2024-03-06T07:38:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37f0a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37ec0af1a0 0x7f37ec153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37f0a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37ec0af580 0x7f37ec09ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37f0a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 0x7f37ec09bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37f0a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec155e70 con 0x7f37ec0cdc20 2024-03-06T07:38:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37f0a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec0b3730 con 0x7f37ec0af1a0 2024-03-06T07:38:54.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37f0a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec155ff0 con 0x7f37ec0af580 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37ea800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 0x7f37ec09bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37ea800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 0x7f37ec09bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51638/0 (socket says 172.21.15.6:51638) 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.957+0000 7f37ea800640 1 -- 172.21.15.6:0/2468750378 learned_addr learned my addr 172.21.15.6:0/2468750378 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37e9e00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37ec0af1a0 0x7f37ec153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37e9400640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37ec0af580 0x7f37ec09ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37ea800640 1 -- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37ec0af580 msgr2=0x7f37ec09ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37ea800640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37ec0af580 0x7f37ec09ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37ea800640 1 -- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37ec0af1a0 msgr2=0x7f37ec153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37ea800640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37ec0af1a0 0x7f37ec153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37ea800640 1 -- 172.21.15.6:0/2468750378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec154d00 con 0x7f37ec0cdc20 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37e9400640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37ec0af580 0x7f37ec09ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:54.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37ea800640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 0x7f37ec09bfd0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f37e400cad0 tx=0x7f37e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.958+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4017070 con 0x7f37ec0cdc20 2024-03-06T07:38:54.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.959+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e400abd0 con 0x7f37ec0cdc20 2024-03-06T07:38:54.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.959+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4012400 con 0x7f37ec0cdc20 2024-03-06T07:38:54.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.959+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec15dce0 con 0x7f37ec0cdc20 2024-03-06T07:38:54.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.959+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec15e2a0 con 0x7f37ec0cdc20 2024-03-06T07:38:54.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.960+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec04f3a0 con 0x7f37ec0cdc20 2024-03-06T07:38:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.961+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37e401a070 con 0x7f37ec0cdc20 2024-03-06T07:38:54.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.962+0000 7f37e2000640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37cc06b4d0 0x7f37cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.962+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37e408a140 con 0x7f37ec0cdc20 2024-03-06T07:38:54.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.962+0000 7f37e9e00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37cc06b4d0 0x7f37cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:54.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.963+0000 7f37e9e00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37cc06b4d0 0x7f37cc06d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f37dc001640 tx=0x7f37dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:54.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:54.966+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37e4058e50 con 0x7f37ec0cdc20 2024-03-06T07:38:55.212 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.210+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37ec155630 con 0x7f37ec0cdc20 2024-03-06T07:38:55.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.211+0000 7f37e2000640 1 -- 172.21.15.6:0/2468750378 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f37e4058750 con 0x7f37ec0cdc20 2024-03-06T07:38:55.213 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:55.214 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37cc06b4d0 msgr2=0x7f37cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37cc06b4d0 0x7f37cc06d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f37dc001640 tx=0x7f37dc009040 comp rx=0 tx=0).stop 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 msgr2=0x7f37ec09bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 0x7f37ec09bfd0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f37e400cad0 tx=0x7f37e400cf90 comp rx=0 tx=0).stop 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 shutdown_connections 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f37cc06b4d0 0x7f37cc06d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f37ec0cdc20 0x7f37ec09bfd0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f37ec0af580 0x7f37ec09ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 --2- 172.21.15.6:0/2468750378 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37ec0af1a0 0x7f37ec153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 >> 172.21.15.6:0/2468750378 conn(0x7f37ec06afb0 msgr2=0x7f37ec0cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 shutdown_connections 2024-03-06T07:38:55.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:55.214+0000 7f37f0a00640 1 -- 172.21.15.6:0/2468750378 wait complete. 2024-03-06T07:38:55.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:55.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:56.227 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:56.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.657+0000 7f0fb6400640 1 Processor -- start 2024-03-06T07:38:56.660 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.658+0000 7f0fb6400640 1 -- start start 2024-03-06T07:38:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.659+0000 7f0fb6400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 0x7f0fb0162520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0fb6400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb0161a80 con 0x7f0fb006c420 2024-03-06T07:38:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0fb6400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb015efa0 con 0x7f0fb006bfc0 2024-03-06T07:38:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0fb6400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb015f280 con 0x7f0fb015dd00 2024-03-06T07:38:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0fafe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0fb006c420 0x7f0fb015e7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58590/0 (socket says 172.21.15.6:58590) 2024-03-06T07:38:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0fafe00640 1 -- 172.21.15.6:0/3432250439 learned_addr learned my addr 172.21.15.6:0/3432250439 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0faf400640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 0x7f0fb0162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843241878 0 0) 0x7f0fb0161a80 con 0x7f0fb006c420 2024-03-06T07:38:56.664 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.660+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f94003520 con 0x7f0fb006c420 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3035018048 0 0) 0x7f0f94003520 con 0x7f0fb006c420 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 msgr2=0x7f0fb0162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 0x7f0fb0162520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 >> v1:172.21.15.178:6789/0 conn(0x7f0fb006bfc0 legacy=0x7f0fb015e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0161d20 con 0x7f0fb006c420 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f0fa8001a10 con 0x7f0fb006c420 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 >> v1:172.21.15.6:6789/0 conn(0x7f0fb006c420 legacy=0x7f0fb015e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94003a50 0x7f0f94003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 0x7f0f94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94004d40 0x7f0f940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb015f280 con 0x7f0f94003a50 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9400bb50 con 0x7f0f94004d40 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9400bd20 con 0x7f0fb015dd00 2024-03-06T07:38:56.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.661+0000 7f0faf400640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94003a50 0x7f0f94003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0fafe00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94004d40 0x7f0f940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0fa7e00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 0x7f0f94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0faf400640 1 -- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 msgr2=0x7f0f94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0faf400640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb015dd00 0x7f0f94004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0faf400640 1 -- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94004d40 msgr2=0x7f0f940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0faf400640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94004d40 0x7f0f940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0faf400640 1 -- 172.21.15.6:0/3432250439 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94003520 con 0x7f0f94003a50 2024-03-06T07:38:56.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.662+0000 7f0fafe00640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94004d40 0x7f0f940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:56.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.663+0000 7f0faf400640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94003a50 0x7f0f94003e70 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009ac0 tx=0x7f0f94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4619dfa58124440 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.663+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0039070 con 0x7f0f94003a50 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.663+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa002c680 con 0x7f0f94003a50 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.663+0000 7f0faea00640 1 -- 172.21.15.6:0/3432250439 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0009690 con 0x7f0f94003a50 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fb6400640 1 -- 172.21.15.6:0/3432250439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94003a50 msgr2=0x7f0f94003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fb6400640 1 --2- 172.21.15.6:0/3432250439 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94003a50 0x7f0f94003e70 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009ac0 tx=0x7f0f94005830 comp rx=0 tx=0).stop 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fafe00640 1 -- 172.21.15.6:0/3432250439 reap_dead start 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fb6400640 1 -- 172.21.15.6:0/3432250439 shutdown_connections 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fb6400640 1 -- 172.21.15.6:0/3432250439 >> 172.21.15.6:0/3432250439 conn(0x7f0fb00714d0 msgr2=0x7f0fb00718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fb6400640 1 -- 172.21.15.6:0/3432250439 shutdown_connections 2024-03-06T07:38:56.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.664+0000 7f0fb6400640 1 -- 172.21.15.6:0/3432250439 wait complete. 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.666+0000 7f0fb6400640 1 Processor -- start 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.666+0000 7f0fb6400640 1 -- start start 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fb6400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94003a50 0x7f0fb00aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fb6400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb0152a30 0x7f0fb0157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fb6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 0x7f0fb0157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fb6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb006f280 con 0x7f0fb00b13f0 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fb6400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb006c890 con 0x7f0f94003a50 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fb6400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0162520 con 0x7f0fb0152a30 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 0x7f0fb0157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fafe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 0x7f0fb0157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51664/0 (socket says 172.21.15.6:51664) 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.667+0000 7f0fafe00640 1 -- 172.21.15.6:0/1176441521 learned_addr learned my addr 172.21.15.6:0/1176441521 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0faf400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94003a50 0x7f0fb00aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fa7e00640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb0152a30 0x7f0fb0157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fafe00640 1 -- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb0152a30 msgr2=0x7f0fb0157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fafe00640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb0152a30 0x7f0fb0157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fafe00640 1 -- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94003a50 msgr2=0x7f0fb00aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fafe00640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94003a50 0x7f0fb00aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fafe00640 1 -- 172.21.15.6:0/1176441521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00cf2e0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0faf400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94003a50 0x7f0fb00aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fafe00640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 0x7f0fb0157f50 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0071110 tx=0x7f0fa800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.668+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8004d60 con 0x7f0fb00b13f0 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.669+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa800bb80 con 0x7f0fb00b13f0 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.669+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb00af790 con 0x7f0fb00b13f0 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.669+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8007110 con 0x7f0fb00b13f0 2024-03-06T07:38:56.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.669+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb01533b0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.670+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb004f3a0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.671+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fa801b020 con 0x7f0fb00b13f0 2024-03-06T07:38:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.672+0000 7f0fa7400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f8c06b500 0x7f0f8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.672+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0fa808a3a0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.672+0000 7f0faf400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f8c06b500 0x7f0f8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.673+0000 7f0faf400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f8c06b500 0x7f0f8c06d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0036000 tx=0x7f0fa0030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:56.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.676+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0fa80591d0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.921+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fb00b34f0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.924 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.922+0000 7f0fa7400640 1 -- 172.21.15.6:0/1176441521 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f0fa8058ad0 con 0x7f0fb00b13f0 2024-03-06T07:38:56.925 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:56.925 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:56.925 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:56.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f8c06b500 msgr2=0x7f0f8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f8c06b500 0x7f0f8c06d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0036000 tx=0x7f0fa0030000 comp rx=0 tx=0).stop 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 msgr2=0x7f0fb0157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 0x7f0fb0157f50 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0071110 tx=0x7f0fa800b4a0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 shutdown_connections 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f0f8c06b500 0x7f0f8c06d9c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fb00b13f0 0x7f0fb0157f50 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f0fb0152a30 0x7f0fb0157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 --2- 172.21.15.6:0/1176441521 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f94003a50 0x7f0fb00aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 >> 172.21.15.6:0/1176441521 conn(0x7f0fb00714d0 msgr2=0x7f0fb006e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 shutdown_connections 2024-03-06T07:38:56.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:56.925+0000 7f0fb6400640 1 -- 172.21.15.6:0/1176441521 wait complete. 2024-03-06T07:38:57.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:57.149 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:58.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:38:58.229 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:38:58.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.375+0000 7fcd12200640 1 Processor -- start 2024-03-06T07:38:58.378 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.376+0000 7fcd12200640 1 -- start start 2024-03-06T07:38:58.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.377+0000 7fcd12200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcd0c15bfb0 0x7fcd0c0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:58.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.377+0000 7fcd12200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c0af000 con 0x7fcd0c15cda0 2024-03-06T07:38:58.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.377+0000 7fcd12200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c0af2e0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.379 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.377+0000 7fcd12200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0c0b3de0 con 0x7fcd0c15bfb0 2024-03-06T07:38:58.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.378+0000 7fcd03e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7fcd0c15bfb0 0x7fcd0c0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:58.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.378+0000 7fcd0b400640 1 --1- >> v1:172.21.15.178:6789/0 conn(0x7fcd0c06afb0 0x7fcd0c06b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.178:6789/0 says I am v1:172.21.15.6:54894/0 (socket says 172.21.15.6:54894) 2024-03-06T07:38:58.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.378+0000 7fcd0b400640 1 -- 172.21.15.6:0/471997295 learned_addr learned my addr 172.21.15.6:0/471997295 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:38:58.380 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.378+0000 7fcd0be00640 1 --1- 172.21.15.6:0/471997295 >> v1:172.21.15.6:6789/0 conn(0x7fcd0c15cda0 0x7fcd0c0b4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58604/0 (socket says 172.21.15.6:58604) 2024-03-06T07:38:58.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 <== mon.1 v1:172.21.15.178:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963031198 0 0) 0x7fcd0c0af2e0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 --> v1:172.21.15.178:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf0003520 con 0x7fcd0c06afb0 2024-03-06T07:38:58.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951538702 0 0) 0x7fcd0c0af000 con 0x7fcd0c15cda0 2024-03-06T07:38:58.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd0c0af2e0 con 0x7fcd0c15cda0 2024-03-06T07:38:58.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd03e00640 1 -- 172.21.15.6:0/471997295 >> v1:172.21.15.178:6789/0 conn(0x7fcd0c06afb0 legacy=0x7fcd0c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd03e00640 1 -- 172.21.15.6:0/471997295 >> v1:172.21.15.6:6789/0 conn(0x7fcd0c15cda0 legacy=0x7fcd0c0b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.382 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.379+0000 7fcd03e00640 1 -- 172.21.15.6:0/471997295 --> v2:172.21.15.178:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c0b4030 con 0x7fcd0c15bfb0 2024-03-06T07:38:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.381+0000 7fcd03e00640 1 --2- 172.21.15.6:0/471997295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15bfb0 0x7fcd0c0ae900 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fccf40097a0 tx=0x7fccf402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1915584dac5bcbee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.381+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4034070 con 0x7fcd0c15bfb0 2024-03-06T07:38:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.381+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf4030610 con 0x7fcd0c15bfb0 2024-03-06T07:38:58.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.381+0000 7fcd0aa00640 1 -- 172.21.15.6:0/471997295 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf40309e0 con 0x7fcd0c15bfb0 2024-03-06T07:38:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 -- 172.21.15.6:0/471997295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15bfb0 msgr2=0x7fcd0c0ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 --2- 172.21.15.6:0/471997295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15bfb0 0x7fcd0c0ae900 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fccf40097a0 tx=0x7fccf402b960 comp rx=0 tx=0).stop 2024-03-06T07:38:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 -- 172.21.15.6:0/471997295 shutdown_connections 2024-03-06T07:38:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 --2- 172.21.15.6:0/471997295 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15bfb0 0x7fcd0c0ae900 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 -- 172.21.15.6:0/471997295 >> 172.21.15.6:0/471997295 conn(0x7fcd0c0b2ae0 msgr2=0x7fcd0c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:58.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 -- 172.21.15.6:0/471997295 shutdown_connections 2024-03-06T07:38:58.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.382+0000 7fcd12200640 1 -- 172.21.15.6:0/471997295 wait complete. 2024-03-06T07:38:58.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.385+0000 7fcd12200640 1 Processor -- start 2024-03-06T07:38:58.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.385+0000 7fcd12200640 1 -- start start 2024-03-06T07:38:58.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd12200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 0x7fcd0c06eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd12200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd0c15bfb0 0x7fcd0c0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd12200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15cda0 0x7fcd0c15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd12200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0b4f40 con 0x7fcd0c06afb0 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd12200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c15d1d0 con 0x7fcd0c15cda0 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd12200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0b50c0 con 0x7fcd0c15bfb0 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.386+0000 7fcd0b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 0x7fcd0c06eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 0x7fcd0c06eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51678/0 (socket says 172.21.15.6:51678) 2024-03-06T07:38:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 -- 172.21.15.6:0/1352973138 learned_addr learned my addr 172.21.15.6:0/1352973138 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:38:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 -- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd0c15bfb0 msgr2=0x7fcd0c0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd0c15bfb0 0x7fcd0c0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 -- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15cda0 msgr2=0x7fcd0c15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0be00640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15cda0 0x7fcd0c15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15cda0 0x7fcd0c15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 -- 172.21.15.6:0/1352973138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c14bff0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0be00640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15cda0 0x7fcd0c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.387+0000 7fcd0b400640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 0x7fcd0c06eae0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c4d0 tx=0x7fccfc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.388+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc017070 con 0x7fcd0c06afb0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.388+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc005ce0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.388+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c0d06b0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.388+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc007270 con 0x7fcd0c06afb0 2024-03-06T07:38:58.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.388+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c0cfbd0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.389+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0c04f3a0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.390+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccfc01a070 con 0x7fcd0c06afb0 2024-03-06T07:38:58.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.391+0000 7fcd03400640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcce806b520 0x7fcce806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:38:58.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.391+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fccfc089580 con 0x7fcd0c06afb0 2024-03-06T07:38:58.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.392+0000 7fcd03e00640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcce806b520 0x7fcce806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:38:58.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.393+0000 7fcd03e00640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcce806b520 0x7fcce806d9e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fccf4036000 tx=0x7fccf4030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:38:58.401 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.400+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccfc057d50 con 0x7fcd0c06afb0 2024-03-06T07:38:58.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.643+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd0c073050 con 0x7fcd0c06afb0 2024-03-06T07:38:58.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.644+0000 7fcd03400640 1 -- 172.21.15.6:0/1352973138 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fccfc05bdf0 con 0x7fcd0c06afb0 2024-03-06T07:38:58.647 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:38:58.647 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:38:58.647 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcce806b520 msgr2=0x7fcce806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcce806b520 0x7fcce806d9e0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fccf4036000 tx=0x7fccf4030000 comp rx=0 tx=0).stop 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 msgr2=0x7fcd0c06eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 0x7fcd0c06eae0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c4d0 tx=0x7fccfc00c990 comp rx=0 tx=0).stop 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 shutdown_connections 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fcce806b520 0x7fcce806d9e0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcd0c15cda0 0x7fcd0c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fcd0c15bfb0 0x7fcd0c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 --2- 172.21.15.6:0/1352973138 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fcd0c06afb0 0x7fcd0c06eae0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.647+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 >> 172.21.15.6:0/1352973138 conn(0x7fcd0c0b2ae0 msgr2=0x7fcd0c074000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.648+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 shutdown_connections 2024-03-06T07:38:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:38:58.648+0000 7fcd12200640 1 -- 172.21.15.6:0/1352973138 wait complete. 2024-03-06T07:38:58.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:38:58.871 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:38:59.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:00.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.093+0000 7fd37e800640 1 Processor -- start 2024-03-06T07:39:00.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.093+0000 7fd37e800640 1 -- start start 2024-03-06T07:39:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.094+0000 7fd37e800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 0x7fd378145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.095+0000 7fd37e800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780d0910 con 0x7fd3781457c0 2024-03-06T07:39:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.095+0000 7fd37e800640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3780d0bf0 con 0x7fd37806b6c0 2024-03-06T07:39:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.095+0000 7fd37e800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780d0ed0 con 0x7fd378144c00 2024-03-06T07:39:00.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.095+0000 7fd37ca00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd3781457c0 0x7fd37806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58606/0 (socket says 172.21.15.6:58606) 2024-03-06T07:39:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.095+0000 7fd37ca00640 1 -- 172.21.15.6:0/2176592519 learned_addr learned my addr 172.21.15.6:0/2176592519 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.095+0000 7fd37d400640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 0x7fd378145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588870267 0 0) 0x7fd3780d0910 con 0x7fd3781457c0 2024-03-06T07:39:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd360003520 con 0x7fd3781457c0 2024-03-06T07:39:00.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2664837382 0 0) 0x7fd360003520 con 0x7fd3781457c0 2024-03-06T07:39:00.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 msgr2=0x7fd378145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:00.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 0x7fd378145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 >> v1:172.21.15.178:6789/0 conn(0x7fd37806b6c0 legacy=0x7fd3780cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3780d11b0 con 0x7fd3781457c0 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd364001670 con 0x7fd3781457c0 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 >> v1:172.21.15.6:6789/0 conn(0x7fd3781457c0 legacy=0x7fd37806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd360003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 0x7fd360004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd360004d40 0x7fd3600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780d0ed0 con 0x7fd3600039e0 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36000baa0 con 0x7fd360004d40 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36000bca0 con 0x7fd378144c00 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.096+0000 7fd37d400640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd360003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.097+0000 7fd37d400640 1 -- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 msgr2=0x7fd360004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.097+0000 7fd37d400640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd378144c00 0x7fd360004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.097+0000 7fd37de00640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd360004d40 0x7fd3600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.097+0000 7fd37d400640 1 -- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd360004d40 msgr2=0x7fd3600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.097+0000 7fd37d400640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd360004d40 0x7fd3600051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.097+0000 7fd37d400640 1 -- 172.21.15.6:0/2176592519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360003520 con 0x7fd3600039e0 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.098+0000 7fd37d400640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd360003e00 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009f90 tx=0x7fd360005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37af2092b684d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c039070 con 0x7fd3600039e0 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c02cc20 con 0x7fd3600039e0 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd377e00640 1 -- 172.21.15.6:0/2176592519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c009c10 con 0x7fd3600039e0 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37e800640 1 -- 172.21.15.6:0/2176592519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 msgr2=0x7fd360003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37e800640 1 --2- 172.21.15.6:0/2176592519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd360003e00 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009f90 tx=0x7fd360005830 comp rx=0 tx=0).stop 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37de00640 1 -- 172.21.15.6:0/2176592519 reap_dead start 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37e800640 1 -- 172.21.15.6:0/2176592519 shutdown_connections 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37e800640 1 -- 172.21.15.6:0/2176592519 >> 172.21.15.6:0/2176592519 conn(0x7fd37806c0c0 msgr2=0x7fd37806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37e800640 1 -- 172.21.15.6:0/2176592519 shutdown_connections 2024-03-06T07:39:00.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.099+0000 7fd37e800640 1 -- 172.21.15.6:0/2176592519 wait complete. 2024-03-06T07:39:00.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.102+0000 7fd37e800640 1 Processor -- start 2024-03-06T07:39:00.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.103+0000 7fd37e800640 1 -- start start 2024-03-06T07:39:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37e800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd3780b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37e800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd37813dd60 0x7fd3780b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37e800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd37813f330 0x7fd37815d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37e800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780d19b0 con 0x7fd3600039e0 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37e800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd378144fe0 con 0x7fd37813dd60 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37e800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37806f820 con 0x7fd37813f330 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd3780b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37d400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd3780b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51696/0 (socket says 172.21.15.6:51696) 2024-03-06T07:39:00.107 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.104+0000 7fd37d400640 1 -- 172.21.15.6:0/2876507135 learned_addr learned my addr 172.21.15.6:0/2876507135 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:00.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.105+0000 7fd37d400640 1 -- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd37813f330 msgr2=0x7fd37815d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.105+0000 7fd37d400640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd37813f330 0x7fd37815d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.105+0000 7fd37d400640 1 -- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd37813dd60 msgr2=0x7fd3780b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.105+0000 7fd37d400640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd37813dd60 0x7fd3780b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.105+0000 7fd37d400640 1 -- 172.21.15.6:0/2876507135 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37815b270 con 0x7fd3600039e0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.105+0000 7fd37d400640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd3780b2d50 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009f90 tx=0x7fd36c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.106+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c039050 con 0x7fd3600039e0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.106+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c03a040 con 0x7fd3600039e0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.106+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c009c10 con 0x7fd3600039e0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.106+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37815b550 con 0x7fd3600039e0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.106+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3780b1610 con 0x7fd3600039e0 2024-03-06T07:39:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.107+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37804f330 con 0x7fd3600039e0 2024-03-06T07:39:00.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.108+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd36c047030 con 0x7fd3600039e0 2024-03-06T07:39:00.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.109+0000 7fd375600640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd35806b4d0 0x7fd35806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:00.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.110+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd36c0ac5c0 con 0x7fd3600039e0 2024-03-06T07:39:00.112 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.110+0000 7fd37ca00640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd35806b4d0 0x7fd35806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:00.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.111+0000 7fd37ca00640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd35806b4d0 0x7fd35806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd3781f6cf0 tx=0x7fd364001580 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:00.115 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.114+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd36c04e060 con 0x7fd3600039e0 2024-03-06T07:39:00.231 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:00.356 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.354+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3780722f0 con 0x7fd3600039e0 2024-03-06T07:39:00.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.355+0000 7fd375600640 1 -- 172.21.15.6:0/2876507135 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fd36c07b3f0 con 0x7fd3600039e0 2024-03-06T07:39:00.358 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:00.358 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:00.358 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd35806b4d0 msgr2=0x7fd35806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd35806b4d0 0x7fd35806d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd3781f6cf0 tx=0x7fd364001580 comp rx=0 tx=0).stop 2024-03-06T07:39:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 msgr2=0x7fd3780b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:00.360 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd3780b2d50 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fd36c009f90 tx=0x7fd36c009750 comp rx=0 tx=0).stop 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 shutdown_connections 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd35806b4d0 0x7fd35806d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd37813dd60 0x7fd3780b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd37813f330 0x7fd37815d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 --2- 172.21.15.6:0/2876507135 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3600039e0 0x7fd3780b2d50 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 >> 172.21.15.6:0/2876507135 conn(0x7fd37806c0c0 msgr2=0x7fd3780b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 shutdown_connections 2024-03-06T07:39:00.361 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:00.358+0000 7fd37e800640 1 -- 172.21.15.6:0/2876507135 wait complete. 2024-03-06T07:39:00.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:00.580 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:01.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:01.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.809+0000 7ffad2600640 1 Processor -- start 2024-03-06T07:39:01.811 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.809+0000 7ffad2600640 1 -- start start 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.811+0000 7ffad2600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc15e0b0 0x7ffacc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.811+0000 7ffad2600640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffacc0d4a10 0x7ffacc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.811+0000 7ffad2600640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffacc0b3710 con 0x7ffacc0d0c80 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.811+0000 7ffad2600640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffacc0b09e0 con 0x7ffacc0d4a10 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.811+0000 7ffad2600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0b39f0 con 0x7ffacc15e0b0 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffacbe00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ffacc0d4a10 0x7ffacc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffad1c00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ffacc0d0c80 0x7ffacc0d1900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58618/0 (socket says 172.21.15.6:58618) 2024-03-06T07:39:01.814 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffad1c00640 1 -- 172.21.15.6:0/2158188255 learned_addr learned my addr 172.21.15.6:0/2158188255 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffad1200640 1 --2- 172.21.15.6:0/2158188255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc15e0b0 0x7ffacc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffacbe00640 1 -- 172.21.15.6:0/2158188255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc15e0b0 msgr2=0x7ffacc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffacbe00640 1 --2- 172.21.15.6:0/2158188255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc15e0b0 0x7ffacc0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffacbe00640 1 -- 172.21.15.6:0/2158188255 >> v1:172.21.15.6:6789/0 conn(0x7ffacc0d0c80 legacy=0x7ffacc0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.815 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.812+0000 7ffacbe00640 1 -- 172.21.15.6:0/2158188255 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc0b3cd0 con 0x7ffacc0d4a10 2024-03-06T07:39:01.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.813+0000 7ffacbe00640 1 --2- 172.21.15.6:0/2158188255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc0d0740 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e09bdd3aafcaee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffacb400640 1 -- 172.21.15.6:0/2158188255 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8035020 con 0x7ffacc0d4a10 2024-03-06T07:39:01.816 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffacb400640 1 -- 172.21.15.6:0/2158188255 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8031810 con 0x7ffacc0d4a10 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffacb400640 1 -- 172.21.15.6:0/2158188255 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8031be0 con 0x7ffacc0d4a10 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 -- 172.21.15.6:0/2158188255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 msgr2=0x7ffacc0d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 --2- 172.21.15.6:0/2158188255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc0d0740 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7ffab80097a0 tx=0x7ffab802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 -- 172.21.15.6:0/2158188255 shutdown_connections 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 --2- 172.21.15.6:0/2158188255 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc0d0740 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 --2- 172.21.15.6:0/2158188255 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc15e0b0 0x7ffacc0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 -- 172.21.15.6:0/2158188255 >> 172.21.15.6:0/2158188255 conn(0x7ffacc0d3d30 msgr2=0x7ffacc0d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 -- 172.21.15.6:0/2158188255 shutdown_connections 2024-03-06T07:39:01.817 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.814+0000 7ffad2600640 1 -- 172.21.15.6:0/2158188255 wait complete. 2024-03-06T07:39:01.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.816+0000 7ffad2600640 1 Processor -- start 2024-03-06T07:39:01.818 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.816+0000 7ffad2600640 1 -- start start 2024-03-06T07:39:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffad2600640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc0d0c80 0x7ffacc0b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.819 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffad2600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc14adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffad2600640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffacc15e0b0 0x7ffacc06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffad2600640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0cfc60 con 0x7ffacc0d4a10 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffad2600640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc15e490 con 0x7ffacc15e0b0 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffad2600640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc0b54e0 con 0x7ffacc0d0c80 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffacbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc14adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffacbe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc14adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51716/0 (socket says 172.21.15.6:51716) 2024-03-06T07:39:01.820 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.817+0000 7ffacbe00640 1 -- 172.21.15.6:0/522224211 learned_addr learned my addr 172.21.15.6:0/522224211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:01.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffacbe00640 1 -- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc0d0c80 msgr2=0x7ffacc0b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.821 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffad1c00640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffacc15e0b0 0x7ffacc06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffad1200640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc0d0c80 0x7ffacc0b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffacbe00640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc0d0c80 0x7ffacc0b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffacbe00640 1 -- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffacc15e0b0 msgr2=0x7ffacc06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffacbe00640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffacc15e0b0 0x7ffacc06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffacbe00640 1 -- 172.21.15.6:0/522224211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc0b8eb0 con 0x7ffacc0d4a10 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffad1c00640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffacc15e0b0 0x7ffacc06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffad1200640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc0d0c80 0x7ffacc0b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.818+0000 7ffacbe00640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc14adf0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7ffab802bec0 tx=0x7ffab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.819+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8041070 con 0x7ffacc0d4a10 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.819+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab803ad00 con 0x7ffacc0d4a10 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.819+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8035040 con 0x7ffacc0d4a10 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.819+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc0b9190 con 0x7ffacc0d4a10 2024-03-06T07:39:01.822 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.819+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc0b9750 con 0x7ffacc0d4a10 2024-03-06T07:39:01.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.820+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc04f330 con 0x7ffacc0d4a10 2024-03-06T07:39:01.823 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.821+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffab8038020 con 0x7ffacc0d4a10 2024-03-06T07:39:01.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.822+0000 7ffac8c00640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffaac06b4d0 0x7ffaac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:01.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.822+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffab80abfd0 con 0x7ffacc0d4a10 2024-03-06T07:39:01.824 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.823+0000 7ffad1200640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffaac06b4d0 0x7ffaac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:01.826 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.824+0000 7ffad1200640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffaac06b4d0 0x7ffaac06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffac000b820 tx=0x7ffac0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:01.832 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:01.831+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffab807ae00 con 0x7ffacc0d4a10 2024-03-06T07:39:02.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.076+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffacc0b9b30 con 0x7ffacc0d4a10 2024-03-06T07:39:02.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.077+0000 7ffac8c00640 1 -- 172.21.15.6:0/522224211 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7ffab807a700 con 0x7ffacc0d4a10 2024-03-06T07:39:02.079 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:02.079 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:02.080 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:02.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffaac06b4d0 msgr2=0x7ffaac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffaac06b4d0 0x7ffaac06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffac000b820 tx=0x7ffac0008040 comp rx=0 tx=0).stop 2024-03-06T07:39:02.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 msgr2=0x7ffacc14adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc14adf0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7ffab802bec0 tx=0x7ffab80046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 shutdown_connections 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7ffaac06b4d0 0x7ffaac06d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ffacc15e0b0 0x7ffacc06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ffacc0d4a10 0x7ffacc14adf0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 --2- 172.21.15.6:0/522224211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7ffacc0d0c80 0x7ffacc0b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 >> 172.21.15.6:0/522224211 conn(0x7ffacc0d3d30 msgr2=0x7ffacc06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 shutdown_connections 2024-03-06T07:39:02.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:02.080+0000 7ffad2600640 1 -- 172.21.15.6:0/522224211 wait complete. 2024-03-06T07:39:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:02.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:02.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:03.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:03.528 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.526+0000 7f462be00640 1 Processor -- start 2024-03-06T07:39:03.529 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.526+0000 7f462be00640 1 -- start start 2024-03-06T07:39:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f462c15e070 0x7f462c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462be00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f462c0cf410 0x7f462c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c0d05c0 0x7f462c0d0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.530 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462be00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462c0d18c0 con 0x7f462c15e070 2024-03-06T07:39:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462be00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462c0d1ba0 con 0x7f462c0cf410 2024-03-06T07:39:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0d1e80 con 0x7f462c0d05c0 2024-03-06T07:39:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f462c15e070 0x7f462c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462aa00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f462c15e070 0x7f462c0d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38682/0 (socket says 172.21.15.6:38682) 2024-03-06T07:39:03.531 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.528+0000 7f462aa00640 1 -- 172.21.15.6:0/3911550341 learned_addr learned my addr 172.21.15.6:0/3911550341 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f4623e00640 1 --2- 172.21.15.6:0/3911550341 >> v2:172.21.15.178:3300/0 conn(0x7f462c0cf410 0x7f462c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f462b400640 1 --2- 172.21.15.6:0/3911550341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c0d05c0 0x7f462c0d0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f462aa00640 1 -- 172.21.15.6:0/3911550341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c0d05c0 msgr2=0x7f462c0d0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f462aa00640 1 --2- 172.21.15.6:0/3911550341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c0d05c0 0x7f462c0d0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f462aa00640 1 -- 172.21.15.6:0/3911550341 >> v2:172.21.15.178:3300/0 conn(0x7f462c0cf410 msgr2=0x7f462c0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f462aa00640 1 --2- 172.21.15.6:0/3911550341 >> v2:172.21.15.178:3300/0 conn(0x7f462c0cf410 0x7f462c0cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f462aa00640 1 -- 172.21.15.6:0/3911550341 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c0d2160 con 0x7f462c15e070 2024-03-06T07:39:03.532 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.529+0000 7f4623e00640 1 --2- 172.21.15.6:0/3911550341 >> v2:172.21.15.178:3300/0 conn(0x7f462c0cf410 0x7f462c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.531+0000 7f462aa00640 1 --2- 172.21.15.6:0/3911550341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c15e070 0x7f462c0d2960 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f462c0b6880 tx=0x7f461c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70032593fd5c766 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.531+0000 7f462a000640 1 -- 172.21.15.6:0/3911550341 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c035020 con 0x7f462c15e070 2024-03-06T07:39:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.531+0000 7f462a000640 1 -- 172.21.15.6:0/3911550341 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c031870 con 0x7f462c15e070 2024-03-06T07:39:03.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.531+0000 7f462a000640 1 -- 172.21.15.6:0/3911550341 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c031c40 con 0x7f462c15e070 2024-03-06T07:39:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 -- 172.21.15.6:0/3911550341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c15e070 msgr2=0x7f462c0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 --2- 172.21.15.6:0/3911550341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c15e070 0x7f462c0d2960 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f462c0b6880 tx=0x7f461c02b910 comp rx=0 tx=0).stop 2024-03-06T07:39:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 -- 172.21.15.6:0/3911550341 shutdown_connections 2024-03-06T07:39:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 --2- 172.21.15.6:0/3911550341 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c0d05c0 0x7f462c0d0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 --2- 172.21.15.6:0/3911550341 >> v2:172.21.15.178:3300/0 conn(0x7f462c0cf410 0x7f462c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 --2- 172.21.15.6:0/3911550341 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c15e070 0x7f462c0d2960 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 -- 172.21.15.6:0/3911550341 >> 172.21.15.6:0/3911550341 conn(0x7f462c08d2a0 msgr2=0x7f462c0d3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 -- 172.21.15.6:0/3911550341 shutdown_connections 2024-03-06T07:39:03.535 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.532+0000 7f462be00640 1 -- 172.21.15.6:0/3911550341 wait complete. 2024-03-06T07:39:03.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.534+0000 7f462be00640 1 Processor -- start 2024-03-06T07:39:03.536 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.534+0000 7f462be00640 1 -- start start 2024-03-06T07:39:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f462be00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f462c0cf410 0x7f462c15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f462be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 0x7f462c0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f462be00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c15e070 0x7f462c0b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f462be00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0d4730 con 0x7f462c0d05c0 2024-03-06T07:39:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f462be00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0d45b0 con 0x7f462c0cf410 2024-03-06T07:39:03.538 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f462be00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0b3630 con 0x7f462c15e070 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f4623e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 0x7f462c0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f4623e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 0x7f462c0b6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38698/0 (socket says 172.21.15.6:38698) 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.535+0000 7f4623e00640 1 -- 172.21.15.6:0/916075993 learned_addr learned my addr 172.21.15.6:0/916075993 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f462aa00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f462c0cf410 0x7f462c15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f462b400640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c15e070 0x7f462c0b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f4623e00640 1 -- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c15e070 msgr2=0x7f462c0b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f4623e00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c15e070 0x7f462c0b6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f4623e00640 1 -- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f462c0cf410 msgr2=0x7f462c15c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f4623e00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f462c0cf410 0x7f462c15c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f4623e00640 1 -- 172.21.15.6:0/916075993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c0cc7a0 con 0x7f462c0d05c0 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f462aa00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f462c0cf410 0x7f462c15c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:03.539 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.536+0000 7f4623e00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 0x7f462c0b6910 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f461400e840 tx=0x7f461400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.537+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461400f040 con 0x7f462c0d05c0 2024-03-06T07:39:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.537+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4614004030 con 0x7f462c0d05c0 2024-03-06T07:39:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.537+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614004400 con 0x7f462c0d05c0 2024-03-06T07:39:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.537+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462c0cbb90 con 0x7f462c0d05c0 2024-03-06T07:39:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.537+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c0cc150 con 0x7f462c0d05c0 2024-03-06T07:39:03.540 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.538+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462c04f330 con 0x7f462c0d05c0 2024-03-06T07:39:03.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.539+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f461400c6d0 con 0x7f462c0d05c0 2024-03-06T07:39:03.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.540+0000 7f4623400640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f460806b4d0 0x7f460806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:03.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.540+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4614089f50 con 0x7f462c0d05c0 2024-03-06T07:39:03.542 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.541+0000 7f462aa00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f460806b4d0 0x7f460806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:03.544 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.542+0000 7f462aa00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f460806b4d0 0x7f460806d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f461c036000 tx=0x7f461c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:03.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.544+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4614017020 con 0x7f462c0d05c0 2024-03-06T07:39:03.784 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.782+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f462c0cdf20 con 0x7f462c0d05c0 2024-03-06T07:39:03.785 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.783+0000 7f4623400640 1 -- 172.21.15.6:0/916075993 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f4614058d90 con 0x7f462c0d05c0 2024-03-06T07:39:03.785 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:03.785 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:03.785 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f460806b4d0 msgr2=0x7f460806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f460806b4d0 0x7f460806d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f461c036000 tx=0x7f461c03c040 comp rx=0 tx=0).stop 2024-03-06T07:39:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 msgr2=0x7f462c0b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:03.787 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 0x7f462c0b6910 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f461400e840 tx=0x7f461400ed00 comp rx=0 tx=0).stop 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 shutdown_connections 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f460806b4d0 0x7f460806d990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f462c15e070 0x7f462c0b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f462c0d05c0 0x7f462c0b6910 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 --2- 172.21.15.6:0/916075993 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f462c0cf410 0x7f462c15c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 >> 172.21.15.6:0/916075993 conn(0x7f462c08d2a0 msgr2=0x7f462c15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 shutdown_connections 2024-03-06T07:39:03.788 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:03.785+0000 7f462be00640 1 -- 172.21.15.6:0/916075993 wait complete. 2024-03-06T07:39:04.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:04.010 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:04.235 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:05.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:05.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.231+0000 7f429e000640 1 Processor -- start 2024-03-06T07:39:05.233 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.231+0000 7f429e000640 1 -- start start 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.232+0000 7f429e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 0x7f429815f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.232+0000 7f429e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f429815fef0 0x7f429806f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.232+0000 7f429e000640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42980b0690 con 0x7f429815bc60 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f429e000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4298160360 con 0x7f429815fef0 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f429e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42980afe10 con 0x7f429815c040 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f4297400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f429815bc60 0x7f42980af2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57636/0 (socket says 172.21.15.6:57636) 2024-03-06T07:39:05.235 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f4297400640 1 -- 172.21.15.6:0/2494492068 learned_addr learned my addr 172.21.15.6:0/2494492068 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:05.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f4297e00640 1 --2- 172.21.15.6:0/2494492068 >> v2:172.21.15.178:3300/0 conn(0x7f429815fef0 0x7f429806f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f428ea00640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 0x7f429815f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763951515 0 0) 0x7f42980b0690 con 0x7f429815bc60 2024-03-06T07:39:05.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.233+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c003520 con 0x7f429815bc60 2024-03-06T07:39:05.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484357297 0 0) 0x7f427c003520 con 0x7f429815bc60 2024-03-06T07:39:05.237 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 msgr2=0x7f429815f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 0x7f429815f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 >> v2:172.21.15.178:3300/0 conn(0x7f429815fef0 msgr2=0x7f429806f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 --2- 172.21.15.6:0/2494492068 >> v2:172.21.15.178:3300/0 conn(0x7f429815fef0 0x7f429806f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42980b00d0 con 0x7f429815bc60 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f4288001670 con 0x7f429815bc60 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 >> v1:172.21.15.6:6789/0 conn(0x7f429815bc60 legacy=0x7f42980af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f427c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f427c004800 0x7f427c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 0x7f427c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.238 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4298160360 con 0x7f427c003a20 2024-03-06T07:39:05.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42980afe10 con 0x7f427c004800 2024-03-06T07:39:05.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c00ba40 con 0x7f429815c040 2024-03-06T07:39:05.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.234+0000 7f4297400640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f427c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.235+0000 7f4297400640 1 -- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 msgr2=0x7f427c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:05.241 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.235+0000 7f4297400640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f429815c040 0x7f427c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.235+0000 7f428ea00640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f427c004800 0x7f427c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.235+0000 7f4297400640 1 -- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f427c004800 msgr2=0x7f427c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.235+0000 7f4297400640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f427c004800 0x7f427c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.235+0000 7f4297400640 1 -- 172.21.15.6:0/2494492068 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c003520 con 0x7f427c003a20 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.236+0000 7f4297400640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f427c003e40 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f428800c240 tx=0x7f427c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8656af8530e220c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.236+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288007d30 con 0x7f427c003a20 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.236+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4288030070 con 0x7f427c003a20 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.236+0000 7f4296a00640 1 -- 172.21.15.6:0/2494492068 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428802cd60 con 0x7f427c003a20 2024-03-06T07:39:05.242 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f429e000640 1 -- 172.21.15.6:0/2494492068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 msgr2=0x7f427c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f429e000640 1 --2- 172.21.15.6:0/2494492068 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f427c003e40 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f428800c240 tx=0x7f427c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f4297e00640 1 -- 172.21.15.6:0/2494492068 reap_dead start 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f429e000640 1 -- 172.21.15.6:0/2494492068 shutdown_connections 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f429e000640 1 -- 172.21.15.6:0/2494492068 >> 172.21.15.6:0/2494492068 conn(0x7f429809aa10 msgr2=0x7f429809ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f429e000640 1 -- 172.21.15.6:0/2494492068 shutdown_connections 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.237+0000 7f429e000640 1 -- 172.21.15.6:0/2494492068 wait complete. 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.239+0000 7f429e000640 1 Processor -- start 2024-03-06T07:39:05.243 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.239+0000 7f429e000640 1 -- start start 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f429e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f4298141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f429e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4298158040 0x7f4298159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f429e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f429815a840 0x7f42981e74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f429e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42980af9c0 con 0x7f427c003a20 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f429e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4298160710 con 0x7f429815a840 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f429e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42980afb40 con 0x7f4298158040 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f4297400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f4298141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.246 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f4297400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f4298141830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38708/0 (socket says 172.21.15.6:38708) 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.240+0000 7f4297400640 1 -- 172.21.15.6:0/1046423191 learned_addr learned my addr 172.21.15.6:0/1046423191 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297e00640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f429815a840 0x7f42981e74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f428ea00640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4298158040 0x7f4298159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297400640 1 -- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4298158040 msgr2=0x7f4298159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297400640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4298158040 0x7f4298159570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297400640 1 -- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f429815a840 msgr2=0x7f42981e74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297400640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f429815a840 0x7f42981e74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297400640 1 -- 172.21.15.6:0/1046423191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298159ab0 con 0x7f427c003a20 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.241+0000 7f4297400640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f4298141830 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f428800c240 tx=0x7f42980b02a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:05.247 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.242+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288007d30 con 0x7f427c003a20 2024-03-06T07:39:05.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.242+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4288004bf0 con 0x7f427c003a20 2024-03-06T07:39:05.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.242+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288001c70 con 0x7f427c003a20 2024-03-06T07:39:05.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.242+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42981584d0 con 0x7f427c003a20 2024-03-06T07:39:05.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.242+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4298158a90 con 0x7f427c003a20 2024-03-06T07:39:05.248 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.243+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429804f3a0 con 0x7f427c003a20 2024-03-06T07:39:05.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.245+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4288047020 con 0x7f427c003a20 2024-03-06T07:39:05.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.246+0000 7f428fe00640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f427406b4d0 0x7f427406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:05.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.246+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f42880ab390 con 0x7f427c003a20 2024-03-06T07:39:05.249 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.247+0000 7f428ea00640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f427406b4d0 0x7f427406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:05.250 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.248+0000 7f428ea00640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f427406b4d0 0x7f427406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f42981f5270 tx=0x7f4280008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:05.258 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.256+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f428807a1c0 con 0x7f427c003a20 2024-03-06T07:39:05.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.498+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f429809b630 con 0x7f427c003a20 2024-03-06T07:39:05.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.499+0000 7f428fe00640 1 -- 172.21.15.6:0/1046423191 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f4288079ac0 con 0x7f427c003a20 2024-03-06T07:39:05.502 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:05.502 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:05.502 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:05.503 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.501+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f427406b4d0 msgr2=0x7f427406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.501+0000 7f429e000640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f427406b4d0 0x7f427406d990 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f42981f5270 tx=0x7f4280008040 comp rx=0 tx=0).stop 2024-03-06T07:39:05.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.501+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 msgr2=0x7f4298141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:05.504 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.501+0000 7f429e000640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f4298141830 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f428800c240 tx=0x7f42980b02a0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 shutdown_connections 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f427406b4d0 0x7f427406d990 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f429815a840 0x7f42981e74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f4298158040 0x7f4298159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 --2- 172.21.15.6:0/1046423191 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f427c003a20 0x7f4298141830 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 >> 172.21.15.6:0/1046423191 conn(0x7f429809aa10 msgr2=0x7f42980b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 shutdown_connections 2024-03-06T07:39:05.505 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:05.502+0000 7f429e000640 1 -- 172.21.15.6:0/1046423191 wait complete. 2024-03-06T07:39:05.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:05.727 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:06.237 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:06.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:06.948 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.946+0000 7f1e2ca00640 1 Processor -- start 2024-03-06T07:39:06.949 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.946+0000 7f1e2ca00640 1 -- start start 2024-03-06T07:39:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.948+0000 7f1e2ca00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e2815e0b0 0x7f1e280cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.951 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.948+0000 7f1e2ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e280d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.948+0000 7f1e2ca00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1e280d0c80 0x7f1e280d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.948+0000 7f1e2ca00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e280b09e0 con 0x7f1e2815e0b0 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.948+0000 7f1e2ca00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e280b3710 con 0x7f1e280d0c80 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.948+0000 7f1e2ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280b39f0 con 0x7f1e280d4a10 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e2815e0b0 0x7f1e280cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1e2815e0b0 0x7f1e280cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38710/0 (socket says 172.21.15.6:38710) 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 -- 172.21.15.6:0/3178779631 learned_addr learned my addr 172.21.15.6:0/3178779631 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e27200640 1 --2- 172.21.15.6:0/3178779631 >> v2:172.21.15.178:3300/0 conn(0x7f1e280d0c80 0x7f1e280d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e25e00640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e280d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.952 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 -- 172.21.15.6:0/3178779631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 msgr2=0x7f1e280d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e280d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 -- 172.21.15.6:0/3178779631 >> v2:172.21.15.178:3300/0 conn(0x7f1e280d0c80 msgr2=0x7f1e280d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 --2- 172.21.15.6:0/3178779631 >> v2:172.21.15.178:3300/0 conn(0x7f1e280d0c80 0x7f1e280d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e26800640 1 -- 172.21.15.6:0/3178779631 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280b3cd0 con 0x7f1e2815e0b0 2024-03-06T07:39:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e27200640 1 --2- 172.21.15.6:0/3178779631 >> v2:172.21.15.178:3300/0 conn(0x7f1e280d0c80 0x7f1e280d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:06.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.949+0000 7f1e25e00640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e280d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.950+0000 7f1e26800640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e2815e0b0 0x7f1e280cf310 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1e280d06c0 tx=0x7f1e1802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a53351bb03c1158 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e25400640 1 -- 172.21.15.6:0/3178779631 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18035020 con 0x7f1e2815e0b0 2024-03-06T07:39:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e25400640 1 -- 172.21.15.6:0/3178779631 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e18031880 con 0x7f1e2815e0b0 2024-03-06T07:39:06.954 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e25400640 1 -- 172.21.15.6:0/3178779631 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18031c50 con 0x7f1e2815e0b0 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 -- 172.21.15.6:0/3178779631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e2815e0b0 msgr2=0x7f1e280cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e2815e0b0 0x7f1e280cf310 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f1e280d06c0 tx=0x7f1e1802b920 comp rx=0 tx=0).stop 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 -- 172.21.15.6:0/3178779631 shutdown_connections 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/3178779631 >> v2:172.21.15.178:3300/0 conn(0x7f1e280d0c80 0x7f1e280d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e280d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/3178779631 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e2815e0b0 0x7f1e280cf310 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.951+0000 7f1e2ca00640 1 -- 172.21.15.6:0/3178779631 >> 172.21.15.6:0/3178779631 conn(0x7f1e280d3d30 msgr2=0x7f1e280d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.952+0000 7f1e2ca00640 1 -- 172.21.15.6:0/3178779631 shutdown_connections 2024-03-06T07:39:06.955 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.952+0000 7f1e2ca00640 1 -- 172.21.15.6:0/3178779631 wait complete. 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.953+0000 7f1e2ca00640 1 Processor -- start 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.953+0000 7f1e2ca00640 1 -- start start 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.954+0000 7f1e2ca00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 0x7f1e280a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.954+0000 7f1e2ca00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e2815d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.954+0000 7f1e2ca00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e2815e0b0 0x7f1e281f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.954+0000 7f1e2ca00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280cfc60 con 0x7f1e280d0c80 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.954+0000 7f1e2ca00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2815e490 con 0x7f1e2815e0b0 2024-03-06T07:39:06.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.954+0000 7f1e2ca00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e280cf850 con 0x7f1e280d4a10 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 0x7f1e280a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 0x7f1e280a1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38720/0 (socket says 172.21.15.6:38720) 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 -- 172.21.15.6:0/1626833087 learned_addr learned my addr 172.21.15.6:0/1626833087 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 -- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 msgr2=0x7f1e2815d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e2815d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 -- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e2815e0b0 msgr2=0x7f1e281f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e2815e0b0 0x7f1e281f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 -- 172.21.15.6:0/1626833087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280caea0 con 0x7f1e280d0c80 2024-03-06T07:39:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.955+0000 7f1e26800640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 0x7f1e280a1460 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f1e1802b960 tx=0x7f1e180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.956+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18040070 con 0x7f1e280d0c80 2024-03-06T07:39:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.956+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1803a420 con 0x7f1e280d0c80 2024-03-06T07:39:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.956+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18035040 con 0x7f1e280d0c80 2024-03-06T07:39:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.956+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e280cb180 con 0x7f1e280d0c80 2024-03-06T07:39:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.956+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e280cb740 con 0x7f1e280d0c80 2024-03-06T07:39:06.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.957+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2804f330 con 0x7f1e280d0c80 2024-03-06T07:39:06.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.958+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e1804b060 con 0x7f1e280d0c80 2024-03-06T07:39:06.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.960+0000 7f1e1ea00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e0806b500 0x7f1e0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:06.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.960+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1e180ac1c0 con 0x7f1e280d0c80 2024-03-06T07:39:06.962 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.960+0000 7f1e25e00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e0806b500 0x7f1e0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:06.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.961+0000 7f1e25e00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e0806b500 0x7f1e0806d9c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1e281601a0 tx=0x7f1e14008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:06.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:06.964+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e1807aff0 con 0x7f1e280d0c80 2024-03-06T07:39:07.213 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.211+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e280cdd10 con 0x7f1e280d0c80 2024-03-06T07:39:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.212+0000 7f1e1ea00640 1 -- 172.21.15.6:0/1626833087 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f1e18030e90 con 0x7f1e280d0c80 2024-03-06T07:39:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:07.214 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:07.214 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.214+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e0806b500 msgr2=0x7f1e0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.214+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e0806b500 0x7f1e0806d9c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f1e281601a0 tx=0x7f1e14008040 comp rx=0 tx=0).stop 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.214+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 msgr2=0x7f1e280a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.214+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 0x7f1e280a1460 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f1e1802b960 tx=0x7f1e180046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 shutdown_connections 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1e0806b500 0x7f1e0806d9c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1e2815e0b0 0x7f1e281f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1e280d4a10 0x7f1e2815d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 --2- 172.21.15.6:0/1626833087 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e280d0c80 0x7f1e280a1460 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 >> 172.21.15.6:0/1626833087 conn(0x7f1e280d3d30 msgr2=0x7f1e280b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:07.217 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 shutdown_connections 2024-03-06T07:39:07.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:07.215+0000 7f1e2ca00640 1 -- 172.21.15.6:0/1626833087 wait complete. 2024-03-06T07:39:07.437 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:07.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:08.239 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:08.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:08.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.664+0000 7f2827200640 1 Processor -- start 2024-03-06T07:39:08.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.664+0000 7f2827200640 1 -- start start 2024-03-06T07:39:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2827200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28200af580 0x7f282015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2827200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28200b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2827200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282015cc40 con 0x7f28200af1a0 2024-03-06T07:39:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2827200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282015cf20 con 0x7f28200af580 2024-03-06T07:39:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2827200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282015d200 con 0x7f28200cdc20 2024-03-06T07:39:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2817400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28200af580 0x7f282015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2817400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f28200af580 0x7f282015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38732/0 (socket says 172.21.15.6:38732) 2024-03-06T07:39:08.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.666+0000 7f2817400640 1 -- 172.21.15.6:0/3922954211 learned_addr learned my addr 172.21.15.6:0/3922954211 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.667+0000 7f2825200640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28200b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.667+0000 7f2817400640 1 -- 172.21.15.6:0/3922954211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 msgr2=0x7f28200b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.667+0000 7f2817400640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28200b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.667+0000 7f2817400640 1 -- 172.21.15.6:0/3922954211 >> v1:172.21.15.6:6789/0 conn(0x7f28200af1a0 legacy=0x7f28200b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.667+0000 7f2817400640 1 -- 172.21.15.6:0/3922954211 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282015d4e0 con 0x7f28200af580 2024-03-06T07:39:08.670 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.667+0000 7f2825200640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28200b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.668+0000 7f2817400640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af580 0x7f282015c600 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f2808009870 tx=0x7f280802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f679c4a38ed650b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.668+0000 7f281f400640 1 -- 172.21.15.6:0/3922954211 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808035020 con 0x7f28200af580 2024-03-06T07:39:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.668+0000 7f281f400640 1 -- 172.21.15.6:0/3922954211 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2808031640 con 0x7f28200af580 2024-03-06T07:39:08.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.668+0000 7f281f400640 1 -- 172.21.15.6:0/3922954211 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280802d040 con 0x7f28200af580 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 -- 172.21.15.6:0/3922954211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af580 msgr2=0x7f282015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af580 0x7f282015c600 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f2808009870 tx=0x7f280802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 -- 172.21.15.6:0/3922954211 shutdown_connections 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28200b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 --2- 172.21.15.6:0/3922954211 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af580 0x7f282015c600 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 -- 172.21.15.6:0/3922954211 >> 172.21.15.6:0/3922954211 conn(0x7f282006afb0 msgr2=0x7f282006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 -- 172.21.15.6:0/3922954211 shutdown_connections 2024-03-06T07:39:08.672 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.669+0000 7f2827200640 1 -- 172.21.15.6:0/3922954211 wait complete. 2024-03-06T07:39:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.671+0000 7f2827200640 1 Processor -- start 2024-03-06T07:39:08.673 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.671+0000 7f2827200640 1 -- start start 2024-03-06T07:39:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f2827200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 0x7f28201f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.674 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f2827200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200af580 0x7f28200b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f2827200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28201f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f2827200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28200b3720 con 0x7f28200af1a0 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f2827200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28200b35a0 con 0x7f28200af580 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f2827200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2820155e70 con 0x7f28200cdc20 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f281fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 0x7f28201f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f281fe00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 0x7f28201f7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38748/0 (socket says 172.21.15.6:38748) 2024-03-06T07:39:08.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.672+0000 7f281fe00640 1 -- 172.21.15.6:0/1631089699 learned_addr learned my addr 172.21.15.6:0/1631089699 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f2825200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28201f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f281fe00640 1 -- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 msgr2=0x7f28201f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f281fe00640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28201f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f281fe00640 1 -- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200af580 msgr2=0x7f28200b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f281fe00640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200af580 0x7f28200b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f281fe00640 1 -- 172.21.15.6:0/1631089699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28201f6f70 con 0x7f28200af1a0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.673+0000 7f281fe00640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 0x7f28201f7a10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f281000c400 tx=0x7f281000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.674+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28100052d0 con 0x7f28200af1a0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.674+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2810005490 con 0x7f28200af1a0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.674+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810007260 con 0x7f28200af1a0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.674+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282015b310 con 0x7f28200af1a0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.674+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282015c2a0 con 0x7f28200af1a0 2024-03-06T07:39:08.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.675+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2820069510 con 0x7f28200af1a0 2024-03-06T07:39:08.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.676+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2810007400 con 0x7f28200af1a0 2024-03-06T07:39:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.677+0000 7f281cc00640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.677+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2810089740 con 0x7f28200af1a0 2024-03-06T07:39:08.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.678+0000 7f2817400640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:08.681 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.679+0000 7f2817400640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27fc06b500 0x7f27fc06d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2808036000 tx=0x7f2808030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:08.684 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.682+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2810058570 con 0x7f28200af1a0 2024-03-06T07:39:08.922 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.920+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28201f3780 con 0x7f28200af1a0 2024-03-06T07:39:08.923 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.921+0000 7f281cc00640 1 -- 172.21.15.6:0/1631089699 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f2810057e70 con 0x7f28200af1a0 2024-03-06T07:39:08.923 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:08.924 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:08.924 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:08.927 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27fc06b500 msgr2=0x7f27fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27fc06b500 0x7f27fc06d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f2808036000 tx=0x7f2808030000 comp rx=0 tx=0).stop 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 msgr2=0x7f28201f7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 0x7f28201f7a10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f281000c400 tx=0x7f281000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 shutdown_connections 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f27fc06b500 0x7f27fc06d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f28200af580 0x7f28200b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f28200cdc20 0x7f28201f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 --2- 172.21.15.6:0/1631089699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f28200af1a0 0x7f28201f7a10 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 >> 172.21.15.6:0/1631089699 conn(0x7f282006afb0 msgr2=0x7f2820153fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.925+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 shutdown_connections 2024-03-06T07:39:08.928 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:08.926+0000 7f2827200640 1 -- 172.21.15.6:0/1631089699 wait complete. 2024-03-06T07:39:09.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:09.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:10.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:10.241 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.379+0000 7f6963200640 1 Processor -- start 2024-03-06T07:39:10.381 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.379+0000 7f6963200640 1 -- start start 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.380+0000 7f6963200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f69640cf080 0x7f69640d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.380+0000 7f6963200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f69640d05e0 0x7f696415d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.380+0000 7f6963200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69640b1370 con 0x7f69640b0f90 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.380+0000 7f6963200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69640d0a10 con 0x7f69640cf080 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.380+0000 7f6963200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69640ccf80 con 0x7f69640d05e0 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f6961e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f69640b0f90 0x7f69640cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57654/0 (socket says 172.21.15.6:57654) 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f6961e00640 1 -- 172.21.15.6:0/2111319654 learned_addr learned my addr 172.21.15.6:0/2111319654 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:10.383 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f695be00640 1 --2- 172.21.15.6:0/2111319654 >> v2:172.21.15.6:3300/0 conn(0x7f69640cf080 0x7f69640d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f6962800640 1 --2- 172.21.15.6:0/2111319654 >> v2:172.21.15.178:3300/0 conn(0x7f69640d05e0 0x7f696415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f695be00640 1 -- 172.21.15.6:0/2111319654 >> v2:172.21.15.178:3300/0 conn(0x7f69640d05e0 msgr2=0x7f696415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f695be00640 1 --2- 172.21.15.6:0/2111319654 >> v2:172.21.15.178:3300/0 conn(0x7f69640d05e0 0x7f696415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.381+0000 7f695be00640 1 -- 172.21.15.6:0/2111319654 >> v1:172.21.15.6:6789/0 conn(0x7f69640b0f90 legacy=0x7f69640cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.382+0000 7f6961400640 1 -- 172.21.15.6:0/2111319654 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301245989 0 0) 0x7f69640b1370 con 0x7f69640b0f90 2024-03-06T07:39:10.384 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.382+0000 7f695be00640 1 -- 172.21.15.6:0/2111319654 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69640cd260 con 0x7f69640cf080 2024-03-06T07:39:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.383+0000 7f695be00640 1 --2- 172.21.15.6:0/2111319654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f69640d00a0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f6964072470 tx=0x7f694c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea44747d3f696591 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.383+0000 7f6961400640 1 -- 172.21.15.6:0/2111319654 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c0315a0 con 0x7f69640cf080 2024-03-06T07:39:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.383+0000 7f6961400640 1 -- 172.21.15.6:0/2111319654 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c037070 con 0x7f69640cf080 2024-03-06T07:39:10.385 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.383+0000 7f6961400640 1 -- 172.21.15.6:0/2111319654 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c031960 con 0x7f69640cf080 2024-03-06T07:39:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 -- 172.21.15.6:0/2111319654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 msgr2=0x7f69640d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 --2- 172.21.15.6:0/2111319654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f69640d00a0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f6964072470 tx=0x7f694c02b840 comp rx=0 tx=0).stop 2024-03-06T07:39:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 -- 172.21.15.6:0/2111319654 shutdown_connections 2024-03-06T07:39:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 --2- 172.21.15.6:0/2111319654 >> v2:172.21.15.178:3300/0 conn(0x7f69640d05e0 0x7f696415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.386 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 --2- 172.21.15.6:0/2111319654 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f69640d00a0 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 -- 172.21.15.6:0/2111319654 >> 172.21.15.6:0/2111319654 conn(0x7f69640d1ae0 msgr2=0x7f69640d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:10.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 -- 172.21.15.6:0/2111319654 shutdown_connections 2024-03-06T07:39:10.387 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.384+0000 7f6963200640 1 -- 172.21.15.6:0/2111319654 wait complete. 2024-03-06T07:39:10.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.386+0000 7f6963200640 1 Processor -- start 2024-03-06T07:39:10.388 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.386+0000 7f6963200640 1 -- start start 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6963200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f69640b0f90 0x7f69640cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6963200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f696414f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6963200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69640d05e0 0x7f69640afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6963200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696415dc90 con 0x7f69640cf080 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6963200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69640cd820 con 0x7f69640d05e0 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6963200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696415de10 con 0x7f69640b0f90 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f695be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f696414f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.389 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f695be00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f696414f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38772/0 (socket says 172.21.15.6:38772) 2024-03-06T07:39:10.390 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f695be00640 1 -- 172.21.15.6:0/2374510327 learned_addr learned my addr 172.21.15.6:0/2374510327 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6962800640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69640d05e0 0x7f69640afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.387+0000 7f6961e00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f69640b0f90 0x7f69640cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f695be00640 1 -- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f69640b0f90 msgr2=0x7f69640cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f695be00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f69640b0f90 0x7f69640cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f695be00640 1 -- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69640d05e0 msgr2=0x7f69640afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f695be00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69640d05e0 0x7f69640afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f695be00640 1 -- 172.21.15.6:0/2374510327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696415e6d0 con 0x7f69640cf080 2024-03-06T07:39:10.391 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f6961e00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f69640b0f90 0x7f69640cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.388+0000 7f695be00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f696414f050 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f694c009890 tx=0x7f694c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.389+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c004230 con 0x7f69640cf080 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.389+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c0043d0 con 0x7f69640cf080 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.389+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c003850 con 0x7f69640cf080 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.389+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696415e9b0 con 0x7f69640cf080 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.389+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696415ef70 con 0x7f69640cf080 2024-03-06T07:39:10.392 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.390+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696404f330 con 0x7f69640cf080 2024-03-06T07:39:10.393 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.391+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f694c0039f0 con 0x7f69640cf080 2024-03-06T07:39:10.394 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.392+0000 7f695a000640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694006b4d0 0x7f694006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:10.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.393+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f694c02f030 con 0x7f69640cf080 2024-03-06T07:39:10.395 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.393+0000 7f6961e00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694006b4d0 0x7f694006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:10.396 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.394+0000 7f6961e00640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694006b4d0 0x7f694006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6954000f60 tx=0x7f6954009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:10.399 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.398+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f694c041070 con 0x7f69640cf080 2024-03-06T07:39:10.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.637+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f696406de10 con 0x7f69640cf080 2024-03-06T07:39:10.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.638+0000 7f695a000640 1 -- 172.21.15.6:0/2374510327 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f694c07b390 con 0x7f69640cf080 2024-03-06T07:39:10.640 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:10.641 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:10.641 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.640+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694006b4d0 msgr2=0x7f694006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.640+0000 7f6963200640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694006b4d0 0x7f694006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f6954000f60 tx=0x7f6954009040 comp rx=0 tx=0).stop 2024-03-06T07:39:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 msgr2=0x7f696414f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:10.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f696414f050 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f694c009890 tx=0x7f694c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 shutdown_connections 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f694006b4d0 0x7f694006d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f69640d05e0 0x7f69640afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69640cf080 0x7f696414f050 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 --2- 172.21.15.6:0/2374510327 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f69640b0f90 0x7f69640cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 >> 172.21.15.6:0/2374510327 conn(0x7f69640d1ae0 msgr2=0x7f696406fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 shutdown_connections 2024-03-06T07:39:10.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:10.641+0000 7f6963200640 1 -- 172.21.15.6:0/2374510327 wait complete. 2024-03-06T07:39:10.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:10.863 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:11.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.092+0000 7f7bb0800640 1 Processor -- start 2024-03-06T07:39:12.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.093+0000 7f7bb0800640 1 -- start start 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.094+0000 7f7bb0800640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ba815e2e0 0x7f7ba80d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.094+0000 7f7bb0800640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f7ba80b50e0 0x7f7ba80d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.094+0000 7f7bb0800640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80b3940 con 0x7f7ba80d0d70 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.094+0000 7f7bb0800640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba80d4e70 con 0x7f7ba815e2e0 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.094+0000 7f7bb0800640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba80b3c40 con 0x7f7ba80b50e0 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bade00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7ba815e2e0 0x7f7ba80d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:12.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bad400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f7ba80d0d70 0x7f7ba80d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50194/0 (socket says 172.21.15.6:50194) 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bad400640 1 -- 172.21.15.6:0/1393060541 learned_addr learned my addr 172.21.15.6:0/1393060541 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bade00640 1 -- 172.21.15.6:0/1393060541 >> v2:172.21.15.178:3300/0 conn(0x7f7ba80b50e0 msgr2=0x7f7ba80d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bae800640 1 --2- 172.21.15.6:0/1393060541 >> v2:172.21.15.178:3300/0 conn(0x7f7ba80b50e0 0x7f7ba80d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bade00640 1 --2- 172.21.15.6:0/1393060541 >> v2:172.21.15.178:3300/0 conn(0x7f7ba80b50e0 0x7f7ba80d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bade00640 1 -- 172.21.15.6:0/1393060541 >> v1:172.21.15.6:6789/0 conn(0x7f7ba80d0d70 legacy=0x7f7ba80d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7baca00640 1 -- 172.21.15.6:0/1393060541 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373568707 0 0) 0x7f7ba80b3940 con 0x7f7ba80d0d70 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bade00640 1 -- 172.21.15.6:0/1393060541 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba80cf580 con 0x7f7ba815e2e0 2024-03-06T07:39:12.098 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.095+0000 7f7bae800640 1 --2- 172.21.15.6:0/1393060541 >> v2:172.21.15.178:3300/0 conn(0x7f7ba80b50e0 0x7f7ba80d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:12.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.096+0000 7f7bade00640 1 --2- 172.21.15.6:0/1393060541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80d0830 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f7ba808e630 tx=0x7f7b9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58be54460c258fdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:12.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.097+0000 7f7baca00640 1 -- 172.21.15.6:0/1393060541 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c035020 con 0x7f7ba815e2e0 2024-03-06T07:39:12.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.097+0000 7f7baca00640 1 -- 172.21.15.6:0/1393060541 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c031830 con 0x7f7ba815e2e0 2024-03-06T07:39:12.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.097+0000 7f7baca00640 1 -- 172.21.15.6:0/1393060541 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c031c00 con 0x7f7ba815e2e0 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.097+0000 7f7bb0800640 1 -- 172.21.15.6:0/1393060541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 msgr2=0x7f7ba80d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.097+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1393060541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80d0830 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f7ba808e630 tx=0x7f7b9c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.098+0000 7f7bb0800640 1 -- 172.21.15.6:0/1393060541 shutdown_connections 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.098+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1393060541 >> v2:172.21.15.178:3300/0 conn(0x7f7ba80b50e0 0x7f7ba80d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.098+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1393060541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80d0830 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.098+0000 7f7bb0800640 1 -- 172.21.15.6:0/1393060541 >> 172.21.15.6:0/1393060541 conn(0x7f7ba80d3f60 msgr2=0x7f7ba80d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.098+0000 7f7bb0800640 1 -- 172.21.15.6:0/1393060541 shutdown_connections 2024-03-06T07:39:12.100 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.098+0000 7f7bb0800640 1 -- 172.21.15.6:0/1393060541 wait complete. 2024-03-06T07:39:12.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.099+0000 7f7bb0800640 1 Processor -- start 2024-03-06T07:39:12.101 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.100+0000 7f7bb0800640 1 -- start start 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.100+0000 7f7bb0800640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ba80b50e0 0x7f7ba814d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:12.103 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.100+0000 7f7bb0800640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7ba80d0d70 0x7f7ba80ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bb0800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bb0800640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba80cfe20 con 0x7f7ba815e2e0 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bb0800640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba80cfb40 con 0x7f7ba80b50e0 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bb0800640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba80d2240 con 0x7f7ba80d0d70 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45940/0 (socket says 172.21.15.6:45940) 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 -- 172.21.15.6:0/1842826595 learned_addr learned my addr 172.21.15.6:0/1842826595 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bad400640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7ba80d0d70 0x7f7ba80ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bade00640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ba80b50e0 0x7f7ba814d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 -- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7ba80d0d70 msgr2=0x7f7ba80ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7ba80d0d70 0x7f7ba80ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 -- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ba80b50e0 msgr2=0x7f7ba814d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ba80b50e0 0x7f7ba814d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bae800640 1 -- 172.21.15.6:0/1842826595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba814df20 con 0x7f7ba815e2e0 2024-03-06T07:39:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.101+0000 7f7bad400640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7ba80d0d70 0x7f7ba80ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:12.105 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.102+0000 7f7bae800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80b25a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7ba400ced0 tx=0x7f7ba400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.102+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4003b90 con 0x7f7ba815e2e0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.102+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba4003d50 con 0x7f7ba815e2e0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.102+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4004260 con 0x7f7ba815e2e0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.102+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba80c9410 con 0x7f7ba815e2e0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.102+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba80c99d0 con 0x7f7ba815e2e0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.104+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba804f3a0 con 0x7f7ba815e2e0 2024-03-06T07:39:12.106 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.105+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ba4004400 con 0x7f7ba815e2e0 2024-03-06T07:39:12.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.106+0000 7f7ba2000640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:12.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.106+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7ba408ac80 con 0x7f7ba815e2e0 2024-03-06T07:39:12.108 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.106+0000 7f7bade00640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:12.110 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.108+0000 7f7bade00640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c036000 tx=0x7f7b9c008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:12.113 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.111+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ba4059af0 con 0x7f7ba815e2e0 2024-03-06T07:39:12.243 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:12.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.409+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ba80d1bd0 con 0x7f7ba815e2e0 2024-03-06T07:39:12.412 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.409+0000 7f7ba2000640 1 -- 172.21.15.6:0/1842826595 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f7ba40593f0 con 0x7f7ba815e2e0 2024-03-06T07:39:12.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:12.413 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:12.413 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b8c06b4d0 msgr2=0x7f7b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c036000 tx=0x7f7b9c008030 comp rx=0 tx=0).stop 2024-03-06T07:39:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 msgr2=0x7f7ba80b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80b25a0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7ba400ced0 tx=0x7f7ba400a710 comp rx=0 tx=0).stop 2024-03-06T07:39:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 shutdown_connections 2024-03-06T07:39:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7b8c06b4d0 0x7f7b8c06d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7ba815e2e0 0x7f7ba80b25a0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7ba80d0d70 0x7f7ba80ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 --2- 172.21.15.6:0/1842826595 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7ba80b50e0 0x7f7ba814d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 >> 172.21.15.6:0/1842826595 conn(0x7f7ba80d3f60 msgr2=0x7f7ba815d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 shutdown_connections 2024-03-06T07:39:12.415 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:12.412+0000 7f7bb0800640 1 -- 172.21.15.6:0/1842826595 wait complete. 2024-03-06T07:39:12.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:12.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:13.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:13.870 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.868+0000 7fc897c00640 1 Processor -- start 2024-03-06T07:39:13.871 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.868+0000 7fc897c00640 1 -- start start 2024-03-06T07:39:13.872 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.870+0000 7fc897c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc89015e2e0 0x7fc8900d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.870+0000 7fc897c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8900b50e0 0x7fc8900d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.870+0000 7fc897c00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8900b3940 con 0x7fc8900d0d70 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.870+0000 7fc897c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8900d4e70 con 0x7fc8900b50e0 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.870+0000 7fc897c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900b3c40 con 0x7fc89015e2e0 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.870+0000 7fc88fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc8900d0d70 0x7fc8900d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50208/0 (socket says 172.21.15.6:50208) 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc88fe00640 1 -- 172.21.15.6:0/1545659945 learned_addr learned my addr 172.21.15.6:0/1545659945 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:13.873 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc895c00640 1 --2- 172.21.15.6:0/1545659945 >> v2:172.21.15.6:3300/0 conn(0x7fc8900b50e0 0x7fc8900d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc895200640 1 --2- 172.21.15.6:0/1545659945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc89015e2e0 0x7fc8900d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc895c00640 1 -- 172.21.15.6:0/1545659945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc89015e2e0 msgr2=0x7fc8900d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc895c00640 1 --2- 172.21.15.6:0/1545659945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc89015e2e0 0x7fc8900d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc895c00640 1 -- 172.21.15.6:0/1545659945 >> v1:172.21.15.6:6789/0 conn(0x7fc8900d0d70 legacy=0x7fc8900d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc88f400640 1 -- 172.21.15.6:0/1545659945 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526733944 0 0) 0x7fc8900b3940 con 0x7fc8900d0d70 2024-03-06T07:39:13.874 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.871+0000 7fc895c00640 1 -- 172.21.15.6:0/1545659945 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8900cf580 con 0x7fc8900b50e0 2024-03-06T07:39:13.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.872+0000 7fc895c00640 1 --2- 172.21.15.6:0/1545659945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8900b50e0 0x7fc8900d11d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc888009d90 tx=0x7fc88802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46c2d542279ed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.875 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc88f400640 1 -- 172.21.15.6:0/1545659945 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888039070 con 0x7fc8900b50e0 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc88f400640 1 -- 172.21.15.6:0/1545659945 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88802cc50 con 0x7fc8900b50e0 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc88f400640 1 -- 172.21.15.6:0/1545659945 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888034440 con 0x7fc8900b50e0 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 -- 172.21.15.6:0/1545659945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8900b50e0 msgr2=0x7fc8900d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 --2- 172.21.15.6:0/1545659945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8900b50e0 0x7fc8900d11d0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fc888009d90 tx=0x7fc88802bea0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 -- 172.21.15.6:0/1545659945 shutdown_connections 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 --2- 172.21.15.6:0/1545659945 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8900b50e0 0x7fc8900d11d0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 --2- 172.21.15.6:0/1545659945 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc89015e2e0 0x7fc8900d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 -- 172.21.15.6:0/1545659945 >> 172.21.15.6:0/1545659945 conn(0x7fc8900d3f60 msgr2=0x7fc8900d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 -- 172.21.15.6:0/1545659945 shutdown_connections 2024-03-06T07:39:13.876 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.873+0000 7fc897c00640 1 -- 172.21.15.6:0/1545659945 wait complete. 2024-03-06T07:39:13.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.875+0000 7fc897c00640 1 Processor -- start 2024-03-06T07:39:13.877 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.875+0000 7fc897c00640 1 -- start start 2024-03-06T07:39:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.876+0000 7fc897c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8900b50e0 0x7fc8900cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.876+0000 7fc897c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8900d0d70 0x7fc890161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.876+0000 7fc897c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 0x7fc89015a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.876+0000 7fc897c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900cfe20 con 0x7fc89015e2e0 2024-03-06T07:39:13.879 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.876+0000 7fc897c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900cfb40 con 0x7fc8900d0d70 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.876+0000 7fc897c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900d2240 con 0x7fc8900b50e0 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 0x7fc89015a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 0x7fc89015a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45960/0 (socket says 172.21.15.6:45960) 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 -- 172.21.15.6:0/503524127 learned_addr learned my addr 172.21.15.6:0/503524127 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc88fe00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8900d0d70 0x7fc890161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895200640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8900b50e0 0x7fc8900cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 -- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8900b50e0 msgr2=0x7fc8900cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8900b50e0 0x7fc8900cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 -- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8900d0d70 msgr2=0x7fc890161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8900d0d70 0x7fc890161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:13.880 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.877+0000 7fc895c00640 1 -- 172.21.15.6:0/503524127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8900cd170 con 0x7fc89015e2e0 2024-03-06T07:39:13.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.878+0000 7fc895c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 0x7fc89015a4f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fc88802c4b0 tx=0x7fc888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.878+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888039050 con 0x7fc89015e2e0 2024-03-06T07:39:13.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.878+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc888045070 con 0x7fc89015e2e0 2024-03-06T07:39:13.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.878+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88803e610 con 0x7fc89015e2e0 2024-03-06T07:39:13.881 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.878+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8900cd450 con 0x7fc89015e2e0 2024-03-06T07:39:13.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.878+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89015aa30 con 0x7fc89015e2e0 2024-03-06T07:39:13.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.879+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89004f3a0 con 0x7fc89015e2e0 2024-03-06T07:39:13.882 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.880+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc88803e7b0 con 0x7fc89015e2e0 2024-03-06T07:39:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.881+0000 7fc88cc00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc87006b4d0 0x7fc87006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:13.883 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.881+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc8880acd20 con 0x7fc89015e2e0 2024-03-06T07:39:13.884 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.882+0000 7fc895200640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc87006b4d0 0x7fc87006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:13.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.883+0000 7fc895200640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc87006b4d0 0x7fc87006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc884006fd0 tx=0x7fc884008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:13.888 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:13.886+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc88807bc70 con 0x7fc89015e2e0 2024-03-06T07:39:14.191 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.189+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8900682b0 con 0x7fc89015e2e0 2024-03-06T07:39:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.191+0000 7fc88cc00640 1 -- 172.21.15.6:0/503524127 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fc88807b570 con 0x7fc89015e2e0 2024-03-06T07:39:14.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:14.193 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:14.193 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.193+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc87006b4d0 msgr2=0x7fc87006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.193+0000 7fc897c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc87006b4d0 0x7fc87006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc884006fd0 tx=0x7fc884008040 comp rx=0 tx=0).stop 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.193+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 msgr2=0x7fc89015a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.193+0000 7fc897c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 0x7fc89015a4f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fc88802c4b0 tx=0x7fc888004a40 comp rx=0 tx=0).stop 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.193+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 shutdown_connections 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fc87006b4d0 0x7fc87006d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.196 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc8900d0d70 0x7fc890161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc89015e2e0 0x7fc89015a4f0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 --2- 172.21.15.6:0/503524127 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fc8900b50e0 0x7fc8900cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 >> 172.21.15.6:0/503524127 conn(0x7fc8900d3f60 msgr2=0x7fc8900d1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 shutdown_connections 2024-03-06T07:39:14.197 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:14.194+0000 7fc897c00640 1 -- 172.21.15.6:0/503524127 wait complete. 2024-03-06T07:39:14.245 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:14.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:14.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:15.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:15.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.632+0000 7fd6d6400640 1 Processor -- start 2024-03-06T07:39:15.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.632+0000 7fd6d6400640 1 -- start start 2024-03-06T07:39:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6d6400640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd6d00d2b40 0x7fd6d00d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6d6400640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d00b3940 con 0x7fd6d00d1480 2024-03-06T07:39:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6d6400640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d00cf580 con 0x7fd6d015e2e0 2024-03-06T07:39:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6d6400640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d00d4e70 con 0x7fd6d00d2b40 2024-03-06T07:39:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6cfe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd6d00d1480 0x7fd6d00b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50224/0 (socket says 172.21.15.6:50224) 2024-03-06T07:39:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6cfe00640 1 -- 172.21.15.6:0/2057054075 learned_addr learned my addr 172.21.15.6:0/2057054075 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.634+0000 7fd6cea00640 1 --2- 172.21.15.6:0/2057054075 >> v2:172.21.15.6:3300/0 conn(0x7fd6d00d2b40 0x7fd6d00d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.635+0000 7fd6ce000640 1 -- 172.21.15.6:0/2057054075 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850414666 0 0) 0x7fd6d00b3940 con 0x7fd6d00d1480 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.635+0000 7fd6ce000640 1 -- 172.21.15.6:0/2057054075 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b4003520 con 0x7fd6d00d1480 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.635+0000 7fd6cea00640 1 -- 172.21.15.6:0/2057054075 >> v1:172.21.15.178:6789/0 conn(0x7fd6d015e2e0 legacy=0x7fd6d00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.635+0000 7fd6cea00640 1 -- 172.21.15.6:0/2057054075 >> v1:172.21.15.6:6789/0 conn(0x7fd6d00d1480 legacy=0x7fd6d00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.635+0000 7fd6cea00640 1 -- 172.21.15.6:0/2057054075 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d00cf860 con 0x7fd6d00d2b40 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.636+0000 7fd6cea00640 1 --2- 172.21.15.6:0/2057054075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d00d0f40 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0097a0 tx=0x7fd6bc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f763d4b19ca902 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.636+0000 7fd6ce000640 1 -- 172.21.15.6:0/2057054075 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc034070 con 0x7fd6d00d2b40 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.636+0000 7fd6ce000640 1 -- 172.21.15.6:0/2057054075 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc030610 con 0x7fd6d00d2b40 2024-03-06T07:39:15.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.636+0000 7fd6ce000640 1 -- 172.21.15.6:0/2057054075 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0309e0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 -- 172.21.15.6:0/2057054075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 msgr2=0x7fd6d00d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 --2- 172.21.15.6:0/2057054075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d00d0f40 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0097a0 tx=0x7fd6bc02b960 comp rx=0 tx=0).stop 2024-03-06T07:39:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 -- 172.21.15.6:0/2057054075 shutdown_connections 2024-03-06T07:39:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 --2- 172.21.15.6:0/2057054075 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d00d0f40 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.639 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 -- 172.21.15.6:0/2057054075 >> 172.21.15.6:0/2057054075 conn(0x7fd6d00d3f60 msgr2=0x7fd6d00d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:15.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 -- 172.21.15.6:0/2057054075 shutdown_connections 2024-03-06T07:39:15.640 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.637+0000 7fd6d6400640 1 -- 172.21.15.6:0/2057054075 wait complete. 2024-03-06T07:39:15.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.639+0000 7fd6d6400640 1 Processor -- start 2024-03-06T07:39:15.641 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.639+0000 7fd6d6400640 1 -- start start 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6d6400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d0153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6d6400640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6d015e2e0 0x7fd6d00cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6d6400640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd6d00b9630 0x7fd6d00c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6d6400640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00cfe20 con 0x7fd6d00d2b40 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6d6400640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00b3c40 con 0x7fd6d015e2e0 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6d6400640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00b47e0 con 0x7fd6d00b9630 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d0153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d0153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:45980/0 (socket says 172.21.15.6:45980) 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.640+0000 7fd6cf400640 1 -- 172.21.15.6:0/1768976319 learned_addr learned my addr 172.21.15.6:0/1768976319 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cea00640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6d015e2e0 0x7fd6d00cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.643 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cf400640 1 -- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd6d00b9630 msgr2=0x7fd6d00c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cfe00640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd6d00b9630 0x7fd6d00c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cf400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd6d00b9630 0x7fd6d00c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cf400640 1 -- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6d015e2e0 msgr2=0x7fd6d00cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cf400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6d015e2e0 0x7fd6d00cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cf400640 1 -- 172.21.15.6:0/1768976319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d00ccdc0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cea00640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6d015e2e0 0x7fd6d00cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:15.644 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6cf400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d0153610 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000c400 tx=0x7fd6c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0017070 con 0x7fd6d00d2b40 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c00052d0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0003830 con 0x7fd6d00d2b40 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.641+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d00cd0a0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.642+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d00cd660 con 0x7fd6d00d2b40 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.643+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d004f3a0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.645 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.644+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6c001a070 con 0x7fd6d00d2b40 2024-03-06T07:39:15.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.645+0000 7fd6c7400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd6b006b4d0 0x7fd6b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:15.647 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.645+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6c00895d0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.648 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.646+0000 7fd6cea00640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd6b006b4d0 0x7fd6b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:15.649 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.647+0000 7fd6cea00640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd6b006b4d0 0x7fd6b006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc036000 tx=0x7fd6bc03b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:15.652 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.650+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6c0058400 con 0x7fd6d00d2b40 2024-03-06T07:39:15.893 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.892+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6d00b9de0 con 0x7fd6d00d2b40 2024-03-06T07:39:15.895 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.892+0000 7fd6c7400640 1 -- 172.21.15.6:0/1768976319 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fd6c0057d00 con 0x7fd6d00d2b40 2024-03-06T07:39:15.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:15.895 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:15.895 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd6b006b4d0 msgr2=0x7fd6b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd6b006b4d0 0x7fd6b006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc036000 tx=0x7fd6bc03b040 comp rx=0 tx=0).stop 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 msgr2=0x7fd6d0153610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d0153610 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000c400 tx=0x7fd6c000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 shutdown_connections 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd6b006b4d0 0x7fd6b006d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd6d015e2e0 0x7fd6d00cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd6d00b9630 0x7fd6d00c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 --2- 172.21.15.6:0/1768976319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd6d00d2b40 0x7fd6d0153610 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.895+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 >> 172.21.15.6:0/1768976319 conn(0x7fd6d00d3f60 msgr2=0x7fd6d00cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.896+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 shutdown_connections 2024-03-06T07:39:15.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:15.896+0000 7fd6d6400640 1 -- 172.21.15.6:0/1768976319 wait complete. 2024-03-06T07:39:16.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:16.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:16.247 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:17.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.341+0000 7fd584e00640 1 Processor -- start 2024-03-06T07:39:17.344 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.342+0000 7fd584e00640 1 -- start start 2024-03-06T07:39:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.343+0000 7fd584e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 0x7fd5800cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.343+0000 7fd584e00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5800b3710 con 0x7fd5800d0e50 2024-03-06T07:39:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.343+0000 7fd584e00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5800b39f0 con 0x7fd5800d4a10 2024-03-06T07:39:17.345 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.343+0000 7fd584e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800b09e0 con 0x7fd58015e0b0 2024-03-06T07:39:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.343+0000 7fd57fe00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd5800d0e50 0x7fd5800b44d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50230/0 (socket says 172.21.15.6:50230) 2024-03-06T07:39:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.343+0000 7fd57fe00640 1 -- 172.21.15.6:0/1782299848 learned_addr learned my addr 172.21.15.6:0/1782299848 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57f400640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 0x7fd5800cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.347 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645168715 0 0) 0x7fd5800b3710 con 0x7fd5800d0e50 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd564003520 con 0x7fd5800d0e50 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401373081 0 0) 0x7fd564003520 con 0x7fd5800d0e50 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 msgr2=0x7fd5800cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 0x7fd5800cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 >> v1:172.21.15.178:6789/0 conn(0x7fd5800d4a10 legacy=0x7fd5800d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5800b3cd0 con 0x7fd5800d0e50 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7fd578001670 con 0x7fd5800d0e50 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 >> v1:172.21.15.6:6789/0 conn(0x7fd5800d0e50 legacy=0x7fd5800b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5640039e0 0x7fd564003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 0x7fd564004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56400bb50 con 0x7fd564004d40 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56400bcd0 con 0x7fd5640039e0 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.344+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56400bea0 con 0x7fd58015e0b0 2024-03-06T07:39:17.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57fe00640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57f400640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5640039e0 0x7fd564003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57ea00640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 0x7fd564004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57fe00640 1 -- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 msgr2=0x7fd564004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57fe00640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd58015e0b0 0x7fd564004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57fe00640 1 -- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5640039e0 msgr2=0x7fd564003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57fe00640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5640039e0 0x7fd564003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.349 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.345+0000 7fd57fe00640 1 -- 172.21.15.6:0/1782299848 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd564003520 con 0x7fd564004d40 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.346+0000 7fd57fe00640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5640051c0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fd57800c240 tx=0x7fd564005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c86308c29c9ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578007d30 con 0x7fd564004d40 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd578030070 con 0x7fd564004d40 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd57e000640 1 -- 172.21.15.6:0/1782299848 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57802cd40 con 0x7fd564004d40 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd584e00640 1 -- 172.21.15.6:0/1782299848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 msgr2=0x7fd5640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd584e00640 1 --2- 172.21.15.6:0/1782299848 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5640051c0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fd57800c240 tx=0x7fd564005830 comp rx=0 tx=0).stop 2024-03-06T07:39:17.350 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd57fe00640 1 -- 172.21.15.6:0/1782299848 reap_dead start 2024-03-06T07:39:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd584e00640 1 -- 172.21.15.6:0/1782299848 shutdown_connections 2024-03-06T07:39:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd584e00640 1 -- 172.21.15.6:0/1782299848 >> 172.21.15.6:0/1782299848 conn(0x7fd5800d3d30 msgr2=0x7fd5800d4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.347+0000 7fd584e00640 1 -- 172.21.15.6:0/1782299848 shutdown_connections 2024-03-06T07:39:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.348+0000 7fd584e00640 1 -- 172.21.15.6:0/1782299848 wait complete. 2024-03-06T07:39:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.349+0000 7fd584e00640 1 Processor -- start 2024-03-06T07:39:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.349+0000 7fd584e00640 1 -- start start 2024-03-06T07:39:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.350+0000 7fd584e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5800b9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.350+0000 7fd584e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5801f6fa0 0x7fd5801f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.350+0000 7fd584e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5800b7300 0x7fd5800b7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.350+0000 7fd584e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800cfc60 con 0x7fd564004d40 2024-03-06T07:39:17.353 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.350+0000 7fd584e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58015e490 con 0x7fd5800b7300 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.350+0000 7fd584e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800cf310 con 0x7fd5801f6fa0 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5800b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5800b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46000/0 (socket says 172.21.15.6:46000) 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 -- 172.21.15.6:0/3450923527 learned_addr learned my addr 172.21.15.6:0/3450923527 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57ea00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5801f6fa0 0x7fd5801f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 -- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5801f6fa0 msgr2=0x7fd5801f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57fe00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5800b7300 0x7fd5800b7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5801f6fa0 0x7fd5801f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 -- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5800b7300 msgr2=0x7fd5800b7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5800b7300 0x7fd5800b7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 -- 172.21.15.6:0/3450923527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5800a1950 con 0x7fd564004d40 2024-03-06T07:39:17.354 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.351+0000 7fd57f400640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5800b9380 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd570009cb0 tx=0x7fd570007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.352+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570014610 con 0x7fd564004d40 2024-03-06T07:39:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.352+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd570019070 con 0x7fd564004d40 2024-03-06T07:39:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.352+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5800b82e0 con 0x7fd564004d40 2024-03-06T07:39:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.352+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700149d0 con 0x7fd564004d40 2024-03-06T07:39:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.352+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5800b88a0 con 0x7fd564004d40 2024-03-06T07:39:17.355 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.353+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58004f330 con 0x7fd564004d40 2024-03-06T07:39:17.357 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.355+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd570014b70 con 0x7fd564004d40 2024-03-06T07:39:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.356+0000 7fd577400640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.356+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd57008ac50 con 0x7fd564004d40 2024-03-06T07:39:17.358 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.356+0000 7fd57ea00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:17.359 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.358+0000 7fd57ea00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd55c06b4d0 0x7fd55c06d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd568007fd0 tx=0x7fd568009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:17.362 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.361+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5700599f0 con 0x7fd564004d40 2024-03-06T07:39:17.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.603+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5800b7cc0 con 0x7fd564004d40 2024-03-06T07:39:17.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.604+0000 7fd577400640 1 -- 172.21.15.6:0/3450923527 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fd5700592f0 con 0x7fd564004d40 2024-03-06T07:39:17.607 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:17.607 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:17.607 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:17.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd55c06b4d0 msgr2=0x7fd55c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd55c06b4d0 0x7fd55c06d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd568007fd0 tx=0x7fd568009040 comp rx=0 tx=0).stop 2024-03-06T07:39:17.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 msgr2=0x7fd5800b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:17.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5800b9380 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd570009cb0 tx=0x7fd570007700 comp rx=0 tx=0).stop 2024-03-06T07:39:17.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 shutdown_connections 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd5800b7300 0x7fd5800b7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fd5801f6fa0 0x7fd5801f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 --2- 172.21.15.6:0/3450923527 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd564004d40 0x7fd5800b9380 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 >> 172.21.15.6:0/3450923527 conn(0x7fd5800d3d30 msgr2=0x7fd58006b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.607+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 shutdown_connections 2024-03-06T07:39:17.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:17.608+0000 7fd584e00640 1 -- 172.21.15.6:0/3450923527 wait complete. 2024-03-06T07:39:17.833 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:17.833 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:18.249 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:18.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:19.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.058+0000 7f194e000640 1 Processor -- start 2024-03-06T07:39:19.060 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.058+0000 7f194e000640 1 -- start start 2024-03-06T07:39:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.061+0000 7f194e000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1948144c00 0x7f1948145280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.061+0000 7f194e000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19481457c0 0x7f194806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.064 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f194e000640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19480d0910 con 0x7f194806b4f0 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f194e000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480d0bf0 con 0x7f19481457c0 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f194e000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480d0ed0 con 0x7f1948144c00 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f1947e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19481457c0 0x7f194806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f194cc00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1948144c00 0x7f1948145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f1947e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f19481457c0 0x7f194806afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46004/0 (socket says 172.21.15.6:46004) 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.062+0000 7f1947e00640 1 -- 172.21.15.6:0/505161078 learned_addr learned my addr 172.21.15.6:0/505161078 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.063+0000 7f1947e00640 1 -- 172.21.15.6:0/505161078 >> v2:172.21.15.178:3300/0 conn(0x7f1948144c00 msgr2=0x7f1948145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:19.065 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.063+0000 7f1947e00640 1 --2- 172.21.15.6:0/505161078 >> v2:172.21.15.178:3300/0 conn(0x7f1948144c00 0x7f1948145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.063+0000 7f1947e00640 1 -- 172.21.15.6:0/505161078 >> v1:172.21.15.178:6789/0 conn(0x7f194806b4f0 legacy=0x7f194815f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.063+0000 7f1947e00640 1 -- 172.21.15.6:0/505161078 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480d11b0 con 0x7f19481457c0 2024-03-06T07:39:19.066 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.065+0000 7f1947e00640 1 --2- 172.21.15.6:0/505161078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19481457c0 0x7f194806afb0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f194806bcf0 tx=0x7f193402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a79d5147f6f595 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.065+0000 7f1947400640 1 -- 172.21.15.6:0/505161078 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934034070 con 0x7f19481457c0 2024-03-06T07:39:19.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.065+0000 7f1947400640 1 -- 172.21.15.6:0/505161078 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1934030860 con 0x7f19481457c0 2024-03-06T07:39:19.067 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.065+0000 7f1947400640 1 -- 172.21.15.6:0/505161078 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1934030c30 con 0x7f19481457c0 2024-03-06T07:39:19.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 -- 172.21.15.6:0/505161078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19481457c0 msgr2=0x7f194806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 --2- 172.21.15.6:0/505161078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19481457c0 0x7f194806afb0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f194806bcf0 tx=0x7f193402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 -- 172.21.15.6:0/505161078 shutdown_connections 2024-03-06T07:39:19.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 --2- 172.21.15.6:0/505161078 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f19481457c0 0x7f194806afb0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 --2- 172.21.15.6:0/505161078 >> v2:172.21.15.178:3300/0 conn(0x7f1948144c00 0x7f1948145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.068 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 -- 172.21.15.6:0/505161078 >> 172.21.15.6:0/505161078 conn(0x7f194806c0c0 msgr2=0x7f194806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:19.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 -- 172.21.15.6:0/505161078 shutdown_connections 2024-03-06T07:39:19.069 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.066+0000 7f194e000640 1 -- 172.21.15.6:0/505161078 wait complete. 2024-03-06T07:39:19.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.069+0000 7f194e000640 1 Processor -- start 2024-03-06T07:39:19.071 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.069+0000 7f194e000640 1 -- start start 2024-03-06T07:39:19.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194e000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 0x7f1948158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194e000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1948144c00 0x7f1948135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194e000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19481457c0 0x7f1948142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194e000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19480d19b0 con 0x7f194806b4f0 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194e000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948144fe0 con 0x7f1948144c00 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194e000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19480d0140 con 0x7f19481457c0 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 0x7f1948158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194cc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 0x7f1948158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46012/0 (socket says 172.21.15.6:46012) 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194cc00640 1 -- 172.21.15.6:0/720130410 learned_addr learned my addr 172.21.15.6:0/720130410 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f1947e00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1948144c00 0x7f1948135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194d600640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19481457c0 0x7f1948142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.071+0000 7f194cc00640 1 -- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19481457c0 msgr2=0x7f1948142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194cc00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19481457c0 0x7f1948142260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194cc00640 1 -- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1948144c00 msgr2=0x7f1948135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194cc00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1948144c00 0x7f1948135320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194cc00640 1 -- 172.21.15.6:0/720130410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19481596e0 con 0x7f194806b4f0 2024-03-06T07:39:19.075 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f1947e00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1948144c00 0x7f1948135320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194d600640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19481457c0 0x7f1948142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194cc00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 0x7f1948158e00 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f193c00e9b0 tx=0x7f193c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c017070 con 0x7f194806b4f0 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c00c9a0 con 0x7f194806b4f0 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c00cd90 con 0x7f194806b4f0 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.072+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19481599c0 con 0x7f194806b4f0 2024-03-06T07:39:19.076 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.073+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19480b10f0 con 0x7f194806b4f0 2024-03-06T07:39:19.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.074+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19480b6e50 con 0x7f194806b4f0 2024-03-06T07:39:19.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.075+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f193c01a070 con 0x7f194806b4f0 2024-03-06T07:39:19.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.076+0000 7f1944c00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f192806b4d0 0x7f192806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:19.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.076+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f193c08a670 con 0x7f194806b4f0 2024-03-06T07:39:19.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.076+0000 7f1947e00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f192806b4d0 0x7f192806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:19.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.077+0000 7f1947e00640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f192806b4d0 0x7f192806d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1934036000 tx=0x7f193403b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:19.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.080+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19480b6e50 con 0x7f194806b4f0 2024-03-06T07:39:19.445 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.443+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f194804f330 con 0x7f194806b4f0 2024-03-06T07:39:19.446 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.444+0000 7f1944c00640 1 -- 172.21.15.6:0/720130410 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f193c059410 con 0x7f194806b4f0 2024-03-06T07:39:19.447 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:19.447 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:19.447 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:19.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.446+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f192806b4d0 msgr2=0x7f192806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f192806b4d0 0x7f192806d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f1934036000 tx=0x7f193403b040 comp rx=0 tx=0).stop 2024-03-06T07:39:19.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 msgr2=0x7f1948158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:19.449 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 0x7f1948158e00 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f193c00e9b0 tx=0x7f193c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 shutdown_connections 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f192806b4d0 0x7f192806d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1948144c00 0x7f1948135320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f19481457c0 0x7f1948142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 --2- 172.21.15.6:0/720130410 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f194806b4f0 0x7f1948158e00 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 >> 172.21.15.6:0/720130410 conn(0x7f194806c0c0 msgr2=0x7f1948143ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 shutdown_connections 2024-03-06T07:39:19.450 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:19.447+0000 7f194e000640 1 -- 172.21.15.6:0/720130410 wait complete. 2024-03-06T07:39:19.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:19.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:20.251 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:20.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:20.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.896+0000 7f66d6a00640 1 Processor -- start 2024-03-06T07:39:20.898 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.896+0000 7f66d6a00640 1 -- start start 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d6a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f66d00af1a0 0x7f66d015c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d6a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af580 0x7f66d00b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d6a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66d00cdc20 0x7f66d00b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d6a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d015cc40 con 0x7f66d00cdc20 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d6a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d015cf20 con 0x7f66d00af1a0 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d6a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d015d200 con 0x7f66d00af580 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.898+0000 7f66d4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66d00cdc20 0x7f66d00b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f66d00cdc20 0x7f66d00b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46026/0 (socket says 172.21.15.6:46026) 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 -- 172.21.15.6:0/4279985218 learned_addr learned my addr 172.21.15.6:0/4279985218 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66cfe00640 1 --2- 172.21.15.6:0/4279985218 >> v2:172.21.15.178:3300/0 conn(0x7f66d00af1a0 0x7f66d015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66c7400640 1 --2- 172.21.15.6:0/4279985218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af580 0x7f66d00b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 -- 172.21.15.6:0/4279985218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af580 msgr2=0x7f66d00b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 --2- 172.21.15.6:0/4279985218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af580 0x7f66d00b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 -- 172.21.15.6:0/4279985218 >> v2:172.21.15.178:3300/0 conn(0x7f66d00af1a0 msgr2=0x7f66d015c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 --2- 172.21.15.6:0/4279985218 >> v2:172.21.15.178:3300/0 conn(0x7f66d00af1a0 0x7f66d015c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.902 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.899+0000 7f66d4a00640 1 -- 172.21.15.6:0/4279985218 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d015d4e0 con 0x7f66d00cdc20 2024-03-06T07:39:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.902+0000 7f66d4a00640 1 --2- 172.21.15.6:0/4279985218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00cdc20 0x7f66d00b2dd0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f66d0074960 tx=0x7f66c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2663b8752037db7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.902+0000 7f66cf400640 1 -- 172.21.15.6:0/4279985218 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c802c9e0 con 0x7f66d00cdc20 2024-03-06T07:39:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.902+0000 7f66cf400640 1 -- 172.21.15.6:0/4279985218 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c802cb80 con 0x7f66d00cdc20 2024-03-06T07:39:20.904 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.902+0000 7f66cf400640 1 -- 172.21.15.6:0/4279985218 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80343f0 con 0x7f66d00cdc20 2024-03-06T07:39:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 -- 172.21.15.6:0/4279985218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00cdc20 msgr2=0x7f66d00b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 --2- 172.21.15.6:0/4279985218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00cdc20 0x7f66d00b2dd0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f66d0074960 tx=0x7f66c802be40 comp rx=0 tx=0).stop 2024-03-06T07:39:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 -- 172.21.15.6:0/4279985218 shutdown_connections 2024-03-06T07:39:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 --2- 172.21.15.6:0/4279985218 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00cdc20 0x7f66d00b2dd0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.905 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 --2- 172.21.15.6:0/4279985218 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af580 0x7f66d00b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 --2- 172.21.15.6:0/4279985218 >> v2:172.21.15.178:3300/0 conn(0x7f66d00af1a0 0x7f66d015c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 -- 172.21.15.6:0/4279985218 >> 172.21.15.6:0/4279985218 conn(0x7f66d006afb0 msgr2=0x7f66d006b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 -- 172.21.15.6:0/4279985218 shutdown_connections 2024-03-06T07:39:20.906 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.903+0000 7f66d6a00640 1 -- 172.21.15.6:0/4279985218 wait complete. 2024-03-06T07:39:20.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.905+0000 7f66d6a00640 1 Processor -- start 2024-03-06T07:39:20.907 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.905+0000 7f66d6a00640 1 -- start start 2024-03-06T07:39:20.909 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66d6a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af1a0 0x7f66d014c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66d6a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 0x7f66d014c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66d6a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d00cdc20 0x7f66d0071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66d6a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b3490 con 0x7f66d00af580 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66d6a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b3310 con 0x7f66d00cdc20 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66d6a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d00b3610 con 0x7f66d00af1a0 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 0x7f66d014c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66c7400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 0x7f66d014c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46032/0 (socket says 172.21.15.6:46032) 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.906+0000 7f66c7400640 1 -- 172.21.15.6:0/2492109947 learned_addr learned my addr 172.21.15.6:0/2492109947 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66cfe00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af1a0 0x7f66d014c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66d4a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d00cdc20 0x7f66d0071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66c7400640 1 -- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af1a0 msgr2=0x7f66d014c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66c7400640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af1a0 0x7f66d014c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66c7400640 1 -- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d00cdc20 msgr2=0x7f66d0071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:20.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66c7400640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d00cdc20 0x7f66d0071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66c7400640 1 -- 172.21.15.6:0/2492109947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d01f3810 con 0x7f66d00af580 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66c7400640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 0x7f66d014c040 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f66b800d800 tx=0x7f66b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.907+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b800b6c0 con 0x7f66d00af580 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.908+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b8019070 con 0x7f66d00af580 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.908+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b800ba80 con 0x7f66d00af580 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.908+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d01f3ad0 con 0x7f66d00af580 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.908+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d01f4090 con 0x7f66d00af580 2024-03-06T07:39:20.911 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.909+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d004f3a0 con 0x7f66d00af580 2024-03-06T07:39:20.912 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.910+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66b800bc20 con 0x7f66d00af580 2024-03-06T07:39:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.911+0000 7f66ccc00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66ac06b4d0 0x7f66ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.911+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f66b808a6f0 con 0x7f66d00af580 2024-03-06T07:39:20.913 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.912+0000 7f66cfe00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66ac06b4d0 0x7f66ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:20.915 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.913+0000 7f66cfe00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66ac06b4d0 0x7f66ac06d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006fd0 tx=0x7f66c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:20.918 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:20.916+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f66b8059530 con 0x7f66d00af580 2024-03-06T07:39:21.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.153+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66d0074c70 con 0x7f66d00af580 2024-03-06T07:39:21.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.154+0000 7f66ccc00640 1 -- 172.21.15.6:0/2492109947 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f66b8058e30 con 0x7f66d00af580 2024-03-06T07:39:21.156 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:21.156 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:21.156 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66ac06b4d0 msgr2=0x7f66ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66ac06b4d0 0x7f66ac06d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f66c0006fd0 tx=0x7f66c0008040 comp rx=0 tx=0).stop 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 msgr2=0x7f66d014c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 0x7f66d014c040 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f66b800d800 tx=0x7f66b800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 shutdown_connections 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f66ac06b4d0 0x7f66ac06d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f66d00cdc20 0x7f66d0071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f66d00af580 0x7f66d014c040 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 --2- 172.21.15.6:0/2492109947 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f66d00af1a0 0x7f66d014c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.156+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 >> 172.21.15.6:0/2492109947 conn(0x7f66d006afb0 msgr2=0x7f66d015c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.157+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 shutdown_connections 2024-03-06T07:39:21.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:21.157+0000 7f66d6a00640 1 -- 172.21.15.6:0/2492109947 wait complete. 2024-03-06T07:39:21.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:21.380 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:22.253 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:22.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:22.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.600+0000 7f9eaa200640 1 Processor -- start 2024-03-06T07:39:22.603 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.601+0000 7f9eaa200640 1 -- start start 2024-03-06T07:39:22.604 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.602+0000 7f9eaa200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f9ea415e1c0 0x7f9ea40b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.602+0000 7f9eaa200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9ea40b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.602+0000 7f9eaa200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea40b3da0 con 0x7f9ea406afb0 2024-03-06T07:39:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.602+0000 7f9eaa200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea40b6060 con 0x7f9ea415e1c0 2024-03-06T07:39:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.602+0000 7f9eaa200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40aff60 con 0x7f9ea406f6b0 2024-03-06T07:39:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.603+0000 7f9ea3e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9ea406afb0 0x7f9ea406da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53438/0 (socket says 172.21.15.6:53438) 2024-03-06T07:39:22.605 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.603+0000 7f9ea3e00640 1 -- 172.21.15.6:0/3503585984 learned_addr learned my addr 172.21.15.6:0/3503585984 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:22.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.603+0000 7f9e9be00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9ea40b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.603+0000 7f9ea3400640 1 --2- 172.21.15.6:0/3503585984 >> v2:172.21.15.178:3300/0 conn(0x7f9ea415e1c0 0x7f9ea40b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.603+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061457481 0 0) 0x7f9ea40b3da0 con 0x7f9ea406afb0 2024-03-06T07:39:22.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.603+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e88003520 con 0x7f9ea406afb0 2024-03-06T07:39:22.606 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1473459130 0 0) 0x7f9e88003520 con 0x7f9ea406afb0 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 msgr2=0x7f9ea40b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9ea40b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 >> v2:172.21.15.178:3300/0 conn(0x7f9ea415e1c0 msgr2=0x7f9ea40b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 --2- 172.21.15.6:0/3503585984 >> v2:172.21.15.178:3300/0 conn(0x7f9ea415e1c0 0x7f9ea40b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea406fb60 con 0x7f9ea406afb0 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f9e9c0019f0 con 0x7f9ea406afb0 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 >> v1:172.21.15.6:6789/0 conn(0x7f9ea406afb0 legacy=0x7f9ea406da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9e88003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9e88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e88004d40 0x7f9e880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b6060 con 0x7f9e88003fa0 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40aff60 con 0x7f9e88004d40 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8800b9f0 con 0x7f9ea406f6b0 2024-03-06T07:39:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9e9be00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9e88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.604+0000 7f9ea3400640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9e88003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9ea3e00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e88004d40 0x7f9e880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9e9be00640 1 -- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 msgr2=0x7f9e88003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9e9be00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9e88003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9e9be00640 1 -- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e88004d40 msgr2=0x7f9e880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9e9be00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e88004d40 0x7f9e880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9e9be00640 1 -- 172.21.15.6:0/3503585984 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88003520 con 0x7f9e88003fa0 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9ea3400640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea406f6b0 0x7f9e88003a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9ea3e00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e88004d40 0x7f9e880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9e9be00640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9e88004800 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009ac0 tx=0x7f9e88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741a8a292460aa30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c039070 con 0x7f9e88003fa0 2024-03-06T07:39:22.609 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e8c02c6b0 con 0x7f9e88003fa0 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9ea2a00640 1 -- 172.21.15.6:0/3503585984 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c009690 con 0x7f9e88003fa0 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9eaa200640 1 -- 172.21.15.6:0/3503585984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 msgr2=0x7f9e88004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.606+0000 7f9eaa200640 1 --2- 172.21.15.6:0/3503585984 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9e88004800 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009ac0 tx=0x7f9e88005830 comp rx=0 tx=0).stop 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.607+0000 7f9ea3e00640 1 -- 172.21.15.6:0/3503585984 reap_dead start 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.607+0000 7f9eaa200640 1 -- 172.21.15.6:0/3503585984 shutdown_connections 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.607+0000 7f9eaa200640 1 -- 172.21.15.6:0/3503585984 >> 172.21.15.6:0/3503585984 conn(0x7f9ea40b2a00 msgr2=0x7f9ea40b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.607+0000 7f9eaa200640 1 -- 172.21.15.6:0/3503585984 shutdown_connections 2024-03-06T07:39:22.610 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.607+0000 7f9eaa200640 1 -- 172.21.15.6:0/3503585984 wait complete. 2024-03-06T07:39:22.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.609+0000 7f9eaa200640 1 Processor -- start 2024-03-06T07:39:22.611 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.609+0000 7f9eaa200640 1 -- start start 2024-03-06T07:39:22.613 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9eaa200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9ea40c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9eaa200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ea415c040 0x7f9ea415c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9eaa200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea41f7550 0x7f9ea41f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9eaa200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea4072b50 con 0x7f9e88003fa0 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9eaa200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea4072870 con 0x7f9ea415c040 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9eaa200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea41e7ce0 con 0x7f9ea41f7550 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9ea3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9ea40c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9ea3400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9ea40c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43184/0 (socket says 172.21.15.6:43184) 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9ea3400640 1 -- 172.21.15.6:0/614316671 learned_addr learned my addr 172.21.15.6:0/614316671 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.610+0000 7f9e9be00640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ea415c040 0x7f9ea415c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3e00640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea41f7550 0x7f9ea41f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3400640 1 -- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea41f7550 msgr2=0x7f9ea41f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3400640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea41f7550 0x7f9ea41f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3400640 1 -- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ea415c040 msgr2=0x7f9ea415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3400640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ea415c040 0x7f9ea415c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.614 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3400640 1 -- 172.21.15.6:0/614316671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea40c8490 con 0x7f9e88003fa0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9e9be00640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ea415c040 0x7f9ea415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3e00640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea41f7550 0x7f9ea41f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9ea3400640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9ea40c7f50 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f9e940109d0 tx=0x7f9e94010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9400f040 con 0x7f9e88003fa0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9400ec20 con 0x7f9e88003fa0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea415ca00 con 0x7f9e88003fa0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.611+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9401c400 con 0x7f9e88003fa0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.612+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea40c8a10 con 0x7f9e88003fa0 2024-03-06T07:39:22.615 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.613+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea404f3a0 con 0x7f9e88003fa0 2024-03-06T07:39:22.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.615+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e9401c5a0 con 0x7f9e88003fa0 2024-03-06T07:39:22.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.616+0000 7f9e9b400640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e8006b500 0x7f9e8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:22.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.616+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e9408a980 con 0x7f9e88003fa0 2024-03-06T07:39:22.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.616+0000 7f9e9be00640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e8006b500 0x7f9e8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:22.620 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.618+0000 7f9e9be00640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e8006b500 0x7f9e8006d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c036000 tx=0x7f9e8c030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:22.622 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.621+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e94018030 con 0x7f9e88003fa0 2024-03-06T07:39:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.859+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea40745e0 con 0x7f9e88003fa0 2024-03-06T07:39:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.860+0000 7f9e9b400640 1 -- 172.21.15.6:0/614316671 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f9e940597b0 con 0x7f9e88003fa0 2024-03-06T07:39:22.862 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:22.862 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:22.862 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.862+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e8006b500 msgr2=0x7f9e8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.862+0000 7f9eaa200640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e8006b500 0x7f9e8006d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c036000 tx=0x7f9e8c030000 comp rx=0 tx=0).stop 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.862+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 msgr2=0x7f9ea40c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.862+0000 7f9eaa200640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9ea40c7f50 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f9e940109d0 tx=0x7f9e94010e90 comp rx=0 tx=0).stop 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 shutdown_connections 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f9e8006b500 0x7f9e8006d9c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9ea415c040 0x7f9ea415c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f9ea41f7550 0x7f9ea41f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 --2- 172.21.15.6:0/614316671 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9e88003fa0 0x7f9ea40c7f50 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 >> 172.21.15.6:0/614316671 conn(0x7f9ea40b2a00 msgr2=0x7f9ea40b18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 shutdown_connections 2024-03-06T07:39:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:22.863+0000 7f9eaa200640 1 -- 172.21.15.6:0/614316671 wait complete. 2024-03-06T07:39:23.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:23.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:24.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:24.255 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:24.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.302+0000 7fdbc5200640 1 Processor -- start 2024-03-06T07:39:24.304 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.302+0000 7fdbc5200640 1 -- start start 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbc5200640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fdbc006b6c0 0x7fdbc00d1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbc5200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc0147210 con 0x7fdbc00b5990 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbc5200640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc0072cd0 con 0x7fdbc00b0530 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbc5200640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00d0bc0 con 0x7fdbc006b6c0 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbbd400640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdbc00b5990 0x7fdbc006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53440/0 (socket says 172.21.15.6:53440) 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbbd400640 1 -- 172.21.15.6:0/2555524630 learned_addr learned my addr 172.21.15.6:0/2555524630 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:24.307 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.305+0000 7fdbbe800640 1 --2- 172.21.15.6:0/2555524630 >> v2:172.21.15.6:3300/0 conn(0x7fdbc006b6c0 0x7fdbc00d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.306+0000 7fdbbca00640 1 -- 172.21.15.6:0/2555524630 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596024045 0 0) 0x7fdbc0147210 con 0x7fdbc00b5990 2024-03-06T07:39:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.306+0000 7fdbbca00640 1 -- 172.21.15.6:0/2555524630 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba4003520 con 0x7fdbc00b5990 2024-03-06T07:39:24.308 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.306+0000 7fdbbe800640 1 -- 172.21.15.6:0/2555524630 >> v1:172.21.15.178:6789/0 conn(0x7fdbc00b0530 legacy=0x7fdbc00725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.306+0000 7fdbbe800640 1 -- 172.21.15.6:0/2555524630 >> v1:172.21.15.6:6789/0 conn(0x7fdbc00b5990 legacy=0x7fdbc006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.306+0000 7fdbbe800640 1 -- 172.21.15.6:0/2555524630 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00b5f20 con 0x7fdbc006b6c0 2024-03-06T07:39:24.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.307+0000 7fdbbe800640 1 --2- 172.21.15.6:0/2555524630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc006b6c0 0x7fdbc00d1660 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009d90 tx=0x7fdbb802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c6bedce92201bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbbca00640 1 -- 172.21.15.6:0/2555524630 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8039070 con 0x7fdbc006b6c0 2024-03-06T07:39:24.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbbca00640 1 -- 172.21.15.6:0/2555524630 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb802cd00 con 0x7fdbc006b6c0 2024-03-06T07:39:24.310 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbbca00640 1 -- 172.21.15.6:0/2555524630 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8034660 con 0x7fdbc006b6c0 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 -- 172.21.15.6:0/2555524630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc006b6c0 msgr2=0x7fdbc00d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2555524630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc006b6c0 0x7fdbc00d1660 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009d90 tx=0x7fdbb802bf50 comp rx=0 tx=0).stop 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 -- 172.21.15.6:0/2555524630 shutdown_connections 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2555524630 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc006b6c0 0x7fdbc00d1660 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 -- 172.21.15.6:0/2555524630 >> 172.21.15.6:0/2555524630 conn(0x7fdbc006c2a0 msgr2=0x7fdbc006c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 -- 172.21.15.6:0/2555524630 shutdown_connections 2024-03-06T07:39:24.311 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.308+0000 7fdbc5200640 1 -- 172.21.15.6:0/2555524630 wait complete. 2024-03-06T07:39:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.310+0000 7fdbc5200640 1 Processor -- start 2024-03-06T07:39:24.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.310+0000 7fdbc5200640 1 -- start start 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.311+0000 7fdbc5200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbc006b6c0 0x7fdbc00c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.311+0000 7fdbc5200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 0x7fdbc00c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.311+0000 7fdbc5200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00b5990 0x7fdbc00c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.311+0000 7fdbc5200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00b6200 con 0x7fdbc00b0530 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.311+0000 7fdbc5200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00d1140 con 0x7fdbc00b5990 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.311+0000 7fdbc5200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc006da90 con 0x7fdbc006b6c0 2024-03-06T07:39:24.314 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 0x7fdbc00c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 0x7fdbc00c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43206/0 (socket says 172.21.15.6:43206) 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 -- 172.21.15.6:0/2494682597 learned_addr learned my addr 172.21.15.6:0/2494682597 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbe800640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00b5990 0x7fdbc00c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbde00640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbc006b6c0 0x7fdbc00c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 -- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbc006b6c0 msgr2=0x7fdbc00c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbc006b6c0 0x7fdbc00c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 -- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00b5990 msgr2=0x7fdbc00c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00b5990 0x7fdbc00c71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 -- 172.21.15.6:0/2494682597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00c66c0 con 0x7fdbc00b0530 2024-03-06T07:39:24.315 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbde00640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbc006b6c0 0x7fdbc00c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.312+0000 7fdbbd400640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 0x7fdbc00c6c80 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c410 tx=0x7fdbac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.313+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac017070 con 0x7fdbc00b0530 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.313+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac005ce0 con 0x7fdbc00b0530 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.313+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0072f0 con 0x7fdbc00b0530 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.313+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc00c69a0 con 0x7fdbc00b0530 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.313+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc01e6b30 con 0x7fdbc00b0530 2024-03-06T07:39:24.316 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.314+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc004f3a0 con 0x7fdbc00b0530 2024-03-06T07:39:24.318 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.316+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdbac01a070 con 0x7fdbc00b0530 2024-03-06T07:39:24.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.317+0000 7fdbb6000640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdba006b4d0 0x7fdba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:24.319 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.317+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdbac089710 con 0x7fdbc00b0530 2024-03-06T07:39:24.320 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.318+0000 7fdbbde00640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdba006b4d0 0x7fdba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:24.321 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.319+0000 7fdbbde00640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdba006b4d0 0x7fdba006d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00af4b0 tx=0x7fdbb0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.322+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdbac058600 con 0x7fdbc00b0530 2024-03-06T07:39:24.566 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.564+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc01469f0 con 0x7fdbc00b0530 2024-03-06T07:39:24.567 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.565+0000 7fdbb6000640 1 -- 172.21.15.6:0/2494682597 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7fdbac057f00 con 0x7fdbc00b0530 2024-03-06T07:39:24.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:24.567 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:24.568 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdba006b4d0 msgr2=0x7fdba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdba006b4d0 0x7fdba006d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00af4b0 tx=0x7fdbb0009040 comp rx=0 tx=0).stop 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 msgr2=0x7fdbc00c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 0x7fdbc00c6c80 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c410 tx=0x7fdbac00c8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 shutdown_connections 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7fdba006b4d0 0x7fdba006d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.570 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc00b5990 0x7fdbc00c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdbc00b0530 0x7fdbc00c6c80 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 --2- 172.21.15.6:0/2494682597 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7fdbc006b6c0 0x7fdbc00c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:24.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 >> 172.21.15.6:0/2494682597 conn(0x7fdbc006c2a0 msgr2=0x7fdbc00b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:24.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 shutdown_connections 2024-03-06T07:39:24.571 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:24.568+0000 7fdbc5200640 1 -- 172.21.15.6:0/2494682597 wait complete. 2024-03-06T07:39:24.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:24.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:24.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:24.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:25.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.068+0000 7f1dd7200640 1 Processor -- start 2024-03-06T07:39:26.070 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.068+0000 7f1dd7200640 1 -- start start 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 0x7f1dd80cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd7200640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f1dd80cce90 0x7f1dd815c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd7200640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd80714d0 con 0x7f1dd8145300 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd7200640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd80cd300 con 0x7f1dd80cce90 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80717b0 con 0x7f1dd8070f10 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd5e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f1dd8145300 0x7f1dd80b49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53456/0 (socket says 172.21.15.6:53456) 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.070+0000 7f1dd5e00640 1 -- 172.21.15.6:0/53878598 learned_addr learned my addr 172.21.15.6:0/53878598 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:26.073 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dcd400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 0x7f1dd80cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd6800640 1 --2- 172.21.15.6:0/53878598 >> v2:172.21.15.178:3300/0 conn(0x7f1dd80cce90 0x7f1dd815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551538410 0 0) 0x7f1dd80714d0 con 0x7f1dd8145300 2024-03-06T07:39:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db8003520 con 0x7f1dd8145300 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1023141597 0 0) 0x7f1db8003520 con 0x7f1dd8145300 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 msgr2=0x7f1dd80cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 0x7f1dd80cc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 >> v2:172.21.15.178:3300/0 conn(0x7f1dd80cce90 msgr2=0x7f1dd815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 --2- 172.21.15.6:0/53878598 >> v2:172.21.15.178:3300/0 conn(0x7f1dd80cce90 0x7f1dd815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8071a90 con 0x7f1dd8145300 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f1dc8001670 con 0x7f1dd8145300 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 >> v1:172.21.15.6:6789/0 conn(0x7f1dd8145300 legacy=0x7f1dd80b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.077 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80039b0 0x7f1db8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1db8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 0x7f1db80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80cd300 con 0x7f1db8004800 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80717b0 con 0x7f1db80039b0 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.071+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db800fdd0 con 0x7f1dd8070f10 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dcd400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1db8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dd5e00640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80039b0 0x7f1db8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dd6800640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 0x7f1db80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dcd400640 1 -- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 msgr2=0x7f1db80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dcd400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd8070f10 0x7f1db80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dcd400640 1 -- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80039b0 msgr2=0x7f1db8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:26.078 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dcd400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1db80039b0 0x7f1db8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.072+0000 7f1dcd400640 1 -- 172.21.15.6:0/53878598 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db8003520 con 0x7f1db8004800 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.073+0000 7f1dcd400640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1db8004c80 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009ac0 tx=0x7f1db8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ce97676733786d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0039070 con 0x7f1db8004800 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc002c6b0 con 0x7f1db8004800 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd5400640 1 -- 172.21.15.6:0/53878598 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0009690 con 0x7f1db8004800 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd7200640 1 -- 172.21.15.6:0/53878598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 msgr2=0x7f1db8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd7200640 1 --2- 172.21.15.6:0/53878598 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1db8004c80 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009ac0 tx=0x7f1db8005830 comp rx=0 tx=0).stop 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd6800640 1 -- 172.21.15.6:0/53878598 reap_dead start 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd7200640 1 -- 172.21.15.6:0/53878598 shutdown_connections 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.074+0000 7f1dd7200640 1 -- 172.21.15.6:0/53878598 >> 172.21.15.6:0/53878598 conn(0x7f1dd806b3d0 msgr2=0x7f1dd806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.075+0000 7f1dd7200640 1 -- 172.21.15.6:0/53878598 shutdown_connections 2024-03-06T07:39:26.079 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.075+0000 7f1dd7200640 1 -- 172.21.15.6:0/53878598 wait complete. 2024-03-06T07:39:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.076+0000 7f1dd7200640 1 Processor -- start 2024-03-06T07:39:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.076+0000 7f1dd7200640 1 -- start start 2024-03-06T07:39:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.077+0000 7f1dd7200640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1dd80b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.077+0000 7f1dd7200640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dd813bb50 0x7f1dd813edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.077+0000 7f1dd7200640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd814ffa0 0x7f1dd8141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.077+0000 7f1dd7200640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80cc950 con 0x7f1db8004800 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.077+0000 7f1dd7200640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd81456e0 con 0x7f1dd813bb50 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.077+0000 7f1dd7200640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd80ccad0 con 0x7f1dd814ffa0 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1dd80b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1dd80b0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43214/0 (socket says 172.21.15.6:43214) 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 -- 172.21.15.6:0/1911251923 learned_addr learned my addr 172.21.15.6:0/1911251923 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 -- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd814ffa0 msgr2=0x7f1dd8141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd814ffa0 0x7f1dd8141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 -- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dd813bb50 msgr2=0x7f1dd813edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dd813bb50 0x7f1dd813edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 -- 172.21.15.6:0/1911251923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd813f530 con 0x7f1db8004800 2024-03-06T07:39:26.083 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.078+0000 7f1dd5e00640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1dd80b0500 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8000c00 tx=0x7f1dc800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.079+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc800c610 con 0x7f1db8004800 2024-03-06T07:39:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.079+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8018070 con 0x7f1db8004800 2024-03-06T07:39:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.079+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc800c9f0 con 0x7f1db8004800 2024-03-06T07:39:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.079+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd8139060 con 0x7f1db8004800 2024-03-06T07:39:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.079+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd8139620 con 0x7f1db8004800 2024-03-06T07:39:26.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.080+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd8139820 con 0x7f1db8004800 2024-03-06T07:39:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.081+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dc800cb90 con 0x7f1db8004800 2024-03-06T07:39:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.082+0000 7f1dcea00640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1db406b500 0x7f1db406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.083+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1dc8089d40 con 0x7f1db8004800 2024-03-06T07:39:26.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.083+0000 7f1dcd400640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1db406b500 0x7f1db406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:26.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.084+0000 7f1dcd400640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1db406b500 0x7f1db406d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0036000 tx=0x7f1dc002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:26.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.086+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1dd8139820 con 0x7f1db8004800 2024-03-06T07:39:26.257 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:26.495 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.493+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dd804f330 con 0x7f1db8004800 2024-03-06T07:39:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.494+0000 7f1dcea00640 1 -- 172.21.15.6:0/1911251923 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f1dc8058b70 con 0x7f1db8004800 2024-03-06T07:39:26.497 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:26.497 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:26.497 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1db406b500 msgr2=0x7f1db406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1db406b500 0x7f1db406d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0036000 tx=0x7f1dc002e000 comp rx=0 tx=0).stop 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 msgr2=0x7f1dd80b0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1dd80b0500 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8000c00 tx=0x7f1dc800ec10 comp rx=0 tx=0).stop 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 shutdown_connections 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f1db406b500 0x7f1db406d9c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1dd813bb50 0x7f1dd813edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.500 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f1dd814ffa0 0x7f1dd8141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 --2- 172.21.15.6:0/1911251923 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1db8004800 0x7f1dd80b0500 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 >> 172.21.15.6:0/1911251923 conn(0x7f1dd806b3d0 msgr2=0x7f1dd80b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 shutdown_connections 2024-03-06T07:39:26.501 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:26.497+0000 7f1dd7200640 1 -- 172.21.15.6:0/1911251923 wait complete. 2024-03-06T07:39:26.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:26.729 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:27.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:39:27.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.951+0000 7f8243a00640 1 Processor -- start 2024-03-06T07:39:27.953 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.951+0000 7f8243a00640 1 -- start start 2024-03-06T07:39:27.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8243a00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f823c0af1a0 0x7f823c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8243a00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f823c0af580 0x7f823c0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8243a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0cdc20 0x7f823c0b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.956 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8243a00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f823c15cc40 con 0x7f823c0af1a0 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8243a00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f823c15cf20 con 0x7f823c0af580 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8243a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c15d200 con 0x7f823c0cdc20 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8241000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f823c0af1a0 0x7f823c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8241000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f823c0af1a0 0x7f823c15c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43226/0 (socket says 172.21.15.6:43226) 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.953+0000 7f8241000640 1 -- 172.21.15.6:0/3044231080 learned_addr learned my addr 172.21.15.6:0/3044231080 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f8241a00640 1 --2- 172.21.15.6:0/3044231080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0cdc20 0x7f823c0b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f823be00640 1 --2- 172.21.15.6:0/3044231080 >> v2:172.21.15.178:3300/0 conn(0x7f823c0af580 0x7f823c0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f8241000640 1 -- 172.21.15.6:0/3044231080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0cdc20 msgr2=0x7f823c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f8241000640 1 --2- 172.21.15.6:0/3044231080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0cdc20 0x7f823c0b2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f8241000640 1 -- 172.21.15.6:0/3044231080 >> v2:172.21.15.178:3300/0 conn(0x7f823c0af580 msgr2=0x7f823c0b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f8241000640 1 --2- 172.21.15.6:0/3044231080 >> v2:172.21.15.178:3300/0 conn(0x7f823c0af580 0x7f823c0b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.957 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.954+0000 7f8241000640 1 -- 172.21.15.6:0/3044231080 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c15d4e0 con 0x7f823c0af1a0 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.955+0000 7f8241000640 1 --2- 172.21.15.6:0/3044231080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0af1a0 0x7f823c15c600 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f823c0b54c0 tx=0x7f823002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806458b8f66a5e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.955+0000 7f823b400640 1 -- 172.21.15.6:0/3044231080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823002e070 con 0x7f823c0af1a0 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f823b400640 1 -- 172.21.15.6:0/3044231080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8230037070 con 0x7f823c0af1a0 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f823b400640 1 -- 172.21.15.6:0/3044231080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8230031a00 con 0x7f823c0af1a0 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 -- 172.21.15.6:0/3044231080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0af1a0 msgr2=0x7f823c15c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 --2- 172.21.15.6:0/3044231080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0af1a0 0x7f823c15c600 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f823c0b54c0 tx=0x7f823002b880 comp rx=0 tx=0).stop 2024-03-06T07:39:27.959 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 -- 172.21.15.6:0/3044231080 shutdown_connections 2024-03-06T07:39:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 --2- 172.21.15.6:0/3044231080 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0cdc20 0x7f823c0b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 --2- 172.21.15.6:0/3044231080 >> v2:172.21.15.178:3300/0 conn(0x7f823c0af580 0x7f823c0b4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 --2- 172.21.15.6:0/3044231080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0af1a0 0x7f823c15c600 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 -- 172.21.15.6:0/3044231080 >> 172.21.15.6:0/3044231080 conn(0x7f823c06afb0 msgr2=0x7f823c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 -- 172.21.15.6:0/3044231080 shutdown_connections 2024-03-06T07:39:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.956+0000 7f8243a00640 1 -- 172.21.15.6:0/3044231080 wait complete. 2024-03-06T07:39:27.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.958+0000 7f8243a00640 1 Processor -- start 2024-03-06T07:39:27.961 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.958+0000 7f8243a00640 1 -- start start 2024-03-06T07:39:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8243a00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0af1a0 0x7f823c09aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8243a00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c0af580 0x7f823c09d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8243a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 0x7f823c14c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8243a00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0b3490 con 0x7f823c0cdc20 2024-03-06T07:39:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8243a00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0b3310 con 0x7f823c0af580 2024-03-06T07:39:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8243a00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f823c0b3610 con 0x7f823c0af1a0 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8241a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 0x7f823c14c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8241a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 0x7f823c14c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43232/0 (socket says 172.21.15.6:43232) 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.959+0000 7f8241a00640 1 -- 172.21.15.6:0/990117664 learned_addr learned my addr 172.21.15.6:0/990117664 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f823be00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c0af580 0x7f823c09d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241000640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0af1a0 0x7f823c09aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241a00640 1 -- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0af1a0 msgr2=0x7f823c09aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0af1a0 0x7f823c09aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241a00640 1 -- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c0af580 msgr2=0x7f823c09d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c0af580 0x7f823c09d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:27.964 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241a00640 1 -- 172.21.15.6:0/990117664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c14cbe0 con 0x7f823c0cdc20 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.960+0000 7f8241a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 0x7f823c14c6a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f8234007e60 tx=0x7f823400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.961+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234014420 con 0x7f823c0cdc20 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.961+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8234019070 con 0x7f823c0cdc20 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.961+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8234014820 con 0x7f823c0cdc20 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.961+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f823c0c93d0 con 0x7f823c0cdc20 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.961+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f823c0c9990 con 0x7f823c0cdc20 2024-03-06T07:39:27.965 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.962+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f823c069510 con 0x7f823c0cdc20 2024-03-06T07:39:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.963+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82340149c0 con 0x7f823c0cdc20 2024-03-06T07:39:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.964+0000 7f8238c00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f821c06b4d0 0x7f821c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.964+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f823408abd0 con 0x7f823c0cdc20 2024-03-06T07:39:27.967 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.965+0000 7f8241000640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f821c06b4d0 0x7f821c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:27.968 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.966+0000 7f8241000640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f821c06b4d0 0x7f821c06d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8230036000 tx=0x7f823003d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:27.971 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:27.969+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82340598e0 con 0x7f823c0cdc20 2024-03-06T07:39:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.212+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f823c0ad090 con 0x7f823c0cdc20 2024-03-06T07:39:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.213+0000 7f8238c00640 1 -- 172.21.15.6:0/990117664 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+3455 (secure 0 0 0) 0x7f82340591e0 con 0x7f823c0cdc20 2024-03-06T07:39:28.215 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:28.215 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":101,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:35:30.498751+0000","modified":"2024-03-06T07:37:59.678200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8355,"mds_1":6871},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6871":{"gid":6871,"name":"c","rank":1,"incarnation":94,"state":"up:stopping","state_seq":5,"addr":"172.21.15.6:6835/941666608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6833","nonce":941666608},{"type":"v1","addr":"172.21.15.6:6835","nonce":941666608}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:36:29.312449+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8355":{"gid":8355,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.6:6834/2669115160","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":2669115160},{"type":"v1","addr":"172.21.15.6:6834","nonce":2669115160}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8355,"qdb_cluster":[8355]},"id":9}]} 2024-03-06T07:39:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:dumped fsmap epoch 101 2024-03-06T07:39:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f821c06b4d0 msgr2=0x7f821c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.218 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f821c06b4d0 0x7f821c06d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8230036000 tx=0x7f823003d040 comp rx=0 tx=0).stop 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 msgr2=0x7f823c14c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 0x7f823c14c6a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f8234007e60 tx=0x7f823400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 shutdown_connections 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f821c06b4d0 0x7f821c06d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f823c0af580 0x7f823c09d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f823c0cdc20 0x7f823c14c6a0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 --2- 172.21.15.6:0/990117664 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f823c0af1a0 0x7f823c09aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 >> 172.21.15.6:0/990117664 conn(0x7f823c06afb0 msgr2=0x7f823c0cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.215+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 shutdown_connections 2024-03-06T07:39:28.219 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:28.216+0000 7f8243a00640 1 -- 172.21.15.6:0/990117664 wait complete. 2024-03-06T07:39:28.259 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:28.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 101, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:35:30.498751+0000', 'modified': '2024-03-06T07:37:59.678200+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8355, 'mds_1': 6871}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6871': {'gid': 6871, 'name': 'c', 'rank': 1, 'incarnation': 94, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.6:6835/941666608', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6833', 'nonce': 941666608}, {'type': 'v1', 'addr': '172.21.15.6:6835', 'nonce': 941666608}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:36:29.312449+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8355': {'gid': 8355, 'name': 'a', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.6:6834/2669115160', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6832', 'nonce': 2669115160}, {'type': 'v1', 'addr': '172.21.15.6:6834', 'nonce': 2669115160}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8355, 'qdb_cluster': [8355]} 2024-03-06T07:39:28.439 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6871:up:stopping 2024-03-06T07:39:29.440 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:35:30.498751+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6871": { "addr": "172.21.15.6:6835/941666608", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6833", "nonce": 941666608, "type": "v2" }, { "addr": "172.21.15.6:6835", "nonce": 941666608, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6871, "incarnation": 94, "join_fscid": -1, "laggy_since": "2024-03-06T07:36:29.312449+0000", "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_8355": { "addr": "172.21.15.6:6834/2669115160", "addrs": { "addrvec": [ { "addr": "172.21.15.6:6832", "nonce": 2669115160, "type": "v2" }, { "addr": "172.21.15.6:6834", "nonce": 2669115160, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8355, "incarnation": 88, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-03-06T07:37:59.678200+0000", "qdb_cluster": [ 8355 ], "qdb_leader": 8355, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8355, "mds_1": 6871 } } } ], "standbys": [] } 2024-03-06T07:39:29.441 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:39:29.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:39:29.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:39:29.679 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:39:29.679 INFO:tasks.cephfs.mount:Terminating background process 2024-03-06T07:39:29.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:29 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:39:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:open_dir_background: keeping dir open 2024-03-06T07:39:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:39:30.296 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:39:30.296 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:39:30.394 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-06T07:39:30.394 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:39:30.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:39:30.662 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.660+0000 7f7e60c00640 1 Processor -- start 2024-03-06T07:39:30.663 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.660+0000 7f7e60c00640 1 -- start start 2024-03-06T07:39:30.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e60c00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e5c161b40 0x7f7e5c0cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e60c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b5060 0x7f7e5c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e60c00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5c0cea50 con 0x7f7e5c0b54a0 2024-03-06T07:39:30.665 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e60c00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0cf900 con 0x7f7e5c161b40 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e60c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0cecf0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e59e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e5c161b40 0x7f7e5c0cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.662+0000 7f7e59e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7e5c161b40 0x7f7e5c0cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43240/0 (socket says 172.21.15.6:43240) 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.663+0000 7f7e59e00640 1 -- 172.21.15.6:0/3169950951 learned_addr learned my addr 172.21.15.6:0/3169950951 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.663+0000 7f7e53e00640 1 --2- 172.21.15.6:0/3169950951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b5060 0x7f7e5c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.663+0000 7f7e59e00640 1 -- 172.21.15.6:0/3169950951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b5060 msgr2=0x7f7e5c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.663+0000 7f7e59e00640 1 --2- 172.21.15.6:0/3169950951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b5060 0x7f7e5c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.663+0000 7f7e59e00640 1 -- 172.21.15.6:0/3169950951 >> v1:172.21.15.178:6789/0 conn(0x7f7e5c0b54a0 legacy=0x7f7e5c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:30.666 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.663+0000 7f7e59e00640 1 -- 172.21.15.6:0/3169950951 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0cefd0 con 0x7f7e5c161b40 2024-03-06T07:39:30.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.665+0000 7f7e59e00640 1 --2- 172.21.15.6:0/3169950951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c161b40 0x7f7e5c0cfd90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009870 tx=0x7f7e4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f33a5f77312e28 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.665+0000 7f7e59400640 1 -- 172.21.15.6:0/3169950951 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c035020 con 0x7f7e5c161b40 2024-03-06T07:39:30.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.665+0000 7f7e59400640 1 -- 172.21.15.6:0/3169950951 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c031640 con 0x7f7e5c161b40 2024-03-06T07:39:30.667 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.665+0000 7f7e59400640 1 -- 172.21.15.6:0/3169950951 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c02d040 con 0x7f7e5c161b40 2024-03-06T07:39:30.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 -- 172.21.15.6:0/3169950951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c161b40 msgr2=0x7f7e5c0cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 --2- 172.21.15.6:0/3169950951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c161b40 0x7f7e5c0cfd90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009870 tx=0x7f7e4c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.668 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 -- 172.21.15.6:0/3169950951 shutdown_connections 2024-03-06T07:39:30.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 --2- 172.21.15.6:0/3169950951 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b5060 0x7f7e5c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 --2- 172.21.15.6:0/3169950951 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c161b40 0x7f7e5c0cfd90 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 -- 172.21.15.6:0/3169950951 >> 172.21.15.6:0/3169950951 conn(0x7f7e5c09d220 msgr2=0x7f7e5c0d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:30.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 -- 172.21.15.6:0/3169950951 shutdown_connections 2024-03-06T07:39:30.669 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.666+0000 7f7e60c00640 1 -- 172.21.15.6:0/3169950951 wait complete. 2024-03-06T07:39:30.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.669+0000 7f7e60c00640 1 Processor -- start 2024-03-06T07:39:30.671 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.669+0000 7f7e60c00640 1 -- start start 2024-03-06T07:39:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e60c00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 0x7f7e5c1f7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e60c00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b54a0 0x7f7e5c0c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e60c00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e5c161b40 0x7f7e5c15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e60c00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0b2ff0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e60c00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0cf590 con 0x7f7e5c161b40 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e60c00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0b3170 con 0x7f7e5c0b54a0 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e59e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 0x7f7e5c1f7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e59e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 0x7f7e5c1f7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43252/0 (socket says 172.21.15.6:43252) 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.671+0000 7f7e59e00640 1 -- 172.21.15.6:0/2446359811 learned_addr learned my addr 172.21.15.6:0/2446359811 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e53e00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b54a0 0x7f7e5c0c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e5a800640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e5c161b40 0x7f7e5c15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e59e00640 1 -- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b54a0 msgr2=0x7f7e5c0c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e59e00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b54a0 0x7f7e5c0c9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e59e00640 1 -- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e5c161b40 msgr2=0x7f7e5c15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e59e00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e5c161b40 0x7f7e5c15f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.676 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e59e00640 1 -- 172.21.15.6:0/2446359811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c1f7880 con 0x7f7e5c0b5060 2024-03-06T07:39:30.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e59e00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 0x7f7e5c1f7340 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0097c0 tx=0x7f7e4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c03aa60 con 0x7f7e5c0b5060 2024-03-06T07:39:30.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.672+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c03ac00 con 0x7f7e5c0b5060 2024-03-06T07:39:30.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.673+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c0c9b30 con 0x7f7e5c0b5060 2024-03-06T07:39:30.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.673+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0443f0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.677 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.673+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c0ca0f0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.674+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c0d1df0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.678 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.675+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e4c04b060 con 0x7f7e5c0b5060 2024-03-06T07:39:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.676+0000 7f7e52000640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e3806b4d0 0x7f7e3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.676+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e4c0abfb0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.676+0000 7f7e53e00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e3806b4d0 0x7f7e3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:30.679 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.678+0000 7f7e53e00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e3806b4d0 0x7f7e3806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c152c80 tx=0x7f7e44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:30.683 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.681+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e4c03f070 con 0x7f7e5c0b5060 2024-03-06T07:39:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.882+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e5c06cc00 con 0x7f7e5c0b5060 2024-03-06T07:39:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.885+0000 7f7e52000640 1 -- 172.21.15.6:0/2446359811 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f7e4c07ade0 con 0x7f7e5c0b5060 2024-03-06T07:39:30.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-06T07:39:30.888 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":130,"fsid":"e57361cb-720d-49c4-a325-3e55e1c43024","created":"2024-03-06T07:27:04.140675+0000","modified":"2024-03-06T07:36:12.441784+0000","last_up_change":"2024-03-06T07:27:16.948529+0000","last_in_change":"2024-03-06T07:27:08.024203+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:27:17.511667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:35:28.426977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:35:29.515406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4d3c7a7e-7b00-4ed6-829d-322f3c73cc69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":129,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6801","nonce":2712143005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6803","nonce":2712143005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6807","nonce":2712143005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2712143005},{"type":"v1","addr":"172.21.15.6:6805","nonce":2712143005}]},"public_addr":"172.21.15.6:6801/2712143005","cluster_addr":"172.21.15.6:6803/2712143005","heartbeat_back_addr":"172.21.15.6:6807/2712143005","heartbeat_front_addr":"172.21.15.6:6805/2712143005","state":["exists","up"]},{"osd":1,"uuid":"be5e1794-8b2f-44e0-a366-01f108e9a562","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6809","nonce":1355309078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6811","nonce":1355309078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6815","nonce":1355309078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":1355309078},{"type":"v1","addr":"172.21.15.6:6813","nonce":1355309078}]},"public_addr":"172.21.15.6:6809/1355309078","cluster_addr":"172.21.15.6:6811/1355309078","heartbeat_back_addr":"172.21.15.6:6815/1355309078","heartbeat_front_addr":"172.21.15.6:6813/1355309078","state":["exists","up"]},{"osd":2,"uuid":"16638d16-3365-4de8-bb6c-b3f4dca67a9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6825","nonce":1963740543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6827","nonce":1963740543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6831","nonce":1963740543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":1963740543},{"type":"v1","addr":"172.21.15.6:6829","nonce":1963740543}]},"public_addr":"172.21.15.6:6825/1963740543","cluster_addr":"172.21.15.6:6827/1963740543","heartbeat_back_addr":"172.21.15.6:6831/1963740543","heartbeat_front_addr":"172.21.15.6:6829/1963740543","state":["exists","up"]},{"osd":3,"uuid":"a8a80788-be86-4b6f-8af6-c4b2fba8ac90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6817","nonce":1271003476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6819","nonce":1271003476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6823","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6824","nonce":1271003476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":1271003476},{"type":"v1","addr":"172.21.15.6:6821","nonce":1271003476}]},"public_addr":"172.21.15.6:6817/1271003476","cluster_addr":"172.21.15.6:6819/1271003476","heartbeat_back_addr":"172.21.15.6:6824/1271003476","heartbeat_front_addr":"172.21.15.6:6821/1271003476","state":["exists","up"]},{"osd":4,"uuid":"04f8a1e4-842b-4372-b1f3-1ef04081a7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6809","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6811","nonce":2203214816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6816","nonce":2203214816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6823","nonce":2203214816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6819","nonce":2203214816},{"type":"v1","addr":"172.21.15.178:6821","nonce":2203214816}]},"public_addr":"172.21.15.178:6811/2203214816","cluster_addr":"172.21.15.178:6816/2203214816","heartbeat_back_addr":"172.21.15.178:6823/2203214816","heartbeat_front_addr":"172.21.15.178:6821/2203214816","state":["exists","up"]},{"osd":5,"uuid":"6b8b8205-1df1-4537-b6c6-3477a9e720ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6801","nonce":1128394266}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6803","nonce":1128394266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6807","nonce":1128394266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":1128394266},{"type":"v1","addr":"172.21.15.178:6805","nonce":1128394266}]},"public_addr":"172.21.15.178:6801/1128394266","cluster_addr":"172.21.15.178:6803/1128394266","heartbeat_back_addr":"172.21.15.178:6807/1128394266","heartbeat_front_addr":"172.21.15.178:6805/1128394266","state":["exists","up"]},{"osd":6,"uuid":"4348e701-3acb-4a7c-b0a9-8951f623e015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6825","nonce":196133891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6827","nonce":196133891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6831","nonce":196133891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":196133891},{"type":"v1","addr":"172.21.15.178:6829","nonce":196133891}]},"public_addr":"172.21.15.178:6825/196133891","cluster_addr":"172.21.15.178:6827/196133891","heartbeat_back_addr":"172.21.15.178:6831/196133891","heartbeat_front_addr":"172.21.15.178:6829/196133891","state":["exists","up"]},{"osd":7,"uuid":"bee5fa45-a2e4-49e5-b380-e32387dbb269","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6810","nonce":56642032}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6813","nonce":56642032}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6820","nonce":56642032}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6815","nonce":56642032},{"type":"v1","addr":"172.21.15.178:6817","nonce":56642032}]},"public_addr":"172.21.15.178:6810/56642032","cluster_addr":"172.21.15.178:6813/56642032","heartbeat_back_addr":"172.21.15.178:6820/56642032","heartbeat_front_addr":"172.21.15.178:6817/56642032","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.277905+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.644190+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.359120+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.568020+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.340248+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.372677+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:27:13.565513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.7","mappings":[{"from":2,"to":0}]},{"pgid":"19.25","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:39:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e3806b4d0 msgr2=0x7f7e3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.890 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e3806b4d0 0x7f7e3806d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c152c80 tx=0x7f7e44008040 comp rx=0 tx=0).stop 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 msgr2=0x7f7e5c1f7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 0x7f7e5c1f7340 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0097c0 tx=0x7f7e4c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 shutdown_connections 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f7e3806b4d0 0x7f7e3806d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e5c161b40 0x7f7e5c15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f7e5c0b54a0 0x7f7e5c0c9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 --2- 172.21.15.6:0/2446359811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7e5c0b5060 0x7f7e5c1f7340 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 >> 172.21.15.6:0/2446359811 conn(0x7f7e5c09d220 msgr2=0x7f7e5c0b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 shutdown_connections 2024-03-06T07:39:30.891 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:30.888+0000 7f7e60c00640 1 -- 172.21.15.6:0/2446359811 wait complete. 2024-03-06T07:39:30.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:27:17.511667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:35:28.426977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:35:29.515406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:39:30.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:39:31.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.145+0000 7f64fe000640 1 Processor -- start 2024-03-06T07:39:31.148 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.146+0000 7f64fe000640 1 -- start start 2024-03-06T07:39:31.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.147+0000 7f64fe000640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f64f8145430 0x7f64f815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.147+0000 7f64fe000640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64f815cc90 0x7f64f80b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.147+0000 7f64fe000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f80b2080 0x7f64f806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.151 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.147+0000 7f64fe000640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f815fe40 con 0x7f64f815cc90 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.147+0000 7f64fe000640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f8160120 con 0x7f64f8145430 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.147+0000 7f64fe000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f8160400 con 0x7f64f80b2080 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.148+0000 7f64f7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64f815cc90 0x7f64f80b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.148+0000 7f64f7e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f64f815cc90 0x7f64f80b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43266/0 (socket says 172.21.15.6:43266) 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.148+0000 7f64f7e00640 1 -- 172.21.15.6:0/3266092002 learned_addr learned my addr 172.21.15.6:0/3266092002 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.148+0000 7f64fd600640 1 --2- 172.21.15.6:0/3266092002 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f80b2080 0x7f64f806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.148+0000 7f64fcc00640 1 --2- 172.21.15.6:0/3266092002 >> v2:172.21.15.178:3300/0 conn(0x7f64f8145430 0x7f64f815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.148+0000 7f64f7e00640 1 -- 172.21.15.6:0/3266092002 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f80b2080 msgr2=0x7f64f806c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.149+0000 7f64f7e00640 1 --2- 172.21.15.6:0/3266092002 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f80b2080 0x7f64f806c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.149+0000 7f64f7e00640 1 -- 172.21.15.6:0/3266092002 >> v2:172.21.15.178:3300/0 conn(0x7f64f8145430 msgr2=0x7f64f815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.149+0000 7f64f7e00640 1 --2- 172.21.15.6:0/3266092002 >> v2:172.21.15.178:3300/0 conn(0x7f64f8145430 0x7f64f815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.149+0000 7f64f7e00640 1 -- 172.21.15.6:0/3266092002 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f81606e0 con 0x7f64f815cc90 2024-03-06T07:39:31.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.150+0000 7f64f7e00640 1 --2- 172.21.15.6:0/3266092002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f80b1b40 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f64f80b2670 tx=0x7f64e802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e355617867925002 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.150+0000 7f64f7400640 1 -- 172.21.15.6:0/3266092002 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8035020 con 0x7f64f815cc90 2024-03-06T07:39:31.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.150+0000 7f64f7400640 1 -- 172.21.15.6:0/3266092002 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e802d040 con 0x7f64f815cc90 2024-03-06T07:39:31.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.150+0000 7f64f7400640 1 -- 172.21.15.6:0/3266092002 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8031a80 con 0x7f64f815cc90 2024-03-06T07:39:31.155 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 -- 172.21.15.6:0/3266092002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 msgr2=0x7f64f80b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 --2- 172.21.15.6:0/3266092002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f80b1b40 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f64f80b2670 tx=0x7f64e802b900 comp rx=0 tx=0).stop 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 -- 172.21.15.6:0/3266092002 shutdown_connections 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 --2- 172.21.15.6:0/3266092002 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f80b2080 0x7f64f806c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 --2- 172.21.15.6:0/3266092002 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f80b1b40 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 --2- 172.21.15.6:0/3266092002 >> v2:172.21.15.178:3300/0 conn(0x7f64f8145430 0x7f64f815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 -- 172.21.15.6:0/3266092002 >> 172.21.15.6:0/3266092002 conn(0x7f64f806b3a0 msgr2=0x7f64f806b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.156 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 -- 172.21.15.6:0/3266092002 shutdown_connections 2024-03-06T07:39:31.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.151+0000 7f64fe000640 1 -- 172.21.15.6:0/3266092002 wait complete. 2024-03-06T07:39:31.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.153+0000 7f64fe000640 1 Processor -- start 2024-03-06T07:39:31.157 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.153+0000 7f64fe000640 1 -- start start 2024-03-06T07:39:31.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fe000640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f80b2080 0x7f64f81423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fe000640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f8145430 0x7f64f80aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.159 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fe000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f81f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fe000640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f8145990 con 0x7f64f815cc90 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fe000640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f8145810 con 0x7f64f80b2080 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fe000640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80cde00 con 0x7f64f8145430 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f81f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fd600640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f81f17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43272/0 (socket says 172.21.15.6:43272) 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fd600640 1 -- 172.21.15.6:0/859306899 learned_addr learned my addr 172.21.15.6:0/859306899 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64fcc00640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f80b2080 0x7f64f81423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.154+0000 7f64f7e00640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f8145430 0x7f64f80aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.160 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fd600640 1 -- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f8145430 msgr2=0x7f64f80aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fd600640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f8145430 0x7f64f80aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fd600640 1 -- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f80b2080 msgr2=0x7f64f81423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fd600640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f80b2080 0x7f64f81423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fd600640 1 -- 172.21.15.6:0/859306899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f80af360 con 0x7f64f815cc90 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fd600640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f81f17c0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f64f0007e40 tx=0x7f64f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0014620 con 0x7f64f815cc90 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f0019070 con 0x7f64f815cc90 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f0014a20 con 0x7f64f815cc90 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f81429c0 con 0x7f64f815cc90 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.155+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f8142f80 con 0x7f64f815cc90 2024-03-06T07:39:31.161 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.156+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f804f330 con 0x7f64f815cc90 2024-03-06T07:39:31.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.158+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64f0014bc0 con 0x7f64f815cc90 2024-03-06T07:39:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.159+0000 7f64f4c00640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64dc06b4d0 0x7f64dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.160+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f64f008abb0 con 0x7f64f815cc90 2024-03-06T07:39:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.160+0000 7f64fcc00640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64dc06b4d0 0x7f64dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.163 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.162+0000 7f64fcc00640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64dc06b4d0 0x7f64dc06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f64ec006fd0 tx=0x7f64ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.166 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.164+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64f0059950 con 0x7f64f815cc90 2024-03-06T07:39:31.368 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.366+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64f813f910 con 0x7f64f815cc90 2024-03-06T07:39:31.369 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.367+0000 7f64f4c00640 1 -- 172.21.15.6:0/859306899 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64f0059250 con 0x7f64f815cc90 2024-03-06T07:39:31.370 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-06T07:39:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.369+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64dc06b4d0 msgr2=0x7f64dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.369+0000 7f64fe000640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64dc06b4d0 0x7f64dc06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f64ec006fd0 tx=0x7f64ec008040 comp rx=0 tx=0).stop 2024-03-06T07:39:31.372 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.369+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 msgr2=0x7f64f81f17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.369+0000 7f64fe000640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f81f17c0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f64f0007e40 tx=0x7f64f000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 shutdown_connections 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f64dc06b4d0 0x7f64dc06d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f64f815cc90 0x7f64f81f17c0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f64f8145430 0x7f64f80aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 --2- 172.21.15.6:0/859306899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64f80b2080 0x7f64f81423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 >> 172.21.15.6:0/859306899 conn(0x7f64f806b3a0 msgr2=0x7f64f815d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 shutdown_connections 2024-03-06T07:39:31.373 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.370+0000 7f64fe000640 1 -- 172.21.15.6:0/859306899 wait complete. 2024-03-06T07:39:31.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:39:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.615+0000 7f6a83e00640 1 Processor -- start 2024-03-06T07:39:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.615+0000 7f6a83e00640 1 -- start start 2024-03-06T07:39:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.618+0000 7f6a83e00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f6a840aef20 0x7f6a840af300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.618+0000 7f6a83e00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a841618e0 0x7f6a840ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.618+0000 7f6a83e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a8415df80 0x7f6a8415e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a83e00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a840aece0 con 0x7f6a841618e0 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a83e00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a840af840 con 0x7f6a840aef20 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a83e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840ae0d0 con 0x7f6a8415df80 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a841618e0 0x7f6a840ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a841618e0 0x7f6a840ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43280/0 (socket says 172.21.15.6:43280) 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 -- 172.21.15.6:0/2385485580 learned_addr learned my addr 172.21.15.6:0/2385485580 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a83400640 1 --2- 172.21.15.6:0/2385485580 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a8415df80 0x7f6a8415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a82a00640 1 --2- 172.21.15.6:0/2385485580 >> v2:172.21.15.178:3300/0 conn(0x7f6a840aef20 0x7f6a840af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 -- 172.21.15.6:0/2385485580 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a8415df80 msgr2=0x7f6a8415e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 --2- 172.21.15.6:0/2385485580 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a8415df80 0x7f6a8415e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 -- 172.21.15.6:0/2385485580 >> v2:172.21.15.178:3300/0 conn(0x7f6a840aef20 msgr2=0x7f6a840af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 --2- 172.21.15.6:0/2385485580 >> v2:172.21.15.178:3300/0 conn(0x7f6a840aef20 0x7f6a840af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.619+0000 7f6a7be00640 1 -- 172.21.15.6:0/2385485580 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a840ae3b0 con 0x7f6a841618e0 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.620+0000 7f6a82a00640 1 --2- 172.21.15.6:0/2385485580 >> v2:172.21.15.178:3300/0 conn(0x7f6a840aef20 0x7f6a840af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.621+0000 7f6a7be00640 1 --2- 172.21.15.6:0/2385485580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a841618e0 0x7f6a840ad880 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f6a84072930 tx=0x7f6a6c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936906215a814cef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.621+0000 7f6a82000640 1 -- 172.21.15.6:0/2385485580 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0317d0 con 0x7f6a841618e0 2024-03-06T07:39:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.621+0000 7f6a82000640 1 -- 172.21.15.6:0/2385485580 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c037070 con 0x7f6a841618e0 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.621+0000 7f6a82000640 1 -- 172.21.15.6:0/2385485580 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c031b90 con 0x7f6a841618e0 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 -- 172.21.15.6:0/2385485580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a841618e0 msgr2=0x7f6a840ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 --2- 172.21.15.6:0/2385485580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a841618e0 0x7f6a840ad880 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f6a84072930 tx=0x7f6a6c02b870 comp rx=0 tx=0).stop 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 -- 172.21.15.6:0/2385485580 shutdown_connections 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 --2- 172.21.15.6:0/2385485580 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a8415df80 0x7f6a8415e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 --2- 172.21.15.6:0/2385485580 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a841618e0 0x7f6a840ad880 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 --2- 172.21.15.6:0/2385485580 >> v2:172.21.15.178:3300/0 conn(0x7f6a840aef20 0x7f6a840af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 -- 172.21.15.6:0/2385485580 >> 172.21.15.6:0/2385485580 conn(0x7f6a840723d0 msgr2=0x7f6a8406afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 -- 172.21.15.6:0/2385485580 shutdown_connections 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.622+0000 7f6a83e00640 1 -- 172.21.15.6:0/2385485580 wait complete. 2024-03-06T07:39:31.630 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.624+0000 7f6a83e00640 1 Processor -- start 2024-03-06T07:39:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.624+0000 7f6a83e00640 1 -- start start 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83e00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 0x7f6a841f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83e00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a8415df80 0x7f6a84142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83e00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a841618e0 0x7f6a84150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83e00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840addc0 con 0x7f6a840aef20 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83e00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840ae970 con 0x7f6a8415df80 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83e00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8415e950 con 0x7f6a841618e0 2024-03-06T07:39:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a82a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 0x7f6a841f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a82a00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 0x7f6a841f7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43290/0 (socket says 172.21.15.6:43290) 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a82a00640 1 -- 172.21.15.6:0/4200149818 learned_addr learned my addr 172.21.15.6:0/4200149818 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a7be00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a8415df80 0x7f6a84142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.626+0000 7f6a83400640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a841618e0 0x7f6a84150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a82a00640 1 -- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a841618e0 msgr2=0x7f6a84150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a82a00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a841618e0 0x7f6a84150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a82a00640 1 -- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a8415df80 msgr2=0x7f6a84142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a82a00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a8415df80 0x7f6a84142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a82a00640 1 -- 172.21.15.6:0/4200149818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a841f45e0 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a83400640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a841618e0 0x7f6a84150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a82a00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 0x7f6a841f7890 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400c7c0 tx=0x7f6a7400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74015070 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a74007a30 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a840b5680 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74007e00 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.627+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a841f6db0 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.629+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8404f330 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.630+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a74018070 con 0x7f6a840aef20 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.631+0000 7f6a7aa00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a6006b4d0 0x7f6a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:31.636 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.631+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6a7408a1d0 con 0x7f6a840aef20 2024-03-06T07:39:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.632+0000 7f6a7be00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a6006b4d0 0x7f6a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:31.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.633+0000 7f6a7be00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a6006b4d0 0x7f6a6006d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c036000 tx=0x7f6a6c03d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:31.638 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.636+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a74058f70 con 0x7f6a840aef20 2024-03-06T07:39:31.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.835+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6a8406c980 con 0x7f6a840aef20 2024-03-06T07:39:31.838 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.836+0000 7f6a7aa00640 1 -- 172.21.15.6:0/4200149818 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a74058870 con 0x7f6a840aef20 2024-03-06T07:39:31.838 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 32 2024-03-06T07:39:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a6006b4d0 msgr2=0x7f6a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a6006b4d0 0x7f6a6006d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c036000 tx=0x7f6a6c03d040 comp rx=0 tx=0).stop 2024-03-06T07:39:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 msgr2=0x7f6a841f7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 0x7f6a841f7890 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400c7c0 tx=0x7f6a7400cc80 comp rx=0 tx=0).stop 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 shutdown_connections 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f6a6006b4d0 0x7f6a6006d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6a8415df80 0x7f6a84142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f6a841618e0 0x7f6a84150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 --2- 172.21.15.6:0/4200149818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a840aef20 0x7f6a841f7890 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 >> 172.21.15.6:0/4200149818 conn(0x7f6a840723d0 msgr2=0x7f6a8406f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 shutdown_connections 2024-03-06T07:39:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:31.839+0000 7f6a83e00640 1 -- 172.21.15.6:0/4200149818 wait complete. 2024-03-06T07:39:31.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:39:32.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.080+0000 7f705fc00640 1 Processor -- start 2024-03-06T07:39:32.082 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.080+0000 7f705fc00640 1 -- start start 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.081+0000 7f705fc00640 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f705815e280 0x7f70580d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.081+0000 7f705fc00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580d0080 con 0x7f70580d2990 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.081+0000 7f705fc00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580cf500 con 0x7f70580d4a00 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.081+0000 7f705fc00640 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70580cf7e0 con 0x7f705815e280 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.082+0000 7f705dc00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f70580d2990 0x7f70580cb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50016/0 (socket says 172.21.15.6:50016) 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.082+0000 7f705dc00640 1 -- 172.21.15.6:0/3878545840 learned_addr learned my addr 172.21.15.6:0/3878545840 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.082+0000 7f705d200640 1 --2- 172.21.15.6:0/3878545840 >> v2:172.21.15.6:3300/0 conn(0x7f705815e280 0x7f70580d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.082+0000 7f7057400640 1 -- 172.21.15.6:0/3878545840 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492974590 0 0) 0x7f70580d0080 con 0x7f70580d2990 2024-03-06T07:39:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.083+0000 7f7057400640 1 -- 172.21.15.6:0/3878545840 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7040003520 con 0x7f70580d2990 2024-03-06T07:39:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.083+0000 7f705d200640 1 -- 172.21.15.6:0/3878545840 >> v1:172.21.15.178:6789/0 conn(0x7f70580d4a00 legacy=0x7f70580d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.083+0000 7f705d200640 1 -- 172.21.15.6:0/3878545840 >> v1:172.21.15.6:6789/0 conn(0x7f70580d2990 legacy=0x7f70580cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.083+0000 7f705d200640 1 -- 172.21.15.6:0/3878545840 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580cfac0 con 0x7f705815e280 2024-03-06T07:39:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.084+0000 7f705d200640 1 --2- 172.21.15.6:0/3878545840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580d1cc0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f704c0097a0 tx=0x7f704c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33734be1df294d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.084+0000 7f7057400640 1 -- 172.21.15.6:0/3878545840 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c035020 con 0x7f705815e280 2024-03-06T07:39:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.084+0000 7f7057400640 1 -- 172.21.15.6:0/3878545840 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c031830 con 0x7f705815e280 2024-03-06T07:39:32.086 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.084+0000 7f7057400640 1 -- 172.21.15.6:0/3878545840 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c031c00 con 0x7f705815e280 2024-03-06T07:39:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 -- 172.21.15.6:0/3878545840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 msgr2=0x7f70580d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 --2- 172.21.15.6:0/3878545840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580d1cc0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f704c0097a0 tx=0x7f704c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 -- 172.21.15.6:0/3878545840 shutdown_connections 2024-03-06T07:39:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 --2- 172.21.15.6:0/3878545840 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580d1cc0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 -- 172.21.15.6:0/3878545840 >> 172.21.15.6:0/3878545840 conn(0x7f70580d3ee0 msgr2=0x7f70580d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 -- 172.21.15.6:0/3878545840 shutdown_connections 2024-03-06T07:39:32.088 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.085+0000 7f705fc00640 1 -- 172.21.15.6:0/3878545840 wait complete. 2024-03-06T07:39:32.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.087+0000 7f705fc00640 1 Processor -- start 2024-03-06T07:39:32.089 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.088+0000 7f705fc00640 1 -- start start 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705fc00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70580d2990 0x7f70580a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705fc00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70580d4a00 0x7f70581577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705fc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705fc00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705815e660 con 0x7f705815e280 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705fc00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70580cb0e0 con 0x7f70580d2990 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705fc00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70580cbd10 con 0x7f70580d4a00 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.089+0000 7f705dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44554/0 (socket says 172.21.15.6:44554) 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 -- 172.21.15.6:0/3752353938 learned_addr learned my addr 172.21.15.6:0/3752353938 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705d200640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70580d2990 0x7f70580a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f7057e00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70580d4a00 0x7f70581577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 -- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70580d4a00 msgr2=0x7f70581577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70580d4a00 0x7f70581577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 -- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70580d2990 msgr2=0x7f70580a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70580d2990 0x7f70580a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 -- 172.21.15.6:0/3752353938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70581f7590 con 0x7f705815e280 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f7057e00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70580d4a00 0x7f70581577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.090+0000 7f705dc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580cc7b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f705000cb70 tx=0x7f705000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.091+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7050017070 con 0x7f705815e280 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.091+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705000ad40 con 0x7f705815e280 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.091+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7050012620 con 0x7f705815e280 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.091+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70581f7870 con 0x7f705815e280 2024-03-06T07:39:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.091+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70581f7e30 con 0x7f705815e280 2024-03-06T07:39:32.095 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.092+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705804f330 con 0x7f705815e280 2024-03-06T07:39:32.096 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.094+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f705001a070 con 0x7f705815e280 2024-03-06T07:39:32.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.095+0000 7f7054c00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f703806b4d0 0x7f703806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.095+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f705008a0d0 con 0x7f705815e280 2024-03-06T07:39:32.097 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.096+0000 7f705d200640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f703806b4d0 0x7f703806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.099 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.097+0000 7f705d200640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f703806b4d0 0x7f703806d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f704c036000 tx=0x7f704c03c040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.102 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.100+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7050058f00 con 0x7f705815e280 2024-03-06T07:39:32.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.306+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70580a1ad0 con 0x7f705815e280 2024-03-06T07:39:32.309 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.307+0000 7f7054c00640 1 -- 172.21.15.6:0/3752353938 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7050058800 con 0x7f705815e280 2024-03-06T07:39:32.309 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 64 2024-03-06T07:39:32.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.309+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f703806b4d0 msgr2=0x7f703806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.309+0000 7f705fc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f703806b4d0 0x7f703806d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f704c036000 tx=0x7f704c03c040 comp rx=0 tx=0).stop 2024-03-06T07:39:32.312 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.309+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 msgr2=0x7f70580cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.309+0000 7f705fc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580cc7b0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f705000cb70 tx=0x7f705000a430 comp rx=0 tx=0).stop 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 shutdown_connections 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f703806b4d0 0x7f703806d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f705815e280 0x7f70580cc7b0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f70580d4a00 0x7f70581577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 --2- 172.21.15.6:0/3752353938 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70580d2990 0x7f70580a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 >> 172.21.15.6:0/3752353938 conn(0x7f70580d3ee0 msgr2=0x7f705806ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 shutdown_connections 2024-03-06T07:39:32.313 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.310+0000 7f705fc00640 1 -- 172.21.15.6:0/3752353938 wait complete. 2024-03-06T07:39:32.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:39:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.539+0000 7f3c8da00640 1 Processor -- start 2024-03-06T07:39:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.539+0000 7f3c8da00640 1 -- start start 2024-03-06T07:39:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c8da00640 1 --2- >> v2:172.21.15.178:3300/0 conn(0x7f3c8815e2a0 0x7f3c880cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c8da00640 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c880d1120 con 0x7f3c8815dec0 2024-03-06T07:39:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c8da00640 1 -- --> v1:172.21.15.178:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c880d1400 con 0x7f3c880cf460 2024-03-06T07:39:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c8da00640 1 -- --> v2:172.21.15.178:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c880b0ea0 con 0x7f3c8815e2a0 2024-03-06T07:39:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c87e00640 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3c8815dec0 0x7f3c880ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50032/0 (socket says 172.21.15.6:50032) 2024-03-06T07:39:32.543 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c87e00640 1 -- 172.21.15.6:0/1239035488 learned_addr learned my addr 172.21.15.6:0/1239035488 (peer_addr_for_me v1:172.21.15.6:0/0) 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.541+0000 7f3c7fe00640 1 --2- 172.21.15.6:0/1239035488 >> v2:172.21.15.178:3300/0 conn(0x7f3c8815e2a0 0x7f3c880cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013269305 0 0) 0x7f3c880d1120 con 0x7f3c8815dec0 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c003520 con 0x7f3c8815dec0 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2078488247 0 0) 0x7f3c6c003520 con 0x7f3c8815dec0 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 >> v2:172.21.15.178:3300/0 conn(0x7f3c8815e2a0 msgr2=0x7f3c880cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 --2- 172.21.15.6:0/1239035488 >> v2:172.21.15.178:3300/0 conn(0x7f3c8815e2a0 0x7f3c880cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 >> v1:172.21.15.178:6789/0 conn(0x7f3c880cf460 legacy=0x7f3c880b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c880b0b30 con 0x7f3c8815dec0 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 89999837 0 0) 0x7f3c78001670 con 0x7f3c8815dec0 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 >> v1:172.21.15.6:6789/0 conn(0x7f3c8815dec0 legacy=0x7f3c880ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c6c0039b0 0x7f3c6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c004800 0x7f3c6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.546 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c6c0051c0 0x7f3c6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.542+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880b0ea0 con 0x7f3c6c0039b0 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c00c060 con 0x7f3c6c004800 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c00c260 con 0x7f3c6c0051c0 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c6c0039b0 0x7f3c6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c7fe00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c004800 0x7f3c6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c8d000640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c6c0051c0 0x7f3c6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87e00640 1 -- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c6c0051c0 msgr2=0x7f3c6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c6c0051c0 0x7f3c6c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87e00640 1 -- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c004800 msgr2=0x7f3c6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c004800 0x7f3c6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c87e00640 1 -- 172.21.15.6:0/1239035488 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c003520 con 0x7f3c6c0039b0 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c7fe00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c004800 0x7f3c6c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:39:32.547 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.543+0000 7f3c8d000640 1 -- 172.21.15.6:0/1239035488 reap_dead start 2024-03-06T07:39:32.548 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.544+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c6c0039b0 0x7f3c6c003dd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800c210 tx=0x7f3c6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e959aa0155ed68c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.545+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78007d30 con 0x7f3c6c0039b0 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.545+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78030070 con 0x7f3c6c0039b0 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.545+0000 7f3c87400640 1 -- 172.21.15.6:0/1239035488 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7802cda0 con 0x7f3c6c0039b0 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.545+0000 7f3c8da00640 1 -- 172.21.15.6:0/1239035488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c6c0039b0 msgr2=0x7f3c6c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.545+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c6c0039b0 0x7f3c6c003dd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800c210 tx=0x7f3c6c005830 comp rx=0 tx=0).stop 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.546+0000 7f3c8da00640 1 -- 172.21.15.6:0/1239035488 shutdown_connections 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.546+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1239035488 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c6c0039b0 0x7f3c6c003dd0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.546+0000 7f3c8da00640 1 -- 172.21.15.6:0/1239035488 >> 172.21.15.6:0/1239035488 conn(0x7f3c880d3d40 msgr2=0x7f3c880d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.546+0000 7f3c8da00640 1 -- 172.21.15.6:0/1239035488 shutdown_connections 2024-03-06T07:39:32.549 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.546+0000 7f3c8da00640 1 -- 172.21.15.6:0/1239035488 wait complete. 2024-03-06T07:39:32.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.549+0000 7f3c8da00640 1 Processor -- start 2024-03-06T07:39:32.551 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.549+0000 7f3c8da00640 1 -- start start 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8da00640 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c0039b0 0x7f3c881f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8da00640 1 --2- >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c880c79c0 0x7f3c880c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8da00640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 0x7f3c880c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8da00640 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880ce990 con 0x7f3c880c99b0 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8da00640 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880ce810 con 0x7f3c6c0039b0 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8da00640 1 -- --> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c880ceb10 con 0x7f3c880c79c0 2024-03-06T07:39:32.553 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 0x7f3c880c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.550+0000 7f3c8d000640 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 0x7f3c880c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44570/0 (socket says 172.21.15.6:44570) 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 -- 172.21.15.6:0/1738735235 learned_addr learned my addr 172.21.15.6:0/1738735235 (peer_addr_for_me v2:172.21.15.6:0/0) 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c0039b0 0x7f3c881f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c7fe00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c880c79c0 0x7f3c880c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 -- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c880c79c0 msgr2=0x7f3c880c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c880c79c0 0x7f3c880c7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 -- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c0039b0 msgr2=0x7f3c881f4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c0039b0 0x7f3c881f4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 -- 172.21.15.6:0/1738735235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c881f4a40 con 0x7f3c880c99b0 2024-03-06T07:39:32.554 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.551+0000 7f3c8d000640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 0x7f3c880c9e30 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000d770 tx=0x7f3c8000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.552+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80018070 con 0x7f3c880c99b0 2024-03-06T07:39:32.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.552+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c80003b70 con 0x7f3c880c99b0 2024-03-06T07:39:32.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.552+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80005070 con 0x7f3c880c99b0 2024-03-06T07:39:32.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.552+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c881f3e70 con 0x7f3c880c99b0 2024-03-06T07:39:32.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.552+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c880ca950 con 0x7f3c880c99b0 2024-03-06T07:39:32.555 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.553+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8804f330 con 0x7f3c880c99b0 2024-03-06T07:39:32.556 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.554+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c8001b070 con 0x7f3c880c99b0 2024-03-06T07:39:32.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.555+0000 7f3c84c00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3c6406b4d0 0x7f3c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:39:32.557 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.556+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3c80089880 con 0x7f3c880c99b0 2024-03-06T07:39:32.558 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.556+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3c6406b4d0 0x7f3c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:39:32.559 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.557+0000 7f3c87e00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3c6406b4d0 0x7f3c6406d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3c78034000 tx=0x7f3c78009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:39:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.559+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c80058750 con 0x7f3c880c99b0 2024-03-06T07:39:32.777 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.776+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3c880d21f0 con 0x7f3c880c99b0 2024-03-06T07:39:32.779 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.776+0000 7f3c84c00640 1 -- 172.21.15.6:0/1738735235 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v130) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3c80058050 con 0x7f3c880c99b0 2024-03-06T07:39:32.779 INFO:teuthology.orchestra.run.smithi006.stderr:listed 0 entries 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3c6406b4d0 msgr2=0x7f3c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3c6406b4d0 0x7f3c6406d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3c78034000 tx=0x7f3c78009030 comp rx=0 tx=0).stop 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 msgr2=0x7f3c880c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 0x7f3c880c9e30 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000d770 tx=0x7f3c8000dc30 comp rx=0 tx=0).stop 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 shutdown_connections 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:6832/3849979937,v1:172.21.15.178:6833/3849979937] conn(0x7f3c6406b4d0 0x7f3c6406d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3c880c99b0 0x7f3c880c9e30 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.782 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3301/0,v1:172.21.15.178:6790/0] conn(0x7f3c880c79c0 0x7f3c880c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 --2- 172.21.15.6:0/1738735235 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3c6c0039b0 0x7f3c881f4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:39:32.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.779+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 >> 172.21.15.6:0/1738735235 conn(0x7f3c880d3d40 msgr2=0x7f3c8806b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:39:32.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.780+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 shutdown_connections 2024-03-06T07:39:32.783 INFO:teuthology.orchestra.run.smithi006.stderr:2024-03-06T07:39:32.780+0000 7f3c8da00640 1 -- 172.21.15.6:0/1738735235 wait complete. 2024-03-06T07:39:32.804 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-06T07:39:32.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:39:32.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-06T07:39:55.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:39:55.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:25.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:25.224 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:55.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:40:55.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:25.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:25.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:55.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:41:55.654 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:25.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:25.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:55.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:42:55.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:26.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:26.093 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:56.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:43:56.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:26.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:26.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:32.813 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-03-06T07:44:32.815 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-06T07:44:56.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:44:56.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:26.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:26.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:56.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:45:56.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:27.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:27.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:57.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:46:57.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:27.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:27.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:57.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:47:57.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:27.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:27.731 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:57.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:48:57.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:28.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:28.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:58.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:49:58.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:28.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:28.313 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:58.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:50:58.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:28.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:28.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:58.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:51:58.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:28.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:28.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:59.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:52:59.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:29.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:29.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:59.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:53:59.340 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:29.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:29.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:59.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:54:59.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:29.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:29.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:59.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:55:59.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:30.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:56:30.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:00.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:00.219 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:30.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:57:30.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:00.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:00.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:30.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:58:30.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:00.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:00.845 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:30.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:59:30.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:01.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:01.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:31.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:00:31.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:01.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:01.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:31.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:01:31.624 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:01.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:01.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:31.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:02:31.916 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:02.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:02.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:32.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:03:32.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:02.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:02.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:32.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:04:32.545 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:02.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:02.682 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:32.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:05:32.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:02.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:02.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:33.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:06:33.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:03.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:03.273 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:33.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:07:33.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:03.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:03.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:33.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:08:33.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:03.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:03.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:33.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:09:33.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:04.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:04.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:34.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:10:34.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:04.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:04.456 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:34.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:11:34.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:04.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:04.748 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:34.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:12:34.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:05.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:05.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:35.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:13:35.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:05.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:05.641 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:35.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:14:35.786 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:05.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:05.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:36.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:15:36.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:06.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:06.245 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:36.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:16:36.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:06.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:06.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:36.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:17:36.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:06.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:06.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:37.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:18:37.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:07.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:07.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:37.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:19:37.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:07.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:07.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:37.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:20:37.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:08.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:08.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:38.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:21:38.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:08.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:08.431 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:38.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:22:38.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:08.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:08.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:38.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:23:38.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:09.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:09.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:39.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:24:39.175 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:09.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:09.321 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:39.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:25:39.470 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:09.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:09.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:39.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:26:39.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:09.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:09.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:40.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:27:40.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:10.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:10.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:40.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:28:40.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:10.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:10.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:40.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:29:40.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:10.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:10.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:40.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:30:40.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:11.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:11.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:41.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:31:41.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:11.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:11.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:41.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:32:41.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:11.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:11.766 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:42.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:33:42.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:12.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:12.227 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:42.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:34:42.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:12.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:12.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:42.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:35:42.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:12.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:12.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:42.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:36:42.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:13.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:13.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:43.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:37:43.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:13.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:13.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:43.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:38:43.895 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:04.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:04.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:04.567882+0000) 2024-03-06T08:39:05.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:05.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:05.568164+0000) 2024-03-06T08:39:06.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:06.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:06.568401+0000) 2024-03-06T08:39:07.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:07.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:07.568677+0000) 2024-03-06T08:39:08.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:08.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:08.568945+0000) 2024-03-06T08:39:09.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:09.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:09.569235+0000) 2024-03-06T08:39:10.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:10.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:10.569511+0000) 2024-03-06T08:39:11.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:11.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:11.569757+0000) 2024-03-06T08:39:12.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:12.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:12.570053+0000) 2024-03-06T08:39:13.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:13.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:13.570283+0000) 2024-03-06T08:39:14.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:14.039 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:14.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:14.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:14.570554+0000) 2024-03-06T08:39:15.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:15.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:15.570832+0000) 2024-03-06T08:39:16.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:16.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:16.571080+0000) 2024-03-06T08:39:17.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:17.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:17.571309+0000) 2024-03-06T08:39:18.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:18.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:18.571540+0000) 2024-03-06T08:39:19.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:19.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:19.571771+0000) 2024-03-06T08:39:20.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:20.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:20.572000+0000) 2024-03-06T08:39:21.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:21.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:21.572291+0000) 2024-03-06T08:39:22.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:22.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:22.572547+0000) 2024-03-06T08:39:23.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:23.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:23.572773+0000) 2024-03-06T08:39:24.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:24.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:24.573072+0000) 2024-03-06T08:39:25.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:25.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:25.573298+0000) 2024-03-06T08:39:26.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:26.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:26.573544+0000) 2024-03-06T08:39:27.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:27.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:27.573765+0000) 2024-03-06T08:39:28.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:28.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:28.574096+0000) 2024-03-06T08:39:29.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:29.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:29.574365+0000) 2024-03-06T08:39:30.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:30.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:30.574618+0000) 2024-03-06T08:39:31.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:31.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:31.574900+0000) 2024-03-06T08:39:32.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:32.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:32.575143+0000) 2024-03-06T08:39:33.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:33.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:33.575362+0000) 2024-03-06T08:39:34.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:34.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:34.575568+0000) 2024-03-06T08:39:35.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:35.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:35.575828+0000) 2024-03-06T08:39:36.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:36.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:36.576120+0000) 2024-03-06T08:39:37.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:37.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:37.576356+0000) 2024-03-06T08:39:38.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:38.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:38.576634+0000) 2024-03-06T08:39:39.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:39.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:39.576913+0000) 2024-03-06T08:39:40.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:40.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:40.577172+0000) 2024-03-06T08:39:41.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:41.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:41.577381+0000) 2024-03-06T08:39:42.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:42.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:42.577614+0000) 2024-03-06T08:39:43.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:43.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:43.577867+0000) 2024-03-06T08:39:44.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:44.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:39:44.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:44.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:44.578159+0000) 2024-03-06T08:39:45.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:45.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:45.578373+0000) 2024-03-06T08:39:46.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:46.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:46.578611+0000) 2024-03-06T08:39:47.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:47.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:47.578829+0000) 2024-03-06T08:39:48.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:48.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:48.579063+0000) 2024-03-06T08:39:49.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:49.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:49.579264+0000) 2024-03-06T08:39:50.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:50.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:50.579531+0000) 2024-03-06T08:39:51.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:51.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:51.579739+0000) 2024-03-06T08:39:52.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:52.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:52.579969+0000) 2024-03-06T08:39:53.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:53.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:53.580270+0000) 2024-03-06T08:39:54.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:54.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:54.580509+0000) 2024-03-06T08:39:55.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:55.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:55.580791+0000) 2024-03-06T08:39:56.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:56.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:56.581060+0000) 2024-03-06T08:39:57.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:57.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:57.581248+0000) 2024-03-06T08:39:58.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:58.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:58.581519+0000) 2024-03-06T08:39:59.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:39:59.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:39:59.581811+0000) 2024-03-06T08:40:00.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:00.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:00.582111+0000) 2024-03-06T08:40:01.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:01.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:01.582328+0000) 2024-03-06T08:40:02.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:02.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:02.582548+0000) 2024-03-06T08:40:03.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:03.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:03.582811+0000) 2024-03-06T08:40:04.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:04.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:04.583112+0000) 2024-03-06T08:40:05.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:05.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:05.583382+0000) 2024-03-06T08:40:06.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:06.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:06.583664+0000) 2024-03-06T08:40:07.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:07.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:07.583884+0000) 2024-03-06T08:40:08.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:08.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:08.584166+0000) 2024-03-06T08:40:09.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:09.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:09.584454+0000) 2024-03-06T08:40:10.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:10.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:10.584743+0000) 2024-03-06T08:40:11.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:11.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:11.585043+0000) 2024-03-06T08:40:12.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:12.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:12.585288+0000) 2024-03-06T08:40:13.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:13.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:13.585562+0000) 2024-03-06T08:40:14.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:14.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:14.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:14.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:14.585798+0000) 2024-03-06T08:40:15.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:15.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:15.586109+0000) 2024-03-06T08:40:16.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:16.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:16.586300+0000) 2024-03-06T08:40:17.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:17.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:17.586561+0000) 2024-03-06T08:40:18.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:18.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:18.586833+0000) 2024-03-06T08:40:19.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:19.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:19.587153+0000) 2024-03-06T08:40:20.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:20.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:20.587415+0000) 2024-03-06T08:40:21.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:21.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:21.587680+0000) 2024-03-06T08:40:22.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:22.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:22.587927+0000) 2024-03-06T08:40:23.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:23.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:23.588183+0000) 2024-03-06T08:40:24.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:24.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:24.588437+0000) 2024-03-06T08:40:25.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:25.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:25.588714+0000) 2024-03-06T08:40:26.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:26.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:26.588990+0000) 2024-03-06T08:40:27.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:27.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:27.589321+0000) 2024-03-06T08:40:28.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:28.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:28.589597+0000) 2024-03-06T08:40:29.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:29.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:29.589873+0000) 2024-03-06T08:40:30.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:30.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:30.590146+0000) 2024-03-06T08:40:31.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:31.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:31.590358+0000) 2024-03-06T08:40:32.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:32.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:32.590597+0000) 2024-03-06T08:40:33.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:33.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:33.590812+0000) 2024-03-06T08:40:34.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:34.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:34.591076+0000) 2024-03-06T08:40:35.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:35.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:35.591304+0000) 2024-03-06T08:40:36.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:36.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:36.591581+0000) 2024-03-06T08:40:37.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:37.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:37.591855+0000) 2024-03-06T08:40:38.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:38.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:38.592155+0000) 2024-03-06T08:40:39.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:39.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:39.592404+0000) 2024-03-06T08:40:40.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:40.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:40.592640+0000) 2024-03-06T08:40:41.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:41.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:41.592889+0000) 2024-03-06T08:40:42.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:42.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:42.593185+0000) 2024-03-06T08:40:43.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:43.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:43.593459+0000) 2024-03-06T08:40:44.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:44.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:40:44.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:44.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:44.593691+0000) 2024-03-06T08:40:45.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:45.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:45.593906+0000) 2024-03-06T08:40:46.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:46.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:46.594148+0000) 2024-03-06T08:40:47.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:47.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:47.594370+0000) 2024-03-06T08:40:48.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:48.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:48.594597+0000) 2024-03-06T08:40:49.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:49.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:49.594835+0000) 2024-03-06T08:40:50.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:50.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:50.595121+0000) 2024-03-06T08:40:51.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:51.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:51.595371+0000) 2024-03-06T08:40:52.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:52.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:52.595602+0000) 2024-03-06T08:40:53.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:53.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:53.595853+0000) 2024-03-06T08:40:54.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:54.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:54.596114+0000) 2024-03-06T08:40:55.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:55.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:55.596350+0000) 2024-03-06T08:40:56.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:56.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:56.596603+0000) 2024-03-06T08:40:57.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:57.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:57.596879+0000) 2024-03-06T08:40:58.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:58.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:58.597164+0000) 2024-03-06T08:40:59.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:40:59.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:40:59.597377+0000) 2024-03-06T08:41:00.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:00.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:00.597596+0000) 2024-03-06T08:41:01.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:01.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:01.597862+0000) 2024-03-06T08:41:02.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:02.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:02.598177+0000) 2024-03-06T08:41:03.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:03.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:03.598418+0000) 2024-03-06T08:41:04.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:04.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:04.598686+0000) 2024-03-06T08:41:05.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:05.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:05.598950+0000) 2024-03-06T08:41:06.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:06.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:06.599214+0000) 2024-03-06T08:41:07.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:07.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:07.599451+0000) 2024-03-06T08:41:08.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:08.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:08.599649+0000) 2024-03-06T08:41:09.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:09.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:09.599841+0000) 2024-03-06T08:41:10.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:10.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:10.600155+0000) 2024-03-06T08:41:11.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:11.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:11.600403+0000) 2024-03-06T08:41:12.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:12.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:12.600653+0000) 2024-03-06T08:41:13.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:13.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:13.600910+0000) 2024-03-06T08:41:14.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:14.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:14.601146+0000) 2024-03-06T08:41:14.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:14.679 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:15.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:15.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:15.601374+0000) 2024-03-06T08:41:16.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:16.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:16.601613+0000) 2024-03-06T08:41:17.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:17.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:17.601840+0000) 2024-03-06T08:41:18.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:18.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:18.602114+0000) 2024-03-06T08:41:19.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:19.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:19.602347+0000) 2024-03-06T08:41:20.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:20.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:20.602583+0000) 2024-03-06T08:41:21.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:21.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:21.602831+0000) 2024-03-06T08:41:22.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:22.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:22.603116+0000) 2024-03-06T08:41:23.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:23.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:23.603347+0000) 2024-03-06T08:41:24.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:24.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:24.603573+0000) 2024-03-06T08:41:25.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:25.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:25.603800+0000) 2024-03-06T08:41:26.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:26.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:26.604080+0000) 2024-03-06T08:41:27.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:27.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:27.604314+0000) 2024-03-06T08:41:28.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:28.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:28.604549+0000) 2024-03-06T08:41:29.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:29.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:29.604762+0000) 2024-03-06T08:41:30.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:30.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:30.605060+0000) 2024-03-06T08:41:31.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:31.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:31.605294+0000) 2024-03-06T08:41:32.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:32.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:32.605554+0000) 2024-03-06T08:41:33.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:33.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:33.605830+0000) 2024-03-06T08:41:34.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:34.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:34.606132+0000) 2024-03-06T08:41:35.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:35.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:35.606345+0000) 2024-03-06T08:41:36.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:36.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:36.606643+0000) 2024-03-06T08:41:37.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:37.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:37.606862+0000) 2024-03-06T08:41:38.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:38.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:38.607146+0000) 2024-03-06T08:41:39.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:39.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:39.607341+0000) 2024-03-06T08:41:40.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:40.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:40.607533+0000) 2024-03-06T08:41:41.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:41.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:41.607785+0000) 2024-03-06T08:41:42.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:42.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:42.608118+0000) 2024-03-06T08:41:43.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:43.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:43.608360+0000) 2024-03-06T08:41:44.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:44.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:44.608587+0000) 2024-03-06T08:41:44.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:44.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:41:45.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:45.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:45.608814+0000) 2024-03-06T08:41:46.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:46.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:46.609102+0000) 2024-03-06T08:41:47.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:47.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:47.609322+0000) 2024-03-06T08:41:48.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:48.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:48.609535+0000) 2024-03-06T08:41:49.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:49.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:49.609760+0000) 2024-03-06T08:41:50.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:50.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:50.609980+0000) 2024-03-06T08:41:51.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:51.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:51.610267+0000) 2024-03-06T08:41:52.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:52.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:52.610498+0000) 2024-03-06T08:41:53.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:53.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:53.610730+0000) 2024-03-06T08:41:54.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:54.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:54.610916+0000) 2024-03-06T08:41:55.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:55.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:55.611124+0000) 2024-03-06T08:41:56.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:56.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:56.611358+0000) 2024-03-06T08:41:57.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:57.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:57.611602+0000) 2024-03-06T08:41:58.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:58.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:58.611887+0000) 2024-03-06T08:41:59.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:41:59.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:41:59.612165+0000) 2024-03-06T08:42:00.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:00.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:00.612443+0000) 2024-03-06T08:42:01.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:01.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:01.612700+0000) 2024-03-06T08:42:02.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:02.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:02.612952+0000) 2024-03-06T08:42:03.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:03.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:03.613170+0000) 2024-03-06T08:42:04.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:04.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:04.613391+0000) 2024-03-06T08:42:05.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:05.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:05.613621+0000) 2024-03-06T08:42:06.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:06.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:06.613855+0000) 2024-03-06T08:42:07.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:07.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:07.614134+0000) 2024-03-06T08:42:08.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:08.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:08.614408+0000) 2024-03-06T08:42:09.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:09.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:09.614691+0000) 2024-03-06T08:42:10.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:10.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:10.614966+0000) 2024-03-06T08:42:11.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:11.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:11.615257+0000) 2024-03-06T08:42:12.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:12.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:12.615504+0000) 2024-03-06T08:42:13.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:13.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:13.615771+0000) 2024-03-06T08:42:14.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:14.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:14.616094+0000) 2024-03-06T08:42:14.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:14.969 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:15.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:15.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:15.616377+0000) 2024-03-06T08:42:16.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:16.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:16.616674+0000) 2024-03-06T08:42:17.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:17.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:17.616939+0000) 2024-03-06T08:42:18.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:18.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:18.617165+0000) 2024-03-06T08:42:19.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:19.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:19.617434+0000) 2024-03-06T08:42:20.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:20.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:20.617688+0000) 2024-03-06T08:42:21.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:21.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:21.617939+0000) 2024-03-06T08:42:22.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:22.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:22.618214+0000) 2024-03-06T08:42:23.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:23.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:23.618427+0000) 2024-03-06T08:42:24.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:24.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:24.618780+0000) 2024-03-06T08:42:25.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:25.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:25.619062+0000) 2024-03-06T08:42:26.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:26.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:26.619312+0000) 2024-03-06T08:42:27.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:27.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:27.619513+0000) 2024-03-06T08:42:28.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:28.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:28.619714+0000) 2024-03-06T08:42:29.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:29.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:29.619919+0000) 2024-03-06T08:42:30.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:30.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:30.620187+0000) 2024-03-06T08:42:31.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:31.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:31.620402+0000) 2024-03-06T08:42:32.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:32.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:32.620639+0000) 2024-03-06T08:42:33.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:33.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:33.620918+0000) 2024-03-06T08:42:34.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:34.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:34.621161+0000) 2024-03-06T08:42:35.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:35.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:35.621410+0000) 2024-03-06T08:42:36.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:36.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:36.621639+0000) 2024-03-06T08:42:37.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:37.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:37.621862+0000) 2024-03-06T08:42:38.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:38.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:38.622122+0000) 2024-03-06T08:42:39.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:39.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:39.622292+0000) 2024-03-06T08:42:40.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:40.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:40.622543+0000) 2024-03-06T08:42:41.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:41.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:41.622771+0000) 2024-03-06T08:42:42.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:42.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:42.623050+0000) 2024-03-06T08:42:43.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:43.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:43.623258+0000) 2024-03-06T08:42:44.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:44.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:44.623488+0000) 2024-03-06T08:42:45.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:45.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:42:45.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:45.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:45.623724+0000) 2024-03-06T08:42:46.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:46.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:46.623964+0000) 2024-03-06T08:42:47.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:47.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:47.624213+0000) 2024-03-06T08:42:48.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:48.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:48.624432+0000) 2024-03-06T08:42:49.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:49.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:49.624746+0000) 2024-03-06T08:42:50.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:50.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:50.625049+0000) 2024-03-06T08:42:51.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:51.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:51.625283+0000) 2024-03-06T08:42:52.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:52.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:52.625531+0000) 2024-03-06T08:42:53.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:53.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:53.625762+0000) 2024-03-06T08:42:54.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:54.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:54.625974+0000) 2024-03-06T08:42:55.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:55.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:55.626266+0000) 2024-03-06T08:42:56.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:56.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:56.626524+0000) 2024-03-06T08:42:57.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:57.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:57.626744+0000) 2024-03-06T08:42:58.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:58.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:58.626952+0000) 2024-03-06T08:42:59.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:42:59.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:42:59.627229+0000) 2024-03-06T08:43:00.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:00.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:00.627423+0000) 2024-03-06T08:43:01.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:01.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:01.627642+0000) 2024-03-06T08:43:02.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:02.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:02.627861+0000) 2024-03-06T08:43:03.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:03.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:03.628116+0000) 2024-03-06T08:43:04.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:04.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:04.628320+0000) 2024-03-06T08:43:05.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:05.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:05.628574+0000) 2024-03-06T08:43:06.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:06.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:06.628845+0000) 2024-03-06T08:43:07.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:07.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:07.629140+0000) 2024-03-06T08:43:08.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:08.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:08.629411+0000) 2024-03-06T08:43:09.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:09.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:09.629633+0000) 2024-03-06T08:43:10.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:10.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:10.629851+0000) 2024-03-06T08:43:11.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:11.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:11.630144+0000) 2024-03-06T08:43:12.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:12.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:12.630343+0000) 2024-03-06T08:43:13.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:13.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:13.630520+0000) 2024-03-06T08:43:14.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:14.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:14.630736+0000) 2024-03-06T08:43:15.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:15.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:15.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:15.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:15.630962+0000) 2024-03-06T08:43:16.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:16.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:16.631202+0000) 2024-03-06T08:43:17.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:17.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:17.631433+0000) 2024-03-06T08:43:18.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:18.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:18.631666+0000) 2024-03-06T08:43:19.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:19.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:19.631939+0000) 2024-03-06T08:43:20.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:20.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:20.632161+0000) 2024-03-06T08:43:21.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:21.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:21.632399+0000) 2024-03-06T08:43:22.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:22.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:22.632632+0000) 2024-03-06T08:43:23.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:23.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:23.632854+0000) 2024-03-06T08:43:24.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:24.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:24.633110+0000) 2024-03-06T08:43:25.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:25.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:25.633340+0000) 2024-03-06T08:43:26.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:26.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:26.633565+0000) 2024-03-06T08:43:27.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:27.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:27.633783+0000) 2024-03-06T08:43:28.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:28.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:28.634042+0000) 2024-03-06T08:43:29.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:29.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:29.634321+0000) 2024-03-06T08:43:30.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:30.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:30.634560+0000) 2024-03-06T08:43:31.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:31.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:31.634780+0000) 2024-03-06T08:43:32.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:32.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:32.635036+0000) 2024-03-06T08:43:33.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:33.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:33.635255+0000) 2024-03-06T08:43:34.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:34.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:34.635512+0000) 2024-03-06T08:43:35.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:35.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:35.635726+0000) 2024-03-06T08:43:36.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:36.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:36.635959+0000) 2024-03-06T08:43:37.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:37.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:37.636277+0000) 2024-03-06T08:43:38.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:38.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:38.636561+0000) 2024-03-06T08:43:39.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:39.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:39.636824+0000) 2024-03-06T08:43:40.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:40.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:40.637152+0000) 2024-03-06T08:43:41.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:41.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:41.637382+0000) 2024-03-06T08:43:42.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:42.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:42.637599+0000) 2024-03-06T08:43:43.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:43.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:43.637836+0000) 2024-03-06T08:43:44.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:44.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:44.638147+0000) 2024-03-06T08:43:45.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:45.463 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:43:45.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:45.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:45.638355+0000) 2024-03-06T08:43:46.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:46.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:46.638580+0000) 2024-03-06T08:43:47.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:47.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:47.638802+0000) 2024-03-06T08:43:48.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:48.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:48.638988+0000) 2024-03-06T08:43:49.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:49.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:49.639303+0000) 2024-03-06T08:43:50.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:50.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:50.639548+0000) 2024-03-06T08:43:51.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:51.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:51.639789+0000) 2024-03-06T08:43:52.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:52.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:52.640092+0000) 2024-03-06T08:43:53.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:53.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:53.640309+0000) 2024-03-06T08:43:54.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:54.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:54.640551+0000) 2024-03-06T08:43:55.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:55.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:55.640792+0000) 2024-03-06T08:43:56.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:56.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:56.641081+0000) 2024-03-06T08:43:57.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:57.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:57.641323+0000) 2024-03-06T08:43:58.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:58.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:58.641606+0000) 2024-03-06T08:43:59.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:43:59.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:43:59.641902+0000) 2024-03-06T08:44:00.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:00.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:00.642127+0000) 2024-03-06T08:44:01.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:01.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:01.642354+0000) 2024-03-06T08:44:02.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:02.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:02.642558+0000) 2024-03-06T08:44:03.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:03.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:03.642765+0000) 2024-03-06T08:44:04.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:04.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:04.642965+0000) 2024-03-06T08:44:05.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:05.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:05.643158+0000) 2024-03-06T08:44:06.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:06.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:06.643390+0000) 2024-03-06T08:44:07.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:07.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:07.643607+0000) 2024-03-06T08:44:08.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:08.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:08.643852+0000) 2024-03-06T08:44:09.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:09.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:09.644134+0000) 2024-03-06T08:44:10.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:10.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:10.644385+0000) 2024-03-06T08:44:11.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:11.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:11.644677+0000) 2024-03-06T08:44:12.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:12.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:12.644900+0000) 2024-03-06T08:44:13.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:13.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:13.645144+0000) 2024-03-06T08:44:14.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:14.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:14.645385+0000) 2024-03-06T08:44:15.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:15.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:15.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:15.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:15.645621+0000) 2024-03-06T08:44:16.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:16.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:16.645855+0000) 2024-03-06T08:44:17.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:17.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:17.646098+0000) 2024-03-06T08:44:18.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:18.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:18.646371+0000) 2024-03-06T08:44:19.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:19.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:19.646623+0000) 2024-03-06T08:44:20.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:20.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:20.646869+0000) 2024-03-06T08:44:21.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:21.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:21.647130+0000) 2024-03-06T08:44:22.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:22.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:22.647371+0000) 2024-03-06T08:44:23.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:23.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:23.647627+0000) 2024-03-06T08:44:24.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:24.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:24.647884+0000) 2024-03-06T08:44:25.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:25.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:25.648155+0000) 2024-03-06T08:44:26.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:26.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:26.648401+0000) 2024-03-06T08:44:27.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:27.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:27.648665+0000) 2024-03-06T08:44:28.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:28.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:28.648938+0000) 2024-03-06T08:44:29.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:29.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:29.649235+0000) 2024-03-06T08:44:30.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:30.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:30.649469+0000) 2024-03-06T08:44:31.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:31.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:31.649698+0000) 2024-03-06T08:44:32.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:32.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:32.649952+0000) 2024-03-06T08:44:33.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:33.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:33.650152+0000) 2024-03-06T08:44:34.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:34.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:34.650385+0000) 2024-03-06T08:44:35.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:35.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:35.650625+0000) 2024-03-06T08:44:36.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:36.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:36.650850+0000) 2024-03-06T08:44:37.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:37.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:37.651122+0000) 2024-03-06T08:44:38.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:38.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:38.651313+0000) 2024-03-06T08:44:39.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:39.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:39.651548+0000) 2024-03-06T08:44:40.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:40.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:40.651800+0000) 2024-03-06T08:44:41.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:41.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:41.652069+0000) 2024-03-06T08:44:42.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:42.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:42.652327+0000) 2024-03-06T08:44:43.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:43.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:43.652579+0000) 2024-03-06T08:44:44.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:44.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:44.652741+0000) 2024-03-06T08:44:45.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:45.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:45.652928+0000) 2024-03-06T08:44:45.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:45.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:44:46.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:46.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:46.653171+0000) 2024-03-06T08:44:47.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:47.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:47.653370+0000) 2024-03-06T08:44:48.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:48.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:48.653599+0000) 2024-03-06T08:44:49.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:49.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:49.653849+0000) 2024-03-06T08:44:50.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:50.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:50.654130+0000) 2024-03-06T08:44:51.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:51.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:51.654283+0000) 2024-03-06T08:44:52.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:52.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:52.654533+0000) 2024-03-06T08:44:53.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:53.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:53.654747+0000) 2024-03-06T08:44:54.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:54.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:54.654960+0000) 2024-03-06T08:44:55.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:55.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:55.655140+0000) 2024-03-06T08:44:56.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:56.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:56.655334+0000) 2024-03-06T08:44:57.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:57.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:57.655580+0000) 2024-03-06T08:44:58.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:58.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:58.655813+0000) 2024-03-06T08:44:59.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:44:59.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:44:59.656127+0000) 2024-03-06T08:45:00.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:00.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:00.656354+0000) 2024-03-06T08:45:01.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:01.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:01.656581+0000) 2024-03-06T08:45:02.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:02.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:02.656825+0000) 2024-03-06T08:45:03.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:03.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:03.657077+0000) 2024-03-06T08:45:04.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:04.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:04.657289+0000) 2024-03-06T08:45:05.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:05.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:05.657505+0000) 2024-03-06T08:45:06.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:06.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:06.657765+0000) 2024-03-06T08:45:07.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:07.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:07.658052+0000) 2024-03-06T08:45:08.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:08.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:08.658263+0000) 2024-03-06T08:45:09.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:09.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:09.658495+0000) 2024-03-06T08:45:10.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:10.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:10.658742+0000) 2024-03-06T08:45:11.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:11.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:11.658967+0000) 2024-03-06T08:45:12.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:12.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:12.659225+0000) 2024-03-06T08:45:13.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:13.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:13.659434+0000) 2024-03-06T08:45:14.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:14.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:14.659653+0000) 2024-03-06T08:45:15.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:15.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:15.659923+0000) 2024-03-06T08:45:15.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:15.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:16.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:16.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:16.660210+0000) 2024-03-06T08:45:17.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:17.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:17.660454+0000) 2024-03-06T08:45:18.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:18.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:18.660694+0000) 2024-03-06T08:45:19.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:19.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:19.660946+0000) 2024-03-06T08:45:20.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:20.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:20.661199+0000) 2024-03-06T08:45:21.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:21.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:21.661417+0000) 2024-03-06T08:45:22.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:22.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:22.661653+0000) 2024-03-06T08:45:23.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:23.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:23.661850+0000) 2024-03-06T08:45:24.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:24.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:24.662075+0000) 2024-03-06T08:45:25.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:25.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:25.662214+0000) 2024-03-06T08:45:26.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:26.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:26.662323+0000) 2024-03-06T08:45:27.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:27.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:27.662530+0000) 2024-03-06T08:45:28.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:28.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:28.662752+0000) 2024-03-06T08:45:29.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:29.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:29.662973+0000) 2024-03-06T08:45:30.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:30.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:30.663205+0000) 2024-03-06T08:45:31.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:31.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:31.663421+0000) 2024-03-06T08:45:32.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:32.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:32.663680+0000) 2024-03-06T08:45:33.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:33.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:33.663866+0000) 2024-03-06T08:45:34.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:34.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:34.664149+0000) 2024-03-06T08:45:35.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:35.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:35.664411+0000) 2024-03-06T08:45:36.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:36.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:36.664659+0000) 2024-03-06T08:45:37.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:37.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:37.664861+0000) 2024-03-06T08:45:38.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:38.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:38.665186+0000) 2024-03-06T08:45:39.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:39.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:39.665478+0000) 2024-03-06T08:45:40.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:40.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:40.665726+0000) 2024-03-06T08:45:41.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:41.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:41.665981+0000) 2024-03-06T08:45:42.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:42.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:42.666263+0000) 2024-03-06T08:45:43.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:43.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:43.666520+0000) 2024-03-06T08:45:44.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:44.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:44.666786+0000) 2024-03-06T08:45:45.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:45.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:45.666984+0000) 2024-03-06T08:45:46.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:46.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:45:46.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:46.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:46.667213+0000) 2024-03-06T08:45:47.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:47.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:47.667430+0000) 2024-03-06T08:45:48.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:48.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:48.667584+0000) 2024-03-06T08:45:49.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:49.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:49.667792+0000) 2024-03-06T08:45:50.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:50.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:50.668051+0000) 2024-03-06T08:45:51.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:51.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:51.668249+0000) 2024-03-06T08:45:52.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:52.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:52.668576+0000) 2024-03-06T08:45:53.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:53.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:53.668854+0000) 2024-03-06T08:45:54.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:54.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:54.669168+0000) 2024-03-06T08:45:55.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:55.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:55.669496+0000) 2024-03-06T08:45:56.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:56.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:56.669745+0000) 2024-03-06T08:45:57.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:57.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:57.669925+0000) 2024-03-06T08:45:58.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:58.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:58.670148+0000) 2024-03-06T08:45:59.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:45:59.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:45:59.670346+0000) 2024-03-06T08:46:00.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:00.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:00.670499+0000) 2024-03-06T08:46:01.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:01.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:01.670719+0000) 2024-03-06T08:46:02.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:02.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:02.670956+0000) 2024-03-06T08:46:03.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:03.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:03.671160+0000) 2024-03-06T08:46:04.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:04.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:04.671361+0000) 2024-03-06T08:46:05.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:05.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:05.671569+0000) 2024-03-06T08:46:06.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:06.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:06.671795+0000) 2024-03-06T08:46:07.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:07.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:07.672057+0000) 2024-03-06T08:46:08.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:08.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:08.672256+0000) 2024-03-06T08:46:09.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:09.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:09.672475+0000) 2024-03-06T08:46:10.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:10.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:10.672702+0000) 2024-03-06T08:46:11.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:11.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:11.672939+0000) 2024-03-06T08:46:12.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:12.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:12.673185+0000) 2024-03-06T08:46:13.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:13.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:13.673432+0000) 2024-03-06T08:46:14.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:14.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:14.673663+0000) 2024-03-06T08:46:15.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:15.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:15.673882+0000) 2024-03-06T08:46:16.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:16.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:16.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:16.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:16.674132+0000) 2024-03-06T08:46:17.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:17.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:17.674323+0000) 2024-03-06T08:46:18.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:18.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:18.674528+0000) 2024-03-06T08:46:19.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:19.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:19.674763+0000) 2024-03-06T08:46:20.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:20.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:20.674987+0000) 2024-03-06T08:46:21.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:21.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:21.675262+0000) 2024-03-06T08:46:22.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:22.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:22.675511+0000) 2024-03-06T08:46:23.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:23.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:23.675927+0000) 2024-03-06T08:46:24.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:24.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:24.676155+0000) 2024-03-06T08:46:25.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:25.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:25.676342+0000) 2024-03-06T08:46:26.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:26.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:26.676556+0000) 2024-03-06T08:46:27.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:27.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:27.676775+0000) 2024-03-06T08:46:28.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:28.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:28.677043+0000) 2024-03-06T08:46:29.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:29.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:29.677248+0000) 2024-03-06T08:46:30.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:30.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:30.677455+0000) 2024-03-06T08:46:31.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:31.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:31.677674+0000) 2024-03-06T08:46:32.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:32.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:32.677913+0000) 2024-03-06T08:46:33.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:33.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:33.678158+0000) 2024-03-06T08:46:34.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:34.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:34.678383+0000) 2024-03-06T08:46:35.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:35.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:35.678592+0000) 2024-03-06T08:46:36.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:36.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:36.678788+0000) 2024-03-06T08:46:37.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:37.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:37.679068+0000) 2024-03-06T08:46:38.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:38.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:38.679323+0000) 2024-03-06T08:46:39.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:39.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:39.679532+0000) 2024-03-06T08:46:40.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:40.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:40.679788+0000) 2024-03-06T08:46:41.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:41.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:41.680031+0000) 2024-03-06T08:46:42.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:42.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:42.680286+0000) 2024-03-06T08:46:43.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:43.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:43.680490+0000) 2024-03-06T08:46:44.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:44.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:44.680712+0000) 2024-03-06T08:46:45.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:45.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:45.680936+0000) 2024-03-06T08:46:46.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:46.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:46:46.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:46.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:46.681200+0000) 2024-03-06T08:46:47.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:47.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:47.681487+0000) 2024-03-06T08:46:48.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:48.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:48.681723+0000) 2024-03-06T08:46:49.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:49.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:49.681976+0000) 2024-03-06T08:46:50.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:50.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:50.682259+0000) 2024-03-06T08:46:51.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:51.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:51.682471+0000) 2024-03-06T08:46:52.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:52.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:52.682723+0000) 2024-03-06T08:46:53.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:53.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:53.682945+0000) 2024-03-06T08:46:54.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:54.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:54.683171+0000) 2024-03-06T08:46:55.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:55.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:55.683377+0000) 2024-03-06T08:46:56.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:56.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:56.683605+0000) 2024-03-06T08:46:57.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:57.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:57.683831+0000) 2024-03-06T08:46:58.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:58.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:58.684107+0000) 2024-03-06T08:46:59.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:46:59.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:46:59.684341+0000) 2024-03-06T08:47:00.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:00.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:00.684548+0000) 2024-03-06T08:47:01.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:01.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:01.684734+0000) 2024-03-06T08:47:02.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:02.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:02.684973+0000) 2024-03-06T08:47:03.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:03.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:03.685154+0000) 2024-03-06T08:47:04.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:04.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:04.685343+0000) 2024-03-06T08:47:05.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:05.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:05.685540+0000) 2024-03-06T08:47:06.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:06.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:06.685771+0000) 2024-03-06T08:47:07.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:07.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:07.685989+0000) 2024-03-06T08:47:08.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:08.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:08.686252+0000) 2024-03-06T08:47:09.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:09.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:09.686464+0000) 2024-03-06T08:47:10.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:10.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:10.686657+0000) 2024-03-06T08:47:11.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:11.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:11.686880+0000) 2024-03-06T08:47:12.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:12.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:12.687132+0000) 2024-03-06T08:47:13.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:13.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:13.687318+0000) 2024-03-06T08:47:14.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:14.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:14.687543+0000) 2024-03-06T08:47:15.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:15.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:15.687788+0000) 2024-03-06T08:47:16.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:16.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:16.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:16.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:16.688071+0000) 2024-03-06T08:47:17.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:17.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:17.688274+0000) 2024-03-06T08:47:18.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:18.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:18.688499+0000) 2024-03-06T08:47:19.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:19.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:19.688742+0000) 2024-03-06T08:47:20.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:20.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:20.688981+0000) 2024-03-06T08:47:21.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:21.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:21.689240+0000) 2024-03-06T08:47:22.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:22.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:22.689533+0000) 2024-03-06T08:47:23.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:23.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:23.689739+0000) 2024-03-06T08:47:24.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:24.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:24.689956+0000) 2024-03-06T08:47:25.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:25.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:25.690161+0000) 2024-03-06T08:47:26.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:26.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:26.690377+0000) 2024-03-06T08:47:27.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:27.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:27.690616+0000) 2024-03-06T08:47:28.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:28.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:28.690846+0000) 2024-03-06T08:47:29.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:29.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:29.691085+0000) 2024-03-06T08:47:30.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:30.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:30.691271+0000) 2024-03-06T08:47:31.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:31.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:31.691493+0000) 2024-03-06T08:47:32.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:32.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:32.691730+0000) 2024-03-06T08:47:33.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:33.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:33.691944+0000) 2024-03-06T08:47:34.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:34.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:34.692161+0000) 2024-03-06T08:47:35.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:35.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:35.692379+0000) 2024-03-06T08:47:36.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:36.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:36.692564+0000) 2024-03-06T08:47:37.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:37.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:37.692800+0000) 2024-03-06T08:47:38.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:38.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:38.692987+0000) 2024-03-06T08:47:39.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:39.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:39.693260+0000) 2024-03-06T08:47:40.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:40.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:40.693477+0000) 2024-03-06T08:47:41.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:41.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:41.693680+0000) 2024-03-06T08:47:42.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:42.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:42.693913+0000) 2024-03-06T08:47:43.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:43.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:43.694147+0000) 2024-03-06T08:47:44.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:44.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:44.694336+0000) 2024-03-06T08:47:45.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:45.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:45.694544+0000) 2024-03-06T08:47:46.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:46.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:47:46.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:46.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:46.694768+0000) 2024-03-06T08:47:47.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:47.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:47.695089+0000) 2024-03-06T08:47:48.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:48.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:48.695364+0000) 2024-03-06T08:47:49.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:49.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:49.695651+0000) 2024-03-06T08:47:50.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:50.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:50.695888+0000) 2024-03-06T08:47:51.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:51.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:51.696153+0000) 2024-03-06T08:47:52.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:52.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:52.696416+0000) 2024-03-06T08:47:53.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:53.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:53.696640+0000) 2024-03-06T08:47:54.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:54.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:54.696864+0000) 2024-03-06T08:47:55.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:55.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:55.697149+0000) 2024-03-06T08:47:56.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:56.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:56.697387+0000) 2024-03-06T08:47:57.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:57.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:57.697674+0000) 2024-03-06T08:47:58.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:58.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:58.697993+0000) 2024-03-06T08:47:59.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:47:59.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:47:59.698372+0000) 2024-03-06T08:48:00.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:00.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:00.698650+0000) 2024-03-06T08:48:01.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:01.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:01.698936+0000) 2024-03-06T08:48:02.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:02.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:02.699234+0000) 2024-03-06T08:48:03.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:03.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:03.699503+0000) 2024-03-06T08:48:04.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:04.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:04.699741+0000) 2024-03-06T08:48:05.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:05.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:05.699885+0000) 2024-03-06T08:48:06.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:06.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:06.700139+0000) 2024-03-06T08:48:07.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:07.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:07.700365+0000) 2024-03-06T08:48:08.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:08.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:08.700635+0000) 2024-03-06T08:48:09.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:09.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:09.700867+0000) 2024-03-06T08:48:10.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:10.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:10.701108+0000) 2024-03-06T08:48:11.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:11.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:11.701320+0000) 2024-03-06T08:48:12.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:12.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:12.701528+0000) 2024-03-06T08:48:13.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:13.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:13.701748+0000) 2024-03-06T08:48:14.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:14.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:14.701977+0000) 2024-03-06T08:48:15.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:15.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:15.702246+0000) 2024-03-06T08:48:16.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:16.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:16.702465+0000) 2024-03-06T08:48:16.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:16.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:17.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:17.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:17.702683+0000) 2024-03-06T08:48:18.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:18.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:18.702887+0000) 2024-03-06T08:48:19.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:19.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:19.703126+0000) 2024-03-06T08:48:20.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:20.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:20.703348+0000) 2024-03-06T08:48:21.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:21.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:21.703569+0000) 2024-03-06T08:48:22.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:22.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:22.703831+0000) 2024-03-06T08:48:23.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:23.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:23.704139+0000) 2024-03-06T08:48:24.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:24.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:24.704386+0000) 2024-03-06T08:48:25.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:25.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:25.704648+0000) 2024-03-06T08:48:26.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:26.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:26.704930+0000) 2024-03-06T08:48:27.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:27.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:27.705247+0000) 2024-03-06T08:48:28.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:28.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:28.705502+0000) 2024-03-06T08:48:29.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:29.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:29.705752+0000) 2024-03-06T08:48:30.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:30.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:30.705974+0000) 2024-03-06T08:48:31.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:31.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:31.706141+0000) 2024-03-06T08:48:32.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:32.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:32.706367+0000) 2024-03-06T08:48:33.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:33.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:33.706588+0000) 2024-03-06T08:48:34.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:34.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:34.706843+0000) 2024-03-06T08:48:35.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:35.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:35.707150+0000) 2024-03-06T08:48:36.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:36.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:36.707396+0000) 2024-03-06T08:48:37.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:37.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:37.707657+0000) 2024-03-06T08:48:38.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:38.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:38.707903+0000) 2024-03-06T08:48:39.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:39.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:39.708193+0000) 2024-03-06T08:48:40.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:40.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:40.708440+0000) 2024-03-06T08:48:41.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:41.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:41.708699+0000) 2024-03-06T08:48:42.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:42.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:42.708980+0000) 2024-03-06T08:48:43.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:43.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:43.709264+0000) 2024-03-06T08:48:44.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:44.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:44.709469+0000) 2024-03-06T08:48:45.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:45.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:45.709687+0000) 2024-03-06T08:48:46.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:46.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:46.709958+0000) 2024-03-06T08:48:46.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:46.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:48:47.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:47.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:47.710228+0000) 2024-03-06T08:48:48.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:48.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:48.710436+0000) 2024-03-06T08:48:49.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:49.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:49.710663+0000) 2024-03-06T08:48:50.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:50.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:50.710887+0000) 2024-03-06T08:48:51.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:51.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:51.711118+0000) 2024-03-06T08:48:52.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:52.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:52.711340+0000) 2024-03-06T08:48:53.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:53.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:53.711600+0000) 2024-03-06T08:48:54.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:54.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:54.711822+0000) 2024-03-06T08:48:55.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:55.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:55.712127+0000) 2024-03-06T08:48:56.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:56.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:56.712365+0000) 2024-03-06T08:48:57.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:57.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:57.712539+0000) 2024-03-06T08:48:58.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:58.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:58.712769+0000) 2024-03-06T08:48:59.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:48:59.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:48:59.712992+0000) 2024-03-06T08:49:00.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:00.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:00.713208+0000) 2024-03-06T08:49:01.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:01.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:01.713399+0000) 2024-03-06T08:49:02.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:02.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:02.713573+0000) 2024-03-06T08:49:03.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:03.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:03.713729+0000) 2024-03-06T08:49:04.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:04.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:04.713858+0000) 2024-03-06T08:49:05.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:05.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:05.714112+0000) 2024-03-06T08:49:06.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:06.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:06.714265+0000) 2024-03-06T08:49:07.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:07.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:07.714471+0000) 2024-03-06T08:49:08.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:08.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:08.714695+0000) 2024-03-06T08:49:09.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:09.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:09.714959+0000) 2024-03-06T08:49:10.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:10.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:10.715128+0000) 2024-03-06T08:49:11.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:11.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:11.715293+0000) 2024-03-06T08:49:12.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:12.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:12.715500+0000) 2024-03-06T08:49:13.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:13.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:13.715727+0000) 2024-03-06T08:49:14.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:14.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:14.715856+0000) 2024-03-06T08:49:15.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:15.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:15.716153+0000) 2024-03-06T08:49:16.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:16.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:16.716382+0000) 2024-03-06T08:49:17.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:17.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:17.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:17.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:17.716608+0000) 2024-03-06T08:49:18.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:18.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:18.716831+0000) 2024-03-06T08:49:19.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:19.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:19.717105+0000) 2024-03-06T08:49:20.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:20.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:20.717324+0000) 2024-03-06T08:49:21.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:21.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:21.717502+0000) 2024-03-06T08:49:22.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:22.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:22.717749+0000) 2024-03-06T08:49:23.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:23.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:23.717987+0000) 2024-03-06T08:49:24.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:24.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:24.718208+0000) 2024-03-06T08:49:25.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:25.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:25.718440+0000) 2024-03-06T08:49:26.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:26.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:26.718663+0000) 2024-03-06T08:49:27.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:27.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:27.718855+0000) 2024-03-06T08:49:28.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:28.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:28.719117+0000) 2024-03-06T08:49:29.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:29.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:29.719353+0000) 2024-03-06T08:49:30.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:30.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:30.719578+0000) 2024-03-06T08:49:31.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:31.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:31.719805+0000) 2024-03-06T08:49:32.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:32.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:32.719989+0000) 2024-03-06T08:49:33.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:33.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:33.720247+0000) 2024-03-06T08:49:34.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:34.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:34.720455+0000) 2024-03-06T08:49:35.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:35.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:35.720679+0000) 2024-03-06T08:49:36.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:36.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:36.720899+0000) 2024-03-06T08:49:37.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:37.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:37.721143+0000) 2024-03-06T08:49:38.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:38.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:38.721353+0000) 2024-03-06T08:49:39.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:39.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:39.721574+0000) 2024-03-06T08:49:40.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:40.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:40.721798+0000) 2024-03-06T08:49:41.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:41.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:41.722045+0000) 2024-03-06T08:49:42.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:42.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:42.722242+0000) 2024-03-06T08:49:43.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:43.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:43.722465+0000) 2024-03-06T08:49:44.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:44.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:44.722657+0000) 2024-03-06T08:49:45.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:45.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:45.722874+0000) 2024-03-06T08:49:46.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:46.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:46.723138+0000) 2024-03-06T08:49:47.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:47.223 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:49:47.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:47.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:47.723353+0000) 2024-03-06T08:49:48.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:48.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:48.723570+0000) 2024-03-06T08:49:49.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:49.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:49.723802+0000) 2024-03-06T08:49:50.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:50.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:50.724038+0000) 2024-03-06T08:49:51.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:51.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:51.724261+0000) 2024-03-06T08:49:52.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:52.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:52.724479+0000) 2024-03-06T08:49:53.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:53.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:53.724700+0000) 2024-03-06T08:49:54.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:54.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:54.724928+0000) 2024-03-06T08:49:55.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:55.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:55.725145+0000) 2024-03-06T08:49:56.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:56.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:56.725333+0000) 2024-03-06T08:49:57.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:57.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:57.725540+0000) 2024-03-06T08:49:58.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:58.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:58.725767+0000) 2024-03-06T08:49:59.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:49:59.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:49:59.725985+0000) 2024-03-06T08:50:00.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:00.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:00.726206+0000) 2024-03-06T08:50:01.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:01.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:01.726394+0000) 2024-03-06T08:50:02.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:02.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:02.726596+0000) 2024-03-06T08:50:03.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:03.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:03.726814+0000) 2024-03-06T08:50:04.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:04.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:04.727072+0000) 2024-03-06T08:50:05.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:05.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:05.727271+0000) 2024-03-06T08:50:06.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:06.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:06.727536+0000) 2024-03-06T08:50:07.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:07.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:07.727778+0000) 2024-03-06T08:50:08.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:08.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:08.728160+0000) 2024-03-06T08:50:09.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:09.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:09.728440+0000) 2024-03-06T08:50:10.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:10.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:10.728719+0000) 2024-03-06T08:50:11.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:11.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:11.728928+0000) 2024-03-06T08:50:12.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:12.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:12.729180+0000) 2024-03-06T08:50:13.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:13.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:13.729427+0000) 2024-03-06T08:50:14.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:14.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:14.729600+0000) 2024-03-06T08:50:15.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:15.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:15.729784+0000) 2024-03-06T08:50:16.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:16.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:16.730098+0000) 2024-03-06T08:50:17.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:17.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:17.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:17.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:17.730377+0000) 2024-03-06T08:50:18.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:18.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:18.730653+0000) 2024-03-06T08:50:19.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:19.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:19.730932+0000) 2024-03-06T08:50:20.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:20.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:20.731158+0000) 2024-03-06T08:50:21.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:21.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:21.731324+0000) 2024-03-06T08:50:22.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:22.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:22.731521+0000) 2024-03-06T08:50:23.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:23.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:23.731729+0000) 2024-03-06T08:50:24.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:24.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:24.731953+0000) 2024-03-06T08:50:25.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:25.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:25.732183+0000) 2024-03-06T08:50:26.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:26.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:26.732381+0000) 2024-03-06T08:50:27.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:27.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:27.732585+0000) 2024-03-06T08:50:28.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:28.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:28.732826+0000) 2024-03-06T08:50:29.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:29.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:29.733120+0000) 2024-03-06T08:50:30.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:30.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:30.733329+0000) 2024-03-06T08:50:31.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:31.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:31.733541+0000) 2024-03-06T08:50:32.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:32.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:32.733740+0000) 2024-03-06T08:50:33.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:33.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:33.733963+0000) 2024-03-06T08:50:34.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:34.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:34.734155+0000) 2024-03-06T08:50:35.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:35.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:35.734371+0000) 2024-03-06T08:50:36.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:36.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:36.734588+0000) 2024-03-06T08:50:37.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:37.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:37.734806+0000) 2024-03-06T08:50:38.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:38.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:38.735072+0000) 2024-03-06T08:50:39.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:39.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:39.735286+0000) 2024-03-06T08:50:40.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:40.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:40.735475+0000) 2024-03-06T08:50:41.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:41.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:41.735695+0000) 2024-03-06T08:50:42.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:42.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:42.735917+0000) 2024-03-06T08:50:43.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:43.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:43.736151+0000) 2024-03-06T08:50:44.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:44.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:44.736339+0000) 2024-03-06T08:50:45.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:45.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:45.736568+0000) 2024-03-06T08:50:46.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:46.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:46.736783+0000) 2024-03-06T08:50:47.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:47.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:50:47.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:47.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:47.737051+0000) 2024-03-06T08:50:48.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:48.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:48.737307+0000) 2024-03-06T08:50:49.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:49.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:49.737537+0000) 2024-03-06T08:50:50.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:50.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:50.737774+0000) 2024-03-06T08:50:51.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:51.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:51.737990+0000) 2024-03-06T08:50:52.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:52.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:52.738310+0000) 2024-03-06T08:50:53.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:53.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:53.738542+0000) 2024-03-06T08:50:54.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:54.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:54.738770+0000) 2024-03-06T08:50:55.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:55.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:55.738990+0000) 2024-03-06T08:50:56.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:56.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:56.739553+0000) 2024-03-06T08:50:57.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:57.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:57.739777+0000) 2024-03-06T08:50:58.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:58.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:58.740049+0000) 2024-03-06T08:50:59.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:50:59.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:50:59.740288+0000) 2024-03-06T08:51:00.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:00.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:00.740519+0000) 2024-03-06T08:51:01.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:01.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:01.740702+0000) 2024-03-06T08:51:02.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:02.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:02.740885+0000) 2024-03-06T08:51:03.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:03.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:03.741117+0000) 2024-03-06T08:51:04.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:04.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:04.741344+0000) 2024-03-06T08:51:05.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:05.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:05.741565+0000) 2024-03-06T08:51:06.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:06.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:06.741793+0000) 2024-03-06T08:51:07.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:07.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:07.741979+0000) 2024-03-06T08:51:08.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:08.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:08.742257+0000) 2024-03-06T08:51:09.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:09.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:09.742497+0000) 2024-03-06T08:51:10.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:10.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:10.742733+0000) 2024-03-06T08:51:11.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:11.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:11.742909+0000) 2024-03-06T08:51:12.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:12.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:12.743132+0000) 2024-03-06T08:51:13.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:13.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:13.743329+0000) 2024-03-06T08:51:14.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:14.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:14.743565+0000) 2024-03-06T08:51:15.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:15.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:15.743801+0000) 2024-03-06T08:51:16.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:16.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:16.744072+0000) 2024-03-06T08:51:17.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:17.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:17.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:17.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:17.744310+0000) 2024-03-06T08:51:18.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:18.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:18.744528+0000) 2024-03-06T08:51:19.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:19.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:19.744758+0000) 2024-03-06T08:51:20.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:20.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:20.745034+0000) 2024-03-06T08:51:21.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:21.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:21.745225+0000) 2024-03-06T08:51:22.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:22.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:22.745424+0000) 2024-03-06T08:51:23.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:23.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:23.745648+0000) 2024-03-06T08:51:24.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:24.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:24.745885+0000) 2024-03-06T08:51:25.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:25.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:25.746117+0000) 2024-03-06T08:51:26.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:26.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:26.746338+0000) 2024-03-06T08:51:27.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:27.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:27.746572+0000) 2024-03-06T08:51:28.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:28.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:28.746794+0000) 2024-03-06T08:51:29.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:29.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:29.747062+0000) 2024-03-06T08:51:30.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:30.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:30.747265+0000) 2024-03-06T08:51:31.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:31.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:31.747480+0000) 2024-03-06T08:51:32.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:32.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:32.747716+0000) 2024-03-06T08:51:33.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:33.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:33.747904+0000) 2024-03-06T08:51:34.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:34.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:34.748062+0000) 2024-03-06T08:51:35.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:35.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:35.748198+0000) 2024-03-06T08:51:36.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:36.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:36.748389+0000) 2024-03-06T08:51:37.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:37.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:37.748597+0000) 2024-03-06T08:51:38.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:38.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:38.748798+0000) 2024-03-06T08:51:39.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:39.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:39.749065+0000) 2024-03-06T08:51:40.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:40.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:40.749260+0000) 2024-03-06T08:51:41.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:41.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:41.749465+0000) 2024-03-06T08:51:42.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:42.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:42.749657+0000) 2024-03-06T08:51:43.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:43.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:43.749815+0000) 2024-03-06T08:51:44.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:44.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:44.750104+0000) 2024-03-06T08:51:45.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:45.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:45.750332+0000) 2024-03-06T08:51:46.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:46.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:46.750555+0000) 2024-03-06T08:51:47.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:47.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:47.750773+0000) 2024-03-06T08:51:47.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:47.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:51:48.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:48.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:48.751050+0000) 2024-03-06T08:51:49.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:49.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:49.751287+0000) 2024-03-06T08:51:50.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:50.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:50.751512+0000) 2024-03-06T08:51:51.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:51.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:51.751690+0000) 2024-03-06T08:51:52.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:52.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:52.751905+0000) 2024-03-06T08:51:53.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:53.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:53.752135+0000) 2024-03-06T08:51:54.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:54.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:54.752362+0000) 2024-03-06T08:51:55.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:55.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:55.752595+0000) 2024-03-06T08:51:56.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:56.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:56.752836+0000) 2024-03-06T08:51:57.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:57.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:57.753104+0000) 2024-03-06T08:51:58.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:58.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:58.753374+0000) 2024-03-06T08:51:59.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:51:59.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:51:59.753624+0000) 2024-03-06T08:52:00.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:00.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:00.753890+0000) 2024-03-06T08:52:01.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:01.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:01.754168+0000) 2024-03-06T08:52:02.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:02.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:02.754417+0000) 2024-03-06T08:52:03.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:03.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:03.754649+0000) 2024-03-06T08:52:04.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:04.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:04.754828+0000) 2024-03-06T08:52:05.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:05.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:05.755079+0000) 2024-03-06T08:52:06.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:06.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:06.755307+0000) 2024-03-06T08:52:07.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:07.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:07.755510+0000) 2024-03-06T08:52:08.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:08.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:08.755755+0000) 2024-03-06T08:52:09.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:09.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:09.755986+0000) 2024-03-06T08:52:10.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:10.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:10.756252+0000) 2024-03-06T08:52:11.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:11.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:11.756461+0000) 2024-03-06T08:52:12.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:12.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:12.756695+0000) 2024-03-06T08:52:13.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:13.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:13.756935+0000) 2024-03-06T08:52:14.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:14.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:14.757154+0000) 2024-03-06T08:52:15.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:15.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:15.757384+0000) 2024-03-06T08:52:16.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:16.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:16.757647+0000) 2024-03-06T08:52:17.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:17.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:17.757922+0000) 2024-03-06T08:52:17.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:17.979 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:18.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:18.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:18.758210+0000) 2024-03-06T08:52:19.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:19.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:19.758434+0000) 2024-03-06T08:52:20.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:20.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:20.758613+0000) 2024-03-06T08:52:21.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:21.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:21.758842+0000) 2024-03-06T08:52:22.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:22.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:22.759126+0000) 2024-03-06T08:52:23.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:23.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:23.759318+0000) 2024-03-06T08:52:24.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:24.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:24.759544+0000) 2024-03-06T08:52:25.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:25.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:25.759802+0000) 2024-03-06T08:52:26.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:26.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:26.760054+0000) 2024-03-06T08:52:27.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:27.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:27.760281+0000) 2024-03-06T08:52:28.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:28.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:28.760432+0000) 2024-03-06T08:52:29.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:29.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:29.760660+0000) 2024-03-06T08:52:30.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:30.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:30.760887+0000) 2024-03-06T08:52:31.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:31.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:31.761116+0000) 2024-03-06T08:52:32.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:32.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:32.761349+0000) 2024-03-06T08:52:33.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:33.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:33.761622+0000) 2024-03-06T08:52:34.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:34.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:34.761867+0000) 2024-03-06T08:52:35.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:35.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:35.762121+0000) 2024-03-06T08:52:36.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:36.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:36.762349+0000) 2024-03-06T08:52:37.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:37.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:37.762606+0000) 2024-03-06T08:52:38.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:38.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:38.762852+0000) 2024-03-06T08:52:39.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:39.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:39.763116+0000) 2024-03-06T08:52:40.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:40.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:40.763379+0000) 2024-03-06T08:52:41.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:41.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:41.763659+0000) 2024-03-06T08:52:42.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:42.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:42.763933+0000) 2024-03-06T08:52:43.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:43.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:43.764266+0000) 2024-03-06T08:52:44.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:44.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:44.764540+0000) 2024-03-06T08:52:45.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:45.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:45.764781+0000) 2024-03-06T08:52:46.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:46.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:46.764978+0000) 2024-03-06T08:52:47.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:47.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:47.765206+0000) 2024-03-06T08:52:48.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:48.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:52:48.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:48.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:48.765409+0000) 2024-03-06T08:52:49.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:49.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:49.765624+0000) 2024-03-06T08:52:50.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:50.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:50.765865+0000) 2024-03-06T08:52:51.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:51.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:51.766119+0000) 2024-03-06T08:52:52.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:52.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:52.766326+0000) 2024-03-06T08:52:53.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:53.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:53.766543+0000) 2024-03-06T08:52:54.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:54.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:54.766775+0000) 2024-03-06T08:52:55.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:55.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:55.766992+0000) 2024-03-06T08:52:56.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:56.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:56.767233+0000) 2024-03-06T08:52:57.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:57.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:57.767434+0000) 2024-03-06T08:52:58.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:58.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:58.767674+0000) 2024-03-06T08:52:59.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:52:59.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:52:59.767889+0000) 2024-03-06T08:53:00.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:00.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:00.768173+0000) 2024-03-06T08:53:01.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:01.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:01.768342+0000) 2024-03-06T08:53:02.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:02.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:02.768566+0000) 2024-03-06T08:53:03.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:03.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:03.768852+0000) 2024-03-06T08:53:04.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:04.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:04.769163+0000) 2024-03-06T08:53:05.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:05.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:05.769395+0000) 2024-03-06T08:53:06.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:06.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:06.769643+0000) 2024-03-06T08:53:07.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:07.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:07.769862+0000) 2024-03-06T08:53:08.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:08.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:08.770141+0000) 2024-03-06T08:53:09.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:09.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:09.770376+0000) 2024-03-06T08:53:10.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:10.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:10.770631+0000) 2024-03-06T08:53:11.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:11.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:11.770832+0000) 2024-03-06T08:53:12.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:12.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:12.771099+0000) 2024-03-06T08:53:13.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:13.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:13.771298+0000) 2024-03-06T08:53:14.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:14.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:14.771531+0000) 2024-03-06T08:53:15.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:15.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:15.771740+0000) 2024-03-06T08:53:16.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:16.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:16.771980+0000) 2024-03-06T08:53:17.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:17.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:17.772246+0000) 2024-03-06T08:53:18.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:18.275 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:18.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:18.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:18.772424+0000) 2024-03-06T08:53:19.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:19.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:19.772654+0000) 2024-03-06T08:53:20.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:20.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:20.772883+0000) 2024-03-06T08:53:21.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:21.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:21.773126+0000) 2024-03-06T08:53:22.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:22.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:22.773337+0000) 2024-03-06T08:53:23.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:23.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:23.773581+0000) 2024-03-06T08:53:24.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:24.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:24.773838+0000) 2024-03-06T08:53:25.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:25.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:25.774100+0000) 2024-03-06T08:53:26.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:26.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:26.774292+0000) 2024-03-06T08:53:27.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:27.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:27.774493+0000) 2024-03-06T08:53:28.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:28.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:28.774732+0000) 2024-03-06T08:53:29.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:29.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:29.774936+0000) 2024-03-06T08:53:30.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:30.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:30.775168+0000) 2024-03-06T08:53:31.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:31.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:31.775408+0000) 2024-03-06T08:53:32.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:32.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:32.775635+0000) 2024-03-06T08:53:33.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:33.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:33.775825+0000) 2024-03-06T08:53:34.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:34.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:34.776117+0000) 2024-03-06T08:53:35.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:35.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:35.776353+0000) 2024-03-06T08:53:36.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:36.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:36.776600+0000) 2024-03-06T08:53:37.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:37.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:37.776805+0000) 2024-03-06T08:53:38.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:38.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:38.777115+0000) 2024-03-06T08:53:39.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:39.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:39.777326+0000) 2024-03-06T08:53:40.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:40.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:40.777573+0000) 2024-03-06T08:53:41.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:41.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:41.777831+0000) 2024-03-06T08:53:42.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:42.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:42.778142+0000) 2024-03-06T08:53:43.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:43.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:43.778402+0000) 2024-03-06T08:53:44.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:44.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:44.778612+0000) 2024-03-06T08:53:45.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:45.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:45.778855+0000) 2024-03-06T08:53:46.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:46.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:46.779132+0000) 2024-03-06T08:53:47.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:47.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:47.779302+0000) 2024-03-06T08:53:48.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:48.412 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:53:48.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:48.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:48.779583+0000) 2024-03-06T08:53:49.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:49.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:49.779833+0000) 2024-03-06T08:53:50.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:50.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:50.780117+0000) 2024-03-06T08:53:51.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:51.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:51.780297+0000) 2024-03-06T08:53:52.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:52.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:52.780500+0000) 2024-03-06T08:53:53.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:53.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:53.780677+0000) 2024-03-06T08:53:54.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:54.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:54.780879+0000) 2024-03-06T08:53:55.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:55.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:55.781120+0000) 2024-03-06T08:53:56.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:56.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:56.781389+0000) 2024-03-06T08:53:57.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:57.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:57.781629+0000) 2024-03-06T08:53:58.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:58.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:58.781882+0000) 2024-03-06T08:53:59.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:53:59.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:53:59.782130+0000) 2024-03-06T08:54:00.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:00.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:00.782375+0000) 2024-03-06T08:54:01.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:01.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:01.782605+0000) 2024-03-06T08:54:02.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:02.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:02.782846+0000) 2024-03-06T08:54:03.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:03.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:03.783125+0000) 2024-03-06T08:54:04.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:04.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:04.783376+0000) 2024-03-06T08:54:05.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:05.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:05.783573+0000) 2024-03-06T08:54:06.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:06.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:06.783800+0000) 2024-03-06T08:54:07.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:07.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:07.784041+0000) 2024-03-06T08:54:08.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:08.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:08.784239+0000) 2024-03-06T08:54:09.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:09.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:09.784456+0000) 2024-03-06T08:54:10.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:10.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:10.784683+0000) 2024-03-06T08:54:11.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:11.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:11.784905+0000) 2024-03-06T08:54:12.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:12.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:12.785142+0000) 2024-03-06T08:54:13.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:13.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:13.785332+0000) 2024-03-06T08:54:14.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:14.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:14.785543+0000) 2024-03-06T08:54:15.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:15.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:15.785763+0000) 2024-03-06T08:54:16.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:16.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:16.785985+0000) 2024-03-06T08:54:17.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:17.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:17.786255+0000) 2024-03-06T08:54:18.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:18.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:18.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:18.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:18.786496+0000) 2024-03-06T08:54:19.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:19.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:19.786744+0000) 2024-03-06T08:54:20.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:20.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:20.786967+0000) 2024-03-06T08:54:21.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:21.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:21.787176+0000) 2024-03-06T08:54:22.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:22.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:22.787395+0000) 2024-03-06T08:54:23.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:23.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:23.787649+0000) 2024-03-06T08:54:24.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:24.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:24.787963+0000) 2024-03-06T08:54:25.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:25.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:25.788187+0000) 2024-03-06T08:54:26.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:26.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:26.788414+0000) 2024-03-06T08:54:27.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:27.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:27.788618+0000) 2024-03-06T08:54:28.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:28.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:28.788837+0000) 2024-03-06T08:54:29.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:29.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:29.789113+0000) 2024-03-06T08:54:30.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:30.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:30.789303+0000) 2024-03-06T08:54:31.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:31.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:31.789502+0000) 2024-03-06T08:54:32.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:32.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:32.789730+0000) 2024-03-06T08:54:33.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:33.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:33.789981+0000) 2024-03-06T08:54:34.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:34.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:34.790254+0000) 2024-03-06T08:54:35.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:35.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:35.790464+0000) 2024-03-06T08:54:36.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:36.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:36.790706+0000) 2024-03-06T08:54:37.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:37.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:37.790931+0000) 2024-03-06T08:54:38.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:38.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:38.791165+0000) 2024-03-06T08:54:39.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:39.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:39.791361+0000) 2024-03-06T08:54:40.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:40.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:40.791560+0000) 2024-03-06T08:54:41.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:41.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:41.791759+0000) 2024-03-06T08:54:42.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:42.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:42.791943+0000) 2024-03-06T08:54:43.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:43.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:43.792162+0000) 2024-03-06T08:54:44.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:44.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:44.792391+0000) 2024-03-06T08:54:45.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:45.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:45.792618+0000) 2024-03-06T08:54:46.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:46.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:46.792846+0000) 2024-03-06T08:54:47.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:47.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:47.793107+0000) 2024-03-06T08:54:48.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:48.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:48.793322+0000) 2024-03-06T08:54:48.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:48.931 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:54:49.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:49.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:49.793589+0000) 2024-03-06T08:54:50.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:50.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:50.793865+0000) 2024-03-06T08:54:51.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:51.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:51.794138+0000) 2024-03-06T08:54:52.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:52.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:52.794363+0000) 2024-03-06T08:54:53.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:53.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:53.794579+0000) 2024-03-06T08:54:54.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:54.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:54.794845+0000) 2024-03-06T08:54:55.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:55.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:55.795083+0000) 2024-03-06T08:54:56.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:56.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:56.795315+0000) 2024-03-06T08:54:57.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:57.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:57.795506+0000) 2024-03-06T08:54:58.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:58.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:58.795732+0000) 2024-03-06T08:54:59.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:54:59.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:54:59.795936+0000) 2024-03-06T08:55:00.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:00.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:00.796138+0000) 2024-03-06T08:55:01.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:01.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:01.796368+0000) 2024-03-06T08:55:02.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:02.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:02.796577+0000) 2024-03-06T08:55:03.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:03.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:03.796854+0000) 2024-03-06T08:55:04.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:04.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:04.797164+0000) 2024-03-06T08:55:05.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:05.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:05.797449+0000) 2024-03-06T08:55:06.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:06.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:06.797725+0000) 2024-03-06T08:55:07.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:07.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:07.798051+0000) 2024-03-06T08:55:08.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:08.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:08.798350+0000) 2024-03-06T08:55:09.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:09.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:09.798638+0000) 2024-03-06T08:55:10.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:10.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:10.798924+0000) 2024-03-06T08:55:11.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:11.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:11.799205+0000) 2024-03-06T08:55:12.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:12.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:12.799545+0000) 2024-03-06T08:55:13.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:13.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:13.799826+0000) 2024-03-06T08:55:14.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:14.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:14.800157+0000) 2024-03-06T08:55:15.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:15.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:15.800429+0000) 2024-03-06T08:55:16.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:16.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:16.800648+0000) 2024-03-06T08:55:17.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:17.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:17.800887+0000) 2024-03-06T08:55:18.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:18.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:18.801156+0000) 2024-03-06T08:55:19.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:19.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:19.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:19.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:19.801419+0000) 2024-03-06T08:55:20.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:20.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:20.801693+0000) 2024-03-06T08:55:21.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:21.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:21.801975+0000) 2024-03-06T08:55:22.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:22.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:22.802302+0000) 2024-03-06T08:55:23.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:23.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:23.802541+0000) 2024-03-06T08:55:24.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:24.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:24.802791+0000) 2024-03-06T08:55:25.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:25.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:25.803054+0000) 2024-03-06T08:55:26.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:26.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:26.803283+0000) 2024-03-06T08:55:27.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:27.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:27.803507+0000) 2024-03-06T08:55:28.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:28.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:28.803755+0000) 2024-03-06T08:55:29.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:29.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:29.803983+0000) 2024-03-06T08:55:30.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:30.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:30.804272+0000) 2024-03-06T08:55:31.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:31.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:31.804560+0000) 2024-03-06T08:55:32.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:32.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:32.804831+0000) 2024-03-06T08:55:33.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:33.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:33.805148+0000) 2024-03-06T08:55:34.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:34.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:34.805389+0000) 2024-03-06T08:55:35.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:35.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:35.805641+0000) 2024-03-06T08:55:36.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:36.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:36.805883+0000) 2024-03-06T08:55:37.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:37.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:37.806102+0000) 2024-03-06T08:55:38.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:38.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:38.806338+0000) 2024-03-06T08:55:39.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:39.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:39.806574+0000) 2024-03-06T08:55:40.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:40.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:40.806797+0000) 2024-03-06T08:55:41.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:41.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:41.807048+0000) 2024-03-06T08:55:42.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:42.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:42.807227+0000) 2024-03-06T08:55:43.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:43.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:43.807440+0000) 2024-03-06T08:55:44.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:44.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:44.807705+0000) 2024-03-06T08:55:45.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:45.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:45.807936+0000) 2024-03-06T08:55:46.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:46.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:46.808206+0000) 2024-03-06T08:55:47.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:47.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:47.808435+0000) 2024-03-06T08:55:48.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:48.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:48.808660+0000) 2024-03-06T08:55:49.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:49.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:55:49.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:49.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:49.808898+0000) 2024-03-06T08:55:50.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:50.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:50.809130+0000) 2024-03-06T08:55:51.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:51.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:51.809416+0000) 2024-03-06T08:55:52.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:52.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:52.809683+0000) 2024-03-06T08:55:53.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:53.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:53.809957+0000) 2024-03-06T08:55:54.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:54.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:54.810210+0000) 2024-03-06T08:55:55.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:55.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:55.810476+0000) 2024-03-06T08:55:56.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:56.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:56.810718+0000) 2024-03-06T08:55:57.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:57.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:57.810976+0000) 2024-03-06T08:55:58.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:58.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:58.811216+0000) 2024-03-06T08:55:59.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:55:59.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:55:59.811448+0000) 2024-03-06T08:56:00.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:00.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:00.811682+0000) 2024-03-06T08:56:01.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:01.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:01.811918+0000) 2024-03-06T08:56:02.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:02.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:02.812144+0000) 2024-03-06T08:56:03.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:03.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:03.812353+0000) 2024-03-06T08:56:04.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:04.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:04.812638+0000) 2024-03-06T08:56:05.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:05.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:05.812920+0000) 2024-03-06T08:56:06.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:06.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:06.813200+0000) 2024-03-06T08:56:07.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:07.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:07.813448+0000) 2024-03-06T08:56:08.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:08.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:08.813676+0000) 2024-03-06T08:56:09.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:09.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:09.813886+0000) 2024-03-06T08:56:10.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:10.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:10.814152+0000) 2024-03-06T08:56:11.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:11.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:11.814383+0000) 2024-03-06T08:56:12.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:12.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:12.814613+0000) 2024-03-06T08:56:13.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:13.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:13.814795+0000) 2024-03-06T08:56:14.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:14.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:14.815050+0000) 2024-03-06T08:56:15.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:15.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:15.815266+0000) 2024-03-06T08:56:16.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:16.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:16.815497+0000) 2024-03-06T08:56:17.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:17.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:17.815714+0000) 2024-03-06T08:56:18.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:18.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:18.815959+0000) 2024-03-06T08:56:19.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:19.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:19.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:19.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:19.816265+0000) 2024-03-06T08:56:20.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:20.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:20.816499+0000) 2024-03-06T08:56:21.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:21.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:21.816710+0000) 2024-03-06T08:56:22.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:22.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:22.816936+0000) 2024-03-06T08:56:23.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:23.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:23.817167+0000) 2024-03-06T08:56:24.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:24.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:24.817395+0000) 2024-03-06T08:56:25.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:25.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:25.817614+0000) 2024-03-06T08:56:26.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:26.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:26.817877+0000) 2024-03-06T08:56:27.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:27.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:27.818151+0000) 2024-03-06T08:56:28.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:28.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:28.818373+0000) 2024-03-06T08:56:29.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:29.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:29.818612+0000) 2024-03-06T08:56:30.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:30.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:30.818841+0000) 2024-03-06T08:56:31.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:31.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:31.819115+0000) 2024-03-06T08:56:32.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:32.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:32.819339+0000) 2024-03-06T08:56:33.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:33.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:33.819523+0000) 2024-03-06T08:56:34.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:34.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:34.819669+0000) 2024-03-06T08:56:35.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:35.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:35.819881+0000) 2024-03-06T08:56:36.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:36.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:36.820122+0000) 2024-03-06T08:56:37.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:37.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:37.820350+0000) 2024-03-06T08:56:38.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:38.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:38.820577+0000) 2024-03-06T08:56:39.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:39.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:39.820815+0000) 2024-03-06T08:56:40.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:40.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:40.821102+0000) 2024-03-06T08:56:41.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:41.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:41.821302+0000) 2024-03-06T08:56:42.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:42.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:42.821545+0000) 2024-03-06T08:56:43.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:43.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:43.821736+0000) 2024-03-06T08:56:44.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:44.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:44.821968+0000) 2024-03-06T08:56:45.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:45.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:45.822173+0000) 2024-03-06T08:56:46.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:46.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:46.822380+0000) 2024-03-06T08:56:47.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:47.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:47.822603+0000) 2024-03-06T08:56:48.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:48.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:48.822803+0000) 2024-03-06T08:56:49.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:49.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:56:49.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:49.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:49.823048+0000) 2024-03-06T08:56:50.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:50.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:50.823265+0000) 2024-03-06T08:56:51.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:51.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:51.823538+0000) 2024-03-06T08:56:52.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:52.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:52.823810+0000) 2024-03-06T08:56:53.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:53.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:53.824124+0000) 2024-03-06T08:56:54.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:54.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:54.824403+0000) 2024-03-06T08:56:55.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:55.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:55.824679+0000) 2024-03-06T08:56:56.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:56.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:56.824954+0000) 2024-03-06T08:56:57.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:57.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:57.825212+0000) 2024-03-06T08:56:58.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:58.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:58.825485+0000) 2024-03-06T08:56:59.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:56:59.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:56:59.825827+0000) 2024-03-06T08:57:00.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:00.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:00.826105+0000) 2024-03-06T08:57:01.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:01.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:01.826352+0000) 2024-03-06T08:57:02.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:02.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:02.826554+0000) 2024-03-06T08:57:03.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:03.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:03.826804+0000) 2024-03-06T08:57:04.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:04.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:04.827046+0000) 2024-03-06T08:57:05.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:05.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:05.827278+0000) 2024-03-06T08:57:06.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:06.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:06.827510+0000) 2024-03-06T08:57:07.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:07.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:07.827731+0000) 2024-03-06T08:57:08.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:08.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:08.827958+0000) 2024-03-06T08:57:09.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:09.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:09.828196+0000) 2024-03-06T08:57:10.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:10.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:10.828418+0000) 2024-03-06T08:57:11.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:11.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:11.828634+0000) 2024-03-06T08:57:12.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:12.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:12.828858+0000) 2024-03-06T08:57:13.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:13.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:13.829109+0000) 2024-03-06T08:57:14.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:14.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:14.829349+0000) 2024-03-06T08:57:15.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:15.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:15.829548+0000) 2024-03-06T08:57:16.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:16.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:16.829755+0000) 2024-03-06T08:57:17.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:17.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:17.829990+0000) 2024-03-06T08:57:18.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:18.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:18.830263+0000) 2024-03-06T08:57:19.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:19.664 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:19.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:19.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:19.830552+0000) 2024-03-06T08:57:20.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:20.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:20.830823+0000) 2024-03-06T08:57:21.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:21.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:21.831135+0000) 2024-03-06T08:57:22.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:22.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:22.831348+0000) 2024-03-06T08:57:23.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:23.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:23.831562+0000) 2024-03-06T08:57:24.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:24.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:24.831805+0000) 2024-03-06T08:57:25.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:25.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:25.832060+0000) 2024-03-06T08:57:26.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:26.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:26.832294+0000) 2024-03-06T08:57:27.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:27.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:27.832542+0000) 2024-03-06T08:57:28.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:28.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:28.832778+0000) 2024-03-06T08:57:29.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:29.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:29.833068+0000) 2024-03-06T08:57:30.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:30.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:30.833295+0000) 2024-03-06T08:57:31.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:31.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:31.833518+0000) 2024-03-06T08:57:32.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:32.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:32.833735+0000) 2024-03-06T08:57:33.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:33.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:33.833949+0000) 2024-03-06T08:57:34.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:34.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:34.834185+0000) 2024-03-06T08:57:35.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:35.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:35.834393+0000) 2024-03-06T08:57:36.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:36.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:36.834646+0000) 2024-03-06T08:57:37.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:37.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:37.834862+0000) 2024-03-06T08:57:38.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:38.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:38.835139+0000) 2024-03-06T08:57:39.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:39.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:39.835384+0000) 2024-03-06T08:57:40.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:40.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:40.835598+0000) 2024-03-06T08:57:41.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:41.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:41.835838+0000) 2024-03-06T08:57:42.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:42.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:42.836106+0000) 2024-03-06T08:57:43.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:43.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:43.836330+0000) 2024-03-06T08:57:44.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:44.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:44.836548+0000) 2024-03-06T08:57:45.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:45.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:45.836788+0000) 2024-03-06T08:57:46.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:46.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:46.837095+0000) 2024-03-06T08:57:47.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:47.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:47.837343+0000) 2024-03-06T08:57:48.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:48.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:48.837576+0000) 2024-03-06T08:57:49.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:49.809 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:57:49.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:49.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:49.837758+0000) 2024-03-06T08:57:50.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:50.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:50.837972+0000) 2024-03-06T08:57:51.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:51.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:51.838195+0000) 2024-03-06T08:57:52.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:52.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:52.838426+0000) 2024-03-06T08:57:53.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:53.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:53.838618+0000) 2024-03-06T08:57:54.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:54.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:54.838856+0000) 2024-03-06T08:57:55.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:55.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:55.839154+0000) 2024-03-06T08:57:56.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:56.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:56.839406+0000) 2024-03-06T08:57:57.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:57.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:57.839654+0000) 2024-03-06T08:57:58.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:58.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:58.839936+0000) 2024-03-06T08:57:59.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:57:59.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:57:59.840200+0000) 2024-03-06T08:58:00.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:00.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:00.840454+0000) 2024-03-06T08:58:01.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:01.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:01.840725+0000) 2024-03-06T08:58:02.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:02.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:02.841000+0000) 2024-03-06T08:58:03.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:03.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:03.841329+0000) 2024-03-06T08:58:04.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:04.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:04.841545+0000) 2024-03-06T08:58:05.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:05.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:05.841754+0000) 2024-03-06T08:58:06.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:06.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:06.841980+0000) 2024-03-06T08:58:07.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:07.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:07.842234+0000) 2024-03-06T08:58:08.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:08.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:08.842494+0000) 2024-03-06T08:58:09.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:09.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:09.842745+0000) 2024-03-06T08:58:10.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:10.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:10.842981+0000) 2024-03-06T08:58:11.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:11.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:11.843253+0000) 2024-03-06T08:58:12.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:12.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:12.843492+0000) 2024-03-06T08:58:13.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:13.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:13.843735+0000) 2024-03-06T08:58:14.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:14.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:14.843931+0000) 2024-03-06T08:58:15.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:15.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:15.844211+0000) 2024-03-06T08:58:16.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:16.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:16.844458+0000) 2024-03-06T08:58:17.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:17.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:17.844618+0000) 2024-03-06T08:58:18.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:18.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:18.844828+0000) 2024-03-06T08:58:19.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:19.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:19.845147+0000) 2024-03-06T08:58:19.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:19.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:20.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:20.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:20.845341+0000) 2024-03-06T08:58:21.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:21.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:21.845565+0000) 2024-03-06T08:58:22.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:22.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:22.845783+0000) 2024-03-06T08:58:23.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:23.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:23.846108+0000) 2024-03-06T08:58:24.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:24.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:24.846407+0000) 2024-03-06T08:58:25.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:25.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:25.846688+0000) 2024-03-06T08:58:26.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:26.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:26.846961+0000) 2024-03-06T08:58:27.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:27.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:27.847288+0000) 2024-03-06T08:58:28.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:28.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:28.847570+0000) 2024-03-06T08:58:29.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:29.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:29.847874+0000) 2024-03-06T08:58:30.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:30.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:30.848133+0000) 2024-03-06T08:58:31.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:31.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:31.848392+0000) 2024-03-06T08:58:32.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:32.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:32.848667+0000) 2024-03-06T08:58:33.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:33.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:33.848943+0000) 2024-03-06T08:58:34.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:34.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:34.849236+0000) 2024-03-06T08:58:35.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:35.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:35.849516+0000) 2024-03-06T08:58:36.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:36.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:36.849785+0000) 2024-03-06T08:58:37.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:37.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:37.850091+0000) 2024-03-06T08:58:38.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:38.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:38.850367+0000) 2024-03-06T08:58:39.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:39.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:39.850667+0000) 2024-03-06T08:58:40.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:40.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:40.850986+0000) 2024-03-06T08:58:41.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:41.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:41.851224+0000) 2024-03-06T08:58:42.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:42.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:42.851437+0000) 2024-03-06T08:58:43.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:43.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:43.851644+0000) 2024-03-06T08:58:44.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:44.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:44.851924+0000) 2024-03-06T08:58:45.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:45.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:45.852204+0000) 2024-03-06T08:58:46.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:46.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:46.852456+0000) 2024-03-06T08:58:47.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:47.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:47.852731+0000) 2024-03-06T08:58:48.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:48.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:48.852993+0000) 2024-03-06T08:58:49.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:49.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:49.853363+0000) 2024-03-06T08:58:50.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:50.110 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:58:50.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:50.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:50.853628+0000) 2024-03-06T08:58:51.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:51.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:51.853885+0000) 2024-03-06T08:58:52.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:52.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:52.854160+0000) 2024-03-06T08:58:53.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:53.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:53.854434+0000) 2024-03-06T08:58:54.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:54.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:54.854720+0000) 2024-03-06T08:58:55.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:55.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:55.854999+0000) 2024-03-06T08:58:56.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:56.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:56.855301+0000) 2024-03-06T08:58:57.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:57.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:57.855532+0000) 2024-03-06T08:58:58.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:58.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:58.855799+0000) 2024-03-06T08:58:59.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:58:59.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:58:59.856110+0000) 2024-03-06T08:59:00.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:00.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:00.856332+0000) 2024-03-06T08:59:01.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:01.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:01.856545+0000) 2024-03-06T08:59:02.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:02.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:02.856731+0000) 2024-03-06T08:59:03.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:03.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:03.856956+0000) 2024-03-06T08:59:04.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:04.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:04.857199+0000) 2024-03-06T08:59:05.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:05.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:05.857429+0000) 2024-03-06T08:59:06.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:06.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:06.857652+0000) 2024-03-06T08:59:07.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:07.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:07.857878+0000) 2024-03-06T08:59:08.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:08.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:08.858141+0000) 2024-03-06T08:59:09.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:09.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:09.858352+0000) 2024-03-06T08:59:10.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:10.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:10.858586+0000) 2024-03-06T08:59:11.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:11.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:11.858804+0000) 2024-03-06T08:59:12.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:12.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:12.859080+0000) 2024-03-06T08:59:13.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:13.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:13.859311+0000) 2024-03-06T08:59:14.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:14.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:14.859546+0000) 2024-03-06T08:59:15.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:15.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:15.859761+0000) 2024-03-06T08:59:16.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:16.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:16.860049+0000) 2024-03-06T08:59:17.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:17.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:17.860284+0000) 2024-03-06T08:59:18.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:18.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:18.860486+0000) 2024-03-06T08:59:19.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:19.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:19.860732+0000) 2024-03-06T08:59:20.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:20.257 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:20.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:20.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:20.860952+0000) 2024-03-06T08:59:21.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:21.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:21.861244+0000) 2024-03-06T08:59:22.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:22.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:22.861490+0000) 2024-03-06T08:59:23.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:23.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:23.861669+0000) 2024-03-06T08:59:24.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:24.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:24.861881+0000) 2024-03-06T08:59:25.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:25.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:25.862131+0000) 2024-03-06T08:59:26.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:26.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:26.862359+0000) 2024-03-06T08:59:27.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:27.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:27.862574+0000) 2024-03-06T08:59:28.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:28.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:28.862901+0000) 2024-03-06T08:59:29.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:29.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:29.863186+0000) 2024-03-06T08:59:30.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:30.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:30.863428+0000) 2024-03-06T08:59:31.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:31.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:31.863704+0000) 2024-03-06T08:59:32.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:32.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:32.863954+0000) 2024-03-06T08:59:33.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:33.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:33.864179+0000) 2024-03-06T08:59:34.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:34.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:34.864410+0000) 2024-03-06T08:59:35.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:35.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:35.864591+0000) 2024-03-06T08:59:36.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:36.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:36.864829+0000) 2024-03-06T08:59:37.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:37.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:37.865092+0000) 2024-03-06T08:59:38.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:38.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:38.865326+0000) 2024-03-06T08:59:39.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:39.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:39.865530+0000) 2024-03-06T08:59:40.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:40.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:40.865758+0000) 2024-03-06T08:59:41.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:41.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:41.865978+0000) 2024-03-06T08:59:42.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:42.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:42.866292+0000) 2024-03-06T08:59:43.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:43.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:43.866497+0000) 2024-03-06T08:59:44.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:44.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:44.866725+0000) 2024-03-06T08:59:45.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:45.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:45.866948+0000) 2024-03-06T08:59:46.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:46.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:46.867148+0000) 2024-03-06T08:59:47.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:47.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:47.867366+0000) 2024-03-06T08:59:48.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:48.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:48.867588+0000) 2024-03-06T08:59:49.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:49.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:49.867818+0000) 2024-03-06T08:59:50.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:50.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T08:59:50.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:50.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:50.868057+0000) 2024-03-06T08:59:51.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:51.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:51.868282+0000) 2024-03-06T08:59:52.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:52.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:52.868510+0000) 2024-03-06T08:59:53.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:53.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:53.868748+0000) 2024-03-06T08:59:54.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:54.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:54.868978+0000) 2024-03-06T08:59:55.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:55.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:55.869250+0000) 2024-03-06T08:59:56.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:56.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:56.869498+0000) 2024-03-06T08:59:57.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:57.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:57.869718+0000) 2024-03-06T08:59:58.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:58.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:58.869946+0000) 2024-03-06T08:59:59.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T08:59:59.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T07:59:59.870181+0000) 2024-03-06T09:00:00.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:00.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:00.870364+0000) 2024-03-06T09:00:01.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:01.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:01.870582+0000) 2024-03-06T09:00:02.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:02.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:02.870812+0000) 2024-03-06T09:00:03.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:03.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:03.871053+0000) 2024-03-06T09:00:04.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:04.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:04.871336+0000) 2024-03-06T09:00:05.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:05.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:05.871602+0000) 2024-03-06T09:00:06.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:06.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:06.871805+0000) 2024-03-06T09:00:07.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:07.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:07.872055+0000) 2024-03-06T09:00:08.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:08.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:08.872290+0000) 2024-03-06T09:00:09.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:09.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:09.872523+0000) 2024-03-06T09:00:10.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:10.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:10.872740+0000) 2024-03-06T09:00:11.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:11.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:11.872968+0000) 2024-03-06T09:00:12.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:12.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:12.873195+0000) 2024-03-06T09:00:13.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:13.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:13.873322+0000) 2024-03-06T09:00:14.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:14.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:14.873539+0000) 2024-03-06T09:00:15.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:15.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:15.873758+0000) 2024-03-06T09:00:16.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:16.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:16.874065+0000) 2024-03-06T09:00:17.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:17.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:17.874322+0000) 2024-03-06T09:00:18.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:18.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:18.874542+0000) 2024-03-06T09:00:19.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:19.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:19.874785+0000) 2024-03-06T09:00:20.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:20.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:20.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:20.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:20.875030+0000) 2024-03-06T09:00:21.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:21.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:21.875258+0000) 2024-03-06T09:00:22.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:22.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:22.875482+0000) 2024-03-06T09:00:23.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:23.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:23.875685+0000) 2024-03-06T09:00:24.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:24.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:24.875913+0000) 2024-03-06T09:00:25.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:25.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:25.876124+0000) 2024-03-06T09:00:26.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:26.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:26.876351+0000) 2024-03-06T09:00:27.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:27.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:27.876580+0000) 2024-03-06T09:00:28.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:28.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:28.876810+0000) 2024-03-06T09:00:29.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:29.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:29.877089+0000) 2024-03-06T09:00:30.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:30.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:30.877329+0000) 2024-03-06T09:00:31.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:31.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:31.877536+0000) 2024-03-06T09:00:32.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:32.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:32.877767+0000) 2024-03-06T09:00:33.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:33.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:33.877994+0000) 2024-03-06T09:00:34.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:34.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:34.878302+0000) 2024-03-06T09:00:35.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:35.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:35.878528+0000) 2024-03-06T09:00:36.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:36.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:36.878712+0000) 2024-03-06T09:00:37.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:37.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:37.878947+0000) 2024-03-06T09:00:38.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:38.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:38.879181+0000) 2024-03-06T09:00:39.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:39.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:39.879418+0000) 2024-03-06T09:00:40.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:40.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:40.879658+0000) 2024-03-06T09:00:41.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:41.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:41.879836+0000) 2024-03-06T09:00:42.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:42.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:42.880136+0000) 2024-03-06T09:00:43.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:43.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:43.880371+0000) 2024-03-06T09:00:44.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:44.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:44.880611+0000) 2024-03-06T09:00:45.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:45.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:45.880847+0000) 2024-03-06T09:00:46.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:46.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:46.881129+0000) 2024-03-06T09:00:47.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:47.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:47.881331+0000) 2024-03-06T09:00:48.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:48.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:48.881567+0000) 2024-03-06T09:00:49.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:49.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:49.881775+0000) 2024-03-06T09:00:50.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:50.709 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:00:50.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:50.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:50.882038+0000) 2024-03-06T09:00:51.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:51.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:51.882293+0000) 2024-03-06T09:00:52.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:52.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:52.882537+0000) 2024-03-06T09:00:53.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:53.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:53.882745+0000) 2024-03-06T09:00:54.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:54.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:54.882974+0000) 2024-03-06T09:00:55.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:55.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:55.883197+0000) 2024-03-06T09:00:56.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:56.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:56.883427+0000) 2024-03-06T09:00:57.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:57.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:57.883646+0000) 2024-03-06T09:00:58.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:58.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:58.883863+0000) 2024-03-06T09:00:59.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:00:59.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:00:59.884117+0000) 2024-03-06T09:01:00.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:00.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:00.884358+0000) 2024-03-06T09:01:01.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:01.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:01.884533+0000) 2024-03-06T09:01:02.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:02.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:02.884765+0000) 2024-03-06T09:01:03.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:03.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:03.884979+0000) 2024-03-06T09:01:04.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:04.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:04.885259+0000) 2024-03-06T09:01:05.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:05.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:05.885463+0000) 2024-03-06T09:01:06.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:06.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:06.885700+0000) 2024-03-06T09:01:07.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:07.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:07.885913+0000) 2024-03-06T09:01:08.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:08.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:08.886167+0000) 2024-03-06T09:01:09.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:09.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:09.886412+0000) 2024-03-06T09:01:10.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:10.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:10.886638+0000) 2024-03-06T09:01:11.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:11.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:11.886919+0000) 2024-03-06T09:01:12.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:12.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:12.887171+0000) 2024-03-06T09:01:13.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:13.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:13.887391+0000) 2024-03-06T09:01:14.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:14.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:14.887632+0000) 2024-03-06T09:01:15.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:15.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:15.887901+0000) 2024-03-06T09:01:16.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:16.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:16.888134+0000) 2024-03-06T09:01:17.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:17.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:17.888360+0000) 2024-03-06T09:01:18.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:18.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:18.888593+0000) 2024-03-06T09:01:19.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:19.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:19.888821+0000) 2024-03-06T09:01:20.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:20.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:20.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:20.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:20.889035+0000) 2024-03-06T09:01:21.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:21.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:21.889267+0000) 2024-03-06T09:01:22.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:22.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:22.889421+0000) 2024-03-06T09:01:23.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:23.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:23.889628+0000) 2024-03-06T09:01:24.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:24.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:24.889857+0000) 2024-03-06T09:01:25.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:25.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:25.890131+0000) 2024-03-06T09:01:26.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:26.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:26.890378+0000) 2024-03-06T09:01:27.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:27.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:27.890601+0000) 2024-03-06T09:01:28.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:28.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:28.890840+0000) 2024-03-06T09:01:29.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:29.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:29.891170+0000) 2024-03-06T09:01:30.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:30.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:30.891450+0000) 2024-03-06T09:01:31.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:31.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:31.891651+0000) 2024-03-06T09:01:32.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:32.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:32.891880+0000) 2024-03-06T09:01:33.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:33.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:33.892124+0000) 2024-03-06T09:01:34.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:34.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:34.892375+0000) 2024-03-06T09:01:35.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:35.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:35.892595+0000) 2024-03-06T09:01:36.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:36.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:36.892836+0000) 2024-03-06T09:01:37.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:37.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:37.893065+0000) 2024-03-06T09:01:38.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:38.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:38.893294+0000) 2024-03-06T09:01:39.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:39.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:39.893525+0000) 2024-03-06T09:01:40.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:40.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:40.893750+0000) 2024-03-06T09:01:41.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:41.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:41.894043+0000) 2024-03-06T09:01:42.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:42.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:42.894270+0000) 2024-03-06T09:01:43.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:43.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:43.894491+0000) 2024-03-06T09:01:44.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:44.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:44.894725+0000) 2024-03-06T09:01:45.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:45.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:45.894950+0000) 2024-03-06T09:01:46.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:46.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:46.895190+0000) 2024-03-06T09:01:47.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:47.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:47.895420+0000) 2024-03-06T09:01:48.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:48.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:48.895657+0000) 2024-03-06T09:01:49.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:49.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:49.895890+0000) 2024-03-06T09:01:50.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:50.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:50.896117+0000) 2024-03-06T09:01:51.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:51.008 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:01:51.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:51.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:51.896324+0000) 2024-03-06T09:01:52.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:52.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:52.896559+0000) 2024-03-06T09:01:53.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:53.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:53.896784+0000) 2024-03-06T09:01:54.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:54.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:54.897035+0000) 2024-03-06T09:01:55.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:55.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:55.897256+0000) 2024-03-06T09:01:56.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:56.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:56.897461+0000) 2024-03-06T09:01:57.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:57.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:57.897674+0000) 2024-03-06T09:01:58.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:58.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:58.897871+0000) 2024-03-06T09:01:59.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:01:59.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:01:59.898148+0000) 2024-03-06T09:02:00.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:00.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:00.898373+0000) 2024-03-06T09:02:01.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:01.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:01.898598+0000) 2024-03-06T09:02:02.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:02.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:02.898809+0000) 2024-03-06T09:02:03.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:03.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:03.899143+0000) 2024-03-06T09:02:04.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:04.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:04.899371+0000) 2024-03-06T09:02:05.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:05.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:05.899595+0000) 2024-03-06T09:02:06.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:06.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:06.899800+0000) 2024-03-06T09:02:07.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:07.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:07.900060+0000) 2024-03-06T09:02:08.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:08.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:08.900309+0000) 2024-03-06T09:02:09.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:09.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:09.900595+0000) 2024-03-06T09:02:10.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:10.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:10.900815+0000) 2024-03-06T09:02:11.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:11.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:11.901125+0000) 2024-03-06T09:02:12.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:12.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:12.901383+0000) 2024-03-06T09:02:13.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:13.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:13.901591+0000) 2024-03-06T09:02:14.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:14.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:14.901816+0000) 2024-03-06T09:02:15.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:15.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:15.902087+0000) 2024-03-06T09:02:16.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:16.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:16.902312+0000) 2024-03-06T09:02:17.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:17.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:17.902565+0000) 2024-03-06T09:02:18.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:18.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:18.902790+0000) 2024-03-06T09:02:19.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:19.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:19.903074+0000) 2024-03-06T09:02:20.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:20.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:20.903303+0000) 2024-03-06T09:02:21.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:21.154 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:21.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:21.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:21.903526+0000) 2024-03-06T09:02:22.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:22.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:22.903804+0000) 2024-03-06T09:02:23.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:23.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:23.904114+0000) 2024-03-06T09:02:24.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:24.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:24.904385+0000) 2024-03-06T09:02:25.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:25.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:25.904625+0000) 2024-03-06T09:02:26.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:26.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:26.904819+0000) 2024-03-06T09:02:27.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:27.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:27.905069+0000) 2024-03-06T09:02:28.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:28.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:28.905297+0000) 2024-03-06T09:02:29.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:29.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:29.905528+0000) 2024-03-06T09:02:30.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:30.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:30.905752+0000) 2024-03-06T09:02:31.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:31.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:31.905975+0000) 2024-03-06T09:02:32.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:32.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:32.906203+0000) 2024-03-06T09:02:33.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:33.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:33.906420+0000) 2024-03-06T09:02:34.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:34.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:34.906652+0000) 2024-03-06T09:02:35.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:35.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:35.906876+0000) 2024-03-06T09:02:36.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:36.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:36.907147+0000) 2024-03-06T09:02:37.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:37.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:37.907372+0000) 2024-03-06T09:02:38.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:38.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:38.907599+0000) 2024-03-06T09:02:39.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:39.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:39.907783+0000) 2024-03-06T09:02:40.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:40.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:40.908084+0000) 2024-03-06T09:02:41.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:41.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:41.908330+0000) 2024-03-06T09:02:42.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:42.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:42.908541+0000) 2024-03-06T09:02:43.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:43.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:43.908695+0000) 2024-03-06T09:02:44.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:44.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:44.908902+0000) 2024-03-06T09:02:45.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:45.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:45.909178+0000) 2024-03-06T09:02:46.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:46.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:46.909422+0000) 2024-03-06T09:02:47.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:47.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:47.909653+0000) 2024-03-06T09:02:48.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:48.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:48.909830+0000) 2024-03-06T09:02:49.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:49.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:49.910102+0000) 2024-03-06T09:02:50.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:50.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:50.910322+0000) 2024-03-06T09:02:51.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:51.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:02:51.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:51.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:51.910549+0000) 2024-03-06T09:02:52.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:52.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:52.910730+0000) 2024-03-06T09:02:53.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:53.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:53.910920+0000) 2024-03-06T09:02:54.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:54.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:54.911175+0000) 2024-03-06T09:02:55.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:55.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:55.911949+0000) 2024-03-06T09:02:56.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:56.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:56.912247+0000) 2024-03-06T09:02:57.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:57.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:57.912470+0000) 2024-03-06T09:02:58.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:58.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:58.912702+0000) 2024-03-06T09:02:59.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:02:59.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:02:59.912935+0000) 2024-03-06T09:03:00.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:00.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:00.913113+0000) 2024-03-06T09:03:01.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:01.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:01.913286+0000) 2024-03-06T09:03:02.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:02.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:02.913513+0000) 2024-03-06T09:03:03.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:03.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:03.913733+0000) 2024-03-06T09:03:04.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:04.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:04.913937+0000) 2024-03-06T09:03:05.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:05.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:05.914164+0000) 2024-03-06T09:03:06.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:06.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:06.914390+0000) 2024-03-06T09:03:07.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:07.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:07.914617+0000) 2024-03-06T09:03:08.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:08.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:08.914845+0000) 2024-03-06T09:03:09.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:09.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:09.915120+0000) 2024-03-06T09:03:10.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:10.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:10.915350+0000) 2024-03-06T09:03:11.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:11.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:11.915567+0000) 2024-03-06T09:03:12.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:12.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:12.915794+0000) 2024-03-06T09:03:13.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:13.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:13.916047+0000) 2024-03-06T09:03:14.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:14.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:14.916282+0000) 2024-03-06T09:03:15.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:15.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:15.916468+0000) 2024-03-06T09:03:16.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:16.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:16.916696+0000) 2024-03-06T09:03:17.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:17.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:17.916925+0000) 2024-03-06T09:03:18.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:18.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:18.917149+0000) 2024-03-06T09:03:19.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:19.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:19.917381+0000) 2024-03-06T09:03:20.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:20.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:20.917618+0000) 2024-03-06T09:03:21.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:21.435 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:21.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:21.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:21.917871+0000) 2024-03-06T09:03:22.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:22.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:22.918143+0000) 2024-03-06T09:03:23.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:23.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:23.918368+0000) 2024-03-06T09:03:24.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:24.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:24.918608+0000) 2024-03-06T09:03:25.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:25.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:25.918863+0000) 2024-03-06T09:03:26.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:26.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:26.919128+0000) 2024-03-06T09:03:27.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:27.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:27.919351+0000) 2024-03-06T09:03:28.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:28.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:28.919548+0000) 2024-03-06T09:03:29.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:29.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:29.919788+0000) 2024-03-06T09:03:30.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:30.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:30.920051+0000) 2024-03-06T09:03:31.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:31.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:31.920287+0000) 2024-03-06T09:03:32.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:32.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:32.920483+0000) 2024-03-06T09:03:33.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:33.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:33.920709+0000) 2024-03-06T09:03:34.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:34.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:34.920933+0000) 2024-03-06T09:03:35.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:35.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:35.921169+0000) 2024-03-06T09:03:36.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:36.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:36.921413+0000) 2024-03-06T09:03:37.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:37.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:37.921643+0000) 2024-03-06T09:03:38.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:38.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:38.921872+0000) 2024-03-06T09:03:39.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:39.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:39.922163+0000) 2024-03-06T09:03:40.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:40.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:40.922412+0000) 2024-03-06T09:03:41.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:41.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:41.922637+0000) 2024-03-06T09:03:42.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:42.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:42.922883+0000) 2024-03-06T09:03:43.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:43.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:43.923125+0000) 2024-03-06T09:03:44.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:44.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:44.923424+0000) 2024-03-06T09:03:45.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:45.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:45.923652+0000) 2024-03-06T09:03:46.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:46.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:46.923904+0000) 2024-03-06T09:03:47.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:47.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:47.924146+0000) 2024-03-06T09:03:48.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:48.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:48.924386+0000) 2024-03-06T09:03:49.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:49.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:49.924601+0000) 2024-03-06T09:03:50.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:50.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:50.924841+0000) 2024-03-06T09:03:51.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:51.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:03:51.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:51.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:51.925142+0000) 2024-03-06T09:03:52.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:52.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:52.925345+0000) 2024-03-06T09:03:53.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:53.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:53.925528+0000) 2024-03-06T09:03:54.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:54.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:54.925762+0000) 2024-03-06T09:03:55.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:55.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:55.926066+0000) 2024-03-06T09:03:56.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:56.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:56.926314+0000) 2024-03-06T09:03:57.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:57.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:57.926547+0000) 2024-03-06T09:03:58.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:58.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:58.926767+0000) 2024-03-06T09:03:59.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:03:59.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:03:59.927067+0000) 2024-03-06T09:04:00.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:00.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:00.927311+0000) 2024-03-06T09:04:01.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:01.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:01.927549+0000) 2024-03-06T09:04:02.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:02.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:02.927822+0000) 2024-03-06T09:04:03.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:03.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:03.928134+0000) 2024-03-06T09:04:04.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:04.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:04.928411+0000) 2024-03-06T09:04:05.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:05.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:05.928692+0000) 2024-03-06T09:04:06.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:06.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:06.928910+0000) 2024-03-06T09:04:07.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:07.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:07.929123+0000) 2024-03-06T09:04:08.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:08.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:08.929333+0000) 2024-03-06T09:04:09.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:09.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:09.929591+0000) 2024-03-06T09:04:10.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:10.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:10.929836+0000) 2024-03-06T09:04:11.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:11.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:11.930154+0000) 2024-03-06T09:04:12.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:12.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:12.930369+0000) 2024-03-06T09:04:13.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:13.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:13.930589+0000) 2024-03-06T09:04:14.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:14.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:14.930816+0000) 2024-03-06T09:04:15.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:15.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:15.931118+0000) 2024-03-06T09:04:16.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:16.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:16.931401+0000) 2024-03-06T09:04:17.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:17.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:17.931669+0000) 2024-03-06T09:04:18.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:18.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:18.931941+0000) 2024-03-06T09:04:19.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:19.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:19.932221+0000) 2024-03-06T09:04:20.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:20.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:20.932442+0000) 2024-03-06T09:04:21.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:21.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:21.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:21.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:21.932658+0000) 2024-03-06T09:04:22.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:22.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:22.932896+0000) 2024-03-06T09:04:23.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:23.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:23.933121+0000) 2024-03-06T09:04:24.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:24.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:24.933358+0000) 2024-03-06T09:04:25.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:25.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:25.933582+0000) 2024-03-06T09:04:26.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:26.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:26.933840+0000) 2024-03-06T09:04:27.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:27.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:27.934128+0000) 2024-03-06T09:04:28.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:28.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:28.934360+0000) 2024-03-06T09:04:29.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:29.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:29.934649+0000) 2024-03-06T09:04:30.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:30.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:30.934914+0000) 2024-03-06T09:04:31.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:31.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:31.935186+0000) 2024-03-06T09:04:32.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:32.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:32.935473+0000) 2024-03-06T09:04:33.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:33.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:33.935661+0000) 2024-03-06T09:04:34.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:34.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:34.935887+0000) 2024-03-06T09:04:35.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:35.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:35.936153+0000) 2024-03-06T09:04:36.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:36.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:36.936427+0000) 2024-03-06T09:04:37.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:37.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:37.936618+0000) 2024-03-06T09:04:38.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:38.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:38.936843+0000) 2024-03-06T09:04:39.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:39.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:39.937080+0000) 2024-03-06T09:04:40.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:40.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:40.937276+0000) 2024-03-06T09:04:41.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:41.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:41.937502+0000) 2024-03-06T09:04:42.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:42.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:42.937728+0000) 2024-03-06T09:04:43.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:43.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:43.937951+0000) 2024-03-06T09:04:44.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:44.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:44.938161+0000) 2024-03-06T09:04:45.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:45.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:45.938395+0000) 2024-03-06T09:04:46.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:46.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:46.938625+0000) 2024-03-06T09:04:47.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:47.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:47.938848+0000) 2024-03-06T09:04:48.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:48.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:48.939112+0000) 2024-03-06T09:04:49.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:49.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:49.939380+0000) 2024-03-06T09:04:50.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:50.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:50.939600+0000) 2024-03-06T09:04:51.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:51.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:51.939824+0000) 2024-03-06T09:04:51.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:51.996 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:04:52.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:52.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:52.940099+0000) 2024-03-06T09:04:53.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:53.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:53.940246+0000) 2024-03-06T09:04:54.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:54.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:54.940456+0000) 2024-03-06T09:04:55.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:55.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:55.940737+0000) 2024-03-06T09:04:56.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:56.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:56.940949+0000) 2024-03-06T09:04:57.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:57.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:57.941172+0000) 2024-03-06T09:04:58.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:58.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:58.941402+0000) 2024-03-06T09:04:59.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:04:59.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:04:59.941635+0000) 2024-03-06T09:05:00.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:00.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:00.941875+0000) 2024-03-06T09:05:01.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:01.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:01.942133+0000) 2024-03-06T09:05:02.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:02.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:02.942358+0000) 2024-03-06T09:05:03.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:03.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:03.942582+0000) 2024-03-06T09:05:04.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:04.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:04.942811+0000) 2024-03-06T09:05:05.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:05.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:05.943085+0000) 2024-03-06T09:05:06.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:06.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:06.943302+0000) 2024-03-06T09:05:07.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:07.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:07.943444+0000) 2024-03-06T09:05:08.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:08.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:08.943660+0000) 2024-03-06T09:05:09.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:09.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:09.943895+0000) 2024-03-06T09:05:10.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:10.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:10.944143+0000) 2024-03-06T09:05:11.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:11.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:11.944322+0000) 2024-03-06T09:05:12.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:12.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:12.944535+0000) 2024-03-06T09:05:13.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:13.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:13.944738+0000) 2024-03-06T09:05:14.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:14.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:14.944961+0000) 2024-03-06T09:05:15.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:15.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:15.945192+0000) 2024-03-06T09:05:16.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:16.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:16.945383+0000) 2024-03-06T09:05:17.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:17.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:17.945608+0000) 2024-03-06T09:05:18.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:18.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:18.945865+0000) 2024-03-06T09:05:19.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:19.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:19.946176+0000) 2024-03-06T09:05:20.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:20.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:20.946407+0000) 2024-03-06T09:05:21.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:21.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:21.946650+0000) 2024-03-06T09:05:22.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:22.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:22.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:22.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:22.946919+0000) 2024-03-06T09:05:23.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:23.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:23.947142+0000) 2024-03-06T09:05:24.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:24.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:24.947341+0000) 2024-03-06T09:05:25.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:25.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:25.947542+0000) 2024-03-06T09:05:26.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:26.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:26.947735+0000) 2024-03-06T09:05:27.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:27.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:27.947963+0000) 2024-03-06T09:05:28.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:28.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:28.948222+0000) 2024-03-06T09:05:29.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:29.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:29.948435+0000) 2024-03-06T09:05:30.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:30.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:30.948644+0000) 2024-03-06T09:05:31.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:31.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:31.948890+0000) 2024-03-06T09:05:32.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:32.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:32.949137+0000) 2024-03-06T09:05:33.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:33.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:33.949378+0000) 2024-03-06T09:05:34.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:34.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:34.949602+0000) 2024-03-06T09:05:35.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:35.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:35.949821+0000) 2024-03-06T09:05:36.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:36.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:36.950042+0000) 2024-03-06T09:05:37.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:37.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:37.950265+0000) 2024-03-06T09:05:38.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:38.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:38.950504+0000) 2024-03-06T09:05:39.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:39.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:39.950764+0000) 2024-03-06T09:05:40.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:40.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:40.951072+0000) 2024-03-06T09:05:41.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:41.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:41.951352+0000) 2024-03-06T09:05:42.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:42.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:42.951628+0000) 2024-03-06T09:05:43.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:43.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:43.951874+0000) 2024-03-06T09:05:44.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:44.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:44.952146+0000) 2024-03-06T09:05:45.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:45.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:45.952338+0000) 2024-03-06T09:05:46.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:46.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:46.952519+0000) 2024-03-06T09:05:47.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:47.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:47.952709+0000) 2024-03-06T09:05:48.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:48.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:48.952896+0000) 2024-03-06T09:05:49.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:49.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:49.953125+0000) 2024-03-06T09:05:50.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:50.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:50.953304+0000) 2024-03-06T09:05:51.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:51.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:51.953525+0000) 2024-03-06T09:05:52.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:52.282 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:05:52.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:52.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:52.953794+0000) 2024-03-06T09:05:53.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:53.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:53.954111+0000) 2024-03-06T09:05:54.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:54.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:54.954384+0000) 2024-03-06T09:05:55.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:55.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:55.954663+0000) 2024-03-06T09:05:56.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:56.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:56.954920+0000) 2024-03-06T09:05:57.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:57.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:57.955170+0000) 2024-03-06T09:05:58.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:58.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:58.955373+0000) 2024-03-06T09:05:59.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:05:59.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:05:59.955610+0000) 2024-03-06T09:06:00.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:00.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:00.955845+0000) 2024-03-06T09:06:01.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:01.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:01.956113+0000) 2024-03-06T09:06:02.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:02.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:02.956324+0000) 2024-03-06T09:06:03.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:03.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:03.956550+0000) 2024-03-06T09:06:04.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:04.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:04.956779+0000) 2024-03-06T09:06:05.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:05.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:05.957020+0000) 2024-03-06T09:06:06.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:06.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:06.957274+0000) 2024-03-06T09:06:07.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:07.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:07.957515+0000) 2024-03-06T09:06:08.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:08.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:08.957754+0000) 2024-03-06T09:06:09.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:09.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:09.957991+0000) 2024-03-06T09:06:10.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:10.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:10.958265+0000) 2024-03-06T09:06:11.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:11.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:11.958493+0000) 2024-03-06T09:06:12.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:12.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:12.958752+0000) 2024-03-06T09:06:13.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:13.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:13.959111+0000) 2024-03-06T09:06:14.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:14.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:14.959343+0000) 2024-03-06T09:06:15.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:15.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:15.959631+0000) 2024-03-06T09:06:16.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:16.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:16.959915+0000) 2024-03-06T09:06:17.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:17.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:17.960202+0000) 2024-03-06T09:06:18.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:18.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:18.960409+0000) 2024-03-06T09:06:19.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:19.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:19.960648+0000) 2024-03-06T09:06:20.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:20.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:20.960867+0000) 2024-03-06T09:06:21.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:21.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:21.961129+0000) 2024-03-06T09:06:22.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:22.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:22.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:22.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:22.961383+0000) 2024-03-06T09:06:23.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:23.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:23.961626+0000) 2024-03-06T09:06:24.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:24.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:24.961878+0000) 2024-03-06T09:06:25.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:25.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:25.962117+0000) 2024-03-06T09:06:26.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:26.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:26.962367+0000) 2024-03-06T09:06:27.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:27.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:27.962613+0000) 2024-03-06T09:06:28.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:28.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:28.962871+0000) 2024-03-06T09:06:29.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:29.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:29.963133+0000) 2024-03-06T09:06:30.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:30.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:30.963378+0000) 2024-03-06T09:06:31.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:31.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:31.963622+0000) 2024-03-06T09:06:32.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:32.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:32.963874+0000) 2024-03-06T09:06:33.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:33.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:33.964119+0000) 2024-03-06T09:06:34.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:34.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:34.964374+0000) 2024-03-06T09:06:35.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:35.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:35.964621+0000) 2024-03-06T09:06:36.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:36.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:36.964870+0000) 2024-03-06T09:06:37.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:37.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:37.965119+0000) 2024-03-06T09:06:38.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:38.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:38.965379+0000) 2024-03-06T09:06:39.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:39.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:39.965624+0000) 2024-03-06T09:06:40.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:40.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:40.965856+0000) 2024-03-06T09:06:41.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:41.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:41.966152+0000) 2024-03-06T09:06:42.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:42.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:42.966364+0000) 2024-03-06T09:06:43.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:43.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:43.966523+0000) 2024-03-06T09:06:44.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:44.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:44.966716+0000) 2024-03-06T09:06:45.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:45.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:45.966944+0000) 2024-03-06T09:06:46.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:46.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:46.967169+0000) 2024-03-06T09:06:47.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:47.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:47.967388+0000) 2024-03-06T09:06:48.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:48.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:48.967573+0000) 2024-03-06T09:06:49.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:49.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:49.967796+0000) 2024-03-06T09:06:50.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:50.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:50.968062+0000) 2024-03-06T09:06:51.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:51.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:51.968304+0000) 2024-03-06T09:06:52.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:52.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:06:52.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:52.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:52.968583+0000) 2024-03-06T09:06:53.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:53.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:53.968865+0000) 2024-03-06T09:06:54.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:54.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:54.969151+0000) 2024-03-06T09:06:55.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:55.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:55.969353+0000) 2024-03-06T09:06:56.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:56.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:56.969587+0000) 2024-03-06T09:06:57.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:57.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:57.969806+0000) 2024-03-06T09:06:58.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:58.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:58.970094+0000) 2024-03-06T09:06:59.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:06:59.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:06:59.970329+0000) 2024-03-06T09:07:00.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:00.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:00.970555+0000) 2024-03-06T09:07:01.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:01.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:01.970828+0000) 2024-03-06T09:07:02.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:02.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:02.971100+0000) 2024-03-06T09:07:03.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:03.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:03.971297+0000) 2024-03-06T09:07:04.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:04.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:04.971535+0000) 2024-03-06T09:07:05.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:05.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:05.971726+0000) 2024-03-06T09:07:06.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:06.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:06.971921+0000) 2024-03-06T09:07:07.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:07.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:07.972081+0000) 2024-03-06T09:07:08.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:08.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:08.972290+0000) 2024-03-06T09:07:09.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:09.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:09.972512+0000) 2024-03-06T09:07:10.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:10.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:10.972774+0000) 2024-03-06T09:07:11.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:11.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:11.973039+0000) 2024-03-06T09:07:12.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:12.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:12.973234+0000) 2024-03-06T09:07:13.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:13.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:13.973450+0000) 2024-03-06T09:07:14.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:14.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:14.973674+0000) 2024-03-06T09:07:15.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:15.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:15.973890+0000) 2024-03-06T09:07:16.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:16.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:16.974122+0000) 2024-03-06T09:07:17.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:17.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:17.974320+0000) 2024-03-06T09:07:18.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:18.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:18.974549+0000) 2024-03-06T09:07:19.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:19.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:19.974779+0000) 2024-03-06T09:07:20.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:20.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:20.974997+0000) 2024-03-06T09:07:21.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:21.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:21.975296+0000) 2024-03-06T09:07:22.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:22.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:22.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:22.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:22.975557+0000) 2024-03-06T09:07:23.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:23.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:23.975805+0000) 2024-03-06T09:07:24.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:24.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:24.976086+0000) 2024-03-06T09:07:25.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:25.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:25.976326+0000) 2024-03-06T09:07:26.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:26.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:26.976514+0000) 2024-03-06T09:07:27.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:27.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:27.976751+0000) 2024-03-06T09:07:28.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:28.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:28.976973+0000) 2024-03-06T09:07:29.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:29.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:29.977284+0000) 2024-03-06T09:07:30.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:30.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:30.977584+0000) 2024-03-06T09:07:31.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:31.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:31.977827+0000) 2024-03-06T09:07:32.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:32.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:32.978114+0000) 2024-03-06T09:07:33.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:33.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:33.978350+0000) 2024-03-06T09:07:34.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:34.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:34.978566+0000) 2024-03-06T09:07:35.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:35.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:35.978830+0000) 2024-03-06T09:07:36.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:36.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:36.979156+0000) 2024-03-06T09:07:37.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:37.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:37.979359+0000) 2024-03-06T09:07:38.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:38.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:38.979582+0000) 2024-03-06T09:07:39.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:39.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:39.979819+0000) 2024-03-06T09:07:40.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:40.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:40.980095+0000) 2024-03-06T09:07:41.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:41.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:41.980357+0000) 2024-03-06T09:07:42.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:42.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:42.980640+0000) 2024-03-06T09:07:43.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:43.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:43.980920+0000) 2024-03-06T09:07:44.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:44.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:44.981194+0000) 2024-03-06T09:07:45.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:45.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:45.981438+0000) 2024-03-06T09:07:46.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:46.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:46.981720+0000) 2024-03-06T09:07:47.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:47.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:47.981993+0000) 2024-03-06T09:07:48.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:48.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:48.982299+0000) 2024-03-06T09:07:49.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:49.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:49.982555+0000) 2024-03-06T09:07:50.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:50.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:50.982771+0000) 2024-03-06T09:07:51.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:51.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:51.982996+0000) 2024-03-06T09:07:52.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:52.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:07:52.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:52.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:52.983244+0000) 2024-03-06T09:07:53.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:53.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:53.983466+0000) 2024-03-06T09:07:54.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:54.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:54.983750+0000) 2024-03-06T09:07:55.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:55.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:55.984105+0000) 2024-03-06T09:07:56.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:56.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:56.984249+0000) 2024-03-06T09:07:57.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:57.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:57.984469+0000) 2024-03-06T09:07:58.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:58.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:58.984626+0000) 2024-03-06T09:07:59.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:07:59.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:07:59.984884+0000) 2024-03-06T09:08:00.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:00.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:00.985127+0000) 2024-03-06T09:08:01.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:01.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:01.985347+0000) 2024-03-06T09:08:02.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:02.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:02.985535+0000) 2024-03-06T09:08:03.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:03.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:03.985797+0000) 2024-03-06T09:08:04.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:04.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:04.986088+0000) 2024-03-06T09:08:05.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:05.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:05.986325+0000) 2024-03-06T09:08:06.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:06.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:06.986511+0000) 2024-03-06T09:08:07.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:07.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:07.986728+0000) 2024-03-06T09:08:08.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:08.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:08.986956+0000) 2024-03-06T09:08:09.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:09.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:09.987188+0000) 2024-03-06T09:08:10.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:10.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:10.987408+0000) 2024-03-06T09:08:11.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:11.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:11.987641+0000) 2024-03-06T09:08:12.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:12.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:12.987873+0000) 2024-03-06T09:08:13.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:13.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:13.988131+0000) 2024-03-06T09:08:14.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:14.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:14.988355+0000) 2024-03-06T09:08:15.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:15.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:15.988548+0000) 2024-03-06T09:08:16.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:16.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:16.988755+0000) 2024-03-06T09:08:17.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:17.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:17.988970+0000) 2024-03-06T09:08:18.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:18.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:18.989217+0000) 2024-03-06T09:08:19.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:19.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:19.989433+0000) 2024-03-06T09:08:20.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:20.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:20.989618+0000) 2024-03-06T09:08:21.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:21.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:21.989864+0000) 2024-03-06T09:08:22.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:22.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:22.990136+0000) 2024-03-06T09:08:23.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:23.048 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:23.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:23.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:23.990355+0000) 2024-03-06T09:08:24.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:24.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:24.990626+0000) 2024-03-06T09:08:25.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:25.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:25.990820+0000) 2024-03-06T09:08:26.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:26.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:26.991100+0000) 2024-03-06T09:08:27.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:27.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:27.991319+0000) 2024-03-06T09:08:28.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:28.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:28.991508+0000) 2024-03-06T09:08:29.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:29.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:29.991739+0000) 2024-03-06T09:08:30.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:30.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:30.991961+0000) 2024-03-06T09:08:31.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:31.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:31.992208+0000) 2024-03-06T09:08:32.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:32.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:32.992483+0000) 2024-03-06T09:08:33.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:33.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:33.992677+0000) 2024-03-06T09:08:34.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:34.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:34.992922+0000) 2024-03-06T09:08:35.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:35.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:35.993159+0000) 2024-03-06T09:08:36.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:36.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:36.993409+0000) 2024-03-06T09:08:37.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:37.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:37.993608+0000) 2024-03-06T09:08:38.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:38.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:38.993869+0000) 2024-03-06T09:08:39.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:39.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:39.994101+0000) 2024-03-06T09:08:40.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:40.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:40.994308+0000) 2024-03-06T09:08:41.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:41.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:41.994519+0000) 2024-03-06T09:08:42.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:42.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:42.994760+0000) 2024-03-06T09:08:43.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:43.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:43.994978+0000) 2024-03-06T09:08:44.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:44.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:44.995215+0000) 2024-03-06T09:08:45.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:45.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:45.995479+0000) 2024-03-06T09:08:46.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:46.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:46.995737+0000) 2024-03-06T09:08:47.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:47.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:47.995920+0000) 2024-03-06T09:08:48.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:48.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:48.996311+0000) 2024-03-06T09:08:49.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:49.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:49.996519+0000) 2024-03-06T09:08:50.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:50.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:50.996755+0000) 2024-03-06T09:08:51.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:51.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:51.996909+0000) 2024-03-06T09:08:52.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:52.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:52.997131+0000) 2024-03-06T09:08:53.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:53.192 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:08:53.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:53.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:53.997372+0000) 2024-03-06T09:08:54.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:54.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:54.997617+0000) 2024-03-06T09:08:55.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:55.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:55.997856+0000) 2024-03-06T09:08:56.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:56.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:56.998147+0000) 2024-03-06T09:08:57.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:57.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:57.998390+0000) 2024-03-06T09:08:58.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:58.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:58.998640+0000) 2024-03-06T09:08:59.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:08:59.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:08:59.998897+0000) 2024-03-06T09:09:00.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:00.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:00.999174+0000) 2024-03-06T09:09:01.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:01.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:01.999395+0000) 2024-03-06T09:09:02.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:02.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:02.999641+0000) 2024-03-06T09:09:03.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:03.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:03.999901+0000) 2024-03-06T09:09:04.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:04.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:05.000155+0000) 2024-03-06T09:09:05.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:05.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:06.000389+0000) 2024-03-06T09:09:06.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:06.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:07.000612+0000) 2024-03-06T09:09:08.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:07.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:08.000844+0000) 2024-03-06T09:09:09.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:09.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:09.001087+0000) 2024-03-06T09:09:10.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:10.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:10.001329+0000) 2024-03-06T09:09:11.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:11.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:11.001525+0000) 2024-03-06T09:09:12.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:12.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:12.001765+0000) 2024-03-06T09:09:13.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:13.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:13.002047+0000) 2024-03-06T09:09:14.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:14.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:14.002296+0000) 2024-03-06T09:09:15.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:15.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:15.002541+0000) 2024-03-06T09:09:16.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:16.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:16.002754+0000) 2024-03-06T09:09:17.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:17.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:17.002989+0000) 2024-03-06T09:09:18.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:18.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:18.003810+0000) 2024-03-06T09:09:19.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:19.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:19.004106+0000) 2024-03-06T09:09:20.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:20.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:20.004351+0000) 2024-03-06T09:09:21.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:21.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:21.004567+0000) 2024-03-06T09:09:22.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:22.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:22.004799+0000) 2024-03-06T09:09:23.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:23.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:23.005058+0000) 2024-03-06T09:09:23.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:23.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:24.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:24.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:24.005288+0000) 2024-03-06T09:09:25.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:25.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:25.005526+0000) 2024-03-06T09:09:26.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:26.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:26.005767+0000) 2024-03-06T09:09:27.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:27.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:27.006068+0000) 2024-03-06T09:09:28.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:28.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:28.006342+0000) 2024-03-06T09:09:29.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:29.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:29.006550+0000) 2024-03-06T09:09:30.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:30.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:30.006782+0000) 2024-03-06T09:09:31.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:31.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:31.006999+0000) 2024-03-06T09:09:32.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:32.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:32.007405+0000) 2024-03-06T09:09:33.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:33.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:33.007642+0000) 2024-03-06T09:09:34.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:34.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:34.007929+0000) 2024-03-06T09:09:35.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:35.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:35.008147+0000) 2024-03-06T09:09:36.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:36.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:36.008333+0000) 2024-03-06T09:09:37.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:37.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:37.008557+0000) 2024-03-06T09:09:38.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:38.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:38.008781+0000) 2024-03-06T09:09:39.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:39.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:39.009029+0000) 2024-03-06T09:09:40.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:40.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:40.009232+0000) 2024-03-06T09:09:41.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:41.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:41.009423+0000) 2024-03-06T09:09:42.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:42.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:42.009659+0000) 2024-03-06T09:09:43.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:43.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:43.009883+0000) 2024-03-06T09:09:44.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:44.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:44.010144+0000) 2024-03-06T09:09:45.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:45.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:45.010380+0000) 2024-03-06T09:09:46.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:46.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:46.010599+0000) 2024-03-06T09:09:47.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:47.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:47.010840+0000) 2024-03-06T09:09:48.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:48.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:48.011110+0000) 2024-03-06T09:09:49.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:49.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:49.011305+0000) 2024-03-06T09:09:50.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:50.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:50.011520+0000) 2024-03-06T09:09:51.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:51.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:51.011739+0000) 2024-03-06T09:09:52.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:52.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:52.011912+0000) 2024-03-06T09:09:53.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:53.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:53.012156+0000) 2024-03-06T09:09:53.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:53.503 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:09:54.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:54.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:54.012390+0000) 2024-03-06T09:09:55.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:55.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:55.012620+0000) 2024-03-06T09:09:56.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:56.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:56.012850+0000) 2024-03-06T09:09:57.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:57.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:57.013132+0000) 2024-03-06T09:09:58.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:58.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:58.013357+0000) 2024-03-06T09:09:59.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:09:59.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:09:59.013580+0000) 2024-03-06T09:10:00.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:00.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:00.013777+0000) 2024-03-06T09:10:01.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:01.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:01.014047+0000) 2024-03-06T09:10:02.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:02.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:02.014273+0000) 2024-03-06T09:10:03.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:03.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:03.014512+0000) 2024-03-06T09:10:04.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:04.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:04.014762+0000) 2024-03-06T09:10:05.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:05.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:05.014992+0000) 2024-03-06T09:10:06.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:06.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:06.015249+0000) 2024-03-06T09:10:07.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:07.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:07.015531+0000) 2024-03-06T09:10:08.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:08.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:08.015769+0000) 2024-03-06T09:10:09.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:09.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:09.015965+0000) 2024-03-06T09:10:10.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:10.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:10.016237+0000) 2024-03-06T09:10:11.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:11.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:11.016461+0000) 2024-03-06T09:10:12.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:12.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:12.016699+0000) 2024-03-06T09:10:13.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:13.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:13.016932+0000) 2024-03-06T09:10:14.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:14.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:14.017158+0000) 2024-03-06T09:10:15.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:15.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:15.017369+0000) 2024-03-06T09:10:16.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:16.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:16.017569+0000) 2024-03-06T09:10:17.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:17.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:17.017781+0000) 2024-03-06T09:10:18.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:18.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:18.017997+0000) 2024-03-06T09:10:19.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:19.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:19.018273+0000) 2024-03-06T09:10:20.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:20.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:20.018495+0000) 2024-03-06T09:10:21.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:21.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:21.018678+0000) 2024-03-06T09:10:22.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:22.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:22.018880+0000) 2024-03-06T09:10:23.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:23.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:23.019129+0000) 2024-03-06T09:10:23.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:23.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:24.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:24.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:24.019326+0000) 2024-03-06T09:10:25.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:25.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:25.019606+0000) 2024-03-06T09:10:26.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:26.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:26.019787+0000) 2024-03-06T09:10:27.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:27.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:27.020055+0000) 2024-03-06T09:10:28.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:28.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:28.020291+0000) 2024-03-06T09:10:29.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:29.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:29.020469+0000) 2024-03-06T09:10:30.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:30.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:30.020708+0000) 2024-03-06T09:10:31.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:31.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:31.020927+0000) 2024-03-06T09:10:32.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:32.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:32.021157+0000) 2024-03-06T09:10:33.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:33.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:33.021423+0000) 2024-03-06T09:10:34.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:34.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:34.021632+0000) 2024-03-06T09:10:35.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:35.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:35.021859+0000) 2024-03-06T09:10:36.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:36.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:36.022127+0000) 2024-03-06T09:10:37.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:37.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:37.022339+0000) 2024-03-06T09:10:38.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:38.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:38.022571+0000) 2024-03-06T09:10:39.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:39.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:39.022805+0000) 2024-03-06T09:10:40.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:40.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:40.023061+0000) 2024-03-06T09:10:41.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:41.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:41.023254+0000) 2024-03-06T09:10:42.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:42.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:42.023510+0000) 2024-03-06T09:10:43.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:43.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:43.023762+0000) 2024-03-06T09:10:44.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:44.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:44.023992+0000) 2024-03-06T09:10:45.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:45.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:45.024290+0000) 2024-03-06T09:10:46.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:46.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:46.024546+0000) 2024-03-06T09:10:47.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:47.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:47.024797+0000) 2024-03-06T09:10:48.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:48.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:48.025043+0000) 2024-03-06T09:10:49.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:49.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:49.025299+0000) 2024-03-06T09:10:50.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:50.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:50.025545+0000) 2024-03-06T09:10:51.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:51.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:51.025790+0000) 2024-03-06T09:10:52.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:52.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:52.025998+0000) 2024-03-06T09:10:53.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:53.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:53.026252+0000) 2024-03-06T09:10:53.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:53.792 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:10:54.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:54.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:54.026505+0000) 2024-03-06T09:10:55.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:55.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:55.026701+0000) 2024-03-06T09:10:56.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:56.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:56.026894+0000) 2024-03-06T09:10:57.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:57.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:57.027147+0000) 2024-03-06T09:10:58.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:58.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:58.027341+0000) 2024-03-06T09:10:59.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:10:59.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:10:59.027557+0000) 2024-03-06T09:11:00.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:00.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:00.027788+0000) 2024-03-06T09:11:01.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:01.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:01.028064+0000) 2024-03-06T09:11:02.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:02.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:02.028257+0000) 2024-03-06T09:11:03.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:03.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:03.028474+0000) 2024-03-06T09:11:04.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:04.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:04.028660+0000) 2024-03-06T09:11:05.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:05.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:05.028895+0000) 2024-03-06T09:11:06.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:06.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:06.029138+0000) 2024-03-06T09:11:07.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:07.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:07.029385+0000) 2024-03-06T09:11:08.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:08.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:08.029630+0000) 2024-03-06T09:11:09.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:09.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:09.029874+0000) 2024-03-06T09:11:10.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:10.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:10.030120+0000) 2024-03-06T09:11:11.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:11.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:11.030318+0000) 2024-03-06T09:11:12.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:12.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:12.030549+0000) 2024-03-06T09:11:13.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:13.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:13.030847+0000) 2024-03-06T09:11:14.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:14.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:14.031116+0000) 2024-03-06T09:11:15.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:15.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:15.031315+0000) 2024-03-06T09:11:16.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:16.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:16.031518+0000) 2024-03-06T09:11:17.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:17.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:17.031773+0000) 2024-03-06T09:11:18.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:18.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:18.032054+0000) 2024-03-06T09:11:19.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:19.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:19.032308+0000) 2024-03-06T09:11:20.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:20.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:20.032549+0000) 2024-03-06T09:11:21.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:21.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:21.032784+0000) 2024-03-06T09:11:22.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:22.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:22.033028+0000) 2024-03-06T09:11:23.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:23.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:23.033255+0000) 2024-03-06T09:11:23.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:23.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:24.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:24.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:24.033468+0000) 2024-03-06T09:11:25.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:25.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:25.033688+0000) 2024-03-06T09:11:26.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:26.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:26.033915+0000) 2024-03-06T09:11:27.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:27.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:27.034142+0000) 2024-03-06T09:11:28.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:28.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:28.034340+0000) 2024-03-06T09:11:29.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:29.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:29.034570+0000) 2024-03-06T09:11:30.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:30.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:30.034798+0000) 2024-03-06T09:11:31.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:31.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:31.035070+0000) 2024-03-06T09:11:32.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:32.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:32.035330+0000) 2024-03-06T09:11:33.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:33.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:33.035613+0000) 2024-03-06T09:11:34.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:34.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:34.035856+0000) 2024-03-06T09:11:35.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:35.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:35.036139+0000) 2024-03-06T09:11:36.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:36.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:36.036359+0000) 2024-03-06T09:11:37.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:37.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:37.036619+0000) 2024-03-06T09:11:38.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:38.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:38.036895+0000) 2024-03-06T09:11:39.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:39.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:39.037151+0000) 2024-03-06T09:11:40.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:40.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:40.037456+0000) 2024-03-06T09:11:41.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:41.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:41.037678+0000) 2024-03-06T09:11:42.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:42.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:42.037898+0000) 2024-03-06T09:11:43.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:43.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:43.038138+0000) 2024-03-06T09:11:44.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:44.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:44.038330+0000) 2024-03-06T09:11:45.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:45.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:45.038565+0000) 2024-03-06T09:11:46.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:46.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:46.038762+0000) 2024-03-06T09:11:47.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:47.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:47.038999+0000) 2024-03-06T09:11:48.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:48.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:48.039263+0000) 2024-03-06T09:11:49.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:49.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:49.039461+0000) 2024-03-06T09:11:50.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:50.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:50.039681+0000) 2024-03-06T09:11:51.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:51.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:51.039899+0000) 2024-03-06T09:11:52.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:52.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:52.040121+0000) 2024-03-06T09:11:53.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:53.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:53.040366+0000) 2024-03-06T09:11:54.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:54.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:54.040589+0000) 2024-03-06T09:11:54.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:54.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:11:55.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:55.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:55.040836+0000) 2024-03-06T09:11:56.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:56.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:56.041077+0000) 2024-03-06T09:11:57.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:57.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:57.041319+0000) 2024-03-06T09:11:58.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:58.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:58.041581+0000) 2024-03-06T09:11:59.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:11:59.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:11:59.041827+0000) 2024-03-06T09:12:00.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:00.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:00.042091+0000) 2024-03-06T09:12:01.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:01.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:01.042345+0000) 2024-03-06T09:12:02.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:02.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:02.042548+0000) 2024-03-06T09:12:03.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:03.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:03.042773+0000) 2024-03-06T09:12:04.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:04.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:04.042947+0000) 2024-03-06T09:12:05.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:05.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:05.043172+0000) 2024-03-06T09:12:06.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:06.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:06.043418+0000) 2024-03-06T09:12:07.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:07.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:07.043665+0000) 2024-03-06T09:12:08.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:08.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:08.043885+0000) 2024-03-06T09:12:09.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:09.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:09.044109+0000) 2024-03-06T09:12:10.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:10.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:10.044332+0000) 2024-03-06T09:12:11.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:11.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:11.044563+0000) 2024-03-06T09:12:12.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:12.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:12.044806+0000) 2024-03-06T09:12:13.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:13.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:13.045131+0000) 2024-03-06T09:12:14.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:14.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:14.045362+0000) 2024-03-06T09:12:15.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:15.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:15.045586+0000) 2024-03-06T09:12:16.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:16.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:16.045803+0000) 2024-03-06T09:12:17.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:17.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:17.046076+0000) 2024-03-06T09:12:18.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:18.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:18.046313+0000) 2024-03-06T09:12:19.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:19.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:19.046556+0000) 2024-03-06T09:12:20.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:20.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:20.046772+0000) 2024-03-06T09:12:21.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:21.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:21.046987+0000) 2024-03-06T09:12:22.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:22.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:22.047237+0000) 2024-03-06T09:12:23.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:23.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:23.047495+0000) 2024-03-06T09:12:24.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:24.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:24.047743+0000) 2024-03-06T09:12:24.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:24.230 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:25.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:25.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:25.047991+0000) 2024-03-06T09:12:26.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:26.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:26.048240+0000) 2024-03-06T09:12:27.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:27.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:27.048513+0000) 2024-03-06T09:12:28.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:28.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:28.048751+0000) 2024-03-06T09:12:29.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:29.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:29.048973+0000) 2024-03-06T09:12:30.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:30.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:30.049283+0000) 2024-03-06T09:12:31.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:31.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:31.049525+0000) 2024-03-06T09:12:32.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:32.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:32.049741+0000) 2024-03-06T09:12:33.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:33.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:33.049968+0000) 2024-03-06T09:12:34.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:34.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:34.050199+0000) 2024-03-06T09:12:35.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:35.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:35.050426+0000) 2024-03-06T09:12:36.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:36.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:36.050622+0000) 2024-03-06T09:12:37.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:37.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:37.050846+0000) 2024-03-06T09:12:38.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:38.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:38.051120+0000) 2024-03-06T09:12:39.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:39.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:39.051342+0000) 2024-03-06T09:12:40.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:40.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:40.051599+0000) 2024-03-06T09:12:41.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:41.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:41.051840+0000) 2024-03-06T09:12:42.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:42.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:42.052102+0000) 2024-03-06T09:12:43.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:43.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:43.052369+0000) 2024-03-06T09:12:44.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:44.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:44.052596+0000) 2024-03-06T09:12:45.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:45.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:45.052778+0000) 2024-03-06T09:12:46.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:46.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:46.052997+0000) 2024-03-06T09:12:47.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:47.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:47.053218+0000) 2024-03-06T09:12:48.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:48.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:48.053415+0000) 2024-03-06T09:12:49.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:49.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:49.053639+0000) 2024-03-06T09:12:50.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:50.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:50.053891+0000) 2024-03-06T09:12:51.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:51.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:51.054158+0000) 2024-03-06T09:12:52.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:52.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:52.054363+0000) 2024-03-06T09:12:53.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:53.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:53.054590+0000) 2024-03-06T09:12:54.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:54.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:54.054823+0000) 2024-03-06T09:12:54.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:54.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:12:55.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:55.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:55.055103+0000) 2024-03-06T09:12:56.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:56.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:56.055297+0000) 2024-03-06T09:12:57.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:57.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:57.055482+0000) 2024-03-06T09:12:58.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:58.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:58.055691+0000) 2024-03-06T09:12:59.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:12:59.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:12:59.055904+0000) 2024-03-06T09:13:00.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:00.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:00.056198+0000) 2024-03-06T09:13:01.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:01.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:01.056388+0000) 2024-03-06T09:13:02.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:02.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:02.056574+0000) 2024-03-06T09:13:03.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:03.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:03.056803+0000) 2024-03-06T09:13:04.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:04.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:04.057062+0000) 2024-03-06T09:13:05.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:05.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:05.057249+0000) 2024-03-06T09:13:06.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:06.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:06.057474+0000) 2024-03-06T09:13:07.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:07.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:07.057705+0000) 2024-03-06T09:13:08.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:08.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:08.057915+0000) 2024-03-06T09:13:09.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:09.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:09.058157+0000) 2024-03-06T09:13:10.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:10.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:10.058388+0000) 2024-03-06T09:13:11.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:11.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:11.058589+0000) 2024-03-06T09:13:12.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:12.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:12.058808+0000) 2024-03-06T09:13:13.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:13.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:13.059072+0000) 2024-03-06T09:13:14.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:14.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:14.059292+0000) 2024-03-06T09:13:15.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:15.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:15.059518+0000) 2024-03-06T09:13:16.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:16.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:16.059742+0000) 2024-03-06T09:13:17.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:17.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:17.059975+0000) 2024-03-06T09:13:18.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:18.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:18.060239+0000) 2024-03-06T09:13:19.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:19.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:19.060466+0000) 2024-03-06T09:13:20.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:20.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:20.060698+0000) 2024-03-06T09:13:21.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:21.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:21.060922+0000) 2024-03-06T09:13:22.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:22.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:22.061135+0000) 2024-03-06T09:13:23.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:23.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:23.061346+0000) 2024-03-06T09:13:24.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:24.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:24.061569+0000) 2024-03-06T09:13:24.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:24.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:25.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:25.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:25.061807+0000) 2024-03-06T09:13:26.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:26.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:26.062098+0000) 2024-03-06T09:13:27.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:27.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:27.062278+0000) 2024-03-06T09:13:28.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:28.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:28.062459+0000) 2024-03-06T09:13:29.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:29.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:29.062736+0000) 2024-03-06T09:13:30.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:30.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:30.063036+0000) 2024-03-06T09:13:31.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:31.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:31.063254+0000) 2024-03-06T09:13:32.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:32.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:32.063469+0000) 2024-03-06T09:13:33.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:33.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:33.063712+0000) 2024-03-06T09:13:34.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:34.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:34.063961+0000) 2024-03-06T09:13:35.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:35.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:35.064184+0000) 2024-03-06T09:13:36.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:36.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:36.064397+0000) 2024-03-06T09:13:37.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:37.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:37.064661+0000) 2024-03-06T09:13:38.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:38.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:38.064921+0000) 2024-03-06T09:13:39.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:39.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:39.065182+0000) 2024-03-06T09:13:40.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:40.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:40.065334+0000) 2024-03-06T09:13:41.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:41.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:41.065553+0000) 2024-03-06T09:13:42.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:42.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:42.065871+0000) 2024-03-06T09:13:43.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:43.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:43.066123+0000) 2024-03-06T09:13:44.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:44.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:44.066276+0000) 2024-03-06T09:13:45.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:45.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:45.066482+0000) 2024-03-06T09:13:46.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:46.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:46.066700+0000) 2024-03-06T09:13:47.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:47.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:47.066963+0000) 2024-03-06T09:13:48.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:48.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:48.067172+0000) 2024-03-06T09:13:49.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:49.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:49.067413+0000) 2024-03-06T09:13:50.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:50.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:50.067705+0000) 2024-03-06T09:13:51.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:51.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:51.067941+0000) 2024-03-06T09:13:52.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:52.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:52.068152+0000) 2024-03-06T09:13:53.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:53.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:53.068370+0000) 2024-03-06T09:13:54.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:54.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:54.068585+0000) 2024-03-06T09:13:54.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:54.678 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:13:55.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:55.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:55.068832+0000) 2024-03-06T09:13:56.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:56.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:56.069062+0000) 2024-03-06T09:13:57.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:57.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:57.069301+0000) 2024-03-06T09:13:58.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:58.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:58.069564+0000) 2024-03-06T09:13:59.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:13:59.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:13:59.069791+0000) 2024-03-06T09:14:00.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:00.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:00.070049+0000) 2024-03-06T09:14:01.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:01.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:01.070246+0000) 2024-03-06T09:14:02.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:02.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:02.070473+0000) 2024-03-06T09:14:03.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:03.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:03.070708+0000) 2024-03-06T09:14:04.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:04.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:04.070934+0000) 2024-03-06T09:14:05.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:05.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:05.071152+0000) 2024-03-06T09:14:06.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:06.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:06.071342+0000) 2024-03-06T09:14:07.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:07.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:07.071549+0000) 2024-03-06T09:14:08.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:08.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:08.071793+0000) 2024-03-06T09:14:09.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:09.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:09.072063+0000) 2024-03-06T09:14:10.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:10.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:10.072259+0000) 2024-03-06T09:14:11.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:11.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:11.072464+0000) 2024-03-06T09:14:12.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:12.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:12.072682+0000) 2024-03-06T09:14:13.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:13.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:13.072914+0000) 2024-03-06T09:14:14.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:14.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:14.073153+0000) 2024-03-06T09:14:15.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:15.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:15.073338+0000) 2024-03-06T09:14:16.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:16.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:16.073544+0000) 2024-03-06T09:14:17.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:17.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:17.073768+0000) 2024-03-06T09:14:18.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:18.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:18.073999+0000) 2024-03-06T09:14:19.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:19.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:19.074226+0000) 2024-03-06T09:14:20.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:20.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:20.074425+0000) 2024-03-06T09:14:21.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:21.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:21.074639+0000) 2024-03-06T09:14:22.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:22.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:22.074869+0000) 2024-03-06T09:14:23.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:23.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:23.075135+0000) 2024-03-06T09:14:24.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:24.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:24.075350+0000) 2024-03-06T09:14:24.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:24.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:25.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:25.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:25.075586+0000) 2024-03-06T09:14:26.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:26.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:26.075787+0000) 2024-03-06T09:14:27.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:27.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:27.076061+0000) 2024-03-06T09:14:28.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:28.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:28.076262+0000) 2024-03-06T09:14:29.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:29.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:29.076480+0000) 2024-03-06T09:14:30.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:30.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:30.076758+0000) 2024-03-06T09:14:31.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:31.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:31.076980+0000) 2024-03-06T09:14:32.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:32.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:32.077194+0000) 2024-03-06T09:14:33.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:33.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:33.077439+0000) 2024-03-06T09:14:34.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:34.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:34.077700+0000) 2024-03-06T09:14:35.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:35.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:35.077942+0000) 2024-03-06T09:14:36.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:36.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:36.078154+0000) 2024-03-06T09:14:37.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:37.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:37.078349+0000) 2024-03-06T09:14:38.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:38.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:38.078550+0000) 2024-03-06T09:14:39.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:39.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:39.078782+0000) 2024-03-06T09:14:40.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:40.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:40.079066+0000) 2024-03-06T09:14:41.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:41.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:41.079258+0000) 2024-03-06T09:14:42.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:42.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:42.079459+0000) 2024-03-06T09:14:43.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:43.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:43.079697+0000) 2024-03-06T09:14:44.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:44.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:44.079925+0000) 2024-03-06T09:14:45.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:45.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:45.080173+0000) 2024-03-06T09:14:46.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:46.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:46.080360+0000) 2024-03-06T09:14:47.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:47.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:47.080566+0000) 2024-03-06T09:14:48.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:48.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:48.080802+0000) 2024-03-06T09:14:49.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:49.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:49.081065+0000) 2024-03-06T09:14:50.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:50.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:50.081262+0000) 2024-03-06T09:14:51.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:51.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:51.081474+0000) 2024-03-06T09:14:52.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:52.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:52.081679+0000) 2024-03-06T09:14:53.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:53.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:53.081918+0000) 2024-03-06T09:14:54.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:54.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:54.082156+0000) 2024-03-06T09:14:54.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:54.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:14:55.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:55.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:55.082369+0000) 2024-03-06T09:14:56.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:56.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:56.082556+0000) 2024-03-06T09:14:57.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:57.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:57.082782+0000) 2024-03-06T09:14:58.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:58.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:58.083031+0000) 2024-03-06T09:14:59.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:14:59.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:14:59.083267+0000) 2024-03-06T09:15:00.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:00.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:00.083489+0000) 2024-03-06T09:15:01.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:01.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:01.083717+0000) 2024-03-06T09:15:02.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:02.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:02.083938+0000) 2024-03-06T09:15:03.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:03.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:03.084169+0000) 2024-03-06T09:15:04.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:04.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:04.084387+0000) 2024-03-06T09:15:05.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:05.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:05.084594+0000) 2024-03-06T09:15:06.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:06.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:06.084809+0000) 2024-03-06T09:15:07.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:07.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:07.085067+0000) 2024-03-06T09:15:08.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:08.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:08.085302+0000) 2024-03-06T09:15:09.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:09.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:09.085520+0000) 2024-03-06T09:15:10.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:10.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:10.085761+0000) 2024-03-06T09:15:11.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:11.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:11.085953+0000) 2024-03-06T09:15:12.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:12.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:12.086203+0000) 2024-03-06T09:15:13.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:13.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:13.086459+0000) 2024-03-06T09:15:14.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:14.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:14.086683+0000) 2024-03-06T09:15:15.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:15.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:15.086904+0000) 2024-03-06T09:15:16.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:16.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:16.087137+0000) 2024-03-06T09:15:17.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:17.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:17.087328+0000) 2024-03-06T09:15:18.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:18.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:18.087563+0000) 2024-03-06T09:15:19.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:19.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:19.087795+0000) 2024-03-06T09:15:20.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:20.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:20.088051+0000) 2024-03-06T09:15:21.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:21.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:21.088287+0000) 2024-03-06T09:15:22.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:22.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:22.088535+0000) 2024-03-06T09:15:23.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:23.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:23.088775+0000) 2024-03-06T09:15:24.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:24.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:24.089091+0000) 2024-03-06T09:15:25.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:25.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:25.089315+0000) 2024-03-06T09:15:25.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:25.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:26.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:26.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:26.089509+0000) 2024-03-06T09:15:27.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:27.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:27.089737+0000) 2024-03-06T09:15:28.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:28.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:28.090045+0000) 2024-03-06T09:15:29.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:29.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:29.090267+0000) 2024-03-06T09:15:30.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:30.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:30.090479+0000) 2024-03-06T09:15:31.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:31.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:31.090739+0000) 2024-03-06T09:15:32.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:32.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:32.090979+0000) 2024-03-06T09:15:33.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:33.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:33.091228+0000) 2024-03-06T09:15:34.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:34.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:34.091440+0000) 2024-03-06T09:15:35.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:35.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:35.091650+0000) 2024-03-06T09:15:36.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:36.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:36.091841+0000) 2024-03-06T09:15:37.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:37.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:37.092070+0000) 2024-03-06T09:15:38.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:38.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:38.092301+0000) 2024-03-06T09:15:39.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:39.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:39.092538+0000) 2024-03-06T09:15:40.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:40.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:40.092771+0000) 2024-03-06T09:15:41.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:41.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:41.092995+0000) 2024-03-06T09:15:42.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:42.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:42.093272+0000) 2024-03-06T09:15:43.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:43.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:43.093493+0000) 2024-03-06T09:15:44.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:44.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:44.093683+0000) 2024-03-06T09:15:45.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:45.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:45.093907+0000) 2024-03-06T09:15:46.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:46.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:46.094119+0000) 2024-03-06T09:15:47.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:47.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:47.094347+0000) 2024-03-06T09:15:48.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:48.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:48.094580+0000) 2024-03-06T09:15:49.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:49.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:49.094813+0000) 2024-03-06T09:15:50.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:50.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:50.095030+0000) 2024-03-06T09:15:51.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:51.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:51.095250+0000) 2024-03-06T09:15:52.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:52.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:52.095475+0000) 2024-03-06T09:15:53.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:53.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:53.095710+0000) 2024-03-06T09:15:54.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:54.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:54.095862+0000) 2024-03-06T09:15:55.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:55.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:55.096121+0000) 2024-03-06T09:15:55.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:55.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:15:56.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:56.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:56.096342+0000) 2024-03-06T09:15:57.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:57.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:57.096503+0000) 2024-03-06T09:15:58.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:58.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:58.096745+0000) 2024-03-06T09:15:59.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:15:59.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:15:59.096989+0000) 2024-03-06T09:16:00.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:00.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:00.097302+0000) 2024-03-06T09:16:01.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:01.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:01.097524+0000) 2024-03-06T09:16:02.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:02.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:02.097742+0000) 2024-03-06T09:16:03.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:03.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:03.098071+0000) 2024-03-06T09:16:04.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:04.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:04.098354+0000) 2024-03-06T09:16:05.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:05.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:05.098616+0000) 2024-03-06T09:16:06.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:06.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:06.098888+0000) 2024-03-06T09:16:07.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:07.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:07.099172+0000) 2024-03-06T09:16:08.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:08.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:08.099472+0000) 2024-03-06T09:16:09.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:09.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:09.099726+0000) 2024-03-06T09:16:10.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:10.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:10.099989+0000) 2024-03-06T09:16:11.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:11.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:11.100222+0000) 2024-03-06T09:16:12.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:12.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:12.100421+0000) 2024-03-06T09:16:13.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:13.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:13.100655+0000) 2024-03-06T09:16:14.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:14.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:14.100922+0000) 2024-03-06T09:16:15.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:15.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:15.101160+0000) 2024-03-06T09:16:16.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:16.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:16.101437+0000) 2024-03-06T09:16:17.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:17.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:17.101742+0000) 2024-03-06T09:16:18.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:18.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:18.102058+0000) 2024-03-06T09:16:19.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:19.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:19.102336+0000) 2024-03-06T09:16:20.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:20.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:20.102573+0000) 2024-03-06T09:16:21.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:21.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:21.102770+0000) 2024-03-06T09:16:22.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:22.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:22.102984+0000) 2024-03-06T09:16:23.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:23.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:23.103253+0000) 2024-03-06T09:16:24.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:24.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:24.103470+0000) 2024-03-06T09:16:25.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:25.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:25.103714+0000) 2024-03-06T09:16:25.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:25.433 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:26.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:26.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:26.103953+0000) 2024-03-06T09:16:27.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:27.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:27.104204+0000) 2024-03-06T09:16:28.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:28.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:28.104406+0000) 2024-03-06T09:16:29.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:29.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:29.104612+0000) 2024-03-06T09:16:30.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:30.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:30.104833+0000) 2024-03-06T09:16:31.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:31.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:31.105056+0000) 2024-03-06T09:16:32.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:32.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:32.105287+0000) 2024-03-06T09:16:33.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:33.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:33.105504+0000) 2024-03-06T09:16:34.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:34.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:34.105768+0000) 2024-03-06T09:16:35.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:35.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:35.106099+0000) 2024-03-06T09:16:36.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:36.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:36.106336+0000) 2024-03-06T09:16:37.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:37.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:37.106598+0000) 2024-03-06T09:16:38.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:38.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:38.106885+0000) 2024-03-06T09:16:39.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:39.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:39.107150+0000) 2024-03-06T09:16:40.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:40.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:40.107438+0000) 2024-03-06T09:16:41.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:41.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:41.107658+0000) 2024-03-06T09:16:42.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:42.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:42.107875+0000) 2024-03-06T09:16:43.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:43.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:43.108177+0000) 2024-03-06T09:16:44.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:44.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:44.108430+0000) 2024-03-06T09:16:45.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:45.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:45.108627+0000) 2024-03-06T09:16:46.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:46.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:46.108865+0000) 2024-03-06T09:16:47.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:47.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:47.109124+0000) 2024-03-06T09:16:48.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:48.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:48.109358+0000) 2024-03-06T09:16:49.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:49.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:49.109589+0000) 2024-03-06T09:16:50.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:50.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:50.109801+0000) 2024-03-06T09:16:51.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:51.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:51.110067+0000) 2024-03-06T09:16:52.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:52.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:52.110313+0000) 2024-03-06T09:16:53.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:53.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:53.110553+0000) 2024-03-06T09:16:54.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:54.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:54.110795+0000) 2024-03-06T09:16:55.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:55.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:55.111074+0000) 2024-03-06T09:16:55.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:55.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:16:56.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:56.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:56.111297+0000) 2024-03-06T09:16:57.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:57.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:57.111502+0000) 2024-03-06T09:16:58.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:58.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:58.111719+0000) 2024-03-06T09:16:59.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:16:59.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:16:59.111920+0000) 2024-03-06T09:17:00.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:00.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:00.112138+0000) 2024-03-06T09:17:01.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:01.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:01.112373+0000) 2024-03-06T09:17:02.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:02.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:02.112595+0000) 2024-03-06T09:17:03.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:03.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:03.112817+0000) 2024-03-06T09:17:04.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:04.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:04.113109+0000) 2024-03-06T09:17:05.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:05.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:05.113326+0000) 2024-03-06T09:17:06.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:06.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:06.113550+0000) 2024-03-06T09:17:07.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:07.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:07.113841+0000) 2024-03-06T09:17:08.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:08.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:08.114118+0000) 2024-03-06T09:17:09.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:09.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:09.114349+0000) 2024-03-06T09:17:10.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:10.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:10.114631+0000) 2024-03-06T09:17:11.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:11.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:11.114851+0000) 2024-03-06T09:17:12.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:12.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:12.115077+0000) 2024-03-06T09:17:13.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:13.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:13.115252+0000) 2024-03-06T09:17:14.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:14.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:14.115469+0000) 2024-03-06T09:17:15.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:15.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:15.115721+0000) 2024-03-06T09:17:16.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:16.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:16.115945+0000) 2024-03-06T09:17:17.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:17.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:17.116175+0000) 2024-03-06T09:17:18.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:18.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:18.116395+0000) 2024-03-06T09:17:19.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:19.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:19.116623+0000) 2024-03-06T09:17:20.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:20.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:20.116850+0000) 2024-03-06T09:17:21.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:21.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:21.117130+0000) 2024-03-06T09:17:22.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:22.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:22.117369+0000) 2024-03-06T09:17:23.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:23.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:23.117594+0000) 2024-03-06T09:17:24.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:24.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:24.117814+0000) 2024-03-06T09:17:25.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:25.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:25.118068+0000) 2024-03-06T09:17:25.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:25.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:26.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:26.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:26.118290+0000) 2024-03-06T09:17:27.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:27.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:27.118478+0000) 2024-03-06T09:17:28.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:28.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:28.118664+0000) 2024-03-06T09:17:29.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:29.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:29.118895+0000) 2024-03-06T09:17:30.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:30.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:30.119130+0000) 2024-03-06T09:17:31.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:31.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:31.119359+0000) 2024-03-06T09:17:32.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:32.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:32.119610+0000) 2024-03-06T09:17:33.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:33.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:33.119856+0000) 2024-03-06T09:17:34.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:34.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:34.120179+0000) 2024-03-06T09:17:35.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:35.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:35.120416+0000) 2024-03-06T09:17:36.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:36.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:36.120647+0000) 2024-03-06T09:17:37.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:37.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:37.120901+0000) 2024-03-06T09:17:38.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:38.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:38.121159+0000) 2024-03-06T09:17:39.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:39.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:39.121407+0000) 2024-03-06T09:17:40.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:40.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:40.121642+0000) 2024-03-06T09:17:41.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:41.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:41.121866+0000) 2024-03-06T09:17:42.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:42.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:42.122134+0000) 2024-03-06T09:17:43.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:43.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:43.122362+0000) 2024-03-06T09:17:44.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:44.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:44.122584+0000) 2024-03-06T09:17:45.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:45.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:45.122807+0000) 2024-03-06T09:17:46.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:46.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:46.123059+0000) 2024-03-06T09:17:47.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:47.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:47.123300+0000) 2024-03-06T09:17:48.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:48.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:48.123546+0000) 2024-03-06T09:17:49.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:49.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:49.123770+0000) 2024-03-06T09:17:50.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:50.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:50.124032+0000) 2024-03-06T09:17:51.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:51.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:51.124258+0000) 2024-03-06T09:17:52.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:52.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:52.124478+0000) 2024-03-06T09:17:53.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:53.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:53.124705+0000) 2024-03-06T09:17:54.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:54.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:54.124927+0000) 2024-03-06T09:17:55.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:55.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:55.125155+0000) 2024-03-06T09:17:55.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:55.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:17:56.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:56.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:56.125421+0000) 2024-03-06T09:17:57.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:57.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:57.125602+0000) 2024-03-06T09:17:58.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:58.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:58.125833+0000) 2024-03-06T09:17:59.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:17:59.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:17:59.126117+0000) 2024-03-06T09:18:00.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:00.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:00.126319+0000) 2024-03-06T09:18:01.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:01.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:01.126523+0000) 2024-03-06T09:18:02.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:02.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:02.126743+0000) 2024-03-06T09:18:03.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:03.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:03.126968+0000) 2024-03-06T09:18:04.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:04.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:04.127235+0000) 2024-03-06T09:18:05.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:05.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:05.127519+0000) 2024-03-06T09:18:06.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:06.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:06.127798+0000) 2024-03-06T09:18:07.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:07.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:07.128113+0000) 2024-03-06T09:18:08.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:08.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:08.128393+0000) 2024-03-06T09:18:09.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:09.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:09.128676+0000) 2024-03-06T09:18:10.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:10.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:10.128913+0000) 2024-03-06T09:18:11.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:11.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:11.129138+0000) 2024-03-06T09:18:12.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:12.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:12.129329+0000) 2024-03-06T09:18:13.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:13.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:13.129554+0000) 2024-03-06T09:18:14.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:14.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:14.129779+0000) 2024-03-06T09:18:15.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:15.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:15.130048+0000) 2024-03-06T09:18:16.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:16.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:16.130235+0000) 2024-03-06T09:18:17.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:17.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:17.130431+0000) 2024-03-06T09:18:18.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:18.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:18.130655+0000) 2024-03-06T09:18:19.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:19.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:19.130920+0000) 2024-03-06T09:18:20.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:20.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:20.131207+0000) 2024-03-06T09:18:21.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:21.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:21.131402+0000) 2024-03-06T09:18:22.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:22.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:22.131641+0000) 2024-03-06T09:18:23.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:23.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:23.131882+0000) 2024-03-06T09:18:24.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:24.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:24.132127+0000) 2024-03-06T09:18:25.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:25.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:25.132366+0000) 2024-03-06T09:18:26.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:26.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:26.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:26.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:26.132603+0000) 2024-03-06T09:18:27.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:27.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:27.132827+0000) 2024-03-06T09:18:28.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:28.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:28.133096+0000) 2024-03-06T09:18:29.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:29.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:29.133350+0000) 2024-03-06T09:18:30.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:30.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:30.133608+0000) 2024-03-06T09:18:31.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:31.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:31.133848+0000) 2024-03-06T09:18:32.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:32.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:32.134099+0000) 2024-03-06T09:18:33.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:33.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:33.134326+0000) 2024-03-06T09:18:34.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:34.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:34.134555+0000) 2024-03-06T09:18:35.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:35.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:35.134784+0000) 2024-03-06T09:18:36.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:36.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:36.134973+0000) 2024-03-06T09:18:37.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:37.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:37.135203+0000) 2024-03-06T09:18:38.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:38.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:38.135433+0000) 2024-03-06T09:18:39.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:39.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:39.135651+0000) 2024-03-06T09:18:40.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:40.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:40.135943+0000) 2024-03-06T09:18:41.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:41.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:41.136179+0000) 2024-03-06T09:18:42.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:42.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:42.136341+0000) 2024-03-06T09:18:43.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:43.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:43.136567+0000) 2024-03-06T09:18:44.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:44.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:44.136790+0000) 2024-03-06T09:18:45.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:45.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:45.137039+0000) 2024-03-06T09:18:46.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:46.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:46.137263+0000) 2024-03-06T09:18:47.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:47.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:47.137482+0000) 2024-03-06T09:18:48.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:48.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:48.137707+0000) 2024-03-06T09:18:49.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:49.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:49.137982+0000) 2024-03-06T09:18:50.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:50.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:50.138292+0000) 2024-03-06T09:18:51.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:51.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:51.138515+0000) 2024-03-06T09:18:52.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:52.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:52.138742+0000) 2024-03-06T09:18:53.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:53.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:53.138969+0000) 2024-03-06T09:18:54.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:54.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:54.139162+0000) 2024-03-06T09:18:55.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:55.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:55.139369+0000) 2024-03-06T09:18:56.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:56.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:56.139612+0000) 2024-03-06T09:18:56.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:56.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:18:57.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:57.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:57.139842+0000) 2024-03-06T09:18:58.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:58.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:58.140108+0000) 2024-03-06T09:18:59.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:18:59.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:18:59.140347+0000) 2024-03-06T09:19:00.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:00.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:00.140579+0000) 2024-03-06T09:19:01.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:01.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:01.140804+0000) 2024-03-06T09:19:02.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:02.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:02.141060+0000) 2024-03-06T09:19:03.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:03.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:03.141288+0000) 2024-03-06T09:19:04.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:04.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:04.141507+0000) 2024-03-06T09:19:05.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:05.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:05.141780+0000) 2024-03-06T09:19:06.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:06.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:06.142105+0000) 2024-03-06T09:19:07.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:07.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:07.142369+0000) 2024-03-06T09:19:08.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:08.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:08.142555+0000) 2024-03-06T09:19:09.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:09.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:09.142756+0000) 2024-03-06T09:19:10.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:10.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:10.142986+0000) 2024-03-06T09:19:11.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:11.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:11.143243+0000) 2024-03-06T09:19:12.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:12.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:12.143458+0000) 2024-03-06T09:19:13.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:13.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:13.143677+0000) 2024-03-06T09:19:14.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:14.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:14.143876+0000) 2024-03-06T09:19:15.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:15.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:15.144127+0000) 2024-03-06T09:19:16.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:16.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:16.144359+0000) 2024-03-06T09:19:17.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:17.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:17.144605+0000) 2024-03-06T09:19:18.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:18.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:18.144829+0000) 2024-03-06T09:19:19.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:19.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:19.145110+0000) 2024-03-06T09:19:20.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:20.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:20.145350+0000) 2024-03-06T09:19:21.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:21.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:21.145579+0000) 2024-03-06T09:19:22.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:22.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:22.145724+0000) 2024-03-06T09:19:23.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:23.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:23.145918+0000) 2024-03-06T09:19:24.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:24.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:24.146201+0000) 2024-03-06T09:19:25.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:25.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:25.146389+0000) 2024-03-06T09:19:26.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:26.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:26.146647+0000) 2024-03-06T09:19:26.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:26.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:27.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:27.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:27.146884+0000) 2024-03-06T09:19:28.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:28.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:28.147051+0000) 2024-03-06T09:19:29.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:29.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:29.147251+0000) 2024-03-06T09:19:30.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:30.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:30.147471+0000) 2024-03-06T09:19:31.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:31.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:31.147695+0000) 2024-03-06T09:19:32.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:32.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:32.147918+0000) 2024-03-06T09:19:33.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:33.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:33.148150+0000) 2024-03-06T09:19:34.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:34.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:34.148316+0000) 2024-03-06T09:19:35.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:35.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:35.148525+0000) 2024-03-06T09:19:36.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:36.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:36.148789+0000) 2024-03-06T09:19:37.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:37.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:37.149036+0000) 2024-03-06T09:19:38.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:38.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:38.149266+0000) 2024-03-06T09:19:39.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:39.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:39.149498+0000) 2024-03-06T09:19:40.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:40.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:40.149739+0000) 2024-03-06T09:19:41.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:41.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:41.149963+0000) 2024-03-06T09:19:42.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:42.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:42.150227+0000) 2024-03-06T09:19:43.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:43.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:43.150508+0000) 2024-03-06T09:19:44.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:44.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:44.150783+0000) 2024-03-06T09:19:45.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:45.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:45.151079+0000) 2024-03-06T09:19:46.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:46.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:46.151324+0000) 2024-03-06T09:19:47.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:47.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:47.151553+0000) 2024-03-06T09:19:48.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:48.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:48.151777+0000) 2024-03-06T09:19:49.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:49.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:49.152076+0000) 2024-03-06T09:19:50.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:50.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:50.152320+0000) 2024-03-06T09:19:51.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:51.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:51.152526+0000) 2024-03-06T09:19:52.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:52.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:52.152747+0000) 2024-03-06T09:19:53.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:53.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:53.152970+0000) 2024-03-06T09:19:54.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:54.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:54.153222+0000) 2024-03-06T09:19:55.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:55.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:55.153507+0000) 2024-03-06T09:19:56.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:56.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:56.153798+0000) 2024-03-06T09:19:56.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:56.495 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:19:57.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:57.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:57.154118+0000) 2024-03-06T09:19:58.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:58.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:58.154398+0000) 2024-03-06T09:19:59.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:19:59.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:19:59.154624+0000) 2024-03-06T09:20:00.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:00.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:00.154829+0000) 2024-03-06T09:20:01.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:01.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:01.154997+0000) 2024-03-06T09:20:02.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:02.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:02.155239+0000) 2024-03-06T09:20:03.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:03.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:03.155457+0000) 2024-03-06T09:20:04.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:04.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:04.155713+0000) 2024-03-06T09:20:05.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:05.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:05.155948+0000) 2024-03-06T09:20:06.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:06.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:06.156185+0000) 2024-03-06T09:20:07.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:07.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:07.156379+0000) 2024-03-06T09:20:08.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:08.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:08.156623+0000) 2024-03-06T09:20:09.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:09.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:09.156826+0000) 2024-03-06T09:20:10.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:10.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:10.157100+0000) 2024-03-06T09:20:11.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:11.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:11.157332+0000) 2024-03-06T09:20:12.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:12.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:12.157588+0000) 2024-03-06T09:20:13.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:13.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:13.157873+0000) 2024-03-06T09:20:14.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:14.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:14.158116+0000) 2024-03-06T09:20:15.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:15.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:15.158344+0000) 2024-03-06T09:20:16.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:16.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:16.158577+0000) 2024-03-06T09:20:17.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:17.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:17.158804+0000) 2024-03-06T09:20:18.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:18.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:18.159056+0000) 2024-03-06T09:20:19.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:19.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:19.159298+0000) 2024-03-06T09:20:20.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:20.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:20.159540+0000) 2024-03-06T09:20:21.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:21.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:21.159760+0000) 2024-03-06T09:20:22.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:22.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:22.159985+0000) 2024-03-06T09:20:23.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:23.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:23.160260+0000) 2024-03-06T09:20:24.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:24.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:24.160520+0000) 2024-03-06T09:20:25.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:25.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:25.160755+0000) 2024-03-06T09:20:26.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:26.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:26.160963+0000) 2024-03-06T09:20:26.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:26.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:27.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:27.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:27.161186+0000) 2024-03-06T09:20:28.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:28.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:28.161469+0000) 2024-03-06T09:20:29.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:29.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:29.161720+0000) 2024-03-06T09:20:30.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:30.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:30.161970+0000) 2024-03-06T09:20:31.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:31.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:31.162303+0000) 2024-03-06T09:20:32.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:32.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:32.162576+0000) 2024-03-06T09:20:33.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:33.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:33.162775+0000) 2024-03-06T09:20:34.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:34.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:34.163039+0000) 2024-03-06T09:20:35.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:35.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:35.163280+0000) 2024-03-06T09:20:36.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:36.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:36.163453+0000) 2024-03-06T09:20:37.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:37.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:37.163658+0000) 2024-03-06T09:20:38.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:38.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:38.163880+0000) 2024-03-06T09:20:39.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:39.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:39.164128+0000) 2024-03-06T09:20:40.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:40.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:40.164359+0000) 2024-03-06T09:20:41.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:41.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:41.164575+0000) 2024-03-06T09:20:42.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:42.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:42.164790+0000) 2024-03-06T09:20:43.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:43.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:43.165054+0000) 2024-03-06T09:20:44.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:44.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:44.165304+0000) 2024-03-06T09:20:45.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:45.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:45.165516+0000) 2024-03-06T09:20:46.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:46.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:46.165737+0000) 2024-03-06T09:20:47.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:47.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:47.165965+0000) 2024-03-06T09:20:48.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:48.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:48.166191+0000) 2024-03-06T09:20:49.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:49.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:49.166431+0000) 2024-03-06T09:20:50.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:50.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:50.166664+0000) 2024-03-06T09:20:51.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:51.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:51.166890+0000) 2024-03-06T09:20:52.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:52.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:52.167132+0000) 2024-03-06T09:20:53.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:53.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:53.167356+0000) 2024-03-06T09:20:54.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:54.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:54.167593+0000) 2024-03-06T09:20:55.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:55.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:55.167738+0000) 2024-03-06T09:20:56.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:56.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:56.167990+0000) 2024-03-06T09:20:56.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:56.814 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:20:57.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:57.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:57.168296+0000) 2024-03-06T09:20:58.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:58.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:58.168576+0000) 2024-03-06T09:20:59.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:20:59.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:20:59.168841+0000) 2024-03-06T09:21:00.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:00.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:00.169123+0000) 2024-03-06T09:21:01.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:01.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:01.169321+0000) 2024-03-06T09:21:02.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:02.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:02.169544+0000) 2024-03-06T09:21:03.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:03.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:03.169746+0000) 2024-03-06T09:21:04.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:04.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:04.169978+0000) 2024-03-06T09:21:05.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:05.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:05.170276+0000) 2024-03-06T09:21:06.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:06.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:06.170557+0000) 2024-03-06T09:21:07.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:07.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:07.170844+0000) 2024-03-06T09:21:08.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:08.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:08.171118+0000) 2024-03-06T09:21:09.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:09.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:09.171404+0000) 2024-03-06T09:21:10.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:10.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:10.171662+0000) 2024-03-06T09:21:11.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:11.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:11.171908+0000) 2024-03-06T09:21:12.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:12.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:12.172088+0000) 2024-03-06T09:21:13.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:13.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:13.172310+0000) 2024-03-06T09:21:14.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:14.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:14.172546+0000) 2024-03-06T09:21:15.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:15.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:15.172785+0000) 2024-03-06T09:21:16.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:16.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:16.173090+0000) 2024-03-06T09:21:17.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:17.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:17.173374+0000) 2024-03-06T09:21:18.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:18.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:18.173609+0000) 2024-03-06T09:21:19.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:19.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:19.173932+0000) 2024-03-06T09:21:20.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:20.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:20.174144+0000) 2024-03-06T09:21:21.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:21.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:21.174419+0000) 2024-03-06T09:21:22.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:22.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:22.174635+0000) 2024-03-06T09:21:23.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:23.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:23.174884+0000) 2024-03-06T09:21:24.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:24.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:24.175081+0000) 2024-03-06T09:21:25.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:25.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:25.175325+0000) 2024-03-06T09:21:26.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:26.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:26.175521+0000) 2024-03-06T09:21:26.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:26.966 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:27.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:27.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:27.175752+0000) 2024-03-06T09:21:28.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:28.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:28.175995+0000) 2024-03-06T09:21:29.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:29.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:29.176295+0000) 2024-03-06T09:21:30.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:30.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:30.176558+0000) 2024-03-06T09:21:31.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:31.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:31.176716+0000) 2024-03-06T09:21:32.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:32.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:32.176939+0000) 2024-03-06T09:21:33.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:33.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:33.177189+0000) 2024-03-06T09:21:34.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:34.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:34.177426+0000) 2024-03-06T09:21:35.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:35.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:35.177657+0000) 2024-03-06T09:21:36.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:36.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:36.177875+0000) 2024-03-06T09:21:37.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:37.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:37.178135+0000) 2024-03-06T09:21:38.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:38.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:38.178356+0000) 2024-03-06T09:21:39.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:39.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:39.178594+0000) 2024-03-06T09:21:40.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:40.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:40.178795+0000) 2024-03-06T09:21:41.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:41.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:41.179076+0000) 2024-03-06T09:21:42.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:42.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:42.179323+0000) 2024-03-06T09:21:43.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:43.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:43.179559+0000) 2024-03-06T09:21:44.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:44.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:44.179843+0000) 2024-03-06T09:21:45.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:45.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:45.180163+0000) 2024-03-06T09:21:46.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:46.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:46.180346+0000) 2024-03-06T09:21:47.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:47.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:47.180527+0000) 2024-03-06T09:21:48.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:48.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:48.180751+0000) 2024-03-06T09:21:49.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:49.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:49.181066+0000) 2024-03-06T09:21:50.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:50.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:50.181356+0000) 2024-03-06T09:21:51.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:51.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:51.181583+0000) 2024-03-06T09:21:52.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:52.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:52.181824+0000) 2024-03-06T09:21:53.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:53.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:53.182114+0000) 2024-03-06T09:21:54.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:54.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:54.182366+0000) 2024-03-06T09:21:55.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:55.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:55.182615+0000) 2024-03-06T09:21:56.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:56.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:56.182842+0000) 2024-03-06T09:21:57.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:57.112 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:21:57.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:57.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:57.183069+0000) 2024-03-06T09:21:58.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:58.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:58.183320+0000) 2024-03-06T09:21:59.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:21:59.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:21:59.183542+0000) 2024-03-06T09:22:00.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:00.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:00.183798+0000) 2024-03-06T09:22:01.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:01.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:01.184096+0000) 2024-03-06T09:22:02.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:02.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:02.184339+0000) 2024-03-06T09:22:03.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:03.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:03.184569+0000) 2024-03-06T09:22:04.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:04.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:04.184816+0000) 2024-03-06T09:22:05.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:05.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:05.185103+0000) 2024-03-06T09:22:06.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:06.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:06.185385+0000) 2024-03-06T09:22:07.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:07.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:07.185652+0000) 2024-03-06T09:22:08.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:08.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:08.185932+0000) 2024-03-06T09:22:09.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:09.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:09.186230+0000) 2024-03-06T09:22:10.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:10.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:10.186413+0000) 2024-03-06T09:22:11.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:11.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:11.186634+0000) 2024-03-06T09:22:12.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:12.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:12.186889+0000) 2024-03-06T09:22:13.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:13.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:13.187114+0000) 2024-03-06T09:22:14.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:14.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:14.187360+0000) 2024-03-06T09:22:15.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:15.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:15.187546+0000) 2024-03-06T09:22:16.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:16.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:16.187763+0000) 2024-03-06T09:22:17.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:17.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:17.188074+0000) 2024-03-06T09:22:18.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:18.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:18.188283+0000) 2024-03-06T09:22:19.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:19.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:19.188495+0000) 2024-03-06T09:22:20.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:20.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:20.188720+0000) 2024-03-06T09:22:21.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:21.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:21.188994+0000) 2024-03-06T09:22:22.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:22.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:22.189315+0000) 2024-03-06T09:22:23.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:23.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:23.189510+0000) 2024-03-06T09:22:24.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:24.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:24.189732+0000) 2024-03-06T09:22:25.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:25.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:25.189960+0000) 2024-03-06T09:22:26.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:26.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:26.190226+0000) 2024-03-06T09:22:27.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:27.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:27.190456+0000) 2024-03-06T09:22:27.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:27.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:28.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:28.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:28.190741+0000) 2024-03-06T09:22:29.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:29.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:29.191072+0000) 2024-03-06T09:22:30.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:30.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:30.191298+0000) 2024-03-06T09:22:31.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:31.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:31.191553+0000) 2024-03-06T09:22:32.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:32.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:32.191830+0000) 2024-03-06T09:22:33.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:33.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:33.192185+0000) 2024-03-06T09:22:34.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:34.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:34.192448+0000) 2024-03-06T09:22:35.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:35.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:35.192720+0000) 2024-03-06T09:22:36.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:36.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:36.192997+0000) 2024-03-06T09:22:37.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:37.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:37.193332+0000) 2024-03-06T09:22:38.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:38.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:38.193613+0000) 2024-03-06T09:22:39.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:39.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:39.193886+0000) 2024-03-06T09:22:40.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:40.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:40.194167+0000) 2024-03-06T09:22:41.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:41.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:41.194437+0000) 2024-03-06T09:22:42.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:42.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:42.194656+0000) 2024-03-06T09:22:43.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:43.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:43.194893+0000) 2024-03-06T09:22:44.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:44.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:44.195126+0000) 2024-03-06T09:22:45.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:45.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:45.195393+0000) 2024-03-06T09:22:46.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:46.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:46.195673+0000) 2024-03-06T09:22:47.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:47.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:47.195907+0000) 2024-03-06T09:22:48.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:48.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:48.196150+0000) 2024-03-06T09:22:49.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:49.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:49.196372+0000) 2024-03-06T09:22:50.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:50.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:50.196600+0000) 2024-03-06T09:22:51.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:51.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:51.196871+0000) 2024-03-06T09:22:52.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:52.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:52.197195+0000) 2024-03-06T09:22:53.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:53.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:53.197442+0000) 2024-03-06T09:22:54.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:54.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:54.197641+0000) 2024-03-06T09:22:55.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:55.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:55.197881+0000) 2024-03-06T09:22:56.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:56.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:56.198122+0000) 2024-03-06T09:22:57.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:57.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:57.198412+0000) 2024-03-06T09:22:57.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:57.436 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:22:58.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:58.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:58.198695+0000) 2024-03-06T09:22:59.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:22:59.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:22:59.198974+0000) 2024-03-06T09:23:00.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:00.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:00.199300+0000) 2024-03-06T09:23:01.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:01.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:01.199492+0000) 2024-03-06T09:23:02.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:02.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:02.199656+0000) 2024-03-06T09:23:03.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:03.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:03.199923+0000) 2024-03-06T09:23:04.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:04.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:04.200165+0000) 2024-03-06T09:23:05.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:05.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:05.200436+0000) 2024-03-06T09:23:06.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:06.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:06.200678+0000) 2024-03-06T09:23:07.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:07.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:07.200909+0000) 2024-03-06T09:23:08.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:08.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:08.201137+0000) 2024-03-06T09:23:09.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:09.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:09.201391+0000) 2024-03-06T09:23:10.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:10.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:10.201649+0000) 2024-03-06T09:23:11.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:11.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:11.201834+0000) 2024-03-06T09:23:12.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:12.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:12.202101+0000) 2024-03-06T09:23:13.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:13.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:13.202295+0000) 2024-03-06T09:23:14.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:14.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:14.202523+0000) 2024-03-06T09:23:15.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:15.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:15.202796+0000) 2024-03-06T09:23:16.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:16.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:16.203043+0000) 2024-03-06T09:23:17.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:17.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:17.203264+0000) 2024-03-06T09:23:18.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:18.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:18.203531+0000) 2024-03-06T09:23:19.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:19.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:19.203808+0000) 2024-03-06T09:23:20.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:20.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:20.204106+0000) 2024-03-06T09:23:21.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:21.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:21.204390+0000) 2024-03-06T09:23:22.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:22.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:22.204633+0000) 2024-03-06T09:23:23.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:23.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:23.204906+0000) 2024-03-06T09:23:24.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:24.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:24.205200+0000) 2024-03-06T09:23:25.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:25.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:25.205485+0000) 2024-03-06T09:23:26.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:26.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:26.205767+0000) 2024-03-06T09:23:27.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:27.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:27.206096+0000) 2024-03-06T09:23:27.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:27.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:28.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:28.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:28.206364+0000) 2024-03-06T09:23:29.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:29.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:29.206640+0000) 2024-03-06T09:23:30.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:30.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:30.206931+0000) 2024-03-06T09:23:31.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:31.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:31.207211+0000) 2024-03-06T09:23:32.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:32.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:32.207458+0000) 2024-03-06T09:23:33.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:33.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:33.207674+0000) 2024-03-06T09:23:34.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:34.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:34.207944+0000) 2024-03-06T09:23:35.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:35.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:35.208219+0000) 2024-03-06T09:23:36.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:36.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:36.208501+0000) 2024-03-06T09:23:37.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:37.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:37.208786+0000) 2024-03-06T09:23:38.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:38.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:38.209111+0000) 2024-03-06T09:23:39.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:39.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:39.209350+0000) 2024-03-06T09:23:40.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:40.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:40.209559+0000) 2024-03-06T09:23:41.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:41.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:41.209839+0000) 2024-03-06T09:23:42.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:42.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:42.210157+0000) 2024-03-06T09:23:43.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:43.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:43.210434+0000) 2024-03-06T09:23:44.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:44.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:44.210672+0000) 2024-03-06T09:23:45.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:45.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:45.210951+0000) 2024-03-06T09:23:46.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:46.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:46.211224+0000) 2024-03-06T09:23:47.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:47.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:47.211508+0000) 2024-03-06T09:23:48.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:48.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:48.211793+0000) 2024-03-06T09:23:49.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:49.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:49.211962+0000) 2024-03-06T09:23:50.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:50.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:50.212218+0000) 2024-03-06T09:23:51.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:51.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:51.212437+0000) 2024-03-06T09:23:52.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:52.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:52.212657+0000) 2024-03-06T09:23:53.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:53.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:53.212879+0000) 2024-03-06T09:23:54.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:54.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:54.213146+0000) 2024-03-06T09:23:55.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:55.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:55.213395+0000) 2024-03-06T09:23:56.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:56.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:56.213652+0000) 2024-03-06T09:23:57.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:57.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:57.213869+0000) 2024-03-06T09:23:57.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:57.864 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:23:58.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:58.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:58.214134+0000) 2024-03-06T09:23:59.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:23:59.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:23:59.214380+0000) 2024-03-06T09:24:00.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:00.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:00.214623+0000) 2024-03-06T09:24:01.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:01.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:01.214811+0000) 2024-03-06T09:24:02.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:02.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:02.215072+0000) 2024-03-06T09:24:03.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:03.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:03.215502+0000) 2024-03-06T09:24:04.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:04.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:04.215740+0000) 2024-03-06T09:24:05.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:05.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:05.215982+0000) 2024-03-06T09:24:06.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:06.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:06.216252+0000) 2024-03-06T09:24:07.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:07.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:07.216495+0000) 2024-03-06T09:24:08.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:08.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:08.216736+0000) 2024-03-06T09:24:09.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:09.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:09.216976+0000) 2024-03-06T09:24:10.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:10.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:10.217300+0000) 2024-03-06T09:24:11.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:11.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:11.217563+0000) 2024-03-06T09:24:12.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:12.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:12.217797+0000) 2024-03-06T09:24:13.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:13.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:13.218057+0000) 2024-03-06T09:24:14.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:14.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:14.218290+0000) 2024-03-06T09:24:15.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:15.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:15.218524+0000) 2024-03-06T09:24:16.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:16.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:16.218742+0000) 2024-03-06T09:24:17.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:17.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:17.218967+0000) 2024-03-06T09:24:18.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:18.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:18.219190+0000) 2024-03-06T09:24:19.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:19.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:19.219403+0000) 2024-03-06T09:24:20.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:20.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:20.219626+0000) 2024-03-06T09:24:21.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:21.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:21.219863+0000) 2024-03-06T09:24:22.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:22.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:22.220054+0000) 2024-03-06T09:24:23.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:23.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:23.220284+0000) 2024-03-06T09:24:24.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:24.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:24.220479+0000) 2024-03-06T09:24:25.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:25.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:25.220713+0000) 2024-03-06T09:24:26.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:26.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:26.220935+0000) 2024-03-06T09:24:27.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:27.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:27.221077+0000) 2024-03-06T09:24:28.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:28.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:28.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:28.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:28.221319+0000) 2024-03-06T09:24:29.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:29.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:29.221523+0000) 2024-03-06T09:24:30.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:30.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:30.221783+0000) 2024-03-06T09:24:31.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:31.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:31.222041+0000) 2024-03-06T09:24:32.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:32.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:32.222275+0000) 2024-03-06T09:24:33.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:33.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:33.222500+0000) 2024-03-06T09:24:34.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:34.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:34.222750+0000) 2024-03-06T09:24:35.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:35.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:35.222904+0000) 2024-03-06T09:24:36.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:36.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:36.223108+0000) 2024-03-06T09:24:37.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:37.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:37.223334+0000) 2024-03-06T09:24:38.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:38.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:38.223613+0000) 2024-03-06T09:24:39.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:39.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:39.223834+0000) 2024-03-06T09:24:40.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:40.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:40.224124+0000) 2024-03-06T09:24:41.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:41.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:41.224309+0000) 2024-03-06T09:24:42.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:42.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:42.224477+0000) 2024-03-06T09:24:43.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:43.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:43.224674+0000) 2024-03-06T09:24:44.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:44.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:44.224875+0000) 2024-03-06T09:24:45.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:45.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:45.225143+0000) 2024-03-06T09:24:46.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:46.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:46.225371+0000) 2024-03-06T09:24:47.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:47.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:47.225600+0000) 2024-03-06T09:24:48.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:48.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:48.225821+0000) 2024-03-06T09:24:49.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:49.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:49.226055+0000) 2024-03-06T09:24:50.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:50.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:50.226261+0000) 2024-03-06T09:24:51.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:51.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:51.226497+0000) 2024-03-06T09:24:52.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:52.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:52.226715+0000) 2024-03-06T09:24:53.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:53.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:53.226941+0000) 2024-03-06T09:24:54.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:54.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:54.227162+0000) 2024-03-06T09:24:55.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:55.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:55.227399+0000) 2024-03-06T09:24:56.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:56.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:56.227580+0000) 2024-03-06T09:24:57.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:57.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:57.227810+0000) 2024-03-06T09:24:58.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:58.185 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:24:58.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:58.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:58.227996+0000) 2024-03-06T09:24:59.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:24:59.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:24:59.228230+0000) 2024-03-06T09:25:00.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:00.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:00.228473+0000) 2024-03-06T09:25:01.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:01.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:01.228696+0000) 2024-03-06T09:25:02.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:02.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:02.228877+0000) 2024-03-06T09:25:03.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:03.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:03.229138+0000) 2024-03-06T09:25:04.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:04.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:04.229387+0000) 2024-03-06T09:25:05.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:05.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:05.229643+0000) 2024-03-06T09:25:06.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:06.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:06.229879+0000) 2024-03-06T09:25:07.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:07.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:07.230125+0000) 2024-03-06T09:25:08.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:08.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:08.230351+0000) 2024-03-06T09:25:09.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:09.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:09.230564+0000) 2024-03-06T09:25:10.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:10.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:10.230833+0000) 2024-03-06T09:25:11.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:11.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:11.231116+0000) 2024-03-06T09:25:12.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:12.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:12.231356+0000) 2024-03-06T09:25:13.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:13.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:13.231597+0000) 2024-03-06T09:25:14.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:14.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:14.231836+0000) 2024-03-06T09:25:15.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:15.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:15.232150+0000) 2024-03-06T09:25:16.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:16.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:16.232398+0000) 2024-03-06T09:25:17.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:17.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:17.232639+0000) 2024-03-06T09:25:18.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:18.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:18.232880+0000) 2024-03-06T09:25:19.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:19.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:19.233118+0000) 2024-03-06T09:25:20.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:20.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:20.233368+0000) 2024-03-06T09:25:21.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:21.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:21.233572+0000) 2024-03-06T09:25:22.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:22.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:22.233752+0000) 2024-03-06T09:25:23.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:23.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:23.233980+0000) 2024-03-06T09:25:24.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:24.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:24.234247+0000) 2024-03-06T09:25:25.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:25.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:25.234489+0000) 2024-03-06T09:25:26.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:26.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:26.234797+0000) 2024-03-06T09:25:27.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:27.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:27.235074+0000) 2024-03-06T09:25:28.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:28.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:28.235298+0000) 2024-03-06T09:25:28.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:28.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:29.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:29.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:29.235533+0000) 2024-03-06T09:25:30.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:30.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:30.235802+0000) 2024-03-06T09:25:31.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:31.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:31.236065+0000) 2024-03-06T09:25:32.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:32.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:32.236261+0000) 2024-03-06T09:25:33.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:33.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:33.236491+0000) 2024-03-06T09:25:34.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:34.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:34.236730+0000) 2024-03-06T09:25:35.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:35.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:35.236972+0000) 2024-03-06T09:25:36.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:36.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:36.237275+0000) 2024-03-06T09:25:37.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:37.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:37.237504+0000) 2024-03-06T09:25:38.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:38.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:38.237677+0000) 2024-03-06T09:25:39.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:39.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:39.237910+0000) 2024-03-06T09:25:40.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:40.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:40.238181+0000) 2024-03-06T09:25:41.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:41.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:41.238406+0000) 2024-03-06T09:25:42.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:42.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:42.238627+0000) 2024-03-06T09:25:43.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:43.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:43.238853+0000) 2024-03-06T09:25:44.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:44.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:44.239035+0000) 2024-03-06T09:25:45.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:45.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:45.239236+0000) 2024-03-06T09:25:46.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:46.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:46.239460+0000) 2024-03-06T09:25:47.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:47.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:47.239686+0000) 2024-03-06T09:25:48.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:48.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:48.239912+0000) 2024-03-06T09:25:49.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:49.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:49.240137+0000) 2024-03-06T09:25:50.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:50.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:50.240387+0000) 2024-03-06T09:25:51.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:51.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:51.240578+0000) 2024-03-06T09:25:52.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:52.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:52.240807+0000) 2024-03-06T09:25:53.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:53.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:53.240993+0000) 2024-03-06T09:25:54.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:54.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:54.241286+0000) 2024-03-06T09:25:55.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:55.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:55.241550+0000) 2024-03-06T09:25:56.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:56.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:56.241790+0000) 2024-03-06T09:25:57.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:57.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:57.242056+0000) 2024-03-06T09:25:58.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:58.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:58.242280+0000) 2024-03-06T09:25:58.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:58.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:25:59.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:25:59.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:25:59.242528+0000) 2024-03-06T09:26:00.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:00.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:00.242788+0000) 2024-03-06T09:26:01.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:01.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:01.242968+0000) 2024-03-06T09:26:02.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:02.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:02.243097+0000) 2024-03-06T09:26:03.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:03.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:03.243224+0000) 2024-03-06T09:26:04.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:04.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:04.243386+0000) 2024-03-06T09:26:05.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:05.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:05.243624+0000) 2024-03-06T09:26:06.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:06.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:06.243842+0000) 2024-03-06T09:26:07.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:07.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:07.244116+0000) 2024-03-06T09:26:08.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:08.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:08.244358+0000) 2024-03-06T09:26:09.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:09.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:09.244605+0000) 2024-03-06T09:26:10.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:10.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:10.244874+0000) 2024-03-06T09:26:11.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:11.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:11.245119+0000) 2024-03-06T09:26:12.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:12.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:12.245349+0000) 2024-03-06T09:26:13.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:13.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:13.245589+0000) 2024-03-06T09:26:14.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:14.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:14.245882+0000) 2024-03-06T09:26:15.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:15.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:15.246182+0000) 2024-03-06T09:26:16.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:16.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:16.246421+0000) 2024-03-06T09:26:17.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:17.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:17.246669+0000) 2024-03-06T09:26:18.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:18.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:18.246905+0000) 2024-03-06T09:26:19.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:19.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:19.247151+0000) 2024-03-06T09:26:20.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:20.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:20.247426+0000) 2024-03-06T09:26:21.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:21.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:21.247663+0000) 2024-03-06T09:26:22.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:22.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:22.247918+0000) 2024-03-06T09:26:23.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:23.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:23.248175+0000) 2024-03-06T09:26:24.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:24.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:24.248389+0000) 2024-03-06T09:26:25.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:25.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:25.248645+0000) 2024-03-06T09:26:26.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:26.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:26.248887+0000) 2024-03-06T09:26:27.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:27.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:27.249132+0000) 2024-03-06T09:26:28.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:28.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:28.249340+0000) 2024-03-06T09:26:28.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:28.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:29.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:29.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:29.249542+0000) 2024-03-06T09:26:30.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:30.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:30.249804+0000) 2024-03-06T09:26:31.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:31.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:31.250083+0000) 2024-03-06T09:26:32.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:32.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:32.250310+0000) 2024-03-06T09:26:33.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:33.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:33.250532+0000) 2024-03-06T09:26:34.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:34.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:34.250755+0000) 2024-03-06T09:26:35.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:35.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:35.251046+0000) 2024-03-06T09:26:36.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:36.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:36.251299+0000) 2024-03-06T09:26:37.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:37.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:37.251523+0000) 2024-03-06T09:26:38.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:38.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:38.251743+0000) 2024-03-06T09:26:39.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:39.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:39.251960+0000) 2024-03-06T09:26:40.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:40.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:40.252277+0000) 2024-03-06T09:26:41.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:41.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:41.252456+0000) 2024-03-06T09:26:42.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:42.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:42.252645+0000) 2024-03-06T09:26:43.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:43.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:43.252868+0000) 2024-03-06T09:26:44.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:44.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:44.253126+0000) 2024-03-06T09:26:45.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:45.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:45.253315+0000) 2024-03-06T09:26:46.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:46.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:46.253534+0000) 2024-03-06T09:26:47.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:47.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:47.253727+0000) 2024-03-06T09:26:48.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:48.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:48.253956+0000) 2024-03-06T09:26:49.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:49.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:49.254148+0000) 2024-03-06T09:26:50.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:50.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:50.254396+0000) 2024-03-06T09:26:51.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:51.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:51.254617+0000) 2024-03-06T09:26:52.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:52.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:52.254840+0000) 2024-03-06T09:26:53.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:53.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:53.255110+0000) 2024-03-06T09:26:54.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:54.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:54.255353+0000) 2024-03-06T09:26:55.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:55.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:55.255500+0000) 2024-03-06T09:26:56.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:56.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:56.255717+0000) 2024-03-06T09:26:57.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:57.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:57.255967+0000) 2024-03-06T09:26:58.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:58.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:58.256219+0000) 2024-03-06T09:26:58.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:58.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:26:59.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:26:59.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:26:59.256432+0000) 2024-03-06T09:27:00.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:00.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:00.256675+0000) 2024-03-06T09:27:01.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:01.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:01.256921+0000) 2024-03-06T09:27:02.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:02.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:02.257161+0000) 2024-03-06T09:27:03.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:03.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:03.257409+0000) 2024-03-06T09:27:04.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:04.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:04.257640+0000) 2024-03-06T09:27:05.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:05.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:05.257840+0000) 2024-03-06T09:27:06.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:06.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:06.258032+0000) 2024-03-06T09:27:07.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:07.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:07.258282+0000) 2024-03-06T09:27:08.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:08.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:08.258520+0000) 2024-03-06T09:27:09.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:09.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:09.258739+0000) 2024-03-06T09:27:10.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:10.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:10.259001+0000) 2024-03-06T09:27:11.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:11.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:11.259278+0000) 2024-03-06T09:27:12.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:12.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:12.259495+0000) 2024-03-06T09:27:13.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:13.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:13.259726+0000) 2024-03-06T09:27:14.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:14.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:14.259946+0000) 2024-03-06T09:27:15.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:15.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:15.260179+0000) 2024-03-06T09:27:16.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:16.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:16.260366+0000) 2024-03-06T09:27:17.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:17.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:17.260612+0000) 2024-03-06T09:27:18.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:18.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:18.260849+0000) 2024-03-06T09:27:19.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:19.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:19.261116+0000) 2024-03-06T09:27:20.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:20.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:20.261344+0000) 2024-03-06T09:27:21.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:21.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:21.261570+0000) 2024-03-06T09:27:22.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:22.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:22.261744+0000) 2024-03-06T09:27:23.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:23.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:23.262048+0000) 2024-03-06T09:27:24.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:24.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:24.262290+0000) 2024-03-06T09:27:25.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:25.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:25.262532+0000) 2024-03-06T09:27:26.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:26.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:26.262736+0000) 2024-03-06T09:27:27.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:27.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:27.262958+0000) 2024-03-06T09:27:28.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:28.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:28.263185+0000) 2024-03-06T09:27:29.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:29.092 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:29.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:29.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:29.263424+0000) 2024-03-06T09:27:30.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:30.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:30.263643+0000) 2024-03-06T09:27:31.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:31.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:31.263749+0000) 2024-03-06T09:27:32.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:32.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:32.263916+0000) 2024-03-06T09:27:33.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:33.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:33.264148+0000) 2024-03-06T09:27:34.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:34.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:34.264331+0000) 2024-03-06T09:27:35.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:35.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:35.264593+0000) 2024-03-06T09:27:36.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:36.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:36.264799+0000) 2024-03-06T09:27:37.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:37.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:37.264985+0000) 2024-03-06T09:27:38.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:38.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:38.265253+0000) 2024-03-06T09:27:39.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:39.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:39.265494+0000) 2024-03-06T09:27:40.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:40.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:40.265710+0000) 2024-03-06T09:27:41.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:41.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:41.265935+0000) 2024-03-06T09:27:42.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:42.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:42.266167+0000) 2024-03-06T09:27:43.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:43.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:43.266414+0000) 2024-03-06T09:27:44.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:44.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:44.266653+0000) 2024-03-06T09:27:45.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:45.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:45.266897+0000) 2024-03-06T09:27:46.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:46.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:46.267133+0000) 2024-03-06T09:27:47.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:47.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:47.267311+0000) 2024-03-06T09:27:48.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:48.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:48.267531+0000) 2024-03-06T09:27:49.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:49.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:49.267752+0000) 2024-03-06T09:27:50.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:50.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:50.268001+0000) 2024-03-06T09:27:51.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:51.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:51.268255+0000) 2024-03-06T09:27:52.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:52.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:52.268478+0000) 2024-03-06T09:27:53.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:53.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:53.268704+0000) 2024-03-06T09:27:54.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:54.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:54.268933+0000) 2024-03-06T09:27:55.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:55.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:55.269163+0000) 2024-03-06T09:27:56.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:56.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:56.269445+0000) 2024-03-06T09:27:57.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:57.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:57.269667+0000) 2024-03-06T09:27:58.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:58.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:58.269849+0000) 2024-03-06T09:27:59.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:59.242 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:27:59.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:27:59.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:27:59.270034+0000) 2024-03-06T09:28:00.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:00.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:00.270216+0000) 2024-03-06T09:28:01.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:01.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:01.270449+0000) 2024-03-06T09:28:02.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:02.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:02.270628+0000) 2024-03-06T09:28:03.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:03.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:03.270816+0000) 2024-03-06T09:28:04.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:04.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:04.271087+0000) 2024-03-06T09:28:05.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:05.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:05.271273+0000) 2024-03-06T09:28:06.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:06.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:06.271474+0000) 2024-03-06T09:28:07.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:07.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:07.271700+0000) 2024-03-06T09:28:08.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:08.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:08.271906+0000) 2024-03-06T09:28:09.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:09.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:09.272152+0000) 2024-03-06T09:28:10.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:10.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:10.272366+0000) 2024-03-06T09:28:11.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:11.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:11.272599+0000) 2024-03-06T09:28:12.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:12.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:12.272748+0000) 2024-03-06T09:28:13.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:13.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:13.272966+0000) 2024-03-06T09:28:14.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:14.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:14.273186+0000) 2024-03-06T09:28:15.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:15.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:15.273428+0000) 2024-03-06T09:28:16.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:16.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:16.273651+0000) 2024-03-06T09:28:17.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:17.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:17.273848+0000) 2024-03-06T09:28:18.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:18.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:18.274116+0000) 2024-03-06T09:28:19.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:19.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:19.274395+0000) 2024-03-06T09:28:20.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:20.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:20.274690+0000) 2024-03-06T09:28:21.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:21.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:21.274932+0000) 2024-03-06T09:28:22.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:22.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:22.275164+0000) 2024-03-06T09:28:23.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:23.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:23.275349+0000) 2024-03-06T09:28:24.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:24.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:24.275577+0000) 2024-03-06T09:28:25.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:25.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:25.275817+0000) 2024-03-06T09:28:26.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:26.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:26.276102+0000) 2024-03-06T09:28:27.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:27.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:27.276342+0000) 2024-03-06T09:28:28.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:28.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:28.276589+0000) 2024-03-06T09:28:29.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:29.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:29.276812+0000) 2024-03-06T09:28:29.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:29.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:30.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:30.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:30.276997+0000) 2024-03-06T09:28:31.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:31.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:31.277271+0000) 2024-03-06T09:28:32.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:32.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:32.277475+0000) 2024-03-06T09:28:33.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:33.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:33.277665+0000) 2024-03-06T09:28:34.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:34.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:34.277906+0000) 2024-03-06T09:28:35.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:35.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:35.278187+0000) 2024-03-06T09:28:36.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:36.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:36.278387+0000) 2024-03-06T09:28:37.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:37.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:37.278632+0000) 2024-03-06T09:28:38.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:38.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:38.278871+0000) 2024-03-06T09:28:39.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:39.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:39.279120+0000) 2024-03-06T09:28:40.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:40.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:40.279362+0000) 2024-03-06T09:28:41.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:41.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:41.279622+0000) 2024-03-06T09:28:42.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:42.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:42.279857+0000) 2024-03-06T09:28:43.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:43.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:43.280126+0000) 2024-03-06T09:28:44.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:44.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:44.280411+0000) 2024-03-06T09:28:45.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:45.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:45.280687+0000) 2024-03-06T09:28:46.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:46.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:46.280956+0000) 2024-03-06T09:28:47.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:47.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:47.281246+0000) 2024-03-06T09:28:48.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:48.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:48.281473+0000) 2024-03-06T09:28:49.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:49.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:49.281692+0000) 2024-03-06T09:28:50.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:50.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:50.281930+0000) 2024-03-06T09:28:51.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:51.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:51.282203+0000) 2024-03-06T09:28:52.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:52.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:52.282442+0000) 2024-03-06T09:28:53.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:53.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:53.282665+0000) 2024-03-06T09:28:54.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:54.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:54.282853+0000) 2024-03-06T09:28:55.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:55.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:55.283097+0000) 2024-03-06T09:28:56.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:56.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:56.283314+0000) 2024-03-06T09:28:57.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:57.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:57.283549+0000) 2024-03-06T09:28:58.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:58.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:58.283787+0000) 2024-03-06T09:28:59.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:28:59.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:28:59.284036+0000) 2024-03-06T09:28:59.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:28:59.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:00.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:00.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:00.284258+0000) 2024-03-06T09:29:01.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:01.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:01.284494+0000) 2024-03-06T09:29:02.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:02.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:02.284718+0000) 2024-03-06T09:29:03.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:03.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:03.284897+0000) 2024-03-06T09:29:04.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:04.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:04.285129+0000) 2024-03-06T09:29:05.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:05.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:05.285326+0000) 2024-03-06T09:29:06.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:06.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:06.285556+0000) 2024-03-06T09:29:07.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:07.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:07.285720+0000) 2024-03-06T09:29:08.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:08.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:08.285843+0000) 2024-03-06T09:29:09.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:09.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:09.286103+0000) 2024-03-06T09:29:10.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:10.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:10.286331+0000) 2024-03-06T09:29:11.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:11.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:11.286551+0000) 2024-03-06T09:29:12.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:12.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:12.286772+0000) 2024-03-06T09:29:13.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:13.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:13.287041+0000) 2024-03-06T09:29:14.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:14.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:14.287243+0000) 2024-03-06T09:29:15.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:15.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:15.287492+0000) 2024-03-06T09:29:16.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:16.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:16.287740+0000) 2024-03-06T09:29:17.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:17.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:17.287974+0000) 2024-03-06T09:29:18.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:18.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:18.288272+0000) 2024-03-06T09:29:19.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:19.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:19.288495+0000) 2024-03-06T09:29:20.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:20.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:20.288728+0000) 2024-03-06T09:29:21.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:21.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:21.288965+0000) 2024-03-06T09:29:22.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:22.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:22.289255+0000) 2024-03-06T09:29:23.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:23.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:23.289455+0000) 2024-03-06T09:29:24.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:24.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:24.289674+0000) 2024-03-06T09:29:25.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:25.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:25.289902+0000) 2024-03-06T09:29:26.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:26.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:26.290132+0000) 2024-03-06T09:29:27.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:27.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:27.290328+0000) 2024-03-06T09:29:28.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:28.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:28.290569+0000) 2024-03-06T09:29:29.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:29.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:29.290821+0000) 2024-03-06T09:29:29.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:29.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:29:30.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:30.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:30.291083+0000) 2024-03-06T09:29:31.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:31.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:31.291280+0000) 2024-03-06T09:29:32.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:32.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:32.291501+0000) 2024-03-06T09:29:33.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:33.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:33.291724+0000) 2024-03-06T09:29:34.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:34.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:34.291946+0000) 2024-03-06T09:29:35.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:35.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:35.292172+0000) 2024-03-06T09:29:36.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:36.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:36.292372+0000) 2024-03-06T09:29:37.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:37.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:37.292674+0000) 2024-03-06T09:29:38.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:38.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:38.292915+0000) 2024-03-06T09:29:39.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:39.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:39.293156+0000) 2024-03-06T09:29:40.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:40.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:40.293385+0000) 2024-03-06T09:29:41.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:41.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:41.293628+0000) 2024-03-06T09:29:42.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:42.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:42.293895+0000) 2024-03-06T09:29:43.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:43.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:43.294141+0000) 2024-03-06T09:29:44.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:44.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:44.294357+0000) 2024-03-06T09:29:45.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:45.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:45.294609+0000) 2024-03-06T09:29:46.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:46.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:46.294845+0000) 2024-03-06T09:29:47.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:47.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:47.295111+0000) 2024-03-06T09:29:48.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:48.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:48.295349+0000) 2024-03-06T09:29:49.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:49.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:49.295572+0000) 2024-03-06T09:29:50.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:50.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:50.295796+0000) 2024-03-06T09:29:51.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:51.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:51.296046+0000) 2024-03-06T09:29:52.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:52.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:52.296303+0000) 2024-03-06T09:29:53.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:53.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:53.296540+0000) 2024-03-06T09:29:54.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:54.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:54.296735+0000) 2024-03-06T09:29:55.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:55.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:55.296986+0000) 2024-03-06T09:29:56.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:56.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:56.297284+0000) 2024-03-06T09:29:57.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:57.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:57.297535+0000) 2024-03-06T09:29:58.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:58.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:58.297772+0000) 2024-03-06T09:29:59.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:29:59.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:29:59.297965+0000) 2024-03-06T09:30:00.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:00.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:00.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:00.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:00.298276+0000) 2024-03-06T09:30:01.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:01.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:01.298500+0000) 2024-03-06T09:30:02.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:02.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:02.298698+0000) 2024-03-06T09:30:03.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:03.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:03.298927+0000) 2024-03-06T09:30:04.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:04.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:04.299149+0000) 2024-03-06T09:30:05.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:05.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:05.299341+0000) 2024-03-06T09:30:06.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:06.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:06.299564+0000) 2024-03-06T09:30:07.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:07.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:07.299784+0000) 2024-03-06T09:30:08.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:08.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:08.299994+0000) 2024-03-06T09:30:09.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:09.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:09.300278+0000) 2024-03-06T09:30:10.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:10.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:10.300496+0000) 2024-03-06T09:30:11.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:11.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:11.300741+0000) 2024-03-06T09:30:12.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:12.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:12.301047+0000) 2024-03-06T09:30:13.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:13.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:13.301313+0000) 2024-03-06T09:30:14.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:14.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:14.301592+0000) 2024-03-06T09:30:15.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:15.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:15.301874+0000) 2024-03-06T09:30:16.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:16.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:16.302190+0000) 2024-03-06T09:30:17.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:17.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:17.302450+0000) 2024-03-06T09:30:18.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:18.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:18.302649+0000) 2024-03-06T09:30:19.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:19.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:19.302890+0000) 2024-03-06T09:30:20.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:20.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:20.303127+0000) 2024-03-06T09:30:21.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:21.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:21.303326+0000) 2024-03-06T09:30:22.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:22.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:22.303552+0000) 2024-03-06T09:30:23.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:23.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:23.303780+0000) 2024-03-06T09:30:24.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:24.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:24.304050+0000) 2024-03-06T09:30:25.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:25.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:25.304298+0000) 2024-03-06T09:30:26.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:26.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:26.304520+0000) 2024-03-06T09:30:27.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:27.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:27.304748+0000) 2024-03-06T09:30:28.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:28.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:28.304956+0000) 2024-03-06T09:30:29.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:29.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:29.305192+0000) 2024-03-06T09:30:30.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:30.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:30.305450+0000) 2024-03-06T09:30:30.321 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:30.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:30:31.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:31.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:31.305706+0000) 2024-03-06T09:30:32.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:32.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:32.305920+0000) 2024-03-06T09:30:33.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:33.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:33.306194+0000) 2024-03-06T09:30:34.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:34.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:34.306458+0000) 2024-03-06T09:30:35.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:35.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:35.306733+0000) 2024-03-06T09:30:36.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:36.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:36.307059+0000) 2024-03-06T09:30:37.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:37.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:37.307302+0000) 2024-03-06T09:30:38.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:38.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:38.307547+0000) 2024-03-06T09:30:39.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:39.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:39.307805+0000) 2024-03-06T09:30:40.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:40.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:40.308127+0000) 2024-03-06T09:30:41.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:41.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:41.308359+0000) 2024-03-06T09:30:42.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:42.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:42.308556+0000) 2024-03-06T09:30:43.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:43.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:43.308807+0000) 2024-03-06T09:30:44.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:44.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:44.309048+0000) 2024-03-06T09:30:45.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:45.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:45.309257+0000) 2024-03-06T09:30:46.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:46.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:46.309462+0000) 2024-03-06T09:30:47.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:47.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:47.309710+0000) 2024-03-06T09:30:48.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:48.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:48.309950+0000) 2024-03-06T09:30:49.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:49.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:49.310199+0000) 2024-03-06T09:30:50.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:50.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:50.310449+0000) 2024-03-06T09:30:51.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:51.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:51.310703+0000) 2024-03-06T09:30:52.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:52.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:52.310951+0000) 2024-03-06T09:30:53.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:53.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:53.311109+0000) 2024-03-06T09:30:54.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:54.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:54.311263+0000) 2024-03-06T09:30:55.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:55.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:55.311516+0000) 2024-03-06T09:30:56.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:56.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:56.311799+0000) 2024-03-06T09:30:57.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:57.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:57.312131+0000) 2024-03-06T09:30:58.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:58.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:58.312367+0000) 2024-03-06T09:30:59.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:30:59.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:30:59.312587+0000) 2024-03-06T09:31:00.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:00.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:00.312832+0000) 2024-03-06T09:31:00.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:00.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:01.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:01.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:01.313126+0000) 2024-03-06T09:31:02.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:02.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:02.313325+0000) 2024-03-06T09:31:03.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:03.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:03.313510+0000) 2024-03-06T09:31:04.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:04.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:04.313731+0000) 2024-03-06T09:31:05.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:05.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:05.313954+0000) 2024-03-06T09:31:06.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:06.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:06.314170+0000) 2024-03-06T09:31:07.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:07.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:07.314380+0000) 2024-03-06T09:31:08.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:08.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:08.314607+0000) 2024-03-06T09:31:09.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:09.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:09.314986+0000) 2024-03-06T09:31:10.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:10.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:10.315313+0000) 2024-03-06T09:31:11.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:11.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:11.315571+0000) 2024-03-06T09:31:12.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:12.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:12.315820+0000) 2024-03-06T09:31:13.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:13.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:13.316101+0000) 2024-03-06T09:31:14.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:14.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:14.316330+0000) 2024-03-06T09:31:15.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:15.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:15.316553+0000) 2024-03-06T09:31:16.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:16.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:16.316798+0000) 2024-03-06T09:31:17.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:17.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:17.317048+0000) 2024-03-06T09:31:18.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:18.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:18.317230+0000) 2024-03-06T09:31:19.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:19.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:19.317444+0000) 2024-03-06T09:31:20.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:20.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:20.317688+0000) 2024-03-06T09:31:21.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:21.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:21.317920+0000) 2024-03-06T09:31:22.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:22.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:22.318138+0000) 2024-03-06T09:31:23.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:23.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:23.318364+0000) 2024-03-06T09:31:24.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:24.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:24.318587+0000) 2024-03-06T09:31:25.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:25.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:25.318809+0000) 2024-03-06T09:31:26.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:26.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:26.319089+0000) 2024-03-06T09:31:27.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:27.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:27.319284+0000) 2024-03-06T09:31:28.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:28.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:28.319505+0000) 2024-03-06T09:31:29.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:29.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:29.319744+0000) 2024-03-06T09:31:30.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:30.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:30.319992+0000) 2024-03-06T09:31:30.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:30.705 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:31:31.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:31.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:31.320269+0000) 2024-03-06T09:31:32.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:32.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:32.320503+0000) 2024-03-06T09:31:33.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:33.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:33.320692+0000) 2024-03-06T09:31:34.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:34.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:34.320877+0000) 2024-03-06T09:31:35.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:35.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:35.321109+0000) 2024-03-06T09:31:36.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:36.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:36.321261+0000) 2024-03-06T09:31:37.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:37.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:37.321484+0000) 2024-03-06T09:31:38.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:38.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:38.321724+0000) 2024-03-06T09:31:39.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:39.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:39.321979+0000) 2024-03-06T09:31:40.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:40.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:40.322249+0000) 2024-03-06T09:31:41.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:41.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:41.322493+0000) 2024-03-06T09:31:42.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:42.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:42.322754+0000) 2024-03-06T09:31:43.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:43.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:43.322991+0000) 2024-03-06T09:31:44.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:44.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:44.323268+0000) 2024-03-06T09:31:45.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:45.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:45.323445+0000) 2024-03-06T09:31:46.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:46.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:46.323680+0000) 2024-03-06T09:31:47.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:47.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:47.323915+0000) 2024-03-06T09:31:48.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:48.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:48.324132+0000) 2024-03-06T09:31:49.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:49.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:49.324357+0000) 2024-03-06T09:31:50.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:50.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:50.324594+0000) 2024-03-06T09:31:51.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:51.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:51.324834+0000) 2024-03-06T09:31:52.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:52.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:52.325101+0000) 2024-03-06T09:31:53.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:53.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:53.325326+0000) 2024-03-06T09:31:54.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:54.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:54.325545+0000) 2024-03-06T09:31:55.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:55.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:55.325777+0000) 2024-03-06T09:31:56.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:56.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:56.326049+0000) 2024-03-06T09:31:57.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:57.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:57.326285+0000) 2024-03-06T09:31:58.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:58.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:58.326524+0000) 2024-03-06T09:31:59.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:31:59.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:31:59.326746+0000) 2024-03-06T09:32:00.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:00.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:00.326982+0000) 2024-03-06T09:32:00.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:00.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:01.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:01.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:01.327346+0000) 2024-03-06T09:32:02.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:02.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:02.327630+0000) 2024-03-06T09:32:03.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:03.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:03.327917+0000) 2024-03-06T09:32:04.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:04.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:04.328206+0000) 2024-03-06T09:32:05.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:05.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:05.328487+0000) 2024-03-06T09:32:06.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:06.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:06.328834+0000) 2024-03-06T09:32:07.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:07.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:07.329101+0000) 2024-03-06T09:32:08.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:08.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:08.329365+0000) 2024-03-06T09:32:09.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:09.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:09.329604+0000) 2024-03-06T09:32:10.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:10.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:10.329822+0000) 2024-03-06T09:32:11.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:11.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:11.330155+0000) 2024-03-06T09:32:12.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:12.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:12.330425+0000) 2024-03-06T09:32:13.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:13.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:13.330696+0000) 2024-03-06T09:32:14.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:14.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:14.330938+0000) 2024-03-06T09:32:15.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:15.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:15.331216+0000) 2024-03-06T09:32:16.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:16.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:16.331512+0000) 2024-03-06T09:32:17.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:17.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:17.331798+0000) 2024-03-06T09:32:18.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:18.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:18.332103+0000) 2024-03-06T09:32:19.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:19.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:19.332368+0000) 2024-03-06T09:32:20.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:20.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:20.332605+0000) 2024-03-06T09:32:21.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:21.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:21.332890+0000) 2024-03-06T09:32:22.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:22.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:22.333166+0000) 2024-03-06T09:32:23.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:23.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:23.333445+0000) 2024-03-06T09:32:24.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:24.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:24.333723+0000) 2024-03-06T09:32:25.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:25.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:25.333915+0000) 2024-03-06T09:32:26.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:26.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:26.334164+0000) 2024-03-06T09:32:27.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:27.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:27.334382+0000) 2024-03-06T09:32:28.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:28.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:28.334628+0000) 2024-03-06T09:32:29.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:29.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:29.334829+0000) 2024-03-06T09:32:30.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:30.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:30.335146+0000) 2024-03-06T09:32:30.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:30.989 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:32:31.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:31.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:31.335398+0000) 2024-03-06T09:32:32.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:32.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:32.335536+0000) 2024-03-06T09:32:33.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:33.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:33.335754+0000) 2024-03-06T09:32:34.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:34.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:34.335974+0000) 2024-03-06T09:32:35.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:35.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:35.336200+0000) 2024-03-06T09:32:36.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:36.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:36.336429+0000) 2024-03-06T09:32:37.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:37.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:37.336698+0000) 2024-03-06T09:32:38.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:38.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:38.336966+0000) 2024-03-06T09:32:39.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:39.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:39.337297+0000) 2024-03-06T09:32:40.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:40.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:40.337587+0000) 2024-03-06T09:32:41.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:41.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:41.337873+0000) 2024-03-06T09:32:42.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:42.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:42.338168+0000) 2024-03-06T09:32:43.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:43.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:43.338366+0000) 2024-03-06T09:32:44.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:44.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:44.338610+0000) 2024-03-06T09:32:45.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:45.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:45.338857+0000) 2024-03-06T09:32:46.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:46.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:46.339140+0000) 2024-03-06T09:32:47.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:47.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:47.339357+0000) 2024-03-06T09:32:48.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:48.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:48.339593+0000) 2024-03-06T09:32:49.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:49.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:49.339848+0000) 2024-03-06T09:32:50.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:50.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:50.340132+0000) 2024-03-06T09:32:51.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:51.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:51.340381+0000) 2024-03-06T09:32:52.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:52.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:52.340623+0000) 2024-03-06T09:32:53.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:53.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:53.340865+0000) 2024-03-06T09:32:54.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:54.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:54.341222+0000) 2024-03-06T09:32:55.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:55.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:55.341449+0000) 2024-03-06T09:32:56.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:56.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:56.341716+0000) 2024-03-06T09:32:57.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:57.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:57.341944+0000) 2024-03-06T09:32:58.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:58.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:58.342157+0000) 2024-03-06T09:32:59.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:32:59.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:32:59.342379+0000) 2024-03-06T09:33:00.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:00.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:00.342613+0000) 2024-03-06T09:33:01.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:01.127 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:01.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:01.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:01.342819+0000) 2024-03-06T09:33:02.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:02.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:02.343102+0000) 2024-03-06T09:33:03.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:03.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:03.343349+0000) 2024-03-06T09:33:04.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:04.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:04.343572+0000) 2024-03-06T09:33:05.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:05.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:05.343812+0000) 2024-03-06T09:33:06.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:06.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:06.344106+0000) 2024-03-06T09:33:07.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:07.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:07.344316+0000) 2024-03-06T09:33:08.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:08.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:08.344479+0000) 2024-03-06T09:33:09.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:09.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:09.344676+0000) 2024-03-06T09:33:10.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:10.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:10.344892+0000) 2024-03-06T09:33:11.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:11.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:11.345136+0000) 2024-03-06T09:33:12.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:12.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:12.345346+0000) 2024-03-06T09:33:13.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:13.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:13.345569+0000) 2024-03-06T09:33:14.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:14.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:14.345791+0000) 2024-03-06T09:33:15.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:15.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:15.346051+0000) 2024-03-06T09:33:16.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:16.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:16.346311+0000) 2024-03-06T09:33:17.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:17.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:17.346578+0000) 2024-03-06T09:33:18.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:18.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:18.346762+0000) 2024-03-06T09:33:19.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:19.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:19.346950+0000) 2024-03-06T09:33:20.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:20.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:20.347163+0000) 2024-03-06T09:33:21.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:21.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:21.347389+0000) 2024-03-06T09:33:22.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:22.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:22.347669+0000) 2024-03-06T09:33:23.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:23.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:23.347914+0000) 2024-03-06T09:33:24.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:24.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:24.348173+0000) 2024-03-06T09:33:25.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:25.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:25.348399+0000) 2024-03-06T09:33:26.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:26.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:26.348630+0000) 2024-03-06T09:33:27.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:27.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:27.348854+0000) 2024-03-06T09:33:28.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:28.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:28.349127+0000) 2024-03-06T09:33:29.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:29.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:29.349337+0000) 2024-03-06T09:33:30.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:30.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:30.349501+0000) 2024-03-06T09:33:31.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:31.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:33:31.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:31.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:31.349716+0000) 2024-03-06T09:33:32.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:32.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:32.349918+0000) 2024-03-06T09:33:33.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:33.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:33.350118+0000) 2024-03-06T09:33:34.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:34.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:34.350342+0000) 2024-03-06T09:33:35.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:35.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:35.350564+0000) 2024-03-06T09:33:36.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:36.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:36.350784+0000) 2024-03-06T09:33:37.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:37.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:37.351086+0000) 2024-03-06T09:33:38.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:38.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:38.351317+0000) 2024-03-06T09:33:39.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:39.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:39.351556+0000) 2024-03-06T09:33:40.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:40.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:40.351795+0000) 2024-03-06T09:33:41.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:41.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:41.352046+0000) 2024-03-06T09:33:42.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:42.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:42.352334+0000) 2024-03-06T09:33:43.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:43.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:43.352556+0000) 2024-03-06T09:33:44.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:44.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:44.352771+0000) 2024-03-06T09:33:45.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:45.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:45.353092+0000) 2024-03-06T09:33:46.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:46.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:46.353341+0000) 2024-03-06T09:33:47.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:47.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:47.353556+0000) 2024-03-06T09:33:48.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:48.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:48.353786+0000) 2024-03-06T09:33:49.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:49.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:49.354037+0000) 2024-03-06T09:33:50.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:50.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:50.354327+0000) 2024-03-06T09:33:51.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:51.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:51.354566+0000) 2024-03-06T09:33:52.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:52.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:52.354796+0000) 2024-03-06T09:33:53.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:53.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:53.355060+0000) 2024-03-06T09:33:54.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:54.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:54.355298+0000) 2024-03-06T09:33:55.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:55.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:55.355549+0000) 2024-03-06T09:33:56.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:56.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:56.355774+0000) 2024-03-06T09:33:57.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:57.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:57.356050+0000) 2024-03-06T09:33:58.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:58.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:58.356257+0000) 2024-03-06T09:33:59.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:33:59.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:33:59.356460+0000) 2024-03-06T09:34:00.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:00.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:00.356676+0000) 2024-03-06T09:34:01.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:01.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:01.356895+0000) 2024-03-06T09:34:01.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:01.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:02.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:02.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:02.357135+0000) 2024-03-06T09:34:03.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:03.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:03.357350+0000) 2024-03-06T09:34:04.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:04.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:04.357606+0000) 2024-03-06T09:34:05.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:05.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:05.357827+0000) 2024-03-06T09:34:06.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:06.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:06.358102+0000) 2024-03-06T09:34:07.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:07.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:07.358341+0000) 2024-03-06T09:34:08.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:08.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:08.358564+0000) 2024-03-06T09:34:09.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:09.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:09.358767+0000) 2024-03-06T09:34:10.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:10.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:10.359047+0000) 2024-03-06T09:34:11.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:11.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:11.359305+0000) 2024-03-06T09:34:12.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:12.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:12.359525+0000) 2024-03-06T09:34:13.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:13.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:13.359712+0000) 2024-03-06T09:34:14.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:14.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:14.359938+0000) 2024-03-06T09:34:15.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:15.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:15.360170+0000) 2024-03-06T09:34:16.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:16.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:16.360384+0000) 2024-03-06T09:34:17.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:17.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:17.360608+0000) 2024-03-06T09:34:18.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:18.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:18.360831+0000) 2024-03-06T09:34:19.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:19.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:19.361106+0000) 2024-03-06T09:34:20.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:20.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:20.361290+0000) 2024-03-06T09:34:21.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:21.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:21.361513+0000) 2024-03-06T09:34:22.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:22.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:22.361737+0000) 2024-03-06T09:34:23.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:23.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:23.361958+0000) 2024-03-06T09:34:24.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:24.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:24.362160+0000) 2024-03-06T09:34:25.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:25.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:25.362392+0000) 2024-03-06T09:34:26.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:26.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:26.362613+0000) 2024-03-06T09:34:27.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:27.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:27.362841+0000) 2024-03-06T09:34:28.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:28.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:28.363118+0000) 2024-03-06T09:34:29.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:29.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:29.363350+0000) 2024-03-06T09:34:30.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:30.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:30.363588+0000) 2024-03-06T09:34:31.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:31.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:31.363813+0000) 2024-03-06T09:34:31.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:31.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:34:32.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:32.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:32.364076+0000) 2024-03-06T09:34:33.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:33.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:33.364302+0000) 2024-03-06T09:34:34.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:34.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:34.364521+0000) 2024-03-06T09:34:35.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:35.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:35.364747+0000) 2024-03-06T09:34:36.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:36.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:36.365060+0000) 2024-03-06T09:34:37.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:37.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:37.365294+0000) 2024-03-06T09:34:38.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:38.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:38.365532+0000) 2024-03-06T09:34:39.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:39.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:39.365759+0000) 2024-03-06T09:34:40.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:40.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:40.365999+0000) 2024-03-06T09:34:41.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:41.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:41.366307+0000) 2024-03-06T09:34:42.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:42.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:42.366529+0000) 2024-03-06T09:34:43.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:43.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:43.366784+0000) 2024-03-06T09:34:44.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:44.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:44.367074+0000) 2024-03-06T09:34:45.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:45.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:45.367318+0000) 2024-03-06T09:34:46.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:46.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:46.367556+0000) 2024-03-06T09:34:47.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:47.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:47.367803+0000) 2024-03-06T09:34:48.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:48.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:48.368072+0000) 2024-03-06T09:34:49.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:49.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:49.368267+0000) 2024-03-06T09:34:50.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:50.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:50.368505+0000) 2024-03-06T09:34:51.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:51.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:51.368745+0000) 2024-03-06T09:34:52.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:52.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:52.368959+0000) 2024-03-06T09:34:53.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:53.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:53.369187+0000) 2024-03-06T09:34:54.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:54.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:54.369406+0000) 2024-03-06T09:34:55.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:55.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:55.369616+0000) 2024-03-06T09:34:56.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:56.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:56.369831+0000) 2024-03-06T09:34:57.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:57.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:57.370122+0000) 2024-03-06T09:34:58.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:58.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:58.370347+0000) 2024-03-06T09:34:59.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:34:59.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:34:59.370535+0000) 2024-03-06T09:35:00.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:00.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:00.370771+0000) 2024-03-06T09:35:01.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:01.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:01.370999+0000) 2024-03-06T09:35:01.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:01.959 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:02.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:02.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:02.371274+0000) 2024-03-06T09:35:03.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:03.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:03.371502+0000) 2024-03-06T09:35:04.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:04.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:04.371726+0000) 2024-03-06T09:35:05.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:05.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:05.371959+0000) 2024-03-06T09:35:06.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:06.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:06.372244+0000) 2024-03-06T09:35:07.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:07.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:07.372482+0000) 2024-03-06T09:35:08.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:08.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:08.372652+0000) 2024-03-06T09:35:09.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:09.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:09.372876+0000) 2024-03-06T09:35:10.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:10.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:10.373111+0000) 2024-03-06T09:35:11.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:11.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:11.373331+0000) 2024-03-06T09:35:12.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:12.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:12.373511+0000) 2024-03-06T09:35:13.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:13.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:13.373736+0000) 2024-03-06T09:35:14.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:14.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:14.374124+0000) 2024-03-06T09:35:15.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:15.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:15.374367+0000) 2024-03-06T09:35:16.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:16.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:16.374596+0000) 2024-03-06T09:35:17.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:17.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:17.374823+0000) 2024-03-06T09:35:18.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:18.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:18.375040+0000) 2024-03-06T09:35:19.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:19.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:19.375268+0000) 2024-03-06T09:35:20.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:20.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:20.375508+0000) 2024-03-06T09:35:21.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:21.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:21.375735+0000) 2024-03-06T09:35:22.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:22.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:22.375999+0000) 2024-03-06T09:35:23.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:23.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:23.376253+0000) 2024-03-06T09:35:24.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:24.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:24.376465+0000) 2024-03-06T09:35:25.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:25.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:25.376702+0000) 2024-03-06T09:35:26.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:26.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:26.376949+0000) 2024-03-06T09:35:27.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:27.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:27.377164+0000) 2024-03-06T09:35:28.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:28.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:28.377408+0000) 2024-03-06T09:35:29.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:29.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:29.377698+0000) 2024-03-06T09:35:30.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:30.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:30.377866+0000) 2024-03-06T09:35:31.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:31.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:31.378120+0000) 2024-03-06T09:35:32.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:32.310 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:35:32.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:32.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:32.378318+0000) 2024-03-06T09:35:33.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:33.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:33.378549+0000) 2024-03-06T09:35:34.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:34.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:34.378768+0000) 2024-03-06T09:35:35.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:35.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:35.378994+0000) 2024-03-06T09:35:36.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:36.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:36.379249+0000) 2024-03-06T09:35:37.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:37.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:37.379479+0000) 2024-03-06T09:35:38.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:38.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:38.379694+0000) 2024-03-06T09:35:39.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:39.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:39.379918+0000) 2024-03-06T09:35:40.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:40.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:40.380238+0000) 2024-03-06T09:35:41.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:41.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:41.380466+0000) 2024-03-06T09:35:42.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:42.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:42.380658+0000) 2024-03-06T09:35:43.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:43.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:43.380902+0000) 2024-03-06T09:35:44.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:44.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:44.381126+0000) 2024-03-06T09:35:45.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:45.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:45.381330+0000) 2024-03-06T09:35:46.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:46.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:46.381543+0000) 2024-03-06T09:35:47.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:47.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:47.381794+0000) 2024-03-06T09:35:48.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:48.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:48.382040+0000) 2024-03-06T09:35:49.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:49.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:49.382263+0000) 2024-03-06T09:35:50.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:50.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:50.382493+0000) 2024-03-06T09:35:51.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:51.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:51.382693+0000) 2024-03-06T09:35:52.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:52.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:52.382839+0000) 2024-03-06T09:35:53.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:53.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:53.383117+0000) 2024-03-06T09:35:54.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:54.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:54.383353+0000) 2024-03-06T09:35:55.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:55.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:55.383584+0000) 2024-03-06T09:35:56.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:56.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:56.383808+0000) 2024-03-06T09:35:57.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:57.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:57.383998+0000) 2024-03-06T09:35:58.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:58.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:58.384219+0000) 2024-03-06T09:35:59.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:35:59.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:35:59.384409+0000) 2024-03-06T09:36:00.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:00.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:00.384626+0000) 2024-03-06T09:36:01.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:01.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:01.384818+0000) 2024-03-06T09:36:02.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:02.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:02.385042+0000) 2024-03-06T09:36:02.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:02.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:03.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:03.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:03.385223+0000) 2024-03-06T09:36:04.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:04.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:04.385411+0000) 2024-03-06T09:36:05.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:05.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:05.385608+0000) 2024-03-06T09:36:06.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:06.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:06.385829+0000) 2024-03-06T09:36:07.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:07.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:07.386100+0000) 2024-03-06T09:36:08.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:08.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:08.386312+0000) 2024-03-06T09:36:09.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:09.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:09.386513+0000) 2024-03-06T09:36:10.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:10.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:10.386752+0000) 2024-03-06T09:36:11.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:11.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:11.386982+0000) 2024-03-06T09:36:12.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:12.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:12.387303+0000) 2024-03-06T09:36:13.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:13.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:13.387525+0000) 2024-03-06T09:36:14.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:14.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:14.387745+0000) 2024-03-06T09:36:15.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:15.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:15.387914+0000) 2024-03-06T09:36:16.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:16.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:16.388142+0000) 2024-03-06T09:36:17.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:17.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:17.388347+0000) 2024-03-06T09:36:18.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:18.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:18.388562+0000) 2024-03-06T09:36:19.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:19.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:19.388809+0000) 2024-03-06T09:36:20.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:20.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:20.389069+0000) 2024-03-06T09:36:21.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:21.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:21.389257+0000) 2024-03-06T09:36:22.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:22.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:22.389472+0000) 2024-03-06T09:36:23.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:23.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:23.389693+0000) 2024-03-06T09:36:24.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:24.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:24.389911+0000) 2024-03-06T09:36:25.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:25.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:25.390140+0000) 2024-03-06T09:36:26.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:26.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:26.390329+0000) 2024-03-06T09:36:27.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:27.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:27.390564+0000) 2024-03-06T09:36:28.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:28.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:28.390792+0000) 2024-03-06T09:36:29.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:29.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:29.391041+0000) 2024-03-06T09:36:30.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:30.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:30.391244+0000) 2024-03-06T09:36:31.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:31.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:31.391485+0000) 2024-03-06T09:36:32.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:32.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:32.391705+0000) 2024-03-06T09:36:32.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:32.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:36:33.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:33.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:33.391934+0000) 2024-03-06T09:36:34.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:34.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:34.392169+0000) 2024-03-06T09:36:35.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:35.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:35.392388+0000) 2024-03-06T09:36:36.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:36.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:36.392561+0000) 2024-03-06T09:36:37.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:37.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:37.392821+0000) 2024-03-06T09:36:38.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:38.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:38.393074+0000) 2024-03-06T09:36:39.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:39.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:39.393291+0000) 2024-03-06T09:36:40.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:40.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:40.393479+0000) 2024-03-06T09:36:41.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:41.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:41.393690+0000) 2024-03-06T09:36:42.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:42.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:42.393920+0000) 2024-03-06T09:36:43.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:43.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:43.394166+0000) 2024-03-06T09:36:44.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:44.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:44.394378+0000) 2024-03-06T09:36:45.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:45.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:45.394598+0000) 2024-03-06T09:36:46.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:46.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:46.394822+0000) 2024-03-06T09:36:47.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:47.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:47.395085+0000) 2024-03-06T09:36:48.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:48.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:48.395269+0000) 2024-03-06T09:36:49.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:49.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:49.395509+0000) 2024-03-06T09:36:50.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:50.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:50.395738+0000) 2024-03-06T09:36:51.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:51.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:51.395974+0000) 2024-03-06T09:36:52.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:52.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:52.396262+0000) 2024-03-06T09:36:53.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:53.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:53.396463+0000) 2024-03-06T09:36:54.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:54.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:54.396687+0000) 2024-03-06T09:36:55.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:55.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:55.396880+0000) 2024-03-06T09:36:56.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:56.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:56.397105+0000) 2024-03-06T09:36:57.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:57.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:57.397338+0000) 2024-03-06T09:36:58.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:58.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:58.397567+0000) 2024-03-06T09:36:59.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:36:59.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:36:59.397812+0000) 2024-03-06T09:37:00.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:00.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:00.398087+0000) 2024-03-06T09:37:01.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:01.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:01.398318+0000) 2024-03-06T09:37:02.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:02.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:02.398561+0000) 2024-03-06T09:37:02.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:02.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:03.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:03.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:03.398787+0000) 2024-03-06T09:37:04.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:04.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:04.399049+0000) 2024-03-06T09:37:05.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:05.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:05.399325+0000) 2024-03-06T09:37:06.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:06.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:06.399553+0000) 2024-03-06T09:37:07.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:07.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:07.399782+0000) 2024-03-06T09:37:08.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:08.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:08.400037+0000) 2024-03-06T09:37:09.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:09.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:09.400275+0000) 2024-03-06T09:37:10.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:10.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:10.400506+0000) 2024-03-06T09:37:11.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:11.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:11.400728+0000) 2024-03-06T09:37:12.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:12.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:12.400960+0000) 2024-03-06T09:37:13.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:13.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:13.401139+0000) 2024-03-06T09:37:14.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:14.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:14.401364+0000) 2024-03-06T09:37:15.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:15.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:15.401590+0000) 2024-03-06T09:37:16.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:16.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:16.401812+0000) 2024-03-06T09:37:17.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:17.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:17.402084+0000) 2024-03-06T09:37:18.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:18.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:18.402320+0000) 2024-03-06T09:37:19.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:19.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:19.402552+0000) 2024-03-06T09:37:20.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:20.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:20.402742+0000) 2024-03-06T09:37:21.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:21.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:21.402979+0000) 2024-03-06T09:37:22.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:22.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:22.403260+0000) 2024-03-06T09:37:23.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:23.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:23.403493+0000) 2024-03-06T09:37:24.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:24.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:24.403715+0000) 2024-03-06T09:37:25.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:25.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:25.403949+0000) 2024-03-06T09:37:26.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:26.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:26.404140+0000) 2024-03-06T09:37:27.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:27.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:27.404351+0000) 2024-03-06T09:37:28.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:28.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:28.404566+0000) 2024-03-06T09:37:29.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:29.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:29.404802+0000) 2024-03-06T09:37:30.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:30.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:30.405056+0000) 2024-03-06T09:37:31.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:31.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:31.405248+0000) 2024-03-06T09:37:32.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:32.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:32.405484+0000) 2024-03-06T09:37:33.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:33.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:37:33.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:33.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:33.405713+0000) 2024-03-06T09:37:34.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:34.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:34.405931+0000) 2024-03-06T09:37:35.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:35.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:35.406163+0000) 2024-03-06T09:37:36.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:36.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:36.406351+0000) 2024-03-06T09:37:37.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:37.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:37.406576+0000) 2024-03-06T09:37:38.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:38.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:38.406766+0000) 2024-03-06T09:37:39.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:39.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:39.406998+0000) 2024-03-06T09:37:40.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:40.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:40.407235+0000) 2024-03-06T09:37:41.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:41.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:41.407450+0000) 2024-03-06T09:37:42.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:42.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:42.407714+0000) 2024-03-06T09:37:43.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:43.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:43.407902+0000) 2024-03-06T09:37:44.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:44.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:44.408136+0000) 2024-03-06T09:37:45.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:45.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:45.408283+0000) 2024-03-06T09:37:46.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:46.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:46.408505+0000) 2024-03-06T09:37:47.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:47.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:47.408778+0000) 2024-03-06T09:37:48.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:48.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:48.409039+0000) 2024-03-06T09:37:49.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:49.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:49.409224+0000) 2024-03-06T09:37:50.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:50.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:50.409459+0000) 2024-03-06T09:37:51.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:51.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:51.409682+0000) 2024-03-06T09:37:52.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:52.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:52.409917+0000) 2024-03-06T09:37:53.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:53.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:53.410194+0000) 2024-03-06T09:37:54.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:54.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:54.410339+0000) 2024-03-06T09:37:55.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:55.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:55.410538+0000) 2024-03-06T09:37:56.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:56.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:56.410750+0000) 2024-03-06T09:37:57.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:57.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:57.411050+0000) 2024-03-06T09:37:58.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:58.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:58.411248+0000) 2024-03-06T09:37:59.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:37:59.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:37:59.411449+0000) 2024-03-06T09:38:00.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:00.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:00.411656+0000) 2024-03-06T09:38:01.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:01.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:01.411874+0000) 2024-03-06T09:38:02.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:02.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:02.412153+0000) 2024-03-06T09:38:03.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:03.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:03.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:03.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:03.412390+0000) 2024-03-06T09:38:04.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:04.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:04.412604+0000) 2024-03-06T09:38:05.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:05.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:05.412828+0000) 2024-03-06T09:38:06.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:06.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:06.413094+0000) 2024-03-06T09:38:07.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:07.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:07.413337+0000) 2024-03-06T09:38:08.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:08.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:08.413510+0000) 2024-03-06T09:38:09.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:09.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:09.413784+0000) 2024-03-06T09:38:10.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:10.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:10.414072+0000) 2024-03-06T09:38:11.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:11.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:11.414368+0000) 2024-03-06T09:38:12.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:12.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:12.414599+0000) 2024-03-06T09:38:13.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:13.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:13.414885+0000) 2024-03-06T09:38:14.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:14.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:14.415144+0000) 2024-03-06T09:38:15.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:15.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:15.415371+0000) 2024-03-06T09:38:16.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:16.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:16.415617+0000) 2024-03-06T09:38:17.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:17.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:17.415846+0000) 2024-03-06T09:38:18.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:18.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:18.416094+0000) 2024-03-06T09:38:19.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:19.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:19.416334+0000) 2024-03-06T09:38:20.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:20.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:20.416638+0000) 2024-03-06T09:38:21.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:21.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:21.416877+0000) 2024-03-06T09:38:22.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:22.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:22.417133+0000) 2024-03-06T09:38:23.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:23.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:23.417327+0000) 2024-03-06T09:38:24.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:24.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:24.417529+0000) 2024-03-06T09:38:25.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:25.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:25.417752+0000) 2024-03-06T09:38:26.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:26.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:26.417976+0000) 2024-03-06T09:38:27.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:27.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:27.418230+0000) 2024-03-06T09:38:28.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:28.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:28.418433+0000) 2024-03-06T09:38:29.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:29.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:29.418700+0000) 2024-03-06T09:38:30.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:30.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:30.418947+0000) 2024-03-06T09:38:31.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:31.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:31.419146+0000) 2024-03-06T09:38:32.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:32.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:32.419409+0000) 2024-03-06T09:38:33.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:33.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:38:33.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:33.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:33.419671+0000) 2024-03-06T09:38:34.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:34.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:34.419926+0000) 2024-03-06T09:38:35.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:35.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:35.420163+0000) 2024-03-06T09:38:36.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:36.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:36.420410+0000) 2024-03-06T09:38:37.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:37.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:37.420652+0000) 2024-03-06T09:38:38.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:38.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:38.420935+0000) 2024-03-06T09:38:39.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:39.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:39.421218+0000) 2024-03-06T09:38:40.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:40.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:40.421509+0000) 2024-03-06T09:38:41.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:41.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:41.421756+0000) 2024-03-06T09:38:42.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:42.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:42.422033+0000) 2024-03-06T09:38:43.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:43.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:43.422256+0000) 2024-03-06T09:38:44.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:44.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:44.422537+0000) 2024-03-06T09:38:45.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:45.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:45.422748+0000) 2024-03-06T09:38:46.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:46.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:46.422967+0000) 2024-03-06T09:38:47.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:47.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:47.423168+0000) 2024-03-06T09:38:48.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:48.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:48.423363+0000) 2024-03-06T09:38:49.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:49.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:49.423582+0000) 2024-03-06T09:38:50.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:50.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:50.423824+0000) 2024-03-06T09:38:51.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:51.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:51.424095+0000) 2024-03-06T09:38:52.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:52.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:52.424349+0000) 2024-03-06T09:38:53.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:53.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:53.424571+0000) 2024-03-06T09:38:54.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:54.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:54.424792+0000) 2024-03-06T09:38:55.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:55.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:55.424965+0000) 2024-03-06T09:38:56.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:56.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:56.425157+0000) 2024-03-06T09:38:57.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:57.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:57.425402+0000) 2024-03-06T09:38:58.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:58.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:58.425628+0000) 2024-03-06T09:38:59.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:38:59.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:38:59.425840+0000) 2024-03-06T09:39:00.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:00.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:00.426115+0000) 2024-03-06T09:39:01.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:01.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:01.426335+0000) 2024-03-06T09:39:02.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:02.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:02.426567+0000) 2024-03-06T09:39:03.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:03.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:03.426796+0000) 2024-03-06T09:39:03.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:03.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:04.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:04.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:04.427051+0000) 2024-03-06T09:39:05.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:05.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:05.427283+0000) 2024-03-06T09:39:06.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:06.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:06.427487+0000) 2024-03-06T09:39:07.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:07.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:07.427725+0000) 2024-03-06T09:39:08.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:08.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:08.427995+0000) 2024-03-06T09:39:09.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:09.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:09.428269+0000) 2024-03-06T09:39:10.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:10.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:10.428488+0000) 2024-03-06T09:39:11.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:11.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:11.428707+0000) 2024-03-06T09:39:12.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:12.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:12.428946+0000) 2024-03-06T09:39:13.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:13.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:13.429197+0000) 2024-03-06T09:39:14.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:14.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:14.429416+0000) 2024-03-06T09:39:15.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:15.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:15.429659+0000) 2024-03-06T09:39:16.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:16.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:16.429864+0000) 2024-03-06T09:39:17.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:17.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:17.430129+0000) 2024-03-06T09:39:18.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:18.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:18.430328+0000) 2024-03-06T09:39:19.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:19.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:19.430549+0000) 2024-03-06T09:39:20.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:20.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:20.430770+0000) 2024-03-06T09:39:21.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:21.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:21.431042+0000) 2024-03-06T09:39:22.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:22.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:22.431294+0000) 2024-03-06T09:39:23.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:23.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:23.431460+0000) 2024-03-06T09:39:24.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:24.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:24.431686+0000) 2024-03-06T09:39:25.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:25.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:25.431892+0000) 2024-03-06T09:39:26.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:26.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:26.432138+0000) 2024-03-06T09:39:27.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:27.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:27.432342+0000) 2024-03-06T09:39:28.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:28.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:28.432558+0000) 2024-03-06T09:39:29.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:29.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:29.432774+0000) 2024-03-06T09:39:30.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:30.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:30.433043+0000) 2024-03-06T09:39:31.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:31.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:31.433267+0000) 2024-03-06T09:39:32.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:32.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:32.433489+0000) 2024-03-06T09:39:33.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:33.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:33.433715+0000) 2024-03-06T09:39:33.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:33.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:39:34.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:34.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:34.433941+0000) 2024-03-06T09:39:35.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:35.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:35.434173+0000) 2024-03-06T09:39:36.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:36.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:36.434389+0000) 2024-03-06T09:39:37.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:37.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:37.434572+0000) 2024-03-06T09:39:38.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:38.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:38.434806+0000) 2024-03-06T09:39:39.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:39.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:39.435075+0000) 2024-03-06T09:39:40.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:40.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:40.435323+0000) 2024-03-06T09:39:41.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:41.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:41.435544+0000) 2024-03-06T09:39:42.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:42.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:42.435762+0000) 2024-03-06T09:39:43.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:43.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:43.435985+0000) 2024-03-06T09:39:44.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:44.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:44.436249+0000) 2024-03-06T09:39:45.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:45.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:45.436498+0000) 2024-03-06T09:39:46.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:46.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:46.436781+0000) 2024-03-06T09:39:47.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:47.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:47.437058+0000) 2024-03-06T09:39:48.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:48.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:48.437279+0000) 2024-03-06T09:39:49.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:49.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:49.437482+0000) 2024-03-06T09:39:50.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:50.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:50.437732+0000) 2024-03-06T09:39:51.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:51.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:51.437976+0000) 2024-03-06T09:39:52.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:52.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:52.438228+0000) 2024-03-06T09:39:53.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:53.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:53.438499+0000) 2024-03-06T09:39:54.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:54.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:54.438870+0000) 2024-03-06T09:39:55.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:55.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:55.439532+0000) 2024-03-06T09:39:56.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:56.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:56.439768+0000) 2024-03-06T09:39:57.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:57.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:57.439999+0000) 2024-03-06T09:39:58.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:58.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:58.440230+0000) 2024-03-06T09:39:59.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:39:59.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:39:59.440450+0000) 2024-03-06T09:40:00.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:00.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:00.440700+0000) 2024-03-06T09:40:01.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:01.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:01.440944+0000) 2024-03-06T09:40:02.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:02.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:02.441165+0000) 2024-03-06T09:40:03.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:03.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:03.441384+0000) 2024-03-06T09:40:04.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:04.172 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:04.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:04.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:04.441621+0000) 2024-03-06T09:40:05.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:05.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:05.441866+0000) 2024-03-06T09:40:06.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:06.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:06.442112+0000) 2024-03-06T09:40:07.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:07.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:07.442306+0000) 2024-03-06T09:40:08.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:08.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:08.442527+0000) 2024-03-06T09:40:09.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:09.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:09.442789+0000) 2024-03-06T09:40:10.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:10.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:10.443069+0000) 2024-03-06T09:40:11.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:11.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:11.443291+0000) 2024-03-06T09:40:12.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:12.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:12.443518+0000) 2024-03-06T09:40:13.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:13.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:13.443767+0000) 2024-03-06T09:40:14.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:14.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:14.444033+0000) 2024-03-06T09:40:15.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:15.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:15.444240+0000) 2024-03-06T09:40:16.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:16.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:16.444461+0000) 2024-03-06T09:40:17.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:17.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:17.444712+0000) 2024-03-06T09:40:18.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:18.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:18.444907+0000) 2024-03-06T09:40:19.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:19.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:19.445149+0000) 2024-03-06T09:40:20.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:20.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:20.445377+0000) 2024-03-06T09:40:21.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:21.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:21.445593+0000) 2024-03-06T09:40:22.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:22.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:22.445865+0000) 2024-03-06T09:40:23.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:23.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:23.446111+0000) 2024-03-06T09:40:24.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:24.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:24.446336+0000) 2024-03-06T09:40:25.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:25.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:25.446544+0000) 2024-03-06T09:40:26.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:26.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:26.446774+0000) 2024-03-06T09:40:27.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:27.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:27.446976+0000) 2024-03-06T09:40:28.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:28.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:28.447210+0000) 2024-03-06T09:40:29.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:29.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:29.447375+0000) 2024-03-06T09:40:30.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:30.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:30.447571+0000) 2024-03-06T09:40:31.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:31.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:31.447859+0000) 2024-03-06T09:40:32.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:32.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:32.448170+0000) 2024-03-06T09:40:33.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:33.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:33.448457+0000) 2024-03-06T09:40:34.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:34.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:40:34.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:34.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:34.448666+0000) 2024-03-06T09:40:35.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:35.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:35.448910+0000) 2024-03-06T09:40:36.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:36.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:36.449179+0000) 2024-03-06T09:40:37.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:37.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:37.449459+0000) 2024-03-06T09:40:38.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:38.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:38.449675+0000) 2024-03-06T09:40:39.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:39.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:39.449900+0000) 2024-03-06T09:40:40.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:40.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:40.450125+0000) 2024-03-06T09:40:41.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:41.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:41.450343+0000) 2024-03-06T09:40:42.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:42.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:42.450592+0000) 2024-03-06T09:40:43.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:43.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:43.450824+0000) 2024-03-06T09:40:44.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:44.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:44.451099+0000) 2024-03-06T09:40:45.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:45.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:45.451343+0000) 2024-03-06T09:40:46.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:46.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:46.451562+0000) 2024-03-06T09:40:47.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:47.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:47.451847+0000) 2024-03-06T09:40:48.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:48.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:48.452160+0000) 2024-03-06T09:40:49.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:49.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:49.452436+0000) 2024-03-06T09:40:50.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:50.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:50.452704+0000) 2024-03-06T09:40:51.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:51.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:51.452969+0000) 2024-03-06T09:40:52.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:52.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:52.453247+0000) 2024-03-06T09:40:53.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:53.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:53.453436+0000) 2024-03-06T09:40:54.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:54.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:54.453665+0000) 2024-03-06T09:40:55.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:55.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:55.453882+0000) 2024-03-06T09:40:56.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:56.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:56.454157+0000) 2024-03-06T09:40:57.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:57.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:57.454430+0000) 2024-03-06T09:40:58.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:58.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:58.454709+0000) 2024-03-06T09:40:59.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:40:59.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:40:59.454986+0000) 2024-03-06T09:41:00.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:00.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:00.455353+0000) 2024-03-06T09:41:01.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:01.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:01.455588+0000) 2024-03-06T09:41:02.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:02.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:02.455810+0000) 2024-03-06T09:41:03.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:03.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:03.456080+0000) 2024-03-06T09:41:04.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:04.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:04.456306+0000) 2024-03-06T09:41:04.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:04.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:05.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:05.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:05.456538+0000) 2024-03-06T09:41:06.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:06.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:06.456766+0000) 2024-03-06T09:41:07.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:07.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:07.457059+0000) 2024-03-06T09:41:08.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:08.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:08.457294+0000) 2024-03-06T09:41:09.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:09.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:09.457556+0000) 2024-03-06T09:41:10.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:10.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:10.457923+0000) 2024-03-06T09:41:11.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:11.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:11.458214+0000) 2024-03-06T09:41:12.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:12.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:12.458492+0000) 2024-03-06T09:41:13.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:13.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:13.458767+0000) 2024-03-06T09:41:14.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:14.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:14.459092+0000) 2024-03-06T09:41:15.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:15.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:15.459367+0000) 2024-03-06T09:41:16.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:16.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:16.459583+0000) 2024-03-06T09:41:17.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:17.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:17.459819+0000) 2024-03-06T09:41:18.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:18.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:18.460091+0000) 2024-03-06T09:41:19.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:19.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:19.460313+0000) 2024-03-06T09:41:20.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:20.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:20.460545+0000) 2024-03-06T09:41:21.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:21.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:21.460742+0000) 2024-03-06T09:41:22.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:22.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:22.460969+0000) 2024-03-06T09:41:23.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:23.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:23.461255+0000) 2024-03-06T09:41:24.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:24.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:24.461472+0000) 2024-03-06T09:41:25.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:25.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:25.461720+0000) 2024-03-06T09:41:26.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:26.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:26.461915+0000) 2024-03-06T09:41:27.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:27.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:27.462155+0000) 2024-03-06T09:41:28.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:28.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:28.462355+0000) 2024-03-06T09:41:29.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:29.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:29.462596+0000) 2024-03-06T09:41:30.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:30.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:30.462906+0000) 2024-03-06T09:41:31.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:31.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:31.463181+0000) 2024-03-06T09:41:32.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:32.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:32.463410+0000) 2024-03-06T09:41:33.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:33.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:33.463659+0000) 2024-03-06T09:41:34.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:34.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:34.463896+0000) 2024-03-06T09:41:34.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:34.779 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:41:35.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:35.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:35.464156+0000) 2024-03-06T09:41:36.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:36.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:36.464374+0000) 2024-03-06T09:41:37.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:37.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:37.464631+0000) 2024-03-06T09:41:38.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:38.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:38.464881+0000) 2024-03-06T09:41:39.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:39.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:39.465122+0000) 2024-03-06T09:41:40.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:40.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:40.465347+0000) 2024-03-06T09:41:41.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:41.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:41.465588+0000) 2024-03-06T09:41:42.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:42.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:42.465780+0000) 2024-03-06T09:41:43.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:43.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:43.466042+0000) 2024-03-06T09:41:44.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:44.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:44.466282+0000) 2024-03-06T09:41:45.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:45.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:45.466508+0000) 2024-03-06T09:41:46.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:46.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:46.466693+0000) 2024-03-06T09:41:47.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:47.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:47.466908+0000) 2024-03-06T09:41:48.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:48.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:48.467139+0000) 2024-03-06T09:41:49.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:49.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:49.467375+0000) 2024-03-06T09:41:50.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:50.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:50.467638+0000) 2024-03-06T09:41:51.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:51.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:51.467861+0000) 2024-03-06T09:41:52.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:52.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:52.468106+0000) 2024-03-06T09:41:53.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:53.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:53.468341+0000) 2024-03-06T09:41:54.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:54.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:54.468516+0000) 2024-03-06T09:41:55.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:55.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:55.468727+0000) 2024-03-06T09:41:56.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:56.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:56.468950+0000) 2024-03-06T09:41:57.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:57.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:57.469149+0000) 2024-03-06T09:41:58.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:58.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:58.469407+0000) 2024-03-06T09:41:59.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:41:59.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:41:59.469628+0000) 2024-03-06T09:42:00.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:00.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:00.469843+0000) 2024-03-06T09:42:01.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:01.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:01.470089+0000) 2024-03-06T09:42:02.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:02.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:02.470315+0000) 2024-03-06T09:42:03.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:03.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:03.470569+0000) 2024-03-06T09:42:04.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:04.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:04.470846+0000) 2024-03-06T09:42:04.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:04.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:05.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:05.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:05.471112+0000) 2024-03-06T09:42:06.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:06.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:06.471309+0000) 2024-03-06T09:42:07.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:07.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:07.471533+0000) 2024-03-06T09:42:08.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:08.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:08.471769+0000) 2024-03-06T09:42:09.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:09.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:09.471995+0000) 2024-03-06T09:42:10.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:10.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:10.472224+0000) 2024-03-06T09:42:11.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:11.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:11.472414+0000) 2024-03-06T09:42:12.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:12.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:12.472629+0000) 2024-03-06T09:42:13.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:13.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:13.472859+0000) 2024-03-06T09:42:14.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:14.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:14.473113+0000) 2024-03-06T09:42:15.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:15.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:15.473330+0000) 2024-03-06T09:42:16.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:16.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:16.473538+0000) 2024-03-06T09:42:17.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:17.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:17.473776+0000) 2024-03-06T09:42:18.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:18.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:18.473992+0000) 2024-03-06T09:42:19.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:19.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:19.474191+0000) 2024-03-06T09:42:20.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:20.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:20.474424+0000) 2024-03-06T09:42:21.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:21.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:21.474642+0000) 2024-03-06T09:42:22.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:22.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:22.474868+0000) 2024-03-06T09:42:23.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:23.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:23.475127+0000) 2024-03-06T09:42:24.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:24.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:24.475349+0000) 2024-03-06T09:42:25.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:25.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:25.475556+0000) 2024-03-06T09:42:26.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:26.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:26.475783+0000) 2024-03-06T09:42:27.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:27.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:27.475987+0000) 2024-03-06T09:42:28.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:28.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:28.476315+0000) 2024-03-06T09:42:29.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:29.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:29.476558+0000) 2024-03-06T09:42:30.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:30.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:30.476812+0000) 2024-03-06T09:42:31.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:31.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:31.477097+0000) 2024-03-06T09:42:32.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:32.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:32.477347+0000) 2024-03-06T09:42:33.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:33.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:33.477594+0000) 2024-03-06T09:42:34.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:34.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:34.477838+0000) 2024-03-06T09:42:35.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:35.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:42:35.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:35.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:35.478073+0000) 2024-03-06T09:42:36.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:36.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:36.478267+0000) 2024-03-06T09:42:37.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:37.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:37.478453+0000) 2024-03-06T09:42:38.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:38.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:38.478687+0000) 2024-03-06T09:42:39.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:39.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:39.478918+0000) 2024-03-06T09:42:40.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:40.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:40.479166+0000) 2024-03-06T09:42:41.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:41.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:41.479396+0000) 2024-03-06T09:42:42.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:42.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:42.479587+0000) 2024-03-06T09:42:43.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:43.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:43.479817+0000) 2024-03-06T09:42:44.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:44.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:44.480065+0000) 2024-03-06T09:42:45.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:45.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:45.480299+0000) 2024-03-06T09:42:46.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:46.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:46.480516+0000) 2024-03-06T09:42:47.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:47.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:47.480730+0000) 2024-03-06T09:42:48.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:48.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:48.480974+0000) 2024-03-06T09:42:49.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:49.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:49.481220+0000) 2024-03-06T09:42:50.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:50.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:50.481495+0000) 2024-03-06T09:42:51.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:51.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:51.481768+0000) 2024-03-06T09:42:52.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:52.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:52.481980+0000) 2024-03-06T09:42:53.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:53.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:53.482253+0000) 2024-03-06T09:42:54.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:54.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:54.482519+0000) 2024-03-06T09:42:55.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:55.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:55.482716+0000) 2024-03-06T09:42:56.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:56.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:56.482909+0000) 2024-03-06T09:42:57.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:57.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:57.483180+0000) 2024-03-06T09:42:58.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:58.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:58.483413+0000) 2024-03-06T09:42:59.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:42:59.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:42:59.483651+0000) 2024-03-06T09:43:00.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:00.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:00.483864+0000) 2024-03-06T09:43:01.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:01.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:01.484125+0000) 2024-03-06T09:43:02.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:02.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:02.484324+0000) 2024-03-06T09:43:03.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:03.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:03.484558+0000) 2024-03-06T09:43:04.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:04.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:04.484782+0000) 2024-03-06T09:43:05.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:05.340 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:05.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:05.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:05.485082+0000) 2024-03-06T09:43:06.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:06.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:06.485253+0000) 2024-03-06T09:43:07.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:07.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:07.485489+0000) 2024-03-06T09:43:08.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:08.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:08.485752+0000) 2024-03-06T09:43:09.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:09.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:09.486091+0000) 2024-03-06T09:43:10.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:10.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:10.486339+0000) 2024-03-06T09:43:11.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:11.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:11.486621+0000) 2024-03-06T09:43:12.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:12.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:12.486895+0000) 2024-03-06T09:43:13.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:13.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:13.487164+0000) 2024-03-06T09:43:14.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:14.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:14.487430+0000) 2024-03-06T09:43:15.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:15.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:15.487701+0000) 2024-03-06T09:43:16.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:16.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:16.487905+0000) 2024-03-06T09:43:17.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:17.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:17.488160+0000) 2024-03-06T09:43:18.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:18.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:18.488421+0000) 2024-03-06T09:43:19.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:19.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:19.488686+0000) 2024-03-06T09:43:20.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:20.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:20.488933+0000) 2024-03-06T09:43:21.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:21.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:21.489165+0000) 2024-03-06T09:43:22.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:22.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:22.489414+0000) 2024-03-06T09:43:23.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:23.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:23.489674+0000) 2024-03-06T09:43:24.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:24.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:24.489899+0000) 2024-03-06T09:43:25.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:25.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:25.490122+0000) 2024-03-06T09:43:26.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:26.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:26.490348+0000) 2024-03-06T09:43:27.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:27.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:27.490539+0000) 2024-03-06T09:43:28.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:28.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:28.490800+0000) 2024-03-06T09:43:29.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:29.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:29.491090+0000) 2024-03-06T09:43:30.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:30.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:30.491364+0000) 2024-03-06T09:43:31.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:31.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:31.491597+0000) 2024-03-06T09:43:32.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:32.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:32.491805+0000) 2024-03-06T09:43:33.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:33.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:33.492100+0000) 2024-03-06T09:43:34.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:34.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:34.492352+0000) 2024-03-06T09:43:35.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:35.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:35.492591+0000) 2024-03-06T09:43:35.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:35.568 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:43:36.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:36.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:36.492839+0000) 2024-03-06T09:43:37.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:37.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:37.493050+0000) 2024-03-06T09:43:38.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:38.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:38.493264+0000) 2024-03-06T09:43:39.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:39.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:39.493482+0000) 2024-03-06T09:43:40.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:40.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:40.493759+0000) 2024-03-06T09:43:41.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:41.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:41.494094+0000) 2024-03-06T09:43:42.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:42.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:42.494373+0000) 2024-03-06T09:43:43.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:43.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:43.494585+0000) 2024-03-06T09:43:44.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:44.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:44.494814+0000) 2024-03-06T09:43:45.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:45.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:45.495099+0000) 2024-03-06T09:43:46.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:46.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:46.495305+0000) 2024-03-06T09:43:47.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:47.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:47.495500+0000) 2024-03-06T09:43:48.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:48.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:48.495711+0000) 2024-03-06T09:43:49.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:49.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:49.495938+0000) 2024-03-06T09:43:50.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:50.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:50.496243+0000) 2024-03-06T09:43:51.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:51.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:51.496464+0000) 2024-03-06T09:43:52.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:52.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:52.496684+0000) 2024-03-06T09:43:53.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:53.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:53.496967+0000) 2024-03-06T09:43:54.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:54.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:54.497181+0000) 2024-03-06T09:43:55.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:55.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:55.497413+0000) 2024-03-06T09:43:56.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:56.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:56.497626+0000) 2024-03-06T09:43:57.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:57.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:57.497826+0000) 2024-03-06T09:43:58.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:58.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:58.498082+0000) 2024-03-06T09:43:59.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:43:59.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:43:59.498276+0000) 2024-03-06T09:44:00.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:00.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:00.498490+0000) 2024-03-06T09:44:01.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:01.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:01.498709+0000) 2024-03-06T09:44:02.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:02.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:02.498930+0000) 2024-03-06T09:44:03.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:03.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:03.499170+0000) 2024-03-06T09:44:04.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:04.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:04.499363+0000) 2024-03-06T09:44:05.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:05.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:05.499608+0000) 2024-03-06T09:44:05.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:05.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:06.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:06.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:06.499847+0000) 2024-03-06T09:44:07.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:07.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:07.500128+0000) 2024-03-06T09:44:08.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:08.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:08.500362+0000) 2024-03-06T09:44:09.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:09.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:09.500546+0000) 2024-03-06T09:44:10.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:10.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:10.500784+0000) 2024-03-06T09:44:11.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:11.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:11.501045+0000) 2024-03-06T09:44:12.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:12.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:12.501294+0000) 2024-03-06T09:44:13.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:13.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:13.501487+0000) 2024-03-06T09:44:14.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:14.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:14.501670+0000) 2024-03-06T09:44:15.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:15.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:15.501894+0000) 2024-03-06T09:44:16.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:16.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:16.502160+0000) 2024-03-06T09:44:17.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:17.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:17.502416+0000) 2024-03-06T09:44:18.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:18.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:18.502657+0000) 2024-03-06T09:44:19.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:19.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:19.502896+0000) 2024-03-06T09:44:20.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:20.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:20.503179+0000) 2024-03-06T09:44:21.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:21.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:21.503423+0000) 2024-03-06T09:44:22.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:22.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:22.503678+0000) 2024-03-06T09:44:23.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:23.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:23.503919+0000) 2024-03-06T09:44:24.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:24.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:24.504157+0000) 2024-03-06T09:44:25.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:25.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:25.504382+0000) 2024-03-06T09:44:26.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:26.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:26.504604+0000) 2024-03-06T09:44:27.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:27.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:27.504836+0000) 2024-03-06T09:44:28.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:28.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:28.505107+0000) 2024-03-06T09:44:29.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:29.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:29.505359+0000) 2024-03-06T09:44:30.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:30.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:30.505648+0000) 2024-03-06T09:44:31.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:31.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:31.505928+0000) 2024-03-06T09:44:32.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:32.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:32.506198+0000) 2024-03-06T09:44:33.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:33.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:33.506403+0000) 2024-03-06T09:44:34.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:34.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:34.506613+0000) 2024-03-06T09:44:35.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:35.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:35.506830+0000) 2024-03-06T09:44:35.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:35.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:44:36.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:36.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:36.507088+0000) 2024-03-06T09:44:37.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:37.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:37.507277+0000) 2024-03-06T09:44:38.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:38.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:38.507493+0000) 2024-03-06T09:44:39.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:39.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:39.507720+0000) 2024-03-06T09:44:40.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:40.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:40.507949+0000) 2024-03-06T09:44:41.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:41.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:41.508151+0000) 2024-03-06T09:44:42.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:42.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:42.508354+0000) 2024-03-06T09:44:43.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:43.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:43.508574+0000) 2024-03-06T09:44:44.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:44.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:44.508814+0000) 2024-03-06T09:44:45.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:45.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:45.509089+0000) 2024-03-06T09:44:46.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:46.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:46.509279+0000) 2024-03-06T09:44:47.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:47.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:47.509484+0000) 2024-03-06T09:44:48.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:48.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:48.509739+0000) 2024-03-06T09:44:49.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:49.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:49.510059+0000) 2024-03-06T09:44:50.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:50.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:50.510331+0000) 2024-03-06T09:44:51.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:51.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:51.510544+0000) 2024-03-06T09:44:52.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:52.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:52.510767+0000) 2024-03-06T09:44:53.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:53.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:53.511085+0000) 2024-03-06T09:44:54.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:54.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:54.511328+0000) 2024-03-06T09:44:55.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:55.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:55.511527+0000) 2024-03-06T09:44:56.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:56.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:56.511787+0000) 2024-03-06T09:44:57.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:57.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:57.512112+0000) 2024-03-06T09:44:58.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:58.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:58.512389+0000) 2024-03-06T09:44:59.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:44:59.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:44:59.512651+0000) 2024-03-06T09:45:00.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:00.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:00.512937+0000) 2024-03-06T09:45:01.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:01.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:01.513211+0000) 2024-03-06T09:45:02.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:02.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:02.513464+0000) 2024-03-06T09:45:03.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:03.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:03.513750+0000) 2024-03-06T09:45:04.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:04.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:04.514063+0000) 2024-03-06T09:45:05.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:05.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:05.514333+0000) 2024-03-06T09:45:06.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:06.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:06.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:06.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:06.514595+0000) 2024-03-06T09:45:07.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:07.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:07.514823+0000) 2024-03-06T09:45:08.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:08.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:08.515090+0000) 2024-03-06T09:45:09.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:09.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:09.515328+0000) 2024-03-06T09:45:10.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:10.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:10.515589+0000) 2024-03-06T09:45:11.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:11.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:11.515879+0000) 2024-03-06T09:45:12.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:12.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:12.516159+0000) 2024-03-06T09:45:13.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:13.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:13.516396+0000) 2024-03-06T09:45:14.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:14.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:14.516660+0000) 2024-03-06T09:45:15.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:15.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:15.516908+0000) 2024-03-06T09:45:16.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:16.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:16.517114+0000) 2024-03-06T09:45:17.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:17.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:17.517304+0000) 2024-03-06T09:45:18.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:18.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:18.517512+0000) 2024-03-06T09:45:19.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:19.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:19.517731+0000) 2024-03-06T09:45:20.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:20.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:20.517951+0000) 2024-03-06T09:45:21.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:21.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:21.518287+0000) 2024-03-06T09:45:22.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:22.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:22.518527+0000) 2024-03-06T09:45:23.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:23.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:23.518808+0000) 2024-03-06T09:45:24.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:24.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:24.519133+0000) 2024-03-06T09:45:25.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:25.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:25.519402+0000) 2024-03-06T09:45:26.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:26.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:26.519678+0000) 2024-03-06T09:45:27.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:27.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:27.519948+0000) 2024-03-06T09:45:28.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:28.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:28.520226+0000) 2024-03-06T09:45:29.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:29.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:29.520493+0000) 2024-03-06T09:45:30.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:30.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:30.520783+0000) 2024-03-06T09:45:31.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:31.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:31.521056+0000) 2024-03-06T09:45:32.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:32.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:32.521307+0000) 2024-03-06T09:45:33.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:33.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:33.521541+0000) 2024-03-06T09:45:34.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:34.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:34.521797+0000) 2024-03-06T09:45:35.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:35.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:35.522081+0000) 2024-03-06T09:45:36.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:36.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:45:36.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:36.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:36.522335+0000) 2024-03-06T09:45:37.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:37.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:37.522598+0000) 2024-03-06T09:45:38.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:38.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:38.522866+0000) 2024-03-06T09:45:39.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:39.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:39.523170+0000) 2024-03-06T09:45:40.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:40.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:40.523478+0000) 2024-03-06T09:45:41.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:41.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:41.523756+0000) 2024-03-06T09:45:42.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:42.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:42.524076+0000) 2024-03-06T09:45:43.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:43.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:43.524365+0000) 2024-03-06T09:45:44.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:44.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:44.524638+0000) 2024-03-06T09:45:45.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:45.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:45.524836+0000) 2024-03-06T09:45:46.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:46.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:46.525079+0000) 2024-03-06T09:45:47.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:47.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:47.525298+0000) 2024-03-06T09:45:48.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:48.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:48.525526+0000) 2024-03-06T09:45:49.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:49.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:49.525762+0000) 2024-03-06T09:45:50.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:50.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:50.526052+0000) 2024-03-06T09:45:51.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:51.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:51.526237+0000) 2024-03-06T09:45:52.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:52.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:52.526450+0000) 2024-03-06T09:45:53.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:53.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:53.526691+0000) 2024-03-06T09:45:54.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:54.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:54.526936+0000) 2024-03-06T09:45:55.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:55.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:55.527164+0000) 2024-03-06T09:45:56.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:56.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:56.527395+0000) 2024-03-06T09:45:57.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:57.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:57.527632+0000) 2024-03-06T09:45:58.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:58.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:58.527851+0000) 2024-03-06T09:45:59.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:45:59.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:45:59.528100+0000) 2024-03-06T09:46:00.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:00.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:00.528320+0000) 2024-03-06T09:46:01.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:01.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:01.528574+0000) 2024-03-06T09:46:02.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:02.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:02.528849+0000) 2024-03-06T09:46:03.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:03.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:03.529123+0000) 2024-03-06T09:46:04.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:04.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:04.529342+0000) 2024-03-06T09:46:05.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:05.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:05.529558+0000) 2024-03-06T09:46:06.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:06.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:06.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:06.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:06.529807+0000) 2024-03-06T09:46:07.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:07.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:07.530097+0000) 2024-03-06T09:46:08.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:08.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:08.530345+0000) 2024-03-06T09:46:09.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:09.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:09.530616+0000) 2024-03-06T09:46:10.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:10.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:10.530851+0000) 2024-03-06T09:46:11.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:11.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:11.531130+0000) 2024-03-06T09:46:12.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:12.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:12.531327+0000) 2024-03-06T09:46:13.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:13.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:13.531560+0000) 2024-03-06T09:46:14.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:14.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:14.531798+0000) 2024-03-06T09:46:15.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:15.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:15.532049+0000) 2024-03-06T09:46:16.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:16.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:16.532270+0000) 2024-03-06T09:46:17.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:17.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:17.532493+0000) 2024-03-06T09:46:18.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:18.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:18.532725+0000) 2024-03-06T09:46:19.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:19.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:19.532955+0000) 2024-03-06T09:46:20.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:20.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:20.533177+0000) 2024-03-06T09:46:21.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:21.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:21.533354+0000) 2024-03-06T09:46:22.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:22.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:22.533586+0000) 2024-03-06T09:46:23.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:23.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:23.533819+0000) 2024-03-06T09:46:24.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:24.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:24.533967+0000) 2024-03-06T09:46:25.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:25.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:25.534163+0000) 2024-03-06T09:46:26.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:26.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:26.534375+0000) 2024-03-06T09:46:27.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:27.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:27.534601+0000) 2024-03-06T09:46:28.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:28.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:28.534826+0000) 2024-03-06T09:46:29.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:29.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:29.535091+0000) 2024-03-06T09:46:30.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:30.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:30.535331+0000) 2024-03-06T09:46:31.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:31.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:31.535535+0000) 2024-03-06T09:46:32.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:32.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:32.535748+0000) 2024-03-06T09:46:33.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:33.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:33.535971+0000) 2024-03-06T09:46:34.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:34.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:34.536179+0000) 2024-03-06T09:46:35.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:35.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:35.536389+0000) 2024-03-06T09:46:36.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:36.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:36.536586+0000) 2024-03-06T09:46:36.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:36.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:46:37.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:37.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:37.536811+0000) 2024-03-06T09:46:38.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:38.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:38.537070+0000) 2024-03-06T09:46:39.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:39.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:39.537265+0000) 2024-03-06T09:46:40.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:40.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:40.537476+0000) 2024-03-06T09:46:41.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:41.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:41.537698+0000) 2024-03-06T09:46:42.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:42.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:42.537922+0000) 2024-03-06T09:46:43.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:43.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:43.538171+0000) 2024-03-06T09:46:44.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:44.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:44.538406+0000) 2024-03-06T09:46:45.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:45.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:45.538641+0000) 2024-03-06T09:46:46.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:46.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:46.538900+0000) 2024-03-06T09:46:47.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:47.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:47.539137+0000) 2024-03-06T09:46:48.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:48.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:48.539365+0000) 2024-03-06T09:46:49.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:49.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:49.539573+0000) 2024-03-06T09:46:50.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:50.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:50.539824+0000) 2024-03-06T09:46:51.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:51.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:51.540070+0000) 2024-03-06T09:46:52.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:52.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:52.540326+0000) 2024-03-06T09:46:53.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:53.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:53.540541+0000) 2024-03-06T09:46:54.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:54.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:54.540801+0000) 2024-03-06T09:46:55.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:55.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:55.541042+0000) 2024-03-06T09:46:56.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:56.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:56.541225+0000) 2024-03-06T09:46:57.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:57.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:57.541445+0000) 2024-03-06T09:46:58.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:58.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:58.541621+0000) 2024-03-06T09:46:59.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:46:59.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:46:59.541835+0000) 2024-03-06T09:47:00.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:00.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:00.542112+0000) 2024-03-06T09:47:01.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:01.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:01.542335+0000) 2024-03-06T09:47:02.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:02.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:02.542566+0000) 2024-03-06T09:47:03.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:03.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:03.542784+0000) 2024-03-06T09:47:04.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:04.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:04.543082+0000) 2024-03-06T09:47:05.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:05.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:05.543321+0000) 2024-03-06T09:47:06.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:06.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:06.543570+0000) 2024-03-06T09:47:06.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:06.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:07.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:07.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:07.543805+0000) 2024-03-06T09:47:08.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:08.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:08.544114+0000) 2024-03-06T09:47:09.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:09.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:09.544345+0000) 2024-03-06T09:47:10.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:10.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:10.544553+0000) 2024-03-06T09:47:11.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:11.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:11.544776+0000) 2024-03-06T09:47:12.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:12.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:12.544997+0000) 2024-03-06T09:47:13.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:13.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:13.545268+0000) 2024-03-06T09:47:14.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:14.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:14.545498+0000) 2024-03-06T09:47:15.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:15.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:15.545701+0000) 2024-03-06T09:47:16.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:16.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:16.545918+0000) 2024-03-06T09:47:17.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:17.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:17.546142+0000) 2024-03-06T09:47:18.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:18.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:18.546333+0000) 2024-03-06T09:47:19.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:19.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:19.546550+0000) 2024-03-06T09:47:20.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:20.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:20.546780+0000) 2024-03-06T09:47:21.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:21.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:21.546995+0000) 2024-03-06T09:47:22.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:22.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:22.547249+0000) 2024-03-06T09:47:23.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:23.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:23.547443+0000) 2024-03-06T09:47:24.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:24.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:24.547648+0000) 2024-03-06T09:47:25.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:25.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:25.547869+0000) 2024-03-06T09:47:26.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:26.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:26.548178+0000) 2024-03-06T09:47:27.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:27.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:27.548425+0000) 2024-03-06T09:47:28.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:28.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:28.548672+0000) 2024-03-06T09:47:29.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:29.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:29.548892+0000) 2024-03-06T09:47:30.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:30.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:30.549142+0000) 2024-03-06T09:47:31.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:31.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:31.549381+0000) 2024-03-06T09:47:32.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:32.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:32.549613+0000) 2024-03-06T09:47:33.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:33.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:33.549836+0000) 2024-03-06T09:47:34.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:34.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:34.550092+0000) 2024-03-06T09:47:35.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:35.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:35.550305+0000) 2024-03-06T09:47:36.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:36.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:36.550500+0000) 2024-03-06T09:47:37.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:37.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:47:37.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:37.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:37.550706+0000) 2024-03-06T09:47:38.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:38.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:38.550944+0000) 2024-03-06T09:47:39.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:39.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:39.551172+0000) 2024-03-06T09:47:40.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:40.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:40.551382+0000) 2024-03-06T09:47:41.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:41.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:41.551555+0000) 2024-03-06T09:47:42.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:42.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:42.551795+0000) 2024-03-06T09:47:43.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:43.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:43.552083+0000) 2024-03-06T09:47:44.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:44.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:44.552300+0000) 2024-03-06T09:47:45.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:45.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:45.552530+0000) 2024-03-06T09:47:46.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:46.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:46.552770+0000) 2024-03-06T09:47:47.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:47.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:47.553038+0000) 2024-03-06T09:47:48.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:48.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:48.553264+0000) 2024-03-06T09:47:49.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:49.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:49.553506+0000) 2024-03-06T09:47:50.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:50.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:50.553772+0000) 2024-03-06T09:47:51.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:51.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:51.554038+0000) 2024-03-06T09:47:52.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:52.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:52.554253+0000) 2024-03-06T09:47:53.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:53.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:53.554472+0000) 2024-03-06T09:47:54.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:54.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:54.554727+0000) 2024-03-06T09:47:55.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:55.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:55.554994+0000) 2024-03-06T09:47:56.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:56.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:56.555322+0000) 2024-03-06T09:47:57.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:57.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:57.555542+0000) 2024-03-06T09:47:58.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:58.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:58.555721+0000) 2024-03-06T09:47:59.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:47:59.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:47:59.555950+0000) 2024-03-06T09:48:00.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:00.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:00.556199+0000) 2024-03-06T09:48:01.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:01.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:01.556411+0000) 2024-03-06T09:48:02.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:02.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:02.556634+0000) 2024-03-06T09:48:03.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:03.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:03.556827+0000) 2024-03-06T09:48:04.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:04.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:04.557111+0000) 2024-03-06T09:48:05.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:05.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:05.557341+0000) 2024-03-06T09:48:06.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:06.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:06.557556+0000) 2024-03-06T09:48:07.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:07.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:07.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:07.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:07.557826+0000) 2024-03-06T09:48:08.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:08.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:08.558134+0000) 2024-03-06T09:48:09.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:09.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:09.558360+0000) 2024-03-06T09:48:10.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:10.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:10.558593+0000) 2024-03-06T09:48:11.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:11.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:11.558777+0000) 2024-03-06T09:48:12.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:12.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:12.558984+0000) 2024-03-06T09:48:13.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:13.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:13.559241+0000) 2024-03-06T09:48:14.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:14.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:14.559482+0000) 2024-03-06T09:48:15.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:15.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:15.559696+0000) 2024-03-06T09:48:16.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:16.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:16.559956+0000) 2024-03-06T09:48:17.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:17.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:17.560176+0000) 2024-03-06T09:48:18.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:18.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:18.560408+0000) 2024-03-06T09:48:19.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:19.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:19.560642+0000) 2024-03-06T09:48:20.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:20.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:20.560890+0000) 2024-03-06T09:48:21.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:21.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:21.561128+0000) 2024-03-06T09:48:22.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:22.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:22.561323+0000) 2024-03-06T09:48:23.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:23.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:23.561551+0000) 2024-03-06T09:48:24.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:24.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:24.561790+0000) 2024-03-06T09:48:25.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:25.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:25.562075+0000) 2024-03-06T09:48:26.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:26.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:26.562297+0000) 2024-03-06T09:48:27.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:27.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:27.562523+0000) 2024-03-06T09:48:28.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:28.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:28.562754+0000) 2024-03-06T09:48:29.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:29.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:29.562987+0000) 2024-03-06T09:48:30.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:30.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:30.563261+0000) 2024-03-06T09:48:31.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:31.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:31.563481+0000) 2024-03-06T09:48:32.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:32.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:32.563712+0000) 2024-03-06T09:48:33.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:33.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:33.563940+0000) 2024-03-06T09:48:34.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:34.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:34.564194+0000) 2024-03-06T09:48:35.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:35.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:35.564427+0000) 2024-03-06T09:48:36.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:36.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:36.564704+0000) 2024-03-06T09:48:37.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:37.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:48:37.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:37.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:37.564946+0000) 2024-03-06T09:48:38.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:38.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:38.565195+0000) 2024-03-06T09:48:39.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:39.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:39.565430+0000) 2024-03-06T09:48:40.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:40.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:40.565647+0000) 2024-03-06T09:48:41.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:41.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:41.565887+0000) 2024-03-06T09:48:42.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:42.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:42.566142+0000) 2024-03-06T09:48:43.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:43.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:43.566388+0000) 2024-03-06T09:48:44.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:44.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:44.566635+0000) 2024-03-06T09:48:45.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:45.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:45.566922+0000) 2024-03-06T09:48:46.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:46.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:46.567194+0000) 2024-03-06T09:48:47.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:47.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:47.567443+0000) 2024-03-06T09:48:48.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:48.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:48.567722+0000) 2024-03-06T09:48:49.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:49.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:49.568083+0000) 2024-03-06T09:48:50.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:50.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:50.568336+0000) 2024-03-06T09:48:51.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:51.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:51.568553+0000) 2024-03-06T09:48:52.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:52.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:52.568760+0000) 2024-03-06T09:48:53.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:53.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:53.568998+0000) 2024-03-06T09:48:54.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:54.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:54.569267+0000) 2024-03-06T09:48:55.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:55.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:55.569513+0000) 2024-03-06T09:48:56.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:56.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:56.569753+0000) 2024-03-06T09:48:57.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:57.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:57.569973+0000) 2024-03-06T09:48:58.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:58.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:58.570194+0000) 2024-03-06T09:48:59.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:48:59.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:48:59.570450+0000) 2024-03-06T09:49:00.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:00.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:00.570715+0000) 2024-03-06T09:49:01.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:01.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:01.570950+0000) 2024-03-06T09:49:02.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:02.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:02.571154+0000) 2024-03-06T09:49:03.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:03.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:03.571347+0000) 2024-03-06T09:49:04.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:04.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:04.571564+0000) 2024-03-06T09:49:05.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:05.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:05.571830+0000) 2024-03-06T09:49:06.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:06.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:06.572132+0000) 2024-03-06T09:49:07.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:07.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:07.572408+0000) 2024-03-06T09:49:07.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:07.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:08.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:08.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:08.572668+0000) 2024-03-06T09:49:09.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:09.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:09.572966+0000) 2024-03-06T09:49:10.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:10.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:10.573252+0000) 2024-03-06T09:49:11.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:11.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:11.573505+0000) 2024-03-06T09:49:12.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:12.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:12.573732+0000) 2024-03-06T09:49:13.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:13.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:13.573958+0000) 2024-03-06T09:49:14.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:14.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:14.574215+0000) 2024-03-06T09:49:15.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:15.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:15.574440+0000) 2024-03-06T09:49:16.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:16.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:16.574673+0000) 2024-03-06T09:49:17.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:17.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:17.574888+0000) 2024-03-06T09:49:18.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:18.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:18.575117+0000) 2024-03-06T09:49:19.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:19.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:19.575346+0000) 2024-03-06T09:49:20.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:20.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:20.575661+0000) 2024-03-06T09:49:21.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:21.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:21.575942+0000) 2024-03-06T09:49:22.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:22.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:22.576212+0000) 2024-03-06T09:49:23.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:23.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:23.576463+0000) 2024-03-06T09:49:24.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:24.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:24.576750+0000) 2024-03-06T09:49:25.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:25.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:25.577047+0000) 2024-03-06T09:49:26.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:26.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:26.577298+0000) 2024-03-06T09:49:27.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:27.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:27.577544+0000) 2024-03-06T09:49:28.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:28.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:28.577806+0000) 2024-03-06T09:49:29.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:29.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:29.578131+0000) 2024-03-06T09:49:30.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:30.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:30.578370+0000) 2024-03-06T09:49:31.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:31.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:31.578574+0000) 2024-03-06T09:49:32.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:32.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:32.578788+0000) 2024-03-06T09:49:33.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:33.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:33.578969+0000) 2024-03-06T09:49:34.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:34.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:34.579182+0000) 2024-03-06T09:49:35.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:35.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:35.579377+0000) 2024-03-06T09:49:36.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:36.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:36.579596+0000) 2024-03-06T09:49:37.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:37.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:37.579818+0000) 2024-03-06T09:49:37.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:37.818 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:49:38.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:38.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:38.580093+0000) 2024-03-06T09:49:39.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:39.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:39.580312+0000) 2024-03-06T09:49:40.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:40.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:40.580525+0000) 2024-03-06T09:49:41.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:41.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:41.580749+0000) 2024-03-06T09:49:42.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:42.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:42.580986+0000) 2024-03-06T09:49:43.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:43.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:43.581275+0000) 2024-03-06T09:49:44.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:44.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:44.581506+0000) 2024-03-06T09:49:45.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:45.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:45.581707+0000) 2024-03-06T09:49:46.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:46.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:46.581934+0000) 2024-03-06T09:49:47.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:47.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:47.582188+0000) 2024-03-06T09:49:48.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:48.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:48.582429+0000) 2024-03-06T09:49:49.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:49.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:49.582681+0000) 2024-03-06T09:49:50.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:50.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:50.582945+0000) 2024-03-06T09:49:51.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:51.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:51.583192+0000) 2024-03-06T09:49:52.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:52.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:52.583428+0000) 2024-03-06T09:49:53.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:53.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:53.583670+0000) 2024-03-06T09:49:54.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:54.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:54.583925+0000) 2024-03-06T09:49:55.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:55.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:55.584162+0000) 2024-03-06T09:49:56.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:56.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:56.584378+0000) 2024-03-06T09:49:57.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:57.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:57.584603+0000) 2024-03-06T09:49:58.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:58.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:58.584857+0000) 2024-03-06T09:49:59.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:49:59.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:49:59.585127+0000) 2024-03-06T09:50:00.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:00.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:00.585319+0000) 2024-03-06T09:50:01.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:01.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:01.585585+0000) 2024-03-06T09:50:02.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:02.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:02.585804+0000) 2024-03-06T09:50:03.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:03.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:03.586101+0000) 2024-03-06T09:50:04.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:04.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:04.586356+0000) 2024-03-06T09:50:05.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:05.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:05.586583+0000) 2024-03-06T09:50:06.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:06.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:06.586820+0000) 2024-03-06T09:50:07.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:07.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:07.587073+0000) 2024-03-06T09:50:07.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:07.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:08.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:08.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:08.587293+0000) 2024-03-06T09:50:09.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:09.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:09.587506+0000) 2024-03-06T09:50:10.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:10.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:10.587763+0000) 2024-03-06T09:50:11.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:11.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:11.588045+0000) 2024-03-06T09:50:12.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:12.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:12.588253+0000) 2024-03-06T09:50:13.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:13.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:13.588478+0000) 2024-03-06T09:50:14.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:14.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:14.588731+0000) 2024-03-06T09:50:15.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:15.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:15.588980+0000) 2024-03-06T09:50:16.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:16.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:16.589235+0000) 2024-03-06T09:50:17.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:17.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:17.589458+0000) 2024-03-06T09:50:18.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:18.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:18.589682+0000) 2024-03-06T09:50:19.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:19.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:19.589928+0000) 2024-03-06T09:50:20.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:20.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:20.590185+0000) 2024-03-06T09:50:21.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:21.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:21.590414+0000) 2024-03-06T09:50:22.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:22.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:22.590642+0000) 2024-03-06T09:50:23.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:23.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:23.590872+0000) 2024-03-06T09:50:24.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:24.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:24.591126+0000) 2024-03-06T09:50:25.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:25.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:25.591441+0000) 2024-03-06T09:50:26.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:26.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:26.591672+0000) 2024-03-06T09:50:27.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:27.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:27.591888+0000) 2024-03-06T09:50:28.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:28.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:28.592159+0000) 2024-03-06T09:50:29.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:29.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:29.592349+0000) 2024-03-06T09:50:30.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:30.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:30.592558+0000) 2024-03-06T09:50:31.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:31.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:31.592786+0000) 2024-03-06T09:50:32.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:32.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:32.593036+0000) 2024-03-06T09:50:33.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:33.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:33.593254+0000) 2024-03-06T09:50:34.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:34.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:34.593481+0000) 2024-03-06T09:50:35.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:35.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:35.593678+0000) 2024-03-06T09:50:36.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:36.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:36.593904+0000) 2024-03-06T09:50:37.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:37.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:37.594172+0000) 2024-03-06T09:50:38.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:38.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:50:38.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:38.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:38.594412+0000) 2024-03-06T09:50:39.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:39.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:39.594633+0000) 2024-03-06T09:50:40.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:40.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:40.594863+0000) 2024-03-06T09:50:41.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:41.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:41.595121+0000) 2024-03-06T09:50:42.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:42.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:42.595317+0000) 2024-03-06T09:50:43.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:43.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:43.595549+0000) 2024-03-06T09:50:44.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:44.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:44.595772+0000) 2024-03-06T09:50:45.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:45.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:45.596049+0000) 2024-03-06T09:50:46.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:46.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:46.596302+0000) 2024-03-06T09:50:47.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:47.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:47.596496+0000) 2024-03-06T09:50:48.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:48.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:48.596742+0000) 2024-03-06T09:50:49.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:49.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:49.596965+0000) 2024-03-06T09:50:50.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:50.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:50.597229+0000) 2024-03-06T09:50:51.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:51.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:51.597455+0000) 2024-03-06T09:50:52.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:52.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:52.597679+0000) 2024-03-06T09:50:53.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:53.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:53.597904+0000) 2024-03-06T09:50:54.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:54.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:54.598127+0000) 2024-03-06T09:50:55.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:55.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:55.598353+0000) 2024-03-06T09:50:56.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:56.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:56.598580+0000) 2024-03-06T09:50:57.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:57.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:57.598805+0000) 2024-03-06T09:50:58.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:58.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:58.599067+0000) 2024-03-06T09:50:59.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:50:59.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:50:59.599296+0000) 2024-03-06T09:51:00.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:00.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:00.599540+0000) 2024-03-06T09:51:01.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:01.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:01.599778+0000) 2024-03-06T09:51:02.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:02.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:02.600036+0000) 2024-03-06T09:51:03.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:03.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:03.600271+0000) 2024-03-06T09:51:04.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:04.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:04.600469+0000) 2024-03-06T09:51:05.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:05.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:05.600720+0000) 2024-03-06T09:51:06.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:06.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:06.600962+0000) 2024-03-06T09:51:07.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:07.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:07.601256+0000) 2024-03-06T09:51:08.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:08.375 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:08.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:08.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:08.601493+0000) 2024-03-06T09:51:09.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:09.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:09.601738+0000) 2024-03-06T09:51:10.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:10.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:10.601948+0000) 2024-03-06T09:51:11.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:11.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:11.602181+0000) 2024-03-06T09:51:12.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:12.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:12.602431+0000) 2024-03-06T09:51:13.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:13.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:13.602667+0000) 2024-03-06T09:51:14.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:14.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:14.602920+0000) 2024-03-06T09:51:15.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:15.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:15.603142+0000) 2024-03-06T09:51:16.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:16.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:16.603356+0000) 2024-03-06T09:51:17.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:17.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:17.603598+0000) 2024-03-06T09:51:18.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:18.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:18.603924+0000) 2024-03-06T09:51:19.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:19.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:19.604160+0000) 2024-03-06T09:51:20.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:20.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:20.604413+0000) 2024-03-06T09:51:21.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:21.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:21.604675+0000) 2024-03-06T09:51:22.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:22.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:22.604953+0000) 2024-03-06T09:51:23.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:23.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:23.605230+0000) 2024-03-06T09:51:24.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:24.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:24.605482+0000) 2024-03-06T09:51:25.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:25.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:25.605730+0000) 2024-03-06T09:51:26.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:26.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:26.605940+0000) 2024-03-06T09:51:27.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:27.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:27.606177+0000) 2024-03-06T09:51:28.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:28.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:28.606442+0000) 2024-03-06T09:51:29.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:29.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:29.606714+0000) 2024-03-06T09:51:30.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:30.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:30.607040+0000) 2024-03-06T09:51:31.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:31.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:31.607326+0000) 2024-03-06T09:51:32.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:32.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:32.607544+0000) 2024-03-06T09:51:33.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:33.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:33.607755+0000) 2024-03-06T09:51:34.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:34.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:34.607991+0000) 2024-03-06T09:51:35.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:35.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:35.608299+0000) 2024-03-06T09:51:36.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:36.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:36.608535+0000) 2024-03-06T09:51:37.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:37.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:37.608756+0000) 2024-03-06T09:51:38.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:38.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:51:38.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:38.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:38.608975+0000) 2024-03-06T09:51:39.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:39.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:39.609170+0000) 2024-03-06T09:51:40.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:40.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:40.609382+0000) 2024-03-06T09:51:41.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:41.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:41.609605+0000) 2024-03-06T09:51:42.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:42.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:42.609849+0000) 2024-03-06T09:51:43.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:43.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:43.610120+0000) 2024-03-06T09:51:44.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:44.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:44.610326+0000) 2024-03-06T09:51:45.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:45.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:45.610571+0000) 2024-03-06T09:51:46.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:46.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:46.610819+0000) 2024-03-06T09:51:47.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:47.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:47.611061+0000) 2024-03-06T09:51:48.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:48.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:48.611259+0000) 2024-03-06T09:51:49.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:49.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:49.611470+0000) 2024-03-06T09:51:50.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:50.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:50.611723+0000) 2024-03-06T09:51:51.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:51.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:51.611957+0000) 2024-03-06T09:51:52.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:52.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:52.612203+0000) 2024-03-06T09:51:53.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:53.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:53.612391+0000) 2024-03-06T09:51:54.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:54.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:54.612606+0000) 2024-03-06T09:51:55.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:55.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:55.612812+0000) 2024-03-06T09:51:56.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:56.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:56.613071+0000) 2024-03-06T09:51:57.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:57.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:57.613308+0000) 2024-03-06T09:51:58.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:58.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:58.613569+0000) 2024-03-06T09:51:59.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:51:59.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:51:59.613810+0000) 2024-03-06T09:52:00.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:00.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:00.614082+0000) 2024-03-06T09:52:01.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:01.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:01.614282+0000) 2024-03-06T09:52:02.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:02.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:02.614500+0000) 2024-03-06T09:52:03.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:03.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:03.614695+0000) 2024-03-06T09:52:04.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:04.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:04.614927+0000) 2024-03-06T09:52:05.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:05.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:05.615170+0000) 2024-03-06T09:52:06.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:06.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:06.615441+0000) 2024-03-06T09:52:07.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:07.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:07.615675+0000) 2024-03-06T09:52:08.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:08.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:08.615894+0000) 2024-03-06T09:52:08.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:08.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:09.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:09.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:09.616139+0000) 2024-03-06T09:52:10.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:10.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:10.616377+0000) 2024-03-06T09:52:11.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:11.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:11.616606+0000) 2024-03-06T09:52:12.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:12.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:12.616834+0000) 2024-03-06T09:52:13.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:13.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:13.617085+0000) 2024-03-06T09:52:14.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:14.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:14.617328+0000) 2024-03-06T09:52:15.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:15.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:15.617527+0000) 2024-03-06T09:52:16.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:16.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:16.617750+0000) 2024-03-06T09:52:17.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:17.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:17.617989+0000) 2024-03-06T09:52:18.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:18.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:18.618246+0000) 2024-03-06T09:52:19.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:19.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:19.618468+0000) 2024-03-06T09:52:20.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:20.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:20.618704+0000) 2024-03-06T09:52:21.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:21.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:21.618929+0000) 2024-03-06T09:52:22.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:22.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:22.619184+0000) 2024-03-06T09:52:23.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:23.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:23.619461+0000) 2024-03-06T09:52:24.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:24.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:24.619731+0000) 2024-03-06T09:52:25.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:25.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:25.620053+0000) 2024-03-06T09:52:26.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:26.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:26.620322+0000) 2024-03-06T09:52:27.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:27.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:27.620570+0000) 2024-03-06T09:52:28.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:28.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:28.620761+0000) 2024-03-06T09:52:29.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:29.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:29.621001+0000) 2024-03-06T09:52:30.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:30.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:30.621240+0000) 2024-03-06T09:52:31.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:31.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:31.621450+0000) 2024-03-06T09:52:32.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:32.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:32.621674+0000) 2024-03-06T09:52:33.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:33.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:33.621935+0000) 2024-03-06T09:52:34.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:34.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:34.622167+0000) 2024-03-06T09:52:35.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:35.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:35.622386+0000) 2024-03-06T09:52:36.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:36.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:36.622622+0000) 2024-03-06T09:52:37.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:37.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:37.622889+0000) 2024-03-06T09:52:38.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:38.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:38.623141+0000) 2024-03-06T09:52:39.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:39.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:52:39.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:39.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:39.623423+0000) 2024-03-06T09:52:40.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:40.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:40.623713+0000) 2024-03-06T09:52:41.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:41.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:41.623981+0000) 2024-03-06T09:52:42.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:42.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:42.624231+0000) 2024-03-06T09:52:43.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:43.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:43.624459+0000) 2024-03-06T09:52:44.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:44.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:44.624724+0000) 2024-03-06T09:52:45.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:45.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:45.624993+0000) 2024-03-06T09:52:46.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:46.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:46.625283+0000) 2024-03-06T09:52:47.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:47.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:47.625476+0000) 2024-03-06T09:52:48.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:48.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:48.625726+0000) 2024-03-06T09:52:49.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:49.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:49.625976+0000) 2024-03-06T09:52:50.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:50.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:50.626247+0000) 2024-03-06T09:52:51.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:51.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:51.626471+0000) 2024-03-06T09:52:52.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:52.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:52.626713+0000) 2024-03-06T09:52:53.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:53.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:53.626958+0000) 2024-03-06T09:52:54.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:54.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:54.627290+0000) 2024-03-06T09:52:55.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:55.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:55.627570+0000) 2024-03-06T09:52:56.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:56.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:56.627857+0000) 2024-03-06T09:52:57.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:57.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:57.628159+0000) 2024-03-06T09:52:58.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:58.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:58.628440+0000) 2024-03-06T09:52:59.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:52:59.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:52:59.628722+0000) 2024-03-06T09:53:00.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:00.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:00.628977+0000) 2024-03-06T09:53:01.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:01.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:01.629201+0000) 2024-03-06T09:53:02.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:02.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:02.629409+0000) 2024-03-06T09:53:03.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:03.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:03.629641+0000) 2024-03-06T09:53:04.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:04.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:04.629880+0000) 2024-03-06T09:53:05.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:05.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:05.630141+0000) 2024-03-06T09:53:06.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:06.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:06.630351+0000) 2024-03-06T09:53:07.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:07.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:07.630582+0000) 2024-03-06T09:53:08.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:08.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:08.630811+0000) 2024-03-06T09:53:09.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:09.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:09.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:09.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:09.631106+0000) 2024-03-06T09:53:10.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:10.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:10.631334+0000) 2024-03-06T09:53:11.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:11.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:11.631543+0000) 2024-03-06T09:53:12.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:12.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:12.631781+0000) 2024-03-06T09:53:13.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:13.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:13.632043+0000) 2024-03-06T09:53:14.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:14.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:14.632277+0000) 2024-03-06T09:53:15.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:15.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:15.632490+0000) 2024-03-06T09:53:16.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:16.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:16.632706+0000) 2024-03-06T09:53:17.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:17.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:17.632950+0000) 2024-03-06T09:53:18.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:18.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:18.633197+0000) 2024-03-06T09:53:19.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:19.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:19.633411+0000) 2024-03-06T09:53:20.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:20.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:20.633657+0000) 2024-03-06T09:53:21.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:21.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:21.633896+0000) 2024-03-06T09:53:22.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:22.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:22.634145+0000) 2024-03-06T09:53:23.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:23.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:23.634305+0000) 2024-03-06T09:53:24.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:24.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:24.634535+0000) 2024-03-06T09:53:25.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:25.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:25.634782+0000) 2024-03-06T09:53:26.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:26.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:26.635000+0000) 2024-03-06T09:53:27.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:27.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:27.635263+0000) 2024-03-06T09:53:28.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:28.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:28.635504+0000) 2024-03-06T09:53:29.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:29.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:29.635742+0000) 2024-03-06T09:53:30.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:30.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:30.636054+0000) 2024-03-06T09:53:31.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:31.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:31.636288+0000) 2024-03-06T09:53:32.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:32.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:32.636516+0000) 2024-03-06T09:53:33.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:33.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:33.636758+0000) 2024-03-06T09:53:34.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:34.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:34.637000+0000) 2024-03-06T09:53:35.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:35.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:35.637246+0000) 2024-03-06T09:53:36.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:36.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:36.637480+0000) 2024-03-06T09:53:37.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:37.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:37.637724+0000) 2024-03-06T09:53:38.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:38.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:38.637970+0000) 2024-03-06T09:53:39.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:39.392 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:53:39.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:39.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:39.638228+0000) 2024-03-06T09:53:40.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:40.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:40.638463+0000) 2024-03-06T09:53:41.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:41.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:41.638693+0000) 2024-03-06T09:53:42.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:42.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:42.638910+0000) 2024-03-06T09:53:43.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:43.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:43.639165+0000) 2024-03-06T09:53:44.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:44.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:44.639384+0000) 2024-03-06T09:53:45.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:45.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:45.639618+0000) 2024-03-06T09:53:46.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:46.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:46.639876+0000) 2024-03-06T09:53:47.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:47.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:47.640148+0000) 2024-03-06T09:53:48.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:48.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:48.640433+0000) 2024-03-06T09:53:49.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:49.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:49.640714+0000) 2024-03-06T09:53:50.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:50.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:50.641045+0000) 2024-03-06T09:53:51.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:51.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:51.641296+0000) 2024-03-06T09:53:52.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:52.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:52.641566+0000) 2024-03-06T09:53:53.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:53.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:53.641750+0000) 2024-03-06T09:53:54.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:54.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:54.641974+0000) 2024-03-06T09:53:55.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:55.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:55.642193+0000) 2024-03-06T09:53:56.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:56.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:56.642465+0000) 2024-03-06T09:53:57.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:57.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:57.642666+0000) 2024-03-06T09:53:58.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:58.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:58.642897+0000) 2024-03-06T09:53:59.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:53:59.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:53:59.643144+0000) 2024-03-06T09:54:00.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:00.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:00.643368+0000) 2024-03-06T09:54:01.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:01.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:01.643588+0000) 2024-03-06T09:54:02.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:02.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:02.643832+0000) 2024-03-06T09:54:03.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:03.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:03.644106+0000) 2024-03-06T09:54:04.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:04.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:04.644308+0000) 2024-03-06T09:54:05.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:05.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:05.644536+0000) 2024-03-06T09:54:06.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:06.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:06.644774+0000) 2024-03-06T09:54:07.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:07.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:07.644996+0000) 2024-03-06T09:54:08.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:08.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:08.645287+0000) 2024-03-06T09:54:09.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:09.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:09.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:09.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:09.645485+0000) 2024-03-06T09:54:10.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:10.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:10.645667+0000) 2024-03-06T09:54:11.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:11.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:11.645926+0000) 2024-03-06T09:54:12.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:12.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:12.646161+0000) 2024-03-06T09:54:13.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:13.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:13.646395+0000) 2024-03-06T09:54:14.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:14.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:14.646624+0000) 2024-03-06T09:54:15.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:15.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:15.646852+0000) 2024-03-06T09:54:16.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:16.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:16.647108+0000) 2024-03-06T09:54:17.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:17.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:17.647344+0000) 2024-03-06T09:54:18.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:18.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:18.647573+0000) 2024-03-06T09:54:19.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:19.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:19.647787+0000) 2024-03-06T09:54:20.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:20.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:20.648106+0000) 2024-03-06T09:54:21.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:21.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:21.648356+0000) 2024-03-06T09:54:22.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:22.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:22.648596+0000) 2024-03-06T09:54:23.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:23.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:23.648845+0000) 2024-03-06T09:54:24.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:24.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:24.649113+0000) 2024-03-06T09:54:25.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:25.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:25.649312+0000) 2024-03-06T09:54:26.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:26.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:26.649534+0000) 2024-03-06T09:54:27.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:27.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:27.649755+0000) 2024-03-06T09:54:28.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:28.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:28.650054+0000) 2024-03-06T09:54:29.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:29.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:29.650253+0000) 2024-03-06T09:54:30.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:30.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:30.650566+0000) 2024-03-06T09:54:31.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:31.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:31.650846+0000) 2024-03-06T09:54:32.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:32.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:32.651152+0000) 2024-03-06T09:54:33.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:33.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:33.651437+0000) 2024-03-06T09:54:34.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:34.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:34.651710+0000) 2024-03-06T09:54:35.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:35.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:35.651910+0000) 2024-03-06T09:54:36.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:36.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:36.652219+0000) 2024-03-06T09:54:37.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:37.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:37.652502+0000) 2024-03-06T09:54:38.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:38.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:38.652784+0000) 2024-03-06T09:54:39.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:39.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:39.653091+0000) 2024-03-06T09:54:39.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:39.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:54:40.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:40.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:40.653352+0000) 2024-03-06T09:54:41.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:41.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:41.653612+0000) 2024-03-06T09:54:42.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:42.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:42.653843+0000) 2024-03-06T09:54:43.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:43.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:43.654127+0000) 2024-03-06T09:54:44.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:44.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:44.654356+0000) 2024-03-06T09:54:45.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:45.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:45.654625+0000) 2024-03-06T09:54:46.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:46.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:46.654901+0000) 2024-03-06T09:54:47.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:47.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:47.655167+0000) 2024-03-06T09:54:48.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:48.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:48.655452+0000) 2024-03-06T09:54:49.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:49.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:49.655750+0000) 2024-03-06T09:54:50.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:50.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:50.656076+0000) 2024-03-06T09:54:51.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:51.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:51.656319+0000) 2024-03-06T09:54:52.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:52.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:52.656590+0000) 2024-03-06T09:54:53.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:53.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:53.656869+0000) 2024-03-06T09:54:54.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:54.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:54.657153+0000) 2024-03-06T09:54:55.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:55.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:55.657372+0000) 2024-03-06T09:54:56.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:56.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:56.657643+0000) 2024-03-06T09:54:57.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:57.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:57.657914+0000) 2024-03-06T09:54:58.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:58.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:58.658204+0000) 2024-03-06T09:54:59.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:54:59.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:54:59.658445+0000) 2024-03-06T09:55:00.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:00.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:00.658759+0000) 2024-03-06T09:55:01.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:01.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:01.658997+0000) 2024-03-06T09:55:02.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:02.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:02.659288+0000) 2024-03-06T09:55:03.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:03.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:03.659505+0000) 2024-03-06T09:55:04.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:04.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:04.659744+0000) 2024-03-06T09:55:05.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:05.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:05.660277+0000) 2024-03-06T09:55:06.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:06.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:06.660687+0000) 2024-03-06T09:55:07.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:07.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:07.660962+0000) 2024-03-06T09:55:08.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:08.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:08.661265+0000) 2024-03-06T09:55:09.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:09.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:09.661474+0000) 2024-03-06T09:55:10.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:10.022 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:10.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:10.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:10.661735+0000) 2024-03-06T09:55:11.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:11.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:11.661964+0000) 2024-03-06T09:55:12.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:12.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:12.662244+0000) 2024-03-06T09:55:13.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:13.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:13.662492+0000) 2024-03-06T09:55:14.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:14.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:14.662734+0000) 2024-03-06T09:55:15.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:15.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:15.663047+0000) 2024-03-06T09:55:16.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:16.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:16.663257+0000) 2024-03-06T09:55:17.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:17.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:17.663476+0000) 2024-03-06T09:55:18.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:18.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:18.663737+0000) 2024-03-06T09:55:19.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:19.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:19.663921+0000) 2024-03-06T09:55:20.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:20.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:20.664201+0000) 2024-03-06T09:55:21.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:21.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:21.664443+0000) 2024-03-06T09:55:22.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:22.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:22.664683+0000) 2024-03-06T09:55:23.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:23.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:23.664908+0000) 2024-03-06T09:55:24.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:24.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:24.665221+0000) 2024-03-06T09:55:25.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:25.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:25.665429+0000) 2024-03-06T09:55:26.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:26.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:26.665619+0000) 2024-03-06T09:55:27.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:27.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:27.665810+0000) 2024-03-06T09:55:28.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:28.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:28.666122+0000) 2024-03-06T09:55:29.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:29.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:29.666407+0000) 2024-03-06T09:55:30.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:30.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:30.666696+0000) 2024-03-06T09:55:31.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:31.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:31.666944+0000) 2024-03-06T09:55:32.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:32.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:32.667175+0000) 2024-03-06T09:55:33.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:33.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:33.667441+0000) 2024-03-06T09:55:34.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:34.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:34.667714+0000) 2024-03-06T09:55:35.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:35.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:35.667998+0000) 2024-03-06T09:55:36.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:36.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:36.668294+0000) 2024-03-06T09:55:37.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:37.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:37.668510+0000) 2024-03-06T09:55:38.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:38.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:38.668805+0000) 2024-03-06T09:55:39.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:39.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:39.669134+0000) 2024-03-06T09:55:40.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:40.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:55:40.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:40.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:40.669433+0000) 2024-03-06T09:55:41.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:41.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:41.669676+0000) 2024-03-06T09:55:42.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:42.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:42.669813+0000) 2024-03-06T09:55:43.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:43.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:43.670070+0000) 2024-03-06T09:55:44.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:44.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:44.670291+0000) 2024-03-06T09:55:45.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:45.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:45.670521+0000) 2024-03-06T09:55:46.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:46.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:46.670744+0000) 2024-03-06T09:55:47.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:47.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:47.670973+0000) 2024-03-06T09:55:48.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:48.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:48.671196+0000) 2024-03-06T09:55:49.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:49.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:49.671444+0000) 2024-03-06T09:55:50.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:50.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:50.671695+0000) 2024-03-06T09:55:51.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:51.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:51.671946+0000) 2024-03-06T09:55:52.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:52.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:52.672176+0000) 2024-03-06T09:55:53.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:53.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:53.672404+0000) 2024-03-06T09:55:54.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:54.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:54.672615+0000) 2024-03-06T09:55:55.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:55.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:55.672838+0000) 2024-03-06T09:55:56.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:56.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:56.673097+0000) 2024-03-06T09:55:57.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:57.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:57.673318+0000) 2024-03-06T09:55:58.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:58.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:58.673552+0000) 2024-03-06T09:55:59.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:55:59.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:55:59.673792+0000) 2024-03-06T09:56:00.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:00.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:00.674115+0000) 2024-03-06T09:56:01.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:01.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:01.674395+0000) 2024-03-06T09:56:02.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:02.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:02.674667+0000) 2024-03-06T09:56:03.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:03.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:03.674829+0000) 2024-03-06T09:56:04.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:04.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:04.675078+0000) 2024-03-06T09:56:05.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:05.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:05.675317+0000) 2024-03-06T09:56:06.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:06.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:06.675578+0000) 2024-03-06T09:56:07.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:07.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:07.675795+0000) 2024-03-06T09:56:08.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:08.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:08.676091+0000) 2024-03-06T09:56:09.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:09.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:09.676335+0000) 2024-03-06T09:56:10.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:10.440 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:10.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:10.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:10.676600+0000) 2024-03-06T09:56:11.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:11.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:11.676836+0000) 2024-03-06T09:56:12.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:12.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:12.677095+0000) 2024-03-06T09:56:13.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:13.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:13.677325+0000) 2024-03-06T09:56:14.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:14.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:14.677547+0000) 2024-03-06T09:56:15.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:15.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:15.677768+0000) 2024-03-06T09:56:16.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:16.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:16.678042+0000) 2024-03-06T09:56:17.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:17.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:17.678300+0000) 2024-03-06T09:56:18.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:18.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:18.678557+0000) 2024-03-06T09:56:19.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:19.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:19.678832+0000) 2024-03-06T09:56:20.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:20.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:20.679143+0000) 2024-03-06T09:56:21.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:21.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:21.679386+0000) 2024-03-06T09:56:22.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:22.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:22.679565+0000) 2024-03-06T09:56:23.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:23.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:23.679795+0000) 2024-03-06T09:56:24.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:24.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:24.679985+0000) 2024-03-06T09:56:25.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:25.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:25.680241+0000) 2024-03-06T09:56:26.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:26.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:26.680483+0000) 2024-03-06T09:56:27.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:27.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:27.680757+0000) 2024-03-06T09:56:28.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:28.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:28.681074+0000) 2024-03-06T09:56:29.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:29.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:29.681363+0000) 2024-03-06T09:56:30.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:30.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:30.681616+0000) 2024-03-06T09:56:31.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:31.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:31.681827+0000) 2024-03-06T09:56:32.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:32.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:32.682149+0000) 2024-03-06T09:56:33.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:33.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:33.682384+0000) 2024-03-06T09:56:34.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:34.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:34.682635+0000) 2024-03-06T09:56:35.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:35.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:35.682911+0000) 2024-03-06T09:56:36.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:36.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:36.683169+0000) 2024-03-06T09:56:37.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:37.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:37.683396+0000) 2024-03-06T09:56:38.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:38.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:38.683663+0000) 2024-03-06T09:56:39.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:39.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:39.683943+0000) 2024-03-06T09:56:40.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:40.652 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:56:40.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:40.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:40.684124+0000) 2024-03-06T09:56:41.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:41.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:41.684333+0000) 2024-03-06T09:56:42.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:42.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:42.684535+0000) 2024-03-06T09:56:43.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:43.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:43.684763+0000) 2024-03-06T09:56:44.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:44.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:44.684989+0000) 2024-03-06T09:56:45.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:45.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:45.685259+0000) 2024-03-06T09:56:46.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:46.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:46.685488+0000) 2024-03-06T09:56:47.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:47.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:47.685711+0000) 2024-03-06T09:56:48.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:48.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:48.685938+0000) 2024-03-06T09:56:49.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:49.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:49.686162+0000) 2024-03-06T09:56:50.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:50.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:50.686408+0000) 2024-03-06T09:56:51.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:51.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:51.686663+0000) 2024-03-06T09:56:52.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:52.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:52.686844+0000) 2024-03-06T09:56:53.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:53.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:53.687131+0000) 2024-03-06T09:56:54.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:54.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:54.687341+0000) 2024-03-06T09:56:55.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:55.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:55.687583+0000) 2024-03-06T09:56:56.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:56.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:56.687833+0000) 2024-03-06T09:56:57.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:57.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:57.688109+0000) 2024-03-06T09:56:58.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:58.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:58.688286+0000) 2024-03-06T09:56:59.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:56:59.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:56:59.688490+0000) 2024-03-06T09:57:00.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:00.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:00.688677+0000) 2024-03-06T09:57:01.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:01.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:01.688906+0000) 2024-03-06T09:57:02.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:02.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:02.689159+0000) 2024-03-06T09:57:03.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:03.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:03.689353+0000) 2024-03-06T09:57:04.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:04.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:04.689618+0000) 2024-03-06T09:57:05.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:05.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:05.689943+0000) 2024-03-06T09:57:06.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:06.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:06.690218+0000) 2024-03-06T09:57:07.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:07.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:07.690465+0000) 2024-03-06T09:57:08.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:08.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:08.690734+0000) 2024-03-06T09:57:09.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:09.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:09.690943+0000) 2024-03-06T09:57:10.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:10.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:10.691209+0000) 2024-03-06T09:57:10.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:10.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:11.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:11.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:11.691502+0000) 2024-03-06T09:57:12.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:12.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:12.691756+0000) 2024-03-06T09:57:13.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:13.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:13.691982+0000) 2024-03-06T09:57:14.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:14.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:14.692254+0000) 2024-03-06T09:57:15.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:15.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:15.692487+0000) 2024-03-06T09:57:16.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:16.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:16.692727+0000) 2024-03-06T09:57:17.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:17.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:17.692977+0000) 2024-03-06T09:57:18.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:18.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:18.693233+0000) 2024-03-06T09:57:19.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:19.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:19.693467+0000) 2024-03-06T09:57:20.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:20.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:20.693678+0000) 2024-03-06T09:57:21.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:21.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:21.693926+0000) 2024-03-06T09:57:22.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:22.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:22.694174+0000) 2024-03-06T09:57:23.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:23.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:23.694410+0000) 2024-03-06T09:57:24.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:24.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:24.694657+0000) 2024-03-06T09:57:25.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:25.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:25.694901+0000) 2024-03-06T09:57:26.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:26.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:26.695171+0000) 2024-03-06T09:57:27.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:27.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:27.695358+0000) 2024-03-06T09:57:28.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:28.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:28.695586+0000) 2024-03-06T09:57:29.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:29.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:29.695822+0000) 2024-03-06T09:57:30.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:30.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:30.696101+0000) 2024-03-06T09:57:31.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:31.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:31.696355+0000) 2024-03-06T09:57:32.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:32.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:32.696607+0000) 2024-03-06T09:57:33.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:33.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:33.696888+0000) 2024-03-06T09:57:34.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:34.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:34.697156+0000) 2024-03-06T09:57:35.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:35.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:35.697438+0000) 2024-03-06T09:57:36.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:36.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:36.697712+0000) 2024-03-06T09:57:37.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:37.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:37.697986+0000) 2024-03-06T09:57:38.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:38.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:38.698285+0000) 2024-03-06T09:57:39.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:39.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:39.698543+0000) 2024-03-06T09:57:40.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:40.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:40.698829+0000) 2024-03-06T09:57:41.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:41.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:57:41.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:41.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:41.699148+0000) 2024-03-06T09:57:42.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:42.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:42.699428+0000) 2024-03-06T09:57:43.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:43.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:43.699713+0000) 2024-03-06T09:57:44.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:44.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:44.699997+0000) 2024-03-06T09:57:45.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:45.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:45.700315+0000) 2024-03-06T09:57:46.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:46.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:46.700584+0000) 2024-03-06T09:57:47.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:47.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:47.700868+0000) 2024-03-06T09:57:48.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:48.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:48.701153+0000) 2024-03-06T09:57:49.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:49.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:49.701432+0000) 2024-03-06T09:57:50.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:50.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:50.701717+0000) 2024-03-06T09:57:51.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:51.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:51.702049+0000) 2024-03-06T09:57:52.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:52.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:52.702341+0000) 2024-03-06T09:57:53.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:53.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:53.702667+0000) 2024-03-06T09:57:54.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:54.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:54.702953+0000) 2024-03-06T09:57:55.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:55.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:55.703232+0000) 2024-03-06T09:57:56.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:56.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:56.703501+0000) 2024-03-06T09:57:57.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:57.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:57.703763+0000) 2024-03-06T09:57:58.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:58.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:58.704072+0000) 2024-03-06T09:57:59.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:57:59.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:57:59.704281+0000) 2024-03-06T09:58:00.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:00.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:00.704486+0000) 2024-03-06T09:58:01.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:01.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:01.704687+0000) 2024-03-06T09:58:02.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:02.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:02.704873+0000) 2024-03-06T09:58:03.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:03.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:03.705138+0000) 2024-03-06T09:58:04.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:04.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:04.705367+0000) 2024-03-06T09:58:05.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:05.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:05.705598+0000) 2024-03-06T09:58:06.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:06.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:06.705873+0000) 2024-03-06T09:58:07.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:07.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:07.706122+0000) 2024-03-06T09:58:08.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:08.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:08.706346+0000) 2024-03-06T09:58:09.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:09.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:09.706573+0000) 2024-03-06T09:58:10.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:10.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:10.706812+0000) 2024-03-06T09:58:11.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:11.271 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:11.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:11.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:11.707107+0000) 2024-03-06T09:58:12.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:12.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:12.707318+0000) 2024-03-06T09:58:13.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:13.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:13.707557+0000) 2024-03-06T09:58:14.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:14.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:14.707802+0000) 2024-03-06T09:58:15.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:15.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:15.708077+0000) 2024-03-06T09:58:16.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:16.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:16.708313+0000) 2024-03-06T09:58:17.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:17.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:17.708585+0000) 2024-03-06T09:58:18.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:18.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:18.708800+0000) 2024-03-06T09:58:19.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:19.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:19.709082+0000) 2024-03-06T09:58:20.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:20.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:20.709350+0000) 2024-03-06T09:58:21.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:21.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:21.709609+0000) 2024-03-06T09:58:22.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:22.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:22.709826+0000) 2024-03-06T09:58:23.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:23.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:23.710096+0000) 2024-03-06T09:58:24.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:24.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:24.710329+0000) 2024-03-06T09:58:25.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:25.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:25.710588+0000) 2024-03-06T09:58:26.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:26.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:26.710819+0000) 2024-03-06T09:58:27.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:27.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:27.711083+0000) 2024-03-06T09:58:28.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:28.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:28.711312+0000) 2024-03-06T09:58:29.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:29.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:29.711517+0000) 2024-03-06T09:58:30.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:30.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:30.711756+0000) 2024-03-06T09:58:31.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:31.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:31.711996+0000) 2024-03-06T09:58:32.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:32.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:32.712298+0000) 2024-03-06T09:58:33.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:33.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:33.712538+0000) 2024-03-06T09:58:34.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:34.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:34.712745+0000) 2024-03-06T09:58:35.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:35.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:35.712971+0000) 2024-03-06T09:58:36.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:36.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:36.713222+0000) 2024-03-06T09:58:37.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:37.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:37.713471+0000) 2024-03-06T09:58:38.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:38.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:38.713718+0000) 2024-03-06T09:58:39.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:39.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:39.713940+0000) 2024-03-06T09:58:40.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:40.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:40.714174+0000) 2024-03-06T09:58:41.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:41.468 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:58:41.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:41.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:41.714406+0000) 2024-03-06T09:58:42.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:42.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:42.714637+0000) 2024-03-06T09:58:43.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:43.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:43.714824+0000) 2024-03-06T09:58:44.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:44.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:44.715043+0000) 2024-03-06T09:58:45.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:45.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:45.715269+0000) 2024-03-06T09:58:46.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:46.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:46.715534+0000) 2024-03-06T09:58:47.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:47.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:47.715745+0000) 2024-03-06T09:58:48.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:48.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:48.715955+0000) 2024-03-06T09:58:49.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:49.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:49.716202+0000) 2024-03-06T09:58:50.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:50.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:50.716462+0000) 2024-03-06T09:58:51.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:51.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:51.716696+0000) 2024-03-06T09:58:52.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:52.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:52.716950+0000) 2024-03-06T09:58:53.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:53.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:53.717186+0000) 2024-03-06T09:58:54.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:54.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:54.717432+0000) 2024-03-06T09:58:55.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:55.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:55.717673+0000) 2024-03-06T09:58:56.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:56.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:56.717922+0000) 2024-03-06T09:58:57.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:57.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:57.718170+0000) 2024-03-06T09:58:58.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:58.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:58.718375+0000) 2024-03-06T09:58:59.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:58:59.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:58:59.718625+0000) 2024-03-06T09:59:00.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:00.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:00.718880+0000) 2024-03-06T09:59:01.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:01.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:01.719155+0000) 2024-03-06T09:59:02.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:02.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:02.719396+0000) 2024-03-06T09:59:03.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:03.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:03.719615+0000) 2024-03-06T09:59:04.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:04.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:04.719858+0000) 2024-03-06T09:59:05.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:05.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:05.720135+0000) 2024-03-06T09:59:06.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:06.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:06.720338+0000) 2024-03-06T09:59:07.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:07.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:07.720573+0000) 2024-03-06T09:59:08.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:08.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:08.720785+0000) 2024-03-06T09:59:09.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:09.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:09.721075+0000) 2024-03-06T09:59:10.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:10.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:10.721318+0000) 2024-03-06T09:59:11.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:11.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:11.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:11.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:11.721499+0000) 2024-03-06T09:59:12.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:12.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:12.721724+0000) 2024-03-06T09:59:13.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:13.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:13.721969+0000) 2024-03-06T09:59:14.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:14.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:14.722229+0000) 2024-03-06T09:59:15.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:15.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:15.722433+0000) 2024-03-06T09:59:16.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:16.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:16.722663+0000) 2024-03-06T09:59:17.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:17.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:17.722921+0000) 2024-03-06T09:59:18.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:18.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:18.723189+0000) 2024-03-06T09:59:19.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:19.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:19.723437+0000) 2024-03-06T09:59:20.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:20.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:20.723671+0000) 2024-03-06T09:59:21.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:21.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:21.723903+0000) 2024-03-06T09:59:22.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:22.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:22.724178+0000) 2024-03-06T09:59:23.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:23.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:23.724452+0000) 2024-03-06T09:59:24.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:24.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:24.724728+0000) 2024-03-06T09:59:25.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:25.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:25.724963+0000) 2024-03-06T09:59:26.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:26.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:26.725262+0000) 2024-03-06T09:59:27.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:27.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:27.725481+0000) 2024-03-06T09:59:28.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:28.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:28.725686+0000) 2024-03-06T09:59:29.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:29.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:29.725870+0000) 2024-03-06T09:59:30.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:30.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:30.726134+0000) 2024-03-06T09:59:31.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:31.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:31.726348+0000) 2024-03-06T09:59:32.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:32.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:32.726579+0000) 2024-03-06T09:59:33.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:33.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:33.726781+0000) 2024-03-06T09:59:34.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:34.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:34.727099+0000) 2024-03-06T09:59:35.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:35.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:35.727316+0000) 2024-03-06T09:59:36.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:36.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:36.727559+0000) 2024-03-06T09:59:37.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:37.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:37.727802+0000) 2024-03-06T09:59:38.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:38.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:38.728116+0000) 2024-03-06T09:59:39.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:39.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:39.728330+0000) 2024-03-06T09:59:40.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:40.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:40.728570+0000) 2024-03-06T09:59:41.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:41.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:41.728801+0000) 2024-03-06T09:59:41.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:41.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T09:59:42.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:42.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:42.729064+0000) 2024-03-06T09:59:43.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:43.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:43.729293+0000) 2024-03-06T09:59:44.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:44.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:44.729522+0000) 2024-03-06T09:59:45.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:45.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:45.729743+0000) 2024-03-06T09:59:46.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:46.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:46.729958+0000) 2024-03-06T09:59:47.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:47.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:47.730186+0000) 2024-03-06T09:59:48.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:48.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:48.730407+0000) 2024-03-06T09:59:49.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:49.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:49.730638+0000) 2024-03-06T09:59:50.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:50.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:50.730892+0000) 2024-03-06T09:59:51.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:51.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:51.731129+0000) 2024-03-06T09:59:52.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:52.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:52.731364+0000) 2024-03-06T09:59:53.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:53.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:53.731588+0000) 2024-03-06T09:59:54.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:54.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:54.731821+0000) 2024-03-06T09:59:55.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:55.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:55.732090+0000) 2024-03-06T09:59:56.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:56.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:56.732337+0000) 2024-03-06T09:59:57.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:57.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:57.732640+0000) 2024-03-06T09:59:58.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:58.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:58.732846+0000) 2024-03-06T09:59:59.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T09:59:59.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T08:59:59.733157+0000) 2024-03-06T10:00:00.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:00.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:00.733368+0000) 2024-03-06T10:00:01.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:01.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:01.733567+0000) 2024-03-06T10:00:02.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:02.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:02.733755+0000) 2024-03-06T10:00:03.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:03.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:03.733946+0000) 2024-03-06T10:00:04.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:04.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:04.734131+0000) 2024-03-06T10:00:05.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:05.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:05.734320+0000) 2024-03-06T10:00:06.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:06.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:06.734528+0000) 2024-03-06T10:00:07.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:07.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:07.734723+0000) 2024-03-06T10:00:08.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:08.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:08.734948+0000) 2024-03-06T10:00:09.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:09.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:09.735180+0000) 2024-03-06T10:00:10.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:10.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:10.735419+0000) 2024-03-06T10:00:11.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:11.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:11.735602+0000) 2024-03-06T10:00:11.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:12.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:12.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:12.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:12.735803+0000) 2024-03-06T10:00:13.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:13.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:13.736050+0000) 2024-03-06T10:00:14.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:14.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:14.736241+0000) 2024-03-06T10:00:15.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:15.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:15.736456+0000) 2024-03-06T10:00:16.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:16.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:16.736667+0000) 2024-03-06T10:00:17.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:17.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:17.736942+0000) 2024-03-06T10:00:18.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:18.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:18.737220+0000) 2024-03-06T10:00:19.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:19.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:19.737424+0000) 2024-03-06T10:00:20.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:20.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:20.737640+0000) 2024-03-06T10:00:21.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:21.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:21.737874+0000) 2024-03-06T10:00:22.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:22.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:22.738184+0000) 2024-03-06T10:00:23.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:23.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:23.738475+0000) 2024-03-06T10:00:24.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:24.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:24.738743+0000) 2024-03-06T10:00:25.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:25.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:25.739061+0000) 2024-03-06T10:00:26.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:26.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:26.739356+0000) 2024-03-06T10:00:27.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:27.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:27.739630+0000) 2024-03-06T10:00:28.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:28.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:28.739903+0000) 2024-03-06T10:00:29.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:29.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:29.740113+0000) 2024-03-06T10:00:30.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:30.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:30.740362+0000) 2024-03-06T10:00:31.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:31.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:31.740605+0000) 2024-03-06T10:00:32.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:32.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:32.740847+0000) 2024-03-06T10:00:33.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:33.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:33.741080+0000) 2024-03-06T10:00:34.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:34.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:34.741307+0000) 2024-03-06T10:00:35.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:35.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:35.741517+0000) 2024-03-06T10:00:36.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:36.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:36.741761+0000) 2024-03-06T10:00:37.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:37.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:37.741978+0000) 2024-03-06T10:00:38.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:38.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:38.742262+0000) 2024-03-06T10:00:39.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:39.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:39.742506+0000) 2024-03-06T10:00:40.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:40.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:40.742729+0000) 2024-03-06T10:00:41.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:41.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:41.742922+0000) 2024-03-06T10:00:42.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:42.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:00:42.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:42.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:42.743158+0000) 2024-03-06T10:00:43.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:43.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:43.743394+0000) 2024-03-06T10:00:44.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:44.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:44.743617+0000) 2024-03-06T10:00:45.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:45.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:45.743810+0000) 2024-03-06T10:00:46.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:46.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:46.744101+0000) 2024-03-06T10:00:47.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:47.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:47.744302+0000) 2024-03-06T10:00:48.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:48.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:48.744573+0000) 2024-03-06T10:00:49.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:49.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:49.744827+0000) 2024-03-06T10:00:50.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:50.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:50.745109+0000) 2024-03-06T10:00:51.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:51.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:51.745314+0000) 2024-03-06T10:00:52.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:52.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:52.745553+0000) 2024-03-06T10:00:53.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:53.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:53.745798+0000) 2024-03-06T10:00:54.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:54.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:54.746077+0000) 2024-03-06T10:00:55.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:55.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:55.746327+0000) 2024-03-06T10:00:56.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:56.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:56.746547+0000) 2024-03-06T10:00:57.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:57.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:57.746784+0000) 2024-03-06T10:00:58.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:58.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:58.747105+0000) 2024-03-06T10:00:59.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:00:59.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:00:59.747382+0000) 2024-03-06T10:01:00.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:00.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:00.747656+0000) 2024-03-06T10:01:01.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:01.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:01.747938+0000) 2024-03-06T10:01:02.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:02.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:02.748215+0000) 2024-03-06T10:01:03.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:03.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:03.748493+0000) 2024-03-06T10:01:04.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:04.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:04.748776+0000) 2024-03-06T10:01:05.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:05.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:05.749109+0000) 2024-03-06T10:01:06.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:06.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:06.749401+0000) 2024-03-06T10:01:07.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:07.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:07.749681+0000) 2024-03-06T10:01:08.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:08.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:08.749964+0000) 2024-03-06T10:01:09.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:09.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:09.750290+0000) 2024-03-06T10:01:10.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:10.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:10.750593+0000) 2024-03-06T10:01:11.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:11.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:11.750888+0000) 2024-03-06T10:01:12.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:12.429 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:12.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:12.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:12.751170+0000) 2024-03-06T10:01:13.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:13.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:13.751445+0000) 2024-03-06T10:01:14.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:14.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:14.751715+0000) 2024-03-06T10:01:15.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:15.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:15.751944+0000) 2024-03-06T10:01:16.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:16.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:16.752266+0000) 2024-03-06T10:01:17.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:17.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:17.752516+0000) 2024-03-06T10:01:18.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:18.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:18.752772+0000) 2024-03-06T10:01:19.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:19.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:19.752991+0000) 2024-03-06T10:01:20.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:20.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:20.753265+0000) 2024-03-06T10:01:21.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:21.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:21.753508+0000) 2024-03-06T10:01:22.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:22.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:22.753739+0000) 2024-03-06T10:01:23.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:23.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:23.753974+0000) 2024-03-06T10:01:24.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:24.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:24.754202+0000) 2024-03-06T10:01:25.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:25.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:25.754383+0000) 2024-03-06T10:01:26.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:26.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:26.754623+0000) 2024-03-06T10:01:27.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:27.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:27.754883+0000) 2024-03-06T10:01:28.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:28.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:28.755155+0000) 2024-03-06T10:01:29.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:29.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:29.755379+0000) 2024-03-06T10:01:30.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:30.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:30.755591+0000) 2024-03-06T10:01:31.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:31.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:31.755829+0000) 2024-03-06T10:01:32.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:32.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:32.756090+0000) 2024-03-06T10:01:33.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:33.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:33.756300+0000) 2024-03-06T10:01:34.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:34.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:34.756526+0000) 2024-03-06T10:01:35.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:35.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:35.756749+0000) 2024-03-06T10:01:36.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:36.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:36.756953+0000) 2024-03-06T10:01:37.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:37.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:37.757201+0000) 2024-03-06T10:01:38.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:38.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:38.757438+0000) 2024-03-06T10:01:39.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:39.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:39.757657+0000) 2024-03-06T10:01:40.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:40.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:40.757892+0000) 2024-03-06T10:01:41.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:41.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:41.758118+0000) 2024-03-06T10:01:42.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:42.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:01:42.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:42.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:42.758322+0000) 2024-03-06T10:01:43.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:43.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:43.758540+0000) 2024-03-06T10:01:44.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:44.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:44.758760+0000) 2024-03-06T10:01:45.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:45.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:45.758982+0000) 2024-03-06T10:01:46.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:46.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:46.759212+0000) 2024-03-06T10:01:47.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:47.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:47.759397+0000) 2024-03-06T10:01:48.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:48.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:48.759642+0000) 2024-03-06T10:01:49.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:49.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:49.759876+0000) 2024-03-06T10:01:50.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:50.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:50.760133+0000) 2024-03-06T10:01:51.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:51.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:51.760378+0000) 2024-03-06T10:01:52.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:52.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:52.760641+0000) 2024-03-06T10:01:53.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:53.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:53.760874+0000) 2024-03-06T10:01:54.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:54.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:54.761127+0000) 2024-03-06T10:01:55.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:55.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:55.761360+0000) 2024-03-06T10:01:56.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:56.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:56.761584+0000) 2024-03-06T10:01:57.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:57.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:57.761817+0000) 2024-03-06T10:01:58.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:58.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:58.762110+0000) 2024-03-06T10:01:59.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:01:59.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:01:59.762358+0000) 2024-03-06T10:02:00.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:00.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:00.762612+0000) 2024-03-06T10:02:01.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:01.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:01.762854+0000) 2024-03-06T10:02:02.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:02.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:02.763153+0000) 2024-03-06T10:02:03.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:03.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:03.763415+0000) 2024-03-06T10:02:04.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:04.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:04.763667+0000) 2024-03-06T10:02:05.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:05.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:05.763909+0000) 2024-03-06T10:02:06.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:06.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:06.764150+0000) 2024-03-06T10:02:07.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:07.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:07.764381+0000) 2024-03-06T10:02:08.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:08.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:08.764566+0000) 2024-03-06T10:02:09.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:09.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:09.764795+0000) 2024-03-06T10:02:10.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:10.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:10.765082+0000) 2024-03-06T10:02:11.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:11.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:11.765326+0000) 2024-03-06T10:02:12.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:12.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:12.765578+0000) 2024-03-06T10:02:12.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:12.787 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:13.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:13.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:13.765779+0000) 2024-03-06T10:02:14.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:14.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:14.766048+0000) 2024-03-06T10:02:15.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:15.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:15.766267+0000) 2024-03-06T10:02:16.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:16.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:16.766486+0000) 2024-03-06T10:02:17.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:17.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:17.766705+0000) 2024-03-06T10:02:18.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:18.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:18.766883+0000) 2024-03-06T10:02:19.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:19.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:19.767139+0000) 2024-03-06T10:02:20.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:20.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:20.767399+0000) 2024-03-06T10:02:21.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:21.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:21.767648+0000) 2024-03-06T10:02:22.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:22.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:22.767897+0000) 2024-03-06T10:02:23.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:23.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:23.768159+0000) 2024-03-06T10:02:24.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:24.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:24.768398+0000) 2024-03-06T10:02:25.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:25.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:25.768642+0000) 2024-03-06T10:02:26.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:26.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:26.768889+0000) 2024-03-06T10:02:27.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:27.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:27.769147+0000) 2024-03-06T10:02:28.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:28.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:28.769407+0000) 2024-03-06T10:02:29.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:29.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:29.769625+0000) 2024-03-06T10:02:30.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:30.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:30.769861+0000) 2024-03-06T10:02:31.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:31.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:31.770080+0000) 2024-03-06T10:02:32.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:32.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:32.770279+0000) 2024-03-06T10:02:33.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:33.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:33.770504+0000) 2024-03-06T10:02:34.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:34.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:34.770710+0000) 2024-03-06T10:02:35.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:35.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:35.770942+0000) 2024-03-06T10:02:36.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:36.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:36.771177+0000) 2024-03-06T10:02:37.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:37.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:37.771413+0000) 2024-03-06T10:02:38.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:38.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:38.771641+0000) 2024-03-06T10:02:39.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:39.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:39.771867+0000) 2024-03-06T10:02:40.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:40.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:40.772152+0000) 2024-03-06T10:02:41.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:41.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:41.772369+0000) 2024-03-06T10:02:42.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:42.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:42.772595+0000) 2024-03-06T10:02:42.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:42.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:02:43.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:43.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:43.772828+0000) 2024-03-06T10:02:44.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:44.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:44.773096+0000) 2024-03-06T10:02:45.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:45.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:45.773308+0000) 2024-03-06T10:02:46.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:46.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:46.773540+0000) 2024-03-06T10:02:47.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:47.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:47.773720+0000) 2024-03-06T10:02:48.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:48.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:48.773920+0000) 2024-03-06T10:02:49.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:49.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:49.774155+0000) 2024-03-06T10:02:50.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:50.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:50.774345+0000) 2024-03-06T10:02:51.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:51.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:51.774572+0000) 2024-03-06T10:02:52.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:52.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:52.774858+0000) 2024-03-06T10:02:53.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:53.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:53.775115+0000) 2024-03-06T10:02:54.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:54.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:54.775339+0000) 2024-03-06T10:02:55.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:55.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:55.775568+0000) 2024-03-06T10:02:56.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:56.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:56.775817+0000) 2024-03-06T10:02:57.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:57.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:57.776128+0000) 2024-03-06T10:02:58.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:58.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:58.776355+0000) 2024-03-06T10:02:59.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:02:59.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:02:59.776606+0000) 2024-03-06T10:03:00.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:00.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:00.776843+0000) 2024-03-06T10:03:01.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:01.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:01.777102+0000) 2024-03-06T10:03:02.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:02.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:02.777340+0000) 2024-03-06T10:03:03.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:03.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:03.777586+0000) 2024-03-06T10:03:04.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:04.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:04.777813+0000) 2024-03-06T10:03:05.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:05.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:05.778090+0000) 2024-03-06T10:03:06.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:06.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:06.778325+0000) 2024-03-06T10:03:07.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:07.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:07.778512+0000) 2024-03-06T10:03:08.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:08.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:08.778759+0000) 2024-03-06T10:03:09.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:09.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:09.778995+0000) 2024-03-06T10:03:10.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:10.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:10.779327+0000) 2024-03-06T10:03:11.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:11.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:11.779532+0000) 2024-03-06T10:03:12.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:12.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:12.779751+0000) 2024-03-06T10:03:13.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:13.105 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:13.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:13.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:13.779982+0000) 2024-03-06T10:03:14.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:14.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:14.780228+0000) 2024-03-06T10:03:15.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:15.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:15.780412+0000) 2024-03-06T10:03:16.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:16.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:16.780639+0000) 2024-03-06T10:03:17.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:17.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:17.780870+0000) 2024-03-06T10:03:18.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:18.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:18.781136+0000) 2024-03-06T10:03:19.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:19.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:19.781374+0000) 2024-03-06T10:03:20.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:20.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:20.781605+0000) 2024-03-06T10:03:21.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:21.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:21.781810+0000) 2024-03-06T10:03:22.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:22.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:22.782097+0000) 2024-03-06T10:03:23.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:23.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:23.782348+0000) 2024-03-06T10:03:24.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:24.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:24.782566+0000) 2024-03-06T10:03:25.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:25.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:25.782815+0000) 2024-03-06T10:03:26.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:26.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:26.783088+0000) 2024-03-06T10:03:27.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:27.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:27.783289+0000) 2024-03-06T10:03:28.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:28.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:28.783479+0000) 2024-03-06T10:03:29.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:29.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:29.783706+0000) 2024-03-06T10:03:30.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:30.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:30.783952+0000) 2024-03-06T10:03:31.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:31.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:31.784153+0000) 2024-03-06T10:03:32.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:32.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:32.784390+0000) 2024-03-06T10:03:33.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:33.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:33.784610+0000) 2024-03-06T10:03:34.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:34.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:34.784850+0000) 2024-03-06T10:03:35.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:35.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:35.785114+0000) 2024-03-06T10:03:36.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:36.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:36.785337+0000) 2024-03-06T10:03:37.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:37.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:37.785562+0000) 2024-03-06T10:03:38.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:38.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:38.785748+0000) 2024-03-06T10:03:39.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:39.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:39.785981+0000) 2024-03-06T10:03:40.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:40.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:40.786242+0000) 2024-03-06T10:03:41.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:41.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:41.786448+0000) 2024-03-06T10:03:42.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:42.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:42.786674+0000) 2024-03-06T10:03:43.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:43.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:03:43.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:43.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:43.786874+0000) 2024-03-06T10:03:44.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:44.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:44.787135+0000) 2024-03-06T10:03:45.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:45.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:45.787413+0000) 2024-03-06T10:03:46.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:46.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:46.787641+0000) 2024-03-06T10:03:47.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:47.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:47.787874+0000) 2024-03-06T10:03:48.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:48.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:48.788100+0000) 2024-03-06T10:03:49.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:49.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:49.788292+0000) 2024-03-06T10:03:50.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:50.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:50.788511+0000) 2024-03-06T10:03:51.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:51.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:51.788735+0000) 2024-03-06T10:03:52.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:52.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:52.788993+0000) 2024-03-06T10:03:53.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:53.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:53.789276+0000) 2024-03-06T10:03:54.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:54.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:54.789501+0000) 2024-03-06T10:03:55.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:55.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:55.789699+0000) 2024-03-06T10:03:56.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:56.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:56.789925+0000) 2024-03-06T10:03:57.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:57.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:57.790162+0000) 2024-03-06T10:03:58.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:58.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:58.790356+0000) 2024-03-06T10:03:59.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:03:59.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:03:59.790576+0000) 2024-03-06T10:04:00.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:00.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:00.790814+0000) 2024-03-06T10:04:01.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:01.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:01.791071+0000) 2024-03-06T10:04:02.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:02.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:02.791327+0000) 2024-03-06T10:04:03.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:03.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:03.791605+0000) 2024-03-06T10:04:04.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:04.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:04.791844+0000) 2024-03-06T10:04:05.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:05.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:05.792110+0000) 2024-03-06T10:04:06.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:06.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:06.792341+0000) 2024-03-06T10:04:07.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:07.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:07.792564+0000) 2024-03-06T10:04:08.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:08.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:08.792784+0000) 2024-03-06T10:04:09.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:09.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:09.793041+0000) 2024-03-06T10:04:10.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:10.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:10.793313+0000) 2024-03-06T10:04:11.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:11.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:11.793542+0000) 2024-03-06T10:04:12.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:12.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:12.793824+0000) 2024-03-06T10:04:13.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:13.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:13.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:13.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:13.794157+0000) 2024-03-06T10:04:14.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:14.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:14.794437+0000) 2024-03-06T10:04:15.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:15.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:15.794712+0000) 2024-03-06T10:04:16.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:16.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:16.794993+0000) 2024-03-06T10:04:17.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:17.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:17.795334+0000) 2024-03-06T10:04:18.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:18.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:18.795613+0000) 2024-03-06T10:04:19.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:19.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:19.795884+0000) 2024-03-06T10:04:20.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:20.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:20.796171+0000) 2024-03-06T10:04:21.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:21.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:21.796452+0000) 2024-03-06T10:04:22.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:22.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:22.796734+0000) 2024-03-06T10:04:23.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:23.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:23.797056+0000) 2024-03-06T10:04:24.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:24.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:24.797330+0000) 2024-03-06T10:04:25.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:25.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:25.797599+0000) 2024-03-06T10:04:26.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:26.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:26.797862+0000) 2024-03-06T10:04:27.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:27.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:27.798151+0000) 2024-03-06T10:04:28.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:28.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:28.798433+0000) 2024-03-06T10:04:29.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:29.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:29.798713+0000) 2024-03-06T10:04:30.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:30.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:30.799045+0000) 2024-03-06T10:04:31.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:31.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:31.799246+0000) 2024-03-06T10:04:32.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:32.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:32.799474+0000) 2024-03-06T10:04:33.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:33.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:33.799762+0000) 2024-03-06T10:04:34.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:34.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:34.799998+0000) 2024-03-06T10:04:35.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:35.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:35.800267+0000) 2024-03-06T10:04:36.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:36.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:36.800494+0000) 2024-03-06T10:04:37.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:37.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:37.800714+0000) 2024-03-06T10:04:38.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:38.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:38.800985+0000) 2024-03-06T10:04:39.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:39.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:39.801288+0000) 2024-03-06T10:04:40.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:40.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:40.801482+0000) 2024-03-06T10:04:41.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:41.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:41.801706+0000) 2024-03-06T10:04:42.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:42.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:42.801929+0000) 2024-03-06T10:04:43.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:43.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:04:43.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:43.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:43.802110+0000) 2024-03-06T10:04:44.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:44.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:44.802334+0000) 2024-03-06T10:04:45.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:45.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:45.802576+0000) 2024-03-06T10:04:46.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:46.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:46.802769+0000) 2024-03-06T10:04:47.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:47.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:47.802940+0000) 2024-03-06T10:04:48.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:48.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:48.803144+0000) 2024-03-06T10:04:49.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:49.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:49.803372+0000) 2024-03-06T10:04:50.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:50.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:50.803613+0000) 2024-03-06T10:04:51.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:51.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:51.803841+0000) 2024-03-06T10:04:52.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:52.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:52.804113+0000) 2024-03-06T10:04:53.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:53.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:53.804334+0000) 2024-03-06T10:04:54.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:54.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:54.804519+0000) 2024-03-06T10:04:55.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:55.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:55.804759+0000) 2024-03-06T10:04:56.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:56.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:56.804980+0000) 2024-03-06T10:04:57.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:57.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:57.805229+0000) 2024-03-06T10:04:58.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:58.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:58.805450+0000) 2024-03-06T10:04:59.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:04:59.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:04:59.805687+0000) 2024-03-06T10:05:00.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:00.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:00.805925+0000) 2024-03-06T10:05:01.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:01.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:01.806174+0000) 2024-03-06T10:05:02.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:02.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:02.806408+0000) 2024-03-06T10:05:03.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:03.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:03.806654+0000) 2024-03-06T10:05:04.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:04.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:04.806895+0000) 2024-03-06T10:05:05.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:05.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:05.807152+0000) 2024-03-06T10:05:06.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:06.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:06.807422+0000) 2024-03-06T10:05:07.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:07.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:07.807701+0000) 2024-03-06T10:05:08.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:08.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:08.807980+0000) 2024-03-06T10:05:09.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:09.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:09.808299+0000) 2024-03-06T10:05:10.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:10.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:10.808596+0000) 2024-03-06T10:05:11.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:11.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:11.808839+0000) 2024-03-06T10:05:12.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:12.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:12.809155+0000) 2024-03-06T10:05:13.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:13.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:13.809435+0000) 2024-03-06T10:05:13.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:13.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:14.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:14.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:14.809630+0000) 2024-03-06T10:05:15.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:15.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:15.809874+0000) 2024-03-06T10:05:16.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:16.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:16.810157+0000) 2024-03-06T10:05:17.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:17.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:17.810448+0000) 2024-03-06T10:05:18.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:18.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:18.810705+0000) 2024-03-06T10:05:19.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:19.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:19.810945+0000) 2024-03-06T10:05:20.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:20.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:20.811274+0000) 2024-03-06T10:05:21.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:21.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:21.811520+0000) 2024-03-06T10:05:22.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:22.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:22.811803+0000) 2024-03-06T10:05:23.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:23.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:23.812121+0000) 2024-03-06T10:05:24.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:24.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:24.812401+0000) 2024-03-06T10:05:25.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:25.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:25.812679+0000) 2024-03-06T10:05:26.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:26.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:26.812919+0000) 2024-03-06T10:05:27.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:27.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:27.813149+0000) 2024-03-06T10:05:28.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:28.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:28.813407+0000) 2024-03-06T10:05:29.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:29.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:29.813661+0000) 2024-03-06T10:05:30.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:30.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:30.813933+0000) 2024-03-06T10:05:31.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:31.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:31.814172+0000) 2024-03-06T10:05:32.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:32.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:32.814419+0000) 2024-03-06T10:05:33.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:33.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:33.814661+0000) 2024-03-06T10:05:34.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:34.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:34.814913+0000) 2024-03-06T10:05:35.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:35.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:35.815139+0000) 2024-03-06T10:05:36.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:36.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:36.815361+0000) 2024-03-06T10:05:37.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:37.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:37.815604+0000) 2024-03-06T10:05:38.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:38.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:38.815841+0000) 2024-03-06T10:05:39.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:39.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:39.816117+0000) 2024-03-06T10:05:40.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:40.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:40.816350+0000) 2024-03-06T10:05:41.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:41.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:41.816542+0000) 2024-03-06T10:05:42.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:42.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:42.816775+0000) 2024-03-06T10:05:43.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:43.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:43.816998+0000) 2024-03-06T10:05:44.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:44.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:05:44.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:44.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:44.817284+0000) 2024-03-06T10:05:45.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:45.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:45.817481+0000) 2024-03-06T10:05:46.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:46.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:46.817720+0000) 2024-03-06T10:05:47.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:47.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:47.817956+0000) 2024-03-06T10:05:48.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:48.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:48.818154+0000) 2024-03-06T10:05:49.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:49.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:49.818378+0000) 2024-03-06T10:05:50.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:50.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:50.818612+0000) 2024-03-06T10:05:51.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:51.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:51.818802+0000) 2024-03-06T10:05:52.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:52.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:52.819062+0000) 2024-03-06T10:05:53.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:53.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:53.819298+0000) 2024-03-06T10:05:54.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:54.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:54.819521+0000) 2024-03-06T10:05:55.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:55.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:55.819729+0000) 2024-03-06T10:05:56.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:56.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:56.819964+0000) 2024-03-06T10:05:57.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:57.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:57.820259+0000) 2024-03-06T10:05:58.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:58.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:58.820487+0000) 2024-03-06T10:05:59.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:05:59.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:05:59.820684+0000) 2024-03-06T10:06:00.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:00.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:00.820897+0000) 2024-03-06T10:06:01.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:01.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:01.821144+0000) 2024-03-06T10:06:02.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:02.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:02.821427+0000) 2024-03-06T10:06:03.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:03.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:03.821709+0000) 2024-03-06T10:06:04.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:04.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:04.821964+0000) 2024-03-06T10:06:05.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:05.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:05.822246+0000) 2024-03-06T10:06:06.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:06.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:06.822455+0000) 2024-03-06T10:06:07.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:07.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:07.822687+0000) 2024-03-06T10:06:08.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:08.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:08.822870+0000) 2024-03-06T10:06:09.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:09.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:09.823166+0000) 2024-03-06T10:06:10.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:10.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:10.823373+0000) 2024-03-06T10:06:11.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:11.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:11.823604+0000) 2024-03-06T10:06:12.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:12.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:12.823838+0000) 2024-03-06T10:06:13.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:13.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:13.824120+0000) 2024-03-06T10:06:14.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:14.367 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:14.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:14.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:14.824379+0000) 2024-03-06T10:06:15.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:15.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:15.824647+0000) 2024-03-06T10:06:16.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:16.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:16.824930+0000) 2024-03-06T10:06:17.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:17.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:17.825223+0000) 2024-03-06T10:06:18.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:18.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:18.825512+0000) 2024-03-06T10:06:19.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:19.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:19.825788+0000) 2024-03-06T10:06:20.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:20.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:20.826143+0000) 2024-03-06T10:06:21.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:21.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:21.826339+0000) 2024-03-06T10:06:22.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:22.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:22.826583+0000) 2024-03-06T10:06:23.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:23.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:23.826849+0000) 2024-03-06T10:06:24.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:24.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:24.827158+0000) 2024-03-06T10:06:25.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:25.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:25.827405+0000) 2024-03-06T10:06:26.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:26.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:26.827632+0000) 2024-03-06T10:06:27.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:27.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:27.827872+0000) 2024-03-06T10:06:28.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:28.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:28.828143+0000) 2024-03-06T10:06:29.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:29.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:29.828329+0000) 2024-03-06T10:06:30.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:30.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:30.828561+0000) 2024-03-06T10:06:31.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:31.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:31.828783+0000) 2024-03-06T10:06:32.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:32.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:32.829040+0000) 2024-03-06T10:06:33.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:33.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:33.829236+0000) 2024-03-06T10:06:34.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:34.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:34.829476+0000) 2024-03-06T10:06:35.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:35.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:35.829725+0000) 2024-03-06T10:06:36.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:36.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:36.829961+0000) 2024-03-06T10:06:37.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:37.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:37.830228+0000) 2024-03-06T10:06:38.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:38.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:38.830464+0000) 2024-03-06T10:06:39.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:39.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:39.830673+0000) 2024-03-06T10:06:40.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:40.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:40.830906+0000) 2024-03-06T10:06:41.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:41.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:41.831127+0000) 2024-03-06T10:06:42.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:42.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:42.831339+0000) 2024-03-06T10:06:43.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:43.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:43.831576+0000) 2024-03-06T10:06:44.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:44.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:06:44.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:44.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:44.831808+0000) 2024-03-06T10:06:45.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:45.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:45.832089+0000) 2024-03-06T10:06:46.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:46.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:46.832317+0000) 2024-03-06T10:06:47.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:47.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:47.832558+0000) 2024-03-06T10:06:48.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:48.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:48.832769+0000) 2024-03-06T10:06:49.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:49.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:49.832999+0000) 2024-03-06T10:06:50.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:50.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:50.833246+0000) 2024-03-06T10:06:51.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:51.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:51.833532+0000) 2024-03-06T10:06:52.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:52.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:52.833799+0000) 2024-03-06T10:06:53.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:53.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:53.834123+0000) 2024-03-06T10:06:54.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:54.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:54.834330+0000) 2024-03-06T10:06:55.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:55.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:55.834564+0000) 2024-03-06T10:06:56.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:56.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:56.834790+0000) 2024-03-06T10:06:57.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:57.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:57.835061+0000) 2024-03-06T10:06:58.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:58.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:58.835305+0000) 2024-03-06T10:06:59.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:06:59.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:06:59.835543+0000) 2024-03-06T10:07:00.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:00.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:00.835804+0000) 2024-03-06T10:07:01.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:01.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:01.835991+0000) 2024-03-06T10:07:02.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:02.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:02.836314+0000) 2024-03-06T10:07:03.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:03.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:03.836507+0000) 2024-03-06T10:07:04.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:04.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:04.836726+0000) 2024-03-06T10:07:05.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:05.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:05.836951+0000) 2024-03-06T10:07:06.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:06.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:06.837175+0000) 2024-03-06T10:07:07.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:07.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:07.837409+0000) 2024-03-06T10:07:08.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:08.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:08.837630+0000) 2024-03-06T10:07:09.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:09.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:09.837914+0000) 2024-03-06T10:07:10.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:10.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:10.838214+0000) 2024-03-06T10:07:11.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:11.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:11.838498+0000) 2024-03-06T10:07:12.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:12.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:12.838750+0000) 2024-03-06T10:07:13.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:13.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:13.838988+0000) 2024-03-06T10:07:14.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:14.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:14.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:14.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:14.839218+0000) 2024-03-06T10:07:15.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:15.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:15.839430+0000) 2024-03-06T10:07:16.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:16.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:16.839686+0000) 2024-03-06T10:07:17.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:17.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:17.839973+0000) 2024-03-06T10:07:18.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:18.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:18.840293+0000) 2024-03-06T10:07:19.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:19.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:19.840502+0000) 2024-03-06T10:07:20.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:20.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:20.840733+0000) 2024-03-06T10:07:21.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:21.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:21.840958+0000) 2024-03-06T10:07:22.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:22.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:22.841187+0000) 2024-03-06T10:07:23.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:23.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:23.841459+0000) 2024-03-06T10:07:24.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:24.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:24.841703+0000) 2024-03-06T10:07:25.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:25.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:25.841925+0000) 2024-03-06T10:07:26.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:26.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:26.842164+0000) 2024-03-06T10:07:27.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:27.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:27.842396+0000) 2024-03-06T10:07:28.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:28.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:28.842636+0000) 2024-03-06T10:07:29.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:29.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:29.842828+0000) 2024-03-06T10:07:30.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:30.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:30.843124+0000) 2024-03-06T10:07:31.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:31.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:31.843349+0000) 2024-03-06T10:07:32.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:32.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:32.843567+0000) 2024-03-06T10:07:33.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:33.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:33.843779+0000) 2024-03-06T10:07:34.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:34.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:34.844000+0000) 2024-03-06T10:07:35.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:35.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:35.844259+0000) 2024-03-06T10:07:36.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:36.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:36.844484+0000) 2024-03-06T10:07:37.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:37.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:37.844715+0000) 2024-03-06T10:07:38.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:38.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:38.844947+0000) 2024-03-06T10:07:39.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:39.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:39.845165+0000) 2024-03-06T10:07:40.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:40.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:40.845421+0000) 2024-03-06T10:07:41.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:41.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:41.845671+0000) 2024-03-06T10:07:42.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:42.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:42.845926+0000) 2024-03-06T10:07:43.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:43.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:43.846165+0000) 2024-03-06T10:07:44.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:44.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:44.846334+0000) 2024-03-06T10:07:45.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:45.010 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:07:45.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:45.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:45.846573+0000) 2024-03-06T10:07:46.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:46.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:46.846782+0000) 2024-03-06T10:07:47.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:47.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:47.846993+0000) 2024-03-06T10:07:48.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:48.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:48.847252+0000) 2024-03-06T10:07:49.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:49.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:49.847486+0000) 2024-03-06T10:07:50.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:50.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:50.847766+0000) 2024-03-06T10:07:51.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:51.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:51.847990+0000) 2024-03-06T10:07:52.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:52.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:52.848244+0000) 2024-03-06T10:07:53.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:53.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:53.848476+0000) 2024-03-06T10:07:54.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:54.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:54.848665+0000) 2024-03-06T10:07:55.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:55.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:55.848904+0000) 2024-03-06T10:07:56.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:56.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:56.849124+0000) 2024-03-06T10:07:57.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:57.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:57.849347+0000) 2024-03-06T10:07:58.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:58.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:58.849543+0000) 2024-03-06T10:07:59.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:07:59.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:07:59.849776+0000) 2024-03-06T10:08:00.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:00.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:00.850059+0000) 2024-03-06T10:08:01.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:01.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:01.850266+0000) 2024-03-06T10:08:02.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:02.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:02.850683+0000) 2024-03-06T10:08:03.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:03.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:03.850933+0000) 2024-03-06T10:08:04.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:04.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:04.851192+0000) 2024-03-06T10:08:05.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:05.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:05.851387+0000) 2024-03-06T10:08:06.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:06.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:06.851633+0000) 2024-03-06T10:08:07.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:07.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:07.851884+0000) 2024-03-06T10:08:08.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:08.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:08.852144+0000) 2024-03-06T10:08:09.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:09.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:09.852388+0000) 2024-03-06T10:08:10.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:10.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:10.852648+0000) 2024-03-06T10:08:11.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:11.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:11.852891+0000) 2024-03-06T10:08:12.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:12.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:12.853143+0000) 2024-03-06T10:08:13.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:13.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:13.853394+0000) 2024-03-06T10:08:14.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:14.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:14.853586+0000) 2024-03-06T10:08:15.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:15.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:15.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:15.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:15.853796+0000) 2024-03-06T10:08:16.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:16.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:16.853998+0000) 2024-03-06T10:08:17.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:17.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:17.854259+0000) 2024-03-06T10:08:18.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:18.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:18.854429+0000) 2024-03-06T10:08:19.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:19.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:19.854658+0000) 2024-03-06T10:08:20.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:20.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:20.854919+0000) 2024-03-06T10:08:21.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:21.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:21.855154+0000) 2024-03-06T10:08:22.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:22.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:22.855409+0000) 2024-03-06T10:08:23.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:23.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:23.855702+0000) 2024-03-06T10:08:24.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:24.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:24.855909+0000) 2024-03-06T10:08:25.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:25.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:25.856118+0000) 2024-03-06T10:08:26.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:26.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:26.856295+0000) 2024-03-06T10:08:27.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:27.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:27.856519+0000) 2024-03-06T10:08:28.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:28.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:28.856755+0000) 2024-03-06T10:08:29.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:29.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:29.856939+0000) 2024-03-06T10:08:30.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:30.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:30.857166+0000) 2024-03-06T10:08:31.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:31.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:31.857384+0000) 2024-03-06T10:08:32.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:32.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:32.857604+0000) 2024-03-06T10:08:33.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:33.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:33.857873+0000) 2024-03-06T10:08:34.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:34.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:34.858062+0000) 2024-03-06T10:08:35.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:35.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:35.858246+0000) 2024-03-06T10:08:36.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:36.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:36.858427+0000) 2024-03-06T10:08:37.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:37.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:37.858619+0000) 2024-03-06T10:08:38.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:38.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:38.858853+0000) 2024-03-06T10:08:39.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:39.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:39.859130+0000) 2024-03-06T10:08:40.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:40.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:40.859398+0000) 2024-03-06T10:08:41.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:41.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:41.859613+0000) 2024-03-06T10:08:42.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:42.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:42.859817+0000) 2024-03-06T10:08:43.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:43.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:43.860051+0000) 2024-03-06T10:08:44.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:44.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:44.860331+0000) 2024-03-06T10:08:45.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:45.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:08:45.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:45.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:45.860596+0000) 2024-03-06T10:08:46.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:46.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:46.860762+0000) 2024-03-06T10:08:47.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:47.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:47.860984+0000) 2024-03-06T10:08:48.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:48.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:48.861289+0000) 2024-03-06T10:08:49.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:49.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:49.861474+0000) 2024-03-06T10:08:50.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:50.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:50.861699+0000) 2024-03-06T10:08:51.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:51.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:51.861930+0000) 2024-03-06T10:08:52.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:52.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:52.862165+0000) 2024-03-06T10:08:53.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:53.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:53.862410+0000) 2024-03-06T10:08:54.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:54.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:54.862664+0000) 2024-03-06T10:08:55.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:55.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:55.862905+0000) 2024-03-06T10:08:56.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:56.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:56.863166+0000) 2024-03-06T10:08:57.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:57.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:57.863369+0000) 2024-03-06T10:08:58.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:58.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:58.863598+0000) 2024-03-06T10:08:59.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:08:59.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:08:59.863826+0000) 2024-03-06T10:09:00.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:00.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:00.864105+0000) 2024-03-06T10:09:01.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:01.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:01.864329+0000) 2024-03-06T10:09:02.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:02.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:02.864549+0000) 2024-03-06T10:09:03.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:03.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:03.864771+0000) 2024-03-06T10:09:04.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:04.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:04.864992+0000) 2024-03-06T10:09:05.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:05.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:05.865248+0000) 2024-03-06T10:09:06.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:06.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:06.865496+0000) 2024-03-06T10:09:07.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:07.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:07.865762+0000) 2024-03-06T10:09:08.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:08.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:08.865973+0000) 2024-03-06T10:09:09.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:09.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:09.866198+0000) 2024-03-06T10:09:10.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:10.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:10.866443+0000) 2024-03-06T10:09:11.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:11.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:11.866674+0000) 2024-03-06T10:09:12.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:12.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:12.866897+0000) 2024-03-06T10:09:13.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:13.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:13.867137+0000) 2024-03-06T10:09:14.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:14.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:14.867360+0000) 2024-03-06T10:09:15.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:15.639 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:15.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:15.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:15.867580+0000) 2024-03-06T10:09:16.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:16.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:16.867813+0000) 2024-03-06T10:09:17.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:17.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:17.868074+0000) 2024-03-06T10:09:18.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:18.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:18.868273+0000) 2024-03-06T10:09:19.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:19.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:19.868491+0000) 2024-03-06T10:09:20.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:20.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:20.868740+0000) 2024-03-06T10:09:21.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:21.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:21.868950+0000) 2024-03-06T10:09:22.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:22.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:22.869202+0000) 2024-03-06T10:09:23.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:23.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:23.869427+0000) 2024-03-06T10:09:24.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:24.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:24.869646+0000) 2024-03-06T10:09:25.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:25.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:25.869886+0000) 2024-03-06T10:09:26.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:26.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:26.870128+0000) 2024-03-06T10:09:27.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:27.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:27.870331+0000) 2024-03-06T10:09:28.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:28.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:28.870520+0000) 2024-03-06T10:09:29.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:29.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:29.870764+0000) 2024-03-06T10:09:30.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:30.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:30.871051+0000) 2024-03-06T10:09:31.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:31.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:31.871276+0000) 2024-03-06T10:09:32.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:32.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:32.871523+0000) 2024-03-06T10:09:33.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:33.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:33.871769+0000) 2024-03-06T10:09:34.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:34.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:34.872062+0000) 2024-03-06T10:09:35.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:35.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:35.872314+0000) 2024-03-06T10:09:36.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:36.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:36.872518+0000) 2024-03-06T10:09:37.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:37.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:37.872794+0000) 2024-03-06T10:09:38.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:38.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:38.873038+0000) 2024-03-06T10:09:39.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:39.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:39.873264+0000) 2024-03-06T10:09:40.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:40.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:40.873495+0000) 2024-03-06T10:09:41.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:41.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:41.873717+0000) 2024-03-06T10:09:42.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:42.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:42.873914+0000) 2024-03-06T10:09:43.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:43.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:43.874158+0000) 2024-03-06T10:09:44.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:44.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:44.874381+0000) 2024-03-06T10:09:45.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:45.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:09:45.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:45.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:45.874561+0000) 2024-03-06T10:09:46.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:46.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:46.874791+0000) 2024-03-06T10:09:47.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:47.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:47.874980+0000) 2024-03-06T10:09:48.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:48.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:48.875228+0000) 2024-03-06T10:09:49.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:49.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:49.875466+0000) 2024-03-06T10:09:50.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:50.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:50.875699+0000) 2024-03-06T10:09:51.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:51.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:51.875921+0000) 2024-03-06T10:09:52.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:52.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:52.876124+0000) 2024-03-06T10:09:53.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:53.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:53.876347+0000) 2024-03-06T10:09:54.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:54.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:54.876578+0000) 2024-03-06T10:09:55.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:55.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:55.876817+0000) 2024-03-06T10:09:56.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:56.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:56.877114+0000) 2024-03-06T10:09:57.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:57.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:57.877323+0000) 2024-03-06T10:09:58.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:58.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:58.877517+0000) 2024-03-06T10:09:59.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:09:59.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:09:59.877741+0000) 2024-03-06T10:10:00.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:00.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:00.877979+0000) 2024-03-06T10:10:01.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:01.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:01.878244+0000) 2024-03-06T10:10:02.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:02.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:02.878475+0000) 2024-03-06T10:10:03.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:03.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:03.878712+0000) 2024-03-06T10:10:04.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:04.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:04.879093+0000) 2024-03-06T10:10:05.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:05.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:05.879354+0000) 2024-03-06T10:10:06.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:06.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:06.879598+0000) 2024-03-06T10:10:07.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:07.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:07.879832+0000) 2024-03-06T10:10:08.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:08.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:08.880145+0000) 2024-03-06T10:10:09.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:09.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:09.880350+0000) 2024-03-06T10:10:10.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:10.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:10.880599+0000) 2024-03-06T10:10:11.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:11.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:11.880823+0000) 2024-03-06T10:10:12.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:12.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:12.881102+0000) 2024-03-06T10:10:13.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:13.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:13.881314+0000) 2024-03-06T10:10:14.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:14.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:14.881542+0000) 2024-03-06T10:10:15.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:15.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:15.881770+0000) 2024-03-06T10:10:16.047 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:16.055 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:16.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:16.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:16.882050+0000) 2024-03-06T10:10:17.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:17.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:17.882300+0000) 2024-03-06T10:10:18.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:18.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:18.882530+0000) 2024-03-06T10:10:19.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:19.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:19.882749+0000) 2024-03-06T10:10:20.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:20.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:20.883080+0000) 2024-03-06T10:10:21.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:21.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:21.883314+0000) 2024-03-06T10:10:22.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:22.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:22.883543+0000) 2024-03-06T10:10:23.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:23.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:23.883776+0000) 2024-03-06T10:10:24.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:24.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:24.883975+0000) 2024-03-06T10:10:25.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:25.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:25.884205+0000) 2024-03-06T10:10:26.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:26.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:26.884436+0000) 2024-03-06T10:10:27.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:27.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:27.884664+0000) 2024-03-06T10:10:28.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:28.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:28.884897+0000) 2024-03-06T10:10:29.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:29.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:29.885131+0000) 2024-03-06T10:10:30.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:30.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:30.885419+0000) 2024-03-06T10:10:31.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:31.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:31.885612+0000) 2024-03-06T10:10:32.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:32.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:32.885827+0000) 2024-03-06T10:10:33.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:33.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:33.886063+0000) 2024-03-06T10:10:34.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:34.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:34.886295+0000) 2024-03-06T10:10:35.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:35.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:35.886513+0000) 2024-03-06T10:10:36.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:36.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:36.886739+0000) 2024-03-06T10:10:37.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:37.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:37.886975+0000) 2024-03-06T10:10:38.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:38.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:38.887217+0000) 2024-03-06T10:10:39.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:39.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:39.887446+0000) 2024-03-06T10:10:40.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:40.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:40.887681+0000) 2024-03-06T10:10:41.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:41.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:41.887908+0000) 2024-03-06T10:10:42.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:42.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:42.888136+0000) 2024-03-06T10:10:43.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:43.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:43.888328+0000) 2024-03-06T10:10:44.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:44.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:44.888544+0000) 2024-03-06T10:10:45.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:45.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:45.888757+0000) 2024-03-06T10:10:46.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:46.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:10:46.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:46.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:46.889243+0000) 2024-03-06T10:10:47.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:47.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:47.889470+0000) 2024-03-06T10:10:48.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:48.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:48.889705+0000) 2024-03-06T10:10:49.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:49.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:49.889935+0000) 2024-03-06T10:10:50.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:50.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:50.890188+0000) 2024-03-06T10:10:51.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:51.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:51.890392+0000) 2024-03-06T10:10:52.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:52.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:52.890586+0000) 2024-03-06T10:10:53.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:53.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:53.890834+0000) 2024-03-06T10:10:54.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:54.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:54.891111+0000) 2024-03-06T10:10:55.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:55.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:55.891326+0000) 2024-03-06T10:10:56.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:56.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:56.891517+0000) 2024-03-06T10:10:57.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:57.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:57.891714+0000) 2024-03-06T10:10:58.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:58.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:58.891947+0000) 2024-03-06T10:10:59.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:10:59.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:10:59.892197+0000) 2024-03-06T10:11:00.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:00.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:00.892430+0000) 2024-03-06T10:11:01.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:01.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:01.892616+0000) 2024-03-06T10:11:02.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:02.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:02.892825+0000) 2024-03-06T10:11:03.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:03.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:03.893077+0000) 2024-03-06T10:11:04.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:04.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:04.893218+0000) 2024-03-06T10:11:05.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:05.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:05.893435+0000) 2024-03-06T10:11:06.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:06.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:06.893668+0000) 2024-03-06T10:11:07.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:07.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:07.893921+0000) 2024-03-06T10:11:08.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:08.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:08.894160+0000) 2024-03-06T10:11:09.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:09.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:09.894389+0000) 2024-03-06T10:11:10.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:10.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:10.894632+0000) 2024-03-06T10:11:11.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:11.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:11.894859+0000) 2024-03-06T10:11:12.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:12.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:12.895113+0000) 2024-03-06T10:11:13.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:13.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:13.895424+0000) 2024-03-06T10:11:14.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:14.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:14.895636+0000) 2024-03-06T10:11:15.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:15.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:15.895881+0000) 2024-03-06T10:11:16.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:16.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:16.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:16.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:16.896132+0000) 2024-03-06T10:11:17.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:17.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:17.896325+0000) 2024-03-06T10:11:18.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:18.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:18.896502+0000) 2024-03-06T10:11:19.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:19.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:19.896747+0000) 2024-03-06T10:11:20.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:20.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:20.897053+0000) 2024-03-06T10:11:21.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:21.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:21.897306+0000) 2024-03-06T10:11:22.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:22.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:22.897530+0000) 2024-03-06T10:11:23.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:23.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:23.897719+0000) 2024-03-06T10:11:24.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:24.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:24.897905+0000) 2024-03-06T10:11:25.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:25.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:25.898106+0000) 2024-03-06T10:11:26.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:26.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:26.898305+0000) 2024-03-06T10:11:27.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:27.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:27.898536+0000) 2024-03-06T10:11:28.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:28.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:28.898754+0000) 2024-03-06T10:11:29.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:29.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:29.898979+0000) 2024-03-06T10:11:30.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:30.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:30.899236+0000) 2024-03-06T10:11:31.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:31.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:31.899432+0000) 2024-03-06T10:11:32.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:32.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:32.899667+0000) 2024-03-06T10:11:33.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:33.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:33.899899+0000) 2024-03-06T10:11:34.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:34.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:34.900152+0000) 2024-03-06T10:11:35.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:35.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:35.900371+0000) 2024-03-06T10:11:36.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:36.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:36.900586+0000) 2024-03-06T10:11:37.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:37.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:37.900797+0000) 2024-03-06T10:11:38.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:38.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:38.901089+0000) 2024-03-06T10:11:39.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:39.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:39.901342+0000) 2024-03-06T10:11:40.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:40.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:40.901537+0000) 2024-03-06T10:11:41.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:41.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:41.901723+0000) 2024-03-06T10:11:42.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:42.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:42.901948+0000) 2024-03-06T10:11:43.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:43.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:43.902192+0000) 2024-03-06T10:11:44.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:44.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:44.902456+0000) 2024-03-06T10:11:45.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:45.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:45.902715+0000) 2024-03-06T10:11:46.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:46.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:11:46.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:46.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:46.902903+0000) 2024-03-06T10:11:47.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:47.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:47.903137+0000) 2024-03-06T10:11:48.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:48.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:48.903379+0000) 2024-03-06T10:11:49.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:49.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:49.903612+0000) 2024-03-06T10:11:50.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:50.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:50.903850+0000) 2024-03-06T10:11:51.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:51.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:51.904122+0000) 2024-03-06T10:11:52.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:52.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:52.904357+0000) 2024-03-06T10:11:53.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:53.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:53.904619+0000) 2024-03-06T10:11:54.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:54.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:54.904856+0000) 2024-03-06T10:11:55.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:55.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:55.905126+0000) 2024-03-06T10:11:56.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:56.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:56.905329+0000) 2024-03-06T10:11:57.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:57.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:57.905555+0000) 2024-03-06T10:11:58.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:58.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:58.905757+0000) 2024-03-06T10:11:59.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:11:59.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:11:59.905945+0000) 2024-03-06T10:12:00.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:00.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:00.906165+0000) 2024-03-06T10:12:01.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:01.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:01.906457+0000) 2024-03-06T10:12:02.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:02.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:02.906693+0000) 2024-03-06T10:12:03.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:03.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:03.906933+0000) 2024-03-06T10:12:04.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:04.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:04.907169+0000) 2024-03-06T10:12:05.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:05.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:05.907368+0000) 2024-03-06T10:12:06.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:06.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:06.907581+0000) 2024-03-06T10:12:07.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:07.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:07.907792+0000) 2024-03-06T10:12:08.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:08.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:08.907976+0000) 2024-03-06T10:12:09.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:09.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:09.908207+0000) 2024-03-06T10:12:10.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:10.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:10.908431+0000) 2024-03-06T10:12:11.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:11.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:11.908691+0000) 2024-03-06T10:12:12.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:12.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:12.908953+0000) 2024-03-06T10:12:13.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:13.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:13.909214+0000) 2024-03-06T10:12:14.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:14.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:14.909410+0000) 2024-03-06T10:12:15.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:15.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:15.909633+0000) 2024-03-06T10:12:16.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:16.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:16.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:16.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:16.909860+0000) 2024-03-06T10:12:17.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:17.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:17.910139+0000) 2024-03-06T10:12:18.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:18.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:18.910377+0000) 2024-03-06T10:12:19.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:19.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:19.910533+0000) 2024-03-06T10:12:20.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:20.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:20.910702+0000) 2024-03-06T10:12:21.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:21.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:21.910949+0000) 2024-03-06T10:12:22.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:22.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:22.911211+0000) 2024-03-06T10:12:23.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:23.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:23.911476+0000) 2024-03-06T10:12:24.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:24.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:24.911711+0000) 2024-03-06T10:12:25.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:25.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:25.911916+0000) 2024-03-06T10:12:26.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:26.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:26.912174+0000) 2024-03-06T10:12:27.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:27.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:27.912421+0000) 2024-03-06T10:12:28.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:28.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:28.912673+0000) 2024-03-06T10:12:29.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:29.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:29.912923+0000) 2024-03-06T10:12:30.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:30.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:30.913167+0000) 2024-03-06T10:12:31.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:31.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:31.913397+0000) 2024-03-06T10:12:32.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:32.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:32.913640+0000) 2024-03-06T10:12:33.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:33.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:33.913899+0000) 2024-03-06T10:12:34.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:34.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:34.914129+0000) 2024-03-06T10:12:35.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:35.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:35.914368+0000) 2024-03-06T10:12:36.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:36.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:36.914543+0000) 2024-03-06T10:12:37.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:37.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:37.914771+0000) 2024-03-06T10:12:38.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:38.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:38.915045+0000) 2024-03-06T10:12:39.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:39.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:39.915286+0000) 2024-03-06T10:12:40.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:40.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:40.915480+0000) 2024-03-06T10:12:41.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:41.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:41.915674+0000) 2024-03-06T10:12:42.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:42.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:42.915896+0000) 2024-03-06T10:12:43.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:43.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:43.916199+0000) 2024-03-06T10:12:44.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:44.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:44.916462+0000) 2024-03-06T10:12:45.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:45.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:45.916690+0000) 2024-03-06T10:12:46.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:46.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:12:46.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:46.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:46.916891+0000) 2024-03-06T10:12:47.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:47.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:47.917158+0000) 2024-03-06T10:12:48.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:48.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:48.917396+0000) 2024-03-06T10:12:49.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:49.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:49.917649+0000) 2024-03-06T10:12:50.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:50.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:50.917884+0000) 2024-03-06T10:12:51.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:51.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:51.918123+0000) 2024-03-06T10:12:52.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:52.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:52.918344+0000) 2024-03-06T10:12:53.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:53.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:53.918588+0000) 2024-03-06T10:12:54.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:54.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:54.918812+0000) 2024-03-06T10:12:55.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:55.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:55.919082+0000) 2024-03-06T10:12:56.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:56.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:56.919294+0000) 2024-03-06T10:12:57.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:57.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:57.919538+0000) 2024-03-06T10:12:58.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:58.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:58.919751+0000) 2024-03-06T10:12:59.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:12:59.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:12:59.919984+0000) 2024-03-06T10:13:00.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:00.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:00.920245+0000) 2024-03-06T10:13:01.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:01.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:01.920434+0000) 2024-03-06T10:13:02.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:02.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:02.920601+0000) 2024-03-06T10:13:03.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:03.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:03.920758+0000) 2024-03-06T10:13:04.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:04.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:04.920987+0000) 2024-03-06T10:13:05.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:05.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:05.921264+0000) 2024-03-06T10:13:06.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:06.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:06.921491+0000) 2024-03-06T10:13:07.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:07.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:07.921721+0000) 2024-03-06T10:13:08.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:08.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:08.921965+0000) 2024-03-06T10:13:09.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:09.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:09.922256+0000) 2024-03-06T10:13:10.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:10.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:10.922466+0000) 2024-03-06T10:13:11.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:11.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:11.922676+0000) 2024-03-06T10:13:12.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:12.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:12.922859+0000) 2024-03-06T10:13:13.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:13.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:13.923097+0000) 2024-03-06T10:13:14.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:14.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:14.923280+0000) 2024-03-06T10:13:15.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:15.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:15.923466+0000) 2024-03-06T10:13:16.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:16.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:16.923676+0000) 2024-03-06T10:13:17.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:17.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:17.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:17.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:17.923923+0000) 2024-03-06T10:13:18.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:18.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:18.924130+0000) 2024-03-06T10:13:19.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:19.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:19.924371+0000) 2024-03-06T10:13:20.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:20.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:20.924615+0000) 2024-03-06T10:13:21.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:21.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:21.924875+0000) 2024-03-06T10:13:22.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:22.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:22.925145+0000) 2024-03-06T10:13:23.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:23.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:23.925392+0000) 2024-03-06T10:13:24.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:24.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:24.925636+0000) 2024-03-06T10:13:25.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:25.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:25.925886+0000) 2024-03-06T10:13:26.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:26.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:26.926148+0000) 2024-03-06T10:13:27.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:27.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:27.926405+0000) 2024-03-06T10:13:28.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:28.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:28.926660+0000) 2024-03-06T10:13:29.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:29.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:29.926908+0000) 2024-03-06T10:13:30.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:30.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:30.927202+0000) 2024-03-06T10:13:31.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:31.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:31.927463+0000) 2024-03-06T10:13:32.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:32.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:32.927719+0000) 2024-03-06T10:13:33.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:33.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:33.927954+0000) 2024-03-06T10:13:34.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:34.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:34.928173+0000) 2024-03-06T10:13:35.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:35.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:35.928368+0000) 2024-03-06T10:13:36.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:36.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:36.928592+0000) 2024-03-06T10:13:37.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:37.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:37.928794+0000) 2024-03-06T10:13:38.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:38.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:38.929061+0000) 2024-03-06T10:13:39.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:39.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:39.929329+0000) 2024-03-06T10:13:40.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:40.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:40.929524+0000) 2024-03-06T10:13:41.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:41.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:41.929722+0000) 2024-03-06T10:13:42.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:42.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:42.929957+0000) 2024-03-06T10:13:43.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:43.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:43.930189+0000) 2024-03-06T10:13:44.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:44.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:44.930436+0000) 2024-03-06T10:13:45.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:45.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:45.930644+0000) 2024-03-06T10:13:46.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:46.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:46.930889+0000) 2024-03-06T10:13:47.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:47.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:13:47.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:47.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:47.931124+0000) 2024-03-06T10:13:48.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:48.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:48.931400+0000) 2024-03-06T10:13:49.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:49.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:49.931649+0000) 2024-03-06T10:13:50.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:50.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:50.931915+0000) 2024-03-06T10:13:51.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:51.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:51.932157+0000) 2024-03-06T10:13:52.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:52.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:52.932404+0000) 2024-03-06T10:13:53.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:53.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:53.932605+0000) 2024-03-06T10:13:54.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:54.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:54.932839+0000) 2024-03-06T10:13:55.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:55.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:55.933143+0000) 2024-03-06T10:13:56.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:56.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:56.933382+0000) 2024-03-06T10:13:57.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:57.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:57.933606+0000) 2024-03-06T10:13:58.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:58.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:58.933798+0000) 2024-03-06T10:13:59.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:13:59.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:13:59.934090+0000) 2024-03-06T10:14:00.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:00.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:00.934323+0000) 2024-03-06T10:14:01.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:01.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:01.934576+0000) 2024-03-06T10:14:02.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:02.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:02.934815+0000) 2024-03-06T10:14:03.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:03.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:03.935110+0000) 2024-03-06T10:14:04.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:04.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:04.935349+0000) 2024-03-06T10:14:05.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:05.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:05.935602+0000) 2024-03-06T10:14:06.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:06.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:06.935773+0000) 2024-03-06T10:14:07.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:07.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:07.935988+0000) 2024-03-06T10:14:08.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:08.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:08.936225+0000) 2024-03-06T10:14:09.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:09.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:09.936437+0000) 2024-03-06T10:14:10.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:10.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:10.936681+0000) 2024-03-06T10:14:11.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:11.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:11.936870+0000) 2024-03-06T10:14:12.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:12.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:12.937103+0000) 2024-03-06T10:14:13.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:13.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:13.937327+0000) 2024-03-06T10:14:14.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:14.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:14.937563+0000) 2024-03-06T10:14:15.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:15.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:15.937781+0000) 2024-03-06T10:14:16.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:16.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:16.938036+0000) 2024-03-06T10:14:17.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:17.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:17.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:17.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:17.938283+0000) 2024-03-06T10:14:18.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:18.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:18.938521+0000) 2024-03-06T10:14:19.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:19.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:19.938760+0000) 2024-03-06T10:14:20.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:20.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:20.939031+0000) 2024-03-06T10:14:21.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:21.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:21.939221+0000) 2024-03-06T10:14:22.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:22.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:22.939474+0000) 2024-03-06T10:14:23.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:23.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:23.939700+0000) 2024-03-06T10:14:24.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:24.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:24.939934+0000) 2024-03-06T10:14:25.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:25.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:25.940177+0000) 2024-03-06T10:14:26.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:26.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:26.940426+0000) 2024-03-06T10:14:27.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:27.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:27.940658+0000) 2024-03-06T10:14:28.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:28.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:28.940898+0000) 2024-03-06T10:14:29.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:29.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:29.941153+0000) 2024-03-06T10:14:30.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:30.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:30.941414+0000) 2024-03-06T10:14:31.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:31.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:31.941635+0000) 2024-03-06T10:14:32.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:32.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:32.941869+0000) 2024-03-06T10:14:33.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:33.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:33.942117+0000) 2024-03-06T10:14:34.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:34.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:34.942380+0000) 2024-03-06T10:14:35.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:35.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:35.942634+0000) 2024-03-06T10:14:36.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:36.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:36.942936+0000) 2024-03-06T10:14:37.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:37.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:37.943181+0000) 2024-03-06T10:14:38.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:38.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:38.943424+0000) 2024-03-06T10:14:39.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:39.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:39.943673+0000) 2024-03-06T10:14:40.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:40.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:40.943906+0000) 2024-03-06T10:14:41.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:41.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:41.944139+0000) 2024-03-06T10:14:42.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:42.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:42.944382+0000) 2024-03-06T10:14:43.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:43.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:43.944615+0000) 2024-03-06T10:14:44.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:44.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:44.944857+0000) 2024-03-06T10:14:45.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:45.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:45.945124+0000) 2024-03-06T10:14:46.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:46.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:46.945352+0000) 2024-03-06T10:14:47.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:47.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:14:47.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:47.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:47.945571+0000) 2024-03-06T10:14:48.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:48.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:48.945818+0000) 2024-03-06T10:14:49.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:49.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:49.946136+0000) 2024-03-06T10:14:50.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:50.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:50.946389+0000) 2024-03-06T10:14:51.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:51.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:51.946632+0000) 2024-03-06T10:14:52.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:52.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:52.946837+0000) 2024-03-06T10:14:53.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:53.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:53.947064+0000) 2024-03-06T10:14:54.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:54.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:54.947259+0000) 2024-03-06T10:14:55.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:55.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:55.947465+0000) 2024-03-06T10:14:56.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:56.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:56.947693+0000) 2024-03-06T10:14:57.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:57.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:57.947925+0000) 2024-03-06T10:14:58.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:58.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:58.948160+0000) 2024-03-06T10:14:59.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:14:59.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:14:59.948387+0000) 2024-03-06T10:15:00.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:00.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:00.948624+0000) 2024-03-06T10:15:01.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:01.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:01.948848+0000) 2024-03-06T10:15:02.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:02.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:02.949094+0000) 2024-03-06T10:15:03.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:03.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:03.949255+0000) 2024-03-06T10:15:04.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:04.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:04.949460+0000) 2024-03-06T10:15:05.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:05.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:05.949688+0000) 2024-03-06T10:15:06.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:06.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:06.949919+0000) 2024-03-06T10:15:07.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:07.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:07.950162+0000) 2024-03-06T10:15:08.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:08.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:08.950388+0000) 2024-03-06T10:15:09.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:09.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:09.950622+0000) 2024-03-06T10:15:10.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:10.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:10.950895+0000) 2024-03-06T10:15:11.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:11.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:11.951129+0000) 2024-03-06T10:15:12.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:12.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:12.951360+0000) 2024-03-06T10:15:13.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:13.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:13.951608+0000) 2024-03-06T10:15:14.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:14.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:14.951844+0000) 2024-03-06T10:15:15.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:15.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:15.952116+0000) 2024-03-06T10:15:16.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:16.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:16.952346+0000) 2024-03-06T10:15:17.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:17.921 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:17.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:17.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:17.952529+0000) 2024-03-06T10:15:18.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:18.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:18.952757+0000) 2024-03-06T10:15:19.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:19.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:19.952967+0000) 2024-03-06T10:15:20.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:20.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:20.953212+0000) 2024-03-06T10:15:21.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:21.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:21.953442+0000) 2024-03-06T10:15:22.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:22.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:22.953688+0000) 2024-03-06T10:15:23.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:23.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:23.953921+0000) 2024-03-06T10:15:24.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:24.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:24.954154+0000) 2024-03-06T10:15:25.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:25.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:25.954340+0000) 2024-03-06T10:15:26.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:26.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:26.954567+0000) 2024-03-06T10:15:27.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:27.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:27.954797+0000) 2024-03-06T10:15:28.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:28.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:28.955075+0000) 2024-03-06T10:15:29.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:29.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:29.955325+0000) 2024-03-06T10:15:30.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:30.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:30.955570+0000) 2024-03-06T10:15:31.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:31.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:31.955813+0000) 2024-03-06T10:15:32.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:32.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:32.956095+0000) 2024-03-06T10:15:33.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:33.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:33.956325+0000) 2024-03-06T10:15:34.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:34.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:34.956562+0000) 2024-03-06T10:15:35.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:35.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:35.956814+0000) 2024-03-06T10:15:36.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:36.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:36.957084+0000) 2024-03-06T10:15:37.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:37.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:37.957320+0000) 2024-03-06T10:15:38.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:38.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:38.957550+0000) 2024-03-06T10:15:39.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:39.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:39.957785+0000) 2024-03-06T10:15:40.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:40.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:40.958044+0000) 2024-03-06T10:15:41.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:41.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:41.958237+0000) 2024-03-06T10:15:42.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:42.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:42.958436+0000) 2024-03-06T10:15:43.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:43.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:43.958674+0000) 2024-03-06T10:15:44.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:44.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:44.958946+0000) 2024-03-06T10:15:45.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:45.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:45.959176+0000) 2024-03-06T10:15:46.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:46.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:46.959413+0000) 2024-03-06T10:15:47.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:47.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:47.959599+0000) 2024-03-06T10:15:48.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:48.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:15:48.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:48.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:48.959873+0000) 2024-03-06T10:15:49.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:49.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:49.960108+0000) 2024-03-06T10:15:50.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:50.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:50.960348+0000) 2024-03-06T10:15:51.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:51.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:51.960572+0000) 2024-03-06T10:15:52.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:52.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:52.960785+0000) 2024-03-06T10:15:53.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:53.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:53.961072+0000) 2024-03-06T10:15:54.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:54.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:54.961310+0000) 2024-03-06T10:15:55.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:55.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:55.961547+0000) 2024-03-06T10:15:56.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:56.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:56.961780+0000) 2024-03-06T10:15:57.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:57.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:57.962043+0000) 2024-03-06T10:15:58.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:58.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:58.962277+0000) 2024-03-06T10:15:59.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:15:59.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:15:59.962466+0000) 2024-03-06T10:16:00.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:00.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:00.962701+0000) 2024-03-06T10:16:01.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:01.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:01.962886+0000) 2024-03-06T10:16:02.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:02.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:02.963122+0000) 2024-03-06T10:16:03.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:03.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:03.963320+0000) 2024-03-06T10:16:04.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:04.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:04.963564+0000) 2024-03-06T10:16:05.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:05.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:05.963794+0000) 2024-03-06T10:16:06.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:06.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:06.964045+0000) 2024-03-06T10:16:07.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:07.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:07.964233+0000) 2024-03-06T10:16:08.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:08.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:08.964422+0000) 2024-03-06T10:16:09.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:09.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:09.964675+0000) 2024-03-06T10:16:10.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:10.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:10.964921+0000) 2024-03-06T10:16:11.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:11.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:11.965138+0000) 2024-03-06T10:16:12.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:12.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:12.965431+0000) 2024-03-06T10:16:13.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:13.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:13.965711+0000) 2024-03-06T10:16:14.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:14.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:14.965918+0000) 2024-03-06T10:16:15.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:15.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:15.966133+0000) 2024-03-06T10:16:16.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:16.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:16.966358+0000) 2024-03-06T10:16:17.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:17.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:17.966510+0000) 2024-03-06T10:16:18.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:18.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:18.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:18.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:18.966743+0000) 2024-03-06T10:16:19.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:19.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:19.966951+0000) 2024-03-06T10:16:20.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:20.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:20.967200+0000) 2024-03-06T10:16:21.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:21.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:21.967414+0000) 2024-03-06T10:16:22.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:22.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:22.967652+0000) 2024-03-06T10:16:23.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:23.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:23.967857+0000) 2024-03-06T10:16:24.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:24.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:24.968136+0000) 2024-03-06T10:16:25.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:25.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:25.968360+0000) 2024-03-06T10:16:26.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:26.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:26.968553+0000) 2024-03-06T10:16:27.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:27.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:27.968788+0000) 2024-03-06T10:16:28.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:28.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:28.969047+0000) 2024-03-06T10:16:29.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:29.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:29.969289+0000) 2024-03-06T10:16:30.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:30.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:30.969529+0000) 2024-03-06T10:16:31.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:31.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:31.969777+0000) 2024-03-06T10:16:32.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:32.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:32.970075+0000) 2024-03-06T10:16:33.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:33.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:33.970336+0000) 2024-03-06T10:16:34.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:34.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:34.970618+0000) 2024-03-06T10:16:35.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:35.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:35.970903+0000) 2024-03-06T10:16:36.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:36.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:36.971190+0000) 2024-03-06T10:16:37.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:37.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:37.971407+0000) 2024-03-06T10:16:38.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:38.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:38.971637+0000) 2024-03-06T10:16:39.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:39.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:39.971876+0000) 2024-03-06T10:16:40.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:40.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:40.972113+0000) 2024-03-06T10:16:41.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:41.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:41.972325+0000) 2024-03-06T10:16:42.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:42.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:42.972601+0000) 2024-03-06T10:16:43.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:43.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:43.972879+0000) 2024-03-06T10:16:44.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:44.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:44.973131+0000) 2024-03-06T10:16:45.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:45.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:45.973371+0000) 2024-03-06T10:16:46.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:46.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:46.973589+0000) 2024-03-06T10:16:47.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:47.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:47.973857+0000) 2024-03-06T10:16:48.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:48.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:16:48.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:48.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:48.974141+0000) 2024-03-06T10:16:49.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:49.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:49.974388+0000) 2024-03-06T10:16:50.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:50.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:50.974581+0000) 2024-03-06T10:16:51.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:51.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:51.974812+0000) 2024-03-06T10:16:52.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:52.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:52.975086+0000) 2024-03-06T10:16:53.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:53.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:53.975318+0000) 2024-03-06T10:16:54.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:54.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:54.975557+0000) 2024-03-06T10:16:55.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:55.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:55.975742+0000) 2024-03-06T10:16:56.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:56.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:56.975961+0000) 2024-03-06T10:16:57.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:57.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:57.976231+0000) 2024-03-06T10:16:58.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:58.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:58.976524+0000) 2024-03-06T10:16:59.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:16:59.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:16:59.976826+0000) 2024-03-06T10:17:00.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:00.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:00.977178+0000) 2024-03-06T10:17:01.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:01.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:01.977342+0000) 2024-03-06T10:17:02.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:02.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:02.977567+0000) 2024-03-06T10:17:03.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:03.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:03.977793+0000) 2024-03-06T10:17:04.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:04.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:04.978072+0000) 2024-03-06T10:17:05.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:05.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:05.978312+0000) 2024-03-06T10:17:06.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:06.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:06.978496+0000) 2024-03-06T10:17:07.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:07.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:07.978727+0000) 2024-03-06T10:17:08.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:08.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:08.978950+0000) 2024-03-06T10:17:09.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:09.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:09.979177+0000) 2024-03-06T10:17:10.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:10.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:10.979360+0000) 2024-03-06T10:17:11.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:11.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:11.979596+0000) 2024-03-06T10:17:12.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:12.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:12.979832+0000) 2024-03-06T10:17:13.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:13.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:13.980099+0000) 2024-03-06T10:17:14.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:14.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:14.980333+0000) 2024-03-06T10:17:15.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:15.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:15.980568+0000) 2024-03-06T10:17:16.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:16.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:16.980802+0000) 2024-03-06T10:17:17.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:17.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:17.981066+0000) 2024-03-06T10:17:18.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:18.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:18.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:18.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:18.981265+0000) 2024-03-06T10:17:19.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:19.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:19.981507+0000) 2024-03-06T10:17:20.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:20.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:20.981743+0000) 2024-03-06T10:17:21.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:21.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:21.981970+0000) 2024-03-06T10:17:22.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:22.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:22.982215+0000) 2024-03-06T10:17:23.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:23.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:23.982454+0000) 2024-03-06T10:17:24.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:24.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:24.982688+0000) 2024-03-06T10:17:25.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:25.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:25.982922+0000) 2024-03-06T10:17:26.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:26.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:26.983151+0000) 2024-03-06T10:17:27.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:27.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:27.983381+0000) 2024-03-06T10:17:28.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:28.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:28.983600+0000) 2024-03-06T10:17:29.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:29.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:29.983853+0000) 2024-03-06T10:17:30.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:30.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:30.984121+0000) 2024-03-06T10:17:31.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:31.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:31.984353+0000) 2024-03-06T10:17:32.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:32.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:32.984574+0000) 2024-03-06T10:17:33.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:33.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:33.984795+0000) 2024-03-06T10:17:34.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:34.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:34.985107+0000) 2024-03-06T10:17:35.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:35.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:35.985340+0000) 2024-03-06T10:17:36.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:36.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:36.985563+0000) 2024-03-06T10:17:37.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:37.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:37.985796+0000) 2024-03-06T10:17:38.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:38.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:38.985993+0000) 2024-03-06T10:17:39.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:39.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:39.986277+0000) 2024-03-06T10:17:40.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:40.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:40.986514+0000) 2024-03-06T10:17:41.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:41.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:41.986746+0000) 2024-03-06T10:17:42.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:42.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:42.986973+0000) 2024-03-06T10:17:43.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:43.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:43.987306+0000) 2024-03-06T10:17:44.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:44.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:44.987599+0000) 2024-03-06T10:17:45.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:45.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:45.987871+0000) 2024-03-06T10:17:46.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:46.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:46.988162+0000) 2024-03-06T10:17:47.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:47.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:47.988352+0000) 2024-03-06T10:17:48.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:48.934 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:17:48.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:48.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:48.988540+0000) 2024-03-06T10:17:49.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:49.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:49.988798+0000) 2024-03-06T10:17:50.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:50.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:50.989152+0000) 2024-03-06T10:17:51.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:51.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:51.989380+0000) 2024-03-06T10:17:52.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:52.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:52.989604+0000) 2024-03-06T10:17:53.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:53.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:53.989836+0000) 2024-03-06T10:17:54.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:54.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:54.990063+0000) 2024-03-06T10:17:55.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:55.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:55.990297+0000) 2024-03-06T10:17:56.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:56.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:56.990573+0000) 2024-03-06T10:17:57.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:57.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:57.990858+0000) 2024-03-06T10:17:58.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:58.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:58.991148+0000) 2024-03-06T10:17:59.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:17:59.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:17:59.991388+0000) 2024-03-06T10:18:00.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:00.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:00.991584+0000) 2024-03-06T10:18:01.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:01.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:01.991824+0000) 2024-03-06T10:18:02.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:02.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:02.991982+0000) 2024-03-06T10:18:03.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:03.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:03.992238+0000) 2024-03-06T10:18:04.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:04.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:04.992440+0000) 2024-03-06T10:18:05.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:05.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:05.992688+0000) 2024-03-06T10:18:06.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:06.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:06.992909+0000) 2024-03-06T10:18:07.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:07.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:07.993122+0000) 2024-03-06T10:18:08.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:08.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:08.993295+0000) 2024-03-06T10:18:09.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:09.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:09.993541+0000) 2024-03-06T10:18:10.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:10.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:10.993783+0000) 2024-03-06T10:18:11.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:11.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:11.994049+0000) 2024-03-06T10:18:12.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:12.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:12.994255+0000) 2024-03-06T10:18:13.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:13.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:13.994444+0000) 2024-03-06T10:18:14.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:14.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:14.994683+0000) 2024-03-06T10:18:15.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:15.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:15.994925+0000) 2024-03-06T10:18:16.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:16.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:16.995144+0000) 2024-03-06T10:18:17.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:17.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:17.995401+0000) 2024-03-06T10:18:18.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:18.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:18.995643+0000) 2024-03-06T10:18:19.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:19.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:19.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:19.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:19.995905+0000) 2024-03-06T10:18:20.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:20.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:20.996155+0000) 2024-03-06T10:18:21.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:21.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:21.996381+0000) 2024-03-06T10:18:22.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:22.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:22.996609+0000) 2024-03-06T10:18:23.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:23.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:23.996840+0000) 2024-03-06T10:18:24.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:24.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:24.997119+0000) 2024-03-06T10:18:25.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:25.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:25.997359+0000) 2024-03-06T10:18:26.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:26.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:26.997548+0000) 2024-03-06T10:18:27.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:27.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:27.997790+0000) 2024-03-06T10:18:28.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:28.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:28.998036+0000) 2024-03-06T10:18:29.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:29.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:29.998253+0000) 2024-03-06T10:18:30.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:30.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:30.998496+0000) 2024-03-06T10:18:31.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:31.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:31.998726+0000) 2024-03-06T10:18:32.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:32.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:32.998951+0000) 2024-03-06T10:18:33.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:33.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:33.999175+0000) 2024-03-06T10:18:34.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:34.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:34.999418+0000) 2024-03-06T10:18:35.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:35.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:35.999673+0000) 2024-03-06T10:18:36.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:36.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:36.999869+0000) 2024-03-06T10:18:37.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:37.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:38.000115+0000) 2024-03-06T10:18:38.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:38.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:39.000342+0000) 2024-03-06T10:18:39.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:39.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:40.000599+0000) 2024-03-06T10:18:40.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:40.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:41.000862+0000) 2024-03-06T10:18:42.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:42.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:42.001151+0000) 2024-03-06T10:18:43.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:43.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:43.001391+0000) 2024-03-06T10:18:44.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:44.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:44.001627+0000) 2024-03-06T10:18:45.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:45.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:45.001860+0000) 2024-03-06T10:18:46.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:46.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:46.002146+0000) 2024-03-06T10:18:47.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:47.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:47.002311+0000) 2024-03-06T10:18:48.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:48.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:48.002510+0000) 2024-03-06T10:18:49.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:49.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:49.002703+0000) 2024-03-06T10:18:49.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:49.299 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:18:50.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:50.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:50.002918+0000) 2024-03-06T10:18:51.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:51.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:51.003141+0000) 2024-03-06T10:18:52.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:52.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:52.003384+0000) 2024-03-06T10:18:53.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:53.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:53.003615+0000) 2024-03-06T10:18:54.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:54.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:54.003865+0000) 2024-03-06T10:18:55.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:55.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:55.004133+0000) 2024-03-06T10:18:56.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:56.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:56.004354+0000) 2024-03-06T10:18:57.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:57.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:57.004552+0000) 2024-03-06T10:18:58.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:58.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:58.004784+0000) 2024-03-06T10:18:59.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:18:59.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:18:59.005001+0000) 2024-03-06T10:19:00.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:00.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:00.005245+0000) 2024-03-06T10:19:01.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:01.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:01.005479+0000) 2024-03-06T10:19:02.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:02.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:02.005701+0000) 2024-03-06T10:19:03.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:03.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:03.005919+0000) 2024-03-06T10:19:04.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:04.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:04.006146+0000) 2024-03-06T10:19:05.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:05.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:05.006358+0000) 2024-03-06T10:19:06.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:06.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:06.006606+0000) 2024-03-06T10:19:07.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:07.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:07.006829+0000) 2024-03-06T10:19:08.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:08.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:08.007109+0000) 2024-03-06T10:19:09.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:09.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:09.007345+0000) 2024-03-06T10:19:10.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:10.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:10.007586+0000) 2024-03-06T10:19:11.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:11.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:11.007853+0000) 2024-03-06T10:19:12.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:12.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:12.008120+0000) 2024-03-06T10:19:13.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:13.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:13.008335+0000) 2024-03-06T10:19:14.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:14.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:14.008550+0000) 2024-03-06T10:19:15.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:15.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:15.008769+0000) 2024-03-06T10:19:16.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:16.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:16.009038+0000) 2024-03-06T10:19:17.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:17.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:17.009259+0000) 2024-03-06T10:19:18.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:18.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:18.009447+0000) 2024-03-06T10:19:19.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:19.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:19.009661+0000) 2024-03-06T10:19:19.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:19.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:20.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:20.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:20.009853+0000) 2024-03-06T10:19:21.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:21.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:21.010133+0000) 2024-03-06T10:19:22.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:22.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:22.010360+0000) 2024-03-06T10:19:23.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:23.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:23.010559+0000) 2024-03-06T10:19:24.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:24.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:24.010818+0000) 2024-03-06T10:19:25.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:25.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:25.011065+0000) 2024-03-06T10:19:26.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:26.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:26.011245+0000) 2024-03-06T10:19:27.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:27.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:27.011416+0000) 2024-03-06T10:19:28.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:28.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:28.011646+0000) 2024-03-06T10:19:29.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:29.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:29.011835+0000) 2024-03-06T10:19:30.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:30.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:30.012112+0000) 2024-03-06T10:19:31.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:31.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:31.012360+0000) 2024-03-06T10:19:32.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:32.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:32.012639+0000) 2024-03-06T10:19:33.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:33.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:33.012916+0000) 2024-03-06T10:19:34.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:34.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:34.013195+0000) 2024-03-06T10:19:35.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:35.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:35.013517+0000) 2024-03-06T10:19:36.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:36.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:36.013795+0000) 2024-03-06T10:19:37.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:37.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:37.014103+0000) 2024-03-06T10:19:38.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:38.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:38.014389+0000) 2024-03-06T10:19:39.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:39.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:39.014671+0000) 2024-03-06T10:19:40.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:40.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:40.014922+0000) 2024-03-06T10:19:41.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:41.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:41.015201+0000) 2024-03-06T10:19:42.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:42.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:42.015468+0000) 2024-03-06T10:19:43.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:43.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:43.015652+0000) 2024-03-06T10:19:44.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:44.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:44.015872+0000) 2024-03-06T10:19:45.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:45.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:45.016168+0000) 2024-03-06T10:19:46.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:46.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:46.016425+0000) 2024-03-06T10:19:47.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:47.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:47.016708+0000) 2024-03-06T10:19:48.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:48.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:48.016957+0000) 2024-03-06T10:19:49.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:49.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:49.017154+0000) 2024-03-06T10:19:49.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:49.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:19:50.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:50.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:50.017384+0000) 2024-03-06T10:19:51.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:51.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:51.017620+0000) 2024-03-06T10:19:52.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:52.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:52.017804+0000) 2024-03-06T10:19:53.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:53.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:53.018065+0000) 2024-03-06T10:19:54.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:54.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:54.018301+0000) 2024-03-06T10:19:55.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:55.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:55.018520+0000) 2024-03-06T10:19:56.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:56.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:56.018714+0000) 2024-03-06T10:19:57.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:57.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:57.018939+0000) 2024-03-06T10:19:58.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:58.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:58.019177+0000) 2024-03-06T10:19:59.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:19:59.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:19:59.019397+0000) 2024-03-06T10:20:00.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:00.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:00.019623+0000) 2024-03-06T10:20:01.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:01.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:01.019874+0000) 2024-03-06T10:20:02.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:02.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:02.020137+0000) 2024-03-06T10:20:03.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:03.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:03.020340+0000) 2024-03-06T10:20:04.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:04.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:04.020583+0000) 2024-03-06T10:20:05.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:05.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:05.020796+0000) 2024-03-06T10:20:06.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:06.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:06.021059+0000) 2024-03-06T10:20:07.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:07.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:07.021281+0000) 2024-03-06T10:20:08.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:08.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:08.021534+0000) 2024-03-06T10:20:09.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:09.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:09.021750+0000) 2024-03-06T10:20:10.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:10.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:10.021979+0000) 2024-03-06T10:20:11.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:11.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:11.022275+0000) 2024-03-06T10:20:12.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:12.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:12.022496+0000) 2024-03-06T10:20:13.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:13.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:13.022717+0000) 2024-03-06T10:20:14.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:14.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:14.022949+0000) 2024-03-06T10:20:15.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:15.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:15.023169+0000) 2024-03-06T10:20:16.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:16.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:16.023386+0000) 2024-03-06T10:20:17.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:17.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:17.023603+0000) 2024-03-06T10:20:18.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:18.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:18.023854+0000) 2024-03-06T10:20:19.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:19.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:19.024124+0000) 2024-03-06T10:20:19.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:19.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:20.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:20.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:20.024345+0000) 2024-03-06T10:20:21.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:21.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:21.024571+0000) 2024-03-06T10:20:22.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:22.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:22.024811+0000) 2024-03-06T10:20:23.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:23.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:23.025058+0000) 2024-03-06T10:20:24.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:24.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:24.025248+0000) 2024-03-06T10:20:25.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:25.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:25.025467+0000) 2024-03-06T10:20:26.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:26.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:26.025698+0000) 2024-03-06T10:20:27.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:27.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:27.025922+0000) 2024-03-06T10:20:28.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:28.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:28.026146+0000) 2024-03-06T10:20:29.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:29.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:29.026362+0000) 2024-03-06T10:20:30.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:30.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:30.026570+0000) 2024-03-06T10:20:31.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:31.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:31.026804+0000) 2024-03-06T10:20:32.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:32.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:32.027066+0000) 2024-03-06T10:20:33.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:33.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:33.027295+0000) 2024-03-06T10:20:34.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:34.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:34.027525+0000) 2024-03-06T10:20:35.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:35.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:35.027744+0000) 2024-03-06T10:20:36.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:36.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:36.027978+0000) 2024-03-06T10:20:37.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:37.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:37.028207+0000) 2024-03-06T10:20:38.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:38.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:38.028437+0000) 2024-03-06T10:20:39.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:39.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:39.028658+0000) 2024-03-06T10:20:40.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:40.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:40.028883+0000) 2024-03-06T10:20:41.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:41.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:41.029118+0000) 2024-03-06T10:20:42.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:42.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:42.029318+0000) 2024-03-06T10:20:43.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:43.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:43.029564+0000) 2024-03-06T10:20:44.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:44.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:44.029817+0000) 2024-03-06T10:20:45.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:45.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:45.030103+0000) 2024-03-06T10:20:46.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:46.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:46.030307+0000) 2024-03-06T10:20:47.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:47.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:47.030537+0000) 2024-03-06T10:20:48.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:48.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:48.030771+0000) 2024-03-06T10:20:49.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:49.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:49.030979+0000) 2024-03-06T10:20:50.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:50.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:50.031237+0000) 2024-03-06T10:20:50.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:50.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:20:51.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:51.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:51.031465+0000) 2024-03-06T10:20:52.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:52.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:52.031700+0000) 2024-03-06T10:20:53.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:53.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:53.031923+0000) 2024-03-06T10:20:54.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:54.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:54.032163+0000) 2024-03-06T10:20:55.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:55.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:55.032399+0000) 2024-03-06T10:20:56.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:56.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:56.032591+0000) 2024-03-06T10:20:57.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:57.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:57.032773+0000) 2024-03-06T10:20:58.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:58.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:58.033043+0000) 2024-03-06T10:20:59.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:20:59.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:20:59.033293+0000) 2024-03-06T10:21:00.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:00.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:00.033501+0000) 2024-03-06T10:21:01.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:01.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:01.033768+0000) 2024-03-06T10:21:02.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:02.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:02.034040+0000) 2024-03-06T10:21:03.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:03.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:03.034260+0000) 2024-03-06T10:21:04.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:04.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:04.034456+0000) 2024-03-06T10:21:05.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:05.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:05.034673+0000) 2024-03-06T10:21:06.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:06.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:06.034908+0000) 2024-03-06T10:21:07.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:07.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:07.035138+0000) 2024-03-06T10:21:08.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:08.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:08.035386+0000) 2024-03-06T10:21:09.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:09.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:09.035635+0000) 2024-03-06T10:21:10.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:10.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:10.035864+0000) 2024-03-06T10:21:11.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:11.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:11.036158+0000) 2024-03-06T10:21:12.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:12.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:12.036389+0000) 2024-03-06T10:21:13.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:13.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:13.036590+0000) 2024-03-06T10:21:14.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:14.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:14.036818+0000) 2024-03-06T10:21:15.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:15.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:15.037094+0000) 2024-03-06T10:21:16.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:16.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:16.037355+0000) 2024-03-06T10:21:17.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:17.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:17.037648+0000) 2024-03-06T10:21:18.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:18.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:18.037875+0000) 2024-03-06T10:21:19.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:19.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:19.038126+0000) 2024-03-06T10:21:20.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:20.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:20.038366+0000) 2024-03-06T10:21:20.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:20.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:21.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:21.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:21.038595+0000) 2024-03-06T10:21:22.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:22.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:22.038824+0000) 2024-03-06T10:21:23.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:23.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:23.039087+0000) 2024-03-06T10:21:24.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:24.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:24.039328+0000) 2024-03-06T10:21:25.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:25.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:25.039546+0000) 2024-03-06T10:21:26.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:26.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:26.039782+0000) 2024-03-06T10:21:27.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:27.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:27.040037+0000) 2024-03-06T10:21:28.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:28.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:28.040274+0000) 2024-03-06T10:21:29.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:29.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:29.040489+0000) 2024-03-06T10:21:30.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:30.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:30.040732+0000) 2024-03-06T10:21:31.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:31.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:31.040986+0000) 2024-03-06T10:21:32.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:32.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:32.041261+0000) 2024-03-06T10:21:33.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:33.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:33.041480+0000) 2024-03-06T10:21:34.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:34.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:34.041708+0000) 2024-03-06T10:21:35.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:35.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:35.041924+0000) 2024-03-06T10:21:36.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:36.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:36.042164+0000) 2024-03-06T10:21:37.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:37.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:37.042381+0000) 2024-03-06T10:21:38.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:38.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:38.042582+0000) 2024-03-06T10:21:39.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:39.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:39.042819+0000) 2024-03-06T10:21:40.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:40.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:40.043119+0000) 2024-03-06T10:21:41.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:41.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:41.043387+0000) 2024-03-06T10:21:42.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:42.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:42.043646+0000) 2024-03-06T10:21:43.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:43.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:43.043896+0000) 2024-03-06T10:21:44.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:44.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:44.044145+0000) 2024-03-06T10:21:45.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:45.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:45.044386+0000) 2024-03-06T10:21:46.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:46.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:46.044647+0000) 2024-03-06T10:21:47.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:47.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:47.044880+0000) 2024-03-06T10:21:48.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:48.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:48.045134+0000) 2024-03-06T10:21:49.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:49.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:49.045371+0000) 2024-03-06T10:21:50.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:50.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:50.045575+0000) 2024-03-06T10:21:50.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:50.475 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:21:51.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:51.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:51.045823+0000) 2024-03-06T10:21:52.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:52.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:52.046093+0000) 2024-03-06T10:21:53.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:53.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:53.046269+0000) 2024-03-06T10:21:54.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:54.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:54.046513+0000) 2024-03-06T10:21:55.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:55.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:55.046701+0000) 2024-03-06T10:21:56.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:56.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:56.046952+0000) 2024-03-06T10:21:57.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:57.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:57.047165+0000) 2024-03-06T10:21:58.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:58.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:58.047422+0000) 2024-03-06T10:21:59.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:21:59.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:21:59.047640+0000) 2024-03-06T10:22:00.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:00.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:00.047869+0000) 2024-03-06T10:22:01.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:01.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:01.048156+0000) 2024-03-06T10:22:02.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:02.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:02.048389+0000) 2024-03-06T10:22:03.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:03.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:03.048571+0000) 2024-03-06T10:22:04.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:04.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:04.048765+0000) 2024-03-06T10:22:05.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:05.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:05.049075+0000) 2024-03-06T10:22:06.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:06.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:06.049320+0000) 2024-03-06T10:22:07.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:07.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:07.049549+0000) 2024-03-06T10:22:08.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:08.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:08.049782+0000) 2024-03-06T10:22:09.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:09.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:09.050001+0000) 2024-03-06T10:22:10.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:10.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:10.050244+0000) 2024-03-06T10:22:11.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:11.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:11.050510+0000) 2024-03-06T10:22:12.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:12.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:12.050757+0000) 2024-03-06T10:22:13.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:13.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:13.050961+0000) 2024-03-06T10:22:14.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:14.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:14.051203+0000) 2024-03-06T10:22:15.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:15.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:15.051432+0000) 2024-03-06T10:22:16.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:16.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:16.051699+0000) 2024-03-06T10:22:17.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:17.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:17.051901+0000) 2024-03-06T10:22:18.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:18.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:18.052165+0000) 2024-03-06T10:22:19.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:19.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:19.052404+0000) 2024-03-06T10:22:20.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:20.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:20.052655+0000) 2024-03-06T10:22:20.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:20.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:21.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:21.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:21.052941+0000) 2024-03-06T10:22:22.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:22.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:22.053186+0000) 2024-03-06T10:22:23.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:23.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:23.053435+0000) 2024-03-06T10:22:24.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:24.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:24.053663+0000) 2024-03-06T10:22:25.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:25.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:25.053838+0000) 2024-03-06T10:22:26.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:26.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:26.054124+0000) 2024-03-06T10:22:27.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:27.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:27.054270+0000) 2024-03-06T10:22:28.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:28.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:28.054409+0000) 2024-03-06T10:22:29.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:29.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:29.054615+0000) 2024-03-06T10:22:30.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:30.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:30.054835+0000) 2024-03-06T10:22:31.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:31.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:31.055120+0000) 2024-03-06T10:22:32.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:32.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:32.055355+0000) 2024-03-06T10:22:33.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:33.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:33.055533+0000) 2024-03-06T10:22:34.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:34.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:34.055767+0000) 2024-03-06T10:22:35.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:35.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:35.055953+0000) 2024-03-06T10:22:36.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:36.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:36.056205+0000) 2024-03-06T10:22:37.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:37.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:37.056342+0000) 2024-03-06T10:22:38.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:38.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:38.056565+0000) 2024-03-06T10:22:39.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:39.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:39.056742+0000) 2024-03-06T10:22:40.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:40.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:40.056941+0000) 2024-03-06T10:22:41.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:41.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:41.057198+0000) 2024-03-06T10:22:42.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:42.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:42.057397+0000) 2024-03-06T10:22:43.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:43.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:43.057601+0000) 2024-03-06T10:22:44.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:44.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:44.057808+0000) 2024-03-06T10:22:45.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:45.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:45.057965+0000) 2024-03-06T10:22:46.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:46.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:46.058192+0000) 2024-03-06T10:22:47.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:47.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:47.058422+0000) 2024-03-06T10:22:48.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:48.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:48.058659+0000) 2024-03-06T10:22:49.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:49.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:49.058879+0000) 2024-03-06T10:22:50.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:50.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:50.059142+0000) 2024-03-06T10:22:50.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:50.765 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:22:51.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:51.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:51.059357+0000) 2024-03-06T10:22:52.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:52.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:52.059589+0000) 2024-03-06T10:22:53.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:53.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:53.059826+0000) 2024-03-06T10:22:54.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:54.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:54.060081+0000) 2024-03-06T10:22:55.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:55.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:55.060326+0000) 2024-03-06T10:22:56.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:56.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:56.060569+0000) 2024-03-06T10:22:57.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:57.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:57.060818+0000) 2024-03-06T10:22:58.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:58.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:58.061157+0000) 2024-03-06T10:22:59.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:22:59.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:22:59.061404+0000) 2024-03-06T10:23:00.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:00.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:00.061652+0000) 2024-03-06T10:23:01.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:01.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:01.061880+0000) 2024-03-06T10:23:02.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:02.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:02.062127+0000) 2024-03-06T10:23:03.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:03.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:03.062374+0000) 2024-03-06T10:23:04.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:04.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:04.062629+0000) 2024-03-06T10:23:05.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:05.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:05.062872+0000) 2024-03-06T10:23:06.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:06.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:06.063172+0000) 2024-03-06T10:23:07.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:07.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:07.063410+0000) 2024-03-06T10:23:08.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:08.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:08.063627+0000) 2024-03-06T10:23:09.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:09.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:09.063861+0000) 2024-03-06T10:23:10.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:10.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:10.064128+0000) 2024-03-06T10:23:11.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:11.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:11.064355+0000) 2024-03-06T10:23:12.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:12.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:12.064572+0000) 2024-03-06T10:23:13.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:13.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:13.064836+0000) 2024-03-06T10:23:14.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:14.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:14.065130+0000) 2024-03-06T10:23:15.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:15.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:15.065354+0000) 2024-03-06T10:23:16.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:16.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:16.065599+0000) 2024-03-06T10:23:17.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:17.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:17.065823+0000) 2024-03-06T10:23:18.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:18.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:18.066061+0000) 2024-03-06T10:23:19.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:19.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:19.066286+0000) 2024-03-06T10:23:20.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:20.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:20.066511+0000) 2024-03-06T10:23:20.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:20.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:21.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:21.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:21.066763+0000) 2024-03-06T10:23:22.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:22.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:22.067000+0000) 2024-03-06T10:23:23.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:23.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:23.067279+0000) 2024-03-06T10:23:24.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:24.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:24.067508+0000) 2024-03-06T10:23:25.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:25.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:25.067739+0000) 2024-03-06T10:23:26.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:26.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:26.067991+0000) 2024-03-06T10:23:27.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:27.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:27.068267+0000) 2024-03-06T10:23:28.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:28.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:28.068502+0000) 2024-03-06T10:23:29.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:29.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:29.068721+0000) 2024-03-06T10:23:30.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:30.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:30.068959+0000) 2024-03-06T10:23:31.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:31.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:31.069198+0000) 2024-03-06T10:23:32.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:32.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:32.069432+0000) 2024-03-06T10:23:33.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:33.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:33.069658+0000) 2024-03-06T10:23:34.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:34.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:34.069904+0000) 2024-03-06T10:23:35.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:35.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:35.070141+0000) 2024-03-06T10:23:36.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:36.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:36.070381+0000) 2024-03-06T10:23:37.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:37.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:37.070592+0000) 2024-03-06T10:23:38.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:38.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:38.070825+0000) 2024-03-06T10:23:39.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:39.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:39.071077+0000) 2024-03-06T10:23:40.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:40.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:40.071226+0000) 2024-03-06T10:23:41.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:41.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:41.071461+0000) 2024-03-06T10:23:42.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:42.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:42.071755+0000) 2024-03-06T10:23:43.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:43.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:43.071991+0000) 2024-03-06T10:23:44.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:44.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:44.072343+0000) 2024-03-06T10:23:45.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:45.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:45.072578+0000) 2024-03-06T10:23:46.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:46.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:46.072880+0000) 2024-03-06T10:23:47.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:47.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:47.073123+0000) 2024-03-06T10:23:48.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:48.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:48.073355+0000) 2024-03-06T10:23:49.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:49.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:49.073618+0000) 2024-03-06T10:23:50.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:50.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:50.073849+0000) 2024-03-06T10:23:51.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:51.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:51.074135+0000) 2024-03-06T10:23:51.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:51.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:23:52.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:52.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:52.074324+0000) 2024-03-06T10:23:53.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:53.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:53.074551+0000) 2024-03-06T10:23:54.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:54.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:54.074809+0000) 2024-03-06T10:23:55.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:55.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:55.075116+0000) 2024-03-06T10:23:56.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:56.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:56.075360+0000) 2024-03-06T10:23:57.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:57.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:57.075598+0000) 2024-03-06T10:23:58.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:58.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:58.075825+0000) 2024-03-06T10:23:59.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:23:59.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:23:59.076093+0000) 2024-03-06T10:24:00.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:00.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:00.076328+0000) 2024-03-06T10:24:01.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:01.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:01.076590+0000) 2024-03-06T10:24:02.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:02.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:02.076803+0000) 2024-03-06T10:24:03.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:03.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:03.077080+0000) 2024-03-06T10:24:04.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:04.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:04.077328+0000) 2024-03-06T10:24:05.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:05.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:05.077520+0000) 2024-03-06T10:24:06.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:06.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:06.077760+0000) 2024-03-06T10:24:07.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:07.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:07.077996+0000) 2024-03-06T10:24:08.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:08.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:08.078212+0000) 2024-03-06T10:24:09.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:09.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:09.078382+0000) 2024-03-06T10:24:10.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:10.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:10.078580+0000) 2024-03-06T10:24:11.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:11.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:11.078838+0000) 2024-03-06T10:24:12.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:12.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:12.079066+0000) 2024-03-06T10:24:13.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:13.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:13.079291+0000) 2024-03-06T10:24:14.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:14.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:14.079518+0000) 2024-03-06T10:24:15.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:15.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:15.079745+0000) 2024-03-06T10:24:16.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:16.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:16.079986+0000) 2024-03-06T10:24:17.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:17.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:17.080280+0000) 2024-03-06T10:24:18.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:18.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:18.080510+0000) 2024-03-06T10:24:19.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:19.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:19.080699+0000) 2024-03-06T10:24:20.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:20.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:20.080944+0000) 2024-03-06T10:24:21.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:21.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:21.081267+0000) 2024-03-06T10:24:21.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:21.391 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:22.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:22.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:22.081474+0000) 2024-03-06T10:24:23.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:23.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:23.081706+0000) 2024-03-06T10:24:24.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:24.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:24.081931+0000) 2024-03-06T10:24:25.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:25.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:25.082144+0000) 2024-03-06T10:24:26.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:26.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:26.082368+0000) 2024-03-06T10:24:27.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:27.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:27.082601+0000) 2024-03-06T10:24:28.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:28.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:28.082836+0000) 2024-03-06T10:24:29.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:29.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:29.083122+0000) 2024-03-06T10:24:30.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:30.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:30.083351+0000) 2024-03-06T10:24:31.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:31.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:31.083580+0000) 2024-03-06T10:24:32.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:32.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:32.083810+0000) 2024-03-06T10:24:33.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:33.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:33.084093+0000) 2024-03-06T10:24:34.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:34.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:34.084372+0000) 2024-03-06T10:24:35.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:35.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:35.084552+0000) 2024-03-06T10:24:36.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:36.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:36.084783+0000) 2024-03-06T10:24:37.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:37.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:37.084967+0000) 2024-03-06T10:24:38.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:38.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:38.085202+0000) 2024-03-06T10:24:39.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:39.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:39.085374+0000) 2024-03-06T10:24:40.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:40.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:40.085590+0000) 2024-03-06T10:24:41.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:41.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:41.085824+0000) 2024-03-06T10:24:42.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:42.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:42.086073+0000) 2024-03-06T10:24:43.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:43.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:43.086316+0000) 2024-03-06T10:24:44.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:44.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:44.086506+0000) 2024-03-06T10:24:45.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:45.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:45.086728+0000) 2024-03-06T10:24:46.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:46.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:46.086973+0000) 2024-03-06T10:24:47.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:47.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:47.087262+0000) 2024-03-06T10:24:48.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:48.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:48.087513+0000) 2024-03-06T10:24:49.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:49.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:49.087706+0000) 2024-03-06T10:24:50.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:50.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:50.087934+0000) 2024-03-06T10:24:51.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:51.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:51.088168+0000) 2024-03-06T10:24:51.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:51.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:24:52.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:52.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:52.088404+0000) 2024-03-06T10:24:53.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:53.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:53.088644+0000) 2024-03-06T10:24:54.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:54.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:54.088877+0000) 2024-03-06T10:24:55.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:55.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:55.089123+0000) 2024-03-06T10:24:56.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:56.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:56.089318+0000) 2024-03-06T10:24:57.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:57.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:57.089488+0000) 2024-03-06T10:24:58.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:58.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:58.089615+0000) 2024-03-06T10:24:59.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:24:59.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:24:59.089829+0000) 2024-03-06T10:25:00.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:00.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:00.090120+0000) 2024-03-06T10:25:01.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:01.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:01.090397+0000) 2024-03-06T10:25:02.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:02.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:02.090624+0000) 2024-03-06T10:25:03.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:03.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:03.090853+0000) 2024-03-06T10:25:04.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:04.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:04.091122+0000) 2024-03-06T10:25:05.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:05.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:05.091352+0000) 2024-03-06T10:25:06.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:06.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:06.091600+0000) 2024-03-06T10:25:07.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:07.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:07.091837+0000) 2024-03-06T10:25:08.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:08.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:08.092125+0000) 2024-03-06T10:25:09.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:09.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:09.092323+0000) 2024-03-06T10:25:10.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:10.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:10.092550+0000) 2024-03-06T10:25:11.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:11.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:11.092795+0000) 2024-03-06T10:25:12.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:12.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:12.093081+0000) 2024-03-06T10:25:13.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:13.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:13.093340+0000) 2024-03-06T10:25:14.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:14.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:14.093542+0000) 2024-03-06T10:25:15.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:15.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:15.093766+0000) 2024-03-06T10:25:16.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:16.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:16.094001+0000) 2024-03-06T10:25:17.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:17.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:17.094231+0000) 2024-03-06T10:25:18.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:18.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:18.094422+0000) 2024-03-06T10:25:19.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:19.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:19.094604+0000) 2024-03-06T10:25:20.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:20.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:20.094835+0000) 2024-03-06T10:25:21.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:21.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:21.095125+0000) 2024-03-06T10:25:21.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:21.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:22.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:22.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:22.095338+0000) 2024-03-06T10:25:23.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:23.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:23.095570+0000) 2024-03-06T10:25:24.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:24.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:24.095808+0000) 2024-03-06T10:25:25.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:25.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:25.096074+0000) 2024-03-06T10:25:26.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:26.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:26.096339+0000) 2024-03-06T10:25:27.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:27.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:27.096664+0000) 2024-03-06T10:25:28.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:28.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:28.096938+0000) 2024-03-06T10:25:29.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:29.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:29.097185+0000) 2024-03-06T10:25:30.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:30.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:30.097441+0000) 2024-03-06T10:25:31.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:31.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:31.097686+0000) 2024-03-06T10:25:32.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:32.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:32.097968+0000) 2024-03-06T10:25:33.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:33.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:33.098287+0000) 2024-03-06T10:25:34.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:34.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:34.098524+0000) 2024-03-06T10:25:35.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:35.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:35.098761+0000) 2024-03-06T10:25:36.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:36.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:36.099047+0000) 2024-03-06T10:25:37.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:37.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:37.099296+0000) 2024-03-06T10:25:38.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:38.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:38.099567+0000) 2024-03-06T10:25:39.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:39.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:39.099818+0000) 2024-03-06T10:25:40.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:40.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:40.100088+0000) 2024-03-06T10:25:41.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:41.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:41.100331+0000) 2024-03-06T10:25:42.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:42.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:42.100562+0000) 2024-03-06T10:25:43.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:43.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:43.100799+0000) 2024-03-06T10:25:44.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:44.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:44.101066+0000) 2024-03-06T10:25:45.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:45.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:45.101254+0000) 2024-03-06T10:25:46.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:46.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:46.101496+0000) 2024-03-06T10:25:47.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:47.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:47.101735+0000) 2024-03-06T10:25:48.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:48.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:48.101989+0000) 2024-03-06T10:25:49.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:49.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:49.102266+0000) 2024-03-06T10:25:50.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:50.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:50.102493+0000) 2024-03-06T10:25:51.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:51.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:51.102721+0000) 2024-03-06T10:25:51.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:51.915 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:25:52.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:52.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:52.102974+0000) 2024-03-06T10:25:53.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:53.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:53.103158+0000) 2024-03-06T10:25:54.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:54.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:54.103356+0000) 2024-03-06T10:25:55.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:55.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:55.103589+0000) 2024-03-06T10:25:56.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:56.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:56.103844+0000) 2024-03-06T10:25:57.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:57.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:57.104129+0000) 2024-03-06T10:25:58.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:58.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:58.104403+0000) 2024-03-06T10:25:59.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:25:59.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:25:59.104651+0000) 2024-03-06T10:26:00.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:00.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:00.104909+0000) 2024-03-06T10:26:01.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:01.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:01.105199+0000) 2024-03-06T10:26:02.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:02.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:02.105477+0000) 2024-03-06T10:26:03.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:03.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:03.105762+0000) 2024-03-06T10:26:04.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:04.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:04.106041+0000) 2024-03-06T10:26:05.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:05.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:05.106275+0000) 2024-03-06T10:26:06.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:06.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:06.106534+0000) 2024-03-06T10:26:07.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:07.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:07.106800+0000) 2024-03-06T10:26:08.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:08.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:08.107106+0000) 2024-03-06T10:26:09.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:09.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:09.107348+0000) 2024-03-06T10:26:10.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:10.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:10.107625+0000) 2024-03-06T10:26:11.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:11.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:11.107908+0000) 2024-03-06T10:26:12.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:12.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:12.108189+0000) 2024-03-06T10:26:13.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:13.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:13.108425+0000) 2024-03-06T10:26:14.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:14.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:14.108675+0000) 2024-03-06T10:26:15.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:15.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:15.108948+0000) 2024-03-06T10:26:16.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:16.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:16.109176+0000) 2024-03-06T10:26:17.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:17.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:17.109359+0000) 2024-03-06T10:26:18.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:18.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:18.109585+0000) 2024-03-06T10:26:19.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:19.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:19.109813+0000) 2024-03-06T10:26:20.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:20.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:20.110080+0000) 2024-03-06T10:26:21.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:21.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:21.110329+0000) 2024-03-06T10:26:22.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:22.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:22.110594+0000) 2024-03-06T10:26:22.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:22.132 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:23.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:23.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:23.110842+0000) 2024-03-06T10:26:24.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:24.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:24.111152+0000) 2024-03-06T10:26:25.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:25.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:25.111362+0000) 2024-03-06T10:26:26.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:26.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:26.111589+0000) 2024-03-06T10:26:27.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:27.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:27.111836+0000) 2024-03-06T10:26:28.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:28.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:28.112108+0000) 2024-03-06T10:26:29.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:29.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:29.112334+0000) 2024-03-06T10:26:30.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:30.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:30.112616+0000) 2024-03-06T10:26:31.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:31.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:31.112798+0000) 2024-03-06T10:26:32.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:32.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:32.113061+0000) 2024-03-06T10:26:33.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:33.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:33.113267+0000) 2024-03-06T10:26:34.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:34.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:34.113484+0000) 2024-03-06T10:26:35.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:35.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:35.113756+0000) 2024-03-06T10:26:36.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:36.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:36.114075+0000) 2024-03-06T10:26:37.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:37.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:37.114366+0000) 2024-03-06T10:26:38.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:38.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:38.114638+0000) 2024-03-06T10:26:39.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:39.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:39.114923+0000) 2024-03-06T10:26:40.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:40.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:40.115152+0000) 2024-03-06T10:26:41.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:41.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:41.115380+0000) 2024-03-06T10:26:42.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:42.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:42.115618+0000) 2024-03-06T10:26:43.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:43.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:43.115890+0000) 2024-03-06T10:26:44.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:44.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:44.116172+0000) 2024-03-06T10:26:45.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:45.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:45.116443+0000) 2024-03-06T10:26:46.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:46.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:46.116717+0000) 2024-03-06T10:26:47.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:47.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:47.116993+0000) 2024-03-06T10:26:48.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:48.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:48.117306+0000) 2024-03-06T10:26:49.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:49.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:49.117577+0000) 2024-03-06T10:26:50.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:50.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:50.117822+0000) 2024-03-06T10:26:51.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:51.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:51.118107+0000) 2024-03-06T10:26:52.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:52.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:52.118370+0000) 2024-03-06T10:26:52.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:52.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:26:53.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:53.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:53.118632+0000) 2024-03-06T10:26:54.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:54.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:54.118860+0000) 2024-03-06T10:26:55.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:55.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:55.119116+0000) 2024-03-06T10:26:56.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:56.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:56.119319+0000) 2024-03-06T10:26:57.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:57.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:57.119510+0000) 2024-03-06T10:26:58.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:58.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:58.119786+0000) 2024-03-06T10:26:59.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:26:59.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:26:59.120116+0000) 2024-03-06T10:27:00.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:00.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:00.120361+0000) 2024-03-06T10:27:01.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:01.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:01.120589+0000) 2024-03-06T10:27:02.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:02.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:02.120819+0000) 2024-03-06T10:27:03.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:03.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:03.121182+0000) 2024-03-06T10:27:04.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:04.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:04.121455+0000) 2024-03-06T10:27:05.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:05.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:05.121738+0000) 2024-03-06T10:27:06.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:06.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:06.122044+0000) 2024-03-06T10:27:07.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:07.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:07.122293+0000) 2024-03-06T10:27:08.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:08.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:08.122566+0000) 2024-03-06T10:27:09.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:09.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:09.122836+0000) 2024-03-06T10:27:10.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:10.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:10.123150+0000) 2024-03-06T10:27:11.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:11.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:11.123421+0000) 2024-03-06T10:27:12.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:12.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:12.123707+0000) 2024-03-06T10:27:13.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:13.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:13.123980+0000) 2024-03-06T10:27:14.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:14.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:14.124301+0000) 2024-03-06T10:27:15.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:15.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:15.124523+0000) 2024-03-06T10:27:16.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:16.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:16.124742+0000) 2024-03-06T10:27:17.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:17.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:17.125056+0000) 2024-03-06T10:27:18.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:18.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:18.125331+0000) 2024-03-06T10:27:19.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:19.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:19.125601+0000) 2024-03-06T10:27:20.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:20.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:20.125845+0000) 2024-03-06T10:27:21.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:21.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:21.126124+0000) 2024-03-06T10:27:22.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:22.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:22.126354+0000) 2024-03-06T10:27:22.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:22.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:23.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:23.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:23.126778+0000) 2024-03-06T10:27:24.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:24.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:24.127064+0000) 2024-03-06T10:27:25.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:25.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:25.127295+0000) 2024-03-06T10:27:26.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:26.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:26.127517+0000) 2024-03-06T10:27:27.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:27.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:27.127782+0000) 2024-03-06T10:27:28.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:28.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:28.128110+0000) 2024-03-06T10:27:29.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:29.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:29.128393+0000) 2024-03-06T10:27:30.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:30.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:30.128664+0000) 2024-03-06T10:27:31.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:31.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:31.128966+0000) 2024-03-06T10:27:32.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:32.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:32.129300+0000) 2024-03-06T10:27:33.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:33.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:33.129512+0000) 2024-03-06T10:27:34.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:34.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:34.129736+0000) 2024-03-06T10:27:35.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:35.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:35.129960+0000) 2024-03-06T10:27:36.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:36.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:36.130248+0000) 2024-03-06T10:27:37.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:37.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:37.130524+0000) 2024-03-06T10:27:38.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:38.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:38.130800+0000) 2024-03-06T10:27:39.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:39.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:39.131051+0000) 2024-03-06T10:27:40.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:40.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:40.131304+0000) 2024-03-06T10:27:41.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:41.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:41.131539+0000) 2024-03-06T10:27:42.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:42.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:42.131767+0000) 2024-03-06T10:27:43.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:43.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:43.131995+0000) 2024-03-06T10:27:44.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:44.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:44.132263+0000) 2024-03-06T10:27:45.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:45.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:45.132489+0000) 2024-03-06T10:27:46.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:46.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:46.132714+0000) 2024-03-06T10:27:47.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:47.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:47.132948+0000) 2024-03-06T10:27:48.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:48.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:48.133126+0000) 2024-03-06T10:27:49.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:49.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:49.133337+0000) 2024-03-06T10:27:50.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:50.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:50.133585+0000) 2024-03-06T10:27:51.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:51.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:51.133818+0000) 2024-03-06T10:27:52.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:52.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:52.134119+0000) 2024-03-06T10:27:52.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:52.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:27:53.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:53.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:53.134393+0000) 2024-03-06T10:27:54.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:54.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:54.134675+0000) 2024-03-06T10:27:55.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:55.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:55.134886+0000) 2024-03-06T10:27:56.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:56.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:56.135128+0000) 2024-03-06T10:27:57.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:57.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:57.135388+0000) 2024-03-06T10:27:58.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:58.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:58.135615+0000) 2024-03-06T10:27:59.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:27:59.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:27:59.135838+0000) 2024-03-06T10:28:00.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:00.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:00.136123+0000) 2024-03-06T10:28:01.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:01.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:01.136349+0000) 2024-03-06T10:28:02.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:02.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:02.136589+0000) 2024-03-06T10:28:03.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:03.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:03.136816+0000) 2024-03-06T10:28:04.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:04.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:04.137084+0000) 2024-03-06T10:28:05.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:05.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:05.137339+0000) 2024-03-06T10:28:06.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:06.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:06.137521+0000) 2024-03-06T10:28:07.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:07.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:07.137726+0000) 2024-03-06T10:28:08.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:08.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:08.137959+0000) 2024-03-06T10:28:09.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:09.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:09.138183+0000) 2024-03-06T10:28:10.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:10.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:10.138385+0000) 2024-03-06T10:28:11.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:11.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:11.138615+0000) 2024-03-06T10:28:12.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:12.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:12.138857+0000) 2024-03-06T10:28:13.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:13.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:13.139119+0000) 2024-03-06T10:28:14.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:14.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:14.139358+0000) 2024-03-06T10:28:15.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:15.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:15.139581+0000) 2024-03-06T10:28:16.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:16.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:16.139781+0000) 2024-03-06T10:28:17.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:17.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:17.139992+0000) 2024-03-06T10:28:18.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:18.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:18.140263+0000) 2024-03-06T10:28:19.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:19.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:19.140491+0000) 2024-03-06T10:28:20.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:20.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:20.140759+0000) 2024-03-06T10:28:21.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:21.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:21.141043+0000) 2024-03-06T10:28:22.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:22.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:22.141335+0000) 2024-03-06T10:28:22.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:22.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:23.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:23.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:23.141637+0000) 2024-03-06T10:28:24.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:24.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:24.141881+0000) 2024-03-06T10:28:25.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:25.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:25.142150+0000) 2024-03-06T10:28:26.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:26.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:26.142425+0000) 2024-03-06T10:28:27.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:27.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:27.142690+0000) 2024-03-06T10:28:28.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:28.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:28.142932+0000) 2024-03-06T10:28:29.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:29.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:29.143203+0000) 2024-03-06T10:28:30.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:30.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:30.143451+0000) 2024-03-06T10:28:31.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:31.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:31.143720+0000) 2024-03-06T10:28:32.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:32.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:32.143934+0000) 2024-03-06T10:28:33.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:33.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:33.144173+0000) 2024-03-06T10:28:34.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:34.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:34.144400+0000) 2024-03-06T10:28:35.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:35.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:35.144636+0000) 2024-03-06T10:28:36.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:36.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:36.144870+0000) 2024-03-06T10:28:37.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:37.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:37.145155+0000) 2024-03-06T10:28:38.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:38.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:38.145383+0000) 2024-03-06T10:28:39.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:39.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:39.145609+0000) 2024-03-06T10:28:40.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:40.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:40.145842+0000) 2024-03-06T10:28:41.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:41.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:41.146114+0000) 2024-03-06T10:28:42.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:42.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:42.146355+0000) 2024-03-06T10:28:43.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:43.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:43.146588+0000) 2024-03-06T10:28:44.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:44.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:44.146817+0000) 2024-03-06T10:28:45.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:45.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:45.147148+0000) 2024-03-06T10:28:46.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:46.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:46.147388+0000) 2024-03-06T10:28:47.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:47.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:47.147624+0000) 2024-03-06T10:28:48.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:48.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:48.147851+0000) 2024-03-06T10:28:49.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:49.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:49.148116+0000) 2024-03-06T10:28:50.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:50.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:50.148384+0000) 2024-03-06T10:28:51.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:51.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:51.148668+0000) 2024-03-06T10:28:52.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:52.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:52.148953+0000) 2024-03-06T10:28:53.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:53.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:28:53.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:53.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:53.149165+0000) 2024-03-06T10:28:54.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:54.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:54.149392+0000) 2024-03-06T10:28:55.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:55.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:55.149634+0000) 2024-03-06T10:28:56.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:56.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:56.149882+0000) 2024-03-06T10:28:57.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:57.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:57.150168+0000) 2024-03-06T10:28:58.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:58.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:58.150412+0000) 2024-03-06T10:28:59.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:28:59.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:28:59.150685+0000) 2024-03-06T10:29:00.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:00.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:00.150934+0000) 2024-03-06T10:29:01.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:01.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:01.151198+0000) 2024-03-06T10:29:02.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:02.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:02.151476+0000) 2024-03-06T10:29:03.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:03.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:03.151718+0000) 2024-03-06T10:29:04.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:04.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:04.151942+0000) 2024-03-06T10:29:05.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:05.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:05.152157+0000) 2024-03-06T10:29:06.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:06.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:06.152369+0000) 2024-03-06T10:29:07.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:07.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:07.152577+0000) 2024-03-06T10:29:08.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:08.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:08.152820+0000) 2024-03-06T10:29:09.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:09.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:09.153069+0000) 2024-03-06T10:29:10.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:10.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:10.153319+0000) 2024-03-06T10:29:11.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:11.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:11.153554+0000) 2024-03-06T10:29:12.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:12.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:12.153760+0000) 2024-03-06T10:29:13.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:13.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:13.153994+0000) 2024-03-06T10:29:14.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:14.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:14.154239+0000) 2024-03-06T10:29:15.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:15.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:15.154468+0000) 2024-03-06T10:29:16.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:16.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:16.154691+0000) 2024-03-06T10:29:17.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:17.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:17.154915+0000) 2024-03-06T10:29:18.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:18.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:18.155193+0000) 2024-03-06T10:29:19.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:19.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:19.155445+0000) 2024-03-06T10:29:20.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:20.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:20.155695+0000) 2024-03-06T10:29:21.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:21.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:21.155959+0000) 2024-03-06T10:29:22.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:22.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:22.156270+0000) 2024-03-06T10:29:23.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:23.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:23.156569+0000) 2024-03-06T10:29:23.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:23.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:24.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:24.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:24.156852+0000) 2024-03-06T10:29:25.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:25.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:25.157135+0000) 2024-03-06T10:29:26.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:26.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:26.157405+0000) 2024-03-06T10:29:27.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:27.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:27.157643+0000) 2024-03-06T10:29:28.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:28.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:28.157867+0000) 2024-03-06T10:29:29.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:29.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:29.158155+0000) 2024-03-06T10:29:30.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:30.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:30.158376+0000) 2024-03-06T10:29:31.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:31.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:31.158609+0000) 2024-03-06T10:29:32.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:32.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:32.158843+0000) 2024-03-06T10:29:33.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:33.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:33.159124+0000) 2024-03-06T10:29:34.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:34.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:34.159390+0000) 2024-03-06T10:29:35.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:35.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:35.159675+0000) 2024-03-06T10:29:36.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:36.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:36.159927+0000) 2024-03-06T10:29:37.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:37.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:37.160178+0000) 2024-03-06T10:29:38.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:38.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:38.160409+0000) 2024-03-06T10:29:39.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:39.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:39.160635+0000) 2024-03-06T10:29:40.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:40.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:40.160868+0000) 2024-03-06T10:29:41.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:41.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:41.161123+0000) 2024-03-06T10:29:42.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:42.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:42.161356+0000) 2024-03-06T10:29:43.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:43.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:43.161594+0000) 2024-03-06T10:29:44.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:44.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:44.161821+0000) 2024-03-06T10:29:45.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:45.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:45.162101+0000) 2024-03-06T10:29:46.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:46.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:46.162308+0000) 2024-03-06T10:29:47.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:47.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:47.162488+0000) 2024-03-06T10:29:48.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:48.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:48.162710+0000) 2024-03-06T10:29:49.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:49.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:49.162920+0000) 2024-03-06T10:29:50.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:50.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:50.163181+0000) 2024-03-06T10:29:51.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:51.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:51.163430+0000) 2024-03-06T10:29:52.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:52.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:52.163670+0000) 2024-03-06T10:29:53.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:53.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:53.163900+0000) 2024-03-06T10:29:53.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:53.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:29:54.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:54.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:54.164165+0000) 2024-03-06T10:29:55.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:55.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:55.164406+0000) 2024-03-06T10:29:56.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:56.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:56.164572+0000) 2024-03-06T10:29:57.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:57.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:57.164799+0000) 2024-03-06T10:29:58.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:58.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:58.165054+0000) 2024-03-06T10:29:59.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:29:59.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:29:59.165335+0000) 2024-03-06T10:30:00.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:00.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:00.165624+0000) 2024-03-06T10:30:01.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:01.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:01.165836+0000) 2024-03-06T10:30:02.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:02.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:02.166120+0000) 2024-03-06T10:30:03.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:03.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:03.166356+0000) 2024-03-06T10:30:04.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:04.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:04.166590+0000) 2024-03-06T10:30:05.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:05.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:05.166893+0000) 2024-03-06T10:30:06.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:06.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:06.167168+0000) 2024-03-06T10:30:07.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:07.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:07.167378+0000) 2024-03-06T10:30:08.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:08.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:08.167588+0000) 2024-03-06T10:30:09.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:09.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:09.167839+0000) 2024-03-06T10:30:10.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:10.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:10.168111+0000) 2024-03-06T10:30:11.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:11.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:11.168361+0000) 2024-03-06T10:30:12.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:12.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:12.168618+0000) 2024-03-06T10:30:13.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:13.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:13.168872+0000) 2024-03-06T10:30:14.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:14.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:14.169144+0000) 2024-03-06T10:30:15.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:15.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:15.169387+0000) 2024-03-06T10:30:16.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:16.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:16.169600+0000) 2024-03-06T10:30:17.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:17.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:17.169821+0000) 2024-03-06T10:30:18.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:18.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:18.170042+0000) 2024-03-06T10:30:19.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:19.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:19.170287+0000) 2024-03-06T10:30:20.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:20.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:20.170546+0000) 2024-03-06T10:30:21.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:21.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:21.170787+0000) 2024-03-06T10:30:22.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:22.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:22.171046+0000) 2024-03-06T10:30:23.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:23.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:23.171271+0000) 2024-03-06T10:30:23.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:23.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:24.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:24.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:24.171552+0000) 2024-03-06T10:30:25.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:25.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:25.171817+0000) 2024-03-06T10:30:26.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:26.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:26.172190+0000) 2024-03-06T10:30:27.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:27.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:27.172408+0000) 2024-03-06T10:30:28.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:28.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:28.172647+0000) 2024-03-06T10:30:29.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:29.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:29.172879+0000) 2024-03-06T10:30:30.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:30.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:30.173155+0000) 2024-03-06T10:30:31.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:31.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:31.173327+0000) 2024-03-06T10:30:32.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:32.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:32.173503+0000) 2024-03-06T10:30:33.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:33.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:33.173700+0000) 2024-03-06T10:30:34.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:34.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:34.173928+0000) 2024-03-06T10:30:35.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:35.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:35.174156+0000) 2024-03-06T10:30:36.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:36.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:36.174386+0000) 2024-03-06T10:30:37.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:37.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:37.174611+0000) 2024-03-06T10:30:38.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:38.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:38.174878+0000) 2024-03-06T10:30:39.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:39.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:39.175138+0000) 2024-03-06T10:30:40.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:40.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:40.175353+0000) 2024-03-06T10:30:41.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:41.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:41.175581+0000) 2024-03-06T10:30:42.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:42.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:42.175795+0000) 2024-03-06T10:30:43.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:43.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:43.176091+0000) 2024-03-06T10:30:44.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:44.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:44.176324+0000) 2024-03-06T10:30:45.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:45.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:45.176555+0000) 2024-03-06T10:30:46.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:46.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:46.176836+0000) 2024-03-06T10:30:47.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:47.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:47.177141+0000) 2024-03-06T10:30:48.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:48.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:48.177356+0000) 2024-03-06T10:30:49.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:49.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:49.177574+0000) 2024-03-06T10:30:50.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:50.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:50.177710+0000) 2024-03-06T10:30:51.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:51.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:51.177935+0000) 2024-03-06T10:30:52.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:52.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:52.178159+0000) 2024-03-06T10:30:53.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:53.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:53.178360+0000) 2024-03-06T10:30:53.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:53.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:30:54.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:54.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:54.178512+0000) 2024-03-06T10:30:55.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:55.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:55.178730+0000) 2024-03-06T10:30:56.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:56.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:56.178938+0000) 2024-03-06T10:30:57.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:57.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:57.179174+0000) 2024-03-06T10:30:58.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:58.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:58.179406+0000) 2024-03-06T10:30:59.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:30:59.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:30:59.179635+0000) 2024-03-06T10:31:00.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:00.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:00.179887+0000) 2024-03-06T10:31:01.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:01.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:01.180178+0000) 2024-03-06T10:31:02.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:02.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:02.180458+0000) 2024-03-06T10:31:03.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:03.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:03.180738+0000) 2024-03-06T10:31:04.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:04.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:04.181057+0000) 2024-03-06T10:31:05.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:05.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:05.181352+0000) 2024-03-06T10:31:06.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:06.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:06.181627+0000) 2024-03-06T10:31:07.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:07.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:07.181852+0000) 2024-03-06T10:31:08.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:08.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:08.182104+0000) 2024-03-06T10:31:09.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:09.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:09.182351+0000) 2024-03-06T10:31:10.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:10.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:10.182611+0000) 2024-03-06T10:31:11.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:11.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:11.182852+0000) 2024-03-06T10:31:12.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:12.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:12.183125+0000) 2024-03-06T10:31:13.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:13.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:13.183361+0000) 2024-03-06T10:31:14.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:14.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:14.183638+0000) 2024-03-06T10:31:15.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:15.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:15.183858+0000) 2024-03-06T10:31:16.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:16.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:16.184127+0000) 2024-03-06T10:31:17.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:17.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:17.184347+0000) 2024-03-06T10:31:18.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:18.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:18.184528+0000) 2024-03-06T10:31:19.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:19.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:19.184753+0000) 2024-03-06T10:31:20.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:20.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:20.184987+0000) 2024-03-06T10:31:21.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:21.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:21.185273+0000) 2024-03-06T10:31:22.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:22.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:22.185519+0000) 2024-03-06T10:31:23.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:23.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:23.185803+0000) 2024-03-06T10:31:24.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:24.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:24.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:24.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:24.186135+0000) 2024-03-06T10:31:25.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:25.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:25.186408+0000) 2024-03-06T10:31:26.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:26.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:26.186689+0000) 2024-03-06T10:31:27.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:27.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:27.186934+0000) 2024-03-06T10:31:28.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:28.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:28.187220+0000) 2024-03-06T10:31:29.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:29.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:29.187501+0000) 2024-03-06T10:31:30.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:30.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:30.187746+0000) 2024-03-06T10:31:31.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:31.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:31.187953+0000) 2024-03-06T10:31:32.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:32.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:32.188144+0000) 2024-03-06T10:31:33.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:33.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:33.188402+0000) 2024-03-06T10:31:34.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:34.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:34.188639+0000) 2024-03-06T10:31:35.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:35.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:35.188893+0000) 2024-03-06T10:31:36.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:36.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:36.189140+0000) 2024-03-06T10:31:37.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:37.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:37.189391+0000) 2024-03-06T10:31:38.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:38.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:38.189590+0000) 2024-03-06T10:31:39.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:39.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:39.189825+0000) 2024-03-06T10:31:40.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:40.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:40.190087+0000) 2024-03-06T10:31:41.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:41.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:41.190322+0000) 2024-03-06T10:31:42.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:42.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:42.190548+0000) 2024-03-06T10:31:43.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:43.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:43.190788+0000) 2024-03-06T10:31:44.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:44.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:44.191096+0000) 2024-03-06T10:31:45.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:45.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:45.191295+0000) 2024-03-06T10:31:46.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:46.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:46.191538+0000) 2024-03-06T10:31:47.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:47.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:47.191781+0000) 2024-03-06T10:31:48.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:48.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:48.192067+0000) 2024-03-06T10:31:49.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:49.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:49.192330+0000) 2024-03-06T10:31:50.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:50.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:50.192530+0000) 2024-03-06T10:31:51.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:51.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:51.192758+0000) 2024-03-06T10:31:52.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:52.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:52.192971+0000) 2024-03-06T10:31:53.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:53.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:53.193218+0000) 2024-03-06T10:31:54.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:54.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:54.193447+0000) 2024-03-06T10:31:54.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:54.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:31:55.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:55.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:55.193676+0000) 2024-03-06T10:31:56.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:56.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:56.193911+0000) 2024-03-06T10:31:57.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:57.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:57.194149+0000) 2024-03-06T10:31:58.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:58.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:58.194361+0000) 2024-03-06T10:31:59.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:31:59.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:31:59.194585+0000) 2024-03-06T10:32:00.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:00.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:00.194813+0000) 2024-03-06T10:32:01.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:01.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:01.195118+0000) 2024-03-06T10:32:02.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:02.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:02.195305+0000) 2024-03-06T10:32:03.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:03.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:03.195553+0000) 2024-03-06T10:32:04.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:04.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:04.195799+0000) 2024-03-06T10:32:05.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:05.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:05.196049+0000) 2024-03-06T10:32:06.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:06.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:06.196263+0000) 2024-03-06T10:32:07.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:07.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:07.196485+0000) 2024-03-06T10:32:08.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:08.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:08.196715+0000) 2024-03-06T10:32:09.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:09.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:09.196928+0000) 2024-03-06T10:32:10.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:10.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:10.197173+0000) 2024-03-06T10:32:11.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:11.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:11.197370+0000) 2024-03-06T10:32:12.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:12.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:12.197558+0000) 2024-03-06T10:32:13.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:13.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:13.197742+0000) 2024-03-06T10:32:14.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:14.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:14.197967+0000) 2024-03-06T10:32:15.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:15.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:15.198200+0000) 2024-03-06T10:32:16.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:16.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:16.198423+0000) 2024-03-06T10:32:17.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:17.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:17.198638+0000) 2024-03-06T10:32:18.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:18.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:18.198876+0000) 2024-03-06T10:32:19.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:19.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:19.199148+0000) 2024-03-06T10:32:20.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:20.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:20.199374+0000) 2024-03-06T10:32:21.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:21.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:21.199570+0000) 2024-03-06T10:32:22.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:22.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:22.199796+0000) 2024-03-06T10:32:23.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:23.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:23.200068+0000) 2024-03-06T10:32:24.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:24.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:24.200286+0000) 2024-03-06T10:32:24.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:24.352 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:25.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:25.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:25.200519+0000) 2024-03-06T10:32:26.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:26.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:26.200745+0000) 2024-03-06T10:32:27.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:27.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:27.200968+0000) 2024-03-06T10:32:28.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:28.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:28.201205+0000) 2024-03-06T10:32:29.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:29.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:29.201432+0000) 2024-03-06T10:32:30.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:30.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:30.201653+0000) 2024-03-06T10:32:31.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:31.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:31.201887+0000) 2024-03-06T10:32:32.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:32.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:32.202131+0000) 2024-03-06T10:32:33.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:33.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:33.202347+0000) 2024-03-06T10:32:34.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:34.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:34.202515+0000) 2024-03-06T10:32:35.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:35.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:35.202746+0000) 2024-03-06T10:32:36.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:36.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:36.203108+0000) 2024-03-06T10:32:37.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:37.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:37.203298+0000) 2024-03-06T10:32:38.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:38.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:38.203451+0000) 2024-03-06T10:32:39.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:39.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:39.203646+0000) 2024-03-06T10:32:40.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:40.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:40.203872+0000) 2024-03-06T10:32:41.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:41.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:41.204136+0000) 2024-03-06T10:32:42.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:42.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:42.204403+0000) 2024-03-06T10:32:43.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:43.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:43.204657+0000) 2024-03-06T10:32:44.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:44.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:44.204839+0000) 2024-03-06T10:32:45.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:45.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:45.205036+0000) 2024-03-06T10:32:46.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:46.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:46.205260+0000) 2024-03-06T10:32:47.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:47.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:47.205463+0000) 2024-03-06T10:32:48.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:48.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:48.205696+0000) 2024-03-06T10:32:49.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:49.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:49.205884+0000) 2024-03-06T10:32:50.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:50.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:50.206140+0000) 2024-03-06T10:32:51.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:51.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:51.206345+0000) 2024-03-06T10:32:52.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:52.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:52.206601+0000) 2024-03-06T10:32:53.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:53.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:53.206852+0000) 2024-03-06T10:32:54.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:54.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:54.207146+0000) 2024-03-06T10:32:54.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:54.565 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:32:55.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:55.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:55.207397+0000) 2024-03-06T10:32:56.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:56.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:56.207622+0000) 2024-03-06T10:32:57.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:57.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:57.207859+0000) 2024-03-06T10:32:58.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:58.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:58.208104+0000) 2024-03-06T10:32:59.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:32:59.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:32:59.208302+0000) 2024-03-06T10:33:00.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:00.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:00.208535+0000) 2024-03-06T10:33:01.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:01.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:01.208838+0000) 2024-03-06T10:33:02.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:02.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:02.209083+0000) 2024-03-06T10:33:03.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:03.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:03.209320+0000) 2024-03-06T10:33:04.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:04.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:04.209545+0000) 2024-03-06T10:33:05.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:05.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:05.209798+0000) 2024-03-06T10:33:06.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:06.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:06.210082+0000) 2024-03-06T10:33:07.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:07.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:07.210312+0000) 2024-03-06T10:33:08.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:08.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:08.210542+0000) 2024-03-06T10:33:09.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:09.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:09.210800+0000) 2024-03-06T10:33:10.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:10.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:10.211065+0000) 2024-03-06T10:33:11.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:11.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:11.211302+0000) 2024-03-06T10:33:12.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:12.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:12.211520+0000) 2024-03-06T10:33:13.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:13.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:13.211752+0000) 2024-03-06T10:33:14.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:14.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:14.212042+0000) 2024-03-06T10:33:15.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:15.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:15.212296+0000) 2024-03-06T10:33:16.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:16.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:16.212527+0000) 2024-03-06T10:33:17.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:17.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:17.212758+0000) 2024-03-06T10:33:18.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:18.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:18.212987+0000) 2024-03-06T10:33:19.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:19.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:19.213242+0000) 2024-03-06T10:33:20.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:20.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:20.213465+0000) 2024-03-06T10:33:21.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:21.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:21.213702+0000) 2024-03-06T10:33:22.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:22.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:22.213929+0000) 2024-03-06T10:33:23.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:23.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:23.214165+0000) 2024-03-06T10:33:24.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:24.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:24.214392+0000) 2024-03-06T10:33:24.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:24.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:25.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:25.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:25.214607+0000) 2024-03-06T10:33:26.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:26.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:26.214842+0000) 2024-03-06T10:33:27.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:27.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:27.215119+0000) 2024-03-06T10:33:28.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:28.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:28.215342+0000) 2024-03-06T10:33:29.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:29.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:29.215570+0000) 2024-03-06T10:33:30.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:30.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:30.215794+0000) 2024-03-06T10:33:31.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:31.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:31.216058+0000) 2024-03-06T10:33:32.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:32.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:32.216292+0000) 2024-03-06T10:33:33.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:33.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:33.216511+0000) 2024-03-06T10:33:34.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:34.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:34.216746+0000) 2024-03-06T10:33:35.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:35.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:35.216973+0000) 2024-03-06T10:33:36.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:36.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:36.217220+0000) 2024-03-06T10:33:37.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:37.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:37.217442+0000) 2024-03-06T10:33:38.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:38.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:38.217685+0000) 2024-03-06T10:33:39.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:39.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:39.217932+0000) 2024-03-06T10:33:40.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:40.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:40.218162+0000) 2024-03-06T10:33:41.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:41.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:41.218361+0000) 2024-03-06T10:33:42.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:42.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:42.218608+0000) 2024-03-06T10:33:43.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:43.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:43.218917+0000) 2024-03-06T10:33:44.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:44.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:44.219177+0000) 2024-03-06T10:33:45.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:45.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:45.219431+0000) 2024-03-06T10:33:46.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:46.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:46.219673+0000) 2024-03-06T10:33:47.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:47.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:47.219923+0000) 2024-03-06T10:33:48.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:48.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:48.220183+0000) 2024-03-06T10:33:49.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:49.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:49.220399+0000) 2024-03-06T10:33:50.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:50.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:50.220646+0000) 2024-03-06T10:33:51.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:51.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:51.220915+0000) 2024-03-06T10:33:52.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:52.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:52.221169+0000) 2024-03-06T10:33:53.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:53.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:53.221425+0000) 2024-03-06T10:33:54.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:54.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:54.221670+0000) 2024-03-06T10:33:54.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:54.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:33:55.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:55.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:55.221933+0000) 2024-03-06T10:33:56.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:56.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:56.222202+0000) 2024-03-06T10:33:57.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:57.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:57.222419+0000) 2024-03-06T10:33:58.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:58.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:58.222671+0000) 2024-03-06T10:33:59.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:33:59.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:33:59.222926+0000) 2024-03-06T10:34:00.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:00.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:00.223161+0000) 2024-03-06T10:34:01.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:01.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:01.223357+0000) 2024-03-06T10:34:02.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:02.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:02.223599+0000) 2024-03-06T10:34:03.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:03.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:03.223834+0000) 2024-03-06T10:34:04.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:04.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:04.224094+0000) 2024-03-06T10:34:05.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:05.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:05.224311+0000) 2024-03-06T10:34:06.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:06.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:06.224510+0000) 2024-03-06T10:34:07.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:07.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:07.224746+0000) 2024-03-06T10:34:08.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:08.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:08.225048+0000) 2024-03-06T10:34:09.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:09.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:09.225295+0000) 2024-03-06T10:34:10.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:10.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:10.225474+0000) 2024-03-06T10:34:11.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:11.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:11.225626+0000) 2024-03-06T10:34:12.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:12.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:12.225957+0000) 2024-03-06T10:34:13.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:13.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:13.226220+0000) 2024-03-06T10:34:14.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:14.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:14.226426+0000) 2024-03-06T10:34:15.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:15.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:15.226651+0000) 2024-03-06T10:34:16.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:16.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:16.226878+0000) 2024-03-06T10:34:17.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:17.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:17.227145+0000) 2024-03-06T10:34:18.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:18.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:18.227383+0000) 2024-03-06T10:34:19.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:19.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:19.227572+0000) 2024-03-06T10:34:20.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:20.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:20.227754+0000) 2024-03-06T10:34:21.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:21.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:21.227949+0000) 2024-03-06T10:34:22.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:22.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:22.228160+0000) 2024-03-06T10:34:23.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:23.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:23.228406+0000) 2024-03-06T10:34:24.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:24.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:24.228618+0000) 2024-03-06T10:34:25.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:25.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:25.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:25.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:25.228860+0000) 2024-03-06T10:34:26.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:26.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:26.229148+0000) 2024-03-06T10:34:27.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:27.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:27.229300+0000) 2024-03-06T10:34:28.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:28.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:28.229539+0000) 2024-03-06T10:34:29.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:29.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:29.229762+0000) 2024-03-06T10:34:30.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:30.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:30.229987+0000) 2024-03-06T10:34:31.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:31.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:31.230301+0000) 2024-03-06T10:34:32.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:32.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:32.230509+0000) 2024-03-06T10:34:33.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:33.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:33.230708+0000) 2024-03-06T10:34:34.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:34.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:34.230950+0000) 2024-03-06T10:34:35.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:35.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:35.231236+0000) 2024-03-06T10:34:36.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:36.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:36.231434+0000) 2024-03-06T10:34:37.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:37.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:37.231661+0000) 2024-03-06T10:34:38.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:38.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:38.231913+0000) 2024-03-06T10:34:39.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:39.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:39.232067+0000) 2024-03-06T10:34:40.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:40.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:40.232284+0000) 2024-03-06T10:34:41.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:41.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:41.232515+0000) 2024-03-06T10:34:42.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:42.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:42.232789+0000) 2024-03-06T10:34:43.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:43.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:43.232988+0000) 2024-03-06T10:34:44.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:44.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:44.233274+0000) 2024-03-06T10:34:45.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:45.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:45.233556+0000) 2024-03-06T10:34:46.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:46.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:46.233827+0000) 2024-03-06T10:34:47.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:47.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:47.234117+0000) 2024-03-06T10:34:48.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:48.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:48.234356+0000) 2024-03-06T10:34:49.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:49.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:49.234588+0000) 2024-03-06T10:34:50.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:50.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:50.234813+0000) 2024-03-06T10:34:51.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:51.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:51.235093+0000) 2024-03-06T10:34:52.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:52.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:52.235345+0000) 2024-03-06T10:34:53.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:53.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:53.235636+0000) 2024-03-06T10:34:54.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:54.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:54.235846+0000) 2024-03-06T10:34:55.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:55.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:55.236095+0000) 2024-03-06T10:34:55.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:55.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:34:56.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:56.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:56.236295+0000) 2024-03-06T10:34:57.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:57.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:57.236501+0000) 2024-03-06T10:34:58.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:58.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:58.236735+0000) 2024-03-06T10:34:59.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:34:59.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:34:59.236959+0000) 2024-03-06T10:35:00.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:00.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:00.237188+0000) 2024-03-06T10:35:01.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:01.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:01.237461+0000) 2024-03-06T10:35:02.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:02.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:02.237667+0000) 2024-03-06T10:35:03.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:03.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:03.237901+0000) 2024-03-06T10:35:04.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:04.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:04.238151+0000) 2024-03-06T10:35:05.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:05.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:05.238337+0000) 2024-03-06T10:35:06.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:06.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:06.238549+0000) 2024-03-06T10:35:07.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:07.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:07.238775+0000) 2024-03-06T10:35:08.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:08.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:08.239033+0000) 2024-03-06T10:35:09.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:09.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:09.239268+0000) 2024-03-06T10:35:10.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:10.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:10.239478+0000) 2024-03-06T10:35:11.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:11.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:11.239696+0000) 2024-03-06T10:35:12.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:12.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:12.239919+0000) 2024-03-06T10:35:13.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:13.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:13.240184+0000) 2024-03-06T10:35:14.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:14.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:14.240474+0000) 2024-03-06T10:35:15.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:15.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:15.240746+0000) 2024-03-06T10:35:16.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:16.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:16.240955+0000) 2024-03-06T10:35:17.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:17.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:17.241173+0000) 2024-03-06T10:35:18.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:18.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:18.241410+0000) 2024-03-06T10:35:19.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:19.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:19.241637+0000) 2024-03-06T10:35:20.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:20.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:20.241866+0000) 2024-03-06T10:35:21.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:21.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:21.242164+0000) 2024-03-06T10:35:22.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:22.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:22.242380+0000) 2024-03-06T10:35:23.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:23.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:23.242574+0000) 2024-03-06T10:35:24.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:24.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:24.242808+0000) 2024-03-06T10:35:25.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:25.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:25.243211+0000) 2024-03-06T10:35:25.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:25.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:26.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:26.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:26.243405+0000) 2024-03-06T10:35:27.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:27.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:27.243625+0000) 2024-03-06T10:35:28.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:28.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:28.243860+0000) 2024-03-06T10:35:29.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:29.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:29.244127+0000) 2024-03-06T10:35:30.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:30.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:30.244353+0000) 2024-03-06T10:35:31.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:31.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:31.244585+0000) 2024-03-06T10:35:32.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:32.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:32.244803+0000) 2024-03-06T10:35:33.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:33.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:33.245101+0000) 2024-03-06T10:35:34.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:34.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:34.245283+0000) 2024-03-06T10:35:35.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:35.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:35.245498+0000) 2024-03-06T10:35:36.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:36.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:36.245724+0000) 2024-03-06T10:35:37.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:37.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:37.245910+0000) 2024-03-06T10:35:38.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:38.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:38.246192+0000) 2024-03-06T10:35:39.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:39.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:39.246442+0000) 2024-03-06T10:35:40.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:40.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:40.246695+0000) 2024-03-06T10:35:41.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:41.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:41.246967+0000) 2024-03-06T10:35:42.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:42.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:42.247202+0000) 2024-03-06T10:35:43.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:43.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:43.247427+0000) 2024-03-06T10:35:44.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:44.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:44.247621+0000) 2024-03-06T10:35:45.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:45.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:45.247840+0000) 2024-03-06T10:35:46.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:46.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:46.248078+0000) 2024-03-06T10:35:47.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:47.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:47.248247+0000) 2024-03-06T10:35:48.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:48.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:48.248436+0000) 2024-03-06T10:35:49.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:49.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:49.248693+0000) 2024-03-06T10:35:50.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:50.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:50.248934+0000) 2024-03-06T10:35:51.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:51.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:51.249255+0000) 2024-03-06T10:35:52.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:52.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:52.249477+0000) 2024-03-06T10:35:53.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:53.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:53.249699+0000) 2024-03-06T10:35:54.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:54.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:54.249878+0000) 2024-03-06T10:35:55.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:55.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:55.250154+0000) 2024-03-06T10:35:55.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:55.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:35:56.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:56.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:56.250366+0000) 2024-03-06T10:35:57.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:57.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:57.250576+0000) 2024-03-06T10:35:58.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:58.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:58.250792+0000) 2024-03-06T10:35:59.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:35:59.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:35:59.251044+0000) 2024-03-06T10:36:00.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:00.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:00.251237+0000) 2024-03-06T10:36:01.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:01.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:01.251454+0000) 2024-03-06T10:36:02.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:02.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:02.251655+0000) 2024-03-06T10:36:03.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:03.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:03.251882+0000) 2024-03-06T10:36:04.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:04.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:04.252145+0000) 2024-03-06T10:36:05.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:05.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:05.252394+0000) 2024-03-06T10:36:06.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:06.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:06.252657+0000) 2024-03-06T10:36:07.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:07.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:07.252931+0000) 2024-03-06T10:36:08.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:08.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:08.253187+0000) 2024-03-06T10:36:09.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:09.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:09.253436+0000) 2024-03-06T10:36:10.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:10.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:10.253658+0000) 2024-03-06T10:36:11.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:11.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:11.253889+0000) 2024-03-06T10:36:12.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:12.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:12.254115+0000) 2024-03-06T10:36:13.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:13.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:13.254349+0000) 2024-03-06T10:36:14.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:14.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:14.254609+0000) 2024-03-06T10:36:15.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:15.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:15.254841+0000) 2024-03-06T10:36:16.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:16.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:16.255101+0000) 2024-03-06T10:36:17.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:17.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:17.255298+0000) 2024-03-06T10:36:18.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:18.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:18.255547+0000) 2024-03-06T10:36:19.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:19.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:19.255776+0000) 2024-03-06T10:36:20.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:20.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:20.256001+0000) 2024-03-06T10:36:21.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:21.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:21.256253+0000) 2024-03-06T10:36:22.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:22.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:22.256492+0000) 2024-03-06T10:36:23.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:23.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:23.256759+0000) 2024-03-06T10:36:24.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:24.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:24.256981+0000) 2024-03-06T10:36:25.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:25.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:25.257155+0000) 2024-03-06T10:36:25.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:25.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:26.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:26.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:26.257343+0000) 2024-03-06T10:36:27.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:27.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:27.257552+0000) 2024-03-06T10:36:28.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:28.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:28.257771+0000) 2024-03-06T10:36:29.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:29.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:29.258035+0000) 2024-03-06T10:36:30.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:30.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:30.258226+0000) 2024-03-06T10:36:31.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:31.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:31.258442+0000) 2024-03-06T10:36:32.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:32.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:32.258642+0000) 2024-03-06T10:36:33.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:33.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:33.258863+0000) 2024-03-06T10:36:34.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:34.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:34.259090+0000) 2024-03-06T10:36:35.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:35.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:35.259272+0000) 2024-03-06T10:36:36.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:36.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:36.259514+0000) 2024-03-06T10:36:37.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:37.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:37.259731+0000) 2024-03-06T10:36:38.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:38.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:38.259953+0000) 2024-03-06T10:36:39.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:39.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:39.260163+0000) 2024-03-06T10:36:40.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:40.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:40.260351+0000) 2024-03-06T10:36:41.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:41.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:41.260572+0000) 2024-03-06T10:36:42.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:42.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:42.260740+0000) 2024-03-06T10:36:43.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:43.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:43.260963+0000) 2024-03-06T10:36:44.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:44.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:44.261168+0000) 2024-03-06T10:36:45.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:45.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:45.261331+0000) 2024-03-06T10:36:46.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:46.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:46.261550+0000) 2024-03-06T10:36:47.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:47.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:47.261784+0000) 2024-03-06T10:36:48.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:48.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:48.262029+0000) 2024-03-06T10:36:49.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:49.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:49.262253+0000) 2024-03-06T10:36:50.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:50.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:50.262457+0000) 2024-03-06T10:36:51.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:51.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:51.262632+0000) 2024-03-06T10:36:52.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:52.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:52.262843+0000) 2024-03-06T10:36:53.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:53.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:53.263081+0000) 2024-03-06T10:36:54.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:54.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:54.263225+0000) 2024-03-06T10:36:55.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:55.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:55.263417+0000) 2024-03-06T10:36:56.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:56.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:36:56.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:56.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:56.263588+0000) 2024-03-06T10:36:57.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:57.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:57.263812+0000) 2024-03-06T10:36:58.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:58.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:58.264068+0000) 2024-03-06T10:36:59.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:36:59.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:36:59.264313+0000) 2024-03-06T10:37:00.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:00.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:00.264499+0000) 2024-03-06T10:37:01.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:01.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:01.264734+0000) 2024-03-06T10:37:02.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:02.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:02.264951+0000) 2024-03-06T10:37:03.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:03.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:03.265159+0000) 2024-03-06T10:37:04.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:04.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:04.265394+0000) 2024-03-06T10:37:05.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:05.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:05.265616+0000) 2024-03-06T10:37:06.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:06.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:06.265833+0000) 2024-03-06T10:37:07.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:07.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:07.266051+0000) 2024-03-06T10:37:08.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:08.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:08.266250+0000) 2024-03-06T10:37:09.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:09.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:09.266487+0000) 2024-03-06T10:37:10.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:10.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:10.266685+0000) 2024-03-06T10:37:11.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:11.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:11.266989+0000) 2024-03-06T10:37:12.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:12.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:12.267249+0000) 2024-03-06T10:37:13.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:13.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:13.267455+0000) 2024-03-06T10:37:14.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:14.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:14.267675+0000) 2024-03-06T10:37:15.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:15.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:15.267904+0000) 2024-03-06T10:37:16.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:16.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:16.268141+0000) 2024-03-06T10:37:17.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:17.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:17.268336+0000) 2024-03-06T10:37:18.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:18.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:18.268542+0000) 2024-03-06T10:37:19.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:19.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:19.268781+0000) 2024-03-06T10:37:20.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:20.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:20.269019+0000) 2024-03-06T10:37:21.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:21.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:21.269220+0000) 2024-03-06T10:37:22.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:22.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:22.269411+0000) 2024-03-06T10:37:23.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:23.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:23.269617+0000) 2024-03-06T10:37:24.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:24.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:24.269839+0000) 2024-03-06T10:37:25.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:25.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:25.270075+0000) 2024-03-06T10:37:26.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:26.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:26.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:26.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:26.270230+0000) 2024-03-06T10:37:27.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:27.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:27.270415+0000) 2024-03-06T10:37:28.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:28.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:28.270606+0000) 2024-03-06T10:37:29.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:29.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:29.270834+0000) 2024-03-06T10:37:30.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:30.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:30.271055+0000) 2024-03-06T10:37:31.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:31.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:31.271297+0000) 2024-03-06T10:37:32.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:32.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:32.271521+0000) 2024-03-06T10:37:33.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:33.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:33.271751+0000) 2024-03-06T10:37:34.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:34.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:34.271970+0000) 2024-03-06T10:37:35.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:35.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:35.272175+0000) 2024-03-06T10:37:36.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:36.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:36.272404+0000) 2024-03-06T10:37:37.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:37.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:37.272649+0000) 2024-03-06T10:37:38.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:38.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:38.272861+0000) 2024-03-06T10:37:39.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:39.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:39.273120+0000) 2024-03-06T10:37:40.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:40.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:40.273314+0000) 2024-03-06T10:37:41.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:41.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:41.273544+0000) 2024-03-06T10:37:42.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:42.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:42.273764+0000) 2024-03-06T10:37:43.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:43.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:43.273993+0000) 2024-03-06T10:37:44.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:44.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:44.274271+0000) 2024-03-06T10:37:45.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:45.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:45.274453+0000) 2024-03-06T10:37:46.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:46.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:46.274678+0000) 2024-03-06T10:37:47.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:47.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:47.274900+0000) 2024-03-06T10:37:48.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:48.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:48.275115+0000) 2024-03-06T10:37:49.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:49.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:49.275314+0000) 2024-03-06T10:37:50.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:50.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:50.275542+0000) 2024-03-06T10:37:51.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:51.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:51.275778+0000) 2024-03-06T10:37:52.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:52.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:52.276044+0000) 2024-03-06T10:37:53.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:53.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:53.276298+0000) 2024-03-06T10:37:54.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:54.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:54.276582+0000) 2024-03-06T10:37:55.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:55.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:55.276809+0000) 2024-03-06T10:37:56.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:56.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:56.276975+0000) 2024-03-06T10:37:56.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:56.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:37:57.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:57.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:57.277223+0000) 2024-03-06T10:37:58.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:58.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:58.277447+0000) 2024-03-06T10:37:59.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:37:59.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:37:59.277689+0000) 2024-03-06T10:38:00.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:00.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:00.277913+0000) 2024-03-06T10:38:01.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:01.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:01.278144+0000) 2024-03-06T10:38:02.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:02.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:02.278337+0000) 2024-03-06T10:38:03.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:03.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:03.278568+0000) 2024-03-06T10:38:04.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:04.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:04.278792+0000) 2024-03-06T10:38:05.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:05.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:05.279059+0000) 2024-03-06T10:38:06.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:06.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:06.279284+0000) 2024-03-06T10:38:07.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:07.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:07.279509+0000) 2024-03-06T10:38:08.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:08.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:08.279654+0000) 2024-03-06T10:38:09.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:09.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:09.279858+0000) 2024-03-06T10:38:10.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:10.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:10.280120+0000) 2024-03-06T10:38:11.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:11.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:11.280364+0000) 2024-03-06T10:38:12.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:12.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:12.280593+0000) 2024-03-06T10:38:13.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:13.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:13.280819+0000) 2024-03-06T10:38:14.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:14.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:14.281037+0000) 2024-03-06T10:38:15.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:15.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:15.281232+0000) 2024-03-06T10:38:16.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:16.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:16.281431+0000) 2024-03-06T10:38:17.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:17.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:17.281708+0000) 2024-03-06T10:38:18.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:18.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:18.281956+0000) 2024-03-06T10:38:19.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:19.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:19.282189+0000) 2024-03-06T10:38:20.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:20.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:20.282413+0000) 2024-03-06T10:38:21.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:21.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:21.282643+0000) 2024-03-06T10:38:22.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:22.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:22.282863+0000) 2024-03-06T10:38:23.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:23.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:23.283117+0000) 2024-03-06T10:38:24.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:24.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:24.283342+0000) 2024-03-06T10:38:25.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:25.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:25.283566+0000) 2024-03-06T10:38:26.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:26.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:26.283790+0000) 2024-03-06T10:38:26.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:26.576 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:27.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:27.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:27.284049+0000) 2024-03-06T10:38:28.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:28.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:28.284280+0000) 2024-03-06T10:38:29.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:29.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:29.284492+0000) 2024-03-06T10:38:30.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:30.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:30.284688+0000) 2024-03-06T10:38:31.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:31.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:31.284916+0000) 2024-03-06T10:38:32.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:32.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:32.285137+0000) 2024-03-06T10:38:33.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:33.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:33.285315+0000) 2024-03-06T10:38:34.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:34.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:34.285546+0000) 2024-03-06T10:38:35.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:35.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:35.285687+0000) 2024-03-06T10:38:36.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:36.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:36.285902+0000) 2024-03-06T10:38:37.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:37.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:37.286142+0000) 2024-03-06T10:38:38.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:38.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:38.286374+0000) 2024-03-06T10:38:39.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:39.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:39.286527+0000) 2024-03-06T10:38:40.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:40.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:40.286746+0000) 2024-03-06T10:38:41.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:41.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:41.286976+0000) 2024-03-06T10:38:42.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:42.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:42.287295+0000) 2024-03-06T10:38:43.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:43.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:43.287525+0000) 2024-03-06T10:38:44.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:44.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:44.287775+0000) 2024-03-06T10:38:45.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:45.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:45.287997+0000) 2024-03-06T10:38:46.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:46.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:46.288270+0000) 2024-03-06T10:38:47.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:47.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:47.288505+0000) 2024-03-06T10:38:48.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:48.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:48.288739+0000) 2024-03-06T10:38:49.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:49.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:49.288982+0000) 2024-03-06T10:38:50.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:50.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:50.289260+0000) 2024-03-06T10:38:51.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:51.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:51.289486+0000) 2024-03-06T10:38:52.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:52.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:52.289758+0000) 2024-03-06T10:38:53.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:53.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:53.290081+0000) 2024-03-06T10:38:54.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:54.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:54.290363+0000) 2024-03-06T10:38:55.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:55.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:55.290640+0000) 2024-03-06T10:38:56.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:56.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:56.290921+0000) 2024-03-06T10:38:56.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:56.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:38:57.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:57.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:57.291183+0000) 2024-03-06T10:38:58.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:58.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:58.291411+0000) 2024-03-06T10:38:59.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:38:59.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:38:59.291656+0000) 2024-03-06T10:39:00.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:00.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:00.291886+0000) 2024-03-06T10:39:01.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:01.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:01.292144+0000) 2024-03-06T10:39:02.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:02.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:02.292322+0000) 2024-03-06T10:39:03.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:03.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:03.292553+0000) 2024-03-06T10:39:04.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:04.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:04.292786+0000) 2024-03-06T10:39:05.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:05.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:05.293078+0000) 2024-03-06T10:39:06.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:06.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:06.293305+0000) 2024-03-06T10:39:07.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:07.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:07.293482+0000) 2024-03-06T10:39:08.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:08.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:08.293694+0000) 2024-03-06T10:39:09.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:09.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:09.293956+0000) 2024-03-06T10:39:10.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:10.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:10.294218+0000) 2024-03-06T10:39:11.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:11.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:11.294514+0000) 2024-03-06T10:39:12.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:12.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:12.294796+0000) 2024-03-06T10:39:13.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:13.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:13.295095+0000) 2024-03-06T10:39:14.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:14.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:14.295348+0000) 2024-03-06T10:39:15.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:15.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:15.295640+0000) 2024-03-06T10:39:16.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:16.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:16.295922+0000) 2024-03-06T10:39:17.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:17.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:17.296129+0000) 2024-03-06T10:39:18.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:18.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:18.296329+0000) 2024-03-06T10:39:19.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:19.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:19.296564+0000) 2024-03-06T10:39:20.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:20.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:20.296747+0000) 2024-03-06T10:39:21.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:21.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:21.296965+0000) 2024-03-06T10:39:22.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:22.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:22.297193+0000) 2024-03-06T10:39:23.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:23.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:23.297421+0000) 2024-03-06T10:39:24.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:24.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:24.297638+0000) 2024-03-06T10:39:25.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:25.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:25.297881+0000) 2024-03-06T10:39:26.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:26.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:26.298127+0000) 2024-03-06T10:39:26.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:26.893 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:27.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:27.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:27.298392+0000) 2024-03-06T10:39:28.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:28.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:28.298616+0000) 2024-03-06T10:39:29.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:29.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:29.298874+0000) 2024-03-06T10:39:30.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:30.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:30.299125+0000) 2024-03-06T10:39:31.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:31.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:31.299360+0000) 2024-03-06T10:39:32.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:32.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:32.299591+0000) 2024-03-06T10:39:33.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:33.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:33.299822+0000) 2024-03-06T10:39:34.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:34.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:34.300084+0000) 2024-03-06T10:39:35.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:35.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:35.300309+0000) 2024-03-06T10:39:36.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:36.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:36.300539+0000) 2024-03-06T10:39:37.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:37.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:37.300787+0000) 2024-03-06T10:39:38.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:38.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:38.301036+0000) 2024-03-06T10:39:39.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:39.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:39.301271+0000) 2024-03-06T10:39:40.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:40.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:40.301475+0000) 2024-03-06T10:39:41.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:41.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:41.301680+0000) 2024-03-06T10:39:42.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:42.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:42.301946+0000) 2024-03-06T10:39:43.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:43.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:43.302188+0000) 2024-03-06T10:39:44.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:44.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:44.302422+0000) 2024-03-06T10:39:45.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:45.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:45.302721+0000) 2024-03-06T10:39:46.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:46.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:46.302986+0000) 2024-03-06T10:39:47.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:47.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:47.303289+0000) 2024-03-06T10:39:48.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:48.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:48.303565+0000) 2024-03-06T10:39:49.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:49.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:49.303853+0000) 2024-03-06T10:39:50.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:50.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:50.304144+0000) 2024-03-06T10:39:51.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:51.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:51.304424+0000) 2024-03-06T10:39:52.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:52.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:52.304702+0000) 2024-03-06T10:39:53.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:53.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:53.304975+0000) 2024-03-06T10:39:54.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:54.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:54.305298+0000) 2024-03-06T10:39:55.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:55.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:55.305573+0000) 2024-03-06T10:39:56.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:56.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:56.305849+0000) 2024-03-06T10:39:57.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:57.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:39:57.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:57.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:57.306104+0000) 2024-03-06T10:39:58.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:58.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:58.306326+0000) 2024-03-06T10:39:59.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:39:59.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:39:59.306606+0000) 2024-03-06T10:40:00.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:00.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:00.306882+0000) 2024-03-06T10:40:01.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:01.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:01.307119+0000) 2024-03-06T10:40:02.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:02.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:02.307363+0000) 2024-03-06T10:40:03.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:03.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:03.307598+0000) 2024-03-06T10:40:04.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:04.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:04.307860+0000) 2024-03-06T10:40:05.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:05.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:05.308106+0000) 2024-03-06T10:40:06.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:06.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:06.308288+0000) 2024-03-06T10:40:07.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:07.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:07.308473+0000) 2024-03-06T10:40:08.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:08.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:08.308697+0000) 2024-03-06T10:40:09.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:09.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:09.308936+0000) 2024-03-06T10:40:10.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:10.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:10.309152+0000) 2024-03-06T10:40:11.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:11.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:11.309402+0000) 2024-03-06T10:40:12.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:12.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:12.309625+0000) 2024-03-06T10:40:13.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:13.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:13.309851+0000) 2024-03-06T10:40:14.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:14.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:14.310118+0000) 2024-03-06T10:40:15.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:15.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:15.310345+0000) 2024-03-06T10:40:16.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:16.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:16.310586+0000) 2024-03-06T10:40:17.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:17.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:17.310806+0000) 2024-03-06T10:40:18.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:18.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:18.311058+0000) 2024-03-06T10:40:19.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:19.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:19.311322+0000) 2024-03-06T10:40:20.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:20.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:20.311552+0000) 2024-03-06T10:40:21.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:21.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:21.311781+0000) 2024-03-06T10:40:22.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:22.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:22.312036+0000) 2024-03-06T10:40:23.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:23.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:23.312273+0000) 2024-03-06T10:40:24.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:24.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:24.312530+0000) 2024-03-06T10:40:25.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:25.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:25.312776+0000) 2024-03-06T10:40:26.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:26.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:26.312997+0000) 2024-03-06T10:40:27.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:27.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:27.313274+0000) 2024-03-06T10:40:27.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:27.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:28.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:28.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:28.313515+0000) 2024-03-06T10:40:29.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:29.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:29.313814+0000) 2024-03-06T10:40:30.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:30.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:30.314071+0000) 2024-03-06T10:40:31.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:31.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:31.314300+0000) 2024-03-06T10:40:32.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:32.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:32.314528+0000) 2024-03-06T10:40:33.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:33.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:33.314730+0000) 2024-03-06T10:40:34.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:34.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:34.314958+0000) 2024-03-06T10:40:35.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:35.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:35.315192+0000) 2024-03-06T10:40:36.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:36.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:36.315383+0000) 2024-03-06T10:40:37.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:37.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:37.315607+0000) 2024-03-06T10:40:38.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:38.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:38.315828+0000) 2024-03-06T10:40:39.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:39.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:39.316060+0000) 2024-03-06T10:40:40.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:40.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:40.316301+0000) 2024-03-06T10:40:41.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:41.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:41.316548+0000) 2024-03-06T10:40:42.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:42.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:42.316774+0000) 2024-03-06T10:40:43.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:43.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:43.317050+0000) 2024-03-06T10:40:44.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:44.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:44.317294+0000) 2024-03-06T10:40:45.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:45.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:45.317541+0000) 2024-03-06T10:40:46.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:46.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:46.317776+0000) 2024-03-06T10:40:47.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:47.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:47.318055+0000) 2024-03-06T10:40:48.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:48.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:48.318308+0000) 2024-03-06T10:40:49.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:49.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:49.318566+0000) 2024-03-06T10:40:50.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:50.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:50.318812+0000) 2024-03-06T10:40:51.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:51.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:51.319085+0000) 2024-03-06T10:40:52.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:52.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:52.319296+0000) 2024-03-06T10:40:53.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:53.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:53.319522+0000) 2024-03-06T10:40:54.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:54.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:54.319725+0000) 2024-03-06T10:40:55.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:55.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:55.319961+0000) 2024-03-06T10:40:56.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:56.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:56.320185+0000) 2024-03-06T10:40:57.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:57.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:57.320415+0000) 2024-03-06T10:40:57.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:57.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:40:58.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:58.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:58.320633+0000) 2024-03-06T10:40:59.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:40:59.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:40:59.320800+0000) 2024-03-06T10:41:00.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:00.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:00.321064+0000) 2024-03-06T10:41:01.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:01.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:01.321269+0000) 2024-03-06T10:41:02.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:02.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:02.321498+0000) 2024-03-06T10:41:03.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:03.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:03.321683+0000) 2024-03-06T10:41:04.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:04.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:04.321917+0000) 2024-03-06T10:41:05.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:05.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:05.322110+0000) 2024-03-06T10:41:06.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:06.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:06.322299+0000) 2024-03-06T10:41:07.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:07.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:07.322524+0000) 2024-03-06T10:41:08.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:08.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:08.322749+0000) 2024-03-06T10:41:09.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:09.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:09.322969+0000) 2024-03-06T10:41:10.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:10.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:10.323208+0000) 2024-03-06T10:41:11.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:11.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:11.323446+0000) 2024-03-06T10:41:12.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:12.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:12.323690+0000) 2024-03-06T10:41:13.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:13.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:13.323968+0000) 2024-03-06T10:41:14.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:14.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:14.324252+0000) 2024-03-06T10:41:15.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:15.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:15.324503+0000) 2024-03-06T10:41:16.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:16.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:16.324754+0000) 2024-03-06T10:41:17.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:17.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:17.325065+0000) 2024-03-06T10:41:18.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:18.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:18.325309+0000) 2024-03-06T10:41:19.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:19.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:19.325564+0000) 2024-03-06T10:41:20.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:20.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:20.325805+0000) 2024-03-06T10:41:21.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:21.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:21.326071+0000) 2024-03-06T10:41:22.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:22.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:22.326248+0000) 2024-03-06T10:41:23.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:23.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:23.326494+0000) 2024-03-06T10:41:24.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:24.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:24.326745+0000) 2024-03-06T10:41:25.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:25.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:25.326979+0000) 2024-03-06T10:41:26.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:26.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:26.327269+0000) 2024-03-06T10:41:27.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:27.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:27.327501+0000) 2024-03-06T10:41:27.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:27.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:28.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:28.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:28.327742+0000) 2024-03-06T10:41:29.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:29.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:29.328053+0000) 2024-03-06T10:41:30.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:30.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:30.328273+0000) 2024-03-06T10:41:31.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:31.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:31.328488+0000) 2024-03-06T10:41:32.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:32.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:32.328744+0000) 2024-03-06T10:41:33.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:33.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:33.328969+0000) 2024-03-06T10:41:34.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:34.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:34.329193+0000) 2024-03-06T10:41:35.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:35.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:35.329421+0000) 2024-03-06T10:41:36.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:36.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:36.329636+0000) 2024-03-06T10:41:37.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:37.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:37.329854+0000) 2024-03-06T10:41:38.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:38.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:38.330147+0000) 2024-03-06T10:41:39.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:39.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:39.330408+0000) 2024-03-06T10:41:40.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:40.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:40.330664+0000) 2024-03-06T10:41:41.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:41.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:41.330923+0000) 2024-03-06T10:41:42.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:42.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:42.331158+0000) 2024-03-06T10:41:43.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:43.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:43.331386+0000) 2024-03-06T10:41:44.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:44.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:44.331624+0000) 2024-03-06T10:41:45.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:45.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:45.331866+0000) 2024-03-06T10:41:46.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:46.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:46.332129+0000) 2024-03-06T10:41:47.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:47.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:47.332375+0000) 2024-03-06T10:41:48.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:48.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:48.332586+0000) 2024-03-06T10:41:49.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:49.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:49.332765+0000) 2024-03-06T10:41:50.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:50.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:50.332952+0000) 2024-03-06T10:41:51.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:51.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:51.333191+0000) 2024-03-06T10:41:52.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:52.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:52.333388+0000) 2024-03-06T10:41:53.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:53.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:53.333609+0000) 2024-03-06T10:41:54.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:54.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:54.333837+0000) 2024-03-06T10:41:55.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:55.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:55.334112+0000) 2024-03-06T10:41:56.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:56.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:56.334349+0000) 2024-03-06T10:41:57.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:57.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:57.334587+0000) 2024-03-06T10:41:57.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:57.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:41:58.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:58.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:58.334823+0000) 2024-03-06T10:41:59.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:41:59.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:41:59.335147+0000) 2024-03-06T10:42:00.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:00.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:00.335378+0000) 2024-03-06T10:42:01.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:01.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:01.335586+0000) 2024-03-06T10:42:02.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:02.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:02.335808+0000) 2024-03-06T10:42:03.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:03.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:03.336086+0000) 2024-03-06T10:42:04.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:04.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:04.336275+0000) 2024-03-06T10:42:05.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:05.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:05.336520+0000) 2024-03-06T10:42:06.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:06.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:06.336742+0000) 2024-03-06T10:42:07.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:07.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:07.336989+0000) 2024-03-06T10:42:08.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:08.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:08.337268+0000) 2024-03-06T10:42:09.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:09.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:09.337491+0000) 2024-03-06T10:42:10.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:10.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:10.337687+0000) 2024-03-06T10:42:11.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:11.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:11.337930+0000) 2024-03-06T10:42:12.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:12.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:12.338165+0000) 2024-03-06T10:42:13.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:13.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:13.338411+0000) 2024-03-06T10:42:14.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:14.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:14.338650+0000) 2024-03-06T10:42:15.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:15.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:15.338902+0000) 2024-03-06T10:42:16.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:16.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:16.339144+0000) 2024-03-06T10:42:17.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:17.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:17.339389+0000) 2024-03-06T10:42:18.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:18.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:18.339629+0000) 2024-03-06T10:42:19.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:19.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:19.339880+0000) 2024-03-06T10:42:20.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:20.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:20.340113+0000) 2024-03-06T10:42:21.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:21.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:21.340365+0000) 2024-03-06T10:42:22.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:22.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:22.340611+0000) 2024-03-06T10:42:23.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:23.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:23.340856+0000) 2024-03-06T10:42:24.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:24.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:24.341115+0000) 2024-03-06T10:42:25.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:25.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:25.341393+0000) 2024-03-06T10:42:26.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:26.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:26.341634+0000) 2024-03-06T10:42:27.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:27.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:27.341878+0000) 2024-03-06T10:42:28.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:28.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:28.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:28.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:28.342131+0000) 2024-03-06T10:42:29.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:29.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:29.342382+0000) 2024-03-06T10:42:30.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:30.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:30.342622+0000) 2024-03-06T10:42:31.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:31.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:31.342874+0000) 2024-03-06T10:42:32.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:32.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:32.343119+0000) 2024-03-06T10:42:33.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:33.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:33.343365+0000) 2024-03-06T10:42:34.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:34.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:34.343610+0000) 2024-03-06T10:42:35.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:35.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:35.343878+0000) 2024-03-06T10:42:36.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:36.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:36.344129+0000) 2024-03-06T10:42:37.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:37.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:37.344346+0000) 2024-03-06T10:42:38.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:38.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:38.344584+0000) 2024-03-06T10:42:39.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:39.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:39.344817+0000) 2024-03-06T10:42:40.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:40.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:40.345094+0000) 2024-03-06T10:42:41.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:41.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:41.345371+0000) 2024-03-06T10:42:42.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:42.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:42.345610+0000) 2024-03-06T10:42:43.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:43.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:43.345878+0000) 2024-03-06T10:42:44.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:44.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:44.346168+0000) 2024-03-06T10:42:45.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:45.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:45.346401+0000) 2024-03-06T10:42:46.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:46.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:46.346619+0000) 2024-03-06T10:42:47.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:47.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:47.346840+0000) 2024-03-06T10:42:48.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:48.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:48.347099+0000) 2024-03-06T10:42:49.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:49.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:49.347322+0000) 2024-03-06T10:42:50.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:50.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:50.347541+0000) 2024-03-06T10:42:51.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:51.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:51.347745+0000) 2024-03-06T10:42:52.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:52.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:52.347972+0000) 2024-03-06T10:42:53.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:53.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:53.348246+0000) 2024-03-06T10:42:54.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:54.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:54.348519+0000) 2024-03-06T10:42:55.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:55.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:55.348800+0000) 2024-03-06T10:42:56.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:56.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:56.349116+0000) 2024-03-06T10:42:57.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:57.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:57.349368+0000) 2024-03-06T10:42:58.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:58.281 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:42:58.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:58.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:58.349587+0000) 2024-03-06T10:42:59.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:42:59.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:42:59.349854+0000) 2024-03-06T10:43:00.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:00.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:00.350052+0000) 2024-03-06T10:43:01.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:01.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:01.350283+0000) 2024-03-06T10:43:02.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:02.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:02.350500+0000) 2024-03-06T10:43:03.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:03.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:03.350766+0000) 2024-03-06T10:43:04.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:04.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:04.350983+0000) 2024-03-06T10:43:05.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:05.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:05.351202+0000) 2024-03-06T10:43:06.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:06.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:06.351423+0000) 2024-03-06T10:43:07.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:07.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:07.351655+0000) 2024-03-06T10:43:08.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:08.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:08.351886+0000) 2024-03-06T10:43:09.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:09.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:09.352132+0000) 2024-03-06T10:43:10.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:10.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:10.352321+0000) 2024-03-06T10:43:11.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:11.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:11.352545+0000) 2024-03-06T10:43:12.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:12.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:12.352725+0000) 2024-03-06T10:43:13.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:13.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:13.352946+0000) 2024-03-06T10:43:14.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:14.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:14.353176+0000) 2024-03-06T10:43:15.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:15.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:15.353362+0000) 2024-03-06T10:43:16.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:16.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:16.353553+0000) 2024-03-06T10:43:17.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:17.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:17.353776+0000) 2024-03-06T10:43:18.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:18.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:18.353951+0000) 2024-03-06T10:43:19.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:19.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:19.354158+0000) 2024-03-06T10:43:20.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:20.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:20.354387+0000) 2024-03-06T10:43:21.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:21.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:21.354605+0000) 2024-03-06T10:43:22.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:22.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:22.354843+0000) 2024-03-06T10:43:23.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:23.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:23.355109+0000) 2024-03-06T10:43:24.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:24.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:24.355365+0000) 2024-03-06T10:43:25.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:25.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:25.355567+0000) 2024-03-06T10:43:26.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:26.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:26.355749+0000) 2024-03-06T10:43:27.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:27.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:27.355974+0000) 2024-03-06T10:43:28.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:28.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:28.356196+0000) 2024-03-06T10:43:28.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:28.509 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:29.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:29.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:29.356428+0000) 2024-03-06T10:43:30.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:30.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:30.356673+0000) 2024-03-06T10:43:31.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:31.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:31.356909+0000) 2024-03-06T10:43:32.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:32.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:32.357152+0000) 2024-03-06T10:43:33.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:33.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:33.357387+0000) 2024-03-06T10:43:34.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:34.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:34.357584+0000) 2024-03-06T10:43:35.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:35.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:35.357785+0000) 2024-03-06T10:43:36.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:36.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:36.358038+0000) 2024-03-06T10:43:37.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:37.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:37.358267+0000) 2024-03-06T10:43:38.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:38.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:38.358469+0000) 2024-03-06T10:43:39.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:39.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:39.358692+0000) 2024-03-06T10:43:40.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:40.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:40.358913+0000) 2024-03-06T10:43:41.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:41.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:41.359170+0000) 2024-03-06T10:43:42.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:42.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:42.359392+0000) 2024-03-06T10:43:43.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:43.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:43.359621+0000) 2024-03-06T10:43:44.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:44.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:44.359834+0000) 2024-03-06T10:43:45.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:45.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:45.360114+0000) 2024-03-06T10:43:46.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:46.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:46.360387+0000) 2024-03-06T10:43:47.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:47.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:47.360600+0000) 2024-03-06T10:43:48.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:48.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:48.360806+0000) 2024-03-06T10:43:49.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:49.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:49.361131+0000) 2024-03-06T10:43:50.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:50.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:50.361401+0000) 2024-03-06T10:43:51.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:51.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:51.361607+0000) 2024-03-06T10:43:52.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:52.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:52.361834+0000) 2024-03-06T10:43:53.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:53.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:53.362099+0000) 2024-03-06T10:43:54.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:54.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:54.362312+0000) 2024-03-06T10:43:55.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:55.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:55.362569+0000) 2024-03-06T10:43:56.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:56.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:56.362769+0000) 2024-03-06T10:43:57.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:57.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:57.362999+0000) 2024-03-06T10:43:58.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:58.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:58.363256+0000) 2024-03-06T10:43:58.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:58.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:43:59.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:43:59.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:43:59.363486+0000) 2024-03-06T10:44:00.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:00.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:00.363673+0000) 2024-03-06T10:44:01.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:01.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:01.363916+0000) 2024-03-06T10:44:02.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:02.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:02.364136+0000) 2024-03-06T10:44:03.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:03.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:03.364368+0000) 2024-03-06T10:44:04.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:04.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:04.364587+0000) 2024-03-06T10:44:05.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:05.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:05.364819+0000) 2024-03-06T10:44:06.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:06.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:06.365079+0000) 2024-03-06T10:44:07.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:07.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:07.365305+0000) 2024-03-06T10:44:08.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:08.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:08.365516+0000) 2024-03-06T10:44:09.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:09.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:09.365743+0000) 2024-03-06T10:44:10.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:10.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:10.365968+0000) 2024-03-06T10:44:11.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:11.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:11.366225+0000) 2024-03-06T10:44:12.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:12.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:12.366433+0000) 2024-03-06T10:44:13.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:13.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:13.366672+0000) 2024-03-06T10:44:14.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:14.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:14.366859+0000) 2024-03-06T10:44:15.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:15.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:15.367123+0000) 2024-03-06T10:44:16.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:16.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:16.367338+0000) 2024-03-06T10:44:17.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:17.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:17.367547+0000) 2024-03-06T10:44:18.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:18.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:18.367771+0000) 2024-03-06T10:44:19.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:19.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:19.367996+0000) 2024-03-06T10:44:20.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:20.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:20.368230+0000) 2024-03-06T10:44:21.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:21.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:21.368417+0000) 2024-03-06T10:44:22.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:22.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:22.368633+0000) 2024-03-06T10:44:23.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:23.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:23.368875+0000) 2024-03-06T10:44:24.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:24.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:24.369133+0000) 2024-03-06T10:44:25.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:25.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:25.369372+0000) 2024-03-06T10:44:26.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:26.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:26.369593+0000) 2024-03-06T10:44:27.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:27.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:27.369789+0000) 2024-03-06T10:44:28.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:28.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:28.370024+0000) 2024-03-06T10:44:28.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:28.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:29.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:29.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:29.370241+0000) 2024-03-06T10:44:30.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:30.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:30.370466+0000) 2024-03-06T10:44:31.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:31.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:31.370717+0000) 2024-03-06T10:44:32.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:32.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:32.370953+0000) 2024-03-06T10:44:33.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:33.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:33.371225+0000) 2024-03-06T10:44:34.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:34.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:34.371418+0000) 2024-03-06T10:44:35.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:35.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:35.371646+0000) 2024-03-06T10:44:36.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:36.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:36.371862+0000) 2024-03-06T10:44:37.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:37.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:37.372123+0000) 2024-03-06T10:44:38.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:38.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:38.372345+0000) 2024-03-06T10:44:39.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:39.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:39.372617+0000) 2024-03-06T10:44:40.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:40.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:40.372845+0000) 2024-03-06T10:44:41.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:41.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:41.373142+0000) 2024-03-06T10:44:42.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:42.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:42.373342+0000) 2024-03-06T10:44:43.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:43.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:43.373571+0000) 2024-03-06T10:44:44.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:44.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:44.373794+0000) 2024-03-06T10:44:45.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:45.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:45.374089+0000) 2024-03-06T10:44:46.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:46.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:46.374321+0000) 2024-03-06T10:44:47.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:47.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:47.374547+0000) 2024-03-06T10:44:48.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:48.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:48.374758+0000) 2024-03-06T10:44:49.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:49.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:49.374957+0000) 2024-03-06T10:44:50.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:50.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:50.375160+0000) 2024-03-06T10:44:51.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:51.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:51.375391+0000) 2024-03-06T10:44:52.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:52.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:52.375614+0000) 2024-03-06T10:44:53.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:53.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:53.375864+0000) 2024-03-06T10:44:54.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:54.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:54.376139+0000) 2024-03-06T10:44:55.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:55.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:55.376384+0000) 2024-03-06T10:44:56.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:56.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:56.376605+0000) 2024-03-06T10:44:57.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:57.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:57.376827+0000) 2024-03-06T10:44:58.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:58.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:58.377086+0000) 2024-03-06T10:44:59.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:59.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:44:59.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:44:59.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:44:59.377301+0000) 2024-03-06T10:45:00.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:00.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:00.377521+0000) 2024-03-06T10:45:01.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:01.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:01.377758+0000) 2024-03-06T10:45:02.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:02.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:02.377995+0000) 2024-03-06T10:45:03.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:03.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:03.378253+0000) 2024-03-06T10:45:04.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:04.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:04.378456+0000) 2024-03-06T10:45:05.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:05.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:05.378689+0000) 2024-03-06T10:45:06.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:06.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:06.378911+0000) 2024-03-06T10:45:07.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:07.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:07.379144+0000) 2024-03-06T10:45:08.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:08.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:08.379367+0000) 2024-03-06T10:45:09.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:09.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:09.379595+0000) 2024-03-06T10:45:10.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:10.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:10.379813+0000) 2024-03-06T10:45:11.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:11.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:11.379949+0000) 2024-03-06T10:45:12.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:12.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:12.380160+0000) 2024-03-06T10:45:13.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:13.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:13.380357+0000) 2024-03-06T10:45:14.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:14.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:14.380578+0000) 2024-03-06T10:45:15.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:15.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:15.380843+0000) 2024-03-06T10:45:16.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:16.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:16.381042+0000) 2024-03-06T10:45:17.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:17.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:17.381283+0000) 2024-03-06T10:45:18.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:18.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:18.381507+0000) 2024-03-06T10:45:19.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:19.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:19.381708+0000) 2024-03-06T10:45:20.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:20.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:20.381948+0000) 2024-03-06T10:45:21.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:21.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:21.382195+0000) 2024-03-06T10:45:22.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:22.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:22.382445+0000) 2024-03-06T10:45:23.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:23.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:23.382689+0000) 2024-03-06T10:45:24.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:24.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:24.382915+0000) 2024-03-06T10:45:25.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:25.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:25.383163+0000) 2024-03-06T10:45:26.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:26.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:26.383382+0000) 2024-03-06T10:45:27.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:27.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:27.383651+0000) 2024-03-06T10:45:28.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:28.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:28.383870+0000) 2024-03-06T10:45:29.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:29.314 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:29.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:29.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:29.384080+0000) 2024-03-06T10:45:30.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:30.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:30.384281+0000) 2024-03-06T10:45:31.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:31.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:31.384516+0000) 2024-03-06T10:45:32.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:32.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:32.384734+0000) 2024-03-06T10:45:33.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:33.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:33.384941+0000) 2024-03-06T10:45:34.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:34.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:34.385162+0000) 2024-03-06T10:45:35.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:35.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:35.385401+0000) 2024-03-06T10:45:36.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:36.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:36.385596+0000) 2024-03-06T10:45:37.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:37.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:37.385820+0000) 2024-03-06T10:45:38.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:38.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:38.386077+0000) 2024-03-06T10:45:39.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:39.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:39.386276+0000) 2024-03-06T10:45:40.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:40.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:40.386469+0000) 2024-03-06T10:45:41.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:41.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:41.386692+0000) 2024-03-06T10:45:42.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:42.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:42.386862+0000) 2024-03-06T10:45:43.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:43.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:43.387115+0000) 2024-03-06T10:45:44.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:44.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:44.387338+0000) 2024-03-06T10:45:45.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:45.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:45.387570+0000) 2024-03-06T10:45:46.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:46.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:46.387748+0000) 2024-03-06T10:45:47.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:47.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:47.387972+0000) 2024-03-06T10:45:48.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:48.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:48.388199+0000) 2024-03-06T10:45:49.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:49.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:49.388395+0000) 2024-03-06T10:45:50.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:50.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:50.388596+0000) 2024-03-06T10:45:51.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:51.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:51.388834+0000) 2024-03-06T10:45:52.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:52.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:52.389052+0000) 2024-03-06T10:45:53.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:53.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:53.389277+0000) 2024-03-06T10:45:54.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:54.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:54.389502+0000) 2024-03-06T10:45:55.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:55.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:55.389691+0000) 2024-03-06T10:45:56.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:56.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:56.389910+0000) 2024-03-06T10:45:57.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:57.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:57.390162+0000) 2024-03-06T10:45:58.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:58.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:58.390399+0000) 2024-03-06T10:45:59.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:45:59.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:45:59.390690+0000) 2024-03-06T10:45:59.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:45:59.530 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:00.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:00.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:00.390930+0000) 2024-03-06T10:46:01.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:01.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:01.391160+0000) 2024-03-06T10:46:02.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:02.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:02.391390+0000) 2024-03-06T10:46:03.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:03.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:03.391618+0000) 2024-03-06T10:46:04.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:04.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:04.391847+0000) 2024-03-06T10:46:05.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:05.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:05.392171+0000) 2024-03-06T10:46:06.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:06.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:06.392454+0000) 2024-03-06T10:46:07.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:07.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:07.392712+0000) 2024-03-06T10:46:08.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:08.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:08.392953+0000) 2024-03-06T10:46:09.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:09.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:09.393252+0000) 2024-03-06T10:46:10.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:10.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:10.393497+0000) 2024-03-06T10:46:11.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:11.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:11.393735+0000) 2024-03-06T10:46:12.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:12.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:12.393976+0000) 2024-03-06T10:46:13.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:13.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:13.394241+0000) 2024-03-06T10:46:14.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:14.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:14.394456+0000) 2024-03-06T10:46:15.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:15.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:15.394727+0000) 2024-03-06T10:46:16.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:16.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:16.394949+0000) 2024-03-06T10:46:17.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:17.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:17.395172+0000) 2024-03-06T10:46:18.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:18.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:18.395391+0000) 2024-03-06T10:46:19.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:19.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:19.395623+0000) 2024-03-06T10:46:20.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:20.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:20.395852+0000) 2024-03-06T10:46:21.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:21.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:21.396107+0000) 2024-03-06T10:46:22.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:22.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:22.396327+0000) 2024-03-06T10:46:23.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:23.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:23.396516+0000) 2024-03-06T10:46:24.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:24.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:24.396759+0000) 2024-03-06T10:46:25.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:25.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:25.396974+0000) 2024-03-06T10:46:26.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:26.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:26.397201+0000) 2024-03-06T10:46:27.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:27.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:27.397383+0000) 2024-03-06T10:46:28.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:28.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:28.397608+0000) 2024-03-06T10:46:29.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:29.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:29.397831+0000) 2024-03-06T10:46:29.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:29.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:30.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:30.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:30.398102+0000) 2024-03-06T10:46:31.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:31.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:31.398344+0000) 2024-03-06T10:46:32.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:32.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:32.398556+0000) 2024-03-06T10:46:33.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:33.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:33.398779+0000) 2024-03-06T10:46:34.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:34.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:34.398991+0000) 2024-03-06T10:46:35.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:35.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:35.399271+0000) 2024-03-06T10:46:36.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:36.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:36.399475+0000) 2024-03-06T10:46:37.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:37.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:37.399731+0000) 2024-03-06T10:46:38.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:38.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:38.399924+0000) 2024-03-06T10:46:39.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:39.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:39.400160+0000) 2024-03-06T10:46:40.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:40.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:40.400391+0000) 2024-03-06T10:46:41.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:41.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:41.400628+0000) 2024-03-06T10:46:42.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:42.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:42.400853+0000) 2024-03-06T10:46:43.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:43.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:43.401114+0000) 2024-03-06T10:46:44.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:44.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:44.401302+0000) 2024-03-06T10:46:45.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:45.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:45.401538+0000) 2024-03-06T10:46:46.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:46.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:46.401757+0000) 2024-03-06T10:46:47.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:47.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:47.401989+0000) 2024-03-06T10:46:48.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:48.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:48.402265+0000) 2024-03-06T10:46:49.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:49.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:49.402498+0000) 2024-03-06T10:46:50.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:50.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:50.402723+0000) 2024-03-06T10:46:51.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:51.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:51.402956+0000) 2024-03-06T10:46:52.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:52.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:52.403179+0000) 2024-03-06T10:46:53.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:53.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:53.403441+0000) 2024-03-06T10:46:54.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:54.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:54.403656+0000) 2024-03-06T10:46:55.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:55.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:55.403892+0000) 2024-03-06T10:46:56.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:56.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:56.404062+0000) 2024-03-06T10:46:57.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:57.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:57.404279+0000) 2024-03-06T10:46:58.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:58.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:58.404494+0000) 2024-03-06T10:46:59.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:46:59.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:46:59.404720+0000) 2024-03-06T10:46:59.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:46:59.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:00.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:00.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:00.404944+0000) 2024-03-06T10:47:01.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:01.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:01.405171+0000) 2024-03-06T10:47:02.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:02.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:02.405417+0000) 2024-03-06T10:47:03.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:03.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:03.405620+0000) 2024-03-06T10:47:04.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:04.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:04.405835+0000) 2024-03-06T10:47:05.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:05.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:05.406056+0000) 2024-03-06T10:47:06.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:06.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:06.406292+0000) 2024-03-06T10:47:07.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:07.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:07.406542+0000) 2024-03-06T10:47:08.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:08.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:08.406753+0000) 2024-03-06T10:47:09.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:09.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:09.406934+0000) 2024-03-06T10:47:10.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:10.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:10.407133+0000) 2024-03-06T10:47:11.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:11.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:11.407377+0000) 2024-03-06T10:47:12.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:12.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:12.407599+0000) 2024-03-06T10:47:13.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:13.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:13.407776+0000) 2024-03-06T10:47:14.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:14.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:14.407996+0000) 2024-03-06T10:47:15.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:15.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:15.408223+0000) 2024-03-06T10:47:16.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:16.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:16.408442+0000) 2024-03-06T10:47:17.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:17.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:17.408662+0000) 2024-03-06T10:47:18.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:18.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:18.408895+0000) 2024-03-06T10:47:19.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:19.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:19.409122+0000) 2024-03-06T10:47:20.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:20.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:20.409311+0000) 2024-03-06T10:47:21.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:21.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:21.409582+0000) 2024-03-06T10:47:22.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:22.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:22.409828+0000) 2024-03-06T10:47:23.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:23.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:23.410067+0000) 2024-03-06T10:47:24.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:24.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:24.410298+0000) 2024-03-06T10:47:25.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:25.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:25.410527+0000) 2024-03-06T10:47:26.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:26.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:26.410753+0000) 2024-03-06T10:47:27.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:27.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:27.410983+0000) 2024-03-06T10:47:28.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:28.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:28.411248+0000) 2024-03-06T10:47:29.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:29.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:29.411456+0000) 2024-03-06T10:47:30.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:30.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:47:30.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:30.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:30.411671+0000) 2024-03-06T10:47:31.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:31.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:31.411947+0000) 2024-03-06T10:47:32.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:32.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:32.412193+0000) 2024-03-06T10:47:33.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:33.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:33.412475+0000) 2024-03-06T10:47:34.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:34.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:34.412758+0000) 2024-03-06T10:47:35.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:35.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:35.413066+0000) 2024-03-06T10:47:36.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:36.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:36.413308+0000) 2024-03-06T10:47:37.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:37.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:37.413588+0000) 2024-03-06T10:47:38.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:38.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:38.413868+0000) 2024-03-06T10:47:39.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:39.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:39.414155+0000) 2024-03-06T10:47:40.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:40.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:40.414290+0000) 2024-03-06T10:47:41.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:41.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:41.414503+0000) 2024-03-06T10:47:42.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:42.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:42.414676+0000) 2024-03-06T10:47:43.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:43.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:43.414897+0000) 2024-03-06T10:47:44.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:44.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:44.415115+0000) 2024-03-06T10:47:45.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:45.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:45.415292+0000) 2024-03-06T10:47:46.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:46.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:46.415499+0000) 2024-03-06T10:47:47.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:47.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:47.415769+0000) 2024-03-06T10:47:48.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:48.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:48.416079+0000) 2024-03-06T10:47:49.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:49.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:49.416370+0000) 2024-03-06T10:47:50.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:50.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:50.416629+0000) 2024-03-06T10:47:51.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:51.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:51.416907+0000) 2024-03-06T10:47:52.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:52.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:52.417188+0000) 2024-03-06T10:47:53.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:53.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:53.417467+0000) 2024-03-06T10:47:54.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:54.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:54.417739+0000) 2024-03-06T10:47:55.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:55.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:55.418055+0000) 2024-03-06T10:47:56.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:56.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:56.418291+0000) 2024-03-06T10:47:57.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:57.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:57.418623+0000) 2024-03-06T10:47:58.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:58.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:58.418851+0000) 2024-03-06T10:47:59.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:47:59.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:47:59.419167+0000) 2024-03-06T10:48:00.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:00.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:00.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:00.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:00.419367+0000) 2024-03-06T10:48:01.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:01.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:01.419568+0000) 2024-03-06T10:48:02.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:02.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:02.419793+0000) 2024-03-06T10:48:03.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:03.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:03.419978+0000) 2024-03-06T10:48:04.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:04.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:04.420257+0000) 2024-03-06T10:48:05.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:05.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:05.420443+0000) 2024-03-06T10:48:06.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:06.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:06.420595+0000) 2024-03-06T10:48:07.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:07.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:07.420810+0000) 2024-03-06T10:48:08.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:08.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:08.421084+0000) 2024-03-06T10:48:09.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:09.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:09.421308+0000) 2024-03-06T10:48:10.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:10.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:10.421495+0000) 2024-03-06T10:48:11.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:11.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:11.421731+0000) 2024-03-06T10:48:12.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:12.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:12.421909+0000) 2024-03-06T10:48:13.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:13.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:13.422138+0000) 2024-03-06T10:48:14.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:14.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:14.422315+0000) 2024-03-06T10:48:15.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:15.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:15.422537+0000) 2024-03-06T10:48:16.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:16.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:16.422738+0000) 2024-03-06T10:48:17.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:17.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:17.422962+0000) 2024-03-06T10:48:18.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:18.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:18.423190+0000) 2024-03-06T10:48:19.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:19.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:19.423371+0000) 2024-03-06T10:48:20.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:20.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:20.423582+0000) 2024-03-06T10:48:21.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:21.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:21.423816+0000) 2024-03-06T10:48:22.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:22.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:22.424079+0000) 2024-03-06T10:48:23.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:23.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:23.424309+0000) 2024-03-06T10:48:24.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:24.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:24.424527+0000) 2024-03-06T10:48:25.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:25.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:25.424663+0000) 2024-03-06T10:48:26.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:26.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:26.424874+0000) 2024-03-06T10:48:27.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:27.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:27.425093+0000) 2024-03-06T10:48:28.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:28.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:28.425325+0000) 2024-03-06T10:48:29.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:29.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:29.425557+0000) 2024-03-06T10:48:30.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:30.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:30.425795+0000) 2024-03-06T10:48:30.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:30.625 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:48:31.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:31.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:31.426113+0000) 2024-03-06T10:48:32.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:32.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:32.426362+0000) 2024-03-06T10:48:33.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:33.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:33.426645+0000) 2024-03-06T10:48:34.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:34.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:34.426911+0000) 2024-03-06T10:48:35.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:35.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:35.427163+0000) 2024-03-06T10:48:36.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:36.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:36.427379+0000) 2024-03-06T10:48:37.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:37.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:37.427626+0000) 2024-03-06T10:48:38.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:38.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:38.427901+0000) 2024-03-06T10:48:39.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:39.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:39.428186+0000) 2024-03-06T10:48:40.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:40.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:40.428457+0000) 2024-03-06T10:48:41.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:41.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:41.428748+0000) 2024-03-06T10:48:42.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:42.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:42.429055+0000) 2024-03-06T10:48:43.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:43.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:43.429340+0000) 2024-03-06T10:48:44.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:44.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:44.429619+0000) 2024-03-06T10:48:45.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:45.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:45.429947+0000) 2024-03-06T10:48:46.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:46.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:46.430227+0000) 2024-03-06T10:48:47.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:47.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:47.430504+0000) 2024-03-06T10:48:48.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:48.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:48.430740+0000) 2024-03-06T10:48:49.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:49.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:49.430964+0000) 2024-03-06T10:48:50.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:50.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:50.431191+0000) 2024-03-06T10:48:51.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:51.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:51.431461+0000) 2024-03-06T10:48:52.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:52.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:52.431722+0000) 2024-03-06T10:48:53.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:53.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:53.431982+0000) 2024-03-06T10:48:54.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:54.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:54.432225+0000) 2024-03-06T10:48:55.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:55.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:55.432425+0000) 2024-03-06T10:48:56.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:56.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:56.432639+0000) 2024-03-06T10:48:57.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:57.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:57.432858+0000) 2024-03-06T10:48:58.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:58.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:58.433127+0000) 2024-03-06T10:48:59.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:48:59.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:48:59.433352+0000) 2024-03-06T10:49:00.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:00.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:00.433569+0000) 2024-03-06T10:49:00.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:00.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:01.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:01.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:01.433804+0000) 2024-03-06T10:49:02.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:02.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:02.433989+0000) 2024-03-06T10:49:03.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:03.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:03.434263+0000) 2024-03-06T10:49:04.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:04.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:04.434521+0000) 2024-03-06T10:49:05.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:05.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:05.434796+0000) 2024-03-06T10:49:06.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:06.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:06.435109+0000) 2024-03-06T10:49:07.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:07.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:07.435376+0000) 2024-03-06T10:49:08.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:08.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:08.435652+0000) 2024-03-06T10:49:09.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:09.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:09.435930+0000) 2024-03-06T10:49:10.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:10.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:10.436211+0000) 2024-03-06T10:49:11.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:11.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:11.436487+0000) 2024-03-06T10:49:12.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:12.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:12.436751+0000) 2024-03-06T10:49:13.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:13.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:13.437046+0000) 2024-03-06T10:49:14.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:14.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:14.437298+0000) 2024-03-06T10:49:15.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:15.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:15.437513+0000) 2024-03-06T10:49:16.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:16.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:16.437688+0000) 2024-03-06T10:49:17.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:17.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:17.437948+0000) 2024-03-06T10:49:18.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:18.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:18.438183+0000) 2024-03-06T10:49:19.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:19.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:19.438385+0000) 2024-03-06T10:49:20.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:20.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:20.438603+0000) 2024-03-06T10:49:21.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:21.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:21.438841+0000) 2024-03-06T10:49:22.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:22.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:22.439108+0000) 2024-03-06T10:49:23.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:23.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:23.439355+0000) 2024-03-06T10:49:24.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:24.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:24.439595+0000) 2024-03-06T10:49:25.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:25.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:25.439829+0000) 2024-03-06T10:49:26.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:26.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:26.440109+0000) 2024-03-06T10:49:27.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:27.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:27.440342+0000) 2024-03-06T10:49:28.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:28.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:28.440579+0000) 2024-03-06T10:49:29.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:29.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:29.440789+0000) 2024-03-06T10:49:30.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:30.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:30.440971+0000) 2024-03-06T10:49:31.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:31.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:49:31.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:31.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:31.441221+0000) 2024-03-06T10:49:32.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:32.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:32.441442+0000) 2024-03-06T10:49:33.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:33.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:33.441654+0000) 2024-03-06T10:49:34.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:34.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:34.441876+0000) 2024-03-06T10:49:35.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:35.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:35.442114+0000) 2024-03-06T10:49:36.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:36.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:36.442322+0000) 2024-03-06T10:49:37.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:37.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:37.442540+0000) 2024-03-06T10:49:38.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:38.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:38.442807+0000) 2024-03-06T10:49:39.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:39.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:39.443064+0000) 2024-03-06T10:49:40.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:40.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:40.443271+0000) 2024-03-06T10:49:41.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:41.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:41.443496+0000) 2024-03-06T10:49:42.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:42.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:42.443715+0000) 2024-03-06T10:49:43.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:43.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:43.443942+0000) 2024-03-06T10:49:44.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:44.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:44.444163+0000) 2024-03-06T10:49:45.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:45.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:45.444335+0000) 2024-03-06T10:49:46.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:46.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:46.444550+0000) 2024-03-06T10:49:47.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:47.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:47.444774+0000) 2024-03-06T10:49:48.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:48.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:48.444998+0000) 2024-03-06T10:49:49.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:49.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:49.445265+0000) 2024-03-06T10:49:50.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:50.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:50.445496+0000) 2024-03-06T10:49:51.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:51.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:51.445700+0000) 2024-03-06T10:49:52.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:52.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:52.445885+0000) 2024-03-06T10:49:53.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:53.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:53.446111+0000) 2024-03-06T10:49:54.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:54.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:54.446339+0000) 2024-03-06T10:49:55.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:55.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:55.446581+0000) 2024-03-06T10:49:56.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:56.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:56.446800+0000) 2024-03-06T10:49:57.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:57.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:57.447102+0000) 2024-03-06T10:49:58.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:58.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:58.447304+0000) 2024-03-06T10:49:59.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:49:59.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:49:59.447539+0000) 2024-03-06T10:50:00.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:00.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:00.447765+0000) 2024-03-06T10:50:01.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:01.215 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:01.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:01.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:01.448068+0000) 2024-03-06T10:50:02.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:02.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:02.448316+0000) 2024-03-06T10:50:03.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:03.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:03.448549+0000) 2024-03-06T10:50:04.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:04.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:04.448777+0000) 2024-03-06T10:50:05.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:05.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:05.449057+0000) 2024-03-06T10:50:06.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:06.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:06.449274+0000) 2024-03-06T10:50:07.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:07.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:07.449504+0000) 2024-03-06T10:50:08.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:08.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:08.449729+0000) 2024-03-06T10:50:09.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:09.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:09.449949+0000) 2024-03-06T10:50:10.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:10.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:10.450182+0000) 2024-03-06T10:50:11.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:11.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:11.450426+0000) 2024-03-06T10:50:12.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:12.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:12.450689+0000) 2024-03-06T10:50:13.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:13.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:13.450936+0000) 2024-03-06T10:50:14.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:14.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:14.451168+0000) 2024-03-06T10:50:15.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:15.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:15.451367+0000) 2024-03-06T10:50:16.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:16.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:16.451568+0000) 2024-03-06T10:50:17.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:17.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:17.451816+0000) 2024-03-06T10:50:18.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:18.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:18.452099+0000) 2024-03-06T10:50:19.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:19.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:19.452351+0000) 2024-03-06T10:50:20.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:20.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:20.452561+0000) 2024-03-06T10:50:21.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:21.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:21.452803+0000) 2024-03-06T10:50:22.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:22.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:22.453046+0000) 2024-03-06T10:50:23.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:23.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:23.453296+0000) 2024-03-06T10:50:24.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:24.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:24.453522+0000) 2024-03-06T10:50:25.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:25.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:25.453757+0000) 2024-03-06T10:50:26.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:26.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:26.454075+0000) 2024-03-06T10:50:27.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:27.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:27.454328+0000) 2024-03-06T10:50:28.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:28.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:28.454558+0000) 2024-03-06T10:50:29.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:29.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:29.454820+0000) 2024-03-06T10:50:30.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:30.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:30.455081+0000) 2024-03-06T10:50:31.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:31.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:50:31.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:31.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:31.455308+0000) 2024-03-06T10:50:32.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:32.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:32.455488+0000) 2024-03-06T10:50:33.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:33.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:33.455723+0000) 2024-03-06T10:50:34.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:34.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:34.455974+0000) 2024-03-06T10:50:35.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:35.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:35.456203+0000) 2024-03-06T10:50:36.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:36.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:36.456420+0000) 2024-03-06T10:50:37.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:37.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:37.456642+0000) 2024-03-06T10:50:38.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:38.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:38.456867+0000) 2024-03-06T10:50:39.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:39.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:39.457133+0000) 2024-03-06T10:50:40.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:40.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:40.457341+0000) 2024-03-06T10:50:41.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:41.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:41.457576+0000) 2024-03-06T10:50:42.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:42.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:42.457800+0000) 2024-03-06T10:50:43.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:43.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:43.458083+0000) 2024-03-06T10:50:44.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:44.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:44.458293+0000) 2024-03-06T10:50:45.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:45.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:45.458472+0000) 2024-03-06T10:50:46.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:46.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:46.458701+0000) 2024-03-06T10:50:47.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:47.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:47.458926+0000) 2024-03-06T10:50:48.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:48.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:48.459143+0000) 2024-03-06T10:50:49.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:49.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:49.459385+0000) 2024-03-06T10:50:50.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:50.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:50.459618+0000) 2024-03-06T10:50:51.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:51.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:51.459882+0000) 2024-03-06T10:50:52.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:52.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:52.460150+0000) 2024-03-06T10:50:53.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:53.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:53.460405+0000) 2024-03-06T10:50:54.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:54.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:54.460626+0000) 2024-03-06T10:50:55.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:55.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:55.460848+0000) 2024-03-06T10:50:56.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:56.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:56.461105+0000) 2024-03-06T10:50:57.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:57.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:57.461330+0000) 2024-03-06T10:50:58.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:58.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:58.461549+0000) 2024-03-06T10:50:59.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:50:59.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:50:59.461781+0000) 2024-03-06T10:51:00.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:00.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:00.462040+0000) 2024-03-06T10:51:01.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:01.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:01.462288+0000) 2024-03-06T10:51:01.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:01.553 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:02.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:02.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:02.462532+0000) 2024-03-06T10:51:03.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:03.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:03.462746+0000) 2024-03-06T10:51:04.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:04.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:04.462973+0000) 2024-03-06T10:51:05.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:05.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:05.463206+0000) 2024-03-06T10:51:06.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:06.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:06.463431+0000) 2024-03-06T10:51:07.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:07.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:07.463659+0000) 2024-03-06T10:51:08.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:08.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:08.463878+0000) 2024-03-06T10:51:09.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:09.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:09.464141+0000) 2024-03-06T10:51:10.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:10.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:10.464360+0000) 2024-03-06T10:51:11.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:11.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:11.464546+0000) 2024-03-06T10:51:12.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:12.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:12.464774+0000) 2024-03-06T10:51:13.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:13.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:13.464955+0000) 2024-03-06T10:51:14.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:14.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:14.465250+0000) 2024-03-06T10:51:15.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:15.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:15.465487+0000) 2024-03-06T10:51:16.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:16.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:16.465735+0000) 2024-03-06T10:51:17.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:17.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:17.465942+0000) 2024-03-06T10:51:18.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:18.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:18.466168+0000) 2024-03-06T10:51:19.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:19.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:19.466363+0000) 2024-03-06T10:51:20.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:20.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:20.466586+0000) 2024-03-06T10:51:21.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:21.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:21.466769+0000) 2024-03-06T10:51:22.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:22.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:22.466972+0000) 2024-03-06T10:51:23.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:23.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:23.467204+0000) 2024-03-06T10:51:24.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:24.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:24.467407+0000) 2024-03-06T10:51:25.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:25.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:25.467597+0000) 2024-03-06T10:51:26.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:26.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:26.467818+0000) 2024-03-06T10:51:27.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:27.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:27.468087+0000) 2024-03-06T10:51:28.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:28.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:28.468305+0000) 2024-03-06T10:51:29.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:29.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:29.468535+0000) 2024-03-06T10:51:30.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:30.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:30.468755+0000) 2024-03-06T10:51:31.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:31.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:31.468982+0000) 2024-03-06T10:51:31.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:31.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:51:32.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:32.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:32.469245+0000) 2024-03-06T10:51:33.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:33.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:33.469469+0000) 2024-03-06T10:51:34.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:34.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:34.469694+0000) 2024-03-06T10:51:35.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:35.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:35.469919+0000) 2024-03-06T10:51:36.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:36.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:36.470167+0000) 2024-03-06T10:51:37.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:37.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:37.470438+0000) 2024-03-06T10:51:38.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:38.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:38.470670+0000) 2024-03-06T10:51:39.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:39.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:39.470903+0000) 2024-03-06T10:51:40.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:40.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:40.471113+0000) 2024-03-06T10:51:41.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:41.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:41.471356+0000) 2024-03-06T10:51:42.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:42.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:42.471578+0000) 2024-03-06T10:51:43.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:43.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:43.471820+0000) 2024-03-06T10:51:44.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:44.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:44.472074+0000) 2024-03-06T10:51:45.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:45.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:45.472331+0000) 2024-03-06T10:51:46.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:46.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:46.472599+0000) 2024-03-06T10:51:47.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:47.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:47.472821+0000) 2024-03-06T10:51:48.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:48.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:48.473081+0000) 2024-03-06T10:51:49.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:49.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:49.473303+0000) 2024-03-06T10:51:50.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:50.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:50.473541+0000) 2024-03-06T10:51:51.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:51.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:51.473785+0000) 2024-03-06T10:51:52.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:52.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:52.473980+0000) 2024-03-06T10:51:53.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:53.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:53.474259+0000) 2024-03-06T10:51:54.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:54.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:54.474482+0000) 2024-03-06T10:51:55.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:55.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:55.474713+0000) 2024-03-06T10:51:56.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:56.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:56.474946+0000) 2024-03-06T10:51:57.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:57.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:57.475183+0000) 2024-03-06T10:51:58.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:58.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:58.475407+0000) 2024-03-06T10:51:59.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:51:59.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:51:59.475637+0000) 2024-03-06T10:52:00.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:00.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:00.475854+0000) 2024-03-06T10:52:01.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:01.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:01.476118+0000) 2024-03-06T10:52:01.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:01.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:02.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:02.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:02.476336+0000) 2024-03-06T10:52:03.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:03.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:03.476590+0000) 2024-03-06T10:52:04.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:04.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:04.476818+0000) 2024-03-06T10:52:05.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:05.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:05.477107+0000) 2024-03-06T10:52:06.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:06.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:06.477344+0000) 2024-03-06T10:52:07.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:07.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:07.477616+0000) 2024-03-06T10:52:08.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:08.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:08.477844+0000) 2024-03-06T10:52:09.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:09.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:09.478114+0000) 2024-03-06T10:52:10.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:10.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:10.478309+0000) 2024-03-06T10:52:11.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:11.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:11.478551+0000) 2024-03-06T10:52:12.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:12.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:12.478768+0000) 2024-03-06T10:52:13.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:13.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:13.478984+0000) 2024-03-06T10:52:14.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:14.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:14.479207+0000) 2024-03-06T10:52:15.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:15.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:15.479442+0000) 2024-03-06T10:52:16.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:16.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:16.479631+0000) 2024-03-06T10:52:17.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:17.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:17.479861+0000) 2024-03-06T10:52:18.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:18.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:18.480124+0000) 2024-03-06T10:52:19.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:19.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:19.480319+0000) 2024-03-06T10:52:20.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:20.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:20.480525+0000) 2024-03-06T10:52:21.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:21.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:21.480794+0000) 2024-03-06T10:52:22.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:22.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:22.481104+0000) 2024-03-06T10:52:23.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:23.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:23.481318+0000) 2024-03-06T10:52:24.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:24.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:24.481539+0000) 2024-03-06T10:52:25.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:25.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:25.481760+0000) 2024-03-06T10:52:26.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:26.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:26.481994+0000) 2024-03-06T10:52:27.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:27.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:27.482298+0000) 2024-03-06T10:52:28.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:28.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:28.482529+0000) 2024-03-06T10:52:29.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:29.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:29.482731+0000) 2024-03-06T10:52:30.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:30.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:30.482947+0000) 2024-03-06T10:52:31.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:31.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:31.483181+0000) 2024-03-06T10:52:32.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:32.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:52:32.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:32.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:32.483400+0000) 2024-03-06T10:52:33.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:33.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:33.483629+0000) 2024-03-06T10:52:34.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:34.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:34.483856+0000) 2024-03-06T10:52:35.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:35.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:35.484117+0000) 2024-03-06T10:52:36.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:36.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:36.484379+0000) 2024-03-06T10:52:37.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:37.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:37.484651+0000) 2024-03-06T10:52:38.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:38.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:38.484908+0000) 2024-03-06T10:52:39.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:39.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:39.485137+0000) 2024-03-06T10:52:40.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:40.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:40.485277+0000) 2024-03-06T10:52:41.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:41.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:41.485463+0000) 2024-03-06T10:52:42.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:42.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:42.485685+0000) 2024-03-06T10:52:43.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:43.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:43.485902+0000) 2024-03-06T10:52:44.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:44.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:44.486162+0000) 2024-03-06T10:52:45.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:45.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:45.486382+0000) 2024-03-06T10:52:46.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:46.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:46.486604+0000) 2024-03-06T10:52:47.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:47.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:47.486834+0000) 2024-03-06T10:52:48.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:48.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:48.487105+0000) 2024-03-06T10:52:49.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:49.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:49.487332+0000) 2024-03-06T10:52:50.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:50.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:50.487503+0000) 2024-03-06T10:52:51.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:51.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:51.487734+0000) 2024-03-06T10:52:52.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:52.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:52.487963+0000) 2024-03-06T10:52:53.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:53.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:53.488252+0000) 2024-03-06T10:52:54.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:54.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:54.488437+0000) 2024-03-06T10:52:55.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:55.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:55.488708+0000) 2024-03-06T10:52:56.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:56.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:56.488926+0000) 2024-03-06T10:52:57.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:57.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:57.489190+0000) 2024-03-06T10:52:58.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:58.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:58.489435+0000) 2024-03-06T10:52:59.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:52:59.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:52:59.489663+0000) 2024-03-06T10:53:00.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:00.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:00.489853+0000) 2024-03-06T10:53:01.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:01.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:01.490132+0000) 2024-03-06T10:53:02.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:02.313 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:02.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:02.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:02.490396+0000) 2024-03-06T10:53:03.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:03.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:03.490623+0000) 2024-03-06T10:53:04.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:04.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:04.490842+0000) 2024-03-06T10:53:05.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:05.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:05.491111+0000) 2024-03-06T10:53:06.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:06.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:06.491268+0000) 2024-03-06T10:53:07.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:07.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:07.491481+0000) 2024-03-06T10:53:08.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:08.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:08.491685+0000) 2024-03-06T10:53:09.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:09.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:09.491861+0000) 2024-03-06T10:53:10.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:10.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:10.492117+0000) 2024-03-06T10:53:11.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:11.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:11.492359+0000) 2024-03-06T10:53:12.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:12.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:12.492598+0000) 2024-03-06T10:53:13.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:13.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:13.492840+0000) 2024-03-06T10:53:14.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:14.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:14.493109+0000) 2024-03-06T10:53:15.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:15.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:15.493340+0000) 2024-03-06T10:53:16.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:16.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:16.493559+0000) 2024-03-06T10:53:17.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:17.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:17.493798+0000) 2024-03-06T10:53:18.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:18.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:18.493974+0000) 2024-03-06T10:53:19.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:19.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:19.494163+0000) 2024-03-06T10:53:20.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:20.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:20.494333+0000) 2024-03-06T10:53:21.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:21.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:21.494559+0000) 2024-03-06T10:53:22.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:22.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:22.494781+0000) 2024-03-06T10:53:23.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:23.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:23.495077+0000) 2024-03-06T10:53:24.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:24.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:24.495315+0000) 2024-03-06T10:53:25.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:25.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:25.495535+0000) 2024-03-06T10:53:26.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:26.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:26.495755+0000) 2024-03-06T10:53:27.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:27.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:27.495982+0000) 2024-03-06T10:53:28.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:28.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:28.496230+0000) 2024-03-06T10:53:29.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:29.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:29.496463+0000) 2024-03-06T10:53:30.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:30.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:30.496655+0000) 2024-03-06T10:53:31.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:31.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:31.496902+0000) 2024-03-06T10:53:32.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:32.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:32.497166+0000) 2024-03-06T10:53:32.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:32.524 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:53:33.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:33.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:33.497419+0000) 2024-03-06T10:53:34.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:34.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:34.497670+0000) 2024-03-06T10:53:35.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:35.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:35.497947+0000) 2024-03-06T10:53:36.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:36.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:36.498226+0000) 2024-03-06T10:53:37.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:37.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:37.498499+0000) 2024-03-06T10:53:38.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:38.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:38.498774+0000) 2024-03-06T10:53:39.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:39.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:39.499087+0000) 2024-03-06T10:53:40.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:40.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:40.499362+0000) 2024-03-06T10:53:41.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:41.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:41.499647+0000) 2024-03-06T10:53:42.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:42.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:42.499918+0000) 2024-03-06T10:53:43.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:43.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:43.500194+0000) 2024-03-06T10:53:44.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:44.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:44.500426+0000) 2024-03-06T10:53:45.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:45.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:45.500652+0000) 2024-03-06T10:53:46.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:46.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:46.500866+0000) 2024-03-06T10:53:47.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:47.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:47.501151+0000) 2024-03-06T10:53:48.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:48.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:48.501392+0000) 2024-03-06T10:53:49.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:49.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:49.501647+0000) 2024-03-06T10:53:50.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:50.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:50.501891+0000) 2024-03-06T10:53:51.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:51.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:51.502128+0000) 2024-03-06T10:53:52.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:52.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:52.502352+0000) 2024-03-06T10:53:53.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:53.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:53.502541+0000) 2024-03-06T10:53:54.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:54.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:54.502767+0000) 2024-03-06T10:53:55.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:55.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:55.503001+0000) 2024-03-06T10:53:56.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:56.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:56.503226+0000) 2024-03-06T10:53:57.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:57.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:57.503472+0000) 2024-03-06T10:53:58.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:58.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:58.503687+0000) 2024-03-06T10:53:59.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:53:59.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:53:59.503934+0000) 2024-03-06T10:54:00.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:00.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:00.504163+0000) 2024-03-06T10:54:01.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:01.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:01.504391+0000) 2024-03-06T10:54:02.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:02.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:02.504621+0000) 2024-03-06T10:54:02.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:02.730 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:03.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:03.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:03.504851+0000) 2024-03-06T10:54:04.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:04.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:04.505115+0000) 2024-03-06T10:54:05.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:05.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:05.505355+0000) 2024-03-06T10:54:06.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:06.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:06.505567+0000) 2024-03-06T10:54:07.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:07.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:07.505828+0000) 2024-03-06T10:54:08.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:08.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:08.506113+0000) 2024-03-06T10:54:09.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:09.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:09.506363+0000) 2024-03-06T10:54:10.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:10.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:10.506553+0000) 2024-03-06T10:54:11.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:11.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:11.506732+0000) 2024-03-06T10:54:12.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:12.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:12.506910+0000) 2024-03-06T10:54:13.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:13.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:13.507118+0000) 2024-03-06T10:54:14.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:14.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:14.507335+0000) 2024-03-06T10:54:15.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:15.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:15.507510+0000) 2024-03-06T10:54:16.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:16.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:16.507721+0000) 2024-03-06T10:54:17.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:17.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:17.507947+0000) 2024-03-06T10:54:18.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:18.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:18.508155+0000) 2024-03-06T10:54:19.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:19.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:19.508380+0000) 2024-03-06T10:54:20.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:20.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:20.508572+0000) 2024-03-06T10:54:21.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:21.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:21.508814+0000) 2024-03-06T10:54:22.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:22.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:22.509075+0000) 2024-03-06T10:54:23.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:23.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:23.509306+0000) 2024-03-06T10:54:24.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:24.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:24.509528+0000) 2024-03-06T10:54:25.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:25.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:25.509715+0000) 2024-03-06T10:54:26.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:26.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:26.509996+0000) 2024-03-06T10:54:27.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:27.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:27.510292+0000) 2024-03-06T10:54:28.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:28.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:28.510502+0000) 2024-03-06T10:54:29.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:29.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:29.510718+0000) 2024-03-06T10:54:30.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:30.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:30.510938+0000) 2024-03-06T10:54:31.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:31.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:31.511213+0000) 2024-03-06T10:54:32.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:32.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:32.511453+0000) 2024-03-06T10:54:32.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:32.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:54:33.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:33.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:33.511680+0000) 2024-03-06T10:54:34.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:34.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:34.511854+0000) 2024-03-06T10:54:35.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:35.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:35.512108+0000) 2024-03-06T10:54:36.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:36.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:36.512326+0000) 2024-03-06T10:54:37.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:37.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:37.512614+0000) 2024-03-06T10:54:38.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:38.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:38.512823+0000) 2024-03-06T10:54:39.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:39.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:39.513059+0000) 2024-03-06T10:54:40.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:40.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:40.513301+0000) 2024-03-06T10:54:41.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:41.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:41.513485+0000) 2024-03-06T10:54:42.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:42.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:42.513714+0000) 2024-03-06T10:54:43.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:43.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:43.513939+0000) 2024-03-06T10:54:44.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:44.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:44.514153+0000) 2024-03-06T10:54:45.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:45.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:45.514338+0000) 2024-03-06T10:54:46.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:46.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:46.514561+0000) 2024-03-06T10:54:47.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:47.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:47.514825+0000) 2024-03-06T10:54:48.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:48.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:48.515089+0000) 2024-03-06T10:54:49.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:49.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:49.515312+0000) 2024-03-06T10:54:50.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:50.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:50.515536+0000) 2024-03-06T10:54:51.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:51.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:51.515730+0000) 2024-03-06T10:54:52.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:52.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:52.515951+0000) 2024-03-06T10:54:53.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:53.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:53.516170+0000) 2024-03-06T10:54:54.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:54.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:54.516385+0000) 2024-03-06T10:54:55.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:55.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:55.516607+0000) 2024-03-06T10:54:56.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:56.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:56.516796+0000) 2024-03-06T10:54:57.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:57.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:57.517072+0000) 2024-03-06T10:54:58.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:58.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:58.517302+0000) 2024-03-06T10:54:59.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:54:59.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:54:59.517535+0000) 2024-03-06T10:55:00.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:00.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:00.517714+0000) 2024-03-06T10:55:01.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:01.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:01.517964+0000) 2024-03-06T10:55:02.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:02.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:02.518142+0000) 2024-03-06T10:55:03.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:03.094 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:03.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:03.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:03.518368+0000) 2024-03-06T10:55:04.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:04.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:04.518585+0000) 2024-03-06T10:55:05.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:05.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:05.518810+0000) 2024-03-06T10:55:06.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:06.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:06.519067+0000) 2024-03-06T10:55:07.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:07.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:07.519297+0000) 2024-03-06T10:55:08.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:08.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:08.519475+0000) 2024-03-06T10:55:09.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:09.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:09.519696+0000) 2024-03-06T10:55:10.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:10.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:10.519919+0000) 2024-03-06T10:55:11.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:11.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:11.520163+0000) 2024-03-06T10:55:12.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:12.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:12.520385+0000) 2024-03-06T10:55:13.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:13.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:13.520586+0000) 2024-03-06T10:55:14.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:14.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:14.520800+0000) 2024-03-06T10:55:15.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:15.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:15.521111+0000) 2024-03-06T10:55:16.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:16.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:16.521354+0000) 2024-03-06T10:55:17.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:17.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:17.521582+0000) 2024-03-06T10:55:18.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:18.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:18.521816+0000) 2024-03-06T10:55:19.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:19.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:19.522059+0000) 2024-03-06T10:55:20.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:20.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:20.522287+0000) 2024-03-06T10:55:21.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:21.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:21.522522+0000) 2024-03-06T10:55:22.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:22.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:22.522735+0000) 2024-03-06T10:55:23.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:23.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:23.523057+0000) 2024-03-06T10:55:24.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:24.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:24.523340+0000) 2024-03-06T10:55:25.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:25.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:25.523665+0000) 2024-03-06T10:55:26.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:26.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:26.523858+0000) 2024-03-06T10:55:27.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:27.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:27.524122+0000) 2024-03-06T10:55:28.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:28.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:28.524345+0000) 2024-03-06T10:55:29.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:29.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:29.524528+0000) 2024-03-06T10:55:30.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:30.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:30.524777+0000) 2024-03-06T10:55:31.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:31.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:31.525063+0000) 2024-03-06T10:55:32.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:32.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:32.525297+0000) 2024-03-06T10:55:33.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:33.263 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:55:33.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:33.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:33.525515+0000) 2024-03-06T10:55:34.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:34.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:34.525722+0000) 2024-03-06T10:55:35.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:35.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:35.525952+0000) 2024-03-06T10:55:36.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:36.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:36.526160+0000) 2024-03-06T10:55:37.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:37.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:37.526349+0000) 2024-03-06T10:55:38.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:38.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:38.526597+0000) 2024-03-06T10:55:39.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:39.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:39.526818+0000) 2024-03-06T10:55:40.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:40.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:40.527083+0000) 2024-03-06T10:55:41.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:41.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:41.527321+0000) 2024-03-06T10:55:42.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:42.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:42.527538+0000) 2024-03-06T10:55:43.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:43.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:43.527773+0000) 2024-03-06T10:55:44.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:44.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:44.527988+0000) 2024-03-06T10:55:45.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:45.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:45.528254+0000) 2024-03-06T10:55:46.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:46.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:46.528471+0000) 2024-03-06T10:55:47.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:47.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:47.528614+0000) 2024-03-06T10:55:48.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:48.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:48.528784+0000) 2024-03-06T10:55:49.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:49.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:49.529036+0000) 2024-03-06T10:55:50.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:50.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:50.529247+0000) 2024-03-06T10:55:51.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:51.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:51.529448+0000) 2024-03-06T10:55:52.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:52.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:52.529687+0000) 2024-03-06T10:55:53.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:53.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:53.529907+0000) 2024-03-06T10:55:54.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:54.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:54.530143+0000) 2024-03-06T10:55:55.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:55.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:55.530352+0000) 2024-03-06T10:55:56.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:56.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:56.530562+0000) 2024-03-06T10:55:57.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:57.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:57.530753+0000) 2024-03-06T10:55:58.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:58.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:58.530978+0000) 2024-03-06T10:55:59.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:55:59.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:55:59.531225+0000) 2024-03-06T10:56:00.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:00.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:00.531442+0000) 2024-03-06T10:56:01.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:01.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:01.531664+0000) 2024-03-06T10:56:02.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:02.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:02.531885+0000) 2024-03-06T10:56:03.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:03.461 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:03.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:03.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:03.532133+0000) 2024-03-06T10:56:04.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:04.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:04.532371+0000) 2024-03-06T10:56:05.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:05.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:05.532616+0000) 2024-03-06T10:56:06.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:06.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:06.532836+0000) 2024-03-06T10:56:07.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:07.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:07.533130+0000) 2024-03-06T10:56:08.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:08.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:08.533344+0000) 2024-03-06T10:56:09.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:09.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:09.533585+0000) 2024-03-06T10:56:10.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:10.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:10.533827+0000) 2024-03-06T10:56:11.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:11.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:11.534113+0000) 2024-03-06T10:56:12.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:12.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:12.534346+0000) 2024-03-06T10:56:13.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:13.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:13.534650+0000) 2024-03-06T10:56:14.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:14.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:14.534890+0000) 2024-03-06T10:56:15.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:15.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:15.535120+0000) 2024-03-06T10:56:16.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:16.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:16.535344+0000) 2024-03-06T10:56:17.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:17.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:17.535582+0000) 2024-03-06T10:56:18.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:18.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:18.535832+0000) 2024-03-06T10:56:19.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:19.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:19.536055+0000) 2024-03-06T10:56:20.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:20.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:20.536240+0000) 2024-03-06T10:56:21.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:21.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:21.536457+0000) 2024-03-06T10:56:22.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:22.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:22.536684+0000) 2024-03-06T10:56:23.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:23.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:23.536906+0000) 2024-03-06T10:56:24.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:24.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:24.537135+0000) 2024-03-06T10:56:25.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:25.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:25.537378+0000) 2024-03-06T10:56:26.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:26.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:26.537594+0000) 2024-03-06T10:56:27.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:27.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:27.537827+0000) 2024-03-06T10:56:28.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:28.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:28.538109+0000) 2024-03-06T10:56:29.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:29.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:29.538360+0000) 2024-03-06T10:56:30.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:30.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:30.538601+0000) 2024-03-06T10:56:31.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:31.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:31.538830+0000) 2024-03-06T10:56:32.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:32.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:32.539097+0000) 2024-03-06T10:56:33.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:33.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:33.539320+0000) 2024-03-06T10:56:33.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:33.657 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:56:34.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:34.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:34.539494+0000) 2024-03-06T10:56:35.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:35.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:35.539721+0000) 2024-03-06T10:56:36.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:36.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:36.539940+0000) 2024-03-06T10:56:37.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:37.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:37.540175+0000) 2024-03-06T10:56:38.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:38.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:38.540395+0000) 2024-03-06T10:56:39.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:39.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:39.540622+0000) 2024-03-06T10:56:40.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:40.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:40.540865+0000) 2024-03-06T10:56:41.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:41.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:41.541120+0000) 2024-03-06T10:56:42.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:42.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:42.541345+0000) 2024-03-06T10:56:43.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:43.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:43.541576+0000) 2024-03-06T10:56:44.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:44.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:44.541797+0000) 2024-03-06T10:56:45.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:45.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:45.541994+0000) 2024-03-06T10:56:46.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:46.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:46.542246+0000) 2024-03-06T10:56:47.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:47.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:47.542486+0000) 2024-03-06T10:56:48.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:48.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:48.542707+0000) 2024-03-06T10:56:49.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:49.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:49.542933+0000) 2024-03-06T10:56:50.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:50.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:50.543156+0000) 2024-03-06T10:56:51.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:51.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:51.543387+0000) 2024-03-06T10:56:52.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:52.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:52.543624+0000) 2024-03-06T10:56:53.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:53.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:53.543825+0000) 2024-03-06T10:56:54.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:54.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:54.544086+0000) 2024-03-06T10:56:55.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:55.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:55.544353+0000) 2024-03-06T10:56:56.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:56.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:56.544628+0000) 2024-03-06T10:56:57.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:57.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:57.544919+0000) 2024-03-06T10:56:58.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:58.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:58.545157+0000) 2024-03-06T10:56:59.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:56:59.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:56:59.545380+0000) 2024-03-06T10:57:00.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:00.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:00.545596+0000) 2024-03-06T10:57:01.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:01.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:01.545776+0000) 2024-03-06T10:57:02.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:02.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:02.546091+0000) 2024-03-06T10:57:03.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:03.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:03.546340+0000) 2024-03-06T10:57:03.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:03.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:04.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:04.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:04.546610+0000) 2024-03-06T10:57:05.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:05.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:05.546815+0000) 2024-03-06T10:57:06.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:06.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:06.547165+0000) 2024-03-06T10:57:07.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:07.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:07.547395+0000) 2024-03-06T10:57:08.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:08.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:08.547612+0000) 2024-03-06T10:57:09.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:09.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:09.547837+0000) 2024-03-06T10:57:10.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:10.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:10.548099+0000) 2024-03-06T10:57:11.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:11.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:11.548303+0000) 2024-03-06T10:57:12.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:12.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:12.548554+0000) 2024-03-06T10:57:13.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:13.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:13.548804+0000) 2024-03-06T10:57:14.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:14.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:14.549050+0000) 2024-03-06T10:57:15.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:15.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:15.549241+0000) 2024-03-06T10:57:16.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:16.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:16.549425+0000) 2024-03-06T10:57:17.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:17.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:17.549702+0000) 2024-03-06T10:57:18.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:18.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:18.549981+0000) 2024-03-06T10:57:19.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:19.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:19.550314+0000) 2024-03-06T10:57:20.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:20.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:20.550591+0000) 2024-03-06T10:57:21.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:21.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:21.550845+0000) 2024-03-06T10:57:22.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:22.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:22.551155+0000) 2024-03-06T10:57:23.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:23.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:23.551445+0000) 2024-03-06T10:57:24.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:24.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:24.551649+0000) 2024-03-06T10:57:25.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:25.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:25.551884+0000) 2024-03-06T10:57:26.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:26.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:26.552114+0000) 2024-03-06T10:57:27.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:27.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:27.552328+0000) 2024-03-06T10:57:28.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:28.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:28.552487+0000) 2024-03-06T10:57:29.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:29.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:29.552703+0000) 2024-03-06T10:57:30.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:30.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:30.552934+0000) 2024-03-06T10:57:31.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:31.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:31.553152+0000) 2024-03-06T10:57:32.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:32.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:32.553375+0000) 2024-03-06T10:57:33.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:33.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:33.553608+0000) 2024-03-06T10:57:34.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:34.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:57:34.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:34.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:34.553832+0000) 2024-03-06T10:57:35.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:35.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:35.554107+0000) 2024-03-06T10:57:36.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:36.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:36.554322+0000) 2024-03-06T10:57:37.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:37.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:37.554565+0000) 2024-03-06T10:57:38.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:38.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:38.554793+0000) 2024-03-06T10:57:39.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:39.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:39.555044+0000) 2024-03-06T10:57:40.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:40.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:40.555268+0000) 2024-03-06T10:57:41.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:41.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:41.555500+0000) 2024-03-06T10:57:42.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:42.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:42.555743+0000) 2024-03-06T10:57:43.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:43.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:43.555973+0000) 2024-03-06T10:57:44.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:44.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:44.556201+0000) 2024-03-06T10:57:45.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:45.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:45.556429+0000) 2024-03-06T10:57:46.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:46.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:46.556652+0000) 2024-03-06T10:57:47.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:47.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:47.556891+0000) 2024-03-06T10:57:48.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:48.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:48.557121+0000) 2024-03-06T10:57:49.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:49.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:49.557346+0000) 2024-03-06T10:57:50.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:50.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:50.557565+0000) 2024-03-06T10:57:51.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:51.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:51.557815+0000) 2024-03-06T10:57:52.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:52.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:52.558091+0000) 2024-03-06T10:57:53.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:53.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:53.558310+0000) 2024-03-06T10:57:54.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:54.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:54.558598+0000) 2024-03-06T10:57:55.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:55.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:55.558827+0000) 2024-03-06T10:57:56.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:56.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:56.559103+0000) 2024-03-06T10:57:57.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:57.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:57.559359+0000) 2024-03-06T10:57:58.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:58.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:58.559596+0000) 2024-03-06T10:57:59.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:57:59.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:57:59.559838+0000) 2024-03-06T10:58:00.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:00.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:00.560119+0000) 2024-03-06T10:58:01.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:01.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:01.560323+0000) 2024-03-06T10:58:02.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:02.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:02.560565+0000) 2024-03-06T10:58:03.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:03.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:03.560793+0000) 2024-03-06T10:58:04.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:04.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:04.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:04.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:04.561058+0000) 2024-03-06T10:58:05.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:05.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:05.561286+0000) 2024-03-06T10:58:06.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:06.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:06.561505+0000) 2024-03-06T10:58:07.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:07.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:07.561754+0000) 2024-03-06T10:58:08.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:08.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:08.561993+0000) 2024-03-06T10:58:09.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:09.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:09.562274+0000) 2024-03-06T10:58:10.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:10.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:10.562501+0000) 2024-03-06T10:58:11.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:11.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:11.562756+0000) 2024-03-06T10:58:12.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:12.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:12.562997+0000) 2024-03-06T10:58:13.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:13.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:13.563294+0000) 2024-03-06T10:58:14.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:14.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:14.563534+0000) 2024-03-06T10:58:15.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:15.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:15.563793+0000) 2024-03-06T10:58:16.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:16.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:16.564059+0000) 2024-03-06T10:58:17.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:17.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:17.564273+0000) 2024-03-06T10:58:18.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:18.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:18.564496+0000) 2024-03-06T10:58:19.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:19.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:19.564728+0000) 2024-03-06T10:58:20.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:20.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:20.564964+0000) 2024-03-06T10:58:21.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:21.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:21.565218+0000) 2024-03-06T10:58:22.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:22.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:22.565458+0000) 2024-03-06T10:58:23.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:23.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:23.565643+0000) 2024-03-06T10:58:24.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:24.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:24.565861+0000) 2024-03-06T10:58:25.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:25.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:25.566084+0000) 2024-03-06T10:58:26.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:26.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:26.566318+0000) 2024-03-06T10:58:27.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:27.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:27.566542+0000) 2024-03-06T10:58:28.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:28.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:28.566721+0000) 2024-03-06T10:58:29.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:29.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:29.566880+0000) 2024-03-06T10:58:30.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:30.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:30.567132+0000) 2024-03-06T10:58:31.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:31.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:31.567367+0000) 2024-03-06T10:58:32.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:32.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:32.567586+0000) 2024-03-06T10:58:33.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:33.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:33.567811+0000) 2024-03-06T10:58:34.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:34.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:58:34.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:34.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:34.568107+0000) 2024-03-06T10:58:35.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:35.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:35.568347+0000) 2024-03-06T10:58:36.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:36.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:36.568526+0000) 2024-03-06T10:58:37.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:37.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:37.568769+0000) 2024-03-06T10:58:38.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:38.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:38.569039+0000) 2024-03-06T10:58:39.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:39.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:39.569301+0000) 2024-03-06T10:58:40.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:40.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:40.569534+0000) 2024-03-06T10:58:41.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:41.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:41.569761+0000) 2024-03-06T10:58:42.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:42.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:42.569995+0000) 2024-03-06T10:58:43.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:43.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:43.570300+0000) 2024-03-06T10:58:44.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:44.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:44.570522+0000) 2024-03-06T10:58:45.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:45.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:45.570746+0000) 2024-03-06T10:58:46.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:46.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:46.570974+0000) 2024-03-06T10:58:47.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:47.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:47.571263+0000) 2024-03-06T10:58:48.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:48.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:48.571478+0000) 2024-03-06T10:58:49.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:49.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:49.571689+0000) 2024-03-06T10:58:50.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:50.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:50.571923+0000) 2024-03-06T10:58:51.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:51.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:51.572168+0000) 2024-03-06T10:58:52.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:52.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:52.572398+0000) 2024-03-06T10:58:53.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:53.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:53.572593+0000) 2024-03-06T10:58:54.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:54.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:54.572813+0000) 2024-03-06T10:58:55.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:55.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:55.573096+0000) 2024-03-06T10:58:56.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:56.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:56.573331+0000) 2024-03-06T10:58:57.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:57.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:57.573574+0000) 2024-03-06T10:58:58.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:58.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:58.573799+0000) 2024-03-06T10:58:59.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:58:59.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:58:59.574090+0000) 2024-03-06T10:59:00.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:00.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:00.574323+0000) 2024-03-06T10:59:01.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:01.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:01.574533+0000) 2024-03-06T10:59:02.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:02.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:02.574759+0000) 2024-03-06T10:59:03.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:03.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:03.574991+0000) 2024-03-06T10:59:04.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:04.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:04.575262+0000) 2024-03-06T10:59:04.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:04.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:05.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:05.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:05.575489+0000) 2024-03-06T10:59:06.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:06.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:06.575673+0000) 2024-03-06T10:59:07.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:07.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:07.575899+0000) 2024-03-06T10:59:08.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:08.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:08.576131+0000) 2024-03-06T10:59:09.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:09.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:09.576361+0000) 2024-03-06T10:59:10.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:10.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:10.576620+0000) 2024-03-06T10:59:11.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:11.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:11.576850+0000) 2024-03-06T10:59:12.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:12.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:12.577113+0000) 2024-03-06T10:59:13.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:13.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:13.577294+0000) 2024-03-06T10:59:14.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:14.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:14.577518+0000) 2024-03-06T10:59:15.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:15.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:15.577695+0000) 2024-03-06T10:59:16.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:16.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:16.577866+0000) 2024-03-06T10:59:17.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:17.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:17.578106+0000) 2024-03-06T10:59:18.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:18.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:18.578295+0000) 2024-03-06T10:59:19.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:19.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:19.578523+0000) 2024-03-06T10:59:20.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:20.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:20.578743+0000) 2024-03-06T10:59:21.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:21.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:21.578966+0000) 2024-03-06T10:59:22.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:22.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:22.579153+0000) 2024-03-06T10:59:23.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:23.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:23.579362+0000) 2024-03-06T10:59:24.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:24.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:24.579579+0000) 2024-03-06T10:59:25.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:25.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:25.579748+0000) 2024-03-06T10:59:26.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:26.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:26.579917+0000) 2024-03-06T10:59:27.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:27.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:27.580119+0000) 2024-03-06T10:59:28.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:28.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:28.580353+0000) 2024-03-06T10:59:29.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:29.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:29.580584+0000) 2024-03-06T10:59:30.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:30.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:30.580806+0000) 2024-03-06T10:59:31.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:31.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:31.581089+0000) 2024-03-06T10:59:32.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:32.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:32.581329+0000) 2024-03-06T10:59:33.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:33.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:33.581554+0000) 2024-03-06T10:59:34.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:34.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:34.581753+0000) 2024-03-06T10:59:34.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:34.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T10:59:35.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:35.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:35.581958+0000) 2024-03-06T10:59:36.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:36.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:36.582194+0000) 2024-03-06T10:59:37.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:37.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:37.582377+0000) 2024-03-06T10:59:38.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:38.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:38.582600+0000) 2024-03-06T10:59:39.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:39.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:39.582833+0000) 2024-03-06T10:59:40.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:40.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:40.583093+0000) 2024-03-06T10:59:41.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:41.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:41.583322+0000) 2024-03-06T10:59:42.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:42.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:42.583544+0000) 2024-03-06T10:59:43.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:43.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:43.583779+0000) 2024-03-06T10:59:44.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:44.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:44.584036+0000) 2024-03-06T10:59:45.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:45.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:45.584279+0000) 2024-03-06T10:59:46.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:46.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:46.584488+0000) 2024-03-06T10:59:47.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:47.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:47.584714+0000) 2024-03-06T10:59:48.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:48.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:48.584889+0000) 2024-03-06T10:59:49.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:49.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:49.585116+0000) 2024-03-06T10:59:50.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:50.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:50.585349+0000) 2024-03-06T10:59:51.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:51.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:51.585541+0000) 2024-03-06T10:59:52.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:52.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:52.585763+0000) 2024-03-06T10:59:53.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:53.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:53.585989+0000) 2024-03-06T10:59:54.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:54.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:54.586274+0000) 2024-03-06T10:59:55.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:55.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:55.586503+0000) 2024-03-06T10:59:56.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:56.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:56.586742+0000) 2024-03-06T10:59:57.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:57.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:57.586965+0000) 2024-03-06T10:59:58.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:58.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:58.587190+0000) 2024-03-06T10:59:59.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T10:59:59.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T09:59:59.587379+0000) 2024-03-06T11:00:00.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:00.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:00.587601+0000) 2024-03-06T11:00:01.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:01.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:01.587829+0000) 2024-03-06T11:00:02.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:02.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:02.588105+0000) 2024-03-06T11:00:03.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:03.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:03.588335+0000) 2024-03-06T11:00:04.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:04.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:04.588557+0000) 2024-03-06T11:00:05.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:05.201 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:05.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:05.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:05.588793+0000) 2024-03-06T11:00:06.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:06.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:06.589047+0000) 2024-03-06T11:00:07.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:07.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:07.589293+0000) 2024-03-06T11:00:08.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:08.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:08.589517+0000) 2024-03-06T11:00:09.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:09.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:09.589735+0000) 2024-03-06T11:00:10.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:10.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:10.589957+0000) 2024-03-06T11:00:11.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:11.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:11.590153+0000) 2024-03-06T11:00:12.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:12.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:12.590376+0000) 2024-03-06T11:00:13.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:13.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:13.590608+0000) 2024-03-06T11:00:14.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:14.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:14.590795+0000) 2024-03-06T11:00:15.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:15.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:15.591071+0000) 2024-03-06T11:00:16.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:16.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:16.591297+0000) 2024-03-06T11:00:17.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:17.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:17.591488+0000) 2024-03-06T11:00:18.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:18.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:18.591712+0000) 2024-03-06T11:00:19.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:19.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:19.591859+0000) 2024-03-06T11:00:20.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:20.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:20.592103+0000) 2024-03-06T11:00:21.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:21.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:21.592369+0000) 2024-03-06T11:00:22.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:22.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:22.592588+0000) 2024-03-06T11:00:23.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:23.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:23.592905+0000) 2024-03-06T11:00:24.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:24.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:24.593161+0000) 2024-03-06T11:00:25.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:25.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:25.593399+0000) 2024-03-06T11:00:26.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:26.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:26.593638+0000) 2024-03-06T11:00:27.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:27.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:27.593870+0000) 2024-03-06T11:00:28.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:28.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:28.594136+0000) 2024-03-06T11:00:29.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:29.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:29.594355+0000) 2024-03-06T11:00:30.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:30.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:30.594522+0000) 2024-03-06T11:00:31.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:31.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:31.594743+0000) 2024-03-06T11:00:32.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:32.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:32.594963+0000) 2024-03-06T11:00:33.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:33.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:33.595263+0000) 2024-03-06T11:00:34.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:34.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:34.595450+0000) 2024-03-06T11:00:35.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:35.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:00:35.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:35.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:35.595700+0000) 2024-03-06T11:00:36.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:36.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:36.595883+0000) 2024-03-06T11:00:37.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:37.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:37.596132+0000) 2024-03-06T11:00:38.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:38.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:38.596327+0000) 2024-03-06T11:00:39.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:39.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:39.596559+0000) 2024-03-06T11:00:40.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:40.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:40.596790+0000) 2024-03-06T11:00:41.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:41.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:41.596974+0000) 2024-03-06T11:00:42.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:42.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:42.597209+0000) 2024-03-06T11:00:43.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:43.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:43.597431+0000) 2024-03-06T11:00:44.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:44.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:44.597612+0000) 2024-03-06T11:00:45.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:45.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:45.597813+0000) 2024-03-06T11:00:46.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:46.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:46.598061+0000) 2024-03-06T11:00:47.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:47.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:47.598292+0000) 2024-03-06T11:00:48.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:48.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:48.598515+0000) 2024-03-06T11:00:49.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:49.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:49.598713+0000) 2024-03-06T11:00:50.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:50.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:50.598940+0000) 2024-03-06T11:00:51.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:51.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:51.599170+0000) 2024-03-06T11:00:52.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:52.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:52.599391+0000) 2024-03-06T11:00:53.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:53.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:53.599612+0000) 2024-03-06T11:00:54.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:54.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:54.599841+0000) 2024-03-06T11:00:55.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:55.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:55.600101+0000) 2024-03-06T11:00:56.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:56.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:56.600322+0000) 2024-03-06T11:00:57.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:57.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:57.600500+0000) 2024-03-06T11:00:58.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:58.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:58.600743+0000) 2024-03-06T11:00:59.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:00:59.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:00:59.600999+0000) 2024-03-06T11:01:00.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:00.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:00.601294+0000) 2024-03-06T11:01:01.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:01.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:01.601547+0000) 2024-03-06T11:01:02.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:02.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:02.601788+0000) 2024-03-06T11:01:03.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:03.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:03.602091+0000) 2024-03-06T11:01:04.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:04.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:04.602267+0000) 2024-03-06T11:01:05.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:05.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:05.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:05.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:05.602444+0000) 2024-03-06T11:01:06.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:06.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:06.602631+0000) 2024-03-06T11:01:07.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:07.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:07.602825+0000) 2024-03-06T11:01:08.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:08.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:08.603059+0000) 2024-03-06T11:01:09.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:09.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:09.603302+0000) 2024-03-06T11:01:10.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:10.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:10.603567+0000) 2024-03-06T11:01:11.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:11.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:11.603879+0000) 2024-03-06T11:01:12.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:12.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:12.604125+0000) 2024-03-06T11:01:13.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:13.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:13.604336+0000) 2024-03-06T11:01:14.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:14.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:14.604558+0000) 2024-03-06T11:01:15.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:15.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:15.604789+0000) 2024-03-06T11:01:16.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:16.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:16.605048+0000) 2024-03-06T11:01:17.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:17.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:17.605245+0000) 2024-03-06T11:01:18.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:18.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:18.605488+0000) 2024-03-06T11:01:19.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:19.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:19.605723+0000) 2024-03-06T11:01:20.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:20.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:20.605947+0000) 2024-03-06T11:01:21.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:21.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:21.606204+0000) 2024-03-06T11:01:22.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:22.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:22.606448+0000) 2024-03-06T11:01:23.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:23.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:23.606692+0000) 2024-03-06T11:01:24.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:24.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:24.606934+0000) 2024-03-06T11:01:25.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:25.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:25.607183+0000) 2024-03-06T11:01:26.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:26.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:26.607420+0000) 2024-03-06T11:01:27.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:27.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:27.607662+0000) 2024-03-06T11:01:28.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:28.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:28.607845+0000) 2024-03-06T11:01:29.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:29.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:29.608113+0000) 2024-03-06T11:01:30.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:30.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:30.608298+0000) 2024-03-06T11:01:31.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:31.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:31.608556+0000) 2024-03-06T11:01:32.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:32.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:32.608799+0000) 2024-03-06T11:01:33.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:33.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:33.609097+0000) 2024-03-06T11:01:34.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:34.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:34.609304+0000) 2024-03-06T11:01:35.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:35.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:35.609544+0000) 2024-03-06T11:01:35.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:35.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:01:36.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:36.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:36.609788+0000) 2024-03-06T11:01:37.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:37.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:37.610063+0000) 2024-03-06T11:01:38.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:38.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:38.610320+0000) 2024-03-06T11:01:39.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:39.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:39.610569+0000) 2024-03-06T11:01:40.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:40.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:40.610789+0000) 2024-03-06T11:01:41.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:41.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:41.611076+0000) 2024-03-06T11:01:42.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:42.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:42.611315+0000) 2024-03-06T11:01:43.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:43.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:43.611570+0000) 2024-03-06T11:01:44.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:44.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:44.611816+0000) 2024-03-06T11:01:45.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:45.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:45.611995+0000) 2024-03-06T11:01:46.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:46.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:46.612200+0000) 2024-03-06T11:01:47.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:47.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:47.612447+0000) 2024-03-06T11:01:48.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:48.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:48.612674+0000) 2024-03-06T11:01:49.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:49.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:49.612869+0000) 2024-03-06T11:01:50.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:50.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:50.613091+0000) 2024-03-06T11:01:51.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:51.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:51.613351+0000) 2024-03-06T11:01:52.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:52.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:52.613597+0000) 2024-03-06T11:01:53.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:53.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:53.613854+0000) 2024-03-06T11:01:54.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:54.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:54.614140+0000) 2024-03-06T11:01:55.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:55.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:55.614380+0000) 2024-03-06T11:01:56.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:56.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:56.614614+0000) 2024-03-06T11:01:57.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:57.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:57.614840+0000) 2024-03-06T11:01:58.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:58.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:58.615119+0000) 2024-03-06T11:01:59.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:01:59.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:01:59.615315+0000) 2024-03-06T11:02:00.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:00.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:00.615543+0000) 2024-03-06T11:02:01.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:01.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:01.615769+0000) 2024-03-06T11:02:02.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:02.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:02.615991+0000) 2024-03-06T11:02:03.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:03.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:03.616203+0000) 2024-03-06T11:02:04.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:04.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:04.616419+0000) 2024-03-06T11:02:05.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:05.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:05.616695+0000) 2024-03-06T11:02:05.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:05.885 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:06.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:06.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:06.616922+0000) 2024-03-06T11:02:07.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:07.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:07.617167+0000) 2024-03-06T11:02:08.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:08.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:08.617364+0000) 2024-03-06T11:02:09.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:09.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:09.617591+0000) 2024-03-06T11:02:10.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:10.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:10.617796+0000) 2024-03-06T11:02:11.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:11.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:11.618078+0000) 2024-03-06T11:02:12.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:12.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:12.618277+0000) 2024-03-06T11:02:13.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:13.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:13.618493+0000) 2024-03-06T11:02:14.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:14.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:14.618728+0000) 2024-03-06T11:02:15.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:15.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:15.618904+0000) 2024-03-06T11:02:16.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:16.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:16.619137+0000) 2024-03-06T11:02:17.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:17.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:17.619356+0000) 2024-03-06T11:02:18.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:18.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:18.619589+0000) 2024-03-06T11:02:19.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:19.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:19.619813+0000) 2024-03-06T11:02:20.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:20.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:20.620089+0000) 2024-03-06T11:02:21.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:21.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:21.620319+0000) 2024-03-06T11:02:22.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:22.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:22.620546+0000) 2024-03-06T11:02:23.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:23.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:23.620788+0000) 2024-03-06T11:02:24.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:24.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:24.621043+0000) 2024-03-06T11:02:25.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:25.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:25.621268+0000) 2024-03-06T11:02:26.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:26.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:26.621483+0000) 2024-03-06T11:02:27.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:27.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:27.621709+0000) 2024-03-06T11:02:28.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:28.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:28.621939+0000) 2024-03-06T11:02:29.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:29.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:29.622171+0000) 2024-03-06T11:02:30.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:30.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:30.622393+0000) 2024-03-06T11:02:31.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:31.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:31.622628+0000) 2024-03-06T11:02:32.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:32.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:32.622849+0000) 2024-03-06T11:02:33.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:33.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:33.623096+0000) 2024-03-06T11:02:34.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:34.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:34.623322+0000) 2024-03-06T11:02:35.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:35.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:35.623583+0000) 2024-03-06T11:02:36.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:36.081 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:02:36.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:36.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:36.623781+0000) 2024-03-06T11:02:37.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:37.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:37.624062+0000) 2024-03-06T11:02:38.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:38.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:38.624311+0000) 2024-03-06T11:02:39.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:39.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:39.624537+0000) 2024-03-06T11:02:40.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:40.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:40.624756+0000) 2024-03-06T11:02:41.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:41.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:41.624987+0000) 2024-03-06T11:02:42.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:42.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:42.625251+0000) 2024-03-06T11:02:43.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:43.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:43.625514+0000) 2024-03-06T11:02:44.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:44.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:44.625757+0000) 2024-03-06T11:02:45.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:45.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:45.625998+0000) 2024-03-06T11:02:46.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:46.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:46.626310+0000) 2024-03-06T11:02:47.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:47.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:47.626556+0000) 2024-03-06T11:02:48.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:48.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:48.626816+0000) 2024-03-06T11:02:49.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:49.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:49.627112+0000) 2024-03-06T11:02:50.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:50.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:50.627349+0000) 2024-03-06T11:02:51.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:51.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:51.627581+0000) 2024-03-06T11:02:52.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:52.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:52.627801+0000) 2024-03-06T11:02:53.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:53.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:53.628097+0000) 2024-03-06T11:02:54.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:54.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:54.628326+0000) 2024-03-06T11:02:55.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:55.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:55.628511+0000) 2024-03-06T11:02:56.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:56.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:56.628734+0000) 2024-03-06T11:02:57.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:57.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:57.628957+0000) 2024-03-06T11:02:58.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:58.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:58.629245+0000) 2024-03-06T11:02:59.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:02:59.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:02:59.629472+0000) 2024-03-06T11:03:00.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:00.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:00.629702+0000) 2024-03-06T11:03:01.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:01.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:01.629924+0000) 2024-03-06T11:03:02.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:02.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:02.630154+0000) 2024-03-06T11:03:03.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:03.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:03.630401+0000) 2024-03-06T11:03:04.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:04.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:04.630600+0000) 2024-03-06T11:03:05.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:05.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:05.630840+0000) 2024-03-06T11:03:06.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:06.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:06.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:06.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:06.631114+0000) 2024-03-06T11:03:07.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:07.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:07.631363+0000) 2024-03-06T11:03:08.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:08.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:08.631619+0000) 2024-03-06T11:03:09.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:09.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:09.631852+0000) 2024-03-06T11:03:10.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:10.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:10.632038+0000) 2024-03-06T11:03:11.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:11.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:11.632323+0000) 2024-03-06T11:03:12.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:12.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:12.632555+0000) 2024-03-06T11:03:13.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:13.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:13.632801+0000) 2024-03-06T11:03:14.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:14.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:14.633066+0000) 2024-03-06T11:03:15.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:15.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:15.633297+0000) 2024-03-06T11:03:16.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:16.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:16.633526+0000) 2024-03-06T11:03:17.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:17.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:17.633755+0000) 2024-03-06T11:03:18.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:18.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:18.633991+0000) 2024-03-06T11:03:19.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:19.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:19.634261+0000) 2024-03-06T11:03:20.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:20.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:20.634485+0000) 2024-03-06T11:03:21.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:21.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:21.634718+0000) 2024-03-06T11:03:22.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:22.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:22.634942+0000) 2024-03-06T11:03:23.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:23.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:23.635201+0000) 2024-03-06T11:03:24.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:24.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:24.635391+0000) 2024-03-06T11:03:25.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:25.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:25.635643+0000) 2024-03-06T11:03:26.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:26.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:26.635900+0000) 2024-03-06T11:03:27.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:27.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:27.636072+0000) 2024-03-06T11:03:28.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:28.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:28.636304+0000) 2024-03-06T11:03:29.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:29.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:29.636536+0000) 2024-03-06T11:03:30.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:30.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:30.636777+0000) 2024-03-06T11:03:31.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:31.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:31.637048+0000) 2024-03-06T11:03:32.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:32.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:32.637266+0000) 2024-03-06T11:03:33.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:33.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:33.637524+0000) 2024-03-06T11:03:34.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:34.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:34.637681+0000) 2024-03-06T11:03:35.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:35.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:35.637905+0000) 2024-03-06T11:03:36.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:36.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:03:36.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:36.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:36.638140+0000) 2024-03-06T11:03:37.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:37.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:37.638347+0000) 2024-03-06T11:03:38.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:38.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:38.638615+0000) 2024-03-06T11:03:39.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:39.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:39.638890+0000) 2024-03-06T11:03:40.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:40.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:40.639150+0000) 2024-03-06T11:03:41.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:41.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:41.639379+0000) 2024-03-06T11:03:42.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:42.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:42.639601+0000) 2024-03-06T11:03:43.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:43.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:43.639857+0000) 2024-03-06T11:03:44.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:44.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:44.640145+0000) 2024-03-06T11:03:45.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:45.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:45.640358+0000) 2024-03-06T11:03:46.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:46.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:46.640722+0000) 2024-03-06T11:03:47.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:47.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:47.641049+0000) 2024-03-06T11:03:48.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:48.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:48.641300+0000) 2024-03-06T11:03:49.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:49.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:49.641541+0000) 2024-03-06T11:03:50.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:50.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:50.641780+0000) 2024-03-06T11:03:51.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:51.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:51.642063+0000) 2024-03-06T11:03:52.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:52.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:52.642284+0000) 2024-03-06T11:03:53.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:53.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:53.642525+0000) 2024-03-06T11:03:54.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:54.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:54.642747+0000) 2024-03-06T11:03:55.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:55.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:55.642996+0000) 2024-03-06T11:03:56.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:56.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:56.643203+0000) 2024-03-06T11:03:57.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:57.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:57.643448+0000) 2024-03-06T11:03:58.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:58.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:58.643675+0000) 2024-03-06T11:03:59.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:03:59.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:03:59.643917+0000) 2024-03-06T11:04:00.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:00.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:00.644155+0000) 2024-03-06T11:04:01.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:01.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:01.644390+0000) 2024-03-06T11:04:02.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:02.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:02.644593+0000) 2024-03-06T11:04:03.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:03.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:03.644830+0000) 2024-03-06T11:04:04.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:04.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:04.645100+0000) 2024-03-06T11:04:05.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:05.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:05.645297+0000) 2024-03-06T11:04:06.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:06.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:06.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:06.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:06.645461+0000) 2024-03-06T11:04:07.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:07.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:07.645686+0000) 2024-03-06T11:04:08.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:08.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:08.645907+0000) 2024-03-06T11:04:09.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:09.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:09.646139+0000) 2024-03-06T11:04:10.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:10.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:10.646333+0000) 2024-03-06T11:04:11.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:11.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:11.646565+0000) 2024-03-06T11:04:12.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:12.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:12.646826+0000) 2024-03-06T11:04:13.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:13.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:13.647097+0000) 2024-03-06T11:04:14.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:14.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:14.647291+0000) 2024-03-06T11:04:15.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:15.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:15.647512+0000) 2024-03-06T11:04:16.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:16.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:16.647736+0000) 2024-03-06T11:04:17.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:17.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:17.647968+0000) 2024-03-06T11:04:18.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:18.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:18.648174+0000) 2024-03-06T11:04:19.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:19.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:19.648402+0000) 2024-03-06T11:04:20.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:20.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:20.648622+0000) 2024-03-06T11:04:21.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:21.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:21.648852+0000) 2024-03-06T11:04:22.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:22.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:22.649104+0000) 2024-03-06T11:04:23.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:23.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:23.649335+0000) 2024-03-06T11:04:24.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:24.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:24.649532+0000) 2024-03-06T11:04:25.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:25.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:25.649747+0000) 2024-03-06T11:04:26.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:26.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:26.649967+0000) 2024-03-06T11:04:27.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:27.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:27.650199+0000) 2024-03-06T11:04:28.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:28.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:28.650434+0000) 2024-03-06T11:04:29.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:29.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:29.650686+0000) 2024-03-06T11:04:30.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:30.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:30.650935+0000) 2024-03-06T11:04:31.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:31.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:31.651162+0000) 2024-03-06T11:04:32.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:32.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:32.651356+0000) 2024-03-06T11:04:33.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:33.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:33.651578+0000) 2024-03-06T11:04:34.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:34.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:34.651881+0000) 2024-03-06T11:04:35.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:35.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:35.652147+0000) 2024-03-06T11:04:36.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:36.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:36.652342+0000) 2024-03-06T11:04:36.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:36.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:04:37.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:37.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:37.652571+0000) 2024-03-06T11:04:38.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:38.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:38.652776+0000) 2024-03-06T11:04:39.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:39.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:39.653046+0000) 2024-03-06T11:04:40.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:40.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:40.653233+0000) 2024-03-06T11:04:41.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:41.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:41.653494+0000) 2024-03-06T11:04:42.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:42.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:42.653713+0000) 2024-03-06T11:04:43.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:43.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:43.653937+0000) 2024-03-06T11:04:44.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:44.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:44.654129+0000) 2024-03-06T11:04:45.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:45.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:45.654322+0000) 2024-03-06T11:04:46.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:46.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:46.654557+0000) 2024-03-06T11:04:47.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:47.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:47.654785+0000) 2024-03-06T11:04:48.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:48.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:48.654986+0000) 2024-03-06T11:04:49.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:49.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:49.655278+0000) 2024-03-06T11:04:50.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:50.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:50.655533+0000) 2024-03-06T11:04:51.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:51.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:51.655774+0000) 2024-03-06T11:04:52.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:52.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:52.655989+0000) 2024-03-06T11:04:53.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:53.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:53.656273+0000) 2024-03-06T11:04:54.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:54.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:54.656493+0000) 2024-03-06T11:04:55.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:55.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:55.656719+0000) 2024-03-06T11:04:56.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:56.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:56.656939+0000) 2024-03-06T11:04:57.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:57.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:57.657182+0000) 2024-03-06T11:04:58.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:58.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:58.657408+0000) 2024-03-06T11:04:59.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:04:59.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:04:59.657656+0000) 2024-03-06T11:05:00.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:00.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:00.657894+0000) 2024-03-06T11:05:01.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:01.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:01.658160+0000) 2024-03-06T11:05:02.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:02.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:02.658376+0000) 2024-03-06T11:05:03.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:03.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:03.658551+0000) 2024-03-06T11:05:04.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:04.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:04.658770+0000) 2024-03-06T11:05:05.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:05.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:05.659050+0000) 2024-03-06T11:05:06.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:06.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:06.659295+0000) 2024-03-06T11:05:07.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:07.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:07.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:07.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:07.659529+0000) 2024-03-06T11:05:08.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:08.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:08.659752+0000) 2024-03-06T11:05:09.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:09.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:09.660058+0000) 2024-03-06T11:05:10.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:10.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:10.660362+0000) 2024-03-06T11:05:11.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:11.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:11.660609+0000) 2024-03-06T11:05:12.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:12.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:12.660829+0000) 2024-03-06T11:05:13.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:13.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:13.661093+0000) 2024-03-06T11:05:14.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:14.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:14.661324+0000) 2024-03-06T11:05:15.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:15.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:15.661598+0000) 2024-03-06T11:05:16.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:16.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:16.661819+0000) 2024-03-06T11:05:17.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:17.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:17.662107+0000) 2024-03-06T11:05:18.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:18.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:18.662333+0000) 2024-03-06T11:05:19.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:19.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:19.662587+0000) 2024-03-06T11:05:20.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:20.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:20.662830+0000) 2024-03-06T11:05:21.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:21.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:21.663107+0000) 2024-03-06T11:05:22.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:22.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:22.663378+0000) 2024-03-06T11:05:23.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:23.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:23.663600+0000) 2024-03-06T11:05:24.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:24.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:24.663823+0000) 2024-03-06T11:05:25.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:25.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:25.664117+0000) 2024-03-06T11:05:26.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:26.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:26.664334+0000) 2024-03-06T11:05:27.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:27.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:27.664572+0000) 2024-03-06T11:05:28.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:28.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:28.664812+0000) 2024-03-06T11:05:29.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:29.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:29.664995+0000) 2024-03-06T11:05:30.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:30.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:30.665260+0000) 2024-03-06T11:05:31.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:31.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:31.665488+0000) 2024-03-06T11:05:32.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:32.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:32.665698+0000) 2024-03-06T11:05:33.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:33.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:33.665917+0000) 2024-03-06T11:05:34.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:34.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:34.666152+0000) 2024-03-06T11:05:35.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:35.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:35.666376+0000) 2024-03-06T11:05:36.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:36.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:36.666601+0000) 2024-03-06T11:05:37.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:37.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:05:37.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:37.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:37.666851+0000) 2024-03-06T11:05:38.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:38.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:38.667136+0000) 2024-03-06T11:05:39.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:39.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:39.667385+0000) 2024-03-06T11:05:40.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:40.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:40.667613+0000) 2024-03-06T11:05:41.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:41.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:41.667860+0000) 2024-03-06T11:05:42.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:42.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:42.668140+0000) 2024-03-06T11:05:43.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:43.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:43.668389+0000) 2024-03-06T11:05:44.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:44.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:44.668640+0000) 2024-03-06T11:05:45.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:45.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:45.668883+0000) 2024-03-06T11:05:46.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:46.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:46.669128+0000) 2024-03-06T11:05:47.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:47.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:47.669322+0000) 2024-03-06T11:05:48.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:48.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:48.669555+0000) 2024-03-06T11:05:49.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:49.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:49.669762+0000) 2024-03-06T11:05:50.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:50.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:50.669975+0000) 2024-03-06T11:05:51.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:51.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:51.670321+0000) 2024-03-06T11:05:52.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:52.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:52.670506+0000) 2024-03-06T11:05:53.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:53.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:53.670726+0000) 2024-03-06T11:05:54.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:54.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:54.670965+0000) 2024-03-06T11:05:55.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:55.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:55.671177+0000) 2024-03-06T11:05:56.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:56.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:56.671353+0000) 2024-03-06T11:05:57.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:57.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:57.671584+0000) 2024-03-06T11:05:58.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:58.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:58.671824+0000) 2024-03-06T11:05:59.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:05:59.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:05:59.672061+0000) 2024-03-06T11:06:00.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:00.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:00.672290+0000) 2024-03-06T11:06:01.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:01.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:01.672531+0000) 2024-03-06T11:06:02.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:02.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:02.672769+0000) 2024-03-06T11:06:03.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:03.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:03.673095+0000) 2024-03-06T11:06:04.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:04.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:04.673303+0000) 2024-03-06T11:06:05.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:05.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:05.673532+0000) 2024-03-06T11:06:06.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:06.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:06.673713+0000) 2024-03-06T11:06:07.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:07.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:07.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:07.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:07.673912+0000) 2024-03-06T11:06:08.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:08.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:08.674148+0000) 2024-03-06T11:06:09.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:09.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:09.674370+0000) 2024-03-06T11:06:10.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:10.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:10.674600+0000) 2024-03-06T11:06:11.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:11.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:11.674830+0000) 2024-03-06T11:06:12.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:12.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:12.675123+0000) 2024-03-06T11:06:13.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:13.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:13.675355+0000) 2024-03-06T11:06:14.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:14.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:14.675539+0000) 2024-03-06T11:06:15.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:15.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:15.675759+0000) 2024-03-06T11:06:16.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:16.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:16.675978+0000) 2024-03-06T11:06:17.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:17.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:17.676250+0000) 2024-03-06T11:06:18.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:18.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:18.676483+0000) 2024-03-06T11:06:19.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:19.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:19.676669+0000) 2024-03-06T11:06:20.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:20.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:20.676902+0000) 2024-03-06T11:06:21.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:21.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:21.677162+0000) 2024-03-06T11:06:22.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:22.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:22.677409+0000) 2024-03-06T11:06:23.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:23.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:23.677645+0000) 2024-03-06T11:06:24.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:24.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:24.677906+0000) 2024-03-06T11:06:25.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:25.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:25.678141+0000) 2024-03-06T11:06:26.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:26.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:26.678345+0000) 2024-03-06T11:06:27.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:27.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:27.678589+0000) 2024-03-06T11:06:28.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:28.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:28.678802+0000) 2024-03-06T11:06:29.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:29.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:29.679069+0000) 2024-03-06T11:06:30.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:30.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:30.679266+0000) 2024-03-06T11:06:31.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:31.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:31.679489+0000) 2024-03-06T11:06:32.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:32.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:32.679711+0000) 2024-03-06T11:06:33.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:33.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:33.679930+0000) 2024-03-06T11:06:34.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:34.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:34.680155+0000) 2024-03-06T11:06:35.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:35.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:35.680423+0000) 2024-03-06T11:06:36.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:36.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:36.680662+0000) 2024-03-06T11:06:37.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:37.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:06:37.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:37.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:37.680862+0000) 2024-03-06T11:06:38.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:38.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:38.681110+0000) 2024-03-06T11:06:39.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:39.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:39.681309+0000) 2024-03-06T11:06:40.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:40.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:40.681481+0000) 2024-03-06T11:06:41.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:41.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:41.681700+0000) 2024-03-06T11:06:42.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:42.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:42.681949+0000) 2024-03-06T11:06:43.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:43.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:43.682188+0000) 2024-03-06T11:06:44.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:44.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:44.682435+0000) 2024-03-06T11:06:45.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:45.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:45.682646+0000) 2024-03-06T11:06:46.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:46.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:46.682870+0000) 2024-03-06T11:06:47.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:47.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:47.683139+0000) 2024-03-06T11:06:48.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:48.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:48.683362+0000) 2024-03-06T11:06:49.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:49.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:49.683544+0000) 2024-03-06T11:06:50.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:50.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:50.683752+0000) 2024-03-06T11:06:51.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:51.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:51.683990+0000) 2024-03-06T11:06:52.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:52.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:52.684222+0000) 2024-03-06T11:06:53.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:53.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:53.684412+0000) 2024-03-06T11:06:54.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:54.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:54.684629+0000) 2024-03-06T11:06:55.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:55.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:55.684856+0000) 2024-03-06T11:06:56.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:56.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:56.685097+0000) 2024-03-06T11:06:57.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:57.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:57.685273+0000) 2024-03-06T11:06:58.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:58.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:58.685475+0000) 2024-03-06T11:06:59.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:06:59.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:06:59.685728+0000) 2024-03-06T11:07:00.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:00.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:00.685970+0000) 2024-03-06T11:07:01.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:01.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:01.686203+0000) 2024-03-06T11:07:02.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:02.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:02.686396+0000) 2024-03-06T11:07:03.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:03.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:03.686655+0000) 2024-03-06T11:07:04.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:04.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:04.686892+0000) 2024-03-06T11:07:05.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:05.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:05.687126+0000) 2024-03-06T11:07:06.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:06.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:06.687321+0000) 2024-03-06T11:07:07.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:07.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:07.687741+0000) 2024-03-06T11:07:07.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:07.805 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:08.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:08.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:08.688099+0000) 2024-03-06T11:07:09.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:09.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:09.688366+0000) 2024-03-06T11:07:10.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:10.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:10.688567+0000) 2024-03-06T11:07:11.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:11.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:11.688814+0000) 2024-03-06T11:07:12.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:12.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:12.689032+0000) 2024-03-06T11:07:13.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:13.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:13.689256+0000) 2024-03-06T11:07:14.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:14.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:14.689467+0000) 2024-03-06T11:07:15.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:15.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:15.689728+0000) 2024-03-06T11:07:16.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:16.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:16.689967+0000) 2024-03-06T11:07:17.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:17.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:17.690250+0000) 2024-03-06T11:07:18.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:18.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:18.690482+0000) 2024-03-06T11:07:19.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:19.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:19.690719+0000) 2024-03-06T11:07:20.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:20.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:20.690939+0000) 2024-03-06T11:07:21.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:21.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:21.691182+0000) 2024-03-06T11:07:22.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:22.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:22.691384+0000) 2024-03-06T11:07:23.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:23.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:23.691618+0000) 2024-03-06T11:07:24.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:24.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:24.691859+0000) 2024-03-06T11:07:25.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:25.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:25.692126+0000) 2024-03-06T11:07:26.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:26.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:26.692348+0000) 2024-03-06T11:07:27.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:27.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:27.692570+0000) 2024-03-06T11:07:28.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:28.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:28.692827+0000) 2024-03-06T11:07:29.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:29.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:29.693097+0000) 2024-03-06T11:07:30.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:30.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:30.693296+0000) 2024-03-06T11:07:31.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:31.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:31.693502+0000) 2024-03-06T11:07:32.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:32.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:32.693725+0000) 2024-03-06T11:07:33.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:33.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:33.693967+0000) 2024-03-06T11:07:34.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:34.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:34.694224+0000) 2024-03-06T11:07:35.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:35.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:35.694418+0000) 2024-03-06T11:07:36.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:36.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:36.694618+0000) 2024-03-06T11:07:37.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:37.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:37.694867+0000) 2024-03-06T11:07:38.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:38.017 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:07:38.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:38.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:38.695156+0000) 2024-03-06T11:07:39.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:39.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:39.695412+0000) 2024-03-06T11:07:40.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:40.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:40.695634+0000) 2024-03-06T11:07:41.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:41.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:41.695880+0000) 2024-03-06T11:07:42.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:42.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:42.696178+0000) 2024-03-06T11:07:43.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:43.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:43.696451+0000) 2024-03-06T11:07:44.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:44.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:44.696711+0000) 2024-03-06T11:07:45.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:45.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:45.696937+0000) 2024-03-06T11:07:46.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:46.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:46.697171+0000) 2024-03-06T11:07:47.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:47.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:47.697402+0000) 2024-03-06T11:07:48.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:48.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:48.697685+0000) 2024-03-06T11:07:49.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:49.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:49.697974+0000) 2024-03-06T11:07:50.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:50.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:50.698228+0000) 2024-03-06T11:07:51.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:51.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:51.698515+0000) 2024-03-06T11:07:52.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:52.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:52.698732+0000) 2024-03-06T11:07:53.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:53.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:53.698967+0000) 2024-03-06T11:07:54.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:54.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:54.699271+0000) 2024-03-06T11:07:55.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:55.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:55.699512+0000) 2024-03-06T11:07:56.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:56.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:56.699705+0000) 2024-03-06T11:07:57.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:57.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:57.699955+0000) 2024-03-06T11:07:58.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:58.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:58.700172+0000) 2024-03-06T11:07:59.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:07:59.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:07:59.700417+0000) 2024-03-06T11:08:00.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:00.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:00.700665+0000) 2024-03-06T11:08:01.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:01.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:01.700898+0000) 2024-03-06T11:08:02.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:02.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:02.701170+0000) 2024-03-06T11:08:03.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:03.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:03.701417+0000) 2024-03-06T11:08:04.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:04.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:04.701670+0000) 2024-03-06T11:08:05.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:05.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:05.701899+0000) 2024-03-06T11:08:06.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:06.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:06.702155+0000) 2024-03-06T11:08:07.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:07.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:07.702415+0000) 2024-03-06T11:08:08.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:08.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:08.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:08.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:08.702663+0000) 2024-03-06T11:08:09.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:09.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:09.702924+0000) 2024-03-06T11:08:10.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:10.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:10.703199+0000) 2024-03-06T11:08:11.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:11.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:11.703469+0000) 2024-03-06T11:08:12.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:12.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:12.703711+0000) 2024-03-06T11:08:13.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:13.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:13.703960+0000) 2024-03-06T11:08:14.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:14.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:14.704259+0000) 2024-03-06T11:08:15.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:15.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:15.704523+0000) 2024-03-06T11:08:16.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:16.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:16.704768+0000) 2024-03-06T11:08:17.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:17.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:17.705062+0000) 2024-03-06T11:08:18.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:18.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:18.705313+0000) 2024-03-06T11:08:19.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:19.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:19.705596+0000) 2024-03-06T11:08:20.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:20.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:20.705835+0000) 2024-03-06T11:08:21.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:21.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:21.706121+0000) 2024-03-06T11:08:22.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:22.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:22.706357+0000) 2024-03-06T11:08:23.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:23.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:23.706580+0000) 2024-03-06T11:08:24.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:24.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:24.706774+0000) 2024-03-06T11:08:25.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:25.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:25.706992+0000) 2024-03-06T11:08:26.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:26.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:26.707217+0000) 2024-03-06T11:08:27.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:27.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:27.707391+0000) 2024-03-06T11:08:28.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:28.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:28.707602+0000) 2024-03-06T11:08:29.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:29.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:29.707872+0000) 2024-03-06T11:08:30.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:30.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:30.708120+0000) 2024-03-06T11:08:31.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:31.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:31.708374+0000) 2024-03-06T11:08:32.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:32.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:32.708631+0000) 2024-03-06T11:08:33.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:33.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:33.708876+0000) 2024-03-06T11:08:34.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:34.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:34.709359+0000) 2024-03-06T11:08:35.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:35.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:35.709592+0000) 2024-03-06T11:08:36.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:36.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:36.709830+0000) 2024-03-06T11:08:37.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:37.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:37.710044+0000) 2024-03-06T11:08:38.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:38.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:08:38.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:38.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:38.710289+0000) 2024-03-06T11:08:39.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:39.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:39.710590+0000) 2024-03-06T11:08:40.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:40.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:40.710826+0000) 2024-03-06T11:08:41.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:41.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:41.711080+0000) 2024-03-06T11:08:42.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:42.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:42.711317+0000) 2024-03-06T11:08:43.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:43.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:43.711557+0000) 2024-03-06T11:08:44.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:44.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:44.711815+0000) 2024-03-06T11:08:45.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:45.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:45.712050+0000) 2024-03-06T11:08:46.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:46.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:46.712311+0000) 2024-03-06T11:08:47.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:47.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:47.712551+0000) 2024-03-06T11:08:48.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:48.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:48.712777+0000) 2024-03-06T11:08:49.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:49.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:49.713060+0000) 2024-03-06T11:08:50.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:50.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:50.713268+0000) 2024-03-06T11:08:51.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:51.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:51.713494+0000) 2024-03-06T11:08:52.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:52.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:52.713670+0000) 2024-03-06T11:08:53.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:53.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:53.713933+0000) 2024-03-06T11:08:54.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:54.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:54.714266+0000) 2024-03-06T11:08:55.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:55.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:55.714649+0000) 2024-03-06T11:08:56.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:56.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:56.714931+0000) 2024-03-06T11:08:57.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:57.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:57.715203+0000) 2024-03-06T11:08:58.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:58.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:58.715411+0000) 2024-03-06T11:08:59.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:08:59.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:08:59.715676+0000) 2024-03-06T11:09:00.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:00.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:00.715900+0000) 2024-03-06T11:09:01.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:01.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:01.716172+0000) 2024-03-06T11:09:02.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:02.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:02.716398+0000) 2024-03-06T11:09:03.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:03.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:03.716583+0000) 2024-03-06T11:09:04.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:04.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:04.716814+0000) 2024-03-06T11:09:05.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:05.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:05.717118+0000) 2024-03-06T11:09:06.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:06.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:06.717402+0000) 2024-03-06T11:09:07.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:07.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:07.717674+0000) 2024-03-06T11:09:08.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:08.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:08.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:08.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:08.717901+0000) 2024-03-06T11:09:09.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:09.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:09.718191+0000) 2024-03-06T11:09:10.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:10.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:10.718410+0000) 2024-03-06T11:09:11.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:11.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:11.718641+0000) 2024-03-06T11:09:12.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:12.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:12.718854+0000) 2024-03-06T11:09:13.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:13.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:13.719117+0000) 2024-03-06T11:09:14.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:14.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:14.719355+0000) 2024-03-06T11:09:15.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:15.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:15.719588+0000) 2024-03-06T11:09:16.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:16.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:16.719815+0000) 2024-03-06T11:09:17.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:17.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:17.720080+0000) 2024-03-06T11:09:18.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:18.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:18.720266+0000) 2024-03-06T11:09:19.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:19.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:19.720504+0000) 2024-03-06T11:09:20.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:20.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:20.720672+0000) 2024-03-06T11:09:21.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:21.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:21.720893+0000) 2024-03-06T11:09:22.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:22.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:22.721109+0000) 2024-03-06T11:09:23.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:23.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:23.721356+0000) 2024-03-06T11:09:24.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:24.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:24.721592+0000) 2024-03-06T11:09:25.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:25.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:25.721845+0000) 2024-03-06T11:09:26.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:26.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:26.722094+0000) 2024-03-06T11:09:27.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:27.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:27.722320+0000) 2024-03-06T11:09:28.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:28.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:28.722561+0000) 2024-03-06T11:09:29.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:29.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:29.722815+0000) 2024-03-06T11:09:30.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:30.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:30.723123+0000) 2024-03-06T11:09:31.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:31.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:31.723423+0000) 2024-03-06T11:09:32.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:32.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:32.723701+0000) 2024-03-06T11:09:33.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:33.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:33.724061+0000) 2024-03-06T11:09:34.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:34.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:34.724309+0000) 2024-03-06T11:09:35.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:35.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:35.724555+0000) 2024-03-06T11:09:36.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:36.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:36.724795+0000) 2024-03-06T11:09:37.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:37.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:37.725067+0000) 2024-03-06T11:09:38.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:38.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:38.725326+0000) 2024-03-06T11:09:38.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:38.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:09:39.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:39.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:39.725492+0000) 2024-03-06T11:09:40.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:40.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:40.725727+0000) 2024-03-06T11:09:41.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:41.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:41.725965+0000) 2024-03-06T11:09:42.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:42.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:42.726165+0000) 2024-03-06T11:09:43.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:43.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:43.726413+0000) 2024-03-06T11:09:44.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:44.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:44.726633+0000) 2024-03-06T11:09:45.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:45.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:45.726896+0000) 2024-03-06T11:09:46.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:46.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:46.727134+0000) 2024-03-06T11:09:47.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:47.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:47.727369+0000) 2024-03-06T11:09:48.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:48.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:48.727571+0000) 2024-03-06T11:09:49.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:49.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:49.727779+0000) 2024-03-06T11:09:50.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:50.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:50.728081+0000) 2024-03-06T11:09:51.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:51.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:51.728327+0000) 2024-03-06T11:09:52.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:52.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:52.728546+0000) 2024-03-06T11:09:53.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:53.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:53.728771+0000) 2024-03-06T11:09:54.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:54.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:54.729061+0000) 2024-03-06T11:09:55.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:55.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:55.729297+0000) 2024-03-06T11:09:56.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:56.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:56.729520+0000) 2024-03-06T11:09:57.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:57.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:57.729745+0000) 2024-03-06T11:09:58.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:58.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:58.729953+0000) 2024-03-06T11:09:59.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:09:59.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:09:59.730239+0000) 2024-03-06T11:10:00.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:00.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:00.730488+0000) 2024-03-06T11:10:01.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:01.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:01.730721+0000) 2024-03-06T11:10:02.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:02.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:02.730997+0000) 2024-03-06T11:10:03.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:03.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:03.731322+0000) 2024-03-06T11:10:04.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:04.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:04.731550+0000) 2024-03-06T11:10:05.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:05.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:05.731786+0000) 2024-03-06T11:10:06.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:06.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:06.732095+0000) 2024-03-06T11:10:07.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:07.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:07.732326+0000) 2024-03-06T11:10:08.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:08.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:08.732579+0000) 2024-03-06T11:10:09.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:09.027 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:09.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:09.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:09.732803+0000) 2024-03-06T11:10:10.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:10.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:10.732983+0000) 2024-03-06T11:10:11.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:11.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:11.733246+0000) 2024-03-06T11:10:12.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:12.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:12.733468+0000) 2024-03-06T11:10:13.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:13.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:13.733694+0000) 2024-03-06T11:10:14.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:14.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:14.733874+0000) 2024-03-06T11:10:15.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:15.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:15.734135+0000) 2024-03-06T11:10:16.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:16.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:16.734329+0000) 2024-03-06T11:10:17.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:17.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:17.734534+0000) 2024-03-06T11:10:18.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:18.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:18.734786+0000) 2024-03-06T11:10:19.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:19.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:19.734971+0000) 2024-03-06T11:10:20.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:20.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:20.735205+0000) 2024-03-06T11:10:21.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:21.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:21.735446+0000) 2024-03-06T11:10:22.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:22.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:22.735671+0000) 2024-03-06T11:10:23.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:23.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:23.735894+0000) 2024-03-06T11:10:24.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:24.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:24.736112+0000) 2024-03-06T11:10:25.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:25.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:25.736327+0000) 2024-03-06T11:10:26.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:26.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:26.736588+0000) 2024-03-06T11:10:27.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:27.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:27.736820+0000) 2024-03-06T11:10:28.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:28.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:28.737084+0000) 2024-03-06T11:10:29.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:29.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:29.737311+0000) 2024-03-06T11:10:30.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:30.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:30.737536+0000) 2024-03-06T11:10:31.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:31.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:31.737771+0000) 2024-03-06T11:10:32.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:32.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:32.737986+0000) 2024-03-06T11:10:33.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:33.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:33.738208+0000) 2024-03-06T11:10:34.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:34.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:34.738436+0000) 2024-03-06T11:10:35.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:35.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:35.738662+0000) 2024-03-06T11:10:36.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:36.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:36.738888+0000) 2024-03-06T11:10:37.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:37.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:37.739116+0000) 2024-03-06T11:10:38.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:38.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:38.739343+0000) 2024-03-06T11:10:39.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:39.231 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:10:39.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:39.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:39.739571+0000) 2024-03-06T11:10:40.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:40.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:40.739747+0000) 2024-03-06T11:10:41.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:41.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:41.740045+0000) 2024-03-06T11:10:42.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:42.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:42.740280+0000) 2024-03-06T11:10:43.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:43.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:43.740582+0000) 2024-03-06T11:10:44.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:44.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:44.740817+0000) 2024-03-06T11:10:45.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:45.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:45.741098+0000) 2024-03-06T11:10:46.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:46.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:46.741351+0000) 2024-03-06T11:10:47.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:47.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:47.741552+0000) 2024-03-06T11:10:48.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:48.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:48.741770+0000) 2024-03-06T11:10:49.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:49.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:49.741956+0000) 2024-03-06T11:10:50.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:50.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:50.742204+0000) 2024-03-06T11:10:51.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:51.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:51.742421+0000) 2024-03-06T11:10:52.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:52.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:52.742647+0000) 2024-03-06T11:10:53.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:53.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:53.742829+0000) 2024-03-06T11:10:54.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:54.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:54.743120+0000) 2024-03-06T11:10:55.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:55.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:55.743337+0000) 2024-03-06T11:10:56.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:56.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:56.743540+0000) 2024-03-06T11:10:57.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:57.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:57.743766+0000) 2024-03-06T11:10:58.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:58.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:58.743992+0000) 2024-03-06T11:10:59.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:10:59.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:10:59.744203+0000) 2024-03-06T11:11:00.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:00.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:00.744428+0000) 2024-03-06T11:11:01.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:01.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:01.744679+0000) 2024-03-06T11:11:02.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:02.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:02.744899+0000) 2024-03-06T11:11:03.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:03.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:03.745129+0000) 2024-03-06T11:11:04.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:04.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:04.745265+0000) 2024-03-06T11:11:05.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:05.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:05.745500+0000) 2024-03-06T11:11:06.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:06.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:06.745706+0000) 2024-03-06T11:11:07.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:07.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:07.745941+0000) 2024-03-06T11:11:08.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:08.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:08.746163+0000) 2024-03-06T11:11:09.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:09.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:09.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:09.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:09.746405+0000) 2024-03-06T11:11:10.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:10.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:10.746622+0000) 2024-03-06T11:11:11.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:11.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:11.746850+0000) 2024-03-06T11:11:12.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:12.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:12.747115+0000) 2024-03-06T11:11:13.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:13.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:13.747335+0000) 2024-03-06T11:11:14.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:14.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:14.747507+0000) 2024-03-06T11:11:15.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:15.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:15.747755+0000) 2024-03-06T11:11:16.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:16.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:16.747983+0000) 2024-03-06T11:11:17.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:17.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:17.748249+0000) 2024-03-06T11:11:18.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:18.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:18.748422+0000) 2024-03-06T11:11:19.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:19.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:19.748722+0000) 2024-03-06T11:11:20.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:20.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:20.748931+0000) 2024-03-06T11:11:21.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:21.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:21.749153+0000) 2024-03-06T11:11:22.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:22.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:22.749383+0000) 2024-03-06T11:11:23.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:23.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:23.749614+0000) 2024-03-06T11:11:24.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:24.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:24.749920+0000) 2024-03-06T11:11:25.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:25.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:25.750178+0000) 2024-03-06T11:11:26.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:26.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:26.750335+0000) 2024-03-06T11:11:27.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:27.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:27.750557+0000) 2024-03-06T11:11:28.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:28.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:28.750766+0000) 2024-03-06T11:11:29.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:29.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:29.750997+0000) 2024-03-06T11:11:30.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:30.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:30.751218+0000) 2024-03-06T11:11:31.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:31.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:31.751412+0000) 2024-03-06T11:11:32.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:32.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:32.751590+0000) 2024-03-06T11:11:33.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:33.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:33.751807+0000) 2024-03-06T11:11:34.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:34.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:34.752093+0000) 2024-03-06T11:11:35.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:35.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:35.752341+0000) 2024-03-06T11:11:36.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:36.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:36.752587+0000) 2024-03-06T11:11:37.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:37.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:37.752801+0000) 2024-03-06T11:11:38.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:38.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:38.753066+0000) 2024-03-06T11:11:39.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:39.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:11:39.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:39.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:39.753236+0000) 2024-03-06T11:11:40.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:40.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:40.753424+0000) 2024-03-06T11:11:41.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:41.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:41.753679+0000) 2024-03-06T11:11:42.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:42.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:42.753925+0000) 2024-03-06T11:11:43.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:43.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:43.754159+0000) 2024-03-06T11:11:44.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:44.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:44.754349+0000) 2024-03-06T11:11:45.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:45.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:45.754586+0000) 2024-03-06T11:11:46.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:46.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:46.754846+0000) 2024-03-06T11:11:47.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:47.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:47.755099+0000) 2024-03-06T11:11:48.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:48.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:48.755360+0000) 2024-03-06T11:11:49.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:49.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:49.755612+0000) 2024-03-06T11:11:50.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:50.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:50.755869+0000) 2024-03-06T11:11:51.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:51.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:51.756126+0000) 2024-03-06T11:11:52.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:52.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:52.756315+0000) 2024-03-06T11:11:53.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:53.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:53.756562+0000) 2024-03-06T11:11:54.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:54.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:54.756804+0000) 2024-03-06T11:11:55.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:55.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:55.757060+0000) 2024-03-06T11:11:56.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:56.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:56.757296+0000) 2024-03-06T11:11:57.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:57.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:57.757540+0000) 2024-03-06T11:11:58.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:58.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:58.757757+0000) 2024-03-06T11:11:59.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:11:59.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:11:59.757977+0000) 2024-03-06T11:12:00.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:00.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:00.758236+0000) 2024-03-06T11:12:01.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:01.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:01.758461+0000) 2024-03-06T11:12:02.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:02.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:02.758671+0000) 2024-03-06T11:12:03.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:03.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:03.758895+0000) 2024-03-06T11:12:04.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:04.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:04.759139+0000) 2024-03-06T11:12:05.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:05.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:05.759411+0000) 2024-03-06T11:12:06.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:06.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:06.759667+0000) 2024-03-06T11:12:07.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:07.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:07.759984+0000) 2024-03-06T11:12:08.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:08.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:08.760275+0000) 2024-03-06T11:12:09.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:09.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:09.760480+0000) 2024-03-06T11:12:09.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:09.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:10.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:10.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:10.760717+0000) 2024-03-06T11:12:11.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:11.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:11.760939+0000) 2024-03-06T11:12:12.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:12.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:12.761214+0000) 2024-03-06T11:12:13.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:13.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:13.761496+0000) 2024-03-06T11:12:14.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:14.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:14.761766+0000) 2024-03-06T11:12:15.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:15.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:15.762050+0000) 2024-03-06T11:12:16.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:16.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:16.762283+0000) 2024-03-06T11:12:17.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:17.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:17.762500+0000) 2024-03-06T11:12:18.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:18.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:18.762726+0000) 2024-03-06T11:12:19.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:19.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:19.762954+0000) 2024-03-06T11:12:20.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:20.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:20.763202+0000) 2024-03-06T11:12:21.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:21.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:21.763407+0000) 2024-03-06T11:12:22.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:22.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:22.763632+0000) 2024-03-06T11:12:23.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:23.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:23.763876+0000) 2024-03-06T11:12:24.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:24.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:24.764141+0000) 2024-03-06T11:12:25.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:25.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:25.764341+0000) 2024-03-06T11:12:26.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:26.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:26.764570+0000) 2024-03-06T11:12:27.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:27.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:27.764803+0000) 2024-03-06T11:12:28.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:28.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:28.765064+0000) 2024-03-06T11:12:29.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:29.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:29.765324+0000) 2024-03-06T11:12:30.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:30.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:30.765590+0000) 2024-03-06T11:12:31.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:31.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:31.765842+0000) 2024-03-06T11:12:32.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:32.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:32.766048+0000) 2024-03-06T11:12:33.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:33.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:33.766266+0000) 2024-03-06T11:12:34.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:34.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:34.766508+0000) 2024-03-06T11:12:35.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:35.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:35.766705+0000) 2024-03-06T11:12:36.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:36.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:36.766860+0000) 2024-03-06T11:12:37.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:37.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:37.767107+0000) 2024-03-06T11:12:38.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:38.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:38.767295+0000) 2024-03-06T11:12:39.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:39.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:39.767526+0000) 2024-03-06T11:12:40.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:40.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:12:40.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:40.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:40.767730+0000) 2024-03-06T11:12:41.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:41.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:41.767962+0000) 2024-03-06T11:12:42.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:42.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:42.768184+0000) 2024-03-06T11:12:43.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:43.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:43.768392+0000) 2024-03-06T11:12:44.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:44.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:44.768669+0000) 2024-03-06T11:12:45.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:45.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:45.768951+0000) 2024-03-06T11:12:46.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:46.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:46.769197+0000) 2024-03-06T11:12:47.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:47.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:47.769429+0000) 2024-03-06T11:12:48.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:48.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:48.769619+0000) 2024-03-06T11:12:49.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:49.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:49.769840+0000) 2024-03-06T11:12:50.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:50.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:50.770162+0000) 2024-03-06T11:12:51.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:51.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:51.770480+0000) 2024-03-06T11:12:52.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:52.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:52.770689+0000) 2024-03-06T11:12:53.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:53.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:53.770924+0000) 2024-03-06T11:12:54.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:54.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:54.771179+0000) 2024-03-06T11:12:55.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:55.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:55.771469+0000) 2024-03-06T11:12:56.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:56.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:56.771736+0000) 2024-03-06T11:12:57.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:57.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:57.772048+0000) 2024-03-06T11:12:58.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:58.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:58.772341+0000) 2024-03-06T11:12:59.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:12:59.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:12:59.772560+0000) 2024-03-06T11:13:00.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:00.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:00.772799+0000) 2024-03-06T11:13:01.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:01.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:01.773050+0000) 2024-03-06T11:13:02.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:02.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:02.773285+0000) 2024-03-06T11:13:03.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:03.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:03.773459+0000) 2024-03-06T11:13:04.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:04.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:04.773623+0000) 2024-03-06T11:13:05.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:05.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:05.773862+0000) 2024-03-06T11:13:06.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:06.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:06.774126+0000) 2024-03-06T11:13:07.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:07.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:07.774382+0000) 2024-03-06T11:13:08.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:08.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:08.774591+0000) 2024-03-06T11:13:09.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:09.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:09.774808+0000) 2024-03-06T11:13:10.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:10.368 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:10.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:10.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:10.775155+0000) 2024-03-06T11:13:11.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:11.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:11.775465+0000) 2024-03-06T11:13:12.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:12.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:12.775757+0000) 2024-03-06T11:13:13.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:13.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:13.776094+0000) 2024-03-06T11:13:14.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:14.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:14.776374+0000) 2024-03-06T11:13:15.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:15.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:15.776626+0000) 2024-03-06T11:13:16.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:16.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:16.776794+0000) 2024-03-06T11:13:17.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:17.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:17.776989+0000) 2024-03-06T11:13:18.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:18.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:18.777224+0000) 2024-03-06T11:13:19.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:19.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:19.777442+0000) 2024-03-06T11:13:20.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:20.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:20.777683+0000) 2024-03-06T11:13:21.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:21.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:21.777941+0000) 2024-03-06T11:13:22.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:22.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:22.778210+0000) 2024-03-06T11:13:23.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:23.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:23.778399+0000) 2024-03-06T11:13:24.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:24.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:24.778645+0000) 2024-03-06T11:13:25.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:25.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:25.778900+0000) 2024-03-06T11:13:26.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:26.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:26.779142+0000) 2024-03-06T11:13:27.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:27.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:27.779342+0000) 2024-03-06T11:13:28.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:28.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:28.779568+0000) 2024-03-06T11:13:29.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:29.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:29.779791+0000) 2024-03-06T11:13:30.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:30.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:30.780069+0000) 2024-03-06T11:13:31.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:31.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:31.780305+0000) 2024-03-06T11:13:32.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:32.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:32.780512+0000) 2024-03-06T11:13:33.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:33.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:33.780754+0000) 2024-03-06T11:13:34.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:34.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:34.780985+0000) 2024-03-06T11:13:35.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:35.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:35.781213+0000) 2024-03-06T11:13:36.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:36.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:36.781389+0000) 2024-03-06T11:13:37.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:37.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:37.781569+0000) 2024-03-06T11:13:38.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:38.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:38.781803+0000) 2024-03-06T11:13:39.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:39.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:39.782081+0000) 2024-03-06T11:13:40.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:40.510 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:13:40.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:40.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:40.782357+0000) 2024-03-06T11:13:41.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:41.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:41.782600+0000) 2024-03-06T11:13:42.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:42.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:42.782807+0000) 2024-03-06T11:13:43.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:43.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:43.782989+0000) 2024-03-06T11:13:44.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:44.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:44.783248+0000) 2024-03-06T11:13:45.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:45.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:45.783505+0000) 2024-03-06T11:13:46.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:46.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:46.783737+0000) 2024-03-06T11:13:47.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:47.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:47.783979+0000) 2024-03-06T11:13:48.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:48.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:48.784244+0000) 2024-03-06T11:13:49.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:49.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:49.784490+0000) 2024-03-06T11:13:50.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:50.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:50.784743+0000) 2024-03-06T11:13:51.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:51.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:51.784982+0000) 2024-03-06T11:13:52.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:52.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:52.785240+0000) 2024-03-06T11:13:53.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:53.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:53.785467+0000) 2024-03-06T11:13:54.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:54.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:54.785714+0000) 2024-03-06T11:13:55.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:55.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:55.785968+0000) 2024-03-06T11:13:56.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:56.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:56.786254+0000) 2024-03-06T11:13:57.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:57.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:57.786481+0000) 2024-03-06T11:13:58.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:58.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:58.786715+0000) 2024-03-06T11:13:59.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:13:59.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:13:59.786925+0000) 2024-03-06T11:14:00.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:00.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:00.787172+0000) 2024-03-06T11:14:01.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:01.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:01.787394+0000) 2024-03-06T11:14:02.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:02.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:02.787631+0000) 2024-03-06T11:14:03.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:03.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:03.787861+0000) 2024-03-06T11:14:04.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:04.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:04.788132+0000) 2024-03-06T11:14:05.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:05.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:05.788329+0000) 2024-03-06T11:14:06.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:06.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:06.788541+0000) 2024-03-06T11:14:07.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:07.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:07.788783+0000) 2024-03-06T11:14:08.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:08.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:08.789049+0000) 2024-03-06T11:14:09.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:09.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:09.789310+0000) 2024-03-06T11:14:10.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:10.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:10.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:10.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:10.789553+0000) 2024-03-06T11:14:11.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:11.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:11.789807+0000) 2024-03-06T11:14:12.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:12.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:12.789999+0000) 2024-03-06T11:14:13.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:13.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:13.790224+0000) 2024-03-06T11:14:14.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:14.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:14.790460+0000) 2024-03-06T11:14:15.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:15.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:15.790720+0000) 2024-03-06T11:14:16.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:16.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:16.790967+0000) 2024-03-06T11:14:17.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:17.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:17.791259+0000) 2024-03-06T11:14:18.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:18.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:18.791494+0000) 2024-03-06T11:14:19.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:19.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:19.791696+0000) 2024-03-06T11:14:20.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:20.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:20.791953+0000) 2024-03-06T11:14:21.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:21.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:21.792253+0000) 2024-03-06T11:14:22.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:22.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:22.792490+0000) 2024-03-06T11:14:23.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:23.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:23.792766+0000) 2024-03-06T11:14:24.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:24.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:24.792991+0000) 2024-03-06T11:14:25.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:25.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:25.793276+0000) 2024-03-06T11:14:26.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:26.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:26.793455+0000) 2024-03-06T11:14:27.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:27.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:27.793644+0000) 2024-03-06T11:14:28.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:28.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:28.793876+0000) 2024-03-06T11:14:29.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:29.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:29.794142+0000) 2024-03-06T11:14:30.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:30.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:30.794395+0000) 2024-03-06T11:14:31.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:31.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:31.794605+0000) 2024-03-06T11:14:32.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:32.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:32.794787+0000) 2024-03-06T11:14:33.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:33.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:33.795067+0000) 2024-03-06T11:14:34.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:34.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:34.795288+0000) 2024-03-06T11:14:35.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:35.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:35.795536+0000) 2024-03-06T11:14:36.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:36.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:36.795757+0000) 2024-03-06T11:14:37.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:37.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:37.796076+0000) 2024-03-06T11:14:38.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:38.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:38.796301+0000) 2024-03-06T11:14:39.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:39.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:39.796529+0000) 2024-03-06T11:14:40.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:40.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:40.796737+0000) 2024-03-06T11:14:40.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:40.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:14:41.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:41.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:41.796976+0000) 2024-03-06T11:14:42.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:42.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:42.797234+0000) 2024-03-06T11:14:43.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:43.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:43.797461+0000) 2024-03-06T11:14:44.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:44.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:44.797638+0000) 2024-03-06T11:14:45.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:45.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:45.797891+0000) 2024-03-06T11:14:46.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:46.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:46.798128+0000) 2024-03-06T11:14:47.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:47.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:47.798357+0000) 2024-03-06T11:14:48.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:48.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:48.798577+0000) 2024-03-06T11:14:49.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:49.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:49.798759+0000) 2024-03-06T11:14:50.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:50.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:50.799042+0000) 2024-03-06T11:14:51.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:51.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:51.799336+0000) 2024-03-06T11:14:52.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:52.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:52.799575+0000) 2024-03-06T11:14:53.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:53.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:53.799824+0000) 2024-03-06T11:14:54.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:54.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:54.800100+0000) 2024-03-06T11:14:55.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:55.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:55.800352+0000) 2024-03-06T11:14:56.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:56.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:56.800556+0000) 2024-03-06T11:14:57.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:57.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:57.800727+0000) 2024-03-06T11:14:58.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:58.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:58.800964+0000) 2024-03-06T11:14:59.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:14:59.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:14:59.801219+0000) 2024-03-06T11:15:00.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:00.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:00.801449+0000) 2024-03-06T11:15:01.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:01.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:01.801682+0000) 2024-03-06T11:15:02.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:02.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:02.801906+0000) 2024-03-06T11:15:03.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:03.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:03.802155+0000) 2024-03-06T11:15:04.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:04.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:04.802391+0000) 2024-03-06T11:15:05.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:05.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:05.802620+0000) 2024-03-06T11:15:06.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:06.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:06.802853+0000) 2024-03-06T11:15:07.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:07.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:07.803115+0000) 2024-03-06T11:15:08.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:08.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:08.803337+0000) 2024-03-06T11:15:09.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:09.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:09.803568+0000) 2024-03-06T11:15:10.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:10.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:10.803796+0000) 2024-03-06T11:15:11.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:11.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:11.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:11.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:11.803990+0000) 2024-03-06T11:15:12.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:12.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:12.804273+0000) 2024-03-06T11:15:13.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:13.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:13.804513+0000) 2024-03-06T11:15:14.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:14.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:14.804683+0000) 2024-03-06T11:15:15.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:15.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:15.804891+0000) 2024-03-06T11:15:16.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:16.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:16.805161+0000) 2024-03-06T11:15:17.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:17.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:17.805409+0000) 2024-03-06T11:15:18.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:18.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:18.805631+0000) 2024-03-06T11:15:19.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:19.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:19.805858+0000) 2024-03-06T11:15:20.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:20.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:20.806123+0000) 2024-03-06T11:15:21.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:21.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:21.806369+0000) 2024-03-06T11:15:22.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:22.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:22.806611+0000) 2024-03-06T11:15:23.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:23.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:23.806852+0000) 2024-03-06T11:15:24.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:24.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:24.807093+0000) 2024-03-06T11:15:25.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:25.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:25.807378+0000) 2024-03-06T11:15:26.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:26.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:26.807614+0000) 2024-03-06T11:15:27.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:27.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:27.807841+0000) 2024-03-06T11:15:28.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:28.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:28.808102+0000) 2024-03-06T11:15:29.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:29.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:29.808346+0000) 2024-03-06T11:15:30.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:30.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:30.808591+0000) 2024-03-06T11:15:31.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:31.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:31.808833+0000) 2024-03-06T11:15:32.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:32.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:32.809090+0000) 2024-03-06T11:15:33.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:33.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:33.809332+0000) 2024-03-06T11:15:34.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:34.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:34.809570+0000) 2024-03-06T11:15:35.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:35.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:35.809817+0000) 2024-03-06T11:15:36.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:36.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:36.810102+0000) 2024-03-06T11:15:37.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:37.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:37.810354+0000) 2024-03-06T11:15:38.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:38.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:38.810576+0000) 2024-03-06T11:15:39.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:39.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:39.810873+0000) 2024-03-06T11:15:40.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:40.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:40.811093+0000) 2024-03-06T11:15:41.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:41.270 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:15:41.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:41.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:41.811333+0000) 2024-03-06T11:15:42.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:42.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:42.811531+0000) 2024-03-06T11:15:43.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:43.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:43.811763+0000) 2024-03-06T11:15:44.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:44.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:44.811991+0000) 2024-03-06T11:15:45.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:45.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:45.812251+0000) 2024-03-06T11:15:46.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:46.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:46.812432+0000) 2024-03-06T11:15:47.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:47.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:47.812677+0000) 2024-03-06T11:15:48.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:48.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:48.812915+0000) 2024-03-06T11:15:49.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:49.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:49.813166+0000) 2024-03-06T11:15:50.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:50.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:50.813408+0000) 2024-03-06T11:15:51.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:51.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:51.813631+0000) 2024-03-06T11:15:52.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:52.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:52.813847+0000) 2024-03-06T11:15:53.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:53.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:53.814096+0000) 2024-03-06T11:15:54.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:54.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:54.814319+0000) 2024-03-06T11:15:55.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:55.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:55.814521+0000) 2024-03-06T11:15:56.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:56.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:56.814761+0000) 2024-03-06T11:15:57.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:57.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:57.815054+0000) 2024-03-06T11:15:58.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:58.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:58.815302+0000) 2024-03-06T11:15:59.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:15:59.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:15:59.815537+0000) 2024-03-06T11:16:00.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:00.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:00.815795+0000) 2024-03-06T11:16:01.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:01.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:01.816089+0000) 2024-03-06T11:16:02.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:02.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:02.816291+0000) 2024-03-06T11:16:03.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:03.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:03.816492+0000) 2024-03-06T11:16:04.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:04.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:04.816715+0000) 2024-03-06T11:16:05.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:05.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:05.816955+0000) 2024-03-06T11:16:06.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:06.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:06.817195+0000) 2024-03-06T11:16:07.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:07.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:07.817442+0000) 2024-03-06T11:16:08.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:08.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:08.817680+0000) 2024-03-06T11:16:09.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:09.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:09.817921+0000) 2024-03-06T11:16:10.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:10.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:10.818177+0000) 2024-03-06T11:16:11.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:11.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:11.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:11.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:11.818423+0000) 2024-03-06T11:16:12.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:12.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:12.818638+0000) 2024-03-06T11:16:13.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:13.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:13.818868+0000) 2024-03-06T11:16:14.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:14.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:14.819116+0000) 2024-03-06T11:16:15.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:15.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:15.819345+0000) 2024-03-06T11:16:16.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:16.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:16.819584+0000) 2024-03-06T11:16:17.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:17.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:17.819829+0000) 2024-03-06T11:16:18.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:18.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:18.820116+0000) 2024-03-06T11:16:19.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:19.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:19.820352+0000) 2024-03-06T11:16:20.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:20.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:20.820579+0000) 2024-03-06T11:16:21.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:21.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:21.820786+0000) 2024-03-06T11:16:22.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:22.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:22.820968+0000) 2024-03-06T11:16:23.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:23.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:23.821259+0000) 2024-03-06T11:16:24.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:24.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:24.821492+0000) 2024-03-06T11:16:25.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:25.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:25.821706+0000) 2024-03-06T11:16:26.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:26.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:26.821903+0000) 2024-03-06T11:16:27.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:27.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:27.822154+0000) 2024-03-06T11:16:28.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:28.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:28.822345+0000) 2024-03-06T11:16:29.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:29.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:29.822579+0000) 2024-03-06T11:16:30.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:30.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:30.822815+0000) 2024-03-06T11:16:31.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:31.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:31.823119+0000) 2024-03-06T11:16:32.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:32.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:32.823355+0000) 2024-03-06T11:16:33.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:33.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:33.823608+0000) 2024-03-06T11:16:34.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:34.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:34.823819+0000) 2024-03-06T11:16:35.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:35.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:35.824106+0000) 2024-03-06T11:16:36.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:36.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:36.824311+0000) 2024-03-06T11:16:37.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:37.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:37.824510+0000) 2024-03-06T11:16:38.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:38.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:38.824728+0000) 2024-03-06T11:16:39.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:39.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:39.824950+0000) 2024-03-06T11:16:40.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:40.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:40.825241+0000) 2024-03-06T11:16:41.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:41.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:16:41.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:41.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:41.825557+0000) 2024-03-06T11:16:42.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:42.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:42.825830+0000) 2024-03-06T11:16:43.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:43.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:43.826154+0000) 2024-03-06T11:16:44.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:44.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:44.826417+0000) 2024-03-06T11:16:45.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:45.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:45.826693+0000) 2024-03-06T11:16:46.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:46.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:46.826909+0000) 2024-03-06T11:16:47.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:47.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:47.827155+0000) 2024-03-06T11:16:48.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:48.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:48.827378+0000) 2024-03-06T11:16:49.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:49.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:49.827586+0000) 2024-03-06T11:16:50.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:50.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:50.827814+0000) 2024-03-06T11:16:51.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:51.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:51.828060+0000) 2024-03-06T11:16:52.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:52.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:52.828253+0000) 2024-03-06T11:16:53.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:53.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:53.828515+0000) 2024-03-06T11:16:54.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:54.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:54.828798+0000) 2024-03-06T11:16:55.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:55.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:55.829138+0000) 2024-03-06T11:16:56.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:56.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:56.829365+0000) 2024-03-06T11:16:57.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:57.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:57.829560+0000) 2024-03-06T11:16:58.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:58.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:58.829796+0000) 2024-03-06T11:16:59.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:16:59.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:16:59.830051+0000) 2024-03-06T11:17:00.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:00.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:00.830298+0000) 2024-03-06T11:17:01.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:01.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:01.830606+0000) 2024-03-06T11:17:02.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:02.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:02.830855+0000) 2024-03-06T11:17:03.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:03.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:03.831119+0000) 2024-03-06T11:17:04.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:04.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:04.831340+0000) 2024-03-06T11:17:05.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:05.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:05.831568+0000) 2024-03-06T11:17:06.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:06.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:06.831814+0000) 2024-03-06T11:17:07.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:07.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:07.832107+0000) 2024-03-06T11:17:08.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:08.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:08.832281+0000) 2024-03-06T11:17:09.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:09.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:09.832473+0000) 2024-03-06T11:17:10.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:10.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:10.832688+0000) 2024-03-06T11:17:11.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:11.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:11.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:11.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:11.832887+0000) 2024-03-06T11:17:12.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:12.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:12.833134+0000) 2024-03-06T11:17:13.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:13.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:13.833391+0000) 2024-03-06T11:17:14.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:14.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:14.833633+0000) 2024-03-06T11:17:15.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:15.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:15.833877+0000) 2024-03-06T11:17:16.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:16.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:16.834114+0000) 2024-03-06T11:17:17.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:17.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:17.834359+0000) 2024-03-06T11:17:18.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:18.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:18.834596+0000) 2024-03-06T11:17:19.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:19.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:19.834858+0000) 2024-03-06T11:17:20.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:20.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:20.835132+0000) 2024-03-06T11:17:21.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:21.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:21.835340+0000) 2024-03-06T11:17:22.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:22.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:22.835520+0000) 2024-03-06T11:17:23.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:23.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:23.835774+0000) 2024-03-06T11:17:24.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:24.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:24.836058+0000) 2024-03-06T11:17:25.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:25.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:25.836274+0000) 2024-03-06T11:17:26.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:26.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:26.836509+0000) 2024-03-06T11:17:27.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:27.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:27.836748+0000) 2024-03-06T11:17:28.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:28.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:28.837061+0000) 2024-03-06T11:17:29.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:29.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:29.837269+0000) 2024-03-06T11:17:30.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:30.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:30.837513+0000) 2024-03-06T11:17:31.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:31.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:31.837767+0000) 2024-03-06T11:17:32.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:32.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:32.838046+0000) 2024-03-06T11:17:33.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:33.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:33.838319+0000) 2024-03-06T11:17:34.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:34.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:34.838608+0000) 2024-03-06T11:17:35.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:35.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:35.838808+0000) 2024-03-06T11:17:36.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:36.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:36.839072+0000) 2024-03-06T11:17:37.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:37.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:37.839356+0000) 2024-03-06T11:17:38.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:38.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:38.839600+0000) 2024-03-06T11:17:39.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:39.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:39.839828+0000) 2024-03-06T11:17:40.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:40.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:40.840129+0000) 2024-03-06T11:17:41.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:41.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:41.840431+0000) 2024-03-06T11:17:42.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:42.046 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:17:42.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:42.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:42.840670+0000) 2024-03-06T11:17:43.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:43.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:43.840949+0000) 2024-03-06T11:17:44.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:44.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:44.841168+0000) 2024-03-06T11:17:45.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:45.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:45.841406+0000) 2024-03-06T11:17:46.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:46.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:46.841655+0000) 2024-03-06T11:17:47.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:47.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:47.841901+0000) 2024-03-06T11:17:48.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:48.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:48.842119+0000) 2024-03-06T11:17:49.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:49.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:49.842297+0000) 2024-03-06T11:17:50.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:50.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:50.842532+0000) 2024-03-06T11:17:51.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:51.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:51.842782+0000) 2024-03-06T11:17:52.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:52.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:52.843047+0000) 2024-03-06T11:17:53.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:53.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:53.843291+0000) 2024-03-06T11:17:54.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:54.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:54.843584+0000) 2024-03-06T11:17:55.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:55.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:55.843830+0000) 2024-03-06T11:17:56.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:56.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:56.844105+0000) 2024-03-06T11:17:57.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:57.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:57.844331+0000) 2024-03-06T11:17:58.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:58.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:58.844544+0000) 2024-03-06T11:17:59.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:17:59.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:17:59.844753+0000) 2024-03-06T11:18:00.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:00.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:00.844976+0000) 2024-03-06T11:18:01.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:01.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:01.845275+0000) 2024-03-06T11:18:02.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:02.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:02.845519+0000) 2024-03-06T11:18:03.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:03.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:03.845767+0000) 2024-03-06T11:18:04.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:04.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:04.846048+0000) 2024-03-06T11:18:05.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:05.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:05.846306+0000) 2024-03-06T11:18:06.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:06.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:06.846534+0000) 2024-03-06T11:18:07.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:07.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:07.846756+0000) 2024-03-06T11:18:08.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:08.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:08.846980+0000) 2024-03-06T11:18:09.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:09.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:09.847247+0000) 2024-03-06T11:18:10.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:10.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:10.847465+0000) 2024-03-06T11:18:11.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:11.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:11.847736+0000) 2024-03-06T11:18:12.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:12.269 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:12.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:12.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:12.848046+0000) 2024-03-06T11:18:13.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:13.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:13.848329+0000) 2024-03-06T11:18:14.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:14.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:14.848606+0000) 2024-03-06T11:18:15.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:15.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:15.848823+0000) 2024-03-06T11:18:16.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:16.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:16.849068+0000) 2024-03-06T11:18:17.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:17.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:17.849319+0000) 2024-03-06T11:18:18.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:18.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:18.849548+0000) 2024-03-06T11:18:19.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:19.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:19.849751+0000) 2024-03-06T11:18:20.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:20.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:20.849991+0000) 2024-03-06T11:18:21.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:21.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:21.850313+0000) 2024-03-06T11:18:22.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:22.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:22.850585+0000) 2024-03-06T11:18:23.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:23.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:23.850865+0000) 2024-03-06T11:18:24.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:24.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:24.851169+0000) 2024-03-06T11:18:25.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:25.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:25.851418+0000) 2024-03-06T11:18:26.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:26.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:26.851653+0000) 2024-03-06T11:18:27.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:27.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:27.851885+0000) 2024-03-06T11:18:28.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:28.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:28.852118+0000) 2024-03-06T11:18:29.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:29.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:29.852351+0000) 2024-03-06T11:18:30.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:30.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:30.852560+0000) 2024-03-06T11:18:31.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:31.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:31.852808+0000) 2024-03-06T11:18:32.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:32.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:32.853086+0000) 2024-03-06T11:18:33.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:33.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:33.853316+0000) 2024-03-06T11:18:34.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:34.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:34.853493+0000) 2024-03-06T11:18:35.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:35.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:35.853745+0000) 2024-03-06T11:18:36.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:36.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:36.853939+0000) 2024-03-06T11:18:37.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:37.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:37.854118+0000) 2024-03-06T11:18:38.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:38.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:38.854365+0000) 2024-03-06T11:18:39.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:39.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:39.854613+0000) 2024-03-06T11:18:40.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:40.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:40.854852+0000) 2024-03-06T11:18:41.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:41.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:41.855147+0000) 2024-03-06T11:18:42.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:42.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:18:42.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:42.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:42.855391+0000) 2024-03-06T11:18:43.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:43.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:43.855582+0000) 2024-03-06T11:18:44.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:44.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:44.855824+0000) 2024-03-06T11:18:45.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:45.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:45.856049+0000) 2024-03-06T11:18:46.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:46.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:46.856301+0000) 2024-03-06T11:18:47.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:47.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:47.856511+0000) 2024-03-06T11:18:48.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:48.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:48.856738+0000) 2024-03-06T11:18:49.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:49.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:49.856943+0000) 2024-03-06T11:18:50.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:50.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:50.857152+0000) 2024-03-06T11:18:51.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:51.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:51.857416+0000) 2024-03-06T11:18:52.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:52.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:52.857616+0000) 2024-03-06T11:18:53.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:53.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:53.857842+0000) 2024-03-06T11:18:54.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:54.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:54.858103+0000) 2024-03-06T11:18:55.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:55.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:55.858314+0000) 2024-03-06T11:18:56.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:56.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:56.858553+0000) 2024-03-06T11:18:57.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:57.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:57.858783+0000) 2024-03-06T11:18:58.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:58.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:58.859036+0000) 2024-03-06T11:18:59.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:18:59.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:18:59.859246+0000) 2024-03-06T11:19:00.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:00.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:00.859482+0000) 2024-03-06T11:19:01.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:01.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:01.859744+0000) 2024-03-06T11:19:02.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:02.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:02.859988+0000) 2024-03-06T11:19:03.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:03.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:03.860197+0000) 2024-03-06T11:19:04.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:04.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:04.860424+0000) 2024-03-06T11:19:05.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:05.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:05.860666+0000) 2024-03-06T11:19:06.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:06.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:06.860920+0000) 2024-03-06T11:19:07.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:07.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:07.861178+0000) 2024-03-06T11:19:08.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:08.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:08.861415+0000) 2024-03-06T11:19:09.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:09.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:09.861654+0000) 2024-03-06T11:19:10.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:10.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:10.861843+0000) 2024-03-06T11:19:11.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:11.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:11.862152+0000) 2024-03-06T11:19:12.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:12.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:12.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:12.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:12.862417+0000) 2024-03-06T11:19:13.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:13.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:13.862620+0000) 2024-03-06T11:19:14.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:14.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:14.862834+0000) 2024-03-06T11:19:15.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:15.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:15.863146+0000) 2024-03-06T11:19:16.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:16.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:16.863429+0000) 2024-03-06T11:19:17.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:17.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:17.863719+0000) 2024-03-06T11:19:18.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:18.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:18.863949+0000) 2024-03-06T11:19:19.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:19.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:19.864210+0000) 2024-03-06T11:19:20.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:20.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:20.864447+0000) 2024-03-06T11:19:21.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:21.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:21.864685+0000) 2024-03-06T11:19:22.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:22.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:22.864886+0000) 2024-03-06T11:19:23.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:23.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:23.865154+0000) 2024-03-06T11:19:24.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:24.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:24.865386+0000) 2024-03-06T11:19:25.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:25.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:25.865604+0000) 2024-03-06T11:19:26.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:26.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:26.865827+0000) 2024-03-06T11:19:27.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:27.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:27.866099+0000) 2024-03-06T11:19:28.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:28.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:28.866319+0000) 2024-03-06T11:19:29.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:29.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:29.866550+0000) 2024-03-06T11:19:30.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:30.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:30.866764+0000) 2024-03-06T11:19:31.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:31.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:31.867062+0000) 2024-03-06T11:19:32.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:32.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:32.867301+0000) 2024-03-06T11:19:33.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:33.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:33.867531+0000) 2024-03-06T11:19:34.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:34.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:34.867754+0000) 2024-03-06T11:19:35.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:35.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:35.868114+0000) 2024-03-06T11:19:36.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:36.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:36.868358+0000) 2024-03-06T11:19:37.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:37.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:37.868569+0000) 2024-03-06T11:19:38.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:38.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:38.868805+0000) 2024-03-06T11:19:39.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:39.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:39.869092+0000) 2024-03-06T11:19:40.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:40.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:40.869324+0000) 2024-03-06T11:19:41.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:41.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:41.869602+0000) 2024-03-06T11:19:42.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:42.755 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:19:42.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:42.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:42.869851+0000) 2024-03-06T11:19:43.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:43.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:43.870119+0000) 2024-03-06T11:19:44.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:44.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:44.870341+0000) 2024-03-06T11:19:45.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:45.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:45.870570+0000) 2024-03-06T11:19:46.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:46.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:46.870811+0000) 2024-03-06T11:19:47.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:47.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:47.871067+0000) 2024-03-06T11:19:48.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:48.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:48.871332+0000) 2024-03-06T11:19:49.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:49.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:49.871597+0000) 2024-03-06T11:19:50.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:50.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:50.871879+0000) 2024-03-06T11:19:51.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:51.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:51.872184+0000) 2024-03-06T11:19:52.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:52.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:52.872468+0000) 2024-03-06T11:19:53.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:53.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:53.872739+0000) 2024-03-06T11:19:54.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:54.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:54.873058+0000) 2024-03-06T11:19:55.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:55.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:55.873237+0000) 2024-03-06T11:19:56.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:56.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:56.873521+0000) 2024-03-06T11:19:57.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:57.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:57.873797+0000) 2024-03-06T11:19:58.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:58.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:58.874079+0000) 2024-03-06T11:19:59.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:19:59.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:19:59.874282+0000) 2024-03-06T11:20:00.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:00.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:00.874508+0000) 2024-03-06T11:20:01.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:01.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:01.874743+0000) 2024-03-06T11:20:02.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:02.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:02.875032+0000) 2024-03-06T11:20:03.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:03.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:03.875282+0000) 2024-03-06T11:20:04.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:04.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:04.875500+0000) 2024-03-06T11:20:05.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:05.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:05.875762+0000) 2024-03-06T11:20:06.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:06.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:06.876029+0000) 2024-03-06T11:20:07.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:07.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:07.876260+0000) 2024-03-06T11:20:08.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:08.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:08.876473+0000) 2024-03-06T11:20:09.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:09.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:09.876705+0000) 2024-03-06T11:20:10.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:10.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:10.876924+0000) 2024-03-06T11:20:11.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:11.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:11.877235+0000) 2024-03-06T11:20:12.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:12.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:12.877515+0000) 2024-03-06T11:20:12.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:13.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:13.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:13.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:13.877794+0000) 2024-03-06T11:20:14.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:14.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:14.878070+0000) 2024-03-06T11:20:15.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:15.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:15.878313+0000) 2024-03-06T11:20:16.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:16.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:16.878610+0000) 2024-03-06T11:20:17.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:17.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:17.878856+0000) 2024-03-06T11:20:18.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:18.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:18.879131+0000) 2024-03-06T11:20:19.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:19.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:19.879376+0000) 2024-03-06T11:20:20.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:20.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:20.879606+0000) 2024-03-06T11:20:21.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:21.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:21.879893+0000) 2024-03-06T11:20:22.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:22.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:22.880166+0000) 2024-03-06T11:20:23.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:23.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:23.880505+0000) 2024-03-06T11:20:24.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:24.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:24.880710+0000) 2024-03-06T11:20:25.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:25.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:25.880953+0000) 2024-03-06T11:20:26.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:26.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:26.881212+0000) 2024-03-06T11:20:27.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:27.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:27.881461+0000) 2024-03-06T11:20:28.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:28.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:28.881698+0000) 2024-03-06T11:20:29.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:29.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:29.881925+0000) 2024-03-06T11:20:30.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:30.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:30.882199+0000) 2024-03-06T11:20:31.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:31.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:31.882484+0000) 2024-03-06T11:20:32.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:32.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:32.882708+0000) 2024-03-06T11:20:33.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:33.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:33.882935+0000) 2024-03-06T11:20:34.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:34.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:34.883166+0000) 2024-03-06T11:20:35.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:35.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:35.883395+0000) 2024-03-06T11:20:36.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:36.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:36.883639+0000) 2024-03-06T11:20:37.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:37.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:37.883827+0000) 2024-03-06T11:20:38.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:38.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:38.884086+0000) 2024-03-06T11:20:39.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:39.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:39.884273+0000) 2024-03-06T11:20:40.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:40.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:40.884534+0000) 2024-03-06T11:20:41.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:41.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:41.884753+0000) 2024-03-06T11:20:42.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:42.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:42.884999+0000) 2024-03-06T11:20:43.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:43.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:20:43.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:43.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:43.885306+0000) 2024-03-06T11:20:44.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:44.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:44.885536+0000) 2024-03-06T11:20:45.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:45.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:45.885783+0000) 2024-03-06T11:20:46.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:46.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:46.886051+0000) 2024-03-06T11:20:47.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:47.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:47.886300+0000) 2024-03-06T11:20:48.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:48.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:48.886532+0000) 2024-03-06T11:20:49.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:49.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:49.886775+0000) 2024-03-06T11:20:50.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:50.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:50.887046+0000) 2024-03-06T11:20:51.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:51.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:51.887324+0000) 2024-03-06T11:20:52.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:52.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:52.887514+0000) 2024-03-06T11:20:53.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:53.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:53.887733+0000) 2024-03-06T11:20:54.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:54.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:54.887950+0000) 2024-03-06T11:20:55.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:55.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:55.888176+0000) 2024-03-06T11:20:56.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:56.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:56.888418+0000) 2024-03-06T11:20:57.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:57.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:57.888581+0000) 2024-03-06T11:20:58.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:58.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:58.888788+0000) 2024-03-06T11:20:59.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:20:59.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:20:59.889125+0000) 2024-03-06T11:21:00.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:00.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:00.889368+0000) 2024-03-06T11:21:01.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:01.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:01.889656+0000) 2024-03-06T11:21:02.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:02.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:02.889932+0000) 2024-03-06T11:21:03.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:03.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:03.890179+0000) 2024-03-06T11:21:04.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:04.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:04.890362+0000) 2024-03-06T11:21:05.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:05.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:05.890603+0000) 2024-03-06T11:21:06.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:06.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:06.890849+0000) 2024-03-06T11:21:07.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:07.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:07.891116+0000) 2024-03-06T11:21:08.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:08.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:08.891328+0000) 2024-03-06T11:21:09.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:09.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:09.891570+0000) 2024-03-06T11:21:10.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:10.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:10.891790+0000) 2024-03-06T11:21:11.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:11.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:11.892083+0000) 2024-03-06T11:21:12.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:12.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:12.892322+0000) 2024-03-06T11:21:13.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:13.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:13.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:13.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:13.892524+0000) 2024-03-06T11:21:14.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:14.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:14.892764+0000) 2024-03-06T11:21:15.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:15.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:15.892979+0000) 2024-03-06T11:21:16.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:16.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:16.893264+0000) 2024-03-06T11:21:17.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:17.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:17.893555+0000) 2024-03-06T11:21:18.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:18.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:18.893758+0000) 2024-03-06T11:21:19.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:19.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:19.893985+0000) 2024-03-06T11:21:20.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:20.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:20.894293+0000) 2024-03-06T11:21:21.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:21.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:21.894556+0000) 2024-03-06T11:21:22.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:22.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:22.894811+0000) 2024-03-06T11:21:23.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:23.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:23.895106+0000) 2024-03-06T11:21:24.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:24.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:24.895343+0000) 2024-03-06T11:21:25.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:25.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:25.895586+0000) 2024-03-06T11:21:26.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:26.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:26.895819+0000) 2024-03-06T11:21:27.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:27.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:27.896124+0000) 2024-03-06T11:21:28.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:28.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:28.896366+0000) 2024-03-06T11:21:29.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:29.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:29.896616+0000) 2024-03-06T11:21:30.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:30.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:30.896865+0000) 2024-03-06T11:21:31.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:31.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:31.897120+0000) 2024-03-06T11:21:32.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:32.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:32.897347+0000) 2024-03-06T11:21:33.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:33.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:33.897622+0000) 2024-03-06T11:21:34.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:34.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:34.897902+0000) 2024-03-06T11:21:35.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:35.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:35.898182+0000) 2024-03-06T11:21:36.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:36.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:36.898470+0000) 2024-03-06T11:21:37.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:37.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:37.898720+0000) 2024-03-06T11:21:38.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:38.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:38.898945+0000) 2024-03-06T11:21:39.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:39.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:39.899174+0000) 2024-03-06T11:21:40.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:40.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:40.899393+0000) 2024-03-06T11:21:41.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:41.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:41.899662+0000) 2024-03-06T11:21:42.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:42.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:42.899858+0000) 2024-03-06T11:21:43.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:43.576 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:21:43.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:43.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:43.900152+0000) 2024-03-06T11:21:44.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:44.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:44.900370+0000) 2024-03-06T11:21:45.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:45.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:45.900594+0000) 2024-03-06T11:21:46.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:46.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:46.900881+0000) 2024-03-06T11:21:47.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:47.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:47.901160+0000) 2024-03-06T11:21:48.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:48.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:48.901440+0000) 2024-03-06T11:21:49.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:49.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:49.901711+0000) 2024-03-06T11:21:50.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:50.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:50.901897+0000) 2024-03-06T11:21:51.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:51.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:51.902129+0000) 2024-03-06T11:21:52.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:52.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:52.902319+0000) 2024-03-06T11:21:53.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:53.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:53.902524+0000) 2024-03-06T11:21:54.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:54.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:54.902765+0000) 2024-03-06T11:21:55.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:55.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:55.903050+0000) 2024-03-06T11:21:56.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:56.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:56.903273+0000) 2024-03-06T11:21:57.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:57.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:57.903491+0000) 2024-03-06T11:21:58.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:58.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:58.903709+0000) 2024-03-06T11:21:59.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:21:59.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:21:59.903934+0000) 2024-03-06T11:22:00.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:00.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:00.904139+0000) 2024-03-06T11:22:01.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:01.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:01.904339+0000) 2024-03-06T11:22:02.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:02.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:02.904563+0000) 2024-03-06T11:22:03.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:03.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:03.904837+0000) 2024-03-06T11:22:04.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:04.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:04.905180+0000) 2024-03-06T11:22:05.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:05.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:05.905514+0000) 2024-03-06T11:22:06.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:06.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:06.905798+0000) 2024-03-06T11:22:07.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:07.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:07.906101+0000) 2024-03-06T11:22:08.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:08.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:08.906344+0000) 2024-03-06T11:22:09.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:09.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:09.906611+0000) 2024-03-06T11:22:10.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:10.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:10.906883+0000) 2024-03-06T11:22:11.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:11.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:11.907205+0000) 2024-03-06T11:22:12.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:12.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:12.907464+0000) 2024-03-06T11:22:13.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:13.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:13.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:13.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:13.907718+0000) 2024-03-06T11:22:14.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:14.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:14.907937+0000) 2024-03-06T11:22:15.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:15.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:15.908211+0000) 2024-03-06T11:22:16.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:16.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:16.908484+0000) 2024-03-06T11:22:17.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:17.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:17.908692+0000) 2024-03-06T11:22:18.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:18.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:18.908924+0000) 2024-03-06T11:22:19.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:19.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:19.909155+0000) 2024-03-06T11:22:20.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:20.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:20.909338+0000) 2024-03-06T11:22:21.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:21.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:21.909573+0000) 2024-03-06T11:22:22.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:22.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:22.909761+0000) 2024-03-06T11:22:23.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:23.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:23.909939+0000) 2024-03-06T11:22:24.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:24.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:24.910163+0000) 2024-03-06T11:22:25.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:25.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:25.910385+0000) 2024-03-06T11:22:26.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:26.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:26.910609+0000) 2024-03-06T11:22:27.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:27.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:27.910882+0000) 2024-03-06T11:22:28.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:28.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:28.911149+0000) 2024-03-06T11:22:29.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:29.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:29.911394+0000) 2024-03-06T11:22:30.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:30.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:30.911634+0000) 2024-03-06T11:22:31.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:31.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:31.911868+0000) 2024-03-06T11:22:32.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:32.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:32.912132+0000) 2024-03-06T11:22:33.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:33.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:33.912394+0000) 2024-03-06T11:22:34.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:34.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:34.912674+0000) 2024-03-06T11:22:35.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:35.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:35.912952+0000) 2024-03-06T11:22:36.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:36.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:36.913238+0000) 2024-03-06T11:22:37.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:37.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:37.913512+0000) 2024-03-06T11:22:38.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:38.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:38.913782+0000) 2024-03-06T11:22:39.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:39.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:39.914132+0000) 2024-03-06T11:22:40.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:40.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:40.914379+0000) 2024-03-06T11:22:41.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:41.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:41.914674+0000) 2024-03-06T11:22:42.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:42.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:42.914954+0000) 2024-03-06T11:22:43.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:43.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:43.915247+0000) 2024-03-06T11:22:44.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:44.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:22:44.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:44.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:44.915525+0000) 2024-03-06T11:22:45.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:45.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:45.915728+0000) 2024-03-06T11:22:46.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:46.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:46.915937+0000) 2024-03-06T11:22:47.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:47.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:47.916217+0000) 2024-03-06T11:22:48.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:48.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:48.916496+0000) 2024-03-06T11:22:49.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:49.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:49.916779+0000) 2024-03-06T11:22:50.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:50.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:50.917100+0000) 2024-03-06T11:22:51.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:51.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:51.917310+0000) 2024-03-06T11:22:52.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:52.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:52.917568+0000) 2024-03-06T11:22:53.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:53.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:53.917904+0000) 2024-03-06T11:22:54.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:54.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:54.918159+0000) 2024-03-06T11:22:55.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:55.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:55.918395+0000) 2024-03-06T11:22:56.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:56.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:56.918687+0000) 2024-03-06T11:22:57.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:57.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:57.918959+0000) 2024-03-06T11:22:58.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:58.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:58.919242+0000) 2024-03-06T11:22:59.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:22:59.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:22:59.919529+0000) 2024-03-06T11:23:00.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:00.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:00.919812+0000) 2024-03-06T11:23:01.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:01.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:01.920136+0000) 2024-03-06T11:23:02.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:02.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:02.920343+0000) 2024-03-06T11:23:03.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:03.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:03.920522+0000) 2024-03-06T11:23:04.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:04.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:04.920709+0000) 2024-03-06T11:23:05.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:05.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:05.920927+0000) 2024-03-06T11:23:06.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:06.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:06.921154+0000) 2024-03-06T11:23:07.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:07.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:07.921371+0000) 2024-03-06T11:23:08.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:08.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:08.921636+0000) 2024-03-06T11:23:09.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:09.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:09.921889+0000) 2024-03-06T11:23:10.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:10.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:10.922111+0000) 2024-03-06T11:23:11.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:11.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:11.922345+0000) 2024-03-06T11:23:12.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:12.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:12.922530+0000) 2024-03-06T11:23:13.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:13.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:13.922757+0000) 2024-03-06T11:23:14.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:14.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:14.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:14.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:14.922991+0000) 2024-03-06T11:23:15.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:15.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:15.923226+0000) 2024-03-06T11:23:16.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:16.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:16.923452+0000) 2024-03-06T11:23:17.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:17.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:17.923710+0000) 2024-03-06T11:23:18.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:18.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:18.923989+0000) 2024-03-06T11:23:19.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:19.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:19.924313+0000) 2024-03-06T11:23:20.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:20.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:20.924527+0000) 2024-03-06T11:23:21.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:21.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:21.924790+0000) 2024-03-06T11:23:22.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:22.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:22.925117+0000) 2024-03-06T11:23:23.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:23.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:23.925422+0000) 2024-03-06T11:23:24.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:24.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:24.925695+0000) 2024-03-06T11:23:25.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:25.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:25.925969+0000) 2024-03-06T11:23:26.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:26.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:26.926269+0000) 2024-03-06T11:23:27.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:27.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:27.926494+0000) 2024-03-06T11:23:28.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:28.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:28.926725+0000) 2024-03-06T11:23:29.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:29.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:29.927037+0000) 2024-03-06T11:23:30.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:30.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:30.927279+0000) 2024-03-06T11:23:31.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:31.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:31.927515+0000) 2024-03-06T11:23:32.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:32.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:32.927793+0000) 2024-03-06T11:23:33.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:33.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:33.928084+0000) 2024-03-06T11:23:34.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:34.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:34.928318+0000) 2024-03-06T11:23:35.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:35.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:35.928544+0000) 2024-03-06T11:23:36.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:36.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:36.928757+0000) 2024-03-06T11:23:37.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:37.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:37.928983+0000) 2024-03-06T11:23:38.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:38.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:38.929232+0000) 2024-03-06T11:23:39.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:39.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:39.929451+0000) 2024-03-06T11:23:40.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:40.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:40.929711+0000) 2024-03-06T11:23:41.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:41.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:41.930044+0000) 2024-03-06T11:23:42.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:42.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:42.930301+0000) 2024-03-06T11:23:43.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:43.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:43.930542+0000) 2024-03-06T11:23:44.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:44.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:23:44.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:44.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:44.930768+0000) 2024-03-06T11:23:45.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:45.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:45.931074+0000) 2024-03-06T11:23:46.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:46.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:46.931414+0000) 2024-03-06T11:23:47.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:47.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:47.931699+0000) 2024-03-06T11:23:48.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:48.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:48.931934+0000) 2024-03-06T11:23:49.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:49.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:49.932183+0000) 2024-03-06T11:23:50.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:50.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:50.932420+0000) 2024-03-06T11:23:51.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:51.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:51.932657+0000) 2024-03-06T11:23:52.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:52.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:52.932901+0000) 2024-03-06T11:23:53.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:53.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:53.933121+0000) 2024-03-06T11:23:54.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:54.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:54.933335+0000) 2024-03-06T11:23:55.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:55.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:55.933613+0000) 2024-03-06T11:23:56.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:56.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:56.933888+0000) 2024-03-06T11:23:57.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:57.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:57.934130+0000) 2024-03-06T11:23:58.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:58.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:58.934346+0000) 2024-03-06T11:23:59.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:23:59.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:23:59.934613+0000) 2024-03-06T11:24:00.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:00.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:00.934898+0000) 2024-03-06T11:24:01.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:01.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:01.935145+0000) 2024-03-06T11:24:02.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:02.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:02.935428+0000) 2024-03-06T11:24:03.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:03.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:03.935703+0000) 2024-03-06T11:24:04.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:04.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:04.935946+0000) 2024-03-06T11:24:05.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:05.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:05.936186+0000) 2024-03-06T11:24:06.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:06.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:06.936377+0000) 2024-03-06T11:24:07.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:07.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:07.936580+0000) 2024-03-06T11:24:08.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:08.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:08.936801+0000) 2024-03-06T11:24:09.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:09.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:09.937095+0000) 2024-03-06T11:24:10.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:10.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:10.937358+0000) 2024-03-06T11:24:11.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:11.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:11.937623+0000) 2024-03-06T11:24:12.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:12.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:12.937905+0000) 2024-03-06T11:24:13.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:13.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:13.938185+0000) 2024-03-06T11:24:14.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:14.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:14.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:14.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:14.938441+0000) 2024-03-06T11:24:15.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:15.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:15.938666+0000) 2024-03-06T11:24:16.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:16.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:16.938887+0000) 2024-03-06T11:24:17.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:17.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:17.939122+0000) 2024-03-06T11:24:18.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:18.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:18.939359+0000) 2024-03-06T11:24:19.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:19.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:19.939574+0000) 2024-03-06T11:24:20.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:20.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:20.939826+0000) 2024-03-06T11:24:21.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:21.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:21.940106+0000) 2024-03-06T11:24:22.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:22.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:22.940360+0000) 2024-03-06T11:24:23.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:23.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:23.940601+0000) 2024-03-06T11:24:24.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:24.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:24.940824+0000) 2024-03-06T11:24:25.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:25.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:25.941106+0000) 2024-03-06T11:24:26.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:26.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:26.941331+0000) 2024-03-06T11:24:27.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:27.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:27.941549+0000) 2024-03-06T11:24:28.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:28.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:28.941827+0000) 2024-03-06T11:24:29.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:29.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:29.942108+0000) 2024-03-06T11:24:30.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:30.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:30.942310+0000) 2024-03-06T11:24:31.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:31.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:31.942485+0000) 2024-03-06T11:24:32.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:32.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:32.942704+0000) 2024-03-06T11:24:33.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:33.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:33.942976+0000) 2024-03-06T11:24:34.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:34.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:34.943276+0000) 2024-03-06T11:24:35.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:35.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:35.943509+0000) 2024-03-06T11:24:36.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:36.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:36.943726+0000) 2024-03-06T11:24:37.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:37.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:37.944045+0000) 2024-03-06T11:24:38.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:38.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:38.944326+0000) 2024-03-06T11:24:39.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:39.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:39.944593+0000) 2024-03-06T11:24:40.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:40.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:40.944847+0000) 2024-03-06T11:24:41.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:41.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:41.945136+0000) 2024-03-06T11:24:42.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:42.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:42.945379+0000) 2024-03-06T11:24:43.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:43.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:43.945598+0000) 2024-03-06T11:24:44.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:44.815 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:24:44.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:44.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:44.945846+0000) 2024-03-06T11:24:45.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:45.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:45.946124+0000) 2024-03-06T11:24:46.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:46.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:46.946348+0000) 2024-03-06T11:24:47.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:47.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:47.946576+0000) 2024-03-06T11:24:48.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:48.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:48.946851+0000) 2024-03-06T11:24:49.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:49.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:49.947166+0000) 2024-03-06T11:24:50.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:50.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:50.947409+0000) 2024-03-06T11:24:51.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:51.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:51.947688+0000) 2024-03-06T11:24:52.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:52.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:52.947921+0000) 2024-03-06T11:24:53.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:53.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:53.948208+0000) 2024-03-06T11:24:54.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:54.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:54.948471+0000) 2024-03-06T11:24:55.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:55.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:55.948706+0000) 2024-03-06T11:24:56.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:56.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:56.948906+0000) 2024-03-06T11:24:57.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:57.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:57.949142+0000) 2024-03-06T11:24:58.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:58.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:58.949390+0000) 2024-03-06T11:24:59.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:24:59.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:24:59.949635+0000) 2024-03-06T11:25:00.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:00.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:00.949879+0000) 2024-03-06T11:25:01.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:01.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:01.950155+0000) 2024-03-06T11:25:02.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:02.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:02.950437+0000) 2024-03-06T11:25:03.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:03.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:03.950661+0000) 2024-03-06T11:25:04.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:04.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:04.950884+0000) 2024-03-06T11:25:05.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:05.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:05.951152+0000) 2024-03-06T11:25:06.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:06.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:06.951397+0000) 2024-03-06T11:25:07.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:07.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:07.951615+0000) 2024-03-06T11:25:08.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:08.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:08.951864+0000) 2024-03-06T11:25:09.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:09.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:09.952104+0000) 2024-03-06T11:25:10.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:10.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:10.952329+0000) 2024-03-06T11:25:11.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:11.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:11.952611+0000) 2024-03-06T11:25:12.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:12.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:12.952905+0000) 2024-03-06T11:25:13.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:13.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:13.953197+0000) 2024-03-06T11:25:14.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:14.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:14.953474+0000) 2024-03-06T11:25:15.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:15.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:15.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:15.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:15.953711+0000) 2024-03-06T11:25:16.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:16.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:16.953847+0000) 2024-03-06T11:25:17.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:17.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:17.954128+0000) 2024-03-06T11:25:18.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:18.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:18.954350+0000) 2024-03-06T11:25:19.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:19.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:19.954583+0000) 2024-03-06T11:25:20.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:20.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:20.954800+0000) 2024-03-06T11:25:21.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:21.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:21.955000+0000) 2024-03-06T11:25:22.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:22.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:22.955282+0000) 2024-03-06T11:25:23.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:23.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:23.955522+0000) 2024-03-06T11:25:24.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:24.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:24.955744+0000) 2024-03-06T11:25:25.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:25.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:25.955999+0000) 2024-03-06T11:25:26.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:26.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:26.956325+0000) 2024-03-06T11:25:27.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:27.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:27.956610+0000) 2024-03-06T11:25:28.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:28.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:28.956873+0000) 2024-03-06T11:25:29.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:29.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:29.957154+0000) 2024-03-06T11:25:30.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:30.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:30.957433+0000) 2024-03-06T11:25:31.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:31.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:31.957732+0000) 2024-03-06T11:25:32.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:32.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:32.958070+0000) 2024-03-06T11:25:33.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:33.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:33.958352+0000) 2024-03-06T11:25:34.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:34.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:34.958614+0000) 2024-03-06T11:25:35.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:35.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:35.958900+0000) 2024-03-06T11:25:36.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:36.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:36.959131+0000) 2024-03-06T11:25:37.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:37.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:37.959372+0000) 2024-03-06T11:25:38.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:38.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:38.959606+0000) 2024-03-06T11:25:39.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:39.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:39.959841+0000) 2024-03-06T11:25:40.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:40.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:40.960100+0000) 2024-03-06T11:25:41.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:41.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:41.960367+0000) 2024-03-06T11:25:42.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:42.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:42.960714+0000) 2024-03-06T11:25:43.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:43.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:43.961108+0000) 2024-03-06T11:25:44.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:44.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:44.961321+0000) 2024-03-06T11:25:45.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:45.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:25:45.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:45.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:45.961572+0000) 2024-03-06T11:25:46.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:46.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:46.961791+0000) 2024-03-06T11:25:47.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:47.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:47.962145+0000) 2024-03-06T11:25:48.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:48.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:48.962384+0000) 2024-03-06T11:25:49.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:49.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:49.962569+0000) 2024-03-06T11:25:50.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:50.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:50.962818+0000) 2024-03-06T11:25:51.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:51.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:51.963125+0000) 2024-03-06T11:25:52.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:52.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:52.963349+0000) 2024-03-06T11:25:53.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:53.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:53.963597+0000) 2024-03-06T11:25:54.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:54.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:54.963832+0000) 2024-03-06T11:25:55.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:55.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:55.964117+0000) 2024-03-06T11:25:56.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:56.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:56.964356+0000) 2024-03-06T11:25:57.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:57.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:57.964627+0000) 2024-03-06T11:25:58.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:58.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:58.964895+0000) 2024-03-06T11:25:59.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:25:59.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:25:59.965146+0000) 2024-03-06T11:26:00.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:00.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:00.965396+0000) 2024-03-06T11:26:01.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:01.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:01.965659+0000) 2024-03-06T11:26:02.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:02.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:02.965907+0000) 2024-03-06T11:26:03.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:03.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:03.966122+0000) 2024-03-06T11:26:04.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:04.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:04.966343+0000) 2024-03-06T11:26:05.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:05.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:05.966575+0000) 2024-03-06T11:26:06.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:06.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:06.966797+0000) 2024-03-06T11:26:07.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:07.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:07.967066+0000) 2024-03-06T11:26:08.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:08.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:08.967301+0000) 2024-03-06T11:26:09.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:09.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:09.967515+0000) 2024-03-06T11:26:10.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:10.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:10.967715+0000) 2024-03-06T11:26:11.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:11.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:11.967951+0000) 2024-03-06T11:26:12.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:12.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:12.968191+0000) 2024-03-06T11:26:13.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:13.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:13.968383+0000) 2024-03-06T11:26:14.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:14.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:14.968608+0000) 2024-03-06T11:26:15.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:15.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:15.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:15.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:15.968891+0000) 2024-03-06T11:26:16.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:16.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:16.969118+0000) 2024-03-06T11:26:17.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:17.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:17.969315+0000) 2024-03-06T11:26:18.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:18.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:18.969542+0000) 2024-03-06T11:26:19.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:19.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:19.969772+0000) 2024-03-06T11:26:20.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:20.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:20.969988+0000) 2024-03-06T11:26:21.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:21.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:21.970286+0000) 2024-03-06T11:26:22.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:22.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:22.970534+0000) 2024-03-06T11:26:23.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:23.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:23.970808+0000) 2024-03-06T11:26:24.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:24.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:24.971085+0000) 2024-03-06T11:26:25.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:25.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:25.971314+0000) 2024-03-06T11:26:26.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:26.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:26.971553+0000) 2024-03-06T11:26:27.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:27.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:27.971813+0000) 2024-03-06T11:26:28.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:28.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:28.972065+0000) 2024-03-06T11:26:29.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:29.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:29.972276+0000) 2024-03-06T11:26:30.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:30.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:30.972474+0000) 2024-03-06T11:26:31.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:31.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:31.972698+0000) 2024-03-06T11:26:32.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:32.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:32.972903+0000) 2024-03-06T11:26:33.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:33.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:33.973151+0000) 2024-03-06T11:26:34.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:34.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:34.973335+0000) 2024-03-06T11:26:35.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:35.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:35.973580+0000) 2024-03-06T11:26:36.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:36.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:36.973753+0000) 2024-03-06T11:26:37.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:37.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:37.973985+0000) 2024-03-06T11:26:38.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:38.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:38.974248+0000) 2024-03-06T11:26:39.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:39.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:39.974472+0000) 2024-03-06T11:26:40.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:40.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:40.974694+0000) 2024-03-06T11:26:41.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:41.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:41.974923+0000) 2024-03-06T11:26:42.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:42.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:42.975154+0000) 2024-03-06T11:26:43.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:43.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:43.975399+0000) 2024-03-06T11:26:44.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:44.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:44.975650+0000) 2024-03-06T11:26:45.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:45.617 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:26:45.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:45.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:45.975883+0000) 2024-03-06T11:26:46.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:46.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:46.976128+0000) 2024-03-06T11:26:47.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:47.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:47.976355+0000) 2024-03-06T11:26:48.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:48.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:48.976610+0000) 2024-03-06T11:26:49.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:49.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:49.976824+0000) 2024-03-06T11:26:50.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:50.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:50.977038+0000) 2024-03-06T11:26:51.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:51.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:51.977280+0000) 2024-03-06T11:26:52.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:52.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:52.977477+0000) 2024-03-06T11:26:53.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:53.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:53.977702+0000) 2024-03-06T11:26:54.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:54.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:54.977928+0000) 2024-03-06T11:26:55.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:55.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:55.978126+0000) 2024-03-06T11:26:56.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:56.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:56.978324+0000) 2024-03-06T11:26:57.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:57.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:57.978539+0000) 2024-03-06T11:26:58.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:58.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:58.978774+0000) 2024-03-06T11:26:59.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:26:59.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:26:59.978979+0000) 2024-03-06T11:27:00.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:00.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:00.979263+0000) 2024-03-06T11:27:01.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:01.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:01.979496+0000) 2024-03-06T11:27:02.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:02.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:02.979731+0000) 2024-03-06T11:27:03.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:03.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:03.980068+0000) 2024-03-06T11:27:04.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:04.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:04.980291+0000) 2024-03-06T11:27:05.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:05.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:05.980533+0000) 2024-03-06T11:27:06.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:06.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:06.980731+0000) 2024-03-06T11:27:07.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:07.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:07.980944+0000) 2024-03-06T11:27:08.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:08.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:08.981189+0000) 2024-03-06T11:27:09.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:09.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:09.981367+0000) 2024-03-06T11:27:10.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:10.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:10.981545+0000) 2024-03-06T11:27:11.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:11.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:11.981779+0000) 2024-03-06T11:27:12.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:12.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:12.981995+0000) 2024-03-06T11:27:13.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:13.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:13.982466+0000) 2024-03-06T11:27:14.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:14.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:14.982718+0000) 2024-03-06T11:27:15.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:15.760 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:15.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:15.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:15.982910+0000) 2024-03-06T11:27:16.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:16.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:16.983142+0000) 2024-03-06T11:27:17.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:17.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:17.983380+0000) 2024-03-06T11:27:18.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:18.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:18.983606+0000) 2024-03-06T11:27:19.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:19.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:19.983839+0000) 2024-03-06T11:27:20.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:20.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:20.984088+0000) 2024-03-06T11:27:21.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:21.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:21.984318+0000) 2024-03-06T11:27:22.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:22.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:22.984544+0000) 2024-03-06T11:27:23.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:23.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:23.984777+0000) 2024-03-06T11:27:24.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:24.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:24.984959+0000) 2024-03-06T11:27:25.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:25.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:25.985261+0000) 2024-03-06T11:27:26.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:26.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:26.985504+0000) 2024-03-06T11:27:27.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:27.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:27.985752+0000) 2024-03-06T11:27:28.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:28.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:28.985991+0000) 2024-03-06T11:27:29.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:29.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:29.986274+0000) 2024-03-06T11:27:30.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:30.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:30.986502+0000) 2024-03-06T11:27:31.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:31.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:31.986760+0000) 2024-03-06T11:27:32.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:32.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:32.986950+0000) 2024-03-06T11:27:33.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:33.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:33.987177+0000) 2024-03-06T11:27:34.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:34.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:34.987402+0000) 2024-03-06T11:27:35.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:35.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:35.987629+0000) 2024-03-06T11:27:36.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:36.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:36.987817+0000) 2024-03-06T11:27:37.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:37.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:37.988056+0000) 2024-03-06T11:27:38.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:38.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:38.988214+0000) 2024-03-06T11:27:39.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:39.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:39.988389+0000) 2024-03-06T11:27:40.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:40.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:40.988607+0000) 2024-03-06T11:27:41.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:41.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:41.988827+0000) 2024-03-06T11:27:42.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:42.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:42.989075+0000) 2024-03-06T11:27:43.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:43.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:43.989275+0000) 2024-03-06T11:27:44.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:44.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:44.989471+0000) 2024-03-06T11:27:45.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:45.990 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:27:45.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:45.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:45.989689+0000) 2024-03-06T11:27:46.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:46.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:46.989927+0000) 2024-03-06T11:27:47.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:47.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:47.990180+0000) 2024-03-06T11:27:48.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:48.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:48.990420+0000) 2024-03-06T11:27:49.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:49.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:49.990644+0000) 2024-03-06T11:27:50.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:50.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:50.990886+0000) 2024-03-06T11:27:51.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:51.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:51.991224+0000) 2024-03-06T11:27:52.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:52.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:52.991426+0000) 2024-03-06T11:27:53.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:53.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:53.991658+0000) 2024-03-06T11:27:54.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:54.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:54.991883+0000) 2024-03-06T11:27:55.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:55.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:55.992151+0000) 2024-03-06T11:27:56.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:56.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:56.992360+0000) 2024-03-06T11:27:57.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:57.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:57.992585+0000) 2024-03-06T11:27:58.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:58.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:58.992804+0000) 2024-03-06T11:27:59.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:27:59.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:27:59.993081+0000) 2024-03-06T11:28:00.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:00.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:00.993275+0000) 2024-03-06T11:28:01.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:01.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:01.993483+0000) 2024-03-06T11:28:02.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:02.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:02.993711+0000) 2024-03-06T11:28:03.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:03.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:03.993897+0000) 2024-03-06T11:28:04.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:04.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:04.994137+0000) 2024-03-06T11:28:05.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:05.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:05.994316+0000) 2024-03-06T11:28:06.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:06.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:06.994499+0000) 2024-03-06T11:28:07.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:07.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:07.994745+0000) 2024-03-06T11:28:08.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:08.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:08.994985+0000) 2024-03-06T11:28:09.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:09.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:09.995312+0000) 2024-03-06T11:28:10.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:10.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:10.995576+0000) 2024-03-06T11:28:11.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:11.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:11.995855+0000) 2024-03-06T11:28:12.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:12.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:12.996154+0000) 2024-03-06T11:28:13.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:13.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:13.996390+0000) 2024-03-06T11:28:14.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:14.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:14.996570+0000) 2024-03-06T11:28:15.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:15.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:15.996802+0000) 2024-03-06T11:28:16.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:16.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:16.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:16.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:16.997098+0000) 2024-03-06T11:28:17.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:17.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:17.997350+0000) 2024-03-06T11:28:18.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:18.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:18.997597+0000) 2024-03-06T11:28:19.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:19.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:19.997875+0000) 2024-03-06T11:28:20.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:20.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:20.998122+0000) 2024-03-06T11:28:21.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:21.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:21.998390+0000) 2024-03-06T11:28:22.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:22.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:22.998627+0000) 2024-03-06T11:28:23.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:23.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:23.998858+0000) 2024-03-06T11:28:24.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:24.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:24.999078+0000) 2024-03-06T11:28:25.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:25.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:25.999303+0000) 2024-03-06T11:28:26.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:26.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:26.999535+0000) 2024-03-06T11:28:27.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:27.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:27.999760+0000) 2024-03-06T11:28:28.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:28.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:28.999969+0000) 2024-03-06T11:28:30.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:29.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:30.000182+0000) 2024-03-06T11:28:31.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:30.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:31.000400+0000) 2024-03-06T11:28:32.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:31.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:32.000695+0000) 2024-03-06T11:28:33.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:32.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:33.000972+0000) 2024-03-06T11:28:34.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:34.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:34.001196+0000) 2024-03-06T11:28:35.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:35.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:35.001427+0000) 2024-03-06T11:28:36.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:36.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:36.001656+0000) 2024-03-06T11:28:37.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:37.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:37.001836+0000) 2024-03-06T11:28:38.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:38.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:38.002116+0000) 2024-03-06T11:28:39.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:39.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:39.002360+0000) 2024-03-06T11:28:40.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:40.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:40.002567+0000) 2024-03-06T11:28:41.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:41.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:41.002765+0000) 2024-03-06T11:28:42.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:42.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:42.003053+0000) 2024-03-06T11:28:43.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:43.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:43.003295+0000) 2024-03-06T11:28:44.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:44.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:44.003544+0000) 2024-03-06T11:28:45.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:45.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:45.003785+0000) 2024-03-06T11:28:46.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:46.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:46.004085+0000) 2024-03-06T11:28:46.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:46.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:28:47.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:47.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:47.004308+0000) 2024-03-06T11:28:48.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:48.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:48.004528+0000) 2024-03-06T11:28:49.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:49.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:49.004752+0000) 2024-03-06T11:28:50.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:50.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:50.004977+0000) 2024-03-06T11:28:51.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:51.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:51.005246+0000) 2024-03-06T11:28:52.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:52.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:52.005510+0000) 2024-03-06T11:28:53.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:53.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:53.005646+0000) 2024-03-06T11:28:54.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:54.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:54.005825+0000) 2024-03-06T11:28:55.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:55.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:55.006103+0000) 2024-03-06T11:28:56.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:56.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:56.006351+0000) 2024-03-06T11:28:57.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:57.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:57.006598+0000) 2024-03-06T11:28:58.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:58.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:58.006842+0000) 2024-03-06T11:28:59.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:28:59.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:28:59.007105+0000) 2024-03-06T11:29:00.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:00.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:00.007366+0000) 2024-03-06T11:29:01.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:01.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:01.007586+0000) 2024-03-06T11:29:02.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:02.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:02.007867+0000) 2024-03-06T11:29:03.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:03.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:03.008106+0000) 2024-03-06T11:29:04.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:04.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:04.008332+0000) 2024-03-06T11:29:05.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:05.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:05.008575+0000) 2024-03-06T11:29:06.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:06.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:06.008852+0000) 2024-03-06T11:29:07.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:07.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:07.009161+0000) 2024-03-06T11:29:08.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:08.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:08.009447+0000) 2024-03-06T11:29:09.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:09.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:09.009717+0000) 2024-03-06T11:29:10.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:10.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:10.009933+0000) 2024-03-06T11:29:11.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:11.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:11.010142+0000) 2024-03-06T11:29:12.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:12.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:12.010375+0000) 2024-03-06T11:29:13.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:13.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:13.010598+0000) 2024-03-06T11:29:14.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:14.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:14.010829+0000) 2024-03-06T11:29:15.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:15.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:15.011105+0000) 2024-03-06T11:29:16.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:16.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:16.011350+0000) 2024-03-06T11:29:16.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:16.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:17.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:17.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:17.011566+0000) 2024-03-06T11:29:18.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:18.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:18.011798+0000) 2024-03-06T11:29:19.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:19.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:19.011992+0000) 2024-03-06T11:29:20.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:20.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:20.012252+0000) 2024-03-06T11:29:21.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:21.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:21.012472+0000) 2024-03-06T11:29:22.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:22.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:22.012722+0000) 2024-03-06T11:29:23.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:23.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:23.012945+0000) 2024-03-06T11:29:24.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:24.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:24.013157+0000) 2024-03-06T11:29:25.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:25.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:25.013388+0000) 2024-03-06T11:29:26.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:26.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:26.013608+0000) 2024-03-06T11:29:27.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:27.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:27.013821+0000) 2024-03-06T11:29:28.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:28.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:28.014093+0000) 2024-03-06T11:29:29.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:29.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:29.014294+0000) 2024-03-06T11:29:30.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:30.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:30.014520+0000) 2024-03-06T11:29:31.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:31.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:31.014690+0000) 2024-03-06T11:29:32.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:32.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:32.014932+0000) 2024-03-06T11:29:33.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:33.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:33.015169+0000) 2024-03-06T11:29:34.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:34.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:34.015403+0000) 2024-03-06T11:29:35.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:35.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:35.015599+0000) 2024-03-06T11:29:36.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:36.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:36.015771+0000) 2024-03-06T11:29:37.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:37.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:37.015991+0000) 2024-03-06T11:29:38.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:38.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:38.016278+0000) 2024-03-06T11:29:39.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:39.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:39.016564+0000) 2024-03-06T11:29:40.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:40.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:40.016797+0000) 2024-03-06T11:29:41.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:41.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:41.017059+0000) 2024-03-06T11:29:42.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:42.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:42.017295+0000) 2024-03-06T11:29:43.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:43.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:43.017520+0000) 2024-03-06T11:29:44.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:44.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:44.017740+0000) 2024-03-06T11:29:45.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:45.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:45.017918+0000) 2024-03-06T11:29:46.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:46.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:46.018131+0000) 2024-03-06T11:29:46.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:46.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:29:47.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:47.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:47.018332+0000) 2024-03-06T11:29:48.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:48.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:48.018527+0000) 2024-03-06T11:29:49.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:49.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:49.018741+0000) 2024-03-06T11:29:50.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:50.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:50.018973+0000) 2024-03-06T11:29:51.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:51.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:51.019189+0000) 2024-03-06T11:29:52.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:52.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:52.019411+0000) 2024-03-06T11:29:53.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:53.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:53.019688+0000) 2024-03-06T11:29:54.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:54.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:54.019958+0000) 2024-03-06T11:29:55.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:55.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:55.020164+0000) 2024-03-06T11:29:56.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:56.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:56.020376+0000) 2024-03-06T11:29:57.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:57.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:57.020578+0000) 2024-03-06T11:29:58.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:58.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:58.020841+0000) 2024-03-06T11:29:59.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:29:59.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:29:59.021151+0000) 2024-03-06T11:30:00.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:00.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:00.021422+0000) 2024-03-06T11:30:01.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:01.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:01.021702+0000) 2024-03-06T11:30:02.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:02.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:02.022046+0000) 2024-03-06T11:30:03.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:03.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:03.022334+0000) 2024-03-06T11:30:04.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:04.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:04.022575+0000) 2024-03-06T11:30:05.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:05.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:05.022864+0000) 2024-03-06T11:30:06.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:06.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:06.023176+0000) 2024-03-06T11:30:07.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:07.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:07.023430+0000) 2024-03-06T11:30:08.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:08.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:08.023632+0000) 2024-03-06T11:30:09.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:09.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:09.023919+0000) 2024-03-06T11:30:10.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:10.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:10.024196+0000) 2024-03-06T11:30:11.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:11.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:11.024457+0000) 2024-03-06T11:30:12.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:12.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:12.024773+0000) 2024-03-06T11:30:13.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:13.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:13.025101+0000) 2024-03-06T11:30:14.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:14.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:14.025386+0000) 2024-03-06T11:30:15.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:15.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:15.025672+0000) 2024-03-06T11:30:16.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:16.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:16.025944+0000) 2024-03-06T11:30:16.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:16.936 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:17.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:17.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:17.026215+0000) 2024-03-06T11:30:18.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:18.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:18.026465+0000) 2024-03-06T11:30:19.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:19.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:19.026711+0000) 2024-03-06T11:30:20.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:20.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:20.026924+0000) 2024-03-06T11:30:21.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:21.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:21.027191+0000) 2024-03-06T11:30:22.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:22.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:22.027500+0000) 2024-03-06T11:30:23.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:23.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:23.027782+0000) 2024-03-06T11:30:24.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:24.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:24.028125+0000) 2024-03-06T11:30:25.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:25.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:25.028324+0000) 2024-03-06T11:30:26.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:26.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:26.028563+0000) 2024-03-06T11:30:27.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:27.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:27.028803+0000) 2024-03-06T11:30:28.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:28.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:28.029097+0000) 2024-03-06T11:30:29.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:29.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:29.029363+0000) 2024-03-06T11:30:30.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:30.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:30.029608+0000) 2024-03-06T11:30:31.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:31.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:31.029844+0000) 2024-03-06T11:30:32.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:32.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:32.030144+0000) 2024-03-06T11:30:33.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:33.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:33.030386+0000) 2024-03-06T11:30:34.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:34.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:34.030644+0000) 2024-03-06T11:30:35.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:35.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:35.030860+0000) 2024-03-06T11:30:36.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:36.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:36.031144+0000) 2024-03-06T11:30:37.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:37.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:37.031377+0000) 2024-03-06T11:30:38.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:38.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:38.031595+0000) 2024-03-06T11:30:39.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:39.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:39.031832+0000) 2024-03-06T11:30:40.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:40.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:40.032102+0000) 2024-03-06T11:30:41.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:41.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:41.032331+0000) 2024-03-06T11:30:42.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:42.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:42.032556+0000) 2024-03-06T11:30:43.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:43.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:43.032808+0000) 2024-03-06T11:30:44.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:44.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:44.033080+0000) 2024-03-06T11:30:45.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:45.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:45.033320+0000) 2024-03-06T11:30:46.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:46.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:46.033536+0000) 2024-03-06T11:30:47.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:47.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:47.033662+0000) 2024-03-06T11:30:47.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:47.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:30:48.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:48.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:48.033802+0000) 2024-03-06T11:30:49.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:49.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:49.034072+0000) 2024-03-06T11:30:50.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:50.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:50.034228+0000) 2024-03-06T11:30:51.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:51.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:51.034435+0000) 2024-03-06T11:30:52.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:52.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:52.034733+0000) 2024-03-06T11:30:53.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:53.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:53.035041+0000) 2024-03-06T11:30:54.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:54.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:54.035313+0000) 2024-03-06T11:30:55.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:55.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:55.035553+0000) 2024-03-06T11:30:56.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:56.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:56.035778+0000) 2024-03-06T11:30:57.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:57.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:57.036001+0000) 2024-03-06T11:30:58.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:58.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:58.036263+0000) 2024-03-06T11:30:59.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:30:59.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:30:59.036532+0000) 2024-03-06T11:31:00.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:00.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:00.036779+0000) 2024-03-06T11:31:01.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:01.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:01.037043+0000) 2024-03-06T11:31:02.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:02.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:02.037268+0000) 2024-03-06T11:31:03.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:03.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:03.037483+0000) 2024-03-06T11:31:04.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:04.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:04.037725+0000) 2024-03-06T11:31:05.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:05.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:05.037948+0000) 2024-03-06T11:31:06.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:06.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:06.038197+0000) 2024-03-06T11:31:07.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:07.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:07.038464+0000) 2024-03-06T11:31:08.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:08.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:08.038712+0000) 2024-03-06T11:31:09.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:09.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:09.038953+0000) 2024-03-06T11:31:10.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:10.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:10.039179+0000) 2024-03-06T11:31:11.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:11.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:11.039399+0000) 2024-03-06T11:31:12.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:12.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:12.039636+0000) 2024-03-06T11:31:13.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:13.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:13.039859+0000) 2024-03-06T11:31:14.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:14.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:14.040143+0000) 2024-03-06T11:31:15.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:15.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:15.040407+0000) 2024-03-06T11:31:16.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:16.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:16.040647+0000) 2024-03-06T11:31:17.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:17.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:17.040925+0000) 2024-03-06T11:31:17.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:17.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:18.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:18.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:18.041150+0000) 2024-03-06T11:31:19.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:19.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:19.041402+0000) 2024-03-06T11:31:20.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:20.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:20.041684+0000) 2024-03-06T11:31:21.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:21.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:21.041922+0000) 2024-03-06T11:31:22.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:22.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:22.042212+0000) 2024-03-06T11:31:23.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:23.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:23.042451+0000) 2024-03-06T11:31:24.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:24.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:24.042698+0000) 2024-03-06T11:31:25.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:25.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:25.042916+0000) 2024-03-06T11:31:26.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:26.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:26.043173+0000) 2024-03-06T11:31:27.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:27.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:27.043386+0000) 2024-03-06T11:31:28.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:28.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:28.043610+0000) 2024-03-06T11:31:29.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:29.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:29.043851+0000) 2024-03-06T11:31:30.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:30.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:30.044134+0000) 2024-03-06T11:31:31.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:31.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:31.044407+0000) 2024-03-06T11:31:32.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:32.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:32.044700+0000) 2024-03-06T11:31:33.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:33.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:33.044940+0000) 2024-03-06T11:31:34.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:34.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:34.045234+0000) 2024-03-06T11:31:35.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:35.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:35.045500+0000) 2024-03-06T11:31:36.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:36.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:36.045709+0000) 2024-03-06T11:31:37.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:37.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:37.045936+0000) 2024-03-06T11:31:38.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:38.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:38.046164+0000) 2024-03-06T11:31:39.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:39.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:39.046395+0000) 2024-03-06T11:31:40.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:40.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:40.046627+0000) 2024-03-06T11:31:41.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:41.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:41.046831+0000) 2024-03-06T11:31:42.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:42.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:42.047111+0000) 2024-03-06T11:31:43.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:43.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:43.047336+0000) 2024-03-06T11:31:44.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:44.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:44.047581+0000) 2024-03-06T11:31:45.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:45.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:45.047806+0000) 2024-03-06T11:31:46.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:46.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:46.048073+0000) 2024-03-06T11:31:47.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:47.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:47.048316+0000) 2024-03-06T11:31:47.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:47.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:31:48.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:48.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:48.048583+0000) 2024-03-06T11:31:49.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:49.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:49.048878+0000) 2024-03-06T11:31:50.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:50.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:50.049185+0000) 2024-03-06T11:31:51.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:51.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:51.049459+0000) 2024-03-06T11:31:52.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:52.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:52.049742+0000) 2024-03-06T11:31:53.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:53.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:53.050060+0000) 2024-03-06T11:31:54.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:54.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:54.050345+0000) 2024-03-06T11:31:55.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:55.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:55.050615+0000) 2024-03-06T11:31:56.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:56.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:56.050898+0000) 2024-03-06T11:31:57.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:57.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:57.051185+0000) 2024-03-06T11:31:58.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:58.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:58.051471+0000) 2024-03-06T11:31:59.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:31:59.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:31:59.051742+0000) 2024-03-06T11:32:00.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:00.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:00.051951+0000) 2024-03-06T11:32:01.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:01.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:01.052173+0000) 2024-03-06T11:32:02.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:02.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:02.052420+0000) 2024-03-06T11:32:03.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:03.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:03.052691+0000) 2024-03-06T11:32:04.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:04.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:04.052926+0000) 2024-03-06T11:32:05.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:05.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:05.053155+0000) 2024-03-06T11:32:06.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:06.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:06.053346+0000) 2024-03-06T11:32:07.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:07.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:07.053567+0000) 2024-03-06T11:32:08.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:08.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:08.053818+0000) 2024-03-06T11:32:09.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:09.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:09.054055+0000) 2024-03-06T11:32:10.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:10.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:10.054289+0000) 2024-03-06T11:32:11.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:11.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:11.054567+0000) 2024-03-06T11:32:12.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:12.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:12.054863+0000) 2024-03-06T11:32:13.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:13.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:13.055168+0000) 2024-03-06T11:32:14.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:14.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:14.055446+0000) 2024-03-06T11:32:15.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:15.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:15.055715+0000) 2024-03-06T11:32:16.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:16.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:16.055993+0000) 2024-03-06T11:32:17.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:17.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:17.056281+0000) 2024-03-06T11:32:17.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:17.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:18.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:18.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:18.056473+0000) 2024-03-06T11:32:19.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:19.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:19.056722+0000) 2024-03-06T11:32:20.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:20.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:20.056973+0000) 2024-03-06T11:32:21.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:21.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:21.057263+0000) 2024-03-06T11:32:22.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:22.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:22.057497+0000) 2024-03-06T11:32:23.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:23.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:23.057696+0000) 2024-03-06T11:32:24.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:24.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:24.057931+0000) 2024-03-06T11:32:25.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:25.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:25.058173+0000) 2024-03-06T11:32:26.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:26.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:26.058363+0000) 2024-03-06T11:32:27.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:27.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:27.058595+0000) 2024-03-06T11:32:28.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:28.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:28.058844+0000) 2024-03-06T11:32:29.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:29.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:29.059129+0000) 2024-03-06T11:32:30.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:30.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:30.059378+0000) 2024-03-06T11:32:31.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:31.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:31.059637+0000) 2024-03-06T11:32:32.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:32.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:32.059894+0000) 2024-03-06T11:32:33.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:33.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:33.060167+0000) 2024-03-06T11:32:34.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:34.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:34.060407+0000) 2024-03-06T11:32:35.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:35.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:35.060677+0000) 2024-03-06T11:32:36.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:36.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:36.060948+0000) 2024-03-06T11:32:37.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:37.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:37.061239+0000) 2024-03-06T11:32:38.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:38.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:38.061522+0000) 2024-03-06T11:32:39.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:39.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:39.061801+0000) 2024-03-06T11:32:40.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:40.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:40.062109+0000) 2024-03-06T11:32:41.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:41.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:41.062348+0000) 2024-03-06T11:32:42.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:42.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:42.062559+0000) 2024-03-06T11:32:43.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:43.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:43.062778+0000) 2024-03-06T11:32:44.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:44.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:44.063052+0000) 2024-03-06T11:32:45.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:45.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:45.063285+0000) 2024-03-06T11:32:46.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:46.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:46.063512+0000) 2024-03-06T11:32:47.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:47.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:47.063694+0000) 2024-03-06T11:32:47.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:47.873 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:32:48.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:48.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:48.063917+0000) 2024-03-06T11:32:49.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:49.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:49.064141+0000) 2024-03-06T11:32:50.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:50.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:50.064385+0000) 2024-03-06T11:32:51.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:51.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:51.064578+0000) 2024-03-06T11:32:52.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:52.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:52.064812+0000) 2024-03-06T11:32:53.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:53.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:53.065076+0000) 2024-03-06T11:32:54.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:54.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:54.065313+0000) 2024-03-06T11:32:55.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:55.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:55.065570+0000) 2024-03-06T11:32:56.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:56.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:56.065855+0000) 2024-03-06T11:32:57.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:57.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:57.066160+0000) 2024-03-06T11:32:58.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:58.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:58.066450+0000) 2024-03-06T11:32:59.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:32:59.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:32:59.066723+0000) 2024-03-06T11:33:00.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:00.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:00.066928+0000) 2024-03-06T11:33:01.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:01.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:01.067144+0000) 2024-03-06T11:33:02.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:02.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:02.067374+0000) 2024-03-06T11:33:03.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:03.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:03.067556+0000) 2024-03-06T11:33:04.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:04.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:04.067778+0000) 2024-03-06T11:33:05.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:05.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:05.068048+0000) 2024-03-06T11:33:06.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:06.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:06.068275+0000) 2024-03-06T11:33:07.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:07.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:07.068495+0000) 2024-03-06T11:33:08.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:08.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:08.068720+0000) 2024-03-06T11:33:09.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:09.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:09.068936+0000) 2024-03-06T11:33:10.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:10.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:10.069179+0000) 2024-03-06T11:33:11.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:11.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:11.069428+0000) 2024-03-06T11:33:12.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:12.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:12.069712+0000) 2024-03-06T11:33:13.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:13.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:13.069994+0000) 2024-03-06T11:33:14.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:14.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:14.070344+0000) 2024-03-06T11:33:15.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:15.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:15.070583+0000) 2024-03-06T11:33:16.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:16.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:16.070775+0000) 2024-03-06T11:33:17.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:17.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:17.070962+0000) 2024-03-06T11:33:18.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:18.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:18.071184+0000) 2024-03-06T11:33:18.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:18.150 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:19.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:19.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:19.071410+0000) 2024-03-06T11:33:20.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:20.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:20.071665+0000) 2024-03-06T11:33:21.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:21.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:21.071904+0000) 2024-03-06T11:33:22.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:22.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:22.072155+0000) 2024-03-06T11:33:23.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:23.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:23.072388+0000) 2024-03-06T11:33:24.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:24.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:24.072662+0000) 2024-03-06T11:33:25.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:25.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:25.072944+0000) 2024-03-06T11:33:26.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:26.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:26.073170+0000) 2024-03-06T11:33:27.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:27.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:27.073454+0000) 2024-03-06T11:33:28.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:28.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:28.073668+0000) 2024-03-06T11:33:29.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:29.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:29.073905+0000) 2024-03-06T11:33:30.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:30.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:30.074144+0000) 2024-03-06T11:33:31.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:31.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:31.074371+0000) 2024-03-06T11:33:32.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:32.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:32.074617+0000) 2024-03-06T11:33:33.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:33.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:33.074859+0000) 2024-03-06T11:33:34.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:34.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:34.075125+0000) 2024-03-06T11:33:35.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:35.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:35.075335+0000) 2024-03-06T11:33:36.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:36.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:36.075617+0000) 2024-03-06T11:33:37.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:37.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:37.075879+0000) 2024-03-06T11:33:38.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:38.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:38.076165+0000) 2024-03-06T11:33:39.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:39.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:39.076431+0000) 2024-03-06T11:33:40.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:40.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:40.076702+0000) 2024-03-06T11:33:41.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:41.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:41.076978+0000) 2024-03-06T11:33:42.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:42.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:42.077335+0000) 2024-03-06T11:33:43.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:43.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:43.077616+0000) 2024-03-06T11:33:44.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:44.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:44.077887+0000) 2024-03-06T11:33:45.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:45.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:45.078128+0000) 2024-03-06T11:33:46.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:46.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:46.078374+0000) 2024-03-06T11:33:47.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:47.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:47.078600+0000) 2024-03-06T11:33:48.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:48.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:48.078815+0000) 2024-03-06T11:33:48.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:48.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:33:49.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:49.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:49.079161+0000) 2024-03-06T11:33:50.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:50.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:50.079450+0000) 2024-03-06T11:33:51.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:51.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:51.079729+0000) 2024-03-06T11:33:52.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:52.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:52.080082+0000) 2024-03-06T11:33:53.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:53.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:53.080335+0000) 2024-03-06T11:33:54.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:54.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:54.080588+0000) 2024-03-06T11:33:55.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:55.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:55.080847+0000) 2024-03-06T11:33:56.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:56.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:56.081162+0000) 2024-03-06T11:33:57.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:57.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:57.081443+0000) 2024-03-06T11:33:58.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:58.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:58.081697+0000) 2024-03-06T11:33:59.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:33:59.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:33:59.081945+0000) 2024-03-06T11:34:00.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:00.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:00.082225+0000) 2024-03-06T11:34:01.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:01.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:01.082508+0000) 2024-03-06T11:34:02.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:02.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:02.082801+0000) 2024-03-06T11:34:03.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:03.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:03.083115+0000) 2024-03-06T11:34:04.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:04.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:04.083400+0000) 2024-03-06T11:34:05.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:05.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:05.083681+0000) 2024-03-06T11:34:06.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:06.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:06.083899+0000) 2024-03-06T11:34:07.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:07.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:07.084141+0000) 2024-03-06T11:34:08.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:08.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:08.084419+0000) 2024-03-06T11:34:09.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:09.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:09.084709+0000) 2024-03-06T11:34:10.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:10.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:10.084971+0000) 2024-03-06T11:34:11.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:11.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:11.085201+0000) 2024-03-06T11:34:12.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:12.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:12.085468+0000) 2024-03-06T11:34:13.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:13.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:13.085745+0000) 2024-03-06T11:34:14.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:14.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:14.086061+0000) 2024-03-06T11:34:15.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:15.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:15.086303+0000) 2024-03-06T11:34:16.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:16.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:16.086516+0000) 2024-03-06T11:34:17.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:17.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:17.086824+0000) 2024-03-06T11:34:18.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:18.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:18.087154+0000) 2024-03-06T11:34:18.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:18.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:19.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:19.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:19.087443+0000) 2024-03-06T11:34:20.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:20.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:20.087716+0000) 2024-03-06T11:34:21.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:21.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:21.088001+0000) 2024-03-06T11:34:22.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:22.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:22.088373+0000) 2024-03-06T11:34:23.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:23.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:23.088611+0000) 2024-03-06T11:34:24.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:24.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:24.088850+0000) 2024-03-06T11:34:25.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:25.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:25.089139+0000) 2024-03-06T11:34:26.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:26.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:26.089370+0000) 2024-03-06T11:34:27.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:27.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:27.089593+0000) 2024-03-06T11:34:28.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:28.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:28.089817+0000) 2024-03-06T11:34:29.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:29.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:29.090061+0000) 2024-03-06T11:34:30.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:30.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:30.090294+0000) 2024-03-06T11:34:31.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:31.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:31.090561+0000) 2024-03-06T11:34:32.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:32.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:32.090840+0000) 2024-03-06T11:34:33.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:33.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:33.091087+0000) 2024-03-06T11:34:34.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:34.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:34.091277+0000) 2024-03-06T11:34:35.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:35.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:35.091463+0000) 2024-03-06T11:34:36.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:36.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:36.091702+0000) 2024-03-06T11:34:37.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:37.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:37.091916+0000) 2024-03-06T11:34:38.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:38.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:38.092205+0000) 2024-03-06T11:34:39.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:39.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:39.092401+0000) 2024-03-06T11:34:40.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:40.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:40.092635+0000) 2024-03-06T11:34:41.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:41.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:41.092863+0000) 2024-03-06T11:34:42.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:42.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:42.093132+0000) 2024-03-06T11:34:43.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:43.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:43.093308+0000) 2024-03-06T11:34:44.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:44.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:44.093495+0000) 2024-03-06T11:34:45.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:45.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:45.093723+0000) 2024-03-06T11:34:46.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:46.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:46.093961+0000) 2024-03-06T11:34:47.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:47.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:47.094152+0000) 2024-03-06T11:34:48.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:48.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:48.094361+0000) 2024-03-06T11:34:48.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:48.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:34:49.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:49.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:49.094604+0000) 2024-03-06T11:34:50.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:50.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:50.094818+0000) 2024-03-06T11:34:51.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:51.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:51.095060+0000) 2024-03-06T11:34:52.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:52.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:52.095289+0000) 2024-03-06T11:34:53.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:53.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:53.095531+0000) 2024-03-06T11:34:54.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:54.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:54.095749+0000) 2024-03-06T11:34:55.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:55.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:55.095991+0000) 2024-03-06T11:34:56.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:56.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:56.096279+0000) 2024-03-06T11:34:57.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:57.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:57.096566+0000) 2024-03-06T11:34:58.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:58.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:58.096846+0000) 2024-03-06T11:34:59.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:34:59.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:34:59.097157+0000) 2024-03-06T11:35:00.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:00.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:00.097440+0000) 2024-03-06T11:35:01.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:01.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:01.097718+0000) 2024-03-06T11:35:02.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:02.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:02.098049+0000) 2024-03-06T11:35:03.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:03.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:03.098329+0000) 2024-03-06T11:35:04.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:04.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:04.098608+0000) 2024-03-06T11:35:05.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:05.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:05.098885+0000) 2024-03-06T11:35:06.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:06.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:06.099155+0000) 2024-03-06T11:35:07.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:07.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:07.099431+0000) 2024-03-06T11:35:08.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:08.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:08.099703+0000) 2024-03-06T11:35:09.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:09.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:09.099981+0000) 2024-03-06T11:35:10.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:10.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:10.100323+0000) 2024-03-06T11:35:11.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:11.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:11.100602+0000) 2024-03-06T11:35:12.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:12.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:12.100873+0000) 2024-03-06T11:35:13.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:13.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:13.101151+0000) 2024-03-06T11:35:14.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:14.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:14.101439+0000) 2024-03-06T11:35:15.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:15.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:15.101720+0000) 2024-03-06T11:35:16.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:16.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:16.101995+0000) 2024-03-06T11:35:17.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:17.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:17.102333+0000) 2024-03-06T11:35:18.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:18.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:18.102560+0000) 2024-03-06T11:35:18.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:18.902 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:19.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:19.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:19.102821+0000) 2024-03-06T11:35:20.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:20.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:20.103121+0000) 2024-03-06T11:35:21.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:21.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:21.103400+0000) 2024-03-06T11:35:22.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:22.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:22.103644+0000) 2024-03-06T11:35:23.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:23.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:23.103868+0000) 2024-03-06T11:35:24.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:24.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:24.104134+0000) 2024-03-06T11:35:25.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:25.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:25.104368+0000) 2024-03-06T11:35:26.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:26.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:26.104570+0000) 2024-03-06T11:35:27.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:27.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:27.104787+0000) 2024-03-06T11:35:28.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:28.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:28.105056+0000) 2024-03-06T11:35:29.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:29.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:29.105276+0000) 2024-03-06T11:35:30.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:30.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:30.105513+0000) 2024-03-06T11:35:31.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:31.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:31.105738+0000) 2024-03-06T11:35:32.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:32.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:32.106103+0000) 2024-03-06T11:35:33.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:33.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:33.106383+0000) 2024-03-06T11:35:34.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:34.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:34.106663+0000) 2024-03-06T11:35:35.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:35.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:35.106937+0000) 2024-03-06T11:35:36.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:36.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:36.107214+0000) 2024-03-06T11:35:37.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:37.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:37.107455+0000) 2024-03-06T11:35:38.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:38.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:38.107742+0000) 2024-03-06T11:35:39.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:39.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:39.108057+0000) 2024-03-06T11:35:40.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:40.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:40.108296+0000) 2024-03-06T11:35:41.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:41.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:41.108481+0000) 2024-03-06T11:35:42.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:42.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:42.108727+0000) 2024-03-06T11:35:43.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:43.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:43.108960+0000) 2024-03-06T11:35:44.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:44.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:44.109191+0000) 2024-03-06T11:35:45.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:45.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:45.109419+0000) 2024-03-06T11:35:46.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:46.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:46.109633+0000) 2024-03-06T11:35:47.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:47.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:47.109873+0000) 2024-03-06T11:35:48.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:48.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:48.110131+0000) 2024-03-06T11:35:49.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:49.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:49.110378+0000) 2024-03-06T11:35:49.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:49.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:35:50.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:50.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:50.110608+0000) 2024-03-06T11:35:51.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:51.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:51.110827+0000) 2024-03-06T11:35:52.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:52.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:52.111107+0000) 2024-03-06T11:35:53.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:53.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:53.111330+0000) 2024-03-06T11:35:54.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:54.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:54.111533+0000) 2024-03-06T11:35:55.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:55.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:55.111726+0000) 2024-03-06T11:35:56.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:56.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:56.111924+0000) 2024-03-06T11:35:57.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:57.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:57.112163+0000) 2024-03-06T11:35:58.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:58.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:58.112392+0000) 2024-03-06T11:35:59.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:35:59.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:35:59.112628+0000) 2024-03-06T11:36:00.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:00.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:00.112870+0000) 2024-03-06T11:36:01.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:01.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:01.113136+0000) 2024-03-06T11:36:02.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:02.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:02.113368+0000) 2024-03-06T11:36:03.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:03.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:03.113585+0000) 2024-03-06T11:36:04.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:04.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:04.113811+0000) 2024-03-06T11:36:05.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:05.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:05.114033+0000) 2024-03-06T11:36:06.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:06.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:06.114268+0000) 2024-03-06T11:36:07.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:07.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:07.114485+0000) 2024-03-06T11:36:08.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:08.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:08.114702+0000) 2024-03-06T11:36:09.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:09.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:09.114916+0000) 2024-03-06T11:36:10.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:10.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:10.115157+0000) 2024-03-06T11:36:11.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:11.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:11.115371+0000) 2024-03-06T11:36:12.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:12.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:12.115613+0000) 2024-03-06T11:36:13.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:13.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:13.115823+0000) 2024-03-06T11:36:14.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:14.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:14.116126+0000) 2024-03-06T11:36:15.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:15.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:15.116320+0000) 2024-03-06T11:36:16.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:16.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:16.116606+0000) 2024-03-06T11:36:17.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:17.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:17.116857+0000) 2024-03-06T11:36:18.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:18.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:18.117131+0000) 2024-03-06T11:36:19.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:19.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:19.117346+0000) 2024-03-06T11:36:19.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:19.378 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:20.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:20.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:20.117538+0000) 2024-03-06T11:36:21.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:21.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:21.117758+0000) 2024-03-06T11:36:22.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:22.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:22.117997+0000) 2024-03-06T11:36:23.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:23.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:23.118250+0000) 2024-03-06T11:36:24.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:24.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:24.118490+0000) 2024-03-06T11:36:25.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:25.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:25.118745+0000) 2024-03-06T11:36:26.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:26.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:26.118989+0000) 2024-03-06T11:36:27.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:27.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:27.119388+0000) 2024-03-06T11:36:28.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:28.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:28.119644+0000) 2024-03-06T11:36:29.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:29.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:29.119861+0000) 2024-03-06T11:36:30.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:30.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:30.120130+0000) 2024-03-06T11:36:31.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:31.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:31.120345+0000) 2024-03-06T11:36:32.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:32.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:32.120577+0000) 2024-03-06T11:36:33.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:33.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:33.120805+0000) 2024-03-06T11:36:34.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:34.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:34.121067+0000) 2024-03-06T11:36:35.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:35.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:35.121314+0000) 2024-03-06T11:36:36.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:36.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:36.121486+0000) 2024-03-06T11:36:37.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:37.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:37.121727+0000) 2024-03-06T11:36:38.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:38.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:38.121967+0000) 2024-03-06T11:36:39.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:39.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:39.122190+0000) 2024-03-06T11:36:40.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:40.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:40.122425+0000) 2024-03-06T11:36:41.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:41.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:41.122642+0000) 2024-03-06T11:36:42.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:42.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:42.122865+0000) 2024-03-06T11:36:43.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:43.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:43.123117+0000) 2024-03-06T11:36:44.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:44.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:44.123342+0000) 2024-03-06T11:36:45.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:45.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:45.123581+0000) 2024-03-06T11:36:46.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:46.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:46.123803+0000) 2024-03-06T11:36:47.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:47.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:47.124051+0000) 2024-03-06T11:36:48.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:48.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:48.124291+0000) 2024-03-06T11:36:49.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:49.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:49.124534+0000) 2024-03-06T11:36:49.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:49.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:36:50.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:50.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:50.124772+0000) 2024-03-06T11:36:51.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:51.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:51.124998+0000) 2024-03-06T11:36:52.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:52.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:52.125344+0000) 2024-03-06T11:36:53.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:53.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:53.125572+0000) 2024-03-06T11:36:54.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:54.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:54.125798+0000) 2024-03-06T11:36:55.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:55.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:55.126048+0000) 2024-03-06T11:36:56.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:56.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:56.126286+0000) 2024-03-06T11:36:57.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:57.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:57.126530+0000) 2024-03-06T11:36:58.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:58.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:58.126751+0000) 2024-03-06T11:36:59.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:36:59.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:36:59.126945+0000) 2024-03-06T11:37:00.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:00.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:00.127192+0000) 2024-03-06T11:37:01.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:01.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:01.127420+0000) 2024-03-06T11:37:02.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:02.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:02.127656+0000) 2024-03-06T11:37:03.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:03.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:03.127874+0000) 2024-03-06T11:37:04.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:04.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:04.128142+0000) 2024-03-06T11:37:05.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:05.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:05.128355+0000) 2024-03-06T11:37:06.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:06.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:06.128574+0000) 2024-03-06T11:37:07.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:07.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:07.128880+0000) 2024-03-06T11:37:08.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:08.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:08.129142+0000) 2024-03-06T11:37:09.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:09.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:09.129368+0000) 2024-03-06T11:37:10.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:10.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:10.129576+0000) 2024-03-06T11:37:11.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:11.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:11.129855+0000) 2024-03-06T11:37:12.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:12.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:12.130162+0000) 2024-03-06T11:37:13.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:13.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:13.130430+0000) 2024-03-06T11:37:14.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:14.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:14.130651+0000) 2024-03-06T11:37:15.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:15.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:15.130883+0000) 2024-03-06T11:37:16.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:16.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:16.131172+0000) 2024-03-06T11:37:17.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:17.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:17.131432+0000) 2024-03-06T11:37:18.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:18.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:18.131680+0000) 2024-03-06T11:37:19.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:19.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:19.131918+0000) 2024-03-06T11:37:19.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:19.718 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:20.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:20.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:20.132162+0000) 2024-03-06T11:37:21.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:21.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:21.132417+0000) 2024-03-06T11:37:22.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:22.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:22.132705+0000) 2024-03-06T11:37:23.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:23.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:23.132976+0000) 2024-03-06T11:37:24.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:24.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:24.133293+0000) 2024-03-06T11:37:25.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:25.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:25.133581+0000) 2024-03-06T11:37:26.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:26.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:26.133827+0000) 2024-03-06T11:37:27.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:27.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:27.134103+0000) 2024-03-06T11:37:28.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:28.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:28.134354+0000) 2024-03-06T11:37:29.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:29.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:29.134567+0000) 2024-03-06T11:37:30.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:30.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:30.134853+0000) 2024-03-06T11:37:31.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:31.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:31.135167+0000) 2024-03-06T11:37:32.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:32.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:32.135467+0000) 2024-03-06T11:37:33.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:33.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:33.135746+0000) 2024-03-06T11:37:34.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:34.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:34.135934+0000) 2024-03-06T11:37:35.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:35.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:35.136188+0000) 2024-03-06T11:37:36.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:36.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:36.136417+0000) 2024-03-06T11:37:37.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:37.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:37.136641+0000) 2024-03-06T11:37:38.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:38.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:38.136866+0000) 2024-03-06T11:37:39.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:39.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:39.137143+0000) 2024-03-06T11:37:40.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:40.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:40.137384+0000) 2024-03-06T11:37:41.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:41.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:41.137603+0000) 2024-03-06T11:37:42.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:42.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:42.137850+0000) 2024-03-06T11:37:43.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:43.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:43.138108+0000) 2024-03-06T11:37:44.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:44.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:44.138314+0000) 2024-03-06T11:37:45.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:45.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:45.138525+0000) 2024-03-06T11:37:46.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:46.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:46.138794+0000) 2024-03-06T11:37:47.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:47.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:47.139091+0000) 2024-03-06T11:37:48.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:48.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:48.139331+0000) 2024-03-06T11:37:49.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:49.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:49.139551+0000) 2024-03-06T11:37:49.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:49.946 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:37:50.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:50.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:50.139802+0000) 2024-03-06T11:37:51.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:51.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:51.140065+0000) 2024-03-06T11:37:52.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:52.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:52.140305+0000) 2024-03-06T11:37:53.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:53.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:53.140527+0000) 2024-03-06T11:37:54.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:54.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:54.140787+0000) 2024-03-06T11:37:55.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:55.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:55.140979+0000) 2024-03-06T11:37:56.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:56.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:56.141258+0000) 2024-03-06T11:37:57.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:57.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:57.141482+0000) 2024-03-06T11:37:58.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:58.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:58.141709+0000) 2024-03-06T11:37:59.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:37:59.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:37:59.141960+0000) 2024-03-06T11:38:00.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:00.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:00.142272+0000) 2024-03-06T11:38:01.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:01.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:01.142483+0000) 2024-03-06T11:38:02.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:02.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:02.142736+0000) 2024-03-06T11:38:03.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:03.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:03.142958+0000) 2024-03-06T11:38:04.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:04.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:04.143221+0000) 2024-03-06T11:38:05.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:05.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:05.143496+0000) 2024-03-06T11:38:06.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:06.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:06.143777+0000) 2024-03-06T11:38:07.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:07.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:07.144110+0000) 2024-03-06T11:38:08.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:08.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:08.144360+0000) 2024-03-06T11:38:09.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:09.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:09.144518+0000) 2024-03-06T11:38:10.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:10.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:10.144748+0000) 2024-03-06T11:38:11.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:11.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:11.144977+0000) 2024-03-06T11:38:12.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:12.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:12.145246+0000) 2024-03-06T11:38:13.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:13.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:13.145466+0000) 2024-03-06T11:38:14.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:14.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:14.145682+0000) 2024-03-06T11:38:15.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:15.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:15.145845+0000) 2024-03-06T11:38:16.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:16.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:16.146165+0000) 2024-03-06T11:38:17.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:17.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:17.146444+0000) 2024-03-06T11:38:18.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:18.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:18.146722+0000) 2024-03-06T11:38:19.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:19.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:19.146917+0000) 2024-03-06T11:38:20.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:20.134 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:20.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:20.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:20.147193+0000) 2024-03-06T11:38:21.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:21.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:21.147454+0000) 2024-03-06T11:38:22.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:22.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:22.147680+0000) 2024-03-06T11:38:23.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:23.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:23.147904+0000) 2024-03-06T11:38:24.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:24.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:24.148147+0000) 2024-03-06T11:38:25.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:25.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:25.148334+0000) 2024-03-06T11:38:26.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:26.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:26.148568+0000) 2024-03-06T11:38:27.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:27.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:27.148790+0000) 2024-03-06T11:38:28.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:28.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:28.149055+0000) 2024-03-06T11:38:29.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:29.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:29.149301+0000) 2024-03-06T11:38:30.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:30.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:30.149526+0000) 2024-03-06T11:38:31.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:31.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:31.149752+0000) 2024-03-06T11:38:32.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:32.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:32.149992+0000) 2024-03-06T11:38:33.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:33.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:33.150246+0000) 2024-03-06T11:38:34.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:34.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:34.150434+0000) 2024-03-06T11:38:35.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:35.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:35.150666+0000) 2024-03-06T11:38:36.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:36.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:36.150892+0000) 2024-03-06T11:38:37.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:37.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:37.151126+0000) 2024-03-06T11:38:38.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:38.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:38.151333+0000) 2024-03-06T11:38:39.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:39.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:39.151557+0000) 2024-03-06T11:38:40.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:40.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:40.151785+0000) 2024-03-06T11:38:41.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:41.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:41.152046+0000) 2024-03-06T11:38:42.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:42.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:42.152302+0000) 2024-03-06T11:38:43.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:43.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:43.152477+0000) 2024-03-06T11:38:44.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:44.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:44.152707+0000) 2024-03-06T11:38:45.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:45.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:45.152925+0000) 2024-03-06T11:38:46.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:46.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:46.153159+0000) 2024-03-06T11:38:47.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:47.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:47.153415+0000) 2024-03-06T11:38:48.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:48.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:48.153771+0000) 2024-03-06T11:38:49.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:49.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:49.154049+0000) 2024-03-06T11:38:50.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:50.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:50.154300+0000) 2024-03-06T11:38:50.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:50.289 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:38:51.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:51.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:51.154527+0000) 2024-03-06T11:38:52.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:52.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:52.154777+0000) 2024-03-06T11:38:53.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:53.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:53.154953+0000) 2024-03-06T11:38:54.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:54.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:54.155244+0000) 2024-03-06T11:38:55.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:55.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:55.155461+0000) 2024-03-06T11:38:56.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:56.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:56.155695+0000) 2024-03-06T11:38:57.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:57.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:57.155914+0000) 2024-03-06T11:38:58.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:58.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:58.156143+0000) 2024-03-06T11:38:59.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:38:59.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:38:59.156375+0000) 2024-03-06T11:39:00.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:00.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:00.156638+0000) 2024-03-06T11:39:01.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:01.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:01.156913+0000) 2024-03-06T11:39:02.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:02.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:02.157208+0000) 2024-03-06T11:39:03.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:03.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:03.157487+0000) 2024-03-06T11:39:04.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:04.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:04.157760+0000) 2024-03-06T11:39:05.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:05.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:05.157995+0000) 2024-03-06T11:39:06.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:06.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:06.158311+0000) 2024-03-06T11:39:07.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:07.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:07.158586+0000) 2024-03-06T11:39:08.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:08.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:08.158824+0000) 2024-03-06T11:39:09.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:09.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:09.159085+0000) 2024-03-06T11:39:10.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:10.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:10.159320+0000) 2024-03-06T11:39:11.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:11.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:11.159597+0000) 2024-03-06T11:39:12.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:12.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:12.159888+0000) 2024-03-06T11:39:13.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:13.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:13.160124+0000) 2024-03-06T11:39:14.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:14.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:14.160353+0000) 2024-03-06T11:39:15.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:15.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:15.160551+0000) 2024-03-06T11:39:16.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:16.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:16.160780+0000) 2024-03-06T11:39:17.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:17.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:17.160998+0000) 2024-03-06T11:39:18.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:18.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:18.161268+0000) 2024-03-06T11:39:19.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:19.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:19.161466+0000) 2024-03-06T11:39:20.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:20.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:20.161702+0000) 2024-03-06T11:39:20.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:20.493 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:21.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:21.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:21.161929+0000) 2024-03-06T11:39:22.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:22.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:22.162177+0000) 2024-03-06T11:39:23.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:23.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:23.162426+0000) 2024-03-06T11:39:24.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:24.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:24.162605+0000) 2024-03-06T11:39:25.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:25.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:25.162852+0000) 2024-03-06T11:39:26.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:26.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:26.163126+0000) 2024-03-06T11:39:27.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:27.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:27.163341+0000) 2024-03-06T11:39:28.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:28.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:28.163564+0000) 2024-03-06T11:39:29.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:29.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:29.163784+0000) 2024-03-06T11:39:30.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:30.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:30.164038+0000) 2024-03-06T11:39:31.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:31.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:31.164351+0000) 2024-03-06T11:39:32.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:32.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:32.164565+0000) 2024-03-06T11:39:33.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:33.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:33.164808+0000) 2024-03-06T11:39:34.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:34.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:34.165090+0000) 2024-03-06T11:39:35.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:35.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:35.165277+0000) 2024-03-06T11:39:36.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:36.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:36.165725+0000) 2024-03-06T11:39:37.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:37.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:37.165965+0000) 2024-03-06T11:39:38.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:38.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:38.166253+0000) 2024-03-06T11:39:39.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:39.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:39.166501+0000) 2024-03-06T11:39:40.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:40.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:40.166784+0000) 2024-03-06T11:39:41.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:41.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:41.167113+0000) 2024-03-06T11:39:42.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:42.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:42.167411+0000) 2024-03-06T11:39:43.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:43.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:43.167651+0000) 2024-03-06T11:39:44.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:44.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:44.167875+0000) 2024-03-06T11:39:45.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:45.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:45.168158+0000) 2024-03-06T11:39:46.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:46.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:46.168367+0000) 2024-03-06T11:39:47.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:47.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:47.168584+0000) 2024-03-06T11:39:48.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:48.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:48.168824+0000) 2024-03-06T11:39:49.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:49.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:49.169115+0000) 2024-03-06T11:39:50.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:50.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:50.169367+0000) 2024-03-06T11:39:50.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:50.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:39:51.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:51.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:51.169598+0000) 2024-03-06T11:39:52.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:52.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:52.169879+0000) 2024-03-06T11:39:53.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:53.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:53.170114+0000) 2024-03-06T11:39:54.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:54.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:54.170362+0000) 2024-03-06T11:39:55.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:55.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:55.170603+0000) 2024-03-06T11:39:56.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:56.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:56.170811+0000) 2024-03-06T11:39:57.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:57.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:57.170992+0000) 2024-03-06T11:39:58.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:58.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:58.171259+0000) 2024-03-06T11:39:59.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:39:59.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:39:59.171507+0000) 2024-03-06T11:40:00.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:00.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:00.171727+0000) 2024-03-06T11:40:01.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:01.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:01.172190+0000) 2024-03-06T11:40:02.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:02.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:02.172408+0000) 2024-03-06T11:40:03.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:03.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:03.172658+0000) 2024-03-06T11:40:04.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:04.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:04.172904+0000) 2024-03-06T11:40:05.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:05.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:05.173161+0000) 2024-03-06T11:40:06.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:06.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:06.173419+0000) 2024-03-06T11:40:07.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:07.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:07.173658+0000) 2024-03-06T11:40:08.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:08.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:08.173926+0000) 2024-03-06T11:40:09.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:09.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:09.174183+0000) 2024-03-06T11:40:10.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:10.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:10.174430+0000) 2024-03-06T11:40:11.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:11.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:11.174634+0000) 2024-03-06T11:40:12.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:12.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:12.174867+0000) 2024-03-06T11:40:13.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:13.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:13.175132+0000) 2024-03-06T11:40:14.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:14.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:14.175361+0000) 2024-03-06T11:40:15.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:15.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:15.175581+0000) 2024-03-06T11:40:16.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:16.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:16.175767+0000) 2024-03-06T11:40:17.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:17.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:17.175981+0000) 2024-03-06T11:40:18.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:18.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:18.176253+0000) 2024-03-06T11:40:19.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:19.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:19.176441+0000) 2024-03-06T11:40:20.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:20.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:20.176693+0000) 2024-03-06T11:40:20.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:20.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:21.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:21.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:21.176933+0000) 2024-03-06T11:40:22.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:22.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:22.177160+0000) 2024-03-06T11:40:23.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:23.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:23.177377+0000) 2024-03-06T11:40:24.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:24.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:24.177606+0000) 2024-03-06T11:40:25.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:25.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:25.177825+0000) 2024-03-06T11:40:26.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:26.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:26.178098+0000) 2024-03-06T11:40:27.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:27.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:27.178273+0000) 2024-03-06T11:40:28.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:28.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:28.178471+0000) 2024-03-06T11:40:29.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:29.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:29.178651+0000) 2024-03-06T11:40:30.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:30.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:30.178870+0000) 2024-03-06T11:40:31.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:31.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:31.179127+0000) 2024-03-06T11:40:32.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:32.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:32.179386+0000) 2024-03-06T11:40:33.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:33.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:33.179607+0000) 2024-03-06T11:40:34.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:34.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:34.179829+0000) 2024-03-06T11:40:35.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:35.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:35.180049+0000) 2024-03-06T11:40:36.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:36.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:36.180306+0000) 2024-03-06T11:40:37.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:37.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:37.180539+0000) 2024-03-06T11:40:38.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:38.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:38.180780+0000) 2024-03-06T11:40:39.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:39.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:39.181077+0000) 2024-03-06T11:40:40.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:40.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:40.181341+0000) 2024-03-06T11:40:41.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:41.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:41.181592+0000) 2024-03-06T11:40:42.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:42.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:42.181851+0000) 2024-03-06T11:40:43.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:43.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:43.182131+0000) 2024-03-06T11:40:44.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:44.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:44.182382+0000) 2024-03-06T11:40:45.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:45.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:45.182620+0000) 2024-03-06T11:40:46.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:46.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:46.182867+0000) 2024-03-06T11:40:47.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:47.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:47.183138+0000) 2024-03-06T11:40:48.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:48.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:48.183389+0000) 2024-03-06T11:40:49.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:49.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:49.183630+0000) 2024-03-06T11:40:50.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:50.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:50.183872+0000) 2024-03-06T11:40:51.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:51.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:40:51.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:51.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:51.184149+0000) 2024-03-06T11:40:52.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:52.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:52.184374+0000) 2024-03-06T11:40:53.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:53.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:53.184569+0000) 2024-03-06T11:40:54.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:54.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:54.184791+0000) 2024-03-06T11:40:55.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:55.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:55.185050+0000) 2024-03-06T11:40:56.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:56.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:56.185320+0000) 2024-03-06T11:40:57.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:57.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:57.185538+0000) 2024-03-06T11:40:58.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:58.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:58.185756+0000) 2024-03-06T11:40:59.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:40:59.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:40:59.185987+0000) 2024-03-06T11:41:00.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:00.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:00.186271+0000) 2024-03-06T11:41:01.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:01.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:01.186504+0000) 2024-03-06T11:41:02.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:02.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:02.186742+0000) 2024-03-06T11:41:03.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:03.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:03.186959+0000) 2024-03-06T11:41:04.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:04.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:04.187195+0000) 2024-03-06T11:41:05.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:05.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:05.187428+0000) 2024-03-06T11:41:06.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:06.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:06.187657+0000) 2024-03-06T11:41:07.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:07.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:07.187838+0000) 2024-03-06T11:41:08.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:08.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:08.188112+0000) 2024-03-06T11:41:09.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:09.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:09.188360+0000) 2024-03-06T11:41:10.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:10.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:10.188611+0000) 2024-03-06T11:41:11.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:11.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:11.188809+0000) 2024-03-06T11:41:12.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:12.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:12.189099+0000) 2024-03-06T11:41:13.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:13.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:13.189327+0000) 2024-03-06T11:41:14.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:14.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:14.189548+0000) 2024-03-06T11:41:15.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:15.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:15.189733+0000) 2024-03-06T11:41:16.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:16.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:16.189930+0000) 2024-03-06T11:41:17.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:17.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:17.190176+0000) 2024-03-06T11:41:18.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:18.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:18.190452+0000) 2024-03-06T11:41:19.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:19.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:19.190698+0000) 2024-03-06T11:41:20.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:20.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:20.190944+0000) 2024-03-06T11:41:21.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:21.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:21.191175+0000) 2024-03-06T11:41:21.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:21.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:22.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:22.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:22.191374+0000) 2024-03-06T11:41:23.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:23.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:23.191624+0000) 2024-03-06T11:41:24.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:24.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:24.191848+0000) 2024-03-06T11:41:25.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:25.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:25.192138+0000) 2024-03-06T11:41:26.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:26.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:26.192395+0000) 2024-03-06T11:41:27.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:27.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:27.192594+0000) 2024-03-06T11:41:28.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:28.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:28.192785+0000) 2024-03-06T11:41:29.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:29.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:29.192982+0000) 2024-03-06T11:41:30.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:30.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:30.193238+0000) 2024-03-06T11:41:31.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:31.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:31.193439+0000) 2024-03-06T11:41:32.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:32.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:32.193669+0000) 2024-03-06T11:41:33.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:33.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:33.193871+0000) 2024-03-06T11:41:34.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:34.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:34.194131+0000) 2024-03-06T11:41:35.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:35.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:35.194317+0000) 2024-03-06T11:41:36.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:36.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:36.194555+0000) 2024-03-06T11:41:37.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:37.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:37.194792+0000) 2024-03-06T11:41:38.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:38.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:38.194989+0000) 2024-03-06T11:41:39.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:39.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:39.195226+0000) 2024-03-06T11:41:40.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:40.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:40.195436+0000) 2024-03-06T11:41:41.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:41.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:41.195675+0000) 2024-03-06T11:41:42.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:42.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:42.195924+0000) 2024-03-06T11:41:43.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:43.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:43.196157+0000) 2024-03-06T11:41:44.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:44.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:44.196355+0000) 2024-03-06T11:41:45.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:45.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:45.196580+0000) 2024-03-06T11:41:46.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:46.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:46.196836+0000) 2024-03-06T11:41:47.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:47.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:47.197096+0000) 2024-03-06T11:41:48.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:48.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:48.197356+0000) 2024-03-06T11:41:49.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:49.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:49.197543+0000) 2024-03-06T11:41:50.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:50.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:50.197775+0000) 2024-03-06T11:41:51.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:51.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:51.198034+0000) 2024-03-06T11:41:51.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:51.501 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:41:52.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:52.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:52.198244+0000) 2024-03-06T11:41:53.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:53.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:53.198462+0000) 2024-03-06T11:41:54.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:54.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:54.198682+0000) 2024-03-06T11:41:55.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:55.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:55.198940+0000) 2024-03-06T11:41:56.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:56.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:56.199170+0000) 2024-03-06T11:41:57.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:57.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:57.199326+0000) 2024-03-06T11:41:58.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:58.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:58.199539+0000) 2024-03-06T11:41:59.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:41:59.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:41:59.199821+0000) 2024-03-06T11:42:00.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:00.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:00.200143+0000) 2024-03-06T11:42:01.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:01.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:01.200436+0000) 2024-03-06T11:42:02.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:02.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:02.200721+0000) 2024-03-06T11:42:03.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:03.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:03.200935+0000) 2024-03-06T11:42:04.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:04.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:04.201158+0000) 2024-03-06T11:42:05.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:05.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:05.201345+0000) 2024-03-06T11:42:06.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:06.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:06.201633+0000) 2024-03-06T11:42:07.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:07.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:07.201858+0000) 2024-03-06T11:42:08.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:08.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:08.202110+0000) 2024-03-06T11:42:09.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:09.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:09.202293+0000) 2024-03-06T11:42:10.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:10.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:10.202499+0000) 2024-03-06T11:42:11.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:11.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:11.202734+0000) 2024-03-06T11:42:12.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:12.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:12.202944+0000) 2024-03-06T11:42:13.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:13.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:13.203162+0000) 2024-03-06T11:42:14.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:14.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:14.203399+0000) 2024-03-06T11:42:15.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:15.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:15.203637+0000) 2024-03-06T11:42:16.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:16.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:16.203910+0000) 2024-03-06T11:42:17.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:17.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:17.204151+0000) 2024-03-06T11:42:18.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:18.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:18.204371+0000) 2024-03-06T11:42:19.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:19.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:19.204552+0000) 2024-03-06T11:42:20.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:20.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:20.204777+0000) 2024-03-06T11:42:21.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:21.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:21.205063+0000) 2024-03-06T11:42:21.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:21.647 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:22.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:22.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:22.205313+0000) 2024-03-06T11:42:23.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:23.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:23.205475+0000) 2024-03-06T11:42:24.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:24.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:24.205699+0000) 2024-03-06T11:42:25.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:25.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:25.205954+0000) 2024-03-06T11:42:26.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:26.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:26.206228+0000) 2024-03-06T11:42:27.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:27.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:27.206423+0000) 2024-03-06T11:42:28.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:28.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:28.206611+0000) 2024-03-06T11:42:29.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:29.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:29.206846+0000) 2024-03-06T11:42:30.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:30.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:30.207120+0000) 2024-03-06T11:42:31.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:31.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:31.207328+0000) 2024-03-06T11:42:32.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:32.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:32.207553+0000) 2024-03-06T11:42:33.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:33.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:33.207773+0000) 2024-03-06T11:42:34.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:34.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:34.208000+0000) 2024-03-06T11:42:35.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:35.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:35.208278+0000) 2024-03-06T11:42:36.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:36.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:36.208479+0000) 2024-03-06T11:42:37.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:37.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:37.208662+0000) 2024-03-06T11:42:38.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:38.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:38.208890+0000) 2024-03-06T11:42:39.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:39.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:39.209127+0000) 2024-03-06T11:42:40.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:40.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:40.209329+0000) 2024-03-06T11:42:41.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:41.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:41.209553+0000) 2024-03-06T11:42:42.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:42.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:42.209792+0000) 2024-03-06T11:42:43.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:43.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:43.210045+0000) 2024-03-06T11:42:44.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:44.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:44.210279+0000) 2024-03-06T11:42:45.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:45.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:45.210482+0000) 2024-03-06T11:42:46.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:46.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:46.210723+0000) 2024-03-06T11:42:47.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:47.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:47.210941+0000) 2024-03-06T11:42:48.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:48.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:48.211149+0000) 2024-03-06T11:42:49.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:49.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:49.211387+0000) 2024-03-06T11:42:50.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:50.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:50.211608+0000) 2024-03-06T11:42:51.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:51.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:51.211855+0000) 2024-03-06T11:42:51.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:51.822 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:42:52.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:52.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:52.212096+0000) 2024-03-06T11:42:53.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:53.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:53.212300+0000) 2024-03-06T11:42:54.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:54.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:54.212511+0000) 2024-03-06T11:42:55.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:55.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:55.212730+0000) 2024-03-06T11:42:56.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:56.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:56.212982+0000) 2024-03-06T11:42:57.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:57.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:57.213258+0000) 2024-03-06T11:42:58.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:58.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:58.213470+0000) 2024-03-06T11:42:59.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:42:59.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:42:59.213746+0000) 2024-03-06T11:43:00.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:00.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:00.213971+0000) 2024-03-06T11:43:01.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:01.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:01.214184+0000) 2024-03-06T11:43:02.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:02.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:02.214381+0000) 2024-03-06T11:43:03.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:03.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:03.214613+0000) 2024-03-06T11:43:04.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:04.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:04.214875+0000) 2024-03-06T11:43:05.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:05.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:05.215139+0000) 2024-03-06T11:43:06.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:06.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:06.215353+0000) 2024-03-06T11:43:07.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:07.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:07.215557+0000) 2024-03-06T11:43:08.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:08.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:08.215780+0000) 2024-03-06T11:43:09.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:09.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:09.216033+0000) 2024-03-06T11:43:10.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:10.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:10.216228+0000) 2024-03-06T11:43:11.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:11.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:11.216443+0000) 2024-03-06T11:43:12.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:12.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:12.216681+0000) 2024-03-06T11:43:13.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:13.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:13.216866+0000) 2024-03-06T11:43:14.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:14.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:14.217142+0000) 2024-03-06T11:43:15.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:15.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:15.217335+0000) 2024-03-06T11:43:16.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:16.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:16.217562+0000) 2024-03-06T11:43:17.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:17.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:17.217782+0000) 2024-03-06T11:43:18.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:18.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:18.218030+0000) 2024-03-06T11:43:19.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:19.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:19.218211+0000) 2024-03-06T11:43:20.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:20.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:20.218408+0000) 2024-03-06T11:43:21.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:21.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:21.218624+0000) 2024-03-06T11:43:22.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:22.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:22.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:22.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:22.218864+0000) 2024-03-06T11:43:23.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:23.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:23.219169+0000) 2024-03-06T11:43:24.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:24.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:24.219353+0000) 2024-03-06T11:43:25.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:25.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:25.219558+0000) 2024-03-06T11:43:26.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:26.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:26.219795+0000) 2024-03-06T11:43:27.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:27.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:27.220068+0000) 2024-03-06T11:43:28.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:28.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:28.220286+0000) 2024-03-06T11:43:29.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:29.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:29.220497+0000) 2024-03-06T11:43:30.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:30.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:30.220704+0000) 2024-03-06T11:43:31.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:31.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:31.220988+0000) 2024-03-06T11:43:32.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:32.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:32.221296+0000) 2024-03-06T11:43:33.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:33.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:33.221546+0000) 2024-03-06T11:43:34.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:34.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:34.221744+0000) 2024-03-06T11:43:35.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:35.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:35.221961+0000) 2024-03-06T11:43:36.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:36.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:36.222219+0000) 2024-03-06T11:43:37.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:37.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:37.222446+0000) 2024-03-06T11:43:38.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:38.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:38.222671+0000) 2024-03-06T11:43:39.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:39.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:39.222841+0000) 2024-03-06T11:43:40.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:40.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:40.223110+0000) 2024-03-06T11:43:41.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:41.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:41.223376+0000) 2024-03-06T11:43:42.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:42.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:42.223619+0000) 2024-03-06T11:43:43.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:43.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:43.223845+0000) 2024-03-06T11:43:44.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:44.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:44.224163+0000) 2024-03-06T11:43:45.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:45.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:45.224443+0000) 2024-03-06T11:43:46.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:46.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:46.224655+0000) 2024-03-06T11:43:47.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:47.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:47.224924+0000) 2024-03-06T11:43:48.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:48.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:48.225199+0000) 2024-03-06T11:43:49.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:49.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:49.225476+0000) 2024-03-06T11:43:50.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:50.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:50.225750+0000) 2024-03-06T11:43:51.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:51.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:51.225970+0000) 2024-03-06T11:43:52.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:52.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:52.226210+0000) 2024-03-06T11:43:52.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:52.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:43:53.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:53.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:53.226412+0000) 2024-03-06T11:43:54.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:54.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:54.226637+0000) 2024-03-06T11:43:55.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:55.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:55.226877+0000) 2024-03-06T11:43:56.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:56.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:56.227140+0000) 2024-03-06T11:43:57.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:57.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:57.227365+0000) 2024-03-06T11:43:58.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:58.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:58.227605+0000) 2024-03-06T11:43:59.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:43:59.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:43:59.227829+0000) 2024-03-06T11:44:00.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:00.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:00.228094+0000) 2024-03-06T11:44:01.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:01.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:01.228317+0000) 2024-03-06T11:44:02.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:02.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:02.228583+0000) 2024-03-06T11:44:03.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:03.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:03.228822+0000) 2024-03-06T11:44:04.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:04.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:04.229110+0000) 2024-03-06T11:44:05.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:05.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:05.229337+0000) 2024-03-06T11:44:06.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:06.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:06.229612+0000) 2024-03-06T11:44:07.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:07.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:07.229862+0000) 2024-03-06T11:44:08.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:08.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:08.230162+0000) 2024-03-06T11:44:09.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:09.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:09.230401+0000) 2024-03-06T11:44:10.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:10.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:10.230632+0000) 2024-03-06T11:44:11.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:11.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:11.230859+0000) 2024-03-06T11:44:12.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:12.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:12.231158+0000) 2024-03-06T11:44:13.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:13.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:13.231396+0000) 2024-03-06T11:44:14.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:14.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:14.231675+0000) 2024-03-06T11:44:15.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:15.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:15.231954+0000) 2024-03-06T11:44:16.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:16.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:16.232135+0000) 2024-03-06T11:44:17.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:17.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:17.232369+0000) 2024-03-06T11:44:18.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:18.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:18.232585+0000) 2024-03-06T11:44:19.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:19.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:19.232857+0000) 2024-03-06T11:44:20.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:20.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:20.233105+0000) 2024-03-06T11:44:21.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:21.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:21.233301+0000) 2024-03-06T11:44:22.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:22.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:22.233523+0000) 2024-03-06T11:44:22.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:22.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:23.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:23.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:23.233750+0000) 2024-03-06T11:44:24.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:24.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:24.233977+0000) 2024-03-06T11:44:25.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:25.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:25.234240+0000) 2024-03-06T11:44:26.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:26.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:26.234467+0000) 2024-03-06T11:44:27.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:27.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:27.234680+0000) 2024-03-06T11:44:28.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:28.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:28.234913+0000) 2024-03-06T11:44:29.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:29.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:29.235166+0000) 2024-03-06T11:44:30.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:30.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:30.235450+0000) 2024-03-06T11:44:31.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:31.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:31.235721+0000) 2024-03-06T11:44:32.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:32.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:32.236056+0000) 2024-03-06T11:44:33.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:33.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:33.236368+0000) 2024-03-06T11:44:34.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:34.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:34.236609+0000) 2024-03-06T11:44:35.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:35.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:35.236863+0000) 2024-03-06T11:44:36.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:36.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:36.237137+0000) 2024-03-06T11:44:37.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:37.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:37.237404+0000) 2024-03-06T11:44:38.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:38.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:38.237681+0000) 2024-03-06T11:44:39.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:39.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:39.237854+0000) 2024-03-06T11:44:40.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:40.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:40.238128+0000) 2024-03-06T11:44:41.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:41.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:41.238386+0000) 2024-03-06T11:44:42.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:42.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:42.238573+0000) 2024-03-06T11:44:43.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:43.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:43.238801+0000) 2024-03-06T11:44:44.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:44.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:44.239069+0000) 2024-03-06T11:44:45.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:45.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:45.239283+0000) 2024-03-06T11:44:46.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:46.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:46.239553+0000) 2024-03-06T11:44:47.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:47.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:47.239839+0000) 2024-03-06T11:44:48.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:48.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:48.240160+0000) 2024-03-06T11:44:49.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:49.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:49.240440+0000) 2024-03-06T11:44:50.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:50.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:50.240677+0000) 2024-03-06T11:44:51.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:51.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:51.240926+0000) 2024-03-06T11:44:52.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:52.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:52.241168+0000) 2024-03-06T11:44:52.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:52.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:44:53.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:53.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:53.241383+0000) 2024-03-06T11:44:54.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:54.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:54.241622+0000) 2024-03-06T11:44:55.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:55.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:55.241822+0000) 2024-03-06T11:44:56.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:56.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:56.242041+0000) 2024-03-06T11:44:57.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:57.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:57.242224+0000) 2024-03-06T11:44:58.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:58.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:58.242443+0000) 2024-03-06T11:44:59.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:44:59.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:44:59.242668+0000) 2024-03-06T11:45:00.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:00.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:00.242893+0000) 2024-03-06T11:45:01.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:01.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:01.243121+0000) 2024-03-06T11:45:02.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:02.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:02.243353+0000) 2024-03-06T11:45:03.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:03.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:03.243570+0000) 2024-03-06T11:45:04.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:04.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:04.243788+0000) 2024-03-06T11:45:05.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:05.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:05.243961+0000) 2024-03-06T11:45:06.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:06.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:06.244186+0000) 2024-03-06T11:45:07.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:07.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:07.244412+0000) 2024-03-06T11:45:08.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:08.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:08.244694+0000) 2024-03-06T11:45:09.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:09.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:09.244974+0000) 2024-03-06T11:45:10.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:10.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:10.245301+0000) 2024-03-06T11:45:11.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:11.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:11.245547+0000) 2024-03-06T11:45:12.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:12.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:12.245785+0000) 2024-03-06T11:45:13.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:13.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:13.246052+0000) 2024-03-06T11:45:14.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:14.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:14.246271+0000) 2024-03-06T11:45:15.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:15.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:15.246494+0000) 2024-03-06T11:45:16.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:16.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:16.246740+0000) 2024-03-06T11:45:17.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:17.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:17.246993+0000) 2024-03-06T11:45:18.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:18.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:18.247282+0000) 2024-03-06T11:45:19.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:19.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:19.247498+0000) 2024-03-06T11:45:20.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:20.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:20.247719+0000) 2024-03-06T11:45:21.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:21.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:21.247940+0000) 2024-03-06T11:45:22.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:22.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:22.248181+0000) 2024-03-06T11:45:22.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:22.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:23.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:23.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:23.248397+0000) 2024-03-06T11:45:24.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:24.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:24.248603+0000) 2024-03-06T11:45:25.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:25.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:25.248826+0000) 2024-03-06T11:45:26.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:26.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:26.249087+0000) 2024-03-06T11:45:27.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:27.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:27.249283+0000) 2024-03-06T11:45:28.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:28.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:28.249604+0000) 2024-03-06T11:45:29.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:29.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:29.249848+0000) 2024-03-06T11:45:30.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:30.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:30.250102+0000) 2024-03-06T11:45:31.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:31.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:31.250333+0000) 2024-03-06T11:45:32.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:32.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:32.250584+0000) 2024-03-06T11:45:33.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:33.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:33.250795+0000) 2024-03-06T11:45:34.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:34.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:34.251069+0000) 2024-03-06T11:45:35.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:35.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:35.251357+0000) 2024-03-06T11:45:36.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:36.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:36.251559+0000) 2024-03-06T11:45:37.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:37.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:37.251830+0000) 2024-03-06T11:45:38.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:38.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:38.252148+0000) 2024-03-06T11:45:39.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:39.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:39.252409+0000) 2024-03-06T11:45:40.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:40.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:40.252694+0000) 2024-03-06T11:45:41.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:41.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:41.252974+0000) 2024-03-06T11:45:42.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:42.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:42.253197+0000) 2024-03-06T11:45:43.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:43.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:43.253468+0000) 2024-03-06T11:45:44.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:44.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:44.253715+0000) 2024-03-06T11:45:45.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:45.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:45.253951+0000) 2024-03-06T11:45:46.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:46.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:46.254213+0000) 2024-03-06T11:45:47.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:47.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:47.254459+0000) 2024-03-06T11:45:48.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:48.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:48.254732+0000) 2024-03-06T11:45:49.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:49.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:49.254960+0000) 2024-03-06T11:45:50.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:50.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:50.255190+0000) 2024-03-06T11:45:51.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:51.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:51.255432+0000) 2024-03-06T11:45:52.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:52.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:52.255687+0000) 2024-03-06T11:45:53.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:53.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:45:53.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:53.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:53.255984+0000) 2024-03-06T11:45:54.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:54.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:54.256299+0000) 2024-03-06T11:45:55.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:55.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:55.256555+0000) 2024-03-06T11:45:56.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:56.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:56.256827+0000) 2024-03-06T11:45:57.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:57.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:57.257133+0000) 2024-03-06T11:45:58.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:58.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:58.257416+0000) 2024-03-06T11:45:59.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:45:59.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:45:59.257643+0000) 2024-03-06T11:46:00.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:00.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:00.257876+0000) 2024-03-06T11:46:01.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:01.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:01.258171+0000) 2024-03-06T11:46:02.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:02.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:02.258471+0000) 2024-03-06T11:46:03.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:03.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:03.258747+0000) 2024-03-06T11:46:04.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:04.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:04.259079+0000) 2024-03-06T11:46:05.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:05.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:05.259339+0000) 2024-03-06T11:46:06.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:06.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:06.259620+0000) 2024-03-06T11:46:07.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:07.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:07.259873+0000) 2024-03-06T11:46:08.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:08.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:08.260122+0000) 2024-03-06T11:46:09.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:09.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:09.260354+0000) 2024-03-06T11:46:10.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:10.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:10.260636+0000) 2024-03-06T11:46:11.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:11.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:11.260877+0000) 2024-03-06T11:46:12.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:12.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:12.261137+0000) 2024-03-06T11:46:13.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:13.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:13.261319+0000) 2024-03-06T11:46:14.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:14.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:14.261509+0000) 2024-03-06T11:46:15.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:15.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:15.261764+0000) 2024-03-06T11:46:16.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:16.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:16.262087+0000) 2024-03-06T11:46:17.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:17.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:17.262313+0000) 2024-03-06T11:46:18.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:18.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:18.262598+0000) 2024-03-06T11:46:19.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:19.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:19.262878+0000) 2024-03-06T11:46:20.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:20.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:20.263134+0000) 2024-03-06T11:46:21.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:21.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:21.263416+0000) 2024-03-06T11:46:22.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:22.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:22.263685+0000) 2024-03-06T11:46:23.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:23.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:23.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:23.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:23.263934+0000) 2024-03-06T11:46:24.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:24.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:24.264165+0000) 2024-03-06T11:46:25.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:25.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:25.264385+0000) 2024-03-06T11:46:26.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:26.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:26.264605+0000) 2024-03-06T11:46:27.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:27.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:27.264836+0000) 2024-03-06T11:46:28.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:28.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:28.265112+0000) 2024-03-06T11:46:29.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:29.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:29.265355+0000) 2024-03-06T11:46:30.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:30.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:30.265574+0000) 2024-03-06T11:46:31.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:31.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:31.265829+0000) 2024-03-06T11:46:32.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:32.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:32.266139+0000) 2024-03-06T11:46:33.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:33.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:33.266367+0000) 2024-03-06T11:46:34.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:34.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:34.266594+0000) 2024-03-06T11:46:35.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:35.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:35.266769+0000) 2024-03-06T11:46:36.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:36.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:36.267037+0000) 2024-03-06T11:46:37.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:37.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:37.267280+0000) 2024-03-06T11:46:38.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:38.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:38.267499+0000) 2024-03-06T11:46:39.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:39.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:39.267720+0000) 2024-03-06T11:46:40.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:40.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:40.267899+0000) 2024-03-06T11:46:41.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:41.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:41.268133+0000) 2024-03-06T11:46:42.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:42.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:42.268295+0000) 2024-03-06T11:46:43.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:43.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:43.268518+0000) 2024-03-06T11:46:44.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:44.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:44.268745+0000) 2024-03-06T11:46:45.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:45.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:45.268969+0000) 2024-03-06T11:46:46.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:46.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:46.269174+0000) 2024-03-06T11:46:47.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:47.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:47.269371+0000) 2024-03-06T11:46:48.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:48.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:48.269585+0000) 2024-03-06T11:46:49.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:49.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:49.269819+0000) 2024-03-06T11:46:50.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:50.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:50.270034+0000) 2024-03-06T11:46:51.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:51.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:51.270203+0000) 2024-03-06T11:46:52.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:52.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:52.270445+0000) 2024-03-06T11:46:53.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:53.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:53.270669+0000) 2024-03-06T11:46:53.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:53.473 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:46:54.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:54.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:54.270902+0000) 2024-03-06T11:46:55.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:55.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:55.271148+0000) 2024-03-06T11:46:56.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:56.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:56.271382+0000) 2024-03-06T11:46:57.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:57.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:57.271620+0000) 2024-03-06T11:46:58.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:58.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:58.271841+0000) 2024-03-06T11:46:59.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:46:59.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:46:59.272158+0000) 2024-03-06T11:47:00.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:00.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:00.272349+0000) 2024-03-06T11:47:01.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:01.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:01.272545+0000) 2024-03-06T11:47:02.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:02.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:02.272786+0000) 2024-03-06T11:47:03.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:03.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:03.273031+0000) 2024-03-06T11:47:04.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:04.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:04.273278+0000) 2024-03-06T11:47:05.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:05.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:05.273478+0000) 2024-03-06T11:47:06.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:06.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:06.273703+0000) 2024-03-06T11:47:07.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:07.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:07.273938+0000) 2024-03-06T11:47:08.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:08.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:08.274168+0000) 2024-03-06T11:47:09.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:09.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:09.274312+0000) 2024-03-06T11:47:10.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:10.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:10.274547+0000) 2024-03-06T11:47:11.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:11.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:11.274788+0000) 2024-03-06T11:47:12.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:12.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:12.275051+0000) 2024-03-06T11:47:13.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:13.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:13.275210+0000) 2024-03-06T11:47:14.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:14.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:14.275430+0000) 2024-03-06T11:47:15.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:15.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:15.275648+0000) 2024-03-06T11:47:16.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:16.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:16.275915+0000) 2024-03-06T11:47:17.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:17.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:17.276215+0000) 2024-03-06T11:47:18.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:18.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:18.276428+0000) 2024-03-06T11:47:19.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:19.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:19.276663+0000) 2024-03-06T11:47:20.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:20.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:20.276912+0000) 2024-03-06T11:47:21.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:21.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:21.277170+0000) 2024-03-06T11:47:22.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:22.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:22.277465+0000) 2024-03-06T11:47:23.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:23.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:23.277708+0000) 2024-03-06T11:47:23.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:23.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:24.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:24.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:24.277976+0000) 2024-03-06T11:47:25.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:25.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:25.278275+0000) 2024-03-06T11:47:26.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:26.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:26.278546+0000) 2024-03-06T11:47:27.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:27.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:27.278762+0000) 2024-03-06T11:47:28.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:28.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:28.278965+0000) 2024-03-06T11:47:29.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:29.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:29.279162+0000) 2024-03-06T11:47:30.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:30.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:30.279365+0000) 2024-03-06T11:47:31.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:31.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:31.279627+0000) 2024-03-06T11:47:32.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:32.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:32.279873+0000) 2024-03-06T11:47:33.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:33.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:33.280122+0000) 2024-03-06T11:47:34.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:34.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:34.280271+0000) 2024-03-06T11:47:35.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:35.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:35.280523+0000) 2024-03-06T11:47:36.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:36.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:36.280756+0000) 2024-03-06T11:47:37.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:37.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:37.281042+0000) 2024-03-06T11:47:38.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:38.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:38.281236+0000) 2024-03-06T11:47:39.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:39.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:39.281465+0000) 2024-03-06T11:47:40.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:40.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:40.281724+0000) 2024-03-06T11:47:41.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:41.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:41.282051+0000) 2024-03-06T11:47:42.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:42.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:42.282386+0000) 2024-03-06T11:47:43.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:43.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:43.282672+0000) 2024-03-06T11:47:44.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:44.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:44.282954+0000) 2024-03-06T11:47:45.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:45.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:45.283191+0000) 2024-03-06T11:47:46.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:46.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:46.283473+0000) 2024-03-06T11:47:47.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:47.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:47.283767+0000) 2024-03-06T11:47:48.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:48.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:48.284096+0000) 2024-03-06T11:47:49.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:49.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:49.284375+0000) 2024-03-06T11:47:50.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:50.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:50.284656+0000) 2024-03-06T11:47:51.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:51.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:51.284934+0000) 2024-03-06T11:47:52.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:52.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:52.285236+0000) 2024-03-06T11:47:53.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:53.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:53.285510+0000) 2024-03-06T11:47:53.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:53.912 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:47:54.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:54.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:54.285746+0000) 2024-03-06T11:47:55.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:55.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:55.285987+0000) 2024-03-06T11:47:56.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:56.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:56.286261+0000) 2024-03-06T11:47:57.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:57.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:57.286505+0000) 2024-03-06T11:47:58.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:58.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:58.286843+0000) 2024-03-06T11:47:59.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:47:59.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:47:59.287134+0000) 2024-03-06T11:48:00.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:00.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:00.287349+0000) 2024-03-06T11:48:01.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:01.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:01.287571+0000) 2024-03-06T11:48:02.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:02.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:02.287854+0000) 2024-03-06T11:48:03.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:03.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:03.288130+0000) 2024-03-06T11:48:04.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:04.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:04.288323+0000) 2024-03-06T11:48:05.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:05.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:05.288521+0000) 2024-03-06T11:48:06.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:06.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:06.288743+0000) 2024-03-06T11:48:07.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:07.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:07.288982+0000) 2024-03-06T11:48:08.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:08.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:08.289230+0000) 2024-03-06T11:48:09.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:09.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:09.289397+0000) 2024-03-06T11:48:10.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:10.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:10.289620+0000) 2024-03-06T11:48:11.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:11.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:11.289843+0000) 2024-03-06T11:48:12.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:12.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:12.290117+0000) 2024-03-06T11:48:13.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:13.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:13.290310+0000) 2024-03-06T11:48:14.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:14.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:14.290524+0000) 2024-03-06T11:48:15.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:15.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:15.290739+0000) 2024-03-06T11:48:16.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:16.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:16.290966+0000) 2024-03-06T11:48:17.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:17.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:17.291173+0000) 2024-03-06T11:48:18.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:18.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:18.291333+0000) 2024-03-06T11:48:19.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:19.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:19.291549+0000) 2024-03-06T11:48:20.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:20.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:20.291765+0000) 2024-03-06T11:48:21.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:21.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:21.291986+0000) 2024-03-06T11:48:22.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:22.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:22.292303+0000) 2024-03-06T11:48:23.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:23.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:23.292584+0000) 2024-03-06T11:48:24.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:24.058 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:24.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:24.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:24.292874+0000) 2024-03-06T11:48:25.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:25.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:25.293178+0000) 2024-03-06T11:48:26.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:26.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:26.293447+0000) 2024-03-06T11:48:27.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:27.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:27.293742+0000) 2024-03-06T11:48:28.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:28.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:28.293992+0000) 2024-03-06T11:48:29.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:29.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:29.294279+0000) 2024-03-06T11:48:30.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:30.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:30.294530+0000) 2024-03-06T11:48:31.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:31.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:31.294783+0000) 2024-03-06T11:48:32.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:32.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:32.295076+0000) 2024-03-06T11:48:33.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:33.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:33.295271+0000) 2024-03-06T11:48:34.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:34.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:34.295495+0000) 2024-03-06T11:48:35.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:35.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:35.295761+0000) 2024-03-06T11:48:36.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:36.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:36.296073+0000) 2024-03-06T11:48:37.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:37.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:37.296258+0000) 2024-03-06T11:48:38.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:38.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:38.296484+0000) 2024-03-06T11:48:39.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:39.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:39.296706+0000) 2024-03-06T11:48:40.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:40.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:40.296976+0000) 2024-03-06T11:48:41.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:41.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:41.297265+0000) 2024-03-06T11:48:42.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:42.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:42.297553+0000) 2024-03-06T11:48:43.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:43.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:43.297796+0000) 2024-03-06T11:48:44.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:44.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:44.298043+0000) 2024-03-06T11:48:45.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:45.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:45.298236+0000) 2024-03-06T11:48:46.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:46.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:46.298440+0000) 2024-03-06T11:48:47.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:47.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:47.298689+0000) 2024-03-06T11:48:48.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:48.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:48.298906+0000) 2024-03-06T11:48:49.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:49.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:49.299136+0000) 2024-03-06T11:48:50.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:50.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:50.299302+0000) 2024-03-06T11:48:51.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:51.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:51.299517+0000) 2024-03-06T11:48:52.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:52.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:52.299770+0000) 2024-03-06T11:48:53.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:53.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:53.300000+0000) 2024-03-06T11:48:54.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:54.262 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:48:54.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:54.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:54.300274+0000) 2024-03-06T11:48:55.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:55.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:55.300517+0000) 2024-03-06T11:48:56.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:56.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:56.300748+0000) 2024-03-06T11:48:57.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:57.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:57.301041+0000) 2024-03-06T11:48:58.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:58.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:58.301269+0000) 2024-03-06T11:48:59.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:48:59.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:48:59.301492+0000) 2024-03-06T11:49:00.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:00.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:00.301719+0000) 2024-03-06T11:49:01.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:01.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:01.301881+0000) 2024-03-06T11:49:02.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:02.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:02.302122+0000) 2024-03-06T11:49:03.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:03.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:03.302332+0000) 2024-03-06T11:49:04.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:04.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:04.302538+0000) 2024-03-06T11:49:05.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:05.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:05.302753+0000) 2024-03-06T11:49:06.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:06.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:06.302982+0000) 2024-03-06T11:49:07.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:07.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:07.303211+0000) 2024-03-06T11:49:08.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:08.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:08.303392+0000) 2024-03-06T11:49:09.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:09.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:09.303592+0000) 2024-03-06T11:49:10.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:10.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:10.303819+0000) 2024-03-06T11:49:11.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:11.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:11.304001+0000) 2024-03-06T11:49:12.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:12.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:12.304285+0000) 2024-03-06T11:49:13.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:13.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:13.304514+0000) 2024-03-06T11:49:14.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:14.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:14.304735+0000) 2024-03-06T11:49:15.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:15.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:15.304942+0000) 2024-03-06T11:49:16.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:16.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:16.305183+0000) 2024-03-06T11:49:17.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:17.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:17.305405+0000) 2024-03-06T11:49:18.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:18.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:18.305573+0000) 2024-03-06T11:49:19.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:19.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:19.305788+0000) 2024-03-06T11:49:20.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:20.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:20.306043+0000) 2024-03-06T11:49:21.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:21.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:21.306257+0000) 2024-03-06T11:49:22.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:22.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:22.306492+0000) 2024-03-06T11:49:23.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:23.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:23.306715+0000) 2024-03-06T11:49:24.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:24.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:24.306878+0000) 2024-03-06T11:49:24.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:24.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:25.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:25.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:25.307124+0000) 2024-03-06T11:49:26.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:26.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:26.307351+0000) 2024-03-06T11:49:27.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:27.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:27.307598+0000) 2024-03-06T11:49:28.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:28.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:28.307827+0000) 2024-03-06T11:49:29.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:29.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:29.308091+0000) 2024-03-06T11:49:30.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:30.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:30.308319+0000) 2024-03-06T11:49:31.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:31.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:31.308548+0000) 2024-03-06T11:49:32.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:32.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:32.308785+0000) 2024-03-06T11:49:33.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:33.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:33.309036+0000) 2024-03-06T11:49:34.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:34.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:34.309322+0000) 2024-03-06T11:49:35.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:35.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:35.309546+0000) 2024-03-06T11:49:36.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:36.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:36.309770+0000) 2024-03-06T11:49:37.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:37.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:37.309960+0000) 2024-03-06T11:49:38.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:38.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:38.310158+0000) 2024-03-06T11:49:39.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:39.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:39.310355+0000) 2024-03-06T11:49:40.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:40.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:40.310576+0000) 2024-03-06T11:49:41.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:41.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:41.310794+0000) 2024-03-06T11:49:42.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:42.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:42.311049+0000) 2024-03-06T11:49:43.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:43.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:43.311275+0000) 2024-03-06T11:49:44.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:44.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:44.311479+0000) 2024-03-06T11:49:45.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:45.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:45.311676+0000) 2024-03-06T11:49:46.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:46.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:46.311894+0000) 2024-03-06T11:49:47.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:47.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:47.312150+0000) 2024-03-06T11:49:48.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:48.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:48.312345+0000) 2024-03-06T11:49:49.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:49.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:49.312503+0000) 2024-03-06T11:49:50.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:50.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:50.312731+0000) 2024-03-06T11:49:51.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:51.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:51.312956+0000) 2024-03-06T11:49:52.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:52.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:52.313153+0000) 2024-03-06T11:49:53.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:53.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:53.313393+0000) 2024-03-06T11:49:54.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:54.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:54.313640+0000) 2024-03-06T11:49:54.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:54.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:49:55.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:55.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:55.313894+0000) 2024-03-06T11:49:56.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:56.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:56.314143+0000) 2024-03-06T11:49:57.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:57.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:57.314363+0000) 2024-03-06T11:49:58.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:58.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:58.314588+0000) 2024-03-06T11:49:59.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:49:59.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:49:59.314803+0000) 2024-03-06T11:50:00.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:00.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:00.314978+0000) 2024-03-06T11:50:01.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:01.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:01.315229+0000) 2024-03-06T11:50:02.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:02.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:02.315473+0000) 2024-03-06T11:50:03.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:03.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:03.315701+0000) 2024-03-06T11:50:04.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:04.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:04.315927+0000) 2024-03-06T11:50:05.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:05.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:05.316165+0000) 2024-03-06T11:50:06.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:06.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:06.316421+0000) 2024-03-06T11:50:07.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:07.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:07.316646+0000) 2024-03-06T11:50:08.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:08.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:08.316891+0000) 2024-03-06T11:50:09.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:09.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:09.317112+0000) 2024-03-06T11:50:10.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:10.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:10.317340+0000) 2024-03-06T11:50:11.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:11.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:11.317501+0000) 2024-03-06T11:50:12.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:12.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:12.317702+0000) 2024-03-06T11:50:13.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:13.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:13.317896+0000) 2024-03-06T11:50:14.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:14.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:14.318123+0000) 2024-03-06T11:50:15.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:15.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:15.318337+0000) 2024-03-06T11:50:16.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:16.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:16.318543+0000) 2024-03-06T11:50:17.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:17.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:17.318758+0000) 2024-03-06T11:50:18.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:18.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:18.318976+0000) 2024-03-06T11:50:19.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:19.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:19.319201+0000) 2024-03-06T11:50:20.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:20.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:20.319395+0000) 2024-03-06T11:50:21.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:21.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:21.319576+0000) 2024-03-06T11:50:22.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:22.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:22.319815+0000) 2024-03-06T11:50:23.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:23.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:23.320109+0000) 2024-03-06T11:50:24.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:24.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:24.320295+0000) 2024-03-06T11:50:24.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:24.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:25.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:25.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:25.320523+0000) 2024-03-06T11:50:26.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:26.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:26.320708+0000) 2024-03-06T11:50:27.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:27.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:27.320983+0000) 2024-03-06T11:50:28.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:28.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:28.321169+0000) 2024-03-06T11:50:29.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:29.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:29.321310+0000) 2024-03-06T11:50:30.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:30.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:30.321519+0000) 2024-03-06T11:50:31.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:31.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:31.321692+0000) 2024-03-06T11:50:32.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:32.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:32.321931+0000) 2024-03-06T11:50:33.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:33.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:33.322155+0000) 2024-03-06T11:50:34.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:34.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:34.322517+0000) 2024-03-06T11:50:35.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:35.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:35.322692+0000) 2024-03-06T11:50:36.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:36.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:36.322934+0000) 2024-03-06T11:50:37.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:37.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:37.323151+0000) 2024-03-06T11:50:38.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:38.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:38.323310+0000) 2024-03-06T11:50:39.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:39.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:39.323539+0000) 2024-03-06T11:50:40.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:40.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:40.323739+0000) 2024-03-06T11:50:41.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:41.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:41.323953+0000) 2024-03-06T11:50:42.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:42.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:42.324198+0000) 2024-03-06T11:50:43.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:43.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:43.324394+0000) 2024-03-06T11:50:44.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:44.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:44.324593+0000) 2024-03-06T11:50:45.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:45.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:45.324817+0000) 2024-03-06T11:50:46.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:46.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:46.325071+0000) 2024-03-06T11:50:47.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:47.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:47.325282+0000) 2024-03-06T11:50:48.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:48.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:48.325506+0000) 2024-03-06T11:50:49.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:49.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:49.325682+0000) 2024-03-06T11:50:50.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:50.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:50.325927+0000) 2024-03-06T11:50:51.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:51.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:51.326143+0000) 2024-03-06T11:50:52.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:52.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:52.326345+0000) 2024-03-06T11:50:53.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:53.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:53.326537+0000) 2024-03-06T11:50:54.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:54.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:54.326769+0000) 2024-03-06T11:50:54.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:54.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:50:55.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:55.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:55.327038+0000) 2024-03-06T11:50:56.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:56.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:56.327210+0000) 2024-03-06T11:50:57.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:57.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:57.327431+0000) 2024-03-06T11:50:58.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:58.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:58.327656+0000) 2024-03-06T11:50:59.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:50:59.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:50:59.327842+0000) 2024-03-06T11:51:00.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:00.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:00.328112+0000) 2024-03-06T11:51:01.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:01.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:01.328269+0000) 2024-03-06T11:51:02.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:02.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:02.328484+0000) 2024-03-06T11:51:03.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:03.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:03.328676+0000) 2024-03-06T11:51:04.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:04.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:04.328911+0000) 2024-03-06T11:51:05.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:05.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:05.329151+0000) 2024-03-06T11:51:06.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:06.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:06.329402+0000) 2024-03-06T11:51:07.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:07.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:07.329631+0000) 2024-03-06T11:51:08.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:08.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:08.329861+0000) 2024-03-06T11:51:09.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:09.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:09.330124+0000) 2024-03-06T11:51:10.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:10.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:10.330356+0000) 2024-03-06T11:51:11.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:11.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:11.330592+0000) 2024-03-06T11:51:12.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:12.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:12.330844+0000) 2024-03-06T11:51:13.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:13.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:13.331127+0000) 2024-03-06T11:51:14.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:14.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:14.331368+0000) 2024-03-06T11:51:15.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:15.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:15.331655+0000) 2024-03-06T11:51:16.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:16.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:16.331881+0000) 2024-03-06T11:51:17.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:17.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:17.332118+0000) 2024-03-06T11:51:18.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:18.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:18.332274+0000) 2024-03-06T11:51:19.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:19.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:19.332492+0000) 2024-03-06T11:51:20.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:20.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:20.332752+0000) 2024-03-06T11:51:21.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:21.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:21.332973+0000) 2024-03-06T11:51:22.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:22.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:22.333201+0000) 2024-03-06T11:51:23.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:23.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:23.333354+0000) 2024-03-06T11:51:24.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:24.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:24.333549+0000) 2024-03-06T11:51:25.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:25.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:25.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:25.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:25.333783+0000) 2024-03-06T11:51:26.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:26.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:26.333996+0000) 2024-03-06T11:51:27.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:27.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:27.334195+0000) 2024-03-06T11:51:28.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:28.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:28.334411+0000) 2024-03-06T11:51:29.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:29.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:29.334650+0000) 2024-03-06T11:51:30.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:30.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:30.334876+0000) 2024-03-06T11:51:31.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:31.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:31.335132+0000) 2024-03-06T11:51:32.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:32.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:32.335367+0000) 2024-03-06T11:51:33.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:33.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:33.335595+0000) 2024-03-06T11:51:34.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:34.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:34.335811+0000) 2024-03-06T11:51:35.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:35.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:35.336068+0000) 2024-03-06T11:51:36.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:36.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:36.336312+0000) 2024-03-06T11:51:37.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:37.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:37.336550+0000) 2024-03-06T11:51:38.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:38.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:38.336758+0000) 2024-03-06T11:51:39.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:39.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:39.336974+0000) 2024-03-06T11:51:40.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:40.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:40.337172+0000) 2024-03-06T11:51:41.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:41.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:41.337362+0000) 2024-03-06T11:51:42.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:42.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:42.337594+0000) 2024-03-06T11:51:43.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:43.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:43.337829+0000) 2024-03-06T11:51:44.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:44.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:44.338094+0000) 2024-03-06T11:51:45.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:45.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:45.338324+0000) 2024-03-06T11:51:46.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:46.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:46.338595+0000) 2024-03-06T11:51:47.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:47.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:47.338872+0000) 2024-03-06T11:51:48.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:48.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:48.339096+0000) 2024-03-06T11:51:49.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:49.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:49.339317+0000) 2024-03-06T11:51:50.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:50.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:50.339538+0000) 2024-03-06T11:51:51.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:51.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:51.339759+0000) 2024-03-06T11:51:52.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:52.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:52.340037+0000) 2024-03-06T11:51:53.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:53.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:53.340278+0000) 2024-03-06T11:51:54.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:54.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:54.340521+0000) 2024-03-06T11:51:55.321 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:55.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:51:55.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:55.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:55.340750+0000) 2024-03-06T11:51:56.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:56.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:56.340938+0000) 2024-03-06T11:51:57.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:57.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:57.341194+0000) 2024-03-06T11:51:58.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:58.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:58.341438+0000) 2024-03-06T11:51:59.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:51:59.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:51:59.341698+0000) 2024-03-06T11:52:00.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:00.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:00.341940+0000) 2024-03-06T11:52:01.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:01.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:01.342183+0000) 2024-03-06T11:52:02.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:02.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:02.342441+0000) 2024-03-06T11:52:03.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:03.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:03.342665+0000) 2024-03-06T11:52:04.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:04.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:04.342873+0000) 2024-03-06T11:52:05.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:05.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:05.343144+0000) 2024-03-06T11:52:06.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:06.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:06.343385+0000) 2024-03-06T11:52:07.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:07.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:07.343636+0000) 2024-03-06T11:52:08.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:08.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:08.343920+0000) 2024-03-06T11:52:09.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:09.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:09.344200+0000) 2024-03-06T11:52:10.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:10.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:10.344445+0000) 2024-03-06T11:52:11.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:11.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:11.344663+0000) 2024-03-06T11:52:12.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:12.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:12.344869+0000) 2024-03-06T11:52:13.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:13.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:13.345146+0000) 2024-03-06T11:52:14.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:14.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:14.345377+0000) 2024-03-06T11:52:15.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:15.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:15.345595+0000) 2024-03-06T11:52:16.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:16.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:16.345821+0000) 2024-03-06T11:52:17.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:17.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:17.346050+0000) 2024-03-06T11:52:18.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:18.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:18.346302+0000) 2024-03-06T11:52:19.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:19.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:19.346562+0000) 2024-03-06T11:52:20.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:20.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:20.346843+0000) 2024-03-06T11:52:21.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:21.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:21.347125+0000) 2024-03-06T11:52:22.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:22.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:22.347398+0000) 2024-03-06T11:52:23.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:23.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:23.347612+0000) 2024-03-06T11:52:24.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:24.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:24.347849+0000) 2024-03-06T11:52:25.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:25.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:25.348120+0000) 2024-03-06T11:52:25.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:25.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:26.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:26.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:26.348349+0000) 2024-03-06T11:52:27.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:27.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:27.348588+0000) 2024-03-06T11:52:28.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:28.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:28.348816+0000) 2024-03-06T11:52:29.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:29.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:29.349104+0000) 2024-03-06T11:52:30.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:30.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:30.349322+0000) 2024-03-06T11:52:31.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:31.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:31.349552+0000) 2024-03-06T11:52:32.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:32.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:32.349833+0000) 2024-03-06T11:52:33.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:33.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:33.350119+0000) 2024-03-06T11:52:34.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:34.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:34.350328+0000) 2024-03-06T11:52:35.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:35.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:35.350556+0000) 2024-03-06T11:52:36.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:36.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:36.350830+0000) 2024-03-06T11:52:37.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:37.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:37.351142+0000) 2024-03-06T11:52:38.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:38.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:38.351418+0000) 2024-03-06T11:52:39.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:39.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:39.351696+0000) 2024-03-06T11:52:40.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:40.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:40.351885+0000) 2024-03-06T11:52:41.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:41.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:41.352086+0000) 2024-03-06T11:52:42.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:42.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:42.352322+0000) 2024-03-06T11:52:43.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:43.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:43.352560+0000) 2024-03-06T11:52:44.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:44.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:44.352799+0000) 2024-03-06T11:52:45.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:45.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:45.353041+0000) 2024-03-06T11:52:46.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:46.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:46.353276+0000) 2024-03-06T11:52:47.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:47.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:47.353493+0000) 2024-03-06T11:52:48.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:48.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:48.353718+0000) 2024-03-06T11:52:49.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:49.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:49.353942+0000) 2024-03-06T11:52:50.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:50.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:50.354178+0000) 2024-03-06T11:52:51.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:51.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:51.354391+0000) 2024-03-06T11:52:52.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:52.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:52.354634+0000) 2024-03-06T11:52:53.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:53.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:53.354858+0000) 2024-03-06T11:52:54.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:54.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:54.355084+0000) 2024-03-06T11:52:55.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:55.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:55.355348+0000) 2024-03-06T11:52:55.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:55.612 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:52:56.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:56.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:56.355688+0000) 2024-03-06T11:52:57.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:57.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:57.355878+0000) 2024-03-06T11:52:58.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:58.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:58.356128+0000) 2024-03-06T11:52:59.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:52:59.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:52:59.356359+0000) 2024-03-06T11:53:00.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:00.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:00.356586+0000) 2024-03-06T11:53:01.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:01.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:01.356840+0000) 2024-03-06T11:53:02.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:02.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:02.357104+0000) 2024-03-06T11:53:03.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:03.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:03.357312+0000) 2024-03-06T11:53:04.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:04.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:04.357536+0000) 2024-03-06T11:53:05.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:05.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:05.357798+0000) 2024-03-06T11:53:06.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:06.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:06.358108+0000) 2024-03-06T11:53:07.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:07.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:07.358354+0000) 2024-03-06T11:53:08.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:08.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:08.358634+0000) 2024-03-06T11:53:09.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:09.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:09.358913+0000) 2024-03-06T11:53:10.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:10.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:10.359196+0000) 2024-03-06T11:53:11.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:11.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:11.359453+0000) 2024-03-06T11:53:12.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:12.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:12.359673+0000) 2024-03-06T11:53:13.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:13.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:13.359912+0000) 2024-03-06T11:53:14.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:14.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:14.360141+0000) 2024-03-06T11:53:15.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:15.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:15.360363+0000) 2024-03-06T11:53:16.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:16.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:16.360545+0000) 2024-03-06T11:53:17.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:17.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:17.360761+0000) 2024-03-06T11:53:18.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:18.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:18.361063+0000) 2024-03-06T11:53:19.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:19.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:19.361309+0000) 2024-03-06T11:53:20.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:20.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:20.361539+0000) 2024-03-06T11:53:21.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:21.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:21.361746+0000) 2024-03-06T11:53:22.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:22.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:22.362047+0000) 2024-03-06T11:53:23.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:23.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:23.362286+0000) 2024-03-06T11:53:24.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:24.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:24.362525+0000) 2024-03-06T11:53:25.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:25.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:25.362736+0000) 2024-03-06T11:53:25.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:25.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:26.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:26.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:26.362949+0000) 2024-03-06T11:53:27.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:27.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:27.363172+0000) 2024-03-06T11:53:28.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:28.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:28.363405+0000) 2024-03-06T11:53:29.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:29.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:29.363623+0000) 2024-03-06T11:53:30.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:30.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:30.364363+0000) 2024-03-06T11:53:31.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:31.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:31.364599+0000) 2024-03-06T11:53:32.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:32.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:32.364832+0000) 2024-03-06T11:53:33.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:33.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:33.365091+0000) 2024-03-06T11:53:34.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:34.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:34.365334+0000) 2024-03-06T11:53:35.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:35.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:35.365515+0000) 2024-03-06T11:53:36.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:36.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:36.365696+0000) 2024-03-06T11:53:37.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:37.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:37.365921+0000) 2024-03-06T11:53:38.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:38.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:38.366186+0000) 2024-03-06T11:53:39.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:39.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:39.366354+0000) 2024-03-06T11:53:40.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:40.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:40.366567+0000) 2024-03-06T11:53:41.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:41.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:41.366796+0000) 2024-03-06T11:53:42.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:42.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:42.367065+0000) 2024-03-06T11:53:43.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:43.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:43.367304+0000) 2024-03-06T11:53:44.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:44.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:44.367565+0000) 2024-03-06T11:53:45.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:45.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:45.367790+0000) 2024-03-06T11:53:46.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:46.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:46.368047+0000) 2024-03-06T11:53:47.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:47.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:47.368289+0000) 2024-03-06T11:53:48.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:48.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:48.368500+0000) 2024-03-06T11:53:49.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:49.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:49.368718+0000) 2024-03-06T11:53:50.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:50.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:50.368951+0000) 2024-03-06T11:53:51.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:51.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:51.369174+0000) 2024-03-06T11:53:52.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:52.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:52.369388+0000) 2024-03-06T11:53:53.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:53.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:53.369613+0000) 2024-03-06T11:53:54.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:54.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:54.369862+0000) 2024-03-06T11:53:55.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:55.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:55.370129+0000) 2024-03-06T11:53:55.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:55.964 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:53:56.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:56.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:56.370357+0000) 2024-03-06T11:53:57.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:57.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:57.370573+0000) 2024-03-06T11:53:58.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:58.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:58.370804+0000) 2024-03-06T11:53:59.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:53:59.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:53:59.371043+0000) 2024-03-06T11:54:00.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:00.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:00.371296+0000) 2024-03-06T11:54:01.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:01.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:01.371524+0000) 2024-03-06T11:54:02.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:02.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:02.371759+0000) 2024-03-06T11:54:03.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:03.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:03.372043+0000) 2024-03-06T11:54:04.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:04.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:04.372273+0000) 2024-03-06T11:54:05.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:05.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:05.372447+0000) 2024-03-06T11:54:06.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:06.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:06.372667+0000) 2024-03-06T11:54:07.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:07.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:07.372876+0000) 2024-03-06T11:54:08.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:08.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:08.373114+0000) 2024-03-06T11:54:09.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:09.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:09.373342+0000) 2024-03-06T11:54:10.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:10.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:10.373541+0000) 2024-03-06T11:54:11.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:11.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:11.373758+0000) 2024-03-06T11:54:12.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:12.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:12.373992+0000) 2024-03-06T11:54:13.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:13.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:13.374279+0000) 2024-03-06T11:54:14.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:14.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:14.374515+0000) 2024-03-06T11:54:15.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:15.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:15.374718+0000) 2024-03-06T11:54:16.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:16.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:16.374947+0000) 2024-03-06T11:54:17.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:17.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:17.375167+0000) 2024-03-06T11:54:18.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:18.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:18.375411+0000) 2024-03-06T11:54:19.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:19.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:19.375652+0000) 2024-03-06T11:54:20.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:20.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:20.375888+0000) 2024-03-06T11:54:21.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:21.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:21.376127+0000) 2024-03-06T11:54:22.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:22.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:22.376374+0000) 2024-03-06T11:54:23.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:23.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:23.376578+0000) 2024-03-06T11:54:24.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:24.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:24.376830+0000) 2024-03-06T11:54:25.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:25.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:25.377109+0000) 2024-03-06T11:54:26.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:26.100 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:26.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:26.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:26.377328+0000) 2024-03-06T11:54:27.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:27.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:27.377521+0000) 2024-03-06T11:54:28.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:28.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:28.377717+0000) 2024-03-06T11:54:29.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:29.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:29.377943+0000) 2024-03-06T11:54:30.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:30.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:30.378245+0000) 2024-03-06T11:54:31.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:31.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:31.378456+0000) 2024-03-06T11:54:32.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:32.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:32.378714+0000) 2024-03-06T11:54:33.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:33.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:33.378931+0000) 2024-03-06T11:54:34.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:34.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:34.379164+0000) 2024-03-06T11:54:35.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:35.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:35.379384+0000) 2024-03-06T11:54:36.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:36.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:36.379597+0000) 2024-03-06T11:54:37.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:37.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:37.379815+0000) 2024-03-06T11:54:38.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:38.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:38.380097+0000) 2024-03-06T11:54:39.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:39.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:39.380320+0000) 2024-03-06T11:54:40.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:40.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:40.380556+0000) 2024-03-06T11:54:41.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:41.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:41.380774+0000) 2024-03-06T11:54:42.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:42.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:42.381036+0000) 2024-03-06T11:54:43.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:43.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:43.381277+0000) 2024-03-06T11:54:44.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:44.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:44.381631+0000) 2024-03-06T11:54:45.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:45.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:45.381865+0000) 2024-03-06T11:54:46.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:46.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:46.382129+0000) 2024-03-06T11:54:47.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:47.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:47.382353+0000) 2024-03-06T11:54:48.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:48.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:48.382590+0000) 2024-03-06T11:54:49.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:49.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:49.382817+0000) 2024-03-06T11:54:50.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:50.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:50.383071+0000) 2024-03-06T11:54:51.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:51.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:51.383294+0000) 2024-03-06T11:54:52.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:52.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:52.383495+0000) 2024-03-06T11:54:53.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:53.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:53.383732+0000) 2024-03-06T11:54:54.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:54.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:54.383992+0000) 2024-03-06T11:54:55.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:55.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:55.384277+0000) 2024-03-06T11:54:56.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:56.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:54:56.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:56.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:56.384457+0000) 2024-03-06T11:54:57.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:57.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:57.384683+0000) 2024-03-06T11:54:58.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:58.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:58.384866+0000) 2024-03-06T11:54:59.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:54:59.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:54:59.385162+0000) 2024-03-06T11:55:00.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:00.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:00.385402+0000) 2024-03-06T11:55:01.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:01.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:01.385554+0000) 2024-03-06T11:55:02.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:02.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:02.385784+0000) 2024-03-06T11:55:03.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:03.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:03.386031+0000) 2024-03-06T11:55:04.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:04.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:04.386263+0000) 2024-03-06T11:55:05.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:05.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:05.386487+0000) 2024-03-06T11:55:06.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:06.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:06.386715+0000) 2024-03-06T11:55:07.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:07.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:07.386952+0000) 2024-03-06T11:55:08.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:08.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:08.387172+0000) 2024-03-06T11:55:09.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:09.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:09.387364+0000) 2024-03-06T11:55:10.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:10.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:10.387617+0000) 2024-03-06T11:55:11.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:11.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:11.387843+0000) 2024-03-06T11:55:12.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:12.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:12.388134+0000) 2024-03-06T11:55:13.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:13.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:13.388364+0000) 2024-03-06T11:55:14.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:14.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:14.388602+0000) 2024-03-06T11:55:15.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:15.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:15.388849+0000) 2024-03-06T11:55:16.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:16.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:16.389125+0000) 2024-03-06T11:55:17.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:17.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:17.389362+0000) 2024-03-06T11:55:18.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:18.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:18.389614+0000) 2024-03-06T11:55:19.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:19.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:19.389854+0000) 2024-03-06T11:55:20.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:20.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:20.390129+0000) 2024-03-06T11:55:21.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:21.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:21.390363+0000) 2024-03-06T11:55:22.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:22.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:22.390615+0000) 2024-03-06T11:55:23.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:23.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:23.390849+0000) 2024-03-06T11:55:24.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:24.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:24.391131+0000) 2024-03-06T11:55:25.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:25.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:25.391362+0000) 2024-03-06T11:55:26.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:26.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:26.391608+0000) 2024-03-06T11:55:26.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:26.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:27.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:27.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:27.391789+0000) 2024-03-06T11:55:28.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:28.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:28.392073+0000) 2024-03-06T11:55:29.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:29.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:29.392317+0000) 2024-03-06T11:55:30.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:30.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:30.392563+0000) 2024-03-06T11:55:31.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:31.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:31.392806+0000) 2024-03-06T11:55:32.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:32.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:32.393078+0000) 2024-03-06T11:55:33.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:33.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:33.393275+0000) 2024-03-06T11:55:34.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:34.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:34.393501+0000) 2024-03-06T11:55:35.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:35.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:35.393716+0000) 2024-03-06T11:55:36.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:36.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:36.393951+0000) 2024-03-06T11:55:37.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:37.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:37.394166+0000) 2024-03-06T11:55:38.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:38.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:38.394383+0000) 2024-03-06T11:55:39.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:39.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:39.394641+0000) 2024-03-06T11:55:40.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:40.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:40.394870+0000) 2024-03-06T11:55:41.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:41.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:41.395121+0000) 2024-03-06T11:55:42.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:42.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:42.395355+0000) 2024-03-06T11:55:43.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:43.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:43.395579+0000) 2024-03-06T11:55:44.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:44.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:44.395765+0000) 2024-03-06T11:55:45.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:45.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:45.395984+0000) 2024-03-06T11:55:46.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:46.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:46.396250+0000) 2024-03-06T11:55:47.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:47.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:47.396478+0000) 2024-03-06T11:55:48.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:48.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:48.396736+0000) 2024-03-06T11:55:49.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:49.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:49.396968+0000) 2024-03-06T11:55:50.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:50.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:50.397219+0000) 2024-03-06T11:55:51.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:51.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:51.397453+0000) 2024-03-06T11:55:52.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:52.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:52.397705+0000) 2024-03-06T11:55:53.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:53.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:53.397910+0000) 2024-03-06T11:55:54.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:54.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:54.398167+0000) 2024-03-06T11:55:55.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:55.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:55.398409+0000) 2024-03-06T11:55:56.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:56.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:56.398637+0000) 2024-03-06T11:55:56.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:56.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:55:57.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:57.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:57.398815+0000) 2024-03-06T11:55:58.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:58.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:58.399095+0000) 2024-03-06T11:55:59.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:55:59.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:55:59.399356+0000) 2024-03-06T11:56:00.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:00.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:00.399599+0000) 2024-03-06T11:56:01.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:01.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:01.399833+0000) 2024-03-06T11:56:02.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:02.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:02.400126+0000) 2024-03-06T11:56:03.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:03.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:03.400352+0000) 2024-03-06T11:56:04.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:04.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:04.400588+0000) 2024-03-06T11:56:05.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:05.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:05.400820+0000) 2024-03-06T11:56:06.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:06.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:06.401106+0000) 2024-03-06T11:56:07.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:07.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:07.401351+0000) 2024-03-06T11:56:08.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:08.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:08.401562+0000) 2024-03-06T11:56:09.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:09.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:09.401818+0000) 2024-03-06T11:56:10.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:10.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:10.402106+0000) 2024-03-06T11:56:11.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:11.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:11.402326+0000) 2024-03-06T11:56:12.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:12.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:12.402560+0000) 2024-03-06T11:56:13.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:13.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:13.402816+0000) 2024-03-06T11:56:14.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:14.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:14.403083+0000) 2024-03-06T11:56:15.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:15.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:15.403299+0000) 2024-03-06T11:56:16.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:16.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:16.403509+0000) 2024-03-06T11:56:17.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:17.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:17.403726+0000) 2024-03-06T11:56:18.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:18.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:18.403952+0000) 2024-03-06T11:56:19.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:19.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:19.404328+0000) 2024-03-06T11:56:20.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:20.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:20.404545+0000) 2024-03-06T11:56:21.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:21.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:21.404745+0000) 2024-03-06T11:56:22.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:22.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:22.404982+0000) 2024-03-06T11:56:23.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:23.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:23.405247+0000) 2024-03-06T11:56:24.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:24.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:24.405481+0000) 2024-03-06T11:56:25.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:25.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:25.405692+0000) 2024-03-06T11:56:26.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:26.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:26.405910+0000) 2024-03-06T11:56:27.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:27.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:27.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:27.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:27.406155+0000) 2024-03-06T11:56:28.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:28.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:28.406382+0000) 2024-03-06T11:56:29.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:29.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:29.406611+0000) 2024-03-06T11:56:30.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:30.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:30.406840+0000) 2024-03-06T11:56:31.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:31.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:31.407095+0000) 2024-03-06T11:56:32.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:32.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:32.407341+0000) 2024-03-06T11:56:33.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:33.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:33.407580+0000) 2024-03-06T11:56:34.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:34.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:34.407812+0000) 2024-03-06T11:56:35.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:35.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:35.408077+0000) 2024-03-06T11:56:36.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:36.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:36.408315+0000) 2024-03-06T11:56:37.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:37.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:37.408534+0000) 2024-03-06T11:56:38.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:38.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:38.408734+0000) 2024-03-06T11:56:39.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:39.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:39.408962+0000) 2024-03-06T11:56:40.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:40.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:40.409169+0000) 2024-03-06T11:56:41.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:41.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:41.409390+0000) 2024-03-06T11:56:42.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:42.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:42.409623+0000) 2024-03-06T11:56:43.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:43.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:43.409800+0000) 2024-03-06T11:56:44.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:44.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:44.410058+0000) 2024-03-06T11:56:45.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:45.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:45.410287+0000) 2024-03-06T11:56:46.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:46.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:46.410516+0000) 2024-03-06T11:56:47.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:47.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:47.410734+0000) 2024-03-06T11:56:48.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:48.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:48.410964+0000) 2024-03-06T11:56:49.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:49.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:49.411201+0000) 2024-03-06T11:56:50.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:50.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:50.411437+0000) 2024-03-06T11:56:51.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:51.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:51.411655+0000) 2024-03-06T11:56:52.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:52.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:52.411894+0000) 2024-03-06T11:56:53.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:53.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:53.412103+0000) 2024-03-06T11:56:54.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:54.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:54.412305+0000) 2024-03-06T11:56:55.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:55.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:55.412518+0000) 2024-03-06T11:56:56.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:56.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:56.412735+0000) 2024-03-06T11:56:57.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:57.171 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:56:57.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:57.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:57.412925+0000) 2024-03-06T11:56:58.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:58.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:58.413162+0000) 2024-03-06T11:56:59.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:56:59.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:56:59.413393+0000) 2024-03-06T11:57:00.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:00.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:00.413624+0000) 2024-03-06T11:57:01.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:01.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:01.413851+0000) 2024-03-06T11:57:02.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:02.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:02.414147+0000) 2024-03-06T11:57:03.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:03.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:03.414370+0000) 2024-03-06T11:57:04.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:04.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:04.414568+0000) 2024-03-06T11:57:05.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:05.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:05.414795+0000) 2024-03-06T11:57:06.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:06.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:06.414975+0000) 2024-03-06T11:57:07.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:07.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:07.415246+0000) 2024-03-06T11:57:08.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:08.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:08.415495+0000) 2024-03-06T11:57:09.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:09.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:09.415752+0000) 2024-03-06T11:57:10.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:10.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:10.415986+0000) 2024-03-06T11:57:11.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:11.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:11.416284+0000) 2024-03-06T11:57:12.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:12.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:12.416518+0000) 2024-03-06T11:57:13.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:13.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:13.416749+0000) 2024-03-06T11:57:14.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:14.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:14.417039+0000) 2024-03-06T11:57:15.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:15.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:15.417277+0000) 2024-03-06T11:57:16.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:16.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:16.417480+0000) 2024-03-06T11:57:17.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:17.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:17.417735+0000) 2024-03-06T11:57:18.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:18.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:18.417980+0000) 2024-03-06T11:57:19.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:19.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:19.418234+0000) 2024-03-06T11:57:20.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:20.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:20.418464+0000) 2024-03-06T11:57:21.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:21.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:21.418683+0000) 2024-03-06T11:57:22.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:22.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:22.418910+0000) 2024-03-06T11:57:23.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:23.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:23.419155+0000) 2024-03-06T11:57:24.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:24.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:24.419388+0000) 2024-03-06T11:57:25.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:25.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:25.419611+0000) 2024-03-06T11:57:26.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:26.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:26.419832+0000) 2024-03-06T11:57:27.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:27.333 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:27.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:27.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:27.420056+0000) 2024-03-06T11:57:28.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:28.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:28.420304+0000) 2024-03-06T11:57:29.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:29.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:29.420533+0000) 2024-03-06T11:57:30.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:30.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:30.420759+0000) 2024-03-06T11:57:31.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:31.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:31.420952+0000) 2024-03-06T11:57:32.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:32.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:32.421212+0000) 2024-03-06T11:57:33.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:33.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:33.421406+0000) 2024-03-06T11:57:34.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:34.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:34.421596+0000) 2024-03-06T11:57:35.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:35.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:35.421828+0000) 2024-03-06T11:57:36.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:36.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:36.422118+0000) 2024-03-06T11:57:37.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:37.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:37.422328+0000) 2024-03-06T11:57:38.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:38.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:38.422569+0000) 2024-03-06T11:57:39.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:39.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:39.422826+0000) 2024-03-06T11:57:40.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:40.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:40.423130+0000) 2024-03-06T11:57:41.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:41.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:41.423348+0000) 2024-03-06T11:57:42.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:42.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:42.423602+0000) 2024-03-06T11:57:43.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:43.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:43.423837+0000) 2024-03-06T11:57:44.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:44.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:44.424128+0000) 2024-03-06T11:57:45.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:45.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:45.424341+0000) 2024-03-06T11:57:46.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:46.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:46.424578+0000) 2024-03-06T11:57:47.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:47.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:47.424798+0000) 2024-03-06T11:57:48.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:48.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:48.425055+0000) 2024-03-06T11:57:49.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:49.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:49.425285+0000) 2024-03-06T11:57:50.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:50.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:50.425470+0000) 2024-03-06T11:57:51.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:51.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:51.425707+0000) 2024-03-06T11:57:52.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:52.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:52.425951+0000) 2024-03-06T11:57:53.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:53.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:53.426158+0000) 2024-03-06T11:57:54.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:54.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:54.426357+0000) 2024-03-06T11:57:55.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:55.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:55.426651+0000) 2024-03-06T11:57:56.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:56.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:56.426885+0000) 2024-03-06T11:57:57.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:57.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:57.427115+0000) 2024-03-06T11:57:57.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:57.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:57:58.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:58.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:58.427312+0000) 2024-03-06T11:57:59.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:57:59.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:57:59.427563+0000) 2024-03-06T11:58:00.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:00.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:00.427783+0000) 2024-03-06T11:58:01.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:01.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:01.428044+0000) 2024-03-06T11:58:02.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:02.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:02.428336+0000) 2024-03-06T11:58:03.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:03.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:03.428487+0000) 2024-03-06T11:58:04.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:04.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:04.428689+0000) 2024-03-06T11:58:05.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:05.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:05.428914+0000) 2024-03-06T11:58:06.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:06.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:06.429157+0000) 2024-03-06T11:58:07.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:07.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:07.429354+0000) 2024-03-06T11:58:08.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:08.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:08.429588+0000) 2024-03-06T11:58:09.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:09.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:09.429806+0000) 2024-03-06T11:58:10.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:10.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:10.430114+0000) 2024-03-06T11:58:11.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:11.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:11.430291+0000) 2024-03-06T11:58:12.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:12.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:12.430528+0000) 2024-03-06T11:58:13.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:13.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:13.430754+0000) 2024-03-06T11:58:14.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:14.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:14.430983+0000) 2024-03-06T11:58:15.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:15.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:15.431245+0000) 2024-03-06T11:58:16.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:16.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:16.431427+0000) 2024-03-06T11:58:17.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:17.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:17.431663+0000) 2024-03-06T11:58:18.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:18.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:18.431903+0000) 2024-03-06T11:58:19.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:19.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:19.432147+0000) 2024-03-06T11:58:20.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:20.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:20.432388+0000) 2024-03-06T11:58:21.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:21.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:21.432626+0000) 2024-03-06T11:58:22.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:22.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:22.432883+0000) 2024-03-06T11:58:23.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:23.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:23.433122+0000) 2024-03-06T11:58:24.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:24.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:24.433326+0000) 2024-03-06T11:58:25.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:25.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:25.433557+0000) 2024-03-06T11:58:26.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:26.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:26.433787+0000) 2024-03-06T11:58:27.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:27.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:27.434040+0000) 2024-03-06T11:58:27.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:27.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:28.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:28.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:28.434211+0000) 2024-03-06T11:58:29.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:29.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:29.434451+0000) 2024-03-06T11:58:30.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:30.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:30.434673+0000) 2024-03-06T11:58:31.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:31.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:31.434864+0000) 2024-03-06T11:58:32.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:32.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:32.435127+0000) 2024-03-06T11:58:33.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:33.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:33.435353+0000) 2024-03-06T11:58:34.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:34.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:34.435549+0000) 2024-03-06T11:58:35.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:35.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:35.435725+0000) 2024-03-06T11:58:36.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:36.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:36.435967+0000) 2024-03-06T11:58:37.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:37.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:37.436162+0000) 2024-03-06T11:58:38.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:38.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:38.436370+0000) 2024-03-06T11:58:39.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:39.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:39.436626+0000) 2024-03-06T11:58:40.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:40.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:40.436878+0000) 2024-03-06T11:58:41.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:41.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:41.437118+0000) 2024-03-06T11:58:42.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:42.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:42.437343+0000) 2024-03-06T11:58:43.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:43.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:43.437638+0000) 2024-03-06T11:58:44.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:44.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:44.437886+0000) 2024-03-06T11:58:45.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:45.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:45.438129+0000) 2024-03-06T11:58:46.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:46.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:46.438343+0000) 2024-03-06T11:58:47.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:47.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:47.438572+0000) 2024-03-06T11:58:48.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:48.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:48.438809+0000) 2024-03-06T11:58:49.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:49.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:49.438991+0000) 2024-03-06T11:58:50.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:50.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:50.439254+0000) 2024-03-06T11:58:51.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:51.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:51.439500+0000) 2024-03-06T11:58:52.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:52.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:52.439753+0000) 2024-03-06T11:58:53.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:53.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:53.439987+0000) 2024-03-06T11:58:54.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:54.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:54.440296+0000) 2024-03-06T11:58:55.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:55.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:55.440503+0000) 2024-03-06T11:58:56.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:56.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:56.440720+0000) 2024-03-06T11:58:57.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:57.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:57.440977+0000) 2024-03-06T11:58:57.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:57.895 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:58:58.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:58.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:58.441246+0000) 2024-03-06T11:58:59.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:58:59.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:58:59.441491+0000) 2024-03-06T11:59:00.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:00.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:00.441696+0000) 2024-03-06T11:59:01.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:01.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:01.441924+0000) 2024-03-06T11:59:02.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:02.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:02.442160+0000) 2024-03-06T11:59:03.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:03.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:03.442387+0000) 2024-03-06T11:59:04.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:04.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:04.442629+0000) 2024-03-06T11:59:05.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:05.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:05.442857+0000) 2024-03-06T11:59:06.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:06.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:06.443120+0000) 2024-03-06T11:59:07.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:07.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:07.443356+0000) 2024-03-06T11:59:08.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:08.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:08.443590+0000) 2024-03-06T11:59:09.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:09.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:09.443801+0000) 2024-03-06T11:59:10.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:10.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:10.444101+0000) 2024-03-06T11:59:11.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:11.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:11.444310+0000) 2024-03-06T11:59:12.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:12.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:12.444571+0000) 2024-03-06T11:59:13.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:13.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:13.444792+0000) 2024-03-06T11:59:14.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:14.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:14.445062+0000) 2024-03-06T11:59:15.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:15.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:15.445268+0000) 2024-03-06T11:59:16.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:16.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:16.445494+0000) 2024-03-06T11:59:17.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:17.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:17.445686+0000) 2024-03-06T11:59:18.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:18.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:18.445929+0000) 2024-03-06T11:59:19.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:19.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:19.446174+0000) 2024-03-06T11:59:20.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:20.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:20.446397+0000) 2024-03-06T11:59:21.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:21.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:21.446633+0000) 2024-03-06T11:59:22.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:22.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:22.446885+0000) 2024-03-06T11:59:23.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:23.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:23.447128+0000) 2024-03-06T11:59:24.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:24.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:24.447349+0000) 2024-03-06T11:59:25.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:25.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:25.447593+0000) 2024-03-06T11:59:26.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:26.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:26.447841+0000) 2024-03-06T11:59:27.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:27.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:27.448119+0000) 2024-03-06T11:59:28.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:28.099 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:28.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:28.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:28.448340+0000) 2024-03-06T11:59:29.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:29.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:29.448572+0000) 2024-03-06T11:59:30.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:30.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:30.448785+0000) 2024-03-06T11:59:31.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:31.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:31.449086+0000) 2024-03-06T11:59:32.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:32.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:32.449319+0000) 2024-03-06T11:59:33.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:33.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:33.449536+0000) 2024-03-06T11:59:34.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:34.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:34.449781+0000) 2024-03-06T11:59:35.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:35.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:35.450050+0000) 2024-03-06T11:59:36.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:36.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:36.450298+0000) 2024-03-06T11:59:37.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:37.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:37.450515+0000) 2024-03-06T11:59:38.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:38.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:38.450739+0000) 2024-03-06T11:59:39.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:39.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:39.450938+0000) 2024-03-06T11:59:40.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:40.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:40.451171+0000) 2024-03-06T11:59:41.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:41.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:41.451399+0000) 2024-03-06T11:59:42.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:42.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:42.451633+0000) 2024-03-06T11:59:43.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:43.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:43.451868+0000) 2024-03-06T11:59:44.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:44.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:44.452127+0000) 2024-03-06T11:59:45.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:45.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:45.452341+0000) 2024-03-06T11:59:46.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:46.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:46.452582+0000) 2024-03-06T11:59:47.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:47.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:47.452831+0000) 2024-03-06T11:59:48.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:48.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:48.453137+0000) 2024-03-06T11:59:49.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:49.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:49.453375+0000) 2024-03-06T11:59:50.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:50.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:50.453621+0000) 2024-03-06T11:59:51.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:51.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:51.453859+0000) 2024-03-06T11:59:52.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:52.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:52.454109+0000) 2024-03-06T11:59:53.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:53.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:53.454330+0000) 2024-03-06T11:59:54.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:54.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:54.454571+0000) 2024-03-06T11:59:55.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:55.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:55.454805+0000) 2024-03-06T11:59:56.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:56.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:56.455074+0000) 2024-03-06T11:59:57.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:57.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:57.455304+0000) 2024-03-06T11:59:58.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:58.329 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T11:59:58.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:58.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:58.455503+0000) 2024-03-06T11:59:59.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T11:59:59.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T10:59:59.455737+0000) 2024-03-06T12:00:00.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:00.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:00.455967+0000) 2024-03-06T12:00:01.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:01.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:01.456197+0000) 2024-03-06T12:00:02.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:02.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:02.456403+0000) 2024-03-06T12:00:03.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:03.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:03.456640+0000) 2024-03-06T12:00:04.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:04.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:04.456905+0000) 2024-03-06T12:00:05.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:05.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:05.457156+0000) 2024-03-06T12:00:06.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:06.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:06.457348+0000) 2024-03-06T12:00:07.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:07.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:07.457566+0000) 2024-03-06T12:00:08.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:08.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:08.457803+0000) 2024-03-06T12:00:09.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:09.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:09.458131+0000) 2024-03-06T12:00:10.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:10.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:10.458356+0000) 2024-03-06T12:00:11.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:11.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:11.458613+0000) 2024-03-06T12:00:12.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:12.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:12.458835+0000) 2024-03-06T12:00:13.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:13.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:13.459086+0000) 2024-03-06T12:00:14.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:14.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:14.459268+0000) 2024-03-06T12:00:15.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:15.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:15.459493+0000) 2024-03-06T12:00:16.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:16.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:16.459723+0000) 2024-03-06T12:00:17.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:17.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:17.459950+0000) 2024-03-06T12:00:18.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:18.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:18.460178+0000) 2024-03-06T12:00:19.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:19.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:19.460429+0000) 2024-03-06T12:00:20.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:20.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:20.460665+0000) 2024-03-06T12:00:21.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:21.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:21.460888+0000) 2024-03-06T12:00:22.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:22.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:22.461126+0000) 2024-03-06T12:00:23.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:23.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:23.461302+0000) 2024-03-06T12:00:24.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:24.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:24.461545+0000) 2024-03-06T12:00:25.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:25.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:25.461782+0000) 2024-03-06T12:00:26.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:26.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:26.462072+0000) 2024-03-06T12:00:27.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:27.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:27.462322+0000) 2024-03-06T12:00:28.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:28.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:28.462574+0000) 2024-03-06T12:00:28.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:28.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:29.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:29.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:29.462852+0000) 2024-03-06T12:00:30.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:30.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:30.463220+0000) 2024-03-06T12:00:31.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:31.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:31.463501+0000) 2024-03-06T12:00:32.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:32.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:32.463814+0000) 2024-03-06T12:00:33.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:33.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:33.464126+0000) 2024-03-06T12:00:34.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:34.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:34.464371+0000) 2024-03-06T12:00:35.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:35.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:35.464648+0000) 2024-03-06T12:00:36.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:36.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:36.464811+0000) 2024-03-06T12:00:37.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:37.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:37.465127+0000) 2024-03-06T12:00:38.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:38.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:38.465344+0000) 2024-03-06T12:00:39.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:39.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:39.465592+0000) 2024-03-06T12:00:40.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:40.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:40.465836+0000) 2024-03-06T12:00:41.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:41.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:41.466136+0000) 2024-03-06T12:00:42.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:42.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:42.466367+0000) 2024-03-06T12:00:43.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:43.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:43.466610+0000) 2024-03-06T12:00:44.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:44.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:44.466858+0000) 2024-03-06T12:00:45.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:45.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:45.467135+0000) 2024-03-06T12:00:46.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:46.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:46.467376+0000) 2024-03-06T12:00:47.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:47.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:47.467627+0000) 2024-03-06T12:00:48.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:48.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:48.467836+0000) 2024-03-06T12:00:49.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:49.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:49.468116+0000) 2024-03-06T12:00:50.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:50.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:50.468300+0000) 2024-03-06T12:00:51.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:51.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:51.468550+0000) 2024-03-06T12:00:52.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:52.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:52.468766+0000) 2024-03-06T12:00:53.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:53.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:53.468989+0000) 2024-03-06T12:00:54.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:54.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:54.469230+0000) 2024-03-06T12:00:55.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:55.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:55.469436+0000) 2024-03-06T12:00:56.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:56.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:56.469660+0000) 2024-03-06T12:00:57.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:57.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:57.469894+0000) 2024-03-06T12:00:58.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:58.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:58.470111+0000) 2024-03-06T12:00:58.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:58.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:00:59.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:00:59.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:00:59.470277+0000) 2024-03-06T12:01:00.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:00.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:00.470502+0000) 2024-03-06T12:01:01.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:01.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:01.470722+0000) 2024-03-06T12:01:02.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:02.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:02.470958+0000) 2024-03-06T12:01:03.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:03.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:03.471163+0000) 2024-03-06T12:01:04.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:04.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:04.471391+0000) 2024-03-06T12:01:05.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:05.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:05.471614+0000) 2024-03-06T12:01:06.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:06.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:06.471839+0000) 2024-03-06T12:01:07.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:07.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:07.472111+0000) 2024-03-06T12:01:08.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:08.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:08.472305+0000) 2024-03-06T12:01:09.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:09.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:09.472525+0000) 2024-03-06T12:01:10.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:10.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:10.472717+0000) 2024-03-06T12:01:11.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:11.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:11.472943+0000) 2024-03-06T12:01:12.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:12.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:12.473145+0000) 2024-03-06T12:01:13.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:13.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:13.473333+0000) 2024-03-06T12:01:14.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:14.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:14.473552+0000) 2024-03-06T12:01:15.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:15.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:15.473780+0000) 2024-03-06T12:01:16.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:16.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:16.474070+0000) 2024-03-06T12:01:17.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:17.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:17.474333+0000) 2024-03-06T12:01:18.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:18.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:18.474651+0000) 2024-03-06T12:01:19.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:19.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:19.474865+0000) 2024-03-06T12:01:20.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:20.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:20.475149+0000) 2024-03-06T12:01:21.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:21.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:21.475377+0000) 2024-03-06T12:01:22.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:22.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:22.475677+0000) 2024-03-06T12:01:23.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:23.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:23.475930+0000) 2024-03-06T12:01:24.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:24.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:24.476194+0000) 2024-03-06T12:01:25.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:25.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:25.476433+0000) 2024-03-06T12:01:26.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:26.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:26.476680+0000) 2024-03-06T12:01:27.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:27.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:27.476907+0000) 2024-03-06T12:01:28.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:28.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:28.477145+0000) 2024-03-06T12:01:28.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:28.884 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:29.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:29.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:29.477383+0000) 2024-03-06T12:01:30.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:30.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:30.477610+0000) 2024-03-06T12:01:31.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:31.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:31.477868+0000) 2024-03-06T12:01:32.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:32.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:32.478158+0000) 2024-03-06T12:01:33.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:33.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:33.478442+0000) 2024-03-06T12:01:34.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:34.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:34.478713+0000) 2024-03-06T12:01:35.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:35.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:35.478918+0000) 2024-03-06T12:01:36.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:36.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:36.479113+0000) 2024-03-06T12:01:37.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:37.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:37.479382+0000) 2024-03-06T12:01:38.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:38.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:38.479622+0000) 2024-03-06T12:01:39.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:39.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:39.479853+0000) 2024-03-06T12:01:40.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:40.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:40.480152+0000) 2024-03-06T12:01:41.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:41.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:41.480327+0000) 2024-03-06T12:01:42.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:42.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:42.480562+0000) 2024-03-06T12:01:43.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:43.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:43.480828+0000) 2024-03-06T12:01:44.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:44.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:44.481146+0000) 2024-03-06T12:01:45.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:45.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:45.481399+0000) 2024-03-06T12:01:46.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:46.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:46.481687+0000) 2024-03-06T12:01:47.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:47.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:47.481972+0000) 2024-03-06T12:01:48.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:48.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:48.482268+0000) 2024-03-06T12:01:49.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:49.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:49.482551+0000) 2024-03-06T12:01:50.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:50.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:50.482804+0000) 2024-03-06T12:01:51.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:51.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:51.483123+0000) 2024-03-06T12:01:52.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:52.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:52.483361+0000) 2024-03-06T12:01:53.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:53.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:53.483645+0000) 2024-03-06T12:01:54.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:54.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:54.483932+0000) 2024-03-06T12:01:55.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:55.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:55.484164+0000) 2024-03-06T12:01:56.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:56.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:56.484360+0000) 2024-03-06T12:01:57.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:57.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:57.484565+0000) 2024-03-06T12:01:58.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:58.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:58.484771+0000) 2024-03-06T12:01:59.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:59.110 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:01:59.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:01:59.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:01:59.484975+0000) 2024-03-06T12:02:00.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:00.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:00.485181+0000) 2024-03-06T12:02:01.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:01.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:01.485395+0000) 2024-03-06T12:02:02.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:02.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:02.485630+0000) 2024-03-06T12:02:03.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:03.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:03.485866+0000) 2024-03-06T12:02:04.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:04.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:04.486135+0000) 2024-03-06T12:02:05.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:05.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:05.486275+0000) 2024-03-06T12:02:06.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:06.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:06.486491+0000) 2024-03-06T12:02:07.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:07.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:07.486722+0000) 2024-03-06T12:02:08.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:08.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:08.486948+0000) 2024-03-06T12:02:09.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:09.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:09.487177+0000) 2024-03-06T12:02:10.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:10.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:10.487406+0000) 2024-03-06T12:02:11.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:11.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:11.487603+0000) 2024-03-06T12:02:12.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:12.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:12.487843+0000) 2024-03-06T12:02:13.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:13.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:13.488095+0000) 2024-03-06T12:02:14.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:14.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:14.488319+0000) 2024-03-06T12:02:15.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:15.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:15.488547+0000) 2024-03-06T12:02:16.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:16.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:16.488777+0000) 2024-03-06T12:02:17.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:17.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:17.489000+0000) 2024-03-06T12:02:18.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:18.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:18.489217+0000) 2024-03-06T12:02:19.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:19.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:19.489437+0000) 2024-03-06T12:02:20.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:20.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:20.489654+0000) 2024-03-06T12:02:21.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:21.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:21.489848+0000) 2024-03-06T12:02:22.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:22.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:22.490121+0000) 2024-03-06T12:02:23.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:23.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:23.490316+0000) 2024-03-06T12:02:24.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:24.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:24.490522+0000) 2024-03-06T12:02:25.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:25.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:25.490704+0000) 2024-03-06T12:02:26.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:26.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:26.490939+0000) 2024-03-06T12:02:27.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:27.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:27.491172+0000) 2024-03-06T12:02:28.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:28.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:28.491407+0000) 2024-03-06T12:02:29.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:29.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:29.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:29.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:29.491647+0000) 2024-03-06T12:02:30.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:30.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:30.491850+0000) 2024-03-06T12:02:31.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:31.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:31.492110+0000) 2024-03-06T12:02:32.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:32.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:32.492349+0000) 2024-03-06T12:02:33.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:33.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:33.492570+0000) 2024-03-06T12:02:34.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:34.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:34.492791+0000) 2024-03-06T12:02:35.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:35.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:35.493055+0000) 2024-03-06T12:02:36.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:36.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:36.493266+0000) 2024-03-06T12:02:37.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:37.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:37.493493+0000) 2024-03-06T12:02:38.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:38.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:38.493722+0000) 2024-03-06T12:02:39.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:39.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:39.493946+0000) 2024-03-06T12:02:40.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:40.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:40.494183+0000) 2024-03-06T12:02:41.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:41.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:41.494450+0000) 2024-03-06T12:02:42.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:42.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:42.494733+0000) 2024-03-06T12:02:43.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:43.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:43.494988+0000) 2024-03-06T12:02:44.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:44.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:44.495302+0000) 2024-03-06T12:02:45.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:45.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:45.495552+0000) 2024-03-06T12:02:46.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:46.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:46.495776+0000) 2024-03-06T12:02:47.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:47.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:47.495970+0000) 2024-03-06T12:02:48.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:48.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:48.496209+0000) 2024-03-06T12:02:49.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:49.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:49.496433+0000) 2024-03-06T12:02:50.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:50.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:50.496632+0000) 2024-03-06T12:02:51.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:51.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:51.496868+0000) 2024-03-06T12:02:52.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:52.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:52.497131+0000) 2024-03-06T12:02:53.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:53.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:53.497355+0000) 2024-03-06T12:02:54.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:54.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:54.497647+0000) 2024-03-06T12:02:55.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:55.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:55.497898+0000) 2024-03-06T12:02:56.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:56.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:56.498139+0000) 2024-03-06T12:02:57.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:57.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:57.498330+0000) 2024-03-06T12:02:58.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:58.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:58.498548+0000) 2024-03-06T12:02:59.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:02:59.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:02:59.498776+0000) 2024-03-06T12:02:59.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:02:59.556 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:00.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:00.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:00.499065+0000) 2024-03-06T12:03:01.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:01.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:01.499263+0000) 2024-03-06T12:03:02.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:02.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:02.499498+0000) 2024-03-06T12:03:03.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:03.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:03.499722+0000) 2024-03-06T12:03:04.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:04.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:04.499951+0000) 2024-03-06T12:03:05.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:05.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:05.500160+0000) 2024-03-06T12:03:06.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:06.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:06.500361+0000) 2024-03-06T12:03:07.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:07.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:07.500589+0000) 2024-03-06T12:03:08.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:08.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:08.500819+0000) 2024-03-06T12:03:09.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:09.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:09.501047+0000) 2024-03-06T12:03:10.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:10.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:10.501271+0000) 2024-03-06T12:03:11.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:11.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:11.501480+0000) 2024-03-06T12:03:12.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:12.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:12.501683+0000) 2024-03-06T12:03:13.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:13.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:13.501881+0000) 2024-03-06T12:03:14.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:14.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:14.502136+0000) 2024-03-06T12:03:15.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:15.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:15.502333+0000) 2024-03-06T12:03:16.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:16.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:16.502535+0000) 2024-03-06T12:03:17.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:17.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:17.502712+0000) 2024-03-06T12:03:18.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:18.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:18.502945+0000) 2024-03-06T12:03:19.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:19.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:19.503164+0000) 2024-03-06T12:03:20.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:20.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:20.503368+0000) 2024-03-06T12:03:21.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:21.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:21.503591+0000) 2024-03-06T12:03:22.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:22.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:22.503811+0000) 2024-03-06T12:03:23.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:23.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:23.504130+0000) 2024-03-06T12:03:24.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:24.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:24.504370+0000) 2024-03-06T12:03:25.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:25.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:25.504552+0000) 2024-03-06T12:03:26.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:26.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:26.504777+0000) 2024-03-06T12:03:27.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:27.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:27.504997+0000) 2024-03-06T12:03:28.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:28.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:28.505293+0000) 2024-03-06T12:03:29.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:29.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:29.505534+0000) 2024-03-06T12:03:29.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:29.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:30.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:30.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:30.505721+0000) 2024-03-06T12:03:31.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:31.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:31.505944+0000) 2024-03-06T12:03:32.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:32.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:32.506193+0000) 2024-03-06T12:03:33.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:33.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:33.506428+0000) 2024-03-06T12:03:34.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:34.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:34.506674+0000) 2024-03-06T12:03:35.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:35.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:35.506862+0000) 2024-03-06T12:03:36.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:36.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:36.507123+0000) 2024-03-06T12:03:37.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:37.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:37.507350+0000) 2024-03-06T12:03:38.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:38.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:38.507529+0000) 2024-03-06T12:03:39.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:39.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:39.507756+0000) 2024-03-06T12:03:40.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:40.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:40.507994+0000) 2024-03-06T12:03:41.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:41.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:41.508249+0000) 2024-03-06T12:03:42.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:42.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:42.508483+0000) 2024-03-06T12:03:43.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:43.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:43.508724+0000) 2024-03-06T12:03:44.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:44.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:44.508980+0000) 2024-03-06T12:03:45.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:45.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:45.509272+0000) 2024-03-06T12:03:46.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:46.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:46.509512+0000) 2024-03-06T12:03:47.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:47.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:47.509802+0000) 2024-03-06T12:03:48.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:48.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:48.510104+0000) 2024-03-06T12:03:49.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:49.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:49.510278+0000) 2024-03-06T12:03:50.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:50.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:50.510509+0000) 2024-03-06T12:03:51.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:51.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:51.510736+0000) 2024-03-06T12:03:52.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:52.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:52.510972+0000) 2024-03-06T12:03:53.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:53.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:53.511161+0000) 2024-03-06T12:03:54.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:54.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:54.511355+0000) 2024-03-06T12:03:55.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:55.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:55.511587+0000) 2024-03-06T12:03:56.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:56.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:56.511769+0000) 2024-03-06T12:03:57.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:57.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:57.511992+0000) 2024-03-06T12:03:58.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:58.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:58.512264+0000) 2024-03-06T12:03:59.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:03:59.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:03:59.512491+0000) 2024-03-06T12:03:59.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:03:59.929 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:00.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:00.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:00.512731+0000) 2024-03-06T12:04:01.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:01.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:01.512960+0000) 2024-03-06T12:04:02.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:02.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:02.513201+0000) 2024-03-06T12:04:03.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:03.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:03.513434+0000) 2024-03-06T12:04:04.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:04.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:04.513692+0000) 2024-03-06T12:04:05.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:05.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:05.513922+0000) 2024-03-06T12:04:06.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:06.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:06.514137+0000) 2024-03-06T12:04:07.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:07.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:07.514383+0000) 2024-03-06T12:04:08.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:08.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:08.514633+0000) 2024-03-06T12:04:09.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:09.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:09.514873+0000) 2024-03-06T12:04:10.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:10.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:10.515133+0000) 2024-03-06T12:04:11.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:11.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:11.515358+0000) 2024-03-06T12:04:12.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:12.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:12.515596+0000) 2024-03-06T12:04:13.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:13.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:13.515774+0000) 2024-03-06T12:04:14.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:14.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:14.515997+0000) 2024-03-06T12:04:15.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:15.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:15.516236+0000) 2024-03-06T12:04:16.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:16.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:16.516429+0000) 2024-03-06T12:04:17.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:17.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:17.516630+0000) 2024-03-06T12:04:18.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:18.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:18.516860+0000) 2024-03-06T12:04:19.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:19.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:19.517159+0000) 2024-03-06T12:04:20.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:20.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:20.517389+0000) 2024-03-06T12:04:21.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:21.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:21.517635+0000) 2024-03-06T12:04:22.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:22.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:22.517886+0000) 2024-03-06T12:04:23.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:23.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:23.518142+0000) 2024-03-06T12:04:24.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:24.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:24.518352+0000) 2024-03-06T12:04:25.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:25.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:25.518603+0000) 2024-03-06T12:04:26.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:26.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:26.518850+0000) 2024-03-06T12:04:27.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:27.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:27.519098+0000) 2024-03-06T12:04:28.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:28.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:28.519343+0000) 2024-03-06T12:04:29.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:29.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:29.519599+0000) 2024-03-06T12:04:30.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:30.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:04:30.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:30.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:30.519827+0000) 2024-03-06T12:04:31.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:31.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:31.520112+0000) 2024-03-06T12:04:32.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:32.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:32.520354+0000) 2024-03-06T12:04:33.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:33.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:33.520560+0000) 2024-03-06T12:04:34.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:34.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:34.520762+0000) 2024-03-06T12:04:35.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:35.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:35.520945+0000) 2024-03-06T12:04:36.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:36.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:36.521146+0000) 2024-03-06T12:04:37.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:37.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:37.521359+0000) 2024-03-06T12:04:38.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:38.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:38.521581+0000) 2024-03-06T12:04:39.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:39.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:39.521807+0000) 2024-03-06T12:04:40.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:40.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:40.522194+0000) 2024-03-06T12:04:41.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:41.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:41.522435+0000) 2024-03-06T12:04:42.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:42.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:42.522663+0000) 2024-03-06T12:04:43.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:43.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:43.522888+0000) 2024-03-06T12:04:44.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:44.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:44.523119+0000) 2024-03-06T12:04:45.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:45.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:45.523351+0000) 2024-03-06T12:04:46.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:46.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:46.523539+0000) 2024-03-06T12:04:47.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:47.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:47.523762+0000) 2024-03-06T12:04:48.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:48.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:48.523987+0000) 2024-03-06T12:04:49.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:49.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:49.524228+0000) 2024-03-06T12:04:50.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:50.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:50.524443+0000) 2024-03-06T12:04:51.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:51.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:51.524668+0000) 2024-03-06T12:04:52.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:52.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:52.524894+0000) 2024-03-06T12:04:53.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:53.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:53.525170+0000) 2024-03-06T12:04:54.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:54.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:54.525363+0000) 2024-03-06T12:04:55.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:55.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:55.525593+0000) 2024-03-06T12:04:56.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:56.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:56.525822+0000) 2024-03-06T12:04:57.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:57.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:57.526112+0000) 2024-03-06T12:04:58.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:58.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:58.526330+0000) 2024-03-06T12:04:59.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:04:59.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:04:59.526584+0000) 2024-03-06T12:05:00.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:00.371 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:00.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:00.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:00.526840+0000) 2024-03-06T12:05:01.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:01.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:01.527127+0000) 2024-03-06T12:05:02.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:02.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:02.527383+0000) 2024-03-06T12:05:03.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:03.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:03.527628+0000) 2024-03-06T12:05:04.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:04.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:04.527875+0000) 2024-03-06T12:05:05.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:05.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:05.528139+0000) 2024-03-06T12:05:06.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:06.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:06.528381+0000) 2024-03-06T12:05:07.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:07.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:07.528635+0000) 2024-03-06T12:05:08.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:08.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:08.528870+0000) 2024-03-06T12:05:09.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:09.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:09.529150+0000) 2024-03-06T12:05:10.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:10.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:10.529363+0000) 2024-03-06T12:05:11.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:11.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:11.529588+0000) 2024-03-06T12:05:12.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:12.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:12.529842+0000) 2024-03-06T12:05:13.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:13.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:13.530164+0000) 2024-03-06T12:05:14.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:14.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:14.530382+0000) 2024-03-06T12:05:15.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:15.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:15.530603+0000) 2024-03-06T12:05:16.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:16.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:16.530782+0000) 2024-03-06T12:05:17.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:17.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:17.531049+0000) 2024-03-06T12:05:18.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:18.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:18.531271+0000) 2024-03-06T12:05:19.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:19.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:19.531492+0000) 2024-03-06T12:05:20.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:20.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:20.531712+0000) 2024-03-06T12:05:21.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:21.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:21.531932+0000) 2024-03-06T12:05:22.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:22.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:22.532167+0000) 2024-03-06T12:05:23.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:23.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:23.532404+0000) 2024-03-06T12:05:24.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:24.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:24.532633+0000) 2024-03-06T12:05:25.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:25.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:25.532875+0000) 2024-03-06T12:05:26.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:26.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:26.533138+0000) 2024-03-06T12:05:27.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:27.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:27.533375+0000) 2024-03-06T12:05:28.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:28.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:28.533675+0000) 2024-03-06T12:05:29.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:29.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:29.533897+0000) 2024-03-06T12:05:30.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:30.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:30.534134+0000) 2024-03-06T12:05:30.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:30.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:05:31.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:31.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:31.534367+0000) 2024-03-06T12:05:32.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:32.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:32.534577+0000) 2024-03-06T12:05:33.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:33.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:33.534806+0000) 2024-03-06T12:05:34.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:34.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:34.535060+0000) 2024-03-06T12:05:35.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:35.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:35.535300+0000) 2024-03-06T12:05:36.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:36.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:36.535541+0000) 2024-03-06T12:05:37.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:37.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:37.535765+0000) 2024-03-06T12:05:38.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:38.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:38.535989+0000) 2024-03-06T12:05:39.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:39.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:39.536274+0000) 2024-03-06T12:05:40.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:40.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:40.536515+0000) 2024-03-06T12:05:41.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:41.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:41.536746+0000) 2024-03-06T12:05:42.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:42.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:42.536984+0000) 2024-03-06T12:05:43.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:43.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:43.537267+0000) 2024-03-06T12:05:44.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:44.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:44.537484+0000) 2024-03-06T12:05:45.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:45.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:45.537677+0000) 2024-03-06T12:05:46.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:46.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:46.537902+0000) 2024-03-06T12:05:47.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:47.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:47.538135+0000) 2024-03-06T12:05:48.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:48.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:48.538354+0000) 2024-03-06T12:05:49.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:49.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:49.538590+0000) 2024-03-06T12:05:50.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:50.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:50.538835+0000) 2024-03-06T12:05:51.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:51.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:51.539100+0000) 2024-03-06T12:05:52.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:52.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:52.539329+0000) 2024-03-06T12:05:53.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:53.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:53.539554+0000) 2024-03-06T12:05:54.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:54.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:54.539802+0000) 2024-03-06T12:05:55.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:55.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:55.540093+0000) 2024-03-06T12:05:56.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:56.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:56.540323+0000) 2024-03-06T12:05:57.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:57.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:57.540548+0000) 2024-03-06T12:05:58.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:58.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:58.540730+0000) 2024-03-06T12:05:59.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:05:59.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:05:59.540926+0000) 2024-03-06T12:06:00.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:00.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:00.541193+0000) 2024-03-06T12:06:00.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:00.753 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:01.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:01.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:01.541445+0000) 2024-03-06T12:06:02.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:02.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:02.541640+0000) 2024-03-06T12:06:03.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:03.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:03.541852+0000) 2024-03-06T12:06:04.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:04.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:04.542151+0000) 2024-03-06T12:06:05.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:05.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:05.542428+0000) 2024-03-06T12:06:06.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:06.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:06.542704+0000) 2024-03-06T12:06:07.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:07.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:07.542985+0000) 2024-03-06T12:06:08.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:08.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:08.543264+0000) 2024-03-06T12:06:09.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:09.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:09.543544+0000) 2024-03-06T12:06:10.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:10.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:10.543837+0000) 2024-03-06T12:06:11.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:11.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:11.544163+0000) 2024-03-06T12:06:12.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:12.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:12.544356+0000) 2024-03-06T12:06:13.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:13.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:13.544545+0000) 2024-03-06T12:06:14.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:14.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:14.544778+0000) 2024-03-06T12:06:15.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:15.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:15.544999+0000) 2024-03-06T12:06:16.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:16.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:16.545260+0000) 2024-03-06T12:06:17.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:17.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:17.545468+0000) 2024-03-06T12:06:18.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:18.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:18.545698+0000) 2024-03-06T12:06:19.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:19.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:19.545918+0000) 2024-03-06T12:06:20.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:20.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:20.546173+0000) 2024-03-06T12:06:21.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:21.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:21.546366+0000) 2024-03-06T12:06:22.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:22.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:22.546554+0000) 2024-03-06T12:06:23.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:23.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:23.546777+0000) 2024-03-06T12:06:24.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:24.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:24.546966+0000) 2024-03-06T12:06:25.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:25.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:25.547166+0000) 2024-03-06T12:06:26.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:26.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:26.547381+0000) 2024-03-06T12:06:27.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:27.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:27.547605+0000) 2024-03-06T12:06:28.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:28.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:28.547831+0000) 2024-03-06T12:06:29.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:29.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:29.548115+0000) 2024-03-06T12:06:30.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:30.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:30.548313+0000) 2024-03-06T12:06:30.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:30.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:06:31.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:31.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:31.548542+0000) 2024-03-06T12:06:32.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:32.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:32.548788+0000) 2024-03-06T12:06:33.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:33.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:33.548996+0000) 2024-03-06T12:06:34.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:34.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:34.549203+0000) 2024-03-06T12:06:35.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:35.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:35.549403+0000) 2024-03-06T12:06:36.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:36.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:36.549626+0000) 2024-03-06T12:06:37.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:37.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:37.549837+0000) 2024-03-06T12:06:38.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:38.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:38.550147+0000) 2024-03-06T12:06:39.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:39.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:39.550353+0000) 2024-03-06T12:06:40.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:40.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:40.550538+0000) 2024-03-06T12:06:41.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:41.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:41.550730+0000) 2024-03-06T12:06:42.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:42.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:42.550963+0000) 2024-03-06T12:06:43.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:43.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:43.551180+0000) 2024-03-06T12:06:44.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:44.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:44.551391+0000) 2024-03-06T12:06:45.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:45.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:45.551615+0000) 2024-03-06T12:06:46.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:46.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:46.551842+0000) 2024-03-06T12:06:47.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:47.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:47.552108+0000) 2024-03-06T12:06:48.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:48.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:48.552295+0000) 2024-03-06T12:06:49.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:49.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:49.552521+0000) 2024-03-06T12:06:50.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:50.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:50.552709+0000) 2024-03-06T12:06:51.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:51.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:51.552936+0000) 2024-03-06T12:06:52.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:52.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:52.553173+0000) 2024-03-06T12:06:53.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:53.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:53.553393+0000) 2024-03-06T12:06:54.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:54.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:54.553625+0000) 2024-03-06T12:06:55.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:55.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:55.553851+0000) 2024-03-06T12:06:56.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:56.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:56.554118+0000) 2024-03-06T12:06:57.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:57.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:57.554310+0000) 2024-03-06T12:06:58.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:58.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:58.554535+0000) 2024-03-06T12:06:59.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:06:59.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:06:59.554719+0000) 2024-03-06T12:07:00.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:00.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:00.554960+0000) 2024-03-06T12:07:01.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:01.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:01.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:01.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:01.555226+0000) 2024-03-06T12:07:02.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:02.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:02.555436+0000) 2024-03-06T12:07:03.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:03.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:03.555659+0000) 2024-03-06T12:07:04.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:04.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:04.555881+0000) 2024-03-06T12:07:05.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:05.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:05.556143+0000) 2024-03-06T12:07:06.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:06.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:06.556362+0000) 2024-03-06T12:07:07.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:07.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:07.556585+0000) 2024-03-06T12:07:08.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:08.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:08.556813+0000) 2024-03-06T12:07:09.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:09.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:09.557089+0000) 2024-03-06T12:07:10.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:10.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:10.557301+0000) 2024-03-06T12:07:11.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:11.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:11.557530+0000) 2024-03-06T12:07:12.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:12.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:12.557730+0000) 2024-03-06T12:07:13.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:13.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:13.557970+0000) 2024-03-06T12:07:14.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:14.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:14.558207+0000) 2024-03-06T12:07:15.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:15.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:15.558399+0000) 2024-03-06T12:07:16.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:16.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:16.558626+0000) 2024-03-06T12:07:17.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:17.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:17.558854+0000) 2024-03-06T12:07:18.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:18.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:18.559123+0000) 2024-03-06T12:07:19.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:19.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:19.559359+0000) 2024-03-06T12:07:20.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:20.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:20.559586+0000) 2024-03-06T12:07:21.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:21.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:21.559812+0000) 2024-03-06T12:07:22.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:22.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:22.560114+0000) 2024-03-06T12:07:23.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:23.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:23.560318+0000) 2024-03-06T12:07:24.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:24.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:24.560534+0000) 2024-03-06T12:07:25.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:25.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:25.560759+0000) 2024-03-06T12:07:26.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:26.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:26.560970+0000) 2024-03-06T12:07:27.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:27.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:27.561217+0000) 2024-03-06T12:07:28.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:28.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:28.561436+0000) 2024-03-06T12:07:29.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:29.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:29.561666+0000) 2024-03-06T12:07:30.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:30.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:30.561901+0000) 2024-03-06T12:07:31.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:31.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:07:31.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:31.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:31.562134+0000) 2024-03-06T12:07:32.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:32.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:32.562369+0000) 2024-03-06T12:07:33.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:33.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:33.562556+0000) 2024-03-06T12:07:34.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:34.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:34.562798+0000) 2024-03-06T12:07:35.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:35.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:35.563074+0000) 2024-03-06T12:07:36.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:36.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:36.563311+0000) 2024-03-06T12:07:37.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:37.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:37.563532+0000) 2024-03-06T12:07:38.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:38.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:38.563738+0000) 2024-03-06T12:07:39.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:39.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:39.563871+0000) 2024-03-06T12:07:40.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:40.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:40.564125+0000) 2024-03-06T12:07:41.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:41.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:41.564368+0000) 2024-03-06T12:07:42.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:42.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:42.564653+0000) 2024-03-06T12:07:43.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:43.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:43.564892+0000) 2024-03-06T12:07:44.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:44.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:44.565130+0000) 2024-03-06T12:07:45.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:45.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:45.565352+0000) 2024-03-06T12:07:46.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:46.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:46.565543+0000) 2024-03-06T12:07:47.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:47.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:47.565765+0000) 2024-03-06T12:07:48.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:48.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:48.565990+0000) 2024-03-06T12:07:49.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:49.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:49.566225+0000) 2024-03-06T12:07:50.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:50.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:50.566426+0000) 2024-03-06T12:07:51.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:51.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:51.566656+0000) 2024-03-06T12:07:52.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:52.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:52.566925+0000) 2024-03-06T12:07:53.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:53.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:53.567098+0000) 2024-03-06T12:07:54.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:54.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:54.567324+0000) 2024-03-06T12:07:55.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:55.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:55.567543+0000) 2024-03-06T12:07:56.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:56.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:56.567672+0000) 2024-03-06T12:07:57.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:57.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:57.567870+0000) 2024-03-06T12:07:58.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:58.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:58.568194+0000) 2024-03-06T12:07:59.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:07:59.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:07:59.568429+0000) 2024-03-06T12:08:00.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:00.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:00.568710+0000) 2024-03-06T12:08:01.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:01.542 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:01.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:01.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:01.568962+0000) 2024-03-06T12:08:02.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:02.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:02.569223+0000) 2024-03-06T12:08:03.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:03.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:03.569452+0000) 2024-03-06T12:08:04.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:04.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:04.569681+0000) 2024-03-06T12:08:05.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:05.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:05.569912+0000) 2024-03-06T12:08:06.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:06.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:06.570184+0000) 2024-03-06T12:08:07.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:07.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:07.570410+0000) 2024-03-06T12:08:08.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:08.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:08.570654+0000) 2024-03-06T12:08:09.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:09.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:09.570869+0000) 2024-03-06T12:08:10.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:10.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:10.571145+0000) 2024-03-06T12:08:11.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:11.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:11.571328+0000) 2024-03-06T12:08:12.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:12.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:12.571595+0000) 2024-03-06T12:08:13.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:13.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:13.571856+0000) 2024-03-06T12:08:14.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:14.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:14.572162+0000) 2024-03-06T12:08:15.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:15.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:15.572398+0000) 2024-03-06T12:08:16.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:16.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:16.572680+0000) 2024-03-06T12:08:17.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:17.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:17.572977+0000) 2024-03-06T12:08:18.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:18.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:18.573202+0000) 2024-03-06T12:08:19.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:19.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:19.573458+0000) 2024-03-06T12:08:20.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:20.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:20.573711+0000) 2024-03-06T12:08:21.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:21.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:21.573946+0000) 2024-03-06T12:08:22.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:22.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:22.574169+0000) 2024-03-06T12:08:23.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:23.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:23.574369+0000) 2024-03-06T12:08:24.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:24.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:24.574606+0000) 2024-03-06T12:08:25.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:25.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:25.574834+0000) 2024-03-06T12:08:26.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:26.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:26.575032+0000) 2024-03-06T12:08:27.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:27.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:27.575257+0000) 2024-03-06T12:08:28.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:28.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:28.575440+0000) 2024-03-06T12:08:29.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:29.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:29.575650+0000) 2024-03-06T12:08:30.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:30.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:30.575876+0000) 2024-03-06T12:08:31.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:31.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:31.576138+0000) 2024-03-06T12:08:31.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:31.683 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:08:32.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:32.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:32.576377+0000) 2024-03-06T12:08:33.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:33.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:33.576575+0000) 2024-03-06T12:08:34.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:34.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:34.576796+0000) 2024-03-06T12:08:35.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:35.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:35.577107+0000) 2024-03-06T12:08:36.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:36.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:36.577307+0000) 2024-03-06T12:08:37.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:37.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:37.577532+0000) 2024-03-06T12:08:38.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:38.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:38.577725+0000) 2024-03-06T12:08:39.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:39.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:39.577908+0000) 2024-03-06T12:08:40.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:40.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:40.578140+0000) 2024-03-06T12:08:41.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:41.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:41.578369+0000) 2024-03-06T12:08:42.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:42.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:42.578603+0000) 2024-03-06T12:08:43.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:43.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:43.578827+0000) 2024-03-06T12:08:44.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:44.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:44.578992+0000) 2024-03-06T12:08:45.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:45.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:45.579244+0000) 2024-03-06T12:08:46.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:46.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:46.579529+0000) 2024-03-06T12:08:47.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:47.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:47.579748+0000) 2024-03-06T12:08:48.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:48.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:48.579968+0000) 2024-03-06T12:08:49.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:49.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:49.580162+0000) 2024-03-06T12:08:50.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:50.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:50.580384+0000) 2024-03-06T12:08:51.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:51.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:51.580615+0000) 2024-03-06T12:08:52.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:52.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:52.580832+0000) 2024-03-06T12:08:53.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:53.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:53.581085+0000) 2024-03-06T12:08:54.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:54.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:54.581284+0000) 2024-03-06T12:08:55.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:55.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:55.581514+0000) 2024-03-06T12:08:56.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:56.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:56.581750+0000) 2024-03-06T12:08:57.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:57.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:57.581973+0000) 2024-03-06T12:08:58.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:58.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:58.582217+0000) 2024-03-06T12:08:59.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:08:59.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:08:59.582442+0000) 2024-03-06T12:09:00.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:00.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:00.582628+0000) 2024-03-06T12:09:01.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:01.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:01.582857+0000) 2024-03-06T12:09:01.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:01.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:02.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:02.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:02.583125+0000) 2024-03-06T12:09:03.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:03.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:03.583342+0000) 2024-03-06T12:09:04.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:04.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:04.583553+0000) 2024-03-06T12:09:05.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:05.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:05.583780+0000) 2024-03-06T12:09:06.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:06.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:06.583965+0000) 2024-03-06T12:09:07.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:07.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:07.584200+0000) 2024-03-06T12:09:08.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:08.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:08.584408+0000) 2024-03-06T12:09:09.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:09.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:09.584636+0000) 2024-03-06T12:09:10.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:10.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:10.584867+0000) 2024-03-06T12:09:11.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:11.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:11.585107+0000) 2024-03-06T12:09:12.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:12.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:12.585338+0000) 2024-03-06T12:09:13.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:13.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:13.585524+0000) 2024-03-06T12:09:14.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:14.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:14.585721+0000) 2024-03-06T12:09:15.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:15.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:15.585948+0000) 2024-03-06T12:09:16.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:16.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:16.586167+0000) 2024-03-06T12:09:17.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:17.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:17.586400+0000) 2024-03-06T12:09:18.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:18.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:18.586647+0000) 2024-03-06T12:09:19.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:19.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:19.586885+0000) 2024-03-06T12:09:20.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:20.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:20.587115+0000) 2024-03-06T12:09:21.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:21.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:21.587326+0000) 2024-03-06T12:09:22.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:22.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:22.587574+0000) 2024-03-06T12:09:23.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:23.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:23.587799+0000) 2024-03-06T12:09:24.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:24.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:24.588063+0000) 2024-03-06T12:09:25.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:25.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:25.588243+0000) 2024-03-06T12:09:26.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:26.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:26.588444+0000) 2024-03-06T12:09:27.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:27.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:27.588668+0000) 2024-03-06T12:09:28.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:28.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:28.588858+0000) 2024-03-06T12:09:29.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:29.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:29.589139+0000) 2024-03-06T12:09:30.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:30.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:30.589354+0000) 2024-03-06T12:09:31.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:31.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:31.589564+0000) 2024-03-06T12:09:32.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:32.113 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:09:32.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:32.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:32.589799+0000) 2024-03-06T12:09:33.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:33.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:33.590064+0000) 2024-03-06T12:09:34.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:34.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:34.590263+0000) 2024-03-06T12:09:35.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:35.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:35.590484+0000) 2024-03-06T12:09:36.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:36.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:36.590720+0000) 2024-03-06T12:09:37.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:37.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:37.590951+0000) 2024-03-06T12:09:38.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:38.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:38.591168+0000) 2024-03-06T12:09:39.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:39.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:39.591421+0000) 2024-03-06T12:09:40.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:40.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:40.591648+0000) 2024-03-06T12:09:41.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:41.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:41.591878+0000) 2024-03-06T12:09:42.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:42.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:42.592127+0000) 2024-03-06T12:09:43.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:43.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:43.592348+0000) 2024-03-06T12:09:44.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:44.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:44.592566+0000) 2024-03-06T12:09:45.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:45.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:45.592803+0000) 2024-03-06T12:09:46.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:46.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:46.592999+0000) 2024-03-06T12:09:47.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:47.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:47.593269+0000) 2024-03-06T12:09:48.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:48.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:48.593495+0000) 2024-03-06T12:09:49.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:49.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:49.593668+0000) 2024-03-06T12:09:50.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:50.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:50.593898+0000) 2024-03-06T12:09:51.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:51.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:51.594151+0000) 2024-03-06T12:09:52.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:52.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:52.594360+0000) 2024-03-06T12:09:53.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:53.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:53.594603+0000) 2024-03-06T12:09:54.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:54.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:54.594853+0000) 2024-03-06T12:09:55.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:55.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:55.595124+0000) 2024-03-06T12:09:56.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:56.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:56.595323+0000) 2024-03-06T12:09:57.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:57.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:57.595540+0000) 2024-03-06T12:09:58.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:58.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:58.595761+0000) 2024-03-06T12:09:59.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:09:59.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:09:59.595990+0000) 2024-03-06T12:10:00.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:00.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:00.596248+0000) 2024-03-06T12:10:01.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:01.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:01.596495+0000) 2024-03-06T12:10:02.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:02.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:02.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:02.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:02.596772+0000) 2024-03-06T12:10:03.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:03.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:03.597074+0000) 2024-03-06T12:10:04.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:04.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:04.597276+0000) 2024-03-06T12:10:05.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:05.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:05.597507+0000) 2024-03-06T12:10:06.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:06.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:06.597767+0000) 2024-03-06T12:10:07.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:07.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:07.597970+0000) 2024-03-06T12:10:08.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:08.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:08.598203+0000) 2024-03-06T12:10:09.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:09.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:09.598447+0000) 2024-03-06T12:10:10.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:10.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:10.598667+0000) 2024-03-06T12:10:11.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:11.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:11.598906+0000) 2024-03-06T12:10:12.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:12.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:12.599154+0000) 2024-03-06T12:10:13.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:13.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:13.599331+0000) 2024-03-06T12:10:14.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:14.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:14.599575+0000) 2024-03-06T12:10:15.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:15.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:15.599758+0000) 2024-03-06T12:10:16.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:16.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:16.599990+0000) 2024-03-06T12:10:17.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:17.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:17.600246+0000) 2024-03-06T12:10:18.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:18.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:18.600477+0000) 2024-03-06T12:10:19.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:19.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:19.600674+0000) 2024-03-06T12:10:20.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:20.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:20.600860+0000) 2024-03-06T12:10:21.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:21.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:21.601161+0000) 2024-03-06T12:10:22.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:22.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:22.601462+0000) 2024-03-06T12:10:23.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:23.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:23.601698+0000) 2024-03-06T12:10:24.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:24.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:24.601961+0000) 2024-03-06T12:10:25.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:25.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:25.602214+0000) 2024-03-06T12:10:26.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:26.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:26.602419+0000) 2024-03-06T12:10:27.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:27.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:27.602642+0000) 2024-03-06T12:10:28.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:28.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:28.602888+0000) 2024-03-06T12:10:29.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:29.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:29.603154+0000) 2024-03-06T12:10:30.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:30.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:30.603333+0000) 2024-03-06T12:10:31.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:31.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:31.603551+0000) 2024-03-06T12:10:32.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:32.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:10:32.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:32.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:32.603767+0000) 2024-03-06T12:10:33.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:33.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:33.603989+0000) 2024-03-06T12:10:34.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:34.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:34.604231+0000) 2024-03-06T12:10:35.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:35.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:35.604414+0000) 2024-03-06T12:10:36.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:36.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:36.604646+0000) 2024-03-06T12:10:37.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:37.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:37.604898+0000) 2024-03-06T12:10:38.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:38.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:38.605146+0000) 2024-03-06T12:10:39.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:39.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:39.605354+0000) 2024-03-06T12:10:40.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:40.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:40.605541+0000) 2024-03-06T12:10:41.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:41.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:41.605787+0000) 2024-03-06T12:10:42.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:42.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:42.606114+0000) 2024-03-06T12:10:43.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:43.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:43.606313+0000) 2024-03-06T12:10:44.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:44.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:44.606558+0000) 2024-03-06T12:10:45.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:45.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:45.606795+0000) 2024-03-06T12:10:46.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:46.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:46.607048+0000) 2024-03-06T12:10:47.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:47.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:47.607268+0000) 2024-03-06T12:10:48.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:48.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:48.607512+0000) 2024-03-06T12:10:49.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:49.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:49.607755+0000) 2024-03-06T12:10:50.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:50.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:50.607999+0000) 2024-03-06T12:10:51.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:51.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:51.608249+0000) 2024-03-06T12:10:52.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:52.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:52.608495+0000) 2024-03-06T12:10:53.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:53.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:53.608700+0000) 2024-03-06T12:10:54.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:54.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:54.608932+0000) 2024-03-06T12:10:55.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:55.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:55.609155+0000) 2024-03-06T12:10:56.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:56.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:56.609358+0000) 2024-03-06T12:10:57.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:57.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:57.609578+0000) 2024-03-06T12:10:58.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:58.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:58.609809+0000) 2024-03-06T12:10:59.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:10:59.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:10:59.610078+0000) 2024-03-06T12:11:00.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:00.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:00.610305+0000) 2024-03-06T12:11:01.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:01.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:01.610527+0000) 2024-03-06T12:11:02.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:02.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:02.610726+0000) 2024-03-06T12:11:02.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:02.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:03.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:03.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:03.610977+0000) 2024-03-06T12:11:04.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:04.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:04.611288+0000) 2024-03-06T12:11:05.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:05.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:05.611533+0000) 2024-03-06T12:11:06.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:06.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:06.611733+0000) 2024-03-06T12:11:07.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:07.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:07.611969+0000) 2024-03-06T12:11:08.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:08.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:08.612178+0000) 2024-03-06T12:11:09.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:09.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:09.612369+0000) 2024-03-06T12:11:10.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:10.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:10.612592+0000) 2024-03-06T12:11:11.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:11.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:11.612828+0000) 2024-03-06T12:11:12.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:12.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:12.613111+0000) 2024-03-06T12:11:13.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:13.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:13.613308+0000) 2024-03-06T12:11:14.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:14.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:14.613520+0000) 2024-03-06T12:11:15.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:15.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:15.613809+0000) 2024-03-06T12:11:16.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:16.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:16.614088+0000) 2024-03-06T12:11:17.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:17.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:17.614293+0000) 2024-03-06T12:11:18.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:18.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:18.614514+0000) 2024-03-06T12:11:19.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:19.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:19.614708+0000) 2024-03-06T12:11:20.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:20.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:20.614949+0000) 2024-03-06T12:11:21.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:21.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:21.615192+0000) 2024-03-06T12:11:22.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:22.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:22.615410+0000) 2024-03-06T12:11:23.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:23.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:23.615617+0000) 2024-03-06T12:11:24.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:24.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:24.615827+0000) 2024-03-06T12:11:25.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:25.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:25.616097+0000) 2024-03-06T12:11:26.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:26.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:26.616317+0000) 2024-03-06T12:11:27.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:27.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:27.616578+0000) 2024-03-06T12:11:28.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:28.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:28.616845+0000) 2024-03-06T12:11:29.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:29.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:29.617098+0000) 2024-03-06T12:11:30.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:30.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:30.617319+0000) 2024-03-06T12:11:31.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:31.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:31.617585+0000) 2024-03-06T12:11:32.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:32.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:32.617845+0000) 2024-03-06T12:11:32.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:32.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:11:33.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:33.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:33.618132+0000) 2024-03-06T12:11:34.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:34.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:34.618346+0000) 2024-03-06T12:11:35.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:35.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:35.618546+0000) 2024-03-06T12:11:36.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:36.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:36.618800+0000) 2024-03-06T12:11:37.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:37.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:37.619058+0000) 2024-03-06T12:11:38.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:38.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:38.619258+0000) 2024-03-06T12:11:39.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:39.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:39.619503+0000) 2024-03-06T12:11:40.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:40.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:40.619745+0000) 2024-03-06T12:11:41.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:41.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:41.619987+0000) 2024-03-06T12:11:42.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:42.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:42.620329+0000) 2024-03-06T12:11:43.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:43.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:43.620568+0000) 2024-03-06T12:11:44.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:44.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:44.620789+0000) 2024-03-06T12:11:45.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:45.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:45.621044+0000) 2024-03-06T12:11:46.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:46.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:46.621286+0000) 2024-03-06T12:11:47.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:47.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:47.621517+0000) 2024-03-06T12:11:48.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:48.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:48.621778+0000) 2024-03-06T12:11:49.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:49.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:49.622059+0000) 2024-03-06T12:11:50.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:50.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:50.622258+0000) 2024-03-06T12:11:51.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:51.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:51.622534+0000) 2024-03-06T12:11:52.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:52.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:52.622824+0000) 2024-03-06T12:11:53.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:53.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:53.623101+0000) 2024-03-06T12:11:54.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:54.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:54.623339+0000) 2024-03-06T12:11:55.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:55.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:55.623537+0000) 2024-03-06T12:11:56.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:56.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:56.623797+0000) 2024-03-06T12:11:57.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:57.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:57.624117+0000) 2024-03-06T12:11:58.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:58.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:58.624364+0000) 2024-03-06T12:11:59.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:11:59.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:11:59.624641+0000) 2024-03-06T12:12:00.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:00.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:00.624912+0000) 2024-03-06T12:12:01.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:01.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:01.625194+0000) 2024-03-06T12:12:02.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:02.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:02.625463+0000) 2024-03-06T12:12:03.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:03.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:03.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:03.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:03.625772+0000) 2024-03-06T12:12:04.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:04.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:04.626110+0000) 2024-03-06T12:12:05.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:05.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:05.626384+0000) 2024-03-06T12:12:06.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:06.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:06.626666+0000) 2024-03-06T12:12:07.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:07.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:07.626872+0000) 2024-03-06T12:12:08.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:08.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:08.627178+0000) 2024-03-06T12:12:09.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:09.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:09.627373+0000) 2024-03-06T12:12:10.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:10.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:10.627597+0000) 2024-03-06T12:12:11.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:11.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:11.627785+0000) 2024-03-06T12:12:12.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:12.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:12.628062+0000) 2024-03-06T12:12:13.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:13.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:13.628309+0000) 2024-03-06T12:12:14.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:14.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:14.628504+0000) 2024-03-06T12:12:15.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:15.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:15.628723+0000) 2024-03-06T12:12:16.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:16.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:16.628943+0000) 2024-03-06T12:12:17.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:17.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:17.629165+0000) 2024-03-06T12:12:18.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:18.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:18.629394+0000) 2024-03-06T12:12:19.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:19.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:19.629655+0000) 2024-03-06T12:12:20.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:20.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:20.629897+0000) 2024-03-06T12:12:21.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:21.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:21.630163+0000) 2024-03-06T12:12:22.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:22.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:22.630396+0000) 2024-03-06T12:12:23.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:23.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:23.630622+0000) 2024-03-06T12:12:24.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:24.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:24.630855+0000) 2024-03-06T12:12:25.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:25.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:25.631099+0000) 2024-03-06T12:12:26.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:26.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:26.631333+0000) 2024-03-06T12:12:27.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:27.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:27.631604+0000) 2024-03-06T12:12:28.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:28.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:28.631888+0000) 2024-03-06T12:12:29.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:29.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:29.632159+0000) 2024-03-06T12:12:30.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:30.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:30.632423+0000) 2024-03-06T12:12:31.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:31.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:31.632664+0000) 2024-03-06T12:12:32.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:32.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:32.632860+0000) 2024-03-06T12:12:33.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:33.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:12:33.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:33.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:33.633130+0000) 2024-03-06T12:12:34.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:34.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:34.633383+0000) 2024-03-06T12:12:35.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:35.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:35.633624+0000) 2024-03-06T12:12:36.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:36.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:36.633851+0000) 2024-03-06T12:12:37.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:37.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:37.634094+0000) 2024-03-06T12:12:38.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:38.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:38.634357+0000) 2024-03-06T12:12:39.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:39.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:39.634596+0000) 2024-03-06T12:12:40.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:40.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:40.634798+0000) 2024-03-06T12:12:41.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:41.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:41.635058+0000) 2024-03-06T12:12:42.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:42.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:42.635276+0000) 2024-03-06T12:12:43.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:43.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:43.635459+0000) 2024-03-06T12:12:44.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:44.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:44.635688+0000) 2024-03-06T12:12:45.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:45.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:45.635913+0000) 2024-03-06T12:12:46.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:46.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:46.636180+0000) 2024-03-06T12:12:47.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:47.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:47.636374+0000) 2024-03-06T12:12:48.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:48.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:48.636599+0000) 2024-03-06T12:12:49.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:49.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:49.636823+0000) 2024-03-06T12:12:50.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:50.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:50.637034+0000) 2024-03-06T12:12:51.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:51.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:51.637285+0000) 2024-03-06T12:12:52.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:52.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:52.637575+0000) 2024-03-06T12:12:53.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:53.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:53.637853+0000) 2024-03-06T12:12:54.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:54.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:54.638109+0000) 2024-03-06T12:12:55.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:55.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:55.638292+0000) 2024-03-06T12:12:56.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:56.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:56.638644+0000) 2024-03-06T12:12:57.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:57.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:57.638890+0000) 2024-03-06T12:12:58.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:58.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:58.639119+0000) 2024-03-06T12:12:59.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:12:59.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:12:59.639325+0000) 2024-03-06T12:13:00.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:00.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:00.639585+0000) 2024-03-06T12:13:01.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:01.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:01.639800+0000) 2024-03-06T12:13:02.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:02.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:02.639992+0000) 2024-03-06T12:13:03.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:03.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:03.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:03.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:03.640233+0000) 2024-03-06T12:13:04.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:04.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:04.640465+0000) 2024-03-06T12:13:05.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:05.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:05.640700+0000) 2024-03-06T12:13:06.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:06.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:06.640923+0000) 2024-03-06T12:13:07.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:07.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:07.641168+0000) 2024-03-06T12:13:08.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:08.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:08.641339+0000) 2024-03-06T12:13:09.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:09.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:09.641564+0000) 2024-03-06T12:13:10.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:10.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:10.641785+0000) 2024-03-06T12:13:11.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:11.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:11.641969+0000) 2024-03-06T12:13:12.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:12.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:12.642177+0000) 2024-03-06T12:13:13.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:13.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:13.642373+0000) 2024-03-06T12:13:14.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:14.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:14.642591+0000) 2024-03-06T12:13:15.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:15.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:15.642863+0000) 2024-03-06T12:13:16.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:16.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:16.643139+0000) 2024-03-06T12:13:17.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:17.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:17.643382+0000) 2024-03-06T12:13:18.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:18.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:18.643607+0000) 2024-03-06T12:13:19.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:19.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:19.643820+0000) 2024-03-06T12:13:20.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:20.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:20.643968+0000) 2024-03-06T12:13:21.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:21.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:21.644199+0000) 2024-03-06T12:13:22.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:22.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:22.644339+0000) 2024-03-06T12:13:23.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:23.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:23.644560+0000) 2024-03-06T12:13:24.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:24.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:24.644801+0000) 2024-03-06T12:13:25.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:25.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:25.645056+0000) 2024-03-06T12:13:26.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:26.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:26.645292+0000) 2024-03-06T12:13:27.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:27.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:27.645522+0000) 2024-03-06T12:13:28.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:28.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:28.645732+0000) 2024-03-06T12:13:29.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:29.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:29.646000+0000) 2024-03-06T12:13:30.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:30.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:30.646297+0000) 2024-03-06T12:13:31.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:31.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:31.646507+0000) 2024-03-06T12:13:32.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:32.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:32.646718+0000) 2024-03-06T12:13:33.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:33.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:33.646943+0000) 2024-03-06T12:13:33.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:33.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:13:34.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:34.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:34.647187+0000) 2024-03-06T12:13:35.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:35.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:35.647418+0000) 2024-03-06T12:13:36.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:36.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:36.647642+0000) 2024-03-06T12:13:37.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:37.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:37.647868+0000) 2024-03-06T12:13:38.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:38.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:38.648142+0000) 2024-03-06T12:13:39.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:39.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:39.648379+0000) 2024-03-06T12:13:40.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:40.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:40.648651+0000) 2024-03-06T12:13:41.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:41.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:41.648931+0000) 2024-03-06T12:13:42.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:42.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:42.649216+0000) 2024-03-06T12:13:43.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:43.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:43.649464+0000) 2024-03-06T12:13:44.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:44.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:44.649744+0000) 2024-03-06T12:13:45.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:45.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:45.649969+0000) 2024-03-06T12:13:46.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:46.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:46.650166+0000) 2024-03-06T12:13:47.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:47.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:47.650427+0000) 2024-03-06T12:13:48.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:48.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:48.650704+0000) 2024-03-06T12:13:49.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:49.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:49.650908+0000) 2024-03-06T12:13:50.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:50.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:50.651146+0000) 2024-03-06T12:13:51.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:51.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:51.651359+0000) 2024-03-06T12:13:52.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:52.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:52.651594+0000) 2024-03-06T12:13:53.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:53.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:53.651815+0000) 2024-03-06T12:13:54.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:54.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:54.652090+0000) 2024-03-06T12:13:55.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:55.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:55.652294+0000) 2024-03-06T12:13:56.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:56.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:56.652474+0000) 2024-03-06T12:13:57.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:57.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:57.652706+0000) 2024-03-06T12:13:58.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:58.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:58.652942+0000) 2024-03-06T12:13:59.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:13:59.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:13:59.653205+0000) 2024-03-06T12:14:00.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:00.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:00.653453+0000) 2024-03-06T12:14:01.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:01.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:01.653732+0000) 2024-03-06T12:14:02.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:02.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:02.654069+0000) 2024-03-06T12:14:03.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:03.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:03.654350+0000) 2024-03-06T12:14:03.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:03.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:04.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:04.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:04.654627+0000) 2024-03-06T12:14:05.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:05.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:05.654861+0000) 2024-03-06T12:14:06.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:06.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:06.655190+0000) 2024-03-06T12:14:07.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:07.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:07.655450+0000) 2024-03-06T12:14:08.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:08.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:08.655736+0000) 2024-03-06T12:14:09.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:09.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:09.656078+0000) 2024-03-06T12:14:10.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:10.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:10.656285+0000) 2024-03-06T12:14:11.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:11.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:11.656513+0000) 2024-03-06T12:14:12.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:12.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:12.656758+0000) 2024-03-06T12:14:13.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:13.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:13.656980+0000) 2024-03-06T12:14:14.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:14.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:14.657258+0000) 2024-03-06T12:14:15.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:15.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:15.657485+0000) 2024-03-06T12:14:16.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:16.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:16.657720+0000) 2024-03-06T12:14:17.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:17.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:17.657954+0000) 2024-03-06T12:14:18.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:18.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:18.658189+0000) 2024-03-06T12:14:19.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:19.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:19.658421+0000) 2024-03-06T12:14:20.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:20.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:20.658668+0000) 2024-03-06T12:14:21.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:21.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:21.658896+0000) 2024-03-06T12:14:22.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:22.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:22.659155+0000) 2024-03-06T12:14:23.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:23.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:23.659319+0000) 2024-03-06T12:14:24.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:24.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:24.659549+0000) 2024-03-06T12:14:25.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:25.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:25.659812+0000) 2024-03-06T12:14:26.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:26.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:26.660084+0000) 2024-03-06T12:14:27.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:27.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:27.660335+0000) 2024-03-06T12:14:28.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:28.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:28.660581+0000) 2024-03-06T12:14:29.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:29.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:29.660822+0000) 2024-03-06T12:14:30.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:30.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:30.661113+0000) 2024-03-06T12:14:31.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:31.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:31.661355+0000) 2024-03-06T12:14:32.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:32.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:32.661619+0000) 2024-03-06T12:14:33.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:33.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:33.661865+0000) 2024-03-06T12:14:34.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:34.120 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:14:34.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:34.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:34.662112+0000) 2024-03-06T12:14:35.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:35.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:35.662340+0000) 2024-03-06T12:14:36.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:36.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:36.662573+0000) 2024-03-06T12:14:37.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:37.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:37.662820+0000) 2024-03-06T12:14:38.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:38.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:38.663082+0000) 2024-03-06T12:14:39.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:39.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:39.663310+0000) 2024-03-06T12:14:40.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:40.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:40.663510+0000) 2024-03-06T12:14:41.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:41.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:41.663687+0000) 2024-03-06T12:14:42.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:42.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:42.663929+0000) 2024-03-06T12:14:43.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:43.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:43.664179+0000) 2024-03-06T12:14:44.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:44.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:44.664367+0000) 2024-03-06T12:14:45.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:45.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:45.664593+0000) 2024-03-06T12:14:46.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:46.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:46.664814+0000) 2024-03-06T12:14:47.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:47.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:47.665083+0000) 2024-03-06T12:14:48.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:48.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:48.665281+0000) 2024-03-06T12:14:49.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:49.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:49.665498+0000) 2024-03-06T12:14:50.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:50.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:50.665719+0000) 2024-03-06T12:14:51.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:51.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:51.665945+0000) 2024-03-06T12:14:52.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:52.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:52.666196+0000) 2024-03-06T12:14:53.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:53.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:53.666427+0000) 2024-03-06T12:14:54.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:54.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:54.666645+0000) 2024-03-06T12:14:55.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:55.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:55.666874+0000) 2024-03-06T12:14:56.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:56.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:56.667142+0000) 2024-03-06T12:14:57.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:57.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:57.667405+0000) 2024-03-06T12:14:58.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:58.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:58.667652+0000) 2024-03-06T12:14:59.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:14:59.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:14:59.667924+0000) 2024-03-06T12:15:00.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:00.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:00.668145+0000) 2024-03-06T12:15:01.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:01.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:01.668365+0000) 2024-03-06T12:15:02.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:02.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:02.668661+0000) 2024-03-06T12:15:03.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:03.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:03.668908+0000) 2024-03-06T12:15:04.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:04.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:04.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:04.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:04.669089+0000) 2024-03-06T12:15:05.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:05.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:05.669210+0000) 2024-03-06T12:15:06.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:06.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:06.669340+0000) 2024-03-06T12:15:07.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:07.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:07.669565+0000) 2024-03-06T12:15:08.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:08.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:08.669786+0000) 2024-03-06T12:15:09.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:09.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:09.670073+0000) 2024-03-06T12:15:10.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:10.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:10.670326+0000) 2024-03-06T12:15:11.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:11.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:11.670609+0000) 2024-03-06T12:15:12.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:12.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:12.670874+0000) 2024-03-06T12:15:13.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:13.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:13.671054+0000) 2024-03-06T12:15:14.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:14.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:14.671272+0000) 2024-03-06T12:15:15.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:15.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:15.671498+0000) 2024-03-06T12:15:16.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:16.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:16.671731+0000) 2024-03-06T12:15:17.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:17.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:17.671917+0000) 2024-03-06T12:15:18.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:18.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:18.672140+0000) 2024-03-06T12:15:19.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:19.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:19.672362+0000) 2024-03-06T12:15:20.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:20.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:20.672587+0000) 2024-03-06T12:15:21.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:21.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:21.672811+0000) 2024-03-06T12:15:22.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:22.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:22.673093+0000) 2024-03-06T12:15:23.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:23.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:23.673318+0000) 2024-03-06T12:15:24.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:24.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:24.673563+0000) 2024-03-06T12:15:25.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:25.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:25.673819+0000) 2024-03-06T12:15:26.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:26.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:26.674131+0000) 2024-03-06T12:15:27.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:27.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:27.674344+0000) 2024-03-06T12:15:28.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:28.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:28.674573+0000) 2024-03-06T12:15:29.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:29.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:29.674799+0000) 2024-03-06T12:15:30.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:30.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:30.674990+0000) 2024-03-06T12:15:31.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:31.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:31.675218+0000) 2024-03-06T12:15:32.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:32.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:32.675453+0000) 2024-03-06T12:15:33.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:33.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:33.675681+0000) 2024-03-06T12:15:34.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:34.520 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:15:34.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:34.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:34.675879+0000) 2024-03-06T12:15:35.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:35.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:35.676151+0000) 2024-03-06T12:15:36.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:36.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:36.676334+0000) 2024-03-06T12:15:37.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:37.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:37.676565+0000) 2024-03-06T12:15:38.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:38.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:38.676775+0000) 2024-03-06T12:15:39.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:39.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:39.677046+0000) 2024-03-06T12:15:40.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:40.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:40.677298+0000) 2024-03-06T12:15:41.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:41.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:41.677523+0000) 2024-03-06T12:15:42.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:42.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:42.677770+0000) 2024-03-06T12:15:43.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:43.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:43.677998+0000) 2024-03-06T12:15:44.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:44.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:44.678235+0000) 2024-03-06T12:15:45.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:45.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:45.678460+0000) 2024-03-06T12:15:46.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:46.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:46.678726+0000) 2024-03-06T12:15:47.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:47.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:47.679052+0000) 2024-03-06T12:15:48.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:48.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:48.679332+0000) 2024-03-06T12:15:49.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:49.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:49.679554+0000) 2024-03-06T12:15:50.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:50.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:50.679837+0000) 2024-03-06T12:15:51.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:51.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:51.680082+0000) 2024-03-06T12:15:52.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:52.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:52.680355+0000) 2024-03-06T12:15:53.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:53.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:53.680581+0000) 2024-03-06T12:15:54.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:54.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:54.680785+0000) 2024-03-06T12:15:55.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:55.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:55.681045+0000) 2024-03-06T12:15:56.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:56.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:56.681258+0000) 2024-03-06T12:15:57.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:57.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:57.681492+0000) 2024-03-06T12:15:58.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:58.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:58.681719+0000) 2024-03-06T12:15:59.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:15:59.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:15:59.681948+0000) 2024-03-06T12:16:00.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:00.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:00.682175+0000) 2024-03-06T12:16:01.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:01.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:01.682357+0000) 2024-03-06T12:16:02.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:02.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:02.682596+0000) 2024-03-06T12:16:03.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:03.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:03.682831+0000) 2024-03-06T12:16:04.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:04.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:04.683099+0000) 2024-03-06T12:16:04.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:04.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:05.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:05.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:05.683328+0000) 2024-03-06T12:16:06.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:06.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:06.683555+0000) 2024-03-06T12:16:07.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:07.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:07.683786+0000) 2024-03-06T12:16:08.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:08.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:08.684049+0000) 2024-03-06T12:16:09.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:09.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:09.684283+0000) 2024-03-06T12:16:10.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:10.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:10.684512+0000) 2024-03-06T12:16:11.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:11.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:11.684738+0000) 2024-03-06T12:16:12.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:12.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:12.684986+0000) 2024-03-06T12:16:13.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:13.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:13.685250+0000) 2024-03-06T12:16:14.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:14.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:14.685543+0000) 2024-03-06T12:16:15.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:15.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:15.685764+0000) 2024-03-06T12:16:16.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:16.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:16.685983+0000) 2024-03-06T12:16:17.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:17.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:17.686259+0000) 2024-03-06T12:16:18.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:18.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:18.686483+0000) 2024-03-06T12:16:19.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:19.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:19.686713+0000) 2024-03-06T12:16:20.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:20.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:20.686942+0000) 2024-03-06T12:16:21.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:21.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:21.687167+0000) 2024-03-06T12:16:22.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:22.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:22.687408+0000) 2024-03-06T12:16:23.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:23.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:23.687638+0000) 2024-03-06T12:16:24.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:24.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:24.687866+0000) 2024-03-06T12:16:25.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:25.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:25.688129+0000) 2024-03-06T12:16:26.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:26.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:26.688346+0000) 2024-03-06T12:16:27.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:27.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:27.688618+0000) 2024-03-06T12:16:28.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:28.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:28.688885+0000) 2024-03-06T12:16:29.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:29.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:29.689169+0000) 2024-03-06T12:16:30.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:30.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:30.689447+0000) 2024-03-06T12:16:31.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:31.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:31.689659+0000) 2024-03-06T12:16:32.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:32.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:32.689913+0000) 2024-03-06T12:16:33.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:33.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:33.690192+0000) 2024-03-06T12:16:34.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:34.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:34.690430+0000) 2024-03-06T12:16:34.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:34.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:16:35.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:35.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:35.690662+0000) 2024-03-06T12:16:36.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:36.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:36.690896+0000) 2024-03-06T12:16:37.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:37.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:37.691160+0000) 2024-03-06T12:16:38.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:38.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:38.691396+0000) 2024-03-06T12:16:39.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:39.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:39.691641+0000) 2024-03-06T12:16:40.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:40.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:40.691863+0000) 2024-03-06T12:16:41.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:41.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:41.692128+0000) 2024-03-06T12:16:42.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:42.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:42.692332+0000) 2024-03-06T12:16:43.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:43.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:43.692562+0000) 2024-03-06T12:16:44.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:44.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:44.692745+0000) 2024-03-06T12:16:45.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:45.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:45.692924+0000) 2024-03-06T12:16:46.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:46.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:46.693150+0000) 2024-03-06T12:16:47.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:47.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:47.693354+0000) 2024-03-06T12:16:48.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:48.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:48.693586+0000) 2024-03-06T12:16:49.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:49.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:49.693816+0000) 2024-03-06T12:16:50.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:50.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:50.694082+0000) 2024-03-06T12:16:51.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:51.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:51.694332+0000) 2024-03-06T12:16:52.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:52.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:52.694583+0000) 2024-03-06T12:16:53.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:53.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:53.694832+0000) 2024-03-06T12:16:54.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:54.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:54.695077+0000) 2024-03-06T12:16:55.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:55.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:55.695277+0000) 2024-03-06T12:16:56.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:56.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:56.695505+0000) 2024-03-06T12:16:57.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:57.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:57.695730+0000) 2024-03-06T12:16:58.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:58.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:58.695921+0000) 2024-03-06T12:16:59.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:16:59.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:16:59.696154+0000) 2024-03-06T12:17:00.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:00.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:00.696348+0000) 2024-03-06T12:17:01.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:01.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:01.696541+0000) 2024-03-06T12:17:02.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:02.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:02.696759+0000) 2024-03-06T12:17:03.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:03.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:03.696966+0000) 2024-03-06T12:17:04.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:04.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:04.697177+0000) 2024-03-06T12:17:05.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:05.107 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:05.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:05.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:05.697427+0000) 2024-03-06T12:17:06.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:06.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:06.697639+0000) 2024-03-06T12:17:07.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:07.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:07.697938+0000) 2024-03-06T12:17:08.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:08.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:08.698171+0000) 2024-03-06T12:17:09.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:09.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:09.698392+0000) 2024-03-06T12:17:10.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:10.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:10.698614+0000) 2024-03-06T12:17:11.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:11.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:11.698836+0000) 2024-03-06T12:17:12.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:12.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:12.699115+0000) 2024-03-06T12:17:13.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:13.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:13.699344+0000) 2024-03-06T12:17:14.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:14.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:14.699574+0000) 2024-03-06T12:17:15.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:15.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:15.699803+0000) 2024-03-06T12:17:16.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:16.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:16.700070+0000) 2024-03-06T12:17:17.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:17.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:17.700283+0000) 2024-03-06T12:17:18.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:18.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:18.700518+0000) 2024-03-06T12:17:19.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:19.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:19.700738+0000) 2024-03-06T12:17:20.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:20.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:20.700966+0000) 2024-03-06T12:17:21.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:21.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:21.701191+0000) 2024-03-06T12:17:22.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:22.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:22.701423+0000) 2024-03-06T12:17:23.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:23.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:23.701642+0000) 2024-03-06T12:17:24.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:24.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:24.701829+0000) 2024-03-06T12:17:25.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:25.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:25.702087+0000) 2024-03-06T12:17:26.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:26.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:26.702310+0000) 2024-03-06T12:17:27.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:27.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:27.702534+0000) 2024-03-06T12:17:28.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:28.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:28.702780+0000) 2024-03-06T12:17:29.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:29.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:29.703050+0000) 2024-03-06T12:17:30.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:30.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:30.703253+0000) 2024-03-06T12:17:31.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:31.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:31.703453+0000) 2024-03-06T12:17:32.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:32.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:32.703698+0000) 2024-03-06T12:17:33.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:33.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:33.703929+0000) 2024-03-06T12:17:34.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:34.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:34.704164+0000) 2024-03-06T12:17:35.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:35.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:17:35.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:35.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:35.704394+0000) 2024-03-06T12:17:36.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:36.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:36.704631+0000) 2024-03-06T12:17:37.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:37.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:37.704853+0000) 2024-03-06T12:17:38.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:38.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:38.705110+0000) 2024-03-06T12:17:39.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:39.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:39.705354+0000) 2024-03-06T12:17:40.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:40.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:40.705572+0000) 2024-03-06T12:17:41.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:41.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:41.705751+0000) 2024-03-06T12:17:42.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:42.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:42.705987+0000) 2024-03-06T12:17:43.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:43.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:43.706241+0000) 2024-03-06T12:17:44.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:44.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:44.706460+0000) 2024-03-06T12:17:45.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:45.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:45.706736+0000) 2024-03-06T12:17:46.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:46.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:46.707047+0000) 2024-03-06T12:17:47.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:47.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:47.707318+0000) 2024-03-06T12:17:48.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:48.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:48.707614+0000) 2024-03-06T12:17:49.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:49.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:49.707855+0000) 2024-03-06T12:17:50.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:50.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:50.708115+0000) 2024-03-06T12:17:51.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:51.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:51.708351+0000) 2024-03-06T12:17:52.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:52.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:52.708600+0000) 2024-03-06T12:17:53.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:53.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:53.708839+0000) 2024-03-06T12:17:54.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:54.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:54.709109+0000) 2024-03-06T12:17:55.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:55.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:55.709403+0000) 2024-03-06T12:17:56.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:56.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:56.709651+0000) 2024-03-06T12:17:57.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:57.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:57.709902+0000) 2024-03-06T12:17:58.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:58.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:58.710149+0000) 2024-03-06T12:17:59.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:17:59.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:17:59.710346+0000) 2024-03-06T12:18:00.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:00.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:00.710589+0000) 2024-03-06T12:18:01.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:01.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:01.710826+0000) 2024-03-06T12:18:02.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:02.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:02.711134+0000) 2024-03-06T12:18:03.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:03.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:03.711376+0000) 2024-03-06T12:18:04.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:04.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:04.711602+0000) 2024-03-06T12:18:05.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:05.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:05.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:05.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:05.711834+0000) 2024-03-06T12:18:06.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:06.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:06.712100+0000) 2024-03-06T12:18:07.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:07.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:07.712327+0000) 2024-03-06T12:18:08.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:08.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:08.712552+0000) 2024-03-06T12:18:09.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:09.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:09.712769+0000) 2024-03-06T12:18:10.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:10.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:10.712979+0000) 2024-03-06T12:18:11.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:11.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:11.713224+0000) 2024-03-06T12:18:12.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:12.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:12.713459+0000) 2024-03-06T12:18:13.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:13.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:13.713681+0000) 2024-03-06T12:18:14.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:14.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:14.713899+0000) 2024-03-06T12:18:15.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:15.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:15.714075+0000) 2024-03-06T12:18:16.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:16.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:16.714295+0000) 2024-03-06T12:18:17.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:17.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:17.714479+0000) 2024-03-06T12:18:18.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:18.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:18.714724+0000) 2024-03-06T12:18:19.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:19.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:19.714958+0000) 2024-03-06T12:18:20.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:20.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:20.715184+0000) 2024-03-06T12:18:21.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:21.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:21.715410+0000) 2024-03-06T12:18:22.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:22.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:22.715685+0000) 2024-03-06T12:18:23.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:23.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:23.715913+0000) 2024-03-06T12:18:24.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:24.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:24.716158+0000) 2024-03-06T12:18:25.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:25.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:25.716362+0000) 2024-03-06T12:18:26.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:26.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:26.716542+0000) 2024-03-06T12:18:27.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:27.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:27.716776+0000) 2024-03-06T12:18:28.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:28.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:28.717041+0000) 2024-03-06T12:18:29.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:29.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:29.717288+0000) 2024-03-06T12:18:30.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:30.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:30.717519+0000) 2024-03-06T12:18:31.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:31.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:31.717744+0000) 2024-03-06T12:18:32.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:32.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:32.717996+0000) 2024-03-06T12:18:33.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:33.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:33.718248+0000) 2024-03-06T12:18:34.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:34.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:34.718469+0000) 2024-03-06T12:18:35.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:35.646 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:18:35.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:35.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:35.718717+0000) 2024-03-06T12:18:36.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:36.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:36.718897+0000) 2024-03-06T12:18:37.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:37.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:37.719126+0000) 2024-03-06T12:18:38.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:38.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:38.719365+0000) 2024-03-06T12:18:39.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:39.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:39.719592+0000) 2024-03-06T12:18:40.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:40.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:40.719817+0000) 2024-03-06T12:18:41.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:41.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:41.720064+0000) 2024-03-06T12:18:42.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:42.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:42.720298+0000) 2024-03-06T12:18:43.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:43.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:43.720533+0000) 2024-03-06T12:18:44.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:44.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:44.720712+0000) 2024-03-06T12:18:45.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:45.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:45.720946+0000) 2024-03-06T12:18:46.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:46.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:46.721218+0000) 2024-03-06T12:18:47.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:47.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:47.721469+0000) 2024-03-06T12:18:48.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:48.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:48.721699+0000) 2024-03-06T12:18:49.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:49.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:49.721912+0000) 2024-03-06T12:18:50.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:50.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:50.722137+0000) 2024-03-06T12:18:51.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:51.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:51.722369+0000) 2024-03-06T12:18:52.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:52.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:52.722614+0000) 2024-03-06T12:18:53.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:53.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:53.722864+0000) 2024-03-06T12:18:54.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:54.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:54.723102+0000) 2024-03-06T12:18:55.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:55.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:55.723328+0000) 2024-03-06T12:18:56.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:56.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:56.723541+0000) 2024-03-06T12:18:57.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:57.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:57.723771+0000) 2024-03-06T12:18:58.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:58.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:58.723995+0000) 2024-03-06T12:18:59.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:18:59.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:18:59.724259+0000) 2024-03-06T12:19:00.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:00.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:00.724440+0000) 2024-03-06T12:19:01.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:01.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:01.724684+0000) 2024-03-06T12:19:02.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:02.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:02.724947+0000) 2024-03-06T12:19:03.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:03.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:03.725193+0000) 2024-03-06T12:19:04.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:04.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:04.725399+0000) 2024-03-06T12:19:05.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:05.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:05.725638+0000) 2024-03-06T12:19:05.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:05.850 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:06.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:06.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:06.725927+0000) 2024-03-06T12:19:07.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:07.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:07.726136+0000) 2024-03-06T12:19:08.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:08.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:08.726356+0000) 2024-03-06T12:19:09.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:09.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:09.726584+0000) 2024-03-06T12:19:10.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:10.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:10.726811+0000) 2024-03-06T12:19:11.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:11.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:11.727051+0000) 2024-03-06T12:19:12.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:12.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:12.727262+0000) 2024-03-06T12:19:13.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:13.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:13.727489+0000) 2024-03-06T12:19:14.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:14.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:14.727731+0000) 2024-03-06T12:19:15.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:15.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:15.727976+0000) 2024-03-06T12:19:16.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:16.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:16.728219+0000) 2024-03-06T12:19:17.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:17.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:17.728401+0000) 2024-03-06T12:19:18.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:18.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:18.728627+0000) 2024-03-06T12:19:19.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:19.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:19.728851+0000) 2024-03-06T12:19:20.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:20.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:20.729136+0000) 2024-03-06T12:19:21.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:21.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:21.729375+0000) 2024-03-06T12:19:22.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:22.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:22.729625+0000) 2024-03-06T12:19:23.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:23.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:23.729862+0000) 2024-03-06T12:19:24.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:24.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:24.730124+0000) 2024-03-06T12:19:25.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:25.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:25.730365+0000) 2024-03-06T12:19:26.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:26.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:26.730610+0000) 2024-03-06T12:19:27.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:27.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:27.730810+0000) 2024-03-06T12:19:28.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:28.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:28.731121+0000) 2024-03-06T12:19:29.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:29.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:29.731362+0000) 2024-03-06T12:19:30.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:30.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:30.731584+0000) 2024-03-06T12:19:31.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:31.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:31.731824+0000) 2024-03-06T12:19:32.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:32.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:32.732096+0000) 2024-03-06T12:19:33.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:33.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:33.732330+0000) 2024-03-06T12:19:34.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:34.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:34.732554+0000) 2024-03-06T12:19:35.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:35.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:35.732751+0000) 2024-03-06T12:19:36.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:36.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:19:36.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:36.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:36.733089+0000) 2024-03-06T12:19:37.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:37.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:37.733324+0000) 2024-03-06T12:19:38.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:38.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:38.733553+0000) 2024-03-06T12:19:39.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:39.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:39.733787+0000) 2024-03-06T12:19:40.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:40.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:40.734060+0000) 2024-03-06T12:19:41.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:41.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:41.734286+0000) 2024-03-06T12:19:42.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:42.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:42.734534+0000) 2024-03-06T12:19:43.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:43.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:43.734771+0000) 2024-03-06T12:19:44.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:44.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:44.735039+0000) 2024-03-06T12:19:45.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:45.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:45.735271+0000) 2024-03-06T12:19:46.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:46.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:46.735497+0000) 2024-03-06T12:19:47.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:47.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:47.735700+0000) 2024-03-06T12:19:48.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:48.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:48.735884+0000) 2024-03-06T12:19:49.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:49.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:49.736126+0000) 2024-03-06T12:19:50.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:50.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:50.736369+0000) 2024-03-06T12:19:51.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:51.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:51.736610+0000) 2024-03-06T12:19:52.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:52.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:52.736861+0000) 2024-03-06T12:19:53.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:53.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:53.737145+0000) 2024-03-06T12:19:54.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:54.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:54.737375+0000) 2024-03-06T12:19:55.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:55.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:55.737613+0000) 2024-03-06T12:19:56.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:56.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:56.737836+0000) 2024-03-06T12:19:57.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:57.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:57.738141+0000) 2024-03-06T12:19:58.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:58.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:58.738392+0000) 2024-03-06T12:19:59.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:19:59.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:19:59.738618+0000) 2024-03-06T12:20:00.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:00.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:00.738843+0000) 2024-03-06T12:20:01.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:01.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:01.739058+0000) 2024-03-06T12:20:02.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:02.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:02.739308+0000) 2024-03-06T12:20:03.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:03.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:03.739562+0000) 2024-03-06T12:20:04.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:04.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:04.739809+0000) 2024-03-06T12:20:05.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:05.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:05.740090+0000) 2024-03-06T12:20:06.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:06.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:06.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:06.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:06.740351+0000) 2024-03-06T12:20:07.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:07.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:07.740594+0000) 2024-03-06T12:20:08.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:08.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:08.740807+0000) 2024-03-06T12:20:09.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:09.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:09.741075+0000) 2024-03-06T12:20:10.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:10.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:10.741332+0000) 2024-03-06T12:20:11.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:11.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:11.741578+0000) 2024-03-06T12:20:12.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:12.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:12.741851+0000) 2024-03-06T12:20:13.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:13.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:13.742104+0000) 2024-03-06T12:20:14.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:14.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:14.742344+0000) 2024-03-06T12:20:15.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:15.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:15.742584+0000) 2024-03-06T12:20:16.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:16.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:16.742828+0000) 2024-03-06T12:20:17.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:17.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:17.743124+0000) 2024-03-06T12:20:18.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:18.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:18.743356+0000) 2024-03-06T12:20:19.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:19.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:19.743609+0000) 2024-03-06T12:20:20.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:20.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:20.743843+0000) 2024-03-06T12:20:21.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:21.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:21.744107+0000) 2024-03-06T12:20:22.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:22.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:22.744345+0000) 2024-03-06T12:20:23.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:23.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:23.744575+0000) 2024-03-06T12:20:24.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:24.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:24.744845+0000) 2024-03-06T12:20:25.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:25.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:25.745114+0000) 2024-03-06T12:20:26.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:26.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:26.745339+0000) 2024-03-06T12:20:27.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:27.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:27.745569+0000) 2024-03-06T12:20:28.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:28.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:28.745801+0000) 2024-03-06T12:20:29.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:29.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:29.746084+0000) 2024-03-06T12:20:30.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:30.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:30.746308+0000) 2024-03-06T12:20:31.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:31.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:31.746510+0000) 2024-03-06T12:20:32.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:32.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:32.746791+0000) 2024-03-06T12:20:33.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:33.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:33.747118+0000) 2024-03-06T12:20:34.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:34.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:34.747403+0000) 2024-03-06T12:20:35.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:35.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:35.747648+0000) 2024-03-06T12:20:36.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:36.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:20:36.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:36.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:36.747885+0000) 2024-03-06T12:20:37.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:37.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:37.748168+0000) 2024-03-06T12:20:38.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:38.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:38.748376+0000) 2024-03-06T12:20:39.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:39.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:39.748589+0000) 2024-03-06T12:20:40.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:40.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:40.748772+0000) 2024-03-06T12:20:41.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:41.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:41.748980+0000) 2024-03-06T12:20:42.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:42.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:42.749246+0000) 2024-03-06T12:20:43.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:43.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:43.749489+0000) 2024-03-06T12:20:44.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:44.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:44.749728+0000) 2024-03-06T12:20:45.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:45.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:45.749976+0000) 2024-03-06T12:20:46.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:46.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:46.750229+0000) 2024-03-06T12:20:47.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:47.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:47.750412+0000) 2024-03-06T12:20:48.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:48.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:48.750637+0000) 2024-03-06T12:20:49.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:49.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:49.750854+0000) 2024-03-06T12:20:50.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:50.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:50.751140+0000) 2024-03-06T12:20:51.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:51.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:51.751338+0000) 2024-03-06T12:20:52.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:52.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:52.751573+0000) 2024-03-06T12:20:53.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:53.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:53.751799+0000) 2024-03-06T12:20:54.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:54.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:54.752053+0000) 2024-03-06T12:20:55.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:55.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:55.752282+0000) 2024-03-06T12:20:56.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:56.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:56.752507+0000) 2024-03-06T12:20:57.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:57.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:57.752689+0000) 2024-03-06T12:20:58.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:58.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:58.752920+0000) 2024-03-06T12:20:59.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:20:59.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:20:59.753172+0000) 2024-03-06T12:21:00.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:00.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:00.753411+0000) 2024-03-06T12:21:01.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:01.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:01.753669+0000) 2024-03-06T12:21:02.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:02.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:02.753906+0000) 2024-03-06T12:21:03.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:03.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:03.754133+0000) 2024-03-06T12:21:04.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:04.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:04.754343+0000) 2024-03-06T12:21:05.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:05.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:05.754567+0000) 2024-03-06T12:21:06.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:06.563 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:06.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:06.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:06.754802+0000) 2024-03-06T12:21:07.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:07.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:07.755084+0000) 2024-03-06T12:21:08.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:08.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:08.755312+0000) 2024-03-06T12:21:09.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:09.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:09.755536+0000) 2024-03-06T12:21:10.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:10.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:10.755760+0000) 2024-03-06T12:21:11.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:11.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:11.755978+0000) 2024-03-06T12:21:12.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:12.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:12.756306+0000) 2024-03-06T12:21:13.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:13.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:13.756540+0000) 2024-03-06T12:21:14.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:14.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:14.756761+0000) 2024-03-06T12:21:15.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:15.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:15.756992+0000) 2024-03-06T12:21:16.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:16.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:16.757281+0000) 2024-03-06T12:21:17.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:17.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:17.757560+0000) 2024-03-06T12:21:18.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:18.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:18.757845+0000) 2024-03-06T12:21:19.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:19.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:19.758161+0000) 2024-03-06T12:21:20.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:20.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:20.758418+0000) 2024-03-06T12:21:21.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:21.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:21.758707+0000) 2024-03-06T12:21:22.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:22.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:22.759059+0000) 2024-03-06T12:21:23.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:23.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:23.759317+0000) 2024-03-06T12:21:24.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:24.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:24.759576+0000) 2024-03-06T12:21:25.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:25.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:25.759847+0000) 2024-03-06T12:21:26.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:26.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:26.760156+0000) 2024-03-06T12:21:27.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:27.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:27.760448+0000) 2024-03-06T12:21:28.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:28.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:28.760738+0000) 2024-03-06T12:21:29.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:29.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:29.761063+0000) 2024-03-06T12:21:30.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:30.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:30.761303+0000) 2024-03-06T12:21:31.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:31.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:31.761589+0000) 2024-03-06T12:21:32.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:32.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:32.761809+0000) 2024-03-06T12:21:33.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:33.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:33.762098+0000) 2024-03-06T12:21:34.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:34.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:34.762340+0000) 2024-03-06T12:21:35.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:35.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:35.762628+0000) 2024-03-06T12:21:36.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:36.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:21:36.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:36.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:36.762795+0000) 2024-03-06T12:21:37.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:37.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:37.763036+0000) 2024-03-06T12:21:38.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:38.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:38.763324+0000) 2024-03-06T12:21:39.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:39.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:39.763606+0000) 2024-03-06T12:21:40.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:40.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:40.763876+0000) 2024-03-06T12:21:41.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:41.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:41.764122+0000) 2024-03-06T12:21:42.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:42.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:42.764365+0000) 2024-03-06T12:21:43.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:43.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:43.764605+0000) 2024-03-06T12:21:44.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:44.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:44.764791+0000) 2024-03-06T12:21:45.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:45.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:45.764983+0000) 2024-03-06T12:21:46.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:46.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:46.765250+0000) 2024-03-06T12:21:47.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:47.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:47.765457+0000) 2024-03-06T12:21:48.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:48.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:48.765690+0000) 2024-03-06T12:21:49.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:49.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:49.765894+0000) 2024-03-06T12:21:50.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:50.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:50.766123+0000) 2024-03-06T12:21:51.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:51.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:51.766347+0000) 2024-03-06T12:21:52.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:52.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:52.766530+0000) 2024-03-06T12:21:53.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:53.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:53.766771+0000) 2024-03-06T12:21:54.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:54.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:54.766979+0000) 2024-03-06T12:21:55.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:55.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:55.767245+0000) 2024-03-06T12:21:56.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:56.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:56.767436+0000) 2024-03-06T12:21:57.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:57.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:57.767658+0000) 2024-03-06T12:21:58.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:58.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:58.767897+0000) 2024-03-06T12:21:59.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:21:59.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:21:59.768150+0000) 2024-03-06T12:22:00.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:00.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:00.768438+0000) 2024-03-06T12:22:01.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:01.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:01.768714+0000) 2024-03-06T12:22:02.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:02.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:02.768922+0000) 2024-03-06T12:22:03.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:03.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:03.769169+0000) 2024-03-06T12:22:04.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:04.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:04.769421+0000) 2024-03-06T12:22:05.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:05.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:05.769643+0000) 2024-03-06T12:22:06.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:06.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:06.769913+0000) 2024-03-06T12:22:06.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:06.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:07.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:07.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:07.770135+0000) 2024-03-06T12:22:08.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:08.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:08.770327+0000) 2024-03-06T12:22:09.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:09.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:09.770551+0000) 2024-03-06T12:22:10.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:10.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:10.770773+0000) 2024-03-06T12:22:11.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:11.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:11.771037+0000) 2024-03-06T12:22:12.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:12.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:12.771276+0000) 2024-03-06T12:22:13.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:13.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:13.771522+0000) 2024-03-06T12:22:14.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:14.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:14.771797+0000) 2024-03-06T12:22:15.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:15.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:15.772111+0000) 2024-03-06T12:22:16.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:16.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:16.772335+0000) 2024-03-06T12:22:17.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:17.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:17.772524+0000) 2024-03-06T12:22:18.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:18.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:18.772767+0000) 2024-03-06T12:22:19.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:19.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:19.772983+0000) 2024-03-06T12:22:20.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:20.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:20.773287+0000) 2024-03-06T12:22:21.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:21.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:21.773487+0000) 2024-03-06T12:22:22.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:22.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:22.773712+0000) 2024-03-06T12:22:23.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:23.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:23.774037+0000) 2024-03-06T12:22:24.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:24.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:24.774295+0000) 2024-03-06T12:22:25.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:25.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:25.774463+0000) 2024-03-06T12:22:26.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:26.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:26.774685+0000) 2024-03-06T12:22:27.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:27.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:27.774941+0000) 2024-03-06T12:22:28.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:28.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:28.775239+0000) 2024-03-06T12:22:29.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:29.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:29.775517+0000) 2024-03-06T12:22:30.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:30.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:30.775800+0000) 2024-03-06T12:22:31.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:31.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:31.776120+0000) 2024-03-06T12:22:32.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:32.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:32.776407+0000) 2024-03-06T12:22:33.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:33.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:33.776700+0000) 2024-03-06T12:22:34.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:34.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:34.776938+0000) 2024-03-06T12:22:35.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:35.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:35.777197+0000) 2024-03-06T12:22:36.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:36.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:36.777406+0000) 2024-03-06T12:22:37.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:37.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:22:37.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:37.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:37.777639+0000) 2024-03-06T12:22:38.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:38.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:38.777886+0000) 2024-03-06T12:22:39.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:39.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:39.778113+0000) 2024-03-06T12:22:40.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:40.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:40.778340+0000) 2024-03-06T12:22:41.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:41.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:41.778568+0000) 2024-03-06T12:22:42.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:42.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:42.778809+0000) 2024-03-06T12:22:43.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:43.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:43.779088+0000) 2024-03-06T12:22:44.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:44.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:44.779308+0000) 2024-03-06T12:22:45.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:45.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:45.779529+0000) 2024-03-06T12:22:46.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:46.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:46.779758+0000) 2024-03-06T12:22:47.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:47.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:47.779981+0000) 2024-03-06T12:22:48.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:48.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:48.780299+0000) 2024-03-06T12:22:49.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:49.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:49.780506+0000) 2024-03-06T12:22:50.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:50.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:50.780734+0000) 2024-03-06T12:22:51.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:51.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:51.780969+0000) 2024-03-06T12:22:52.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:52.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:52.781274+0000) 2024-03-06T12:22:53.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:53.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:53.781574+0000) 2024-03-06T12:22:54.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:54.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:54.781855+0000) 2024-03-06T12:22:55.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:55.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:55.782168+0000) 2024-03-06T12:22:56.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:56.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:56.782451+0000) 2024-03-06T12:22:57.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:57.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:57.782706+0000) 2024-03-06T12:22:58.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:58.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:58.782968+0000) 2024-03-06T12:22:59.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:22:59.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:22:59.783256+0000) 2024-03-06T12:23:00.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:00.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:00.783430+0000) 2024-03-06T12:23:01.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:01.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:01.783653+0000) 2024-03-06T12:23:02.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:02.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:02.783910+0000) 2024-03-06T12:23:03.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:03.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:03.784183+0000) 2024-03-06T12:23:04.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:04.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:04.784389+0000) 2024-03-06T12:23:05.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:05.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:05.784594+0000) 2024-03-06T12:23:06.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:06.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:06.784790+0000) 2024-03-06T12:23:07.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:07.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:07.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:07.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:07.785050+0000) 2024-03-06T12:23:08.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:08.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:08.785238+0000) 2024-03-06T12:23:09.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:09.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:09.785444+0000) 2024-03-06T12:23:10.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:10.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:10.785682+0000) 2024-03-06T12:23:11.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:11.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:11.785911+0000) 2024-03-06T12:23:12.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:12.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:12.786127+0000) 2024-03-06T12:23:13.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:13.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:13.786341+0000) 2024-03-06T12:23:14.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:14.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:14.786572+0000) 2024-03-06T12:23:15.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:15.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:15.786783+0000) 2024-03-06T12:23:16.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:16.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:16.787049+0000) 2024-03-06T12:23:17.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:17.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:17.787284+0000) 2024-03-06T12:23:18.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:18.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:18.787529+0000) 2024-03-06T12:23:19.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:19.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:19.787772+0000) 2024-03-06T12:23:20.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:20.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:20.788035+0000) 2024-03-06T12:23:21.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:21.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:21.788202+0000) 2024-03-06T12:23:22.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:22.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:22.788436+0000) 2024-03-06T12:23:23.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:23.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:23.788651+0000) 2024-03-06T12:23:24.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:24.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:24.788920+0000) 2024-03-06T12:23:25.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:25.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:25.789172+0000) 2024-03-06T12:23:26.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:26.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:26.789451+0000) 2024-03-06T12:23:27.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:27.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:27.789695+0000) 2024-03-06T12:23:28.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:28.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:28.789946+0000) 2024-03-06T12:23:29.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:29.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:29.790165+0000) 2024-03-06T12:23:30.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:30.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:30.790435+0000) 2024-03-06T12:23:31.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:31.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:31.790720+0000) 2024-03-06T12:23:32.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:32.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:32.791055+0000) 2024-03-06T12:23:33.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:33.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:33.791275+0000) 2024-03-06T12:23:34.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:34.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:34.791528+0000) 2024-03-06T12:23:35.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:35.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:35.791803+0000) 2024-03-06T12:23:36.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:36.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:36.792047+0000) 2024-03-06T12:23:37.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:37.459 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:23:37.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:37.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:37.792249+0000) 2024-03-06T12:23:38.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:38.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:38.792471+0000) 2024-03-06T12:23:39.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:39.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:39.792700+0000) 2024-03-06T12:23:40.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:40.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:40.792829+0000) 2024-03-06T12:23:41.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:41.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:41.793075+0000) 2024-03-06T12:23:42.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:42.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:42.793342+0000) 2024-03-06T12:23:43.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:43.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:43.793556+0000) 2024-03-06T12:23:44.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:44.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:44.793782+0000) 2024-03-06T12:23:45.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:45.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:45.794031+0000) 2024-03-06T12:23:46.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:46.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:46.794266+0000) 2024-03-06T12:23:47.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:47.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:47.794492+0000) 2024-03-06T12:23:48.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:48.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:48.794718+0000) 2024-03-06T12:23:49.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:49.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:49.794943+0000) 2024-03-06T12:23:50.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:50.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:50.795178+0000) 2024-03-06T12:23:51.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:51.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:51.795393+0000) 2024-03-06T12:23:52.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:52.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:52.795587+0000) 2024-03-06T12:23:53.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:53.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:53.795787+0000) 2024-03-06T12:23:54.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:54.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:54.796044+0000) 2024-03-06T12:23:55.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:55.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:55.796258+0000) 2024-03-06T12:23:56.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:56.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:56.796476+0000) 2024-03-06T12:23:57.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:57.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:57.796700+0000) 2024-03-06T12:23:58.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:58.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:58.796920+0000) 2024-03-06T12:23:59.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:23:59.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:23:59.797153+0000) 2024-03-06T12:24:00.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:00.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:00.797400+0000) 2024-03-06T12:24:01.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:01.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:01.797580+0000) 2024-03-06T12:24:02.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:02.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:02.797767+0000) 2024-03-06T12:24:03.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:03.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:03.797989+0000) 2024-03-06T12:24:04.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:04.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:04.798237+0000) 2024-03-06T12:24:05.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:05.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:05.798501+0000) 2024-03-06T12:24:06.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:06.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:06.798736+0000) 2024-03-06T12:24:07.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:07.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:07.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:07.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:07.799041+0000) 2024-03-06T12:24:08.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:08.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:08.799324+0000) 2024-03-06T12:24:09.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:09.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:09.799562+0000) 2024-03-06T12:24:10.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:10.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:10.799787+0000) 2024-03-06T12:24:11.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:11.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:11.800114+0000) 2024-03-06T12:24:12.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:12.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:12.800397+0000) 2024-03-06T12:24:13.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:13.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:13.800642+0000) 2024-03-06T12:24:14.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:14.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:14.800887+0000) 2024-03-06T12:24:15.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:15.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:15.801176+0000) 2024-03-06T12:24:16.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:16.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:16.801447+0000) 2024-03-06T12:24:17.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:17.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:17.801727+0000) 2024-03-06T12:24:18.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:18.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:18.802043+0000) 2024-03-06T12:24:19.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:19.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:19.802326+0000) 2024-03-06T12:24:20.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:20.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:20.802568+0000) 2024-03-06T12:24:21.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:21.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:21.802820+0000) 2024-03-06T12:24:22.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:22.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:22.803153+0000) 2024-03-06T12:24:23.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:23.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:23.803375+0000) 2024-03-06T12:24:24.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:24.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:24.803642+0000) 2024-03-06T12:24:25.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:25.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:25.803814+0000) 2024-03-06T12:24:26.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:26.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:26.804081+0000) 2024-03-06T12:24:27.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:27.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:27.804323+0000) 2024-03-06T12:24:28.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:28.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:28.804574+0000) 2024-03-06T12:24:29.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:29.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:29.804775+0000) 2024-03-06T12:24:30.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:30.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:30.804966+0000) 2024-03-06T12:24:31.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:31.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:31.805239+0000) 2024-03-06T12:24:32.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:32.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:32.805474+0000) 2024-03-06T12:24:33.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:33.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:33.805692+0000) 2024-03-06T12:24:34.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:34.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:34.805921+0000) 2024-03-06T12:24:35.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:35.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:35.806205+0000) 2024-03-06T12:24:36.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:36.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:36.806403+0000) 2024-03-06T12:24:37.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:37.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:37.806591+0000) 2024-03-06T12:24:37.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:37.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:24:38.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:38.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:38.806822+0000) 2024-03-06T12:24:39.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:39.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:39.807126+0000) 2024-03-06T12:24:40.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:40.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:40.807369+0000) 2024-03-06T12:24:41.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:41.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:41.807551+0000) 2024-03-06T12:24:42.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:42.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:42.807810+0000) 2024-03-06T12:24:43.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:43.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:43.808095+0000) 2024-03-06T12:24:44.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:44.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:44.808345+0000) 2024-03-06T12:24:45.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:45.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:45.808578+0000) 2024-03-06T12:24:46.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:46.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:46.808763+0000) 2024-03-06T12:24:47.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:47.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:47.809047+0000) 2024-03-06T12:24:48.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:48.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:48.809294+0000) 2024-03-06T12:24:49.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:49.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:49.809506+0000) 2024-03-06T12:24:50.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:50.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:50.809742+0000) 2024-03-06T12:24:51.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:51.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:51.809977+0000) 2024-03-06T12:24:52.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:52.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:52.810307+0000) 2024-03-06T12:24:53.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:53.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:53.810548+0000) 2024-03-06T12:24:54.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:54.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:54.810790+0000) 2024-03-06T12:24:55.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:55.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:55.811062+0000) 2024-03-06T12:24:56.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:56.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:56.811299+0000) 2024-03-06T12:24:57.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:57.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:57.811496+0000) 2024-03-06T12:24:58.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:58.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:58.811715+0000) 2024-03-06T12:24:59.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:24:59.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:24:59.811946+0000) 2024-03-06T12:25:00.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:00.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:00.812181+0000) 2024-03-06T12:25:01.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:01.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:01.812403+0000) 2024-03-06T12:25:02.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:02.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:02.812639+0000) 2024-03-06T12:25:03.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:03.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:03.812802+0000) 2024-03-06T12:25:04.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:04.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:04.813060+0000) 2024-03-06T12:25:05.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:05.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:05.813289+0000) 2024-03-06T12:25:06.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:06.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:06.813506+0000) 2024-03-06T12:25:07.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:07.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:07.813715+0000) 2024-03-06T12:25:08.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:08.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:08.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:08.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:08.813972+0000) 2024-03-06T12:25:09.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:09.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:09.814233+0000) 2024-03-06T12:25:10.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:10.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:10.814455+0000) 2024-03-06T12:25:11.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:11.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:11.814636+0000) 2024-03-06T12:25:12.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:12.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:12.814895+0000) 2024-03-06T12:25:13.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:13.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:13.815151+0000) 2024-03-06T12:25:14.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:14.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:14.815400+0000) 2024-03-06T12:25:15.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:15.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:15.815658+0000) 2024-03-06T12:25:16.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:16.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:16.815906+0000) 2024-03-06T12:25:17.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:17.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:17.816157+0000) 2024-03-06T12:25:18.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:18.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:18.816368+0000) 2024-03-06T12:25:19.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:19.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:19.816613+0000) 2024-03-06T12:25:20.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:20.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:20.816863+0000) 2024-03-06T12:25:21.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:21.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:21.817101+0000) 2024-03-06T12:25:22.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:22.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:22.817334+0000) 2024-03-06T12:25:23.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:23.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:23.817587+0000) 2024-03-06T12:25:24.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:24.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:24.817815+0000) 2024-03-06T12:25:25.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:25.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:25.818079+0000) 2024-03-06T12:25:26.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:26.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:26.818302+0000) 2024-03-06T12:25:27.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:27.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:27.818525+0000) 2024-03-06T12:25:28.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:28.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:28.818750+0000) 2024-03-06T12:25:29.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:29.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:29.819051+0000) 2024-03-06T12:25:30.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:30.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:30.819292+0000) 2024-03-06T12:25:31.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:31.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:31.819527+0000) 2024-03-06T12:25:32.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:32.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:32.819780+0000) 2024-03-06T12:25:33.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:33.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:33.820046+0000) 2024-03-06T12:25:34.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:34.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:34.820300+0000) 2024-03-06T12:25:35.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:35.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:35.820535+0000) 2024-03-06T12:25:36.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:36.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:36.820764+0000) 2024-03-06T12:25:37.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:37.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:37.820983+0000) 2024-03-06T12:25:38.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:38.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:25:38.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:38.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:38.821260+0000) 2024-03-06T12:25:39.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:39.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:39.821504+0000) 2024-03-06T12:25:40.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:40.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:40.821693+0000) 2024-03-06T12:25:41.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:41.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:41.821919+0000) 2024-03-06T12:25:42.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:42.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:42.822154+0000) 2024-03-06T12:25:43.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:43.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:43.822372+0000) 2024-03-06T12:25:44.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:44.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:44.822603+0000) 2024-03-06T12:25:45.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:45.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:45.822811+0000) 2024-03-06T12:25:46.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:46.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:46.823081+0000) 2024-03-06T12:25:47.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:47.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:47.823314+0000) 2024-03-06T12:25:48.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:48.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:48.823536+0000) 2024-03-06T12:25:49.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:49.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:49.823756+0000) 2024-03-06T12:25:50.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:50.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:50.823991+0000) 2024-03-06T12:25:51.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:51.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:51.824264+0000) 2024-03-06T12:25:52.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:52.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:52.824516+0000) 2024-03-06T12:25:53.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:53.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:53.824729+0000) 2024-03-06T12:25:54.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:54.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:54.824949+0000) 2024-03-06T12:25:55.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:55.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:55.825186+0000) 2024-03-06T12:25:56.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:56.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:56.825387+0000) 2024-03-06T12:25:57.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:57.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:57.825633+0000) 2024-03-06T12:25:58.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:58.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:58.825865+0000) 2024-03-06T12:25:59.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:25:59.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:25:59.826132+0000) 2024-03-06T12:26:00.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:00.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:00.826374+0000) 2024-03-06T12:26:01.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:01.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:01.826614+0000) 2024-03-06T12:26:02.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:02.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:02.826843+0000) 2024-03-06T12:26:03.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:03.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:03.827124+0000) 2024-03-06T12:26:04.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:04.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:04.827318+0000) 2024-03-06T12:26:05.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:05.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:05.827530+0000) 2024-03-06T12:26:06.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:06.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:06.827705+0000) 2024-03-06T12:26:07.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:07.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:07.827935+0000) 2024-03-06T12:26:08.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:08.377 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:08.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:08.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:08.828186+0000) 2024-03-06T12:26:09.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:09.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:09.828408+0000) 2024-03-06T12:26:10.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:10.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:10.828648+0000) 2024-03-06T12:26:11.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:11.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:11.828851+0000) 2024-03-06T12:26:12.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:12.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:12.829117+0000) 2024-03-06T12:26:13.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:13.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:13.829339+0000) 2024-03-06T12:26:14.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:14.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:14.829527+0000) 2024-03-06T12:26:15.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:15.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:15.829728+0000) 2024-03-06T12:26:16.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:16.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:16.829954+0000) 2024-03-06T12:26:17.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:17.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:17.830204+0000) 2024-03-06T12:26:18.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:18.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:18.830445+0000) 2024-03-06T12:26:19.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:19.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:19.830679+0000) 2024-03-06T12:26:20.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:20.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:20.830902+0000) 2024-03-06T12:26:21.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:21.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:21.831126+0000) 2024-03-06T12:26:22.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:22.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:22.831362+0000) 2024-03-06T12:26:23.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:23.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:23.831589+0000) 2024-03-06T12:26:24.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:24.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:24.831816+0000) 2024-03-06T12:26:25.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:25.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:25.832097+0000) 2024-03-06T12:26:26.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:26.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:26.832343+0000) 2024-03-06T12:26:27.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:27.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:27.832583+0000) 2024-03-06T12:26:28.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:28.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:28.832806+0000) 2024-03-06T12:26:29.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:29.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:29.833074+0000) 2024-03-06T12:26:30.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:30.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:30.833267+0000) 2024-03-06T12:26:31.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:31.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:31.833410+0000) 2024-03-06T12:26:32.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:32.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:32.833630+0000) 2024-03-06T12:26:33.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:33.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:33.833843+0000) 2024-03-06T12:26:34.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:34.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:34.834131+0000) 2024-03-06T12:26:35.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:35.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:35.834311+0000) 2024-03-06T12:26:36.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:36.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:36.834532+0000) 2024-03-06T12:26:37.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:37.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:37.834750+0000) 2024-03-06T12:26:38.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:38.581 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:26:38.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:38.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:38.834970+0000) 2024-03-06T12:26:39.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:39.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:39.835190+0000) 2024-03-06T12:26:40.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:40.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:40.835378+0000) 2024-03-06T12:26:41.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:41.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:41.835599+0000) 2024-03-06T12:26:42.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:42.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:42.835835+0000) 2024-03-06T12:26:43.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:43.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:43.836072+0000) 2024-03-06T12:26:44.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:44.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:44.836304+0000) 2024-03-06T12:26:45.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:45.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:45.836527+0000) 2024-03-06T12:26:46.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:46.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:46.836769+0000) 2024-03-06T12:26:47.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:47.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:47.837063+0000) 2024-03-06T12:26:48.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:48.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:48.837314+0000) 2024-03-06T12:26:49.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:49.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:49.837568+0000) 2024-03-06T12:26:50.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:50.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:50.837804+0000) 2024-03-06T12:26:51.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:51.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:51.838072+0000) 2024-03-06T12:26:52.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:52.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:52.838348+0000) 2024-03-06T12:26:53.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:53.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:53.838590+0000) 2024-03-06T12:26:54.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:54.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:54.838839+0000) 2024-03-06T12:26:55.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:55.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:55.839140+0000) 2024-03-06T12:26:56.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:56.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:56.839382+0000) 2024-03-06T12:26:57.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:57.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:57.839620+0000) 2024-03-06T12:26:58.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:58.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:58.839840+0000) 2024-03-06T12:26:59.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:26:59.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:26:59.840074+0000) 2024-03-06T12:27:00.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:00.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:00.840302+0000) 2024-03-06T12:27:01.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:01.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:01.840581+0000) 2024-03-06T12:27:02.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:02.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:02.840886+0000) 2024-03-06T12:27:03.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:03.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:03.841130+0000) 2024-03-06T12:27:04.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:04.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:04.841356+0000) 2024-03-06T12:27:05.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:05.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:05.841574+0000) 2024-03-06T12:27:06.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:06.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:06.841838+0000) 2024-03-06T12:27:07.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:07.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:07.842117+0000) 2024-03-06T12:27:08.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:08.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:08.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:08.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:08.842295+0000) 2024-03-06T12:27:09.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:09.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:09.842521+0000) 2024-03-06T12:27:10.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:10.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:10.842844+0000) 2024-03-06T12:27:11.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:11.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:11.843156+0000) 2024-03-06T12:27:12.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:12.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:12.843450+0000) 2024-03-06T12:27:13.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:13.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:13.843678+0000) 2024-03-06T12:27:14.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:14.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:14.843859+0000) 2024-03-06T12:27:15.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:15.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:15.844131+0000) 2024-03-06T12:27:16.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:16.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:16.844366+0000) 2024-03-06T12:27:17.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:17.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:17.844621+0000) 2024-03-06T12:27:18.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:18.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:18.844875+0000) 2024-03-06T12:27:19.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:19.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:19.845183+0000) 2024-03-06T12:27:20.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:20.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:20.845445+0000) 2024-03-06T12:27:21.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:21.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:21.845713+0000) 2024-03-06T12:27:22.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:22.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:22.845998+0000) 2024-03-06T12:27:23.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:23.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:23.846286+0000) 2024-03-06T12:27:24.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:24.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:24.846547+0000) 2024-03-06T12:27:25.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:25.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:25.846785+0000) 2024-03-06T12:27:26.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:26.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:26.847052+0000) 2024-03-06T12:27:27.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:27.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:27.847315+0000) 2024-03-06T12:27:28.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:28.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:28.847520+0000) 2024-03-06T12:27:29.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:29.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:29.847757+0000) 2024-03-06T12:27:30.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:30.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:30.847982+0000) 2024-03-06T12:27:31.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:31.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:31.848260+0000) 2024-03-06T12:27:32.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:32.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:32.848521+0000) 2024-03-06T12:27:33.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:33.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:33.848722+0000) 2024-03-06T12:27:34.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:34.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:34.848952+0000) 2024-03-06T12:27:35.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:35.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:35.849183+0000) 2024-03-06T12:27:36.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:36.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:36.849384+0000) 2024-03-06T12:27:37.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:37.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:37.849592+0000) 2024-03-06T12:27:38.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:38.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:38.849806+0000) 2024-03-06T12:27:38.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:39.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:27:39.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:39.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:39.850079+0000) 2024-03-06T12:27:40.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:40.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:40.850277+0000) 2024-03-06T12:27:41.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:41.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:41.850473+0000) 2024-03-06T12:27:42.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:42.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:42.850720+0000) 2024-03-06T12:27:43.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:43.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:43.850947+0000) 2024-03-06T12:27:44.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:44.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:44.851184+0000) 2024-03-06T12:27:45.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:45.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:45.851412+0000) 2024-03-06T12:27:46.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:46.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:46.851626+0000) 2024-03-06T12:27:47.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:47.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:47.851882+0000) 2024-03-06T12:27:48.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:48.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:48.852159+0000) 2024-03-06T12:27:49.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:49.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:49.852390+0000) 2024-03-06T12:27:50.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:50.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:50.852608+0000) 2024-03-06T12:27:51.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:51.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:51.852813+0000) 2024-03-06T12:27:52.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:52.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:52.853085+0000) 2024-03-06T12:27:53.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:53.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:53.853351+0000) 2024-03-06T12:27:54.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:54.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:54.853581+0000) 2024-03-06T12:27:55.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:55.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:55.853791+0000) 2024-03-06T12:27:56.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:56.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:56.853994+0000) 2024-03-06T12:27:57.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:57.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:57.854235+0000) 2024-03-06T12:27:58.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:58.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:58.854482+0000) 2024-03-06T12:27:59.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:27:59.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:27:59.854744+0000) 2024-03-06T12:28:00.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:00.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:00.854998+0000) 2024-03-06T12:28:01.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:01.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:01.855260+0000) 2024-03-06T12:28:02.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:02.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:02.855490+0000) 2024-03-06T12:28:03.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:03.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:03.855698+0000) 2024-03-06T12:28:04.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:04.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:04.855915+0000) 2024-03-06T12:28:05.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:05.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:05.856165+0000) 2024-03-06T12:28:06.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:06.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:06.856387+0000) 2024-03-06T12:28:07.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:07.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:07.856576+0000) 2024-03-06T12:28:08.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:08.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:08.856817+0000) 2024-03-06T12:28:09.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:09.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:09.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:09.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:09.857108+0000) 2024-03-06T12:28:10.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:10.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:10.857350+0000) 2024-03-06T12:28:11.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:11.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:11.857599+0000) 2024-03-06T12:28:12.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:12.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:12.857849+0000) 2024-03-06T12:28:13.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:13.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:13.858140+0000) 2024-03-06T12:28:14.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:14.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:14.858392+0000) 2024-03-06T12:28:15.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:15.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:15.858632+0000) 2024-03-06T12:28:16.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:16.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:16.858816+0000) 2024-03-06T12:28:17.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:17.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:17.859073+0000) 2024-03-06T12:28:18.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:18.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:18.859308+0000) 2024-03-06T12:28:19.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:19.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:19.859574+0000) 2024-03-06T12:28:20.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:20.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:20.859808+0000) 2024-03-06T12:28:21.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:21.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:21.860040+0000) 2024-03-06T12:28:22.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:22.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:22.860264+0000) 2024-03-06T12:28:23.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:23.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:23.860489+0000) 2024-03-06T12:28:24.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:24.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:24.860727+0000) 2024-03-06T12:28:25.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:25.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:25.860971+0000) 2024-03-06T12:28:26.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:26.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:26.861208+0000) 2024-03-06T12:28:27.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:27.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:27.861460+0000) 2024-03-06T12:28:28.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:28.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:28.861719+0000) 2024-03-06T12:28:29.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:29.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:29.861982+0000) 2024-03-06T12:28:30.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:30.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:30.862249+0000) 2024-03-06T12:28:31.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:31.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:31.862424+0000) 2024-03-06T12:28:32.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:32.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:32.862610+0000) 2024-03-06T12:28:33.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:33.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:33.862838+0000) 2024-03-06T12:28:34.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:34.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:34.863041+0000) 2024-03-06T12:28:35.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:35.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:35.863275+0000) 2024-03-06T12:28:36.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:36.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:36.863448+0000) 2024-03-06T12:28:37.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:37.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:37.863643+0000) 2024-03-06T12:28:38.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:38.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:38.863861+0000) 2024-03-06T12:28:39.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:39.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:28:39.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:39.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:39.864136+0000) 2024-03-06T12:28:40.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:40.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:40.864420+0000) 2024-03-06T12:28:41.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:41.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:41.864609+0000) 2024-03-06T12:28:42.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:42.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:42.864818+0000) 2024-03-06T12:28:43.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:43.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:43.865079+0000) 2024-03-06T12:28:44.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:44.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:44.865319+0000) 2024-03-06T12:28:45.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:45.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:45.865524+0000) 2024-03-06T12:28:46.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:46.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:46.865762+0000) 2024-03-06T12:28:47.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:47.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:47.865951+0000) 2024-03-06T12:28:48.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:48.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:48.866183+0000) 2024-03-06T12:28:49.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:49.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:49.866406+0000) 2024-03-06T12:28:50.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:50.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:50.866634+0000) 2024-03-06T12:28:51.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:51.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:51.866813+0000) 2024-03-06T12:28:52.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:52.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:52.867066+0000) 2024-03-06T12:28:53.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:53.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:53.867300+0000) 2024-03-06T12:28:54.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:54.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:54.867518+0000) 2024-03-06T12:28:55.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:55.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:55.867747+0000) 2024-03-06T12:28:56.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:56.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:56.867923+0000) 2024-03-06T12:28:57.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:57.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:57.868125+0000) 2024-03-06T12:28:58.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:58.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:58.868308+0000) 2024-03-06T12:28:59.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:28:59.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:28:59.868543+0000) 2024-03-06T12:29:00.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:00.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:00.868769+0000) 2024-03-06T12:29:01.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:01.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:01.868990+0000) 2024-03-06T12:29:02.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:02.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:02.869273+0000) 2024-03-06T12:29:03.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:03.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:03.869494+0000) 2024-03-06T12:29:04.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:04.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:04.869722+0000) 2024-03-06T12:29:05.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:05.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:05.869949+0000) 2024-03-06T12:29:06.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:06.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:06.870178+0000) 2024-03-06T12:29:07.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:07.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:07.870373+0000) 2024-03-06T12:29:08.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:08.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:08.870602+0000) 2024-03-06T12:29:09.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:09.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:09.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:09.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:09.870833+0000) 2024-03-06T12:29:10.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:10.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:10.871098+0000) 2024-03-06T12:29:11.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:11.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:11.871274+0000) 2024-03-06T12:29:12.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:12.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:12.871506+0000) 2024-03-06T12:29:13.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:13.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:13.871720+0000) 2024-03-06T12:29:14.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:14.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:14.871892+0000) 2024-03-06T12:29:15.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:15.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:15.872165+0000) 2024-03-06T12:29:16.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:16.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:16.872412+0000) 2024-03-06T12:29:17.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:17.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:17.872638+0000) 2024-03-06T12:29:18.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:18.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:18.872863+0000) 2024-03-06T12:29:19.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:19.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:19.873115+0000) 2024-03-06T12:29:20.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:20.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:20.873347+0000) 2024-03-06T12:29:21.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:21.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:21.873571+0000) 2024-03-06T12:29:22.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:22.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:22.873801+0000) 2024-03-06T12:29:23.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:23.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:23.874061+0000) 2024-03-06T12:29:24.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:24.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:24.874295+0000) 2024-03-06T12:29:25.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:25.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:25.874523+0000) 2024-03-06T12:29:26.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:26.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:26.874660+0000) 2024-03-06T12:29:27.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:27.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:27.874924+0000) 2024-03-06T12:29:28.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:28.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:28.875200+0000) 2024-03-06T12:29:29.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:29.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:29.875472+0000) 2024-03-06T12:29:30.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:30.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:30.875749+0000) 2024-03-06T12:29:31.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:31.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:31.875942+0000) 2024-03-06T12:29:32.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:32.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:32.876140+0000) 2024-03-06T12:29:33.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:33.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:33.876372+0000) 2024-03-06T12:29:34.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:34.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:34.876686+0000) 2024-03-06T12:29:35.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:35.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:35.876957+0000) 2024-03-06T12:29:36.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:36.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:36.877191+0000) 2024-03-06T12:29:37.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:37.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:37.877377+0000) 2024-03-06T12:29:38.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:38.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:38.877599+0000) 2024-03-06T12:29:39.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:39.838 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:29:39.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:39.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:39.877835+0000) 2024-03-06T12:29:40.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:40.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:40.878110+0000) 2024-03-06T12:29:41.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:41.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:41.878338+0000) 2024-03-06T12:29:42.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:42.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:42.878531+0000) 2024-03-06T12:29:43.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:43.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:43.878724+0000) 2024-03-06T12:29:44.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:44.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:44.878950+0000) 2024-03-06T12:29:45.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:45.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:45.879177+0000) 2024-03-06T12:29:46.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:46.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:46.879392+0000) 2024-03-06T12:29:47.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:47.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:47.879654+0000) 2024-03-06T12:29:48.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:48.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:48.879857+0000) 2024-03-06T12:29:49.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:49.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:49.880120+0000) 2024-03-06T12:29:50.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:50.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:50.880315+0000) 2024-03-06T12:29:51.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:51.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:51.880530+0000) 2024-03-06T12:29:52.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:52.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:52.880732+0000) 2024-03-06T12:29:53.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:53.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:53.880972+0000) 2024-03-06T12:29:54.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:54.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:54.881257+0000) 2024-03-06T12:29:55.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:55.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:55.881445+0000) 2024-03-06T12:29:56.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:56.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:56.881604+0000) 2024-03-06T12:29:57.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:57.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:57.881830+0000) 2024-03-06T12:29:58.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:58.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:58.881988+0000) 2024-03-06T12:29:59.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:29:59.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:29:59.882183+0000) 2024-03-06T12:30:00.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:00.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:00.882416+0000) 2024-03-06T12:30:01.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:01.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:01.882598+0000) 2024-03-06T12:30:02.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:02.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:02.882821+0000) 2024-03-06T12:30:03.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:03.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:03.883089+0000) 2024-03-06T12:30:04.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:04.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:04.883293+0000) 2024-03-06T12:30:05.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:05.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:05.883525+0000) 2024-03-06T12:30:06.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:06.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:06.883770+0000) 2024-03-06T12:30:07.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:07.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:07.883968+0000) 2024-03-06T12:30:08.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:08.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:08.884258+0000) 2024-03-06T12:30:09.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:09.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:09.884501+0000) 2024-03-06T12:30:10.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:10.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:10.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:10.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:10.884745+0000) 2024-03-06T12:30:11.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:11.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:11.884945+0000) 2024-03-06T12:30:12.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:12.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:12.885110+0000) 2024-03-06T12:30:13.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:13.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:13.885326+0000) 2024-03-06T12:30:14.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:14.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:14.885559+0000) 2024-03-06T12:30:15.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:15.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:15.885791+0000) 2024-03-06T12:30:16.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:16.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:16.886060+0000) 2024-03-06T12:30:17.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:17.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:17.886296+0000) 2024-03-06T12:30:18.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:18.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:18.886517+0000) 2024-03-06T12:30:19.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:19.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:19.886740+0000) 2024-03-06T12:30:20.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:20.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:20.886898+0000) 2024-03-06T12:30:21.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:21.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:21.887039+0000) 2024-03-06T12:30:22.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:22.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:22.887222+0000) 2024-03-06T12:30:23.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:23.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:23.887445+0000) 2024-03-06T12:30:24.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:24.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:24.887660+0000) 2024-03-06T12:30:25.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:25.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:25.887890+0000) 2024-03-06T12:30:26.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:26.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:26.888110+0000) 2024-03-06T12:30:27.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:27.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:27.888350+0000) 2024-03-06T12:30:28.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:28.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:28.888575+0000) 2024-03-06T12:30:29.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:29.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:29.888802+0000) 2024-03-06T12:30:30.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:30.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:30.889111+0000) 2024-03-06T12:30:31.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:31.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:31.889317+0000) 2024-03-06T12:30:32.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:32.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:32.889514+0000) 2024-03-06T12:30:33.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:33.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:33.889739+0000) 2024-03-06T12:30:34.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:34.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:34.889966+0000) 2024-03-06T12:30:35.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:35.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:35.890230+0000) 2024-03-06T12:30:36.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:36.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:36.890458+0000) 2024-03-06T12:30:37.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:37.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:37.890683+0000) 2024-03-06T12:30:38.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:38.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:38.890911+0000) 2024-03-06T12:30:39.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:39.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:39.891141+0000) 2024-03-06T12:30:40.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:40.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:30:40.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:40.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:40.891366+0000) 2024-03-06T12:30:41.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:41.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:41.891569+0000) 2024-03-06T12:30:42.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:42.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:42.891818+0000) 2024-03-06T12:30:43.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:43.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:43.892045+0000) 2024-03-06T12:30:44.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:44.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:44.892274+0000) 2024-03-06T12:30:45.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:45.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:45.892488+0000) 2024-03-06T12:30:46.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:46.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:46.892714+0000) 2024-03-06T12:30:47.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:47.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:47.892961+0000) 2024-03-06T12:30:48.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:48.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:48.893183+0000) 2024-03-06T12:30:49.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:49.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:49.893368+0000) 2024-03-06T12:30:50.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:50.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:50.893566+0000) 2024-03-06T12:30:51.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:51.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:51.893760+0000) 2024-03-06T12:30:52.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:52.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:52.893923+0000) 2024-03-06T12:30:53.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:53.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:53.894149+0000) 2024-03-06T12:30:54.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:54.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:54.894384+0000) 2024-03-06T12:30:55.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:55.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:55.894624+0000) 2024-03-06T12:30:56.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:56.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:56.894854+0000) 2024-03-06T12:30:57.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:57.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:57.895126+0000) 2024-03-06T12:30:58.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:58.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:58.895363+0000) 2024-03-06T12:30:59.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:30:59.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:30:59.895602+0000) 2024-03-06T12:31:00.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:00.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:00.895850+0000) 2024-03-06T12:31:01.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:01.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:01.896055+0000) 2024-03-06T12:31:02.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:02.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:02.896323+0000) 2024-03-06T12:31:03.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:03.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:03.896568+0000) 2024-03-06T12:31:04.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:04.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:04.896816+0000) 2024-03-06T12:31:05.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:05.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:05.897121+0000) 2024-03-06T12:31:06.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:06.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:06.897366+0000) 2024-03-06T12:31:07.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:07.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:07.897600+0000) 2024-03-06T12:31:08.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:08.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:08.897796+0000) 2024-03-06T12:31:09.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:09.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:09.898058+0000) 2024-03-06T12:31:10.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:10.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:10.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:10.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:10.898289+0000) 2024-03-06T12:31:11.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:11.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:11.898447+0000) 2024-03-06T12:31:12.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:12.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:12.898702+0000) 2024-03-06T12:31:13.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:13.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:13.898943+0000) 2024-03-06T12:31:14.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:14.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:14.899152+0000) 2024-03-06T12:31:15.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:15.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:15.899408+0000) 2024-03-06T12:31:16.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:16.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:16.899660+0000) 2024-03-06T12:31:17.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:17.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:17.899915+0000) 2024-03-06T12:31:18.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:18.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:18.900164+0000) 2024-03-06T12:31:19.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:19.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:19.900392+0000) 2024-03-06T12:31:20.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:20.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:20.900670+0000) 2024-03-06T12:31:21.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:21.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:21.900894+0000) 2024-03-06T12:31:22.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:22.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:22.901135+0000) 2024-03-06T12:31:23.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:23.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:23.901342+0000) 2024-03-06T12:31:24.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:24.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:24.901583+0000) 2024-03-06T12:31:25.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:25.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:25.901815+0000) 2024-03-06T12:31:26.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:26.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:26.901973+0000) 2024-03-06T12:31:27.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:27.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:27.902151+0000) 2024-03-06T12:31:28.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:28.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:28.902340+0000) 2024-03-06T12:31:29.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:29.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:29.902569+0000) 2024-03-06T12:31:30.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:30.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:30.902795+0000) 2024-03-06T12:31:31.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:31.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:31.902995+0000) 2024-03-06T12:31:32.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:32.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:32.903302+0000) 2024-03-06T12:31:33.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:33.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:33.903490+0000) 2024-03-06T12:31:34.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:34.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:34.903722+0000) 2024-03-06T12:31:35.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:35.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:35.903984+0000) 2024-03-06T12:31:36.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:36.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:36.904303+0000) 2024-03-06T12:31:37.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:37.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:37.904534+0000) 2024-03-06T12:31:38.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:38.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:38.904778+0000) 2024-03-06T12:31:39.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:39.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:39.905121+0000) 2024-03-06T12:31:40.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:40.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:31:40.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:40.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:40.905346+0000) 2024-03-06T12:31:41.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:41.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:41.905605+0000) 2024-03-06T12:31:42.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:42.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:42.905892+0000) 2024-03-06T12:31:43.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:43.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:43.906177+0000) 2024-03-06T12:31:44.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:44.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:44.906382+0000) 2024-03-06T12:31:45.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:45.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:45.906604+0000) 2024-03-06T12:31:46.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:46.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:46.906803+0000) 2024-03-06T12:31:47.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:47.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:47.907072+0000) 2024-03-06T12:31:48.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:48.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:48.907209+0000) 2024-03-06T12:31:49.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:49.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:49.907390+0000) 2024-03-06T12:31:50.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:50.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:50.907615+0000) 2024-03-06T12:31:51.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:51.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:51.907831+0000) 2024-03-06T12:31:52.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:52.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:52.908102+0000) 2024-03-06T12:31:53.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:53.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:53.908309+0000) 2024-03-06T12:31:54.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:54.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:54.908546+0000) 2024-03-06T12:31:55.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:55.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:55.908789+0000) 2024-03-06T12:31:56.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:56.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:56.909040+0000) 2024-03-06T12:31:57.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:57.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:57.909271+0000) 2024-03-06T12:31:58.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:58.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:58.909498+0000) 2024-03-06T12:31:59.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:31:59.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:31:59.909696+0000) 2024-03-06T12:32:00.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:00.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:00.909886+0000) 2024-03-06T12:32:01.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:01.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:01.910097+0000) 2024-03-06T12:32:02.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:02.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:02.910321+0000) 2024-03-06T12:32:03.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:03.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:03.910532+0000) 2024-03-06T12:32:04.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:04.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:04.910748+0000) 2024-03-06T12:32:05.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:05.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:05.911045+0000) 2024-03-06T12:32:06.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:06.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:06.911262+0000) 2024-03-06T12:32:07.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:07.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:07.911481+0000) 2024-03-06T12:32:08.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:08.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:08.911803+0000) 2024-03-06T12:32:09.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:09.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:09.912099+0000) 2024-03-06T12:32:10.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:10.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:10.912385+0000) 2024-03-06T12:32:10.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:10.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:11.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:11.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:11.912656+0000) 2024-03-06T12:32:12.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:12.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:12.912876+0000) 2024-03-06T12:32:13.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:13.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:13.913148+0000) 2024-03-06T12:32:14.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:14.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:14.913366+0000) 2024-03-06T12:32:15.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:15.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:15.913614+0000) 2024-03-06T12:32:16.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:16.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:16.913835+0000) 2024-03-06T12:32:17.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:17.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:17.914102+0000) 2024-03-06T12:32:18.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:18.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:18.914349+0000) 2024-03-06T12:32:19.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:19.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:19.914589+0000) 2024-03-06T12:32:20.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:20.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:20.914845+0000) 2024-03-06T12:32:21.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:21.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:21.915128+0000) 2024-03-06T12:32:22.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:22.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:22.915319+0000) 2024-03-06T12:32:23.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:23.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:23.915545+0000) 2024-03-06T12:32:24.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:24.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:24.915802+0000) 2024-03-06T12:32:25.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:25.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:25.916072+0000) 2024-03-06T12:32:26.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:26.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:26.916264+0000) 2024-03-06T12:32:27.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:27.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:27.916499+0000) 2024-03-06T12:32:28.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:28.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:28.916749+0000) 2024-03-06T12:32:29.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:29.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:29.916982+0000) 2024-03-06T12:32:30.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:30.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:30.917239+0000) 2024-03-06T12:32:31.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:31.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:31.917459+0000) 2024-03-06T12:32:32.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:32.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:32.917699+0000) 2024-03-06T12:32:33.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:33.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:33.917948+0000) 2024-03-06T12:32:34.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:34.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:34.918170+0000) 2024-03-06T12:32:35.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:35.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:35.918401+0000) 2024-03-06T12:32:36.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:36.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:36.918631+0000) 2024-03-06T12:32:37.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:37.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:37.918834+0000) 2024-03-06T12:32:38.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:38.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:38.919110+0000) 2024-03-06T12:32:39.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:39.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:39.919330+0000) 2024-03-06T12:32:40.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:40.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:40.919539+0000) 2024-03-06T12:32:41.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:41.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:32:41.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:41.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:41.919771+0000) 2024-03-06T12:32:42.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:42.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:42.919969+0000) 2024-03-06T12:32:43.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:43.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:43.920155+0000) 2024-03-06T12:32:44.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:44.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:44.920355+0000) 2024-03-06T12:32:45.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:45.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:45.920551+0000) 2024-03-06T12:32:46.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:46.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:46.920739+0000) 2024-03-06T12:32:47.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:47.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:47.920971+0000) 2024-03-06T12:32:48.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:48.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:48.921176+0000) 2024-03-06T12:32:49.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:49.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:49.921364+0000) 2024-03-06T12:32:50.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:50.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:50.921578+0000) 2024-03-06T12:32:51.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:51.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:51.921852+0000) 2024-03-06T12:32:52.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:52.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:52.922124+0000) 2024-03-06T12:32:53.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:53.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:53.922343+0000) 2024-03-06T12:32:54.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:54.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:54.922631+0000) 2024-03-06T12:32:55.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:55.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:55.922919+0000) 2024-03-06T12:32:56.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:56.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:56.923163+0000) 2024-03-06T12:32:57.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:57.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:57.923380+0000) 2024-03-06T12:32:58.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:58.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:58.923665+0000) 2024-03-06T12:32:59.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:32:59.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:32:59.923867+0000) 2024-03-06T12:33:00.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:00.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:00.924135+0000) 2024-03-06T12:33:01.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:01.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:01.924418+0000) 2024-03-06T12:33:02.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:02.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:02.924669+0000) 2024-03-06T12:33:03.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:03.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:03.924903+0000) 2024-03-06T12:33:04.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:04.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:04.925162+0000) 2024-03-06T12:33:05.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:05.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:05.925448+0000) 2024-03-06T12:33:06.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:06.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:06.925700+0000) 2024-03-06T12:33:07.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:07.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:07.925942+0000) 2024-03-06T12:33:08.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:08.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:08.926170+0000) 2024-03-06T12:33:09.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:09.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:09.926397+0000) 2024-03-06T12:33:10.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:10.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:10.926633+0000) 2024-03-06T12:33:11.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:11.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:11.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:11.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:11.926896+0000) 2024-03-06T12:33:12.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:12.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:12.927159+0000) 2024-03-06T12:33:13.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:13.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:13.927335+0000) 2024-03-06T12:33:14.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:14.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:14.927547+0000) 2024-03-06T12:33:15.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:15.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:15.927745+0000) 2024-03-06T12:33:16.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:16.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:16.927966+0000) 2024-03-06T12:33:17.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:17.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:17.928161+0000) 2024-03-06T12:33:18.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:18.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:18.928382+0000) 2024-03-06T12:33:19.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:19.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:19.928621+0000) 2024-03-06T12:33:20.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:20.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:20.928851+0000) 2024-03-06T12:33:21.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:21.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:21.929070+0000) 2024-03-06T12:33:22.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:22.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:22.929371+0000) 2024-03-06T12:33:23.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:23.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:23.929645+0000) 2024-03-06T12:33:24.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:24.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:24.929934+0000) 2024-03-06T12:33:25.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:25.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:25.930228+0000) 2024-03-06T12:33:26.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:26.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:26.930509+0000) 2024-03-06T12:33:27.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:27.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:27.930793+0000) 2024-03-06T12:33:28.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:28.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:28.931119+0000) 2024-03-06T12:33:29.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:29.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:29.931394+0000) 2024-03-06T12:33:30.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:30.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:30.931684+0000) 2024-03-06T12:33:31.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:31.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:31.931971+0000) 2024-03-06T12:33:32.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:32.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:32.932307+0000) 2024-03-06T12:33:33.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:33.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:33.932586+0000) 2024-03-06T12:33:34.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:34.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:34.932852+0000) 2024-03-06T12:33:35.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:35.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:35.933151+0000) 2024-03-06T12:33:36.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:36.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:36.933433+0000) 2024-03-06T12:33:37.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:37.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:37.933720+0000) 2024-03-06T12:33:38.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:38.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:38.934051+0000) 2024-03-06T12:33:39.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:39.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:39.934341+0000) 2024-03-06T12:33:40.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:40.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:40.934585+0000) 2024-03-06T12:33:41.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:41.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:33:41.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:41.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:41.934796+0000) 2024-03-06T12:33:42.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:42.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:42.935072+0000) 2024-03-06T12:33:43.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:43.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:43.935329+0000) 2024-03-06T12:33:44.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:44.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:44.935597+0000) 2024-03-06T12:33:45.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:45.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:45.935882+0000) 2024-03-06T12:33:46.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:46.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:46.936159+0000) 2024-03-06T12:33:47.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:47.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:47.936433+0000) 2024-03-06T12:33:48.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:48.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:48.936715+0000) 2024-03-06T12:33:49.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:49.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:49.936983+0000) 2024-03-06T12:33:50.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:50.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:50.937275+0000) 2024-03-06T12:33:51.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:51.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:51.937532+0000) 2024-03-06T12:33:52.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:52.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:52.937783+0000) 2024-03-06T12:33:53.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:53.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:53.938033+0000) 2024-03-06T12:33:54.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:54.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:54.938197+0000) 2024-03-06T12:33:55.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:55.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:55.938438+0000) 2024-03-06T12:33:56.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:56.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:56.938655+0000) 2024-03-06T12:33:57.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:57.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:57.938850+0000) 2024-03-06T12:33:58.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:58.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:58.939132+0000) 2024-03-06T12:33:59.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:33:59.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:33:59.939387+0000) 2024-03-06T12:34:00.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:00.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:00.939657+0000) 2024-03-06T12:34:01.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:01.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:01.939905+0000) 2024-03-06T12:34:02.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:02.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:02.940150+0000) 2024-03-06T12:34:03.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:03.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:03.940320+0000) 2024-03-06T12:34:04.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:04.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:04.940552+0000) 2024-03-06T12:34:05.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:05.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:05.940789+0000) 2024-03-06T12:34:06.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:06.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:06.941041+0000) 2024-03-06T12:34:07.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:07.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:07.941268+0000) 2024-03-06T12:34:08.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:08.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:08.941489+0000) 2024-03-06T12:34:09.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:09.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:09.941698+0000) 2024-03-06T12:34:10.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:10.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:10.941926+0000) 2024-03-06T12:34:11.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:11.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:11.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:11.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:11.942169+0000) 2024-03-06T12:34:12.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:12.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:12.942445+0000) 2024-03-06T12:34:13.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:13.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:13.942731+0000) 2024-03-06T12:34:14.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:14.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:14.943107+0000) 2024-03-06T12:34:15.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:15.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:15.943365+0000) 2024-03-06T12:34:16.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:16.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:16.943615+0000) 2024-03-06T12:34:17.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:17.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:17.943893+0000) 2024-03-06T12:34:18.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:18.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:18.944157+0000) 2024-03-06T12:34:19.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:19.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:19.944455+0000) 2024-03-06T12:34:20.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:20.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:20.944688+0000) 2024-03-06T12:34:21.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:21.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:21.944905+0000) 2024-03-06T12:34:22.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:22.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:22.945143+0000) 2024-03-06T12:34:23.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:23.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:23.945369+0000) 2024-03-06T12:34:24.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:24.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:24.945596+0000) 2024-03-06T12:34:25.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:25.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:25.945800+0000) 2024-03-06T12:34:26.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:26.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:26.945991+0000) 2024-03-06T12:34:27.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:27.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:27.946291+0000) 2024-03-06T12:34:28.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:28.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:28.946528+0000) 2024-03-06T12:34:29.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:29.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:29.946792+0000) 2024-03-06T12:34:30.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:30.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:30.947052+0000) 2024-03-06T12:34:31.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:31.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:31.947251+0000) 2024-03-06T12:34:32.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:32.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:32.947467+0000) 2024-03-06T12:34:33.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:33.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:33.947693+0000) 2024-03-06T12:34:34.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:34.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:34.947915+0000) 2024-03-06T12:34:35.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:35.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:35.948216+0000) 2024-03-06T12:34:36.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:36.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:36.948474+0000) 2024-03-06T12:34:37.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:37.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:37.948711+0000) 2024-03-06T12:34:38.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:38.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:38.949044+0000) 2024-03-06T12:34:39.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:39.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:39.949306+0000) 2024-03-06T12:34:40.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:40.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:40.949536+0000) 2024-03-06T12:34:41.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:41.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:34:41.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:41.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:41.949761+0000) 2024-03-06T12:34:42.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:42.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:42.949983+0000) 2024-03-06T12:34:43.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:43.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:43.950291+0000) 2024-03-06T12:34:44.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:44.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:44.950542+0000) 2024-03-06T12:34:45.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:45.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:45.950764+0000) 2024-03-06T12:34:46.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:46.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:46.950991+0000) 2024-03-06T12:34:47.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:47.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:47.951288+0000) 2024-03-06T12:34:48.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:48.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:48.951553+0000) 2024-03-06T12:34:49.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:49.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:49.951829+0000) 2024-03-06T12:34:50.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:50.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:50.952104+0000) 2024-03-06T12:34:51.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:51.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:51.952381+0000) 2024-03-06T12:34:52.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:52.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:52.952644+0000) 2024-03-06T12:34:53.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:53.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:53.952871+0000) 2024-03-06T12:34:54.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:54.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:54.953143+0000) 2024-03-06T12:34:55.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:55.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:55.953334+0000) 2024-03-06T12:34:56.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:56.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:56.953591+0000) 2024-03-06T12:34:57.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:57.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:57.953810+0000) 2024-03-06T12:34:58.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:58.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:58.954099+0000) 2024-03-06T12:34:59.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:34:59.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:34:59.954328+0000) 2024-03-06T12:35:00.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:00.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:00.954526+0000) 2024-03-06T12:35:01.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:01.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:01.954751+0000) 2024-03-06T12:35:02.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:02.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:02.954900+0000) 2024-03-06T12:35:03.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:03.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:03.955129+0000) 2024-03-06T12:35:04.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:04.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:04.955370+0000) 2024-03-06T12:35:05.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:05.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:05.955592+0000) 2024-03-06T12:35:06.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:06.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:06.955814+0000) 2024-03-06T12:35:07.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:07.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:07.956061+0000) 2024-03-06T12:35:08.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:08.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:08.956337+0000) 2024-03-06T12:35:09.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:09.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:09.956610+0000) 2024-03-06T12:35:10.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:10.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:10.956877+0000) 2024-03-06T12:35:11.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:11.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:11.957123+0000) 2024-03-06T12:35:12.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:12.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:12.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:12.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:12.957361+0000) 2024-03-06T12:35:13.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:13.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:13.957579+0000) 2024-03-06T12:35:14.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:14.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:14.957799+0000) 2024-03-06T12:35:15.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:15.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:15.957970+0000) 2024-03-06T12:35:16.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:16.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:16.958194+0000) 2024-03-06T12:35:17.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:17.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:17.958427+0000) 2024-03-06T12:35:18.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:18.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:18.958637+0000) 2024-03-06T12:35:19.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:19.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:19.958912+0000) 2024-03-06T12:35:20.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:20.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:20.959192+0000) 2024-03-06T12:35:21.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:21.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:21.959468+0000) 2024-03-06T12:35:22.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:22.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:22.959722+0000) 2024-03-06T12:35:23.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:23.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:23.959940+0000) 2024-03-06T12:35:24.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:24.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:24.960167+0000) 2024-03-06T12:35:25.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:25.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:25.960392+0000) 2024-03-06T12:35:26.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:26.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:26.960689+0000) 2024-03-06T12:35:27.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:27.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:27.960937+0000) 2024-03-06T12:35:28.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:28.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:28.961184+0000) 2024-03-06T12:35:29.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:29.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:29.961406+0000) 2024-03-06T12:35:30.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:30.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:30.961650+0000) 2024-03-06T12:35:31.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:31.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:31.961884+0000) 2024-03-06T12:35:32.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:32.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:32.962162+0000) 2024-03-06T12:35:33.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:33.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:33.962392+0000) 2024-03-06T12:35:34.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:34.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:34.962610+0000) 2024-03-06T12:35:35.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:35.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:35.962829+0000) 2024-03-06T12:35:36.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:36.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:36.963091+0000) 2024-03-06T12:35:37.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:37.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:37.963306+0000) 2024-03-06T12:35:38.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:38.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:38.963510+0000) 2024-03-06T12:35:39.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:39.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:39.963763+0000) 2024-03-06T12:35:40.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:40.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:40.963984+0000) 2024-03-06T12:35:41.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:41.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:41.964248+0000) 2024-03-06T12:35:42.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:42.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:35:42.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:42.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:42.964491+0000) 2024-03-06T12:35:43.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:43.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:43.964719+0000) 2024-03-06T12:35:44.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:44.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:44.964939+0000) 2024-03-06T12:35:45.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:45.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:45.965072+0000) 2024-03-06T12:35:46.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:46.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:46.965302+0000) 2024-03-06T12:35:47.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:47.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:47.965514+0000) 2024-03-06T12:35:48.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:48.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:48.965720+0000) 2024-03-06T12:35:49.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:49.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:49.965950+0000) 2024-03-06T12:35:50.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:50.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:50.966186+0000) 2024-03-06T12:35:51.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:51.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:51.966430+0000) 2024-03-06T12:35:52.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:52.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:52.966668+0000) 2024-03-06T12:35:53.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:53.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:53.966892+0000) 2024-03-06T12:35:54.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:54.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:54.967123+0000) 2024-03-06T12:35:55.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:55.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:55.967364+0000) 2024-03-06T12:35:56.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:56.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:56.967593+0000) 2024-03-06T12:35:57.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:57.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:57.967824+0000) 2024-03-06T12:35:58.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:58.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:58.968069+0000) 2024-03-06T12:35:59.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:35:59.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:35:59.968321+0000) 2024-03-06T12:36:00.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:00.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:00.968544+0000) 2024-03-06T12:36:01.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:01.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:01.968771+0000) 2024-03-06T12:36:02.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:02.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:02.969051+0000) 2024-03-06T12:36:03.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:03.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:03.969294+0000) 2024-03-06T12:36:04.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:04.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:04.969516+0000) 2024-03-06T12:36:05.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:05.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:05.969742+0000) 2024-03-06T12:36:06.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:06.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:06.969964+0000) 2024-03-06T12:36:07.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:07.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:07.970206+0000) 2024-03-06T12:36:08.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:08.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:08.970434+0000) 2024-03-06T12:36:09.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:09.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:09.970655+0000) 2024-03-06T12:36:10.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:10.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:10.970910+0000) 2024-03-06T12:36:11.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:11.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:11.971159+0000) 2024-03-06T12:36:12.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:12.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:12.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:12.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:12.971406+0000) 2024-03-06T12:36:13.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:13.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:13.971616+0000) 2024-03-06T12:36:14.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:14.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:14.971816+0000) 2024-03-06T12:36:15.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:15.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:15.972093+0000) 2024-03-06T12:36:16.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:16.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:16.972330+0000) 2024-03-06T12:36:17.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:17.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:17.972569+0000) 2024-03-06T12:36:18.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:18.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:18.972757+0000) 2024-03-06T12:36:19.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:19.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:19.973099+0000) 2024-03-06T12:36:20.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:20.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:20.973302+0000) 2024-03-06T12:36:21.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:21.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:21.973553+0000) 2024-03-06T12:36:22.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:22.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:22.973834+0000) 2024-03-06T12:36:23.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:23.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:23.974101+0000) 2024-03-06T12:36:24.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:24.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:24.974327+0000) 2024-03-06T12:36:25.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:25.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:25.974558+0000) 2024-03-06T12:36:26.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:26.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:26.974778+0000) 2024-03-06T12:36:27.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:27.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:27.975040+0000) 2024-03-06T12:36:28.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:28.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:28.975293+0000) 2024-03-06T12:36:29.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:29.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:29.975531+0000) 2024-03-06T12:36:30.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:30.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:30.975757+0000) 2024-03-06T12:36:31.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:31.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:31.975990+0000) 2024-03-06T12:36:32.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:32.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:32.976265+0000) 2024-03-06T12:36:33.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:33.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:33.976494+0000) 2024-03-06T12:36:34.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:34.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:34.976730+0000) 2024-03-06T12:36:35.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:35.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:35.976971+0000) 2024-03-06T12:36:36.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:36.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:36.977164+0000) 2024-03-06T12:36:37.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:37.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:37.977392+0000) 2024-03-06T12:36:38.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:38.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:38.977626+0000) 2024-03-06T12:36:39.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:39.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:39.977823+0000) 2024-03-06T12:36:40.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:40.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:40.978105+0000) 2024-03-06T12:36:41.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:41.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:41.978333+0000) 2024-03-06T12:36:42.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:42.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:36:42.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:42.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:42.978555+0000) 2024-03-06T12:36:43.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:43.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:43.978775+0000) 2024-03-06T12:36:44.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:44.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:44.978999+0000) 2024-03-06T12:36:45.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:45.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:45.979261+0000) 2024-03-06T12:36:46.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:46.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:46.979517+0000) 2024-03-06T12:36:47.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:47.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:47.979761+0000) 2024-03-06T12:36:48.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:48.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:48.979949+0000) 2024-03-06T12:36:49.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:49.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:49.980168+0000) 2024-03-06T12:36:50.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:50.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:50.980394+0000) 2024-03-06T12:36:51.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:51.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:51.980615+0000) 2024-03-06T12:36:52.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:52.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:52.980844+0000) 2024-03-06T12:36:53.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:53.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:53.981123+0000) 2024-03-06T12:36:54.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:54.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:54.981361+0000) 2024-03-06T12:36:55.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:55.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:55.981592+0000) 2024-03-06T12:36:56.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:56.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:56.981729+0000) 2024-03-06T12:36:57.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:57.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:57.981963+0000) 2024-03-06T12:36:58.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:58.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:58.982206+0000) 2024-03-06T12:36:59.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:36:59.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:36:59.982463+0000) 2024-03-06T12:37:00.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:00.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:00.982685+0000) 2024-03-06T12:37:01.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:01.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:01.982844+0000) 2024-03-06T12:37:02.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:02.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:02.983176+0000) 2024-03-06T12:37:03.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:03.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:03.983403+0000) 2024-03-06T12:37:04.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:04.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:04.983649+0000) 2024-03-06T12:37:05.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:05.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:05.983891+0000) 2024-03-06T12:37:06.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:06.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:06.984152+0000) 2024-03-06T12:37:07.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:07.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:07.984383+0000) 2024-03-06T12:37:08.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:08.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:08.984616+0000) 2024-03-06T12:37:09.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:09.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:09.984855+0000) 2024-03-06T12:37:10.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:10.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:10.985145+0000) 2024-03-06T12:37:11.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:11.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:11.985402+0000) 2024-03-06T12:37:12.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:12.888 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:12.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:12.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:12.985598+0000) 2024-03-06T12:37:13.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:13.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:13.985819+0000) 2024-03-06T12:37:14.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:14.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:14.986105+0000) 2024-03-06T12:37:15.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:15.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:15.986320+0000) 2024-03-06T12:37:16.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:16.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:16.986555+0000) 2024-03-06T12:37:17.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:17.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:17.986783+0000) 2024-03-06T12:37:18.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:18.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:18.987041+0000) 2024-03-06T12:37:19.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:19.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:19.987233+0000) 2024-03-06T12:37:20.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:20.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:20.987485+0000) 2024-03-06T12:37:21.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:21.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:21.987740+0000) 2024-03-06T12:37:22.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:22.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:22.988060+0000) 2024-03-06T12:37:23.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:23.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:23.988352+0000) 2024-03-06T12:37:24.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:24.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:24.988620+0000) 2024-03-06T12:37:25.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:25.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:25.988826+0000) 2024-03-06T12:37:26.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:26.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:26.989104+0000) 2024-03-06T12:37:27.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:27.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:27.989301+0000) 2024-03-06T12:37:28.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:28.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:28.989535+0000) 2024-03-06T12:37:29.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:29.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:29.989768+0000) 2024-03-06T12:37:30.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:30.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:30.989975+0000) 2024-03-06T12:37:31.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:31.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:31.990178+0000) 2024-03-06T12:37:32.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:32.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:32.990379+0000) 2024-03-06T12:37:33.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:33.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:33.990577+0000) 2024-03-06T12:37:34.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:34.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:34.990831+0000) 2024-03-06T12:37:35.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:35.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:35.991110+0000) 2024-03-06T12:37:36.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:36.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:36.991343+0000) 2024-03-06T12:37:37.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:37.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:37.991564+0000) 2024-03-06T12:37:38.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:38.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:38.991819+0000) 2024-03-06T12:37:39.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:39.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:39.991955+0000) 2024-03-06T12:37:40.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:40.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:40.992115+0000) 2024-03-06T12:37:41.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:41.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:41.992337+0000) 2024-03-06T12:37:42.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:42.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:42.992576+0000) 2024-03-06T12:37:43.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:43.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:37:43.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:43.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:43.992799+0000) 2024-03-06T12:37:44.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:44.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:44.993064+0000) 2024-03-06T12:37:45.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:45.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:45.993264+0000) 2024-03-06T12:37:46.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:46.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:46.993494+0000) 2024-03-06T12:37:47.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:47.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:47.993744+0000) 2024-03-06T12:37:48.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:48.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:48.993966+0000) 2024-03-06T12:37:49.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:49.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:49.994203+0000) 2024-03-06T12:37:50.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:50.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:50.994458+0000) 2024-03-06T12:37:51.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:51.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:51.994715+0000) 2024-03-06T12:37:52.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:52.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:52.994999+0000) 2024-03-06T12:37:53.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:53.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:53.995253+0000) 2024-03-06T12:37:54.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:54.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:54.995481+0000) 2024-03-06T12:37:55.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:55.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:55.995761+0000) 2024-03-06T12:37:56.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:56.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:56.996000+0000) 2024-03-06T12:37:57.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:57.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:57.996258+0000) 2024-03-06T12:37:58.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:58.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:58.996463+0000) 2024-03-06T12:37:59.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:37:59.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:37:59.996607+0000) 2024-03-06T12:38:00.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:00.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:00.996836+0000) 2024-03-06T12:38:01.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:01.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:01.997142+0000) 2024-03-06T12:38:02.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:02.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:02.997381+0000) 2024-03-06T12:38:03.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:03.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:03.997661+0000) 2024-03-06T12:38:04.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:04.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:04.997938+0000) 2024-03-06T12:38:05.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:05.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:05.998217+0000) 2024-03-06T12:38:06.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:06.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:06.998488+0000) 2024-03-06T12:38:07.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:07.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:07.998690+0000) 2024-03-06T12:38:08.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:08.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:08.999186+0000) 2024-03-06T12:38:09.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:09.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:09.999572+0000) 2024-03-06T12:38:10.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:10.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:10.999841+0000) 2024-03-06T12:38:11.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:11.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:12.000157+0000) 2024-03-06T12:38:12.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:12.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:13.000383+0000) 2024-03-06T12:38:13.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:13.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:13.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:13.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:14.000657+0000) 2024-03-06T12:38:14.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:14.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:15.000848+0000) 2024-03-06T12:38:15.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:16.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:16.001138+0000) 2024-03-06T12:38:16.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:17.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:17.001405+0000) 2024-03-06T12:38:17.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:18.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:18.001678+0000) 2024-03-06T12:38:18.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:19.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:19.001916+0000) 2024-03-06T12:38:19.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:20.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:20.002148+0000) 2024-03-06T12:38:20.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:21.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:21.002423+0000) 2024-03-06T12:38:21.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:22.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:22.002685+0000) 2024-03-06T12:38:22.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:23.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:23.002976+0000) 2024-03-06T12:38:23.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:24.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:24.003183+0000) 2024-03-06T12:38:24.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:25.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:25.003380+0000) 2024-03-06T12:38:25.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:26.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:26.003602+0000) 2024-03-06T12:38:26.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:27.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:27.003812+0000) 2024-03-06T12:38:27.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:28.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:28.004137+0000) 2024-03-06T12:38:28.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:29.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:29.004312+0000) 2024-03-06T12:38:29.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:30.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:30.004526+0000) 2024-03-06T12:38:30.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:31.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:31.004744+0000) 2024-03-06T12:38:31.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:32.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:32.004980+0000) 2024-03-06T12:38:32.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:33.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:33.005198+0000) 2024-03-06T12:38:33.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:34.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:34.005390+0000) 2024-03-06T12:38:34.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:35.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:35.005604+0000) 2024-03-06T12:38:35.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:36.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:36.005838+0000) 2024-03-06T12:38:36.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:37.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:37.006084+0000) 2024-03-06T12:38:37.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:38.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:38.006275+0000) 2024-03-06T12:38:38.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:39.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:39.006497+0000) 2024-03-06T12:38:39.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:40.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:40.006720+0000) 2024-03-06T12:38:40.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:41.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:41.006941+0000) 2024-03-06T12:38:41.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:42.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:42.007148+0000) 2024-03-06T12:38:42.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:43.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:43.007372+0000) 2024-03-06T12:38:43.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:43.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:38:43.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:44.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:44.007586+0000) 2024-03-06T12:38:44.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:45.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:45.007801+0000) 2024-03-06T12:38:45.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:46.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:46.008063+0000) 2024-03-06T12:38:47.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:47.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:47.008255+0000) 2024-03-06T12:38:48.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:48.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:48.008441+0000) 2024-03-06T12:38:49.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:49.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:49.008712+0000) 2024-03-06T12:38:50.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:50.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:50.008932+0000) 2024-03-06T12:38:51.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:51.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:51.009159+0000) 2024-03-06T12:38:52.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:52.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:52.009405+0000) 2024-03-06T12:38:53.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:53.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:53.009636+0000) 2024-03-06T12:38:54.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:54.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:54.009859+0000) 2024-03-06T12:38:55.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:55.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:55.010121+0000) 2024-03-06T12:38:56.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:56.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:56.010341+0000) 2024-03-06T12:38:57.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:57.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:57.010564+0000) 2024-03-06T12:38:58.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:58.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:58.010752+0000) 2024-03-06T12:38:59.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:38:59.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:38:59.010969+0000) 2024-03-06T12:39:00.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:00.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:00.011221+0000) 2024-03-06T12:39:01.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:01.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:01.011432+0000) 2024-03-06T12:39:02.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:02.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:02.011665+0000) 2024-03-06T12:39:03.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:03.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:03.011917+0000) 2024-03-06T12:39:04.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:04.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:04.012145+0000) 2024-03-06T12:39:05.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:05.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:05.012366+0000) 2024-03-06T12:39:06.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:06.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:06.012598+0000) 2024-03-06T12:39:07.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:07.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:07.012789+0000) 2024-03-06T12:39:08.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:08.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:08.012992+0000) 2024-03-06T12:39:09.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:09.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:09.013180+0000) 2024-03-06T12:39:10.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:10.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:10.013405+0000) 2024-03-06T12:39:11.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:11.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:11.013610+0000) 2024-03-06T12:39:12.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:12.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:12.013852+0000) 2024-03-06T12:39:13.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:13.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:13.014103+0000) 2024-03-06T12:39:13.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:13.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:14.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:14.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:14.014302+0000) 2024-03-06T12:39:15.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:15.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:15.014518+0000) 2024-03-06T12:39:16.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:16.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:16.014773+0000) 2024-03-06T12:39:17.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:17.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:17.014998+0000) 2024-03-06T12:39:18.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:18.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:18.015267+0000) 2024-03-06T12:39:19.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:19.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:19.015491+0000) 2024-03-06T12:39:20.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:20.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:20.015711+0000) 2024-03-06T12:39:21.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:21.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:21.015934+0000) 2024-03-06T12:39:22.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:22.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:22.016164+0000) 2024-03-06T12:39:23.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:23.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:23.016449+0000) 2024-03-06T12:39:24.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:24.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:24.016694+0000) 2024-03-06T12:39:25.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:25.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:25.016939+0000) 2024-03-06T12:39:26.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:26.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:26.017186+0000) 2024-03-06T12:39:27.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:27.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:27.017423+0000) 2024-03-06T12:39:28.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:28.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:28.017646+0000) 2024-03-06T12:39:29.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:29.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:29.017866+0000) 2024-03-06T12:39:30.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:30.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:30.018120+0000) 2024-03-06T12:39:31.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:31.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:31.018356+0000) 2024-03-06T12:39:32.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:32.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:32.018614+0000) 2024-03-06T12:39:33.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:33.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:33.018878+0000) 2024-03-06T12:39:34.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:34.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:34.019129+0000) 2024-03-06T12:39:35.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:35.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:35.019364+0000) 2024-03-06T12:39:36.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:36.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:36.019585+0000) 2024-03-06T12:39:37.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:37.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:37.019875+0000) 2024-03-06T12:39:38.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:38.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:38.020122+0000) 2024-03-06T12:39:39.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:39.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:39.020361+0000) 2024-03-06T12:39:40.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:40.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:40.020556+0000) 2024-03-06T12:39:41.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:41.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:41.020779+0000) 2024-03-06T12:39:42.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:42.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:42.021078+0000) 2024-03-06T12:39:43.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:43.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:43.021347+0000) 2024-03-06T12:39:43.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:43.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:39:44.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:44.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:44.021541+0000) 2024-03-06T12:39:45.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:45.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:45.021734+0000) 2024-03-06T12:39:46.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:46.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:46.022043+0000) 2024-03-06T12:39:47.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:47.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:47.022241+0000) 2024-03-06T12:39:48.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:48.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:48.022441+0000) 2024-03-06T12:39:49.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:49.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:49.022632+0000) 2024-03-06T12:39:50.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:50.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:50.022816+0000) 2024-03-06T12:39:51.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:51.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:51.023080+0000) 2024-03-06T12:39:52.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:52.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:52.023354+0000) 2024-03-06T12:39:53.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:53.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:53.023657+0000) 2024-03-06T12:39:54.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:54.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:54.023939+0000) 2024-03-06T12:39:55.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:55.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:55.024335+0000) 2024-03-06T12:39:56.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:56.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:56.024611+0000) 2024-03-06T12:39:57.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:57.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:57.024869+0000) 2024-03-06T12:39:58.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:58.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:58.025131+0000) 2024-03-06T12:39:59.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:39:59.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:39:59.025353+0000) 2024-03-06T12:40:00.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:00.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:00.025560+0000) 2024-03-06T12:40:01.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:01.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:01.025782+0000) 2024-03-06T12:40:02.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:02.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:02.026041+0000) 2024-03-06T12:40:03.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:03.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:03.026261+0000) 2024-03-06T12:40:04.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:04.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:04.026495+0000) 2024-03-06T12:40:05.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:05.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:05.026717+0000) 2024-03-06T12:40:06.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:06.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:06.026949+0000) 2024-03-06T12:40:07.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:07.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:07.027180+0000) 2024-03-06T12:40:08.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:08.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:08.027412+0000) 2024-03-06T12:40:09.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:09.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:09.027620+0000) 2024-03-06T12:40:10.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:10.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:10.027806+0000) 2024-03-06T12:40:11.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:11.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:11.028054+0000) 2024-03-06T12:40:12.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:12.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:12.028292+0000) 2024-03-06T12:40:13.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:13.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:13.028520+0000) 2024-03-06T12:40:13.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:14.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:14.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:14.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:14.028717+0000) 2024-03-06T12:40:15.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:15.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:15.028949+0000) 2024-03-06T12:40:16.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:16.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:16.029164+0000) 2024-03-06T12:40:17.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:17.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:17.029356+0000) 2024-03-06T12:40:18.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:18.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:18.029584+0000) 2024-03-06T12:40:19.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:19.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:19.029803+0000) 2024-03-06T12:40:20.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:20.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:20.030095+0000) 2024-03-06T12:40:21.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:21.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:21.030287+0000) 2024-03-06T12:40:22.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:22.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:22.030508+0000) 2024-03-06T12:40:23.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:23.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:23.030738+0000) 2024-03-06T12:40:24.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:24.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:24.030963+0000) 2024-03-06T12:40:25.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:25.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:25.031234+0000) 2024-03-06T12:40:26.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:26.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:26.031460+0000) 2024-03-06T12:40:27.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:27.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:27.031689+0000) 2024-03-06T12:40:28.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:28.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:28.031917+0000) 2024-03-06T12:40:29.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:29.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:29.032145+0000) 2024-03-06T12:40:30.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:30.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:30.032368+0000) 2024-03-06T12:40:31.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:31.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:31.032585+0000) 2024-03-06T12:40:32.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:32.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:32.032807+0000) 2024-03-06T12:40:33.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:33.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:33.033065+0000) 2024-03-06T12:40:34.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:34.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:34.033295+0000) 2024-03-06T12:40:35.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:35.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:35.033473+0000) 2024-03-06T12:40:36.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:36.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:36.033706+0000) 2024-03-06T12:40:37.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:37.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:37.033929+0000) 2024-03-06T12:40:38.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:38.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:38.034168+0000) 2024-03-06T12:40:39.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:39.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:39.034331+0000) 2024-03-06T12:40:40.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:40.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:40.034536+0000) 2024-03-06T12:40:41.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:41.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:41.034757+0000) 2024-03-06T12:40:42.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:42.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:42.034989+0000) 2024-03-06T12:40:43.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:43.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:43.035257+0000) 2024-03-06T12:40:44.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:44.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:44.035453+0000) 2024-03-06T12:40:44.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:44.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:40:45.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:45.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:45.035684+0000) 2024-03-06T12:40:46.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:46.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:46.035903+0000) 2024-03-06T12:40:47.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:47.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:47.036131+0000) 2024-03-06T12:40:48.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:48.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:48.036358+0000) 2024-03-06T12:40:49.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:49.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:49.036575+0000) 2024-03-06T12:40:50.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:50.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:50.036796+0000) 2024-03-06T12:40:51.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:51.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:51.037053+0000) 2024-03-06T12:40:52.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:52.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:52.037289+0000) 2024-03-06T12:40:53.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:53.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:53.037516+0000) 2024-03-06T12:40:54.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:54.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:54.037739+0000) 2024-03-06T12:40:55.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:55.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:55.037958+0000) 2024-03-06T12:40:56.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:56.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:56.038166+0000) 2024-03-06T12:40:57.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:57.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:57.038351+0000) 2024-03-06T12:40:58.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:58.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:58.038543+0000) 2024-03-06T12:40:59.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:40:59.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:40:59.038780+0000) 2024-03-06T12:41:00.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:00.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:00.039057+0000) 2024-03-06T12:41:01.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:01.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:01.039292+0000) 2024-03-06T12:41:02.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:02.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:02.039523+0000) 2024-03-06T12:41:03.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:03.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:03.039780+0000) 2024-03-06T12:41:04.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:04.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:04.040063+0000) 2024-03-06T12:41:05.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:05.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:05.040237+0000) 2024-03-06T12:41:06.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:06.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:06.040471+0000) 2024-03-06T12:41:07.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:07.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:07.040673+0000) 2024-03-06T12:41:08.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:08.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:08.040912+0000) 2024-03-06T12:41:09.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:09.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:09.041135+0000) 2024-03-06T12:41:10.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:10.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:10.041328+0000) 2024-03-06T12:41:11.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:11.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:11.041547+0000) 2024-03-06T12:41:12.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:12.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:12.041786+0000) 2024-03-06T12:41:13.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:13.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:13.042040+0000) 2024-03-06T12:41:14.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:14.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:14.042250+0000) 2024-03-06T12:41:14.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:14.430 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:15.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:15.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:15.042489+0000) 2024-03-06T12:41:16.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:16.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:16.042734+0000) 2024-03-06T12:41:17.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:17.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:17.042972+0000) 2024-03-06T12:41:18.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:18.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:18.043213+0000) 2024-03-06T12:41:19.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:19.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:19.043436+0000) 2024-03-06T12:41:20.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:20.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:20.043622+0000) 2024-03-06T12:41:21.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:21.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:21.043860+0000) 2024-03-06T12:41:22.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:22.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:22.044154+0000) 2024-03-06T12:41:23.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:23.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:23.044365+0000) 2024-03-06T12:41:24.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:24.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:24.044585+0000) 2024-03-06T12:41:25.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:25.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:25.044789+0000) 2024-03-06T12:41:26.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:26.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:26.044978+0000) 2024-03-06T12:41:27.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:27.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:27.045229+0000) 2024-03-06T12:41:28.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:28.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:28.045474+0000) 2024-03-06T12:41:29.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:29.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:29.045713+0000) 2024-03-06T12:41:30.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:30.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:30.045910+0000) 2024-03-06T12:41:31.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:31.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:31.046160+0000) 2024-03-06T12:41:32.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:32.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:32.046396+0000) 2024-03-06T12:41:33.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:33.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:33.046626+0000) 2024-03-06T12:41:34.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:34.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:34.046850+0000) 2024-03-06T12:41:35.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:35.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:35.047113+0000) 2024-03-06T12:41:36.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:36.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:36.047334+0000) 2024-03-06T12:41:37.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:37.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:37.047553+0000) 2024-03-06T12:41:38.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:38.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:38.047780+0000) 2024-03-06T12:41:39.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:39.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:39.048035+0000) 2024-03-06T12:41:40.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:40.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:40.048236+0000) 2024-03-06T12:41:41.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:41.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:41.048461+0000) 2024-03-06T12:41:42.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:42.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:42.048690+0000) 2024-03-06T12:41:43.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:43.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:43.048918+0000) 2024-03-06T12:41:44.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:44.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:44.049136+0000) 2024-03-06T12:41:44.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:44.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:41:45.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:45.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:45.049373+0000) 2024-03-06T12:41:46.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:46.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:46.049616+0000) 2024-03-06T12:41:47.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:47.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:47.049846+0000) 2024-03-06T12:41:48.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:48.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:48.050126+0000) 2024-03-06T12:41:49.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:49.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:49.050347+0000) 2024-03-06T12:41:50.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:50.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:50.050571+0000) 2024-03-06T12:41:51.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:51.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:51.050822+0000) 2024-03-06T12:41:52.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:52.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:52.051099+0000) 2024-03-06T12:41:53.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:53.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:53.051266+0000) 2024-03-06T12:41:54.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:54.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:54.051444+0000) 2024-03-06T12:41:55.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:55.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:55.051620+0000) 2024-03-06T12:41:56.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:56.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:56.051830+0000) 2024-03-06T12:41:57.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:57.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:57.052091+0000) 2024-03-06T12:41:58.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:58.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:58.052287+0000) 2024-03-06T12:41:59.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:41:59.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:41:59.052505+0000) 2024-03-06T12:42:00.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:00.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:00.052720+0000) 2024-03-06T12:42:01.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:01.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:01.052936+0000) 2024-03-06T12:42:02.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:02.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:02.053162+0000) 2024-03-06T12:42:03.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:03.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:03.053392+0000) 2024-03-06T12:42:04.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:04.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:04.053625+0000) 2024-03-06T12:42:05.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:05.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:05.053847+0000) 2024-03-06T12:42:06.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:06.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:06.054119+0000) 2024-03-06T12:42:07.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:07.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:07.054383+0000) 2024-03-06T12:42:08.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:08.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:08.054623+0000) 2024-03-06T12:42:09.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:09.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:09.054841+0000) 2024-03-06T12:42:10.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:10.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:10.055081+0000) 2024-03-06T12:42:11.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:11.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:11.055300+0000) 2024-03-06T12:42:12.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:12.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:12.055535+0000) 2024-03-06T12:42:13.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:13.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:13.055716+0000) 2024-03-06T12:42:14.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:14.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:14.055967+0000) 2024-03-06T12:42:14.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:14.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:15.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:15.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:15.056233+0000) 2024-03-06T12:42:16.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:16.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:16.056483+0000) 2024-03-06T12:42:17.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:17.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:17.056717+0000) 2024-03-06T12:42:18.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:18.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:18.056968+0000) 2024-03-06T12:42:19.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:19.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:19.057226+0000) 2024-03-06T12:42:20.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:20.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:20.057479+0000) 2024-03-06T12:42:21.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:21.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:21.057714+0000) 2024-03-06T12:42:22.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:22.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:22.057955+0000) 2024-03-06T12:42:23.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:23.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:23.058170+0000) 2024-03-06T12:42:24.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:24.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:24.058389+0000) 2024-03-06T12:42:25.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:25.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:25.058632+0000) 2024-03-06T12:42:26.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:26.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:26.058856+0000) 2024-03-06T12:42:27.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:27.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:27.059140+0000) 2024-03-06T12:42:28.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:28.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:28.059391+0000) 2024-03-06T12:42:29.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:29.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:29.059626+0000) 2024-03-06T12:42:30.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:30.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:30.059841+0000) 2024-03-06T12:42:31.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:31.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:31.060151+0000) 2024-03-06T12:42:32.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:32.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:32.060434+0000) 2024-03-06T12:42:33.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:33.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:33.060717+0000) 2024-03-06T12:42:34.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:34.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:34.061000+0000) 2024-03-06T12:42:35.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:35.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:35.061324+0000) 2024-03-06T12:42:36.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:36.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:36.061527+0000) 2024-03-06T12:42:37.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:37.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:37.061752+0000) 2024-03-06T12:42:38.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:38.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:38.061965+0000) 2024-03-06T12:42:39.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:39.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:39.062157+0000) 2024-03-06T12:42:40.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:40.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:40.062398+0000) 2024-03-06T12:42:41.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:41.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:41.062633+0000) 2024-03-06T12:42:42.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:42.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:42.062850+0000) 2024-03-06T12:42:43.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:43.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:43.063123+0000) 2024-03-06T12:42:44.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:44.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:44.063325+0000) 2024-03-06T12:42:45.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:45.059 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:42:45.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:45.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:45.063529+0000) 2024-03-06T12:42:46.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:46.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:46.063733+0000) 2024-03-06T12:42:47.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:47.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:47.063965+0000) 2024-03-06T12:42:48.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:48.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:48.064220+0000) 2024-03-06T12:42:49.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:49.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:49.064435+0000) 2024-03-06T12:42:50.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:50.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:50.064656+0000) 2024-03-06T12:42:51.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:51.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:51.064866+0000) 2024-03-06T12:42:52.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:52.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:52.065127+0000) 2024-03-06T12:42:53.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:53.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:53.065352+0000) 2024-03-06T12:42:54.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:54.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:54.065567+0000) 2024-03-06T12:42:55.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:55.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:55.065851+0000) 2024-03-06T12:42:56.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:56.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:56.066122+0000) 2024-03-06T12:42:57.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:57.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:57.066323+0000) 2024-03-06T12:42:58.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:58.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:58.066599+0000) 2024-03-06T12:42:59.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:42:59.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:42:59.066889+0000) 2024-03-06T12:43:00.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:00.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:00.067172+0000) 2024-03-06T12:43:01.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:01.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:01.067397+0000) 2024-03-06T12:43:02.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:02.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:02.067618+0000) 2024-03-06T12:43:03.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:03.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:03.067861+0000) 2024-03-06T12:43:04.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:04.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:04.068116+0000) 2024-03-06T12:43:05.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:05.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:05.068345+0000) 2024-03-06T12:43:06.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:06.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:06.068578+0000) 2024-03-06T12:43:07.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:07.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:07.068787+0000) 2024-03-06T12:43:08.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:08.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:08.068940+0000) 2024-03-06T12:43:09.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:09.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:09.069150+0000) 2024-03-06T12:43:10.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:10.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:10.069410+0000) 2024-03-06T12:43:11.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:11.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:11.069656+0000) 2024-03-06T12:43:12.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:12.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:12.069881+0000) 2024-03-06T12:43:13.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:13.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:13.070118+0000) 2024-03-06T12:43:14.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:14.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:14.070314+0000) 2024-03-06T12:43:15.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:15.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:15.070532+0000) 2024-03-06T12:43:15.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:15.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:16.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:16.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:16.070752+0000) 2024-03-06T12:43:17.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:17.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:17.070981+0000) 2024-03-06T12:43:18.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:18.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:18.071289+0000) 2024-03-06T12:43:19.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:19.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:19.071527+0000) 2024-03-06T12:43:20.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:20.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:20.071743+0000) 2024-03-06T12:43:21.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:21.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:21.071958+0000) 2024-03-06T12:43:22.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:22.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:22.072195+0000) 2024-03-06T12:43:23.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:23.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:23.072456+0000) 2024-03-06T12:43:24.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:24.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:24.072742+0000) 2024-03-06T12:43:25.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:25.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:25.072966+0000) 2024-03-06T12:43:26.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:26.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:26.073245+0000) 2024-03-06T12:43:27.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:27.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:27.073528+0000) 2024-03-06T12:43:28.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:28.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:28.073818+0000) 2024-03-06T12:43:29.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:29.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:29.074126+0000) 2024-03-06T12:43:30.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:30.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:30.074328+0000) 2024-03-06T12:43:31.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:31.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:31.074511+0000) 2024-03-06T12:43:32.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:32.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:32.074779+0000) 2024-03-06T12:43:33.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:33.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:33.075040+0000) 2024-03-06T12:43:34.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:34.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:34.075276+0000) 2024-03-06T12:43:35.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:35.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:35.075527+0000) 2024-03-06T12:43:36.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:36.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:36.075767+0000) 2024-03-06T12:43:37.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:37.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:37.076043+0000) 2024-03-06T12:43:38.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:38.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:38.076336+0000) 2024-03-06T12:43:39.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:39.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:39.076606+0000) 2024-03-06T12:43:40.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:40.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:40.076848+0000) 2024-03-06T12:43:41.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:41.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:41.077128+0000) 2024-03-06T12:43:42.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:42.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:42.077385+0000) 2024-03-06T12:43:43.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:43.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:43.077669+0000) 2024-03-06T12:43:44.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:44.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:44.077942+0000) 2024-03-06T12:43:45.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:45.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:45.078188+0000) 2024-03-06T12:43:45.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:45.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:43:46.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:46.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:46.078449+0000) 2024-03-06T12:43:47.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:47.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:47.078679+0000) 2024-03-06T12:43:48.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:48.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:48.078948+0000) 2024-03-06T12:43:49.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:49.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:49.079172+0000) 2024-03-06T12:43:50.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:50.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:50.079360+0000) 2024-03-06T12:43:51.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:51.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:51.079562+0000) 2024-03-06T12:43:52.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:52.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:52.079767+0000) 2024-03-06T12:43:53.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:53.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:53.079955+0000) 2024-03-06T12:43:54.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:54.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:54.080123+0000) 2024-03-06T12:43:55.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:55.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:55.080308+0000) 2024-03-06T12:43:56.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:56.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:56.080543+0000) 2024-03-06T12:43:57.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:57.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:57.080758+0000) 2024-03-06T12:43:58.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:58.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:58.081036+0000) 2024-03-06T12:43:59.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:43:59.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:43:59.081239+0000) 2024-03-06T12:44:00.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:00.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:00.081480+0000) 2024-03-06T12:44:01.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:01.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:01.081718+0000) 2024-03-06T12:44:02.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:02.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:02.081914+0000) 2024-03-06T12:44:03.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:03.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:03.082152+0000) 2024-03-06T12:44:04.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:04.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:04.082363+0000) 2024-03-06T12:44:05.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:05.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:05.082595+0000) 2024-03-06T12:44:06.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:06.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:06.082820+0000) 2024-03-06T12:44:07.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:07.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:07.083073+0000) 2024-03-06T12:44:08.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:08.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:08.083271+0000) 2024-03-06T12:44:09.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:09.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:09.083493+0000) 2024-03-06T12:44:10.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:10.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:10.083731+0000) 2024-03-06T12:44:11.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:11.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:11.083992+0000) 2024-03-06T12:44:12.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:12.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:12.084245+0000) 2024-03-06T12:44:13.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:13.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:13.084496+0000) 2024-03-06T12:44:14.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:14.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:14.084750+0000) 2024-03-06T12:44:15.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:15.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:15.084997+0000) 2024-03-06T12:44:15.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:15.659 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:16.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:16.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:16.085297+0000) 2024-03-06T12:44:17.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:17.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:17.085507+0000) 2024-03-06T12:44:18.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:18.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:18.085765+0000) 2024-03-06T12:44:19.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:19.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:19.086033+0000) 2024-03-06T12:44:20.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:20.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:20.086271+0000) 2024-03-06T12:44:21.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:21.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:21.086493+0000) 2024-03-06T12:44:22.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:22.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:22.086720+0000) 2024-03-06T12:44:23.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:23.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:23.086940+0000) 2024-03-06T12:44:24.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:24.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:24.087184+0000) 2024-03-06T12:44:25.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:25.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:25.087417+0000) 2024-03-06T12:44:26.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:26.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:26.087659+0000) 2024-03-06T12:44:27.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:27.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:27.087884+0000) 2024-03-06T12:44:28.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:28.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:28.088172+0000) 2024-03-06T12:44:29.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:29.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:29.088396+0000) 2024-03-06T12:44:30.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:30.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:30.088640+0000) 2024-03-06T12:44:31.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:31.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:31.088888+0000) 2024-03-06T12:44:32.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:32.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:32.089115+0000) 2024-03-06T12:44:33.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:33.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:33.089312+0000) 2024-03-06T12:44:34.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:34.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:34.089594+0000) 2024-03-06T12:44:35.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:35.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:35.089859+0000) 2024-03-06T12:44:36.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:36.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:36.090228+0000) 2024-03-06T12:44:37.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:37.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:37.090458+0000) 2024-03-06T12:44:38.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:38.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:38.090714+0000) 2024-03-06T12:44:39.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:39.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:39.090935+0000) 2024-03-06T12:44:40.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:40.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:40.091164+0000) 2024-03-06T12:44:41.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:41.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:41.091396+0000) 2024-03-06T12:44:42.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:42.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:42.091643+0000) 2024-03-06T12:44:43.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:43.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:43.091892+0000) 2024-03-06T12:44:44.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:44.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:44.092152+0000) 2024-03-06T12:44:45.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:45.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:45.092342+0000) 2024-03-06T12:44:45.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:45.894 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:44:46.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:46.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:46.092584+0000) 2024-03-06T12:44:47.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:47.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:47.092805+0000) 2024-03-06T12:44:48.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:48.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:48.093119+0000) 2024-03-06T12:44:49.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:49.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:49.093312+0000) 2024-03-06T12:44:50.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:50.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:50.093554+0000) 2024-03-06T12:44:51.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:51.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:51.093778+0000) 2024-03-06T12:44:52.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:52.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:52.094086+0000) 2024-03-06T12:44:53.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:53.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:53.094378+0000) 2024-03-06T12:44:54.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:54.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:54.094568+0000) 2024-03-06T12:44:55.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:55.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:55.094791+0000) 2024-03-06T12:44:56.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:56.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:56.095072+0000) 2024-03-06T12:44:57.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:57.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:57.095289+0000) 2024-03-06T12:44:58.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:58.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:58.095527+0000) 2024-03-06T12:44:59.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:44:59.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:44:59.095771+0000) 2024-03-06T12:45:00.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:00.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:00.096062+0000) 2024-03-06T12:45:01.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:01.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:01.096305+0000) 2024-03-06T12:45:02.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:02.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:02.096491+0000) 2024-03-06T12:45:03.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:03.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:03.096730+0000) 2024-03-06T12:45:04.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:04.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:04.096962+0000) 2024-03-06T12:45:05.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:05.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:05.097115+0000) 2024-03-06T12:45:06.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:06.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:06.097353+0000) 2024-03-06T12:45:07.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:07.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:07.097572+0000) 2024-03-06T12:45:08.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:08.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:08.097819+0000) 2024-03-06T12:45:09.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:09.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:09.098102+0000) 2024-03-06T12:45:10.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:10.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:10.098339+0000) 2024-03-06T12:45:11.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:11.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:11.098586+0000) 2024-03-06T12:45:12.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:12.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:12.098811+0000) 2024-03-06T12:45:13.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:13.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:13.099092+0000) 2024-03-06T12:45:14.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:14.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:14.099322+0000) 2024-03-06T12:45:15.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:15.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:15.099508+0000) 2024-03-06T12:45:16.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:16.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:16.099697+0000) 2024-03-06T12:45:16.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:16.109 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:17.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:17.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:17.099921+0000) 2024-03-06T12:45:18.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:18.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:18.100161+0000) 2024-03-06T12:45:19.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:19.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:19.100339+0000) 2024-03-06T12:45:20.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:20.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:20.100566+0000) 2024-03-06T12:45:21.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:21.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:21.100785+0000) 2024-03-06T12:45:22.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:22.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:22.101043+0000) 2024-03-06T12:45:23.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:23.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:23.101261+0000) 2024-03-06T12:45:24.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:24.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:24.101494+0000) 2024-03-06T12:45:25.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:25.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:25.101683+0000) 2024-03-06T12:45:26.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:26.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:26.101913+0000) 2024-03-06T12:45:27.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:27.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:27.102139+0000) 2024-03-06T12:45:28.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:28.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:28.102389+0000) 2024-03-06T12:45:29.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:29.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:29.102596+0000) 2024-03-06T12:45:30.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:30.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:30.102816+0000) 2024-03-06T12:45:31.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:31.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:31.103039+0000) 2024-03-06T12:45:32.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:32.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:32.103252+0000) 2024-03-06T12:45:33.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:33.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:33.103488+0000) 2024-03-06T12:45:34.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:34.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:34.103718+0000) 2024-03-06T12:45:35.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:35.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:35.103907+0000) 2024-03-06T12:45:36.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:36.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:36.104144+0000) 2024-03-06T12:45:37.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:37.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:37.104397+0000) 2024-03-06T12:45:38.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:38.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:38.104647+0000) 2024-03-06T12:45:39.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:39.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:39.104894+0000) 2024-03-06T12:45:40.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:40.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:40.105140+0000) 2024-03-06T12:45:41.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:41.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:41.105373+0000) 2024-03-06T12:45:42.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:42.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:42.105630+0000) 2024-03-06T12:45:43.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:43.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:43.105858+0000) 2024-03-06T12:45:44.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:44.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:44.106119+0000) 2024-03-06T12:45:45.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:45.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:45.106347+0000) 2024-03-06T12:45:46.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:46.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:46.106582+0000) 2024-03-06T12:45:46.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:46.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:45:47.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:47.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:47.106800+0000) 2024-03-06T12:45:48.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:48.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:48.107052+0000) 2024-03-06T12:45:49.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:49.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:49.107288+0000) 2024-03-06T12:45:50.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:50.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:50.107520+0000) 2024-03-06T12:45:51.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:51.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:51.107777+0000) 2024-03-06T12:45:52.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:52.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:52.107972+0000) 2024-03-06T12:45:53.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:53.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:53.108209+0000) 2024-03-06T12:45:54.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:54.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:54.108411+0000) 2024-03-06T12:45:55.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:55.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:55.108593+0000) 2024-03-06T12:45:56.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:56.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:56.108796+0000) 2024-03-06T12:45:57.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:57.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:57.109049+0000) 2024-03-06T12:45:58.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:58.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:58.109290+0000) 2024-03-06T12:45:59.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:45:59.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:45:59.109536+0000) 2024-03-06T12:46:00.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:00.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:00.109755+0000) 2024-03-06T12:46:01.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:01.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:01.110022+0000) 2024-03-06T12:46:02.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:02.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:02.110219+0000) 2024-03-06T12:46:03.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:03.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:03.110426+0000) 2024-03-06T12:46:04.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:04.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:04.110642+0000) 2024-03-06T12:46:05.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:05.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:05.110835+0000) 2024-03-06T12:46:06.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:06.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:06.111055+0000) 2024-03-06T12:46:07.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:07.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:07.111279+0000) 2024-03-06T12:46:08.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:08.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:08.111521+0000) 2024-03-06T12:46:09.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:09.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:09.111756+0000) 2024-03-06T12:46:10.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:10.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:10.112041+0000) 2024-03-06T12:46:11.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:11.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:11.112268+0000) 2024-03-06T12:46:12.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:12.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:12.112467+0000) 2024-03-06T12:46:13.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:13.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:13.112692+0000) 2024-03-06T12:46:14.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:14.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:14.112965+0000) 2024-03-06T12:46:15.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:15.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:15.113263+0000) 2024-03-06T12:46:16.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:16.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:16.113503+0000) 2024-03-06T12:46:16.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:16.498 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:17.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:17.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:17.113792+0000) 2024-03-06T12:46:18.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:18.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:18.113993+0000) 2024-03-06T12:46:19.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:19.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:19.114241+0000) 2024-03-06T12:46:20.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:20.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:20.114467+0000) 2024-03-06T12:46:21.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:21.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:21.114678+0000) 2024-03-06T12:46:22.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:22.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:22.114920+0000) 2024-03-06T12:46:23.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:23.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:23.115173+0000) 2024-03-06T12:46:24.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:24.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:24.115420+0000) 2024-03-06T12:46:25.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:25.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:25.115650+0000) 2024-03-06T12:46:26.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:26.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:26.115931+0000) 2024-03-06T12:46:27.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:27.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:27.116156+0000) 2024-03-06T12:46:28.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:28.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:28.116355+0000) 2024-03-06T12:46:29.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:29.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:29.116570+0000) 2024-03-06T12:46:30.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:30.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:30.116790+0000) 2024-03-06T12:46:31.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:31.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:31.117039+0000) 2024-03-06T12:46:32.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:32.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:32.117273+0000) 2024-03-06T12:46:33.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:33.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:33.117499+0000) 2024-03-06T12:46:34.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:34.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:34.117725+0000) 2024-03-06T12:46:35.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:35.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:35.117945+0000) 2024-03-06T12:46:36.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:36.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:36.118256+0000) 2024-03-06T12:46:37.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:37.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:37.118452+0000) 2024-03-06T12:46:38.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:38.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:38.118679+0000) 2024-03-06T12:46:39.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:39.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:39.118867+0000) 2024-03-06T12:46:40.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:40.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:40.119146+0000) 2024-03-06T12:46:41.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:41.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:41.119379+0000) 2024-03-06T12:46:42.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:42.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:42.119627+0000) 2024-03-06T12:46:43.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:43.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:43.119876+0000) 2024-03-06T12:46:44.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:44.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:44.120144+0000) 2024-03-06T12:46:45.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:45.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:45.120355+0000) 2024-03-06T12:46:46.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:46.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:46.120597+0000) 2024-03-06T12:46:46.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:46.758 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:46:47.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:47.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:47.120787+0000) 2024-03-06T12:46:48.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:48.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:48.121063+0000) 2024-03-06T12:46:49.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:49.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:49.121277+0000) 2024-03-06T12:46:50.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:50.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:50.121525+0000) 2024-03-06T12:46:51.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:51.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:51.121758+0000) 2024-03-06T12:46:52.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:52.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:52.122067+0000) 2024-03-06T12:46:53.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:53.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:53.122303+0000) 2024-03-06T12:46:54.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:54.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:54.122536+0000) 2024-03-06T12:46:55.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:55.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:55.122771+0000) 2024-03-06T12:46:56.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:56.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:56.122994+0000) 2024-03-06T12:46:57.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:57.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:57.123233+0000) 2024-03-06T12:46:58.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:58.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:58.123468+0000) 2024-03-06T12:46:59.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:46:59.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:46:59.123647+0000) 2024-03-06T12:47:00.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:00.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:00.123850+0000) 2024-03-06T12:47:01.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:01.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:01.124130+0000) 2024-03-06T12:47:02.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:02.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:02.124379+0000) 2024-03-06T12:47:03.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:03.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:03.124608+0000) 2024-03-06T12:47:04.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:04.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:04.124829+0000) 2024-03-06T12:47:05.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:05.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:05.125166+0000) 2024-03-06T12:47:06.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:06.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:06.125365+0000) 2024-03-06T12:47:07.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:07.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:07.125625+0000) 2024-03-06T12:47:08.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:08.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:08.125909+0000) 2024-03-06T12:47:09.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:09.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:09.126189+0000) 2024-03-06T12:47:10.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:10.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:10.126421+0000) 2024-03-06T12:47:11.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:11.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:11.126686+0000) 2024-03-06T12:47:12.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:12.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:12.126925+0000) 2024-03-06T12:47:13.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:13.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:13.127164+0000) 2024-03-06T12:47:14.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:14.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:14.127397+0000) 2024-03-06T12:47:15.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:15.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:15.127638+0000) 2024-03-06T12:47:16.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:16.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:16.127870+0000) 2024-03-06T12:47:16.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:16.914 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:17.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:17.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:17.128122+0000) 2024-03-06T12:47:18.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:18.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:18.128335+0000) 2024-03-06T12:47:19.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:19.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:19.128529+0000) 2024-03-06T12:47:20.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:20.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:20.128728+0000) 2024-03-06T12:47:21.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:21.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:21.128959+0000) 2024-03-06T12:47:22.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:22.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:22.129185+0000) 2024-03-06T12:47:23.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:23.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:23.129414+0000) 2024-03-06T12:47:24.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:24.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:24.129688+0000) 2024-03-06T12:47:25.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:25.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:25.129938+0000) 2024-03-06T12:47:26.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:26.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:26.130172+0000) 2024-03-06T12:47:27.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:27.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:27.130365+0000) 2024-03-06T12:47:28.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:28.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:28.130610+0000) 2024-03-06T12:47:29.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:29.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:29.130890+0000) 2024-03-06T12:47:30.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:30.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:30.131166+0000) 2024-03-06T12:47:31.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:31.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:31.131436+0000) 2024-03-06T12:47:32.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:32.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:32.131688+0000) 2024-03-06T12:47:33.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:33.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:33.131945+0000) 2024-03-06T12:47:34.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:34.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:34.132162+0000) 2024-03-06T12:47:35.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:35.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:35.132352+0000) 2024-03-06T12:47:36.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:36.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:36.132538+0000) 2024-03-06T12:47:37.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:37.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:37.132778+0000) 2024-03-06T12:47:38.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:38.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:38.132976+0000) 2024-03-06T12:47:39.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:39.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:39.133218+0000) 2024-03-06T12:47:40.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:40.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:40.133450+0000) 2024-03-06T12:47:41.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:41.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:41.133665+0000) 2024-03-06T12:47:42.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:42.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:42.133884+0000) 2024-03-06T12:47:43.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:43.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:43.134140+0000) 2024-03-06T12:47:44.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:44.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:44.134339+0000) 2024-03-06T12:47:45.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:45.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:45.134554+0000) 2024-03-06T12:47:46.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:46.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:46.134813+0000) 2024-03-06T12:47:47.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:47.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:47.135065+0000) 2024-03-06T12:47:47.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:47.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:47:48.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:48.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:48.135386+0000) 2024-03-06T12:47:49.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:49.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:49.135664+0000) 2024-03-06T12:47:50.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:50.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:50.135935+0000) 2024-03-06T12:47:51.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:51.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:51.136190+0000) 2024-03-06T12:47:52.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:52.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:52.136474+0000) 2024-03-06T12:47:53.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:53.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:53.136811+0000) 2024-03-06T12:47:54.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:54.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:54.137136+0000) 2024-03-06T12:47:55.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:55.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:55.137404+0000) 2024-03-06T12:47:56.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:56.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:56.137692+0000) 2024-03-06T12:47:57.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:57.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:57.137970+0000) 2024-03-06T12:47:58.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:58.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:58.138203+0000) 2024-03-06T12:47:59.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:47:59.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:47:59.138448+0000) 2024-03-06T12:48:00.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:00.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:00.138702+0000) 2024-03-06T12:48:01.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:01.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:01.138944+0000) 2024-03-06T12:48:02.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:02.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:02.139152+0000) 2024-03-06T12:48:03.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:03.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:03.139345+0000) 2024-03-06T12:48:04.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:04.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:04.139543+0000) 2024-03-06T12:48:05.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:05.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:05.139748+0000) 2024-03-06T12:48:06.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:06.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:06.139937+0000) 2024-03-06T12:48:07.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:07.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:07.140160+0000) 2024-03-06T12:48:08.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:08.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:08.140357+0000) 2024-03-06T12:48:09.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:09.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:09.140578+0000) 2024-03-06T12:48:10.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:10.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:10.140771+0000) 2024-03-06T12:48:11.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:11.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:11.140970+0000) 2024-03-06T12:48:12.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:12.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:12.141260+0000) 2024-03-06T12:48:13.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:13.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:13.141513+0000) 2024-03-06T12:48:14.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:14.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:14.141750+0000) 2024-03-06T12:48:15.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:15.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:15.141982+0000) 2024-03-06T12:48:16.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:16.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:16.142268+0000) 2024-03-06T12:48:17.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:17.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:17.142510+0000) 2024-03-06T12:48:17.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:17.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:18.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:18.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:18.142755+0000) 2024-03-06T12:48:19.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:19.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:19.142990+0000) 2024-03-06T12:48:20.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:20.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:20.143280+0000) 2024-03-06T12:48:21.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:21.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:21.143516+0000) 2024-03-06T12:48:22.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:22.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:22.143718+0000) 2024-03-06T12:48:23.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:23.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:23.143940+0000) 2024-03-06T12:48:24.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:24.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:24.144172+0000) 2024-03-06T12:48:25.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:25.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:25.144361+0000) 2024-03-06T12:48:26.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:26.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:26.144581+0000) 2024-03-06T12:48:27.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:27.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:27.144801+0000) 2024-03-06T12:48:28.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:28.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:28.145102+0000) 2024-03-06T12:48:29.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:29.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:29.145297+0000) 2024-03-06T12:48:30.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:30.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:30.145520+0000) 2024-03-06T12:48:31.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:31.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:31.145736+0000) 2024-03-06T12:48:32.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:32.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:32.145957+0000) 2024-03-06T12:48:33.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:33.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:33.146160+0000) 2024-03-06T12:48:34.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:34.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:34.146398+0000) 2024-03-06T12:48:35.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:35.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:35.146573+0000) 2024-03-06T12:48:36.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:36.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:36.146826+0000) 2024-03-06T12:48:37.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:37.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:37.147141+0000) 2024-03-06T12:48:38.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:38.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:38.147427+0000) 2024-03-06T12:48:39.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:39.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:39.147693+0000) 2024-03-06T12:48:40.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:40.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:40.147973+0000) 2024-03-06T12:48:41.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:41.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:41.148290+0000) 2024-03-06T12:48:42.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:42.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:42.148519+0000) 2024-03-06T12:48:43.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:43.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:43.148811+0000) 2024-03-06T12:48:44.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:44.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:44.149123+0000) 2024-03-06T12:48:45.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:45.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:45.149408+0000) 2024-03-06T12:48:46.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:46.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:46.149646+0000) 2024-03-06T12:48:47.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:47.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:47.149968+0000) 2024-03-06T12:48:47.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:47.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:48:48.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:48.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:48.150257+0000) 2024-03-06T12:48:49.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:49.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:49.150474+0000) 2024-03-06T12:48:50.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:50.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:50.150707+0000) 2024-03-06T12:48:51.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:51.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:51.150938+0000) 2024-03-06T12:48:52.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:52.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:52.151175+0000) 2024-03-06T12:48:53.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:53.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:53.151406+0000) 2024-03-06T12:48:54.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:54.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:54.151647+0000) 2024-03-06T12:48:55.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:55.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:55.151903+0000) 2024-03-06T12:48:56.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:56.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:56.152188+0000) 2024-03-06T12:48:57.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:57.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:57.152456+0000) 2024-03-06T12:48:58.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:58.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:58.152732+0000) 2024-03-06T12:48:59.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:48:59.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:48:59.152997+0000) 2024-03-06T12:49:00.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:00.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:00.153280+0000) 2024-03-06T12:49:01.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:01.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:01.153484+0000) 2024-03-06T12:49:02.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:02.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:02.153711+0000) 2024-03-06T12:49:03.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:03.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:03.153984+0000) 2024-03-06T12:49:04.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:04.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:04.154329+0000) 2024-03-06T12:49:05.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:05.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:05.154598+0000) 2024-03-06T12:49:06.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:06.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:06.154859+0000) 2024-03-06T12:49:07.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:07.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:07.155179+0000) 2024-03-06T12:49:08.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:08.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:08.155429+0000) 2024-03-06T12:49:09.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:09.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:09.155660+0000) 2024-03-06T12:49:10.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:10.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:10.155891+0000) 2024-03-06T12:49:11.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:11.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:11.156121+0000) 2024-03-06T12:49:12.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:12.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:12.156363+0000) 2024-03-06T12:49:13.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:13.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:13.156587+0000) 2024-03-06T12:49:14.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:14.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:14.156838+0000) 2024-03-06T12:49:15.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:15.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:15.157098+0000) 2024-03-06T12:49:16.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:16.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:16.157374+0000) 2024-03-06T12:49:17.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:17.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:17.157654+0000) 2024-03-06T12:49:17.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:17.715 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:18.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:18.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:18.157939+0000) 2024-03-06T12:49:19.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:19.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:19.158154+0000) 2024-03-06T12:49:20.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:20.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:20.158382+0000) 2024-03-06T12:49:21.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:21.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:21.158599+0000) 2024-03-06T12:49:22.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:22.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:22.158820+0000) 2024-03-06T12:49:23.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:23.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:23.159112+0000) 2024-03-06T12:49:24.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:24.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:24.159353+0000) 2024-03-06T12:49:25.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:25.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:25.159590+0000) 2024-03-06T12:49:26.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:26.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:26.159814+0000) 2024-03-06T12:49:27.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:27.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:27.160084+0000) 2024-03-06T12:49:28.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:28.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:28.160310+0000) 2024-03-06T12:49:29.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:29.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:29.160537+0000) 2024-03-06T12:49:30.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:30.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:30.160771+0000) 2024-03-06T12:49:31.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:31.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:31.161055+0000) 2024-03-06T12:49:32.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:32.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:32.161321+0000) 2024-03-06T12:49:33.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:33.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:33.161554+0000) 2024-03-06T12:49:34.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:34.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:34.161764+0000) 2024-03-06T12:49:35.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:35.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:35.161979+0000) 2024-03-06T12:49:36.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:36.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:36.162240+0000) 2024-03-06T12:49:37.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:37.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:37.162439+0000) 2024-03-06T12:49:38.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:38.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:38.162663+0000) 2024-03-06T12:49:39.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:39.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:39.162895+0000) 2024-03-06T12:49:40.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:40.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:40.163266+0000) 2024-03-06T12:49:41.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:41.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:41.163499+0000) 2024-03-06T12:49:42.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:42.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:42.163726+0000) 2024-03-06T12:49:43.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:43.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:43.164051+0000) 2024-03-06T12:49:44.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:44.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:44.164318+0000) 2024-03-06T12:49:45.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:45.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:45.164596+0000) 2024-03-06T12:49:46.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:46.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:46.164882+0000) 2024-03-06T12:49:47.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:47.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:47.165151+0000) 2024-03-06T12:49:47.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:47.860 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:49:48.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:48.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:48.165439+0000) 2024-03-06T12:49:49.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:49.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:49.165709+0000) 2024-03-06T12:49:50.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:50.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:50.165990+0000) 2024-03-06T12:49:51.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:51.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:51.166243+0000) 2024-03-06T12:49:52.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:52.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:52.166427+0000) 2024-03-06T12:49:53.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:53.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:53.166654+0000) 2024-03-06T12:49:54.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:54.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:54.166890+0000) 2024-03-06T12:49:55.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:55.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:55.167115+0000) 2024-03-06T12:49:56.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:56.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:56.167323+0000) 2024-03-06T12:49:57.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:57.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:57.167539+0000) 2024-03-06T12:49:58.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:58.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:58.167770+0000) 2024-03-06T12:49:59.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:49:59.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:49:59.168040+0000) 2024-03-06T12:50:00.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:00.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:00.168244+0000) 2024-03-06T12:50:01.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:01.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:01.168448+0000) 2024-03-06T12:50:02.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:02.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:02.168727+0000) 2024-03-06T12:50:03.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:03.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:03.168973+0000) 2024-03-06T12:50:04.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:04.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:04.169275+0000) 2024-03-06T12:50:05.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:05.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:05.169517+0000) 2024-03-06T12:50:06.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:06.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:06.169788+0000) 2024-03-06T12:50:07.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:07.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:07.170092+0000) 2024-03-06T12:50:08.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:08.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:08.170340+0000) 2024-03-06T12:50:09.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:09.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:09.170578+0000) 2024-03-06T12:50:10.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:10.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:10.170868+0000) 2024-03-06T12:50:11.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:11.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:11.171147+0000) 2024-03-06T12:50:12.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:12.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:12.171332+0000) 2024-03-06T12:50:13.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:13.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:13.171583+0000) 2024-03-06T12:50:14.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:14.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:14.171837+0000) 2024-03-06T12:50:15.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:15.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:15.172106+0000) 2024-03-06T12:50:16.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:16.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:16.172323+0000) 2024-03-06T12:50:17.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:17.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:17.172542+0000) 2024-03-06T12:50:18.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:18.098 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:18.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:18.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:18.172817+0000) 2024-03-06T12:50:19.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:19.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:19.173082+0000) 2024-03-06T12:50:20.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:20.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:20.173305+0000) 2024-03-06T12:50:21.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:21.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:21.173534+0000) 2024-03-06T12:50:22.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:22.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:22.173781+0000) 2024-03-06T12:50:23.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:23.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:23.174131+0000) 2024-03-06T12:50:24.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:24.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:24.174344+0000) 2024-03-06T12:50:25.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:25.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:25.174570+0000) 2024-03-06T12:50:26.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:26.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:26.174789+0000) 2024-03-06T12:50:27.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:27.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:27.175040+0000) 2024-03-06T12:50:28.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:28.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:28.175281+0000) 2024-03-06T12:50:29.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:29.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:29.175464+0000) 2024-03-06T12:50:30.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:30.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:30.175708+0000) 2024-03-06T12:50:31.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:31.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:31.175947+0000) 2024-03-06T12:50:32.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:32.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:32.176171+0000) 2024-03-06T12:50:33.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:33.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:33.176393+0000) 2024-03-06T12:50:34.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:34.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:34.176633+0000) 2024-03-06T12:50:35.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:35.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:35.176837+0000) 2024-03-06T12:50:36.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:36.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:36.177116+0000) 2024-03-06T12:50:37.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:37.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:37.177337+0000) 2024-03-06T12:50:38.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:38.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:38.177585+0000) 2024-03-06T12:50:39.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:39.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:39.177876+0000) 2024-03-06T12:50:40.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:40.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:40.178173+0000) 2024-03-06T12:50:41.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:41.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:41.178443+0000) 2024-03-06T12:50:42.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:42.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:42.178639+0000) 2024-03-06T12:50:43.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:43.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:43.178837+0000) 2024-03-06T12:50:44.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:44.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:44.179128+0000) 2024-03-06T12:50:45.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:45.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:45.179413+0000) 2024-03-06T12:50:46.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:46.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:46.179698+0000) 2024-03-06T12:50:47.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:47.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:47.179979+0000) 2024-03-06T12:50:48.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:48.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:48.180306+0000) 2024-03-06T12:50:48.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:48.301 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:50:49.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:49.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:49.180605+0000) 2024-03-06T12:50:50.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:50.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:50.180880+0000) 2024-03-06T12:50:51.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:51.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:51.181163+0000) 2024-03-06T12:50:52.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:52.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:52.181416+0000) 2024-03-06T12:50:53.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:53.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:53.181653+0000) 2024-03-06T12:50:54.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:54.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:54.181906+0000) 2024-03-06T12:50:55.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:55.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:55.182152+0000) 2024-03-06T12:50:56.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:56.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:56.182420+0000) 2024-03-06T12:50:57.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:57.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:57.182700+0000) 2024-03-06T12:50:58.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:58.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:58.182988+0000) 2024-03-06T12:50:59.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:50:59.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:50:59.183239+0000) 2024-03-06T12:51:00.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:00.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:00.183443+0000) 2024-03-06T12:51:01.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:01.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:01.183658+0000) 2024-03-06T12:51:02.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:02.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:02.183883+0000) 2024-03-06T12:51:03.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:03.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:03.184108+0000) 2024-03-06T12:51:04.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:04.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:04.184379+0000) 2024-03-06T12:51:05.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:05.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:05.184615+0000) 2024-03-06T12:51:06.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:06.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:06.184904+0000) 2024-03-06T12:51:07.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:07.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:07.185135+0000) 2024-03-06T12:51:08.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:08.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:08.185323+0000) 2024-03-06T12:51:09.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:09.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:09.185557+0000) 2024-03-06T12:51:10.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:10.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:10.185788+0000) 2024-03-06T12:51:11.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:11.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:11.186026+0000) 2024-03-06T12:51:12.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:12.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:12.186224+0000) 2024-03-06T12:51:13.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:13.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:13.186449+0000) 2024-03-06T12:51:14.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:14.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:14.186673+0000) 2024-03-06T12:51:15.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:15.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:15.186877+0000) 2024-03-06T12:51:16.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:16.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:16.187230+0000) 2024-03-06T12:51:17.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:17.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:17.187468+0000) 2024-03-06T12:51:18.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:18.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:18.187670+0000) 2024-03-06T12:51:18.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:18.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:19.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:19.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:19.187893+0000) 2024-03-06T12:51:20.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:20.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:20.188123+0000) 2024-03-06T12:51:21.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:21.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:21.188315+0000) 2024-03-06T12:51:22.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:22.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:22.188537+0000) 2024-03-06T12:51:23.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:23.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:23.188732+0000) 2024-03-06T12:51:24.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:24.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:24.188965+0000) 2024-03-06T12:51:25.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:25.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:25.189177+0000) 2024-03-06T12:51:26.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:26.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:26.189406+0000) 2024-03-06T12:51:27.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:27.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:27.189635+0000) 2024-03-06T12:51:28.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:28.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:28.189872+0000) 2024-03-06T12:51:29.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:29.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:29.190123+0000) 2024-03-06T12:51:30.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:30.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:30.190349+0000) 2024-03-06T12:51:31.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:31.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:31.190587+0000) 2024-03-06T12:51:32.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:32.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:32.190825+0000) 2024-03-06T12:51:33.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:33.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:33.191125+0000) 2024-03-06T12:51:34.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:34.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:34.191387+0000) 2024-03-06T12:51:35.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:35.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:35.191627+0000) 2024-03-06T12:51:36.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:36.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:36.191852+0000) 2024-03-06T12:51:37.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:37.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:37.192141+0000) 2024-03-06T12:51:38.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:38.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:38.192385+0000) 2024-03-06T12:51:39.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:39.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:39.192648+0000) 2024-03-06T12:51:40.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:40.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:40.192870+0000) 2024-03-06T12:51:41.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:41.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:41.193121+0000) 2024-03-06T12:51:42.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:42.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:42.193353+0000) 2024-03-06T12:51:43.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:43.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:43.193586+0000) 2024-03-06T12:51:44.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:44.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:44.193767+0000) 2024-03-06T12:51:45.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:45.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:45.193984+0000) 2024-03-06T12:51:46.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:46.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:46.194241+0000) 2024-03-06T12:51:47.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:47.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:47.194481+0000) 2024-03-06T12:51:48.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:48.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:48.194733+0000) 2024-03-06T12:51:48.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:48.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:51:49.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:49.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:49.194915+0000) 2024-03-06T12:51:50.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:50.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:50.195149+0000) 2024-03-06T12:51:51.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:51.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:51.195400+0000) 2024-03-06T12:51:52.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:52.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:52.195616+0000) 2024-03-06T12:51:53.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:53.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:53.195873+0000) 2024-03-06T12:51:54.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:54.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:54.196135+0000) 2024-03-06T12:51:55.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:55.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:55.196374+0000) 2024-03-06T12:51:56.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:56.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:56.196626+0000) 2024-03-06T12:51:57.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:57.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:57.196865+0000) 2024-03-06T12:51:58.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:58.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:58.197129+0000) 2024-03-06T12:51:59.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:51:59.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:51:59.197350+0000) 2024-03-06T12:52:00.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:00.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:00.197536+0000) 2024-03-06T12:52:01.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:01.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:01.197769+0000) 2024-03-06T12:52:02.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:02.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:02.197993+0000) 2024-03-06T12:52:03.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:03.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:03.198264+0000) 2024-03-06T12:52:04.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:04.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:04.198569+0000) 2024-03-06T12:52:05.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:05.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:05.198794+0000) 2024-03-06T12:52:06.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:06.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:06.199053+0000) 2024-03-06T12:52:07.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:07.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:07.199298+0000) 2024-03-06T12:52:08.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:08.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:08.199577+0000) 2024-03-06T12:52:09.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:09.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:09.199845+0000) 2024-03-06T12:52:10.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:10.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:10.200157+0000) 2024-03-06T12:52:11.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:11.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:11.200397+0000) 2024-03-06T12:52:12.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:12.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:12.200603+0000) 2024-03-06T12:52:13.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:13.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:13.200853+0000) 2024-03-06T12:52:14.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:14.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:14.201111+0000) 2024-03-06T12:52:15.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:15.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:15.201350+0000) 2024-03-06T12:52:16.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:16.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:16.201591+0000) 2024-03-06T12:52:17.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:17.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:17.201849+0000) 2024-03-06T12:52:18.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:18.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:18.202123+0000) 2024-03-06T12:52:18.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:18.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:19.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:19.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:19.202354+0000) 2024-03-06T12:52:20.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:20.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:20.202634+0000) 2024-03-06T12:52:21.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:21.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:21.202911+0000) 2024-03-06T12:52:22.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:22.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:22.203157+0000) 2024-03-06T12:52:23.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:23.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:23.203350+0000) 2024-03-06T12:52:24.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:24.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:24.203597+0000) 2024-03-06T12:52:25.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:25.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:25.203839+0000) 2024-03-06T12:52:26.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:26.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:26.204115+0000) 2024-03-06T12:52:27.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:27.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:27.204353+0000) 2024-03-06T12:52:28.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:28.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:28.204600+0000) 2024-03-06T12:52:29.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:29.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:29.204808+0000) 2024-03-06T12:52:30.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:30.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:30.205127+0000) 2024-03-06T12:52:31.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:31.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:31.205369+0000) 2024-03-06T12:52:32.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:32.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:32.205582+0000) 2024-03-06T12:52:33.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:33.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:33.205835+0000) 2024-03-06T12:52:34.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:34.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:34.206121+0000) 2024-03-06T12:52:35.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:35.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:35.206361+0000) 2024-03-06T12:52:36.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:36.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:36.206603+0000) 2024-03-06T12:52:37.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:37.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:37.206810+0000) 2024-03-06T12:52:38.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:38.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:38.207090+0000) 2024-03-06T12:52:39.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:39.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:39.207283+0000) 2024-03-06T12:52:40.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:40.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:40.207523+0000) 2024-03-06T12:52:41.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:41.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:41.207720+0000) 2024-03-06T12:52:42.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:42.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:42.207988+0000) 2024-03-06T12:52:43.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:43.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:43.208330+0000) 2024-03-06T12:52:44.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:44.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:44.208573+0000) 2024-03-06T12:52:45.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:45.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:45.208783+0000) 2024-03-06T12:52:46.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:46.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:46.209090+0000) 2024-03-06T12:52:47.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:47.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:47.209269+0000) 2024-03-06T12:52:48.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:48.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:48.209490+0000) 2024-03-06T12:52:49.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:49.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:52:49.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:49.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:49.209713+0000) 2024-03-06T12:52:50.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:50.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:50.209942+0000) 2024-03-06T12:52:51.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:51.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:51.210164+0000) 2024-03-06T12:52:52.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:52.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:52.210409+0000) 2024-03-06T12:52:53.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:53.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:53.210697+0000) 2024-03-06T12:52:54.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:54.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:54.210975+0000) 2024-03-06T12:52:55.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:55.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:55.211305+0000) 2024-03-06T12:52:56.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:56.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:56.211587+0000) 2024-03-06T12:52:57.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:57.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:57.211825+0000) 2024-03-06T12:52:58.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:58.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:58.212094+0000) 2024-03-06T12:52:59.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:52:59.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:52:59.212273+0000) 2024-03-06T12:53:00.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:00.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:00.212538+0000) 2024-03-06T12:53:01.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:01.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:01.212773+0000) 2024-03-06T12:53:02.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:02.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:02.213053+0000) 2024-03-06T12:53:03.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:03.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:03.213310+0000) 2024-03-06T12:53:04.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:04.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:04.213553+0000) 2024-03-06T12:53:05.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:05.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:05.213743+0000) 2024-03-06T12:53:06.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:06.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:06.213993+0000) 2024-03-06T12:53:07.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:07.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:07.214250+0000) 2024-03-06T12:53:08.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:08.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:08.214491+0000) 2024-03-06T12:53:09.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:09.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:09.214696+0000) 2024-03-06T12:53:10.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:10.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:10.214934+0000) 2024-03-06T12:53:11.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:11.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:11.215166+0000) 2024-03-06T12:53:12.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:12.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:12.215392+0000) 2024-03-06T12:53:13.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:13.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:13.215625+0000) 2024-03-06T12:53:14.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:14.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:14.215853+0000) 2024-03-06T12:53:15.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:15.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:15.216062+0000) 2024-03-06T12:53:16.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:16.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:16.216300+0000) 2024-03-06T12:53:17.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:17.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:17.216515+0000) 2024-03-06T12:53:18.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:18.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:18.216763+0000) 2024-03-06T12:53:19.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:19.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:19.216991+0000) 2024-03-06T12:53:19.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:19.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:20.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:20.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:20.217256+0000) 2024-03-06T12:53:21.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:21.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:21.217447+0000) 2024-03-06T12:53:22.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:22.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:22.217667+0000) 2024-03-06T12:53:23.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:23.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:23.217892+0000) 2024-03-06T12:53:24.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:24.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:24.218136+0000) 2024-03-06T12:53:25.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:25.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:25.218355+0000) 2024-03-06T12:53:26.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:26.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:26.218570+0000) 2024-03-06T12:53:27.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:27.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:27.218774+0000) 2024-03-06T12:53:28.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:28.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:28.218996+0000) 2024-03-06T12:53:29.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:29.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:29.219323+0000) 2024-03-06T12:53:30.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:30.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:30.219583+0000) 2024-03-06T12:53:31.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:31.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:31.219819+0000) 2024-03-06T12:53:32.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:32.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:32.220088+0000) 2024-03-06T12:53:33.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:33.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:33.220322+0000) 2024-03-06T12:53:34.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:34.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:34.220543+0000) 2024-03-06T12:53:35.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:35.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:35.220751+0000) 2024-03-06T12:53:36.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:36.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:36.221080+0000) 2024-03-06T12:53:37.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:37.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:37.221361+0000) 2024-03-06T12:53:38.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:38.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:38.221569+0000) 2024-03-06T12:53:39.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:39.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:39.221797+0000) 2024-03-06T12:53:40.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:40.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:40.222051+0000) 2024-03-06T12:53:41.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:41.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:41.222270+0000) 2024-03-06T12:53:42.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:42.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:42.222536+0000) 2024-03-06T12:53:43.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:43.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:43.222789+0000) 2024-03-06T12:53:44.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:44.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:44.223055+0000) 2024-03-06T12:53:45.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:45.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:45.223313+0000) 2024-03-06T12:53:46.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:46.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:46.223554+0000) 2024-03-06T12:53:47.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:47.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:47.223774+0000) 2024-03-06T12:53:48.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:48.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:48.223980+0000) 2024-03-06T12:53:49.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:49.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:49.224235+0000) 2024-03-06T12:53:49.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:49.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:53:50.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:50.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:50.224457+0000) 2024-03-06T12:53:51.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:51.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:51.224784+0000) 2024-03-06T12:53:52.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:52.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:52.225103+0000) 2024-03-06T12:53:53.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:53.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:53.225396+0000) 2024-03-06T12:53:54.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:54.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:54.225675+0000) 2024-03-06T12:53:55.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:55.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:55.225907+0000) 2024-03-06T12:53:56.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:56.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:56.226199+0000) 2024-03-06T12:53:57.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:57.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:57.226459+0000) 2024-03-06T12:53:58.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:58.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:58.226738+0000) 2024-03-06T12:53:59.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:53:59.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:53:59.226986+0000) 2024-03-06T12:54:00.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:00.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:00.227284+0000) 2024-03-06T12:54:01.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:01.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:01.227542+0000) 2024-03-06T12:54:02.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:02.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:02.227769+0000) 2024-03-06T12:54:03.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:03.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:03.227999+0000) 2024-03-06T12:54:04.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:04.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:04.228294+0000) 2024-03-06T12:54:05.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:05.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:05.228639+0000) 2024-03-06T12:54:06.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:06.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:06.228881+0000) 2024-03-06T12:54:07.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:07.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:07.229131+0000) 2024-03-06T12:54:08.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:08.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:08.229347+0000) 2024-03-06T12:54:09.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:09.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:09.229574+0000) 2024-03-06T12:54:10.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:10.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:10.229806+0000) 2024-03-06T12:54:11.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:11.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:11.230074+0000) 2024-03-06T12:54:12.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:12.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:12.230318+0000) 2024-03-06T12:54:13.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:13.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:13.230564+0000) 2024-03-06T12:54:14.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:14.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:14.230786+0000) 2024-03-06T12:54:15.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:15.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:15.231062+0000) 2024-03-06T12:54:16.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:16.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:16.231317+0000) 2024-03-06T12:54:17.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:17.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:17.231534+0000) 2024-03-06T12:54:18.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:18.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:18.231762+0000) 2024-03-06T12:54:19.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:19.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:19.231987+0000) 2024-03-06T12:54:19.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:19.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:20.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:20.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:20.232294+0000) 2024-03-06T12:54:21.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:21.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:21.232516+0000) 2024-03-06T12:54:22.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:22.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:22.232740+0000) 2024-03-06T12:54:23.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:23.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:23.232950+0000) 2024-03-06T12:54:24.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:24.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:24.233166+0000) 2024-03-06T12:54:25.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:25.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:25.233388+0000) 2024-03-06T12:54:26.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:26.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:26.233578+0000) 2024-03-06T12:54:27.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:27.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:27.233761+0000) 2024-03-06T12:54:28.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:28.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:28.233996+0000) 2024-03-06T12:54:29.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:29.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:29.234242+0000) 2024-03-06T12:54:30.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:30.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:30.234494+0000) 2024-03-06T12:54:31.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:31.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:31.234714+0000) 2024-03-06T12:54:32.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:32.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:32.234947+0000) 2024-03-06T12:54:33.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:33.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:33.235214+0000) 2024-03-06T12:54:34.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:34.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:34.235438+0000) 2024-03-06T12:54:35.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:35.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:35.235664+0000) 2024-03-06T12:54:36.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:36.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:36.235856+0000) 2024-03-06T12:54:37.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:37.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:37.236079+0000) 2024-03-06T12:54:38.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:38.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:38.236306+0000) 2024-03-06T12:54:39.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:39.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:39.236522+0000) 2024-03-06T12:54:40.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:40.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:40.236754+0000) 2024-03-06T12:54:41.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:41.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:41.236972+0000) 2024-03-06T12:54:42.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:42.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:42.237129+0000) 2024-03-06T12:54:43.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:43.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:43.237351+0000) 2024-03-06T12:54:44.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:44.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:44.237582+0000) 2024-03-06T12:54:45.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:45.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:45.237803+0000) 2024-03-06T12:54:46.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:46.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:46.238064+0000) 2024-03-06T12:54:47.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:47.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:47.238292+0000) 2024-03-06T12:54:48.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:48.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:48.238535+0000) 2024-03-06T12:54:49.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:49.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:49.238777+0000) 2024-03-06T12:54:49.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:49.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:54:50.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:50.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:50.239049+0000) 2024-03-06T12:54:51.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:51.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:51.239209+0000) 2024-03-06T12:54:52.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:52.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:52.239409+0000) 2024-03-06T12:54:53.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:53.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:53.239599+0000) 2024-03-06T12:54:54.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:54.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:54.239844+0000) 2024-03-06T12:54:55.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:55.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:55.240121+0000) 2024-03-06T12:54:56.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:56.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:56.240365+0000) 2024-03-06T12:54:57.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:57.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:57.240605+0000) 2024-03-06T12:54:58.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:58.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:58.240846+0000) 2024-03-06T12:54:59.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:54:59.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:54:59.241120+0000) 2024-03-06T12:55:00.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:00.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:00.241386+0000) 2024-03-06T12:55:01.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:01.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:01.241622+0000) 2024-03-06T12:55:02.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:02.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:02.241857+0000) 2024-03-06T12:55:03.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:03.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:03.242105+0000) 2024-03-06T12:55:04.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:04.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:04.242331+0000) 2024-03-06T12:55:05.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:05.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:05.242560+0000) 2024-03-06T12:55:06.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:06.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:06.242793+0000) 2024-03-06T12:55:07.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:07.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:07.243068+0000) 2024-03-06T12:55:08.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:08.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:08.243323+0000) 2024-03-06T12:55:09.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:09.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:09.243531+0000) 2024-03-06T12:55:10.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:10.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:10.243745+0000) 2024-03-06T12:55:11.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:11.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:11.243968+0000) 2024-03-06T12:55:12.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:12.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:12.244197+0000) 2024-03-06T12:55:13.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:13.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:13.244393+0000) 2024-03-06T12:55:14.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:14.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:14.244581+0000) 2024-03-06T12:55:15.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:15.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:15.244803+0000) 2024-03-06T12:55:16.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:16.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:16.245073+0000) 2024-03-06T12:55:17.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:17.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:17.245308+0000) 2024-03-06T12:55:18.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:18.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:18.245550+0000) 2024-03-06T12:55:19.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:19.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:19.245786+0000) 2024-03-06T12:55:20.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:20.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:20.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:20.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:20.246039+0000) 2024-03-06T12:55:21.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:21.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:21.246283+0000) 2024-03-06T12:55:22.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:22.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:22.246504+0000) 2024-03-06T12:55:23.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:23.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:23.246707+0000) 2024-03-06T12:55:24.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:24.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:24.246936+0000) 2024-03-06T12:55:25.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:25.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:25.247164+0000) 2024-03-06T12:55:26.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:26.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:26.247389+0000) 2024-03-06T12:55:27.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:27.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:27.247583+0000) 2024-03-06T12:55:28.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:28.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:28.247769+0000) 2024-03-06T12:55:29.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:29.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:29.247989+0000) 2024-03-06T12:55:30.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:30.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:30.248239+0000) 2024-03-06T12:55:31.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:31.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:31.248441+0000) 2024-03-06T12:55:32.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:32.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:32.248654+0000) 2024-03-06T12:55:33.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:33.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:33.248881+0000) 2024-03-06T12:55:34.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:34.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:34.249141+0000) 2024-03-06T12:55:35.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:35.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:35.249334+0000) 2024-03-06T12:55:36.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:36.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:36.249525+0000) 2024-03-06T12:55:37.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:37.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:37.249746+0000) 2024-03-06T12:55:38.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:38.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:38.249996+0000) 2024-03-06T12:55:39.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:39.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:39.250262+0000) 2024-03-06T12:55:40.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:40.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:40.250501+0000) 2024-03-06T12:55:41.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:41.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:41.250716+0000) 2024-03-06T12:55:42.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:42.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:42.250933+0000) 2024-03-06T12:55:43.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:43.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:43.251157+0000) 2024-03-06T12:55:44.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:44.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:44.251418+0000) 2024-03-06T12:55:45.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:45.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:45.251653+0000) 2024-03-06T12:55:46.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:46.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:46.251827+0000) 2024-03-06T12:55:47.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:47.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:47.252038+0000) 2024-03-06T12:55:48.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:48.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:48.252281+0000) 2024-03-06T12:55:49.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:49.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:49.252479+0000) 2024-03-06T12:55:50.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:50.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:50.252694+0000) 2024-03-06T12:55:50.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:50.342 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:55:51.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:51.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:51.252916+0000) 2024-03-06T12:55:52.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:52.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:52.253138+0000) 2024-03-06T12:55:53.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:53.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:53.253397+0000) 2024-03-06T12:55:54.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:54.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:54.253680+0000) 2024-03-06T12:55:55.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:55.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:55.253956+0000) 2024-03-06T12:55:56.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:56.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:56.254246+0000) 2024-03-06T12:55:57.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:57.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:57.254524+0000) 2024-03-06T12:55:58.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:58.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:58.254808+0000) 2024-03-06T12:55:59.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:55:59.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:55:59.255116+0000) 2024-03-06T12:56:00.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:00.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:00.255311+0000) 2024-03-06T12:56:01.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:01.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:01.255507+0000) 2024-03-06T12:56:02.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:02.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:02.255775+0000) 2024-03-06T12:56:03.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:03.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:03.256069+0000) 2024-03-06T12:56:04.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:04.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:04.256362+0000) 2024-03-06T12:56:05.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:05.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:05.256657+0000) 2024-03-06T12:56:06.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:06.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:06.256911+0000) 2024-03-06T12:56:07.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:07.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:07.257191+0000) 2024-03-06T12:56:08.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:08.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:08.257474+0000) 2024-03-06T12:56:09.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:09.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:09.257707+0000) 2024-03-06T12:56:10.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:10.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:10.257963+0000) 2024-03-06T12:56:11.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:11.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:11.258209+0000) 2024-03-06T12:56:12.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:12.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:12.258350+0000) 2024-03-06T12:56:13.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:13.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:13.258588+0000) 2024-03-06T12:56:14.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:14.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:14.258786+0000) 2024-03-06T12:56:15.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:15.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:15.259106+0000) 2024-03-06T12:56:16.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:16.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:16.259300+0000) 2024-03-06T12:56:17.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:17.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:17.259500+0000) 2024-03-06T12:56:18.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:18.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:18.259721+0000) 2024-03-06T12:56:19.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:19.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:19.259943+0000) 2024-03-06T12:56:20.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:20.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:20.260171+0000) 2024-03-06T12:56:20.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:20.555 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:21.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:21.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:21.260346+0000) 2024-03-06T12:56:22.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:22.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:22.260571+0000) 2024-03-06T12:56:23.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:23.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:23.260800+0000) 2024-03-06T12:56:24.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:24.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:24.261076+0000) 2024-03-06T12:56:25.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:25.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:25.261323+0000) 2024-03-06T12:56:26.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:26.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:26.261552+0000) 2024-03-06T12:56:27.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:27.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:27.261779+0000) 2024-03-06T12:56:28.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:28.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:28.262043+0000) 2024-03-06T12:56:29.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:29.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:29.262267+0000) 2024-03-06T12:56:30.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:30.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:30.262482+0000) 2024-03-06T12:56:31.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:31.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:31.262702+0000) 2024-03-06T12:56:32.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:32.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:32.262925+0000) 2024-03-06T12:56:33.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:33.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:33.263169+0000) 2024-03-06T12:56:34.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:34.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:34.263389+0000) 2024-03-06T12:56:35.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:35.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:35.263636+0000) 2024-03-06T12:56:36.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:36.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:36.263855+0000) 2024-03-06T12:56:37.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:37.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:37.264099+0000) 2024-03-06T12:56:38.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:38.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:38.264325+0000) 2024-03-06T12:56:39.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:39.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:39.264539+0000) 2024-03-06T12:56:40.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:40.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:40.264757+0000) 2024-03-06T12:56:41.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:41.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:41.264984+0000) 2024-03-06T12:56:42.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:42.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:42.265213+0000) 2024-03-06T12:56:43.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:43.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:43.265413+0000) 2024-03-06T12:56:44.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:44.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:44.265598+0000) 2024-03-06T12:56:45.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:45.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:45.265860+0000) 2024-03-06T12:56:46.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:46.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:46.266149+0000) 2024-03-06T12:56:47.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:47.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:47.266391+0000) 2024-03-06T12:56:48.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:48.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:48.266637+0000) 2024-03-06T12:56:49.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:49.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:49.266875+0000) 2024-03-06T12:56:50.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:50.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:50.267130+0000) 2024-03-06T12:56:50.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:50.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:56:51.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:51.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:51.267371+0000) 2024-03-06T12:56:52.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:52.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:52.267617+0000) 2024-03-06T12:56:53.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:53.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:53.267871+0000) 2024-03-06T12:56:54.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:54.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:54.268115+0000) 2024-03-06T12:56:55.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:55.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:55.268355+0000) 2024-03-06T12:56:56.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:56.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:56.268602+0000) 2024-03-06T12:56:57.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:57.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:57.268845+0000) 2024-03-06T12:56:58.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:58.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:58.269122+0000) 2024-03-06T12:56:59.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:56:59.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:56:59.269361+0000) 2024-03-06T12:57:00.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:00.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:00.269624+0000) 2024-03-06T12:57:01.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:01.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:01.269887+0000) 2024-03-06T12:57:02.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:02.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:02.270151+0000) 2024-03-06T12:57:03.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:03.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:03.270398+0000) 2024-03-06T12:57:04.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:04.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:04.270635+0000) 2024-03-06T12:57:05.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:05.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:05.270851+0000) 2024-03-06T12:57:06.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:06.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:06.271133+0000) 2024-03-06T12:57:07.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:07.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:07.271370+0000) 2024-03-06T12:57:08.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:08.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:08.271631+0000) 2024-03-06T12:57:09.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:09.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:09.271855+0000) 2024-03-06T12:57:10.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:10.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:10.272119+0000) 2024-03-06T12:57:11.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:11.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:11.272347+0000) 2024-03-06T12:57:12.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:12.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:12.272584+0000) 2024-03-06T12:57:13.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:13.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:13.272809+0000) 2024-03-06T12:57:14.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:14.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:14.273092+0000) 2024-03-06T12:57:15.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:15.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:15.273320+0000) 2024-03-06T12:57:16.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:16.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:16.273536+0000) 2024-03-06T12:57:17.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:17.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:17.273768+0000) 2024-03-06T12:57:18.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:18.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:18.274042+0000) 2024-03-06T12:57:19.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:19.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:19.274286+0000) 2024-03-06T12:57:20.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:20.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:20.274524+0000) 2024-03-06T12:57:20.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:20.930 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:21.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:21.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:21.274767+0000) 2024-03-06T12:57:22.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:22.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:22.275075+0000) 2024-03-06T12:57:23.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:23.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:23.275362+0000) 2024-03-06T12:57:24.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:24.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:24.275618+0000) 2024-03-06T12:57:25.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:25.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:25.275855+0000) 2024-03-06T12:57:26.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:26.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:26.276144+0000) 2024-03-06T12:57:27.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:27.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:27.276383+0000) 2024-03-06T12:57:28.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:28.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:28.276631+0000) 2024-03-06T12:57:29.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:29.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:29.276872+0000) 2024-03-06T12:57:30.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:30.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:30.277158+0000) 2024-03-06T12:57:31.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:31.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:31.277413+0000) 2024-03-06T12:57:32.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:32.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:32.277596+0000) 2024-03-06T12:57:33.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:33.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:33.277821+0000) 2024-03-06T12:57:34.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:34.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:34.278064+0000) 2024-03-06T12:57:35.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:35.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:35.278250+0000) 2024-03-06T12:57:36.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:36.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:36.278477+0000) 2024-03-06T12:57:37.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:37.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:37.278710+0000) 2024-03-06T12:57:38.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:38.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:38.278958+0000) 2024-03-06T12:57:39.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:39.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:39.279149+0000) 2024-03-06T12:57:40.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:40.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:40.279347+0000) 2024-03-06T12:57:41.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:41.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:41.279535+0000) 2024-03-06T12:57:42.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:42.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:42.279773+0000) 2024-03-06T12:57:43.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:43.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:43.280041+0000) 2024-03-06T12:57:44.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:44.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:44.280267+0000) 2024-03-06T12:57:45.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:45.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:45.280480+0000) 2024-03-06T12:57:46.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:46.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:46.280700+0000) 2024-03-06T12:57:47.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:47.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:47.280918+0000) 2024-03-06T12:57:48.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:48.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:48.281150+0000) 2024-03-06T12:57:49.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:49.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:49.281371+0000) 2024-03-06T12:57:50.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:50.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:50.281595+0000) 2024-03-06T12:57:51.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:51.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:57:51.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:51.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:51.281823+0000) 2024-03-06T12:57:52.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:52.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:52.282073+0000) 2024-03-06T12:57:53.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:53.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:53.282230+0000) 2024-03-06T12:57:54.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:54.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:54.282434+0000) 2024-03-06T12:57:55.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:55.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:55.282639+0000) 2024-03-06T12:57:56.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:56.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:56.282874+0000) 2024-03-06T12:57:57.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:57.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:57.283115+0000) 2024-03-06T12:57:58.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:58.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:58.283339+0000) 2024-03-06T12:57:59.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:57:59.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:57:59.283511+0000) 2024-03-06T12:58:00.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:00.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:00.283732+0000) 2024-03-06T12:58:01.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:01.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:01.284108+0000) 2024-03-06T12:58:02.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:02.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:02.284318+0000) 2024-03-06T12:58:03.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:03.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:03.284531+0000) 2024-03-06T12:58:04.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:04.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:04.284721+0000) 2024-03-06T12:58:05.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:05.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:05.284951+0000) 2024-03-06T12:58:06.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:06.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:06.285177+0000) 2024-03-06T12:58:07.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:07.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:07.285399+0000) 2024-03-06T12:58:08.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:08.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:08.285602+0000) 2024-03-06T12:58:09.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:09.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:09.285829+0000) 2024-03-06T12:58:10.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:10.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:10.286089+0000) 2024-03-06T12:58:11.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:11.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:11.286281+0000) 2024-03-06T12:58:12.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:12.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:12.286514+0000) 2024-03-06T12:58:13.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:13.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:13.286749+0000) 2024-03-06T12:58:14.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:14.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:14.286981+0000) 2024-03-06T12:58:15.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:15.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:15.287247+0000) 2024-03-06T12:58:16.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:16.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:16.287487+0000) 2024-03-06T12:58:17.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:17.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:17.287710+0000) 2024-03-06T12:58:18.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:18.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:18.287934+0000) 2024-03-06T12:58:19.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:19.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:19.288161+0000) 2024-03-06T12:58:20.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:20.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:20.288392+0000) 2024-03-06T12:58:21.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:21.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:21.288622+0000) 2024-03-06T12:58:21.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:21.340 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:22.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:22.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:22.288843+0000) 2024-03-06T12:58:23.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:23.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:23.289110+0000) 2024-03-06T12:58:24.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:24.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:24.289334+0000) 2024-03-06T12:58:25.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:25.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:25.289547+0000) 2024-03-06T12:58:26.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:26.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:26.289775+0000) 2024-03-06T12:58:27.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:27.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:27.289992+0000) 2024-03-06T12:58:28.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:28.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:28.290299+0000) 2024-03-06T12:58:29.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:29.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:29.290525+0000) 2024-03-06T12:58:30.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:30.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:30.290755+0000) 2024-03-06T12:58:31.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:31.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:31.290972+0000) 2024-03-06T12:58:32.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:32.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:32.291272+0000) 2024-03-06T12:58:33.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:33.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:33.291500+0000) 2024-03-06T12:58:34.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:34.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:34.291725+0000) 2024-03-06T12:58:35.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:35.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:35.291963+0000) 2024-03-06T12:58:36.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:36.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:36.292266+0000) 2024-03-06T12:58:37.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:37.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:37.292443+0000) 2024-03-06T12:58:38.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:38.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:38.292652+0000) 2024-03-06T12:58:39.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:39.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:39.292873+0000) 2024-03-06T12:58:40.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:40.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:40.293131+0000) 2024-03-06T12:58:41.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:41.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:41.293317+0000) 2024-03-06T12:58:42.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:42.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:42.293519+0000) 2024-03-06T12:58:43.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:43.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:43.293747+0000) 2024-03-06T12:58:44.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:44.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:44.293977+0000) 2024-03-06T12:58:45.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:45.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:45.294150+0000) 2024-03-06T12:58:46.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:46.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:46.294350+0000) 2024-03-06T12:58:47.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:47.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:47.294553+0000) 2024-03-06T12:58:48.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:48.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:48.294774+0000) 2024-03-06T12:58:49.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:49.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:49.294993+0000) 2024-03-06T12:58:50.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:50.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:50.295211+0000) 2024-03-06T12:58:51.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:51.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:51.295397+0000) 2024-03-06T12:58:51.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:51.544 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:58:52.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:52.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:52.295615+0000) 2024-03-06T12:58:53.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:53.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:53.295857+0000) 2024-03-06T12:58:54.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:54.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:54.296120+0000) 2024-03-06T12:58:55.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:55.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:55.296332+0000) 2024-03-06T12:58:56.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:56.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:56.296546+0000) 2024-03-06T12:58:57.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:57.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:57.296772+0000) 2024-03-06T12:58:58.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:58.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:58.296987+0000) 2024-03-06T12:58:59.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:58:59.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:58:59.297213+0000) 2024-03-06T12:59:00.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:00.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:00.297411+0000) 2024-03-06T12:59:01.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:01.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:01.297619+0000) 2024-03-06T12:59:02.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:02.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:02.297854+0000) 2024-03-06T12:59:03.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:03.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:03.298172+0000) 2024-03-06T12:59:04.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:04.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:04.298372+0000) 2024-03-06T12:59:05.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:05.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:05.298584+0000) 2024-03-06T12:59:06.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:06.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:06.298781+0000) 2024-03-06T12:59:07.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:07.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:07.298972+0000) 2024-03-06T12:59:08.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:08.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:08.299212+0000) 2024-03-06T12:59:09.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:09.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:09.299443+0000) 2024-03-06T12:59:10.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:10.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:10.299700+0000) 2024-03-06T12:59:11.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:11.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:11.299946+0000) 2024-03-06T12:59:12.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:12.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:12.300172+0000) 2024-03-06T12:59:13.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:13.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:13.300411+0000) 2024-03-06T12:59:14.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:14.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:14.300674+0000) 2024-03-06T12:59:15.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:15.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:15.300951+0000) 2024-03-06T12:59:16.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:16.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:16.301217+0000) 2024-03-06T12:59:17.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:17.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:17.301458+0000) 2024-03-06T12:59:18.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:18.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:18.301724+0000) 2024-03-06T12:59:19.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:19.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:19.301909+0000) 2024-03-06T12:59:20.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:20.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:20.302143+0000) 2024-03-06T12:59:21.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:21.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:21.302321+0000) 2024-03-06T12:59:21.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:21.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:22.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:22.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:22.302589+0000) 2024-03-06T12:59:23.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:23.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:23.302791+0000) 2024-03-06T12:59:24.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:24.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:24.303052+0000) 2024-03-06T12:59:25.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:25.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:25.303248+0000) 2024-03-06T12:59:26.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:26.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:26.303491+0000) 2024-03-06T12:59:27.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:27.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:27.303718+0000) 2024-03-06T12:59:28.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:28.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:28.303936+0000) 2024-03-06T12:59:29.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:29.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:29.304164+0000) 2024-03-06T12:59:30.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:30.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:30.304384+0000) 2024-03-06T12:59:31.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:31.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:31.304606+0000) 2024-03-06T12:59:32.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:32.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:32.304913+0000) 2024-03-06T12:59:33.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:33.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:33.305130+0000) 2024-03-06T12:59:34.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:34.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:34.305342+0000) 2024-03-06T12:59:35.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:35.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:35.305558+0000) 2024-03-06T12:59:36.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:36.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:36.305809+0000) 2024-03-06T12:59:37.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:37.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:37.306074+0000) 2024-03-06T12:59:38.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:38.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:38.306321+0000) 2024-03-06T12:59:39.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:39.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:39.306507+0000) 2024-03-06T12:59:40.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:40.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:40.306735+0000) 2024-03-06T12:59:41.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:41.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:41.306953+0000) 2024-03-06T12:59:42.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:42.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:42.307253+0000) 2024-03-06T12:59:43.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:43.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:43.307465+0000) 2024-03-06T12:59:44.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:44.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:44.307733+0000) 2024-03-06T12:59:45.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:45.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:45.307969+0000) 2024-03-06T12:59:46.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:46.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:46.308201+0000) 2024-03-06T12:59:47.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:47.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:47.308386+0000) 2024-03-06T12:59:48.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:48.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:48.308617+0000) 2024-03-06T12:59:49.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:49.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:49.308839+0000) 2024-03-06T12:59:50.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:50.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:50.309121+0000) 2024-03-06T12:59:51.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:51.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:51.309346+0000) 2024-03-06T12:59:51.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:51.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T12:59:52.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:52.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:52.309555+0000) 2024-03-06T12:59:53.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:53.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:53.309804+0000) 2024-03-06T12:59:54.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:54.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:54.310065+0000) 2024-03-06T12:59:55.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:55.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:55.310287+0000) 2024-03-06T12:59:56.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:56.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:56.310517+0000) 2024-03-06T12:59:57.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:57.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:57.310766+0000) 2024-03-06T12:59:58.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:58.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:58.311074+0000) 2024-03-06T12:59:59.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T12:59:59.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T11:59:59.311324+0000) 2024-03-06T13:00:00.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:00.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:00.311587+0000) 2024-03-06T13:00:01.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:01.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:01.311855+0000) 2024-03-06T13:00:02.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:02.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:02.312158+0000) 2024-03-06T13:00:03.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:03.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:03.312417+0000) 2024-03-06T13:00:04.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:04.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:04.312650+0000) 2024-03-06T13:00:05.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:05.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:05.312887+0000) 2024-03-06T13:00:06.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:06.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:06.313151+0000) 2024-03-06T13:00:07.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:07.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:07.313343+0000) 2024-03-06T13:00:08.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:08.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:08.313588+0000) 2024-03-06T13:00:09.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:09.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:09.313835+0000) 2024-03-06T13:00:10.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:10.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:10.314051+0000) 2024-03-06T13:00:11.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:11.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:11.314280+0000) 2024-03-06T13:00:12.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:12.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:12.314525+0000) 2024-03-06T13:00:13.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:13.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:13.314757+0000) 2024-03-06T13:00:14.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:14.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:14.314978+0000) 2024-03-06T13:00:15.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:15.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:15.315192+0000) 2024-03-06T13:00:16.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:16.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:16.315392+0000) 2024-03-06T13:00:17.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:17.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:17.315598+0000) 2024-03-06T13:00:18.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:18.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:18.315844+0000) 2024-03-06T13:00:19.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:19.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:19.316104+0000) 2024-03-06T13:00:20.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:20.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:20.316346+0000) 2024-03-06T13:00:21.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:21.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:21.316569+0000) 2024-03-06T13:00:21.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:22.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:22.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:22.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:22.316832+0000) 2024-03-06T13:00:23.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:23.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:23.317076+0000) 2024-03-06T13:00:24.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:24.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:24.317305+0000) 2024-03-06T13:00:25.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:25.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:25.317497+0000) 2024-03-06T13:00:26.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:26.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:26.317748+0000) 2024-03-06T13:00:27.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:27.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:27.318042+0000) 2024-03-06T13:00:28.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:28.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:28.318308+0000) 2024-03-06T13:00:29.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:29.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:29.318582+0000) 2024-03-06T13:00:30.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:30.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:30.318859+0000) 2024-03-06T13:00:31.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:31.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:31.319153+0000) 2024-03-06T13:00:32.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:32.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:32.319439+0000) 2024-03-06T13:00:33.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:33.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:33.319729+0000) 2024-03-06T13:00:34.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:34.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:34.320049+0000) 2024-03-06T13:00:35.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:35.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:35.320220+0000) 2024-03-06T13:00:36.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:36.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:36.320445+0000) 2024-03-06T13:00:37.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:37.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:37.320692+0000) 2024-03-06T13:00:38.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:38.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:38.320913+0000) 2024-03-06T13:00:39.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:39.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:39.321129+0000) 2024-03-06T13:00:40.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:40.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:40.321323+0000) 2024-03-06T13:00:41.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:41.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:41.321529+0000) 2024-03-06T13:00:42.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:42.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:42.321742+0000) 2024-03-06T13:00:43.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:43.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:43.321931+0000) 2024-03-06T13:00:44.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:44.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:44.322180+0000) 2024-03-06T13:00:45.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:45.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:45.322415+0000) 2024-03-06T13:00:46.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:46.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:46.322658+0000) 2024-03-06T13:00:47.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:47.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:47.322895+0000) 2024-03-06T13:00:48.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:48.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:48.323146+0000) 2024-03-06T13:00:49.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:49.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:49.323325+0000) 2024-03-06T13:00:50.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:50.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:50.323561+0000) 2024-03-06T13:00:51.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:51.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:51.323828+0000) 2024-03-06T13:00:52.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:52.237 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:00:52.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:52.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:52.324065+0000) 2024-03-06T13:00:53.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:53.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:53.324282+0000) 2024-03-06T13:00:54.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:54.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:54.324454+0000) 2024-03-06T13:00:55.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:55.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:55.324626+0000) 2024-03-06T13:00:56.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:56.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:56.324861+0000) 2024-03-06T13:00:57.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:57.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:57.325122+0000) 2024-03-06T13:00:58.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:58.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:58.325347+0000) 2024-03-06T13:00:59.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:00:59.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:00:59.325531+0000) 2024-03-06T13:01:00.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:00.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:00.325769+0000) 2024-03-06T13:01:01.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:01.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:01.325992+0000) 2024-03-06T13:01:02.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:02.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:02.326263+0000) 2024-03-06T13:01:03.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:03.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:03.326497+0000) 2024-03-06T13:01:04.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:04.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:04.326725+0000) 2024-03-06T13:01:05.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:05.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:05.326904+0000) 2024-03-06T13:01:06.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:06.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:06.327185+0000) 2024-03-06T13:01:07.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:07.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:07.327385+0000) 2024-03-06T13:01:08.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:08.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:08.327616+0000) 2024-03-06T13:01:09.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:09.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:09.327833+0000) 2024-03-06T13:01:10.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:10.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:10.328095+0000) 2024-03-06T13:01:11.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:11.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:11.328288+0000) 2024-03-06T13:01:12.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:12.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:12.328492+0000) 2024-03-06T13:01:13.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:13.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:13.328713+0000) 2024-03-06T13:01:14.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:14.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:14.328933+0000) 2024-03-06T13:01:15.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:15.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:15.329150+0000) 2024-03-06T13:01:16.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:16.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:16.329344+0000) 2024-03-06T13:01:17.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:17.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:17.329564+0000) 2024-03-06T13:01:18.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:18.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:18.329801+0000) 2024-03-06T13:01:19.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:19.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:19.330098+0000) 2024-03-06T13:01:20.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:20.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:20.330284+0000) 2024-03-06T13:01:21.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:21.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:21.330496+0000) 2024-03-06T13:01:22.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:22.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:22.330731+0000) 2024-03-06T13:01:22.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:22.432 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:23.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:23.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:23.330980+0000) 2024-03-06T13:01:24.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:24.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:24.331205+0000) 2024-03-06T13:01:25.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:25.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:25.331390+0000) 2024-03-06T13:01:26.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:26.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:26.331581+0000) 2024-03-06T13:01:27.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:27.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:27.331767+0000) 2024-03-06T13:01:28.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:28.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:28.331994+0000) 2024-03-06T13:01:29.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:29.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:29.332250+0000) 2024-03-06T13:01:30.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:30.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:30.332452+0000) 2024-03-06T13:01:31.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:31.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:31.332672+0000) 2024-03-06T13:01:32.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:32.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:32.332898+0000) 2024-03-06T13:01:33.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:33.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:33.333142+0000) 2024-03-06T13:01:34.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:34.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:34.333362+0000) 2024-03-06T13:01:35.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:35.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:35.333565+0000) 2024-03-06T13:01:36.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:36.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:36.333802+0000) 2024-03-06T13:01:37.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:37.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:37.334049+0000) 2024-03-06T13:01:38.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:38.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:38.334240+0000) 2024-03-06T13:01:39.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:39.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:39.334455+0000) 2024-03-06T13:01:40.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:40.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:40.334705+0000) 2024-03-06T13:01:41.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:41.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:41.334953+0000) 2024-03-06T13:01:42.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:42.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:42.335152+0000) 2024-03-06T13:01:43.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:43.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:43.335345+0000) 2024-03-06T13:01:44.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:44.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:44.335565+0000) 2024-03-06T13:01:45.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:45.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:45.335786+0000) 2024-03-06T13:01:46.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:46.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:46.336059+0000) 2024-03-06T13:01:47.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:47.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:47.336249+0000) 2024-03-06T13:01:48.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:48.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:48.336460+0000) 2024-03-06T13:01:49.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:49.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:49.336665+0000) 2024-03-06T13:01:50.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:50.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:50.336890+0000) 2024-03-06T13:01:51.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:51.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:51.337142+0000) 2024-03-06T13:01:52.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:52.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:52.337366+0000) 2024-03-06T13:01:52.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:52.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:01:53.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:53.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:53.337583+0000) 2024-03-06T13:01:54.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:54.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:54.337812+0000) 2024-03-06T13:01:55.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:55.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:55.338089+0000) 2024-03-06T13:01:56.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:56.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:56.338307+0000) 2024-03-06T13:01:57.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:57.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:57.338513+0000) 2024-03-06T13:01:58.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:58.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:58.338731+0000) 2024-03-06T13:01:59.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:01:59.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:01:59.338975+0000) 2024-03-06T13:02:00.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:00.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:00.339155+0000) 2024-03-06T13:02:01.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:01.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:01.339367+0000) 2024-03-06T13:02:02.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:02.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:02.339572+0000) 2024-03-06T13:02:03.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:03.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:03.339800+0000) 2024-03-06T13:02:04.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:04.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:04.339979+0000) 2024-03-06T13:02:05.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:05.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:05.340280+0000) 2024-03-06T13:02:06.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:06.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:06.340523+0000) 2024-03-06T13:02:07.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:07.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:07.340840+0000) 2024-03-06T13:02:08.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:08.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:08.341070+0000) 2024-03-06T13:02:09.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:09.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:09.341290+0000) 2024-03-06T13:02:10.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:10.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:10.341510+0000) 2024-03-06T13:02:11.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:11.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:11.341733+0000) 2024-03-06T13:02:12.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:12.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:12.341959+0000) 2024-03-06T13:02:13.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:13.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:13.342110+0000) 2024-03-06T13:02:14.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:14.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:14.342302+0000) 2024-03-06T13:02:15.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:15.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:15.342522+0000) 2024-03-06T13:02:16.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:16.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:16.342795+0000) 2024-03-06T13:02:17.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:17.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:17.343115+0000) 2024-03-06T13:02:18.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:18.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:18.343386+0000) 2024-03-06T13:02:19.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:19.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:19.343640+0000) 2024-03-06T13:02:20.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:20.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:20.343853+0000) 2024-03-06T13:02:21.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:21.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:21.344127+0000) 2024-03-06T13:02:22.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:22.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:22.344358+0000) 2024-03-06T13:02:22.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:22.775 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:23.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:23.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:23.344582+0000) 2024-03-06T13:02:24.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:24.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:24.344806+0000) 2024-03-06T13:02:25.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:25.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:25.345044+0000) 2024-03-06T13:02:26.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:26.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:26.345264+0000) 2024-03-06T13:02:27.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:27.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:27.345472+0000) 2024-03-06T13:02:28.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:28.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:28.345711+0000) 2024-03-06T13:02:29.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:29.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:29.345887+0000) 2024-03-06T13:02:30.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:30.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:30.346115+0000) 2024-03-06T13:02:31.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:31.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:31.346339+0000) 2024-03-06T13:02:32.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:32.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:32.346565+0000) 2024-03-06T13:02:33.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:33.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:33.346837+0000) 2024-03-06T13:02:34.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:34.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:34.347138+0000) 2024-03-06T13:02:35.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:35.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:35.347346+0000) 2024-03-06T13:02:36.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:36.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:36.347619+0000) 2024-03-06T13:02:37.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:37.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:37.347859+0000) 2024-03-06T13:02:38.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:38.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:38.348117+0000) 2024-03-06T13:02:39.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:39.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:39.348336+0000) 2024-03-06T13:02:40.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:40.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:40.348589+0000) 2024-03-06T13:02:41.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:41.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:41.348815+0000) 2024-03-06T13:02:42.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:42.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:42.349078+0000) 2024-03-06T13:02:43.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:43.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:43.349278+0000) 2024-03-06T13:02:44.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:44.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:44.349491+0000) 2024-03-06T13:02:45.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:45.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:45.349675+0000) 2024-03-06T13:02:46.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:46.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:46.349896+0000) 2024-03-06T13:02:47.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:47.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:47.350129+0000) 2024-03-06T13:02:48.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:48.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:48.350374+0000) 2024-03-06T13:02:49.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:49.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:49.350623+0000) 2024-03-06T13:02:50.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:50.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:50.350846+0000) 2024-03-06T13:02:51.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:51.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:51.351152+0000) 2024-03-06T13:02:52.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:52.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:52.351367+0000) 2024-03-06T13:02:52.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:52.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:02:53.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:53.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:53.351591+0000) 2024-03-06T13:02:54.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:54.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:54.351818+0000) 2024-03-06T13:02:55.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:55.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:55.352064+0000) 2024-03-06T13:02:56.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:56.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:56.352290+0000) 2024-03-06T13:02:57.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:57.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:57.352508+0000) 2024-03-06T13:02:58.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:58.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:58.352737+0000) 2024-03-06T13:02:59.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:02:59.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:02:59.352955+0000) 2024-03-06T13:03:00.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:00.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:00.353298+0000) 2024-03-06T13:03:01.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:01.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:01.353579+0000) 2024-03-06T13:03:02.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:02.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:02.353780+0000) 2024-03-06T13:03:03.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:03.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:03.353997+0000) 2024-03-06T13:03:04.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:04.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:04.354218+0000) 2024-03-06T13:03:05.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:05.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:05.354429+0000) 2024-03-06T13:03:06.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:06.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:06.354711+0000) 2024-03-06T13:03:07.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:07.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:07.354981+0000) 2024-03-06T13:03:08.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:08.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:08.355314+0000) 2024-03-06T13:03:09.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:09.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:09.355553+0000) 2024-03-06T13:03:10.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:10.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:10.355803+0000) 2024-03-06T13:03:11.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:11.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:11.356097+0000) 2024-03-06T13:03:12.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:12.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:12.356367+0000) 2024-03-06T13:03:13.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:13.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:13.356656+0000) 2024-03-06T13:03:14.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:14.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:14.356930+0000) 2024-03-06T13:03:15.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:15.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:15.357210+0000) 2024-03-06T13:03:16.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:16.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:16.357489+0000) 2024-03-06T13:03:17.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:17.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:17.357771+0000) 2024-03-06T13:03:18.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:18.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:18.358049+0000) 2024-03-06T13:03:19.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:19.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:19.358337+0000) 2024-03-06T13:03:20.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:20.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:20.358610+0000) 2024-03-06T13:03:21.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:21.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:21.358866+0000) 2024-03-06T13:03:22.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:22.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:22.359139+0000) 2024-03-06T13:03:23.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:23.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:23.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:23.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:23.359358+0000) 2024-03-06T13:03:24.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:24.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:24.359583+0000) 2024-03-06T13:03:25.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:25.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:25.359865+0000) 2024-03-06T13:03:26.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:26.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:26.360153+0000) 2024-03-06T13:03:27.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:27.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:27.360421+0000) 2024-03-06T13:03:28.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:28.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:28.360628+0000) 2024-03-06T13:03:29.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:29.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:29.360857+0000) 2024-03-06T13:03:30.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:30.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:30.361159+0000) 2024-03-06T13:03:31.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:31.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:31.361438+0000) 2024-03-06T13:03:32.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:32.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:32.361719+0000) 2024-03-06T13:03:33.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:33.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:33.362044+0000) 2024-03-06T13:03:34.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:34.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:34.362328+0000) 2024-03-06T13:03:35.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:35.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:35.362606+0000) 2024-03-06T13:03:36.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:36.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:36.362821+0000) 2024-03-06T13:03:37.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:37.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:37.363078+0000) 2024-03-06T13:03:38.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:38.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:38.363311+0000) 2024-03-06T13:03:39.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:39.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:39.363552+0000) 2024-03-06T13:03:40.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:40.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:40.363774+0000) 2024-03-06T13:03:41.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:41.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:41.364106+0000) 2024-03-06T13:03:42.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:42.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:42.364355+0000) 2024-03-06T13:03:43.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:43.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:43.364642+0000) 2024-03-06T13:03:44.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:44.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:44.364865+0000) 2024-03-06T13:03:45.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:45.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:45.365112+0000) 2024-03-06T13:03:46.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:46.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:46.365255+0000) 2024-03-06T13:03:47.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:47.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:47.365481+0000) 2024-03-06T13:03:48.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:48.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:48.365703+0000) 2024-03-06T13:03:49.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:49.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:49.365946+0000) 2024-03-06T13:03:50.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:50.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:50.366165+0000) 2024-03-06T13:03:51.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:51.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:51.366384+0000) 2024-03-06T13:03:52.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:52.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:52.366584+0000) 2024-03-06T13:03:53.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:53.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:03:53.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:53.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:53.366834+0000) 2024-03-06T13:03:54.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:54.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:54.367100+0000) 2024-03-06T13:03:55.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:55.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:55.367277+0000) 2024-03-06T13:03:56.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:56.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:56.367489+0000) 2024-03-06T13:03:57.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:57.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:57.367718+0000) 2024-03-06T13:03:58.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:58.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:58.367978+0000) 2024-03-06T13:03:59.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:03:59.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:03:59.368229+0000) 2024-03-06T13:04:00.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:00.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:00.368421+0000) 2024-03-06T13:04:01.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:01.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:01.368623+0000) 2024-03-06T13:04:02.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:02.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:02.368846+0000) 2024-03-06T13:04:03.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:03.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:03.369114+0000) 2024-03-06T13:04:04.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:04.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:04.369355+0000) 2024-03-06T13:04:05.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:05.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:05.369573+0000) 2024-03-06T13:04:06.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:06.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:06.369805+0000) 2024-03-06T13:04:07.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:07.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:07.370066+0000) 2024-03-06T13:04:08.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:08.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:08.370308+0000) 2024-03-06T13:04:09.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:09.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:09.370554+0000) 2024-03-06T13:04:10.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:10.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:10.370765+0000) 2024-03-06T13:04:11.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:11.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:11.371030+0000) 2024-03-06T13:04:12.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:12.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:12.371239+0000) 2024-03-06T13:04:13.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:13.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:13.371457+0000) 2024-03-06T13:04:14.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:14.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:14.371655+0000) 2024-03-06T13:04:15.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:15.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:15.371861+0000) 2024-03-06T13:04:16.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:16.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:16.372060+0000) 2024-03-06T13:04:17.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:17.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:17.372283+0000) 2024-03-06T13:04:18.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:18.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:18.372503+0000) 2024-03-06T13:04:19.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:19.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:19.372680+0000) 2024-03-06T13:04:20.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:20.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:20.372905+0000) 2024-03-06T13:04:21.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:21.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:21.373143+0000) 2024-03-06T13:04:22.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:22.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:22.373331+0000) 2024-03-06T13:04:23.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:23.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:23.373556+0000) 2024-03-06T13:04:23.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:23.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:24.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:24.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:24.373768+0000) 2024-03-06T13:04:25.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:25.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:25.373941+0000) 2024-03-06T13:04:26.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:26.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:26.374160+0000) 2024-03-06T13:04:27.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:27.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:27.374350+0000) 2024-03-06T13:04:28.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:28.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:28.374535+0000) 2024-03-06T13:04:29.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:29.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:29.374763+0000) 2024-03-06T13:04:30.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:30.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:30.374997+0000) 2024-03-06T13:04:31.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:31.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:31.375226+0000) 2024-03-06T13:04:32.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:32.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:32.375426+0000) 2024-03-06T13:04:33.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:33.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:33.375637+0000) 2024-03-06T13:04:34.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:34.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:34.375858+0000) 2024-03-06T13:04:35.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:35.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:35.376121+0000) 2024-03-06T13:04:36.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:36.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:36.376367+0000) 2024-03-06T13:04:37.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:37.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:37.376596+0000) 2024-03-06T13:04:38.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:38.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:38.376820+0000) 2024-03-06T13:04:39.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:39.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:39.377082+0000) 2024-03-06T13:04:40.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:40.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:40.377301+0000) 2024-03-06T13:04:41.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:41.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:41.377526+0000) 2024-03-06T13:04:42.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:42.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:42.377723+0000) 2024-03-06T13:04:43.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:43.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:43.377953+0000) 2024-03-06T13:04:44.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:44.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:44.378172+0000) 2024-03-06T13:04:45.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:45.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:45.378394+0000) 2024-03-06T13:04:46.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:46.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:46.378622+0000) 2024-03-06T13:04:47.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:47.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:47.378849+0000) 2024-03-06T13:04:48.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:48.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:48.379093+0000) 2024-03-06T13:04:49.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:49.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:49.379279+0000) 2024-03-06T13:04:50.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:50.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:50.379496+0000) 2024-03-06T13:04:51.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:51.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:51.379719+0000) 2024-03-06T13:04:52.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:52.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:52.379948+0000) 2024-03-06T13:04:53.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:53.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:53.380179+0000) 2024-03-06T13:04:53.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:53.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:04:54.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:54.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:54.380409+0000) 2024-03-06T13:04:55.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:55.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:55.380627+0000) 2024-03-06T13:04:56.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:56.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:56.380855+0000) 2024-03-06T13:04:57.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:57.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:57.381128+0000) 2024-03-06T13:04:58.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:58.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:58.381341+0000) 2024-03-06T13:04:59.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:04:59.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:04:59.381542+0000) 2024-03-06T13:05:00.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:00.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:00.381772+0000) 2024-03-06T13:05:01.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:01.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:01.381988+0000) 2024-03-06T13:05:02.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:02.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:02.382241+0000) 2024-03-06T13:05:03.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:03.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:03.382463+0000) 2024-03-06T13:05:04.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:04.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:04.382722+0000) 2024-03-06T13:05:05.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:05.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:05.382956+0000) 2024-03-06T13:05:06.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:06.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:06.383155+0000) 2024-03-06T13:05:07.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:07.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:07.383356+0000) 2024-03-06T13:05:08.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:08.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:08.383547+0000) 2024-03-06T13:05:09.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:09.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:09.383777+0000) 2024-03-06T13:05:10.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:10.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:10.384000+0000) 2024-03-06T13:05:11.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:11.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:11.384285+0000) 2024-03-06T13:05:12.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:12.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:12.384510+0000) 2024-03-06T13:05:13.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:13.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:13.384747+0000) 2024-03-06T13:05:14.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:14.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:14.384981+0000) 2024-03-06T13:05:15.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:15.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:15.385245+0000) 2024-03-06T13:05:16.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:16.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:16.385432+0000) 2024-03-06T13:05:17.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:17.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:17.385636+0000) 2024-03-06T13:05:18.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:18.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:18.385860+0000) 2024-03-06T13:05:19.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:19.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:19.386119+0000) 2024-03-06T13:05:20.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:20.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:20.386343+0000) 2024-03-06T13:05:21.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:21.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:21.386689+0000) 2024-03-06T13:05:22.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:22.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:22.386940+0000) 2024-03-06T13:05:23.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:23.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:23.387172+0000) 2024-03-06T13:05:23.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:23.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:24.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:24.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:24.387374+0000) 2024-03-06T13:05:25.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:25.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:25.387558+0000) 2024-03-06T13:05:26.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:26.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:26.387781+0000) 2024-03-06T13:05:27.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:27.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:27.387963+0000) 2024-03-06T13:05:28.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:28.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:28.388192+0000) 2024-03-06T13:05:29.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:29.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:29.388470+0000) 2024-03-06T13:05:30.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:30.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:30.388654+0000) 2024-03-06T13:05:31.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:31.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:31.388847+0000) 2024-03-06T13:05:32.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:32.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:32.389138+0000) 2024-03-06T13:05:33.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:33.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:33.389373+0000) 2024-03-06T13:05:34.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:34.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:34.389598+0000) 2024-03-06T13:05:35.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:35.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:35.389857+0000) 2024-03-06T13:05:36.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:36.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:36.390119+0000) 2024-03-06T13:05:37.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:37.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:37.390359+0000) 2024-03-06T13:05:38.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:38.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:38.390585+0000) 2024-03-06T13:05:39.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:39.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:39.390804+0000) 2024-03-06T13:05:40.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:40.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:40.391051+0000) 2024-03-06T13:05:41.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:41.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:41.391206+0000) 2024-03-06T13:05:42.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:42.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:42.391434+0000) 2024-03-06T13:05:43.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:43.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:43.391684+0000) 2024-03-06T13:05:44.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:44.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:44.391879+0000) 2024-03-06T13:05:45.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:45.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:45.392120+0000) 2024-03-06T13:05:46.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:46.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:46.392317+0000) 2024-03-06T13:05:47.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:47.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:47.392528+0000) 2024-03-06T13:05:48.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:48.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:48.392761+0000) 2024-03-06T13:05:49.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:49.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:49.392965+0000) 2024-03-06T13:05:50.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:50.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:50.393199+0000) 2024-03-06T13:05:51.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:51.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:51.393360+0000) 2024-03-06T13:05:52.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:52.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:52.393588+0000) 2024-03-06T13:05:53.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:53.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:53.393776+0000) 2024-03-06T13:05:54.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:54.169 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:05:54.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:54.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:54.394044+0000) 2024-03-06T13:05:55.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:55.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:55.394241+0000) 2024-03-06T13:05:56.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:56.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:56.394457+0000) 2024-03-06T13:05:57.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:57.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:57.394661+0000) 2024-03-06T13:05:58.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:58.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:58.394882+0000) 2024-03-06T13:05:59.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:05:59.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:05:59.395104+0000) 2024-03-06T13:06:00.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:00.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:00.395298+0000) 2024-03-06T13:06:01.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:01.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:01.395499+0000) 2024-03-06T13:06:02.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:02.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:02.395710+0000) 2024-03-06T13:06:03.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:03.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:03.395942+0000) 2024-03-06T13:06:04.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:04.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:04.396174+0000) 2024-03-06T13:06:05.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:05.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:05.396357+0000) 2024-03-06T13:06:06.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:06.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:06.396547+0000) 2024-03-06T13:06:07.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:07.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:07.396711+0000) 2024-03-06T13:06:08.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:08.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:08.396932+0000) 2024-03-06T13:06:09.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:09.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:09.397154+0000) 2024-03-06T13:06:10.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:10.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:10.397361+0000) 2024-03-06T13:06:11.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:11.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:11.397593+0000) 2024-03-06T13:06:12.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:12.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:12.397846+0000) 2024-03-06T13:06:13.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:13.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:13.398140+0000) 2024-03-06T13:06:14.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:14.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:14.398321+0000) 2024-03-06T13:06:15.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:15.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:15.398519+0000) 2024-03-06T13:06:16.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:16.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:16.398738+0000) 2024-03-06T13:06:17.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:17.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:17.399040+0000) 2024-03-06T13:06:18.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:18.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:18.399234+0000) 2024-03-06T13:06:19.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:19.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:19.399421+0000) 2024-03-06T13:06:20.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:20.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:20.399639+0000) 2024-03-06T13:06:21.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:21.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:21.399885+0000) 2024-03-06T13:06:22.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:22.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:22.400129+0000) 2024-03-06T13:06:23.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:23.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:23.400328+0000) 2024-03-06T13:06:24.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:24.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:24.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:24.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:24.400541+0000) 2024-03-06T13:06:25.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:25.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:25.400777+0000) 2024-03-06T13:06:26.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:26.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:26.401000+0000) 2024-03-06T13:06:27.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:27.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:27.401261+0000) 2024-03-06T13:06:28.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:28.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:28.401456+0000) 2024-03-06T13:06:29.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:29.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:29.401635+0000) 2024-03-06T13:06:30.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:30.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:30.401853+0000) 2024-03-06T13:06:31.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:31.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:31.402094+0000) 2024-03-06T13:06:32.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:32.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:32.402303+0000) 2024-03-06T13:06:33.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:33.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:33.402542+0000) 2024-03-06T13:06:34.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:34.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:34.402787+0000) 2024-03-06T13:06:35.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:35.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:35.403031+0000) 2024-03-06T13:06:36.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:36.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:36.403231+0000) 2024-03-06T13:06:37.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:37.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:37.403430+0000) 2024-03-06T13:06:38.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:38.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:38.403636+0000) 2024-03-06T13:06:39.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:39.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:39.403855+0000) 2024-03-06T13:06:40.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:40.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:40.404148+0000) 2024-03-06T13:06:41.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:41.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:41.404335+0000) 2024-03-06T13:06:42.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:42.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:42.404545+0000) 2024-03-06T13:06:43.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:43.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:43.404782+0000) 2024-03-06T13:06:44.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:44.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:44.405046+0000) 2024-03-06T13:06:45.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:45.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:45.405266+0000) 2024-03-06T13:06:46.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:46.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:46.405485+0000) 2024-03-06T13:06:47.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:47.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:47.405771+0000) 2024-03-06T13:06:48.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:48.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:48.406069+0000) 2024-03-06T13:06:49.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:49.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:49.406290+0000) 2024-03-06T13:06:50.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:50.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:50.406454+0000) 2024-03-06T13:06:51.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:51.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:51.406655+0000) 2024-03-06T13:06:52.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:52.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:52.406901+0000) 2024-03-06T13:06:53.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:53.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:53.407142+0000) 2024-03-06T13:06:54.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:54.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:54.407337+0000) 2024-03-06T13:06:54.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:54.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:06:55.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:55.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:55.407523+0000) 2024-03-06T13:06:56.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:56.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:56.407767+0000) 2024-03-06T13:06:57.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:57.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:57.408060+0000) 2024-03-06T13:06:58.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:58.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:58.408282+0000) 2024-03-06T13:06:59.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:06:59.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:06:59.408520+0000) 2024-03-06T13:07:00.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:00.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:00.408755+0000) 2024-03-06T13:07:01.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:01.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:01.408989+0000) 2024-03-06T13:07:02.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:02.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:02.409265+0000) 2024-03-06T13:07:03.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:03.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:03.409494+0000) 2024-03-06T13:07:04.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:04.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:04.409716+0000) 2024-03-06T13:07:05.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:05.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:05.409986+0000) 2024-03-06T13:07:06.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:06.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:06.410199+0000) 2024-03-06T13:07:07.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:07.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:07.410423+0000) 2024-03-06T13:07:08.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:08.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:08.410674+0000) 2024-03-06T13:07:09.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:09.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:09.410897+0000) 2024-03-06T13:07:10.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:10.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:10.411122+0000) 2024-03-06T13:07:11.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:11.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:11.411341+0000) 2024-03-06T13:07:12.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:12.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:12.411582+0000) 2024-03-06T13:07:13.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:13.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:13.411778+0000) 2024-03-06T13:07:14.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:14.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:14.411963+0000) 2024-03-06T13:07:15.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:15.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:15.412238+0000) 2024-03-06T13:07:16.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:16.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:16.412487+0000) 2024-03-06T13:07:17.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:17.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:17.412749+0000) 2024-03-06T13:07:18.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:18.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:18.412994+0000) 2024-03-06T13:07:19.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:19.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:19.413232+0000) 2024-03-06T13:07:20.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:20.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:20.413472+0000) 2024-03-06T13:07:21.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:21.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:21.413711+0000) 2024-03-06T13:07:22.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:22.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:22.413964+0000) 2024-03-06T13:07:23.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:23.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:23.414183+0000) 2024-03-06T13:07:24.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:24.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:24.414412+0000) 2024-03-06T13:07:24.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:24.800 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:25.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:25.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:25.414602+0000) 2024-03-06T13:07:26.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:26.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:26.414780+0000) 2024-03-06T13:07:27.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:27.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:27.415084+0000) 2024-03-06T13:07:28.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:28.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:28.415297+0000) 2024-03-06T13:07:29.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:29.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:29.415517+0000) 2024-03-06T13:07:30.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:30.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:30.415762+0000) 2024-03-06T13:07:31.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:31.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:31.415998+0000) 2024-03-06T13:07:32.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:32.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:32.416265+0000) 2024-03-06T13:07:33.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:33.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:33.416497+0000) 2024-03-06T13:07:34.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:34.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:34.416744+0000) 2024-03-06T13:07:35.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:35.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:35.416979+0000) 2024-03-06T13:07:36.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:36.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:36.417265+0000) 2024-03-06T13:07:37.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:37.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:37.417508+0000) 2024-03-06T13:07:38.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:38.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:38.417751+0000) 2024-03-06T13:07:39.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:39.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:39.417986+0000) 2024-03-06T13:07:40.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:40.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:40.418234+0000) 2024-03-06T13:07:41.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:41.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:41.418451+0000) 2024-03-06T13:07:42.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:42.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:42.418682+0000) 2024-03-06T13:07:43.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:43.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:43.418935+0000) 2024-03-06T13:07:44.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:44.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:44.419156+0000) 2024-03-06T13:07:45.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:45.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:45.419395+0000) 2024-03-06T13:07:46.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:46.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:46.419613+0000) 2024-03-06T13:07:47.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:47.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:47.419872+0000) 2024-03-06T13:07:48.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:48.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:48.420132+0000) 2024-03-06T13:07:49.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:49.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:49.420291+0000) 2024-03-06T13:07:50.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:50.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:50.420491+0000) 2024-03-06T13:07:51.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:51.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:51.420713+0000) 2024-03-06T13:07:52.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:52.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:52.420950+0000) 2024-03-06T13:07:53.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:53.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:53.421193+0000) 2024-03-06T13:07:54.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:54.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:54.421414+0000) 2024-03-06T13:07:54.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:54.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:07:55.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:55.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:55.421675+0000) 2024-03-06T13:07:56.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:56.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:56.421905+0000) 2024-03-06T13:07:57.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:57.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:57.422139+0000) 2024-03-06T13:07:58.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:58.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:58.422330+0000) 2024-03-06T13:07:59.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:07:59.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:07:59.422545+0000) 2024-03-06T13:08:00.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:00.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:00.422771+0000) 2024-03-06T13:08:01.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:01.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:01.422992+0000) 2024-03-06T13:08:02.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:02.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:02.423198+0000) 2024-03-06T13:08:03.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:03.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:03.423430+0000) 2024-03-06T13:08:04.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:04.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:04.423691+0000) 2024-03-06T13:08:05.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:05.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:05.423876+0000) 2024-03-06T13:08:06.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:06.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:06.424133+0000) 2024-03-06T13:08:07.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:07.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:07.424356+0000) 2024-03-06T13:08:08.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:08.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:08.424585+0000) 2024-03-06T13:08:09.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:09.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:09.424808+0000) 2024-03-06T13:08:10.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:10.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:10.425072+0000) 2024-03-06T13:08:11.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:11.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:11.425300+0000) 2024-03-06T13:08:12.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:12.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:12.425535+0000) 2024-03-06T13:08:13.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:13.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:13.425779+0000) 2024-03-06T13:08:14.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:14.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:14.426057+0000) 2024-03-06T13:08:15.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:15.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:15.426204+0000) 2024-03-06T13:08:16.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:16.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:16.426392+0000) 2024-03-06T13:08:17.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:17.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:17.426599+0000) 2024-03-06T13:08:18.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:18.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:18.426825+0000) 2024-03-06T13:08:19.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:19.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:19.427078+0000) 2024-03-06T13:08:20.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:20.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:20.427275+0000) 2024-03-06T13:08:21.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:21.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:21.427478+0000) 2024-03-06T13:08:22.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:22.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:22.427688+0000) 2024-03-06T13:08:23.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:23.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:23.427925+0000) 2024-03-06T13:08:24.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:24.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:24.428158+0000) 2024-03-06T13:08:25.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:25.174 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:25.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:25.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:25.428405+0000) 2024-03-06T13:08:26.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:26.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:26.428616+0000) 2024-03-06T13:08:27.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:27.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:27.428770+0000) 2024-03-06T13:08:28.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:28.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:28.429041+0000) 2024-03-06T13:08:29.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:29.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:29.429251+0000) 2024-03-06T13:08:30.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:30.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:30.429439+0000) 2024-03-06T13:08:31.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:31.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:31.429642+0000) 2024-03-06T13:08:32.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:32.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:32.429861+0000) 2024-03-06T13:08:33.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:33.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:33.430126+0000) 2024-03-06T13:08:34.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:34.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:34.430307+0000) 2024-03-06T13:08:35.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:35.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:35.430511+0000) 2024-03-06T13:08:36.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:36.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:36.430736+0000) 2024-03-06T13:08:37.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:37.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:37.430960+0000) 2024-03-06T13:08:38.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:38.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:38.431165+0000) 2024-03-06T13:08:39.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:39.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:39.431373+0000) 2024-03-06T13:08:40.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:40.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:40.431585+0000) 2024-03-06T13:08:41.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:41.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:41.431813+0000) 2024-03-06T13:08:42.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:42.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:42.432065+0000) 2024-03-06T13:08:43.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:43.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:43.432296+0000) 2024-03-06T13:08:44.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:44.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:44.432529+0000) 2024-03-06T13:08:45.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:45.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:45.432758+0000) 2024-03-06T13:08:46.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:46.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:46.432996+0000) 2024-03-06T13:08:47.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:47.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:47.433337+0000) 2024-03-06T13:08:48.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:48.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:48.433584+0000) 2024-03-06T13:08:49.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:49.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:49.433823+0000) 2024-03-06T13:08:50.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:50.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:50.434106+0000) 2024-03-06T13:08:51.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:51.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:51.434339+0000) 2024-03-06T13:08:52.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:52.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:52.434593+0000) 2024-03-06T13:08:53.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:53.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:53.434858+0000) 2024-03-06T13:08:54.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:54.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:54.435129+0000) 2024-03-06T13:08:55.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:55.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:08:55.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:55.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:55.435383+0000) 2024-03-06T13:08:56.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:56.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:56.435627+0000) 2024-03-06T13:08:57.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:57.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:57.435853+0000) 2024-03-06T13:08:58.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:58.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:58.436125+0000) 2024-03-06T13:08:59.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:08:59.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:08:59.436340+0000) 2024-03-06T13:09:00.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:00.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:00.436568+0000) 2024-03-06T13:09:01.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:01.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:01.436803+0000) 2024-03-06T13:09:02.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:02.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:02.437074+0000) 2024-03-06T13:09:03.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:03.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:03.437236+0000) 2024-03-06T13:09:04.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:04.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:04.437463+0000) 2024-03-06T13:09:05.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:05.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:05.437689+0000) 2024-03-06T13:09:06.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:06.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:06.437907+0000) 2024-03-06T13:09:07.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:07.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:07.438148+0000) 2024-03-06T13:09:08.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:08.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:08.438361+0000) 2024-03-06T13:09:09.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:09.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:09.438600+0000) 2024-03-06T13:09:10.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:10.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:10.438840+0000) 2024-03-06T13:09:11.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:11.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:11.439113+0000) 2024-03-06T13:09:12.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:12.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:12.439356+0000) 2024-03-06T13:09:13.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:13.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:13.439636+0000) 2024-03-06T13:09:14.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:14.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:14.439875+0000) 2024-03-06T13:09:15.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:15.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:15.440148+0000) 2024-03-06T13:09:16.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:16.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:16.440339+0000) 2024-03-06T13:09:17.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:17.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:17.440543+0000) 2024-03-06T13:09:18.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:18.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:18.440793+0000) 2024-03-06T13:09:19.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:19.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:19.441050+0000) 2024-03-06T13:09:20.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:20.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:20.441224+0000) 2024-03-06T13:09:21.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:21.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:21.441407+0000) 2024-03-06T13:09:22.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:22.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:22.441655+0000) 2024-03-06T13:09:23.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:23.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:23.441873+0000) 2024-03-06T13:09:24.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:24.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:24.442122+0000) 2024-03-06T13:09:25.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:25.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:25.442332+0000) 2024-03-06T13:09:25.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:25.615 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:26.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:26.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:26.442618+0000) 2024-03-06T13:09:27.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:27.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:27.442874+0000) 2024-03-06T13:09:28.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:28.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:28.443151+0000) 2024-03-06T13:09:29.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:29.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:29.443404+0000) 2024-03-06T13:09:30.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:30.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:30.443661+0000) 2024-03-06T13:09:31.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:31.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:31.443859+0000) 2024-03-06T13:09:32.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:32.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:32.444145+0000) 2024-03-06T13:09:33.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:33.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:33.444418+0000) 2024-03-06T13:09:34.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:34.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:34.444659+0000) 2024-03-06T13:09:35.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:35.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:35.444959+0000) 2024-03-06T13:09:36.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:36.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:36.445185+0000) 2024-03-06T13:09:37.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:37.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:37.445408+0000) 2024-03-06T13:09:38.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:38.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:38.445633+0000) 2024-03-06T13:09:39.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:39.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:39.445885+0000) 2024-03-06T13:09:40.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:40.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:40.446117+0000) 2024-03-06T13:09:41.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:41.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:41.446272+0000) 2024-03-06T13:09:42.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:42.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:42.446490+0000) 2024-03-06T13:09:43.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:43.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:43.446738+0000) 2024-03-06T13:09:44.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:44.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:44.446979+0000) 2024-03-06T13:09:45.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:45.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:45.447239+0000) 2024-03-06T13:09:46.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:46.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:46.447443+0000) 2024-03-06T13:09:47.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:47.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:47.447656+0000) 2024-03-06T13:09:48.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:48.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:48.447920+0000) 2024-03-06T13:09:49.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:49.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:49.448143+0000) 2024-03-06T13:09:50.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:50.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:50.448370+0000) 2024-03-06T13:09:51.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:51.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:51.448594+0000) 2024-03-06T13:09:52.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:52.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:52.448812+0000) 2024-03-06T13:09:53.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:53.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:53.449082+0000) 2024-03-06T13:09:54.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:54.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:54.449263+0000) 2024-03-06T13:09:55.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:55.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:55.449498+0000) 2024-03-06T13:09:55.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:55.743 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:09:56.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:56.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:56.449751+0000) 2024-03-06T13:09:57.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:57.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:57.449947+0000) 2024-03-06T13:09:58.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:58.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:58.450159+0000) 2024-03-06T13:09:59.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:09:59.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:09:59.450388+0000) 2024-03-06T13:10:00.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:00.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:00.450657+0000) 2024-03-06T13:10:01.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:01.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:01.450936+0000) 2024-03-06T13:10:02.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:02.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:02.451144+0000) 2024-03-06T13:10:03.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:03.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:03.451348+0000) 2024-03-06T13:10:04.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:04.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:04.451592+0000) 2024-03-06T13:10:05.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:05.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:05.451830+0000) 2024-03-06T13:10:06.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:06.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:06.452089+0000) 2024-03-06T13:10:07.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:07.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:07.452317+0000) 2024-03-06T13:10:08.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:08.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:08.452552+0000) 2024-03-06T13:10:09.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:09.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:09.452767+0000) 2024-03-06T13:10:10.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:10.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:10.453039+0000) 2024-03-06T13:10:11.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:11.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:11.453280+0000) 2024-03-06T13:10:12.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:12.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:12.453482+0000) 2024-03-06T13:10:13.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:13.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:13.453683+0000) 2024-03-06T13:10:14.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:14.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:14.453877+0000) 2024-03-06T13:10:15.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:15.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:15.454128+0000) 2024-03-06T13:10:16.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:16.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:16.454358+0000) 2024-03-06T13:10:17.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:17.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:17.454606+0000) 2024-03-06T13:10:18.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:18.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:18.454849+0000) 2024-03-06T13:10:19.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:19.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:19.455119+0000) 2024-03-06T13:10:20.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:20.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:20.455357+0000) 2024-03-06T13:10:21.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:21.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:21.455578+0000) 2024-03-06T13:10:22.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:22.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:22.455822+0000) 2024-03-06T13:10:23.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:23.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:23.456121+0000) 2024-03-06T13:10:24.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:24.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:24.456345+0000) 2024-03-06T13:10:25.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:25.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:25.456609+0000) 2024-03-06T13:10:25.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:25.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:26.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:26.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:26.456834+0000) 2024-03-06T13:10:27.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:27.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:27.457080+0000) 2024-03-06T13:10:28.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:28.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:28.457385+0000) 2024-03-06T13:10:29.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:29.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:29.457610+0000) 2024-03-06T13:10:30.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:30.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:30.457829+0000) 2024-03-06T13:10:31.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:31.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:31.458081+0000) 2024-03-06T13:10:32.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:32.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:32.458270+0000) 2024-03-06T13:10:33.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:33.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:33.458509+0000) 2024-03-06T13:10:34.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:34.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:34.458752+0000) 2024-03-06T13:10:35.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:35.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:35.458968+0000) 2024-03-06T13:10:36.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:36.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:36.459154+0000) 2024-03-06T13:10:37.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:37.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:37.459332+0000) 2024-03-06T13:10:38.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:38.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:38.459574+0000) 2024-03-06T13:10:39.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:39.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:39.459805+0000) 2024-03-06T13:10:40.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:40.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:40.460076+0000) 2024-03-06T13:10:41.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:41.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:41.460305+0000) 2024-03-06T13:10:42.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:42.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:42.460527+0000) 2024-03-06T13:10:43.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:43.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:43.460697+0000) 2024-03-06T13:10:44.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:44.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:44.460874+0000) 2024-03-06T13:10:45.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:45.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:45.461103+0000) 2024-03-06T13:10:46.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:46.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:46.461328+0000) 2024-03-06T13:10:47.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:47.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:47.461546+0000) 2024-03-06T13:10:48.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:48.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:48.461799+0000) 2024-03-06T13:10:49.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:49.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:49.462069+0000) 2024-03-06T13:10:50.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:50.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:50.462278+0000) 2024-03-06T13:10:51.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:51.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:51.462489+0000) 2024-03-06T13:10:52.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:52.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:52.462718+0000) 2024-03-06T13:10:53.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:53.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:53.462936+0000) 2024-03-06T13:10:54.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:54.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:54.463153+0000) 2024-03-06T13:10:55.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:55.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:55.463336+0000) 2024-03-06T13:10:56.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:56.129 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:10:56.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:56.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:56.463544+0000) 2024-03-06T13:10:57.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:57.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:57.463802+0000) 2024-03-06T13:10:58.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:58.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:58.464090+0000) 2024-03-06T13:10:59.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:10:59.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:10:59.464282+0000) 2024-03-06T13:11:00.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:00.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:00.464464+0000) 2024-03-06T13:11:01.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:01.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:01.464644+0000) 2024-03-06T13:11:02.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:02.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:02.464865+0000) 2024-03-06T13:11:03.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:03.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:03.465081+0000) 2024-03-06T13:11:04.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:04.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:04.465224+0000) 2024-03-06T13:11:05.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:05.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:05.465416+0000) 2024-03-06T13:11:06.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:06.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:06.465635+0000) 2024-03-06T13:11:07.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:07.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:07.465867+0000) 2024-03-06T13:11:08.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:08.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:08.466147+0000) 2024-03-06T13:11:09.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:09.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:09.466338+0000) 2024-03-06T13:11:10.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:10.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:10.466528+0000) 2024-03-06T13:11:11.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:11.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:11.466728+0000) 2024-03-06T13:11:12.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:12.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:12.466944+0000) 2024-03-06T13:11:13.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:13.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:13.467170+0000) 2024-03-06T13:11:14.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:14.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:14.467393+0000) 2024-03-06T13:11:15.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:15.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:15.467599+0000) 2024-03-06T13:11:16.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:16.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:16.467896+0000) 2024-03-06T13:11:17.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:17.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:17.468175+0000) 2024-03-06T13:11:18.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:18.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:18.468438+0000) 2024-03-06T13:11:19.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:19.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:19.468723+0000) 2024-03-06T13:11:20.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:20.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:20.469036+0000) 2024-03-06T13:11:21.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:21.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:21.469286+0000) 2024-03-06T13:11:22.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:22.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:22.469503+0000) 2024-03-06T13:11:23.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:23.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:23.469748+0000) 2024-03-06T13:11:24.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:24.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:24.469983+0000) 2024-03-06T13:11:25.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:25.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:25.470184+0000) 2024-03-06T13:11:26.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:26.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:26.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:26.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:26.470341+0000) 2024-03-06T13:11:27.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:27.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:27.470560+0000) 2024-03-06T13:11:28.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:28.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:28.470782+0000) 2024-03-06T13:11:29.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:29.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:29.470956+0000) 2024-03-06T13:11:30.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:30.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:30.471168+0000) 2024-03-06T13:11:31.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:31.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:31.471367+0000) 2024-03-06T13:11:32.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:32.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:32.471577+0000) 2024-03-06T13:11:33.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:33.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:33.471813+0000) 2024-03-06T13:11:34.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:34.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:34.472077+0000) 2024-03-06T13:11:35.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:35.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:35.472299+0000) 2024-03-06T13:11:36.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:36.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:36.472523+0000) 2024-03-06T13:11:37.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:37.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:37.472751+0000) 2024-03-06T13:11:38.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:38.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:38.472987+0000) 2024-03-06T13:11:39.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:39.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:39.473213+0000) 2024-03-06T13:11:40.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:40.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:40.473441+0000) 2024-03-06T13:11:41.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:41.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:41.473642+0000) 2024-03-06T13:11:42.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:42.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:42.473826+0000) 2024-03-06T13:11:43.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:43.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:43.474119+0000) 2024-03-06T13:11:44.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:44.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:44.474343+0000) 2024-03-06T13:11:45.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:45.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:45.474524+0000) 2024-03-06T13:11:46.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:46.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:46.474707+0000) 2024-03-06T13:11:47.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:47.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:47.474932+0000) 2024-03-06T13:11:48.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:48.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:48.475175+0000) 2024-03-06T13:11:49.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:49.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:49.475331+0000) 2024-03-06T13:11:50.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:50.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:50.475555+0000) 2024-03-06T13:11:51.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:51.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:51.475783+0000) 2024-03-06T13:11:52.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:52.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:52.476032+0000) 2024-03-06T13:11:53.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:53.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:53.476282+0000) 2024-03-06T13:11:54.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:54.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:54.476500+0000) 2024-03-06T13:11:55.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:55.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:55.476739+0000) 2024-03-06T13:11:56.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:56.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:56.477063+0000) 2024-03-06T13:11:56.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:56.579 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:11:57.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:57.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:57.477270+0000) 2024-03-06T13:11:58.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:58.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:58.477510+0000) 2024-03-06T13:11:59.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:11:59.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:11:59.477732+0000) 2024-03-06T13:12:00.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:00.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:00.477989+0000) 2024-03-06T13:12:01.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:01.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:01.478213+0000) 2024-03-06T13:12:02.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:02.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:02.478431+0000) 2024-03-06T13:12:03.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:03.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:03.478675+0000) 2024-03-06T13:12:04.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:04.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:04.478941+0000) 2024-03-06T13:12:05.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:05.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:05.479345+0000) 2024-03-06T13:12:06.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:06.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:06.479621+0000) 2024-03-06T13:12:07.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:07.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:07.479907+0000) 2024-03-06T13:12:08.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:08.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:08.480178+0000) 2024-03-06T13:12:09.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:09.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:09.480423+0000) 2024-03-06T13:12:10.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:10.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:10.480645+0000) 2024-03-06T13:12:11.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:11.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:11.480906+0000) 2024-03-06T13:12:12.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:12.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:12.481193+0000) 2024-03-06T13:12:13.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:13.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:13.481520+0000) 2024-03-06T13:12:14.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:14.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:14.481743+0000) 2024-03-06T13:12:15.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:15.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:15.481976+0000) 2024-03-06T13:12:16.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:16.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:16.482243+0000) 2024-03-06T13:12:17.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:17.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:17.482466+0000) 2024-03-06T13:12:18.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:18.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:18.482701+0000) 2024-03-06T13:12:19.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:19.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:19.482945+0000) 2024-03-06T13:12:20.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:20.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:20.483192+0000) 2024-03-06T13:12:21.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:21.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:21.483345+0000) 2024-03-06T13:12:22.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:22.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:22.483543+0000) 2024-03-06T13:12:23.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:23.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:23.483779+0000) 2024-03-06T13:12:24.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:24.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:24.484027+0000) 2024-03-06T13:12:25.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:25.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:25.484211+0000) 2024-03-06T13:12:26.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:26.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:26.484489+0000) 2024-03-06T13:12:26.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:26.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:27.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:27.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:27.484771+0000) 2024-03-06T13:12:28.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:28.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:28.484983+0000) 2024-03-06T13:12:29.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:29.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:29.485175+0000) 2024-03-06T13:12:30.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:30.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:30.485407+0000) 2024-03-06T13:12:31.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:31.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:31.485618+0000) 2024-03-06T13:12:32.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:32.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:32.485887+0000) 2024-03-06T13:12:33.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:33.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:33.486168+0000) 2024-03-06T13:12:34.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:34.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:34.486394+0000) 2024-03-06T13:12:35.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:35.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:35.486614+0000) 2024-03-06T13:12:36.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:36.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:36.486801+0000) 2024-03-06T13:12:37.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:37.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:37.487039+0000) 2024-03-06T13:12:38.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:38.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:38.487252+0000) 2024-03-06T13:12:39.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:39.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:39.487479+0000) 2024-03-06T13:12:40.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:40.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:40.487705+0000) 2024-03-06T13:12:41.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:41.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:41.487929+0000) 2024-03-06T13:12:42.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:42.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:42.488149+0000) 2024-03-06T13:12:43.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:43.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:43.488378+0000) 2024-03-06T13:12:44.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:44.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:44.488591+0000) 2024-03-06T13:12:45.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:45.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:45.488805+0000) 2024-03-06T13:12:46.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:46.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:46.489040+0000) 2024-03-06T13:12:47.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:47.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:47.489234+0000) 2024-03-06T13:12:48.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:48.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:48.489483+0000) 2024-03-06T13:12:49.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:49.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:49.489686+0000) 2024-03-06T13:12:50.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:50.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:50.489897+0000) 2024-03-06T13:12:51.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:51.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:51.490116+0000) 2024-03-06T13:12:52.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:52.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:52.490347+0000) 2024-03-06T13:12:53.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:53.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:53.490629+0000) 2024-03-06T13:12:54.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:54.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:54.490846+0000) 2024-03-06T13:12:55.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:55.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:55.491058+0000) 2024-03-06T13:12:56.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:56.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:56.491211+0000) 2024-03-06T13:12:56.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:56.986 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:12:57.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:57.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:57.491470+0000) 2024-03-06T13:12:58.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:58.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:58.491716+0000) 2024-03-06T13:12:59.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:12:59.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:12:59.491943+0000) 2024-03-06T13:13:00.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:00.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:00.492160+0000) 2024-03-06T13:13:01.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:01.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:01.492380+0000) 2024-03-06T13:13:02.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:02.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:02.492603+0000) 2024-03-06T13:13:03.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:03.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:03.492886+0000) 2024-03-06T13:13:04.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:04.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:04.493122+0000) 2024-03-06T13:13:05.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:05.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:05.493344+0000) 2024-03-06T13:13:06.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:06.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:06.493566+0000) 2024-03-06T13:13:07.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:07.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:07.493752+0000) 2024-03-06T13:13:08.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:08.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:08.493964+0000) 2024-03-06T13:13:09.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:09.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:09.494218+0000) 2024-03-06T13:13:10.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:10.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:10.494437+0000) 2024-03-06T13:13:11.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:11.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:11.494711+0000) 2024-03-06T13:13:12.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:12.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:12.494980+0000) 2024-03-06T13:13:13.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:13.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:13.495297+0000) 2024-03-06T13:13:14.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:14.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:14.495563+0000) 2024-03-06T13:13:15.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:15.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:15.495842+0000) 2024-03-06T13:13:16.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:16.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:16.496156+0000) 2024-03-06T13:13:17.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:17.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:17.496442+0000) 2024-03-06T13:13:18.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:18.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:18.496715+0000) 2024-03-06T13:13:19.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:19.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:19.496950+0000) 2024-03-06T13:13:20.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:20.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:20.497291+0000) 2024-03-06T13:13:21.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:21.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:21.497528+0000) 2024-03-06T13:13:22.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:22.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:22.497730+0000) 2024-03-06T13:13:23.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:23.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:23.497983+0000) 2024-03-06T13:13:24.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:24.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:24.498282+0000) 2024-03-06T13:13:25.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:25.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:25.498490+0000) 2024-03-06T13:13:26.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:26.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:26.498755+0000) 2024-03-06T13:13:27.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:27.167 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:27.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:27.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:27.499102+0000) 2024-03-06T13:13:28.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:28.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:28.499400+0000) 2024-03-06T13:13:29.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:29.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:29.499669+0000) 2024-03-06T13:13:30.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:30.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:30.499946+0000) 2024-03-06T13:13:31.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:31.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:31.500221+0000) 2024-03-06T13:13:32.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:32.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:32.500493+0000) 2024-03-06T13:13:33.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:33.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:33.500696+0000) 2024-03-06T13:13:34.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:34.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:34.500912+0000) 2024-03-06T13:13:35.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:35.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:35.501190+0000) 2024-03-06T13:13:36.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:36.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:36.501401+0000) 2024-03-06T13:13:37.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:37.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:37.501605+0000) 2024-03-06T13:13:38.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:38.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:38.501860+0000) 2024-03-06T13:13:39.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:39.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:39.502096+0000) 2024-03-06T13:13:40.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:40.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:40.502331+0000) 2024-03-06T13:13:41.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:41.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:41.502548+0000) 2024-03-06T13:13:42.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:42.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:42.502762+0000) 2024-03-06T13:13:43.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:43.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:43.502966+0000) 2024-03-06T13:13:44.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:44.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:44.503215+0000) 2024-03-06T13:13:45.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:45.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:45.503451+0000) 2024-03-06T13:13:46.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:46.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:46.503655+0000) 2024-03-06T13:13:47.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:47.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:47.503869+0000) 2024-03-06T13:13:48.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:48.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:48.504148+0000) 2024-03-06T13:13:49.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:49.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:49.504371+0000) 2024-03-06T13:13:50.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:50.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:50.504599+0000) 2024-03-06T13:13:51.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:51.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:51.504825+0000) 2024-03-06T13:13:52.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:52.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:52.505097+0000) 2024-03-06T13:13:53.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:53.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:53.505332+0000) 2024-03-06T13:13:54.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:54.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:54.505562+0000) 2024-03-06T13:13:55.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:55.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:55.505786+0000) 2024-03-06T13:13:56.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:56.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:56.505996+0000) 2024-03-06T13:13:57.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:57.386 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:13:57.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:57.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:57.506296+0000) 2024-03-06T13:13:58.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:58.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:58.506535+0000) 2024-03-06T13:13:59.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:13:59.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:13:59.506761+0000) 2024-03-06T13:14:00.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:00.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:00.506977+0000) 2024-03-06T13:14:01.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:01.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:01.507248+0000) 2024-03-06T13:14:02.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:02.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:02.507502+0000) 2024-03-06T13:14:03.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:03.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:03.507811+0000) 2024-03-06T13:14:04.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:04.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:04.508119+0000) 2024-03-06T13:14:05.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:05.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:05.508357+0000) 2024-03-06T13:14:06.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:06.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:06.508642+0000) 2024-03-06T13:14:07.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:07.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:07.508919+0000) 2024-03-06T13:14:08.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:08.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:08.509211+0000) 2024-03-06T13:14:09.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:09.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:09.509448+0000) 2024-03-06T13:14:10.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:10.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:10.509676+0000) 2024-03-06T13:14:11.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:11.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:11.509902+0000) 2024-03-06T13:14:12.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:12.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:12.510143+0000) 2024-03-06T13:14:13.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:13.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:13.510337+0000) 2024-03-06T13:14:14.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:14.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:14.510563+0000) 2024-03-06T13:14:15.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:15.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:15.510799+0000) 2024-03-06T13:14:16.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:16.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:16.511069+0000) 2024-03-06T13:14:17.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:17.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:17.511271+0000) 2024-03-06T13:14:18.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:18.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:18.511505+0000) 2024-03-06T13:14:19.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:19.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:19.511727+0000) 2024-03-06T13:14:20.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:20.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:20.511979+0000) 2024-03-06T13:14:21.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:21.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:21.512277+0000) 2024-03-06T13:14:22.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:22.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:22.512500+0000) 2024-03-06T13:14:23.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:23.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:23.512750+0000) 2024-03-06T13:14:24.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:24.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:24.513035+0000) 2024-03-06T13:14:25.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:25.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:25.513290+0000) 2024-03-06T13:14:26.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:26.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:26.513510+0000) 2024-03-06T13:14:27.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:27.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:27.513745+0000) 2024-03-06T13:14:27.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:27.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:28.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:28.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:28.514035+0000) 2024-03-06T13:14:29.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:29.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:29.514261+0000) 2024-03-06T13:14:30.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:30.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:30.514494+0000) 2024-03-06T13:14:31.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:31.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:31.514733+0000) 2024-03-06T13:14:32.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:32.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:32.514971+0000) 2024-03-06T13:14:33.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:33.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:33.515271+0000) 2024-03-06T13:14:34.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:34.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:34.515501+0000) 2024-03-06T13:14:35.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:35.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:35.515715+0000) 2024-03-06T13:14:36.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:36.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:36.515962+0000) 2024-03-06T13:14:37.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:37.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:37.516191+0000) 2024-03-06T13:14:38.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:38.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:38.516424+0000) 2024-03-06T13:14:39.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:39.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:39.516645+0000) 2024-03-06T13:14:40.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:40.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:40.516875+0000) 2024-03-06T13:14:41.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:41.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:41.517128+0000) 2024-03-06T13:14:42.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:42.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:42.517356+0000) 2024-03-06T13:14:43.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:43.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:43.517600+0000) 2024-03-06T13:14:44.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:44.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:44.517820+0000) 2024-03-06T13:14:45.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:45.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:45.518076+0000) 2024-03-06T13:14:46.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:46.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:46.518264+0000) 2024-03-06T13:14:47.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:47.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:47.518483+0000) 2024-03-06T13:14:48.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:48.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:48.518729+0000) 2024-03-06T13:14:49.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:49.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:49.518949+0000) 2024-03-06T13:14:50.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:50.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:50.519152+0000) 2024-03-06T13:14:51.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:51.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:51.519404+0000) 2024-03-06T13:14:52.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:52.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:52.519645+0000) 2024-03-06T13:14:53.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:53.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:53.519866+0000) 2024-03-06T13:14:54.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:54.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:54.520115+0000) 2024-03-06T13:14:55.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:55.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:55.520322+0000) 2024-03-06T13:14:56.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:56.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:56.520560+0000) 2024-03-06T13:14:57.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:57.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:57.520794+0000) 2024-03-06T13:14:57.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:57.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:14:58.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:58.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:58.521058+0000) 2024-03-06T13:14:59.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:14:59.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:14:59.521299+0000) 2024-03-06T13:15:00.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:00.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:00.521533+0000) 2024-03-06T13:15:01.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:01.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:01.521874+0000) 2024-03-06T13:15:02.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:02.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:02.522071+0000) 2024-03-06T13:15:03.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:03.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:03.522288+0000) 2024-03-06T13:15:04.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:04.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:04.522474+0000) 2024-03-06T13:15:05.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:05.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:05.522695+0000) 2024-03-06T13:15:06.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:06.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:06.522916+0000) 2024-03-06T13:15:07.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:07.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:07.523140+0000) 2024-03-06T13:15:08.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:08.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:08.523317+0000) 2024-03-06T13:15:09.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:09.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:09.523536+0000) 2024-03-06T13:15:10.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:10.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:10.523777+0000) 2024-03-06T13:15:11.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:11.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:11.524047+0000) 2024-03-06T13:15:12.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:12.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:12.524258+0000) 2024-03-06T13:15:13.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:13.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:13.524516+0000) 2024-03-06T13:15:14.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:14.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:14.524761+0000) 2024-03-06T13:15:15.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:15.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:15.524995+0000) 2024-03-06T13:15:16.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:16.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:16.525243+0000) 2024-03-06T13:15:17.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:17.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:17.525480+0000) 2024-03-06T13:15:18.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:18.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:18.525701+0000) 2024-03-06T13:15:19.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:19.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:19.525938+0000) 2024-03-06T13:15:20.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:20.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:20.526142+0000) 2024-03-06T13:15:21.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:21.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:21.526367+0000) 2024-03-06T13:15:22.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:22.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:22.526584+0000) 2024-03-06T13:15:23.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:23.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:23.526819+0000) 2024-03-06T13:15:24.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:24.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:24.527124+0000) 2024-03-06T13:15:25.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:25.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:25.527362+0000) 2024-03-06T13:15:26.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:26.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:26.527575+0000) 2024-03-06T13:15:27.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:27.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:27.527866+0000) 2024-03-06T13:15:27.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:27.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:28.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:28.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:28.528164+0000) 2024-03-06T13:15:29.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:29.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:29.528401+0000) 2024-03-06T13:15:30.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:30.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:30.528657+0000) 2024-03-06T13:15:31.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:31.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:31.528939+0000) 2024-03-06T13:15:32.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:32.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:32.529170+0000) 2024-03-06T13:15:33.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:33.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:33.529423+0000) 2024-03-06T13:15:34.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:34.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:34.529706+0000) 2024-03-06T13:15:35.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:35.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:35.529953+0000) 2024-03-06T13:15:36.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:36.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:36.530169+0000) 2024-03-06T13:15:37.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:37.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:37.530411+0000) 2024-03-06T13:15:38.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:38.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:38.530613+0000) 2024-03-06T13:15:39.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:39.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:39.530848+0000) 2024-03-06T13:15:40.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:40.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:40.531121+0000) 2024-03-06T13:15:41.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:41.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:41.531313+0000) 2024-03-06T13:15:42.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:42.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:42.531516+0000) 2024-03-06T13:15:43.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:43.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:43.531743+0000) 2024-03-06T13:15:44.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:44.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:44.531969+0000) 2024-03-06T13:15:45.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:45.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:45.532166+0000) 2024-03-06T13:15:46.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:46.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:46.532362+0000) 2024-03-06T13:15:47.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:47.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:47.532610+0000) 2024-03-06T13:15:48.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:48.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:48.532887+0000) 2024-03-06T13:15:49.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:49.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:49.533155+0000) 2024-03-06T13:15:50.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:50.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:50.533399+0000) 2024-03-06T13:15:51.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:51.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:51.533659+0000) 2024-03-06T13:15:52.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:52.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:52.533849+0000) 2024-03-06T13:15:53.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:53.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:53.534086+0000) 2024-03-06T13:15:54.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:54.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:54.534291+0000) 2024-03-06T13:15:55.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:55.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:55.534521+0000) 2024-03-06T13:15:56.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:56.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:56.534779+0000) 2024-03-06T13:15:57.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:57.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:57.535051+0000) 2024-03-06T13:15:58.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:58.149 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:15:58.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:58.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:58.535300+0000) 2024-03-06T13:15:59.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:15:59.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:15:59.535524+0000) 2024-03-06T13:16:00.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:00.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:00.535760+0000) 2024-03-06T13:16:01.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:01.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:01.535980+0000) 2024-03-06T13:16:02.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:02.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:02.536223+0000) 2024-03-06T13:16:03.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:03.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:03.536452+0000) 2024-03-06T13:16:04.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:04.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:04.536663+0000) 2024-03-06T13:16:05.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:05.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:05.536881+0000) 2024-03-06T13:16:06.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:06.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:06.537118+0000) 2024-03-06T13:16:07.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:07.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:07.537259+0000) 2024-03-06T13:16:08.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:08.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:08.537477+0000) 2024-03-06T13:16:09.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:09.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:09.537724+0000) 2024-03-06T13:16:10.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:10.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:10.537984+0000) 2024-03-06T13:16:11.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:11.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:11.538312+0000) 2024-03-06T13:16:12.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:12.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:12.538541+0000) 2024-03-06T13:16:13.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:13.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:13.538756+0000) 2024-03-06T13:16:14.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:14.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:14.538901+0000) 2024-03-06T13:16:15.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:15.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:15.539139+0000) 2024-03-06T13:16:16.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:16.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:16.539347+0000) 2024-03-06T13:16:17.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:17.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:17.539601+0000) 2024-03-06T13:16:18.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:18.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:18.539841+0000) 2024-03-06T13:16:19.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:19.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:19.540149+0000) 2024-03-06T13:16:20.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:20.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:20.540412+0000) 2024-03-06T13:16:21.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:21.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:21.540622+0000) 2024-03-06T13:16:22.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:22.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:22.540859+0000) 2024-03-06T13:16:23.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:23.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:23.541151+0000) 2024-03-06T13:16:24.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:24.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:24.541402+0000) 2024-03-06T13:16:25.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:25.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:25.541640+0000) 2024-03-06T13:16:26.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:26.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:26.541879+0000) 2024-03-06T13:16:27.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:27.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:27.542159+0000) 2024-03-06T13:16:28.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:28.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:28.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:28.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:28.542387+0000) 2024-03-06T13:16:29.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:29.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:29.542620+0000) 2024-03-06T13:16:30.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:30.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:30.542861+0000) 2024-03-06T13:16:31.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:31.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:31.543130+0000) 2024-03-06T13:16:32.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:32.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:32.543337+0000) 2024-03-06T13:16:33.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:33.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:33.543579+0000) 2024-03-06T13:16:34.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:34.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:34.543836+0000) 2024-03-06T13:16:35.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:35.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:35.544098+0000) 2024-03-06T13:16:36.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:36.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:36.544278+0000) 2024-03-06T13:16:37.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:37.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:37.544497+0000) 2024-03-06T13:16:38.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:38.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:38.544757+0000) 2024-03-06T13:16:39.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:39.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:39.544983+0000) 2024-03-06T13:16:40.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:40.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:40.545200+0000) 2024-03-06T13:16:41.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:41.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:41.545437+0000) 2024-03-06T13:16:42.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:42.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:42.545660+0000) 2024-03-06T13:16:43.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:43.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:43.545893+0000) 2024-03-06T13:16:44.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:44.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:44.546172+0000) 2024-03-06T13:16:45.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:45.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:45.546376+0000) 2024-03-06T13:16:46.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:46.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:46.546613+0000) 2024-03-06T13:16:47.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:47.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:47.546845+0000) 2024-03-06T13:16:48.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:48.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:48.547133+0000) 2024-03-06T13:16:49.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:49.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:49.547383+0000) 2024-03-06T13:16:50.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:50.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:50.547584+0000) 2024-03-06T13:16:51.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:51.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:51.547802+0000) 2024-03-06T13:16:52.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:52.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:52.548064+0000) 2024-03-06T13:16:53.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:53.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:53.548298+0000) 2024-03-06T13:16:54.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:54.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:54.548491+0000) 2024-03-06T13:16:55.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:55.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:55.548709+0000) 2024-03-06T13:16:56.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:56.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:56.548930+0000) 2024-03-06T13:16:57.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:57.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:57.549154+0000) 2024-03-06T13:16:58.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:58.529 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:16:58.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:58.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:58.549325+0000) 2024-03-06T13:16:59.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:16:59.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:16:59.549548+0000) 2024-03-06T13:17:00.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:00.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:00.549746+0000) 2024-03-06T13:17:01.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:01.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:01.549964+0000) 2024-03-06T13:17:02.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:02.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:02.550271+0000) 2024-03-06T13:17:03.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:03.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:03.550557+0000) 2024-03-06T13:17:04.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:04.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:04.550783+0000) 2024-03-06T13:17:05.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:05.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:05.550967+0000) 2024-03-06T13:17:06.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:06.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:06.551195+0000) 2024-03-06T13:17:07.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:07.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:07.551431+0000) 2024-03-06T13:17:08.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:08.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:08.551673+0000) 2024-03-06T13:17:09.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:09.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:09.551905+0000) 2024-03-06T13:17:10.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:10.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:10.552148+0000) 2024-03-06T13:17:11.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:11.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:11.552335+0000) 2024-03-06T13:17:12.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:12.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:12.552559+0000) 2024-03-06T13:17:13.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:13.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:13.552788+0000) 2024-03-06T13:17:14.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:14.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:14.553064+0000) 2024-03-06T13:17:15.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:15.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:15.553307+0000) 2024-03-06T13:17:16.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:16.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:16.553551+0000) 2024-03-06T13:17:17.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:17.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:17.553768+0000) 2024-03-06T13:17:18.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:18.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:18.553999+0000) 2024-03-06T13:17:19.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:19.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:19.554271+0000) 2024-03-06T13:17:20.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:20.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:20.554500+0000) 2024-03-06T13:17:21.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:21.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:21.554756+0000) 2024-03-06T13:17:22.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:22.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:22.554984+0000) 2024-03-06T13:17:23.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:23.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:23.555271+0000) 2024-03-06T13:17:24.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:24.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:24.555538+0000) 2024-03-06T13:17:25.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:25.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:25.555765+0000) 2024-03-06T13:17:26.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:26.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:26.555986+0000) 2024-03-06T13:17:27.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:27.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:27.556243+0000) 2024-03-06T13:17:28.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:28.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:28.556463+0000) 2024-03-06T13:17:28.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:28.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:29.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:29.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:29.556705+0000) 2024-03-06T13:17:30.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:30.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:30.556947+0000) 2024-03-06T13:17:31.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:31.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:31.557172+0000) 2024-03-06T13:17:32.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:32.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:32.557401+0000) 2024-03-06T13:17:33.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:33.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:33.557634+0000) 2024-03-06T13:17:34.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:34.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:34.557879+0000) 2024-03-06T13:17:35.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:35.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:35.558111+0000) 2024-03-06T13:17:36.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:36.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:36.558305+0000) 2024-03-06T13:17:37.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:37.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:37.558517+0000) 2024-03-06T13:17:38.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:38.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:38.558700+0000) 2024-03-06T13:17:39.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:39.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:39.558953+0000) 2024-03-06T13:17:40.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:40.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:40.559160+0000) 2024-03-06T13:17:41.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:41.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:41.559367+0000) 2024-03-06T13:17:42.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:42.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:42.559586+0000) 2024-03-06T13:17:43.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:43.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:43.559831+0000) 2024-03-06T13:17:44.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:44.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:44.560135+0000) 2024-03-06T13:17:45.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:45.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:45.560348+0000) 2024-03-06T13:17:46.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:46.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:46.560572+0000) 2024-03-06T13:17:47.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:47.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:47.560809+0000) 2024-03-06T13:17:48.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:48.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:48.561081+0000) 2024-03-06T13:17:49.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:49.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:49.561337+0000) 2024-03-06T13:17:50.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:50.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:50.561589+0000) 2024-03-06T13:17:51.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:51.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:51.561833+0000) 2024-03-06T13:17:52.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:52.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:52.562103+0000) 2024-03-06T13:17:53.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:53.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:53.562370+0000) 2024-03-06T13:17:54.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:54.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:54.562615+0000) 2024-03-06T13:17:55.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:55.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:55.562828+0000) 2024-03-06T13:17:56.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:56.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:56.563211+0000) 2024-03-06T13:17:57.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:57.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:57.563479+0000) 2024-03-06T13:17:58.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:58.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:58.563631+0000) 2024-03-06T13:17:58.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:58.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:17:59.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:17:59.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:17:59.563865+0000) 2024-03-06T13:18:00.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:00.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:00.564129+0000) 2024-03-06T13:18:01.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:01.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:01.564351+0000) 2024-03-06T13:18:02.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:02.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:02.564559+0000) 2024-03-06T13:18:03.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:03.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:03.564770+0000) 2024-03-06T13:18:04.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:04.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:04.564981+0000) 2024-03-06T13:18:05.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:05.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:05.565230+0000) 2024-03-06T13:18:06.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:06.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:06.565458+0000) 2024-03-06T13:18:07.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:07.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:07.565679+0000) 2024-03-06T13:18:08.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:08.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:08.565870+0000) 2024-03-06T13:18:09.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:09.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:09.566142+0000) 2024-03-06T13:18:10.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:10.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:10.566430+0000) 2024-03-06T13:18:11.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:11.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:11.566680+0000) 2024-03-06T13:18:12.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:12.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:12.566904+0000) 2024-03-06T13:18:13.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:13.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:13.567130+0000) 2024-03-06T13:18:14.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:14.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:14.567375+0000) 2024-03-06T13:18:15.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:15.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:15.567580+0000) 2024-03-06T13:18:16.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:16.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:16.567806+0000) 2024-03-06T13:18:17.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:17.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:17.568065+0000) 2024-03-06T13:18:18.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:18.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:18.568319+0000) 2024-03-06T13:18:19.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:19.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:19.568620+0000) 2024-03-06T13:18:20.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:20.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:20.568893+0000) 2024-03-06T13:18:21.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:21.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:21.569130+0000) 2024-03-06T13:18:22.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:22.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:22.569314+0000) 2024-03-06T13:18:23.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:23.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:23.569529+0000) 2024-03-06T13:18:24.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:24.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:24.569744+0000) 2024-03-06T13:18:25.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:25.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:25.569968+0000) 2024-03-06T13:18:26.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:26.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:26.570184+0000) 2024-03-06T13:18:27.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:27.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:27.570385+0000) 2024-03-06T13:18:28.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:28.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:28.570633+0000) 2024-03-06T13:18:29.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:29.145 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:29.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:29.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:29.570856+0000) 2024-03-06T13:18:30.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:30.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:30.571127+0000) 2024-03-06T13:18:31.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:31.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:31.571331+0000) 2024-03-06T13:18:32.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:32.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:32.571573+0000) 2024-03-06T13:18:33.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:33.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:33.571802+0000) 2024-03-06T13:18:34.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:34.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:34.572093+0000) 2024-03-06T13:18:35.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:35.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:35.572331+0000) 2024-03-06T13:18:36.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:36.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:36.572559+0000) 2024-03-06T13:18:37.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:37.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:37.572776+0000) 2024-03-06T13:18:38.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:38.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:38.572998+0000) 2024-03-06T13:18:39.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:39.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:39.573233+0000) 2024-03-06T13:18:40.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:40.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:40.573431+0000) 2024-03-06T13:18:41.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:41.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:41.573637+0000) 2024-03-06T13:18:42.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:42.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:42.573852+0000) 2024-03-06T13:18:43.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:43.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:43.574108+0000) 2024-03-06T13:18:44.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:44.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:44.574402+0000) 2024-03-06T13:18:45.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:45.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:45.574609+0000) 2024-03-06T13:18:46.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:46.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:46.574830+0000) 2024-03-06T13:18:47.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:47.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:47.575079+0000) 2024-03-06T13:18:48.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:48.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:48.575301+0000) 2024-03-06T13:18:49.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:49.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:49.575553+0000) 2024-03-06T13:18:50.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:50.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:50.575760+0000) 2024-03-06T13:18:51.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:51.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:51.575989+0000) 2024-03-06T13:18:52.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:52.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:52.576241+0000) 2024-03-06T13:18:53.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:53.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:53.576471+0000) 2024-03-06T13:18:54.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:54.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:54.576745+0000) 2024-03-06T13:18:55.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:55.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:55.576983+0000) 2024-03-06T13:18:56.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:56.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:56.577226+0000) 2024-03-06T13:18:57.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:57.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:57.577457+0000) 2024-03-06T13:18:58.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:58.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:58.577711+0000) 2024-03-06T13:18:59.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:59.371 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:18:59.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:18:59.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:18:59.577965+0000) 2024-03-06T13:19:00.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:00.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:00.578220+0000) 2024-03-06T13:19:01.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:01.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:01.578460+0000) 2024-03-06T13:19:02.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:02.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:02.578702+0000) 2024-03-06T13:19:03.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:03.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:03.578967+0000) 2024-03-06T13:19:04.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:04.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:04.579229+0000) 2024-03-06T13:19:05.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:05.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:05.579431+0000) 2024-03-06T13:19:06.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:06.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:06.579627+0000) 2024-03-06T13:19:07.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:07.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:07.579843+0000) 2024-03-06T13:19:08.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:08.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:08.580117+0000) 2024-03-06T13:19:09.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:09.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:09.580282+0000) 2024-03-06T13:19:10.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:10.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:10.580493+0000) 2024-03-06T13:19:11.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:11.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:11.580742+0000) 2024-03-06T13:19:12.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:12.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:12.580979+0000) 2024-03-06T13:19:13.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:13.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:13.581222+0000) 2024-03-06T13:19:14.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:14.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:14.581440+0000) 2024-03-06T13:19:15.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:15.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:15.581671+0000) 2024-03-06T13:19:16.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:16.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:16.581923+0000) 2024-03-06T13:19:17.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:17.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:17.582147+0000) 2024-03-06T13:19:18.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:18.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:18.582369+0000) 2024-03-06T13:19:19.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:19.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:19.582607+0000) 2024-03-06T13:19:20.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:20.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:20.582839+0000) 2024-03-06T13:19:21.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:21.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:21.583102+0000) 2024-03-06T13:19:22.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:22.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:22.583322+0000) 2024-03-06T13:19:23.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:23.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:23.583553+0000) 2024-03-06T13:19:24.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:24.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:24.583792+0000) 2024-03-06T13:19:25.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:25.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:25.584068+0000) 2024-03-06T13:19:26.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:26.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:26.584299+0000) 2024-03-06T13:19:27.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:27.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:27.584433+0000) 2024-03-06T13:19:28.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:28.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:28.584605+0000) 2024-03-06T13:19:29.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:29.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:29.584747+0000) 2024-03-06T13:19:29.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:29.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:30.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:30.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:30.584968+0000) 2024-03-06T13:19:31.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:31.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:31.585170+0000) 2024-03-06T13:19:32.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:32.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:32.585362+0000) 2024-03-06T13:19:33.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:33.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:33.585552+0000) 2024-03-06T13:19:34.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:34.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:34.585786+0000) 2024-03-06T13:19:35.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:35.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:35.586052+0000) 2024-03-06T13:19:36.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:36.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:36.586260+0000) 2024-03-06T13:19:37.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:37.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:37.586484+0000) 2024-03-06T13:19:38.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:38.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:38.586725+0000) 2024-03-06T13:19:39.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:39.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:39.586948+0000) 2024-03-06T13:19:40.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:40.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:40.587174+0000) 2024-03-06T13:19:41.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:41.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:41.587367+0000) 2024-03-06T13:19:42.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:42.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:42.587571+0000) 2024-03-06T13:19:43.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:43.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:43.587806+0000) 2024-03-06T13:19:44.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:44.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:44.588075+0000) 2024-03-06T13:19:45.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:45.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:45.588266+0000) 2024-03-06T13:19:46.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:46.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:46.588472+0000) 2024-03-06T13:19:47.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:47.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:47.588752+0000) 2024-03-06T13:19:48.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:48.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:48.589049+0000) 2024-03-06T13:19:49.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:49.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:49.589295+0000) 2024-03-06T13:19:50.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:50.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:50.589501+0000) 2024-03-06T13:19:51.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:51.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:51.589725+0000) 2024-03-06T13:19:52.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:52.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:52.589946+0000) 2024-03-06T13:19:53.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:53.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:53.590181+0000) 2024-03-06T13:19:54.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:54.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:54.590422+0000) 2024-03-06T13:19:55.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:55.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:55.590672+0000) 2024-03-06T13:19:56.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:56.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:56.590917+0000) 2024-03-06T13:19:57.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:57.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:57.591162+0000) 2024-03-06T13:19:58.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:58.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:58.591411+0000) 2024-03-06T13:19:59.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:19:59.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:19:59.591660+0000) 2024-03-06T13:19:59.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:19:59.780 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:00.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:00.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:00.591920+0000) 2024-03-06T13:20:01.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:01.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:01.592143+0000) 2024-03-06T13:20:02.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:02.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:02.592359+0000) 2024-03-06T13:20:03.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:03.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:03.592597+0000) 2024-03-06T13:20:04.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:04.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:04.592815+0000) 2024-03-06T13:20:05.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:05.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:05.593124+0000) 2024-03-06T13:20:06.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:06.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:06.593296+0000) 2024-03-06T13:20:07.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:07.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:07.593528+0000) 2024-03-06T13:20:08.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:08.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:08.593755+0000) 2024-03-06T13:20:09.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:09.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:09.593929+0000) 2024-03-06T13:20:10.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:10.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:10.594158+0000) 2024-03-06T13:20:11.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:11.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:11.594348+0000) 2024-03-06T13:20:12.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:12.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:12.594557+0000) 2024-03-06T13:20:13.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:13.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:13.594790+0000) 2024-03-06T13:20:14.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:14.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:14.595033+0000) 2024-03-06T13:20:15.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:15.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:15.595203+0000) 2024-03-06T13:20:16.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:16.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:16.595436+0000) 2024-03-06T13:20:17.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:17.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:17.595672+0000) 2024-03-06T13:20:18.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:18.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:18.595894+0000) 2024-03-06T13:20:19.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:19.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:19.596112+0000) 2024-03-06T13:20:20.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:20.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:20.596322+0000) 2024-03-06T13:20:21.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:21.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:21.596564+0000) 2024-03-06T13:20:22.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:22.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:22.596700+0000) 2024-03-06T13:20:23.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:23.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:23.596917+0000) 2024-03-06T13:20:24.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:24.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:24.597151+0000) 2024-03-06T13:20:25.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:25.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:25.597384+0000) 2024-03-06T13:20:26.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:26.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:26.597627+0000) 2024-03-06T13:20:27.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:27.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:27.597845+0000) 2024-03-06T13:20:28.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:28.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:28.598120+0000) 2024-03-06T13:20:29.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:29.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:29.598340+0000) 2024-03-06T13:20:29.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:29.975 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:20:30.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:30.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:30.598575+0000) 2024-03-06T13:20:31.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:31.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:31.598851+0000) 2024-03-06T13:20:32.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:32.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:32.599115+0000) 2024-03-06T13:20:33.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:33.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:33.599341+0000) 2024-03-06T13:20:34.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:34.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:34.599600+0000) 2024-03-06T13:20:35.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:35.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:35.599821+0000) 2024-03-06T13:20:36.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:36.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:36.600093+0000) 2024-03-06T13:20:37.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:37.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:37.600314+0000) 2024-03-06T13:20:38.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:38.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:38.600540+0000) 2024-03-06T13:20:39.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:39.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:39.600758+0000) 2024-03-06T13:20:40.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:40.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:40.600946+0000) 2024-03-06T13:20:41.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:41.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:41.601176+0000) 2024-03-06T13:20:42.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:42.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:42.601396+0000) 2024-03-06T13:20:43.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:43.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:43.601651+0000) 2024-03-06T13:20:44.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:44.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:44.601881+0000) 2024-03-06T13:20:45.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:45.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:45.602119+0000) 2024-03-06T13:20:46.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:46.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:46.602296+0000) 2024-03-06T13:20:47.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:47.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:47.602518+0000) 2024-03-06T13:20:48.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:48.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:48.602741+0000) 2024-03-06T13:20:49.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:49.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:49.602962+0000) 2024-03-06T13:20:50.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:50.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:50.603194+0000) 2024-03-06T13:20:51.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:51.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:51.603433+0000) 2024-03-06T13:20:52.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:52.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:52.603669+0000) 2024-03-06T13:20:53.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:53.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:53.603898+0000) 2024-03-06T13:20:54.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:54.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:54.604104+0000) 2024-03-06T13:20:55.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:55.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:55.604333+0000) 2024-03-06T13:20:56.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:56.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:56.604546+0000) 2024-03-06T13:20:57.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:57.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:57.604741+0000) 2024-03-06T13:20:58.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:58.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:58.604962+0000) 2024-03-06T13:20:59.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:20:59.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:20:59.605213+0000) 2024-03-06T13:21:00.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:00.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:00.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:00.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:00.605419+0000) 2024-03-06T13:21:01.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:01.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:01.605648+0000) 2024-03-06T13:21:02.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:02.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:02.605873+0000) 2024-03-06T13:21:03.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:03.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:03.606124+0000) 2024-03-06T13:21:04.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:04.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:04.606316+0000) 2024-03-06T13:21:05.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:05.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:05.606523+0000) 2024-03-06T13:21:06.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:06.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:06.606742+0000) 2024-03-06T13:21:07.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:07.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:07.606965+0000) 2024-03-06T13:21:08.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:08.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:08.607172+0000) 2024-03-06T13:21:09.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:09.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:09.607329+0000) 2024-03-06T13:21:10.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:10.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:10.607546+0000) 2024-03-06T13:21:11.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:11.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:11.607756+0000) 2024-03-06T13:21:12.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:12.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:12.607931+0000) 2024-03-06T13:21:13.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:13.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:13.608201+0000) 2024-03-06T13:21:14.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:14.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:14.608418+0000) 2024-03-06T13:21:15.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:15.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:15.608608+0000) 2024-03-06T13:21:16.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:16.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:16.608872+0000) 2024-03-06T13:21:17.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:17.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:17.609122+0000) 2024-03-06T13:21:18.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:18.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:18.609341+0000) 2024-03-06T13:21:19.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:19.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:19.609557+0000) 2024-03-06T13:21:20.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:20.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:20.609791+0000) 2024-03-06T13:21:21.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:21.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:21.610037+0000) 2024-03-06T13:21:22.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:22.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:22.610268+0000) 2024-03-06T13:21:23.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:23.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:23.610494+0000) 2024-03-06T13:21:24.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:24.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:24.610705+0000) 2024-03-06T13:21:25.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:25.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:25.610957+0000) 2024-03-06T13:21:26.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:26.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:26.611193+0000) 2024-03-06T13:21:27.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:27.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:27.611403+0000) 2024-03-06T13:21:28.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:28.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:28.611625+0000) 2024-03-06T13:21:29.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:29.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:29.611853+0000) 2024-03-06T13:21:30.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:30.284 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:21:30.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:30.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:30.612144+0000) 2024-03-06T13:21:31.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:31.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:31.612369+0000) 2024-03-06T13:21:32.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:32.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:32.612601+0000) 2024-03-06T13:21:33.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:33.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:33.612823+0000) 2024-03-06T13:21:34.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:34.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:34.613107+0000) 2024-03-06T13:21:35.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:35.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:35.613345+0000) 2024-03-06T13:21:36.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:36.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:36.613578+0000) 2024-03-06T13:21:37.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:37.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:37.613762+0000) 2024-03-06T13:21:38.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:38.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:38.613998+0000) 2024-03-06T13:21:39.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:39.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:39.614306+0000) 2024-03-06T13:21:40.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:40.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:40.614560+0000) 2024-03-06T13:21:41.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:41.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:41.614788+0000) 2024-03-06T13:21:42.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:42.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:42.615042+0000) 2024-03-06T13:21:43.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:43.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:43.615294+0000) 2024-03-06T13:21:44.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:44.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:44.615480+0000) 2024-03-06T13:21:45.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:45.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:45.615686+0000) 2024-03-06T13:21:46.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:46.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:46.615909+0000) 2024-03-06T13:21:47.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:47.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:47.616142+0000) 2024-03-06T13:21:48.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:48.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:48.616377+0000) 2024-03-06T13:21:49.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:49.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:49.616615+0000) 2024-03-06T13:21:50.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:50.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:50.616861+0000) 2024-03-06T13:21:51.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:51.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:51.617104+0000) 2024-03-06T13:21:52.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:52.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:52.617318+0000) 2024-03-06T13:21:53.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:53.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:53.617549+0000) 2024-03-06T13:21:54.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:54.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:54.617769+0000) 2024-03-06T13:21:55.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:55.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:55.617983+0000) 2024-03-06T13:21:56.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:56.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:56.618272+0000) 2024-03-06T13:21:57.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:57.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:57.618445+0000) 2024-03-06T13:21:58.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:58.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:58.618665+0000) 2024-03-06T13:21:59.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:21:59.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:21:59.618886+0000) 2024-03-06T13:22:00.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:00.505 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:00.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:00.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:00.619086+0000) 2024-03-06T13:22:01.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:01.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:01.619321+0000) 2024-03-06T13:22:02.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:02.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:02.619533+0000) 2024-03-06T13:22:03.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:03.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:03.619761+0000) 2024-03-06T13:22:04.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:04.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:04.619965+0000) 2024-03-06T13:22:05.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:05.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:05.620160+0000) 2024-03-06T13:22:06.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:06.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:06.620396+0000) 2024-03-06T13:22:07.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:07.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:07.620605+0000) 2024-03-06T13:22:08.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:08.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:08.620827+0000) 2024-03-06T13:22:09.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:09.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:09.621077+0000) 2024-03-06T13:22:10.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:10.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:10.621279+0000) 2024-03-06T13:22:11.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:11.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:11.621503+0000) 2024-03-06T13:22:12.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:12.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:12.621736+0000) 2024-03-06T13:22:13.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:13.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:13.621990+0000) 2024-03-06T13:22:14.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:14.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:14.622230+0000) 2024-03-06T13:22:15.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:15.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:15.622421+0000) 2024-03-06T13:22:16.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:16.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:16.622657+0000) 2024-03-06T13:22:17.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:17.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:17.622887+0000) 2024-03-06T13:22:18.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:18.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:18.623129+0000) 2024-03-06T13:22:19.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:19.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:19.623350+0000) 2024-03-06T13:22:20.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:20.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:20.623566+0000) 2024-03-06T13:22:21.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:21.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:21.623807+0000) 2024-03-06T13:22:22.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:22.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:22.624058+0000) 2024-03-06T13:22:23.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:23.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:23.624267+0000) 2024-03-06T13:22:24.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:24.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:24.624491+0000) 2024-03-06T13:22:25.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:25.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:25.624754+0000) 2024-03-06T13:22:26.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:26.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:26.625084+0000) 2024-03-06T13:22:27.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:27.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:27.625326+0000) 2024-03-06T13:22:28.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:28.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:28.625603+0000) 2024-03-06T13:22:29.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:29.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:29.625798+0000) 2024-03-06T13:22:30.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:30.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:30.626061+0000) 2024-03-06T13:22:30.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:30.660 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:22:31.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:31.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:31.626259+0000) 2024-03-06T13:22:32.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:32.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:32.626488+0000) 2024-03-06T13:22:33.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:33.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:33.626725+0000) 2024-03-06T13:22:34.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:34.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:34.626958+0000) 2024-03-06T13:22:35.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:35.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:35.627169+0000) 2024-03-06T13:22:36.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:36.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:36.627362+0000) 2024-03-06T13:22:37.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:37.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:37.627562+0000) 2024-03-06T13:22:38.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:38.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:38.627749+0000) 2024-03-06T13:22:39.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:39.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:39.627969+0000) 2024-03-06T13:22:40.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:40.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:40.628208+0000) 2024-03-06T13:22:41.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:41.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:41.628402+0000) 2024-03-06T13:22:42.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:42.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:42.628622+0000) 2024-03-06T13:22:43.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:43.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:43.628888+0000) 2024-03-06T13:22:44.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:44.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:44.629120+0000) 2024-03-06T13:22:45.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:45.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:45.629361+0000) 2024-03-06T13:22:46.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:46.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:46.629569+0000) 2024-03-06T13:22:47.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:47.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:47.629802+0000) 2024-03-06T13:22:48.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:48.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:48.630088+0000) 2024-03-06T13:22:49.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:49.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:49.630275+0000) 2024-03-06T13:22:50.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:50.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:50.630491+0000) 2024-03-06T13:22:51.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:51.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:51.630707+0000) 2024-03-06T13:22:52.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:52.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:52.630953+0000) 2024-03-06T13:22:53.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:53.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:53.631179+0000) 2024-03-06T13:22:54.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:54.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:54.631411+0000) 2024-03-06T13:22:55.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:55.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:55.631693+0000) 2024-03-06T13:22:56.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:56.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:56.631929+0000) 2024-03-06T13:22:57.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:57.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:57.632201+0000) 2024-03-06T13:22:58.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:58.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:58.632478+0000) 2024-03-06T13:22:59.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:22:59.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:22:59.632759+0000) 2024-03-06T13:23:00.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:00.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:00.633056+0000) 2024-03-06T13:23:00.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:00.872 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:01.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:01.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:01.633339+0000) 2024-03-06T13:23:02.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:02.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:02.633597+0000) 2024-03-06T13:23:03.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:03.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:03.633827+0000) 2024-03-06T13:23:04.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:04.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:04.634117+0000) 2024-03-06T13:23:05.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:05.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:05.634303+0000) 2024-03-06T13:23:06.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:06.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:06.634530+0000) 2024-03-06T13:23:07.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:07.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:07.634733+0000) 2024-03-06T13:23:08.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:08.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:08.634990+0000) 2024-03-06T13:23:09.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:09.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:09.635247+0000) 2024-03-06T13:23:10.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:10.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:10.635469+0000) 2024-03-06T13:23:11.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:11.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:11.635689+0000) 2024-03-06T13:23:12.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:12.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:12.635912+0000) 2024-03-06T13:23:13.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:13.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:13.636157+0000) 2024-03-06T13:23:14.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:14.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:14.636340+0000) 2024-03-06T13:23:15.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:15.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:15.636556+0000) 2024-03-06T13:23:16.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:16.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:16.636789+0000) 2024-03-06T13:23:17.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:17.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:17.637063+0000) 2024-03-06T13:23:18.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:18.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:18.637350+0000) 2024-03-06T13:23:19.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:19.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:19.637616+0000) 2024-03-06T13:23:20.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:20.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:20.637910+0000) 2024-03-06T13:23:21.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:21.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:21.638191+0000) 2024-03-06T13:23:22.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:22.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:22.638478+0000) 2024-03-06T13:23:23.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:23.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:23.638725+0000) 2024-03-06T13:23:24.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:24.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:24.638953+0000) 2024-03-06T13:23:25.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:25.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:25.639192+0000) 2024-03-06T13:23:26.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:26.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:26.639346+0000) 2024-03-06T13:23:27.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:27.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:27.639532+0000) 2024-03-06T13:23:28.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:28.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:28.639757+0000) 2024-03-06T13:23:29.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:29.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:29.640046+0000) 2024-03-06T13:23:30.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:30.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:30.640338+0000) 2024-03-06T13:23:31.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:31.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:23:31.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:31.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:31.640612+0000) 2024-03-06T13:23:32.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:32.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:32.640771+0000) 2024-03-06T13:23:33.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:33.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:33.640992+0000) 2024-03-06T13:23:34.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:34.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:34.641189+0000) 2024-03-06T13:23:35.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:35.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:35.641338+0000) 2024-03-06T13:23:36.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:36.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:36.641562+0000) 2024-03-06T13:23:37.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:37.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:37.641771+0000) 2024-03-06T13:23:38.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:38.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:38.641978+0000) 2024-03-06T13:23:39.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:39.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:39.642231+0000) 2024-03-06T13:23:40.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:40.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:40.642470+0000) 2024-03-06T13:23:41.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:41.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:41.642719+0000) 2024-03-06T13:23:42.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:42.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:42.643001+0000) 2024-03-06T13:23:43.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:43.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:43.643358+0000) 2024-03-06T13:23:44.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:44.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:44.643628+0000) 2024-03-06T13:23:45.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:45.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:45.643914+0000) 2024-03-06T13:23:46.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:46.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:46.644162+0000) 2024-03-06T13:23:47.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:47.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:47.644348+0000) 2024-03-06T13:23:48.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:48.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:48.644612+0000) 2024-03-06T13:23:49.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:49.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:49.644832+0000) 2024-03-06T13:23:50.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:50.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:50.645103+0000) 2024-03-06T13:23:51.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:51.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:51.645327+0000) 2024-03-06T13:23:52.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:52.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:52.645533+0000) 2024-03-06T13:23:53.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:53.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:53.645765+0000) 2024-03-06T13:23:54.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:54.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:54.646051+0000) 2024-03-06T13:23:55.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:55.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:55.646296+0000) 2024-03-06T13:23:56.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:56.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:56.646507+0000) 2024-03-06T13:23:57.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:57.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:57.646712+0000) 2024-03-06T13:23:58.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:58.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:58.646984+0000) 2024-03-06T13:23:59.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:23:59.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:23:59.647313+0000) 2024-03-06T13:24:00.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:00.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:00.647528+0000) 2024-03-06T13:24:01.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:01.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:01.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:01.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:01.647779+0000) 2024-03-06T13:24:02.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:02.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:02.648046+0000) 2024-03-06T13:24:03.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:03.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:03.648255+0000) 2024-03-06T13:24:04.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:04.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:04.648443+0000) 2024-03-06T13:24:05.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:05.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:05.648667+0000) 2024-03-06T13:24:06.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:06.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:06.648907+0000) 2024-03-06T13:24:07.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:07.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:07.649129+0000) 2024-03-06T13:24:08.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:08.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:08.649291+0000) 2024-03-06T13:24:09.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:09.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:09.649494+0000) 2024-03-06T13:24:10.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:10.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:10.649731+0000) 2024-03-06T13:24:11.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:11.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:11.649954+0000) 2024-03-06T13:24:12.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:12.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:12.650158+0000) 2024-03-06T13:24:13.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:13.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:13.650348+0000) 2024-03-06T13:24:14.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:14.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:14.650552+0000) 2024-03-06T13:24:15.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:15.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:15.650792+0000) 2024-03-06T13:24:16.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:16.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:16.651056+0000) 2024-03-06T13:24:17.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:17.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:17.651343+0000) 2024-03-06T13:24:18.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:18.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:18.651596+0000) 2024-03-06T13:24:19.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:19.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:19.651873+0000) 2024-03-06T13:24:20.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:20.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:20.652135+0000) 2024-03-06T13:24:21.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:21.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:21.652310+0000) 2024-03-06T13:24:22.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:22.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:22.652522+0000) 2024-03-06T13:24:23.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:23.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:23.652751+0000) 2024-03-06T13:24:24.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:24.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:24.652972+0000) 2024-03-06T13:24:25.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:25.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:25.653187+0000) 2024-03-06T13:24:26.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:26.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:26.653387+0000) 2024-03-06T13:24:27.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:27.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:27.653598+0000) 2024-03-06T13:24:28.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:28.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:28.653831+0000) 2024-03-06T13:24:29.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:29.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:29.654077+0000) 2024-03-06T13:24:30.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:30.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:30.654282+0000) 2024-03-06T13:24:31.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:31.452 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:24:31.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:31.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:31.654470+0000) 2024-03-06T13:24:32.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:32.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:32.654690+0000) 2024-03-06T13:24:33.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:33.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:33.654915+0000) 2024-03-06T13:24:34.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:34.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:34.655139+0000) 2024-03-06T13:24:35.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:35.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:35.655348+0000) 2024-03-06T13:24:36.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:36.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:36.655636+0000) 2024-03-06T13:24:37.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:37.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:37.655876+0000) 2024-03-06T13:24:38.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:38.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:38.656134+0000) 2024-03-06T13:24:39.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:39.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:39.656355+0000) 2024-03-06T13:24:40.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:40.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:40.656621+0000) 2024-03-06T13:24:41.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:41.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:41.656949+0000) 2024-03-06T13:24:42.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:42.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:42.657144+0000) 2024-03-06T13:24:43.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:43.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:43.657372+0000) 2024-03-06T13:24:44.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:44.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:44.657624+0000) 2024-03-06T13:24:45.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:45.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:45.657881+0000) 2024-03-06T13:24:46.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:46.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:46.658170+0000) 2024-03-06T13:24:47.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:47.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:47.658445+0000) 2024-03-06T13:24:48.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:48.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:48.658715+0000) 2024-03-06T13:24:49.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:49.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:49.658992+0000) 2024-03-06T13:24:50.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:50.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:50.659229+0000) 2024-03-06T13:24:51.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:51.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:51.659451+0000) 2024-03-06T13:24:52.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:52.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:52.659684+0000) 2024-03-06T13:24:53.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:53.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:53.659914+0000) 2024-03-06T13:24:54.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:54.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:54.660157+0000) 2024-03-06T13:24:55.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:55.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:55.660468+0000) 2024-03-06T13:24:56.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:56.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:56.660752+0000) 2024-03-06T13:24:57.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:57.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:57.660952+0000) 2024-03-06T13:24:58.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:58.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:58.661165+0000) 2024-03-06T13:24:59.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:24:59.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:24:59.661400+0000) 2024-03-06T13:25:00.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:00.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:00.661643+0000) 2024-03-06T13:25:01.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:01.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:01.661891+0000) 2024-03-06T13:25:01.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:01.673 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:02.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:02.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:02.662155+0000) 2024-03-06T13:25:03.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:03.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:03.662386+0000) 2024-03-06T13:25:04.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:04.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:04.662614+0000) 2024-03-06T13:25:05.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:05.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:05.662869+0000) 2024-03-06T13:25:06.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:06.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:06.663132+0000) 2024-03-06T13:25:07.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:07.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:07.663323+0000) 2024-03-06T13:25:08.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:08.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:08.663543+0000) 2024-03-06T13:25:09.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:09.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:09.663750+0000) 2024-03-06T13:25:10.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:10.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:10.663946+0000) 2024-03-06T13:25:11.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:11.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:11.664164+0000) 2024-03-06T13:25:12.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:12.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:12.664316+0000) 2024-03-06T13:25:13.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:13.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:13.664500+0000) 2024-03-06T13:25:14.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:14.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:14.664728+0000) 2024-03-06T13:25:15.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:15.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:15.664938+0000) 2024-03-06T13:25:16.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:16.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:16.665217+0000) 2024-03-06T13:25:17.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:17.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:17.665453+0000) 2024-03-06T13:25:18.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:18.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:18.665749+0000) 2024-03-06T13:25:19.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:19.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:19.665988+0000) 2024-03-06T13:25:20.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:20.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:20.666249+0000) 2024-03-06T13:25:21.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:21.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:21.666481+0000) 2024-03-06T13:25:22.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:22.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:22.666707+0000) 2024-03-06T13:25:23.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:23.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:23.666923+0000) 2024-03-06T13:25:24.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:24.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:24.667190+0000) 2024-03-06T13:25:25.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:25.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:25.667369+0000) 2024-03-06T13:25:26.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:26.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:26.667561+0000) 2024-03-06T13:25:27.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:27.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:27.667769+0000) 2024-03-06T13:25:28.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:28.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:28.667960+0000) 2024-03-06T13:25:29.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:29.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:29.668165+0000) 2024-03-06T13:25:30.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:30.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:30.668361+0000) 2024-03-06T13:25:31.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:31.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:31.668581+0000) 2024-03-06T13:25:31.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:31.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:25:32.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:32.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:32.668831+0000) 2024-03-06T13:25:33.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:33.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:33.669111+0000) 2024-03-06T13:25:34.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:34.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:34.669319+0000) 2024-03-06T13:25:35.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:35.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:35.669541+0000) 2024-03-06T13:25:36.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:36.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:36.669776+0000) 2024-03-06T13:25:37.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:37.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:37.670076+0000) 2024-03-06T13:25:38.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:38.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:38.670256+0000) 2024-03-06T13:25:39.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:39.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:39.670461+0000) 2024-03-06T13:25:40.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:40.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:40.670706+0000) 2024-03-06T13:25:41.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:41.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:41.670943+0000) 2024-03-06T13:25:42.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:42.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:42.671161+0000) 2024-03-06T13:25:43.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:43.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:43.671408+0000) 2024-03-06T13:25:44.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:44.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:44.671652+0000) 2024-03-06T13:25:45.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:45.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:45.671860+0000) 2024-03-06T13:25:46.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:46.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:46.672153+0000) 2024-03-06T13:25:47.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:47.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:47.672355+0000) 2024-03-06T13:25:48.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:48.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:48.672554+0000) 2024-03-06T13:25:49.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:49.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:49.672787+0000) 2024-03-06T13:25:50.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:50.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:50.673046+0000) 2024-03-06T13:25:51.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:51.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:51.673278+0000) 2024-03-06T13:25:52.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:52.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:52.673491+0000) 2024-03-06T13:25:53.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:53.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:53.673686+0000) 2024-03-06T13:25:54.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:54.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:54.673937+0000) 2024-03-06T13:25:55.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:55.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:55.674152+0000) 2024-03-06T13:25:56.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:56.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:56.674421+0000) 2024-03-06T13:25:57.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:57.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:57.674695+0000) 2024-03-06T13:25:58.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:58.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:58.674966+0000) 2024-03-06T13:25:59.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:25:59.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:25:59.675245+0000) 2024-03-06T13:26:00.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:00.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:00.675500+0000) 2024-03-06T13:26:01.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:01.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:01.675762+0000) 2024-03-06T13:26:01.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:01.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:02.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:02.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:02.676075+0000) 2024-03-06T13:26:03.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:03.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:03.676298+0000) 2024-03-06T13:26:04.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:04.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:04.676537+0000) 2024-03-06T13:26:05.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:05.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:05.676768+0000) 2024-03-06T13:26:06.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:06.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:06.676987+0000) 2024-03-06T13:26:07.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:07.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:07.677244+0000) 2024-03-06T13:26:08.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:08.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:08.677471+0000) 2024-03-06T13:26:09.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:09.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:09.677612+0000) 2024-03-06T13:26:10.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:10.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:10.677797+0000) 2024-03-06T13:26:11.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:11.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:11.678057+0000) 2024-03-06T13:26:12.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:12.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:12.678377+0000) 2024-03-06T13:26:13.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:13.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:13.678667+0000) 2024-03-06T13:26:14.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:14.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:14.678949+0000) 2024-03-06T13:26:15.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:15.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:15.679225+0000) 2024-03-06T13:26:16.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:16.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:16.679477+0000) 2024-03-06T13:26:17.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:17.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:17.679744+0000) 2024-03-06T13:26:18.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:18.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:18.679959+0000) 2024-03-06T13:26:19.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:19.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:19.680130+0000) 2024-03-06T13:26:20.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:20.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:20.680336+0000) 2024-03-06T13:26:21.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:21.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:21.680557+0000) 2024-03-06T13:26:22.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:22.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:22.680780+0000) 2024-03-06T13:26:23.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:23.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:23.681031+0000) 2024-03-06T13:26:24.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:24.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:24.681224+0000) 2024-03-06T13:26:25.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:25.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:25.681438+0000) 2024-03-06T13:26:26.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:26.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:26.681703+0000) 2024-03-06T13:26:27.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:27.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:27.681917+0000) 2024-03-06T13:26:28.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:28.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:28.682155+0000) 2024-03-06T13:26:29.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:29.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:29.682378+0000) 2024-03-06T13:26:30.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:30.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:30.682617+0000) 2024-03-06T13:26:31.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:31.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:31.682871+0000) 2024-03-06T13:26:32.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:32.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:26:32.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:32.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:32.683125+0000) 2024-03-06T13:26:33.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:33.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:33.683320+0000) 2024-03-06T13:26:34.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:34.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:34.683554+0000) 2024-03-06T13:26:35.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:35.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:35.683736+0000) 2024-03-06T13:26:36.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:36.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:36.683970+0000) 2024-03-06T13:26:37.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:37.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:37.684218+0000) 2024-03-06T13:26:38.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:38.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:38.684442+0000) 2024-03-06T13:26:39.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:39.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:39.684676+0000) 2024-03-06T13:26:40.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:40.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:40.684914+0000) 2024-03-06T13:26:41.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:41.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:41.685153+0000) 2024-03-06T13:26:42.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:42.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:42.685346+0000) 2024-03-06T13:26:43.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:43.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:43.685560+0000) 2024-03-06T13:26:44.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:44.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:44.685800+0000) 2024-03-06T13:26:45.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:45.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:45.686047+0000) 2024-03-06T13:26:46.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:46.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:46.686249+0000) 2024-03-06T13:26:47.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:47.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:47.686468+0000) 2024-03-06T13:26:48.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:48.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:48.686712+0000) 2024-03-06T13:26:49.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:49.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:49.686947+0000) 2024-03-06T13:26:50.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:50.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:50.687158+0000) 2024-03-06T13:26:51.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:51.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:51.687363+0000) 2024-03-06T13:26:52.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:52.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:52.687622+0000) 2024-03-06T13:26:53.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:53.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:53.687905+0000) 2024-03-06T13:26:54.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:54.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:54.688181+0000) 2024-03-06T13:26:55.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:55.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:55.688421+0000) 2024-03-06T13:26:56.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:56.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:56.688696+0000) 2024-03-06T13:26:57.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:57.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:57.688962+0000) 2024-03-06T13:26:58.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:58.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:58.689240+0000) 2024-03-06T13:26:59.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:26:59.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:26:59.689501+0000) 2024-03-06T13:27:00.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:00.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:00.689708+0000) 2024-03-06T13:27:01.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:01.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:01.689937+0000) 2024-03-06T13:27:02.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:02.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:02.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:02.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:02.690224+0000) 2024-03-06T13:27:03.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:03.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:03.690516+0000) 2024-03-06T13:27:04.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:04.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:04.690778+0000) 2024-03-06T13:27:05.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:05.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:05.691043+0000) 2024-03-06T13:27:06.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:06.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:06.691313+0000) 2024-03-06T13:27:07.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:07.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:07.691525+0000) 2024-03-06T13:27:08.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:08.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:08.691778+0000) 2024-03-06T13:27:09.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:09.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:09.692082+0000) 2024-03-06T13:27:10.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:10.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:10.692330+0000) 2024-03-06T13:27:11.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:11.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:11.692608+0000) 2024-03-06T13:27:12.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:12.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:12.692882+0000) 2024-03-06T13:27:13.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:13.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:13.693129+0000) 2024-03-06T13:27:14.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:14.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:14.693290+0000) 2024-03-06T13:27:15.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:15.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:15.693493+0000) 2024-03-06T13:27:16.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:16.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:16.693737+0000) 2024-03-06T13:27:17.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:17.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:17.693938+0000) 2024-03-06T13:27:18.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:18.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:18.694209+0000) 2024-03-06T13:27:19.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:19.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:19.694444+0000) 2024-03-06T13:27:20.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:20.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:20.694716+0000) 2024-03-06T13:27:21.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:21.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:21.694955+0000) 2024-03-06T13:27:22.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:22.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:22.695214+0000) 2024-03-06T13:27:23.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:23.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:23.695482+0000) 2024-03-06T13:27:24.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:24.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:24.695757+0000) 2024-03-06T13:27:25.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:25.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:25.696057+0000) 2024-03-06T13:27:26.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:26.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:26.696259+0000) 2024-03-06T13:27:27.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:27.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:27.696480+0000) 2024-03-06T13:27:28.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:28.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:28.696760+0000) 2024-03-06T13:27:29.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:29.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:29.697066+0000) 2024-03-06T13:27:30.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:30.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:30.697297+0000) 2024-03-06T13:27:31.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:31.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:31.697549+0000) 2024-03-06T13:27:32.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:32.571 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:27:32.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:32.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:32.697836+0000) 2024-03-06T13:27:33.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:33.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:33.698115+0000) 2024-03-06T13:27:34.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:34.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:34.698313+0000) 2024-03-06T13:27:35.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:35.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:35.698523+0000) 2024-03-06T13:27:36.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:36.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:36.698797+0000) 2024-03-06T13:27:37.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:37.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:37.699099+0000) 2024-03-06T13:27:38.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:38.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:38.699345+0000) 2024-03-06T13:27:39.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:39.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:39.699612+0000) 2024-03-06T13:27:40.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:40.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:40.699895+0000) 2024-03-06T13:27:41.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:41.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:41.700145+0000) 2024-03-06T13:27:42.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:42.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:42.700359+0000) 2024-03-06T13:27:43.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:43.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:43.700585+0000) 2024-03-06T13:27:44.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:44.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:44.700820+0000) 2024-03-06T13:27:45.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:45.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:45.701131+0000) 2024-03-06T13:27:46.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:46.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:46.701408+0000) 2024-03-06T13:27:47.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:47.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:47.701643+0000) 2024-03-06T13:27:48.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:48.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:48.701904+0000) 2024-03-06T13:27:49.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:49.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:49.702154+0000) 2024-03-06T13:27:50.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:50.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:50.702323+0000) 2024-03-06T13:27:51.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:51.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:51.702535+0000) 2024-03-06T13:27:52.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:52.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:52.702759+0000) 2024-03-06T13:27:53.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:53.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:53.702992+0000) 2024-03-06T13:27:54.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:54.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:54.703211+0000) 2024-03-06T13:27:55.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:55.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:55.703432+0000) 2024-03-06T13:27:56.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:56.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:56.703643+0000) 2024-03-06T13:27:57.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:57.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:57.703864+0000) 2024-03-06T13:27:58.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:58.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:58.704123+0000) 2024-03-06T13:27:59.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:27:59.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:27:59.704380+0000) 2024-03-06T13:28:00.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:00.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:00.704649+0000) 2024-03-06T13:28:01.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:01.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:01.704932+0000) 2024-03-06T13:28:02.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:02.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:02.705213+0000) 2024-03-06T13:28:02.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:02.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:03.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:03.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:03.705469+0000) 2024-03-06T13:28:04.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:04.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:04.705676+0000) 2024-03-06T13:28:05.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:05.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:05.705907+0000) 2024-03-06T13:28:06.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:06.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:06.706189+0000) 2024-03-06T13:28:07.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:07.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:07.706467+0000) 2024-03-06T13:28:08.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:08.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:08.706750+0000) 2024-03-06T13:28:09.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:09.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:09.707045+0000) 2024-03-06T13:28:10.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:10.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:10.707325+0000) 2024-03-06T13:28:11.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:11.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:11.707608+0000) 2024-03-06T13:28:12.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:12.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:12.707814+0000) 2024-03-06T13:28:13.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:13.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:13.708107+0000) 2024-03-06T13:28:14.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:14.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:14.708307+0000) 2024-03-06T13:28:15.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:15.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:15.708563+0000) 2024-03-06T13:28:16.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:16.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:16.708847+0000) 2024-03-06T13:28:17.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:17.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:17.709126+0000) 2024-03-06T13:28:18.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:18.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:18.709364+0000) 2024-03-06T13:28:19.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:19.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:19.709545+0000) 2024-03-06T13:28:20.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:20.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:20.709790+0000) 2024-03-06T13:28:21.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:21.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:21.710078+0000) 2024-03-06T13:28:22.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:22.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:22.710301+0000) 2024-03-06T13:28:23.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:23.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:23.710505+0000) 2024-03-06T13:28:24.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:24.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:24.710737+0000) 2024-03-06T13:28:25.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:25.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:25.710939+0000) 2024-03-06T13:28:26.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:26.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:26.711169+0000) 2024-03-06T13:28:27.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:27.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:27.711404+0000) 2024-03-06T13:28:28.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:28.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:28.711654+0000) 2024-03-06T13:28:29.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:29.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:29.711941+0000) 2024-03-06T13:28:30.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:30.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:30.712182+0000) 2024-03-06T13:28:31.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:31.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:31.712338+0000) 2024-03-06T13:28:32.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:32.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:32.712559+0000) 2024-03-06T13:28:32.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:33.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:28:33.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:33.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:33.712828+0000) 2024-03-06T13:28:34.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:34.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:34.713047+0000) 2024-03-06T13:28:35.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:35.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:35.713289+0000) 2024-03-06T13:28:36.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:36.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:36.713511+0000) 2024-03-06T13:28:37.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:37.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:37.713750+0000) 2024-03-06T13:28:38.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:38.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:38.713988+0000) 2024-03-06T13:28:39.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:39.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:39.714233+0000) 2024-03-06T13:28:40.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:40.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:40.714455+0000) 2024-03-06T13:28:41.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:41.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:41.714670+0000) 2024-03-06T13:28:42.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:42.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:42.714900+0000) 2024-03-06T13:28:43.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:43.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:43.715132+0000) 2024-03-06T13:28:44.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:44.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:44.715369+0000) 2024-03-06T13:28:45.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:45.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:45.715597+0000) 2024-03-06T13:28:46.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:46.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:46.715851+0000) 2024-03-06T13:28:47.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:47.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:47.716217+0000) 2024-03-06T13:28:48.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:48.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:48.716459+0000) 2024-03-06T13:28:49.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:49.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:49.716703+0000) 2024-03-06T13:28:50.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:50.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:50.716946+0000) 2024-03-06T13:28:51.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:51.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:51.717177+0000) 2024-03-06T13:28:52.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:52.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:52.717424+0000) 2024-03-06T13:28:53.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:53.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:53.717654+0000) 2024-03-06T13:28:54.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:54.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:54.717894+0000) 2024-03-06T13:28:55.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:55.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:55.718127+0000) 2024-03-06T13:28:56.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:56.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:56.718347+0000) 2024-03-06T13:28:57.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:57.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:57.718574+0000) 2024-03-06T13:28:58.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:58.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:58.718809+0000) 2024-03-06T13:28:59.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:28:59.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:28:59.719085+0000) 2024-03-06T13:29:00.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:00.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:00.719343+0000) 2024-03-06T13:29:01.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:01.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:01.719612+0000) 2024-03-06T13:29:02.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:02.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:02.719820+0000) 2024-03-06T13:29:03.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:03.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:03.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:03.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:03.720112+0000) 2024-03-06T13:29:04.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:04.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:04.720355+0000) 2024-03-06T13:29:05.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:05.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:05.720580+0000) 2024-03-06T13:29:06.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:06.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:06.720831+0000) 2024-03-06T13:29:07.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:07.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:07.721111+0000) 2024-03-06T13:29:08.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:08.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:08.721326+0000) 2024-03-06T13:29:09.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:09.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:09.721556+0000) 2024-03-06T13:29:10.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:10.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:10.721802+0000) 2024-03-06T13:29:11.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:11.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:11.722086+0000) 2024-03-06T13:29:12.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:12.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:12.722292+0000) 2024-03-06T13:29:13.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:13.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:13.722533+0000) 2024-03-06T13:29:14.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:14.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:14.722781+0000) 2024-03-06T13:29:15.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:15.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:15.723032+0000) 2024-03-06T13:29:16.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:16.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:16.723268+0000) 2024-03-06T13:29:17.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:17.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:17.723481+0000) 2024-03-06T13:29:18.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:18.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:18.723726+0000) 2024-03-06T13:29:19.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:19.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:19.723923+0000) 2024-03-06T13:29:20.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:20.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:20.724149+0000) 2024-03-06T13:29:21.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:21.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:21.724372+0000) 2024-03-06T13:29:22.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:22.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:22.724576+0000) 2024-03-06T13:29:23.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:23.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:23.724808+0000) 2024-03-06T13:29:24.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:24.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:24.725062+0000) 2024-03-06T13:29:25.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:25.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:25.725227+0000) 2024-03-06T13:29:26.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:26.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:26.725437+0000) 2024-03-06T13:29:27.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:27.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:27.725645+0000) 2024-03-06T13:29:28.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:28.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:28.725874+0000) 2024-03-06T13:29:29.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:29.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:29.726139+0000) 2024-03-06T13:29:30.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:30.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:30.726386+0000) 2024-03-06T13:29:31.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:31.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:31.726688+0000) 2024-03-06T13:29:32.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:32.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:32.726937+0000) 2024-03-06T13:29:33.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:33.358 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:29:33.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:33.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:33.727219+0000) 2024-03-06T13:29:34.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:34.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:34.727438+0000) 2024-03-06T13:29:35.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:35.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:35.727667+0000) 2024-03-06T13:29:36.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:36.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:36.727917+0000) 2024-03-06T13:29:37.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:37.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:37.728157+0000) 2024-03-06T13:29:38.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:38.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:38.728401+0000) 2024-03-06T13:29:39.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:39.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:39.728641+0000) 2024-03-06T13:29:40.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:40.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:40.728956+0000) 2024-03-06T13:29:41.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:41.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:41.729216+0000) 2024-03-06T13:29:42.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:42.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:42.729455+0000) 2024-03-06T13:29:43.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:43.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:43.729665+0000) 2024-03-06T13:29:44.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:44.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:44.729892+0000) 2024-03-06T13:29:45.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:45.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:45.730109+0000) 2024-03-06T13:29:46.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:46.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:46.730310+0000) 2024-03-06T13:29:47.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:47.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:47.730531+0000) 2024-03-06T13:29:48.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:48.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:48.730758+0000) 2024-03-06T13:29:49.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:49.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:49.730940+0000) 2024-03-06T13:29:50.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:50.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:50.731306+0000) 2024-03-06T13:29:51.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:51.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:51.731565+0000) 2024-03-06T13:29:52.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:52.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:52.731776+0000) 2024-03-06T13:29:53.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:53.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:53.731939+0000) 2024-03-06T13:29:54.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:54.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:54.732153+0000) 2024-03-06T13:29:55.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:55.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:55.732379+0000) 2024-03-06T13:29:56.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:56.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:56.732569+0000) 2024-03-06T13:29:57.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:57.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:57.733189+0000) 2024-03-06T13:29:58.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:58.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:58.733420+0000) 2024-03-06T13:29:59.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:29:59.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:29:59.733615+0000) 2024-03-06T13:30:00.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:00.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:00.733844+0000) 2024-03-06T13:30:01.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:01.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:01.734136+0000) 2024-03-06T13:30:02.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:02.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:02.734321+0000) 2024-03-06T13:30:03.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:03.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:03.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:03.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:03.734546+0000) 2024-03-06T13:30:04.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:04.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:04.734779+0000) 2024-03-06T13:30:05.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:05.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:05.734994+0000) 2024-03-06T13:30:06.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:06.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:06.735226+0000) 2024-03-06T13:30:07.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:07.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:07.735417+0000) 2024-03-06T13:30:08.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:08.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:08.735821+0000) 2024-03-06T13:30:09.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:09.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:09.736128+0000) 2024-03-06T13:30:10.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:10.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:10.736402+0000) 2024-03-06T13:30:11.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:11.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:11.736657+0000) 2024-03-06T13:30:12.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:12.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:12.736934+0000) 2024-03-06T13:30:13.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:13.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:13.737231+0000) 2024-03-06T13:30:14.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:14.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:14.737513+0000) 2024-03-06T13:30:15.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:15.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:15.737797+0000) 2024-03-06T13:30:16.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:16.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:16.738131+0000) 2024-03-06T13:30:17.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:17.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:17.738373+0000) 2024-03-06T13:30:18.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:18.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:18.738611+0000) 2024-03-06T13:30:19.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:19.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:19.738834+0000) 2024-03-06T13:30:20.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:20.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:20.739129+0000) 2024-03-06T13:30:21.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:21.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:21.739336+0000) 2024-03-06T13:30:22.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:22.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:22.739542+0000) 2024-03-06T13:30:23.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:23.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:23.739849+0000) 2024-03-06T13:30:24.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:24.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:24.740142+0000) 2024-03-06T13:30:25.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:25.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:25.740358+0000) 2024-03-06T13:30:26.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:26.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:26.740578+0000) 2024-03-06T13:30:27.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:27.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:27.740803+0000) 2024-03-06T13:30:28.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:28.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:28.741059+0000) 2024-03-06T13:30:29.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:29.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:29.741241+0000) 2024-03-06T13:30:30.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:30.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:30.741410+0000) 2024-03-06T13:30:31.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:31.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:31.741628+0000) 2024-03-06T13:30:32.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:32.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:32.741862+0000) 2024-03-06T13:30:33.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:33.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:30:33.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:33.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:33.742077+0000) 2024-03-06T13:30:34.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:34.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:34.742305+0000) 2024-03-06T13:30:35.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:35.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:35.742556+0000) 2024-03-06T13:30:36.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:36.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:36.742801+0000) 2024-03-06T13:30:37.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:37.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:37.743055+0000) 2024-03-06T13:30:38.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:38.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:38.743282+0000) 2024-03-06T13:30:39.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:39.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:39.743508+0000) 2024-03-06T13:30:40.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:40.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:40.743727+0000) 2024-03-06T13:30:41.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:41.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:41.743961+0000) 2024-03-06T13:30:42.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:42.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:42.744257+0000) 2024-03-06T13:30:43.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:43.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:43.744496+0000) 2024-03-06T13:30:44.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:44.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:44.744707+0000) 2024-03-06T13:30:45.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:45.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:45.744928+0000) 2024-03-06T13:30:46.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:46.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:46.745162+0000) 2024-03-06T13:30:47.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:47.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:47.745355+0000) 2024-03-06T13:30:48.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:48.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:48.745558+0000) 2024-03-06T13:30:49.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:49.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:49.745731+0000) 2024-03-06T13:30:50.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:50.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:50.745954+0000) 2024-03-06T13:30:51.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:51.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:51.746127+0000) 2024-03-06T13:30:52.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:52.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:52.746360+0000) 2024-03-06T13:30:53.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:53.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:53.746565+0000) 2024-03-06T13:30:54.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:54.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:54.746763+0000) 2024-03-06T13:30:55.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:55.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:55.747067+0000) 2024-03-06T13:30:56.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:56.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:56.747315+0000) 2024-03-06T13:30:57.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:57.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:57.747531+0000) 2024-03-06T13:30:58.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:58.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:58.747751+0000) 2024-03-06T13:30:59.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:30:59.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:30:59.747927+0000) 2024-03-06T13:31:00.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:00.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:00.748167+0000) 2024-03-06T13:31:01.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:01.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:01.748393+0000) 2024-03-06T13:31:02.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:02.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:02.748617+0000) 2024-03-06T13:31:03.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:03.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:03.748844+0000) 2024-03-06T13:31:03.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:03.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:04.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:04.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:04.749076+0000) 2024-03-06T13:31:05.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:05.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:05.749306+0000) 2024-03-06T13:31:06.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:06.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:06.749543+0000) 2024-03-06T13:31:07.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:07.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:07.749768+0000) 2024-03-06T13:31:08.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:08.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:08.750043+0000) 2024-03-06T13:31:09.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:09.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:09.750273+0000) 2024-03-06T13:31:10.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:10.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:10.750499+0000) 2024-03-06T13:31:11.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:11.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:11.750737+0000) 2024-03-06T13:31:12.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:12.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:12.750973+0000) 2024-03-06T13:31:13.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:13.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:13.751255+0000) 2024-03-06T13:31:14.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:14.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:14.751482+0000) 2024-03-06T13:31:15.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:15.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:15.751695+0000) 2024-03-06T13:31:16.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:16.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:16.751960+0000) 2024-03-06T13:31:17.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:17.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:17.752130+0000) 2024-03-06T13:31:18.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:18.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:18.752305+0000) 2024-03-06T13:31:19.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:19.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:19.752531+0000) 2024-03-06T13:31:20.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:20.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:20.752763+0000) 2024-03-06T13:31:21.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:21.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:21.752985+0000) 2024-03-06T13:31:22.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:22.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:22.753269+0000) 2024-03-06T13:31:23.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:23.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:23.753483+0000) 2024-03-06T13:31:24.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:24.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:24.753711+0000) 2024-03-06T13:31:25.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:25.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:25.753939+0000) 2024-03-06T13:31:26.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:26.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:26.754163+0000) 2024-03-06T13:31:27.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:27.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:27.754429+0000) 2024-03-06T13:31:28.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:28.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:28.754664+0000) 2024-03-06T13:31:29.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:29.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:29.754901+0000) 2024-03-06T13:31:30.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:30.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:30.755159+0000) 2024-03-06T13:31:31.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:31.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:31.755398+0000) 2024-03-06T13:31:32.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:32.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:32.755625+0000) 2024-03-06T13:31:33.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:33.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:33.755863+0000) 2024-03-06T13:31:34.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:34.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:31:34.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:34.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:34.756147+0000) 2024-03-06T13:31:35.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:35.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:35.756394+0000) 2024-03-06T13:31:36.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:36.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:36.756577+0000) 2024-03-06T13:31:37.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:37.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:37.756765+0000) 2024-03-06T13:31:38.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:38.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:38.756915+0000) 2024-03-06T13:31:39.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:39.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:39.757144+0000) 2024-03-06T13:31:40.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:40.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:40.757403+0000) 2024-03-06T13:31:41.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:41.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:41.757645+0000) 2024-03-06T13:31:42.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:42.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:42.757839+0000) 2024-03-06T13:31:43.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:43.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:43.758111+0000) 2024-03-06T13:31:44.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:44.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:44.758337+0000) 2024-03-06T13:31:45.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:45.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:45.758573+0000) 2024-03-06T13:31:46.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:46.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:46.758795+0000) 2024-03-06T13:31:47.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:47.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:47.758992+0000) 2024-03-06T13:31:48.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:48.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:48.759265+0000) 2024-03-06T13:31:49.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:49.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:49.759479+0000) 2024-03-06T13:31:50.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:50.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:50.759683+0000) 2024-03-06T13:31:51.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:51.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:51.759907+0000) 2024-03-06T13:31:52.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:52.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:52.760142+0000) 2024-03-06T13:31:53.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:53.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:53.760389+0000) 2024-03-06T13:31:54.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:54.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:54.760629+0000) 2024-03-06T13:31:55.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:55.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:55.760867+0000) 2024-03-06T13:31:56.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:56.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:56.761111+0000) 2024-03-06T13:31:57.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:57.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:57.761313+0000) 2024-03-06T13:31:58.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:58.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:58.761543+0000) 2024-03-06T13:31:59.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:31:59.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:31:59.761740+0000) 2024-03-06T13:32:00.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:00.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:00.761979+0000) 2024-03-06T13:32:01.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:01.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:01.762289+0000) 2024-03-06T13:32:02.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:02.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:02.762539+0000) 2024-03-06T13:32:03.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:03.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:03.762800+0000) 2024-03-06T13:32:04.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:04.291 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:04.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:04.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:04.763067+0000) 2024-03-06T13:32:05.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:05.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:05.763304+0000) 2024-03-06T13:32:06.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:06.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:06.763522+0000) 2024-03-06T13:32:07.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:07.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:07.763745+0000) 2024-03-06T13:32:08.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:08.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:08.763962+0000) 2024-03-06T13:32:09.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:09.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:09.764304+0000) 2024-03-06T13:32:10.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:10.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:10.764534+0000) 2024-03-06T13:32:11.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:11.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:11.764785+0000) 2024-03-06T13:32:12.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:12.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:12.765066+0000) 2024-03-06T13:32:13.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:13.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:13.765314+0000) 2024-03-06T13:32:14.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:14.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:14.765554+0000) 2024-03-06T13:32:15.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:15.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:15.765788+0000) 2024-03-06T13:32:16.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:16.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:16.766069+0000) 2024-03-06T13:32:17.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:17.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:17.766310+0000) 2024-03-06T13:32:18.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:18.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:18.766550+0000) 2024-03-06T13:32:19.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:19.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:19.766791+0000) 2024-03-06T13:32:20.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:20.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:20.767039+0000) 2024-03-06T13:32:21.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:21.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:21.767278+0000) 2024-03-06T13:32:22.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:22.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:22.767526+0000) 2024-03-06T13:32:23.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:23.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:23.767687+0000) 2024-03-06T13:32:24.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:24.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:24.767884+0000) 2024-03-06T13:32:25.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:25.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:25.768101+0000) 2024-03-06T13:32:26.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:26.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:26.768314+0000) 2024-03-06T13:32:27.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:27.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:27.768534+0000) 2024-03-06T13:32:28.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:28.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:28.768765+0000) 2024-03-06T13:32:29.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:29.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:29.768988+0000) 2024-03-06T13:32:30.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:30.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:30.769254+0000) 2024-03-06T13:32:31.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:31.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:31.769478+0000) 2024-03-06T13:32:32.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:32.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:32.769708+0000) 2024-03-06T13:32:33.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:33.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:33.769945+0000) 2024-03-06T13:32:34.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:34.512 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:32:34.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:34.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:34.770187+0000) 2024-03-06T13:32:35.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:35.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:35.770403+0000) 2024-03-06T13:32:36.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:36.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:36.770660+0000) 2024-03-06T13:32:37.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:37.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:37.770890+0000) 2024-03-06T13:32:38.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:38.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:38.771123+0000) 2024-03-06T13:32:39.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:39.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:39.771347+0000) 2024-03-06T13:32:40.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:40.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:40.771575+0000) 2024-03-06T13:32:41.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:41.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:41.771803+0000) 2024-03-06T13:32:42.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:42.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:42.772090+0000) 2024-03-06T13:32:43.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:43.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:43.772364+0000) 2024-03-06T13:32:44.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:44.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:44.772592+0000) 2024-03-06T13:32:45.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:45.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:45.772812+0000) 2024-03-06T13:32:46.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:46.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:46.773061+0000) 2024-03-06T13:32:47.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:47.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:47.773241+0000) 2024-03-06T13:32:48.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:48.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:48.773464+0000) 2024-03-06T13:32:49.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:49.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:49.773680+0000) 2024-03-06T13:32:50.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:50.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:50.773908+0000) 2024-03-06T13:32:51.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:51.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:51.774149+0000) 2024-03-06T13:32:52.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:52.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:52.774399+0000) 2024-03-06T13:32:53.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:53.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:53.774646+0000) 2024-03-06T13:32:54.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:54.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:54.774881+0000) 2024-03-06T13:32:55.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:55.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:55.775124+0000) 2024-03-06T13:32:56.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:56.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:56.775359+0000) 2024-03-06T13:32:57.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:57.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:57.775640+0000) 2024-03-06T13:32:58.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:58.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:58.775855+0000) 2024-03-06T13:32:59.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:32:59.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:32:59.776116+0000) 2024-03-06T13:33:00.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:00.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:00.776344+0000) 2024-03-06T13:33:01.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:01.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:01.776586+0000) 2024-03-06T13:33:02.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:02.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:02.776822+0000) 2024-03-06T13:33:03.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:03.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:03.777103+0000) 2024-03-06T13:33:04.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:04.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:04.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:04.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:04.777274+0000) 2024-03-06T13:33:05.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:05.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:05.777488+0000) 2024-03-06T13:33:06.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:06.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:06.777712+0000) 2024-03-06T13:33:07.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:07.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:07.777943+0000) 2024-03-06T13:33:08.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:08.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:08.778183+0000) 2024-03-06T13:33:09.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:09.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:09.778413+0000) 2024-03-06T13:33:10.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:10.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:10.778659+0000) 2024-03-06T13:33:11.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:11.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:11.778921+0000) 2024-03-06T13:33:12.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:12.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:12.779173+0000) 2024-03-06T13:33:13.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:13.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:13.779425+0000) 2024-03-06T13:33:14.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:14.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:14.779654+0000) 2024-03-06T13:33:15.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:15.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:15.779902+0000) 2024-03-06T13:33:16.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:16.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:16.780151+0000) 2024-03-06T13:33:17.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:17.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:17.780395+0000) 2024-03-06T13:33:18.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:18.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:18.780615+0000) 2024-03-06T13:33:19.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:19.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:19.780846+0000) 2024-03-06T13:33:20.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:20.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:20.781111+0000) 2024-03-06T13:33:21.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:21.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:21.781326+0000) 2024-03-06T13:33:22.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:22.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:22.781525+0000) 2024-03-06T13:33:23.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:23.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:23.781714+0000) 2024-03-06T13:33:24.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:24.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:24.781944+0000) 2024-03-06T13:33:25.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:25.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:25.782169+0000) 2024-03-06T13:33:26.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:26.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:26.782360+0000) 2024-03-06T13:33:27.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:27.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:27.782598+0000) 2024-03-06T13:33:28.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:28.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:28.782850+0000) 2024-03-06T13:33:29.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:29.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:29.783103+0000) 2024-03-06T13:33:30.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:30.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:30.783288+0000) 2024-03-06T13:33:31.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:31.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:31.783464+0000) 2024-03-06T13:33:32.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:32.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:32.783655+0000) 2024-03-06T13:33:33.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:33.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:33.783904+0000) 2024-03-06T13:33:34.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:34.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:34.784129+0000) 2024-03-06T13:33:34.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:34.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:33:35.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:35.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:35.784363+0000) 2024-03-06T13:33:36.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:36.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:36.784591+0000) 2024-03-06T13:33:37.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:37.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:37.784816+0000) 2024-03-06T13:33:38.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:38.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:38.785082+0000) 2024-03-06T13:33:39.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:39.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:39.785277+0000) 2024-03-06T13:33:40.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:40.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:40.785481+0000) 2024-03-06T13:33:41.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:41.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:41.785692+0000) 2024-03-06T13:33:42.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:42.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:42.785889+0000) 2024-03-06T13:33:43.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:43.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:43.786131+0000) 2024-03-06T13:33:44.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:44.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:44.786327+0000) 2024-03-06T13:33:45.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:45.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:45.786600+0000) 2024-03-06T13:33:46.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:46.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:46.786824+0000) 2024-03-06T13:33:47.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:47.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:47.787098+0000) 2024-03-06T13:33:48.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:48.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:48.787359+0000) 2024-03-06T13:33:49.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:49.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:49.787641+0000) 2024-03-06T13:33:50.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:50.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:50.787908+0000) 2024-03-06T13:33:51.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:51.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:51.788180+0000) 2024-03-06T13:33:52.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:52.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:52.788460+0000) 2024-03-06T13:33:53.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:53.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:53.788704+0000) 2024-03-06T13:33:54.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:54.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:54.788986+0000) 2024-03-06T13:33:55.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:55.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:55.789292+0000) 2024-03-06T13:33:56.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:56.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:56.789555+0000) 2024-03-06T13:33:57.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:57.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:57.789822+0000) 2024-03-06T13:33:58.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:58.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:58.790063+0000) 2024-03-06T13:33:59.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:33:59.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:33:59.790298+0000) 2024-03-06T13:34:00.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:00.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:00.790525+0000) 2024-03-06T13:34:01.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:01.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:01.790759+0000) 2024-03-06T13:34:02.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:02.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:02.790971+0000) 2024-03-06T13:34:03.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:03.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:03.791211+0000) 2024-03-06T13:34:04.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:04.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:04.791393+0000) 2024-03-06T13:34:05.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:05.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:05.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:05.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:05.791622+0000) 2024-03-06T13:34:06.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:06.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:06.791846+0000) 2024-03-06T13:34:07.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:07.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:07.792115+0000) 2024-03-06T13:34:08.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:08.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:08.792337+0000) 2024-03-06T13:34:09.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:09.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:09.792571+0000) 2024-03-06T13:34:10.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:10.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:10.792814+0000) 2024-03-06T13:34:11.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:11.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:11.793081+0000) 2024-03-06T13:34:12.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:12.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:12.793337+0000) 2024-03-06T13:34:13.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:13.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:13.793577+0000) 2024-03-06T13:34:14.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:14.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:14.793795+0000) 2024-03-06T13:34:15.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:15.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:15.794061+0000) 2024-03-06T13:34:16.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:16.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:16.794288+0000) 2024-03-06T13:34:17.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:17.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:17.794530+0000) 2024-03-06T13:34:18.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:18.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:18.794761+0000) 2024-03-06T13:34:19.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:19.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:19.794984+0000) 2024-03-06T13:34:20.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:20.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:20.795219+0000) 2024-03-06T13:34:21.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:21.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:21.795412+0000) 2024-03-06T13:34:22.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:22.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:22.795596+0000) 2024-03-06T13:34:23.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:23.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:23.795854+0000) 2024-03-06T13:34:24.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:24.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:24.796111+0000) 2024-03-06T13:34:25.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:25.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:25.796354+0000) 2024-03-06T13:34:26.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:26.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:26.796564+0000) 2024-03-06T13:34:27.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:27.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:27.796797+0000) 2024-03-06T13:34:28.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:28.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:28.797060+0000) 2024-03-06T13:34:29.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:29.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:29.797287+0000) 2024-03-06T13:34:30.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:30.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:30.797533+0000) 2024-03-06T13:34:31.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:31.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:31.797755+0000) 2024-03-06T13:34:32.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:32.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:32.797990+0000) 2024-03-06T13:34:33.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:33.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:33.798254+0000) 2024-03-06T13:34:34.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:34.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:34.798516+0000) 2024-03-06T13:34:35.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:35.387 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:34:35.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:35.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:35.798752+0000) 2024-03-06T13:34:36.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:36.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:36.798990+0000) 2024-03-06T13:34:37.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:37.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:37.799257+0000) 2024-03-06T13:34:38.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:38.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:38.799478+0000) 2024-03-06T13:34:39.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:39.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:39.799700+0000) 2024-03-06T13:34:40.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:40.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:40.799926+0000) 2024-03-06T13:34:41.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:41.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:41.800163+0000) 2024-03-06T13:34:42.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:42.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:42.800403+0000) 2024-03-06T13:34:43.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:43.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:43.800638+0000) 2024-03-06T13:34:44.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:44.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:44.800867+0000) 2024-03-06T13:34:45.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:45.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:45.801135+0000) 2024-03-06T13:34:46.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:46.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:46.801403+0000) 2024-03-06T13:34:47.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:47.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:47.801641+0000) 2024-03-06T13:34:48.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:48.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:48.801841+0000) 2024-03-06T13:34:49.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:49.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:49.802103+0000) 2024-03-06T13:34:50.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:50.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:50.802334+0000) 2024-03-06T13:34:51.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:51.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:51.802534+0000) 2024-03-06T13:34:52.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:52.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:52.802776+0000) 2024-03-06T13:34:53.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:53.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:53.803044+0000) 2024-03-06T13:34:54.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:54.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:54.803235+0000) 2024-03-06T13:34:55.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:55.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:55.803455+0000) 2024-03-06T13:34:56.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:56.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:56.803679+0000) 2024-03-06T13:34:57.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:57.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:57.803913+0000) 2024-03-06T13:34:58.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:58.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:58.804145+0000) 2024-03-06T13:34:59.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:34:59.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:34:59.804372+0000) 2024-03-06T13:35:00.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:00.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:00.804601+0000) 2024-03-06T13:35:01.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:01.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:01.804827+0000) 2024-03-06T13:35:02.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:02.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:02.805082+0000) 2024-03-06T13:35:03.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:03.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:03.805315+0000) 2024-03-06T13:35:04.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:04.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:04.805540+0000) 2024-03-06T13:35:05.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:05.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:05.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:05.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:05.805800+0000) 2024-03-06T13:35:06.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:06.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:06.806077+0000) 2024-03-06T13:35:07.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:07.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:07.806327+0000) 2024-03-06T13:35:08.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:08.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:08.806561+0000) 2024-03-06T13:35:09.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:09.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:09.806787+0000) 2024-03-06T13:35:10.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:10.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:10.806991+0000) 2024-03-06T13:35:11.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:11.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:11.807217+0000) 2024-03-06T13:35:12.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:12.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:12.807411+0000) 2024-03-06T13:35:13.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:13.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:13.807641+0000) 2024-03-06T13:35:14.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:14.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:14.807847+0000) 2024-03-06T13:35:15.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:15.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:15.808127+0000) 2024-03-06T13:35:16.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:16.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:16.808327+0000) 2024-03-06T13:35:17.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:17.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:17.808577+0000) 2024-03-06T13:35:18.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:18.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:18.808802+0000) 2024-03-06T13:35:19.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:19.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:19.809056+0000) 2024-03-06T13:35:20.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:20.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:20.809338+0000) 2024-03-06T13:35:21.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:21.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:21.809566+0000) 2024-03-06T13:35:22.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:22.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:22.809808+0000) 2024-03-06T13:35:23.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:23.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:23.810080+0000) 2024-03-06T13:35:24.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:24.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:24.810301+0000) 2024-03-06T13:35:25.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:25.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:25.810530+0000) 2024-03-06T13:35:26.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:26.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:26.810754+0000) 2024-03-06T13:35:27.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:27.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:27.811081+0000) 2024-03-06T13:35:28.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:28.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:28.811345+0000) 2024-03-06T13:35:29.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:29.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:29.811568+0000) 2024-03-06T13:35:30.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:30.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:30.811803+0000) 2024-03-06T13:35:31.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:31.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:31.812077+0000) 2024-03-06T13:35:32.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:32.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:32.812273+0000) 2024-03-06T13:35:33.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:33.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:33.812508+0000) 2024-03-06T13:35:34.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:34.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:34.812700+0000) 2024-03-06T13:35:35.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:35.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:35:35.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:35.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:35.812882+0000) 2024-03-06T13:35:36.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:36.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:36.813118+0000) 2024-03-06T13:35:37.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:37.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:37.813357+0000) 2024-03-06T13:35:38.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:38.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:38.813578+0000) 2024-03-06T13:35:39.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:39.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:39.813819+0000) 2024-03-06T13:35:40.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:40.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:40.814127+0000) 2024-03-06T13:35:41.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:41.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:41.814316+0000) 2024-03-06T13:35:42.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:42.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:42.814560+0000) 2024-03-06T13:35:43.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:43.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:43.814830+0000) 2024-03-06T13:35:44.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:44.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:44.815045+0000) 2024-03-06T13:35:45.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:45.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:45.815267+0000) 2024-03-06T13:35:46.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:46.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:46.815520+0000) 2024-03-06T13:35:47.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:47.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:47.815770+0000) 2024-03-06T13:35:48.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:48.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:48.816047+0000) 2024-03-06T13:35:49.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:49.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:49.816270+0000) 2024-03-06T13:35:50.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:50.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:50.816495+0000) 2024-03-06T13:35:51.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:51.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:51.816718+0000) 2024-03-06T13:35:52.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:52.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:52.816960+0000) 2024-03-06T13:35:53.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:53.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:53.817204+0000) 2024-03-06T13:35:54.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:54.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:54.817422+0000) 2024-03-06T13:35:55.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:55.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:55.817653+0000) 2024-03-06T13:35:56.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:56.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:56.817865+0000) 2024-03-06T13:35:57.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:57.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:57.818166+0000) 2024-03-06T13:35:58.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:58.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:58.818427+0000) 2024-03-06T13:35:59.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:35:59.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:35:59.818698+0000) 2024-03-06T13:36:00.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:00.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:00.818966+0000) 2024-03-06T13:36:01.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:01.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:01.819290+0000) 2024-03-06T13:36:02.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:02.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:02.819576+0000) 2024-03-06T13:36:03.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:03.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:03.819799+0000) 2024-03-06T13:36:04.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:04.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:04.820072+0000) 2024-03-06T13:36:05.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:05.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:05.820296+0000) 2024-03-06T13:36:05.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:05.967 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:06.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:06.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:06.820533+0000) 2024-03-06T13:36:07.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:07.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:07.820790+0000) 2024-03-06T13:36:08.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:08.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:08.821062+0000) 2024-03-06T13:36:09.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:09.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:09.821288+0000) 2024-03-06T13:36:10.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:10.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:10.821511+0000) 2024-03-06T13:36:11.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:11.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:11.821707+0000) 2024-03-06T13:36:12.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:12.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:12.821994+0000) 2024-03-06T13:36:13.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:13.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:13.822354+0000) 2024-03-06T13:36:14.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:14.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:14.822617+0000) 2024-03-06T13:36:15.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:15.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:15.822908+0000) 2024-03-06T13:36:16.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:16.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:16.823128+0000) 2024-03-06T13:36:17.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:17.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:17.823406+0000) 2024-03-06T13:36:18.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:18.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:18.823669+0000) 2024-03-06T13:36:19.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:19.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:19.823871+0000) 2024-03-06T13:36:20.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:20.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:20.824144+0000) 2024-03-06T13:36:21.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:21.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:21.824363+0000) 2024-03-06T13:36:22.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:22.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:22.824617+0000) 2024-03-06T13:36:23.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:23.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:23.824855+0000) 2024-03-06T13:36:24.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:24.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:24.825116+0000) 2024-03-06T13:36:25.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:25.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:25.825312+0000) 2024-03-06T13:36:26.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:26.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:26.825546+0000) 2024-03-06T13:36:27.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:27.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:27.825774+0000) 2024-03-06T13:36:28.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:28.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:28.826047+0000) 2024-03-06T13:36:29.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:29.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:29.826280+0000) 2024-03-06T13:36:30.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:30.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:30.826505+0000) 2024-03-06T13:36:31.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:31.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:31.826742+0000) 2024-03-06T13:36:32.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:32.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:32.826965+0000) 2024-03-06T13:36:33.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:33.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:33.827216+0000) 2024-03-06T13:36:34.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:34.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:34.827446+0000) 2024-03-06T13:36:35.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:35.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:35.827668+0000) 2024-03-06T13:36:36.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:36.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:36:36.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:36.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:36.827901+0000) 2024-03-06T13:36:37.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:37.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:37.828150+0000) 2024-03-06T13:36:38.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:38.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:38.828413+0000) 2024-03-06T13:36:39.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:39.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:39.828616+0000) 2024-03-06T13:36:40.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:40.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:40.828809+0000) 2024-03-06T13:36:41.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:41.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:41.829073+0000) 2024-03-06T13:36:42.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:42.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:42.829313+0000) 2024-03-06T13:36:43.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:43.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:43.829572+0000) 2024-03-06T13:36:44.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:44.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:44.829813+0000) 2024-03-06T13:36:45.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:45.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:45.830076+0000) 2024-03-06T13:36:46.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:46.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:46.830267+0000) 2024-03-06T13:36:47.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:47.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:47.830500+0000) 2024-03-06T13:36:48.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:48.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:48.830729+0000) 2024-03-06T13:36:49.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:49.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:49.830954+0000) 2024-03-06T13:36:50.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:50.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:50.831204+0000) 2024-03-06T13:36:51.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:51.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:51.831389+0000) 2024-03-06T13:36:52.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:52.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:52.831617+0000) 2024-03-06T13:36:53.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:53.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:53.831840+0000) 2024-03-06T13:36:54.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:54.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:54.832105+0000) 2024-03-06T13:36:55.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:55.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:55.832323+0000) 2024-03-06T13:36:56.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:56.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:56.832509+0000) 2024-03-06T13:36:57.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:57.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:57.832749+0000) 2024-03-06T13:36:58.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:58.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:58.832971+0000) 2024-03-06T13:36:59.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:36:59.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:36:59.833201+0000) 2024-03-06T13:37:00.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:00.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:00.833422+0000) 2024-03-06T13:37:01.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:01.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:01.833633+0000) 2024-03-06T13:37:02.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:02.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:02.833872+0000) 2024-03-06T13:37:03.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:03.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:03.834105+0000) 2024-03-06T13:37:04.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:04.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:04.834285+0000) 2024-03-06T13:37:05.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:05.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:05.834519+0000) 2024-03-06T13:37:06.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:06.327 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:06.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:06.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:06.834743+0000) 2024-03-06T13:37:07.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:07.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:07.834971+0000) 2024-03-06T13:37:08.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:08.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:08.835260+0000) 2024-03-06T13:37:09.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:09.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:09.835430+0000) 2024-03-06T13:37:10.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:10.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:10.835658+0000) 2024-03-06T13:37:11.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:11.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:11.835876+0000) 2024-03-06T13:37:12.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:12.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:12.836125+0000) 2024-03-06T13:37:13.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:13.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:13.836356+0000) 2024-03-06T13:37:14.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:14.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:14.836575+0000) 2024-03-06T13:37:15.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:15.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:15.836796+0000) 2024-03-06T13:37:16.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:16.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:16.837062+0000) 2024-03-06T13:37:17.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:17.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:17.837282+0000) 2024-03-06T13:37:18.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:18.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:18.837410+0000) 2024-03-06T13:37:19.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:19.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:19.837570+0000) 2024-03-06T13:37:20.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:20.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:20.837792+0000) 2024-03-06T13:37:21.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:21.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:21.838037+0000) 2024-03-06T13:37:22.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:22.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:22.838225+0000) 2024-03-06T13:37:23.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:23.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:23.838449+0000) 2024-03-06T13:37:24.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:24.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:24.838648+0000) 2024-03-06T13:37:25.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:25.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:25.838867+0000) 2024-03-06T13:37:26.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:26.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:26.839126+0000) 2024-03-06T13:37:27.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:27.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:27.839307+0000) 2024-03-06T13:37:28.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:28.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:28.839535+0000) 2024-03-06T13:37:29.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:29.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:29.839761+0000) 2024-03-06T13:37:30.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:30.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:30.839977+0000) 2024-03-06T13:37:31.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:31.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:31.840214+0000) 2024-03-06T13:37:32.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:32.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:32.840471+0000) 2024-03-06T13:37:33.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:33.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:33.840702+0000) 2024-03-06T13:37:34.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:34.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:34.840910+0000) 2024-03-06T13:37:35.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:35.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:35.841144+0000) 2024-03-06T13:37:36.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:36.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:37:36.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:36.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:36.841382+0000) 2024-03-06T13:37:37.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:37.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:37.841615+0000) 2024-03-06T13:37:38.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:38.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:38.841843+0000) 2024-03-06T13:37:39.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:39.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:39.842083+0000) 2024-03-06T13:37:40.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:40.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:40.842274+0000) 2024-03-06T13:37:41.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:41.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:41.842486+0000) 2024-03-06T13:37:42.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:42.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:42.842713+0000) 2024-03-06T13:37:43.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:43.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:43.842952+0000) 2024-03-06T13:37:44.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:44.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:44.843149+0000) 2024-03-06T13:37:45.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:45.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:45.843370+0000) 2024-03-06T13:37:46.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:46.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:46.843595+0000) 2024-03-06T13:37:47.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:47.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:47.843828+0000) 2024-03-06T13:37:48.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:48.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:48.844053+0000) 2024-03-06T13:37:49.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:49.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:49.844275+0000) 2024-03-06T13:37:50.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:50.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:50.844503+0000) 2024-03-06T13:37:51.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:51.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:51.844731+0000) 2024-03-06T13:37:52.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:52.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:52.844955+0000) 2024-03-06T13:37:53.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:53.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:53.845184+0000) 2024-03-06T13:37:54.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:54.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:54.845444+0000) 2024-03-06T13:37:55.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:55.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:55.845685+0000) 2024-03-06T13:37:56.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:56.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:56.845992+0000) 2024-03-06T13:37:57.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:57.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:57.846273+0000) 2024-03-06T13:37:58.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:58.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:58.846500+0000) 2024-03-06T13:37:59.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:37:59.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:37:59.846721+0000) 2024-03-06T13:38:00.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:00.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:00.846909+0000) 2024-03-06T13:38:01.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:01.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:01.847163+0000) 2024-03-06T13:38:02.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:02.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:02.847409+0000) 2024-03-06T13:38:03.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:03.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:03.847628+0000) 2024-03-06T13:38:04.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:04.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:04.847814+0000) 2024-03-06T13:38:05.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:05.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:05.848064+0000) 2024-03-06T13:38:06.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:06.737 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:06.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:06.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:06.848220+0000) 2024-03-06T13:38:07.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:07.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:07.848438+0000) 2024-03-06T13:38:08.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:08.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:08.848667+0000) 2024-03-06T13:38:09.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:09.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:09.848892+0000) 2024-03-06T13:38:10.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:10.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:10.849123+0000) 2024-03-06T13:38:11.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:11.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:11.849345+0000) 2024-03-06T13:38:12.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:12.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:12.849536+0000) 2024-03-06T13:38:13.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:13.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:13.849715+0000) 2024-03-06T13:38:14.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:14.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:14.849922+0000) 2024-03-06T13:38:15.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:15.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:15.850159+0000) 2024-03-06T13:38:16.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:16.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:16.850398+0000) 2024-03-06T13:38:17.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:17.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:17.850628+0000) 2024-03-06T13:38:18.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:18.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:18.850861+0000) 2024-03-06T13:38:19.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:19.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:19.851122+0000) 2024-03-06T13:38:20.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:20.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:20.851342+0000) 2024-03-06T13:38:21.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:21.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:21.851514+0000) 2024-03-06T13:38:22.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:22.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:22.851736+0000) 2024-03-06T13:38:23.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:23.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:23.851974+0000) 2024-03-06T13:38:24.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:24.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:24.852200+0000) 2024-03-06T13:38:25.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:25.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:25.852423+0000) 2024-03-06T13:38:26.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:26.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:26.852644+0000) 2024-03-06T13:38:27.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:27.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:27.852883+0000) 2024-03-06T13:38:28.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:28.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:28.853171+0000) 2024-03-06T13:38:29.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:29.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:29.853397+0000) 2024-03-06T13:38:30.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:30.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:30.853596+0000) 2024-03-06T13:38:31.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:31.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:31.853785+0000) 2024-03-06T13:38:32.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:32.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:32.853999+0000) 2024-03-06T13:38:33.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:33.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:33.854280+0000) 2024-03-06T13:38:34.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:34.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:34.854514+0000) 2024-03-06T13:38:35.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:35.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:35.854773+0000) 2024-03-06T13:38:36.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:36.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:36.854966+0000) 2024-03-06T13:38:36.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:36.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:38:37.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:37.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:37.855185+0000) 2024-03-06T13:38:38.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:38.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:38.855414+0000) 2024-03-06T13:38:39.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:39.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:39.855638+0000) 2024-03-06T13:38:40.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:40.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:40.855858+0000) 2024-03-06T13:38:41.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:41.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:41.856120+0000) 2024-03-06T13:38:42.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:42.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:42.856365+0000) 2024-03-06T13:38:43.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:43.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:43.856604+0000) 2024-03-06T13:38:44.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:44.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:44.856877+0000) 2024-03-06T13:38:45.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:45.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:45.857123+0000) 2024-03-06T13:38:46.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:46.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:46.857349+0000) 2024-03-06T13:38:47.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:47.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:47.857553+0000) 2024-03-06T13:38:48.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:48.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:48.857794+0000) 2024-03-06T13:38:49.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:49.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:49.858058+0000) 2024-03-06T13:38:50.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:50.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:50.858277+0000) 2024-03-06T13:38:51.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:51.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:51.858503+0000) 2024-03-06T13:38:52.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:52.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:52.858730+0000) 2024-03-06T13:38:53.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:53.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:53.858969+0000) 2024-03-06T13:38:54.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:54.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:54.859207+0000) 2024-03-06T13:38:55.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:55.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:55.859434+0000) 2024-03-06T13:38:56.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:56.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:56.859697+0000) 2024-03-06T13:38:57.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:57.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:57.859909+0000) 2024-03-06T13:38:58.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:58.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:58.860201+0000) 2024-03-06T13:38:59.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:38:59.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:38:59.860449+0000) 2024-03-06T13:39:00.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:00.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:00.860675+0000) 2024-03-06T13:39:01.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:01.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:01.860920+0000) 2024-03-06T13:39:02.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:02.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:02.861163+0000) 2024-03-06T13:39:03.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:03.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:03.861372+0000) 2024-03-06T13:39:04.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:04.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:04.861569+0000) 2024-03-06T13:39:05.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:05.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:05.861839+0000) 2024-03-06T13:39:06.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:06.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:06.862149+0000) 2024-03-06T13:39:07.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:07.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:07.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:07.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:07.862440+0000) 2024-03-06T13:39:08.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:08.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:08.862715+0000) 2024-03-06T13:39:09.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:09.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:09.862910+0000) 2024-03-06T13:39:10.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:10.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:10.863133+0000) 2024-03-06T13:39:11.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:11.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:11.863367+0000) 2024-03-06T13:39:12.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:12.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:12.863654+0000) 2024-03-06T13:39:13.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:13.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:13.863871+0000) 2024-03-06T13:39:14.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:14.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:14.864139+0000) 2024-03-06T13:39:15.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:15.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:15.864367+0000) 2024-03-06T13:39:16.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:16.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:16.864590+0000) 2024-03-06T13:39:17.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:17.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:17.864825+0000) 2024-03-06T13:39:18.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:18.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:18.865093+0000) 2024-03-06T13:39:19.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:19.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:19.865321+0000) 2024-03-06T13:39:20.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:20.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:20.865500+0000) 2024-03-06T13:39:21.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:21.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:21.865721+0000) 2024-03-06T13:39:22.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:22.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:22.865959+0000) 2024-03-06T13:39:23.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:23.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:23.866211+0000) 2024-03-06T13:39:24.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:24.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:24.866408+0000) 2024-03-06T13:39:25.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:25.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:25.866637+0000) 2024-03-06T13:39:26.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:26.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:26.866824+0000) 2024-03-06T13:39:27.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:27.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:27.867108+0000) 2024-03-06T13:39:28.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:28.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:28.867304+0000) 2024-03-06T13:39:29.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:29.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:29.867540+0000) 2024-03-06T13:39:30.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:30.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:30.867765+0000) 2024-03-06T13:39:31.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:31.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:31.867949+0000) 2024-03-06T13:39:32.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:32.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:32.868197+0000) 2024-03-06T13:39:33.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:33.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:33.868415+0000) 2024-03-06T13:39:34.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:34.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:34.868613+0000) 2024-03-06T13:39:35.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:35.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:35.868839+0000) 2024-03-06T13:39:36.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:36.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:36.869147+0000) 2024-03-06T13:39:37.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:37.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:39:37.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:37.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:37.869483+0000) 2024-03-06T13:39:38.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:38.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:38.869748+0000) 2024-03-06T13:39:39.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:39.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:39.870061+0000) 2024-03-06T13:39:40.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:40.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:40.870338+0000) 2024-03-06T13:39:41.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:41.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:41.870624+0000) 2024-03-06T13:39:42.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:42.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:42.870914+0000) 2024-03-06T13:39:43.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:43.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:43.871216+0000) 2024-03-06T13:39:44.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:44.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:44.871505+0000) 2024-03-06T13:39:45.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:45.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:45.871786+0000) 2024-03-06T13:39:46.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:46.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:46.872041+0000) 2024-03-06T13:39:47.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:47.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:47.872283+0000) 2024-03-06T13:39:48.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:48.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:48.872526+0000) 2024-03-06T13:39:49.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:49.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:49.872715+0000) 2024-03-06T13:39:50.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:50.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:50.872943+0000) 2024-03-06T13:39:51.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:51.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:51.873155+0000) 2024-03-06T13:39:52.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:52.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:52.873378+0000) 2024-03-06T13:39:53.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:53.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:53.873608+0000) 2024-03-06T13:39:54.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:54.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:54.873833+0000) 2024-03-06T13:39:55.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:55.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:55.874080+0000) 2024-03-06T13:39:56.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:56.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:56.874271+0000) 2024-03-06T13:39:57.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:57.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:57.874491+0000) 2024-03-06T13:39:58.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:58.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:58.874684+0000) 2024-03-06T13:39:59.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:39:59.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:39:59.874918+0000) 2024-03-06T13:40:00.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:00.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:00.875160+0000) 2024-03-06T13:40:01.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:01.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:01.875350+0000) 2024-03-06T13:40:02.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:02.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:02.875577+0000) 2024-03-06T13:40:03.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:03.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:03.875827+0000) 2024-03-06T13:40:04.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:04.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:04.876086+0000) 2024-03-06T13:40:05.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:05.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:05.876310+0000) 2024-03-06T13:40:06.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:06.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:06.876502+0000) 2024-03-06T13:40:07.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:07.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:07.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:07.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:07.876684+0000) 2024-03-06T13:40:08.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:08.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:08.876915+0000) 2024-03-06T13:40:09.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:09.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:09.877143+0000) 2024-03-06T13:40:10.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:10.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:10.877373+0000) 2024-03-06T13:40:11.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:11.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:11.877606+0000) 2024-03-06T13:40:12.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:12.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:12.877821+0000) 2024-03-06T13:40:13.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:13.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:13.878120+0000) 2024-03-06T13:40:14.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:14.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:14.878352+0000) 2024-03-06T13:40:15.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:15.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:15.878547+0000) 2024-03-06T13:40:16.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:16.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:16.878727+0000) 2024-03-06T13:40:17.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:17.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:17.878956+0000) 2024-03-06T13:40:18.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:18.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:18.879188+0000) 2024-03-06T13:40:19.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:19.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:19.879421+0000) 2024-03-06T13:40:20.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:20.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:20.879672+0000) 2024-03-06T13:40:21.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:21.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:21.879912+0000) 2024-03-06T13:40:22.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:22.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:22.880160+0000) 2024-03-06T13:40:23.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:23.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:23.880381+0000) 2024-03-06T13:40:24.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:24.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:24.880596+0000) 2024-03-06T13:40:25.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:25.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:25.880874+0000) 2024-03-06T13:40:26.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:26.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:26.881134+0000) 2024-03-06T13:40:27.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:27.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:27.881360+0000) 2024-03-06T13:40:28.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:28.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:28.881599+0000) 2024-03-06T13:40:29.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:29.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:29.881821+0000) 2024-03-06T13:40:30.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:30.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:30.882056+0000) 2024-03-06T13:40:31.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:31.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:31.882289+0000) 2024-03-06T13:40:32.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:32.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:32.882515+0000) 2024-03-06T13:40:33.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:33.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:33.882754+0000) 2024-03-06T13:40:34.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:34.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:34.882954+0000) 2024-03-06T13:40:35.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:35.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:35.883138+0000) 2024-03-06T13:40:36.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:36.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:36.883382+0000) 2024-03-06T13:40:37.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:37.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:40:37.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:37.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:37.883654+0000) 2024-03-06T13:40:38.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:38.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:38.883875+0000) 2024-03-06T13:40:39.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:39.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:39.884139+0000) 2024-03-06T13:40:40.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:40.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:40.884367+0000) 2024-03-06T13:40:41.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:41.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:41.884545+0000) 2024-03-06T13:40:42.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:42.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:42.884763+0000) 2024-03-06T13:40:43.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:43.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:43.885053+0000) 2024-03-06T13:40:44.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:44.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:44.885322+0000) 2024-03-06T13:40:45.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:45.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:45.885562+0000) 2024-03-06T13:40:46.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:46.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:46.885787+0000) 2024-03-06T13:40:47.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:47.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:47.886029+0000) 2024-03-06T13:40:48.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:48.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:48.886236+0000) 2024-03-06T13:40:49.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:49.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:49.886461+0000) 2024-03-06T13:40:50.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:50.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:50.886678+0000) 2024-03-06T13:40:51.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:51.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:51.886932+0000) 2024-03-06T13:40:52.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:52.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:52.887158+0000) 2024-03-06T13:40:53.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:53.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:53.887375+0000) 2024-03-06T13:40:54.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:54.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:54.887573+0000) 2024-03-06T13:40:55.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:55.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:55.887805+0000) 2024-03-06T13:40:56.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:56.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:56.888050+0000) 2024-03-06T13:40:57.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:57.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:57.888284+0000) 2024-03-06T13:40:58.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:58.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:58.888509+0000) 2024-03-06T13:40:59.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:40:59.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:40:59.888735+0000) 2024-03-06T13:41:00.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:00.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:00.888958+0000) 2024-03-06T13:41:01.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:01.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:01.889181+0000) 2024-03-06T13:41:02.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:02.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:02.889405+0000) 2024-03-06T13:41:03.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:03.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:03.889688+0000) 2024-03-06T13:41:04.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:04.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:04.889867+0000) 2024-03-06T13:41:05.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:05.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:05.890132+0000) 2024-03-06T13:41:06.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:06.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:06.890361+0000) 2024-03-06T13:41:07.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:07.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:07.890580+0000) 2024-03-06T13:41:07.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:07.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:08.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:08.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:08.890843+0000) 2024-03-06T13:41:09.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:09.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:09.891098+0000) 2024-03-06T13:41:10.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:10.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:10.891352+0000) 2024-03-06T13:41:11.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:11.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:11.891596+0000) 2024-03-06T13:41:12.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:12.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:12.891810+0000) 2024-03-06T13:41:13.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:13.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:13.892159+0000) 2024-03-06T13:41:14.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:14.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:14.892385+0000) 2024-03-06T13:41:15.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:15.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:15.892603+0000) 2024-03-06T13:41:16.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:16.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:16.892879+0000) 2024-03-06T13:41:17.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:17.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:17.893087+0000) 2024-03-06T13:41:18.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:18.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:18.893307+0000) 2024-03-06T13:41:19.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:19.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:19.893535+0000) 2024-03-06T13:41:20.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:20.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:20.893766+0000) 2024-03-06T13:41:21.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:21.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:21.893977+0000) 2024-03-06T13:41:22.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:22.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:22.894243+0000) 2024-03-06T13:41:23.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:23.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:23.894497+0000) 2024-03-06T13:41:24.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:24.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:24.894728+0000) 2024-03-06T13:41:25.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:25.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:25.894946+0000) 2024-03-06T13:41:26.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:26.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:26.895171+0000) 2024-03-06T13:41:27.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:27.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:27.895341+0000) 2024-03-06T13:41:28.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:28.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:28.895555+0000) 2024-03-06T13:41:29.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:29.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:29.895781+0000) 2024-03-06T13:41:30.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:30.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:30.896042+0000) 2024-03-06T13:41:31.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:31.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:31.896241+0000) 2024-03-06T13:41:32.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:32.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:32.896463+0000) 2024-03-06T13:41:33.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:33.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:33.896738+0000) 2024-03-06T13:41:34.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:34.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:34.896999+0000) 2024-03-06T13:41:35.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:35.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:35.897256+0000) 2024-03-06T13:41:36.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:36.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:36.897531+0000) 2024-03-06T13:41:37.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:37.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:37.897769+0000) 2024-03-06T13:41:38.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:38.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:41:38.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:38.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:38.898090+0000) 2024-03-06T13:41:39.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:39.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:39.898301+0000) 2024-03-06T13:41:40.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:40.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:40.898530+0000) 2024-03-06T13:41:41.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:41.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:41.898733+0000) 2024-03-06T13:41:42.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:42.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:42.898952+0000) 2024-03-06T13:41:43.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:43.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:43.899178+0000) 2024-03-06T13:41:44.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:44.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:44.899389+0000) 2024-03-06T13:41:45.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:45.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:45.899617+0000) 2024-03-06T13:41:46.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:46.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:46.899872+0000) 2024-03-06T13:41:47.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:47.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:47.900147+0000) 2024-03-06T13:41:48.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:48.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:48.900367+0000) 2024-03-06T13:41:49.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:49.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:49.900590+0000) 2024-03-06T13:41:50.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:50.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:50.900813+0000) 2024-03-06T13:41:51.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:51.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:51.901099+0000) 2024-03-06T13:41:52.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:52.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:52.901294+0000) 2024-03-06T13:41:53.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:53.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:53.901545+0000) 2024-03-06T13:41:54.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:54.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:54.901768+0000) 2024-03-06T13:41:55.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:55.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:55.902046+0000) 2024-03-06T13:41:56.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:56.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:56.902280+0000) 2024-03-06T13:41:57.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:57.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:57.902465+0000) 2024-03-06T13:41:58.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:58.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:58.902716+0000) 2024-03-06T13:41:59.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:41:59.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:41:59.902946+0000) 2024-03-06T13:42:00.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:00.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:00.903158+0000) 2024-03-06T13:42:01.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:01.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:01.903335+0000) 2024-03-06T13:42:02.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:02.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:02.903568+0000) 2024-03-06T13:42:03.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:03.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:03.903804+0000) 2024-03-06T13:42:04.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:04.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:04.904114+0000) 2024-03-06T13:42:05.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:05.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:05.904358+0000) 2024-03-06T13:42:06.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:06.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:06.904583+0000) 2024-03-06T13:42:07.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:07.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:07.904875+0000) 2024-03-06T13:42:08.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:08.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:08.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:08.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:08.905131+0000) 2024-03-06T13:42:09.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:09.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:09.905369+0000) 2024-03-06T13:42:10.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:10.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:10.905616+0000) 2024-03-06T13:42:11.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:11.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:11.905871+0000) 2024-03-06T13:42:12.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:12.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:12.906187+0000) 2024-03-06T13:42:13.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:13.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:13.906458+0000) 2024-03-06T13:42:14.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:14.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:14.906705+0000) 2024-03-06T13:42:15.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:15.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:15.906927+0000) 2024-03-06T13:42:16.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:16.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:16.907159+0000) 2024-03-06T13:42:17.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:17.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:17.907385+0000) 2024-03-06T13:42:18.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:18.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:18.907615+0000) 2024-03-06T13:42:19.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:19.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:19.907832+0000) 2024-03-06T13:42:20.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:20.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:20.908114+0000) 2024-03-06T13:42:21.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:21.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:21.908334+0000) 2024-03-06T13:42:22.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:22.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:22.908520+0000) 2024-03-06T13:42:23.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:23.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:23.908803+0000) 2024-03-06T13:42:24.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:24.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:24.909081+0000) 2024-03-06T13:42:25.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:25.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:25.909329+0000) 2024-03-06T13:42:26.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:26.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:26.909554+0000) 2024-03-06T13:42:27.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:27.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:27.909776+0000) 2024-03-06T13:42:28.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:28.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:28.909996+0000) 2024-03-06T13:42:29.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:29.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:29.910262+0000) 2024-03-06T13:42:30.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:30.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:30.910498+0000) 2024-03-06T13:42:31.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:31.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:31.910718+0000) 2024-03-06T13:42:32.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:32.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:32.910950+0000) 2024-03-06T13:42:33.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:33.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:33.911199+0000) 2024-03-06T13:42:34.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:34.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:34.911430+0000) 2024-03-06T13:42:35.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:35.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:35.911659+0000) 2024-03-06T13:42:36.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:36.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:36.911895+0000) 2024-03-06T13:42:37.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:37.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:37.912117+0000) 2024-03-06T13:42:38.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:38.497 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:42:38.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:38.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:38.912362+0000) 2024-03-06T13:42:39.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:39.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:39.912571+0000) 2024-03-06T13:42:40.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:40.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:40.912773+0000) 2024-03-06T13:42:41.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:41.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:41.912996+0000) 2024-03-06T13:42:42.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:42.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:42.913258+0000) 2024-03-06T13:42:43.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:43.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:43.913490+0000) 2024-03-06T13:42:44.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:44.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:44.913724+0000) 2024-03-06T13:42:45.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:45.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:45.913938+0000) 2024-03-06T13:42:46.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:46.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:46.914207+0000) 2024-03-06T13:42:47.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:47.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:47.914449+0000) 2024-03-06T13:42:48.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:48.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:48.914719+0000) 2024-03-06T13:42:49.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:49.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:49.914996+0000) 2024-03-06T13:42:50.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:50.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:50.915254+0000) 2024-03-06T13:42:51.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:51.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:51.915480+0000) 2024-03-06T13:42:52.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:52.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:52.915702+0000) 2024-03-06T13:42:53.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:53.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:53.915974+0000) 2024-03-06T13:42:54.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:54.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:54.916264+0000) 2024-03-06T13:42:55.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:55.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:55.916536+0000) 2024-03-06T13:42:56.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:56.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:56.916762+0000) 2024-03-06T13:42:57.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:57.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:57.916980+0000) 2024-03-06T13:42:58.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:58.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:58.917249+0000) 2024-03-06T13:42:59.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:42:59.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:42:59.917472+0000) 2024-03-06T13:43:00.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:00.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:00.917682+0000) 2024-03-06T13:43:01.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:01.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:01.917963+0000) 2024-03-06T13:43:02.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:02.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:02.918257+0000) 2024-03-06T13:43:03.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:03.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:03.918546+0000) 2024-03-06T13:43:04.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:04.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:04.918815+0000) 2024-03-06T13:43:05.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:05.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:05.919133+0000) 2024-03-06T13:43:06.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:06.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:06.919391+0000) 2024-03-06T13:43:07.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:07.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:07.919636+0000) 2024-03-06T13:43:08.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:08.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:08.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:08.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:08.919896+0000) 2024-03-06T13:43:09.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:09.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:09.920122+0000) 2024-03-06T13:43:10.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:10.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:10.920318+0000) 2024-03-06T13:43:11.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:11.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:11.920561+0000) 2024-03-06T13:43:12.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:12.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:12.920768+0000) 2024-03-06T13:43:13.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:13.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:13.921044+0000) 2024-03-06T13:43:14.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:14.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:14.921241+0000) 2024-03-06T13:43:15.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:15.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:15.921511+0000) 2024-03-06T13:43:16.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:16.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:16.921789+0000) 2024-03-06T13:43:17.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:17.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:17.922128+0000) 2024-03-06T13:43:18.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:18.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:18.922376+0000) 2024-03-06T13:43:19.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:19.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:19.922651+0000) 2024-03-06T13:43:20.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:20.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:20.922883+0000) 2024-03-06T13:43:21.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:21.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:21.923153+0000) 2024-03-06T13:43:22.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:22.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:22.923351+0000) 2024-03-06T13:43:23.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:23.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:23.923567+0000) 2024-03-06T13:43:24.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:24.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:24.923789+0000) 2024-03-06T13:43:25.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:25.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:25.923973+0000) 2024-03-06T13:43:26.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:26.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:26.924179+0000) 2024-03-06T13:43:27.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:27.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:27.924416+0000) 2024-03-06T13:43:28.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:28.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:28.924646+0000) 2024-03-06T13:43:29.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:29.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:29.924873+0000) 2024-03-06T13:43:30.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:30.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:30.925134+0000) 2024-03-06T13:43:31.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:31.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:31.925378+0000) 2024-03-06T13:43:32.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:32.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:32.925596+0000) 2024-03-06T13:43:33.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:33.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:33.925856+0000) 2024-03-06T13:43:34.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:34.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:34.926184+0000) 2024-03-06T13:43:35.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:35.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:35.926401+0000) 2024-03-06T13:43:36.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:36.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:36.926619+0000) 2024-03-06T13:43:37.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:37.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:37.926842+0000) 2024-03-06T13:43:38.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:38.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:43:38.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:38.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:38.927063+0000) 2024-03-06T13:43:39.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:39.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:39.927292+0000) 2024-03-06T13:43:40.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:40.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:40.927536+0000) 2024-03-06T13:43:41.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:41.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:41.927781+0000) 2024-03-06T13:43:42.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:42.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:42.928057+0000) 2024-03-06T13:43:43.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:43.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:43.928380+0000) 2024-03-06T13:43:44.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:44.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:44.928664+0000) 2024-03-06T13:43:45.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:45.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:45.928934+0000) 2024-03-06T13:43:46.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:46.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:46.929187+0000) 2024-03-06T13:43:47.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:47.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:47.929399+0000) 2024-03-06T13:43:48.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:48.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:48.929638+0000) 2024-03-06T13:43:49.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:49.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:49.929888+0000) 2024-03-06T13:43:50.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:50.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:50.930121+0000) 2024-03-06T13:43:51.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:51.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:51.930354+0000) 2024-03-06T13:43:52.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:52.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:52.930505+0000) 2024-03-06T13:43:53.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:53.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:53.930762+0000) 2024-03-06T13:43:54.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:54.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:54.931052+0000) 2024-03-06T13:43:55.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:55.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:55.931280+0000) 2024-03-06T13:43:56.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:56.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:56.931483+0000) 2024-03-06T13:43:57.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:57.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:57.931708+0000) 2024-03-06T13:43:58.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:58.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:58.931935+0000) 2024-03-06T13:43:59.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:43:59.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:43:59.932179+0000) 2024-03-06T13:44:00.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:00.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:00.932408+0000) 2024-03-06T13:44:01.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:01.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:01.932627+0000) 2024-03-06T13:44:02.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:02.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:02.932852+0000) 2024-03-06T13:44:03.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:03.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:03.933134+0000) 2024-03-06T13:44:04.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:04.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:04.933397+0000) 2024-03-06T13:44:05.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:05.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:05.933644+0000) 2024-03-06T13:44:06.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:06.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:06.933894+0000) 2024-03-06T13:44:07.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:07.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:07.934123+0000) 2024-03-06T13:44:08.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:08.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:08.934350+0000) 2024-03-06T13:44:09.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:09.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:09.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:09.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:09.934584+0000) 2024-03-06T13:44:10.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:10.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:10.934807+0000) 2024-03-06T13:44:11.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:11.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:11.935100+0000) 2024-03-06T13:44:12.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:12.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:12.935338+0000) 2024-03-06T13:44:13.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:13.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:13.935573+0000) 2024-03-06T13:44:14.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:14.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:14.935805+0000) 2024-03-06T13:44:15.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:15.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:15.936070+0000) 2024-03-06T13:44:16.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:16.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:16.936300+0000) 2024-03-06T13:44:17.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:17.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:17.936525+0000) 2024-03-06T13:44:18.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:18.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:18.936743+0000) 2024-03-06T13:44:19.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:19.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:19.937000+0000) 2024-03-06T13:44:20.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:20.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:20.937288+0000) 2024-03-06T13:44:21.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:21.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:21.937520+0000) 2024-03-06T13:44:22.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:22.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:22.937751+0000) 2024-03-06T13:44:23.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:23.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:23.938049+0000) 2024-03-06T13:44:24.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:24.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:24.938316+0000) 2024-03-06T13:44:25.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:25.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:25.938540+0000) 2024-03-06T13:44:26.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:26.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:26.938765+0000) 2024-03-06T13:44:27.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:27.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:27.938995+0000) 2024-03-06T13:44:28.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:28.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:28.939261+0000) 2024-03-06T13:44:29.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:29.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:29.939504+0000) 2024-03-06T13:44:30.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:30.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:30.939748+0000) 2024-03-06T13:44:31.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:31.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:31.939993+0000) 2024-03-06T13:44:32.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:32.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:32.940245+0000) 2024-03-06T13:44:33.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:33.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:33.940471+0000) 2024-03-06T13:44:34.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:34.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:34.940701+0000) 2024-03-06T13:44:35.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:35.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:35.940930+0000) 2024-03-06T13:44:36.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:36.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:36.941242+0000) 2024-03-06T13:44:37.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:37.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:37.941514+0000) 2024-03-06T13:44:38.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:38.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:38.941779+0000) 2024-03-06T13:44:39.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:39.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:44:39.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:39.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:39.942108+0000) 2024-03-06T13:44:40.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:40.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:40.942361+0000) 2024-03-06T13:44:41.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:41.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:41.942643+0000) 2024-03-06T13:44:42.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:42.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:42.942857+0000) 2024-03-06T13:44:43.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:43.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:43.943184+0000) 2024-03-06T13:44:44.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:44.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:44.943443+0000) 2024-03-06T13:44:45.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:45.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:45.943728+0000) 2024-03-06T13:44:46.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:46.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:46.943976+0000) 2024-03-06T13:44:47.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:47.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:47.944292+0000) 2024-03-06T13:44:48.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:48.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:48.944557+0000) 2024-03-06T13:44:49.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:49.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:49.944841+0000) 2024-03-06T13:44:50.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:50.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:50.945158+0000) 2024-03-06T13:44:51.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:51.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:51.945409+0000) 2024-03-06T13:44:52.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:52.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:52.945682+0000) 2024-03-06T13:44:53.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:53.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:53.945963+0000) 2024-03-06T13:44:54.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:54.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:54.946225+0000) 2024-03-06T13:44:55.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:55.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:55.946473+0000) 2024-03-06T13:44:56.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:56.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:56.946704+0000) 2024-03-06T13:44:57.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:57.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:57.946934+0000) 2024-03-06T13:44:58.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:58.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:58.947160+0000) 2024-03-06T13:44:59.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:44:59.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:44:59.947406+0000) 2024-03-06T13:45:00.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:00.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:00.947640+0000) 2024-03-06T13:45:01.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:01.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:01.947881+0000) 2024-03-06T13:45:02.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:02.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:02.948122+0000) 2024-03-06T13:45:03.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:03.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:03.948370+0000) 2024-03-06T13:45:04.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:04.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:04.948655+0000) 2024-03-06T13:45:05.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:05.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:05.948938+0000) 2024-03-06T13:45:06.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:06.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:06.949196+0000) 2024-03-06T13:45:07.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:07.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:07.949401+0000) 2024-03-06T13:45:08.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:08.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:08.949602+0000) 2024-03-06T13:45:09.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:09.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:09.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:09.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:09.949836+0000) 2024-03-06T13:45:10.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:10.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:10.950054+0000) 2024-03-06T13:45:11.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:11.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:11.950247+0000) 2024-03-06T13:45:12.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:12.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:12.950444+0000) 2024-03-06T13:45:13.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:13.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:13.950671+0000) 2024-03-06T13:45:14.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:14.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:14.950904+0000) 2024-03-06T13:45:15.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:15.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:15.951136+0000) 2024-03-06T13:45:16.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:16.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:16.951365+0000) 2024-03-06T13:45:17.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:17.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:17.951577+0000) 2024-03-06T13:45:18.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:18.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:18.951798+0000) 2024-03-06T13:45:19.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:19.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:19.952089+0000) 2024-03-06T13:45:20.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:20.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:20.952315+0000) 2024-03-06T13:45:21.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:21.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:21.952537+0000) 2024-03-06T13:45:22.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:22.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:22.952756+0000) 2024-03-06T13:45:23.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:23.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:23.952972+0000) 2024-03-06T13:45:24.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:24.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:24.953265+0000) 2024-03-06T13:45:25.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:25.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:25.953486+0000) 2024-03-06T13:45:26.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:26.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:26.953711+0000) 2024-03-06T13:45:27.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:27.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:27.953904+0000) 2024-03-06T13:45:28.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:28.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:28.954146+0000) 2024-03-06T13:45:29.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:29.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:29.954325+0000) 2024-03-06T13:45:30.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:30.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:30.954551+0000) 2024-03-06T13:45:31.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:31.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:31.954785+0000) 2024-03-06T13:45:32.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:32.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:32.955040+0000) 2024-03-06T13:45:33.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:33.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:33.955267+0000) 2024-03-06T13:45:34.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:34.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:34.955489+0000) 2024-03-06T13:45:35.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:35.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:35.955705+0000) 2024-03-06T13:45:36.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:36.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:36.955885+0000) 2024-03-06T13:45:37.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:37.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:37.956127+0000) 2024-03-06T13:45:38.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:38.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:38.956369+0000) 2024-03-06T13:45:39.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:39.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:45:39.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:39.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:39.956632+0000) 2024-03-06T13:45:40.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:40.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:40.956883+0000) 2024-03-06T13:45:41.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:41.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:41.957122+0000) 2024-03-06T13:45:42.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:42.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:42.957369+0000) 2024-03-06T13:45:43.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:43.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:43.957598+0000) 2024-03-06T13:45:44.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:44.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:44.957856+0000) 2024-03-06T13:45:45.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:45.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:45.958131+0000) 2024-03-06T13:45:46.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:46.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:46.958283+0000) 2024-03-06T13:45:47.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:47.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:47.958511+0000) 2024-03-06T13:45:48.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:48.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:48.958736+0000) 2024-03-06T13:45:49.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:49.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:49.958931+0000) 2024-03-06T13:45:50.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:50.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:50.959162+0000) 2024-03-06T13:45:51.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:51.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:51.959393+0000) 2024-03-06T13:45:52.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:52.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:52.959539+0000) 2024-03-06T13:45:53.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:53.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:53.959744+0000) 2024-03-06T13:45:54.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:54.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:54.959954+0000) 2024-03-06T13:45:55.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:55.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:55.960131+0000) 2024-03-06T13:45:56.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:56.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:56.960379+0000) 2024-03-06T13:45:57.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:57.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:57.960598+0000) 2024-03-06T13:45:58.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:58.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:58.960862+0000) 2024-03-06T13:45:59.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:45:59.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:45:59.961137+0000) 2024-03-06T13:46:00.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:00.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:00.961352+0000) 2024-03-06T13:46:01.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:01.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:01.961577+0000) 2024-03-06T13:46:02.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:02.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:02.961810+0000) 2024-03-06T13:46:03.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:03.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:03.962090+0000) 2024-03-06T13:46:04.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:04.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:04.962303+0000) 2024-03-06T13:46:05.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:05.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:05.962482+0000) 2024-03-06T13:46:06.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:06.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:06.962713+0000) 2024-03-06T13:46:07.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:07.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:07.962964+0000) 2024-03-06T13:46:08.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:08.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:08.963151+0000) 2024-03-06T13:46:09.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:09.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:09.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:09.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:09.963361+0000) 2024-03-06T13:46:10.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:10.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:10.963595+0000) 2024-03-06T13:46:11.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:11.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:11.963828+0000) 2024-03-06T13:46:12.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:12.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:12.964106+0000) 2024-03-06T13:46:13.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:13.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:13.964351+0000) 2024-03-06T13:46:14.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:14.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:14.964608+0000) 2024-03-06T13:46:15.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:15.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:15.964829+0000) 2024-03-06T13:46:16.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:16.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:16.965117+0000) 2024-03-06T13:46:17.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:17.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:17.965315+0000) 2024-03-06T13:46:18.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:18.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:18.965543+0000) 2024-03-06T13:46:19.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:19.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:19.965734+0000) 2024-03-06T13:46:20.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:20.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:20.965928+0000) 2024-03-06T13:46:21.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:21.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:21.966187+0000) 2024-03-06T13:46:22.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:22.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:22.966411+0000) 2024-03-06T13:46:23.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:23.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:23.966621+0000) 2024-03-06T13:46:24.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:24.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:24.966880+0000) 2024-03-06T13:46:25.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:25.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:25.967176+0000) 2024-03-06T13:46:26.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:26.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:26.967460+0000) 2024-03-06T13:46:27.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:27.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:27.967695+0000) 2024-03-06T13:46:28.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:28.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:28.967951+0000) 2024-03-06T13:46:29.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:29.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:29.968212+0000) 2024-03-06T13:46:30.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:30.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:30.968454+0000) 2024-03-06T13:46:31.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:31.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:31.968704+0000) 2024-03-06T13:46:32.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:32.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:32.968955+0000) 2024-03-06T13:46:33.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:33.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:33.969188+0000) 2024-03-06T13:46:34.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:34.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:34.969434+0000) 2024-03-06T13:46:35.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:35.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:35.969664+0000) 2024-03-06T13:46:36.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:36.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:36.969877+0000) 2024-03-06T13:46:37.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:37.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:37.970152+0000) 2024-03-06T13:46:38.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:38.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:38.970341+0000) 2024-03-06T13:46:39.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:39.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:39.970599+0000) 2024-03-06T13:46:39.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:39.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:46:40.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:40.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:40.970855+0000) 2024-03-06T13:46:41.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:41.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:41.971132+0000) 2024-03-06T13:46:42.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:42.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:42.971384+0000) 2024-03-06T13:46:43.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:43.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:43.971629+0000) 2024-03-06T13:46:44.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:44.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:44.971898+0000) 2024-03-06T13:46:45.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:45.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:45.972160+0000) 2024-03-06T13:46:46.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:46.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:46.972393+0000) 2024-03-06T13:46:47.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:47.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:47.972635+0000) 2024-03-06T13:46:48.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:48.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:48.972877+0000) 2024-03-06T13:46:49.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:49.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:49.973138+0000) 2024-03-06T13:46:50.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:50.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:50.973396+0000) 2024-03-06T13:46:51.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:51.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:51.973575+0000) 2024-03-06T13:46:52.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:52.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:52.973783+0000) 2024-03-06T13:46:53.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:53.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:53.973978+0000) 2024-03-06T13:46:54.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:54.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:54.974276+0000) 2024-03-06T13:46:55.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:55.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:55.974507+0000) 2024-03-06T13:46:56.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:56.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:56.974738+0000) 2024-03-06T13:46:57.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:57.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:57.974959+0000) 2024-03-06T13:46:58.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:58.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:58.975181+0000) 2024-03-06T13:46:59.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:46:59.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:46:59.975438+0000) 2024-03-06T13:47:00.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:00.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:00.975703+0000) 2024-03-06T13:47:01.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:01.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:01.975920+0000) 2024-03-06T13:47:02.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:02.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:02.976125+0000) 2024-03-06T13:47:03.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:03.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:03.976362+0000) 2024-03-06T13:47:04.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:04.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:04.976621+0000) 2024-03-06T13:47:05.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:05.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:05.976848+0000) 2024-03-06T13:47:06.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:06.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:06.977108+0000) 2024-03-06T13:47:07.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:07.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:07.977335+0000) 2024-03-06T13:47:08.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:08.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:08.977572+0000) 2024-03-06T13:47:09.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:09.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:09.977802+0000) 2024-03-06T13:47:10.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:10.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:10.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:10.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:10.978077+0000) 2024-03-06T13:47:11.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:11.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:11.978354+0000) 2024-03-06T13:47:12.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:12.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:12.978551+0000) 2024-03-06T13:47:13.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:13.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:13.978780+0000) 2024-03-06T13:47:14.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:14.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:14.979038+0000) 2024-03-06T13:47:15.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:15.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:15.979265+0000) 2024-03-06T13:47:16.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:16.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:16.979494+0000) 2024-03-06T13:47:17.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:17.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:17.979724+0000) 2024-03-06T13:47:18.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:18.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:18.979945+0000) 2024-03-06T13:47:19.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:19.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:19.980192+0000) 2024-03-06T13:47:20.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:20.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:20.980419+0000) 2024-03-06T13:47:21.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:21.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:21.980655+0000) 2024-03-06T13:47:22.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:22.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:22.980878+0000) 2024-03-06T13:47:23.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:23.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:23.981162+0000) 2024-03-06T13:47:24.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:24.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:24.981342+0000) 2024-03-06T13:47:25.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:25.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:25.981571+0000) 2024-03-06T13:47:26.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:26.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:26.981799+0000) 2024-03-06T13:47:27.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:27.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:27.981982+0000) 2024-03-06T13:47:28.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:28.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:28.982251+0000) 2024-03-06T13:47:29.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:29.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:29.982488+0000) 2024-03-06T13:47:30.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:30.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:30.982717+0000) 2024-03-06T13:47:31.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:31.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:31.982975+0000) 2024-03-06T13:47:32.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:32.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:32.983275+0000) 2024-03-06T13:47:33.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:33.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:33.983495+0000) 2024-03-06T13:47:34.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:34.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:34.983739+0000) 2024-03-06T13:47:35.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:35.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:35.983964+0000) 2024-03-06T13:47:36.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:36.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:36.984165+0000) 2024-03-06T13:47:37.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:37.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:37.984345+0000) 2024-03-06T13:47:38.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:38.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:38.984569+0000) 2024-03-06T13:47:39.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:39.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:39.984826+0000) 2024-03-06T13:47:40.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:40.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:47:40.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:40.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:40.985088+0000) 2024-03-06T13:47:41.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:41.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:41.985327+0000) 2024-03-06T13:47:42.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:42.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:42.985588+0000) 2024-03-06T13:47:43.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:43.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:43.985861+0000) 2024-03-06T13:47:44.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:44.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:44.986157+0000) 2024-03-06T13:47:45.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:45.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:45.986351+0000) 2024-03-06T13:47:46.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:46.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:46.986582+0000) 2024-03-06T13:47:47.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:47.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:47.986832+0000) 2024-03-06T13:47:48.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:48.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:48.987094+0000) 2024-03-06T13:47:49.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:49.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:49.987332+0000) 2024-03-06T13:47:50.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:50.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:50.987583+0000) 2024-03-06T13:47:51.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:51.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:51.987783+0000) 2024-03-06T13:47:52.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:52.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:52.988035+0000) 2024-03-06T13:47:53.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:53.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:53.988312+0000) 2024-03-06T13:47:54.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:54.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:54.988545+0000) 2024-03-06T13:47:55.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:55.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:55.988770+0000) 2024-03-06T13:47:56.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:56.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:56.989046+0000) 2024-03-06T13:47:57.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:57.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:57.989271+0000) 2024-03-06T13:47:58.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:58.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:58.989528+0000) 2024-03-06T13:47:59.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:47:59.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:47:59.989763+0000) 2024-03-06T13:48:00.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:00.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:00.990047+0000) 2024-03-06T13:48:01.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:01.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:01.990235+0000) 2024-03-06T13:48:02.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:02.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:02.990454+0000) 2024-03-06T13:48:03.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:03.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:03.990653+0000) 2024-03-06T13:48:04.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:04.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:04.990907+0000) 2024-03-06T13:48:05.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:05.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:05.991155+0000) 2024-03-06T13:48:06.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:06.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:06.991400+0000) 2024-03-06T13:48:07.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:07.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:07.991644+0000) 2024-03-06T13:48:08.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:08.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:08.991891+0000) 2024-03-06T13:48:09.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:09.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:09.992139+0000) 2024-03-06T13:48:10.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:10.555 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:10.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:10.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:10.992377+0000) 2024-03-06T13:48:11.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:11.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:11.992611+0000) 2024-03-06T13:48:12.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:12.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:12.992855+0000) 2024-03-06T13:48:13.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:13.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:13.993132+0000) 2024-03-06T13:48:14.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:14.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:14.993366+0000) 2024-03-06T13:48:15.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:15.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:15.993579+0000) 2024-03-06T13:48:16.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:16.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:16.993801+0000) 2024-03-06T13:48:17.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:17.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:17.994111+0000) 2024-03-06T13:48:18.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:18.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:18.994307+0000) 2024-03-06T13:48:19.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:19.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:19.994551+0000) 2024-03-06T13:48:20.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:20.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:20.994825+0000) 2024-03-06T13:48:21.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:21.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:21.995131+0000) 2024-03-06T13:48:22.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:22.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:22.995325+0000) 2024-03-06T13:48:23.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:23.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:23.995543+0000) 2024-03-06T13:48:24.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:24.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:24.995741+0000) 2024-03-06T13:48:25.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:25.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:25.995912+0000) 2024-03-06T13:48:26.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:26.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:26.996160+0000) 2024-03-06T13:48:27.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:27.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:27.996372+0000) 2024-03-06T13:48:28.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:28.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:28.996573+0000) 2024-03-06T13:48:29.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:29.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:29.996795+0000) 2024-03-06T13:48:30.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:30.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:30.996982+0000) 2024-03-06T13:48:31.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:31.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:31.997272+0000) 2024-03-06T13:48:32.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:32.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:32.997559+0000) 2024-03-06T13:48:33.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:33.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:33.997816+0000) 2024-03-06T13:48:34.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:34.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:34.998137+0000) 2024-03-06T13:48:35.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:35.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:35.998332+0000) 2024-03-06T13:48:36.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:36.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:36.998600+0000) 2024-03-06T13:48:37.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:37.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:37.998882+0000) 2024-03-06T13:48:38.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:38.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:38.999160+0000) 2024-03-06T13:48:39.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:39.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:39.999389+0000) 2024-03-06T13:48:40.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:40.732 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:48:40.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:40.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:40.999615+0000) 2024-03-06T13:48:41.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:41.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:41.999891+0000) 2024-03-06T13:48:42.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:42.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:43.000134+0000) 2024-03-06T13:48:43.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:43.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:44.000405+0000) 2024-03-06T13:48:44.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:44.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:45.000640+0000) 2024-03-06T13:48:45.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:45.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:46.000872+0000) 2024-03-06T13:48:46.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:47.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:47.001138+0000) 2024-03-06T13:48:47.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:48.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:48.001353+0000) 2024-03-06T13:48:48.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:49.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:49.001593+0000) 2024-03-06T13:48:49.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:50.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:50.001839+0000) 2024-03-06T13:48:50.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:51.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:51.002120+0000) 2024-03-06T13:48:52.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:52.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:52.002320+0000) 2024-03-06T13:48:53.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:53.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:53.002576+0000) 2024-03-06T13:48:54.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:54.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:54.002870+0000) 2024-03-06T13:48:55.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:55.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:55.003108+0000) 2024-03-06T13:48:56.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:56.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:56.003289+0000) 2024-03-06T13:48:57.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:57.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:57.003540+0000) 2024-03-06T13:48:58.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:58.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:58.003818+0000) 2024-03-06T13:48:59.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:48:59.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:48:59.004139+0000) 2024-03-06T13:49:00.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:00.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:00.004350+0000) 2024-03-06T13:49:01.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:01.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:01.004585+0000) 2024-03-06T13:49:02.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:02.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:02.004768+0000) 2024-03-06T13:49:03.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:03.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:03.004992+0000) 2024-03-06T13:49:04.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:04.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:04.005241+0000) 2024-03-06T13:49:05.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:05.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:05.005421+0000) 2024-03-06T13:49:06.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:06.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:06.005661+0000) 2024-03-06T13:49:07.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:07.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:07.005856+0000) 2024-03-06T13:49:08.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:08.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:08.006115+0000) 2024-03-06T13:49:09.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:09.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:09.006355+0000) 2024-03-06T13:49:10.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:10.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:10.006589+0000) 2024-03-06T13:49:10.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:10.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:11.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:11.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:11.006757+0000) 2024-03-06T13:49:12.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:12.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:12.006991+0000) 2024-03-06T13:49:13.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:13.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:13.007309+0000) 2024-03-06T13:49:14.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:14.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:14.007549+0000) 2024-03-06T13:49:15.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:15.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:15.007785+0000) 2024-03-06T13:49:16.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:16.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:16.008060+0000) 2024-03-06T13:49:17.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:17.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:17.008304+0000) 2024-03-06T13:49:18.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:18.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:18.008546+0000) 2024-03-06T13:49:19.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:19.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:19.008764+0000) 2024-03-06T13:49:20.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:20.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:20.009070+0000) 2024-03-06T13:49:21.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:21.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:21.009294+0000) 2024-03-06T13:49:22.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:22.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:22.009521+0000) 2024-03-06T13:49:23.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:23.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:23.009755+0000) 2024-03-06T13:49:24.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:24.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:24.009979+0000) 2024-03-06T13:49:25.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:25.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:25.010299+0000) 2024-03-06T13:49:26.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:26.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:26.010577+0000) 2024-03-06T13:49:27.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:27.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:27.010860+0000) 2024-03-06T13:49:28.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:28.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:28.011166+0000) 2024-03-06T13:49:29.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:29.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:29.011355+0000) 2024-03-06T13:49:30.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:30.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:30.011580+0000) 2024-03-06T13:49:31.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:31.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:31.011805+0000) 2024-03-06T13:49:32.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:32.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:32.012126+0000) 2024-03-06T13:49:33.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:33.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:33.012336+0000) 2024-03-06T13:49:34.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:34.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:34.012539+0000) 2024-03-06T13:49:35.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:35.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:35.012756+0000) 2024-03-06T13:49:36.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:36.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:36.013087+0000) 2024-03-06T13:49:37.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:37.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:37.013333+0000) 2024-03-06T13:49:38.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:38.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:38.013576+0000) 2024-03-06T13:49:39.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:39.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:39.013786+0000) 2024-03-06T13:49:40.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:40.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:40.014045+0000) 2024-03-06T13:49:41.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:41.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:41.014271+0000) 2024-03-06T13:49:41.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:41.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:49:42.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:42.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:42.014479+0000) 2024-03-06T13:49:43.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:43.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:43.014702+0000) 2024-03-06T13:49:44.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:44.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:44.014939+0000) 2024-03-06T13:49:45.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:45.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:45.015154+0000) 2024-03-06T13:49:46.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:46.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:46.015390+0000) 2024-03-06T13:49:47.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:47.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:47.015742+0000) 2024-03-06T13:49:48.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:48.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:48.015987+0000) 2024-03-06T13:49:49.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:49.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:49.016217+0000) 2024-03-06T13:49:50.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:50.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:50.016438+0000) 2024-03-06T13:49:51.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:51.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:51.016629+0000) 2024-03-06T13:49:52.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:52.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:52.016851+0000) 2024-03-06T13:49:53.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:53.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:53.017113+0000) 2024-03-06T13:49:54.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:54.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:54.017346+0000) 2024-03-06T13:49:55.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:55.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:55.017530+0000) 2024-03-06T13:49:56.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:56.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:56.017754+0000) 2024-03-06T13:49:57.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:57.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:57.017982+0000) 2024-03-06T13:49:58.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:58.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:58.018242+0000) 2024-03-06T13:49:59.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:49:59.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:49:59.018453+0000) 2024-03-06T13:50:00.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:00.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:00.018673+0000) 2024-03-06T13:50:01.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:01.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:01.018901+0000) 2024-03-06T13:50:02.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:02.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:02.019150+0000) 2024-03-06T13:50:03.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:03.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:03.019351+0000) 2024-03-06T13:50:04.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:04.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:04.019536+0000) 2024-03-06T13:50:05.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:05.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:05.019781+0000) 2024-03-06T13:50:06.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:06.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:06.020062+0000) 2024-03-06T13:50:07.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:07.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:07.020231+0000) 2024-03-06T13:50:08.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:08.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:08.020426+0000) 2024-03-06T13:50:09.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:09.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:09.020656+0000) 2024-03-06T13:50:10.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:10.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:10.020900+0000) 2024-03-06T13:50:11.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:11.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:11.021138+0000) 2024-03-06T13:50:11.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:11.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:12.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:12.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:12.021342+0000) 2024-03-06T13:50:13.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:13.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:13.021544+0000) 2024-03-06T13:50:14.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:14.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:14.021792+0000) 2024-03-06T13:50:15.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:15.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:15.022035+0000) 2024-03-06T13:50:16.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:16.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:16.022221+0000) 2024-03-06T13:50:17.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:17.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:17.022390+0000) 2024-03-06T13:50:18.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:18.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:18.022613+0000) 2024-03-06T13:50:19.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:19.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:19.022831+0000) 2024-03-06T13:50:20.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:20.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:20.023061+0000) 2024-03-06T13:50:21.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:21.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:21.023328+0000) 2024-03-06T13:50:22.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:22.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:22.023552+0000) 2024-03-06T13:50:23.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:23.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:23.023778+0000) 2024-03-06T13:50:24.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:24.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:24.023975+0000) 2024-03-06T13:50:25.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:25.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:25.024153+0000) 2024-03-06T13:50:26.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:26.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:26.024347+0000) 2024-03-06T13:50:27.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:27.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:27.024575+0000) 2024-03-06T13:50:28.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:28.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:28.024793+0000) 2024-03-06T13:50:29.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:29.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:29.025118+0000) 2024-03-06T13:50:30.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:30.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:30.025315+0000) 2024-03-06T13:50:31.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:31.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:31.025512+0000) 2024-03-06T13:50:32.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:32.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:32.025743+0000) 2024-03-06T13:50:33.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:33.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:33.025964+0000) 2024-03-06T13:50:34.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:34.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:34.026308+0000) 2024-03-06T13:50:35.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:35.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:35.026589+0000) 2024-03-06T13:50:36.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:36.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:36.026865+0000) 2024-03-06T13:50:37.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:37.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:37.027123+0000) 2024-03-06T13:50:38.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:38.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:38.027310+0000) 2024-03-06T13:50:39.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:39.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:39.027537+0000) 2024-03-06T13:50:40.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:40.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:40.027740+0000) 2024-03-06T13:50:41.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:41.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:41.027977+0000) 2024-03-06T13:50:41.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:41.428 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:50:42.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:42.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:42.028228+0000) 2024-03-06T13:50:43.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:43.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:43.028417+0000) 2024-03-06T13:50:44.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:44.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:44.028652+0000) 2024-03-06T13:50:45.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:45.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:45.028882+0000) 2024-03-06T13:50:46.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:46.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:46.029151+0000) 2024-03-06T13:50:47.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:47.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:47.029419+0000) 2024-03-06T13:50:48.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:48.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:48.029698+0000) 2024-03-06T13:50:49.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:49.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:49.029979+0000) 2024-03-06T13:50:50.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:50.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:50.030299+0000) 2024-03-06T13:50:51.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:51.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:51.030578+0000) 2024-03-06T13:50:52.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:52.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:52.030849+0000) 2024-03-06T13:50:53.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:53.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:53.031170+0000) 2024-03-06T13:50:54.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:54.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:54.031502+0000) 2024-03-06T13:50:55.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:55.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:55.031715+0000) 2024-03-06T13:50:56.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:56.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:56.031945+0000) 2024-03-06T13:50:57.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:57.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:57.032179+0000) 2024-03-06T13:50:58.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:58.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:58.032426+0000) 2024-03-06T13:50:59.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:50:59.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:50:59.032688+0000) 2024-03-06T13:51:00.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:00.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:00.032911+0000) 2024-03-06T13:51:01.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:01.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:01.033181+0000) 2024-03-06T13:51:02.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:02.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:02.033421+0000) 2024-03-06T13:51:03.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:03.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:03.033641+0000) 2024-03-06T13:51:04.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:04.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:04.033878+0000) 2024-03-06T13:51:05.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:05.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:05.034134+0000) 2024-03-06T13:51:06.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:06.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:06.034370+0000) 2024-03-06T13:51:07.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:07.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:07.034592+0000) 2024-03-06T13:51:08.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:08.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:08.034833+0000) 2024-03-06T13:51:09.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:09.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:09.035095+0000) 2024-03-06T13:51:10.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:10.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:10.035343+0000) 2024-03-06T13:51:11.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:11.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:11.035591+0000) 2024-03-06T13:51:11.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:11.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:12.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:12.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:12.035793+0000) 2024-03-06T13:51:13.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:13.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:13.035983+0000) 2024-03-06T13:51:14.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:14.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:14.036310+0000) 2024-03-06T13:51:15.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:15.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:15.036557+0000) 2024-03-06T13:51:16.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:16.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:16.036808+0000) 2024-03-06T13:51:17.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:17.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:17.037094+0000) 2024-03-06T13:51:18.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:18.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:18.037315+0000) 2024-03-06T13:51:19.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:19.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:19.037564+0000) 2024-03-06T13:51:20.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:20.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:20.037817+0000) 2024-03-06T13:51:21.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:21.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:21.038131+0000) 2024-03-06T13:51:22.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:22.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:22.038448+0000) 2024-03-06T13:51:23.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:23.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:23.038646+0000) 2024-03-06T13:51:24.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:24.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:24.038879+0000) 2024-03-06T13:51:25.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:25.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:25.039136+0000) 2024-03-06T13:51:26.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:26.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:26.039383+0000) 2024-03-06T13:51:27.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:27.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:27.039615+0000) 2024-03-06T13:51:28.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:28.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:28.039851+0000) 2024-03-06T13:51:29.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:29.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:29.040128+0000) 2024-03-06T13:51:30.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:30.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:30.040375+0000) 2024-03-06T13:51:31.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:31.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:31.040609+0000) 2024-03-06T13:51:32.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:32.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:32.040844+0000) 2024-03-06T13:51:33.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:33.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:33.041064+0000) 2024-03-06T13:51:34.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:34.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:34.041318+0000) 2024-03-06T13:51:35.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:35.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:35.041537+0000) 2024-03-06T13:51:36.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:36.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:36.041769+0000) 2024-03-06T13:51:37.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:37.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:37.042044+0000) 2024-03-06T13:51:38.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:38.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:38.042301+0000) 2024-03-06T13:51:39.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:39.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:39.042513+0000) 2024-03-06T13:51:40.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:40.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:40.042728+0000) 2024-03-06T13:51:41.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:41.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:41.042983+0000) 2024-03-06T13:51:41.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:41.744 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:51:42.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:42.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:42.043248+0000) 2024-03-06T13:51:43.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:43.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:43.043477+0000) 2024-03-06T13:51:44.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:44.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:44.043713+0000) 2024-03-06T13:51:45.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:45.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:45.043912+0000) 2024-03-06T13:51:46.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:46.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:46.044148+0000) 2024-03-06T13:51:47.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:47.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:47.044339+0000) 2024-03-06T13:51:48.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:48.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:48.044568+0000) 2024-03-06T13:51:49.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:49.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:49.044751+0000) 2024-03-06T13:51:50.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:50.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:50.044942+0000) 2024-03-06T13:51:51.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:51.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:51.045181+0000) 2024-03-06T13:51:52.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:52.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:52.045429+0000) 2024-03-06T13:51:53.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:53.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:53.045670+0000) 2024-03-06T13:51:54.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:54.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:54.045902+0000) 2024-03-06T13:51:55.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:55.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:55.046123+0000) 2024-03-06T13:51:56.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:56.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:56.046359+0000) 2024-03-06T13:51:57.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:57.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:57.046600+0000) 2024-03-06T13:51:58.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:58.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:58.046811+0000) 2024-03-06T13:51:59.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:51:59.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:51:59.047095+0000) 2024-03-06T13:52:00.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:00.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:00.047308+0000) 2024-03-06T13:52:01.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:01.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:01.047564+0000) 2024-03-06T13:52:02.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:02.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:02.047812+0000) 2024-03-06T13:52:03.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:03.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:03.048101+0000) 2024-03-06T13:52:04.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:04.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:04.048366+0000) 2024-03-06T13:52:05.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:05.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:05.048665+0000) 2024-03-06T13:52:06.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:06.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:06.048923+0000) 2024-03-06T13:52:07.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:07.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:07.049102+0000) 2024-03-06T13:52:08.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:08.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:08.049339+0000) 2024-03-06T13:52:09.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:09.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:09.049572+0000) 2024-03-06T13:52:10.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:10.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:10.049777+0000) 2024-03-06T13:52:11.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:11.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:11.050049+0000) 2024-03-06T13:52:11.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:11.892 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:12.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:12.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:12.050235+0000) 2024-03-06T13:52:13.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:13.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:13.050449+0000) 2024-03-06T13:52:14.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:14.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:14.050681+0000) 2024-03-06T13:52:15.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:15.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:15.050933+0000) 2024-03-06T13:52:16.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:16.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:16.051170+0000) 2024-03-06T13:52:17.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:17.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:17.051417+0000) 2024-03-06T13:52:18.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:18.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:18.051664+0000) 2024-03-06T13:52:19.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:19.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:19.051862+0000) 2024-03-06T13:52:20.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:20.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:20.052136+0000) 2024-03-06T13:52:21.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:21.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:21.052395+0000) 2024-03-06T13:52:22.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:22.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:22.052644+0000) 2024-03-06T13:52:23.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:23.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:23.052844+0000) 2024-03-06T13:52:24.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:24.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:24.053126+0000) 2024-03-06T13:52:25.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:25.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:25.053368+0000) 2024-03-06T13:52:26.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:26.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:26.053622+0000) 2024-03-06T13:52:27.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:27.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:27.053817+0000) 2024-03-06T13:52:28.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:28.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:28.054098+0000) 2024-03-06T13:52:29.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:29.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:29.054335+0000) 2024-03-06T13:52:30.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:30.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:30.054586+0000) 2024-03-06T13:52:31.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:31.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:31.054818+0000) 2024-03-06T13:52:32.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:32.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:32.055122+0000) 2024-03-06T13:52:33.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:33.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:33.055352+0000) 2024-03-06T13:52:34.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:34.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:34.055559+0000) 2024-03-06T13:52:35.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:35.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:35.055812+0000) 2024-03-06T13:52:36.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:36.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:36.056071+0000) 2024-03-06T13:52:37.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:37.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:37.056282+0000) 2024-03-06T13:52:38.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:38.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:38.056509+0000) 2024-03-06T13:52:39.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:39.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:39.056729+0000) 2024-03-06T13:52:40.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:40.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:40.056954+0000) 2024-03-06T13:52:41.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:41.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:41.057204+0000) 2024-03-06T13:52:42.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:42.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:42.057391+0000) 2024-03-06T13:52:42.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:42.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:52:43.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:43.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:43.057624+0000) 2024-03-06T13:52:44.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:44.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:44.057864+0000) 2024-03-06T13:52:45.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:45.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:45.058127+0000) 2024-03-06T13:52:46.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:46.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:46.058315+0000) 2024-03-06T13:52:47.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:47.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:47.058494+0000) 2024-03-06T13:52:48.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:48.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:48.058716+0000) 2024-03-06T13:52:49.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:49.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:49.058963+0000) 2024-03-06T13:52:50.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:50.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:50.059162+0000) 2024-03-06T13:52:51.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:51.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:51.059395+0000) 2024-03-06T13:52:52.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:52.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:52.059667+0000) 2024-03-06T13:52:53.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:53.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:53.059939+0000) 2024-03-06T13:52:54.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:54.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:54.060178+0000) 2024-03-06T13:52:55.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:55.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:55.060403+0000) 2024-03-06T13:52:56.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:56.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:56.060640+0000) 2024-03-06T13:52:57.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:57.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:57.060908+0000) 2024-03-06T13:52:58.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:58.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:58.061175+0000) 2024-03-06T13:52:59.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:52:59.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:52:59.061409+0000) 2024-03-06T13:53:00.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:00.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:00.061614+0000) 2024-03-06T13:53:01.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:01.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:01.061856+0000) 2024-03-06T13:53:02.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:02.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:02.062101+0000) 2024-03-06T13:53:03.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:03.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:03.062321+0000) 2024-03-06T13:53:04.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:04.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:04.062562+0000) 2024-03-06T13:53:05.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:05.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:05.062754+0000) 2024-03-06T13:53:06.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:06.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:06.062997+0000) 2024-03-06T13:53:07.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:07.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:07.063289+0000) 2024-03-06T13:53:08.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:08.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:08.063528+0000) 2024-03-06T13:53:09.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:09.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:09.063785+0000) 2024-03-06T13:53:10.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:10.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:10.063995+0000) 2024-03-06T13:53:11.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:11.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:11.064290+0000) 2024-03-06T13:53:12.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:12.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:12.064511+0000) 2024-03-06T13:53:12.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:12.232 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:13.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:13.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:13.064743+0000) 2024-03-06T13:53:14.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:14.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:14.064979+0000) 2024-03-06T13:53:15.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:15.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:15.065251+0000) 2024-03-06T13:53:16.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:16.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:16.065496+0000) 2024-03-06T13:53:17.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:17.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:17.065704+0000) 2024-03-06T13:53:18.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:18.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:18.065895+0000) 2024-03-06T13:53:19.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:19.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:19.066134+0000) 2024-03-06T13:53:20.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:20.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:20.066336+0000) 2024-03-06T13:53:21.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:21.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:21.066563+0000) 2024-03-06T13:53:22.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:22.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:22.066790+0000) 2024-03-06T13:53:23.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:23.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:23.066974+0000) 2024-03-06T13:53:24.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:24.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:24.067212+0000) 2024-03-06T13:53:25.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:25.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:25.067397+0000) 2024-03-06T13:53:26.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:26.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:26.067617+0000) 2024-03-06T13:53:27.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:27.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:27.067852+0000) 2024-03-06T13:53:28.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:28.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:28.068103+0000) 2024-03-06T13:53:29.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:29.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:29.068294+0000) 2024-03-06T13:53:30.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:30.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:30.068502+0000) 2024-03-06T13:53:31.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:31.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:31.068713+0000) 2024-03-06T13:53:32.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:32.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:32.068905+0000) 2024-03-06T13:53:33.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:33.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:33.069136+0000) 2024-03-06T13:53:34.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:34.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:34.069334+0000) 2024-03-06T13:53:35.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:35.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:35.069578+0000) 2024-03-06T13:53:36.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:36.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:36.069850+0000) 2024-03-06T13:53:37.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:37.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:37.070129+0000) 2024-03-06T13:53:38.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:38.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:38.070363+0000) 2024-03-06T13:53:39.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:39.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:39.070582+0000) 2024-03-06T13:53:40.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:40.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:40.070785+0000) 2024-03-06T13:53:41.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:41.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:41.071053+0000) 2024-03-06T13:53:42.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:42.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:42.071250+0000) 2024-03-06T13:53:42.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:42.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:53:43.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:43.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:43.071467+0000) 2024-03-06T13:53:44.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:44.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:44.071716+0000) 2024-03-06T13:53:45.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:45.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:45.071932+0000) 2024-03-06T13:53:46.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:46.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:46.072177+0000) 2024-03-06T13:53:47.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:47.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:47.072379+0000) 2024-03-06T13:53:48.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:48.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:48.072587+0000) 2024-03-06T13:53:49.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:49.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:49.072820+0000) 2024-03-06T13:53:50.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:50.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:50.072995+0000) 2024-03-06T13:53:51.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:51.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:51.073273+0000) 2024-03-06T13:53:52.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:52.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:52.073502+0000) 2024-03-06T13:53:53.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:53.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:53.073728+0000) 2024-03-06T13:53:54.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:54.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:54.073970+0000) 2024-03-06T13:53:55.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:55.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:55.074166+0000) 2024-03-06T13:53:56.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:56.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:56.074379+0000) 2024-03-06T13:53:57.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:57.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:57.074609+0000) 2024-03-06T13:53:58.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:58.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:58.074825+0000) 2024-03-06T13:53:59.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:53:59.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:53:59.075088+0000) 2024-03-06T13:54:00.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:00.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:00.075324+0000) 2024-03-06T13:54:01.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:01.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:01.075566+0000) 2024-03-06T13:54:02.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:02.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:02.075789+0000) 2024-03-06T13:54:03.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:03.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:03.076036+0000) 2024-03-06T13:54:04.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:04.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:04.076289+0000) 2024-03-06T13:54:05.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:05.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:05.076470+0000) 2024-03-06T13:54:06.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:06.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:06.076689+0000) 2024-03-06T13:54:07.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:07.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:07.076917+0000) 2024-03-06T13:54:08.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:08.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:08.077130+0000) 2024-03-06T13:54:09.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:09.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:09.077320+0000) 2024-03-06T13:54:10.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:10.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:10.077559+0000) 2024-03-06T13:54:11.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:11.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:11.077798+0000) 2024-03-06T13:54:12.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:12.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:12.078069+0000) 2024-03-06T13:54:12.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:12.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:13.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:13.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:13.078274+0000) 2024-03-06T13:54:14.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:14.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:14.078493+0000) 2024-03-06T13:54:15.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:15.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:15.078697+0000) 2024-03-06T13:54:16.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:16.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:16.078885+0000) 2024-03-06T13:54:17.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:17.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:17.079113+0000) 2024-03-06T13:54:18.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:18.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:18.079320+0000) 2024-03-06T13:54:19.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:19.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:19.079569+0000) 2024-03-06T13:54:20.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:20.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:20.079809+0000) 2024-03-06T13:54:21.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:21.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:21.080074+0000) 2024-03-06T13:54:22.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:22.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:22.080268+0000) 2024-03-06T13:54:23.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:23.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:23.080495+0000) 2024-03-06T13:54:24.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:24.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:24.080694+0000) 2024-03-06T13:54:25.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:25.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:25.080922+0000) 2024-03-06T13:54:26.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:26.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:26.081146+0000) 2024-03-06T13:54:27.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:27.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:27.081347+0000) 2024-03-06T13:54:28.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:28.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:28.081533+0000) 2024-03-06T13:54:29.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:29.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:29.081745+0000) 2024-03-06T13:54:30.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:30.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:30.081931+0000) 2024-03-06T13:54:31.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:31.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:31.082199+0000) 2024-03-06T13:54:32.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:32.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:32.082446+0000) 2024-03-06T13:54:33.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:33.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:33.082678+0000) 2024-03-06T13:54:34.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:34.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:34.082945+0000) 2024-03-06T13:54:35.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:35.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:35.083171+0000) 2024-03-06T13:54:36.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:36.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:36.083392+0000) 2024-03-06T13:54:37.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:37.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:37.083642+0000) 2024-03-06T13:54:38.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:38.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:38.083840+0000) 2024-03-06T13:54:39.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:39.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:39.084108+0000) 2024-03-06T13:54:40.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:40.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:40.084398+0000) 2024-03-06T13:54:41.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:41.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:41.084622+0000) 2024-03-06T13:54:42.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:42.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:42.084806+0000) 2024-03-06T13:54:42.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:42.808 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:54:43.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:43.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:43.085057+0000) 2024-03-06T13:54:44.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:44.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:44.085260+0000) 2024-03-06T13:54:45.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:45.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:45.085473+0000) 2024-03-06T13:54:46.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:46.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:46.085693+0000) 2024-03-06T13:54:47.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:47.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:47.085920+0000) 2024-03-06T13:54:48.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:48.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:48.086164+0000) 2024-03-06T13:54:49.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:49.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:49.086374+0000) 2024-03-06T13:54:50.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:50.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:50.086556+0000) 2024-03-06T13:54:51.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:51.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:51.086781+0000) 2024-03-06T13:54:52.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:52.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:52.087047+0000) 2024-03-06T13:54:53.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:53.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:53.087314+0000) 2024-03-06T13:54:54.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:54.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:54.087529+0000) 2024-03-06T13:54:55.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:55.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:55.087737+0000) 2024-03-06T13:54:56.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:56.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:56.087972+0000) 2024-03-06T13:54:57.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:57.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:57.088257+0000) 2024-03-06T13:54:58.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:58.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:58.088514+0000) 2024-03-06T13:54:59.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:54:59.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:54:59.088757+0000) 2024-03-06T13:55:00.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:00.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:00.088969+0000) 2024-03-06T13:55:01.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:01.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:01.089257+0000) 2024-03-06T13:55:02.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:02.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:02.089499+0000) 2024-03-06T13:55:03.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:03.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:03.089706+0000) 2024-03-06T13:55:04.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:04.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:04.089939+0000) 2024-03-06T13:55:05.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:05.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:05.090151+0000) 2024-03-06T13:55:06.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:06.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:06.090338+0000) 2024-03-06T13:55:07.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:07.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:07.090561+0000) 2024-03-06T13:55:08.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:08.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:08.090770+0000) 2024-03-06T13:55:09.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:09.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:09.090989+0000) 2024-03-06T13:55:10.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:10.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:10.091199+0000) 2024-03-06T13:55:11.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:11.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:11.091441+0000) 2024-03-06T13:55:12.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:12.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:12.091683+0000) 2024-03-06T13:55:13.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:13.024 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:13.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:13.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:13.091877+0000) 2024-03-06T13:55:14.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:14.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:14.092114+0000) 2024-03-06T13:55:15.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:15.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:15.092352+0000) 2024-03-06T13:55:16.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:16.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:16.092549+0000) 2024-03-06T13:55:17.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:17.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:17.092745+0000) 2024-03-06T13:55:18.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:18.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:18.092974+0000) 2024-03-06T13:55:19.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:19.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:19.093247+0000) 2024-03-06T13:55:20.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:20.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:20.093451+0000) 2024-03-06T13:55:21.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:21.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:21.093699+0000) 2024-03-06T13:55:22.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:22.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:22.093953+0000) 2024-03-06T13:55:23.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:23.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:23.094212+0000) 2024-03-06T13:55:24.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:24.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:24.094466+0000) 2024-03-06T13:55:25.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:25.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:25.094725+0000) 2024-03-06T13:55:26.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:26.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:26.094982+0000) 2024-03-06T13:55:27.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:27.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:27.095282+0000) 2024-03-06T13:55:28.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:28.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:28.095528+0000) 2024-03-06T13:55:29.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:29.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:29.095775+0000) 2024-03-06T13:55:30.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:30.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:30.096049+0000) 2024-03-06T13:55:31.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:31.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:31.096270+0000) 2024-03-06T13:55:32.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:32.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:32.096483+0000) 2024-03-06T13:55:33.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:33.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:33.096714+0000) 2024-03-06T13:55:34.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:34.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:34.096943+0000) 2024-03-06T13:55:35.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:35.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:35.097156+0000) 2024-03-06T13:55:36.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:36.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:36.097349+0000) 2024-03-06T13:55:37.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:37.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:37.097552+0000) 2024-03-06T13:55:38.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:38.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:38.097806+0000) 2024-03-06T13:55:39.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:39.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:39.098096+0000) 2024-03-06T13:55:40.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:40.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:40.098306+0000) 2024-03-06T13:55:41.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:41.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:41.098543+0000) 2024-03-06T13:55:42.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:42.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:42.098778+0000) 2024-03-06T13:55:43.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:43.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:43.099061+0000) 2024-03-06T13:55:43.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:43.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:55:44.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:44.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:44.099299+0000) 2024-03-06T13:55:45.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:45.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:45.099544+0000) 2024-03-06T13:55:46.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:46.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:46.099803+0000) 2024-03-06T13:55:47.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:47.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:47.100041+0000) 2024-03-06T13:55:48.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:48.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:48.100239+0000) 2024-03-06T13:55:49.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:49.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:49.100438+0000) 2024-03-06T13:55:50.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:50.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:50.100652+0000) 2024-03-06T13:55:51.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:51.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:51.100880+0000) 2024-03-06T13:55:52.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:52.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:52.101147+0000) 2024-03-06T13:55:53.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:53.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:53.101389+0000) 2024-03-06T13:55:54.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:54.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:54.101644+0000) 2024-03-06T13:55:55.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:55.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:55.101877+0000) 2024-03-06T13:55:56.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:56.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:56.102141+0000) 2024-03-06T13:55:57.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:57.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:57.102367+0000) 2024-03-06T13:55:58.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:58.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:58.102568+0000) 2024-03-06T13:55:59.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:55:59.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:55:59.102822+0000) 2024-03-06T13:56:00.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:00.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:00.103097+0000) 2024-03-06T13:56:01.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:01.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:01.103316+0000) 2024-03-06T13:56:02.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:02.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:02.103532+0000) 2024-03-06T13:56:03.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:03.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:03.103771+0000) 2024-03-06T13:56:04.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:04.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:04.104001+0000) 2024-03-06T13:56:05.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:05.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:05.104239+0000) 2024-03-06T13:56:06.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:06.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:06.104447+0000) 2024-03-06T13:56:07.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:07.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:07.104713+0000) 2024-03-06T13:56:08.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:08.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:08.104950+0000) 2024-03-06T13:56:09.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:09.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:09.105155+0000) 2024-03-06T13:56:10.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:10.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:10.105402+0000) 2024-03-06T13:56:11.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:11.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:11.105596+0000) 2024-03-06T13:56:12.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:12.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:12.105837+0000) 2024-03-06T13:56:13.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:13.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:13.106113+0000) 2024-03-06T13:56:13.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:13.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:14.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:14.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:14.106349+0000) 2024-03-06T13:56:15.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:15.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:15.106592+0000) 2024-03-06T13:56:16.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:16.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:16.106788+0000) 2024-03-06T13:56:17.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:17.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:17.107043+0000) 2024-03-06T13:56:18.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:18.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:18.107282+0000) 2024-03-06T13:56:19.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:19.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:19.107521+0000) 2024-03-06T13:56:20.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:20.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:20.107742+0000) 2024-03-06T13:56:21.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:21.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:21.107969+0000) 2024-03-06T13:56:22.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:22.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:22.108177+0000) 2024-03-06T13:56:23.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:23.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:23.108365+0000) 2024-03-06T13:56:24.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:24.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:24.108579+0000) 2024-03-06T13:56:25.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:25.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:25.108799+0000) 2024-03-06T13:56:26.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:26.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:26.109046+0000) 2024-03-06T13:56:27.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:27.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:27.109248+0000) 2024-03-06T13:56:28.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:28.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:28.109473+0000) 2024-03-06T13:56:29.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:29.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:29.109707+0000) 2024-03-06T13:56:30.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:30.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:30.109933+0000) 2024-03-06T13:56:31.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:31.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:31.110163+0000) 2024-03-06T13:56:32.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:32.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:32.110388+0000) 2024-03-06T13:56:33.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:33.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:33.110570+0000) 2024-03-06T13:56:34.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:34.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:34.110775+0000) 2024-03-06T13:56:35.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:35.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:35.111044+0000) 2024-03-06T13:56:36.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:36.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:36.111295+0000) 2024-03-06T13:56:37.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:37.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:37.111540+0000) 2024-03-06T13:56:38.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:38.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:38.111788+0000) 2024-03-06T13:56:39.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:39.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:39.112075+0000) 2024-03-06T13:56:40.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:40.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:40.112316+0000) 2024-03-06T13:56:41.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:41.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:41.112549+0000) 2024-03-06T13:56:42.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:42.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:42.112797+0000) 2024-03-06T13:56:43.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:43.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:43.113084+0000) 2024-03-06T13:56:43.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:43.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:56:44.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:44.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:44.113347+0000) 2024-03-06T13:56:45.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:45.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:45.113555+0000) 2024-03-06T13:56:46.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:46.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:46.113774+0000) 2024-03-06T13:56:47.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:47.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:47.114050+0000) 2024-03-06T13:56:48.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:48.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:48.114265+0000) 2024-03-06T13:56:49.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:49.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:49.114503+0000) 2024-03-06T13:56:50.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:50.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:50.114744+0000) 2024-03-06T13:56:51.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:51.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:51.114987+0000) 2024-03-06T13:56:52.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:52.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:52.115297+0000) 2024-03-06T13:56:53.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:53.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:53.115547+0000) 2024-03-06T13:56:54.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:54.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:54.115801+0000) 2024-03-06T13:56:55.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:55.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:55.115999+0000) 2024-03-06T13:56:56.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:56.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:56.116229+0000) 2024-03-06T13:56:57.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:57.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:57.116463+0000) 2024-03-06T13:56:58.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:58.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:58.116668+0000) 2024-03-06T13:56:59.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:56:59.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:56:59.116887+0000) 2024-03-06T13:57:00.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:00.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:00.117108+0000) 2024-03-06T13:57:01.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:01.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:01.117347+0000) 2024-03-06T13:57:02.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:02.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:02.117577+0000) 2024-03-06T13:57:03.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:03.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:03.117800+0000) 2024-03-06T13:57:04.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:04.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:04.118122+0000) 2024-03-06T13:57:05.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:05.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:05.118310+0000) 2024-03-06T13:57:06.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:06.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:06.118529+0000) 2024-03-06T13:57:07.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:07.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:07.118726+0000) 2024-03-06T13:57:08.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:08.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:08.118948+0000) 2024-03-06T13:57:09.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:09.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:09.119181+0000) 2024-03-06T13:57:10.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:10.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:10.119400+0000) 2024-03-06T13:57:11.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:11.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:11.119616+0000) 2024-03-06T13:57:12.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:12.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:12.119817+0000) 2024-03-06T13:57:13.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:13.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:13.120116+0000) 2024-03-06T13:57:13.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:13.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:14.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:14.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:14.120408+0000) 2024-03-06T13:57:15.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:15.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:15.120640+0000) 2024-03-06T13:57:16.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:16.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:16.120873+0000) 2024-03-06T13:57:17.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:17.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:17.121140+0000) 2024-03-06T13:57:18.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:18.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:18.121340+0000) 2024-03-06T13:57:19.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:19.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:19.121548+0000) 2024-03-06T13:57:20.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:20.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:20.121812+0000) 2024-03-06T13:57:21.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:21.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:21.122072+0000) 2024-03-06T13:57:22.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:22.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:22.122340+0000) 2024-03-06T13:57:23.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:23.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:23.122562+0000) 2024-03-06T13:57:24.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:24.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:24.122822+0000) 2024-03-06T13:57:25.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:25.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:25.123093+0000) 2024-03-06T13:57:26.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:26.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:26.123333+0000) 2024-03-06T13:57:27.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:27.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:27.123579+0000) 2024-03-06T13:57:28.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:28.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:28.123790+0000) 2024-03-06T13:57:29.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:29.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:29.124063+0000) 2024-03-06T13:57:30.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:30.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:30.124290+0000) 2024-03-06T13:57:31.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:31.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:31.124519+0000) 2024-03-06T13:57:32.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:32.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:32.124708+0000) 2024-03-06T13:57:33.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:33.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:33.124945+0000) 2024-03-06T13:57:34.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:34.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:34.125206+0000) 2024-03-06T13:57:35.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:35.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:35.125428+0000) 2024-03-06T13:57:36.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:36.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:36.125660+0000) 2024-03-06T13:57:37.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:37.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:37.125900+0000) 2024-03-06T13:57:38.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:38.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:38.126124+0000) 2024-03-06T13:57:39.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:39.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:39.126366+0000) 2024-03-06T13:57:40.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:40.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:40.126575+0000) 2024-03-06T13:57:41.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:41.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:41.126777+0000) 2024-03-06T13:57:42.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:42.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:42.126999+0000) 2024-03-06T13:57:43.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:43.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:43.127246+0000) 2024-03-06T13:57:44.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:44.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:57:44.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:44.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:44.127499+0000) 2024-03-06T13:57:45.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:45.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:45.127692+0000) 2024-03-06T13:57:46.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:46.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:46.127927+0000) 2024-03-06T13:57:47.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:47.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:47.128178+0000) 2024-03-06T13:57:48.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:48.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:48.128403+0000) 2024-03-06T13:57:49.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:49.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:49.128642+0000) 2024-03-06T13:57:50.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:50.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:50.128822+0000) 2024-03-06T13:57:51.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:51.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:51.129044+0000) 2024-03-06T13:57:52.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:52.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:52.129292+0000) 2024-03-06T13:57:53.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:53.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:53.129533+0000) 2024-03-06T13:57:54.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:54.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:54.129751+0000) 2024-03-06T13:57:55.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:55.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:55.129938+0000) 2024-03-06T13:57:56.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:56.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:56.130170+0000) 2024-03-06T13:57:57.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:57.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:57.130442+0000) 2024-03-06T13:57:58.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:58.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:58.130665+0000) 2024-03-06T13:57:59.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:57:59.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:57:59.130895+0000) 2024-03-06T13:58:00.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:00.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:00.131116+0000) 2024-03-06T13:58:01.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:01.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:01.131304+0000) 2024-03-06T13:58:02.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:02.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:02.131505+0000) 2024-03-06T13:58:03.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:03.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:03.131732+0000) 2024-03-06T13:58:04.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:04.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:04.131923+0000) 2024-03-06T13:58:05.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:05.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:05.132159+0000) 2024-03-06T13:58:06.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:06.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:06.132398+0000) 2024-03-06T13:58:07.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:07.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:07.132659+0000) 2024-03-06T13:58:08.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:08.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:08.132886+0000) 2024-03-06T13:58:09.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:09.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:09.133111+0000) 2024-03-06T13:58:10.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:10.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:10.133334+0000) 2024-03-06T13:58:11.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:11.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:11.133556+0000) 2024-03-06T13:58:12.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:12.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:12.133812+0000) 2024-03-06T13:58:13.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:13.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:13.134057+0000) 2024-03-06T13:58:14.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:14.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:14.134296+0000) 2024-03-06T13:58:14.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:14.251 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:15.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:15.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:15.134480+0000) 2024-03-06T13:58:16.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:16.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:16.134699+0000) 2024-03-06T13:58:17.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:17.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:17.134970+0000) 2024-03-06T13:58:18.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:18.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:18.135178+0000) 2024-03-06T13:58:19.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:19.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:19.135409+0000) 2024-03-06T13:58:20.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:20.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:20.135656+0000) 2024-03-06T13:58:21.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:21.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:21.135889+0000) 2024-03-06T13:58:22.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:22.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:22.136134+0000) 2024-03-06T13:58:23.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:23.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:23.136335+0000) 2024-03-06T13:58:24.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:24.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:24.136558+0000) 2024-03-06T13:58:25.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:25.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:25.136782+0000) 2024-03-06T13:58:26.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:26.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:26.136996+0000) 2024-03-06T13:58:27.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:27.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:27.137333+0000) 2024-03-06T13:58:28.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:28.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:28.137576+0000) 2024-03-06T13:58:29.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:29.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:29.137798+0000) 2024-03-06T13:58:30.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:30.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:30.138071+0000) 2024-03-06T13:58:31.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:31.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:31.138302+0000) 2024-03-06T13:58:32.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:32.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:32.138504+0000) 2024-03-06T13:58:33.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:33.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:33.138722+0000) 2024-03-06T13:58:34.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:34.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:34.138925+0000) 2024-03-06T13:58:35.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:35.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:35.139153+0000) 2024-03-06T13:58:36.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:36.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:36.139377+0000) 2024-03-06T13:58:37.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:37.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:37.139621+0000) 2024-03-06T13:58:38.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:38.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:38.139846+0000) 2024-03-06T13:58:39.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:39.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:39.140114+0000) 2024-03-06T13:58:40.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:40.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:40.140348+0000) 2024-03-06T13:58:41.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:41.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:41.140549+0000) 2024-03-06T13:58:42.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:42.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:42.140788+0000) 2024-03-06T13:58:43.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:43.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:43.141055+0000) 2024-03-06T13:58:44.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:44.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:44.141293+0000) 2024-03-06T13:58:44.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:44.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:58:45.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:45.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:45.141509+0000) 2024-03-06T13:58:46.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:46.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:46.141761+0000) 2024-03-06T13:58:47.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:47.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:47.141963+0000) 2024-03-06T13:58:48.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:48.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:48.142152+0000) 2024-03-06T13:58:49.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:49.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:49.142376+0000) 2024-03-06T13:58:50.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:50.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:50.142650+0000) 2024-03-06T13:58:51.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:51.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:51.142876+0000) 2024-03-06T13:58:52.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:52.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:52.143184+0000) 2024-03-06T13:58:53.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:53.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:53.143446+0000) 2024-03-06T13:58:54.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:54.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:54.143662+0000) 2024-03-06T13:58:55.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:55.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:55.143897+0000) 2024-03-06T13:58:56.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:56.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:56.144136+0000) 2024-03-06T13:58:57.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:57.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:57.144318+0000) 2024-03-06T13:58:58.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:58.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:58.144530+0000) 2024-03-06T13:58:59.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:58:59.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:58:59.144767+0000) 2024-03-06T13:59:00.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:00.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:00.144996+0000) 2024-03-06T13:59:01.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:01.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:01.145252+0000) 2024-03-06T13:59:02.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:02.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:02.145479+0000) 2024-03-06T13:59:03.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:03.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:03.145710+0000) 2024-03-06T13:59:04.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:04.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:04.145948+0000) 2024-03-06T13:59:05.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:05.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:05.146187+0000) 2024-03-06T13:59:06.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:06.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:06.146446+0000) 2024-03-06T13:59:07.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:07.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:07.146701+0000) 2024-03-06T13:59:08.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:08.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:08.146933+0000) 2024-03-06T13:59:09.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:09.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:09.147162+0000) 2024-03-06T13:59:10.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:10.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:10.147394+0000) 2024-03-06T13:59:11.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:11.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:11.147636+0000) 2024-03-06T13:59:12.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:12.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:12.147875+0000) 2024-03-06T13:59:13.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:13.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:13.148138+0000) 2024-03-06T13:59:14.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:14.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:14.148372+0000) 2024-03-06T13:59:14.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:14.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:15.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:15.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:15.148606+0000) 2024-03-06T13:59:16.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:16.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:16.148790+0000) 2024-03-06T13:59:17.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:17.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:17.149040+0000) 2024-03-06T13:59:18.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:18.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:18.149283+0000) 2024-03-06T13:59:19.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:19.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:19.149456+0000) 2024-03-06T13:59:20.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:20.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:20.149677+0000) 2024-03-06T13:59:21.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:21.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:21.149907+0000) 2024-03-06T13:59:22.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:22.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:22.150165+0000) 2024-03-06T13:59:23.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:23.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:23.150419+0000) 2024-03-06T13:59:24.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:24.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:24.150656+0000) 2024-03-06T13:59:25.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:25.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:25.150888+0000) 2024-03-06T13:59:26.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:26.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:26.151166+0000) 2024-03-06T13:59:27.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:27.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:27.151400+0000) 2024-03-06T13:59:28.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:28.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:28.151620+0000) 2024-03-06T13:59:29.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:29.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:29.151807+0000) 2024-03-06T13:59:30.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:30.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:30.152069+0000) 2024-03-06T13:59:31.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:31.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:31.152260+0000) 2024-03-06T13:59:32.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:32.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:32.152491+0000) 2024-03-06T13:59:33.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:33.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:33.152681+0000) 2024-03-06T13:59:34.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:34.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:34.152912+0000) 2024-03-06T13:59:35.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:35.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:35.153146+0000) 2024-03-06T13:59:36.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:36.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:36.153367+0000) 2024-03-06T13:59:37.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:37.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:37.153590+0000) 2024-03-06T13:59:38.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:38.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:38.153905+0000) 2024-03-06T13:59:39.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:39.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:39.154167+0000) 2024-03-06T13:59:40.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:40.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:40.154393+0000) 2024-03-06T13:59:41.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:41.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:41.154623+0000) 2024-03-06T13:59:42.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:42.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:42.154832+0000) 2024-03-06T13:59:43.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:43.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:43.155107+0000) 2024-03-06T13:59:44.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:44.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:44.155340+0000) 2024-03-06T13:59:44.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:44.880 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T13:59:45.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:45.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:45.155540+0000) 2024-03-06T13:59:46.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:46.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:46.155778+0000) 2024-03-06T13:59:47.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:47.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:47.156038+0000) 2024-03-06T13:59:48.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:48.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:48.156270+0000) 2024-03-06T13:59:49.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:49.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:49.156506+0000) 2024-03-06T13:59:50.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:50.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:50.156728+0000) 2024-03-06T13:59:51.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:51.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:51.156950+0000) 2024-03-06T13:59:52.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:52.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:52.157179+0000) 2024-03-06T13:59:53.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:53.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:53.157410+0000) 2024-03-06T13:59:54.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:54.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:54.157658+0000) 2024-03-06T13:59:55.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:55.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:55.157881+0000) 2024-03-06T13:59:56.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:56.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:56.158147+0000) 2024-03-06T13:59:57.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:57.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:57.158372+0000) 2024-03-06T13:59:58.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:58.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:58.158584+0000) 2024-03-06T13:59:59.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T13:59:59.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T12:59:59.158767+0000) 2024-03-06T14:00:00.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:00.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:00.158991+0000) 2024-03-06T14:00:01.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:01.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:01.159243+0000) 2024-03-06T14:00:02.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:02.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:02.159481+0000) 2024-03-06T14:00:03.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:03.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:03.159712+0000) 2024-03-06T14:00:04.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:04.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:04.159937+0000) 2024-03-06T14:00:05.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:05.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:05.160175+0000) 2024-03-06T14:00:06.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:06.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:06.160369+0000) 2024-03-06T14:00:07.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:07.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:07.160615+0000) 2024-03-06T14:00:08.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:08.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:08.160793+0000) 2024-03-06T14:00:09.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:09.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:09.161038+0000) 2024-03-06T14:00:10.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:10.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:10.161262+0000) 2024-03-06T14:00:11.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:11.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:11.161487+0000) 2024-03-06T14:00:12.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:12.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:12.161686+0000) 2024-03-06T14:00:13.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:13.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:13.161939+0000) 2024-03-06T14:00:14.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:14.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:14.162202+0000) 2024-03-06T14:00:15.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:15.091 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:15.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:15.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:15.162385+0000) 2024-03-06T14:00:16.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:16.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:16.162598+0000) 2024-03-06T14:00:17.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:17.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:17.162849+0000) 2024-03-06T14:00:18.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:18.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:18.163158+0000) 2024-03-06T14:00:19.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:19.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:19.163413+0000) 2024-03-06T14:00:20.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:20.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:20.163614+0000) 2024-03-06T14:00:21.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:21.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:21.163860+0000) 2024-03-06T14:00:22.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:22.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:22.164134+0000) 2024-03-06T14:00:23.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:23.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:23.164366+0000) 2024-03-06T14:00:24.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:24.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:24.164593+0000) 2024-03-06T14:00:25.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:25.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:25.164844+0000) 2024-03-06T14:00:26.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:26.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:26.165162+0000) 2024-03-06T14:00:27.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:27.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:27.165395+0000) 2024-03-06T14:00:28.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:28.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:28.165621+0000) 2024-03-06T14:00:29.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:29.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:29.165847+0000) 2024-03-06T14:00:30.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:30.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:30.166110+0000) 2024-03-06T14:00:31.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:31.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:31.166330+0000) 2024-03-06T14:00:32.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:32.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:32.166550+0000) 2024-03-06T14:00:33.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:33.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:33.166784+0000) 2024-03-06T14:00:34.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:34.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:34.167047+0000) 2024-03-06T14:00:35.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:35.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:35.167271+0000) 2024-03-06T14:00:36.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:36.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:36.167442+0000) 2024-03-06T14:00:37.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:37.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:37.167676+0000) 2024-03-06T14:00:38.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:38.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:38.167904+0000) 2024-03-06T14:00:39.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:39.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:39.168124+0000) 2024-03-06T14:00:40.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:40.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:40.168371+0000) 2024-03-06T14:00:41.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:41.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:41.168574+0000) 2024-03-06T14:00:42.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:42.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:42.168802+0000) 2024-03-06T14:00:43.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:43.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:43.168997+0000) 2024-03-06T14:00:44.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:44.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:44.169266+0000) 2024-03-06T14:00:45.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:45.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:45.169518+0000) 2024-03-06T14:00:45.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:45.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:00:46.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:46.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:46.169744+0000) 2024-03-06T14:00:47.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:47.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:47.169964+0000) 2024-03-06T14:00:48.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:48.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:48.170179+0000) 2024-03-06T14:00:49.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:49.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:49.170378+0000) 2024-03-06T14:00:50.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:50.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:50.170599+0000) 2024-03-06T14:00:51.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:51.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:51.170808+0000) 2024-03-06T14:00:52.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:52.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:52.171096+0000) 2024-03-06T14:00:53.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:53.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:53.171330+0000) 2024-03-06T14:00:54.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:54.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:54.171565+0000) 2024-03-06T14:00:55.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:55.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:55.171788+0000) 2024-03-06T14:00:56.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:56.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:56.172044+0000) 2024-03-06T14:00:57.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:57.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:57.172308+0000) 2024-03-06T14:00:58.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:58.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:58.172538+0000) 2024-03-06T14:00:59.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:00:59.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:00:59.172746+0000) 2024-03-06T14:01:00.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:00.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:00.172976+0000) 2024-03-06T14:01:01.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:01.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:01.173202+0000) 2024-03-06T14:01:02.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:02.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:02.173434+0000) 2024-03-06T14:01:03.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:03.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:03.173654+0000) 2024-03-06T14:01:04.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:04.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:04.173859+0000) 2024-03-06T14:01:05.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:05.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:05.174130+0000) 2024-03-06T14:01:06.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:06.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:06.174357+0000) 2024-03-06T14:01:07.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:07.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:07.174581+0000) 2024-03-06T14:01:08.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:08.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:08.174798+0000) 2024-03-06T14:01:09.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:09.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:09.175065+0000) 2024-03-06T14:01:10.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:10.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:10.175289+0000) 2024-03-06T14:01:11.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:11.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:11.175469+0000) 2024-03-06T14:01:12.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:12.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:12.175681+0000) 2024-03-06T14:01:13.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:13.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:13.175866+0000) 2024-03-06T14:01:14.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:14.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:14.176134+0000) 2024-03-06T14:01:15.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:15.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:15.176359+0000) 2024-03-06T14:01:15.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:15.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:16.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:16.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:16.176585+0000) 2024-03-06T14:01:17.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:17.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:17.176800+0000) 2024-03-06T14:01:18.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:18.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:18.177057+0000) 2024-03-06T14:01:19.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:19.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:19.177267+0000) 2024-03-06T14:01:20.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:20.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:20.177504+0000) 2024-03-06T14:01:21.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:21.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:21.177724+0000) 2024-03-06T14:01:22.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:22.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:22.177942+0000) 2024-03-06T14:01:23.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:23.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:23.178168+0000) 2024-03-06T14:01:24.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:24.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:24.178402+0000) 2024-03-06T14:01:25.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:25.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:25.178638+0000) 2024-03-06T14:01:26.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:26.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:26.178858+0000) 2024-03-06T14:01:27.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:27.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:27.179120+0000) 2024-03-06T14:01:28.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:28.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:28.179346+0000) 2024-03-06T14:01:29.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:29.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:29.179530+0000) 2024-03-06T14:01:30.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:30.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:30.179748+0000) 2024-03-06T14:01:31.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:31.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:31.179970+0000) 2024-03-06T14:01:32.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:32.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:32.180190+0000) 2024-03-06T14:01:33.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:33.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:33.180425+0000) 2024-03-06T14:01:34.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:34.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:34.180647+0000) 2024-03-06T14:01:35.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:35.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:35.180858+0000) 2024-03-06T14:01:36.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:36.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:36.181095+0000) 2024-03-06T14:01:37.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:37.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:37.181334+0000) 2024-03-06T14:01:38.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:38.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:38.181591+0000) 2024-03-06T14:01:39.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:39.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:39.181825+0000) 2024-03-06T14:01:40.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:40.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:40.182096+0000) 2024-03-06T14:01:41.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:41.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:41.182333+0000) 2024-03-06T14:01:42.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:42.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:42.182553+0000) 2024-03-06T14:01:43.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:43.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:43.182772+0000) 2024-03-06T14:01:44.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:44.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:44.183063+0000) 2024-03-06T14:01:45.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:45.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:45.183254+0000) 2024-03-06T14:01:45.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:45.632 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:01:46.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:46.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:46.183492+0000) 2024-03-06T14:01:47.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:47.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:47.183716+0000) 2024-03-06T14:01:48.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:48.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:48.183952+0000) 2024-03-06T14:01:49.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:49.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:49.184156+0000) 2024-03-06T14:01:50.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:50.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:50.184381+0000) 2024-03-06T14:01:51.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:51.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:51.184606+0000) 2024-03-06T14:01:52.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:52.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:52.184834+0000) 2024-03-06T14:01:53.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:53.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:53.185107+0000) 2024-03-06T14:01:54.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:54.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:54.185352+0000) 2024-03-06T14:01:55.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:55.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:55.185577+0000) 2024-03-06T14:01:56.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:56.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:56.185800+0000) 2024-03-06T14:01:57.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:57.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:57.186066+0000) 2024-03-06T14:01:58.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:58.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:58.186311+0000) 2024-03-06T14:01:59.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:01:59.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:01:59.186559+0000) 2024-03-06T14:02:00.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:00.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:00.186791+0000) 2024-03-06T14:02:01.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:01.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:01.187094+0000) 2024-03-06T14:02:02.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:02.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:02.187282+0000) 2024-03-06T14:02:03.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:03.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:03.187880+0000) 2024-03-06T14:02:04.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:04.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:04.188188+0000) 2024-03-06T14:02:05.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:05.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:05.188457+0000) 2024-03-06T14:02:06.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:06.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:06.188737+0000) 2024-03-06T14:02:07.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:07.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:07.189056+0000) 2024-03-06T14:02:08.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:08.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:08.189401+0000) 2024-03-06T14:02:09.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:09.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:09.189684+0000) 2024-03-06T14:02:10.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:10.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:10.189957+0000) 2024-03-06T14:02:11.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:11.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:11.190335+0000) 2024-03-06T14:02:12.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:12.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:12.190615+0000) 2024-03-06T14:02:13.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:13.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:13.190871+0000) 2024-03-06T14:02:14.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:14.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:14.191156+0000) 2024-03-06T14:02:15.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:15.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:15.191442+0000) 2024-03-06T14:02:15.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:15.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:16.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:16.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:16.191696+0000) 2024-03-06T14:02:17.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:17.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:17.191955+0000) 2024-03-06T14:02:18.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:18.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:18.192119+0000) 2024-03-06T14:02:19.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:19.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:19.192284+0000) 2024-03-06T14:02:20.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:20.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:20.192521+0000) 2024-03-06T14:02:21.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:21.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:21.192795+0000) 2024-03-06T14:02:22.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:22.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:22.193116+0000) 2024-03-06T14:02:23.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:23.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:23.193363+0000) 2024-03-06T14:02:24.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:24.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:24.193655+0000) 2024-03-06T14:02:25.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:25.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:25.193935+0000) 2024-03-06T14:02:26.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:26.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:26.194215+0000) 2024-03-06T14:02:27.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:27.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:27.194474+0000) 2024-03-06T14:02:28.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:28.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:28.194703+0000) 2024-03-06T14:02:29.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:29.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:29.194938+0000) 2024-03-06T14:02:30.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:30.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:30.195163+0000) 2024-03-06T14:02:31.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:31.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:31.195352+0000) 2024-03-06T14:02:32.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:32.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:32.195578+0000) 2024-03-06T14:02:33.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:33.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:33.195781+0000) 2024-03-06T14:02:34.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:34.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:34.196060+0000) 2024-03-06T14:02:35.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:35.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:35.196278+0000) 2024-03-06T14:02:36.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:36.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:36.196501+0000) 2024-03-06T14:02:37.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:37.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:37.196715+0000) 2024-03-06T14:02:38.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:38.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:38.196960+0000) 2024-03-06T14:02:39.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:39.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:39.197187+0000) 2024-03-06T14:02:40.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:40.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:40.197423+0000) 2024-03-06T14:02:41.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:41.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:41.197676+0000) 2024-03-06T14:02:42.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:42.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:42.197900+0000) 2024-03-06T14:02:43.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:43.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:43.198177+0000) 2024-03-06T14:02:44.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:44.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:44.198496+0000) 2024-03-06T14:02:45.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:45.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:45.198771+0000) 2024-03-06T14:02:46.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:46.071 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:02:46.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:46.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:46.198967+0000) 2024-03-06T14:02:47.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:47.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:47.199189+0000) 2024-03-06T14:02:48.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:48.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:48.199470+0000) 2024-03-06T14:02:49.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:49.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:49.199766+0000) 2024-03-06T14:02:50.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:50.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:50.200087+0000) 2024-03-06T14:02:51.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:51.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:51.200325+0000) 2024-03-06T14:02:52.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:52.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:52.200638+0000) 2024-03-06T14:02:53.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:53.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:53.200888+0000) 2024-03-06T14:02:54.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:54.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:54.201118+0000) 2024-03-06T14:02:55.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:55.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:55.201314+0000) 2024-03-06T14:02:56.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:56.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:56.201601+0000) 2024-03-06T14:02:57.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:57.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:57.201824+0000) 2024-03-06T14:02:58.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:58.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:58.202105+0000) 2024-03-06T14:02:59.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:02:59.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:02:59.202348+0000) 2024-03-06T14:03:00.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:00.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:00.202660+0000) 2024-03-06T14:03:01.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:01.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:01.202873+0000) 2024-03-06T14:03:02.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:02.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:02.203127+0000) 2024-03-06T14:03:03.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:03.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:03.203372+0000) 2024-03-06T14:03:04.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:04.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:04.203635+0000) 2024-03-06T14:03:05.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:05.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:05.203920+0000) 2024-03-06T14:03:06.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:06.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:06.204168+0000) 2024-03-06T14:03:07.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:07.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:07.204455+0000) 2024-03-06T14:03:08.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:08.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:08.204714+0000) 2024-03-06T14:03:09.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:09.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:09.204994+0000) 2024-03-06T14:03:10.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:10.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:10.205293+0000) 2024-03-06T14:03:11.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:11.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:11.205516+0000) 2024-03-06T14:03:12.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:12.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:12.205744+0000) 2024-03-06T14:03:13.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:13.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:13.205976+0000) 2024-03-06T14:03:14.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:14.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:14.206268+0000) 2024-03-06T14:03:15.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:15.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:15.206531+0000) 2024-03-06T14:03:16.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:16.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:16.206787+0000) 2024-03-06T14:03:16.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:16.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:17.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:17.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:17.207111+0000) 2024-03-06T14:03:18.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:18.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:18.207366+0000) 2024-03-06T14:03:19.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:19.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:19.207595+0000) 2024-03-06T14:03:20.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:20.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:20.207858+0000) 2024-03-06T14:03:21.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:21.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:21.208160+0000) 2024-03-06T14:03:22.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:22.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:22.208372+0000) 2024-03-06T14:03:23.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:23.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:23.208589+0000) 2024-03-06T14:03:24.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:24.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:24.208849+0000) 2024-03-06T14:03:25.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:25.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:25.209143+0000) 2024-03-06T14:03:26.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:26.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:26.209384+0000) 2024-03-06T14:03:27.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:27.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:27.209632+0000) 2024-03-06T14:03:28.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:28.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:28.209887+0000) 2024-03-06T14:03:29.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:29.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:29.210135+0000) 2024-03-06T14:03:30.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:30.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:30.210348+0000) 2024-03-06T14:03:31.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:31.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:31.210597+0000) 2024-03-06T14:03:32.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:32.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:32.210892+0000) 2024-03-06T14:03:33.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:33.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:33.211142+0000) 2024-03-06T14:03:34.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:34.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:34.211380+0000) 2024-03-06T14:03:35.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:35.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:35.211607+0000) 2024-03-06T14:03:36.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:36.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:36.211830+0000) 2024-03-06T14:03:37.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:37.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:37.212106+0000) 2024-03-06T14:03:38.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:38.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:38.212364+0000) 2024-03-06T14:03:39.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:39.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:39.212622+0000) 2024-03-06T14:03:40.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:40.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:40.212896+0000) 2024-03-06T14:03:41.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:41.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:41.213130+0000) 2024-03-06T14:03:42.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:42.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:42.213354+0000) 2024-03-06T14:03:43.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:43.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:43.213597+0000) 2024-03-06T14:03:44.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:44.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:44.213809+0000) 2024-03-06T14:03:45.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:45.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:45.214079+0000) 2024-03-06T14:03:46.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:46.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:46.214276+0000) 2024-03-06T14:03:46.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:46.496 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:03:47.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:47.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:47.214500+0000) 2024-03-06T14:03:48.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:48.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:48.214726+0000) 2024-03-06T14:03:49.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:49.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:49.215048+0000) 2024-03-06T14:03:50.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:50.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:50.215282+0000) 2024-03-06T14:03:51.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:51.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:51.215510+0000) 2024-03-06T14:03:52.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:52.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:52.215746+0000) 2024-03-06T14:03:53.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:53.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:53.215980+0000) 2024-03-06T14:03:54.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:54.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:54.216274+0000) 2024-03-06T14:03:55.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:55.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:55.216504+0000) 2024-03-06T14:03:56.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:56.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:56.216702+0000) 2024-03-06T14:03:57.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:57.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:57.216891+0000) 2024-03-06T14:03:58.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:58.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:58.217142+0000) 2024-03-06T14:03:59.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:03:59.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:03:59.217328+0000) 2024-03-06T14:04:00.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:00.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:00.217566+0000) 2024-03-06T14:04:01.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:01.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:01.217797+0000) 2024-03-06T14:04:02.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:02.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:02.218070+0000) 2024-03-06T14:04:03.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:03.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:03.218337+0000) 2024-03-06T14:04:04.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:04.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:04.218505+0000) 2024-03-06T14:04:05.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:05.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:05.218727+0000) 2024-03-06T14:04:06.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:06.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:06.218978+0000) 2024-03-06T14:04:07.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:07.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:07.219242+0000) 2024-03-06T14:04:08.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:08.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:08.219540+0000) 2024-03-06T14:04:09.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:09.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:09.219791+0000) 2024-03-06T14:04:10.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:10.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:10.220058+0000) 2024-03-06T14:04:11.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:11.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:11.220284+0000) 2024-03-06T14:04:12.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:12.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:12.220468+0000) 2024-03-06T14:04:13.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:13.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:13.220679+0000) 2024-03-06T14:04:14.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:14.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:14.220847+0000) 2024-03-06T14:04:15.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:15.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:15.221114+0000) 2024-03-06T14:04:16.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:16.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:16.221357+0000) 2024-03-06T14:04:16.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:16.650 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:17.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:17.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:17.221612+0000) 2024-03-06T14:04:18.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:18.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:18.221888+0000) 2024-03-06T14:04:19.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:19.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:19.222297+0000) 2024-03-06T14:04:20.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:20.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:20.222545+0000) 2024-03-06T14:04:21.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:21.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:21.222713+0000) 2024-03-06T14:04:22.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:22.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:22.222929+0000) 2024-03-06T14:04:23.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:23.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:23.223194+0000) 2024-03-06T14:04:24.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:24.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:24.223458+0000) 2024-03-06T14:04:25.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:25.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:25.223703+0000) 2024-03-06T14:04:26.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:26.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:26.223901+0000) 2024-03-06T14:04:27.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:27.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:27.224173+0000) 2024-03-06T14:04:28.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:28.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:28.224450+0000) 2024-03-06T14:04:29.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:29.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:29.224661+0000) 2024-03-06T14:04:30.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:30.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:30.224891+0000) 2024-03-06T14:04:31.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:31.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:31.225135+0000) 2024-03-06T14:04:32.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:32.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:32.225378+0000) 2024-03-06T14:04:33.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:33.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:33.225645+0000) 2024-03-06T14:04:34.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:34.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:34.225911+0000) 2024-03-06T14:04:35.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:35.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:35.226154+0000) 2024-03-06T14:04:36.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:36.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:36.226371+0000) 2024-03-06T14:04:37.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:37.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:37.226674+0000) 2024-03-06T14:04:38.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:38.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:38.226940+0000) 2024-03-06T14:04:39.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:39.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:39.227194+0000) 2024-03-06T14:04:40.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:40.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:40.227430+0000) 2024-03-06T14:04:41.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:41.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:41.227675+0000) 2024-03-06T14:04:42.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:42.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:42.227899+0000) 2024-03-06T14:04:43.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:43.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:43.228155+0000) 2024-03-06T14:04:44.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:44.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:44.228381+0000) 2024-03-06T14:04:45.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:45.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:45.228571+0000) 2024-03-06T14:04:46.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:46.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:46.228812+0000) 2024-03-06T14:04:46.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:46.787 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:04:47.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:47.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:47.229101+0000) 2024-03-06T14:04:48.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:48.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:48.229338+0000) 2024-03-06T14:04:49.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:49.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:49.229580+0000) 2024-03-06T14:04:50.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:50.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:50.229758+0000) 2024-03-06T14:04:51.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:51.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:51.229994+0000) 2024-03-06T14:04:52.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:52.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:52.230302+0000) 2024-03-06T14:04:53.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:53.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:53.230497+0000) 2024-03-06T14:04:54.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:54.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:54.230756+0000) 2024-03-06T14:04:55.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:55.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:55.231040+0000) 2024-03-06T14:04:56.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:56.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:56.231228+0000) 2024-03-06T14:04:57.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:57.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:57.231460+0000) 2024-03-06T14:04:58.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:58.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:58.231689+0000) 2024-03-06T14:04:59.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:04:59.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:04:59.231921+0000) 2024-03-06T14:05:00.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:00.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:00.232177+0000) 2024-03-06T14:05:01.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:01.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:01.232400+0000) 2024-03-06T14:05:02.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:02.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:02.232622+0000) 2024-03-06T14:05:03.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:03.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:03.232805+0000) 2024-03-06T14:05:04.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:04.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:04.232996+0000) 2024-03-06T14:05:05.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:05.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:05.233290+0000) 2024-03-06T14:05:06.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:06.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:06.233543+0000) 2024-03-06T14:05:07.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:07.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:07.233776+0000) 2024-03-06T14:05:08.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:08.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:08.234054+0000) 2024-03-06T14:05:09.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:09.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:09.234327+0000) 2024-03-06T14:05:10.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:10.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:10.234592+0000) 2024-03-06T14:05:11.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:11.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:11.234866+0000) 2024-03-06T14:05:12.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:12.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:12.235161+0000) 2024-03-06T14:05:13.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:13.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:13.235412+0000) 2024-03-06T14:05:14.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:14.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:14.235677+0000) 2024-03-06T14:05:15.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:15.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:15.235935+0000) 2024-03-06T14:05:16.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:16.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:16.236184+0000) 2024-03-06T14:05:16.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:17.000 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:17.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:17.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:17.236425+0000) 2024-03-06T14:05:18.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:18.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:18.236643+0000) 2024-03-06T14:05:19.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:19.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:19.236909+0000) 2024-03-06T14:05:20.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:20.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:20.237155+0000) 2024-03-06T14:05:21.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:21.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:21.237378+0000) 2024-03-06T14:05:22.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:22.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:22.237582+0000) 2024-03-06T14:05:23.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:23.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:23.237784+0000) 2024-03-06T14:05:24.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:24.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:24.238061+0000) 2024-03-06T14:05:25.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:25.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:25.238333+0000) 2024-03-06T14:05:26.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:26.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:26.238556+0000) 2024-03-06T14:05:27.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:27.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:27.238783+0000) 2024-03-06T14:05:28.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:28.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:28.238935+0000) 2024-03-06T14:05:29.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:29.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:29.239185+0000) 2024-03-06T14:05:30.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:30.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:30.239458+0000) 2024-03-06T14:05:31.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:31.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:31.239707+0000) 2024-03-06T14:05:32.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:32.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:32.239946+0000) 2024-03-06T14:05:33.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:33.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:33.240204+0000) 2024-03-06T14:05:34.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:34.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:34.240415+0000) 2024-03-06T14:05:35.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:35.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:35.240638+0000) 2024-03-06T14:05:36.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:36.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:36.240863+0000) 2024-03-06T14:05:37.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:37.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:37.241119+0000) 2024-03-06T14:05:38.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:38.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:38.241353+0000) 2024-03-06T14:05:39.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:39.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:39.241568+0000) 2024-03-06T14:05:40.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:40.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:40.241797+0000) 2024-03-06T14:05:41.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:41.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:41.242058+0000) 2024-03-06T14:05:42.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:42.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:42.242285+0000) 2024-03-06T14:05:43.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:43.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:43.242512+0000) 2024-03-06T14:05:44.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:44.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:44.242729+0000) 2024-03-06T14:05:45.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:45.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:45.242955+0000) 2024-03-06T14:05:46.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:46.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:46.243181+0000) 2024-03-06T14:05:47.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:47.205 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:05:47.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:47.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:47.243410+0000) 2024-03-06T14:05:48.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:48.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:48.243600+0000) 2024-03-06T14:05:49.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:49.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:49.243779+0000) 2024-03-06T14:05:50.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:50.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:50.244050+0000) 2024-03-06T14:05:51.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:51.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:51.244296+0000) 2024-03-06T14:05:52.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:52.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:52.244525+0000) 2024-03-06T14:05:53.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:53.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:53.244765+0000) 2024-03-06T14:05:54.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:54.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:54.245066+0000) 2024-03-06T14:05:55.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:55.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:55.245295+0000) 2024-03-06T14:05:56.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:56.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:56.245538+0000) 2024-03-06T14:05:57.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:57.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:57.245785+0000) 2024-03-06T14:05:58.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:58.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:58.245965+0000) 2024-03-06T14:05:59.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:05:59.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:05:59.246171+0000) 2024-03-06T14:06:00.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:00.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:00.246425+0000) 2024-03-06T14:06:01.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:01.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:01.246666+0000) 2024-03-06T14:06:02.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:02.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:02.246908+0000) 2024-03-06T14:06:03.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:03.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:03.247239+0000) 2024-03-06T14:06:04.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:04.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:04.247536+0000) 2024-03-06T14:06:05.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:05.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:05.247778+0000) 2024-03-06T14:06:06.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:06.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:06.248049+0000) 2024-03-06T14:06:07.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:07.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:07.248296+0000) 2024-03-06T14:06:08.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:08.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:08.248553+0000) 2024-03-06T14:06:09.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:09.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:09.248820+0000) 2024-03-06T14:06:10.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:10.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:10.249096+0000) 2024-03-06T14:06:11.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:11.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:11.249347+0000) 2024-03-06T14:06:12.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:12.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:12.249593+0000) 2024-03-06T14:06:13.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:13.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:13.249821+0000) 2024-03-06T14:06:14.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:14.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:14.250096+0000) 2024-03-06T14:06:15.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:15.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:15.250343+0000) 2024-03-06T14:06:16.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:16.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:16.250579+0000) 2024-03-06T14:06:17.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:17.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:17.250884+0000) 2024-03-06T14:06:17.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:17.400 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:18.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:18.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:18.251178+0000) 2024-03-06T14:06:19.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:19.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:19.251382+0000) 2024-03-06T14:06:20.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:20.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:20.251617+0000) 2024-03-06T14:06:21.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:21.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:21.251802+0000) 2024-03-06T14:06:22.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:22.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:22.252119+0000) 2024-03-06T14:06:23.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:23.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:23.252398+0000) 2024-03-06T14:06:24.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:24.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:24.252677+0000) 2024-03-06T14:06:25.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:25.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:25.252928+0000) 2024-03-06T14:06:26.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:26.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:26.253173+0000) 2024-03-06T14:06:27.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:27.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:27.253438+0000) 2024-03-06T14:06:28.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:28.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:28.253723+0000) 2024-03-06T14:06:29.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:29.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:29.254057+0000) 2024-03-06T14:06:30.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:30.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:30.254358+0000) 2024-03-06T14:06:31.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:31.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:31.254638+0000) 2024-03-06T14:06:32.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:32.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:32.254847+0000) 2024-03-06T14:06:33.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:33.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:33.255127+0000) 2024-03-06T14:06:34.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:34.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:34.255336+0000) 2024-03-06T14:06:35.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:35.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:35.255608+0000) 2024-03-06T14:06:36.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:36.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:36.255876+0000) 2024-03-06T14:06:37.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:37.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:37.256125+0000) 2024-03-06T14:06:38.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:38.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:38.256363+0000) 2024-03-06T14:06:39.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:39.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:39.256591+0000) 2024-03-06T14:06:40.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:40.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:40.256828+0000) 2024-03-06T14:06:41.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:41.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:41.257043+0000) 2024-03-06T14:06:42.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:42.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:42.257242+0000) 2024-03-06T14:06:43.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:43.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:43.257489+0000) 2024-03-06T14:06:44.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:44.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:44.257746+0000) 2024-03-06T14:06:45.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:45.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:45.257996+0000) 2024-03-06T14:06:46.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:46.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:46.258278+0000) 2024-03-06T14:06:47.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:47.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:47.258521+0000) 2024-03-06T14:06:47.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:47.605 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:06:48.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:48.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:48.258771+0000) 2024-03-06T14:06:49.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:49.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:49.259049+0000) 2024-03-06T14:06:50.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:50.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:50.259297+0000) 2024-03-06T14:06:51.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:51.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:51.259547+0000) 2024-03-06T14:06:52.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:52.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:52.259782+0000) 2024-03-06T14:06:53.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:53.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:53.259971+0000) 2024-03-06T14:06:54.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:54.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:54.260222+0000) 2024-03-06T14:06:55.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:55.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:55.260447+0000) 2024-03-06T14:06:56.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:56.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:56.260631+0000) 2024-03-06T14:06:57.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:57.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:57.260867+0000) 2024-03-06T14:06:58.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:58.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:58.261116+0000) 2024-03-06T14:06:59.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:06:59.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:06:59.261349+0000) 2024-03-06T14:07:00.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:00.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:00.261573+0000) 2024-03-06T14:07:01.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:01.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:01.261789+0000) 2024-03-06T14:07:02.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:02.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:02.262049+0000) 2024-03-06T14:07:03.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:03.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:03.262309+0000) 2024-03-06T14:07:04.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:04.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:04.262580+0000) 2024-03-06T14:07:05.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:05.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:05.262828+0000) 2024-03-06T14:07:06.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:06.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:06.263145+0000) 2024-03-06T14:07:07.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:07.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:07.263353+0000) 2024-03-06T14:07:08.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:08.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:08.263561+0000) 2024-03-06T14:07:09.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:09.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:09.263811+0000) 2024-03-06T14:07:10.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:10.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:10.264065+0000) 2024-03-06T14:07:11.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:11.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:11.264281+0000) 2024-03-06T14:07:12.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:12.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:12.264518+0000) 2024-03-06T14:07:13.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:13.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:13.264801+0000) 2024-03-06T14:07:14.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:14.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:14.265135+0000) 2024-03-06T14:07:15.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:15.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:15.265421+0000) 2024-03-06T14:07:16.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:16.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:16.265711+0000) 2024-03-06T14:07:17.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:17.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:17.265927+0000) 2024-03-06T14:07:17.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:17.759 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:18.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:18.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:18.266193+0000) 2024-03-06T14:07:19.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:19.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:19.266415+0000) 2024-03-06T14:07:20.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:20.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:20.266625+0000) 2024-03-06T14:07:21.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:21.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:21.266854+0000) 2024-03-06T14:07:22.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:22.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:22.267119+0000) 2024-03-06T14:07:23.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:23.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:23.267352+0000) 2024-03-06T14:07:24.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:24.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:24.267594+0000) 2024-03-06T14:07:25.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:25.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:25.267823+0000) 2024-03-06T14:07:26.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:26.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:26.268084+0000) 2024-03-06T14:07:27.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:27.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:27.268314+0000) 2024-03-06T14:07:28.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:28.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:28.268541+0000) 2024-03-06T14:07:29.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:29.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:29.268768+0000) 2024-03-06T14:07:30.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:30.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:30.268992+0000) 2024-03-06T14:07:31.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:31.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:31.269234+0000) 2024-03-06T14:07:32.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:32.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:32.269469+0000) 2024-03-06T14:07:33.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:33.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:33.269708+0000) 2024-03-06T14:07:34.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:34.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:34.269950+0000) 2024-03-06T14:07:35.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:35.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:35.270172+0000) 2024-03-06T14:07:36.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:36.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:36.270393+0000) 2024-03-06T14:07:37.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:37.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:37.270653+0000) 2024-03-06T14:07:38.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:38.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:38.270874+0000) 2024-03-06T14:07:39.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:39.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:39.271189+0000) 2024-03-06T14:07:40.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:40.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:40.271438+0000) 2024-03-06T14:07:41.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:41.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:41.271688+0000) 2024-03-06T14:07:42.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:42.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:42.271874+0000) 2024-03-06T14:07:43.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:43.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:43.272171+0000) 2024-03-06T14:07:44.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:44.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:44.272461+0000) 2024-03-06T14:07:45.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:45.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:45.272736+0000) 2024-03-06T14:07:46.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:46.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:46.273059+0000) 2024-03-06T14:07:47.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:47.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:47.273343+0000) 2024-03-06T14:07:47.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:47.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:07:48.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:48.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:48.273640+0000) 2024-03-06T14:07:49.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:49.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:49.273865+0000) 2024-03-06T14:07:50.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:50.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:50.274174+0000) 2024-03-06T14:07:51.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:51.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:51.274401+0000) 2024-03-06T14:07:52.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:52.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:52.274685+0000) 2024-03-06T14:07:53.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:53.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:53.274963+0000) 2024-03-06T14:07:54.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:54.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:54.275352+0000) 2024-03-06T14:07:55.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:55.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:55.275636+0000) 2024-03-06T14:07:56.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:56.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:56.275920+0000) 2024-03-06T14:07:57.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:57.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:57.276193+0000) 2024-03-06T14:07:58.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:58.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:58.276432+0000) 2024-03-06T14:07:59.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:07:59.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:07:59.276688+0000) 2024-03-06T14:08:00.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:00.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:00.276938+0000) 2024-03-06T14:08:01.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:01.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:01.277185+0000) 2024-03-06T14:08:02.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:02.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:02.277462+0000) 2024-03-06T14:08:03.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:03.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:03.277747+0000) 2024-03-06T14:08:04.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:04.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:04.278099+0000) 2024-03-06T14:08:05.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:05.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:05.278376+0000) 2024-03-06T14:08:06.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:06.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:06.278654+0000) 2024-03-06T14:08:07.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:07.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:07.278898+0000) 2024-03-06T14:08:08.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:08.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:08.279185+0000) 2024-03-06T14:08:09.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:09.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:09.279435+0000) 2024-03-06T14:08:10.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:10.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:10.279690+0000) 2024-03-06T14:08:11.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:11.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:11.279936+0000) 2024-03-06T14:08:12.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:12.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:12.280131+0000) 2024-03-06T14:08:13.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:13.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:13.280347+0000) 2024-03-06T14:08:14.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:14.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:14.280622+0000) 2024-03-06T14:08:15.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:15.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:15.280874+0000) 2024-03-06T14:08:16.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:16.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:16.281143+0000) 2024-03-06T14:08:17.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:17.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:17.281343+0000) 2024-03-06T14:08:18.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:18.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:18.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:18.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:18.281563+0000) 2024-03-06T14:08:19.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:19.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:19.281799+0000) 2024-03-06T14:08:20.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:20.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:20.282048+0000) 2024-03-06T14:08:21.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:21.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:21.282281+0000) 2024-03-06T14:08:22.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:22.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:22.282561+0000) 2024-03-06T14:08:23.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:23.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:23.282840+0000) 2024-03-06T14:08:24.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:24.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:24.283172+0000) 2024-03-06T14:08:25.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:25.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:25.283452+0000) 2024-03-06T14:08:26.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:26.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:26.283656+0000) 2024-03-06T14:08:27.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:27.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:27.283898+0000) 2024-03-06T14:08:28.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:28.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:28.284143+0000) 2024-03-06T14:08:29.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:29.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:29.284379+0000) 2024-03-06T14:08:30.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:30.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:30.284658+0000) 2024-03-06T14:08:31.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:31.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:31.284936+0000) 2024-03-06T14:08:32.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:32.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:32.285214+0000) 2024-03-06T14:08:33.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:33.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:33.285495+0000) 2024-03-06T14:08:34.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:34.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:34.285692+0000) 2024-03-06T14:08:35.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:35.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:35.285943+0000) 2024-03-06T14:08:36.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:36.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:36.286202+0000) 2024-03-06T14:08:37.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:37.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:37.286444+0000) 2024-03-06T14:08:38.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:38.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:38.286675+0000) 2024-03-06T14:08:39.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:39.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:39.286918+0000) 2024-03-06T14:08:40.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:40.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:40.287145+0000) 2024-03-06T14:08:41.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:41.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:41.287378+0000) 2024-03-06T14:08:42.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:42.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:42.287613+0000) 2024-03-06T14:08:43.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:43.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:43.287820+0000) 2024-03-06T14:08:44.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:44.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:44.288077+0000) 2024-03-06T14:08:45.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:45.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:45.288310+0000) 2024-03-06T14:08:46.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:46.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:46.288516+0000) 2024-03-06T14:08:47.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:47.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:47.288709+0000) 2024-03-06T14:08:48.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:48.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:48.288967+0000) 2024-03-06T14:08:48.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:48.356 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:08:49.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:49.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:49.289214+0000) 2024-03-06T14:08:50.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:50.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:50.289426+0000) 2024-03-06T14:08:51.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:51.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:51.289687+0000) 2024-03-06T14:08:52.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:52.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:52.289935+0000) 2024-03-06T14:08:53.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:53.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:53.290160+0000) 2024-03-06T14:08:54.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:54.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:54.290412+0000) 2024-03-06T14:08:55.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:55.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:55.290641+0000) 2024-03-06T14:08:56.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:56.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:56.290874+0000) 2024-03-06T14:08:57.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:57.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:57.291138+0000) 2024-03-06T14:08:58.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:58.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:58.291375+0000) 2024-03-06T14:08:59.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:08:59.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:08:59.291657+0000) 2024-03-06T14:09:00.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:00.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:00.292175+0000) 2024-03-06T14:09:01.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:01.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:01.292427+0000) 2024-03-06T14:09:02.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:02.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:02.292692+0000) 2024-03-06T14:09:03.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:03.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:03.292920+0000) 2024-03-06T14:09:04.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:04.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:04.293169+0000) 2024-03-06T14:09:05.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:05.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:05.293359+0000) 2024-03-06T14:09:06.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:06.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:06.293594+0000) 2024-03-06T14:09:07.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:07.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:07.293841+0000) 2024-03-06T14:09:08.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:08.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:08.294119+0000) 2024-03-06T14:09:09.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:09.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:09.294379+0000) 2024-03-06T14:09:10.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:10.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:10.294623+0000) 2024-03-06T14:09:11.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:11.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:11.294877+0000) 2024-03-06T14:09:12.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:12.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:12.295109+0000) 2024-03-06T14:09:13.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:13.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:13.295348+0000) 2024-03-06T14:09:14.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:14.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:14.295628+0000) 2024-03-06T14:09:15.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:15.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:15.295832+0000) 2024-03-06T14:09:16.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:16.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:16.296122+0000) 2024-03-06T14:09:17.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:17.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:17.296362+0000) 2024-03-06T14:09:18.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:18.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:18.296594+0000) 2024-03-06T14:09:18.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:18.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:19.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:19.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:19.296850+0000) 2024-03-06T14:09:20.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:20.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:20.297133+0000) 2024-03-06T14:09:21.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:21.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:21.297376+0000) 2024-03-06T14:09:22.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:22.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:22.297616+0000) 2024-03-06T14:09:23.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:23.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:23.297848+0000) 2024-03-06T14:09:24.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:24.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:24.298157+0000) 2024-03-06T14:09:25.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:25.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:25.298370+0000) 2024-03-06T14:09:26.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:26.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:26.298598+0000) 2024-03-06T14:09:27.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:27.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:27.298842+0000) 2024-03-06T14:09:28.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:28.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:28.299120+0000) 2024-03-06T14:09:29.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:29.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:29.299380+0000) 2024-03-06T14:09:30.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:30.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:30.299627+0000) 2024-03-06T14:09:31.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:31.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:31.299871+0000) 2024-03-06T14:09:32.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:32.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:32.300141+0000) 2024-03-06T14:09:33.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:33.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:33.300346+0000) 2024-03-06T14:09:34.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:34.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:34.300603+0000) 2024-03-06T14:09:35.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:35.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:35.300834+0000) 2024-03-06T14:09:36.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:36.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:36.301117+0000) 2024-03-06T14:09:37.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:37.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:37.301359+0000) 2024-03-06T14:09:38.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:38.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:38.301594+0000) 2024-03-06T14:09:39.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:39.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:39.301856+0000) 2024-03-06T14:09:40.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:40.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:40.302162+0000) 2024-03-06T14:09:41.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:41.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:41.302412+0000) 2024-03-06T14:09:42.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:42.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:42.302642+0000) 2024-03-06T14:09:43.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:43.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:43.302887+0000) 2024-03-06T14:09:44.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:44.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:44.303163+0000) 2024-03-06T14:09:45.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:45.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:45.303362+0000) 2024-03-06T14:09:46.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:46.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:46.303590+0000) 2024-03-06T14:09:47.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:47.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:47.303808+0000) 2024-03-06T14:09:48.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:48.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:48.304089+0000) 2024-03-06T14:09:48.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:48.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:09:49.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:49.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:49.304373+0000) 2024-03-06T14:09:50.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:50.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:50.304596+0000) 2024-03-06T14:09:51.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:51.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:51.304831+0000) 2024-03-06T14:09:52.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:52.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:52.305100+0000) 2024-03-06T14:09:53.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:53.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:53.305324+0000) 2024-03-06T14:09:54.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:54.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:54.305589+0000) 2024-03-06T14:09:55.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:55.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:55.305834+0000) 2024-03-06T14:09:56.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:56.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:56.306137+0000) 2024-03-06T14:09:57.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:57.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:57.306346+0000) 2024-03-06T14:09:58.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:58.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:58.306572+0000) 2024-03-06T14:09:59.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:09:59.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:09:59.306813+0000) 2024-03-06T14:10:00.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:00.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:00.307092+0000) 2024-03-06T14:10:01.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:01.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:01.307284+0000) 2024-03-06T14:10:02.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:02.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:02.307517+0000) 2024-03-06T14:10:03.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:03.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:03.307745+0000) 2024-03-06T14:10:04.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:04.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:04.307994+0000) 2024-03-06T14:10:05.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:05.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:05.308254+0000) 2024-03-06T14:10:06.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:06.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:06.308485+0000) 2024-03-06T14:10:07.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:07.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:07.308701+0000) 2024-03-06T14:10:08.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:08.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:08.308931+0000) 2024-03-06T14:10:09.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:09.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:09.309181+0000) 2024-03-06T14:10:10.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:10.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:10.309394+0000) 2024-03-06T14:10:11.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:11.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:11.309626+0000) 2024-03-06T14:10:12.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:12.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:12.309861+0000) 2024-03-06T14:10:13.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:13.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:13.310154+0000) 2024-03-06T14:10:14.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:14.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:14.310410+0000) 2024-03-06T14:10:15.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:15.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:15.310641+0000) 2024-03-06T14:10:16.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:16.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:16.310860+0000) 2024-03-06T14:10:17.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:17.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:17.311110+0000) 2024-03-06T14:10:18.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:18.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:18.311352+0000) 2024-03-06T14:10:18.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:18.870 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:19.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:19.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:19.311652+0000) 2024-03-06T14:10:20.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:20.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:20.311928+0000) 2024-03-06T14:10:21.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:21.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:21.312203+0000) 2024-03-06T14:10:22.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:22.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:22.312438+0000) 2024-03-06T14:10:23.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:23.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:23.312665+0000) 2024-03-06T14:10:24.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:24.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:24.313040+0000) 2024-03-06T14:10:25.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:25.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:25.313302+0000) 2024-03-06T14:10:26.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:26.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:26.313539+0000) 2024-03-06T14:10:27.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:27.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:27.313757+0000) 2024-03-06T14:10:28.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:28.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:28.313975+0000) 2024-03-06T14:10:29.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:29.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:29.314303+0000) 2024-03-06T14:10:30.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:30.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:30.314585+0000) 2024-03-06T14:10:31.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:31.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:31.314872+0000) 2024-03-06T14:10:32.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:32.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:32.315134+0000) 2024-03-06T14:10:33.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:33.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:33.315341+0000) 2024-03-06T14:10:34.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:34.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:34.315588+0000) 2024-03-06T14:10:35.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:35.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:35.315826+0000) 2024-03-06T14:10:36.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:36.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:36.316094+0000) 2024-03-06T14:10:37.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:37.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:37.316309+0000) 2024-03-06T14:10:38.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:38.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:38.316541+0000) 2024-03-06T14:10:39.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:39.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:39.316772+0000) 2024-03-06T14:10:40.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:40.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:40.316999+0000) 2024-03-06T14:10:41.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:41.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:41.317223+0000) 2024-03-06T14:10:42.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:42.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:42.317417+0000) 2024-03-06T14:10:43.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:43.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:43.317641+0000) 2024-03-06T14:10:44.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:44.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:44.317861+0000) 2024-03-06T14:10:45.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:45.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:45.318154+0000) 2024-03-06T14:10:46.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:46.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:46.318374+0000) 2024-03-06T14:10:47.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:47.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:47.318661+0000) 2024-03-06T14:10:48.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:48.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:48.318943+0000) 2024-03-06T14:10:49.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:49.080 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:10:49.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:49.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:49.319174+0000) 2024-03-06T14:10:50.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:50.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:50.319378+0000) 2024-03-06T14:10:51.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:51.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:51.319640+0000) 2024-03-06T14:10:52.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:52.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:52.319846+0000) 2024-03-06T14:10:53.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:53.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:53.320115+0000) 2024-03-06T14:10:54.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:54.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:54.320346+0000) 2024-03-06T14:10:55.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:55.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:55.320569+0000) 2024-03-06T14:10:56.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:56.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:56.320839+0000) 2024-03-06T14:10:57.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:57.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:57.321147+0000) 2024-03-06T14:10:58.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:58.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:58.321388+0000) 2024-03-06T14:10:59.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:10:59.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:10:59.321634+0000) 2024-03-06T14:11:00.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:00.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:00.321925+0000) 2024-03-06T14:11:01.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:01.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:01.322193+0000) 2024-03-06T14:11:02.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:02.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:02.322407+0000) 2024-03-06T14:11:03.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:03.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:03.322620+0000) 2024-03-06T14:11:04.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:04.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:04.322834+0000) 2024-03-06T14:11:05.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:05.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:05.323119+0000) 2024-03-06T14:11:06.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:06.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:06.323312+0000) 2024-03-06T14:11:07.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:07.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:07.323537+0000) 2024-03-06T14:11:08.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:08.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:08.323795+0000) 2024-03-06T14:11:09.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:09.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:09.324054+0000) 2024-03-06T14:11:10.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:10.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:10.324261+0000) 2024-03-06T14:11:11.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:11.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:11.324445+0000) 2024-03-06T14:11:12.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:12.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:12.324663+0000) 2024-03-06T14:11:13.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:13.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:13.324908+0000) 2024-03-06T14:11:14.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:14.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:14.325173+0000) 2024-03-06T14:11:15.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:15.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:15.325340+0000) 2024-03-06T14:11:16.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:16.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:16.325571+0000) 2024-03-06T14:11:17.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:17.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:17.325829+0000) 2024-03-06T14:11:18.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:18.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:18.326132+0000) 2024-03-06T14:11:19.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:19.276 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:19.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:19.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:19.326271+0000) 2024-03-06T14:11:20.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:20.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:20.326456+0000) 2024-03-06T14:11:21.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:21.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:21.326691+0000) 2024-03-06T14:11:22.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:22.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:22.326873+0000) 2024-03-06T14:11:23.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:23.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:23.327145+0000) 2024-03-06T14:11:24.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:24.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:24.327391+0000) 2024-03-06T14:11:25.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:25.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:25.327600+0000) 2024-03-06T14:11:26.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:26.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:26.327823+0000) 2024-03-06T14:11:27.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:27.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:27.327997+0000) 2024-03-06T14:11:28.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:28.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:28.328254+0000) 2024-03-06T14:11:29.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:29.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:29.328535+0000) 2024-03-06T14:11:30.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:30.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:30.328822+0000) 2024-03-06T14:11:31.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:31.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:31.329102+0000) 2024-03-06T14:11:32.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:32.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:32.329305+0000) 2024-03-06T14:11:33.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:33.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:33.329530+0000) 2024-03-06T14:11:34.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:34.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:34.329766+0000) 2024-03-06T14:11:35.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:35.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:35.329986+0000) 2024-03-06T14:11:36.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:36.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:36.330321+0000) 2024-03-06T14:11:37.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:37.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:37.330608+0000) 2024-03-06T14:11:38.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:38.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:38.330850+0000) 2024-03-06T14:11:39.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:39.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:39.331144+0000) 2024-03-06T14:11:40.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:40.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:40.331355+0000) 2024-03-06T14:11:41.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:41.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:41.331566+0000) 2024-03-06T14:11:42.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:42.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:42.331812+0000) 2024-03-06T14:11:43.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:43.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:43.332113+0000) 2024-03-06T14:11:44.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:44.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:44.332360+0000) 2024-03-06T14:11:45.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:45.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:45.332595+0000) 2024-03-06T14:11:46.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:46.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:46.332860+0000) 2024-03-06T14:11:47.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:47.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:47.333115+0000) 2024-03-06T14:11:48.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:48.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:48.333360+0000) 2024-03-06T14:11:49.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:49.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:49.333599+0000) 2024-03-06T14:11:49.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:49.414 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:11:50.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:50.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:50.333799+0000) 2024-03-06T14:11:51.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:51.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:51.334076+0000) 2024-03-06T14:11:52.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:52.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:52.334301+0000) 2024-03-06T14:11:53.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:53.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:53.334507+0000) 2024-03-06T14:11:54.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:54.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:54.334768+0000) 2024-03-06T14:11:55.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:55.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:55.334999+0000) 2024-03-06T14:11:56.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:56.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:56.335250+0000) 2024-03-06T14:11:57.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:57.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:57.335488+0000) 2024-03-06T14:11:58.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:58.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:58.335723+0000) 2024-03-06T14:11:59.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:11:59.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:11:59.335958+0000) 2024-03-06T14:12:00.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:00.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:00.336197+0000) 2024-03-06T14:12:01.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:01.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:01.336435+0000) 2024-03-06T14:12:02.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:02.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:02.336670+0000) 2024-03-06T14:12:03.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:03.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:03.336917+0000) 2024-03-06T14:12:04.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:04.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:04.337184+0000) 2024-03-06T14:12:05.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:05.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:05.337482+0000) 2024-03-06T14:12:06.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:06.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:06.337758+0000) 2024-03-06T14:12:07.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:07.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:07.338080+0000) 2024-03-06T14:12:08.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:08.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:08.338316+0000) 2024-03-06T14:12:09.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:09.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:09.338588+0000) 2024-03-06T14:12:10.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:10.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:10.338869+0000) 2024-03-06T14:12:11.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:11.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:11.339152+0000) 2024-03-06T14:12:12.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:12.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:12.339386+0000) 2024-03-06T14:12:13.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:13.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:13.339598+0000) 2024-03-06T14:12:14.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:14.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:14.339871+0000) 2024-03-06T14:12:15.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:15.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:15.340144+0000) 2024-03-06T14:12:16.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:16.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:16.340395+0000) 2024-03-06T14:12:17.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:17.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:17.340623+0000) 2024-03-06T14:12:18.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:18.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:18.340843+0000) 2024-03-06T14:12:19.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:19.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:19.341174+0000) 2024-03-06T14:12:19.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:19.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:20.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:20.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:20.341424+0000) 2024-03-06T14:12:21.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:21.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:21.341655+0000) 2024-03-06T14:12:22.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:22.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:22.341852+0000) 2024-03-06T14:12:23.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:23.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:23.342131+0000) 2024-03-06T14:12:24.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:24.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:24.342360+0000) 2024-03-06T14:12:25.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:25.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:25.342555+0000) 2024-03-06T14:12:26.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:26.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:26.342795+0000) 2024-03-06T14:12:27.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:27.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:27.343124+0000) 2024-03-06T14:12:28.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:28.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:28.343367+0000) 2024-03-06T14:12:29.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:29.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:29.343565+0000) 2024-03-06T14:12:30.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:30.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:30.343800+0000) 2024-03-06T14:12:31.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:31.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:31.344065+0000) 2024-03-06T14:12:32.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:32.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:32.344266+0000) 2024-03-06T14:12:33.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:33.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:33.344510+0000) 2024-03-06T14:12:34.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:34.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:34.344741+0000) 2024-03-06T14:12:35.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:35.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:35.344986+0000) 2024-03-06T14:12:36.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:36.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:36.345241+0000) 2024-03-06T14:12:37.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:37.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:37.345483+0000) 2024-03-06T14:12:38.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:38.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:38.345720+0000) 2024-03-06T14:12:39.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:39.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:39.345965+0000) 2024-03-06T14:12:40.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:40.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:40.346252+0000) 2024-03-06T14:12:41.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:41.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:41.346501+0000) 2024-03-06T14:12:42.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:42.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:42.346703+0000) 2024-03-06T14:12:43.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:43.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:43.346926+0000) 2024-03-06T14:12:44.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:44.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:44.347164+0000) 2024-03-06T14:12:45.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:45.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:45.347387+0000) 2024-03-06T14:12:46.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:46.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:46.347625+0000) 2024-03-06T14:12:47.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:47.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:47.347834+0000) 2024-03-06T14:12:48.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:48.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:48.348116+0000) 2024-03-06T14:12:49.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:49.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:49.348360+0000) 2024-03-06T14:12:49.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:49.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:12:50.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:50.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:50.348642+0000) 2024-03-06T14:12:51.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:51.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:51.348915+0000) 2024-03-06T14:12:52.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:52.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:52.349158+0000) 2024-03-06T14:12:53.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:53.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:53.349387+0000) 2024-03-06T14:12:54.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:54.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:54.349597+0000) 2024-03-06T14:12:55.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:55.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:55.349825+0000) 2024-03-06T14:12:56.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:56.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:56.350116+0000) 2024-03-06T14:12:57.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:57.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:57.350357+0000) 2024-03-06T14:12:58.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:58.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:58.350602+0000) 2024-03-06T14:12:59.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:12:59.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:12:59.350881+0000) 2024-03-06T14:13:00.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:00.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:00.351170+0000) 2024-03-06T14:13:01.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:01.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:01.351416+0000) 2024-03-06T14:13:02.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:02.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:02.351678+0000) 2024-03-06T14:13:03.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:03.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:03.351945+0000) 2024-03-06T14:13:04.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:04.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:04.352204+0000) 2024-03-06T14:13:05.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:05.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:05.352457+0000) 2024-03-06T14:13:06.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:06.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:06.352670+0000) 2024-03-06T14:13:07.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:07.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:07.352891+0000) 2024-03-06T14:13:08.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:08.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:08.353135+0000) 2024-03-06T14:13:09.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:09.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:09.353399+0000) 2024-03-06T14:13:10.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:10.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:10.353622+0000) 2024-03-06T14:13:11.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:11.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:11.353856+0000) 2024-03-06T14:13:12.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:12.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:12.354107+0000) 2024-03-06T14:13:13.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:13.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:13.354352+0000) 2024-03-06T14:13:14.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:14.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:14.354617+0000) 2024-03-06T14:13:15.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:15.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:15.354893+0000) 2024-03-06T14:13:16.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:16.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:16.355174+0000) 2024-03-06T14:13:17.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:17.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:17.355448+0000) 2024-03-06T14:13:18.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:18.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:18.355698+0000) 2024-03-06T14:13:19.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:19.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:19.355919+0000) 2024-03-06T14:13:20.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:20.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:20.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:20.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:20.356135+0000) 2024-03-06T14:13:21.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:21.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:21.356385+0000) 2024-03-06T14:13:22.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:22.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:22.356623+0000) 2024-03-06T14:13:23.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:23.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:23.356870+0000) 2024-03-06T14:13:24.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:24.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:24.357152+0000) 2024-03-06T14:13:25.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:25.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:25.357389+0000) 2024-03-06T14:13:26.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:26.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:26.357605+0000) 2024-03-06T14:13:27.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:27.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:27.357810+0000) 2024-03-06T14:13:28.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:28.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:28.358030+0000) 2024-03-06T14:13:29.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:29.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:29.358228+0000) 2024-03-06T14:13:30.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:30.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:30.358413+0000) 2024-03-06T14:13:31.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:31.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:31.358629+0000) 2024-03-06T14:13:32.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:32.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:32.358907+0000) 2024-03-06T14:13:33.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:33.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:33.359186+0000) 2024-03-06T14:13:34.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:34.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:34.359434+0000) 2024-03-06T14:13:35.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:35.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:35.359692+0000) 2024-03-06T14:13:36.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:36.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:36.359957+0000) 2024-03-06T14:13:37.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:37.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:37.360165+0000) 2024-03-06T14:13:38.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:38.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:38.360384+0000) 2024-03-06T14:13:39.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:39.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:39.360610+0000) 2024-03-06T14:13:40.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:40.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:40.360831+0000) 2024-03-06T14:13:41.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:41.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:41.361109+0000) 2024-03-06T14:13:42.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:42.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:42.361338+0000) 2024-03-06T14:13:43.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:43.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:43.361617+0000) 2024-03-06T14:13:44.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:44.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:44.361905+0000) 2024-03-06T14:13:45.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:45.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:45.362195+0000) 2024-03-06T14:13:46.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:46.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:46.362435+0000) 2024-03-06T14:13:47.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:47.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:47.362653+0000) 2024-03-06T14:13:48.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:48.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:48.362909+0000) 2024-03-06T14:13:49.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:49.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:49.363190+0000) 2024-03-06T14:13:50.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:50.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:13:50.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:50.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:50.363471+0000) 2024-03-06T14:13:51.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:51.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:51.363735+0000) 2024-03-06T14:13:52.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:52.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:52.364042+0000) 2024-03-06T14:13:53.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:53.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:53.364327+0000) 2024-03-06T14:13:54.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:54.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:54.364617+0000) 2024-03-06T14:13:55.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:55.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:55.364909+0000) 2024-03-06T14:13:56.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:56.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:56.365194+0000) 2024-03-06T14:13:57.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:57.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:57.365401+0000) 2024-03-06T14:13:58.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:58.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:58.365632+0000) 2024-03-06T14:13:59.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:13:59.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:13:59.365852+0000) 2024-03-06T14:14:00.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:00.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:00.366115+0000) 2024-03-06T14:14:01.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:01.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:01.366340+0000) 2024-03-06T14:14:02.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:02.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:02.366610+0000) 2024-03-06T14:14:03.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:03.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:03.366822+0000) 2024-03-06T14:14:04.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:04.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:04.367132+0000) 2024-03-06T14:14:05.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:05.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:05.367335+0000) 2024-03-06T14:14:06.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:06.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:06.367557+0000) 2024-03-06T14:14:07.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:07.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:07.367771+0000) 2024-03-06T14:14:08.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:08.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:08.367960+0000) 2024-03-06T14:14:09.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:09.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:09.368222+0000) 2024-03-06T14:14:10.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:10.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:10.368505+0000) 2024-03-06T14:14:11.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:11.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:11.368751+0000) 2024-03-06T14:14:12.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:12.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:12.369047+0000) 2024-03-06T14:14:13.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:13.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:13.369322+0000) 2024-03-06T14:14:14.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:14.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:14.369618+0000) 2024-03-06T14:14:15.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:15.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:15.369905+0000) 2024-03-06T14:14:16.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:16.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:16.370193+0000) 2024-03-06T14:14:17.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:17.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:17.370442+0000) 2024-03-06T14:14:18.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:18.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:18.370633+0000) 2024-03-06T14:14:19.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:19.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:19.370836+0000) 2024-03-06T14:14:20.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:20.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:20.371142+0000) 2024-03-06T14:14:20.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:20.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:21.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:21.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:21.371436+0000) 2024-03-06T14:14:22.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:22.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:22.371726+0000) 2024-03-06T14:14:23.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:23.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:23.372047+0000) 2024-03-06T14:14:24.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:24.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:24.372313+0000) 2024-03-06T14:14:25.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:25.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:25.372597+0000) 2024-03-06T14:14:26.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:26.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:26.372888+0000) 2024-03-06T14:14:27.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:27.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:27.373171+0000) 2024-03-06T14:14:28.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:28.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:28.373429+0000) 2024-03-06T14:14:29.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:29.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:29.373691+0000) 2024-03-06T14:14:30.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:30.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:30.373970+0000) 2024-03-06T14:14:31.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:31.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:31.374235+0000) 2024-03-06T14:14:32.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:32.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:32.374405+0000) 2024-03-06T14:14:33.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:33.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:33.374619+0000) 2024-03-06T14:14:34.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:34.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:34.374897+0000) 2024-03-06T14:14:35.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:35.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:35.375138+0000) 2024-03-06T14:14:36.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:36.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:36.375337+0000) 2024-03-06T14:14:37.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:37.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:37.375555+0000) 2024-03-06T14:14:38.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:38.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:38.375774+0000) 2024-03-06T14:14:39.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:39.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:39.376047+0000) 2024-03-06T14:14:40.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:40.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:40.376267+0000) 2024-03-06T14:14:41.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:41.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:41.376492+0000) 2024-03-06T14:14:42.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:42.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:42.376714+0000) 2024-03-06T14:14:43.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:43.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:43.376945+0000) 2024-03-06T14:14:44.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:44.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:44.377167+0000) 2024-03-06T14:14:45.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:45.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:45.377400+0000) 2024-03-06T14:14:46.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:46.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:46.377681+0000) 2024-03-06T14:14:47.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:47.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:47.377925+0000) 2024-03-06T14:14:48.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:48.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:48.378155+0000) 2024-03-06T14:14:49.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:49.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:49.378318+0000) 2024-03-06T14:14:50.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:50.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:50.378547+0000) 2024-03-06T14:14:50.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:50.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:14:51.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:51.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:51.378766+0000) 2024-03-06T14:14:52.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:52.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:52.378977+0000) 2024-03-06T14:14:53.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:53.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:53.379245+0000) 2024-03-06T14:14:54.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:54.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:54.379502+0000) 2024-03-06T14:14:55.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:55.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:55.379720+0000) 2024-03-06T14:14:56.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:56.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:56.379910+0000) 2024-03-06T14:14:57.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:57.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:57.380126+0000) 2024-03-06T14:14:58.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:58.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:58.380355+0000) 2024-03-06T14:14:59.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:14:59.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:14:59.380589+0000) 2024-03-06T14:15:00.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:00.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:00.380827+0000) 2024-03-06T14:15:01.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:01.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:01.381136+0000) 2024-03-06T14:15:02.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:02.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:02.381423+0000) 2024-03-06T14:15:03.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:03.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:03.381708+0000) 2024-03-06T14:15:04.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:04.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:04.381988+0000) 2024-03-06T14:15:05.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:05.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:05.382230+0000) 2024-03-06T14:15:06.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:06.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:06.382469+0000) 2024-03-06T14:15:07.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:07.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:07.382710+0000) 2024-03-06T14:15:08.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:08.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:08.382947+0000) 2024-03-06T14:15:09.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:09.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:09.383169+0000) 2024-03-06T14:15:10.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:10.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:10.383398+0000) 2024-03-06T14:15:11.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:11.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:11.383636+0000) 2024-03-06T14:15:12.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:12.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:12.383871+0000) 2024-03-06T14:15:13.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:13.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:13.384121+0000) 2024-03-06T14:15:14.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:14.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:14.384367+0000) 2024-03-06T14:15:15.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:15.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:15.384620+0000) 2024-03-06T14:15:16.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:16.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:16.384867+0000) 2024-03-06T14:15:17.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:17.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:17.385155+0000) 2024-03-06T14:15:18.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:18.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:18.385369+0000) 2024-03-06T14:15:19.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:19.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:19.385609+0000) 2024-03-06T14:15:20.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:20.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:20.385871+0000) 2024-03-06T14:15:20.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:20.826 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:21.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:21.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:21.386124+0000) 2024-03-06T14:15:22.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:22.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:22.386373+0000) 2024-03-06T14:15:23.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:23.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:23.386612+0000) 2024-03-06T14:15:24.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:24.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:24.386832+0000) 2024-03-06T14:15:25.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:25.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:25.387125+0000) 2024-03-06T14:15:26.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:26.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:26.387333+0000) 2024-03-06T14:15:27.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:27.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:27.387568+0000) 2024-03-06T14:15:28.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:28.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:28.387819+0000) 2024-03-06T14:15:29.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:29.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:29.388067+0000) 2024-03-06T14:15:30.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:30.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:30.388300+0000) 2024-03-06T14:15:31.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:31.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:31.388534+0000) 2024-03-06T14:15:32.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:32.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:32.388787+0000) 2024-03-06T14:15:33.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:33.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:33.389066+0000) 2024-03-06T14:15:34.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:34.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:34.389234+0000) 2024-03-06T14:15:35.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:35.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:35.389472+0000) 2024-03-06T14:15:36.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:36.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:36.389682+0000) 2024-03-06T14:15:37.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:37.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:37.389908+0000) 2024-03-06T14:15:38.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:38.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:38.390144+0000) 2024-03-06T14:15:39.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:39.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:39.390347+0000) 2024-03-06T14:15:40.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:40.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:40.390568+0000) 2024-03-06T14:15:41.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:41.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:41.390805+0000) 2024-03-06T14:15:42.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:42.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:42.391049+0000) 2024-03-06T14:15:43.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:43.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:43.391271+0000) 2024-03-06T14:15:44.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:44.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:44.391472+0000) 2024-03-06T14:15:45.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:45.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:45.391723+0000) 2024-03-06T14:15:46.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:46.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:46.391927+0000) 2024-03-06T14:15:47.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:47.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:47.392146+0000) 2024-03-06T14:15:48.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:48.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:48.392331+0000) 2024-03-06T14:15:49.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:49.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:49.392549+0000) 2024-03-06T14:15:50.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:50.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:50.392853+0000) 2024-03-06T14:15:50.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:50.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:15:51.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:51.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:51.393166+0000) 2024-03-06T14:15:52.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:52.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:52.393448+0000) 2024-03-06T14:15:53.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:53.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:53.393719+0000) 2024-03-06T14:15:54.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:54.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:54.393977+0000) 2024-03-06T14:15:55.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:55.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:55.394248+0000) 2024-03-06T14:15:56.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:56.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:56.394478+0000) 2024-03-06T14:15:57.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:57.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:57.394725+0000) 2024-03-06T14:15:58.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:58.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:58.395053+0000) 2024-03-06T14:15:59.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:15:59.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:15:59.395294+0000) 2024-03-06T14:16:00.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:00.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:00.395582+0000) 2024-03-06T14:16:01.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:01.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:01.395865+0000) 2024-03-06T14:16:02.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:02.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:02.396107+0000) 2024-03-06T14:16:03.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:03.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:03.396379+0000) 2024-03-06T14:16:04.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:04.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:04.396642+0000) 2024-03-06T14:16:05.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:05.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:05.396868+0000) 2024-03-06T14:16:06.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:06.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:06.397110+0000) 2024-03-06T14:16:07.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:07.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:07.397346+0000) 2024-03-06T14:16:08.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:08.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:08.397607+0000) 2024-03-06T14:16:09.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:09.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:09.397884+0000) 2024-03-06T14:16:10.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:10.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:10.398124+0000) 2024-03-06T14:16:11.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:11.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:11.398403+0000) 2024-03-06T14:16:12.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:12.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:12.398609+0000) 2024-03-06T14:16:13.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:13.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:13.398832+0000) 2024-03-06T14:16:14.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:14.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:14.399141+0000) 2024-03-06T14:16:15.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:15.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:15.399339+0000) 2024-03-06T14:16:16.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:16.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:16.399616+0000) 2024-03-06T14:16:17.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:17.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:17.399823+0000) 2024-03-06T14:16:18.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:18.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:18.400110+0000) 2024-03-06T14:16:19.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:19.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:19.400361+0000) 2024-03-06T14:16:20.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:20.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:20.400638+0000) 2024-03-06T14:16:21.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:21.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:21.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:21.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:21.400930+0000) 2024-03-06T14:16:22.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:22.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:22.401164+0000) 2024-03-06T14:16:23.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:23.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:23.401388+0000) 2024-03-06T14:16:24.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:24.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:24.401617+0000) 2024-03-06T14:16:25.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:25.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:25.401844+0000) 2024-03-06T14:16:26.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:26.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:26.402125+0000) 2024-03-06T14:16:27.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:27.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:27.402317+0000) 2024-03-06T14:16:28.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:28.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:28.402494+0000) 2024-03-06T14:16:29.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:29.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:29.402766+0000) 2024-03-06T14:16:30.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:30.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:30.402965+0000) 2024-03-06T14:16:31.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:31.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:31.403173+0000) 2024-03-06T14:16:32.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:32.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:32.403400+0000) 2024-03-06T14:16:33.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:33.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:33.403626+0000) 2024-03-06T14:16:34.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:34.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:34.403865+0000) 2024-03-06T14:16:35.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:35.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:35.404130+0000) 2024-03-06T14:16:36.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:36.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:36.404342+0000) 2024-03-06T14:16:37.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:37.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:37.404614+0000) 2024-03-06T14:16:38.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:38.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:38.404895+0000) 2024-03-06T14:16:39.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:39.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:39.405147+0000) 2024-03-06T14:16:40.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:40.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:40.405375+0000) 2024-03-06T14:16:41.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:41.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:41.405591+0000) 2024-03-06T14:16:42.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:42.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:42.405864+0000) 2024-03-06T14:16:43.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:43.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:43.406135+0000) 2024-03-06T14:16:44.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:44.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:44.406348+0000) 2024-03-06T14:16:45.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:45.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:45.406600+0000) 2024-03-06T14:16:46.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:46.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:46.406841+0000) 2024-03-06T14:16:47.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:47.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:47.407125+0000) 2024-03-06T14:16:48.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:48.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:48.407323+0000) 2024-03-06T14:16:49.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:49.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:49.407558+0000) 2024-03-06T14:16:50.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:50.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:50.407835+0000) 2024-03-06T14:16:51.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:51.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:16:51.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:51.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:51.408165+0000) 2024-03-06T14:16:52.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:52.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:52.408439+0000) 2024-03-06T14:16:53.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:53.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:53.408654+0000) 2024-03-06T14:16:54.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:54.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:54.408886+0000) 2024-03-06T14:16:55.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:55.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:55.409124+0000) 2024-03-06T14:16:56.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:56.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:56.409348+0000) 2024-03-06T14:16:57.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:57.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:57.409564+0000) 2024-03-06T14:16:58.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:58.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:58.409746+0000) 2024-03-06T14:16:59.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:16:59.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:16:59.409971+0000) 2024-03-06T14:17:00.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:00.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:00.410264+0000) 2024-03-06T14:17:01.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:01.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:01.410443+0000) 2024-03-06T14:17:02.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:02.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:02.410620+0000) 2024-03-06T14:17:03.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:03.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:03.410839+0000) 2024-03-06T14:17:04.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:04.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:04.411064+0000) 2024-03-06T14:17:05.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:05.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:05.411261+0000) 2024-03-06T14:17:06.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:06.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:06.411486+0000) 2024-03-06T14:17:07.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:07.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:07.411665+0000) 2024-03-06T14:17:08.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:08.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:08.411840+0000) 2024-03-06T14:17:09.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:09.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:09.412161+0000) 2024-03-06T14:17:10.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:10.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:10.412389+0000) 2024-03-06T14:17:11.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:11.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:11.412630+0000) 2024-03-06T14:17:12.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:12.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:12.412846+0000) 2024-03-06T14:17:13.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:13.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:13.413111+0000) 2024-03-06T14:17:14.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:14.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:14.413310+0000) 2024-03-06T14:17:15.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:15.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:15.413533+0000) 2024-03-06T14:17:16.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:16.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:16.413754+0000) 2024-03-06T14:17:17.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:17.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:17.413989+0000) 2024-03-06T14:17:18.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:18.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:18.414212+0000) 2024-03-06T14:17:19.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:19.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:19.414429+0000) 2024-03-06T14:17:20.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:20.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:20.414612+0000) 2024-03-06T14:17:21.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:21.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:21.414844+0000) 2024-03-06T14:17:21.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:21.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:22.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:22.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:22.415135+0000) 2024-03-06T14:17:23.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:23.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:23.415346+0000) 2024-03-06T14:17:24.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:24.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:24.415596+0000) 2024-03-06T14:17:25.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:25.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:25.415795+0000) 2024-03-06T14:17:26.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:26.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:26.416044+0000) 2024-03-06T14:17:27.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:27.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:27.416304+0000) 2024-03-06T14:17:28.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:28.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:28.416543+0000) 2024-03-06T14:17:29.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:29.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:29.416752+0000) 2024-03-06T14:17:30.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:30.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:30.417057+0000) 2024-03-06T14:17:31.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:31.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:31.417306+0000) 2024-03-06T14:17:32.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:32.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:32.417578+0000) 2024-03-06T14:17:33.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:33.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:33.417852+0000) 2024-03-06T14:17:34.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:34.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:34.418151+0000) 2024-03-06T14:17:35.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:35.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:35.418391+0000) 2024-03-06T14:17:36.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:36.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:36.418658+0000) 2024-03-06T14:17:37.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:37.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:37.418894+0000) 2024-03-06T14:17:38.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:38.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:38.419182+0000) 2024-03-06T14:17:39.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:39.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:39.419430+0000) 2024-03-06T14:17:40.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:40.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:40.419710+0000) 2024-03-06T14:17:41.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:41.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:41.419994+0000) 2024-03-06T14:17:42.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:42.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:42.420278+0000) 2024-03-06T14:17:43.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:43.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:43.420546+0000) 2024-03-06T14:17:44.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:44.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:44.420791+0000) 2024-03-06T14:17:45.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:45.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:45.420994+0000) 2024-03-06T14:17:46.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:46.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:46.421289+0000) 2024-03-06T14:17:47.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:47.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:47.421519+0000) 2024-03-06T14:17:48.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:48.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:48.421762+0000) 2024-03-06T14:17:49.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:49.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:49.422049+0000) 2024-03-06T14:17:50.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:50.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:50.422321+0000) 2024-03-06T14:17:51.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:51.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:51.422553+0000) 2024-03-06T14:17:51.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:51.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:17:52.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:52.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:52.422759+0000) 2024-03-06T14:17:53.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:53.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:53.422960+0000) 2024-03-06T14:17:54.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:54.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:54.423174+0000) 2024-03-06T14:17:55.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:55.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:55.423345+0000) 2024-03-06T14:17:56.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:56.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:56.423583+0000) 2024-03-06T14:17:57.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:57.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:57.423761+0000) 2024-03-06T14:17:58.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:58.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:58.423977+0000) 2024-03-06T14:17:59.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:17:59.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:17:59.424217+0000) 2024-03-06T14:18:00.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:00.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:00.424393+0000) 2024-03-06T14:18:01.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:01.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:01.424621+0000) 2024-03-06T14:18:02.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:02.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:02.424898+0000) 2024-03-06T14:18:03.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:03.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:03.425142+0000) 2024-03-06T14:18:04.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:04.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:04.425400+0000) 2024-03-06T14:18:05.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:05.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:05.425616+0000) 2024-03-06T14:18:06.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:06.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:06.425859+0000) 2024-03-06T14:18:07.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:07.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:07.426114+0000) 2024-03-06T14:18:08.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:08.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:08.426371+0000) 2024-03-06T14:18:09.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:09.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:09.426612+0000) 2024-03-06T14:18:10.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:10.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:10.426839+0000) 2024-03-06T14:18:11.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:11.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:11.427116+0000) 2024-03-06T14:18:12.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:12.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:12.427361+0000) 2024-03-06T14:18:13.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:13.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:13.427606+0000) 2024-03-06T14:18:14.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:14.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:14.427857+0000) 2024-03-06T14:18:15.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:15.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:15.428138+0000) 2024-03-06T14:18:16.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:16.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:16.428383+0000) 2024-03-06T14:18:17.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:17.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:17.428611+0000) 2024-03-06T14:18:18.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:18.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:18.428853+0000) 2024-03-06T14:18:19.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:19.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:19.429127+0000) 2024-03-06T14:18:20.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:20.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:20.429371+0000) 2024-03-06T14:18:21.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:21.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:21.429610+0000) 2024-03-06T14:18:21.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:21.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:22.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:22.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:22.429855+0000) 2024-03-06T14:18:23.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:23.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:23.430123+0000) 2024-03-06T14:18:24.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:24.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:24.430370+0000) 2024-03-06T14:18:25.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:25.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:25.430597+0000) 2024-03-06T14:18:26.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:26.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:26.430807+0000) 2024-03-06T14:18:27.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:27.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:27.431106+0000) 2024-03-06T14:18:28.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:28.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:28.431339+0000) 2024-03-06T14:18:29.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:29.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:29.431589+0000) 2024-03-06T14:18:30.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:30.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:30.431859+0000) 2024-03-06T14:18:31.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:31.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:31.432163+0000) 2024-03-06T14:18:32.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:32.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:32.432441+0000) 2024-03-06T14:18:33.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:33.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:33.432708+0000) 2024-03-06T14:18:34.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:34.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:34.432992+0000) 2024-03-06T14:18:35.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:35.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:35.433332+0000) 2024-03-06T14:18:36.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:36.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:36.433555+0000) 2024-03-06T14:18:37.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:37.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:37.433782+0000) 2024-03-06T14:18:38.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:38.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:38.434067+0000) 2024-03-06T14:18:39.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:39.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:39.434243+0000) 2024-03-06T14:18:40.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:40.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:40.434470+0000) 2024-03-06T14:18:41.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:41.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:41.434691+0000) 2024-03-06T14:18:42.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:42.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:42.434910+0000) 2024-03-06T14:18:43.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:43.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:43.435129+0000) 2024-03-06T14:18:44.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:44.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:44.435340+0000) 2024-03-06T14:18:45.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:45.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:45.435619+0000) 2024-03-06T14:18:46.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:46.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:46.435853+0000) 2024-03-06T14:18:47.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:47.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:47.436095+0000) 2024-03-06T14:18:48.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:48.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:48.436323+0000) 2024-03-06T14:18:49.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:49.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:49.436586+0000) 2024-03-06T14:18:50.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:50.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:50.436808+0000) 2024-03-06T14:18:51.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:51.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:51.437091+0000) 2024-03-06T14:18:51.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:51.980 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:18:52.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:52.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:52.437310+0000) 2024-03-06T14:18:53.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:53.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:53.437532+0000) 2024-03-06T14:18:54.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:54.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:54.437761+0000) 2024-03-06T14:18:55.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:55.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:55.437984+0000) 2024-03-06T14:18:56.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:56.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:56.438262+0000) 2024-03-06T14:18:57.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:57.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:57.438488+0000) 2024-03-06T14:18:58.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:58.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:58.438705+0000) 2024-03-06T14:18:59.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:18:59.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:18:59.438931+0000) 2024-03-06T14:19:00.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:00.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:00.439205+0000) 2024-03-06T14:19:01.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:01.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:01.439441+0000) 2024-03-06T14:19:02.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:02.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:02.439663+0000) 2024-03-06T14:19:03.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:03.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:03.439866+0000) 2024-03-06T14:19:04.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:04.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:04.440127+0000) 2024-03-06T14:19:05.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:05.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:05.440348+0000) 2024-03-06T14:19:06.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:06.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:06.440588+0000) 2024-03-06T14:19:07.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:07.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:07.440812+0000) 2024-03-06T14:19:08.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:08.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:08.441076+0000) 2024-03-06T14:19:09.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:09.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:09.441300+0000) 2024-03-06T14:19:10.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:10.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:10.441517+0000) 2024-03-06T14:19:11.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:11.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:11.441751+0000) 2024-03-06T14:19:12.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:12.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:12.442050+0000) 2024-03-06T14:19:13.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:13.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:13.442287+0000) 2024-03-06T14:19:14.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:14.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:14.442506+0000) 2024-03-06T14:19:15.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:15.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:15.442703+0000) 2024-03-06T14:19:16.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:16.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:16.442950+0000) 2024-03-06T14:19:17.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:17.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:17.443174+0000) 2024-03-06T14:19:18.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:18.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:18.443393+0000) 2024-03-06T14:19:19.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:19.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:19.443576+0000) 2024-03-06T14:19:20.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:20.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:20.443800+0000) 2024-03-06T14:19:21.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:21.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:21.444067+0000) 2024-03-06T14:19:22.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:22.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:22.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:22.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:22.444266+0000) 2024-03-06T14:19:23.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:23.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:23.444476+0000) 2024-03-06T14:19:24.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:24.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:24.444691+0000) 2024-03-06T14:19:25.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:25.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:25.444937+0000) 2024-03-06T14:19:26.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:26.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:26.445200+0000) 2024-03-06T14:19:27.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:27.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:27.445444+0000) 2024-03-06T14:19:28.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:28.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:28.445663+0000) 2024-03-06T14:19:29.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:29.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:29.445820+0000) 2024-03-06T14:19:30.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:30.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:30.446079+0000) 2024-03-06T14:19:31.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:31.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:31.446274+0000) 2024-03-06T14:19:32.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:32.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:32.446492+0000) 2024-03-06T14:19:33.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:33.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:33.446833+0000) 2024-03-06T14:19:34.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:34.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:34.447162+0000) 2024-03-06T14:19:35.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:35.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:35.447453+0000) 2024-03-06T14:19:36.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:36.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:36.447746+0000) 2024-03-06T14:19:37.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:37.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:37.448044+0000) 2024-03-06T14:19:38.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:38.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:38.448254+0000) 2024-03-06T14:19:39.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:39.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:39.448495+0000) 2024-03-06T14:19:40.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:40.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:40.448713+0000) 2024-03-06T14:19:41.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:41.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:41.448972+0000) 2024-03-06T14:19:42.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:42.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:42.449248+0000) 2024-03-06T14:19:43.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:43.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:43.449465+0000) 2024-03-06T14:19:44.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:44.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:44.449692+0000) 2024-03-06T14:19:45.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:45.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:45.449919+0000) 2024-03-06T14:19:46.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:46.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:46.450171+0000) 2024-03-06T14:19:47.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:47.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:47.450370+0000) 2024-03-06T14:19:48.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:48.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:48.450592+0000) 2024-03-06T14:19:49.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:49.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:49.450814+0000) 2024-03-06T14:19:50.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:50.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:50.451070+0000) 2024-03-06T14:19:51.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:51.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:51.451315+0000) 2024-03-06T14:19:52.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:52.357 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:19:52.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:52.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:52.451510+0000) 2024-03-06T14:19:53.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:53.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:53.451714+0000) 2024-03-06T14:19:54.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:54.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:54.451971+0000) 2024-03-06T14:19:55.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:55.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:55.452208+0000) 2024-03-06T14:19:56.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:56.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:56.452440+0000) 2024-03-06T14:19:57.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:57.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:57.452636+0000) 2024-03-06T14:19:58.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:58.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:58.452856+0000) 2024-03-06T14:19:59.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:19:59.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:19:59.453112+0000) 2024-03-06T14:20:00.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:00.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:00.453342+0000) 2024-03-06T14:20:01.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:01.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:01.453566+0000) 2024-03-06T14:20:02.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:02.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:02.453779+0000) 2024-03-06T14:20:03.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:03.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:03.454050+0000) 2024-03-06T14:20:04.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:04.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:04.454307+0000) 2024-03-06T14:20:05.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:05.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:05.454532+0000) 2024-03-06T14:20:06.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:06.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:06.454762+0000) 2024-03-06T14:20:07.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:07.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:07.454992+0000) 2024-03-06T14:20:08.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:08.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:08.455275+0000) 2024-03-06T14:20:09.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:09.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:09.455500+0000) 2024-03-06T14:20:10.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:10.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:10.455726+0000) 2024-03-06T14:20:11.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:11.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:11.455979+0000) 2024-03-06T14:20:12.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:12.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:12.456261+0000) 2024-03-06T14:20:13.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:13.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:13.456482+0000) 2024-03-06T14:20:14.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:14.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:14.456709+0000) 2024-03-06T14:20:15.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:15.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:15.456931+0000) 2024-03-06T14:20:16.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:16.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:16.457173+0000) 2024-03-06T14:20:17.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:17.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:17.457389+0000) 2024-03-06T14:20:18.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:18.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:18.457613+0000) 2024-03-06T14:20:19.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:19.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:19.457865+0000) 2024-03-06T14:20:20.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:20.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:20.458134+0000) 2024-03-06T14:20:21.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:21.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:21.458364+0000) 2024-03-06T14:20:22.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:22.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:22.458583+0000) 2024-03-06T14:20:22.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:22.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:23.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:23.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:23.458810+0000) 2024-03-06T14:20:24.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:24.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:24.459083+0000) 2024-03-06T14:20:25.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:25.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:25.459314+0000) 2024-03-06T14:20:26.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:26.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:26.459555+0000) 2024-03-06T14:20:27.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:27.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:27.459784+0000) 2024-03-06T14:20:28.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:28.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:28.460039+0000) 2024-03-06T14:20:29.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:29.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:29.460275+0000) 2024-03-06T14:20:30.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:30.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:30.460495+0000) 2024-03-06T14:20:31.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:31.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:31.460730+0000) 2024-03-06T14:20:32.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:32.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:32.460953+0000) 2024-03-06T14:20:33.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:33.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:33.461193+0000) 2024-03-06T14:20:34.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:34.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:34.461401+0000) 2024-03-06T14:20:35.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:35.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:35.461628+0000) 2024-03-06T14:20:36.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:36.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:36.461881+0000) 2024-03-06T14:20:37.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:37.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:37.462137+0000) 2024-03-06T14:20:38.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:38.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:38.462371+0000) 2024-03-06T14:20:39.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:39.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:39.462599+0000) 2024-03-06T14:20:40.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:40.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:40.462822+0000) 2024-03-06T14:20:41.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:41.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:41.463110+0000) 2024-03-06T14:20:42.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:42.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:42.463321+0000) 2024-03-06T14:20:43.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:43.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:43.463531+0000) 2024-03-06T14:20:44.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:44.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:44.463774+0000) 2024-03-06T14:20:45.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:45.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:45.463960+0000) 2024-03-06T14:20:46.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:46.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:46.464199+0000) 2024-03-06T14:20:47.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:47.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:47.464394+0000) 2024-03-06T14:20:48.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:48.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:48.464588+0000) 2024-03-06T14:20:49.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:49.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:49.464799+0000) 2024-03-06T14:20:50.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:50.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:50.465043+0000) 2024-03-06T14:20:51.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:51.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:51.465280+0000) 2024-03-06T14:20:52.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:52.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:52.465498+0000) 2024-03-06T14:20:52.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:52.712 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:20:53.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:53.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:53.465728+0000) 2024-03-06T14:20:54.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:54.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:54.465958+0000) 2024-03-06T14:20:55.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:55.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:55.466179+0000) 2024-03-06T14:20:56.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:56.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:56.466387+0000) 2024-03-06T14:20:57.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:57.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:57.466634+0000) 2024-03-06T14:20:58.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:58.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:58.466885+0000) 2024-03-06T14:20:59.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:20:59.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:20:59.467149+0000) 2024-03-06T14:21:00.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:00.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:00.467380+0000) 2024-03-06T14:21:01.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:01.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:01.467657+0000) 2024-03-06T14:21:02.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:02.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:02.467892+0000) 2024-03-06T14:21:03.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:03.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:03.468130+0000) 2024-03-06T14:21:04.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:04.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:04.468370+0000) 2024-03-06T14:21:05.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:05.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:05.468570+0000) 2024-03-06T14:21:06.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:06.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:06.468843+0000) 2024-03-06T14:21:07.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:07.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:07.469111+0000) 2024-03-06T14:21:08.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:08.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:08.469346+0000) 2024-03-06T14:21:09.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:09.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:09.469573+0000) 2024-03-06T14:21:10.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:10.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:10.469777+0000) 2024-03-06T14:21:11.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:11.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:11.470065+0000) 2024-03-06T14:21:12.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:12.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:12.470297+0000) 2024-03-06T14:21:13.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:13.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:13.470528+0000) 2024-03-06T14:21:14.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:14.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:14.470779+0000) 2024-03-06T14:21:15.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:15.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:15.471026+0000) 2024-03-06T14:21:16.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:16.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:16.471231+0000) 2024-03-06T14:21:17.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:17.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:17.471465+0000) 2024-03-06T14:21:18.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:18.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:18.471629+0000) 2024-03-06T14:21:19.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:19.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:19.471849+0000) 2024-03-06T14:21:20.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:20.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:20.472294+0000) 2024-03-06T14:21:21.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:21.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:21.472486+0000) 2024-03-06T14:21:22.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:22.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:22.472720+0000) 2024-03-06T14:21:22.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:22.849 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:23.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:23.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:23.472953+0000) 2024-03-06T14:21:24.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:24.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:24.473190+0000) 2024-03-06T14:21:25.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:25.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:25.473427+0000) 2024-03-06T14:21:26.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:26.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:26.473674+0000) 2024-03-06T14:21:27.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:27.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:27.473898+0000) 2024-03-06T14:21:28.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:28.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:28.474125+0000) 2024-03-06T14:21:29.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:29.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:29.474339+0000) 2024-03-06T14:21:30.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:30.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:30.474584+0000) 2024-03-06T14:21:31.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:31.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:31.474819+0000) 2024-03-06T14:21:32.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:32.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:32.474985+0000) 2024-03-06T14:21:33.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:33.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:33.475248+0000) 2024-03-06T14:21:34.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:34.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:34.475488+0000) 2024-03-06T14:21:35.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:35.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:35.475712+0000) 2024-03-06T14:21:36.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:36.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:36.475947+0000) 2024-03-06T14:21:37.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:37.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:37.476199+0000) 2024-03-06T14:21:38.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:38.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:38.476438+0000) 2024-03-06T14:21:39.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:39.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:39.476684+0000) 2024-03-06T14:21:40.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:40.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:40.476926+0000) 2024-03-06T14:21:41.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:41.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:41.477189+0000) 2024-03-06T14:21:42.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:42.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:42.477429+0000) 2024-03-06T14:21:43.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:43.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:43.477677+0000) 2024-03-06T14:21:44.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:44.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:44.477910+0000) 2024-03-06T14:21:45.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:45.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:45.478146+0000) 2024-03-06T14:21:46.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:46.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:46.478344+0000) 2024-03-06T14:21:47.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:47.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:47.478564+0000) 2024-03-06T14:21:48.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:48.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:48.478771+0000) 2024-03-06T14:21:49.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:49.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:49.479001+0000) 2024-03-06T14:21:50.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:50.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:50.479264+0000) 2024-03-06T14:21:51.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:51.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:51.479490+0000) 2024-03-06T14:21:52.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:52.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:52.479717+0000) 2024-03-06T14:21:52.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:52.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:21:53.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:53.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:53.479952+0000) 2024-03-06T14:21:54.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:54.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:54.480210+0000) 2024-03-06T14:21:55.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:55.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:55.480456+0000) 2024-03-06T14:21:56.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:56.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:56.480676+0000) 2024-03-06T14:21:57.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:57.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:57.480939+0000) 2024-03-06T14:21:58.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:58.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:58.481157+0000) 2024-03-06T14:21:59.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:21:59.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:21:59.481383+0000) 2024-03-06T14:22:00.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:00.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:00.481609+0000) 2024-03-06T14:22:01.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:01.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:01.481843+0000) 2024-03-06T14:22:02.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:02.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:02.482122+0000) 2024-03-06T14:22:03.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:03.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:03.482360+0000) 2024-03-06T14:22:04.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:04.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:04.482622+0000) 2024-03-06T14:22:05.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:05.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:05.482844+0000) 2024-03-06T14:22:06.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:06.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:06.483118+0000) 2024-03-06T14:22:07.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:07.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:07.483350+0000) 2024-03-06T14:22:08.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:08.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:08.483625+0000) 2024-03-06T14:22:09.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:09.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:09.483848+0000) 2024-03-06T14:22:10.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:10.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:10.484124+0000) 2024-03-06T14:22:11.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:11.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:11.484353+0000) 2024-03-06T14:22:12.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:12.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:12.484577+0000) 2024-03-06T14:22:13.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:13.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:13.484809+0000) 2024-03-06T14:22:14.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:14.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:14.485067+0000) 2024-03-06T14:22:15.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:15.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:15.485261+0000) 2024-03-06T14:22:16.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:16.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:16.485439+0000) 2024-03-06T14:22:17.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:17.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:17.485685+0000) 2024-03-06T14:22:18.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:18.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:18.485929+0000) 2024-03-06T14:22:19.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:19.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:19.486184+0000) 2024-03-06T14:22:20.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:20.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:20.486389+0000) 2024-03-06T14:22:21.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:21.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:21.486613+0000) 2024-03-06T14:22:22.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:22.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:22.486833+0000) 2024-03-06T14:22:23.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:23.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:23.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:23.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:23.487100+0000) 2024-03-06T14:22:24.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:24.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:24.487344+0000) 2024-03-06T14:22:25.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:25.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:25.487565+0000) 2024-03-06T14:22:26.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:26.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:26.487817+0000) 2024-03-06T14:22:27.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:27.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:27.488074+0000) 2024-03-06T14:22:28.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:28.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:28.488317+0000) 2024-03-06T14:22:29.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:29.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:29.488544+0000) 2024-03-06T14:22:30.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:30.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:30.488756+0000) 2024-03-06T14:22:31.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:31.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:31.488938+0000) 2024-03-06T14:22:32.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:32.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:32.489190+0000) 2024-03-06T14:22:33.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:33.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:33.489426+0000) 2024-03-06T14:22:34.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:34.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:34.489678+0000) 2024-03-06T14:22:35.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:35.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:35.489909+0000) 2024-03-06T14:22:36.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:36.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:36.490194+0000) 2024-03-06T14:22:37.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:37.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:37.490428+0000) 2024-03-06T14:22:38.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:38.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:38.490649+0000) 2024-03-06T14:22:39.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:39.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:39.490916+0000) 2024-03-06T14:22:40.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:40.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:40.491144+0000) 2024-03-06T14:22:41.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:41.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:41.491404+0000) 2024-03-06T14:22:42.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:42.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:42.491621+0000) 2024-03-06T14:22:43.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:43.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:43.491879+0000) 2024-03-06T14:22:44.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:44.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:44.492135+0000) 2024-03-06T14:22:45.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:45.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:45.492362+0000) 2024-03-06T14:22:46.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:46.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:46.492548+0000) 2024-03-06T14:22:47.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:47.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:47.492780+0000) 2024-03-06T14:22:48.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:48.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:48.493047+0000) 2024-03-06T14:22:49.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:49.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:49.493333+0000) 2024-03-06T14:22:50.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:50.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:50.493614+0000) 2024-03-06T14:22:51.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:51.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:51.493803+0000) 2024-03-06T14:22:52.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:52.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:52.494060+0000) 2024-03-06T14:22:53.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:53.393 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:22:53.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:53.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:53.494272+0000) 2024-03-06T14:22:54.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:54.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:54.494476+0000) 2024-03-06T14:22:55.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:55.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:55.494696+0000) 2024-03-06T14:22:56.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:56.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:56.494901+0000) 2024-03-06T14:22:57.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:57.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:57.495134+0000) 2024-03-06T14:22:58.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:58.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:58.495356+0000) 2024-03-06T14:22:59.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:22:59.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:22:59.495572+0000) 2024-03-06T14:23:00.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:00.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:00.495835+0000) 2024-03-06T14:23:01.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:01.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:01.496111+0000) 2024-03-06T14:23:02.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:02.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:02.496338+0000) 2024-03-06T14:23:03.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:03.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:03.496515+0000) 2024-03-06T14:23:04.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:04.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:04.496760+0000) 2024-03-06T14:23:05.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:05.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:05.496983+0000) 2024-03-06T14:23:06.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:06.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:06.497258+0000) 2024-03-06T14:23:07.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:07.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:07.497475+0000) 2024-03-06T14:23:08.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:08.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:08.497737+0000) 2024-03-06T14:23:09.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:09.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:09.497971+0000) 2024-03-06T14:23:10.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:10.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:10.498210+0000) 2024-03-06T14:23:11.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:11.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:11.498456+0000) 2024-03-06T14:23:12.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:12.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:12.498672+0000) 2024-03-06T14:23:13.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:13.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:13.498919+0000) 2024-03-06T14:23:14.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:14.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:14.499155+0000) 2024-03-06T14:23:15.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:15.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:15.499383+0000) 2024-03-06T14:23:16.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:16.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:16.499612+0000) 2024-03-06T14:23:17.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:17.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:17.499801+0000) 2024-03-06T14:23:18.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:18.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:18.500063+0000) 2024-03-06T14:23:19.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:19.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:19.500292+0000) 2024-03-06T14:23:20.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:20.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:20.500508+0000) 2024-03-06T14:23:21.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:21.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:21.500732+0000) 2024-03-06T14:23:22.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:22.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:22.500960+0000) 2024-03-06T14:23:23.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:23.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:23.501197+0000) 2024-03-06T14:23:23.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:23.607 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:24.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:24.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:24.501439+0000) 2024-03-06T14:23:25.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:25.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:25.501676+0000) 2024-03-06T14:23:26.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:26.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:26.501876+0000) 2024-03-06T14:23:27.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:27.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:27.502142+0000) 2024-03-06T14:23:28.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:28.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:28.502369+0000) 2024-03-06T14:23:29.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:29.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:29.502604+0000) 2024-03-06T14:23:30.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:30.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:30.502824+0000) 2024-03-06T14:23:31.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:31.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:31.503095+0000) 2024-03-06T14:23:32.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:32.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:32.503321+0000) 2024-03-06T14:23:33.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:33.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:33.503562+0000) 2024-03-06T14:23:34.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:34.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:34.503755+0000) 2024-03-06T14:23:35.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:35.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:35.503994+0000) 2024-03-06T14:23:36.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:36.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:36.504227+0000) 2024-03-06T14:23:37.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:37.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:37.504450+0000) 2024-03-06T14:23:38.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:38.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:38.504671+0000) 2024-03-06T14:23:39.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:39.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:39.504895+0000) 2024-03-06T14:23:40.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:40.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:40.505129+0000) 2024-03-06T14:23:41.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:41.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:41.505361+0000) 2024-03-06T14:23:42.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:42.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:42.505584+0000) 2024-03-06T14:23:43.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:43.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:43.505792+0000) 2024-03-06T14:23:44.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:44.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:44.506105+0000) 2024-03-06T14:23:45.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:45.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:45.506292+0000) 2024-03-06T14:23:46.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:46.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:46.506512+0000) 2024-03-06T14:23:47.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:47.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:47.506734+0000) 2024-03-06T14:23:48.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:48.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:48.506949+0000) 2024-03-06T14:23:49.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:49.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:49.507146+0000) 2024-03-06T14:23:50.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:50.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:50.507379+0000) 2024-03-06T14:23:51.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:51.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:51.507608+0000) 2024-03-06T14:23:52.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:52.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:52.507828+0000) 2024-03-06T14:23:53.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:53.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:53.508039+0000) 2024-03-06T14:23:53.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:53.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:23:54.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:54.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:54.508295+0000) 2024-03-06T14:23:55.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:55.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:55.508534+0000) 2024-03-06T14:23:56.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:56.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:56.508784+0000) 2024-03-06T14:23:57.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:57.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:57.509047+0000) 2024-03-06T14:23:58.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:58.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:58.509309+0000) 2024-03-06T14:23:59.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:23:59.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:23:59.509542+0000) 2024-03-06T14:24:00.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:00.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:00.509770+0000) 2024-03-06T14:24:01.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:01.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:01.509992+0000) 2024-03-06T14:24:02.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:02.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:02.510219+0000) 2024-03-06T14:24:03.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:03.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:03.510448+0000) 2024-03-06T14:24:04.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:04.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:04.510691+0000) 2024-03-06T14:24:05.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:05.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:05.510919+0000) 2024-03-06T14:24:06.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:06.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:06.511145+0000) 2024-03-06T14:24:07.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:07.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:07.511336+0000) 2024-03-06T14:24:08.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:08.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:08.511558+0000) 2024-03-06T14:24:09.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:09.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:09.511764+0000) 2024-03-06T14:24:10.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:10.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:10.511991+0000) 2024-03-06T14:24:11.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:11.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:11.512307+0000) 2024-03-06T14:24:12.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:12.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:12.512554+0000) 2024-03-06T14:24:13.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:13.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:13.512793+0000) 2024-03-06T14:24:14.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:14.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:14.513070+0000) 2024-03-06T14:24:15.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:15.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:15.513296+0000) 2024-03-06T14:24:16.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:16.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:16.513519+0000) 2024-03-06T14:24:17.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:17.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:17.513744+0000) 2024-03-06T14:24:18.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:18.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:18.513965+0000) 2024-03-06T14:24:19.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:19.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:19.514196+0000) 2024-03-06T14:24:20.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:20.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:20.514424+0000) 2024-03-06T14:24:21.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:21.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:21.514657+0000) 2024-03-06T14:24:22.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:22.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:22.514835+0000) 2024-03-06T14:24:23.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:23.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:23.515110+0000) 2024-03-06T14:24:23.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:23.948 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:24.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:24.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:24.515335+0000) 2024-03-06T14:24:25.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:25.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:25.515540+0000) 2024-03-06T14:24:26.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:26.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:26.515777+0000) 2024-03-06T14:24:27.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:27.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:27.516106+0000) 2024-03-06T14:24:28.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:28.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:28.516347+0000) 2024-03-06T14:24:29.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:29.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:29.516614+0000) 2024-03-06T14:24:30.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:30.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:30.516890+0000) 2024-03-06T14:24:31.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:31.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:31.517143+0000) 2024-03-06T14:24:32.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:32.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:32.517417+0000) 2024-03-06T14:24:33.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:33.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:33.517711+0000) 2024-03-06T14:24:34.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:34.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:34.518057+0000) 2024-03-06T14:24:35.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:35.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:35.518344+0000) 2024-03-06T14:24:36.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:36.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:36.518578+0000) 2024-03-06T14:24:37.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:37.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:37.518906+0000) 2024-03-06T14:24:38.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:38.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:38.519140+0000) 2024-03-06T14:24:39.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:39.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:39.519364+0000) 2024-03-06T14:24:40.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:40.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:40.519584+0000) 2024-03-06T14:24:41.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:41.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:41.519811+0000) 2024-03-06T14:24:42.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:42.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:42.520064+0000) 2024-03-06T14:24:43.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:43.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:43.520245+0000) 2024-03-06T14:24:44.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:44.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:44.520482+0000) 2024-03-06T14:24:45.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:45.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:45.520684+0000) 2024-03-06T14:24:46.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:46.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:46.520905+0000) 2024-03-06T14:24:47.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:47.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:47.521162+0000) 2024-03-06T14:24:48.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:48.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:48.521431+0000) 2024-03-06T14:24:49.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:49.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:49.521699+0000) 2024-03-06T14:24:50.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:50.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:50.521977+0000) 2024-03-06T14:24:51.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:51.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:51.522310+0000) 2024-03-06T14:24:52.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:52.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:52.522581+0000) 2024-03-06T14:24:53.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:53.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:53.522821+0000) 2024-03-06T14:24:54.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:54.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:24:54.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:54.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:54.523107+0000) 2024-03-06T14:24:55.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:55.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:55.523337+0000) 2024-03-06T14:24:56.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:56.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:56.523559+0000) 2024-03-06T14:24:57.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:57.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:57.523759+0000) 2024-03-06T14:24:58.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:58.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:58.524071+0000) 2024-03-06T14:24:59.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:24:59.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:24:59.524313+0000) 2024-03-06T14:25:00.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:00.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:00.524557+0000) 2024-03-06T14:25:01.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:01.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:01.524781+0000) 2024-03-06T14:25:02.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:02.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:02.525037+0000) 2024-03-06T14:25:03.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:03.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:03.525264+0000) 2024-03-06T14:25:04.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:04.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:04.525490+0000) 2024-03-06T14:25:05.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:05.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:05.525713+0000) 2024-03-06T14:25:06.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:06.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:06.525937+0000) 2024-03-06T14:25:07.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:07.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:07.526161+0000) 2024-03-06T14:25:08.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:08.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:08.526385+0000) 2024-03-06T14:25:09.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:09.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:09.526620+0000) 2024-03-06T14:25:10.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:10.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:10.526836+0000) 2024-03-06T14:25:11.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:11.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:11.527107+0000) 2024-03-06T14:25:12.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:12.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:12.527313+0000) 2024-03-06T14:25:13.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:13.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:13.527532+0000) 2024-03-06T14:25:14.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:14.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:14.527740+0000) 2024-03-06T14:25:15.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:15.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:15.527989+0000) 2024-03-06T14:25:16.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:16.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:16.528221+0000) 2024-03-06T14:25:17.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:17.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:17.528412+0000) 2024-03-06T14:25:18.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:18.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:18.528630+0000) 2024-03-06T14:25:19.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:19.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:19.528849+0000) 2024-03-06T14:25:20.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:20.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:20.529114+0000) 2024-03-06T14:25:21.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:21.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:21.529281+0000) 2024-03-06T14:25:22.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:22.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:22.529477+0000) 2024-03-06T14:25:23.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:23.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:23.529705+0000) 2024-03-06T14:25:24.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:24.445 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:24.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:24.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:24.529939+0000) 2024-03-06T14:25:25.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:25.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:25.530154+0000) 2024-03-06T14:25:26.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:26.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:26.530340+0000) 2024-03-06T14:25:27.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:27.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:27.530580+0000) 2024-03-06T14:25:28.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:28.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:28.530731+0000) 2024-03-06T14:25:29.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:29.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:29.530941+0000) 2024-03-06T14:25:30.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:30.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:30.531293+0000) 2024-03-06T14:25:31.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:31.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:31.531474+0000) 2024-03-06T14:25:32.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:32.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:32.531705+0000) 2024-03-06T14:25:33.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:33.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:33.531945+0000) 2024-03-06T14:25:34.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:34.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:34.532179+0000) 2024-03-06T14:25:35.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:35.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:35.532398+0000) 2024-03-06T14:25:36.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:36.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:36.532605+0000) 2024-03-06T14:25:37.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:37.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:37.532883+0000) 2024-03-06T14:25:38.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:38.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:38.533165+0000) 2024-03-06T14:25:39.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:39.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:39.533403+0000) 2024-03-06T14:25:40.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:40.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:40.533627+0000) 2024-03-06T14:25:41.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:41.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:41.533853+0000) 2024-03-06T14:25:42.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:42.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:42.534150+0000) 2024-03-06T14:25:43.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:43.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:43.534435+0000) 2024-03-06T14:25:44.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:44.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:44.534738+0000) 2024-03-06T14:25:45.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:45.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:45.535055+0000) 2024-03-06T14:25:46.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:46.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:46.535338+0000) 2024-03-06T14:25:47.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:47.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:47.535536+0000) 2024-03-06T14:25:48.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:48.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:48.535771+0000) 2024-03-06T14:25:49.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:49.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:49.536057+0000) 2024-03-06T14:25:50.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:50.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:50.536346+0000) 2024-03-06T14:25:51.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:51.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:51.536627+0000) 2024-03-06T14:25:52.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:52.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:52.536916+0000) 2024-03-06T14:25:53.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:53.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:53.537193+0000) 2024-03-06T14:25:54.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:54.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:54.537416+0000) 2024-03-06T14:25:54.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:54.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:25:55.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:55.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:55.537678+0000) 2024-03-06T14:25:56.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:56.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:56.537951+0000) 2024-03-06T14:25:57.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:57.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:57.538249+0000) 2024-03-06T14:25:58.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:58.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:58.538525+0000) 2024-03-06T14:25:59.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:25:59.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:25:59.538811+0000) 2024-03-06T14:26:00.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:00.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:00.539123+0000) 2024-03-06T14:26:01.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:01.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:01.539369+0000) 2024-03-06T14:26:02.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:02.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:02.539604+0000) 2024-03-06T14:26:03.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:03.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:03.539827+0000) 2024-03-06T14:26:04.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:04.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:04.540153+0000) 2024-03-06T14:26:05.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:05.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:05.540428+0000) 2024-03-06T14:26:06.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:06.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:06.540660+0000) 2024-03-06T14:26:07.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:07.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:07.540949+0000) 2024-03-06T14:26:08.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:08.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:08.541214+0000) 2024-03-06T14:26:09.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:09.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:09.541494+0000) 2024-03-06T14:26:10.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:10.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:10.541770+0000) 2024-03-06T14:26:11.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:11.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:11.541974+0000) 2024-03-06T14:26:12.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:12.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:12.542210+0000) 2024-03-06T14:26:13.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:13.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:13.542414+0000) 2024-03-06T14:26:14.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:14.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:14.542647+0000) 2024-03-06T14:26:15.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:15.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:15.542913+0000) 2024-03-06T14:26:16.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:16.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:16.543150+0000) 2024-03-06T14:26:17.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:17.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:17.543383+0000) 2024-03-06T14:26:18.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:18.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:18.543662+0000) 2024-03-06T14:26:19.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:19.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:19.543888+0000) 2024-03-06T14:26:20.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:20.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:20.544119+0000) 2024-03-06T14:26:21.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:21.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:21.544347+0000) 2024-03-06T14:26:22.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:22.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:22.544592+0000) 2024-03-06T14:26:23.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:23.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:23.544810+0000) 2024-03-06T14:26:24.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:24.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:24.545123+0000) 2024-03-06T14:26:24.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:24.816 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:25.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:25.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:25.545331+0000) 2024-03-06T14:26:26.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:26.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:26.546052+0000) 2024-03-06T14:26:27.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:27.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:27.546349+0000) 2024-03-06T14:26:28.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:28.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:28.546568+0000) 2024-03-06T14:26:29.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:29.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:29.546800+0000) 2024-03-06T14:26:30.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:30.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:30.547042+0000) 2024-03-06T14:26:31.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:31.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:31.547274+0000) 2024-03-06T14:26:32.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:32.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:32.547531+0000) 2024-03-06T14:26:33.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:33.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:33.547767+0000) 2024-03-06T14:26:34.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:34.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:34.548074+0000) 2024-03-06T14:26:35.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:35.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:35.548317+0000) 2024-03-06T14:26:36.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:36.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:36.548603+0000) 2024-03-06T14:26:37.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:37.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:37.548904+0000) 2024-03-06T14:26:38.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:38.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:38.549191+0000) 2024-03-06T14:26:39.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:39.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:39.549401+0000) 2024-03-06T14:26:40.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:40.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:40.549609+0000) 2024-03-06T14:26:41.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:41.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:41.549825+0000) 2024-03-06T14:26:42.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:42.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:42.550098+0000) 2024-03-06T14:26:43.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:43.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:43.550292+0000) 2024-03-06T14:26:44.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:44.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:44.550518+0000) 2024-03-06T14:26:45.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:45.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:45.550793+0000) 2024-03-06T14:26:46.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:46.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:46.551130+0000) 2024-03-06T14:26:47.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:47.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:47.551381+0000) 2024-03-06T14:26:48.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:48.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:48.551588+0000) 2024-03-06T14:26:49.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:49.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:49.551816+0000) 2024-03-06T14:26:50.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:50.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:50.552063+0000) 2024-03-06T14:26:51.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:51.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:51.552245+0000) 2024-03-06T14:26:52.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:52.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:52.552541+0000) 2024-03-06T14:26:53.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:53.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:53.552755+0000) 2024-03-06T14:26:54.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:54.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:54.552990+0000) 2024-03-06T14:26:55.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:55.011 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:26:55.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:55.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:55.553206+0000) 2024-03-06T14:26:56.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:56.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:56.553421+0000) 2024-03-06T14:26:57.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:57.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:57.553672+0000) 2024-03-06T14:26:58.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:58.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:58.553907+0000) 2024-03-06T14:26:59.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:26:59.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:26:59.554150+0000) 2024-03-06T14:27:00.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:00.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:00.554345+0000) 2024-03-06T14:27:01.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:01.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:01.554553+0000) 2024-03-06T14:27:02.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:02.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:02.554799+0000) 2024-03-06T14:27:03.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:03.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:03.554988+0000) 2024-03-06T14:27:04.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:04.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:04.555193+0000) 2024-03-06T14:27:05.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:05.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:05.555437+0000) 2024-03-06T14:27:06.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:06.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:06.555664+0000) 2024-03-06T14:27:07.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:07.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:07.555949+0000) 2024-03-06T14:27:08.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:08.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:08.556155+0000) 2024-03-06T14:27:09.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:09.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:09.556394+0000) 2024-03-06T14:27:10.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:10.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:10.556649+0000) 2024-03-06T14:27:11.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:11.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:11.556888+0000) 2024-03-06T14:27:12.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:12.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:12.557170+0000) 2024-03-06T14:27:13.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:13.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:13.557364+0000) 2024-03-06T14:27:14.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:14.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:14.557582+0000) 2024-03-06T14:27:15.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:15.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:15.557848+0000) 2024-03-06T14:27:16.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:16.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:16.558161+0000) 2024-03-06T14:27:17.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:17.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:17.558458+0000) 2024-03-06T14:27:18.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:18.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:18.558703+0000) 2024-03-06T14:27:19.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:19.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:19.558927+0000) 2024-03-06T14:27:20.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:20.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:20.559182+0000) 2024-03-06T14:27:21.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:21.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:21.559376+0000) 2024-03-06T14:27:22.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:22.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:22.559564+0000) 2024-03-06T14:27:23.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:23.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:23.559812+0000) 2024-03-06T14:27:24.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:24.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:24.560091+0000) 2024-03-06T14:27:25.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:25.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:25.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:25.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:25.560294+0000) 2024-03-06T14:27:26.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:26.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:26.560518+0000) 2024-03-06T14:27:27.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:27.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:27.560742+0000) 2024-03-06T14:27:28.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:28.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:28.560966+0000) 2024-03-06T14:27:29.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:29.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:29.561101+0000) 2024-03-06T14:27:30.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:30.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:30.561234+0000) 2024-03-06T14:27:31.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:31.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:31.561417+0000) 2024-03-06T14:27:32.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:32.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:32.561646+0000) 2024-03-06T14:27:33.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:33.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:33.561849+0000) 2024-03-06T14:27:34.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:34.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:34.562135+0000) 2024-03-06T14:27:35.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:35.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:35.562325+0000) 2024-03-06T14:27:36.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:36.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:36.562550+0000) 2024-03-06T14:27:37.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:37.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:37.562726+0000) 2024-03-06T14:27:38.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:38.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:38.562978+0000) 2024-03-06T14:27:39.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:39.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:39.563260+0000) 2024-03-06T14:27:40.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:40.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:40.563500+0000) 2024-03-06T14:27:41.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:41.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:41.563727+0000) 2024-03-06T14:27:42.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:42.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:42.563956+0000) 2024-03-06T14:27:43.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:43.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:43.564160+0000) 2024-03-06T14:27:44.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:44.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:44.564372+0000) 2024-03-06T14:27:45.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:45.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:45.564588+0000) 2024-03-06T14:27:46.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:46.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:46.564812+0000) 2024-03-06T14:27:47.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:47.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:47.565087+0000) 2024-03-06T14:27:48.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:48.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:48.565261+0000) 2024-03-06T14:27:49.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:49.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:49.565488+0000) 2024-03-06T14:27:50.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:50.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:50.565692+0000) 2024-03-06T14:27:51.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:51.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:51.565896+0000) 2024-03-06T14:27:52.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:52.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:52.566150+0000) 2024-03-06T14:27:53.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:53.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:53.566387+0000) 2024-03-06T14:27:54.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:54.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:54.566601+0000) 2024-03-06T14:27:55.338 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:55.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:27:55.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:55.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:55.566830+0000) 2024-03-06T14:27:56.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:56.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:56.567099+0000) 2024-03-06T14:27:57.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:57.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:57.567282+0000) 2024-03-06T14:27:58.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:58.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:58.567457+0000) 2024-03-06T14:27:59.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:27:59.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:27:59.567695+0000) 2024-03-06T14:28:00.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:00.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:00.567877+0000) 2024-03-06T14:28:01.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:01.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:01.568142+0000) 2024-03-06T14:28:02.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:02.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:02.568323+0000) 2024-03-06T14:28:03.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:03.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:03.568544+0000) 2024-03-06T14:28:04.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:04.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:04.568775+0000) 2024-03-06T14:28:05.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:05.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:05.568996+0000) 2024-03-06T14:28:06.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:06.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:06.569285+0000) 2024-03-06T14:28:07.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:07.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:07.569517+0000) 2024-03-06T14:28:08.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:08.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:08.569753+0000) 2024-03-06T14:28:09.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:09.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:09.569940+0000) 2024-03-06T14:28:10.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:10.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:10.570202+0000) 2024-03-06T14:28:11.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:11.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:11.570399+0000) 2024-03-06T14:28:12.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:12.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:12.570606+0000) 2024-03-06T14:28:13.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:13.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:13.570842+0000) 2024-03-06T14:28:14.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:14.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:14.571124+0000) 2024-03-06T14:28:15.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:15.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:15.571319+0000) 2024-03-06T14:28:16.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:16.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:16.571533+0000) 2024-03-06T14:28:17.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:17.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:17.571750+0000) 2024-03-06T14:28:18.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:18.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:18.571963+0000) 2024-03-06T14:28:19.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:19.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:19.572158+0000) 2024-03-06T14:28:20.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:20.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:20.572390+0000) 2024-03-06T14:28:21.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:21.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:21.572602+0000) 2024-03-06T14:28:22.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:22.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:22.572822+0000) 2024-03-06T14:28:23.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:23.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:23.573072+0000) 2024-03-06T14:28:24.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:24.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:24.573272+0000) 2024-03-06T14:28:25.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:25.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:25.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:25.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:25.573459+0000) 2024-03-06T14:28:26.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:26.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:26.573684+0000) 2024-03-06T14:28:27.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:27.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:27.573867+0000) 2024-03-06T14:28:28.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:28.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:28.574138+0000) 2024-03-06T14:28:29.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:29.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:29.574363+0000) 2024-03-06T14:28:30.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:30.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:30.574586+0000) 2024-03-06T14:28:31.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:31.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:31.574815+0000) 2024-03-06T14:28:32.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:32.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:32.575067+0000) 2024-03-06T14:28:33.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:33.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:33.575288+0000) 2024-03-06T14:28:34.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:34.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:34.575476+0000) 2024-03-06T14:28:35.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:35.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:35.575721+0000) 2024-03-06T14:28:36.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:36.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:36.575932+0000) 2024-03-06T14:28:37.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:37.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:37.576152+0000) 2024-03-06T14:28:38.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:38.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:38.576347+0000) 2024-03-06T14:28:39.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:39.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:39.576552+0000) 2024-03-06T14:28:40.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:40.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:40.576770+0000) 2024-03-06T14:28:41.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:41.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:41.576993+0000) 2024-03-06T14:28:42.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:42.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:42.577213+0000) 2024-03-06T14:28:43.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:43.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:43.577458+0000) 2024-03-06T14:28:44.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:44.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:44.577701+0000) 2024-03-06T14:28:45.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:45.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:45.577924+0000) 2024-03-06T14:28:46.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:46.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:46.578160+0000) 2024-03-06T14:28:47.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:47.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:47.578381+0000) 2024-03-06T14:28:48.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:48.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:48.578664+0000) 2024-03-06T14:28:49.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:49.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:49.578898+0000) 2024-03-06T14:28:50.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:50.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:50.579144+0000) 2024-03-06T14:28:51.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:51.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:51.579338+0000) 2024-03-06T14:28:52.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:52.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:52.579559+0000) 2024-03-06T14:28:53.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:53.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:53.579747+0000) 2024-03-06T14:28:54.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:54.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:54.579977+0000) 2024-03-06T14:28:55.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:55.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:55.580163+0000) 2024-03-06T14:28:55.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:55.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:28:56.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:56.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:56.580338+0000) 2024-03-06T14:28:57.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:57.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:57.580555+0000) 2024-03-06T14:28:58.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:58.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:58.580794+0000) 2024-03-06T14:28:59.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:28:59.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:28:59.580974+0000) 2024-03-06T14:29:00.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:00.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:00.581217+0000) 2024-03-06T14:29:01.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:01.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:01.581452+0000) 2024-03-06T14:29:02.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:02.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:02.581702+0000) 2024-03-06T14:29:03.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:03.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:03.581935+0000) 2024-03-06T14:29:04.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:04.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:04.582162+0000) 2024-03-06T14:29:05.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:05.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:05.582385+0000) 2024-03-06T14:29:06.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:06.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:06.582615+0000) 2024-03-06T14:29:07.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:07.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:07.582828+0000) 2024-03-06T14:29:08.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:08.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:08.583126+0000) 2024-03-06T14:29:09.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:09.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:09.583325+0000) 2024-03-06T14:29:10.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:10.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:10.583549+0000) 2024-03-06T14:29:11.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:11.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:11.583780+0000) 2024-03-06T14:29:12.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:12.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:12.584047+0000) 2024-03-06T14:29:13.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:13.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:13.584293+0000) 2024-03-06T14:29:14.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:14.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:14.584527+0000) 2024-03-06T14:29:15.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:15.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:15.584754+0000) 2024-03-06T14:29:16.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:16.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:16.584982+0000) 2024-03-06T14:29:17.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:17.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:17.585275+0000) 2024-03-06T14:29:18.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:18.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:18.585521+0000) 2024-03-06T14:29:19.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:19.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:19.585762+0000) 2024-03-06T14:29:20.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:20.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:20.586000+0000) 2024-03-06T14:29:21.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:21.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:21.586271+0000) 2024-03-06T14:29:22.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:22.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:22.586549+0000) 2024-03-06T14:29:23.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:23.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:23.586829+0000) 2024-03-06T14:29:24.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:24.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:24.587170+0000) 2024-03-06T14:29:25.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:25.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:25.587419+0000) 2024-03-06T14:29:25.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:25.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:26.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:26.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:26.587704+0000) 2024-03-06T14:29:27.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:27.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:27.587988+0000) 2024-03-06T14:29:28.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:28.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:28.588282+0000) 2024-03-06T14:29:29.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:29.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:29.588568+0000) 2024-03-06T14:29:30.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:30.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:30.588810+0000) 2024-03-06T14:29:31.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:31.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:31.589126+0000) 2024-03-06T14:29:32.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:32.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:32.589326+0000) 2024-03-06T14:29:33.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:33.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:33.589521+0000) 2024-03-06T14:29:34.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:34.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:34.589755+0000) 2024-03-06T14:29:35.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:35.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:35.589971+0000) 2024-03-06T14:29:36.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:36.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:36.590226+0000) 2024-03-06T14:29:37.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:37.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:37.590414+0000) 2024-03-06T14:29:38.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:38.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:38.590641+0000) 2024-03-06T14:29:39.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:39.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:39.590872+0000) 2024-03-06T14:29:40.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:40.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:40.591137+0000) 2024-03-06T14:29:41.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:41.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:41.591339+0000) 2024-03-06T14:29:42.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:42.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:42.591514+0000) 2024-03-06T14:29:43.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:43.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:43.591740+0000) 2024-03-06T14:29:44.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:44.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:44.591976+0000) 2024-03-06T14:29:45.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:45.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:45.592196+0000) 2024-03-06T14:29:46.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:46.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:46.592429+0000) 2024-03-06T14:29:47.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:47.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:47.592671+0000) 2024-03-06T14:29:48.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:48.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:48.592900+0000) 2024-03-06T14:29:49.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:49.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:49.593141+0000) 2024-03-06T14:29:50.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:50.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:50.593359+0000) 2024-03-06T14:29:51.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:51.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:51.593584+0000) 2024-03-06T14:29:52.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:52.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:52.593793+0000) 2024-03-06T14:29:53.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:53.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:53.594091+0000) 2024-03-06T14:29:54.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:54.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:54.594288+0000) 2024-03-06T14:29:55.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:55.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:55.594512+0000) 2024-03-06T14:29:56.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:56.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:29:56.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:56.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:56.594738+0000) 2024-03-06T14:29:57.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:57.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:57.594972+0000) 2024-03-06T14:29:58.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:58.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:58.595218+0000) 2024-03-06T14:29:59.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:29:59.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:29:59.595368+0000) 2024-03-06T14:30:00.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:00.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:00.595576+0000) 2024-03-06T14:30:01.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:01.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:01.595826+0000) 2024-03-06T14:30:02.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:02.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:02.596084+0000) 2024-03-06T14:30:03.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:03.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:03.596299+0000) 2024-03-06T14:30:04.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:04.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:04.596529+0000) 2024-03-06T14:30:05.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:05.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:05.596768+0000) 2024-03-06T14:30:06.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:06.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:06.597089+0000) 2024-03-06T14:30:07.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:07.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:07.597333+0000) 2024-03-06T14:30:08.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:08.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:08.597620+0000) 2024-03-06T14:30:09.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:09.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:09.597868+0000) 2024-03-06T14:30:10.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:10.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:10.598148+0000) 2024-03-06T14:30:11.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:11.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:11.598401+0000) 2024-03-06T14:30:12.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:12.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:12.598671+0000) 2024-03-06T14:30:13.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:13.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:13.598954+0000) 2024-03-06T14:30:14.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:14.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:14.599280+0000) 2024-03-06T14:30:15.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:15.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:15.599520+0000) 2024-03-06T14:30:16.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:16.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:16.599803+0000) 2024-03-06T14:30:17.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:17.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:17.600043+0000) 2024-03-06T14:30:18.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:18.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:18.600261+0000) 2024-03-06T14:30:19.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:19.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:19.600478+0000) 2024-03-06T14:30:20.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:20.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:20.600711+0000) 2024-03-06T14:30:21.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:21.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:21.600933+0000) 2024-03-06T14:30:22.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:22.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:22.601170+0000) 2024-03-06T14:30:23.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:23.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:23.601369+0000) 2024-03-06T14:30:24.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:24.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:24.601597+0000) 2024-03-06T14:30:25.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:25.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:25.601822+0000) 2024-03-06T14:30:26.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:26.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:26.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:26.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:26.602094+0000) 2024-03-06T14:30:27.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:27.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:27.602348+0000) 2024-03-06T14:30:28.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:28.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:28.602578+0000) 2024-03-06T14:30:29.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:29.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:29.602871+0000) 2024-03-06T14:30:30.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:30.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:30.603110+0000) 2024-03-06T14:30:31.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:31.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:31.603322+0000) 2024-03-06T14:30:32.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:32.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:32.603543+0000) 2024-03-06T14:30:33.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:33.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:33.603772+0000) 2024-03-06T14:30:34.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:34.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:34.603993+0000) 2024-03-06T14:30:35.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:35.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:35.604311+0000) 2024-03-06T14:30:36.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:36.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:36.604517+0000) 2024-03-06T14:30:37.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:37.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:37.604706+0000) 2024-03-06T14:30:38.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:38.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:38.604931+0000) 2024-03-06T14:30:39.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:39.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:39.605204+0000) 2024-03-06T14:30:40.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:40.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:40.605456+0000) 2024-03-06T14:30:41.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:41.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:41.605699+0000) 2024-03-06T14:30:42.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:42.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:42.605961+0000) 2024-03-06T14:30:43.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:43.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:43.606216+0000) 2024-03-06T14:30:44.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:44.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:44.606440+0000) 2024-03-06T14:30:45.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:45.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:45.606686+0000) 2024-03-06T14:30:46.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:46.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:46.606925+0000) 2024-03-06T14:30:47.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:47.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:47.607187+0000) 2024-03-06T14:30:48.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:48.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:48.607414+0000) 2024-03-06T14:30:49.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:49.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:49.607624+0000) 2024-03-06T14:30:50.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:50.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:50.607848+0000) 2024-03-06T14:30:51.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:51.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:51.608114+0000) 2024-03-06T14:30:52.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:52.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:52.608333+0000) 2024-03-06T14:30:53.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:53.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:53.608536+0000) 2024-03-06T14:30:54.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:54.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:54.608756+0000) 2024-03-06T14:30:55.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:55.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:55.608984+0000) 2024-03-06T14:30:56.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:56.455 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:30:56.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:56.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:56.609308+0000) 2024-03-06T14:30:57.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:57.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:57.609573+0000) 2024-03-06T14:30:58.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:58.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:58.609859+0000) 2024-03-06T14:30:59.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:30:59.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:30:59.610170+0000) 2024-03-06T14:31:00.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:00.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:00.610408+0000) 2024-03-06T14:31:01.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:01.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:01.610684+0000) 2024-03-06T14:31:02.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:02.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:02.610954+0000) 2024-03-06T14:31:03.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:03.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:03.611222+0000) 2024-03-06T14:31:04.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:04.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:04.611449+0000) 2024-03-06T14:31:05.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:05.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:05.611673+0000) 2024-03-06T14:31:06.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:06.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:06.611938+0000) 2024-03-06T14:31:07.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:07.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:07.612156+0000) 2024-03-06T14:31:08.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:08.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:08.612405+0000) 2024-03-06T14:31:09.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:09.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:09.612631+0000) 2024-03-06T14:31:10.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:10.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:10.612862+0000) 2024-03-06T14:31:11.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:11.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:11.613143+0000) 2024-03-06T14:31:12.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:12.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:12.613336+0000) 2024-03-06T14:31:13.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:13.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:13.613567+0000) 2024-03-06T14:31:14.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:14.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:14.613865+0000) 2024-03-06T14:31:15.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:15.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:15.614142+0000) 2024-03-06T14:31:16.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:16.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:16.614417+0000) 2024-03-06T14:31:17.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:17.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:17.614742+0000) 2024-03-06T14:31:18.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:18.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:18.615061+0000) 2024-03-06T14:31:19.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:19.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:19.615272+0000) 2024-03-06T14:31:20.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:20.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:20.615488+0000) 2024-03-06T14:31:21.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:21.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:21.615717+0000) 2024-03-06T14:31:22.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:22.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:22.615976+0000) 2024-03-06T14:31:23.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:23.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:23.616277+0000) 2024-03-06T14:31:24.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:24.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:24.616494+0000) 2024-03-06T14:31:25.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:25.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:25.616717+0000) 2024-03-06T14:31:26.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:26.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:26.616941+0000) 2024-03-06T14:31:26.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:26.685 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:27.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:27.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:27.617197+0000) 2024-03-06T14:31:28.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:28.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:28.617435+0000) 2024-03-06T14:31:29.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:29.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:29.617665+0000) 2024-03-06T14:31:30.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:30.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:30.617924+0000) 2024-03-06T14:31:31.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:31.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:31.618160+0000) 2024-03-06T14:31:32.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:32.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:32.618371+0000) 2024-03-06T14:31:33.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:33.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:33.618591+0000) 2024-03-06T14:31:34.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:34.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:34.618872+0000) 2024-03-06T14:31:35.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:35.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:35.619153+0000) 2024-03-06T14:31:36.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:36.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:36.619397+0000) 2024-03-06T14:31:37.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:37.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:37.619650+0000) 2024-03-06T14:31:38.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:38.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:38.619932+0000) 2024-03-06T14:31:39.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:39.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:39.620171+0000) 2024-03-06T14:31:40.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:40.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:40.620398+0000) 2024-03-06T14:31:41.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:41.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:41.620625+0000) 2024-03-06T14:31:42.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:42.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:42.620841+0000) 2024-03-06T14:31:43.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:43.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:43.621124+0000) 2024-03-06T14:31:44.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:44.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:44.621358+0000) 2024-03-06T14:31:45.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:45.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:45.621555+0000) 2024-03-06T14:31:46.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:46.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:46.621778+0000) 2024-03-06T14:31:47.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:47.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:47.622045+0000) 2024-03-06T14:31:48.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:48.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:48.622296+0000) 2024-03-06T14:31:49.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:49.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:49.622537+0000) 2024-03-06T14:31:50.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:50.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:50.622748+0000) 2024-03-06T14:31:51.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:51.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:51.622963+0000) 2024-03-06T14:31:52.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:52.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:52.623149+0000) 2024-03-06T14:31:53.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:53.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:53.623397+0000) 2024-03-06T14:31:54.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:54.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:54.623622+0000) 2024-03-06T14:31:55.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:55.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:55.623852+0000) 2024-03-06T14:31:56.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:56.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:56.624113+0000) 2024-03-06T14:31:56.914 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:56.922 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:31:57.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:57.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:57.624338+0000) 2024-03-06T14:31:58.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:58.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:58.624583+0000) 2024-03-06T14:31:59.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:31:59.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:31:59.624800+0000) 2024-03-06T14:32:00.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:00.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:00.625044+0000) 2024-03-06T14:32:01.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:01.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:01.625273+0000) 2024-03-06T14:32:02.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:02.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:02.625460+0000) 2024-03-06T14:32:03.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:03.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:03.625712+0000) 2024-03-06T14:32:04.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:04.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:04.625947+0000) 2024-03-06T14:32:05.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:05.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:05.626238+0000) 2024-03-06T14:32:06.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:06.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:06.626460+0000) 2024-03-06T14:32:07.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:07.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:07.626695+0000) 2024-03-06T14:32:08.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:08.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:08.626964+0000) 2024-03-06T14:32:09.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:09.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:09.627204+0000) 2024-03-06T14:32:10.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:10.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:10.627421+0000) 2024-03-06T14:32:11.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:11.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:11.627647+0000) 2024-03-06T14:32:12.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:12.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:12.627873+0000) 2024-03-06T14:32:13.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:13.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:13.628179+0000) 2024-03-06T14:32:14.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:14.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:14.628422+0000) 2024-03-06T14:32:15.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:15.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:15.628680+0000) 2024-03-06T14:32:16.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:16.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:16.628953+0000) 2024-03-06T14:32:17.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:17.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:17.629248+0000) 2024-03-06T14:32:18.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:18.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:18.629534+0000) 2024-03-06T14:32:19.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:19.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:19.629804+0000) 2024-03-06T14:32:20.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:20.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:20.630128+0000) 2024-03-06T14:32:21.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:21.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:21.630373+0000) 2024-03-06T14:32:22.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:22.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:22.630584+0000) 2024-03-06T14:32:23.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:23.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:23.630837+0000) 2024-03-06T14:32:24.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:24.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:24.631046+0000) 2024-03-06T14:32:25.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:25.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:25.631262+0000) 2024-03-06T14:32:26.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:26.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:26.631458+0000) 2024-03-06T14:32:27.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:27.067 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:27.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:27.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:27.631682+0000) 2024-03-06T14:32:28.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:28.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:28.631895+0000) 2024-03-06T14:32:29.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:29.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:29.632149+0000) 2024-03-06T14:32:30.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:30.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:30.632334+0000) 2024-03-06T14:32:31.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:31.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:31.632561+0000) 2024-03-06T14:32:32.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:32.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:32.632786+0000) 2024-03-06T14:32:33.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:33.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:33.632962+0000) 2024-03-06T14:32:34.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:34.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:34.633178+0000) 2024-03-06T14:32:35.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:35.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:35.633428+0000) 2024-03-06T14:32:36.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:36.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:36.633674+0000) 2024-03-06T14:32:37.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:37.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:37.633883+0000) 2024-03-06T14:32:38.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:38.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:38.634167+0000) 2024-03-06T14:32:39.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:39.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:39.634364+0000) 2024-03-06T14:32:40.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:40.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:40.634551+0000) 2024-03-06T14:32:41.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:41.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:41.634776+0000) 2024-03-06T14:32:42.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:42.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:42.635050+0000) 2024-03-06T14:32:43.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:43.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:43.635269+0000) 2024-03-06T14:32:44.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:44.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:44.635510+0000) 2024-03-06T14:32:45.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:45.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:45.635705+0000) 2024-03-06T14:32:46.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:46.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:46.635934+0000) 2024-03-06T14:32:47.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:47.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:47.636194+0000) 2024-03-06T14:32:48.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:48.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:48.636464+0000) 2024-03-06T14:32:49.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:49.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:49.636717+0000) 2024-03-06T14:32:50.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:50.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:50.636953+0000) 2024-03-06T14:32:51.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:51.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:51.637218+0000) 2024-03-06T14:32:52.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:52.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:52.637459+0000) 2024-03-06T14:32:53.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:53.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:53.637668+0000) 2024-03-06T14:32:54.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:54.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:54.637887+0000) 2024-03-06T14:32:55.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:55.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:55.638121+0000) 2024-03-06T14:32:56.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:56.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:56.638327+0000) 2024-03-06T14:32:57.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:57.280 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:32:57.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:57.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:57.638550+0000) 2024-03-06T14:32:58.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:58.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:58.638777+0000) 2024-03-06T14:32:59.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:32:59.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:32:59.639000+0000) 2024-03-06T14:33:00.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:00.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:00.639294+0000) 2024-03-06T14:33:01.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:01.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:01.639569+0000) 2024-03-06T14:33:02.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:02.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:02.639803+0000) 2024-03-06T14:33:03.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:03.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:03.640091+0000) 2024-03-06T14:33:04.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:04.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:04.640310+0000) 2024-03-06T14:33:05.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:05.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:05.640530+0000) 2024-03-06T14:33:06.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:06.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:06.640733+0000) 2024-03-06T14:33:07.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:07.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:07.640968+0000) 2024-03-06T14:33:08.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:08.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:08.641220+0000) 2024-03-06T14:33:09.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:09.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:09.641461+0000) 2024-03-06T14:33:10.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:10.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:10.641698+0000) 2024-03-06T14:33:11.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:11.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:11.641940+0000) 2024-03-06T14:33:12.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:12.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:12.642159+0000) 2024-03-06T14:33:13.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:13.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:13.642385+0000) 2024-03-06T14:33:14.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:14.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:14.642630+0000) 2024-03-06T14:33:15.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:15.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:15.642870+0000) 2024-03-06T14:33:16.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:16.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:16.643154+0000) 2024-03-06T14:33:17.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:17.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:17.643394+0000) 2024-03-06T14:33:18.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:18.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:18.643672+0000) 2024-03-06T14:33:19.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:19.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:19.643941+0000) 2024-03-06T14:33:20.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:20.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:20.644214+0000) 2024-03-06T14:33:21.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:21.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:21.644462+0000) 2024-03-06T14:33:22.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:22.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:22.644735+0000) 2024-03-06T14:33:23.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:23.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:23.645059+0000) 2024-03-06T14:33:24.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:24.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:24.645370+0000) 2024-03-06T14:33:25.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:25.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:25.645585+0000) 2024-03-06T14:33:26.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:26.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:26.645722+0000) 2024-03-06T14:33:27.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:27.484 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:27.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:27.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:27.645963+0000) 2024-03-06T14:33:28.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:28.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:28.646210+0000) 2024-03-06T14:33:29.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:29.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:29.646414+0000) 2024-03-06T14:33:30.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:30.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:30.646687+0000) 2024-03-06T14:33:31.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:31.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:31.646959+0000) 2024-03-06T14:33:32.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:32.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:32.647206+0000) 2024-03-06T14:33:33.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:33.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:33.647432+0000) 2024-03-06T14:33:34.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:34.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:34.647699+0000) 2024-03-06T14:33:35.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:35.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:35.647977+0000) 2024-03-06T14:33:36.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:36.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:36.648282+0000) 2024-03-06T14:33:37.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:37.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:37.648567+0000) 2024-03-06T14:33:38.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:38.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:38.648849+0000) 2024-03-06T14:33:39.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:39.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:39.649164+0000) 2024-03-06T14:33:40.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:40.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:40.649399+0000) 2024-03-06T14:33:41.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:41.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:41.649683+0000) 2024-03-06T14:33:42.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:42.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:42.649962+0000) 2024-03-06T14:33:43.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:43.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:43.650255+0000) 2024-03-06T14:33:44.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:44.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:44.650515+0000) 2024-03-06T14:33:45.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:45.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:45.650746+0000) 2024-03-06T14:33:46.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:46.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:46.650984+0000) 2024-03-06T14:33:47.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:47.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:47.651259+0000) 2024-03-06T14:33:48.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:48.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:48.651524+0000) 2024-03-06T14:33:49.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:49.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:49.651771+0000) 2024-03-06T14:33:50.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:50.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:50.652052+0000) 2024-03-06T14:33:51.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:51.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:51.652305+0000) 2024-03-06T14:33:52.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:52.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:52.652503+0000) 2024-03-06T14:33:53.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:53.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:53.652737+0000) 2024-03-06T14:33:54.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:54.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:54.652994+0000) 2024-03-06T14:33:55.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:55.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:55.653282+0000) 2024-03-06T14:33:56.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:56.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:56.653525+0000) 2024-03-06T14:33:57.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:57.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:57.653773+0000) 2024-03-06T14:33:57.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:57.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:33:58.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:58.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:58.654054+0000) 2024-03-06T14:33:59.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:33:59.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:33:59.654307+0000) 2024-03-06T14:34:00.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:00.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:00.654536+0000) 2024-03-06T14:34:01.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:01.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:01.654731+0000) 2024-03-06T14:34:02.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:02.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:02.654954+0000) 2024-03-06T14:34:03.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:03.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:03.655157+0000) 2024-03-06T14:34:04.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:04.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:04.655369+0000) 2024-03-06T14:34:05.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:05.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:05.655576+0000) 2024-03-06T14:34:06.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:06.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:06.655802+0000) 2024-03-06T14:34:07.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:07.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:07.656070+0000) 2024-03-06T14:34:08.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:08.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:08.656296+0000) 2024-03-06T14:34:09.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:09.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:09.656482+0000) 2024-03-06T14:34:10.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:10.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:10.656698+0000) 2024-03-06T14:34:11.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:11.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:11.656963+0000) 2024-03-06T14:34:12.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:12.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:12.657227+0000) 2024-03-06T14:34:13.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:13.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:13.657450+0000) 2024-03-06T14:34:14.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:14.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:14.657670+0000) 2024-03-06T14:34:15.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:15.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:15.657919+0000) 2024-03-06T14:34:16.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:16.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:16.658164+0000) 2024-03-06T14:34:17.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:17.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:17.658381+0000) 2024-03-06T14:34:18.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:18.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:18.658600+0000) 2024-03-06T14:34:19.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:19.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:19.658836+0000) 2024-03-06T14:34:20.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:20.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:20.659106+0000) 2024-03-06T14:34:21.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:21.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:21.659333+0000) 2024-03-06T14:34:22.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:22.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:22.659508+0000) 2024-03-06T14:34:23.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:23.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:23.659753+0000) 2024-03-06T14:34:24.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:24.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:24.660045+0000) 2024-03-06T14:34:25.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:25.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:25.660271+0000) 2024-03-06T14:34:26.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:26.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:26.660515+0000) 2024-03-06T14:34:27.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:27.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:27.660695+0000) 2024-03-06T14:34:27.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:27.868 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:28.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:28.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:28.660923+0000) 2024-03-06T14:34:29.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:29.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:29.661154+0000) 2024-03-06T14:34:30.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:30.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:30.661374+0000) 2024-03-06T14:34:31.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:31.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:31.661633+0000) 2024-03-06T14:34:32.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:32.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:32.661841+0000) 2024-03-06T14:34:33.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:33.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:33.662095+0000) 2024-03-06T14:34:34.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:34.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:34.662330+0000) 2024-03-06T14:34:35.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:35.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:35.662560+0000) 2024-03-06T14:34:36.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:36.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:36.662783+0000) 2024-03-06T14:34:37.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:37.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:37.663061+0000) 2024-03-06T14:34:38.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:38.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:38.663259+0000) 2024-03-06T14:34:39.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:39.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:39.663486+0000) 2024-03-06T14:34:40.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:40.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:40.663709+0000) 2024-03-06T14:34:41.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:41.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:41.663888+0000) 2024-03-06T14:34:42.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:42.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:42.664116+0000) 2024-03-06T14:34:43.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:43.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:43.664362+0000) 2024-03-06T14:34:44.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:44.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:44.664619+0000) 2024-03-06T14:34:45.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:45.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:45.664838+0000) 2024-03-06T14:34:46.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:46.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:46.665125+0000) 2024-03-06T14:34:47.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:47.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:47.665389+0000) 2024-03-06T14:34:48.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:48.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:48.665611+0000) 2024-03-06T14:34:49.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:49.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:49.665824+0000) 2024-03-06T14:34:50.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:50.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:50.666058+0000) 2024-03-06T14:34:51.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:51.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:51.666254+0000) 2024-03-06T14:34:52.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:52.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:52.666484+0000) 2024-03-06T14:34:53.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:53.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:53.666669+0000) 2024-03-06T14:34:54.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:54.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:54.666906+0000) 2024-03-06T14:34:55.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:55.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:55.667156+0000) 2024-03-06T14:34:56.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:56.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:56.667401+0000) 2024-03-06T14:34:57.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:57.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:57.667645+0000) 2024-03-06T14:34:58.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:58.070 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:34:58.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:58.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:58.667884+0000) 2024-03-06T14:34:59.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:34:59.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:34:59.668112+0000) 2024-03-06T14:35:00.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:00.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:00.668342+0000) 2024-03-06T14:35:01.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:01.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:01.668568+0000) 2024-03-06T14:35:02.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:02.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:02.668787+0000) 2024-03-06T14:35:03.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:03.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:03.669036+0000) 2024-03-06T14:35:04.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:04.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:04.669242+0000) 2024-03-06T14:35:05.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:05.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:05.669449+0000) 2024-03-06T14:35:06.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:06.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:06.669675+0000) 2024-03-06T14:35:07.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:07.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:07.669926+0000) 2024-03-06T14:35:08.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:08.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:08.670150+0000) 2024-03-06T14:35:09.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:09.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:09.670387+0000) 2024-03-06T14:35:10.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:10.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:10.670615+0000) 2024-03-06T14:35:11.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:11.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:11.670846+0000) 2024-03-06T14:35:12.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:12.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:12.671114+0000) 2024-03-06T14:35:13.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:13.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:13.671333+0000) 2024-03-06T14:35:14.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:14.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:14.671576+0000) 2024-03-06T14:35:15.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:15.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:15.671768+0000) 2024-03-06T14:35:16.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:16.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:16.672036+0000) 2024-03-06T14:35:17.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:17.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:17.672197+0000) 2024-03-06T14:35:18.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:18.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:18.672410+0000) 2024-03-06T14:35:19.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:19.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:19.672650+0000) 2024-03-06T14:35:20.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:20.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:20.672880+0000) 2024-03-06T14:35:21.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:21.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:21.673144+0000) 2024-03-06T14:35:22.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:22.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:22.673373+0000) 2024-03-06T14:35:23.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:23.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:23.673606+0000) 2024-03-06T14:35:24.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:24.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:24.673847+0000) 2024-03-06T14:35:25.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:25.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:25.674147+0000) 2024-03-06T14:35:26.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:26.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:26.674353+0000) 2024-03-06T14:35:27.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:27.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:27.674576+0000) 2024-03-06T14:35:28.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:28.301 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:28.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:28.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:28.674862+0000) 2024-03-06T14:35:29.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:29.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:29.675141+0000) 2024-03-06T14:35:30.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:30.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:30.675322+0000) 2024-03-06T14:35:31.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:31.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:31.675547+0000) 2024-03-06T14:35:32.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:32.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:32.675772+0000) 2024-03-06T14:35:33.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:33.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:33.676001+0000) 2024-03-06T14:35:34.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:34.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:34.676209+0000) 2024-03-06T14:35:35.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:35.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:35.676453+0000) 2024-03-06T14:35:36.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:36.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:36.676697+0000) 2024-03-06T14:35:37.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:37.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:37.676916+0000) 2024-03-06T14:35:38.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:38.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:38.677141+0000) 2024-03-06T14:35:39.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:39.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:39.677345+0000) 2024-03-06T14:35:40.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:40.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:40.677573+0000) 2024-03-06T14:35:41.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:41.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:41.677801+0000) 2024-03-06T14:35:42.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:42.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:42.678057+0000) 2024-03-06T14:35:43.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:43.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:43.678253+0000) 2024-03-06T14:35:44.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:44.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:44.678495+0000) 2024-03-06T14:35:45.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:45.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:45.678713+0000) 2024-03-06T14:35:46.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:46.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:46.678931+0000) 2024-03-06T14:35:47.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:47.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:47.679163+0000) 2024-03-06T14:35:48.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:48.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:48.679365+0000) 2024-03-06T14:35:49.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:49.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:49.679599+0000) 2024-03-06T14:35:50.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:50.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:50.679824+0000) 2024-03-06T14:35:51.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:51.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:51.680112+0000) 2024-03-06T14:35:52.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:52.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:52.680316+0000) 2024-03-06T14:35:53.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:53.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:53.680507+0000) 2024-03-06T14:35:54.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:54.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:54.680718+0000) 2024-03-06T14:35:55.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:55.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:55.680925+0000) 2024-03-06T14:35:56.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:56.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:56.681160+0000) 2024-03-06T14:35:57.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:57.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:57.681385+0000) 2024-03-06T14:35:58.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:58.504 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:35:58.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:58.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:58.681555+0000) 2024-03-06T14:35:59.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:35:59.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:35:59.681784+0000) 2024-03-06T14:36:00.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:00.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:00.682001+0000) 2024-03-06T14:36:01.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:01.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:01.682241+0000) 2024-03-06T14:36:02.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:02.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:02.682455+0000) 2024-03-06T14:36:03.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:03.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:03.682684+0000) 2024-03-06T14:36:04.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:04.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:04.682926+0000) 2024-03-06T14:36:05.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:05.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:05.683143+0000) 2024-03-06T14:36:06.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:06.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:06.683321+0000) 2024-03-06T14:36:07.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:07.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:07.683542+0000) 2024-03-06T14:36:08.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:08.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:08.683761+0000) 2024-03-06T14:36:09.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:09.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:09.683987+0000) 2024-03-06T14:36:10.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:10.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:10.684258+0000) 2024-03-06T14:36:11.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:11.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:11.684486+0000) 2024-03-06T14:36:12.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:12.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:12.684716+0000) 2024-03-06T14:36:13.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:13.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:13.684934+0000) 2024-03-06T14:36:14.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:14.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:14.685198+0000) 2024-03-06T14:36:15.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:15.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:15.685424+0000) 2024-03-06T14:36:16.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:16.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:16.685671+0000) 2024-03-06T14:36:17.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:17.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:17.685893+0000) 2024-03-06T14:36:18.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:18.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:18.686114+0000) 2024-03-06T14:36:19.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:19.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:19.686313+0000) 2024-03-06T14:36:20.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:20.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:20.686502+0000) 2024-03-06T14:36:21.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:21.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:21.686717+0000) 2024-03-06T14:36:22.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:22.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:22.686937+0000) 2024-03-06T14:36:23.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:23.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:23.687184+0000) 2024-03-06T14:36:24.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:24.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:24.687454+0000) 2024-03-06T14:36:25.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:25.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:25.687702+0000) 2024-03-06T14:36:26.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:26.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:26.687869+0000) 2024-03-06T14:36:27.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:27.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:27.688097+0000) 2024-03-06T14:36:28.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:28.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:28.688292+0000) 2024-03-06T14:36:28.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:28.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:29.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:29.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:29.688532+0000) 2024-03-06T14:36:30.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:30.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:30.688758+0000) 2024-03-06T14:36:31.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:31.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:31.688987+0000) 2024-03-06T14:36:32.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:32.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:32.689233+0000) 2024-03-06T14:36:33.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:33.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:33.689452+0000) 2024-03-06T14:36:34.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:34.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:34.689705+0000) 2024-03-06T14:36:35.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:35.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:35.689892+0000) 2024-03-06T14:36:36.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:36.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:36.690128+0000) 2024-03-06T14:36:37.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:37.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:37.690396+0000) 2024-03-06T14:36:38.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:38.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:38.690619+0000) 2024-03-06T14:36:39.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:39.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:39.690843+0000) 2024-03-06T14:36:40.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:40.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:40.691119+0000) 2024-03-06T14:36:41.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:41.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:41.691366+0000) 2024-03-06T14:36:42.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:42.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:42.691541+0000) 2024-03-06T14:36:43.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:43.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:43.691797+0000) 2024-03-06T14:36:44.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:44.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:44.692162+0000) 2024-03-06T14:36:45.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:45.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:45.692429+0000) 2024-03-06T14:36:46.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:46.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:46.692703+0000) 2024-03-06T14:36:47.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:47.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:47.692980+0000) 2024-03-06T14:36:48.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:48.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:48.693246+0000) 2024-03-06T14:36:49.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:49.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:49.693510+0000) 2024-03-06T14:36:50.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:50.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:50.693787+0000) 2024-03-06T14:36:51.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:51.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:51.694114+0000) 2024-03-06T14:36:52.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:52.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:52.694358+0000) 2024-03-06T14:36:53.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:53.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:53.694636+0000) 2024-03-06T14:36:54.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:54.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:54.694932+0000) 2024-03-06T14:36:55.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:55.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:55.695193+0000) 2024-03-06T14:36:56.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:56.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:56.695441+0000) 2024-03-06T14:36:57.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:57.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:57.695718+0000) 2024-03-06T14:36:58.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:58.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:58.695971+0000) 2024-03-06T14:36:58.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:58.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:36:59.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:36:59.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:36:59.696181+0000) 2024-03-06T14:37:00.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:00.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:00.696386+0000) 2024-03-06T14:37:01.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:01.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:01.696612+0000) 2024-03-06T14:37:02.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:02.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:02.696833+0000) 2024-03-06T14:37:03.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:03.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:03.697104+0000) 2024-03-06T14:37:04.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:04.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:04.697376+0000) 2024-03-06T14:37:05.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:05.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:05.697610+0000) 2024-03-06T14:37:06.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:06.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:06.697848+0000) 2024-03-06T14:37:07.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:07.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:07.698162+0000) 2024-03-06T14:37:08.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:08.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:08.698404+0000) 2024-03-06T14:37:09.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:09.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:09.698657+0000) 2024-03-06T14:37:10.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:10.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:10.698898+0000) 2024-03-06T14:37:11.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:11.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:11.699146+0000) 2024-03-06T14:37:12.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:12.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:12.699424+0000) 2024-03-06T14:37:13.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:13.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:13.699708+0000) 2024-03-06T14:37:14.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:14.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:14.700054+0000) 2024-03-06T14:37:15.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:15.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:15.700321+0000) 2024-03-06T14:37:16.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:16.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:16.700565+0000) 2024-03-06T14:37:17.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:17.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:17.700814+0000) 2024-03-06T14:37:18.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:18.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:18.701103+0000) 2024-03-06T14:37:19.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:19.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:19.701383+0000) 2024-03-06T14:37:20.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:20.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:20.701655+0000) 2024-03-06T14:37:21.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:21.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:21.701870+0000) 2024-03-06T14:37:22.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:22.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:22.702157+0000) 2024-03-06T14:37:23.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:23.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:23.702407+0000) 2024-03-06T14:37:24.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:24.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:24.702677+0000) 2024-03-06T14:37:25.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:25.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:25.702952+0000) 2024-03-06T14:37:26.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:26.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:26.703235+0000) 2024-03-06T14:37:27.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:27.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:27.703523+0000) 2024-03-06T14:37:28.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:28.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:28.703767+0000) 2024-03-06T14:37:29.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:29.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:29.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:29.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:29.704080+0000) 2024-03-06T14:37:30.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:30.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:30.704355+0000) 2024-03-06T14:37:31.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:31.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:31.704543+0000) 2024-03-06T14:37:32.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:32.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:32.704763+0000) 2024-03-06T14:37:33.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:33.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:33.704970+0000) 2024-03-06T14:37:34.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:34.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:34.705229+0000) 2024-03-06T14:37:35.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:35.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:35.705470+0000) 2024-03-06T14:37:36.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:36.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:36.705713+0000) 2024-03-06T14:37:37.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:37.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:37.705951+0000) 2024-03-06T14:37:38.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:38.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:38.706169+0000) 2024-03-06T14:37:39.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:39.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:39.706421+0000) 2024-03-06T14:37:40.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:40.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:40.706694+0000) 2024-03-06T14:37:41.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:41.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:41.706971+0000) 2024-03-06T14:37:42.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:42.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:42.707266+0000) 2024-03-06T14:37:43.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:43.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:43.707499+0000) 2024-03-06T14:37:44.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:44.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:44.707739+0000) 2024-03-06T14:37:45.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:45.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:45.707976+0000) 2024-03-06T14:37:46.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:46.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:46.708241+0000) 2024-03-06T14:37:47.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:47.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:47.708466+0000) 2024-03-06T14:37:48.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:48.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:48.708673+0000) 2024-03-06T14:37:49.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:49.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:49.708934+0000) 2024-03-06T14:37:50.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:50.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:50.709189+0000) 2024-03-06T14:37:51.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:51.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:51.709443+0000) 2024-03-06T14:37:52.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:52.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:52.709679+0000) 2024-03-06T14:37:53.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:53.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:53.709936+0000) 2024-03-06T14:37:54.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:54.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:54.710234+0000) 2024-03-06T14:37:55.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:55.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:55.710527+0000) 2024-03-06T14:37:56.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:56.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:56.710700+0000) 2024-03-06T14:37:57.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:57.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:57.710935+0000) 2024-03-06T14:37:58.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:58.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:58.711159+0000) 2024-03-06T14:37:59.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:59.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:37:59.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:37:59.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:37:59.711380+0000) 2024-03-06T14:38:00.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:00.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:00.711593+0000) 2024-03-06T14:38:01.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:01.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:01.711805+0000) 2024-03-06T14:38:02.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:02.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:02.712077+0000) 2024-03-06T14:38:03.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:03.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:03.712315+0000) 2024-03-06T14:38:04.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:04.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:04.712564+0000) 2024-03-06T14:38:05.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:05.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:05.712783+0000) 2024-03-06T14:38:06.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:06.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:06.713048+0000) 2024-03-06T14:38:07.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:07.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:07.713313+0000) 2024-03-06T14:38:08.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:08.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:08.713565+0000) 2024-03-06T14:38:09.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:09.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:09.713808+0000) 2024-03-06T14:38:10.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:10.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:10.714061+0000) 2024-03-06T14:38:11.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:11.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:11.714279+0000) 2024-03-06T14:38:12.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:12.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:12.714540+0000) 2024-03-06T14:38:13.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:13.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:13.714769+0000) 2024-03-06T14:38:14.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:14.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:14.715128+0000) 2024-03-06T14:38:15.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:15.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:15.715401+0000) 2024-03-06T14:38:16.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:16.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:16.715674+0000) 2024-03-06T14:38:17.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:17.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:17.715940+0000) 2024-03-06T14:38:18.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:18.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:18.716194+0000) 2024-03-06T14:38:19.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:19.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:19.716451+0000) 2024-03-06T14:38:20.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:20.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:20.716733+0000) 2024-03-06T14:38:21.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:21.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:21.717056+0000) 2024-03-06T14:38:22.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:22.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:22.717303+0000) 2024-03-06T14:38:23.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:23.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:23.717581+0000) 2024-03-06T14:38:24.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:24.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:24.717852+0000) 2024-03-06T14:38:25.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:25.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:25.718130+0000) 2024-03-06T14:38:26.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:26.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:26.718410+0000) 2024-03-06T14:38:27.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:27.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:27.718652+0000) 2024-03-06T14:38:28.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:28.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:28.718928+0000) 2024-03-06T14:38:29.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:29.426 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:29.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:29.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:29.719229+0000) 2024-03-06T14:38:30.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:30.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:30.719475+0000) 2024-03-06T14:38:31.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:31.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:31.719717+0000) 2024-03-06T14:38:32.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:32.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:32.719937+0000) 2024-03-06T14:38:33.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:33.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:33.720148+0000) 2024-03-06T14:38:34.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:34.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:34.720430+0000) 2024-03-06T14:38:35.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:35.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:35.720707+0000) 2024-03-06T14:38:36.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:36.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:36.720984+0000) 2024-03-06T14:38:37.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:37.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:37.721246+0000) 2024-03-06T14:38:38.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:38.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:38.721449+0000) 2024-03-06T14:38:39.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:39.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:39.721714+0000) 2024-03-06T14:38:40.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:40.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:40.721962+0000) 2024-03-06T14:38:41.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:41.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:41.722174+0000) 2024-03-06T14:38:42.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:42.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:42.722362+0000) 2024-03-06T14:38:43.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:43.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:43.722611+0000) 2024-03-06T14:38:44.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:44.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:44.722910+0000) 2024-03-06T14:38:45.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:45.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:45.723245+0000) 2024-03-06T14:38:46.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:46.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:46.723522+0000) 2024-03-06T14:38:47.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:47.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:47.723778+0000) 2024-03-06T14:38:48.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:48.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:48.724104+0000) 2024-03-06T14:38:49.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:49.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:49.724295+0000) 2024-03-06T14:38:50.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:50.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:50.724562+0000) 2024-03-06T14:38:51.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:51.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:51.724838+0000) 2024-03-06T14:38:52.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:52.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:52.725139+0000) 2024-03-06T14:38:53.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:53.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:53.725408+0000) 2024-03-06T14:38:54.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:54.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:54.725675+0000) 2024-03-06T14:38:55.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:55.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:55.725925+0000) 2024-03-06T14:38:56.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:56.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:56.726171+0000) 2024-03-06T14:38:57.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:57.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:57.726394+0000) 2024-03-06T14:38:58.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:58.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:58.726613+0000) 2024-03-06T14:38:59.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:59.648 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:38:59.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:38:59.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:38:59.726813+0000) 2024-03-06T14:39:00.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:00.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:00.727115+0000) 2024-03-06T14:39:01.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:01.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:01.727390+0000) 2024-03-06T14:39:02.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:02.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:02.727614+0000) 2024-03-06T14:39:03.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:03.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:03.727884+0000) 2024-03-06T14:39:04.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:04.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:04.728146+0000) 2024-03-06T14:39:05.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:05.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:05.728345+0000) 2024-03-06T14:39:06.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:06.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:06.728570+0000) 2024-03-06T14:39:07.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:07.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:07.728793+0000) 2024-03-06T14:39:08.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:08.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:08.729051+0000) 2024-03-06T14:39:09.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:09.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:09.729272+0000) 2024-03-06T14:39:10.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:10.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:10.729492+0000) 2024-03-06T14:39:11.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:11.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:11.729733+0000) 2024-03-06T14:39:12.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:12.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:12.729986+0000) 2024-03-06T14:39:13.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:13.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:13.730335+0000) 2024-03-06T14:39:14.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:14.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:14.730621+0000) 2024-03-06T14:39:15.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:15.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:15.730876+0000) 2024-03-06T14:39:16.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:16.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:16.731150+0000) 2024-03-06T14:39:17.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:17.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:17.731387+0000) 2024-03-06T14:39:18.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:18.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:18.731602+0000) 2024-03-06T14:39:19.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:19.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:19.731832+0000) 2024-03-06T14:39:20.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:20.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:20.732121+0000) 2024-03-06T14:39:21.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:21.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:21.732355+0000) 2024-03-06T14:39:22.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:22.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:22.732589+0000) 2024-03-06T14:39:23.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:23.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:23.732814+0000) 2024-03-06T14:39:24.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:24.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:24.733147+0000) 2024-03-06T14:39:25.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:25.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:25.733348+0000) 2024-03-06T14:39:26.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:26.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:26.733539+0000) 2024-03-06T14:39:27.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:27.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:27.733758+0000) 2024-03-06T14:39:28.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:28.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:28.733911+0000) 2024-03-06T14:39:29.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:29.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:29.734176+0000) 2024-03-06T14:39:29.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:29.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:39:30.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:30.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:30.734453+0000) 2024-03-06T14:39:31.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:31.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:31.734726+0000) 2024-03-06T14:39:32.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:32.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:32.734942+0000) 2024-03-06T14:39:33.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:33.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:33.735166+0000) 2024-03-06T14:39:34.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:34.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:34.735398+0000) 2024-03-06T14:39:35.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:35.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:35.735621+0000) 2024-03-06T14:39:36.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:36.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:36.735903+0000) 2024-03-06T14:39:37.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:37.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:37.736146+0000) 2024-03-06T14:39:38.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:38.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:38.736327+0000) 2024-03-06T14:39:39.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:39.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:39.736551+0000) 2024-03-06T14:39:40.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:40.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:40.736813+0000) 2024-03-06T14:39:41.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:41.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:41.737026+0000) 2024-03-06T14:39:42.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:42.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:42.737258+0000) 2024-03-06T14:39:43.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:43.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:43.737502+0000) 2024-03-06T14:39:44.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:44.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:44.737760+0000) 2024-03-06T14:39:45.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:45.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:45.738067+0000) 2024-03-06T14:39:46.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:46.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:46.738349+0000) 2024-03-06T14:39:47.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:47.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:47.738551+0000) 2024-03-06T14:39:48.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:48.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:48.738807+0000) 2024-03-06T14:39:49.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:49.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:49.739095+0000) 2024-03-06T14:39:50.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:50.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:50.739325+0000) 2024-03-06T14:39:51.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:51.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:51.739545+0000) 2024-03-06T14:39:52.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:52.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:52.739821+0000) 2024-03-06T14:39:53.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:53.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:53.740110+0000) 2024-03-06T14:39:54.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:54.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:54.740413+0000) 2024-03-06T14:39:55.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:55.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:55.740690+0000) 2024-03-06T14:39:56.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:56.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:56.740966+0000) 2024-03-06T14:39:57.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:57.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:57.741219+0000) 2024-03-06T14:39:58.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:58.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:58.741444+0000) 2024-03-06T14:39:59.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:39:59.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:39:59.741725+0000) 2024-03-06T14:39:59.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:00.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:00.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:00.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:00.742000+0000) 2024-03-06T14:40:01.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:01.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:01.742297+0000) 2024-03-06T14:40:02.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:02.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:02.742829+0000) 2024-03-06T14:40:03.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:03.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:03.743158+0000) 2024-03-06T14:40:04.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:04.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:04.743474+0000) 2024-03-06T14:40:05.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:05.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:05.743744+0000) 2024-03-06T14:40:06.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:06.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:06.744054+0000) 2024-03-06T14:40:07.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:07.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:07.744316+0000) 2024-03-06T14:40:08.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:08.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:08.744526+0000) 2024-03-06T14:40:09.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:09.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:09.744752+0000) 2024-03-06T14:40:10.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:10.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:10.744980+0000) 2024-03-06T14:40:11.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:11.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:11.745222+0000) 2024-03-06T14:40:12.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:12.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:12.745442+0000) 2024-03-06T14:40:13.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:13.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:13.745720+0000) 2024-03-06T14:40:14.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:14.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:14.745999+0000) 2024-03-06T14:40:15.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:15.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:15.746300+0000) 2024-03-06T14:40:16.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:16.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:16.746554+0000) 2024-03-06T14:40:17.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:17.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:17.746837+0000) 2024-03-06T14:40:18.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:18.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:18.747079+0000) 2024-03-06T14:40:19.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:19.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:19.747312+0000) 2024-03-06T14:40:20.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:20.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:20.747559+0000) 2024-03-06T14:40:21.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:21.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:21.747797+0000) 2024-03-06T14:40:22.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:22.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:22.748076+0000) 2024-03-06T14:40:23.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:23.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:23.748322+0000) 2024-03-06T14:40:24.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:24.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:24.748572+0000) 2024-03-06T14:40:25.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:25.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:25.748806+0000) 2024-03-06T14:40:26.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:26.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:26.749113+0000) 2024-03-06T14:40:27.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:27.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:27.749344+0000) 2024-03-06T14:40:28.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:28.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:28.749621+0000) 2024-03-06T14:40:29.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:29.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:29.749894+0000) 2024-03-06T14:40:30.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:30.177 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:40:30.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:30.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:30.750177+0000) 2024-03-06T14:40:31.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:31.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:31.750428+0000) 2024-03-06T14:40:32.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:32.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:32.750699+0000) 2024-03-06T14:40:33.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:33.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:33.750980+0000) 2024-03-06T14:40:34.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:34.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:34.751299+0000) 2024-03-06T14:40:35.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:35.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:35.751507+0000) 2024-03-06T14:40:36.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:36.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:36.751733+0000) 2024-03-06T14:40:37.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:37.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:37.751982+0000) 2024-03-06T14:40:38.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:38.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:38.752301+0000) 2024-03-06T14:40:39.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:39.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:39.752580+0000) 2024-03-06T14:40:40.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:40.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:40.752877+0000) 2024-03-06T14:40:41.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:41.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:41.753147+0000) 2024-03-06T14:40:42.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:42.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:42.753388+0000) 2024-03-06T14:40:43.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:43.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:43.753652+0000) 2024-03-06T14:40:44.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:44.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:44.753933+0000) 2024-03-06T14:40:45.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:45.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:45.754161+0000) 2024-03-06T14:40:46.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:46.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:46.754358+0000) 2024-03-06T14:40:47.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:47.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:47.754677+0000) 2024-03-06T14:40:48.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:48.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:48.754952+0000) 2024-03-06T14:40:49.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:49.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:49.755198+0000) 2024-03-06T14:40:50.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:50.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:50.755452+0000) 2024-03-06T14:40:51.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:51.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:51.755693+0000) 2024-03-06T14:40:52.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:52.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:52.755975+0000) 2024-03-06T14:40:53.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:53.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:53.756256+0000) 2024-03-06T14:40:54.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:54.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:54.756503+0000) 2024-03-06T14:40:55.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:55.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:55.756782+0000) 2024-03-06T14:40:56.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:56.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:56.757102+0000) 2024-03-06T14:40:57.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:57.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:57.757343+0000) 2024-03-06T14:40:58.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:58.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:58.757580+0000) 2024-03-06T14:40:59.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:40:59.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:40:59.757797+0000) 2024-03-06T14:41:00.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:00.381 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:00.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:00.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:00.758054+0000) 2024-03-06T14:41:01.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:01.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:01.758296+0000) 2024-03-06T14:41:02.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:02.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:02.758556+0000) 2024-03-06T14:41:03.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:03.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:03.758833+0000) 2024-03-06T14:41:04.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:04.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:04.759156+0000) 2024-03-06T14:41:05.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:05.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:05.759392+0000) 2024-03-06T14:41:06.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:06.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:06.759610+0000) 2024-03-06T14:41:07.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:07.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:07.759838+0000) 2024-03-06T14:41:08.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:08.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:08.760159+0000) 2024-03-06T14:41:09.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:09.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:09.760400+0000) 2024-03-06T14:41:10.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:10.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:10.760614+0000) 2024-03-06T14:41:11.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:11.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:11.760841+0000) 2024-03-06T14:41:12.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:12.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:12.761147+0000) 2024-03-06T14:41:13.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:13.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:13.761424+0000) 2024-03-06T14:41:14.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:14.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:14.761777+0000) 2024-03-06T14:41:15.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:15.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:15.762295+0000) 2024-03-06T14:41:16.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:16.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:16.762547+0000) 2024-03-06T14:41:17.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:17.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:17.762768+0000) 2024-03-06T14:41:18.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:18.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:18.762983+0000) 2024-03-06T14:41:19.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:19.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:19.763459+0000) 2024-03-06T14:41:20.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:20.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:20.763686+0000) 2024-03-06T14:41:21.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:21.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:21.763867+0000) 2024-03-06T14:41:22.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:22.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:22.764090+0000) 2024-03-06T14:41:23.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:23.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:23.764294+0000) 2024-03-06T14:41:24.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:24.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:24.764544+0000) 2024-03-06T14:41:25.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:25.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:25.764737+0000) 2024-03-06T14:41:26.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:26.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:26.764963+0000) 2024-03-06T14:41:27.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:27.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:27.765198+0000) 2024-03-06T14:41:28.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:28.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:28.765429+0000) 2024-03-06T14:41:29.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:29.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:29.765651+0000) 2024-03-06T14:41:30.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:30.536 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:41:30.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:30.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:30.765886+0000) 2024-03-06T14:41:31.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:31.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:31.766110+0000) 2024-03-06T14:41:32.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:32.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:32.766337+0000) 2024-03-06T14:41:33.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:33.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:33.766558+0000) 2024-03-06T14:41:34.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:34.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:34.766791+0000) 2024-03-06T14:41:35.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:35.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:35.767042+0000) 2024-03-06T14:41:36.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:36.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:36.767277+0000) 2024-03-06T14:41:37.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:37.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:37.767519+0000) 2024-03-06T14:41:38.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:38.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:38.767748+0000) 2024-03-06T14:41:39.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:39.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:39.767969+0000) 2024-03-06T14:41:40.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:40.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:40.768264+0000) 2024-03-06T14:41:41.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:41.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:41.768488+0000) 2024-03-06T14:41:42.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:42.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:42.768679+0000) 2024-03-06T14:41:43.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:43.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:43.768918+0000) 2024-03-06T14:41:44.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:44.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:44.769146+0000) 2024-03-06T14:41:45.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:45.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:45.769366+0000) 2024-03-06T14:41:46.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:46.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:46.769623+0000) 2024-03-06T14:41:47.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:47.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:47.769807+0000) 2024-03-06T14:41:48.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:48.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:48.770063+0000) 2024-03-06T14:41:49.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:49.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:49.770285+0000) 2024-03-06T14:41:50.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:50.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:50.770510+0000) 2024-03-06T14:41:51.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:51.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:51.770746+0000) 2024-03-06T14:41:52.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:52.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:52.770970+0000) 2024-03-06T14:41:53.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:53.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:53.771199+0000) 2024-03-06T14:41:54.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:54.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:54.771441+0000) 2024-03-06T14:41:55.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:55.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:55.771664+0000) 2024-03-06T14:41:56.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:56.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:56.771904+0000) 2024-03-06T14:41:57.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:57.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:57.772131+0000) 2024-03-06T14:41:58.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:58.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:58.772368+0000) 2024-03-06T14:41:59.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:41:59.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:41:59.772562+0000) 2024-03-06T14:42:00.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:00.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:00.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:00.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:00.772723+0000) 2024-03-06T14:42:01.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:01.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:01.772934+0000) 2024-03-06T14:42:02.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:02.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:02.773214+0000) 2024-03-06T14:42:03.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:03.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:03.773431+0000) 2024-03-06T14:42:04.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:04.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:04.773656+0000) 2024-03-06T14:42:05.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:05.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:05.773910+0000) 2024-03-06T14:42:06.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:06.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:06.774173+0000) 2024-03-06T14:42:07.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:07.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:07.774410+0000) 2024-03-06T14:42:08.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:08.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:08.774672+0000) 2024-03-06T14:42:09.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:09.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:09.774919+0000) 2024-03-06T14:42:10.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:10.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:10.775178+0000) 2024-03-06T14:42:11.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:11.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:11.775442+0000) 2024-03-06T14:42:12.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:12.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:12.775648+0000) 2024-03-06T14:42:13.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:13.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:13.775868+0000) 2024-03-06T14:42:14.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:14.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:14.776099+0000) 2024-03-06T14:42:15.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:15.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:15.776332+0000) 2024-03-06T14:42:16.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:16.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:16.776576+0000) 2024-03-06T14:42:17.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:17.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:17.776797+0000) 2024-03-06T14:42:18.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:18.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:18.777049+0000) 2024-03-06T14:42:19.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:19.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:19.777272+0000) 2024-03-06T14:42:20.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:20.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:20.777472+0000) 2024-03-06T14:42:21.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:21.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:21.777706+0000) 2024-03-06T14:42:22.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:22.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:22.778001+0000) 2024-03-06T14:42:23.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:23.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:23.778226+0000) 2024-03-06T14:42:24.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:24.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:24.778427+0000) 2024-03-06T14:42:25.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:25.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:25.778629+0000) 2024-03-06T14:42:26.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:26.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:26.778855+0000) 2024-03-06T14:42:27.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:27.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:27.779112+0000) 2024-03-06T14:42:28.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:28.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:28.779310+0000) 2024-03-06T14:42:29.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:29.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:29.779541+0000) 2024-03-06T14:42:30.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:30.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:30.779797+0000) 2024-03-06T14:42:30.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:30.995 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:42:31.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:31.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:31.779996+0000) 2024-03-06T14:42:32.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:32.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:32.780246+0000) 2024-03-06T14:42:33.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:33.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:33.780432+0000) 2024-03-06T14:42:34.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:34.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:34.780620+0000) 2024-03-06T14:42:35.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:35.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:35.780837+0000) 2024-03-06T14:42:36.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:36.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:36.781098+0000) 2024-03-06T14:42:37.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:37.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:37.781348+0000) 2024-03-06T14:42:38.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:38.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:38.781595+0000) 2024-03-06T14:42:39.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:39.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:39.781823+0000) 2024-03-06T14:42:40.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:40.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:40.782053+0000) 2024-03-06T14:42:41.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:41.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:41.782281+0000) 2024-03-06T14:42:42.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:42.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:42.782521+0000) 2024-03-06T14:42:43.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:43.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:43.782724+0000) 2024-03-06T14:42:44.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:44.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:44.782967+0000) 2024-03-06T14:42:45.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:45.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:45.783163+0000) 2024-03-06T14:42:46.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:46.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:46.783357+0000) 2024-03-06T14:42:47.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:47.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:47.783559+0000) 2024-03-06T14:42:48.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:48.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:48.783777+0000) 2024-03-06T14:42:49.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:49.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:49.783991+0000) 2024-03-06T14:42:50.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:50.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:50.784244+0000) 2024-03-06T14:42:51.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:51.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:51.784454+0000) 2024-03-06T14:42:52.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:52.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:52.784663+0000) 2024-03-06T14:42:53.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:53.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:53.784880+0000) 2024-03-06T14:42:54.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:54.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:54.785111+0000) 2024-03-06T14:42:55.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:55.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:55.785319+0000) 2024-03-06T14:42:56.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:56.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:56.785570+0000) 2024-03-06T14:42:57.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:57.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:57.785793+0000) 2024-03-06T14:42:58.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:58.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:58.786049+0000) 2024-03-06T14:42:59.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:42:59.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:42:59.786294+0000) 2024-03-06T14:43:00.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:00.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:00.786580+0000) 2024-03-06T14:43:01.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:01.150 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:01.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:01.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:01.786853+0000) 2024-03-06T14:43:02.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:02.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:02.787148+0000) 2024-03-06T14:43:03.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:03.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:03.787391+0000) 2024-03-06T14:43:04.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:04.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:04.787620+0000) 2024-03-06T14:43:05.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:05.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:05.787844+0000) 2024-03-06T14:43:06.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:06.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:06.788056+0000) 2024-03-06T14:43:07.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:07.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:07.788285+0000) 2024-03-06T14:43:08.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:08.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:08.788506+0000) 2024-03-06T14:43:09.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:09.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:09.788752+0000) 2024-03-06T14:43:10.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:10.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:10.788983+0000) 2024-03-06T14:43:11.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:11.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:11.789221+0000) 2024-03-06T14:43:12.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:12.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:12.789440+0000) 2024-03-06T14:43:13.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:13.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:13.789617+0000) 2024-03-06T14:43:14.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:14.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:14.789860+0000) 2024-03-06T14:43:15.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:15.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:15.790048+0000) 2024-03-06T14:43:16.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:16.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:16.790249+0000) 2024-03-06T14:43:17.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:17.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:17.790484+0000) 2024-03-06T14:43:18.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:18.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:18.790706+0000) 2024-03-06T14:43:19.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:19.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:19.790921+0000) 2024-03-06T14:43:20.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:20.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:20.791151+0000) 2024-03-06T14:43:21.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:21.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:21.791336+0000) 2024-03-06T14:43:22.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:22.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:22.791546+0000) 2024-03-06T14:43:23.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:23.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:23.791743+0000) 2024-03-06T14:43:24.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:24.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:24.791980+0000) 2024-03-06T14:43:25.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:25.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:25.792174+0000) 2024-03-06T14:43:26.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:26.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:26.792359+0000) 2024-03-06T14:43:27.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:27.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:27.792542+0000) 2024-03-06T14:43:28.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:28.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:28.792764+0000) 2024-03-06T14:43:29.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:29.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:29.792983+0000) 2024-03-06T14:43:30.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:30.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:30.793255+0000) 2024-03-06T14:43:31.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:31.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:43:31.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:31.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:31.793491+0000) 2024-03-06T14:43:32.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:32.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:32.793764+0000) 2024-03-06T14:43:33.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:33.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:33.794085+0000) 2024-03-06T14:43:34.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:34.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:34.794331+0000) 2024-03-06T14:43:35.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:35.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:35.794603+0000) 2024-03-06T14:43:36.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:36.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:36.794849+0000) 2024-03-06T14:43:37.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:37.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:37.795140+0000) 2024-03-06T14:43:38.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:38.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:38.795388+0000) 2024-03-06T14:43:39.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:39.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:39.795673+0000) 2024-03-06T14:43:40.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:40.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:40.795938+0000) 2024-03-06T14:43:41.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:41.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:41.796165+0000) 2024-03-06T14:43:42.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:42.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:42.796390+0000) 2024-03-06T14:43:43.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:43.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:43.796602+0000) 2024-03-06T14:43:44.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:44.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:44.796829+0000) 2024-03-06T14:43:45.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:45.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:45.797095+0000) 2024-03-06T14:43:46.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:46.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:46.797293+0000) 2024-03-06T14:43:47.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:47.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:47.797510+0000) 2024-03-06T14:43:48.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:48.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:48.797731+0000) 2024-03-06T14:43:49.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:49.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:49.798033+0000) 2024-03-06T14:43:50.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:50.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:50.798281+0000) 2024-03-06T14:43:51.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:51.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:51.798512+0000) 2024-03-06T14:43:52.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:52.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:52.798729+0000) 2024-03-06T14:43:53.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:53.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:53.798942+0000) 2024-03-06T14:43:54.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:54.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:54.799238+0000) 2024-03-06T14:43:55.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:55.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:55.799518+0000) 2024-03-06T14:43:56.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:56.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:56.799795+0000) 2024-03-06T14:43:57.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:57.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:57.800104+0000) 2024-03-06T14:43:58.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:58.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:58.800383+0000) 2024-03-06T14:43:59.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:43:59.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:43:59.800661+0000) 2024-03-06T14:44:00.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:00.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:00.800878+0000) 2024-03-06T14:44:01.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:01.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:01.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:01.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:01.801110+0000) 2024-03-06T14:44:02.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:02.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:02.801331+0000) 2024-03-06T14:44:03.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:03.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:03.801605+0000) 2024-03-06T14:44:04.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:04.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:04.801868+0000) 2024-03-06T14:44:05.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:05.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:05.802113+0000) 2024-03-06T14:44:06.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:06.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:06.802305+0000) 2024-03-06T14:44:07.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:07.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:07.802503+0000) 2024-03-06T14:44:08.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:08.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:08.802709+0000) 2024-03-06T14:44:09.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:09.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:09.802923+0000) 2024-03-06T14:44:10.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:10.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:10.803174+0000) 2024-03-06T14:44:11.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:11.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:11.803392+0000) 2024-03-06T14:44:12.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:12.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:12.803651+0000) 2024-03-06T14:44:13.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:13.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:13.803872+0000) 2024-03-06T14:44:14.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:14.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:14.804164+0000) 2024-03-06T14:44:15.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:15.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:15.804397+0000) 2024-03-06T14:44:16.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:16.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:16.804630+0000) 2024-03-06T14:44:17.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:17.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:17.804922+0000) 2024-03-06T14:44:18.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:18.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:18.805199+0000) 2024-03-06T14:44:19.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:19.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:19.805485+0000) 2024-03-06T14:44:20.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:20.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:20.805735+0000) 2024-03-06T14:44:21.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:21.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:21.805986+0000) 2024-03-06T14:44:22.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:22.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:22.806295+0000) 2024-03-06T14:44:23.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:23.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:23.806572+0000) 2024-03-06T14:44:24.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:24.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:24.806861+0000) 2024-03-06T14:44:25.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:25.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:25.807085+0000) 2024-03-06T14:44:26.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:26.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:26.807291+0000) 2024-03-06T14:44:27.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:27.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:27.807496+0000) 2024-03-06T14:44:28.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:28.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:28.807716+0000) 2024-03-06T14:44:29.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:29.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:29.807939+0000) 2024-03-06T14:44:30.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:30.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:30.808209+0000) 2024-03-06T14:44:31.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:31.696 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:44:31.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:31.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:31.808461+0000) 2024-03-06T14:44:32.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:32.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:32.808688+0000) 2024-03-06T14:44:33.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:33.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:33.808891+0000) 2024-03-06T14:44:34.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:34.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:34.809150+0000) 2024-03-06T14:44:35.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:35.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:35.809368+0000) 2024-03-06T14:44:36.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:36.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:36.809584+0000) 2024-03-06T14:44:37.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:37.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:37.809813+0000) 2024-03-06T14:44:38.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:38.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:38.810076+0000) 2024-03-06T14:44:39.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:39.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:39.810296+0000) 2024-03-06T14:44:40.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:40.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:40.810510+0000) 2024-03-06T14:44:41.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:41.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:41.810761+0000) 2024-03-06T14:44:42.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:42.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:42.811086+0000) 2024-03-06T14:44:43.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:43.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:43.811308+0000) 2024-03-06T14:44:44.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:44.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:44.811544+0000) 2024-03-06T14:44:45.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:45.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:45.811770+0000) 2024-03-06T14:44:46.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:46.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:46.812001+0000) 2024-03-06T14:44:47.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:47.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:47.812255+0000) 2024-03-06T14:44:48.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:48.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:48.812498+0000) 2024-03-06T14:44:49.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:49.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:49.812765+0000) 2024-03-06T14:44:50.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:50.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:50.813034+0000) 2024-03-06T14:44:51.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:51.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:51.813261+0000) 2024-03-06T14:44:52.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:52.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:52.813443+0000) 2024-03-06T14:44:53.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:53.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:53.813689+0000) 2024-03-06T14:44:54.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:54.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:54.813980+0000) 2024-03-06T14:44:55.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:55.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:55.814417+0000) 2024-03-06T14:44:56.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:56.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:56.814670+0000) 2024-03-06T14:44:57.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:57.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:57.814904+0000) 2024-03-06T14:44:58.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:58.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:58.815140+0000) 2024-03-06T14:44:59.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:44:59.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:44:59.815325+0000) 2024-03-06T14:45:00.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:00.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:00.815534+0000) 2024-03-06T14:45:01.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:01.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:01.815757+0000) 2024-03-06T14:45:01.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:01.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:02.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:02.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:02.815990+0000) 2024-03-06T14:45:03.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:03.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:03.816267+0000) 2024-03-06T14:45:04.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:04.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:04.816470+0000) 2024-03-06T14:45:05.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:05.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:05.816688+0000) 2024-03-06T14:45:06.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:06.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:06.816915+0000) 2024-03-06T14:45:07.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:07.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:07.817137+0000) 2024-03-06T14:45:08.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:08.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:08.817338+0000) 2024-03-06T14:45:09.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:09.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:09.817575+0000) 2024-03-06T14:45:10.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:10.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:10.817795+0000) 2024-03-06T14:45:11.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:11.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:11.818046+0000) 2024-03-06T14:45:12.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:12.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:12.818270+0000) 2024-03-06T14:45:13.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:13.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:13.818492+0000) 2024-03-06T14:45:14.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:14.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:14.818726+0000) 2024-03-06T14:45:15.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:15.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:15.818948+0000) 2024-03-06T14:45:16.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:16.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:16.819158+0000) 2024-03-06T14:45:17.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:17.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:17.819401+0000) 2024-03-06T14:45:18.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:18.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:18.819616+0000) 2024-03-06T14:45:19.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:19.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:19.819802+0000) 2024-03-06T14:45:20.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:20.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:20.820063+0000) 2024-03-06T14:45:21.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:21.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:21.820252+0000) 2024-03-06T14:45:22.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:22.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:22.820422+0000) 2024-03-06T14:45:23.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:23.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:23.820638+0000) 2024-03-06T14:45:24.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:24.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:24.820876+0000) 2024-03-06T14:45:25.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:25.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:25.821178+0000) 2024-03-06T14:45:26.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:26.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:26.821408+0000) 2024-03-06T14:45:27.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:27.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:27.821631+0000) 2024-03-06T14:45:28.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:28.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:28.821833+0000) 2024-03-06T14:45:29.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:29.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:29.822084+0000) 2024-03-06T14:45:30.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:30.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:30.822309+0000) 2024-03-06T14:45:31.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:31.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:31.822508+0000) 2024-03-06T14:45:32.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:32.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:45:32.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:32.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:32.822735+0000) 2024-03-06T14:45:33.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:33.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:33.822953+0000) 2024-03-06T14:45:34.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:34.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:34.823201+0000) 2024-03-06T14:45:35.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:35.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:35.823448+0000) 2024-03-06T14:45:36.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:36.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:36.823680+0000) 2024-03-06T14:45:37.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:37.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:37.823900+0000) 2024-03-06T14:45:38.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:38.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:38.824115+0000) 2024-03-06T14:45:39.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:39.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:39.824303+0000) 2024-03-06T14:45:40.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:40.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:40.824485+0000) 2024-03-06T14:45:41.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:41.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:41.824683+0000) 2024-03-06T14:45:42.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:42.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:42.824916+0000) 2024-03-06T14:45:43.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:43.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:43.825145+0000) 2024-03-06T14:45:44.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:44.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:44.825310+0000) 2024-03-06T14:45:45.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:45.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:45.825522+0000) 2024-03-06T14:45:46.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:46.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:46.825711+0000) 2024-03-06T14:45:47.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:47.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:47.825929+0000) 2024-03-06T14:45:48.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:48.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:48.826168+0000) 2024-03-06T14:45:49.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:49.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:49.826389+0000) 2024-03-06T14:45:50.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:50.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:50.826608+0000) 2024-03-06T14:45:51.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:51.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:51.826831+0000) 2024-03-06T14:45:52.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:52.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:52.827094+0000) 2024-03-06T14:45:53.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:53.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:53.827314+0000) 2024-03-06T14:45:54.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:54.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:54.827540+0000) 2024-03-06T14:45:55.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:55.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:55.827751+0000) 2024-03-06T14:45:56.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:56.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:56.827965+0000) 2024-03-06T14:45:57.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:57.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:57.828154+0000) 2024-03-06T14:45:58.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:58.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:58.828340+0000) 2024-03-06T14:45:59.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:45:59.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:45:59.828520+0000) 2024-03-06T14:46:00.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:00.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:00.828744+0000) 2024-03-06T14:46:01.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:01.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:01.828921+0000) 2024-03-06T14:46:02.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:02.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:02.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:02.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:02.829134+0000) 2024-03-06T14:46:03.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:03.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:03.829346+0000) 2024-03-06T14:46:04.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:04.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:04.829565+0000) 2024-03-06T14:46:05.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:05.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:05.829786+0000) 2024-03-06T14:46:06.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:06.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:06.830049+0000) 2024-03-06T14:46:07.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:07.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:07.830236+0000) 2024-03-06T14:46:08.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:08.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:08.830425+0000) 2024-03-06T14:46:09.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:09.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:09.830611+0000) 2024-03-06T14:46:10.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:10.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:10.830843+0000) 2024-03-06T14:46:11.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:11.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:11.831093+0000) 2024-03-06T14:46:12.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:12.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:12.831286+0000) 2024-03-06T14:46:13.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:13.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:13.831577+0000) 2024-03-06T14:46:14.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:14.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:14.831820+0000) 2024-03-06T14:46:15.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:15.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:15.832086+0000) 2024-03-06T14:46:16.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:16.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:16.832282+0000) 2024-03-06T14:46:17.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:17.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:17.832489+0000) 2024-03-06T14:46:18.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:18.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:18.832692+0000) 2024-03-06T14:46:19.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:19.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:19.832925+0000) 2024-03-06T14:46:20.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:20.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:20.833168+0000) 2024-03-06T14:46:21.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:21.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:21.833420+0000) 2024-03-06T14:46:22.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:22.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:22.833652+0000) 2024-03-06T14:46:23.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:23.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:23.833873+0000) 2024-03-06T14:46:24.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:24.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:24.834109+0000) 2024-03-06T14:46:25.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:25.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:25.834329+0000) 2024-03-06T14:46:26.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:26.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:26.834521+0000) 2024-03-06T14:46:27.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:27.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:27.834770+0000) 2024-03-06T14:46:28.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:28.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:28.835066+0000) 2024-03-06T14:46:29.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:29.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:29.835267+0000) 2024-03-06T14:46:30.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:30.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:30.835484+0000) 2024-03-06T14:46:31.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:31.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:31.835774+0000) 2024-03-06T14:46:32.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:32.663 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:46:32.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:32.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:32.836097+0000) 2024-03-06T14:46:33.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:33.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:33.836303+0000) 2024-03-06T14:46:34.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:34.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:34.836546+0000) 2024-03-06T14:46:35.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:35.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:35.836780+0000) 2024-03-06T14:46:36.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:36.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:36.837103+0000) 2024-03-06T14:46:37.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:37.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:37.837316+0000) 2024-03-06T14:46:38.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:38.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:38.837528+0000) 2024-03-06T14:46:39.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:39.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:39.837744+0000) 2024-03-06T14:46:40.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:40.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:40.837996+0000) 2024-03-06T14:46:41.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:41.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:41.838266+0000) 2024-03-06T14:46:42.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:42.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:42.838473+0000) 2024-03-06T14:46:43.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:43.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:43.838692+0000) 2024-03-06T14:46:44.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:44.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:44.838959+0000) 2024-03-06T14:46:45.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:45.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:45.839189+0000) 2024-03-06T14:46:46.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:46.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:46.839424+0000) 2024-03-06T14:46:47.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:47.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:47.839689+0000) 2024-03-06T14:46:48.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:48.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:48.839972+0000) 2024-03-06T14:46:49.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:49.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:49.840256+0000) 2024-03-06T14:46:50.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:50.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:50.840488+0000) 2024-03-06T14:46:51.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:51.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:51.840762+0000) 2024-03-06T14:46:52.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:52.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:52.841080+0000) 2024-03-06T14:46:53.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:53.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:53.841360+0000) 2024-03-06T14:46:54.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:54.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:54.841639+0000) 2024-03-06T14:46:55.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:55.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:55.841871+0000) 2024-03-06T14:46:56.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:56.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:56.842144+0000) 2024-03-06T14:46:57.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:57.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:57.842328+0000) 2024-03-06T14:46:58.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:58.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:58.842550+0000) 2024-03-06T14:46:59.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:46:59.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:46:59.842773+0000) 2024-03-06T14:47:00.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:00.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:00.842999+0000) 2024-03-06T14:47:01.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:01.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:01.843226+0000) 2024-03-06T14:47:02.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:02.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:02.843415+0000) 2024-03-06T14:47:02.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:02.858 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:03.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:03.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:03.843585+0000) 2024-03-06T14:47:04.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:04.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:04.843810+0000) 2024-03-06T14:47:05.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:05.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:05.844105+0000) 2024-03-06T14:47:06.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:06.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:06.844393+0000) 2024-03-06T14:47:07.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:07.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:07.844659+0000) 2024-03-06T14:47:08.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:08.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:08.844854+0000) 2024-03-06T14:47:09.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:09.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:09.845127+0000) 2024-03-06T14:47:10.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:10.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:10.845347+0000) 2024-03-06T14:47:11.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:11.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:11.845611+0000) 2024-03-06T14:47:12.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:12.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:12.845843+0000) 2024-03-06T14:47:13.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:13.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:13.846091+0000) 2024-03-06T14:47:14.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:14.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:14.846363+0000) 2024-03-06T14:47:15.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:15.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:15.846629+0000) 2024-03-06T14:47:16.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:16.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:16.846851+0000) 2024-03-06T14:47:17.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:17.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:17.847127+0000) 2024-03-06T14:47:18.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:18.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:18.847368+0000) 2024-03-06T14:47:19.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:19.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:19.847608+0000) 2024-03-06T14:47:20.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:20.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:20.847816+0000) 2024-03-06T14:47:21.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:21.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:21.848060+0000) 2024-03-06T14:47:22.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:22.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:22.848254+0000) 2024-03-06T14:47:23.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:23.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:23.848491+0000) 2024-03-06T14:47:24.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:24.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:24.848738+0000) 2024-03-06T14:47:25.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:25.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:25.848974+0000) 2024-03-06T14:47:26.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:26.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:26.849277+0000) 2024-03-06T14:47:27.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:27.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:27.849519+0000) 2024-03-06T14:47:28.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:28.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:28.849768+0000) 2024-03-06T14:47:29.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:29.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:29.850039+0000) 2024-03-06T14:47:30.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:30.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:30.850279+0000) 2024-03-06T14:47:31.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:31.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:31.850520+0000) 2024-03-06T14:47:32.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:32.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:32.850765+0000) 2024-03-06T14:47:33.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:33.020 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:47:33.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:33.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:33.851040+0000) 2024-03-06T14:47:34.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:34.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:34.851303+0000) 2024-03-06T14:47:35.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:35.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:35.851543+0000) 2024-03-06T14:47:36.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:36.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:36.851792+0000) 2024-03-06T14:47:37.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:37.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:37.852073+0000) 2024-03-06T14:47:38.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:38.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:38.852344+0000) 2024-03-06T14:47:39.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:39.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:39.852558+0000) 2024-03-06T14:47:40.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:40.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:40.852785+0000) 2024-03-06T14:47:41.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:41.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:41.853081+0000) 2024-03-06T14:47:42.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:42.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:42.853275+0000) 2024-03-06T14:47:43.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:43.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:43.853508+0000) 2024-03-06T14:47:44.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:44.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:44.853742+0000) 2024-03-06T14:47:45.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:45.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:45.853966+0000) 2024-03-06T14:47:46.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:46.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:46.854204+0000) 2024-03-06T14:47:47.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:47.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:47.854408+0000) 2024-03-06T14:47:48.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:48.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:48.854632+0000) 2024-03-06T14:47:49.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:49.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:49.854857+0000) 2024-03-06T14:47:50.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:50.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:50.855113+0000) 2024-03-06T14:47:51.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:51.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:51.855351+0000) 2024-03-06T14:47:52.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:52.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:52.855578+0000) 2024-03-06T14:47:53.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:53.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:53.855804+0000) 2024-03-06T14:47:54.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:54.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:54.856057+0000) 2024-03-06T14:47:55.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:55.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:55.856284+0000) 2024-03-06T14:47:56.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:56.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:56.856586+0000) 2024-03-06T14:47:57.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:57.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:57.856821+0000) 2024-03-06T14:47:58.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:58.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:58.857065+0000) 2024-03-06T14:47:59.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:47:59.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:47:59.857312+0000) 2024-03-06T14:48:00.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:00.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:00.857591+0000) 2024-03-06T14:48:01.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:01.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:01.857760+0000) 2024-03-06T14:48:02.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:02.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:02.857985+0000) 2024-03-06T14:48:03.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:03.241 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:03.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:03.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:03.858235+0000) 2024-03-06T14:48:04.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:04.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:04.858447+0000) 2024-03-06T14:48:05.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:05.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:05.858659+0000) 2024-03-06T14:48:06.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:06.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:06.858891+0000) 2024-03-06T14:48:07.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:07.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:07.859126+0000) 2024-03-06T14:48:08.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:08.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:08.859316+0000) 2024-03-06T14:48:09.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:09.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:09.859548+0000) 2024-03-06T14:48:10.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:10.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:10.859788+0000) 2024-03-06T14:48:11.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:11.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:11.859985+0000) 2024-03-06T14:48:12.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:12.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:12.860257+0000) 2024-03-06T14:48:13.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:13.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:13.860480+0000) 2024-03-06T14:48:14.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:14.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:14.860741+0000) 2024-03-06T14:48:15.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:15.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:15.860979+0000) 2024-03-06T14:48:16.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:16.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:16.861274+0000) 2024-03-06T14:48:17.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:17.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:17.861511+0000) 2024-03-06T14:48:18.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:18.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:18.861758+0000) 2024-03-06T14:48:19.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:19.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:19.861994+0000) 2024-03-06T14:48:20.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:20.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:20.862289+0000) 2024-03-06T14:48:21.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:21.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:21.862533+0000) 2024-03-06T14:48:22.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:22.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:22.862781+0000) 2024-03-06T14:48:23.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:23.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:23.863047+0000) 2024-03-06T14:48:24.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:24.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:24.863280+0000) 2024-03-06T14:48:25.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:25.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:25.863501+0000) 2024-03-06T14:48:26.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:26.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:26.863752+0000) 2024-03-06T14:48:27.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:27.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:27.863967+0000) 2024-03-06T14:48:28.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:28.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:28.864235+0000) 2024-03-06T14:48:29.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:29.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:29.864462+0000) 2024-03-06T14:48:30.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:30.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:30.864708+0000) 2024-03-06T14:48:31.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:31.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:31.864939+0000) 2024-03-06T14:48:32.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:32.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:32.865171+0000) 2024-03-06T14:48:33.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:33.402 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:48:33.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:33.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:33.865416+0000) 2024-03-06T14:48:34.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:34.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:34.865653+0000) 2024-03-06T14:48:35.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:35.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:35.865882+0000) 2024-03-06T14:48:36.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:36.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:36.866120+0000) 2024-03-06T14:48:37.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:37.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:37.866342+0000) 2024-03-06T14:48:38.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:38.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:38.866534+0000) 2024-03-06T14:48:39.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:39.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:39.866754+0000) 2024-03-06T14:48:40.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:40.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:40.866983+0000) 2024-03-06T14:48:41.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:41.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:41.867275+0000) 2024-03-06T14:48:42.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:42.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:42.867531+0000) 2024-03-06T14:48:43.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:43.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:43.867781+0000) 2024-03-06T14:48:44.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:44.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:44.868078+0000) 2024-03-06T14:48:45.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:45.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:45.868302+0000) 2024-03-06T14:48:46.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:46.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:46.868497+0000) 2024-03-06T14:48:47.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:47.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:47.868711+0000) 2024-03-06T14:48:48.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:48.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:48.869041+0000) 2024-03-06T14:48:49.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:49.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:49.869248+0000) 2024-03-06T14:48:50.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:50.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:50.869461+0000) 2024-03-06T14:48:51.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:51.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:51.869684+0000) 2024-03-06T14:48:52.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:52.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:52.869928+0000) 2024-03-06T14:48:53.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:53.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:53.870166+0000) 2024-03-06T14:48:54.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:54.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:54.870422+0000) 2024-03-06T14:48:55.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:55.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:55.870623+0000) 2024-03-06T14:48:56.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:56.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:56.870852+0000) 2024-03-06T14:48:57.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:57.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:57.871096+0000) 2024-03-06T14:48:58.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:58.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:58.871292+0000) 2024-03-06T14:48:59.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:48:59.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:48:59.871492+0000) 2024-03-06T14:49:00.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:00.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:00.871708+0000) 2024-03-06T14:49:01.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:01.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:01.871932+0000) 2024-03-06T14:49:02.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:02.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:02.872133+0000) 2024-03-06T14:49:03.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:03.618 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:03.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:03.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:03.872310+0000) 2024-03-06T14:49:04.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:04.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:04.872500+0000) 2024-03-06T14:49:05.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:05.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:05.872682+0000) 2024-03-06T14:49:06.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:06.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:06.872919+0000) 2024-03-06T14:49:07.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:07.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:07.873146+0000) 2024-03-06T14:49:08.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:08.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:08.873319+0000) 2024-03-06T14:49:09.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:09.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:09.873530+0000) 2024-03-06T14:49:10.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:10.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:10.873707+0000) 2024-03-06T14:49:11.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:11.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:11.873932+0000) 2024-03-06T14:49:12.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:12.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:12.874164+0000) 2024-03-06T14:49:13.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:13.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:13.874381+0000) 2024-03-06T14:49:14.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:14.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:14.874571+0000) 2024-03-06T14:49:15.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:15.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:15.874787+0000) 2024-03-06T14:49:16.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:16.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:16.875074+0000) 2024-03-06T14:49:17.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:17.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:17.875286+0000) 2024-03-06T14:49:18.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:18.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:18.875506+0000) 2024-03-06T14:49:19.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:19.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:19.875715+0000) 2024-03-06T14:49:20.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:20.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:20.875905+0000) 2024-03-06T14:49:21.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:21.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:21.876131+0000) 2024-03-06T14:49:22.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:22.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:22.876326+0000) 2024-03-06T14:49:23.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:23.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:23.876539+0000) 2024-03-06T14:49:24.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:24.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:24.876768+0000) 2024-03-06T14:49:25.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:25.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:25.876966+0000) 2024-03-06T14:49:26.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:26.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:26.877197+0000) 2024-03-06T14:49:27.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:27.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:27.877433+0000) 2024-03-06T14:49:28.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:28.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:28.877700+0000) 2024-03-06T14:49:29.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:29.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:29.877894+0000) 2024-03-06T14:49:30.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:30.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:30.878170+0000) 2024-03-06T14:49:31.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:31.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:31.878406+0000) 2024-03-06T14:49:32.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:32.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:32.878628+0000) 2024-03-06T14:49:33.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:33.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:49:33.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:33.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:33.878814+0000) 2024-03-06T14:49:34.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:34.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:34.879079+0000) 2024-03-06T14:49:35.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:35.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:35.879299+0000) 2024-03-06T14:49:36.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:36.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:36.879518+0000) 2024-03-06T14:49:37.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:37.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:37.879739+0000) 2024-03-06T14:49:38.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:38.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:38.879967+0000) 2024-03-06T14:49:39.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:39.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:39.880189+0000) 2024-03-06T14:49:40.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:40.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:40.880378+0000) 2024-03-06T14:49:41.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:41.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:41.880604+0000) 2024-03-06T14:49:42.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:42.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:42.880826+0000) 2024-03-06T14:49:43.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:43.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:43.881081+0000) 2024-03-06T14:49:44.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:44.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:44.881321+0000) 2024-03-06T14:49:45.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:45.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:45.881541+0000) 2024-03-06T14:49:46.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:46.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:46.881779+0000) 2024-03-06T14:49:47.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:47.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:47.881999+0000) 2024-03-06T14:49:48.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:48.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:48.882246+0000) 2024-03-06T14:49:49.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:49.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:49.882446+0000) 2024-03-06T14:49:50.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:50.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:50.882653+0000) 2024-03-06T14:49:51.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:51.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:51.882890+0000) 2024-03-06T14:49:52.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:52.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:52.883133+0000) 2024-03-06T14:49:53.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:53.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:53.883322+0000) 2024-03-06T14:49:54.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:54.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:54.883533+0000) 2024-03-06T14:49:55.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:55.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:55.883756+0000) 2024-03-06T14:49:56.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:56.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:56.883969+0000) 2024-03-06T14:49:57.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:57.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:57.884155+0000) 2024-03-06T14:49:58.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:58.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:58.884398+0000) 2024-03-06T14:49:59.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:49:59.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:49:59.884634+0000) 2024-03-06T14:50:00.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:00.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:00.884882+0000) 2024-03-06T14:50:01.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:01.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:01.885170+0000) 2024-03-06T14:50:02.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:02.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:02.885401+0000) 2024-03-06T14:50:03.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:03.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:03.885621+0000) 2024-03-06T14:50:04.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:04.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:04.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:04.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:04.885861+0000) 2024-03-06T14:50:05.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:05.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:05.886137+0000) 2024-03-06T14:50:06.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:06.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:06.886377+0000) 2024-03-06T14:50:07.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:07.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:07.886599+0000) 2024-03-06T14:50:08.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:08.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:08.886843+0000) 2024-03-06T14:50:09.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:09.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:09.887118+0000) 2024-03-06T14:50:10.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:10.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:10.887342+0000) 2024-03-06T14:50:11.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:11.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:11.887580+0000) 2024-03-06T14:50:12.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:12.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:12.887810+0000) 2024-03-06T14:50:13.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:13.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:13.888069+0000) 2024-03-06T14:50:14.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:14.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:14.888273+0000) 2024-03-06T14:50:15.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:15.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:15.888444+0000) 2024-03-06T14:50:16.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:16.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:16.888657+0000) 2024-03-06T14:50:17.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:17.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:17.888875+0000) 2024-03-06T14:50:18.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:18.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:18.889113+0000) 2024-03-06T14:50:19.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:19.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:19.889303+0000) 2024-03-06T14:50:20.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:20.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:20.889509+0000) 2024-03-06T14:50:21.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:21.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:21.889732+0000) 2024-03-06T14:50:22.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:22.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:22.889953+0000) 2024-03-06T14:50:23.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:23.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:23.890120+0000) 2024-03-06T14:50:24.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:24.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:24.890369+0000) 2024-03-06T14:50:25.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:25.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:25.890589+0000) 2024-03-06T14:50:26.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:26.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:26.890822+0000) 2024-03-06T14:50:27.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:27.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:27.891085+0000) 2024-03-06T14:50:28.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:28.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:28.891277+0000) 2024-03-06T14:50:29.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:29.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:29.891497+0000) 2024-03-06T14:50:30.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:30.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:30.891676+0000) 2024-03-06T14:50:31.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:31.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:31.891895+0000) 2024-03-06T14:50:32.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:32.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:32.892128+0000) 2024-03-06T14:50:33.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:33.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:33.892364+0000) 2024-03-06T14:50:34.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:34.304 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:50:34.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:34.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:34.892629+0000) 2024-03-06T14:50:35.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:35.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:35.892854+0000) 2024-03-06T14:50:36.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:36.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:36.893049+0000) 2024-03-06T14:50:37.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:37.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:37.893240+0000) 2024-03-06T14:50:38.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:38.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:38.893457+0000) 2024-03-06T14:50:39.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:39.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:39.893698+0000) 2024-03-06T14:50:40.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:40.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:40.893927+0000) 2024-03-06T14:50:41.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:41.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:41.894149+0000) 2024-03-06T14:50:42.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:42.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:42.894338+0000) 2024-03-06T14:50:43.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:43.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:43.894568+0000) 2024-03-06T14:50:44.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:44.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:44.894802+0000) 2024-03-06T14:50:45.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:45.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:45.895052+0000) 2024-03-06T14:50:46.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:46.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:46.895245+0000) 2024-03-06T14:50:47.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:47.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:47.895452+0000) 2024-03-06T14:50:48.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:48.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:48.895673+0000) 2024-03-06T14:50:49.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:49.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:49.895908+0000) 2024-03-06T14:50:50.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:50.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:50.896149+0000) 2024-03-06T14:50:51.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:51.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:51.896347+0000) 2024-03-06T14:50:52.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:52.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:52.896558+0000) 2024-03-06T14:50:53.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:53.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:53.896775+0000) 2024-03-06T14:50:54.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:54.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:54.897036+0000) 2024-03-06T14:50:55.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:55.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:55.897258+0000) 2024-03-06T14:50:56.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:56.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:56.897484+0000) 2024-03-06T14:50:57.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:57.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:57.897710+0000) 2024-03-06T14:50:58.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:58.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:58.897930+0000) 2024-03-06T14:50:59.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:50:59.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:50:59.898185+0000) 2024-03-06T14:51:00.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:00.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:00.898411+0000) 2024-03-06T14:51:01.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:01.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:01.898641+0000) 2024-03-06T14:51:02.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:02.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:02.898875+0000) 2024-03-06T14:51:03.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:03.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:03.899099+0000) 2024-03-06T14:51:04.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:04.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:04.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:04.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:04.899333+0000) 2024-03-06T14:51:05.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:05.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:05.899534+0000) 2024-03-06T14:51:06.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:06.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:06.899724+0000) 2024-03-06T14:51:07.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:07.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:07.899964+0000) 2024-03-06T14:51:08.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:08.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:08.900168+0000) 2024-03-06T14:51:09.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:09.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:09.900353+0000) 2024-03-06T14:51:10.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:10.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:10.900531+0000) 2024-03-06T14:51:11.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:11.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:11.900752+0000) 2024-03-06T14:51:12.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:12.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:12.900984+0000) 2024-03-06T14:51:13.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:13.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:13.901241+0000) 2024-03-06T14:51:14.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:14.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:14.901492+0000) 2024-03-06T14:51:15.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:15.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:15.901768+0000) 2024-03-06T14:51:16.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:16.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:16.901995+0000) 2024-03-06T14:51:17.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:17.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:17.902327+0000) 2024-03-06T14:51:18.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:18.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:18.902533+0000) 2024-03-06T14:51:19.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:19.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:19.902758+0000) 2024-03-06T14:51:20.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:20.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:20.902954+0000) 2024-03-06T14:51:21.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:21.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:21.903173+0000) 2024-03-06T14:51:22.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:22.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:22.903394+0000) 2024-03-06T14:51:23.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:23.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:23.903580+0000) 2024-03-06T14:51:24.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:24.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:24.903770+0000) 2024-03-06T14:51:25.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:25.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:25.903990+0000) 2024-03-06T14:51:26.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:26.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:26.904246+0000) 2024-03-06T14:51:27.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:27.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:27.904461+0000) 2024-03-06T14:51:28.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:28.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:28.904705+0000) 2024-03-06T14:51:29.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:29.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:29.904932+0000) 2024-03-06T14:51:30.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:30.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:30.905153+0000) 2024-03-06T14:51:31.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:31.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:31.905403+0000) 2024-03-06T14:51:32.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:32.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:32.905626+0000) 2024-03-06T14:51:33.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:33.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:33.905849+0000) 2024-03-06T14:51:34.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:34.795 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:51:34.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:34.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:34.906120+0000) 2024-03-06T14:51:35.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:35.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:35.906324+0000) 2024-03-06T14:51:36.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:36.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:36.906579+0000) 2024-03-06T14:51:37.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:37.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:37.906819+0000) 2024-03-06T14:51:38.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:38.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:38.907092+0000) 2024-03-06T14:51:39.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:39.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:39.907308+0000) 2024-03-06T14:51:40.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:40.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:40.907493+0000) 2024-03-06T14:51:41.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:41.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:41.907713+0000) 2024-03-06T14:51:42.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:42.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:42.907931+0000) 2024-03-06T14:51:43.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:43.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:43.908151+0000) 2024-03-06T14:51:44.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:44.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:44.908386+0000) 2024-03-06T14:51:45.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:45.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:45.908590+0000) 2024-03-06T14:51:46.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:46.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:46.908814+0000) 2024-03-06T14:51:47.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:47.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:47.909067+0000) 2024-03-06T14:51:48.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:48.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:48.909267+0000) 2024-03-06T14:51:49.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:49.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:49.909475+0000) 2024-03-06T14:51:50.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:50.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:50.909716+0000) 2024-03-06T14:51:51.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:51.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:51.909933+0000) 2024-03-06T14:51:52.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:52.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:52.910151+0000) 2024-03-06T14:51:53.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:53.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:53.910375+0000) 2024-03-06T14:51:54.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:54.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:54.910630+0000) 2024-03-06T14:51:55.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:55.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:55.910819+0000) 2024-03-06T14:51:56.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:56.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:56.911079+0000) 2024-03-06T14:51:57.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:57.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:57.911312+0000) 2024-03-06T14:51:58.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:58.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:58.911521+0000) 2024-03-06T14:51:59.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:51:59.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:51:59.911745+0000) 2024-03-06T14:52:00.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:00.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:00.911969+0000) 2024-03-06T14:52:01.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:01.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:01.912155+0000) 2024-03-06T14:52:02.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:02.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:02.912342+0000) 2024-03-06T14:52:03.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:03.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:03.912560+0000) 2024-03-06T14:52:04.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:04.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:04.912790+0000) 2024-03-06T14:52:04.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:04.945 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:05.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:05.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:05.912993+0000) 2024-03-06T14:52:06.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:06.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:06.913238+0000) 2024-03-06T14:52:07.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:07.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:07.913470+0000) 2024-03-06T14:52:08.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:08.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:08.913712+0000) 2024-03-06T14:52:09.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:09.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:09.913934+0000) 2024-03-06T14:52:10.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:10.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:10.914273+0000) 2024-03-06T14:52:11.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:11.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:11.914496+0000) 2024-03-06T14:52:12.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:12.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:12.914686+0000) 2024-03-06T14:52:13.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:13.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:13.914926+0000) 2024-03-06T14:52:14.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:14.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:14.915172+0000) 2024-03-06T14:52:15.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:15.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:15.915409+0000) 2024-03-06T14:52:16.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:16.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:16.915660+0000) 2024-03-06T14:52:17.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:17.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:17.915908+0000) 2024-03-06T14:52:18.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:18.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:18.916170+0000) 2024-03-06T14:52:19.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:19.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:19.916418+0000) 2024-03-06T14:52:20.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:20.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:20.916660+0000) 2024-03-06T14:52:21.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:21.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:21.916914+0000) 2024-03-06T14:52:22.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:22.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:22.917161+0000) 2024-03-06T14:52:23.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:23.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:23.917378+0000) 2024-03-06T14:52:24.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:24.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:24.917615+0000) 2024-03-06T14:52:25.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:25.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:25.917835+0000) 2024-03-06T14:52:26.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:26.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:26.918112+0000) 2024-03-06T14:52:27.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:27.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:27.918361+0000) 2024-03-06T14:52:28.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:28.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:28.918610+0000) 2024-03-06T14:52:29.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:29.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:29.918855+0000) 2024-03-06T14:52:30.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:30.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:30.919123+0000) 2024-03-06T14:52:31.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:31.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:31.919372+0000) 2024-03-06T14:52:32.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:32.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:32.919612+0000) 2024-03-06T14:52:33.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:33.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:33.919851+0000) 2024-03-06T14:52:34.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:34.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:34.920132+0000) 2024-03-06T14:52:35.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:35.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:52:35.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:35.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:35.920340+0000) 2024-03-06T14:52:36.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:36.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:36.920619+0000) 2024-03-06T14:52:37.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:37.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:37.920886+0000) 2024-03-06T14:52:38.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:38.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:38.921161+0000) 2024-03-06T14:52:39.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:39.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:39.921398+0000) 2024-03-06T14:52:40.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:40.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:40.921682+0000) 2024-03-06T14:52:41.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:41.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:41.921962+0000) 2024-03-06T14:52:42.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:42.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:42.922247+0000) 2024-03-06T14:52:43.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:43.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:43.922477+0000) 2024-03-06T14:52:44.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:44.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:44.922743+0000) 2024-03-06T14:52:45.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:45.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:45.922982+0000) 2024-03-06T14:52:46.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:46.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:46.923281+0000) 2024-03-06T14:52:47.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:47.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:47.923513+0000) 2024-03-06T14:52:48.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:48.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:48.923783+0000) 2024-03-06T14:52:49.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:49.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:49.924089+0000) 2024-03-06T14:52:50.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:50.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:50.924339+0000) 2024-03-06T14:52:51.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:51.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:51.924597+0000) 2024-03-06T14:52:52.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:52.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:52.924851+0000) 2024-03-06T14:52:53.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:53.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:53.925149+0000) 2024-03-06T14:52:54.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:54.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:54.925420+0000) 2024-03-06T14:52:55.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:55.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:55.925690+0000) 2024-03-06T14:52:56.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:56.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:56.925961+0000) 2024-03-06T14:52:57.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:57.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:57.926298+0000) 2024-03-06T14:52:58.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:58.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:58.926624+0000) 2024-03-06T14:52:59.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:52:59.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:52:59.926894+0000) 2024-03-06T14:53:00.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:00.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:00.927127+0000) 2024-03-06T14:53:01.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:01.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:01.927294+0000) 2024-03-06T14:53:02.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:02.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:02.927546+0000) 2024-03-06T14:53:03.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:03.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:03.927777+0000) 2024-03-06T14:53:04.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:04.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:04.928063+0000) 2024-03-06T14:53:05.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:05.366 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:05.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:05.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:05.928305+0000) 2024-03-06T14:53:06.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:06.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:06.928551+0000) 2024-03-06T14:53:07.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:07.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:07.928781+0000) 2024-03-06T14:53:08.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:08.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:08.929074+0000) 2024-03-06T14:53:09.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:09.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:09.929273+0000) 2024-03-06T14:53:10.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:10.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:10.929490+0000) 2024-03-06T14:53:11.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:11.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:11.929707+0000) 2024-03-06T14:53:12.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:12.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:12.929964+0000) 2024-03-06T14:53:13.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:13.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:13.930277+0000) 2024-03-06T14:53:14.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:14.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:14.930503+0000) 2024-03-06T14:53:15.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:15.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:15.930748+0000) 2024-03-06T14:53:16.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:16.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:16.931042+0000) 2024-03-06T14:53:17.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:17.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:17.931297+0000) 2024-03-06T14:53:18.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:18.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:18.931551+0000) 2024-03-06T14:53:19.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:19.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:19.931788+0000) 2024-03-06T14:53:20.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:20.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:20.932000+0000) 2024-03-06T14:53:21.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:21.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:21.932231+0000) 2024-03-06T14:53:22.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:22.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:22.932471+0000) 2024-03-06T14:53:23.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:23.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:23.932697+0000) 2024-03-06T14:53:24.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:24.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:24.932944+0000) 2024-03-06T14:53:25.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:25.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:25.933179+0000) 2024-03-06T14:53:26.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:26.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:26.933418+0000) 2024-03-06T14:53:27.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:27.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:27.933673+0000) 2024-03-06T14:53:28.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:28.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:28.933854+0000) 2024-03-06T14:53:29.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:29.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:29.934100+0000) 2024-03-06T14:53:30.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:30.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:30.934332+0000) 2024-03-06T14:53:31.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:31.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:31.934479+0000) 2024-03-06T14:53:32.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:32.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:32.934704+0000) 2024-03-06T14:53:33.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:33.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:33.934939+0000) 2024-03-06T14:53:34.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:34.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:34.935164+0000) 2024-03-06T14:53:35.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:35.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:53:35.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:35.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:35.935311+0000) 2024-03-06T14:53:36.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:36.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:36.935493+0000) 2024-03-06T14:53:37.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:37.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:37.935713+0000) 2024-03-06T14:53:38.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:38.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:38.935918+0000) 2024-03-06T14:53:39.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:39.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:39.936154+0000) 2024-03-06T14:53:40.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:40.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:40.936405+0000) 2024-03-06T14:53:41.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:41.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:41.936667+0000) 2024-03-06T14:53:42.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:42.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:42.936873+0000) 2024-03-06T14:53:43.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:43.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:43.937127+0000) 2024-03-06T14:53:44.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:44.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:44.937358+0000) 2024-03-06T14:53:45.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:45.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:45.937558+0000) 2024-03-06T14:53:46.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:46.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:46.937786+0000) 2024-03-06T14:53:47.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:47.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:47.938056+0000) 2024-03-06T14:53:48.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:48.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:48.938270+0000) 2024-03-06T14:53:49.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:49.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:49.938485+0000) 2024-03-06T14:53:50.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:50.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:50.938712+0000) 2024-03-06T14:53:51.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:51.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:51.938894+0000) 2024-03-06T14:53:52.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:52.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:52.939139+0000) 2024-03-06T14:53:53.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:53.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:53.939367+0000) 2024-03-06T14:53:54.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:54.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:54.939575+0000) 2024-03-06T14:53:55.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:55.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:55.939784+0000) 2024-03-06T14:53:56.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:56.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:56.940046+0000) 2024-03-06T14:53:57.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:57.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:57.940249+0000) 2024-03-06T14:53:58.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:58.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:58.940453+0000) 2024-03-06T14:53:59.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:53:59.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:53:59.940713+0000) 2024-03-06T14:54:00.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:00.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:00.940922+0000) 2024-03-06T14:54:01.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:01.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:01.941202+0000) 2024-03-06T14:54:02.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:02.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:02.941491+0000) 2024-03-06T14:54:03.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:03.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:03.941773+0000) 2024-03-06T14:54:04.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:04.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:04.942117+0000) 2024-03-06T14:54:05.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:05.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:05.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:05.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:05.942423+0000) 2024-03-06T14:54:06.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:06.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:06.942633+0000) 2024-03-06T14:54:07.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:07.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:07.942822+0000) 2024-03-06T14:54:08.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:08.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:08.943042+0000) 2024-03-06T14:54:09.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:09.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:09.943268+0000) 2024-03-06T14:54:10.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:10.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:10.943485+0000) 2024-03-06T14:54:11.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:11.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:11.943668+0000) 2024-03-06T14:54:12.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:12.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:12.943904+0000) 2024-03-06T14:54:13.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:13.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:13.944137+0000) 2024-03-06T14:54:14.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:14.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:14.944356+0000) 2024-03-06T14:54:15.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:15.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:15.944613+0000) 2024-03-06T14:54:16.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:16.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:16.944891+0000) 2024-03-06T14:54:17.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:17.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:17.945184+0000) 2024-03-06T14:54:18.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:18.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:18.945460+0000) 2024-03-06T14:54:19.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:19.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:19.945735+0000) 2024-03-06T14:54:20.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:20.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:20.946101+0000) 2024-03-06T14:54:21.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:21.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:21.946365+0000) 2024-03-06T14:54:22.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:22.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:22.946650+0000) 2024-03-06T14:54:23.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:23.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:23.946933+0000) 2024-03-06T14:54:24.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:24.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:24.947213+0000) 2024-03-06T14:54:25.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:25.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:25.947488+0000) 2024-03-06T14:54:26.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:26.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:26.947776+0000) 2024-03-06T14:54:27.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:27.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:27.948099+0000) 2024-03-06T14:54:28.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:28.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:28.948305+0000) 2024-03-06T14:54:29.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:29.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:29.948544+0000) 2024-03-06T14:54:30.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:30.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:30.948787+0000) 2024-03-06T14:54:31.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:31.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:31.949057+0000) 2024-03-06T14:54:32.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:32.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:32.949340+0000) 2024-03-06T14:54:33.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:33.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:33.949591+0000) 2024-03-06T14:54:34.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:34.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:34.949879+0000) 2024-03-06T14:54:35.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:35.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:35.950159+0000) 2024-03-06T14:54:35.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:35.999 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:54:36.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:36.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:36.950436+0000) 2024-03-06T14:54:37.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:37.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:37.950707+0000) 2024-03-06T14:54:38.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:38.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:38.950981+0000) 2024-03-06T14:54:39.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:39.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:39.951311+0000) 2024-03-06T14:54:40.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:40.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:40.951576+0000) 2024-03-06T14:54:41.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:41.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:41.951847+0000) 2024-03-06T14:54:42.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:42.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:42.952159+0000) 2024-03-06T14:54:43.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:43.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:43.952372+0000) 2024-03-06T14:54:44.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:44.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:44.952600+0000) 2024-03-06T14:54:45.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:45.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:45.952871+0000) 2024-03-06T14:54:46.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:46.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:46.953136+0000) 2024-03-06T14:54:47.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:47.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:47.953408+0000) 2024-03-06T14:54:48.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:48.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:48.953683+0000) 2024-03-06T14:54:49.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:49.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:49.953959+0000) 2024-03-06T14:54:50.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:50.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:50.954190+0000) 2024-03-06T14:54:51.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:51.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:51.954474+0000) 2024-03-06T14:54:52.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:52.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:52.954700+0000) 2024-03-06T14:54:53.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:53.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:53.954989+0000) 2024-03-06T14:54:54.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:54.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:54.955345+0000) 2024-03-06T14:54:55.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:55.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:55.955550+0000) 2024-03-06T14:54:56.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:56.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:56.955795+0000) 2024-03-06T14:54:57.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:57.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:57.956092+0000) 2024-03-06T14:54:58.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:58.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:58.956361+0000) 2024-03-06T14:54:59.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:54:59.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:54:59.956608+0000) 2024-03-06T14:55:00.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:00.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:00.956882+0000) 2024-03-06T14:55:01.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:01.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:01.957155+0000) 2024-03-06T14:55:02.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:02.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:02.957421+0000) 2024-03-06T14:55:03.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:03.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:03.957673+0000) 2024-03-06T14:55:04.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:04.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:04.957912+0000) 2024-03-06T14:55:05.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:05.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:05.958146+0000) 2024-03-06T14:55:06.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:06.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:06.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:06.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:06.958378+0000) 2024-03-06T14:55:07.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:07.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:07.958643+0000) 2024-03-06T14:55:08.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:08.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:08.958873+0000) 2024-03-06T14:55:09.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:09.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:09.959137+0000) 2024-03-06T14:55:10.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:10.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:10.959364+0000) 2024-03-06T14:55:11.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:11.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:11.959588+0000) 2024-03-06T14:55:12.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:12.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:12.959836+0000) 2024-03-06T14:55:13.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:13.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:13.960094+0000) 2024-03-06T14:55:14.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:14.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:14.960335+0000) 2024-03-06T14:55:15.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:15.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:15.960571+0000) 2024-03-06T14:55:16.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:16.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:16.960818+0000) 2024-03-06T14:55:17.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:17.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:17.961074+0000) 2024-03-06T14:55:18.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:18.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:18.961314+0000) 2024-03-06T14:55:19.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:19.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:19.961576+0000) 2024-03-06T14:55:20.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:20.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:20.961830+0000) 2024-03-06T14:55:21.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:21.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:21.962123+0000) 2024-03-06T14:55:22.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:22.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:22.962408+0000) 2024-03-06T14:55:23.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:23.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:23.962651+0000) 2024-03-06T14:55:24.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:24.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:24.962933+0000) 2024-03-06T14:55:25.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:25.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:25.963173+0000) 2024-03-06T14:55:26.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:26.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:26.963430+0000) 2024-03-06T14:55:27.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:27.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:27.963640+0000) 2024-03-06T14:55:28.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:28.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:28.963967+0000) 2024-03-06T14:55:29.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:29.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:29.964298+0000) 2024-03-06T14:55:30.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:30.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:30.964582+0000) 2024-03-06T14:55:31.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:31.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:31.964865+0000) 2024-03-06T14:55:32.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:32.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:32.965170+0000) 2024-03-06T14:55:33.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:33.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:33.965458+0000) 2024-03-06T14:55:34.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:34.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:34.965718+0000) 2024-03-06T14:55:35.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:35.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:35.965956+0000) 2024-03-06T14:55:36.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:36.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:55:36.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:36.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:36.966184+0000) 2024-03-06T14:55:37.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:37.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:37.966448+0000) 2024-03-06T14:55:38.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:38.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:38.966725+0000) 2024-03-06T14:55:39.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:39.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:39.966990+0000) 2024-03-06T14:55:40.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:40.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:40.967287+0000) 2024-03-06T14:55:41.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:41.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:41.967494+0000) 2024-03-06T14:55:42.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:42.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:42.967695+0000) 2024-03-06T14:55:43.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:43.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:43.967915+0000) 2024-03-06T14:55:44.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:44.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:44.968155+0000) 2024-03-06T14:55:45.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:45.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:45.968379+0000) 2024-03-06T14:55:46.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:46.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:46.968605+0000) 2024-03-06T14:55:47.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:47.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:47.968848+0000) 2024-03-06T14:55:48.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:48.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:48.969163+0000) 2024-03-06T14:55:49.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:49.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:49.969361+0000) 2024-03-06T14:55:50.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:50.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:50.969590+0000) 2024-03-06T14:55:51.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:51.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:51.969818+0000) 2024-03-06T14:55:52.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:52.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:52.970048+0000) 2024-03-06T14:55:53.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:53.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:53.970276+0000) 2024-03-06T14:55:54.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:54.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:54.970521+0000) 2024-03-06T14:55:55.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:55.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:55.970739+0000) 2024-03-06T14:55:56.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:56.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:56.970972+0000) 2024-03-06T14:55:57.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:57.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:57.971190+0000) 2024-03-06T14:55:58.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:58.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:58.971426+0000) 2024-03-06T14:55:59.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:55:59.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:55:59.971674+0000) 2024-03-06T14:56:00.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:00.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:00.971882+0000) 2024-03-06T14:56:01.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:01.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:01.972135+0000) 2024-03-06T14:56:02.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:02.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:02.972313+0000) 2024-03-06T14:56:03.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:03.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:03.972513+0000) 2024-03-06T14:56:04.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:04.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:04.972882+0000) 2024-03-06T14:56:05.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:05.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:05.973159+0000) 2024-03-06T14:56:06.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:06.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:06.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:06.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:06.973409+0000) 2024-03-06T14:56:07.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:07.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:07.973662+0000) 2024-03-06T14:56:08.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:08.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:08.973879+0000) 2024-03-06T14:56:09.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:09.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:09.974119+0000) 2024-03-06T14:56:10.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:10.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:10.974354+0000) 2024-03-06T14:56:11.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:11.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:11.974574+0000) 2024-03-06T14:56:12.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:12.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:12.974780+0000) 2024-03-06T14:56:13.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:13.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:13.975039+0000) 2024-03-06T14:56:14.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:14.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:14.975322+0000) 2024-03-06T14:56:15.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:15.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:15.975554+0000) 2024-03-06T14:56:16.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:16.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:16.975876+0000) 2024-03-06T14:56:17.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:17.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:17.976120+0000) 2024-03-06T14:56:18.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:18.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:18.976375+0000) 2024-03-06T14:56:19.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:19.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:19.976657+0000) 2024-03-06T14:56:20.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:20.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:20.976894+0000) 2024-03-06T14:56:21.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:21.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:21.977118+0000) 2024-03-06T14:56:22.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:22.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:22.977340+0000) 2024-03-06T14:56:23.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:23.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:23.977602+0000) 2024-03-06T14:56:24.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:24.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:24.977888+0000) 2024-03-06T14:56:25.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:25.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:25.978160+0000) 2024-03-06T14:56:26.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:26.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:26.978433+0000) 2024-03-06T14:56:27.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:27.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:27.978668+0000) 2024-03-06T14:56:28.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:28.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:28.978886+0000) 2024-03-06T14:56:29.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:29.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:29.979108+0000) 2024-03-06T14:56:30.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:30.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:30.979335+0000) 2024-03-06T14:56:31.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:31.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:31.979567+0000) 2024-03-06T14:56:32.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:32.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:32.979786+0000) 2024-03-06T14:56:33.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:33.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:33.980036+0000) 2024-03-06T14:56:34.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:34.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:34.980305+0000) 2024-03-06T14:56:35.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:35.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:35.980583+0000) 2024-03-06T14:56:36.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:36.867 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:56:36.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:36.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:36.980869+0000) 2024-03-06T14:56:37.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:37.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:37.981113+0000) 2024-03-06T14:56:38.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:38.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:38.981312+0000) 2024-03-06T14:56:39.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:39.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:39.981486+0000) 2024-03-06T14:56:40.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:40.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:40.981731+0000) 2024-03-06T14:56:41.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:41.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:41.981971+0000) 2024-03-06T14:56:42.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:42.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:42.982164+0000) 2024-03-06T14:56:43.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:43.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:43.982384+0000) 2024-03-06T14:56:44.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:44.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:44.982628+0000) 2024-03-06T14:56:45.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:45.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:45.982864+0000) 2024-03-06T14:56:46.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:46.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:46.983132+0000) 2024-03-06T14:56:47.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:47.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:47.983364+0000) 2024-03-06T14:56:48.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:48.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:48.983615+0000) 2024-03-06T14:56:49.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:49.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:49.983842+0000) 2024-03-06T14:56:50.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:50.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:50.984141+0000) 2024-03-06T14:56:51.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:51.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:51.984430+0000) 2024-03-06T14:56:52.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:52.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:52.984703+0000) 2024-03-06T14:56:53.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:53.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:53.984944+0000) 2024-03-06T14:56:54.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:54.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:54.985279+0000) 2024-03-06T14:56:55.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:55.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:55.985506+0000) 2024-03-06T14:56:56.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:56.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:56.985737+0000) 2024-03-06T14:56:57.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:57.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:57.985971+0000) 2024-03-06T14:56:58.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:58.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:58.986243+0000) 2024-03-06T14:56:59.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:56:59.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:56:59.986489+0000) 2024-03-06T14:57:00.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:00.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:00.986720+0000) 2024-03-06T14:57:01.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:01.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:01.986958+0000) 2024-03-06T14:57:02.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:02.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:02.987192+0000) 2024-03-06T14:57:03.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:03.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:03.987427+0000) 2024-03-06T14:57:04.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:04.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:04.987717+0000) 2024-03-06T14:57:05.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:05.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:05.987983+0000) 2024-03-06T14:57:06.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:06.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:06.988287+0000) 2024-03-06T14:57:07.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:07.041 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:07.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:07.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:07.988570+0000) 2024-03-06T14:57:08.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:08.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:08.988844+0000) 2024-03-06T14:57:09.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:09.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:09.989214+0000) 2024-03-06T14:57:10.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:10.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:10.989494+0000) 2024-03-06T14:57:11.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:11.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:11.989728+0000) 2024-03-06T14:57:12.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:12.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:12.989965+0000) 2024-03-06T14:57:13.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:13.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:13.990292+0000) 2024-03-06T14:57:14.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:14.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:14.990547+0000) 2024-03-06T14:57:15.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:15.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:15.990765+0000) 2024-03-06T14:57:16.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:16.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:16.991074+0000) 2024-03-06T14:57:17.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:17.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:17.991319+0000) 2024-03-06T14:57:18.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:18.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:18.991591+0000) 2024-03-06T14:57:19.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:19.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:19.991826+0000) 2024-03-06T14:57:20.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:20.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:20.992093+0000) 2024-03-06T14:57:21.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:21.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:21.992334+0000) 2024-03-06T14:57:22.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:22.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:22.992579+0000) 2024-03-06T14:57:23.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:23.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:23.992800+0000) 2024-03-06T14:57:24.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:24.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:24.993086+0000) 2024-03-06T14:57:25.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:25.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:25.993334+0000) 2024-03-06T14:57:26.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:26.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:26.993531+0000) 2024-03-06T14:57:27.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:27.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:27.993762+0000) 2024-03-06T14:57:28.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:28.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:28.993994+0000) 2024-03-06T14:57:29.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:29.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:29.994282+0000) 2024-03-06T14:57:30.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:30.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:30.994526+0000) 2024-03-06T14:57:31.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:31.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:31.994809+0000) 2024-03-06T14:57:32.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:32.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:32.995125+0000) 2024-03-06T14:57:33.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:33.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:33.995379+0000) 2024-03-06T14:57:34.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:34.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:34.995615+0000) 2024-03-06T14:57:35.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:35.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:35.995832+0000) 2024-03-06T14:57:36.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:36.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:36.996132+0000) 2024-03-06T14:57:37.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:37.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:57:38.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:37.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:37.996412+0000) 2024-03-06T14:57:39.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:38.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:38.996686+0000) 2024-03-06T14:57:40.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:39.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:39.996965+0000) 2024-03-06T14:57:41.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:40.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:40.997276+0000) 2024-03-06T14:57:42.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:41.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:41.997554+0000) 2024-03-06T14:57:43.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:42.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:42.997832+0000) 2024-03-06T14:57:44.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:43.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:43.998143+0000) 2024-03-06T14:57:45.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:44.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:44.998416+0000) 2024-03-06T14:57:46.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:45.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:45.998630+0000) 2024-03-06T14:57:47.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:46.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:46.998872+0000) 2024-03-06T14:57:48.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:47.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:47.999152+0000) 2024-03-06T14:57:49.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:48.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:48.999400+0000) 2024-03-06T14:57:50.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:49.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:49.999634+0000) 2024-03-06T14:57:51.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:50.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:50.999872+0000) 2024-03-06T14:57:52.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:51.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:52.000109+0000) 2024-03-06T14:57:53.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:52.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:53.000351+0000) 2024-03-06T14:57:54.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:53.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:54.000576+0000) 2024-03-06T14:57:55.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:54.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:55.000814+0000) 2024-03-06T14:57:56.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:56.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:56.001075+0000) 2024-03-06T14:57:57.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:57.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:57.001316+0000) 2024-03-06T14:57:58.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:58.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:58.001571+0000) 2024-03-06T14:57:59.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:57:59.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:57:59.001855+0000) 2024-03-06T14:58:00.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:00.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:00.002161+0000) 2024-03-06T14:58:01.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:01.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:01.002436+0000) 2024-03-06T14:58:02.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:02.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:02.002681+0000) 2024-03-06T14:58:03.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:03.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:03.002870+0000) 2024-03-06T14:58:04.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:04.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:04.003138+0000) 2024-03-06T14:58:05.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:05.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:05.003407+0000) 2024-03-06T14:58:06.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:06.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:06.003680+0000) 2024-03-06T14:58:07.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:07.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:07.003946+0000) 2024-03-06T14:58:07.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:07.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:08.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:08.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:08.004223+0000) 2024-03-06T14:58:09.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:09.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:09.004470+0000) 2024-03-06T14:58:10.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:10.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:10.004735+0000) 2024-03-06T14:58:11.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:11.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:11.004954+0000) 2024-03-06T14:58:12.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:12.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:12.005179+0000) 2024-03-06T14:58:13.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:13.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:13.005406+0000) 2024-03-06T14:58:14.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:14.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:14.005634+0000) 2024-03-06T14:58:15.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:15.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:15.005855+0000) 2024-03-06T14:58:16.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:16.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:16.006148+0000) 2024-03-06T14:58:17.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:17.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:17.006348+0000) 2024-03-06T14:58:18.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:18.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:18.006560+0000) 2024-03-06T14:58:19.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:19.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:19.006794+0000) 2024-03-06T14:58:20.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:20.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:20.007079+0000) 2024-03-06T14:58:21.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:21.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:21.007292+0000) 2024-03-06T14:58:22.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:22.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:22.007553+0000) 2024-03-06T14:58:23.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:23.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:23.007830+0000) 2024-03-06T14:58:24.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:24.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:24.008134+0000) 2024-03-06T14:58:25.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:25.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:25.008424+0000) 2024-03-06T14:58:26.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:26.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:26.008689+0000) 2024-03-06T14:58:27.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:27.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:27.008970+0000) 2024-03-06T14:58:28.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:28.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:28.009309+0000) 2024-03-06T14:58:29.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:29.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:29.009527+0000) 2024-03-06T14:58:30.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:30.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:30.009786+0000) 2024-03-06T14:58:31.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:31.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:31.010119+0000) 2024-03-06T14:58:32.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:32.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:32.010406+0000) 2024-03-06T14:58:33.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:33.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:33.010616+0000) 2024-03-06T14:58:34.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:34.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:34.010835+0000) 2024-03-06T14:58:35.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:35.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:35.011109+0000) 2024-03-06T14:58:36.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:36.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:36.011338+0000) 2024-03-06T14:58:37.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:37.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:37.011569+0000) 2024-03-06T14:58:37.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:37.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:58:38.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:38.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:38.011795+0000) 2024-03-06T14:58:39.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:39.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:39.012064+0000) 2024-03-06T14:58:40.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:40.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:40.012285+0000) 2024-03-06T14:58:41.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:41.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:41.012514+0000) 2024-03-06T14:58:42.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:42.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:42.012752+0000) 2024-03-06T14:58:43.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:43.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:43.012999+0000) 2024-03-06T14:58:44.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:44.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:44.013272+0000) 2024-03-06T14:58:45.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:45.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:45.013520+0000) 2024-03-06T14:58:46.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:46.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:46.013842+0000) 2024-03-06T14:58:47.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:47.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:47.014121+0000) 2024-03-06T14:58:48.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:48.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:48.014342+0000) 2024-03-06T14:58:49.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:49.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:49.014574+0000) 2024-03-06T14:58:50.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:50.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:50.014793+0000) 2024-03-06T14:58:51.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:51.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:51.015046+0000) 2024-03-06T14:58:52.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:52.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:52.015271+0000) 2024-03-06T14:58:53.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:53.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:53.015500+0000) 2024-03-06T14:58:54.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:54.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:54.015720+0000) 2024-03-06T14:58:55.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:55.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:55.016055+0000) 2024-03-06T14:58:56.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:56.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:56.016348+0000) 2024-03-06T14:58:57.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:57.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:57.016620+0000) 2024-03-06T14:58:58.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:58.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:58.016850+0000) 2024-03-06T14:58:59.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:58:59.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:58:59.017135+0000) 2024-03-06T14:59:00.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:00.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:00.017322+0000) 2024-03-06T14:59:01.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:01.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:01.017558+0000) 2024-03-06T14:59:02.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:02.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:02.017778+0000) 2024-03-06T14:59:03.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:03.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:03.018028+0000) 2024-03-06T14:59:04.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:04.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:04.018271+0000) 2024-03-06T14:59:05.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:05.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:05.018507+0000) 2024-03-06T14:59:06.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:06.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:06.018736+0000) 2024-03-06T14:59:07.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:07.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:07.018944+0000) 2024-03-06T14:59:07.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:07.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:08.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:08.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:08.019184+0000) 2024-03-06T14:59:09.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:09.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:09.019427+0000) 2024-03-06T14:59:10.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:10.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:10.019653+0000) 2024-03-06T14:59:11.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:11.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:11.019875+0000) 2024-03-06T14:59:12.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:12.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:12.020134+0000) 2024-03-06T14:59:13.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:13.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:13.020326+0000) 2024-03-06T14:59:14.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:14.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:14.020534+0000) 2024-03-06T14:59:15.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:15.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:15.020739+0000) 2024-03-06T14:59:16.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:16.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:16.020939+0000) 2024-03-06T14:59:17.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:17.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:17.021174+0000) 2024-03-06T14:59:18.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:18.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:18.021388+0000) 2024-03-06T14:59:19.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:19.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:19.021621+0000) 2024-03-06T14:59:20.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:20.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:20.021834+0000) 2024-03-06T14:59:21.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:21.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:21.022096+0000) 2024-03-06T14:59:22.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:22.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:22.022344+0000) 2024-03-06T14:59:23.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:23.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:23.022530+0000) 2024-03-06T14:59:24.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:24.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:24.022754+0000) 2024-03-06T14:59:25.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:25.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:25.022990+0000) 2024-03-06T14:59:26.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:26.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:26.023251+0000) 2024-03-06T14:59:27.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:27.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:27.023473+0000) 2024-03-06T14:59:28.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:28.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:28.023694+0000) 2024-03-06T14:59:29.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:29.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:29.023932+0000) 2024-03-06T14:59:30.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:30.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:30.024162+0000) 2024-03-06T14:59:31.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:31.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:31.024405+0000) 2024-03-06T14:59:32.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:32.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:32.024630+0000) 2024-03-06T14:59:33.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:33.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:33.024861+0000) 2024-03-06T14:59:34.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:34.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:34.025094+0000) 2024-03-06T14:59:35.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:35.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:35.025304+0000) 2024-03-06T14:59:36.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:36.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:36.025538+0000) 2024-03-06T14:59:37.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:37.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:37.025787+0000) 2024-03-06T14:59:38.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:38.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:38.026036+0000) 2024-03-06T14:59:38.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:38.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T14:59:39.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:39.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:39.026348+0000) 2024-03-06T14:59:40.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:40.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:40.026590+0000) 2024-03-06T14:59:41.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:41.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:41.026836+0000) 2024-03-06T14:59:42.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:42.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:42.027061+0000) 2024-03-06T14:59:43.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:43.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:43.027294+0000) 2024-03-06T14:59:44.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:44.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:44.027521+0000) 2024-03-06T14:59:45.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:45.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:45.027761+0000) 2024-03-06T14:59:46.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:46.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:46.027987+0000) 2024-03-06T14:59:47.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:47.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:47.028277+0000) 2024-03-06T14:59:48.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:48.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:48.028478+0000) 2024-03-06T14:59:49.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:49.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:49.028710+0000) 2024-03-06T14:59:50.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:50.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:50.028921+0000) 2024-03-06T14:59:51.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:51.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:51.029166+0000) 2024-03-06T14:59:52.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:52.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:52.029342+0000) 2024-03-06T14:59:53.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:53.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:53.029599+0000) 2024-03-06T14:59:54.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:54.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:54.029844+0000) 2024-03-06T14:59:55.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:55.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:55.030110+0000) 2024-03-06T14:59:56.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:56.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:56.030334+0000) 2024-03-06T14:59:57.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:57.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:57.030518+0000) 2024-03-06T14:59:58.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:58.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:58.030755+0000) 2024-03-06T14:59:59.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T14:59:59.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T13:59:59.031051+0000) 2024-03-06T15:00:00.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:00.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:00.031308+0000) 2024-03-06T15:00:01.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:01.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:01.031552+0000) 2024-03-06T15:00:02.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:02.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:02.031783+0000) 2024-03-06T15:00:03.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:03.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:03.032113+0000) 2024-03-06T15:00:04.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:04.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:04.032364+0000) 2024-03-06T15:00:05.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:05.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:05.032656+0000) 2024-03-06T15:00:06.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:06.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:06.032933+0000) 2024-03-06T15:00:07.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:07.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:07.033169+0000) 2024-03-06T15:00:08.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:08.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:08.033381+0000) 2024-03-06T15:00:08.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:08.204 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:09.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:09.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:09.033639+0000) 2024-03-06T15:00:10.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:10.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:10.033874+0000) 2024-03-06T15:00:11.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:11.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:11.034125+0000) 2024-03-06T15:00:12.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:12.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:12.034362+0000) 2024-03-06T15:00:13.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:13.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:13.034620+0000) 2024-03-06T15:00:14.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:14.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:14.034796+0000) 2024-03-06T15:00:15.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:15.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:15.035029+0000) 2024-03-06T15:00:16.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:16.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:16.035222+0000) 2024-03-06T15:00:17.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:17.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:17.035427+0000) 2024-03-06T15:00:18.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:18.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:18.035689+0000) 2024-03-06T15:00:19.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:19.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:19.035980+0000) 2024-03-06T15:00:20.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:20.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:20.036312+0000) 2024-03-06T15:00:21.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:21.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:21.036587+0000) 2024-03-06T15:00:22.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:22.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:22.036837+0000) 2024-03-06T15:00:23.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:23.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:23.037122+0000) 2024-03-06T15:00:24.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:24.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:24.037369+0000) 2024-03-06T15:00:25.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:25.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:25.037609+0000) 2024-03-06T15:00:26.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:26.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:26.037847+0000) 2024-03-06T15:00:27.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:27.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:27.038181+0000) 2024-03-06T15:00:28.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:28.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:28.038405+0000) 2024-03-06T15:00:29.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:29.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:29.038650+0000) 2024-03-06T15:00:30.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:30.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:30.038916+0000) 2024-03-06T15:00:31.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:31.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:31.039168+0000) 2024-03-06T15:00:32.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:32.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:32.039410+0000) 2024-03-06T15:00:33.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:33.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:33.039664+0000) 2024-03-06T15:00:34.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:34.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:34.039923+0000) 2024-03-06T15:00:35.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:35.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:35.040200+0000) 2024-03-06T15:00:36.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:36.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:36.040474+0000) 2024-03-06T15:00:37.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:37.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:37.040748+0000) 2024-03-06T15:00:38.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:38.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:38.041066+0000) 2024-03-06T15:00:38.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:38.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:00:39.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:39.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:39.041359+0000) 2024-03-06T15:00:40.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:40.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:40.041640+0000) 2024-03-06T15:00:41.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:41.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:41.041884+0000) 2024-03-06T15:00:42.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:42.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:42.042158+0000) 2024-03-06T15:00:43.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:43.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:43.042440+0000) 2024-03-06T15:00:44.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:44.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:44.042725+0000) 2024-03-06T15:00:45.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:45.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:45.042951+0000) 2024-03-06T15:00:46.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:46.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:46.043203+0000) 2024-03-06T15:00:47.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:47.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:47.043476+0000) 2024-03-06T15:00:48.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:48.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:48.043734+0000) 2024-03-06T15:00:49.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:49.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:49.043999+0000) 2024-03-06T15:00:50.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:50.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:50.044254+0000) 2024-03-06T15:00:51.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:51.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:51.044372+0000) 2024-03-06T15:00:52.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:52.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:52.044583+0000) 2024-03-06T15:00:53.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:53.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:53.044808+0000) 2024-03-06T15:00:54.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:54.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:54.045048+0000) 2024-03-06T15:00:55.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:55.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:55.045292+0000) 2024-03-06T15:00:56.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:56.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:56.045518+0000) 2024-03-06T15:00:57.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:57.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:57.045742+0000) 2024-03-06T15:00:58.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:58.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:58.045923+0000) 2024-03-06T15:00:59.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:00:59.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:00:59.046132+0000) 2024-03-06T15:01:00.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:00.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:00.046295+0000) 2024-03-06T15:01:01.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:01.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:01.046474+0000) 2024-03-06T15:01:02.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:02.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:02.046709+0000) 2024-03-06T15:01:03.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:03.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:03.046935+0000) 2024-03-06T15:01:04.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:04.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:04.047166+0000) 2024-03-06T15:01:05.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:05.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:05.047409+0000) 2024-03-06T15:01:06.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:06.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:06.047587+0000) 2024-03-06T15:01:07.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:07.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:07.047768+0000) 2024-03-06T15:01:08.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:08.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:08.047980+0000) 2024-03-06T15:01:08.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:08.540 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:09.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:09.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:09.048255+0000) 2024-03-06T15:01:10.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:10.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:10.048490+0000) 2024-03-06T15:01:11.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:11.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:11.048713+0000) 2024-03-06T15:01:12.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:12.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:12.048909+0000) 2024-03-06T15:01:13.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:13.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:13.049139+0000) 2024-03-06T15:01:14.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:14.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:14.049366+0000) 2024-03-06T15:01:15.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:15.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:15.049653+0000) 2024-03-06T15:01:16.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:16.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:16.049875+0000) 2024-03-06T15:01:17.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:17.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:17.050142+0000) 2024-03-06T15:01:18.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:18.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:18.050361+0000) 2024-03-06T15:01:19.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:19.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:19.050589+0000) 2024-03-06T15:01:20.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:20.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:20.050812+0000) 2024-03-06T15:01:21.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:21.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:21.051095+0000) 2024-03-06T15:01:22.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:22.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:22.051346+0000) 2024-03-06T15:01:23.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:23.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:23.051571+0000) 2024-03-06T15:01:24.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:24.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:24.051828+0000) 2024-03-06T15:01:25.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:25.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:25.052056+0000) 2024-03-06T15:01:26.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:26.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:26.052254+0000) 2024-03-06T15:01:27.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:27.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:27.052482+0000) 2024-03-06T15:01:28.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:28.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:28.052695+0000) 2024-03-06T15:01:29.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:29.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:29.052927+0000) 2024-03-06T15:01:30.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:30.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:30.053149+0000) 2024-03-06T15:01:31.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:31.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:31.053377+0000) 2024-03-06T15:01:32.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:32.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:32.053579+0000) 2024-03-06T15:01:33.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:33.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:33.053806+0000) 2024-03-06T15:01:34.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:34.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:34.054038+0000) 2024-03-06T15:01:35.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:35.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:35.054300+0000) 2024-03-06T15:01:36.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:36.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:36.054516+0000) 2024-03-06T15:01:37.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:37.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:37.054781+0000) 2024-03-06T15:01:38.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:38.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:38.054997+0000) 2024-03-06T15:01:38.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:38.754 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:01:39.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:39.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:39.055207+0000) 2024-03-06T15:01:40.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:40.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:40.055442+0000) 2024-03-06T15:01:41.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:41.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:41.055675+0000) 2024-03-06T15:01:42.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:42.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:42.055904+0000) 2024-03-06T15:01:43.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:43.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:43.056217+0000) 2024-03-06T15:01:44.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:44.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:44.056488+0000) 2024-03-06T15:01:45.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:45.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:45.056702+0000) 2024-03-06T15:01:46.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:46.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:46.056919+0000) 2024-03-06T15:01:47.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:47.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:47.057147+0000) 2024-03-06T15:01:48.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:48.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:48.057362+0000) 2024-03-06T15:01:49.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:49.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:49.057595+0000) 2024-03-06T15:01:50.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:50.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:50.057815+0000) 2024-03-06T15:01:51.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:51.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:51.058073+0000) 2024-03-06T15:01:52.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:52.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:52.058297+0000) 2024-03-06T15:01:53.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:53.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:53.058573+0000) 2024-03-06T15:01:54.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:54.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:54.058852+0000) 2024-03-06T15:01:55.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:55.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:55.059155+0000) 2024-03-06T15:01:56.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:56.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:56.059384+0000) 2024-03-06T15:01:57.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:57.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:57.059629+0000) 2024-03-06T15:01:58.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:58.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:58.059911+0000) 2024-03-06T15:01:59.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:01:59.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:01:59.060215+0000) 2024-03-06T15:02:00.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:00.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:00.060452+0000) 2024-03-06T15:02:01.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:01.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:01.060730+0000) 2024-03-06T15:02:02.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:02.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:02.060995+0000) 2024-03-06T15:02:03.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:03.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:03.061279+0000) 2024-03-06T15:02:04.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:04.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:04.061533+0000) 2024-03-06T15:02:05.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:05.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:05.061752+0000) 2024-03-06T15:02:06.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:06.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:06.061949+0000) 2024-03-06T15:02:07.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:07.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:07.062179+0000) 2024-03-06T15:02:08.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:08.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:08.062397+0000) 2024-03-06T15:02:08.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:08.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:09.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:09.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:09.062689+0000) 2024-03-06T15:02:10.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:10.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:10.062943+0000) 2024-03-06T15:02:11.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:11.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:11.063185+0000) 2024-03-06T15:02:12.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:12.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:12.063420+0000) 2024-03-06T15:02:13.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:13.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:13.063681+0000) 2024-03-06T15:02:14.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:14.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:14.063924+0000) 2024-03-06T15:02:15.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:15.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:15.064181+0000) 2024-03-06T15:02:16.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:16.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:16.064383+0000) 2024-03-06T15:02:17.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:17.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:17.064623+0000) 2024-03-06T15:02:18.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:18.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:18.064852+0000) 2024-03-06T15:02:19.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:19.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:19.065156+0000) 2024-03-06T15:02:20.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:20.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:20.065434+0000) 2024-03-06T15:02:21.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:21.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:21.065702+0000) 2024-03-06T15:02:22.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:22.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:22.065973+0000) 2024-03-06T15:02:23.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:23.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:23.066289+0000) 2024-03-06T15:02:24.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:24.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:24.066541+0000) 2024-03-06T15:02:25.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:25.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:25.066834+0000) 2024-03-06T15:02:26.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:26.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:26.067108+0000) 2024-03-06T15:02:27.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:27.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:27.067368+0000) 2024-03-06T15:02:28.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:28.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:28.067553+0000) 2024-03-06T15:02:29.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:29.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:29.067782+0000) 2024-03-06T15:02:30.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:30.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:30.068039+0000) 2024-03-06T15:02:31.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:31.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:31.068273+0000) 2024-03-06T15:02:32.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:32.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:32.068489+0000) 2024-03-06T15:02:33.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:33.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:33.068709+0000) 2024-03-06T15:02:34.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:34.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:34.068941+0000) 2024-03-06T15:02:35.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:35.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:35.069181+0000) 2024-03-06T15:02:36.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:36.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:36.069365+0000) 2024-03-06T15:02:37.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:37.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:37.069593+0000) 2024-03-06T15:02:38.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:38.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:38.069817+0000) 2024-03-06T15:02:39.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:39.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:39.070033+0000) 2024-03-06T15:02:39.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:39.122 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:02:40.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:40.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:40.070262+0000) 2024-03-06T15:02:41.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:41.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:41.070509+0000) 2024-03-06T15:02:42.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:42.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:42.070731+0000) 2024-03-06T15:02:43.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:43.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:43.070933+0000) 2024-03-06T15:02:44.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:44.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:44.071174+0000) 2024-03-06T15:02:45.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:45.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:45.071390+0000) 2024-03-06T15:02:46.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:46.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:46.071601+0000) 2024-03-06T15:02:47.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:47.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:47.071863+0000) 2024-03-06T15:02:48.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:48.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:48.072148+0000) 2024-03-06T15:02:49.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:49.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:49.072365+0000) 2024-03-06T15:02:50.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:50.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:50.072584+0000) 2024-03-06T15:02:51.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:51.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:51.072820+0000) 2024-03-06T15:02:52.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:52.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:52.073090+0000) 2024-03-06T15:02:53.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:53.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:53.073353+0000) 2024-03-06T15:02:54.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:54.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:54.073597+0000) 2024-03-06T15:02:55.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:55.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:55.073849+0000) 2024-03-06T15:02:56.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:56.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:56.074131+0000) 2024-03-06T15:02:57.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:57.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:57.074419+0000) 2024-03-06T15:02:58.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:58.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:58.074655+0000) 2024-03-06T15:02:59.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:02:59.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:02:59.074888+0000) 2024-03-06T15:03:00.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:00.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:00.075158+0000) 2024-03-06T15:03:01.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:01.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:01.075403+0000) 2024-03-06T15:03:02.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:02.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:02.075645+0000) 2024-03-06T15:03:03.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:03.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:03.075890+0000) 2024-03-06T15:03:04.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:04.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:04.076132+0000) 2024-03-06T15:03:05.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:05.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:05.076390+0000) 2024-03-06T15:03:06.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:06.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:06.076619+0000) 2024-03-06T15:03:07.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:07.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:07.076862+0000) 2024-03-06T15:03:08.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:08.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:08.077118+0000) 2024-03-06T15:03:09.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:09.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:09.077359+0000) 2024-03-06T15:03:09.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:09.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:10.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:10.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:10.077570+0000) 2024-03-06T15:03:11.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:11.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:11.077809+0000) 2024-03-06T15:03:12.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:12.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:12.078053+0000) 2024-03-06T15:03:13.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:13.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:13.078296+0000) 2024-03-06T15:03:14.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:14.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:14.078545+0000) 2024-03-06T15:03:15.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:15.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:15.078788+0000) 2024-03-06T15:03:16.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:16.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:16.079060+0000) 2024-03-06T15:03:17.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:17.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:17.079310+0000) 2024-03-06T15:03:18.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:18.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:18.079549+0000) 2024-03-06T15:03:19.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:19.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:19.079793+0000) 2024-03-06T15:03:20.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:20.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:20.080057+0000) 2024-03-06T15:03:21.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:21.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:21.080306+0000) 2024-03-06T15:03:22.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:22.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:22.080547+0000) 2024-03-06T15:03:23.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:23.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:23.080800+0000) 2024-03-06T15:03:24.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:24.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:24.081037+0000) 2024-03-06T15:03:25.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:25.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:25.081272+0000) 2024-03-06T15:03:26.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:26.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:26.081516+0000) 2024-03-06T15:03:27.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:27.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:27.081745+0000) 2024-03-06T15:03:28.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:28.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:28.081964+0000) 2024-03-06T15:03:29.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:29.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:29.082192+0000) 2024-03-06T15:03:30.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:30.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:30.082432+0000) 2024-03-06T15:03:31.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:31.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:31.082651+0000) 2024-03-06T15:03:32.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:32.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:32.082874+0000) 2024-03-06T15:03:33.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:33.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:33.083150+0000) 2024-03-06T15:03:34.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:34.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:34.083378+0000) 2024-03-06T15:03:35.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:35.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:35.083664+0000) 2024-03-06T15:03:36.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:36.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:36.083897+0000) 2024-03-06T15:03:37.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:37.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:37.084140+0000) 2024-03-06T15:03:38.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:38.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:38.084382+0000) 2024-03-06T15:03:39.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:39.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:39.084607+0000) 2024-03-06T15:03:39.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:39.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:03:40.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:40.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:40.084831+0000) 2024-03-06T15:03:41.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:41.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:41.085095+0000) 2024-03-06T15:03:42.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:42.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:42.085336+0000) 2024-03-06T15:03:43.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:43.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:43.085516+0000) 2024-03-06T15:03:44.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:44.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:44.085735+0000) 2024-03-06T15:03:45.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:45.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:45.086095+0000) 2024-03-06T15:03:46.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:46.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:46.086334+0000) 2024-03-06T15:03:47.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:47.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:47.086559+0000) 2024-03-06T15:03:48.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:48.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:48.086783+0000) 2024-03-06T15:03:49.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:49.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:49.086979+0000) 2024-03-06T15:03:50.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:50.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:50.087228+0000) 2024-03-06T15:03:51.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:51.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:51.087447+0000) 2024-03-06T15:03:52.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:52.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:52.087634+0000) 2024-03-06T15:03:53.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:53.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:53.087907+0000) 2024-03-06T15:03:54.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:54.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:54.088129+0000) 2024-03-06T15:03:55.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:55.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:55.088382+0000) 2024-03-06T15:03:56.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:56.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:56.088632+0000) 2024-03-06T15:03:57.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:57.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:57.088877+0000) 2024-03-06T15:03:58.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:58.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:58.089113+0000) 2024-03-06T15:03:59.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:03:59.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:03:59.089333+0000) 2024-03-06T15:04:00.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:00.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:00.089554+0000) 2024-03-06T15:04:01.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:01.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:01.089781+0000) 2024-03-06T15:04:02.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:02.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:02.090046+0000) 2024-03-06T15:04:03.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:03.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:03.090308+0000) 2024-03-06T15:04:04.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:04.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:04.090502+0000) 2024-03-06T15:04:05.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:05.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:05.090736+0000) 2024-03-06T15:04:06.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:06.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:06.090965+0000) 2024-03-06T15:04:07.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:07.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:07.091184+0000) 2024-03-06T15:04:08.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:08.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:08.091402+0000) 2024-03-06T15:04:09.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:09.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:09.091597+0000) 2024-03-06T15:04:09.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:09.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:10.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:10.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:10.091792+0000) 2024-03-06T15:04:11.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:11.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:11.091988+0000) 2024-03-06T15:04:12.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:12.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:12.092265+0000) 2024-03-06T15:04:13.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:13.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:13.092495+0000) 2024-03-06T15:04:14.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:14.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:14.092724+0000) 2024-03-06T15:04:15.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:15.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:15.092961+0000) 2024-03-06T15:04:16.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:16.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:16.093178+0000) 2024-03-06T15:04:17.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:17.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:17.093419+0000) 2024-03-06T15:04:18.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:18.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:18.093653+0000) 2024-03-06T15:04:19.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:19.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:19.093891+0000) 2024-03-06T15:04:20.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:20.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:20.094162+0000) 2024-03-06T15:04:21.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:21.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:21.094414+0000) 2024-03-06T15:04:22.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:22.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:22.094636+0000) 2024-03-06T15:04:23.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:23.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:23.094823+0000) 2024-03-06T15:04:24.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:24.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:24.095076+0000) 2024-03-06T15:04:25.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:25.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:25.095323+0000) 2024-03-06T15:04:26.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:26.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:26.095543+0000) 2024-03-06T15:04:27.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:27.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:27.095788+0000) 2024-03-06T15:04:28.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:28.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:28.096049+0000) 2024-03-06T15:04:29.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:29.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:29.096276+0000) 2024-03-06T15:04:30.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:30.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:30.096507+0000) 2024-03-06T15:04:31.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:31.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:31.096719+0000) 2024-03-06T15:04:32.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:32.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:32.096951+0000) 2024-03-06T15:04:33.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:33.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:33.097237+0000) 2024-03-06T15:04:34.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:34.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:34.097478+0000) 2024-03-06T15:04:35.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:35.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:35.097705+0000) 2024-03-06T15:04:36.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:36.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:36.097885+0000) 2024-03-06T15:04:37.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:37.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:37.098113+0000) 2024-03-06T15:04:38.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:38.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:38.098361+0000) 2024-03-06T15:04:39.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:39.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:39.098588+0000) 2024-03-06T15:04:39.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:39.915 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:04:40.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:40.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:40.098826+0000) 2024-03-06T15:04:41.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:41.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:41.099069+0000) 2024-03-06T15:04:42.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:42.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:42.099255+0000) 2024-03-06T15:04:43.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:43.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:43.099481+0000) 2024-03-06T15:04:44.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:44.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:44.099700+0000) 2024-03-06T15:04:45.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:45.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:45.099985+0000) 2024-03-06T15:04:46.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:46.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:46.100246+0000) 2024-03-06T15:04:47.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:47.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:47.100458+0000) 2024-03-06T15:04:48.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:48.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:48.100682+0000) 2024-03-06T15:04:49.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:49.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:49.100914+0000) 2024-03-06T15:04:50.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:50.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:50.101145+0000) 2024-03-06T15:04:51.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:51.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:51.101369+0000) 2024-03-06T15:04:52.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:52.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:52.101595+0000) 2024-03-06T15:04:53.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:53.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:53.101820+0000) 2024-03-06T15:04:54.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:54.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:54.101997+0000) 2024-03-06T15:04:55.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:55.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:55.102322+0000) 2024-03-06T15:04:56.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:56.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:56.102560+0000) 2024-03-06T15:04:57.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:57.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:57.102813+0000) 2024-03-06T15:04:58.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:58.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:58.103109+0000) 2024-03-06T15:04:59.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:04:59.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:04:59.103375+0000) 2024-03-06T15:05:00.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:00.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:00.103593+0000) 2024-03-06T15:05:01.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:01.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:01.103786+0000) 2024-03-06T15:05:02.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:02.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:02.104037+0000) 2024-03-06T15:05:03.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:03.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:03.104268+0000) 2024-03-06T15:05:04.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:04.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:04.104485+0000) 2024-03-06T15:05:05.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:05.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:05.104733+0000) 2024-03-06T15:05:06.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:06.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:06.104954+0000) 2024-03-06T15:05:07.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:07.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:07.105185+0000) 2024-03-06T15:05:08.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:08.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:08.105410+0000) 2024-03-06T15:05:09.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:09.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:09.105654+0000) 2024-03-06T15:05:10.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:10.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:10.105885+0000) 2024-03-06T15:05:10.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:10.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:11.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:11.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:11.106142+0000) 2024-03-06T15:05:12.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:12.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:12.106352+0000) 2024-03-06T15:05:13.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:13.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:13.106559+0000) 2024-03-06T15:05:14.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:14.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:14.106790+0000) 2024-03-06T15:05:15.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:15.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:15.107071+0000) 2024-03-06T15:05:16.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:16.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:16.107306+0000) 2024-03-06T15:05:17.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:17.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:17.107534+0000) 2024-03-06T15:05:18.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:18.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:18.107778+0000) 2024-03-06T15:05:19.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:19.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:19.107950+0000) 2024-03-06T15:05:20.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:20.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:20.108177+0000) 2024-03-06T15:05:21.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:21.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:21.108424+0000) 2024-03-06T15:05:22.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:22.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:22.108660+0000) 2024-03-06T15:05:23.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:23.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:23.108903+0000) 2024-03-06T15:05:24.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:24.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:24.109133+0000) 2024-03-06T15:05:25.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:25.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:25.109406+0000) 2024-03-06T15:05:26.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:26.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:26.109705+0000) 2024-03-06T15:05:27.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:27.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:27.109949+0000) 2024-03-06T15:05:28.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:28.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:28.110164+0000) 2024-03-06T15:05:29.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:29.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:29.110361+0000) 2024-03-06T15:05:30.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:30.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:30.110594+0000) 2024-03-06T15:05:31.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:31.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:31.110852+0000) 2024-03-06T15:05:32.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:32.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:32.111073+0000) 2024-03-06T15:05:33.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:33.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:33.111307+0000) 2024-03-06T15:05:34.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:34.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:34.111530+0000) 2024-03-06T15:05:35.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:35.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:35.111797+0000) 2024-03-06T15:05:36.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:36.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:36.112036+0000) 2024-03-06T15:05:37.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:37.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:37.112278+0000) 2024-03-06T15:05:38.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:38.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:38.112507+0000) 2024-03-06T15:05:39.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:39.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:39.112714+0000) 2024-03-06T15:05:40.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:40.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:40.112957+0000) 2024-03-06T15:05:40.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:40.349 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:05:41.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:41.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:41.113192+0000) 2024-03-06T15:05:42.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:42.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:42.113419+0000) 2024-03-06T15:05:43.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:43.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:43.113648+0000) 2024-03-06T15:05:44.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:44.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:44.113886+0000) 2024-03-06T15:05:45.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:45.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:45.114132+0000) 2024-03-06T15:05:46.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:46.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:46.114339+0000) 2024-03-06T15:05:47.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:47.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:47.114563+0000) 2024-03-06T15:05:48.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:48.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:48.114787+0000) 2024-03-06T15:05:49.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:49.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:49.115046+0000) 2024-03-06T15:05:50.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:50.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:50.115244+0000) 2024-03-06T15:05:51.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:51.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:51.115471+0000) 2024-03-06T15:05:52.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:52.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:52.115745+0000) 2024-03-06T15:05:53.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:53.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:53.116047+0000) 2024-03-06T15:05:54.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:54.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:54.116338+0000) 2024-03-06T15:05:55.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:55.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:55.116642+0000) 2024-03-06T15:05:56.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:56.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:56.116928+0000) 2024-03-06T15:05:57.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:57.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:57.117201+0000) 2024-03-06T15:05:58.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:58.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:58.117451+0000) 2024-03-06T15:05:59.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:05:59.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:05:59.117726+0000) 2024-03-06T15:06:00.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:00.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:00.117940+0000) 2024-03-06T15:06:01.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:01.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:01.118192+0000) 2024-03-06T15:06:02.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:02.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:02.118438+0000) 2024-03-06T15:06:03.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:03.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:03.118659+0000) 2024-03-06T15:06:04.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:04.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:04.118884+0000) 2024-03-06T15:06:05.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:05.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:05.119169+0000) 2024-03-06T15:06:06.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:06.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:06.119325+0000) 2024-03-06T15:06:07.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:07.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:07.119548+0000) 2024-03-06T15:06:08.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:08.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:08.119770+0000) 2024-03-06T15:06:09.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:09.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:09.119992+0000) 2024-03-06T15:06:10.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:10.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:10.120269+0000) 2024-03-06T15:06:10.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:10.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:11.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:11.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:11.120483+0000) 2024-03-06T15:06:12.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:12.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:12.120677+0000) 2024-03-06T15:06:13.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:13.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:13.120913+0000) 2024-03-06T15:06:14.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:14.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:14.121153+0000) 2024-03-06T15:06:15.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:15.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:15.121568+0000) 2024-03-06T15:06:16.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:16.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:16.121817+0000) 2024-03-06T15:06:17.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:17.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:17.122081+0000) 2024-03-06T15:06:18.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:18.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:18.122319+0000) 2024-03-06T15:06:19.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:19.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:19.122543+0000) 2024-03-06T15:06:20.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:20.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:20.122752+0000) 2024-03-06T15:06:21.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:21.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:21.122944+0000) 2024-03-06T15:06:22.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:22.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:22.123173+0000) 2024-03-06T15:06:23.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:23.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:23.123402+0000) 2024-03-06T15:06:24.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:24.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:24.123630+0000) 2024-03-06T15:06:25.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:25.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:25.123847+0000) 2024-03-06T15:06:26.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:26.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:26.124127+0000) 2024-03-06T15:06:27.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:27.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:27.124378+0000) 2024-03-06T15:06:28.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:28.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:28.124616+0000) 2024-03-06T15:06:29.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:29.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:29.124864+0000) 2024-03-06T15:06:30.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:30.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:30.125164+0000) 2024-03-06T15:06:31.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:31.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:31.125423+0000) 2024-03-06T15:06:32.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:32.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:32.125680+0000) 2024-03-06T15:06:33.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:33.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:33.125926+0000) 2024-03-06T15:06:34.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:34.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:34.126158+0000) 2024-03-06T15:06:35.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:35.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:35.126428+0000) 2024-03-06T15:06:36.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:36.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:36.126686+0000) 2024-03-06T15:06:37.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:37.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:37.126901+0000) 2024-03-06T15:06:38.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:38.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:38.127146+0000) 2024-03-06T15:06:39.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:39.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:39.127359+0000) 2024-03-06T15:06:40.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:40.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:40.127608+0000) 2024-03-06T15:06:40.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:40.734 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:06:41.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:41.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:41.127853+0000) 2024-03-06T15:06:42.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:42.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:42.128087+0000) 2024-03-06T15:06:43.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:43.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:43.128285+0000) 2024-03-06T15:06:44.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:44.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:44.128511+0000) 2024-03-06T15:06:45.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:45.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:45.128749+0000) 2024-03-06T15:06:46.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:46.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:46.129057+0000) 2024-03-06T15:06:47.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:47.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:47.129356+0000) 2024-03-06T15:06:48.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:48.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:48.129643+0000) 2024-03-06T15:06:49.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:49.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:49.129923+0000) 2024-03-06T15:06:50.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:50.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:50.130208+0000) 2024-03-06T15:06:51.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:51.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:51.130459+0000) 2024-03-06T15:06:52.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:52.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:52.130681+0000) 2024-03-06T15:06:53.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:53.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:53.130941+0000) 2024-03-06T15:06:54.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:54.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:54.131153+0000) 2024-03-06T15:06:55.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:55.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:55.131388+0000) 2024-03-06T15:06:56.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:56.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:56.131611+0000) 2024-03-06T15:06:57.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:57.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:57.131835+0000) 2024-03-06T15:06:58.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:58.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:58.132108+0000) 2024-03-06T15:06:59.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:06:59.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:06:59.132327+0000) 2024-03-06T15:07:00.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:00.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:00.132524+0000) 2024-03-06T15:07:01.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:01.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:01.132753+0000) 2024-03-06T15:07:02.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:02.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:02.132982+0000) 2024-03-06T15:07:03.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:03.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:03.133229+0000) 2024-03-06T15:07:04.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:04.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:04.133450+0000) 2024-03-06T15:07:05.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:05.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:05.133690+0000) 2024-03-06T15:07:06.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:06.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:06.133905+0000) 2024-03-06T15:07:07.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:07.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:07.134264+0000) 2024-03-06T15:07:08.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:08.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:08.134447+0000) 2024-03-06T15:07:09.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:09.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:09.134683+0000) 2024-03-06T15:07:10.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:10.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:10.134969+0000) 2024-03-06T15:07:10.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:10.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:11.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:11.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:11.135299+0000) 2024-03-06T15:07:12.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:12.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:12.135569+0000) 2024-03-06T15:07:13.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:13.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:13.135848+0000) 2024-03-06T15:07:14.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:14.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:14.136132+0000) 2024-03-06T15:07:15.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:15.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:15.136398+0000) 2024-03-06T15:07:16.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:16.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:16.136637+0000) 2024-03-06T15:07:17.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:17.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:17.136880+0000) 2024-03-06T15:07:18.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:18.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:18.137126+0000) 2024-03-06T15:07:19.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:19.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:19.137335+0000) 2024-03-06T15:07:20.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:20.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:20.137562+0000) 2024-03-06T15:07:21.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:21.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:21.137801+0000) 2024-03-06T15:07:22.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:22.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:22.138065+0000) 2024-03-06T15:07:23.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:23.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:23.138339+0000) 2024-03-06T15:07:24.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:24.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:24.138551+0000) 2024-03-06T15:07:25.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:25.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:25.138789+0000) 2024-03-06T15:07:26.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:26.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:26.138976+0000) 2024-03-06T15:07:27.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:27.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:27.139239+0000) 2024-03-06T15:07:28.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:28.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:28.139437+0000) 2024-03-06T15:07:29.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:29.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:29.139673+0000) 2024-03-06T15:07:30.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:30.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:30.139909+0000) 2024-03-06T15:07:31.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:31.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:31.140149+0000) 2024-03-06T15:07:32.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:32.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:32.140345+0000) 2024-03-06T15:07:33.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:33.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:33.140553+0000) 2024-03-06T15:07:34.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:34.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:34.140782+0000) 2024-03-06T15:07:35.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:35.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:35.141035+0000) 2024-03-06T15:07:36.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:36.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:36.141229+0000) 2024-03-06T15:07:37.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:37.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:37.141412+0000) 2024-03-06T15:07:38.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:38.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:38.141952+0000) 2024-03-06T15:07:39.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:39.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:39.142199+0000) 2024-03-06T15:07:40.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:40.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:40.142415+0000) 2024-03-06T15:07:41.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:41.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:07:41.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:41.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:41.142604+0000) 2024-03-06T15:07:42.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:42.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:42.142773+0000) 2024-03-06T15:07:43.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:43.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:43.142982+0000) 2024-03-06T15:07:44.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:44.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:44.143249+0000) 2024-03-06T15:07:45.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:45.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:45.143531+0000) 2024-03-06T15:07:46.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:46.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:46.143753+0000) 2024-03-06T15:07:47.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:47.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:47.143999+0000) 2024-03-06T15:07:48.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:48.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:48.144274+0000) 2024-03-06T15:07:49.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:49.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:49.144469+0000) 2024-03-06T15:07:50.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:50.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:50.144633+0000) 2024-03-06T15:07:51.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:51.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:51.144831+0000) 2024-03-06T15:07:52.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:52.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:52.145105+0000) 2024-03-06T15:07:53.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:53.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:53.145328+0000) 2024-03-06T15:07:54.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:54.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:54.145501+0000) 2024-03-06T15:07:55.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:55.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:55.145792+0000) 2024-03-06T15:07:56.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:56.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:56.146082+0000) 2024-03-06T15:07:57.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:57.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:57.146324+0000) 2024-03-06T15:07:58.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:58.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:58.146499+0000) 2024-03-06T15:07:59.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:07:59.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:07:59.146720+0000) 2024-03-06T15:08:00.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:00.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:00.146944+0000) 2024-03-06T15:08:01.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:01.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:01.147183+0000) 2024-03-06T15:08:02.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:02.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:02.147741+0000) 2024-03-06T15:08:03.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:03.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:03.148079+0000) 2024-03-06T15:08:04.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:04.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:04.148321+0000) 2024-03-06T15:08:05.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:05.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:05.148623+0000) 2024-03-06T15:08:06.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:06.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:06.148893+0000) 2024-03-06T15:08:07.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:07.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:07.149176+0000) 2024-03-06T15:08:08.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:08.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:08.149424+0000) 2024-03-06T15:08:09.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:09.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:09.149699+0000) 2024-03-06T15:08:10.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:10.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:10.149976+0000) 2024-03-06T15:08:11.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:11.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:11.150271+0000) 2024-03-06T15:08:11.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:11.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:12.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:12.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:12.150541+0000) 2024-03-06T15:08:13.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:13.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:13.150813+0000) 2024-03-06T15:08:14.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:14.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:14.151120+0000) 2024-03-06T15:08:15.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:15.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:15.151381+0000) 2024-03-06T15:08:16.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:16.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:16.151659+0000) 2024-03-06T15:08:17.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:17.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:17.151936+0000) 2024-03-06T15:08:18.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:18.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:18.152216+0000) 2024-03-06T15:08:19.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:19.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:19.152486+0000) 2024-03-06T15:08:20.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:20.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:20.152764+0000) 2024-03-06T15:08:21.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:21.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:21.153085+0000) 2024-03-06T15:08:22.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:22.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:22.153337+0000) 2024-03-06T15:08:23.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:23.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:23.153621+0000) 2024-03-06T15:08:24.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:24.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:24.153898+0000) 2024-03-06T15:08:25.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:25.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:25.154190+0000) 2024-03-06T15:08:26.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:26.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:26.154396+0000) 2024-03-06T15:08:27.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:27.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:27.154629+0000) 2024-03-06T15:08:28.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:28.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:28.154842+0000) 2024-03-06T15:08:29.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:29.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:29.155159+0000) 2024-03-06T15:08:30.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:30.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:30.155335+0000) 2024-03-06T15:08:31.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:31.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:31.155565+0000) 2024-03-06T15:08:32.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:32.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:32.155813+0000) 2024-03-06T15:08:33.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:33.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:33.156069+0000) 2024-03-06T15:08:34.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:34.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:34.156323+0000) 2024-03-06T15:08:35.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:35.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:35.156557+0000) 2024-03-06T15:08:36.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:36.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:36.156776+0000) 2024-03-06T15:08:37.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:37.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:37.157059+0000) 2024-03-06T15:08:38.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:38.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:38.157314+0000) 2024-03-06T15:08:39.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:39.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:39.157577+0000) 2024-03-06T15:08:40.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:40.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:40.157851+0000) 2024-03-06T15:08:41.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:41.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:41.158164+0000) 2024-03-06T15:08:41.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:41.425 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:08:42.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:42.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:42.158409+0000) 2024-03-06T15:08:43.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:43.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:43.158734+0000) 2024-03-06T15:08:44.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:44.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:44.159051+0000) 2024-03-06T15:08:45.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:45.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:45.159352+0000) 2024-03-06T15:08:46.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:46.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:46.159618+0000) 2024-03-06T15:08:47.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:47.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:47.159862+0000) 2024-03-06T15:08:48.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:48.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:48.160132+0000) 2024-03-06T15:08:49.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:49.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:49.160317+0000) 2024-03-06T15:08:50.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:50.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:50.160578+0000) 2024-03-06T15:08:51.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:51.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:51.160806+0000) 2024-03-06T15:08:52.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:52.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:52.161062+0000) 2024-03-06T15:08:53.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:53.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:53.161285+0000) 2024-03-06T15:08:54.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:54.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:54.161558+0000) 2024-03-06T15:08:55.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:55.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:55.161842+0000) 2024-03-06T15:08:56.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:56.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:56.162131+0000) 2024-03-06T15:08:57.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:57.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:57.162375+0000) 2024-03-06T15:08:58.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:58.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:58.162611+0000) 2024-03-06T15:08:59.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:08:59.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:08:59.162855+0000) 2024-03-06T15:09:00.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:00.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:00.163060+0000) 2024-03-06T15:09:01.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:01.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:01.163293+0000) 2024-03-06T15:09:02.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:02.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:02.163518+0000) 2024-03-06T15:09:03.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:03.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:03.163748+0000) 2024-03-06T15:09:04.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:04.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:04.163965+0000) 2024-03-06T15:09:05.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:05.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:05.164207+0000) 2024-03-06T15:09:06.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:06.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:06.164447+0000) 2024-03-06T15:09:07.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:07.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:07.164669+0000) 2024-03-06T15:09:08.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:08.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:08.164905+0000) 2024-03-06T15:09:09.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:09.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:09.165162+0000) 2024-03-06T15:09:10.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:10.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:10.165353+0000) 2024-03-06T15:09:11.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:11.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:11.165580+0000) 2024-03-06T15:09:11.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:11.619 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:12.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:12.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:12.165802+0000) 2024-03-06T15:09:13.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:13.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:13.165997+0000) 2024-03-06T15:09:14.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:14.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:14.166265+0000) 2024-03-06T15:09:15.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:15.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:15.166509+0000) 2024-03-06T15:09:16.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:16.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:16.166731+0000) 2024-03-06T15:09:17.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:17.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:17.166958+0000) 2024-03-06T15:09:18.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:18.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:18.167195+0000) 2024-03-06T15:09:19.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:19.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:19.167411+0000) 2024-03-06T15:09:20.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:20.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:20.167663+0000) 2024-03-06T15:09:21.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:21.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:21.167896+0000) 2024-03-06T15:09:22.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:22.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:22.168120+0000) 2024-03-06T15:09:23.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:23.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:23.168311+0000) 2024-03-06T15:09:24.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:24.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:24.168545+0000) 2024-03-06T15:09:25.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:25.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:25.168803+0000) 2024-03-06T15:09:26.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:26.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:26.169050+0000) 2024-03-06T15:09:27.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:27.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:27.169265+0000) 2024-03-06T15:09:28.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:28.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:28.169473+0000) 2024-03-06T15:09:29.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:29.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:29.169694+0000) 2024-03-06T15:09:30.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:30.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:30.169955+0000) 2024-03-06T15:09:31.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:31.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:31.170185+0000) 2024-03-06T15:09:32.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:32.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:32.170411+0000) 2024-03-06T15:09:33.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:33.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:33.170651+0000) 2024-03-06T15:09:34.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:34.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:34.170889+0000) 2024-03-06T15:09:35.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:35.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:35.171131+0000) 2024-03-06T15:09:36.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:36.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:36.171317+0000) 2024-03-06T15:09:37.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:37.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:37.171644+0000) 2024-03-06T15:09:38.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:38.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:38.171924+0000) 2024-03-06T15:09:39.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:39.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:39.172197+0000) 2024-03-06T15:09:40.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:40.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:40.172442+0000) 2024-03-06T15:09:41.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:41.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:41.172717+0000) 2024-03-06T15:09:41.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:41.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:09:42.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:42.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:42.172996+0000) 2024-03-06T15:09:43.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:43.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:43.173297+0000) 2024-03-06T15:09:44.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:44.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:44.173562+0000) 2024-03-06T15:09:45.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:45.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:45.173847+0000) 2024-03-06T15:09:46.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:46.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:46.174171+0000) 2024-03-06T15:09:47.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:47.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:47.174421+0000) 2024-03-06T15:09:48.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:48.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:48.174690+0000) 2024-03-06T15:09:49.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:49.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:49.174893+0000) 2024-03-06T15:09:50.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:50.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:50.175114+0000) 2024-03-06T15:09:51.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:51.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:51.175347+0000) 2024-03-06T15:09:52.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:52.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:52.175582+0000) 2024-03-06T15:09:53.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:53.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:53.175820+0000) 2024-03-06T15:09:54.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:54.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:54.176072+0000) 2024-03-06T15:09:55.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:55.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:55.176325+0000) 2024-03-06T15:09:56.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:56.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:56.176554+0000) 2024-03-06T15:09:57.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:57.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:57.176782+0000) 2024-03-06T15:09:58.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:58.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:58.177048+0000) 2024-03-06T15:09:59.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:09:59.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:09:59.177252+0000) 2024-03-06T15:10:00.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:00.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:00.177531+0000) 2024-03-06T15:10:01.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:01.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:01.177816+0000) 2024-03-06T15:10:02.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:02.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:02.178112+0000) 2024-03-06T15:10:03.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:03.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:03.178373+0000) 2024-03-06T15:10:04.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:04.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:04.178609+0000) 2024-03-06T15:10:05.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:05.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:05.178903+0000) 2024-03-06T15:10:06.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:06.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:06.179152+0000) 2024-03-06T15:10:07.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:07.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:07.179347+0000) 2024-03-06T15:10:08.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:08.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:08.179576+0000) 2024-03-06T15:10:09.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:09.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:09.179814+0000) 2024-03-06T15:10:10.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:10.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:10.180092+0000) 2024-03-06T15:10:11.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:11.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:11.180326+0000) 2024-03-06T15:10:11.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:11.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:12.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:12.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:12.180558+0000) 2024-03-06T15:10:13.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:13.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:13.180789+0000) 2024-03-06T15:10:14.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:14.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:14.180973+0000) 2024-03-06T15:10:15.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:15.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:15.181237+0000) 2024-03-06T15:10:16.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:16.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:16.181461+0000) 2024-03-06T15:10:17.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:17.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:17.181671+0000) 2024-03-06T15:10:18.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:18.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:18.181848+0000) 2024-03-06T15:10:19.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:19.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:19.182130+0000) 2024-03-06T15:10:20.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:20.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:20.182319+0000) 2024-03-06T15:10:21.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:21.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:21.182548+0000) 2024-03-06T15:10:22.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:22.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:22.182775+0000) 2024-03-06T15:10:23.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:23.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:23.183000+0000) 2024-03-06T15:10:24.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:24.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:24.183279+0000) 2024-03-06T15:10:25.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:25.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:25.183544+0000) 2024-03-06T15:10:26.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:26.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:26.183808+0000) 2024-03-06T15:10:27.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:27.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:27.184111+0000) 2024-03-06T15:10:28.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:28.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:28.184311+0000) 2024-03-06T15:10:29.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:29.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:29.184539+0000) 2024-03-06T15:10:30.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:30.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:30.184815+0000) 2024-03-06T15:10:31.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:31.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:31.185092+0000) 2024-03-06T15:10:32.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:32.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:32.185274+0000) 2024-03-06T15:10:33.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:33.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:33.185543+0000) 2024-03-06T15:10:34.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:34.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:34.185816+0000) 2024-03-06T15:10:35.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:35.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:35.186066+0000) 2024-03-06T15:10:36.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:36.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:36.186298+0000) 2024-03-06T15:10:37.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:37.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:37.186528+0000) 2024-03-06T15:10:38.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:38.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:38.186763+0000) 2024-03-06T15:10:39.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:39.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:39.186986+0000) 2024-03-06T15:10:40.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:40.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:40.187250+0000) 2024-03-06T15:10:41.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:41.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:41.187486+0000) 2024-03-06T15:10:42.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:42.106 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:10:42.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:42.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:42.187709+0000) 2024-03-06T15:10:43.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:43.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:43.187947+0000) 2024-03-06T15:10:44.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:44.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:44.188264+0000) 2024-03-06T15:10:45.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:45.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:45.188501+0000) 2024-03-06T15:10:46.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:46.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:46.188729+0000) 2024-03-06T15:10:47.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:47.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:47.188885+0000) 2024-03-06T15:10:48.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:48.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:48.189110+0000) 2024-03-06T15:10:49.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:49.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:49.189341+0000) 2024-03-06T15:10:50.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:50.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:50.189549+0000) 2024-03-06T15:10:51.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:51.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:51.189763+0000) 2024-03-06T15:10:52.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:52.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:52.189986+0000) 2024-03-06T15:10:53.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:53.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:53.190248+0000) 2024-03-06T15:10:54.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:54.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:54.190467+0000) 2024-03-06T15:10:55.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:55.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:55.190699+0000) 2024-03-06T15:10:56.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:56.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:56.190927+0000) 2024-03-06T15:10:57.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:57.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:57.191165+0000) 2024-03-06T15:10:58.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:58.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:58.191388+0000) 2024-03-06T15:10:59.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:10:59.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:10:59.191565+0000) 2024-03-06T15:11:00.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:00.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:00.191783+0000) 2024-03-06T15:11:01.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:01.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:01.191985+0000) 2024-03-06T15:11:02.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:02.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:02.192243+0000) 2024-03-06T15:11:03.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:03.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:03.192461+0000) 2024-03-06T15:11:04.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:04.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:04.192681+0000) 2024-03-06T15:11:05.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:05.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:05.192914+0000) 2024-03-06T15:11:06.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:06.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:06.193128+0000) 2024-03-06T15:11:07.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:07.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:07.193365+0000) 2024-03-06T15:11:08.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:08.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:08.193597+0000) 2024-03-06T15:11:09.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:09.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:09.193818+0000) 2024-03-06T15:11:10.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:10.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:10.194086+0000) 2024-03-06T15:11:11.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:11.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:11.194288+0000) 2024-03-06T15:11:12.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:12.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:12.194517+0000) 2024-03-06T15:11:12.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:12.295 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:13.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:13.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:13.194794+0000) 2024-03-06T15:11:14.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:14.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:14.195042+0000) 2024-03-06T15:11:15.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:15.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:15.195250+0000) 2024-03-06T15:11:16.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:16.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:16.195495+0000) 2024-03-06T15:11:17.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:17.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:17.195683+0000) 2024-03-06T15:11:18.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:18.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:18.195903+0000) 2024-03-06T15:11:19.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:19.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:19.196127+0000) 2024-03-06T15:11:20.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:20.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:20.196346+0000) 2024-03-06T15:11:21.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:21.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:21.196576+0000) 2024-03-06T15:11:22.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:22.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:22.196779+0000) 2024-03-06T15:11:23.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:23.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:23.196994+0000) 2024-03-06T15:11:24.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:24.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:24.197246+0000) 2024-03-06T15:11:25.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:25.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:25.197438+0000) 2024-03-06T15:11:26.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:26.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:26.197673+0000) 2024-03-06T15:11:27.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:27.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:27.197856+0000) 2024-03-06T15:11:28.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:28.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:28.198108+0000) 2024-03-06T15:11:29.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:29.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:29.198305+0000) 2024-03-06T15:11:30.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:30.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:30.198524+0000) 2024-03-06T15:11:31.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:31.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:31.198728+0000) 2024-03-06T15:11:32.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:32.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:32.198918+0000) 2024-03-06T15:11:33.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:33.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:33.199141+0000) 2024-03-06T15:11:34.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:34.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:34.199362+0000) 2024-03-06T15:11:35.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:35.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:35.199593+0000) 2024-03-06T15:11:36.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:36.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:36.199828+0000) 2024-03-06T15:11:37.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:37.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:37.200096+0000) 2024-03-06T15:11:38.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:38.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:38.200317+0000) 2024-03-06T15:11:39.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:39.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:39.200539+0000) 2024-03-06T15:11:40.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:40.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:40.200761+0000) 2024-03-06T15:11:41.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:41.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:41.200995+0000) 2024-03-06T15:11:42.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:42.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:42.201289+0000) 2024-03-06T15:11:42.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:42.448 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:11:43.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:43.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:43.201488+0000) 2024-03-06T15:11:44.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:44.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:44.201717+0000) 2024-03-06T15:11:45.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:45.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:45.201976+0000) 2024-03-06T15:11:46.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:46.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:46.202216+0000) 2024-03-06T15:11:47.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:47.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:47.202449+0000) 2024-03-06T15:11:48.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:48.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:48.202633+0000) 2024-03-06T15:11:49.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:49.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:49.202862+0000) 2024-03-06T15:11:50.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:50.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:50.203124+0000) 2024-03-06T15:11:51.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:51.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:51.203365+0000) 2024-03-06T15:11:52.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:52.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:52.203596+0000) 2024-03-06T15:11:53.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:53.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:53.203821+0000) 2024-03-06T15:11:54.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:54.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:54.204063+0000) 2024-03-06T15:11:55.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:55.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:55.204339+0000) 2024-03-06T15:11:56.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:56.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:56.204590+0000) 2024-03-06T15:11:57.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:57.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:57.204775+0000) 2024-03-06T15:11:58.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:58.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:58.204952+0000) 2024-03-06T15:11:59.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:11:59.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:11:59.205190+0000) 2024-03-06T15:12:00.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:00.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:00.205377+0000) 2024-03-06T15:12:01.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:01.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:01.205569+0000) 2024-03-06T15:12:02.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:02.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:02.205793+0000) 2024-03-06T15:12:03.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:03.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:03.206045+0000) 2024-03-06T15:12:04.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:04.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:04.206267+0000) 2024-03-06T15:12:05.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:05.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:05.206503+0000) 2024-03-06T15:12:06.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:06.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:06.206764+0000) 2024-03-06T15:12:07.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:07.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:07.206957+0000) 2024-03-06T15:12:08.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:08.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:08.207159+0000) 2024-03-06T15:12:09.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:09.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:09.207387+0000) 2024-03-06T15:12:10.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:10.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:10.207566+0000) 2024-03-06T15:12:11.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:11.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:11.207811+0000) 2024-03-06T15:12:12.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:12.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:12.208078+0000) 2024-03-06T15:12:12.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:12.603 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:13.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:13.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:13.208308+0000) 2024-03-06T15:12:14.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:14.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:14.208529+0000) 2024-03-06T15:12:15.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:15.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:15.208773+0000) 2024-03-06T15:12:16.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:16.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:16.209045+0000) 2024-03-06T15:12:17.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:17.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:17.209305+0000) 2024-03-06T15:12:18.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:18.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:18.209523+0000) 2024-03-06T15:12:19.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:19.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:19.209730+0000) 2024-03-06T15:12:20.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:20.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:20.209937+0000) 2024-03-06T15:12:21.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:21.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:21.210166+0000) 2024-03-06T15:12:22.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:22.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:22.210423+0000) 2024-03-06T15:12:23.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:23.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:23.210645+0000) 2024-03-06T15:12:24.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:24.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:24.210830+0000) 2024-03-06T15:12:25.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:25.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:25.211104+0000) 2024-03-06T15:12:26.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:26.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:26.211344+0000) 2024-03-06T15:12:27.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:27.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:27.211602+0000) 2024-03-06T15:12:28.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:28.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:28.211812+0000) 2024-03-06T15:12:29.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:29.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:29.212076+0000) 2024-03-06T15:12:30.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:30.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:30.212325+0000) 2024-03-06T15:12:31.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:31.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:31.212555+0000) 2024-03-06T15:12:32.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:32.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:32.212784+0000) 2024-03-06T15:12:33.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:33.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:33.213038+0000) 2024-03-06T15:12:34.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:34.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:34.213260+0000) 2024-03-06T15:12:35.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:35.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:35.213498+0000) 2024-03-06T15:12:36.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:36.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:36.213748+0000) 2024-03-06T15:12:37.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:37.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:37.213978+0000) 2024-03-06T15:12:38.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:38.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:38.214239+0000) 2024-03-06T15:12:39.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:39.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:39.214468+0000) 2024-03-06T15:12:40.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:40.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:40.214677+0000) 2024-03-06T15:12:41.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:41.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:41.214894+0000) 2024-03-06T15:12:42.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:42.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:42.215120+0000) 2024-03-06T15:12:42.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:42.824 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:12:43.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:43.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:43.215358+0000) 2024-03-06T15:12:44.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:44.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:44.215578+0000) 2024-03-06T15:12:45.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:45.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:45.215808+0000) 2024-03-06T15:12:46.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:46.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:46.216085+0000) 2024-03-06T15:12:47.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:47.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:47.216339+0000) 2024-03-06T15:12:48.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:48.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:48.216587+0000) 2024-03-06T15:12:49.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:49.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:49.216776+0000) 2024-03-06T15:12:50.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:50.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:50.216974+0000) 2024-03-06T15:12:51.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:51.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:51.217257+0000) 2024-03-06T15:12:52.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:52.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:52.217442+0000) 2024-03-06T15:12:53.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:53.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:53.217641+0000) 2024-03-06T15:12:54.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:54.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:54.217913+0000) 2024-03-06T15:12:55.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:55.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:55.218114+0000) 2024-03-06T15:12:56.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:56.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:56.218352+0000) 2024-03-06T15:12:57.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:57.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:57.218578+0000) 2024-03-06T15:12:58.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:58.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:58.218802+0000) 2024-03-06T15:12:59.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:12:59.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:12:59.219080+0000) 2024-03-06T15:13:00.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:00.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:00.219309+0000) 2024-03-06T15:13:01.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:01.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:01.219569+0000) 2024-03-06T15:13:02.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:02.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:02.219792+0000) 2024-03-06T15:13:03.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:03.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:03.220047+0000) 2024-03-06T15:13:04.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:04.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:04.220225+0000) 2024-03-06T15:13:05.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:05.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:05.220462+0000) 2024-03-06T15:13:06.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:06.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:06.220669+0000) 2024-03-06T15:13:07.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:07.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:07.220868+0000) 2024-03-06T15:13:08.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:08.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:08.221136+0000) 2024-03-06T15:13:09.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:09.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:09.221357+0000) 2024-03-06T15:13:10.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:10.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:10.221552+0000) 2024-03-06T15:13:11.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:11.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:11.221788+0000) 2024-03-06T15:13:12.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:12.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:12.222054+0000) 2024-03-06T15:13:12.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:12.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:13.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:13.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:13.222248+0000) 2024-03-06T15:13:14.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:14.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:14.222468+0000) 2024-03-06T15:13:15.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:15.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:15.222694+0000) 2024-03-06T15:13:16.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:16.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:16.222926+0000) 2024-03-06T15:13:17.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:17.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:17.223209+0000) 2024-03-06T15:13:18.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:18.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:18.223447+0000) 2024-03-06T15:13:19.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:19.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:19.223694+0000) 2024-03-06T15:13:20.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:20.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:20.223852+0000) 2024-03-06T15:13:21.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:21.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:21.224120+0000) 2024-03-06T15:13:22.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:22.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:22.224346+0000) 2024-03-06T15:13:23.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:23.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:23.224570+0000) 2024-03-06T15:13:24.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:24.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:24.224811+0000) 2024-03-06T15:13:25.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:25.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:25.225052+0000) 2024-03-06T15:13:26.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:26.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:26.225301+0000) 2024-03-06T15:13:27.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:27.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:27.225532+0000) 2024-03-06T15:13:28.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:28.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:28.225772+0000) 2024-03-06T15:13:29.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:29.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:29.226052+0000) 2024-03-06T15:13:30.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:30.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:30.226294+0000) 2024-03-06T15:13:31.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:31.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:31.226569+0000) 2024-03-06T15:13:32.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:32.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:32.226842+0000) 2024-03-06T15:13:33.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:33.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:33.227120+0000) 2024-03-06T15:13:34.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:34.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:34.227373+0000) 2024-03-06T15:13:35.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:35.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:35.227673+0000) 2024-03-06T15:13:36.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:36.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:36.227867+0000) 2024-03-06T15:13:37.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:37.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:37.228110+0000) 2024-03-06T15:13:38.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:38.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:38.228330+0000) 2024-03-06T15:13:39.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:39.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:39.228569+0000) 2024-03-06T15:13:40.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:40.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:40.228829+0000) 2024-03-06T15:13:41.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:41.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:41.229106+0000) 2024-03-06T15:13:42.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:42.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:42.229346+0000) 2024-03-06T15:13:43.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:43.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:13:43.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:43.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:43.229540+0000) 2024-03-06T15:13:44.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:44.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:44.229734+0000) 2024-03-06T15:13:45.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:45.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:45.229970+0000) 2024-03-06T15:13:46.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:46.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:46.230184+0000) 2024-03-06T15:13:47.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:47.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:47.230393+0000) 2024-03-06T15:13:48.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:48.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:48.230662+0000) 2024-03-06T15:13:49.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:49.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:49.230932+0000) 2024-03-06T15:13:50.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:50.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:50.231156+0000) 2024-03-06T15:13:51.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:51.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:51.231377+0000) 2024-03-06T15:13:52.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:52.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:52.231641+0000) 2024-03-06T15:13:53.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:53.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:53.231872+0000) 2024-03-06T15:13:54.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:54.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:54.232147+0000) 2024-03-06T15:13:55.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:55.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:55.232429+0000) 2024-03-06T15:13:56.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:56.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:56.232710+0000) 2024-03-06T15:13:57.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:57.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:57.232938+0000) 2024-03-06T15:13:58.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:58.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:58.233213+0000) 2024-03-06T15:13:59.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:13:59.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:13:59.233470+0000) 2024-03-06T15:14:00.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:00.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:00.233746+0000) 2024-03-06T15:14:01.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:01.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:01.234064+0000) 2024-03-06T15:14:02.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:02.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:02.234338+0000) 2024-03-06T15:14:03.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:03.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:03.234612+0000) 2024-03-06T15:14:04.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:04.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:04.234879+0000) 2024-03-06T15:14:05.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:05.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:05.235165+0000) 2024-03-06T15:14:06.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:06.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:06.235433+0000) 2024-03-06T15:14:07.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:07.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:07.235705+0000) 2024-03-06T15:14:08.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:08.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:08.235903+0000) 2024-03-06T15:14:09.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:09.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:09.236147+0000) 2024-03-06T15:14:10.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:10.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:10.236378+0000) 2024-03-06T15:14:11.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:11.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:11.236671+0000) 2024-03-06T15:14:12.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:12.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:12.236961+0000) 2024-03-06T15:14:13.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:13.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:13.237274+0000) 2024-03-06T15:14:13.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:13.360 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:14.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:14.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:14.237560+0000) 2024-03-06T15:14:15.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:15.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:15.237871+0000) 2024-03-06T15:14:16.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:16.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:16.238150+0000) 2024-03-06T15:14:17.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:17.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:17.238424+0000) 2024-03-06T15:14:18.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:18.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:18.238660+0000) 2024-03-06T15:14:19.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:19.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:19.238931+0000) 2024-03-06T15:14:20.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:20.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:20.239175+0000) 2024-03-06T15:14:21.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:21.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:21.239420+0000) 2024-03-06T15:14:22.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:22.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:22.239669+0000) 2024-03-06T15:14:23.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:23.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:23.239913+0000) 2024-03-06T15:14:24.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:24.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:24.240194+0000) 2024-03-06T15:14:25.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:25.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:25.240499+0000) 2024-03-06T15:14:26.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:26.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:26.240787+0000) 2024-03-06T15:14:27.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:27.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:27.241112+0000) 2024-03-06T15:14:28.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:28.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:28.241358+0000) 2024-03-06T15:14:29.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:29.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:29.241566+0000) 2024-03-06T15:14:30.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:30.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:30.241745+0000) 2024-03-06T15:14:31.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:31.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:31.241912+0000) 2024-03-06T15:14:32.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:32.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:32.242166+0000) 2024-03-06T15:14:33.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:33.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:33.242412+0000) 2024-03-06T15:14:34.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:34.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:34.242650+0000) 2024-03-06T15:14:35.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:35.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:35.242910+0000) 2024-03-06T15:14:36.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:36.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:36.243190+0000) 2024-03-06T15:14:37.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:37.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:37.243467+0000) 2024-03-06T15:14:38.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:38.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:38.243707+0000) 2024-03-06T15:14:39.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:39.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:39.243920+0000) 2024-03-06T15:14:40.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:40.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:40.244155+0000) 2024-03-06T15:14:41.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:41.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:41.244479+0000) 2024-03-06T15:14:42.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:42.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:42.244710+0000) 2024-03-06T15:14:43.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:43.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:43.244975+0000) 2024-03-06T15:14:43.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:43.593 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:14:44.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:44.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:44.245285+0000) 2024-03-06T15:14:45.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:45.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:45.245595+0000) 2024-03-06T15:14:46.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:46.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:46.245863+0000) 2024-03-06T15:14:47.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:47.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:47.246093+0000) 2024-03-06T15:14:48.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:48.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:48.246232+0000) 2024-03-06T15:14:49.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:49.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:49.246446+0000) 2024-03-06T15:14:50.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:50.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:50.246664+0000) 2024-03-06T15:14:51.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:51.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:51.246903+0000) 2024-03-06T15:14:52.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:52.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:52.247136+0000) 2024-03-06T15:14:53.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:53.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:53.247349+0000) 2024-03-06T15:14:54.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:54.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:54.247577+0000) 2024-03-06T15:14:55.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:55.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:55.247776+0000) 2024-03-06T15:14:56.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:56.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:56.247960+0000) 2024-03-06T15:14:57.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:57.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:57.248187+0000) 2024-03-06T15:14:58.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:58.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:58.248412+0000) 2024-03-06T15:14:59.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:14:59.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:14:59.248594+0000) 2024-03-06T15:15:00.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:00.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:00.248811+0000) 2024-03-06T15:15:01.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:01.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:01.249060+0000) 2024-03-06T15:15:02.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:02.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:02.249287+0000) 2024-03-06T15:15:03.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:03.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:03.249512+0000) 2024-03-06T15:15:04.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:04.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:04.249738+0000) 2024-03-06T15:15:05.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:05.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:05.249977+0000) 2024-03-06T15:15:06.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:06.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:06.250177+0000) 2024-03-06T15:15:07.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:07.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:07.250415+0000) 2024-03-06T15:15:08.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:08.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:08.250645+0000) 2024-03-06T15:15:09.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:09.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:09.250842+0000) 2024-03-06T15:15:10.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:10.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:10.251106+0000) 2024-03-06T15:15:11.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:11.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:11.251322+0000) 2024-03-06T15:15:12.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:12.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:12.251526+0000) 2024-03-06T15:15:13.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:13.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:13.251758+0000) 2024-03-06T15:15:13.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:13.801 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:14.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:14.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:14.251981+0000) 2024-03-06T15:15:15.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:15.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:15.252261+0000) 2024-03-06T15:15:16.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:16.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:16.252463+0000) 2024-03-06T15:15:17.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:17.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:17.252637+0000) 2024-03-06T15:15:18.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:18.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:18.252837+0000) 2024-03-06T15:15:19.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:19.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:19.253108+0000) 2024-03-06T15:15:20.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:20.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:20.253332+0000) 2024-03-06T15:15:21.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:21.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:21.253555+0000) 2024-03-06T15:15:22.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:22.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:22.253755+0000) 2024-03-06T15:15:23.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:23.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:23.253999+0000) 2024-03-06T15:15:24.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:24.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:24.254275+0000) 2024-03-06T15:15:25.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:25.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:25.254465+0000) 2024-03-06T15:15:26.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:26.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:26.254646+0000) 2024-03-06T15:15:27.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:27.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:27.254937+0000) 2024-03-06T15:15:28.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:28.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:28.255217+0000) 2024-03-06T15:15:29.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:29.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:29.255542+0000) 2024-03-06T15:15:30.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:30.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:30.255786+0000) 2024-03-06T15:15:31.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:31.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:31.256043+0000) 2024-03-06T15:15:32.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:32.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:32.256287+0000) 2024-03-06T15:15:33.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:33.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:33.256556+0000) 2024-03-06T15:15:34.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:34.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:34.256831+0000) 2024-03-06T15:15:35.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:35.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:35.257147+0000) 2024-03-06T15:15:36.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:36.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:36.257424+0000) 2024-03-06T15:15:37.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:37.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:37.257644+0000) 2024-03-06T15:15:38.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:38.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:38.257851+0000) 2024-03-06T15:15:39.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:39.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:39.258119+0000) 2024-03-06T15:15:40.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:40.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:40.258332+0000) 2024-03-06T15:15:41.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:41.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:41.258558+0000) 2024-03-06T15:15:42.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:42.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:42.258776+0000) 2024-03-06T15:15:43.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:43.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:43.259051+0000) 2024-03-06T15:15:43.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:43.955 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:15:44.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:44.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:44.259316+0000) 2024-03-06T15:15:45.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:45.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:45.259581+0000) 2024-03-06T15:15:46.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:46.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:46.259806+0000) 2024-03-06T15:15:47.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:47.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:47.260116+0000) 2024-03-06T15:15:48.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:48.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:48.260358+0000) 2024-03-06T15:15:49.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:49.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:49.260592+0000) 2024-03-06T15:15:50.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:50.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:50.260822+0000) 2024-03-06T15:15:51.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:51.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:51.261122+0000) 2024-03-06T15:15:52.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:52.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:52.261376+0000) 2024-03-06T15:15:53.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:53.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:53.261650+0000) 2024-03-06T15:15:54.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:54.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:54.261915+0000) 2024-03-06T15:15:55.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:55.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:55.262195+0000) 2024-03-06T15:15:56.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:56.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:56.262474+0000) 2024-03-06T15:15:57.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:57.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:57.262707+0000) 2024-03-06T15:15:58.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:58.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:58.262903+0000) 2024-03-06T15:15:59.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:15:59.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:15:59.263132+0000) 2024-03-06T15:16:00.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:00.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:00.263349+0000) 2024-03-06T15:16:01.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:01.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:01.263590+0000) 2024-03-06T15:16:02.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:02.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:02.263820+0000) 2024-03-06T15:16:03.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:03.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:03.264140+0000) 2024-03-06T15:16:04.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:04.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:04.264346+0000) 2024-03-06T15:16:05.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:05.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:05.264577+0000) 2024-03-06T15:16:06.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:06.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:06.264844+0000) 2024-03-06T15:16:07.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:07.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:07.265148+0000) 2024-03-06T15:16:08.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:08.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:08.265388+0000) 2024-03-06T15:16:09.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:09.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:09.265654+0000) 2024-03-06T15:16:10.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:10.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:10.265934+0000) 2024-03-06T15:16:11.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:11.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:11.266224+0000) 2024-03-06T15:16:12.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:12.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:12.266443+0000) 2024-03-06T15:16:13.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:13.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:13.266653+0000) 2024-03-06T15:16:14.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:14.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:14.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:14.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:14.266871+0000) 2024-03-06T15:16:15.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:15.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:15.267118+0000) 2024-03-06T15:16:16.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:16.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:16.267341+0000) 2024-03-06T15:16:17.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:17.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:17.267614+0000) 2024-03-06T15:16:18.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:18.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:18.267894+0000) 2024-03-06T15:16:19.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:19.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:19.268180+0000) 2024-03-06T15:16:20.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:20.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:20.268455+0000) 2024-03-06T15:16:21.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:21.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:21.268726+0000) 2024-03-06T15:16:22.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:22.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:22.268999+0000) 2024-03-06T15:16:23.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:23.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:23.269304+0000) 2024-03-06T15:16:24.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:24.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:24.269522+0000) 2024-03-06T15:16:25.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:25.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:25.269797+0000) 2024-03-06T15:16:26.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:26.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:26.270120+0000) 2024-03-06T15:16:27.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:27.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:27.270378+0000) 2024-03-06T15:16:28.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:28.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:28.270616+0000) 2024-03-06T15:16:29.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:29.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:29.270820+0000) 2024-03-06T15:16:30.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:30.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:30.271036+0000) 2024-03-06T15:16:31.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:31.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:31.271281+0000) 2024-03-06T15:16:32.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:32.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:32.271517+0000) 2024-03-06T15:16:33.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:33.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:33.271712+0000) 2024-03-06T15:16:34.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:34.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:34.271896+0000) 2024-03-06T15:16:35.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:35.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:35.272135+0000) 2024-03-06T15:16:36.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:36.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:36.272269+0000) 2024-03-06T15:16:37.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:37.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:37.272438+0000) 2024-03-06T15:16:38.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:38.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:38.272653+0000) 2024-03-06T15:16:39.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:39.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:39.272885+0000) 2024-03-06T15:16:40.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:40.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:40.273115+0000) 2024-03-06T15:16:41.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:41.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:41.273349+0000) 2024-03-06T15:16:42.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:42.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:42.273565+0000) 2024-03-06T15:16:43.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:43.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:43.273757+0000) 2024-03-06T15:16:44.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:44.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:44.273975+0000) 2024-03-06T15:16:44.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:44.370 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:16:45.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:45.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:45.274247+0000) 2024-03-06T15:16:46.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:46.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:46.274509+0000) 2024-03-06T15:16:47.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:47.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:47.274789+0000) 2024-03-06T15:16:48.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:48.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:48.275065+0000) 2024-03-06T15:16:49.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:49.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:49.275263+0000) 2024-03-06T15:16:50.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:50.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:50.275481+0000) 2024-03-06T15:16:51.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:51.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:51.275757+0000) 2024-03-06T15:16:52.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:52.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:52.275970+0000) 2024-03-06T15:16:53.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:53.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:53.276202+0000) 2024-03-06T15:16:54.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:54.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:54.276389+0000) 2024-03-06T15:16:55.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:55.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:55.276615+0000) 2024-03-06T15:16:56.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:56.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:56.276814+0000) 2024-03-06T15:16:57.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:57.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:57.277084+0000) 2024-03-06T15:16:58.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:58.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:58.277297+0000) 2024-03-06T15:16:59.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:16:59.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:16:59.277573+0000) 2024-03-06T15:17:00.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:00.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:00.277767+0000) 2024-03-06T15:17:01.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:01.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:01.278055+0000) 2024-03-06T15:17:02.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:02.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:02.278306+0000) 2024-03-06T15:17:03.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:03.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:03.278595+0000) 2024-03-06T15:17:04.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:04.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:04.278799+0000) 2024-03-06T15:17:05.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:05.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:05.279130+0000) 2024-03-06T15:17:06.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:06.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:06.279340+0000) 2024-03-06T15:17:07.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:07.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:07.279568+0000) 2024-03-06T15:17:08.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:08.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:08.279791+0000) 2024-03-06T15:17:09.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:09.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:09.280064+0000) 2024-03-06T15:17:10.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:10.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:10.280255+0000) 2024-03-06T15:17:11.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:11.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:11.280478+0000) 2024-03-06T15:17:12.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:12.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:12.280722+0000) 2024-03-06T15:17:13.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:13.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:13.280948+0000) 2024-03-06T15:17:14.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:14.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:14.281171+0000) 2024-03-06T15:17:14.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:14.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:15.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:15.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:15.281428+0000) 2024-03-06T15:17:16.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:16.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:16.281661+0000) 2024-03-06T15:17:17.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:17.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:17.281893+0000) 2024-03-06T15:17:18.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:18.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:18.282134+0000) 2024-03-06T15:17:19.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:19.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:19.282345+0000) 2024-03-06T15:17:20.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:20.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:20.282607+0000) 2024-03-06T15:17:21.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:21.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:21.282862+0000) 2024-03-06T15:17:22.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:22.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:22.283147+0000) 2024-03-06T15:17:23.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:23.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:23.283393+0000) 2024-03-06T15:17:24.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:24.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:24.283634+0000) 2024-03-06T15:17:25.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:25.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:25.283891+0000) 2024-03-06T15:17:26.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:26.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:26.284133+0000) 2024-03-06T15:17:27.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:27.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:27.284390+0000) 2024-03-06T15:17:28.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:28.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:28.284628+0000) 2024-03-06T15:17:29.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:29.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:29.284876+0000) 2024-03-06T15:17:30.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:30.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:30.285122+0000) 2024-03-06T15:17:31.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:31.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:31.285379+0000) 2024-03-06T15:17:32.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:32.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:32.285647+0000) 2024-03-06T15:17:33.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:33.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:33.285901+0000) 2024-03-06T15:17:34.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:34.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:34.286137+0000) 2024-03-06T15:17:35.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:35.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:35.286370+0000) 2024-03-06T15:17:36.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:36.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:36.286643+0000) 2024-03-06T15:17:37.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:37.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:37.286943+0000) 2024-03-06T15:17:38.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:38.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:38.287190+0000) 2024-03-06T15:17:39.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:39.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:39.287420+0000) 2024-03-06T15:17:40.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:40.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:40.287650+0000) 2024-03-06T15:17:41.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:41.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:41.287855+0000) 2024-03-06T15:17:42.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:42.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:42.288148+0000) 2024-03-06T15:17:43.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:43.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:43.288343+0000) 2024-03-06T15:17:44.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:44.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:44.288558+0000) 2024-03-06T15:17:44.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:44.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:17:45.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:45.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:45.288788+0000) 2024-03-06T15:17:46.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:46.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:46.289112+0000) 2024-03-06T15:17:47.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:47.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:47.289348+0000) 2024-03-06T15:17:48.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:48.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:48.289526+0000) 2024-03-06T15:17:49.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:49.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:49.289793+0000) 2024-03-06T15:17:50.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:50.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:50.290111+0000) 2024-03-06T15:17:51.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:51.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:51.290396+0000) 2024-03-06T15:17:52.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:52.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:52.290702+0000) 2024-03-06T15:17:53.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:53.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:53.290960+0000) 2024-03-06T15:17:54.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:54.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:54.291221+0000) 2024-03-06T15:17:55.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:55.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:55.291521+0000) 2024-03-06T15:17:56.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:56.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:56.291800+0000) 2024-03-06T15:17:57.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:57.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:57.292141+0000) 2024-03-06T15:17:58.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:58.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:58.292471+0000) 2024-03-06T15:17:59.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:17:59.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:17:59.292760+0000) 2024-03-06T15:18:00.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:00.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:00.293078+0000) 2024-03-06T15:18:01.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:01.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:01.293361+0000) 2024-03-06T15:18:02.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:02.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:02.293652+0000) 2024-03-06T15:18:03.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:03.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:03.293910+0000) 2024-03-06T15:18:04.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:04.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:04.294146+0000) 2024-03-06T15:18:05.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:05.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:05.294389+0000) 2024-03-06T15:18:06.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:06.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:06.294605+0000) 2024-03-06T15:18:07.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:07.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:07.294840+0000) 2024-03-06T15:18:08.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:08.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:08.295110+0000) 2024-03-06T15:18:09.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:09.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:09.295337+0000) 2024-03-06T15:18:10.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:10.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:10.295618+0000) 2024-03-06T15:18:11.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:11.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:11.295903+0000) 2024-03-06T15:18:12.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:12.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:12.296182+0000) 2024-03-06T15:18:13.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:13.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:13.296452+0000) 2024-03-06T15:18:14.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:14.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:14.296697+0000) 2024-03-06T15:18:14.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:15.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:15.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:15.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:15.296945+0000) 2024-03-06T15:18:16.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:16.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:16.297211+0000) 2024-03-06T15:18:17.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:17.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:17.297471+0000) 2024-03-06T15:18:18.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:18.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:18.297701+0000) 2024-03-06T15:18:19.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:19.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:19.297973+0000) 2024-03-06T15:18:20.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:20.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:20.298245+0000) 2024-03-06T15:18:21.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:21.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:21.298499+0000) 2024-03-06T15:18:22.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:22.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:22.298736+0000) 2024-03-06T15:18:23.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:23.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:23.298995+0000) 2024-03-06T15:18:24.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:24.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:24.299311+0000) 2024-03-06T15:18:25.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:25.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:25.299578+0000) 2024-03-06T15:18:26.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:26.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:26.299862+0000) 2024-03-06T15:18:27.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:27.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:27.300180+0000) 2024-03-06T15:18:28.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:28.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:28.300449+0000) 2024-03-06T15:18:29.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:29.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:29.300730+0000) 2024-03-06T15:18:30.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:30.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:30.301055+0000) 2024-03-06T15:18:31.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:31.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:31.301304+0000) 2024-03-06T15:18:32.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:32.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:32.301546+0000) 2024-03-06T15:18:33.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:33.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:33.301795+0000) 2024-03-06T15:18:34.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:34.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:34.302052+0000) 2024-03-06T15:18:35.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:35.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:35.302315+0000) 2024-03-06T15:18:36.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:36.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:36.302598+0000) 2024-03-06T15:18:37.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:37.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:37.302892+0000) 2024-03-06T15:18:38.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:38.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:38.303179+0000) 2024-03-06T15:18:39.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:39.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:39.303434+0000) 2024-03-06T15:18:40.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:40.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:40.303720+0000) 2024-03-06T15:18:41.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:41.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:41.303969+0000) 2024-03-06T15:18:42.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:42.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:42.304182+0000) 2024-03-06T15:18:43.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:43.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:43.304414+0000) 2024-03-06T15:18:44.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:44.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:44.304623+0000) 2024-03-06T15:18:45.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:45.182 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:18:45.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:45.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:45.304885+0000) 2024-03-06T15:18:46.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:46.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:46.305122+0000) 2024-03-06T15:18:47.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:47.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:47.305367+0000) 2024-03-06T15:18:48.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:48.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:48.305615+0000) 2024-03-06T15:18:49.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:49.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:49.305882+0000) 2024-03-06T15:18:50.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:50.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:50.306113+0000) 2024-03-06T15:18:51.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:51.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:51.306396+0000) 2024-03-06T15:18:52.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:52.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:52.306717+0000) 2024-03-06T15:18:53.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:53.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:53.306953+0000) 2024-03-06T15:18:54.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:54.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:54.307176+0000) 2024-03-06T15:18:55.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:55.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:55.307398+0000) 2024-03-06T15:18:56.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:56.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:56.307604+0000) 2024-03-06T15:18:57.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:57.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:57.307850+0000) 2024-03-06T15:18:58.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:58.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:58.308065+0000) 2024-03-06T15:18:59.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:18:59.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:18:59.308296+0000) 2024-03-06T15:19:00.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:00.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:00.308529+0000) 2024-03-06T15:19:01.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:01.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:01.308751+0000) 2024-03-06T15:19:02.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:02.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:02.308954+0000) 2024-03-06T15:19:03.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:03.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:03.309163+0000) 2024-03-06T15:19:04.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:04.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:04.309390+0000) 2024-03-06T15:19:05.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:05.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:05.309651+0000) 2024-03-06T15:19:06.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:06.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:06.309858+0000) 2024-03-06T15:19:07.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:07.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:07.310129+0000) 2024-03-06T15:19:08.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:08.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:08.310323+0000) 2024-03-06T15:19:09.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:09.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:09.310517+0000) 2024-03-06T15:19:10.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:10.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:10.310701+0000) 2024-03-06T15:19:11.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:11.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:11.310939+0000) 2024-03-06T15:19:12.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:12.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:12.311194+0000) 2024-03-06T15:19:13.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:13.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:13.311462+0000) 2024-03-06T15:19:14.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:14.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:14.311739+0000) 2024-03-06T15:19:15.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:15.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:15.311965+0000) 2024-03-06T15:19:15.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:15.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:16.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:16.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:16.312244+0000) 2024-03-06T15:19:17.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:17.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:17.312489+0000) 2024-03-06T15:19:18.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:18.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:18.312752+0000) 2024-03-06T15:19:19.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:19.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:19.312980+0000) 2024-03-06T15:19:20.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:20.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:20.313251+0000) 2024-03-06T15:19:21.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:21.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:21.313527+0000) 2024-03-06T15:19:22.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:22.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:22.313734+0000) 2024-03-06T15:19:23.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:23.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:23.313965+0000) 2024-03-06T15:19:24.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:24.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:24.314190+0000) 2024-03-06T15:19:25.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:25.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:25.314459+0000) 2024-03-06T15:19:26.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:26.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:26.314714+0000) 2024-03-06T15:19:27.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:27.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:27.314913+0000) 2024-03-06T15:19:28.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:28.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:28.315145+0000) 2024-03-06T15:19:29.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:29.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:29.315376+0000) 2024-03-06T15:19:30.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:30.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:30.315606+0000) 2024-03-06T15:19:31.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:31.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:31.315836+0000) 2024-03-06T15:19:32.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:32.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:32.316075+0000) 2024-03-06T15:19:33.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:33.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:33.316325+0000) 2024-03-06T15:19:34.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:34.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:34.316581+0000) 2024-03-06T15:19:35.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:35.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:35.316782+0000) 2024-03-06T15:19:36.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:36.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:36.317000+0000) 2024-03-06T15:19:37.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:37.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:37.317276+0000) 2024-03-06T15:19:38.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:38.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:38.317508+0000) 2024-03-06T15:19:39.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:39.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:39.317784+0000) 2024-03-06T15:19:40.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:40.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:40.318070+0000) 2024-03-06T15:19:41.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:41.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:41.318388+0000) 2024-03-06T15:19:42.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:42.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:42.318601+0000) 2024-03-06T15:19:43.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:43.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:43.318830+0000) 2024-03-06T15:19:44.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:44.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:44.319098+0000) 2024-03-06T15:19:45.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:45.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:45.319352+0000) 2024-03-06T15:19:45.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:45.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:19:46.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:46.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:46.319574+0000) 2024-03-06T15:19:47.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:47.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:47.319776+0000) 2024-03-06T15:19:48.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:48.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:48.319999+0000) 2024-03-06T15:19:49.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:49.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:49.320301+0000) 2024-03-06T15:19:50.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:50.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:50.320535+0000) 2024-03-06T15:19:51.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:51.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:51.320784+0000) 2024-03-06T15:19:52.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:52.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:52.320998+0000) 2024-03-06T15:19:53.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:53.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:53.321287+0000) 2024-03-06T15:19:54.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:54.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:54.321507+0000) 2024-03-06T15:19:55.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:55.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:55.321756+0000) 2024-03-06T15:19:56.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:56.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:56.321992+0000) 2024-03-06T15:19:57.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:57.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:57.322305+0000) 2024-03-06T15:19:58.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:58.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:58.322531+0000) 2024-03-06T15:19:59.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:19:59.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:19:59.322729+0000) 2024-03-06T15:20:00.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:00.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:00.322906+0000) 2024-03-06T15:20:01.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:01.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:01.323149+0000) 2024-03-06T15:20:02.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:02.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:02.323377+0000) 2024-03-06T15:20:03.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:03.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:03.323607+0000) 2024-03-06T15:20:04.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:04.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:04.323839+0000) 2024-03-06T15:20:05.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:05.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:05.324116+0000) 2024-03-06T15:20:06.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:06.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:06.324327+0000) 2024-03-06T15:20:07.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:07.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:07.324522+0000) 2024-03-06T15:20:08.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:08.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:08.324759+0000) 2024-03-06T15:20:09.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:09.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:09.324981+0000) 2024-03-06T15:20:10.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:10.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:10.325221+0000) 2024-03-06T15:20:11.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:11.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:11.325451+0000) 2024-03-06T15:20:12.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:12.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:12.325726+0000) 2024-03-06T15:20:13.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:13.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:13.326049+0000) 2024-03-06T15:20:14.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:14.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:14.326346+0000) 2024-03-06T15:20:15.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:15.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:15.326640+0000) 2024-03-06T15:20:15.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:15.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:16.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:16.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:16.326902+0000) 2024-03-06T15:20:17.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:17.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:17.327148+0000) 2024-03-06T15:20:18.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:18.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:18.327371+0000) 2024-03-06T15:20:19.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:19.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:19.327592+0000) 2024-03-06T15:20:20.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:20.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:20.327827+0000) 2024-03-06T15:20:21.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:21.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:21.328056+0000) 2024-03-06T15:20:22.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:22.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:22.328334+0000) 2024-03-06T15:20:23.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:23.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:23.328598+0000) 2024-03-06T15:20:24.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:24.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:24.328795+0000) 2024-03-06T15:20:25.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:25.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:25.329059+0000) 2024-03-06T15:20:26.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:26.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:26.329284+0000) 2024-03-06T15:20:27.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:27.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:27.329523+0000) 2024-03-06T15:20:28.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:28.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:28.329747+0000) 2024-03-06T15:20:29.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:29.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:29.329947+0000) 2024-03-06T15:20:30.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:30.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:30.330173+0000) 2024-03-06T15:20:31.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:31.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:31.330410+0000) 2024-03-06T15:20:32.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:32.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:32.330624+0000) 2024-03-06T15:20:33.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:33.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:33.330849+0000) 2024-03-06T15:20:34.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:34.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:34.331117+0000) 2024-03-06T15:20:35.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:35.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:35.331350+0000) 2024-03-06T15:20:36.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:36.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:36.331573+0000) 2024-03-06T15:20:37.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:37.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:37.331761+0000) 2024-03-06T15:20:38.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:38.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:38.331950+0000) 2024-03-06T15:20:39.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:39.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:39.332192+0000) 2024-03-06T15:20:40.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:40.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:40.332386+0000) 2024-03-06T15:20:41.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:41.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:41.332592+0000) 2024-03-06T15:20:42.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:42.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:42.332805+0000) 2024-03-06T15:20:43.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:43.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:43.332995+0000) 2024-03-06T15:20:44.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:44.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:44.333278+0000) 2024-03-06T15:20:45.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:45.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:45.333575+0000) 2024-03-06T15:20:45.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:45.976 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:20:46.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:46.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:46.333838+0000) 2024-03-06T15:20:47.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:47.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:47.334129+0000) 2024-03-06T15:20:48.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:48.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:48.334385+0000) 2024-03-06T15:20:49.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:49.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:49.334579+0000) 2024-03-06T15:20:50.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:50.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:50.334790+0000) 2024-03-06T15:20:51.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:51.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:51.334958+0000) 2024-03-06T15:20:52.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:52.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:52.335146+0000) 2024-03-06T15:20:53.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:53.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:53.335371+0000) 2024-03-06T15:20:54.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:54.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:54.335613+0000) 2024-03-06T15:20:55.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:55.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:55.335865+0000) 2024-03-06T15:20:56.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:56.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:56.336121+0000) 2024-03-06T15:20:57.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:57.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:57.336372+0000) 2024-03-06T15:20:58.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:58.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:58.336594+0000) 2024-03-06T15:20:59.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:20:59.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:20:59.336837+0000) 2024-03-06T15:21:00.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:00.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:00.337113+0000) 2024-03-06T15:21:01.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:01.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:01.337368+0000) 2024-03-06T15:21:02.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:02.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:02.337609+0000) 2024-03-06T15:21:03.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:03.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:03.337928+0000) 2024-03-06T15:21:04.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:04.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:04.338161+0000) 2024-03-06T15:21:05.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:05.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:05.338379+0000) 2024-03-06T15:21:06.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:06.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:06.338618+0000) 2024-03-06T15:21:07.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:07.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:07.338868+0000) 2024-03-06T15:21:08.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:08.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:08.339133+0000) 2024-03-06T15:21:09.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:09.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:09.339347+0000) 2024-03-06T15:21:10.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:10.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:10.339752+0000) 2024-03-06T15:21:11.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:11.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:11.340048+0000) 2024-03-06T15:21:12.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:12.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:12.340317+0000) 2024-03-06T15:21:13.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:13.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:13.340574+0000) 2024-03-06T15:21:14.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:14.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:14.340842+0000) 2024-03-06T15:21:15.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:15.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:15.341163+0000) 2024-03-06T15:21:16.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:16.195 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:16.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:16.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:16.341407+0000) 2024-03-06T15:21:17.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:17.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:17.341635+0000) 2024-03-06T15:21:18.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:18.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:18.341862+0000) 2024-03-06T15:21:19.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:19.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:19.342130+0000) 2024-03-06T15:21:20.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:20.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:20.342322+0000) 2024-03-06T15:21:21.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:21.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:21.342537+0000) 2024-03-06T15:21:22.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:22.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:22.342741+0000) 2024-03-06T15:21:23.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:23.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:23.342979+0000) 2024-03-06T15:21:24.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:24.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:24.343261+0000) 2024-03-06T15:21:25.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:25.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:25.343457+0000) 2024-03-06T15:21:26.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:26.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:26.343687+0000) 2024-03-06T15:21:27.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:27.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:27.343917+0000) 2024-03-06T15:21:28.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:28.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:28.344146+0000) 2024-03-06T15:21:29.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:29.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:29.344407+0000) 2024-03-06T15:21:30.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:30.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:30.344659+0000) 2024-03-06T15:21:31.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:31.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:31.344885+0000) 2024-03-06T15:21:32.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:32.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:32.345108+0000) 2024-03-06T15:21:33.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:33.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:33.345313+0000) 2024-03-06T15:21:34.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:34.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:34.345499+0000) 2024-03-06T15:21:35.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:35.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:35.345682+0000) 2024-03-06T15:21:36.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:36.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:36.345907+0000) 2024-03-06T15:21:37.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:37.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:37.346173+0000) 2024-03-06T15:21:38.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:38.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:38.346424+0000) 2024-03-06T15:21:39.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:39.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:39.346640+0000) 2024-03-06T15:21:40.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:40.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:40.346859+0000) 2024-03-06T15:21:41.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:41.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:41.347172+0000) 2024-03-06T15:21:42.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:42.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:42.347440+0000) 2024-03-06T15:21:43.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:43.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:43.347699+0000) 2024-03-06T15:21:44.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:44.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:44.347960+0000) 2024-03-06T15:21:45.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:45.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:45.348194+0000) 2024-03-06T15:21:46.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:46.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:46.348414+0000) 2024-03-06T15:21:46.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:46.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:21:47.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:47.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:47.348640+0000) 2024-03-06T15:21:48.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:48.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:48.348909+0000) 2024-03-06T15:21:49.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:49.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:49.349118+0000) 2024-03-06T15:21:50.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:50.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:50.349304+0000) 2024-03-06T15:21:51.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:51.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:51.349527+0000) 2024-03-06T15:21:52.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:52.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:52.349746+0000) 2024-03-06T15:21:53.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:53.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:53.350048+0000) 2024-03-06T15:21:54.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:54.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:54.350268+0000) 2024-03-06T15:21:55.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:55.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:55.350465+0000) 2024-03-06T15:21:56.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:56.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:56.350682+0000) 2024-03-06T15:21:57.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:57.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:57.350908+0000) 2024-03-06T15:21:58.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:58.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:58.351189+0000) 2024-03-06T15:21:59.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:21:59.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:21:59.351484+0000) 2024-03-06T15:22:00.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:00.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:00.351767+0000) 2024-03-06T15:22:01.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:01.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:01.352079+0000) 2024-03-06T15:22:02.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:02.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:02.352355+0000) 2024-03-06T15:22:03.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:03.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:03.352621+0000) 2024-03-06T15:22:04.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:04.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:04.352889+0000) 2024-03-06T15:22:05.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:05.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:05.353183+0000) 2024-03-06T15:22:06.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:06.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:06.353463+0000) 2024-03-06T15:22:07.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:07.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:07.353749+0000) 2024-03-06T15:22:08.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:08.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:08.354047+0000) 2024-03-06T15:22:09.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:09.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:09.354375+0000) 2024-03-06T15:22:10.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:10.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:10.354607+0000) 2024-03-06T15:22:11.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:11.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:11.354849+0000) 2024-03-06T15:22:12.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:12.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:12.355114+0000) 2024-03-06T15:22:13.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:13.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:13.355319+0000) 2024-03-06T15:22:14.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:14.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:14.355526+0000) 2024-03-06T15:22:15.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:15.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:15.355765+0000) 2024-03-06T15:22:16.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:16.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:16.355980+0000) 2024-03-06T15:22:16.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:16.637 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:17.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:17.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:17.356241+0000) 2024-03-06T15:22:18.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:18.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:18.356448+0000) 2024-03-06T15:22:19.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:19.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:19.356678+0000) 2024-03-06T15:22:20.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:20.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:20.356901+0000) 2024-03-06T15:22:21.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:21.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:21.357134+0000) 2024-03-06T15:22:22.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:22.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:22.357391+0000) 2024-03-06T15:22:23.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:23.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:23.357649+0000) 2024-03-06T15:22:24.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:24.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:24.357873+0000) 2024-03-06T15:22:25.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:25.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:25.358114+0000) 2024-03-06T15:22:26.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:26.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:26.358329+0000) 2024-03-06T15:22:27.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:27.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:27.358473+0000) 2024-03-06T15:22:28.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:28.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:28.358706+0000) 2024-03-06T15:22:29.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:29.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:29.358938+0000) 2024-03-06T15:22:30.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:30.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:30.359159+0000) 2024-03-06T15:22:31.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:31.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:31.359322+0000) 2024-03-06T15:22:32.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:32.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:32.359577+0000) 2024-03-06T15:22:33.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:33.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:33.359833+0000) 2024-03-06T15:22:34.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:34.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:34.360099+0000) 2024-03-06T15:22:35.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:35.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:35.360370+0000) 2024-03-06T15:22:36.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:36.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:36.360612+0000) 2024-03-06T15:22:37.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:37.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:37.360836+0000) 2024-03-06T15:22:38.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:38.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:38.361105+0000) 2024-03-06T15:22:39.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:39.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:39.361303+0000) 2024-03-06T15:22:40.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:40.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:40.361530+0000) 2024-03-06T15:22:41.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:41.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:41.361763+0000) 2024-03-06T15:22:42.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:42.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:42.361965+0000) 2024-03-06T15:22:43.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:43.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:43.362165+0000) 2024-03-06T15:22:44.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:44.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:44.362353+0000) 2024-03-06T15:22:45.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:45.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:45.362558+0000) 2024-03-06T15:22:46.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:46.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:46.362801+0000) 2024-03-06T15:22:46.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:46.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:22:47.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:47.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:47.363058+0000) 2024-03-06T15:22:48.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:48.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:48.363262+0000) 2024-03-06T15:22:49.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:49.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:49.363490+0000) 2024-03-06T15:22:50.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:50.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:50.363688+0000) 2024-03-06T15:22:51.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:51.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:51.363945+0000) 2024-03-06T15:22:52.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:52.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:52.364189+0000) 2024-03-06T15:22:53.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:53.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:53.364430+0000) 2024-03-06T15:22:54.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:54.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:54.364675+0000) 2024-03-06T15:22:55.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:55.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:55.364972+0000) 2024-03-06T15:22:56.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:56.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:56.365304+0000) 2024-03-06T15:22:57.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:57.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:57.365640+0000) 2024-03-06T15:22:58.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:58.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:58.365929+0000) 2024-03-06T15:22:59.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:22:59.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:22:59.366210+0000) 2024-03-06T15:23:00.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:00.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:00.366474+0000) 2024-03-06T15:23:01.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:01.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:01.366752+0000) 2024-03-06T15:23:02.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:02.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:02.367062+0000) 2024-03-06T15:23:03.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:03.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:03.367324+0000) 2024-03-06T15:23:04.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:04.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:04.367602+0000) 2024-03-06T15:23:05.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:05.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:05.367871+0000) 2024-03-06T15:23:06.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:06.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:06.368111+0000) 2024-03-06T15:23:07.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:07.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:07.368324+0000) 2024-03-06T15:23:08.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:08.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:08.368605+0000) 2024-03-06T15:23:09.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:09.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:09.368821+0000) 2024-03-06T15:23:10.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:10.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:10.368980+0000) 2024-03-06T15:23:11.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:11.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:11.369201+0000) 2024-03-06T15:23:12.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:12.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:12.369432+0000) 2024-03-06T15:23:13.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:13.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:13.369685+0000) 2024-03-06T15:23:14.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:14.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:14.369907+0000) 2024-03-06T15:23:15.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:15.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:15.370148+0000) 2024-03-06T15:23:16.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:16.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:16.370333+0000) 2024-03-06T15:23:16.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:16.910 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:17.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:17.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:17.370531+0000) 2024-03-06T15:23:18.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:18.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:18.370786+0000) 2024-03-06T15:23:19.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:19.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:19.370936+0000) 2024-03-06T15:23:20.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:20.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:20.371147+0000) 2024-03-06T15:23:21.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:21.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:21.371331+0000) 2024-03-06T15:23:22.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:22.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:22.371554+0000) 2024-03-06T15:23:23.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:23.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:23.371754+0000) 2024-03-06T15:23:24.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:24.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:24.371978+0000) 2024-03-06T15:23:25.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:25.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:25.372266+0000) 2024-03-06T15:23:26.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:26.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:26.372454+0000) 2024-03-06T15:23:27.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:27.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:27.372651+0000) 2024-03-06T15:23:28.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:28.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:28.372865+0000) 2024-03-06T15:23:29.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:29.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:29.373151+0000) 2024-03-06T15:23:30.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:30.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:30.373389+0000) 2024-03-06T15:23:31.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:31.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:31.373623+0000) 2024-03-06T15:23:32.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:32.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:32.373845+0000) 2024-03-06T15:23:33.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:33.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:33.374119+0000) 2024-03-06T15:23:34.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:34.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:34.374308+0000) 2024-03-06T15:23:35.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:35.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:35.374533+0000) 2024-03-06T15:23:36.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:36.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:36.374750+0000) 2024-03-06T15:23:37.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:37.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:37.374970+0000) 2024-03-06T15:23:38.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:38.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:38.375182+0000) 2024-03-06T15:23:39.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:39.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:39.375388+0000) 2024-03-06T15:23:40.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:40.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:40.375590+0000) 2024-03-06T15:23:41.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:41.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:41.375821+0000) 2024-03-06T15:23:42.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:42.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:42.376044+0000) 2024-03-06T15:23:43.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:43.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:43.376278+0000) 2024-03-06T15:23:44.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:44.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:44.376474+0000) 2024-03-06T15:23:45.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:45.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:45.376709+0000) 2024-03-06T15:23:46.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:46.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:46.376941+0000) 2024-03-06T15:23:47.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:47.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:23:47.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:47.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:47.377158+0000) 2024-03-06T15:23:48.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:48.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:48.377367+0000) 2024-03-06T15:23:49.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:49.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:49.377608+0000) 2024-03-06T15:23:50.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:50.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:50.378043+0000) 2024-03-06T15:23:51.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:51.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:51.378235+0000) 2024-03-06T15:23:52.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:52.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:52.378424+0000) 2024-03-06T15:23:53.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:53.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:53.378645+0000) 2024-03-06T15:23:54.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:54.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:54.378880+0000) 2024-03-06T15:23:55.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:55.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:55.379134+0000) 2024-03-06T15:23:56.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:56.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:56.379362+0000) 2024-03-06T15:23:57.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:57.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:57.379606+0000) 2024-03-06T15:23:58.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:58.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:58.379849+0000) 2024-03-06T15:23:59.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:23:59.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:23:59.380123+0000) 2024-03-06T15:24:00.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:00.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:00.380355+0000) 2024-03-06T15:24:01.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:01.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:01.380537+0000) 2024-03-06T15:24:02.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:02.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:02.380722+0000) 2024-03-06T15:24:03.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:03.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:03.380902+0000) 2024-03-06T15:24:04.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:04.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:04.381152+0000) 2024-03-06T15:24:05.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:05.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:05.381418+0000) 2024-03-06T15:24:06.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:06.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:06.381650+0000) 2024-03-06T15:24:07.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:07.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:07.381857+0000) 2024-03-06T15:24:08.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:08.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:08.382124+0000) 2024-03-06T15:24:09.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:09.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:09.382370+0000) 2024-03-06T15:24:10.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:10.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:10.382588+0000) 2024-03-06T15:24:11.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:11.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:11.382831+0000) 2024-03-06T15:24:12.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:12.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:12.383111+0000) 2024-03-06T15:24:13.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:13.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:13.383327+0000) 2024-03-06T15:24:14.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:14.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:14.383571+0000) 2024-03-06T15:24:15.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:15.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:15.383826+0000) 2024-03-06T15:24:16.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:16.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:16.384098+0000) 2024-03-06T15:24:17.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:17.252 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:17.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:17.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:17.384377+0000) 2024-03-06T15:24:18.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:18.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:18.384627+0000) 2024-03-06T15:24:19.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:19.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:19.384868+0000) 2024-03-06T15:24:20.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:20.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:20.385085+0000) 2024-03-06T15:24:21.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:21.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:21.385322+0000) 2024-03-06T15:24:22.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:22.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:22.385562+0000) 2024-03-06T15:24:23.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:23.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:23.385797+0000) 2024-03-06T15:24:24.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:24.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:24.386075+0000) 2024-03-06T15:24:25.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:25.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:25.386302+0000) 2024-03-06T15:24:26.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:26.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:26.386530+0000) 2024-03-06T15:24:27.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:27.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:27.386757+0000) 2024-03-06T15:24:28.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:28.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:28.386977+0000) 2024-03-06T15:24:29.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:29.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:29.387208+0000) 2024-03-06T15:24:30.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:30.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:30.387394+0000) 2024-03-06T15:24:31.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:31.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:31.387604+0000) 2024-03-06T15:24:32.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:32.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:32.387819+0000) 2024-03-06T15:24:33.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:33.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:33.388133+0000) 2024-03-06T15:24:34.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:34.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:34.388319+0000) 2024-03-06T15:24:35.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:35.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:35.388532+0000) 2024-03-06T15:24:36.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:36.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:36.388749+0000) 2024-03-06T15:24:37.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:37.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:37.388974+0000) 2024-03-06T15:24:38.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:38.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:38.389202+0000) 2024-03-06T15:24:39.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:39.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:39.389400+0000) 2024-03-06T15:24:40.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:40.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:40.389630+0000) 2024-03-06T15:24:41.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:41.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:41.389855+0000) 2024-03-06T15:24:42.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:42.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:42.390123+0000) 2024-03-06T15:24:43.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:43.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:43.390351+0000) 2024-03-06T15:24:44.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:44.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:44.390575+0000) 2024-03-06T15:24:45.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:45.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:45.390816+0000) 2024-03-06T15:24:46.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:46.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:46.390999+0000) 2024-03-06T15:24:47.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:47.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:47.391275+0000) 2024-03-06T15:24:47.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:47.464 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:24:48.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:48.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:48.391498+0000) 2024-03-06T15:24:49.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:49.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:49.391726+0000) 2024-03-06T15:24:50.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:50.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:50.391952+0000) 2024-03-06T15:24:51.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:51.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:51.392161+0000) 2024-03-06T15:24:52.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:52.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:52.392348+0000) 2024-03-06T15:24:53.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:53.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:53.392586+0000) 2024-03-06T15:24:54.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:54.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:54.392830+0000) 2024-03-06T15:24:55.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:55.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:55.393106+0000) 2024-03-06T15:24:56.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:56.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:56.393307+0000) 2024-03-06T15:24:57.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:57.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:57.393538+0000) 2024-03-06T15:24:58.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:58.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:58.393757+0000) 2024-03-06T15:24:59.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:24:59.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:24:59.393989+0000) 2024-03-06T15:25:00.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:00.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:00.394209+0000) 2024-03-06T15:25:01.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:01.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:01.394405+0000) 2024-03-06T15:25:02.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:02.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:02.394626+0000) 2024-03-06T15:25:03.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:03.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:03.394832+0000) 2024-03-06T15:25:04.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:04.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:04.395095+0000) 2024-03-06T15:25:05.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:05.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:05.395342+0000) 2024-03-06T15:25:06.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:06.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:06.395551+0000) 2024-03-06T15:25:07.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:07.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:07.395774+0000) 2024-03-06T15:25:08.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:08.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:08.396043+0000) 2024-03-06T15:25:09.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:09.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:09.396288+0000) 2024-03-06T15:25:10.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:10.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:10.396516+0000) 2024-03-06T15:25:11.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:11.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:11.396721+0000) 2024-03-06T15:25:12.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:12.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:12.396993+0000) 2024-03-06T15:25:13.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:13.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:13.397248+0000) 2024-03-06T15:25:14.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:14.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:14.397472+0000) 2024-03-06T15:25:15.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:15.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:15.397710+0000) 2024-03-06T15:25:16.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:16.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:16.397939+0000) 2024-03-06T15:25:17.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:17.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:17.398178+0000) 2024-03-06T15:25:17.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:17.677 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:18.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:18.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:18.398378+0000) 2024-03-06T15:25:19.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:19.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:19.398594+0000) 2024-03-06T15:25:20.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:20.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:20.398844+0000) 2024-03-06T15:25:21.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:21.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:21.399144+0000) 2024-03-06T15:25:22.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:22.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:22.399382+0000) 2024-03-06T15:25:23.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:23.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:23.399622+0000) 2024-03-06T15:25:24.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:24.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:24.399841+0000) 2024-03-06T15:25:25.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:25.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:25.400143+0000) 2024-03-06T15:25:26.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:26.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:26.400431+0000) 2024-03-06T15:25:27.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:27.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:27.400637+0000) 2024-03-06T15:25:28.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:28.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:28.400858+0000) 2024-03-06T15:25:29.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:29.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:29.401150+0000) 2024-03-06T15:25:30.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:30.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:30.401353+0000) 2024-03-06T15:25:31.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:31.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:31.401549+0000) 2024-03-06T15:25:32.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:32.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:32.401789+0000) 2024-03-06T15:25:33.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:33.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:33.402082+0000) 2024-03-06T15:25:34.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:34.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:34.402310+0000) 2024-03-06T15:25:35.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:35.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:35.402557+0000) 2024-03-06T15:25:36.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:36.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:36.402788+0000) 2024-03-06T15:25:37.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:37.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:37.403076+0000) 2024-03-06T15:25:38.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:38.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:38.403316+0000) 2024-03-06T15:25:39.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:39.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:39.403582+0000) 2024-03-06T15:25:40.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:40.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:40.403764+0000) 2024-03-06T15:25:41.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:41.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:41.403929+0000) 2024-03-06T15:25:42.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:42.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:42.404145+0000) 2024-03-06T15:25:43.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:43.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:43.404335+0000) 2024-03-06T15:25:44.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:44.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:44.404549+0000) 2024-03-06T15:25:45.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:45.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:45.404740+0000) 2024-03-06T15:25:46.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:46.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:46.404977+0000) 2024-03-06T15:25:47.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:47.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:47.405265+0000) 2024-03-06T15:25:47.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:47.855 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:25:48.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:48.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:48.405487+0000) 2024-03-06T15:25:49.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:49.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:49.405654+0000) 2024-03-06T15:25:50.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:50.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:50.405844+0000) 2024-03-06T15:25:51.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:51.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:51.406118+0000) 2024-03-06T15:25:52.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:52.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:52.406307+0000) 2024-03-06T15:25:53.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:53.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:53.406514+0000) 2024-03-06T15:25:54.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:54.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:54.406739+0000) 2024-03-06T15:25:55.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:55.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:55.406971+0000) 2024-03-06T15:25:56.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:56.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:56.407163+0000) 2024-03-06T15:25:57.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:57.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:57.407413+0000) 2024-03-06T15:25:58.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:58.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:58.407600+0000) 2024-03-06T15:25:59.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:25:59.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:25:59.407828+0000) 2024-03-06T15:26:00.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:00.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:00.408049+0000) 2024-03-06T15:26:01.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:01.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:01.408299+0000) 2024-03-06T15:26:02.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:02.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:02.408543+0000) 2024-03-06T15:26:03.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:03.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:03.408791+0000) 2024-03-06T15:26:04.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:04.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:04.409039+0000) 2024-03-06T15:26:05.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:05.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:05.409221+0000) 2024-03-06T15:26:06.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:06.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:06.409467+0000) 2024-03-06T15:26:07.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:07.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:07.409651+0000) 2024-03-06T15:26:08.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:08.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:08.409823+0000) 2024-03-06T15:26:09.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:09.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:09.410132+0000) 2024-03-06T15:26:10.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:10.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:10.410306+0000) 2024-03-06T15:26:11.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:11.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:11.410551+0000) 2024-03-06T15:26:12.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:12.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:12.410826+0000) 2024-03-06T15:26:13.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:13.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:13.411124+0000) 2024-03-06T15:26:14.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:14.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:14.411369+0000) 2024-03-06T15:26:15.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:15.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:15.411622+0000) 2024-03-06T15:26:16.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:16.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:16.411868+0000) 2024-03-06T15:26:17.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:17.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:17.412130+0000) 2024-03-06T15:26:18.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:18.052 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:18.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:18.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:18.412382+0000) 2024-03-06T15:26:19.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:19.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:19.412696+0000) 2024-03-06T15:26:20.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:20.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:20.412877+0000) 2024-03-06T15:26:21.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:21.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:21.413136+0000) 2024-03-06T15:26:22.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:22.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:22.413343+0000) 2024-03-06T15:26:23.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:23.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:23.413584+0000) 2024-03-06T15:26:24.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:24.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:24.413847+0000) 2024-03-06T15:26:25.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:25.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:25.414110+0000) 2024-03-06T15:26:26.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:26.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:26.414277+0000) 2024-03-06T15:26:27.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:27.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:27.414479+0000) 2024-03-06T15:26:28.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:28.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:28.414697+0000) 2024-03-06T15:26:29.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:29.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:29.414894+0000) 2024-03-06T15:26:30.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:30.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:30.415123+0000) 2024-03-06T15:26:31.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:31.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:31.415377+0000) 2024-03-06T15:26:32.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:32.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:32.415616+0000) 2024-03-06T15:26:33.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:33.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:33.415860+0000) 2024-03-06T15:26:34.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:34.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:34.416133+0000) 2024-03-06T15:26:35.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:35.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:35.416366+0000) 2024-03-06T15:26:36.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:36.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:36.416612+0000) 2024-03-06T15:26:37.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:37.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:37.416871+0000) 2024-03-06T15:26:38.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:38.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:38.417131+0000) 2024-03-06T15:26:39.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:39.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:39.417389+0000) 2024-03-06T15:26:40.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:40.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:40.417614+0000) 2024-03-06T15:26:41.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:41.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:41.417805+0000) 2024-03-06T15:26:42.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:42.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:42.418059+0000) 2024-03-06T15:26:43.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:43.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:43.418266+0000) 2024-03-06T15:26:44.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:44.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:44.418509+0000) 2024-03-06T15:26:45.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:45.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:45.418735+0000) 2024-03-06T15:26:46.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:46.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:46.418925+0000) 2024-03-06T15:26:47.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:47.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:47.419153+0000) 2024-03-06T15:26:48.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:48.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:26:48.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:48.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:48.419393+0000) 2024-03-06T15:26:49.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:49.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:49.419591+0000) 2024-03-06T15:26:50.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:50.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:50.419811+0000) 2024-03-06T15:26:51.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:51.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:51.420089+0000) 2024-03-06T15:26:52.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:52.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:52.420288+0000) 2024-03-06T15:26:53.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:53.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:53.420541+0000) 2024-03-06T15:26:54.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:54.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:54.420817+0000) 2024-03-06T15:26:55.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:55.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:55.421131+0000) 2024-03-06T15:26:56.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:56.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:56.421375+0000) 2024-03-06T15:26:57.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:57.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:57.421618+0000) 2024-03-06T15:26:58.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:58.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:58.421827+0000) 2024-03-06T15:26:59.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:26:59.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:26:59.422134+0000) 2024-03-06T15:27:00.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:00.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:00.422387+0000) 2024-03-06T15:27:01.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:01.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:01.422642+0000) 2024-03-06T15:27:02.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:02.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:02.422886+0000) 2024-03-06T15:27:03.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:03.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:03.423132+0000) 2024-03-06T15:27:04.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:04.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:04.423381+0000) 2024-03-06T15:27:05.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:05.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:05.423645+0000) 2024-03-06T15:27:06.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:06.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:06.423836+0000) 2024-03-06T15:27:07.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:07.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:07.424163+0000) 2024-03-06T15:27:08.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:08.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:08.424358+0000) 2024-03-06T15:27:09.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:09.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:09.424578+0000) 2024-03-06T15:27:10.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:10.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:10.424799+0000) 2024-03-06T15:27:11.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:11.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:11.425052+0000) 2024-03-06T15:27:12.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:12.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:12.425246+0000) 2024-03-06T15:27:13.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:13.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:13.425472+0000) 2024-03-06T15:27:14.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:14.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:14.425692+0000) 2024-03-06T15:27:15.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:15.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:15.425937+0000) 2024-03-06T15:27:16.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:16.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:16.426196+0000) 2024-03-06T15:27:17.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:17.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:17.426448+0000) 2024-03-06T15:27:18.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:18.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:18.426669+0000) 2024-03-06T15:27:18.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:18.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:19.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:19.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:19.426943+0000) 2024-03-06T15:27:20.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:20.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:20.427222+0000) 2024-03-06T15:27:21.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:21.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:21.427512+0000) 2024-03-06T15:27:22.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:22.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:22.427791+0000) 2024-03-06T15:27:23.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:23.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:23.427990+0000) 2024-03-06T15:27:24.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:24.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:24.428273+0000) 2024-03-06T15:27:25.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:25.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:25.428567+0000) 2024-03-06T15:27:26.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:26.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:26.428810+0000) 2024-03-06T15:27:27.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:27.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:27.429076+0000) 2024-03-06T15:27:28.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:28.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:28.429212+0000) 2024-03-06T15:27:29.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:29.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:29.429392+0000) 2024-03-06T15:27:30.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:30.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:30.429574+0000) 2024-03-06T15:27:31.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:31.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:31.429795+0000) 2024-03-06T15:27:32.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:32.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:32.430106+0000) 2024-03-06T15:27:33.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:33.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:33.430307+0000) 2024-03-06T15:27:34.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:34.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:34.430549+0000) 2024-03-06T15:27:35.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:35.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:35.430798+0000) 2024-03-06T15:27:36.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:36.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:36.431056+0000) 2024-03-06T15:27:37.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:37.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:37.431279+0000) 2024-03-06T15:27:38.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:38.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:38.431448+0000) 2024-03-06T15:27:39.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:39.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:39.431682+0000) 2024-03-06T15:27:40.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:40.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:40.431904+0000) 2024-03-06T15:27:41.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:41.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:41.432176+0000) 2024-03-06T15:27:42.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:42.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:42.432428+0000) 2024-03-06T15:27:43.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:43.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:43.432651+0000) 2024-03-06T15:27:44.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:44.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:44.432832+0000) 2024-03-06T15:27:45.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:45.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:45.433113+0000) 2024-03-06T15:27:46.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:46.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:46.433330+0000) 2024-03-06T15:27:47.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:47.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:47.433564+0000) 2024-03-06T15:27:48.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:48.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:48.433786+0000) 2024-03-06T15:27:48.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:48.672 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:27:49.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:49.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:49.434057+0000) 2024-03-06T15:27:50.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:50.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:50.434287+0000) 2024-03-06T15:27:51.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:51.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:51.434513+0000) 2024-03-06T15:27:52.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:52.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:52.434754+0000) 2024-03-06T15:27:53.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:53.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:53.434993+0000) 2024-03-06T15:27:54.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:54.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:54.435300+0000) 2024-03-06T15:27:55.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:55.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:55.435593+0000) 2024-03-06T15:27:56.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:56.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:56.435818+0000) 2024-03-06T15:27:57.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:57.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:57.436089+0000) 2024-03-06T15:27:58.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:58.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:58.436319+0000) 2024-03-06T15:27:59.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:27:59.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:27:59.436558+0000) 2024-03-06T15:28:00.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:00.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:00.436781+0000) 2024-03-06T15:28:01.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:01.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:01.437039+0000) 2024-03-06T15:28:02.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:02.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:02.437268+0000) 2024-03-06T15:28:03.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:03.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:03.437495+0000) 2024-03-06T15:28:04.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:04.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:04.437761+0000) 2024-03-06T15:28:05.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:05.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:05.438053+0000) 2024-03-06T15:28:06.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:06.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:06.438285+0000) 2024-03-06T15:28:07.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:07.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:07.438528+0000) 2024-03-06T15:28:08.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:08.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:08.438761+0000) 2024-03-06T15:28:09.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:09.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:09.438998+0000) 2024-03-06T15:28:10.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:10.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:10.439275+0000) 2024-03-06T15:28:11.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:11.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:11.439505+0000) 2024-03-06T15:28:12.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:12.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:12.439713+0000) 2024-03-06T15:28:13.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:13.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:13.439939+0000) 2024-03-06T15:28:14.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:14.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:14.440181+0000) 2024-03-06T15:28:15.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:15.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:15.440460+0000) 2024-03-06T15:28:16.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:16.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:16.440682+0000) 2024-03-06T15:28:17.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:17.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:17.440906+0000) 2024-03-06T15:28:18.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:18.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:18.441170+0000) 2024-03-06T15:28:18.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:18.893 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:19.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:19.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:19.441463+0000) 2024-03-06T15:28:20.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:20.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:20.441761+0000) 2024-03-06T15:28:21.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:21.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:21.442083+0000) 2024-03-06T15:28:22.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:22.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:22.442354+0000) 2024-03-06T15:28:23.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:23.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:23.442597+0000) 2024-03-06T15:28:24.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:24.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:24.442848+0000) 2024-03-06T15:28:25.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:25.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:25.443109+0000) 2024-03-06T15:28:26.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:26.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:26.443331+0000) 2024-03-06T15:28:27.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:27.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:27.443574+0000) 2024-03-06T15:28:28.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:28.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:28.443824+0000) 2024-03-06T15:28:29.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:29.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:29.444115+0000) 2024-03-06T15:28:30.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:30.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:30.444345+0000) 2024-03-06T15:28:31.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:31.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:31.444628+0000) 2024-03-06T15:28:32.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:32.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:32.444841+0000) 2024-03-06T15:28:33.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:33.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:33.445137+0000) 2024-03-06T15:28:34.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:34.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:34.445402+0000) 2024-03-06T15:28:35.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:35.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:35.445680+0000) 2024-03-06T15:28:36.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:36.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:36.445951+0000) 2024-03-06T15:28:37.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:37.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:37.446237+0000) 2024-03-06T15:28:38.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:38.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:38.446474+0000) 2024-03-06T15:28:39.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:39.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:39.446729+0000) 2024-03-06T15:28:40.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:40.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:40.446897+0000) 2024-03-06T15:28:41.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:41.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:41.447129+0000) 2024-03-06T15:28:42.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:42.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:42.447312+0000) 2024-03-06T15:28:43.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:43.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:43.447542+0000) 2024-03-06T15:28:44.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:44.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:44.447832+0000) 2024-03-06T15:28:45.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:45.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:45.448166+0000) 2024-03-06T15:28:46.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:46.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:46.448449+0000) 2024-03-06T15:28:47.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:47.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:47.448729+0000) 2024-03-06T15:28:48.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:48.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:48.448981+0000) 2024-03-06T15:28:49.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:49.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:28:49.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:49.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:49.449371+0000) 2024-03-06T15:28:50.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:50.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:50.449649+0000) 2024-03-06T15:28:51.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:51.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:51.449916+0000) 2024-03-06T15:28:52.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:52.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:52.450155+0000) 2024-03-06T15:28:53.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:53.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:53.450329+0000) 2024-03-06T15:28:54.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:54.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:54.450579+0000) 2024-03-06T15:28:55.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:55.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:55.450821+0000) 2024-03-06T15:28:56.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:56.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:56.451074+0000) 2024-03-06T15:28:57.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:57.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:57.451294+0000) 2024-03-06T15:28:58.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:58.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:58.451514+0000) 2024-03-06T15:28:59.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:28:59.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:28:59.451772+0000) 2024-03-06T15:29:00.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:00.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:00.451956+0000) 2024-03-06T15:29:01.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:01.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:01.452214+0000) 2024-03-06T15:29:02.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:02.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:02.452492+0000) 2024-03-06T15:29:03.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:03.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:03.452765+0000) 2024-03-06T15:29:04.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:04.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:04.453103+0000) 2024-03-06T15:29:05.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:05.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:05.453320+0000) 2024-03-06T15:29:06.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:06.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:06.453534+0000) 2024-03-06T15:29:07.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:07.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:07.453797+0000) 2024-03-06T15:29:08.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:08.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:08.454050+0000) 2024-03-06T15:29:09.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:09.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:09.454327+0000) 2024-03-06T15:29:10.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:10.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:10.454569+0000) 2024-03-06T15:29:11.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:11.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:11.454813+0000) 2024-03-06T15:29:12.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:12.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:12.455080+0000) 2024-03-06T15:29:13.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:13.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:13.455368+0000) 2024-03-06T15:29:14.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:14.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:14.455664+0000) 2024-03-06T15:29:15.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:15.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:15.455906+0000) 2024-03-06T15:29:16.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:16.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:16.456147+0000) 2024-03-06T15:29:17.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:17.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:17.456382+0000) 2024-03-06T15:29:18.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:18.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:18.456634+0000) 2024-03-06T15:29:19.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:19.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:19.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:19.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:19.456842+0000) 2024-03-06T15:29:20.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:20.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:20.457058+0000) 2024-03-06T15:29:21.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:21.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:21.457296+0000) 2024-03-06T15:29:22.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:22.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:22.457514+0000) 2024-03-06T15:29:23.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:23.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:23.457800+0000) 2024-03-06T15:29:24.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:24.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:24.458043+0000) 2024-03-06T15:29:25.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:25.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:25.458260+0000) 2024-03-06T15:29:26.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:26.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:26.458491+0000) 2024-03-06T15:29:27.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:27.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:27.458687+0000) 2024-03-06T15:29:28.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:28.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:28.458912+0000) 2024-03-06T15:29:29.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:29.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:29.459195+0000) 2024-03-06T15:29:30.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:30.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:30.459404+0000) 2024-03-06T15:29:31.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:31.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:31.459612+0000) 2024-03-06T15:29:32.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:32.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:32.459871+0000) 2024-03-06T15:29:33.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:33.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:33.460144+0000) 2024-03-06T15:29:34.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:34.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:34.460354+0000) 2024-03-06T15:29:35.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:35.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:35.460592+0000) 2024-03-06T15:29:36.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:36.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:36.460784+0000) 2024-03-06T15:29:37.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:37.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:37.461110+0000) 2024-03-06T15:29:38.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:38.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:38.461317+0000) 2024-03-06T15:29:39.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:39.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:39.461721+0000) 2024-03-06T15:29:40.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:40.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:40.461961+0000) 2024-03-06T15:29:41.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:41.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:41.462204+0000) 2024-03-06T15:29:42.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:42.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:42.462435+0000) 2024-03-06T15:29:43.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:43.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:43.462684+0000) 2024-03-06T15:29:44.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:44.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:44.462939+0000) 2024-03-06T15:29:45.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:45.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:45.463216+0000) 2024-03-06T15:29:46.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:46.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:46.463437+0000) 2024-03-06T15:29:47.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:47.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:47.463650+0000) 2024-03-06T15:29:48.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:48.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:48.463911+0000) 2024-03-06T15:29:49.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:49.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:49.464170+0000) 2024-03-06T15:29:49.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:49.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:29:50.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:50.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:50.464406+0000) 2024-03-06T15:29:51.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:51.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:51.464608+0000) 2024-03-06T15:29:52.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:52.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:52.464829+0000) 2024-03-06T15:29:53.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:53.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:53.465154+0000) 2024-03-06T15:29:54.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:54.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:54.465412+0000) 2024-03-06T15:29:55.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:55.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:55.465694+0000) 2024-03-06T15:29:56.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:56.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:56.466061+0000) 2024-03-06T15:29:57.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:57.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:57.466323+0000) 2024-03-06T15:29:58.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:58.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:58.466518+0000) 2024-03-06T15:29:59.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:29:59.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:29:59.466760+0000) 2024-03-06T15:30:00.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:00.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:00.466987+0000) 2024-03-06T15:30:01.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:01.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:01.467205+0000) 2024-03-06T15:30:02.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:02.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:02.467423+0000) 2024-03-06T15:30:03.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:03.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:03.467615+0000) 2024-03-06T15:30:04.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:04.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:04.467915+0000) 2024-03-06T15:30:05.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:05.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:05.468174+0000) 2024-03-06T15:30:06.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:06.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:06.468364+0000) 2024-03-06T15:30:07.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:07.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:07.468579+0000) 2024-03-06T15:30:08.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:08.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:08.468835+0000) 2024-03-06T15:30:09.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:09.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:09.469137+0000) 2024-03-06T15:30:10.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:10.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:10.469381+0000) 2024-03-06T15:30:11.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:11.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:11.469642+0000) 2024-03-06T15:30:12.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:12.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:12.469917+0000) 2024-03-06T15:30:13.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:13.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:13.470191+0000) 2024-03-06T15:30:14.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:14.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:14.470359+0000) 2024-03-06T15:30:15.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:15.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:15.470595+0000) 2024-03-06T15:30:16.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:16.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:16.470841+0000) 2024-03-06T15:30:17.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:17.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:17.471126+0000) 2024-03-06T15:30:18.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:18.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:18.471367+0000) 2024-03-06T15:30:19.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:19.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:19.471608+0000) 2024-03-06T15:30:19.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:19.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:20.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:20.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:20.471875+0000) 2024-03-06T15:30:21.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:21.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:21.472128+0000) 2024-03-06T15:30:22.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:22.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:22.472371+0000) 2024-03-06T15:30:23.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:23.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:23.472627+0000) 2024-03-06T15:30:24.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:24.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:24.472865+0000) 2024-03-06T15:30:25.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:25.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:25.473190+0000) 2024-03-06T15:30:26.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:26.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:26.473411+0000) 2024-03-06T15:30:27.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:27.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:27.473640+0000) 2024-03-06T15:30:28.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:28.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:28.474169+0000) 2024-03-06T15:30:29.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:29.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:29.474385+0000) 2024-03-06T15:30:30.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:30.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:30.474579+0000) 2024-03-06T15:30:31.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:31.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:31.474759+0000) 2024-03-06T15:30:32.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:32.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:32.474976+0000) 2024-03-06T15:30:33.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:33.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:33.475245+0000) 2024-03-06T15:30:34.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:34.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:34.475474+0000) 2024-03-06T15:30:35.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:35.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:35.475725+0000) 2024-03-06T15:30:36.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:36.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:36.475957+0000) 2024-03-06T15:30:37.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:37.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:37.476215+0000) 2024-03-06T15:30:38.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:38.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:38.476413+0000) 2024-03-06T15:30:39.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:39.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:39.476642+0000) 2024-03-06T15:30:40.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:40.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:40.476877+0000) 2024-03-06T15:30:41.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:41.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:41.477139+0000) 2024-03-06T15:30:42.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:42.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:42.477362+0000) 2024-03-06T15:30:43.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:43.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:43.477590+0000) 2024-03-06T15:30:44.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:44.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:44.477855+0000) 2024-03-06T15:30:45.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:45.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:45.478143+0000) 2024-03-06T15:30:46.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:46.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:46.478327+0000) 2024-03-06T15:30:47.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:47.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:47.478552+0000) 2024-03-06T15:30:48.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:48.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:48.478782+0000) 2024-03-06T15:30:49.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:49.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:49.479050+0000) 2024-03-06T15:30:49.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:49.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:30:50.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:50.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:50.479265+0000) 2024-03-06T15:30:51.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:51.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:51.479482+0000) 2024-03-06T15:30:52.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:52.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:52.479682+0000) 2024-03-06T15:30:53.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:53.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:53.479909+0000) 2024-03-06T15:30:54.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:54.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:54.480156+0000) 2024-03-06T15:30:55.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:55.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:55.480436+0000) 2024-03-06T15:30:56.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:56.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:56.480644+0000) 2024-03-06T15:30:57.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:57.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:57.480929+0000) 2024-03-06T15:30:58.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:58.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:58.481174+0000) 2024-03-06T15:30:59.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:30:59.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:30:59.481407+0000) 2024-03-06T15:31:00.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:00.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:00.481650+0000) 2024-03-06T15:31:01.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:01.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:01.481906+0000) 2024-03-06T15:31:02.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:02.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:02.482166+0000) 2024-03-06T15:31:03.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:03.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:03.482408+0000) 2024-03-06T15:31:04.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:04.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:04.482632+0000) 2024-03-06T15:31:05.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:05.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:05.482876+0000) 2024-03-06T15:31:06.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:06.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:06.483115+0000) 2024-03-06T15:31:07.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:07.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:07.483357+0000) 2024-03-06T15:31:08.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:08.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:08.483566+0000) 2024-03-06T15:31:09.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:09.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:09.483802+0000) 2024-03-06T15:31:10.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:10.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:10.484062+0000) 2024-03-06T15:31:11.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:11.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:11.484294+0000) 2024-03-06T15:31:12.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:12.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:12.484541+0000) 2024-03-06T15:31:13.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:13.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:13.484781+0000) 2024-03-06T15:31:14.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:14.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:14.485057+0000) 2024-03-06T15:31:15.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:15.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:15.485284+0000) 2024-03-06T15:31:16.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:16.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:16.485480+0000) 2024-03-06T15:31:17.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:17.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:17.485715+0000) 2024-03-06T15:31:18.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:18.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:18.485903+0000) 2024-03-06T15:31:19.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:19.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:19.486129+0000) 2024-03-06T15:31:20.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:20.047 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:20.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:20.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:20.486362+0000) 2024-03-06T15:31:21.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:21.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:21.486600+0000) 2024-03-06T15:31:22.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:22.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:22.486834+0000) 2024-03-06T15:31:23.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:23.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:23.487100+0000) 2024-03-06T15:31:24.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:24.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:24.487297+0000) 2024-03-06T15:31:25.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:25.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:25.487520+0000) 2024-03-06T15:31:26.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:26.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:26.487754+0000) 2024-03-06T15:31:27.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:27.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:27.487981+0000) 2024-03-06T15:31:28.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:28.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:28.488195+0000) 2024-03-06T15:31:29.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:29.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:29.488426+0000) 2024-03-06T15:31:30.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:30.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:30.488652+0000) 2024-03-06T15:31:31.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:31.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:31.488861+0000) 2024-03-06T15:31:32.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:32.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:32.489114+0000) 2024-03-06T15:31:33.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:33.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:33.489340+0000) 2024-03-06T15:31:34.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:34.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:34.489568+0000) 2024-03-06T15:31:35.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:35.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:35.489819+0000) 2024-03-06T15:31:36.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:36.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:36.490103+0000) 2024-03-06T15:31:37.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:37.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:37.490337+0000) 2024-03-06T15:31:38.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:38.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:38.490617+0000) 2024-03-06T15:31:39.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:39.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:39.490848+0000) 2024-03-06T15:31:40.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:40.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:40.491158+0000) 2024-03-06T15:31:41.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:41.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:41.491415+0000) 2024-03-06T15:31:42.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:42.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:42.491684+0000) 2024-03-06T15:31:43.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:43.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:43.491965+0000) 2024-03-06T15:31:44.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:44.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:44.492221+0000) 2024-03-06T15:31:45.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:45.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:45.492475+0000) 2024-03-06T15:31:46.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:46.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:46.492744+0000) 2024-03-06T15:31:47.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:47.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:47.493057+0000) 2024-03-06T15:31:48.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:48.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:48.493312+0000) 2024-03-06T15:31:49.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:49.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:49.493596+0000) 2024-03-06T15:31:50.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:50.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:31:50.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:50.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:50.493886+0000) 2024-03-06T15:31:51.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:51.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:51.494182+0000) 2024-03-06T15:31:52.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:52.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:52.494419+0000) 2024-03-06T15:31:53.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:53.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:53.494702+0000) 2024-03-06T15:31:54.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:54.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:54.494979+0000) 2024-03-06T15:31:55.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:55.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:55.495286+0000) 2024-03-06T15:31:56.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:56.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:56.495541+0000) 2024-03-06T15:31:57.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:57.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:57.495823+0000) 2024-03-06T15:31:58.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:58.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:58.496133+0000) 2024-03-06T15:31:59.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:31:59.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:31:59.496365+0000) 2024-03-06T15:32:00.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:00.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:00.496594+0000) 2024-03-06T15:32:01.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:01.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:01.496826+0000) 2024-03-06T15:32:02.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:02.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:02.497114+0000) 2024-03-06T15:32:03.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:03.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:03.497328+0000) 2024-03-06T15:32:04.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:04.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:04.497538+0000) 2024-03-06T15:32:05.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:05.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:05.497773+0000) 2024-03-06T15:32:06.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:06.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:06.498139+0000) 2024-03-06T15:32:07.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:07.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:07.498348+0000) 2024-03-06T15:32:08.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:08.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:08.498577+0000) 2024-03-06T15:32:09.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:09.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:09.498805+0000) 2024-03-06T15:32:10.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:10.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:10.499053+0000) 2024-03-06T15:32:11.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:11.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:11.499289+0000) 2024-03-06T15:32:12.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:12.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:12.499522+0000) 2024-03-06T15:32:13.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:13.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:13.499744+0000) 2024-03-06T15:32:14.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:14.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:14.499965+0000) 2024-03-06T15:32:15.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:15.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:15.500184+0000) 2024-03-06T15:32:16.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:16.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:16.500380+0000) 2024-03-06T15:32:17.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:17.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:17.500600+0000) 2024-03-06T15:32:18.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:18.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:18.500842+0000) 2024-03-06T15:32:19.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:19.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:19.501068+0000) 2024-03-06T15:32:20.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:20.325 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:20.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:20.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:20.501322+0000) 2024-03-06T15:32:21.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:21.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:21.501561+0000) 2024-03-06T15:32:22.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:22.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:22.501807+0000) 2024-03-06T15:32:23.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:23.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:23.502113+0000) 2024-03-06T15:32:24.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:24.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:24.502271+0000) 2024-03-06T15:32:25.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:25.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:25.502495+0000) 2024-03-06T15:32:26.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:26.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:26.502716+0000) 2024-03-06T15:32:27.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:27.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:27.502908+0000) 2024-03-06T15:32:28.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:28.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:28.503150+0000) 2024-03-06T15:32:29.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:29.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:29.503386+0000) 2024-03-06T15:32:30.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:30.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:30.503641+0000) 2024-03-06T15:32:31.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:31.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:31.503866+0000) 2024-03-06T15:32:32.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:32.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:32.504105+0000) 2024-03-06T15:32:33.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:33.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:33.504382+0000) 2024-03-06T15:32:34.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:34.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:34.504635+0000) 2024-03-06T15:32:35.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:35.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:35.504931+0000) 2024-03-06T15:32:36.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:36.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:36.505195+0000) 2024-03-06T15:32:37.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:37.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:37.505440+0000) 2024-03-06T15:32:38.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:38.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:38.505694+0000) 2024-03-06T15:32:39.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:39.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:39.505932+0000) 2024-03-06T15:32:40.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:40.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:40.506209+0000) 2024-03-06T15:32:41.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:41.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:41.506454+0000) 2024-03-06T15:32:42.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:42.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:42.506711+0000) 2024-03-06T15:32:43.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:43.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:43.506954+0000) 2024-03-06T15:32:44.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:44.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:44.507167+0000) 2024-03-06T15:32:45.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:45.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:45.507459+0000) 2024-03-06T15:32:46.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:46.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:46.507739+0000) 2024-03-06T15:32:47.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:47.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:47.507945+0000) 2024-03-06T15:32:48.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:48.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:48.508172+0000) 2024-03-06T15:32:49.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:49.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:49.508432+0000) 2024-03-06T15:32:50.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:50.465 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:32:50.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:50.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:50.508712+0000) 2024-03-06T15:32:51.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:51.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:51.508993+0000) 2024-03-06T15:32:52.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:52.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:52.509298+0000) 2024-03-06T15:32:53.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:53.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:53.509525+0000) 2024-03-06T15:32:54.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:54.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:54.509729+0000) 2024-03-06T15:32:55.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:55.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:55.509973+0000) 2024-03-06T15:32:56.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:56.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:56.510153+0000) 2024-03-06T15:32:57.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:57.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:57.510369+0000) 2024-03-06T15:32:58.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:58.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:58.510653+0000) 2024-03-06T15:32:59.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:32:59.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:32:59.510905+0000) 2024-03-06T15:33:00.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:00.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:00.511187+0000) 2024-03-06T15:33:01.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:01.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:01.511401+0000) 2024-03-06T15:33:02.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:02.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:02.511661+0000) 2024-03-06T15:33:03.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:03.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:03.511894+0000) 2024-03-06T15:33:04.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:04.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:04.512147+0000) 2024-03-06T15:33:05.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:05.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:05.512425+0000) 2024-03-06T15:33:06.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:06.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:06.512700+0000) 2024-03-06T15:33:07.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:07.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:07.512965+0000) 2024-03-06T15:33:08.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:08.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:08.513320+0000) 2024-03-06T15:33:09.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:09.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:09.513610+0000) 2024-03-06T15:33:10.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:10.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:10.513900+0000) 2024-03-06T15:33:11.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:11.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:11.514180+0000) 2024-03-06T15:33:12.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:12.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:12.514452+0000) 2024-03-06T15:33:13.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:13.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:13.514739+0000) 2024-03-06T15:33:14.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:14.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:14.514999+0000) 2024-03-06T15:33:15.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:15.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:15.515306+0000) 2024-03-06T15:33:16.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:16.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:16.515574+0000) 2024-03-06T15:33:17.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:17.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:17.515790+0000) 2024-03-06T15:33:18.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:18.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:18.515973+0000) 2024-03-06T15:33:19.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:19.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:19.516214+0000) 2024-03-06T15:33:20.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:20.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:20.516452+0000) 2024-03-06T15:33:20.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:20.627 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:21.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:21.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:21.516738+0000) 2024-03-06T15:33:22.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:22.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:22.517044+0000) 2024-03-06T15:33:23.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:23.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:23.517232+0000) 2024-03-06T15:33:24.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:24.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:24.517470+0000) 2024-03-06T15:33:25.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:25.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:25.517750+0000) 2024-03-06T15:33:26.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:26.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:26.517998+0000) 2024-03-06T15:33:27.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:27.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:27.518223+0000) 2024-03-06T15:33:28.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:28.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:28.518424+0000) 2024-03-06T15:33:29.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:29.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:29.518632+0000) 2024-03-06T15:33:30.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:30.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:30.518876+0000) 2024-03-06T15:33:31.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:31.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:31.519154+0000) 2024-03-06T15:33:32.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:32.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:32.519406+0000) 2024-03-06T15:33:33.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:33.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:33.519616+0000) 2024-03-06T15:33:34.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:34.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:34.519840+0000) 2024-03-06T15:33:35.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:35.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:35.520159+0000) 2024-03-06T15:33:36.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:36.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:36.520372+0000) 2024-03-06T15:33:37.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:37.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:37.520591+0000) 2024-03-06T15:33:38.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:38.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:38.520784+0000) 2024-03-06T15:33:39.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:39.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:39.521065+0000) 2024-03-06T15:33:40.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:40.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:40.521278+0000) 2024-03-06T15:33:41.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:41.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:41.521491+0000) 2024-03-06T15:33:42.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:42.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:42.521702+0000) 2024-03-06T15:33:43.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:43.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:43.521938+0000) 2024-03-06T15:33:44.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:44.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:44.522168+0000) 2024-03-06T15:33:45.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:45.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:45.522472+0000) 2024-03-06T15:33:46.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:46.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:46.522760+0000) 2024-03-06T15:33:47.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:47.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:47.523121+0000) 2024-03-06T15:33:48.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:48.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:48.523343+0000) 2024-03-06T15:33:49.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:49.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:49.523600+0000) 2024-03-06T15:33:50.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:50.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:50.523873+0000) 2024-03-06T15:33:50.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:50.769 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:33:51.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:51.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:51.524161+0000) 2024-03-06T15:33:52.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:52.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:52.524438+0000) 2024-03-06T15:33:53.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:53.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:53.524683+0000) 2024-03-06T15:33:54.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:54.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:54.524901+0000) 2024-03-06T15:33:55.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:55.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:55.525134+0000) 2024-03-06T15:33:56.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:56.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:56.525329+0000) 2024-03-06T15:33:57.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:57.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:57.525504+0000) 2024-03-06T15:33:58.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:58.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:58.525738+0000) 2024-03-06T15:33:59.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:33:59.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:33:59.525961+0000) 2024-03-06T15:34:00.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:00.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:00.526177+0000) 2024-03-06T15:34:01.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:01.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:01.526379+0000) 2024-03-06T15:34:02.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:02.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:02.526600+0000) 2024-03-06T15:34:03.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:03.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:03.526860+0000) 2024-03-06T15:34:04.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:04.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:04.527109+0000) 2024-03-06T15:34:05.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:05.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:05.527351+0000) 2024-03-06T15:34:06.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:06.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:06.527579+0000) 2024-03-06T15:34:07.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:07.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:07.527803+0000) 2024-03-06T15:34:08.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:08.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:08.528049+0000) 2024-03-06T15:34:09.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:09.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:09.528307+0000) 2024-03-06T15:34:10.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:10.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:10.528572+0000) 2024-03-06T15:34:11.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:11.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:11.528823+0000) 2024-03-06T15:34:12.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:12.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:12.529122+0000) 2024-03-06T15:34:13.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:13.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:13.529347+0000) 2024-03-06T15:34:14.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:14.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:14.529588+0000) 2024-03-06T15:34:15.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:15.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:15.529854+0000) 2024-03-06T15:34:16.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:16.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:16.530115+0000) 2024-03-06T15:34:17.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:17.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:17.530357+0000) 2024-03-06T15:34:18.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:18.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:18.530570+0000) 2024-03-06T15:34:19.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:19.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:19.530822+0000) 2024-03-06T15:34:20.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:20.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:20.531102+0000) 2024-03-06T15:34:20.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:20.926 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:21.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:21.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:21.531342+0000) 2024-03-06T15:34:22.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:22.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:22.531504+0000) 2024-03-06T15:34:23.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:23.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:23.531689+0000) 2024-03-06T15:34:24.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:24.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:24.531886+0000) 2024-03-06T15:34:25.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:25.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:25.532138+0000) 2024-03-06T15:34:26.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:26.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:26.532319+0000) 2024-03-06T15:34:27.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:27.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:27.532566+0000) 2024-03-06T15:34:28.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:28.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:28.532826+0000) 2024-03-06T15:34:29.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:29.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:29.533127+0000) 2024-03-06T15:34:30.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:30.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:30.533381+0000) 2024-03-06T15:34:31.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:31.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:31.533653+0000) 2024-03-06T15:34:32.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:32.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:32.533898+0000) 2024-03-06T15:34:33.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:33.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:33.534159+0000) 2024-03-06T15:34:34.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:34.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:34.534371+0000) 2024-03-06T15:34:35.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:35.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:35.534654+0000) 2024-03-06T15:34:36.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:36.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:36.534895+0000) 2024-03-06T15:34:37.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:37.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:37.535142+0000) 2024-03-06T15:34:38.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:38.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:38.535340+0000) 2024-03-06T15:34:39.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:39.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:39.535551+0000) 2024-03-06T15:34:40.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:40.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:40.535797+0000) 2024-03-06T15:34:41.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:41.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:41.536057+0000) 2024-03-06T15:34:42.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:42.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:42.536257+0000) 2024-03-06T15:34:43.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:43.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:43.536507+0000) 2024-03-06T15:34:44.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:44.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:44.536775+0000) 2024-03-06T15:34:45.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:45.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:45.537049+0000) 2024-03-06T15:34:46.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:46.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:46.537289+0000) 2024-03-06T15:34:47.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:47.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:47.537538+0000) 2024-03-06T15:34:48.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:48.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:48.537718+0000) 2024-03-06T15:34:49.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:49.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:49.537877+0000) 2024-03-06T15:34:50.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:50.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:50.538140+0000) 2024-03-06T15:34:51.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:51.082 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:34:51.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:51.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:51.538377+0000) 2024-03-06T15:34:52.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:52.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:52.538605+0000) 2024-03-06T15:34:53.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:53.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:53.538891+0000) 2024-03-06T15:34:54.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:54.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:54.539141+0000) 2024-03-06T15:34:55.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:55.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:55.539390+0000) 2024-03-06T15:34:56.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:56.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:56.539617+0000) 2024-03-06T15:34:57.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:57.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:57.539848+0000) 2024-03-06T15:34:58.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:58.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:58.540107+0000) 2024-03-06T15:34:59.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:34:59.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:34:59.540351+0000) 2024-03-06T15:35:00.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:00.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:00.540591+0000) 2024-03-06T15:35:01.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:01.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:01.540845+0000) 2024-03-06T15:35:02.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:02.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:02.541134+0000) 2024-03-06T15:35:03.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:03.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:03.541372+0000) 2024-03-06T15:35:04.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:04.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:04.541630+0000) 2024-03-06T15:35:05.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:05.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:05.541882+0000) 2024-03-06T15:35:06.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:06.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:06.542114+0000) 2024-03-06T15:35:07.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:07.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:07.542373+0000) 2024-03-06T15:35:08.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:08.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:08.542538+0000) 2024-03-06T15:35:09.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:09.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:09.542769+0000) 2024-03-06T15:35:10.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:10.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:10.543042+0000) 2024-03-06T15:35:11.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:11.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:11.543291+0000) 2024-03-06T15:35:12.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:12.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:12.543524+0000) 2024-03-06T15:35:13.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:13.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:13.543779+0000) 2024-03-06T15:35:14.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:14.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:14.544106+0000) 2024-03-06T15:35:15.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:15.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:15.544428+0000) 2024-03-06T15:35:16.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:16.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:16.544707+0000) 2024-03-06T15:35:17.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:17.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:17.544952+0000) 2024-03-06T15:35:18.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:18.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:18.545201+0000) 2024-03-06T15:35:19.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:19.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:19.545471+0000) 2024-03-06T15:35:20.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:20.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:20.545749+0000) 2024-03-06T15:35:21.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:21.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:21.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:21.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:21.546064+0000) 2024-03-06T15:35:22.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:22.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:22.546353+0000) 2024-03-06T15:35:23.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:23.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:23.546686+0000) 2024-03-06T15:35:24.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:24.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:24.546937+0000) 2024-03-06T15:35:25.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:25.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:25.547163+0000) 2024-03-06T15:35:26.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:26.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:26.547377+0000) 2024-03-06T15:35:27.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:27.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:27.547605+0000) 2024-03-06T15:35:28.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:28.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:28.547815+0000) 2024-03-06T15:35:29.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:29.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:29.548078+0000) 2024-03-06T15:35:30.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:30.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:30.548321+0000) 2024-03-06T15:35:31.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:31.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:31.548526+0000) 2024-03-06T15:35:32.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:32.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:32.548734+0000) 2024-03-06T15:35:33.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:33.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:33.548999+0000) 2024-03-06T15:35:34.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:34.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:34.549285+0000) 2024-03-06T15:35:35.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:35.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:35.549558+0000) 2024-03-06T15:35:36.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:36.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:36.549837+0000) 2024-03-06T15:35:37.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:37.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:37.550137+0000) 2024-03-06T15:35:38.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:38.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:38.550324+0000) 2024-03-06T15:35:39.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:39.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:39.550558+0000) 2024-03-06T15:35:40.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:40.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:40.550779+0000) 2024-03-06T15:35:41.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:41.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:41.551087+0000) 2024-03-06T15:35:42.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:42.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:42.551341+0000) 2024-03-06T15:35:43.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:43.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:43.551626+0000) 2024-03-06T15:35:44.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:44.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:44.551918+0000) 2024-03-06T15:35:45.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:45.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:45.552205+0000) 2024-03-06T15:35:46.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:46.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:46.552482+0000) 2024-03-06T15:35:47.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:47.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:47.552770+0000) 2024-03-06T15:35:48.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:48.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:48.553046+0000) 2024-03-06T15:35:49.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:49.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:49.553300+0000) 2024-03-06T15:35:50.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:50.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:50.553545+0000) 2024-03-06T15:35:51.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:51.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:35:51.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:51.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:51.553774+0000) 2024-03-06T15:35:52.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:52.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:52.554063+0000) 2024-03-06T15:35:53.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:53.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:53.554299+0000) 2024-03-06T15:35:54.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:54.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:54.554543+0000) 2024-03-06T15:35:55.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:55.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:55.554818+0000) 2024-03-06T15:35:56.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:56.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:56.555133+0000) 2024-03-06T15:35:57.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:57.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:57.555370+0000) 2024-03-06T15:35:58.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:58.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:58.555633+0000) 2024-03-06T15:35:59.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:35:59.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:35:59.555915+0000) 2024-03-06T15:36:00.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:00.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:00.556192+0000) 2024-03-06T15:36:01.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:01.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:01.556443+0000) 2024-03-06T15:36:02.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:02.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:02.556724+0000) 2024-03-06T15:36:03.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:03.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:03.556992+0000) 2024-03-06T15:36:04.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:04.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:04.557289+0000) 2024-03-06T15:36:05.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:05.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:05.557572+0000) 2024-03-06T15:36:06.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:06.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:06.557851+0000) 2024-03-06T15:36:07.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:07.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:07.558120+0000) 2024-03-06T15:36:08.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:08.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:08.558346+0000) 2024-03-06T15:36:09.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:09.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:09.558548+0000) 2024-03-06T15:36:10.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:10.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:10.558763+0000) 2024-03-06T15:36:11.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:11.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:11.559040+0000) 2024-03-06T15:36:12.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:12.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:12.559241+0000) 2024-03-06T15:36:13.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:13.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:13.559506+0000) 2024-03-06T15:36:14.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:14.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:14.559771+0000) 2024-03-06T15:36:15.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:15.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:15.560052+0000) 2024-03-06T15:36:16.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:16.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:16.560300+0000) 2024-03-06T15:36:17.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:17.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:17.560603+0000) 2024-03-06T15:36:18.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:18.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:18.560842+0000) 2024-03-06T15:36:19.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:19.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:19.561122+0000) 2024-03-06T15:36:20.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:20.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:20.561370+0000) 2024-03-06T15:36:21.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:21.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:21.561600+0000) 2024-03-06T15:36:21.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:21.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:22.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:22.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:22.561831+0000) 2024-03-06T15:36:23.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:23.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:23.562099+0000) 2024-03-06T15:36:24.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:24.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:24.562336+0000) 2024-03-06T15:36:25.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:25.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:25.562536+0000) 2024-03-06T15:36:26.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:26.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:26.562761+0000) 2024-03-06T15:36:27.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:27.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:27.563046+0000) 2024-03-06T15:36:28.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:28.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:28.563302+0000) 2024-03-06T15:36:29.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:29.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:29.563563+0000) 2024-03-06T15:36:30.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:30.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:30.563840+0000) 2024-03-06T15:36:31.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:31.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:31.564143+0000) 2024-03-06T15:36:32.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:32.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:32.564390+0000) 2024-03-06T15:36:33.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:33.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:33.564657+0000) 2024-03-06T15:36:34.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:34.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:34.564935+0000) 2024-03-06T15:36:35.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:35.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:35.565198+0000) 2024-03-06T15:36:36.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:36.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:36.565450+0000) 2024-03-06T15:36:37.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:37.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:37.565631+0000) 2024-03-06T15:36:38.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:38.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:38.565802+0000) 2024-03-06T15:36:39.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:39.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:39.565983+0000) 2024-03-06T15:36:40.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:40.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:40.566213+0000) 2024-03-06T15:36:41.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:41.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:41.566411+0000) 2024-03-06T15:36:42.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:42.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:42.566624+0000) 2024-03-06T15:36:43.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:43.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:43.566840+0000) 2024-03-06T15:36:44.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:44.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:44.567116+0000) 2024-03-06T15:36:45.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:45.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:45.567321+0000) 2024-03-06T15:36:46.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:46.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:46.567535+0000) 2024-03-06T15:36:47.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:47.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:47.567764+0000) 2024-03-06T15:36:48.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:48.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:48.567982+0000) 2024-03-06T15:36:49.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:49.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:49.568204+0000) 2024-03-06T15:36:50.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:50.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:50.568397+0000) 2024-03-06T15:36:51.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:51.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:51.568559+0000) 2024-03-06T15:36:51.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:51.859 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:36:52.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:52.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:52.568844+0000) 2024-03-06T15:36:53.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:53.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:53.569134+0000) 2024-03-06T15:36:54.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:54.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:54.569376+0000) 2024-03-06T15:36:55.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:55.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:55.569615+0000) 2024-03-06T15:36:56.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:56.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:56.569801+0000) 2024-03-06T15:36:57.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:57.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:57.570049+0000) 2024-03-06T15:36:58.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:58.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:58.570271+0000) 2024-03-06T15:36:59.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:36:59.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:36:59.570508+0000) 2024-03-06T15:37:00.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:00.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:00.570794+0000) 2024-03-06T15:37:01.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:01.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:01.571127+0000) 2024-03-06T15:37:02.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:02.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:02.571410+0000) 2024-03-06T15:37:03.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:03.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:03.571656+0000) 2024-03-06T15:37:04.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:04.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:04.571882+0000) 2024-03-06T15:37:05.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:05.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:05.572141+0000) 2024-03-06T15:37:06.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:06.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:06.572418+0000) 2024-03-06T15:37:07.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:07.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:07.572710+0000) 2024-03-06T15:37:08.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:08.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:08.572916+0000) 2024-03-06T15:37:09.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:09.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:09.573151+0000) 2024-03-06T15:37:10.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:10.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:10.573403+0000) 2024-03-06T15:37:11.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:11.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:11.573639+0000) 2024-03-06T15:37:12.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:12.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:12.573879+0000) 2024-03-06T15:37:13.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:13.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:13.574126+0000) 2024-03-06T15:37:14.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:14.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:14.574330+0000) 2024-03-06T15:37:15.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:15.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:15.574560+0000) 2024-03-06T15:37:16.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:16.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:16.574836+0000) 2024-03-06T15:37:17.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:17.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:17.575119+0000) 2024-03-06T15:37:18.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:18.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:18.575333+0000) 2024-03-06T15:37:19.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:19.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:19.575535+0000) 2024-03-06T15:37:20.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:20.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:20.575758+0000) 2024-03-06T15:37:21.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:21.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:21.576036+0000) 2024-03-06T15:37:22.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:22.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:22.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:22.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:22.576270+0000) 2024-03-06T15:37:23.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:23.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:23.576496+0000) 2024-03-06T15:37:24.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:24.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:24.576792+0000) 2024-03-06T15:37:25.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:25.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:25.577095+0000) 2024-03-06T15:37:26.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:26.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:26.577291+0000) 2024-03-06T15:37:27.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:27.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:27.577503+0000) 2024-03-06T15:37:28.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:28.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:28.577771+0000) 2024-03-06T15:37:29.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:29.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:29.578096+0000) 2024-03-06T15:37:30.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:30.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:30.578304+0000) 2024-03-06T15:37:31.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:31.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:31.578523+0000) 2024-03-06T15:37:32.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:32.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:32.578738+0000) 2024-03-06T15:37:33.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:33.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:33.579040+0000) 2024-03-06T15:37:34.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:34.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:34.579261+0000) 2024-03-06T15:37:35.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:35.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:35.579503+0000) 2024-03-06T15:37:36.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:36.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:36.579753+0000) 2024-03-06T15:37:37.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:37.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:37.579991+0000) 2024-03-06T15:37:38.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:38.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:38.580252+0000) 2024-03-06T15:37:39.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:39.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:39.580534+0000) 2024-03-06T15:37:40.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:40.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:40.580740+0000) 2024-03-06T15:37:41.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:41.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:41.580973+0000) 2024-03-06T15:37:42.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:42.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:42.581168+0000) 2024-03-06T15:37:43.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:43.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:43.581400+0000) 2024-03-06T15:37:44.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:44.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:44.581623+0000) 2024-03-06T15:37:45.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:45.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:45.581859+0000) 2024-03-06T15:37:46.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:46.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:46.582140+0000) 2024-03-06T15:37:47.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:47.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:47.582358+0000) 2024-03-06T15:37:48.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:48.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:48.582594+0000) 2024-03-06T15:37:49.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:49.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:49.582832+0000) 2024-03-06T15:37:50.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:50.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:50.583067+0000) 2024-03-06T15:37:51.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:51.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:51.583302+0000) 2024-03-06T15:37:52.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:52.296 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:37:52.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:52.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:52.583531+0000) 2024-03-06T15:37:53.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:53.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:53.583809+0000) 2024-03-06T15:37:54.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:54.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:54.584047+0000) 2024-03-06T15:37:55.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:55.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:55.584294+0000) 2024-03-06T15:37:56.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:56.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:56.584524+0000) 2024-03-06T15:37:57.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:57.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:57.584709+0000) 2024-03-06T15:37:58.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:58.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:58.584978+0000) 2024-03-06T15:37:59.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:37:59.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:37:59.585313+0000) 2024-03-06T15:38:00.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:00.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:00.585596+0000) 2024-03-06T15:38:01.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:01.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:01.585822+0000) 2024-03-06T15:38:02.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:02.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:02.586126+0000) 2024-03-06T15:38:03.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:03.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:03.586377+0000) 2024-03-06T15:38:04.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:04.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:04.586628+0000) 2024-03-06T15:38:05.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:05.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:05.586917+0000) 2024-03-06T15:38:06.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:06.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:06.587199+0000) 2024-03-06T15:38:07.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:07.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:07.587414+0000) 2024-03-06T15:38:08.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:08.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:08.587669+0000) 2024-03-06T15:38:09.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:09.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:09.587924+0000) 2024-03-06T15:38:10.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:10.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:10.588167+0000) 2024-03-06T15:38:11.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:11.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:11.588426+0000) 2024-03-06T15:38:12.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:12.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:12.588706+0000) 2024-03-06T15:38:13.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:13.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:13.588982+0000) 2024-03-06T15:38:14.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:14.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:14.589223+0000) 2024-03-06T15:38:15.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:15.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:15.589420+0000) 2024-03-06T15:38:16.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:16.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:16.589640+0000) 2024-03-06T15:38:17.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:17.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:17.589883+0000) 2024-03-06T15:38:18.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:18.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:18.590119+0000) 2024-03-06T15:38:19.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:19.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:19.590360+0000) 2024-03-06T15:38:20.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:20.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:20.590642+0000) 2024-03-06T15:38:21.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:21.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:21.590895+0000) 2024-03-06T15:38:22.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:22.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:22.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:22.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:22.591144+0000) 2024-03-06T15:38:23.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:23.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:23.591345+0000) 2024-03-06T15:38:24.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:24.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:24.591535+0000) 2024-03-06T15:38:25.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:25.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:25.591769+0000) 2024-03-06T15:38:26.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:26.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:26.592019+0000) 2024-03-06T15:38:27.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:27.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:27.592935+0000) 2024-03-06T15:38:28.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:28.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:28.593280+0000) 2024-03-06T15:38:29.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:29.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:29.593536+0000) 2024-03-06T15:38:30.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:30.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:30.593796+0000) 2024-03-06T15:38:31.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:31.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:31.594088+0000) 2024-03-06T15:38:32.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:32.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:32.594421+0000) 2024-03-06T15:38:33.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:33.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:33.594656+0000) 2024-03-06T15:38:34.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:34.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:34.594888+0000) 2024-03-06T15:38:35.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:35.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:35.595130+0000) 2024-03-06T15:38:36.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:36.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:36.595338+0000) 2024-03-06T15:38:37.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:37.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:37.595526+0000) 2024-03-06T15:38:38.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:38.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:38.595747+0000) 2024-03-06T15:38:39.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:39.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:39.595969+0000) 2024-03-06T15:38:40.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:40.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:40.596161+0000) 2024-03-06T15:38:41.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:41.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:41.596380+0000) 2024-03-06T15:38:42.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:42.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:42.596647+0000) 2024-03-06T15:38:43.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:43.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:43.596870+0000) 2024-03-06T15:38:44.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:44.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:44.597136+0000) 2024-03-06T15:38:45.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:45.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:45.597334+0000) 2024-03-06T15:38:46.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:46.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:46.597575+0000) 2024-03-06T15:38:47.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:47.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:47.597805+0000) 2024-03-06T15:38:48.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:48.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:48.598056+0000) 2024-03-06T15:38:49.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:49.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:49.598282+0000) 2024-03-06T15:38:50.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:50.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:50.598505+0000) 2024-03-06T15:38:51.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:51.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:51.598718+0000) 2024-03-06T15:38:52.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:52.599 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:38:52.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:52.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:52.598963+0000) 2024-03-06T15:38:53.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:53.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:53.599198+0000) 2024-03-06T15:38:54.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:54.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:54.599417+0000) 2024-03-06T15:38:55.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:55.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:55.599647+0000) 2024-03-06T15:38:56.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:56.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:56.599856+0000) 2024-03-06T15:38:57.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:57.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:57.600096+0000) 2024-03-06T15:38:58.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:58.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:58.600288+0000) 2024-03-06T15:38:59.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:38:59.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:38:59.600474+0000) 2024-03-06T15:39:00.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:00.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:00.600699+0000) 2024-03-06T15:39:01.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:01.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:01.600945+0000) 2024-03-06T15:39:02.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:02.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:02.601168+0000) 2024-03-06T15:39:03.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:03.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:03.601422+0000) 2024-03-06T15:39:04.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:04.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:04.601594+0000) 2024-03-06T15:39:05.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:05.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:05.601849+0000) 2024-03-06T15:39:06.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:06.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:06.602138+0000) 2024-03-06T15:39:07.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:07.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:07.602341+0000) 2024-03-06T15:39:08.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:08.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:08.602569+0000) 2024-03-06T15:39:09.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:09.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:09.602853+0000) 2024-03-06T15:39:10.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:10.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:10.603164+0000) 2024-03-06T15:39:11.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:11.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:11.603451+0000) 2024-03-06T15:39:12.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:12.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:12.603726+0000) 2024-03-06T15:39:13.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:13.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:13.603997+0000) 2024-03-06T15:39:14.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:14.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:14.604297+0000) 2024-03-06T15:39:15.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:15.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:15.604535+0000) 2024-03-06T15:39:16.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:16.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:16.604775+0000) 2024-03-06T15:39:17.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:17.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:17.605059+0000) 2024-03-06T15:39:18.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:18.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:18.605290+0000) 2024-03-06T15:39:19.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:19.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:19.605511+0000) 2024-03-06T15:39:20.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:20.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:20.605724+0000) 2024-03-06T15:39:21.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:21.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:21.605926+0000) 2024-03-06T15:39:22.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:22.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:22.606197+0000) 2024-03-06T15:39:22.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:22.749 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:23.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:23.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:23.606462+0000) 2024-03-06T15:39:24.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:24.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:24.606725+0000) 2024-03-06T15:39:25.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:25.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:25.606963+0000) 2024-03-06T15:39:26.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:26.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:26.607215+0000) 2024-03-06T15:39:27.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:27.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:27.607456+0000) 2024-03-06T15:39:28.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:28.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:28.607697+0000) 2024-03-06T15:39:29.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:29.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:29.607921+0000) 2024-03-06T15:39:30.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:30.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:30.608140+0000) 2024-03-06T15:39:31.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:31.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:31.608377+0000) 2024-03-06T15:39:32.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:32.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:32.608644+0000) 2024-03-06T15:39:33.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:33.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:33.608843+0000) 2024-03-06T15:39:34.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:34.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:34.609096+0000) 2024-03-06T15:39:35.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:35.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:35.609387+0000) 2024-03-06T15:39:36.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:36.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:36.609655+0000) 2024-03-06T15:39:37.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:37.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:37.609912+0000) 2024-03-06T15:39:38.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:38.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:38.610194+0000) 2024-03-06T15:39:39.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:39.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:39.610480+0000) 2024-03-06T15:39:40.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:40.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:40.610750+0000) 2024-03-06T15:39:41.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:41.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:41.611077+0000) 2024-03-06T15:39:42.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:42.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:42.611299+0000) 2024-03-06T15:39:43.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:43.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:43.611546+0000) 2024-03-06T15:39:44.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:44.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:44.611757+0000) 2024-03-06T15:39:45.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:45.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:45.612044+0000) 2024-03-06T15:39:46.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:46.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:46.612303+0000) 2024-03-06T15:39:47.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:47.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:47.612537+0000) 2024-03-06T15:39:48.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:48.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:48.612770+0000) 2024-03-06T15:39:49.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:49.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:49.613036+0000) 2024-03-06T15:39:50.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:50.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:50.613276+0000) 2024-03-06T15:39:51.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:51.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:51.613531+0000) 2024-03-06T15:39:52.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:52.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:52.613758+0000) 2024-03-06T15:39:52.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:52.887 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:39:53.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:53.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:53.613983+0000) 2024-03-06T15:39:54.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:54.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:54.614233+0000) 2024-03-06T15:39:55.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:55.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:55.614463+0000) 2024-03-06T15:39:56.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:56.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:56.614705+0000) 2024-03-06T15:39:57.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:57.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:57.614930+0000) 2024-03-06T15:39:58.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:58.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:58.615158+0000) 2024-03-06T15:39:59.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:39:59.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:39:59.615411+0000) 2024-03-06T15:40:00.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:00.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:00.615640+0000) 2024-03-06T15:40:01.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:01.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:01.615876+0000) 2024-03-06T15:40:02.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:02.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:02.616144+0000) 2024-03-06T15:40:03.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:03.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:03.616380+0000) 2024-03-06T15:40:04.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:04.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:04.616588+0000) 2024-03-06T15:40:05.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:05.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:05.616841+0000) 2024-03-06T15:40:06.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:06.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:06.617146+0000) 2024-03-06T15:40:07.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:07.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:07.617355+0000) 2024-03-06T15:40:08.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:08.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:08.617606+0000) 2024-03-06T15:40:09.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:09.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:09.617859+0000) 2024-03-06T15:40:10.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:10.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:10.618108+0000) 2024-03-06T15:40:11.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:11.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:11.618353+0000) 2024-03-06T15:40:12.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:12.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:12.618574+0000) 2024-03-06T15:40:13.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:13.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:13.618808+0000) 2024-03-06T15:40:14.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:14.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:14.619051+0000) 2024-03-06T15:40:15.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:15.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:15.619243+0000) 2024-03-06T15:40:16.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:16.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:16.619491+0000) 2024-03-06T15:40:17.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:17.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:17.619715+0000) 2024-03-06T15:40:18.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:18.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:18.619941+0000) 2024-03-06T15:40:19.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:19.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:19.620170+0000) 2024-03-06T15:40:20.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:20.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:20.620358+0000) 2024-03-06T15:40:21.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:21.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:21.620593+0000) 2024-03-06T15:40:22.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:22.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:22.620817+0000) 2024-03-06T15:40:23.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:23.037 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:23.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:23.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:23.621083+0000) 2024-03-06T15:40:24.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:24.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:24.621304+0000) 2024-03-06T15:40:25.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:25.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:25.621538+0000) 2024-03-06T15:40:26.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:26.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:26.621792+0000) 2024-03-06T15:40:27.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:27.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:27.622049+0000) 2024-03-06T15:40:28.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:28.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:28.622252+0000) 2024-03-06T15:40:29.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:29.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:29.622551+0000) 2024-03-06T15:40:30.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:30.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:30.622788+0000) 2024-03-06T15:40:31.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:31.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:31.623094+0000) 2024-03-06T15:40:32.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:32.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:32.623350+0000) 2024-03-06T15:40:33.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:33.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:33.623562+0000) 2024-03-06T15:40:34.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:34.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:34.623783+0000) 2024-03-06T15:40:35.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:35.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:35.623993+0000) 2024-03-06T15:40:36.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:36.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:36.624258+0000) 2024-03-06T15:40:37.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:37.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:37.624472+0000) 2024-03-06T15:40:38.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:38.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:38.624714+0000) 2024-03-06T15:40:39.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:39.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:39.624940+0000) 2024-03-06T15:40:40.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:40.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:40.625136+0000) 2024-03-06T15:40:41.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:41.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:41.625344+0000) 2024-03-06T15:40:42.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:42.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:42.625551+0000) 2024-03-06T15:40:43.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:43.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:43.625768+0000) 2024-03-06T15:40:44.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:44.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:44.625999+0000) 2024-03-06T15:40:45.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:45.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:45.626227+0000) 2024-03-06T15:40:46.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:46.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:46.626451+0000) 2024-03-06T15:40:47.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:47.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:47.626663+0000) 2024-03-06T15:40:48.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:48.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:48.626862+0000) 2024-03-06T15:40:49.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:49.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:49.627138+0000) 2024-03-06T15:40:50.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:50.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:50.627398+0000) 2024-03-06T15:40:51.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:51.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:51.627640+0000) 2024-03-06T15:40:52.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:52.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:52.627833+0000) 2024-03-06T15:40:53.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:53.253 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:40:53.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:53.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:53.628068+0000) 2024-03-06T15:40:54.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:54.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:54.628263+0000) 2024-03-06T15:40:55.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:55.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:55.628515+0000) 2024-03-06T15:40:56.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:56.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:56.628775+0000) 2024-03-06T15:40:57.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:57.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:57.629061+0000) 2024-03-06T15:40:58.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:58.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:58.629308+0000) 2024-03-06T15:40:59.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:40:59.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:40:59.629569+0000) 2024-03-06T15:41:00.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:00.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:00.629809+0000) 2024-03-06T15:41:01.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:01.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:01.630118+0000) 2024-03-06T15:41:02.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:02.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:02.630347+0000) 2024-03-06T15:41:03.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:03.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:03.630546+0000) 2024-03-06T15:41:04.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:04.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:04.630771+0000) 2024-03-06T15:41:05.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:05.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:05.631037+0000) 2024-03-06T15:41:06.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:06.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:06.631295+0000) 2024-03-06T15:41:07.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:07.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:07.631530+0000) 2024-03-06T15:41:08.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:08.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:08.631728+0000) 2024-03-06T15:41:09.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:09.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:09.631953+0000) 2024-03-06T15:41:10.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:10.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:10.632202+0000) 2024-03-06T15:41:11.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:11.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:11.632443+0000) 2024-03-06T15:41:12.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:12.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:12.632632+0000) 2024-03-06T15:41:13.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:13.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:13.632858+0000) 2024-03-06T15:41:14.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:14.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:14.633121+0000) 2024-03-06T15:41:15.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:15.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:15.633330+0000) 2024-03-06T15:41:16.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:16.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:16.633576+0000) 2024-03-06T15:41:17.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:17.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:17.633811+0000) 2024-03-06T15:41:18.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:18.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:18.634083+0000) 2024-03-06T15:41:19.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:19.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:19.634315+0000) 2024-03-06T15:41:20.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:20.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:20.634517+0000) 2024-03-06T15:41:21.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:21.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:21.634767+0000) 2024-03-06T15:41:22.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:22.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:22.635108+0000) 2024-03-06T15:41:23.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:23.450 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:23.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:23.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:23.635290+0000) 2024-03-06T15:41:24.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:24.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:24.635516+0000) 2024-03-06T15:41:25.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:25.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:25.635754+0000) 2024-03-06T15:41:26.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:26.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:26.635980+0000) 2024-03-06T15:41:27.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:27.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:27.636260+0000) 2024-03-06T15:41:28.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:28.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:28.636489+0000) 2024-03-06T15:41:29.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:29.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:29.636718+0000) 2024-03-06T15:41:30.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:30.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:30.636902+0000) 2024-03-06T15:41:31.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:31.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:31.637158+0000) 2024-03-06T15:41:32.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:32.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:32.637406+0000) 2024-03-06T15:41:33.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:33.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:33.637656+0000) 2024-03-06T15:41:34.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:34.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:34.637957+0000) 2024-03-06T15:41:35.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:35.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:35.638249+0000) 2024-03-06T15:41:36.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:36.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:36.638510+0000) 2024-03-06T15:41:37.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:37.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:37.638759+0000) 2024-03-06T15:41:38.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:38.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:38.638993+0000) 2024-03-06T15:41:39.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:39.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:39.639279+0000) 2024-03-06T15:41:40.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:40.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:40.639518+0000) 2024-03-06T15:41:41.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:41.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:41.639747+0000) 2024-03-06T15:41:42.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:42.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:42.639980+0000) 2024-03-06T15:41:43.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:43.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:43.640201+0000) 2024-03-06T15:41:44.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:44.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:44.640398+0000) 2024-03-06T15:41:45.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:45.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:45.640610+0000) 2024-03-06T15:41:46.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:46.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:46.640820+0000) 2024-03-06T15:41:47.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:47.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:47.641071+0000) 2024-03-06T15:41:48.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:48.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:48.641313+0000) 2024-03-06T15:41:49.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:49.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:49.641531+0000) 2024-03-06T15:41:50.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:50.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:50.641713+0000) 2024-03-06T15:41:51.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:51.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:51.641941+0000) 2024-03-06T15:41:52.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:52.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:52.642205+0000) 2024-03-06T15:41:53.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:53.602 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:41:53.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:53.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:53.642420+0000) 2024-03-06T15:41:54.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:54.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:54.642675+0000) 2024-03-06T15:41:55.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:55.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:55.642987+0000) 2024-03-06T15:41:56.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:56.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:56.643367+0000) 2024-03-06T15:41:57.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:57.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:57.643650+0000) 2024-03-06T15:41:58.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:58.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:58.643885+0000) 2024-03-06T15:41:59.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:41:59.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:41:59.644176+0000) 2024-03-06T15:42:00.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:00.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:00.644403+0000) 2024-03-06T15:42:01.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:01.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:01.644632+0000) 2024-03-06T15:42:02.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:02.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:02.644858+0000) 2024-03-06T15:42:03.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:03.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:03.645148+0000) 2024-03-06T15:42:04.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:04.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:04.645387+0000) 2024-03-06T15:42:05.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:05.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:05.645637+0000) 2024-03-06T15:42:06.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:06.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:06.645916+0000) 2024-03-06T15:42:07.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:07.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:07.646206+0000) 2024-03-06T15:42:08.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:08.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:08.646419+0000) 2024-03-06T15:42:09.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:09.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:09.646646+0000) 2024-03-06T15:42:10.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:10.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:10.646862+0000) 2024-03-06T15:42:11.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:11.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:11.647170+0000) 2024-03-06T15:42:12.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:12.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:12.647451+0000) 2024-03-06T15:42:13.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:13.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:13.647719+0000) 2024-03-06T15:42:14.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:14.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:14.647962+0000) 2024-03-06T15:42:15.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:15.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:15.648189+0000) 2024-03-06T15:42:16.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:16.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:16.648410+0000) 2024-03-06T15:42:17.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:17.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:17.648689+0000) 2024-03-06T15:42:18.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:18.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:18.648927+0000) 2024-03-06T15:42:19.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:19.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:19.649160+0000) 2024-03-06T15:42:20.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:20.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:20.649390+0000) 2024-03-06T15:42:21.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:21.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:21.649613+0000) 2024-03-06T15:42:22.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:22.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:22.649884+0000) 2024-03-06T15:42:23.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:23.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:23.650160+0000) 2024-03-06T15:42:23.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:23.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:24.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:24.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:24.650434+0000) 2024-03-06T15:42:25.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:25.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:25.650734+0000) 2024-03-06T15:42:26.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:26.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:26.651052+0000) 2024-03-06T15:42:27.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:27.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:27.651238+0000) 2024-03-06T15:42:28.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:28.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:28.651464+0000) 2024-03-06T15:42:29.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:29.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:29.651745+0000) 2024-03-06T15:42:30.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:30.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:30.651996+0000) 2024-03-06T15:42:31.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:31.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:31.652344+0000) 2024-03-06T15:42:32.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:32.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:32.652615+0000) 2024-03-06T15:42:33.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:33.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:33.652824+0000) 2024-03-06T15:42:34.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:34.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:34.653083+0000) 2024-03-06T15:42:35.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:35.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:35.653318+0000) 2024-03-06T15:42:36.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:36.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:36.653588+0000) 2024-03-06T15:42:37.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:37.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:37.653884+0000) 2024-03-06T15:42:38.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:38.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:38.654167+0000) 2024-03-06T15:42:39.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:39.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:39.654416+0000) 2024-03-06T15:42:40.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:40.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:40.654646+0000) 2024-03-06T15:42:41.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:41.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:41.654893+0000) 2024-03-06T15:42:42.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:42.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:42.655176+0000) 2024-03-06T15:42:43.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:43.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:43.655428+0000) 2024-03-06T15:42:44.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:44.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:44.655654+0000) 2024-03-06T15:42:45.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:45.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:45.655887+0000) 2024-03-06T15:42:46.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:46.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:46.656117+0000) 2024-03-06T15:42:47.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:47.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:47.656352+0000) 2024-03-06T15:42:48.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:48.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:48.656580+0000) 2024-03-06T15:42:49.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:49.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:49.656838+0000) 2024-03-06T15:42:50.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:50.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:50.657157+0000) 2024-03-06T15:42:51.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:51.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:51.657445+0000) 2024-03-06T15:42:52.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:52.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:52.657652+0000) 2024-03-06T15:42:53.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:53.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:53.657916+0000) 2024-03-06T15:42:53.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:53.871 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:42:54.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:54.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:54.658150+0000) 2024-03-06T15:42:55.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:55.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:55.658384+0000) 2024-03-06T15:42:56.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:56.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:56.658642+0000) 2024-03-06T15:42:57.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:57.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:57.658918+0000) 2024-03-06T15:42:58.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:58.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:58.659183+0000) 2024-03-06T15:42:59.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:42:59.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:42:59.659396+0000) 2024-03-06T15:43:00.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:00.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:00.659635+0000) 2024-03-06T15:43:01.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:01.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:01.659851+0000) 2024-03-06T15:43:02.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:02.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:02.660133+0000) 2024-03-06T15:43:03.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:03.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:03.660363+0000) 2024-03-06T15:43:04.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:04.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:04.660585+0000) 2024-03-06T15:43:05.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:05.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:05.660839+0000) 2024-03-06T15:43:06.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:06.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:06.661044+0000) 2024-03-06T15:43:07.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:07.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:07.661288+0000) 2024-03-06T15:43:08.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:08.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:08.661477+0000) 2024-03-06T15:43:09.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:09.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:09.661712+0000) 2024-03-06T15:43:10.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:10.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:10.661930+0000) 2024-03-06T15:43:11.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:11.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:11.662208+0000) 2024-03-06T15:43:12.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:12.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:12.662454+0000) 2024-03-06T15:43:13.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:13.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:13.662735+0000) 2024-03-06T15:43:14.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:14.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:14.663042+0000) 2024-03-06T15:43:15.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:15.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:15.663305+0000) 2024-03-06T15:43:16.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:16.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:16.663568+0000) 2024-03-06T15:43:17.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:17.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:17.663842+0000) 2024-03-06T15:43:18.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:18.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:18.664146+0000) 2024-03-06T15:43:19.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:19.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:19.664405+0000) 2024-03-06T15:43:20.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:20.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:20.664626+0000) 2024-03-06T15:43:21.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:21.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:21.664849+0000) 2024-03-06T15:43:22.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:22.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:22.665146+0000) 2024-03-06T15:43:23.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:23.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:23.665396+0000) 2024-03-06T15:43:24.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:24.013 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:24.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:24.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:24.665660+0000) 2024-03-06T15:43:25.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:25.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:25.665941+0000) 2024-03-06T15:43:26.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:26.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:26.666166+0000) 2024-03-06T15:43:27.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:27.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:27.666404+0000) 2024-03-06T15:43:28.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:28.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:28.666638+0000) 2024-03-06T15:43:29.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:29.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:29.666934+0000) 2024-03-06T15:43:30.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:30.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:30.667206+0000) 2024-03-06T15:43:31.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:31.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:31.667415+0000) 2024-03-06T15:43:32.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:32.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:32.667648+0000) 2024-03-06T15:43:33.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:33.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:33.667888+0000) 2024-03-06T15:43:34.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:34.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:34.668114+0000) 2024-03-06T15:43:35.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:35.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:35.668360+0000) 2024-03-06T15:43:36.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:36.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:36.668626+0000) 2024-03-06T15:43:37.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:37.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:37.668904+0000) 2024-03-06T15:43:38.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:38.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:38.669181+0000) 2024-03-06T15:43:39.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:39.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:39.669435+0000) 2024-03-06T15:43:40.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:40.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:40.669644+0000) 2024-03-06T15:43:41.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:41.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:41.669879+0000) 2024-03-06T15:43:42.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:42.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:42.670122+0000) 2024-03-06T15:43:43.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:43.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:43.670360+0000) 2024-03-06T15:43:44.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:44.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:44.670585+0000) 2024-03-06T15:43:45.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:45.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:45.670820+0000) 2024-03-06T15:43:46.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:46.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:46.671092+0000) 2024-03-06T15:43:47.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:47.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:47.671337+0000) 2024-03-06T15:43:48.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:48.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:48.671573+0000) 2024-03-06T15:43:49.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:49.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:49.671853+0000) 2024-03-06T15:43:50.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:50.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:50.672158+0000) 2024-03-06T15:43:51.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:51.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:51.672434+0000) 2024-03-06T15:43:52.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:52.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:52.672706+0000) 2024-03-06T15:43:53.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:53.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:53.672990+0000) 2024-03-06T15:43:54.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:54.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:43:54.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:54.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:54.673276+0000) 2024-03-06T15:43:55.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:55.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:55.673553+0000) 2024-03-06T15:43:56.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:56.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:56.673843+0000) 2024-03-06T15:43:57.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:57.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:57.674152+0000) 2024-03-06T15:43:58.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:58.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:58.674416+0000) 2024-03-06T15:43:59.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:43:59.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:43:59.674687+0000) 2024-03-06T15:44:00.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:00.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:00.674966+0000) 2024-03-06T15:44:01.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:01.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:01.675248+0000) 2024-03-06T15:44:02.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:02.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:02.675515+0000) 2024-03-06T15:44:03.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:03.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:03.675764+0000) 2024-03-06T15:44:04.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:04.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:04.676071+0000) 2024-03-06T15:44:05.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:05.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:05.676376+0000) 2024-03-06T15:44:06.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:06.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:06.676640+0000) 2024-03-06T15:44:07.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:07.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:07.676919+0000) 2024-03-06T15:44:08.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:08.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:08.677200+0000) 2024-03-06T15:44:09.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:09.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:09.677429+0000) 2024-03-06T15:44:10.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:10.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:10.677675+0000) 2024-03-06T15:44:11.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:11.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:11.677865+0000) 2024-03-06T15:44:12.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:12.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:12.678156+0000) 2024-03-06T15:44:13.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:13.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:13.678391+0000) 2024-03-06T15:44:14.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:14.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:14.678616+0000) 2024-03-06T15:44:15.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:15.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:15.678862+0000) 2024-03-06T15:44:16.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:16.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:16.679118+0000) 2024-03-06T15:44:17.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:17.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:17.679354+0000) 2024-03-06T15:44:18.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:18.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:18.679579+0000) 2024-03-06T15:44:19.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:19.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:19.679787+0000) 2024-03-06T15:44:20.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:20.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:20.680041+0000) 2024-03-06T15:44:21.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:21.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:21.680236+0000) 2024-03-06T15:44:22.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:22.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:22.680482+0000) 2024-03-06T15:44:23.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:23.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:23.680727+0000) 2024-03-06T15:44:24.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:24.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:24.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:24.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:24.680943+0000) 2024-03-06T15:44:25.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:25.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:25.681227+0000) 2024-03-06T15:44:26.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:26.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:26.681400+0000) 2024-03-06T15:44:27.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:27.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:27.681630+0000) 2024-03-06T15:44:28.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:28.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:28.681858+0000) 2024-03-06T15:44:29.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:29.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:29.682145+0000) 2024-03-06T15:44:30.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:30.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:30.682416+0000) 2024-03-06T15:44:31.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:31.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:31.682700+0000) 2024-03-06T15:44:32.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:32.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:32.682970+0000) 2024-03-06T15:44:33.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:33.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:33.683271+0000) 2024-03-06T15:44:34.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:34.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:34.683478+0000) 2024-03-06T15:44:35.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:35.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:35.683711+0000) 2024-03-06T15:44:36.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:36.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:36.683939+0000) 2024-03-06T15:44:37.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:37.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:37.684191+0000) 2024-03-06T15:44:38.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:38.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:38.684423+0000) 2024-03-06T15:44:39.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:39.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:39.684700+0000) 2024-03-06T15:44:40.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:40.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:40.684922+0000) 2024-03-06T15:44:41.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:41.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:41.685156+0000) 2024-03-06T15:44:42.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:42.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:42.685392+0000) 2024-03-06T15:44:43.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:43.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:43.685621+0000) 2024-03-06T15:44:44.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:44.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:44.685859+0000) 2024-03-06T15:44:45.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:45.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:45.686146+0000) 2024-03-06T15:44:46.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:46.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:46.686417+0000) 2024-03-06T15:44:47.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:47.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:47.686703+0000) 2024-03-06T15:44:48.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:48.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:48.686938+0000) 2024-03-06T15:44:49.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:49.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:49.687195+0000) 2024-03-06T15:44:50.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:50.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:50.687455+0000) 2024-03-06T15:44:51.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:51.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:51.687738+0000) 2024-03-06T15:44:52.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:52.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:52.688066+0000) 2024-03-06T15:44:53.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:53.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:53.688301+0000) 2024-03-06T15:44:54.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:54.494 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:44:54.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:54.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:54.688536+0000) 2024-03-06T15:44:55.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:55.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:55.688822+0000) 2024-03-06T15:44:56.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:56.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:56.689060+0000) 2024-03-06T15:44:57.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:57.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:57.689304+0000) 2024-03-06T15:44:58.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:58.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:58.689529+0000) 2024-03-06T15:44:59.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:44:59.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:44:59.689752+0000) 2024-03-06T15:45:00.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:00.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:00.689982+0000) 2024-03-06T15:45:01.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:01.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:01.690209+0000) 2024-03-06T15:45:02.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:02.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:02.690461+0000) 2024-03-06T15:45:03.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:03.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:03.690698+0000) 2024-03-06T15:45:04.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:04.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:04.690938+0000) 2024-03-06T15:45:05.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:05.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:05.691143+0000) 2024-03-06T15:45:06.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:06.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:06.691329+0000) 2024-03-06T15:45:07.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:07.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:07.691560+0000) 2024-03-06T15:45:08.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:08.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:08.691796+0000) 2024-03-06T15:45:09.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:09.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:09.692047+0000) 2024-03-06T15:45:10.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:10.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:10.692242+0000) 2024-03-06T15:45:11.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:11.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:11.692445+0000) 2024-03-06T15:45:12.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:12.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:12.692674+0000) 2024-03-06T15:45:13.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:13.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:13.692895+0000) 2024-03-06T15:45:14.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:14.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:14.693126+0000) 2024-03-06T15:45:15.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:15.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:15.693364+0000) 2024-03-06T15:45:16.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:16.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:16.693589+0000) 2024-03-06T15:45:17.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:17.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:17.693830+0000) 2024-03-06T15:45:18.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:18.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:18.694089+0000) 2024-03-06T15:45:19.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:19.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:19.694323+0000) 2024-03-06T15:45:20.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:20.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:20.694603+0000) 2024-03-06T15:45:21.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:21.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:21.694869+0000) 2024-03-06T15:45:22.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:22.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:22.695173+0000) 2024-03-06T15:45:23.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:23.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:23.695418+0000) 2024-03-06T15:45:24.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:24.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:24.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:24.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:24.695597+0000) 2024-03-06T15:45:25.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:25.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:25.695855+0000) 2024-03-06T15:45:26.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:26.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:26.696100+0000) 2024-03-06T15:45:27.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:27.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:27.696363+0000) 2024-03-06T15:45:28.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:28.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:28.696558+0000) 2024-03-06T15:45:29.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:29.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:29.696794+0000) 2024-03-06T15:45:30.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:30.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:30.697042+0000) 2024-03-06T15:45:31.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:31.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:31.697288+0000) 2024-03-06T15:45:32.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:32.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:32.697522+0000) 2024-03-06T15:45:33.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:33.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:33.697744+0000) 2024-03-06T15:45:34.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:34.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:34.698064+0000) 2024-03-06T15:45:35.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:35.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:35.698329+0000) 2024-03-06T15:45:36.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:36.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:36.698592+0000) 2024-03-06T15:45:37.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:37.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:37.698854+0000) 2024-03-06T15:45:38.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:38.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:38.699140+0000) 2024-03-06T15:45:39.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:39.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:39.699333+0000) 2024-03-06T15:45:40.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:40.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:40.699540+0000) 2024-03-06T15:45:41.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:41.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:41.699772+0000) 2024-03-06T15:45:42.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:42.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:42.700038+0000) 2024-03-06T15:45:43.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:43.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:43.700233+0000) 2024-03-06T15:45:44.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:44.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:44.700444+0000) 2024-03-06T15:45:45.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:45.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:45.700677+0000) 2024-03-06T15:45:46.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:46.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:46.700895+0000) 2024-03-06T15:45:47.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:47.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:47.701149+0000) 2024-03-06T15:45:48.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:48.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:48.701385+0000) 2024-03-06T15:45:49.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:49.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:49.701594+0000) 2024-03-06T15:45:50.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:50.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:50.701781+0000) 2024-03-06T15:45:51.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:51.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:51.702048+0000) 2024-03-06T15:45:52.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:52.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:52.702276+0000) 2024-03-06T15:45:53.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:53.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:53.702496+0000) 2024-03-06T15:45:54.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:54.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:54.702693+0000) 2024-03-06T15:45:54.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:54.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:45:55.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:55.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:55.702899+0000) 2024-03-06T15:45:56.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:56.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:56.703125+0000) 2024-03-06T15:45:57.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:57.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:57.703366+0000) 2024-03-06T15:45:58.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:58.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:58.703547+0000) 2024-03-06T15:45:59.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:45:59.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:45:59.703761+0000) 2024-03-06T15:46:00.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:00.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:00.703982+0000) 2024-03-06T15:46:01.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:01.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:01.704210+0000) 2024-03-06T15:46:02.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:02.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:02.704416+0000) 2024-03-06T15:46:03.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:03.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:03.704629+0000) 2024-03-06T15:46:04.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:04.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:04.704856+0000) 2024-03-06T15:46:05.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:05.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:05.705134+0000) 2024-03-06T15:46:06.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:06.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:06.705319+0000) 2024-03-06T15:46:07.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:07.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:07.705542+0000) 2024-03-06T15:46:08.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:08.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:08.705761+0000) 2024-03-06T15:46:09.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:09.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:09.705986+0000) 2024-03-06T15:46:10.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:10.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:10.706230+0000) 2024-03-06T15:46:11.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:11.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:11.706434+0000) 2024-03-06T15:46:12.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:12.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:12.706657+0000) 2024-03-06T15:46:13.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:13.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:13.706880+0000) 2024-03-06T15:46:14.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:14.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:14.707123+0000) 2024-03-06T15:46:15.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:15.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:15.707355+0000) 2024-03-06T15:46:16.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:16.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:16.707553+0000) 2024-03-06T15:46:17.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:17.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:17.707802+0000) 2024-03-06T15:46:18.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:18.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:18.708061+0000) 2024-03-06T15:46:19.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:19.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:19.708259+0000) 2024-03-06T15:46:20.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:20.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:20.708469+0000) 2024-03-06T15:46:21.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:21.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:21.708743+0000) 2024-03-06T15:46:22.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:22.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:22.708981+0000) 2024-03-06T15:46:23.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:23.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:23.709236+0000) 2024-03-06T15:46:24.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:24.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:24.709452+0000) 2024-03-06T15:46:24.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:24.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:25.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:25.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:25.709671+0000) 2024-03-06T15:46:26.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:26.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:26.709895+0000) 2024-03-06T15:46:27.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:27.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:27.710160+0000) 2024-03-06T15:46:28.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:28.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:28.710370+0000) 2024-03-06T15:46:29.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:29.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:29.710655+0000) 2024-03-06T15:46:30.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:30.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:30.710935+0000) 2024-03-06T15:46:31.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:31.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:31.711170+0000) 2024-03-06T15:46:32.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:32.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:32.711404+0000) 2024-03-06T15:46:33.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:33.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:33.711616+0000) 2024-03-06T15:46:34.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:34.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:34.711806+0000) 2024-03-06T15:46:35.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:35.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:35.712091+0000) 2024-03-06T15:46:36.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:36.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:36.712282+0000) 2024-03-06T15:46:37.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:37.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:37.712508+0000) 2024-03-06T15:46:38.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:38.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:38.712730+0000) 2024-03-06T15:46:39.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:39.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:39.712955+0000) 2024-03-06T15:46:40.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:40.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:40.713155+0000) 2024-03-06T15:46:41.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:41.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:41.713346+0000) 2024-03-06T15:46:42.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:42.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:42.713568+0000) 2024-03-06T15:46:43.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:43.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:43.713803+0000) 2024-03-06T15:46:44.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:44.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:44.714078+0000) 2024-03-06T15:46:45.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:45.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:45.714282+0000) 2024-03-06T15:46:46.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:46.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:46.714487+0000) 2024-03-06T15:46:47.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:47.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:47.714704+0000) 2024-03-06T15:46:48.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:48.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:48.714927+0000) 2024-03-06T15:46:49.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:49.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:49.715163+0000) 2024-03-06T15:46:50.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:50.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:50.715361+0000) 2024-03-06T15:46:51.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:51.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:51.715583+0000) 2024-03-06T15:46:52.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:52.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:52.715823+0000) 2024-03-06T15:46:53.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:53.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:53.716091+0000) 2024-03-06T15:46:54.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:54.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:54.716283+0000) 2024-03-06T15:46:55.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:55.147 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:46:55.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:55.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:55.716510+0000) 2024-03-06T15:46:56.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:56.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:56.716733+0000) 2024-03-06T15:46:57.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:57.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:57.716965+0000) 2024-03-06T15:46:58.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:58.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:58.717156+0000) 2024-03-06T15:46:59.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:46:59.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:46:59.717401+0000) 2024-03-06T15:47:00.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:00.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:00.717643+0000) 2024-03-06T15:47:01.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:01.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:01.717927+0000) 2024-03-06T15:47:02.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:02.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:02.718174+0000) 2024-03-06T15:47:03.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:03.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:03.718458+0000) 2024-03-06T15:47:04.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:04.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:04.718651+0000) 2024-03-06T15:47:05.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:05.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:05.718898+0000) 2024-03-06T15:47:06.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:06.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:06.719146+0000) 2024-03-06T15:47:07.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:07.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:07.719376+0000) 2024-03-06T15:47:08.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:08.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:08.719620+0000) 2024-03-06T15:47:09.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:09.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:09.719852+0000) 2024-03-06T15:47:10.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:10.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:10.720054+0000) 2024-03-06T15:47:11.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:11.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:11.720247+0000) 2024-03-06T15:47:12.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:12.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:12.720467+0000) 2024-03-06T15:47:13.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:13.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:13.720696+0000) 2024-03-06T15:47:14.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:14.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:14.720918+0000) 2024-03-06T15:47:15.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:15.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:15.721177+0000) 2024-03-06T15:47:16.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:16.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:16.721341+0000) 2024-03-06T15:47:17.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:17.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:17.721550+0000) 2024-03-06T15:47:18.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:18.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:18.721777+0000) 2024-03-06T15:47:19.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:19.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:19.721973+0000) 2024-03-06T15:47:20.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:20.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:20.722201+0000) 2024-03-06T15:47:21.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:21.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:21.722428+0000) 2024-03-06T15:47:22.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:22.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:22.722663+0000) 2024-03-06T15:47:23.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:23.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:23.722896+0000) 2024-03-06T15:47:24.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:24.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:24.723076+0000) 2024-03-06T15:47:25.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:25.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:25.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:25.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:25.723290+0000) 2024-03-06T15:47:26.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:26.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:26.723471+0000) 2024-03-06T15:47:27.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:27.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:27.723692+0000) 2024-03-06T15:47:28.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:28.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:28.723928+0000) 2024-03-06T15:47:29.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:29.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:29.724168+0000) 2024-03-06T15:47:30.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:30.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:30.724387+0000) 2024-03-06T15:47:31.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:31.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:31.724624+0000) 2024-03-06T15:47:32.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:32.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:32.724872+0000) 2024-03-06T15:47:33.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:33.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:33.725125+0000) 2024-03-06T15:47:34.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:34.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:34.725401+0000) 2024-03-06T15:47:35.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:35.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:35.725682+0000) 2024-03-06T15:47:36.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:36.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:36.725961+0000) 2024-03-06T15:47:37.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:37.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:37.726264+0000) 2024-03-06T15:47:38.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:38.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:38.726539+0000) 2024-03-06T15:47:39.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:39.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:39.726821+0000) 2024-03-06T15:47:40.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:40.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:40.727142+0000) 2024-03-06T15:47:41.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:41.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:41.727317+0000) 2024-03-06T15:47:42.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:42.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:42.727573+0000) 2024-03-06T15:47:43.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:43.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:43.727836+0000) 2024-03-06T15:47:44.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:44.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:44.728135+0000) 2024-03-06T15:47:45.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:45.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:45.728414+0000) 2024-03-06T15:47:46.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:46.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:46.728683+0000) 2024-03-06T15:47:47.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:47.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:47.728934+0000) 2024-03-06T15:47:48.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:48.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:48.729204+0000) 2024-03-06T15:47:49.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:49.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:49.729419+0000) 2024-03-06T15:47:50.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:50.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:50.729606+0000) 2024-03-06T15:47:51.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:51.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:51.729826+0000) 2024-03-06T15:47:52.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:52.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:52.730070+0000) 2024-03-06T15:47:53.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:53.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:53.730277+0000) 2024-03-06T15:47:54.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:54.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:54.730497+0000) 2024-03-06T15:47:55.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:55.533 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:47:55.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:55.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:55.730726+0000) 2024-03-06T15:47:56.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:56.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:56.730978+0000) 2024-03-06T15:47:57.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:57.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:57.731270+0000) 2024-03-06T15:47:58.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:58.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:58.731490+0000) 2024-03-06T15:47:59.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:47:59.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:47:59.731732+0000) 2024-03-06T15:48:00.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:00.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:00.731970+0000) 2024-03-06T15:48:01.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:01.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:01.732218+0000) 2024-03-06T15:48:02.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:02.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:02.732428+0000) 2024-03-06T15:48:03.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:03.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:03.732660+0000) 2024-03-06T15:48:04.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:04.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:04.732889+0000) 2024-03-06T15:48:05.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:05.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:05.733153+0000) 2024-03-06T15:48:06.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:06.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:06.733339+0000) 2024-03-06T15:48:07.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:07.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:07.733561+0000) 2024-03-06T15:48:08.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:08.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:08.733797+0000) 2024-03-06T15:48:09.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:09.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:09.734037+0000) 2024-03-06T15:48:10.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:10.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:10.734261+0000) 2024-03-06T15:48:11.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:11.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:11.734477+0000) 2024-03-06T15:48:12.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:12.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:12.734704+0000) 2024-03-06T15:48:13.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:13.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:13.734927+0000) 2024-03-06T15:48:14.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:14.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:14.735153+0000) 2024-03-06T15:48:15.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:15.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:15.735354+0000) 2024-03-06T15:48:16.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:16.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:16.735549+0000) 2024-03-06T15:48:17.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:17.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:17.735792+0000) 2024-03-06T15:48:18.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:18.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:18.736087+0000) 2024-03-06T15:48:19.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:19.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:19.736280+0000) 2024-03-06T15:48:20.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:20.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:20.736495+0000) 2024-03-06T15:48:21.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:21.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:21.736731+0000) 2024-03-06T15:48:22.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:22.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:22.736953+0000) 2024-03-06T15:48:23.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:23.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:23.737174+0000) 2024-03-06T15:48:24.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:24.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:24.737406+0000) 2024-03-06T15:48:25.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:25.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:25.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:25.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:25.737689+0000) 2024-03-06T15:48:26.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:26.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:26.737958+0000) 2024-03-06T15:48:27.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:27.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:27.738164+0000) 2024-03-06T15:48:28.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:28.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:28.738412+0000) 2024-03-06T15:48:29.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:29.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:29.738691+0000) 2024-03-06T15:48:30.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:30.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:30.738969+0000) 2024-03-06T15:48:31.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:31.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:31.739264+0000) 2024-03-06T15:48:32.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:32.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:32.739506+0000) 2024-03-06T15:48:33.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:33.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:33.739791+0000) 2024-03-06T15:48:34.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:34.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:34.740055+0000) 2024-03-06T15:48:35.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:35.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:35.740350+0000) 2024-03-06T15:48:36.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:36.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:36.740570+0000) 2024-03-06T15:48:37.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:37.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:37.740748+0000) 2024-03-06T15:48:38.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:38.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:38.740974+0000) 2024-03-06T15:48:39.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:39.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:39.741202+0000) 2024-03-06T15:48:40.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:40.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:40.741402+0000) 2024-03-06T15:48:41.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:41.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:41.741604+0000) 2024-03-06T15:48:42.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:42.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:42.741819+0000) 2024-03-06T15:48:43.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:43.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:43.742090+0000) 2024-03-06T15:48:44.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:44.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:44.742333+0000) 2024-03-06T15:48:45.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:45.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:45.742572+0000) 2024-03-06T15:48:46.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:46.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:46.742809+0000) 2024-03-06T15:48:47.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:47.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:47.743081+0000) 2024-03-06T15:48:48.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:48.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:48.743310+0000) 2024-03-06T15:48:49.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:49.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:49.743505+0000) 2024-03-06T15:48:50.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:50.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:50.743788+0000) 2024-03-06T15:48:51.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:51.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:51.743972+0000) 2024-03-06T15:48:52.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:52.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:52.744274+0000) 2024-03-06T15:48:53.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:53.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:53.744543+0000) 2024-03-06T15:48:54.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:54.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:54.744770+0000) 2024-03-06T15:48:55.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:55.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:55.745124+0000) 2024-03-06T15:48:55.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:55.907 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:48:56.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:56.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:56.745416+0000) 2024-03-06T15:48:57.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:57.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:57.745611+0000) 2024-03-06T15:48:58.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:58.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:58.745856+0000) 2024-03-06T15:48:59.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:48:59.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:48:59.746129+0000) 2024-03-06T15:49:00.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:00.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:00.746342+0000) 2024-03-06T15:49:01.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:01.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:01.746552+0000) 2024-03-06T15:49:02.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:02.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:02.746771+0000) 2024-03-06T15:49:03.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:03.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:03.747054+0000) 2024-03-06T15:49:04.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:04.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:04.747292+0000) 2024-03-06T15:49:05.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:05.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:05.747522+0000) 2024-03-06T15:49:06.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:06.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:06.747768+0000) 2024-03-06T15:49:07.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:07.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:07.748049+0000) 2024-03-06T15:49:08.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:08.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:08.748366+0000) 2024-03-06T15:49:09.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:09.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:09.748645+0000) 2024-03-06T15:49:10.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:10.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:10.748928+0000) 2024-03-06T15:49:11.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:11.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:11.749204+0000) 2024-03-06T15:49:12.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:12.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:12.749434+0000) 2024-03-06T15:49:13.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:13.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:13.749675+0000) 2024-03-06T15:49:14.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:14.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:14.749914+0000) 2024-03-06T15:49:15.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:15.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:15.750210+0000) 2024-03-06T15:49:16.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:16.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:16.750487+0000) 2024-03-06T15:49:17.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:17.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:17.750693+0000) 2024-03-06T15:49:18.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:18.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:18.750941+0000) 2024-03-06T15:49:19.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:19.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:19.751166+0000) 2024-03-06T15:49:20.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:20.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:20.751404+0000) 2024-03-06T15:49:21.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:21.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:21.751626+0000) 2024-03-06T15:49:22.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:22.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:22.751865+0000) 2024-03-06T15:49:23.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:23.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:23.752166+0000) 2024-03-06T15:49:24.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:24.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:24.752444+0000) 2024-03-06T15:49:25.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:25.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:25.752651+0000) 2024-03-06T15:49:26.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:26.103 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:26.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:26.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:26.752934+0000) 2024-03-06T15:49:27.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:27.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:27.753167+0000) 2024-03-06T15:49:28.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:28.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:28.753450+0000) 2024-03-06T15:49:29.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:29.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:29.753728+0000) 2024-03-06T15:49:30.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:30.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:30.754056+0000) 2024-03-06T15:49:31.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:31.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:31.754304+0000) 2024-03-06T15:49:32.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:32.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:32.754605+0000) 2024-03-06T15:49:33.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:33.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:33.754887+0000) 2024-03-06T15:49:34.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:34.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:34.755160+0000) 2024-03-06T15:49:35.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:35.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:35.755451+0000) 2024-03-06T15:49:36.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:36.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:36.755665+0000) 2024-03-06T15:49:37.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:37.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:37.755910+0000) 2024-03-06T15:49:38.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:38.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:38.756277+0000) 2024-03-06T15:49:39.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:39.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:39.756534+0000) 2024-03-06T15:49:40.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:40.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:40.756798+0000) 2024-03-06T15:49:41.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:41.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:41.757115+0000) 2024-03-06T15:49:42.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:42.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:42.757401+0000) 2024-03-06T15:49:43.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:43.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:43.757697+0000) 2024-03-06T15:49:44.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:44.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:44.757938+0000) 2024-03-06T15:49:45.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:45.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:45.758209+0000) 2024-03-06T15:49:46.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:46.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:46.758490+0000) 2024-03-06T15:49:47.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:47.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:47.758707+0000) 2024-03-06T15:49:48.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:48.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:48.758953+0000) 2024-03-06T15:49:49.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:49.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:49.759202+0000) 2024-03-06T15:49:50.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:50.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:50.759447+0000) 2024-03-06T15:49:51.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:51.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:51.759687+0000) 2024-03-06T15:49:52.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:52.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:52.759888+0000) 2024-03-06T15:49:53.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:53.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:53.760153+0000) 2024-03-06T15:49:54.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:54.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:54.760426+0000) 2024-03-06T15:49:55.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:55.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:55.760714+0000) 2024-03-06T15:49:56.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:56.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:49:56.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:56.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:56.760950+0000) 2024-03-06T15:49:57.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:57.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:57.761189+0000) 2024-03-06T15:49:58.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:58.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:58.761362+0000) 2024-03-06T15:49:59.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:49:59.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:49:59.761591+0000) 2024-03-06T15:50:00.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:00.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:00.761809+0000) 2024-03-06T15:50:01.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:01.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:01.762069+0000) 2024-03-06T15:50:02.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:02.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:02.762273+0000) 2024-03-06T15:50:03.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:03.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:03.762547+0000) 2024-03-06T15:50:04.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:04.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:04.762811+0000) 2024-03-06T15:50:05.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:05.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:05.763129+0000) 2024-03-06T15:50:06.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:06.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:06.763357+0000) 2024-03-06T15:50:07.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:07.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:07.763571+0000) 2024-03-06T15:50:08.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:08.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:08.763812+0000) 2024-03-06T15:50:09.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:09.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:09.764100+0000) 2024-03-06T15:50:10.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:10.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:10.764313+0000) 2024-03-06T15:50:11.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:11.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:11.764549+0000) 2024-03-06T15:50:12.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:12.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:12.764785+0000) 2024-03-06T15:50:13.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:13.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:13.765108+0000) 2024-03-06T15:50:14.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:14.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:14.765390+0000) 2024-03-06T15:50:15.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:15.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:15.765675+0000) 2024-03-06T15:50:16.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:16.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:16.765954+0000) 2024-03-06T15:50:17.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:17.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:17.766216+0000) 2024-03-06T15:50:18.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:18.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:18.766448+0000) 2024-03-06T15:50:19.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:19.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:19.766675+0000) 2024-03-06T15:50:20.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:20.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:20.766940+0000) 2024-03-06T15:50:21.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:21.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:21.767221+0000) 2024-03-06T15:50:22.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:22.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:22.767396+0000) 2024-03-06T15:50:23.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:23.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:23.767643+0000) 2024-03-06T15:50:24.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:24.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:24.767884+0000) 2024-03-06T15:50:25.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:25.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:25.768119+0000) 2024-03-06T15:50:26.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:26.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:26.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:26.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:26.768412+0000) 2024-03-06T15:50:27.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:27.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:27.768659+0000) 2024-03-06T15:50:28.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:28.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:28.768900+0000) 2024-03-06T15:50:29.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:29.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:29.769157+0000) 2024-03-06T15:50:30.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:30.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:30.769365+0000) 2024-03-06T15:50:31.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:31.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:31.769600+0000) 2024-03-06T15:50:32.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:32.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:32.769823+0000) 2024-03-06T15:50:33.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:33.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:33.770104+0000) 2024-03-06T15:50:34.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:34.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:34.770329+0000) 2024-03-06T15:50:35.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:35.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:35.770528+0000) 2024-03-06T15:50:36.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:36.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:36.770763+0000) 2024-03-06T15:50:37.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:37.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:37.770989+0000) 2024-03-06T15:50:38.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:38.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:38.771269+0000) 2024-03-06T15:50:39.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:39.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:39.771471+0000) 2024-03-06T15:50:40.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:40.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:40.771689+0000) 2024-03-06T15:50:41.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:41.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:41.771916+0000) 2024-03-06T15:50:42.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:42.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:42.772161+0000) 2024-03-06T15:50:43.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:43.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:43.772399+0000) 2024-03-06T15:50:44.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:44.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:44.772642+0000) 2024-03-06T15:50:45.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:45.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:45.772898+0000) 2024-03-06T15:50:46.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:46.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:46.773122+0000) 2024-03-06T15:50:47.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:47.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:47.773380+0000) 2024-03-06T15:50:48.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:48.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:48.773653+0000) 2024-03-06T15:50:49.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:49.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:49.773895+0000) 2024-03-06T15:50:50.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:50.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:50.774124+0000) 2024-03-06T15:50:51.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:51.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:51.774332+0000) 2024-03-06T15:50:52.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:52.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:52.774544+0000) 2024-03-06T15:50:53.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:53.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:53.774783+0000) 2024-03-06T15:50:54.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:54.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:54.775113+0000) 2024-03-06T15:50:55.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:55.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:55.775358+0000) 2024-03-06T15:50:56.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:56.618 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:50:56.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:56.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:56.775600+0000) 2024-03-06T15:50:57.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:57.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:57.775870+0000) 2024-03-06T15:50:58.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:58.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:58.776172+0000) 2024-03-06T15:50:59.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:50:59.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:50:59.776394+0000) 2024-03-06T15:51:00.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:00.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:00.776610+0000) 2024-03-06T15:51:01.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:01.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:01.776835+0000) 2024-03-06T15:51:02.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:02.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:02.777143+0000) 2024-03-06T15:51:03.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:03.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:03.777400+0000) 2024-03-06T15:51:04.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:04.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:04.777688+0000) 2024-03-06T15:51:05.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:05.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:05.777983+0000) 2024-03-06T15:51:06.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:06.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:06.778283+0000) 2024-03-06T15:51:07.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:07.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:07.778520+0000) 2024-03-06T15:51:08.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:08.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:08.778822+0000) 2024-03-06T15:51:09.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:09.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:09.779128+0000) 2024-03-06T15:51:10.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:10.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:10.779413+0000) 2024-03-06T15:51:11.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:11.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:11.779675+0000) 2024-03-06T15:51:12.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:12.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:12.779950+0000) 2024-03-06T15:51:13.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:13.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:13.780237+0000) 2024-03-06T15:51:14.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:14.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:14.780497+0000) 2024-03-06T15:51:15.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:15.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:15.780792+0000) 2024-03-06T15:51:16.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:16.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:16.781100+0000) 2024-03-06T15:51:17.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:17.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:17.781346+0000) 2024-03-06T15:51:18.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:18.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:18.781570+0000) 2024-03-06T15:51:19.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:19.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:19.781846+0000) 2024-03-06T15:51:20.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:20.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:20.782146+0000) 2024-03-06T15:51:21.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:21.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:21.782424+0000) 2024-03-06T15:51:22.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:22.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:22.782623+0000) 2024-03-06T15:51:23.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:23.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:23.782855+0000) 2024-03-06T15:51:24.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:24.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:24.783148+0000) 2024-03-06T15:51:25.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:25.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:25.783401+0000) 2024-03-06T15:51:26.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:26.757 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:26.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:26.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:26.783659+0000) 2024-03-06T15:51:27.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:27.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:27.783913+0000) 2024-03-06T15:51:28.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:28.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:28.784211+0000) 2024-03-06T15:51:29.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:29.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:29.784448+0000) 2024-03-06T15:51:30.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:30.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:30.784715+0000) 2024-03-06T15:51:31.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:31.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:31.784970+0000) 2024-03-06T15:51:32.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:32.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:32.785262+0000) 2024-03-06T15:51:33.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:33.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:33.785505+0000) 2024-03-06T15:51:34.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:34.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:34.785733+0000) 2024-03-06T15:51:35.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:35.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:35.785997+0000) 2024-03-06T15:51:36.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:36.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:36.786290+0000) 2024-03-06T15:51:37.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:37.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:37.786515+0000) 2024-03-06T15:51:38.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:38.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:38.786773+0000) 2024-03-06T15:51:39.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:39.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:39.787069+0000) 2024-03-06T15:51:40.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:40.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:40.787372+0000) 2024-03-06T15:51:41.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:41.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:41.787610+0000) 2024-03-06T15:51:42.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:42.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:42.787850+0000) 2024-03-06T15:51:43.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:43.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:43.788112+0000) 2024-03-06T15:51:44.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:44.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:44.788339+0000) 2024-03-06T15:51:45.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:45.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:45.788581+0000) 2024-03-06T15:51:46.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:46.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:46.788811+0000) 2024-03-06T15:51:47.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:47.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:47.789092+0000) 2024-03-06T15:51:48.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:48.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:48.789342+0000) 2024-03-06T15:51:49.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:49.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:49.789575+0000) 2024-03-06T15:51:50.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:50.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:50.789815+0000) 2024-03-06T15:51:51.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:51.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:51.790095+0000) 2024-03-06T15:51:52.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:52.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:52.790327+0000) 2024-03-06T15:51:53.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:53.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:53.790599+0000) 2024-03-06T15:51:54.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:54.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:54.790876+0000) 2024-03-06T15:51:55.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:55.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:55.791186+0000) 2024-03-06T15:51:56.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:56.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:56.791556+0000) 2024-03-06T15:51:56.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:56.900 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:51:57.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:57.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:57.791833+0000) 2024-03-06T15:51:58.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:58.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:58.792140+0000) 2024-03-06T15:51:59.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:51:59.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:51:59.792390+0000) 2024-03-06T15:52:00.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:00.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:00.792631+0000) 2024-03-06T15:52:01.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:01.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:01.792872+0000) 2024-03-06T15:52:02.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:02.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:02.793116+0000) 2024-03-06T15:52:03.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:03.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:03.793342+0000) 2024-03-06T15:52:04.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:04.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:04.793573+0000) 2024-03-06T15:52:05.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:05.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:05.793784+0000) 2024-03-06T15:52:06.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:06.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:06.793991+0000) 2024-03-06T15:52:07.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:07.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:07.794289+0000) 2024-03-06T15:52:08.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:08.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:08.794529+0000) 2024-03-06T15:52:09.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:09.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:09.794767+0000) 2024-03-06T15:52:10.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:10.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:10.795037+0000) 2024-03-06T15:52:11.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:11.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:11.795271+0000) 2024-03-06T15:52:12.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:12.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:12.795493+0000) 2024-03-06T15:52:13.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:13.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:13.795710+0000) 2024-03-06T15:52:14.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:14.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:14.795946+0000) 2024-03-06T15:52:15.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:15.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:15.796176+0000) 2024-03-06T15:52:16.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:16.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:16.796402+0000) 2024-03-06T15:52:17.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:17.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:17.796631+0000) 2024-03-06T15:52:18.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:18.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:18.796868+0000) 2024-03-06T15:52:19.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:19.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:19.797135+0000) 2024-03-06T15:52:20.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:20.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:20.797360+0000) 2024-03-06T15:52:21.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:21.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:21.797583+0000) 2024-03-06T15:52:22.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:22.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:22.797803+0000) 2024-03-06T15:52:23.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:23.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:23.798065+0000) 2024-03-06T15:52:24.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:24.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:24.798308+0000) 2024-03-06T15:52:25.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:25.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:25.798480+0000) 2024-03-06T15:52:26.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:26.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:26.798686+0000) 2024-03-06T15:52:27.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:27.119 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:27.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:27.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:27.798911+0000) 2024-03-06T15:52:28.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:28.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:28.799146+0000) 2024-03-06T15:52:29.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:29.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:29.799378+0000) 2024-03-06T15:52:30.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:30.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:30.799608+0000) 2024-03-06T15:52:31.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:31.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:31.799843+0000) 2024-03-06T15:52:32.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:32.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:32.800118+0000) 2024-03-06T15:52:33.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:33.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:33.800356+0000) 2024-03-06T15:52:34.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:34.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:34.800559+0000) 2024-03-06T15:52:35.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:35.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:35.800812+0000) 2024-03-06T15:52:36.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:36.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:36.801072+0000) 2024-03-06T15:52:37.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:37.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:37.801277+0000) 2024-03-06T15:52:38.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:38.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:38.801500+0000) 2024-03-06T15:52:39.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:39.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:39.801731+0000) 2024-03-06T15:52:40.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:40.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:40.801966+0000) 2024-03-06T15:52:41.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:41.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:41.802212+0000) 2024-03-06T15:52:42.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:42.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:42.802416+0000) 2024-03-06T15:52:43.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:43.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:43.802664+0000) 2024-03-06T15:52:44.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:44.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:44.802920+0000) 2024-03-06T15:52:45.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:45.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:45.803175+0000) 2024-03-06T15:52:46.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:46.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:46.803392+0000) 2024-03-06T15:52:47.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:47.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:47.803630+0000) 2024-03-06T15:52:48.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:48.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:48.803885+0000) 2024-03-06T15:52:49.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:49.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:49.804154+0000) 2024-03-06T15:52:50.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:50.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:50.804361+0000) 2024-03-06T15:52:51.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:51.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:51.804600+0000) 2024-03-06T15:52:52.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:52.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:52.804848+0000) 2024-03-06T15:52:53.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:53.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:53.805141+0000) 2024-03-06T15:52:54.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:54.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:54.805346+0000) 2024-03-06T15:52:55.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:55.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:55.805635+0000) 2024-03-06T15:52:56.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:56.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:56.805855+0000) 2024-03-06T15:52:57.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:57.391 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:52:57.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:57.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:57.806128+0000) 2024-03-06T15:52:58.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:58.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:58.806344+0000) 2024-03-06T15:52:59.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:52:59.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:52:59.806568+0000) 2024-03-06T15:53:00.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:00.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:00.806771+0000) 2024-03-06T15:53:01.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:01.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:01.806963+0000) 2024-03-06T15:53:02.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:02.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:02.807159+0000) 2024-03-06T15:53:03.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:03.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:03.807368+0000) 2024-03-06T15:53:04.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:04.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:04.807581+0000) 2024-03-06T15:53:05.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:05.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:05.807832+0000) 2024-03-06T15:53:06.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:06.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:06.808100+0000) 2024-03-06T15:53:07.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:07.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:07.808312+0000) 2024-03-06T15:53:08.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:08.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:08.808551+0000) 2024-03-06T15:53:09.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:09.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:09.808785+0000) 2024-03-06T15:53:10.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:10.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:10.809045+0000) 2024-03-06T15:53:11.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:11.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:11.809278+0000) 2024-03-06T15:53:12.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:12.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:12.809487+0000) 2024-03-06T15:53:13.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:13.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:13.809722+0000) 2024-03-06T15:53:14.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:14.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:14.809965+0000) 2024-03-06T15:53:15.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:15.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:15.810241+0000) 2024-03-06T15:53:16.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:16.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:16.810469+0000) 2024-03-06T15:53:17.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:17.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:17.810671+0000) 2024-03-06T15:53:18.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:18.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:18.810862+0000) 2024-03-06T15:53:19.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:19.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:19.811103+0000) 2024-03-06T15:53:20.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:20.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:20.811317+0000) 2024-03-06T15:53:21.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:21.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:21.811538+0000) 2024-03-06T15:53:22.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:22.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:22.811779+0000) 2024-03-06T15:53:23.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:23.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:23.812043+0000) 2024-03-06T15:53:24.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:24.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:24.812255+0000) 2024-03-06T15:53:25.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:25.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:25.812456+0000) 2024-03-06T15:53:26.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:26.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:26.812676+0000) 2024-03-06T15:53:27.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:27.595 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:27.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:27.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:27.812860+0000) 2024-03-06T15:53:28.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:28.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:28.813121+0000) 2024-03-06T15:53:29.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:29.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:29.813336+0000) 2024-03-06T15:53:30.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:30.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:30.813571+0000) 2024-03-06T15:53:31.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:31.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:31.813770+0000) 2024-03-06T15:53:32.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:32.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:32.813993+0000) 2024-03-06T15:53:33.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:33.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:33.814253+0000) 2024-03-06T15:53:34.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:34.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:34.814476+0000) 2024-03-06T15:53:35.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:35.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:35.814705+0000) 2024-03-06T15:53:36.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:36.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:36.814930+0000) 2024-03-06T15:53:37.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:37.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:37.815153+0000) 2024-03-06T15:53:38.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:38.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:38.815370+0000) 2024-03-06T15:53:39.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:39.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:39.815574+0000) 2024-03-06T15:53:40.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:40.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:40.815817+0000) 2024-03-06T15:53:41.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:41.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:41.816099+0000) 2024-03-06T15:53:42.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:42.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:42.816314+0000) 2024-03-06T15:53:43.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:43.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:43.816534+0000) 2024-03-06T15:53:44.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:44.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:44.816739+0000) 2024-03-06T15:53:45.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:45.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:45.817047+0000) 2024-03-06T15:53:46.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:46.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:46.817283+0000) 2024-03-06T15:53:47.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:47.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:47.817521+0000) 2024-03-06T15:53:48.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:48.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:48.817738+0000) 2024-03-06T15:53:49.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:49.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:49.818075+0000) 2024-03-06T15:53:50.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:50.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:50.818297+0000) 2024-03-06T15:53:51.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:51.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:51.818523+0000) 2024-03-06T15:53:52.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:52.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:52.818764+0000) 2024-03-06T15:53:53.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:53.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:53.819001+0000) 2024-03-06T15:53:54.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:54.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:54.819285+0000) 2024-03-06T15:53:55.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:55.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:55.819497+0000) 2024-03-06T15:53:56.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:56.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:56.819725+0000) 2024-03-06T15:53:57.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:57.815 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:53:57.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:57.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:57.819926+0000) 2024-03-06T15:53:58.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:58.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:58.820172+0000) 2024-03-06T15:53:59.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:53:59.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:53:59.820374+0000) 2024-03-06T15:54:00.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:00.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:00.820610+0000) 2024-03-06T15:54:01.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:01.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:01.820839+0000) 2024-03-06T15:54:02.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:02.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:02.821084+0000) 2024-03-06T15:54:03.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:03.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:03.821277+0000) 2024-03-06T15:54:04.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:04.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:04.821492+0000) 2024-03-06T15:54:05.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:05.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:05.821733+0000) 2024-03-06T15:54:06.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:06.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:06.821949+0000) 2024-03-06T15:54:07.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:07.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:07.822170+0000) 2024-03-06T15:54:08.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:08.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:08.822320+0000) 2024-03-06T15:54:09.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:09.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:09.822551+0000) 2024-03-06T15:54:10.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:10.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:10.822788+0000) 2024-03-06T15:54:11.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:11.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:11.823051+0000) 2024-03-06T15:54:12.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:12.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:12.823282+0000) 2024-03-06T15:54:13.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:13.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:13.823638+0000) 2024-03-06T15:54:14.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:14.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:14.823905+0000) 2024-03-06T15:54:15.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:15.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:15.824152+0000) 2024-03-06T15:54:16.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:16.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:16.824386+0000) 2024-03-06T15:54:17.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:17.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:17.824617+0000) 2024-03-06T15:54:18.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:18.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:18.824844+0000) 2024-03-06T15:54:19.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:19.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:19.825136+0000) 2024-03-06T15:54:20.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:20.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:20.825367+0000) 2024-03-06T15:54:21.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:21.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:21.825610+0000) 2024-03-06T15:54:22.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:22.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:22.825874+0000) 2024-03-06T15:54:23.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:23.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:23.826107+0000) 2024-03-06T15:54:24.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:24.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:24.826337+0000) 2024-03-06T15:54:25.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:25.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:25.826570+0000) 2024-03-06T15:54:26.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:26.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:26.826792+0000) 2024-03-06T15:54:27.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:27.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:27.827069+0000) 2024-03-06T15:54:28.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:28.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:28.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:28.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:28.827311+0000) 2024-03-06T15:54:29.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:29.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:29.827531+0000) 2024-03-06T15:54:30.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:30.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:30.827714+0000) 2024-03-06T15:54:31.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:31.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:31.827941+0000) 2024-03-06T15:54:32.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:32.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:32.828151+0000) 2024-03-06T15:54:33.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:33.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:33.828346+0000) 2024-03-06T15:54:34.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:34.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:34.828556+0000) 2024-03-06T15:54:35.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:35.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:35.828803+0000) 2024-03-06T15:54:36.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:36.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:36.829069+0000) 2024-03-06T15:54:37.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:37.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:37.829357+0000) 2024-03-06T15:54:38.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:38.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:38.829572+0000) 2024-03-06T15:54:39.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:39.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:39.829805+0000) 2024-03-06T15:54:40.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:40.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:40.830042+0000) 2024-03-06T15:54:41.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:41.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:41.830268+0000) 2024-03-06T15:54:42.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:42.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:42.830538+0000) 2024-03-06T15:54:43.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:43.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:43.830765+0000) 2024-03-06T15:54:44.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:44.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:44.830976+0000) 2024-03-06T15:54:45.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:45.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:45.831208+0000) 2024-03-06T15:54:46.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:46.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:46.831456+0000) 2024-03-06T15:54:47.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:47.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:47.831701+0000) 2024-03-06T15:54:48.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:48.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:48.831938+0000) 2024-03-06T15:54:49.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:49.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:49.832164+0000) 2024-03-06T15:54:50.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:50.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:50.832402+0000) 2024-03-06T15:54:51.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:51.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:51.832654+0000) 2024-03-06T15:54:52.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:52.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:52.832898+0000) 2024-03-06T15:54:53.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:53.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:53.833144+0000) 2024-03-06T15:54:54.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:54.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:54.833364+0000) 2024-03-06T15:54:55.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:55.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:55.833628+0000) 2024-03-06T15:54:56.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:56.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:56.833852+0000) 2024-03-06T15:54:57.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:57.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:57.834069+0000) 2024-03-06T15:54:58.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:58.260 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:54:58.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:58.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:58.834273+0000) 2024-03-06T15:54:59.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:54:59.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:54:59.834475+0000) 2024-03-06T15:55:00.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:00.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:00.834690+0000) 2024-03-06T15:55:01.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:01.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:01.834914+0000) 2024-03-06T15:55:02.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:02.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:02.835162+0000) 2024-03-06T15:55:03.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:03.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:03.835355+0000) 2024-03-06T15:55:04.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:04.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:04.835569+0000) 2024-03-06T15:55:05.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:05.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:05.835805+0000) 2024-03-06T15:55:06.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:06.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:06.836067+0000) 2024-03-06T15:55:07.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:07.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:07.836291+0000) 2024-03-06T15:55:08.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:08.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:08.836503+0000) 2024-03-06T15:55:09.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:09.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:09.836741+0000) 2024-03-06T15:55:10.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:10.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:10.836928+0000) 2024-03-06T15:55:11.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:11.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:11.837157+0000) 2024-03-06T15:55:12.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:12.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:12.837382+0000) 2024-03-06T15:55:13.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:13.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:13.837606+0000) 2024-03-06T15:55:14.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:14.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:14.837822+0000) 2024-03-06T15:55:15.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:15.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:15.838096+0000) 2024-03-06T15:55:16.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:16.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:16.838326+0000) 2024-03-06T15:55:17.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:17.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:17.838542+0000) 2024-03-06T15:55:18.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:18.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:18.838743+0000) 2024-03-06T15:55:19.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:19.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:19.838977+0000) 2024-03-06T15:55:20.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:20.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:20.839231+0000) 2024-03-06T15:55:21.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:21.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:21.839435+0000) 2024-03-06T15:55:22.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:22.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:22.839668+0000) 2024-03-06T15:55:23.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:23.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:23.839908+0000) 2024-03-06T15:55:24.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:24.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:24.840158+0000) 2024-03-06T15:55:25.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:25.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:25.840443+0000) 2024-03-06T15:55:26.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:26.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:26.840669+0000) 2024-03-06T15:55:27.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:27.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:27.840891+0000) 2024-03-06T15:55:28.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:28.480 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:28.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:28.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:28.841138+0000) 2024-03-06T15:55:29.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:29.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:29.841372+0000) 2024-03-06T15:55:30.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:30.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:30.841594+0000) 2024-03-06T15:55:31.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:31.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:31.841835+0000) 2024-03-06T15:55:32.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:32.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:32.842081+0000) 2024-03-06T15:55:33.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:33.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:33.842287+0000) 2024-03-06T15:55:34.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:34.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:34.842545+0000) 2024-03-06T15:55:35.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:35.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:35.842802+0000) 2024-03-06T15:55:36.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:36.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:36.843067+0000) 2024-03-06T15:55:37.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:37.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:37.843276+0000) 2024-03-06T15:55:38.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:38.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:38.843544+0000) 2024-03-06T15:55:39.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:39.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:39.843795+0000) 2024-03-06T15:55:40.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:40.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:40.844068+0000) 2024-03-06T15:55:41.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:41.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:41.844268+0000) 2024-03-06T15:55:42.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:42.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:42.844482+0000) 2024-03-06T15:55:43.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:43.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:43.844717+0000) 2024-03-06T15:55:44.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:44.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:44.844954+0000) 2024-03-06T15:55:45.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:45.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:45.845192+0000) 2024-03-06T15:55:46.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:46.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:46.845385+0000) 2024-03-06T15:55:47.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:47.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:47.845589+0000) 2024-03-06T15:55:48.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:48.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:48.845813+0000) 2024-03-06T15:55:49.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:49.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:49.846044+0000) 2024-03-06T15:55:50.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:50.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:50.846284+0000) 2024-03-06T15:55:51.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:51.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:51.846520+0000) 2024-03-06T15:55:52.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:52.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:52.846760+0000) 2024-03-06T15:55:53.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:53.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:53.846998+0000) 2024-03-06T15:55:54.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:54.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:54.847240+0000) 2024-03-06T15:55:55.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:55.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:55.847457+0000) 2024-03-06T15:55:56.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:56.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:56.847713+0000) 2024-03-06T15:55:57.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:57.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:57.847992+0000) 2024-03-06T15:55:58.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:58.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:55:58.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:58.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:58.848294+0000) 2024-03-06T15:55:59.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:55:59.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:55:59.848534+0000) 2024-03-06T15:56:00.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:00.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:00.848759+0000) 2024-03-06T15:56:01.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:01.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:01.849042+0000) 2024-03-06T15:56:02.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:02.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:02.849251+0000) 2024-03-06T15:56:03.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:03.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:03.849532+0000) 2024-03-06T15:56:04.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:04.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:04.849798+0000) 2024-03-06T15:56:05.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:05.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:05.850074+0000) 2024-03-06T15:56:06.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:06.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:06.850266+0000) 2024-03-06T15:56:07.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:07.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:07.850526+0000) 2024-03-06T15:56:08.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:08.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:08.850728+0000) 2024-03-06T15:56:09.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:09.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:09.850994+0000) 2024-03-06T15:56:10.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:10.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:10.851289+0000) 2024-03-06T15:56:11.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:11.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:11.851522+0000) 2024-03-06T15:56:12.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:12.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:12.851770+0000) 2024-03-06T15:56:13.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:13.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:13.852086+0000) 2024-03-06T15:56:14.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:14.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:14.852320+0000) 2024-03-06T15:56:15.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:15.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:15.852551+0000) 2024-03-06T15:56:16.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:16.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:16.852730+0000) 2024-03-06T15:56:17.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:17.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:17.852974+0000) 2024-03-06T15:56:18.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:18.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:18.853293+0000) 2024-03-06T15:56:19.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:19.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:19.853531+0000) 2024-03-06T15:56:20.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:20.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:20.853783+0000) 2024-03-06T15:56:21.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:21.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:21.854039+0000) 2024-03-06T15:56:22.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:22.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:22.854277+0000) 2024-03-06T15:56:23.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:23.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:23.854527+0000) 2024-03-06T15:56:24.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:24.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:24.854817+0000) 2024-03-06T15:56:25.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:25.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:25.855115+0000) 2024-03-06T15:56:26.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:26.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:26.855315+0000) 2024-03-06T15:56:27.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:27.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:27.855550+0000) 2024-03-06T15:56:28.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:28.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:28.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:28.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:28.855729+0000) 2024-03-06T15:56:29.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:29.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:29.855998+0000) 2024-03-06T15:56:30.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:30.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:30.856325+0000) 2024-03-06T15:56:31.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:31.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:31.856597+0000) 2024-03-06T15:56:32.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:32.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:32.856860+0000) 2024-03-06T15:56:33.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:33.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:33.857148+0000) 2024-03-06T15:56:34.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:34.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:34.857409+0000) 2024-03-06T15:56:35.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:35.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:35.857715+0000) 2024-03-06T15:56:36.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:36.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:36.857923+0000) 2024-03-06T15:56:37.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:37.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:37.858146+0000) 2024-03-06T15:56:38.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:38.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:38.858411+0000) 2024-03-06T15:56:39.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:39.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:39.858685+0000) 2024-03-06T15:56:40.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:40.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:40.858963+0000) 2024-03-06T15:56:41.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:41.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:41.859243+0000) 2024-03-06T15:56:42.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:42.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:42.859507+0000) 2024-03-06T15:56:43.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:43.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:43.859778+0000) 2024-03-06T15:56:44.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:44.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:44.860116+0000) 2024-03-06T15:56:45.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:45.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:45.860369+0000) 2024-03-06T15:56:46.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:46.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:46.860600+0000) 2024-03-06T15:56:47.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:47.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:47.860822+0000) 2024-03-06T15:56:48.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:48.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:48.861088+0000) 2024-03-06T15:56:49.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:49.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:49.861340+0000) 2024-03-06T15:56:50.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:50.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:50.861586+0000) 2024-03-06T15:56:51.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:51.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:51.861835+0000) 2024-03-06T15:56:52.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:52.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:52.862140+0000) 2024-03-06T15:56:53.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:53.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:53.862383+0000) 2024-03-06T15:56:54.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:54.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:54.862639+0000) 2024-03-06T15:56:55.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:55.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:55.862890+0000) 2024-03-06T15:56:56.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:56.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:56.863140+0000) 2024-03-06T15:56:57.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:57.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:57.863382+0000) 2024-03-06T15:56:58.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:58.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:58.863652+0000) 2024-03-06T15:56:59.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:59.063 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:56:59.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:56:59.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:56:59.863910+0000) 2024-03-06T15:57:00.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:00.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:00.864142+0000) 2024-03-06T15:57:01.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:01.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:01.864352+0000) 2024-03-06T15:57:02.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:02.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:02.864606+0000) 2024-03-06T15:57:03.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:03.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:03.864855+0000) 2024-03-06T15:57:04.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:04.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:04.865119+0000) 2024-03-06T15:57:05.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:05.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:05.865388+0000) 2024-03-06T15:57:06.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:06.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:06.865723+0000) 2024-03-06T15:57:07.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:07.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:07.866032+0000) 2024-03-06T15:57:08.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:08.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:08.866351+0000) 2024-03-06T15:57:09.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:09.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:09.866562+0000) 2024-03-06T15:57:10.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:10.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:10.866789+0000) 2024-03-06T15:57:11.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:11.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:11.867071+0000) 2024-03-06T15:57:12.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:12.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:12.867273+0000) 2024-03-06T15:57:13.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:13.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:13.867492+0000) 2024-03-06T15:57:14.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:14.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:14.867762+0000) 2024-03-06T15:57:15.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:15.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:15.868095+0000) 2024-03-06T15:57:16.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:16.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:16.868386+0000) 2024-03-06T15:57:17.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:17.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:17.868631+0000) 2024-03-06T15:57:18.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:18.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:18.868833+0000) 2024-03-06T15:57:19.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:19.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:19.869116+0000) 2024-03-06T15:57:20.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:20.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:20.869345+0000) 2024-03-06T15:57:21.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:21.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:21.869572+0000) 2024-03-06T15:57:22.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:22.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:22.869822+0000) 2024-03-06T15:57:23.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:23.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:23.870114+0000) 2024-03-06T15:57:24.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:24.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:24.870349+0000) 2024-03-06T15:57:25.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:25.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:25.870513+0000) 2024-03-06T15:57:26.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:26.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:26.870742+0000) 2024-03-06T15:57:27.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:27.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:27.871070+0000) 2024-03-06T15:57:28.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:28.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:28.871281+0000) 2024-03-06T15:57:29.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:29.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:29.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:29.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:29.871550+0000) 2024-03-06T15:57:30.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:30.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:30.871834+0000) 2024-03-06T15:57:31.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:31.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:31.872120+0000) 2024-03-06T15:57:32.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:32.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:32.872369+0000) 2024-03-06T15:57:33.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:33.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:33.872567+0000) 2024-03-06T15:57:34.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:34.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:34.872843+0000) 2024-03-06T15:57:35.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:35.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:35.873137+0000) 2024-03-06T15:57:36.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:36.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:36.873357+0000) 2024-03-06T15:57:37.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:37.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:37.873613+0000) 2024-03-06T15:57:38.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:38.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:38.873835+0000) 2024-03-06T15:57:39.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:39.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:39.874141+0000) 2024-03-06T15:57:40.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:40.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:40.874326+0000) 2024-03-06T15:57:41.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:41.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:41.874580+0000) 2024-03-06T15:57:42.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:42.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:42.874825+0000) 2024-03-06T15:57:43.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:43.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:43.875150+0000) 2024-03-06T15:57:44.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:44.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:44.875383+0000) 2024-03-06T15:57:45.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:45.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:45.875655+0000) 2024-03-06T15:57:46.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:46.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:46.875891+0000) 2024-03-06T15:57:47.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:47.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:47.876131+0000) 2024-03-06T15:57:48.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:48.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:48.876359+0000) 2024-03-06T15:57:49.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:49.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:49.876586+0000) 2024-03-06T15:57:50.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:50.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:50.876828+0000) 2024-03-06T15:57:51.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:51.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:51.877109+0000) 2024-03-06T15:57:52.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:52.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:52.877361+0000) 2024-03-06T15:57:53.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:53.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:53.877603+0000) 2024-03-06T15:57:54.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:54.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:54.877914+0000) 2024-03-06T15:57:55.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:55.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:55.878186+0000) 2024-03-06T15:57:56.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:56.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:56.878438+0000) 2024-03-06T15:57:57.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:57.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:57.878665+0000) 2024-03-06T15:57:58.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:58.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:58.878854+0000) 2024-03-06T15:57:59.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:59.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:57:59.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:57:59.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:57:59.879131+0000) 2024-03-06T15:58:00.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:00.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:00.879315+0000) 2024-03-06T15:58:01.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:01.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:01.879542+0000) 2024-03-06T15:58:02.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:02.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:02.879806+0000) 2024-03-06T15:58:03.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:03.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:03.879969+0000) 2024-03-06T15:58:04.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:04.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:04.880193+0000) 2024-03-06T15:58:05.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:05.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:05.880409+0000) 2024-03-06T15:58:06.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:06.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:06.880613+0000) 2024-03-06T15:58:07.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:07.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:07.880903+0000) 2024-03-06T15:58:08.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:08.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:08.881183+0000) 2024-03-06T15:58:09.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:09.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:09.881448+0000) 2024-03-06T15:58:10.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:10.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:10.881684+0000) 2024-03-06T15:58:11.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:11.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:11.881926+0000) 2024-03-06T15:58:12.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:12.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:12.882191+0000) 2024-03-06T15:58:13.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:13.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:13.882478+0000) 2024-03-06T15:58:14.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:14.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:14.882712+0000) 2024-03-06T15:58:15.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:15.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:15.882954+0000) 2024-03-06T15:58:16.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:16.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:16.883183+0000) 2024-03-06T15:58:17.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:17.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:17.883449+0000) 2024-03-06T15:58:18.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:18.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:18.883645+0000) 2024-03-06T15:58:19.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:19.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:19.883889+0000) 2024-03-06T15:58:20.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:20.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:20.884122+0000) 2024-03-06T15:58:21.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:21.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:21.884352+0000) 2024-03-06T15:58:22.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:22.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:22.884581+0000) 2024-03-06T15:58:23.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:23.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:23.884805+0000) 2024-03-06T15:58:24.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:24.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:24.885091+0000) 2024-03-06T15:58:25.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:25.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:25.885308+0000) 2024-03-06T15:58:26.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:26.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:26.885527+0000) 2024-03-06T15:58:27.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:27.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:27.885751+0000) 2024-03-06T15:58:28.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:28.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:28.885975+0000) 2024-03-06T15:58:29.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:29.741 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:29.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:29.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:29.886200+0000) 2024-03-06T15:58:30.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:30.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:30.886417+0000) 2024-03-06T15:58:31.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:31.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:31.886631+0000) 2024-03-06T15:58:32.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:32.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:32.886860+0000) 2024-03-06T15:58:33.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:33.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:33.887200+0000) 2024-03-06T15:58:34.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:34.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:34.887474+0000) 2024-03-06T15:58:35.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:35.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:35.887833+0000) 2024-03-06T15:58:36.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:36.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:36.888077+0000) 2024-03-06T15:58:37.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:37.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:37.888252+0000) 2024-03-06T15:58:38.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:38.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:38.888430+0000) 2024-03-06T15:58:39.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:39.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:39.888626+0000) 2024-03-06T15:58:40.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:40.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:40.888852+0000) 2024-03-06T15:58:41.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:41.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:41.889130+0000) 2024-03-06T15:58:42.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:42.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:42.889364+0000) 2024-03-06T15:58:43.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:43.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:43.889621+0000) 2024-03-06T15:58:44.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:44.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:44.889870+0000) 2024-03-06T15:58:45.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:45.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:45.890149+0000) 2024-03-06T15:58:46.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:46.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:46.890372+0000) 2024-03-06T15:58:47.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:47.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:47.890661+0000) 2024-03-06T15:58:48.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:48.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:48.890883+0000) 2024-03-06T15:58:49.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:49.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:49.891118+0000) 2024-03-06T15:58:50.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:50.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:50.891342+0000) 2024-03-06T15:58:51.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:51.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:51.891571+0000) 2024-03-06T15:58:52.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:52.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:52.891795+0000) 2024-03-06T15:58:53.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:53.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:53.892045+0000) 2024-03-06T15:58:54.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:54.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:54.892275+0000) 2024-03-06T15:58:55.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:55.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:55.892491+0000) 2024-03-06T15:58:56.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:56.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:56.892715+0000) 2024-03-06T15:58:57.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:57.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:57.892941+0000) 2024-03-06T15:58:58.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:58.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:58.893176+0000) 2024-03-06T15:58:59.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:58:59.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:58:59.893373+0000) 2024-03-06T15:58:59.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:58:59.954 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:00.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:00.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:00.893594+0000) 2024-03-06T15:59:01.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:01.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:01.893815+0000) 2024-03-06T15:59:02.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:02.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:02.894077+0000) 2024-03-06T15:59:03.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:03.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:03.894264+0000) 2024-03-06T15:59:04.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:04.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:04.894475+0000) 2024-03-06T15:59:05.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:05.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:05.894722+0000) 2024-03-06T15:59:06.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:06.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:06.894928+0000) 2024-03-06T15:59:07.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:07.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:07.895169+0000) 2024-03-06T15:59:08.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:08.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:08.895445+0000) 2024-03-06T15:59:09.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:09.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:09.895726+0000) 2024-03-06T15:59:10.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:10.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:10.895979+0000) 2024-03-06T15:59:11.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:11.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:11.896284+0000) 2024-03-06T15:59:12.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:12.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:12.896568+0000) 2024-03-06T15:59:13.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:13.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:13.896770+0000) 2024-03-06T15:59:14.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:14.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:14.896995+0000) 2024-03-06T15:59:15.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:15.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:15.897268+0000) 2024-03-06T15:59:16.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:16.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:16.897556+0000) 2024-03-06T15:59:17.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:17.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:17.897780+0000) 2024-03-06T15:59:18.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:18.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:18.897998+0000) 2024-03-06T15:59:19.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:19.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:19.898218+0000) 2024-03-06T15:59:20.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:20.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:20.898431+0000) 2024-03-06T15:59:21.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:21.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:21.898678+0000) 2024-03-06T15:59:22.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:22.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:22.898861+0000) 2024-03-06T15:59:23.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:23.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:23.899118+0000) 2024-03-06T15:59:24.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:24.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:24.899347+0000) 2024-03-06T15:59:25.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:25.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:25.899585+0000) 2024-03-06T15:59:26.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:26.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:26.899810+0000) 2024-03-06T15:59:27.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:27.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:27.900102+0000) 2024-03-06T15:59:28.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:28.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:28.900325+0000) 2024-03-06T15:59:29.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:29.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:29.900544+0000) 2024-03-06T15:59:30.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:30.090 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T15:59:30.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:30.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:30.900793+0000) 2024-03-06T15:59:31.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:31.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:31.901059+0000) 2024-03-06T15:59:32.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:32.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:32.901266+0000) 2024-03-06T15:59:33.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:33.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:33.901440+0000) 2024-03-06T15:59:34.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:34.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:34.901641+0000) 2024-03-06T15:59:35.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:35.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:35.901922+0000) 2024-03-06T15:59:36.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:36.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:36.902207+0000) 2024-03-06T15:59:37.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:37.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:37.902489+0000) 2024-03-06T15:59:38.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:38.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:38.902774+0000) 2024-03-06T15:59:39.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:39.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:39.903106+0000) 2024-03-06T15:59:40.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:40.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:40.903384+0000) 2024-03-06T15:59:41.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:41.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:41.903714+0000) 2024-03-06T15:59:42.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:42.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:42.903982+0000) 2024-03-06T15:59:43.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:43.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:43.904264+0000) 2024-03-06T15:59:44.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:44.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:44.904547+0000) 2024-03-06T15:59:45.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:45.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:45.904766+0000) 2024-03-06T15:59:46.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:46.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:46.904995+0000) 2024-03-06T15:59:47.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:47.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:47.905270+0000) 2024-03-06T15:59:48.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:48.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:48.905494+0000) 2024-03-06T15:59:49.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:49.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:49.905714+0000) 2024-03-06T15:59:50.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:50.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:50.905925+0000) 2024-03-06T15:59:51.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:51.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:51.906163+0000) 2024-03-06T15:59:52.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:52.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:52.906390+0000) 2024-03-06T15:59:53.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:53.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:53.906610+0000) 2024-03-06T15:59:54.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:54.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:54.906846+0000) 2024-03-06T15:59:55.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:55.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:55.907127+0000) 2024-03-06T15:59:56.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:56.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:56.907353+0000) 2024-03-06T15:59:57.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:57.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:57.907583+0000) 2024-03-06T15:59:58.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:58.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:58.907804+0000) 2024-03-06T15:59:59.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T15:59:59.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T14:59:59.907993+0000) 2024-03-06T16:00:00.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:00.255 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:00.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:00.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:00.908295+0000) 2024-03-06T16:00:01.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:01.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:01.908535+0000) 2024-03-06T16:00:02.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:02.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:02.908770+0000) 2024-03-06T16:00:03.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:03.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:03.909058+0000) 2024-03-06T16:00:04.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:04.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:04.909298+0000) 2024-03-06T16:00:05.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:05.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:05.909521+0000) 2024-03-06T16:00:06.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:06.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:06.909704+0000) 2024-03-06T16:00:07.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:07.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:07.909930+0000) 2024-03-06T16:00:08.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:08.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:08.910149+0000) 2024-03-06T16:00:09.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:09.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:09.910375+0000) 2024-03-06T16:00:10.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:10.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:10.910585+0000) 2024-03-06T16:00:11.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:11.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:11.910816+0000) 2024-03-06T16:00:12.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:12.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:12.911079+0000) 2024-03-06T16:00:13.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:13.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:13.911272+0000) 2024-03-06T16:00:14.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:14.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:14.911487+0000) 2024-03-06T16:00:15.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:15.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:15.911714+0000) 2024-03-06T16:00:16.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:16.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:16.911944+0000) 2024-03-06T16:00:17.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:17.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:17.912166+0000) 2024-03-06T16:00:18.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:18.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:18.912405+0000) 2024-03-06T16:00:19.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:19.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:19.912658+0000) 2024-03-06T16:00:20.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:20.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:20.912851+0000) 2024-03-06T16:00:21.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:21.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:21.913133+0000) 2024-03-06T16:00:22.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:22.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:22.913374+0000) 2024-03-06T16:00:23.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:23.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:23.913653+0000) 2024-03-06T16:00:24.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:24.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:24.913872+0000) 2024-03-06T16:00:25.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:25.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:25.914165+0000) 2024-03-06T16:00:26.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:26.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:26.914416+0000) 2024-03-06T16:00:27.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:27.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:27.914674+0000) 2024-03-06T16:00:28.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:28.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:28.914906+0000) 2024-03-06T16:00:29.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:29.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:29.915132+0000) 2024-03-06T16:00:30.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:30.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:00:30.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:30.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:30.915361+0000) 2024-03-06T16:00:31.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:31.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:31.915597+0000) 2024-03-06T16:00:32.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:32.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:32.915815+0000) 2024-03-06T16:00:33.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:33.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:33.916075+0000) 2024-03-06T16:00:34.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:34.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:34.916304+0000) 2024-03-06T16:00:35.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:35.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:35.916477+0000) 2024-03-06T16:00:36.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:36.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:36.916667+0000) 2024-03-06T16:00:37.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:37.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:37.916851+0000) 2024-03-06T16:00:38.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:38.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:38.917118+0000) 2024-03-06T16:00:39.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:39.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:39.917366+0000) 2024-03-06T16:00:40.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:40.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:40.917613+0000) 2024-03-06T16:00:41.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:41.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:41.917842+0000) 2024-03-06T16:00:42.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:42.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:42.918077+0000) 2024-03-06T16:00:43.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:43.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:43.918300+0000) 2024-03-06T16:00:44.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:44.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:44.918532+0000) 2024-03-06T16:00:45.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:45.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:45.918779+0000) 2024-03-06T16:00:46.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:46.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:46.919032+0000) 2024-03-06T16:00:47.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:47.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:47.919261+0000) 2024-03-06T16:00:48.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:48.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:48.919474+0000) 2024-03-06T16:00:49.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:49.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:49.919648+0000) 2024-03-06T16:00:50.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:50.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:50.919878+0000) 2024-03-06T16:00:51.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:51.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:51.920136+0000) 2024-03-06T16:00:52.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:52.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:52.920403+0000) 2024-03-06T16:00:53.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:53.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:53.920682+0000) 2024-03-06T16:00:54.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:54.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:54.920963+0000) 2024-03-06T16:00:55.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:55.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:55.921296+0000) 2024-03-06T16:00:56.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:56.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:56.921511+0000) 2024-03-06T16:00:57.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:57.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:57.921729+0000) 2024-03-06T16:00:58.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:58.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:58.921920+0000) 2024-03-06T16:00:59.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:00:59.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:00:59.922168+0000) 2024-03-06T16:01:00.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:00.548 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:00.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:00.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:00.922440+0000) 2024-03-06T16:01:01.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:01.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:01.922636+0000) 2024-03-06T16:01:02.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:02.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:02.922848+0000) 2024-03-06T16:01:03.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:03.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:03.923109+0000) 2024-03-06T16:01:04.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:04.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:04.923332+0000) 2024-03-06T16:01:05.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:05.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:05.923597+0000) 2024-03-06T16:01:06.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:06.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:06.923830+0000) 2024-03-06T16:01:07.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:07.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:07.924059+0000) 2024-03-06T16:01:08.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:08.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:08.924328+0000) 2024-03-06T16:01:09.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:09.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:09.924612+0000) 2024-03-06T16:01:10.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:10.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:10.924889+0000) 2024-03-06T16:01:11.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:11.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:11.925166+0000) 2024-03-06T16:01:12.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:12.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:12.925375+0000) 2024-03-06T16:01:13.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:13.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:13.925599+0000) 2024-03-06T16:01:14.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:14.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:14.925825+0000) 2024-03-06T16:01:15.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:15.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:15.926118+0000) 2024-03-06T16:01:16.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:16.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:16.926410+0000) 2024-03-06T16:01:17.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:17.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:17.926730+0000) 2024-03-06T16:01:18.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:18.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:18.927001+0000) 2024-03-06T16:01:19.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:19.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:19.927324+0000) 2024-03-06T16:01:20.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:20.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:20.927554+0000) 2024-03-06T16:01:21.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:21.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:21.927755+0000) 2024-03-06T16:01:22.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:22.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:22.927980+0000) 2024-03-06T16:01:23.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:23.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:23.928253+0000) 2024-03-06T16:01:24.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:24.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:24.928484+0000) 2024-03-06T16:01:25.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:25.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:25.928729+0000) 2024-03-06T16:01:26.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:26.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:26.928955+0000) 2024-03-06T16:01:27.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:27.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:27.929188+0000) 2024-03-06T16:01:28.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:28.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:28.929415+0000) 2024-03-06T16:01:29.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:29.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:29.929634+0000) 2024-03-06T16:01:30.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:30.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:01:30.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:30.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:30.929844+0000) 2024-03-06T16:01:31.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:31.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:31.930111+0000) 2024-03-06T16:01:32.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:32.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:32.930338+0000) 2024-03-06T16:01:33.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:33.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:33.930524+0000) 2024-03-06T16:01:34.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:34.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:34.930762+0000) 2024-03-06T16:01:35.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:35.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:35.931121+0000) 2024-03-06T16:01:36.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:36.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:36.931381+0000) 2024-03-06T16:01:37.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:37.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:37.931602+0000) 2024-03-06T16:01:38.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:38.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:38.931842+0000) 2024-03-06T16:01:39.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:39.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:39.932124+0000) 2024-03-06T16:01:40.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:40.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:40.932326+0000) 2024-03-06T16:01:41.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:41.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:41.932563+0000) 2024-03-06T16:01:42.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:42.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:42.932808+0000) 2024-03-06T16:01:43.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:43.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:43.933099+0000) 2024-03-06T16:01:44.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:44.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:44.933326+0000) 2024-03-06T16:01:45.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:45.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:45.933581+0000) 2024-03-06T16:01:46.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:46.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:46.933804+0000) 2024-03-06T16:01:47.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:47.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:47.934066+0000) 2024-03-06T16:01:48.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:48.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:48.934302+0000) 2024-03-06T16:01:49.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:49.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:49.934508+0000) 2024-03-06T16:01:50.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:50.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:50.934736+0000) 2024-03-06T16:01:51.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:51.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:51.934976+0000) 2024-03-06T16:01:52.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:52.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:52.935269+0000) 2024-03-06T16:01:53.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:53.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:53.935514+0000) 2024-03-06T16:01:54.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:54.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:54.935733+0000) 2024-03-06T16:01:55.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:55.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:55.935991+0000) 2024-03-06T16:01:56.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:56.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:56.936261+0000) 2024-03-06T16:01:57.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:57.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:57.936492+0000) 2024-03-06T16:01:58.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:58.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:58.936730+0000) 2024-03-06T16:01:59.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:01:59.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:01:59.936957+0000) 2024-03-06T16:02:00.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:00.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:00.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:00.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:00.937117+0000) 2024-03-06T16:02:01.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:01.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:01.937371+0000) 2024-03-06T16:02:02.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:02.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:02.937618+0000) 2024-03-06T16:02:03.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:03.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:03.937902+0000) 2024-03-06T16:02:04.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:04.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:04.938197+0000) 2024-03-06T16:02:05.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:05.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:05.938552+0000) 2024-03-06T16:02:06.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:06.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:06.938747+0000) 2024-03-06T16:02:07.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:07.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:07.938982+0000) 2024-03-06T16:02:08.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:08.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:08.939304+0000) 2024-03-06T16:02:09.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:09.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:09.939545+0000) 2024-03-06T16:02:10.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:10.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:10.939778+0000) 2024-03-06T16:02:11.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:11.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:11.939974+0000) 2024-03-06T16:02:12.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:12.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:12.940142+0000) 2024-03-06T16:02:13.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:13.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:13.940371+0000) 2024-03-06T16:02:14.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:14.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:14.940595+0000) 2024-03-06T16:02:15.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:15.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:15.940836+0000) 2024-03-06T16:02:16.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:16.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:16.941106+0000) 2024-03-06T16:02:17.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:17.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:17.941301+0000) 2024-03-06T16:02:18.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:18.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:18.941524+0000) 2024-03-06T16:02:19.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:19.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:19.941754+0000) 2024-03-06T16:02:20.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:20.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:20.941989+0000) 2024-03-06T16:02:21.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:21.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:21.942286+0000) 2024-03-06T16:02:22.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:22.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:22.942527+0000) 2024-03-06T16:02:23.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:23.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:23.942808+0000) 2024-03-06T16:02:24.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:24.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:24.943103+0000) 2024-03-06T16:02:25.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:25.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:25.943400+0000) 2024-03-06T16:02:26.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:26.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:26.943592+0000) 2024-03-06T16:02:27.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:27.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:27.943813+0000) 2024-03-06T16:02:28.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:28.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:28.944100+0000) 2024-03-06T16:02:29.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:29.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:29.944300+0000) 2024-03-06T16:02:30.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:30.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:30.944548+0000) 2024-03-06T16:02:31.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:31.014 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:02:31.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:31.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:31.944779+0000) 2024-03-06T16:02:32.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:32.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:32.945058+0000) 2024-03-06T16:02:33.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:33.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:33.945288+0000) 2024-03-06T16:02:34.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:34.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:34.945513+0000) 2024-03-06T16:02:35.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:35.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:35.945760+0000) 2024-03-06T16:02:36.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:36.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:36.945981+0000) 2024-03-06T16:02:37.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:37.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:37.946244+0000) 2024-03-06T16:02:38.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:38.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:38.946466+0000) 2024-03-06T16:02:39.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:39.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:39.946641+0000) 2024-03-06T16:02:40.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:40.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:40.946924+0000) 2024-03-06T16:02:41.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:41.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:41.947157+0000) 2024-03-06T16:02:42.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:42.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:42.947379+0000) 2024-03-06T16:02:43.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:43.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:43.947607+0000) 2024-03-06T16:02:44.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:44.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:44.947869+0000) 2024-03-06T16:02:45.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:45.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:45.948152+0000) 2024-03-06T16:02:46.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:46.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:46.948378+0000) 2024-03-06T16:02:47.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:47.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:47.948610+0000) 2024-03-06T16:02:48.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:48.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:48.948834+0000) 2024-03-06T16:02:49.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:49.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:49.949054+0000) 2024-03-06T16:02:50.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:50.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:50.949266+0000) 2024-03-06T16:02:51.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:51.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:51.949499+0000) 2024-03-06T16:02:52.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:52.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:52.949726+0000) 2024-03-06T16:02:53.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:53.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:53.949957+0000) 2024-03-06T16:02:54.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:54.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:54.950192+0000) 2024-03-06T16:02:55.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:55.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:55.950466+0000) 2024-03-06T16:02:56.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:56.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:56.950651+0000) 2024-03-06T16:02:57.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:57.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:57.950927+0000) 2024-03-06T16:02:58.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:58.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:58.951188+0000) 2024-03-06T16:02:59.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:02:59.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:02:59.951460+0000) 2024-03-06T16:03:00.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:00.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:00.951743+0000) 2024-03-06T16:03:01.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:01.161 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:01.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:01.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:01.952033+0000) 2024-03-06T16:03:02.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:02.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:02.952259+0000) 2024-03-06T16:03:03.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:03.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:03.952531+0000) 2024-03-06T16:03:04.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:04.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:04.952814+0000) 2024-03-06T16:03:05.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:05.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:05.953131+0000) 2024-03-06T16:03:06.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:06.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:06.953371+0000) 2024-03-06T16:03:07.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:07.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:07.953613+0000) 2024-03-06T16:03:08.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:08.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:08.953843+0000) 2024-03-06T16:03:09.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:09.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:09.954105+0000) 2024-03-06T16:03:10.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:10.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:10.954340+0000) 2024-03-06T16:03:11.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:11.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:11.954535+0000) 2024-03-06T16:03:12.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:12.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:12.954794+0000) 2024-03-06T16:03:13.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:13.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:13.955086+0000) 2024-03-06T16:03:14.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:14.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:14.955311+0000) 2024-03-06T16:03:15.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:15.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:15.955558+0000) 2024-03-06T16:03:16.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:16.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:16.955809+0000) 2024-03-06T16:03:17.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:17.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:17.956099+0000) 2024-03-06T16:03:18.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:18.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:18.956307+0000) 2024-03-06T16:03:19.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:19.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:19.956567+0000) 2024-03-06T16:03:20.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:20.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:20.956800+0000) 2024-03-06T16:03:21.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:21.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:21.957086+0000) 2024-03-06T16:03:22.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:22.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:22.957291+0000) 2024-03-06T16:03:23.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:23.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:23.957493+0000) 2024-03-06T16:03:24.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:24.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:24.957728+0000) 2024-03-06T16:03:25.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:25.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:25.957988+0000) 2024-03-06T16:03:26.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:26.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:26.958257+0000) 2024-03-06T16:03:27.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:27.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:27.958466+0000) 2024-03-06T16:03:28.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:28.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:28.958736+0000) 2024-03-06T16:03:29.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:29.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:29.958976+0000) 2024-03-06T16:03:30.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:30.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:30.959285+0000) 2024-03-06T16:03:31.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:31.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:03:31.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:31.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:31.959505+0000) 2024-03-06T16:03:32.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:32.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:32.959723+0000) 2024-03-06T16:03:33.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:33.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:33.959973+0000) 2024-03-06T16:03:34.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:34.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:34.960210+0000) 2024-03-06T16:03:35.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:35.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:35.960440+0000) 2024-03-06T16:03:36.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:36.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:36.960716+0000) 2024-03-06T16:03:37.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:37.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:37.960954+0000) 2024-03-06T16:03:38.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:38.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:38.961148+0000) 2024-03-06T16:03:39.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:39.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:39.961395+0000) 2024-03-06T16:03:40.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:40.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:40.961612+0000) 2024-03-06T16:03:41.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:41.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:41.961839+0000) 2024-03-06T16:03:42.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:42.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:42.962129+0000) 2024-03-06T16:03:43.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:43.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:43.962353+0000) 2024-03-06T16:03:44.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:44.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:44.962576+0000) 2024-03-06T16:03:45.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:45.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:45.962827+0000) 2024-03-06T16:03:46.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:46.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:46.963131+0000) 2024-03-06T16:03:47.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:47.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:47.963351+0000) 2024-03-06T16:03:48.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:48.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:48.963574+0000) 2024-03-06T16:03:49.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:49.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:49.963791+0000) 2024-03-06T16:03:50.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:50.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:50.964074+0000) 2024-03-06T16:03:51.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:51.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:51.964356+0000) 2024-03-06T16:03:52.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:52.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:52.964569+0000) 2024-03-06T16:03:53.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:53.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:53.964817+0000) 2024-03-06T16:03:54.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:54.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:54.965111+0000) 2024-03-06T16:03:55.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:55.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:55.965318+0000) 2024-03-06T16:03:56.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:56.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:56.965539+0000) 2024-03-06T16:03:57.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:57.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:57.965762+0000) 2024-03-06T16:03:58.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:58.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:58.965990+0000) 2024-03-06T16:03:59.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:03:59.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:03:59.966232+0000) 2024-03-06T16:04:00.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:00.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:00.966430+0000) 2024-03-06T16:04:01.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:01.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:01.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:01.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:01.966621+0000) 2024-03-06T16:04:02.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:02.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:02.966859+0000) 2024-03-06T16:04:03.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:03.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:03.967145+0000) 2024-03-06T16:04:04.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:04.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:04.967366+0000) 2024-03-06T16:04:05.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:05.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:05.967629+0000) 2024-03-06T16:04:06.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:06.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:06.967874+0000) 2024-03-06T16:04:07.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:07.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:07.968128+0000) 2024-03-06T16:04:08.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:08.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:08.968344+0000) 2024-03-06T16:04:09.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:09.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:09.968591+0000) 2024-03-06T16:04:10.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:10.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:10.968790+0000) 2024-03-06T16:04:11.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:11.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:11.969064+0000) 2024-03-06T16:04:12.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:12.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:12.969277+0000) 2024-03-06T16:04:13.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:13.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:13.969509+0000) 2024-03-06T16:04:14.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:14.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:14.969727+0000) 2024-03-06T16:04:15.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:15.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:15.969977+0000) 2024-03-06T16:04:16.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:16.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:16.970245+0000) 2024-03-06T16:04:17.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:17.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:17.970509+0000) 2024-03-06T16:04:18.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:18.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:18.970784+0000) 2024-03-06T16:04:19.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:19.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:19.971000+0000) 2024-03-06T16:04:20.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:20.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:20.971260+0000) 2024-03-06T16:04:21.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:21.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:21.971475+0000) 2024-03-06T16:04:22.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:22.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:22.971717+0000) 2024-03-06T16:04:23.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:23.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:23.971989+0000) 2024-03-06T16:04:24.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:24.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:24.972253+0000) 2024-03-06T16:04:25.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:25.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:25.972495+0000) 2024-03-06T16:04:26.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:26.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:26.972722+0000) 2024-03-06T16:04:27.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:27.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:27.972945+0000) 2024-03-06T16:04:28.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:28.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:28.973210+0000) 2024-03-06T16:04:29.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:29.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:29.973447+0000) 2024-03-06T16:04:30.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:30.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:30.973693+0000) 2024-03-06T16:04:31.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:31.649 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:04:31.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:31.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:31.973979+0000) 2024-03-06T16:04:32.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:32.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:32.974305+0000) 2024-03-06T16:04:33.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:33.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:33.974521+0000) 2024-03-06T16:04:34.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:34.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:34.974799+0000) 2024-03-06T16:04:35.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:35.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:35.975061+0000) 2024-03-06T16:04:36.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:36.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:36.975295+0000) 2024-03-06T16:04:37.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:37.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:37.975513+0000) 2024-03-06T16:04:38.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:38.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:38.975755+0000) 2024-03-06T16:04:39.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:39.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:39.976030+0000) 2024-03-06T16:04:40.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:40.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:40.976253+0000) 2024-03-06T16:04:41.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:41.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:41.976486+0000) 2024-03-06T16:04:42.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:42.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:42.976706+0000) 2024-03-06T16:04:43.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:43.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:43.976932+0000) 2024-03-06T16:04:44.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:44.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:44.977158+0000) 2024-03-06T16:04:45.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:45.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:45.977391+0000) 2024-03-06T16:04:46.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:46.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:46.977609+0000) 2024-03-06T16:04:47.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:47.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:47.977828+0000) 2024-03-06T16:04:48.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:48.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:48.978088+0000) 2024-03-06T16:04:49.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:49.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:49.978312+0000) 2024-03-06T16:04:50.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:50.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:50.978530+0000) 2024-03-06T16:04:51.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:51.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:51.978782+0000) 2024-03-06T16:04:52.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:52.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:52.979055+0000) 2024-03-06T16:04:53.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:53.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:53.979311+0000) 2024-03-06T16:04:54.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:54.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:54.979536+0000) 2024-03-06T16:04:55.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:55.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:55.979793+0000) 2024-03-06T16:04:56.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:56.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:56.979995+0000) 2024-03-06T16:04:57.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:57.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:57.980263+0000) 2024-03-06T16:04:58.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:58.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:58.980452+0000) 2024-03-06T16:04:59.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:04:59.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:04:59.980677+0000) 2024-03-06T16:05:00.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:00.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:00.980902+0000) 2024-03-06T16:05:01.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:01.777 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:01.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:01.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:01.981150+0000) 2024-03-06T16:05:02.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:02.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:02.981368+0000) 2024-03-06T16:05:03.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:03.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:03.981598+0000) 2024-03-06T16:05:04.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:04.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:04.981818+0000) 2024-03-06T16:05:05.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:05.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:05.982094+0000) 2024-03-06T16:05:06.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:06.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:06.982293+0000) 2024-03-06T16:05:07.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:07.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:07.982481+0000) 2024-03-06T16:05:08.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:08.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:08.982674+0000) 2024-03-06T16:05:09.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:09.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:09.982904+0000) 2024-03-06T16:05:10.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:10.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:10.983137+0000) 2024-03-06T16:05:11.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:11.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:11.983368+0000) 2024-03-06T16:05:12.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:12.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:12.983577+0000) 2024-03-06T16:05:13.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:13.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:13.983771+0000) 2024-03-06T16:05:14.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:14.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:14.983999+0000) 2024-03-06T16:05:15.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:15.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:15.984299+0000) 2024-03-06T16:05:16.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:16.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:16.984528+0000) 2024-03-06T16:05:17.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:17.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:17.984773+0000) 2024-03-06T16:05:18.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:18.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:18.985041+0000) 2024-03-06T16:05:19.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:19.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:19.985274+0000) 2024-03-06T16:05:20.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:20.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:20.985460+0000) 2024-03-06T16:05:21.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:21.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:21.985675+0000) 2024-03-06T16:05:22.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:22.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:22.985955+0000) 2024-03-06T16:05:23.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:23.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:23.986178+0000) 2024-03-06T16:05:24.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:24.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:24.986403+0000) 2024-03-06T16:05:25.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:25.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:25.986648+0000) 2024-03-06T16:05:26.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:26.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:26.986888+0000) 2024-03-06T16:05:27.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:27.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:27.987119+0000) 2024-03-06T16:05:28.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:28.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:28.987281+0000) 2024-03-06T16:05:29.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:29.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:29.987489+0000) 2024-03-06T16:05:30.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:30.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:30.987675+0000) 2024-03-06T16:05:31.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:31.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:05:31.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:31.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:31.987929+0000) 2024-03-06T16:05:32.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:32.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:32.988189+0000) 2024-03-06T16:05:33.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:33.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:33.988415+0000) 2024-03-06T16:05:34.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:34.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:34.988619+0000) 2024-03-06T16:05:35.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:35.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:35.988829+0000) 2024-03-06T16:05:36.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:36.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:36.989115+0000) 2024-03-06T16:05:37.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:37.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:37.989353+0000) 2024-03-06T16:05:38.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:38.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:38.989579+0000) 2024-03-06T16:05:39.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:39.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:39.989805+0000) 2024-03-06T16:05:40.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:40.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:40.990057+0000) 2024-03-06T16:05:41.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:41.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:41.990306+0000) 2024-03-06T16:05:42.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:42.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:42.990524+0000) 2024-03-06T16:05:43.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:43.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:43.990777+0000) 2024-03-06T16:05:44.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:44.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:44.990994+0000) 2024-03-06T16:05:45.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:45.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:45.991280+0000) 2024-03-06T16:05:46.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:46.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:46.991530+0000) 2024-03-06T16:05:47.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:47.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:47.991718+0000) 2024-03-06T16:05:48.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:48.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:48.991910+0000) 2024-03-06T16:05:49.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:49.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:49.992167+0000) 2024-03-06T16:05:50.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:50.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:50.992388+0000) 2024-03-06T16:05:51.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:51.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:51.992650+0000) 2024-03-06T16:05:52.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:52.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:52.992892+0000) 2024-03-06T16:05:53.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:53.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:53.993121+0000) 2024-03-06T16:05:54.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:54.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:54.993350+0000) 2024-03-06T16:05:55.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:55.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:55.993574+0000) 2024-03-06T16:05:56.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:56.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:56.993765+0000) 2024-03-06T16:05:57.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:57.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:57.993989+0000) 2024-03-06T16:05:58.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:58.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:58.994256+0000) 2024-03-06T16:05:59.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:05:59.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:05:59.994510+0000) 2024-03-06T16:06:00.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:00.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:00.994699+0000) 2024-03-06T16:06:01.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:01.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:01.994929+0000) 2024-03-06T16:06:02.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:02.073 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:02.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:02.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:02.995165+0000) 2024-03-06T16:06:03.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:03.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:03.995372+0000) 2024-03-06T16:06:04.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:04.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:04.995639+0000) 2024-03-06T16:06:05.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:05.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:05.995875+0000) 2024-03-06T16:06:06.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:06.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:06.996107+0000) 2024-03-06T16:06:07.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:07.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:07.996325+0000) 2024-03-06T16:06:08.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:08.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:08.996572+0000) 2024-03-06T16:06:09.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:09.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:09.996789+0000) 2024-03-06T16:06:10.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:10.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:10.996999+0000) 2024-03-06T16:06:11.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:11.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:11.997316+0000) 2024-03-06T16:06:12.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:12.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:12.997525+0000) 2024-03-06T16:06:13.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:13.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:13.997759+0000) 2024-03-06T16:06:14.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:14.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:14.997993+0000) 2024-03-06T16:06:15.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:15.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:15.998354+0000) 2024-03-06T16:06:16.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:16.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:16.998576+0000) 2024-03-06T16:06:17.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:17.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:17.998757+0000) 2024-03-06T16:06:18.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:18.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:18.998992+0000) 2024-03-06T16:06:19.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:19.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:19.999273+0000) 2024-03-06T16:06:20.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:20.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:20.999498+0000) 2024-03-06T16:06:21.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:21.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:21.999746+0000) 2024-03-06T16:06:22.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:22.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:22.999862+0000) 2024-03-06T16:06:23.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:23.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:24.000119+0000) 2024-03-06T16:06:24.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:24.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:25.000330+0000) 2024-03-06T16:06:25.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:25.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:26.000584+0000) 2024-03-06T16:06:26.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:26.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:27.000807+0000) 2024-03-06T16:06:27.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:28.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:28.000993+0000) 2024-03-06T16:06:28.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:29.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:29.001262+0000) 2024-03-06T16:06:29.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:30.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:30.001521+0000) 2024-03-06T16:06:30.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:31.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:31.001748+0000) 2024-03-06T16:06:31.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:32.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:32.001986+0000) 2024-03-06T16:06:32.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:32.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:06:32.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:33.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:33.002228+0000) 2024-03-06T16:06:33.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:34.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:34.002468+0000) 2024-03-06T16:06:34.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:35.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:35.002712+0000) 2024-03-06T16:06:35.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:36.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:36.002927+0000) 2024-03-06T16:06:36.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:37.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:37.003161+0000) 2024-03-06T16:06:37.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:38.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:38.003411+0000) 2024-03-06T16:06:38.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:39.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:39.003686+0000) 2024-03-06T16:06:39.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:40.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:40.003959+0000) 2024-03-06T16:06:40.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:41.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:41.004242+0000) 2024-03-06T16:06:41.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:42.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:42.004520+0000) 2024-03-06T16:06:42.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:43.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:43.004769+0000) 2024-03-06T16:06:43.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:44.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:44.005095+0000) 2024-03-06T16:06:44.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:45.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:45.005338+0000) 2024-03-06T16:06:45.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:46.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:46.005571+0000) 2024-03-06T16:06:46.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:47.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:47.005869+0000) 2024-03-06T16:06:47.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:48.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:48.006119+0000) 2024-03-06T16:06:48.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:49.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:49.006318+0000) 2024-03-06T16:06:49.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:50.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:50.006498+0000) 2024-03-06T16:06:50.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:51.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:51.006694+0000) 2024-03-06T16:06:51.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:52.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:52.006939+0000) 2024-03-06T16:06:52.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:53.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:53.007161+0000) 2024-03-06T16:06:53.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:54.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:54.007388+0000) 2024-03-06T16:06:54.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:55.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:55.007598+0000) 2024-03-06T16:06:55.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:56.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:56.007948+0000) 2024-03-06T16:06:56.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:57.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:57.008124+0000) 2024-03-06T16:06:58.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:58.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:58.008336+0000) 2024-03-06T16:06:59.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:06:59.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:06:59.008561+0000) 2024-03-06T16:07:00.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:00.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:00.008787+0000) 2024-03-06T16:07:01.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:01.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:01.008978+0000) 2024-03-06T16:07:02.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:02.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:02.009249+0000) 2024-03-06T16:07:02.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:02.369 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:03.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:03.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:03.009470+0000) 2024-03-06T16:07:04.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:04.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:04.009697+0000) 2024-03-06T16:07:05.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:05.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:05.009917+0000) 2024-03-06T16:07:06.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:06.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:06.010149+0000) 2024-03-06T16:07:07.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:07.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:07.010377+0000) 2024-03-06T16:07:08.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:08.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:08.010611+0000) 2024-03-06T16:07:09.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:09.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:09.010831+0000) 2024-03-06T16:07:10.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:10.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:10.011150+0000) 2024-03-06T16:07:11.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:11.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:11.011386+0000) 2024-03-06T16:07:12.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:12.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:12.011620+0000) 2024-03-06T16:07:13.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:13.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:13.011884+0000) 2024-03-06T16:07:14.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:14.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:14.012116+0000) 2024-03-06T16:07:15.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:15.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:15.012335+0000) 2024-03-06T16:07:16.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:16.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:16.012583+0000) 2024-03-06T16:07:17.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:17.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:17.012817+0000) 2024-03-06T16:07:18.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:18.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:18.013070+0000) 2024-03-06T16:07:19.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:19.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:19.013295+0000) 2024-03-06T16:07:20.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:20.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:20.013528+0000) 2024-03-06T16:07:21.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:21.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:21.013747+0000) 2024-03-06T16:07:22.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:22.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:22.013937+0000) 2024-03-06T16:07:23.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:23.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:23.014174+0000) 2024-03-06T16:07:24.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:24.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:24.014389+0000) 2024-03-06T16:07:25.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:25.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:25.014605+0000) 2024-03-06T16:07:26.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:26.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:26.014859+0000) 2024-03-06T16:07:27.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:27.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:27.015134+0000) 2024-03-06T16:07:28.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:28.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:28.015379+0000) 2024-03-06T16:07:29.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:29.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:29.015601+0000) 2024-03-06T16:07:30.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:30.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:30.015844+0000) 2024-03-06T16:07:31.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:31.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:31.016115+0000) 2024-03-06T16:07:32.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:32.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:32.016361+0000) 2024-03-06T16:07:32.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:32.508 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:07:33.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:33.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:33.016586+0000) 2024-03-06T16:07:34.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:34.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:34.016808+0000) 2024-03-06T16:07:35.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:35.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:35.017092+0000) 2024-03-06T16:07:36.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:36.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:36.017331+0000) 2024-03-06T16:07:37.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:37.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:37.017567+0000) 2024-03-06T16:07:38.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:38.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:38.017779+0000) 2024-03-06T16:07:39.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:39.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:39.017971+0000) 2024-03-06T16:07:40.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:40.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:40.018237+0000) 2024-03-06T16:07:41.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:41.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:41.018394+0000) 2024-03-06T16:07:42.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:42.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:42.018572+0000) 2024-03-06T16:07:43.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:43.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:43.018793+0000) 2024-03-06T16:07:44.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:44.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:44.019044+0000) 2024-03-06T16:07:45.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:45.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:45.019260+0000) 2024-03-06T16:07:46.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:46.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:46.019442+0000) 2024-03-06T16:07:47.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:47.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:47.019665+0000) 2024-03-06T16:07:48.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:48.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:48.019893+0000) 2024-03-06T16:07:49.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:49.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:49.020169+0000) 2024-03-06T16:07:50.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:50.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:50.020371+0000) 2024-03-06T16:07:51.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:51.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:51.020488+0000) 2024-03-06T16:07:52.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:52.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:52.020717+0000) 2024-03-06T16:07:53.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:53.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:53.020958+0000) 2024-03-06T16:07:54.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:54.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:54.021189+0000) 2024-03-06T16:07:55.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:55.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:55.021412+0000) 2024-03-06T16:07:56.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:56.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:56.021686+0000) 2024-03-06T16:07:57.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:57.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:57.021910+0000) 2024-03-06T16:07:58.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:58.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:58.022143+0000) 2024-03-06T16:07:59.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:07:59.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:07:59.022336+0000) 2024-03-06T16:08:00.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:00.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:00.022549+0000) 2024-03-06T16:08:01.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:01.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:01.022771+0000) 2024-03-06T16:08:02.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:02.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:02.023062+0000) 2024-03-06T16:08:02.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:02.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:03.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:03.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:03.023399+0000) 2024-03-06T16:08:04.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:04.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:04.023684+0000) 2024-03-06T16:08:05.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:05.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:05.023885+0000) 2024-03-06T16:08:06.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:06.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:06.024167+0000) 2024-03-06T16:08:07.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:07.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:07.024428+0000) 2024-03-06T16:08:08.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:08.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:08.024669+0000) 2024-03-06T16:08:09.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:09.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:09.024946+0000) 2024-03-06T16:08:10.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:10.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:10.025207+0000) 2024-03-06T16:08:11.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:11.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:11.025488+0000) 2024-03-06T16:08:12.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:12.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:12.025706+0000) 2024-03-06T16:08:13.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:13.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:13.025946+0000) 2024-03-06T16:08:14.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:14.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:14.026152+0000) 2024-03-06T16:08:15.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:15.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:15.026357+0000) 2024-03-06T16:08:16.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:16.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:16.026631+0000) 2024-03-06T16:08:17.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:17.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:17.026881+0000) 2024-03-06T16:08:18.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:18.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:18.027182+0000) 2024-03-06T16:08:19.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:19.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:19.027405+0000) 2024-03-06T16:08:20.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:20.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:20.027673+0000) 2024-03-06T16:08:21.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:21.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:21.027942+0000) 2024-03-06T16:08:22.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:22.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:22.028225+0000) 2024-03-06T16:08:23.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:23.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:23.028427+0000) 2024-03-06T16:08:24.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:24.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:24.028651+0000) 2024-03-06T16:08:25.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:25.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:25.028871+0000) 2024-03-06T16:08:26.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:26.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:26.029120+0000) 2024-03-06T16:08:27.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:27.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:27.029358+0000) 2024-03-06T16:08:28.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:28.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:28.029577+0000) 2024-03-06T16:08:29.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:29.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:29.029802+0000) 2024-03-06T16:08:30.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:30.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:30.030053+0000) 2024-03-06T16:08:31.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:31.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:31.030236+0000) 2024-03-06T16:08:32.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:32.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:32.030467+0000) 2024-03-06T16:08:32.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:32.789 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:08:33.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:33.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:33.030684+0000) 2024-03-06T16:08:34.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:34.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:34.030962+0000) 2024-03-06T16:08:35.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:35.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:35.031199+0000) 2024-03-06T16:08:36.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:36.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:36.031459+0000) 2024-03-06T16:08:37.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:37.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:37.031745+0000) 2024-03-06T16:08:38.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:38.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:38.032072+0000) 2024-03-06T16:08:39.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:39.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:39.032337+0000) 2024-03-06T16:08:40.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:40.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:40.032620+0000) 2024-03-06T16:08:41.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:41.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:41.032901+0000) 2024-03-06T16:08:42.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:42.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:42.033168+0000) 2024-03-06T16:08:43.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:43.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:43.033389+0000) 2024-03-06T16:08:44.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:44.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:44.033613+0000) 2024-03-06T16:08:45.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:45.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:45.033810+0000) 2024-03-06T16:08:46.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:46.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:46.034176+0000) 2024-03-06T16:08:47.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:47.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:47.034454+0000) 2024-03-06T16:08:48.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:48.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:48.034653+0000) 2024-03-06T16:08:49.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:49.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:49.034876+0000) 2024-03-06T16:08:50.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:50.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:50.035093+0000) 2024-03-06T16:08:51.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:51.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:51.035298+0000) 2024-03-06T16:08:52.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:52.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:52.035568+0000) 2024-03-06T16:08:53.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:53.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:53.035844+0000) 2024-03-06T16:08:54.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:54.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:54.036144+0000) 2024-03-06T16:08:55.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:55.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:55.036385+0000) 2024-03-06T16:08:56.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:56.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:56.036629+0000) 2024-03-06T16:08:57.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:57.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:57.036887+0000) 2024-03-06T16:08:58.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:58.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:58.037120+0000) 2024-03-06T16:08:59.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:08:59.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:08:59.037351+0000) 2024-03-06T16:09:00.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:00.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:00.037587+0000) 2024-03-06T16:09:01.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:01.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:01.037855+0000) 2024-03-06T16:09:02.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:02.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:02.038161+0000) 2024-03-06T16:09:02.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:02.936 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:03.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:03.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:03.038351+0000) 2024-03-06T16:09:04.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:04.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:04.038580+0000) 2024-03-06T16:09:05.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:05.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:05.038819+0000) 2024-03-06T16:09:06.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:06.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:06.039094+0000) 2024-03-06T16:09:07.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:07.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:07.039336+0000) 2024-03-06T16:09:08.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:08.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:08.039600+0000) 2024-03-06T16:09:09.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:09.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:09.039769+0000) 2024-03-06T16:09:10.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:10.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:10.039986+0000) 2024-03-06T16:09:11.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:11.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:11.040234+0000) 2024-03-06T16:09:12.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:12.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:12.040463+0000) 2024-03-06T16:09:13.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:13.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:13.040659+0000) 2024-03-06T16:09:14.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:14.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:14.040882+0000) 2024-03-06T16:09:15.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:15.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:15.041111+0000) 2024-03-06T16:09:16.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:16.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:16.041352+0000) 2024-03-06T16:09:17.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:17.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:17.041610+0000) 2024-03-06T16:09:18.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:18.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:18.041772+0000) 2024-03-06T16:09:19.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:19.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:19.041985+0000) 2024-03-06T16:09:20.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:20.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:20.042227+0000) 2024-03-06T16:09:21.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:21.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:21.042452+0000) 2024-03-06T16:09:22.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:22.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:22.042716+0000) 2024-03-06T16:09:23.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:23.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:23.042936+0000) 2024-03-06T16:09:24.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:24.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:24.043185+0000) 2024-03-06T16:09:25.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:25.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:25.043422+0000) 2024-03-06T16:09:26.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:26.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:26.043716+0000) 2024-03-06T16:09:27.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:27.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:27.044053+0000) 2024-03-06T16:09:28.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:28.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:28.044336+0000) 2024-03-06T16:09:29.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:29.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:29.044610+0000) 2024-03-06T16:09:30.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:30.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:30.044888+0000) 2024-03-06T16:09:31.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:31.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:31.045175+0000) 2024-03-06T16:09:32.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:32.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:32.045452+0000) 2024-03-06T16:09:33.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:33.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:33.045731+0000) 2024-03-06T16:09:33.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:33.083 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:09:34.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:34.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:34.046085+0000) 2024-03-06T16:09:35.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:35.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:35.046289+0000) 2024-03-06T16:09:36.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:36.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:36.046516+0000) 2024-03-06T16:09:37.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:37.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:37.046703+0000) 2024-03-06T16:09:38.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:38.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:38.046856+0000) 2024-03-06T16:09:39.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:39.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:39.047152+0000) 2024-03-06T16:09:40.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:40.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:40.047383+0000) 2024-03-06T16:09:41.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:41.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:41.047609+0000) 2024-03-06T16:09:42.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:42.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:42.047823+0000) 2024-03-06T16:09:43.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:43.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:43.048104+0000) 2024-03-06T16:09:44.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:44.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:44.048341+0000) 2024-03-06T16:09:45.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:45.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:45.048539+0000) 2024-03-06T16:09:46.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:46.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:46.048746+0000) 2024-03-06T16:09:47.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:47.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:47.048965+0000) 2024-03-06T16:09:48.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:48.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:48.049140+0000) 2024-03-06T16:09:49.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:49.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:49.049356+0000) 2024-03-06T16:09:50.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:50.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:50.049583+0000) 2024-03-06T16:09:51.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:51.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:51.049802+0000) 2024-03-06T16:09:52.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:52.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:52.050070+0000) 2024-03-06T16:09:53.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:53.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:53.050292+0000) 2024-03-06T16:09:54.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:54.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:54.050471+0000) 2024-03-06T16:09:55.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:55.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:55.050695+0000) 2024-03-06T16:09:56.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:56.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:56.050904+0000) 2024-03-06T16:09:57.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:57.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:57.051161+0000) 2024-03-06T16:09:58.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:58.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:58.051413+0000) 2024-03-06T16:09:59.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:09:59.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:09:59.051613+0000) 2024-03-06T16:10:00.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:00.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:00.051816+0000) 2024-03-06T16:10:01.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:01.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:01.052044+0000) 2024-03-06T16:10:02.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:02.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:02.052292+0000) 2024-03-06T16:10:03.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:03.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:03.052575+0000) 2024-03-06T16:10:03.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:03.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:04.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:04.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:04.052862+0000) 2024-03-06T16:10:05.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:05.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:05.053144+0000) 2024-03-06T16:10:06.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:06.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:06.053406+0000) 2024-03-06T16:10:07.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:07.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:07.053664+0000) 2024-03-06T16:10:08.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:08.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:08.053934+0000) 2024-03-06T16:10:09.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:09.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:09.054194+0000) 2024-03-06T16:10:10.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:10.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:10.054432+0000) 2024-03-06T16:10:11.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:11.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:11.054689+0000) 2024-03-06T16:10:12.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:12.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:12.054901+0000) 2024-03-06T16:10:13.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:13.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:13.055153+0000) 2024-03-06T16:10:14.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:14.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:14.055381+0000) 2024-03-06T16:10:15.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:15.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:15.055637+0000) 2024-03-06T16:10:16.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:16.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:16.055880+0000) 2024-03-06T16:10:17.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:17.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:17.056149+0000) 2024-03-06T16:10:18.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:18.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:18.056390+0000) 2024-03-06T16:10:19.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:19.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:19.056606+0000) 2024-03-06T16:10:20.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:20.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:20.056810+0000) 2024-03-06T16:10:21.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:21.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:21.057062+0000) 2024-03-06T16:10:22.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:22.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:22.057293+0000) 2024-03-06T16:10:23.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:23.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:23.057520+0000) 2024-03-06T16:10:24.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:24.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:24.057747+0000) 2024-03-06T16:10:25.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:25.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:25.057967+0000) 2024-03-06T16:10:26.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:26.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:26.058233+0000) 2024-03-06T16:10:27.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:27.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:27.058453+0000) 2024-03-06T16:10:28.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:28.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:28.058678+0000) 2024-03-06T16:10:29.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:29.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:29.058855+0000) 2024-03-06T16:10:30.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:30.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:30.059132+0000) 2024-03-06T16:10:31.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:31.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:31.059367+0000) 2024-03-06T16:10:32.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:32.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:32.059600+0000) 2024-03-06T16:10:33.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:33.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:33.059825+0000) 2024-03-06T16:10:33.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:33.362 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:10:34.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:34.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:34.060097+0000) 2024-03-06T16:10:35.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:35.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:35.060323+0000) 2024-03-06T16:10:36.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:36.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:36.060562+0000) 2024-03-06T16:10:37.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:37.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:37.060719+0000) 2024-03-06T16:10:38.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:38.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:38.060928+0000) 2024-03-06T16:10:39.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:39.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:39.061184+0000) 2024-03-06T16:10:40.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:40.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:40.061405+0000) 2024-03-06T16:10:41.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:41.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:41.061642+0000) 2024-03-06T16:10:42.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:42.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:42.061841+0000) 2024-03-06T16:10:43.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:43.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:43.062116+0000) 2024-03-06T16:10:44.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:44.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:44.062315+0000) 2024-03-06T16:10:45.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:45.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:45.062554+0000) 2024-03-06T16:10:46.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:46.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:46.062833+0000) 2024-03-06T16:10:47.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:47.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:47.063122+0000) 2024-03-06T16:10:48.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:48.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:48.063341+0000) 2024-03-06T16:10:49.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:49.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:49.063535+0000) 2024-03-06T16:10:50.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:50.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:50.063754+0000) 2024-03-06T16:10:51.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:51.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:51.063975+0000) 2024-03-06T16:10:52.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:52.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:52.064178+0000) 2024-03-06T16:10:53.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:53.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:53.064398+0000) 2024-03-06T16:10:54.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:54.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:54.064622+0000) 2024-03-06T16:10:55.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:55.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:55.064838+0000) 2024-03-06T16:10:56.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:56.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:56.065067+0000) 2024-03-06T16:10:57.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:57.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:57.065283+0000) 2024-03-06T16:10:58.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:58.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:58.065505+0000) 2024-03-06T16:10:59.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:10:59.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:10:59.065737+0000) 2024-03-06T16:11:00.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:00.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:00.065961+0000) 2024-03-06T16:11:01.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:01.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:01.066168+0000) 2024-03-06T16:11:02.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:02.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:02.066426+0000) 2024-03-06T16:11:03.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:03.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:03.066664+0000) 2024-03-06T16:11:03.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:03.527 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:04.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:04.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:04.066892+0000) 2024-03-06T16:11:05.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:05.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:05.067133+0000) 2024-03-06T16:11:06.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:06.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:06.067352+0000) 2024-03-06T16:11:07.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:07.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:07.067604+0000) 2024-03-06T16:11:08.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:08.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:08.067854+0000) 2024-03-06T16:11:09.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:09.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:09.068104+0000) 2024-03-06T16:11:10.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:10.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:10.068338+0000) 2024-03-06T16:11:11.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:11.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:11.068508+0000) 2024-03-06T16:11:12.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:12.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:12.068729+0000) 2024-03-06T16:11:13.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:13.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:13.068949+0000) 2024-03-06T16:11:14.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:14.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:14.069149+0000) 2024-03-06T16:11:15.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:15.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:15.069392+0000) 2024-03-06T16:11:16.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:16.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:16.069618+0000) 2024-03-06T16:11:17.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:17.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:17.069862+0000) 2024-03-06T16:11:18.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:18.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:18.070135+0000) 2024-03-06T16:11:19.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:19.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:19.070337+0000) 2024-03-06T16:11:20.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:20.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:20.070574+0000) 2024-03-06T16:11:21.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:21.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:21.070821+0000) 2024-03-06T16:11:22.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:22.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:22.071098+0000) 2024-03-06T16:11:23.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:23.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:23.071306+0000) 2024-03-06T16:11:24.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:24.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:24.071535+0000) 2024-03-06T16:11:25.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:25.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:25.071793+0000) 2024-03-06T16:11:26.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:26.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:26.072102+0000) 2024-03-06T16:11:27.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:27.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:27.072347+0000) 2024-03-06T16:11:28.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:28.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:28.072546+0000) 2024-03-06T16:11:29.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:29.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:29.072716+0000) 2024-03-06T16:11:30.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:30.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:30.072906+0000) 2024-03-06T16:11:31.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:31.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:31.073139+0000) 2024-03-06T16:11:32.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:32.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:32.073317+0000) 2024-03-06T16:11:33.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:33.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:33.073545+0000) 2024-03-06T16:11:33.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:33.700 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:11:34.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:34.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:34.073785+0000) 2024-03-06T16:11:35.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:35.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:35.074034+0000) 2024-03-06T16:11:36.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:36.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:36.074319+0000) 2024-03-06T16:11:37.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:37.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:37.074547+0000) 2024-03-06T16:11:38.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:38.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:38.074776+0000) 2024-03-06T16:11:39.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:39.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:39.074999+0000) 2024-03-06T16:11:40.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:40.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:40.075267+0000) 2024-03-06T16:11:41.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:41.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:41.075460+0000) 2024-03-06T16:11:42.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:42.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:42.075690+0000) 2024-03-06T16:11:43.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:43.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:43.075927+0000) 2024-03-06T16:11:44.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:44.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:44.076152+0000) 2024-03-06T16:11:45.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:45.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:45.076375+0000) 2024-03-06T16:11:46.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:46.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:46.076605+0000) 2024-03-06T16:11:47.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:47.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:47.076827+0000) 2024-03-06T16:11:48.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:48.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:48.077097+0000) 2024-03-06T16:11:49.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:49.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:49.077314+0000) 2024-03-06T16:11:50.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:50.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:50.077524+0000) 2024-03-06T16:11:51.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:51.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:51.077750+0000) 2024-03-06T16:11:52.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:52.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:52.077974+0000) 2024-03-06T16:11:53.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:53.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:53.078217+0000) 2024-03-06T16:11:54.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:54.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:54.078408+0000) 2024-03-06T16:11:55.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:55.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:55.078629+0000) 2024-03-06T16:11:56.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:56.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:56.078862+0000) 2024-03-06T16:11:57.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:57.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:57.079115+0000) 2024-03-06T16:11:58.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:58.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:58.079311+0000) 2024-03-06T16:11:59.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:11:59.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:11:59.079520+0000) 2024-03-06T16:12:00.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:00.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:00.079716+0000) 2024-03-06T16:12:01.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:01.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:01.079896+0000) 2024-03-06T16:12:02.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:02.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:02.080191+0000) 2024-03-06T16:12:03.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:03.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:03.080440+0000) 2024-03-06T16:12:03.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:03.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:04.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:04.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:04.080685+0000) 2024-03-06T16:12:05.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:05.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:05.080908+0000) 2024-03-06T16:12:06.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:06.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:06.081154+0000) 2024-03-06T16:12:07.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:07.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:07.081391+0000) 2024-03-06T16:12:08.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:08.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:08.081748+0000) 2024-03-06T16:12:09.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:09.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:09.081996+0000) 2024-03-06T16:12:10.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:10.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:10.082264+0000) 2024-03-06T16:12:11.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:11.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:11.082490+0000) 2024-03-06T16:12:12.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:12.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:12.082711+0000) 2024-03-06T16:12:13.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:13.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:13.082934+0000) 2024-03-06T16:12:14.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:14.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:14.083163+0000) 2024-03-06T16:12:15.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:15.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:15.083382+0000) 2024-03-06T16:12:16.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:16.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:16.083650+0000) 2024-03-06T16:12:17.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:17.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:17.083897+0000) 2024-03-06T16:12:18.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:18.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:18.084166+0000) 2024-03-06T16:12:19.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:19.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:19.084417+0000) 2024-03-06T16:12:20.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:20.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:20.084664+0000) 2024-03-06T16:12:21.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:21.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:21.084896+0000) 2024-03-06T16:12:22.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:22.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:22.085123+0000) 2024-03-06T16:12:23.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:23.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:23.085314+0000) 2024-03-06T16:12:24.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:24.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:24.085544+0000) 2024-03-06T16:12:25.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:25.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:25.085764+0000) 2024-03-06T16:12:26.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:26.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:26.085960+0000) 2024-03-06T16:12:27.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:27.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:27.086191+0000) 2024-03-06T16:12:28.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:28.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:28.086370+0000) 2024-03-06T16:12:29.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:29.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:29.086598+0000) 2024-03-06T16:12:30.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:30.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:30.086822+0000) 2024-03-06T16:12:31.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:31.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:31.087087+0000) 2024-03-06T16:12:32.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:32.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:32.087315+0000) 2024-03-06T16:12:33.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:33.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:33.087536+0000) 2024-03-06T16:12:33.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:33.987 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:12:34.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:34.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:34.087666+0000) 2024-03-06T16:12:35.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:35.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:35.087882+0000) 2024-03-06T16:12:36.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:36.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:36.088136+0000) 2024-03-06T16:12:37.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:37.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:37.088363+0000) 2024-03-06T16:12:38.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:38.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:38.088581+0000) 2024-03-06T16:12:39.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:39.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:39.088806+0000) 2024-03-06T16:12:40.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:40.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:40.089073+0000) 2024-03-06T16:12:41.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:41.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:41.089315+0000) 2024-03-06T16:12:42.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:42.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:42.089552+0000) 2024-03-06T16:12:43.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:43.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:43.089800+0000) 2024-03-06T16:12:44.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:44.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:44.090082+0000) 2024-03-06T16:12:45.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:45.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:45.090308+0000) 2024-03-06T16:12:46.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:46.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:46.090523+0000) 2024-03-06T16:12:47.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:47.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:47.090730+0000) 2024-03-06T16:12:48.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:48.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:48.090971+0000) 2024-03-06T16:12:49.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:49.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:49.091207+0000) 2024-03-06T16:12:50.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:50.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:50.091413+0000) 2024-03-06T16:12:51.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:51.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:51.091620+0000) 2024-03-06T16:12:52.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:52.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:52.091816+0000) 2024-03-06T16:12:53.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:53.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:53.092104+0000) 2024-03-06T16:12:54.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:54.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:54.092367+0000) 2024-03-06T16:12:55.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:55.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:55.092621+0000) 2024-03-06T16:12:56.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:56.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:56.092880+0000) 2024-03-06T16:12:57.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:57.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:57.093123+0000) 2024-03-06T16:12:58.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:58.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:58.093335+0000) 2024-03-06T16:12:59.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:12:59.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:12:59.093547+0000) 2024-03-06T16:13:00.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:00.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:00.093790+0000) 2024-03-06T16:13:01.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:01.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:01.094059+0000) 2024-03-06T16:13:02.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:02.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:02.094321+0000) 2024-03-06T16:13:03.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:03.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:03.094514+0000) 2024-03-06T16:13:04.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:04.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:04.094750+0000) 2024-03-06T16:13:04.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:04.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:05.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:05.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:05.094998+0000) 2024-03-06T16:13:06.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:06.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:06.095321+0000) 2024-03-06T16:13:07.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:07.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:07.095568+0000) 2024-03-06T16:13:08.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:08.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:08.095807+0000) 2024-03-06T16:13:09.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:09.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:09.096117+0000) 2024-03-06T16:13:10.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:10.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:10.096362+0000) 2024-03-06T16:13:11.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:11.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:11.096585+0000) 2024-03-06T16:13:12.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:12.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:12.096828+0000) 2024-03-06T16:13:13.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:13.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:13.097126+0000) 2024-03-06T16:13:14.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:14.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:14.097374+0000) 2024-03-06T16:13:15.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:15.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:15.097620+0000) 2024-03-06T16:13:16.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:16.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:16.097839+0000) 2024-03-06T16:13:17.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:17.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:17.098093+0000) 2024-03-06T16:13:18.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:18.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:18.098333+0000) 2024-03-06T16:13:19.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:19.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:19.098561+0000) 2024-03-06T16:13:20.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:20.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:20.098782+0000) 2024-03-06T16:13:21.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:21.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:21.098943+0000) 2024-03-06T16:13:22.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:22.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:22.099167+0000) 2024-03-06T16:13:23.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:23.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:23.099400+0000) 2024-03-06T16:13:24.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:24.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:24.099650+0000) 2024-03-06T16:13:25.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:25.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:25.099854+0000) 2024-03-06T16:13:26.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:26.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:26.100126+0000) 2024-03-06T16:13:27.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:27.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:27.100312+0000) 2024-03-06T16:13:28.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:28.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:28.100609+0000) 2024-03-06T16:13:29.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:29.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:29.100857+0000) 2024-03-06T16:13:30.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:30.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:30.101097+0000) 2024-03-06T16:13:31.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:31.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:31.101273+0000) 2024-03-06T16:13:32.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:32.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:32.101511+0000) 2024-03-06T16:13:33.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:33.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:33.101768+0000) 2024-03-06T16:13:34.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:34.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:34.101993+0000) 2024-03-06T16:13:34.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:34.266 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:13:35.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:35.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:35.102233+0000) 2024-03-06T16:13:36.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:36.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:36.102470+0000) 2024-03-06T16:13:37.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:37.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:37.102681+0000) 2024-03-06T16:13:38.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:38.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:38.102908+0000) 2024-03-06T16:13:39.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:39.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:39.103131+0000) 2024-03-06T16:13:40.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:40.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:40.103348+0000) 2024-03-06T16:13:41.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:41.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:41.103585+0000) 2024-03-06T16:13:42.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:42.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:42.103811+0000) 2024-03-06T16:13:43.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:43.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:43.104039+0000) 2024-03-06T16:13:44.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:44.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:44.104287+0000) 2024-03-06T16:13:45.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:45.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:45.104528+0000) 2024-03-06T16:13:46.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:46.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:46.104786+0000) 2024-03-06T16:13:47.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:47.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:47.105059+0000) 2024-03-06T16:13:48.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:48.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:48.105309+0000) 2024-03-06T16:13:49.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:49.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:49.105529+0000) 2024-03-06T16:13:50.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:50.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:50.105732+0000) 2024-03-06T16:13:51.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:51.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:51.105956+0000) 2024-03-06T16:13:52.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:52.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:52.106215+0000) 2024-03-06T16:13:53.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:53.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:53.106472+0000) 2024-03-06T16:13:54.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:54.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:54.106643+0000) 2024-03-06T16:13:55.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:55.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:55.106864+0000) 2024-03-06T16:13:56.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:56.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:56.107133+0000) 2024-03-06T16:13:57.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:57.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:57.107362+0000) 2024-03-06T16:13:58.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:58.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:58.107600+0000) 2024-03-06T16:13:59.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:13:59.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:13:59.107823+0000) 2024-03-06T16:14:00.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:00.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:00.108080+0000) 2024-03-06T16:14:01.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:01.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:01.108299+0000) 2024-03-06T16:14:02.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:02.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:02.108525+0000) 2024-03-06T16:14:03.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:03.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:03.108758+0000) 2024-03-06T16:14:04.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:04.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:04.108984+0000) 2024-03-06T16:14:04.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:04.400 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:05.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:05.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:05.109254+0000) 2024-03-06T16:14:06.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:06.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:06.109499+0000) 2024-03-06T16:14:07.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:07.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:07.109720+0000) 2024-03-06T16:14:08.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:08.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:08.109966+0000) 2024-03-06T16:14:09.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:09.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:09.110188+0000) 2024-03-06T16:14:10.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:10.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:10.110367+0000) 2024-03-06T16:14:11.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:11.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:11.110608+0000) 2024-03-06T16:14:12.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:12.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:12.110855+0000) 2024-03-06T16:14:13.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:13.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:13.111149+0000) 2024-03-06T16:14:14.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:14.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:14.111347+0000) 2024-03-06T16:14:15.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:15.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:15.111545+0000) 2024-03-06T16:14:16.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:16.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:16.111780+0000) 2024-03-06T16:14:17.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:17.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:17.112038+0000) 2024-03-06T16:14:18.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:18.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:18.112276+0000) 2024-03-06T16:14:19.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:19.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:19.112517+0000) 2024-03-06T16:14:20.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:20.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:20.112737+0000) 2024-03-06T16:14:21.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:21.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:21.112915+0000) 2024-03-06T16:14:22.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:22.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:22.113144+0000) 2024-03-06T16:14:23.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:23.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:23.113414+0000) 2024-03-06T16:14:24.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:24.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:24.113698+0000) 2024-03-06T16:14:25.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:25.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:25.113979+0000) 2024-03-06T16:14:26.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:26.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:26.114353+0000) 2024-03-06T16:14:27.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:27.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:27.114630+0000) 2024-03-06T16:14:28.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:28.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:28.114896+0000) 2024-03-06T16:14:29.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:29.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:29.115172+0000) 2024-03-06T16:14:30.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:30.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:30.115372+0000) 2024-03-06T16:14:31.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:31.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:31.115573+0000) 2024-03-06T16:14:32.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:32.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:32.115796+0000) 2024-03-06T16:14:33.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:33.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:33.116060+0000) 2024-03-06T16:14:34.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:34.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:34.116338+0000) 2024-03-06T16:14:34.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:34.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:14:35.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:35.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:35.116559+0000) 2024-03-06T16:14:36.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:36.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:36.116806+0000) 2024-03-06T16:14:37.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:37.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:37.117087+0000) 2024-03-06T16:14:38.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:38.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:38.117385+0000) 2024-03-06T16:14:39.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:39.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:39.117608+0000) 2024-03-06T16:14:40.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:40.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:40.117841+0000) 2024-03-06T16:14:41.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:41.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:41.118110+0000) 2024-03-06T16:14:42.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:42.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:42.118333+0000) 2024-03-06T16:14:43.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:43.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:43.118577+0000) 2024-03-06T16:14:44.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:44.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:44.118812+0000) 2024-03-06T16:14:45.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:45.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:45.119078+0000) 2024-03-06T16:14:46.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:46.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:46.119312+0000) 2024-03-06T16:14:47.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:47.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:47.119539+0000) 2024-03-06T16:14:48.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:48.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:48.119812+0000) 2024-03-06T16:14:49.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:49.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:49.120086+0000) 2024-03-06T16:14:50.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:50.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:50.120307+0000) 2024-03-06T16:14:51.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:51.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:51.120547+0000) 2024-03-06T16:14:52.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:52.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:52.120773+0000) 2024-03-06T16:14:53.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:53.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:53.121042+0000) 2024-03-06T16:14:54.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:54.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:54.121277+0000) 2024-03-06T16:14:55.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:55.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:55.121501+0000) 2024-03-06T16:14:56.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:56.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:56.121728+0000) 2024-03-06T16:14:57.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:57.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:57.121952+0000) 2024-03-06T16:14:58.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:58.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:58.122198+0000) 2024-03-06T16:14:59.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:14:59.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:14:59.122426+0000) 2024-03-06T16:15:00.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:00.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:00.122658+0000) 2024-03-06T16:15:01.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:01.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:01.122883+0000) 2024-03-06T16:15:02.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:02.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:02.123112+0000) 2024-03-06T16:15:03.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:03.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:03.123345+0000) 2024-03-06T16:15:04.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:04.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:04.123548+0000) 2024-03-06T16:15:04.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:04.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:05.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:05.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:05.123782+0000) 2024-03-06T16:15:06.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:06.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:06.124053+0000) 2024-03-06T16:15:07.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:07.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:07.124236+0000) 2024-03-06T16:15:08.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:08.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:08.124498+0000) 2024-03-06T16:15:09.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:09.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:09.124740+0000) 2024-03-06T16:15:10.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:10.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:10.124986+0000) 2024-03-06T16:15:11.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:11.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:11.125283+0000) 2024-03-06T16:15:12.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:12.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:12.125532+0000) 2024-03-06T16:15:13.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:13.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:13.125788+0000) 2024-03-06T16:15:14.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:14.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:14.126069+0000) 2024-03-06T16:15:15.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:15.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:15.126319+0000) 2024-03-06T16:15:16.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:16.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:16.126600+0000) 2024-03-06T16:15:17.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:17.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:17.126838+0000) 2024-03-06T16:15:18.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:18.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:18.127131+0000) 2024-03-06T16:15:19.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:19.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:19.127358+0000) 2024-03-06T16:15:20.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:20.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:20.127564+0000) 2024-03-06T16:15:21.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:21.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:21.127791+0000) 2024-03-06T16:15:22.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:22.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:22.128102+0000) 2024-03-06T16:15:23.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:23.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:23.128357+0000) 2024-03-06T16:15:24.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:24.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:24.128568+0000) 2024-03-06T16:15:25.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:25.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:25.128826+0000) 2024-03-06T16:15:26.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:26.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:26.129063+0000) 2024-03-06T16:15:27.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:27.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:27.129303+0000) 2024-03-06T16:15:28.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:28.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:28.129530+0000) 2024-03-06T16:15:29.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:29.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:29.129759+0000) 2024-03-06T16:15:30.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:30.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:30.129975+0000) 2024-03-06T16:15:31.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:31.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:31.130298+0000) 2024-03-06T16:15:32.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:32.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:32.130548+0000) 2024-03-06T16:15:33.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:33.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:33.130789+0000) 2024-03-06T16:15:34.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:34.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:34.131054+0000) 2024-03-06T16:15:34.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:34.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:15:35.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:35.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:35.131244+0000) 2024-03-06T16:15:36.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:36.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:36.131450+0000) 2024-03-06T16:15:37.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:37.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:37.131669+0000) 2024-03-06T16:15:38.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:38.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:38.131911+0000) 2024-03-06T16:15:39.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:39.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:39.132158+0000) 2024-03-06T16:15:40.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:40.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:40.132413+0000) 2024-03-06T16:15:41.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:41.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:41.132672+0000) 2024-03-06T16:15:42.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:42.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:42.132926+0000) 2024-03-06T16:15:43.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:43.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:43.133214+0000) 2024-03-06T16:15:44.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:44.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:44.133488+0000) 2024-03-06T16:15:45.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:45.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:45.133698+0000) 2024-03-06T16:15:46.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:46.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:46.133931+0000) 2024-03-06T16:15:47.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:47.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:47.134160+0000) 2024-03-06T16:15:48.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:48.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:48.134384+0000) 2024-03-06T16:15:49.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:49.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:49.134637+0000) 2024-03-06T16:15:50.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:50.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:50.134869+0000) 2024-03-06T16:15:51.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:51.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:51.135114+0000) 2024-03-06T16:15:52.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:52.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:52.135322+0000) 2024-03-06T16:15:53.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:53.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:53.135546+0000) 2024-03-06T16:15:54.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:54.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:54.135755+0000) 2024-03-06T16:15:55.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:55.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:55.135994+0000) 2024-03-06T16:15:56.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:56.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:56.136285+0000) 2024-03-06T16:15:57.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:57.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:57.136538+0000) 2024-03-06T16:15:58.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:58.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:58.136790+0000) 2024-03-06T16:15:59.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:15:59.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:15:59.137078+0000) 2024-03-06T16:16:00.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:00.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:00.137298+0000) 2024-03-06T16:16:01.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:01.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:01.137519+0000) 2024-03-06T16:16:02.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:02.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:02.137758+0000) 2024-03-06T16:16:03.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:03.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:03.138046+0000) 2024-03-06T16:16:04.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:04.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:04.138296+0000) 2024-03-06T16:16:04.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:05.003 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:05.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:05.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:05.138535+0000) 2024-03-06T16:16:06.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:06.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:06.138785+0000) 2024-03-06T16:16:07.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:07.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:07.139080+0000) 2024-03-06T16:16:08.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:08.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:08.139332+0000) 2024-03-06T16:16:09.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:09.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:09.139576+0000) 2024-03-06T16:16:10.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:10.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:10.139818+0000) 2024-03-06T16:16:11.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:11.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:11.140109+0000) 2024-03-06T16:16:12.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:12.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:12.140354+0000) 2024-03-06T16:16:13.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:13.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:13.140606+0000) 2024-03-06T16:16:14.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:14.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:14.140855+0000) 2024-03-06T16:16:15.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:15.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:15.141118+0000) 2024-03-06T16:16:16.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:16.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:16.141315+0000) 2024-03-06T16:16:17.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:17.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:17.141543+0000) 2024-03-06T16:16:18.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:18.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:18.141781+0000) 2024-03-06T16:16:19.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:19.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:19.142149+0000) 2024-03-06T16:16:20.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:20.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:20.142350+0000) 2024-03-06T16:16:21.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:21.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:21.142589+0000) 2024-03-06T16:16:22.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:22.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:22.142820+0000) 2024-03-06T16:16:23.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:23.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:23.143109+0000) 2024-03-06T16:16:24.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:24.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:24.143331+0000) 2024-03-06T16:16:25.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:25.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:25.143575+0000) 2024-03-06T16:16:26.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:26.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:26.143826+0000) 2024-03-06T16:16:27.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:27.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:27.144116+0000) 2024-03-06T16:16:28.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:28.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:28.144356+0000) 2024-03-06T16:16:29.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:29.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:29.144599+0000) 2024-03-06T16:16:30.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:30.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:30.144821+0000) 2024-03-06T16:16:31.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:31.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:31.145131+0000) 2024-03-06T16:16:32.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:32.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:32.145361+0000) 2024-03-06T16:16:33.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:33.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:33.145556+0000) 2024-03-06T16:16:34.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:34.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:34.145793+0000) 2024-03-06T16:16:35.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:35.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:35.146054+0000) 2024-03-06T16:16:35.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:35.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:16:36.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:36.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:36.146297+0000) 2024-03-06T16:16:37.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:37.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:37.146553+0000) 2024-03-06T16:16:38.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:38.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:38.146780+0000) 2024-03-06T16:16:39.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:39.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:39.147040+0000) 2024-03-06T16:16:40.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:40.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:40.147274+0000) 2024-03-06T16:16:41.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:41.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:41.147495+0000) 2024-03-06T16:16:42.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:42.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:42.147740+0000) 2024-03-06T16:16:43.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:43.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:43.147966+0000) 2024-03-06T16:16:44.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:44.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:44.148160+0000) 2024-03-06T16:16:45.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:45.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:45.148409+0000) 2024-03-06T16:16:46.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:46.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:46.148670+0000) 2024-03-06T16:16:47.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:47.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:47.148910+0000) 2024-03-06T16:16:48.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:48.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:48.149150+0000) 2024-03-06T16:16:49.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:49.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:49.149369+0000) 2024-03-06T16:16:50.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:50.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:50.149573+0000) 2024-03-06T16:16:51.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:51.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:51.149794+0000) 2024-03-06T16:16:52.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:52.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:52.150067+0000) 2024-03-06T16:16:53.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:53.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:53.150300+0000) 2024-03-06T16:16:54.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:54.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:54.150528+0000) 2024-03-06T16:16:55.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:55.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:55.150741+0000) 2024-03-06T16:16:56.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:56.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:56.151040+0000) 2024-03-06T16:16:57.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:57.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:57.151271+0000) 2024-03-06T16:16:58.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:58.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:58.151487+0000) 2024-03-06T16:16:59.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:16:59.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:16:59.151713+0000) 2024-03-06T16:17:00.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:00.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:00.151939+0000) 2024-03-06T16:17:01.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:01.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:01.152177+0000) 2024-03-06T16:17:02.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:02.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:02.152402+0000) 2024-03-06T16:17:03.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:03.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:03.152625+0000) 2024-03-06T16:17:04.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:04.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:04.152859+0000) 2024-03-06T16:17:05.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:05.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:05.153131+0000) 2024-03-06T16:17:05.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:05.297 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:06.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:06.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:06.153369+0000) 2024-03-06T16:17:07.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:07.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:07.153606+0000) 2024-03-06T16:17:08.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:08.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:08.153848+0000) 2024-03-06T16:17:09.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:09.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:09.154102+0000) 2024-03-06T16:17:10.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:10.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:10.154325+0000) 2024-03-06T16:17:11.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:11.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:11.154569+0000) 2024-03-06T16:17:12.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:12.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:12.154815+0000) 2024-03-06T16:17:13.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:13.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:13.155102+0000) 2024-03-06T16:17:14.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:14.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:14.155359+0000) 2024-03-06T16:17:15.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:15.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:15.155560+0000) 2024-03-06T16:17:16.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:16.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:16.155795+0000) 2024-03-06T16:17:17.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:17.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:17.156059+0000) 2024-03-06T16:17:18.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:18.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:18.156283+0000) 2024-03-06T16:17:19.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:19.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:19.156507+0000) 2024-03-06T16:17:20.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:20.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:20.156763+0000) 2024-03-06T16:17:21.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:21.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:21.157001+0000) 2024-03-06T16:17:22.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:22.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:22.157275+0000) 2024-03-06T16:17:23.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:23.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:23.157518+0000) 2024-03-06T16:17:24.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:24.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:24.157760+0000) 2024-03-06T16:17:25.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:25.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:25.158001+0000) 2024-03-06T16:17:26.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:26.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:26.158291+0000) 2024-03-06T16:17:27.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:27.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:27.158525+0000) 2024-03-06T16:17:28.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:28.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:28.158760+0000) 2024-03-06T16:17:29.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:29.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:29.158980+0000) 2024-03-06T16:17:30.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:30.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:30.159226+0000) 2024-03-06T16:17:31.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:31.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:31.159454+0000) 2024-03-06T16:17:32.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:32.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:32.159681+0000) 2024-03-06T16:17:33.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:33.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:33.159853+0000) 2024-03-06T16:17:34.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:34.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:34.160125+0000) 2024-03-06T16:17:35.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:35.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:35.160319+0000) 2024-03-06T16:17:35.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:35.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:17:36.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:36.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:36.160553+0000) 2024-03-06T16:17:37.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:37.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:37.160777+0000) 2024-03-06T16:17:38.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:38.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:38.160970+0000) 2024-03-06T16:17:39.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:39.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:39.161219+0000) 2024-03-06T16:17:40.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:40.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:40.161477+0000) 2024-03-06T16:17:41.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:41.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:41.161734+0000) 2024-03-06T16:17:42.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:42.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:42.161986+0000) 2024-03-06T16:17:43.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:43.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:43.162323+0000) 2024-03-06T16:17:44.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:44.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:44.162603+0000) 2024-03-06T16:17:45.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:45.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:45.162808+0000) 2024-03-06T16:17:46.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:46.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:46.163120+0000) 2024-03-06T16:17:47.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:47.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:47.163360+0000) 2024-03-06T16:17:48.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:48.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:48.163584+0000) 2024-03-06T16:17:49.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:49.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:49.163829+0000) 2024-03-06T16:17:50.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:50.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:50.164123+0000) 2024-03-06T16:17:51.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:51.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:51.164375+0000) 2024-03-06T16:17:52.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:52.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:52.164613+0000) 2024-03-06T16:17:53.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:53.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:53.164833+0000) 2024-03-06T16:17:54.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:54.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:54.165141+0000) 2024-03-06T16:17:55.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:55.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:55.165394+0000) 2024-03-06T16:17:56.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:56.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:56.165679+0000) 2024-03-06T16:17:57.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:57.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:57.165921+0000) 2024-03-06T16:17:58.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:58.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:58.166155+0000) 2024-03-06T16:17:59.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:17:59.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:17:59.166375+0000) 2024-03-06T16:18:00.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:00.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:00.166632+0000) 2024-03-06T16:18:01.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:01.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:01.166870+0000) 2024-03-06T16:18:02.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:02.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:02.167124+0000) 2024-03-06T16:18:03.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:03.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:03.167334+0000) 2024-03-06T16:18:04.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:04.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:04.167579+0000) 2024-03-06T16:18:05.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:05.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:05.167822+0000) 2024-03-06T16:18:05.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:05.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:06.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:06.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:06.168156+0000) 2024-03-06T16:18:07.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:07.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:07.168369+0000) 2024-03-06T16:18:08.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:08.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:08.168606+0000) 2024-03-06T16:18:09.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:09.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:09.168851+0000) 2024-03-06T16:18:10.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:10.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:10.169146+0000) 2024-03-06T16:18:11.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:11.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:11.169400+0000) 2024-03-06T16:18:12.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:12.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:12.169644+0000) 2024-03-06T16:18:13.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:13.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:13.169852+0000) 2024-03-06T16:18:14.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:14.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:14.170138+0000) 2024-03-06T16:18:15.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:15.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:15.170377+0000) 2024-03-06T16:18:16.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:16.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:16.170664+0000) 2024-03-06T16:18:17.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:17.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:17.170864+0000) 2024-03-06T16:18:18.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:18.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:18.171131+0000) 2024-03-06T16:18:19.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:19.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:19.171325+0000) 2024-03-06T16:18:20.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:20.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:20.171550+0000) 2024-03-06T16:18:21.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:21.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:21.171772+0000) 2024-03-06T16:18:22.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:22.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:22.172085+0000) 2024-03-06T16:18:23.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:23.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:23.172315+0000) 2024-03-06T16:18:24.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:24.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:24.172553+0000) 2024-03-06T16:18:25.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:25.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:25.172769+0000) 2024-03-06T16:18:26.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:26.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:26.173054+0000) 2024-03-06T16:18:27.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:27.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:27.173303+0000) 2024-03-06T16:18:28.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:28.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:28.173535+0000) 2024-03-06T16:18:29.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:29.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:29.173773+0000) 2024-03-06T16:18:30.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:30.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:30.173995+0000) 2024-03-06T16:18:31.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:31.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:31.174196+0000) 2024-03-06T16:18:32.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:32.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:32.174400+0000) 2024-03-06T16:18:33.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:33.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:33.174616+0000) 2024-03-06T16:18:34.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:34.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:34.174849+0000) 2024-03-06T16:18:35.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:35.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:35.175120+0000) 2024-03-06T16:18:35.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:35.723 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:18:36.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:36.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:36.175341+0000) 2024-03-06T16:18:37.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:37.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:37.175579+0000) 2024-03-06T16:18:38.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:38.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:38.175810+0000) 2024-03-06T16:18:39.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:39.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:39.176113+0000) 2024-03-06T16:18:40.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:40.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:40.176328+0000) 2024-03-06T16:18:41.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:41.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:41.176563+0000) 2024-03-06T16:18:42.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:42.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:42.176804+0000) 2024-03-06T16:18:43.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:43.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:43.177115+0000) 2024-03-06T16:18:44.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:44.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:44.177345+0000) 2024-03-06T16:18:45.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:45.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:45.177549+0000) 2024-03-06T16:18:46.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:46.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:46.177796+0000) 2024-03-06T16:18:47.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:47.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:47.178096+0000) 2024-03-06T16:18:48.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:48.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:48.178371+0000) 2024-03-06T16:18:49.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:49.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:49.178581+0000) 2024-03-06T16:18:50.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:50.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:50.178803+0000) 2024-03-06T16:18:51.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:51.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:51.179071+0000) 2024-03-06T16:18:52.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:52.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:52.179238+0000) 2024-03-06T16:18:53.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:53.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:53.179473+0000) 2024-03-06T16:18:54.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:54.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:54.179727+0000) 2024-03-06T16:18:55.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:55.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:55.179970+0000) 2024-03-06T16:18:56.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:56.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:56.180289+0000) 2024-03-06T16:18:57.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:57.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:57.180540+0000) 2024-03-06T16:18:58.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:58.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:58.180761+0000) 2024-03-06T16:18:59.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:18:59.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:18:59.181041+0000) 2024-03-06T16:19:00.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:00.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:00.181284+0000) 2024-03-06T16:19:01.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:01.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:01.181482+0000) 2024-03-06T16:19:02.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:02.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:02.181707+0000) 2024-03-06T16:19:03.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:03.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:03.181948+0000) 2024-03-06T16:19:04.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:04.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:04.182175+0000) 2024-03-06T16:19:05.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:05.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:05.182367+0000) 2024-03-06T16:19:05.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:05.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:06.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:06.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:06.182586+0000) 2024-03-06T16:19:07.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:07.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:07.182791+0000) 2024-03-06T16:19:08.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:08.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:08.182979+0000) 2024-03-06T16:19:09.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:09.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:09.183218+0000) 2024-03-06T16:19:10.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:10.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:10.183406+0000) 2024-03-06T16:19:11.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:11.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:11.183636+0000) 2024-03-06T16:19:12.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:12.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:12.183850+0000) 2024-03-06T16:19:13.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:13.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:13.184120+0000) 2024-03-06T16:19:14.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:14.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:14.184370+0000) 2024-03-06T16:19:15.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:15.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:15.184619+0000) 2024-03-06T16:19:16.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:16.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:16.184857+0000) 2024-03-06T16:19:17.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:17.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:17.185089+0000) 2024-03-06T16:19:18.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:18.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:18.185287+0000) 2024-03-06T16:19:19.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:19.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:19.185490+0000) 2024-03-06T16:19:20.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:20.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:20.185715+0000) 2024-03-06T16:19:21.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:21.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:21.185958+0000) 2024-03-06T16:19:22.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:22.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:22.186161+0000) 2024-03-06T16:19:23.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:23.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:23.186355+0000) 2024-03-06T16:19:24.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:24.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:24.186548+0000) 2024-03-06T16:19:25.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:25.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:25.186773+0000) 2024-03-06T16:19:26.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:26.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:26.187034+0000) 2024-03-06T16:19:27.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:27.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:27.187245+0000) 2024-03-06T16:19:28.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:28.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:28.187490+0000) 2024-03-06T16:19:29.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:29.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:29.187750+0000) 2024-03-06T16:19:30.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:30.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:30.187975+0000) 2024-03-06T16:19:31.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:31.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:31.188236+0000) 2024-03-06T16:19:32.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:32.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:32.188485+0000) 2024-03-06T16:19:33.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:33.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:33.188718+0000) 2024-03-06T16:19:34.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:34.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:34.188974+0000) 2024-03-06T16:19:35.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:35.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:35.189235+0000) 2024-03-06T16:19:36.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:36.077 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:19:36.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:36.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:36.189462+0000) 2024-03-06T16:19:37.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:37.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:37.189673+0000) 2024-03-06T16:19:38.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:38.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:38.189892+0000) 2024-03-06T16:19:39.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:39.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:39.190128+0000) 2024-03-06T16:19:40.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:40.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:40.190333+0000) 2024-03-06T16:19:41.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:41.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:41.190558+0000) 2024-03-06T16:19:42.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:42.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:42.190781+0000) 2024-03-06T16:19:43.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:43.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:43.191052+0000) 2024-03-06T16:19:44.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:44.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:44.191271+0000) 2024-03-06T16:19:45.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:45.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:45.191495+0000) 2024-03-06T16:19:46.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:46.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:46.191689+0000) 2024-03-06T16:19:47.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:47.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:47.191916+0000) 2024-03-06T16:19:48.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:48.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:48.192139+0000) 2024-03-06T16:19:49.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:49.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:49.192370+0000) 2024-03-06T16:19:50.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:50.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:50.192601+0000) 2024-03-06T16:19:51.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:51.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:51.192824+0000) 2024-03-06T16:19:52.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:52.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:52.193127+0000) 2024-03-06T16:19:53.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:53.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:53.193328+0000) 2024-03-06T16:19:54.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:54.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:54.193567+0000) 2024-03-06T16:19:55.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:55.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:55.193789+0000) 2024-03-06T16:19:56.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:56.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:56.194079+0000) 2024-03-06T16:19:57.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:57.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:57.194319+0000) 2024-03-06T16:19:58.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:58.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:58.194525+0000) 2024-03-06T16:19:59.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:19:59.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:19:59.194768+0000) 2024-03-06T16:20:00.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:00.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:00.194993+0000) 2024-03-06T16:20:01.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:01.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:01.195224+0000) 2024-03-06T16:20:02.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:02.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:02.195454+0000) 2024-03-06T16:20:03.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:03.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:03.195676+0000) 2024-03-06T16:20:04.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:04.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:04.195907+0000) 2024-03-06T16:20:05.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:05.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:05.196134+0000) 2024-03-06T16:20:06.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:06.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:06.196367+0000) 2024-03-06T16:20:06.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:06.216 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:07.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:07.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:07.196565+0000) 2024-03-06T16:20:08.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:08.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:08.196749+0000) 2024-03-06T16:20:09.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:09.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:09.197057+0000) 2024-03-06T16:20:10.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:10.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:10.197272+0000) 2024-03-06T16:20:11.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:11.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:11.197489+0000) 2024-03-06T16:20:12.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:12.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:12.197698+0000) 2024-03-06T16:20:13.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:13.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:13.197925+0000) 2024-03-06T16:20:14.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:14.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:14.198169+0000) 2024-03-06T16:20:15.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:15.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:15.198395+0000) 2024-03-06T16:20:16.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:16.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:16.198595+0000) 2024-03-06T16:20:17.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:17.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:17.198833+0000) 2024-03-06T16:20:18.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:18.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:18.199050+0000) 2024-03-06T16:20:19.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:19.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:19.199306+0000) 2024-03-06T16:20:20.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:20.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:20.199546+0000) 2024-03-06T16:20:21.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:21.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:21.199810+0000) 2024-03-06T16:20:22.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:22.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:22.200168+0000) 2024-03-06T16:20:23.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:23.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:23.200452+0000) 2024-03-06T16:20:24.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:24.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:24.200750+0000) 2024-03-06T16:20:25.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:25.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:25.201069+0000) 2024-03-06T16:20:26.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:26.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:26.201376+0000) 2024-03-06T16:20:27.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:27.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:27.201664+0000) 2024-03-06T16:20:28.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:28.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:28.201884+0000) 2024-03-06T16:20:29.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:29.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:29.202131+0000) 2024-03-06T16:20:30.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:30.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:30.202397+0000) 2024-03-06T16:20:31.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:31.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:31.202630+0000) 2024-03-06T16:20:32.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:32.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:32.202864+0000) 2024-03-06T16:20:33.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:33.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:33.203106+0000) 2024-03-06T16:20:34.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:34.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:34.203342+0000) 2024-03-06T16:20:35.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:35.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:35.203593+0000) 2024-03-06T16:20:36.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:36.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:36.203857+0000) 2024-03-06T16:20:36.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:36.361 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:20:37.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:37.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:37.204128+0000) 2024-03-06T16:20:38.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:38.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:38.204322+0000) 2024-03-06T16:20:39.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:39.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:39.204519+0000) 2024-03-06T16:20:40.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:40.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:40.204765+0000) 2024-03-06T16:20:41.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:41.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:41.205068+0000) 2024-03-06T16:20:42.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:42.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:42.205306+0000) 2024-03-06T16:20:43.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:43.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:43.205543+0000) 2024-03-06T16:20:44.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:44.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:44.205777+0000) 2024-03-06T16:20:45.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:45.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:45.206055+0000) 2024-03-06T16:20:46.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:46.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:46.206270+0000) 2024-03-06T16:20:47.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:47.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:47.206492+0000) 2024-03-06T16:20:48.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:48.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:48.206715+0000) 2024-03-06T16:20:49.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:49.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:49.206955+0000) 2024-03-06T16:20:50.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:50.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:50.207160+0000) 2024-03-06T16:20:51.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:51.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:51.207347+0000) 2024-03-06T16:20:52.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:52.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:52.207575+0000) 2024-03-06T16:20:53.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:53.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:53.207796+0000) 2024-03-06T16:20:54.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:54.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:54.208066+0000) 2024-03-06T16:20:55.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:55.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:55.208293+0000) 2024-03-06T16:20:56.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:56.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:56.208532+0000) 2024-03-06T16:20:57.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:57.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:57.208770+0000) 2024-03-06T16:20:58.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:58.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:58.208985+0000) 2024-03-06T16:20:59.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:20:59.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:20:59.209233+0000) 2024-03-06T16:21:00.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:00.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:00.209464+0000) 2024-03-06T16:21:01.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:01.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:01.209678+0000) 2024-03-06T16:21:02.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:02.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:02.209916+0000) 2024-03-06T16:21:03.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:03.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:03.210133+0000) 2024-03-06T16:21:04.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:04.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:04.210331+0000) 2024-03-06T16:21:05.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:05.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:05.210554+0000) 2024-03-06T16:21:06.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:06.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:06.210796+0000) 2024-03-06T16:21:06.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:06.522 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:07.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:07.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:07.211090+0000) 2024-03-06T16:21:08.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:08.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:08.211272+0000) 2024-03-06T16:21:09.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:09.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:09.211505+0000) 2024-03-06T16:21:10.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:10.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:10.211756+0000) 2024-03-06T16:21:11.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:11.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:11.212050+0000) 2024-03-06T16:21:12.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:12.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:12.212303+0000) 2024-03-06T16:21:13.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:13.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:13.212505+0000) 2024-03-06T16:21:14.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:14.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:14.212782+0000) 2024-03-06T16:21:15.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:15.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:15.213113+0000) 2024-03-06T16:21:16.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:16.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:16.213303+0000) 2024-03-06T16:21:17.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:17.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:17.213534+0000) 2024-03-06T16:21:18.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:18.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:18.213780+0000) 2024-03-06T16:21:19.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:19.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:19.214049+0000) 2024-03-06T16:21:20.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:20.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:20.214254+0000) 2024-03-06T16:21:21.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:21.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:21.214474+0000) 2024-03-06T16:21:22.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:22.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:22.214698+0000) 2024-03-06T16:21:23.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:23.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:23.214918+0000) 2024-03-06T16:21:24.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:24.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:24.215147+0000) 2024-03-06T16:21:25.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:25.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:25.215344+0000) 2024-03-06T16:21:26.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:26.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:26.215572+0000) 2024-03-06T16:21:27.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:27.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:27.215794+0000) 2024-03-06T16:21:28.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:28.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:28.216056+0000) 2024-03-06T16:21:29.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:29.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:29.216263+0000) 2024-03-06T16:21:30.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:30.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:30.216495+0000) 2024-03-06T16:21:31.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:31.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:31.216722+0000) 2024-03-06T16:21:32.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:32.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:32.216956+0000) 2024-03-06T16:21:33.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:33.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:33.217177+0000) 2024-03-06T16:21:34.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:34.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:34.217431+0000) 2024-03-06T16:21:35.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:35.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:35.217659+0000) 2024-03-06T16:21:36.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:36.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:36.217896+0000) 2024-03-06T16:21:36.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:36.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:21:37.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:37.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:37.218152+0000) 2024-03-06T16:21:38.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:38.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:38.218333+0000) 2024-03-06T16:21:39.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:39.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:39.218565+0000) 2024-03-06T16:21:40.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:40.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:40.218794+0000) 2024-03-06T16:21:41.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:41.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:41.219046+0000) 2024-03-06T16:21:42.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:42.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:42.219286+0000) 2024-03-06T16:21:43.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:43.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:43.219569+0000) 2024-03-06T16:21:44.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:44.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:44.219822+0000) 2024-03-06T16:21:45.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:45.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:45.220067+0000) 2024-03-06T16:21:46.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:46.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:46.220350+0000) 2024-03-06T16:21:47.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:47.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:47.220630+0000) 2024-03-06T16:21:48.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:48.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:48.220901+0000) 2024-03-06T16:21:49.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:49.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:49.221159+0000) 2024-03-06T16:21:50.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:50.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:50.221362+0000) 2024-03-06T16:21:51.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:51.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:51.221588+0000) 2024-03-06T16:21:52.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:52.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:52.221814+0000) 2024-03-06T16:21:53.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:53.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:53.222040+0000) 2024-03-06T16:21:54.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:54.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:54.222244+0000) 2024-03-06T16:21:55.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:55.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:55.222466+0000) 2024-03-06T16:21:56.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:56.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:56.222696+0000) 2024-03-06T16:21:57.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:57.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:57.222898+0000) 2024-03-06T16:21:58.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:58.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:58.223134+0000) 2024-03-06T16:21:59.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:21:59.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:21:59.223332+0000) 2024-03-06T16:22:00.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:00.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:00.223561+0000) 2024-03-06T16:22:01.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:01.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:01.223785+0000) 2024-03-06T16:22:02.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:02.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:02.224043+0000) 2024-03-06T16:22:03.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:03.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:03.224244+0000) 2024-03-06T16:22:04.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:04.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:04.224472+0000) 2024-03-06T16:22:05.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:05.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:05.224696+0000) 2024-03-06T16:22:06.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:06.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:06.224982+0000) 2024-03-06T16:22:06.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:06.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:07.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:07.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:07.225236+0000) 2024-03-06T16:22:08.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:08.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:08.225457+0000) 2024-03-06T16:22:09.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:09.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:09.225722+0000) 2024-03-06T16:22:10.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:10.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:10.225995+0000) 2024-03-06T16:22:11.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:11.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:11.226343+0000) 2024-03-06T16:22:12.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:12.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:12.226606+0000) 2024-03-06T16:22:13.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:13.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:13.226889+0000) 2024-03-06T16:22:14.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:14.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:14.227142+0000) 2024-03-06T16:22:15.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:15.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:15.227347+0000) 2024-03-06T16:22:16.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:16.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:16.227579+0000) 2024-03-06T16:22:17.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:17.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:17.227796+0000) 2024-03-06T16:22:18.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:18.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:18.228065+0000) 2024-03-06T16:22:19.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:19.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:19.228265+0000) 2024-03-06T16:22:20.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:20.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:20.228489+0000) 2024-03-06T16:22:21.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:21.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:21.228693+0000) 2024-03-06T16:22:22.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:22.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:22.228918+0000) 2024-03-06T16:22:23.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:23.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:23.229144+0000) 2024-03-06T16:22:24.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:24.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:24.229319+0000) 2024-03-06T16:22:25.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:25.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:25.229550+0000) 2024-03-06T16:22:26.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:26.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:26.229809+0000) 2024-03-06T16:22:27.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:27.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:27.230105+0000) 2024-03-06T16:22:28.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:28.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:28.230349+0000) 2024-03-06T16:22:29.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:29.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:29.230617+0000) 2024-03-06T16:22:30.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:30.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:30.230895+0000) 2024-03-06T16:22:31.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:31.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:31.231181+0000) 2024-03-06T16:22:32.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:32.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:32.231413+0000) 2024-03-06T16:22:33.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:33.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:33.231705+0000) 2024-03-06T16:22:34.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:34.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:34.231988+0000) 2024-03-06T16:22:35.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:35.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:35.232326+0000) 2024-03-06T16:22:36.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:36.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:36.232580+0000) 2024-03-06T16:22:37.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:37.016 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:22:37.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:37.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:37.232779+0000) 2024-03-06T16:22:38.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:38.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:38.233041+0000) 2024-03-06T16:22:39.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:39.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:39.233252+0000) 2024-03-06T16:22:40.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:40.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:40.233477+0000) 2024-03-06T16:22:41.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:41.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:41.233699+0000) 2024-03-06T16:22:42.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:42.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:42.233920+0000) 2024-03-06T16:22:43.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:43.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:43.234155+0000) 2024-03-06T16:22:44.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:44.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:44.234354+0000) 2024-03-06T16:22:45.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:45.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:45.234580+0000) 2024-03-06T16:22:46.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:46.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:46.234818+0000) 2024-03-06T16:22:47.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:47.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:47.235111+0000) 2024-03-06T16:22:48.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:48.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:48.235309+0000) 2024-03-06T16:22:49.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:49.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:49.235531+0000) 2024-03-06T16:22:50.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:50.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:50.235757+0000) 2024-03-06T16:22:51.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:51.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:51.235976+0000) 2024-03-06T16:22:52.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:52.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:52.236158+0000) 2024-03-06T16:22:53.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:53.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:53.236422+0000) 2024-03-06T16:22:54.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:54.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:54.236694+0000) 2024-03-06T16:22:55.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:55.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:55.236964+0000) 2024-03-06T16:22:56.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:56.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:56.237290+0000) 2024-03-06T16:22:57.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:57.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:57.237574+0000) 2024-03-06T16:22:58.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:58.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:58.237770+0000) 2024-03-06T16:22:59.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:22:59.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:22:59.238117+0000) 2024-03-06T16:23:00.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:00.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:00.238328+0000) 2024-03-06T16:23:01.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:01.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:01.238555+0000) 2024-03-06T16:23:02.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:02.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:02.238837+0000) 2024-03-06T16:23:03.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:03.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:03.239104+0000) 2024-03-06T16:23:04.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:04.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:04.239329+0000) 2024-03-06T16:23:05.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:05.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:05.239560+0000) 2024-03-06T16:23:06.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:06.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:06.239754+0000) 2024-03-06T16:23:07.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:07.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:07.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:07.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:07.239932+0000) 2024-03-06T16:23:08.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:08.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:08.240161+0000) 2024-03-06T16:23:09.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:09.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:09.240383+0000) 2024-03-06T16:23:10.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:10.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:10.240608+0000) 2024-03-06T16:23:11.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:11.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:11.240813+0000) 2024-03-06T16:23:12.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:12.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:12.241083+0000) 2024-03-06T16:23:13.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:13.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:13.241305+0000) 2024-03-06T16:23:14.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:14.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:14.241522+0000) 2024-03-06T16:23:15.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:15.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:15.241754+0000) 2024-03-06T16:23:16.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:16.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:16.241997+0000) 2024-03-06T16:23:17.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:17.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:17.242270+0000) 2024-03-06T16:23:18.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:18.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:18.242534+0000) 2024-03-06T16:23:19.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:19.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:19.242775+0000) 2024-03-06T16:23:20.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:20.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:20.243041+0000) 2024-03-06T16:23:21.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:21.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:21.243282+0000) 2024-03-06T16:23:22.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:22.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:22.243516+0000) 2024-03-06T16:23:23.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:23.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:23.243781+0000) 2024-03-06T16:23:24.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:24.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:24.243981+0000) 2024-03-06T16:23:25.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:25.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:25.244233+0000) 2024-03-06T16:23:26.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:26.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:26.244493+0000) 2024-03-06T16:23:27.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:27.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:27.244720+0000) 2024-03-06T16:23:28.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:28.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:28.245054+0000) 2024-03-06T16:23:29.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:29.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:29.245341+0000) 2024-03-06T16:23:30.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:30.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:30.245555+0000) 2024-03-06T16:23:31.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:31.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:31.245789+0000) 2024-03-06T16:23:32.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:32.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:32.246093+0000) 2024-03-06T16:23:33.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:33.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:33.246309+0000) 2024-03-06T16:23:34.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:34.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:34.246535+0000) 2024-03-06T16:23:35.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:35.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:35.246734+0000) 2024-03-06T16:23:36.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:36.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:36.246967+0000) 2024-03-06T16:23:37.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:37.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:37.247170+0000) 2024-03-06T16:23:37.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:37.365 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:23:38.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:38.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:38.247391+0000) 2024-03-06T16:23:39.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:39.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:39.247643+0000) 2024-03-06T16:23:40.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:40.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:40.247889+0000) 2024-03-06T16:23:41.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:41.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:41.248146+0000) 2024-03-06T16:23:42.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:42.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:42.248352+0000) 2024-03-06T16:23:43.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:43.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:43.248574+0000) 2024-03-06T16:23:44.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:44.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:44.248766+0000) 2024-03-06T16:23:45.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:45.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:45.249046+0000) 2024-03-06T16:23:46.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:46.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:46.249261+0000) 2024-03-06T16:23:47.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:47.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:47.249536+0000) 2024-03-06T16:23:48.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:48.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:48.249745+0000) 2024-03-06T16:23:49.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:49.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:49.249985+0000) 2024-03-06T16:23:50.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:50.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:50.250272+0000) 2024-03-06T16:23:51.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:51.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:51.250464+0000) 2024-03-06T16:23:52.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:52.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:52.250693+0000) 2024-03-06T16:23:53.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:53.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:53.250884+0000) 2024-03-06T16:23:54.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:54.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:54.251140+0000) 2024-03-06T16:23:55.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:55.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:55.251341+0000) 2024-03-06T16:23:56.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:56.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:56.251613+0000) 2024-03-06T16:23:57.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:57.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:57.251864+0000) 2024-03-06T16:23:58.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:58.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:58.252124+0000) 2024-03-06T16:23:59.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:23:59.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:23:59.252331+0000) 2024-03-06T16:24:00.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:00.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:00.252552+0000) 2024-03-06T16:24:01.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:01.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:01.252783+0000) 2024-03-06T16:24:02.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:02.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:02.253045+0000) 2024-03-06T16:24:03.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:03.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:03.253268+0000) 2024-03-06T16:24:04.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:04.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:04.253490+0000) 2024-03-06T16:24:05.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:05.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:05.253724+0000) 2024-03-06T16:24:06.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:06.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:06.253959+0000) 2024-03-06T16:24:07.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:07.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:07.254161+0000) 2024-03-06T16:24:07.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:07.586 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:08.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:08.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:08.254383+0000) 2024-03-06T16:24:09.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:09.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:09.254603+0000) 2024-03-06T16:24:10.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:10.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:10.254830+0000) 2024-03-06T16:24:11.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:11.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:11.255118+0000) 2024-03-06T16:24:12.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:12.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:12.255339+0000) 2024-03-06T16:24:13.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:13.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:13.255554+0000) 2024-03-06T16:24:14.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:14.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:14.255801+0000) 2024-03-06T16:24:15.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:15.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:15.256066+0000) 2024-03-06T16:24:16.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:16.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:16.256298+0000) 2024-03-06T16:24:17.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:17.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:17.256562+0000) 2024-03-06T16:24:18.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:18.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:18.256774+0000) 2024-03-06T16:24:19.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:19.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:19.256996+0000) 2024-03-06T16:24:20.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:20.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:20.257254+0000) 2024-03-06T16:24:21.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:21.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:21.257448+0000) 2024-03-06T16:24:22.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:22.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:22.257664+0000) 2024-03-06T16:24:23.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:23.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:23.257877+0000) 2024-03-06T16:24:24.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:24.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:24.258148+0000) 2024-03-06T16:24:25.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:25.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:25.258365+0000) 2024-03-06T16:24:26.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:26.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:26.258594+0000) 2024-03-06T16:24:27.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:27.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:27.258806+0000) 2024-03-06T16:24:28.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:28.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:28.259124+0000) 2024-03-06T16:24:29.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:29.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:29.259405+0000) 2024-03-06T16:24:30.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:30.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:30.259696+0000) 2024-03-06T16:24:31.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:31.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:31.259934+0000) 2024-03-06T16:24:32.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:32.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:32.260154+0000) 2024-03-06T16:24:33.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:33.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:33.260341+0000) 2024-03-06T16:24:34.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:34.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:34.260561+0000) 2024-03-06T16:24:35.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:35.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:35.260874+0000) 2024-03-06T16:24:36.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:36.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:36.261118+0000) 2024-03-06T16:24:37.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:37.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:37.261314+0000) 2024-03-06T16:24:37.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:37.733 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:24:38.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:38.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:38.261563+0000) 2024-03-06T16:24:39.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:39.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:39.261806+0000) 2024-03-06T16:24:40.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:40.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:40.262105+0000) 2024-03-06T16:24:41.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:41.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:41.262359+0000) 2024-03-06T16:24:42.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:42.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:42.262592+0000) 2024-03-06T16:24:43.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:43.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:43.262823+0000) 2024-03-06T16:24:44.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:44.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:44.263112+0000) 2024-03-06T16:24:45.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:45.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:45.263366+0000) 2024-03-06T16:24:46.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:46.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:46.263618+0000) 2024-03-06T16:24:47.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:47.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:47.263859+0000) 2024-03-06T16:24:48.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:48.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:48.264078+0000) 2024-03-06T16:24:49.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:49.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:49.264254+0000) 2024-03-06T16:24:50.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:50.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:50.264451+0000) 2024-03-06T16:24:51.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:51.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:51.264676+0000) 2024-03-06T16:24:52.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:52.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:52.264914+0000) 2024-03-06T16:24:53.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:53.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:53.265164+0000) 2024-03-06T16:24:54.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:54.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:54.265353+0000) 2024-03-06T16:24:55.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:55.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:55.265577+0000) 2024-03-06T16:24:56.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:56.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:56.265818+0000) 2024-03-06T16:24:57.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:57.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:57.266126+0000) 2024-03-06T16:24:58.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:58.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:58.266306+0000) 2024-03-06T16:24:59.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:24:59.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:24:59.266532+0000) 2024-03-06T16:25:00.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:00.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:00.266765+0000) 2024-03-06T16:25:01.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:01.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:01.266991+0000) 2024-03-06T16:25:02.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:02.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:02.267262+0000) 2024-03-06T16:25:03.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:03.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:03.267484+0000) 2024-03-06T16:25:04.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:04.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:04.267706+0000) 2024-03-06T16:25:05.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:05.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:05.267945+0000) 2024-03-06T16:25:06.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:06.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:06.268185+0000) 2024-03-06T16:25:07.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:07.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:07.268416+0000) 2024-03-06T16:25:07.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:07.950 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:08.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:08.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:08.268665+0000) 2024-03-06T16:25:09.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:09.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:09.268893+0000) 2024-03-06T16:25:10.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:10.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:10.269089+0000) 2024-03-06T16:25:11.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:11.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:11.269303+0000) 2024-03-06T16:25:12.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:12.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:12.269532+0000) 2024-03-06T16:25:13.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:13.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:13.269752+0000) 2024-03-06T16:25:14.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:14.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:14.269971+0000) 2024-03-06T16:25:15.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:15.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:15.270206+0000) 2024-03-06T16:25:16.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:16.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:16.270447+0000) 2024-03-06T16:25:17.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:17.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:17.270665+0000) 2024-03-06T16:25:18.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:18.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:18.270894+0000) 2024-03-06T16:25:19.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:19.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:19.271119+0000) 2024-03-06T16:25:20.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:20.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:20.271364+0000) 2024-03-06T16:25:21.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:21.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:21.271590+0000) 2024-03-06T16:25:22.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:22.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:22.271813+0000) 2024-03-06T16:25:23.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:23.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:23.272068+0000) 2024-03-06T16:25:24.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:24.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:24.272289+0000) 2024-03-06T16:25:25.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:25.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:25.272525+0000) 2024-03-06T16:25:26.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:26.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:26.272769+0000) 2024-03-06T16:25:27.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:27.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:27.273050+0000) 2024-03-06T16:25:28.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:28.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:28.273335+0000) 2024-03-06T16:25:29.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:29.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:29.273533+0000) 2024-03-06T16:25:30.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:30.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:30.273770+0000) 2024-03-06T16:25:31.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:31.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:31.273999+0000) 2024-03-06T16:25:32.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:32.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:32.274295+0000) 2024-03-06T16:25:33.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:33.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:33.274482+0000) 2024-03-06T16:25:34.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:34.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:34.274728+0000) 2024-03-06T16:25:35.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:35.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:35.274978+0000) 2024-03-06T16:25:36.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:36.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:36.275245+0000) 2024-03-06T16:25:37.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:37.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:37.275489+0000) 2024-03-06T16:25:38.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:38.148 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:25:38.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:38.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:38.275712+0000) 2024-03-06T16:25:39.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:39.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:39.275931+0000) 2024-03-06T16:25:40.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:40.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:40.276174+0000) 2024-03-06T16:25:41.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:41.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:41.276381+0000) 2024-03-06T16:25:42.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:42.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:42.276586+0000) 2024-03-06T16:25:43.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:43.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:43.276823+0000) 2024-03-06T16:25:44.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:44.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:44.277109+0000) 2024-03-06T16:25:45.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:45.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:45.277359+0000) 2024-03-06T16:25:46.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:46.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:46.277602+0000) 2024-03-06T16:25:47.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:47.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:47.277786+0000) 2024-03-06T16:25:48.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:48.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:48.278079+0000) 2024-03-06T16:25:49.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:49.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:49.278335+0000) 2024-03-06T16:25:50.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:50.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:50.278558+0000) 2024-03-06T16:25:51.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:51.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:51.278781+0000) 2024-03-06T16:25:52.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:52.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:52.278987+0000) 2024-03-06T16:25:53.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:53.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:53.279226+0000) 2024-03-06T16:25:54.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:54.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:54.279449+0000) 2024-03-06T16:25:55.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:55.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:55.279704+0000) 2024-03-06T16:25:56.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:56.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:56.279942+0000) 2024-03-06T16:25:57.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:57.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:57.280168+0000) 2024-03-06T16:25:58.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:58.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:58.280376+0000) 2024-03-06T16:25:59.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:25:59.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:25:59.280619+0000) 2024-03-06T16:26:00.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:00.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:00.280874+0000) 2024-03-06T16:26:01.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:01.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:01.281134+0000) 2024-03-06T16:26:02.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:02.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:02.281341+0000) 2024-03-06T16:26:03.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:03.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:03.281519+0000) 2024-03-06T16:26:04.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:04.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:04.281753+0000) 2024-03-06T16:26:05.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:05.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:05.281998+0000) 2024-03-06T16:26:06.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:06.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:06.282285+0000) 2024-03-06T16:26:07.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:07.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:07.282505+0000) 2024-03-06T16:26:08.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:08.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:08.282716+0000) 2024-03-06T16:26:08.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:08.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:09.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:09.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:09.282932+0000) 2024-03-06T16:26:10.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:10.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:10.283183+0000) 2024-03-06T16:26:11.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:11.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:11.283346+0000) 2024-03-06T16:26:12.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:12.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:12.283585+0000) 2024-03-06T16:26:13.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:13.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:13.283815+0000) 2024-03-06T16:26:14.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:14.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:14.284110+0000) 2024-03-06T16:26:15.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:15.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:15.284334+0000) 2024-03-06T16:26:16.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:16.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:16.284829+0000) 2024-03-06T16:26:17.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:17.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:17.285120+0000) 2024-03-06T16:26:18.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:18.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:18.285350+0000) 2024-03-06T16:26:19.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:19.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:19.285557+0000) 2024-03-06T16:26:20.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:20.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:20.285786+0000) 2024-03-06T16:26:21.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:21.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:21.285984+0000) 2024-03-06T16:26:22.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:22.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:22.286262+0000) 2024-03-06T16:26:23.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:23.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:23.286464+0000) 2024-03-06T16:26:24.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:24.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:24.286704+0000) 2024-03-06T16:26:25.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:25.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:25.286956+0000) 2024-03-06T16:26:26.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:26.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:26.287228+0000) 2024-03-06T16:26:27.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:27.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:27.287434+0000) 2024-03-06T16:26:28.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:28.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:28.287663+0000) 2024-03-06T16:26:29.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:29.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:29.287906+0000) 2024-03-06T16:26:30.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:30.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:30.288137+0000) 2024-03-06T16:26:31.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:31.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:31.288364+0000) 2024-03-06T16:26:32.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:32.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:32.288596+0000) 2024-03-06T16:26:33.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:33.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:33.288770+0000) 2024-03-06T16:26:34.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:34.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:34.289040+0000) 2024-03-06T16:26:35.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:35.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:35.289271+0000) 2024-03-06T16:26:36.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:36.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:36.289522+0000) 2024-03-06T16:26:37.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:37.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:37.289765+0000) 2024-03-06T16:26:38.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:38.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:38.290049+0000) 2024-03-06T16:26:38.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:38.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:26:39.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:39.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:39.290280+0000) 2024-03-06T16:26:40.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:40.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:40.290513+0000) 2024-03-06T16:26:41.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:41.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:41.290740+0000) 2024-03-06T16:26:42.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:42.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:42.290969+0000) 2024-03-06T16:26:43.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:43.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:43.291213+0000) 2024-03-06T16:26:44.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:44.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:44.291457+0000) 2024-03-06T16:26:45.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:45.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:45.291705+0000) 2024-03-06T16:26:46.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:46.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:46.291964+0000) 2024-03-06T16:26:47.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:47.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:47.292205+0000) 2024-03-06T16:26:48.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:48.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:48.292443+0000) 2024-03-06T16:26:49.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:49.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:49.292651+0000) 2024-03-06T16:26:50.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:50.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:50.292881+0000) 2024-03-06T16:26:51.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:51.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:51.293124+0000) 2024-03-06T16:26:52.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:52.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:52.293331+0000) 2024-03-06T16:26:53.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:53.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:53.293564+0000) 2024-03-06T16:26:54.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:54.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:54.293802+0000) 2024-03-06T16:26:55.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:55.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:55.294080+0000) 2024-03-06T16:26:56.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:56.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:56.294328+0000) 2024-03-06T16:26:57.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:57.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:57.294522+0000) 2024-03-06T16:26:58.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:58.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:58.294743+0000) 2024-03-06T16:26:59.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:26:59.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:26:59.294991+0000) 2024-03-06T16:27:00.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:00.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:00.295287+0000) 2024-03-06T16:27:01.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:01.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:01.295541+0000) 2024-03-06T16:27:02.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:02.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:02.295736+0000) 2024-03-06T16:27:03.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:03.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:03.295960+0000) 2024-03-06T16:27:04.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:04.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:04.296265+0000) 2024-03-06T16:27:05.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:05.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:05.296432+0000) 2024-03-06T16:27:06.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:06.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:06.296652+0000) 2024-03-06T16:27:07.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:07.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:07.296840+0000) 2024-03-06T16:27:08.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:08.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:08.297123+0000) 2024-03-06T16:27:08.709 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:08.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:09.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:09.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:09.297341+0000) 2024-03-06T16:27:10.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:10.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:10.297575+0000) 2024-03-06T16:27:11.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:11.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:11.297806+0000) 2024-03-06T16:27:12.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:12.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:12.298094+0000) 2024-03-06T16:27:13.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:13.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:13.298283+0000) 2024-03-06T16:27:14.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:14.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:14.298497+0000) 2024-03-06T16:27:15.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:15.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:15.298723+0000) 2024-03-06T16:27:16.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:16.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:16.298961+0000) 2024-03-06T16:27:17.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:17.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:17.299173+0000) 2024-03-06T16:27:18.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:18.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:18.299381+0000) 2024-03-06T16:27:19.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:19.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:19.299609+0000) 2024-03-06T16:27:20.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:20.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:20.299835+0000) 2024-03-06T16:27:21.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:21.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:21.300117+0000) 2024-03-06T16:27:22.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:22.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:22.300326+0000) 2024-03-06T16:27:23.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:23.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:23.300524+0000) 2024-03-06T16:27:24.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:24.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:24.300745+0000) 2024-03-06T16:27:25.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:25.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:25.300933+0000) 2024-03-06T16:27:26.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:26.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:26.301170+0000) 2024-03-06T16:27:27.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:27.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:27.301394+0000) 2024-03-06T16:27:28.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:28.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:28.301630+0000) 2024-03-06T16:27:29.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:29.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:29.301876+0000) 2024-03-06T16:27:30.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:30.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:30.302128+0000) 2024-03-06T16:27:31.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:31.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:31.302375+0000) 2024-03-06T16:27:32.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:32.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:32.302629+0000) 2024-03-06T16:27:33.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:33.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:33.302849+0000) 2024-03-06T16:27:34.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:34.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:34.303113+0000) 2024-03-06T16:27:35.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:35.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:35.303302+0000) 2024-03-06T16:27:36.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:36.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:36.303530+0000) 2024-03-06T16:27:37.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:37.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:37.303754+0000) 2024-03-06T16:27:38.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:38.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:38.303973+0000) 2024-03-06T16:27:38.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:38.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:27:39.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:39.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:39.304264+0000) 2024-03-06T16:27:40.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:40.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:40.304493+0000) 2024-03-06T16:27:41.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:41.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:41.304675+0000) 2024-03-06T16:27:42.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:42.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:42.304921+0000) 2024-03-06T16:27:43.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:43.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:43.305157+0000) 2024-03-06T16:27:44.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:44.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:44.305385+0000) 2024-03-06T16:27:45.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:45.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:45.305622+0000) 2024-03-06T16:27:46.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:46.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:46.305875+0000) 2024-03-06T16:27:47.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:47.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:47.306124+0000) 2024-03-06T16:27:48.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:48.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:48.306333+0000) 2024-03-06T16:27:49.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:49.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:49.306573+0000) 2024-03-06T16:27:50.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:50.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:50.306812+0000) 2024-03-06T16:27:51.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:51.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:51.307053+0000) 2024-03-06T16:27:52.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:52.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:52.307282+0000) 2024-03-06T16:27:53.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:53.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:53.307500+0000) 2024-03-06T16:27:54.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:54.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:54.307745+0000) 2024-03-06T16:27:55.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:55.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:55.308043+0000) 2024-03-06T16:27:56.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:56.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:56.308293+0000) 2024-03-06T16:27:57.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:57.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:57.308490+0000) 2024-03-06T16:27:58.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:58.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:58.308781+0000) 2024-03-06T16:27:59.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:27:59.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:27:59.308980+0000) 2024-03-06T16:28:00.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:00.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:00.309253+0000) 2024-03-06T16:28:01.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:01.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:01.309494+0000) 2024-03-06T16:28:02.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:02.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:02.309725+0000) 2024-03-06T16:28:03.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:03.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:03.309958+0000) 2024-03-06T16:28:04.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:04.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:04.310164+0000) 2024-03-06T16:28:05.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:05.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:05.310367+0000) 2024-03-06T16:28:06.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:06.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:06.310626+0000) 2024-03-06T16:28:07.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:07.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:07.310880+0000) 2024-03-06T16:28:08.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:08.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:08.311129+0000) 2024-03-06T16:28:08.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:09.005 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:09.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:09.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:09.311352+0000) 2024-03-06T16:28:10.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:10.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:10.311605+0000) 2024-03-06T16:28:11.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:11.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:11.311859+0000) 2024-03-06T16:28:12.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:12.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:12.312143+0000) 2024-03-06T16:28:13.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:13.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:13.312349+0000) 2024-03-06T16:28:14.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:14.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:14.312545+0000) 2024-03-06T16:28:15.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:15.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:15.312766+0000) 2024-03-06T16:28:16.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:16.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:16.312960+0000) 2024-03-06T16:28:17.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:17.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:17.313254+0000) 2024-03-06T16:28:18.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:18.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:18.313489+0000) 2024-03-06T16:28:19.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:19.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:19.313691+0000) 2024-03-06T16:28:20.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:20.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:20.313915+0000) 2024-03-06T16:28:21.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:21.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:21.314153+0000) 2024-03-06T16:28:22.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:22.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:22.314372+0000) 2024-03-06T16:28:23.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:23.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:23.314566+0000) 2024-03-06T16:28:24.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:24.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:24.314741+0000) 2024-03-06T16:28:25.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:25.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:25.314938+0000) 2024-03-06T16:28:26.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:26.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:26.315196+0000) 2024-03-06T16:28:27.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:27.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:27.315418+0000) 2024-03-06T16:28:28.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:28.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:28.315644+0000) 2024-03-06T16:28:29.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:29.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:29.315869+0000) 2024-03-06T16:28:30.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:30.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:30.316130+0000) 2024-03-06T16:28:31.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:31.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:31.316325+0000) 2024-03-06T16:28:32.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:32.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:32.316551+0000) 2024-03-06T16:28:33.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:33.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:33.316750+0000) 2024-03-06T16:28:34.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:34.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:34.317035+0000) 2024-03-06T16:28:35.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:35.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:35.317286+0000) 2024-03-06T16:28:36.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:36.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:36.317523+0000) 2024-03-06T16:28:37.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:37.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:37.317749+0000) 2024-03-06T16:28:38.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:38.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:38.317968+0000) 2024-03-06T16:28:39.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:39.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:28:39.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:39.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:39.318263+0000) 2024-03-06T16:28:40.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:40.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:40.318470+0000) 2024-03-06T16:28:41.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:41.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:41.318711+0000) 2024-03-06T16:28:42.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:42.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:42.318954+0000) 2024-03-06T16:28:43.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:43.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:43.319177+0000) 2024-03-06T16:28:44.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:44.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:44.319379+0000) 2024-03-06T16:28:45.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:45.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:45.319615+0000) 2024-03-06T16:28:46.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:46.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:46.319943+0000) 2024-03-06T16:28:47.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:47.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:47.320195+0000) 2024-03-06T16:28:48.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:48.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:48.320416+0000) 2024-03-06T16:28:49.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:49.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:49.320642+0000) 2024-03-06T16:28:50.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:50.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:50.320863+0000) 2024-03-06T16:28:51.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:51.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:51.321130+0000) 2024-03-06T16:28:52.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:52.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:52.321367+0000) 2024-03-06T16:28:53.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:53.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:53.321570+0000) 2024-03-06T16:28:54.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:54.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:54.321796+0000) 2024-03-06T16:28:55.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:55.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:55.322049+0000) 2024-03-06T16:28:56.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:56.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:56.322303+0000) 2024-03-06T16:28:57.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:57.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:57.322533+0000) 2024-03-06T16:28:58.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:58.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:58.322721+0000) 2024-03-06T16:28:59.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:28:59.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:28:59.322950+0000) 2024-03-06T16:29:00.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:00.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:00.323176+0000) 2024-03-06T16:29:01.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:01.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:01.323405+0000) 2024-03-06T16:29:02.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:02.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:02.323634+0000) 2024-03-06T16:29:03.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:03.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:03.323859+0000) 2024-03-06T16:29:04.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:04.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:04.324136+0000) 2024-03-06T16:29:05.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:05.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:05.324378+0000) 2024-03-06T16:29:06.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:06.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:06.324575+0000) 2024-03-06T16:29:07.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:07.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:07.324825+0000) 2024-03-06T16:29:08.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:08.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:08.325130+0000) 2024-03-06T16:29:09.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:09.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:09.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:09.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:09.325347+0000) 2024-03-06T16:29:10.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:10.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:10.325558+0000) 2024-03-06T16:29:11.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:11.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:11.325791+0000) 2024-03-06T16:29:12.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:12.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:12.326042+0000) 2024-03-06T16:29:13.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:13.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:13.326262+0000) 2024-03-06T16:29:14.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:14.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:14.326464+0000) 2024-03-06T16:29:15.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:15.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:15.326686+0000) 2024-03-06T16:29:16.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:16.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:16.326940+0000) 2024-03-06T16:29:17.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:17.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:17.327161+0000) 2024-03-06T16:29:18.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:18.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:18.327377+0000) 2024-03-06T16:29:19.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:19.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:19.327631+0000) 2024-03-06T16:29:20.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:20.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:20.327870+0000) 2024-03-06T16:29:21.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:21.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:21.328129+0000) 2024-03-06T16:29:22.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:22.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:22.328358+0000) 2024-03-06T16:29:23.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:23.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:23.328582+0000) 2024-03-06T16:29:24.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:24.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:24.328810+0000) 2024-03-06T16:29:25.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:25.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:25.329073+0000) 2024-03-06T16:29:26.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:26.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:26.329312+0000) 2024-03-06T16:29:27.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:27.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:27.329532+0000) 2024-03-06T16:29:28.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:28.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:28.329717+0000) 2024-03-06T16:29:29.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:29.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:29.329950+0000) 2024-03-06T16:29:30.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:30.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:30.330178+0000) 2024-03-06T16:29:31.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:31.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:31.330343+0000) 2024-03-06T16:29:32.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:32.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:32.330570+0000) 2024-03-06T16:29:33.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:33.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:33.330787+0000) 2024-03-06T16:29:34.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:34.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:34.331054+0000) 2024-03-06T16:29:35.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:35.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:35.331252+0000) 2024-03-06T16:29:36.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:36.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:36.331460+0000) 2024-03-06T16:29:37.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:37.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:37.331655+0000) 2024-03-06T16:29:38.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:38.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:38.331876+0000) 2024-03-06T16:29:39.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:39.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:39.332106+0000) 2024-03-06T16:29:39.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:39.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:29:40.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:40.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:40.332306+0000) 2024-03-06T16:29:41.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:41.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:41.332548+0000) 2024-03-06T16:29:42.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:42.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:42.332791+0000) 2024-03-06T16:29:43.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:43.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:43.333044+0000) 2024-03-06T16:29:44.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:44.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:44.333236+0000) 2024-03-06T16:29:45.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:45.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:45.333462+0000) 2024-03-06T16:29:46.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:46.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:46.333661+0000) 2024-03-06T16:29:47.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:47.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:47.333884+0000) 2024-03-06T16:29:48.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:48.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:48.334149+0000) 2024-03-06T16:29:49.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:49.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:49.334368+0000) 2024-03-06T16:29:50.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:50.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:50.334592+0000) 2024-03-06T16:29:51.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:51.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:51.334796+0000) 2024-03-06T16:29:52.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:52.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:52.335054+0000) 2024-03-06T16:29:53.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:53.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:53.335276+0000) 2024-03-06T16:29:54.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:54.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:54.335500+0000) 2024-03-06T16:29:55.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:55.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:55.335713+0000) 2024-03-06T16:29:56.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:56.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:56.335963+0000) 2024-03-06T16:29:57.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:57.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:57.336200+0000) 2024-03-06T16:29:58.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:58.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:58.336411+0000) 2024-03-06T16:29:59.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:29:59.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:29:59.336637+0000) 2024-03-06T16:30:00.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:00.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:00.336848+0000) 2024-03-06T16:30:01.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:01.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:01.337115+0000) 2024-03-06T16:30:02.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:02.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:02.337329+0000) 2024-03-06T16:30:03.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:03.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:03.337524+0000) 2024-03-06T16:30:04.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:04.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:04.337760+0000) 2024-03-06T16:30:05.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:05.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:05.337986+0000) 2024-03-06T16:30:06.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:06.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:06.338223+0000) 2024-03-06T16:30:07.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:07.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:07.338448+0000) 2024-03-06T16:30:08.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:08.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:08.338699+0000) 2024-03-06T16:30:09.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:09.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:09.338898+0000) 2024-03-06T16:30:09.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:09.561 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:10.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:10.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:10.339135+0000) 2024-03-06T16:30:11.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:11.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:11.339396+0000) 2024-03-06T16:30:12.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:12.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:12.339641+0000) 2024-03-06T16:30:13.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:13.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:13.339862+0000) 2024-03-06T16:30:14.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:14.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:14.340081+0000) 2024-03-06T16:30:15.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:15.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:15.340318+0000) 2024-03-06T16:30:16.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:16.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:16.340579+0000) 2024-03-06T16:30:17.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:17.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:17.340825+0000) 2024-03-06T16:30:18.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:18.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:18.340990+0000) 2024-03-06T16:30:19.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:19.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:19.341264+0000) 2024-03-06T16:30:20.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:20.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:20.341488+0000) 2024-03-06T16:30:21.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:21.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:21.341717+0000) 2024-03-06T16:30:22.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:22.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:22.341941+0000) 2024-03-06T16:30:23.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:23.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:23.342164+0000) 2024-03-06T16:30:24.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:24.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:24.342366+0000) 2024-03-06T16:30:25.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:25.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:25.342542+0000) 2024-03-06T16:30:26.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:26.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:26.342785+0000) 2024-03-06T16:30:27.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:27.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:27.343044+0000) 2024-03-06T16:30:28.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:28.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:28.343269+0000) 2024-03-06T16:30:29.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:29.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:29.343494+0000) 2024-03-06T16:30:30.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:30.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:30.343708+0000) 2024-03-06T16:30:31.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:31.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:31.343941+0000) 2024-03-06T16:30:32.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:32.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:32.344250+0000) 2024-03-06T16:30:33.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:33.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:33.344454+0000) 2024-03-06T16:30:34.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:34.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:34.344681+0000) 2024-03-06T16:30:35.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:35.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:35.344905+0000) 2024-03-06T16:30:36.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:36.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:36.345146+0000) 2024-03-06T16:30:37.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:37.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:37.345347+0000) 2024-03-06T16:30:38.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:38.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:38.345542+0000) 2024-03-06T16:30:39.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:39.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:39.345762+0000) 2024-03-06T16:30:39.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:39.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:30:40.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:40.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:40.345993+0000) 2024-03-06T16:30:41.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:41.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:41.346276+0000) 2024-03-06T16:30:42.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:42.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:42.346499+0000) 2024-03-06T16:30:43.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:43.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:43.346721+0000) 2024-03-06T16:30:44.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:44.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:44.346929+0000) 2024-03-06T16:30:45.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:45.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:45.347153+0000) 2024-03-06T16:30:46.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:46.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:46.347401+0000) 2024-03-06T16:30:47.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:47.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:47.347599+0000) 2024-03-06T16:30:48.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:48.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:48.347817+0000) 2024-03-06T16:30:49.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:49.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:49.348077+0000) 2024-03-06T16:30:50.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:50.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:50.348252+0000) 2024-03-06T16:30:51.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:51.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:51.348489+0000) 2024-03-06T16:30:52.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:52.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:52.348712+0000) 2024-03-06T16:30:53.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:53.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:53.348937+0000) 2024-03-06T16:30:54.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:54.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:54.349162+0000) 2024-03-06T16:30:55.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:55.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:55.349359+0000) 2024-03-06T16:30:56.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:56.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:56.349611+0000) 2024-03-06T16:30:57.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:57.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:57.349853+0000) 2024-03-06T16:30:58.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:58.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:58.350124+0000) 2024-03-06T16:30:59.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:30:59.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:30:59.350329+0000) 2024-03-06T16:31:00.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:00.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:00.350551+0000) 2024-03-06T16:31:01.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:01.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:01.350781+0000) 2024-03-06T16:31:02.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:02.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:02.351048+0000) 2024-03-06T16:31:03.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:03.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:03.351257+0000) 2024-03-06T16:31:04.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:04.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:04.351430+0000) 2024-03-06T16:31:05.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:05.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:05.351654+0000) 2024-03-06T16:31:06.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:06.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:06.351899+0000) 2024-03-06T16:31:07.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:07.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:07.352121+0000) 2024-03-06T16:31:08.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:08.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:08.352319+0000) 2024-03-06T16:31:09.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:09.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:09.352550+0000) 2024-03-06T16:31:09.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:09.938 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:10.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:10.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:10.352799+0000) 2024-03-06T16:31:11.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:11.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:11.353080+0000) 2024-03-06T16:31:12.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:12.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:12.353302+0000) 2024-03-06T16:31:13.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:13.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:13.353523+0000) 2024-03-06T16:31:14.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:14.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:14.353743+0000) 2024-03-06T16:31:15.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:15.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:15.353979+0000) 2024-03-06T16:31:16.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:16.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:16.354218+0000) 2024-03-06T16:31:17.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:17.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:17.354413+0000) 2024-03-06T16:31:18.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:18.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:18.354653+0000) 2024-03-06T16:31:19.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:19.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:19.354934+0000) 2024-03-06T16:31:20.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:20.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:20.355194+0000) 2024-03-06T16:31:21.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:21.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:21.355483+0000) 2024-03-06T16:31:22.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:22.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:22.355761+0000) 2024-03-06T16:31:23.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:23.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:23.355995+0000) 2024-03-06T16:31:24.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:24.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:24.356229+0000) 2024-03-06T16:31:25.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:25.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:25.356417+0000) 2024-03-06T16:31:26.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:26.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:26.356672+0000) 2024-03-06T16:31:27.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:27.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:27.356898+0000) 2024-03-06T16:31:28.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:28.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:28.357088+0000) 2024-03-06T16:31:29.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:29.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:29.357310+0000) 2024-03-06T16:31:30.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:30.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:30.357521+0000) 2024-03-06T16:31:31.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:31.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:31.357724+0000) 2024-03-06T16:31:32.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:32.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:32.357907+0000) 2024-03-06T16:31:33.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:33.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:33.358136+0000) 2024-03-06T16:31:34.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:34.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:34.358354+0000) 2024-03-06T16:31:35.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:35.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:35.358647+0000) 2024-03-06T16:31:36.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:36.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:36.358900+0000) 2024-03-06T16:31:37.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:37.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:37.359169+0000) 2024-03-06T16:31:38.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:38.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:38.359367+0000) 2024-03-06T16:31:39.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:39.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:39.359591+0000) 2024-03-06T16:31:40.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:40.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:31:40.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:40.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:40.359857+0000) 2024-03-06T16:31:41.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:41.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:41.360184+0000) 2024-03-06T16:31:42.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:42.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:42.360452+0000) 2024-03-06T16:31:43.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:43.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:43.360689+0000) 2024-03-06T16:31:44.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:44.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:44.360954+0000) 2024-03-06T16:31:45.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:45.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:45.361209+0000) 2024-03-06T16:31:46.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:46.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:46.361537+0000) 2024-03-06T16:31:47.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:47.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:47.361793+0000) 2024-03-06T16:31:48.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:48.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:48.362106+0000) 2024-03-06T16:31:49.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:49.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:49.362384+0000) 2024-03-06T16:31:50.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:50.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:50.362670+0000) 2024-03-06T16:31:51.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:51.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:51.362970+0000) 2024-03-06T16:31:52.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:52.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:52.363264+0000) 2024-03-06T16:31:53.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:53.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:53.363503+0000) 2024-03-06T16:31:54.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:54.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:54.363722+0000) 2024-03-06T16:31:55.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:55.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:55.363948+0000) 2024-03-06T16:31:56.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:56.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:56.364191+0000) 2024-03-06T16:31:57.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:57.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:57.364422+0000) 2024-03-06T16:31:58.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:58.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:58.364653+0000) 2024-03-06T16:31:59.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:31:59.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:31:59.364883+0000) 2024-03-06T16:32:00.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:00.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:00.365128+0000) 2024-03-06T16:32:01.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:01.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:01.365391+0000) 2024-03-06T16:32:02.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:02.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:02.365625+0000) 2024-03-06T16:32:03.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:03.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:03.365943+0000) 2024-03-06T16:32:04.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:04.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:04.366231+0000) 2024-03-06T16:32:05.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:05.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:05.366512+0000) 2024-03-06T16:32:06.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:06.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:06.366810+0000) 2024-03-06T16:32:07.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:07.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:07.367132+0000) 2024-03-06T16:32:08.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:08.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:08.367312+0000) 2024-03-06T16:32:09.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:09.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:09.367523+0000) 2024-03-06T16:32:10.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:10.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:10.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:10.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:10.367765+0000) 2024-03-06T16:32:11.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:11.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:11.368056+0000) 2024-03-06T16:32:12.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:12.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:12.368281+0000) 2024-03-06T16:32:13.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:13.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:13.368505+0000) 2024-03-06T16:32:14.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:14.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:14.368715+0000) 2024-03-06T16:32:15.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:15.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:15.368978+0000) 2024-03-06T16:32:16.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:16.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:16.369214+0000) 2024-03-06T16:32:17.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:17.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:17.369377+0000) 2024-03-06T16:32:18.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:18.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:18.369588+0000) 2024-03-06T16:32:19.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:19.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:19.369818+0000) 2024-03-06T16:32:20.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:20.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:20.370134+0000) 2024-03-06T16:32:21.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:21.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:21.370413+0000) 2024-03-06T16:32:22.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:22.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:22.370668+0000) 2024-03-06T16:32:23.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:23.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:23.370933+0000) 2024-03-06T16:32:24.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:24.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:24.371158+0000) 2024-03-06T16:32:25.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:25.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:25.371315+0000) 2024-03-06T16:32:26.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:26.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:26.371547+0000) 2024-03-06T16:32:27.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:27.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:27.371782+0000) 2024-03-06T16:32:28.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:28.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:28.371953+0000) 2024-03-06T16:32:29.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:29.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:29.372140+0000) 2024-03-06T16:32:30.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:30.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:30.372363+0000) 2024-03-06T16:32:31.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:31.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:31.372594+0000) 2024-03-06T16:32:32.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:32.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:32.372746+0000) 2024-03-06T16:32:33.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:33.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:33.372972+0000) 2024-03-06T16:32:34.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:34.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:34.373230+0000) 2024-03-06T16:32:35.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:35.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:35.373501+0000) 2024-03-06T16:32:36.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:36.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:36.373783+0000) 2024-03-06T16:32:37.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:37.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:37.374118+0000) 2024-03-06T16:32:38.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:38.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:38.374341+0000) 2024-03-06T16:32:39.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:39.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:39.374559+0000) 2024-03-06T16:32:40.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:40.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:40.374776+0000) 2024-03-06T16:32:40.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:40.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:32:41.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:41.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:41.375041+0000) 2024-03-06T16:32:42.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:42.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:42.375271+0000) 2024-03-06T16:32:43.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:43.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:43.375531+0000) 2024-03-06T16:32:44.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:44.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:44.375808+0000) 2024-03-06T16:32:45.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:45.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:45.376128+0000) 2024-03-06T16:32:46.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:46.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:46.376319+0000) 2024-03-06T16:32:47.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:47.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:47.376554+0000) 2024-03-06T16:32:48.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:48.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:48.376777+0000) 2024-03-06T16:32:49.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:49.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:49.377044+0000) 2024-03-06T16:32:50.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:50.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:50.377247+0000) 2024-03-06T16:32:51.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:51.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:51.377409+0000) 2024-03-06T16:32:52.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:52.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:52.377628+0000) 2024-03-06T16:32:53.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:53.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:53.377844+0000) 2024-03-06T16:32:54.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:54.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:54.378118+0000) 2024-03-06T16:32:55.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:55.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:55.378382+0000) 2024-03-06T16:32:56.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:56.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:56.378710+0000) 2024-03-06T16:32:57.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:57.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:57.378977+0000) 2024-03-06T16:32:58.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:58.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:58.379327+0000) 2024-03-06T16:32:59.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:32:59.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:32:59.379532+0000) 2024-03-06T16:33:00.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:00.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:00.379751+0000) 2024-03-06T16:33:01.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:01.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:01.379965+0000) 2024-03-06T16:33:02.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:02.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:02.380284+0000) 2024-03-06T16:33:03.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:03.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:03.380557+0000) 2024-03-06T16:33:04.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:04.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:04.380821+0000) 2024-03-06T16:33:05.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:05.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:05.381153+0000) 2024-03-06T16:33:06.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:06.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:06.381415+0000) 2024-03-06T16:33:07.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:07.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:07.381669+0000) 2024-03-06T16:33:08.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:08.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:08.381915+0000) 2024-03-06T16:33:09.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:09.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:09.382101+0000) 2024-03-06T16:33:10.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:10.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:10.382340+0000) 2024-03-06T16:33:10.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:10.675 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:11.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:11.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:11.382623+0000) 2024-03-06T16:33:12.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:12.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:12.382827+0000) 2024-03-06T16:33:13.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:13.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:13.383001+0000) 2024-03-06T16:33:14.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:14.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:14.383263+0000) 2024-03-06T16:33:15.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:15.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:15.383501+0000) 2024-03-06T16:33:16.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:16.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:16.383803+0000) 2024-03-06T16:33:17.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:17.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:17.384130+0000) 2024-03-06T16:33:18.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:18.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:18.384356+0000) 2024-03-06T16:33:19.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:19.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:19.384624+0000) 2024-03-06T16:33:20.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:20.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:20.384828+0000) 2024-03-06T16:33:21.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:21.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:21.385058+0000) 2024-03-06T16:33:22.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:22.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:22.385261+0000) 2024-03-06T16:33:23.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:23.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:23.385431+0000) 2024-03-06T16:33:24.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:24.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:24.385650+0000) 2024-03-06T16:33:25.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:25.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:25.385798+0000) 2024-03-06T16:33:26.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:26.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:26.385957+0000) 2024-03-06T16:33:27.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:27.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:27.386167+0000) 2024-03-06T16:33:28.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:28.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:28.386352+0000) 2024-03-06T16:33:29.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:29.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:29.386561+0000) 2024-03-06T16:33:30.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:30.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:30.386801+0000) 2024-03-06T16:33:31.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:31.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:31.387096+0000) 2024-03-06T16:33:32.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:32.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:32.387319+0000) 2024-03-06T16:33:33.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:33.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:33.387518+0000) 2024-03-06T16:33:34.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:34.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:34.387731+0000) 2024-03-06T16:33:35.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:35.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:35.387955+0000) 2024-03-06T16:33:36.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:36.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:36.388191+0000) 2024-03-06T16:33:37.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:37.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:37.388406+0000) 2024-03-06T16:33:38.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:38.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:38.388585+0000) 2024-03-06T16:33:39.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:39.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:39.388854+0000) 2024-03-06T16:33:40.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:40.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:40.389131+0000) 2024-03-06T16:33:40.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:40.827 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:33:41.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:41.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:41.389379+0000) 2024-03-06T16:33:42.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:42.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:42.389609+0000) 2024-03-06T16:33:43.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:43.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:43.389772+0000) 2024-03-06T16:33:44.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:44.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:44.389973+0000) 2024-03-06T16:33:45.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:45.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:45.390207+0000) 2024-03-06T16:33:46.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:46.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:46.390404+0000) 2024-03-06T16:33:47.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:47.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:47.390668+0000) 2024-03-06T16:33:48.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:48.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:48.390933+0000) 2024-03-06T16:33:49.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:49.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:49.391207+0000) 2024-03-06T16:33:50.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:50.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:50.391531+0000) 2024-03-06T16:33:51.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:51.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:51.391774+0000) 2024-03-06T16:33:52.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:52.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:52.391995+0000) 2024-03-06T16:33:53.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:53.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:53.392292+0000) 2024-03-06T16:33:54.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:54.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:54.392562+0000) 2024-03-06T16:33:55.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:55.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:55.392832+0000) 2024-03-06T16:33:56.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:56.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:56.393160+0000) 2024-03-06T16:33:57.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:57.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:57.393442+0000) 2024-03-06T16:33:58.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:58.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:58.393709+0000) 2024-03-06T16:33:59.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:33:59.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:33:59.393987+0000) 2024-03-06T16:34:00.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:00.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:00.394243+0000) 2024-03-06T16:34:01.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:01.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:01.394473+0000) 2024-03-06T16:34:02.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:02.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:02.394661+0000) 2024-03-06T16:34:03.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:03.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:03.394890+0000) 2024-03-06T16:34:04.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:04.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:04.395124+0000) 2024-03-06T16:34:05.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:05.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:05.395323+0000) 2024-03-06T16:34:06.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:06.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:06.395528+0000) 2024-03-06T16:34:07.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:07.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:07.395754+0000) 2024-03-06T16:34:08.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:08.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:08.395979+0000) 2024-03-06T16:34:09.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:09.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:09.396212+0000) 2024-03-06T16:34:10.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:10.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:10.396395+0000) 2024-03-06T16:34:11.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:11.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:11.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:11.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:11.396627+0000) 2024-03-06T16:34:12.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:12.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:12.396852+0000) 2024-03-06T16:34:13.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:13.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:13.397120+0000) 2024-03-06T16:34:14.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:14.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:14.397337+0000) 2024-03-06T16:34:15.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:15.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:15.397567+0000) 2024-03-06T16:34:16.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:16.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:16.397832+0000) 2024-03-06T16:34:17.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:17.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:17.398129+0000) 2024-03-06T16:34:18.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:18.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:18.398259+0000) 2024-03-06T16:34:19.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:19.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:19.398469+0000) 2024-03-06T16:34:20.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:20.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:20.398594+0000) 2024-03-06T16:34:21.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:21.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:21.398812+0000) 2024-03-06T16:34:22.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:22.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:22.399123+0000) 2024-03-06T16:34:23.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:23.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:23.399389+0000) 2024-03-06T16:34:24.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:24.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:24.399652+0000) 2024-03-06T16:34:25.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:25.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:25.399933+0000) 2024-03-06T16:34:26.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:26.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:26.400222+0000) 2024-03-06T16:34:27.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:27.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:27.400499+0000) 2024-03-06T16:34:28.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:28.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:28.400778+0000) 2024-03-06T16:34:29.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:29.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:29.401054+0000) 2024-03-06T16:34:30.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:30.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:30.401305+0000) 2024-03-06T16:34:31.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:31.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:31.401529+0000) 2024-03-06T16:34:32.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:32.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:32.401769+0000) 2024-03-06T16:34:33.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:33.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:33.402034+0000) 2024-03-06T16:34:34.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:34.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:34.402251+0000) 2024-03-06T16:34:35.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:35.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:35.402487+0000) 2024-03-06T16:34:36.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:36.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:36.402716+0000) 2024-03-06T16:34:37.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:37.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:37.402992+0000) 2024-03-06T16:34:38.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:38.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:38.403306+0000) 2024-03-06T16:34:39.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:39.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:39.403570+0000) 2024-03-06T16:34:40.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:40.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:40.403825+0000) 2024-03-06T16:34:41.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:41.243 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:34:41.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:41.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:41.404116+0000) 2024-03-06T16:34:42.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:42.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:42.404359+0000) 2024-03-06T16:34:43.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:43.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:43.404678+0000) 2024-03-06T16:34:44.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:44.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:44.404863+0000) 2024-03-06T16:34:45.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:45.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:45.405045+0000) 2024-03-06T16:34:46.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:46.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:46.405277+0000) 2024-03-06T16:34:47.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:47.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:47.405500+0000) 2024-03-06T16:34:48.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:48.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:48.405723+0000) 2024-03-06T16:34:49.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:49.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:49.405965+0000) 2024-03-06T16:34:50.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:50.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:50.406152+0000) 2024-03-06T16:34:51.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:51.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:51.406355+0000) 2024-03-06T16:34:52.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:52.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:52.406589+0000) 2024-03-06T16:34:53.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:53.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:53.406804+0000) 2024-03-06T16:34:54.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:54.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:54.407087+0000) 2024-03-06T16:34:55.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:55.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:55.407288+0000) 2024-03-06T16:34:56.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:56.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:56.407515+0000) 2024-03-06T16:34:57.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:57.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:57.407743+0000) 2024-03-06T16:34:58.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:58.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:58.407958+0000) 2024-03-06T16:34:59.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:34:59.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:34:59.408168+0000) 2024-03-06T16:35:00.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:00.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:00.408393+0000) 2024-03-06T16:35:01.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:01.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:01.408623+0000) 2024-03-06T16:35:02.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:02.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:02.408855+0000) 2024-03-06T16:35:03.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:03.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:03.409113+0000) 2024-03-06T16:35:04.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:04.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:04.409306+0000) 2024-03-06T16:35:05.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:05.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:05.409604+0000) 2024-03-06T16:35:06.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:06.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:06.409860+0000) 2024-03-06T16:35:07.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:07.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:07.410164+0000) 2024-03-06T16:35:08.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:08.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:08.410357+0000) 2024-03-06T16:35:09.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:09.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:09.410553+0000) 2024-03-06T16:35:10.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:10.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:10.410786+0000) 2024-03-06T16:35:11.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:11.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:11.411040+0000) 2024-03-06T16:35:11.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:11.454 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:12.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:12.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:12.411302+0000) 2024-03-06T16:35:13.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:13.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:13.411489+0000) 2024-03-06T16:35:14.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:14.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:14.411706+0000) 2024-03-06T16:35:15.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:15.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:15.411908+0000) 2024-03-06T16:35:16.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:16.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:16.412125+0000) 2024-03-06T16:35:17.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:17.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:17.412355+0000) 2024-03-06T16:35:18.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:18.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:18.412574+0000) 2024-03-06T16:35:19.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:19.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:19.412803+0000) 2024-03-06T16:35:20.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:20.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:20.413046+0000) 2024-03-06T16:35:21.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:21.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:21.413270+0000) 2024-03-06T16:35:22.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:22.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:22.413533+0000) 2024-03-06T16:35:23.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:23.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:23.413780+0000) 2024-03-06T16:35:24.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:24.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:24.414047+0000) 2024-03-06T16:35:25.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:25.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:25.414261+0000) 2024-03-06T16:35:26.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:26.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:26.414516+0000) 2024-03-06T16:35:27.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:27.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:27.414749+0000) 2024-03-06T16:35:28.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:28.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:28.414973+0000) 2024-03-06T16:35:29.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:29.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:29.415199+0000) 2024-03-06T16:35:30.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:30.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:30.415408+0000) 2024-03-06T16:35:31.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:31.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:31.415638+0000) 2024-03-06T16:35:32.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:32.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:32.415863+0000) 2024-03-06T16:35:33.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:33.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:33.416134+0000) 2024-03-06T16:35:34.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:34.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:34.416368+0000) 2024-03-06T16:35:35.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:35.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:35.416613+0000) 2024-03-06T16:35:36.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:36.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:36.416843+0000) 2024-03-06T16:35:37.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:37.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:37.417112+0000) 2024-03-06T16:35:38.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:38.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:38.417327+0000) 2024-03-06T16:35:39.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:39.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:39.417550+0000) 2024-03-06T16:35:40.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:40.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:40.417776+0000) 2024-03-06T16:35:41.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:41.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:41.417988+0000) 2024-03-06T16:35:41.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:41.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:35:42.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:42.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:42.418254+0000) 2024-03-06T16:35:43.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:43.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:43.418499+0000) 2024-03-06T16:35:44.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:44.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:44.418729+0000) 2024-03-06T16:35:45.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:45.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:45.418926+0000) 2024-03-06T16:35:46.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:46.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:46.419128+0000) 2024-03-06T16:35:47.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:47.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:47.419355+0000) 2024-03-06T16:35:48.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:48.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:48.419602+0000) 2024-03-06T16:35:49.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:49.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:49.419865+0000) 2024-03-06T16:35:50.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:50.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:50.420099+0000) 2024-03-06T16:35:51.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:51.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:51.420328+0000) 2024-03-06T16:35:52.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:52.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:52.420516+0000) 2024-03-06T16:35:53.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:53.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:53.420734+0000) 2024-03-06T16:35:54.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:54.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:54.420944+0000) 2024-03-06T16:35:55.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:55.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:55.421179+0000) 2024-03-06T16:35:56.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:56.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:56.421371+0000) 2024-03-06T16:35:57.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:57.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:57.421598+0000) 2024-03-06T16:35:58.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:58.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:58.421814+0000) 2024-03-06T16:35:59.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:35:59.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:35:59.422079+0000) 2024-03-06T16:36:00.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:00.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:00.422293+0000) 2024-03-06T16:36:01.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:01.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:01.422481+0000) 2024-03-06T16:36:02.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:02.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:02.422702+0000) 2024-03-06T16:36:03.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:03.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:03.422921+0000) 2024-03-06T16:36:04.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:04.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:04.423143+0000) 2024-03-06T16:36:05.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:05.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:05.423318+0000) 2024-03-06T16:36:06.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:06.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:06.423541+0000) 2024-03-06T16:36:07.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:07.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:07.423777+0000) 2024-03-06T16:36:08.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:08.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:08.424096+0000) 2024-03-06T16:36:09.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:09.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:09.424367+0000) 2024-03-06T16:36:10.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:10.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:10.424645+0000) 2024-03-06T16:36:11.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:11.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:11.424929+0000) 2024-03-06T16:36:11.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:11.783 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:12.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:12.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:12.425225+0000) 2024-03-06T16:36:13.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:13.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:13.425460+0000) 2024-03-06T16:36:14.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:14.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:14.425754+0000) 2024-03-06T16:36:15.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:15.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:15.426078+0000) 2024-03-06T16:36:16.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:16.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:16.426360+0000) 2024-03-06T16:36:17.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:17.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:17.426605+0000) 2024-03-06T16:36:18.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:18.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:18.426882+0000) 2024-03-06T16:36:19.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:19.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:19.427145+0000) 2024-03-06T16:36:20.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:20.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:20.427392+0000) 2024-03-06T16:36:21.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:21.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:21.427628+0000) 2024-03-06T16:36:22.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:22.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:22.427861+0000) 2024-03-06T16:36:23.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:23.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:23.428110+0000) 2024-03-06T16:36:24.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:24.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:24.428271+0000) 2024-03-06T16:36:25.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:25.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:25.428501+0000) 2024-03-06T16:36:26.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:26.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:26.428713+0000) 2024-03-06T16:36:27.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:27.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:27.428954+0000) 2024-03-06T16:36:28.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:28.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:28.429151+0000) 2024-03-06T16:36:29.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:29.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:29.429384+0000) 2024-03-06T16:36:30.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:30.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:30.429612+0000) 2024-03-06T16:36:31.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:31.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:31.429838+0000) 2024-03-06T16:36:32.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:32.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:32.430090+0000) 2024-03-06T16:36:33.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:33.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:33.430309+0000) 2024-03-06T16:36:34.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:34.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:34.430514+0000) 2024-03-06T16:36:35.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:35.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:35.430731+0000) 2024-03-06T16:36:36.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:36.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:36.430980+0000) 2024-03-06T16:36:37.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:37.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:37.431271+0000) 2024-03-06T16:36:38.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:38.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:38.431492+0000) 2024-03-06T16:36:39.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:39.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:39.431672+0000) 2024-03-06T16:36:40.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:40.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:40.431867+0000) 2024-03-06T16:36:41.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:41.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:41.432126+0000) 2024-03-06T16:36:41.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:41.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:36:42.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:42.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:42.432355+0000) 2024-03-06T16:36:43.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:43.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:43.432556+0000) 2024-03-06T16:36:44.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:44.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:44.432737+0000) 2024-03-06T16:36:45.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:45.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:45.432960+0000) 2024-03-06T16:36:46.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:46.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:46.433221+0000) 2024-03-06T16:36:47.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:47.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:47.433507+0000) 2024-03-06T16:36:48.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:48.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:48.433772+0000) 2024-03-06T16:36:49.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:49.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:49.434086+0000) 2024-03-06T16:36:50.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:50.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:50.434342+0000) 2024-03-06T16:36:51.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:51.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:51.434613+0000) 2024-03-06T16:36:52.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:52.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:52.434901+0000) 2024-03-06T16:36:53.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:53.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:53.435185+0000) 2024-03-06T16:36:54.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:54.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:54.435467+0000) 2024-03-06T16:36:55.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:55.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:55.435708+0000) 2024-03-06T16:36:56.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:56.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:56.435938+0000) 2024-03-06T16:36:57.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:57.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:57.436171+0000) 2024-03-06T16:36:58.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:58.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:58.436397+0000) 2024-03-06T16:36:59.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:36:59.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:36:59.436629+0000) 2024-03-06T16:37:00.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:00.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:00.436848+0000) 2024-03-06T16:37:01.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:01.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:01.437065+0000) 2024-03-06T16:37:02.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:02.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:02.437306+0000) 2024-03-06T16:37:03.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:03.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:03.437484+0000) 2024-03-06T16:37:04.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:04.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:04.437714+0000) 2024-03-06T16:37:05.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:05.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:05.437862+0000) 2024-03-06T16:37:06.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:06.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:06.438135+0000) 2024-03-06T16:37:07.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:07.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:07.438353+0000) 2024-03-06T16:37:08.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:08.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:08.438556+0000) 2024-03-06T16:37:09.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:09.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:09.438743+0000) 2024-03-06T16:37:10.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:10.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:10.438966+0000) 2024-03-06T16:37:11.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:11.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:11.439159+0000) 2024-03-06T16:37:12.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:12.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:12.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:12.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:12.439410+0000) 2024-03-06T16:37:13.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:13.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:13.439613+0000) 2024-03-06T16:37:14.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:14.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:14.439828+0000) 2024-03-06T16:37:15.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:15.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:15.440080+0000) 2024-03-06T16:37:16.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:16.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:16.440334+0000) 2024-03-06T16:37:17.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:17.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:17.440572+0000) 2024-03-06T16:37:18.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:18.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:18.440783+0000) 2024-03-06T16:37:19.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:19.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:19.441000+0000) 2024-03-06T16:37:20.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:20.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:20.441254+0000) 2024-03-06T16:37:21.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:21.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:21.441433+0000) 2024-03-06T16:37:22.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:22.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:22.441654+0000) 2024-03-06T16:37:23.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:23.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:23.441900+0000) 2024-03-06T16:37:24.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:24.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:24.442175+0000) 2024-03-06T16:37:25.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:25.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:25.442423+0000) 2024-03-06T16:37:26.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:26.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:26.442630+0000) 2024-03-06T16:37:27.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:27.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:27.442873+0000) 2024-03-06T16:37:28.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:28.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:28.443145+0000) 2024-03-06T16:37:29.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:29.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:29.443384+0000) 2024-03-06T16:37:30.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:30.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:30.443624+0000) 2024-03-06T16:37:31.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:31.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:31.443842+0000) 2024-03-06T16:37:32.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:32.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:32.444136+0000) 2024-03-06T16:37:33.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:33.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:33.444380+0000) 2024-03-06T16:37:34.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:34.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:34.444649+0000) 2024-03-06T16:37:35.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:35.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:35.444840+0000) 2024-03-06T16:37:36.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:36.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:36.445114+0000) 2024-03-06T16:37:37.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:37.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:37.445367+0000) 2024-03-06T16:37:38.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:38.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:38.445586+0000) 2024-03-06T16:37:39.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:39.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:39.445819+0000) 2024-03-06T16:37:40.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:40.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:40.446110+0000) 2024-03-06T16:37:41.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:41.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:41.446357+0000) 2024-03-06T16:37:42.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:42.377 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:37:42.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:42.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:42.446560+0000) 2024-03-06T16:37:43.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:43.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:43.446772+0000) 2024-03-06T16:37:44.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:44.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:44.447000+0000) 2024-03-06T16:37:45.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:45.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:45.447218+0000) 2024-03-06T16:37:46.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:46.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:46.447413+0000) 2024-03-06T16:37:47.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:47.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:47.447635+0000) 2024-03-06T16:37:48.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:48.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:48.447857+0000) 2024-03-06T16:37:49.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:49.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:49.448110+0000) 2024-03-06T16:37:50.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:50.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:50.448365+0000) 2024-03-06T16:37:51.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:51.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:51.448645+0000) 2024-03-06T16:37:52.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:52.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:52.448931+0000) 2024-03-06T16:37:53.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:53.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:53.449197+0000) 2024-03-06T16:37:54.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:54.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:54.449477+0000) 2024-03-06T16:37:55.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:55.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:55.449749+0000) 2024-03-06T16:37:56.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:56.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:56.450068+0000) 2024-03-06T16:37:57.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:57.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:57.450342+0000) 2024-03-06T16:37:58.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:58.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:58.450556+0000) 2024-03-06T16:37:59.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:37:59.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:37:59.450751+0000) 2024-03-06T16:38:00.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:00.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:00.451059+0000) 2024-03-06T16:38:01.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:01.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:01.451249+0000) 2024-03-06T16:38:02.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:02.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:02.451488+0000) 2024-03-06T16:38:03.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:03.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:03.451738+0000) 2024-03-06T16:38:04.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:04.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:04.451961+0000) 2024-03-06T16:38:05.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:05.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:05.452189+0000) 2024-03-06T16:38:06.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:06.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:06.452427+0000) 2024-03-06T16:38:07.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:07.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:07.452622+0000) 2024-03-06T16:38:08.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:08.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:08.452798+0000) 2024-03-06T16:38:09.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:09.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:09.453046+0000) 2024-03-06T16:38:10.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:10.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:10.453259+0000) 2024-03-06T16:38:11.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:11.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:11.453515+0000) 2024-03-06T16:38:12.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:12.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:12.453764+0000) 2024-03-06T16:38:12.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:12.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:13.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:13.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:13.453947+0000) 2024-03-06T16:38:14.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:14.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:14.454152+0000) 2024-03-06T16:38:15.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:15.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:15.454348+0000) 2024-03-06T16:38:16.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:16.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:16.454567+0000) 2024-03-06T16:38:17.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:17.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:17.454790+0000) 2024-03-06T16:38:18.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:18.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:18.454996+0000) 2024-03-06T16:38:19.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:19.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:19.455244+0000) 2024-03-06T16:38:20.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:20.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:20.455486+0000) 2024-03-06T16:38:21.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:21.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:21.455683+0000) 2024-03-06T16:38:22.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:22.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:22.455912+0000) 2024-03-06T16:38:23.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:23.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:23.456144+0000) 2024-03-06T16:38:24.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:24.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:24.456367+0000) 2024-03-06T16:38:25.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:25.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:25.456561+0000) 2024-03-06T16:38:26.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:26.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:26.456791+0000) 2024-03-06T16:38:27.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:27.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:27.456988+0000) 2024-03-06T16:38:28.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:28.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:28.457253+0000) 2024-03-06T16:38:29.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:29.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:29.457467+0000) 2024-03-06T16:38:30.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:30.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:30.457693+0000) 2024-03-06T16:38:31.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:31.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:31.457933+0000) 2024-03-06T16:38:32.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:32.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:32.458155+0000) 2024-03-06T16:38:33.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:33.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:33.458341+0000) 2024-03-06T16:38:34.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:34.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:34.458553+0000) 2024-03-06T16:38:35.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:35.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:35.458779+0000) 2024-03-06T16:38:36.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:36.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:36.459063+0000) 2024-03-06T16:38:37.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:37.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:37.459269+0000) 2024-03-06T16:38:38.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:38.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:38.459486+0000) 2024-03-06T16:38:39.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:39.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:39.459678+0000) 2024-03-06T16:38:40.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:40.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:40.459903+0000) 2024-03-06T16:38:41.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:41.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:41.460137+0000) 2024-03-06T16:38:42.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:42.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:42.460338+0000) 2024-03-06T16:38:42.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:42.776 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:38:43.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:43.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:43.460574+0000) 2024-03-06T16:38:44.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:44.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:44.460795+0000) 2024-03-06T16:38:45.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:45.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:45.461048+0000) 2024-03-06T16:38:46.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:46.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:46.461258+0000) 2024-03-06T16:38:47.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:47.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:47.461443+0000) 2024-03-06T16:38:48.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:48.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:48.461701+0000) 2024-03-06T16:38:49.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:49.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:49.461920+0000) 2024-03-06T16:38:50.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:50.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:50.462153+0000) 2024-03-06T16:38:51.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:51.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:51.462340+0000) 2024-03-06T16:38:52.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:52.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:52.462547+0000) 2024-03-06T16:38:53.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:53.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:53.462769+0000) 2024-03-06T16:38:54.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:54.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:54.462995+0000) 2024-03-06T16:38:55.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:55.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:55.463264+0000) 2024-03-06T16:38:56.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:56.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:56.463458+0000) 2024-03-06T16:38:57.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:57.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:57.463657+0000) 2024-03-06T16:38:58.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:58.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:58.463878+0000) 2024-03-06T16:38:59.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:38:59.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:38:59.464123+0000) 2024-03-06T16:39:00.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:00.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:00.464310+0000) 2024-03-06T16:39:01.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:01.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:01.464515+0000) 2024-03-06T16:39:02.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:02.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:02.464693+0000) 2024-03-06T16:39:03.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:03.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:03.464913+0000) 2024-03-06T16:39:04.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:04.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:04.465125+0000) 2024-03-06T16:39:05.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:05.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:05.465337+0000) 2024-03-06T16:39:06.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:06.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:06.465568+0000) 2024-03-06T16:39:07.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:07.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:07.465836+0000) 2024-03-06T16:39:08.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:08.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:08.466071+0000) 2024-03-06T16:39:09.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:09.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:09.466310+0000) 2024-03-06T16:39:10.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:10.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:10.466570+0000) 2024-03-06T16:39:11.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:11.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:11.466785+0000) 2024-03-06T16:39:12.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:12.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:12.467060+0000) 2024-03-06T16:39:12.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:12.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:13.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:13.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:13.467270+0000) 2024-03-06T16:39:14.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:14.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:14.467538+0000) 2024-03-06T16:39:15.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:15.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:15.467810+0000) 2024-03-06T16:39:16.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:16.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:16.468083+0000) 2024-03-06T16:39:17.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:17.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:17.468323+0000) 2024-03-06T16:39:18.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:18.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:18.468547+0000) 2024-03-06T16:39:19.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:19.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:19.468767+0000) 2024-03-06T16:39:20.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:20.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:20.469042+0000) 2024-03-06T16:39:21.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:21.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:21.469323+0000) 2024-03-06T16:39:22.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:22.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:22.469491+0000) 2024-03-06T16:39:23.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:23.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:23.469643+0000) 2024-03-06T16:39:24.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:24.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:24.469852+0000) 2024-03-06T16:39:25.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:25.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:25.470093+0000) 2024-03-06T16:39:26.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:26.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:26.470336+0000) 2024-03-06T16:39:27.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:27.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:27.470557+0000) 2024-03-06T16:39:28.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:28.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:28.470817+0000) 2024-03-06T16:39:29.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:29.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:29.471128+0000) 2024-03-06T16:39:30.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:30.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:30.471393+0000) 2024-03-06T16:39:31.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:31.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:31.471716+0000) 2024-03-06T16:39:32.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:32.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:32.471986+0000) 2024-03-06T16:39:33.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:33.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:33.472296+0000) 2024-03-06T16:39:34.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:34.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:34.472573+0000) 2024-03-06T16:39:35.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:35.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:35.472858+0000) 2024-03-06T16:39:36.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:36.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:36.473149+0000) 2024-03-06T16:39:37.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:37.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:37.473317+0000) 2024-03-06T16:39:38.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:38.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:38.473537+0000) 2024-03-06T16:39:39.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:39.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:39.473793+0000) 2024-03-06T16:39:40.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:40.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:40.474039+0000) 2024-03-06T16:39:41.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:41.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:41.474292+0000) 2024-03-06T16:39:42.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:42.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:42.474535+0000) 2024-03-06T16:39:43.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:43.125 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:39:43.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:43.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:43.474778+0000) 2024-03-06T16:39:44.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:44.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:44.474984+0000) 2024-03-06T16:39:45.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:45.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:45.475204+0000) 2024-03-06T16:39:46.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:46.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:46.475401+0000) 2024-03-06T16:39:47.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:47.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:47.475607+0000) 2024-03-06T16:39:48.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:48.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:48.475832+0000) 2024-03-06T16:39:49.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:49.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:49.476120+0000) 2024-03-06T16:39:50.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:50.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:50.476343+0000) 2024-03-06T16:39:51.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:51.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:51.476544+0000) 2024-03-06T16:39:52.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:52.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:52.476746+0000) 2024-03-06T16:39:53.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:53.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:53.476974+0000) 2024-03-06T16:39:54.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:54.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:54.477154+0000) 2024-03-06T16:39:55.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:55.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:55.477353+0000) 2024-03-06T16:39:56.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:56.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:56.477534+0000) 2024-03-06T16:39:57.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:57.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:57.477700+0000) 2024-03-06T16:39:58.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:58.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:58.477925+0000) 2024-03-06T16:39:59.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:39:59.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:39:59.478143+0000) 2024-03-06T16:40:00.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:00.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:00.478329+0000) 2024-03-06T16:40:01.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:01.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:01.478533+0000) 2024-03-06T16:40:02.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:02.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:02.478751+0000) 2024-03-06T16:40:03.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:03.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:03.478973+0000) 2024-03-06T16:40:04.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:04.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:04.479163+0000) 2024-03-06T16:40:05.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:05.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:05.479360+0000) 2024-03-06T16:40:06.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:06.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:06.479577+0000) 2024-03-06T16:40:07.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:07.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:07.479794+0000) 2024-03-06T16:40:08.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:08.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:08.480039+0000) 2024-03-06T16:40:09.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:09.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:09.480229+0000) 2024-03-06T16:40:10.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:10.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:10.480414+0000) 2024-03-06T16:40:11.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:11.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:11.480617+0000) 2024-03-06T16:40:12.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:12.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:12.480834+0000) 2024-03-06T16:40:13.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:13.256 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:13.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:13.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:13.481097+0000) 2024-03-06T16:40:14.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:14.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:14.481285+0000) 2024-03-06T16:40:15.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:15.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:15.481494+0000) 2024-03-06T16:40:16.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:16.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:16.481719+0000) 2024-03-06T16:40:17.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:17.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:17.481940+0000) 2024-03-06T16:40:18.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:18.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:18.482162+0000) 2024-03-06T16:40:19.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:19.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:19.482352+0000) 2024-03-06T16:40:20.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:20.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:20.482555+0000) 2024-03-06T16:40:21.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:21.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:21.482780+0000) 2024-03-06T16:40:22.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:22.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:22.482995+0000) 2024-03-06T16:40:23.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:23.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:23.483259+0000) 2024-03-06T16:40:24.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:24.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:24.483515+0000) 2024-03-06T16:40:25.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:25.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:25.483733+0000) 2024-03-06T16:40:26.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:26.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:26.483956+0000) 2024-03-06T16:40:27.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:27.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:27.484130+0000) 2024-03-06T16:40:28.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:28.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:28.484368+0000) 2024-03-06T16:40:29.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:29.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:29.484602+0000) 2024-03-06T16:40:30.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:30.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:30.484794+0000) 2024-03-06T16:40:31.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:31.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:31.485047+0000) 2024-03-06T16:40:32.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:32.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:32.485228+0000) 2024-03-06T16:40:33.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:33.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:33.485459+0000) 2024-03-06T16:40:34.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:34.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:34.485676+0000) 2024-03-06T16:40:35.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:35.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:35.485913+0000) 2024-03-06T16:40:36.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:36.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:36.486175+0000) 2024-03-06T16:40:37.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:37.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:37.486367+0000) 2024-03-06T16:40:38.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:38.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:38.486553+0000) 2024-03-06T16:40:39.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:39.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:39.486785+0000) 2024-03-06T16:40:40.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:40.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:40.487070+0000) 2024-03-06T16:40:41.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:41.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:41.487263+0000) 2024-03-06T16:40:42.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:42.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:42.487465+0000) 2024-03-06T16:40:43.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:43.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:40:43.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:43.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:43.487662+0000) 2024-03-06T16:40:44.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:44.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:44.487882+0000) 2024-03-06T16:40:45.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:45.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:45.488159+0000) 2024-03-06T16:40:46.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:46.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:46.488422+0000) 2024-03-06T16:40:47.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:47.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:47.488666+0000) 2024-03-06T16:40:48.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:48.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:48.488910+0000) 2024-03-06T16:40:49.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:49.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:49.489139+0000) 2024-03-06T16:40:50.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:50.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:50.489387+0000) 2024-03-06T16:40:51.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:51.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:51.489626+0000) 2024-03-06T16:40:52.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:52.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:52.489879+0000) 2024-03-06T16:40:53.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:53.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:53.490159+0000) 2024-03-06T16:40:54.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:54.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:54.490335+0000) 2024-03-06T16:40:55.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:55.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:55.490541+0000) 2024-03-06T16:40:56.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:56.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:56.490771+0000) 2024-03-06T16:40:57.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:57.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:57.491001+0000) 2024-03-06T16:40:58.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:58.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:58.491225+0000) 2024-03-06T16:40:59.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:40:59.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:40:59.491414+0000) 2024-03-06T16:41:00.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:00.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:00.491603+0000) 2024-03-06T16:41:01.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:01.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:01.491842+0000) 2024-03-06T16:41:02.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:02.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:02.492106+0000) 2024-03-06T16:41:03.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:03.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:03.492333+0000) 2024-03-06T16:41:04.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:04.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:04.492556+0000) 2024-03-06T16:41:05.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:05.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:05.492786+0000) 2024-03-06T16:41:06.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:06.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:06.493034+0000) 2024-03-06T16:41:07.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:07.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:07.493270+0000) 2024-03-06T16:41:08.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:08.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:08.493491+0000) 2024-03-06T16:41:09.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:09.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:09.493718+0000) 2024-03-06T16:41:10.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:10.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:10.493936+0000) 2024-03-06T16:41:11.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:11.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:11.494153+0000) 2024-03-06T16:41:12.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:12.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:12.494342+0000) 2024-03-06T16:41:13.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:13.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:13.494549+0000) 2024-03-06T16:41:13.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:13.613 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:14.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:14.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:14.494723+0000) 2024-03-06T16:41:15.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:15.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:15.494950+0000) 2024-03-06T16:41:16.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:16.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:16.495187+0000) 2024-03-06T16:41:17.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:17.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:17.495403+0000) 2024-03-06T16:41:18.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:18.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:18.495623+0000) 2024-03-06T16:41:19.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:19.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:19.495845+0000) 2024-03-06T16:41:20.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:20.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:20.496095+0000) 2024-03-06T16:41:21.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:21.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:21.496277+0000) 2024-03-06T16:41:22.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:22.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:22.496506+0000) 2024-03-06T16:41:23.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:23.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:23.496760+0000) 2024-03-06T16:41:24.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:24.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:24.496994+0000) 2024-03-06T16:41:25.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:25.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:25.497268+0000) 2024-03-06T16:41:26.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:26.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:26.497511+0000) 2024-03-06T16:41:27.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:27.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:27.497745+0000) 2024-03-06T16:41:28.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:28.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:28.497984+0000) 2024-03-06T16:41:29.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:29.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:29.498281+0000) 2024-03-06T16:41:30.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:30.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:30.498463+0000) 2024-03-06T16:41:31.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:31.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:31.498699+0000) 2024-03-06T16:41:32.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:32.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:32.498926+0000) 2024-03-06T16:41:33.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:33.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:33.499172+0000) 2024-03-06T16:41:34.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:34.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:34.499374+0000) 2024-03-06T16:41:35.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:35.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:35.499584+0000) 2024-03-06T16:41:36.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:36.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:36.499826+0000) 2024-03-06T16:41:37.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:37.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:37.500091+0000) 2024-03-06T16:41:38.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:38.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:38.500295+0000) 2024-03-06T16:41:39.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:39.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:39.500516+0000) 2024-03-06T16:41:40.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:40.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:40.500690+0000) 2024-03-06T16:41:41.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:41.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:41.500886+0000) 2024-03-06T16:41:42.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:42.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:42.501119+0000) 2024-03-06T16:41:43.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:43.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:43.501317+0000) 2024-03-06T16:41:43.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:43.817 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:41:44.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:44.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:44.501499+0000) 2024-03-06T16:41:45.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:45.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:45.501750+0000) 2024-03-06T16:41:46.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:46.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:46.501995+0000) 2024-03-06T16:41:47.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:47.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:47.502259+0000) 2024-03-06T16:41:48.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:48.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:48.502493+0000) 2024-03-06T16:41:49.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:49.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:49.502685+0000) 2024-03-06T16:41:50.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:50.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:50.502885+0000) 2024-03-06T16:41:51.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:51.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:51.503120+0000) 2024-03-06T16:41:52.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:52.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:52.503302+0000) 2024-03-06T16:41:53.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:53.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:53.503519+0000) 2024-03-06T16:41:54.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:54.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:54.503766+0000) 2024-03-06T16:41:55.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:55.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:55.503995+0000) 2024-03-06T16:41:56.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:56.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:56.504247+0000) 2024-03-06T16:41:57.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:57.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:57.504453+0000) 2024-03-06T16:41:58.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:58.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:58.504654+0000) 2024-03-06T16:41:59.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:41:59.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:41:59.504874+0000) 2024-03-06T16:42:00.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:00.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:00.505119+0000) 2024-03-06T16:42:01.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:01.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:01.505366+0000) 2024-03-06T16:42:02.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:02.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:02.505567+0000) 2024-03-06T16:42:03.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:03.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:03.505795+0000) 2024-03-06T16:42:04.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:04.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:04.506043+0000) 2024-03-06T16:42:05.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:05.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:05.506269+0000) 2024-03-06T16:42:06.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:06.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:06.506497+0000) 2024-03-06T16:42:07.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:07.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:07.506719+0000) 2024-03-06T16:42:08.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:08.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:08.506937+0000) 2024-03-06T16:42:09.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:09.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:09.507151+0000) 2024-03-06T16:42:10.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:10.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:10.507358+0000) 2024-03-06T16:42:11.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:11.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:11.507637+0000) 2024-03-06T16:42:12.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:12.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:12.507920+0000) 2024-03-06T16:42:13.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:13.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:13.508176+0000) 2024-03-06T16:42:13.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:13.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:14.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:14.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:14.508386+0000) 2024-03-06T16:42:15.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:15.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:15.508597+0000) 2024-03-06T16:42:16.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:16.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:16.508882+0000) 2024-03-06T16:42:17.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:17.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:17.509177+0000) 2024-03-06T16:42:18.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:18.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:18.509406+0000) 2024-03-06T16:42:19.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:19.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:19.509619+0000) 2024-03-06T16:42:20.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:20.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:20.509902+0000) 2024-03-06T16:42:21.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:21.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:21.510159+0000) 2024-03-06T16:42:22.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:22.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:22.510395+0000) 2024-03-06T16:42:23.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:23.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:23.510631+0000) 2024-03-06T16:42:24.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:24.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:24.510893+0000) 2024-03-06T16:42:25.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:25.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:25.511175+0000) 2024-03-06T16:42:26.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:26.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:26.511458+0000) 2024-03-06T16:42:27.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:27.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:27.511729+0000) 2024-03-06T16:42:28.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:28.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:28.511928+0000) 2024-03-06T16:42:29.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:29.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:29.512175+0000) 2024-03-06T16:42:30.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:30.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:30.512411+0000) 2024-03-06T16:42:31.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:31.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:31.512605+0000) 2024-03-06T16:42:32.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:32.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:32.512834+0000) 2024-03-06T16:42:33.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:33.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:33.513123+0000) 2024-03-06T16:42:34.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:34.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:34.513349+0000) 2024-03-06T16:42:35.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:35.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:35.513572+0000) 2024-03-06T16:42:36.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:36.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:36.513819+0000) 2024-03-06T16:42:37.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:37.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:37.514093+0000) 2024-03-06T16:42:38.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:38.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:38.514330+0000) 2024-03-06T16:42:39.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:39.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:39.514592+0000) 2024-03-06T16:42:40.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:40.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:40.514837+0000) 2024-03-06T16:42:41.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:41.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:41.515049+0000) 2024-03-06T16:42:42.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:42.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:42.515215+0000) 2024-03-06T16:42:43.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:43.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:43.515387+0000) 2024-03-06T16:42:44.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:44.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:42:44.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:44.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:44.515612+0000) 2024-03-06T16:42:45.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:45.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:45.515856+0000) 2024-03-06T16:42:46.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:46.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:46.516114+0000) 2024-03-06T16:42:47.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:47.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:47.516329+0000) 2024-03-06T16:42:48.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:48.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:48.516548+0000) 2024-03-06T16:42:49.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:49.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:49.516771+0000) 2024-03-06T16:42:50.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:50.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:50.516984+0000) 2024-03-06T16:42:51.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:51.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:51.517322+0000) 2024-03-06T16:42:52.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:52.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:52.517552+0000) 2024-03-06T16:42:53.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:53.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:53.517762+0000) 2024-03-06T16:42:54.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:54.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:54.517991+0000) 2024-03-06T16:42:55.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:55.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:55.518215+0000) 2024-03-06T16:42:56.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:56.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:56.518410+0000) 2024-03-06T16:42:57.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:57.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:57.518629+0000) 2024-03-06T16:42:58.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:58.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:58.518884+0000) 2024-03-06T16:42:59.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:42:59.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:42:59.519178+0000) 2024-03-06T16:43:00.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:00.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:00.519415+0000) 2024-03-06T16:43:01.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:01.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:01.519646+0000) 2024-03-06T16:43:02.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:02.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:02.519889+0000) 2024-03-06T16:43:03.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:03.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:03.520147+0000) 2024-03-06T16:43:04.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:04.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:04.520375+0000) 2024-03-06T16:43:05.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:05.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:05.520588+0000) 2024-03-06T16:43:06.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:06.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:06.520816+0000) 2024-03-06T16:43:07.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:07.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:07.521076+0000) 2024-03-06T16:43:08.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:08.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:08.521315+0000) 2024-03-06T16:43:09.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:09.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:09.521563+0000) 2024-03-06T16:43:10.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:10.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:10.521785+0000) 2024-03-06T16:43:11.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:11.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:11.522037+0000) 2024-03-06T16:43:12.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:12.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:12.522266+0000) 2024-03-06T16:43:13.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:13.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:13.522539+0000) 2024-03-06T16:43:14.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:14.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:14.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:14.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:14.522781+0000) 2024-03-06T16:43:15.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:15.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:15.523072+0000) 2024-03-06T16:43:16.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:16.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:16.523326+0000) 2024-03-06T16:43:17.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:17.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:17.523550+0000) 2024-03-06T16:43:18.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:18.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:18.523778+0000) 2024-03-06T16:43:19.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:19.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:19.524044+0000) 2024-03-06T16:43:20.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:20.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:20.524240+0000) 2024-03-06T16:43:21.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:21.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:21.524467+0000) 2024-03-06T16:43:22.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:22.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:22.524685+0000) 2024-03-06T16:43:23.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:23.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:23.524918+0000) 2024-03-06T16:43:24.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:24.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:24.525128+0000) 2024-03-06T16:43:25.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:25.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:25.525287+0000) 2024-03-06T16:43:26.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:26.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:26.525559+0000) 2024-03-06T16:43:27.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:27.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:27.525769+0000) 2024-03-06T16:43:28.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:28.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:28.525991+0000) 2024-03-06T16:43:29.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:29.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:29.526212+0000) 2024-03-06T16:43:30.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:30.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:30.526396+0000) 2024-03-06T16:43:31.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:31.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:31.526613+0000) 2024-03-06T16:43:32.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:32.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:32.526863+0000) 2024-03-06T16:43:33.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:33.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:33.527115+0000) 2024-03-06T16:43:34.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:34.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:34.527342+0000) 2024-03-06T16:43:35.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:35.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:35.527576+0000) 2024-03-06T16:43:36.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:36.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:36.527849+0000) 2024-03-06T16:43:37.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:37.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:37.528102+0000) 2024-03-06T16:43:38.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:38.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:38.528288+0000) 2024-03-06T16:43:39.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:39.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:39.528504+0000) 2024-03-06T16:43:40.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:40.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:40.528743+0000) 2024-03-06T16:43:41.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:41.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:41.528997+0000) 2024-03-06T16:43:42.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:42.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:42.529331+0000) 2024-03-06T16:43:43.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:43.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:43.529537+0000) 2024-03-06T16:43:44.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:44.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:44.529756+0000) 2024-03-06T16:43:44.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:44.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:43:45.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:45.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:45.529983+0000) 2024-03-06T16:43:46.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:46.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:46.530203+0000) 2024-03-06T16:43:47.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:47.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:47.530442+0000) 2024-03-06T16:43:48.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:48.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:48.530649+0000) 2024-03-06T16:43:49.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:49.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:49.530855+0000) 2024-03-06T16:43:50.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:50.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:50.531089+0000) 2024-03-06T16:43:51.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:51.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:51.531282+0000) 2024-03-06T16:43:52.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:52.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:52.531490+0000) 2024-03-06T16:43:53.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:53.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:53.531700+0000) 2024-03-06T16:43:54.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:54.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:54.531914+0000) 2024-03-06T16:43:55.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:55.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:55.532144+0000) 2024-03-06T16:43:56.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:56.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:56.532453+0000) 2024-03-06T16:43:57.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:57.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:57.532663+0000) 2024-03-06T16:43:58.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:58.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:58.532880+0000) 2024-03-06T16:43:59.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:43:59.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:43:59.533150+0000) 2024-03-06T16:44:00.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:00.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:00.533393+0000) 2024-03-06T16:44:01.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:01.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:01.533632+0000) 2024-03-06T16:44:02.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:02.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:02.533852+0000) 2024-03-06T16:44:03.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:03.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:03.534120+0000) 2024-03-06T16:44:04.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:04.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:04.534341+0000) 2024-03-06T16:44:05.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:05.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:05.534558+0000) 2024-03-06T16:44:06.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:06.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:06.534766+0000) 2024-03-06T16:44:07.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:07.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:07.534988+0000) 2024-03-06T16:44:08.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:08.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:08.535281+0000) 2024-03-06T16:44:09.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:09.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:09.535537+0000) 2024-03-06T16:44:10.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:10.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:10.535818+0000) 2024-03-06T16:44:11.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:11.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:11.536136+0000) 2024-03-06T16:44:12.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:12.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:12.536386+0000) 2024-03-06T16:44:13.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:13.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:13.536662+0000) 2024-03-06T16:44:14.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:14.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:14.536876+0000) 2024-03-06T16:44:14.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:14.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:15.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:15.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:15.537160+0000) 2024-03-06T16:44:16.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:16.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:16.537341+0000) 2024-03-06T16:44:17.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:17.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:17.537549+0000) 2024-03-06T16:44:18.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:18.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:18.537764+0000) 2024-03-06T16:44:19.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:19.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:19.538001+0000) 2024-03-06T16:44:20.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:20.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:20.538252+0000) 2024-03-06T16:44:21.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:21.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:21.538458+0000) 2024-03-06T16:44:22.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:22.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:22.538689+0000) 2024-03-06T16:44:23.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:23.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:23.538968+0000) 2024-03-06T16:44:24.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:24.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:24.539288+0000) 2024-03-06T16:44:25.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:25.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:25.539551+0000) 2024-03-06T16:44:26.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:26.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:26.539827+0000) 2024-03-06T16:44:27.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:27.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:27.540116+0000) 2024-03-06T16:44:28.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:28.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:28.540282+0000) 2024-03-06T16:44:29.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:29.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:29.540489+0000) 2024-03-06T16:44:30.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:30.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:30.540742+0000) 2024-03-06T16:44:31.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:31.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:31.540958+0000) 2024-03-06T16:44:32.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:32.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:32.541156+0000) 2024-03-06T16:44:33.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:33.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:33.541374+0000) 2024-03-06T16:44:34.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:34.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:34.541575+0000) 2024-03-06T16:44:35.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:35.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:35.541803+0000) 2024-03-06T16:44:36.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:36.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:36.542121+0000) 2024-03-06T16:44:37.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:37.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:37.542382+0000) 2024-03-06T16:44:38.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:38.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:38.542577+0000) 2024-03-06T16:44:39.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:39.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:39.542786+0000) 2024-03-06T16:44:40.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:40.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:40.543026+0000) 2024-03-06T16:44:41.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:41.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:41.543263+0000) 2024-03-06T16:44:42.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:42.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:42.543518+0000) 2024-03-06T16:44:43.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:43.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:43.543774+0000) 2024-03-06T16:44:44.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:44.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:44.543998+0000) 2024-03-06T16:44:45.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:45.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:44:45.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:45.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:45.544239+0000) 2024-03-06T16:44:46.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:46.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:46.544460+0000) 2024-03-06T16:44:47.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:47.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:47.544671+0000) 2024-03-06T16:44:48.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:48.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:48.544882+0000) 2024-03-06T16:44:49.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:49.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:49.545149+0000) 2024-03-06T16:44:50.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:50.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:50.545336+0000) 2024-03-06T16:44:51.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:51.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:51.545519+0000) 2024-03-06T16:44:52.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:52.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:52.545765+0000) 2024-03-06T16:44:53.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:53.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:53.545990+0000) 2024-03-06T16:44:54.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:54.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:54.546238+0000) 2024-03-06T16:44:55.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:55.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:55.546434+0000) 2024-03-06T16:44:56.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:56.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:56.546647+0000) 2024-03-06T16:44:57.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:57.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:57.546882+0000) 2024-03-06T16:44:58.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:58.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:58.547117+0000) 2024-03-06T16:44:59.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:44:59.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:44:59.547311+0000) 2024-03-06T16:45:00.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:00.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:00.547528+0000) 2024-03-06T16:45:01.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:01.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:01.547771+0000) 2024-03-06T16:45:02.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:02.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:02.547991+0000) 2024-03-06T16:45:03.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:03.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:03.548242+0000) 2024-03-06T16:45:04.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:04.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:04.548463+0000) 2024-03-06T16:45:05.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:05.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:05.548693+0000) 2024-03-06T16:45:06.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:06.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:06.548917+0000) 2024-03-06T16:45:07.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:07.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:07.549156+0000) 2024-03-06T16:45:08.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:08.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:08.549338+0000) 2024-03-06T16:45:09.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:09.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:09.549555+0000) 2024-03-06T16:45:10.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:10.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:10.549772+0000) 2024-03-06T16:45:11.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:11.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:11.549995+0000) 2024-03-06T16:45:12.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:12.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:12.550257+0000) 2024-03-06T16:45:13.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:13.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:13.550498+0000) 2024-03-06T16:45:14.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:14.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:14.550715+0000) 2024-03-06T16:45:15.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:15.180 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:15.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:15.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:15.550954+0000) 2024-03-06T16:45:16.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:16.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:16.551186+0000) 2024-03-06T16:45:17.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:17.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:17.551387+0000) 2024-03-06T16:45:18.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:18.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:18.551596+0000) 2024-03-06T16:45:19.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:19.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:19.551822+0000) 2024-03-06T16:45:20.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:20.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:20.552074+0000) 2024-03-06T16:45:21.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:21.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:21.552266+0000) 2024-03-06T16:45:22.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:22.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:22.552498+0000) 2024-03-06T16:45:23.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:23.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:23.552673+0000) 2024-03-06T16:45:24.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:24.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:24.552895+0000) 2024-03-06T16:45:25.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:25.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:25.553120+0000) 2024-03-06T16:45:26.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:26.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:26.553301+0000) 2024-03-06T16:45:27.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:27.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:27.553525+0000) 2024-03-06T16:45:28.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:28.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:28.553796+0000) 2024-03-06T16:45:29.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:29.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:29.554052+0000) 2024-03-06T16:45:30.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:30.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:30.554240+0000) 2024-03-06T16:45:31.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:31.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:31.554466+0000) 2024-03-06T16:45:32.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:32.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:32.554684+0000) 2024-03-06T16:45:33.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:33.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:33.554911+0000) 2024-03-06T16:45:34.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:34.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:34.555152+0000) 2024-03-06T16:45:35.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:35.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:35.555397+0000) 2024-03-06T16:45:36.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:36.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:36.555651+0000) 2024-03-06T16:45:37.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:37.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:37.555886+0000) 2024-03-06T16:45:38.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:38.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:38.556106+0000) 2024-03-06T16:45:39.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:39.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:39.556301+0000) 2024-03-06T16:45:40.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:40.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:40.556513+0000) 2024-03-06T16:45:41.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:41.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:41.556740+0000) 2024-03-06T16:45:42.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:42.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:42.556957+0000) 2024-03-06T16:45:43.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:43.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:43.557177+0000) 2024-03-06T16:45:44.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:44.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:44.557405+0000) 2024-03-06T16:45:45.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:45.354 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:45:45.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:45.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:45.557623+0000) 2024-03-06T16:45:46.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:46.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:46.557855+0000) 2024-03-06T16:45:47.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:47.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:47.558126+0000) 2024-03-06T16:45:48.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:48.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:48.558276+0000) 2024-03-06T16:45:49.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:49.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:49.558485+0000) 2024-03-06T16:45:50.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:50.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:50.558708+0000) 2024-03-06T16:45:51.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:51.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:51.558880+0000) 2024-03-06T16:45:52.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:52.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:52.559106+0000) 2024-03-06T16:45:53.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:53.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:53.559293+0000) 2024-03-06T16:45:54.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:54.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:54.559510+0000) 2024-03-06T16:45:55.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:55.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:55.559750+0000) 2024-03-06T16:45:56.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:56.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:56.559987+0000) 2024-03-06T16:45:57.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:57.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:57.560234+0000) 2024-03-06T16:45:58.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:58.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:58.560412+0000) 2024-03-06T16:45:59.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:45:59.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:45:59.560592+0000) 2024-03-06T16:46:00.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:00.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:00.560809+0000) 2024-03-06T16:46:01.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:01.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:01.560992+0000) 2024-03-06T16:46:02.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:02.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:02.561216+0000) 2024-03-06T16:46:03.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:03.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:03.561403+0000) 2024-03-06T16:46:04.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:04.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:04.561604+0000) 2024-03-06T16:46:05.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:05.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:05.561845+0000) 2024-03-06T16:46:06.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:06.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:06.562106+0000) 2024-03-06T16:46:07.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:07.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:07.562298+0000) 2024-03-06T16:46:08.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:08.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:08.562499+0000) 2024-03-06T16:46:09.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:09.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:09.562713+0000) 2024-03-06T16:46:10.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:10.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:10.562899+0000) 2024-03-06T16:46:11.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:11.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:11.563082+0000) 2024-03-06T16:46:12.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:12.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:12.563243+0000) 2024-03-06T16:46:13.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:13.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:13.563451+0000) 2024-03-06T16:46:14.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:14.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:14.563677+0000) 2024-03-06T16:46:15.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:15.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:15.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:15.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:15.563907+0000) 2024-03-06T16:46:16.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:16.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:16.564220+0000) 2024-03-06T16:46:17.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:17.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:17.564443+0000) 2024-03-06T16:46:18.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:18.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:18.564644+0000) 2024-03-06T16:46:19.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:19.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:19.564872+0000) 2024-03-06T16:46:20.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:20.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:20.565028+0000) 2024-03-06T16:46:21.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:21.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:21.565192+0000) 2024-03-06T16:46:22.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:22.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:22.565410+0000) 2024-03-06T16:46:23.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:23.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:23.565598+0000) 2024-03-06T16:46:24.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:24.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:24.565830+0000) 2024-03-06T16:46:25.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:25.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:25.566099+0000) 2024-03-06T16:46:26.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:26.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:26.566333+0000) 2024-03-06T16:46:27.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:27.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:27.566547+0000) 2024-03-06T16:46:28.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:28.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:28.566770+0000) 2024-03-06T16:46:29.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:29.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:29.567045+0000) 2024-03-06T16:46:30.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:30.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:30.567292+0000) 2024-03-06T16:46:31.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:31.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:31.567515+0000) 2024-03-06T16:46:32.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:32.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:32.567737+0000) 2024-03-06T16:46:33.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:33.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:33.567959+0000) 2024-03-06T16:46:34.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:34.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:34.568187+0000) 2024-03-06T16:46:35.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:35.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:35.568413+0000) 2024-03-06T16:46:36.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:36.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:36.568685+0000) 2024-03-06T16:46:37.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:37.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:37.568958+0000) 2024-03-06T16:46:38.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:38.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:38.569147+0000) 2024-03-06T16:46:39.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:39.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:39.569340+0000) 2024-03-06T16:46:40.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:40.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:40.569549+0000) 2024-03-06T16:46:41.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:41.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:41.569770+0000) 2024-03-06T16:46:42.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:42.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:42.569985+0000) 2024-03-06T16:46:43.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:43.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:43.570244+0000) 2024-03-06T16:46:44.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:44.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:44.570496+0000) 2024-03-06T16:46:45.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:45.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:45.570716+0000) 2024-03-06T16:46:45.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:45.699 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:46:46.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:46.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:46.570949+0000) 2024-03-06T16:46:47.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:47.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:47.571149+0000) 2024-03-06T16:46:48.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:48.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:48.571339+0000) 2024-03-06T16:46:49.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:49.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:49.571557+0000) 2024-03-06T16:46:50.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:50.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:50.571777+0000) 2024-03-06T16:46:51.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:51.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:51.571996+0000) 2024-03-06T16:46:52.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:52.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:52.572209+0000) 2024-03-06T16:46:53.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:53.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:53.572396+0000) 2024-03-06T16:46:54.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:54.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:54.572597+0000) 2024-03-06T16:46:55.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:55.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:55.572816+0000) 2024-03-06T16:46:56.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:56.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:56.573043+0000) 2024-03-06T16:46:57.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:57.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:57.573231+0000) 2024-03-06T16:46:58.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:58.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:58.573407+0000) 2024-03-06T16:46:59.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:46:59.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:46:59.573646+0000) 2024-03-06T16:47:00.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:00.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:00.573837+0000) 2024-03-06T16:47:01.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:01.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:01.574111+0000) 2024-03-06T16:47:02.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:02.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:02.574300+0000) 2024-03-06T16:47:03.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:03.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:03.574541+0000) 2024-03-06T16:47:04.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:04.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:04.574739+0000) 2024-03-06T16:47:05.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:05.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:05.574959+0000) 2024-03-06T16:47:06.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:06.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:06.575180+0000) 2024-03-06T16:47:07.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:07.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:07.575341+0000) 2024-03-06T16:47:08.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:08.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:08.575504+0000) 2024-03-06T16:47:09.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:09.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:09.575728+0000) 2024-03-06T16:47:10.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:10.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:10.575999+0000) 2024-03-06T16:47:11.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:11.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:11.576236+0000) 2024-03-06T16:47:12.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:12.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:12.576463+0000) 2024-03-06T16:47:13.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:13.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:13.576657+0000) 2024-03-06T16:47:14.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:14.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:14.576877+0000) 2024-03-06T16:47:15.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:15.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:15.577120+0000) 2024-03-06T16:47:15.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:15.913 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:16.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:16.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:16.577351+0000) 2024-03-06T16:47:17.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:17.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:17.577563+0000) 2024-03-06T16:47:18.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:18.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:18.577797+0000) 2024-03-06T16:47:19.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:19.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:19.578093+0000) 2024-03-06T16:47:20.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:20.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:20.578304+0000) 2024-03-06T16:47:21.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:21.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:21.578543+0000) 2024-03-06T16:47:22.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:22.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:22.578765+0000) 2024-03-06T16:47:23.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:23.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:23.578986+0000) 2024-03-06T16:47:24.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:24.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:24.579219+0000) 2024-03-06T16:47:25.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:25.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:25.579443+0000) 2024-03-06T16:47:26.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:26.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:26.579625+0000) 2024-03-06T16:47:27.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:27.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:27.579805+0000) 2024-03-06T16:47:28.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:28.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:28.580062+0000) 2024-03-06T16:47:29.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:29.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:29.580294+0000) 2024-03-06T16:47:30.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:30.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:30.580512+0000) 2024-03-06T16:47:31.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:31.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:31.580736+0000) 2024-03-06T16:47:32.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:32.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:32.580954+0000) 2024-03-06T16:47:33.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:33.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:33.581253+0000) 2024-03-06T16:47:34.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:34.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:34.581511+0000) 2024-03-06T16:47:35.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:35.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:35.581793+0000) 2024-03-06T16:47:36.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:36.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:36.582129+0000) 2024-03-06T16:47:37.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:37.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:37.582411+0000) 2024-03-06T16:47:38.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:38.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:38.582689+0000) 2024-03-06T16:47:39.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:39.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:39.582958+0000) 2024-03-06T16:47:40.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:40.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:40.583240+0000) 2024-03-06T16:47:41.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:41.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:41.583493+0000) 2024-03-06T16:47:42.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:42.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:42.583733+0000) 2024-03-06T16:47:43.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:43.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:43.583974+0000) 2024-03-06T16:47:44.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:44.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:44.584196+0000) 2024-03-06T16:47:45.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:45.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:45.584345+0000) 2024-03-06T16:47:46.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:46.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:47:46.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:46.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:46.584535+0000) 2024-03-06T16:47:47.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:47.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:47.584779+0000) 2024-03-06T16:47:48.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:48.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:48.585043+0000) 2024-03-06T16:47:49.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:49.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:49.585307+0000) 2024-03-06T16:47:50.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:50.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:50.585531+0000) 2024-03-06T16:47:51.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:51.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:51.585731+0000) 2024-03-06T16:47:52.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:52.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:52.585955+0000) 2024-03-06T16:47:53.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:53.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:53.586176+0000) 2024-03-06T16:47:54.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:54.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:54.586390+0000) 2024-03-06T16:47:55.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:55.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:55.586611+0000) 2024-03-06T16:47:56.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:56.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:56.586802+0000) 2024-03-06T16:47:57.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:57.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:57.587060+0000) 2024-03-06T16:47:58.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:58.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:58.587391+0000) 2024-03-06T16:47:59.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:47:59.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:47:59.587670+0000) 2024-03-06T16:48:00.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:00.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:00.587886+0000) 2024-03-06T16:48:01.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:01.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:01.588168+0000) 2024-03-06T16:48:02.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:02.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:02.588447+0000) 2024-03-06T16:48:03.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:03.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:03.588712+0000) 2024-03-06T16:48:04.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:04.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:04.588910+0000) 2024-03-06T16:48:05.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:05.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:05.589147+0000) 2024-03-06T16:48:06.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:06.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:06.589352+0000) 2024-03-06T16:48:07.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:07.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:07.589623+0000) 2024-03-06T16:48:08.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:08.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:08.589884+0000) 2024-03-06T16:48:09.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:09.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:09.590164+0000) 2024-03-06T16:48:10.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:10.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:10.590440+0000) 2024-03-06T16:48:11.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:11.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:11.590720+0000) 2024-03-06T16:48:12.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:12.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:12.590998+0000) 2024-03-06T16:48:13.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:13.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:13.591330+0000) 2024-03-06T16:48:14.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:14.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:14.591598+0000) 2024-03-06T16:48:15.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:15.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:15.591839+0000) 2024-03-06T16:48:16.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:16.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:16.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:16.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:16.592118+0000) 2024-03-06T16:48:17.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:17.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:17.592386+0000) 2024-03-06T16:48:18.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:18.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:18.592665+0000) 2024-03-06T16:48:19.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:19.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:19.592953+0000) 2024-03-06T16:48:20.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:20.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:20.593237+0000) 2024-03-06T16:48:21.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:21.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:21.593473+0000) 2024-03-06T16:48:22.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:22.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:22.593735+0000) 2024-03-06T16:48:23.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:23.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:23.594046+0000) 2024-03-06T16:48:24.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:24.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:24.594334+0000) 2024-03-06T16:48:25.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:25.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:25.594617+0000) 2024-03-06T16:48:26.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:26.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:26.594859+0000) 2024-03-06T16:48:27.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:27.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:27.595178+0000) 2024-03-06T16:48:28.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:28.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:28.595459+0000) 2024-03-06T16:48:29.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:29.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:29.595705+0000) 2024-03-06T16:48:30.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:30.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:30.595977+0000) 2024-03-06T16:48:31.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:31.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:31.596284+0000) 2024-03-06T16:48:32.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:32.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:32.596550+0000) 2024-03-06T16:48:33.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:33.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:33.596833+0000) 2024-03-06T16:48:34.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:34.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:34.597152+0000) 2024-03-06T16:48:35.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:35.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:35.597396+0000) 2024-03-06T16:48:36.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:36.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:36.597680+0000) 2024-03-06T16:48:37.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:37.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:37.597951+0000) 2024-03-06T16:48:38.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:38.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:38.598197+0000) 2024-03-06T16:48:39.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:39.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:39.598492+0000) 2024-03-06T16:48:40.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:40.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:40.598695+0000) 2024-03-06T16:48:41.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:41.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:41.598913+0000) 2024-03-06T16:48:42.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:42.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:42.599154+0000) 2024-03-06T16:48:43.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:43.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:43.599342+0000) 2024-03-06T16:48:44.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:44.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:44.599569+0000) 2024-03-06T16:48:45.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:45.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:45.599788+0000) 2024-03-06T16:48:46.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:46.472 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:48:46.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:46.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:46.600137+0000) 2024-03-06T16:48:47.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:47.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:47.600346+0000) 2024-03-06T16:48:48.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:48.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:48.600549+0000) 2024-03-06T16:48:49.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:49.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:49.600783+0000) 2024-03-06T16:48:50.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:50.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:50.601046+0000) 2024-03-06T16:48:51.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:51.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:51.601278+0000) 2024-03-06T16:48:52.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:52.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:52.601480+0000) 2024-03-06T16:48:53.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:53.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:53.601706+0000) 2024-03-06T16:48:54.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:54.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:54.601942+0000) 2024-03-06T16:48:55.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:55.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:55.602168+0000) 2024-03-06T16:48:56.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:56.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:56.602369+0000) 2024-03-06T16:48:57.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:57.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:57.602577+0000) 2024-03-06T16:48:58.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:58.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:58.602793+0000) 2024-03-06T16:48:59.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:48:59.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:48:59.603041+0000) 2024-03-06T16:49:00.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:00.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:00.603241+0000) 2024-03-06T16:49:01.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:01.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:01.603448+0000) 2024-03-06T16:49:02.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:02.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:02.603672+0000) 2024-03-06T16:49:03.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:03.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:03.603898+0000) 2024-03-06T16:49:04.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:04.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:04.604153+0000) 2024-03-06T16:49:05.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:05.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:05.604348+0000) 2024-03-06T16:49:06.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:06.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:06.604557+0000) 2024-03-06T16:49:07.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:07.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:07.604783+0000) 2024-03-06T16:49:08.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:08.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:08.605039+0000) 2024-03-06T16:49:09.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:09.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:09.605274+0000) 2024-03-06T16:49:10.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:10.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:10.605479+0000) 2024-03-06T16:49:11.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:11.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:11.605751+0000) 2024-03-06T16:49:12.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:12.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:12.605995+0000) 2024-03-06T16:49:13.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:13.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:13.606310+0000) 2024-03-06T16:49:14.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:14.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:14.606558+0000) 2024-03-06T16:49:15.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:15.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:15.606802+0000) 2024-03-06T16:49:16.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:16.610 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:16.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:16.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:16.606988+0000) 2024-03-06T16:49:17.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:17.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:17.607273+0000) 2024-03-06T16:49:18.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:18.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:18.607542+0000) 2024-03-06T16:49:19.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:19.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:19.607825+0000) 2024-03-06T16:49:20.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:20.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:20.608152+0000) 2024-03-06T16:49:21.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:21.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:21.608402+0000) 2024-03-06T16:49:22.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:22.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:22.608647+0000) 2024-03-06T16:49:23.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:23.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:23.608842+0000) 2024-03-06T16:49:24.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:24.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:24.609055+0000) 2024-03-06T16:49:25.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:25.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:25.609320+0000) 2024-03-06T16:49:26.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:26.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:26.609530+0000) 2024-03-06T16:49:27.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:27.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:27.609771+0000) 2024-03-06T16:49:28.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:28.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:28.609971+0000) 2024-03-06T16:49:29.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:29.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:29.610207+0000) 2024-03-06T16:49:30.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:30.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:30.610439+0000) 2024-03-06T16:49:31.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:31.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:31.610660+0000) 2024-03-06T16:49:32.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:32.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:32.610909+0000) 2024-03-06T16:49:33.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:33.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:33.611128+0000) 2024-03-06T16:49:34.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:34.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:34.611357+0000) 2024-03-06T16:49:35.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:35.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:35.611618+0000) 2024-03-06T16:49:36.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:36.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:36.611857+0000) 2024-03-06T16:49:37.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:37.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:37.612120+0000) 2024-03-06T16:49:38.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:38.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:38.612337+0000) 2024-03-06T16:49:39.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:39.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:39.612584+0000) 2024-03-06T16:49:40.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:40.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:40.612828+0000) 2024-03-06T16:49:41.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:41.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:41.613132+0000) 2024-03-06T16:49:42.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:42.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:42.613405+0000) 2024-03-06T16:49:43.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:43.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:43.613646+0000) 2024-03-06T16:49:44.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:44.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:44.613922+0000) 2024-03-06T16:49:45.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:45.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:45.614202+0000) 2024-03-06T16:49:46.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:46.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:46.614480+0000) 2024-03-06T16:49:46.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:46.762 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:49:47.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:47.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:47.614641+0000) 2024-03-06T16:49:48.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:48.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:48.614852+0000) 2024-03-06T16:49:49.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:49.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:49.614988+0000) 2024-03-06T16:49:50.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:50.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:50.615227+0000) 2024-03-06T16:49:51.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:51.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:51.615446+0000) 2024-03-06T16:49:52.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:52.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:52.615706+0000) 2024-03-06T16:49:53.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:53.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:53.615983+0000) 2024-03-06T16:49:54.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:54.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:54.616275+0000) 2024-03-06T16:49:55.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:55.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:55.616489+0000) 2024-03-06T16:49:56.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:56.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:56.616782+0000) 2024-03-06T16:49:57.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:57.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:57.617083+0000) 2024-03-06T16:49:58.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:58.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:58.617325+0000) 2024-03-06T16:49:59.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:49:59.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:49:59.617606+0000) 2024-03-06T16:50:00.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:00.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:00.617876+0000) 2024-03-06T16:50:01.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:01.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:01.618144+0000) 2024-03-06T16:50:02.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:02.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:02.618311+0000) 2024-03-06T16:50:03.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:03.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:03.618530+0000) 2024-03-06T16:50:04.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:04.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:04.618738+0000) 2024-03-06T16:50:05.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:05.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:05.618992+0000) 2024-03-06T16:50:06.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:06.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:06.619273+0000) 2024-03-06T16:50:07.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:07.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:07.619480+0000) 2024-03-06T16:50:08.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:08.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:08.619690+0000) 2024-03-06T16:50:09.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:09.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:09.619872+0000) 2024-03-06T16:50:10.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:10.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:10.620115+0000) 2024-03-06T16:50:11.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:11.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:11.620343+0000) 2024-03-06T16:50:12.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:12.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:12.620598+0000) 2024-03-06T16:50:13.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:13.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:13.620791+0000) 2024-03-06T16:50:14.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:14.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:14.621036+0000) 2024-03-06T16:50:15.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:15.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:15.621234+0000) 2024-03-06T16:50:16.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:16.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:16.621463+0000) 2024-03-06T16:50:16.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:16.924 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:17.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:17.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:17.621690+0000) 2024-03-06T16:50:18.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:18.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:18.621917+0000) 2024-03-06T16:50:19.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:19.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:19.622169+0000) 2024-03-06T16:50:20.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:20.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:20.622386+0000) 2024-03-06T16:50:21.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:21.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:21.622572+0000) 2024-03-06T16:50:22.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:22.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:22.622794+0000) 2024-03-06T16:50:23.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:23.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:23.623035+0000) 2024-03-06T16:50:24.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:24.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:24.623236+0000) 2024-03-06T16:50:25.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:25.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:25.623463+0000) 2024-03-06T16:50:26.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:26.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:26.623719+0000) 2024-03-06T16:50:27.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:27.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:27.624033+0000) 2024-03-06T16:50:28.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:28.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:28.624224+0000) 2024-03-06T16:50:29.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:29.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:29.624412+0000) 2024-03-06T16:50:30.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:30.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:30.624648+0000) 2024-03-06T16:50:31.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:31.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:31.624888+0000) 2024-03-06T16:50:32.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:32.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:32.625124+0000) 2024-03-06T16:50:33.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:33.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:33.625343+0000) 2024-03-06T16:50:34.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:34.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:34.625601+0000) 2024-03-06T16:50:35.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:35.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:35.625829+0000) 2024-03-06T16:50:36.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:36.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:36.626095+0000) 2024-03-06T16:50:37.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:37.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:37.626318+0000) 2024-03-06T16:50:38.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:38.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:38.626572+0000) 2024-03-06T16:50:39.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:39.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:39.626792+0000) 2024-03-06T16:50:40.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:40.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:40.627033+0000) 2024-03-06T16:50:41.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:41.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:41.627226+0000) 2024-03-06T16:50:42.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:42.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:42.627410+0000) 2024-03-06T16:50:43.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:43.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:43.627611+0000) 2024-03-06T16:50:44.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:44.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:44.627842+0000) 2024-03-06T16:50:45.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:45.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:45.628085+0000) 2024-03-06T16:50:46.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:46.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:46.628303+0000) 2024-03-06T16:50:47.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:47.072 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:50:47.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:47.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:47.628534+0000) 2024-03-06T16:50:48.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:48.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:48.628757+0000) 2024-03-06T16:50:49.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:49.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:49.628918+0000) 2024-03-06T16:50:50.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:50.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:50.629141+0000) 2024-03-06T16:50:51.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:51.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:51.629377+0000) 2024-03-06T16:50:52.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:52.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:52.629606+0000) 2024-03-06T16:50:53.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:53.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:53.629837+0000) 2024-03-06T16:50:54.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:54.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:54.630120+0000) 2024-03-06T16:50:55.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:55.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:55.630370+0000) 2024-03-06T16:50:56.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:56.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:56.630582+0000) 2024-03-06T16:50:57.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:57.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:57.630819+0000) 2024-03-06T16:50:58.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:58.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:58.631001+0000) 2024-03-06T16:50:59.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:50:59.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:50:59.631267+0000) 2024-03-06T16:51:00.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:00.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:00.631503+0000) 2024-03-06T16:51:01.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:01.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:01.631730+0000) 2024-03-06T16:51:02.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:02.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:02.631950+0000) 2024-03-06T16:51:03.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:03.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:03.632173+0000) 2024-03-06T16:51:04.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:04.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:04.632403+0000) 2024-03-06T16:51:05.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:05.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:05.632681+0000) 2024-03-06T16:51:06.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:06.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:06.632972+0000) 2024-03-06T16:51:07.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:07.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:07.633298+0000) 2024-03-06T16:51:08.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:08.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:08.633576+0000) 2024-03-06T16:51:09.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:09.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:09.633761+0000) 2024-03-06T16:51:10.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:10.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:10.633983+0000) 2024-03-06T16:51:11.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:11.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:11.634217+0000) 2024-03-06T16:51:12.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:12.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:12.634367+0000) 2024-03-06T16:51:13.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:13.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:13.634574+0000) 2024-03-06T16:51:14.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:14.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:14.634792+0000) 2024-03-06T16:51:15.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:15.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:15.635109+0000) 2024-03-06T16:51:16.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:16.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:16.635311+0000) 2024-03-06T16:51:17.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:17.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:17.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:17.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:17.635528+0000) 2024-03-06T16:51:18.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:18.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:18.635758+0000) 2024-03-06T16:51:19.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:19.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:19.635983+0000) 2024-03-06T16:51:20.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:20.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:20.636240+0000) 2024-03-06T16:51:21.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:21.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:21.636445+0000) 2024-03-06T16:51:22.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:22.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:22.636647+0000) 2024-03-06T16:51:23.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:23.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:23.636883+0000) 2024-03-06T16:51:24.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:24.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:24.637140+0000) 2024-03-06T16:51:25.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:25.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:25.637337+0000) 2024-03-06T16:51:26.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:26.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:26.637564+0000) 2024-03-06T16:51:27.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:27.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:27.637799+0000) 2024-03-06T16:51:28.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:28.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:28.638052+0000) 2024-03-06T16:51:29.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:29.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:29.638242+0000) 2024-03-06T16:51:30.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:30.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:30.638442+0000) 2024-03-06T16:51:31.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:31.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:31.638644+0000) 2024-03-06T16:51:32.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:32.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:32.638864+0000) 2024-03-06T16:51:33.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:33.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:33.639090+0000) 2024-03-06T16:51:34.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:34.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:34.639343+0000) 2024-03-06T16:51:35.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:35.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:35.639548+0000) 2024-03-06T16:51:36.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:36.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:36.639782+0000) 2024-03-06T16:51:37.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:37.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:37.640001+0000) 2024-03-06T16:51:38.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:38.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:38.640211+0000) 2024-03-06T16:51:39.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:39.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:39.640404+0000) 2024-03-06T16:51:40.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:40.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:40.640604+0000) 2024-03-06T16:51:41.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:41.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:41.640829+0000) 2024-03-06T16:51:42.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:42.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:42.641094+0000) 2024-03-06T16:51:43.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:43.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:43.641296+0000) 2024-03-06T16:51:44.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:44.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:44.641497+0000) 2024-03-06T16:51:45.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:45.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:45.641735+0000) 2024-03-06T16:51:46.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:46.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:46.641930+0000) 2024-03-06T16:51:47.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:47.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:51:47.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:47.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:47.642168+0000) 2024-03-06T16:51:48.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:48.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:48.642356+0000) 2024-03-06T16:51:49.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:49.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:49.642535+0000) 2024-03-06T16:51:50.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:50.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:50.642752+0000) 2024-03-06T16:51:51.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:51.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:51.642973+0000) 2024-03-06T16:51:52.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:52.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:52.643176+0000) 2024-03-06T16:51:53.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:53.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:53.643364+0000) 2024-03-06T16:51:54.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:54.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:54.643537+0000) 2024-03-06T16:51:55.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:55.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:55.643761+0000) 2024-03-06T16:51:56.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:56.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:56.643993+0000) 2024-03-06T16:51:57.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:57.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:57.644226+0000) 2024-03-06T16:51:58.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:58.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:58.644410+0000) 2024-03-06T16:51:59.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:51:59.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:51:59.644627+0000) 2024-03-06T16:52:00.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:00.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:00.644844+0000) 2024-03-06T16:52:01.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:01.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:01.645092+0000) 2024-03-06T16:52:02.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:02.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:02.645284+0000) 2024-03-06T16:52:03.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:03.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:03.645460+0000) 2024-03-06T16:52:04.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:04.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:04.645662+0000) 2024-03-06T16:52:05.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:05.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:05.645895+0000) 2024-03-06T16:52:06.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:06.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:06.646092+0000) 2024-03-06T16:52:07.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:07.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:07.646323+0000) 2024-03-06T16:52:08.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:08.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:08.646594+0000) 2024-03-06T16:52:09.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:09.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:09.646814+0000) 2024-03-06T16:52:10.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:10.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:10.647075+0000) 2024-03-06T16:52:11.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:11.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:11.647309+0000) 2024-03-06T16:52:12.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:12.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:12.647571+0000) 2024-03-06T16:52:13.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:13.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:13.647844+0000) 2024-03-06T16:52:14.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:14.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:14.648149+0000) 2024-03-06T16:52:15.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:15.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:15.648393+0000) 2024-03-06T16:52:16.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:16.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:16.648691+0000) 2024-03-06T16:52:17.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:17.516 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:17.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:17.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:17.648942+0000) 2024-03-06T16:52:18.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:18.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:18.649201+0000) 2024-03-06T16:52:19.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:19.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:19.649446+0000) 2024-03-06T16:52:20.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:20.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:20.649692+0000) 2024-03-06T16:52:21.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:21.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:21.649896+0000) 2024-03-06T16:52:22.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:22.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:22.650162+0000) 2024-03-06T16:52:23.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:23.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:23.650422+0000) 2024-03-06T16:52:24.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:24.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:24.650623+0000) 2024-03-06T16:52:25.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:25.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:25.650852+0000) 2024-03-06T16:52:26.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:26.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:26.651151+0000) 2024-03-06T16:52:27.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:27.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:27.651365+0000) 2024-03-06T16:52:28.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:28.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:28.651572+0000) 2024-03-06T16:52:29.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:29.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:29.651793+0000) 2024-03-06T16:52:30.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:30.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:30.651996+0000) 2024-03-06T16:52:31.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:31.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:31.652336+0000) 2024-03-06T16:52:32.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:32.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:32.652595+0000) 2024-03-06T16:52:33.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:33.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:33.652875+0000) 2024-03-06T16:52:34.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:34.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:34.653165+0000) 2024-03-06T16:52:35.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:35.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:35.653433+0000) 2024-03-06T16:52:36.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:36.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:36.653678+0000) 2024-03-06T16:52:37.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:37.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:37.653923+0000) 2024-03-06T16:52:38.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:38.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:38.654207+0000) 2024-03-06T16:52:39.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:39.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:39.654456+0000) 2024-03-06T16:52:40.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:40.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:40.654692+0000) 2024-03-06T16:52:41.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:41.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:41.654944+0000) 2024-03-06T16:52:42.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:42.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:42.655168+0000) 2024-03-06T16:52:43.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:43.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:43.655403+0000) 2024-03-06T16:52:44.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:44.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:44.655637+0000) 2024-03-06T16:52:45.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:45.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:45.655853+0000) 2024-03-06T16:52:46.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:46.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:46.656120+0000) 2024-03-06T16:52:47.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:47.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:52:47.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:47.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:47.656374+0000) 2024-03-06T16:52:48.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:48.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:48.656626+0000) 2024-03-06T16:52:49.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:49.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:49.656875+0000) 2024-03-06T16:52:50.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:50.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:50.657090+0000) 2024-03-06T16:52:51.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:51.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:51.657279+0000) 2024-03-06T16:52:52.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:52.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:52.657483+0000) 2024-03-06T16:52:53.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:53.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:53.657703+0000) 2024-03-06T16:52:54.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:54.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:54.657910+0000) 2024-03-06T16:52:55.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:55.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:55.658165+0000) 2024-03-06T16:52:56.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:56.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:56.658418+0000) 2024-03-06T16:52:57.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:57.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:57.658598+0000) 2024-03-06T16:52:58.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:58.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:58.658810+0000) 2024-03-06T16:52:59.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:52:59.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:52:59.659065+0000) 2024-03-06T16:53:00.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:00.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:00.659279+0000) 2024-03-06T16:53:01.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:01.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:01.659479+0000) 2024-03-06T16:53:02.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:02.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:02.659640+0000) 2024-03-06T16:53:03.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:03.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:03.659858+0000) 2024-03-06T16:53:04.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:04.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:04.660108+0000) 2024-03-06T16:53:05.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:05.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:05.660300+0000) 2024-03-06T16:53:06.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:06.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:06.660508+0000) 2024-03-06T16:53:07.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:07.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:07.660746+0000) 2024-03-06T16:53:08.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:08.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:08.660982+0000) 2024-03-06T16:53:09.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:09.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:09.661243+0000) 2024-03-06T16:53:10.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:10.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:10.661443+0000) 2024-03-06T16:53:11.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:11.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:11.661648+0000) 2024-03-06T16:53:12.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:12.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:12.661882+0000) 2024-03-06T16:53:13.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:13.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:13.662141+0000) 2024-03-06T16:53:14.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:14.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:14.662328+0000) 2024-03-06T16:53:15.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:15.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:15.662539+0000) 2024-03-06T16:53:16.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:16.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:16.662769+0000) 2024-03-06T16:53:17.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:17.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:17.662997+0000) 2024-03-06T16:53:17.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:17.794 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:18.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:18.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:18.663203+0000) 2024-03-06T16:53:19.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:19.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:19.663411+0000) 2024-03-06T16:53:20.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:20.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:20.663657+0000) 2024-03-06T16:53:21.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:21.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:21.663814+0000) 2024-03-06T16:53:22.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:22.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:22.664102+0000) 2024-03-06T16:53:23.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:23.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:23.664324+0000) 2024-03-06T16:53:24.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:24.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:24.664546+0000) 2024-03-06T16:53:25.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:25.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:25.664781+0000) 2024-03-06T16:53:26.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:26.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:26.665066+0000) 2024-03-06T16:53:27.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:27.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:27.665263+0000) 2024-03-06T16:53:28.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:28.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:28.665470+0000) 2024-03-06T16:53:29.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:29.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:29.665698+0000) 2024-03-06T16:53:30.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:30.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:30.665925+0000) 2024-03-06T16:53:31.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:31.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:31.666142+0000) 2024-03-06T16:53:32.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:32.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:32.666406+0000) 2024-03-06T16:53:33.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:33.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:33.666628+0000) 2024-03-06T16:53:34.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:34.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:34.666846+0000) 2024-03-06T16:53:35.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:35.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:35.667104+0000) 2024-03-06T16:53:36.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:36.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:36.667298+0000) 2024-03-06T16:53:37.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:37.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:37.667455+0000) 2024-03-06T16:53:38.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:38.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:38.667650+0000) 2024-03-06T16:53:39.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:39.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:39.667876+0000) 2024-03-06T16:53:40.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:40.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:40.668130+0000) 2024-03-06T16:53:41.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:41.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:41.668326+0000) 2024-03-06T16:53:42.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:42.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:42.668510+0000) 2024-03-06T16:53:43.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:43.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:43.668706+0000) 2024-03-06T16:53:44.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:44.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:44.668961+0000) 2024-03-06T16:53:45.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:45.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:45.669203+0000) 2024-03-06T16:53:46.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:46.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:46.669418+0000) 2024-03-06T16:53:47.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:47.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:47.669655+0000) 2024-03-06T16:53:47.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:47.943 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:53:48.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:48.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:48.669847+0000) 2024-03-06T16:53:49.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:49.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:49.670071+0000) 2024-03-06T16:53:50.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:50.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:50.670295+0000) 2024-03-06T16:53:51.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:51.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:51.670500+0000) 2024-03-06T16:53:52.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:52.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:52.670682+0000) 2024-03-06T16:53:53.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:53.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:53.670900+0000) 2024-03-06T16:53:54.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:54.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:54.671157+0000) 2024-03-06T16:53:55.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:55.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:55.671406+0000) 2024-03-06T16:53:56.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:56.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:56.671626+0000) 2024-03-06T16:53:57.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:57.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:57.671818+0000) 2024-03-06T16:53:58.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:58.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:58.672037+0000) 2024-03-06T16:53:59.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:53:59.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:53:59.672246+0000) 2024-03-06T16:54:00.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:00.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:00.672499+0000) 2024-03-06T16:54:01.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:01.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:01.672701+0000) 2024-03-06T16:54:02.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:02.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:02.672960+0000) 2024-03-06T16:54:03.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:03.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:03.673243+0000) 2024-03-06T16:54:04.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:04.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:04.673501+0000) 2024-03-06T16:54:05.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:05.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:05.673787+0000) 2024-03-06T16:54:06.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:06.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:06.674098+0000) 2024-03-06T16:54:07.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:07.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:07.674349+0000) 2024-03-06T16:54:08.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:08.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:08.674583+0000) 2024-03-06T16:54:09.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:09.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:09.674844+0000) 2024-03-06T16:54:10.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:10.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:10.675153+0000) 2024-03-06T16:54:11.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:11.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:11.675429+0000) 2024-03-06T16:54:12.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:12.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:12.675701+0000) 2024-03-06T16:54:13.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:13.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:13.675979+0000) 2024-03-06T16:54:14.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:14.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:14.676273+0000) 2024-03-06T16:54:15.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:15.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:15.676558+0000) 2024-03-06T16:54:16.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:16.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:16.676848+0000) 2024-03-06T16:54:17.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:17.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:17.677166+0000) 2024-03-06T16:54:18.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:18.085 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:18.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:18.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:18.677449+0000) 2024-03-06T16:54:19.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:19.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:19.677723+0000) 2024-03-06T16:54:20.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:20.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:20.677943+0000) 2024-03-06T16:54:21.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:21.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:21.678154+0000) 2024-03-06T16:54:22.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:22.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:22.678352+0000) 2024-03-06T16:54:23.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:23.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:23.678569+0000) 2024-03-06T16:54:24.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:24.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:24.678814+0000) 2024-03-06T16:54:25.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:25.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:25.679120+0000) 2024-03-06T16:54:26.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:26.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:26.679341+0000) 2024-03-06T16:54:27.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:27.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:27.679570+0000) 2024-03-06T16:54:28.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:28.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:28.679790+0000) 2024-03-06T16:54:29.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:29.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:29.679969+0000) 2024-03-06T16:54:30.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:30.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:30.680165+0000) 2024-03-06T16:54:31.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:31.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:31.680397+0000) 2024-03-06T16:54:32.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:32.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:32.680602+0000) 2024-03-06T16:54:33.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:33.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:33.680830+0000) 2024-03-06T16:54:34.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:34.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:34.681076+0000) 2024-03-06T16:54:35.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:35.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:35.681277+0000) 2024-03-06T16:54:36.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:36.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:36.681488+0000) 2024-03-06T16:54:37.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:37.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:37.681713+0000) 2024-03-06T16:54:38.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:38.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:38.681982+0000) 2024-03-06T16:54:39.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:39.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:39.682255+0000) 2024-03-06T16:54:40.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:40.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:40.682475+0000) 2024-03-06T16:54:41.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:41.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:41.682695+0000) 2024-03-06T16:54:42.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:42.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:42.682962+0000) 2024-03-06T16:54:43.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:43.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:43.683203+0000) 2024-03-06T16:54:44.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:44.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:44.683397+0000) 2024-03-06T16:54:45.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:45.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:45.683616+0000) 2024-03-06T16:54:46.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:46.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:46.683849+0000) 2024-03-06T16:54:47.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:47.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:47.684100+0000) 2024-03-06T16:54:48.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:48.239 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:54:48.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:48.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:48.684326+0000) 2024-03-06T16:54:49.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:49.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:49.684547+0000) 2024-03-06T16:54:50.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:50.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:50.684795+0000) 2024-03-06T16:54:51.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:51.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:51.685077+0000) 2024-03-06T16:54:52.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:52.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:52.685288+0000) 2024-03-06T16:54:53.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:53.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:53.685498+0000) 2024-03-06T16:54:54.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:54.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:54.685722+0000) 2024-03-06T16:54:55.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:55.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:55.685947+0000) 2024-03-06T16:54:56.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:56.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:56.686173+0000) 2024-03-06T16:54:57.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:57.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:57.686361+0000) 2024-03-06T16:54:58.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:58.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:58.686552+0000) 2024-03-06T16:54:59.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:54:59.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:54:59.686786+0000) 2024-03-06T16:55:00.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:00.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:00.687035+0000) 2024-03-06T16:55:01.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:01.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:01.687227+0000) 2024-03-06T16:55:02.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:02.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:02.687447+0000) 2024-03-06T16:55:03.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:03.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:03.687695+0000) 2024-03-06T16:55:04.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:04.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:04.687949+0000) 2024-03-06T16:55:05.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:05.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:05.688177+0000) 2024-03-06T16:55:06.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:06.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:06.688374+0000) 2024-03-06T16:55:07.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:07.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:07.688557+0000) 2024-03-06T16:55:08.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:08.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:08.688776+0000) 2024-03-06T16:55:09.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:09.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:09.689001+0000) 2024-03-06T16:55:10.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:10.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:10.689239+0000) 2024-03-06T16:55:11.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:11.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:11.689436+0000) 2024-03-06T16:55:12.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:12.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:12.689612+0000) 2024-03-06T16:55:13.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:13.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:13.689817+0000) 2024-03-06T16:55:14.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:14.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:14.690110+0000) 2024-03-06T16:55:15.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:15.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:15.690299+0000) 2024-03-06T16:55:16.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:16.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:16.690527+0000) 2024-03-06T16:55:17.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:17.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:17.690754+0000) 2024-03-06T16:55:18.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:18.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:18.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:18.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:18.690972+0000) 2024-03-06T16:55:19.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:19.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:19.691171+0000) 2024-03-06T16:55:20.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:20.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:20.691359+0000) 2024-03-06T16:55:21.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:21.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:21.691582+0000) 2024-03-06T16:55:22.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:22.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:22.691792+0000) 2024-03-06T16:55:23.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:23.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:23.691990+0000) 2024-03-06T16:55:24.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:24.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:24.692238+0000) 2024-03-06T16:55:25.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:25.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:25.692462+0000) 2024-03-06T16:55:26.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:26.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:26.692740+0000) 2024-03-06T16:55:27.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:27.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:27.692965+0000) 2024-03-06T16:55:28.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:28.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:28.693150+0000) 2024-03-06T16:55:29.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:29.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:29.693338+0000) 2024-03-06T16:55:30.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:30.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:30.693561+0000) 2024-03-06T16:55:31.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:31.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:31.693776+0000) 2024-03-06T16:55:32.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:32.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:32.693991+0000) 2024-03-06T16:55:33.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:33.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:33.694257+0000) 2024-03-06T16:55:34.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:34.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:34.694465+0000) 2024-03-06T16:55:35.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:35.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:35.694679+0000) 2024-03-06T16:55:36.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:36.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:36.694940+0000) 2024-03-06T16:55:37.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:37.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:37.695182+0000) 2024-03-06T16:55:38.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:38.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:38.695363+0000) 2024-03-06T16:55:39.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:39.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:39.695569+0000) 2024-03-06T16:55:40.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:40.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:40.695791+0000) 2024-03-06T16:55:41.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:41.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:41.696037+0000) 2024-03-06T16:55:42.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:42.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:42.696233+0000) 2024-03-06T16:55:43.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:43.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:43.696417+0000) 2024-03-06T16:55:44.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:44.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:44.696636+0000) 2024-03-06T16:55:45.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:45.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:45.696853+0000) 2024-03-06T16:55:46.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:46.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:46.697093+0000) 2024-03-06T16:55:47.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:47.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:47.697249+0000) 2024-03-06T16:55:48.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:48.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:55:48.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:48.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:48.697491+0000) 2024-03-06T16:55:49.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:49.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:49.697743+0000) 2024-03-06T16:55:50.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:50.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:50.697979+0000) 2024-03-06T16:55:51.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:51.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:51.698266+0000) 2024-03-06T16:55:52.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:52.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:52.698471+0000) 2024-03-06T16:55:53.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:53.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:53.698699+0000) 2024-03-06T16:55:54.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:54.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:54.698915+0000) 2024-03-06T16:55:55.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:55.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:55.699136+0000) 2024-03-06T16:55:56.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:56.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:56.699337+0000) 2024-03-06T16:55:57.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:57.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:57.699564+0000) 2024-03-06T16:55:58.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:58.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:58.699773+0000) 2024-03-06T16:55:59.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:55:59.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:55:59.699987+0000) 2024-03-06T16:56:00.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:00.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:00.700204+0000) 2024-03-06T16:56:01.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:01.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:01.700395+0000) 2024-03-06T16:56:02.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:02.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:02.700641+0000) 2024-03-06T16:56:03.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:03.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:03.700880+0000) 2024-03-06T16:56:04.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:04.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:04.701160+0000) 2024-03-06T16:56:05.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:05.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:05.701407+0000) 2024-03-06T16:56:06.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:06.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:06.701670+0000) 2024-03-06T16:56:07.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:07.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:07.701906+0000) 2024-03-06T16:56:08.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:08.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:08.702173+0000) 2024-03-06T16:56:09.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:09.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:09.702387+0000) 2024-03-06T16:56:10.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:10.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:10.702574+0000) 2024-03-06T16:56:11.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:11.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:11.702763+0000) 2024-03-06T16:56:12.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:12.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:12.702982+0000) 2024-03-06T16:56:13.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:13.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:13.703251+0000) 2024-03-06T16:56:14.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:14.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:14.703532+0000) 2024-03-06T16:56:15.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:15.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:15.703754+0000) 2024-03-06T16:56:16.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:16.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:16.703980+0000) 2024-03-06T16:56:17.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:17.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:17.704214+0000) 2024-03-06T16:56:18.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:18.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:18.704399+0000) 2024-03-06T16:56:18.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:18.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:19.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:19.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:19.704609+0000) 2024-03-06T16:56:20.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:20.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:20.704834+0000) 2024-03-06T16:56:21.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:21.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:21.705121+0000) 2024-03-06T16:56:22.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:22.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:22.705355+0000) 2024-03-06T16:56:23.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:23.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:23.705586+0000) 2024-03-06T16:56:24.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:24.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:24.705826+0000) 2024-03-06T16:56:25.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:25.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:25.706129+0000) 2024-03-06T16:56:26.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:26.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:26.706401+0000) 2024-03-06T16:56:27.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:27.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:27.706625+0000) 2024-03-06T16:56:28.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:28.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:28.706813+0000) 2024-03-06T16:56:29.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:29.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:29.707072+0000) 2024-03-06T16:56:30.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:30.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:30.707297+0000) 2024-03-06T16:56:31.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:31.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:31.707527+0000) 2024-03-06T16:56:32.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:32.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:32.707753+0000) 2024-03-06T16:56:33.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:33.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:33.707970+0000) 2024-03-06T16:56:34.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:34.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:34.708197+0000) 2024-03-06T16:56:35.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:35.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:35.708424+0000) 2024-03-06T16:56:36.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:36.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:36.708669+0000) 2024-03-06T16:56:37.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:37.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:37.708909+0000) 2024-03-06T16:56:38.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:38.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:38.709136+0000) 2024-03-06T16:56:39.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:39.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:39.709363+0000) 2024-03-06T16:56:40.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:40.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:40.709589+0000) 2024-03-06T16:56:41.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:41.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:41.709820+0000) 2024-03-06T16:56:42.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:42.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:42.710099+0000) 2024-03-06T16:56:43.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:43.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:43.710333+0000) 2024-03-06T16:56:44.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:44.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:44.710553+0000) 2024-03-06T16:56:45.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:45.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:45.710796+0000) 2024-03-06T16:56:46.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:46.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:46.711071+0000) 2024-03-06T16:56:47.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:47.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:47.711354+0000) 2024-03-06T16:56:48.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:48.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:48.711592+0000) 2024-03-06T16:56:48.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:48.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:56:49.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:49.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:49.711815+0000) 2024-03-06T16:56:50.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:50.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:50.712029+0000) 2024-03-06T16:56:51.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:51.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:51.712224+0000) 2024-03-06T16:56:52.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:52.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:52.712413+0000) 2024-03-06T16:56:53.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:53.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:53.712653+0000) 2024-03-06T16:56:54.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:54.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:54.712886+0000) 2024-03-06T16:56:55.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:55.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:55.713159+0000) 2024-03-06T16:56:56.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:56.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:56.713395+0000) 2024-03-06T16:56:57.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:57.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:57.713595+0000) 2024-03-06T16:56:58.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:58.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:58.713826+0000) 2024-03-06T16:56:59.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:56:59.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:56:59.714149+0000) 2024-03-06T16:57:00.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:00.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:00.714359+0000) 2024-03-06T16:57:01.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:01.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:01.714590+0000) 2024-03-06T16:57:02.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:02.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:02.714797+0000) 2024-03-06T16:57:03.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:03.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:03.714979+0000) 2024-03-06T16:57:04.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:04.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:04.715232+0000) 2024-03-06T16:57:05.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:05.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:05.715473+0000) 2024-03-06T16:57:06.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:06.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:06.715695+0000) 2024-03-06T16:57:07.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:07.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:07.715950+0000) 2024-03-06T16:57:08.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:08.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:08.716230+0000) 2024-03-06T16:57:09.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:09.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:09.716423+0000) 2024-03-06T16:57:10.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:10.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:10.716652+0000) 2024-03-06T16:57:11.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:11.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:11.716883+0000) 2024-03-06T16:57:12.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:12.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:12.717123+0000) 2024-03-06T16:57:13.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:13.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:13.717368+0000) 2024-03-06T16:57:14.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:14.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:14.717613+0000) 2024-03-06T16:57:15.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:15.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:15.717866+0000) 2024-03-06T16:57:16.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:16.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:16.718127+0000) 2024-03-06T16:57:17.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:17.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:17.718334+0000) 2024-03-06T16:57:18.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:18.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:18.718582+0000) 2024-03-06T16:57:19.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:19.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:19.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:19.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:19.718812+0000) 2024-03-06T16:57:20.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:20.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:20.719065+0000) 2024-03-06T16:57:21.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:21.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:21.719225+0000) 2024-03-06T16:57:22.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:22.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:22.719444+0000) 2024-03-06T16:57:23.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:23.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:23.719676+0000) 2024-03-06T16:57:24.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:24.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:24.719983+0000) 2024-03-06T16:57:25.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:25.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:25.720241+0000) 2024-03-06T16:57:26.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:26.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:26.720428+0000) 2024-03-06T16:57:27.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:27.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:27.720651+0000) 2024-03-06T16:57:28.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:28.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:28.720872+0000) 2024-03-06T16:57:29.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:29.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:29.721127+0000) 2024-03-06T16:57:30.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:30.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:30.721280+0000) 2024-03-06T16:57:31.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:31.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:31.721546+0000) 2024-03-06T16:57:32.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:32.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:32.721828+0000) 2024-03-06T16:57:33.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:33.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:33.722115+0000) 2024-03-06T16:57:34.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:34.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:34.722293+0000) 2024-03-06T16:57:35.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:35.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:35.722497+0000) 2024-03-06T16:57:36.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:36.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:36.722741+0000) 2024-03-06T16:57:37.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:37.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:37.722972+0000) 2024-03-06T16:57:38.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:38.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:38.723200+0000) 2024-03-06T16:57:39.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:39.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:39.723374+0000) 2024-03-06T16:57:40.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:40.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:40.723738+0000) 2024-03-06T16:57:41.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:41.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:41.723992+0000) 2024-03-06T16:57:42.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:42.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:42.724251+0000) 2024-03-06T16:57:43.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:43.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:43.724479+0000) 2024-03-06T16:57:44.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:44.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:44.724706+0000) 2024-03-06T16:57:45.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:45.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:45.724932+0000) 2024-03-06T16:57:46.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:46.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:46.725192+0000) 2024-03-06T16:57:47.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:47.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:47.725397+0000) 2024-03-06T16:57:48.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:48.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:48.725650+0000) 2024-03-06T16:57:49.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:49.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:57:49.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:49.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:49.725913+0000) 2024-03-06T16:57:50.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:50.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:50.726137+0000) 2024-03-06T16:57:51.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:51.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:51.726384+0000) 2024-03-06T16:57:52.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:52.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:52.726655+0000) 2024-03-06T16:57:53.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:53.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:53.726936+0000) 2024-03-06T16:57:54.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:54.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:54.727171+0000) 2024-03-06T16:57:55.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:55.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:55.727509+0000) 2024-03-06T16:57:56.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:56.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:56.727774+0000) 2024-03-06T16:57:57.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:57.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:57.728052+0000) 2024-03-06T16:57:58.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:58.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:58.728336+0000) 2024-03-06T16:57:59.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:57:59.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:57:59.728616+0000) 2024-03-06T16:58:00.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:00.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:00.728894+0000) 2024-03-06T16:58:01.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:01.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:01.729126+0000) 2024-03-06T16:58:02.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:02.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:02.729345+0000) 2024-03-06T16:58:03.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:03.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:03.729607+0000) 2024-03-06T16:58:04.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:04.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:04.729854+0000) 2024-03-06T16:58:05.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:05.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:05.730149+0000) 2024-03-06T16:58:06.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:06.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:06.730358+0000) 2024-03-06T16:58:07.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:07.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:07.730580+0000) 2024-03-06T16:58:08.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:08.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:08.730801+0000) 2024-03-06T16:58:09.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:09.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:09.731065+0000) 2024-03-06T16:58:10.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:10.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:10.731292+0000) 2024-03-06T16:58:11.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:11.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:11.731549+0000) 2024-03-06T16:58:12.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:12.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:12.731783+0000) 2024-03-06T16:58:13.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:13.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:13.732115+0000) 2024-03-06T16:58:14.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:14.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:14.732377+0000) 2024-03-06T16:58:15.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:15.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:15.732653+0000) 2024-03-06T16:58:16.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:16.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:16.732960+0000) 2024-03-06T16:58:17.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:17.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:17.733244+0000) 2024-03-06T16:58:18.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:18.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:18.733471+0000) 2024-03-06T16:58:19.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:19.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:19.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:19.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:19.733743+0000) 2024-03-06T16:58:20.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:20.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:20.733981+0000) 2024-03-06T16:58:21.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:21.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:21.734308+0000) 2024-03-06T16:58:22.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:22.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:22.734550+0000) 2024-03-06T16:58:23.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:23.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:23.734772+0000) 2024-03-06T16:58:24.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:24.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:24.734996+0000) 2024-03-06T16:58:25.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:25.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:25.735249+0000) 2024-03-06T16:58:26.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:26.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:26.735516+0000) 2024-03-06T16:58:27.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:27.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:27.735746+0000) 2024-03-06T16:58:28.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:28.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:28.735983+0000) 2024-03-06T16:58:29.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:29.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:29.736252+0000) 2024-03-06T16:58:30.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:30.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:30.736454+0000) 2024-03-06T16:58:31.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:31.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:31.736660+0000) 2024-03-06T16:58:32.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:32.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:32.736911+0000) 2024-03-06T16:58:33.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:33.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:33.737166+0000) 2024-03-06T16:58:34.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:34.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:34.737412+0000) 2024-03-06T16:58:35.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:35.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:35.737639+0000) 2024-03-06T16:58:36.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:36.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:36.737921+0000) 2024-03-06T16:58:37.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:37.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:37.738189+0000) 2024-03-06T16:58:38.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:38.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:38.738433+0000) 2024-03-06T16:58:39.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:39.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:39.738689+0000) 2024-03-06T16:58:40.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:40.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:40.738940+0000) 2024-03-06T16:58:41.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:41.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:41.739191+0000) 2024-03-06T16:58:42.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:42.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:42.739435+0000) 2024-03-06T16:58:43.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:43.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:43.739692+0000) 2024-03-06T16:58:44.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:44.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:44.739908+0000) 2024-03-06T16:58:45.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:45.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:45.740153+0000) 2024-03-06T16:58:46.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:46.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:46.740398+0000) 2024-03-06T16:58:47.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:47.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:47.740622+0000) 2024-03-06T16:58:48.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:48.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:48.740832+0000) 2024-03-06T16:58:49.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:49.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:49.741118+0000) 2024-03-06T16:58:49.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:49.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:58:50.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:50.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:50.741319+0000) 2024-03-06T16:58:51.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:51.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:51.741553+0000) 2024-03-06T16:58:52.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:52.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:52.741723+0000) 2024-03-06T16:58:53.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:53.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:53.741949+0000) 2024-03-06T16:58:54.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:54.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:54.742174+0000) 2024-03-06T16:58:55.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:55.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:55.742397+0000) 2024-03-06T16:58:56.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:56.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:56.742640+0000) 2024-03-06T16:58:57.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:57.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:57.742879+0000) 2024-03-06T16:58:58.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:58.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:58.743141+0000) 2024-03-06T16:58:59.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:58:59.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:58:59.743369+0000) 2024-03-06T16:59:00.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:00.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:00.743560+0000) 2024-03-06T16:59:01.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:01.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:01.743796+0000) 2024-03-06T16:59:02.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:02.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:02.744381+0000) 2024-03-06T16:59:03.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:03.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:03.744602+0000) 2024-03-06T16:59:04.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:04.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:04.744821+0000) 2024-03-06T16:59:05.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:05.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:05.745078+0000) 2024-03-06T16:59:06.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:06.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:06.745286+0000) 2024-03-06T16:59:07.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:07.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:07.745493+0000) 2024-03-06T16:59:08.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:08.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:08.745689+0000) 2024-03-06T16:59:09.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:09.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:09.745909+0000) 2024-03-06T16:59:10.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:10.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:10.746144+0000) 2024-03-06T16:59:11.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:11.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:11.746340+0000) 2024-03-06T16:59:12.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:12.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:12.746543+0000) 2024-03-06T16:59:13.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:13.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:13.746743+0000) 2024-03-06T16:59:14.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:14.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:14.746960+0000) 2024-03-06T16:59:15.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:15.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:15.747179+0000) 2024-03-06T16:59:16.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:16.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:16.747394+0000) 2024-03-06T16:59:17.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:17.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:17.747615+0000) 2024-03-06T16:59:18.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:18.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:18.747831+0000) 2024-03-06T16:59:19.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:19.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:19.748118+0000) 2024-03-06T16:59:19.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:19.901 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:20.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:20.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:20.748339+0000) 2024-03-06T16:59:21.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:21.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:21.748552+0000) 2024-03-06T16:59:22.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:22.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:22.748805+0000) 2024-03-06T16:59:23.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:23.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:23.749063+0000) 2024-03-06T16:59:24.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:24.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:24.749274+0000) 2024-03-06T16:59:25.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:25.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:25.749496+0000) 2024-03-06T16:59:26.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:26.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:26.749745+0000) 2024-03-06T16:59:27.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:27.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:27.749988+0000) 2024-03-06T16:59:28.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:28.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:28.750239+0000) 2024-03-06T16:59:29.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:29.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:29.750426+0000) 2024-03-06T16:59:30.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:30.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:30.750624+0000) 2024-03-06T16:59:31.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:31.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:31.750820+0000) 2024-03-06T16:59:32.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:32.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:32.751065+0000) 2024-03-06T16:59:33.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:33.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:33.751289+0000) 2024-03-06T16:59:34.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:34.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:34.751464+0000) 2024-03-06T16:59:35.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:35.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:35.751687+0000) 2024-03-06T16:59:36.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:36.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:36.751948+0000) 2024-03-06T16:59:37.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:37.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:37.752171+0000) 2024-03-06T16:59:38.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:38.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:38.752361+0000) 2024-03-06T16:59:39.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:39.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:39.752538+0000) 2024-03-06T16:59:40.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:40.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:40.752718+0000) 2024-03-06T16:59:41.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:41.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:41.752951+0000) 2024-03-06T16:59:42.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:42.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:42.753173+0000) 2024-03-06T16:59:43.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:43.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:43.753409+0000) 2024-03-06T16:59:44.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:44.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:44.753697+0000) 2024-03-06T16:59:45.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:45.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:45.753981+0000) 2024-03-06T16:59:46.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:46.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:46.754243+0000) 2024-03-06T16:59:47.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:47.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:47.754453+0000) 2024-03-06T16:59:48.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:48.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:48.754663+0000) 2024-03-06T16:59:49.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:49.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:49.754883+0000) 2024-03-06T16:59:50.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:50.049 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T16:59:50.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:50.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:50.755112+0000) 2024-03-06T16:59:51.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:51.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:51.755313+0000) 2024-03-06T16:59:52.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:52.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:52.755531+0000) 2024-03-06T16:59:53.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:53.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:53.755737+0000) 2024-03-06T16:59:54.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:54.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:54.755969+0000) 2024-03-06T16:59:55.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:55.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:55.756165+0000) 2024-03-06T16:59:56.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:56.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:56.756359+0000) 2024-03-06T16:59:57.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:57.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:57.756584+0000) 2024-03-06T16:59:58.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:58.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:58.756741+0000) 2024-03-06T16:59:59.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T16:59:59.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T15:59:59.756955+0000) 2024-03-06T17:00:00.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:00.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:00.757166+0000) 2024-03-06T17:00:01.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:01.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:01.757375+0000) 2024-03-06T17:00:02.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:02.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:02.757589+0000) 2024-03-06T17:00:03.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:03.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:03.757787+0000) 2024-03-06T17:00:04.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:04.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:04.758051+0000) 2024-03-06T17:00:05.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:05.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:05.758240+0000) 2024-03-06T17:00:06.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:06.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:06.758470+0000) 2024-03-06T17:00:07.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:07.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:07.758707+0000) 2024-03-06T17:00:08.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:08.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:08.758935+0000) 2024-03-06T17:00:09.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:09.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:09.759163+0000) 2024-03-06T17:00:10.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:10.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:10.759380+0000) 2024-03-06T17:00:11.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:11.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:11.759655+0000) 2024-03-06T17:00:12.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:12.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:12.759942+0000) 2024-03-06T17:00:13.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:13.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:13.760179+0000) 2024-03-06T17:00:14.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:14.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:14.760403+0000) 2024-03-06T17:00:15.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:15.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:15.760609+0000) 2024-03-06T17:00:16.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:16.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:16.760847+0000) 2024-03-06T17:00:17.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:17.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:17.761065+0000) 2024-03-06T17:00:18.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:18.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:18.761255+0000) 2024-03-06T17:00:19.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:19.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:19.761480+0000) 2024-03-06T17:00:20.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:20.285 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:20.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:20.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:20.761678+0000) 2024-03-06T17:00:21.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:21.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:21.761873+0000) 2024-03-06T17:00:22.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:22.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:22.762094+0000) 2024-03-06T17:00:23.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:23.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:23.762301+0000) 2024-03-06T17:00:24.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:24.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:24.762499+0000) 2024-03-06T17:00:25.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:25.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:25.762718+0000) 2024-03-06T17:00:26.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:26.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:26.762938+0000) 2024-03-06T17:00:27.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:27.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:27.763162+0000) 2024-03-06T17:00:28.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:28.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:28.763428+0000) 2024-03-06T17:00:29.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:29.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:29.763692+0000) 2024-03-06T17:00:30.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:30.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:30.763971+0000) 2024-03-06T17:00:31.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:31.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:31.764224+0000) 2024-03-06T17:00:32.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:32.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:32.764448+0000) 2024-03-06T17:00:33.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:33.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:33.764634+0000) 2024-03-06T17:00:34.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:34.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:34.764855+0000) 2024-03-06T17:00:35.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:35.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:35.765106+0000) 2024-03-06T17:00:36.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:36.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:36.765313+0000) 2024-03-06T17:00:37.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:37.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:37.765513+0000) 2024-03-06T17:00:38.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:38.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:38.765730+0000) 2024-03-06T17:00:39.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:39.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:39.765980+0000) 2024-03-06T17:00:40.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:40.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:40.766213+0000) 2024-03-06T17:00:41.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:41.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:41.766410+0000) 2024-03-06T17:00:42.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:42.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:42.766611+0000) 2024-03-06T17:00:43.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:43.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:43.766811+0000) 2024-03-06T17:00:44.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:44.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:44.767069+0000) 2024-03-06T17:00:45.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:45.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:45.767271+0000) 2024-03-06T17:00:46.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:46.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:46.767504+0000) 2024-03-06T17:00:47.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:47.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:47.767773+0000) 2024-03-06T17:00:48.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:48.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:48.768048+0000) 2024-03-06T17:00:49.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:49.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:49.768274+0000) 2024-03-06T17:00:50.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:50.514 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:00:50.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:50.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:50.768501+0000) 2024-03-06T17:00:51.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:51.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:51.768693+0000) 2024-03-06T17:00:52.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:52.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:52.768895+0000) 2024-03-06T17:00:53.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:53.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:53.769124+0000) 2024-03-06T17:00:54.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:54.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:54.769313+0000) 2024-03-06T17:00:55.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:55.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:55.769516+0000) 2024-03-06T17:00:56.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:56.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:56.769744+0000) 2024-03-06T17:00:57.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:57.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:57.769970+0000) 2024-03-06T17:00:58.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:58.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:58.770152+0000) 2024-03-06T17:00:59.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:00:59.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:00:59.770382+0000) 2024-03-06T17:01:00.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:00.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:00.770604+0000) 2024-03-06T17:01:01.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:01.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:01.770837+0000) 2024-03-06T17:01:02.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:02.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:02.771094+0000) 2024-03-06T17:01:03.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:03.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:03.771288+0000) 2024-03-06T17:01:04.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:04.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:04.771489+0000) 2024-03-06T17:01:05.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:05.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:05.771707+0000) 2024-03-06T17:01:06.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:06.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:06.772001+0000) 2024-03-06T17:01:07.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:07.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:07.772303+0000) 2024-03-06T17:01:08.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:08.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:08.772531+0000) 2024-03-06T17:01:09.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:09.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:09.772705+0000) 2024-03-06T17:01:10.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:10.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:10.772924+0000) 2024-03-06T17:01:11.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:11.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:11.773154+0000) 2024-03-06T17:01:12.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:12.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:12.773354+0000) 2024-03-06T17:01:13.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:13.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:13.773571+0000) 2024-03-06T17:01:14.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:14.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:14.773758+0000) 2024-03-06T17:01:15.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:15.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:15.773987+0000) 2024-03-06T17:01:16.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:16.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:16.774232+0000) 2024-03-06T17:01:17.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:17.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:17.774441+0000) 2024-03-06T17:01:18.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:18.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:18.774630+0000) 2024-03-06T17:01:19.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:19.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:19.774855+0000) 2024-03-06T17:01:20.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:20.661 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:20.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:20.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:20.775127+0000) 2024-03-06T17:01:21.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:21.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:21.775364+0000) 2024-03-06T17:01:22.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:22.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:22.775556+0000) 2024-03-06T17:01:23.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:23.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:23.775844+0000) 2024-03-06T17:01:24.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:24.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:24.776126+0000) 2024-03-06T17:01:25.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:25.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:25.776346+0000) 2024-03-06T17:01:26.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:26.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:26.776564+0000) 2024-03-06T17:01:27.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:27.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:27.776769+0000) 2024-03-06T17:01:28.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:28.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:28.777069+0000) 2024-03-06T17:01:29.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:29.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:29.777311+0000) 2024-03-06T17:01:30.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:30.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:30.777510+0000) 2024-03-06T17:01:31.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:31.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:31.777691+0000) 2024-03-06T17:01:32.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:32.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:32.777912+0000) 2024-03-06T17:01:33.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:33.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:33.778149+0000) 2024-03-06T17:01:34.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:34.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:34.778397+0000) 2024-03-06T17:01:35.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:35.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:35.778662+0000) 2024-03-06T17:01:36.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:36.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:36.778955+0000) 2024-03-06T17:01:37.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:37.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:37.779216+0000) 2024-03-06T17:01:38.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:38.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:38.779454+0000) 2024-03-06T17:01:39.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:39.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:39.779693+0000) 2024-03-06T17:01:40.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:40.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:40.779906+0000) 2024-03-06T17:01:41.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:41.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:41.780121+0000) 2024-03-06T17:01:42.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:42.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:42.780350+0000) 2024-03-06T17:01:43.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:43.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:43.780562+0000) 2024-03-06T17:01:44.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:44.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:44.780749+0000) 2024-03-06T17:01:45.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:45.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:45.780991+0000) 2024-03-06T17:01:46.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:46.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:46.781262+0000) 2024-03-06T17:01:47.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:47.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:47.781496+0000) 2024-03-06T17:01:48.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:48.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:48.781708+0000) 2024-03-06T17:01:49.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:49.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:49.781960+0000) 2024-03-06T17:01:50.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:50.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:50.782153+0000) 2024-03-06T17:01:50.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:50.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:01:51.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:51.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:51.782355+0000) 2024-03-06T17:01:52.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:52.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:52.782565+0000) 2024-03-06T17:01:53.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:53.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:53.782784+0000) 2024-03-06T17:01:54.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:54.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:54.783094+0000) 2024-03-06T17:01:55.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:55.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:55.783337+0000) 2024-03-06T17:01:56.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:56.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:56.783556+0000) 2024-03-06T17:01:57.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:57.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:57.783789+0000) 2024-03-06T17:01:58.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:58.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:58.784048+0000) 2024-03-06T17:01:59.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:01:59.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:01:59.784237+0000) 2024-03-06T17:02:00.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:00.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:00.784469+0000) 2024-03-06T17:02:01.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:01.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:01.784688+0000) 2024-03-06T17:02:02.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:02.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:02.784915+0000) 2024-03-06T17:02:03.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:03.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:03.785148+0000) 2024-03-06T17:02:04.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:04.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:04.785353+0000) 2024-03-06T17:02:05.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:05.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:05.785558+0000) 2024-03-06T17:02:06.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:06.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:06.785813+0000) 2024-03-06T17:02:07.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:07.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:07.786075+0000) 2024-03-06T17:02:08.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:08.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:08.786263+0000) 2024-03-06T17:02:09.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:09.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:09.786510+0000) 2024-03-06T17:02:10.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:10.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:10.786749+0000) 2024-03-06T17:02:11.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:11.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:11.786988+0000) 2024-03-06T17:02:12.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:12.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:12.787184+0000) 2024-03-06T17:02:13.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:13.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:13.787376+0000) 2024-03-06T17:02:14.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:14.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:14.787616+0000) 2024-03-06T17:02:15.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:15.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:15.787860+0000) 2024-03-06T17:02:16.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:16.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:16.788166+0000) 2024-03-06T17:02:17.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:17.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:17.788416+0000) 2024-03-06T17:02:18.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:18.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:18.788641+0000) 2024-03-06T17:02:19.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:19.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:19.788922+0000) 2024-03-06T17:02:20.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:20.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:20.789161+0000) 2024-03-06T17:02:21.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:21.043 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:21.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:21.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:21.789391+0000) 2024-03-06T17:02:22.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:22.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:22.789618+0000) 2024-03-06T17:02:23.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:23.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:23.789842+0000) 2024-03-06T17:02:24.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:24.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:24.790108+0000) 2024-03-06T17:02:25.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:25.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:25.790337+0000) 2024-03-06T17:02:26.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:26.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:26.790562+0000) 2024-03-06T17:02:27.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:27.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:27.790800+0000) 2024-03-06T17:02:28.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:28.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:28.791087+0000) 2024-03-06T17:02:29.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:29.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:29.791312+0000) 2024-03-06T17:02:30.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:30.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:30.791530+0000) 2024-03-06T17:02:31.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:31.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:31.791760+0000) 2024-03-06T17:02:32.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:32.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:32.791956+0000) 2024-03-06T17:02:33.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:33.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:33.792175+0000) 2024-03-06T17:02:34.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:34.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:34.792420+0000) 2024-03-06T17:02:35.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:35.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:35.792593+0000) 2024-03-06T17:02:36.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:36.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:36.792813+0000) 2024-03-06T17:02:37.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:37.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:37.792956+0000) 2024-03-06T17:02:38.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:38.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:38.793164+0000) 2024-03-06T17:02:39.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:39.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:39.793318+0000) 2024-03-06T17:02:40.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:40.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:40.793540+0000) 2024-03-06T17:02:41.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:41.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:41.793762+0000) 2024-03-06T17:02:42.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:42.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:42.793940+0000) 2024-03-06T17:02:43.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:43.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:43.794191+0000) 2024-03-06T17:02:44.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:44.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:44.794401+0000) 2024-03-06T17:02:45.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:45.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:45.794643+0000) 2024-03-06T17:02:46.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:46.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:46.794883+0000) 2024-03-06T17:02:47.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:47.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:47.795148+0000) 2024-03-06T17:02:48.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:48.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:48.795374+0000) 2024-03-06T17:02:49.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:49.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:49.795611+0000) 2024-03-06T17:02:50.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:50.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:50.795790+0000) 2024-03-06T17:02:51.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:51.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:02:51.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:51.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:51.796073+0000) 2024-03-06T17:02:52.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:52.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:52.796284+0000) 2024-03-06T17:02:53.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:53.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:53.796520+0000) 2024-03-06T17:02:54.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:54.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:54.796818+0000) 2024-03-06T17:02:55.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:55.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:55.797108+0000) 2024-03-06T17:02:56.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:56.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:56.797370+0000) 2024-03-06T17:02:57.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:57.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:57.797609+0000) 2024-03-06T17:02:58.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:58.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:58.797844+0000) 2024-03-06T17:02:59.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:02:59.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:02:59.798125+0000) 2024-03-06T17:03:00.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:00.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:00.798331+0000) 2024-03-06T17:03:01.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:01.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:01.798533+0000) 2024-03-06T17:03:02.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:02.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:02.798752+0000) 2024-03-06T17:03:03.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:03.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:03.798988+0000) 2024-03-06T17:03:04.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:04.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:04.799258+0000) 2024-03-06T17:03:05.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:05.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:05.799486+0000) 2024-03-06T17:03:06.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:06.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:06.799736+0000) 2024-03-06T17:03:07.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:07.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:07.799986+0000) 2024-03-06T17:03:08.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:08.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:08.800245+0000) 2024-03-06T17:03:09.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:09.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:09.800462+0000) 2024-03-06T17:03:10.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:10.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:10.800698+0000) 2024-03-06T17:03:11.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:11.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:11.800937+0000) 2024-03-06T17:03:12.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:12.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:12.801151+0000) 2024-03-06T17:03:13.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:13.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:13.801357+0000) 2024-03-06T17:03:14.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:14.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:14.801577+0000) 2024-03-06T17:03:15.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:15.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:15.801823+0000) 2024-03-06T17:03:16.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:16.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:16.802105+0000) 2024-03-06T17:03:17.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:17.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:17.802316+0000) 2024-03-06T17:03:18.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:18.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:18.802534+0000) 2024-03-06T17:03:19.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:19.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:19.802745+0000) 2024-03-06T17:03:20.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:20.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:20.802984+0000) 2024-03-06T17:03:21.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:21.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:21.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:21.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:21.803216+0000) 2024-03-06T17:03:22.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:22.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:22.803402+0000) 2024-03-06T17:03:23.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:23.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:23.803640+0000) 2024-03-06T17:03:24.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:24.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:24.803864+0000) 2024-03-06T17:03:25.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:25.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:25.804139+0000) 2024-03-06T17:03:26.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:26.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:26.804371+0000) 2024-03-06T17:03:27.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:27.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:27.804573+0000) 2024-03-06T17:03:28.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:28.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:28.804809+0000) 2024-03-06T17:03:29.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:29.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:29.805069+0000) 2024-03-06T17:03:30.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:30.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:30.805290+0000) 2024-03-06T17:03:31.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:31.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:31.805528+0000) 2024-03-06T17:03:32.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:32.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:32.805754+0000) 2024-03-06T17:03:33.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:33.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:33.805970+0000) 2024-03-06T17:03:34.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:34.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:34.806165+0000) 2024-03-06T17:03:35.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:35.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:35.806369+0000) 2024-03-06T17:03:36.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:36.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:36.806602+0000) 2024-03-06T17:03:37.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:37.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:37.806845+0000) 2024-03-06T17:03:38.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:38.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:38.807072+0000) 2024-03-06T17:03:39.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:39.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:39.807271+0000) 2024-03-06T17:03:40.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:40.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:40.807492+0000) 2024-03-06T17:03:41.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:41.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:41.807713+0000) 2024-03-06T17:03:42.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:42.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:42.807990+0000) 2024-03-06T17:03:43.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:43.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:43.808240+0000) 2024-03-06T17:03:44.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:44.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:44.808474+0000) 2024-03-06T17:03:45.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:45.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:45.808719+0000) 2024-03-06T17:03:46.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:46.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:46.808935+0000) 2024-03-06T17:03:47.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:47.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:47.809188+0000) 2024-03-06T17:03:48.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:48.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:48.809434+0000) 2024-03-06T17:03:49.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:49.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:49.809654+0000) 2024-03-06T17:03:50.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:50.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:50.809886+0000) 2024-03-06T17:03:51.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:51.694 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:03:51.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:51.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:51.810152+0000) 2024-03-06T17:03:52.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:52.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:52.810378+0000) 2024-03-06T17:03:53.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:53.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:53.810582+0000) 2024-03-06T17:03:54.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:54.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:54.810800+0000) 2024-03-06T17:03:55.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:55.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:55.811069+0000) 2024-03-06T17:03:56.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:56.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:56.811274+0000) 2024-03-06T17:03:57.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:57.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:57.811473+0000) 2024-03-06T17:03:58.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:58.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:58.811693+0000) 2024-03-06T17:03:59.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:03:59.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:03:59.811929+0000) 2024-03-06T17:04:00.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:00.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:00.812150+0000) 2024-03-06T17:04:01.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:01.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:01.812358+0000) 2024-03-06T17:04:02.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:02.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:02.812588+0000) 2024-03-06T17:04:03.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:03.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:03.812816+0000) 2024-03-06T17:04:04.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:04.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:04.813099+0000) 2024-03-06T17:04:05.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:05.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:05.813284+0000) 2024-03-06T17:04:06.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:06.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:06.813515+0000) 2024-03-06T17:04:07.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:07.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:07.813784+0000) 2024-03-06T17:04:08.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:08.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:08.814111+0000) 2024-03-06T17:04:09.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:09.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:09.814342+0000) 2024-03-06T17:04:10.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:10.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:10.814540+0000) 2024-03-06T17:04:11.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:11.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:11.814763+0000) 2024-03-06T17:04:12.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:12.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:12.814979+0000) 2024-03-06T17:04:13.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:13.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:13.815312+0000) 2024-03-06T17:04:14.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:14.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:14.815580+0000) 2024-03-06T17:04:15.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:15.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:15.815850+0000) 2024-03-06T17:04:16.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:16.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:16.816166+0000) 2024-03-06T17:04:17.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:17.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:17.816442+0000) 2024-03-06T17:04:18.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:18.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:18.816644+0000) 2024-03-06T17:04:19.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:19.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:19.816883+0000) 2024-03-06T17:04:20.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:20.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:20.817119+0000) 2024-03-06T17:04:21.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:21.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:21.817287+0000) 2024-03-06T17:04:21.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:21.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:22.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:22.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:22.817500+0000) 2024-03-06T17:04:23.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:23.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:23.817762+0000) 2024-03-06T17:04:24.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:24.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:24.818074+0000) 2024-03-06T17:04:25.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:25.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:25.818356+0000) 2024-03-06T17:04:26.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:26.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:26.818653+0000) 2024-03-06T17:04:27.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:27.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:27.818870+0000) 2024-03-06T17:04:28.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:28.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:28.819124+0000) 2024-03-06T17:04:29.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:29.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:29.819314+0000) 2024-03-06T17:04:30.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:30.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:30.819489+0000) 2024-03-06T17:04:31.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:31.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:31.819727+0000) 2024-03-06T17:04:32.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:32.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:32.819963+0000) 2024-03-06T17:04:33.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:33.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:33.820194+0000) 2024-03-06T17:04:34.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:34.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:34.820384+0000) 2024-03-06T17:04:35.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:35.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:35.820578+0000) 2024-03-06T17:04:36.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:36.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:36.820819+0000) 2024-03-06T17:04:37.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:37.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:37.821114+0000) 2024-03-06T17:04:38.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:38.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:38.821325+0000) 2024-03-06T17:04:39.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:39.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:39.821547+0000) 2024-03-06T17:04:40.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:40.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:40.821842+0000) 2024-03-06T17:04:41.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:41.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:41.822122+0000) 2024-03-06T17:04:42.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:42.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:42.822299+0000) 2024-03-06T17:04:43.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:43.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:43.822523+0000) 2024-03-06T17:04:44.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:44.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:44.822751+0000) 2024-03-06T17:04:45.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:45.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:45.822982+0000) 2024-03-06T17:04:46.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:46.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:46.823256+0000) 2024-03-06T17:04:47.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:47.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:47.823521+0000) 2024-03-06T17:04:48.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:48.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:48.823703+0000) 2024-03-06T17:04:49.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:49.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:49.823852+0000) 2024-03-06T17:04:50.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:50.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:50.824124+0000) 2024-03-06T17:04:51.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:51.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:51.824410+0000) 2024-03-06T17:04:51.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:51.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:04:52.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:52.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:52.824695+0000) 2024-03-06T17:04:53.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:53.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:53.824981+0000) 2024-03-06T17:04:54.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:54.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:54.825303+0000) 2024-03-06T17:04:55.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:55.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:55.825504+0000) 2024-03-06T17:04:56.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:56.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:56.825753+0000) 2024-03-06T17:04:57.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:57.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:57.825988+0000) 2024-03-06T17:04:58.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:58.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:58.826277+0000) 2024-03-06T17:04:59.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:04:59.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:04:59.826515+0000) 2024-03-06T17:05:00.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:00.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:00.826712+0000) 2024-03-06T17:05:01.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:01.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:01.826934+0000) 2024-03-06T17:05:02.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:02.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:02.827173+0000) 2024-03-06T17:05:03.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:03.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:03.827395+0000) 2024-03-06T17:05:04.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:04.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:04.827569+0000) 2024-03-06T17:05:05.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:05.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:05.827794+0000) 2024-03-06T17:05:06.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:06.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:06.828068+0000) 2024-03-06T17:05:07.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:07.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:07.828305+0000) 2024-03-06T17:05:08.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:08.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:08.828541+0000) 2024-03-06T17:05:09.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:09.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:09.828769+0000) 2024-03-06T17:05:10.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:10.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:10.829091+0000) 2024-03-06T17:05:11.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:11.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:11.829323+0000) 2024-03-06T17:05:12.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:12.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:12.829615+0000) 2024-03-06T17:05:13.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:13.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:13.829817+0000) 2024-03-06T17:05:14.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:14.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:14.830096+0000) 2024-03-06T17:05:15.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:15.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:15.830300+0000) 2024-03-06T17:05:16.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:16.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:16.830542+0000) 2024-03-06T17:05:17.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:17.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:17.830798+0000) 2024-03-06T17:05:18.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:18.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:18.831078+0000) 2024-03-06T17:05:19.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:19.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:19.831319+0000) 2024-03-06T17:05:20.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:20.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:20.831555+0000) 2024-03-06T17:05:21.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:21.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:21.831794+0000) 2024-03-06T17:05:22.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:22.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:22.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:22.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:22.832092+0000) 2024-03-06T17:05:23.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:23.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:23.832294+0000) 2024-03-06T17:05:24.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:24.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:24.832540+0000) 2024-03-06T17:05:25.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:25.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:25.832739+0000) 2024-03-06T17:05:26.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:26.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:26.833045+0000) 2024-03-06T17:05:27.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:27.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:27.833316+0000) 2024-03-06T17:05:28.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:28.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:28.833629+0000) 2024-03-06T17:05:29.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:29.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:29.833846+0000) 2024-03-06T17:05:30.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:30.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:30.834149+0000) 2024-03-06T17:05:31.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:31.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:31.834403+0000) 2024-03-06T17:05:32.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:32.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:32.834596+0000) 2024-03-06T17:05:33.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:33.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:33.834812+0000) 2024-03-06T17:05:34.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:34.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:34.835074+0000) 2024-03-06T17:05:35.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:35.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:35.835299+0000) 2024-03-06T17:05:36.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:36.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:36.835533+0000) 2024-03-06T17:05:37.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:37.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:37.835745+0000) 2024-03-06T17:05:38.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:38.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:38.835968+0000) 2024-03-06T17:05:39.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:39.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:39.836180+0000) 2024-03-06T17:05:40.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:40.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:40.836411+0000) 2024-03-06T17:05:41.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:41.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:41.836632+0000) 2024-03-06T17:05:42.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:42.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:42.836890+0000) 2024-03-06T17:05:43.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:43.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:43.837130+0000) 2024-03-06T17:05:44.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:44.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:44.837320+0000) 2024-03-06T17:05:45.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:45.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:45.837523+0000) 2024-03-06T17:05:46.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:46.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:46.837797+0000) 2024-03-06T17:05:47.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:47.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:47.838065+0000) 2024-03-06T17:05:48.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:48.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:48.838247+0000) 2024-03-06T17:05:49.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:49.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:49.838473+0000) 2024-03-06T17:05:50.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:50.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:50.838737+0000) 2024-03-06T17:05:51.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:51.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:51.838969+0000) 2024-03-06T17:05:52.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:52.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:05:52.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:52.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:52.839170+0000) 2024-03-06T17:05:53.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:53.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:53.839384+0000) 2024-03-06T17:05:54.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:54.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:54.839600+0000) 2024-03-06T17:05:55.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:55.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:55.839826+0000) 2024-03-06T17:05:56.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:56.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:56.840114+0000) 2024-03-06T17:05:57.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:57.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:57.840346+0000) 2024-03-06T17:05:58.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:58.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:58.840628+0000) 2024-03-06T17:05:59.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:05:59.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:05:59.840911+0000) 2024-03-06T17:06:00.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:00.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:00.841189+0000) 2024-03-06T17:06:01.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:01.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:01.841435+0000) 2024-03-06T17:06:02.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:02.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:02.841645+0000) 2024-03-06T17:06:03.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:03.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:03.841890+0000) 2024-03-06T17:06:04.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:04.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:04.842126+0000) 2024-03-06T17:06:05.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:05.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:05.842366+0000) 2024-03-06T17:06:06.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:06.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:06.842629+0000) 2024-03-06T17:06:07.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:07.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:07.842855+0000) 2024-03-06T17:06:08.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:08.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:08.843100+0000) 2024-03-06T17:06:09.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:09.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:09.843350+0000) 2024-03-06T17:06:10.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:10.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:10.843592+0000) 2024-03-06T17:06:11.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:11.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:11.843895+0000) 2024-03-06T17:06:12.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:12.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:12.844146+0000) 2024-03-06T17:06:13.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:13.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:13.844382+0000) 2024-03-06T17:06:14.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:14.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:14.844610+0000) 2024-03-06T17:06:15.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:15.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:15.844857+0000) 2024-03-06T17:06:16.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:16.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:16.845105+0000) 2024-03-06T17:06:17.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:17.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:17.845298+0000) 2024-03-06T17:06:18.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:18.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:18.845492+0000) 2024-03-06T17:06:19.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:19.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:19.845713+0000) 2024-03-06T17:06:20.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:20.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:20.845938+0000) 2024-03-06T17:06:21.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:21.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:21.846162+0000) 2024-03-06T17:06:22.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:22.636 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:22.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:22.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:22.846380+0000) 2024-03-06T17:06:23.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:23.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:23.846581+0000) 2024-03-06T17:06:24.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:24.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:24.846803+0000) 2024-03-06T17:06:25.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:25.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:25.846993+0000) 2024-03-06T17:06:26.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:26.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:26.847276+0000) 2024-03-06T17:06:27.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:27.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:27.847524+0000) 2024-03-06T17:06:28.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:28.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:28.847769+0000) 2024-03-06T17:06:29.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:29.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:29.847973+0000) 2024-03-06T17:06:30.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:30.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:30.848206+0000) 2024-03-06T17:06:31.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:31.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:31.848434+0000) 2024-03-06T17:06:32.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:32.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:32.848661+0000) 2024-03-06T17:06:33.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:33.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:33.848887+0000) 2024-03-06T17:06:34.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:34.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:34.849067+0000) 2024-03-06T17:06:35.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:35.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:35.849254+0000) 2024-03-06T17:06:36.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:36.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:36.849488+0000) 2024-03-06T17:06:37.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:37.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:37.849687+0000) 2024-03-06T17:06:38.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:38.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:38.849903+0000) 2024-03-06T17:06:39.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:39.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:39.850132+0000) 2024-03-06T17:06:40.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:40.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:40.850279+0000) 2024-03-06T17:06:41.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:41.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:41.850511+0000) 2024-03-06T17:06:42.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:42.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:42.850733+0000) 2024-03-06T17:06:43.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:43.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:43.850967+0000) 2024-03-06T17:06:44.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:44.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:44.851254+0000) 2024-03-06T17:06:45.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:45.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:45.851476+0000) 2024-03-06T17:06:46.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:46.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:46.851680+0000) 2024-03-06T17:06:47.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:47.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:47.851898+0000) 2024-03-06T17:06:48.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:48.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:48.852148+0000) 2024-03-06T17:06:49.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:49.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:49.852326+0000) 2024-03-06T17:06:50.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:50.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:50.852517+0000) 2024-03-06T17:06:51.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:51.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:51.852751+0000) 2024-03-06T17:06:52.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:52.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:06:52.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:52.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:52.852925+0000) 2024-03-06T17:06:53.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:53.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:53.853127+0000) 2024-03-06T17:06:54.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:54.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:54.853327+0000) 2024-03-06T17:06:55.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:55.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:55.853554+0000) 2024-03-06T17:06:56.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:56.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:56.853786+0000) 2024-03-06T17:06:57.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:57.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:57.854034+0000) 2024-03-06T17:06:58.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:58.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:58.854259+0000) 2024-03-06T17:06:59.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:06:59.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:06:59.854520+0000) 2024-03-06T17:07:00.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:00.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:00.854745+0000) 2024-03-06T17:07:01.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:01.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:01.855039+0000) 2024-03-06T17:07:02.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:02.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:02.855275+0000) 2024-03-06T17:07:03.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:03.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:03.855503+0000) 2024-03-06T17:07:04.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:04.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:04.855782+0000) 2024-03-06T17:07:05.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:05.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:05.856038+0000) 2024-03-06T17:07:06.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:06.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:06.856239+0000) 2024-03-06T17:07:07.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:07.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:07.856457+0000) 2024-03-06T17:07:08.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:08.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:08.856679+0000) 2024-03-06T17:07:09.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:09.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:09.856904+0000) 2024-03-06T17:07:10.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:10.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:10.857141+0000) 2024-03-06T17:07:11.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:11.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:11.857302+0000) 2024-03-06T17:07:12.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:12.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:12.857508+0000) 2024-03-06T17:07:13.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:13.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:13.857731+0000) 2024-03-06T17:07:14.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:14.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:14.857949+0000) 2024-03-06T17:07:15.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:15.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:15.858167+0000) 2024-03-06T17:07:16.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:16.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:16.858398+0000) 2024-03-06T17:07:17.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:17.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:17.858582+0000) 2024-03-06T17:07:18.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:18.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:18.858801+0000) 2024-03-06T17:07:19.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:19.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:19.859065+0000) 2024-03-06T17:07:20.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:20.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:20.859282+0000) 2024-03-06T17:07:21.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:21.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:21.859503+0000) 2024-03-06T17:07:22.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:22.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:22.859717+0000) 2024-03-06T17:07:23.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:23.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:23.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:23.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:23.859914+0000) 2024-03-06T17:07:24.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:24.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:24.860156+0000) 2024-03-06T17:07:25.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:25.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:25.860400+0000) 2024-03-06T17:07:26.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:26.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:26.860660+0000) 2024-03-06T17:07:27.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:27.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:27.860917+0000) 2024-03-06T17:07:28.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:28.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:28.861197+0000) 2024-03-06T17:07:29.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:29.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:29.861403+0000) 2024-03-06T17:07:30.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:30.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:30.861612+0000) 2024-03-06T17:07:31.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:31.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:31.861833+0000) 2024-03-06T17:07:32.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:32.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:32.862095+0000) 2024-03-06T17:07:33.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:33.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:33.862326+0000) 2024-03-06T17:07:34.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:34.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:34.862549+0000) 2024-03-06T17:07:35.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:35.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:35.862775+0000) 2024-03-06T17:07:36.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:36.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:36.863034+0000) 2024-03-06T17:07:37.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:37.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:37.863226+0000) 2024-03-06T17:07:38.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:38.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:38.863416+0000) 2024-03-06T17:07:39.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:39.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:39.863624+0000) 2024-03-06T17:07:40.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:40.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:40.863850+0000) 2024-03-06T17:07:41.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:41.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:41.864099+0000) 2024-03-06T17:07:42.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:42.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:42.864286+0000) 2024-03-06T17:07:43.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:43.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:43.864513+0000) 2024-03-06T17:07:44.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:44.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:44.864731+0000) 2024-03-06T17:07:45.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:45.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:45.864949+0000) 2024-03-06T17:07:46.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:46.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:46.865179+0000) 2024-03-06T17:07:47.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:47.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:47.865371+0000) 2024-03-06T17:07:48.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:48.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:48.865567+0000) 2024-03-06T17:07:49.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:49.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:49.865784+0000) 2024-03-06T17:07:50.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:50.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:50.865963+0000) 2024-03-06T17:07:51.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:51.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:51.866202+0000) 2024-03-06T17:07:52.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:52.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:52.866391+0000) 2024-03-06T17:07:53.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:53.208 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:07:53.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:53.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:53.866604+0000) 2024-03-06T17:07:54.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:54.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:54.866819+0000) 2024-03-06T17:07:55.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:55.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:55.867093+0000) 2024-03-06T17:07:56.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:56.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:56.867323+0000) 2024-03-06T17:07:57.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:57.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:57.867587+0000) 2024-03-06T17:07:58.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:58.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:58.867829+0000) 2024-03-06T17:07:59.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:07:59.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:07:59.868085+0000) 2024-03-06T17:08:00.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:00.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:00.868304+0000) 2024-03-06T17:08:01.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:01.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:01.868568+0000) 2024-03-06T17:08:02.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:02.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:02.868858+0000) 2024-03-06T17:08:03.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:03.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:03.869161+0000) 2024-03-06T17:08:04.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:04.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:04.869373+0000) 2024-03-06T17:08:05.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:05.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:05.869621+0000) 2024-03-06T17:08:06.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:06.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:06.869867+0000) 2024-03-06T17:08:07.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:07.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:07.870119+0000) 2024-03-06T17:08:08.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:08.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:08.870308+0000) 2024-03-06T17:08:09.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:09.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:09.870493+0000) 2024-03-06T17:08:10.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:10.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:10.870753+0000) 2024-03-06T17:08:11.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:11.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:11.871039+0000) 2024-03-06T17:08:12.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:12.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:12.871258+0000) 2024-03-06T17:08:13.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:13.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:13.871480+0000) 2024-03-06T17:08:14.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:14.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:14.871685+0000) 2024-03-06T17:08:15.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:15.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:15.871908+0000) 2024-03-06T17:08:16.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:16.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:16.872138+0000) 2024-03-06T17:08:17.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:17.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:17.872383+0000) 2024-03-06T17:08:18.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:18.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:18.872606+0000) 2024-03-06T17:08:19.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:19.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:19.872825+0000) 2024-03-06T17:08:20.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:20.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:20.873139+0000) 2024-03-06T17:08:21.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:21.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:21.873393+0000) 2024-03-06T17:08:22.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:22.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:22.873659+0000) 2024-03-06T17:08:23.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:23.345 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:23.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:23.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:23.873942+0000) 2024-03-06T17:08:24.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:24.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:24.874195+0000) 2024-03-06T17:08:25.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:25.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:25.874357+0000) 2024-03-06T17:08:26.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:26.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:26.874603+0000) 2024-03-06T17:08:27.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:27.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:27.874832+0000) 2024-03-06T17:08:28.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:28.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:28.875138+0000) 2024-03-06T17:08:29.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:29.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:29.875376+0000) 2024-03-06T17:08:30.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:30.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:30.875623+0000) 2024-03-06T17:08:31.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:31.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:31.875901+0000) 2024-03-06T17:08:32.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:32.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:32.876194+0000) 2024-03-06T17:08:33.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:33.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:33.876452+0000) 2024-03-06T17:08:34.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:34.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:34.876700+0000) 2024-03-06T17:08:35.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:35.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:35.876989+0000) 2024-03-06T17:08:36.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:36.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:36.877267+0000) 2024-03-06T17:08:37.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:37.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:37.877457+0000) 2024-03-06T17:08:38.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:38.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:38.877684+0000) 2024-03-06T17:08:39.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:39.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:39.877855+0000) 2024-03-06T17:08:40.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:40.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:40.878121+0000) 2024-03-06T17:08:41.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:41.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:41.878347+0000) 2024-03-06T17:08:42.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:42.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:42.878571+0000) 2024-03-06T17:08:43.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:43.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:43.878816+0000) 2024-03-06T17:08:44.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:44.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:44.879109+0000) 2024-03-06T17:08:45.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:45.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:45.879398+0000) 2024-03-06T17:08:46.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:46.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:46.879671+0000) 2024-03-06T17:08:47.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:47.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:47.879953+0000) 2024-03-06T17:08:48.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:48.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:48.880197+0000) 2024-03-06T17:08:49.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:49.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:49.880434+0000) 2024-03-06T17:08:50.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:50.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:50.880667+0000) 2024-03-06T17:08:51.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:51.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:51.880945+0000) 2024-03-06T17:08:52.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:52.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:52.881168+0000) 2024-03-06T17:08:53.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:53.502 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:08:53.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:53.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:53.881337+0000) 2024-03-06T17:08:54.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:54.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:54.881559+0000) 2024-03-06T17:08:55.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:55.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:55.881788+0000) 2024-03-06T17:08:56.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:56.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:56.882075+0000) 2024-03-06T17:08:57.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:57.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:57.882322+0000) 2024-03-06T17:08:58.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:58.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:58.882565+0000) 2024-03-06T17:08:59.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:08:59.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:08:59.882783+0000) 2024-03-06T17:09:00.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:00.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:00.883059+0000) 2024-03-06T17:09:01.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:01.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:01.883289+0000) 2024-03-06T17:09:02.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:02.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:02.883516+0000) 2024-03-06T17:09:03.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:03.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:03.883747+0000) 2024-03-06T17:09:04.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:04.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:04.883977+0000) 2024-03-06T17:09:05.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:05.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:05.884155+0000) 2024-03-06T17:09:06.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:06.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:06.884393+0000) 2024-03-06T17:09:07.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:07.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:07.884584+0000) 2024-03-06T17:09:08.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:08.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:08.884822+0000) 2024-03-06T17:09:09.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:09.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:09.885106+0000) 2024-03-06T17:09:10.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:10.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:10.885329+0000) 2024-03-06T17:09:11.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:11.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:11.885541+0000) 2024-03-06T17:09:12.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:12.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:12.885774+0000) 2024-03-06T17:09:13.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:13.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:13.886046+0000) 2024-03-06T17:09:14.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:14.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:14.886244+0000) 2024-03-06T17:09:15.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:15.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:15.886463+0000) 2024-03-06T17:09:16.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:16.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:16.886745+0000) 2024-03-06T17:09:17.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:17.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:17.887045+0000) 2024-03-06T17:09:18.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:18.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:18.887286+0000) 2024-03-06T17:09:19.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:19.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:19.887548+0000) 2024-03-06T17:09:20.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:20.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:20.887784+0000) 2024-03-06T17:09:21.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:21.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:21.888074+0000) 2024-03-06T17:09:22.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:22.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:22.888310+0000) 2024-03-06T17:09:23.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:23.703 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:23.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:23.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:23.888498+0000) 2024-03-06T17:09:24.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:24.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:24.888772+0000) 2024-03-06T17:09:25.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:25.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:25.889083+0000) 2024-03-06T17:09:26.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:26.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:26.889376+0000) 2024-03-06T17:09:27.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:27.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:27.889656+0000) 2024-03-06T17:09:28.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:28.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:28.889900+0000) 2024-03-06T17:09:29.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:29.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:29.890141+0000) 2024-03-06T17:09:30.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:30.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:30.890327+0000) 2024-03-06T17:09:31.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:31.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:31.890550+0000) 2024-03-06T17:09:32.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:32.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:32.890790+0000) 2024-03-06T17:09:33.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:33.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:33.890973+0000) 2024-03-06T17:09:34.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:34.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:34.891133+0000) 2024-03-06T17:09:35.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:35.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:35.891373+0000) 2024-03-06T17:09:36.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:36.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:36.891637+0000) 2024-03-06T17:09:37.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:37.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:37.891877+0000) 2024-03-06T17:09:38.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:38.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:38.892136+0000) 2024-03-06T17:09:39.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:39.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:39.892432+0000) 2024-03-06T17:09:40.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:40.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:40.892604+0000) 2024-03-06T17:09:41.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:41.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:41.892818+0000) 2024-03-06T17:09:42.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:42.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:42.893115+0000) 2024-03-06T17:09:43.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:43.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:43.893341+0000) 2024-03-06T17:09:44.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:44.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:44.893563+0000) 2024-03-06T17:09:45.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:45.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:45.893787+0000) 2024-03-06T17:09:46.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:46.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:46.894048+0000) 2024-03-06T17:09:47.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:47.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:47.894278+0000) 2024-03-06T17:09:48.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:48.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:48.894491+0000) 2024-03-06T17:09:49.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:49.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:49.894658+0000) 2024-03-06T17:09:50.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:50.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:50.894894+0000) 2024-03-06T17:09:51.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:51.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:51.895149+0000) 2024-03-06T17:09:52.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:52.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:52.895352+0000) 2024-03-06T17:09:53.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:53.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:09:53.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:53.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:53.895552+0000) 2024-03-06T17:09:54.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:54.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:54.895782+0000) 2024-03-06T17:09:55.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:55.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:55.895958+0000) 2024-03-06T17:09:56.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:56.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:56.896177+0000) 2024-03-06T17:09:57.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:57.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:57.896405+0000) 2024-03-06T17:09:58.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:58.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:58.896642+0000) 2024-03-06T17:09:59.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:09:59.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:09:59.896870+0000) 2024-03-06T17:10:00.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:00.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:00.897143+0000) 2024-03-06T17:10:01.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:01.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:01.897383+0000) 2024-03-06T17:10:02.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:02.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:02.897561+0000) 2024-03-06T17:10:03.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:03.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:03.897785+0000) 2024-03-06T17:10:04.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:04.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:04.898059+0000) 2024-03-06T17:10:05.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:05.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:05.898291+0000) 2024-03-06T17:10:06.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:06.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:06.898507+0000) 2024-03-06T17:10:07.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:07.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:07.898734+0000) 2024-03-06T17:10:08.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:08.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:08.898962+0000) 2024-03-06T17:10:09.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:09.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:09.899201+0000) 2024-03-06T17:10:10.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:10.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:10.899425+0000) 2024-03-06T17:10:11.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:11.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:11.899685+0000) 2024-03-06T17:10:12.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:12.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:12.899912+0000) 2024-03-06T17:10:13.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:13.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:13.900153+0000) 2024-03-06T17:10:14.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:14.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:14.900368+0000) 2024-03-06T17:10:15.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:15.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:15.900601+0000) 2024-03-06T17:10:16.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:16.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:16.900795+0000) 2024-03-06T17:10:17.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:17.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:17.901059+0000) 2024-03-06T17:10:18.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:18.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:18.901312+0000) 2024-03-06T17:10:19.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:19.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:19.901551+0000) 2024-03-06T17:10:20.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:20.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:20.901757+0000) 2024-03-06T17:10:21.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:21.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:21.902073+0000) 2024-03-06T17:10:22.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:22.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:22.902390+0000) 2024-03-06T17:10:23.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:23.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:23.902600+0000) 2024-03-06T17:10:24.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:24.096 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:24.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:24.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:24.902780+0000) 2024-03-06T17:10:25.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:25.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:25.903036+0000) 2024-03-06T17:10:26.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:26.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:26.903277+0000) 2024-03-06T17:10:27.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:27.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:27.903485+0000) 2024-03-06T17:10:28.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:28.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:28.903704+0000) 2024-03-06T17:10:29.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:29.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:29.903925+0000) 2024-03-06T17:10:30.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:30.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:30.904167+0000) 2024-03-06T17:10:31.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:31.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:31.904356+0000) 2024-03-06T17:10:32.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:32.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:32.904576+0000) 2024-03-06T17:10:33.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:33.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:33.904816+0000) 2024-03-06T17:10:34.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:34.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:34.905073+0000) 2024-03-06T17:10:35.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:35.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:35.905264+0000) 2024-03-06T17:10:36.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:36.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:36.905463+0000) 2024-03-06T17:10:37.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:37.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:37.905692+0000) 2024-03-06T17:10:38.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:38.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:38.905941+0000) 2024-03-06T17:10:39.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:39.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:39.906163+0000) 2024-03-06T17:10:40.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:40.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:40.906350+0000) 2024-03-06T17:10:41.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:41.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:41.906569+0000) 2024-03-06T17:10:42.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:42.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:42.906731+0000) 2024-03-06T17:10:43.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:43.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:43.906916+0000) 2024-03-06T17:10:44.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:44.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:44.907125+0000) 2024-03-06T17:10:45.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:45.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:45.907342+0000) 2024-03-06T17:10:46.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:46.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:46.907556+0000) 2024-03-06T17:10:47.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:47.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:47.907773+0000) 2024-03-06T17:10:48.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:48.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:48.908049+0000) 2024-03-06T17:10:49.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:49.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:49.908288+0000) 2024-03-06T17:10:50.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:50.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:50.908539+0000) 2024-03-06T17:10:51.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:51.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:51.908705+0000) 2024-03-06T17:10:52.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:52.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:52.908893+0000) 2024-03-06T17:10:53.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:53.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:53.909151+0000) 2024-03-06T17:10:54.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:54.233 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:10:54.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:54.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:54.909377+0000) 2024-03-06T17:10:55.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:55.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:55.909600+0000) 2024-03-06T17:10:56.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:56.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:56.909787+0000) 2024-03-06T17:10:57.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:57.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:57.909986+0000) 2024-03-06T17:10:58.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:58.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:58.910276+0000) 2024-03-06T17:10:59.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:10:59.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:10:59.910515+0000) 2024-03-06T17:11:00.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:00.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:00.910765+0000) 2024-03-06T17:11:01.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:01.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:01.910963+0000) 2024-03-06T17:11:02.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:02.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:02.911212+0000) 2024-03-06T17:11:03.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:03.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:03.911432+0000) 2024-03-06T17:11:04.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:04.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:04.911697+0000) 2024-03-06T17:11:05.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:05.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:05.911908+0000) 2024-03-06T17:11:06.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:06.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:06.912163+0000) 2024-03-06T17:11:07.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:07.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:07.912406+0000) 2024-03-06T17:11:08.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:08.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:08.912671+0000) 2024-03-06T17:11:09.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:09.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:09.912895+0000) 2024-03-06T17:11:10.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:10.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:10.913228+0000) 2024-03-06T17:11:11.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:11.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:11.913477+0000) 2024-03-06T17:11:12.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:12.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:12.913719+0000) 2024-03-06T17:11:13.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:13.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:13.913996+0000) 2024-03-06T17:11:14.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:14.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:14.914262+0000) 2024-03-06T17:11:15.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:15.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:15.914512+0000) 2024-03-06T17:11:16.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:16.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:16.914769+0000) 2024-03-06T17:11:17.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:17.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:17.915043+0000) 2024-03-06T17:11:18.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:18.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:18.915344+0000) 2024-03-06T17:11:19.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:19.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:19.915616+0000) 2024-03-06T17:11:20.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:20.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:20.915880+0000) 2024-03-06T17:11:21.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:21.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:21.916158+0000) 2024-03-06T17:11:22.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:22.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:22.916403+0000) 2024-03-06T17:11:23.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:23.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:23.916643+0000) 2024-03-06T17:11:24.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:24.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:24.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:24.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:24.916854+0000) 2024-03-06T17:11:25.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:25.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:25.917083+0000) 2024-03-06T17:11:26.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:26.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:26.917338+0000) 2024-03-06T17:11:27.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:27.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:27.917543+0000) 2024-03-06T17:11:28.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:28.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:28.917760+0000) 2024-03-06T17:11:29.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:29.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:29.917984+0000) 2024-03-06T17:11:30.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:30.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:30.918200+0000) 2024-03-06T17:11:31.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:31.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:31.918421+0000) 2024-03-06T17:11:32.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:32.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:32.918644+0000) 2024-03-06T17:11:33.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:33.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:33.918876+0000) 2024-03-06T17:11:34.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:34.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:34.919139+0000) 2024-03-06T17:11:35.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:35.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:35.919340+0000) 2024-03-06T17:11:36.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:36.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:36.919524+0000) 2024-03-06T17:11:37.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:37.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:37.919704+0000) 2024-03-06T17:11:38.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:38.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:38.919937+0000) 2024-03-06T17:11:39.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:39.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:39.920166+0000) 2024-03-06T17:11:40.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:40.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:40.920352+0000) 2024-03-06T17:11:41.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:41.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:41.920632+0000) 2024-03-06T17:11:42.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:42.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:42.920901+0000) 2024-03-06T17:11:43.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:43.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:43.921179+0000) 2024-03-06T17:11:44.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:44.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:44.921415+0000) 2024-03-06T17:11:45.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:45.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:45.921662+0000) 2024-03-06T17:11:46.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:46.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:46.921831+0000) 2024-03-06T17:11:47.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:47.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:47.922085+0000) 2024-03-06T17:11:48.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:48.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:48.922312+0000) 2024-03-06T17:11:49.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:49.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:49.922568+0000) 2024-03-06T17:11:50.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:50.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:50.922842+0000) 2024-03-06T17:11:51.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:51.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:51.923153+0000) 2024-03-06T17:11:52.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:52.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:52.923393+0000) 2024-03-06T17:11:53.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:53.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:53.923581+0000) 2024-03-06T17:11:54.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:54.642 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:11:54.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:54.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:54.923770+0000) 2024-03-06T17:11:55.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:55.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:55.923952+0000) 2024-03-06T17:11:56.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:56.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:56.924157+0000) 2024-03-06T17:11:57.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:57.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:57.924345+0000) 2024-03-06T17:11:58.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:58.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:58.924550+0000) 2024-03-06T17:11:59.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:11:59.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:11:59.924825+0000) 2024-03-06T17:12:00.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:00.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:00.925075+0000) 2024-03-06T17:12:01.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:01.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:01.925263+0000) 2024-03-06T17:12:02.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:02.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:02.925464+0000) 2024-03-06T17:12:03.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:03.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:03.925682+0000) 2024-03-06T17:12:04.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:04.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:04.925876+0000) 2024-03-06T17:12:05.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:05.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:05.926132+0000) 2024-03-06T17:12:06.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:06.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:06.926418+0000) 2024-03-06T17:12:07.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:07.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:07.926627+0000) 2024-03-06T17:12:08.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:08.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:08.926933+0000) 2024-03-06T17:12:09.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:09.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:09.927192+0000) 2024-03-06T17:12:10.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:10.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:10.927401+0000) 2024-03-06T17:12:11.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:11.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:11.927696+0000) 2024-03-06T17:12:12.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:12.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:12.927978+0000) 2024-03-06T17:12:13.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:13.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:13.928356+0000) 2024-03-06T17:12:14.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:14.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:14.928555+0000) 2024-03-06T17:12:15.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:15.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:15.928778+0000) 2024-03-06T17:12:16.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:16.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:16.929058+0000) 2024-03-06T17:12:17.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:17.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:17.929252+0000) 2024-03-06T17:12:18.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:18.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:18.929431+0000) 2024-03-06T17:12:19.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:19.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:19.929688+0000) 2024-03-06T17:12:20.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:20.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:20.929929+0000) 2024-03-06T17:12:21.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:21.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:21.930161+0000) 2024-03-06T17:12:22.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:22.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:22.930383+0000) 2024-03-06T17:12:23.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:23.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:23.930602+0000) 2024-03-06T17:12:24.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:24.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:24.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:24.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:24.930809+0000) 2024-03-06T17:12:25.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:25.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:25.931106+0000) 2024-03-06T17:12:26.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:26.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:26.931398+0000) 2024-03-06T17:12:27.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:27.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:27.931657+0000) 2024-03-06T17:12:28.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:28.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:28.931886+0000) 2024-03-06T17:12:29.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:29.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:29.932131+0000) 2024-03-06T17:12:30.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:30.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:30.932286+0000) 2024-03-06T17:12:31.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:31.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:31.932495+0000) 2024-03-06T17:12:32.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:32.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:32.932703+0000) 2024-03-06T17:12:33.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:33.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:33.932932+0000) 2024-03-06T17:12:34.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:34.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:34.933156+0000) 2024-03-06T17:12:35.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:35.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:35.933352+0000) 2024-03-06T17:12:36.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:36.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:36.933590+0000) 2024-03-06T17:12:37.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:37.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:37.933801+0000) 2024-03-06T17:12:38.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:38.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:38.934046+0000) 2024-03-06T17:12:39.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:39.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:39.934244+0000) 2024-03-06T17:12:40.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:40.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:40.934446+0000) 2024-03-06T17:12:41.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:41.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:41.934629+0000) 2024-03-06T17:12:42.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:42.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:42.934892+0000) 2024-03-06T17:12:43.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:43.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:43.935144+0000) 2024-03-06T17:12:44.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:44.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:44.935378+0000) 2024-03-06T17:12:45.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:45.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:45.935652+0000) 2024-03-06T17:12:46.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:46.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:46.935940+0000) 2024-03-06T17:12:47.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:47.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:47.936195+0000) 2024-03-06T17:12:48.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:48.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:48.936475+0000) 2024-03-06T17:12:49.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:49.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:49.936720+0000) 2024-03-06T17:12:50.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:50.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:50.936907+0000) 2024-03-06T17:12:51.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:51.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:51.937177+0000) 2024-03-06T17:12:52.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:52.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:52.937382+0000) 2024-03-06T17:12:53.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:53.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:53.937659+0000) 2024-03-06T17:12:54.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:54.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:54.937929+0000) 2024-03-06T17:12:55.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:55.042 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:12:55.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:55.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:55.938206+0000) 2024-03-06T17:12:56.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:56.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:56.938551+0000) 2024-03-06T17:12:57.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:57.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:57.938798+0000) 2024-03-06T17:12:58.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:58.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:58.939048+0000) 2024-03-06T17:12:59.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:12:59.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:12:59.939248+0000) 2024-03-06T17:13:00.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:00.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:00.939508+0000) 2024-03-06T17:13:01.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:01.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:01.939795+0000) 2024-03-06T17:13:02.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:02.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:02.940061+0000) 2024-03-06T17:13:03.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:03.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:03.940335+0000) 2024-03-06T17:13:04.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:04.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:04.940619+0000) 2024-03-06T17:13:05.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:05.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:05.940902+0000) 2024-03-06T17:13:06.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:06.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:06.941192+0000) 2024-03-06T17:13:07.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:07.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:07.941472+0000) 2024-03-06T17:13:08.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:08.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:08.941765+0000) 2024-03-06T17:13:09.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:09.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:09.942096+0000) 2024-03-06T17:13:10.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:10.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:10.942267+0000) 2024-03-06T17:13:11.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:11.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:11.942500+0000) 2024-03-06T17:13:12.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:12.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:12.942732+0000) 2024-03-06T17:13:13.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:13.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:13.942963+0000) 2024-03-06T17:13:14.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:14.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:14.943282+0000) 2024-03-06T17:13:15.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:15.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:15.943477+0000) 2024-03-06T17:13:16.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:16.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:16.943693+0000) 2024-03-06T17:13:17.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:17.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:17.943881+0000) 2024-03-06T17:13:18.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:18.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:18.944126+0000) 2024-03-06T17:13:19.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:19.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:19.944363+0000) 2024-03-06T17:13:20.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:20.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:20.944581+0000) 2024-03-06T17:13:21.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:21.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:21.944831+0000) 2024-03-06T17:13:22.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:22.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:22.945047+0000) 2024-03-06T17:13:23.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:23.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:23.945273+0000) 2024-03-06T17:13:24.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:24.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:24.945511+0000) 2024-03-06T17:13:25.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:25.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:25.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:25.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:25.945704+0000) 2024-03-06T17:13:26.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:26.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:26.945938+0000) 2024-03-06T17:13:27.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:27.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:27.946175+0000) 2024-03-06T17:13:28.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:28.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:28.946417+0000) 2024-03-06T17:13:29.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:29.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:29.946680+0000) 2024-03-06T17:13:30.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:30.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:30.946841+0000) 2024-03-06T17:13:31.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:31.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:31.947100+0000) 2024-03-06T17:13:32.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:32.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:32.947257+0000) 2024-03-06T17:13:33.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:33.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:33.947482+0000) 2024-03-06T17:13:34.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:34.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:34.947717+0000) 2024-03-06T17:13:35.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:35.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:35.947940+0000) 2024-03-06T17:13:36.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:36.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:36.948161+0000) 2024-03-06T17:13:37.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:37.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:37.948410+0000) 2024-03-06T17:13:38.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:38.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:38.948624+0000) 2024-03-06T17:13:39.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:39.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:39.948855+0000) 2024-03-06T17:13:40.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:40.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:40.949127+0000) 2024-03-06T17:13:41.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:41.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:41.949345+0000) 2024-03-06T17:13:42.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:42.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:42.949590+0000) 2024-03-06T17:13:43.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:43.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:43.949836+0000) 2024-03-06T17:13:44.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:44.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:44.950117+0000) 2024-03-06T17:13:45.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:45.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:45.950317+0000) 2024-03-06T17:13:46.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:46.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:46.950542+0000) 2024-03-06T17:13:47.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:47.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:47.950775+0000) 2024-03-06T17:13:48.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:48.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:48.950988+0000) 2024-03-06T17:13:49.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:49.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:49.951242+0000) 2024-03-06T17:13:50.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:50.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:50.951500+0000) 2024-03-06T17:13:51.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:51.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:51.951720+0000) 2024-03-06T17:13:52.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:52.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:52.951932+0000) 2024-03-06T17:13:53.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:53.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:53.952152+0000) 2024-03-06T17:13:54.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:54.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:54.952339+0000) 2024-03-06T17:13:55.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:55.347 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:13:55.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:55.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:55.952552+0000) 2024-03-06T17:13:56.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:56.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:56.952786+0000) 2024-03-06T17:13:57.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:57.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:57.953030+0000) 2024-03-06T17:13:58.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:58.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:58.953215+0000) 2024-03-06T17:13:59.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:13:59.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:13:59.953408+0000) 2024-03-06T17:14:00.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:00.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:00.953615+0000) 2024-03-06T17:14:01.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:01.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:01.953840+0000) 2024-03-06T17:14:02.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:02.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:02.954086+0000) 2024-03-06T17:14:03.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:03.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:03.954309+0000) 2024-03-06T17:14:04.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:04.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:04.954526+0000) 2024-03-06T17:14:05.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:05.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:05.954778+0000) 2024-03-06T17:14:06.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:06.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:06.955034+0000) 2024-03-06T17:14:07.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:07.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:07.955224+0000) 2024-03-06T17:14:08.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:08.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:08.955429+0000) 2024-03-06T17:14:09.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:09.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:09.955654+0000) 2024-03-06T17:14:10.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:10.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:10.955894+0000) 2024-03-06T17:14:11.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:11.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:11.956149+0000) 2024-03-06T17:14:12.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:12.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:12.956371+0000) 2024-03-06T17:14:13.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:13.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:13.956592+0000) 2024-03-06T17:14:14.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:14.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:14.956816+0000) 2024-03-06T17:14:15.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:15.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:15.957078+0000) 2024-03-06T17:14:16.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:16.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:16.957281+0000) 2024-03-06T17:14:17.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:17.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:17.957482+0000) 2024-03-06T17:14:18.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:18.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:18.957721+0000) 2024-03-06T17:14:19.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:19.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:19.957966+0000) 2024-03-06T17:14:20.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:20.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:20.958162+0000) 2024-03-06T17:14:21.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:21.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:21.958326+0000) 2024-03-06T17:14:22.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:22.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:22.958564+0000) 2024-03-06T17:14:23.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:23.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:23.958789+0000) 2024-03-06T17:14:24.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:24.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:24.959040+0000) 2024-03-06T17:14:25.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:25.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:25.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:25.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:25.959281+0000) 2024-03-06T17:14:26.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:26.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:26.959532+0000) 2024-03-06T17:14:27.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:27.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:27.959779+0000) 2024-03-06T17:14:28.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:28.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:28.960124+0000) 2024-03-06T17:14:29.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:29.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:29.960369+0000) 2024-03-06T17:14:30.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:30.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:30.960575+0000) 2024-03-06T17:14:31.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:31.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:31.960827+0000) 2024-03-06T17:14:32.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:32.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:32.961105+0000) 2024-03-06T17:14:33.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:33.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:33.961345+0000) 2024-03-06T17:14:34.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:34.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:34.961543+0000) 2024-03-06T17:14:35.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:35.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:35.961751+0000) 2024-03-06T17:14:36.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:36.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:36.961942+0000) 2024-03-06T17:14:37.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:37.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:37.962145+0000) 2024-03-06T17:14:38.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:38.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:38.962382+0000) 2024-03-06T17:14:39.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:39.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:39.962592+0000) 2024-03-06T17:14:40.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:40.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:40.962795+0000) 2024-03-06T17:14:41.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:41.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:41.963034+0000) 2024-03-06T17:14:42.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:42.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:42.963193+0000) 2024-03-06T17:14:43.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:43.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:43.963431+0000) 2024-03-06T17:14:44.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:44.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:44.963675+0000) 2024-03-06T17:14:45.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:45.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:45.963847+0000) 2024-03-06T17:14:46.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:46.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:46.964119+0000) 2024-03-06T17:14:47.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:47.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:47.964312+0000) 2024-03-06T17:14:48.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:48.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:48.964522+0000) 2024-03-06T17:14:49.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:49.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:49.964712+0000) 2024-03-06T17:14:50.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:50.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:50.964908+0000) 2024-03-06T17:14:51.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:51.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:51.965153+0000) 2024-03-06T17:14:52.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:52.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:52.965353+0000) 2024-03-06T17:14:53.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:53.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:53.965581+0000) 2024-03-06T17:14:54.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:54.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:54.965780+0000) 2024-03-06T17:14:55.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:55.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:14:55.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:55.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:55.966057+0000) 2024-03-06T17:14:56.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:56.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:56.966256+0000) 2024-03-06T17:14:57.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:57.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:57.966491+0000) 2024-03-06T17:14:58.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:58.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:58.966676+0000) 2024-03-06T17:14:59.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:14:59.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:14:59.966840+0000) 2024-03-06T17:15:00.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:00.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:00.967107+0000) 2024-03-06T17:15:01.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:01.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:01.967327+0000) 2024-03-06T17:15:02.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:02.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:02.967550+0000) 2024-03-06T17:15:03.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:03.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:03.967771+0000) 2024-03-06T17:15:04.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:04.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:04.968047+0000) 2024-03-06T17:15:05.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:05.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:05.968222+0000) 2024-03-06T17:15:06.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:06.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:06.968428+0000) 2024-03-06T17:15:07.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:07.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:07.968643+0000) 2024-03-06T17:15:08.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:08.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:08.968837+0000) 2024-03-06T17:15:09.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:09.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:09.969114+0000) 2024-03-06T17:15:10.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:10.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:10.969302+0000) 2024-03-06T17:15:11.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:11.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:11.969638+0000) 2024-03-06T17:15:12.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:12.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:12.969820+0000) 2024-03-06T17:15:13.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:13.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:13.970083+0000) 2024-03-06T17:15:14.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:14.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:14.970288+0000) 2024-03-06T17:15:15.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:15.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:15.970506+0000) 2024-03-06T17:15:16.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:16.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:16.970721+0000) 2024-03-06T17:15:17.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:17.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:17.970942+0000) 2024-03-06T17:15:18.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:18.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:18.971163+0000) 2024-03-06T17:15:19.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:19.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:19.971393+0000) 2024-03-06T17:15:20.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:20.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:20.971627+0000) 2024-03-06T17:15:21.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:21.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:21.971877+0000) 2024-03-06T17:15:22.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:22.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:22.972127+0000) 2024-03-06T17:15:23.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:23.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:23.972334+0000) 2024-03-06T17:15:24.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:24.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:24.972559+0000) 2024-03-06T17:15:25.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:25.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:25.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:25.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:25.972805+0000) 2024-03-06T17:15:26.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:26.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:26.973077+0000) 2024-03-06T17:15:27.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:27.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:27.973281+0000) 2024-03-06T17:15:28.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:28.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:28.973498+0000) 2024-03-06T17:15:29.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:29.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:29.973715+0000) 2024-03-06T17:15:30.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:30.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:30.973939+0000) 2024-03-06T17:15:31.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:31.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:31.974169+0000) 2024-03-06T17:15:32.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:32.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:32.974363+0000) 2024-03-06T17:15:33.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:33.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:33.974564+0000) 2024-03-06T17:15:34.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:34.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:34.974794+0000) 2024-03-06T17:15:35.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:35.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:35.975053+0000) 2024-03-06T17:15:36.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:36.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:36.975300+0000) 2024-03-06T17:15:37.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:37.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:37.975525+0000) 2024-03-06T17:15:38.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:38.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:38.975748+0000) 2024-03-06T17:15:39.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:39.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:39.975935+0000) 2024-03-06T17:15:40.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:40.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:40.976171+0000) 2024-03-06T17:15:41.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:41.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:41.976326+0000) 2024-03-06T17:15:42.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:42.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:42.976536+0000) 2024-03-06T17:15:43.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:43.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:43.976754+0000) 2024-03-06T17:15:44.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:44.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:44.976979+0000) 2024-03-06T17:15:45.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:45.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:45.977191+0000) 2024-03-06T17:15:46.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:46.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:46.977392+0000) 2024-03-06T17:15:47.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:47.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:47.977599+0000) 2024-03-06T17:15:48.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:48.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:48.977805+0000) 2024-03-06T17:15:49.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:49.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:49.978068+0000) 2024-03-06T17:15:50.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:50.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:50.978295+0000) 2024-03-06T17:15:51.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:51.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:51.978501+0000) 2024-03-06T17:15:52.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:52.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:52.978729+0000) 2024-03-06T17:15:53.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:53.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:53.978955+0000) 2024-03-06T17:15:54.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:54.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:54.979196+0000) 2024-03-06T17:15:55.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:55.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:55.979385+0000) 2024-03-06T17:15:56.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:56.117 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:15:56.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:56.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:56.979619+0000) 2024-03-06T17:15:57.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:57.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:57.979910+0000) 2024-03-06T17:15:58.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:58.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:58.980176+0000) 2024-03-06T17:15:59.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:15:59.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:15:59.980453+0000) 2024-03-06T17:16:00.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:00.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:00.980693+0000) 2024-03-06T17:16:01.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:01.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:01.980943+0000) 2024-03-06T17:16:02.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:02.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:02.981240+0000) 2024-03-06T17:16:03.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:03.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:03.981472+0000) 2024-03-06T17:16:04.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:04.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:04.981678+0000) 2024-03-06T17:16:05.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:05.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:05.981922+0000) 2024-03-06T17:16:06.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:06.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:06.982152+0000) 2024-03-06T17:16:07.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:07.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:07.982374+0000) 2024-03-06T17:16:08.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:08.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:08.982597+0000) 2024-03-06T17:16:09.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:09.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:09.982795+0000) 2024-03-06T17:16:10.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:10.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:10.983050+0000) 2024-03-06T17:16:11.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:11.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:11.983280+0000) 2024-03-06T17:16:12.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:12.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:12.983522+0000) 2024-03-06T17:16:13.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:13.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:13.983740+0000) 2024-03-06T17:16:14.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:14.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:14.983968+0000) 2024-03-06T17:16:15.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:15.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:15.984215+0000) 2024-03-06T17:16:16.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:16.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:16.984416+0000) 2024-03-06T17:16:17.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:17.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:17.984593+0000) 2024-03-06T17:16:18.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:18.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:18.984817+0000) 2024-03-06T17:16:19.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:19.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:19.985124+0000) 2024-03-06T17:16:20.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:20.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:20.985323+0000) 2024-03-06T17:16:21.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:21.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:21.985551+0000) 2024-03-06T17:16:22.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:22.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:22.985767+0000) 2024-03-06T17:16:23.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:23.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:23.985992+0000) 2024-03-06T17:16:24.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:24.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:24.986240+0000) 2024-03-06T17:16:25.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:25.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:25.986453+0000) 2024-03-06T17:16:26.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:26.323 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:26.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:26.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:26.986673+0000) 2024-03-06T17:16:27.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:27.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:27.986905+0000) 2024-03-06T17:16:28.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:28.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:28.987139+0000) 2024-03-06T17:16:29.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:29.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:29.987338+0000) 2024-03-06T17:16:30.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:30.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:30.987546+0000) 2024-03-06T17:16:31.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:31.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:31.987768+0000) 2024-03-06T17:16:32.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:32.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:32.987987+0000) 2024-03-06T17:16:33.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:33.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:33.988231+0000) 2024-03-06T17:16:34.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:34.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:34.988466+0000) 2024-03-06T17:16:35.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:35.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:35.988685+0000) 2024-03-06T17:16:36.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:36.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:36.988959+0000) 2024-03-06T17:16:37.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:37.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:37.989193+0000) 2024-03-06T17:16:38.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:38.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:38.989427+0000) 2024-03-06T17:16:39.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:39.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:39.989702+0000) 2024-03-06T17:16:40.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:40.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:40.989972+0000) 2024-03-06T17:16:41.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:41.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:41.990262+0000) 2024-03-06T17:16:42.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:42.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:42.990476+0000) 2024-03-06T17:16:43.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:43.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:43.990724+0000) 2024-03-06T17:16:44.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:44.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:44.990960+0000) 2024-03-06T17:16:45.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:45.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:45.991158+0000) 2024-03-06T17:16:46.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:46.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:46.991406+0000) 2024-03-06T17:16:47.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:47.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:47.991644+0000) 2024-03-06T17:16:48.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:48.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:48.991886+0000) 2024-03-06T17:16:49.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:49.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:49.992146+0000) 2024-03-06T17:16:50.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:50.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:50.992339+0000) 2024-03-06T17:16:51.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:51.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:51.992543+0000) 2024-03-06T17:16:52.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:52.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:52.992742+0000) 2024-03-06T17:16:53.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:53.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:53.992960+0000) 2024-03-06T17:16:54.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:54.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:54.993201+0000) 2024-03-06T17:16:55.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:55.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:55.993434+0000) 2024-03-06T17:16:56.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:56.543 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:16:56.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:56.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:56.993681+0000) 2024-03-06T17:16:57.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:57.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:57.993949+0000) 2024-03-06T17:16:58.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:58.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:58.994182+0000) 2024-03-06T17:16:59.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:16:59.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:16:59.994412+0000) 2024-03-06T17:17:00.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:00.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:00.994604+0000) 2024-03-06T17:17:01.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:01.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:01.994868+0000) 2024-03-06T17:17:02.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:02.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:02.995122+0000) 2024-03-06T17:17:03.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:03.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:03.995340+0000) 2024-03-06T17:17:04.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:04.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:04.995580+0000) 2024-03-06T17:17:05.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:05.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:05.995804+0000) 2024-03-06T17:17:06.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:06.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:06.996054+0000) 2024-03-06T17:17:07.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:07.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:07.996297+0000) 2024-03-06T17:17:08.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:08.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:08.996531+0000) 2024-03-06T17:17:09.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:09.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:09.996792+0000) 2024-03-06T17:17:10.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:10.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:10.997038+0000) 2024-03-06T17:17:11.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:11.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:11.997238+0000) 2024-03-06T17:17:12.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:12.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:12.997450+0000) 2024-03-06T17:17:13.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:13.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:13.997653+0000) 2024-03-06T17:17:15.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:14.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:14.997842+0000) 2024-03-06T17:17:16.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:15.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:15.998113+0000) 2024-03-06T17:17:17.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:16.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:16.998346+0000) 2024-03-06T17:17:18.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:17.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:17.998569+0000) 2024-03-06T17:17:19.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:18.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:18.998793+0000) 2024-03-06T17:17:20.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:19.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:19.999087+0000) 2024-03-06T17:17:21.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:20.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:20.999335+0000) 2024-03-06T17:17:22.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:21.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:21.999598+0000) 2024-03-06T17:17:23.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:22.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:22.999853+0000) 2024-03-06T17:17:24.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:23.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:24.000128+0000) 2024-03-06T17:17:25.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:24.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:25.000377+0000) 2024-03-06T17:17:26.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:25.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:26.000611+0000) 2024-03-06T17:17:26.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:26.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:27.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:26.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:27.000836+0000) 2024-03-06T17:17:28.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:28.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:28.001062+0000) 2024-03-06T17:17:29.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:29.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:29.001252+0000) 2024-03-06T17:17:30.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:30.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:30.001460+0000) 2024-03-06T17:17:31.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:31.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:31.001641+0000) 2024-03-06T17:17:32.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:32.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:32.001867+0000) 2024-03-06T17:17:33.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:33.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:33.002107+0000) 2024-03-06T17:17:34.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:34.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:34.002293+0000) 2024-03-06T17:17:35.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:35.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:35.002500+0000) 2024-03-06T17:17:36.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:36.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:36.002735+0000) 2024-03-06T17:17:37.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:37.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:37.002965+0000) 2024-03-06T17:17:38.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:38.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:38.003208+0000) 2024-03-06T17:17:39.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:39.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:39.003391+0000) 2024-03-06T17:17:40.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:40.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:40.003612+0000) 2024-03-06T17:17:41.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:41.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:41.003855+0000) 2024-03-06T17:17:42.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:42.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:42.004097+0000) 2024-03-06T17:17:43.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:43.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:43.004315+0000) 2024-03-06T17:17:44.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:44.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:44.004582+0000) 2024-03-06T17:17:45.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:45.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:45.004822+0000) 2024-03-06T17:17:46.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:46.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:46.005088+0000) 2024-03-06T17:17:47.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:47.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:47.005290+0000) 2024-03-06T17:17:48.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:48.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:48.005505+0000) 2024-03-06T17:17:49.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:49.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:49.005712+0000) 2024-03-06T17:17:50.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:50.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:50.005880+0000) 2024-03-06T17:17:51.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:51.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:51.006092+0000) 2024-03-06T17:17:52.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:52.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:52.006328+0000) 2024-03-06T17:17:53.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:53.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:53.006563+0000) 2024-03-06T17:17:54.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:54.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:54.006782+0000) 2024-03-06T17:17:55.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:55.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:55.007000+0000) 2024-03-06T17:17:56.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:56.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:56.007327+0000) 2024-03-06T17:17:56.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:56.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:17:57.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:57.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:57.007551+0000) 2024-03-06T17:17:58.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:58.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:58.007785+0000) 2024-03-06T17:17:59.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:17:59.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:17:59.008038+0000) 2024-03-06T17:18:00.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:00.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:00.008265+0000) 2024-03-06T17:18:01.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:01.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:01.008461+0000) 2024-03-06T17:18:02.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:02.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:02.008687+0000) 2024-03-06T17:18:03.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:03.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:03.008913+0000) 2024-03-06T17:18:04.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:04.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:04.009140+0000) 2024-03-06T17:18:05.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:05.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:05.009285+0000) 2024-03-06T17:18:06.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:06.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:06.009466+0000) 2024-03-06T17:18:07.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:07.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:07.009703+0000) 2024-03-06T17:18:08.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:08.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:08.009928+0000) 2024-03-06T17:18:09.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:09.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:09.010189+0000) 2024-03-06T17:18:10.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:10.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:10.010409+0000) 2024-03-06T17:18:11.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:11.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:11.010606+0000) 2024-03-06T17:18:12.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:12.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:12.010834+0000) 2024-03-06T17:18:13.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:13.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:13.011075+0000) 2024-03-06T17:18:14.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:14.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:14.011306+0000) 2024-03-06T17:18:15.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:15.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:15.011523+0000) 2024-03-06T17:18:16.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:16.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:16.011777+0000) 2024-03-06T17:18:17.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:17.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:17.012062+0000) 2024-03-06T17:18:18.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:18.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:18.012293+0000) 2024-03-06T17:18:19.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:19.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:19.012530+0000) 2024-03-06T17:18:20.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:20.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:20.012772+0000) 2024-03-06T17:18:21.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:21.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:21.013031+0000) 2024-03-06T17:18:22.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:22.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:22.013259+0000) 2024-03-06T17:18:23.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:23.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:23.013479+0000) 2024-03-06T17:18:24.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:24.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:24.013727+0000) 2024-03-06T17:18:25.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:25.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:25.013989+0000) 2024-03-06T17:18:26.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:26.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:26.014258+0000) 2024-03-06T17:18:27.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:27.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:27.014479+0000) 2024-03-06T17:18:27.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:27.031 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:28.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:28.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:28.014715+0000) 2024-03-06T17:18:29.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:29.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:29.014925+0000) 2024-03-06T17:18:30.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:30.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:30.015165+0000) 2024-03-06T17:18:31.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:31.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:31.015343+0000) 2024-03-06T17:18:32.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:32.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:32.015570+0000) 2024-03-06T17:18:33.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:33.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:33.015773+0000) 2024-03-06T17:18:34.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:34.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:34.015987+0000) 2024-03-06T17:18:35.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:35.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:35.016194+0000) 2024-03-06T17:18:36.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:36.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:36.016424+0000) 2024-03-06T17:18:37.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:37.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:37.016691+0000) 2024-03-06T17:18:38.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:38.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:38.016933+0000) 2024-03-06T17:18:39.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:39.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:39.017163+0000) 2024-03-06T17:18:40.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:40.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:40.017313+0000) 2024-03-06T17:18:41.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:41.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:41.017510+0000) 2024-03-06T17:18:42.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:42.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:42.017739+0000) 2024-03-06T17:18:43.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:43.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:43.017991+0000) 2024-03-06T17:18:44.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:44.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:44.018222+0000) 2024-03-06T17:18:45.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:45.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:45.018434+0000) 2024-03-06T17:18:46.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:46.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:46.018679+0000) 2024-03-06T17:18:47.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:47.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:47.018919+0000) 2024-03-06T17:18:48.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:48.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:48.019162+0000) 2024-03-06T17:18:49.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:49.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:49.019406+0000) 2024-03-06T17:18:50.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:50.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:50.019654+0000) 2024-03-06T17:18:51.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:51.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:51.019890+0000) 2024-03-06T17:18:52.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:52.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:52.020119+0000) 2024-03-06T17:18:53.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:53.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:53.020345+0000) 2024-03-06T17:18:54.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:54.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:54.020571+0000) 2024-03-06T17:18:55.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:55.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:55.020794+0000) 2024-03-06T17:18:56.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:56.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:56.021061+0000) 2024-03-06T17:18:57.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:57.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:57.021349+0000) 2024-03-06T17:18:57.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:57.244 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:18:58.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:58.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:58.021603+0000) 2024-03-06T17:18:59.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:18:59.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:18:59.021832+0000) 2024-03-06T17:19:00.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:00.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:00.022105+0000) 2024-03-06T17:19:01.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:01.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:01.022341+0000) 2024-03-06T17:19:02.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:02.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:02.022586+0000) 2024-03-06T17:19:03.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:03.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:03.022803+0000) 2024-03-06T17:19:04.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:04.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:04.023046+0000) 2024-03-06T17:19:05.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:05.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:05.023280+0000) 2024-03-06T17:19:06.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:06.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:06.023514+0000) 2024-03-06T17:19:07.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:07.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:07.023775+0000) 2024-03-06T17:19:08.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:08.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:08.024032+0000) 2024-03-06T17:19:09.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:09.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:09.024195+0000) 2024-03-06T17:19:10.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:10.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:10.024385+0000) 2024-03-06T17:19:11.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:11.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:11.024612+0000) 2024-03-06T17:19:12.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:12.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:12.024833+0000) 2024-03-06T17:19:13.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:13.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:13.025102+0000) 2024-03-06T17:19:14.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:14.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:14.025254+0000) 2024-03-06T17:19:15.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:15.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:15.025445+0000) 2024-03-06T17:19:16.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:16.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:16.025664+0000) 2024-03-06T17:19:17.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:17.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:17.025901+0000) 2024-03-06T17:19:18.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:18.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:18.026134+0000) 2024-03-06T17:19:19.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:19.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:19.026323+0000) 2024-03-06T17:19:20.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:20.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:20.026587+0000) 2024-03-06T17:19:21.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:21.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:21.026823+0000) 2024-03-06T17:19:22.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:22.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:22.027085+0000) 2024-03-06T17:19:23.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:23.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:23.027309+0000) 2024-03-06T17:19:24.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:24.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:24.027516+0000) 2024-03-06T17:19:25.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:25.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:25.027755+0000) 2024-03-06T17:19:26.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:26.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:26.027981+0000) 2024-03-06T17:19:27.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:27.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:27.028219+0000) 2024-03-06T17:19:27.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:27.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:28.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:28.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:28.028421+0000) 2024-03-06T17:19:29.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:29.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:29.028631+0000) 2024-03-06T17:19:30.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:30.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:30.028851+0000) 2024-03-06T17:19:31.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:31.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:31.029125+0000) 2024-03-06T17:19:32.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:32.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:32.029337+0000) 2024-03-06T17:19:33.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:33.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:33.029564+0000) 2024-03-06T17:19:34.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:34.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:34.029795+0000) 2024-03-06T17:19:35.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:35.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:35.030055+0000) 2024-03-06T17:19:36.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:36.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:36.030307+0000) 2024-03-06T17:19:37.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:37.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:37.030615+0000) 2024-03-06T17:19:38.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:38.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:38.030855+0000) 2024-03-06T17:19:39.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:39.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:39.031117+0000) 2024-03-06T17:19:40.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:40.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:40.031366+0000) 2024-03-06T17:19:41.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:41.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:41.031606+0000) 2024-03-06T17:19:42.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:42.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:42.031845+0000) 2024-03-06T17:19:43.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:43.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:43.032069+0000) 2024-03-06T17:19:44.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:44.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:44.032288+0000) 2024-03-06T17:19:45.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:45.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:45.032531+0000) 2024-03-06T17:19:46.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:46.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:46.032769+0000) 2024-03-06T17:19:47.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:47.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:47.033000+0000) 2024-03-06T17:19:48.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:48.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:48.033240+0000) 2024-03-06T17:19:49.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:49.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:49.033451+0000) 2024-03-06T17:19:50.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:50.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:50.033644+0000) 2024-03-06T17:19:51.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:51.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:51.033856+0000) 2024-03-06T17:19:52.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:52.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:52.034094+0000) 2024-03-06T17:19:53.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:53.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:53.034294+0000) 2024-03-06T17:19:54.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:54.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:54.034501+0000) 2024-03-06T17:19:55.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:55.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:55.034722+0000) 2024-03-06T17:19:56.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:56.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:56.034913+0000) 2024-03-06T17:19:57.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:57.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:57.035156+0000) 2024-03-06T17:19:57.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:57.574 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:19:58.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:58.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:58.035387+0000) 2024-03-06T17:19:59.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:19:59.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:19:59.035610+0000) 2024-03-06T17:20:00.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:00.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:00.035848+0000) 2024-03-06T17:20:01.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:01.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:01.036098+0000) 2024-03-06T17:20:02.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:02.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:02.036320+0000) 2024-03-06T17:20:03.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:03.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:03.036548+0000) 2024-03-06T17:20:04.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:04.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:04.036752+0000) 2024-03-06T17:20:05.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:05.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:05.036995+0000) 2024-03-06T17:20:06.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:06.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:06.037263+0000) 2024-03-06T17:20:07.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:07.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:07.037480+0000) 2024-03-06T17:20:08.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:08.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:08.037721+0000) 2024-03-06T17:20:09.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:09.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:09.037967+0000) 2024-03-06T17:20:10.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:10.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:10.038183+0000) 2024-03-06T17:20:11.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:11.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:11.038402+0000) 2024-03-06T17:20:12.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:12.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:12.038652+0000) 2024-03-06T17:20:13.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:13.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:13.038877+0000) 2024-03-06T17:20:14.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:14.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:14.039132+0000) 2024-03-06T17:20:15.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:15.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:15.039279+0000) 2024-03-06T17:20:16.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:16.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:16.039508+0000) 2024-03-06T17:20:17.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:17.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:17.039761+0000) 2024-03-06T17:20:18.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:18.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:18.039983+0000) 2024-03-06T17:20:19.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:19.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:19.040195+0000) 2024-03-06T17:20:20.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:20.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:20.040428+0000) 2024-03-06T17:20:21.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:21.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:21.040652+0000) 2024-03-06T17:20:22.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:22.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:22.040891+0000) 2024-03-06T17:20:23.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:23.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:23.041137+0000) 2024-03-06T17:20:24.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:24.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:24.041322+0000) 2024-03-06T17:20:25.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:25.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:25.041560+0000) 2024-03-06T17:20:26.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:26.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:26.041785+0000) 2024-03-06T17:20:27.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:27.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:27.042046+0000) 2024-03-06T17:20:27.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:27.793 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:28.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:28.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:28.042250+0000) 2024-03-06T17:20:29.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:29.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:29.042480+0000) 2024-03-06T17:20:30.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:30.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:30.042686+0000) 2024-03-06T17:20:31.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:31.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:31.042924+0000) 2024-03-06T17:20:32.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:32.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:32.043164+0000) 2024-03-06T17:20:33.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:33.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:33.043371+0000) 2024-03-06T17:20:34.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:34.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:34.043561+0000) 2024-03-06T17:20:35.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:35.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:35.043778+0000) 2024-03-06T17:20:36.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:36.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:36.044045+0000) 2024-03-06T17:20:37.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:37.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:37.044212+0000) 2024-03-06T17:20:38.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:38.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:38.044401+0000) 2024-03-06T17:20:39.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:39.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:39.044621+0000) 2024-03-06T17:20:40.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:40.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:40.044846+0000) 2024-03-06T17:20:41.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:41.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:41.045115+0000) 2024-03-06T17:20:42.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:42.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:42.045339+0000) 2024-03-06T17:20:43.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:43.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:43.045568+0000) 2024-03-06T17:20:44.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:44.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:44.045762+0000) 2024-03-06T17:20:45.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:45.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:45.045988+0000) 2024-03-06T17:20:46.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:46.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:46.046217+0000) 2024-03-06T17:20:47.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:47.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:47.046448+0000) 2024-03-06T17:20:48.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:48.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:48.046656+0000) 2024-03-06T17:20:49.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:49.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:49.046878+0000) 2024-03-06T17:20:50.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:50.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:50.047130+0000) 2024-03-06T17:20:51.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:51.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:51.047341+0000) 2024-03-06T17:20:52.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:52.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:52.047552+0000) 2024-03-06T17:20:53.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:53.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:53.047780+0000) 2024-03-06T17:20:54.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:54.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:54.048000+0000) 2024-03-06T17:20:55.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:55.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:55.048217+0000) 2024-03-06T17:20:56.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:56.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:56.048418+0000) 2024-03-06T17:20:57.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:57.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:57.048638+0000) 2024-03-06T17:20:57.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:58.004 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:20:58.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:58.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:58.048820+0000) 2024-03-06T17:20:59.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:20:59.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:20:59.048986+0000) 2024-03-06T17:21:00.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:00.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:00.049257+0000) 2024-03-06T17:21:01.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:01.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:01.049487+0000) 2024-03-06T17:21:02.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:02.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:02.049712+0000) 2024-03-06T17:21:03.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:03.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:03.049951+0000) 2024-03-06T17:21:04.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:04.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:04.050196+0000) 2024-03-06T17:21:05.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:05.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:05.050398+0000) 2024-03-06T17:21:06.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:06.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:06.050612+0000) 2024-03-06T17:21:07.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:07.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:07.050843+0000) 2024-03-06T17:21:08.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:08.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:08.051116+0000) 2024-03-06T17:21:09.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:09.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:09.051347+0000) 2024-03-06T17:21:10.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:10.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:10.051586+0000) 2024-03-06T17:21:11.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:11.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:11.051831+0000) 2024-03-06T17:21:12.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:12.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:12.052102+0000) 2024-03-06T17:21:13.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:13.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:13.052288+0000) 2024-03-06T17:21:14.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:14.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:14.052492+0000) 2024-03-06T17:21:15.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:15.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:15.052709+0000) 2024-03-06T17:21:16.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:16.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:16.052935+0000) 2024-03-06T17:21:17.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:17.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:17.053155+0000) 2024-03-06T17:21:18.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:18.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:18.053378+0000) 2024-03-06T17:21:19.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:19.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:19.053620+0000) 2024-03-06T17:21:20.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:20.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:20.053846+0000) 2024-03-06T17:21:21.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:21.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:21.054106+0000) 2024-03-06T17:21:22.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:22.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:22.054329+0000) 2024-03-06T17:21:23.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:23.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:23.054527+0000) 2024-03-06T17:21:24.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:24.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:24.054755+0000) 2024-03-06T17:21:25.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:25.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:25.054955+0000) 2024-03-06T17:21:26.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:26.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:26.055194+0000) 2024-03-06T17:21:27.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:27.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:27.055426+0000) 2024-03-06T17:21:28.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:28.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:28.055660+0000) 2024-03-06T17:21:28.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:28.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:29.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:29.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:29.055887+0000) 2024-03-06T17:21:30.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:30.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:30.056110+0000) 2024-03-06T17:21:31.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:31.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:31.056311+0000) 2024-03-06T17:21:32.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:32.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:32.056544+0000) 2024-03-06T17:21:33.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:33.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:33.056776+0000) 2024-03-06T17:21:34.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:34.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:34.057052+0000) 2024-03-06T17:21:35.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:35.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:35.057246+0000) 2024-03-06T17:21:36.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:36.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:36.057453+0000) 2024-03-06T17:21:37.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:37.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:37.057685+0000) 2024-03-06T17:21:38.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:38.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:38.057907+0000) 2024-03-06T17:21:39.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:39.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:39.058144+0000) 2024-03-06T17:21:40.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:40.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:40.058340+0000) 2024-03-06T17:21:41.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:41.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:41.058547+0000) 2024-03-06T17:21:42.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:42.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:42.058767+0000) 2024-03-06T17:21:43.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:43.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:43.058989+0000) 2024-03-06T17:21:44.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:44.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:44.059204+0000) 2024-03-06T17:21:45.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:45.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:45.059393+0000) 2024-03-06T17:21:46.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:46.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:46.059612+0000) 2024-03-06T17:21:47.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:47.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:47.059849+0000) 2024-03-06T17:21:48.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:48.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:48.060117+0000) 2024-03-06T17:21:49.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:49.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:49.060300+0000) 2024-03-06T17:21:50.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:50.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:50.060506+0000) 2024-03-06T17:21:51.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:51.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:51.060737+0000) 2024-03-06T17:21:52.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:52.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:52.060967+0000) 2024-03-06T17:21:53.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:53.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:53.061184+0000) 2024-03-06T17:21:54.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:54.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:54.061427+0000) 2024-03-06T17:21:55.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:55.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:55.061645+0000) 2024-03-06T17:21:56.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:56.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:56.061877+0000) 2024-03-06T17:21:57.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:57.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:57.062132+0000) 2024-03-06T17:21:58.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:58.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:58.062335+0000) 2024-03-06T17:21:58.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:58.331 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:21:59.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:21:59.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:21:59.062537+0000) 2024-03-06T17:22:00.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:00.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:00.062763+0000) 2024-03-06T17:22:01.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:01.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:01.063028+0000) 2024-03-06T17:22:02.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:02.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:02.063223+0000) 2024-03-06T17:22:03.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:03.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:03.063409+0000) 2024-03-06T17:22:04.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:04.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:04.063621+0000) 2024-03-06T17:22:05.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:05.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:05.063842+0000) 2024-03-06T17:22:06.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:06.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:06.064073+0000) 2024-03-06T17:22:07.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:07.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:07.064364+0000) 2024-03-06T17:22:08.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:08.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:08.064592+0000) 2024-03-06T17:22:09.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:09.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:09.064808+0000) 2024-03-06T17:22:10.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:10.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:10.065068+0000) 2024-03-06T17:22:11.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:11.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:11.065237+0000) 2024-03-06T17:22:12.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:12.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:12.065470+0000) 2024-03-06T17:22:13.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:13.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:13.065695+0000) 2024-03-06T17:22:14.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:14.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:14.065918+0000) 2024-03-06T17:22:15.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:15.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:15.066142+0000) 2024-03-06T17:22:16.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:16.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:16.066401+0000) 2024-03-06T17:22:17.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:17.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:17.066656+0000) 2024-03-06T17:22:18.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:18.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:18.066846+0000) 2024-03-06T17:22:19.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:19.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:19.067107+0000) 2024-03-06T17:22:20.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:20.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:20.067337+0000) 2024-03-06T17:22:21.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:21.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:21.067532+0000) 2024-03-06T17:22:22.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:22.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:22.067766+0000) 2024-03-06T17:22:23.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:23.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:23.067987+0000) 2024-03-06T17:22:24.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:24.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:24.068214+0000) 2024-03-06T17:22:25.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:25.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:25.068393+0000) 2024-03-06T17:22:26.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:26.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:26.068594+0000) 2024-03-06T17:22:27.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:27.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:27.068828+0000) 2024-03-06T17:22:28.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:28.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:28.069090+0000) 2024-03-06T17:22:28.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:28.486 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:29.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:29.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:29.069286+0000) 2024-03-06T17:22:30.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:30.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:30.069493+0000) 2024-03-06T17:22:31.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:31.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:31.069739+0000) 2024-03-06T17:22:32.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:32.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:32.069963+0000) 2024-03-06T17:22:33.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:33.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:33.070157+0000) 2024-03-06T17:22:34.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:34.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:34.070344+0000) 2024-03-06T17:22:35.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:35.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:35.070549+0000) 2024-03-06T17:22:36.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:36.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:36.070794+0000) 2024-03-06T17:22:37.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:37.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:37.071075+0000) 2024-03-06T17:22:38.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:38.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:38.071267+0000) 2024-03-06T17:22:39.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:39.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:39.071479+0000) 2024-03-06T17:22:40.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:40.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:40.071720+0000) 2024-03-06T17:22:41.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:41.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:41.071991+0000) 2024-03-06T17:22:42.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:42.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:42.072225+0000) 2024-03-06T17:22:43.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:43.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:43.072450+0000) 2024-03-06T17:22:44.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:44.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:44.072716+0000) 2024-03-06T17:22:45.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:45.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:45.072897+0000) 2024-03-06T17:22:46.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:46.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:46.073147+0000) 2024-03-06T17:22:47.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:47.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:47.073352+0000) 2024-03-06T17:22:48.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:48.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:48.073566+0000) 2024-03-06T17:22:49.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:49.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:49.073789+0000) 2024-03-06T17:22:50.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:50.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:50.074035+0000) 2024-03-06T17:22:51.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:51.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:51.074242+0000) 2024-03-06T17:22:52.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:52.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:52.074433+0000) 2024-03-06T17:22:53.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:53.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:53.074660+0000) 2024-03-06T17:22:54.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:54.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:54.074883+0000) 2024-03-06T17:22:55.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:55.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:55.075139+0000) 2024-03-06T17:22:56.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:56.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:56.075340+0000) 2024-03-06T17:22:57.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:57.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:57.075569+0000) 2024-03-06T17:22:58.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:58.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:58.075798+0000) 2024-03-06T17:22:58.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:58.644 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:22:59.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:22:59.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:22:59.075991+0000) 2024-03-06T17:23:00.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:00.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:00.076273+0000) 2024-03-06T17:23:01.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:01.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:01.076491+0000) 2024-03-06T17:23:02.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:02.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:02.076683+0000) 2024-03-06T17:23:03.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:03.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:03.076899+0000) 2024-03-06T17:23:04.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:04.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:04.077150+0000) 2024-03-06T17:23:05.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:05.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:05.077405+0000) 2024-03-06T17:23:06.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:06.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:06.077598+0000) 2024-03-06T17:23:07.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:07.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:07.077849+0000) 2024-03-06T17:23:08.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:08.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:08.078116+0000) 2024-03-06T17:23:09.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:09.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:09.078309+0000) 2024-03-06T17:23:10.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:10.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:10.078494+0000) 2024-03-06T17:23:11.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:11.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:11.078727+0000) 2024-03-06T17:23:12.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:12.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:12.078924+0000) 2024-03-06T17:23:13.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:13.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:13.079156+0000) 2024-03-06T17:23:14.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:14.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:14.079341+0000) 2024-03-06T17:23:15.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:15.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:15.079565+0000) 2024-03-06T17:23:16.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:16.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:16.079800+0000) 2024-03-06T17:23:17.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:17.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:17.080088+0000) 2024-03-06T17:23:18.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:18.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:18.080289+0000) 2024-03-06T17:23:19.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:19.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:19.080486+0000) 2024-03-06T17:23:20.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:20.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:20.080706+0000) 2024-03-06T17:23:21.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:21.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:21.080932+0000) 2024-03-06T17:23:22.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:22.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:22.081174+0000) 2024-03-06T17:23:23.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:23.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:23.081397+0000) 2024-03-06T17:23:24.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:24.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:24.081621+0000) 2024-03-06T17:23:25.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:25.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:25.081805+0000) 2024-03-06T17:23:26.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:26.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:26.082055+0000) 2024-03-06T17:23:27.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:27.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:27.082266+0000) 2024-03-06T17:23:28.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:28.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:28.082476+0000) 2024-03-06T17:23:28.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:28.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:29.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:29.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:29.082712+0000) 2024-03-06T17:23:30.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:30.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:30.082918+0000) 2024-03-06T17:23:31.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:31.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:31.083141+0000) 2024-03-06T17:23:32.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:32.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:32.083311+0000) 2024-03-06T17:23:33.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:33.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:33.083554+0000) 2024-03-06T17:23:34.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:34.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:34.083803+0000) 2024-03-06T17:23:35.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:35.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:35.084046+0000) 2024-03-06T17:23:36.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:36.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:36.084255+0000) 2024-03-06T17:23:37.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:37.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:37.084513+0000) 2024-03-06T17:23:38.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:38.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:38.084733+0000) 2024-03-06T17:23:39.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:39.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:39.084964+0000) 2024-03-06T17:23:40.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:40.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:40.085230+0000) 2024-03-06T17:23:41.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:41.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:41.085451+0000) 2024-03-06T17:23:42.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:42.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:42.085722+0000) 2024-03-06T17:23:43.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:43.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:43.085954+0000) 2024-03-06T17:23:44.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:44.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:44.086218+0000) 2024-03-06T17:23:45.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:45.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:45.086384+0000) 2024-03-06T17:23:46.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:46.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:46.086605+0000) 2024-03-06T17:23:47.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:47.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:47.086833+0000) 2024-03-06T17:23:48.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:48.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:48.087113+0000) 2024-03-06T17:23:49.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:49.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:49.087309+0000) 2024-03-06T17:23:50.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:50.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:50.087531+0000) 2024-03-06T17:23:51.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:51.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:51.087808+0000) 2024-03-06T17:23:52.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:52.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:52.088113+0000) 2024-03-06T17:23:53.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:53.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:53.088360+0000) 2024-03-06T17:23:54.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:54.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:54.088643+0000) 2024-03-06T17:23:55.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:55.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:55.088840+0000) 2024-03-06T17:23:56.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:56.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:56.089097+0000) 2024-03-06T17:23:57.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:57.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:57.089321+0000) 2024-03-06T17:23:58.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:58.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:58.089546+0000) 2024-03-06T17:23:59.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:59.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:23:59.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:23:59.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:23:59.089751+0000) 2024-03-06T17:24:00.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:00.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:00.090055+0000) 2024-03-06T17:24:01.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:01.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:01.090341+0000) 2024-03-06T17:24:02.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:02.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:02.090620+0000) 2024-03-06T17:24:03.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:03.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:03.090865+0000) 2024-03-06T17:24:04.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:04.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:04.091119+0000) 2024-03-06T17:24:05.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:05.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:05.091354+0000) 2024-03-06T17:24:06.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:06.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:06.091640+0000) 2024-03-06T17:24:07.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:07.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:07.091853+0000) 2024-03-06T17:24:08.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:08.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:08.092116+0000) 2024-03-06T17:24:09.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:09.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:09.092327+0000) 2024-03-06T17:24:10.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:10.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:10.092563+0000) 2024-03-06T17:24:11.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:11.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:11.092775+0000) 2024-03-06T17:24:12.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:12.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:12.092978+0000) 2024-03-06T17:24:13.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:13.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:13.093204+0000) 2024-03-06T17:24:14.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:14.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:14.093391+0000) 2024-03-06T17:24:15.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:15.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:15.093595+0000) 2024-03-06T17:24:16.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:16.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:16.093812+0000) 2024-03-06T17:24:17.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:17.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:17.094072+0000) 2024-03-06T17:24:18.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:18.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:18.094289+0000) 2024-03-06T17:24:19.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:19.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:19.094488+0000) 2024-03-06T17:24:20.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:20.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:20.094761+0000) 2024-03-06T17:24:21.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:21.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:21.095001+0000) 2024-03-06T17:24:22.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:22.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:22.095223+0000) 2024-03-06T17:24:23.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:23.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:23.095434+0000) 2024-03-06T17:24:24.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:24.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:24.095686+0000) 2024-03-06T17:24:25.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:25.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:25.095906+0000) 2024-03-06T17:24:26.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:26.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:26.096129+0000) 2024-03-06T17:24:27.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:27.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:27.096324+0000) 2024-03-06T17:24:28.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:28.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:28.096500+0000) 2024-03-06T17:24:29.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:29.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:29.096710+0000) 2024-03-06T17:24:29.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:29.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:30.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:30.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:30.096927+0000) 2024-03-06T17:24:31.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:31.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:31.097162+0000) 2024-03-06T17:24:32.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:32.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:32.097374+0000) 2024-03-06T17:24:33.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:33.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:33.097597+0000) 2024-03-06T17:24:34.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:34.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:34.097814+0000) 2024-03-06T17:24:35.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:35.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:35.098055+0000) 2024-03-06T17:24:36.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:36.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:36.098305+0000) 2024-03-06T17:24:37.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:37.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:37.098576+0000) 2024-03-06T17:24:38.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:38.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:38.098803+0000) 2024-03-06T17:24:39.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:39.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:39.099067+0000) 2024-03-06T17:24:40.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:40.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:40.099297+0000) 2024-03-06T17:24:41.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:41.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:41.099519+0000) 2024-03-06T17:24:42.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:42.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:42.099748+0000) 2024-03-06T17:24:43.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:43.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:43.099953+0000) 2024-03-06T17:24:44.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:44.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:44.100184+0000) 2024-03-06T17:24:45.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:45.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:45.100403+0000) 2024-03-06T17:24:46.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:46.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:46.100620+0000) 2024-03-06T17:24:47.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:47.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:47.100852+0000) 2024-03-06T17:24:48.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:48.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:48.101109+0000) 2024-03-06T17:24:49.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:49.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:49.101336+0000) 2024-03-06T17:24:50.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:50.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:50.101534+0000) 2024-03-06T17:24:51.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:51.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:51.101752+0000) 2024-03-06T17:24:52.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:52.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:52.101983+0000) 2024-03-06T17:24:53.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:53.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:53.102189+0000) 2024-03-06T17:24:54.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:54.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:54.102424+0000) 2024-03-06T17:24:55.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:55.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:55.102668+0000) 2024-03-06T17:24:56.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:56.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:56.102906+0000) 2024-03-06T17:24:57.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:57.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:57.103158+0000) 2024-03-06T17:24:58.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:58.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:58.103390+0000) 2024-03-06T17:24:59.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:24:59.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:24:59.103600+0000) 2024-03-06T17:24:59.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:24:59.488 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:00.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:00.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:00.103821+0000) 2024-03-06T17:25:01.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:01.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:01.104046+0000) 2024-03-06T17:25:02.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:02.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:02.104271+0000) 2024-03-06T17:25:03.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:03.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:03.104489+0000) 2024-03-06T17:25:04.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:04.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:04.104711+0000) 2024-03-06T17:25:05.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:05.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:05.104939+0000) 2024-03-06T17:25:06.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:06.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:06.105160+0000) 2024-03-06T17:25:07.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:07.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:07.105391+0000) 2024-03-06T17:25:08.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:08.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:08.105617+0000) 2024-03-06T17:25:09.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:09.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:09.105851+0000) 2024-03-06T17:25:10.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:10.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:10.106102+0000) 2024-03-06T17:25:11.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:11.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:11.106325+0000) 2024-03-06T17:25:12.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:12.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:12.106550+0000) 2024-03-06T17:25:13.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:13.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:13.106785+0000) 2024-03-06T17:25:14.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:14.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:14.107066+0000) 2024-03-06T17:25:15.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:15.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:15.107311+0000) 2024-03-06T17:25:16.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:16.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:16.107531+0000) 2024-03-06T17:25:17.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:17.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:17.107791+0000) 2024-03-06T17:25:18.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:18.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:18.107978+0000) 2024-03-06T17:25:19.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:19.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:19.108178+0000) 2024-03-06T17:25:20.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:20.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:20.108402+0000) 2024-03-06T17:25:21.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:21.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:21.108624+0000) 2024-03-06T17:25:22.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:22.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:22.108852+0000) 2024-03-06T17:25:23.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:23.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:23.109108+0000) 2024-03-06T17:25:24.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:24.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:24.109385+0000) 2024-03-06T17:25:25.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:25.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:25.109600+0000) 2024-03-06T17:25:26.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:26.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:26.109806+0000) 2024-03-06T17:25:27.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:27.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:27.110081+0000) 2024-03-06T17:25:28.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:28.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:28.110308+0000) 2024-03-06T17:25:29.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:29.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:29.110506+0000) 2024-03-06T17:25:29.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:29.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:30.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:30.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:30.110748+0000) 2024-03-06T17:25:31.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:31.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:31.110931+0000) 2024-03-06T17:25:32.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:32.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:32.111159+0000) 2024-03-06T17:25:33.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:33.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:33.111340+0000) 2024-03-06T17:25:34.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:34.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:34.111560+0000) 2024-03-06T17:25:35.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:35.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:35.111747+0000) 2024-03-06T17:25:36.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:36.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:36.111970+0000) 2024-03-06T17:25:37.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:37.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:37.112167+0000) 2024-03-06T17:25:38.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:38.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:38.112415+0000) 2024-03-06T17:25:39.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:39.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:39.112658+0000) 2024-03-06T17:25:40.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:40.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:40.112919+0000) 2024-03-06T17:25:41.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:41.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:41.113169+0000) 2024-03-06T17:25:42.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:42.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:42.113349+0000) 2024-03-06T17:25:43.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:43.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:43.113555+0000) 2024-03-06T17:25:44.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:44.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:44.113772+0000) 2024-03-06T17:25:45.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:45.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:45.114022+0000) 2024-03-06T17:25:46.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:46.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:46.114216+0000) 2024-03-06T17:25:47.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:47.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:47.114422+0000) 2024-03-06T17:25:48.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:48.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:48.114632+0000) 2024-03-06T17:25:49.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:49.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:49.114861+0000) 2024-03-06T17:25:50.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:50.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:50.115150+0000) 2024-03-06T17:25:51.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:51.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:51.115387+0000) 2024-03-06T17:25:52.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:52.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:52.115598+0000) 2024-03-06T17:25:53.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:53.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:53.115815+0000) 2024-03-06T17:25:54.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:54.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:54.116090+0000) 2024-03-06T17:25:55.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:55.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:55.116321+0000) 2024-03-06T17:25:56.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:56.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:56.116567+0000) 2024-03-06T17:25:57.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:57.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:57.116805+0000) 2024-03-06T17:25:58.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:58.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:58.117058+0000) 2024-03-06T17:25:59.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:25:59.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:25:59.117279+0000) 2024-03-06T17:25:59.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:25:59.897 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:00.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:00.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:00.117490+0000) 2024-03-06T17:26:01.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:01.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:01.117713+0000) 2024-03-06T17:26:02.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:02.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:02.117944+0000) 2024-03-06T17:26:03.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:03.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:03.118153+0000) 2024-03-06T17:26:04.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:04.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:04.118341+0000) 2024-03-06T17:26:05.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:05.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:05.118563+0000) 2024-03-06T17:26:06.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:06.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:06.118803+0000) 2024-03-06T17:26:07.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:07.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:07.119099+0000) 2024-03-06T17:26:08.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:08.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:08.119339+0000) 2024-03-06T17:26:09.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:09.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:09.119589+0000) 2024-03-06T17:26:10.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:10.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:10.119818+0000) 2024-03-06T17:26:11.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:11.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:11.120118+0000) 2024-03-06T17:26:12.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:12.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:12.120397+0000) 2024-03-06T17:26:13.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:13.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:13.120632+0000) 2024-03-06T17:26:14.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:14.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:14.120908+0000) 2024-03-06T17:26:15.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:15.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:15.121175+0000) 2024-03-06T17:26:16.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:16.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:16.121404+0000) 2024-03-06T17:26:17.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:17.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:17.121709+0000) 2024-03-06T17:26:18.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:18.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:18.121934+0000) 2024-03-06T17:26:19.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:19.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:19.122197+0000) 2024-03-06T17:26:20.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:20.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:20.122369+0000) 2024-03-06T17:26:21.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:21.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:21.122553+0000) 2024-03-06T17:26:22.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:22.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:22.122735+0000) 2024-03-06T17:26:23.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:23.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:23.122958+0000) 2024-03-06T17:26:24.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:24.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:24.123182+0000) 2024-03-06T17:26:25.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:25.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:25.123333+0000) 2024-03-06T17:26:26.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:26.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:26.123523+0000) 2024-03-06T17:26:27.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:27.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:27.123754+0000) 2024-03-06T17:26:28.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:28.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:28.123948+0000) 2024-03-06T17:26:29.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:29.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:29.124146+0000) 2024-03-06T17:26:30.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:30.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:26:30.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:30.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:30.124319+0000) 2024-03-06T17:26:31.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:31.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:31.124510+0000) 2024-03-06T17:26:32.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:32.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:32.124748+0000) 2024-03-06T17:26:33.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:33.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:33.124987+0000) 2024-03-06T17:26:34.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:34.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:34.125253+0000) 2024-03-06T17:26:35.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:35.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:35.125471+0000) 2024-03-06T17:26:36.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:36.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:36.125693+0000) 2024-03-06T17:26:37.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:37.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:37.125932+0000) 2024-03-06T17:26:38.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:38.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:38.126154+0000) 2024-03-06T17:26:39.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:39.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:39.126376+0000) 2024-03-06T17:26:40.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:40.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:40.126601+0000) 2024-03-06T17:26:41.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:41.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:41.126818+0000) 2024-03-06T17:26:42.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:42.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:42.127083+0000) 2024-03-06T17:26:43.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:43.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:43.127283+0000) 2024-03-06T17:26:44.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:44.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:44.127509+0000) 2024-03-06T17:26:45.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:45.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:45.127725+0000) 2024-03-06T17:26:46.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:46.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:46.127981+0000) 2024-03-06T17:26:47.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:47.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:47.128342+0000) 2024-03-06T17:26:48.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:48.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:48.128575+0000) 2024-03-06T17:26:49.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:49.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:49.128804+0000) 2024-03-06T17:26:50.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:50.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:50.129080+0000) 2024-03-06T17:26:51.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:51.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:51.129227+0000) 2024-03-06T17:26:52.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:52.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:52.129395+0000) 2024-03-06T17:26:53.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:53.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:53.129595+0000) 2024-03-06T17:26:54.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:54.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:54.129810+0000) 2024-03-06T17:26:55.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:55.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:55.130090+0000) 2024-03-06T17:26:56.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:56.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:56.130331+0000) 2024-03-06T17:26:57.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:57.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:57.130555+0000) 2024-03-06T17:26:58.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:58.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:58.130795+0000) 2024-03-06T17:26:59.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:26:59.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:26:59.131042+0000) 2024-03-06T17:27:00.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:00.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:00.131271+0000) 2024-03-06T17:27:00.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:00.322 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:01.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:01.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:01.131504+0000) 2024-03-06T17:27:02.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:02.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:02.131740+0000) 2024-03-06T17:27:03.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:03.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:03.131945+0000) 2024-03-06T17:27:04.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:04.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:04.132154+0000) 2024-03-06T17:27:05.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:05.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:05.132343+0000) 2024-03-06T17:27:06.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:06.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:06.132476+0000) 2024-03-06T17:27:07.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:07.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:07.132707+0000) 2024-03-06T17:27:08.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:08.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:08.132966+0000) 2024-03-06T17:27:09.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:09.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:09.133285+0000) 2024-03-06T17:27:10.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:10.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:10.133571+0000) 2024-03-06T17:27:11.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:11.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:11.133825+0000) 2024-03-06T17:27:12.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:12.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:12.134141+0000) 2024-03-06T17:27:13.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:13.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:13.134305+0000) 2024-03-06T17:27:14.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:14.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:14.134564+0000) 2024-03-06T17:27:15.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:15.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:15.134837+0000) 2024-03-06T17:27:16.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:16.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:16.135129+0000) 2024-03-06T17:27:17.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:17.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:17.135317+0000) 2024-03-06T17:27:18.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:18.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:18.135544+0000) 2024-03-06T17:27:19.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:19.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:19.135820+0000) 2024-03-06T17:27:20.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:20.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:20.136128+0000) 2024-03-06T17:27:21.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:21.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:21.136399+0000) 2024-03-06T17:27:22.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:22.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:22.136688+0000) 2024-03-06T17:27:23.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:23.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:23.136926+0000) 2024-03-06T17:27:24.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:24.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:24.137157+0000) 2024-03-06T17:27:25.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:25.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:25.137401+0000) 2024-03-06T17:27:26.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:26.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:26.137608+0000) 2024-03-06T17:27:27.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:27.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:27.137889+0000) 2024-03-06T17:27:28.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:28.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:28.138195+0000) 2024-03-06T17:27:29.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:29.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:29.138432+0000) 2024-03-06T17:27:30.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:30.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:30.138711+0000) 2024-03-06T17:27:30.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:30.487 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:27:31.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:31.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:31.138904+0000) 2024-03-06T17:27:32.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:32.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:32.139154+0000) 2024-03-06T17:27:33.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:33.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:33.139347+0000) 2024-03-06T17:27:34.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:34.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:34.139576+0000) 2024-03-06T17:27:35.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:35.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:35.139797+0000) 2024-03-06T17:27:36.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:36.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:36.140111+0000) 2024-03-06T17:27:37.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:37.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:37.140407+0000) 2024-03-06T17:27:38.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:38.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:38.140671+0000) 2024-03-06T17:27:39.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:39.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:39.140947+0000) 2024-03-06T17:27:40.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:40.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:40.141150+0000) 2024-03-06T17:27:41.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:41.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:41.141374+0000) 2024-03-06T17:27:42.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:42.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:42.141617+0000) 2024-03-06T17:27:43.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:43.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:43.141840+0000) 2024-03-06T17:27:44.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:44.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:44.142094+0000) 2024-03-06T17:27:45.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:45.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:45.142281+0000) 2024-03-06T17:27:46.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:46.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:46.142508+0000) 2024-03-06T17:27:47.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:47.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:47.142711+0000) 2024-03-06T17:27:48.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:48.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:48.142930+0000) 2024-03-06T17:27:49.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:49.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:49.143198+0000) 2024-03-06T17:27:50.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:50.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:50.143484+0000) 2024-03-06T17:27:51.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:51.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:51.143767+0000) 2024-03-06T17:27:52.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:52.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:52.144081+0000) 2024-03-06T17:27:53.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:53.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:53.144355+0000) 2024-03-06T17:27:54.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:54.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:54.144635+0000) 2024-03-06T17:27:55.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:55.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:55.144926+0000) 2024-03-06T17:27:56.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:56.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:56.145165+0000) 2024-03-06T17:27:57.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:57.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:57.145412+0000) 2024-03-06T17:27:58.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:58.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:58.145590+0000) 2024-03-06T17:27:59.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:27:59.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:27:59.145799+0000) 2024-03-06T17:28:00.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:00.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:00.146062+0000) 2024-03-06T17:28:00.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:00.656 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:01.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:01.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:01.146254+0000) 2024-03-06T17:28:02.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:02.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:02.146491+0000) 2024-03-06T17:28:03.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:03.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:03.146692+0000) 2024-03-06T17:28:04.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:04.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:04.146917+0000) 2024-03-06T17:28:05.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:05.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:05.147156+0000) 2024-03-06T17:28:06.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:06.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:06.147350+0000) 2024-03-06T17:28:07.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:07.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:07.147588+0000) 2024-03-06T17:28:08.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:08.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:08.147812+0000) 2024-03-06T17:28:09.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:09.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:09.148064+0000) 2024-03-06T17:28:10.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:10.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:10.148295+0000) 2024-03-06T17:28:11.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:11.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:11.148512+0000) 2024-03-06T17:28:12.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:12.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:12.148755+0000) 2024-03-06T17:28:13.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:13.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:13.149035+0000) 2024-03-06T17:28:14.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:14.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:14.149242+0000) 2024-03-06T17:28:15.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:15.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:15.149443+0000) 2024-03-06T17:28:16.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:16.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:16.149648+0000) 2024-03-06T17:28:17.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:17.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:17.149898+0000) 2024-03-06T17:28:18.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:18.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:18.150142+0000) 2024-03-06T17:28:19.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:19.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:19.150335+0000) 2024-03-06T17:28:20.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:20.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:20.150547+0000) 2024-03-06T17:28:21.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:21.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:21.150777+0000) 2024-03-06T17:28:22.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:22.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:22.151028+0000) 2024-03-06T17:28:23.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:23.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:23.151226+0000) 2024-03-06T17:28:24.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:24.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:24.151419+0000) 2024-03-06T17:28:25.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:25.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:25.151624+0000) 2024-03-06T17:28:26.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:26.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:26.151848+0000) 2024-03-06T17:28:27.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:27.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:27.152115+0000) 2024-03-06T17:28:28.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:28.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:28.152329+0000) 2024-03-06T17:28:29.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:29.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:29.152564+0000) 2024-03-06T17:28:30.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:30.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:30.152792+0000) 2024-03-06T17:28:30.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:30.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:28:31.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:31.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:31.152986+0000) 2024-03-06T17:28:32.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:32.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:32.153214+0000) 2024-03-06T17:28:33.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:33.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:33.153493+0000) 2024-03-06T17:28:34.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:34.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:34.153775+0000) 2024-03-06T17:28:35.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:35.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:35.154101+0000) 2024-03-06T17:28:36.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:36.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:36.154360+0000) 2024-03-06T17:28:37.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:37.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:37.154665+0000) 2024-03-06T17:28:38.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:38.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:38.154945+0000) 2024-03-06T17:28:39.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:39.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:39.155223+0000) 2024-03-06T17:28:40.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:40.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:40.155447+0000) 2024-03-06T17:28:41.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:41.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:41.155686+0000) 2024-03-06T17:28:42.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:42.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:42.155884+0000) 2024-03-06T17:28:43.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:43.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:43.156090+0000) 2024-03-06T17:28:44.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:44.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:44.156282+0000) 2024-03-06T17:28:45.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:45.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:45.156470+0000) 2024-03-06T17:28:46.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:46.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:46.156694+0000) 2024-03-06T17:28:47.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:47.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:47.157041+0000) 2024-03-06T17:28:48.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:48.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:48.157282+0000) 2024-03-06T17:28:49.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:49.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:49.157517+0000) 2024-03-06T17:28:50.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:50.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:50.157738+0000) 2024-03-06T17:28:51.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:51.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:51.157982+0000) 2024-03-06T17:28:52.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:52.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:52.158315+0000) 2024-03-06T17:28:53.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:53.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:53.158591+0000) 2024-03-06T17:28:54.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:54.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:54.158871+0000) 2024-03-06T17:28:55.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:55.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:55.159113+0000) 2024-03-06T17:28:56.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:56.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:56.159327+0000) 2024-03-06T17:28:57.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:57.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:57.159545+0000) 2024-03-06T17:28:58.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:58.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:58.159742+0000) 2024-03-06T17:28:59.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:28:59.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:28:59.159972+0000) 2024-03-06T17:29:00.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:00.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:00.160159+0000) 2024-03-06T17:29:01.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:01.055 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:01.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:01.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:01.160370+0000) 2024-03-06T17:29:02.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:02.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:02.160607+0000) 2024-03-06T17:29:03.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:03.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:03.160788+0000) 2024-03-06T17:29:04.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:04.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:04.161037+0000) 2024-03-06T17:29:05.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:05.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:05.161270+0000) 2024-03-06T17:29:06.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:06.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:06.161564+0000) 2024-03-06T17:29:07.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:07.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:07.161858+0000) 2024-03-06T17:29:08.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:08.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:08.162078+0000) 2024-03-06T17:29:09.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:09.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:09.162287+0000) 2024-03-06T17:29:10.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:10.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:10.162500+0000) 2024-03-06T17:29:11.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:11.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:11.162681+0000) 2024-03-06T17:29:12.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:12.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:12.162894+0000) 2024-03-06T17:29:13.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:13.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:13.163124+0000) 2024-03-06T17:29:14.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:14.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:14.163343+0000) 2024-03-06T17:29:15.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:15.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:15.163543+0000) 2024-03-06T17:29:16.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:16.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:16.163762+0000) 2024-03-06T17:29:17.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:17.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:17.164037+0000) 2024-03-06T17:29:18.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:18.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:18.164232+0000) 2024-03-06T17:29:19.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:19.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:19.164429+0000) 2024-03-06T17:29:20.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:20.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:20.164612+0000) 2024-03-06T17:29:21.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:21.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:21.164859+0000) 2024-03-06T17:29:22.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:22.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:22.165172+0000) 2024-03-06T17:29:23.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:23.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:23.165449+0000) 2024-03-06T17:29:24.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:24.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:24.165726+0000) 2024-03-06T17:29:25.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:25.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:25.165995+0000) 2024-03-06T17:29:26.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:26.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:26.166313+0000) 2024-03-06T17:29:27.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:27.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:27.166603+0000) 2024-03-06T17:29:28.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:28.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:28.166826+0000) 2024-03-06T17:29:29.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:29.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:29.167090+0000) 2024-03-06T17:29:30.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:30.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:30.167315+0000) 2024-03-06T17:29:31.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:31.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:31.167519+0000) 2024-03-06T17:29:31.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:31.268 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:29:32.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:32.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:32.167727+0000) 2024-03-06T17:29:33.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:33.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:33.167972+0000) 2024-03-06T17:29:34.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:34.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:34.168146+0000) 2024-03-06T17:29:35.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:35.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:35.168376+0000) 2024-03-06T17:29:36.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:36.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:36.168596+0000) 2024-03-06T17:29:37.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:37.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:37.168812+0000) 2024-03-06T17:29:38.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:38.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:38.169062+0000) 2024-03-06T17:29:39.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:39.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:39.169263+0000) 2024-03-06T17:29:40.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:40.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:40.169478+0000) 2024-03-06T17:29:41.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:41.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:41.169661+0000) 2024-03-06T17:29:42.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:42.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:42.169883+0000) 2024-03-06T17:29:43.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:43.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:43.170116+0000) 2024-03-06T17:29:44.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:44.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:44.170298+0000) 2024-03-06T17:29:45.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:45.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:45.170566+0000) 2024-03-06T17:29:46.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:46.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:46.170842+0000) 2024-03-06T17:29:47.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:47.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:47.171082+0000) 2024-03-06T17:29:48.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:48.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:48.171279+0000) 2024-03-06T17:29:49.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:49.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:49.171484+0000) 2024-03-06T17:29:50.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:50.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:50.171767+0000) 2024-03-06T17:29:51.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:51.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:51.172066+0000) 2024-03-06T17:29:52.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:52.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:52.172346+0000) 2024-03-06T17:29:53.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:53.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:53.172615+0000) 2024-03-06T17:29:54.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:54.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:54.172895+0000) 2024-03-06T17:29:55.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:55.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:55.173169+0000) 2024-03-06T17:29:56.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:56.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:56.173444+0000) 2024-03-06T17:29:57.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:57.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:57.173727+0000) 2024-03-06T17:29:58.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:58.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:58.174000+0000) 2024-03-06T17:29:59.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:29:59.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:29:59.174284+0000) 2024-03-06T17:30:00.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:00.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:00.174566+0000) 2024-03-06T17:30:01.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:01.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:01.174830+0000) 2024-03-06T17:30:01.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:01.411 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:02.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:02.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:02.175148+0000) 2024-03-06T17:30:03.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:03.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:03.175397+0000) 2024-03-06T17:30:04.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:04.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:04.175666+0000) 2024-03-06T17:30:05.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:05.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:05.175947+0000) 2024-03-06T17:30:06.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:06.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:06.176229+0000) 2024-03-06T17:30:07.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:07.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:07.176500+0000) 2024-03-06T17:30:08.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:08.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:08.176775+0000) 2024-03-06T17:30:09.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:09.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:09.177098+0000) 2024-03-06T17:30:10.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:10.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:10.177317+0000) 2024-03-06T17:30:11.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:11.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:11.177514+0000) 2024-03-06T17:30:12.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:12.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:12.177748+0000) 2024-03-06T17:30:13.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:13.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:13.178052+0000) 2024-03-06T17:30:14.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:14.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:14.178342+0000) 2024-03-06T17:30:15.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:15.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:15.178623+0000) 2024-03-06T17:30:16.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:16.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:16.178906+0000) 2024-03-06T17:30:17.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:17.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:17.179204+0000) 2024-03-06T17:30:18.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:18.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:18.179450+0000) 2024-03-06T17:30:19.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:19.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:19.179646+0000) 2024-03-06T17:30:20.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:20.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:20.179832+0000) 2024-03-06T17:30:21.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:21.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:21.180071+0000) 2024-03-06T17:30:22.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:22.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:22.180297+0000) 2024-03-06T17:30:23.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:23.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:23.180623+0000) 2024-03-06T17:30:24.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:24.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:24.180845+0000) 2024-03-06T17:30:25.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:25.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:25.181133+0000) 2024-03-06T17:30:26.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:26.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:26.181345+0000) 2024-03-06T17:30:27.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:27.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:27.181573+0000) 2024-03-06T17:30:28.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:28.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:28.181789+0000) 2024-03-06T17:30:29.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:29.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:29.181978+0000) 2024-03-06T17:30:30.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:30.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:30.182168+0000) 2024-03-06T17:30:31.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:31.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:31.182346+0000) 2024-03-06T17:30:31.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:31.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:30:32.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:32.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:32.182549+0000) 2024-03-06T17:30:33.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:33.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:33.182795+0000) 2024-03-06T17:30:34.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:34.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:34.183038+0000) 2024-03-06T17:30:35.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:35.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:35.183227+0000) 2024-03-06T17:30:36.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:36.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:36.183451+0000) 2024-03-06T17:30:37.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:37.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:37.183741+0000) 2024-03-06T17:30:38.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:38.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:38.184069+0000) 2024-03-06T17:30:39.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:39.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:39.184350+0000) 2024-03-06T17:30:40.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:40.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:40.184595+0000) 2024-03-06T17:30:41.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:41.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:41.184803+0000) 2024-03-06T17:30:42.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:42.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:42.185075+0000) 2024-03-06T17:30:43.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:43.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:43.185315+0000) 2024-03-06T17:30:44.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:44.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:44.185534+0000) 2024-03-06T17:30:45.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:45.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:45.185760+0000) 2024-03-06T17:30:46.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:46.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:46.185988+0000) 2024-03-06T17:30:47.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:47.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:47.186219+0000) 2024-03-06T17:30:48.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:48.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:48.186419+0000) 2024-03-06T17:30:49.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:49.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:49.186644+0000) 2024-03-06T17:30:50.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:50.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:50.186923+0000) 2024-03-06T17:30:51.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:51.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:51.187154+0000) 2024-03-06T17:30:52.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:52.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:52.187362+0000) 2024-03-06T17:30:53.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:53.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:53.187599+0000) 2024-03-06T17:30:54.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:54.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:54.187784+0000) 2024-03-06T17:30:55.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:55.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:55.188033+0000) 2024-03-06T17:30:56.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:56.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:56.188261+0000) 2024-03-06T17:30:57.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:57.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:57.188488+0000) 2024-03-06T17:30:58.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:58.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:58.188749+0000) 2024-03-06T17:30:59.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:30:59.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:30:59.188969+0000) 2024-03-06T17:31:00.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:00.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:00.189167+0000) 2024-03-06T17:31:01.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:01.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:01.189355+0000) 2024-03-06T17:31:01.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:01.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:02.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:02.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:02.189593+0000) 2024-03-06T17:31:03.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:03.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:03.189834+0000) 2024-03-06T17:31:04.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:04.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:04.190104+0000) 2024-03-06T17:31:05.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:05.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:05.190327+0000) 2024-03-06T17:31:06.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:06.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:06.190507+0000) 2024-03-06T17:31:07.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:07.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:07.190740+0000) 2024-03-06T17:31:08.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:08.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:08.190972+0000) 2024-03-06T17:31:09.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:09.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:09.191163+0000) 2024-03-06T17:31:10.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:10.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:10.191310+0000) 2024-03-06T17:31:11.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:11.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:11.191534+0000) 2024-03-06T17:31:12.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:12.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:12.191757+0000) 2024-03-06T17:31:13.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:13.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:13.191939+0000) 2024-03-06T17:31:14.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:14.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:14.192168+0000) 2024-03-06T17:31:15.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:15.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:15.192376+0000) 2024-03-06T17:31:16.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:16.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:16.192597+0000) 2024-03-06T17:31:17.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:17.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:17.192827+0000) 2024-03-06T17:31:18.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:18.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:18.193082+0000) 2024-03-06T17:31:19.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:19.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:19.193267+0000) 2024-03-06T17:31:20.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:20.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:20.193477+0000) 2024-03-06T17:31:21.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:21.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:21.193732+0000) 2024-03-06T17:31:22.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:22.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:22.193953+0000) 2024-03-06T17:31:23.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:23.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:23.194168+0000) 2024-03-06T17:31:24.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:24.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:24.194421+0000) 2024-03-06T17:31:25.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:25.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:25.194624+0000) 2024-03-06T17:31:26.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:26.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:26.194860+0000) 2024-03-06T17:31:27.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:27.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:27.195115+0000) 2024-03-06T17:31:28.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:28.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:28.195389+0000) 2024-03-06T17:31:29.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:29.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:29.195648+0000) 2024-03-06T17:31:30.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:30.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:30.195874+0000) 2024-03-06T17:31:31.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:31.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:31.196127+0000) 2024-03-06T17:31:31.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:31.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:31:32.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:32.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:32.196334+0000) 2024-03-06T17:31:33.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:33.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:33.196572+0000) 2024-03-06T17:31:34.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:34.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:34.196813+0000) 2024-03-06T17:31:35.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:35.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:35.197092+0000) 2024-03-06T17:31:36.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:36.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:36.197316+0000) 2024-03-06T17:31:37.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:37.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:37.197549+0000) 2024-03-06T17:31:38.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:38.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:38.197807+0000) 2024-03-06T17:31:39.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:39.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:39.198087+0000) 2024-03-06T17:31:40.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:40.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:40.198279+0000) 2024-03-06T17:31:41.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:41.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:41.198483+0000) 2024-03-06T17:31:42.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:42.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:42.198675+0000) 2024-03-06T17:31:43.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:43.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:43.198904+0000) 2024-03-06T17:31:44.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:44.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:44.199135+0000) 2024-03-06T17:31:45.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:45.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:45.199322+0000) 2024-03-06T17:31:46.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:46.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:46.199540+0000) 2024-03-06T17:31:47.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:47.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:47.199767+0000) 2024-03-06T17:31:48.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:48.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:48.199995+0000) 2024-03-06T17:31:49.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:49.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:49.200258+0000) 2024-03-06T17:31:50.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:50.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:50.200460+0000) 2024-03-06T17:31:51.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:51.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:51.200682+0000) 2024-03-06T17:31:52.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:52.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:52.200907+0000) 2024-03-06T17:31:53.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:53.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:53.201141+0000) 2024-03-06T17:31:54.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:54.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:54.201364+0000) 2024-03-06T17:31:55.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:55.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:55.201559+0000) 2024-03-06T17:31:56.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:56.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:56.201792+0000) 2024-03-06T17:31:57.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:57.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:57.202109+0000) 2024-03-06T17:31:58.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:58.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:58.202335+0000) 2024-03-06T17:31:59.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:31:59.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:31:59.202524+0000) 2024-03-06T17:32:00.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:00.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:00.202761+0000) 2024-03-06T17:32:01.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:01.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:01.202952+0000) 2024-03-06T17:32:02.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:02.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:02.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:02.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:02.203131+0000) 2024-03-06T17:32:03.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:03.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:03.203367+0000) 2024-03-06T17:32:04.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:04.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:04.203610+0000) 2024-03-06T17:32:05.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:05.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:05.203788+0000) 2024-03-06T17:32:06.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:06.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:06.203995+0000) 2024-03-06T17:32:07.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:07.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:07.204259+0000) 2024-03-06T17:32:08.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:08.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:08.204507+0000) 2024-03-06T17:32:09.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:09.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:09.204811+0000) 2024-03-06T17:32:10.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:10.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:10.205092+0000) 2024-03-06T17:32:11.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:11.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:11.205324+0000) 2024-03-06T17:32:12.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:12.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:12.205566+0000) 2024-03-06T17:32:13.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:13.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:13.205811+0000) 2024-03-06T17:32:14.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:14.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:14.206097+0000) 2024-03-06T17:32:15.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:15.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:15.206335+0000) 2024-03-06T17:32:16.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:16.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:16.206557+0000) 2024-03-06T17:32:17.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:17.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:17.206819+0000) 2024-03-06T17:32:18.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:18.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:18.207110+0000) 2024-03-06T17:32:19.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:19.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:19.207297+0000) 2024-03-06T17:32:20.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:20.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:20.207519+0000) 2024-03-06T17:32:21.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:21.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:21.207706+0000) 2024-03-06T17:32:22.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:22.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:22.207947+0000) 2024-03-06T17:32:23.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:23.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:23.208192+0000) 2024-03-06T17:32:24.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:24.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:24.208365+0000) 2024-03-06T17:32:25.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:25.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:25.208570+0000) 2024-03-06T17:32:26.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:26.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:26.208797+0000) 2024-03-06T17:32:27.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:27.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:27.209057+0000) 2024-03-06T17:32:28.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:28.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:28.209297+0000) 2024-03-06T17:32:29.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:29.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:29.209522+0000) 2024-03-06T17:32:30.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:30.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:30.209745+0000) 2024-03-06T17:32:31.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:31.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:31.209978+0000) 2024-03-06T17:32:32.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:32.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:32.210232+0000) 2024-03-06T17:32:32.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:32.309 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:32:33.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:33.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:33.210450+0000) 2024-03-06T17:32:34.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:34.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:34.210674+0000) 2024-03-06T17:32:35.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:35.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:35.210902+0000) 2024-03-06T17:32:36.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:36.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:36.211168+0000) 2024-03-06T17:32:37.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:37.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:37.211376+0000) 2024-03-06T17:32:38.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:38.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:38.211586+0000) 2024-03-06T17:32:39.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:39.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:39.211800+0000) 2024-03-06T17:32:40.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:40.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:40.212077+0000) 2024-03-06T17:32:41.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:41.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:41.212277+0000) 2024-03-06T17:32:42.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:42.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:42.212459+0000) 2024-03-06T17:32:43.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:43.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:43.212709+0000) 2024-03-06T17:32:44.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:44.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:44.212966+0000) 2024-03-06T17:32:45.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:45.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:45.213263+0000) 2024-03-06T17:32:46.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:46.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:46.213450+0000) 2024-03-06T17:32:47.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:47.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:47.213660+0000) 2024-03-06T17:32:48.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:48.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:48.213903+0000) 2024-03-06T17:32:49.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:49.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:49.214144+0000) 2024-03-06T17:32:50.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:50.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:50.214339+0000) 2024-03-06T17:32:51.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:51.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:51.214542+0000) 2024-03-06T17:32:52.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:52.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:52.214821+0000) 2024-03-06T17:32:53.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:53.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:53.215099+0000) 2024-03-06T17:32:54.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:54.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:54.215297+0000) 2024-03-06T17:32:55.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:55.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:55.215499+0000) 2024-03-06T17:32:56.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:56.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:56.215689+0000) 2024-03-06T17:32:57.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:57.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:57.215914+0000) 2024-03-06T17:32:58.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:58.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:58.216089+0000) 2024-03-06T17:32:59.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:32:59.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:32:59.216316+0000) 2024-03-06T17:33:00.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:00.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:00.216544+0000) 2024-03-06T17:33:01.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:01.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:01.216783+0000) 2024-03-06T17:33:02.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:02.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:02.217037+0000) 2024-03-06T17:33:02.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:02.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:03.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:03.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:03.217247+0000) 2024-03-06T17:33:04.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:04.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:04.217484+0000) 2024-03-06T17:33:05.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:05.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:05.217708+0000) 2024-03-06T17:33:06.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:06.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:06.217936+0000) 2024-03-06T17:33:07.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:07.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:07.218168+0000) 2024-03-06T17:33:08.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:08.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:08.218402+0000) 2024-03-06T17:33:09.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:09.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:09.218567+0000) 2024-03-06T17:33:10.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:10.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:10.218783+0000) 2024-03-06T17:33:11.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:11.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:11.219029+0000) 2024-03-06T17:33:12.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:12.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:12.219221+0000) 2024-03-06T17:33:13.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:13.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:13.219422+0000) 2024-03-06T17:33:14.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:14.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:14.219607+0000) 2024-03-06T17:33:15.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:15.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:15.219829+0000) 2024-03-06T17:33:16.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:16.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:16.220097+0000) 2024-03-06T17:33:17.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:17.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:17.220336+0000) 2024-03-06T17:33:18.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:18.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:18.220587+0000) 2024-03-06T17:33:19.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:19.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:19.220802+0000) 2024-03-06T17:33:20.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:20.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:20.221043+0000) 2024-03-06T17:33:21.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:21.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:21.221229+0000) 2024-03-06T17:33:22.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:22.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:22.221401+0000) 2024-03-06T17:33:23.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:23.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:23.221593+0000) 2024-03-06T17:33:24.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:24.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:24.221774+0000) 2024-03-06T17:33:25.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:25.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:25.221997+0000) 2024-03-06T17:33:26.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:26.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:26.222268+0000) 2024-03-06T17:33:27.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:27.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:27.222472+0000) 2024-03-06T17:33:28.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:28.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:28.222636+0000) 2024-03-06T17:33:29.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:29.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:29.222805+0000) 2024-03-06T17:33:30.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:30.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:30.223065+0000) 2024-03-06T17:33:31.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:31.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:31.223317+0000) 2024-03-06T17:33:32.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:32.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:32.223541+0000) 2024-03-06T17:33:32.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:32.592 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:33:33.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:33.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:33.223750+0000) 2024-03-06T17:33:34.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:34.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:34.223985+0000) 2024-03-06T17:33:35.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:35.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:35.224251+0000) 2024-03-06T17:33:36.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:36.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:36.224482+0000) 2024-03-06T17:33:37.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:37.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:37.224716+0000) 2024-03-06T17:33:38.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:38.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:38.224951+0000) 2024-03-06T17:33:39.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:39.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:39.225179+0000) 2024-03-06T17:33:40.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:40.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:40.225409+0000) 2024-03-06T17:33:41.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:41.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:41.225634+0000) 2024-03-06T17:33:42.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:42.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:42.225864+0000) 2024-03-06T17:33:43.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:43.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:43.226137+0000) 2024-03-06T17:33:44.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:44.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:44.226267+0000) 2024-03-06T17:33:45.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:45.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:45.226550+0000) 2024-03-06T17:33:46.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:46.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:46.226776+0000) 2024-03-06T17:33:47.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:47.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:47.227045+0000) 2024-03-06T17:33:48.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:48.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:48.227253+0000) 2024-03-06T17:33:49.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:49.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:49.227459+0000) 2024-03-06T17:33:50.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:50.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:50.227682+0000) 2024-03-06T17:33:51.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:51.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:51.227947+0000) 2024-03-06T17:33:52.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:52.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:52.228172+0000) 2024-03-06T17:33:53.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:53.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:53.228415+0000) 2024-03-06T17:33:54.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:54.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:54.228622+0000) 2024-03-06T17:33:55.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:55.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:55.228856+0000) 2024-03-06T17:33:56.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:56.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:56.229136+0000) 2024-03-06T17:33:57.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:57.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:57.229366+0000) 2024-03-06T17:33:58.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:58.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:58.229616+0000) 2024-03-06T17:33:59.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:33:59.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:33:59.229836+0000) 2024-03-06T17:34:00.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:00.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:00.230081+0000) 2024-03-06T17:34:01.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:01.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:01.230268+0000) 2024-03-06T17:34:02.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:02.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:02.230501+0000) 2024-03-06T17:34:02.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:02.736 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:03.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:03.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:03.230742+0000) 2024-03-06T17:34:04.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:04.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:04.230970+0000) 2024-03-06T17:34:05.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:05.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:05.231194+0000) 2024-03-06T17:34:06.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:06.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:06.231388+0000) 2024-03-06T17:34:07.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:07.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:07.231611+0000) 2024-03-06T17:34:08.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:08.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:08.231848+0000) 2024-03-06T17:34:09.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:09.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:09.232106+0000) 2024-03-06T17:34:10.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:10.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:10.232317+0000) 2024-03-06T17:34:11.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:11.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:11.232531+0000) 2024-03-06T17:34:12.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:12.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:12.232794+0000) 2024-03-06T17:34:13.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:13.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:13.233108+0000) 2024-03-06T17:34:14.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:14.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:14.233313+0000) 2024-03-06T17:34:15.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:15.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:15.233503+0000) 2024-03-06T17:34:16.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:16.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:16.233733+0000) 2024-03-06T17:34:17.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:17.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:17.233965+0000) 2024-03-06T17:34:18.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:18.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:18.234155+0000) 2024-03-06T17:34:19.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:19.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:19.234370+0000) 2024-03-06T17:34:20.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:20.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:20.234615+0000) 2024-03-06T17:34:21.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:21.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:21.234796+0000) 2024-03-06T17:34:22.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:22.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:22.235066+0000) 2024-03-06T17:34:23.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:23.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:23.235264+0000) 2024-03-06T17:34:24.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:24.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:24.235490+0000) 2024-03-06T17:34:25.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:25.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:25.235704+0000) 2024-03-06T17:34:26.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:26.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:26.235961+0000) 2024-03-06T17:34:27.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:27.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:27.236144+0000) 2024-03-06T17:34:28.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:28.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:28.236385+0000) 2024-03-06T17:34:29.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:29.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:29.236626+0000) 2024-03-06T17:34:30.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:30.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:30.236878+0000) 2024-03-06T17:34:31.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:31.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:31.237132+0000) 2024-03-06T17:34:32.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:32.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:32.237358+0000) 2024-03-06T17:34:32.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:32.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:34:33.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:33.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:33.237569+0000) 2024-03-06T17:34:34.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:34.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:34.237769+0000) 2024-03-06T17:34:35.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:35.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:35.237990+0000) 2024-03-06T17:34:36.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:36.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:36.238212+0000) 2024-03-06T17:34:37.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:37.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:37.238421+0000) 2024-03-06T17:34:38.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:38.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:38.238627+0000) 2024-03-06T17:34:39.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:39.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:39.238858+0000) 2024-03-06T17:34:40.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:40.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:40.239084+0000) 2024-03-06T17:34:41.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:41.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:41.239282+0000) 2024-03-06T17:34:42.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:42.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:42.239505+0000) 2024-03-06T17:34:43.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:43.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:43.239733+0000) 2024-03-06T17:34:44.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:44.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:44.239905+0000) 2024-03-06T17:34:45.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:45.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:45.240120+0000) 2024-03-06T17:34:46.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:46.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:46.240290+0000) 2024-03-06T17:34:47.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:47.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:47.240446+0000) 2024-03-06T17:34:48.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:48.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:48.240658+0000) 2024-03-06T17:34:49.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:49.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:49.240882+0000) 2024-03-06T17:34:50.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:50.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:50.241129+0000) 2024-03-06T17:34:51.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:51.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:51.241310+0000) 2024-03-06T17:34:52.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:52.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:52.241495+0000) 2024-03-06T17:34:53.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:53.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:53.241726+0000) 2024-03-06T17:34:54.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:54.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:54.241934+0000) 2024-03-06T17:34:55.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:55.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:55.242140+0000) 2024-03-06T17:34:56.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:56.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:56.242370+0000) 2024-03-06T17:34:57.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:57.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:57.242601+0000) 2024-03-06T17:34:58.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:58.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:58.242801+0000) 2024-03-06T17:34:59.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:34:59.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:34:59.243061+0000) 2024-03-06T17:35:00.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:00.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:00.243257+0000) 2024-03-06T17:35:01.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:01.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:01.243460+0000) 2024-03-06T17:35:02.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:02.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:02.243664+0000) 2024-03-06T17:35:03.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:03.164 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:03.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:03.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:03.243848+0000) 2024-03-06T17:35:04.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:04.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:04.244104+0000) 2024-03-06T17:35:05.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:05.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:05.244309+0000) 2024-03-06T17:35:06.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:06.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:06.244526+0000) 2024-03-06T17:35:07.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:07.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:07.244783+0000) 2024-03-06T17:35:08.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:08.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:08.245032+0000) 2024-03-06T17:35:09.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:09.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:09.245211+0000) 2024-03-06T17:35:10.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:10.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:10.245404+0000) 2024-03-06T17:35:11.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:11.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:11.245609+0000) 2024-03-06T17:35:12.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:12.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:12.245785+0000) 2024-03-06T17:35:13.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:13.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:13.246031+0000) 2024-03-06T17:35:14.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:14.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:14.246274+0000) 2024-03-06T17:35:15.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:15.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:15.246494+0000) 2024-03-06T17:35:16.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:16.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:16.246724+0000) 2024-03-06T17:35:17.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:17.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:17.246928+0000) 2024-03-06T17:35:18.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:18.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:18.247171+0000) 2024-03-06T17:35:19.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:19.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:19.247375+0000) 2024-03-06T17:35:20.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:20.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:20.247597+0000) 2024-03-06T17:35:21.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:21.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:21.247835+0000) 2024-03-06T17:35:22.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:22.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:22.248106+0000) 2024-03-06T17:35:23.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:23.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:23.248311+0000) 2024-03-06T17:35:24.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:24.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:24.248536+0000) 2024-03-06T17:35:25.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:25.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:25.248739+0000) 2024-03-06T17:35:26.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:26.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:26.248962+0000) 2024-03-06T17:35:27.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:27.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:27.249182+0000) 2024-03-06T17:35:28.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:28.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:28.249419+0000) 2024-03-06T17:35:29.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:29.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:29.249686+0000) 2024-03-06T17:35:30.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:30.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:30.249917+0000) 2024-03-06T17:35:31.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:31.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:31.250140+0000) 2024-03-06T17:35:32.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:32.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:32.250390+0000) 2024-03-06T17:35:33.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:33.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:33.250594+0000) 2024-03-06T17:35:33.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:33.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:35:34.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:34.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:34.250814+0000) 2024-03-06T17:35:35.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:35.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:35.251085+0000) 2024-03-06T17:35:36.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:36.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:36.251269+0000) 2024-03-06T17:35:37.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:37.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:37.251506+0000) 2024-03-06T17:35:38.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:38.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:38.251732+0000) 2024-03-06T17:35:39.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:39.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:39.251954+0000) 2024-03-06T17:35:40.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:40.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:40.252187+0000) 2024-03-06T17:35:41.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:41.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:41.252374+0000) 2024-03-06T17:35:42.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:42.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:42.252580+0000) 2024-03-06T17:35:43.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:43.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:43.252806+0000) 2024-03-06T17:35:44.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:44.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:44.252998+0000) 2024-03-06T17:35:45.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:45.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:45.253211+0000) 2024-03-06T17:35:46.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:46.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:46.253397+0000) 2024-03-06T17:35:47.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:47.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:47.253604+0000) 2024-03-06T17:35:48.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:48.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:48.253955+0000) 2024-03-06T17:35:49.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:49.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:49.254181+0000) 2024-03-06T17:35:50.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:50.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:50.254372+0000) 2024-03-06T17:35:51.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:51.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:51.254598+0000) 2024-03-06T17:35:52.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:52.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:52.254823+0000) 2024-03-06T17:35:53.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:53.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:53.255092+0000) 2024-03-06T17:35:54.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:54.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:54.255328+0000) 2024-03-06T17:35:55.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:55.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:55.255578+0000) 2024-03-06T17:35:56.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:56.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:56.255800+0000) 2024-03-06T17:35:57.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:57.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:57.256079+0000) 2024-03-06T17:35:58.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:58.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:58.256305+0000) 2024-03-06T17:35:59.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:35:59.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:35:59.256551+0000) 2024-03-06T17:36:00.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:00.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:00.256759+0000) 2024-03-06T17:36:01.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:01.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:01.256986+0000) 2024-03-06T17:36:02.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:02.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:02.257186+0000) 2024-03-06T17:36:03.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:03.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:03.257372+0000) 2024-03-06T17:36:03.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:03.528 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:04.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:04.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:04.257586+0000) 2024-03-06T17:36:05.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:05.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:05.257813+0000) 2024-03-06T17:36:06.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:06.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:06.258027+0000) 2024-03-06T17:36:07.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:07.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:07.258262+0000) 2024-03-06T17:36:08.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:08.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:08.258464+0000) 2024-03-06T17:36:09.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:09.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:09.258669+0000) 2024-03-06T17:36:10.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:10.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:10.258896+0000) 2024-03-06T17:36:11.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:11.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:11.259105+0000) 2024-03-06T17:36:12.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:12.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:12.259302+0000) 2024-03-06T17:36:13.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:13.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:13.259509+0000) 2024-03-06T17:36:14.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:14.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:14.259721+0000) 2024-03-06T17:36:15.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:15.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:15.259936+0000) 2024-03-06T17:36:16.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:16.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:16.260159+0000) 2024-03-06T17:36:17.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:17.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:17.260359+0000) 2024-03-06T17:36:18.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:18.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:18.260563+0000) 2024-03-06T17:36:19.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:19.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:19.260758+0000) 2024-03-06T17:36:20.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:20.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:20.261115+0000) 2024-03-06T17:36:21.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:21.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:21.261355+0000) 2024-03-06T17:36:22.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:22.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:22.261611+0000) 2024-03-06T17:36:23.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:23.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:23.261833+0000) 2024-03-06T17:36:24.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:24.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:24.262107+0000) 2024-03-06T17:36:25.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:25.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:25.262385+0000) 2024-03-06T17:36:26.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:26.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:26.262609+0000) 2024-03-06T17:36:27.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:27.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:27.262832+0000) 2024-03-06T17:36:28.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:28.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:28.263093+0000) 2024-03-06T17:36:29.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:29.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:29.263302+0000) 2024-03-06T17:36:30.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:30.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:30.263515+0000) 2024-03-06T17:36:31.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:31.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:31.263731+0000) 2024-03-06T17:36:32.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:32.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:32.263959+0000) 2024-03-06T17:36:33.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:33.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:33.264151+0000) 2024-03-06T17:36:33.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:33.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:36:34.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:34.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:34.264351+0000) 2024-03-06T17:36:35.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:35.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:35.264549+0000) 2024-03-06T17:36:36.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:36.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:36.264803+0000) 2024-03-06T17:36:37.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:37.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:37.265095+0000) 2024-03-06T17:36:38.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:38.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:38.265300+0000) 2024-03-06T17:36:39.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:39.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:39.265486+0000) 2024-03-06T17:36:40.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:40.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:40.265709+0000) 2024-03-06T17:36:41.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:41.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:41.265898+0000) 2024-03-06T17:36:42.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:42.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:42.266125+0000) 2024-03-06T17:36:43.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:43.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:43.266322+0000) 2024-03-06T17:36:44.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:44.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:44.266547+0000) 2024-03-06T17:36:45.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:45.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:45.266749+0000) 2024-03-06T17:36:46.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:46.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:46.266963+0000) 2024-03-06T17:36:47.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:47.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:47.267230+0000) 2024-03-06T17:36:48.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:48.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:48.267431+0000) 2024-03-06T17:36:49.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:49.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:49.267616+0000) 2024-03-06T17:36:50.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:50.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:50.267847+0000) 2024-03-06T17:36:51.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:51.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:51.268111+0000) 2024-03-06T17:36:52.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:52.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:52.268315+0000) 2024-03-06T17:36:53.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:53.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:53.268532+0000) 2024-03-06T17:36:54.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:54.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:54.268786+0000) 2024-03-06T17:36:55.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:55.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:55.269050+0000) 2024-03-06T17:36:56.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:56.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:56.269258+0000) 2024-03-06T17:36:57.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:57.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:57.269471+0000) 2024-03-06T17:36:58.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:58.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:58.269689+0000) 2024-03-06T17:36:59.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:36:59.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:36:59.269876+0000) 2024-03-06T17:37:00.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:00.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:00.270150+0000) 2024-03-06T17:37:01.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:01.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:01.270361+0000) 2024-03-06T17:37:02.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:02.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:02.270584+0000) 2024-03-06T17:37:03.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:03.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:03.270855+0000) 2024-03-06T17:37:03.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:03.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:04.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:04.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:04.271125+0000) 2024-03-06T17:37:05.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:05.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:05.271315+0000) 2024-03-06T17:37:06.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:06.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:06.271535+0000) 2024-03-06T17:37:07.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:07.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:07.271784+0000) 2024-03-06T17:37:08.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:08.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:08.271957+0000) 2024-03-06T17:37:09.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:09.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:09.272174+0000) 2024-03-06T17:37:10.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:10.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:10.272380+0000) 2024-03-06T17:37:11.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:11.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:11.272598+0000) 2024-03-06T17:37:12.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:12.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:12.272819+0000) 2024-03-06T17:37:13.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:13.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:13.273084+0000) 2024-03-06T17:37:14.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:14.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:14.273284+0000) 2024-03-06T17:37:15.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:15.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:15.273510+0000) 2024-03-06T17:37:16.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:16.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:16.273725+0000) 2024-03-06T17:37:17.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:17.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:17.273958+0000) 2024-03-06T17:37:18.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:18.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:18.274189+0000) 2024-03-06T17:37:19.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:19.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:19.274405+0000) 2024-03-06T17:37:20.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:20.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:20.274629+0000) 2024-03-06T17:37:21.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:21.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:21.274845+0000) 2024-03-06T17:37:22.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:22.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:22.275078+0000) 2024-03-06T17:37:23.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:23.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:23.275291+0000) 2024-03-06T17:37:24.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:24.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:24.275529+0000) 2024-03-06T17:37:25.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:25.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:25.275752+0000) 2024-03-06T17:37:26.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:26.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:26.275973+0000) 2024-03-06T17:37:27.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:27.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:27.276206+0000) 2024-03-06T17:37:28.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:28.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:28.276392+0000) 2024-03-06T17:37:29.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:29.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:29.276605+0000) 2024-03-06T17:37:30.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:30.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:30.276828+0000) 2024-03-06T17:37:31.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:31.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:31.277071+0000) 2024-03-06T17:37:32.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:32.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:32.277291+0000) 2024-03-06T17:37:33.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:33.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:33.277531+0000) 2024-03-06T17:37:34.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:34.050 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:37:34.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:34.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:34.277755+0000) 2024-03-06T17:37:35.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:35.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:35.277927+0000) 2024-03-06T17:37:36.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:36.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:36.278157+0000) 2024-03-06T17:37:37.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:37.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:37.278363+0000) 2024-03-06T17:37:38.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:38.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:38.278575+0000) 2024-03-06T17:37:39.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:39.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:39.278807+0000) 2024-03-06T17:37:40.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:40.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:40.279039+0000) 2024-03-06T17:37:41.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:41.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:41.279272+0000) 2024-03-06T17:37:42.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:42.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:42.279498+0000) 2024-03-06T17:37:43.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:43.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:43.279720+0000) 2024-03-06T17:37:44.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:44.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:44.279948+0000) 2024-03-06T17:37:45.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:45.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:45.280202+0000) 2024-03-06T17:37:46.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:46.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:46.280482+0000) 2024-03-06T17:37:47.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:47.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:47.280772+0000) 2024-03-06T17:37:48.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:48.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:48.281092+0000) 2024-03-06T17:37:49.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:49.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:49.281340+0000) 2024-03-06T17:37:50.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:50.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:50.281561+0000) 2024-03-06T17:37:51.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:51.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:51.281873+0000) 2024-03-06T17:37:52.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:52.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:52.282156+0000) 2024-03-06T17:37:53.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:53.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:53.282468+0000) 2024-03-06T17:37:54.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:54.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:54.282720+0000) 2024-03-06T17:37:55.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:55.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:55.282948+0000) 2024-03-06T17:37:56.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:56.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:56.283290+0000) 2024-03-06T17:37:57.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:57.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:57.283573+0000) 2024-03-06T17:37:58.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:58.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:58.283857+0000) 2024-03-06T17:37:59.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:37:59.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:37:59.284096+0000) 2024-03-06T17:38:00.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:00.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:00.284322+0000) 2024-03-06T17:38:01.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:01.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:01.284544+0000) 2024-03-06T17:38:02.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:02.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:02.284729+0000) 2024-03-06T17:38:03.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:03.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:03.284917+0000) 2024-03-06T17:38:04.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:04.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:04.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:04.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:04.285155+0000) 2024-03-06T17:38:05.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:05.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:05.285371+0000) 2024-03-06T17:38:06.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:06.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:06.285618+0000) 2024-03-06T17:38:07.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:07.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:07.285864+0000) 2024-03-06T17:38:08.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:08.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:08.286110+0000) 2024-03-06T17:38:09.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:09.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:09.286310+0000) 2024-03-06T17:38:10.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:10.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:10.286553+0000) 2024-03-06T17:38:11.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:11.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:11.286787+0000) 2024-03-06T17:38:12.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:12.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:12.287061+0000) 2024-03-06T17:38:13.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:13.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:13.287251+0000) 2024-03-06T17:38:14.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:14.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:14.287469+0000) 2024-03-06T17:38:15.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:15.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:15.287681+0000) 2024-03-06T17:38:16.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:16.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:16.287906+0000) 2024-03-06T17:38:17.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:17.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:17.288107+0000) 2024-03-06T17:38:18.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:18.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:18.288307+0000) 2024-03-06T17:38:19.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:19.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:19.288528+0000) 2024-03-06T17:38:20.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:20.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:20.288728+0000) 2024-03-06T17:38:21.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:21.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:21.288963+0000) 2024-03-06T17:38:22.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:22.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:22.289194+0000) 2024-03-06T17:38:23.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:23.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:23.289418+0000) 2024-03-06T17:38:24.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:24.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:24.289607+0000) 2024-03-06T17:38:25.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:25.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:25.289823+0000) 2024-03-06T17:38:26.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:26.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:26.290079+0000) 2024-03-06T17:38:27.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:27.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:27.290308+0000) 2024-03-06T17:38:28.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:28.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:28.290498+0000) 2024-03-06T17:38:29.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:29.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:29.290732+0000) 2024-03-06T17:38:30.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:30.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:30.290977+0000) 2024-03-06T17:38:31.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:31.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:31.291256+0000) 2024-03-06T17:38:32.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:32.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:32.291469+0000) 2024-03-06T17:38:33.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:33.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:33.291656+0000) 2024-03-06T17:38:34.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:34.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:34.291895+0000) 2024-03-06T17:38:34.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:34.351 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:38:35.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:35.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:35.292159+0000) 2024-03-06T17:38:36.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:36.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:36.292406+0000) 2024-03-06T17:38:37.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:37.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:37.292640+0000) 2024-03-06T17:38:38.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:38.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:38.292860+0000) 2024-03-06T17:38:39.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:39.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:39.293128+0000) 2024-03-06T17:38:40.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:40.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:40.293357+0000) 2024-03-06T17:38:41.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:41.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:41.293571+0000) 2024-03-06T17:38:42.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:42.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:42.293817+0000) 2024-03-06T17:38:43.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:43.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:43.294037+0000) 2024-03-06T17:38:44.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:44.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:44.294282+0000) 2024-03-06T17:38:45.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:45.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:45.294511+0000) 2024-03-06T17:38:46.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:46.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:46.294743+0000) 2024-03-06T17:38:47.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:47.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:47.294971+0000) 2024-03-06T17:38:48.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:48.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:48.295164+0000) 2024-03-06T17:38:49.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:49.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:49.295385+0000) 2024-03-06T17:38:50.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:50.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:50.295590+0000) 2024-03-06T17:38:51.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:51.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:51.295774+0000) 2024-03-06T17:38:52.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:52.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:52.295993+0000) 2024-03-06T17:38:53.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:53.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:53.296231+0000) 2024-03-06T17:38:54.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:54.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:54.296423+0000) 2024-03-06T17:38:55.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:55.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:55.296647+0000) 2024-03-06T17:38:56.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:56.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:56.296867+0000) 2024-03-06T17:38:57.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:57.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:57.297120+0000) 2024-03-06T17:38:58.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:58.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:58.297330+0000) 2024-03-06T17:38:59.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:38:59.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:38:59.297562+0000) 2024-03-06T17:39:00.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:00.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:00.297725+0000) 2024-03-06T17:39:01.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:01.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:01.297937+0000) 2024-03-06T17:39:02.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:02.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:02.298178+0000) 2024-03-06T17:39:03.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:03.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:03.298407+0000) 2024-03-06T17:39:04.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:04.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:04.298589+0000) 2024-03-06T17:39:04.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:04.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:05.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:05.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:05.298809+0000) 2024-03-06T17:39:06.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:06.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:06.298996+0000) 2024-03-06T17:39:07.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:07.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:07.299235+0000) 2024-03-06T17:39:08.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:08.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:08.299449+0000) 2024-03-06T17:39:09.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:09.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:09.299662+0000) 2024-03-06T17:39:10.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:10.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:10.299875+0000) 2024-03-06T17:39:11.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:11.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:11.300120+0000) 2024-03-06T17:39:12.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:12.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:12.300309+0000) 2024-03-06T17:39:13.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:13.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:13.300515+0000) 2024-03-06T17:39:14.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:14.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:14.300741+0000) 2024-03-06T17:39:15.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:15.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:15.300960+0000) 2024-03-06T17:39:16.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:16.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:16.301174+0000) 2024-03-06T17:39:17.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:17.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:17.301415+0000) 2024-03-06T17:39:18.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:18.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:18.301638+0000) 2024-03-06T17:39:19.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:19.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:19.301866+0000) 2024-03-06T17:39:20.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:20.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:20.302120+0000) 2024-03-06T17:39:21.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:21.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:21.302352+0000) 2024-03-06T17:39:22.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:22.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:22.302608+0000) 2024-03-06T17:39:23.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:23.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:23.302844+0000) 2024-03-06T17:39:24.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:24.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:24.303082+0000) 2024-03-06T17:39:25.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:25.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:25.303281+0000) 2024-03-06T17:39:26.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:26.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:26.303493+0000) 2024-03-06T17:39:27.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:27.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:27.303724+0000) 2024-03-06T17:39:28.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:28.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:28.303934+0000) 2024-03-06T17:39:29.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:29.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:29.304167+0000) 2024-03-06T17:39:30.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:30.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:30.304414+0000) 2024-03-06T17:39:31.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:31.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:31.304609+0000) 2024-03-06T17:39:32.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:32.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:32.304841+0000) 2024-03-06T17:39:33.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:33.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:33.305093+0000) 2024-03-06T17:39:34.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:34.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:34.305266+0000) 2024-03-06T17:39:34.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:34.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:39:35.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:35.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:35.305448+0000) 2024-03-06T17:39:36.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:36.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:36.305651+0000) 2024-03-06T17:39:37.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:37.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:37.305966+0000) 2024-03-06T17:39:38.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:38.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:38.306218+0000) 2024-03-06T17:39:39.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:39.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:39.306429+0000) 2024-03-06T17:39:40.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:40.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:40.306639+0000) 2024-03-06T17:39:41.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:41.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:41.306873+0000) 2024-03-06T17:39:42.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:42.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:42.307118+0000) 2024-03-06T17:39:43.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:43.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:43.307326+0000) 2024-03-06T17:39:44.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:44.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:44.307562+0000) 2024-03-06T17:39:45.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:45.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:45.307788+0000) 2024-03-06T17:39:46.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:46.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:46.308038+0000) 2024-03-06T17:39:47.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:47.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:47.308223+0000) 2024-03-06T17:39:48.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:48.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:48.308410+0000) 2024-03-06T17:39:49.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:49.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:49.308631+0000) 2024-03-06T17:39:50.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:50.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:50.308855+0000) 2024-03-06T17:39:51.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:51.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:51.309099+0000) 2024-03-06T17:39:52.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:52.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:52.309264+0000) 2024-03-06T17:39:53.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:53.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:53.309507+0000) 2024-03-06T17:39:54.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:54.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:54.309730+0000) 2024-03-06T17:39:55.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:55.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:55.309967+0000) 2024-03-06T17:39:56.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:56.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:56.310174+0000) 2024-03-06T17:39:57.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:57.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:57.310393+0000) 2024-03-06T17:39:58.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:58.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:58.310609+0000) 2024-03-06T17:39:59.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:39:59.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:39:59.310828+0000) 2024-03-06T17:40:00.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:00.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:00.311096+0000) 2024-03-06T17:40:01.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:01.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:01.311317+0000) 2024-03-06T17:40:02.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:02.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:02.311554+0000) 2024-03-06T17:40:03.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:03.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:03.311782+0000) 2024-03-06T17:40:04.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:04.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:04.312057+0000) 2024-03-06T17:40:04.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:04.904 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:05.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:05.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:05.312308+0000) 2024-03-06T17:40:06.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:06.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:06.312555+0000) 2024-03-06T17:40:07.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:07.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:07.312802+0000) 2024-03-06T17:40:08.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:08.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:08.313049+0000) 2024-03-06T17:40:09.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:09.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:09.313305+0000) 2024-03-06T17:40:10.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:10.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:10.313547+0000) 2024-03-06T17:40:11.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:11.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:11.313696+0000) 2024-03-06T17:40:12.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:12.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:12.313880+0000) 2024-03-06T17:40:13.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:13.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:13.314147+0000) 2024-03-06T17:40:14.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:14.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:14.314396+0000) 2024-03-06T17:40:15.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:15.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:15.314636+0000) 2024-03-06T17:40:16.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:16.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:16.314875+0000) 2024-03-06T17:40:17.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:17.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:17.315126+0000) 2024-03-06T17:40:18.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:18.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:18.315361+0000) 2024-03-06T17:40:19.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:19.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:19.315611+0000) 2024-03-06T17:40:20.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:20.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:20.315841+0000) 2024-03-06T17:40:21.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:21.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:21.316104+0000) 2024-03-06T17:40:22.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:22.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:22.316330+0000) 2024-03-06T17:40:23.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:23.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:23.316546+0000) 2024-03-06T17:40:24.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:24.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:24.316775+0000) 2024-03-06T17:40:25.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:25.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:25.317048+0000) 2024-03-06T17:40:26.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:26.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:26.317297+0000) 2024-03-06T17:40:27.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:27.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:27.317543+0000) 2024-03-06T17:40:28.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:28.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:28.317723+0000) 2024-03-06T17:40:29.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:29.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:29.317958+0000) 2024-03-06T17:40:30.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:30.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:30.318176+0000) 2024-03-06T17:40:31.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:31.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:31.318416+0000) 2024-03-06T17:40:32.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:32.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:32.318619+0000) 2024-03-06T17:40:33.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:33.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:33.318844+0000) 2024-03-06T17:40:34.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:34.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:34.319129+0000) 2024-03-06T17:40:35.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:35.124 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:40:35.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:35.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:35.319339+0000) 2024-03-06T17:40:36.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:36.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:36.319552+0000) 2024-03-06T17:40:37.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:37.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:37.319791+0000) 2024-03-06T17:40:38.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:38.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:38.319987+0000) 2024-03-06T17:40:39.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:39.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:39.320208+0000) 2024-03-06T17:40:40.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:40.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:40.320402+0000) 2024-03-06T17:40:41.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:41.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:41.320599+0000) 2024-03-06T17:40:42.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:42.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:42.320821+0000) 2024-03-06T17:40:43.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:43.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:43.321054+0000) 2024-03-06T17:40:44.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:44.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:44.321246+0000) 2024-03-06T17:40:45.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:45.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:45.321392+0000) 2024-03-06T17:40:46.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:46.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:46.321616+0000) 2024-03-06T17:40:47.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:47.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:47.321861+0000) 2024-03-06T17:40:48.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:48.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:48.322094+0000) 2024-03-06T17:40:49.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:49.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:49.322292+0000) 2024-03-06T17:40:50.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:50.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:50.322497+0000) 2024-03-06T17:40:51.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:51.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:51.322723+0000) 2024-03-06T17:40:52.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:52.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:52.322956+0000) 2024-03-06T17:40:53.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:53.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:53.323184+0000) 2024-03-06T17:40:54.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:54.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:54.323399+0000) 2024-03-06T17:40:55.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:55.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:55.323621+0000) 2024-03-06T17:40:56.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:56.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:56.323853+0000) 2024-03-06T17:40:57.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:57.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:57.324137+0000) 2024-03-06T17:40:58.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:58.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:58.324345+0000) 2024-03-06T17:40:59.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:40:59.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:40:59.324567+0000) 2024-03-06T17:41:00.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:00.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:00.324825+0000) 2024-03-06T17:41:01.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:01.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:01.325092+0000) 2024-03-06T17:41:02.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:02.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:02.325315+0000) 2024-03-06T17:41:03.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:03.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:03.325533+0000) 2024-03-06T17:41:04.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:04.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:04.325781+0000) 2024-03-06T17:41:05.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:05.271 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:05.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:05.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:05.325970+0000) 2024-03-06T17:41:06.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:06.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:06.326155+0000) 2024-03-06T17:41:07.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:07.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:07.326354+0000) 2024-03-06T17:41:08.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:08.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:08.326524+0000) 2024-03-06T17:41:09.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:09.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:09.326772+0000) 2024-03-06T17:41:10.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:10.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:10.327046+0000) 2024-03-06T17:41:11.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:11.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:11.327265+0000) 2024-03-06T17:41:12.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:12.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:12.327500+0000) 2024-03-06T17:41:13.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:13.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:13.327746+0000) 2024-03-06T17:41:14.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:14.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:14.327977+0000) 2024-03-06T17:41:15.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:15.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:15.328295+0000) 2024-03-06T17:41:16.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:16.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:16.328559+0000) 2024-03-06T17:41:17.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:17.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:17.328832+0000) 2024-03-06T17:41:18.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:18.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:18.329138+0000) 2024-03-06T17:41:19.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:19.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:19.329367+0000) 2024-03-06T17:41:20.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:20.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:20.329638+0000) 2024-03-06T17:41:21.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:21.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:21.329909+0000) 2024-03-06T17:41:22.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:22.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:22.330180+0000) 2024-03-06T17:41:23.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:23.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:23.330461+0000) 2024-03-06T17:41:24.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:24.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:24.330669+0000) 2024-03-06T17:41:25.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:25.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:25.330910+0000) 2024-03-06T17:41:26.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:26.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:26.331158+0000) 2024-03-06T17:41:27.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:27.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:27.331381+0000) 2024-03-06T17:41:28.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:28.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:28.331617+0000) 2024-03-06T17:41:29.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:29.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:29.331822+0000) 2024-03-06T17:41:30.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:30.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:30.332102+0000) 2024-03-06T17:41:31.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:31.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:31.332339+0000) 2024-03-06T17:41:32.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:32.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:32.332588+0000) 2024-03-06T17:41:33.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:33.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:33.332822+0000) 2024-03-06T17:41:34.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:34.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:34.333096+0000) 2024-03-06T17:41:35.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:35.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:35.333349+0000) 2024-03-06T17:41:35.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:35.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:41:36.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:36.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:36.333574+0000) 2024-03-06T17:41:37.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:37.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:37.333785+0000) 2024-03-06T17:41:38.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:38.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:38.334037+0000) 2024-03-06T17:41:39.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:39.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:39.334257+0000) 2024-03-06T17:41:40.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:40.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:40.334485+0000) 2024-03-06T17:41:41.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:41.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:41.334707+0000) 2024-03-06T17:41:42.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:42.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:42.334928+0000) 2024-03-06T17:41:43.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:43.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:43.335148+0000) 2024-03-06T17:41:44.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:44.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:44.335416+0000) 2024-03-06T17:41:45.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:45.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:45.335689+0000) 2024-03-06T17:41:46.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:46.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:46.335967+0000) 2024-03-06T17:41:47.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:47.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:47.336260+0000) 2024-03-06T17:41:48.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:48.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:48.336540+0000) 2024-03-06T17:41:49.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:49.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:49.336808+0000) 2024-03-06T17:41:50.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:50.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:50.337110+0000) 2024-03-06T17:41:51.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:51.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:51.337375+0000) 2024-03-06T17:41:52.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:52.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:52.337651+0000) 2024-03-06T17:41:53.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:53.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:53.337850+0000) 2024-03-06T17:41:54.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:54.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:54.338109+0000) 2024-03-06T17:41:55.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:55.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:55.338328+0000) 2024-03-06T17:41:56.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:56.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:56.338556+0000) 2024-03-06T17:41:57.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:57.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:57.338802+0000) 2024-03-06T17:41:58.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:58.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:58.339061+0000) 2024-03-06T17:41:59.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:41:59.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:41:59.339287+0000) 2024-03-06T17:42:00.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:00.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:00.339514+0000) 2024-03-06T17:42:01.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:01.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:01.339740+0000) 2024-03-06T17:42:02.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:02.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:02.339974+0000) 2024-03-06T17:42:03.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:03.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:03.340178+0000) 2024-03-06T17:42:04.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:04.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:04.340401+0000) 2024-03-06T17:42:05.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:05.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:05.340627+0000) 2024-03-06T17:42:05.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:05.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:06.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:06.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:06.340786+0000) 2024-03-06T17:42:07.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:07.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:07.341038+0000) 2024-03-06T17:42:08.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:08.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:08.341266+0000) 2024-03-06T17:42:09.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:09.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:09.341484+0000) 2024-03-06T17:42:10.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:10.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:10.341714+0000) 2024-03-06T17:42:11.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:11.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:11.341933+0000) 2024-03-06T17:42:12.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:12.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:12.342204+0000) 2024-03-06T17:42:13.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:13.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:13.342442+0000) 2024-03-06T17:42:14.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:14.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:14.342620+0000) 2024-03-06T17:42:15.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:15.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:15.342857+0000) 2024-03-06T17:42:16.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:16.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:16.343117+0000) 2024-03-06T17:42:17.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:17.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:17.343350+0000) 2024-03-06T17:42:18.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:18.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:18.343564+0000) 2024-03-06T17:42:19.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:19.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:19.343775+0000) 2024-03-06T17:42:20.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:20.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:20.344100+0000) 2024-03-06T17:42:21.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:21.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:21.344295+0000) 2024-03-06T17:42:22.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:22.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:22.344534+0000) 2024-03-06T17:42:23.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:23.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:23.344771+0000) 2024-03-06T17:42:24.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:24.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:24.345040+0000) 2024-03-06T17:42:25.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:25.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:25.345295+0000) 2024-03-06T17:42:26.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:26.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:26.345476+0000) 2024-03-06T17:42:27.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:27.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:27.345705+0000) 2024-03-06T17:42:28.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:28.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:28.345935+0000) 2024-03-06T17:42:29.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:29.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:29.346188+0000) 2024-03-06T17:42:30.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:30.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:30.346458+0000) 2024-03-06T17:42:31.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:31.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:31.346669+0000) 2024-03-06T17:42:32.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:32.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:32.346904+0000) 2024-03-06T17:42:33.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:33.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:33.347134+0000) 2024-03-06T17:42:34.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:34.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:34.347344+0000) 2024-03-06T17:42:35.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:35.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:35.347584+0000) 2024-03-06T17:42:35.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:35.711 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:42:36.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:36.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:36.347827+0000) 2024-03-06T17:42:37.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:37.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:37.348106+0000) 2024-03-06T17:42:38.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:38.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:38.348343+0000) 2024-03-06T17:42:39.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:39.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:39.348587+0000) 2024-03-06T17:42:40.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:40.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:40.348846+0000) 2024-03-06T17:42:41.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:41.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:41.349115+0000) 2024-03-06T17:42:42.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:42.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:42.349376+0000) 2024-03-06T17:42:43.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:43.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:43.349656+0000) 2024-03-06T17:42:44.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:44.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:44.349934+0000) 2024-03-06T17:42:45.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:45.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:45.350183+0000) 2024-03-06T17:42:46.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:46.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:46.350431+0000) 2024-03-06T17:42:47.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:47.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:47.350620+0000) 2024-03-06T17:42:48.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:48.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:48.350850+0000) 2024-03-06T17:42:49.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:49.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:49.351123+0000) 2024-03-06T17:42:50.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:50.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:50.351358+0000) 2024-03-06T17:42:51.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:51.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:51.351598+0000) 2024-03-06T17:42:52.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:52.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:52.351832+0000) 2024-03-06T17:42:53.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:53.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:53.352139+0000) 2024-03-06T17:42:54.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:54.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:54.352408+0000) 2024-03-06T17:42:55.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:55.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:55.352656+0000) 2024-03-06T17:42:56.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:56.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:56.352926+0000) 2024-03-06T17:42:57.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:57.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:57.353219+0000) 2024-03-06T17:42:58.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:58.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:58.353478+0000) 2024-03-06T17:42:59.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:42:59.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:42:59.353752+0000) 2024-03-06T17:43:00.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:00.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:00.354086+0000) 2024-03-06T17:43:01.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:01.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:01.354288+0000) 2024-03-06T17:43:02.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:02.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:02.354516+0000) 2024-03-06T17:43:03.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:03.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:03.354693+0000) 2024-03-06T17:43:04.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:04.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:04.354955+0000) 2024-03-06T17:43:05.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:05.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:05.355282+0000) 2024-03-06T17:43:05.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:05.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:06.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:06.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:06.355550+0000) 2024-03-06T17:43:07.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:07.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:07.355833+0000) 2024-03-06T17:43:08.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:08.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:08.356098+0000) 2024-03-06T17:43:09.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:09.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:09.356353+0000) 2024-03-06T17:43:10.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:10.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:10.356611+0000) 2024-03-06T17:43:11.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:11.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:11.356878+0000) 2024-03-06T17:43:12.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:12.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:12.357147+0000) 2024-03-06T17:43:13.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:13.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:13.357386+0000) 2024-03-06T17:43:14.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:14.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:14.357623+0000) 2024-03-06T17:43:15.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:15.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:15.357870+0000) 2024-03-06T17:43:16.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:16.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:16.358051+0000) 2024-03-06T17:43:17.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:17.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:17.358310+0000) 2024-03-06T17:43:18.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:18.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:18.358529+0000) 2024-03-06T17:43:19.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:19.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:19.358786+0000) 2024-03-06T17:43:20.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:20.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:20.359088+0000) 2024-03-06T17:43:21.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:21.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:21.359328+0000) 2024-03-06T17:43:22.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:22.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:22.359584+0000) 2024-03-06T17:43:23.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:23.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:23.359823+0000) 2024-03-06T17:43:24.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:24.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:24.360095+0000) 2024-03-06T17:43:25.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:25.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:25.360339+0000) 2024-03-06T17:43:26.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:26.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:26.360581+0000) 2024-03-06T17:43:27.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:27.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:27.360837+0000) 2024-03-06T17:43:28.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:28.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:28.361100+0000) 2024-03-06T17:43:29.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:29.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:29.361340+0000) 2024-03-06T17:43:30.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:30.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:30.361588+0000) 2024-03-06T17:43:31.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:31.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:31.361822+0000) 2024-03-06T17:43:32.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:32.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:32.362124+0000) 2024-03-06T17:43:33.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:33.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:33.362368+0000) 2024-03-06T17:43:34.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:34.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:34.362645+0000) 2024-03-06T17:43:35.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:35.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:35.362918+0000) 2024-03-06T17:43:35.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:35.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:43:36.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:36.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:36.363205+0000) 2024-03-06T17:43:37.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:37.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:37.363507+0000) 2024-03-06T17:43:38.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:38.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:38.363797+0000) 2024-03-06T17:43:39.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:39.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:39.364034+0000) 2024-03-06T17:43:40.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:40.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:40.364315+0000) 2024-03-06T17:43:41.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:41.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:41.364596+0000) 2024-03-06T17:43:42.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:42.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:42.364848+0000) 2024-03-06T17:43:43.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:43.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:43.365114+0000) 2024-03-06T17:43:44.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:44.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:44.365299+0000) 2024-03-06T17:43:45.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:45.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:45.365531+0000) 2024-03-06T17:43:46.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:46.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:46.365757+0000) 2024-03-06T17:43:47.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:47.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:47.365972+0000) 2024-03-06T17:43:48.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:48.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:48.366357+0000) 2024-03-06T17:43:49.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:49.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:49.366610+0000) 2024-03-06T17:43:50.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:50.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:50.366860+0000) 2024-03-06T17:43:51.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:51.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:51.367100+0000) 2024-03-06T17:43:52.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:52.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:52.367335+0000) 2024-03-06T17:43:53.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:53.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:53.367559+0000) 2024-03-06T17:43:54.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:54.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:54.367759+0000) 2024-03-06T17:43:55.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:55.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:55.367991+0000) 2024-03-06T17:43:56.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:56.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:56.368308+0000) 2024-03-06T17:43:57.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:57.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:57.368600+0000) 2024-03-06T17:43:58.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:58.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:58.368887+0000) 2024-03-06T17:43:59.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:43:59.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:43:59.369160+0000) 2024-03-06T17:44:00.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:00.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:00.369399+0000) 2024-03-06T17:44:01.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:01.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:01.369620+0000) 2024-03-06T17:44:02.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:02.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:02.369843+0000) 2024-03-06T17:44:03.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:03.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:03.370111+0000) 2024-03-06T17:44:04.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:04.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:04.370298+0000) 2024-03-06T17:44:05.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:05.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:05.370533+0000) 2024-03-06T17:44:06.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:06.141 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:06.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:06.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:06.370741+0000) 2024-03-06T17:44:07.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:07.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:07.370979+0000) 2024-03-06T17:44:08.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:08.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:08.371225+0000) 2024-03-06T17:44:09.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:09.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:09.371415+0000) 2024-03-06T17:44:10.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:10.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:10.371646+0000) 2024-03-06T17:44:11.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:11.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:11.371870+0000) 2024-03-06T17:44:12.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:12.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:12.372138+0000) 2024-03-06T17:44:13.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:13.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:13.372361+0000) 2024-03-06T17:44:14.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:14.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:14.372584+0000) 2024-03-06T17:44:15.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:15.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:15.372820+0000) 2024-03-06T17:44:16.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:16.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:16.373063+0000) 2024-03-06T17:44:17.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:17.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:17.373322+0000) 2024-03-06T17:44:18.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:18.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:18.373540+0000) 2024-03-06T17:44:19.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:19.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:19.373708+0000) 2024-03-06T17:44:20.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:20.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:20.373894+0000) 2024-03-06T17:44:21.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:21.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:21.374133+0000) 2024-03-06T17:44:22.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:22.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:22.374351+0000) 2024-03-06T17:44:23.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:23.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:23.374581+0000) 2024-03-06T17:44:24.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:24.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:24.374804+0000) 2024-03-06T17:44:25.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:25.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:25.375086+0000) 2024-03-06T17:44:26.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:26.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:26.375288+0000) 2024-03-06T17:44:27.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:27.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:27.375559+0000) 2024-03-06T17:44:28.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:28.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:28.375805+0000) 2024-03-06T17:44:29.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:29.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:29.376072+0000) 2024-03-06T17:44:30.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:30.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:30.376303+0000) 2024-03-06T17:44:31.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:31.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:31.376526+0000) 2024-03-06T17:44:32.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:32.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:32.376764+0000) 2024-03-06T17:44:33.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:33.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:33.377000+0000) 2024-03-06T17:44:34.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:34.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:34.377271+0000) 2024-03-06T17:44:35.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:35.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:35.377523+0000) 2024-03-06T17:44:36.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:36.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:44:36.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:36.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:36.377861+0000) 2024-03-06T17:44:37.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:37.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:37.378153+0000) 2024-03-06T17:44:38.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:38.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:38.378436+0000) 2024-03-06T17:44:39.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:39.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:39.378711+0000) 2024-03-06T17:44:40.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:40.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:40.378991+0000) 2024-03-06T17:44:41.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:41.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:41.379298+0000) 2024-03-06T17:44:42.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:42.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:42.379533+0000) 2024-03-06T17:44:43.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:43.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:43.379793+0000) 2024-03-06T17:44:44.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:44.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:44.380082+0000) 2024-03-06T17:44:45.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:45.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:45.380336+0000) 2024-03-06T17:44:46.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:46.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:46.380560+0000) 2024-03-06T17:44:47.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:47.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:47.380815+0000) 2024-03-06T17:44:48.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:48.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:48.381094+0000) 2024-03-06T17:44:49.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:49.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:49.381329+0000) 2024-03-06T17:44:50.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:50.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:50.381559+0000) 2024-03-06T17:44:51.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:51.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:51.381744+0000) 2024-03-06T17:44:52.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:52.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:52.381985+0000) 2024-03-06T17:44:53.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:53.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:53.382227+0000) 2024-03-06T17:44:54.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:54.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:54.382422+0000) 2024-03-06T17:44:55.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:55.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:55.382648+0000) 2024-03-06T17:44:56.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:56.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:56.382873+0000) 2024-03-06T17:44:57.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:57.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:57.383140+0000) 2024-03-06T17:44:58.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:58.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:58.383322+0000) 2024-03-06T17:44:59.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:44:59.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:44:59.383548+0000) 2024-03-06T17:45:00.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:00.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:00.383785+0000) 2024-03-06T17:45:01.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:01.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:01.383965+0000) 2024-03-06T17:45:02.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:02.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:02.384241+0000) 2024-03-06T17:45:03.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:03.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:03.384474+0000) 2024-03-06T17:45:04.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:04.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:04.384698+0000) 2024-03-06T17:45:05.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:05.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:05.384949+0000) 2024-03-06T17:45:06.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:06.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:06.385202+0000) 2024-03-06T17:45:06.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:06.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:07.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:07.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:07.385401+0000) 2024-03-06T17:45:08.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:08.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:08.385591+0000) 2024-03-06T17:45:09.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:09.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:09.385820+0000) 2024-03-06T17:45:10.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:10.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:10.386105+0000) 2024-03-06T17:45:11.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:11.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:11.386346+0000) 2024-03-06T17:45:12.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:12.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:12.386588+0000) 2024-03-06T17:45:13.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:13.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:13.386860+0000) 2024-03-06T17:45:14.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:14.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:14.387162+0000) 2024-03-06T17:45:15.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:15.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:15.387447+0000) 2024-03-06T17:45:16.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:16.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:16.387735+0000) 2024-03-06T17:45:17.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:17.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:17.388038+0000) 2024-03-06T17:45:18.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:18.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:18.388286+0000) 2024-03-06T17:45:19.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:19.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:19.388458+0000) 2024-03-06T17:45:20.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:20.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:20.388686+0000) 2024-03-06T17:45:21.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:21.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:21.388909+0000) 2024-03-06T17:45:22.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:22.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:22.389148+0000) 2024-03-06T17:45:23.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:23.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:23.389360+0000) 2024-03-06T17:45:24.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:24.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:24.389599+0000) 2024-03-06T17:45:25.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:25.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:25.389839+0000) 2024-03-06T17:45:26.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:26.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:26.390115+0000) 2024-03-06T17:45:27.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:27.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:27.390331+0000) 2024-03-06T17:45:28.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:28.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:28.390611+0000) 2024-03-06T17:45:29.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:29.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:29.390917+0000) 2024-03-06T17:45:30.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:30.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:30.391176+0000) 2024-03-06T17:45:31.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:31.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:31.391435+0000) 2024-03-06T17:45:32.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:32.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:32.391679+0000) 2024-03-06T17:45:33.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:33.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:33.391919+0000) 2024-03-06T17:45:34.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:34.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:34.392169+0000) 2024-03-06T17:45:35.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:35.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:35.392318+0000) 2024-03-06T17:45:36.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:36.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:36.392513+0000) 2024-03-06T17:45:36.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:36.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:45:37.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:37.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:37.392709+0000) 2024-03-06T17:45:38.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:38.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:38.392888+0000) 2024-03-06T17:45:39.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:39.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:39.393142+0000) 2024-03-06T17:45:40.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:40.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:40.393347+0000) 2024-03-06T17:45:41.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:41.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:41.393584+0000) 2024-03-06T17:45:42.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:42.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:42.393838+0000) 2024-03-06T17:45:43.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:43.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:43.394132+0000) 2024-03-06T17:45:44.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:44.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:44.394360+0000) 2024-03-06T17:45:45.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:45.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:45.394595+0000) 2024-03-06T17:45:46.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:46.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:46.394769+0000) 2024-03-06T17:45:47.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:47.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:47.394959+0000) 2024-03-06T17:45:48.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:48.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:48.395188+0000) 2024-03-06T17:45:49.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:49.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:49.395448+0000) 2024-03-06T17:45:50.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:50.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:50.395737+0000) 2024-03-06T17:45:51.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:51.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:51.396056+0000) 2024-03-06T17:45:52.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:52.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:52.396287+0000) 2024-03-06T17:45:53.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:53.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:53.396524+0000) 2024-03-06T17:45:54.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:54.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:54.396756+0000) 2024-03-06T17:45:55.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:55.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:55.396990+0000) 2024-03-06T17:45:56.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:56.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:56.397256+0000) 2024-03-06T17:45:57.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:57.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:57.397489+0000) 2024-03-06T17:45:58.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:58.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:58.397722+0000) 2024-03-06T17:45:59.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:45:59.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:45:59.397954+0000) 2024-03-06T17:46:00.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:00.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:00.398245+0000) 2024-03-06T17:46:01.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:01.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:01.398489+0000) 2024-03-06T17:46:02.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:02.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:02.398736+0000) 2024-03-06T17:46:03.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:03.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:03.398908+0000) 2024-03-06T17:46:04.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:04.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:04.399130+0000) 2024-03-06T17:46:05.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:05.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:05.399293+0000) 2024-03-06T17:46:06.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:06.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:06.399419+0000) 2024-03-06T17:46:06.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:06.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:07.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:07.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:07.399649+0000) 2024-03-06T17:46:08.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:08.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:08.399871+0000) 2024-03-06T17:46:09.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:09.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:09.400143+0000) 2024-03-06T17:46:10.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:10.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:10.400346+0000) 2024-03-06T17:46:11.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:11.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:11.400568+0000) 2024-03-06T17:46:12.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:12.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:12.400807+0000) 2024-03-06T17:46:13.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:13.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:13.401057+0000) 2024-03-06T17:46:14.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:14.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:14.401341+0000) 2024-03-06T17:46:15.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:15.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:15.401577+0000) 2024-03-06T17:46:16.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:16.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:16.401873+0000) 2024-03-06T17:46:17.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:17.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:17.402188+0000) 2024-03-06T17:46:18.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:18.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:18.402380+0000) 2024-03-06T17:46:19.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:19.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:19.402584+0000) 2024-03-06T17:46:20.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:20.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:20.402833+0000) 2024-03-06T17:46:21.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:21.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:21.403115+0000) 2024-03-06T17:46:22.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:22.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:22.403293+0000) 2024-03-06T17:46:23.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:23.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:23.403497+0000) 2024-03-06T17:46:24.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:24.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:24.403721+0000) 2024-03-06T17:46:25.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:25.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:25.403939+0000) 2024-03-06T17:46:26.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:26.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:26.404158+0000) 2024-03-06T17:46:27.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:27.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:27.404353+0000) 2024-03-06T17:46:28.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:28.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:28.404561+0000) 2024-03-06T17:46:29.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:29.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:29.404789+0000) 2024-03-06T17:46:30.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:30.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:30.404978+0000) 2024-03-06T17:46:31.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:31.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:31.405207+0000) 2024-03-06T17:46:32.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:32.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:32.405398+0000) 2024-03-06T17:46:33.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:33.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:33.405610+0000) 2024-03-06T17:46:34.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:34.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:34.405853+0000) 2024-03-06T17:46:35.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:35.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:35.406110+0000) 2024-03-06T17:46:36.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:36.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:36.406358+0000) 2024-03-06T17:46:36.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:36.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:46:37.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:37.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:37.406619+0000) 2024-03-06T17:46:38.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:38.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:38.406844+0000) 2024-03-06T17:46:39.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:39.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:39.407095+0000) 2024-03-06T17:46:40.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:40.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:40.407285+0000) 2024-03-06T17:46:41.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:41.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:41.407501+0000) 2024-03-06T17:46:42.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:42.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:42.407723+0000) 2024-03-06T17:46:43.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:43.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:43.407937+0000) 2024-03-06T17:46:44.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:44.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:44.408163+0000) 2024-03-06T17:46:45.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:45.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:45.408352+0000) 2024-03-06T17:46:46.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:46.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:46.408544+0000) 2024-03-06T17:46:47.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:47.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:47.408801+0000) 2024-03-06T17:46:48.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:48.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:48.409056+0000) 2024-03-06T17:46:49.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:49.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:49.409285+0000) 2024-03-06T17:46:50.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:50.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:50.409507+0000) 2024-03-06T17:46:51.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:51.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:51.409761+0000) 2024-03-06T17:46:52.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:52.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:52.409965+0000) 2024-03-06T17:46:53.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:53.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:53.410188+0000) 2024-03-06T17:46:54.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:54.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:54.410414+0000) 2024-03-06T17:46:55.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:55.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:55.410636+0000) 2024-03-06T17:46:56.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:56.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:56.410873+0000) 2024-03-06T17:46:57.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:57.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:57.411130+0000) 2024-03-06T17:46:58.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:58.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:58.411351+0000) 2024-03-06T17:46:59.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:46:59.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:46:59.411567+0000) 2024-03-06T17:47:00.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:00.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:00.411843+0000) 2024-03-06T17:47:01.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:01.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:01.412117+0000) 2024-03-06T17:47:02.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:02.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:02.412341+0000) 2024-03-06T17:47:03.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:03.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:03.412563+0000) 2024-03-06T17:47:04.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:04.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:04.412791+0000) 2024-03-06T17:47:05.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:05.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:05.413099+0000) 2024-03-06T17:47:06.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:06.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:06.413336+0000) 2024-03-06T17:47:07.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:07.028 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:07.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:07.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:07.413588+0000) 2024-03-06T17:47:08.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:08.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:08.413810+0000) 2024-03-06T17:47:09.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:09.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:09.414092+0000) 2024-03-06T17:47:10.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:10.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:10.414348+0000) 2024-03-06T17:47:11.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:11.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:11.414617+0000) 2024-03-06T17:47:12.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:12.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:12.414904+0000) 2024-03-06T17:47:13.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:13.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:13.415160+0000) 2024-03-06T17:47:14.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:14.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:14.415396+0000) 2024-03-06T17:47:15.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:15.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:15.415647+0000) 2024-03-06T17:47:16.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:16.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:16.415924+0000) 2024-03-06T17:47:17.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:17.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:17.416134+0000) 2024-03-06T17:47:18.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:18.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:18.416339+0000) 2024-03-06T17:47:19.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:19.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:19.416564+0000) 2024-03-06T17:47:20.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:20.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:20.416779+0000) 2024-03-06T17:47:21.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:21.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:21.417037+0000) 2024-03-06T17:47:22.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:22.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:22.417288+0000) 2024-03-06T17:47:23.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:23.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:23.417527+0000) 2024-03-06T17:47:24.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:24.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:24.417788+0000) 2024-03-06T17:47:25.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:25.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:25.418034+0000) 2024-03-06T17:47:26.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:26.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:26.418287+0000) 2024-03-06T17:47:27.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:27.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:27.418527+0000) 2024-03-06T17:47:28.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:28.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:28.418762+0000) 2024-03-06T17:47:29.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:29.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:29.419046+0000) 2024-03-06T17:47:30.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:30.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:30.419233+0000) 2024-03-06T17:47:31.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:31.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:31.419473+0000) 2024-03-06T17:47:32.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:32.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:32.419715+0000) 2024-03-06T17:47:33.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:33.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:33.419978+0000) 2024-03-06T17:47:34.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:34.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:34.420242+0000) 2024-03-06T17:47:35.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:35.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:35.420516+0000) 2024-03-06T17:47:36.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:36.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:36.420766+0000) 2024-03-06T17:47:37.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:37.178 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:47:37.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:37.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:37.420993+0000) 2024-03-06T17:47:38.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:38.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:38.421226+0000) 2024-03-06T17:47:39.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:39.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:39.421412+0000) 2024-03-06T17:47:40.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:40.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:40.421629+0000) 2024-03-06T17:47:41.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:41.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:41.421849+0000) 2024-03-06T17:47:42.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:42.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:42.422116+0000) 2024-03-06T17:47:43.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:43.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:43.422310+0000) 2024-03-06T17:47:44.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:44.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:44.422514+0000) 2024-03-06T17:47:45.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:45.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:45.422736+0000) 2024-03-06T17:47:46.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:46.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:46.422966+0000) 2024-03-06T17:47:47.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:47.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:47.423227+0000) 2024-03-06T17:47:48.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:48.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:48.423415+0000) 2024-03-06T17:47:49.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:49.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:49.423657+0000) 2024-03-06T17:47:50.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:50.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:50.423922+0000) 2024-03-06T17:47:51.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:51.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:51.424204+0000) 2024-03-06T17:47:52.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:52.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:52.424454+0000) 2024-03-06T17:47:53.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:53.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:53.424731+0000) 2024-03-06T17:47:54.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:54.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:54.425040+0000) 2024-03-06T17:47:55.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:55.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:55.425274+0000) 2024-03-06T17:47:56.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:56.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:56.425546+0000) 2024-03-06T17:47:57.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:57.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:57.425777+0000) 2024-03-06T17:47:58.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:58.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:58.425991+0000) 2024-03-06T17:47:59.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:47:59.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:47:59.426261+0000) 2024-03-06T17:48:00.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:00.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:00.426475+0000) 2024-03-06T17:48:01.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:01.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:01.426711+0000) 2024-03-06T17:48:02.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:02.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:02.426930+0000) 2024-03-06T17:48:03.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:03.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:03.427149+0000) 2024-03-06T17:48:04.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:04.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:04.427383+0000) 2024-03-06T17:48:05.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:05.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:05.427619+0000) 2024-03-06T17:48:06.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:06.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:06.427874+0000) 2024-03-06T17:48:07.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:07.315 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:07.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:07.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:07.428164+0000) 2024-03-06T17:48:08.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:08.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:08.428355+0000) 2024-03-06T17:48:09.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:09.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:09.428579+0000) 2024-03-06T17:48:10.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:10.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:10.428800+0000) 2024-03-06T17:48:11.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:11.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:11.429070+0000) 2024-03-06T17:48:12.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:12.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:12.429267+0000) 2024-03-06T17:48:13.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:13.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:13.429475+0000) 2024-03-06T17:48:14.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:14.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:14.429697+0000) 2024-03-06T17:48:15.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:15.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:15.429919+0000) 2024-03-06T17:48:16.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:16.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:16.430174+0000) 2024-03-06T17:48:17.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:17.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:17.430388+0000) 2024-03-06T17:48:18.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:18.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:18.430621+0000) 2024-03-06T17:48:19.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:19.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:19.430844+0000) 2024-03-06T17:48:20.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:20.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:20.431128+0000) 2024-03-06T17:48:21.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:21.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:21.431380+0000) 2024-03-06T17:48:22.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:22.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:22.431605+0000) 2024-03-06T17:48:23.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:23.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:23.431824+0000) 2024-03-06T17:48:24.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:24.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:24.432080+0000) 2024-03-06T17:48:25.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:25.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:25.432300+0000) 2024-03-06T17:48:26.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:26.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:26.432513+0000) 2024-03-06T17:48:27.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:27.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:27.432738+0000) 2024-03-06T17:48:28.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:28.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:28.432927+0000) 2024-03-06T17:48:29.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:29.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:29.433123+0000) 2024-03-06T17:48:30.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:30.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:30.433345+0000) 2024-03-06T17:48:31.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:31.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:31.433582+0000) 2024-03-06T17:48:32.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:32.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:32.433811+0000) 2024-03-06T17:48:33.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:33.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:33.434065+0000) 2024-03-06T17:48:34.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:34.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:34.434313+0000) 2024-03-06T17:48:35.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:35.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:35.434535+0000) 2024-03-06T17:48:36.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:36.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:36.434785+0000) 2024-03-06T17:48:37.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:37.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:37.435044+0000) 2024-03-06T17:48:37.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:37.453 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:48:38.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:38.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:38.435286+0000) 2024-03-06T17:48:39.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:39.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:39.435492+0000) 2024-03-06T17:48:40.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:40.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:40.435682+0000) 2024-03-06T17:48:41.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:41.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:41.435923+0000) 2024-03-06T17:48:42.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:42.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:42.436164+0000) 2024-03-06T17:48:43.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:43.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:43.436390+0000) 2024-03-06T17:48:44.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:44.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:44.436629+0000) 2024-03-06T17:48:45.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:45.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:45.436855+0000) 2024-03-06T17:48:46.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:46.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:46.437136+0000) 2024-03-06T17:48:47.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:47.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:47.437388+0000) 2024-03-06T17:48:48.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:48.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:48.437594+0000) 2024-03-06T17:48:49.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:49.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:49.437815+0000) 2024-03-06T17:48:50.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:50.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:50.438085+0000) 2024-03-06T17:48:51.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:51.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:51.438290+0000) 2024-03-06T17:48:52.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:52.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:52.438544+0000) 2024-03-06T17:48:53.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:53.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:53.438786+0000) 2024-03-06T17:48:54.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:54.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:54.439032+0000) 2024-03-06T17:48:55.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:55.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:55.439244+0000) 2024-03-06T17:48:56.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:56.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:56.439486+0000) 2024-03-06T17:48:57.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:57.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:57.439738+0000) 2024-03-06T17:48:58.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:58.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:58.440001+0000) 2024-03-06T17:48:59.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:48:59.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:48:59.440271+0000) 2024-03-06T17:49:00.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:00.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:00.440487+0000) 2024-03-06T17:49:01.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:01.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:01.440753+0000) 2024-03-06T17:49:02.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:02.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:02.440948+0000) 2024-03-06T17:49:03.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:03.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:03.441175+0000) 2024-03-06T17:49:04.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:04.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:04.441412+0000) 2024-03-06T17:49:05.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:05.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:05.441649+0000) 2024-03-06T17:49:06.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:06.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:06.441909+0000) 2024-03-06T17:49:07.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:07.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:07.442173+0000) 2024-03-06T17:49:07.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:07.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:08.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:08.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:08.442399+0000) 2024-03-06T17:49:09.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:09.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:09.442638+0000) 2024-03-06T17:49:10.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:10.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:10.442830+0000) 2024-03-06T17:49:11.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:11.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:11.443074+0000) 2024-03-06T17:49:12.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:12.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:12.443332+0000) 2024-03-06T17:49:13.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:13.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:13.443565+0000) 2024-03-06T17:49:14.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:14.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:14.443817+0000) 2024-03-06T17:49:15.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:15.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:15.444046+0000) 2024-03-06T17:49:16.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:16.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:16.444254+0000) 2024-03-06T17:49:17.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:17.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:17.444541+0000) 2024-03-06T17:49:18.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:18.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:18.444890+0000) 2024-03-06T17:49:19.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:19.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:19.445137+0000) 2024-03-06T17:49:20.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:20.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:20.445345+0000) 2024-03-06T17:49:21.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:21.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:21.445583+0000) 2024-03-06T17:49:22.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:22.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:22.445839+0000) 2024-03-06T17:49:23.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:23.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:23.446121+0000) 2024-03-06T17:49:24.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:24.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:24.446325+0000) 2024-03-06T17:49:25.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:25.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:25.446551+0000) 2024-03-06T17:49:26.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:26.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:26.446807+0000) 2024-03-06T17:49:27.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:27.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:27.447096+0000) 2024-03-06T17:49:28.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:28.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:28.447296+0000) 2024-03-06T17:49:29.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:29.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:29.447517+0000) 2024-03-06T17:49:30.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:30.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:30.447761+0000) 2024-03-06T17:49:31.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:31.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:31.448050+0000) 2024-03-06T17:49:32.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:32.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:32.448248+0000) 2024-03-06T17:49:33.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:33.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:33.448483+0000) 2024-03-06T17:49:34.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:34.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:34.448678+0000) 2024-03-06T17:49:35.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:35.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:35.448952+0000) 2024-03-06T17:49:36.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:36.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:36.449184+0000) 2024-03-06T17:49:37.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:37.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:37.449436+0000) 2024-03-06T17:49:37.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:37.726 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:49:38.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:38.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:38.449677+0000) 2024-03-06T17:49:39.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:39.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:39.449914+0000) 2024-03-06T17:49:40.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:40.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:40.450139+0000) 2024-03-06T17:49:41.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:41.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:41.450353+0000) 2024-03-06T17:49:42.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:42.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:42.450561+0000) 2024-03-06T17:49:43.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:43.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:43.450789+0000) 2024-03-06T17:49:44.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:44.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:44.451038+0000) 2024-03-06T17:49:45.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:45.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:45.451272+0000) 2024-03-06T17:49:46.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:46.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:46.451544+0000) 2024-03-06T17:49:47.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:47.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:47.451795+0000) 2024-03-06T17:49:48.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:48.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:48.452061+0000) 2024-03-06T17:49:49.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:49.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:49.452267+0000) 2024-03-06T17:49:50.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:50.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:50.452502+0000) 2024-03-06T17:49:51.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:51.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:51.452748+0000) 2024-03-06T17:49:52.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:52.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:52.452978+0000) 2024-03-06T17:49:53.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:53.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:53.453246+0000) 2024-03-06T17:49:54.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:54.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:54.453447+0000) 2024-03-06T17:49:55.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:55.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:55.453664+0000) 2024-03-06T17:49:56.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:56.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:56.453927+0000) 2024-03-06T17:49:57.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:57.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:57.454148+0000) 2024-03-06T17:49:58.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:58.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:58.454384+0000) 2024-03-06T17:49:59.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:49:59.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:49:59.454626+0000) 2024-03-06T17:50:00.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:00.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:00.454871+0000) 2024-03-06T17:50:01.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:01.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:01.455151+0000) 2024-03-06T17:50:02.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:02.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:02.455343+0000) 2024-03-06T17:50:03.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:03.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:03.455520+0000) 2024-03-06T17:50:04.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:04.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:04.455748+0000) 2024-03-06T17:50:05.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:05.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:05.455986+0000) 2024-03-06T17:50:06.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:06.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:06.456246+0000) 2024-03-06T17:50:07.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:07.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:07.456437+0000) 2024-03-06T17:50:07.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:07.866 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:08.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:08.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:08.456689+0000) 2024-03-06T17:50:09.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:09.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:09.456930+0000) 2024-03-06T17:50:10.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:10.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:10.457167+0000) 2024-03-06T17:50:11.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:11.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:11.457383+0000) 2024-03-06T17:50:12.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:12.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:12.457619+0000) 2024-03-06T17:50:13.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:13.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:13.457867+0000) 2024-03-06T17:50:14.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:14.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:14.458141+0000) 2024-03-06T17:50:15.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:15.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:15.458332+0000) 2024-03-06T17:50:16.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:16.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:16.458579+0000) 2024-03-06T17:50:17.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:17.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:17.458821+0000) 2024-03-06T17:50:18.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:18.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:18.459082+0000) 2024-03-06T17:50:19.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:19.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:19.459303+0000) 2024-03-06T17:50:20.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:20.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:20.459527+0000) 2024-03-06T17:50:21.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:21.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:21.459773+0000) 2024-03-06T17:50:22.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:22.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:22.460049+0000) 2024-03-06T17:50:23.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:23.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:23.460290+0000) 2024-03-06T17:50:24.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:24.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:24.460519+0000) 2024-03-06T17:50:25.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:25.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:25.460745+0000) 2024-03-06T17:50:26.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:26.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:26.460947+0000) 2024-03-06T17:50:27.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:27.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:27.461202+0000) 2024-03-06T17:50:28.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:28.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:28.461481+0000) 2024-03-06T17:50:29.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:29.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:29.461659+0000) 2024-03-06T17:50:30.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:30.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:30.461902+0000) 2024-03-06T17:50:31.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:31.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:31.462172+0000) 2024-03-06T17:50:32.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:32.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:32.462400+0000) 2024-03-06T17:50:33.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:33.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:33.463077+0000) 2024-03-06T17:50:34.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:34.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:34.463362+0000) 2024-03-06T17:50:35.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:35.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:35.463604+0000) 2024-03-06T17:50:36.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:36.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:36.463869+0000) 2024-03-06T17:50:37.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:37.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:37.464169+0000) 2024-03-06T17:50:38.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:38.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:50:38.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:38.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:38.464424+0000) 2024-03-06T17:50:39.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:39.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:39.464713+0000) 2024-03-06T17:50:40.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:40.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:40.464941+0000) 2024-03-06T17:50:41.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:41.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:41.465233+0000) 2024-03-06T17:50:42.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:42.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:42.465488+0000) 2024-03-06T17:50:43.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:43.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:43.465725+0000) 2024-03-06T17:50:44.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:44.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:44.465982+0000) 2024-03-06T17:50:45.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:45.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:45.466269+0000) 2024-03-06T17:50:46.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:46.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:46.466567+0000) 2024-03-06T17:50:47.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:47.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:47.466782+0000) 2024-03-06T17:50:48.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:48.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:48.467039+0000) 2024-03-06T17:50:49.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:49.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:49.467247+0000) 2024-03-06T17:50:50.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:50.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:50.467430+0000) 2024-03-06T17:50:51.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:51.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:51.467708+0000) 2024-03-06T17:50:52.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:52.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:52.467959+0000) 2024-03-06T17:50:53.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:53.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:53.468190+0000) 2024-03-06T17:50:54.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:54.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:54.468475+0000) 2024-03-06T17:50:55.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:55.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:55.468747+0000) 2024-03-06T17:50:56.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:56.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:56.469104+0000) 2024-03-06T17:50:57.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:57.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:57.469387+0000) 2024-03-06T17:50:58.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:58.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:58.469663+0000) 2024-03-06T17:50:59.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:50:59.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:50:59.469939+0000) 2024-03-06T17:51:00.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:00.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:00.470186+0000) 2024-03-06T17:51:01.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:01.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:01.470532+0000) 2024-03-06T17:51:02.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:02.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:02.470778+0000) 2024-03-06T17:51:03.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:03.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:03.470996+0000) 2024-03-06T17:51:04.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:04.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:04.471218+0000) 2024-03-06T17:51:05.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:05.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:05.471437+0000) 2024-03-06T17:51:06.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:06.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:06.471684+0000) 2024-03-06T17:51:07.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:07.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:07.471897+0000) 2024-03-06T17:51:08.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:08.163 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:08.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:08.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:08.472125+0000) 2024-03-06T17:51:09.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:09.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:09.472363+0000) 2024-03-06T17:51:10.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:10.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:10.472553+0000) 2024-03-06T17:51:11.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:11.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:11.472783+0000) 2024-03-06T17:51:12.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:12.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:12.473060+0000) 2024-03-06T17:51:13.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:13.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:13.473313+0000) 2024-03-06T17:51:14.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:14.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:14.473551+0000) 2024-03-06T17:51:15.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:15.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:15.473750+0000) 2024-03-06T17:51:16.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:16.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:16.474124+0000) 2024-03-06T17:51:17.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:17.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:17.474358+0000) 2024-03-06T17:51:18.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:18.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:18.474578+0000) 2024-03-06T17:51:19.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:19.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:19.474793+0000) 2024-03-06T17:51:20.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:20.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:20.475052+0000) 2024-03-06T17:51:21.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:21.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:21.475284+0000) 2024-03-06T17:51:22.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:22.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:22.475535+0000) 2024-03-06T17:51:23.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:23.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:23.475782+0000) 2024-03-06T17:51:24.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:24.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:24.475987+0000) 2024-03-06T17:51:25.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:25.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:25.476255+0000) 2024-03-06T17:51:26.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:26.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:26.476454+0000) 2024-03-06T17:51:27.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:27.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:27.476696+0000) 2024-03-06T17:51:28.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:28.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:28.476941+0000) 2024-03-06T17:51:29.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:29.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:29.477163+0000) 2024-03-06T17:51:30.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:30.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:30.477391+0000) 2024-03-06T17:51:31.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:31.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:31.477587+0000) 2024-03-06T17:51:32.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:32.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:32.477813+0000) 2024-03-06T17:51:33.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:33.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:33.478042+0000) 2024-03-06T17:51:34.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:34.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:34.478294+0000) 2024-03-06T17:51:35.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:35.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:35.478519+0000) 2024-03-06T17:51:36.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:36.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:36.478790+0000) 2024-03-06T17:51:37.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:37.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:37.479118+0000) 2024-03-06T17:51:38.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:38.364 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:51:38.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:38.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:38.479349+0000) 2024-03-06T17:51:39.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:39.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:39.479582+0000) 2024-03-06T17:51:40.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:40.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:40.479849+0000) 2024-03-06T17:51:41.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:41.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:41.480167+0000) 2024-03-06T17:51:42.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:42.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:42.480434+0000) 2024-03-06T17:51:43.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:43.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:43.480696+0000) 2024-03-06T17:51:44.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:44.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:44.480976+0000) 2024-03-06T17:51:45.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:45.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:45.481313+0000) 2024-03-06T17:51:46.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:46.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:46.481600+0000) 2024-03-06T17:51:47.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:47.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:47.481865+0000) 2024-03-06T17:51:48.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:48.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:48.482119+0000) 2024-03-06T17:51:49.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:49.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:49.482332+0000) 2024-03-06T17:51:50.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:50.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:50.482568+0000) 2024-03-06T17:51:51.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:51.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:51.482778+0000) 2024-03-06T17:51:52.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:52.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:52.482994+0000) 2024-03-06T17:51:53.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:53.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:53.483331+0000) 2024-03-06T17:51:54.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:54.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:54.483610+0000) 2024-03-06T17:51:55.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:55.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:55.483890+0000) 2024-03-06T17:51:56.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:56.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:56.484173+0000) 2024-03-06T17:51:57.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:57.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:57.484491+0000) 2024-03-06T17:51:58.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:58.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:58.484754+0000) 2024-03-06T17:51:59.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:51:59.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:51:59.484994+0000) 2024-03-06T17:52:00.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:00.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:00.485231+0000) 2024-03-06T17:52:01.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:01.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:01.485454+0000) 2024-03-06T17:52:02.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:02.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:02.485666+0000) 2024-03-06T17:52:03.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:03.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:03.485887+0000) 2024-03-06T17:52:04.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:04.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:04.486238+0000) 2024-03-06T17:52:05.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:05.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:05.486506+0000) 2024-03-06T17:52:06.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:06.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:06.486707+0000) 2024-03-06T17:52:07.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:07.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:07.486936+0000) 2024-03-06T17:52:08.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:08.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:08.487154+0000) 2024-03-06T17:52:08.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:08.692 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:09.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:09.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:09.487348+0000) 2024-03-06T17:52:10.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:10.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:10.487571+0000) 2024-03-06T17:52:11.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:11.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:11.487791+0000) 2024-03-06T17:52:12.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:12.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:12.488052+0000) 2024-03-06T17:52:13.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:13.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:13.488248+0000) 2024-03-06T17:52:14.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:14.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:14.488429+0000) 2024-03-06T17:52:15.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:15.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:15.488639+0000) 2024-03-06T17:52:16.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:16.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:16.488856+0000) 2024-03-06T17:52:17.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:17.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:17.489133+0000) 2024-03-06T17:52:18.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:18.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:18.489307+0000) 2024-03-06T17:52:19.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:19.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:19.489499+0000) 2024-03-06T17:52:20.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:20.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:20.489724+0000) 2024-03-06T17:52:21.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:21.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:21.489945+0000) 2024-03-06T17:52:22.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:22.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:22.490165+0000) 2024-03-06T17:52:23.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:23.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:23.490386+0000) 2024-03-06T17:52:24.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:24.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:24.490603+0000) 2024-03-06T17:52:25.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:25.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:25.490763+0000) 2024-03-06T17:52:26.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:26.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:26.490934+0000) 2024-03-06T17:52:27.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:27.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:27.491163+0000) 2024-03-06T17:52:28.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:28.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:28.491380+0000) 2024-03-06T17:52:29.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:29.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:29.491630+0000) 2024-03-06T17:52:30.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:30.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:30.491905+0000) 2024-03-06T17:52:31.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:31.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:31.492150+0000) 2024-03-06T17:52:32.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:32.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:32.492345+0000) 2024-03-06T17:52:33.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:33.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:33.492563+0000) 2024-03-06T17:52:34.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:34.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:34.492783+0000) 2024-03-06T17:52:35.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:35.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:35.492999+0000) 2024-03-06T17:52:36.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:36.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:36.493248+0000) 2024-03-06T17:52:37.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:37.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:37.493460+0000) 2024-03-06T17:52:38.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:38.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:38.493671+0000) 2024-03-06T17:52:38.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:38.847 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:52:39.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:39.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:39.493945+0000) 2024-03-06T17:52:40.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:40.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:40.494207+0000) 2024-03-06T17:52:41.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:41.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:41.494459+0000) 2024-03-06T17:52:42.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:42.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:42.494731+0000) 2024-03-06T17:52:43.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:43.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:43.495042+0000) 2024-03-06T17:52:44.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:44.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:44.495320+0000) 2024-03-06T17:52:45.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:45.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:45.495530+0000) 2024-03-06T17:52:46.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:46.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:46.495754+0000) 2024-03-06T17:52:47.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:47.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:47.495973+0000) 2024-03-06T17:52:48.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:48.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:48.496201+0000) 2024-03-06T17:52:49.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:49.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:49.496392+0000) 2024-03-06T17:52:50.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:50.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:50.496626+0000) 2024-03-06T17:52:51.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:51.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:51.496844+0000) 2024-03-06T17:52:52.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:52.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:52.497123+0000) 2024-03-06T17:52:53.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:53.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:53.497358+0000) 2024-03-06T17:52:54.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:54.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:54.497639+0000) 2024-03-06T17:52:55.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:55.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:55.497912+0000) 2024-03-06T17:52:56.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:56.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:56.498183+0000) 2024-03-06T17:52:57.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:57.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:57.498505+0000) 2024-03-06T17:52:58.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:58.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:58.498772+0000) 2024-03-06T17:52:59.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:52:59.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:52:59.499092+0000) 2024-03-06T17:53:00.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:00.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:00.499369+0000) 2024-03-06T17:53:01.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:01.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:01.499649+0000) 2024-03-06T17:53:02.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:02.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:02.499885+0000) 2024-03-06T17:53:03.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:03.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:03.500152+0000) 2024-03-06T17:53:04.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:04.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:04.500369+0000) 2024-03-06T17:53:05.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:05.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:05.500588+0000) 2024-03-06T17:53:06.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:06.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:06.500767+0000) 2024-03-06T17:53:07.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:07.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:07.501001+0000) 2024-03-06T17:53:08.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:08.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:08.501242+0000) 2024-03-06T17:53:09.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:09.034 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:09.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:09.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:09.501475+0000) 2024-03-06T17:53:10.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:10.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:10.501708+0000) 2024-03-06T17:53:11.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:11.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:11.501930+0000) 2024-03-06T17:53:12.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:12.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:12.502129+0000) 2024-03-06T17:53:13.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:13.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:13.502304+0000) 2024-03-06T17:53:14.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:14.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:14.502525+0000) 2024-03-06T17:53:15.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:15.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:15.502751+0000) 2024-03-06T17:53:16.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:16.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:16.502986+0000) 2024-03-06T17:53:17.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:17.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:17.503282+0000) 2024-03-06T17:53:18.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:18.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:18.503547+0000) 2024-03-06T17:53:19.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:19.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:19.503742+0000) 2024-03-06T17:53:20.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:20.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:20.503966+0000) 2024-03-06T17:53:21.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:21.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:21.504198+0000) 2024-03-06T17:53:22.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:22.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:22.504433+0000) 2024-03-06T17:53:23.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:23.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:23.504647+0000) 2024-03-06T17:53:24.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:24.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:24.504871+0000) 2024-03-06T17:53:25.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:25.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:25.505149+0000) 2024-03-06T17:53:26.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:26.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:26.505377+0000) 2024-03-06T17:53:27.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:27.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:27.505657+0000) 2024-03-06T17:53:28.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:28.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:28.505941+0000) 2024-03-06T17:53:29.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:29.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:29.506209+0000) 2024-03-06T17:53:30.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:30.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:30.506456+0000) 2024-03-06T17:53:31.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:31.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:31.506727+0000) 2024-03-06T17:53:32.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:32.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:32.507039+0000) 2024-03-06T17:53:33.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:33.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:33.507274+0000) 2024-03-06T17:53:34.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:34.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:34.507559+0000) 2024-03-06T17:53:35.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:35.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:35.507835+0000) 2024-03-06T17:53:36.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:36.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:36.508137+0000) 2024-03-06T17:53:37.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:37.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:37.508407+0000) 2024-03-06T17:53:38.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:38.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:38.508684+0000) 2024-03-06T17:53:39.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:39.248 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:53:39.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:39.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:39.508932+0000) 2024-03-06T17:53:40.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:40.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:40.509178+0000) 2024-03-06T17:53:41.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:41.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:41.509405+0000) 2024-03-06T17:53:42.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:42.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:42.509656+0000) 2024-03-06T17:53:43.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:43.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:43.509883+0000) 2024-03-06T17:53:44.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:44.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:44.510107+0000) 2024-03-06T17:53:45.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:45.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:45.510310+0000) 2024-03-06T17:53:46.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:46.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:46.510533+0000) 2024-03-06T17:53:47.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:47.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:47.510767+0000) 2024-03-06T17:53:48.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:48.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:48.510995+0000) 2024-03-06T17:53:49.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:49.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:49.511227+0000) 2024-03-06T17:53:50.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:50.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:50.511462+0000) 2024-03-06T17:53:51.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:51.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:51.511684+0000) 2024-03-06T17:53:52.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:52.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:52.511890+0000) 2024-03-06T17:53:53.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:53.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:53.512134+0000) 2024-03-06T17:53:54.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:54.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:54.512298+0000) 2024-03-06T17:53:55.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:55.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:55.512517+0000) 2024-03-06T17:53:56.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:56.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:56.512751+0000) 2024-03-06T17:53:57.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:57.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:57.512977+0000) 2024-03-06T17:53:58.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:58.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:58.513228+0000) 2024-03-06T17:53:59.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:53:59.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:53:59.513414+0000) 2024-03-06T17:54:00.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:00.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:00.513642+0000) 2024-03-06T17:54:01.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:01.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:01.513866+0000) 2024-03-06T17:54:02.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:02.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:02.514173+0000) 2024-03-06T17:54:03.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:03.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:03.514407+0000) 2024-03-06T17:54:04.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:04.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:04.514673+0000) 2024-03-06T17:54:05.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:05.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:05.514880+0000) 2024-03-06T17:54:06.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:06.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:06.515083+0000) 2024-03-06T17:54:07.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:07.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:07.515324+0000) 2024-03-06T17:54:08.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:08.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:08.515541+0000) 2024-03-06T17:54:09.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:09.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:09.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:09.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:09.515737+0000) 2024-03-06T17:54:10.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:10.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:10.515910+0000) 2024-03-06T17:54:11.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:11.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:11.516132+0000) 2024-03-06T17:54:12.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:12.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:12.516374+0000) 2024-03-06T17:54:13.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:13.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:13.516598+0000) 2024-03-06T17:54:14.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:14.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:14.516822+0000) 2024-03-06T17:54:15.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:15.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:15.517084+0000) 2024-03-06T17:54:16.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:16.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:16.517319+0000) 2024-03-06T17:54:17.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:17.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:17.517568+0000) 2024-03-06T17:54:18.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:18.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:18.517792+0000) 2024-03-06T17:54:19.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:19.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:19.518044+0000) 2024-03-06T17:54:20.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:20.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:20.518272+0000) 2024-03-06T17:54:21.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:21.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:21.518496+0000) 2024-03-06T17:54:22.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:22.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:22.518707+0000) 2024-03-06T17:54:23.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:23.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:23.518927+0000) 2024-03-06T17:54:24.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:24.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:24.519163+0000) 2024-03-06T17:54:25.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:25.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:25.519382+0000) 2024-03-06T17:54:26.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:26.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:26.519608+0000) 2024-03-06T17:54:27.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:27.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:27.519867+0000) 2024-03-06T17:54:28.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:28.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:28.520159+0000) 2024-03-06T17:54:29.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:29.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:29.520383+0000) 2024-03-06T17:54:30.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:30.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:30.520632+0000) 2024-03-06T17:54:31.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:31.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:31.520841+0000) 2024-03-06T17:54:32.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:32.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:32.521135+0000) 2024-03-06T17:54:33.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:33.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:33.521427+0000) 2024-03-06T17:54:34.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:34.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:34.521660+0000) 2024-03-06T17:54:35.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:35.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:35.521882+0000) 2024-03-06T17:54:36.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:36.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:36.522111+0000) 2024-03-06T17:54:37.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:37.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:37.522362+0000) 2024-03-06T17:54:38.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:38.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:38.522544+0000) 2024-03-06T17:54:39.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:39.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:39.522720+0000) 2024-03-06T17:54:39.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:39.673 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:54:40.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:40.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:40.522945+0000) 2024-03-06T17:54:41.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:41.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:41.523164+0000) 2024-03-06T17:54:42.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:42.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:42.523406+0000) 2024-03-06T17:54:43.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:43.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:43.523597+0000) 2024-03-06T17:54:44.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:44.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:44.523836+0000) 2024-03-06T17:54:45.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:45.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:45.524063+0000) 2024-03-06T17:54:46.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:46.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:46.524301+0000) 2024-03-06T17:54:47.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:47.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:47.524552+0000) 2024-03-06T17:54:48.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:48.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:48.524778+0000) 2024-03-06T17:54:49.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:49.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:49.525000+0000) 2024-03-06T17:54:50.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:50.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:50.525233+0000) 2024-03-06T17:54:51.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:51.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:51.525425+0000) 2024-03-06T17:54:52.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:52.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:52.525670+0000) 2024-03-06T17:54:53.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:53.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:53.525894+0000) 2024-03-06T17:54:54.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:54.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:54.526132+0000) 2024-03-06T17:54:55.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:55.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:55.526334+0000) 2024-03-06T17:54:56.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:56.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:56.526570+0000) 2024-03-06T17:54:57.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:57.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:57.526822+0000) 2024-03-06T17:54:58.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:58.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:58.527089+0000) 2024-03-06T17:54:59.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:54:59.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:54:59.527317+0000) 2024-03-06T17:55:00.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:00.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:00.527546+0000) 2024-03-06T17:55:01.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:01.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:01.527753+0000) 2024-03-06T17:55:02.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:02.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:02.527991+0000) 2024-03-06T17:55:03.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:03.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:03.528253+0000) 2024-03-06T17:55:04.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:04.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:04.528458+0000) 2024-03-06T17:55:05.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:05.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:05.528681+0000) 2024-03-06T17:55:06.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:06.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:06.528905+0000) 2024-03-06T17:55:07.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:07.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:07.529166+0000) 2024-03-06T17:55:08.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:08.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:08.529394+0000) 2024-03-06T17:55:09.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:09.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:09.529620+0000) 2024-03-06T17:55:09.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:09.836 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:10.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:10.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:10.529801+0000) 2024-03-06T17:55:11.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:11.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:11.530069+0000) 2024-03-06T17:55:12.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:12.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:12.530300+0000) 2024-03-06T17:55:13.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:13.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:13.530486+0000) 2024-03-06T17:55:14.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:14.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:14.530681+0000) 2024-03-06T17:55:15.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:15.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:15.530866+0000) 2024-03-06T17:55:16.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:16.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:16.531138+0000) 2024-03-06T17:55:17.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:17.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:17.531349+0000) 2024-03-06T17:55:18.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:18.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:18.531537+0000) 2024-03-06T17:55:19.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:19.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:19.531762+0000) 2024-03-06T17:55:20.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:20.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:20.531942+0000) 2024-03-06T17:55:21.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:21.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:21.532160+0000) 2024-03-06T17:55:22.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:22.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:22.532400+0000) 2024-03-06T17:55:23.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:23.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:23.532621+0000) 2024-03-06T17:55:24.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:24.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:24.532839+0000) 2024-03-06T17:55:25.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:25.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:25.533097+0000) 2024-03-06T17:55:26.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:26.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:26.533326+0000) 2024-03-06T17:55:27.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:27.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:27.533755+0000) 2024-03-06T17:55:28.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:28.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:28.533893+0000) 2024-03-06T17:55:29.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:29.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:29.534107+0000) 2024-03-06T17:55:30.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:30.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:30.534356+0000) 2024-03-06T17:55:31.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:31.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:31.534578+0000) 2024-03-06T17:55:32.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:32.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:32.534783+0000) 2024-03-06T17:55:33.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:33.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:33.535043+0000) 2024-03-06T17:55:34.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:34.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:34.535255+0000) 2024-03-06T17:55:35.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:35.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:35.535480+0000) 2024-03-06T17:55:36.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:36.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:36.535699+0000) 2024-03-06T17:55:37.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:37.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:37.535980+0000) 2024-03-06T17:55:38.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:38.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:38.536226+0000) 2024-03-06T17:55:39.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:39.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:39.536418+0000) 2024-03-06T17:55:39.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:39.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:55:40.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:40.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:40.536659+0000) 2024-03-06T17:55:41.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:41.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:41.536883+0000) 2024-03-06T17:55:42.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:42.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:42.537155+0000) 2024-03-06T17:55:43.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:43.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:43.537392+0000) 2024-03-06T17:55:44.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:44.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:44.537632+0000) 2024-03-06T17:55:45.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:45.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:45.537868+0000) 2024-03-06T17:55:46.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:46.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:46.538082+0000) 2024-03-06T17:55:47.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:47.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:47.538315+0000) 2024-03-06T17:55:48.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:48.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:48.538560+0000) 2024-03-06T17:55:49.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:49.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:49.538819+0000) 2024-03-06T17:55:50.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:50.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:50.539119+0000) 2024-03-06T17:55:51.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:51.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:51.539328+0000) 2024-03-06T17:55:52.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:52.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:52.539560+0000) 2024-03-06T17:55:53.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:53.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:53.539816+0000) 2024-03-06T17:55:54.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:54.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:54.540044+0000) 2024-03-06T17:55:55.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:55.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:55.540219+0000) 2024-03-06T17:55:56.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:56.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:56.540455+0000) 2024-03-06T17:55:57.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:57.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:57.540726+0000) 2024-03-06T17:55:58.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:58.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:58.540992+0000) 2024-03-06T17:55:59.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:55:59.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:55:59.541292+0000) 2024-03-06T17:56:00.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:00.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:00.541515+0000) 2024-03-06T17:56:01.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:01.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:01.541689+0000) 2024-03-06T17:56:02.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:02.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:02.541940+0000) 2024-03-06T17:56:03.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:03.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:03.542166+0000) 2024-03-06T17:56:04.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:04.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:04.542389+0000) 2024-03-06T17:56:05.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:05.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:05.542607+0000) 2024-03-06T17:56:06.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:06.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:06.542838+0000) 2024-03-06T17:56:07.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:07.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:07.543125+0000) 2024-03-06T17:56:08.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:08.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:08.543348+0000) 2024-03-06T17:56:09.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:09.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:09.543577+0000) 2024-03-06T17:56:10.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:10.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:10.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:10.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:10.543840+0000) 2024-03-06T17:56:11.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:11.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:11.544073+0000) 2024-03-06T17:56:12.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:12.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:12.544321+0000) 2024-03-06T17:56:13.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:13.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:13.544551+0000) 2024-03-06T17:56:14.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:14.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:14.544820+0000) 2024-03-06T17:56:15.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:15.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:15.545187+0000) 2024-03-06T17:56:16.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:16.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:16.545481+0000) 2024-03-06T17:56:17.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:17.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:17.545794+0000) 2024-03-06T17:56:18.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:18.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:18.546112+0000) 2024-03-06T17:56:19.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:19.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:19.546368+0000) 2024-03-06T17:56:20.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:20.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:20.546607+0000) 2024-03-06T17:56:21.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:21.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:21.546827+0000) 2024-03-06T17:56:22.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:22.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:22.547161+0000) 2024-03-06T17:56:23.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:23.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:23.547436+0000) 2024-03-06T17:56:24.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:24.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:24.547715+0000) 2024-03-06T17:56:25.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:25.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:25.547953+0000) 2024-03-06T17:56:26.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:26.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:26.548178+0000) 2024-03-06T17:56:27.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:27.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:27.548436+0000) 2024-03-06T17:56:28.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:28.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:28.548659+0000) 2024-03-06T17:56:29.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:29.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:29.548917+0000) 2024-03-06T17:56:30.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:30.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:30.549167+0000) 2024-03-06T17:56:31.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:31.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:31.549358+0000) 2024-03-06T17:56:32.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:32.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:32.549607+0000) 2024-03-06T17:56:33.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:33.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:33.549846+0000) 2024-03-06T17:56:34.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:34.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:34.550144+0000) 2024-03-06T17:56:35.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:35.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:35.550339+0000) 2024-03-06T17:56:36.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:36.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:36.550553+0000) 2024-03-06T17:56:37.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:37.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:37.550844+0000) 2024-03-06T17:56:38.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:38.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:38.551119+0000) 2024-03-06T17:56:39.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:39.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:39.551324+0000) 2024-03-06T17:56:40.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:40.398 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:56:40.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:40.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:40.551555+0000) 2024-03-06T17:56:41.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:41.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:41.551789+0000) 2024-03-06T17:56:42.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:42.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:42.552062+0000) 2024-03-06T17:56:43.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:43.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:43.552292+0000) 2024-03-06T17:56:44.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:44.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:44.552519+0000) 2024-03-06T17:56:45.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:45.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:45.552706+0000) 2024-03-06T17:56:46.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:46.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:46.552936+0000) 2024-03-06T17:56:47.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:47.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:47.553231+0000) 2024-03-06T17:56:48.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:48.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:48.553486+0000) 2024-03-06T17:56:49.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:49.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:49.553729+0000) 2024-03-06T17:56:50.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:50.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:50.553974+0000) 2024-03-06T17:56:51.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:51.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:51.554320+0000) 2024-03-06T17:56:52.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:52.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:52.554567+0000) 2024-03-06T17:56:53.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:53.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:53.554808+0000) 2024-03-06T17:56:54.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:54.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:54.555041+0000) 2024-03-06T17:56:55.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:55.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:55.555233+0000) 2024-03-06T17:56:56.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:56.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:56.555458+0000) 2024-03-06T17:56:57.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:57.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:57.555692+0000) 2024-03-06T17:56:58.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:58.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:58.555970+0000) 2024-03-06T17:56:59.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:56:59.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:56:59.556243+0000) 2024-03-06T17:57:00.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:00.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:00.556521+0000) 2024-03-06T17:57:01.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:01.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:01.556789+0000) 2024-03-06T17:57:02.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:02.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:02.557111+0000) 2024-03-06T17:57:03.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:03.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:03.557403+0000) 2024-03-06T17:57:04.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:04.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:04.557634+0000) 2024-03-06T17:57:05.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:05.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:05.557859+0000) 2024-03-06T17:57:06.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:06.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:06.558121+0000) 2024-03-06T17:57:07.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:07.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:07.558369+0000) 2024-03-06T17:57:08.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:08.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:08.558612+0000) 2024-03-06T17:57:09.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:09.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:09.558870+0000) 2024-03-06T17:57:10.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:10.552 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:10.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:10.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:10.559159+0000) 2024-03-06T17:57:11.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:11.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:11.559397+0000) 2024-03-06T17:57:12.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:12.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:12.559576+0000) 2024-03-06T17:57:13.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:13.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:13.559806+0000) 2024-03-06T17:57:14.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:14.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:14.560076+0000) 2024-03-06T17:57:15.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:15.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:15.560300+0000) 2024-03-06T17:57:16.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:16.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:16.560534+0000) 2024-03-06T17:57:17.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:17.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:17.560769+0000) 2024-03-06T17:57:18.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:18.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:18.560965+0000) 2024-03-06T17:57:19.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:19.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:19.561238+0000) 2024-03-06T17:57:20.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:20.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:20.561465+0000) 2024-03-06T17:57:21.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:21.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:21.561650+0000) 2024-03-06T17:57:22.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:22.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:22.561872+0000) 2024-03-06T17:57:23.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:23.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:23.562132+0000) 2024-03-06T17:57:24.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:24.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:24.562350+0000) 2024-03-06T17:57:25.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:25.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:25.562576+0000) 2024-03-06T17:57:26.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:26.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:26.562793+0000) 2024-03-06T17:57:27.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:27.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:27.563055+0000) 2024-03-06T17:57:28.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:28.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:28.563307+0000) 2024-03-06T17:57:29.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:29.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:29.563539+0000) 2024-03-06T17:57:30.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:30.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:30.563722+0000) 2024-03-06T17:57:31.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:31.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:31.563981+0000) 2024-03-06T17:57:32.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:32.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:32.564315+0000) 2024-03-06T17:57:33.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:33.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:33.564557+0000) 2024-03-06T17:57:34.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:34.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:34.564779+0000) 2024-03-06T17:57:35.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:35.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:35.565069+0000) 2024-03-06T17:57:36.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:36.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:36.565313+0000) 2024-03-06T17:57:37.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:37.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:37.565602+0000) 2024-03-06T17:57:38.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:38.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:38.565839+0000) 2024-03-06T17:57:39.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:39.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:39.566142+0000) 2024-03-06T17:57:40.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:40.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:40.566389+0000) 2024-03-06T17:57:40.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:40.698 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:57:41.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:41.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:41.566584+0000) 2024-03-06T17:57:42.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:42.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:42.566862+0000) 2024-03-06T17:57:43.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:43.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:43.567113+0000) 2024-03-06T17:57:44.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:44.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:44.567333+0000) 2024-03-06T17:57:45.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:45.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:45.567569+0000) 2024-03-06T17:57:46.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:46.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:46.567815+0000) 2024-03-06T17:57:47.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:47.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:47.568082+0000) 2024-03-06T17:57:48.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:48.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:48.568304+0000) 2024-03-06T17:57:49.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:49.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:49.568529+0000) 2024-03-06T17:57:50.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:50.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:50.568743+0000) 2024-03-06T17:57:51.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:51.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:51.568996+0000) 2024-03-06T17:57:52.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:52.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:52.569302+0000) 2024-03-06T17:57:53.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:53.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:53.569508+0000) 2024-03-06T17:57:54.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:54.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:54.569731+0000) 2024-03-06T17:57:55.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:55.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:55.569990+0000) 2024-03-06T17:57:56.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:56.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:56.570249+0000) 2024-03-06T17:57:57.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:57.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:57.570519+0000) 2024-03-06T17:57:58.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:58.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:58.570752+0000) 2024-03-06T17:57:59.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:57:59.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:57:59.570991+0000) 2024-03-06T17:58:00.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:00.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:00.571267+0000) 2024-03-06T17:58:01.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:01.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:01.571535+0000) 2024-03-06T17:58:02.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:02.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:02.571768+0000) 2024-03-06T17:58:03.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:03.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:03.572098+0000) 2024-03-06T17:58:04.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:04.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:04.572383+0000) 2024-03-06T17:58:05.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:05.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:05.572664+0000) 2024-03-06T17:58:06.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:06.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:06.572950+0000) 2024-03-06T17:58:07.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:07.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:07.573271+0000) 2024-03-06T17:58:08.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:08.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:08.573543+0000) 2024-03-06T17:58:09.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:09.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:09.573821+0000) 2024-03-06T17:58:10.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:10.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:10.574135+0000) 2024-03-06T17:58:10.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:10.841 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:11.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:11.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:11.574347+0000) 2024-03-06T17:58:12.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:12.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:12.574580+0000) 2024-03-06T17:58:13.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:13.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:13.574799+0000) 2024-03-06T17:58:14.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:14.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:14.575022+0000) 2024-03-06T17:58:15.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:15.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:15.575241+0000) 2024-03-06T17:58:16.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:16.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:16.575466+0000) 2024-03-06T17:58:17.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:17.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:17.575706+0000) 2024-03-06T17:58:18.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:18.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:18.575925+0000) 2024-03-06T17:58:19.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:19.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:19.576161+0000) 2024-03-06T17:58:20.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:20.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:20.576607+0000) 2024-03-06T17:58:21.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:21.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:21.576890+0000) 2024-03-06T17:58:22.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:22.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:22.577170+0000) 2024-03-06T17:58:23.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:23.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:23.577440+0000) 2024-03-06T17:58:24.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:24.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:24.577649+0000) 2024-03-06T17:58:25.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:25.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:25.577870+0000) 2024-03-06T17:58:26.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:26.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:26.578126+0000) 2024-03-06T17:58:27.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:27.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:27.578355+0000) 2024-03-06T17:58:28.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:28.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:28.578581+0000) 2024-03-06T17:58:29.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:29.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:29.578860+0000) 2024-03-06T17:58:30.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:30.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:30.579177+0000) 2024-03-06T17:58:31.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:31.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:31.579459+0000) 2024-03-06T17:58:32.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:32.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:32.579737+0000) 2024-03-06T17:58:33.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:33.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:33.579939+0000) 2024-03-06T17:58:34.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:34.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:34.580183+0000) 2024-03-06T17:58:35.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:35.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:35.580401+0000) 2024-03-06T17:58:36.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:36.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:36.580572+0000) 2024-03-06T17:58:37.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:37.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:37.580828+0000) 2024-03-06T17:58:38.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:38.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:38.581107+0000) 2024-03-06T17:58:39.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:39.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:39.581292+0000) 2024-03-06T17:58:40.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:40.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:40.581518+0000) 2024-03-06T17:58:40.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:41.007 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:58:41.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:41.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:41.581734+0000) 2024-03-06T17:58:42.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:42.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:42.581976+0000) 2024-03-06T17:58:43.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:43.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:43.582204+0000) 2024-03-06T17:58:44.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:44.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:44.582432+0000) 2024-03-06T17:58:45.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:45.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:45.582674+0000) 2024-03-06T17:58:46.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:46.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:46.582877+0000) 2024-03-06T17:58:47.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:47.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:47.583132+0000) 2024-03-06T17:58:48.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:48.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:48.583336+0000) 2024-03-06T17:58:49.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:49.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:49.583627+0000) 2024-03-06T17:58:50.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:50.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:50.583842+0000) 2024-03-06T17:58:51.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:51.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:51.584073+0000) 2024-03-06T17:58:52.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:52.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:52.584308+0000) 2024-03-06T17:58:53.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:53.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:53.584536+0000) 2024-03-06T17:58:54.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:54.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:54.584761+0000) 2024-03-06T17:58:55.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:55.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:55.584981+0000) 2024-03-06T17:58:56.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:56.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:56.585235+0000) 2024-03-06T17:58:57.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:57.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:57.585470+0000) 2024-03-06T17:58:58.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:58.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:58.585710+0000) 2024-03-06T17:58:59.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:58:59.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:58:59.585955+0000) 2024-03-06T17:59:00.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:00.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:00.586241+0000) 2024-03-06T17:59:01.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:01.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:01.586441+0000) 2024-03-06T17:59:02.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:02.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:02.586637+0000) 2024-03-06T17:59:03.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:03.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:03.586881+0000) 2024-03-06T17:59:04.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:04.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:04.587116+0000) 2024-03-06T17:59:05.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:05.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:05.587308+0000) 2024-03-06T17:59:06.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:06.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:06.587529+0000) 2024-03-06T17:59:07.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:07.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:07.587760+0000) 2024-03-06T17:59:08.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:08.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:08.588057+0000) 2024-03-06T17:59:09.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:09.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:09.588343+0000) 2024-03-06T17:59:10.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:10.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:10.588614+0000) 2024-03-06T17:59:11.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:11.153 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:11.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:11.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:11.588906+0000) 2024-03-06T17:59:12.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:12.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:12.589194+0000) 2024-03-06T17:59:13.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:13.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:13.589443+0000) 2024-03-06T17:59:14.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:14.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:14.589685+0000) 2024-03-06T17:59:15.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:15.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:15.589921+0000) 2024-03-06T17:59:16.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:16.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:16.590164+0000) 2024-03-06T17:59:17.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:17.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:17.590395+0000) 2024-03-06T17:59:18.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:18.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:18.590626+0000) 2024-03-06T17:59:19.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:19.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:19.590844+0000) 2024-03-06T17:59:20.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:20.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:20.591113+0000) 2024-03-06T17:59:21.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:21.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:21.591336+0000) 2024-03-06T17:59:22.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:22.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:22.591577+0000) 2024-03-06T17:59:23.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:23.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:23.591780+0000) 2024-03-06T17:59:24.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:24.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:24.591970+0000) 2024-03-06T17:59:25.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:25.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:25.592201+0000) 2024-03-06T17:59:26.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:26.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:26.592437+0000) 2024-03-06T17:59:27.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:27.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:27.592677+0000) 2024-03-06T17:59:28.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:28.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:28.592902+0000) 2024-03-06T17:59:29.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:29.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:29.593118+0000) 2024-03-06T17:59:30.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:30.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:30.593284+0000) 2024-03-06T17:59:31.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:31.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:31.593504+0000) 2024-03-06T17:59:32.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:32.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:32.593788+0000) 2024-03-06T17:59:33.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:33.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:33.594039+0000) 2024-03-06T17:59:34.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:34.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:34.594234+0000) 2024-03-06T17:59:35.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:35.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:35.594443+0000) 2024-03-06T17:59:36.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:36.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:36.594691+0000) 2024-03-06T17:59:37.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:37.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:37.594925+0000) 2024-03-06T17:59:38.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:38.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:38.595148+0000) 2024-03-06T17:59:39.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:39.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:39.595380+0000) 2024-03-06T17:59:40.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:40.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:40.595633+0000) 2024-03-06T17:59:41.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:41.290 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T17:59:41.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:41.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:41.595798+0000) 2024-03-06T17:59:42.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:42.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:42.596036+0000) 2024-03-06T17:59:43.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:43.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:43.596249+0000) 2024-03-06T17:59:44.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:44.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:44.596443+0000) 2024-03-06T17:59:45.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:45.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:45.596620+0000) 2024-03-06T17:59:46.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:46.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:46.596827+0000) 2024-03-06T17:59:47.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:47.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:47.597085+0000) 2024-03-06T17:59:48.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:48.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:48.597319+0000) 2024-03-06T17:59:49.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:49.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:49.597563+0000) 2024-03-06T17:59:50.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:50.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:50.597744+0000) 2024-03-06T17:59:51.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:51.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:51.597964+0000) 2024-03-06T17:59:52.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:52.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:52.598200+0000) 2024-03-06T17:59:53.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:53.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:53.598455+0000) 2024-03-06T17:59:54.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:54.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:54.598704+0000) 2024-03-06T17:59:55.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:55.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:55.598945+0000) 2024-03-06T17:59:56.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:56.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:56.599172+0000) 2024-03-06T17:59:57.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:57.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:57.599408+0000) 2024-03-06T17:59:58.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:58.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:58.599700+0000) 2024-03-06T17:59:59.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T17:59:59.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T16:59:59.599985+0000) 2024-03-06T18:00:00.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:00.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:00.600301+0000) 2024-03-06T18:00:01.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:01.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:01.600575+0000) 2024-03-06T18:00:02.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:02.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:02.600840+0000) 2024-03-06T18:00:03.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:03.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:03.601158+0000) 2024-03-06T18:00:04.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:04.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:04.601408+0000) 2024-03-06T18:00:05.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:05.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:05.601649+0000) 2024-03-06T18:00:06.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:06.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:06.601862+0000) 2024-03-06T18:00:07.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:07.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:07.602094+0000) 2024-03-06T18:00:08.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:08.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:08.602331+0000) 2024-03-06T18:00:09.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:09.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:09.602515+0000) 2024-03-06T18:00:10.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:10.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:10.602695+0000) 2024-03-06T18:00:11.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:11.442 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:11.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:11.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:11.602940+0000) 2024-03-06T18:00:12.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:12.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:12.603175+0000) 2024-03-06T18:00:13.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:13.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:13.603394+0000) 2024-03-06T18:00:14.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:14.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:14.603651+0000) 2024-03-06T18:00:15.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:15.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:15.603894+0000) 2024-03-06T18:00:16.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:16.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:16.604141+0000) 2024-03-06T18:00:17.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:17.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:17.604372+0000) 2024-03-06T18:00:18.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:18.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:18.604611+0000) 2024-03-06T18:00:19.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:19.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:19.604875+0000) 2024-03-06T18:00:20.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:20.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:20.605129+0000) 2024-03-06T18:00:21.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:21.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:21.605353+0000) 2024-03-06T18:00:22.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:22.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:22.605576+0000) 2024-03-06T18:00:23.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:23.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:23.605834+0000) 2024-03-06T18:00:24.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:24.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:24.606134+0000) 2024-03-06T18:00:25.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:25.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:25.606458+0000) 2024-03-06T18:00:26.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:26.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:26.606686+0000) 2024-03-06T18:00:27.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:27.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:27.606934+0000) 2024-03-06T18:00:28.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:28.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:28.607170+0000) 2024-03-06T18:00:29.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:29.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:29.607410+0000) 2024-03-06T18:00:30.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:30.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:30.607622+0000) 2024-03-06T18:00:31.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:31.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:31.607778+0000) 2024-03-06T18:00:32.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:32.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:32.607949+0000) 2024-03-06T18:00:33.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:33.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:33.608183+0000) 2024-03-06T18:00:34.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:34.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:34.608413+0000) 2024-03-06T18:00:35.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:35.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:35.608636+0000) 2024-03-06T18:00:36.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:36.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:36.608856+0000) 2024-03-06T18:00:37.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:37.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:37.609124+0000) 2024-03-06T18:00:38.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:38.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:38.609347+0000) 2024-03-06T18:00:39.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:39.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:39.609570+0000) 2024-03-06T18:00:40.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:40.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:40.609799+0000) 2024-03-06T18:00:41.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:41.583 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:00:41.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:41.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:41.610032+0000) 2024-03-06T18:00:42.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:42.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:42.610234+0000) 2024-03-06T18:00:43.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:43.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:43.610418+0000) 2024-03-06T18:00:44.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:44.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:44.610679+0000) 2024-03-06T18:00:45.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:45.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:45.610962+0000) 2024-03-06T18:00:46.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:46.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:46.611250+0000) 2024-03-06T18:00:47.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:47.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:47.611489+0000) 2024-03-06T18:00:48.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:48.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:48.611733+0000) 2024-03-06T18:00:49.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:49.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:49.611966+0000) 2024-03-06T18:00:50.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:50.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:50.612264+0000) 2024-03-06T18:00:51.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:51.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:51.612541+0000) 2024-03-06T18:00:52.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:52.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:52.612814+0000) 2024-03-06T18:00:53.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:53.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:53.613129+0000) 2024-03-06T18:00:54.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:54.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:54.613409+0000) 2024-03-06T18:00:55.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:55.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:55.613613+0000) 2024-03-06T18:00:56.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:56.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:56.613832+0000) 2024-03-06T18:00:57.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:57.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:57.614145+0000) 2024-03-06T18:00:58.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:58.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:58.614422+0000) 2024-03-06T18:00:59.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:00:59.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:00:59.614713+0000) 2024-03-06T18:01:00.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:00.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:00.614919+0000) 2024-03-06T18:01:01.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:01.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:01.615172+0000) 2024-03-06T18:01:02.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:02.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:02.615423+0000) 2024-03-06T18:01:03.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:03.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:03.615717+0000) 2024-03-06T18:01:04.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:04.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:04.615937+0000) 2024-03-06T18:01:05.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:05.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:05.616165+0000) 2024-03-06T18:01:06.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:06.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:06.616413+0000) 2024-03-06T18:01:07.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:07.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:07.616636+0000) 2024-03-06T18:01:08.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:08.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:08.616862+0000) 2024-03-06T18:01:09.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:09.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:09.617115+0000) 2024-03-06T18:01:10.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:10.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:10.617333+0000) 2024-03-06T18:01:11.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:11.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:11.617550+0000) 2024-03-06T18:01:11.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:11.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:12.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:12.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:12.617780+0000) 2024-03-06T18:01:13.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:13.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:13.618044+0000) 2024-03-06T18:01:14.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:14.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:14.618283+0000) 2024-03-06T18:01:15.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:15.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:15.618498+0000) 2024-03-06T18:01:16.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:16.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:16.618725+0000) 2024-03-06T18:01:17.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:17.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:17.618973+0000) 2024-03-06T18:01:18.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:18.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:18.619221+0000) 2024-03-06T18:01:19.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:19.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:19.619429+0000) 2024-03-06T18:01:20.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:20.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:20.619666+0000) 2024-03-06T18:01:21.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:21.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:21.619892+0000) 2024-03-06T18:01:22.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:22.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:22.620115+0000) 2024-03-06T18:01:23.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:23.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:23.620359+0000) 2024-03-06T18:01:24.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:24.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:24.620585+0000) 2024-03-06T18:01:25.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:25.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:25.620811+0000) 2024-03-06T18:01:26.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:26.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:26.621055+0000) 2024-03-06T18:01:27.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:27.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:27.621286+0000) 2024-03-06T18:01:28.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:28.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:28.621529+0000) 2024-03-06T18:01:29.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:29.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:29.621753+0000) 2024-03-06T18:01:30.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:30.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:30.621972+0000) 2024-03-06T18:01:31.622 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:31.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:31.622210+0000) 2024-03-06T18:01:32.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:32.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:32.622453+0000) 2024-03-06T18:01:33.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:33.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:33.622730+0000) 2024-03-06T18:01:34.623 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:34.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:34.623044+0000) 2024-03-06T18:01:35.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:35.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:35.623328+0000) 2024-03-06T18:01:36.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:36.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:36.623563+0000) 2024-03-06T18:01:37.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:37.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:37.623815+0000) 2024-03-06T18:01:38.624 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:38.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:38.624109+0000) 2024-03-06T18:01:39.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:39.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:39.624302+0000) 2024-03-06T18:01:40.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:40.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:40.624526+0000) 2024-03-06T18:01:41.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:41.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:41.624704+0000) 2024-03-06T18:01:41.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:41.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:01:42.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:42.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:42.624926+0000) 2024-03-06T18:01:43.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:43.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:43.625147+0000) 2024-03-06T18:01:44.625 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:44.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:44.625365+0000) 2024-03-06T18:01:45.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:45.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:45.625570+0000) 2024-03-06T18:01:46.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:46.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:46.625792+0000) 2024-03-06T18:01:47.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:47.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:47.626047+0000) 2024-03-06T18:01:48.626 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:48.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:48.626297+0000) 2024-03-06T18:01:49.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:49.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:49.626531+0000) 2024-03-06T18:01:50.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:50.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:50.626762+0000) 2024-03-06T18:01:51.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:51.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:51.626991+0000) 2024-03-06T18:01:52.627 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:52.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:52.627253+0000) 2024-03-06T18:01:53.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:53.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:53.627465+0000) 2024-03-06T18:01:54.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:54.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:54.627683+0000) 2024-03-06T18:01:55.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:55.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:55.627938+0000) 2024-03-06T18:01:56.628 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:56.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:56.628210+0000) 2024-03-06T18:01:57.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:57.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:57.628481+0000) 2024-03-06T18:01:58.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:58.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:58.628752+0000) 2024-03-06T18:01:59.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:01:59.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:01:59.628992+0000) 2024-03-06T18:02:00.629 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:00.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:00.629248+0000) 2024-03-06T18:02:01.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:01.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:01.629467+0000) 2024-03-06T18:02:02.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:02.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:02.629695+0000) 2024-03-06T18:02:03.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:03.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:03.629935+0000) 2024-03-06T18:02:04.630 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:04.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:04.630173+0000) 2024-03-06T18:02:05.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:05.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:05.630435+0000) 2024-03-06T18:02:06.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:06.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:06.630708+0000) 2024-03-06T18:02:07.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:07.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:07.630932+0000) 2024-03-06T18:02:08.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:08.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:08.631181+0000) 2024-03-06T18:02:09.631 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:09.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:09.631406+0000) 2024-03-06T18:02:10.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:10.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:10.631666+0000) 2024-03-06T18:02:11.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:11.630+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:11.631904+0000) 2024-03-06T18:02:12.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:12.035 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:12.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:12.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:12.632189+0000) 2024-03-06T18:02:13.632 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:13.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:13.632438+0000) 2024-03-06T18:02:14.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:14.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:14.632703+0000) 2024-03-06T18:02:15.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:15.631+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:15.632977+0000) 2024-03-06T18:02:16.633 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:16.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:16.633279+0000) 2024-03-06T18:02:17.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:17.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:17.633528+0000) 2024-03-06T18:02:18.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:18.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:18.633748+0000) 2024-03-06T18:02:19.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:19.632+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:19.633973+0000) 2024-03-06T18:02:20.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:20.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:20.634225+0000) 2024-03-06T18:02:21.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:21.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:21.634449+0000) 2024-03-06T18:02:22.634 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:22.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:22.634674+0000) 2024-03-06T18:02:23.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:23.633+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:23.634884+0000) 2024-03-06T18:02:24.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:24.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:24.635109+0000) 2024-03-06T18:02:25.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:25.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:25.635336+0000) 2024-03-06T18:02:26.635 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:26.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:26.635564+0000) 2024-03-06T18:02:27.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:27.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:27.635768+0000) 2024-03-06T18:02:28.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:28.634+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:28.635956+0000) 2024-03-06T18:02:29.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:29.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:29.636157+0000) 2024-03-06T18:02:30.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:30.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:30.636364+0000) 2024-03-06T18:02:31.636 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:31.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:31.636536+0000) 2024-03-06T18:02:32.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:32.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:32.636749+0000) 2024-03-06T18:02:33.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:33.635+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:33.636969+0000) 2024-03-06T18:02:34.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:34.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:34.637199+0000) 2024-03-06T18:02:35.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:35.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:35.637418+0000) 2024-03-06T18:02:36.637 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:36.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:36.637638+0000) 2024-03-06T18:02:37.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:37.636+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:37.637868+0000) 2024-03-06T18:02:38.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:38.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:38.638122+0000) 2024-03-06T18:02:39.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:39.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:39.638346+0000) 2024-03-06T18:02:40.638 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:40.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:40.638563+0000) 2024-03-06T18:02:41.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:41.637+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:41.638814+0000) 2024-03-06T18:02:42.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:42.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:02:42.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:42.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:42.639086+0000) 2024-03-06T18:02:43.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:43.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:43.639320+0000) 2024-03-06T18:02:44.639 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:44.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:44.639557+0000) 2024-03-06T18:02:45.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:45.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:45.639743+0000) 2024-03-06T18:02:46.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:46.638+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:46.639972+0000) 2024-03-06T18:02:47.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:47.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:47.640223+0000) 2024-03-06T18:02:48.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:48.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:48.640479+0000) 2024-03-06T18:02:49.640 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:49.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:49.640701+0000) 2024-03-06T18:02:50.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:50.639+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:50.640875+0000) 2024-03-06T18:02:51.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:51.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:51.641097+0000) 2024-03-06T18:02:52.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:52.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:52.641322+0000) 2024-03-06T18:02:53.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:53.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:53.641508+0000) 2024-03-06T18:02:54.641 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:54.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:54.641692+0000) 2024-03-06T18:02:55.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:55.640+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:55.641934+0000) 2024-03-06T18:02:56.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:56.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:56.642173+0000) 2024-03-06T18:02:57.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:57.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:57.642428+0000) 2024-03-06T18:02:58.642 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:58.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:58.642655+0000) 2024-03-06T18:02:59.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:02:59.641+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:02:59.642868+0000) 2024-03-06T18:03:00.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:00.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:00.643175+0000) 2024-03-06T18:03:01.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:01.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:01.643396+0000) 2024-03-06T18:03:02.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:02.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:02.643621+0000) 2024-03-06T18:03:03.643 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:03.642+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:03.643835+0000) 2024-03-06T18:03:04.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:04.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:04.644103+0000) 2024-03-06T18:03:05.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:05.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:05.644331+0000) 2024-03-06T18:03:06.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:06.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:06.644549+0000) 2024-03-06T18:03:07.644 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:07.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:07.644784+0000) 2024-03-06T18:03:08.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:08.643+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:08.644986+0000) 2024-03-06T18:03:09.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:09.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:09.645234+0000) 2024-03-06T18:03:10.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:10.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:10.645452+0000) 2024-03-06T18:03:11.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:11.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:11.645634+0000) 2024-03-06T18:03:12.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:12.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:12.645 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:12.644+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:12.645842+0000) 2024-03-06T18:03:13.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:13.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:13.646072+0000) 2024-03-06T18:03:14.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:14.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:14.646270+0000) 2024-03-06T18:03:15.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:15.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:15.646488+0000) 2024-03-06T18:03:16.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:16.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:16.646715+0000) 2024-03-06T18:03:17.646 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:17.645+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:17.646944+0000) 2024-03-06T18:03:18.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:18.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:18.647164+0000) 2024-03-06T18:03:19.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:19.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:19.647385+0000) 2024-03-06T18:03:20.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:20.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:20.647605+0000) 2024-03-06T18:03:21.647 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:21.646+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:21.647829+0000) 2024-03-06T18:03:22.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:22.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:22.648091+0000) 2024-03-06T18:03:23.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:23.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:23.648276+0000) 2024-03-06T18:03:24.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:24.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:24.648460+0000) 2024-03-06T18:03:25.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:25.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:25.648686+0000) 2024-03-06T18:03:26.648 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:26.647+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:26.648910+0000) 2024-03-06T18:03:27.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:27.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:27.649138+0000) 2024-03-06T18:03:28.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:28.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:28.649322+0000) 2024-03-06T18:03:29.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:29.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:29.649549+0000) 2024-03-06T18:03:30.649 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:30.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:30.649748+0000) 2024-03-06T18:03:31.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:31.648+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:31.649946+0000) 2024-03-06T18:03:32.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:32.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:32.650202+0000) 2024-03-06T18:03:33.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:33.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:33.650437+0000) 2024-03-06T18:03:34.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:34.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:34.650691+0000) 2024-03-06T18:03:35.650 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:35.649+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:35.650883+0000) 2024-03-06T18:03:36.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:36.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:36.651140+0000) 2024-03-06T18:03:37.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:37.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:37.651345+0000) 2024-03-06T18:03:38.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:38.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:38.651546+0000) 2024-03-06T18:03:39.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:39.650+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:39.651767+0000) 2024-03-06T18:03:40.651 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:40.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:40.651997+0000) 2024-03-06T18:03:41.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:41.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:41.652273+0000) 2024-03-06T18:03:42.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:42.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:03:42.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:42.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:42.652531+0000) 2024-03-06T18:03:43.652 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:43.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:43.652793+0000) 2024-03-06T18:03:44.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:44.651+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:44.652986+0000) 2024-03-06T18:03:45.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:45.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:45.653243+0000) 2024-03-06T18:03:46.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:46.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:46.653461+0000) 2024-03-06T18:03:47.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:47.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:47.653675+0000) 2024-03-06T18:03:48.653 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:48.652+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:48.653895+0000) 2024-03-06T18:03:49.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:49.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:49.654169+0000) 2024-03-06T18:03:50.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:50.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:50.654423+0000) 2024-03-06T18:03:51.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:51.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:51.654661+0000) 2024-03-06T18:03:52.654 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:52.653+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:52.654927+0000) 2024-03-06T18:03:53.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:53.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:53.655152+0000) 2024-03-06T18:03:54.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:54.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:54.655398+0000) 2024-03-06T18:03:55.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:55.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:55.655632+0000) 2024-03-06T18:03:56.655 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:56.654+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:56.655890+0000) 2024-03-06T18:03:57.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:57.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:57.656152+0000) 2024-03-06T18:03:58.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:58.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:58.656326+0000) 2024-03-06T18:03:59.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:03:59.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:03:59.656562+0000) 2024-03-06T18:04:00.656 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:00.655+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:00.656788+0000) 2024-03-06T18:04:01.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:01.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:01.657041+0000) 2024-03-06T18:04:02.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:02.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:02.657297+0000) 2024-03-06T18:04:03.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:03.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:03.657517+0000) 2024-03-06T18:04:04.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:04.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:04.657741+0000) 2024-03-06T18:04:05.657 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:05.656+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:05.657961+0000) 2024-03-06T18:04:06.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:06.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:06.658165+0000) 2024-03-06T18:04:07.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:07.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:07.658447+0000) 2024-03-06T18:04:08.658 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:08.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:08.658727+0000) 2024-03-06T18:04:09.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:09.657+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:09.658973+0000) 2024-03-06T18:04:10.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:10.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:10.659203+0000) 2024-03-06T18:04:11.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:11.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:11.659433+0000) 2024-03-06T18:04:12.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:12.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:12.659652+0000) 2024-03-06T18:04:12.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:12.735 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:13.659 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:13.658+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:13.659916+0000) 2024-03-06T18:04:14.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:14.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:14.660204+0000) 2024-03-06T18:04:15.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:15.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:15.660489+0000) 2024-03-06T18:04:16.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:16.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:16.660709+0000) 2024-03-06T18:04:17.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:17.659+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:17.660954+0000) 2024-03-06T18:04:18.660 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:18.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:18.661202+0000) 2024-03-06T18:04:19.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:19.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:19.661478+0000) 2024-03-06T18:04:20.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:20.660+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:20.661744+0000) 2024-03-06T18:04:21.661 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:21.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:21.662060+0000) 2024-03-06T18:04:22.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:22.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:22.662305+0000) 2024-03-06T18:04:23.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:23.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:23.662597+0000) 2024-03-06T18:04:24.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:24.661+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:24.662880+0000) 2024-03-06T18:04:25.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:25.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:25.663119+0000) 2024-03-06T18:04:26.662 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:26.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:26.663319+0000) 2024-03-06T18:04:27.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:27.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:27.663545+0000) 2024-03-06T18:04:28.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:28.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:28.663726+0000) 2024-03-06T18:04:29.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:29.662+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:29.663958+0000) 2024-03-06T18:04:30.663 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:30.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:30.664180+0000) 2024-03-06T18:04:31.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:31.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:31.664459+0000) 2024-03-06T18:04:32.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:32.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:32.664682+0000) 2024-03-06T18:04:33.664 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:33.663+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:33.664912+0000) 2024-03-06T18:04:34.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:34.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:34.665146+0000) 2024-03-06T18:04:35.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:35.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:35.665369+0000) 2024-03-06T18:04:36.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:36.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:36.665592+0000) 2024-03-06T18:04:37.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:37.664+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:37.665825+0000) 2024-03-06T18:04:38.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:38.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:38.666071+0000) 2024-03-06T18:04:39.665 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:39.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:39.666328+0000) 2024-03-06T18:04:40.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:40.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:40.666523+0000) 2024-03-06T18:04:41.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:41.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:41.666763+0000) 2024-03-06T18:04:42.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:42.665+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:42.666989+0000) 2024-03-06T18:04:42.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:42.949 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:04:43.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:43.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:43.667218+0000) 2024-03-06T18:04:44.666 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:44.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:44.667463+0000) 2024-03-06T18:04:45.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:45.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:45.667686+0000) 2024-03-06T18:04:46.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:46.666+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:46.667866+0000) 2024-03-06T18:04:47.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:47.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:47.668128+0000) 2024-03-06T18:04:48.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:48.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:48.668342+0000) 2024-03-06T18:04:49.667 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:49.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:49.668579+0000) 2024-03-06T18:04:50.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:50.667+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:50.668793+0000) 2024-03-06T18:04:51.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:51.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:51.668994+0000) 2024-03-06T18:04:52.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:52.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:52.669208+0000) 2024-03-06T18:04:53.668 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:53.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:53.669423+0000) 2024-03-06T18:04:54.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:54.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:54.669649+0000) 2024-03-06T18:04:55.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:55.668+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:55.669872+0000) 2024-03-06T18:04:56.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:56.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:56.670067+0000) 2024-03-06T18:04:57.669 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:57.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:57.670295+0000) 2024-03-06T18:04:58.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:58.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:58.670515+0000) 2024-03-06T18:04:59.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:04:59.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:04:59.670752+0000) 2024-03-06T18:05:00.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:00.669+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:00.670965+0000) 2024-03-06T18:05:01.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:01.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:01.671267+0000) 2024-03-06T18:05:02.670 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:02.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:02.671446+0000) 2024-03-06T18:05:03.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:03.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:03.671665+0000) 2024-03-06T18:05:04.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:04.670+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:04.671932+0000) 2024-03-06T18:05:05.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:05.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:05.672159+0000) 2024-03-06T18:05:06.671 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:06.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:06.672399+0000) 2024-03-06T18:05:07.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:07.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:07.672630+0000) 2024-03-06T18:05:08.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:08.671+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:08.672836+0000) 2024-03-06T18:05:09.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:09.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:09.673131+0000) 2024-03-06T18:05:10.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:10.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:10.673351+0000) 2024-03-06T18:05:11.672 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:11.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:11.673579+0000) 2024-03-06T18:05:12.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:12.672+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:12.673798+0000) 2024-03-06T18:05:13.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:13.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:13.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:13.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:13.674053+0000) 2024-03-06T18:05:14.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:14.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:14.674289+0000) 2024-03-06T18:05:15.673 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:15.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:15.674518+0000) 2024-03-06T18:05:16.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:16.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:16.674711+0000) 2024-03-06T18:05:17.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:17.673+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:17.674923+0000) 2024-03-06T18:05:18.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:18.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:18.675134+0000) 2024-03-06T18:05:19.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:19.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:19.675351+0000) 2024-03-06T18:05:20.674 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:20.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:20.675530+0000) 2024-03-06T18:05:21.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:21.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:21.675758+0000) 2024-03-06T18:05:22.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:22.674+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:22.675936+0000) 2024-03-06T18:05:23.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:23.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:23.676160+0000) 2024-03-06T18:05:24.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:24.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:24.676395+0000) 2024-03-06T18:05:25.675 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:25.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:25.676620+0000) 2024-03-06T18:05:26.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:26.675+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:26.676817+0000) 2024-03-06T18:05:27.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:27.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:27.677083+0000) 2024-03-06T18:05:28.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:28.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:28.677275+0000) 2024-03-06T18:05:29.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:29.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:29.677531+0000) 2024-03-06T18:05:30.676 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:30.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:30.677754+0000) 2024-03-06T18:05:31.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:31.676+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:31.677955+0000) 2024-03-06T18:05:32.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:32.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:32.678177+0000) 2024-03-06T18:05:33.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:33.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:33.678389+0000) 2024-03-06T18:05:34.677 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:34.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:34.678618+0000) 2024-03-06T18:05:35.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:35.677+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:35.678839+0000) 2024-03-06T18:05:36.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:36.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:36.679102+0000) 2024-03-06T18:05:37.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:37.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:37.679326+0000) 2024-03-06T18:05:38.678 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:38.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:38.679536+0000) 2024-03-06T18:05:39.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:39.678+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:39.679779+0000) 2024-03-06T18:05:40.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:40.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:40.680059+0000) 2024-03-06T18:05:41.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:41.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:41.680266+0000) 2024-03-06T18:05:42.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:42.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:42.680489+0000) 2024-03-06T18:05:43.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:43.335 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:05:43.679 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:43.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:43.680729+0000) 2024-03-06T18:05:44.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:44.679+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:44.680933+0000) 2024-03-06T18:05:45.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:45.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:45.681138+0000) 2024-03-06T18:05:46.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:46.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:46.681314+0000) 2024-03-06T18:05:47.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:47.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:47.681545+0000) 2024-03-06T18:05:48.680 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:48.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:48.681750+0000) 2024-03-06T18:05:49.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:49.680+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:49.681954+0000) 2024-03-06T18:05:50.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:50.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:50.682169+0000) 2024-03-06T18:05:51.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:51.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:51.682403+0000) 2024-03-06T18:05:52.681 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:52.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:52.682643+0000) 2024-03-06T18:05:53.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:53.681+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:53.682861+0000) 2024-03-06T18:05:54.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:54.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:54.683140+0000) 2024-03-06T18:05:55.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:55.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:55.683330+0000) 2024-03-06T18:05:56.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:56.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:56.683485+0000) 2024-03-06T18:05:57.682 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:57.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:57.683696+0000) 2024-03-06T18:05:58.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:58.682+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:58.683934+0000) 2024-03-06T18:05:59.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:05:59.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:05:59.684190+0000) 2024-03-06T18:06:00.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:00.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:00.684372+0000) 2024-03-06T18:06:01.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:01.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:01.684607+0000) 2024-03-06T18:06:02.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:02.683+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:02.684841+0000) 2024-03-06T18:06:03.683 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:03.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:03.685054+0000) 2024-03-06T18:06:04.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:04.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:04.685311+0000) 2024-03-06T18:06:05.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:05.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:05.685555+0000) 2024-03-06T18:06:06.684 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:06.684+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:06.685763+0000) 2024-03-06T18:06:07.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:07.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:07.686085+0000) 2024-03-06T18:06:08.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:08.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:08.686326+0000) 2024-03-06T18:06:09.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:09.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:09.686590+0000) 2024-03-06T18:06:10.685 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:10.685+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:10.686823+0000) 2024-03-06T18:06:11.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:11.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:11.687111+0000) 2024-03-06T18:06:12.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:12.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:12.687317+0000) 2024-03-06T18:06:13.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:13.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:13.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:13.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:13.687543+0000) 2024-03-06T18:06:14.686 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:14.686+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:14.687793+0000) 2024-03-06T18:06:15.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:15.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:15.688087+0000) 2024-03-06T18:06:16.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:16.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:16.688313+0000) 2024-03-06T18:06:17.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:17.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:17.688615+0000) 2024-03-06T18:06:18.687 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:18.687+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:18.688799+0000) 2024-03-06T18:06:19.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:19.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:19.688999+0000) 2024-03-06T18:06:20.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:20.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:20.689295+0000) 2024-03-06T18:06:21.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:21.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:21.689550+0000) 2024-03-06T18:06:22.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:22.688+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:22.689786+0000) 2024-03-06T18:06:23.688 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:23.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:23.690043+0000) 2024-03-06T18:06:24.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:24.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:24.690300+0000) 2024-03-06T18:06:25.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:25.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:25.690550+0000) 2024-03-06T18:06:26.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:26.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:26.690771+0000) 2024-03-06T18:06:27.689 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:27.689+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:27.690949+0000) 2024-03-06T18:06:28.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:28.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:28.691179+0000) 2024-03-06T18:06:29.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:29.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:29.691438+0000) 2024-03-06T18:06:30.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:30.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:30.691671+0000) 2024-03-06T18:06:31.690 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:31.690+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:31.691897+0000) 2024-03-06T18:06:32.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:32.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:32.692107+0000) 2024-03-06T18:06:33.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:33.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:33.692322+0000) 2024-03-06T18:06:34.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:34.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:34.692526+0000) 2024-03-06T18:06:35.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:35.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:35.692706+0000) 2024-03-06T18:06:36.691 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:36.691+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:36.692893+0000) 2024-03-06T18:06:37.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:37.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:37.693119+0000) 2024-03-06T18:06:38.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:38.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:38.693333+0000) 2024-03-06T18:06:39.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:39.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:39.693580+0000) 2024-03-06T18:06:40.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:40.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:40.693802+0000) 2024-03-06T18:06:41.692 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:41.692+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:41.693989+0000) 2024-03-06T18:06:42.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:42.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:42.694260+0000) 2024-03-06T18:06:43.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:43.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:43.694491+0000) 2024-03-06T18:06:43.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:43.764 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:06:44.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:44.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:44.694730+0000) 2024-03-06T18:06:45.693 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:45.693+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:45.694953+0000) 2024-03-06T18:06:46.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:46.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:46.695180+0000) 2024-03-06T18:06:47.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:47.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:47.695378+0000) 2024-03-06T18:06:48.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:48.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:48.695595+0000) 2024-03-06T18:06:49.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:49.694+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:49.695782+0000) 2024-03-06T18:06:50.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:50.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:50.696033+0000) 2024-03-06T18:06:51.694 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:51.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:51.696178+0000) 2024-03-06T18:06:52.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:52.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:52.696371+0000) 2024-03-06T18:06:53.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:53.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:53.696586+0000) 2024-03-06T18:06:54.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:54.695+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:54.696853+0000) 2024-03-06T18:06:55.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:55.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:55.697063+0000) 2024-03-06T18:06:56.695 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:56.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:56.697292+0000) 2024-03-06T18:06:57.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:57.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:57.697496+0000) 2024-03-06T18:06:58.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:58.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:58.697714+0000) 2024-03-06T18:06:59.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:06:59.696+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:06:59.697955+0000) 2024-03-06T18:07:00.696 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:00.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:00.698129+0000) 2024-03-06T18:07:01.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:01.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:01.698317+0000) 2024-03-06T18:07:02.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:02.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:02.698529+0000) 2024-03-06T18:07:03.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:03.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:03.698731+0000) 2024-03-06T18:07:04.697 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:04.697+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:04.698980+0000) 2024-03-06T18:07:05.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:05.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:05.699251+0000) 2024-03-06T18:07:06.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:06.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:06.699488+0000) 2024-03-06T18:07:07.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:07.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:07.699719+0000) 2024-03-06T18:07:08.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:08.698+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:08.699868+0000) 2024-03-06T18:07:09.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:09.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:09.700131+0000) 2024-03-06T18:07:10.698 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:10.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:10.700282+0000) 2024-03-06T18:07:11.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:11.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:11.700574+0000) 2024-03-06T18:07:12.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:12.699+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:12.700809+0000) 2024-03-06T18:07:13.699 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:13.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:13.701075+0000) 2024-03-06T18:07:13.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:13.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:14.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:14.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:14.701345+0000) 2024-03-06T18:07:15.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:15.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:15.701542+0000) 2024-03-06T18:07:16.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:16.700+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:16.701790+0000) 2024-03-06T18:07:17.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:17.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:17.702073+0000) 2024-03-06T18:07:18.700 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:18.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:18.702316+0000) 2024-03-06T18:07:19.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:19.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:19.702578+0000) 2024-03-06T18:07:20.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:20.701+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:20.702818+0000) 2024-03-06T18:07:21.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:21.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:21.703090+0000) 2024-03-06T18:07:22.701 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:22.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:22.703300+0000) 2024-03-06T18:07:23.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:23.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:23.703482+0000) 2024-03-06T18:07:24.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:24.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:24.703683+0000) 2024-03-06T18:07:25.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:25.702+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:25.703899+0000) 2024-03-06T18:07:26.702 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:26.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:26.704151+0000) 2024-03-06T18:07:27.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:27.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:27.704398+0000) 2024-03-06T18:07:28.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:28.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:28.704632+0000) 2024-03-06T18:07:29.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:29.703+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:29.704904+0000) 2024-03-06T18:07:30.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:30.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:30.705139+0000) 2024-03-06T18:07:31.703 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:31.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:31.705358+0000) 2024-03-06T18:07:32.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:32.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:32.705535+0000) 2024-03-06T18:07:33.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:33.704+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:33.705754+0000) 2024-03-06T18:07:34.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:34.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:34.706040+0000) 2024-03-06T18:07:35.704 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:35.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:35.706270+0000) 2024-03-06T18:07:36.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:36.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:36.706498+0000) 2024-03-06T18:07:37.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:37.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:37.706733+0000) 2024-03-06T18:07:38.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:38.705+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:38.706948+0000) 2024-03-06T18:07:39.705 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:39.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:39.707195+0000) 2024-03-06T18:07:40.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:40.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:40.707427+0000) 2024-03-06T18:07:41.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:41.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:41.707656+0000) 2024-03-06T18:07:42.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:42.706+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:42.707881+0000) 2024-03-06T18:07:43.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:43.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:43.708132+0000) 2024-03-06T18:07:44.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:44.114 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:07:44.706 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:44.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:44.708382+0000) 2024-03-06T18:07:45.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:45.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:45.708566+0000) 2024-03-06T18:07:46.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:46.707+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:46.708772+0000) 2024-03-06T18:07:47.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:47.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:47.709041+0000) 2024-03-06T18:07:48.707 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:48.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:48.709287+0000) 2024-03-06T18:07:49.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:49.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:49.709540+0000) 2024-03-06T18:07:50.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:50.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:50.709769+0000) 2024-03-06T18:07:51.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:51.708+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:51.709947+0000) 2024-03-06T18:07:52.708 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:52.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:52.710183+0000) 2024-03-06T18:07:53.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:53.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:53.710481+0000) 2024-03-06T18:07:54.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:54.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:54.710736+0000) 2024-03-06T18:07:55.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:55.709+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:55.710957+0000) 2024-03-06T18:07:56.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:56.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:56.711186+0000) 2024-03-06T18:07:57.709 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:57.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:57.711387+0000) 2024-03-06T18:07:58.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:58.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:58.711626+0000) 2024-03-06T18:07:59.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:07:59.710+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:07:59.711893+0000) 2024-03-06T18:08:00.710 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:00.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:00.712148+0000) 2024-03-06T18:08:01.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:01.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:01.712389+0000) 2024-03-06T18:08:02.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:02.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:02.712633+0000) 2024-03-06T18:08:03.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:03.711+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:03.712877+0000) 2024-03-06T18:08:04.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:04.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:04.713145+0000) 2024-03-06T18:08:05.711 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:05.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:05.713371+0000) 2024-03-06T18:08:06.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:06.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:06.713619+0000) 2024-03-06T18:08:07.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:07.712+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:07.713867+0000) 2024-03-06T18:08:08.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:08.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:08.714130+0000) 2024-03-06T18:08:09.712 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:09.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:09.714404+0000) 2024-03-06T18:08:10.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:10.713+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:10.714667+0000) 2024-03-06T18:08:11.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:11.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:11.715113+0000) 2024-03-06T18:08:12.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:12.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:12.715336+0000) 2024-03-06T18:08:13.713 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:13.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:13.715555+0000) 2024-03-06T18:08:14.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:14.318 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:14.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:14.714+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:14.715794+0000) 2024-03-06T18:08:15.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:15.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:15.716000+0000) 2024-03-06T18:08:16.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:16.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:16.716244+0000) 2024-03-06T18:08:17.714 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:17.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:17.716484+0000) 2024-03-06T18:08:18.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:18.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:18.716700+0000) 2024-03-06T18:08:19.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:19.715+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:19.716931+0000) 2024-03-06T18:08:20.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:20.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:20.717163+0000) 2024-03-06T18:08:21.715 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:21.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:21.717432+0000) 2024-03-06T18:08:22.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:22.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:22.717676+0000) 2024-03-06T18:08:23.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:23.716+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:23.717875+0000) 2024-03-06T18:08:24.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:24.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:24.718138+0000) 2024-03-06T18:08:25.716 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:25.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:25.718376+0000) 2024-03-06T18:08:26.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:26.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:26.718612+0000) 2024-03-06T18:08:27.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:27.717+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:27.718842+0000) 2024-03-06T18:08:28.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:28.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:28.719126+0000) 2024-03-06T18:08:29.717 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:29.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:29.719337+0000) 2024-03-06T18:08:30.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:30.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:30.719621+0000) 2024-03-06T18:08:31.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:31.718+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:31.719869+0000) 2024-03-06T18:08:32.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:32.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:32.720149+0000) 2024-03-06T18:08:33.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:33.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:33.720341+0000) 2024-03-06T18:08:34.718 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:34.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:34.720535+0000) 2024-03-06T18:08:35.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:35.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:35.720762+0000) 2024-03-06T18:08:36.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:36.719+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:36.720962+0000) 2024-03-06T18:08:37.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:37.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:37.721265+0000) 2024-03-06T18:08:38.719 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:38.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:38.721506+0000) 2024-03-06T18:08:39.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:39.720+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:39.721747+0000) 2024-03-06T18:08:40.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:40.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:40.721998+0000) 2024-03-06T18:08:41.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:41.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:41.722272+0000) 2024-03-06T18:08:42.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:42.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:42.722517+0000) 2024-03-06T18:08:43.720 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:43.721+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:43.722728+0000) 2024-03-06T18:08:44.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:44.531 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:08:44.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:44.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:44.722991+0000) 2024-03-06T18:08:45.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:45.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:45.723253+0000) 2024-03-06T18:08:46.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:46.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:46.723482+0000) 2024-03-06T18:08:47.721 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:47.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:47.723718+0000) 2024-03-06T18:08:48.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:48.722+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:48.723929+0000) 2024-03-06T18:08:49.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:49.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:49.724116+0000) 2024-03-06T18:08:50.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:50.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:50.724353+0000) 2024-03-06T18:08:51.722 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:51.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:51.724577+0000) 2024-03-06T18:08:52.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:52.723+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:52.724763+0000) 2024-03-06T18:08:53.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:53.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:53.725038+0000) 2024-03-06T18:08:54.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:54.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:54.725296+0000) 2024-03-06T18:08:55.723 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:55.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:55.725546+0000) 2024-03-06T18:08:56.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:56.724+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:56.725788+0000) 2024-03-06T18:08:57.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:57.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:57.726123+0000) 2024-03-06T18:08:58.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:58.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:58.726339+0000) 2024-03-06T18:08:59.724 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:08:59.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:08:59.726585+0000) 2024-03-06T18:09:00.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:00.725+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:00.726834+0000) 2024-03-06T18:09:01.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:01.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:01.727138+0000) 2024-03-06T18:09:02.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:02.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:02.727356+0000) 2024-03-06T18:09:03.725 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:03.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:03.727595+0000) 2024-03-06T18:09:04.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:04.726+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:04.727837+0000) 2024-03-06T18:09:05.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:05.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:05.728127+0000) 2024-03-06T18:09:06.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:06.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:06.728344+0000) 2024-03-06T18:09:07.726 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:07.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:07.728560+0000) 2024-03-06T18:09:08.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:08.727+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:08.728783+0000) 2024-03-06T18:09:09.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:09.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:09.729033+0000) 2024-03-06T18:09:10.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:10.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:10.729274+0000) 2024-03-06T18:09:11.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:11.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:11.729479+0000) 2024-03-06T18:09:12.727 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:12.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:12.729707+0000) 2024-03-06T18:09:13.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:13.728+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:13.729884+0000) 2024-03-06T18:09:14.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:14.676 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:14.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:14.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:14.730117+0000) 2024-03-06T18:09:15.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:15.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:15.730354+0000) 2024-03-06T18:09:16.728 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:16.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:16.730616+0000) 2024-03-06T18:09:17.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:17.729+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:17.730829+0000) 2024-03-06T18:09:18.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:18.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:18.731095+0000) 2024-03-06T18:09:19.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:19.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:19.731322+0000) 2024-03-06T18:09:20.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:20.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:20.731576+0000) 2024-03-06T18:09:21.729 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:21.730+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:21.731852+0000) 2024-03-06T18:09:22.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:22.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:22.732128+0000) 2024-03-06T18:09:23.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:23.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:23.732347+0000) 2024-03-06T18:09:24.730 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:24.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:24.732582+0000) 2024-03-06T18:09:25.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:25.731+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:25.732806+0000) 2024-03-06T18:09:26.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:26.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:26.733121+0000) 2024-03-06T18:09:27.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:27.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:27.733371+0000) 2024-03-06T18:09:28.731 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:28.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:28.733646+0000) 2024-03-06T18:09:29.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:29.732+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:29.733918+0000) 2024-03-06T18:09:30.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:30.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:30.734209+0000) 2024-03-06T18:09:31.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:31.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:31.734454+0000) 2024-03-06T18:09:32.732 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:32.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:32.734704+0000) 2024-03-06T18:09:33.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:33.733+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:33.734913+0000) 2024-03-06T18:09:34.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:34.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:34.735150+0000) 2024-03-06T18:09:35.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:35.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:35.735352+0000) 2024-03-06T18:09:36.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:36.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:36.735584+0000) 2024-03-06T18:09:37.733 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:37.734+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:37.735821+0000) 2024-03-06T18:09:38.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:38.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:38.736056+0000) 2024-03-06T18:09:39.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:39.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:39.736320+0000) 2024-03-06T18:09:40.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:40.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:40.736626+0000) 2024-03-06T18:09:41.734 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:41.735+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:41.736842+0000) 2024-03-06T18:09:42.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:42.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:42.737157+0000) 2024-03-06T18:09:43.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:43.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:43.737392+0000) 2024-03-06T18:09:44.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:44.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:44.737672+0000) 2024-03-06T18:09:44.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:44.906 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:09:45.735 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:45.736+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:45.737963+0000) 2024-03-06T18:09:46.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:46.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:46.738267+0000) 2024-03-06T18:09:47.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:47.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:47.738525+0000) 2024-03-06T18:09:48.736 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:48.737+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:48.738824+0000) 2024-03-06T18:09:49.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:49.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:49.739075+0000) 2024-03-06T18:09:50.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:50.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:50.739243+0000) 2024-03-06T18:09:51.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:51.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:51.739504+0000) 2024-03-06T18:09:52.737 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:52.738+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:52.739786+0000) 2024-03-06T18:09:53.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:53.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:53.740093+0000) 2024-03-06T18:09:54.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:54.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:54.740343+0000) 2024-03-06T18:09:55.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:55.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:55.740583+0000) 2024-03-06T18:09:56.738 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:56.739+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:56.740812+0000) 2024-03-06T18:09:57.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:57.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:57.741080+0000) 2024-03-06T18:09:58.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:58.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:58.741320+0000) 2024-03-06T18:09:59.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:09:59.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:09:59.741523+0000) 2024-03-06T18:10:00.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:00.740+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:00.741765+0000) 2024-03-06T18:10:01.739 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:01.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:01.741993+0000) 2024-03-06T18:10:02.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:02.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:02.742269+0000) 2024-03-06T18:10:03.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:03.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:03.742471+0000) 2024-03-06T18:10:04.740 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:04.741+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:04.742745+0000) 2024-03-06T18:10:05.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:05.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:05.743074+0000) 2024-03-06T18:10:06.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:06.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:06.743336+0000) 2024-03-06T18:10:07.741 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:07.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:07.743649+0000) 2024-03-06T18:10:08.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:08.742+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:08.743923+0000) 2024-03-06T18:10:09.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:09.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:09.744161+0000) 2024-03-06T18:10:10.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:10.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:10.744402+0000) 2024-03-06T18:10:11.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:11.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:11.744633+0000) 2024-03-06T18:10:12.742 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:12.743+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:12.744889+0000) 2024-03-06T18:10:13.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:13.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:13.745140+0000) 2024-03-06T18:10:14.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:14.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:14.745347+0000) 2024-03-06T18:10:15.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:15.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:15.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:15.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:15.745579+0000) 2024-03-06T18:10:16.743 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:16.744+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:16.745809+0000) 2024-03-06T18:10:17.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:17.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:17.746092+0000) 2024-03-06T18:10:18.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:18.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:18.746275+0000) 2024-03-06T18:10:19.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:19.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:19.746498+0000) 2024-03-06T18:10:20.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:20.745+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:20.746743+0000) 2024-03-06T18:10:21.744 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:21.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:21.746988+0000) 2024-03-06T18:10:22.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:22.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:22.747286+0000) 2024-03-06T18:10:23.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:23.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:23.747518+0000) 2024-03-06T18:10:24.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:24.746+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:24.747764+0000) 2024-03-06T18:10:25.745 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:25.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:25.748036+0000) 2024-03-06T18:10:26.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:26.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:26.748267+0000) 2024-03-06T18:10:27.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:27.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:27.748504+0000) 2024-03-06T18:10:28.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:28.747+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:28.748790+0000) 2024-03-06T18:10:29.746 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:29.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:29.749040+0000) 2024-03-06T18:10:30.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:30.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:30.749252+0000) 2024-03-06T18:10:31.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:31.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:31.749471+0000) 2024-03-06T18:10:32.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:32.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:32.749699+0000) 2024-03-06T18:10:33.747 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:33.748+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:33.749928+0000) 2024-03-06T18:10:34.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:34.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:34.750205+0000) 2024-03-06T18:10:35.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:35.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:35.750481+0000) 2024-03-06T18:10:36.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:36.749+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:36.750740+0000) 2024-03-06T18:10:37.748 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:37.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:37.750991+0000) 2024-03-06T18:10:38.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:38.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:38.751287+0000) 2024-03-06T18:10:39.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:39.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:39.751511+0000) 2024-03-06T18:10:40.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:40.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:40.751756+0000) 2024-03-06T18:10:41.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:41.750+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:41.751957+0000) 2024-03-06T18:10:42.749 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:42.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:42.752185+0000) 2024-03-06T18:10:43.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:43.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:43.752406+0000) 2024-03-06T18:10:44.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:44.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:44.752635+0000) 2024-03-06T18:10:45.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:45.348 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:10:45.750 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:45.751+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:45.752872+0000) 2024-03-06T18:10:46.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:46.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:46.753136+0000) 2024-03-06T18:10:47.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:47.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:47.753353+0000) 2024-03-06T18:10:48.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:48.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:48.753606+0000) 2024-03-06T18:10:49.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:49.752+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:49.753851+0000) 2024-03-06T18:10:50.751 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:50.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:50.754121+0000) 2024-03-06T18:10:51.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:51.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:51.754285+0000) 2024-03-06T18:10:52.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:52.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:52.754512+0000) 2024-03-06T18:10:53.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:53.753+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:53.754753+0000) 2024-03-06T18:10:54.752 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:54.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:54.755000+0000) 2024-03-06T18:10:55.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:55.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:55.755289+0000) 2024-03-06T18:10:56.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:56.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:56.755522+0000) 2024-03-06T18:10:57.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:57.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:57.755758+0000) 2024-03-06T18:10:58.753 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:58.754+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:58.755977+0000) 2024-03-06T18:10:59.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:10:59.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:10:59.756202+0000) 2024-03-06T18:11:00.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:00.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:00.756439+0000) 2024-03-06T18:11:01.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:01.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:01.756683+0000) 2024-03-06T18:11:02.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:02.755+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:02.756880+0000) 2024-03-06T18:11:03.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:03.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:03.757106+0000) 2024-03-06T18:11:04.754 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:04.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:04.757292+0000) 2024-03-06T18:11:05.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:05.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:05.757516+0000) 2024-03-06T18:11:06.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:06.756+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:06.757751+0000) 2024-03-06T18:11:07.755 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:07.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:07.758045+0000) 2024-03-06T18:11:08.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:08.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:08.758254+0000) 2024-03-06T18:11:09.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:09.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:09.758493+0000) 2024-03-06T18:11:10.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:10.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:10.758732+0000) 2024-03-06T18:11:11.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:11.757+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:11.758990+0000) 2024-03-06T18:11:12.756 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:12.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:12.759232+0000) 2024-03-06T18:11:13.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:13.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:13.759451+0000) 2024-03-06T18:11:14.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:14.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:14.759672+0000) 2024-03-06T18:11:15.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:15.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:15.757 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:15.758+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:15.759936+0000) 2024-03-06T18:11:16.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:16.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:16.760243+0000) 2024-03-06T18:11:17.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:17.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:17.760454+0000) 2024-03-06T18:11:18.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:18.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:18.760680+0000) 2024-03-06T18:11:19.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:19.759+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:19.760913+0000) 2024-03-06T18:11:20.758 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:20.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:20.761165+0000) 2024-03-06T18:11:21.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:21.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:21.761387+0000) 2024-03-06T18:11:22.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:22.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:22.761556+0000) 2024-03-06T18:11:23.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:23.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:23.761733+0000) 2024-03-06T18:11:24.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:24.760+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:24.761965+0000) 2024-03-06T18:11:25.759 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:25.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:25.762230+0000) 2024-03-06T18:11:26.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:26.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:26.762491+0000) 2024-03-06T18:11:27.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:27.761+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:27.762775+0000) 2024-03-06T18:11:28.760 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:28.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:28.763103+0000) 2024-03-06T18:11:29.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:29.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:29.763345+0000) 2024-03-06T18:11:30.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:30.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:30.763631+0000) 2024-03-06T18:11:31.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:31.762+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:31.763917+0000) 2024-03-06T18:11:32.761 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:32.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:32.764198+0000) 2024-03-06T18:11:33.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:33.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:33.764414+0000) 2024-03-06T18:11:34.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:34.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:34.764641+0000) 2024-03-06T18:11:35.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:35.763+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:35.764908+0000) 2024-03-06T18:11:36.762 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:36.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:36.765158+0000) 2024-03-06T18:11:37.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:37.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:37.765433+0000) 2024-03-06T18:11:38.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:38.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:38.765716+0000) 2024-03-06T18:11:39.763 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:39.764+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:39.765981+0000) 2024-03-06T18:11:40.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:40.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:40.766299+0000) 2024-03-06T18:11:41.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:41.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:41.766539+0000) 2024-03-06T18:11:42.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:42.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:42.766719+0000) 2024-03-06T18:11:43.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:43.765+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:43.766959+0000) 2024-03-06T18:11:44.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:44.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:44.767193+0000) 2024-03-06T18:11:45.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:45.729 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:11:45.764 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:45.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:45.767418+0000) 2024-03-06T18:11:46.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:46.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:46.767646+0000) 2024-03-06T18:11:47.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:47.766+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:47.767880+0000) 2024-03-06T18:11:48.765 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:48.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:48.768146+0000) 2024-03-06T18:11:49.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:49.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:49.768396+0000) 2024-03-06T18:11:50.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:50.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:50.768640+0000) 2024-03-06T18:11:51.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:51.767+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:51.768886+0000) 2024-03-06T18:11:52.766 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:52.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:52.769115+0000) 2024-03-06T18:11:53.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:53.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:53.769358+0000) 2024-03-06T18:11:54.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:54.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:54.769602+0000) 2024-03-06T18:11:55.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:55.768+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:55.769835+0000) 2024-03-06T18:11:56.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:56.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:56.770133+0000) 2024-03-06T18:11:57.767 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:57.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:57.770355+0000) 2024-03-06T18:11:58.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:58.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:58.770586+0000) 2024-03-06T18:11:59.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:11:59.769+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:11:59.770822+0000) 2024-03-06T18:12:00.768 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:00.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:00.771132+0000) 2024-03-06T18:12:01.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:01.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:01.771373+0000) 2024-03-06T18:12:02.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:02.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:02.771616+0000) 2024-03-06T18:12:03.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:03.770+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:03.771855+0000) 2024-03-06T18:12:04.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:04.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:04.772142+0000) 2024-03-06T18:12:05.769 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:05.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:05.772404+0000) 2024-03-06T18:12:06.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:06.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:06.772656+0000) 2024-03-06T18:12:07.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:07.771+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:07.772905+0000) 2024-03-06T18:12:08.770 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:08.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:08.773115+0000) 2024-03-06T18:12:09.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:09.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:09.773421+0000) 2024-03-06T18:12:10.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:10.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:10.773714+0000) 2024-03-06T18:12:11.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:11.772+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:11.773986+0000) 2024-03-06T18:12:12.771 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:12.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:12.774273+0000) 2024-03-06T18:12:13.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:13.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:13.774552+0000) 2024-03-06T18:12:14.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:14.773+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:14.774833+0000) 2024-03-06T18:12:15.772 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:15.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:15.775128+0000) 2024-03-06T18:12:15.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:15.911 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:16.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:16.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:16.775336+0000) 2024-03-06T18:12:17.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:17.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:17.775552+0000) 2024-03-06T18:12:18.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:18.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:18.775773+0000) 2024-03-06T18:12:19.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:19.774+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:19.775992+0000) 2024-03-06T18:12:20.773 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:20.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:20.776277+0000) 2024-03-06T18:12:21.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:21.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:21.776495+0000) 2024-03-06T18:12:22.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:22.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:22.776738+0000) 2024-03-06T18:12:23.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:23.775+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:23.776954+0000) 2024-03-06T18:12:24.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:24.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:24.777157+0000) 2024-03-06T18:12:25.774 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:25.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:25.777394+0000) 2024-03-06T18:12:26.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:26.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:26.777619+0000) 2024-03-06T18:12:27.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:27.776+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:27.777879+0000) 2024-03-06T18:12:28.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:28.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:28.778144+0000) 2024-03-06T18:12:29.775 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:29.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:29.778362+0000) 2024-03-06T18:12:30.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:30.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:30.778628+0000) 2024-03-06T18:12:31.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:31.777+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:31.778872+0000) 2024-03-06T18:12:32.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:32.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:32.779096+0000) 2024-03-06T18:12:33.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:33.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:33.779286+0000) 2024-03-06T18:12:34.776 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:34.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:34.779505+0000) 2024-03-06T18:12:35.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:35.778+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:35.779751+0000) 2024-03-06T18:12:36.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:36.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:36.780085+0000) 2024-03-06T18:12:37.777 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:37.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:37.780374+0000) 2024-03-06T18:12:38.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:38.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:38.780623+0000) 2024-03-06T18:12:39.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:39.779+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:39.780808+0000) 2024-03-06T18:12:40.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:40.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:40.781043+0000) 2024-03-06T18:12:41.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:41.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:41.781260+0000) 2024-03-06T18:12:42.778 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:42.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:42.781502+0000) 2024-03-06T18:12:43.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:43.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:43.781733+0000) 2024-03-06T18:12:44.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:44.780+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:44.781976+0000) 2024-03-06T18:12:45.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:45.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:45.782263+0000) 2024-03-06T18:12:46.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:46.139 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:12:46.779 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:46.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:46.782517+0000) 2024-03-06T18:12:47.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:47.781+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:47.782773+0000) 2024-03-06T18:12:48.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:48.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:48.783045+0000) 2024-03-06T18:12:49.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:49.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:49.783257+0000) 2024-03-06T18:12:50.780 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:50.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:50.783511+0000) 2024-03-06T18:12:51.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:51.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:51.783748+0000) 2024-03-06T18:12:52.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:52.782+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:52.783986+0000) 2024-03-06T18:12:53.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:53.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:53.784215+0000) 2024-03-06T18:12:54.781 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:54.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:54.784440+0000) 2024-03-06T18:12:55.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:55.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:55.784674+0000) 2024-03-06T18:12:56.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:56.783+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:56.784904+0000) 2024-03-06T18:12:57.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:57.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:57.785214+0000) 2024-03-06T18:12:58.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:58.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:58.785438+0000) 2024-03-06T18:12:59.782 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:12:59.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:12:59.785677+0000) 2024-03-06T18:13:00.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:00.784+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:00.785935+0000) 2024-03-06T18:13:01.783 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:01.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:01.786165+0000) 2024-03-06T18:13:02.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:02.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:02.786417+0000) 2024-03-06T18:13:03.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:03.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:03.786629+0000) 2024-03-06T18:13:04.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:04.785+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:04.786820+0000) 2024-03-06T18:13:05.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:05.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:05.787139+0000) 2024-03-06T18:13:06.784 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:06.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:06.787413+0000) 2024-03-06T18:13:07.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:07.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:07.787635+0000) 2024-03-06T18:13:08.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:08.786+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:08.787852+0000) 2024-03-06T18:13:09.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:09.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:09.788142+0000) 2024-03-06T18:13:10.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:10.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:10.788363+0000) 2024-03-06T18:13:11.785 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:11.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:11.788600+0000) 2024-03-06T18:13:12.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:12.787+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:12.788843+0000) 2024-03-06T18:13:13.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:13.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:13.789112+0000) 2024-03-06T18:13:14.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:14.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:14.789363+0000) 2024-03-06T18:13:15.786 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:15.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:15.789650+0000) 2024-03-06T18:13:16.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:16.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:16.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:16.788+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:16.789938+0000) 2024-03-06T18:13:17.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:17.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:17.790181+0000) 2024-03-06T18:13:18.787 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:18.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:18.790386+0000) 2024-03-06T18:13:19.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:19.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:19.790631+0000) 2024-03-06T18:13:20.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:20.789+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:20.790895+0000) 2024-03-06T18:13:21.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:21.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:21.791170+0000) 2024-03-06T18:13:22.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:22.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:22.791418+0000) 2024-03-06T18:13:23.788 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:23.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:23.791638+0000) 2024-03-06T18:13:24.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:24.790+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:24.791826+0000) 2024-03-06T18:13:25.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:25.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:25.792118+0000) 2024-03-06T18:13:26.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:26.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:26.792313+0000) 2024-03-06T18:13:27.789 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:27.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:27.792552+0000) 2024-03-06T18:13:28.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:28.791+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:28.792772+0000) 2024-03-06T18:13:29.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:29.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:29.792995+0000) 2024-03-06T18:13:30.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:30.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:30.793259+0000) 2024-03-06T18:13:31.790 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:31.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:31.793504+0000) 2024-03-06T18:13:32.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:32.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:32.793729+0000) 2024-03-06T18:13:33.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:33.792+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:33.793923+0000) 2024-03-06T18:13:34.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:34.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:34.794152+0000) 2024-03-06T18:13:35.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:35.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:35.794433+0000) 2024-03-06T18:13:36.791 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:36.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:36.794664+0000) 2024-03-06T18:13:37.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:37.793+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:37.794944+0000) 2024-03-06T18:13:38.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:38.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:38.795162+0000) 2024-03-06T18:13:39.792 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:39.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:39.795381+0000) 2024-03-06T18:13:40.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:40.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:40.795655+0000) 2024-03-06T18:13:41.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:41.794+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:41.795914+0000) 2024-03-06T18:13:42.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:42.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:42.796137+0000) 2024-03-06T18:13:43.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:43.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:43.796357+0000) 2024-03-06T18:13:44.793 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:44.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:44.796576+0000) 2024-03-06T18:13:45.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:45.795+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:45.796867+0000) 2024-03-06T18:13:46.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:46.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:13:46.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:46.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:46.797118+0000) 2024-03-06T18:13:47.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:47.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:47.797338+0000) 2024-03-06T18:13:48.794 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:48.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:48.797555+0000) 2024-03-06T18:13:49.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:49.796+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:49.797822+0000) 2024-03-06T18:13:50.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:50.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:50.798156+0000) 2024-03-06T18:13:51.795 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:51.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:51.798427+0000) 2024-03-06T18:13:52.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:52.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:52.798711+0000) 2024-03-06T18:13:53.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:53.797+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:53.798988+0000) 2024-03-06T18:13:54.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:54.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:54.799312+0000) 2024-03-06T18:13:55.796 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:55.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:55.799597+0000) 2024-03-06T18:13:56.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:56.798+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:56.799880+0000) 2024-03-06T18:13:57.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:57.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:57.800159+0000) 2024-03-06T18:13:58.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:58.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:58.800438+0000) 2024-03-06T18:13:59.797 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:13:59.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:13:59.800681+0000) 2024-03-06T18:14:00.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:00.799+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:00.800967+0000) 2024-03-06T18:14:01.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:01.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:01.801272+0000) 2024-03-06T18:14:02.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:02.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:02.801479+0000) 2024-03-06T18:14:03.798 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:03.800+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:03.801744+0000) 2024-03-06T18:14:04.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:04.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:04.802061+0000) 2024-03-06T18:14:05.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:05.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:05.802345+0000) 2024-03-06T18:14:06.799 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:06.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:06.802622+0000) 2024-03-06T18:14:07.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:07.801+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:07.802903+0000) 2024-03-06T18:14:08.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:08.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:08.803189+0000) 2024-03-06T18:14:09.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:09.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:09.803427+0000) 2024-03-06T18:14:10.800 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:10.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:10.803690+0000) 2024-03-06T18:14:11.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:11.802+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:11.803943+0000) 2024-03-06T18:14:12.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:12.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:12.804222+0000) 2024-03-06T18:14:13.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:13.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:13.804495+0000) 2024-03-06T18:14:14.801 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:14.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:14.804694+0000) 2024-03-06T18:14:15.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:15.803+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:15.804919+0000) 2024-03-06T18:14:16.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:16.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:16.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:16.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:16.805172+0000) 2024-03-06T18:14:17.802 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:17.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:17.805465+0000) 2024-03-06T18:14:18.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:18.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:18.805710+0000) 2024-03-06T18:14:19.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:19.804+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:19.805974+0000) 2024-03-06T18:14:20.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:20.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:20.806303+0000) 2024-03-06T18:14:21.803 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:21.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:21.806569+0000) 2024-03-06T18:14:22.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:22.805+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:22.806842+0000) 2024-03-06T18:14:23.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:23.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:23.807129+0000) 2024-03-06T18:14:24.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:24.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:24.807373+0000) 2024-03-06T18:14:25.804 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:25.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:25.807521+0000) 2024-03-06T18:14:26.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:26.806+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:26.807769+0000) 2024-03-06T18:14:27.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:27.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:27.808061+0000) 2024-03-06T18:14:28.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:28.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:28.808288+0000) 2024-03-06T18:14:29.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:29.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:29.808493+0000) 2024-03-06T18:14:30.805 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:30.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:30.808689+0000) 2024-03-06T18:14:31.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:31.807+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:31.808928+0000) 2024-03-06T18:14:32.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:32.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:32.809190+0000) 2024-03-06T18:14:33.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:33.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:33.809404+0000) 2024-03-06T18:14:34.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:34.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:34.809587+0000) 2024-03-06T18:14:35.806 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:35.808+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:35.809793+0000) 2024-03-06T18:14:36.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:36.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:36.810063+0000) 2024-03-06T18:14:37.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:37.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:37.810270+0000) 2024-03-06T18:14:38.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:38.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:38.810469+0000) 2024-03-06T18:14:39.807 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:39.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:39.810696+0000) 2024-03-06T18:14:40.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:40.809+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:40.810894+0000) 2024-03-06T18:14:41.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:41.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:41.811119+0000) 2024-03-06T18:14:42.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:42.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:42.811277+0000) 2024-03-06T18:14:43.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:43.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:43.811497+0000) 2024-03-06T18:14:44.808 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:44.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:44.811727+0000) 2024-03-06T18:14:45.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:45.810+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:45.811956+0000) 2024-03-06T18:14:46.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:46.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:46.812191+0000) 2024-03-06T18:14:46.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:46.882 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:14:47.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:47.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:47.812391+0000) 2024-03-06T18:14:48.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:48.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:48.812630+0000) 2024-03-06T18:14:49.809 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:49.811+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:49.812877+0000) 2024-03-06T18:14:50.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:50.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:50.813118+0000) 2024-03-06T18:14:51.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:51.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:51.813399+0000) 2024-03-06T18:14:52.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:52.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:52.813667+0000) 2024-03-06T18:14:53.810 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:53.812+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:53.813897+0000) 2024-03-06T18:14:54.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:54.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:54.814140+0000) 2024-03-06T18:14:55.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:55.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:55.814382+0000) 2024-03-06T18:14:56.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:56.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:56.814647+0000) 2024-03-06T18:14:57.811 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:57.813+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:57.814875+0000) 2024-03-06T18:14:58.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:58.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:58.815134+0000) 2024-03-06T18:14:59.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:14:59.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:14:59.815349+0000) 2024-03-06T18:15:00.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:00.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:00.815598+0000) 2024-03-06T18:15:01.812 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:01.814+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:01.815823+0000) 2024-03-06T18:15:02.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:02.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:02.816124+0000) 2024-03-06T18:15:03.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:03.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:03.816346+0000) 2024-03-06T18:15:04.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:04.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:04.816534+0000) 2024-03-06T18:15:05.813 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:05.815+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:05.816778+0000) 2024-03-06T18:15:06.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:06.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:06.817076+0000) 2024-03-06T18:15:07.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:07.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:07.817328+0000) 2024-03-06T18:15:08.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:08.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:08.817528+0000) 2024-03-06T18:15:09.814 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:09.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:09.817756+0000) 2024-03-06T18:15:10.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:10.816+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:10.817982+0000) 2024-03-06T18:15:11.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:11.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:11.818312+0000) 2024-03-06T18:15:12.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:12.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:12.818597+0000) 2024-03-06T18:15:13.815 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:13.817+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:13.818881+0000) 2024-03-06T18:15:14.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:14.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:14.819176+0000) 2024-03-06T18:15:15.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:15.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:15.819357+0000) 2024-03-06T18:15:16.816 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:16.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:16.819596+0000) 2024-03-06T18:15:17.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:17.069 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:17.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:17.818+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:17.819832+0000) 2024-03-06T18:15:18.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:18.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:18.820114+0000) 2024-03-06T18:15:19.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:19.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:19.820411+0000) 2024-03-06T18:15:20.817 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:20.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:20.820645+0000) 2024-03-06T18:15:21.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:21.819+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:21.820884+0000) 2024-03-06T18:15:22.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:22.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:22.821137+0000) 2024-03-06T18:15:23.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:23.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:23.821328+0000) 2024-03-06T18:15:24.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:24.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:24.821540+0000) 2024-03-06T18:15:25.818 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:25.820+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:25.821791+0000) 2024-03-06T18:15:26.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:26.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:26.822069+0000) 2024-03-06T18:15:27.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:27.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:27.822329+0000) 2024-03-06T18:15:28.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:28.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:28.822553+0000) 2024-03-06T18:15:29.819 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:29.821+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:29.822775+0000) 2024-03-06T18:15:30.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:30.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:30.823001+0000) 2024-03-06T18:15:31.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:31.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:31.823258+0000) 2024-03-06T18:15:32.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:32.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:32.823495+0000) 2024-03-06T18:15:33.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:33.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:33.823680+0000) 2024-03-06T18:15:34.820 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:34.822+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:34.823931+0000) 2024-03-06T18:15:35.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:35.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:35.824178+0000) 2024-03-06T18:15:36.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:36.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:36.824431+0000) 2024-03-06T18:15:37.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:37.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:37.824662+0000) 2024-03-06T18:15:38.821 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:38.823+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:38.824868+0000) 2024-03-06T18:15:39.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:39.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:39.825117+0000) 2024-03-06T18:15:40.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:40.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:40.825319+0000) 2024-03-06T18:15:41.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:41.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:41.825533+0000) 2024-03-06T18:15:42.822 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:42.824+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:42.825775+0000) 2024-03-06T18:15:43.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:43.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:43.826052+0000) 2024-03-06T18:15:44.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:44.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:44.826285+0000) 2024-03-06T18:15:45.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:45.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:45.826509+0000) 2024-03-06T18:15:46.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:46.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:46.826737+0000) 2024-03-06T18:15:47.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:47.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:15:47.823 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:47.825+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:47.826984+0000) 2024-03-06T18:15:48.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:48.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:48.827213+0000) 2024-03-06T18:15:49.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:49.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:49.827404+0000) 2024-03-06T18:15:50.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:50.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:50.827613+0000) 2024-03-06T18:15:51.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:51.826+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:51.827882+0000) 2024-03-06T18:15:52.824 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:52.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:52.828181+0000) 2024-03-06T18:15:53.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:53.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:53.828432+0000) 2024-03-06T18:15:54.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:54.827+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:54.828714+0000) 2024-03-06T18:15:55.825 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:55.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:55.828995+0000) 2024-03-06T18:15:56.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:56.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:56.829256+0000) 2024-03-06T18:15:57.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:57.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:57.829498+0000) 2024-03-06T18:15:58.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:58.828+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:58.829723+0000) 2024-03-06T18:15:59.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:15:59.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:15:59.829900+0000) 2024-03-06T18:16:00.826 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:00.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:00.830140+0000) 2024-03-06T18:16:01.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:01.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:01.830388+0000) 2024-03-06T18:16:02.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:02.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:02.830663+0000) 2024-03-06T18:16:03.827 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:03.829+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:03.830941+0000) 2024-03-06T18:16:04.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:04.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:04.831177+0000) 2024-03-06T18:16:05.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:05.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:05.831399+0000) 2024-03-06T18:16:06.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:06.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:06.831630+0000) 2024-03-06T18:16:07.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:07.830+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:07.831857+0000) 2024-03-06T18:16:08.828 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:08.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:08.832143+0000) 2024-03-06T18:16:09.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:09.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:09.832339+0000) 2024-03-06T18:16:10.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:10.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:10.832562+0000) 2024-03-06T18:16:11.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:11.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:11.832795+0000) 2024-03-06T18:16:12.829 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:12.831+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:12.832973+0000) 2024-03-06T18:16:13.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:13.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:13.833233+0000) 2024-03-06T18:16:14.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:14.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:14.833444+0000) 2024-03-06T18:16:15.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:15.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:15.833702+0000) 2024-03-06T18:16:16.830 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:16.832+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:16.833909+0000) 2024-03-06T18:16:17.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:17.419 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:17.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:17.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:17.834188+0000) 2024-03-06T18:16:18.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:18.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:18.834369+0000) 2024-03-06T18:16:19.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:19.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:19.834556+0000) 2024-03-06T18:16:20.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:20.833+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:20.834785+0000) 2024-03-06T18:16:21.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:21.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:21.835082+0000) 2024-03-06T18:16:22.831 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:22.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:22.835277+0000) 2024-03-06T18:16:23.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:23.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:23.835504+0000) 2024-03-06T18:16:24.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:24.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:24.835724+0000) 2024-03-06T18:16:25.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:25.834+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:25.835959+0000) 2024-03-06T18:16:26.832 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:26.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:26.836190+0000) 2024-03-06T18:16:27.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:27.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:27.836387+0000) 2024-03-06T18:16:28.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:28.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:28.836604+0000) 2024-03-06T18:16:29.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:29.835+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:29.836817+0000) 2024-03-06T18:16:30.833 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:30.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:30.837093+0000) 2024-03-06T18:16:31.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:31.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:31.837304+0000) 2024-03-06T18:16:32.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:32.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:32.837533+0000) 2024-03-06T18:16:33.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:33.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:33.837748+0000) 2024-03-06T18:16:34.834 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:34.836+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:34.837977+0000) 2024-03-06T18:16:35.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:35.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:35.838244+0000) 2024-03-06T18:16:36.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:36.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:36.838487+0000) 2024-03-06T18:16:37.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:37.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:37.838723+0000) 2024-03-06T18:16:38.835 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:38.837+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:38.838951+0000) 2024-03-06T18:16:39.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:39.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:39.839144+0000) 2024-03-06T18:16:40.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:40.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:40.839338+0000) 2024-03-06T18:16:41.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:41.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:41.839570+0000) 2024-03-06T18:16:42.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:42.838+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:42.839800+0000) 2024-03-06T18:16:43.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:43.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:43.840053+0000) 2024-03-06T18:16:44.836 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:44.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:44.840258+0000) 2024-03-06T18:16:45.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:45.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:45.840480+0000) 2024-03-06T18:16:46.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:46.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:46.840715+0000) 2024-03-06T18:16:47.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:47.623 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:16:47.837 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:47.839+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:47.840886+0000) 2024-03-06T18:16:48.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:48.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:48.841144+0000) 2024-03-06T18:16:49.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:49.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:49.841288+0000) 2024-03-06T18:16:50.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:50.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:50.841526+0000) 2024-03-06T18:16:51.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:51.840+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:51.841760+0000) 2024-03-06T18:16:52.838 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:52.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:52.841994+0000) 2024-03-06T18:16:53.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:53.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:53.842248+0000) 2024-03-06T18:16:54.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:54.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:54.842484+0000) 2024-03-06T18:16:55.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:55.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:55.842685+0000) 2024-03-06T18:16:56.839 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:56.841+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:56.842891+0000) 2024-03-06T18:16:57.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:57.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:57.843149+0000) 2024-03-06T18:16:58.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:58.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:58.843333+0000) 2024-03-06T18:16:59.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:16:59.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:16:59.843531+0000) 2024-03-06T18:17:00.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:00.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:00.843756+0000) 2024-03-06T18:17:01.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:01.842+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:01.843944+0000) 2024-03-06T18:17:02.840 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:02.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:02.844193+0000) 2024-03-06T18:17:03.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:03.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:03.844376+0000) 2024-03-06T18:17:04.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:04.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:04.844597+0000) 2024-03-06T18:17:05.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:05.843+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:05.844832+0000) 2024-03-06T18:17:06.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:06.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:06.845113+0000) 2024-03-06T18:17:07.841 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:07.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:07.845317+0000) 2024-03-06T18:17:08.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:08.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:08.845610+0000) 2024-03-06T18:17:09.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:09.844+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:09.845848+0000) 2024-03-06T18:17:10.842 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:10.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:10.846062+0000) 2024-03-06T18:17:11.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:11.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:11.846321+0000) 2024-03-06T18:17:12.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:12.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:12.846576+0000) 2024-03-06T18:17:13.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:13.845+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:13.846812+0000) 2024-03-06T18:17:14.843 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:14.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:14.847096+0000) 2024-03-06T18:17:15.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:15.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:15.847309+0000) 2024-03-06T18:17:16.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:16.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:16.847548+0000) 2024-03-06T18:17:17.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:17.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:17.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:17.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:17.847716+0000) 2024-03-06T18:17:18.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:18.846+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:18.847967+0000) 2024-03-06T18:17:19.844 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:19.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:19.848252+0000) 2024-03-06T18:17:20.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:20.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:20.848516+0000) 2024-03-06T18:17:21.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:21.847+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:21.848799+0000) 2024-03-06T18:17:22.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:22.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:22.849097+0000) 2024-03-06T18:17:23.845 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:23.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:23.849323+0000) 2024-03-06T18:17:24.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:24.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:24.849524+0000) 2024-03-06T18:17:25.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:25.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:25.849728+0000) 2024-03-06T18:17:26.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:26.848+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:26.849981+0000) 2024-03-06T18:17:27.846 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:27.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:27.850235+0000) 2024-03-06T18:17:28.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:28.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:28.850457+0000) 2024-03-06T18:17:29.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:29.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:29.850633+0000) 2024-03-06T18:17:30.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:30.849+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:30.850878+0000) 2024-03-06T18:17:31.847 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:31.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:31.851138+0000) 2024-03-06T18:17:32.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:32.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:32.851324+0000) 2024-03-06T18:17:33.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:33.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:33.851581+0000) 2024-03-06T18:17:34.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:34.850+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:34.851823+0000) 2024-03-06T18:17:35.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:35.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:35.852105+0000) 2024-03-06T18:17:36.848 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:36.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:36.852322+0000) 2024-03-06T18:17:37.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:37.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:37.852576+0000) 2024-03-06T18:17:38.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:38.851+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:38.852796+0000) 2024-03-06T18:17:39.849 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:39.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:39.853092+0000) 2024-03-06T18:17:40.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:40.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:40.853296+0000) 2024-03-06T18:17:41.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:41.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:41.853516+0000) 2024-03-06T18:17:42.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:42.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:42.853739+0000) 2024-03-06T18:17:43.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:43.852+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:43.853957+0000) 2024-03-06T18:17:44.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:44.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:44.854162+0000) 2024-03-06T18:17:45.850 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:45.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:45.854364+0000) 2024-03-06T18:17:46.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:46.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:46.854597+0000) 2024-03-06T18:17:47.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:47.853+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:47.854863+0000) 2024-03-06T18:17:48.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:48.032 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:17:48.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:48.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:48.855135+0000) 2024-03-06T18:17:49.851 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:49.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:49.855336+0000) 2024-03-06T18:17:50.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:50.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:50.855554+0000) 2024-03-06T18:17:51.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:51.854+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:51.855830+0000) 2024-03-06T18:17:52.852 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:52.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:52.856128+0000) 2024-03-06T18:17:53.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:53.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:53.856329+0000) 2024-03-06T18:17:54.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:54.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:54.856553+0000) 2024-03-06T18:17:55.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:55.855+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:55.856773+0000) 2024-03-06T18:17:56.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:56.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:56.857123+0000) 2024-03-06T18:17:57.853 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:57.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:57.857332+0000) 2024-03-06T18:17:58.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:58.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:58.857568+0000) 2024-03-06T18:17:59.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:17:59.856+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:17:59.857789+0000) 2024-03-06T18:18:00.854 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:00.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:00.858065+0000) 2024-03-06T18:18:01.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:01.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:01.858269+0000) 2024-03-06T18:18:02.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:02.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:02.858495+0000) 2024-03-06T18:18:03.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:03.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:03.858699+0000) 2024-03-06T18:18:04.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:04.857+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:04.858924+0000) 2024-03-06T18:18:05.855 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:05.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:05.859175+0000) 2024-03-06T18:18:06.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:06.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:06.859378+0000) 2024-03-06T18:18:07.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:07.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:07.859584+0000) 2024-03-06T18:18:08.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:08.858+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:08.859828+0000) 2024-03-06T18:18:09.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:09.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:09.860056+0000) 2024-03-06T18:18:10.856 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:10.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:10.860281+0000) 2024-03-06T18:18:11.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:11.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:11.860525+0000) 2024-03-06T18:18:12.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:12.859+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:12.860762+0000) 2024-03-06T18:18:13.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:13.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:13.860996+0000) 2024-03-06T18:18:14.857 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:14.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:14.861245+0000) 2024-03-06T18:18:15.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:15.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:15.861505+0000) 2024-03-06T18:18:16.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:16.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:16.861753+0000) 2024-03-06T18:18:17.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:17.860+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:17.861990+0000) 2024-03-06T18:18:18.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:18.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:18.858 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:18.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:18.862236+0000) 2024-03-06T18:18:19.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:19.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:19.862486+0000) 2024-03-06T18:18:20.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:20.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:20.862707+0000) 2024-03-06T18:18:21.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:21.861+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:21.862934+0000) 2024-03-06T18:18:22.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:22.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:22.863169+0000) 2024-03-06T18:18:23.859 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:23.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:23.863408+0000) 2024-03-06T18:18:24.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:24.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:24.863650+0000) 2024-03-06T18:18:25.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:25.862+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:25.863871+0000) 2024-03-06T18:18:26.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:26.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:26.864139+0000) 2024-03-06T18:18:27.860 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:27.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:27.864341+0000) 2024-03-06T18:18:28.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:28.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:28.864549+0000) 2024-03-06T18:18:29.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:29.863+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:29.864788+0000) 2024-03-06T18:18:30.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:30.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:30.865070+0000) 2024-03-06T18:18:31.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:31.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:31.865326+0000) 2024-03-06T18:18:32.861 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:32.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:32.865534+0000) 2024-03-06T18:18:33.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:33.864+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:33.865759+0000) 2024-03-06T18:18:34.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:34.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:34.865991+0000) 2024-03-06T18:18:35.862 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:35.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:35.866305+0000) 2024-03-06T18:18:36.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:36.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:36.866542+0000) 2024-03-06T18:18:37.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:37.865+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:37.866775+0000) 2024-03-06T18:18:38.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:38.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:38.866999+0000) 2024-03-06T18:18:39.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:39.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:39.867227+0000) 2024-03-06T18:18:40.863 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:40.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:40.867417+0000) 2024-03-06T18:18:41.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:41.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:41.867667+0000) 2024-03-06T18:18:42.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:42.866+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:42.867901+0000) 2024-03-06T18:18:43.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:43.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:43.868129+0000) 2024-03-06T18:18:44.864 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:44.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:44.868403+0000) 2024-03-06T18:18:45.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:45.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:45.868614+0000) 2024-03-06T18:18:46.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:46.867+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:46.868850+0000) 2024-03-06T18:18:47.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:47.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:47.869158+0000) 2024-03-06T18:18:48.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:48.423 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:18:48.865 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:48.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:48.869363+0000) 2024-03-06T18:18:49.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:49.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:49.869594+0000) 2024-03-06T18:18:50.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:50.868+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:50.869818+0000) 2024-03-06T18:18:51.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:51.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:51.870117+0000) 2024-03-06T18:18:52.866 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:52.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:52.870306+0000) 2024-03-06T18:18:53.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:53.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:53.870549+0000) 2024-03-06T18:18:54.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:54.869+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:54.870769+0000) 2024-03-06T18:18:55.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:55.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:55.870996+0000) 2024-03-06T18:18:56.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:56.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:56.871233+0000) 2024-03-06T18:18:57.867 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:57.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:57.871455+0000) 2024-03-06T18:18:58.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:58.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:58.871674+0000) 2024-03-06T18:18:59.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:18:59.870+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:18:59.871893+0000) 2024-03-06T18:19:00.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:00.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:00.872140+0000) 2024-03-06T18:19:01.868 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:01.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:01.872352+0000) 2024-03-06T18:19:02.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:02.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:02.872570+0000) 2024-03-06T18:19:03.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:03.871+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:03.872788+0000) 2024-03-06T18:19:04.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:04.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:04.873071+0000) 2024-03-06T18:19:05.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:05.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:05.873263+0000) 2024-03-06T18:19:06.869 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:06.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:06.873532+0000) 2024-03-06T18:19:07.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:07.872+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:07.873777+0000) 2024-03-06T18:19:08.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:08.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:08.874038+0000) 2024-03-06T18:19:09.870 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:09.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:09.874244+0000) 2024-03-06T18:19:10.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:10.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:10.874476+0000) 2024-03-06T18:19:11.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:11.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:11.874726+0000) 2024-03-06T18:19:12.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:12.873+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:12.874956+0000) 2024-03-06T18:19:13.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:13.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:13.875228+0000) 2024-03-06T18:19:14.871 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:14.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:14.875421+0000) 2024-03-06T18:19:15.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:15.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:15.875636+0000) 2024-03-06T18:19:16.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:16.874+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:16.875884+0000) 2024-03-06T18:19:17.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:17.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:17.876146+0000) 2024-03-06T18:19:18.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:18.570 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:18.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:18.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:18.876354+0000) 2024-03-06T18:19:19.872 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:19.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:19.876563+0000) 2024-03-06T18:19:20.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:20.875+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:20.876787+0000) 2024-03-06T18:19:21.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:21.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:21.877041+0000) 2024-03-06T18:19:22.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:22.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:22.877242+0000) 2024-03-06T18:19:23.873 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:23.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:23.877461+0000) 2024-03-06T18:19:24.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:24.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:24.877667+0000) 2024-03-06T18:19:25.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:25.876+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:25.877889+0000) 2024-03-06T18:19:26.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:26.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:26.878138+0000) 2024-03-06T18:19:27.874 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:27.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:27.878408+0000) 2024-03-06T18:19:28.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:28.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:28.878602+0000) 2024-03-06T18:19:29.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:29.877+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:29.878852+0000) 2024-03-06T18:19:30.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:30.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:30.879129+0000) 2024-03-06T18:19:31.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:31.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:31.879329+0000) 2024-03-06T18:19:32.875 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:32.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:32.879570+0000) 2024-03-06T18:19:33.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:33.878+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:33.879776+0000) 2024-03-06T18:19:34.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:34.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:34.880096+0000) 2024-03-06T18:19:35.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:35.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:35.880274+0000) 2024-03-06T18:19:36.876 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:36.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:36.880520+0000) 2024-03-06T18:19:37.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:37.879+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:37.880805+0000) 2024-03-06T18:19:38.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:38.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:38.881062+0000) 2024-03-06T18:19:39.877 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:39.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:39.881251+0000) 2024-03-06T18:19:40.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:40.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:40.881466+0000) 2024-03-06T18:19:41.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:41.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:41.881697+0000) 2024-03-06T18:19:42.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:42.880+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:42.881925+0000) 2024-03-06T18:19:43.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:43.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:43.882153+0000) 2024-03-06T18:19:44.878 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:44.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:44.882343+0000) 2024-03-06T18:19:45.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:45.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:45.882583+0000) 2024-03-06T18:19:46.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:46.881+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:46.882808+0000) 2024-03-06T18:19:47.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:47.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:47.883095+0000) 2024-03-06T18:19:48.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:48.706 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:19:48.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:48.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:48.883341+0000) 2024-03-06T18:19:49.879 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:49.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:49.883567+0000) 2024-03-06T18:19:50.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:50.882+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:50.883795+0000) 2024-03-06T18:19:51.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:51.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:51.884054+0000) 2024-03-06T18:19:52.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:52.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:52.884274+0000) 2024-03-06T18:19:53.880 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:53.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:53.884501+0000) 2024-03-06T18:19:54.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:54.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:54.884724+0000) 2024-03-06T18:19:55.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:55.883+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:55.884959+0000) 2024-03-06T18:19:56.881 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:56.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:56.885197+0000) 2024-03-06T18:19:57.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:57.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:57.885481+0000) 2024-03-06T18:19:58.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:58.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:58.885728+0000) 2024-03-06T18:19:59.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:19:59.884+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:19:59.885967+0000) 2024-03-06T18:20:00.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:00.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:00.886169+0000) 2024-03-06T18:20:01.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:01.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:01.886375+0000) 2024-03-06T18:20:02.882 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:02.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:02.886619+0000) 2024-03-06T18:20:03.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:03.885+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:03.886841+0000) 2024-03-06T18:20:04.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:04.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:04.887140+0000) 2024-03-06T18:20:05.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:05.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:05.887332+0000) 2024-03-06T18:20:06.883 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:06.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:06.887574+0000) 2024-03-06T18:20:07.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:07.886+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:07.887822+0000) 2024-03-06T18:20:08.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:08.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:08.888064+0000) 2024-03-06T18:20:09.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:09.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:09.888314+0000) 2024-03-06T18:20:10.884 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:10.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:10.888515+0000) 2024-03-06T18:20:11.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:11.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:11.888755+0000) 2024-03-06T18:20:12.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:12.887+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:12.888954+0000) 2024-03-06T18:20:13.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:13.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:13.889170+0000) 2024-03-06T18:20:14.885 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:14.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:14.889393+0000) 2024-03-06T18:20:15.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:15.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:15.889637+0000) 2024-03-06T18:20:16.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:16.888+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:16.889863+0000) 2024-03-06T18:20:17.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:17.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:17.890150+0000) 2024-03-06T18:20:18.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:18.840 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:18.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:18.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:18.890347+0000) 2024-03-06T18:20:19.886 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:19.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:19.890587+0000) 2024-03-06T18:20:20.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:20.889+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:20.890837+0000) 2024-03-06T18:20:21.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:21.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:21.891141+0000) 2024-03-06T18:20:22.887 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:22.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:22.891393+0000) 2024-03-06T18:20:23.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:23.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:23.891630+0000) 2024-03-06T18:20:24.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:24.890+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:24.891852+0000) 2024-03-06T18:20:25.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:25.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:25.892162+0000) 2024-03-06T18:20:26.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:26.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:26.892413+0000) 2024-03-06T18:20:27.888 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:27.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:27.892653+0000) 2024-03-06T18:20:28.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:28.891+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:28.892880+0000) 2024-03-06T18:20:29.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:29.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:29.893122+0000) 2024-03-06T18:20:30.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:30.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:30.893356+0000) 2024-03-06T18:20:31.889 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:31.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:31.893571+0000) 2024-03-06T18:20:32.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:32.892+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:32.893829+0000) 2024-03-06T18:20:33.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:33.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:33.894094+0000) 2024-03-06T18:20:34.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:34.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:34.894283+0000) 2024-03-06T18:20:35.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:35.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:35.894493+0000) 2024-03-06T18:20:36.890 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:36.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:36.894721+0000) 2024-03-06T18:20:37.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:37.893+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:37.894962+0000) 2024-03-06T18:20:38.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:38.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:38.895216+0000) 2024-03-06T18:20:39.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:39.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:39.895404+0000) 2024-03-06T18:20:40.891 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:40.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:40.895616+0000) 2024-03-06T18:20:41.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:41.894+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:41.895843+0000) 2024-03-06T18:20:42.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:42.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:42.896116+0000) 2024-03-06T18:20:43.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:43.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:43.896303+0000) 2024-03-06T18:20:44.892 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:44.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:44.896512+0000) 2024-03-06T18:20:45.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:45.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:45.896707+0000) 2024-03-06T18:20:46.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:46.895+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:46.896933+0000) 2024-03-06T18:20:47.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:47.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:47.897164+0000) 2024-03-06T18:20:48.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:48.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:48.897355+0000) 2024-03-06T18:20:48.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:48.973 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:20:49.893 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:49.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:49.897544+0000) 2024-03-06T18:20:50.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:50.896+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:50.897808+0000) 2024-03-06T18:20:51.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:51.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:51.898083+0000) 2024-03-06T18:20:52.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:52.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:52.898286+0000) 2024-03-06T18:20:53.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:53.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:53.898517+0000) 2024-03-06T18:20:54.894 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:54.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:54.898741+0000) 2024-03-06T18:20:55.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:55.897+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:55.898972+0000) 2024-03-06T18:20:56.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:56.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:56.899174+0000) 2024-03-06T18:20:57.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:57.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:57.899388+0000) 2024-03-06T18:20:58.895 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:58.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:58.899598+0000) 2024-03-06T18:20:59.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:20:59.898+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:20:59.899819+0000) 2024-03-06T18:21:00.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:00.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:00.900131+0000) 2024-03-06T18:21:01.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:01.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:01.900344+0000) 2024-03-06T18:21:02.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:02.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:02.900549+0000) 2024-03-06T18:21:03.896 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:03.899+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:03.900810+0000) 2024-03-06T18:21:04.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:04.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:04.901043+0000) 2024-03-06T18:21:05.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:05.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:05.901270+0000) 2024-03-06T18:21:06.897 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:06.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:06.901514+0000) 2024-03-06T18:21:07.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:07.900+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:07.901785+0000) 2024-03-06T18:21:08.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:08.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:08.902064+0000) 2024-03-06T18:21:09.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:09.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:09.902271+0000) 2024-03-06T18:21:10.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:10.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:10.902465+0000) 2024-03-06T18:21:11.898 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:11.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:11.902674+0000) 2024-03-06T18:21:12.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:12.901+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:12.902916+0000) 2024-03-06T18:21:13.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:13.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:13.903216+0000) 2024-03-06T18:21:14.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:14.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:14.903406+0000) 2024-03-06T18:21:15.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:15.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:15.903602+0000) 2024-03-06T18:21:16.899 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:16.902+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:16.903833+0000) 2024-03-06T18:21:17.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:17.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:17.904119+0000) 2024-03-06T18:21:18.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:18.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:18.904316+0000) 2024-03-06T18:21:19.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:19.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:19.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:19.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:19.904550+0000) 2024-03-06T18:21:20.900 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:20.903+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:20.904751+0000) 2024-03-06T18:21:21.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:21.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:21.905049+0000) 2024-03-06T18:21:22.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:22.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:22.905261+0000) 2024-03-06T18:21:23.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:23.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:23.905473+0000) 2024-03-06T18:21:24.901 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:24.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:24.905713+0000) 2024-03-06T18:21:25.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:25.904+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:25.905938+0000) 2024-03-06T18:21:26.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:26.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:26.906181+0000) 2024-03-06T18:21:27.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:27.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:27.906412+0000) 2024-03-06T18:21:28.902 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:28.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:28.906635+0000) 2024-03-06T18:21:29.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:29.905+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:29.906885+0000) 2024-03-06T18:21:30.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:30.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:30.907127+0000) 2024-03-06T18:21:31.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:31.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:31.907314+0000) 2024-03-06T18:21:32.903 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:32.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:32.907535+0000) 2024-03-06T18:21:33.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:33.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:33.907766+0000) 2024-03-06T18:21:34.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:34.906+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:34.907969+0000) 2024-03-06T18:21:35.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:35.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:35.908257+0000) 2024-03-06T18:21:36.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:36.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:36.908420+0000) 2024-03-06T18:21:37.904 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:37.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:37.908677+0000) 2024-03-06T18:21:38.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:38.907+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:38.908923+0000) 2024-03-06T18:21:39.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:39.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:39.909158+0000) 2024-03-06T18:21:40.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:40.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:40.909367+0000) 2024-03-06T18:21:41.905 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:41.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:41.909620+0000) 2024-03-06T18:21:42.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:42.908+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:42.909854+0000) 2024-03-06T18:21:43.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:43.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:43.910149+0000) 2024-03-06T18:21:44.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:44.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:44.910353+0000) 2024-03-06T18:21:45.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:45.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:45.910577+0000) 2024-03-06T18:21:46.906 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:46.909+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:46.910774+0000) 2024-03-06T18:21:47.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:47.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:47.911077+0000) 2024-03-06T18:21:48.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:48.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:48.911289+0000) 2024-03-06T18:21:49.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:49.339 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:21:49.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:49.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:49.911552+0000) 2024-03-06T18:21:50.907 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:50.910+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:50.911808+0000) 2024-03-06T18:21:51.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:51.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:51.912085+0000) 2024-03-06T18:21:52.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:52.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:52.912368+0000) 2024-03-06T18:21:53.908 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:53.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:53.912597+0000) 2024-03-06T18:21:54.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:54.911+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:54.912835+0000) 2024-03-06T18:21:55.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:55.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:55.913068+0000) 2024-03-06T18:21:56.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:56.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:56.913289+0000) 2024-03-06T18:21:57.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:57.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:57.913545+0000) 2024-03-06T18:21:58.909 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:58.912+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:58.913811+0000) 2024-03-06T18:21:59.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:21:59.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:21:59.914105+0000) 2024-03-06T18:22:00.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:00.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:00.914396+0000) 2024-03-06T18:22:01.910 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:01.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:01.914672+0000) 2024-03-06T18:22:02.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:02.913+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:02.914932+0000) 2024-03-06T18:22:03.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:03.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:03.915148+0000) 2024-03-06T18:22:04.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:04.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:04.915371+0000) 2024-03-06T18:22:05.911 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:05.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:05.915613+0000) 2024-03-06T18:22:06.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:06.914+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:06.915872+0000) 2024-03-06T18:22:07.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:07.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:07.916182+0000) 2024-03-06T18:22:08.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:08.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:08.916377+0000) 2024-03-06T18:22:09.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:09.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:09.916581+0000) 2024-03-06T18:22:10.912 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:10.915+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:10.916803+0000) 2024-03-06T18:22:11.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:11.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:11.917050+0000) 2024-03-06T18:22:12.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:12.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:12.917278+0000) 2024-03-06T18:22:13.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:13.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:13.917475+0000) 2024-03-06T18:22:14.913 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:14.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:14.917703+0000) 2024-03-06T18:22:15.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:15.916+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:15.917925+0000) 2024-03-06T18:22:16.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:16.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:16.918145+0000) 2024-03-06T18:22:17.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:17.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:17.918357+0000) 2024-03-06T18:22:18.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:18.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:18.918561+0000) 2024-03-06T18:22:19.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:19.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:19.914 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:19.917+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:19.918795+0000) 2024-03-06T18:22:20.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:20.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:20.919049+0000) 2024-03-06T18:22:21.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:21.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:21.919273+0000) 2024-03-06T18:22:22.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:22.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:22.919491+0000) 2024-03-06T18:22:23.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:23.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:23.919711+0000) 2024-03-06T18:22:24.915 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:24.918+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:24.919934+0000) 2024-03-06T18:22:25.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:25.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:25.920164+0000) 2024-03-06T18:22:26.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:26.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:26.920373+0000) 2024-03-06T18:22:27.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:27.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:27.920619+0000) 2024-03-06T18:22:28.916 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:28.919+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:28.920816+0000) 2024-03-06T18:22:29.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:29.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:29.920996+0000) 2024-03-06T18:22:30.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:30.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:30.921225+0000) 2024-03-06T18:22:31.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:31.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:31.921443+0000) 2024-03-06T18:22:32.917 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:32.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:32.921685+0000) 2024-03-06T18:22:33.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:33.920+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:33.921922+0000) 2024-03-06T18:22:34.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:34.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:34.922159+0000) 2024-03-06T18:22:35.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:35.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:35.922348+0000) 2024-03-06T18:22:36.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:36.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:36.922539+0000) 2024-03-06T18:22:37.918 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:37.921+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:37.922786+0000) 2024-03-06T18:22:38.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:38.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:38.923040+0000) 2024-03-06T18:22:39.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:39.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:39.923269+0000) 2024-03-06T18:22:40.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:40.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:40.923494+0000) 2024-03-06T18:22:41.919 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:41.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:41.923754+0000) 2024-03-06T18:22:42.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:42.922+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:42.923989+0000) 2024-03-06T18:22:43.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:43.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:43.924256+0000) 2024-03-06T18:22:44.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:44.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:44.924479+0000) 2024-03-06T18:22:45.920 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:45.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:45.924700+0000) 2024-03-06T18:22:46.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:46.923+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:46.924955+0000) 2024-03-06T18:22:47.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:47.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:47.925212+0000) 2024-03-06T18:22:48.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:48.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:48.925450+0000) 2024-03-06T18:22:49.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:49.774 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:22:49.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:49.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:49.925711+0000) 2024-03-06T18:22:50.921 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:50.924+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:50.925931+0000) 2024-03-06T18:22:51.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:51.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:51.926172+0000) 2024-03-06T18:22:52.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:52.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:52.926422+0000) 2024-03-06T18:22:53.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:53.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:53.926662+0000) 2024-03-06T18:22:54.922 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:54.925+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:54.926876+0000) 2024-03-06T18:22:55.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:55.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:55.927197+0000) 2024-03-06T18:22:56.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:56.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:56.927426+0000) 2024-03-06T18:22:57.923 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:57.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:57.927680+0000) 2024-03-06T18:22:58.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:58.926+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:58.927907+0000) 2024-03-06T18:22:59.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:22:59.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:22:59.928140+0000) 2024-03-06T18:23:00.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:00.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:00.928364+0000) 2024-03-06T18:23:01.924 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:01.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:01.928620+0000) 2024-03-06T18:23:02.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:02.927+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:02.928889+0000) 2024-03-06T18:23:03.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:03.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:03.929119+0000) 2024-03-06T18:23:04.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:04.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:04.929344+0000) 2024-03-06T18:23:05.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:05.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:05.929551+0000) 2024-03-06T18:23:06.925 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:06.928+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:06.929778+0000) 2024-03-06T18:23:07.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:07.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:07.930040+0000) 2024-03-06T18:23:08.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:08.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:08.930235+0000) 2024-03-06T18:23:09.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:09.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:09.930459+0000) 2024-03-06T18:23:10.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:10.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:10.930642+0000) 2024-03-06T18:23:11.926 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:11.929+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:11.930870+0000) 2024-03-06T18:23:12.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:12.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:12.931130+0000) 2024-03-06T18:23:13.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:13.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:13.931352+0000) 2024-03-06T18:23:14.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:14.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:14.931585+0000) 2024-03-06T18:23:15.927 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:15.930+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:15.931816+0000) 2024-03-06T18:23:16.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:16.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:16.932093+0000) 2024-03-06T18:23:17.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:17.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:17.932347+0000) 2024-03-06T18:23:18.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:18.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:18.932569+0000) 2024-03-06T18:23:19.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:19.918 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:19.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:19.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:19.932754+0000) 2024-03-06T18:23:20.928 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:20.931+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:20.932977+0000) 2024-03-06T18:23:21.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:21.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:21.933240+0000) 2024-03-06T18:23:22.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:22.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:22.933478+0000) 2024-03-06T18:23:23.929 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:23.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:23.933708+0000) 2024-03-06T18:23:24.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:24.932+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:24.933933+0000) 2024-03-06T18:23:25.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:25.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:25.934161+0000) 2024-03-06T18:23:26.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:26.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:26.934394+0000) 2024-03-06T18:23:27.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:27.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:27.934644+0000) 2024-03-06T18:23:28.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:28.933+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:28.934881+0000) 2024-03-06T18:23:29.930 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:29.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:29.935096+0000) 2024-03-06T18:23:30.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:30.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:30.935289+0000) 2024-03-06T18:23:31.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:31.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:31.935493+0000) 2024-03-06T18:23:32.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:32.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:32.935734+0000) 2024-03-06T18:23:33.931 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:33.934+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:33.935953+0000) 2024-03-06T18:23:34.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:34.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:34.936250+0000) 2024-03-06T18:23:35.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:35.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:35.936482+0000) 2024-03-06T18:23:36.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:36.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:36.936708+0000) 2024-03-06T18:23:37.932 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:37.935+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:37.936973+0000) 2024-03-06T18:23:38.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:38.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:38.937225+0000) 2024-03-06T18:23:39.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:39.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:39.937417+0000) 2024-03-06T18:23:40.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:40.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:40.937625+0000) 2024-03-06T18:23:41.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:41.936+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:41.937823+0000) 2024-03-06T18:23:42.933 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:42.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:42.938043+0000) 2024-03-06T18:23:43.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:43.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:43.938320+0000) 2024-03-06T18:23:44.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:44.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:44.938543+0000) 2024-03-06T18:23:45.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:45.937+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:45.938768+0000) 2024-03-06T18:23:46.934 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:46.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:46.939052+0000) 2024-03-06T18:23:47.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:47.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:47.939360+0000) 2024-03-06T18:23:48.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:48.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:48.939616+0000) 2024-03-06T18:23:49.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:49.938+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:49.939798+0000) 2024-03-06T18:23:50.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:50.131 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:23:50.935 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:50.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:50.940001+0000) 2024-03-06T18:23:51.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:51.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:51.940244+0000) 2024-03-06T18:23:52.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:52.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:52.940483+0000) 2024-03-06T18:23:53.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:53.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:53.940706+0000) 2024-03-06T18:23:54.936 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:54.939+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:54.940933+0000) 2024-03-06T18:23:55.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:55.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:55.941168+0000) 2024-03-06T18:23:56.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:56.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:56.941368+0000) 2024-03-06T18:23:57.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:57.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:57.941635+0000) 2024-03-06T18:23:58.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:58.940+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:58.941843+0000) 2024-03-06T18:23:59.937 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:23:59.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:23:59.942106+0000) 2024-03-06T18:24:00.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:00.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:00.942341+0000) 2024-03-06T18:24:01.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:01.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:01.942595+0000) 2024-03-06T18:24:02.938 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:02.941+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:02.942849+0000) 2024-03-06T18:24:03.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:03.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:03.943116+0000) 2024-03-06T18:24:04.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:04.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:04.943370+0000) 2024-03-06T18:24:05.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:05.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:05.943627+0000) 2024-03-06T18:24:06.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:06.942+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:06.943875+0000) 2024-03-06T18:24:07.939 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:07.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:07.944139+0000) 2024-03-06T18:24:08.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:08.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:08.944328+0000) 2024-03-06T18:24:09.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:09.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:09.944551+0000) 2024-03-06T18:24:10.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:10.943+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:10.944790+0000) 2024-03-06T18:24:11.940 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:11.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:11.945065+0000) 2024-03-06T18:24:12.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:12.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:12.945324+0000) 2024-03-06T18:24:13.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:13.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:13.945516+0000) 2024-03-06T18:24:14.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:14.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:14.945745+0000) 2024-03-06T18:24:15.941 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:15.944+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:15.945973+0000) 2024-03-06T18:24:16.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:16.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:16.946227+0000) 2024-03-06T18:24:17.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:17.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:17.946530+0000) 2024-03-06T18:24:18.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:18.945+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:18.946784+0000) 2024-03-06T18:24:19.942 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:19.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:19.947034+0000) 2024-03-06T18:24:20.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:20.272 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:20.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:20.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:20.947291+0000) 2024-03-06T18:24:21.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:21.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:21.947536+0000) 2024-03-06T18:24:22.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:22.946+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:22.947785+0000) 2024-03-06T18:24:23.943 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:23.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:23.948053+0000) 2024-03-06T18:24:24.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:24.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:24.948279+0000) 2024-03-06T18:24:25.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:25.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:25.948510+0000) 2024-03-06T18:24:26.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:26.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:26.948735+0000) 2024-03-06T18:24:27.944 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:27.947+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:27.948980+0000) 2024-03-06T18:24:28.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:28.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:28.949202+0000) 2024-03-06T18:24:29.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:29.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:29.949424+0000) 2024-03-06T18:24:30.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:30.948+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:30.949722+0000) 2024-03-06T18:24:31.945 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:31.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:31.950000+0000) 2024-03-06T18:24:32.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:32.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:32.950363+0000) 2024-03-06T18:24:33.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:33.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:33.950601+0000) 2024-03-06T18:24:34.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:34.949+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:34.950792+0000) 2024-03-06T18:24:35.946 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:35.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:35.951046+0000) 2024-03-06T18:24:36.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:36.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:36.951280+0000) 2024-03-06T18:24:37.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:37.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:37.951551+0000) 2024-03-06T18:24:38.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:38.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:38.951747+0000) 2024-03-06T18:24:39.947 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:39.950+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:39.951972+0000) 2024-03-06T18:24:40.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:40.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:40.952204+0000) 2024-03-06T18:24:41.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:41.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:41.952522+0000) 2024-03-06T18:24:42.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:42.951+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:42.952771+0000) 2024-03-06T18:24:43.948 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:43.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:43.952991+0000) 2024-03-06T18:24:44.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:44.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:44.953264+0000) 2024-03-06T18:24:45.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:45.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:45.953501+0000) 2024-03-06T18:24:46.949 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:46.952+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:46.953752+0000) 2024-03-06T18:24:47.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:47.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:47.954048+0000) 2024-03-06T18:24:48.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:48.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:48.954263+0000) 2024-03-06T18:24:49.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:49.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:49.954512+0000) 2024-03-06T18:24:50.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:50.523 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:24:50.950 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:50.953+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:50.954771+0000) 2024-03-06T18:24:51.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:51.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:51.955068+0000) 2024-03-06T18:24:52.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:52.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:52.955284+0000) 2024-03-06T18:24:53.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:53.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:53.955487+0000) 2024-03-06T18:24:54.951 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:54.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:54.955714+0000) 2024-03-06T18:24:55.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:55.954+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:55.955953+0000) 2024-03-06T18:24:56.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:56.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:56.956184+0000) 2024-03-06T18:24:57.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:57.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:57.956434+0000) 2024-03-06T18:24:58.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:58.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:58.956659+0000) 2024-03-06T18:24:59.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:24:59.955+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:24:59.956881+0000) 2024-03-06T18:25:00.952 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:00.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:00.957042+0000) 2024-03-06T18:25:01.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:01.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:01.957263+0000) 2024-03-06T18:25:02.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:02.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:02.957511+0000) 2024-03-06T18:25:03.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:03.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:03.957735+0000) 2024-03-06T18:25:04.953 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:04.956+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:04.957965+0000) 2024-03-06T18:25:05.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:05.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:05.958154+0000) 2024-03-06T18:25:06.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:06.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:06.958392+0000) 2024-03-06T18:25:07.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:07.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:07.958628+0000) 2024-03-06T18:25:08.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:08.957+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:08.958864+0000) 2024-03-06T18:25:09.954 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:09.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:09.959151+0000) 2024-03-06T18:25:10.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:10.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:10.959370+0000) 2024-03-06T18:25:11.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:11.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:11.959599+0000) 2024-03-06T18:25:12.955 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:12.958+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:12.959832+0000) 2024-03-06T18:25:13.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:13.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:13.960103+0000) 2024-03-06T18:25:14.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:14.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:14.960325+0000) 2024-03-06T18:25:15.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:15.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:15.960556+0000) 2024-03-06T18:25:16.956 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:16.959+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:16.960815+0000) 2024-03-06T18:25:17.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:17.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:17.961119+0000) 2024-03-06T18:25:18.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:18.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:18.961348+0000) 2024-03-06T18:25:19.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:19.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:19.961572+0000) 2024-03-06T18:25:20.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:20.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:20.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:20.960+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:20.961828+0000) 2024-03-06T18:25:21.957 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:21.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:21.962066+0000) 2024-03-06T18:25:22.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:22.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:22.962318+0000) 2024-03-06T18:25:23.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:23.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:23.962539+0000) 2024-03-06T18:25:24.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:24.961+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:24.962824+0000) 2024-03-06T18:25:25.958 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:25.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:25.963085+0000) 2024-03-06T18:25:26.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:26.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:26.963321+0000) 2024-03-06T18:25:27.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:27.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:27.963562+0000) 2024-03-06T18:25:28.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:28.962+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:28.963789+0000) 2024-03-06T18:25:29.959 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:29.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:29.964053+0000) 2024-03-06T18:25:30.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:30.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:30.964260+0000) 2024-03-06T18:25:31.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:31.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:31.964478+0000) 2024-03-06T18:25:32.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:32.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:32.964709+0000) 2024-03-06T18:25:33.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:33.963+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:33.964934+0000) 2024-03-06T18:25:34.960 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:34.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:34.965168+0000) 2024-03-06T18:25:35.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:35.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:35.965385+0000) 2024-03-06T18:25:36.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:36.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:36.965610+0000) 2024-03-06T18:25:37.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:37.964+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:37.965845+0000) 2024-03-06T18:25:38.961 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:38.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:38.966069+0000) 2024-03-06T18:25:39.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:39.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:39.966300+0000) 2024-03-06T18:25:40.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:40.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:40.966573+0000) 2024-03-06T18:25:41.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:41.965+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:41.966809+0000) 2024-03-06T18:25:42.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:42.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:42.966997+0000) 2024-03-06T18:25:43.962 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:43.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:43.967290+0000) 2024-03-06T18:25:44.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:44.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:44.967514+0000) 2024-03-06T18:25:45.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:45.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:45.967738+0000) 2024-03-06T18:25:46.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:46.966+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:46.967931+0000) 2024-03-06T18:25:47.963 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:47.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:47.968170+0000) 2024-03-06T18:25:48.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:48.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:48.968395+0000) 2024-03-06T18:25:49.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:49.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:49.968613+0000) 2024-03-06T18:25:50.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:50.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:50.967+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:50.968840+0000) 2024-03-06T18:25:50.984 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:25:51.964 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:51.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:51.969161+0000) 2024-03-06T18:25:52.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:52.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:52.969445+0000) 2024-03-06T18:25:53.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:53.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:53.969689+0000) 2024-03-06T18:25:54.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:54.968+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:54.969906+0000) 2024-03-06T18:25:55.965 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:55.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:55.970168+0000) 2024-03-06T18:25:56.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:56.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:56.970414+0000) 2024-03-06T18:25:57.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:57.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:57.970609+0000) 2024-03-06T18:25:58.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:58.969+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:58.970843+0000) 2024-03-06T18:25:59.966 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:25:59.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:25:59.971107+0000) 2024-03-06T18:26:00.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:00.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:00.971335+0000) 2024-03-06T18:26:01.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:01.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:01.971568+0000) 2024-03-06T18:26:02.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:02.970+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:02.971802+0000) 2024-03-06T18:26:03.967 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:03.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:03.972061+0000) 2024-03-06T18:26:04.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:04.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:04.972296+0000) 2024-03-06T18:26:05.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:05.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:05.972525+0000) 2024-03-06T18:26:06.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:06.971+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:06.972754+0000) 2024-03-06T18:26:07.968 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:07.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:07.972997+0000) 2024-03-06T18:26:08.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:08.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:08.973269+0000) 2024-03-06T18:26:09.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:09.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:09.973495+0000) 2024-03-06T18:26:10.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:10.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:10.973672+0000) 2024-03-06T18:26:11.969 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:11.972+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:11.973948+0000) 2024-03-06T18:26:12.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:12.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:12.974232+0000) 2024-03-06T18:26:13.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:13.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:13.974466+0000) 2024-03-06T18:26:14.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:14.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:14.974643+0000) 2024-03-06T18:26:15.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:15.973+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:15.974915+0000) 2024-03-06T18:26:16.970 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:16.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:16.975199+0000) 2024-03-06T18:26:17.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:17.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:17.975487+0000) 2024-03-06T18:26:18.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:18.974+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:18.975767+0000) 2024-03-06T18:26:19.971 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:19.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:19.976095+0000) 2024-03-06T18:26:20.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:20.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:20.976356+0000) 2024-03-06T18:26:21.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:21.193 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:21.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:21.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:21.976615+0000) 2024-03-06T18:26:22.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:22.975+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:22.976845+0000) 2024-03-06T18:26:23.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:23.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:23.977123+0000) 2024-03-06T18:26:24.972 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:24.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:24.977318+0000) 2024-03-06T18:26:25.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:25.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:25.977557+0000) 2024-03-06T18:26:26.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:26.976+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:26.977811+0000) 2024-03-06T18:26:27.973 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:27.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:27.978090+0000) 2024-03-06T18:26:28.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:28.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:28.978329+0000) 2024-03-06T18:26:29.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:29.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:29.978577+0000) 2024-03-06T18:26:30.974 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:30.977+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:30.978847+0000) 2024-03-06T18:26:31.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:31.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:31.979167+0000) 2024-03-06T18:26:32.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:32.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:32.979439+0000) 2024-03-06T18:26:33.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:33.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:33.979682+0000) 2024-03-06T18:26:34.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:34.978+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:34.979931+0000) 2024-03-06T18:26:35.975 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:35.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:35.980244+0000) 2024-03-06T18:26:36.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:36.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:36.980492+0000) 2024-03-06T18:26:37.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:37.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:37.980734+0000) 2024-03-06T18:26:38.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:38.979+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:38.980940+0000) 2024-03-06T18:26:39.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:39.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:39.981127+0000) 2024-03-06T18:26:40.976 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:40.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:40.981349+0000) 2024-03-06T18:26:41.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:41.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:41.981618+0000) 2024-03-06T18:26:42.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:42.980+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:42.981892+0000) 2024-03-06T18:26:43.977 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:43.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:43.982214+0000) 2024-03-06T18:26:44.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:44.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:44.982486+0000) 2024-03-06T18:26:45.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:45.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:45.982727+0000) 2024-03-06T18:26:46.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:46.981+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:46.982946+0000) 2024-03-06T18:26:47.978 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:47.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:47.983251+0000) 2024-03-06T18:26:48.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:48.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:48.983480+0000) 2024-03-06T18:26:49.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:49.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:49.983704+0000) 2024-03-06T18:26:50.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:50.982+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:50.983932+0000) 2024-03-06T18:26:51.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:51.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:26:51.979 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:51.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:51.984151+0000) 2024-03-06T18:26:52.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:52.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:52.984436+0000) 2024-03-06T18:26:53.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:53.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:53.984711+0000) 2024-03-06T18:26:54.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:54.983+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:54.984928+0000) 2024-03-06T18:26:55.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:55.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:55.985157+0000) 2024-03-06T18:26:56.980 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:56.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:56.985390+0000) 2024-03-06T18:26:57.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:57.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:57.985621+0000) 2024-03-06T18:26:58.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:58.984+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:58.985850+0000) 2024-03-06T18:26:59.981 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:26:59.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:26:59.986073+0000) 2024-03-06T18:27:00.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:00.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:00.986299+0000) 2024-03-06T18:27:01.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:01.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:01.986527+0000) 2024-03-06T18:27:02.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:02.985+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:02.986761+0000) 2024-03-06T18:27:03.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:03.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:03.987040+0000) 2024-03-06T18:27:04.982 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:04.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:04.987252+0000) 2024-03-06T18:27:05.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:05.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:05.987562+0000) 2024-03-06T18:27:06.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:06.986+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:06.987801+0000) 2024-03-06T18:27:07.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:07.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:07.988104+0000) 2024-03-06T18:27:08.983 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:08.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:08.988335+0000) 2024-03-06T18:27:09.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:09.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:09.988556+0000) 2024-03-06T18:27:10.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:10.987+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:10.988804+0000) 2024-03-06T18:27:11.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:11.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:11.988994+0000) 2024-03-06T18:27:12.984 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:12.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:12.989216+0000) 2024-03-06T18:27:13.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:13.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:13.989406+0000) 2024-03-06T18:27:14.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:14.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:14.989581+0000) 2024-03-06T18:27:15.985 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:15.988+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:15.989765+0000) 2024-03-06T18:27:16.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:16.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:16.990302+0000) 2024-03-06T18:27:17.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:17.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:17.990702+0000) 2024-03-06T18:27:18.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:18.989+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:18.990931+0000) 2024-03-06T18:27:19.986 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:19.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:19.991152+0000) 2024-03-06T18:27:20.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:20.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:20.991347+0000) 2024-03-06T18:27:21.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:21.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:21.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:21.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:21.991546+0000) 2024-03-06T18:27:22.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:22.990+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:22.991776+0000) 2024-03-06T18:27:23.987 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:23.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:23.992037+0000) 2024-03-06T18:27:24.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:24.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:24.992285+0000) 2024-03-06T18:27:25.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:25.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:25.992527+0000) 2024-03-06T18:27:26.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:26.991+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:26.992766+0000) 2024-03-06T18:27:27.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:27.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:27.993057+0000) 2024-03-06T18:27:28.988 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:28.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:28.993301+0000) 2024-03-06T18:27:29.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:29.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:29.993540+0000) 2024-03-06T18:27:30.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:30.992+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:30.993777+0000) 2024-03-06T18:27:31.989 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:31.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:31.994061+0000) 2024-03-06T18:27:32.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:32.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:32.994320+0000) 2024-03-06T18:27:33.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:33.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:33.994563+0000) 2024-03-06T18:27:34.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:34.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:34.994750+0000) 2024-03-06T18:27:35.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:35.993+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:35.994984+0000) 2024-03-06T18:27:36.990 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:36.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:36.995226+0000) 2024-03-06T18:27:37.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:37.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:37.995419+0000) 2024-03-06T18:27:38.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:38.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:38.995644+0000) 2024-03-06T18:27:39.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:39.994+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:39.995877+0000) 2024-03-06T18:27:40.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:40.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:40.996182+0000) 2024-03-06T18:27:41.991 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:41.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:41.996399+0000) 2024-03-06T18:27:42.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:42.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:42.996589+0000) 2024-03-06T18:27:43.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:43.995+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:43.996809+0000) 2024-03-06T18:27:44.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:44.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:44.997074+0000) 2024-03-06T18:27:45.992 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:45.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:45.997323+0000) 2024-03-06T18:27:46.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:46.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:46.997591+0000) 2024-03-06T18:27:47.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:47.996+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:47.997875+0000) 2024-03-06T18:27:48.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:48.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:48.998173+0000) 2024-03-06T18:27:49.993 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:49.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:49.998420+0000) 2024-03-06T18:27:50.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:50.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:50.998650+0000) 2024-03-06T18:27:51.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:51.851 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:27:51.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:51.997+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:51.998895+0000) 2024-03-06T18:27:52.994 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:52.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:52.999146+0000) 2024-03-06T18:27:53.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:53.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:53.999406+0000) 2024-03-06T18:27:54.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:54.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:54.999639+0000) 2024-03-06T18:27:55.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:55.998+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:55.999824+0000) 2024-03-06T18:27:56.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:56.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:57.000056+0000) 2024-03-06T18:27:57.995 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:57.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:58.000334+0000) 2024-03-06T18:27:58.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:58.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:27:59.000560+0000) 2024-03-06T18:27:59.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:27:59.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:00.000795+0000) 2024-03-06T18:28:00.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:00.999+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:01.000968+0000) 2024-03-06T18:28:01.996 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:02.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:02.001213+0000) 2024-03-06T18:28:02.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:03.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:03.001458+0000) 2024-03-06T18:28:03.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:04.000+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:04.001695+0000) 2024-03-06T18:28:04.997 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:05.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:05.002323+0000) 2024-03-06T18:28:05.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:06.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:06.002571+0000) 2024-03-06T18:28:06.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:07.001+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:07.002802+0000) 2024-03-06T18:28:07.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:08.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:08.003057+0000) 2024-03-06T18:28:08.998 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:09.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:09.003291+0000) 2024-03-06T18:28:09.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:10.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:10.003481+0000) 2024-03-06T18:28:10.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:11.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:11.003700+0000) 2024-03-06T18:28:11.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:12.002+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:12.003951+0000) 2024-03-06T18:28:12.999 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:13.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:13.004361+0000) 2024-03-06T18:28:14.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:14.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:14.004637+0000) 2024-03-06T18:28:15.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:15.003+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:15.004877+0000) 2024-03-06T18:28:16.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:16.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:16.005133+0000) 2024-03-06T18:28:17.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:17.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:17.005352+0000) 2024-03-06T18:28:18.000 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:18.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:18.005542+0000) 2024-03-06T18:28:19.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:19.004+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:19.005786+0000) 2024-03-06T18:28:20.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:20.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:20.006078+0000) 2024-03-06T18:28:21.001 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:21.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:21.006371+0000) 2024-03-06T18:28:22.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:22.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:22.006613+0000) 2024-03-06T18:28:22.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:22.044 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:23.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:23.005+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:23.006849+0000) 2024-03-06T18:28:24.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:24.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:24.007137+0000) 2024-03-06T18:28:25.002 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:25.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:25.007382+0000) 2024-03-06T18:28:26.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:26.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:26.007589+0000) 2024-03-06T18:28:27.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:27.006+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:27.007816+0000) 2024-03-06T18:28:28.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:28.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:28.008131+0000) 2024-03-06T18:28:29.003 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:29.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:29.008347+0000) 2024-03-06T18:28:30.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:30.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:30.008608+0000) 2024-03-06T18:28:31.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:31.007+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:31.008850+0000) 2024-03-06T18:28:32.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:32.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:32.009163+0000) 2024-03-06T18:28:33.004 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:33.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:33.009442+0000) 2024-03-06T18:28:34.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:34.008+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:34.009736+0000) 2024-03-06T18:28:35.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:35.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:35.010065+0000) 2024-03-06T18:28:36.005 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:36.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:36.010343+0000) 2024-03-06T18:28:37.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:37.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:37.010585+0000) 2024-03-06T18:28:38.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:38.009+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:38.010819+0000) 2024-03-06T18:28:39.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:39.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:39.011101+0000) 2024-03-06T18:28:40.006 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:40.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:40.011309+0000) 2024-03-06T18:28:41.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:41.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:41.011548+0000) 2024-03-06T18:28:42.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:42.010+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:42.011777+0000) 2024-03-06T18:28:43.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:43.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:43.011997+0000) 2024-03-06T18:28:44.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:44.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:44.012267+0000) 2024-03-06T18:28:45.007 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:45.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:45.012478+0000) 2024-03-06T18:28:46.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:46.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:46.012741+0000) 2024-03-06T18:28:47.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:47.011+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:47.012976+0000) 2024-03-06T18:28:48.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:48.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:48.013252+0000) 2024-03-06T18:28:49.008 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:49.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:49.013470+0000) 2024-03-06T18:28:50.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:50.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:50.013662+0000) 2024-03-06T18:28:51.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:51.012+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:51.013871+0000) 2024-03-06T18:28:52.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:52.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:52.014092+0000) 2024-03-06T18:28:52.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:52.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:28:53.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:53.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:53.014355+0000) 2024-03-06T18:28:54.009 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:54.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:54.014554+0000) 2024-03-06T18:28:55.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:55.013+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:55.014820+0000) 2024-03-06T18:28:56.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:56.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:56.015134+0000) 2024-03-06T18:28:57.010 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:57.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:57.015384+0000) 2024-03-06T18:28:58.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:58.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:58.015659+0000) 2024-03-06T18:28:59.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:28:59.014+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:28:59.015917+0000) 2024-03-06T18:29:00.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:00.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:00.016153+0000) 2024-03-06T18:29:01.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:01.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:01.016338+0000) 2024-03-06T18:29:02.011 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:02.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:02.016564+0000) 2024-03-06T18:29:03.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:03.015+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:03.016833+0000) 2024-03-06T18:29:04.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:04.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:04.017114+0000) 2024-03-06T18:29:05.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:05.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:05.017327+0000) 2024-03-06T18:29:06.012 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:06.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:06.017542+0000) 2024-03-06T18:29:07.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:07.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:07.017766+0000) 2024-03-06T18:29:08.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:08.016+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:08.017961+0000) 2024-03-06T18:29:09.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:09.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:09.018209+0000) 2024-03-06T18:29:10.013 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:10.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:10.018430+0000) 2024-03-06T18:29:11.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:11.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:11.018661+0000) 2024-03-06T18:29:12.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:12.017+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:12.018886+0000) 2024-03-06T18:29:13.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:13.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:13.019118+0000) 2024-03-06T18:29:14.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:14.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:14.019360+0000) 2024-03-06T18:29:15.014 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:15.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:15.019601+0000) 2024-03-06T18:29:16.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:16.018+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:16.019873+0000) 2024-03-06T18:29:17.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:17.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:17.020147+0000) 2024-03-06T18:29:18.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:18.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:18.020339+0000) 2024-03-06T18:29:19.015 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:19.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:19.020562+0000) 2024-03-06T18:29:20.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:20.019+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:20.020802+0000) 2024-03-06T18:29:21.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:21.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:21.021077+0000) 2024-03-06T18:29:22.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:22.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:22.021289+0000) 2024-03-06T18:29:22.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:22.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:23.016 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:23.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:23.021506+0000) 2024-03-06T18:29:24.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:24.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:24.021741+0000) 2024-03-06T18:29:25.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:25.020+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:25.021963+0000) 2024-03-06T18:29:26.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:26.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:26.022194+0000) 2024-03-06T18:29:27.017 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:27.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:27.022448+0000) 2024-03-06T18:29:28.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:28.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:28.022757+0000) 2024-03-06T18:29:29.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:29.021+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:29.022989+0000) 2024-03-06T18:29:30.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:30.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:30.023271+0000) 2024-03-06T18:29:31.018 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:31.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:31.023505+0000) 2024-03-06T18:29:32.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:32.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:32.023733+0000) 2024-03-06T18:29:33.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:33.022+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:33.023967+0000) 2024-03-06T18:29:34.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:34.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:34.024243+0000) 2024-03-06T18:29:35.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:35.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:35.024504+0000) 2024-03-06T18:29:36.019 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:36.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:36.024734+0000) 2024-03-06T18:29:37.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:37.023+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:37.024916+0000) 2024-03-06T18:29:38.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:38.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:38.025171+0000) 2024-03-06T18:29:39.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:39.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:39.025416+0000) 2024-03-06T18:29:40.020 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:40.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:40.025606+0000) 2024-03-06T18:29:41.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:41.024+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:41.025914+0000) 2024-03-06T18:29:42.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:42.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:42.026159+0000) 2024-03-06T18:29:43.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:43.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:43.026397+0000) 2024-03-06T18:29:44.021 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:44.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:44.026657+0000) 2024-03-06T18:29:45.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:45.025+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:45.026911+0000) 2024-03-06T18:29:46.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:46.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:46.027162+0000) 2024-03-06T18:29:47.022 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:47.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:47.027452+0000) 2024-03-06T18:29:48.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:48.026+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:48.027745+0000) 2024-03-06T18:29:49.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:49.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:49.028049+0000) 2024-03-06T18:29:50.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:50.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:50.028353+0000) 2024-03-06T18:29:51.023 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:51.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:51.028577+0000) 2024-03-06T18:29:52.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:52.027+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:52.028860+0000) 2024-03-06T18:29:52.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:52.598 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:29:53.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:53.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:53.029182+0000) 2024-03-06T18:29:54.024 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:54.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:54.029482+0000) 2024-03-06T18:29:55.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:55.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:55.029717+0000) 2024-03-06T18:29:56.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:56.028+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:56.029961+0000) 2024-03-06T18:29:57.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:57.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:57.030205+0000) 2024-03-06T18:29:58.025 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:58.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:58.030425+0000) 2024-03-06T18:29:59.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:29:59.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:29:59.030699+0000) 2024-03-06T18:30:00.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:00.029+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:00.030970+0000) 2024-03-06T18:30:01.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:01.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:01.031306+0000) 2024-03-06T18:30:02.026 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:02.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:02.031578+0000) 2024-03-06T18:30:03.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:03.030+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:03.031870+0000) 2024-03-06T18:30:04.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:04.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:04.032154+0000) 2024-03-06T18:30:05.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:05.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:05.032366+0000) 2024-03-06T18:30:06.027 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:06.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:06.032582+0000) 2024-03-06T18:30:07.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:07.031+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:07.032850+0000) 2024-03-06T18:30:08.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:08.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:08.033163+0000) 2024-03-06T18:30:09.028 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:09.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:09.033386+0000) 2024-03-06T18:30:10.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:10.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:10.033598+0000) 2024-03-06T18:30:11.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:11.032+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:11.033825+0000) 2024-03-06T18:30:12.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:12.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:12.034082+0000) 2024-03-06T18:30:13.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:13.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:13.034311+0000) 2024-03-06T18:30:14.029 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:14.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:14.034604+0000) 2024-03-06T18:30:15.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:15.033+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:15.034888+0000) 2024-03-06T18:30:16.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:16.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:16.035162+0000) 2024-03-06T18:30:17.030 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:17.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:17.035448+0000) 2024-03-06T18:30:18.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:18.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:18.035719+0000) 2024-03-06T18:30:19.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:19.034+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:19.035986+0000) 2024-03-06T18:30:20.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:20.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:20.036273+0000) 2024-03-06T18:30:21.031 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:21.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:21.036526+0000) 2024-03-06T18:30:22.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:22.035+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:22.036757+0000) 2024-03-06T18:30:22.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:22.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:23.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:23.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:23.037001+0000) 2024-03-06T18:30:24.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:24.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:24.037386+0000) 2024-03-06T18:30:25.032 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:25.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:25.037634+0000) 2024-03-06T18:30:26.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:26.036+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:26.037902+0000) 2024-03-06T18:30:27.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:27.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:27.038192+0000) 2024-03-06T18:30:28.033 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:28.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:28.038499+0000) 2024-03-06T18:30:29.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:29.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:29.038760+0000) 2024-03-06T18:30:30.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:30.037+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:30.038990+0000) 2024-03-06T18:30:31.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:31.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:31.039244+0000) 2024-03-06T18:30:32.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:32.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:32.039479+0000) 2024-03-06T18:30:33.034 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:33.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:33.039695+0000) 2024-03-06T18:30:34.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:34.038+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:34.039917+0000) 2024-03-06T18:30:35.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:35.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:35.040180+0000) 2024-03-06T18:30:36.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:36.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:36.040407+0000) 2024-03-06T18:30:37.035 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:37.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:37.040648+0000) 2024-03-06T18:30:38.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:38.039+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:38.040889+0000) 2024-03-06T18:30:39.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:39.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:39.041165+0000) 2024-03-06T18:30:40.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:40.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:40.041413+0000) 2024-03-06T18:30:41.036 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:41.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:41.041669+0000) 2024-03-06T18:30:42.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:42.040+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:42.041953+0000) 2024-03-06T18:30:43.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:43.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:43.042160+0000) 2024-03-06T18:30:44.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:44.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:44.042386+0000) 2024-03-06T18:30:45.037 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:45.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:45.042609+0000) 2024-03-06T18:30:46.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:46.041+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:46.042800+0000) 2024-03-06T18:30:47.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:47.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:47.043089+0000) 2024-03-06T18:30:48.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:48.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:48.043319+0000) 2024-03-06T18:30:49.038 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:49.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:49.043553+0000) 2024-03-06T18:30:50.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:50.042+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:50.043785+0000) 2024-03-06T18:30:51.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:51.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:51.044047+0000) 2024-03-06T18:30:52.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:52.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:52.044284+0000) 2024-03-06T18:30:53.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:53.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:53.044496+0000) 2024-03-06T18:30:53.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:53.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:30:54.039 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:54.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:54.044729+0000) 2024-03-06T18:30:55.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:55.043+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:55.044984+0000) 2024-03-06T18:30:56.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:56.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:56.045323+0000) 2024-03-06T18:30:57.040 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:57.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:57.045607+0000) 2024-03-06T18:30:58.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:58.044+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:58.045865+0000) 2024-03-06T18:30:59.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:30:59.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:30:59.046153+0000) 2024-03-06T18:31:00.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:00.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:00.046425+0000) 2024-03-06T18:31:01.041 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:01.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:01.046709+0000) 2024-03-06T18:31:02.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:02.045+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:02.046979+0000) 2024-03-06T18:31:03.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:03.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:03.047281+0000) 2024-03-06T18:31:04.042 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:04.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:04.047576+0000) 2024-03-06T18:31:05.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:05.046+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:05.047855+0000) 2024-03-06T18:31:06.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:06.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:06.048151+0000) 2024-03-06T18:31:07.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:07.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:07.048327+0000) 2024-03-06T18:31:08.043 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:08.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:08.048561+0000) 2024-03-06T18:31:09.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:09.047+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:09.048798+0000) 2024-03-06T18:31:10.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:10.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:10.049077+0000) 2024-03-06T18:31:11.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:11.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:11.049308+0000) 2024-03-06T18:31:12.044 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:12.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:12.049593+0000) 2024-03-06T18:31:13.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:13.048+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:13.049817+0000) 2024-03-06T18:31:14.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:14.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:14.050056+0000) 2024-03-06T18:31:15.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:15.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:15.050257+0000) 2024-03-06T18:31:16.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:16.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:16.050472+0000) 2024-03-06T18:31:17.045 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:17.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:17.050669+0000) 2024-03-06T18:31:18.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:18.049+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:18.050852+0000) 2024-03-06T18:31:19.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:19.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:19.051103+0000) 2024-03-06T18:31:20.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:20.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:20.051332+0000) 2024-03-06T18:31:21.046 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:21.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:21.051560+0000) 2024-03-06T18:31:22.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:22.050+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:22.051792+0000) 2024-03-06T18:31:23.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:23.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:23.052047+0000) 2024-03-06T18:31:23.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:23.320 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:24.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:24.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:24.052308+0000) 2024-03-06T18:31:25.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:25.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:25.052528+0000) 2024-03-06T18:31:26.047 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:26.051+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:26.052780+0000) 2024-03-06T18:31:27.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:27.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:27.053039+0000) 2024-03-06T18:31:28.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:28.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:28.053291+0000) 2024-03-06T18:31:29.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:29.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:29.053513+0000) 2024-03-06T18:31:30.048 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:30.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:30.053746+0000) 2024-03-06T18:31:31.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:31.052+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:31.053938+0000) 2024-03-06T18:31:32.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:32.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:32.054176+0000) 2024-03-06T18:31:33.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:33.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:33.054414+0000) 2024-03-06T18:31:34.049 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:34.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:34.054624+0000) 2024-03-06T18:31:35.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:35.053+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:35.054851+0000) 2024-03-06T18:31:36.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:36.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:36.055117+0000) 2024-03-06T18:31:37.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:37.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:37.055335+0000) 2024-03-06T18:31:38.050 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:38.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:38.055637+0000) 2024-03-06T18:31:39.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:39.054+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:39.055919+0000) 2024-03-06T18:31:40.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:40.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:40.056170+0000) 2024-03-06T18:31:41.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:41.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:41.056401+0000) 2024-03-06T18:31:42.051 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:42.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:42.056624+0000) 2024-03-06T18:31:43.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:43.055+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:43.056857+0000) 2024-03-06T18:31:44.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:44.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:44.057155+0000) 2024-03-06T18:31:45.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:45.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:45.057395+0000) 2024-03-06T18:31:46.052 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:46.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:46.057647+0000) 2024-03-06T18:31:47.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:47.056+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:47.057878+0000) 2024-03-06T18:31:48.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:48.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:48.058154+0000) 2024-03-06T18:31:49.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:49.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:49.058400+0000) 2024-03-06T18:31:50.053 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:50.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:50.058621+0000) 2024-03-06T18:31:51.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:51.057+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:51.058845+0000) 2024-03-06T18:31:52.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:52.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:52.059129+0000) 2024-03-06T18:31:53.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:53.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:53.059365+0000) 2024-03-06T18:31:53.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:53.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:31:54.054 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:54.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:54.059604+0000) 2024-03-06T18:31:55.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:55.058+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:55.059838+0000) 2024-03-06T18:31:56.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:56.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:56.060116+0000) 2024-03-06T18:31:57.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:57.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:57.060364+0000) 2024-03-06T18:31:58.055 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:58.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:58.060660+0000) 2024-03-06T18:31:59.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:31:59.059+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:31:59.060896+0000) 2024-03-06T18:32:00.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:00.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:00.061130+0000) 2024-03-06T18:32:01.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:01.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:01.061326+0000) 2024-03-06T18:32:02.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:02.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:02.061533+0000) 2024-03-06T18:32:03.056 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:03.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:03.061754+0000) 2024-03-06T18:32:04.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:04.060+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:04.061949+0000) 2024-03-06T18:32:05.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:05.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:05.062196+0000) 2024-03-06T18:32:06.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:06.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:06.062419+0000) 2024-03-06T18:32:07.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:07.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:07.062650+0000) 2024-03-06T18:32:08.057 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:08.061+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:08.062886+0000) 2024-03-06T18:32:09.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:09.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:09.063109+0000) 2024-03-06T18:32:10.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:10.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:10.063346+0000) 2024-03-06T18:32:11.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:11.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:11.063616+0000) 2024-03-06T18:32:12.058 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:12.062+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:12.063900+0000) 2024-03-06T18:32:13.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:13.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:13.064171+0000) 2024-03-06T18:32:14.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:14.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:14.064387+0000) 2024-03-06T18:32:15.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:15.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:15.064610+0000) 2024-03-06T18:32:16.059 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:16.063+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:16.064840+0000) 2024-03-06T18:32:17.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:17.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:17.065076+0000) 2024-03-06T18:32:18.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:18.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:18.065289+0000) 2024-03-06T18:32:19.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:19.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:19.065503+0000) 2024-03-06T18:32:20.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:20.064+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:20.065740+0000) 2024-03-06T18:32:21.060 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:21.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:21.065989+0000) 2024-03-06T18:32:22.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:22.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:22.066238+0000) 2024-03-06T18:32:23.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:23.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:23.066457+0000) 2024-03-06T18:32:23.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:23.728 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:24.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:24.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:24.066688+0000) 2024-03-06T18:32:25.061 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:25.065+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:25.066943+0000) 2024-03-06T18:32:26.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:26.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:26.067172+0000) 2024-03-06T18:32:27.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:27.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:27.067414+0000) 2024-03-06T18:32:28.062 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:28.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:28.067671+0000) 2024-03-06T18:32:29.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:29.066+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:29.067922+0000) 2024-03-06T18:32:30.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:30.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:30.068171+0000) 2024-03-06T18:32:31.063 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:31.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:31.068414+0000) 2024-03-06T18:32:32.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:32.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:32.068659+0000) 2024-03-06T18:32:33.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:33.067+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:33.068895+0000) 2024-03-06T18:32:34.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:34.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:34.069137+0000) 2024-03-06T18:32:35.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:35.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:35.069344+0000) 2024-03-06T18:32:36.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:36.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:36.069587+0000) 2024-03-06T18:32:37.064 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:37.068+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:37.069844+0000) 2024-03-06T18:32:38.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:38.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:38.070123+0000) 2024-03-06T18:32:39.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:39.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:39.070357+0000) 2024-03-06T18:32:40.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:40.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:40.070606+0000) 2024-03-06T18:32:41.065 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:41.069+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:41.070841+0000) 2024-03-06T18:32:42.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:42.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:42.071112+0000) 2024-03-06T18:32:43.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:43.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:43.071345+0000) 2024-03-06T18:32:44.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:44.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:44.071581+0000) 2024-03-06T18:32:45.066 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:45.070+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:45.071816+0000) 2024-03-06T18:32:46.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:46.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:46.071998+0000) 2024-03-06T18:32:47.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:47.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:47.072218+0000) 2024-03-06T18:32:48.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:48.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:48.072420+0000) 2024-03-06T18:32:49.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:49.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:49.072622+0000) 2024-03-06T18:32:50.067 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:50.071+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:50.072859+0000) 2024-03-06T18:32:51.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:51.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:51.073121+0000) 2024-03-06T18:32:52.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:52.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:52.073313+0000) 2024-03-06T18:32:53.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:53.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:53.073535+0000) 2024-03-06T18:32:53.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:53.923 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:32:54.068 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:54.072+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:54.073762+0000) 2024-03-06T18:32:55.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:55.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:55.074034+0000) 2024-03-06T18:32:56.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:56.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:56.074268+0000) 2024-03-06T18:32:57.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:57.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:57.074462+0000) 2024-03-06T18:32:58.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:58.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:58.074755+0000) 2024-03-06T18:32:59.069 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:32:59.073+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:32:59.074974+0000) 2024-03-06T18:33:00.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:00.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:00.075270+0000) 2024-03-06T18:33:01.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:01.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:01.075531+0000) 2024-03-06T18:33:02.070 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:02.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:02.075759+0000) 2024-03-06T18:33:03.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:03.074+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:03.075987+0000) 2024-03-06T18:33:04.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:04.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:04.076288+0000) 2024-03-06T18:33:05.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:05.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:05.076548+0000) 2024-03-06T18:33:06.071 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:06.075+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:06.076789+0000) 2024-03-06T18:33:07.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:07.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:07.077089+0000) 2024-03-06T18:33:08.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:08.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:08.077355+0000) 2024-03-06T18:33:09.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:09.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:09.077564+0000) 2024-03-06T18:33:10.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:10.076+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:10.077793+0000) 2024-03-06T18:33:11.072 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:11.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:11.078037+0000) 2024-03-06T18:33:12.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:12.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:12.078266+0000) 2024-03-06T18:33:13.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:13.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:13.078541+0000) 2024-03-06T18:33:14.073 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:14.077+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:14.078741+0000) 2024-03-06T18:33:15.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:15.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:15.079000+0000) 2024-03-06T18:33:16.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:16.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:16.079295+0000) 2024-03-06T18:33:17.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:17.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:17.079546+0000) 2024-03-06T18:33:18.074 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:18.078+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:18.079805+0000) 2024-03-06T18:33:19.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:19.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:19.080081+0000) 2024-03-06T18:33:20.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:20.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:20.080301+0000) 2024-03-06T18:33:21.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:21.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:21.080553+0000) 2024-03-06T18:33:22.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:22.079+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:22.080796+0000) 2024-03-06T18:33:23.075 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:23.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:23.081075+0000) 2024-03-06T18:33:24.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:24.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:24.081319+0000) 2024-03-06T18:33:24.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:24.136 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:25.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:25.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:25.081576+0000) 2024-03-06T18:33:26.076 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:26.080+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:26.081825+0000) 2024-03-06T18:33:27.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:27.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:27.082117+0000) 2024-03-06T18:33:28.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:28.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:28.082378+0000) 2024-03-06T18:33:29.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:29.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:29.082621+0000) 2024-03-06T18:33:30.077 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:30.081+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:30.082868+0000) 2024-03-06T18:33:31.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:31.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:31.083115+0000) 2024-03-06T18:33:32.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:32.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:32.083367+0000) 2024-03-06T18:33:33.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:33.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:33.083624+0000) 2024-03-06T18:33:34.078 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:34.082+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:34.083847+0000) 2024-03-06T18:33:35.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:35.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:35.084135+0000) 2024-03-06T18:33:36.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:36.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:36.084376+0000) 2024-03-06T18:33:37.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:37.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:37.084617+0000) 2024-03-06T18:33:38.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:38.083+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:38.084876+0000) 2024-03-06T18:33:39.079 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:39.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:39.085113+0000) 2024-03-06T18:33:40.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:40.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:40.085363+0000) 2024-03-06T18:33:41.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:41.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:41.085626+0000) 2024-03-06T18:33:42.080 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:42.084+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:42.085847+0000) 2024-03-06T18:33:43.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:43.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:43.086070+0000) 2024-03-06T18:33:44.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:44.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:44.086262+0000) 2024-03-06T18:33:45.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:45.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:45.086477+0000) 2024-03-06T18:33:46.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:46.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:46.086721+0000) 2024-03-06T18:33:47.081 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:47.085+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:47.086943+0000) 2024-03-06T18:33:48.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:48.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:48.087158+0000) 2024-03-06T18:33:49.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:49.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:49.087393+0000) 2024-03-06T18:33:50.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:50.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:50.087612+0000) 2024-03-06T18:33:51.082 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:51.086+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:51.087836+0000) 2024-03-06T18:33:52.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:52.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:52.088135+0000) 2024-03-06T18:33:53.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:53.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:53.088365+0000) 2024-03-06T18:33:54.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:54.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:54.088585+0000) 2024-03-06T18:33:54.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:54.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:33:55.083 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:55.087+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:55.088850+0000) 2024-03-06T18:33:56.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:56.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:56.089135+0000) 2024-03-06T18:33:57.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:57.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:57.089360+0000) 2024-03-06T18:33:58.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:58.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:58.089582+0000) 2024-03-06T18:33:59.084 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:33:59.088+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:33:59.089815+0000) 2024-03-06T18:34:00.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:00.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:00.090082+0000) 2024-03-06T18:34:01.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:01.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:01.090278+0000) 2024-03-06T18:34:02.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:02.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:02.090509+0000) 2024-03-06T18:34:03.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:03.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:03.090706+0000) 2024-03-06T18:34:04.085 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:04.089+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:04.090960+0000) 2024-03-06T18:34:05.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:05.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:05.091258+0000) 2024-03-06T18:34:06.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:06.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:06.091505+0000) 2024-03-06T18:34:07.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:07.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:07.091705+0000) 2024-03-06T18:34:08.086 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:08.090+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:08.091967+0000) 2024-03-06T18:34:09.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:09.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:09.092271+0000) 2024-03-06T18:34:10.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:10.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:10.092547+0000) 2024-03-06T18:34:11.087 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:11.091+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:11.092793+0000) 2024-03-06T18:34:12.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:12.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:12.093075+0000) 2024-03-06T18:34:13.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:13.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:13.093336+0000) 2024-03-06T18:34:14.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:14.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:14.093568+0000) 2024-03-06T18:34:15.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:15.092+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:15.093820+0000) 2024-03-06T18:34:16.088 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:16.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:16.094036+0000) 2024-03-06T18:34:17.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:17.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:17.094237+0000) 2024-03-06T18:34:18.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:18.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:18.094444+0000) 2024-03-06T18:34:19.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:19.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:19.094651+0000) 2024-03-06T18:34:20.089 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:20.093+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:20.094873+0000) 2024-03-06T18:34:21.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:21.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:21.095133+0000) 2024-03-06T18:34:22.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:22.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:22.095344+0000) 2024-03-06T18:34:23.090 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:23.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:23.095566+0000) 2024-03-06T18:34:24.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:24.094+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:24.095807+0000) 2024-03-06T18:34:24.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:24.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:25.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:25.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:25.096089+0000) 2024-03-06T18:34:26.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:26.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:26.096303+0000) 2024-03-06T18:34:27.091 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:27.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:27.096542+0000) 2024-03-06T18:34:28.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:28.095+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:28.096822+0000) 2024-03-06T18:34:29.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:29.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:29.097143+0000) 2024-03-06T18:34:30.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:30.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:30.097378+0000) 2024-03-06T18:34:31.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:31.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:31.097624+0000) 2024-03-06T18:34:32.092 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:32.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:32.097831+0000) 2024-03-06T18:34:33.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:33.096+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:33.097984+0000) 2024-03-06T18:34:34.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:34.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:34.098293+0000) 2024-03-06T18:34:35.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:35.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:35.098521+0000) 2024-03-06T18:34:36.093 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:36.097+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:36.098762+0000) 2024-03-06T18:34:37.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:37.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:37.098994+0000) 2024-03-06T18:34:38.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:38.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:38.099247+0000) 2024-03-06T18:34:39.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:39.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:39.099470+0000) 2024-03-06T18:34:40.094 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:40.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:40.099687+0000) 2024-03-06T18:34:41.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:41.098+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:41.099918+0000) 2024-03-06T18:34:42.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:42.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:42.100160+0000) 2024-03-06T18:34:43.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:43.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:43.100369+0000) 2024-03-06T18:34:44.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:44.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:44.100590+0000) 2024-03-06T18:34:45.095 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:45.099+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:45.100850+0000) 2024-03-06T18:34:46.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:46.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:46.101126+0000) 2024-03-06T18:34:47.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:47.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:47.101323+0000) 2024-03-06T18:34:48.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:48.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:48.101535+0000) 2024-03-06T18:34:49.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:49.100+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:49.101756+0000) 2024-03-06T18:34:50.096 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:50.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:50.101999+0000) 2024-03-06T18:34:51.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:51.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:51.102234+0000) 2024-03-06T18:34:52.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:52.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:52.102437+0000) 2024-03-06T18:34:53.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:53.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:53.102696+0000) 2024-03-06T18:34:54.097 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:54.101+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:54.102902+0000) 2024-03-06T18:34:54.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:54.665 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:34:55.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:55.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:55.103171+0000) 2024-03-06T18:34:56.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:56.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:56.103388+0000) 2024-03-06T18:34:57.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:57.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:57.103595+0000) 2024-03-06T18:34:58.098 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:58.102+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:58.103806+0000) 2024-03-06T18:34:59.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:34:59.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:34:59.104050+0000) 2024-03-06T18:35:00.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:00.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:00.104263+0000) 2024-03-06T18:35:01.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:01.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:01.104518+0000) 2024-03-06T18:35:02.099 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:02.103+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:02.104764+0000) 2024-03-06T18:35:03.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:03.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:03.105044+0000) 2024-03-06T18:35:04.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:04.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:04.105284+0000) 2024-03-06T18:35:05.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:05.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:05.105532+0000) 2024-03-06T18:35:06.100 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:06.104+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:06.105787+0000) 2024-03-06T18:35:07.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:07.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:07.106064+0000) 2024-03-06T18:35:08.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:08.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:08.106324+0000) 2024-03-06T18:35:09.101 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:09.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:09.106574+0000) 2024-03-06T18:35:10.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:10.105+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:10.106848+0000) 2024-03-06T18:35:11.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:11.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:11.107119+0000) 2024-03-06T18:35:12.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:12.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:12.107332+0000) 2024-03-06T18:35:13.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:13.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:13.107608+0000) 2024-03-06T18:35:14.102 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:14.106+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:14.107853+0000) 2024-03-06T18:35:15.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:15.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:15.108105+0000) 2024-03-06T18:35:16.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:16.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:16.108351+0000) 2024-03-06T18:35:17.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:17.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:17.108608+0000) 2024-03-06T18:35:18.103 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:18.107+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:18.108859+0000) 2024-03-06T18:35:19.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:19.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:19.109095+0000) 2024-03-06T18:35:20.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:20.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:20.109317+0000) 2024-03-06T18:35:21.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:21.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:21.109579+0000) 2024-03-06T18:35:22.104 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:22.108+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:22.109829+0000) 2024-03-06T18:35:23.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:23.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:23.110103+0000) 2024-03-06T18:35:24.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:24.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:24.110347+0000) 2024-03-06T18:35:24.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:24.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:25.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:25.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:25.110571+0000) 2024-03-06T18:35:26.105 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:26.109+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:26.110812+0000) 2024-03-06T18:35:27.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:27.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:27.111049+0000) 2024-03-06T18:35:28.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:28.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:28.111288+0000) 2024-03-06T18:35:29.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:29.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:29.111466+0000) 2024-03-06T18:35:30.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:30.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:30.111673+0000) 2024-03-06T18:35:31.106 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:31.110+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:31.111862+0000) 2024-03-06T18:35:32.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:32.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:32.112101+0000) 2024-03-06T18:35:33.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:33.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:33.112301+0000) 2024-03-06T18:35:34.107 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:34.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:34.112556+0000) 2024-03-06T18:35:35.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:35.111+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:35.112803+0000) 2024-03-06T18:35:36.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:36.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:36.113078+0000) 2024-03-06T18:35:37.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:37.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:37.113292+0000) 2024-03-06T18:35:38.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:38.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:38.113530+0000) 2024-03-06T18:35:39.108 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:39.112+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:39.113756+0000) 2024-03-06T18:35:40.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:40.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:40.114057+0000) 2024-03-06T18:35:41.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:41.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:41.114318+0000) 2024-03-06T18:35:42.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:42.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:42.114546+0000) 2024-03-06T18:35:43.109 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:43.113+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:43.114773+0000) 2024-03-06T18:35:44.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:44.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:44.115059+0000) 2024-03-06T18:35:45.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:45.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:45.115305+0000) 2024-03-06T18:35:46.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:46.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:46.115551+0000) 2024-03-06T18:35:47.110 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:47.114+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:47.115784+0000) 2024-03-06T18:35:48.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:48.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:48.116038+0000) 2024-03-06T18:35:49.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:49.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:49.116276+0000) 2024-03-06T18:35:50.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:50.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:50.116481+0000) 2024-03-06T18:35:51.111 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:51.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:51.116723+0000) 2024-03-06T18:35:52.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:52.115+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:52.116894+0000) 2024-03-06T18:35:53.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:53.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:53.117135+0000) 2024-03-06T18:35:54.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:54.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:54.117361+0000) 2024-03-06T18:35:54.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:54.939 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:35:55.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:55.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:55.117612+0000) 2024-03-06T18:35:56.112 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:56.116+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:56.117795+0000) 2024-03-06T18:35:57.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:57.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:57.118070+0000) 2024-03-06T18:35:58.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:58.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:58.118298+0000) 2024-03-06T18:35:59.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:35:59.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:35:59.118518+0000) 2024-03-06T18:36:00.113 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:00.117+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:00.118788+0000) 2024-03-06T18:36:01.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:01.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:01.119057+0000) 2024-03-06T18:36:02.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:02.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:02.119259+0000) 2024-03-06T18:36:03.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:03.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:03.119486+0000) 2024-03-06T18:36:04.114 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:04.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:04.119712+0000) 2024-03-06T18:36:05.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:05.118+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:05.119983+0000) 2024-03-06T18:36:06.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:06.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:06.120281+0000) 2024-03-06T18:36:07.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:07.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:07.120527+0000) 2024-03-06T18:36:08.115 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:08.119+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:08.120777+0000) 2024-03-06T18:36:09.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:09.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:09.121049+0000) 2024-03-06T18:36:10.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:10.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:10.121258+0000) 2024-03-06T18:36:11.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:11.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:11.121496+0000) 2024-03-06T18:36:12.116 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:12.120+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:12.121732+0000) 2024-03-06T18:36:13.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:13.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:13.121997+0000) 2024-03-06T18:36:14.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:14.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:14.122259+0000) 2024-03-06T18:36:15.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:15.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:15.122483+0000) 2024-03-06T18:36:16.117 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:16.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:16.122726+0000) 2024-03-06T18:36:17.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:17.121+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:17.122966+0000) 2024-03-06T18:36:18.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:18.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:18.123265+0000) 2024-03-06T18:36:19.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:19.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:19.123498+0000) 2024-03-06T18:36:20.118 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:20.122+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:20.123726+0000) 2024-03-06T18:36:21.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:21.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:21.124051+0000) 2024-03-06T18:36:22.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:22.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:22.124247+0000) 2024-03-06T18:36:23.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:23.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:23.124457+0000) 2024-03-06T18:36:24.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:24.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:24.124678+0000) 2024-03-06T18:36:25.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:25.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:25.119 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:25.123+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:25.124855+0000) 2024-03-06T18:36:26.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:26.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:26.125109+0000) 2024-03-06T18:36:27.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:27.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:27.125336+0000) 2024-03-06T18:36:28.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:28.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:28.125543+0000) 2024-03-06T18:36:29.120 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:29.124+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:29.125774+0000) 2024-03-06T18:36:30.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:30.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:30.126063+0000) 2024-03-06T18:36:31.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:31.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:31.126333+0000) 2024-03-06T18:36:32.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:32.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:32.126582+0000) 2024-03-06T18:36:33.121 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:33.125+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:33.126822+0000) 2024-03-06T18:36:34.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:34.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:34.127060+0000) 2024-03-06T18:36:35.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:35.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:35.127269+0000) 2024-03-06T18:36:36.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:36.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:36.127503+0000) 2024-03-06T18:36:37.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:37.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:37.127717+0000) 2024-03-06T18:36:38.122 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:38.126+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:38.127912+0000) 2024-03-06T18:36:39.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:39.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:39.128135+0000) 2024-03-06T18:36:40.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:40.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:40.128407+0000) 2024-03-06T18:36:41.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:41.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:41.128638+0000) 2024-03-06T18:36:42.123 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:42.127+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:42.128872+0000) 2024-03-06T18:36:43.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:43.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:43.129129+0000) 2024-03-06T18:36:44.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:44.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:44.129328+0000) 2024-03-06T18:36:45.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:45.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:45.129562+0000) 2024-03-06T18:36:46.124 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:46.128+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:46.129793+0000) 2024-03-06T18:36:47.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:47.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:47.130062+0000) 2024-03-06T18:36:48.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:48.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:48.130277+0000) 2024-03-06T18:36:49.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:49.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:49.130501+0000) 2024-03-06T18:36:50.125 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:50.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:50.130733+0000) 2024-03-06T18:36:51.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:51.129+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:51.130964+0000) 2024-03-06T18:36:52.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:52.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:52.131173+0000) 2024-03-06T18:36:53.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:53.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:53.131351+0000) 2024-03-06T18:36:54.126 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:54.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:54.131580+0000) 2024-03-06T18:36:55.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:55.130+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:55.131815+0000) 2024-03-06T18:36:55.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:55.206 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:36:56.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:56.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:56.132074+0000) 2024-03-06T18:36:57.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:57.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:57.132310+0000) 2024-03-06T18:36:58.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:58.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:58.132522+0000) 2024-03-06T18:36:59.127 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:36:59.131+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:36:59.132761+0000) 2024-03-06T18:37:00.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:00.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:00.133046+0000) 2024-03-06T18:37:01.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:01.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:01.133307+0000) 2024-03-06T18:37:02.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:02.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:02.133555+0000) 2024-03-06T18:37:03.128 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:03.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:03.133793+0000) 2024-03-06T18:37:04.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:04.132+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:04.133985+0000) 2024-03-06T18:37:05.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:05.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:05.134242+0000) 2024-03-06T18:37:06.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:06.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:06.134459+0000) 2024-03-06T18:37:07.129 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:07.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:07.134683+0000) 2024-03-06T18:37:08.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:08.133+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:08.134910+0000) 2024-03-06T18:37:09.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:09.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:09.135215+0000) 2024-03-06T18:37:10.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:10.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:10.135368+0000) 2024-03-06T18:37:11.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:11.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:11.135618+0000) 2024-03-06T18:37:12.130 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:12.134+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:12.135862+0000) 2024-03-06T18:37:13.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:13.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:13.136136+0000) 2024-03-06T18:37:14.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:14.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:14.136378+0000) 2024-03-06T18:37:15.131 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:15.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:15.136624+0000) 2024-03-06T18:37:16.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:16.135+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:16.136894+0000) 2024-03-06T18:37:17.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:17.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:17.137143+0000) 2024-03-06T18:37:18.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:18.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:18.137399+0000) 2024-03-06T18:37:19.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:19.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:19.137638+0000) 2024-03-06T18:37:20.132 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:20.136+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:20.137880+0000) 2024-03-06T18:37:21.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:21.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:21.138125+0000) 2024-03-06T18:37:22.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:22.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:22.138317+0000) 2024-03-06T18:37:23.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:23.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:23.138525+0000) 2024-03-06T18:37:24.133 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:24.137+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:24.138753+0000) 2024-03-06T18:37:25.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:25.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:25.139078+0000) 2024-03-06T18:37:25.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:25.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:26.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:26.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:26.139330+0000) 2024-03-06T18:37:27.134 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:27.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:27.139617+0000) 2024-03-06T18:37:28.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:28.138+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:28.139910+0000) 2024-03-06T18:37:29.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:29.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:29.140186+0000) 2024-03-06T18:37:30.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:30.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:30.140424+0000) 2024-03-06T18:37:31.135 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:31.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:31.140691+0000) 2024-03-06T18:37:32.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:32.139+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:32.140941+0000) 2024-03-06T18:37:33.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:33.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:33.141197+0000) 2024-03-06T18:37:34.136 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:34.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:34.141430+0000) 2024-03-06T18:37:35.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:35.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:35.141668+0000) 2024-03-06T18:37:36.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:36.140+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:36.141874+0000) 2024-03-06T18:37:37.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:37.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:37.142172+0000) 2024-03-06T18:37:38.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:38.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:38.142435+0000) 2024-03-06T18:37:39.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:39.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:39.142649+0000) 2024-03-06T18:37:40.137 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:40.141+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:40.142876+0000) 2024-03-06T18:37:41.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:41.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:41.143109+0000) 2024-03-06T18:37:42.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:42.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:42.143356+0000) 2024-03-06T18:37:43.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:43.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:43.143601+0000) 2024-03-06T18:37:44.138 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:44.142+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:44.143839+0000) 2024-03-06T18:37:45.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:45.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:45.144112+0000) 2024-03-06T18:37:46.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:46.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:46.144324+0000) 2024-03-06T18:37:47.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:47.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:47.144524+0000) 2024-03-06T18:37:48.139 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:48.143+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:48.144769+0000) 2024-03-06T18:37:49.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:49.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:49.144995+0000) 2024-03-06T18:37:50.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:50.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:50.145268+0000) 2024-03-06T18:37:51.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:51.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:51.145529+0000) 2024-03-06T18:37:52.140 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:52.144+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:52.145772+0000) 2024-03-06T18:37:53.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:53.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:53.145997+0000) 2024-03-06T18:37:54.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:54.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:54.146223+0000) 2024-03-06T18:37:55.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:55.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:55.146453+0000) 2024-03-06T18:37:55.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:55.640 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:37:56.141 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:56.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:56.146665+0000) 2024-03-06T18:37:57.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:57.145+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:57.146958+0000) 2024-03-06T18:37:58.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:58.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:58.147214+0000) 2024-03-06T18:37:59.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:37:59.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:37:59.147442+0000) 2024-03-06T18:38:00.142 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:00.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:00.147670+0000) 2024-03-06T18:38:01.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:01.146+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:01.147903+0000) 2024-03-06T18:38:02.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:02.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:02.148135+0000) 2024-03-06T18:38:03.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:03.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:03.148332+0000) 2024-03-06T18:38:04.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:04.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:04.148550+0000) 2024-03-06T18:38:05.143 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:05.147+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:05.148779+0000) 2024-03-06T18:38:06.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:06.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:06.149034+0000) 2024-03-06T18:38:07.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:07.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:07.149207+0000) 2024-03-06T18:38:08.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:08.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:08.149410+0000) 2024-03-06T18:38:09.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:09.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:09.149623+0000) 2024-03-06T18:38:10.144 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:10.148+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:10.149840+0000) 2024-03-06T18:38:11.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:11.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:11.150132+0000) 2024-03-06T18:38:12.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:12.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:12.150380+0000) 2024-03-06T18:38:13.145 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:13.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:13.150654+0000) 2024-03-06T18:38:14.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:14.149+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:14.150890+0000) 2024-03-06T18:38:15.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:15.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:15.151114+0000) 2024-03-06T18:38:16.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:16.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:16.151307+0000) 2024-03-06T18:38:17.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:17.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:17.151542+0000) 2024-03-06T18:38:18.146 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:18.150+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:18.151805+0000) 2024-03-06T18:38:19.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:19.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:19.152061+0000) 2024-03-06T18:38:20.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:20.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:20.152265+0000) 2024-03-06T18:38:21.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:21.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:21.152532+0000) 2024-03-06T18:38:22.147 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:22.151+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:22.152811+0000) 2024-03-06T18:38:23.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:23.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:23.153127+0000) 2024-03-06T18:38:24.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:24.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:24.153334+0000) 2024-03-06T18:38:25.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:25.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:25.153538+0000) 2024-03-06T18:38:25.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:25.883 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:26.148 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:26.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:26.153735+0000) 2024-03-06T18:38:27.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:27.152+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:27.153963+0000) 2024-03-06T18:38:28.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:28.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:28.154205+0000) 2024-03-06T18:38:29.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:29.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:29.154401+0000) 2024-03-06T18:38:30.149 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:30.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:30.154626+0000) 2024-03-06T18:38:31.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:31.153+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:31.154861+0000) 2024-03-06T18:38:32.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:32.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:32.155128+0000) 2024-03-06T18:38:33.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:33.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:33.155377+0000) 2024-03-06T18:38:34.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:34.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:34.155633+0000) 2024-03-06T18:38:35.150 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:35.154+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:35.155859+0000) 2024-03-06T18:38:36.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:36.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:36.156135+0000) 2024-03-06T18:38:37.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:37.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:37.156333+0000) 2024-03-06T18:38:38.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:38.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:38.156568+0000) 2024-03-06T18:38:39.151 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:39.155+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:39.156765+0000) 2024-03-06T18:38:40.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:40.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:40.157037+0000) 2024-03-06T18:38:41.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:41.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:41.157280+0000) 2024-03-06T18:38:42.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:42.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:42.157522+0000) 2024-03-06T18:38:43.152 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:43.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:43.157755+0000) 2024-03-06T18:38:44.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:44.156+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:44.157981+0000) 2024-03-06T18:38:45.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:45.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:45.158253+0000) 2024-03-06T18:38:46.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:46.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:46.158482+0000) 2024-03-06T18:38:47.153 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:47.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:47.158707+0000) 2024-03-06T18:38:48.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:48.157+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:48.158919+0000) 2024-03-06T18:38:49.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:49.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:49.159144+0000) 2024-03-06T18:38:50.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:50.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:50.159371+0000) 2024-03-06T18:38:51.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:51.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:51.159613+0000) 2024-03-06T18:38:52.154 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:52.158+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:52.159827+0000) 2024-03-06T18:38:53.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:53.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:53.160100+0000) 2024-03-06T18:38:54.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:54.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:54.160325+0000) 2024-03-06T18:38:55.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:55.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:55.160580+0000) 2024-03-06T18:38:56.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:56.115 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:38:56.155 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:56.159+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:56.160796+0000) 2024-03-06T18:38:57.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:57.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:57.161041+0000) 2024-03-06T18:38:58.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:58.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:58.161261+0000) 2024-03-06T18:38:59.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:38:59.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:38:59.161486+0000) 2024-03-06T18:39:00.156 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:00.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:00.161717+0000) 2024-03-06T18:39:01.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:01.160+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:01.161972+0000) 2024-03-06T18:39:02.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:02.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:02.162236+0000) 2024-03-06T18:39:03.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:03.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:03.162481+0000) 2024-03-06T18:39:04.157 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:04.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:04.162716+0000) 2024-03-06T18:39:05.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:05.161+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:05.162968+0000) 2024-03-06T18:39:06.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:06.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:06.163211+0000) 2024-03-06T18:39:07.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:07.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:07.163415+0000) 2024-03-06T18:39:08.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:08.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:08.163637+0000) 2024-03-06T18:39:09.158 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:09.162+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:09.163870+0000) 2024-03-06T18:39:10.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:10.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:10.164120+0000) 2024-03-06T18:39:11.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:11.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:11.164363+0000) 2024-03-06T18:39:12.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:12.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:12.164588+0000) 2024-03-06T18:39:13.159 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:13.163+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:13.164812+0000) 2024-03-06T18:39:14.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:14.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:14.165058+0000) 2024-03-06T18:39:15.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:15.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:15.165259+0000) 2024-03-06T18:39:16.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:16.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:16.165490+0000) 2024-03-06T18:39:17.160 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:17.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:17.165720+0000) 2024-03-06T18:39:18.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:18.164+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:18.165957+0000) 2024-03-06T18:39:19.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:19.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:19.166151+0000) 2024-03-06T18:39:20.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:20.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:20.166345+0000) 2024-03-06T18:39:21.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:21.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:21.166542+0000) 2024-03-06T18:39:22.161 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:22.165+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:22.166776+0000) 2024-03-06T18:39:23.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:23.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:23.166998+0000) 2024-03-06T18:39:24.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:24.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:24.167271+0000) 2024-03-06T18:39:25.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:25.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:25.167507+0000) 2024-03-06T18:39:26.162 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:26.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:26.167746+0000) 2024-03-06T18:39:26.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:26.336 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:27.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:27.166+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:27.167977+0000) 2024-03-06T18:39:28.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:28.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:28.168266+0000) 2024-03-06T18:39:29.163 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:29.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:29.168453+0000) 2024-03-06T18:39:30.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:30.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:30.168669+0000) 2024-03-06T18:39:31.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:31.167+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:31.168901+0000) 2024-03-06T18:39:32.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:32.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:32.169163+0000) 2024-03-06T18:39:33.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:33.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:33.169401+0000) 2024-03-06T18:39:34.164 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:34.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:34.169606+0000) 2024-03-06T18:39:35.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:35.168+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:35.169825+0000) 2024-03-06T18:39:36.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:36.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:36.170084+0000) 2024-03-06T18:39:37.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:37.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:37.170289+0000) 2024-03-06T18:39:38.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:38.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:38.170556+0000) 2024-03-06T18:39:39.165 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:39.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:39.170788+0000) 2024-03-06T18:39:40.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:40.169+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:40.170975+0000) 2024-03-06T18:39:41.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:41.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:41.171221+0000) 2024-03-06T18:39:42.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:42.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:42.171456+0000) 2024-03-06T18:39:43.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:43.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:43.171695+0000) 2024-03-06T18:39:44.166 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:44.170+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:44.171897+0000) 2024-03-06T18:39:45.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:45.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:45.172135+0000) 2024-03-06T18:39:46.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:46.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:46.172341+0000) 2024-03-06T18:39:47.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:47.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:47.172532+0000) 2024-03-06T18:39:48.167 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:48.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:48.172754+0000) 2024-03-06T18:39:49.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:49.171+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:49.172988+0000) 2024-03-06T18:39:50.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:50.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:50.173266+0000) 2024-03-06T18:39:51.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:51.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:51.173514+0000) 2024-03-06T18:39:52.168 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:52.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:52.173741+0000) 2024-03-06T18:39:53.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:53.172+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:53.173970+0000) 2024-03-06T18:39:54.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:54.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:54.174217+0000) 2024-03-06T18:39:55.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:55.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:55.174449+0000) 2024-03-06T18:39:56.169 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:56.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:56.174694+0000) 2024-03-06T18:39:56.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:56.479 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:39:57.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:57.173+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:57.174937+0000) 2024-03-06T18:39:58.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:58.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:58.175178+0000) 2024-03-06T18:39:59.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:39:59.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:39:59.175401+0000) 2024-03-06T18:40:00.170 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:00.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:00.175640+0000) 2024-03-06T18:40:01.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:01.174+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:01.175876+0000) 2024-03-06T18:40:02.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:02.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:02.176126+0000) 2024-03-06T18:40:03.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:03.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:03.176354+0000) 2024-03-06T18:40:04.171 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:04.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:04.176563+0000) 2024-03-06T18:40:05.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:05.175+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:05.176806+0000) 2024-03-06T18:40:06.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:06.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:06.177085+0000) 2024-03-06T18:40:07.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:07.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:07.177325+0000) 2024-03-06T18:40:08.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:08.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:08.177540+0000) 2024-03-06T18:40:09.172 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:09.176+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:09.177764+0000) 2024-03-06T18:40:10.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:10.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:10.177997+0000) 2024-03-06T18:40:11.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:11.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:11.178279+0000) 2024-03-06T18:40:12.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:12.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:12.178519+0000) 2024-03-06T18:40:13.173 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:13.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:13.178756+0000) 2024-03-06T18:40:14.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:14.177+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:14.178982+0000) 2024-03-06T18:40:15.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:15.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:15.179249+0000) 2024-03-06T18:40:16.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:16.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:16.179496+0000) 2024-03-06T18:40:17.174 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:17.178+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:17.179739+0000) 2024-03-06T18:40:18.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:18.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:18.180001+0000) 2024-03-06T18:40:19.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:19.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:19.180258+0000) 2024-03-06T18:40:20.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:20.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:20.180492+0000) 2024-03-06T18:40:21.175 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:21.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:21.180715+0000) 2024-03-06T18:40:22.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:22.179+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:22.180975+0000) 2024-03-06T18:40:23.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:23.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:23.181183+0000) 2024-03-06T18:40:24.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:24.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:24.181420+0000) 2024-03-06T18:40:25.176 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:25.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:25.181645+0000) 2024-03-06T18:40:26.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:26.180+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:26.181946+0000) 2024-03-06T18:40:26.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:26.693 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:27.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:27.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:27.182157+0000) 2024-03-06T18:40:28.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:28.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:28.182414+0000) 2024-03-06T18:40:29.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:29.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:29.182657+0000) 2024-03-06T18:40:30.177 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:30.181+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:30.182835+0000) 2024-03-06T18:40:31.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:31.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:31.183131+0000) 2024-03-06T18:40:32.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:32.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:32.183361+0000) 2024-03-06T18:40:33.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:33.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:33.183592+0000) 2024-03-06T18:40:34.178 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:34.182+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:34.183822+0000) 2024-03-06T18:40:35.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:35.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:35.184085+0000) 2024-03-06T18:40:36.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:36.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:36.184325+0000) 2024-03-06T18:40:37.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:37.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:37.184602+0000) 2024-03-06T18:40:38.179 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:38.183+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:38.184874+0000) 2024-03-06T18:40:39.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:39.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:39.185128+0000) 2024-03-06T18:40:40.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:40.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:40.185313+0000) 2024-03-06T18:40:41.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:41.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:41.185554+0000) 2024-03-06T18:40:42.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:42.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:42.185742+0000) 2024-03-06T18:40:43.180 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:43.184+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:43.185985+0000) 2024-03-06T18:40:44.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:44.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:44.186243+0000) 2024-03-06T18:40:45.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:45.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:45.186444+0000) 2024-03-06T18:40:46.181 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:46.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:46.186689+0000) 2024-03-06T18:40:47.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:47.185+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:47.186918+0000) 2024-03-06T18:40:48.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:48.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:48.187233+0000) 2024-03-06T18:40:49.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:49.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:49.187484+0000) 2024-03-06T18:40:50.182 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:50.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:50.187712+0000) 2024-03-06T18:40:51.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:51.186+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:51.187951+0000) 2024-03-06T18:40:52.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:52.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:52.188197+0000) 2024-03-06T18:40:53.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:53.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:53.188448+0000) 2024-03-06T18:40:54.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:54.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:54.188679+0000) 2024-03-06T18:40:55.183 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:55.187+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:55.188873+0000) 2024-03-06T18:40:56.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:56.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:56.189161+0000) 2024-03-06T18:40:56.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:56.833 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:40:57.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:57.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:57.189320+0000) 2024-03-06T18:40:58.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:58.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:58.189542+0000) 2024-03-06T18:40:59.184 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:40:59.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:40:59.189759+0000) 2024-03-06T18:41:00.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:00.188+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:00.189988+0000) 2024-03-06T18:41:01.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:01.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:01.190259+0000) 2024-03-06T18:41:02.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:02.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:02.190490+0000) 2024-03-06T18:41:03.185 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:03.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:03.190734+0000) 2024-03-06T18:41:04.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:04.189+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:04.190959+0000) 2024-03-06T18:41:05.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:05.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:05.191193+0000) 2024-03-06T18:41:06.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:06.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:06.191404+0000) 2024-03-06T18:41:07.186 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:07.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:07.191615+0000) 2024-03-06T18:41:08.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:08.190+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:08.191857+0000) 2024-03-06T18:41:09.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:09.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:09.192119+0000) 2024-03-06T18:41:10.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:10.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:10.192320+0000) 2024-03-06T18:41:11.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:11.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:11.192539+0000) 2024-03-06T18:41:12.187 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:12.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:12.192772+0000) 2024-03-06T18:41:13.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:13.191+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:13.192964+0000) 2024-03-06T18:41:14.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:14.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:14.193197+0000) 2024-03-06T18:41:15.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:15.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:15.193449+0000) 2024-03-06T18:41:16.188 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:16.192+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:16.193741+0000) 2024-03-06T18:41:17.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:17.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:17.194043+0000) 2024-03-06T18:41:18.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:18.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:18.194261+0000) 2024-03-06T18:41:19.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:19.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:19.194528+0000) 2024-03-06T18:41:20.189 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:20.193+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:20.194781+0000) 2024-03-06T18:41:21.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:21.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:21.195048+0000) 2024-03-06T18:41:22.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:22.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:22.195288+0000) 2024-03-06T18:41:23.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:23.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:23.195511+0000) 2024-03-06T18:41:24.190 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:24.194+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:24.195741+0000) 2024-03-06T18:41:25.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:25.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:25.195991+0000) 2024-03-06T18:41:26.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:26.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:26.196316+0000) 2024-03-06T18:41:26.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:26.977 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:27.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:27.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:27.196579+0000) 2024-03-06T18:41:28.191 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:28.195+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:28.196832+0000) 2024-03-06T18:41:29.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:29.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:29.197117+0000) 2024-03-06T18:41:30.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:30.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:30.197318+0000) 2024-03-06T18:41:31.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:31.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:31.197573+0000) 2024-03-06T18:41:32.192 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:32.196+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:32.197824+0000) 2024-03-06T18:41:33.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:33.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:33.198070+0000) 2024-03-06T18:41:34.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:34.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:34.198231+0000) 2024-03-06T18:41:35.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:35.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:35.198437+0000) 2024-03-06T18:41:36.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:36.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:36.198661+0000) 2024-03-06T18:41:37.193 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:37.197+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:37.198886+0000) 2024-03-06T18:41:38.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:38.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:38.199134+0000) 2024-03-06T18:41:39.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:39.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:39.199368+0000) 2024-03-06T18:41:40.194 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:40.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:40.199640+0000) 2024-03-06T18:41:41.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:41.198+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:41.199928+0000) 2024-03-06T18:41:42.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:42.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:42.200178+0000) 2024-03-06T18:41:43.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:43.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:43.200411+0000) 2024-03-06T18:41:44.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:44.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:44.200630+0000) 2024-03-06T18:41:45.195 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:45.199+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:45.200882+0000) 2024-03-06T18:41:46.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:46.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:46.201159+0000) 2024-03-06T18:41:47.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:47.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:47.201365+0000) 2024-03-06T18:41:48.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:48.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:48.201587+0000) 2024-03-06T18:41:49.196 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:49.200+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:49.201855+0000) 2024-03-06T18:41:50.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:50.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:50.202170+0000) 2024-03-06T18:41:51.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:51.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:51.202408+0000) 2024-03-06T18:41:52.197 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:52.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:52.202668+0000) 2024-03-06T18:41:53.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:53.201+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:53.202982+0000) 2024-03-06T18:41:54.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:54.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:54.203277+0000) 2024-03-06T18:41:55.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:55.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:55.203538+0000) 2024-03-06T18:41:56.198 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:56.202+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:56.203805+0000) 2024-03-06T18:41:57.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:57.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:57.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:57.204126+0000) 2024-03-06T18:41:57.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:41:58.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:58.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:58.204418+0000) 2024-03-06T18:41:59.199 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:41:59.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:41:59.204685+0000) 2024-03-06T18:42:00.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:00.203+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:00.204946+0000) 2024-03-06T18:42:01.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:01.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:01.205202+0000) 2024-03-06T18:42:02.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:02.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:02.205459+0000) 2024-03-06T18:42:03.200 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:03.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:03.205739+0000) 2024-03-06T18:42:04.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:04.204+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:04.205975+0000) 2024-03-06T18:42:05.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:05.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:05.206170+0000) 2024-03-06T18:42:06.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:06.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:06.206389+0000) 2024-03-06T18:42:07.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:07.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:07.206643+0000) 2024-03-06T18:42:08.201 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:08.205+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:08.206983+0000) 2024-03-06T18:42:09.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:09.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:09.207260+0000) 2024-03-06T18:42:10.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:10.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:10.207502+0000) 2024-03-06T18:42:11.202 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:11.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:11.207713+0000) 2024-03-06T18:42:12.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:12.206+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:12.207952+0000) 2024-03-06T18:42:13.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:13.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:13.208203+0000) 2024-03-06T18:42:14.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:14.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:14.208436+0000) 2024-03-06T18:42:15.203 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:15.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:15.208676+0000) 2024-03-06T18:42:16.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:16.207+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:16.208907+0000) 2024-03-06T18:42:17.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:17.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:17.209141+0000) 2024-03-06T18:42:18.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:18.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:18.209384+0000) 2024-03-06T18:42:19.204 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:19.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:19.209623+0000) 2024-03-06T18:42:20.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:20.208+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:20.209864+0000) 2024-03-06T18:42:21.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:21.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:21.210142+0000) 2024-03-06T18:42:22.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:22.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:22.210386+0000) 2024-03-06T18:42:23.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:23.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:23.210621+0000) 2024-03-06T18:42:24.205 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:24.209+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:24.210826+0000) 2024-03-06T18:42:25.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:25.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:25.211109+0000) 2024-03-06T18:42:26.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:26.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:26.211358+0000) 2024-03-06T18:42:27.206 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:27.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:27.211612+0000) 2024-03-06T18:42:27.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:27.344 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:28.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:28.210+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:28.211902+0000) 2024-03-06T18:42:29.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:29.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:29.212125+0000) 2024-03-06T18:42:30.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:30.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:30.212353+0000) 2024-03-06T18:42:31.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:31.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:31.212597+0000) 2024-03-06T18:42:32.207 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:32.211+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:32.212824+0000) 2024-03-06T18:42:33.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:33.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:33.213042+0000) 2024-03-06T18:42:34.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:34.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:34.213260+0000) 2024-03-06T18:42:35.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:35.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:35.213474+0000) 2024-03-06T18:42:36.208 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:36.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:36.213686+0000) 2024-03-06T18:42:37.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:37.212+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:37.213925+0000) 2024-03-06T18:42:38.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:38.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:38.214147+0000) 2024-03-06T18:42:39.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:39.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:39.214340+0000) 2024-03-06T18:42:40.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:40.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:40.214566+0000) 2024-03-06T18:42:41.209 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:41.213+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:41.214787+0000) 2024-03-06T18:42:42.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:42.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:42.215084+0000) 2024-03-06T18:42:43.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:43.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:43.215339+0000) 2024-03-06T18:42:44.210 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:44.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:44.215565+0000) 2024-03-06T18:42:45.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:45.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:45.215782+0000) 2024-03-06T18:42:46.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:46.214+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:46.215983+0000) 2024-03-06T18:42:47.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:47.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:47.216302+0000) 2024-03-06T18:42:48.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:48.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:48.216549+0000) 2024-03-06T18:42:49.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:49.215+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:49.216784+0000) 2024-03-06T18:42:50.211 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:50.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:50.217050+0000) 2024-03-06T18:42:51.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:51.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:51.217244+0000) 2024-03-06T18:42:52.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:52.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:52.217457+0000) 2024-03-06T18:42:53.212 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:53.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:53.217644+0000) 2024-03-06T18:42:54.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:54.216+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:54.217864+0000) 2024-03-06T18:42:55.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:55.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:55.218123+0000) 2024-03-06T18:42:56.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:56.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:56.218405+0000) 2024-03-06T18:42:57.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:57.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:57.218600+0000) 2024-03-06T18:42:57.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:57.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:42:58.213 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:58.217+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:58.218881+0000) 2024-03-06T18:42:59.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:42:59.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:42:59.219138+0000) 2024-03-06T18:43:00.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:00.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:00.219366+0000) 2024-03-06T18:43:01.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:01.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:01.219566+0000) 2024-03-06T18:43:02.214 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:02.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:02.219763+0000) 2024-03-06T18:43:03.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:03.218+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:03.219974+0000) 2024-03-06T18:43:04.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:04.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:04.220237+0000) 2024-03-06T18:43:05.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:05.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:05.220458+0000) 2024-03-06T18:43:06.215 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:06.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:06.220662+0000) 2024-03-06T18:43:07.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:07.219+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:07.220885+0000) 2024-03-06T18:43:08.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:08.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:08.221144+0000) 2024-03-06T18:43:09.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:09.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:09.221336+0000) 2024-03-06T18:43:10.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:10.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:10.221553+0000) 2024-03-06T18:43:11.216 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:11.220+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:11.221768+0000) 2024-03-06T18:43:12.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:12.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:12.221996+0000) 2024-03-06T18:43:13.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:13.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:13.222212+0000) 2024-03-06T18:43:14.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:14.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:14.222403+0000) 2024-03-06T18:43:15.217 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:15.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:15.222602+0000) 2024-03-06T18:43:16.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:16.221+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:16.222832+0000) 2024-03-06T18:43:17.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:17.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:17.223090+0000) 2024-03-06T18:43:18.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:18.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:18.223348+0000) 2024-03-06T18:43:19.218 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:19.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:19.223552+0000) 2024-03-06T18:43:20.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:20.222+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:20.223798+0000) 2024-03-06T18:43:21.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:21.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:21.224039+0000) 2024-03-06T18:43:22.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:22.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:22.224269+0000) 2024-03-06T18:43:23.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:23.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:23.224471+0000) 2024-03-06T18:43:24.219 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:24.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:24.224715+0000) 2024-03-06T18:43:25.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:25.223+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:25.224949+0000) 2024-03-06T18:43:26.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:26.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:26.225206+0000) 2024-03-06T18:43:27.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:27.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:27.225421+0000) 2024-03-06T18:43:27.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:27.716 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:28.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:28.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:28.225648+0000) 2024-03-06T18:43:29.220 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:29.224+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:29.225881+0000) 2024-03-06T18:43:30.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:30.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:30.226131+0000) 2024-03-06T18:43:31.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:31.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:31.226330+0000) 2024-03-06T18:43:32.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:32.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:32.226597+0000) 2024-03-06T18:43:33.221 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:33.225+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:33.226813+0000) 2024-03-06T18:43:34.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:34.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:34.227076+0000) 2024-03-06T18:43:35.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:35.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:35.227261+0000) 2024-03-06T18:43:36.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:36.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:36.227463+0000) 2024-03-06T18:43:37.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:37.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:37.227637+0000) 2024-03-06T18:43:38.222 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:38.226+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:38.227880+0000) 2024-03-06T18:43:39.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:39.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:39.228119+0000) 2024-03-06T18:43:40.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:40.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:40.228354+0000) 2024-03-06T18:43:41.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:41.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:41.228598+0000) 2024-03-06T18:43:42.223 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:42.227+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:42.228782+0000) 2024-03-06T18:43:43.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:43.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:43.229036+0000) 2024-03-06T18:43:44.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:44.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:44.229249+0000) 2024-03-06T18:43:45.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:45.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:45.229495+0000) 2024-03-06T18:43:46.224 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:46.228+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:46.229774+0000) 2024-03-06T18:43:47.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:47.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:47.230083+0000) 2024-03-06T18:43:48.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:48.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:48.230304+0000) 2024-03-06T18:43:49.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:49.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:49.230581+0000) 2024-03-06T18:43:50.225 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:50.229+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:50.230815+0000) 2024-03-06T18:43:51.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:51.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:51.231073+0000) 2024-03-06T18:43:52.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:52.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:52.231268+0000) 2024-03-06T18:43:53.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:53.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:53.231527+0000) 2024-03-06T18:43:54.226 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:54.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:54.231771+0000) 2024-03-06T18:43:55.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:55.230+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:55.231975+0000) 2024-03-06T18:43:56.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:56.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:56.232254+0000) 2024-03-06T18:43:57.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:57.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:57.232517+0000) 2024-03-06T18:43:57.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:57.925 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:43:58.227 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:58.231+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:58.232812+0000) 2024-03-06T18:43:59.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:43:59.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:43:59.233080+0000) 2024-03-06T18:44:00.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:00.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:00.233313+0000) 2024-03-06T18:44:01.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:01.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:01.233515+0000) 2024-03-06T18:44:02.228 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:02.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:02.233754+0000) 2024-03-06T18:44:03.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:03.232+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:03.233962+0000) 2024-03-06T18:44:04.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:04.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:04.234161+0000) 2024-03-06T18:44:05.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:05.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:05.234392+0000) 2024-03-06T18:44:06.229 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:06.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:06.234647+0000) 2024-03-06T18:44:07.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:07.233+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:07.234926+0000) 2024-03-06T18:44:08.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:08.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:08.235209+0000) 2024-03-06T18:44:09.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:09.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:09.235418+0000) 2024-03-06T18:44:10.230 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:10.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:10.235644+0000) 2024-03-06T18:44:11.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:11.234+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:11.235868+0000) 2024-03-06T18:44:12.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:12.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:12.236150+0000) 2024-03-06T18:44:13.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:13.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:13.236394+0000) 2024-03-06T18:44:14.231 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:14.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:14.236620+0000) 2024-03-06T18:44:15.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:15.235+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:15.236843+0000) 2024-03-06T18:44:16.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:16.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:16.237125+0000) 2024-03-06T18:44:17.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:17.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:17.237331+0000) 2024-03-06T18:44:18.232 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:18.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:18.237571+0000) 2024-03-06T18:44:19.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:19.236+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:19.237813+0000) 2024-03-06T18:44:20.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:20.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:20.238077+0000) 2024-03-06T18:44:21.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:21.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:21.238300+0000) 2024-03-06T18:44:22.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:22.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:22.238541+0000) 2024-03-06T18:44:23.233 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:23.237+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:23.238760+0000) 2024-03-06T18:44:24.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:24.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:24.238999+0000) 2024-03-06T18:44:25.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:25.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:25.239212+0000) 2024-03-06T18:44:26.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:26.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:26.239436+0000) 2024-03-06T18:44:27.234 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:27.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:27.239648+0000) 2024-03-06T18:44:28.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:28.061 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:28.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:28.238+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:28.239884+0000) 2024-03-06T18:44:29.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:29.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:29.240095+0000) 2024-03-06T18:44:30.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:30.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:30.240293+0000) 2024-03-06T18:44:31.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:31.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:31.240499+0000) 2024-03-06T18:44:32.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:32.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:32.240753+0000) 2024-03-06T18:44:33.235 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:33.239+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:33.240979+0000) 2024-03-06T18:44:34.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:34.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:34.241163+0000) 2024-03-06T18:44:35.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:35.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:35.241380+0000) 2024-03-06T18:44:36.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:36.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:36.241639+0000) 2024-03-06T18:44:37.236 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:37.240+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:37.241962+0000) 2024-03-06T18:44:38.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:38.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:38.242244+0000) 2024-03-06T18:44:39.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:39.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:39.242492+0000) 2024-03-06T18:44:40.237 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:40.241+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:40.242772+0000) 2024-03-06T18:44:41.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:41.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:41.243103+0000) 2024-03-06T18:44:42.238 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:42.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:42.243354+0000) 2024-03-06T18:44:43.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:43.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:43.243610+0000) 2024-03-06T18:44:44.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:44.242+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:44.243903+0000) 2024-03-06T18:44:45.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:45.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:45.244202+0000) 2024-03-06T18:44:46.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:46.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:46.244428+0000) 2024-03-06T18:44:47.239 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:47.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:47.244678+0000) 2024-03-06T18:44:48.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:48.243+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:48.244922+0000) 2024-03-06T18:44:49.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:49.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:49.245129+0000) 2024-03-06T18:44:50.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:50.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:50.245368+0000) 2024-03-06T18:44:51.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:51.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:51.245599+0000) 2024-03-06T18:44:52.240 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:52.244+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:52.245846+0000) 2024-03-06T18:44:53.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:53.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:53.246050+0000) 2024-03-06T18:44:54.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:54.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:54.246221+0000) 2024-03-06T18:44:55.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:55.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:55.246415+0000) 2024-03-06T18:44:56.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:56.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:56.246651+0000) 2024-03-06T18:44:57.241 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:57.245+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:57.246876+0000) 2024-03-06T18:44:58.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:58.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:58.247161+0000) 2024-03-06T18:44:58.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:58.265 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:44:59.242 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:44:59.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:44:59.247373+0000) 2024-03-06T18:45:00.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:00.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:00.247604+0000) 2024-03-06T18:45:01.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:01.246+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:01.247826+0000) 2024-03-06T18:45:02.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:02.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:02.248111+0000) 2024-03-06T18:45:03.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:03.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:03.248314+0000) 2024-03-06T18:45:04.243 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:04.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:04.248520+0000) 2024-03-06T18:45:05.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:05.247+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:05.248758+0000) 2024-03-06T18:45:06.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:06.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:06.249039+0000) 2024-03-06T18:45:07.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:07.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:07.249266+0000) 2024-03-06T18:45:08.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:08.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:08.249489+0000) 2024-03-06T18:45:09.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:09.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:09.249718+0000) 2024-03-06T18:45:10.244 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:10.248+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:10.249981+0000) 2024-03-06T18:45:11.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:11.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:11.250191+0000) 2024-03-06T18:45:12.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:12.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:12.250392+0000) 2024-03-06T18:45:13.245 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:13.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:13.250603+0000) 2024-03-06T18:45:14.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:14.249+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:14.250810+0000) 2024-03-06T18:45:15.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:15.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:15.251060+0000) 2024-03-06T18:45:16.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:16.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:16.251255+0000) 2024-03-06T18:45:17.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:17.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:17.251462+0000) 2024-03-06T18:45:18.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:18.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:18.251682+0000) 2024-03-06T18:45:19.246 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:19.250+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:19.251896+0000) 2024-03-06T18:45:20.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:20.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:20.252144+0000) 2024-03-06T18:45:21.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:21.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:21.252336+0000) 2024-03-06T18:45:22.247 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:22.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:22.252565+0000) 2024-03-06T18:45:23.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:23.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:23.252735+0000) 2024-03-06T18:45:24.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:24.251+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:24.252972+0000) 2024-03-06T18:45:25.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:25.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:25.253218+0000) 2024-03-06T18:45:26.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:26.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:26.253456+0000) 2024-03-06T18:45:27.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:27.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:27.253678+0000) 2024-03-06T18:45:28.248 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:28.252+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:28.253940+0000) 2024-03-06T18:45:28.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:28.478 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:29.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:29.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:29.254163+0000) 2024-03-06T18:45:30.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:30.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:30.254398+0000) 2024-03-06T18:45:31.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:31.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:31.254664+0000) 2024-03-06T18:45:32.249 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:32.253+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:32.254883+0000) 2024-03-06T18:45:33.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:33.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:33.255137+0000) 2024-03-06T18:45:34.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:34.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:34.255396+0000) 2024-03-06T18:45:35.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:35.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:35.255608+0000) 2024-03-06T18:45:36.250 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:36.254+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:36.255796+0000) 2024-03-06T18:45:37.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:37.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:37.256053+0000) 2024-03-06T18:45:38.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:38.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:38.256290+0000) 2024-03-06T18:45:39.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:39.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:39.256479+0000) 2024-03-06T18:45:40.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:40.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:40.256737+0000) 2024-03-06T18:45:41.251 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:41.255+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:41.256959+0000) 2024-03-06T18:45:42.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:42.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:42.257198+0000) 2024-03-06T18:45:43.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:43.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:43.257422+0000) 2024-03-06T18:45:44.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:44.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:44.257650+0000) 2024-03-06T18:45:45.252 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:45.256+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:45.257866+0000) 2024-03-06T18:45:46.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:46.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:46.258151+0000) 2024-03-06T18:45:47.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:47.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:47.258376+0000) 2024-03-06T18:45:48.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:48.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:48.258641+0000) 2024-03-06T18:45:49.253 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:49.257+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:49.258860+0000) 2024-03-06T18:45:50.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:50.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:50.259124+0000) 2024-03-06T18:45:51.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:51.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:51.259345+0000) 2024-03-06T18:45:52.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:52.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:52.259582+0000) 2024-03-06T18:45:53.254 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:53.258+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:53.259818+0000) 2024-03-06T18:45:54.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:54.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:54.260075+0000) 2024-03-06T18:45:55.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:55.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:55.260252+0000) 2024-03-06T18:45:56.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:56.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:56.260486+0000) 2024-03-06T18:45:57.255 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:57.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:57.260713+0000) 2024-03-06T18:45:58.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:58.259+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:58.260950+0000) 2024-03-06T18:45:58.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:58.616 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:45:59.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:45:59.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:45:59.261173+0000) 2024-03-06T18:46:00.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:00.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:00.261415+0000) 2024-03-06T18:46:01.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:01.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:01.261639+0000) 2024-03-06T18:46:02.256 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:02.260+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:02.261852+0000) 2024-03-06T18:46:03.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:03.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:03.262107+0000) 2024-03-06T18:46:04.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:04.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:04.262336+0000) 2024-03-06T18:46:05.257 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:05.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:05.262609+0000) 2024-03-06T18:46:06.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:06.261+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:06.262902+0000) 2024-03-06T18:46:07.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:07.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:07.263197+0000) 2024-03-06T18:46:08.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:08.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:08.263438+0000) 2024-03-06T18:46:09.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:09.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:09.263646+0000) 2024-03-06T18:46:10.258 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:10.262+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:10.263873+0000) 2024-03-06T18:46:11.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:11.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:11.264144+0000) 2024-03-06T18:46:12.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:12.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:12.264411+0000) 2024-03-06T18:46:13.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:13.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:13.264693+0000) 2024-03-06T18:46:14.259 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:14.263+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:14.264927+0000) 2024-03-06T18:46:15.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:15.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:15.265189+0000) 2024-03-06T18:46:16.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:16.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:16.265429+0000) 2024-03-06T18:46:17.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:17.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:17.265662+0000) 2024-03-06T18:46:18.260 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:18.264+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:18.265955+0000) 2024-03-06T18:46:19.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:19.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:19.266184+0000) 2024-03-06T18:46:20.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:20.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:20.266456+0000) 2024-03-06T18:46:21.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:21.265+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:21.266712+0000) 2024-03-06T18:46:22.261 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:22.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:22.266991+0000) 2024-03-06T18:46:23.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:23.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:23.267227+0000) 2024-03-06T18:46:24.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:24.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:24.267474+0000) 2024-03-06T18:46:25.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:25.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:25.267669+0000) 2024-03-06T18:46:26.262 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:26.266+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:26.267899+0000) 2024-03-06T18:46:27.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:27.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:27.268140+0000) 2024-03-06T18:46:28.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:28.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:28.268445+0000) 2024-03-06T18:46:28.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:28.756 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:29.263 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:29.267+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:29.268716+0000) 2024-03-06T18:46:30.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:30.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:30.268997+0000) 2024-03-06T18:46:31.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:31.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:31.269310+0000) 2024-03-06T18:46:32.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:32.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:32.269609+0000) 2024-03-06T18:46:33.264 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:33.268+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:33.269890+0000) 2024-03-06T18:46:34.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:34.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:34.270163+0000) 2024-03-06T18:46:35.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:35.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:35.270445+0000) 2024-03-06T18:46:36.265 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:36.269+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:36.270721+0000) 2024-03-06T18:46:37.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:37.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:37.271041+0000) 2024-03-06T18:46:38.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:38.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:38.271351+0000) 2024-03-06T18:46:39.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:39.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:39.271585+0000) 2024-03-06T18:46:40.266 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:40.270+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:40.271814+0000) 2024-03-06T18:46:41.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:41.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:41.272067+0000) 2024-03-06T18:46:42.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:42.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:42.272306+0000) 2024-03-06T18:46:43.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:43.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:43.272520+0000) 2024-03-06T18:46:44.267 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:44.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:44.272750+0000) 2024-03-06T18:46:45.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:45.271+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:45.272958+0000) 2024-03-06T18:46:46.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:46.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:46.273211+0000) 2024-03-06T18:46:47.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:47.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:47.273452+0000) 2024-03-06T18:46:48.268 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:48.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:48.273715+0000) 2024-03-06T18:46:49.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:49.272+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:49.273965+0000) 2024-03-06T18:46:50.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:50.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:50.274278+0000) 2024-03-06T18:46:51.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:51.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:51.274549+0000) 2024-03-06T18:46:52.269 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:52.273+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:52.274799+0000) 2024-03-06T18:46:53.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:53.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:53.275051+0000) 2024-03-06T18:46:54.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:54.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:54.275289+0000) 2024-03-06T18:46:55.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:55.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:55.275478+0000) 2024-03-06T18:46:56.270 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:56.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:56.275719+0000) 2024-03-06T18:46:57.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:57.274+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:57.275954+0000) 2024-03-06T18:46:58.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:58.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:58.276199+0000) 2024-03-06T18:46:58.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:58.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:46:59.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:46:59.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:46:59.276442+0000) 2024-03-06T18:47:00.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:00.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:00.276656+0000) 2024-03-06T18:47:01.271 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:01.275+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:01.276887+0000) 2024-03-06T18:47:02.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:02.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:02.277139+0000) 2024-03-06T18:47:03.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:03.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:03.277347+0000) 2024-03-06T18:47:04.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:04.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:04.277595+0000) 2024-03-06T18:47:05.272 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:05.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:05.277800+0000) 2024-03-06T18:47:06.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:06.276+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:06.277989+0000) 2024-03-06T18:47:07.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:07.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:07.278251+0000) 2024-03-06T18:47:08.273 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:08.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:08.278526+0000) 2024-03-06T18:47:09.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:09.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:09.278748+0000) 2024-03-06T18:47:10.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:10.277+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:10.278973+0000) 2024-03-06T18:47:11.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:11.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:11.279152+0000) 2024-03-06T18:47:12.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:12.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:12.279363+0000) 2024-03-06T18:47:13.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:13.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:13.279564+0000) 2024-03-06T18:47:14.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:14.278+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:14.279795+0000) 2024-03-06T18:47:15.274 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:15.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:15.280047+0000) 2024-03-06T18:47:16.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:16.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:16.280232+0000) 2024-03-06T18:47:17.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:17.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:17.280409+0000) 2024-03-06T18:47:18.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:18.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:18.280631+0000) 2024-03-06T18:47:19.275 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:19.279+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:19.280855+0000) 2024-03-06T18:47:20.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:20.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:20.281105+0000) 2024-03-06T18:47:21.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:21.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:21.281286+0000) 2024-03-06T18:47:22.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:22.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:22.281488+0000) 2024-03-06T18:47:23.276 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:23.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:23.281709+0000) 2024-03-06T18:47:24.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:24.280+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:24.281938+0000) 2024-03-06T18:47:25.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:25.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:25.282162+0000) 2024-03-06T18:47:26.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:26.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:26.282388+0000) 2024-03-06T18:47:27.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:27.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:27.282558+0000) 2024-03-06T18:47:28.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:28.281+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:28.282777+0000) 2024-03-06T18:47:29.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:29.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:29.277 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:29.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:29.283028+0000) 2024-03-06T18:47:30.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:30.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:30.283221+0000) 2024-03-06T18:47:31.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:31.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:31.283437+0000) 2024-03-06T18:47:32.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:32.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:32.283704+0000) 2024-03-06T18:47:33.278 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:33.282+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:33.283908+0000) 2024-03-06T18:47:34.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:34.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:34.284141+0000) 2024-03-06T18:47:35.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:35.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:35.284360+0000) 2024-03-06T18:47:36.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:36.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:36.284590+0000) 2024-03-06T18:47:37.279 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:37.283+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:37.284811+0000) 2024-03-06T18:47:38.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:38.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:38.285091+0000) 2024-03-06T18:47:39.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:39.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:39.285316+0000) 2024-03-06T18:47:40.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:40.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:40.285505+0000) 2024-03-06T18:47:41.280 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:41.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:41.285726+0000) 2024-03-06T18:47:42.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:42.284+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:42.285957+0000) 2024-03-06T18:47:43.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:43.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:43.286200+0000) 2024-03-06T18:47:44.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:44.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:44.286412+0000) 2024-03-06T18:47:45.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:45.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:45.286609+0000) 2024-03-06T18:47:46.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:46.285+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:46.286837+0000) 2024-03-06T18:47:47.281 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:47.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:47.287128+0000) 2024-03-06T18:47:48.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:48.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:48.287384+0000) 2024-03-06T18:47:49.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:49.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:49.287600+0000) 2024-03-06T18:47:50.282 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:50.286+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:50.287827+0000) 2024-03-06T18:47:51.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:51.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:51.288086+0000) 2024-03-06T18:47:52.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:52.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:52.288283+0000) 2024-03-06T18:47:53.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:53.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:53.288493+0000) 2024-03-06T18:47:54.283 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:54.287+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:54.288781+0000) 2024-03-06T18:47:55.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:55.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:55.289065+0000) 2024-03-06T18:47:56.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:56.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:56.289313+0000) 2024-03-06T18:47:57.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:57.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:57.289538+0000) 2024-03-06T18:47:58.284 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:58.288+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:58.289791+0000) 2024-03-06T18:47:59.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:59.166 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:47:59.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:47:59.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:47:59.290087+0000) 2024-03-06T18:48:00.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:00.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:00.290344+0000) 2024-03-06T18:48:01.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:01.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:01.290556+0000) 2024-03-06T18:48:02.285 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:02.289+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:02.290782+0000) 2024-03-06T18:48:03.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:03.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:03.291042+0000) 2024-03-06T18:48:04.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:04.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:04.291219+0000) 2024-03-06T18:48:05.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:05.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:05.291410+0000) 2024-03-06T18:48:06.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:06.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:06.291640+0000) 2024-03-06T18:48:07.286 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:07.290+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:07.291868+0000) 2024-03-06T18:48:08.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:08.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:08.292150+0000) 2024-03-06T18:48:09.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:09.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:09.292382+0000) 2024-03-06T18:48:10.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:10.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:10.292590+0000) 2024-03-06T18:48:11.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:11.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:11.292804+0000) 2024-03-06T18:48:12.287 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:12.291+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:12.292983+0000) 2024-03-06T18:48:13.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:13.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:13.293168+0000) 2024-03-06T18:48:14.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:14.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:14.293361+0000) 2024-03-06T18:48:15.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:15.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:15.293562+0000) 2024-03-06T18:48:16.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:16.292+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:16.293807+0000) 2024-03-06T18:48:17.288 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:17.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:17.294084+0000) 2024-03-06T18:48:18.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:18.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:18.294351+0000) 2024-03-06T18:48:19.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:19.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:19.294584+0000) 2024-03-06T18:48:20.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:20.293+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:20.294850+0000) 2024-03-06T18:48:21.289 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:21.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:21.295090+0000) 2024-03-06T18:48:22.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:22.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:22.295298+0000) 2024-03-06T18:48:23.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:23.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:23.295511+0000) 2024-03-06T18:48:24.290 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:24.294+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:24.295813+0000) 2024-03-06T18:48:25.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:25.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:25.296099+0000) 2024-03-06T18:48:26.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:26.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:26.296346+0000) 2024-03-06T18:48:27.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:27.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:27.296628+0000) 2024-03-06T18:48:28.291 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:28.295+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:28.296898+0000) 2024-03-06T18:48:29.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:29.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:29.297184+0000) 2024-03-06T18:48:29.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:29.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:30.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:30.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:30.297445+0000) 2024-03-06T18:48:31.292 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:31.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:31.297697+0000) 2024-03-06T18:48:32.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:32.296+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:32.297975+0000) 2024-03-06T18:48:33.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:33.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:33.298214+0000) 2024-03-06T18:48:34.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:34.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:34.298408+0000) 2024-03-06T18:48:35.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:35.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:35.298609+0000) 2024-03-06T18:48:36.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:36.297+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:36.298835+0000) 2024-03-06T18:48:37.293 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:37.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:37.299100+0000) 2024-03-06T18:48:38.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:38.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:38.299316+0000) 2024-03-06T18:48:39.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:39.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:39.299539+0000) 2024-03-06T18:48:40.294 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:40.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:40.299765+0000) 2024-03-06T18:48:41.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:41.298+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:41.299986+0000) 2024-03-06T18:48:42.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:42.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:42.300252+0000) 2024-03-06T18:48:43.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:43.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:43.300476+0000) 2024-03-06T18:48:44.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:44.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:44.300693+0000) 2024-03-06T18:48:45.295 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:45.299+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:45.300958+0000) 2024-03-06T18:48:46.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:46.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:46.301195+0000) 2024-03-06T18:48:47.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:47.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:47.301430+0000) 2024-03-06T18:48:48.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:48.300+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:48.301763+0000) 2024-03-06T18:48:49.296 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:49.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:49.301992+0000) 2024-03-06T18:48:50.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:50.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:50.302277+0000) 2024-03-06T18:48:51.297 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:51.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:51.302553+0000) 2024-03-06T18:48:52.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:52.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:52.302768+0000) 2024-03-06T18:48:53.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:53.301+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:53.302939+0000) 2024-03-06T18:48:54.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:54.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:54.303164+0000) 2024-03-06T18:48:55.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:55.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:55.303398+0000) 2024-03-06T18:48:56.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:56.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:56.303645+0000) 2024-03-06T18:48:57.298 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:57.302+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:57.303880+0000) 2024-03-06T18:48:58.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:58.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:58.304166+0000) 2024-03-06T18:48:59.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:48:59.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:48:59.304370+0000) 2024-03-06T18:48:59.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:48:59.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:00.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:00.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:00.304607+0000) 2024-03-06T18:49:01.299 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:01.303+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:01.304829+0000) 2024-03-06T18:49:02.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:02.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:02.305141+0000) 2024-03-06T18:49:03.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:03.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:03.305402+0000) 2024-03-06T18:49:04.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:04.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:04.305629+0000) 2024-03-06T18:49:05.300 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:05.304+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:05.305851+0000) 2024-03-06T18:49:06.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:06.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:06.306102+0000) 2024-03-06T18:49:07.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:07.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:07.306289+0000) 2024-03-06T18:49:08.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:08.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:08.306475+0000) 2024-03-06T18:49:09.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:09.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:09.306705+0000) 2024-03-06T18:49:10.301 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:10.305+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:10.306932+0000) 2024-03-06T18:49:11.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:11.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:11.307203+0000) 2024-03-06T18:49:12.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:12.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:12.307463+0000) 2024-03-06T18:49:13.302 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:13.306+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:13.307744+0000) 2024-03-06T18:49:14.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:14.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:14.308049+0000) 2024-03-06T18:49:15.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:15.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:15.308257+0000) 2024-03-06T18:49:16.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:16.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:16.308466+0000) 2024-03-06T18:49:17.303 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:17.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:17.308717+0000) 2024-03-06T18:49:18.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:18.307+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:18.308973+0000) 2024-03-06T18:49:19.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:19.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:19.309162+0000) 2024-03-06T18:49:20.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:20.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:20.309383+0000) 2024-03-06T18:49:21.304 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:21.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:21.309607+0000) 2024-03-06T18:49:22.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:22.308+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:22.309830+0000) 2024-03-06T18:49:23.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:23.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:23.310083+0000) 2024-03-06T18:49:24.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:24.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:24.310325+0000) 2024-03-06T18:49:25.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:25.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:25.310546+0000) 2024-03-06T18:49:26.305 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:26.309+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:26.310770+0000) 2024-03-06T18:49:27.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:27.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:27.310994+0000) 2024-03-06T18:49:28.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:28.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:28.311235+0000) 2024-03-06T18:49:29.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:29.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:29.311453+0000) 2024-03-06T18:49:29.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:29.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:30.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:30.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:30.311682+0000) 2024-03-06T18:49:31.306 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:31.310+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:31.311926+0000) 2024-03-06T18:49:32.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:32.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:32.312167+0000) 2024-03-06T18:49:33.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:33.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:33.312377+0000) 2024-03-06T18:49:34.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:34.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:34.312600+0000) 2024-03-06T18:49:35.307 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:35.311+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:35.312852+0000) 2024-03-06T18:49:36.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:36.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:36.313158+0000) 2024-03-06T18:49:37.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:37.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:37.313375+0000) 2024-03-06T18:49:38.308 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:38.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:38.313596+0000) 2024-03-06T18:49:39.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:39.312+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:39.313822+0000) 2024-03-06T18:49:40.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:40.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:40.314095+0000) 2024-03-06T18:49:41.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:41.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:41.314285+0000) 2024-03-06T18:49:42.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:42.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:42.314492+0000) 2024-03-06T18:49:43.309 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:43.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:43.314689+0000) 2024-03-06T18:49:44.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:44.313+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:44.314911+0000) 2024-03-06T18:49:45.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:45.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:45.315147+0000) 2024-03-06T18:49:46.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:46.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:46.315371+0000) 2024-03-06T18:49:47.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:47.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:47.315559+0000) 2024-03-06T18:49:48.310 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:48.314+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:48.315781+0000) 2024-03-06T18:49:49.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:49.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:49.316000+0000) 2024-03-06T18:49:50.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:50.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:50.316227+0000) 2024-03-06T18:49:51.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:51.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:51.316419+0000) 2024-03-06T18:49:52.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:52.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:52.316632+0000) 2024-03-06T18:49:53.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:53.315+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:53.316850+0000) 2024-03-06T18:49:54.311 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:54.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:54.317093+0000) 2024-03-06T18:49:55.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:55.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:55.317311+0000) 2024-03-06T18:49:56.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:56.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:56.317556+0000) 2024-03-06T18:49:57.312 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:57.316+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:57.317760+0000) 2024-03-06T18:49:58.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:58.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:58.317989+0000) 2024-03-06T18:49:59.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:49:59.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:49:59.318227+0000) 2024-03-06T18:49:59.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:49:59.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:00.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:00.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:00.318423+0000) 2024-03-06T18:50:01.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:01.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:01.318634+0000) 2024-03-06T18:50:02.313 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:02.317+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:02.318830+0000) 2024-03-06T18:50:03.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:03.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:03.319085+0000) 2024-03-06T18:50:04.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:04.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:04.319287+0000) 2024-03-06T18:50:05.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:05.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:05.319491+0000) 2024-03-06T18:50:06.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:06.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:06.319716+0000) 2024-03-06T18:50:07.314 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:07.318+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:07.319936+0000) 2024-03-06T18:50:08.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:08.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:08.320176+0000) 2024-03-06T18:50:09.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:09.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:09.320366+0000) 2024-03-06T18:50:10.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:10.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:10.320594+0000) 2024-03-06T18:50:11.315 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:11.319+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:11.320781+0000) 2024-03-06T18:50:12.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:12.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:12.321052+0000) 2024-03-06T18:50:13.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:13.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:13.321215+0000) 2024-03-06T18:50:14.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:14.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:14.321412+0000) 2024-03-06T18:50:15.316 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:15.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:15.321618+0000) 2024-03-06T18:50:16.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:16.320+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:16.321850+0000) 2024-03-06T18:50:17.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:17.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:17.322109+0000) 2024-03-06T18:50:18.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:18.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:18.322323+0000) 2024-03-06T18:50:19.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:19.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:19.322550+0000) 2024-03-06T18:50:20.317 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:20.321+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:20.322787+0000) 2024-03-06T18:50:21.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:21.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:21.323107+0000) 2024-03-06T18:50:22.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:22.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:22.323334+0000) 2024-03-06T18:50:23.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:23.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:23.323573+0000) 2024-03-06T18:50:24.318 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:24.322+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:24.323881+0000) 2024-03-06T18:50:25.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:25.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:25.324155+0000) 2024-03-06T18:50:26.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:26.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:26.324368+0000) 2024-03-06T18:50:27.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:27.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:27.324597+0000) 2024-03-06T18:50:28.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:28.323+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:28.324889+0000) 2024-03-06T18:50:29.319 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:29.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:29.325163+0000) 2024-03-06T18:50:29.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:29.874 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:50:30.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:30.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:30.325442+0000) 2024-03-06T18:50:31.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:31.324+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:31.325716+0000) 2024-03-06T18:50:32.320 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:32.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:32.326056+0000) 2024-03-06T18:50:33.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:33.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:33.326338+0000) 2024-03-06T18:50:34.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:34.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:34.326618+0000) 2024-03-06T18:50:35.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:35.325+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:35.326884+0000) 2024-03-06T18:50:36.321 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:36.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:36.327137+0000) 2024-03-06T18:50:37.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:37.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:37.327293+0000) 2024-03-06T18:50:38.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:38.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:38.327517+0000) 2024-03-06T18:50:39.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:39.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:39.327738+0000) 2024-03-06T18:50:40.322 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:40.326+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:40.327979+0000) 2024-03-06T18:50:41.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:41.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:41.328211+0000) 2024-03-06T18:50:42.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:42.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:42.328449+0000) 2024-03-06T18:50:43.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:43.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:43.328675+0000) 2024-03-06T18:50:44.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:44.327+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:44.328870+0000) 2024-03-06T18:50:45.323 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:45.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:45.329097+0000) 2024-03-06T18:50:46.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:46.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:46.329331+0000) 2024-03-06T18:50:47.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:47.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:47.329568+0000) 2024-03-06T18:50:48.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:48.328+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:48.329825+0000) 2024-03-06T18:50:49.324 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:49.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:49.329995+0000) 2024-03-06T18:50:50.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:50.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:50.330225+0000) 2024-03-06T18:50:51.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:51.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:51.330441+0000) 2024-03-06T18:50:52.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:52.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:52.330670+0000) 2024-03-06T18:50:53.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:53.329+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:53.330891+0000) 2024-03-06T18:50:54.325 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:54.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:54.331117+0000) 2024-03-06T18:50:55.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:55.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:55.331337+0000) 2024-03-06T18:50:56.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:56.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:56.331581+0000) 2024-03-06T18:50:57.326 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:57.330+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:57.331801+0000) 2024-03-06T18:50:58.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:58.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:58.332077+0000) 2024-03-06T18:50:59.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:50:59.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:50:59.332306+0000) 2024-03-06T18:51:00.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:00.019 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:00.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:00.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:00.332518+0000) 2024-03-06T18:51:01.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:01.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:01.332747+0000) 2024-03-06T18:51:02.327 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:02.331+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:02.332972+0000) 2024-03-06T18:51:03.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:03.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:03.333198+0000) 2024-03-06T18:51:04.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:04.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:04.333392+0000) 2024-03-06T18:51:05.328 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:05.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:05.333600+0000) 2024-03-06T18:51:06.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:06.332+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:06.333836+0000) 2024-03-06T18:51:07.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:07.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:07.334079+0000) 2024-03-06T18:51:08.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:08.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:08.334291+0000) 2024-03-06T18:51:09.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:09.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:09.334483+0000) 2024-03-06T18:51:10.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:10.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:10.334703+0000) 2024-03-06T18:51:11.329 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:11.333+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:11.334917+0000) 2024-03-06T18:51:12.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:12.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:12.335132+0000) 2024-03-06T18:51:13.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:13.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:13.335352+0000) 2024-03-06T18:51:14.330 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:14.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:14.335570+0000) 2024-03-06T18:51:15.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:15.334+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:15.335781+0000) 2024-03-06T18:51:16.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:16.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:16.336043+0000) 2024-03-06T18:51:17.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:17.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:17.336269+0000) 2024-03-06T18:51:18.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:18.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:18.336501+0000) 2024-03-06T18:51:19.331 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:19.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:19.336739+0000) 2024-03-06T18:51:20.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:20.335+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:20.336966+0000) 2024-03-06T18:51:21.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:21.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:21.337194+0000) 2024-03-06T18:51:22.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:22.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:22.337481+0000) 2024-03-06T18:51:23.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:23.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:23.337703+0000) 2024-03-06T18:51:24.332 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:24.336+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:24.337940+0000) 2024-03-06T18:51:25.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:25.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:25.338155+0000) 2024-03-06T18:51:26.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:26.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:26.338405+0000) 2024-03-06T18:51:27.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:27.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:27.338616+0000) 2024-03-06T18:51:28.333 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:28.337+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:28.338866+0000) 2024-03-06T18:51:29.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:29.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:29.339124+0000) 2024-03-06T18:51:30.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:30.224 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:51:30.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:30.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:30.339292+0000) 2024-03-06T18:51:31.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:31.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:31.339460+0000) 2024-03-06T18:51:32.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:32.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:32.339707+0000) 2024-03-06T18:51:33.334 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:33.338+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:33.339943+0000) 2024-03-06T18:51:34.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:34.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:34.340174+0000) 2024-03-06T18:51:35.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:35.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:35.340379+0000) 2024-03-06T18:51:36.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:36.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:36.340612+0000) 2024-03-06T18:51:37.335 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:37.339+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:37.340823+0000) 2024-03-06T18:51:38.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:38.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:38.341122+0000) 2024-03-06T18:51:39.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:39.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:39.341352+0000) 2024-03-06T18:51:40.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:40.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:40.341586+0000) 2024-03-06T18:51:41.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:41.340+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:41.341810+0000) 2024-03-06T18:51:42.336 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:42.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:42.342059+0000) 2024-03-06T18:51:43.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:43.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:43.342273+0000) 2024-03-06T18:51:44.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:44.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:44.342528+0000) 2024-03-06T18:51:45.337 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:45.341+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:45.342801+0000) 2024-03-06T18:51:46.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:46.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:46.343133+0000) 2024-03-06T18:51:47.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:47.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:47.343415+0000) 2024-03-06T18:51:48.338 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:48.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:48.343643+0000) 2024-03-06T18:51:49.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:49.342+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:49.343857+0000) 2024-03-06T18:51:50.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:50.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:50.344144+0000) 2024-03-06T18:51:51.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:51.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:51.344361+0000) 2024-03-06T18:51:52.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:52.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:52.344583+0000) 2024-03-06T18:51:53.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:53.343+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:53.344844+0000) 2024-03-06T18:51:54.339 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:54.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:54.345096+0000) 2024-03-06T18:51:55.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:55.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:55.345294+0000) 2024-03-06T18:51:56.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:56.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:56.345524+0000) 2024-03-06T18:51:57.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:57.344+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:57.345743+0000) 2024-03-06T18:51:58.340 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:58.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:58.346043+0000) 2024-03-06T18:51:59.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:51:59.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:51:59.346258+0000) 2024-03-06T18:52:00.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:00.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:00.346507+0000) 2024-03-06T18:52:00.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:00.434 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:01.341 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:01.345+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:01.346763+0000) 2024-03-06T18:52:02.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:02.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:02.347001+0000) 2024-03-06T18:52:03.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:03.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:03.347239+0000) 2024-03-06T18:52:04.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:04.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:04.347469+0000) 2024-03-06T18:52:05.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:05.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:05.347743+0000) 2024-03-06T18:52:06.342 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:06.346+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:06.347967+0000) 2024-03-06T18:52:07.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:07.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:07.348254+0000) 2024-03-06T18:52:08.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:08.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:08.348503+0000) 2024-03-06T18:52:09.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:09.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:09.348725+0000) 2024-03-06T18:52:10.343 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:10.347+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:10.348963+0000) 2024-03-06T18:52:11.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:11.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:11.349251+0000) 2024-03-06T18:52:12.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:12.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:12.349503+0000) 2024-03-06T18:52:13.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:13.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:13.349760+0000) 2024-03-06T18:52:14.344 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:14.348+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:14.349980+0000) 2024-03-06T18:52:15.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:15.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:15.350224+0000) 2024-03-06T18:52:16.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:16.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:16.350489+0000) 2024-03-06T18:52:17.345 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:17.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:17.350687+0000) 2024-03-06T18:52:18.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:18.349+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:18.350941+0000) 2024-03-06T18:52:19.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:19.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:19.351179+0000) 2024-03-06T18:52:20.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:20.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:20.351405+0000) 2024-03-06T18:52:21.346 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:21.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:21.351632+0000) 2024-03-06T18:52:22.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:22.350+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:22.351865+0000) 2024-03-06T18:52:23.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:23.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:23.352132+0000) 2024-03-06T18:52:24.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:24.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:24.352356+0000) 2024-03-06T18:52:25.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:25.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:25.352571+0000) 2024-03-06T18:52:26.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:26.351+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:26.352804+0000) 2024-03-06T18:52:27.347 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:27.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:27.353057+0000) 2024-03-06T18:52:28.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:28.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:28.353320+0000) 2024-03-06T18:52:29.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:29.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:29.353524+0000) 2024-03-06T18:52:30.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:30.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:30.353756+0000) 2024-03-06T18:52:30.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:30.572 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:52:31.348 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:31.352+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:31.353968+0000) 2024-03-06T18:52:32.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:32.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:32.354237+0000) 2024-03-06T18:52:33.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:33.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:33.354464+0000) 2024-03-06T18:52:34.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:34.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:34.354694+0000) 2024-03-06T18:52:35.349 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:35.353+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:35.354921+0000) 2024-03-06T18:52:36.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:36.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:36.355171+0000) 2024-03-06T18:52:37.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:37.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:37.355391+0000) 2024-03-06T18:52:38.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:38.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:38.355642+0000) 2024-03-06T18:52:39.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:39.354+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:39.355868+0000) 2024-03-06T18:52:40.350 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:40.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:40.356131+0000) 2024-03-06T18:52:41.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:41.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:41.356362+0000) 2024-03-06T18:52:42.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:42.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:42.356599+0000) 2024-03-06T18:52:43.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:43.355+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:43.356866+0000) 2024-03-06T18:52:44.351 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:44.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:44.357131+0000) 2024-03-06T18:52:45.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:45.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:45.357311+0000) 2024-03-06T18:52:46.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:46.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:46.357522+0000) 2024-03-06T18:52:47.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:47.356+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:47.357763+0000) 2024-03-06T18:52:48.352 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:48.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:48.358091+0000) 2024-03-06T18:52:49.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:49.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:49.358301+0000) 2024-03-06T18:52:50.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:50.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:50.358510+0000) 2024-03-06T18:52:51.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:51.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:51.358726+0000) 2024-03-06T18:52:52.353 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:52.357+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:52.358968+0000) 2024-03-06T18:52:53.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:53.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:53.359298+0000) 2024-03-06T18:52:54.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:54.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:54.359508+0000) 2024-03-06T18:52:55.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:55.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:55.359725+0000) 2024-03-06T18:52:56.354 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:56.358+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:56.359946+0000) 2024-03-06T18:52:57.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:57.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:57.360122+0000) 2024-03-06T18:52:58.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:58.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:58.360375+0000) 2024-03-06T18:52:59.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:52:59.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:52:59.360578+0000) 2024-03-06T18:53:00.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:00.359+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:00.360811+0000) 2024-03-06T18:53:00.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:00.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:01.355 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:01.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:01.361078+0000) 2024-03-06T18:53:02.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:02.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:02.361316+0000) 2024-03-06T18:53:03.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:03.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:03.361553+0000) 2024-03-06T18:53:04.356 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:04.360+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:04.361783+0000) 2024-03-06T18:53:05.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:05.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:05.362038+0000) 2024-03-06T18:53:06.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:06.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:06.362283+0000) 2024-03-06T18:53:07.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:07.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:07.362510+0000) 2024-03-06T18:53:08.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:08.361+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:08.362749+0000) 2024-03-06T18:53:09.357 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:09.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:09.363000+0000) 2024-03-06T18:53:10.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:10.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:10.363312+0000) 2024-03-06T18:53:11.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:11.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:11.363540+0000) 2024-03-06T18:53:12.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:12.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:12.363766+0000) 2024-03-06T18:53:13.358 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:13.362+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:13.363985+0000) 2024-03-06T18:53:14.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:14.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:14.364256+0000) 2024-03-06T18:53:15.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:15.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:15.364465+0000) 2024-03-06T18:53:16.359 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:16.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:16.364690+0000) 2024-03-06T18:53:17.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:17.363+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:17.364895+0000) 2024-03-06T18:53:18.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:18.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:18.365148+0000) 2024-03-06T18:53:19.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:19.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:19.365374+0000) 2024-03-06T18:53:20.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:20.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:20.365601+0000) 2024-03-06T18:53:21.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:21.364+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:21.365828+0000) 2024-03-06T18:53:22.360 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:22.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:22.366065+0000) 2024-03-06T18:53:23.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:23.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:23.366284+0000) 2024-03-06T18:53:24.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:24.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:24.366513+0000) 2024-03-06T18:53:25.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:25.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:25.366732+0000) 2024-03-06T18:53:26.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:26.365+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:26.366935+0000) 2024-03-06T18:53:27.361 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:27.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:27.367162+0000) 2024-03-06T18:53:28.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:28.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:28.367394+0000) 2024-03-06T18:53:29.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:29.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:29.367659+0000) 2024-03-06T18:53:30.362 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:30.366+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:30.367905+0000) 2024-03-06T18:53:30.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:30.886 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:53:31.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:31.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:31.368147+0000) 2024-03-06T18:53:32.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:32.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:32.368437+0000) 2024-03-06T18:53:33.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:33.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:33.368716+0000) 2024-03-06T18:53:34.363 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:34.367+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:34.368956+0000) 2024-03-06T18:53:35.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:35.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:35.369121+0000) 2024-03-06T18:53:36.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:36.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:36.369315+0000) 2024-03-06T18:53:37.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:37.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:37.369514+0000) 2024-03-06T18:53:38.364 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:38.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:38.369746+0000) 2024-03-06T18:53:39.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:39.368+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:39.369961+0000) 2024-03-06T18:53:40.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:40.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:40.370199+0000) 2024-03-06T18:53:41.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:41.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:41.370387+0000) 2024-03-06T18:53:42.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:42.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:42.370582+0000) 2024-03-06T18:53:43.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:43.369+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:43.370797+0000) 2024-03-06T18:53:44.365 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:44.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:44.371037+0000) 2024-03-06T18:53:45.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:45.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:45.371222+0000) 2024-03-06T18:53:46.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:46.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:46.371501+0000) 2024-03-06T18:53:47.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:47.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:47.371676+0000) 2024-03-06T18:53:48.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:48.370+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:48.371920+0000) 2024-03-06T18:53:49.366 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:49.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:49.372158+0000) 2024-03-06T18:53:50.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:50.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:50.372386+0000) 2024-03-06T18:53:51.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:51.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:51.372613+0000) 2024-03-06T18:53:52.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:52.371+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:52.372848+0000) 2024-03-06T18:53:53.367 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:53.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:53.373095+0000) 2024-03-06T18:53:54.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:54.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:54.373250+0000) 2024-03-06T18:53:55.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:55.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:55.373452+0000) 2024-03-06T18:53:56.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:56.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:56.373660+0000) 2024-03-06T18:53:57.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:57.372+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:57.373882+0000) 2024-03-06T18:53:58.368 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:58.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:58.374134+0000) 2024-03-06T18:53:59.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:53:59.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:53:59.374283+0000) 2024-03-06T18:54:00.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:00.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:00.374506+0000) 2024-03-06T18:54:01.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:01.025 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:01.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:01.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:01.374686+0000) 2024-03-06T18:54:02.369 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:02.373+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:02.374866+0000) 2024-03-06T18:54:03.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:03.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:03.375121+0000) 2024-03-06T18:54:04.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:04.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:04.375345+0000) 2024-03-06T18:54:05.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:05.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:05.375559+0000) 2024-03-06T18:54:06.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:06.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:06.375756+0000) 2024-03-06T18:54:07.370 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:07.374+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:07.375970+0000) 2024-03-06T18:54:08.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:08.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:08.376240+0000) 2024-03-06T18:54:09.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:09.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:09.376506+0000) 2024-03-06T18:54:10.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:10.375+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:10.376764+0000) 2024-03-06T18:54:11.371 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:11.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:11.377077+0000) 2024-03-06T18:54:12.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:12.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:12.377357+0000) 2024-03-06T18:54:13.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:13.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:13.377586+0000) 2024-03-06T18:54:14.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:14.376+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:14.377838+0000) 2024-03-06T18:54:15.372 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:15.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:15.378081+0000) 2024-03-06T18:54:16.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:16.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:16.378312+0000) 2024-03-06T18:54:17.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:17.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:17.378533+0000) 2024-03-06T18:54:18.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:18.377+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:18.378775+0000) 2024-03-06T18:54:19.373 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:19.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:19.379001+0000) 2024-03-06T18:54:20.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:20.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:20.379267+0000) 2024-03-06T18:54:21.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:21.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:21.379524+0000) 2024-03-06T18:54:22.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:22.378+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:22.379799+0000) 2024-03-06T18:54:23.374 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:23.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:23.380058+0000) 2024-03-06T18:54:24.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:24.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:24.380300+0000) 2024-03-06T18:54:25.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:25.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:25.380530+0000) 2024-03-06T18:54:26.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:26.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:26.380711+0000) 2024-03-06T18:54:27.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:27.379+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:27.380927+0000) 2024-03-06T18:54:28.375 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:28.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:28.381165+0000) 2024-03-06T18:54:29.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:29.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:29.381426+0000) 2024-03-06T18:54:30.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:30.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:30.381624+0000) 2024-03-06T18:54:31.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:31.173 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:54:31.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:31.380+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:31.381819+0000) 2024-03-06T18:54:32.376 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:32.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:32.382072+0000) 2024-03-06T18:54:33.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:33.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:33.382298+0000) 2024-03-06T18:54:34.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:34.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:34.382501+0000) 2024-03-06T18:54:35.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:35.381+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:35.382781+0000) 2024-03-06T18:54:36.377 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:36.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:36.383114+0000) 2024-03-06T18:54:37.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:37.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:37.383317+0000) 2024-03-06T18:54:38.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:38.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:38.383557+0000) 2024-03-06T18:54:39.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:39.382+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:39.383835+0000) 2024-03-06T18:54:40.378 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:40.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:40.384123+0000) 2024-03-06T18:54:41.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:41.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:41.384319+0000) 2024-03-06T18:54:42.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:42.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:42.384557+0000) 2024-03-06T18:54:43.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:43.383+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:43.384779+0000) 2024-03-06T18:54:44.379 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:44.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:44.385038+0000) 2024-03-06T18:54:45.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:45.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:45.385272+0000) 2024-03-06T18:54:46.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:46.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:46.385504+0000) 2024-03-06T18:54:47.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:47.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:47.385731+0000) 2024-03-06T18:54:48.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:48.384+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:48.385968+0000) 2024-03-06T18:54:49.380 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:49.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:49.386193+0000) 2024-03-06T18:54:50.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:50.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:50.386423+0000) 2024-03-06T18:54:51.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:51.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:51.386604+0000) 2024-03-06T18:54:52.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:52.385+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:52.386824+0000) 2024-03-06T18:54:53.381 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:53.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:53.387112+0000) 2024-03-06T18:54:54.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:54.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:54.387347+0000) 2024-03-06T18:54:55.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:55.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:55.387566+0000) 2024-03-06T18:54:56.382 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:56.386+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:56.387807+0000) 2024-03-06T18:54:57.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:57.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:57.388053+0000) 2024-03-06T18:54:58.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:58.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:58.388295+0000) 2024-03-06T18:54:59.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:54:59.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:54:59.388518+0000) 2024-03-06T18:55:00.383 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:00.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:00.388746+0000) 2024-03-06T18:55:01.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:01.308 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:01.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:01.387+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:01.388928+0000) 2024-03-06T18:55:02.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:02.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:02.389172+0000) 2024-03-06T18:55:03.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:03.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:03.389409+0000) 2024-03-06T18:55:04.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:04.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:04.389667+0000) 2024-03-06T18:55:05.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:05.388+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:05.389930+0000) 2024-03-06T18:55:06.384 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:06.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:06.390175+0000) 2024-03-06T18:55:07.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:07.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:07.390395+0000) 2024-03-06T18:55:08.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:08.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:08.390638+0000) 2024-03-06T18:55:09.385 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:09.389+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:09.390865+0000) 2024-03-06T18:55:10.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:10.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:10.391118+0000) 2024-03-06T18:55:11.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:11.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:11.391348+0000) 2024-03-06T18:55:12.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:12.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:12.391579+0000) 2024-03-06T18:55:13.386 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:13.390+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:13.391802+0000) 2024-03-06T18:55:14.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:14.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:14.392074+0000) 2024-03-06T18:55:15.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:15.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:15.392298+0000) 2024-03-06T18:55:16.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:16.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:16.392547+0000) 2024-03-06T18:55:17.387 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:17.391+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:17.392783+0000) 2024-03-06T18:55:18.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:18.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:18.393074+0000) 2024-03-06T18:55:19.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:19.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:19.393310+0000) 2024-03-06T18:55:20.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:20.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:20.393547+0000) 2024-03-06T18:55:21.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:21.392+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:21.393780+0000) 2024-03-06T18:55:22.388 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:22.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:22.394072+0000) 2024-03-06T18:55:23.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:23.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:23.394308+0000) 2024-03-06T18:55:24.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:24.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:24.394557+0000) 2024-03-06T18:55:25.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:25.393+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:25.394794+0000) 2024-03-06T18:55:26.389 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:26.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:26.395076+0000) 2024-03-06T18:55:27.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:27.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:27.395315+0000) 2024-03-06T18:55:28.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:28.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:28.395568+0000) 2024-03-06T18:55:29.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:29.394+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:29.395803+0000) 2024-03-06T18:55:30.390 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:30.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:30.396092+0000) 2024-03-06T18:55:31.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:31.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:31.396346+0000) 2024-03-06T18:55:31.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:31.449 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:55:32.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:32.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:32.396616+0000) 2024-03-06T18:55:33.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:33.395+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:33.396938+0000) 2024-03-06T18:55:34.391 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:34.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:34.397187+0000) 2024-03-06T18:55:35.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:35.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:35.397395+0000) 2024-03-06T18:55:36.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:36.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:36.397660+0000) 2024-03-06T18:55:37.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:37.396+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:37.397856+0000) 2024-03-06T18:55:38.392 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:38.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:38.398119+0000) 2024-03-06T18:55:39.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:39.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:39.398360+0000) 2024-03-06T18:55:40.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:40.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:40.398613+0000) 2024-03-06T18:55:41.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:41.397+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:41.398845+0000) 2024-03-06T18:55:42.393 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:42.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:42.399156+0000) 2024-03-06T18:55:43.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:43.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:43.399383+0000) 2024-03-06T18:55:44.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:44.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:44.399640+0000) 2024-03-06T18:55:45.394 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:45.398+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:45.399910+0000) 2024-03-06T18:55:46.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:46.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:46.400158+0000) 2024-03-06T18:55:47.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:47.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:47.400390+0000) 2024-03-06T18:55:48.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:48.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:48.400578+0000) 2024-03-06T18:55:49.395 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:49.399+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:49.400798+0000) 2024-03-06T18:55:50.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:50.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:50.401060+0000) 2024-03-06T18:55:51.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:51.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:51.401290+0000) 2024-03-06T18:55:52.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:52.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:52.401524+0000) 2024-03-06T18:55:53.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:53.400+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:53.401770+0000) 2024-03-06T18:55:54.396 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:54.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:54.402054+0000) 2024-03-06T18:55:55.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:55.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:55.402293+0000) 2024-03-06T18:55:56.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:56.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:56.402547+0000) 2024-03-06T18:55:57.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:57.401+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:57.402789+0000) 2024-03-06T18:55:58.397 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:58.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:58.402994+0000) 2024-03-06T18:55:59.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:55:59.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:55:59.403260+0000) 2024-03-06T18:56:00.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:00.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:00.403537+0000) 2024-03-06T18:56:01.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:01.402+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:01.403811+0000) 2024-03-06T18:56:01.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:01.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:02.398 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:02.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:02.404116+0000) 2024-03-06T18:56:03.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:03.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:03.404349+0000) 2024-03-06T18:56:04.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:04.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:04.404612+0000) 2024-03-06T18:56:05.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:05.403+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:05.404849+0000) 2024-03-06T18:56:06.399 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:06.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:06.405085+0000) 2024-03-06T18:56:07.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:07.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:07.405303+0000) 2024-03-06T18:56:08.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:08.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:08.405591+0000) 2024-03-06T18:56:09.400 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:09.404+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:09.405837+0000) 2024-03-06T18:56:10.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:10.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:10.406120+0000) 2024-03-06T18:56:11.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:11.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:11.406359+0000) 2024-03-06T18:56:12.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:12.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:12.406560+0000) 2024-03-06T18:56:13.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:13.405+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:13.406781+0000) 2024-03-06T18:56:14.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:14.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:14.406993+0000) 2024-03-06T18:56:15.401 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:15.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:15.407245+0000) 2024-03-06T18:56:16.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:16.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:16.407481+0000) 2024-03-06T18:56:17.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:17.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:17.407697+0000) 2024-03-06T18:56:18.402 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:18.406+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:18.407933+0000) 2024-03-06T18:56:19.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:19.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:19.408158+0000) 2024-03-06T18:56:20.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:20.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:20.408384+0000) 2024-03-06T18:56:21.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:21.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:21.408565+0000) 2024-03-06T18:56:22.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:22.407+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:22.408791+0000) 2024-03-06T18:56:23.403 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:23.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:23.409054+0000) 2024-03-06T18:56:24.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:24.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:24.409324+0000) 2024-03-06T18:56:25.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:25.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:25.409529+0000) 2024-03-06T18:56:26.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:26.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:26.409720+0000) 2024-03-06T18:56:27.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:27.408+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:27.409948+0000) 2024-03-06T18:56:28.404 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:28.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:28.410166+0000) 2024-03-06T18:56:29.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:29.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:29.410379+0000) 2024-03-06T18:56:30.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:30.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:30.410653+0000) 2024-03-06T18:56:31.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:31.409+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:31.410858+0000) 2024-03-06T18:56:31.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:31.837 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:56:32.405 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:32.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:32.411147+0000) 2024-03-06T18:56:33.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:33.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:33.411339+0000) 2024-03-06T18:56:34.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:34.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:34.411552+0000) 2024-03-06T18:56:35.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:35.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:35.411761+0000) 2024-03-06T18:56:36.406 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:36.410+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:36.411978+0000) 2024-03-06T18:56:37.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:37.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:37.412241+0000) 2024-03-06T18:56:38.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:38.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:38.412472+0000) 2024-03-06T18:56:39.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:39.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:39.412696+0000) 2024-03-06T18:56:40.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:40.411+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:40.412927+0000) 2024-03-06T18:56:41.407 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:41.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:41.413152+0000) 2024-03-06T18:56:42.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:42.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:42.413378+0000) 2024-03-06T18:56:43.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:43.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:43.413601+0000) 2024-03-06T18:56:44.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:44.412+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:44.413838+0000) 2024-03-06T18:56:45.408 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:45.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:45.414107+0000) 2024-03-06T18:56:46.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:46.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:46.414341+0000) 2024-03-06T18:56:47.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:47.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:47.414598+0000) 2024-03-06T18:56:48.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:48.413+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:48.414850+0000) 2024-03-06T18:56:49.409 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:49.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:49.415149+0000) 2024-03-06T18:56:50.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:50.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:50.415366+0000) 2024-03-06T18:56:51.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:51.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:51.415569+0000) 2024-03-06T18:56:52.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:52.414+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:52.415798+0000) 2024-03-06T18:56:53.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:53.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:53.416058+0000) 2024-03-06T18:56:54.410 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:54.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:54.416263+0000) 2024-03-06T18:56:55.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:55.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:55.416466+0000) 2024-03-06T18:56:56.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:56.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:56.416693+0000) 2024-03-06T18:56:57.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:57.415+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:57.416885+0000) 2024-03-06T18:56:58.411 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:58.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:58.417149+0000) 2024-03-06T18:56:59.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:56:59.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:56:59.417341+0000) 2024-03-06T18:57:00.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:00.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:00.417542+0000) 2024-03-06T18:57:01.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:01.416+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:01.417753+0000) 2024-03-06T18:57:01.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:01.978 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:02.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:02.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:02.417996+0000) 2024-03-06T18:57:03.412 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:03.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:03.418243+0000) 2024-03-06T18:57:04.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:04.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:04.418489+0000) 2024-03-06T18:57:05.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:05.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:05.418705+0000) 2024-03-06T18:57:06.413 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:06.417+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:06.418927+0000) 2024-03-06T18:57:07.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:07.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:07.419149+0000) 2024-03-06T18:57:08.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:08.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:08.419357+0000) 2024-03-06T18:57:09.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:09.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:09.419567+0000) 2024-03-06T18:57:10.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:10.418+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:10.419809+0000) 2024-03-06T18:57:11.414 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:11.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:11.420082+0000) 2024-03-06T18:57:12.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:12.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:12.420295+0000) 2024-03-06T18:57:13.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:13.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:13.420529+0000) 2024-03-06T18:57:14.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:14.419+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:14.420788+0000) 2024-03-06T18:57:15.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:15.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:15.421043+0000) 2024-03-06T18:57:16.415 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:16.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:16.421257+0000) 2024-03-06T18:57:17.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:17.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:17.421479+0000) 2024-03-06T18:57:18.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:18.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:18.421712+0000) 2024-03-06T18:57:19.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:19.420+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:19.421929+0000) 2024-03-06T18:57:20.416 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:20.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:20.422173+0000) 2024-03-06T18:57:21.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:21.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:21.422349+0000) 2024-03-06T18:57:22.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:22.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:22.422581+0000) 2024-03-06T18:57:23.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:23.421+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:23.422812+0000) 2024-03-06T18:57:24.417 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:24.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:24.423122+0000) 2024-03-06T18:57:25.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:25.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:25.423349+0000) 2024-03-06T18:57:26.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:26.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:26.423578+0000) 2024-03-06T18:57:27.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:27.422+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:27.423813+0000) 2024-03-06T18:57:28.418 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:28.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:28.424089+0000) 2024-03-06T18:57:29.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:29.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:29.424291+0000) 2024-03-06T18:57:30.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:30.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:30.424510+0000) 2024-03-06T18:57:31.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:31.423+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:31.424744+0000) 2024-03-06T18:57:32.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:32.186 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:57:32.419 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:32.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:32.425043+0000) 2024-03-06T18:57:33.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:33.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:33.425257+0000) 2024-03-06T18:57:34.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:34.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:34.425490+0000) 2024-03-06T18:57:35.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:35.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:35.425711+0000) 2024-03-06T18:57:36.420 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:36.424+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:36.425933+0000) 2024-03-06T18:57:37.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:37.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:37.426197+0000) 2024-03-06T18:57:38.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:38.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:38.426386+0000) 2024-03-06T18:57:39.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:39.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:39.426641+0000) 2024-03-06T18:57:40.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:40.425+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:40.426854+0000) 2024-03-06T18:57:41.421 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:41.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:41.427130+0000) 2024-03-06T18:57:42.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:42.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:42.427352+0000) 2024-03-06T18:57:43.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:43.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:43.427532+0000) 2024-03-06T18:57:44.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:44.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:44.427749+0000) 2024-03-06T18:57:45.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:45.426+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:45.427942+0000) 2024-03-06T18:57:46.422 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:46.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:46.428175+0000) 2024-03-06T18:57:47.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:47.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:47.428407+0000) 2024-03-06T18:57:48.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:48.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:48.428652+0000) 2024-03-06T18:57:49.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:49.427+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:49.428908+0000) 2024-03-06T18:57:50.423 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:50.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:50.429190+0000) 2024-03-06T18:57:51.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:51.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:51.429467+0000) 2024-03-06T18:57:52.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:52.428+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:52.429792+0000) 2024-03-06T18:57:53.424 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:53.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:53.430052+0000) 2024-03-06T18:57:54.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:54.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:54.430326+0000) 2024-03-06T18:57:55.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:55.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:55.430546+0000) 2024-03-06T18:57:56.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:56.429+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:56.430775+0000) 2024-03-06T18:57:57.425 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:57.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:57.431001+0000) 2024-03-06T18:57:58.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:58.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:58.431289+0000) 2024-03-06T18:57:59.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:57:59.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:57:59.431527+0000) 2024-03-06T18:58:00.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:00.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:00.431755+0000) 2024-03-06T18:58:01.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:01.430+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:01.431931+0000) 2024-03-06T18:58:02.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:02.416 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:02.426 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:02.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:02.432209+0000) 2024-03-06T18:58:03.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:03.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:03.432487+0000) 2024-03-06T18:58:04.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:04.431+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:04.432789+0000) 2024-03-06T18:58:05.427 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:05.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:05.433061+0000) 2024-03-06T18:58:06.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:06.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:06.433303+0000) 2024-03-06T18:58:07.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:07.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:07.433561+0000) 2024-03-06T18:58:08.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:08.432+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:08.433799+0000) 2024-03-06T18:58:09.428 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:09.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:09.434095+0000) 2024-03-06T18:58:10.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:10.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:10.434334+0000) 2024-03-06T18:58:11.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:11.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:11.434551+0000) 2024-03-06T18:58:12.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:12.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:12.434758+0000) 2024-03-06T18:58:13.429 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:13.433+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:13.434990+0000) 2024-03-06T18:58:14.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:14.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:14.435281+0000) 2024-03-06T18:58:15.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:15.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:15.435464+0000) 2024-03-06T18:58:16.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:16.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:16.435673+0000) 2024-03-06T18:58:17.430 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:17.434+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:17.435832+0000) 2024-03-06T18:58:18.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:18.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:18.436129+0000) 2024-03-06T18:58:19.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:19.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:19.436346+0000) 2024-03-06T18:58:20.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:20.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:20.436524+0000) 2024-03-06T18:58:21.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:21.435+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:21.436762+0000) 2024-03-06T18:58:22.431 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:22.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:22.437038+0000) 2024-03-06T18:58:23.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:23.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:23.437249+0000) 2024-03-06T18:58:24.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:24.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:24.437517+0000) 2024-03-06T18:58:25.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:25.436+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:25.437753+0000) 2024-03-06T18:58:26.432 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:26.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:26.437998+0000) 2024-03-06T18:58:27.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:27.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:27.438301+0000) 2024-03-06T18:58:28.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:28.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:28.438538+0000) 2024-03-06T18:58:29.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:29.437+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:29.438783+0000) 2024-03-06T18:58:30.433 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:30.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:30.439054+0000) 2024-03-06T18:58:31.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:31.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:31.439294+0000) 2024-03-06T18:58:32.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:32.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:32.439517+0000) 2024-03-06T18:58:32.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:32.631 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:58:33.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:33.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:33.439726+0000) 2024-03-06T18:58:34.434 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:34.438+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:34.439990+0000) 2024-03-06T18:58:35.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:35.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:35.440211+0000) 2024-03-06T18:58:36.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:36.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:36.440405+0000) 2024-03-06T18:58:37.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:37.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:37.440616+0000) 2024-03-06T18:58:38.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:38.439+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:38.440861+0000) 2024-03-06T18:58:39.435 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:39.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:39.441120+0000) 2024-03-06T18:58:40.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:40.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:40.441373+0000) 2024-03-06T18:58:41.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:41.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:41.441595+0000) 2024-03-06T18:58:42.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:42.440+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:42.441840+0000) 2024-03-06T18:58:43.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:43.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:43.442069+0000) 2024-03-06T18:58:44.436 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:44.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:44.442312+0000) 2024-03-06T18:58:45.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:45.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:45.442561+0000) 2024-03-06T18:58:46.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:46.441+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:46.442789+0000) 2024-03-06T18:58:47.437 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:47.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:47.443052+0000) 2024-03-06T18:58:48.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:48.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:48.443278+0000) 2024-03-06T18:58:49.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:49.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:49.443506+0000) 2024-03-06T18:58:50.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:50.442+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:50.443784+0000) 2024-03-06T18:58:51.438 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:51.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:51.444054+0000) 2024-03-06T18:58:52.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:52.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:52.444282+0000) 2024-03-06T18:58:53.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:53.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:53.444504+0000) 2024-03-06T18:58:54.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:54.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:54.444737+0000) 2024-03-06T18:58:55.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:55.443+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:55.444976+0000) 2024-03-06T18:58:56.439 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:56.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:56.445234+0000) 2024-03-06T18:58:57.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:57.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:57.445443+0000) 2024-03-06T18:58:58.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:58.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:58.445680+0000) 2024-03-06T18:58:59.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:58:59.444+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:58:59.445924+0000) 2024-03-06T18:59:00.440 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:00.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:00.446164+0000) 2024-03-06T18:59:01.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:01.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:01.446373+0000) 2024-03-06T18:59:02.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:02.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:02.446596+0000) 2024-03-06T18:59:02.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:02.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:03.441 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:03.445+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:03.446839+0000) 2024-03-06T18:59:04.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:04.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:04.447133+0000) 2024-03-06T18:59:05.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:05.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:05.447368+0000) 2024-03-06T18:59:06.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:06.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:06.447586+0000) 2024-03-06T18:59:07.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:07.446+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:07.447792+0000) 2024-03-06T18:59:08.442 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:08.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:08.448065+0000) 2024-03-06T18:59:09.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:09.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:09.448305+0000) 2024-03-06T18:59:10.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:10.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:10.448586+0000) 2024-03-06T18:59:11.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:11.447+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:11.448871+0000) 2024-03-06T18:59:12.443 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:12.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:12.449143+0000) 2024-03-06T18:59:13.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:13.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:13.449317+0000) 2024-03-06T18:59:14.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:14.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:14.449553+0000) 2024-03-06T18:59:15.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:15.448+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:15.449767+0000) 2024-03-06T18:59:16.444 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:16.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:16.450048+0000) 2024-03-06T18:59:17.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:17.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:17.450250+0000) 2024-03-06T18:59:18.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:18.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:18.450557+0000) 2024-03-06T18:59:19.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:19.449+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:19.450795+0000) 2024-03-06T18:59:20.445 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:20.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:20.451049+0000) 2024-03-06T18:59:21.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:21.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:21.451289+0000) 2024-03-06T18:59:22.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:22.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:22.451516+0000) 2024-03-06T18:59:23.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:23.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:23.451742+0000) 2024-03-06T18:59:24.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:24.450+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:24.451939+0000) 2024-03-06T18:59:25.446 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:25.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:25.452174+0000) 2024-03-06T18:59:26.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:26.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:26.452401+0000) 2024-03-06T18:59:27.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:27.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:27.452623+0000) 2024-03-06T18:59:28.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:28.451+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:28.452862+0000) 2024-03-06T18:59:29.447 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:29.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:29.453122+0000) 2024-03-06T18:59:30.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:30.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:30.453352+0000) 2024-03-06T18:59:31.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:31.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:31.453585+0000) 2024-03-06T18:59:32.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:32.452+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:32.453813+0000) 2024-03-06T18:59:32.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:32.937 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T18:59:33.448 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:33.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:33.454072+0000) 2024-03-06T18:59:34.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:34.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:34.454301+0000) 2024-03-06T18:59:35.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:35.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:35.454490+0000) 2024-03-06T18:59:36.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:36.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:36.454733+0000) 2024-03-06T18:59:37.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:37.453+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:37.454959+0000) 2024-03-06T18:59:38.449 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:38.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:38.455258+0000) 2024-03-06T18:59:39.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:39.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:39.455480+0000) 2024-03-06T18:59:40.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:40.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:40.455712+0000) 2024-03-06T18:59:41.450 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:41.454+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:41.455908+0000) 2024-03-06T18:59:42.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:42.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:42.456149+0000) 2024-03-06T18:59:43.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:43.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:43.456372+0000) 2024-03-06T18:59:44.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:44.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:44.456581+0000) 2024-03-06T18:59:45.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:45.455+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:45.456807+0000) 2024-03-06T18:59:46.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:46.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:46.457032+0000) 2024-03-06T18:59:47.451 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:47.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:47.457278+0000) 2024-03-06T18:59:48.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:48.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:48.457523+0000) 2024-03-06T18:59:49.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:49.456+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:49.457768+0000) 2024-03-06T18:59:50.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:50.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:50.458037+0000) 2024-03-06T18:59:51.452 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:51.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:51.458281+0000) 2024-03-06T18:59:52.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:52.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:52.458498+0000) 2024-03-06T18:59:53.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:53.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:53.458714+0000) 2024-03-06T18:59:54.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:54.457+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:54.458912+0000) 2024-03-06T18:59:55.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:55.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:55.459146+0000) 2024-03-06T18:59:56.453 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:56.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:56.459298+0000) 2024-03-06T18:59:57.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:57.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:57.459524+0000) 2024-03-06T18:59:58.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:58.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:58.459765+0000) 2024-03-06T18:59:59.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T18:59:59.458+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T17:59:59.459982+0000) 2024-03-06T19:00:00.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:00.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:00.460242+0000) 2024-03-06T19:00:01.454 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:01.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:01.460472+0000) 2024-03-06T19:00:02.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:02.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:02.460687+0000) 2024-03-06T19:00:03.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:03.078 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:03.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:03.459+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:03.460887+0000) 2024-03-06T19:00:04.455 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:04.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:04.461143+0000) 2024-03-06T19:00:05.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:05.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:05.461362+0000) 2024-03-06T19:00:06.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:06.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:06.461619+0000) 2024-03-06T19:00:07.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:07.460+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:07.461854+0000) 2024-03-06T19:00:08.456 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:08.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:08.462130+0000) 2024-03-06T19:00:09.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:09.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:09.462302+0000) 2024-03-06T19:00:10.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:10.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:10.462534+0000) 2024-03-06T19:00:11.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:11.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:11.462730+0000) 2024-03-06T19:00:12.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:12.461+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:12.462956+0000) 2024-03-06T19:00:13.457 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:13.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:13.463159+0000) 2024-03-06T19:00:14.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:14.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:14.463349+0000) 2024-03-06T19:00:15.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:15.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:15.463553+0000) 2024-03-06T19:00:16.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:16.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:16.463785+0000) 2024-03-06T19:00:17.458 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:17.462+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:17.463972+0000) 2024-03-06T19:00:18.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:18.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:18.464273+0000) 2024-03-06T19:00:19.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:19.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:19.464488+0000) 2024-03-06T19:00:20.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:20.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:20.464745+0000) 2024-03-06T19:00:21.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:21.463+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:21.464982+0000) 2024-03-06T19:00:22.459 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:22.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:22.465285+0000) 2024-03-06T19:00:23.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:23.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:23.465528+0000) 2024-03-06T19:00:24.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:24.464+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:24.465773+0000) 2024-03-06T19:00:25.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:25.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:25.466052+0000) 2024-03-06T19:00:26.460 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:26.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:26.466333+0000) 2024-03-06T19:00:27.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:27.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:27.466555+0000) 2024-03-06T19:00:28.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:28.465+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:28.466806+0000) 2024-03-06T19:00:29.461 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:29.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:29.467118+0000) 2024-03-06T19:00:30.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:30.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:30.467411+0000) 2024-03-06T19:00:31.462 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:31.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:31.467655+0000) 2024-03-06T19:00:32.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:32.466+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:32.467945+0000) 2024-03-06T19:00:33.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:33.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:00:33.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:33.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:33.468195+0000) 2024-03-06T19:00:34.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:34.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:34.468449+0000) 2024-03-06T19:00:35.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:35.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:35.468672+0000) 2024-03-06T19:00:36.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:36.467+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:36.468861+0000) 2024-03-06T19:00:37.463 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:37.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:37.469127+0000) 2024-03-06T19:00:38.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:38.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:38.469370+0000) 2024-03-06T19:00:39.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:39.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:39.469574+0000) 2024-03-06T19:00:40.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:40.468+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:40.469814+0000) 2024-03-06T19:00:41.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:41.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:41.470081+0000) 2024-03-06T19:00:42.464 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:42.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:42.470282+0000) 2024-03-06T19:00:43.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:43.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:43.470507+0000) 2024-03-06T19:00:44.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:44.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:44.470725+0000) 2024-03-06T19:00:45.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:45.469+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:45.470972+0000) 2024-03-06T19:00:46.465 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:46.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:46.471243+0000) 2024-03-06T19:00:47.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:47.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:47.471460+0000) 2024-03-06T19:00:48.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:48.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:48.471682+0000) 2024-03-06T19:00:49.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:49.470+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:49.471906+0000) 2024-03-06T19:00:50.466 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:50.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:50.472152+0000) 2024-03-06T19:00:51.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:51.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:51.472348+0000) 2024-03-06T19:00:52.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:52.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:52.472577+0000) 2024-03-06T19:00:53.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:53.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:53.472755+0000) 2024-03-06T19:00:54.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:54.471+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:54.472940+0000) 2024-03-06T19:00:55.467 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:55.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:55.473184+0000) 2024-03-06T19:00:56.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:56.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:56.473447+0000) 2024-03-06T19:00:57.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:57.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:57.473704+0000) 2024-03-06T19:00:58.468 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:58.472+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:58.473919+0000) 2024-03-06T19:00:59.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:00:59.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:00:59.474199+0000) 2024-03-06T19:01:00.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:00.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:00.474466+0000) 2024-03-06T19:01:01.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:01.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:01.474693+0000) 2024-03-06T19:01:02.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:02.473+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:02.474929+0000) 2024-03-06T19:01:03.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:03.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:03.475151+0000) 2024-03-06T19:01:03.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:03.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:04.469 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:04.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:04.475367+0000) 2024-03-06T19:01:05.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:05.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:05.475628+0000) 2024-03-06T19:01:06.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:06.474+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:06.475914+0000) 2024-03-06T19:01:07.470 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:07.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:07.476203+0000) 2024-03-06T19:01:08.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:08.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:08.476507+0000) 2024-03-06T19:01:09.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:09.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:09.476705+0000) 2024-03-06T19:01:10.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:10.475+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:10.476955+0000) 2024-03-06T19:01:11.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:11.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:11.477180+0000) 2024-03-06T19:01:12.471 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:12.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:12.477393+0000) 2024-03-06T19:01:13.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:13.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:13.477620+0000) 2024-03-06T19:01:14.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:14.476+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:14.477915+0000) 2024-03-06T19:01:15.472 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:15.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:15.478162+0000) 2024-03-06T19:01:16.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:16.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:16.478399+0000) 2024-03-06T19:01:17.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:17.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:17.478632+0000) 2024-03-06T19:01:18.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:18.477+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:18.478883+0000) 2024-03-06T19:01:19.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:19.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:19.479127+0000) 2024-03-06T19:01:20.473 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:20.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:20.479344+0000) 2024-03-06T19:01:21.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:21.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:21.479563+0000) 2024-03-06T19:01:22.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:22.478+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:22.479807+0000) 2024-03-06T19:01:23.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:23.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:23.480104+0000) 2024-03-06T19:01:24.474 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:24.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:24.480321+0000) 2024-03-06T19:01:25.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:25.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:25.480549+0000) 2024-03-06T19:01:26.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:26.479+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:26.480791+0000) 2024-03-06T19:01:27.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:27.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:27.481056+0000) 2024-03-06T19:01:28.475 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:28.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:28.481312+0000) 2024-03-06T19:01:29.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:29.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:29.481531+0000) 2024-03-06T19:01:30.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:30.480+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:30.481792+0000) 2024-03-06T19:01:31.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:31.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:31.482054+0000) 2024-03-06T19:01:32.476 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:32.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:32.482268+0000) 2024-03-06T19:01:33.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:33.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:33.482473+0000) 2024-03-06T19:01:33.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:33.751 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:01:34.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:34.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:34.482693+0000) 2024-03-06T19:01:35.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:35.481+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:35.482935+0000) 2024-03-06T19:01:36.477 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:36.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:36.483165+0000) 2024-03-06T19:01:37.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:37.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:37.483350+0000) 2024-03-06T19:01:38.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:38.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:38.483568+0000) 2024-03-06T19:01:39.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:39.482+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:39.483750+0000) 2024-03-06T19:01:40.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:40.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:40.483992+0000) 2024-03-06T19:01:41.478 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:41.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:41.484311+0000) 2024-03-06T19:01:42.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:42.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:42.484595+0000) 2024-03-06T19:01:43.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:43.483+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:43.484798+0000) 2024-03-06T19:01:44.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:44.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:44.485036+0000) 2024-03-06T19:01:45.479 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:45.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:45.485258+0000) 2024-03-06T19:01:46.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:46.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:46.485492+0000) 2024-03-06T19:01:47.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:47.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:47.485724+0000) 2024-03-06T19:01:48.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:48.484+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:48.485955+0000) 2024-03-06T19:01:49.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:49.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:49.486159+0000) 2024-03-06T19:01:50.480 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:50.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:50.486389+0000) 2024-03-06T19:01:51.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:51.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:51.486591+0000) 2024-03-06T19:01:52.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:52.485+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:52.486820+0000) 2024-03-06T19:01:53.481 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:53.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:53.487065+0000) 2024-03-06T19:01:54.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:54.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:54.487328+0000) 2024-03-06T19:01:55.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:55.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:55.487587+0000) 2024-03-06T19:01:56.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:56.486+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:56.487822+0000) 2024-03-06T19:01:57.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:57.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:57.488038+0000) 2024-03-06T19:01:58.482 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:58.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:58.488287+0000) 2024-03-06T19:01:59.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:01:59.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:01:59.488479+0000) 2024-03-06T19:02:00.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:00.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:00.488710+0000) 2024-03-06T19:02:01.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:01.487+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:01.488925+0000) 2024-03-06T19:02:02.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:02.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:02.489151+0000) 2024-03-06T19:02:03.483 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:03.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:03.489395+0000) 2024-03-06T19:02:03.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:03.981 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:04.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:04.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:04.489629+0000) 2024-03-06T19:02:05.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:05.488+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:05.489821+0000) 2024-03-06T19:02:06.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:06.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:06.490091+0000) 2024-03-06T19:02:07.484 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:07.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:07.490322+0000) 2024-03-06T19:02:08.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:08.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:08.490559+0000) 2024-03-06T19:02:09.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:09.489+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:09.490765+0000) 2024-03-06T19:02:10.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:10.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:10.491036+0000) 2024-03-06T19:02:11.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:11.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:11.491240+0000) 2024-03-06T19:02:12.485 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:12.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:12.491467+0000) 2024-03-06T19:02:13.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:13.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:13.491685+0000) 2024-03-06T19:02:14.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:14.490+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:14.491921+0000) 2024-03-06T19:02:15.486 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:15.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:15.492160+0000) 2024-03-06T19:02:16.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:16.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:16.492415+0000) 2024-03-06T19:02:17.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:17.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:17.492640+0000) 2024-03-06T19:02:18.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:18.491+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:18.492909+0000) 2024-03-06T19:02:19.487 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:19.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:19.493189+0000) 2024-03-06T19:02:20.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:20.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:20.493468+0000) 2024-03-06T19:02:21.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:21.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:21.493666+0000) 2024-03-06T19:02:22.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:22.492+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:22.493880+0000) 2024-03-06T19:02:23.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:23.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:23.494158+0000) 2024-03-06T19:02:24.488 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:24.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:24.494429+0000) 2024-03-06T19:02:25.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:25.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:25.494669+0000) 2024-03-06T19:02:26.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:26.493+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:26.494912+0000) 2024-03-06T19:02:27.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:27.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:27.495122+0000) 2024-03-06T19:02:28.489 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:28.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:28.495334+0000) 2024-03-06T19:02:29.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:29.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:29.495556+0000) 2024-03-06T19:02:30.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:30.494+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:30.495789+0000) 2024-03-06T19:02:31.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:31.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:31.496038+0000) 2024-03-06T19:02:32.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:32.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:32.496244+0000) 2024-03-06T19:02:33.490 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:33.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:33.496464+0000) 2024-03-06T19:02:34.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:34.203 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:02:34.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:34.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:34.496687+0000) 2024-03-06T19:02:35.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:35.495+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:35.496914+0000) 2024-03-06T19:02:36.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:36.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:36.497160+0000) 2024-03-06T19:02:37.491 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:37.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:37.497310+0000) 2024-03-06T19:02:38.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:38.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:38.497524+0000) 2024-03-06T19:02:39.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:39.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:39.497750+0000) 2024-03-06T19:02:40.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:40.496+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:40.497979+0000) 2024-03-06T19:02:41.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:41.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:41.498193+0000) 2024-03-06T19:02:42.492 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:42.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:42.498390+0000) 2024-03-06T19:02:43.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:43.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:43.498609+0000) 2024-03-06T19:02:44.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:44.497+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:44.498821+0000) 2024-03-06T19:02:45.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:45.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:45.499087+0000) 2024-03-06T19:02:46.493 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:46.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:46.499283+0000) 2024-03-06T19:02:47.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:47.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:47.499502+0000) 2024-03-06T19:02:48.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:48.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:48.499733+0000) 2024-03-06T19:02:49.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:49.498+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:49.499948+0000) 2024-03-06T19:02:50.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:50.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:50.500167+0000) 2024-03-06T19:02:51.494 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:51.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:51.500361+0000) 2024-03-06T19:02:52.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:52.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:52.500508+0000) 2024-03-06T19:02:53.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:53.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:53.500733+0000) 2024-03-06T19:02:54.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:54.499+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:54.500921+0000) 2024-03-06T19:02:55.495 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:55.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:55.501143+0000) 2024-03-06T19:02:56.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:56.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:56.501337+0000) 2024-03-06T19:02:57.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:57.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:57.501520+0000) 2024-03-06T19:02:58.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:58.500+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:58.501767+0000) 2024-03-06T19:02:59.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:02:59.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:02:59.502044+0000) 2024-03-06T19:03:00.496 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:00.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:00.502236+0000) 2024-03-06T19:03:01.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:01.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:01.502550+0000) 2024-03-06T19:03:02.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:02.501+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:02.502773+0000) 2024-03-06T19:03:03.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:03.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:03.502997+0000) 2024-03-06T19:03:04.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:04.424 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:04.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:04.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:04.503248+0000) 2024-03-06T19:03:05.497 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:05.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:05.503434+0000) 2024-03-06T19:03:06.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:06.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:06.503669+0000) 2024-03-06T19:03:07.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:07.502+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:07.503893+0000) 2024-03-06T19:03:08.498 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:08.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:08.504135+0000) 2024-03-06T19:03:09.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:09.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:09.504359+0000) 2024-03-06T19:03:10.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:10.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:10.504600+0000) 2024-03-06T19:03:11.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:11.503+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:11.504824+0000) 2024-03-06T19:03:12.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:12.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:12.505095+0000) 2024-03-06T19:03:13.499 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:13.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:13.505329+0000) 2024-03-06T19:03:14.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:14.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:14.505516+0000) 2024-03-06T19:03:15.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:15.504+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:15.505748+0000) 2024-03-06T19:03:16.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:16.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:16.505991+0000) 2024-03-06T19:03:17.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:17.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:17.506264+0000) 2024-03-06T19:03:18.500 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:18.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:18.506524+0000) 2024-03-06T19:03:19.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:19.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:19.506730+0000) 2024-03-06T19:03:20.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:20.505+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:20.506971+0000) 2024-03-06T19:03:21.501 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:21.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:21.507213+0000) 2024-03-06T19:03:22.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:22.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:22.507431+0000) 2024-03-06T19:03:23.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:23.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:23.507650+0000) 2024-03-06T19:03:24.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:24.506+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:24.507879+0000) 2024-03-06T19:03:25.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:25.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:25.508156+0000) 2024-03-06T19:03:26.502 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:26.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:26.508392+0000) 2024-03-06T19:03:27.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:27.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:27.508592+0000) 2024-03-06T19:03:28.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:28.507+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:28.508848+0000) 2024-03-06T19:03:29.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:29.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:29.509110+0000) 2024-03-06T19:03:30.503 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:30.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:30.509356+0000) 2024-03-06T19:03:31.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:31.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:31.509581+0000) 2024-03-06T19:03:32.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:32.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:32.509769+0000) 2024-03-06T19:03:33.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:33.508+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:33.509985+0000) 2024-03-06T19:03:34.504 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:34.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:34.510289+0000) 2024-03-06T19:03:34.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:34.569 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:03:35.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:35.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:35.510546+0000) 2024-03-06T19:03:36.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:36.509+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:36.510779+0000) 2024-03-06T19:03:37.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:37.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:37.511040+0000) 2024-03-06T19:03:38.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:38.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:38.511263+0000) 2024-03-06T19:03:39.505 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:39.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:39.511465+0000) 2024-03-06T19:03:40.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:40.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:40.511673+0000) 2024-03-06T19:03:41.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:41.510+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:41.511899+0000) 2024-03-06T19:03:42.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:42.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:42.512157+0000) 2024-03-06T19:03:43.506 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:43.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:43.512345+0000) 2024-03-06T19:03:44.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:44.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:44.512556+0000) 2024-03-06T19:03:45.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:45.511+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:45.512799+0000) 2024-03-06T19:03:46.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:46.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:46.513069+0000) 2024-03-06T19:03:47.507 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:47.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:47.513289+0000) 2024-03-06T19:03:48.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:48.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:48.513532+0000) 2024-03-06T19:03:49.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:49.512+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:49.513859+0000) 2024-03-06T19:03:50.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:50.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:50.514133+0000) 2024-03-06T19:03:51.508 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:51.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:51.514328+0000) 2024-03-06T19:03:52.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:52.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:52.514556+0000) 2024-03-06T19:03:53.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:53.513+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:53.514781+0000) 2024-03-06T19:03:54.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:54.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:54.515046+0000) 2024-03-06T19:03:55.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:55.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:55.515267+0000) 2024-03-06T19:03:56.509 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:56.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:56.515474+0000) 2024-03-06T19:03:57.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:57.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:57.515685+0000) 2024-03-06T19:03:58.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:58.514+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:58.515931+0000) 2024-03-06T19:03:59.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:03:59.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:03:59.516160+0000) 2024-03-06T19:04:00.510 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:00.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:00.516381+0000) 2024-03-06T19:04:01.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:01.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:01.516591+0000) 2024-03-06T19:04:02.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:02.515+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:02.516815+0000) 2024-03-06T19:04:03.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:03.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:03.517063+0000) 2024-03-06T19:04:04.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:04.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:04.517257+0000) 2024-03-06T19:04:04.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:04.708 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:05.511 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:05.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:05.517510+0000) 2024-03-06T19:04:06.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:06.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:06.517755+0000) 2024-03-06T19:04:07.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:07.516+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:07.517976+0000) 2024-03-06T19:04:08.512 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:08.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:08.518203+0000) 2024-03-06T19:04:09.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:09.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:09.518392+0000) 2024-03-06T19:04:10.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:10.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:10.518671+0000) 2024-03-06T19:04:11.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:11.517+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:11.518949+0000) 2024-03-06T19:04:12.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:12.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:12.519221+0000) 2024-03-06T19:04:13.513 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:13.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:13.519454+0000) 2024-03-06T19:04:14.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:14.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:14.519684+0000) 2024-03-06T19:04:15.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:15.518+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:15.519877+0000) 2024-03-06T19:04:16.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:16.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:16.520136+0000) 2024-03-06T19:04:17.514 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:17.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:17.520357+0000) 2024-03-06T19:04:18.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:18.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:18.520612+0000) 2024-03-06T19:04:19.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:19.519+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:19.520855+0000) 2024-03-06T19:04:20.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:20.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:20.521166+0000) 2024-03-06T19:04:21.515 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:21.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:21.521427+0000) 2024-03-06T19:04:22.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:22.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:22.521683+0000) 2024-03-06T19:04:23.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:23.520+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:23.521924+0000) 2024-03-06T19:04:24.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:24.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:24.522167+0000) 2024-03-06T19:04:25.516 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:25.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:25.522397+0000) 2024-03-06T19:04:26.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:26.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:26.522611+0000) 2024-03-06T19:04:27.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:27.521+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:27.522804+0000) 2024-03-06T19:04:28.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:28.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:28.523061+0000) 2024-03-06T19:04:29.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:29.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:29.523302+0000) 2024-03-06T19:04:30.517 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:30.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:30.523497+0000) 2024-03-06T19:04:31.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:31.522+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:31.523718+0000) 2024-03-06T19:04:32.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:32.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:32.523990+0000) 2024-03-06T19:04:33.518 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:33.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:33.524268+0000) 2024-03-06T19:04:34.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:34.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:34.524452+0000) 2024-03-06T19:04:34.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:34.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:04:35.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:35.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:35.524697+0000) 2024-03-06T19:04:36.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:36.523+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:36.524946+0000) 2024-03-06T19:04:37.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:37.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:37.525237+0000) 2024-03-06T19:04:38.519 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:38.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:38.525437+0000) 2024-03-06T19:04:39.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:39.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:39.525629+0000) 2024-03-06T19:04:40.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:40.524+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:40.525857+0000) 2024-03-06T19:04:41.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:41.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:41.526112+0000) 2024-03-06T19:04:42.520 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:42.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:42.526313+0000) 2024-03-06T19:04:43.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:43.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:43.526520+0000) 2024-03-06T19:04:44.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:44.525+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:44.526807+0000) 2024-03-06T19:04:45.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:45.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:45.527069+0000) 2024-03-06T19:04:46.521 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:46.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:46.527352+0000) 2024-03-06T19:04:47.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:47.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:47.527631+0000) 2024-03-06T19:04:48.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:48.526+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:48.527839+0000) 2024-03-06T19:04:49.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:49.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:49.528119+0000) 2024-03-06T19:04:50.522 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:50.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:50.528361+0000) 2024-03-06T19:04:51.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:51.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:51.528604+0000) 2024-03-06T19:04:52.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:52.527+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:52.528844+0000) 2024-03-06T19:04:53.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:53.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:53.529129+0000) 2024-03-06T19:04:54.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:54.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:54.529326+0000) 2024-03-06T19:04:55.523 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:55.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:55.529530+0000) 2024-03-06T19:04:56.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:56.528+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:56.529770+0000) 2024-03-06T19:04:57.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:57.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:57.530063+0000) 2024-03-06T19:04:58.524 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:58.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:58.530311+0000) 2024-03-06T19:04:59.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:04:59.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:04:59.530560+0000) 2024-03-06T19:05:00.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:00.529+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:00.530824+0000) 2024-03-06T19:05:01.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:01.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:01.531110+0000) 2024-03-06T19:05:02.525 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:02.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:02.531320+0000) 2024-03-06T19:05:03.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:03.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:03.531556+0000) 2024-03-06T19:05:04.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:04.530+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:04.531779+0000) 2024-03-06T19:05:04.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:04.997 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:05.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:05.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:05.532029+0000) 2024-03-06T19:05:06.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:06.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:06.532224+0000) 2024-03-06T19:05:07.526 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:07.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:07.532444+0000) 2024-03-06T19:05:08.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:08.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:08.532688+0000) 2024-03-06T19:05:09.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:09.531+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:09.532929+0000) 2024-03-06T19:05:10.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:10.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:10.533186+0000) 2024-03-06T19:05:11.527 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:11.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:11.533462+0000) 2024-03-06T19:05:12.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:12.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:12.533674+0000) 2024-03-06T19:05:13.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:13.532+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:13.533894+0000) 2024-03-06T19:05:14.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:14.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:14.534167+0000) 2024-03-06T19:05:15.528 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:15.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:15.534399+0000) 2024-03-06T19:05:16.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:16.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:16.534665+0000) 2024-03-06T19:05:17.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:17.533+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:17.534894+0000) 2024-03-06T19:05:18.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:18.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:18.535140+0000) 2024-03-06T19:05:19.529 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:19.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:19.535405+0000) 2024-03-06T19:05:20.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:20.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:20.535736+0000) 2024-03-06T19:05:21.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:21.534+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:21.535976+0000) 2024-03-06T19:05:22.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:22.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:22.536233+0000) 2024-03-06T19:05:23.530 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:23.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:23.536488+0000) 2024-03-06T19:05:24.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:24.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:24.536747+0000) 2024-03-06T19:05:25.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:25.535+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:25.536985+0000) 2024-03-06T19:05:26.531 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:26.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:26.537252+0000) 2024-03-06T19:05:27.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:27.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:27.537492+0000) 2024-03-06T19:05:28.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:28.536+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:28.537760+0000) 2024-03-06T19:05:29.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:29.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:29.538095+0000) 2024-03-06T19:05:30.532 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:30.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:30.538371+0000) 2024-03-06T19:05:31.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:31.537+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:31.538656+0000) 2024-03-06T19:05:32.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:32.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:32.538992+0000) 2024-03-06T19:05:33.533 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:33.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:33.539330+0000) 2024-03-06T19:05:34.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:34.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:34.539586+0000) 2024-03-06T19:05:35.136 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:35.144 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:05:35.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:35.538+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:35.539825+0000) 2024-03-06T19:05:36.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:36.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:36.540155+0000) 2024-03-06T19:05:37.534 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:37.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:37.540323+0000) 2024-03-06T19:05:38.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:38.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:38.540558+0000) 2024-03-06T19:05:39.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:39.539+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:39.540783+0000) 2024-03-06T19:05:40.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:40.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:40.541057+0000) 2024-03-06T19:05:41.535 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:41.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:41.541319+0000) 2024-03-06T19:05:42.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:42.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:42.541597+0000) 2024-03-06T19:05:43.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:43.540+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:43.541801+0000) 2024-03-06T19:05:44.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:44.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:44.542061+0000) 2024-03-06T19:05:45.536 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:45.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:45.542288+0000) 2024-03-06T19:05:46.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:46.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:46.542526+0000) 2024-03-06T19:05:47.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:47.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:47.542728+0000) 2024-03-06T19:05:48.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:48.541+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:48.542924+0000) 2024-03-06T19:05:49.537 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:49.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:49.543192+0000) 2024-03-06T19:05:50.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:50.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:50.543443+0000) 2024-03-06T19:05:51.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:51.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:51.543709+0000) 2024-03-06T19:05:52.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:52.542+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:52.543987+0000) 2024-03-06T19:05:53.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:53.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:53.544267+0000) 2024-03-06T19:05:54.538 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:54.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:54.544538+0000) 2024-03-06T19:05:55.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:55.543+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:55.544806+0000) 2024-03-06T19:05:56.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:56.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:56.545132+0000) 2024-03-06T19:05:57.539 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:57.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:57.545376+0000) 2024-03-06T19:05:58.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:58.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:58.545642+0000) 2024-03-06T19:05:59.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:05:59.544+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:05:59.545907+0000) 2024-03-06T19:06:00.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:00.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:00.546201+0000) 2024-03-06T19:06:01.540 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:01.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:01.546450+0000) 2024-03-06T19:06:02.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:02.545+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:02.546717+0000) 2024-03-06T19:06:03.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:03.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:03.546997+0000) 2024-03-06T19:06:04.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:04.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:04.547265+0000) 2024-03-06T19:06:05.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:05.280 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:05.541 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:05.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:05.547511+0000) 2024-03-06T19:06:06.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:06.546+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:06.547768+0000) 2024-03-06T19:06:07.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:07.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:07.547994+0000) 2024-03-06T19:06:08.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:08.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:08.548278+0000) 2024-03-06T19:06:09.542 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:09.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:09.548500+0000) 2024-03-06T19:06:10.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:10.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:10.548703+0000) 2024-03-06T19:06:11.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:11.547+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:11.548939+0000) 2024-03-06T19:06:12.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:12.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:12.549181+0000) 2024-03-06T19:06:13.543 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:13.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:13.549438+0000) 2024-03-06T19:06:14.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:14.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:14.549673+0000) 2024-03-06T19:06:15.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:15.548+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:15.549901+0000) 2024-03-06T19:06:16.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:16.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:16.550136+0000) 2024-03-06T19:06:17.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:17.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:17.550359+0000) 2024-03-06T19:06:18.544 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:18.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:18.550641+0000) 2024-03-06T19:06:19.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:19.549+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:19.550875+0000) 2024-03-06T19:06:20.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:20.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:20.551144+0000) 2024-03-06T19:06:21.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:21.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:21.551351+0000) 2024-03-06T19:06:22.545 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:22.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:22.551556+0000) 2024-03-06T19:06:23.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:23.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:23.551727+0000) 2024-03-06T19:06:24.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:24.550+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:24.551987+0000) 2024-03-06T19:06:25.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:25.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:25.552205+0000) 2024-03-06T19:06:26.546 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:26.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:26.552403+0000) 2024-03-06T19:06:27.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:27.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:27.552657+0000) 2024-03-06T19:06:28.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:28.551+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:28.552944+0000) 2024-03-06T19:06:29.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:29.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:29.553184+0000) 2024-03-06T19:06:30.547 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:30.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:30.553427+0000) 2024-03-06T19:06:31.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:31.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:31.553688+0000) 2024-03-06T19:06:32.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:32.552+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:32.553971+0000) 2024-03-06T19:06:33.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:33.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:33.554224+0000) 2024-03-06T19:06:34.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:34.553+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:34.554469+0000) 2024-03-06T19:06:35.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:35.417 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:06:35.548 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:35.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:35.554742+0000) 2024-03-06T19:06:36.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:36.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:36.555063+0000) 2024-03-06T19:06:37.549 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:37.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:37.555364+0000) 2024-03-06T19:06:38.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:38.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:38.555632+0000) 2024-03-06T19:06:39.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:39.554+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:39.555898+0000) 2024-03-06T19:06:40.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:40.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:40.556167+0000) 2024-03-06T19:06:41.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:41.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:41.556437+0000) 2024-03-06T19:06:42.550 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:42.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:42.556647+0000) 2024-03-06T19:06:43.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:43.555+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:43.556865+0000) 2024-03-06T19:06:44.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:44.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:44.557114+0000) 2024-03-06T19:06:45.551 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:45.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:45.557331+0000) 2024-03-06T19:06:46.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:46.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:46.557540+0000) 2024-03-06T19:06:47.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:47.556+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:47.557753+0000) 2024-03-06T19:06:48.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:48.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:48.558056+0000) 2024-03-06T19:06:49.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:49.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:49.558307+0000) 2024-03-06T19:06:50.552 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:50.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:50.558530+0000) 2024-03-06T19:06:51.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:51.557+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:51.558759+0000) 2024-03-06T19:06:52.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:52.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:52.558993+0000) 2024-03-06T19:06:53.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:53.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:53.559249+0000) 2024-03-06T19:06:54.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:54.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:54.559468+0000) 2024-03-06T19:06:55.553 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:55.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:55.559684+0000) 2024-03-06T19:06:56.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:56.558+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:56.559921+0000) 2024-03-06T19:06:57.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:57.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:57.560133+0000) 2024-03-06T19:06:58.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:58.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:58.560363+0000) 2024-03-06T19:06:59.554 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:06:59.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:06:59.560596+0000) 2024-03-06T19:07:00.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:00.559+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:00.560848+0000) 2024-03-06T19:07:01.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:01.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:01.561112+0000) 2024-03-06T19:07:02.555 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:02.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:02.561341+0000) 2024-03-06T19:07:03.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:03.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:03.561552+0000) 2024-03-06T19:07:04.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:04.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:04.561732+0000) 2024-03-06T19:07:05.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:05.560+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:05.561921+0000) 2024-03-06T19:07:05.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:05.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:06.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:06.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:06.562206+0000) 2024-03-06T19:07:07.556 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:07.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:07.562402+0000) 2024-03-06T19:07:08.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:08.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:08.562692+0000) 2024-03-06T19:07:09.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:09.561+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:09.562971+0000) 2024-03-06T19:07:10.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:10.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:10.563261+0000) 2024-03-06T19:07:11.557 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:11.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:11.563517+0000) 2024-03-06T19:07:12.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:12.562+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:12.563794+0000) 2024-03-06T19:07:13.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:13.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:13.564099+0000) 2024-03-06T19:07:14.558 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:14.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:14.564382+0000) 2024-03-06T19:07:15.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:15.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:15.564614+0000) 2024-03-06T19:07:16.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:16.563+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:16.564888+0000) 2024-03-06T19:07:17.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:17.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:17.565177+0000) 2024-03-06T19:07:18.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:18.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:18.565443+0000) 2024-03-06T19:07:19.559 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:19.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:19.565679+0000) 2024-03-06T19:07:20.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:20.564+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:20.565899+0000) 2024-03-06T19:07:21.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:21.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:21.566178+0000) 2024-03-06T19:07:22.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:22.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:22.566431+0000) 2024-03-06T19:07:23.560 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:23.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:23.566654+0000) 2024-03-06T19:07:24.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:24.565+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:24.566879+0000) 2024-03-06T19:07:25.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:25.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:25.567132+0000) 2024-03-06T19:07:26.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:26.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:26.567344+0000) 2024-03-06T19:07:27.561 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:27.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:27.567566+0000) 2024-03-06T19:07:28.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:28.566+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:28.567815+0000) 2024-03-06T19:07:29.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:29.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:29.568098+0000) 2024-03-06T19:07:30.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:30.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:30.568287+0000) 2024-03-06T19:07:31.562 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:31.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:31.568500+0000) 2024-03-06T19:07:32.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:32.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:32.568737+0000) 2024-03-06T19:07:33.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:33.567+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:33.568957+0000) 2024-03-06T19:07:34.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:34.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:34.569171+0000) 2024-03-06T19:07:35.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:35.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:35.569366+0000) 2024-03-06T19:07:35.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:35.767 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:07:36.563 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:36.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:36.569585+0000) 2024-03-06T19:07:37.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:37.568+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:37.569809+0000) 2024-03-06T19:07:38.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:38.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:38.570073+0000) 2024-03-06T19:07:39.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:39.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:39.570264+0000) 2024-03-06T19:07:40.564 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:40.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:40.570474+0000) 2024-03-06T19:07:41.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:41.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:41.570691+0000) 2024-03-06T19:07:42.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:42.569+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:42.570917+0000) 2024-03-06T19:07:43.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:43.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:43.571141+0000) 2024-03-06T19:07:44.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:44.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:44.571336+0000) 2024-03-06T19:07:45.565 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:45.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:45.571546+0000) 2024-03-06T19:07:46.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:46.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:46.571770+0000) 2024-03-06T19:07:47.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:47.570+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:47.571990+0000) 2024-03-06T19:07:48.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:48.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:48.572224+0000) 2024-03-06T19:07:49.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:49.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:49.572451+0000) 2024-03-06T19:07:50.566 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:50.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:50.572660+0000) 2024-03-06T19:07:51.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:51.571+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:51.572894+0000) 2024-03-06T19:07:52.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:52.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:52.573131+0000) 2024-03-06T19:07:53.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:53.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:53.573323+0000) 2024-03-06T19:07:54.567 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:54.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:54.573535+0000) 2024-03-06T19:07:55.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:55.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:55.573759+0000) 2024-03-06T19:07:56.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:56.572+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:56.573939+0000) 2024-03-06T19:07:57.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:57.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:57.574166+0000) 2024-03-06T19:07:58.568 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:58.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:58.574367+0000) 2024-03-06T19:07:59.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:07:59.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:07:59.574567+0000) 2024-03-06T19:08:00.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:00.573+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:00.574847+0000) 2024-03-06T19:08:01.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:01.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:01.575104+0000) 2024-03-06T19:08:02.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:02.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:02.575345+0000) 2024-03-06T19:08:03.569 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:03.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:03.575608+0000) 2024-03-06T19:08:04.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:04.574+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:04.575851+0000) 2024-03-06T19:08:05.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:05.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:05.576119+0000) 2024-03-06T19:08:05.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:05.972 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:06.570 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:06.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:06.576343+0000) 2024-03-06T19:08:07.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:07.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:07.576564+0000) 2024-03-06T19:08:08.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:08.575+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:08.576802+0000) 2024-03-06T19:08:09.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:09.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:09.577053+0000) 2024-03-06T19:08:10.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:10.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:10.577255+0000) 2024-03-06T19:08:11.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:11.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:11.577480+0000) 2024-03-06T19:08:12.571 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:12.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:12.577720+0000) 2024-03-06T19:08:13.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:13.576+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:13.577923+0000) 2024-03-06T19:08:14.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:14.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:14.578163+0000) 2024-03-06T19:08:15.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:15.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:15.578376+0000) 2024-03-06T19:08:16.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:16.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:16.578628+0000) 2024-03-06T19:08:17.572 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:17.577+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:17.578869+0000) 2024-03-06T19:08:18.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:18.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:18.579127+0000) 2024-03-06T19:08:19.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:19.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:19.579364+0000) 2024-03-06T19:08:20.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:20.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:20.579563+0000) 2024-03-06T19:08:21.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:21.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:21.579754+0000) 2024-03-06T19:08:22.573 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:22.578+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:22.579967+0000) 2024-03-06T19:08:23.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:23.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:23.580152+0000) 2024-03-06T19:08:24.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:24.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:24.580351+0000) 2024-03-06T19:08:25.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:25.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:25.580555+0000) 2024-03-06T19:08:26.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:26.579+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:26.580791+0000) 2024-03-06T19:08:27.574 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:27.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:27.581028+0000) 2024-03-06T19:08:28.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:28.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:28.581267+0000) 2024-03-06T19:08:29.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:29.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:29.581495+0000) 2024-03-06T19:08:30.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:30.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:30.581681+0000) 2024-03-06T19:08:31.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:31.580+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:31.581891+0000) 2024-03-06T19:08:32.575 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:32.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:32.582123+0000) 2024-03-06T19:08:33.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:33.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:33.582312+0000) 2024-03-06T19:08:34.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:34.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:34.582523+0000) 2024-03-06T19:08:35.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:35.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:35.582741+0000) 2024-03-06T19:08:36.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:36.183 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:08:36.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:36.581+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:36.582975+0000) 2024-03-06T19:08:37.576 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:37.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:37.583276+0000) 2024-03-06T19:08:38.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:38.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:38.583563+0000) 2024-03-06T19:08:39.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:39.582+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:39.583844+0000) 2024-03-06T19:08:40.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:40.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:40.584162+0000) 2024-03-06T19:08:41.577 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:41.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:41.584342+0000) 2024-03-06T19:08:42.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:42.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:42.584598+0000) 2024-03-06T19:08:43.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:43.583+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:43.584843+0000) 2024-03-06T19:08:44.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:44.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:44.585118+0000) 2024-03-06T19:08:45.578 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:45.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:45.585309+0000) 2024-03-06T19:08:46.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:46.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:46.585532+0000) 2024-03-06T19:08:47.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:47.584+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:47.585738+0000) 2024-03-06T19:08:48.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:48.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:48.586083+0000) 2024-03-06T19:08:49.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:49.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:49.586278+0000) 2024-03-06T19:08:50.579 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:50.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:50.586464+0000) 2024-03-06T19:08:51.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:51.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:51.586695+0000) 2024-03-06T19:08:52.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:52.585+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:52.586924+0000) 2024-03-06T19:08:53.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:53.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:53.587166+0000) 2024-03-06T19:08:54.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:54.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:54.587363+0000) 2024-03-06T19:08:55.580 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:55.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:55.587563+0000) 2024-03-06T19:08:56.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:56.586+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:56.587798+0000) 2024-03-06T19:08:57.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:57.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:57.588036+0000) 2024-03-06T19:08:58.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:58.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:58.588243+0000) 2024-03-06T19:08:59.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:08:59.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:08:59.588451+0000) 2024-03-06T19:09:00.581 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:00.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:00.588637+0000) 2024-03-06T19:09:01.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:01.587+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:01.588876+0000) 2024-03-06T19:09:02.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:02.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:02.589148+0000) 2024-03-06T19:09:03.582 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:03.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:03.589346+0000) 2024-03-06T19:09:04.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:04.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:04.589592+0000) 2024-03-06T19:09:05.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:05.588+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:05.589828+0000) 2024-03-06T19:09:06.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:06.321 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:06.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:06.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:06.590061+0000) 2024-03-06T19:09:07.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:07.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:07.590275+0000) 2024-03-06T19:09:08.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:08.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:08.590533+0000) 2024-03-06T19:09:09.583 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:09.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:09.590758+0000) 2024-03-06T19:09:10.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:10.589+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:10.590987+0000) 2024-03-06T19:09:11.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:11.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:11.591275+0000) 2024-03-06T19:09:12.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:12.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:12.591484+0000) 2024-03-06T19:09:13.584 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:13.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:13.591698+0000) 2024-03-06T19:09:14.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:14.590+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:14.591939+0000) 2024-03-06T19:09:15.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:15.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:15.592202+0000) 2024-03-06T19:09:16.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:16.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:16.592500+0000) 2024-03-06T19:09:17.585 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:17.591+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:17.592774+0000) 2024-03-06T19:09:18.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:18.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:18.593113+0000) 2024-03-06T19:09:19.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:19.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:19.593377+0000) 2024-03-06T19:09:20.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:20.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:20.593652+0000) 2024-03-06T19:09:21.586 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:21.592+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:21.593913+0000) 2024-03-06T19:09:22.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:22.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:22.594123+0000) 2024-03-06T19:09:23.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:23.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:23.594384+0000) 2024-03-06T19:09:24.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:24.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:24.594656+0000) 2024-03-06T19:09:25.587 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:25.593+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:25.594916+0000) 2024-03-06T19:09:26.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:26.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:26.595180+0000) 2024-03-06T19:09:27.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:27.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:27.595364+0000) 2024-03-06T19:09:28.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:28.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:28.595578+0000) 2024-03-06T19:09:29.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:29.594+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:29.595814+0000) 2024-03-06T19:09:30.588 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:30.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:30.596066+0000) 2024-03-06T19:09:31.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:31.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:31.596282+0000) 2024-03-06T19:09:32.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:32.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:32.596509+0000) 2024-03-06T19:09:33.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:33.595+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:33.596757+0000) 2024-03-06T19:09:34.589 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:34.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:34.597053+0000) 2024-03-06T19:09:35.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:35.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:35.597309+0000) 2024-03-06T19:09:36.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:36.458 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:09:36.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:36.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:36.597535+0000) 2024-03-06T19:09:37.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:37.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:37.597776+0000) 2024-03-06T19:09:38.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:38.596+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:38.597973+0000) 2024-03-06T19:09:39.590 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:39.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:39.598172+0000) 2024-03-06T19:09:40.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:40.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:40.598401+0000) 2024-03-06T19:09:41.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:41.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:41.598630+0000) 2024-03-06T19:09:42.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:42.597+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:42.598876+0000) 2024-03-06T19:09:43.591 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:43.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:43.599112+0000) 2024-03-06T19:09:44.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:44.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:44.599336+0000) 2024-03-06T19:09:45.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:45.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:45.599556+0000) 2024-03-06T19:09:46.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:46.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:46.599783+0000) 2024-03-06T19:09:47.592 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:47.598+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:47.599988+0000) 2024-03-06T19:09:48.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:48.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:48.600208+0000) 2024-03-06T19:09:49.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:49.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:49.600399+0000) 2024-03-06T19:09:50.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:50.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:50.600625+0000) 2024-03-06T19:09:51.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:51.599+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:51.600861+0000) 2024-03-06T19:09:52.593 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:52.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:52.601134+0000) 2024-03-06T19:09:53.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:53.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:53.601350+0000) 2024-03-06T19:09:54.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:54.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:54.601586+0000) 2024-03-06T19:09:55.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:55.600+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:55.601805+0000) 2024-03-06T19:09:56.594 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:56.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:56.602049+0000) 2024-03-06T19:09:57.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:57.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:57.602290+0000) 2024-03-06T19:09:58.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:58.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:58.602534+0000) 2024-03-06T19:09:59.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:09:59.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:09:59.602750+0000) 2024-03-06T19:10:00.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:00.601+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:00.602933+0000) 2024-03-06T19:10:01.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:01.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:01.603167+0000) 2024-03-06T19:10:02.595 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:02.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:02.603364+0000) 2024-03-06T19:10:03.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:03.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:03.603570+0000) 2024-03-06T19:10:04.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:04.602+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:04.603801+0000) 2024-03-06T19:10:05.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:05.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:05.604062+0000) 2024-03-06T19:10:06.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:06.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:06.604267+0000) 2024-03-06T19:10:06.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:06.669 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:07.596 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:07.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:07.604482+0000) 2024-03-06T19:10:08.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:08.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:08.604720+0000) 2024-03-06T19:10:09.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:09.603+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:09.604912+0000) 2024-03-06T19:10:10.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:10.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:10.605139+0000) 2024-03-06T19:10:11.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:11.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:11.605396+0000) 2024-03-06T19:10:12.597 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:12.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:12.605641+0000) 2024-03-06T19:10:13.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:13.604+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:13.605876+0000) 2024-03-06T19:10:14.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:14.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:14.606145+0000) 2024-03-06T19:10:15.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:15.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:15.606374+0000) 2024-03-06T19:10:16.598 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:16.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:16.606619+0000) 2024-03-06T19:10:17.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:17.605+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:17.606869+0000) 2024-03-06T19:10:18.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:18.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:18.607142+0000) 2024-03-06T19:10:19.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:19.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:19.607390+0000) 2024-03-06T19:10:20.599 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:20.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:20.607632+0000) 2024-03-06T19:10:21.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:21.606+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:21.607861+0000) 2024-03-06T19:10:22.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:22.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:22.608111+0000) 2024-03-06T19:10:23.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:23.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:23.608353+0000) 2024-03-06T19:10:24.600 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:24.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:24.608581+0000) 2024-03-06T19:10:25.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:25.607+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:25.608802+0000) 2024-03-06T19:10:26.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:26.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:26.609062+0000) 2024-03-06T19:10:27.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:27.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:27.609297+0000) 2024-03-06T19:10:28.601 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:28.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:28.609543+0000) 2024-03-06T19:10:29.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:29.608+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:29.609816+0000) 2024-03-06T19:10:30.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:30.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:30.610057+0000) 2024-03-06T19:10:31.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:31.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:31.610294+0000) 2024-03-06T19:10:32.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:32.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:32.610512+0000) 2024-03-06T19:10:33.602 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:33.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:33.610703+0000) 2024-03-06T19:10:34.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:34.609+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:34.610932+0000) 2024-03-06T19:10:35.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:35.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:35.611165+0000) 2024-03-06T19:10:36.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:36.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:36.611396+0000) 2024-03-06T19:10:36.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:36.806 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:10:37.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:37.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:37.611616+0000) 2024-03-06T19:10:38.603 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:38.610+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:38.611863+0000) 2024-03-06T19:10:39.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:39.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:39.612112+0000) 2024-03-06T19:10:40.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:40.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:40.612359+0000) 2024-03-06T19:10:41.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:41.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:41.612596+0000) 2024-03-06T19:10:42.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:42.611+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:42.612845+0000) 2024-03-06T19:10:43.604 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:43.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:43.613128+0000) 2024-03-06T19:10:44.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:44.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:44.613372+0000) 2024-03-06T19:10:45.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:45.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:45.613612+0000) 2024-03-06T19:10:46.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:46.612+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:46.613855+0000) 2024-03-06T19:10:47.605 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:47.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:47.614135+0000) 2024-03-06T19:10:48.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:48.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:48.614365+0000) 2024-03-06T19:10:49.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:49.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:49.614579+0000) 2024-03-06T19:10:50.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:50.613+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:50.614823+0000) 2024-03-06T19:10:51.606 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:51.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:51.615082+0000) 2024-03-06T19:10:52.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:52.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:52.615318+0000) 2024-03-06T19:10:53.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:53.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:53.615530+0000) 2024-03-06T19:10:54.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:54.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:54.615760+0000) 2024-03-06T19:10:55.607 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:55.614+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:55.615979+0000) 2024-03-06T19:10:56.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:56.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:56.616262+0000) 2024-03-06T19:10:57.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:57.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:57.616508+0000) 2024-03-06T19:10:58.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:58.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:58.616768+0000) 2024-03-06T19:10:59.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:10:59.615+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:10:59.616968+0000) 2024-03-06T19:11:00.608 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:00.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:00.617168+0000) 2024-03-06T19:11:01.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:01.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:01.617398+0000) 2024-03-06T19:11:02.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:02.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:02.617662+0000) 2024-03-06T19:11:03.609 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:03.616+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:03.617946+0000) 2024-03-06T19:11:04.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:04.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:04.618185+0000) 2024-03-06T19:11:05.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:05.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:05.618394+0000) 2024-03-06T19:11:06.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:06.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:06.618660+0000) 2024-03-06T19:11:06.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:06.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:07.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:07.617+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:07.618947+0000) 2024-03-06T19:11:08.610 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:08.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:08.619221+0000) 2024-03-06T19:11:09.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:09.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:09.619455+0000) 2024-03-06T19:11:10.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:10.618+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:10.619737+0000) 2024-03-06T19:11:11.611 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:11.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:11.620048+0000) 2024-03-06T19:11:12.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:12.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:12.620308+0000) 2024-03-06T19:11:13.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:13.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:13.620575+0000) 2024-03-06T19:11:14.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:14.619+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:14.620813+0000) 2024-03-06T19:11:15.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:15.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:15.621105+0000) 2024-03-06T19:11:16.612 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:16.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:16.621355+0000) 2024-03-06T19:11:17.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:17.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:17.621665+0000) 2024-03-06T19:11:18.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:18.620+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:18.621954+0000) 2024-03-06T19:11:19.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:19.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:19.622209+0000) 2024-03-06T19:11:20.613 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:20.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:20.622484+0000) 2024-03-06T19:11:21.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:21.621+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:21.622770+0000) 2024-03-06T19:11:22.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:22.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:22.623077+0000) 2024-03-06T19:11:23.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:23.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:23.623304+0000) 2024-03-06T19:11:24.614 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:24.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:24.623510+0000) 2024-03-06T19:11:25.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:25.622+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:25.623752+0000) 2024-03-06T19:11:26.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:26.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:26.623999+0000) 2024-03-06T19:11:27.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:27.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:27.624288+0000) 2024-03-06T19:11:28.615 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:28.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:28.624549+0000) 2024-03-06T19:11:29.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:29.623+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:29.624777+0000) 2024-03-06T19:11:30.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:30.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:30.625049+0000) 2024-03-06T19:11:31.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:31.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:31.625279+0000) 2024-03-06T19:11:32.616 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:32.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:32.625548+0000) 2024-03-06T19:11:33.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:33.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:33.625731+0000) 2024-03-06T19:11:34.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:34.624+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:34.625928+0000) 2024-03-06T19:11:35.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:35.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:35.626157+0000) 2024-03-06T19:11:36.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:36.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:36.626370+0000) 2024-03-06T19:11:37.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:37.118 DEBUG:teuthology.orchestra.run.smithi178:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T19:11:37.617 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:37.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:37.626582+0000) 2024-03-06T19:11:38.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:38.625+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:38.626821+0000) 2024-03-06T19:11:39.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:39.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:39.627097+0000) 2024-03-06T19:11:40.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:40.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:40.627287+0000) 2024-03-06T19:11:41.618 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:41.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:41.627533+0000) 2024-03-06T19:11:42.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:42.626+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:42.627785+0000) 2024-03-06T19:11:43.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:43.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:43.628046+0000) 2024-03-06T19:11:44.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:44.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:44.628272+0000) 2024-03-06T19:11:45.619 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:45.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:45.628509+0000) 2024-03-06T19:11:46.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:46.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:46.628762+0000) 2024-03-06T19:11:47.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:47.627+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:47.628972+0000) 2024-03-06T19:11:48.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:48.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:48.629214+0000) 2024-03-06T19:11:49.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:49.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:49.629411+0000) 2024-03-06T19:11:50.620 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:50.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:50.629640+0000) 2024-03-06T19:11:51.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:51.628+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:51.629878+0000) 2024-03-06T19:11:52.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:52.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:52.630160+0000) 2024-03-06T19:11:53.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:53.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:53.630397+0000) 2024-03-06T19:11:54.621 INFO:tasks.ceph.mds.c.smithi006.stderr:2024-03-06T19:11:54.629+0000 7f6bf0600640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-03-06T18:11:54.630655+0000) 2024-03-06T19:11:55.507 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-03-06T19:11:55.507 DEBUG:teuthology.task.console_log:Killing console logger for smithi006 2024-03-06T19:11:55.508 DEBUG:teuthology.task.console_log:Killing console logger for smithi178 2024-03-06T19:11:55.508 DEBUG:teuthology.exit:Finished running handlers